AI Trajectory

Trajectory: (Expand/Collapse All)
Seq	Timestamp	Type	Name	Duration
0/0	2026/10/19 19:57	flow	patch-triage	2m Args: null Results: { "EnableConfigs": [ "PAGE_RCU_FREE" ], "FocusSymbols": [ "filemap_read_rcu", "filemap_read_rcu_batch", "filemap_read_accessed", "filemap_mark_hot", "rcu_free_defer", "rcu_free_workfn", "rcu_free_expedite_workfn", "rcu_free_drain", "rcu_free_sync", "rcu_free_wait" ], "KMSANReasoning": "The patch series implements lockless page cache reads using RCU (CONFIG_PAGE_RCU_FREE) and defers the physical freeing of hot folios until after an RCU grace period.\n\nFrom a memory safety and sanitizer perspective:\n1. No uninitialized memory is exposed: All local variables and data structures (such as struct rcu_free_pcpu, list heads, and loop variables) are properly initialized before use.\n2. Kernel-to-user copies: The fast path in `filemap_read_rcu_copy()` copies file data from uptodate folios bounded strictly by the file's `i_size` and page boundaries, identically to the existing slow-path page cache read mechanisms; no uninitialized kernel structure fields or padding are leaked to user space.\n3. KMSAN hooks: The patch properly defers `kmsan_free_page()` in `__free_pages_prepare()` until the folio is actually released after the RCU grace period, ensuring KMSAN page shadow tracking remains accurate.\n4. Risk surface: The primary risks associated with this lockless RCU page cache reader patch are concurrency races (e.g. during folio splitting, truncation, migration, or hwpoison), potential use-after-free bugs, reference-counting anomalies, and locking/deadlock issues. These failure modes are covered by KASAN, LOCKDEP, and KCSAN.\n\nBecause the changes do not introduce or expose uninitialized memory reads or kernel information leaks, a dedicated KMSAN fuzzing session is not warranted.", "NeedsKMSAN": false, "Reasoning": "The patch introduces lockless page cache reads and deferred RCU page freeing (PAGE_RCU_FREE). Readers can copy from hot page cache folios under RCU without taking a folio reference count, while page freeing defers memory recycling until an RCU grace period completes. This touches core MM logic (page allocator free paths, slowpath reclaim retry, memory failure, and page cache read paths in filemap and shmem) which is reachable through standard user-space file reading operations on 64-bit systems.", "WorthFuzzing": true }
1/1	2026/10/19 19:57	action	read-patch-diff	0m Args: null Results: { "PatchDiff": "commit dddf59c0f0df6bc96619a328ac462a9dfff93355\nAuthor: syz-cluster \u003ctriage@syzkaller.com\u003e\nDate: Mon Oct 19 19:57:32 2026 +0000\n\n syz-cluster: applied patch under review\n\ndiff --git a/fs/fuse/dev.c b/fs/fuse/dev.c\nindex 4fec31fc0b845..526929c7b3f1a 100644\n--- a/fs/fuse/dev.c\n+++ b/fs/fuse/dev.c\n@@ -1152,6 +1152,10 @@ static int fuse_try_move_folio(struct fuse_copy_state cs, struct folio foliop\n \n \tnewfolio = page_folio(buf-\u003epage);\n \n+\t/ A lockless page cache read may still be copying from it /\n+\tif (folio_test_rcu_free(newfolio))\n+\t\tgoto out_fallback_unlock;\n+\n \tfolio_clear_uptodate(newfolio);\n \tfolio_clear_mappedtodisk(newfolio);\n \ndiff --git a/fs/splice.c b/fs/splice.c\nindex 9d8f63e2fd1ab..0133c2348da03 100644\n--- a/fs/splice.c\n+++ b/fs/splice.c\n@@ -55,56 +55,6 @@ static noinline void pipe_clear_nowait(struct file file)\n \t} while (!try_cmpxchg(\u0026file-\u003ef_mode, \u0026fmode, fmode \u0026 ~FMODE_NOWAIT));\n }\n \n-/\n- Attempt to steal a page from a pipe buffer. This should perhaps go into\n- * a vm helper function, it's already simplified quite a bit by the\n- * addition of remove_mapping(). If success is returned, the caller may\n- * attempt to reuse this page for another destination.\n- /\n-static bool page_cache_pipe_buf_try_steal(struct pipe_inode_info pipe,\n-\t\tstruct pipe_buffer buf)\n-{\n-\tstruct folio folio = page_folio(buf-\u003epage);\n-\tstruct address_space mapping;\n-\n-\tfolio_lock(folio);\n-\n-\tmapping = folio_mapping(folio);\n-\tif (mapping) {\n-\t\tWARN_ON(!folio_test_uptodate(folio));\n-\n-\t\t/\n-\t\t * At least for ext2 with nobh option, we need to wait on\n-\t\t * writeback completing on this folio, since we'll remove it\n-\t\t * from the pagecache. Otherwise truncate wont wait on the\n-\t\t * folio, allowing the disk blocks to be reused by someone else\n-\t\t * before we actually wrote our data to them. fs corruption\n-\t\t * ensues.\n-\t\t /\n-\t\tfolio_wait_writeback(folio);\n-\n-\t\tif (!filemap_release_folio(folio, GFP_KERNEL))\n-\t\t\tgoto out_unlock;\n-\n-\t\t/\n-\t\t * If we succeeded in removing the mapping, set LRU flag\n-\t\t * and return good.\n-\t\t /\n-\t\tif (remove_mapping(mapping, folio)) {\n-\t\t\tbuf-\u003eflags \|= PIPE_BUF_FLAG_LRU;\n-\t\t\treturn true;\n-\t\t}\n-\t}\n-\n-\t/\n-\t * Raced with truncate or failed to remove folio from current\n-\t * address space, unlock and return failure.\n-\t /\n-out_unlock:\n-\tfolio_unlock(folio);\n-\treturn false;\n-}\n-\n static void page_cache_pipe_buf_release(struct pipe_inode_info pipe,\n \t\t\t\t\tstruct pipe_buffer buf)\n {\n@@ -155,7 +105,6 @@ static int page_cache_pipe_buf_confirm(struct pipe_inode_info pipe,\n const struct pipe_buf_operations page_cache_pipe_buf_ops = {\n \t.confirm\t= page_cache_pipe_buf_confirm,\n \t.release\t= page_cache_pipe_buf_release,\n-\t.try_steal\t= page_cache_pipe_buf_try_steal,\n \t.get\t\t= generic_pipe_buf_get,\n };\n \ndiff --git a/include/linux/mmzone.h b/include/linux/mmzone.h\nindex 94f9c3ff54160..3336b9e5ee999 100644\n--- a/include/linux/mmzone.h\n+++ b/include/linux/mmzone.h\n@@ -228,6 +228,7 @@ enum zone_stat_item {\n \tNR_ZSPAGES,\t\t/* allocated in zsmalloc /\n #endif\n \tNR_FREE_CMA_PAGES,\n+\tNR_FREE_RCU_PAGES,\t/ freed, waiting for an RCU grace period /\n #ifdef CONFIG_UNACCEPTED_MEMORY\n \tNR_UNACCEPTED,\n #endif\ndiff --git a/include/linux/page-flags.h b/include/linux/page-flags.h\nindex 7a863572adce7..4916b5713a74b 100644\n--- a/include/linux/page-flags.h\n+++ b/include/linux/page-flags.h\n@@ -111,6 +111,9 @@ enum pageflags {\n \tPG_swapbacked,\t\t/ Page is backed by RAM/swap /\n \tPG_unevictable,\t\t/ Page is \"unevictable\" /\n \tPG_dropbehind,\t\t/ drop pages on IO completion /\n+#ifdef CONFIG_PAGE_RCU_FREE\n+\tPG_rcu_free,\t\t/ free only after an RCU grace period /\n+#endif\n #ifdef CONFIG_MMU\n \tPG_mlocked,\t\t/ Page is vma mlocked /\n #endif\n@@ -602,6 +605,27 @@ FOLIO_FLAG(dropbehind, FOLIO_HEAD_PAGE)\n \tFOLIO_TEST_CLEAR_FLAG(dropbehind, FOLIO_HEAD_PAGE)\n \t__FOLIO_SET_FLAG(dropbehind, FOLIO_HEAD_PAGE)\n \n+/\n+ * A lockless lookup that sees PG_rcu_free and then still finds the folio\n+ * where it looked may read it until rcu_read_unlock(). Set under a\n+ * reference. Cleared by the free, or when the folio's inode is evicted:\n+ * no lockless lookup can reach it then. Not for hugetlb or\n+ * free_pages_nolock().\n+ /\n+#ifdef CONFIG_PAGE_RCU_FREE\n+FOLIO_TEST_FLAG(rcu_free, FOLIO_HEAD_PAGE)\n+\tFOLIO_TEST_SET_FLAG(rcu_free, FOLIO_HEAD_PAGE)\n+\tFOLIO_CLEAR_FLAG(rcu_free, FOLIO_HEAD_PAGE)\n+\t__FOLIO_CLEAR_FLAG(rcu_free, FOLIO_HEAD_PAGE)\n+#define __PG_RCU_FREE (1UL \u003c\u003c PG_rcu_free)\n+#else\n+FOLIO_TEST_FLAG_FALSE(rcu_free)\n+\tFOLIO_TEST_SET_FLAG_FALSE(rcu_free)\n+\tFOLIO_CLEAR_FLAG_NOOP(rcu_free)\n+\t__FOLIO_CLEAR_FLAG_NOOP(rcu_free)\n+#define __PG_RCU_FREE 0\n+#endif\n+\n #ifdef CONFIG_HIGHMEM\n /\n * Must use a macro here due to header dependency issues. page_zone() is not\ndiff --git a/include/linux/pagemap.h b/include/linux/pagemap.h\nindex 0adfa6605653d..6c7f58803eee4 100644\n--- a/include/linux/pagemap.h\n+++ b/include/linux/pagemap.h\n@@ -210,6 +210,7 @@ enum mapping_flags {\n \tAS_WRITEBACK_MAY_DEADLOCK_ON_RECLAIM = 9,\n \tAS_KERNEL_FILE = 10,\t/* mapping for a fake kernel file that shouldn't\n \t\t\t\t account usage to user cgroups /\n+\tAS_RCU_READ = 11,\t/ a folio has had PG_rcu_free /\n \t/ Bits 16-25 are used for FOLIO_ORDER /\n \tAS_FOLIO_ORDER_BITS = 5,\n \tAS_FOLIO_ORDER_MIN = 16,\n@@ -269,6 +270,17 @@ static inline bool mapping_unevictable(const struct address_space mapping)\n \treturn mapping \u0026\u0026 test_bit(AS_UNEVICTABLE, \u0026mapping-\u003eflags);\n }\n \n+static inline void mapping_set_rcu_read(struct address_space mapping)\n+{\n+\tif (!test_bit(AS_RCU_READ, \u0026mapping-\u003eflags))\n+\t\tset_bit(AS_RCU_READ, \u0026mapping-\u003eflags);\n+}\n+\n+static inline bool mapping_rcu_read(const struct address_space mapping)\n+{\n+\treturn test_bit(AS_RCU_READ, \u0026mapping-\u003eflags);\n+}\n+\n static inline void mapping_set_exiting(struct address_space mapping)\n {\n \tset_bit(AS_EXITING, \u0026mapping-\u003eflags);\ndiff --git a/include/linux/swap.h b/include/linux/swap.h\nindex 5658a1634b85e..67aa5271ae6cb 100644\n--- a/include/linux/swap.h\n+++ b/include/linux/swap.h\n@@ -295,7 +295,7 @@ extern unsigned long totalreserve_pages;\n \n / linux/mm/folio.c /\n void folio_add_lru(struct folio folio);\n-void folio_mark_accessed(struct folio folio);\n+bool folio_mark_accessed(struct folio folio);\n void lru_add_drain_all(void);\n \n enum lru_cache_drained {\ndiff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h\nindex 2628ccda076a0..c1a63f070aa71 100644\n--- a/include/linux/vm_event_item.h\n+++ b/include/linux/vm_event_item.h\n@@ -36,6 +36,8 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,\n \t\tFOR_ALL_ZONES(ALLOCSTALL)\n \t\tFOR_ALL_ZONES(PGSCAN_SKIP)\n \t\tPGFREE, PGACTIVATE, PGDEACTIVATE, PGLAZYFREE,\n+\t\tPGFREE_RCU, PGFREE_RCU_EXPEDITE, PGFREE_RCU_DRAIN,\n+\t\tPGFREE_RCU_MARK,\n \t\tPGFAULT, PGMAJFAULT,\n \t\tPGLAZYFREED,\n \t\tPGREUSE,\ndiff --git a/include/trace/events/mmflags.h b/include/trace/events/mmflags.h\nindex 935893e5ea53b..47c3411a75e8f 100644\n--- a/include/trace/events/mmflags.h\n+++ b/include/trace/events/mmflags.h\n@@ -117,6 +117,12 @@ TRACE_DEFINE_ENUM(___GFP_LAST_BIT);\n #define IF_HAVE_PG_IDLE(_name)\n #endif\n \n+#ifdef CONFIG_PAGE_RCU_FREE\n+#define IF_HAVE_PG_RCU_FREE(_name) ,{1UL \u003c\u003c PG_##_name, __stringify(_name)}\n+#else\n+#define IF_HAVE_PG_RCU_FREE(_name)\n+#endif\n+\n #ifdef CONFIG_ARCH_USES_PG_ARCH_2\n #define IF_HAVE_PG_ARCH_2(_name) ,{1UL \u003c\u003c PG_##_name, __stringify(_name)}\n #else\n@@ -152,6 +158,7 @@ TRACE_DEFINE_ENUM(___GFP_LAST_BIT);\n \tDEF_PAGEFLAG_NAME(swapbacked),\t\t\t\t\t\\\n \tDEF_PAGEFLAG_NAME(unevictable),\t\t\t\t\t\\\n \tDEF_PAGEFLAG_NAME(dropbehind)\t\t\t\t\t\\\n+IF_HAVE_PG_RCU_FREE(rcu_free)\t\t\t\t\t\t\\\n IF_HAVE_PG_MLOCK(mlocked)\t\t\t\t\t\t\\\n IF_HAVE_PG_HWPOISON(hwpoison)\t\t\t\t\t\t\\\n IF_HAVE_PG_IDLE(idle)\t\t\t\t\t\t\t\\\ndiff --git a/mm/Kconfig b/mm/Kconfig\nindex 604c58199acbf..1ad1cb7969e79 100644\n--- a/mm/Kconfig\n+++ b/mm/Kconfig\n@@ -718,6 +718,21 @@ config PCP_BATCH_SCALE_MAX\n \t latency. This option sets the upper limit of scale factor to limit\n \t the maximum latency.\n \n+config PAGE_RCU_FREE\n+\tdef_bool 64BIT\n+\thelp\n+\t Reads from the page cache can copy from folios that are read\n+\t again and again without taking a reference on them, so small\n+\t reads of one file from many CPUs do not contend on the folio's\n+\t reference count. The cost is that memory freed from such\n+\t folios becomes reusable only after an RCU grace period.\n+\n+\t 64-bit only: it takes a page flag, and those are scarce on\n+\t 32-bit.\n+\n+\t Writing 0 to lockless_page_cache_read in debugfs turns these\n+\t reads off.\n+\n config PHYS_ADDR_T_64BIT\n \tdef_bool 64BIT\n \ndiff --git a/mm/filemap.c b/mm/filemap.c\nindex 00fd89cf6f550..7021ed2ab6c9c 100644\n--- a/mm/filemap.c\n+++ b/mm/filemap.c\n@@ -48,6 +48,7 @@\n #include \u003clinux/rcupdate_wait.h\u003e\n #include \u003clinux/sched/mm.h\u003e\n #include \u003clinux/sysctl.h\u003e\n+#include \u003clinux/debugfs.h\u003e\n #include \u003clinux/pgalloc.h\u003e\n \n #include \u003casm/tlbflush.h\u003e\n@@ -138,6 +139,9 @@ static void page_cache_delete(struct address_space mapping,\n \tnr = folio_nr_pages(folio);\n \n \tVM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);\n+\t/ No lockless reader can be in an evicted inode /\n+\tif (folio_test_rcu_free(folio) \u0026\u0026 mapping_exiting(mapping))\n+\t\tfolio_clear_rcu_free(folio);\n \n \txas_store(\u0026xas, shadow);\n \txas_init_marks(\u0026xas);\n@@ -306,6 +310,9 @@ static void page_cache_delete_batch(struct address_space mapping,\n \t\t}\n \n \t\tWARN_ON_ONCE(!folio_test_locked(folio));\n+\t\t/* No lockless reader can be in an evicted inode /\n+\t\tif (folio_test_rcu_free(folio) \u0026\u0026 mapping_exiting(mapping))\n+\t\t\tfolio_clear_rcu_free(folio);\n \n \t\tfolio-\u003emapping = NULL;\n \t\t/ Leave folio-\u003eindex set: truncation lookup relies on it /\n@@ -2759,6 +2766,101 @@ static inline bool pos_same_folio(loff_t pos1, loff_t pos2, struct folio folio)\n \treturn (pos1 \u003e\u003e shift == pos2 \u003e\u003e shift);\n }\n \n+#ifdef CONFIG_PAGE_RCU_FREE\n+/* Writing 0 to lockless_page_cache_read in debugfs turns the reads off /\n+static DEFINE_STATIC_KEY_TRUE(lockless_read_key);\n+\n+static bool lockless_read_enabled(void)\n+{\n+\treturn static_branch_likely(\u0026lockless_read_key);\n+}\n+\n+#ifdef CONFIG_DEBUG_FS\n+static int lockless_read_get(void data, u64 val)\n+{\n+\tval = static_key_enabled(\u0026lockless_read_key);\n+\treturn 0;\n+}\n+\n+static int lockless_read_set(void data, u64 val)\n+{\n+\tif (val \u003e 1)\n+\t\treturn -EINVAL;\n+\tif (val)\n+\t\tstatic_branch_enable(\u0026lockless_read_key);\n+\telse\n+\t\tstatic_branch_disable(\u0026lockless_read_key);\n+\treturn 0;\n+}\n+DEFINE_DEBUGFS_ATTRIBUTE(lockless_read_fops, lockless_read_get,\n+\t\t\t lockless_read_set, \"%llu\\n\");\n+\n+static int __init lockless_read_debugfs_init(void)\n+{\n+\tdebugfs_create_file_unsafe(\"lockless_page_cache_read\", 0644, NULL, NULL,\n+\t\t\t\t \u0026lockless_read_fops);\n+\treturn 0;\n+}\n+late_initcall(lockless_read_debugfs_init);\n+#endif\n+#else\n+static bool lockless_read_enabled(void)\n+{\n+\treturn false;\n+}\n+#endif\n+\n+/\n+ * Reads keep coming back to @folio: free it only after an RCU grace period\n+ * from now on. Not if its next read needs a reference anyway. Not if it\n+ * is unevictable: folio_mark_accessed() calls such a folio hot after at\n+ * most one access, so every ramfs folio that is read would get the flag.\n+ /\n+static void filemap_mark_hot(struct folio folio)\n+{\n+\tstruct address_space mapping;\n+\n+\tif (!lockless_read_enabled())\n+\t\treturn;\n+\tif (folio_test_rcu_free(folio) \|\| !folio_test_uptodate(folio) \|\|\n+\t folio_test_dropbehind(folio) \|\| folio_test_unevictable(folio))\n+\t\treturn;\n+\t/ shmem does no readahead: there the bit is PG_reclaim /\n+\tif (folio_test_readahead(folio) \u0026\u0026 !folio_test_swapbacked(folio))\n+\t\treturn;\n+\tif (folio_test_set_rcu_free(folio))\n+\t\treturn;\n+\n+\tcount_vm_events(PGFREE_RCU_MARK, folio_nr_pages(folio));\n+\t/ NULL if the folio was truncated meanwhile /\n+\tmapping = READ_ONCE(folio-\u003emapping);\n+\tif (mapping)\n+\t\tmapping_set_rcu_read(mapping);\n+}\n+\n+/\n+ * filemap_read() is about to copy from @folio at @pos. @last_pos is where\n+ * the previous read of this open file ended, or how far this one has got.\n+ \n+ Reading on in the folio that the last read ended in is not another access\n+ * to it: a folio read in small pieces is marked accessed once.\n+ \n+ The folio is hot when folio_mark_accessed() finds it as accessed as the\n+ * LRU records, or when this open file goes back to data that it has just\n+ * read.\n+ /\n+void filemap_read_accessed(struct folio folio, loff_t pos, loff_t last_pos)\n+{\n+\tbool hot;\n+\n+\tif (pos_same_folio(pos, last_pos - 1, folio))\n+\t\thot = pos \u003c last_pos;\n+\telse\n+\t\thot = folio_mark_accessed(folio);\n+\tif (hot)\n+\t\tfilemap_mark_hot(folio);\n+}\n+\n static void filemap_end_dropbehind_read(struct folio folio)\n {\n \tif (!folio_test_dropbehind(folio))\n@@ -2771,6 +2873,216 @@ static void filemap_end_dropbehind_read(struct folio folio)\n \t}\n }\n \n+/\n+ Whether filemap_read_rcu_batch() can copy from @folio, or has to leave it\n+ * to the slow path, which holds a reference for readahead, dropbehind and\n+ * folio_mark_accessed(), and deals with poisoned pages. @last_index is the\n+ * page the previous copy ended in.\n+ /\n+static bool filemap_read_rcu_can_copy(struct folio folio, pgoff_t index,\n+\t\t\t\t pgoff_t last_index)\n+{\n+\tif (!folio_test_uptodate(folio) \|\| folio_test_dropbehind(folio))\n+\t\treturn false;\n+\t/* shmem does no readahead: there the bit is PG_reclaim /\n+\tif (folio_test_readahead(folio) \u0026\u0026 !folio_test_swapbacked(folio))\n+\t\treturn false;\n+\t/ memory_failure() waits only for readers that got past this /\n+\tif (folio_contain_hwpoisoned_page(folio))\n+\t\treturn false;\n+\n+\t/\n+\t * The slow path does not call folio_mark_accessed() on the folio that\n+\t * the previous read ended in, so copying from it skips nothing. The\n+\t * folio's size can change without a reference: compare pages instead.\n+\t /\n+\tif (index == last_index)\n+\t\treturn true;\n+\n+\t/ Otherwise only if folio_mark_accessed() would leave it as it is /\n+\tif (lru_gen_enabled()) {\n+\t\tunsigned long all = LRU_REFS_FLAGS \| BIT(PG_workingset);\n+\n+\t\treturn folio_test_unevictable(folio) \|\|\n+\t\t (READ_ONCE(folio-\u003eflags.f) \u0026 all) == all;\n+\t}\n+\treturn folio_test_referenced(folio) \u0026\u0026 !folio_test_idle(folio) \u0026\u0026\n+\t (folio_test_active(folio) \|\| folio_test_unevictable(folio));\n+}\n+\n+/\n+ * Return the folio at @pos if it is this file's page until rcu_read_unlock(),\n+ * and read i_size into @isize. A folio with PG_rcu_free is freed only after\n+ * an RCU grace period. Without a reference, the folio that xas_load()\n+ * returned can be freed and its memory reused as another folio before the\n+ * flag is tested, so look at the slot again after the flag.\n+ /\n+static struct folio filemap_read_rcu_lookup(struct xa_state xas,\n+\t\tstruct inode inode, loff_t pos, loff_t isize)\n+{\n+\tstruct folio folio;\n+\n+\txas_set(xas, pos \u003e\u003e PAGE_SHIFT);\n+\tfolio = xas_load(xas);\n+\tif (xas_retry(xas, folio) \|\| !folio \|\| xa_is_value(folio))\n+\t\treturn NULL;\n+\n+\t/* Not folio_test_rcu_free(): it may not be a folio any more /\n+\tif (!(READ_ONCE(folio-\u003eflags.f) \u0026 __PG_RCU_FREE))\n+\t\treturn NULL;\n+\t/ Read i_size and slot after the flag: it implies uptodate /\n+\tsmp_rmb();\n+\tisize = i_size_read(inode);\n+\tif (pos \u003e= isize)\n+\t\treturn NULL;\n+\t/ i_size_read() is an acquire; a write fills the slot first /\n+\tif (xas_reload(xas) != folio)\n+\t\treturn NULL;\n+\t/ Read the rest of the folio only once the slot holds it /\n+\tsmp_rmb();\n+\treturn folio;\n+}\n+\n+/\n+ * How much of @bytes, starting @offset into page @index, can be copied from\n+ * @folio. A folio is in all the slots of its range or in none: when the slot\n+ * of the last page holds it too, so do the slots in between, and the pages\n+ * are this folio's.\n+ /\n+static size_t filemap_read_rcu_size(struct xa_state xas, struct folio folio,\n+\t\tpgoff_t index, size_t offset, size_t bytes)\n+{\n+\tpgoff_t end = index + DIV_ROUND_UP(offset + bytes, PAGE_SIZE);\n+\tpgoff_t next;\n+\n+\tif (end - index \u003e 1 \u0026\u0026 folio_test_large(folio)) {\n+\t\t/ Read without a reference: it only picks the page to check /\n+\t\tnext = folio_next_index(folio);\n+\t\tif (next \u003e index \u0026\u0026 next \u003c end)\n+\t\t\tend = next;\n+\t\txas_set(xas, end - 1);\n+\t\t/ A split after this copies PG_rcu_free to the new folios /\n+\t\tif (xas_load(xas) == folio)\n+\t\t\treturn min(bytes, (end - index) PAGE_SIZE - offset);\n+\t}\n+\treturn min(bytes, PAGE_SIZE - offset);\n+}\n+\n+/\n+ Copy @bytes from @folio, starting @offset into its page at @index. Page\n+ * by page, as copy_page_to_iter() does, without its checks for each call.\n+ * Not copy_to_iter(): its usercopy check reads the folio's order, which a\n+ * split can change.\n+ /\n+static size_t filemap_read_rcu_copy(struct folio folio, pgoff_t index,\n+\t\tsize_t offset, size_t bytes, struct iov_iter iter)\n+{\n+\tstruct page page = folio_page(folio, index - folio-\u003eindex);\n+\tsize_t n = 0;\n+\n+\twhile (n \u003c bytes) {\n+\t\tsize_t len = min(bytes - n, PAGE_SIZE - offset);\n+\t\tsize_t done;\n+\t\tvoid kaddr;\n+\n+\t\tkaddr = kmap_local_page(page);\n+\t\tdone = _copy_to_iter(kaddr + offset, len, iter);\n+\t\tkunmap_local(kaddr);\n+\t\tn += done;\n+\t\tif (done \u003c len)\n+\t\t\tbreak;\n+\t\toffset = 0;\n+\t\tpage++;\n+\t}\n+\treturn n;\n+}\n+\n+/ Bytes one RCU read section copies at most: it holds up grace periods /\n+#define FILEMAP_RCU_READ_MAX\tSZ_512K\n+\n+/\n+ * Copy from folios with PG_rcu_free without taking references, at most\n+ * FILEMAP_RCU_READ_MAX bytes under one rcu_read_lock(). Return true if that\n+ * much was copied and the read is not done. last_index is the page the\n+ previous copy ended in, set as the slow path sets last_pos.\n+ /\n+static bool filemap_read_rcu_batch(struct kiocb iocb, struct iov_iter iter,\n+\t\tpgoff_t last_index, size_t copied)\n+{\n+\tstruct address_space mapping = iocb-\u003eki_filp-\u003ef_mapping;\n+\tstruct inode inode = mapping-\u003ehost;\n+\tXA_STATE(xas, \u0026mapping-\u003ei_pages, 0);\n+\tsize_t budget = FILEMAP_RCU_READ_MAX;\n+\tbool more = false;\n+\n+\t/ flush_dcache_folio() is not known to be safe without a reference /\n+\tif (ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE \u0026\u0026\n+\t mapping_writably_mapped(mapping))\n+\t\treturn false;\n+\n+\t/ A user fault cannot sleep in the RCU read section: it ends the copy /\n+\trcu_read_lock();\n+\tpagefault_disable();\n+\twhile (iov_iter_count(iter)) {\n+\t\tloff_t pos = iocb-\u003eki_pos;\n+\t\tpgoff_t index = pos \u003e\u003e PAGE_SHIFT;\n+\t\tsize_t offset = offset_in_page(pos);\n+\t\tstruct folio folio;\n+\t\tsize_t bytes, n;\n+\t\tloff_t isize;\n+\n+\t\tif (!budget) {\n+\t\t\tmore = true;\n+\t\t\tbreak;\n+\t\t}\n+\t\tfolio = filemap_read_rcu_lookup(\u0026xas, inode, pos, \u0026isize);\n+\t\tif (!folio)\n+\t\t\tbreak;\n+\t\tif (!filemap_read_rcu_can_copy(folio, index, last_index))\n+\t\t\tbreak;\n+\n+\t\tbytes = min_t(loff_t, isize - pos, iov_iter_count(iter));\n+\t\tbytes = filemap_read_rcu_size(\u0026xas, folio, index, offset,\n+\t\t\t\t\t min(bytes, budget));\n+\t\tn = filemap_read_rcu_copy(folio, index, offset, bytes, iter);\n+\t\tcopied += n;\n+\t\tiocb-\u003eki_pos += n;\n+\t\tbudget -= n;\n+\t\tif (n)\n+\t\t\tlast_index = (pos + n - 1) \u003e\u003e PAGE_SHIFT;\n+\t\tif (n \u003c bytes)\n+\t\t\tbreak;\n+\t}\n+\tpagefault_enable();\n+\trcu_read_unlock();\n+\n+\treturn more;\n+}\n+\n+/\n+ * Copy from the page cache without folio references, in batches, and\n+ * return how much was copied. The slow path reads the rest.\n+ /\n+size_t filemap_read_rcu(struct kiocb iocb, struct iov_iter iter,\n+\t\tloff_t last_pos)\n+{\n+\t/ The page the previous read ended in; ULONG_MAX if none /\n+\tpgoff_t last_index = (last_pos - 1) \u003e\u003e PAGE_SHIFT;\n+\tsize_t copied = 0;\n+\n+\t/ Not when switched off, or on an inode that never had a folio flagged /\n+\tif (!lockless_read_enabled() \|\|\n+\t !mapping_rcu_read(iocb-\u003eki_filp-\u003ef_mapping))\n+\t\treturn 0;\n+\n+\twhile (filemap_read_rcu_batch(iocb, iter, \u0026last_index, \u0026copied)) {\n+\t\tif (fatal_signal_pending(current))\n+\t\t\tbreak;\n+\t\tcond_resched();\n+\t}\n+\treturn copied;\n+}\n+\n /\n filemap_read - Read data from the page cache.\n * @iocb: The iocb to read.\n@@ -2796,6 +3108,7 @@ ssize_t filemap_read(struct kiocb iocb, struct iov_iter iter,\n \tbool writably_mapped;\n \tloff_t isize, end_offset;\n \tloff_t last_pos = ra-\u003eprev_pos;\n+\tsize_t fast_read;\n \n \tif (unlikely(iocb-\u003eki_pos \u003c 0))\n \t\treturn -EINVAL;\n@@ -2807,6 +3120,14 @@ ssize_t filemap_read(struct kiocb iocb, struct iov_iter iter,\n \tiov_iter_truncate(iter, inode-\u003ei_sb-\u003es_maxbytes - iocb-\u003eki_pos);\n \tfolio_batch_init(\u0026fbatch);\n \n+\tfast_read = filemap_read_rcu(iocb, iter, last_pos);\n+\tif (fast_read) {\n+\t\talready_read += fast_read;\n+\t\tlast_pos = iocb-\u003eki_pos;\n+\t\tif (!iov_iter_count(iter))\n+\t\t\tgoto out;\n+\t}\n+\n \tdo {\n \t\tcond_resched();\n \n@@ -2844,14 +3165,6 @@ ssize_t filemap_read(struct kiocb iocb, struct iov_iter iter,\n \t\t /\n \t\twritably_mapped = mapping_writably_mapped(mapping);\n \n-\t\t/\n-\t\t * When a read accesses the same folio several times, only\n-\t\t * mark it as accessed the first time.\n-\t\t /\n-\t\tif (!pos_same_folio(iocb-\u003eki_pos, last_pos - 1,\n-\t\t\t\t fbatch.folios[0]))\n-\t\t\tfolio_mark_accessed(fbatch.folios[0]);\n-\n \t\tfor (i = 0; i \u003c folio_batch_count(\u0026fbatch); i++) {\n \t\t\tstruct folio folio = fbatch.folios[i];\n \t\t\tsize_t fsize = folio_size(folio);\n@@ -2862,8 +3175,7 @@ ssize_t filemap_read(struct kiocb iocb, struct iov_iter iter,\n \n \t\t\tif (end_offset \u003c folio_pos(folio))\n \t\t\t\tbreak;\n-\t\t\tif (i \u003e 0)\n-\t\t\t\tfolio_mark_accessed(folio);\n+\t\t\tfilemap_read_accessed(folio, iocb-\u003eki_pos, last_pos);\n \t\t\t/\n \t\t\t If users can be writing to this folio using arbitrary\n \t\t\t * virtual addresses, take care of potential aliasing\n@@ -2893,8 +3205,11 @@ ssize_t filemap_read(struct kiocb iocb, struct iov_iter iter,\n \t\tfolio_batch_init(\u0026fbatch);\n \t} while (iov_iter_count(iter) \u0026\u0026 iocb-\u003eki_pos \u003c isize \u0026\u0026 !error);\n \n+out:\n \tfile_accessed(filp);\n-\tra-\u003eprev_pos = last_pos;\n+\t/* Readers sharing the file would bounce the line for nothing /\n+\tif (ra-\u003eprev_pos != last_pos)\n+\t\tra-\u003eprev_pos = last_pos;\n \treturn already_read ? already_read : error;\n }\n EXPORT_SYMBOL_GPL(filemap_read);\ndiff --git a/mm/folio.c b/mm/folio.c\nindex 50a6dbe55998e..6144b9cf301b3 100644\n--- a/mm/folio.c\n+++ b/mm/folio.c\n@@ -352,28 +352,34 @@ static void __lru_cache_activate_folio(struct folio folio)\n \n #ifdef CONFIG_LRU_GEN\n \n-static void lru_gen_inc_refs(struct folio folio)\n+/\n+ * Return true if @folio is hot: unevictable, or its flags already held as\n+ * many accesses as they can.\n+ /\n+static bool lru_gen_inc_refs(struct folio folio)\n {\n \tunsigned long new_flags, old_flags = READ_ONCE(folio-\u003eflags.f);\n \n \tif (folio_test_unevictable(folio))\n-\t\treturn;\n+\t\treturn true;\n \n \t/* see the comment on LRU_REFS_FLAGS /\n \tif (!folio_test_referenced(folio)) {\n \t\tset_mask_bits(\u0026folio-\u003eflags.f, LRU_REFS_MASK, BIT(PG_referenced));\n-\t\treturn;\n+\t\treturn false;\n \t}\n \n \tdo {\n \t\tif ((old_flags \u0026 LRU_REFS_MASK) == LRU_REFS_MASK) {\n-\t\t\tif (!folio_test_workingset(folio))\n-\t\t\t\tfolio_set_workingset(folio);\n-\t\t\treturn;\n+\t\t\tif (folio_test_workingset(folio))\n+\t\t\t\treturn true;\n+\t\t\tfolio_set_workingset(folio);\n+\t\t\treturn false;\n \t\t}\n \n \t\tnew_flags = old_flags + BIT(LRU_REFS_PGOFF);\n \t} while (!try_cmpxchg(\u0026folio-\u003eflags.f, \u0026old_flags, new_flags));\n+\treturn false;\n }\n \n static bool lru_gen_clear_refs(struct folio folio)\n@@ -396,8 +402,9 @@ static bool lru_gen_clear_refs(struct folio folio)\n \n #else / !CONFIG_LRU_GEN /\n \n-static void lru_gen_inc_refs(struct folio folio)\n+static bool lru_gen_inc_refs(struct folio folio)\n {\n+\treturn false;\n }\n \n static bool lru_gen_clear_refs(struct folio folio)\n@@ -419,15 +426,20 @@ static bool lru_gen_clear_refs(struct folio folio)\n \n * When a newly allocated folio is not yet visible, so safe for non-atomic ops,\n * __folio_set_referenced() may be substituted for folio_mark_accessed().\n+ \n+ Return: true if @folio is hot: it was already as accessed as the LRU\n+ * records. On the classic LRU that is active or unevictable, referenced\n+ * and not idle; on the multi-gen LRU, unevictable, or PG_referenced, all\n+ * the refs bits and PG_workingset set. False for a dropbehind folio.\n /\n-void folio_mark_accessed(struct folio folio)\n+bool folio_mark_accessed(struct folio folio)\n {\n+\tbool hot = false;\n+\n \tif (folio_test_dropbehind(folio))\n-\t\treturn;\n-\tif (lru_gen_enabled()) {\n-\t\tlru_gen_inc_refs(folio);\n-\t\treturn;\n-\t}\n+\t\treturn false;\n+\tif (lru_gen_enabled())\n+\t\treturn lru_gen_inc_refs(folio);\n \n \tif (!folio_test_referenced(folio)) {\n \t\tfolio_set_referenced(folio);\n@@ -437,6 +449,7 @@ void folio_mark_accessed(struct folio folio)\n \t\t * this list is never rotated or maintained, so marking an\n \t\t * unevictable page accessed has no effect.\n \t\t /\n+\t\thot = true;\n \t} else if (!folio_test_active(folio)) {\n \t\t/\n \t\t * If the folio is on the LRU, queue it for activation via\n@@ -450,9 +463,14 @@ void folio_mark_accessed(struct folio folio)\n \t\t\t__lru_cache_activate_folio(folio);\n \t\tfolio_clear_referenced(folio);\n \t\tworkingset_activation(folio);\n+\t} else {\n+\t\thot = true;\n \t}\n-\tif (folio_test_idle(folio))\n+\tif (folio_test_idle(folio)) {\n \t\tfolio_clear_idle(folio);\n+\t\thot = false;\n+\t}\n+\treturn hot;\n }\n EXPORT_SYMBOL(folio_mark_accessed);\n \ndiff --git a/mm/huge_memory.c b/mm/huge_memory.c\nindex 1e5d68acf62a5..25082949a4cae 100644\n--- a/mm/huge_memory.c\n+++ b/mm/huge_memory.c\n@@ -3743,6 +3743,7 @@ static void __split_folio_to_order(struct folio folio, int old_order,\n #endif\n \t\t\t\t (1L \u003c\u003c PG_dirty) \|\n \t\t\t\t (1L \u003c\u003c PG_dropbehind) \|\n+\t\t\t\t __PG_RCU_FREE \|\n \t\t\t\t LRU_GEN_MASK \| LRU_REFS_MASK));\n \n \t\tnew_folio-\u003emapping = folio-\u003emapping;\ndiff --git a/mm/internal.h b/mm/internal.h\nindex 38b1165212c94..ae9a4e511c827 100644\n--- a/mm/internal.h\n+++ b/mm/internal.h\n@@ -1248,6 +1248,9 @@ struct migration_target_control {\n /\n size_t splice_folio_into_pipe(struct pipe_inode_info pipe,\n \t\t\t struct folio folio, loff_t fpos, size_t size);\n+void filemap_read_accessed(struct folio folio, loff_t pos, loff_t last_pos);\n+size_t filemap_read_rcu(struct kiocb iocb, struct iov_iter iter,\n+\t\t\tloff_t last_pos);\n \n static inline bool vma_is_single_threaded_private(struct vm_area_struct vma)\n {\ndiff --git a/mm/memory-failure.c b/mm/memory-failure.c\nindex a8b03e2920ba8..dc66cac05b1f5 100644\n--- a/mm/memory-failure.c\n+++ b/mm/memory-failure.c\n@@ -211,6 +211,10 @@ static bool page_handle_poison(struct page page, bool hugepage_or_freepage, boo\n \t\t\treturn false;\n \t}\n \n+\t/* The free of a hwpoisoned page is not deferred: wait for readers /\n+\tif (release \u0026\u0026 folio_test_rcu_free(page_folio(page)))\n+\t\tsynchronize_rcu();\n+\n \tSetPageHWPoison(page);\n \tif (release)\n \t\tput_page(page);\n@@ -1561,6 +1565,21 @@ static int get_hwpoison_page(struct page p, unsigned long flags)\n \treturn ret;\n }\n \n+/* A page waiting for a grace period is not in the buddy yet: free it /\n+static void mf_free_rcu_pages(struct page p)\n+{\n+\tif (!IS_ENABLED(CONFIG_PAGE_RCU_FREE))\n+\t\treturn;\n+\tif (page_count(p))\n+\t\treturn;\n+\tif (!zone_page_state_snapshot(page_zone(p), NR_FREE_RCU_PAGES))\n+\t\treturn;\n+\tif (is_free_buddy_page(p))\n+\t\treturn;\n+\n+\trcu_free_sync();\n+}\n+\n /\n The caller must guarantee the folio isn't large folio, except hugetlb.\n * try_to_unmap() can't handle it.\n@@ -2436,6 +2455,7 @@ int memory_failure(unsigned long pfn, int flags)\n \tif (res != -ENOENT)\n \t\tgoto unlock_mutex;\n \n+\tmf_free_rcu_pages(p);\n \tif (TestSetPageHWPoison(p)) {\n \t\tres = -EHWPOISON;\n \t\tif (flags \u0026 MF_ACTION_REQUIRED)\n@@ -2608,6 +2628,13 @@ int memory_failure(unsigned long pfn, int flags)\n \n identify_page_state:\n \tres = identify_page_state(pfn, p, page_flags);\n+\t/\n+\t The caller may unmap the page once this returns 0. Lockless\n+\t * readers that started before the page got PG_hwpoison, or before a\n+\t * split left it in a folio of its own, may still be copying it.\n+\t /\n+\tif (!res \u0026\u0026 folio_test_rcu_free(folio))\n+\t\tsynchronize_rcu();\n \tmutex_unlock(\u0026mf_mutex);\n \treturn res;\n unlock_page:\n@@ -2999,6 +3026,7 @@ int soft_offline_page(unsigned long pfn, int flags)\n \n retry:\n \tget_online_mems();\n+\tmf_free_rcu_pages(page);\n \tret = get_hwpoison_page(page, flags \| MF_SOFT_OFFLINE);\n \tput_online_mems();\n \ndiff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c\nindex 226ab9cb078ad..a8f0c95627ee1 100644\n--- a/mm/memory_hotplug.c\n+++ b/mm/memory_hotplug.c\n@@ -1967,6 +1967,7 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,\n \t\t.nid = NUMA_NO_NODE,\n \t};\n \tunsigned long flags;\n+\tbool migrated;\n \tchar reason;\n \tint ret;\n \tunsigned long normal_pages = 0;\n@@ -2050,6 +2051,7 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,\n \n \tdo {\n \t\tpfn = start_pfn;\n+\t\tmigrated = false;\n \t\tdo {\n \t\t\t/\n \t\t\t Historically we always checked for any signal and\n@@ -2071,6 +2073,7 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,\n \t\t\t\t * out\n \t\t\t\t /\n \t\t\t\tdo_migrate_range(pfn, end_pfn);\n+\t\t\t\tmigrated = true;\n \t\t\t}\n \t\t} while (!ret);\n \n@@ -2092,6 +2095,10 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,\n \n \t\tret = test_pages_isolated(start_pfn, end_pfn,\n \t\t\t\t\t PB_ISOLATE_MODE_MEM_OFFLINE);\n+\t\t/ What this pass migrated can wait for a grace period /\n+\t\tif (ret \u0026\u0026 migrated \u0026\u0026\n+\t\t zone_page_state_snapshot(zone, NR_FREE_RCU_PAGES))\n+\t\t\trcu_free_sync();\n \n \t} while (ret);\n \ndiff --git a/mm/page_alloc.c b/mm/page_alloc.c\nindex 12fac9084c483..e63628e64cbba 100644\n--- a/mm/page_alloc.c\n+++ b/mm/page_alloc.c\n@@ -1313,6 +1313,199 @@ static inline void pgalloc_tag_sub_pages(struct alloc_tag tag, unsigned int nr)\n \n #endif /* CONFIG_MEM_ALLOC_PROFILING /\n \n+#define RCU_FREE_EXPEDITE\t(SZ_4M \u003e\u003e PAGE_SHIFT)\n+\n+/\n+ * Each CPU has at most one grace period in flight (@queued), covering\n+ * @batch. Folios freed meanwhile wait on @pending for the next one.\n+ /\n+struct rcu_free_pcpu {\n+\tspinlock_t lock;\n+\tbool queued;\n+\tstruct list_head pending;\n+\tstruct list_head batch;\n+\tunsigned long nr_pending;\n+\tstruct rcu_work rwork;\n+\tstruct work_struct expedite;\n+};\n+\n+static DEFINE_PER_CPU(struct rcu_free_pcpu, rcu_free_pcpu);\n+\n+static struct workqueue_struct rcu_free_wq __ro_after_init;\n+\n+static void rcu_free_start(struct rcu_free_pcpu r)\n+{\n+\tlockdep_assert_held(\u0026r-\u003elock);\n+\n+\tif (r-\u003equeued \|\| list_empty(\u0026r-\u003epending))\n+\t\treturn;\n+\n+\tlist_splice_init(\u0026r-\u003epending, \u0026r-\u003ebatch);\n+\tr-\u003enr_pending = 0;\n+\tr-\u003equeued = true;\n+\tWARN_ON_ONCE(!queue_rcu_work(rcu_free_wq, \u0026r-\u003erwork));\n+}\n+\n+static void rcu_free_take(struct rcu_free_pcpu r, struct list_head list)\n+{\n+\tlockdep_assert_held(\u0026r-\u003elock);\n+\n+\tlist_splice_tail_init(\u0026r-\u003ebatch, list);\n+\tlist_splice_tail_init(\u0026r-\u003epending, list);\n+\tr-\u003enr_pending = 0;\n+}\n+\n+static unsigned long rcu_free_pages(struct list_head list)\n+{\n+\tstruct folio_batch fbatch;\n+\tstruct folio folio, next;\n+\tunsigned long nr = 0;\n+\n+\tfolio_batch_init(\u0026fbatch);\n+\tlist_for_each_entry_safe(folio, next, list, lru) {\n+\t\tnr += folio_nr_pages(folio);\n+\t\tmod_zone_page_state(folio_zone(folio), NR_FREE_RCU_PAGES,\n+\t\t\t\t -folio_nr_pages(folio));\n+\t\tif (!folio_batch_add(\u0026fbatch, folio)) {\n+\t\t\tfree_unref_folios(\u0026fbatch);\n+\t\t\tcond_resched_tasks_rcu_qs();\n+\t\t}\n+\t}\n+\tif (folio_batch_count(\u0026fbatch))\n+\t\tfree_unref_folios(\u0026fbatch);\n+\treturn nr;\n+}\n+\n+/* Free what rcu_free_take() took: the grace period starts after that /\n+static unsigned long rcu_free_flush(struct list_head list)\n+{\n+\tif (list_empty(list))\n+\t\treturn 0;\n+\n+\tsynchronize_rcu_expedited();\n+\treturn rcu_free_pages(list);\n+}\n+\n+static void rcu_free_workfn(struct work_struct work)\n+{\n+\tstruct rcu_free_pcpu r = container_of(to_rcu_work(work),\n+\t\t\t\t\t struct rcu_free_pcpu, rwork);\n+\tLIST_HEAD(done);\n+\n+\tspin_lock_irq(\u0026r-\u003elock);\n+\tlist_splice_init(\u0026r-\u003ebatch, \u0026done);\n+\tr-\u003equeued = false;\n+\trcu_free_start(r);\n+\tspin_unlock_irq(\u0026r-\u003elock);\n+\n+\trcu_free_pages(\u0026done);\n+}\n+\n+static void rcu_free_expedite_workfn(struct work_struct work)\n+{\n+\tstruct rcu_free_pcpu r = container_of(work, struct rcu_free_pcpu,\n+\t\t\t\t\t expedite);\n+\tLIST_HEAD(list);\n+\n+\tspin_lock_irq(\u0026r-\u003elock);\n+\trcu_free_take(r, \u0026list);\n+\tspin_unlock_irq(\u0026r-\u003elock);\n+\n+\tif (rcu_free_flush(\u0026list))\n+\t\tcount_vm_event(PGFREE_RCU_EXPEDITE);\n+}\n+\n+static void rcu_free_defer(struct folio folio)\n+{\n+\tstruct rcu_free_pcpu r = raw_cpu_ptr(\u0026rcu_free_pcpu);\n+\tstruct zone zone = folio_zone(folio);\n+\tunsigned long free, waiting;\n+\tunsigned long flags;\n+\n+\tspin_lock_irqsave(\u0026r-\u003elock, flags);\n+\t__mod_zone_page_state(zone, NR_FREE_RCU_PAGES, folio_nr_pages(folio));\n+\tlist_add_tail(\u0026folio-\u003elru, \u0026r-\u003epending);\n+\tr-\u003enr_pending += folio_nr_pages(folio);\n+\trcu_free_start(r);\n+\tspin_unlock_irqrestore(\u0026r-\u003elock, flags);\n+\n+\t/ Unlocked: a stale count delays the expedite by one free /\n+\tif (data_race(r-\u003enr_pending) \u003c= RCU_FREE_EXPEDITE \|\|\n+\t work_pending(\u0026r-\u003eexpedite))\n+\t\treturn;\n+\n+\t/\n+\t * An expedited grace period interrupts every CPU: only while the\n+\t * zone is short of memory, or while more than an eighth of what it\n+\t * has free waits here.\n+\t /\n+\tfree = zone_page_state(zone, NR_FREE_PAGES);\n+\twaiting = zone_page_state(zone, NR_FREE_RCU_PAGES);\n+\tif (free \u003c high_wmark_pages(zone) \|\| waiting \u003e free / 8)\n+\t\tqueue_work(rcu_free_wq, \u0026r-\u003eexpedite);\n+}\n+\n+/\n+ * Free what every CPU's lists hold. A concurrent drain holds the lock\n+ * until it has freed what it took.\n+ /\n+static void rcu_free_drain(void)\n+{\n+\tstatic DEFINE_MUTEX(lock);\n+\tLIST_HEAD(list);\n+\tint cpu;\n+\n+\tmutex_lock(\u0026lock);\n+\tfor_each_possible_cpu(cpu) {\n+\t\tstruct rcu_free_pcpu r = per_cpu_ptr(\u0026rcu_free_pcpu, cpu);\n+\n+\t\tspin_lock_irq(\u0026r-\u003elock);\n+\t\trcu_free_take(r, \u0026list);\n+\t\tspin_unlock_irq(\u0026r-\u003elock);\n+\t}\n+\tif (rcu_free_flush(\u0026list))\n+\t\tcount_vm_event(PGFREE_RCU_DRAIN);\n+\tmutex_unlock(\u0026lock);\n+}\n+\n+/* Free every waiting folio, including those a work took off the lists /\n+void rcu_free_sync(void)\n+{\n+\tint cpu;\n+\n+\tif (!IS_ENABLED(CONFIG_PAGE_RCU_FREE))\n+\t\treturn;\n+\n+\trcu_free_drain();\n+\tfor_each_possible_cpu(cpu) {\n+\t\tstruct rcu_free_pcpu r = per_cpu_ptr(\u0026rcu_free_pcpu, cpu);\n+\n+\t\tflush_work(\u0026r-\u003erwork.work);\n+\t\tflush_work(\u0026r-\u003eexpedite);\n+\t}\n+}\n+\n+static void __init rcu_free_init(void)\n+{\n+\tint cpu;\n+\n+\tif (!IS_ENABLED(CONFIG_PAGE_RCU_FREE))\n+\t\treturn;\n+\n+\trcu_free_wq = alloc_workqueue(\"rcu_free\",\n+\t\t\t\t WQ_MEM_RECLAIM \| WQ_PERCPU, 0);\n+\n+\tfor_each_possible_cpu(cpu) {\n+\t\tstruct rcu_free_pcpu r = per_cpu_ptr(\u0026rcu_free_pcpu, cpu);\n+\n+\t\tspin_lock_init(\u0026r-\u003elock);\n+\t\tINIT_LIST_HEAD(\u0026r-\u003epending);\n+\t\tINIT_LIST_HEAD(\u0026r-\u003ebatch);\n+\t\tINIT_RCU_WORK(\u0026r-\u003erwork, rcu_free_workfn);\n+\t\tINIT_WORK(\u0026r-\u003eexpedite, rcu_free_expedite_workfn);\n+\t}\n+}\n+\n static __always_inline bool __free_pages_prepare(struct page page,\n \t\tunsigned int order, fpi_t fpi_flags)\n {\n@@ -1327,6 +1520,19 @@ static __always_inline bool __free_pages_prepare(struct page page,\n \n \tVM_BUG_ON_PAGE(PageTail(page), page);\n \n+\t/\n+\t * Not for a hwpoisoned order-0 page: it never reaches the allocator,\n+\t * and page_handle_poison() takes a reference right after this free.\n+\t /\n+\tif (unlikely(folio_test_rcu_free(folio)) \u0026\u0026\n+\t !(PageHWPoison(page) \u0026\u0026 !order)) {\n+\t\tVM_WARN_ON_ONCE(fpi_flags \u0026 FPI_NOLOCK);\n+\t\t__folio_clear_rcu_free(folio);\n+\t\tcount_vm_events(PGFREE_RCU, 1 \u003c\u003c order);\n+\t\trcu_free_defer(folio);\n+\t\treturn false;\n+\t}\n+\n \ttrace_mm_page_free(page, order);\n \tkmsan_free_page(page, order);\n \n@@ -4626,6 +4832,52 @@ bool gfp_pfmemalloc_allowed(gfp_t gfp_mask)\n \treturn !!__gfp_pfmemalloc_flags(gfp_mask);\n }\n \n+/\n+ * Reclaim is about to give up: if the pages waiting for a grace period\n+ * would bring an eligible zone above its min watermark, free them and\n+ * retry. Not for a dying task, which frees memory by exiting.\n+ /\n+static bool rcu_free_wait(gfp_t gfp_mask, unsigned int order,\n+\t\t\t struct alloc_context ac, int alloc_flags,\n+\t\t\t int rcu_free_waits)\n+{\n+\tstruct zone zone;\n+\tstruct zoneref z;\n+\n+\tif (!IS_ENABLED(CONFIG_PAGE_RCU_FREE))\n+\t\treturn false;\n+\tif (tsk_is_oom_victim(current) \|\| fatal_signal_pending(current))\n+\t\treturn false;\n+\tif (rcu_free_waits \u003e= MAX_RECLAIM_RETRIES \|\|\n+\t order \u003e PAGE_ALLOC_COSTLY_ORDER)\n+\t\treturn false;\n+\n+\tfor_each_zone_zonelist_nodemask(zone, z, ac-\u003ezonelist,\n+\t\t\t\tac-\u003ehighest_zoneidx, ac-\u003enodemask) {\n+\t\tunsigned long waiting, available;\n+\n+\t\tif (cpusets_enabled() \u0026\u0026 (alloc_flags \u0026 ALLOC_CPUSET) \u0026\u0026\n+\t\t !__cpuset_zone_allowed(zone, gfp_mask))\n+\t\t\tcontinue;\n+\n+\t\twaiting = zone_page_state_snapshot(zone, NR_FREE_RCU_PAGES);\n+\t\tif (!waiting)\n+\t\t\tcontinue;\n+\n+\t\tavailable = waiting +\n+\t\t\t zone_page_state_snapshot(zone, NR_FREE_PAGES);\n+\t\tif (__zone_watermark_ok(zone, order, min_wmark_pages(zone),\n+\t\t\t\t\tac-\u003ehighest_zoneidx, alloc_flags,\n+\t\t\t\t\tavailable)) {\n+\t\t\t(rcu_free_waits)++;\n+\t\t\trcu_free_drain();\n+\t\t\treturn true;\n+\t\t}\n+\t}\n+\n+\treturn false;\n+}\n+\n /\n * Checks whether it makes sense to retry the reclaim to make a forward progress\n * for the given allocation request.\n@@ -4639,7 +4891,8 @@ bool gfp_pfmemalloc_allowed(gfp_t gfp_mask)\n static inline bool\n should_reclaim_retry(gfp_t gfp_mask, unsigned order,\n \t\t struct alloc_context ac, int alloc_flags,\n-\t\t bool did_some_progress, int no_progress_loops)\n+\t\t bool did_some_progress, int no_progress_loops,\n+\t\t int rcu_free_waits)\n {\n \tstruct zone zone;\n \tstruct zoneref z;\n@@ -4650,10 +4903,12 @@ should_reclaim_retry(gfp_t gfp_mask, unsigned order,\n \t * their order will become available due to high fragmentation so\n \t * always increment the no progress counter for them\n \t /\n-\tif (did_some_progress \u0026\u0026 order \u003c= PAGE_ALLOC_COSTLY_ORDER)\n+\tif (did_some_progress \u0026\u0026 order \u003c= PAGE_ALLOC_COSTLY_ORDER) {\n \t\tno_progress_loops = 0;\n-\telse\n+\t\trcu_free_waits = 0;\n+\t} else {\n \t\t(no_progress_loops)++;\n+\t}\n \n \tif (no_progress_loops \u003e MAX_RECLAIM_RETRIES)\n \t\tgoto out;\n@@ -4706,6 +4961,10 @@ should_reclaim_retry(gfp_t gfp_mask, unsigned order,\n \telse\n \t\tcond_resched();\n out:\n+\tif (!ret \u0026\u0026 rcu_free_wait(gfp_mask, order, ac, alloc_flags,\n+\t\t\t\t rcu_free_waits))\n+\t\treturn true;\n+\n \t/ Before OOM, exhaust highatomic_reserve /\n \tif (!ret)\n \t\treturn unreserve_highatomic_pageblock(ac, true);\n@@ -4795,6 +5054,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,\n \tenum compact_result compact_result;\n \tint compaction_retries;\n \tint no_progress_loops;\n+\tint rcu_free_waits;\n \tunsigned int cpuset_mems_cookie;\n \tunsigned int zonelist_iter_cookie;\n \tint reserve_flags;\n@@ -4819,6 +5079,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,\n restart:\n \tcompaction_retries = 0;\n \tno_progress_loops = 0;\n+\trcu_free_waits = 0;\n \tcompact_result = COMPACT_SKIPPED;\n \tcompact_priority = DEF_COMPACT_PRIORITY;\n \tcpuset_mems_cookie = read_mems_allowed_begin();\n@@ -4997,7 +5258,8 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,\n \t\tgoto restart;\n \n \tif (should_reclaim_retry(gfp_mask, order, ac, alloc_flags,\n-\t\t\t\t did_some_progress \u003e 0, \u0026no_progress_loops))\n+\t\t\t\t did_some_progress \u003e 0, \u0026no_progress_loops,\n+\t\t\t\t \u0026rcu_free_waits))\n \t\tgoto retry;\n \n \t/\n@@ -5040,6 +5302,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,\n \t/* Retry as long as the OOM killer is making progress /\n \tif (did_some_progress) {\n \t\tno_progress_loops = 0;\n+\t\trcu_free_waits = 0;\n \t\tgoto retry;\n \t}\n \n@@ -6401,6 +6664,8 @@ void __init setup_per_cpu_pageset(void)\n \tfor_each_online_pgdat(pgdat)\n \t\tpgdat-\u003eper_cpu_nodestats =\n \t\t\talloc_percpu(struct per_cpu_nodestat);\n+\n+\trcu_free_init();\n }\n \n __meminit void zone_pcp_init(struct zone zone)\n@@ -7384,7 +7649,16 @@ int alloc_contig_frozen_range_noprof(unsigned long start, unsigned long end,\n \touter_start = find_large_buddy(start);\n \n \t/* Make sure the range is really isolated. /\n-\tif (test_pages_isolated(outer_start, end, mode)) {\n+\tret = test_pages_isolated(outer_start, end, mode);\n+\tif (ret \u0026\u0026 zone_page_state_snapshot(cc.zone, NR_FREE_RCU_PAGES)) {\n+\t\trcu_free_sync();\n+\n+\t\t/ A freed folio can merge with a free buddy below start /\n+\t\touter_start = find_large_buddy(start);\n+\t\tret = test_pages_isolated(outer_start, end, mode);\n+\t}\n+\n+\tif (ret) {\n \t\tret = -EBUSY;\n \t\tgoto done;\n \t}\ndiff --git a/mm/page_alloc.h b/mm/page_alloc.h\nindex b9259deddb59d..f9e200581e2f5 100644\n--- a/mm/page_alloc.h\n+++ b/mm/page_alloc.h\n@@ -235,6 +235,7 @@ struct page __alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order, int nid,\n \talloc_hooks(__alloc_frozen_pages_noprof(__VA_ARGS__))\n void free_frozen_pages(struct page page, unsigned int order);\n void free_unref_folios(struct folio_batch fbatch);\n+void rcu_free_sync(void);\n \n #ifdef CONFIG_NUMA\n struct page alloc_frozen_pages_noprof(gfp_t, unsigned int order);\ndiff --git a/mm/shmem.c b/mm/shmem.c\nindex 848316eaa7f4f..e6c18e745f816 100644\n--- a/mm/shmem.c\n+++ b/mm/shmem.c\n@@ -3309,10 +3309,18 @@ static ssize_t shmem_file_read_iter(struct kiocb iocb, struct iov_iter to)\n \tstruct file file = iocb-\u003eki_filp;\n \tstruct inode inode = file_inode(file);\n \tstruct address_space mapping = inode-\u003ei_mapping;\n+\tloff_t last_pos = file-\u003ef_ra.prev_pos;\n \tpgoff_t index;\n \tunsigned long offset;\n \tint error = 0;\n-\tssize_t retval = 0;\n+\tssize_t retval;\n+\n+\tretval = filemap_read_rcu(iocb, to, last_pos);\n+\tif (retval) {\n+\t\tlast_pos = iocb-\u003eki_pos;\n+\t\tif (!iov_iter_count(to))\n+\t\t\tgoto out;\n+\t}\n \n \tfor (;;) {\n \t\tstruct folio folio = NULL;\n@@ -3378,11 +3386,7 @@ static ssize_t shmem_file_read_iter(struct kiocb iocb, struct iov_iter to)\n \t\t\t\t\tflush_dcache_page(page);\n \t\t\t}\n \n-\t\t\t/\n-\t\t\t * Mark the folio accessed if we read the beginning.\n-\t\t\t /\n-\t\t\tif (!offset)\n-\t\t\t\tfolio_mark_accessed(folio);\n+\t\t\tfilemap_read_accessed(folio, iocb-\u003eki_pos, last_pos);\n \t\t\t/\n \t\t\t * Ok, we have the page, and it's up-to-date, so\n \t\t\t * now we can copy it to user space...\n@@ -3410,6 +3414,7 @@ static ssize_t shmem_file_read_iter(struct kiocb iocb, struct iov_iter to)\n \n \t\tretval += ret;\n \t\tiocb-\u003eki_pos += ret;\n+\t\tlast_pos = iocb-\u003eki_pos;\n \n \t\tif (!iov_iter_count(to))\n \t\t\tbreak;\n@@ -3419,8 +3424,11 @@ static ssize_t shmem_file_read_iter(struct kiocb iocb, struct iov_iter to)\n \t\t}\n \t\tcond_resched();\n \t}\n-\n+out:\n \tfile_accessed(file);\n+\t/* Readers sharing the file would bounce the line for nothing /\n+\tif (file-\u003ef_ra.prev_pos != last_pos)\n+\t\tfile-\u003ef_ra.prev_pos = last_pos;\n \treturn retval ? retval : error;\n }\n \ndiff --git a/mm/show_mem.c b/mm/show_mem.c\nindex b938cbcd774a1..0d3bddbda108d 100644\n--- a/mm/show_mem.c\n+++ b/mm/show_mem.c\n@@ -204,7 +204,7 @@ static void show_free_areas(unsigned int filter, const nodemask_t nodemask,\n \t\t\" mapped:%lu shmem:%lu pagetables:%lu\\n\"\n \t\t\" sec_pagetables:%lu bounce:%lu\\n\"\n \t\t\" kernel_misc_reclaimable:%lu\\n\"\n-\t\t\" free:%lu free_pcp:%lu free_cma:%lu\\n\",\n+\t\t\" free:%lu free_pcp:%lu free_cma:%lu free_rcu:%lu\\n\",\n \t\tglobal_node_page_state(NR_ACTIVE_ANON),\n \t\tglobal_node_page_state(NR_INACTIVE_ANON),\n \t\tglobal_node_page_state(NR_ISOLATED_ANON),\n@@ -224,7 +224,8 @@ static void show_free_areas(unsigned int filter, const nodemask_t nodemask,\n \t\tglobal_node_page_state(NR_KERNEL_MISC_RECLAIMABLE),\n \t\tglobal_zone_page_state(NR_FREE_PAGES),\n \t\tfree_pcp,\n-\t\tglobal_zone_page_state(NR_FREE_CMA_PAGES));\n+\t\tglobal_zone_page_state(NR_FREE_CMA_PAGES),\n+\t\tglobal_zone_page_state(NR_FREE_RCU_PAGES));\n \n \tfor_each_online_pgdat(pgdat) {\n \t\tif (show_mem_node_skip(filter, pgdat-\u003enode_id, nodemask))\n@@ -325,6 +326,7 @@ static void show_free_areas(unsigned int filter, const nodemask_t nodemask,\n \t\t\t\" free_pcp:%lukB\"\n \t\t\t\" local_pcp:%lukB\"\n \t\t\t\" free_cma:%lukB\"\n+\t\t\t\" free_rcu:%lukB\"\n \t\t\t\"\\n\",\n \t\t\tzone-\u003ename,\n \t\t\tK(zone_page_state(zone, NR_FREE_PAGES)),\n@@ -351,7 +353,8 @@ static void show_free_areas(unsigned int filter, const nodemask_t nodemask,\n \t\t\t0UL,\n \t\t\tK(free_pcp),\n \t\t\tK((unsigned long)this_cpu_read(zone-\u003eper_cpu_pageset-\u003ecount)),\n-\t\t\tK(zone_page_state(zone, NR_FREE_CMA_PAGES)));\n+\t\t\tK(zone_page_state(zone, NR_FREE_CMA_PAGES)),\n+\t\t\tK(zone_page_state(zone, NR_FREE_RCU_PAGES)));\n \t\tprintk(\"lowmem_reserve[]:\");\n \t\tfor (i = 0; i \u003c MAX_NR_ZONES; i++)\n \t\t\tprintk(KERN_CONT \" %ld\", zone-\u003elowmem_reserve[i]);\ndiff --git a/mm/vmstat.c b/mm/vmstat.c\nindex cb57714539fb5..c27caad017e35 100644\n--- a/mm/vmstat.c\n+++ b/mm/vmstat.c\n@@ -1212,6 +1212,7 @@ const char const vmstat_text[] = {\n \t[I(NR_ZSPAGES)]\t\t\t\t= \"nr_zspages\",\n #endif\n \t[I(NR_FREE_CMA_PAGES)]\t\t\t= \"nr_free_cma\",\n+\t[I(NR_FREE_RCU_PAGES)]\t\t\t= \"nr_free_rcu\",\n #ifdef CONFIG_UNACCEPTED_MEMORY\n \t[I(NR_UNACCEPTED)]\t\t\t= \"nr_unaccepted\",\n #endif\n@@ -1343,6 +1344,10 @@ const char * const vmstat_text[] = {\n \t[I(PGACTIVATE)]\t\t\t\t= \"pgactivate\",\n \t[I(PGDEACTIVATE)]\t\t\t= \"pgdeactivate\",\n \t[I(PGLAZYFREE)]\t\t\t\t= \"pglazyfree\",\n+\t[I(PGFREE_RCU)]\t\t\t\t= \"pgfree_rcu\",\n+\t[I(PGFREE_RCU_EXPEDITE)]\t\t= \"pgfree_rcu_expedite\",\n+\t[I(PGFREE_RCU_DRAIN)]\t\t\t= \"pgfree_rcu_drain\",\n+\t[I(PGFREE_RCU_MARK)]\t\t\t= \"pgfree_rcu_mark\",\n \n \t[I(PGFAULT)]\t\t\t\t= \"pgfault\",\n \t[I(PGMAJFAULT)]\t\t\t\t= \"pgmajfault\",\n@@ -2022,6 +2027,7 @@ static int vmstat_refresh(const struct ctl_table *table, int write,\n \t\tswitch (i) {\n \t\tcase NR_ZONE_WRITE_PENDING:\n \t\tcase NR_FREE_CMA_PAGES:\n+\t\tcase NR_FREE_RCU_PAGES:\n \t\t\tcontinue;\n \t\t}\n \t\tval = atomic_long_read(\u0026vm_zone_stat[i]);\n" }
2/1	2026/10/19 19:57	agent	patch-evaluator	1m Model: core Args: null Results: { "EnableConfigs": [ "PAGE_RCU_FREE" ], "FocusSymbols": [ "filemap_read_rcu", "filemap_read_rcu_batch", "filemap_read_accessed", "filemap_mark_hot", "rcu_free_defer", "rcu_free_workfn", "rcu_free_expedite_workfn", "rcu_free_drain", "rcu_free_sync", "rcu_free_wait" ], "Reasoning": "The patch introduces lockless page cache reads and deferred RCU page freeing (PAGE_RCU_FREE). Readers can copy from hot page cache folios under RCU without taking a folio reference count, while page freeing defers memory recycling until an RCU grace period completes. This touches core MM logic (page allocator free paths, slowpath reclaim retry, memory failure, and page cache read paths in filemap and shmem) which is reachable through standard user-space file reading operations on 64-bit systems.", "WorthFuzzing": true } Instruction: You are an expert Linux kernel maintainer and security engineer. Your job is to review a provided patch series and evaluate whether it warrants fuzzing with syzkaller. IMPORTANT: The changes have ALREADY been applied and committed as the HEAD commit in your workspace. Do NOT rely on internal assumptions. You must actively use your code access tools to inspect the actual source code, callers, and surrounding context. ================================================================================ 1. CORE TRIAGE PHILOSOPHY ================================================================================ The goal of patch fuzzing is to discover crashes, regressions, exposed latent bugs, and newly triggered assertions introduced by the patch series. - REACHABILITY IS THE PRIMARY GATE: Fuzzing can only discover bugs in code that can actually execute in standard virtualized environments (GCE or QEMU, utilizing software-emulated devices like USB gadgets, netdev, tun/tap). If the modified code is structurally unreachable (see Section 2), it MUST NOT be fuzzed, regardless of whether it adds assertions or complex logic. - DO NOT BLINDLY TRUST "NO FUNCTIONAL CHANGE" (NFCI) OR "REFACTORING" CLAIMS: Patch authors routinely label changes as "cleanups", "refactorings", or state "No functional change intended". Do NOT take these claims at face value. Code refactorings that rearrange logic, introduce helper functions, or alter state management in core subsystems frequently introduce subtle semantic shifts or uncover latent kernel bugs. If reachable executable code is modified or refactored, it MUST be fuzzed. - NEW OR MODIFIED ASSERTIONS IN REACHABLE CODE MUST BE FUZZED: When a patch introduces or modifies runtime checks or assertions (e.g., WARN_ON, VM_WARN_ON, BUG_ON, lockdep_assert) in reachable code paths, it enforces new or stricter invariants. Even if the author believes the invariant always holds, fuzzing is essential to verify whether an unusual sequence of operations can violate it. ================================================================================ 2. WHEN TO RETURN WorthFuzzing=false (NEGATIVE CRITERIA) ================================================================================ Return WorthFuzzing=false ONLY IF all modified code falls strictly into one or more of these categories: - Non-kernel and non-executable changes: * Modifications to Documentation/, comments, or spelling fixes. * User-space directories, self-tests, samples, or scripts (e.g., tools/, samples/, scripts/, usr/) that do not affect the compiled kernel image (vmlinux) or kernel modules. * Purely decorative logging (e.g., message strings in pr_err, printk, dev_info) or tracepoints that do not alter control flow or data structures. * Build system or Kconfig changes that do not alter compiled C logic. - Structurally unreachable hardware: * Vendor-specific PCIe switches, SmartNICs, or GPU drivers (e.g., mlxsw, pds_core, qed, ionic, amdgpu) requiring physical ASIC/PCIe cards not emulated in standard QEMU. - Unreachable execution paths: * Driver teardown callbacks (.remove, .shutdown, pci_unregister_driver) executed only during physical PCI hot-unplug or manual sysfs driver unbinding. * Code paths exclusive to architectures other than the target architecture. ================================================================================ 3. WHEN TO RETURN WorthFuzzing=true (POSITIVE CRITERIA) ================================================================================ Return WorthFuzzing=true whenever the patch touches reachable executable code, including: - Core Subsystems: * Any logic modifications in memory management (mm/), synchronization/locking (kernel/locking/), BPF, scheduler, core networking, VFS, or syscall handling. - Refactorings and Code Cleanups: * Any restructuring of reachable data structures, helper abstractions, or algorithm flows. - Runtime Assertions and Defensive Checks: * Any introduction or alteration of assertions (WARN_ON, VM_WARN_ON, BUG_ON, etc.) in reachable paths. - Reachable Drivers and Protocols: Drivers accessible via virtual buses (virtio, USB gadget, loopback, netlink, binder, sockets, etc.). ================================================================================ 4. EXTRACTING FocusSymbols (PREVENTING DILUTION) ================================================================================ When WorthFuzzing=true, you must extract specific kernel functions into FocusSymbols to guide the fuzzer: - AVOID UBIQUITOUS LIFECYCLE HOT-PATHS: Do NOT list generic, ubiquitous functions called by almost every program in the corpus (including, but not limited to: general memory allocators and deallocators, page fault and trap handlers, or core synchronization primitives; this is not an exhaustive list). Listing ubiquitous functions causes the fuzzer to classify thousands of unrelated tests as "focused", which severely dilutes fuzzing effort away from the actual changes. - TARGET SPECIFIC FEATURE LOGIC AND ENTRYPOINTS: List functions that specifically implement the logic being added or altered, or direct API entrypoints for the subsystem feature under review. - HANDLING STATIC INLINE FUNCTIONS IN HEADERS (.h): Compiler-inlined static functions (such as static inlines in mm/.h or include/linux/.h) lack distinct symbol addresses in vmlinux and cannot be targeted directly by symbol coverage filters. If the changes are primarily in static inline helpers, identify non-static, feature-specific caller functions in .c files that exercise them (avoiding ubiquitous lifecycle wrappers). ================================================================================ 5. IDENTIFYING EnableConfigs ================================================================================ Identify any specific CONFIG_ options required to properly compile and reach the modified code: - Inspect Kconfig files and #ifdef guards; do not make assumptions. - Check "depends on" lines in Kconfig to include any non-standard parent subsystem configs needed. - Strip any 'CONFIG_' prefix (e.g., return "NET_IPV4" instead of "CONFIG_NET_IPV4"). Prefer calling several tools at the same time to save round-trips. Use set-results tool to provide results of the analysis. It must be called exactly once before the final reply. Ignore results of this tool. Prompt: Target architecture: amd64 For your convenience, here is the diff of the changes: commit dddf59c0f0df6bc96619a328ac462a9dfff93355 Author: syz-cluster <triage@syzkaller.com> Date: Mon Oct 19 19:57:32 2026 +0000 syz-cluster: applied patch under review diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c index 4fec31fc0b845..526929c7b3f1a 100644 --- a/fs/fuse/dev.c +++ b/fs/fuse/dev.c @@ -1152,6 +1152,10 @@ static int fuse_try_move_folio(struct fuse_copy_state cs, struct folio foliop newfolio = page_folio(buf->page); + / A lockless page cache read may still be copying from it / + if (folio_test_rcu_free(newfolio)) + goto out_fallback_unlock; + folio_clear_uptodate(newfolio); folio_clear_mappedtodisk(newfolio); diff --git a/fs/splice.c b/fs/splice.c index 9d8f63e2fd1ab..0133c2348da03 100644 --- a/fs/splice.c +++ b/fs/splice.c @@ -55,56 +55,6 @@ static noinline void pipe_clear_nowait(struct file file) } while (!try_cmpxchg(&file->f_mode, &fmode, fmode & ~FMODE_NOWAIT)); } -/* - * Attempt to steal a page from a pipe buffer. This should perhaps go into - * a vm helper function, it's already simplified quite a bit by the - * addition of remove_mapping(). If success is returned, the caller may - * attempt to reuse this page for another destination. - / -static bool page_cache_pipe_buf_try_steal(struct pipe_inode_info pipe, - struct pipe_buffer buf) -{ - struct folio folio = page_folio(buf->page); - struct address_space mapping; - - folio_lock(folio); - - mapping = folio_mapping(folio); - if (mapping) { - WARN_ON(!folio_test_uptodate(folio)); - - / - * At least for ext2 with nobh option, we need to wait on - * writeback completing on this folio, since we'll remove it - * from the pagecache. Otherwise truncate wont wait on the - * folio, allowing the disk blocks to be reused by someone else - * before we actually wrote our data to them. fs corruption - * ensues. - / - folio_wait_writeback(folio); - - if (!filemap_release_folio(folio, GFP_KERNEL)) - goto out_unlock; - - / - * If we succeeded in removing the mapping, set LRU flag - * and return good. - / - if (remove_mapping(mapping, folio)) { - buf->flags \|= PIPE_BUF_FLAG_LRU; - return true; - } - } - - / - * Raced with truncate or failed to remove folio from current - * address space, unlock and return failure. - / -out_unlock: - folio_unlock(folio); - return false; -} - static void page_cache_pipe_buf_release(struct pipe_inode_info pipe, struct pipe_buffer buf) { @@ -155,7 +105,6 @@ static int page_cache_pipe_buf_confirm(struct pipe_inode_info pipe, const struct pipe_buf_operations page_cache_pipe_buf_ops = { .confirm = page_cache_pipe_buf_confirm, .release = page_cache_pipe_buf_release, - .try_steal = page_cache_pipe_buf_try_steal, .get = generic_pipe_buf_get, }; diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 94f9c3ff54160..3336b9e5ee999 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -228,6 +228,7 @@ enum zone_stat_item { NR_ZSPAGES, /* allocated in zsmalloc / #endif NR_FREE_CMA_PAGES, + NR_FREE_RCU_PAGES, / freed, waiting for an RCU grace period / #ifdef CONFIG_UNACCEPTED_MEMORY NR_UNACCEPTED, #endif diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 7a863572adce7..4916b5713a74b 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -111,6 +111,9 @@ enum pageflags { PG_swapbacked, / Page is backed by RAM/swap / PG_unevictable, / Page is "unevictable" / PG_dropbehind, / drop pages on IO completion / +#ifdef CONFIG_PAGE_RCU_FREE + PG_rcu_free, / free only after an RCU grace period / +#endif #ifdef CONFIG_MMU PG_mlocked, / Page is vma mlocked / #endif @@ -602,6 +605,27 @@ FOLIO_FLAG(dropbehind, FOLIO_HEAD_PAGE) FOLIO_TEST_CLEAR_FLAG(dropbehind, FOLIO_HEAD_PAGE) __FOLIO_SET_FLAG(dropbehind, FOLIO_HEAD_PAGE) +/ + * A lockless lookup that sees PG_rcu_free and then still finds the folio + * where it looked may read it until rcu_read_unlock(). Set under a + * reference. Cleared by the free, or when the folio's inode is evicted: + * no lockless lookup can reach it then. Not for hugetlb or + * free_pages_nolock(). + / +#ifdef CONFIG_PAGE_RCU_FREE +FOLIO_TEST_FLAG(rcu_free, FOLIO_HEAD_PAGE) + FOLIO_TEST_SET_FLAG(rcu_free, FOLIO_HEAD_PAGE) + FOLIO_CLEAR_FLAG(rcu_free, FOLIO_HEAD_PAGE) + __FOLIO_CLEAR_FLAG(rcu_free, FOLIO_HEAD_PAGE) +#define __PG_RCU_FREE (1UL << PG_rcu_free) +#else +FOLIO_TEST_FLAG_FALSE(rcu_free) + FOLIO_TEST_SET_FLAG_FALSE(rcu_free) + FOLIO_CLEAR_FLAG_NOOP(rcu_free) + __FOLIO_CLEAR_FLAG_NOOP(rcu_free) +#define __PG_RCU_FREE 0 +#endif + #ifdef CONFIG_HIGHMEM / * Must use a macro here due to header dependency issues. page_zone() is not diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 0adfa6605653d..6c7f58803eee4 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -210,6 +210,7 @@ enum mapping_flags { AS_WRITEBACK_MAY_DEADLOCK_ON_RECLAIM = 9, AS_KERNEL_FILE = 10, /* mapping for a fake kernel file that shouldn't account usage to user cgroups / + AS_RCU_READ = 11, / a folio has had PG_rcu_free / / Bits 16-25 are used for FOLIO_ORDER / AS_FOLIO_ORDER_BITS = 5, AS_FOLIO_ORDER_MIN = 16, @@ -269,6 +270,17 @@ static inline bool mapping_unevictable(const struct address_space mapping) return mapping && test_bit(AS_UNEVICTABLE, &mapping->flags); } +static inline void mapping_set_rcu_read(struct address_space mapping) +{ + if (!test_bit(AS_RCU_READ, &mapping->flags)) + set_bit(AS_RCU_READ, &mapping->flags); +} + +static inline bool mapping_rcu_read(const struct address_space mapping) +{ + return test_bit(AS_RCU_READ, &mapping->flags); +} + static inline void mapping_set_exiting(struct address_space mapping) { set_bit(AS_EXITING, &mapping->flags); diff --git a/include/linux/swap.h b/include/linux/swap.h index 5658a1634b85e..67aa5271ae6cb 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -295,7 +295,7 @@ extern unsigned long totalreserve_pages; / linux/mm/folio.c / void folio_add_lru(struct folio folio); -void folio_mark_accessed(struct folio folio); +bool folio_mark_accessed(struct folio folio); void lru_add_drain_all(void); enum lru_cache_drained { diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index 2628ccda076a0..c1a63f070aa71 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -36,6 +36,8 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, FOR_ALL_ZONES(ALLOCSTALL) FOR_ALL_ZONES(PGSCAN_SKIP) PGFREE, PGACTIVATE, PGDEACTIVATE, PGLAZYFREE, + PGFREE_RCU, PGFREE_RCU_EXPEDITE, PGFREE_RCU_DRAIN, + PGFREE_RCU_MARK, PGFAULT, PGMAJFAULT, PGLAZYFREED, PGREUSE, diff --git a/include/trace/events/mmflags.h b/include/trace/events/mmflags.h index 935893e5ea53b..47c3411a75e8f 100644 --- a/include/trace/events/mmflags.h +++ b/include/trace/events/mmflags.h @@ -117,6 +117,12 @@ TRACE_DEFINE_ENUM(___GFP_LAST_BIT); #define IF_HAVE_PG_IDLE(_name) #endif +#ifdef CONFIG_PAGE_RCU_FREE +#define IF_HAVE_PG_RCU_FREE(_name) ,{1UL << PG_##_name, __stringify(_name)} +#else +#define IF_HAVE_PG_RCU_FREE(_name) +#endif + #ifdef CONFIG_ARCH_USES_PG_ARCH_2 #define IF_HAVE_PG_ARCH_2(_name) ,{1UL << PG_##_name, __stringify(_name)} #else @@ -152,6 +158,7 @@ TRACE_DEFINE_ENUM(___GFP_LAST_BIT); DEF_PAGEFLAG_NAME(swapbacked), \ DEF_PAGEFLAG_NAME(unevictable), \ DEF_PAGEFLAG_NAME(dropbehind) \ +IF_HAVE_PG_RCU_FREE(rcu_free) \ IF_HAVE_PG_MLOCK(mlocked) \ IF_HAVE_PG_HWPOISON(hwpoison) \ IF_HAVE_PG_IDLE(idle) \ diff --git a/mm/Kconfig b/mm/Kconfig index 604c58199acbf..1ad1cb7969e79 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -718,6 +718,21 @@ config PCP_BATCH_SCALE_MAX latency. This option sets the upper limit of scale factor to limit the maximum latency. +config PAGE_RCU_FREE + def_bool 64BIT + help + Reads from the page cache can copy from folios that are read + again and again without taking a reference on them, so small + reads of one file from many CPUs do not contend on the folio's + reference count. The cost is that memory freed from such + folios becomes reusable only after an RCU grace period. + + 64-bit only: it takes a page flag, and those are scarce on + 32-bit. + + Writing 0 to lockless_page_cache_read in debugfs turns these + reads off. + config PHYS_ADDR_T_64BIT def_bool 64BIT diff --git a/mm/filemap.c b/mm/filemap.c index 00fd89cf6f550..7021ed2ab6c9c 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -48,6 +48,7 @@ #include <linux/rcupdate_wait.h> #include <linux/sched/mm.h> #include <linux/sysctl.h> +#include <linux/debugfs.h> #include <linux/pgalloc.h> #include <asm/tlbflush.h> @@ -138,6 +139,9 @@ static void page_cache_delete(struct address_space mapping, nr = folio_nr_pages(folio); VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); + / No lockless reader can be in an evicted inode / + if (folio_test_rcu_free(folio) && mapping_exiting(mapping)) + folio_clear_rcu_free(folio); xas_store(&xas, shadow); xas_init_marks(&xas); @@ -306,6 +310,9 @@ static void page_cache_delete_batch(struct address_space mapping, } WARN_ON_ONCE(!folio_test_locked(folio)); + /* No lockless reader can be in an evicted inode / + if (folio_test_rcu_free(folio) && mapping_exiting(mapping)) + folio_clear_rcu_free(folio); folio->mapping = NULL; / Leave folio->index set: truncation lookup relies on it / @@ -2759,6 +2766,101 @@ static inline bool pos_same_folio(loff_t pos1, loff_t pos2, struct folio folio) return (pos1 >> shift == pos2 >> shift); } +#ifdef CONFIG_PAGE_RCU_FREE +/* Writing 0 to lockless_page_cache_read in debugfs turns the reads off / +static DEFINE_STATIC_KEY_TRUE(lockless_read_key); + +static bool lockless_read_enabled(void) +{ + return static_branch_likely(&lockless_read_key); +} + +#ifdef CONFIG_DEBUG_FS +static int lockless_read_get(void data, u64 val) +{ + val = static_key_enabled(&lockless_read_key); + return 0; +} + +static int lockless_read_set(void data, u64 val) +{ + if (val > 1) + return -EINVAL; + if (val) + static_branch_enable(&lockless_read_key); + else + static_branch_disable(&lockless_read_key); + return 0; +} +DEFINE_DEBUGFS_ATTRIBUTE(lockless_read_fops, lockless_read_get, + lockless_read_set, "%llu\n"); + +static int __init lockless_read_debugfs_init(void) +{ + debugfs_create_file_unsafe("lockless_page_cache_read", 0644, NULL, NULL, + &lockless_read_fops); + return 0; +} +late_initcall(lockless_read_debugfs_init); +#endif +#else +static bool lockless_read_enabled(void) +{ + return false; +} +#endif + +/ + * Reads keep coming back to @folio: free it only after an RCU grace period + * from now on. Not if its next read needs a reference anyway. Not if it + * is unevictable: folio_mark_accessed() calls such a folio hot after at + * most one access, so every ramfs folio that is read would get the flag. + / +static void filemap_mark_hot(struct folio folio) +{ + struct address_space mapping; + + if (!lockless_read_enabled()) + return; + if (folio_test_rcu_free(folio) \|\| !folio_test_uptodate(folio) \|\| + folio_test_dropbehind(folio) \|\| folio_test_unevictable(folio)) + return; + / shmem does no readahead: there the bit is PG_reclaim / + if (folio_test_readahead(folio) && !folio_test_swapbacked(folio)) + return; + if (folio_test_set_rcu_free(folio)) + return; + + count_vm_events(PGFREE_RCU_MARK, folio_nr_pages(folio)); + / NULL if the folio was truncated meanwhile / + mapping = READ_ONCE(folio->mapping); + if (mapping) + mapping_set_rcu_read(mapping); +} + +/ + * filemap_read() is about to copy from @folio at @pos. @last_pos is where + * the previous read of this open file ended, or how far this one has got. + * + * Reading on in the folio that the last read ended in is not another access + * to it: a folio read in small pieces is marked accessed once. + * + * The folio is hot when folio_mark_accessed() finds it as accessed as the + * LRU records, or when this open file goes back to data that it has just + * read. + / +void filemap_read_accessed(struct folio folio, loff_t pos, loff_t last_pos) +{ + bool hot; + + if (pos_same_folio(pos, last_pos - 1, folio)) + hot = pos < last_pos; + else + hot = folio_mark_accessed(folio); + if (hot) + filemap_mark_hot(folio); +} + static void filemap_end_dropbehind_read(struct folio folio) { if (!folio_test_dropbehind(folio)) @@ -2771,6 +2873,216 @@ static void filemap_end_dropbehind_read(struct folio folio) } } +/* + * Whether filemap_read_rcu_batch() can copy from @folio, or has to leave it + * to the slow path, which holds a reference for readahead, dropbehind and + * folio_mark_accessed(), and deals with poisoned pages. @last_index is the + * page the previous copy ended in. + / +static bool filemap_read_rcu_can_copy(struct folio folio, pgoff_t index, + pgoff_t last_index) +{ + if (!folio_test_uptodate(folio) \|\| folio_test_dropbehind(folio)) + return false; + /* shmem does no readahead: there the bit is PG_reclaim / + if (folio_test_readahead(folio) && !folio_test_swapbacked(folio)) + return false; + / memory_failure() waits only for readers that got past this / + if (folio_contain_hwpoisoned_page(folio)) + return false; + + / + * The slow path does not call folio_mark_accessed() on the folio that + * the previous read ended in, so copying from it skips nothing. The + * folio's size can change without a reference: compare pages instead. + / + if (index == last_index) + return true; + + / Otherwise only if folio_mark_accessed() would leave it as it is / + if (lru_gen_enabled()) { + unsigned long all = LRU_REFS_FLAGS \| BIT(PG_workingset); + + return folio_test_unevictable(folio) \|\| + (READ_ONCE(folio->flags.f) & all) == all; + } + return folio_test_referenced(folio) && !folio_test_idle(folio) && + (folio_test_active(folio) \|\| folio_test_unevictable(folio)); +} + +/ + * Return the folio at @pos if it is this file's page until rcu_read_unlock(), + * and read i_size into @isize. A folio with PG_rcu_free is freed only after + * an RCU grace period. Without a reference, the folio that xas_load() + * returned can be freed and its memory reused as another folio before the + * flag is tested, so look at the slot again after the flag. + / +static struct folio filemap_read_rcu_lookup(struct xa_state xas, + struct inode inode, loff_t pos, loff_t isize) +{ + struct folio folio; + + xas_set(xas, pos >> PAGE_SHIFT); + folio = xas_load(xas); + if (xas_retry(xas, folio) \|\| !folio \|\| xa_is_value(folio)) + return NULL; + + /* Not folio_test_rcu_free(): it may not be a folio any more / + if (!(READ_ONCE(folio->flags.f) & __PG_RCU_FREE)) + return NULL; + / Read i_size and slot after the flag: it implies uptodate / + smp_rmb(); + isize = i_size_read(inode); + if (pos >= isize) + return NULL; + / i_size_read() is an acquire; a write fills the slot first / + if (xas_reload(xas) != folio) + return NULL; + / Read the rest of the folio only once the slot holds it / + smp_rmb(); + return folio; +} + +/ + * How much of @bytes, starting @offset into page @index, can be copied from + * @folio. A folio is in all the slots of its range or in none: when the slot + * of the last page holds it too, so do the slots in between, and the pages + * are this folio's. + / +static size_t filemap_read_rcu_size(struct xa_state xas, struct folio folio, + pgoff_t index, size_t offset, size_t bytes) +{ + pgoff_t end = index + DIV_ROUND_UP(offset + bytes, PAGE_SIZE); + pgoff_t next; + + if (end - index > 1 && folio_test_large(folio)) { + / Read without a reference: it only picks the page to check / + next = folio_next_index(folio); + if (next > index && next < end) + end = next; + xas_set(xas, end - 1); + / A split after this copies PG_rcu_free to the new folios / + if (xas_load(xas) == folio) + return min(bytes, (end - index) PAGE_SIZE - offset); + } + return min(bytes, PAGE_SIZE - offset); +} + +/* + * Copy @bytes from @folio, starting @offset into its page at @index. Page + * by page, as copy_page_to_iter() does, without its checks for each call. + * Not copy_to_iter(): its usercopy check reads the folio's order, which a + * split can change. + / +static size_t filemap_read_rcu_copy(struct folio folio, pgoff_t index, + size_t offset, size_t bytes, struct iov_iter iter) +{ + struct page page = folio_page(folio, index - folio->index); + size_t n = 0; + + while (n < bytes) { + size_t len = min(bytes - n, PAGE_SIZE - offset); + size_t done; + void kaddr; + + kaddr = kmap_local_page(page); + done = _copy_to_iter(kaddr + offset, len, iter); + kunmap_local(kaddr); + n += done; + if (done < len) + break; + offset = 0; + page++; + } + return n; +} + +/ Bytes one RCU read section copies at most: it holds up grace periods / +#define FILEMAP_RCU_READ_MAX SZ_512K + +/ + * Copy from folios with PG_rcu_free without taking references, at most + * FILEMAP_RCU_READ_MAX bytes under one rcu_read_lock(). Return true if that + * much was copied and the read is not done. last_index is the page the + previous copy ended in, set as the slow path sets last_pos. + / +static bool filemap_read_rcu_batch(struct kiocb iocb, struct iov_iter iter, + pgoff_t last_index, size_t copied) +{ + struct address_space mapping = iocb->ki_filp->f_mapping; + struct inode inode = mapping->host; + XA_STATE(xas, &mapping->i_pages, 0); + size_t budget = FILEMAP_RCU_READ_MAX; + bool more = false; + + / flush_dcache_folio() is not known to be safe without a reference / + if (ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE && + mapping_writably_mapped(mapping)) + return false; + + / A user fault cannot sleep in the RCU read section: it ends the copy / + rcu_read_lock(); + pagefault_disable(); + while (iov_iter_count(iter)) { + loff_t pos = iocb->ki_pos; + pgoff_t index = pos >> PAGE_SHIFT; + size_t offset = offset_in_page(pos); + struct folio folio; + size_t bytes, n; + loff_t isize; + + if (!budget) { + more = true; + break; + } + folio = filemap_read_rcu_lookup(&xas, inode, pos, &isize); + if (!folio) + break; + if (!filemap_read_rcu_can_copy(folio, index, last_index)) + break; + + bytes = min_t(loff_t, isize - pos, iov_iter_count(iter)); + bytes = filemap_read_rcu_size(&xas, folio, index, offset, + min(bytes, budget)); + n = filemap_read_rcu_copy(folio, index, offset, bytes, iter); + copied += n; + iocb->ki_pos += n; + budget -= n; + if (n) + last_index = (pos + n - 1) >> PAGE_SHIFT; + if (n < bytes) + break; + } + pagefault_enable(); + rcu_read_unlock(); + + return more; +} + +/ + * Copy from the page cache without folio references, in batches, and + * return how much was copied. The slow path reads the rest. + / +size_t filemap_read_rcu(struct kiocb iocb, struct iov_iter iter, + loff_t last_pos) +{ + / The page the previous read ended in; ULONG_MAX if none / + pgoff_t last_index = (last_pos - 1) >> PAGE_SHIFT; + size_t copied = 0; + + / Not when switched off, or on an inode that never had a folio flagged / + if (!lockless_read_enabled() \|\| + !mapping_rcu_read(iocb->ki_filp->f_mapping)) + return 0; + + while (filemap_read_rcu_batch(iocb, iter, &last_index, &copied)) { + if (fatal_signal_pending(current)) + break; + cond_resched(); + } + return copied; +} + /* * filemap_read - Read data from the page cache. * @iocb: The iocb to read. @@ -2796,6 +3108,7 @@ ssize_t filemap_read(struct kiocb iocb, struct iov_iter iter, bool writably_mapped; loff_t isize, end_offset; loff_t last_pos = ra->prev_pos; + size_t fast_read; if (unlikely(iocb->ki_pos < 0)) return -EINVAL; @@ -2807,6 +3120,14 @@ ssize_t filemap_read(struct kiocb iocb, struct iov_iter iter, iov_iter_truncate(iter, inode->i_sb->s_maxbytes - iocb->ki_pos); folio_batch_init(&fbatch); + fast_read = filemap_read_rcu(iocb, iter, last_pos); + if (fast_read) { + already_read += fast_read; + last_pos = iocb->ki_pos; + if (!iov_iter_count(iter)) + goto out; + } + do { cond_resched(); @@ -2844,14 +3165,6 @@ ssize_t filemap_read(struct kiocb iocb, struct iov_iter iter, / writably_mapped = mapping_writably_mapped(mapping); - / - * When a read accesses the same folio several times, only - * mark it as accessed the first time. - / - if (!pos_same_folio(iocb->ki_pos, last_pos - 1, - fbatch.folios[0])) - folio_mark_accessed(fbatch.folios[0]); - for (i = 0; i < folio_batch_count(&fbatch); i++) { struct folio folio = fbatch.folios[i]; size_t fsize = folio_size(folio); @@ -2862,8 +3175,7 @@ ssize_t filemap_read(struct kiocb iocb, struct iov_iter iter, if (end_offset < folio_pos(folio)) break; - if (i > 0) - folio_mark_accessed(folio); + filemap_read_accessed(folio, iocb->ki_pos, last_pos); /* * If users can be writing to this folio using arbitrary * virtual addresses, take care of potential aliasing @@ -2893,8 +3205,11 @@ ssize_t filemap_read(struct kiocb iocb, struct iov_iter iter, folio_batch_init(&fbatch); } while (iov_iter_count(iter) && iocb->ki_pos < isize && !error); +out: file_accessed(filp); - ra->prev_pos = last_pos; + /* Readers sharing the file would bounce the line for nothing / + if (ra->prev_pos != last_pos) + ra->prev_pos = last_pos; return already_read ? already_read : error; } EXPORT_SYMBOL_GPL(filemap_read); diff --git a/mm/folio.c b/mm/folio.c index 50a6dbe55998e..6144b9cf301b3 100644 --- a/mm/folio.c +++ b/mm/folio.c @@ -352,28 +352,34 @@ static void __lru_cache_activate_folio(struct folio folio) #ifdef CONFIG_LRU_GEN -static void lru_gen_inc_refs(struct folio folio) +/ + * Return true if @folio is hot: unevictable, or its flags already held as + * many accesses as they can. + / +static bool lru_gen_inc_refs(struct folio folio) { unsigned long new_flags, old_flags = READ_ONCE(folio->flags.f); if (folio_test_unevictable(folio)) - return; + return true; /* see the comment on LRU_REFS_FLAGS / if (!folio_test_referenced(folio)) { set_mask_bits(&folio->flags.f, LRU_REFS_MASK, BIT(PG_referenced)); - return; + return false; } do { if ((old_flags & LRU_REFS_MASK) == LRU_REFS_MASK) { - if (!folio_test_workingset(folio)) - folio_set_workingset(folio); - return; + if (folio_test_workingset(folio)) + return true; + folio_set_workingset(folio); + return false; } new_flags = old_flags + BIT(LRU_REFS_PGOFF); } while (!try_cmpxchg(&folio->flags.f, &old_flags, new_flags)); + return false; } static bool lru_gen_clear_refs(struct folio folio) @@ -396,8 +402,9 @@ static bool lru_gen_clear_refs(struct folio folio) #else / !CONFIG_LRU_GEN / -static void lru_gen_inc_refs(struct folio folio) +static bool lru_gen_inc_refs(struct folio folio) { + return false; } static bool lru_gen_clear_refs(struct folio folio) @@ -419,15 +426,20 @@ static bool lru_gen_clear_refs(struct folio folio) * When a newly allocated folio is not yet visible, so safe for non-atomic ops, * __folio_set_referenced() may be substituted for folio_mark_accessed(). + * + * Return: true if @folio is hot: it was already as accessed as the LRU + * records. On the classic LRU that is active or unevictable, referenced + * and not idle; on the multi-gen LRU, unevictable, or PG_referenced, all + * the refs bits and PG_workingset set. False for a dropbehind folio. / -void folio_mark_accessed(struct folio folio) +bool folio_mark_accessed(struct folio folio) { + bool hot = false; + if (folio_test_dropbehind(folio)) - return; - if (lru_gen_enabled()) { - lru_gen_inc_refs(folio); - return; - } + return false; + if (lru_gen_enabled()) + return lru_gen_inc_refs(folio); if (!folio_test_referenced(folio)) { folio_set_referenced(folio); @@ -437,6 +449,7 @@ void folio_mark_accessed(struct folio folio) * this list is never rotated or maintained, so marking an * unevictable page accessed has no effect. / + hot = true; } else if (!folio_test_active(folio)) { / * If the folio is on the LRU, queue it for activation via @@ -450,9 +463,14 @@ void folio_mark_accessed(struct folio folio) __lru_cache_activate_folio(folio); folio_clear_referenced(folio); workingset_activation(folio); + } else { + hot = true; } - if (folio_test_idle(folio)) + if (folio_test_idle(folio)) { folio_clear_idle(folio); + hot = false; + } + return hot; } EXPORT_SYMBOL(folio_mark_accessed); diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 1e5d68acf62a5..25082949a4cae 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3743,6 +3743,7 @@ static void __split_folio_to_order(struct folio folio, int old_order, #endif (1L << PG_dirty) \| (1L << PG_dropbehind) \| + __PG_RCU_FREE \| LRU_GEN_MASK \| LRU_REFS_MASK)); new_folio->mapping = folio->mapping; diff --git a/mm/internal.h b/mm/internal.h index 38b1165212c94..ae9a4e511c827 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1248,6 +1248,9 @@ struct migration_target_control { / size_t splice_folio_into_pipe(struct pipe_inode_info pipe, struct folio folio, loff_t fpos, size_t size); +void filemap_read_accessed(struct folio folio, loff_t pos, loff_t last_pos); +size_t filemap_read_rcu(struct kiocb iocb, struct iov_iter iter, + loff_t last_pos); static inline bool vma_is_single_threaded_private(struct vm_area_struct vma) { diff --git a/mm/memory-failure.c b/mm/memory-failure.c index a8b03e2920ba8..dc66cac05b1f5 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -211,6 +211,10 @@ static bool page_handle_poison(struct page page, bool hugepage_or_freepage, boo return false; } + /* The free of a hwpoisoned page is not deferred: wait for readers / + if (release && folio_test_rcu_free(page_folio(page))) + synchronize_rcu(); + SetPageHWPoison(page); if (release) put_page(page); @@ -1561,6 +1565,21 @@ static int get_hwpoison_page(struct page p, unsigned long flags) return ret; } +/* A page waiting for a grace period is not in the buddy yet: free it / +static void mf_free_rcu_pages(struct page p) +{ + if (!IS_ENABLED(CONFIG_PAGE_RCU_FREE)) + return; + if (page_count(p)) + return; + if (!zone_page_state_snapshot(page_zone(p), NR_FREE_RCU_PAGES)) + return; + if (is_free_buddy_page(p)) + return; + + rcu_free_sync(); +} + /* * The caller must guarantee the folio isn't large folio, except hugetlb. * try_to_unmap() can't handle it. @@ -2436,6 +2455,7 @@ int memory_failure(unsigned long pfn, int flags) if (res != -ENOENT) goto unlock_mutex; + mf_free_rcu_pages(p); if (TestSetPageHWPoison(p)) { res = -EHWPOISON; if (flags & MF_ACTION_REQUIRED) @@ -2608,6 +2628,13 @@ int memory_failure(unsigned long pfn, int flags) identify_page_state: res = identify_page_state(pfn, p, page_flags); + /* + * The caller may unmap the page once this returns 0. Lockless + * readers that started before the page got PG_hwpoison, or before a + * split left it in a folio of its own, may still be copying it. + / + if (!res && folio_test_rcu_free(folio)) + synchronize_rcu(); mutex_unlock(&mf_mutex); return res; unlock_page: @@ -2999,6 +3026,7 @@ int soft_offline_page(unsigned long pfn, int flags) retry: get_online_mems(); + mf_free_rcu_pages(page); ret = get_hwpoison_page(page, flags \| MF_SOFT_OFFLINE); put_online_mems(); diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index 226ab9cb078ad..a8f0c95627ee1 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -1967,6 +1967,7 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages, .nid = NUMA_NO_NODE, }; unsigned long flags; + bool migrated; char reason; int ret; unsigned long normal_pages = 0; @@ -2050,6 +2051,7 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages, do { pfn = start_pfn; + migrated = false; do { /* * Historically we always checked for any signal and @@ -2071,6 +2073,7 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages, * out / do_migrate_range(pfn, end_pfn); + migrated = true; } } while (!ret); @@ -2092,6 +2095,10 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages, ret = test_pages_isolated(start_pfn, end_pfn, PB_ISOLATE_MODE_MEM_OFFLINE); + / What this pass migrated can wait for a grace period / + if (ret && migrated && + zone_page_state_snapshot(zone, NR_FREE_RCU_PAGES)) + rcu_free_sync(); } while (ret); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 12fac9084c483..e63628e64cbba 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1313,6 +1313,199 @@ static inline void pgalloc_tag_sub_pages(struct alloc_tag tag, unsigned int nr) #endif /* CONFIG_MEM_ALLOC_PROFILING / +#define RCU_FREE_EXPEDITE (SZ_4M >> PAGE_SHIFT) + +/ + * Each CPU has at most one grace period in flight (@queued), covering + * @batch. Folios freed meanwhile wait on @pending for the next one. + / +struct rcu_free_pcpu { + spinlock_t lock; + bool queued; + struct list_head pending; + struct list_head batch; + unsigned long nr_pending; + struct rcu_work rwork; + struct work_struct expedite; +}; + +static DEFINE_PER_CPU(struct rcu_free_pcpu, rcu_free_pcpu); + +static struct workqueue_struct rcu_free_wq __ro_after_init; + +static void rcu_free_start(struct rcu_free_pcpu r) +{ + lockdep_assert_held(&r->lock); + + if (r->queued \|\| list_empty(&r->pending)) + return; + + list_splice_init(&r->pending, &r->batch); + r->nr_pending = 0; + r->queued = true; + WARN_ON_ONCE(!queue_rcu_work(rcu_free_wq, &r->rwork)); +} + +static void rcu_free_take(struct rcu_free_pcpu r, struct list_head list) +{ + lockdep_assert_held(&r->lock); + + list_splice_tail_init(&r->batch, list); + list_splice_tail_init(&r->pending, list); + r->nr_pending = 0; +} + +static unsigned long rcu_free_pages(struct list_head list) +{ + struct folio_batch fbatch; + struct folio folio, next; + unsigned long nr = 0; + + folio_batch_init(&fbatch); + list_for_each_entry_safe(folio, next, list, lru) { + nr += folio_nr_pages(folio); + mod_zone_page_state(folio_zone(folio), NR_FREE_RCU_PAGES, + -folio_nr_pages(folio)); + if (!folio_batch_add(&fbatch, folio)) { + free_unref_folios(&fbatch); + cond_resched_tasks_rcu_qs(); + } + } + if (folio_batch_count(&fbatch)) + free_unref_folios(&fbatch); + return nr; +} + +/* Free what rcu_free_take() took: the grace period starts after that / +static unsigned long rcu_free_flush(struct list_head list) +{ + if (list_empty(list)) + return 0; + + synchronize_rcu_expedited(); + return rcu_free_pages(list); +} + +static void rcu_free_workfn(struct work_struct work) +{ + struct rcu_free_pcpu r = container_of(to_rcu_work(work), + struct rcu_free_pcpu, rwork); + LIST_HEAD(done); + + spin_lock_irq(&r->lock); + list_splice_init(&r->batch, &done); + r->queued = false; + rcu_free_start(r); + spin_unlock_irq(&r->lock); + + rcu_free_pages(&done); +} + +static void rcu_free_expedite_workfn(struct work_struct work) +{ + struct rcu_free_pcpu r = container_of(work, struct rcu_free_pcpu, + expedite); + LIST_HEAD(list); + + spin_lock_irq(&r->lock); + rcu_free_take(r, &list); + spin_unlock_irq(&r->lock); + + if (rcu_free_flush(&list)) + count_vm_event(PGFREE_RCU_EXPEDITE); +} + +static void rcu_free_defer(struct folio folio) +{ + struct rcu_free_pcpu r = raw_cpu_ptr(&rcu_free_pcpu); + struct zone zone = folio_zone(folio); + unsigned long free, waiting; + unsigned long flags; + + spin_lock_irqsave(&r->lock, flags); + __mod_zone_page_state(zone, NR_FREE_RCU_PAGES, folio_nr_pages(folio)); + list_add_tail(&folio->lru, &r->pending); + r->nr_pending += folio_nr_pages(folio); + rcu_free_start(r); + spin_unlock_irqrestore(&r->lock, flags); + + / Unlocked: a stale count delays the expedite by one free / + if (data_race(r->nr_pending) <= RCU_FREE_EXPEDITE \|\| + work_pending(&r->expedite)) + return; + + / + * An expedited grace period interrupts every CPU: only while the + * zone is short of memory, or while more than an eighth of what it + * has free waits here. + / + free = zone_page_state(zone, NR_FREE_PAGES); + waiting = zone_page_state(zone, NR_FREE_RCU_PAGES); + if (free < high_wmark_pages(zone) \|\| waiting > free / 8) + queue_work(rcu_free_wq, &r->expedite); +} + +/ + * Free what every CPU's lists hold. A concurrent drain holds the lock + * until it has freed what it took. + / +static void rcu_free_drain(void) +{ + static DEFINE_MUTEX(lock); + LIST_HEAD(list); + int cpu; + + mutex_lock(&lock); + for_each_possible_cpu(cpu) { + struct rcu_free_pcpu r = per_cpu_ptr(&rcu_free_pcpu, cpu); + + spin_lock_irq(&r->lock); + rcu_free_take(r, &list); + spin_unlock_irq(&r->lock); + } + if (rcu_free_flush(&list)) + count_vm_event(PGFREE_RCU_DRAIN); + mutex_unlock(&lock); +} + +/* Free every waiting folio, including those a work took off the lists / +void rcu_free_sync(void) +{ + int cpu; + + if (!IS_ENABLED(CONFIG_PAGE_RCU_FREE)) + return; + + rcu_free_drain(); + for_each_possible_cpu(cpu) { + struct rcu_free_pcpu r = per_cpu_ptr(&rcu_free_pcpu, cpu); + + flush_work(&r->rwork.work); + flush_work(&r->expedite); + } +} + +static void __init rcu_free_init(void) +{ + int cpu; + + if (!IS_ENABLED(CONFIG_PAGE_RCU_FREE)) + return; + + rcu_free_wq = alloc_workqueue("rcu_free", + WQ_MEM_RECLAIM \| WQ_PERCPU, 0); + + for_each_possible_cpu(cpu) { + struct rcu_free_pcpu r = per_cpu_ptr(&rcu_free_pcpu, cpu); + + spin_lock_init(&r->lock); + INIT_LIST_HEAD(&r->pending); + INIT_LIST_HEAD(&r->batch); + INIT_RCU_WORK(&r->rwork, rcu_free_workfn); + INIT_WORK(&r->expedite, rcu_free_expedite_workfn); + } +} + static __always_inline bool __free_pages_prepare(struct page page, unsigned int order, fpi_t fpi_flags) { @@ -1327,6 +1520,19 @@ static __always_inline bool __free_pages_prepare(struct page page, VM_BUG_ON_PAGE(PageTail(page), page); + / + * Not for a hwpoisoned order-0 page: it never reaches the allocator, + * and page_handle_poison() takes a reference right after this free. + / + if (unlikely(folio_test_rcu_free(folio)) && + !(PageHWPoison(page) && !order)) { + VM_WARN_ON_ONCE(fpi_flags & FPI_NOLOCK); + __folio_clear_rcu_free(folio); + count_vm_events(PGFREE_RCU, 1 << order); + rcu_free_defer(folio); + return false; + } + trace_mm_page_free(page, order); kmsan_free_page(page, order); @@ -4626,6 +4832,52 @@ bool gfp_pfmemalloc_allowed(gfp_t gfp_mask) return !!__gfp_pfmemalloc_flags(gfp_mask); } +/ + * Reclaim is about to give up: if the pages waiting for a grace period + * would bring an eligible zone above its min watermark, free them and + * retry. Not for a dying task, which frees memory by exiting. + / +static bool rcu_free_wait(gfp_t gfp_mask, unsigned int order, + struct alloc_context ac, int alloc_flags, + int rcu_free_waits) +{ + struct zone zone; + struct zoneref z; + + if (!IS_ENABLED(CONFIG_PAGE_RCU_FREE)) + return false; + if (tsk_is_oom_victim(current) \|\| fatal_signal_pending(current)) + return false; + if (rcu_free_waits >= MAX_RECLAIM_RETRIES \|\| + order > PAGE_ALLOC_COSTLY_ORDER) + return false; + + for_each_zone_zonelist_nodemask(zone, z, ac->zonelist, + ac->highest_zoneidx, ac->nodemask) { + unsigned long waiting, available; + + if (cpusets_enabled() && (alloc_flags & ALLOC_CPUSET) && + !__cpuset_zone_allowed(zone, gfp_mask)) + continue; + + waiting = zone_page_state_snapshot(zone, NR_FREE_RCU_PAGES); + if (!waiting) + continue; + + available = waiting + + zone_page_state_snapshot(zone, NR_FREE_PAGES); + if (__zone_watermark_ok(zone, order, min_wmark_pages(zone), + ac->highest_zoneidx, alloc_flags, + available)) { + (rcu_free_waits)++; + rcu_free_drain(); + return true; + } + } + + return false; +} + / * Checks whether it makes sense to retry the reclaim to make a forward progress * for the given allocation request. @@ -4639,7 +4891,8 @@ bool gfp_pfmemalloc_allowed(gfp_t gfp_mask) static inline bool should_reclaim_retry(gfp_t gfp_mask, unsigned order, struct alloc_context ac, int alloc_flags, - bool did_some_progress, int no_progress_loops) + bool did_some_progress, int no_progress_loops, + int rcu_free_waits) { struct zone zone; struct zoneref z; @@ -4650,10 +4903,12 @@ should_reclaim_retry(gfp_t gfp_mask, unsigned order, * their order will become available due to high fragmentation so * always increment the no progress counter for them / - if (did_some_progress && order <= PAGE_ALLOC_COSTLY_ORDER) + if (did_some_progress && order <= PAGE_ALLOC_COSTLY_ORDER) { no_progress_loops = 0; - else + rcu_free_waits = 0; + } else { (no_progress_loops)++; + } if (no_progress_loops > MAX_RECLAIM_RETRIES) goto out; @@ -4706,6 +4961,10 @@ should_reclaim_retry(gfp_t gfp_mask, unsigned order, else cond_resched(); out: + if (!ret && rcu_free_wait(gfp_mask, order, ac, alloc_flags, + rcu_free_waits)) + return true; + / Before OOM, exhaust highatomic_reserve / if (!ret) return unreserve_highatomic_pageblock(ac, true); @@ -4795,6 +5054,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, enum compact_result compact_result; int compaction_retries; int no_progress_loops; + int rcu_free_waits; unsigned int cpuset_mems_cookie; unsigned int zonelist_iter_cookie; int reserve_flags; @@ -4819,6 +5079,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, restart: compaction_retries = 0; no_progress_loops = 0; + rcu_free_waits = 0; compact_result = COMPACT_SKIPPED; compact_priority = DEF_COMPACT_PRIORITY; cpuset_mems_cookie = read_mems_allowed_begin(); @@ -4997,7 +5258,8 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, goto restart; if (should_reclaim_retry(gfp_mask, order, ac, alloc_flags, - did_some_progress > 0, &no_progress_loops)) + did_some_progress > 0, &no_progress_loops, + &rcu_free_waits)) goto retry; / @@ -5040,6 +5302,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, /* Retry as long as the OOM killer is making progress / if (did_some_progress) { no_progress_loops = 0; + rcu_free_waits = 0; goto retry; } @@ -6401,6 +6664,8 @@ void __init setup_per_cpu_pageset(void) for_each_online_pgdat(pgdat) pgdat->per_cpu_nodestats = alloc_percpu(struct per_cpu_nodestat); + + rcu_free_init(); } __meminit void zone_pcp_init(struct zone zone) @@ -7384,7 +7649,16 @@ int alloc_contig_frozen_range_noprof(unsigned long start, unsigned long end, outer_start = find_large_buddy(start); /* Make sure the range is really isolated. / - if (test_pages_isolated(outer_start, end, mode)) { + ret = test_pages_isolated(outer_start, end, mode); + if (ret && zone_page_state_snapshot(cc.zone, NR_FREE_RCU_PAGES)) { + rcu_free_sync(); + + / A freed folio can merge with a free buddy below start / + outer_start = find_large_buddy(start); + ret = test_pages_isolated(outer_start, end, mode); + } + + if (ret) { ret = -EBUSY; goto done; } diff --git a/mm/page_alloc.h b/mm/page_alloc.h index b9259deddb59d..f9e200581e2f5 100644 --- a/mm/page_alloc.h +++ b/mm/page_alloc.h @@ -235,6 +235,7 @@ struct page __alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order, int nid, alloc_hooks(__alloc_frozen_pages_noprof(__VA_ARGS__)) void free_frozen_pages(struct page page, unsigned int order); void free_unref_folios(struct folio_batch fbatch); +void rcu_free_sync(void); #ifdef CONFIG_NUMA struct page alloc_frozen_pages_noprof(gfp_t, unsigned int order); diff --git a/mm/shmem.c b/mm/shmem.c index 848316eaa7f4f..e6c18e745f816 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -3309,10 +3309,18 @@ static ssize_t shmem_file_read_iter(struct kiocb iocb, struct iov_iter to) struct file file = iocb->ki_filp; struct inode inode = file_inode(file); struct address_space mapping = inode->i_mapping; + loff_t last_pos = file->f_ra.prev_pos; pgoff_t index; unsigned long offset; int error = 0; - ssize_t retval = 0; + ssize_t retval; + + retval = filemap_read_rcu(iocb, to, last_pos); + if (retval) { + last_pos = iocb->ki_pos; + if (!iov_iter_count(to)) + goto out; + } for (;;) { struct folio folio = NULL; @@ -3378,11 +3386,7 @@ static ssize_t shmem_file_read_iter(struct kiocb iocb, struct iov_iter to) flush_dcache_page(page); } - / - * Mark the folio accessed if we read the beginning. - / - if (!offset) - folio_mark_accessed(folio); + filemap_read_accessed(folio, iocb->ki_pos, last_pos); / * Ok, we have the page, and it's up-to-date, so * now we can copy it to user space... @@ -3410,6 +3414,7 @@ static ssize_t shmem_file_read_iter(struct kiocb iocb, struct iov_iter to) retval += ret; iocb->ki_pos += ret; + last_pos = iocb->ki_pos; if (!iov_iter_count(to)) break; @@ -3419,8 +3424,11 @@ static ssize_t shmem_file_read_iter(struct kiocb iocb, struct iov_iter to) } cond_resched(); } - +out: file_accessed(file); + /* Readers sharing the file would bounce the line for nothing / + if (file->f_ra.prev_pos != last_pos) + file->f_ra.prev_pos = last_pos; return retval ? retval : error; } diff --git a/mm/show_mem.c b/mm/show_mem.c index b938cbcd774a1..0d3bddbda108d 100644 --- a/mm/show_mem.c +++ b/mm/show_mem.c @@ -204,7 +204,7 @@ static void show_free_areas(unsigned int filter, const nodemask_t nodemask, " mapped:%lu shmem:%lu pagetables:%lu\n" " sec_pagetables:%lu bounce:%lu\n" " kernel_misc_reclaimable:%lu\n" - " free:%lu free_pcp:%lu free_cma:%lu\n", + " free:%lu free_pcp:%lu free_cma:%lu free_rcu:%lu\n", global_node_page_state(NR_ACTIVE_ANON), global_node_page_state(NR_INACTIVE_ANON), global_node_page_state(NR_ISOLATED_ANON), @@ -224,7 +224,8 @@ static void show_free_areas(unsigned int filter, const nodemask_t nodemask, global_node_page_state(NR_KERNEL_MISC_RECLAIMABLE), global_zone_page_state(NR_FREE_PAGES), free_pcp, - global_zone_page_state(NR_FREE_CMA_PAGES)); + global_zone_page_state(NR_FREE_CMA_PAGES), + global_zone_page_state(NR_FREE_RCU_PAGES)); for_each_online_pgdat(pgdat) { if (show_mem_node_skip(filter, pgdat->node_id, nodemask)) @@ -325,6 +326,7 @@ static void show_free_areas(unsigned int filter, const nodemask_t nodemask, " free_pcp:%lukB" " local_pcp:%lukB" " free_cma:%lukB" + " free_rcu:%lukB" "\n", zone->name, K(zone_page_state(zone, NR_FREE_PAGES)), @@ -351,7 +353,8 @@ static void show_free_areas(unsigned int filter, const nodemask_t nodemask, 0UL, K(free_pcp), K((unsigned long)this_cpu_read(zone->per_cpu_pageset->count)), - K(zone_page_state(zone, NR_FREE_CMA_PAGES))); + K(zone_page_state(zone, NR_FREE_CMA_PAGES)), + K(zone_page_state(zone, NR_FREE_RCU_PAGES))); printk("lowmem_reserve[]:"); for (i = 0; i < MAX_NR_ZONES; i++) printk(KERN_CONT " %ld", zone->lowmem_reserve[i]); diff --git a/mm/vmstat.c b/mm/vmstat.c index cb57714539fb5..c27caad017e35 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1212,6 +1212,7 @@ const char const vmstat_text[] = { [I(NR_ZSPAGES)] = "nr_zspages", #endif [I(NR_FREE_CMA_PAGES)] = "nr_free_cma", + [I(NR_FREE_RCU_PAGES)] = "nr_free_rcu", #ifdef CONFIG_UNACCEPTED_MEMORY [I(NR_UNACCEPTED)] = "nr_unaccepted", #endif @@ -1343,6 +1344,10 @@ const char * const vmstat_text[] = { [I(PGACTIVATE)] = "pgactivate", [I(PGDEACTIVATE)] = "pgdeactivate", [I(PGLAZYFREE)] = "pglazyfree", + [I(PGFREE_RCU)] = "pgfree_rcu", + [I(PGFREE_RCU_EXPEDITE)] = "pgfree_rcu_expedite", + [I(PGFREE_RCU_DRAIN)] = "pgfree_rcu_drain", + [I(PGFREE_RCU_MARK)] = "pgfree_rcu_mark", [I(PGFAULT)] = "pgfault", [I(PGMAJFAULT)] = "pgmajfault", @@ -2022,6 +2027,7 @@ static int vmstat_refresh(const struct ctl_table *table, int write, switch (i) { case NR_ZONE_WRITE_PENDING: case NR_FREE_CMA_PAGES: + case NR_FREE_RCU_PAGES: continue; } val = atomic_long_read(&vm_zone_stat[i]);
3/2	2026/10/19 19:57	llm	patch-evaluator	0m Model: gemini-3.8-flash Args: null Results: null Tokens: input: 20822 output: 32 thoughts: 1231 Thoughts: Analyzing the Lockless Page Cache Reads and RCU Page Freeing Patch Okay, let's dive into this patch. My immediate thought is, wow, this is a significant undertaking, essentially introducing lockless page cache reads and RCU-based page freeing. This touches core memory management and the VFS layer, which means it will affect all standard filesystems and shmem! We are talking about major changes to `filemap_read`, `__free_pages_prepare`, and reclaim retry logic, page splitting... a massive undertaking. I see it's enabling `PAGE_RCU_FREE` by default for 64-bit architectures (which is probably most of our targets). That means this is going to be exercised... a lot. Standard read() syscalls on regular files (ext4, tmpfs/shmem, etc.) will trigger this new path. Good to know it's reachable -- this is clearly something we need to pay very close attention to. Specifically, I need to understand the implications of: * `fuse_try_move_folio`: introduces rcu checks * `filemap_read_rcu`: fast path that will be used by all normal read operations * `__free_pages_prepare`: introduces rcu handling for page freeing * `filemap_mark_hot`: sets rcu_free and as_rcu_read flags. * The `rcu_free_*` functions in `page_alloc.c`--these seem to be the new infrastructure. My initial assessment? Absolutely, `WorthFuzzing = true`. This is a huge change and has the potential for interesting interactions. Now, the next step is to nail down the precise scope. What are the key non-ubiquitous entrypoints or functions to focus on? What configuration options do we need to ensure this code path is active during testing? Let's get some greps and checks going to get this clarified.
4/2	2026/10/19 19:57	tool	grepper	0m Args: { "Expression": "config PAGE_RCU_FREE", "PathPrefixes": [ "mm/Kconfig" ] } Results: { "Output": "mm/Kconfig=710=config PCP_BATCH_SCALE_MAX\n--\nmm/Kconfig-720-\nmm/Kconfig:721:config PAGE_RCU_FREE\nmm/Kconfig-722-\tdef_bool 64BIT\n" }
5/2	2026/10/19 19:57	llm	patch-evaluator	0m Model: gemini-3.8-flash Args: null Results: null Tokens: input: 22583 output: 36 thoughts: 1222 Thoughts: Analyzing the Impact of CONFIG_PAGE_RCU_FREE Okay, let's dive into this `CONFIG_PAGE_RCU_FREE` thing. First, I need to confirm that this is a 64-bit feature, which checks out given the `def_bool 64BIT` dependency. I'm also looking for any other configuration options that are involved in this change. I see `CONFIG_LRU_GEN` is touched, but that's in `folio.c`, and the code supports both with and without `LRU_GEN`. `CONFIG_DEBUG_FS` is related for a debugfs attribute, but that's not the core. Now, let's check the function symbols that are introduced or modified with this feature. Checking `filemap.c`, I see a couple of functions: `filemap_read_rcu`, `filemap_read_rcu_batch`, `filemap_read_accessed`, `filemap_mark_hot`, and `filemap_read`. The key one here is `filemap_read_rcu` -- that's the entry point! But I need to be sure; is `filemap_read_rcu` actually a symbol in vmlinux? Looking into `mm/filemap.c`, and seeing it's not declared static, and looking at the header, `mm/internal.h`, confirms that it's exported as a global symbol. Great! And, `filemap_read_accessed` is also not static, and it's also declared in `mm/internal.h`. What about `folio.c`? `folio_mark_accessed` is ubiquitous, called everywhere. I need to avoid that per the guidelines. Next, I look at `mm/page_alloc.c` and notice several RCU-related functions: `rcu_free_defer`, `rcu_free_start`, `rcu_free_take`, `rcu_free_pages`, `rcu_free_flush`, `rcu_free_workfn`, `rcu_free_expedite_workfn`, `rcu_free_drain`, `rcu_free_sync`, `rcu_free_wait`, and `rcu_free_init`. Now, most of these are static, but wait... workqueue callbacks! Functions such as `rcu_free_workfn`, and `rcu_free_expedite_workfn` definitely have symbols in vmlinux because their addresses are passed to workqueues, so they cannot be inlined! `rcu_free_sync` is also non-static. I need to note that. Let's see: `filemap_read_rcu` is called inside `shmem_file_read_iter` and `filemap_read`. I also need to note `fs/fuse/dev.c`, which contains `fuse_try_move_folio` with a check for `folio_test_rcu_free`. And let's not forget about memory failures and `soft_offline_page` which call `mf_free_rcu_pages`. I should probably do a quick read on the relevant sections of `mm/page_alloc.c` around these `rcu_free_*` functions too, just to be thorough.
6/2	2026/10/19 19:57	tool	read-file	0m Args: { "File": "mm/page_alloc.c", "FirstLine": 1340, "LineCount": 50 } Results: { "Contents": "1340:\t\tif (r-\u003equeued \|\| list_empty(\u0026r-\u003epending))\n1341:\t\t\treturn;\n1342:\t\n1343:\t\tlist_splice_init(\u0026r-\u003epending, \u0026r-\u003ebatch);\n1344:\t\tr-\u003enr_pending = 0;\n1345:\t\tr-\u003equeued = true;\n1346:\t\tWARN_ON_ONCE(!queue_rcu_work(rcu_free_wq, \u0026r-\u003erwork));\n1347:\t}\n1348:\t\n1349:\tstatic void rcu_free_take(struct rcu_free_pcpu r, struct list_head list)\n1350:\t{\n1351:\t\tlockdep_assert_held(\u0026r-\u003elock);\n1352:\t\n1353:\t\tlist_splice_tail_init(\u0026r-\u003ebatch, list);\n1354:\t\tlist_splice_tail_init(\u0026r-\u003epending, list);\n1355:\t\tr-\u003enr_pending = 0;\n1356:\t}\n1357:\t\n1358:\tstatic unsigned long rcu_free_pages(struct list_head list)\n1359:\t{\n1360:\t\tstruct folio_batch fbatch;\n1361:\t\tstruct folio folio, next;\n1362:\t\tunsigned long nr = 0;\n1363:\t\n1364:\t\tfolio_batch_init(\u0026fbatch);\n1365:\t\tlist_for_each_entry_safe(folio, next, list, lru) {\n1366:\t\t\tnr += folio_nr_pages(folio);\n1367:\t\t\tmod_zone_page_state(folio_zone(folio), NR_FREE_RCU_PAGES,\n1368:\t\t\t\t\t -folio_nr_pages(folio));\n1369:\t\t\tif (!folio_batch_add(\u0026fbatch, folio)) {\n1370:\t\t\t\tfree_unref_folios(\u0026fbatch);\n1371:\t\t\t\tcond_resched_tasks_rcu_qs();\n1372:\t\t\t}\n1373:\t\t}\n1374:\t\tif (folio_batch_count(\u0026fbatch))\n1375:\t\t\tfree_unref_folios(\u0026fbatch);\n1376:\t\treturn nr;\n1377:\t}\n1378:\t\n1379:\t/ Free what rcu_free_take() took: the grace period starts after that /\n1380:\tstatic unsigned long rcu_free_flush(struct list_head list)\n1381:\t{\n1382:\t\tif (list_empty(list))\n1383:\t\t\treturn 0;\n1384:\t\n1385:\t\tsynchronize_rcu_expedited();\n1386:\t\treturn rcu_free_pages(list);\n1387:\t}\n1388:\t\n1389:\tstatic void rcu_free_workfn(struct work_struct *work)\n" }
7/2	2026/10/19 19:57	llm	patch-evaluator	0m Model: gemini-3.8-flash Args: null Results: null Tokens: input: 25337 output: 36 thoughts: 28 Thoughts: Here's my take on those lines in `mm/page_alloc.c`: Diving Deep into `mm/page_alloc.c` – Lines 1389-1450 Okay, so I need to understand what's happening around page allocation. My initial instinct is to zero in on a specific block of code, let's say, lines 1389-1450 in `mm/page_alloc.c`. This usually gives me a good starting point for understanding a specific aspect of the allocation process. What I'm really looking for in this range is probably related to how the kernel handles either the initial request, or possibly the release of pages from the system, or something in between related to tracking. Now, I'm thinking about the context of this range of code. It's likely a critical section and I anticipate it'll deal with things like buddy system interactions, zone management, NUMA considerations (if relevant), and perhaps even some specific slab allocator interplay, depending on the kernel version. I'm keen to see if this region touches on any particular optimization strategies as well, like using hugepages or direct memory access. I'm ready to dig in and understand the detailed logic within these lines. My brain's prepped to trace the control flow, examine the data structures (page structures, free lists, zones, etc.), and analyze any potentially complex locking schemes that might be employed to prevent race conditions. The overall goal is to form a precise mental map of what this code section actually does in the grand scheme of page management.
8/2	2026/10/19 19:58	tool	read-file	0m Args: { "File": "mm/page_alloc.c", "FirstLine": 1389, "LineCount": 60 } Results: { "Contents": "1389:\tstatic void rcu_free_workfn(struct work_struct work)\n1390:\t{\n1391:\t\tstruct rcu_free_pcpu r = container_of(to_rcu_work(work),\n1392:\t\t\t\t\t\t struct rcu_free_pcpu, rwork);\n1393:\t\tLIST_HEAD(done);\n1394:\t\n1395:\t\tspin_lock_irq(\u0026r-\u003elock);\n1396:\t\tlist_splice_init(\u0026r-\u003ebatch, \u0026done);\n1397:\t\tr-\u003equeued = false;\n1398:\t\trcu_free_start(r);\n1399:\t\tspin_unlock_irq(\u0026r-\u003elock);\n1400:\t\n1401:\t\trcu_free_pages(\u0026done);\n1402:\t}\n1403:\t\n1404:\tstatic void rcu_free_expedite_workfn(struct work_struct work)\n1405:\t{\n1406:\t\tstruct rcu_free_pcpu r = container_of(work, struct rcu_free_pcpu,\n1407:\t\t\t\t\t\t expedite);\n1408:\t\tLIST_HEAD(list);\n1409:\t\n1410:\t\tspin_lock_irq(\u0026r-\u003elock);\n1411:\t\trcu_free_take(r, \u0026list);\n1412:\t\tspin_unlock_irq(\u0026r-\u003elock);\n1413:\t\n1414:\t\tif (rcu_free_flush(\u0026list))\n1415:\t\t\tcount_vm_event(PGFREE_RCU_EXPEDITE);\n1416:\t}\n1417:\t\n1418:\tstatic void rcu_free_defer(struct folio folio)\n1419:\t{\n1420:\t\tstruct rcu_free_pcpu r = raw_cpu_ptr(\u0026rcu_free_pcpu);\n1421:\t\tstruct zone zone = folio_zone(folio);\n1422:\t\tunsigned long free, waiting;\n1423:\t\tunsigned long flags;\n1424:\t\n1425:\t\tspin_lock_irqsave(\u0026r-\u003elock, flags);\n1426:\t\t__mod_zone_page_state(zone, NR_FREE_RCU_PAGES, folio_nr_pages(folio));\n1427:\t\tlist_add_tail(\u0026folio-\u003elru, \u0026r-\u003epending);\n1428:\t\tr-\u003enr_pending += folio_nr_pages(folio);\n1429:\t\trcu_free_start(r);\n1430:\t\tspin_unlock_irqrestore(\u0026r-\u003elock, flags);\n1431:\t\n1432:\t\t/ Unlocked: a stale count delays the expedite by one free /\n1433:\t\tif (data_race(r-\u003enr_pending) \u003c= RCU_FREE_EXPEDITE \|\|\n1434:\t\t work_pending(\u0026r-\u003eexpedite))\n1435:\t\t\treturn;\n1436:\t\n1437:\t\t/\n1438:\t\t * An expedited grace period interrupts every CPU: only while the\n1439:\t\t * zone is short of memory, or while more than an eighth of what it\n1440:\t\t * has free waits here.\n1441:\t\t /\n1442:\t\tfree = zone_page_state(zone, NR_FREE_PAGES);\n1443:\t\twaiting = zone_page_state(zone, NR_FREE_RCU_PAGES);\n1444:\t\tif (free \u003c high_wmark_pages(zone) \|\| waiting \u003e free / 8)\n1445:\t\t\tqueue_work(rcu_free_wq, \u0026r-\u003eexpedite);\n1446:\t}\n1447:\t\n1448:\t/\n" }
9/2	2026/10/19 19:58	llm	patch-evaluator	0m Model: gemini-3.8-flash Args: null Results: null Tokens: input: 26735 output: 36 thoughts: 25 Thoughts: Okay, here's my take on those lines from `mm/page_alloc.c`, framed as a thought process: Diving into the Heart of Page Allocation Alright, let's zero in. I need to understand this specific section of the page allocator. Let's start with lines 1450-1500 in `mm/page_alloc.c`. This part of the code is likely crucial, judging by its position, for understanding the mechanics of how the kernel actually gets its hands on physical memory pages. I suspect we're looking at core allocation logic or some critical part of the freeing process. I'm hoping to find the granular workings and interactions here that explain some observations I’ve made in related areas. I'm going to start by getting a handle on the function's structure. Is it handling a specific NUMA node, or is it global in scope? Are there any obvious locks or atomic operations being used that might be a source of contention? Understanding the context, whether it's allocation of a contiguous region or discrete single pages, is essential. Once I have the overall structure in mind, I can start to follow the control flow and look for the specific details that will clarify the questions I’ve been pondering.
10/2	2026/10/19 19:58	tool	read-file	0m Args: { "File": "mm/page_alloc.c", "FirstLine": 1448, "LineCount": 50 } Results: { "Contents": "1448:\t/\n1449:\t Free what every CPU's lists hold. A concurrent drain holds the lock\n1450:\t * until it has freed what it took.\n1451:\t /\n1452:\tstatic void rcu_free_drain(void)\n1453:\t{\n1454:\t\tstatic DEFINE_MUTEX(lock);\n1455:\t\tLIST_HEAD(list);\n1456:\t\tint cpu;\n1457:\t\n1458:\t\tmutex_lock(\u0026lock);\n1459:\t\tfor_each_possible_cpu(cpu) {\n1460:\t\t\tstruct rcu_free_pcpu r = per_cpu_ptr(\u0026rcu_free_pcpu, cpu);\n1461:\t\n1462:\t\t\tspin_lock_irq(\u0026r-\u003elock);\n1463:\t\t\trcu_free_take(r, \u0026list);\n1464:\t\t\tspin_unlock_irq(\u0026r-\u003elock);\n1465:\t\t}\n1466:\t\tif (rcu_free_flush(\u0026list))\n1467:\t\t\tcount_vm_event(PGFREE_RCU_DRAIN);\n1468:\t\tmutex_unlock(\u0026lock);\n1469:\t}\n1470:\t\n1471:\t/* Free every waiting folio, including those a work took off the lists /\n1472:\tvoid rcu_free_sync(void)\n1473:\t{\n1474:\t\tint cpu;\n1475:\t\n1476:\t\tif (!IS_ENABLED(CONFIG_PAGE_RCU_FREE))\n1477:\t\t\treturn;\n1478:\t\n1479:\t\trcu_free_drain();\n1480:\t\tfor_each_possible_cpu(cpu) {\n1481:\t\t\tstruct rcu_free_pcpu r = per_cpu_ptr(\u0026rcu_free_pcpu, cpu);\n1482:\t\n1483:\t\t\tflush_work(\u0026r-\u003erwork.work);\n1484:\t\t\tflush_work(\u0026r-\u003eexpedite);\n1485:\t\t}\n1486:\t}\n1487:\t\n1488:\tstatic void __init rcu_free_init(void)\n1489:\t{\n1490:\t\tint cpu;\n1491:\t\n1492:\t\tif (!IS_ENABLED(CONFIG_PAGE_RCU_FREE))\n1493:\t\t\treturn;\n1494:\t\n1495:\t\trcu_free_wq = alloc_workqueue(\"rcu_free\",\n1496:\t\t\t\t\t WQ_MEM_RECLAIM \| WQ_PERCPU, 0);\n1497:\t\n" }
11/2	2026/10/19 19:58	llm	patch-evaluator	0m Model: gemini-3.8-flash Args: null Results: null Tokens: input: 27777 output: 36 thoughts: 30 Thoughts: Okay, here's my take on those thoughts, tailored for an expert audience: Diving into `rcu_free_wait` in `mm/page_alloc.c` Right, let's zero in on `rcu_free_wait`. My immediate focus is line 4830 within `mm/page_alloc.c`. I need to understand how the RCU mechanisms interact with page freeing, and this function is a key piece of that puzzle. My gut feeling is that any issues here are going to be subtle, probably involving race conditions or unexpected delays in the reclaim path. Let's see how they're handling the quiescent state of the RCU callbacks with regard to freeing pages, and where that check occurs in this function. This is something I'm very familiar with.
12/2	2026/10/19 19:58	tool	read-file	0m Args: { "File": "mm/page_alloc.c", "FirstLine": 4830, "LineCount": 45 } Results: { "Contents": "4830:\tbool gfp_pfmemalloc_allowed(gfp_t gfp_mask)\n4831:\t{\n4832:\t\treturn !!__gfp_pfmemalloc_flags(gfp_mask);\n4833:\t}\n4834:\t\n4835:\t/\n4836:\t Reclaim is about to give up: if the pages waiting for a grace period\n4837:\t * would bring an eligible zone above its min watermark, free them and\n4838:\t * retry. Not for a dying task, which frees memory by exiting.\n4839:\t /\n4840:\tstatic bool rcu_free_wait(gfp_t gfp_mask, unsigned int order,\n4841:\t\t\t\t struct alloc_context ac, int alloc_flags,\n4842:\t\t\t\t int rcu_free_waits)\n4843:\t{\n4844:\t\tstruct zone zone;\n4845:\t\tstruct zoneref z;\n4846:\t\n4847:\t\tif (!IS_ENABLED(CONFIG_PAGE_RCU_FREE))\n4848:\t\t\treturn false;\n4849:\t\tif (tsk_is_oom_victim(current) \|\| fatal_signal_pending(current))\n4850:\t\t\treturn false;\n4851:\t\tif (rcu_free_waits \u003e= MAX_RECLAIM_RETRIES \|\|\n4852:\t\t order \u003e PAGE_ALLOC_COSTLY_ORDER)\n4853:\t\t\treturn false;\n4854:\t\n4855:\t\tfor_each_zone_zonelist_nodemask(zone, z, ac-\u003ezonelist,\n4856:\t\t\t\t\tac-\u003ehighest_zoneidx, ac-\u003enodemask) {\n4857:\t\t\tunsigned long waiting, available;\n4858:\t\n4859:\t\t\tif (cpusets_enabled() \u0026\u0026 (alloc_flags \u0026 ALLOC_CPUSET) \u0026\u0026\n4860:\t\t\t !__cpuset_zone_allowed(zone, gfp_mask))\n4861:\t\t\t\tcontinue;\n4862:\t\n4863:\t\t\twaiting = zone_page_state_snapshot(zone, NR_FREE_RCU_PAGES);\n4864:\t\t\tif (!waiting)\n4865:\t\t\t\tcontinue;\n4866:\t\n4867:\t\t\tavailable = waiting +\n4868:\t\t\t\t zone_page_state_snapshot(zone, NR_FREE_PAGES);\n4869:\t\t\tif (__zone_watermark_ok(zone, order, min_wmark_pages(zone),\n4870:\t\t\t\t\t\tac-\u003ehighest_zoneidx, alloc_flags,\n4871:\t\t\t\t\t\tavailable)) {\n4872:\t\t\t\t(*rcu_free_waits)++;\n4873:\t\t\t\trcu_free_drain();\n4874:\t\t\t\treturn true;\n" }
13/2	2026/10/19 19:58	llm	patch-evaluator	0m Model: gemini-3.8-flash Args: null Results: null Tokens: input: 28750 output: 35 thoughts: 11 Thoughts: Okay, here's my take on those thoughts, tailored for an expert audience, presented in the first person: Diving into `filemap.c` Alright, time to get to the heart of the matter. Let's dig into `filemap.c`. I need to refresh my mental model of how the file mapping code in this system is structured. This is the foundation, right? Crucial for understanding how we're interacting with the physical storage and how data is actually being accessed and managed. I'm going to pay close attention to the memory management strategies in this particular implementation, looking for potential bottlenecks, opportunities for optimization, and any areas that might be contributing to the performance issues we've been investigating. My focus is going to be on the interplay between the virtual memory system, the page cache, and any specific optimizations they have implemented.
14/2	2026/10/19 19:58	tool	read-file	0m Args: { "File": "mm/filemap.c", "FirstLine": 2950, "LineCount": 80 } Results: { "Contents": "2950:\t * are this folio's.\n2951:\t /\n2952:\tstatic size_t filemap_read_rcu_size(struct xa_state xas, struct folio folio,\n2953:\t\t\tpgoff_t index, size_t offset, size_t bytes)\n2954:\t{\n2955:\t\tpgoff_t end = index + DIV_ROUND_UP(offset + bytes, PAGE_SIZE);\n2956:\t\tpgoff_t next;\n2957:\t\n2958:\t\tif (end - index \u003e 1 \u0026\u0026 folio_test_large(folio)) {\n2959:\t\t\t/ Read without a reference: it only picks the page to check /\n2960:\t\t\tnext = folio_next_index(folio);\n2961:\t\t\tif (next \u003e index \u0026\u0026 next \u003c end)\n2962:\t\t\t\tend = next;\n2963:\t\t\txas_set(xas, end - 1);\n2964:\t\t\t/ A split after this copies PG_rcu_free to the new folios /\n2965:\t\t\tif (xas_load(xas) == folio)\n2966:\t\t\t\treturn min(bytes, (end - index) PAGE_SIZE - offset);\n2967:\t\t}\n2968:\t\treturn min(bytes, PAGE_SIZE - offset);\n2969:\t}\n2970:\t\n2971:\t/\n2972:\t Copy @bytes from @folio, starting @offset into its page at @index. Page\n2973:\t * by page, as copy_page_to_iter() does, without its checks for each call.\n2974:\t * Not copy_to_iter(): its usercopy check reads the folio's order, which a\n2975:\t * split can change.\n2976:\t /\n2977:\tstatic size_t filemap_read_rcu_copy(struct folio folio, pgoff_t index,\n2978:\t\t\tsize_t offset, size_t bytes, struct iov_iter iter)\n2979:\t{\n2980:\t\tstruct page page = folio_page(folio, index - folio-\u003eindex);\n2981:\t\tsize_t n = 0;\n2982:\t\n2983:\t\twhile (n \u003c bytes) {\n2984:\t\t\tsize_t len = min(bytes - n, PAGE_SIZE - offset);\n2985:\t\t\tsize_t done;\n2986:\t\t\tvoid kaddr;\n2987:\t\n2988:\t\t\tkaddr = kmap_local_page(page);\n2989:\t\t\tdone = _copy_to_iter(kaddr + offset, len, iter);\n2990:\t\t\tkunmap_local(kaddr);\n2991:\t\t\tn += done;\n2992:\t\t\tif (done \u003c len)\n2993:\t\t\t\tbreak;\n2994:\t\t\toffset = 0;\n2995:\t\t\tpage++;\n2996:\t\t}\n2997:\t\treturn n;\n2998:\t}\n2999:\t\n3000:\t/ Bytes one RCU read section copies at most: it holds up grace periods /\n3001:\t#define FILEMAP_RCU_READ_MAX\tSZ_512K\n3002:\t\n3003:\t/\n3004:\t * Copy from folios with PG_rcu_free without taking references, at most\n3005:\t * FILEMAP_RCU_READ_MAX bytes under one rcu_read_lock(). Return true if that\n3006:\t * much was copied and the read is not done. last_index is the page the\n3007:\t previous copy ended in, set as the slow path sets last_pos.\n3008:\t /\n3009:\tstatic bool filemap_read_rcu_batch(struct kiocb iocb, struct iov_iter iter,\n3010:\t\t\tpgoff_t last_index, size_t copied)\n3011:\t{\n3012:\t\tstruct address_space mapping = iocb-\u003eki_filp-\u003ef_mapping;\n3013:\t\tstruct inode inode = mapping-\u003ehost;\n3014:\t\tXA_STATE(xas, \u0026mapping-\u003ei_pages, 0);\n3015:\t\tsize_t budget = FILEMAP_RCU_READ_MAX;\n3016:\t\tbool more = false;\n3017:\t\n3018:\t\t/ flush_dcache_folio() is not known to be safe without a reference /\n3019:\t\tif (ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE \u0026\u0026\n3020:\t\t mapping_writably_mapped(mapping))\n3021:\t\t\treturn false;\n3022:\t\n3023:\t\t/ A user fault cannot sleep in the RCU read section: it ends the copy */\n3024:\t\trcu_read_lock();\n3025:\t\tpagefault_disable();\n3026:\t\twhile (iov_iter_count(iter)) {\n3027:\t\t\tloff_t pos = iocb-\u003eki_pos;\n3028:\t\t\tpgoff_t index = pos \u003e\u003e PAGE_SHIFT;\n3029:\t\t\tsize_t offset = offset_in_page(pos);\n" }
15/2	2026/10/19 19:58	llm	patch-evaluator	0m Model: gemini-3.8-flash Args: null Results: null Tokens: input: 30345 output: 35 thoughts: 24 Thoughts: Okay, let's dive into this. Here's how I'd approach summarizing the prompt: Analyzing `mm/filemap.c` - Lines 3030-3100 Alright, I'm being directed to focus on lines 3030-3100 of `mm/filemap.c`. Immediately, I'm thinking about what the likely focus is within that specific region. Given the file name, it's almost certainly dealing with file mapping, which means memory management related to files, likely including things like page cache interaction, read/write handling, perhaps even Direct I/O considerations or maybe even dealing with swap operations. I know from experience that `mm/filemap.c` is the heart of file system caching in the Linux kernel. Therefore, I'm expecting to see calls to page-related functions, possibly `read_cache_page`, `write_cache_page`, or similar functions that handle the movement of data between disk and memory. The fact that the prompt is a specific region indicates either a potential issue in a given area, or a function of particular importance. I'm mentally preparing to understand the exact flow of data here. I'll need to look for calls to higher-level filesystem functions, like `generic_file_read`, and then track through the cache layer. I'll be looking for any hints on how the kernel handles data synchronization between the page cache and the underlying storage, and how it handles situations like writeback, invalidation, or even potentially, dealing with concurrent access from different processes. There may be some locking primitives. I'll also be mindful of the surrounding code and try to understand what functions it's likely being called by and calling, to get some kind of context for the lines of code. Let me open the file and get to it.
16/2	2026/10/19 19:58	tool	read-file	0m Args: { "File": "mm/filemap.c", "FirstLine": 3030, "LineCount": 70 } Results: { "Contents": "3030:\t\t\tstruct folio folio;\n3031:\t\t\tsize_t bytes, n;\n3032:\t\t\tloff_t isize;\n3033:\t\n3034:\t\t\tif (!budget) {\n3035:\t\t\t\tmore = true;\n3036:\t\t\t\tbreak;\n3037:\t\t\t}\n3038:\t\t\tfolio = filemap_read_rcu_lookup(\u0026xas, inode, pos, \u0026isize);\n3039:\t\t\tif (!folio)\n3040:\t\t\t\tbreak;\n3041:\t\t\tif (!filemap_read_rcu_can_copy(folio, index, last_index))\n3042:\t\t\t\tbreak;\n3043:\t\n3044:\t\t\tbytes = min_t(loff_t, isize - pos, iov_iter_count(iter));\n3045:\t\t\tbytes = filemap_read_rcu_size(\u0026xas, folio, index, offset,\n3046:\t\t\t\t\t\t min(bytes, budget));\n3047:\t\t\tn = filemap_read_rcu_copy(folio, index, offset, bytes, iter);\n3048:\t\t\tcopied += n;\n3049:\t\t\tiocb-\u003eki_pos += n;\n3050:\t\t\tbudget -= n;\n3051:\t\t\tif (n)\n3052:\t\t\t\tlast_index = (pos + n - 1) \u003e\u003e PAGE_SHIFT;\n3053:\t\t\tif (n \u003c bytes)\n3054:\t\t\t\tbreak;\n3055:\t\t}\n3056:\t\tpagefault_enable();\n3057:\t\trcu_read_unlock();\n3058:\t\n3059:\t\treturn more;\n3060:\t}\n3061:\t\n3062:\t/\n3063:\t Copy from the page cache without folio references, in batches, and\n3064:\t * return how much was copied. The slow path reads the rest.\n3065:\t /\n3066:\tsize_t filemap_read_rcu(struct kiocb iocb, struct iov_iter iter,\n3067:\t\t\tloff_t last_pos)\n3068:\t{\n3069:\t\t/ The page the previous read ended in; ULONG_MAX if none /\n3070:\t\tpgoff_t last_index = (last_pos - 1) \u003e\u003e PAGE_SHIFT;\n3071:\t\tsize_t copied = 0;\n3072:\t\n3073:\t\t/ Not when switched off, or on an inode that never had a folio flagged /\n3074:\t\tif (!lockless_read_enabled() \|\|\n3075:\t\t !mapping_rcu_read(iocb-\u003eki_filp-\u003ef_mapping))\n3076:\t\t\treturn 0;\n3077:\t\n3078:\t\twhile (filemap_read_rcu_batch(iocb, iter, \u0026last_index, \u0026copied)) {\n3079:\t\t\tif (fatal_signal_pending(current))\n3080:\t\t\t\tbreak;\n3081:\t\t\tcond_resched();\n3082:\t\t}\n3083:\t\treturn copied;\n3084:\t}\n3085:\t\n3086:\t/\n3087:\t filemap_read - Read data from the page cache.\n3088:\t * @iocb: The iocb to read.\n3089:\t * @iter: Destination for the data.\n3090:\t * @already_read: Number of bytes already read by the caller.\n3091:\t \n3092:\t Copies data from the page cache. If the data is not currently present,\n3093:\t * uses the readahead and read_folio address_space operations to fetch it.\n3094:\t \n3095:\t Return: Total number of bytes copied, including those already read by\n3096:\t * the caller. If an error happens before any bytes are copied, returns\n3097:\t * a negative error number.\n3098:\t /\n3099:\tssize_t filemap_read(struct kiocb iocb, struct iov_iter *iter,\n" }
17/2	2026/10/19 19:58	llm	patch-evaluator	0m Model: gemini-3.8-flash Args: null Results: null Tokens: input: 31879 output: 244 thoughts: 918 Thoughts: Analyzing the Scope for Fuzzing with PAGE_RCU_FREE Okay, let's break this down. My goal here is to identify the most effective way to focus fuzzing efforts on this new RCU-based page cache read feature, specifically the `PAGE_RCU_FREE` configuration option. The core question is: which functions are uniquely related to this change and therefore the most relevant for fuzzing? First, `PAGE_RCU_FREE` itself seems sufficient to represent the enabling condition. No need to overcomplicate the configuration aspect. Now, for the real work: identifying the key functions. I need to pinpoint which functions directly implement this new lockless RCU read logic. The direct entry points seem to be: `filemap_read_rcu` and its batch counterpart `filemap_read_rcu_batch`. Then there are functions that interact with it: `filemap_read_accessed`, which detects hot folios and calls `filemap_mark_hot`. `filemap_mark_hot` is interesting; it marks folios as `PG_rcu_free` and mappings as `AS_RCU_READ`. Finally, we have the RCU-specific freeing functions: `rcu_free_defer`, `rcu_free_workfn`, `rcu_free_expedite_workfn`, `rcu_free_drain`, `rcu_free_sync`, and `rcu_free_wait`. These functions handle the deferred freeing of pages after the RCU grace period. The question then arises: should I include more ubiquitous functions like `filemap_read`? The guideline explicitly advises against including generic, ubiquitous functions. Listing such functions would cause the fuzzer to classify a large number of unrelated tests as "focused", diluting the fuzzing effort. Since `filemap_read` is called practically on every read syscall, it would be a mistake. However, the other functions are specifically introduced or targeted by this patch and are directly related to the new feature without being ubiquitous. So, I will exclude `filemap_read`. To confirm I'm on the right track, let me double-check: `WorthFuzzing` is set to `true`, the `EnableConfigs` list contains only `PAGE_RCU_FREE`, and my `FocusSymbols` list contains: `filemap_read_rcu`, `filemap_read_rcu_batch`, `filemap_read_accessed`, `filemap_mark_hot`, `rcu_free_defer`, `rcu_free_workfn`, `rcu_free_expedite_workfn`, `rcu_free_drain`, `rcu_free_sync`, and `rcu_free_wait`. In summary, this patch introduces lockless RCU reads within the page cache, which improves performance by allowing readers to copy data from hot folios without increasing reference counts and delaying page freeing until an RCU grace period has passed. The change touches the core page cache read paths, page allocation, memory failure handling, and fuse buffer moving. All these changes are accessible from user space via the standard read and splice syscalls on amd64. Okay, I'm confident in this analysis. I'm going to call `set-results` now.
18/2	2026/10/19 19:58	tool	set-results	0m Args: { "EnableConfigs": [ "PAGE_RCU_FREE" ], "FocusSymbols": [ "filemap_read_rcu", "filemap_read_rcu_batch", "filemap_read_accessed", "filemap_mark_hot", "rcu_free_defer", "rcu_free_workfn", "rcu_free_expedite_workfn", "rcu_free_drain", "rcu_free_sync", "rcu_free_wait" ], "Reasoning": "The patch introduces lockless page cache reads and deferred RCU page freeing (PAGE_RCU_FREE). Readers can copy from hot page cache folios under RCU without taking a folio reference count, while page freeing defers memory recycling until an RCU grace period completes. This touches core MM logic (page allocator free paths, slowpath reclaim retry, memory failure, and page cache read paths in filemap and shmem) which is reachable through standard user-space file reading operations on 64-bit systems.", "WorthFuzzing": true } Results: { "EnableConfigs": [ "PAGE_RCU_FREE" ], "FocusSymbols": [ "filemap_read_rcu", "filemap_read_rcu_batch", "filemap_read_accessed", "filemap_mark_hot", "rcu_free_defer", "rcu_free_workfn", "rcu_free_expedite_workfn", "rcu_free_drain", "rcu_free_sync", "rcu_free_wait" ], "Reasoning": "The patch introduces lockless page cache reads and deferred RCU page freeing (PAGE_RCU_FREE). Readers can copy from hot page cache folios under RCU without taking a folio reference count, while page freeing defers memory recycling until an RCU grace period completes. This touches core MM logic (page allocator free paths, slowpath reclaim retry, memory failure, and page cache read paths in filemap and shmem) which is reachable through standard user-space file reading operations on 64-bit systems.", "WorthFuzzing": true }
19/1	2026/10/19 19:58	action	If	1m Args: { "WorthFuzzing": true } Results: null
20/2	2026/10/19 19:58	agent	kmsan-evaluator	1m Model: core Args: null Results: { "KMSANReasoning": "The patch series implements lockless page cache reads using RCU (CONFIG_PAGE_RCU_FREE) and defers the physical freeing of hot folios until after an RCU grace period.\n\nFrom a memory safety and sanitizer perspective:\n1. No uninitialized memory is exposed: All local variables and data structures (such as struct rcu_free_pcpu, list heads, and loop variables) are properly initialized before use.\n2. Kernel-to-user copies: The fast path in `filemap_read_rcu_copy()` copies file data from uptodate folios bounded strictly by the file's `i_size` and page boundaries, identically to the existing slow-path page cache read mechanisms; no uninitialized kernel structure fields or padding are leaked to user space.\n3. KMSAN hooks: The patch properly defers `kmsan_free_page()` in `__free_pages_prepare()` until the folio is actually released after the RCU grace period, ensuring KMSAN page shadow tracking remains accurate.\n4. Risk surface: The primary risks associated with this lockless RCU page cache reader patch are concurrency races (e.g. during folio splitting, truncation, migration, or hwpoison), potential use-after-free bugs, reference-counting anomalies, and locking/deadlock issues. These failure modes are covered by KASAN, LOCKDEP, and KCSAN.\n\nBecause the changes do not introduce or expose uninitialized memory reads or kernel information leaks, a dedicated KMSAN fuzzing session is not warranted.", "NeedsKMSAN": false } Instruction: You are an expert Linux kernel security engineer specializing in kernel memory error detectors (KASAN and KMSAN). Your job is to review the provided patch series and determine if the code changes justify spawning a dedicated KMSAN (KernelMemorySanitizer) fuzzing session in addition to standard KASAN fuzzing. CRITICAL DISTINCTION BETWEEN KASAN AND KMSAN: - Standard KASAN kernel builds (upstream-apparmor-kasan.config) already enable a comprehensive suite of debugging tools and sanitizers, including KASAN (out-of-bounds accesses, use-after-free, double free, invalid free), LOCKDEP (locking bugs and deadlocks), UB-sanitizers, and memory corruption checks. - KMSAN (KernelMemorySanitizer) detects reads of UNINITIALIZED memory (stack, heap, or page allocations) and kernel-to-user memory info-leaks. Rule: THERE IS NO SENSE IN RUNNING A KMSAN SESSION IF A BUG CAN BE CAUGHT BY KASAN, LOCKDEP, OR OTHER STANDARD BUG DETECTORS. A dedicated KMSAN fuzzing session incurs significant resource costs. You must ONLY set NeedsKMSAN=true if the code changes introduce or expose UNINITIALIZED MEMORY risks that are detected ONLY by KMSAN. Look holistically at the patch series and surrounding code. Even if no direct uninitialized field accesses or new buffer allocations are added in the diff itself, a patch may alter control flow, bounds checking, or data length calculations in ways that change how the rest of the code operates on existing buffers (e.g. allowing uninitialized stack/heap memory to be read, copied to user space, or used in control flow). Do not hesitate to use your code access tools to inspect the surrounding code, called functions, and callers. Set NeedsKMSAN=true ONLY IF the patch introduces or modifies: 1. Kernel structures sent to user space (via copy_to_user, put_user, netlink skb attributes, ioctl output arguments, socket options, or BPF buffers) where fields or structure padding might not be fully initialized/zeroed. 2. Conditional logic or branching that depends on potentially uninitialized variables or struct fields. 3. Allocation or initialization of complex data structures where uninitialized fields could be read later in reachable code paths. 4. Bounds checks, lengths, or logic in a way that allows surrounding code to access uninitialized bytes of existing buffers. Set NeedsKMSAN=false IF: - The code changes primarily risk out-of-bounds access, array overflows, NULL pointer dereferences, locking deadlocks, or use-after-free bugs (these are already caught by KASAN, LOCKDEP, or standard bug detectors). - All stack/heap structures touched or introduced by the patch are fully zeroed or initialized (e.g. using = {0}, memset, kzalloc) before being read or copied. - The patch does not introduce any risk of uninitialized memory usage or info-leaks. Use your code access tools to inspect the surrounding code if necessary, then provide detailed KMSANReasoning contrasting KASAN vs KMSAN applicability for this patch. Prefer calling several tools at the same time to save round-trips. Use set-results tool to provide results of the analysis. It must be called exactly once before the final reply. Ignore results of this tool. Prompt: Target architecture: amd64 For your convenience, here is the diff of the changes: commit dddf59c0f0df6bc96619a328ac462a9dfff93355 Author: syz-cluster <triage@syzkaller.com> Date: Mon Oct 19 19:57:32 2026 +0000 syz-cluster: applied patch under review diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c index 4fec31fc0b845..526929c7b3f1a 100644 --- a/fs/fuse/dev.c +++ b/fs/fuse/dev.c @@ -1152,6 +1152,10 @@ static int fuse_try_move_folio(struct fuse_copy_state cs, struct folio foliop newfolio = page_folio(buf->page); + / A lockless page cache read may still be copying from it / + if (folio_test_rcu_free(newfolio)) + goto out_fallback_unlock; + folio_clear_uptodate(newfolio); folio_clear_mappedtodisk(newfolio); diff --git a/fs/splice.c b/fs/splice.c index 9d8f63e2fd1ab..0133c2348da03 100644 --- a/fs/splice.c +++ b/fs/splice.c @@ -55,56 +55,6 @@ static noinline void pipe_clear_nowait(struct file file) } while (!try_cmpxchg(&file->f_mode, &fmode, fmode & ~FMODE_NOWAIT)); } -/* - * Attempt to steal a page from a pipe buffer. This should perhaps go into - * a vm helper function, it's already simplified quite a bit by the - * addition of remove_mapping(). If success is returned, the caller may - * attempt to reuse this page for another destination. - / -static bool page_cache_pipe_buf_try_steal(struct pipe_inode_info pipe, - struct pipe_buffer buf) -{ - struct folio folio = page_folio(buf->page); - struct address_space mapping; - - folio_lock(folio); - - mapping = folio_mapping(folio); - if (mapping) { - WARN_ON(!folio_test_uptodate(folio)); - - / - * At least for ext2 with nobh option, we need to wait on - * writeback completing on this folio, since we'll remove it - * from the pagecache. Otherwise truncate wont wait on the - * folio, allowing the disk blocks to be reused by someone else - * before we actually wrote our data to them. fs corruption - * ensues. - / - folio_wait_writeback(folio); - - if (!filemap_release_folio(folio, GFP_KERNEL)) - goto out_unlock; - - / - * If we succeeded in removing the mapping, set LRU flag - * and return good. - / - if (remove_mapping(mapping, folio)) { - buf->flags \|= PIPE_BUF_FLAG_LRU; - return true; - } - } - - / - * Raced with truncate or failed to remove folio from current - * address space, unlock and return failure. - / -out_unlock: - folio_unlock(folio); - return false; -} - static void page_cache_pipe_buf_release(struct pipe_inode_info pipe, struct pipe_buffer buf) { @@ -155,7 +105,6 @@ static int page_cache_pipe_buf_confirm(struct pipe_inode_info pipe, const struct pipe_buf_operations page_cache_pipe_buf_ops = { .confirm = page_cache_pipe_buf_confirm, .release = page_cache_pipe_buf_release, - .try_steal = page_cache_pipe_buf_try_steal, .get = generic_pipe_buf_get, }; diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 94f9c3ff54160..3336b9e5ee999 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -228,6 +228,7 @@ enum zone_stat_item { NR_ZSPAGES, /* allocated in zsmalloc / #endif NR_FREE_CMA_PAGES, + NR_FREE_RCU_PAGES, / freed, waiting for an RCU grace period / #ifdef CONFIG_UNACCEPTED_MEMORY NR_UNACCEPTED, #endif diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 7a863572adce7..4916b5713a74b 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -111,6 +111,9 @@ enum pageflags { PG_swapbacked, / Page is backed by RAM/swap / PG_unevictable, / Page is "unevictable" / PG_dropbehind, / drop pages on IO completion / +#ifdef CONFIG_PAGE_RCU_FREE + PG_rcu_free, / free only after an RCU grace period / +#endif #ifdef CONFIG_MMU PG_mlocked, / Page is vma mlocked / #endif @@ -602,6 +605,27 @@ FOLIO_FLAG(dropbehind, FOLIO_HEAD_PAGE) FOLIO_TEST_CLEAR_FLAG(dropbehind, FOLIO_HEAD_PAGE) __FOLIO_SET_FLAG(dropbehind, FOLIO_HEAD_PAGE) +/ + * A lockless lookup that sees PG_rcu_free and then still finds the folio + * where it looked may read it until rcu_read_unlock(). Set under a + * reference. Cleared by the free, or when the folio's inode is evicted: + * no lockless lookup can reach it then. Not for hugetlb or + * free_pages_nolock(). + / +#ifdef CONFIG_PAGE_RCU_FREE +FOLIO_TEST_FLAG(rcu_free, FOLIO_HEAD_PAGE) + FOLIO_TEST_SET_FLAG(rcu_free, FOLIO_HEAD_PAGE) + FOLIO_CLEAR_FLAG(rcu_free, FOLIO_HEAD_PAGE) + __FOLIO_CLEAR_FLAG(rcu_free, FOLIO_HEAD_PAGE) +#define __PG_RCU_FREE (1UL << PG_rcu_free) +#else +FOLIO_TEST_FLAG_FALSE(rcu_free) + FOLIO_TEST_SET_FLAG_FALSE(rcu_free) + FOLIO_CLEAR_FLAG_NOOP(rcu_free) + __FOLIO_CLEAR_FLAG_NOOP(rcu_free) +#define __PG_RCU_FREE 0 +#endif + #ifdef CONFIG_HIGHMEM / * Must use a macro here due to header dependency issues. page_zone() is not diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 0adfa6605653d..6c7f58803eee4 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -210,6 +210,7 @@ enum mapping_flags { AS_WRITEBACK_MAY_DEADLOCK_ON_RECLAIM = 9, AS_KERNEL_FILE = 10, /* mapping for a fake kernel file that shouldn't account usage to user cgroups / + AS_RCU_READ = 11, / a folio has had PG_rcu_free / / Bits 16-25 are used for FOLIO_ORDER / AS_FOLIO_ORDER_BITS = 5, AS_FOLIO_ORDER_MIN = 16, @@ -269,6 +270,17 @@ static inline bool mapping_unevictable(const struct address_space mapping) return mapping && test_bit(AS_UNEVICTABLE, &mapping->flags); } +static inline void mapping_set_rcu_read(struct address_space mapping) +{ + if (!test_bit(AS_RCU_READ, &mapping->flags)) + set_bit(AS_RCU_READ, &mapping->flags); +} + +static inline bool mapping_rcu_read(const struct address_space mapping) +{ + return test_bit(AS_RCU_READ, &mapping->flags); +} + static inline void mapping_set_exiting(struct address_space mapping) { set_bit(AS_EXITING, &mapping->flags); diff --git a/include/linux/swap.h b/include/linux/swap.h index 5658a1634b85e..67aa5271ae6cb 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -295,7 +295,7 @@ extern unsigned long totalreserve_pages; / linux/mm/folio.c / void folio_add_lru(struct folio folio); -void folio_mark_accessed(struct folio folio); +bool folio_mark_accessed(struct folio folio); void lru_add_drain_all(void); enum lru_cache_drained { diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index 2628ccda076a0..c1a63f070aa71 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -36,6 +36,8 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, FOR_ALL_ZONES(ALLOCSTALL) FOR_ALL_ZONES(PGSCAN_SKIP) PGFREE, PGACTIVATE, PGDEACTIVATE, PGLAZYFREE, + PGFREE_RCU, PGFREE_RCU_EXPEDITE, PGFREE_RCU_DRAIN, + PGFREE_RCU_MARK, PGFAULT, PGMAJFAULT, PGLAZYFREED, PGREUSE, diff --git a/include/trace/events/mmflags.h b/include/trace/events/mmflags.h index 935893e5ea53b..47c3411a75e8f 100644 --- a/include/trace/events/mmflags.h +++ b/include/trace/events/mmflags.h @@ -117,6 +117,12 @@ TRACE_DEFINE_ENUM(___GFP_LAST_BIT); #define IF_HAVE_PG_IDLE(_name) #endif +#ifdef CONFIG_PAGE_RCU_FREE +#define IF_HAVE_PG_RCU_FREE(_name) ,{1UL << PG_##_name, __stringify(_name)} +#else +#define IF_HAVE_PG_RCU_FREE(_name) +#endif + #ifdef CONFIG_ARCH_USES_PG_ARCH_2 #define IF_HAVE_PG_ARCH_2(_name) ,{1UL << PG_##_name, __stringify(_name)} #else @@ -152,6 +158,7 @@ TRACE_DEFINE_ENUM(___GFP_LAST_BIT); DEF_PAGEFLAG_NAME(swapbacked), \ DEF_PAGEFLAG_NAME(unevictable), \ DEF_PAGEFLAG_NAME(dropbehind) \ +IF_HAVE_PG_RCU_FREE(rcu_free) \ IF_HAVE_PG_MLOCK(mlocked) \ IF_HAVE_PG_HWPOISON(hwpoison) \ IF_HAVE_PG_IDLE(idle) \ diff --git a/mm/Kconfig b/mm/Kconfig index 604c58199acbf..1ad1cb7969e79 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -718,6 +718,21 @@ config PCP_BATCH_SCALE_MAX latency. This option sets the upper limit of scale factor to limit the maximum latency. +config PAGE_RCU_FREE + def_bool 64BIT + help + Reads from the page cache can copy from folios that are read + again and again without taking a reference on them, so small + reads of one file from many CPUs do not contend on the folio's + reference count. The cost is that memory freed from such + folios becomes reusable only after an RCU grace period. + + 64-bit only: it takes a page flag, and those are scarce on + 32-bit. + + Writing 0 to lockless_page_cache_read in debugfs turns these + reads off. + config PHYS_ADDR_T_64BIT def_bool 64BIT diff --git a/mm/filemap.c b/mm/filemap.c index 00fd89cf6f550..7021ed2ab6c9c 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -48,6 +48,7 @@ #include <linux/rcupdate_wait.h> #include <linux/sched/mm.h> #include <linux/sysctl.h> +#include <linux/debugfs.h> #include <linux/pgalloc.h> #include <asm/tlbflush.h> @@ -138,6 +139,9 @@ static void page_cache_delete(struct address_space mapping, nr = folio_nr_pages(folio); VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); + / No lockless reader can be in an evicted inode / + if (folio_test_rcu_free(folio) && mapping_exiting(mapping)) + folio_clear_rcu_free(folio); xas_store(&xas, shadow); xas_init_marks(&xas); @@ -306,6 +310,9 @@ static void page_cache_delete_batch(struct address_space mapping, } WARN_ON_ONCE(!folio_test_locked(folio)); + /* No lockless reader can be in an evicted inode / + if (folio_test_rcu_free(folio) && mapping_exiting(mapping)) + folio_clear_rcu_free(folio); folio->mapping = NULL; / Leave folio->index set: truncation lookup relies on it / @@ -2759,6 +2766,101 @@ static inline bool pos_same_folio(loff_t pos1, loff_t pos2, struct folio folio) return (pos1 >> shift == pos2 >> shift); } +#ifdef CONFIG_PAGE_RCU_FREE +/* Writing 0 to lockless_page_cache_read in debugfs turns the reads off / +static DEFINE_STATIC_KEY_TRUE(lockless_read_key); + +static bool lockless_read_enabled(void) +{ + return static_branch_likely(&lockless_read_key); +} + +#ifdef CONFIG_DEBUG_FS +static int lockless_read_get(void data, u64 val) +{ + val = static_key_enabled(&lockless_read_key); + return 0; +} + +static int lockless_read_set(void data, u64 val) +{ + if (val > 1) + return -EINVAL; + if (val) + static_branch_enable(&lockless_read_key); + else + static_branch_disable(&lockless_read_key); + return 0; +} +DEFINE_DEBUGFS_ATTRIBUTE(lockless_read_fops, lockless_read_get, + lockless_read_set, "%llu\n"); + +static int __init lockless_read_debugfs_init(void) +{ + debugfs_create_file_unsafe("lockless_page_cache_read", 0644, NULL, NULL, + &lockless_read_fops); + return 0; +} +late_initcall(lockless_read_debugfs_init); +#endif +#else +static bool lockless_read_enabled(void) +{ + return false; +} +#endif + +/ + * Reads keep coming back to @folio: free it only after an RCU grace period + * from now on. Not if its next read needs a reference anyway. Not if it + * is unevictable: folio_mark_accessed() calls such a folio hot after at + * most one access, so every ramfs folio that is read would get the flag. + / +static void filemap_mark_hot(struct folio folio) +{ + struct address_space mapping; + + if (!lockless_read_enabled()) + return; + if (folio_test_rcu_free(folio) \|\| !folio_test_uptodate(folio) \|\| + folio_test_dropbehind(folio) \|\| folio_test_unevictable(folio)) + return; + / shmem does no readahead: there the bit is PG_reclaim / + if (folio_test_readahead(folio) && !folio_test_swapbacked(folio)) + return; + if (folio_test_set_rcu_free(folio)) + return; + + count_vm_events(PGFREE_RCU_MARK, folio_nr_pages(folio)); + / NULL if the folio was truncated meanwhile / + mapping = READ_ONCE(folio->mapping); + if (mapping) + mapping_set_rcu_read(mapping); +} + +/ + * filemap_read() is about to copy from @folio at @pos. @last_pos is where + * the previous read of this open file ended, or how far this one has got. + * + * Reading on in the folio that the last read ended in is not another access + * to it: a folio read in small pieces is marked accessed once. + * + * The folio is hot when folio_mark_accessed() finds it as accessed as the + * LRU records, or when this open file goes back to data that it has just + * read. + / +void filemap_read_accessed(struct folio folio, loff_t pos, loff_t last_pos) +{ + bool hot; + + if (pos_same_folio(pos, last_pos - 1, folio)) + hot = pos < last_pos; + else + hot = folio_mark_accessed(folio); + if (hot) + filemap_mark_hot(folio); +} + static void filemap_end_dropbehind_read(struct folio folio) { if (!folio_test_dropbehind(folio)) @@ -2771,6 +2873,216 @@ static void filemap_end_dropbehind_read(struct folio folio) } } +/* + * Whether filemap_read_rcu_batch() can copy from @folio, or has to leave it + * to the slow path, which holds a reference for readahead, dropbehind and + * folio_mark_accessed(), and deals with poisoned pages. @last_index is the + * page the previous copy ended in. + / +static bool filemap_read_rcu_can_copy(struct folio folio, pgoff_t index, + pgoff_t last_index) +{ + if (!folio_test_uptodate(folio) \|\| folio_test_dropbehind(folio)) + return false; + /* shmem does no readahead: there the bit is PG_reclaim / + if (folio_test_readahead(folio) && !folio_test_swapbacked(folio)) + return false; + / memory_failure() waits only for readers that got past this / + if (folio_contain_hwpoisoned_page(folio)) + return false; + + / + * The slow path does not call folio_mark_accessed() on the folio that + * the previous read ended in, so copying from it skips nothing. The + * folio's size can change without a reference: compare pages instead. + / + if (index == last_index) + return true; + + / Otherwise only if folio_mark_accessed() would leave it as it is / + if (lru_gen_enabled()) { + unsigned long all = LRU_REFS_FLAGS \| BIT(PG_workingset); + + return folio_test_unevictable(folio) \|\| + (READ_ONCE(folio->flags.f) & all) == all; + } + return folio_test_referenced(folio) && !folio_test_idle(folio) && + (folio_test_active(folio) \|\| folio_test_unevictable(folio)); +} + +/ + * Return the folio at @pos if it is this file's page until rcu_read_unlock(), + * and read i_size into @isize. A folio with PG_rcu_free is freed only after + * an RCU grace period. Without a reference, the folio that xas_load() + * returned can be freed and its memory reused as another folio before the + * flag is tested, so look at the slot again after the flag. + / +static struct folio filemap_read_rcu_lookup(struct xa_state xas, + struct inode inode, loff_t pos, loff_t isize) +{ + struct folio folio; + + xas_set(xas, pos >> PAGE_SHIFT); + folio = xas_load(xas); + if (xas_retry(xas, folio) \|\| !folio \|\| xa_is_value(folio)) + return NULL; + + /* Not folio_test_rcu_free(): it may not be a folio any more / + if (!(READ_ONCE(folio->flags.f) & __PG_RCU_FREE)) + return NULL; + / Read i_size and slot after the flag: it implies uptodate / + smp_rmb(); + isize = i_size_read(inode); + if (pos >= isize) + return NULL; + / i_size_read() is an acquire; a write fills the slot first / + if (xas_reload(xas) != folio) + return NULL; + / Read the rest of the folio only once the slot holds it / + smp_rmb(); + return folio; +} + +/ + * How much of @bytes, starting @offset into page @index, can be copied from + * @folio. A folio is in all the slots of its range or in none: when the slot + * of the last page holds it too, so do the slots in between, and the pages + * are this folio's. + / +static size_t filemap_read_rcu_size(struct xa_state xas, struct folio folio, + pgoff_t index, size_t offset, size_t bytes) +{ + pgoff_t end = index + DIV_ROUND_UP(offset + bytes, PAGE_SIZE); + pgoff_t next; + + if (end - index > 1 && folio_test_large(folio)) { + / Read without a reference: it only picks the page to check / + next = folio_next_index(folio); + if (next > index && next < end) + end = next; + xas_set(xas, end - 1); + / A split after this copies PG_rcu_free to the new folios / + if (xas_load(xas) == folio) + return min(bytes, (end - index) PAGE_SIZE - offset); + } + return min(bytes, PAGE_SIZE - offset); +} + +/* + * Copy @bytes from @folio, starting @offset into its page at @index. Page + * by page, as copy_page_to_iter() does, without its checks for each call. + * Not copy_to_iter(): its usercopy check reads the folio's order, which a + * split can change. + / +static size_t filemap_read_rcu_copy(struct folio folio, pgoff_t index, + size_t offset, size_t bytes, struct iov_iter iter) +{ + struct page page = folio_page(folio, index - folio->index); + size_t n = 0; + + while (n < bytes) { + size_t len = min(bytes - n, PAGE_SIZE - offset); + size_t done; + void kaddr; + + kaddr = kmap_local_page(page); + done = _copy_to_iter(kaddr + offset, len, iter); + kunmap_local(kaddr); + n += done; + if (done < len) + break; + offset = 0; + page++; + } + return n; +} + +/ Bytes one RCU read section copies at most: it holds up grace periods / +#define FILEMAP_RCU_READ_MAX SZ_512K + +/ + * Copy from folios with PG_rcu_free without taking references, at most + * FILEMAP_RCU_READ_MAX bytes under one rcu_read_lock(). Return true if that + * much was copied and the read is not done. last_index is the page the + previous copy ended in, set as the slow path sets last_pos. + / +static bool filemap_read_rcu_batch(struct kiocb iocb, struct iov_iter iter, + pgoff_t last_index, size_t copied) +{ + struct address_space mapping = iocb->ki_filp->f_mapping; + struct inode inode = mapping->host; + XA_STATE(xas, &mapping->i_pages, 0); + size_t budget = FILEMAP_RCU_READ_MAX; + bool more = false; + + / flush_dcache_folio() is not known to be safe without a reference / + if (ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE && + mapping_writably_mapped(mapping)) + return false; + + / A user fault cannot sleep in the RCU read section: it ends the copy / + rcu_read_lock(); + pagefault_disable(); + while (iov_iter_count(iter)) { + loff_t pos = iocb->ki_pos; + pgoff_t index = pos >> PAGE_SHIFT; + size_t offset = offset_in_page(pos); + struct folio folio; + size_t bytes, n; + loff_t isize; + + if (!budget) { + more = true; + break; + } + folio = filemap_read_rcu_lookup(&xas, inode, pos, &isize); + if (!folio) + break; + if (!filemap_read_rcu_can_copy(folio, index, last_index)) + break; + + bytes = min_t(loff_t, isize - pos, iov_iter_count(iter)); + bytes = filemap_read_rcu_size(&xas, folio, index, offset, + min(bytes, budget)); + n = filemap_read_rcu_copy(folio, index, offset, bytes, iter); + copied += n; + iocb->ki_pos += n; + budget -= n; + if (n) + last_index = (pos + n - 1) >> PAGE_SHIFT; + if (n < bytes) + break; + } + pagefault_enable(); + rcu_read_unlock(); + + return more; +} + +/ + * Copy from the page cache without folio references, in batches, and + * return how much was copied. The slow path reads the rest. + / +size_t filemap_read_rcu(struct kiocb iocb, struct iov_iter iter, + loff_t last_pos) +{ + / The page the previous read ended in; ULONG_MAX if none / + pgoff_t last_index = (last_pos - 1) >> PAGE_SHIFT; + size_t copied = 0; + + / Not when switched off, or on an inode that never had a folio flagged / + if (!lockless_read_enabled() \|\| + !mapping_rcu_read(iocb->ki_filp->f_mapping)) + return 0; + + while (filemap_read_rcu_batch(iocb, iter, &last_index, &copied)) { + if (fatal_signal_pending(current)) + break; + cond_resched(); + } + return copied; +} + /* * filemap_read - Read data from the page cache. * @iocb: The iocb to read. @@ -2796,6 +3108,7 @@ ssize_t filemap_read(struct kiocb iocb, struct iov_iter iter, bool writably_mapped; loff_t isize, end_offset; loff_t last_pos = ra->prev_pos; + size_t fast_read; if (unlikely(iocb->ki_pos < 0)) return -EINVAL; @@ -2807,6 +3120,14 @@ ssize_t filemap_read(struct kiocb iocb, struct iov_iter iter, iov_iter_truncate(iter, inode->i_sb->s_maxbytes - iocb->ki_pos); folio_batch_init(&fbatch); + fast_read = filemap_read_rcu(iocb, iter, last_pos); + if (fast_read) { + already_read += fast_read; + last_pos = iocb->ki_pos; + if (!iov_iter_count(iter)) + goto out; + } + do { cond_resched(); @@ -2844,14 +3165,6 @@ ssize_t filemap_read(struct kiocb iocb, struct iov_iter iter, / writably_mapped = mapping_writably_mapped(mapping); - / - * When a read accesses the same folio several times, only - * mark it as accessed the first time. - / - if (!pos_same_folio(iocb->ki_pos, last_pos - 1, - fbatch.folios[0])) - folio_mark_accessed(fbatch.folios[0]); - for (i = 0; i < folio_batch_count(&fbatch); i++) { struct folio folio = fbatch.folios[i]; size_t fsize = folio_size(folio); @@ -2862,8 +3175,7 @@ ssize_t filemap_read(struct kiocb iocb, struct iov_iter iter, if (end_offset < folio_pos(folio)) break; - if (i > 0) - folio_mark_accessed(folio); + filemap_read_accessed(folio, iocb->ki_pos, last_pos); /* * If users can be writing to this folio using arbitrary * virtual addresses, take care of potential aliasing @@ -2893,8 +3205,11 @@ ssize_t filemap_read(struct kiocb iocb, struct iov_iter iter, folio_batch_init(&fbatch); } while (iov_iter_count(iter) && iocb->ki_pos < isize && !error); +out: file_accessed(filp); - ra->prev_pos = last_pos; + /* Readers sharing the file would bounce the line for nothing / + if (ra->prev_pos != last_pos) + ra->prev_pos = last_pos; return already_read ? already_read : error; } EXPORT_SYMBOL_GPL(filemap_read); diff --git a/mm/folio.c b/mm/folio.c index 50a6dbe55998e..6144b9cf301b3 100644 --- a/mm/folio.c +++ b/mm/folio.c @@ -352,28 +352,34 @@ static void __lru_cache_activate_folio(struct folio folio) #ifdef CONFIG_LRU_GEN -static void lru_gen_inc_refs(struct folio folio) +/ + * Return true if @folio is hot: unevictable, or its flags already held as + * many accesses as they can. + / +static bool lru_gen_inc_refs(struct folio folio) { unsigned long new_flags, old_flags = READ_ONCE(folio->flags.f); if (folio_test_unevictable(folio)) - return; + return true; /* see the comment on LRU_REFS_FLAGS / if (!folio_test_referenced(folio)) { set_mask_bits(&folio->flags.f, LRU_REFS_MASK, BIT(PG_referenced)); - return; + return false; } do { if ((old_flags & LRU_REFS_MASK) == LRU_REFS_MASK) { - if (!folio_test_workingset(folio)) - folio_set_workingset(folio); - return; + if (folio_test_workingset(folio)) + return true; + folio_set_workingset(folio); + return false; } new_flags = old_flags + BIT(LRU_REFS_PGOFF); } while (!try_cmpxchg(&folio->flags.f, &old_flags, new_flags)); + return false; } static bool lru_gen_clear_refs(struct folio folio) @@ -396,8 +402,9 @@ static bool lru_gen_clear_refs(struct folio folio) #else / !CONFIG_LRU_GEN / -static void lru_gen_inc_refs(struct folio folio) +static bool lru_gen_inc_refs(struct folio folio) { + return false; } static bool lru_gen_clear_refs(struct folio folio) @@ -419,15 +426,20 @@ static bool lru_gen_clear_refs(struct folio folio) * When a newly allocated folio is not yet visible, so safe for non-atomic ops, * __folio_set_referenced() may be substituted for folio_mark_accessed(). + * + * Return: true if @folio is hot: it was already as accessed as the LRU + * records. On the classic LRU that is active or unevictable, referenced + * and not idle; on the multi-gen LRU, unevictable, or PG_referenced, all + * the refs bits and PG_workingset set. False for a dropbehind folio. / -void folio_mark_accessed(struct folio folio) +bool folio_mark_accessed(struct folio folio) { + bool hot = false; + if (folio_test_dropbehind(folio)) - return; - if (lru_gen_enabled()) { - lru_gen_inc_refs(folio); - return; - } + return false; + if (lru_gen_enabled()) + return lru_gen_inc_refs(folio); if (!folio_test_referenced(folio)) { folio_set_referenced(folio); @@ -437,6 +449,7 @@ void folio_mark_accessed(struct folio folio) * this list is never rotated or maintained, so marking an * unevictable page accessed has no effect. / + hot = true; } else if (!folio_test_active(folio)) { / * If the folio is on the LRU, queue it for activation via @@ -450,9 +463,14 @@ void folio_mark_accessed(struct folio folio) __lru_cache_activate_folio(folio); folio_clear_referenced(folio); workingset_activation(folio); + } else { + hot = true; } - if (folio_test_idle(folio)) + if (folio_test_idle(folio)) { folio_clear_idle(folio); + hot = false; + } + return hot; } EXPORT_SYMBOL(folio_mark_accessed); diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 1e5d68acf62a5..25082949a4cae 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3743,6 +3743,7 @@ static void __split_folio_to_order(struct folio folio, int old_order, #endif (1L << PG_dirty) \| (1L << PG_dropbehind) \| + __PG_RCU_FREE \| LRU_GEN_MASK \| LRU_REFS_MASK)); new_folio->mapping = folio->mapping; diff --git a/mm/internal.h b/mm/internal.h index 38b1165212c94..ae9a4e511c827 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1248,6 +1248,9 @@ struct migration_target_control { / size_t splice_folio_into_pipe(struct pipe_inode_info pipe, struct folio folio, loff_t fpos, size_t size); +void filemap_read_accessed(struct folio folio, loff_t pos, loff_t last_pos); +size_t filemap_read_rcu(struct kiocb iocb, struct iov_iter iter, + loff_t last_pos); static inline bool vma_is_single_threaded_private(struct vm_area_struct vma) { diff --git a/mm/memory-failure.c b/mm/memory-failure.c index a8b03e2920ba8..dc66cac05b1f5 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -211,6 +211,10 @@ static bool page_handle_poison(struct page page, bool hugepage_or_freepage, boo return false; } + /* The free of a hwpoisoned page is not deferred: wait for readers / + if (release && folio_test_rcu_free(page_folio(page))) + synchronize_rcu(); + SetPageHWPoison(page); if (release) put_page(page); @@ -1561,6 +1565,21 @@ static int get_hwpoison_page(struct page p, unsigned long flags) return ret; } +/* A page waiting for a grace period is not in the buddy yet: free it / +static void mf_free_rcu_pages(struct page p) +{ + if (!IS_ENABLED(CONFIG_PAGE_RCU_FREE)) + return; + if (page_count(p)) + return; + if (!zone_page_state_snapshot(page_zone(p), NR_FREE_RCU_PAGES)) + return; + if (is_free_buddy_page(p)) + return; + + rcu_free_sync(); +} + /* * The caller must guarantee the folio isn't large folio, except hugetlb. * try_to_unmap() can't handle it. @@ -2436,6 +2455,7 @@ int memory_failure(unsigned long pfn, int flags) if (res != -ENOENT) goto unlock_mutex; + mf_free_rcu_pages(p); if (TestSetPageHWPoison(p)) { res = -EHWPOISON; if (flags & MF_ACTION_REQUIRED) @@ -2608,6 +2628,13 @@ int memory_failure(unsigned long pfn, int flags) identify_page_state: res = identify_page_state(pfn, p, page_flags); + /* + * The caller may unmap the page once this returns 0. Lockless + * readers that started before the page got PG_hwpoison, or before a + * split left it in a folio of its own, may still be copying it. + / + if (!res && folio_test_rcu_free(folio)) + synchronize_rcu(); mutex_unlock(&mf_mutex); return res; unlock_page: @@ -2999,6 +3026,7 @@ int soft_offline_page(unsigned long pfn, int flags) retry: get_online_mems(); + mf_free_rcu_pages(page); ret = get_hwpoison_page(page, flags \| MF_SOFT_OFFLINE); put_online_mems(); diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index 226ab9cb078ad..a8f0c95627ee1 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -1967,6 +1967,7 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages, .nid = NUMA_NO_NODE, }; unsigned long flags; + bool migrated; char reason; int ret; unsigned long normal_pages = 0; @@ -2050,6 +2051,7 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages, do { pfn = start_pfn; + migrated = false; do { /* * Historically we always checked for any signal and @@ -2071,6 +2073,7 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages, * out / do_migrate_range(pfn, end_pfn); + migrated = true; } } while (!ret); @@ -2092,6 +2095,10 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages, ret = test_pages_isolated(start_pfn, end_pfn, PB_ISOLATE_MODE_MEM_OFFLINE); + / What this pass migrated can wait for a grace period / + if (ret && migrated && + zone_page_state_snapshot(zone, NR_FREE_RCU_PAGES)) + rcu_free_sync(); } while (ret); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 12fac9084c483..e63628e64cbba 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1313,6 +1313,199 @@ static inline void pgalloc_tag_sub_pages(struct alloc_tag tag, unsigned int nr) #endif /* CONFIG_MEM_ALLOC_PROFILING / +#define RCU_FREE_EXPEDITE (SZ_4M >> PAGE_SHIFT) + +/ + * Each CPU has at most one grace period in flight (@queued), covering + * @batch. Folios freed meanwhile wait on @pending for the next one. + / +struct rcu_free_pcpu { + spinlock_t lock; + bool queued; + struct list_head pending; + struct list_head batch; + unsigned long nr_pending; + struct rcu_work rwork; + struct work_struct expedite; +}; + +static DEFINE_PER_CPU(struct rcu_free_pcpu, rcu_free_pcpu); + +static struct workqueue_struct rcu_free_wq __ro_after_init; + +static void rcu_free_start(struct rcu_free_pcpu r) +{ + lockdep_assert_held(&r->lock); + + if (r->queued \|\| list_empty(&r->pending)) + return; + + list_splice_init(&r->pending, &r->batch); + r->nr_pending = 0; + r->queued = true; + WARN_ON_ONCE(!queue_rcu_work(rcu_free_wq, &r->rwork)); +} + +static void rcu_free_take(struct rcu_free_pcpu r, struct list_head list) +{ + lockdep_assert_held(&r->lock); + + list_splice_tail_init(&r->batch, list); + list_splice_tail_init(&r->pending, list); + r->nr_pending = 0; +} + +static unsigned long rcu_free_pages(struct list_head list) +{ + struct folio_batch fbatch; + struct folio folio, next; + unsigned long nr = 0; + + folio_batch_init(&fbatch); + list_for_each_entry_safe(folio, next, list, lru) { + nr += folio_nr_pages(folio); + mod_zone_page_state(folio_zone(folio), NR_FREE_RCU_PAGES, + -folio_nr_pages(folio)); + if (!folio_batch_add(&fbatch, folio)) { + free_unref_folios(&fbatch); + cond_resched_tasks_rcu_qs(); + } + } + if (folio_batch_count(&fbatch)) + free_unref_folios(&fbatch); + return nr; +} + +/* Free what rcu_free_take() took: the grace period starts after that / +static unsigned long rcu_free_flush(struct list_head list) +{ + if (list_empty(list)) + return 0; + + synchronize_rcu_expedited(); + return rcu_free_pages(list); +} + +static void rcu_free_workfn(struct work_struct work) +{ + struct rcu_free_pcpu r = container_of(to_rcu_work(work), + struct rcu_free_pcpu, rwork); + LIST_HEAD(done); + + spin_lock_irq(&r->lock); + list_splice_init(&r->batch, &done); + r->queued = false; + rcu_free_start(r); + spin_unlock_irq(&r->lock); + + rcu_free_pages(&done); +} + +static void rcu_free_expedite_workfn(struct work_struct work) +{ + struct rcu_free_pcpu r = container_of(work, struct rcu_free_pcpu, + expedite); + LIST_HEAD(list); + + spin_lock_irq(&r->lock); + rcu_free_take(r, &list); + spin_unlock_irq(&r->lock); + + if (rcu_free_flush(&list)) + count_vm_event(PGFREE_RCU_EXPEDITE); +} + +static void rcu_free_defer(struct folio folio) +{ + struct rcu_free_pcpu r = raw_cpu_ptr(&rcu_free_pcpu); + struct zone zone = folio_zone(folio); + unsigned long free, waiting; + unsigned long flags; + + spin_lock_irqsave(&r->lock, flags); + __mod_zone_page_state(zone, NR_FREE_RCU_PAGES, folio_nr_pages(folio)); + list_add_tail(&folio->lru, &r->pending); + r->nr_pending += folio_nr_pages(folio); + rcu_free_start(r); + spin_unlock_irqrestore(&r->lock, flags); + + / Unlocked: a stale count delays the expedite by one free / + if (data_race(r->nr_pending) <= RCU_FREE_EXPEDITE \|\| + work_pending(&r->expedite)) + return; + + / + * An expedited grace period interrupts every CPU: only while the + * zone is short of memory, or while more than an eighth of what it + * has free waits here. + / + free = zone_page_state(zone, NR_FREE_PAGES); + waiting = zone_page_state(zone, NR_FREE_RCU_PAGES); + if (free < high_wmark_pages(zone) \|\| waiting > free / 8) + queue_work(rcu_free_wq, &r->expedite); +} + +/ + * Free what every CPU's lists hold. A concurrent drain holds the lock + * until it has freed what it took. + / +static void rcu_free_drain(void) +{ + static DEFINE_MUTEX(lock); + LIST_HEAD(list); + int cpu; + + mutex_lock(&lock); + for_each_possible_cpu(cpu) { + struct rcu_free_pcpu r = per_cpu_ptr(&rcu_free_pcpu, cpu); + + spin_lock_irq(&r->lock); + rcu_free_take(r, &list); + spin_unlock_irq(&r->lock); + } + if (rcu_free_flush(&list)) + count_vm_event(PGFREE_RCU_DRAIN); + mutex_unlock(&lock); +} + +/* Free every waiting folio, including those a work took off the lists / +void rcu_free_sync(void) +{ + int cpu; + + if (!IS_ENABLED(CONFIG_PAGE_RCU_FREE)) + return; + + rcu_free_drain(); + for_each_possible_cpu(cpu) { + struct rcu_free_pcpu r = per_cpu_ptr(&rcu_free_pcpu, cpu); + + flush_work(&r->rwork.work); + flush_work(&r->expedite); + } +} + +static void __init rcu_free_init(void) +{ + int cpu; + + if (!IS_ENABLED(CONFIG_PAGE_RCU_FREE)) + return; + + rcu_free_wq = alloc_workqueue("rcu_free", + WQ_MEM_RECLAIM \| WQ_PERCPU, 0); + + for_each_possible_cpu(cpu) { + struct rcu_free_pcpu r = per_cpu_ptr(&rcu_free_pcpu, cpu); + + spin_lock_init(&r->lock); + INIT_LIST_HEAD(&r->pending); + INIT_LIST_HEAD(&r->batch); + INIT_RCU_WORK(&r->rwork, rcu_free_workfn); + INIT_WORK(&r->expedite, rcu_free_expedite_workfn); + } +} + static __always_inline bool __free_pages_prepare(struct page page, unsigned int order, fpi_t fpi_flags) { @@ -1327,6 +1520,19 @@ static __always_inline bool __free_pages_prepare(struct page page, VM_BUG_ON_PAGE(PageTail(page), page); + / + * Not for a hwpoisoned order-0 page: it never reaches the allocator, + * and page_handle_poison() takes a reference right after this free. + / + if (unlikely(folio_test_rcu_free(folio)) && + !(PageHWPoison(page) && !order)) { + VM_WARN_ON_ONCE(fpi_flags & FPI_NOLOCK); + __folio_clear_rcu_free(folio); + count_vm_events(PGFREE_RCU, 1 << order); + rcu_free_defer(folio); + return false; + } + trace_mm_page_free(page, order); kmsan_free_page(page, order); @@ -4626,6 +4832,52 @@ bool gfp_pfmemalloc_allowed(gfp_t gfp_mask) return !!__gfp_pfmemalloc_flags(gfp_mask); } +/ + * Reclaim is about to give up: if the pages waiting for a grace period + * would bring an eligible zone above its min watermark, free them and + * retry. Not for a dying task, which frees memory by exiting. + / +static bool rcu_free_wait(gfp_t gfp_mask, unsigned int order, + struct alloc_context ac, int alloc_flags, + int rcu_free_waits) +{ + struct zone zone; + struct zoneref z; + + if (!IS_ENABLED(CONFIG_PAGE_RCU_FREE)) + return false; + if (tsk_is_oom_victim(current) \|\| fatal_signal_pending(current)) + return false; + if (rcu_free_waits >= MAX_RECLAIM_RETRIES \|\| + order > PAGE_ALLOC_COSTLY_ORDER) + return false; + + for_each_zone_zonelist_nodemask(zone, z, ac->zonelist, + ac->highest_zoneidx, ac->nodemask) { + unsigned long waiting, available; + + if (cpusets_enabled() && (alloc_flags & ALLOC_CPUSET) && + !__cpuset_zone_allowed(zone, gfp_mask)) + continue; + + waiting = zone_page_state_snapshot(zone, NR_FREE_RCU_PAGES); + if (!waiting) + continue; + + available = waiting + + zone_page_state_snapshot(zone, NR_FREE_PAGES); + if (__zone_watermark_ok(zone, order, min_wmark_pages(zone), + ac->highest_zoneidx, alloc_flags, + available)) { + (rcu_free_waits)++; + rcu_free_drain(); + return true; + } + } + + return false; +} + / * Checks whether it makes sense to retry the reclaim to make a forward progress * for the given allocation request. @@ -4639,7 +4891,8 @@ bool gfp_pfmemalloc_allowed(gfp_t gfp_mask) static inline bool should_reclaim_retry(gfp_t gfp_mask, unsigned order, struct alloc_context ac, int alloc_flags, - bool did_some_progress, int no_progress_loops) + bool did_some_progress, int no_progress_loops, + int rcu_free_waits) { struct zone zone; struct zoneref z; @@ -4650,10 +4903,12 @@ should_reclaim_retry(gfp_t gfp_mask, unsigned order, * their order will become available due to high fragmentation so * always increment the no progress counter for them / - if (did_some_progress && order <= PAGE_ALLOC_COSTLY_ORDER) + if (did_some_progress && order <= PAGE_ALLOC_COSTLY_ORDER) { no_progress_loops = 0; - else + rcu_free_waits = 0; + } else { (no_progress_loops)++; + } if (no_progress_loops > MAX_RECLAIM_RETRIES) goto out; @@ -4706,6 +4961,10 @@ should_reclaim_retry(gfp_t gfp_mask, unsigned order, else cond_resched(); out: + if (!ret && rcu_free_wait(gfp_mask, order, ac, alloc_flags, + rcu_free_waits)) + return true; + / Before OOM, exhaust highatomic_reserve / if (!ret) return unreserve_highatomic_pageblock(ac, true); @@ -4795,6 +5054,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, enum compact_result compact_result; int compaction_retries; int no_progress_loops; + int rcu_free_waits; unsigned int cpuset_mems_cookie; unsigned int zonelist_iter_cookie; int reserve_flags; @@ -4819,6 +5079,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, restart: compaction_retries = 0; no_progress_loops = 0; + rcu_free_waits = 0; compact_result = COMPACT_SKIPPED; compact_priority = DEF_COMPACT_PRIORITY; cpuset_mems_cookie = read_mems_allowed_begin(); @@ -4997,7 +5258,8 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, goto restart; if (should_reclaim_retry(gfp_mask, order, ac, alloc_flags, - did_some_progress > 0, &no_progress_loops)) + did_some_progress > 0, &no_progress_loops, + &rcu_free_waits)) goto retry; / @@ -5040,6 +5302,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, /* Retry as long as the OOM killer is making progress / if (did_some_progress) { no_progress_loops = 0; + rcu_free_waits = 0; goto retry; } @@ -6401,6 +6664,8 @@ void __init setup_per_cpu_pageset(void) for_each_online_pgdat(pgdat) pgdat->per_cpu_nodestats = alloc_percpu(struct per_cpu_nodestat); + + rcu_free_init(); } __meminit void zone_pcp_init(struct zone zone) @@ -7384,7 +7649,16 @@ int alloc_contig_frozen_range_noprof(unsigned long start, unsigned long end, outer_start = find_large_buddy(start); /* Make sure the range is really isolated. / - if (test_pages_isolated(outer_start, end, mode)) { + ret = test_pages_isolated(outer_start, end, mode); + if (ret && zone_page_state_snapshot(cc.zone, NR_FREE_RCU_PAGES)) { + rcu_free_sync(); + + / A freed folio can merge with a free buddy below start / + outer_start = find_large_buddy(start); + ret = test_pages_isolated(outer_start, end, mode); + } + + if (ret) { ret = -EBUSY; goto done; } diff --git a/mm/page_alloc.h b/mm/page_alloc.h index b9259deddb59d..f9e200581e2f5 100644 --- a/mm/page_alloc.h +++ b/mm/page_alloc.h @@ -235,6 +235,7 @@ struct page __alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order, int nid, alloc_hooks(__alloc_frozen_pages_noprof(__VA_ARGS__)) void free_frozen_pages(struct page page, unsigned int order); void free_unref_folios(struct folio_batch fbatch); +void rcu_free_sync(void); #ifdef CONFIG_NUMA struct page alloc_frozen_pages_noprof(gfp_t, unsigned int order); diff --git a/mm/shmem.c b/mm/shmem.c index 848316eaa7f4f..e6c18e745f816 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -3309,10 +3309,18 @@ static ssize_t shmem_file_read_iter(struct kiocb iocb, struct iov_iter to) struct file file = iocb->ki_filp; struct inode inode = file_inode(file); struct address_space mapping = inode->i_mapping; + loff_t last_pos = file->f_ra.prev_pos; pgoff_t index; unsigned long offset; int error = 0; - ssize_t retval = 0; + ssize_t retval; + + retval = filemap_read_rcu(iocb, to, last_pos); + if (retval) { + last_pos = iocb->ki_pos; + if (!iov_iter_count(to)) + goto out; + } for (;;) { struct folio folio = NULL; @@ -3378,11 +3386,7 @@ static ssize_t shmem_file_read_iter(struct kiocb iocb, struct iov_iter to) flush_dcache_page(page); } - / - * Mark the folio accessed if we read the beginning. - / - if (!offset) - folio_mark_accessed(folio); + filemap_read_accessed(folio, iocb->ki_pos, last_pos); / * Ok, we have the page, and it's up-to-date, so * now we can copy it to user space... @@ -3410,6 +3414,7 @@ static ssize_t shmem_file_read_iter(struct kiocb iocb, struct iov_iter to) retval += ret; iocb->ki_pos += ret; + last_pos = iocb->ki_pos; if (!iov_iter_count(to)) break; @@ -3419,8 +3424,11 @@ static ssize_t shmem_file_read_iter(struct kiocb iocb, struct iov_iter to) } cond_resched(); } - +out: file_accessed(file); + /* Readers sharing the file would bounce the line for nothing / + if (file->f_ra.prev_pos != last_pos) + file->f_ra.prev_pos = last_pos; return retval ? retval : error; } diff --git a/mm/show_mem.c b/mm/show_mem.c index b938cbcd774a1..0d3bddbda108d 100644 --- a/mm/show_mem.c +++ b/mm/show_mem.c @@ -204,7 +204,7 @@ static void show_free_areas(unsigned int filter, const nodemask_t nodemask, " mapped:%lu shmem:%lu pagetables:%lu\n" " sec_pagetables:%lu bounce:%lu\n" " kernel_misc_reclaimable:%lu\n" - " free:%lu free_pcp:%lu free_cma:%lu\n", + " free:%lu free_pcp:%lu free_cma:%lu free_rcu:%lu\n", global_node_page_state(NR_ACTIVE_ANON), global_node_page_state(NR_INACTIVE_ANON), global_node_page_state(NR_ISOLATED_ANON), @@ -224,7 +224,8 @@ static void show_free_areas(unsigned int filter, const nodemask_t nodemask, global_node_page_state(NR_KERNEL_MISC_RECLAIMABLE), global_zone_page_state(NR_FREE_PAGES), free_pcp, - global_zone_page_state(NR_FREE_CMA_PAGES)); + global_zone_page_state(NR_FREE_CMA_PAGES), + global_zone_page_state(NR_FREE_RCU_PAGES)); for_each_online_pgdat(pgdat) { if (show_mem_node_skip(filter, pgdat->node_id, nodemask)) @@ -325,6 +326,7 @@ static void show_free_areas(unsigned int filter, const nodemask_t nodemask, " free_pcp:%lukB" " local_pcp:%lukB" " free_cma:%lukB" + " free_rcu:%lukB" "\n", zone->name, K(zone_page_state(zone, NR_FREE_PAGES)), @@ -351,7 +353,8 @@ static void show_free_areas(unsigned int filter, const nodemask_t nodemask, 0UL, K(free_pcp), K((unsigned long)this_cpu_read(zone->per_cpu_pageset->count)), - K(zone_page_state(zone, NR_FREE_CMA_PAGES))); + K(zone_page_state(zone, NR_FREE_CMA_PAGES)), + K(zone_page_state(zone, NR_FREE_RCU_PAGES))); printk("lowmem_reserve[]:"); for (i = 0; i < MAX_NR_ZONES; i++) printk(KERN_CONT " %ld", zone->lowmem_reserve[i]); diff --git a/mm/vmstat.c b/mm/vmstat.c index cb57714539fb5..c27caad017e35 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1212,6 +1212,7 @@ const char const vmstat_text[] = { [I(NR_ZSPAGES)] = "nr_zspages", #endif [I(NR_FREE_CMA_PAGES)] = "nr_free_cma", + [I(NR_FREE_RCU_PAGES)] = "nr_free_rcu", #ifdef CONFIG_UNACCEPTED_MEMORY [I(NR_UNACCEPTED)] = "nr_unaccepted", #endif @@ -1343,6 +1344,10 @@ const char * const vmstat_text[] = { [I(PGACTIVATE)] = "pgactivate", [I(PGDEACTIVATE)] = "pgdeactivate", [I(PGLAZYFREE)] = "pglazyfree", + [I(PGFREE_RCU)] = "pgfree_rcu", + [I(PGFREE_RCU_EXPEDITE)] = "pgfree_rcu_expedite", + [I(PGFREE_RCU_DRAIN)] = "pgfree_rcu_drain", + [I(PGFREE_RCU_MARK)] = "pgfree_rcu_mark", [I(PGFAULT)] = "pgfault", [I(PGMAJFAULT)] = "pgmajfault", @@ -2022,6 +2027,7 @@ static int vmstat_refresh(const struct ctl_table *table, int write, switch (i) { case NR_ZONE_WRITE_PENDING: case NR_FREE_CMA_PAGES: + case NR_FREE_RCU_PAGES: continue; } val = atomic_long_read(&vm_zone_stat[i]);
21/3	2026/10/19 19:58	llm	kmsan-evaluator	0m Model: gemini-3.8-flash Args: null Results: null Tokens: input: 20034 output: 30 thoughts: 871 Thoughts: KMSAN Fuzzing for RCU-Free Folios: Initial Assessment Okay, so the task is to determine if a dedicated KMSAN fuzzing session is warranted for this patch series introducing lockless page cache reads using RCU. First, I need to fully grasp the patch's changes. It looks like it's a fairly invasive change touching several core memory management subsystems. Here's the gist: we're adding RCU-based freeing for folios, enabled by `CONFIG_PAGE_RCU_FREE`. The changes propagate through various files: `fuse/dev.c` (specifically `fuse_try_move_folio` which checks the `rcu_free` flag), `splice.c` (removing `page_cache_pipe_buf_try_steal`), and several header files. We're seeing new `NR_FREE_RCU_PAGES`, `PG_rcu_free`, `AS_RCU_READ`, and related vm events being introduced. The `mm/filemap.c` file is heavily modified to handle RCU-free folios during eviction and deletion, including new functions like `filemap_read_rcu_batch` and changes to `filemap_read` to utilize `filemap_read_rcu`. Other areas like `mm/folio.c`, `mm/huge_memory.c`, `mm/memory-failure.c`, and `mm/memory_hotplug.c` are updated to account for this RCU-based freeing. However, the critical point, and the one that sparks my immediate concern, is in `mm/page_alloc.c`, specifically the function `__free_pages_prepare`. Here's where things get interesting. The code includes a check for `folio_test_rcu_free(folio)`. If that flag is set, the freeing is deferred to RCU via `rcu_free_defer()`. Critically, the function returns `false` before the call to `kmsan_free_page()`! This means KMSAN won't see the initial "free" of the page. Then, at a later time, when RCU callback runs (`rcu_free_pages`), it will call `free_unref_folios(&fbatch);`. That's where the actual free happens, but it's now decoupled from the original "free" location that KMSAN usually tracks. My immediate thought is that this introduces a significant potential for race conditions or use-after-free scenarios that KMSAN might not catch in its standard execution. I need to dive into `free_unref_folios` to understand exactly what happens there to be certain, but the deferral behavior is screaming "possible KMSAN blind spot." We need to see what `free_unref_folios` does in order to be confident.
22/3	2026/10/19 19:58	tool	grepper	0m Args: { "Expression": "void free_unref_folios", "PathPrefixes": [ "mm/" ] } Results: { "Output": "mm/page_alloc.c=3208=void free_frozen_pages_nolock(struct page page, unsigned int order)\n--\nmm/page_alloc.c-3215- /\nmm/page_alloc.c:3216:void free_unref_folios(struct folio_batch folios)\nmm/page_alloc.c-3217-{\n--\nmm/page_alloc.h=236=void free_frozen_pages(struct page page, unsigned int order);\nmm/page_alloc.h:237:void free_unref_folios(struct folio_batch *fbatch);\nmm/page_alloc.h-238-void rcu_free_sync(void);\n" }
23/3	2026/10/19 19:58	llm	kmsan-evaluator	0m Model: gemini-3.8-flash Args: null Results: null Tokens: input: 21743 output: 36 thoughts: 22 Thoughts: Here's a breakdown of my thought process: Diving into `free_unref_folios` Okay, so I'm focusing in on `free_unref_folios` within `mm/page_alloc.c`. This suggests we're dealing with the core mechanics of memory management, specifically, releasing folio objects that are no longer actively referenced. I need to understand this function. What's the context? What calls it? What's its relationship to the broader folio lifecycle, and, by extension, the page and slab allocators? How does it handle potential race conditions? I'm assuming it has to be carefully managed because we are releasing memory objects. I am expecting this to be a key area for performance optimization and debugging memory leaks or fragmentation issues. I need to understand this at a low level.
24/3	2026/10/19 19:59	tool	read-file	0m Args: { "File": "mm/page_alloc.c", "FirstLine": 3210, "LineCount": 50 } Results: { "Contents": "3210:\t\t__free_frozen_pages(page, order, FPI_NOLOCK);\n3211:\t}\n3212:\t\n3213:\t/\n3214:\t Free a batch of folios\n3215:\t /\n3216:\tvoid free_unref_folios(struct folio_batch folios)\n3217:\t{\n3218:\t\tstruct per_cpu_pages pcp = NULL;\n3219:\t\tstruct zone locked_zone = NULL;\n3220:\t\tint i, j;\n3221:\t\n3222:\t\t/* Prepare folios for freeing /\n3223:\t\tfor (i = 0, j = 0; i \u003c folios-\u003enr; i++) {\n3224:\t\t\tstruct folio folio = folios-\u003efolios[i];\n3225:\t\t\tunsigned long pfn = folio_pfn(folio);\n3226:\t\t\tunsigned int order = folio_order(folio);\n3227:\t\n3228:\t\t\tif (!__free_pages_prepare(\u0026folio-\u003epage, order, FPI_NONE))\n3229:\t\t\t\tcontinue;\n3230:\t\t\t/\n3231:\t\t\t Free orders not handled on the PCP directly to the\n3232:\t\t\t * allocator.\n3233:\t\t\t /\n3234:\t\t\tif (!pcp_allowed_order(order)) {\n3235:\t\t\t\tfree_one_page(folio_zone(folio), \u0026folio-\u003epage,\n3236:\t\t\t\t\t pfn, order, FPI_NONE);\n3237:\t\t\t\tcontinue;\n3238:\t\t\t}\n3239:\t\t\tfolio-\u003eprivate = (void )(unsigned long)order;\n3240:\t\t\tif (j != i)\n3241:\t\t\t\tfolios-\u003efolios[j] = folio;\n3242:\t\t\tj++;\n3243:\t\t}\n3244:\t\tfolios-\u003enr = j;\n3245:\t\n3246:\t\tfor (i = 0; i \u003c folios-\u003enr; i++) {\n3247:\t\t\tstruct folio folio = folios-\u003efolios[i];\n3248:\t\t\tstruct zone zone = folio_zone(folio);\n3249:\t\t\tunsigned long pfn = folio_pfn(folio);\n3250:\t\t\tunsigned int order = (unsigned long)folio-\u003eprivate;\n3251:\t\t\tint migratetype;\n3252:\t\n3253:\t\t\tfolio-\u003eprivate = NULL;\n3254:\t\t\tmigratetype = get_pfnblock_migratetype(\u0026folio-\u003epage, pfn);\n3255:\t\n3256:\t\t\t/* Different zone requires a different pcp lock */\n3257:\t\t\tif (zone != locked_zone \|\|\n3258:\t\t\t is_migrate_isolate(migratetype)) {\n3259:\t\t\t\tif (pcp) {\n" }
25/3	2026/10/19 19:59	llm	kmsan-evaluator	0m Model: gemini-3.8-flash Args: null Results: null Tokens: input: 22720 output: 28 thoughts: 2844 Thoughts: KMSAN Assessment of Lockless Page Cache Reads Okay, so `free_unref_folios` calls `__free_pages_prepare`, and that looks like the KMSAN tracking point. Since `__folio_clear_rcu_free` happens earlier, `folio_test_rcu_free` is false the second time around, and `kmsan_free_page` is called right when the page actually gets freed. That's good; it preserves the KMSAN tracking. Now, let's refresh our memory on the KMSAN vs. KASAN rules: KMSAN is for uninitialized memory reads, while KASAN and other tools handle things like out-of-bounds access and use-after-free. The critical thing is, if KASAN (or something else) catches it, KMSAN isn't needed. We only use KMSAN for uninitialized memory risks. We must examine the surrounding code carefully! This patch implements lockless page cache reads using RCU. Users read from the page cache via `filemap_read` and `shmem_file_read_iter`, which calls `filemap_read_rcu`. This looks up folios in the xarray under RCU read lock and then relies on `PG_rcu_free` and a grace period. `filemap_read_rcu_copy` then uses `kmap_local_page` and `_copy_to_iter`. The crucial question is whether `_copy_to_iter` could copy uninitialized memory. Let's trace it. It checks for `!folio_test_uptodate(folio)` and ensures that, before marking pages as "hot". The `pos >= isize` check in `filemap_read_rcu_lookup` limits the copy size, ensuring it doesn't exceed the file size. Also, the `filemap_read_rcu_size` limits the `bytes` copied to the smaller value of `isize` or the current page. So, regular* `filemap_read` copies initialized memory that either comes from disk or is user written. Now, what if the file is truncated or hole punched? The folio is removed via `page_cache_delete` or `remove_mapping`. If a page is reused after freeing, `__free_pages_prepare` ensures that it's deferred until after the RCU grace period (so the folio is not freed to buddy or PCP). This prevents it from being reallocated for something else while someone is reading it! That’s smart. Now, what about use-after-free or races with truncate? Any such issue would be a use-after-free/data race/synchronization bug caught by KASAN. And, if the folio is in the RCU grace period, `kasan_free_pages` and `kmsan_free_page` haven't run on it yet. So, does this patch introduce any uninitialized memory? Let's check: 1. `struct rcu_free_pcpu`: Static per-cpu struct, initialized at boot. BSS (zeroed). 2. `struct folio_batch fbatch`, `XA_STATE(xas, &mapping->i_pages, 0)`, `LIST_HEAD(done)`, `LIST_HEAD(list)` are also zeroed or initialized. Are kernel structures being sent to userspace? No, not really. The only user copy happens with `_copy_to_iter`, copying file data in `filemap_read_rcu_copy` or `copy_folio_to_iter`, exactly as it already did in the slow path of `filemap_read`. Could `filemap_read_rcu_copy` read past the end of a page or uninitialized bytes? It uses `kmap_local_page` and calls `_copy_to_iter` and a loop. It then ensures that `index - folio->index` is valid (or OOB). The `bytes` variable is calculated by min of `isize - pos` (or an equivalent). Page cache pages are required to be uptodate. Uptodate page cache folios always have initialized contents up to `i_size`. And filesystems are good about zeroing out pages on write/read. Wait, does KMSAN track page cache pages? I need to search the kernel for KMSAN hooks in the page cache or page_alloc to be sure, but I don't see any likely candidates in this patch!
26/3	2026/10/19 19:59	tool	grepper	0m Args: { "Expression": "kmsan_.page", "PathPrefixes": [ "mm/" ] } Results:* { "Output": "mm/kmsan/core.c=191=void kmsan_internal_set_shadow_origin(void addr, size_t size, int b,\n--\nmm/kmsan/core.c-238-\nmm/kmsan/core.c:239:struct page kmsan_vmalloc_to_page_or_null(void vaddr)\nmm/kmsan/core.c-240-{\n--\nmm/kmsan/hooks.c=110=void kmsan_kfree_large(const void ptr)\n--\nmm/kmsan/hooks.c-118-\tKMSAN_WARN_ON(ptr != page_address(page));\nmm/kmsan/hooks.c:119:\tkmsan_internal_poison_memory((void )ptr, page_size(page),\nmm/kmsan/hooks.c-120-\t\t\t\t GFP_KERNEL \u0026 ~(__GFP_RECLAIM),\n--\nmm/kmsan/hooks.c=137=void kmsan_vunmap_range_noflush(unsigned long start, unsigned long end)\n--\nmm/kmsan/hooks.c-149- /\nmm/kmsan/hooks.c:150:int kmsan_ioremap_page_range(unsigned long start, unsigned long end,\nmm/kmsan/hooks.c-151-\t\t\t phys_addr_t phys_addr, pgprot_t prot,\n--\nmm/kmsan/hooks.c-219-\nmm/kmsan/hooks.c:220:void kmsan_iounmap_page_range(unsigned long start, unsigned long end)\nmm/kmsan/hooks.c-221-{\n--\nmm/kmsan/hooks.c-234-\t i++, v_shadow += PAGE_SIZE, v_origin += PAGE_SIZE) {\nmm/kmsan/hooks.c:235:\t\tshadow = kmsan_vmalloc_to_page_or_null((void )v_shadow);\nmm/kmsan/hooks.c:236:\t\torigin = kmsan_vmalloc_to_page_or_null((void )v_origin);\nmm/kmsan/hooks.c-237-\t\t__vunmap_range_noflush(v_shadow, vmalloc_shadow(end));\n--\nmm/kmsan/hooks.c=316=EXPORT_SYMBOL_GPL(kmsan_handle_urb);\nmm/kmsan/hooks.c-317-\nmm/kmsan/hooks.c:318:static void kmsan_handle_dma_page(const void addr, size_t size,\nmm/kmsan/hooks.c-319-\t\t\t\t enum dma_data_direction dir)\n--\nmm/kmsan/hooks.c=342=void kmsan_handle_dma(phys_addr_t phys, size_t size,\n--\nmm/kmsan/hooks.c-358-\t\tto_go = min(PAGE_SIZE - page_offset, (u64)size);\nmm/kmsan/hooks.c:359:\t\tkmsan_handle_dma_page((void )addr, to_go, dir);\nmm/kmsan/hooks.c-360-\t\taddr += to_go;\n--\nmm/kmsan/init.c=101=static struct metadata_page_pair held_back[NR_PAGE_ORDERS] __initdata;\n--\nmm/kmsan/init.c-106- * We store the pointers to the returned blocks of pages in held_back[] grouped\nmm/kmsan/init.c:107: * by their order: when kmsan_memblock_free_pages() is called for the first\nmm/kmsan/init.c-108- * time with a certain order, it is reserved as a shadow block, for the second\n--\nmm/kmsan/init.c-115- /\nmm/kmsan/init.c:116:bool kmsan_memblock_free_pages(struct page page, unsigned int order)\nmm/kmsan/init.c-117-{\n--\nmm/kmsan/init.c-129-\torigin = held_back[order].origin;\nmm/kmsan/init.c:130:\tkmsan_setup_meta(page, shadow, origin, order);\nmm/kmsan/init.c-131-\n--\nmm/kmsan/init.c=168=static void do_collection(void)\n--\nmm/kmsan/init.c-175-\t\torigin = smallstack_pop(\u0026collect);\nmm/kmsan/init.c:176:\t\tkmsan_setup_meta(page, shadow, origin, collect.order);\nmm/kmsan/init.c-177-\t\t__free_pages_core(page, collect.order, MEMINIT_EARLY);\n--\nmm/kmsan/kmsan.h=165=void kmsan_internal_check_memory(void addr, size_t size,\n--\nmm/kmsan/kmsan.h-167-\nmm/kmsan/kmsan.h:168:struct page kmsan_vmalloc_to_page_or_null(void vaddr);\nmm/kmsan/kmsan.h:169:void kmsan_setup_meta(struct page page, struct page shadow,\nmm/kmsan/kmsan.h-170-\t\t struct page origin, int order);\n--\nmm/kmsan/kmsan_test.c=316=static void test_init_kmsan_vmap_vunmap(struct kunit test)\n--\nmm/kmsan/kmsan_test.c-330-\tfor (int i = 0; i \u003c npages; i++)\nmm/kmsan/kmsan_test.c:331:\t\tkmsan_check_memory(page_address(pages[i]), PAGE_SIZE);\nmm/kmsan/kmsan_test.c-332-\n--\nmm/kmsan/shadow.c=125=void kmsan_get_metadata(void address, bool is_origin)\n--\nmm/kmsan/shadow.c-151-\nmm/kmsan/shadow.c:152:void kmsan_copy_page_meta(struct page dst, struct page src)\nmm/kmsan/shadow.c-153-{\n--\nmm/kmsan/shadow.c-158-\tif (!src \|\| !page_has_metadata(src)) {\nmm/kmsan/shadow.c:159:\t\tkmsan_internal_unpoison_memory(page_address(dst), PAGE_SIZE,\nmm/kmsan/shadow.c-160-\t\t\t\t\t /checked/ false);\n--\nmm/kmsan/shadow.c-168-}\nmm/kmsan/shadow.c:169:EXPORT_SYMBOL(kmsan_copy_page_meta);\nmm/kmsan/shadow.c-170-\nmm/kmsan/shadow.c:171:void kmsan_alloc_page(struct page page, unsigned int order, gfp_t flags)\nmm/kmsan/shadow.c-172-{\n--\nmm/kmsan/shadow.c-205-\nmm/kmsan/shadow.c:206:void kmsan_free_page(struct page page, unsigned int order)\nmm/kmsan/shadow.c-207-{\n--\nmm/kmsan/shadow.c-210-\tkmsan_enter_runtime();\nmm/kmsan/shadow.c:211:\tkmsan_internal_poison_memory(page_address(page), PAGE_SIZE \u003c\u003c order,\nmm/kmsan/shadow.c-212-\t\t\t\t GFP_KERNEL \u0026 ~(__GFP_RECLAIM),\n--\nmm/kmsan/shadow.c-216-\nmm/kmsan/shadow.c:217:int kmsan_vmap_pages_range_noflush(unsigned long start, unsigned long end,\nmm/kmsan/shadow.c-218-\t\t\t\t pgprot_t prot, struct page pages,\n--\nmm/kmsan/shadow.c=276=void __init kmsan_init_alloc_meta_for_range(void start, void end)\n--\nmm/kmsan/shadow.c-298-\nmm/kmsan/shadow.c:299:void kmsan_setup_meta(struct page page, struct page shadow,\nmm/kmsan/shadow.c-300-\t\t struct page origin, int order)\n--\nmm/memory.c=3985=static vm_fault_t wp_page_copy(struct vm_fault vmf)\n--\nmm/memory.c-4029-\t\t}\nmm/memory.c:4030:\t\tkmsan_copy_page_meta(\u0026new_folio-\u003epage, vmf-\u003epage);\nmm/memory.c-4031-\t}\n--\nmm/mm_init.c=2409=void __init memblock_free_pages(unsigned long pfn, unsigned int order)\n--\nmm/mm_init.c-2419-\nmm/mm_init.c:2420:\tif (!kmsan_memblock_free_pages(page, order)) {\nmm/mm_init.c-2421-\t\t/ KMSAN will take care of these pages. /\n--\nmm/page_alloc.c=1509=static __always_inline bool __free_pages_prepare(struct page page,\n--\nmm/page_alloc.c-1536-\ttrace_mm_page_free(page, order);\nmm/page_alloc.c:1537:\tkmsan_free_page(page, order);\nmm/page_alloc.c-1538-\n--\nmm/page_alloc.c=5606=static inline bool alloc_nolock_allowed(void)\n--\nmm/page_alloc.c-5631- \nmm/page_alloc.c:5632: Specify __GFP_ZERO to make sure that call to kmsan_alloc_page() below\nmm/page_alloc.c-5633- * is safe in any context. Also zeroing the page is mandatory for\n--\nmm/page_alloc.c=5646=struct page __alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order,\n--\nmm/page_alloc.c-5722-\ttrace_mm_page_alloc(page, order, alloc_gfp, ac.migratetype);\nmm/page_alloc.c:5723:\tkmsan_alloc_page(page, order, alloc_gfp);\nmm/page_alloc.c-5724-\n--\nmm/vmalloc.c=355=int vmap_page_range(unsigned long addr, unsigned long end,\n--\nmm/vmalloc.c-363-\tif (!err)\nmm/vmalloc.c:364:\t\terr = kmsan_ioremap_page_range(addr, end, phys_addr, prot,\nmm/vmalloc.c-365-\t\t\t\t\t ioremap_max_page_shift);\n--\nmm/vmalloc.c=708=int vmap_pages_range_noflush(unsigned long addr, unsigned long end,\n--\nmm/vmalloc.c-711-{\nmm/vmalloc.c:712:\tint ret = kmsan_vmap_pages_range_noflush(addr, end, prot, pages,\nmm/vmalloc.c-713-\t\t\t\t\t\tpage_shift, gfp_mask);\n--\nmm/zsmalloc.c=1841=static int zs_page_migrate(struct page newpage, struct page *page,\n--\nmm/zsmalloc.c-1910-\tcopy_page(d_addr, s_addr);\nmm/zsmalloc.c:1911:\tkmsan_copy_page_meta(zpdesc_page(newzpdesc), zpdesc_page(zpdesc));\nmm/zsmalloc.c-1912-\tkunmap_local(d_addr);\n" }
27/3	2026/10/19 19:59	llm	kmsan-evaluator	0m Model: gemini-3.8-flash Args: null Results: null Tokens: input: 29566 output: 350 thoughts: 2796 Thoughts: KMSAN Assessment: NeedsKMSAN = false Okay, let's break this down. The patch delays the KMSAN poisoning of pages freed via RCU. The key here is the timing: before the patch, `kmsan_free_page` was called immediately when a folio was freed, which would be problematic because an RCU reader might still be accessing the page during the RCU grace period. The patch correctly delays the KMSAN poisoning to after the grace period to avoid false positives from KMSAN. The core change is about synchronization and ensuring memory validity during RCU grace periods, not introducing uninitialized memory bugs. Now, let's systematically check for uninitialized memory risks. We have to consider four key criteria: 1. Kernel structures to user space: There are no kernel structures directly sent to userspace. File page contents are copied to user space iterators, limited by `i_size` and page boundaries, and that is not a kernel structure. 2. Conditional logic and uninitialized variables: I've carefully reviewed all the relevant code paths (`filemap_read_rcu_lookup`, `filemap_read`, `rcu_free_pcpu`, `rcu_free_wait`, `rcu_free_defer`, and others). There are no uninitialized variables or struct fields used in conditional logic. All variables are either initialized at declaration, initialized through function calls (e.g., `rcu_free_init`), or explicitly set to an initial value (`false` for `migrated`, `0` for `retval`). 3. Complex data structure allocation/initialization: There are no new heap allocations of complex data structures. Per-CPU structures are initialized using standard initialization macros (`INIT_LIST_HEAD`, `spin_lock_init`, etc.). 4. Bounds checks and uninitialized bytes: The lockless reads only read from up-to-date folios, within file `i_size` and within page bounds. This eliminates the risk of reading uninitialized bytes from existing buffers. Therefore, the patch doesn't introduce any risk of uninitialized memory usage. The patch is about addressing the existing concurrency problems in the page cache subsystem by adding proper synchronization. The failure modes, however, are important: concurrency or race conditions, folio refcounting issues, deadlocks, use-after-free, and memory leaks. These are classic concurrency and memory management bugs. The relevant bug detectors are KASAN (for use-after-free and memory corruption), LOCKDEP (for locking bugs and deadlocks), KMEMLEAK (for memory leaks), and KCSAN (for race conditions and data races). These standard bug detectors cover any introduced risks, not KMSAN. I've also double-checked all relevant functions and code paths to ensure I have not missed anything: `shmem_file_read_iter`, `filemap_read`, `offline_pages`, `rcu_free_wait`, `rcu_free_defer`, `rcu_free_drain`, and `fuse_try_move_folio` to make sure all variables are properly initialized. In all cases, all variables, local and struct members, are fully initialized before use. In summary, this patch changes synchronization to prevent use-after-free conditions. Standard bug detectors will catch any issues in the code that are not caught by KMSAN, which only detects uninitialized memory bugs. Therefore, `NeedsKMSAN = false`.
28/3	2026/10/19 20:00	tool	set-results	0m Args: { "KMSANReasoning": "The patch series implements lockless page cache reads using RCU (CONFIG_PAGE_RCU_FREE) and defers the physical freeing of hot folios until after an RCU grace period.\n\nFrom a memory safety and sanitizer perspective:\n1. No uninitialized memory is exposed: All local variables and data structures (such as struct rcu_free_pcpu, list heads, and loop variables) are properly initialized before use.\n2. Kernel-to-user copies: The fast path in `filemap_read_rcu_copy()` copies file data from uptodate folios bounded strictly by the file's `i_size` and page boundaries, identically to the existing slow-path page cache read mechanisms; no uninitialized kernel structure fields or padding are leaked to user space.\n3. KMSAN hooks: The patch properly defers `kmsan_free_page()` in `__free_pages_prepare()` until the folio is actually released after the RCU grace period, ensuring KMSAN page shadow tracking remains accurate.\n4. Risk surface: The primary risks associated with this lockless RCU page cache reader patch are concurrency races (e.g. during folio splitting, truncation, migration, or hwpoison), potential use-after-free bugs, reference-counting anomalies, and locking/deadlock issues. These failure modes are covered by KASAN, LOCKDEP, and KCSAN.\n\nBecause the changes do not introduce or expose uninitialized memory reads or kernel information leaks, a dedicated KMSAN fuzzing session is not warranted.", "NeedsKMSAN": false } Results: { "KMSANReasoning": "The patch series implements lockless page cache reads using RCU (CONFIG_PAGE_RCU_FREE) and defers the physical freeing of hot folios until after an RCU grace period.\n\nFrom a memory safety and sanitizer perspective:\n1. No uninitialized memory is exposed: All local variables and data structures (such as struct rcu_free_pcpu, list heads, and loop variables) are properly initialized before use.\n2. Kernel-to-user copies: The fast path in `filemap_read_rcu_copy()` copies file data from uptodate folios bounded strictly by the file's `i_size` and page boundaries, identically to the existing slow-path page cache read mechanisms; no uninitialized kernel structure fields or padding are leaked to user space.\n3. KMSAN hooks: The patch properly defers `kmsan_free_page()` in `__free_pages_prepare()` until the folio is actually released after the RCU grace period, ensuring KMSAN page shadow tracking remains accurate.\n4. Risk surface: The primary risks associated with this lockless RCU page cache reader patch are concurrency races (e.g. during folio splitting, truncation, migration, or hwpoison), potential use-after-free bugs, reference-counting anomalies, and locking/deadlock issues. These failure modes are covered by KASAN, LOCKDEP, and KCSAN.\n\nBecause the changes do not introduce or expose uninitialized memory reads or kernel information leaks, a dedicated KMSAN fuzzing session is not warranted.", "NeedsKMSAN": false }

{
  "EnableConfigs": [
    "PAGE_RCU_FREE"
  ],
  "FocusSymbols": [
    "filemap_read_rcu",
    "filemap_read_rcu_batch",
    "filemap_read_accessed",
    "filemap_mark_hot",
    "rcu_free_defer",
    "rcu_free_workfn",
    "rcu_free_expedite_workfn",
    "rcu_free_drain",
    "rcu_free_sync",
    "rcu_free_wait"
  ],
  "KMSANReasoning": "The patch series implements lockless page cache reads using RCU (CONFIG_PAGE_RCU_FREE) and defers the physical freeing of hot folios until after an RCU grace period.\n\nFrom a memory safety and sanitizer perspective:\n1. No uninitialized memory is exposed: All local variables and data structures (such as struct rcu_free_pcpu, list heads, and loop variables) are properly initialized before use.\n2. Kernel-to-user copies: The fast path in `filemap_read_rcu_copy()` copies file data from uptodate folios bounded strictly by the file's `i_size` and page boundaries, identically to the existing slow-path page cache read mechanisms; no uninitialized kernel structure fields or padding are leaked to user space.\n3. KMSAN hooks: The patch properly defers `kmsan_free_page()` in `__free_pages_prepare()` until the folio is actually released after the RCU grace period, ensuring KMSAN page shadow tracking remains accurate.\n4. Risk surface: The primary risks associated with this lockless RCU page cache reader patch are concurrency races (e.g. during folio splitting, truncation, migration, or hwpoison), potential use-after-free bugs, reference-counting anomalies, and locking/deadlock issues. These failure modes are covered by KASAN, LOCKDEP, and KCSAN.\n\nBecause the changes do not introduce or expose uninitialized memory reads or kernel information leaks, a dedicated KMSAN fuzzing session is not warranted.",
  "NeedsKMSAN": false,
  "Reasoning": "The patch introduces lockless page cache reads and deferred RCU page freeing (PAGE_RCU_FREE). Readers can copy from hot page cache folios under RCU without taking a folio reference count, while page freeing defers memory recycling until an RCU grace period completes. This touches core MM logic (page allocator free paths, slowpath reclaim retry, memory failure, and page cache read paths in filemap and shmem) which is reachable through standard user-space file reading operations on 64-bit systems.",
  "WorthFuzzing": true
}

{
  "PatchDiff": "commit dddf59c0f0df6bc96619a328ac462a9dfff93355\nAuthor: syz-cluster \u003ctriage@syzkaller.com\u003e\nDate:   Mon Oct 19 19:57:32 2026 +0000\n\n    syz-cluster: applied patch under review\n\ndiff --git a/fs/fuse/dev.c b/fs/fuse/dev.c\nindex 4fec31fc0b845..526929c7b3f1a 100644\n--- a/fs/fuse/dev.c\n+++ b/fs/fuse/dev.c\n@@ -1152,6 +1152,10 @@ static int fuse_try_move_folio(struct fuse_copy_state *cs, struct folio **foliop\n \n \tnewfolio = page_folio(buf-\u003epage);\n \n+\t/* A lockless page cache read may still be copying from it */\n+\tif (folio_test_rcu_free(newfolio))\n+\t\tgoto out_fallback_unlock;\n+\n \tfolio_clear_uptodate(newfolio);\n \tfolio_clear_mappedtodisk(newfolio);\n \ndiff --git a/fs/splice.c b/fs/splice.c\nindex 9d8f63e2fd1ab..0133c2348da03 100644\n--- a/fs/splice.c\n+++ b/fs/splice.c\n@@ -55,56 +55,6 @@ static noinline void pipe_clear_nowait(struct file *file)\n \t} while (!try_cmpxchg(\u0026file-\u003ef_mode, \u0026fmode, fmode \u0026 ~FMODE_NOWAIT));\n }\n \n-/*\n- * Attempt to steal a page from a pipe buffer. This should perhaps go into\n- * a vm helper function, it's already simplified quite a bit by the\n- * addition of remove_mapping(). If success is returned, the caller may\n- * attempt to reuse this page for another destination.\n- */\n-static bool page_cache_pipe_buf_try_steal(struct pipe_inode_info *pipe,\n-\t\tstruct pipe_buffer *buf)\n-{\n-\tstruct folio *folio = page_folio(buf-\u003epage);\n-\tstruct address_space *mapping;\n-\n-\tfolio_lock(folio);\n-\n-\tmapping = folio_mapping(folio);\n-\tif (mapping) {\n-\t\tWARN_ON(!folio_test_uptodate(folio));\n-\n-\t\t/*\n-\t\t * At least for ext2 with nobh option, we need to wait on\n-\t\t * writeback completing on this folio, since we'll remove it\n-\t\t * from the pagecache.  Otherwise truncate wont wait on the\n-\t\t * folio, allowing the disk blocks to be reused by someone else\n-\t\t * before we actually wrote our data to them. fs corruption\n-\t\t * ensues.\n-\t\t */\n-\t\tfolio_wait_writeback(folio);\n-\n-\t\tif (!filemap_release_folio(folio, GFP_KERNEL))\n-\t\t\tgoto out_unlock;\n-\n-\t\t/*\n-\t\t * If we succeeded in removing the mapping, set LRU flag\n-\t\t * and return good.\n-\t\t */\n-\t\tif (remove_mapping(mapping, folio)) {\n-\t\t\tbuf-\u003eflags |= PIPE_BUF_FLAG_LRU;\n-\t\t\treturn true;\n-\t\t}\n-\t}\n-\n-\t/*\n-\t * Raced with truncate or failed to remove folio from current\n-\t * address space, unlock and return failure.\n-\t */\n-out_unlock:\n-\tfolio_unlock(folio);\n-\treturn false;\n-}\n-\n static void page_cache_pipe_buf_release(struct pipe_inode_info *pipe,\n \t\t\t\t\tstruct pipe_buffer *buf)\n {\n@@ -155,7 +105,6 @@ static int page_cache_pipe_buf_confirm(struct pipe_inode_info *pipe,\n const struct pipe_buf_operations page_cache_pipe_buf_ops = {\n \t.confirm\t= page_cache_pipe_buf_confirm,\n \t.release\t= page_cache_pipe_buf_release,\n-\t.try_steal\t= page_cache_pipe_buf_try_steal,\n \t.get\t\t= generic_pipe_buf_get,\n };\n \ndiff --git a/include/linux/mmzone.h b/include/linux/mmzone.h\nindex 94f9c3ff54160..3336b9e5ee999 100644\n--- a/include/linux/mmzone.h\n+++ b/include/linux/mmzone.h\n@@ -228,6 +228,7 @@ enum zone_stat_item {\n \tNR_ZSPAGES,\t\t/* allocated in zsmalloc */\n #endif\n \tNR_FREE_CMA_PAGES,\n+\tNR_FREE_RCU_PAGES,\t/* freed, waiting for an RCU grace period */\n #ifdef CONFIG_UNACCEPTED_MEMORY\n \tNR_UNACCEPTED,\n #endif\ndiff --git a/include/linux/page-flags.h b/include/linux/page-flags.h\nindex 7a863572adce7..4916b5713a74b 100644\n--- a/include/linux/page-flags.h\n+++ b/include/linux/page-flags.h\n@@ -111,6 +111,9 @@ enum pageflags {\n \tPG_swapbacked,\t\t/* Page is backed by RAM/swap */\n \tPG_unevictable,\t\t/* Page is \"unevictable\"  */\n \tPG_dropbehind,\t\t/* drop pages on IO completion */\n+#ifdef CONFIG_PAGE_RCU_FREE\n+\tPG_rcu_free,\t\t/* free only after an RCU grace period */\n+#endif\n #ifdef CONFIG_MMU\n \tPG_mlocked,\t\t/* Page is vma mlocked */\n #endif\n@@ -602,6 +605,27 @@ FOLIO_FLAG(dropbehind, FOLIO_HEAD_PAGE)\n \tFOLIO_TEST_CLEAR_FLAG(dropbehind, FOLIO_HEAD_PAGE)\n \t__FOLIO_SET_FLAG(dropbehind, FOLIO_HEAD_PAGE)\n \n+/*\n+ * A lockless lookup that sees PG_rcu_free and then still finds the folio\n+ * where it looked may read it until rcu_read_unlock().  Set under a\n+ * reference.  Cleared by the free, or when the folio's inode is evicted:\n+ * no lockless lookup can reach it then.  Not for hugetlb or\n+ * free_pages_nolock().\n+ */\n+#ifdef CONFIG_PAGE_RCU_FREE\n+FOLIO_TEST_FLAG(rcu_free, FOLIO_HEAD_PAGE)\n+\tFOLIO_TEST_SET_FLAG(rcu_free, FOLIO_HEAD_PAGE)\n+\tFOLIO_CLEAR_FLAG(rcu_free, FOLIO_HEAD_PAGE)\n+\t__FOLIO_CLEAR_FLAG(rcu_free, FOLIO_HEAD_PAGE)\n+#define __PG_RCU_FREE (1UL \u003c\u003c PG_rcu_free)\n+#else\n+FOLIO_TEST_FLAG_FALSE(rcu_free)\n+\tFOLIO_TEST_SET_FLAG_FALSE(rcu_free)\n+\tFOLIO_CLEAR_FLAG_NOOP(rcu_free)\n+\t__FOLIO_CLEAR_FLAG_NOOP(rcu_free)\n+#define __PG_RCU_FREE 0\n+#endif\n+\n #ifdef CONFIG_HIGHMEM\n /*\n  * Must use a macro here due to header dependency issues. page_zone() is not\ndiff --git a/include/linux/pagemap.h b/include/linux/pagemap.h\nindex 0adfa6605653d..6c7f58803eee4 100644\n--- a/include/linux/pagemap.h\n+++ b/include/linux/pagemap.h\n@@ -210,6 +210,7 @@ enum mapping_flags {\n \tAS_WRITEBACK_MAY_DEADLOCK_ON_RECLAIM = 9,\n \tAS_KERNEL_FILE = 10,\t/* mapping for a fake kernel file that shouldn't\n \t\t\t\t   account usage to user cgroups */\n+\tAS_RCU_READ = 11,\t/* a folio has had PG_rcu_free */\n \t/* Bits 16-25 are used for FOLIO_ORDER */\n \tAS_FOLIO_ORDER_BITS = 5,\n \tAS_FOLIO_ORDER_MIN = 16,\n@@ -269,6 +270,17 @@ static inline bool mapping_unevictable(const struct address_space *mapping)\n \treturn mapping \u0026\u0026 test_bit(AS_UNEVICTABLE, \u0026mapping-\u003eflags);\n }\n \n+static inline void mapping_set_rcu_read(struct address_space *mapping)\n+{\n+\tif (!test_bit(AS_RCU_READ, \u0026mapping-\u003eflags))\n+\t\tset_bit(AS_RCU_READ, \u0026mapping-\u003eflags);\n+}\n+\n+static inline bool mapping_rcu_read(const struct address_space *mapping)\n+{\n+\treturn test_bit(AS_RCU_READ, \u0026mapping-\u003eflags);\n+}\n+\n static inline void mapping_set_exiting(struct address_space *mapping)\n {\n \tset_bit(AS_EXITING, \u0026mapping-\u003eflags);\ndiff --git a/include/linux/swap.h b/include/linux/swap.h\nindex 5658a1634b85e..67aa5271ae6cb 100644\n--- a/include/linux/swap.h\n+++ b/include/linux/swap.h\n@@ -295,7 +295,7 @@ extern unsigned long totalreserve_pages;\n \n /* linux/mm/folio.c */\n void folio_add_lru(struct folio *folio);\n-void folio_mark_accessed(struct folio *folio);\n+bool folio_mark_accessed(struct folio *folio);\n void lru_add_drain_all(void);\n \n enum lru_cache_drained {\ndiff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h\nindex 2628ccda076a0..c1a63f070aa71 100644\n--- a/include/linux/vm_event_item.h\n+++ b/include/linux/vm_event_item.h\n@@ -36,6 +36,8 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,\n \t\tFOR_ALL_ZONES(ALLOCSTALL)\n \t\tFOR_ALL_ZONES(PGSCAN_SKIP)\n \t\tPGFREE, PGACTIVATE, PGDEACTIVATE, PGLAZYFREE,\n+\t\tPGFREE_RCU, PGFREE_RCU_EXPEDITE, PGFREE_RCU_DRAIN,\n+\t\tPGFREE_RCU_MARK,\n \t\tPGFAULT, PGMAJFAULT,\n \t\tPGLAZYFREED,\n \t\tPGREUSE,\ndiff --git a/include/trace/events/mmflags.h b/include/trace/events/mmflags.h\nindex 935893e5ea53b..47c3411a75e8f 100644\n--- a/include/trace/events/mmflags.h\n+++ b/include/trace/events/mmflags.h\n@@ -117,6 +117,12 @@ TRACE_DEFINE_ENUM(___GFP_LAST_BIT);\n #define IF_HAVE_PG_IDLE(_name)\n #endif\n \n+#ifdef CONFIG_PAGE_RCU_FREE\n+#define IF_HAVE_PG_RCU_FREE(_name) ,{1UL \u003c\u003c PG_##_name, __stringify(_name)}\n+#else\n+#define IF_HAVE_PG_RCU_FREE(_name)\n+#endif\n+\n #ifdef CONFIG_ARCH_USES_PG_ARCH_2\n #define IF_HAVE_PG_ARCH_2(_name) ,{1UL \u003c\u003c PG_##_name, __stringify(_name)}\n #else\n@@ -152,6 +158,7 @@ TRACE_DEFINE_ENUM(___GFP_LAST_BIT);\n \tDEF_PAGEFLAG_NAME(swapbacked),\t\t\t\t\t\\\n \tDEF_PAGEFLAG_NAME(unevictable),\t\t\t\t\t\\\n \tDEF_PAGEFLAG_NAME(dropbehind)\t\t\t\t\t\\\n+IF_HAVE_PG_RCU_FREE(rcu_free)\t\t\t\t\t\t\\\n IF_HAVE_PG_MLOCK(mlocked)\t\t\t\t\t\t\\\n IF_HAVE_PG_HWPOISON(hwpoison)\t\t\t\t\t\t\\\n IF_HAVE_PG_IDLE(idle)\t\t\t\t\t\t\t\\\ndiff --git a/mm/Kconfig b/mm/Kconfig\nindex 604c58199acbf..1ad1cb7969e79 100644\n--- a/mm/Kconfig\n+++ b/mm/Kconfig\n@@ -718,6 +718,21 @@ config PCP_BATCH_SCALE_MAX\n \t  latency.  This option sets the upper limit of scale factor to limit\n \t  the maximum latency.\n \n+config PAGE_RCU_FREE\n+\tdef_bool 64BIT\n+\thelp\n+\t  Reads from the page cache can copy from folios that are read\n+\t  again and again without taking a reference on them, so small\n+\t  reads of one file from many CPUs do not contend on the folio's\n+\t  reference count.  The cost is that memory freed from such\n+\t  folios becomes reusable only after an RCU grace period.\n+\n+\t  64-bit only: it takes a page flag, and those are scarce on\n+\t  32-bit.\n+\n+\t  Writing 0 to lockless_page_cache_read in debugfs turns these\n+\t  reads off.\n+\n config PHYS_ADDR_T_64BIT\n \tdef_bool 64BIT\n \ndiff --git a/mm/filemap.c b/mm/filemap.c\nindex 00fd89cf6f550..7021ed2ab6c9c 100644\n--- a/mm/filemap.c\n+++ b/mm/filemap.c\n@@ -48,6 +48,7 @@\n #include \u003clinux/rcupdate_wait.h\u003e\n #include \u003clinux/sched/mm.h\u003e\n #include \u003clinux/sysctl.h\u003e\n+#include \u003clinux/debugfs.h\u003e\n #include \u003clinux/pgalloc.h\u003e\n \n #include \u003casm/tlbflush.h\u003e\n@@ -138,6 +139,9 @@ static void page_cache_delete(struct address_space *mapping,\n \tnr = folio_nr_pages(folio);\n \n \tVM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);\n+\t/* No lockless reader can be in an evicted inode */\n+\tif (folio_test_rcu_free(folio) \u0026\u0026 mapping_exiting(mapping))\n+\t\tfolio_clear_rcu_free(folio);\n \n \txas_store(\u0026xas, shadow);\n \txas_init_marks(\u0026xas);\n@@ -306,6 +310,9 @@ static void page_cache_delete_batch(struct address_space *mapping,\n \t\t}\n \n \t\tWARN_ON_ONCE(!folio_test_locked(folio));\n+\t\t/* No lockless reader can be in an evicted inode */\n+\t\tif (folio_test_rcu_free(folio) \u0026\u0026 mapping_exiting(mapping))\n+\t\t\tfolio_clear_rcu_free(folio);\n \n \t\tfolio-\u003emapping = NULL;\n \t\t/* Leave folio-\u003eindex set: truncation lookup relies on it */\n@@ -2759,6 +2766,101 @@ static inline bool pos_same_folio(loff_t pos1, loff_t pos2, struct folio *folio)\n \treturn (pos1 \u003e\u003e shift == pos2 \u003e\u003e shift);\n }\n \n+#ifdef CONFIG_PAGE_RCU_FREE\n+/* Writing 0 to lockless_page_cache_read in debugfs turns the reads off */\n+static DEFINE_STATIC_KEY_TRUE(lockless_read_key);\n+\n+static bool lockless_read_enabled(void)\n+{\n+\treturn static_branch_likely(\u0026lockless_read_key);\n+}\n+\n+#ifdef CONFIG_DEBUG_FS\n+static int lockless_read_get(void *data, u64 *val)\n+{\n+\t*val = static_key_enabled(\u0026lockless_read_key);\n+\treturn 0;\n+}\n+\n+static int lockless_read_set(void *data, u64 val)\n+{\n+\tif (val \u003e 1)\n+\t\treturn -EINVAL;\n+\tif (val)\n+\t\tstatic_branch_enable(\u0026lockless_read_key);\n+\telse\n+\t\tstatic_branch_disable(\u0026lockless_read_key);\n+\treturn 0;\n+}\n+DEFINE_DEBUGFS_ATTRIBUTE(lockless_read_fops, lockless_read_get,\n+\t\t\t lockless_read_set, \"%llu\\n\");\n+\n+static int __init lockless_read_debugfs_init(void)\n+{\n+\tdebugfs_create_file_unsafe(\"lockless_page_cache_read\", 0644, NULL, NULL,\n+\t\t\t\t   \u0026lockless_read_fops);\n+\treturn 0;\n+}\n+late_initcall(lockless_read_debugfs_init);\n+#endif\n+#else\n+static bool lockless_read_enabled(void)\n+{\n+\treturn false;\n+}\n+#endif\n+\n+/*\n+ * Reads keep coming back to @folio: free it only after an RCU grace period\n+ * from now on.  Not if its next read needs a reference anyway.  Not if it\n+ * is unevictable: folio_mark_accessed() calls such a folio hot after at\n+ * most one access, so every ramfs folio that is read would get the flag.\n+ */\n+static void filemap_mark_hot(struct folio *folio)\n+{\n+\tstruct address_space *mapping;\n+\n+\tif (!lockless_read_enabled())\n+\t\treturn;\n+\tif (folio_test_rcu_free(folio) || !folio_test_uptodate(folio) ||\n+\t    folio_test_dropbehind(folio) || folio_test_unevictable(folio))\n+\t\treturn;\n+\t/* shmem does no readahead: there the bit is PG_reclaim */\n+\tif (folio_test_readahead(folio) \u0026\u0026 !folio_test_swapbacked(folio))\n+\t\treturn;\n+\tif (folio_test_set_rcu_free(folio))\n+\t\treturn;\n+\n+\tcount_vm_events(PGFREE_RCU_MARK, folio_nr_pages(folio));\n+\t/* NULL if the folio was truncated meanwhile */\n+\tmapping = READ_ONCE(folio-\u003emapping);\n+\tif (mapping)\n+\t\tmapping_set_rcu_read(mapping);\n+}\n+\n+/*\n+ * filemap_read() is about to copy from @folio at @pos.  @last_pos is where\n+ * the previous read of this open file ended, or how far this one has got.\n+ *\n+ * Reading on in the folio that the last read ended in is not another access\n+ * to it: a folio read in small pieces is marked accessed once.\n+ *\n+ * The folio is hot when folio_mark_accessed() finds it as accessed as the\n+ * LRU records, or when this open file goes back to data that it has just\n+ * read.\n+ */\n+void filemap_read_accessed(struct folio *folio, loff_t pos, loff_t last_pos)\n+{\n+\tbool hot;\n+\n+\tif (pos_same_folio(pos, last_pos - 1, folio))\n+\t\thot = pos \u003c last_pos;\n+\telse\n+\t\thot = folio_mark_accessed(folio);\n+\tif (hot)\n+\t\tfilemap_mark_hot(folio);\n+}\n+\n static void filemap_end_dropbehind_read(struct folio *folio)\n {\n \tif (!folio_test_dropbehind(folio))\n@@ -2771,6 +2873,216 @@ static void filemap_end_dropbehind_read(struct folio *folio)\n \t}\n }\n \n+/*\n+ * Whether filemap_read_rcu_batch() can copy from @folio, or has to leave it\n+ * to the slow path, which holds a reference for readahead, dropbehind and\n+ * folio_mark_accessed(), and deals with poisoned pages.  @last_index is the\n+ * page the previous copy ended in.\n+ */\n+static bool filemap_read_rcu_can_copy(struct folio *folio, pgoff_t index,\n+\t\t\t\t      pgoff_t last_index)\n+{\n+\tif (!folio_test_uptodate(folio) || folio_test_dropbehind(folio))\n+\t\treturn false;\n+\t/* shmem does no readahead: there the bit is PG_reclaim */\n+\tif (folio_test_readahead(folio) \u0026\u0026 !folio_test_swapbacked(folio))\n+\t\treturn false;\n+\t/* memory_failure() waits only for readers that got past this */\n+\tif (folio_contain_hwpoisoned_page(folio))\n+\t\treturn false;\n+\n+\t/*\n+\t * The slow path does not call folio_mark_accessed() on the folio that\n+\t * the previous read ended in, so copying from it skips nothing.  The\n+\t * folio's size can change without a reference: compare pages instead.\n+\t */\n+\tif (index == last_index)\n+\t\treturn true;\n+\n+\t/* Otherwise only if folio_mark_accessed() would leave it as it is */\n+\tif (lru_gen_enabled()) {\n+\t\tunsigned long all = LRU_REFS_FLAGS | BIT(PG_workingset);\n+\n+\t\treturn folio_test_unevictable(folio) ||\n+\t\t       (READ_ONCE(folio-\u003eflags.f) \u0026 all) == all;\n+\t}\n+\treturn folio_test_referenced(folio) \u0026\u0026 !folio_test_idle(folio) \u0026\u0026\n+\t       (folio_test_active(folio) || folio_test_unevictable(folio));\n+}\n+\n+/*\n+ * Return the folio at @pos if it is this file's page until rcu_read_unlock(),\n+ * and read i_size into @isize.  A folio with PG_rcu_free is freed only after\n+ * an RCU grace period.  Without a reference, the folio that xas_load()\n+ * returned can be freed and its memory reused as another folio before the\n+ * flag is tested, so look at the slot again after the flag.\n+ */\n+static struct folio *filemap_read_rcu_lookup(struct xa_state *xas,\n+\t\tstruct inode *inode, loff_t pos, loff_t *isize)\n+{\n+\tstruct folio *folio;\n+\n+\txas_set(xas, pos \u003e\u003e PAGE_SHIFT);\n+\tfolio = xas_load(xas);\n+\tif (xas_retry(xas, folio) || !folio || xa_is_value(folio))\n+\t\treturn NULL;\n+\n+\t/* Not folio_test_rcu_free(): it may not be a folio any more */\n+\tif (!(READ_ONCE(folio-\u003eflags.f) \u0026 __PG_RCU_FREE))\n+\t\treturn NULL;\n+\t/* Read i_size and slot after the flag: it implies uptodate */\n+\tsmp_rmb();\n+\t*isize = i_size_read(inode);\n+\tif (pos \u003e= *isize)\n+\t\treturn NULL;\n+\t/* i_size_read() is an acquire; a write fills the slot first */\n+\tif (xas_reload(xas) != folio)\n+\t\treturn NULL;\n+\t/* Read the rest of the folio only once the slot holds it */\n+\tsmp_rmb();\n+\treturn folio;\n+}\n+\n+/*\n+ * How much of @bytes, starting @offset into page @index, can be copied from\n+ * @folio.  A folio is in all the slots of its range or in none: when the slot\n+ * of the last page holds it too, so do the slots in between, and the pages\n+ * are this folio's.\n+ */\n+static size_t filemap_read_rcu_size(struct xa_state *xas, struct folio *folio,\n+\t\tpgoff_t index, size_t offset, size_t bytes)\n+{\n+\tpgoff_t end = index + DIV_ROUND_UP(offset + bytes, PAGE_SIZE);\n+\tpgoff_t next;\n+\n+\tif (end - index \u003e 1 \u0026\u0026 folio_test_large(folio)) {\n+\t\t/* Read without a reference: it only picks the page to check */\n+\t\tnext = folio_next_index(folio);\n+\t\tif (next \u003e index \u0026\u0026 next \u003c end)\n+\t\t\tend = next;\n+\t\txas_set(xas, end - 1);\n+\t\t/* A split after this copies PG_rcu_free to the new folios */\n+\t\tif (xas_load(xas) == folio)\n+\t\t\treturn min(bytes, (end - index) * PAGE_SIZE - offset);\n+\t}\n+\treturn min(bytes, PAGE_SIZE - offset);\n+}\n+\n+/*\n+ * Copy @bytes from @folio, starting @offset into its page at @index.  Page\n+ * by page, as copy_page_to_iter() does, without its checks for each call.\n+ * Not copy_to_iter(): its usercopy check reads the folio's order, which a\n+ * split can change.\n+ */\n+static size_t filemap_read_rcu_copy(struct folio *folio, pgoff_t index,\n+\t\tsize_t offset, size_t bytes, struct iov_iter *iter)\n+{\n+\tstruct page *page = folio_page(folio, index - folio-\u003eindex);\n+\tsize_t n = 0;\n+\n+\twhile (n \u003c bytes) {\n+\t\tsize_t len = min(bytes - n, PAGE_SIZE - offset);\n+\t\tsize_t done;\n+\t\tvoid *kaddr;\n+\n+\t\tkaddr = kmap_local_page(page);\n+\t\tdone = _copy_to_iter(kaddr + offset, len, iter);\n+\t\tkunmap_local(kaddr);\n+\t\tn += done;\n+\t\tif (done \u003c len)\n+\t\t\tbreak;\n+\t\toffset = 0;\n+\t\tpage++;\n+\t}\n+\treturn n;\n+}\n+\n+/* Bytes one RCU read section copies at most: it holds up grace periods */\n+#define FILEMAP_RCU_READ_MAX\tSZ_512K\n+\n+/*\n+ * Copy from folios with PG_rcu_free without taking references, at most\n+ * FILEMAP_RCU_READ_MAX bytes under one rcu_read_lock().  Return true if that\n+ * much was copied and the read is not done.  *last_index is the page the\n+ * previous copy ended in, set as the slow path sets last_pos.\n+ */\n+static bool filemap_read_rcu_batch(struct kiocb *iocb, struct iov_iter *iter,\n+\t\tpgoff_t *last_index, size_t *copied)\n+{\n+\tstruct address_space *mapping = iocb-\u003eki_filp-\u003ef_mapping;\n+\tstruct inode *inode = mapping-\u003ehost;\n+\tXA_STATE(xas, \u0026mapping-\u003ei_pages, 0);\n+\tsize_t budget = FILEMAP_RCU_READ_MAX;\n+\tbool more = false;\n+\n+\t/* flush_dcache_folio() is not known to be safe without a reference */\n+\tif (ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE \u0026\u0026\n+\t    mapping_writably_mapped(mapping))\n+\t\treturn false;\n+\n+\t/* A user fault cannot sleep in the RCU read section: it ends the copy */\n+\trcu_read_lock();\n+\tpagefault_disable();\n+\twhile (iov_iter_count(iter)) {\n+\t\tloff_t pos = iocb-\u003eki_pos;\n+\t\tpgoff_t index = pos \u003e\u003e PAGE_SHIFT;\n+\t\tsize_t offset = offset_in_page(pos);\n+\t\tstruct folio *folio;\n+\t\tsize_t bytes, n;\n+\t\tloff_t isize;\n+\n+\t\tif (!budget) {\n+\t\t\tmore = true;\n+\t\t\tbreak;\n+\t\t}\n+\t\tfolio = filemap_read_rcu_lookup(\u0026xas, inode, pos, \u0026isize);\n+\t\tif (!folio)\n+\t\t\tbreak;\n+\t\tif (!filemap_read_rcu_can_copy(folio, index, *last_index))\n+\t\t\tbreak;\n+\n+\t\tbytes = min_t(loff_t, isize - pos, iov_iter_count(iter));\n+\t\tbytes = filemap_read_rcu_size(\u0026xas, folio, index, offset,\n+\t\t\t\t\t      min(bytes, budget));\n+\t\tn = filemap_read_rcu_copy(folio, index, offset, bytes, iter);\n+\t\t*copied += n;\n+\t\tiocb-\u003eki_pos += n;\n+\t\tbudget -= n;\n+\t\tif (n)\n+\t\t\t*last_index = (pos + n - 1) \u003e\u003e PAGE_SHIFT;\n+\t\tif (n \u003c bytes)\n+\t\t\tbreak;\n+\t}\n+\tpagefault_enable();\n+\trcu_read_unlock();\n+\n+\treturn more;\n+}\n+\n+/*\n+ * Copy from the page cache without folio references, in batches, and\n+ * return how much was copied.  The slow path reads the rest.\n+ */\n+size_t filemap_read_rcu(struct kiocb *iocb, struct iov_iter *iter,\n+\t\tloff_t last_pos)\n+{\n+\t/* The page the previous read ended in; ULONG_MAX if none */\n+\tpgoff_t last_index = (last_pos - 1) \u003e\u003e PAGE_SHIFT;\n+\tsize_t copied = 0;\n+\n+\t/* Not when switched off, or on an inode that never had a folio flagged */\n+\tif (!lockless_read_enabled() ||\n+\t    !mapping_rcu_read(iocb-\u003eki_filp-\u003ef_mapping))\n+\t\treturn 0;\n+\n+\twhile (filemap_read_rcu_batch(iocb, iter, \u0026last_index, \u0026copied)) {\n+\t\tif (fatal_signal_pending(current))\n+\t\t\tbreak;\n+\t\tcond_resched();\n+\t}\n+\treturn copied;\n+}\n+\n /**\n  * filemap_read - Read data from the page cache.\n  * @iocb: The iocb to read.\n@@ -2796,6 +3108,7 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter,\n \tbool writably_mapped;\n \tloff_t isize, end_offset;\n \tloff_t last_pos = ra-\u003eprev_pos;\n+\tsize_t fast_read;\n \n \tif (unlikely(iocb-\u003eki_pos \u003c 0))\n \t\treturn -EINVAL;\n@@ -2807,6 +3120,14 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter,\n \tiov_iter_truncate(iter, inode-\u003ei_sb-\u003es_maxbytes - iocb-\u003eki_pos);\n \tfolio_batch_init(\u0026fbatch);\n \n+\tfast_read = filemap_read_rcu(iocb, iter, last_pos);\n+\tif (fast_read) {\n+\t\talready_read += fast_read;\n+\t\tlast_pos = iocb-\u003eki_pos;\n+\t\tif (!iov_iter_count(iter))\n+\t\t\tgoto out;\n+\t}\n+\n \tdo {\n \t\tcond_resched();\n \n@@ -2844,14 +3165,6 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter,\n \t\t */\n \t\twritably_mapped = mapping_writably_mapped(mapping);\n \n-\t\t/*\n-\t\t * When a read accesses the same folio several times, only\n-\t\t * mark it as accessed the first time.\n-\t\t */\n-\t\tif (!pos_same_folio(iocb-\u003eki_pos, last_pos - 1,\n-\t\t\t\t    fbatch.folios[0]))\n-\t\t\tfolio_mark_accessed(fbatch.folios[0]);\n-\n \t\tfor (i = 0; i \u003c folio_batch_count(\u0026fbatch); i++) {\n \t\t\tstruct folio *folio = fbatch.folios[i];\n \t\t\tsize_t fsize = folio_size(folio);\n@@ -2862,8 +3175,7 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter,\n \n \t\t\tif (end_offset \u003c folio_pos(folio))\n \t\t\t\tbreak;\n-\t\t\tif (i \u003e 0)\n-\t\t\t\tfolio_mark_accessed(folio);\n+\t\t\tfilemap_read_accessed(folio, iocb-\u003eki_pos, last_pos);\n \t\t\t/*\n \t\t\t * If users can be writing to this folio using arbitrary\n \t\t\t * virtual addresses, take care of potential aliasing\n@@ -2893,8 +3205,11 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter,\n \t\tfolio_batch_init(\u0026fbatch);\n \t} while (iov_iter_count(iter) \u0026\u0026 iocb-\u003eki_pos \u003c isize \u0026\u0026 !error);\n \n+out:\n \tfile_accessed(filp);\n-\tra-\u003eprev_pos = last_pos;\n+\t/* Readers sharing the file would bounce the line for nothing */\n+\tif (ra-\u003eprev_pos != last_pos)\n+\t\tra-\u003eprev_pos = last_pos;\n \treturn already_read ? already_read : error;\n }\n EXPORT_SYMBOL_GPL(filemap_read);\ndiff --git a/mm/folio.c b/mm/folio.c\nindex 50a6dbe55998e..6144b9cf301b3 100644\n--- a/mm/folio.c\n+++ b/mm/folio.c\n@@ -352,28 +352,34 @@ static void __lru_cache_activate_folio(struct folio *folio)\n \n #ifdef CONFIG_LRU_GEN\n \n-static void lru_gen_inc_refs(struct folio *folio)\n+/*\n+ * Return true if @folio is hot: unevictable, or its flags already held as\n+ * many accesses as they can.\n+ */\n+static bool lru_gen_inc_refs(struct folio *folio)\n {\n \tunsigned long new_flags, old_flags = READ_ONCE(folio-\u003eflags.f);\n \n \tif (folio_test_unevictable(folio))\n-\t\treturn;\n+\t\treturn true;\n \n \t/* see the comment on LRU_REFS_FLAGS */\n \tif (!folio_test_referenced(folio)) {\n \t\tset_mask_bits(\u0026folio-\u003eflags.f, LRU_REFS_MASK, BIT(PG_referenced));\n-\t\treturn;\n+\t\treturn false;\n \t}\n \n \tdo {\n \t\tif ((old_flags \u0026 LRU_REFS_MASK) == LRU_REFS_MASK) {\n-\t\t\tif (!folio_test_workingset(folio))\n-\t\t\t\tfolio_set_workingset(folio);\n-\t\t\treturn;\n+\t\t\tif (folio_test_workingset(folio))\n+\t\t\t\treturn true;\n+\t\t\tfolio_set_workingset(folio);\n+\t\t\treturn false;\n \t\t}\n \n \t\tnew_flags = old_flags + BIT(LRU_REFS_PGOFF);\n \t} while (!try_cmpxchg(\u0026folio-\u003eflags.f, \u0026old_flags, new_flags));\n+\treturn false;\n }\n \n static bool lru_gen_clear_refs(struct folio *folio)\n@@ -396,8 +402,9 @@ static bool lru_gen_clear_refs(struct folio *folio)\n \n #else /* !CONFIG_LRU_GEN */\n \n-static void lru_gen_inc_refs(struct folio *folio)\n+static bool lru_gen_inc_refs(struct folio *folio)\n {\n+\treturn false;\n }\n \n static bool lru_gen_clear_refs(struct folio *folio)\n@@ -419,15 +426,20 @@ static bool lru_gen_clear_refs(struct folio *folio)\n  *\n  * When a newly allocated folio is not yet visible, so safe for non-atomic ops,\n  * __folio_set_referenced() may be substituted for folio_mark_accessed().\n+ *\n+ * Return: true if @folio is hot: it was already as accessed as the LRU\n+ * records.  On the classic LRU that is active or unevictable, referenced\n+ * and not idle; on the multi-gen LRU, unevictable, or PG_referenced, all\n+ * the refs bits and PG_workingset set.  False for a dropbehind folio.\n  */\n-void folio_mark_accessed(struct folio *folio)\n+bool folio_mark_accessed(struct folio *folio)\n {\n+\tbool hot = false;\n+\n \tif (folio_test_dropbehind(folio))\n-\t\treturn;\n-\tif (lru_gen_enabled()) {\n-\t\tlru_gen_inc_refs(folio);\n-\t\treturn;\n-\t}\n+\t\treturn false;\n+\tif (lru_gen_enabled())\n+\t\treturn lru_gen_inc_refs(folio);\n \n \tif (!folio_test_referenced(folio)) {\n \t\tfolio_set_referenced(folio);\n@@ -437,6 +449,7 @@ void folio_mark_accessed(struct folio *folio)\n \t\t * this list is never rotated or maintained, so marking an\n \t\t * unevictable page accessed has no effect.\n \t\t */\n+\t\thot = true;\n \t} else if (!folio_test_active(folio)) {\n \t\t/*\n \t\t * If the folio is on the LRU, queue it for activation via\n@@ -450,9 +463,14 @@ void folio_mark_accessed(struct folio *folio)\n \t\t\t__lru_cache_activate_folio(folio);\n \t\tfolio_clear_referenced(folio);\n \t\tworkingset_activation(folio);\n+\t} else {\n+\t\thot = true;\n \t}\n-\tif (folio_test_idle(folio))\n+\tif (folio_test_idle(folio)) {\n \t\tfolio_clear_idle(folio);\n+\t\thot = false;\n+\t}\n+\treturn hot;\n }\n EXPORT_SYMBOL(folio_mark_accessed);\n \ndiff --git a/mm/huge_memory.c b/mm/huge_memory.c\nindex 1e5d68acf62a5..25082949a4cae 100644\n--- a/mm/huge_memory.c\n+++ b/mm/huge_memory.c\n@@ -3743,6 +3743,7 @@ static void __split_folio_to_order(struct folio *folio, int old_order,\n #endif\n \t\t\t\t (1L \u003c\u003c PG_dirty) |\n \t\t\t\t (1L \u003c\u003c PG_dropbehind) |\n+\t\t\t\t __PG_RCU_FREE |\n \t\t\t\t LRU_GEN_MASK | LRU_REFS_MASK));\n \n \t\tnew_folio-\u003emapping = folio-\u003emapping;\ndiff --git a/mm/internal.h b/mm/internal.h\nindex 38b1165212c94..ae9a4e511c827 100644\n--- a/mm/internal.h\n+++ b/mm/internal.h\n@@ -1248,6 +1248,9 @@ struct migration_target_control {\n  */\n size_t splice_folio_into_pipe(struct pipe_inode_info *pipe,\n \t\t\t      struct folio *folio, loff_t fpos, size_t size);\n+void filemap_read_accessed(struct folio *folio, loff_t pos, loff_t last_pos);\n+size_t filemap_read_rcu(struct kiocb *iocb, struct iov_iter *iter,\n+\t\t\tloff_t last_pos);\n \n static inline bool vma_is_single_threaded_private(struct vm_area_struct *vma)\n {\ndiff --git a/mm/memory-failure.c b/mm/memory-failure.c\nindex a8b03e2920ba8..dc66cac05b1f5 100644\n--- a/mm/memory-failure.c\n+++ b/mm/memory-failure.c\n@@ -211,6 +211,10 @@ static bool page_handle_poison(struct page *page, bool hugepage_or_freepage, boo\n \t\t\treturn false;\n \t}\n \n+\t/* The free of a hwpoisoned page is not deferred: wait for readers */\n+\tif (release \u0026\u0026 folio_test_rcu_free(page_folio(page)))\n+\t\tsynchronize_rcu();\n+\n \tSetPageHWPoison(page);\n \tif (release)\n \t\tput_page(page);\n@@ -1561,6 +1565,21 @@ static int get_hwpoison_page(struct page *p, unsigned long flags)\n \treturn ret;\n }\n \n+/* A page waiting for a grace period is not in the buddy yet: free it */\n+static void mf_free_rcu_pages(struct page *p)\n+{\n+\tif (!IS_ENABLED(CONFIG_PAGE_RCU_FREE))\n+\t\treturn;\n+\tif (page_count(p))\n+\t\treturn;\n+\tif (!zone_page_state_snapshot(page_zone(p), NR_FREE_RCU_PAGES))\n+\t\treturn;\n+\tif (is_free_buddy_page(p))\n+\t\treturn;\n+\n+\trcu_free_sync();\n+}\n+\n /*\n  * The caller must guarantee the folio isn't large folio, except hugetlb.\n  * try_to_unmap() can't handle it.\n@@ -2436,6 +2455,7 @@ int memory_failure(unsigned long pfn, int flags)\n \tif (res != -ENOENT)\n \t\tgoto unlock_mutex;\n \n+\tmf_free_rcu_pages(p);\n \tif (TestSetPageHWPoison(p)) {\n \t\tres = -EHWPOISON;\n \t\tif (flags \u0026 MF_ACTION_REQUIRED)\n@@ -2608,6 +2628,13 @@ int memory_failure(unsigned long pfn, int flags)\n \n identify_page_state:\n \tres = identify_page_state(pfn, p, page_flags);\n+\t/*\n+\t * The caller may unmap the page once this returns 0.  Lockless\n+\t * readers that started before the page got PG_hwpoison, or before a\n+\t * split left it in a folio of its own, may still be copying it.\n+\t */\n+\tif (!res \u0026\u0026 folio_test_rcu_free(folio))\n+\t\tsynchronize_rcu();\n \tmutex_unlock(\u0026mf_mutex);\n \treturn res;\n unlock_page:\n@@ -2999,6 +3026,7 @@ int soft_offline_page(unsigned long pfn, int flags)\n \n retry:\n \tget_online_mems();\n+\tmf_free_rcu_pages(page);\n \tret = get_hwpoison_page(page, flags | MF_SOFT_OFFLINE);\n \tput_online_mems();\n \ndiff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c\nindex 226ab9cb078ad..a8f0c95627ee1 100644\n--- a/mm/memory_hotplug.c\n+++ b/mm/memory_hotplug.c\n@@ -1967,6 +1967,7 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,\n \t\t.nid = NUMA_NO_NODE,\n \t};\n \tunsigned long flags;\n+\tbool migrated;\n \tchar *reason;\n \tint ret;\n \tunsigned long normal_pages = 0;\n@@ -2050,6 +2051,7 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,\n \n \tdo {\n \t\tpfn = start_pfn;\n+\t\tmigrated = false;\n \t\tdo {\n \t\t\t/*\n \t\t\t * Historically we always checked for any signal and\n@@ -2071,6 +2073,7 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,\n \t\t\t\t * out\n \t\t\t\t */\n \t\t\t\tdo_migrate_range(pfn, end_pfn);\n+\t\t\t\tmigrated = true;\n \t\t\t}\n \t\t} while (!ret);\n \n@@ -2092,6 +2095,10 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,\n \n \t\tret = test_pages_isolated(start_pfn, end_pfn,\n \t\t\t\t\t  PB_ISOLATE_MODE_MEM_OFFLINE);\n+\t\t/* What this pass migrated can wait for a grace period */\n+\t\tif (ret \u0026\u0026 migrated \u0026\u0026\n+\t\t    zone_page_state_snapshot(zone, NR_FREE_RCU_PAGES))\n+\t\t\trcu_free_sync();\n \n \t} while (ret);\n \ndiff --git a/mm/page_alloc.c b/mm/page_alloc.c\nindex 12fac9084c483..e63628e64cbba 100644\n--- a/mm/page_alloc.c\n+++ b/mm/page_alloc.c\n@@ -1313,6 +1313,199 @@ static inline void pgalloc_tag_sub_pages(struct alloc_tag *tag, unsigned int nr)\n \n #endif /* CONFIG_MEM_ALLOC_PROFILING */\n \n+#define RCU_FREE_EXPEDITE\t(SZ_4M \u003e\u003e PAGE_SHIFT)\n+\n+/*\n+ * Each CPU has at most one grace period in flight (@queued), covering\n+ * @batch.  Folios freed meanwhile wait on @pending for the next one.\n+ */\n+struct rcu_free_pcpu {\n+\tspinlock_t lock;\n+\tbool queued;\n+\tstruct list_head pending;\n+\tstruct list_head batch;\n+\tunsigned long nr_pending;\n+\tstruct rcu_work rwork;\n+\tstruct work_struct expedite;\n+};\n+\n+static DEFINE_PER_CPU(struct rcu_free_pcpu, rcu_free_pcpu);\n+\n+static struct workqueue_struct *rcu_free_wq __ro_after_init;\n+\n+static void rcu_free_start(struct rcu_free_pcpu *r)\n+{\n+\tlockdep_assert_held(\u0026r-\u003elock);\n+\n+\tif (r-\u003equeued || list_empty(\u0026r-\u003epending))\n+\t\treturn;\n+\n+\tlist_splice_init(\u0026r-\u003epending, \u0026r-\u003ebatch);\n+\tr-\u003enr_pending = 0;\n+\tr-\u003equeued = true;\n+\tWARN_ON_ONCE(!queue_rcu_work(rcu_free_wq, \u0026r-\u003erwork));\n+}\n+\n+static void rcu_free_take(struct rcu_free_pcpu *r, struct list_head *list)\n+{\n+\tlockdep_assert_held(\u0026r-\u003elock);\n+\n+\tlist_splice_tail_init(\u0026r-\u003ebatch, list);\n+\tlist_splice_tail_init(\u0026r-\u003epending, list);\n+\tr-\u003enr_pending = 0;\n+}\n+\n+static unsigned long rcu_free_pages(struct list_head *list)\n+{\n+\tstruct folio_batch fbatch;\n+\tstruct folio *folio, *next;\n+\tunsigned long nr = 0;\n+\n+\tfolio_batch_init(\u0026fbatch);\n+\tlist_for_each_entry_safe(folio, next, list, lru) {\n+\t\tnr += folio_nr_pages(folio);\n+\t\tmod_zone_page_state(folio_zone(folio), NR_FREE_RCU_PAGES,\n+\t\t\t\t    -folio_nr_pages(folio));\n+\t\tif (!folio_batch_add(\u0026fbatch, folio)) {\n+\t\t\tfree_unref_folios(\u0026fbatch);\n+\t\t\tcond_resched_tasks_rcu_qs();\n+\t\t}\n+\t}\n+\tif (folio_batch_count(\u0026fbatch))\n+\t\tfree_unref_folios(\u0026fbatch);\n+\treturn nr;\n+}\n+\n+/* Free what rcu_free_take() took: the grace period starts after that */\n+static unsigned long rcu_free_flush(struct list_head *list)\n+{\n+\tif (list_empty(list))\n+\t\treturn 0;\n+\n+\tsynchronize_rcu_expedited();\n+\treturn rcu_free_pages(list);\n+}\n+\n+static void rcu_free_workfn(struct work_struct *work)\n+{\n+\tstruct rcu_free_pcpu *r = container_of(to_rcu_work(work),\n+\t\t\t\t\t       struct rcu_free_pcpu, rwork);\n+\tLIST_HEAD(done);\n+\n+\tspin_lock_irq(\u0026r-\u003elock);\n+\tlist_splice_init(\u0026r-\u003ebatch, \u0026done);\n+\tr-\u003equeued = false;\n+\trcu_free_start(r);\n+\tspin_unlock_irq(\u0026r-\u003elock);\n+\n+\trcu_free_pages(\u0026done);\n+}\n+\n+static void rcu_free_expedite_workfn(struct work_struct *work)\n+{\n+\tstruct rcu_free_pcpu *r = container_of(work, struct rcu_free_pcpu,\n+\t\t\t\t\t       expedite);\n+\tLIST_HEAD(list);\n+\n+\tspin_lock_irq(\u0026r-\u003elock);\n+\trcu_free_take(r, \u0026list);\n+\tspin_unlock_irq(\u0026r-\u003elock);\n+\n+\tif (rcu_free_flush(\u0026list))\n+\t\tcount_vm_event(PGFREE_RCU_EXPEDITE);\n+}\n+\n+static void rcu_free_defer(struct folio *folio)\n+{\n+\tstruct rcu_free_pcpu *r = raw_cpu_ptr(\u0026rcu_free_pcpu);\n+\tstruct zone *zone = folio_zone(folio);\n+\tunsigned long free, waiting;\n+\tunsigned long flags;\n+\n+\tspin_lock_irqsave(\u0026r-\u003elock, flags);\n+\t__mod_zone_page_state(zone, NR_FREE_RCU_PAGES, folio_nr_pages(folio));\n+\tlist_add_tail(\u0026folio-\u003elru, \u0026r-\u003epending);\n+\tr-\u003enr_pending += folio_nr_pages(folio);\n+\trcu_free_start(r);\n+\tspin_unlock_irqrestore(\u0026r-\u003elock, flags);\n+\n+\t/* Unlocked: a stale count delays the expedite by one free */\n+\tif (data_race(r-\u003enr_pending) \u003c= RCU_FREE_EXPEDITE ||\n+\t    work_pending(\u0026r-\u003eexpedite))\n+\t\treturn;\n+\n+\t/*\n+\t * An expedited grace period interrupts every CPU: only while the\n+\t * zone is short of memory, or while more than an eighth of what it\n+\t * has free waits here.\n+\t */\n+\tfree = zone_page_state(zone, NR_FREE_PAGES);\n+\twaiting = zone_page_state(zone, NR_FREE_RCU_PAGES);\n+\tif (free \u003c high_wmark_pages(zone) || waiting \u003e free / 8)\n+\t\tqueue_work(rcu_free_wq, \u0026r-\u003eexpedite);\n+}\n+\n+/*\n+ * Free what every CPU's lists hold.  A concurrent drain holds the lock\n+ * until it has freed what it took.\n+ */\n+static void rcu_free_drain(void)\n+{\n+\tstatic DEFINE_MUTEX(lock);\n+\tLIST_HEAD(list);\n+\tint cpu;\n+\n+\tmutex_lock(\u0026lock);\n+\tfor_each_possible_cpu(cpu) {\n+\t\tstruct rcu_free_pcpu *r = per_cpu_ptr(\u0026rcu_free_pcpu, cpu);\n+\n+\t\tspin_lock_irq(\u0026r-\u003elock);\n+\t\trcu_free_take(r, \u0026list);\n+\t\tspin_unlock_irq(\u0026r-\u003elock);\n+\t}\n+\tif (rcu_free_flush(\u0026list))\n+\t\tcount_vm_event(PGFREE_RCU_DRAIN);\n+\tmutex_unlock(\u0026lock);\n+}\n+\n+/* Free every waiting folio, including those a work took off the lists */\n+void rcu_free_sync(void)\n+{\n+\tint cpu;\n+\n+\tif (!IS_ENABLED(CONFIG_PAGE_RCU_FREE))\n+\t\treturn;\n+\n+\trcu_free_drain();\n+\tfor_each_possible_cpu(cpu) {\n+\t\tstruct rcu_free_pcpu *r = per_cpu_ptr(\u0026rcu_free_pcpu, cpu);\n+\n+\t\tflush_work(\u0026r-\u003erwork.work);\n+\t\tflush_work(\u0026r-\u003eexpedite);\n+\t}\n+}\n+\n+static void __init rcu_free_init(void)\n+{\n+\tint cpu;\n+\n+\tif (!IS_ENABLED(CONFIG_PAGE_RCU_FREE))\n+\t\treturn;\n+\n+\trcu_free_wq = alloc_workqueue(\"rcu_free\",\n+\t\t\t\t      WQ_MEM_RECLAIM | WQ_PERCPU, 0);\n+\n+\tfor_each_possible_cpu(cpu) {\n+\t\tstruct rcu_free_pcpu *r = per_cpu_ptr(\u0026rcu_free_pcpu, cpu);\n+\n+\t\tspin_lock_init(\u0026r-\u003elock);\n+\t\tINIT_LIST_HEAD(\u0026r-\u003epending);\n+\t\tINIT_LIST_HEAD(\u0026r-\u003ebatch);\n+\t\tINIT_RCU_WORK(\u0026r-\u003erwork, rcu_free_workfn);\n+\t\tINIT_WORK(\u0026r-\u003eexpedite, rcu_free_expedite_workfn);\n+\t}\n+}\n+\n static __always_inline bool __free_pages_prepare(struct page *page,\n \t\tunsigned int order, fpi_t fpi_flags)\n {\n@@ -1327,6 +1520,19 @@ static __always_inline bool __free_pages_prepare(struct page *page,\n \n \tVM_BUG_ON_PAGE(PageTail(page), page);\n \n+\t/*\n+\t * Not for a hwpoisoned order-0 page: it never reaches the allocator,\n+\t * and page_handle_poison() takes a reference right after this free.\n+\t */\n+\tif (unlikely(folio_test_rcu_free(folio)) \u0026\u0026\n+\t    !(PageHWPoison(page) \u0026\u0026 !order)) {\n+\t\tVM_WARN_ON_ONCE(fpi_flags \u0026 FPI_NOLOCK);\n+\t\t__folio_clear_rcu_free(folio);\n+\t\tcount_vm_events(PGFREE_RCU, 1 \u003c\u003c order);\n+\t\trcu_free_defer(folio);\n+\t\treturn false;\n+\t}\n+\n \ttrace_mm_page_free(page, order);\n \tkmsan_free_page(page, order);\n \n@@ -4626,6 +4832,52 @@ bool gfp_pfmemalloc_allowed(gfp_t gfp_mask)\n \treturn !!__gfp_pfmemalloc_flags(gfp_mask);\n }\n \n+/*\n+ * Reclaim is about to give up: if the pages waiting for a grace period\n+ * would bring an eligible zone above its min watermark, free them and\n+ * retry.  Not for a dying task, which frees memory by exiting.\n+ */\n+static bool rcu_free_wait(gfp_t gfp_mask, unsigned int order,\n+\t\t\t  struct alloc_context *ac, int alloc_flags,\n+\t\t\t  int *rcu_free_waits)\n+{\n+\tstruct zone *zone;\n+\tstruct zoneref *z;\n+\n+\tif (!IS_ENABLED(CONFIG_PAGE_RCU_FREE))\n+\t\treturn false;\n+\tif (tsk_is_oom_victim(current) || fatal_signal_pending(current))\n+\t\treturn false;\n+\tif (*rcu_free_waits \u003e= MAX_RECLAIM_RETRIES ||\n+\t    order \u003e PAGE_ALLOC_COSTLY_ORDER)\n+\t\treturn false;\n+\n+\tfor_each_zone_zonelist_nodemask(zone, z, ac-\u003ezonelist,\n+\t\t\t\tac-\u003ehighest_zoneidx, ac-\u003enodemask) {\n+\t\tunsigned long waiting, available;\n+\n+\t\tif (cpusets_enabled() \u0026\u0026 (alloc_flags \u0026 ALLOC_CPUSET) \u0026\u0026\n+\t\t    !__cpuset_zone_allowed(zone, gfp_mask))\n+\t\t\tcontinue;\n+\n+\t\twaiting = zone_page_state_snapshot(zone, NR_FREE_RCU_PAGES);\n+\t\tif (!waiting)\n+\t\t\tcontinue;\n+\n+\t\tavailable = waiting +\n+\t\t\t    zone_page_state_snapshot(zone, NR_FREE_PAGES);\n+\t\tif (__zone_watermark_ok(zone, order, min_wmark_pages(zone),\n+\t\t\t\t\tac-\u003ehighest_zoneidx, alloc_flags,\n+\t\t\t\t\tavailable)) {\n+\t\t\t(*rcu_free_waits)++;\n+\t\t\trcu_free_drain();\n+\t\t\treturn true;\n+\t\t}\n+\t}\n+\n+\treturn false;\n+}\n+\n /*\n  * Checks whether it makes sense to retry the reclaim to make a forward progress\n  * for the given allocation request.\n@@ -4639,7 +4891,8 @@ bool gfp_pfmemalloc_allowed(gfp_t gfp_mask)\n static inline bool\n should_reclaim_retry(gfp_t gfp_mask, unsigned order,\n \t\t     struct alloc_context *ac, int alloc_flags,\n-\t\t     bool did_some_progress, int *no_progress_loops)\n+\t\t     bool did_some_progress, int *no_progress_loops,\n+\t\t     int *rcu_free_waits)\n {\n \tstruct zone *zone;\n \tstruct zoneref *z;\n@@ -4650,10 +4903,12 @@ should_reclaim_retry(gfp_t gfp_mask, unsigned order,\n \t * their order will become available due to high fragmentation so\n \t * always increment the no progress counter for them\n \t */\n-\tif (did_some_progress \u0026\u0026 order \u003c= PAGE_ALLOC_COSTLY_ORDER)\n+\tif (did_some_progress \u0026\u0026 order \u003c= PAGE_ALLOC_COSTLY_ORDER) {\n \t\t*no_progress_loops = 0;\n-\telse\n+\t\t*rcu_free_waits = 0;\n+\t} else {\n \t\t(*no_progress_loops)++;\n+\t}\n \n \tif (*no_progress_loops \u003e MAX_RECLAIM_RETRIES)\n \t\tgoto out;\n@@ -4706,6 +4961,10 @@ should_reclaim_retry(gfp_t gfp_mask, unsigned order,\n \telse\n \t\tcond_resched();\n out:\n+\tif (!ret \u0026\u0026 rcu_free_wait(gfp_mask, order, ac, alloc_flags,\n+\t\t\t\t  rcu_free_waits))\n+\t\treturn true;\n+\n \t/* Before OOM, exhaust highatomic_reserve */\n \tif (!ret)\n \t\treturn unreserve_highatomic_pageblock(ac, true);\n@@ -4795,6 +5054,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,\n \tenum compact_result compact_result;\n \tint compaction_retries;\n \tint no_progress_loops;\n+\tint rcu_free_waits;\n \tunsigned int cpuset_mems_cookie;\n \tunsigned int zonelist_iter_cookie;\n \tint reserve_flags;\n@@ -4819,6 +5079,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,\n restart:\n \tcompaction_retries = 0;\n \tno_progress_loops = 0;\n+\trcu_free_waits = 0;\n \tcompact_result = COMPACT_SKIPPED;\n \tcompact_priority = DEF_COMPACT_PRIORITY;\n \tcpuset_mems_cookie = read_mems_allowed_begin();\n@@ -4997,7 +5258,8 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,\n \t\tgoto restart;\n \n \tif (should_reclaim_retry(gfp_mask, order, ac, alloc_flags,\n-\t\t\t\t did_some_progress \u003e 0, \u0026no_progress_loops))\n+\t\t\t\t did_some_progress \u003e 0, \u0026no_progress_loops,\n+\t\t\t\t \u0026rcu_free_waits))\n \t\tgoto retry;\n \n \t/*\n@@ -5040,6 +5302,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,\n \t/* Retry as long as the OOM killer is making progress */\n \tif (did_some_progress) {\n \t\tno_progress_loops = 0;\n+\t\trcu_free_waits = 0;\n \t\tgoto retry;\n \t}\n \n@@ -6401,6 +6664,8 @@ void __init setup_per_cpu_pageset(void)\n \tfor_each_online_pgdat(pgdat)\n \t\tpgdat-\u003eper_cpu_nodestats =\n \t\t\talloc_percpu(struct per_cpu_nodestat);\n+\n+\trcu_free_init();\n }\n \n __meminit void zone_pcp_init(struct zone *zone)\n@@ -7384,7 +7649,16 @@ int alloc_contig_frozen_range_noprof(unsigned long start, unsigned long end,\n \touter_start = find_large_buddy(start);\n \n \t/* Make sure the range is really isolated. */\n-\tif (test_pages_isolated(outer_start, end, mode)) {\n+\tret = test_pages_isolated(outer_start, end, mode);\n+\tif (ret \u0026\u0026 zone_page_state_snapshot(cc.zone, NR_FREE_RCU_PAGES)) {\n+\t\trcu_free_sync();\n+\n+\t\t/* A freed folio can merge with a free buddy below start */\n+\t\touter_start = find_large_buddy(start);\n+\t\tret = test_pages_isolated(outer_start, end, mode);\n+\t}\n+\n+\tif (ret) {\n \t\tret = -EBUSY;\n \t\tgoto done;\n \t}\ndiff --git a/mm/page_alloc.h b/mm/page_alloc.h\nindex b9259deddb59d..f9e200581e2f5 100644\n--- a/mm/page_alloc.h\n+++ b/mm/page_alloc.h\n@@ -235,6 +235,7 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order, int nid,\n \talloc_hooks(__alloc_frozen_pages_noprof(__VA_ARGS__))\n void free_frozen_pages(struct page *page, unsigned int order);\n void free_unref_folios(struct folio_batch *fbatch);\n+void rcu_free_sync(void);\n \n #ifdef CONFIG_NUMA\n struct page *alloc_frozen_pages_noprof(gfp_t, unsigned int order);\ndiff --git a/mm/shmem.c b/mm/shmem.c\nindex 848316eaa7f4f..e6c18e745f816 100644\n--- a/mm/shmem.c\n+++ b/mm/shmem.c\n@@ -3309,10 +3309,18 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)\n \tstruct file *file = iocb-\u003eki_filp;\n \tstruct inode *inode = file_inode(file);\n \tstruct address_space *mapping = inode-\u003ei_mapping;\n+\tloff_t last_pos = file-\u003ef_ra.prev_pos;\n \tpgoff_t index;\n \tunsigned long offset;\n \tint error = 0;\n-\tssize_t retval = 0;\n+\tssize_t retval;\n+\n+\tretval = filemap_read_rcu(iocb, to, last_pos);\n+\tif (retval) {\n+\t\tlast_pos = iocb-\u003eki_pos;\n+\t\tif (!iov_iter_count(to))\n+\t\t\tgoto out;\n+\t}\n \n \tfor (;;) {\n \t\tstruct folio *folio = NULL;\n@@ -3378,11 +3386,7 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)\n \t\t\t\t\tflush_dcache_page(page);\n \t\t\t}\n \n-\t\t\t/*\n-\t\t\t * Mark the folio accessed if we read the beginning.\n-\t\t\t */\n-\t\t\tif (!offset)\n-\t\t\t\tfolio_mark_accessed(folio);\n+\t\t\tfilemap_read_accessed(folio, iocb-\u003eki_pos, last_pos);\n \t\t\t/*\n \t\t\t * Ok, we have the page, and it's up-to-date, so\n \t\t\t * now we can copy it to user space...\n@@ -3410,6 +3414,7 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)\n \n \t\tretval += ret;\n \t\tiocb-\u003eki_pos += ret;\n+\t\tlast_pos = iocb-\u003eki_pos;\n \n \t\tif (!iov_iter_count(to))\n \t\t\tbreak;\n@@ -3419,8 +3424,11 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)\n \t\t}\n \t\tcond_resched();\n \t}\n-\n+out:\n \tfile_accessed(file);\n+\t/* Readers sharing the file would bounce the line for nothing */\n+\tif (file-\u003ef_ra.prev_pos != last_pos)\n+\t\tfile-\u003ef_ra.prev_pos = last_pos;\n \treturn retval ? retval : error;\n }\n \ndiff --git a/mm/show_mem.c b/mm/show_mem.c\nindex b938cbcd774a1..0d3bddbda108d 100644\n--- a/mm/show_mem.c\n+++ b/mm/show_mem.c\n@@ -204,7 +204,7 @@ static void show_free_areas(unsigned int filter, const nodemask_t *nodemask,\n \t\t\" mapped:%lu shmem:%lu pagetables:%lu\\n\"\n \t\t\" sec_pagetables:%lu bounce:%lu\\n\"\n \t\t\" kernel_misc_reclaimable:%lu\\n\"\n-\t\t\" free:%lu free_pcp:%lu free_cma:%lu\\n\",\n+\t\t\" free:%lu free_pcp:%lu free_cma:%lu free_rcu:%lu\\n\",\n \t\tglobal_node_page_state(NR_ACTIVE_ANON),\n \t\tglobal_node_page_state(NR_INACTIVE_ANON),\n \t\tglobal_node_page_state(NR_ISOLATED_ANON),\n@@ -224,7 +224,8 @@ static void show_free_areas(unsigned int filter, const nodemask_t *nodemask,\n \t\tglobal_node_page_state(NR_KERNEL_MISC_RECLAIMABLE),\n \t\tglobal_zone_page_state(NR_FREE_PAGES),\n \t\tfree_pcp,\n-\t\tglobal_zone_page_state(NR_FREE_CMA_PAGES));\n+\t\tglobal_zone_page_state(NR_FREE_CMA_PAGES),\n+\t\tglobal_zone_page_state(NR_FREE_RCU_PAGES));\n \n \tfor_each_online_pgdat(pgdat) {\n \t\tif (show_mem_node_skip(filter, pgdat-\u003enode_id, nodemask))\n@@ -325,6 +326,7 @@ static void show_free_areas(unsigned int filter, const nodemask_t *nodemask,\n \t\t\t\" free_pcp:%lukB\"\n \t\t\t\" local_pcp:%lukB\"\n \t\t\t\" free_cma:%lukB\"\n+\t\t\t\" free_rcu:%lukB\"\n \t\t\t\"\\n\",\n \t\t\tzone-\u003ename,\n \t\t\tK(zone_page_state(zone, NR_FREE_PAGES)),\n@@ -351,7 +353,8 @@ static void show_free_areas(unsigned int filter, const nodemask_t *nodemask,\n \t\t\t0UL,\n \t\t\tK(free_pcp),\n \t\t\tK((unsigned long)this_cpu_read(zone-\u003eper_cpu_pageset-\u003ecount)),\n-\t\t\tK(zone_page_state(zone, NR_FREE_CMA_PAGES)));\n+\t\t\tK(zone_page_state(zone, NR_FREE_CMA_PAGES)),\n+\t\t\tK(zone_page_state(zone, NR_FREE_RCU_PAGES)));\n \t\tprintk(\"lowmem_reserve[]:\");\n \t\tfor (i = 0; i \u003c MAX_NR_ZONES; i++)\n \t\t\tprintk(KERN_CONT \" %ld\", zone-\u003elowmem_reserve[i]);\ndiff --git a/mm/vmstat.c b/mm/vmstat.c\nindex cb57714539fb5..c27caad017e35 100644\n--- a/mm/vmstat.c\n+++ b/mm/vmstat.c\n@@ -1212,6 +1212,7 @@ const char * const vmstat_text[] = {\n \t[I(NR_ZSPAGES)]\t\t\t\t= \"nr_zspages\",\n #endif\n \t[I(NR_FREE_CMA_PAGES)]\t\t\t= \"nr_free_cma\",\n+\t[I(NR_FREE_RCU_PAGES)]\t\t\t= \"nr_free_rcu\",\n #ifdef CONFIG_UNACCEPTED_MEMORY\n \t[I(NR_UNACCEPTED)]\t\t\t= \"nr_unaccepted\",\n #endif\n@@ -1343,6 +1344,10 @@ const char * const vmstat_text[] = {\n \t[I(PGACTIVATE)]\t\t\t\t= \"pgactivate\",\n \t[I(PGDEACTIVATE)]\t\t\t= \"pgdeactivate\",\n \t[I(PGLAZYFREE)]\t\t\t\t= \"pglazyfree\",\n+\t[I(PGFREE_RCU)]\t\t\t\t= \"pgfree_rcu\",\n+\t[I(PGFREE_RCU_EXPEDITE)]\t\t= \"pgfree_rcu_expedite\",\n+\t[I(PGFREE_RCU_DRAIN)]\t\t\t= \"pgfree_rcu_drain\",\n+\t[I(PGFREE_RCU_MARK)]\t\t\t= \"pgfree_rcu_mark\",\n \n \t[I(PGFAULT)]\t\t\t\t= \"pgfault\",\n \t[I(PGMAJFAULT)]\t\t\t\t= \"pgmajfault\",\n@@ -2022,6 +2027,7 @@ static int vmstat_refresh(const struct ctl_table *table, int write,\n \t\tswitch (i) {\n \t\tcase NR_ZONE_WRITE_PENDING:\n \t\tcase NR_FREE_CMA_PAGES:\n+\t\tcase NR_FREE_RCU_PAGES:\n \t\t\tcontinue;\n \t\t}\n \t\tval = atomic_long_read(\u0026vm_zone_stat[i]);\n"
}

{
  "EnableConfigs": [
    "PAGE_RCU_FREE"
  ],
  "FocusSymbols": [
    "filemap_read_rcu",
    "filemap_read_rcu_batch",
    "filemap_read_accessed",
    "filemap_mark_hot",
    "rcu_free_defer",
    "rcu_free_workfn",
    "rcu_free_expedite_workfn",
    "rcu_free_drain",
    "rcu_free_sync",
    "rcu_free_wait"
  ],
  "Reasoning": "The patch introduces lockless page cache reads and deferred RCU page freeing (PAGE_RCU_FREE). Readers can copy from hot page cache folios under RCU without taking a folio reference count, while page freeing defers memory recycling until an RCU grace period completes. This touches core MM logic (page allocator free paths, slowpath reclaim retry, memory failure, and page cache read paths in filemap and shmem) which is reachable through standard user-space file reading operations on 64-bit systems.",
  "WorthFuzzing": true
}

You are an expert Linux kernel maintainer and security engineer.
Your job is to review a provided patch series and evaluate whether it warrants fuzzing with syzkaller.

IMPORTANT: The changes have ALREADY been applied and committed as the HEAD commit in
your workspace. Do NOT rely on internal assumptions. You must actively use your code access
tools to inspect the actual source code, callers, and surrounding context.

================================================================================
1. CORE TRIAGE PHILOSOPHY
================================================================================
The goal of patch fuzzing is to discover crashes, regressions, exposed latent bugs,
and newly triggered assertions introduced by the patch series.

- REACHABILITY IS THE PRIMARY GATE:
  Fuzzing can only discover bugs in code that can actually execute in standard virtualized
  environments (GCE or QEMU, utilizing software-emulated devices like USB gadgets, netdev, tun/tap).
  If the modified code is structurally unreachable (see Section 2), it MUST NOT be fuzzed,
  regardless of whether it adds assertions or complex logic.

- DO NOT BLINDLY TRUST "NO FUNCTIONAL CHANGE" (NFCI) OR "REFACTORING" CLAIMS:
  Patch authors routinely label changes as "cleanups", "refactorings", or state
  "No functional change intended". Do NOT take these claims at face value.
  Code refactorings that rearrange logic, introduce helper functions, or alter state management
  in core subsystems frequently introduce subtle semantic shifts or uncover latent kernel bugs.
  If reachable executable code is modified or refactored, it MUST be fuzzed.

- NEW OR MODIFIED ASSERTIONS IN REACHABLE CODE MUST BE FUZZED:
  When a patch introduces or modifies runtime checks or assertions (e.g., WARN_ON*, VM_WARN_ON*,
  BUG_ON*, lockdep_assert*) in reachable code paths, it enforces new or stricter invariants.
  Even if the author believes the invariant always holds, fuzzing is essential to verify whether
  an unusual sequence of operations can violate it.

================================================================================
2. WHEN TO RETURN WorthFuzzing=false (NEGATIVE CRITERIA)
================================================================================
Return WorthFuzzing=false ONLY IF all modified code falls strictly into one or more of these categories:

- Non-kernel and non-executable changes:
  * Modifications to Documentation/, comments, or spelling fixes.
  * User-space directories, self-tests, samples, or scripts (e.g., tools/, samples/, scripts/, usr/)
    that do not affect the compiled kernel image (vmlinux) or kernel modules.
  * Purely decorative logging (e.g., message strings in pr_err, printk, dev_info) or tracepoints
    that do not alter control flow or data structures.
  * Build system or Kconfig changes that do not alter compiled C logic.
- Structurally unreachable hardware:
  * Vendor-specific PCIe switches, SmartNICs, or GPU drivers (e.g., mlxsw, pds_core, qed,
    ionic, amdgpu) requiring physical ASIC/PCIe cards not emulated in standard QEMU.
- Unreachable execution paths:
  * Driver teardown callbacks (.remove, .shutdown, pci_unregister_driver) executed only during
    physical PCI hot-unplug or manual sysfs driver unbinding.
  * Code paths exclusive to architectures other than the target architecture.

================================================================================
3. WHEN TO RETURN WorthFuzzing=true (POSITIVE CRITERIA)
================================================================================
Return WorthFuzzing=true whenever the patch touches reachable executable code, including:
- Core Subsystems:
  * Any logic modifications in memory management (mm/), synchronization/locking (kernel/locking/),
    BPF, scheduler, core networking, VFS, or syscall handling.
- Refactorings and Code Cleanups:
  * Any restructuring of reachable data structures, helper abstractions, or algorithm flows.
- Runtime Assertions and Defensive Checks:
  * Any introduction or alteration of assertions (WARN_ON*, VM_WARN_ON*, BUG_ON*, etc.) in reachable paths.
- Reachable Drivers and Protocols:
  * Drivers accessible via virtual buses (virtio, USB gadget, loopback, netlink, binder, sockets, etc.).

================================================================================
4. EXTRACTING FocusSymbols (PREVENTING DILUTION)
================================================================================
When WorthFuzzing=true, you must extract specific kernel functions into FocusSymbols to guide the fuzzer:

- AVOID UBIQUITOUS LIFECYCLE HOT-PATHS:
  Do NOT list generic, ubiquitous functions called by almost every program in the corpus
  (including, but not limited to: general memory allocators and deallocators, page fault
  and trap handlers, or core synchronization primitives; this is not an exhaustive list).
  Listing ubiquitous functions causes the fuzzer to classify thousands of unrelated tests as "focused",
  which severely dilutes fuzzing effort away from the actual changes.

- TARGET SPECIFIC FEATURE LOGIC AND ENTRYPOINTS:
  List functions that specifically implement the logic being added or altered, or direct API entrypoints
  for the subsystem feature under review.

- HANDLING STATIC INLINE FUNCTIONS IN HEADERS (.h):
  Compiler-inlined static functions (such as static inlines in mm/*.h or include/linux/*.h) lack
  distinct symbol addresses in vmlinux and cannot be targeted directly by symbol coverage filters.
  If the changes are primarily in static inline helpers, identify non-static, feature-specific caller
  functions in .c files that exercise them (avoiding ubiquitous lifecycle wrappers).

================================================================================
5. IDENTIFYING EnableConfigs
================================================================================
Identify any specific CONFIG_ options required to properly compile and reach the modified code:
- Inspect Kconfig files and #ifdef guards; do not make assumptions.
- Check "depends on" lines in Kconfig to include any non-standard parent subsystem configs needed.
- Strip any 'CONFIG_' prefix (e.g., return "NET_IPV4" instead of "CONFIG_NET_IPV4").
Prefer calling several tools at the same time to save round-trips.


Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.

Target architecture: amd64

For your convenience, here is the diff of the changes:
commit dddf59c0f0df6bc96619a328ac462a9dfff93355
Author: syz-cluster <triage@syzkaller.com>
Date:   Mon Oct 19 19:57:32 2026 +0000

    syz-cluster: applied patch under review

diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c
index 4fec31fc0b845..526929c7b3f1a 100644
--- a/fs/fuse/dev.c
+++ b/fs/fuse/dev.c
@@ -1152,6 +1152,10 @@ static int fuse_try_move_folio(struct fuse_copy_state *cs, struct folio **foliop
 
 	newfolio = page_folio(buf->page);
 
+	/* A lockless page cache read may still be copying from it */
+	if (folio_test_rcu_free(newfolio))
+		goto out_fallback_unlock;
+
 	folio_clear_uptodate(newfolio);
 	folio_clear_mappedtodisk(newfolio);
 
diff --git a/fs/splice.c b/fs/splice.c
index 9d8f63e2fd1ab..0133c2348da03 100644
--- a/fs/splice.c
+++ b/fs/splice.c
@@ -55,56 +55,6 @@ static noinline void pipe_clear_nowait(struct file *file)
 	} while (!try_cmpxchg(&file->f_mode, &fmode, fmode & ~FMODE_NOWAIT));
 }
 
-/*
- * Attempt to steal a page from a pipe buffer. This should perhaps go into
- * a vm helper function, it's already simplified quite a bit by the
- * addition of remove_mapping(). If success is returned, the caller may
- * attempt to reuse this page for another destination.
- */
-static bool page_cache_pipe_buf_try_steal(struct pipe_inode_info *pipe,
-		struct pipe_buffer *buf)
-{
-	struct folio *folio = page_folio(buf->page);
-	struct address_space *mapping;
-
-	folio_lock(folio);
-
-	mapping = folio_mapping(folio);
-	if (mapping) {
-		WARN_ON(!folio_test_uptodate(folio));
-
-		/*
-		 * At least for ext2 with nobh option, we need to wait on
-		 * writeback completing on this folio, since we'll remove it
-		 * from the pagecache.  Otherwise truncate wont wait on the
-		 * folio, allowing the disk blocks to be reused by someone else
-		 * before we actually wrote our data to them. fs corruption
-		 * ensues.
-		 */
-		folio_wait_writeback(folio);
-
-		if (!filemap_release_folio(folio, GFP_KERNEL))
-			goto out_unlock;
-
-		/*
-		 * If we succeeded in removing the mapping, set LRU flag
-		 * and return good.
-		 */
-		if (remove_mapping(mapping, folio)) {
-			buf->flags |= PIPE_BUF_FLAG_LRU;
-			return true;
-		}
-	}
-
-	/*
-	 * Raced with truncate or failed to remove folio from current
-	 * address space, unlock and return failure.
-	 */
-out_unlock:
-	folio_unlock(folio);
-	return false;
-}
-
 static void page_cache_pipe_buf_release(struct pipe_inode_info *pipe,
 					struct pipe_buffer *buf)
 {
@@ -155,7 +105,6 @@ static int page_cache_pipe_buf_confirm(struct pipe_inode_info *pipe,
 const struct pipe_buf_operations page_cache_pipe_buf_ops = {
 	.confirm	= page_cache_pipe_buf_confirm,
 	.release	= page_cache_pipe_buf_release,
-	.try_steal	= page_cache_pipe_buf_try_steal,
 	.get		= generic_pipe_buf_get,
 };
 
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 94f9c3ff54160..3336b9e5ee999 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -228,6 +228,7 @@ enum zone_stat_item {
 	NR_ZSPAGES,		/* allocated in zsmalloc */
 #endif
 	NR_FREE_CMA_PAGES,
+	NR_FREE_RCU_PAGES,	/* freed, waiting for an RCU grace period */
 #ifdef CONFIG_UNACCEPTED_MEMORY
 	NR_UNACCEPTED,
 #endif
diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
index 7a863572adce7..4916b5713a74b 100644
--- a/include/linux/page-flags.h
+++ b/include/linux/page-flags.h
@@ -111,6 +111,9 @@ enum pageflags {
 	PG_swapbacked,		/* Page is backed by RAM/swap */
 	PG_unevictable,		/* Page is "unevictable"  */
 	PG_dropbehind,		/* drop pages on IO completion */
+#ifdef CONFIG_PAGE_RCU_FREE
+	PG_rcu_free,		/* free only after an RCU grace period */
+#endif
 #ifdef CONFIG_MMU
 	PG_mlocked,		/* Page is vma mlocked */
 #endif
@@ -602,6 +605,27 @@ FOLIO_FLAG(dropbehind, FOLIO_HEAD_PAGE)
 	FOLIO_TEST_CLEAR_FLAG(dropbehind, FOLIO_HEAD_PAGE)
 	__FOLIO_SET_FLAG(dropbehind, FOLIO_HEAD_PAGE)
 
+/*
+ * A lockless lookup that sees PG_rcu_free and then still finds the folio
+ * where it looked may read it until rcu_read_unlock().  Set under a
+ * reference.  Cleared by the free, or when the folio's inode is evicted:
+ * no lockless lookup can reach it then.  Not for hugetlb or
+ * free_pages_nolock().
+ */
+#ifdef CONFIG_PAGE_RCU_FREE
+FOLIO_TEST_FLAG(rcu_free, FOLIO_HEAD_PAGE)
+	FOLIO_TEST_SET_FLAG(rcu_free, FOLIO_HEAD_PAGE)
+	FOLIO_CLEAR_FLAG(rcu_free, FOLIO_HEAD_PAGE)
+	__FOLIO_CLEAR_FLAG(rcu_free, FOLIO_HEAD_PAGE)
+#define __PG_RCU_FREE (1UL << PG_rcu_free)
+#else
+FOLIO_TEST_FLAG_FALSE(rcu_free)
+	FOLIO_TEST_SET_FLAG_FALSE(rcu_free)
+	FOLIO_CLEAR_FLAG_NOOP(rcu_free)
+	__FOLIO_CLEAR_FLAG_NOOP(rcu_free)
+#define __PG_RCU_FREE 0
+#endif
+
 #ifdef CONFIG_HIGHMEM
 /*
  * Must use a macro here due to header dependency issues. page_zone() is not
diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 0adfa6605653d..6c7f58803eee4 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -210,6 +210,7 @@ enum mapping_flags {
 	AS_WRITEBACK_MAY_DEADLOCK_ON_RECLAIM = 9,
 	AS_KERNEL_FILE = 10,	/* mapping for a fake kernel file that shouldn't
 				   account usage to user cgroups */
+	AS_RCU_READ = 11,	/* a folio has had PG_rcu_free */
 	/* Bits 16-25 are used for FOLIO_ORDER */
 	AS_FOLIO_ORDER_BITS = 5,
 	AS_FOLIO_ORDER_MIN = 16,
@@ -269,6 +270,17 @@ static inline bool mapping_unevictable(const struct address_space *mapping)
 	return mapping && test_bit(AS_UNEVICTABLE, &mapping->flags);
 }
 
+static inline void mapping_set_rcu_read(struct address_space *mapping)
+{
+	if (!test_bit(AS_RCU_READ, &mapping->flags))
+		set_bit(AS_RCU_READ, &mapping->flags);
+}
+
+static inline bool mapping_rcu_read(const struct address_space *mapping)
+{
+	return test_bit(AS_RCU_READ, &mapping->flags);
+}
+
 static inline void mapping_set_exiting(struct address_space *mapping)
 {
 	set_bit(AS_EXITING, &mapping->flags);
diff --git a/include/linux/swap.h b/include/linux/swap.h
index 5658a1634b85e..67aa5271ae6cb 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -295,7 +295,7 @@ extern unsigned long totalreserve_pages;
 
 /* linux/mm/folio.c */
 void folio_add_lru(struct folio *folio);
-void folio_mark_accessed(struct folio *folio);
+bool folio_mark_accessed(struct folio *folio);
 void lru_add_drain_all(void);
 
 enum lru_cache_drained {
diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h
index 2628ccda076a0..c1a63f070aa71 100644
--- a/include/linux/vm_event_item.h
+++ b/include/linux/vm_event_item.h
@@ -36,6 +36,8 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
 		FOR_ALL_ZONES(ALLOCSTALL)
 		FOR_ALL_ZONES(PGSCAN_SKIP)
 		PGFREE, PGACTIVATE, PGDEACTIVATE, PGLAZYFREE,
+		PGFREE_RCU, PGFREE_RCU_EXPEDITE, PGFREE_RCU_DRAIN,
+		PGFREE_RCU_MARK,
 		PGFAULT, PGMAJFAULT,
 		PGLAZYFREED,
 		PGREUSE,
diff --git a/include/trace/events/mmflags.h b/include/trace/events/mmflags.h
index 935893e5ea53b..47c3411a75e8f 100644
--- a/include/trace/events/mmflags.h
+++ b/include/trace/events/mmflags.h
@@ -117,6 +117,12 @@ TRACE_DEFINE_ENUM(___GFP_LAST_BIT);
 #define IF_HAVE_PG_IDLE(_name)
 #endif
 
+#ifdef CONFIG_PAGE_RCU_FREE
+#define IF_HAVE_PG_RCU_FREE(_name) ,{1UL << PG_##_name, __stringify(_name)}
+#else
+#define IF_HAVE_PG_RCU_FREE(_name)
+#endif
+
 #ifdef CONFIG_ARCH_USES_PG_ARCH_2
 #define IF_HAVE_PG_ARCH_2(_name) ,{1UL << PG_##_name, __stringify(_name)}
 #else
@@ -152,6 +158,7 @@ TRACE_DEFINE_ENUM(___GFP_LAST_BIT);
 	DEF_PAGEFLAG_NAME(swapbacked),					\
 	DEF_PAGEFLAG_NAME(unevictable),					\
 	DEF_PAGEFLAG_NAME(dropbehind)					\
+IF_HAVE_PG_RCU_FREE(rcu_free)						\
 IF_HAVE_PG_MLOCK(mlocked)						\
 IF_HAVE_PG_HWPOISON(hwpoison)						\
 IF_HAVE_PG_IDLE(idle)							\
diff --git a/mm/Kconfig b/mm/Kconfig
index 604c58199acbf..1ad1cb7969e79 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -718,6 +718,21 @@ config PCP_BATCH_SCALE_MAX
 	  latency.  This option sets the upper limit of scale factor to limit
 	  the maximum latency.
 
+config PAGE_RCU_FREE
+	def_bool 64BIT
+	help
+	  Reads from the page cache can copy from folios that are read
+	  again and again without taking a reference on them, so small
+	  reads of one file from many CPUs do not contend on the folio's
+	  reference count.  The cost is that memory freed from such
+	  folios becomes reusable only after an RCU grace period.
+
+	  64-bit only: it takes a page flag, and those are scarce on
+	  32-bit.
+
+	  Writing 0 to lockless_page_cache_read in debugfs turns these
+	  reads off.
+
 config PHYS_ADDR_T_64BIT
 	def_bool 64BIT
 
diff --git a/mm/filemap.c b/mm/filemap.c
index 00fd89cf6f550..7021ed2ab6c9c 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -48,6 +48,7 @@
 #include <linux/rcupdate_wait.h>
 #include <linux/sched/mm.h>
 #include <linux/sysctl.h>
+#include <linux/debugfs.h>
 #include <linux/pgalloc.h>
 
 #include <asm/tlbflush.h>
@@ -138,6 +139,9 @@ static void page_cache_delete(struct address_space *mapping,
 	nr = folio_nr_pages(folio);
 
 	VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
+	/* No lockless reader can be in an evicted inode */
+	if (folio_test_rcu_free(folio) && mapping_exiting(mapping))
+		folio_clear_rcu_free(folio);
 
 	xas_store(&xas, shadow);
 	xas_init_marks(&xas);
@@ -306,6 +310,9 @@ static void page_cache_delete_batch(struct address_space *mapping,
 		}
 
 		WARN_ON_ONCE(!folio_test_locked(folio));
+		/* No lockless reader can be in an evicted inode */
+		if (folio_test_rcu_free(folio) && mapping_exiting(mapping))
+			folio_clear_rcu_free(folio);
 
 		folio->mapping = NULL;
 		/* Leave folio->index set: truncation lookup relies on it */
@@ -2759,6 +2766,101 @@ static inline bool pos_same_folio(loff_t pos1, loff_t pos2, struct folio *folio)
 	return (pos1 >> shift == pos2 >> shift);
 }
 
+#ifdef CONFIG_PAGE_RCU_FREE
+/* Writing 0 to lockless_page_cache_read in debugfs turns the reads off */
+static DEFINE_STATIC_KEY_TRUE(lockless_read_key);
+
+static bool lockless_read_enabled(void)
+{
+	return static_branch_likely(&lockless_read_key);
+}
+
+#ifdef CONFIG_DEBUG_FS
+static int lockless_read_get(void *data, u64 *val)
+{
+	*val = static_key_enabled(&lockless_read_key);
+	return 0;
+}
+
+static int lockless_read_set(void *data, u64 val)
+{
+	if (val > 1)
+		return -EINVAL;
+	if (val)
+		static_branch_enable(&lockless_read_key);
+	else
+		static_branch_disable(&lockless_read_key);
+	return 0;
+}
+DEFINE_DEBUGFS_ATTRIBUTE(lockless_read_fops, lockless_read_get,
+			 lockless_read_set, "%llu\n");
+
+static int __init lockless_read_debugfs_init(void)
+{
+	debugfs_create_file_unsafe("lockless_page_cache_read", 0644, NULL, NULL,
+				   &lockless_read_fops);
+	return 0;
+}
+late_initcall(lockless_read_debugfs_init);
+#endif
+#else
+static bool lockless_read_enabled(void)
+{
+	return false;
+}
+#endif
+
+/*
+ * Reads keep coming back to @folio: free it only after an RCU grace period
+ * from now on.  Not if its next read needs a reference anyway.  Not if it
+ * is unevictable: folio_mark_accessed() calls such a folio hot after at
+ * most one access, so every ramfs folio that is read would get the flag.
+ */
+static void filemap_mark_hot(struct folio *folio)
+{
+	struct address_space *mapping;
+
+	if (!lockless_read_enabled())
+		return;
+	if (folio_test_rcu_free(folio) || !folio_test_uptodate(folio) ||
+	    folio_test_dropbehind(folio) || folio_test_unevictable(folio))
+		return;
+	/* shmem does no readahead: there the bit is PG_reclaim */
+	if (folio_test_readahead(folio) && !folio_test_swapbacked(folio))
+		return;
+	if (folio_test_set_rcu_free(folio))
+		return;
+
+	count_vm_events(PGFREE_RCU_MARK, folio_nr_pages(folio));
+	/* NULL if the folio was truncated meanwhile */
+	mapping = READ_ONCE(folio->mapping);
+	if (mapping)
+		mapping_set_rcu_read(mapping);
+}
+
+/*
+ * filemap_read() is about to copy from @folio at @pos.  @last_pos is where
+ * the previous read of this open file ended, or how far this one has got.
+ *
+ * Reading on in the folio that the last read ended in is not another access
+ * to it: a folio read in small pieces is marked accessed once.
+ *
+ * The folio is hot when folio_mark_accessed() finds it as accessed as the
+ * LRU records, or when this open file goes back to data that it has just
+ * read.
+ */
+void filemap_read_accessed(struct folio *folio, loff_t pos, loff_t last_pos)
+{
+	bool hot;
+
+	if (pos_same_folio(pos, last_pos - 1, folio))
+		hot = pos < last_pos;
+	else
+		hot = folio_mark_accessed(folio);
+	if (hot)
+		filemap_mark_hot(folio);
+}
+
 static void filemap_end_dropbehind_read(struct folio *folio)
 {
 	if (!folio_test_dropbehind(folio))
@@ -2771,6 +2873,216 @@ static void filemap_end_dropbehind_read(struct folio *folio)
 	}
 }
 
+/*
+ * Whether filemap_read_rcu_batch() can copy from @folio, or has to leave it
+ * to the slow path, which holds a reference for readahead, dropbehind and
+ * folio_mark_accessed(), and deals with poisoned pages.  @last_index is the
+ * page the previous copy ended in.
+ */
+static bool filemap_read_rcu_can_copy(struct folio *folio, pgoff_t index,
+				      pgoff_t last_index)
+{
+	if (!folio_test_uptodate(folio) || folio_test_dropbehind(folio))
+		return false;
+	/* shmem does no readahead: there the bit is PG_reclaim */
+	if (folio_test_readahead(folio) && !folio_test_swapbacked(folio))
+		return false;
+	/* memory_failure() waits only for readers that got past this */
+	if (folio_contain_hwpoisoned_page(folio))
+		return false;
+
+	/*
+	 * The slow path does not call folio_mark_accessed() on the folio that
+	 * the previous read ended in, so copying from it skips nothing.  The
+	 * folio's size can change without a reference: compare pages instead.
+	 */
+	if (index == last_index)
+		return true;
+
+	/* Otherwise only if folio_mark_accessed() would leave it as it is */
+	if (lru_gen_enabled()) {
+		unsigned long all = LRU_REFS_FLAGS | BIT(PG_workingset);
+
+		return folio_test_unevictable(folio) ||
+		       (READ_ONCE(folio->flags.f) & all) == all;
+	}
+	return folio_test_referenced(folio) && !folio_test_idle(folio) &&
+	       (folio_test_active(folio) || folio_test_unevictable(folio));
+}
+
+/*
+ * Return the folio at @pos if it is this file's page until rcu_read_unlock(),
+ * and read i_size into @isize.  A folio with PG_rcu_free is freed only after
+ * an RCU grace period.  Without a reference, the folio that xas_load()
+ * returned can be freed and its memory reused as another folio before the
+ * flag is tested, so look at the slot again after the flag.
+ */
+static struct folio *filemap_read_rcu_lookup(struct xa_state *xas,
+		struct inode *inode, loff_t pos, loff_t *isize)
+{
+	struct folio *folio;
+
+	xas_set(xas, pos >> PAGE_SHIFT);
+	folio = xas_load(xas);
+	if (xas_retry(xas, folio) || !folio || xa_is_value(folio))
+		return NULL;
+
+	/* Not folio_test_rcu_free(): it may not be a folio any more */
+	if (!(READ_ONCE(folio->flags.f) & __PG_RCU_FREE))
+		return NULL;
+	/* Read i_size and slot after the flag: it implies uptodate */
+	smp_rmb();
+	*isize = i_size_read(inode);
+	if (pos >= *isize)
+		return NULL;
+	/* i_size_read() is an acquire; a write fills the slot first */
+	if (xas_reload(xas) != folio)
+		return NULL;
+	/* Read the rest of the folio only once the slot holds it */
+	smp_rmb();
+	return folio;
+}
+
+/*
+ * How much of @bytes, starting @offset into page @index, can be copied from
+ * @folio.  A folio is in all the slots of its range or in none: when the slot
+ * of the last page holds it too, so do the slots in between, and the pages
+ * are this folio's.
+ */
+static size_t filemap_read_rcu_size(struct xa_state *xas, struct folio *folio,
+		pgoff_t index, size_t offset, size_t bytes)
+{
+	pgoff_t end = index + DIV_ROUND_UP(offset + bytes, PAGE_SIZE);
+	pgoff_t next;
+
+	if (end - index > 1 && folio_test_large(folio)) {
+		/* Read without a reference: it only picks the page to check */
+		next = folio_next_index(folio);
+		if (next > index && next < end)
+			end = next;
+		xas_set(xas, end - 1);
+		/* A split after this copies PG_rcu_free to the new folios */
+		if (xas_load(xas) == folio)
+			return min(bytes, (end - index) * PAGE_SIZE - offset);
+	}
+	return min(bytes, PAGE_SIZE - offset);
+}
+
+/*
+ * Copy @bytes from @folio, starting @offset into its page at @index.  Page
+ * by page, as copy_page_to_iter() does, without its checks for each call.
+ * Not copy_to_iter(): its usercopy check reads the folio's order, which a
+ * split can change.
+ */
+static size_t filemap_read_rcu_copy(struct folio *folio, pgoff_t index,
+		size_t offset, size_t bytes, struct iov_iter *iter)
+{
+	struct page *page = folio_page(folio, index - folio->index);
+	size_t n = 0;
+
+	while (n < bytes) {
+		size_t len = min(bytes - n, PAGE_SIZE - offset);
+		size_t done;
+		void *kaddr;
+
+		kaddr = kmap_local_page(page);
+		done = _copy_to_iter(kaddr + offset, len, iter);
+		kunmap_local(kaddr);
+		n += done;
+		if (done < len)
+			break;
+		offset = 0;
+		page++;
+	}
+	return n;
+}
+
+/* Bytes one RCU read section copies at most: it holds up grace periods */
+#define FILEMAP_RCU_READ_MAX	SZ_512K
+
+/*
+ * Copy from folios with PG_rcu_free without taking references, at most
+ * FILEMAP_RCU_READ_MAX bytes under one rcu_read_lock().  Return true if that
+ * much was copied and the read is not done.  *last_index is the page the
+ * previous copy ended in, set as the slow path sets last_pos.
+ */
+static bool filemap_read_rcu_batch(struct kiocb *iocb, struct iov_iter *iter,
+		pgoff_t *last_index, size_t *copied)
+{
+	struct address_space *mapping = iocb->ki_filp->f_mapping;
+	struct inode *inode = mapping->host;
+	XA_STATE(xas, &mapping->i_pages, 0);
+	size_t budget = FILEMAP_RCU_READ_MAX;
+	bool more = false;
+
+	/* flush_dcache_folio() is not known to be safe without a reference */
+	if (ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE &&
+	    mapping_writably_mapped(mapping))
+		return false;
+
+	/* A user fault cannot sleep in the RCU read section: it ends the copy */
+	rcu_read_lock();
+	pagefault_disable();
+	while (iov_iter_count(iter)) {
+		loff_t pos = iocb->ki_pos;
+		pgoff_t index = pos >> PAGE_SHIFT;
+		size_t offset = offset_in_page(pos);
+		struct folio *folio;
+		size_t bytes, n;
+		loff_t isize;
+
+		if (!budget) {
+			more = true;
+			break;
+		}
+		folio = filemap_read_rcu_lookup(&xas, inode, pos, &isize);
+		if (!folio)
+			break;
+		if (!filemap_read_rcu_can_copy(folio, index, *last_index))
+			break;
+
+		bytes = min_t(loff_t, isize - pos, iov_iter_count(iter));
+		bytes = filemap_read_rcu_size(&xas, folio, index, offset,
+					      min(bytes, budget));
+		n = filemap_read_rcu_copy(folio, index, offset, bytes, iter);
+		*copied += n;
+		iocb->ki_pos += n;
+		budget -= n;
+		if (n)
+			*last_index = (pos + n - 1) >> PAGE_SHIFT;
+		if (n < bytes)
+			break;
+	}
+	pagefault_enable();
+	rcu_read_unlock();
+
+	return more;
+}
+
+/*
+ * Copy from the page cache without folio references, in batches, and
+ * return how much was copied.  The slow path reads the rest.
+ */
+size_t filemap_read_rcu(struct kiocb *iocb, struct iov_iter *iter,
+		loff_t last_pos)
+{
+	/* The page the previous read ended in; ULONG_MAX if none */
+	pgoff_t last_index = (last_pos - 1) >> PAGE_SHIFT;
+	size_t copied = 0;
+
+	/* Not when switched off, or on an inode that never had a folio flagged */
+	if (!lockless_read_enabled() ||
+	    !mapping_rcu_read(iocb->ki_filp->f_mapping))
+		return 0;
+
+	while (filemap_read_rcu_batch(iocb, iter, &last_index, &copied)) {
+		if (fatal_signal_pending(current))
+			break;
+		cond_resched();
+	}
+	return copied;
+}
+
 /**
  * filemap_read - Read data from the page cache.
  * @iocb: The iocb to read.
@@ -2796,6 +3108,7 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter,
 	bool writably_mapped;
 	loff_t isize, end_offset;
 	loff_t last_pos = ra->prev_pos;
+	size_t fast_read;
 
 	if (unlikely(iocb->ki_pos < 0))
 		return -EINVAL;
@@ -2807,6 +3120,14 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter,
 	iov_iter_truncate(iter, inode->i_sb->s_maxbytes - iocb->ki_pos);
 	folio_batch_init(&fbatch);
 
+	fast_read = filemap_read_rcu(iocb, iter, last_pos);
+	if (fast_read) {
+		already_read += fast_read;
+		last_pos = iocb->ki_pos;
+		if (!iov_iter_count(iter))
+			goto out;
+	}
+
 	do {
 		cond_resched();
 
@@ -2844,14 +3165,6 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter,
 		 */
 		writably_mapped = mapping_writably_mapped(mapping);
 
-		/*
-		 * When a read accesses the same folio several times, only
-		 * mark it as accessed the first time.
-		 */
-		if (!pos_same_folio(iocb->ki_pos, last_pos - 1,
-				    fbatch.folios[0]))
-			folio_mark_accessed(fbatch.folios[0]);
-
 		for (i = 0; i < folio_batch_count(&fbatch); i++) {
 			struct folio *folio = fbatch.folios[i];
 			size_t fsize = folio_size(folio);
@@ -2862,8 +3175,7 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter,
 
 			if (end_offset < folio_pos(folio))
 				break;
-			if (i > 0)
-				folio_mark_accessed(folio);
+			filemap_read_accessed(folio, iocb->ki_pos, last_pos);
 			/*
 			 * If users can be writing to this folio using arbitrary
 			 * virtual addresses, take care of potential aliasing
@@ -2893,8 +3205,11 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter,
 		folio_batch_init(&fbatch);
 	} while (iov_iter_count(iter) && iocb->ki_pos < isize && !error);
 
+out:
 	file_accessed(filp);
-	ra->prev_pos = last_pos;
+	/* Readers sharing the file would bounce the line for nothing */
+	if (ra->prev_pos != last_pos)
+		ra->prev_pos = last_pos;
 	return already_read ? already_read : error;
 }
 EXPORT_SYMBOL_GPL(filemap_read);
diff --git a/mm/folio.c b/mm/folio.c
index 50a6dbe55998e..6144b9cf301b3 100644
--- a/mm/folio.c
+++ b/mm/folio.c
@@ -352,28 +352,34 @@ static void __lru_cache_activate_folio(struct folio *folio)
 
 #ifdef CONFIG_LRU_GEN
 
-static void lru_gen_inc_refs(struct folio *folio)
+/*
+ * Return true if @folio is hot: unevictable, or its flags already held as
+ * many accesses as they can.
+ */
+static bool lru_gen_inc_refs(struct folio *folio)
 {
 	unsigned long new_flags, old_flags = READ_ONCE(folio->flags.f);
 
 	if (folio_test_unevictable(folio))
-		return;
+		return true;
 
 	/* see the comment on LRU_REFS_FLAGS */
 	if (!folio_test_referenced(folio)) {
 		set_mask_bits(&folio->flags.f, LRU_REFS_MASK, BIT(PG_referenced));
-		return;
+		return false;
 	}
 
 	do {
 		if ((old_flags & LRU_REFS_MASK) == LRU_REFS_MASK) {
-			if (!folio_test_workingset(folio))
-				folio_set_workingset(folio);
-			return;
+			if (folio_test_workingset(folio))
+				return true;
+			folio_set_workingset(folio);
+			return false;
 		}
 
 		new_flags = old_flags + BIT(LRU_REFS_PGOFF);
 	} while (!try_cmpxchg(&folio->flags.f, &old_flags, new_flags));
+	return false;
 }
 
 static bool lru_gen_clear_refs(struct folio *folio)
@@ -396,8 +402,9 @@ static bool lru_gen_clear_refs(struct folio *folio)
 
 #else /* !CONFIG_LRU_GEN */
 
-static void lru_gen_inc_refs(struct folio *folio)
+static bool lru_gen_inc_refs(struct folio *folio)
 {
+	return false;
 }
 
 static bool lru_gen_clear_refs(struct folio *folio)
@@ -419,15 +426,20 @@ static bool lru_gen_clear_refs(struct folio *folio)
  *
  * When a newly allocated folio is not yet visible, so safe for non-atomic ops,
  * __folio_set_referenced() may be substituted for folio_mark_accessed().
+ *
+ * Return: true if @folio is hot: it was already as accessed as the LRU
+ * records.  On the classic LRU that is active or unevictable, referenced
+ * and not idle; on the multi-gen LRU, unevictable, or PG_referenced, all
+ * the refs bits and PG_workingset set.  False for a dropbehind folio.
  */
-void folio_mark_accessed(struct folio *folio)
+bool folio_mark_accessed(struct folio *folio)
 {
+	bool hot = false;
+
 	if (folio_test_dropbehind(folio))
-		return;
-	if (lru_gen_enabled()) {
-		lru_gen_inc_refs(folio);
-		return;
-	}
+		return false;
+	if (lru_gen_enabled())
+		return lru_gen_inc_refs(folio);
 
 	if (!folio_test_referenced(folio)) {
 		folio_set_referenced(folio);
@@ -437,6 +449,7 @@ void folio_mark_accessed(struct folio *folio)
 		 * this list is never rotated or maintained, so marking an
 		 * unevictable page accessed has no effect.
 		 */
+		hot = true;
 	} else if (!folio_test_active(folio)) {
 		/*
 		 * If the folio is on the LRU, queue it for activation via
@@ -450,9 +463,14 @@ void folio_mark_accessed(struct folio *folio)
 			__lru_cache_activate_folio(folio);
 		folio_clear_referenced(folio);
 		workingset_activation(folio);
+	} else {
+		hot = true;
 	}
-	if (folio_test_idle(folio))
+	if (folio_test_idle(folio)) {
 		folio_clear_idle(folio);
+		hot = false;
+	}
+	return hot;
 }
 EXPORT_SYMBOL(folio_mark_accessed);
 
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 1e5d68acf62a5..25082949a4cae 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -3743,6 +3743,7 @@ static void __split_folio_to_order(struct folio *folio, int old_order,
 #endif
 				 (1L << PG_dirty) |
 				 (1L << PG_dropbehind) |
+				 __PG_RCU_FREE |
 				 LRU_GEN_MASK | LRU_REFS_MASK));
 
 		new_folio->mapping = folio->mapping;
diff --git a/mm/internal.h b/mm/internal.h
index 38b1165212c94..ae9a4e511c827 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -1248,6 +1248,9 @@ struct migration_target_control {
  */
 size_t splice_folio_into_pipe(struct pipe_inode_info *pipe,
 			      struct folio *folio, loff_t fpos, size_t size);
+void filemap_read_accessed(struct folio *folio, loff_t pos, loff_t last_pos);
+size_t filemap_read_rcu(struct kiocb *iocb, struct iov_iter *iter,
+			loff_t last_pos);
 
 static inline bool vma_is_single_threaded_private(struct vm_area_struct *vma)
 {
diff --git a/mm/memory-failure.c b/mm/memory-failure.c
index a8b03e2920ba8..dc66cac05b1f5 100644
--- a/mm/memory-failure.c
+++ b/mm/memory-failure.c
@@ -211,6 +211,10 @@ static bool page_handle_poison(struct page *page, bool hugepage_or_freepage, boo
 			return false;
 	}
 
+	/* The free of a hwpoisoned page is not deferred: wait for readers */
+	if (release && folio_test_rcu_free(page_folio(page)))
+		synchronize_rcu();
+
 	SetPageHWPoison(page);
 	if (release)
 		put_page(page);
@@ -1561,6 +1565,21 @@ static int get_hwpoison_page(struct page *p, unsigned long flags)
 	return ret;
 }
 
+/* A page waiting for a grace period is not in the buddy yet: free it */
+static void mf_free_rcu_pages(struct page *p)
+{
+	if (!IS_ENABLED(CONFIG_PAGE_RCU_FREE))
+		return;
+	if (page_count(p))
+		return;
+	if (!zone_page_state_snapshot(page_zone(p), NR_FREE_RCU_PAGES))
+		return;
+	if (is_free_buddy_page(p))
+		return;
+
+	rcu_free_sync();
+}
+
 /*
  * The caller must guarantee the folio isn't large folio, except hugetlb.
  * try_to_unmap() can't handle it.
@@ -2436,6 +2455,7 @@ int memory_failure(unsigned long pfn, int flags)
 	if (res != -ENOENT)
 		goto unlock_mutex;
 
+	mf_free_rcu_pages(p);
 	if (TestSetPageHWPoison(p)) {
 		res = -EHWPOISON;
 		if (flags & MF_ACTION_REQUIRED)
@@ -2608,6 +2628,13 @@ int memory_failure(unsigned long pfn, int flags)
 
 identify_page_state:
 	res = identify_page_state(pfn, p, page_flags);
+	/*
+	 * The caller may unmap the page once this returns 0.  Lockless
+	 * readers that started before the page got PG_hwpoison, or before a
+	 * split left it in a folio of its own, may still be copying it.
+	 */
+	if (!res && folio_test_rcu_free(folio))
+		synchronize_rcu();
 	mutex_unlock(&mf_mutex);
 	return res;
 unlock_page:
@@ -2999,6 +3026,7 @@ int soft_offline_page(unsigned long pfn, int flags)
 
 retry:
 	get_online_mems();
+	mf_free_rcu_pages(page);
 	ret = get_hwpoison_page(page, flags | MF_SOFT_OFFLINE);
 	put_online_mems();
 
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 226ab9cb078ad..a8f0c95627ee1 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -1967,6 +1967,7 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,
 		.nid = NUMA_NO_NODE,
 	};
 	unsigned long flags;
+	bool migrated;
 	char *reason;
 	int ret;
 	unsigned long normal_pages = 0;
@@ -2050,6 +2051,7 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,
 
 	do {
 		pfn = start_pfn;
+		migrated = false;
 		do {
 			/*
 			 * Historically we always checked for any signal and
@@ -2071,6 +2073,7 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,
 				 * out
 				 */
 				do_migrate_range(pfn, end_pfn);
+				migrated = true;
 			}
 		} while (!ret);
 
@@ -2092,6 +2095,10 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,
 
 		ret = test_pages_isolated(start_pfn, end_pfn,
 					  PB_ISOLATE_MODE_MEM_OFFLINE);
+		/* What this pass migrated can wait for a grace period */
+		if (ret && migrated &&
+		    zone_page_state_snapshot(zone, NR_FREE_RCU_PAGES))
+			rcu_free_sync();
 
 	} while (ret);
 
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 12fac9084c483..e63628e64cbba 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1313,6 +1313,199 @@ static inline void pgalloc_tag_sub_pages(struct alloc_tag *tag, unsigned int nr)
 
 #endif /* CONFIG_MEM_ALLOC_PROFILING */
 
+#define RCU_FREE_EXPEDITE	(SZ_4M >> PAGE_SHIFT)
+
+/*
+ * Each CPU has at most one grace period in flight (@queued), covering
+ * @batch.  Folios freed meanwhile wait on @pending for the next one.
+ */
+struct rcu_free_pcpu {
+	spinlock_t lock;
+	bool queued;
+	struct list_head pending;
+	struct list_head batch;
+	unsigned long nr_pending;
+	struct rcu_work rwork;
+	struct work_struct expedite;
+};
+
+static DEFINE_PER_CPU(struct rcu_free_pcpu, rcu_free_pcpu);
+
+static struct workqueue_struct *rcu_free_wq __ro_after_init;
+
+static void rcu_free_start(struct rcu_free_pcpu *r)
+{
+	lockdep_assert_held(&r->lock);
+
+	if (r->queued || list_empty(&r->pending))
+		return;
+
+	list_splice_init(&r->pending, &r->batch);
+	r->nr_pending = 0;
+	r->queued = true;
+	WARN_ON_ONCE(!queue_rcu_work(rcu_free_wq, &r->rwork));
+}
+
+static void rcu_free_take(struct rcu_free_pcpu *r, struct list_head *list)
+{
+	lockdep_assert_held(&r->lock);
+
+	list_splice_tail_init(&r->batch, list);
+	list_splice_tail_init(&r->pending, list);
+	r->nr_pending = 0;
+}
+
+static unsigned long rcu_free_pages(struct list_head *list)
+{
+	struct folio_batch fbatch;
+	struct folio *folio, *next;
+	unsigned long nr = 0;
+
+	folio_batch_init(&fbatch);
+	list_for_each_entry_safe(folio, next, list, lru) {
+		nr += folio_nr_pages(folio);
+		mod_zone_page_state(folio_zone(folio), NR_FREE_RCU_PAGES,
+				    -folio_nr_pages(folio));
+		if (!folio_batch_add(&fbatch, folio)) {
+			free_unref_folios(&fbatch);
+			cond_resched_tasks_rcu_qs();
+		}
+	}
+	if (folio_batch_count(&fbatch))
+		free_unref_folios(&fbatch);
+	return nr;
+}
+
+/* Free what rcu_free_take() took: the grace period starts after that */
+static unsigned long rcu_free_flush(struct list_head *list)
+{
+	if (list_empty(list))
+		return 0;
+
+	synchronize_rcu_expedited();
+	return rcu_free_pages(list);
+}
+
+static void rcu_free_workfn(struct work_struct *work)
+{
+	struct rcu_free_pcpu *r = container_of(to_rcu_work(work),
+					       struct rcu_free_pcpu, rwork);
+	LIST_HEAD(done);
+
+	spin_lock_irq(&r->lock);
+	list_splice_init(&r->batch, &done);
+	r->queued = false;
+	rcu_free_start(r);
+	spin_unlock_irq(&r->lock);
+
+	rcu_free_pages(&done);
+}
+
+static void rcu_free_expedite_workfn(struct work_struct *work)
+{
+	struct rcu_free_pcpu *r = container_of(work, struct rcu_free_pcpu,
+					       expedite);
+	LIST_HEAD(list);
+
+	spin_lock_irq(&r->lock);
+	rcu_free_take(r, &list);
+	spin_unlock_irq(&r->lock);
+
+	if (rcu_free_flush(&list))
+		count_vm_event(PGFREE_RCU_EXPEDITE);
+}
+
+static void rcu_free_defer(struct folio *folio)
+{
+	struct rcu_free_pcpu *r = raw_cpu_ptr(&rcu_free_pcpu);
+	struct zone *zone = folio_zone(folio);
+	unsigned long free, waiting;
+	unsigned long flags;
+
+	spin_lock_irqsave(&r->lock, flags);
+	__mod_zone_page_state(zone, NR_FREE_RCU_PAGES, folio_nr_pages(folio));
+	list_add_tail(&folio->lru, &r->pending);
+	r->nr_pending += folio_nr_pages(folio);
+	rcu_free_start(r);
+	spin_unlock_irqrestore(&r->lock, flags);
+
+	/* Unlocked: a stale count delays the expedite by one free */
+	if (data_race(r->nr_pending) <= RCU_FREE_EXPEDITE ||
+	    work_pending(&r->expedite))
+		return;
+
+	/*
+	 * An expedited grace period interrupts every CPU: only while the
+	 * zone is short of memory, or while more than an eighth of what it
+	 * has free waits here.
+	 */
+	free = zone_page_state(zone, NR_FREE_PAGES);
+	waiting = zone_page_state(zone, NR_FREE_RCU_PAGES);
+	if (free < high_wmark_pages(zone) || waiting > free / 8)
+		queue_work(rcu_free_wq, &r->expedite);
+}
+
+/*
+ * Free what every CPU's lists hold.  A concurrent drain holds the lock
+ * until it has freed what it took.
+ */
+static void rcu_free_drain(void)
+{
+	static DEFINE_MUTEX(lock);
+	LIST_HEAD(list);
+	int cpu;
+
+	mutex_lock(&lock);
+	for_each_possible_cpu(cpu) {
+		struct rcu_free_pcpu *r = per_cpu_ptr(&rcu_free_pcpu, cpu);
+
+		spin_lock_irq(&r->lock);
+		rcu_free_take(r, &list);
+		spin_unlock_irq(&r->lock);
+	}
+	if (rcu_free_flush(&list))
+		count_vm_event(PGFREE_RCU_DRAIN);
+	mutex_unlock(&lock);
+}
+
+/* Free every waiting folio, including those a work took off the lists */
+void rcu_free_sync(void)
+{
+	int cpu;
+
+	if (!IS_ENABLED(CONFIG_PAGE_RCU_FREE))
+		return;
+
+	rcu_free_drain();
+	for_each_possible_cpu(cpu) {
+		struct rcu_free_pcpu *r = per_cpu_ptr(&rcu_free_pcpu, cpu);
+
+		flush_work(&r->rwork.work);
+		flush_work(&r->expedite);
+	}
+}
+
+static void __init rcu_free_init(void)
+{
+	int cpu;
+
+	if (!IS_ENABLED(CONFIG_PAGE_RCU_FREE))
+		return;
+
+	rcu_free_wq = alloc_workqueue("rcu_free",
+				      WQ_MEM_RECLAIM | WQ_PERCPU, 0);
+
+	for_each_possible_cpu(cpu) {
+		struct rcu_free_pcpu *r = per_cpu_ptr(&rcu_free_pcpu, cpu);
+
+		spin_lock_init(&r->lock);
+		INIT_LIST_HEAD(&r->pending);
+		INIT_LIST_HEAD(&r->batch);
+		INIT_RCU_WORK(&r->rwork, rcu_free_workfn);
+		INIT_WORK(&r->expedite, rcu_free_expedite_workfn);
+	}
+}
+
 static __always_inline bool __free_pages_prepare(struct page *page,
 		unsigned int order, fpi_t fpi_flags)
 {
@@ -1327,6 +1520,19 @@ static __always_inline bool __free_pages_prepare(struct page *page,
 
 	VM_BUG_ON_PAGE(PageTail(page), page);
 
+	/*
+	 * Not for a hwpoisoned order-0 page: it never reaches the allocator,
+	 * and page_handle_poison() takes a reference right after this free.
+	 */
+	if (unlikely(folio_test_rcu_free(folio)) &&
+	    !(PageHWPoison(page) && !order)) {
+		VM_WARN_ON_ONCE(fpi_flags & FPI_NOLOCK);
+		__folio_clear_rcu_free(folio);
+		count_vm_events(PGFREE_RCU, 1 << order);
+		rcu_free_defer(folio);
+		return false;
+	}
+
 	trace_mm_page_free(page, order);
 	kmsan_free_page(page, order);
 
@@ -4626,6 +4832,52 @@ bool gfp_pfmemalloc_allowed(gfp_t gfp_mask)
 	return !!__gfp_pfmemalloc_flags(gfp_mask);
 }
 
+/*
+ * Reclaim is about to give up: if the pages waiting for a grace period
+ * would bring an eligible zone above its min watermark, free them and
+ * retry.  Not for a dying task, which frees memory by exiting.
+ */
+static bool rcu_free_wait(gfp_t gfp_mask, unsigned int order,
+			  struct alloc_context *ac, int alloc_flags,
+			  int *rcu_free_waits)
+{
+	struct zone *zone;
+	struct zoneref *z;
+
+	if (!IS_ENABLED(CONFIG_PAGE_RCU_FREE))
+		return false;
+	if (tsk_is_oom_victim(current) || fatal_signal_pending(current))
+		return false;
+	if (*rcu_free_waits >= MAX_RECLAIM_RETRIES ||
+	    order > PAGE_ALLOC_COSTLY_ORDER)
+		return false;
+
+	for_each_zone_zonelist_nodemask(zone, z, ac->zonelist,
+				ac->highest_zoneidx, ac->nodemask) {
+		unsigned long waiting, available;
+
+		if (cpusets_enabled() && (alloc_flags & ALLOC_CPUSET) &&
+		    !__cpuset_zone_allowed(zone, gfp_mask))
+			continue;
+
+		waiting = zone_page_state_snapshot(zone, NR_FREE_RCU_PAGES);
+		if (!waiting)
+			continue;
+
+		available = waiting +
+			    zone_page_state_snapshot(zone, NR_FREE_PAGES);
+		if (__zone_watermark_ok(zone, order, min_wmark_pages(zone),
+					ac->highest_zoneidx, alloc_flags,
+					available)) {
+			(*rcu_free_waits)++;
+			rcu_free_drain();
+			return true;
+		}
+	}
+
+	return false;
+}
+
 /*
  * Checks whether it makes sense to retry the reclaim to make a forward progress
  * for the given allocation request.
@@ -4639,7 +4891,8 @@ bool gfp_pfmemalloc_allowed(gfp_t gfp_mask)
 static inline bool
 should_reclaim_retry(gfp_t gfp_mask, unsigned order,
 		     struct alloc_context *ac, int alloc_flags,
-		     bool did_some_progress, int *no_progress_loops)
+		     bool did_some_progress, int *no_progress_loops,
+		     int *rcu_free_waits)
 {
 	struct zone *zone;
 	struct zoneref *z;
@@ -4650,10 +4903,12 @@ should_reclaim_retry(gfp_t gfp_mask, unsigned order,
 	 * their order will become available due to high fragmentation so
 	 * always increment the no progress counter for them
 	 */
-	if (did_some_progress && order <= PAGE_ALLOC_COSTLY_ORDER)
+	if (did_some_progress && order <= PAGE_ALLOC_COSTLY_ORDER) {
 		*no_progress_loops = 0;
-	else
+		*rcu_free_waits = 0;
+	} else {
 		(*no_progress_loops)++;
+	}
 
 	if (*no_progress_loops > MAX_RECLAIM_RETRIES)
 		goto out;
@@ -4706,6 +4961,10 @@ should_reclaim_retry(gfp_t gfp_mask, unsigned order,
 	else
 		cond_resched();
 out:
+	if (!ret && rcu_free_wait(gfp_mask, order, ac, alloc_flags,
+				  rcu_free_waits))
+		return true;
+
 	/* Before OOM, exhaust highatomic_reserve */
 	if (!ret)
 		return unreserve_highatomic_pageblock(ac, true);
@@ -4795,6 +5054,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	enum compact_result compact_result;
 	int compaction_retries;
 	int no_progress_loops;
+	int rcu_free_waits;
 	unsigned int cpuset_mems_cookie;
 	unsigned int zonelist_iter_cookie;
 	int reserve_flags;
@@ -4819,6 +5079,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 restart:
 	compaction_retries = 0;
 	no_progress_loops = 0;
+	rcu_free_waits = 0;
 	compact_result = COMPACT_SKIPPED;
 	compact_priority = DEF_COMPACT_PRIORITY;
 	cpuset_mems_cookie = read_mems_allowed_begin();
@@ -4997,7 +5258,8 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 		goto restart;
 
 	if (should_reclaim_retry(gfp_mask, order, ac, alloc_flags,
-				 did_some_progress > 0, &no_progress_loops))
+				 did_some_progress > 0, &no_progress_loops,
+				 &rcu_free_waits))
 		goto retry;
 
 	/*
@@ -5040,6 +5302,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	/* Retry as long as the OOM killer is making progress */
 	if (did_some_progress) {
 		no_progress_loops = 0;
+		rcu_free_waits = 0;
 		goto retry;
 	}
 
@@ -6401,6 +6664,8 @@ void __init setup_per_cpu_pageset(void)
 	for_each_online_pgdat(pgdat)
 		pgdat->per_cpu_nodestats =
 			alloc_percpu(struct per_cpu_nodestat);
+
+	rcu_free_init();
 }
 
 __meminit void zone_pcp_init(struct zone *zone)
@@ -7384,7 +7649,16 @@ int alloc_contig_frozen_range_noprof(unsigned long start, unsigned long end,
 	outer_start = find_large_buddy(start);
 
 	/* Make sure the range is really isolated. */
-	if (test_pages_isolated(outer_start, end, mode)) {
+	ret = test_pages_isolated(outer_start, end, mode);
+	if (ret && zone_page_state_snapshot(cc.zone, NR_FREE_RCU_PAGES)) {
+		rcu_free_sync();
+
+		/* A freed folio can merge with a free buddy below start */
+		outer_start = find_large_buddy(start);
+		ret = test_pages_isolated(outer_start, end, mode);
+	}
+
+	if (ret) {
 		ret = -EBUSY;
 		goto done;
 	}
diff --git a/mm/page_alloc.h b/mm/page_alloc.h
index b9259deddb59d..f9e200581e2f5 100644
--- a/mm/page_alloc.h
+++ b/mm/page_alloc.h
@@ -235,6 +235,7 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order, int nid,
 	alloc_hooks(__alloc_frozen_pages_noprof(__VA_ARGS__))
 void free_frozen_pages(struct page *page, unsigned int order);
 void free_unref_folios(struct folio_batch *fbatch);
+void rcu_free_sync(void);
 
 #ifdef CONFIG_NUMA
 struct page *alloc_frozen_pages_noprof(gfp_t, unsigned int order);
diff --git a/mm/shmem.c b/mm/shmem.c
index 848316eaa7f4f..e6c18e745f816 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -3309,10 +3309,18 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
 	struct file *file = iocb->ki_filp;
 	struct inode *inode = file_inode(file);
 	struct address_space *mapping = inode->i_mapping;
+	loff_t last_pos = file->f_ra.prev_pos;
 	pgoff_t index;
 	unsigned long offset;
 	int error = 0;
-	ssize_t retval = 0;
+	ssize_t retval;
+
+	retval = filemap_read_rcu(iocb, to, last_pos);
+	if (retval) {
+		last_pos = iocb->ki_pos;
+		if (!iov_iter_count(to))
+			goto out;
+	}
 
 	for (;;) {
 		struct folio *folio = NULL;
@@ -3378,11 +3386,7 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
 					flush_dcache_page(page);
 			}
 
-			/*
-			 * Mark the folio accessed if we read the beginning.
-			 */
-			if (!offset)
-				folio_mark_accessed(folio);
+			filemap_read_accessed(folio, iocb->ki_pos, last_pos);
 			/*
 			 * Ok, we have the page, and it's up-to-date, so
 			 * now we can copy it to user space...
@@ -3410,6 +3414,7 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
 
 		retval += ret;
 		iocb->ki_pos += ret;
+		last_pos = iocb->ki_pos;
 
 		if (!iov_iter_count(to))
 			break;
@@ -3419,8 +3424,11 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
 		}
 		cond_resched();
 	}
-
+out:
 	file_accessed(file);
+	/* Readers sharing the file would bounce the line for nothing */
+	if (file->f_ra.prev_pos != last_pos)
+		file->f_ra.prev_pos = last_pos;
 	return retval ? retval : error;
 }
 
diff --git a/mm/show_mem.c b/mm/show_mem.c
index b938cbcd774a1..0d3bddbda108d 100644
--- a/mm/show_mem.c
+++ b/mm/show_mem.c
@@ -204,7 +204,7 @@ static void show_free_areas(unsigned int filter, const nodemask_t *nodemask,
 		" mapped:%lu shmem:%lu pagetables:%lu\n"
 		" sec_pagetables:%lu bounce:%lu\n"
 		" kernel_misc_reclaimable:%lu\n"
-		" free:%lu free_pcp:%lu free_cma:%lu\n",
+		" free:%lu free_pcp:%lu free_cma:%lu free_rcu:%lu\n",
 		global_node_page_state(NR_ACTIVE_ANON),
 		global_node_page_state(NR_INACTIVE_ANON),
 		global_node_page_state(NR_ISOLATED_ANON),
@@ -224,7 +224,8 @@ static void show_free_areas(unsigned int filter, const nodemask_t *nodemask,
 		global_node_page_state(NR_KERNEL_MISC_RECLAIMABLE),
 		global_zone_page_state(NR_FREE_PAGES),
 		free_pcp,
-		global_zone_page_state(NR_FREE_CMA_PAGES));
+		global_zone_page_state(NR_FREE_CMA_PAGES),
+		global_zone_page_state(NR_FREE_RCU_PAGES));
 
 	for_each_online_pgdat(pgdat) {
 		if (show_mem_node_skip(filter, pgdat->node_id, nodemask))
@@ -325,6 +326,7 @@ static void show_free_areas(unsigned int filter, const nodemask_t *nodemask,
 			" free_pcp:%lukB"
 			" local_pcp:%lukB"
 			" free_cma:%lukB"
+			" free_rcu:%lukB"
 			"\n",
 			zone->name,
 			K(zone_page_state(zone, NR_FREE_PAGES)),
@@ -351,7 +353,8 @@ static void show_free_areas(unsigned int filter, const nodemask_t *nodemask,
 			0UL,
 			K(free_pcp),
 			K((unsigned long)this_cpu_read(zone->per_cpu_pageset->count)),
-			K(zone_page_state(zone, NR_FREE_CMA_PAGES)));
+			K(zone_page_state(zone, NR_FREE_CMA_PAGES)),
+			K(zone_page_state(zone, NR_FREE_RCU_PAGES)));
 		printk("lowmem_reserve[]:");
 		for (i = 0; i < MAX_NR_ZONES; i++)
 			printk(KERN_CONT " %ld", zone->lowmem_reserve[i]);
diff --git a/mm/vmstat.c b/mm/vmstat.c
index cb57714539fb5..c27caad017e35 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -1212,6 +1212,7 @@ const char * const vmstat_text[] = {
 	[I(NR_ZSPAGES)]				= "nr_zspages",
 #endif
 	[I(NR_FREE_CMA_PAGES)]			= "nr_free_cma",
+	[I(NR_FREE_RCU_PAGES)]			= "nr_free_rcu",
 #ifdef CONFIG_UNACCEPTED_MEMORY
 	[I(NR_UNACCEPTED)]			= "nr_unaccepted",
 #endif
@@ -1343,6 +1344,10 @@ const char * const vmstat_text[] = {
 	[I(PGACTIVATE)]				= "pgactivate",
 	[I(PGDEACTIVATE)]			= "pgdeactivate",
 	[I(PGLAZYFREE)]				= "pglazyfree",
+	[I(PGFREE_RCU)]				= "pgfree_rcu",
+	[I(PGFREE_RCU_EXPEDITE)]		= "pgfree_rcu_expedite",
+	[I(PGFREE_RCU_DRAIN)]			= "pgfree_rcu_drain",
+	[I(PGFREE_RCU_MARK)]			= "pgfree_rcu_mark",
 
 	[I(PGFAULT)]				= "pgfault",
 	[I(PGMAJFAULT)]				= "pgmajfault",
@@ -2022,6 +2027,7 @@ static int vmstat_refresh(const struct ctl_table *table, int write,
 		switch (i) {
 		case NR_ZONE_WRITE_PENDING:
 		case NR_FREE_CMA_PAGES:
+		case NR_FREE_RCU_PAGES:
 			continue;
 		}
 		val = atomic_long_read(&vm_zone_stat[i]);

							input: 20822
							output: 32
							thoughts: 1231

**Analyzing the Lockless Page Cache Reads and RCU Page Freeing Patch**

Okay, let's dive into this patch. My immediate thought is, wow, this is a significant undertaking, essentially introducing lockless page cache reads and RCU-based page freeing. This touches *core* memory management and the VFS layer, which means it will affect all standard filesystems and shmem! We are talking about major changes to `filemap_read`, `__free_pages_prepare`, and reclaim retry logic, page splitting... a massive undertaking.

I see it's enabling `PAGE_RCU_FREE` by default for 64-bit architectures (which is probably most of our targets). That means this is going to be exercised... *a lot*. Standard read() syscalls on regular files (ext4, tmpfs/shmem, etc.) will trigger this new path. Good to know it's *reachable* -- this is clearly something we need to pay very close attention to.

Specifically, I need to understand the implications of:
* `fuse_try_move_folio`: introduces rcu checks
* `filemap_read_rcu`: fast path that will be used by all normal read operations
* `__free_pages_prepare`: introduces rcu handling for page freeing
* `filemap_mark_hot`: sets rcu_free and as_rcu_read flags.
* The `rcu_free_*` functions in `page_alloc.c`--these seem to be the new infrastructure.

My initial assessment? Absolutely, `WorthFuzzing = true`. This is a huge change and has the potential for interesting interactions. Now, the next step is to nail down the precise scope. What are the key non-ubiquitous entrypoints or functions to focus on? What configuration options do we need to ensure this code path is active during testing? Let's get some greps and checks going to get this clarified.

{
  "Expression": "config PAGE_RCU_FREE",
  "PathPrefixes": [
    "mm/Kconfig"
  ]
}

{
  "Output": "mm/Kconfig=710=config PCP_BATCH_SCALE_MAX\n--\nmm/Kconfig-720-\nmm/Kconfig:721:config PAGE_RCU_FREE\nmm/Kconfig-722-\tdef_bool 64BIT\n"
}

							input: 22583
							output: 36
							thoughts: 1222

**Analyzing the Impact of CONFIG_PAGE_RCU_FREE**

Okay, let's dive into this `CONFIG_PAGE_RCU_FREE` thing. First, I need to confirm that this is a 64-bit feature, which checks out given the `def_bool 64BIT` dependency. I'm also looking for any other configuration options that are involved in this change. I see `CONFIG_LRU_GEN` is touched, but that's in `folio.c`, and the code supports both with and without `LRU_GEN`. `CONFIG_DEBUG_FS` is related for a debugfs attribute, but that's not the core.

Now, let's check the function symbols that are introduced or modified with this feature. Checking `filemap.c`, I see a couple of functions: `filemap_read_rcu`, `filemap_read_rcu_batch`, `filemap_read_accessed`, `filemap_mark_hot`, and `filemap_read`. The key one here is `filemap_read_rcu` -- that's the entry point! But I need to be sure; is `filemap_read_rcu` actually a symbol in vmlinux? Looking into `mm/filemap.c`, and seeing it's not declared static, and looking at the header, `mm/internal.h`, confirms that it's exported as a global symbol. Great! And, `filemap_read_accessed` is also not static, and it's also declared in `mm/internal.h`.

What about `folio.c`? `folio_mark_accessed` is ubiquitous, called everywhere. I need to avoid that per the guidelines.

Next, I look at `mm/page_alloc.c` and notice several RCU-related functions: `rcu_free_defer`, `rcu_free_start`, `rcu_free_take`, `rcu_free_pages`, `rcu_free_flush`, `rcu_free_workfn`, `rcu_free_expedite_workfn`, `rcu_free_drain`, `rcu_free_sync`, `rcu_free_wait`, and `rcu_free_init`. Now, most of these are static, but wait... workqueue callbacks! Functions such as `rcu_free_workfn`, and `rcu_free_expedite_workfn` definitely have symbols in vmlinux because their addresses are passed to workqueues, so they cannot be inlined! `rcu_free_sync` is also non-static. I need to note that.

Let's see: `filemap_read_rcu` is called inside `shmem_file_read_iter` and `filemap_read`. I also need to note `fs/fuse/dev.c`, which contains `fuse_try_move_folio` with a check for `folio_test_rcu_free`. And let's not forget about memory failures and `soft_offline_page` which call `mf_free_rcu_pages`.

I should probably do a quick read on the relevant sections of `mm/page_alloc.c` around these `rcu_free_*` functions too, just to be thorough.

{
  "File": "mm/page_alloc.c",
  "FirstLine": 1340,
  "LineCount": 50
}

{
  "Contents": "1340:\t\tif (r-\u003equeued || list_empty(\u0026r-\u003epending))\n1341:\t\t\treturn;\n1342:\t\n1343:\t\tlist_splice_init(\u0026r-\u003epending, \u0026r-\u003ebatch);\n1344:\t\tr-\u003enr_pending = 0;\n1345:\t\tr-\u003equeued = true;\n1346:\t\tWARN_ON_ONCE(!queue_rcu_work(rcu_free_wq, \u0026r-\u003erwork));\n1347:\t}\n1348:\t\n1349:\tstatic void rcu_free_take(struct rcu_free_pcpu *r, struct list_head *list)\n1350:\t{\n1351:\t\tlockdep_assert_held(\u0026r-\u003elock);\n1352:\t\n1353:\t\tlist_splice_tail_init(\u0026r-\u003ebatch, list);\n1354:\t\tlist_splice_tail_init(\u0026r-\u003epending, list);\n1355:\t\tr-\u003enr_pending = 0;\n1356:\t}\n1357:\t\n1358:\tstatic unsigned long rcu_free_pages(struct list_head *list)\n1359:\t{\n1360:\t\tstruct folio_batch fbatch;\n1361:\t\tstruct folio *folio, *next;\n1362:\t\tunsigned long nr = 0;\n1363:\t\n1364:\t\tfolio_batch_init(\u0026fbatch);\n1365:\t\tlist_for_each_entry_safe(folio, next, list, lru) {\n1366:\t\t\tnr += folio_nr_pages(folio);\n1367:\t\t\tmod_zone_page_state(folio_zone(folio), NR_FREE_RCU_PAGES,\n1368:\t\t\t\t\t    -folio_nr_pages(folio));\n1369:\t\t\tif (!folio_batch_add(\u0026fbatch, folio)) {\n1370:\t\t\t\tfree_unref_folios(\u0026fbatch);\n1371:\t\t\t\tcond_resched_tasks_rcu_qs();\n1372:\t\t\t}\n1373:\t\t}\n1374:\t\tif (folio_batch_count(\u0026fbatch))\n1375:\t\t\tfree_unref_folios(\u0026fbatch);\n1376:\t\treturn nr;\n1377:\t}\n1378:\t\n1379:\t/* Free what rcu_free_take() took: the grace period starts after that */\n1380:\tstatic unsigned long rcu_free_flush(struct list_head *list)\n1381:\t{\n1382:\t\tif (list_empty(list))\n1383:\t\t\treturn 0;\n1384:\t\n1385:\t\tsynchronize_rcu_expedited();\n1386:\t\treturn rcu_free_pages(list);\n1387:\t}\n1388:\t\n1389:\tstatic void rcu_free_workfn(struct work_struct *work)\n"
}

							input: 25337
							output: 36
							thoughts: 28

Here's my take on those lines in `mm/page_alloc.c`:

**Diving Deep into `mm/page_alloc.c` – Lines 1389-1450**

Okay, so I need to understand what's happening around page allocation. My initial instinct is to zero in on a specific block of code, let's say, lines 1389-1450 in `mm/page_alloc.c`. This usually gives me a good starting point for understanding a specific aspect of the allocation process. What I'm really looking for in this range is probably related to how the kernel handles either the initial request, or possibly the release of pages from the system, or something in between related to tracking.

Now, I'm thinking about the context of this range of code. It's likely a critical section and I anticipate it'll deal with things like buddy system interactions, zone management, NUMA considerations (if relevant), and perhaps even some specific slab allocator interplay, depending on the kernel version. I'm keen to see if this region touches on any particular optimization strategies as well, like using hugepages or direct memory access.

I'm ready to dig in and understand the detailed logic within these lines. My brain's prepped to trace the control flow, examine the data structures (page structures, free lists, zones, etc.), and analyze any potentially complex locking schemes that might be employed to prevent race conditions. The overall goal is to form a precise mental map of what this code section actually *does* in the grand scheme of page management.

{
  "File": "mm/page_alloc.c",
  "FirstLine": 1389,
  "LineCount": 60
}

{
  "Contents": "1389:\tstatic void rcu_free_workfn(struct work_struct *work)\n1390:\t{\n1391:\t\tstruct rcu_free_pcpu *r = container_of(to_rcu_work(work),\n1392:\t\t\t\t\t\t       struct rcu_free_pcpu, rwork);\n1393:\t\tLIST_HEAD(done);\n1394:\t\n1395:\t\tspin_lock_irq(\u0026r-\u003elock);\n1396:\t\tlist_splice_init(\u0026r-\u003ebatch, \u0026done);\n1397:\t\tr-\u003equeued = false;\n1398:\t\trcu_free_start(r);\n1399:\t\tspin_unlock_irq(\u0026r-\u003elock);\n1400:\t\n1401:\t\trcu_free_pages(\u0026done);\n1402:\t}\n1403:\t\n1404:\tstatic void rcu_free_expedite_workfn(struct work_struct *work)\n1405:\t{\n1406:\t\tstruct rcu_free_pcpu *r = container_of(work, struct rcu_free_pcpu,\n1407:\t\t\t\t\t\t       expedite);\n1408:\t\tLIST_HEAD(list);\n1409:\t\n1410:\t\tspin_lock_irq(\u0026r-\u003elock);\n1411:\t\trcu_free_take(r, \u0026list);\n1412:\t\tspin_unlock_irq(\u0026r-\u003elock);\n1413:\t\n1414:\t\tif (rcu_free_flush(\u0026list))\n1415:\t\t\tcount_vm_event(PGFREE_RCU_EXPEDITE);\n1416:\t}\n1417:\t\n1418:\tstatic void rcu_free_defer(struct folio *folio)\n1419:\t{\n1420:\t\tstruct rcu_free_pcpu *r = raw_cpu_ptr(\u0026rcu_free_pcpu);\n1421:\t\tstruct zone *zone = folio_zone(folio);\n1422:\t\tunsigned long free, waiting;\n1423:\t\tunsigned long flags;\n1424:\t\n1425:\t\tspin_lock_irqsave(\u0026r-\u003elock, flags);\n1426:\t\t__mod_zone_page_state(zone, NR_FREE_RCU_PAGES, folio_nr_pages(folio));\n1427:\t\tlist_add_tail(\u0026folio-\u003elru, \u0026r-\u003epending);\n1428:\t\tr-\u003enr_pending += folio_nr_pages(folio);\n1429:\t\trcu_free_start(r);\n1430:\t\tspin_unlock_irqrestore(\u0026r-\u003elock, flags);\n1431:\t\n1432:\t\t/* Unlocked: a stale count delays the expedite by one free */\n1433:\t\tif (data_race(r-\u003enr_pending) \u003c= RCU_FREE_EXPEDITE ||\n1434:\t\t    work_pending(\u0026r-\u003eexpedite))\n1435:\t\t\treturn;\n1436:\t\n1437:\t\t/*\n1438:\t\t * An expedited grace period interrupts every CPU: only while the\n1439:\t\t * zone is short of memory, or while more than an eighth of what it\n1440:\t\t * has free waits here.\n1441:\t\t */\n1442:\t\tfree = zone_page_state(zone, NR_FREE_PAGES);\n1443:\t\twaiting = zone_page_state(zone, NR_FREE_RCU_PAGES);\n1444:\t\tif (free \u003c high_wmark_pages(zone) || waiting \u003e free / 8)\n1445:\t\t\tqueue_work(rcu_free_wq, \u0026r-\u003eexpedite);\n1446:\t}\n1447:\t\n1448:\t/*\n"
}

							input: 26735
							output: 36
							thoughts: 25

Okay, here's my take on those lines from `mm/page_alloc.c`, framed as a thought process:

**Diving into the Heart of Page Allocation**

Alright, let's zero in. I need to understand this specific section of the page allocator. Let's start with lines 1450-1500 in `mm/page_alloc.c`. This part of the code is likely crucial, judging by its position, for understanding the mechanics of how the kernel actually gets its hands on physical memory pages. I suspect we're looking at core allocation logic or some critical part of the freeing process. I'm hoping to find the granular workings and interactions here that explain some observations I’ve made in related areas. I'm going to start by getting a handle on the function's structure. Is it handling a specific NUMA node, or is it global in scope? Are there any obvious locks or atomic operations being used that might be a source of contention? Understanding the context, whether it's allocation of a contiguous region or discrete single pages, is essential. Once I have the overall structure in mind, I can start to follow the control flow and look for the specific details that will clarify the questions I’ve been pondering.

{
  "File": "mm/page_alloc.c",
  "FirstLine": 1448,
  "LineCount": 50
}

{
  "Contents": "1448:\t/*\n1449:\t * Free what every CPU's lists hold.  A concurrent drain holds the lock\n1450:\t * until it has freed what it took.\n1451:\t */\n1452:\tstatic void rcu_free_drain(void)\n1453:\t{\n1454:\t\tstatic DEFINE_MUTEX(lock);\n1455:\t\tLIST_HEAD(list);\n1456:\t\tint cpu;\n1457:\t\n1458:\t\tmutex_lock(\u0026lock);\n1459:\t\tfor_each_possible_cpu(cpu) {\n1460:\t\t\tstruct rcu_free_pcpu *r = per_cpu_ptr(\u0026rcu_free_pcpu, cpu);\n1461:\t\n1462:\t\t\tspin_lock_irq(\u0026r-\u003elock);\n1463:\t\t\trcu_free_take(r, \u0026list);\n1464:\t\t\tspin_unlock_irq(\u0026r-\u003elock);\n1465:\t\t}\n1466:\t\tif (rcu_free_flush(\u0026list))\n1467:\t\t\tcount_vm_event(PGFREE_RCU_DRAIN);\n1468:\t\tmutex_unlock(\u0026lock);\n1469:\t}\n1470:\t\n1471:\t/* Free every waiting folio, including those a work took off the lists */\n1472:\tvoid rcu_free_sync(void)\n1473:\t{\n1474:\t\tint cpu;\n1475:\t\n1476:\t\tif (!IS_ENABLED(CONFIG_PAGE_RCU_FREE))\n1477:\t\t\treturn;\n1478:\t\n1479:\t\trcu_free_drain();\n1480:\t\tfor_each_possible_cpu(cpu) {\n1481:\t\t\tstruct rcu_free_pcpu *r = per_cpu_ptr(\u0026rcu_free_pcpu, cpu);\n1482:\t\n1483:\t\t\tflush_work(\u0026r-\u003erwork.work);\n1484:\t\t\tflush_work(\u0026r-\u003eexpedite);\n1485:\t\t}\n1486:\t}\n1487:\t\n1488:\tstatic void __init rcu_free_init(void)\n1489:\t{\n1490:\t\tint cpu;\n1491:\t\n1492:\t\tif (!IS_ENABLED(CONFIG_PAGE_RCU_FREE))\n1493:\t\t\treturn;\n1494:\t\n1495:\t\trcu_free_wq = alloc_workqueue(\"rcu_free\",\n1496:\t\t\t\t\t      WQ_MEM_RECLAIM | WQ_PERCPU, 0);\n1497:\t\n"
}

							input: 27777
							output: 36
							thoughts: 30

Okay, here's my take on those thoughts, tailored for an expert audience:

**Diving into `rcu_free_wait` in `mm/page_alloc.c`**

Right, let's zero in on `rcu_free_wait`. My immediate focus is line 4830 within `mm/page_alloc.c`. I need to understand how the RCU mechanisms interact with page freeing, and this function is a key piece of that puzzle. My gut feeling is that any issues here are going to be subtle, probably involving race conditions or unexpected delays in the reclaim path. Let's see how they're handling the quiescent state of the RCU callbacks with regard to freeing pages, and where that check occurs in this function. This is something I'm very familiar with.

{
  "File": "mm/page_alloc.c",
  "FirstLine": 4830,
  "LineCount": 45
}

{
  "Contents": "4830:\tbool gfp_pfmemalloc_allowed(gfp_t gfp_mask)\n4831:\t{\n4832:\t\treturn !!__gfp_pfmemalloc_flags(gfp_mask);\n4833:\t}\n4834:\t\n4835:\t/*\n4836:\t * Reclaim is about to give up: if the pages waiting for a grace period\n4837:\t * would bring an eligible zone above its min watermark, free them and\n4838:\t * retry.  Not for a dying task, which frees memory by exiting.\n4839:\t */\n4840:\tstatic bool rcu_free_wait(gfp_t gfp_mask, unsigned int order,\n4841:\t\t\t\t  struct alloc_context *ac, int alloc_flags,\n4842:\t\t\t\t  int *rcu_free_waits)\n4843:\t{\n4844:\t\tstruct zone *zone;\n4845:\t\tstruct zoneref *z;\n4846:\t\n4847:\t\tif (!IS_ENABLED(CONFIG_PAGE_RCU_FREE))\n4848:\t\t\treturn false;\n4849:\t\tif (tsk_is_oom_victim(current) || fatal_signal_pending(current))\n4850:\t\t\treturn false;\n4851:\t\tif (*rcu_free_waits \u003e= MAX_RECLAIM_RETRIES ||\n4852:\t\t    order \u003e PAGE_ALLOC_COSTLY_ORDER)\n4853:\t\t\treturn false;\n4854:\t\n4855:\t\tfor_each_zone_zonelist_nodemask(zone, z, ac-\u003ezonelist,\n4856:\t\t\t\t\tac-\u003ehighest_zoneidx, ac-\u003enodemask) {\n4857:\t\t\tunsigned long waiting, available;\n4858:\t\n4859:\t\t\tif (cpusets_enabled() \u0026\u0026 (alloc_flags \u0026 ALLOC_CPUSET) \u0026\u0026\n4860:\t\t\t    !__cpuset_zone_allowed(zone, gfp_mask))\n4861:\t\t\t\tcontinue;\n4862:\t\n4863:\t\t\twaiting = zone_page_state_snapshot(zone, NR_FREE_RCU_PAGES);\n4864:\t\t\tif (!waiting)\n4865:\t\t\t\tcontinue;\n4866:\t\n4867:\t\t\tavailable = waiting +\n4868:\t\t\t\t    zone_page_state_snapshot(zone, NR_FREE_PAGES);\n4869:\t\t\tif (__zone_watermark_ok(zone, order, min_wmark_pages(zone),\n4870:\t\t\t\t\t\tac-\u003ehighest_zoneidx, alloc_flags,\n4871:\t\t\t\t\t\tavailable)) {\n4872:\t\t\t\t(*rcu_free_waits)++;\n4873:\t\t\t\trcu_free_drain();\n4874:\t\t\t\treturn true;\n"
}

							input: 28750
							output: 35
							thoughts: 11

Okay, here's my take on those thoughts, tailored for an expert audience, presented in the first person:

**Diving into `filemap.c`**

Alright, time to get to the heart of the matter. Let's dig into `filemap.c`. I need to refresh my mental model of how the file mapping code in this system is structured. This is the foundation, right? Crucial for understanding how we're interacting with the physical storage and how data is actually being accessed and managed. I'm going to pay close attention to the memory management strategies in this particular implementation, looking for potential bottlenecks, opportunities for optimization, and any areas that might be contributing to the performance issues we've been investigating. My focus is going to be on the interplay between the virtual memory system, the page cache, and any specific optimizations they have implemented.

{
  "File": "mm/filemap.c",
  "FirstLine": 2950,
  "LineCount": 80
}

{
  "Contents": "2950:\t * are this folio's.\n2951:\t */\n2952:\tstatic size_t filemap_read_rcu_size(struct xa_state *xas, struct folio *folio,\n2953:\t\t\tpgoff_t index, size_t offset, size_t bytes)\n2954:\t{\n2955:\t\tpgoff_t end = index + DIV_ROUND_UP(offset + bytes, PAGE_SIZE);\n2956:\t\tpgoff_t next;\n2957:\t\n2958:\t\tif (end - index \u003e 1 \u0026\u0026 folio_test_large(folio)) {\n2959:\t\t\t/* Read without a reference: it only picks the page to check */\n2960:\t\t\tnext = folio_next_index(folio);\n2961:\t\t\tif (next \u003e index \u0026\u0026 next \u003c end)\n2962:\t\t\t\tend = next;\n2963:\t\t\txas_set(xas, end - 1);\n2964:\t\t\t/* A split after this copies PG_rcu_free to the new folios */\n2965:\t\t\tif (xas_load(xas) == folio)\n2966:\t\t\t\treturn min(bytes, (end - index) * PAGE_SIZE - offset);\n2967:\t\t}\n2968:\t\treturn min(bytes, PAGE_SIZE - offset);\n2969:\t}\n2970:\t\n2971:\t/*\n2972:\t * Copy @bytes from @folio, starting @offset into its page at @index.  Page\n2973:\t * by page, as copy_page_to_iter() does, without its checks for each call.\n2974:\t * Not copy_to_iter(): its usercopy check reads the folio's order, which a\n2975:\t * split can change.\n2976:\t */\n2977:\tstatic size_t filemap_read_rcu_copy(struct folio *folio, pgoff_t index,\n2978:\t\t\tsize_t offset, size_t bytes, struct iov_iter *iter)\n2979:\t{\n2980:\t\tstruct page *page = folio_page(folio, index - folio-\u003eindex);\n2981:\t\tsize_t n = 0;\n2982:\t\n2983:\t\twhile (n \u003c bytes) {\n2984:\t\t\tsize_t len = min(bytes - n, PAGE_SIZE - offset);\n2985:\t\t\tsize_t done;\n2986:\t\t\tvoid *kaddr;\n2987:\t\n2988:\t\t\tkaddr = kmap_local_page(page);\n2989:\t\t\tdone = _copy_to_iter(kaddr + offset, len, iter);\n2990:\t\t\tkunmap_local(kaddr);\n2991:\t\t\tn += done;\n2992:\t\t\tif (done \u003c len)\n2993:\t\t\t\tbreak;\n2994:\t\t\toffset = 0;\n2995:\t\t\tpage++;\n2996:\t\t}\n2997:\t\treturn n;\n2998:\t}\n2999:\t\n3000:\t/* Bytes one RCU read section copies at most: it holds up grace periods */\n3001:\t#define FILEMAP_RCU_READ_MAX\tSZ_512K\n3002:\t\n3003:\t/*\n3004:\t * Copy from folios with PG_rcu_free without taking references, at most\n3005:\t * FILEMAP_RCU_READ_MAX bytes under one rcu_read_lock().  Return true if that\n3006:\t * much was copied and the read is not done.  *last_index is the page the\n3007:\t * previous copy ended in, set as the slow path sets last_pos.\n3008:\t */\n3009:\tstatic bool filemap_read_rcu_batch(struct kiocb *iocb, struct iov_iter *iter,\n3010:\t\t\tpgoff_t *last_index, size_t *copied)\n3011:\t{\n3012:\t\tstruct address_space *mapping = iocb-\u003eki_filp-\u003ef_mapping;\n3013:\t\tstruct inode *inode = mapping-\u003ehost;\n3014:\t\tXA_STATE(xas, \u0026mapping-\u003ei_pages, 0);\n3015:\t\tsize_t budget = FILEMAP_RCU_READ_MAX;\n3016:\t\tbool more = false;\n3017:\t\n3018:\t\t/* flush_dcache_folio() is not known to be safe without a reference */\n3019:\t\tif (ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE \u0026\u0026\n3020:\t\t    mapping_writably_mapped(mapping))\n3021:\t\t\treturn false;\n3022:\t\n3023:\t\t/* A user fault cannot sleep in the RCU read section: it ends the copy */\n3024:\t\trcu_read_lock();\n3025:\t\tpagefault_disable();\n3026:\t\twhile (iov_iter_count(iter)) {\n3027:\t\t\tloff_t pos = iocb-\u003eki_pos;\n3028:\t\t\tpgoff_t index = pos \u003e\u003e PAGE_SHIFT;\n3029:\t\t\tsize_t offset = offset_in_page(pos);\n"
}

							input: 30345
							output: 35
							thoughts: 24

Okay, let's dive into this. Here's how I'd approach summarizing the prompt:

**Analyzing `mm/filemap.c` - Lines 3030-3100**

Alright, I'm being directed to focus on lines 3030-3100 of `mm/filemap.c`. Immediately, I'm thinking about what the likely focus is within that specific region. Given the file name, it's almost certainly dealing with file mapping, which means memory management related to files, likely including things like page cache interaction, read/write handling, perhaps even Direct I/O considerations or maybe even dealing with swap operations. I know from experience that `mm/filemap.c` is the heart of file system caching in the Linux kernel. Therefore, I'm expecting to see calls to page-related functions, possibly `read_cache_page`, `write_cache_page`, or similar functions that handle the movement of data between disk and memory. The fact that the prompt is a specific region indicates either a potential issue in a given area, or a function of particular importance.

I'm mentally preparing to understand the exact flow of data here. I'll need to look for calls to higher-level filesystem functions, like `generic_file_read`, and then track through the cache layer. I'll be looking for any hints on how the kernel handles data synchronization between the page cache and the underlying storage, and how it handles situations like writeback, invalidation, or even potentially, dealing with concurrent access from different processes. There may be some locking primitives. I'll also be mindful of the surrounding code and try to understand what functions it's likely being called by and calling, to get some kind of context for the lines of code.

Let me open the file and get to it.

{
  "File": "mm/filemap.c",
  "FirstLine": 3030,
  "LineCount": 70
}

{
  "Contents": "3030:\t\t\tstruct folio *folio;\n3031:\t\t\tsize_t bytes, n;\n3032:\t\t\tloff_t isize;\n3033:\t\n3034:\t\t\tif (!budget) {\n3035:\t\t\t\tmore = true;\n3036:\t\t\t\tbreak;\n3037:\t\t\t}\n3038:\t\t\tfolio = filemap_read_rcu_lookup(\u0026xas, inode, pos, \u0026isize);\n3039:\t\t\tif (!folio)\n3040:\t\t\t\tbreak;\n3041:\t\t\tif (!filemap_read_rcu_can_copy(folio, index, *last_index))\n3042:\t\t\t\tbreak;\n3043:\t\n3044:\t\t\tbytes = min_t(loff_t, isize - pos, iov_iter_count(iter));\n3045:\t\t\tbytes = filemap_read_rcu_size(\u0026xas, folio, index, offset,\n3046:\t\t\t\t\t\t      min(bytes, budget));\n3047:\t\t\tn = filemap_read_rcu_copy(folio, index, offset, bytes, iter);\n3048:\t\t\t*copied += n;\n3049:\t\t\tiocb-\u003eki_pos += n;\n3050:\t\t\tbudget -= n;\n3051:\t\t\tif (n)\n3052:\t\t\t\t*last_index = (pos + n - 1) \u003e\u003e PAGE_SHIFT;\n3053:\t\t\tif (n \u003c bytes)\n3054:\t\t\t\tbreak;\n3055:\t\t}\n3056:\t\tpagefault_enable();\n3057:\t\trcu_read_unlock();\n3058:\t\n3059:\t\treturn more;\n3060:\t}\n3061:\t\n3062:\t/*\n3063:\t * Copy from the page cache without folio references, in batches, and\n3064:\t * return how much was copied.  The slow path reads the rest.\n3065:\t */\n3066:\tsize_t filemap_read_rcu(struct kiocb *iocb, struct iov_iter *iter,\n3067:\t\t\tloff_t last_pos)\n3068:\t{\n3069:\t\t/* The page the previous read ended in; ULONG_MAX if none */\n3070:\t\tpgoff_t last_index = (last_pos - 1) \u003e\u003e PAGE_SHIFT;\n3071:\t\tsize_t copied = 0;\n3072:\t\n3073:\t\t/* Not when switched off, or on an inode that never had a folio flagged */\n3074:\t\tif (!lockless_read_enabled() ||\n3075:\t\t    !mapping_rcu_read(iocb-\u003eki_filp-\u003ef_mapping))\n3076:\t\t\treturn 0;\n3077:\t\n3078:\t\twhile (filemap_read_rcu_batch(iocb, iter, \u0026last_index, \u0026copied)) {\n3079:\t\t\tif (fatal_signal_pending(current))\n3080:\t\t\t\tbreak;\n3081:\t\t\tcond_resched();\n3082:\t\t}\n3083:\t\treturn copied;\n3084:\t}\n3085:\t\n3086:\t/**\n3087:\t * filemap_read - Read data from the page cache.\n3088:\t * @iocb: The iocb to read.\n3089:\t * @iter: Destination for the data.\n3090:\t * @already_read: Number of bytes already read by the caller.\n3091:\t *\n3092:\t * Copies data from the page cache.  If the data is not currently present,\n3093:\t * uses the readahead and read_folio address_space operations to fetch it.\n3094:\t *\n3095:\t * Return: Total number of bytes copied, including those already read by\n3096:\t * the caller.  If an error happens before any bytes are copied, returns\n3097:\t * a negative error number.\n3098:\t */\n3099:\tssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter,\n"
}

							input: 31879
							output: 244
							thoughts: 918

**Analyzing the Scope for Fuzzing with PAGE_RCU_FREE**

Okay, let's break this down. My goal here is to identify the most effective way to focus fuzzing efforts on this new RCU-based page cache read feature, specifically the `PAGE_RCU_FREE` configuration option. The core question is: which functions are *uniquely* related to this change and therefore the most relevant for fuzzing?

First, `PAGE_RCU_FREE` itself seems sufficient to represent the enabling condition. No need to overcomplicate the configuration aspect. Now, for the real work: identifying the key functions. I need to pinpoint which functions *directly* implement this new lockless RCU read logic.

The direct entry points seem to be: `filemap_read_rcu` and its batch counterpart `filemap_read_rcu_batch`. Then there are functions that interact with it: `filemap_read_accessed`, which detects hot folios and calls `filemap_mark_hot`. `filemap_mark_hot` is interesting; it marks folios as `PG_rcu_free` and mappings as `AS_RCU_READ`. Finally, we have the RCU-specific freeing functions: `rcu_free_defer`, `rcu_free_workfn`, `rcu_free_expedite_workfn`, `rcu_free_drain`, `rcu_free_sync`, and `rcu_free_wait`. These functions handle the deferred freeing of pages after the RCU grace period.

The question then arises: should I include more ubiquitous functions like `filemap_read`? The guideline explicitly advises against including generic, ubiquitous functions. Listing such functions would cause the fuzzer to classify a large number of unrelated tests as "focused", diluting the fuzzing effort. Since `filemap_read` *is* called practically on every read syscall, it would be a mistake. However, the other functions are *specifically* introduced or targeted by this patch and are directly related to the new feature without being ubiquitous. So, I will exclude `filemap_read`.

To confirm I'm on the right track, let me double-check: `WorthFuzzing` is set to `true`, the `EnableConfigs` list contains only `PAGE_RCU_FREE`, and my `FocusSymbols` list contains: `filemap_read_rcu`, `filemap_read_rcu_batch`, `filemap_read_accessed`, `filemap_mark_hot`, `rcu_free_defer`, `rcu_free_workfn`, `rcu_free_expedite_workfn`, `rcu_free_drain`, `rcu_free_sync`, and `rcu_free_wait`.

In summary, this patch introduces lockless RCU reads within the page cache, which improves performance by allowing readers to copy data from hot folios without increasing reference counts and delaying page freeing until an RCU grace period has passed. The change touches the core page cache read paths, page allocation, memory failure handling, and fuse buffer moving. All these changes are accessible from user space via the standard read and splice syscalls on amd64.

Okay, I'm confident in this analysis. I'm going to call `set-results` now.

{
  "EnableConfigs": [
    "PAGE_RCU_FREE"
  ],
  "FocusSymbols": [
    "filemap_read_rcu",
    "filemap_read_rcu_batch",
    "filemap_read_accessed",
    "filemap_mark_hot",
    "rcu_free_defer",
    "rcu_free_workfn",
    "rcu_free_expedite_workfn",
    "rcu_free_drain",
    "rcu_free_sync",
    "rcu_free_wait"
  ],
  "Reasoning": "The patch introduces lockless page cache reads and deferred RCU page freeing (PAGE_RCU_FREE). Readers can copy from hot page cache folios under RCU without taking a folio reference count, while page freeing defers memory recycling until an RCU grace period completes. This touches core MM logic (page allocator free paths, slowpath reclaim retry, memory failure, and page cache read paths in filemap and shmem) which is reachable through standard user-space file reading operations on 64-bit systems.",
  "WorthFuzzing": true
}

{
  "EnableConfigs": [
    "PAGE_RCU_FREE"
  ],
  "FocusSymbols": [
    "filemap_read_rcu",
    "filemap_read_rcu_batch",
    "filemap_read_accessed",
    "filemap_mark_hot",
    "rcu_free_defer",
    "rcu_free_workfn",
    "rcu_free_expedite_workfn",
    "rcu_free_drain",
    "rcu_free_sync",
    "rcu_free_wait"
  ],
  "Reasoning": "The patch introduces lockless page cache reads and deferred RCU page freeing (PAGE_RCU_FREE). Readers can copy from hot page cache folios under RCU without taking a folio reference count, while page freeing defers memory recycling until an RCU grace period completes. This touches core MM logic (page allocator free paths, slowpath reclaim retry, memory failure, and page cache read paths in filemap and shmem) which is reachable through standard user-space file reading operations on 64-bit systems.",
  "WorthFuzzing": true
}

{
  "WorthFuzzing": true
}

{
  "KMSANReasoning": "The patch series implements lockless page cache reads using RCU (CONFIG_PAGE_RCU_FREE) and defers the physical freeing of hot folios until after an RCU grace period.\n\nFrom a memory safety and sanitizer perspective:\n1. No uninitialized memory is exposed: All local variables and data structures (such as struct rcu_free_pcpu, list heads, and loop variables) are properly initialized before use.\n2. Kernel-to-user copies: The fast path in `filemap_read_rcu_copy()` copies file data from uptodate folios bounded strictly by the file's `i_size` and page boundaries, identically to the existing slow-path page cache read mechanisms; no uninitialized kernel structure fields or padding are leaked to user space.\n3. KMSAN hooks: The patch properly defers `kmsan_free_page()` in `__free_pages_prepare()` until the folio is actually released after the RCU grace period, ensuring KMSAN page shadow tracking remains accurate.\n4. Risk surface: The primary risks associated with this lockless RCU page cache reader patch are concurrency races (e.g. during folio splitting, truncation, migration, or hwpoison), potential use-after-free bugs, reference-counting anomalies, and locking/deadlock issues. These failure modes are covered by KASAN, LOCKDEP, and KCSAN.\n\nBecause the changes do not introduce or expose uninitialized memory reads or kernel information leaks, a dedicated KMSAN fuzzing session is not warranted.",
  "NeedsKMSAN": false
}

You are an expert Linux kernel security engineer specializing in kernel memory
error detectors (KASAN and KMSAN). Your job is to review the provided patch series and
determine if the code changes justify spawning a dedicated KMSAN (KernelMemorySanitizer)
fuzzing session in addition to standard KASAN fuzzing.

CRITICAL DISTINCTION BETWEEN KASAN AND KMSAN:
- Standard KASAN kernel builds (upstream-apparmor-kasan.config) already enable
  a comprehensive suite of debugging tools and sanitizers, including KASAN
  (out-of-bounds accesses, use-after-free, double free, invalid free), LOCKDEP
  (locking bugs and deadlocks), UB-sanitizers, and memory corruption checks.
- KMSAN (KernelMemorySanitizer) detects reads of UNINITIALIZED memory (stack, heap,
  or page allocations) and kernel-to-user memory info-leaks.

Rule: THERE IS NO SENSE IN RUNNING A KMSAN SESSION IF A BUG CAN BE CAUGHT BY KASAN,
LOCKDEP, OR OTHER STANDARD BUG DETECTORS.
A dedicated KMSAN fuzzing session incurs significant resource costs. You must ONLY
set NeedsKMSAN=true if the code changes introduce or expose UNINITIALIZED MEMORY risks
that are detected ONLY by KMSAN.

Look holistically at the patch series and surrounding code. Even if no direct
uninitialized field accesses or new buffer allocations are added in the diff itself,
a patch may alter control flow, bounds checking, or data length calculations in ways
that change how the rest of the code operates on existing buffers (e.g. allowing
uninitialized stack/heap memory to be read, copied to user space, or used in control
flow). Do not hesitate to use your code access tools to inspect the surrounding code,
called functions, and callers.

Set NeedsKMSAN=true ONLY IF the patch introduces or modifies:
1. Kernel structures sent to user space (via copy_to_user, put_user, netlink skb
   attributes, ioctl output arguments, socket options, or BPF buffers) where fields
   or structure padding might not be fully initialized/zeroed.
2. Conditional logic or branching that depends on potentially uninitialized variables
   or struct fields.
3. Allocation or initialization of complex data structures where uninitialized fields
   could be read later in reachable code paths.
4. Bounds checks, lengths, or logic in a way that allows surrounding code to access
   uninitialized bytes of existing buffers.

Set NeedsKMSAN=false IF:
- The code changes primarily risk out-of-bounds access, array overflows, NULL pointer
  dereferences, locking deadlocks, or use-after-free bugs (these are already caught
  by KASAN, LOCKDEP, or standard bug detectors).
- All stack/heap structures touched or introduced by the patch are fully zeroed
  or initialized (e.g. using = {0}, memset, kzalloc) before being read or copied.
- The patch does not introduce any risk of uninitialized memory usage or info-leaks.

Use your code access tools to inspect the surrounding code if necessary, then provide
detailed KMSANReasoning contrasting KASAN vs KMSAN applicability for this patch.
Prefer calling several tools at the same time to save round-trips.


Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.

Target architecture: amd64

For your convenience, here is the diff of the changes:
commit dddf59c0f0df6bc96619a328ac462a9dfff93355
Author: syz-cluster <triage@syzkaller.com>
Date:   Mon Oct 19 19:57:32 2026 +0000

    syz-cluster: applied patch under review

diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c
index 4fec31fc0b845..526929c7b3f1a 100644
--- a/fs/fuse/dev.c
+++ b/fs/fuse/dev.c
@@ -1152,6 +1152,10 @@ static int fuse_try_move_folio(struct fuse_copy_state *cs, struct folio **foliop
 
 	newfolio = page_folio(buf->page);
 
+	/* A lockless page cache read may still be copying from it */
+	if (folio_test_rcu_free(newfolio))
+		goto out_fallback_unlock;
+
 	folio_clear_uptodate(newfolio);
 	folio_clear_mappedtodisk(newfolio);
 
diff --git a/fs/splice.c b/fs/splice.c
index 9d8f63e2fd1ab..0133c2348da03 100644
--- a/fs/splice.c
+++ b/fs/splice.c
@@ -55,56 +55,6 @@ static noinline void pipe_clear_nowait(struct file *file)
 	} while (!try_cmpxchg(&file->f_mode, &fmode, fmode & ~FMODE_NOWAIT));
 }
 
-/*
- * Attempt to steal a page from a pipe buffer. This should perhaps go into
- * a vm helper function, it's already simplified quite a bit by the
- * addition of remove_mapping(). If success is returned, the caller may
- * attempt to reuse this page for another destination.
- */
-static bool page_cache_pipe_buf_try_steal(struct pipe_inode_info *pipe,
-		struct pipe_buffer *buf)
-{
-	struct folio *folio = page_folio(buf->page);
-	struct address_space *mapping;
-
-	folio_lock(folio);
-
-	mapping = folio_mapping(folio);
-	if (mapping) {
-		WARN_ON(!folio_test_uptodate(folio));
-
-		/*
-		 * At least for ext2 with nobh option, we need to wait on
-		 * writeback completing on this folio, since we'll remove it
-		 * from the pagecache.  Otherwise truncate wont wait on the
-		 * folio, allowing the disk blocks to be reused by someone else
-		 * before we actually wrote our data to them. fs corruption
-		 * ensues.
-		 */
-		folio_wait_writeback(folio);
-
-		if (!filemap_release_folio(folio, GFP_KERNEL))
-			goto out_unlock;
-
-		/*
-		 * If we succeeded in removing the mapping, set LRU flag
-		 * and return good.
-		 */
-		if (remove_mapping(mapping, folio)) {
-			buf->flags |= PIPE_BUF_FLAG_LRU;
-			return true;
-		}
-	}
-
-	/*
-	 * Raced with truncate or failed to remove folio from current
-	 * address space, unlock and return failure.
-	 */
-out_unlock:
-	folio_unlock(folio);
-	return false;
-}
-
 static void page_cache_pipe_buf_release(struct pipe_inode_info *pipe,
 					struct pipe_buffer *buf)
 {
@@ -155,7 +105,6 @@ static int page_cache_pipe_buf_confirm(struct pipe_inode_info *pipe,
 const struct pipe_buf_operations page_cache_pipe_buf_ops = {
 	.confirm	= page_cache_pipe_buf_confirm,
 	.release	= page_cache_pipe_buf_release,
-	.try_steal	= page_cache_pipe_buf_try_steal,
 	.get		= generic_pipe_buf_get,
 };
 
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 94f9c3ff54160..3336b9e5ee999 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -228,6 +228,7 @@ enum zone_stat_item {
 	NR_ZSPAGES,		/* allocated in zsmalloc */
 #endif
 	NR_FREE_CMA_PAGES,
+	NR_FREE_RCU_PAGES,	/* freed, waiting for an RCU grace period */
 #ifdef CONFIG_UNACCEPTED_MEMORY
 	NR_UNACCEPTED,
 #endif
diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
index 7a863572adce7..4916b5713a74b 100644
--- a/include/linux/page-flags.h
+++ b/include/linux/page-flags.h
@@ -111,6 +111,9 @@ enum pageflags {
 	PG_swapbacked,		/* Page is backed by RAM/swap */
 	PG_unevictable,		/* Page is "unevictable"  */
 	PG_dropbehind,		/* drop pages on IO completion */
+#ifdef CONFIG_PAGE_RCU_FREE
+	PG_rcu_free,		/* free only after an RCU grace period */
+#endif
 #ifdef CONFIG_MMU
 	PG_mlocked,		/* Page is vma mlocked */
 #endif
@@ -602,6 +605,27 @@ FOLIO_FLAG(dropbehind, FOLIO_HEAD_PAGE)
 	FOLIO_TEST_CLEAR_FLAG(dropbehind, FOLIO_HEAD_PAGE)
 	__FOLIO_SET_FLAG(dropbehind, FOLIO_HEAD_PAGE)
 
+/*
+ * A lockless lookup that sees PG_rcu_free and then still finds the folio
+ * where it looked may read it until rcu_read_unlock().  Set under a
+ * reference.  Cleared by the free, or when the folio's inode is evicted:
+ * no lockless lookup can reach it then.  Not for hugetlb or
+ * free_pages_nolock().
+ */
+#ifdef CONFIG_PAGE_RCU_FREE
+FOLIO_TEST_FLAG(rcu_free, FOLIO_HEAD_PAGE)
+	FOLIO_TEST_SET_FLAG(rcu_free, FOLIO_HEAD_PAGE)
+	FOLIO_CLEAR_FLAG(rcu_free, FOLIO_HEAD_PAGE)
+	__FOLIO_CLEAR_FLAG(rcu_free, FOLIO_HEAD_PAGE)
+#define __PG_RCU_FREE (1UL << PG_rcu_free)
+#else
+FOLIO_TEST_FLAG_FALSE(rcu_free)
+	FOLIO_TEST_SET_FLAG_FALSE(rcu_free)
+	FOLIO_CLEAR_FLAG_NOOP(rcu_free)
+	__FOLIO_CLEAR_FLAG_NOOP(rcu_free)
+#define __PG_RCU_FREE 0
+#endif
+
 #ifdef CONFIG_HIGHMEM
 /*
  * Must use a macro here due to header dependency issues. page_zone() is not
diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 0adfa6605653d..6c7f58803eee4 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -210,6 +210,7 @@ enum mapping_flags {
 	AS_WRITEBACK_MAY_DEADLOCK_ON_RECLAIM = 9,
 	AS_KERNEL_FILE = 10,	/* mapping for a fake kernel file that shouldn't
 				   account usage to user cgroups */
+	AS_RCU_READ = 11,	/* a folio has had PG_rcu_free */
 	/* Bits 16-25 are used for FOLIO_ORDER */
 	AS_FOLIO_ORDER_BITS = 5,
 	AS_FOLIO_ORDER_MIN = 16,
@@ -269,6 +270,17 @@ static inline bool mapping_unevictable(const struct address_space *mapping)
 	return mapping && test_bit(AS_UNEVICTABLE, &mapping->flags);
 }
 
+static inline void mapping_set_rcu_read(struct address_space *mapping)
+{
+	if (!test_bit(AS_RCU_READ, &mapping->flags))
+		set_bit(AS_RCU_READ, &mapping->flags);
+}
+
+static inline bool mapping_rcu_read(const struct address_space *mapping)
+{
+	return test_bit(AS_RCU_READ, &mapping->flags);
+}
+
 static inline void mapping_set_exiting(struct address_space *mapping)
 {
 	set_bit(AS_EXITING, &mapping->flags);
diff --git a/include/linux/swap.h b/include/linux/swap.h
index 5658a1634b85e..67aa5271ae6cb 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -295,7 +295,7 @@ extern unsigned long totalreserve_pages;
 
 /* linux/mm/folio.c */
 void folio_add_lru(struct folio *folio);
-void folio_mark_accessed(struct folio *folio);
+bool folio_mark_accessed(struct folio *folio);
 void lru_add_drain_all(void);
 
 enum lru_cache_drained {
diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h
index 2628ccda076a0..c1a63f070aa71 100644
--- a/include/linux/vm_event_item.h
+++ b/include/linux/vm_event_item.h
@@ -36,6 +36,8 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
 		FOR_ALL_ZONES(ALLOCSTALL)
 		FOR_ALL_ZONES(PGSCAN_SKIP)
 		PGFREE, PGACTIVATE, PGDEACTIVATE, PGLAZYFREE,
+		PGFREE_RCU, PGFREE_RCU_EXPEDITE, PGFREE_RCU_DRAIN,
+		PGFREE_RCU_MARK,
 		PGFAULT, PGMAJFAULT,
 		PGLAZYFREED,
 		PGREUSE,
diff --git a/include/trace/events/mmflags.h b/include/trace/events/mmflags.h
index 935893e5ea53b..47c3411a75e8f 100644
--- a/include/trace/events/mmflags.h
+++ b/include/trace/events/mmflags.h
@@ -117,6 +117,12 @@ TRACE_DEFINE_ENUM(___GFP_LAST_BIT);
 #define IF_HAVE_PG_IDLE(_name)
 #endif
 
+#ifdef CONFIG_PAGE_RCU_FREE
+#define IF_HAVE_PG_RCU_FREE(_name) ,{1UL << PG_##_name, __stringify(_name)}
+#else
+#define IF_HAVE_PG_RCU_FREE(_name)
+#endif
+
 #ifdef CONFIG_ARCH_USES_PG_ARCH_2
 #define IF_HAVE_PG_ARCH_2(_name) ,{1UL << PG_##_name, __stringify(_name)}
 #else
@@ -152,6 +158,7 @@ TRACE_DEFINE_ENUM(___GFP_LAST_BIT);
 	DEF_PAGEFLAG_NAME(swapbacked),					\
 	DEF_PAGEFLAG_NAME(unevictable),					\
 	DEF_PAGEFLAG_NAME(dropbehind)					\
+IF_HAVE_PG_RCU_FREE(rcu_free)						\
 IF_HAVE_PG_MLOCK(mlocked)						\
 IF_HAVE_PG_HWPOISON(hwpoison)						\
 IF_HAVE_PG_IDLE(idle)							\
diff --git a/mm/Kconfig b/mm/Kconfig
index 604c58199acbf..1ad1cb7969e79 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -718,6 +718,21 @@ config PCP_BATCH_SCALE_MAX
 	  latency.  This option sets the upper limit of scale factor to limit
 	  the maximum latency.
 
+config PAGE_RCU_FREE
+	def_bool 64BIT
+	help
+	  Reads from the page cache can copy from folios that are read
+	  again and again without taking a reference on them, so small
+	  reads of one file from many CPUs do not contend on the folio's
+	  reference count.  The cost is that memory freed from such
+	  folios becomes reusable only after an RCU grace period.
+
+	  64-bit only: it takes a page flag, and those are scarce on
+	  32-bit.
+
+	  Writing 0 to lockless_page_cache_read in debugfs turns these
+	  reads off.
+
 config PHYS_ADDR_T_64BIT
 	def_bool 64BIT
 
diff --git a/mm/filemap.c b/mm/filemap.c
index 00fd89cf6f550..7021ed2ab6c9c 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -48,6 +48,7 @@
 #include <linux/rcupdate_wait.h>
 #include <linux/sched/mm.h>
 #include <linux/sysctl.h>
+#include <linux/debugfs.h>
 #include <linux/pgalloc.h>
 
 #include <asm/tlbflush.h>
@@ -138,6 +139,9 @@ static void page_cache_delete(struct address_space *mapping,
 	nr = folio_nr_pages(folio);
 
 	VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
+	/* No lockless reader can be in an evicted inode */
+	if (folio_test_rcu_free(folio) && mapping_exiting(mapping))
+		folio_clear_rcu_free(folio);
 
 	xas_store(&xas, shadow);
 	xas_init_marks(&xas);
@@ -306,6 +310,9 @@ static void page_cache_delete_batch(struct address_space *mapping,
 		}
 
 		WARN_ON_ONCE(!folio_test_locked(folio));
+		/* No lockless reader can be in an evicted inode */
+		if (folio_test_rcu_free(folio) && mapping_exiting(mapping))
+			folio_clear_rcu_free(folio);
 
 		folio->mapping = NULL;
 		/* Leave folio->index set: truncation lookup relies on it */
@@ -2759,6 +2766,101 @@ static inline bool pos_same_folio(loff_t pos1, loff_t pos2, struct folio *folio)
 	return (pos1 >> shift == pos2 >> shift);
 }
 
+#ifdef CONFIG_PAGE_RCU_FREE
+/* Writing 0 to lockless_page_cache_read in debugfs turns the reads off */
+static DEFINE_STATIC_KEY_TRUE(lockless_read_key);
+
+static bool lockless_read_enabled(void)
+{
+	return static_branch_likely(&lockless_read_key);
+}
+
+#ifdef CONFIG_DEBUG_FS
+static int lockless_read_get(void *data, u64 *val)
+{
+	*val = static_key_enabled(&lockless_read_key);
+	return 0;
+}
+
+static int lockless_read_set(void *data, u64 val)
+{
+	if (val > 1)
+		return -EINVAL;
+	if (val)
+		static_branch_enable(&lockless_read_key);
+	else
+		static_branch_disable(&lockless_read_key);
+	return 0;
+}
+DEFINE_DEBUGFS_ATTRIBUTE(lockless_read_fops, lockless_read_get,
+			 lockless_read_set, "%llu\n");
+
+static int __init lockless_read_debugfs_init(void)
+{
+	debugfs_create_file_unsafe("lockless_page_cache_read", 0644, NULL, NULL,
+				   &lockless_read_fops);
+	return 0;
+}
+late_initcall(lockless_read_debugfs_init);
+#endif
+#else
+static bool lockless_read_enabled(void)
+{
+	return false;
+}
+#endif
+
+/*
+ * Reads keep coming back to @folio: free it only after an RCU grace period
+ * from now on.  Not if its next read needs a reference anyway.  Not if it
+ * is unevictable: folio_mark_accessed() calls such a folio hot after at
+ * most one access, so every ramfs folio that is read would get the flag.
+ */
+static void filemap_mark_hot(struct folio *folio)
+{
+	struct address_space *mapping;
+
+	if (!lockless_read_enabled())
+		return;
+	if (folio_test_rcu_free(folio) || !folio_test_uptodate(folio) ||
+	    folio_test_dropbehind(folio) || folio_test_unevictable(folio))
+		return;
+	/* shmem does no readahead: there the bit is PG_reclaim */
+	if (folio_test_readahead(folio) && !folio_test_swapbacked(folio))
+		return;
+	if (folio_test_set_rcu_free(folio))
+		return;
+
+	count_vm_events(PGFREE_RCU_MARK, folio_nr_pages(folio));
+	/* NULL if the folio was truncated meanwhile */
+	mapping = READ_ONCE(folio->mapping);
+	if (mapping)
+		mapping_set_rcu_read(mapping);
+}
+
+/*
+ * filemap_read() is about to copy from @folio at @pos.  @last_pos is where
+ * the previous read of this open file ended, or how far this one has got.
+ *
+ * Reading on in the folio that the last read ended in is not another access
+ * to it: a folio read in small pieces is marked accessed once.
+ *
+ * The folio is hot when folio_mark_accessed() finds it as accessed as the
+ * LRU records, or when this open file goes back to data that it has just
+ * read.
+ */
+void filemap_read_accessed(struct folio *folio, loff_t pos, loff_t last_pos)
+{
+	bool hot;
+
+	if (pos_same_folio(pos, last_pos - 1, folio))
+		hot = pos < last_pos;
+	else
+		hot = folio_mark_accessed(folio);
+	if (hot)
+		filemap_mark_hot(folio);
+}
+
 static void filemap_end_dropbehind_read(struct folio *folio)
 {
 	if (!folio_test_dropbehind(folio))
@@ -2771,6 +2873,216 @@ static void filemap_end_dropbehind_read(struct folio *folio)
 	}
 }
 
+/*
+ * Whether filemap_read_rcu_batch() can copy from @folio, or has to leave it
+ * to the slow path, which holds a reference for readahead, dropbehind and
+ * folio_mark_accessed(), and deals with poisoned pages.  @last_index is the
+ * page the previous copy ended in.
+ */
+static bool filemap_read_rcu_can_copy(struct folio *folio, pgoff_t index,
+				      pgoff_t last_index)
+{
+	if (!folio_test_uptodate(folio) || folio_test_dropbehind(folio))
+		return false;
+	/* shmem does no readahead: there the bit is PG_reclaim */
+	if (folio_test_readahead(folio) && !folio_test_swapbacked(folio))
+		return false;
+	/* memory_failure() waits only for readers that got past this */
+	if (folio_contain_hwpoisoned_page(folio))
+		return false;
+
+	/*
+	 * The slow path does not call folio_mark_accessed() on the folio that
+	 * the previous read ended in, so copying from it skips nothing.  The
+	 * folio's size can change without a reference: compare pages instead.
+	 */
+	if (index == last_index)
+		return true;
+
+	/* Otherwise only if folio_mark_accessed() would leave it as it is */
+	if (lru_gen_enabled()) {
+		unsigned long all = LRU_REFS_FLAGS | BIT(PG_workingset);
+
+		return folio_test_unevictable(folio) ||
+		       (READ_ONCE(folio->flags.f) & all) == all;
+	}
+	return folio_test_referenced(folio) && !folio_test_idle(folio) &&
+	       (folio_test_active(folio) || folio_test_unevictable(folio));
+}
+
+/*
+ * Return the folio at @pos if it is this file's page until rcu_read_unlock(),
+ * and read i_size into @isize.  A folio with PG_rcu_free is freed only after
+ * an RCU grace period.  Without a reference, the folio that xas_load()
+ * returned can be freed and its memory reused as another folio before the
+ * flag is tested, so look at the slot again after the flag.
+ */
+static struct folio *filemap_read_rcu_lookup(struct xa_state *xas,
+		struct inode *inode, loff_t pos, loff_t *isize)
+{
+	struct folio *folio;
+
+	xas_set(xas, pos >> PAGE_SHIFT);
+	folio = xas_load(xas);
+	if (xas_retry(xas, folio) || !folio || xa_is_value(folio))
+		return NULL;
+
+	/* Not folio_test_rcu_free(): it may not be a folio any more */
+	if (!(READ_ONCE(folio->flags.f) & __PG_RCU_FREE))
+		return NULL;
+	/* Read i_size and slot after the flag: it implies uptodate */
+	smp_rmb();
+	*isize = i_size_read(inode);
+	if (pos >= *isize)
+		return NULL;
+	/* i_size_read() is an acquire; a write fills the slot first */
+	if (xas_reload(xas) != folio)
+		return NULL;
+	/* Read the rest of the folio only once the slot holds it */
+	smp_rmb();
+	return folio;
+}
+
+/*
+ * How much of @bytes, starting @offset into page @index, can be copied from
+ * @folio.  A folio is in all the slots of its range or in none: when the slot
+ * of the last page holds it too, so do the slots in between, and the pages
+ * are this folio's.
+ */
+static size_t filemap_read_rcu_size(struct xa_state *xas, struct folio *folio,
+		pgoff_t index, size_t offset, size_t bytes)
+{
+	pgoff_t end = index + DIV_ROUND_UP(offset + bytes, PAGE_SIZE);
+	pgoff_t next;
+
+	if (end - index > 1 && folio_test_large(folio)) {
+		/* Read without a reference: it only picks the page to check */
+		next = folio_next_index(folio);
+		if (next > index && next < end)
+			end = next;
+		xas_set(xas, end - 1);
+		/* A split after this copies PG_rcu_free to the new folios */
+		if (xas_load(xas) == folio)
+			return min(bytes, (end - index) * PAGE_SIZE - offset);
+	}
+	return min(bytes, PAGE_SIZE - offset);
+}
+
+/*
+ * Copy @bytes from @folio, starting @offset into its page at @index.  Page
+ * by page, as copy_page_to_iter() does, without its checks for each call.
+ * Not copy_to_iter(): its usercopy check reads the folio's order, which a
+ * split can change.
+ */
+static size_t filemap_read_rcu_copy(struct folio *folio, pgoff_t index,
+		size_t offset, size_t bytes, struct iov_iter *iter)
+{
+	struct page *page = folio_page(folio, index - folio->index);
+	size_t n = 0;
+
+	while (n < bytes) {
+		size_t len = min(bytes - n, PAGE_SIZE - offset);
+		size_t done;
+		void *kaddr;
+
+		kaddr = kmap_local_page(page);
+		done = _copy_to_iter(kaddr + offset, len, iter);
+		kunmap_local(kaddr);
+		n += done;
+		if (done < len)
+			break;
+		offset = 0;
+		page++;
+	}
+	return n;
+}
+
+/* Bytes one RCU read section copies at most: it holds up grace periods */
+#define FILEMAP_RCU_READ_MAX	SZ_512K
+
+/*
+ * Copy from folios with PG_rcu_free without taking references, at most
+ * FILEMAP_RCU_READ_MAX bytes under one rcu_read_lock().  Return true if that
+ * much was copied and the read is not done.  *last_index is the page the
+ * previous copy ended in, set as the slow path sets last_pos.
+ */
+static bool filemap_read_rcu_batch(struct kiocb *iocb, struct iov_iter *iter,
+		pgoff_t *last_index, size_t *copied)
+{
+	struct address_space *mapping = iocb->ki_filp->f_mapping;
+	struct inode *inode = mapping->host;
+	XA_STATE(xas, &mapping->i_pages, 0);
+	size_t budget = FILEMAP_RCU_READ_MAX;
+	bool more = false;
+
+	/* flush_dcache_folio() is not known to be safe without a reference */
+	if (ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE &&
+	    mapping_writably_mapped(mapping))
+		return false;
+
+	/* A user fault cannot sleep in the RCU read section: it ends the copy */
+	rcu_read_lock();
+	pagefault_disable();
+	while (iov_iter_count(iter)) {
+		loff_t pos = iocb->ki_pos;
+		pgoff_t index = pos >> PAGE_SHIFT;
+		size_t offset = offset_in_page(pos);
+		struct folio *folio;
+		size_t bytes, n;
+		loff_t isize;
+
+		if (!budget) {
+			more = true;
+			break;
+		}
+		folio = filemap_read_rcu_lookup(&xas, inode, pos, &isize);
+		if (!folio)
+			break;
+		if (!filemap_read_rcu_can_copy(folio, index, *last_index))
+			break;
+
+		bytes = min_t(loff_t, isize - pos, iov_iter_count(iter));
+		bytes = filemap_read_rcu_size(&xas, folio, index, offset,
+					      min(bytes, budget));
+		n = filemap_read_rcu_copy(folio, index, offset, bytes, iter);
+		*copied += n;
+		iocb->ki_pos += n;
+		budget -= n;
+		if (n)
+			*last_index = (pos + n - 1) >> PAGE_SHIFT;
+		if (n < bytes)
+			break;
+	}
+	pagefault_enable();
+	rcu_read_unlock();
+
+	return more;
+}
+
+/*
+ * Copy from the page cache without folio references, in batches, and
+ * return how much was copied.  The slow path reads the rest.
+ */
+size_t filemap_read_rcu(struct kiocb *iocb, struct iov_iter *iter,
+		loff_t last_pos)
+{
+	/* The page the previous read ended in; ULONG_MAX if none */
+	pgoff_t last_index = (last_pos - 1) >> PAGE_SHIFT;
+	size_t copied = 0;
+
+	/* Not when switched off, or on an inode that never had a folio flagged */
+	if (!lockless_read_enabled() ||
+	    !mapping_rcu_read(iocb->ki_filp->f_mapping))
+		return 0;
+
+	while (filemap_read_rcu_batch(iocb, iter, &last_index, &copied)) {
+		if (fatal_signal_pending(current))
+			break;
+		cond_resched();
+	}
+	return copied;
+}
+
 /**
  * filemap_read - Read data from the page cache.
  * @iocb: The iocb to read.
@@ -2796,6 +3108,7 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter,
 	bool writably_mapped;
 	loff_t isize, end_offset;
 	loff_t last_pos = ra->prev_pos;
+	size_t fast_read;
 
 	if (unlikely(iocb->ki_pos < 0))
 		return -EINVAL;
@@ -2807,6 +3120,14 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter,
 	iov_iter_truncate(iter, inode->i_sb->s_maxbytes - iocb->ki_pos);
 	folio_batch_init(&fbatch);
 
+	fast_read = filemap_read_rcu(iocb, iter, last_pos);
+	if (fast_read) {
+		already_read += fast_read;
+		last_pos = iocb->ki_pos;
+		if (!iov_iter_count(iter))
+			goto out;
+	}
+
 	do {
 		cond_resched();
 
@@ -2844,14 +3165,6 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter,
 		 */
 		writably_mapped = mapping_writably_mapped(mapping);
 
-		/*
-		 * When a read accesses the same folio several times, only
-		 * mark it as accessed the first time.
-		 */
-		if (!pos_same_folio(iocb->ki_pos, last_pos - 1,
-				    fbatch.folios[0]))
-			folio_mark_accessed(fbatch.folios[0]);
-
 		for (i = 0; i < folio_batch_count(&fbatch); i++) {
 			struct folio *folio = fbatch.folios[i];
 			size_t fsize = folio_size(folio);
@@ -2862,8 +3175,7 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter,
 
 			if (end_offset < folio_pos(folio))
 				break;
-			if (i > 0)
-				folio_mark_accessed(folio);
+			filemap_read_accessed(folio, iocb->ki_pos, last_pos);
 			/*
 			 * If users can be writing to this folio using arbitrary
 			 * virtual addresses, take care of potential aliasing
@@ -2893,8 +3205,11 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter,
 		folio_batch_init(&fbatch);
 	} while (iov_iter_count(iter) && iocb->ki_pos < isize && !error);
 
+out:
 	file_accessed(filp);
-	ra->prev_pos = last_pos;
+	/* Readers sharing the file would bounce the line for nothing */
+	if (ra->prev_pos != last_pos)
+		ra->prev_pos = last_pos;
 	return already_read ? already_read : error;
 }
 EXPORT_SYMBOL_GPL(filemap_read);
diff --git a/mm/folio.c b/mm/folio.c
index 50a6dbe55998e..6144b9cf301b3 100644
--- a/mm/folio.c
+++ b/mm/folio.c
@@ -352,28 +352,34 @@ static void __lru_cache_activate_folio(struct folio *folio)
 
 #ifdef CONFIG_LRU_GEN
 
-static void lru_gen_inc_refs(struct folio *folio)
+/*
+ * Return true if @folio is hot: unevictable, or its flags already held as
+ * many accesses as they can.
+ */
+static bool lru_gen_inc_refs(struct folio *folio)
 {
 	unsigned long new_flags, old_flags = READ_ONCE(folio->flags.f);
 
 	if (folio_test_unevictable(folio))
-		return;
+		return true;
 
 	/* see the comment on LRU_REFS_FLAGS */
 	if (!folio_test_referenced(folio)) {
 		set_mask_bits(&folio->flags.f, LRU_REFS_MASK, BIT(PG_referenced));
-		return;
+		return false;
 	}
 
 	do {
 		if ((old_flags & LRU_REFS_MASK) == LRU_REFS_MASK) {
-			if (!folio_test_workingset(folio))
-				folio_set_workingset(folio);
-			return;
+			if (folio_test_workingset(folio))
+				return true;
+			folio_set_workingset(folio);
+			return false;
 		}
 
 		new_flags = old_flags + BIT(LRU_REFS_PGOFF);
 	} while (!try_cmpxchg(&folio->flags.f, &old_flags, new_flags));
+	return false;
 }
 
 static bool lru_gen_clear_refs(struct folio *folio)
@@ -396,8 +402,9 @@ static bool lru_gen_clear_refs(struct folio *folio)
 
 #else /* !CONFIG_LRU_GEN */
 
-static void lru_gen_inc_refs(struct folio *folio)
+static bool lru_gen_inc_refs(struct folio *folio)
 {
+	return false;
 }
 
 static bool lru_gen_clear_refs(struct folio *folio)
@@ -419,15 +426,20 @@ static bool lru_gen_clear_refs(struct folio *folio)
  *
  * When a newly allocated folio is not yet visible, so safe for non-atomic ops,
  * __folio_set_referenced() may be substituted for folio_mark_accessed().
+ *
+ * Return: true if @folio is hot: it was already as accessed as the LRU
+ * records.  On the classic LRU that is active or unevictable, referenced
+ * and not idle; on the multi-gen LRU, unevictable, or PG_referenced, all
+ * the refs bits and PG_workingset set.  False for a dropbehind folio.
  */
-void folio_mark_accessed(struct folio *folio)
+bool folio_mark_accessed(struct folio *folio)
 {
+	bool hot = false;
+
 	if (folio_test_dropbehind(folio))
-		return;
-	if (lru_gen_enabled()) {
-		lru_gen_inc_refs(folio);
-		return;
-	}
+		return false;
+	if (lru_gen_enabled())
+		return lru_gen_inc_refs(folio);
 
 	if (!folio_test_referenced(folio)) {
 		folio_set_referenced(folio);
@@ -437,6 +449,7 @@ void folio_mark_accessed(struct folio *folio)
 		 * this list is never rotated or maintained, so marking an
 		 * unevictable page accessed has no effect.
 		 */
+		hot = true;
 	} else if (!folio_test_active(folio)) {
 		/*
 		 * If the folio is on the LRU, queue it for activation via
@@ -450,9 +463,14 @@ void folio_mark_accessed(struct folio *folio)
 			__lru_cache_activate_folio(folio);
 		folio_clear_referenced(folio);
 		workingset_activation(folio);
+	} else {
+		hot = true;
 	}
-	if (folio_test_idle(folio))
+	if (folio_test_idle(folio)) {
 		folio_clear_idle(folio);
+		hot = false;
+	}
+	return hot;
 }
 EXPORT_SYMBOL(folio_mark_accessed);
 
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 1e5d68acf62a5..25082949a4cae 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -3743,6 +3743,7 @@ static void __split_folio_to_order(struct folio *folio, int old_order,
 #endif
 				 (1L << PG_dirty) |
 				 (1L << PG_dropbehind) |
+				 __PG_RCU_FREE |
 				 LRU_GEN_MASK | LRU_REFS_MASK));
 
 		new_folio->mapping = folio->mapping;
diff --git a/mm/internal.h b/mm/internal.h
index 38b1165212c94..ae9a4e511c827 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -1248,6 +1248,9 @@ struct migration_target_control {
  */
 size_t splice_folio_into_pipe(struct pipe_inode_info *pipe,
 			      struct folio *folio, loff_t fpos, size_t size);
+void filemap_read_accessed(struct folio *folio, loff_t pos, loff_t last_pos);
+size_t filemap_read_rcu(struct kiocb *iocb, struct iov_iter *iter,
+			loff_t last_pos);
 
 static inline bool vma_is_single_threaded_private(struct vm_area_struct *vma)
 {
diff --git a/mm/memory-failure.c b/mm/memory-failure.c
index a8b03e2920ba8..dc66cac05b1f5 100644
--- a/mm/memory-failure.c
+++ b/mm/memory-failure.c
@@ -211,6 +211,10 @@ static bool page_handle_poison(struct page *page, bool hugepage_or_freepage, boo
 			return false;
 	}
 
+	/* The free of a hwpoisoned page is not deferred: wait for readers */
+	if (release && folio_test_rcu_free(page_folio(page)))
+		synchronize_rcu();
+
 	SetPageHWPoison(page);
 	if (release)
 		put_page(page);
@@ -1561,6 +1565,21 @@ static int get_hwpoison_page(struct page *p, unsigned long flags)
 	return ret;
 }
 
+/* A page waiting for a grace period is not in the buddy yet: free it */
+static void mf_free_rcu_pages(struct page *p)
+{
+	if (!IS_ENABLED(CONFIG_PAGE_RCU_FREE))
+		return;
+	if (page_count(p))
+		return;
+	if (!zone_page_state_snapshot(page_zone(p), NR_FREE_RCU_PAGES))
+		return;
+	if (is_free_buddy_page(p))
+		return;
+
+	rcu_free_sync();
+}
+
 /*
  * The caller must guarantee the folio isn't large folio, except hugetlb.
  * try_to_unmap() can't handle it.
@@ -2436,6 +2455,7 @@ int memory_failure(unsigned long pfn, int flags)
 	if (res != -ENOENT)
 		goto unlock_mutex;
 
+	mf_free_rcu_pages(p);
 	if (TestSetPageHWPoison(p)) {
 		res = -EHWPOISON;
 		if (flags & MF_ACTION_REQUIRED)
@@ -2608,6 +2628,13 @@ int memory_failure(unsigned long pfn, int flags)
 
 identify_page_state:
 	res = identify_page_state(pfn, p, page_flags);
+	/*
+	 * The caller may unmap the page once this returns 0.  Lockless
+	 * readers that started before the page got PG_hwpoison, or before a
+	 * split left it in a folio of its own, may still be copying it.
+	 */
+	if (!res && folio_test_rcu_free(folio))
+		synchronize_rcu();
 	mutex_unlock(&mf_mutex);
 	return res;
 unlock_page:
@@ -2999,6 +3026,7 @@ int soft_offline_page(unsigned long pfn, int flags)
 
 retry:
 	get_online_mems();
+	mf_free_rcu_pages(page);
 	ret = get_hwpoison_page(page, flags | MF_SOFT_OFFLINE);
 	put_online_mems();
 
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 226ab9cb078ad..a8f0c95627ee1 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -1967,6 +1967,7 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,
 		.nid = NUMA_NO_NODE,
 	};
 	unsigned long flags;
+	bool migrated;
 	char *reason;
 	int ret;
 	unsigned long normal_pages = 0;
@@ -2050,6 +2051,7 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,
 
 	do {
 		pfn = start_pfn;
+		migrated = false;
 		do {
 			/*
 			 * Historically we always checked for any signal and
@@ -2071,6 +2073,7 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,
 				 * out
 				 */
 				do_migrate_range(pfn, end_pfn);
+				migrated = true;
 			}
 		} while (!ret);
 
@@ -2092,6 +2095,10 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,
 
 		ret = test_pages_isolated(start_pfn, end_pfn,
 					  PB_ISOLATE_MODE_MEM_OFFLINE);
+		/* What this pass migrated can wait for a grace period */
+		if (ret && migrated &&
+		    zone_page_state_snapshot(zone, NR_FREE_RCU_PAGES))
+			rcu_free_sync();
 
 	} while (ret);
 
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 12fac9084c483..e63628e64cbba 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1313,6 +1313,199 @@ static inline void pgalloc_tag_sub_pages(struct alloc_tag *tag, unsigned int nr)
 
 #endif /* CONFIG_MEM_ALLOC_PROFILING */
 
+#define RCU_FREE_EXPEDITE	(SZ_4M >> PAGE_SHIFT)
+
+/*
+ * Each CPU has at most one grace period in flight (@queued), covering
+ * @batch.  Folios freed meanwhile wait on @pending for the next one.
+ */
+struct rcu_free_pcpu {
+	spinlock_t lock;
+	bool queued;
+	struct list_head pending;
+	struct list_head batch;
+	unsigned long nr_pending;
+	struct rcu_work rwork;
+	struct work_struct expedite;
+};
+
+static DEFINE_PER_CPU(struct rcu_free_pcpu, rcu_free_pcpu);
+
+static struct workqueue_struct *rcu_free_wq __ro_after_init;
+
+static void rcu_free_start(struct rcu_free_pcpu *r)
+{
+	lockdep_assert_held(&r->lock);
+
+	if (r->queued || list_empty(&r->pending))
+		return;
+
+	list_splice_init(&r->pending, &r->batch);
+	r->nr_pending = 0;
+	r->queued = true;
+	WARN_ON_ONCE(!queue_rcu_work(rcu_free_wq, &r->rwork));
+}
+
+static void rcu_free_take(struct rcu_free_pcpu *r, struct list_head *list)
+{
+	lockdep_assert_held(&r->lock);
+
+	list_splice_tail_init(&r->batch, list);
+	list_splice_tail_init(&r->pending, list);
+	r->nr_pending = 0;
+}
+
+static unsigned long rcu_free_pages(struct list_head *list)
+{
+	struct folio_batch fbatch;
+	struct folio *folio, *next;
+	unsigned long nr = 0;
+
+	folio_batch_init(&fbatch);
+	list_for_each_entry_safe(folio, next, list, lru) {
+		nr += folio_nr_pages(folio);
+		mod_zone_page_state(folio_zone(folio), NR_FREE_RCU_PAGES,
+				    -folio_nr_pages(folio));
+		if (!folio_batch_add(&fbatch, folio)) {
+			free_unref_folios(&fbatch);
+			cond_resched_tasks_rcu_qs();
+		}
+	}
+	if (folio_batch_count(&fbatch))
+		free_unref_folios(&fbatch);
+	return nr;
+}
+
+/* Free what rcu_free_take() took: the grace period starts after that */
+static unsigned long rcu_free_flush(struct list_head *list)
+{
+	if (list_empty(list))
+		return 0;
+
+	synchronize_rcu_expedited();
+	return rcu_free_pages(list);
+}
+
+static void rcu_free_workfn(struct work_struct *work)
+{
+	struct rcu_free_pcpu *r = container_of(to_rcu_work(work),
+					       struct rcu_free_pcpu, rwork);
+	LIST_HEAD(done);
+
+	spin_lock_irq(&r->lock);
+	list_splice_init(&r->batch, &done);
+	r->queued = false;
+	rcu_free_start(r);
+	spin_unlock_irq(&r->lock);
+
+	rcu_free_pages(&done);
+}
+
+static void rcu_free_expedite_workfn(struct work_struct *work)
+{
+	struct rcu_free_pcpu *r = container_of(work, struct rcu_free_pcpu,
+					       expedite);
+	LIST_HEAD(list);
+
+	spin_lock_irq(&r->lock);
+	rcu_free_take(r, &list);
+	spin_unlock_irq(&r->lock);
+
+	if (rcu_free_flush(&list))
+		count_vm_event(PGFREE_RCU_EXPEDITE);
+}
+
+static void rcu_free_defer(struct folio *folio)
+{
+	struct rcu_free_pcpu *r = raw_cpu_ptr(&rcu_free_pcpu);
+	struct zone *zone = folio_zone(folio);
+	unsigned long free, waiting;
+	unsigned long flags;
+
+	spin_lock_irqsave(&r->lock, flags);
+	__mod_zone_page_state(zone, NR_FREE_RCU_PAGES, folio_nr_pages(folio));
+	list_add_tail(&folio->lru, &r->pending);
+	r->nr_pending += folio_nr_pages(folio);
+	rcu_free_start(r);
+	spin_unlock_irqrestore(&r->lock, flags);
+
+	/* Unlocked: a stale count delays the expedite by one free */
+	if (data_race(r->nr_pending) <= RCU_FREE_EXPEDITE ||
+	    work_pending(&r->expedite))
+		return;
+
+	/*
+	 * An expedited grace period interrupts every CPU: only while the
+	 * zone is short of memory, or while more than an eighth of what it
+	 * has free waits here.
+	 */
+	free = zone_page_state(zone, NR_FREE_PAGES);
+	waiting = zone_page_state(zone, NR_FREE_RCU_PAGES);
+	if (free < high_wmark_pages(zone) || waiting > free / 8)
+		queue_work(rcu_free_wq, &r->expedite);
+}
+
+/*
+ * Free what every CPU's lists hold.  A concurrent drain holds the lock
+ * until it has freed what it took.
+ */
+static void rcu_free_drain(void)
+{
+	static DEFINE_MUTEX(lock);
+	LIST_HEAD(list);
+	int cpu;
+
+	mutex_lock(&lock);
+	for_each_possible_cpu(cpu) {
+		struct rcu_free_pcpu *r = per_cpu_ptr(&rcu_free_pcpu, cpu);
+
+		spin_lock_irq(&r->lock);
+		rcu_free_take(r, &list);
+		spin_unlock_irq(&r->lock);
+	}
+	if (rcu_free_flush(&list))
+		count_vm_event(PGFREE_RCU_DRAIN);
+	mutex_unlock(&lock);
+}
+
+/* Free every waiting folio, including those a work took off the lists */
+void rcu_free_sync(void)
+{
+	int cpu;
+
+	if (!IS_ENABLED(CONFIG_PAGE_RCU_FREE))
+		return;
+
+	rcu_free_drain();
+	for_each_possible_cpu(cpu) {
+		struct rcu_free_pcpu *r = per_cpu_ptr(&rcu_free_pcpu, cpu);
+
+		flush_work(&r->rwork.work);
+		flush_work(&r->expedite);
+	}
+}
+
+static void __init rcu_free_init(void)
+{
+	int cpu;
+
+	if (!IS_ENABLED(CONFIG_PAGE_RCU_FREE))
+		return;
+
+	rcu_free_wq = alloc_workqueue("rcu_free",
+				      WQ_MEM_RECLAIM | WQ_PERCPU, 0);
+
+	for_each_possible_cpu(cpu) {
+		struct rcu_free_pcpu *r = per_cpu_ptr(&rcu_free_pcpu, cpu);
+
+		spin_lock_init(&r->lock);
+		INIT_LIST_HEAD(&r->pending);
+		INIT_LIST_HEAD(&r->batch);
+		INIT_RCU_WORK(&r->rwork, rcu_free_workfn);
+		INIT_WORK(&r->expedite, rcu_free_expedite_workfn);
+	}
+}
+
 static __always_inline bool __free_pages_prepare(struct page *page,
 		unsigned int order, fpi_t fpi_flags)
 {
@@ -1327,6 +1520,19 @@ static __always_inline bool __free_pages_prepare(struct page *page,
 
 	VM_BUG_ON_PAGE(PageTail(page), page);
 
+	/*
+	 * Not for a hwpoisoned order-0 page: it never reaches the allocator,
+	 * and page_handle_poison() takes a reference right after this free.
+	 */
+	if (unlikely(folio_test_rcu_free(folio)) &&
+	    !(PageHWPoison(page) && !order)) {
+		VM_WARN_ON_ONCE(fpi_flags & FPI_NOLOCK);
+		__folio_clear_rcu_free(folio);
+		count_vm_events(PGFREE_RCU, 1 << order);
+		rcu_free_defer(folio);
+		return false;
+	}
+
 	trace_mm_page_free(page, order);
 	kmsan_free_page(page, order);
 
@@ -4626,6 +4832,52 @@ bool gfp_pfmemalloc_allowed(gfp_t gfp_mask)
 	return !!__gfp_pfmemalloc_flags(gfp_mask);
 }
 
+/*
+ * Reclaim is about to give up: if the pages waiting for a grace period
+ * would bring an eligible zone above its min watermark, free them and
+ * retry.  Not for a dying task, which frees memory by exiting.
+ */
+static bool rcu_free_wait(gfp_t gfp_mask, unsigned int order,
+			  struct alloc_context *ac, int alloc_flags,
+			  int *rcu_free_waits)
+{
+	struct zone *zone;
+	struct zoneref *z;
+
+	if (!IS_ENABLED(CONFIG_PAGE_RCU_FREE))
+		return false;
+	if (tsk_is_oom_victim(current) || fatal_signal_pending(current))
+		return false;
+	if (*rcu_free_waits >= MAX_RECLAIM_RETRIES ||
+	    order > PAGE_ALLOC_COSTLY_ORDER)
+		return false;
+
+	for_each_zone_zonelist_nodemask(zone, z, ac->zonelist,
+				ac->highest_zoneidx, ac->nodemask) {
+		unsigned long waiting, available;
+
+		if (cpusets_enabled() && (alloc_flags & ALLOC_CPUSET) &&
+		    !__cpuset_zone_allowed(zone, gfp_mask))
+			continue;
+
+		waiting = zone_page_state_snapshot(zone, NR_FREE_RCU_PAGES);
+		if (!waiting)
+			continue;
+
+		available = waiting +
+			    zone_page_state_snapshot(zone, NR_FREE_PAGES);
+		if (__zone_watermark_ok(zone, order, min_wmark_pages(zone),
+					ac->highest_zoneidx, alloc_flags,
+					available)) {
+			(*rcu_free_waits)++;
+			rcu_free_drain();
+			return true;
+		}
+	}
+
+	return false;
+}
+
 /*
  * Checks whether it makes sense to retry the reclaim to make a forward progress
  * for the given allocation request.
@@ -4639,7 +4891,8 @@ bool gfp_pfmemalloc_allowed(gfp_t gfp_mask)
 static inline bool
 should_reclaim_retry(gfp_t gfp_mask, unsigned order,
 		     struct alloc_context *ac, int alloc_flags,
-		     bool did_some_progress, int *no_progress_loops)
+		     bool did_some_progress, int *no_progress_loops,
+		     int *rcu_free_waits)
 {
 	struct zone *zone;
 	struct zoneref *z;
@@ -4650,10 +4903,12 @@ should_reclaim_retry(gfp_t gfp_mask, unsigned order,
 	 * their order will become available due to high fragmentation so
 	 * always increment the no progress counter for them
 	 */
-	if (did_some_progress && order <= PAGE_ALLOC_COSTLY_ORDER)
+	if (did_some_progress && order <= PAGE_ALLOC_COSTLY_ORDER) {
 		*no_progress_loops = 0;
-	else
+		*rcu_free_waits = 0;
+	} else {
 		(*no_progress_loops)++;
+	}
 
 	if (*no_progress_loops > MAX_RECLAIM_RETRIES)
 		goto out;
@@ -4706,6 +4961,10 @@ should_reclaim_retry(gfp_t gfp_mask, unsigned order,
 	else
 		cond_resched();
 out:
+	if (!ret && rcu_free_wait(gfp_mask, order, ac, alloc_flags,
+				  rcu_free_waits))
+		return true;
+
 	/* Before OOM, exhaust highatomic_reserve */
 	if (!ret)
 		return unreserve_highatomic_pageblock(ac, true);
@@ -4795,6 +5054,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	enum compact_result compact_result;
 	int compaction_retries;
 	int no_progress_loops;
+	int rcu_free_waits;
 	unsigned int cpuset_mems_cookie;
 	unsigned int zonelist_iter_cookie;
 	int reserve_flags;
@@ -4819,6 +5079,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 restart:
 	compaction_retries = 0;
 	no_progress_loops = 0;
+	rcu_free_waits = 0;
 	compact_result = COMPACT_SKIPPED;
 	compact_priority = DEF_COMPACT_PRIORITY;
 	cpuset_mems_cookie = read_mems_allowed_begin();
@@ -4997,7 +5258,8 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 		goto restart;
 
 	if (should_reclaim_retry(gfp_mask, order, ac, alloc_flags,
-				 did_some_progress > 0, &no_progress_loops))
+				 did_some_progress > 0, &no_progress_loops,
+				 &rcu_free_waits))
 		goto retry;
 
 	/*
@@ -5040,6 +5302,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	/* Retry as long as the OOM killer is making progress */
 	if (did_some_progress) {
 		no_progress_loops = 0;
+		rcu_free_waits = 0;
 		goto retry;
 	}
 
@@ -6401,6 +6664,8 @@ void __init setup_per_cpu_pageset(void)
 	for_each_online_pgdat(pgdat)
 		pgdat->per_cpu_nodestats =
 			alloc_percpu(struct per_cpu_nodestat);
+
+	rcu_free_init();
 }
 
 __meminit void zone_pcp_init(struct zone *zone)
@@ -7384,7 +7649,16 @@ int alloc_contig_frozen_range_noprof(unsigned long start, unsigned long end,
 	outer_start = find_large_buddy(start);
 
 	/* Make sure the range is really isolated. */
-	if (test_pages_isolated(outer_start, end, mode)) {
+	ret = test_pages_isolated(outer_start, end, mode);
+	if (ret && zone_page_state_snapshot(cc.zone, NR_FREE_RCU_PAGES)) {
+		rcu_free_sync();
+
+		/* A freed folio can merge with a free buddy below start */
+		outer_start = find_large_buddy(start);
+		ret = test_pages_isolated(outer_start, end, mode);
+	}
+
+	if (ret) {
 		ret = -EBUSY;
 		goto done;
 	}
diff --git a/mm/page_alloc.h b/mm/page_alloc.h
index b9259deddb59d..f9e200581e2f5 100644
--- a/mm/page_alloc.h
+++ b/mm/page_alloc.h
@@ -235,6 +235,7 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order, int nid,
 	alloc_hooks(__alloc_frozen_pages_noprof(__VA_ARGS__))
 void free_frozen_pages(struct page *page, unsigned int order);
 void free_unref_folios(struct folio_batch *fbatch);
+void rcu_free_sync(void);
 
 #ifdef CONFIG_NUMA
 struct page *alloc_frozen_pages_noprof(gfp_t, unsigned int order);
diff --git a/mm/shmem.c b/mm/shmem.c
index 848316eaa7f4f..e6c18e745f816 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -3309,10 +3309,18 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
 	struct file *file = iocb->ki_filp;
 	struct inode *inode = file_inode(file);
 	struct address_space *mapping = inode->i_mapping;
+	loff_t last_pos = file->f_ra.prev_pos;
 	pgoff_t index;
 	unsigned long offset;
 	int error = 0;
-	ssize_t retval = 0;
+	ssize_t retval;
+
+	retval = filemap_read_rcu(iocb, to, last_pos);
+	if (retval) {
+		last_pos = iocb->ki_pos;
+		if (!iov_iter_count(to))
+			goto out;
+	}
 
 	for (;;) {
 		struct folio *folio = NULL;
@@ -3378,11 +3386,7 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
 					flush_dcache_page(page);
 			}
 
-			/*
-			 * Mark the folio accessed if we read the beginning.
-			 */
-			if (!offset)
-				folio_mark_accessed(folio);
+			filemap_read_accessed(folio, iocb->ki_pos, last_pos);
 			/*
 			 * Ok, we have the page, and it's up-to-date, so
 			 * now we can copy it to user space...
@@ -3410,6 +3414,7 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
 
 		retval += ret;
 		iocb->ki_pos += ret;
+		last_pos = iocb->ki_pos;
 
 		if (!iov_iter_count(to))
 			break;
@@ -3419,8 +3424,11 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
 		}
 		cond_resched();
 	}
-
+out:
 	file_accessed(file);
+	/* Readers sharing the file would bounce the line for nothing */
+	if (file->f_ra.prev_pos != last_pos)
+		file->f_ra.prev_pos = last_pos;
 	return retval ? retval : error;
 }
 
diff --git a/mm/show_mem.c b/mm/show_mem.c
index b938cbcd774a1..0d3bddbda108d 100644
--- a/mm/show_mem.c
+++ b/mm/show_mem.c
@@ -204,7 +204,7 @@ static void show_free_areas(unsigned int filter, const nodemask_t *nodemask,
 		" mapped:%lu shmem:%lu pagetables:%lu\n"
 		" sec_pagetables:%lu bounce:%lu\n"
 		" kernel_misc_reclaimable:%lu\n"
-		" free:%lu free_pcp:%lu free_cma:%lu\n",
+		" free:%lu free_pcp:%lu free_cma:%lu free_rcu:%lu\n",
 		global_node_page_state(NR_ACTIVE_ANON),
 		global_node_page_state(NR_INACTIVE_ANON),
 		global_node_page_state(NR_ISOLATED_ANON),
@@ -224,7 +224,8 @@ static void show_free_areas(unsigned int filter, const nodemask_t *nodemask,
 		global_node_page_state(NR_KERNEL_MISC_RECLAIMABLE),
 		global_zone_page_state(NR_FREE_PAGES),
 		free_pcp,
-		global_zone_page_state(NR_FREE_CMA_PAGES));
+		global_zone_page_state(NR_FREE_CMA_PAGES),
+		global_zone_page_state(NR_FREE_RCU_PAGES));
 
 	for_each_online_pgdat(pgdat) {
 		if (show_mem_node_skip(filter, pgdat->node_id, nodemask))
@@ -325,6 +326,7 @@ static void show_free_areas(unsigned int filter, const nodemask_t *nodemask,
 			" free_pcp:%lukB"
 			" local_pcp:%lukB"
 			" free_cma:%lukB"
+			" free_rcu:%lukB"
 			"\n",
 			zone->name,
 			K(zone_page_state(zone, NR_FREE_PAGES)),
@@ -351,7 +353,8 @@ static void show_free_areas(unsigned int filter, const nodemask_t *nodemask,
 			0UL,
 			K(free_pcp),
 			K((unsigned long)this_cpu_read(zone->per_cpu_pageset->count)),
-			K(zone_page_state(zone, NR_FREE_CMA_PAGES)));
+			K(zone_page_state(zone, NR_FREE_CMA_PAGES)),
+			K(zone_page_state(zone, NR_FREE_RCU_PAGES)));
 		printk("lowmem_reserve[]:");
 		for (i = 0; i < MAX_NR_ZONES; i++)
 			printk(KERN_CONT " %ld", zone->lowmem_reserve[i]);
diff --git a/mm/vmstat.c b/mm/vmstat.c
index cb57714539fb5..c27caad017e35 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -1212,6 +1212,7 @@ const char * const vmstat_text[] = {
 	[I(NR_ZSPAGES)]				= "nr_zspages",
 #endif
 	[I(NR_FREE_CMA_PAGES)]			= "nr_free_cma",
+	[I(NR_FREE_RCU_PAGES)]			= "nr_free_rcu",
 #ifdef CONFIG_UNACCEPTED_MEMORY
 	[I(NR_UNACCEPTED)]			= "nr_unaccepted",
 #endif
@@ -1343,6 +1344,10 @@ const char * const vmstat_text[] = {
 	[I(PGACTIVATE)]				= "pgactivate",
 	[I(PGDEACTIVATE)]			= "pgdeactivate",
 	[I(PGLAZYFREE)]				= "pglazyfree",
+	[I(PGFREE_RCU)]				= "pgfree_rcu",
+	[I(PGFREE_RCU_EXPEDITE)]		= "pgfree_rcu_expedite",
+	[I(PGFREE_RCU_DRAIN)]			= "pgfree_rcu_drain",
+	[I(PGFREE_RCU_MARK)]			= "pgfree_rcu_mark",
 
 	[I(PGFAULT)]				= "pgfault",
 	[I(PGMAJFAULT)]				= "pgmajfault",
@@ -2022,6 +2027,7 @@ static int vmstat_refresh(const struct ctl_table *table, int write,
 		switch (i) {
 		case NR_ZONE_WRITE_PENDING:
 		case NR_FREE_CMA_PAGES:
+		case NR_FREE_RCU_PAGES:
 			continue;
 		}
 		val = atomic_long_read(&vm_zone_stat[i]);

							input: 20034
							output: 30
							thoughts: 871

**KMSAN Fuzzing for RCU-Free Folios: Initial Assessment**

Okay, so the task is to determine if a dedicated KMSAN fuzzing session is warranted for this patch series introducing lockless page cache reads using RCU. First, I need to fully grasp the patch's changes. It looks like it's a fairly invasive change touching several core memory management subsystems.

Here's the gist: we're adding RCU-based freeing for folios, enabled by `CONFIG_PAGE_RCU_FREE`. The changes propagate through various files: `fuse/dev.c` (specifically `fuse_try_move_folio` which checks the `rcu_free` flag), `splice.c` (removing `page_cache_pipe_buf_try_steal`), and several header files. We're seeing new `NR_FREE_RCU_PAGES`, `PG_rcu_free`, `AS_RCU_READ`, and related vm events being introduced. The `mm/filemap.c` file is heavily modified to handle RCU-free folios during eviction and deletion, including new functions like `filemap_read_rcu_batch` and changes to `filemap_read` to utilize `filemap_read_rcu`. Other areas like `mm/folio.c`, `mm/huge_memory.c`, `mm/memory-failure.c`, and `mm/memory_hotplug.c` are updated to account for this RCU-based freeing.

However, the critical point, and the one that sparks my immediate concern, is in `mm/page_alloc.c`, specifically the function `__free_pages_prepare`. Here's where things get interesting. The code includes a check for `folio_test_rcu_free(folio)`. If that flag is set, the freeing is *deferred* to RCU via `rcu_free_defer()`. Critically, the function returns `false` *before* the call to `kmsan_free_page()`! This means KMSAN won't see the initial "free" of the page. Then, at a later time, when RCU callback runs (`rcu_free_pages`), it will call `free_unref_folios(&fbatch);`. *That's* where the actual free happens, but it's now decoupled from the original "free" location that KMSAN usually tracks. My immediate thought is that this introduces a significant potential for race conditions or use-after-free scenarios that KMSAN might not catch in its standard execution. I need to dive into `free_unref_folios` to understand exactly what happens there to be certain, but the deferral behavior is screaming "possible KMSAN blind spot." We need to see what `free_unref_folios` does in order to be confident.

{
  "Expression": "void free_unref_folios",
  "PathPrefixes": [
    "mm/"
  ]
}

{
  "Output": "mm/page_alloc.c=3208=void free_frozen_pages_nolock(struct page *page, unsigned int order)\n--\nmm/page_alloc.c-3215- */\nmm/page_alloc.c:3216:void free_unref_folios(struct folio_batch *folios)\nmm/page_alloc.c-3217-{\n--\nmm/page_alloc.h=236=void free_frozen_pages(struct page *page, unsigned int order);\nmm/page_alloc.h:237:void free_unref_folios(struct folio_batch *fbatch);\nmm/page_alloc.h-238-void rcu_free_sync(void);\n"
}

							input: 21743
							output: 36
							thoughts: 22

Here's a breakdown of my thought process:

**Diving into `free_unref_folios`**

Okay, so I'm focusing in on `free_unref_folios` within `mm/page_alloc.c`. This suggests we're dealing with the core mechanics of memory management, specifically, releasing folio objects that are no longer actively referenced. I need to understand this function. What's the context? What calls it? What's its relationship to the broader folio lifecycle, and, by extension, the page and slab allocators? How does it handle potential race conditions? I'm assuming it has to be carefully managed because we are releasing memory objects. I am expecting this to be a key area for performance optimization and debugging memory leaks or fragmentation issues. I need to understand this at a low level.

{
  "File": "mm/page_alloc.c",
  "FirstLine": 3210,
  "LineCount": 50
}

{
  "Contents": "3210:\t\t__free_frozen_pages(page, order, FPI_NOLOCK);\n3211:\t}\n3212:\t\n3213:\t/*\n3214:\t * Free a batch of folios\n3215:\t */\n3216:\tvoid free_unref_folios(struct folio_batch *folios)\n3217:\t{\n3218:\t\tstruct per_cpu_pages *pcp = NULL;\n3219:\t\tstruct zone *locked_zone = NULL;\n3220:\t\tint i, j;\n3221:\t\n3222:\t\t/* Prepare folios for freeing */\n3223:\t\tfor (i = 0, j = 0; i \u003c folios-\u003enr; i++) {\n3224:\t\t\tstruct folio *folio = folios-\u003efolios[i];\n3225:\t\t\tunsigned long pfn = folio_pfn(folio);\n3226:\t\t\tunsigned int order = folio_order(folio);\n3227:\t\n3228:\t\t\tif (!__free_pages_prepare(\u0026folio-\u003epage, order, FPI_NONE))\n3229:\t\t\t\tcontinue;\n3230:\t\t\t/*\n3231:\t\t\t * Free orders not handled on the PCP directly to the\n3232:\t\t\t * allocator.\n3233:\t\t\t */\n3234:\t\t\tif (!pcp_allowed_order(order)) {\n3235:\t\t\t\tfree_one_page(folio_zone(folio), \u0026folio-\u003epage,\n3236:\t\t\t\t\t      pfn, order, FPI_NONE);\n3237:\t\t\t\tcontinue;\n3238:\t\t\t}\n3239:\t\t\tfolio-\u003eprivate = (void *)(unsigned long)order;\n3240:\t\t\tif (j != i)\n3241:\t\t\t\tfolios-\u003efolios[j] = folio;\n3242:\t\t\tj++;\n3243:\t\t}\n3244:\t\tfolios-\u003enr = j;\n3245:\t\n3246:\t\tfor (i = 0; i \u003c folios-\u003enr; i++) {\n3247:\t\t\tstruct folio *folio = folios-\u003efolios[i];\n3248:\t\t\tstruct zone *zone = folio_zone(folio);\n3249:\t\t\tunsigned long pfn = folio_pfn(folio);\n3250:\t\t\tunsigned int order = (unsigned long)folio-\u003eprivate;\n3251:\t\t\tint migratetype;\n3252:\t\n3253:\t\t\tfolio-\u003eprivate = NULL;\n3254:\t\t\tmigratetype = get_pfnblock_migratetype(\u0026folio-\u003epage, pfn);\n3255:\t\n3256:\t\t\t/* Different zone requires a different pcp lock */\n3257:\t\t\tif (zone != locked_zone ||\n3258:\t\t\t    is_migrate_isolate(migratetype)) {\n3259:\t\t\t\tif (pcp) {\n"
}

							input: 22720
							output: 28
							thoughts: 2844

**KMSAN Assessment of Lockless Page Cache Reads**

Okay, so `free_unref_folios` calls `__free_pages_prepare`, and that looks like the KMSAN tracking point. Since `__folio_clear_rcu_free` happens earlier, `folio_test_rcu_free` is false the second time around, and `kmsan_free_page` is called right when the page actually gets freed. That's good; it preserves the KMSAN tracking.

Now, let's refresh our memory on the KMSAN vs. KASAN rules: KMSAN is for *uninitialized* memory reads, while KASAN and other tools handle things like out-of-bounds access and use-after-free. The critical thing is, if KASAN (or something else) catches it, KMSAN isn't needed. We only use KMSAN for uninitialized memory risks. We must examine the surrounding code carefully!

This patch implements lockless page cache reads using RCU. Users read from the page cache via `filemap_read` and `shmem_file_read_iter`, which calls `filemap_read_rcu`. This looks up folios in the xarray under RCU read lock and then relies on `PG_rcu_free` and a grace period. `filemap_read_rcu_copy` then uses `kmap_local_page` and `_copy_to_iter`.

The crucial question is whether `_copy_to_iter` could copy uninitialized memory. Let's trace it. It checks for `!folio_test_uptodate(folio)` and ensures that, *before* marking pages as "hot". The `pos >= *isize` check in `filemap_read_rcu_lookup` limits the copy size, ensuring it doesn't exceed the file size. Also, the `filemap_read_rcu_size` limits the `bytes` copied to the smaller value of `isize` or the current page.

So, *regular* `filemap_read` copies initialized memory that either comes from disk or is user written. Now, what if the file is truncated or hole punched? The folio is removed via `page_cache_delete` or `remove_mapping`. If a page is reused after freeing, `__free_pages_prepare` ensures that it's deferred until after the RCU grace period (so the folio is *not* freed to buddy or PCP). This prevents it from being reallocated for something else while someone is reading it! That’s smart.

Now, what about use-after-free or races with truncate? Any such issue would be a use-after-free/data race/synchronization bug caught by KASAN. And, if the folio is in the RCU grace period, `kasan_free_pages` and `kmsan_free_page` haven't run on it yet.

So, does this patch introduce *any* uninitialized memory? Let's check:
1. `struct rcu_free_pcpu`: Static per-cpu struct, initialized at boot. BSS (zeroed).
2. `struct folio_batch fbatch`, `XA_STATE(xas, &mapping->i_pages, 0)`, `LIST_HEAD(done)`, `LIST_HEAD(list)` are also zeroed or initialized.

Are kernel structures being sent to userspace? No, not really. The only user copy happens with `_copy_to_iter`, copying file data in `filemap_read_rcu_copy` or `copy_folio_to_iter`, exactly as it already did in the slow path of `filemap_read`.

Could `filemap_read_rcu_copy` read past the end of a page or uninitialized bytes? It uses `kmap_local_page` and calls `_copy_to_iter` and a loop. It then ensures that `index - folio->index` is valid (or OOB). The `bytes` variable is calculated by min of `isize - pos` (or an equivalent). Page cache pages are required to be uptodate. Uptodate page cache folios *always* have initialized contents up to `i_size`. And filesystems are good about zeroing out pages on write/read.

Wait, does KMSAN track page cache pages? I need to search the kernel for KMSAN hooks in the page cache or page_alloc to be sure, but I don't see any likely candidates in this patch!

{
  "Expression": "kmsan_.*page",
  "PathPrefixes": [
    "mm/"
  ]
}

{
  "Output": "mm/kmsan/core.c=191=void kmsan_internal_set_shadow_origin(void *addr, size_t size, int b,\n--\nmm/kmsan/core.c-238-\nmm/kmsan/core.c:239:struct page *kmsan_vmalloc_to_page_or_null(void *vaddr)\nmm/kmsan/core.c-240-{\n--\nmm/kmsan/hooks.c=110=void kmsan_kfree_large(const void *ptr)\n--\nmm/kmsan/hooks.c-118-\tKMSAN_WARN_ON(ptr != page_address(page));\nmm/kmsan/hooks.c:119:\tkmsan_internal_poison_memory((void *)ptr, page_size(page),\nmm/kmsan/hooks.c-120-\t\t\t\t     GFP_KERNEL \u0026 ~(__GFP_RECLAIM),\n--\nmm/kmsan/hooks.c=137=void kmsan_vunmap_range_noflush(unsigned long start, unsigned long end)\n--\nmm/kmsan/hooks.c-149- */\nmm/kmsan/hooks.c:150:int kmsan_ioremap_page_range(unsigned long start, unsigned long end,\nmm/kmsan/hooks.c-151-\t\t\t     phys_addr_t phys_addr, pgprot_t prot,\n--\nmm/kmsan/hooks.c-219-\nmm/kmsan/hooks.c:220:void kmsan_iounmap_page_range(unsigned long start, unsigned long end)\nmm/kmsan/hooks.c-221-{\n--\nmm/kmsan/hooks.c-234-\t     i++, v_shadow += PAGE_SIZE, v_origin += PAGE_SIZE) {\nmm/kmsan/hooks.c:235:\t\tshadow = kmsan_vmalloc_to_page_or_null((void *)v_shadow);\nmm/kmsan/hooks.c:236:\t\torigin = kmsan_vmalloc_to_page_or_null((void *)v_origin);\nmm/kmsan/hooks.c-237-\t\t__vunmap_range_noflush(v_shadow, vmalloc_shadow(end));\n--\nmm/kmsan/hooks.c=316=EXPORT_SYMBOL_GPL(kmsan_handle_urb);\nmm/kmsan/hooks.c-317-\nmm/kmsan/hooks.c:318:static void kmsan_handle_dma_page(const void *addr, size_t size,\nmm/kmsan/hooks.c-319-\t\t\t\t  enum dma_data_direction dir)\n--\nmm/kmsan/hooks.c=342=void kmsan_handle_dma(phys_addr_t phys, size_t size,\n--\nmm/kmsan/hooks.c-358-\t\tto_go = min(PAGE_SIZE - page_offset, (u64)size);\nmm/kmsan/hooks.c:359:\t\tkmsan_handle_dma_page((void *)addr, to_go, dir);\nmm/kmsan/hooks.c-360-\t\taddr += to_go;\n--\nmm/kmsan/init.c=101=static struct metadata_page_pair held_back[NR_PAGE_ORDERS] __initdata;\n--\nmm/kmsan/init.c-106- * We store the pointers to the returned blocks of pages in held_back[] grouped\nmm/kmsan/init.c:107: * by their order: when kmsan_memblock_free_pages() is called for the first\nmm/kmsan/init.c-108- * time with a certain order, it is reserved as a shadow block, for the second\n--\nmm/kmsan/init.c-115- */\nmm/kmsan/init.c:116:bool kmsan_memblock_free_pages(struct page *page, unsigned int order)\nmm/kmsan/init.c-117-{\n--\nmm/kmsan/init.c-129-\torigin = held_back[order].origin;\nmm/kmsan/init.c:130:\tkmsan_setup_meta(page, shadow, origin, order);\nmm/kmsan/init.c-131-\n--\nmm/kmsan/init.c=168=static void do_collection(void)\n--\nmm/kmsan/init.c-175-\t\torigin = smallstack_pop(\u0026collect);\nmm/kmsan/init.c:176:\t\tkmsan_setup_meta(page, shadow, origin, collect.order);\nmm/kmsan/init.c-177-\t\t__free_pages_core(page, collect.order, MEMINIT_EARLY);\n--\nmm/kmsan/kmsan.h=165=void kmsan_internal_check_memory(void *addr, size_t size,\n--\nmm/kmsan/kmsan.h-167-\nmm/kmsan/kmsan.h:168:struct page *kmsan_vmalloc_to_page_or_null(void *vaddr);\nmm/kmsan/kmsan.h:169:void kmsan_setup_meta(struct page *page, struct page *shadow,\nmm/kmsan/kmsan.h-170-\t\t      struct page *origin, int order);\n--\nmm/kmsan/kmsan_test.c=316=static void test_init_kmsan_vmap_vunmap(struct kunit *test)\n--\nmm/kmsan/kmsan_test.c-330-\tfor (int i = 0; i \u003c npages; i++)\nmm/kmsan/kmsan_test.c:331:\t\tkmsan_check_memory(page_address(pages[i]), PAGE_SIZE);\nmm/kmsan/kmsan_test.c-332-\n--\nmm/kmsan/shadow.c=125=void *kmsan_get_metadata(void *address, bool is_origin)\n--\nmm/kmsan/shadow.c-151-\nmm/kmsan/shadow.c:152:void kmsan_copy_page_meta(struct page *dst, struct page *src)\nmm/kmsan/shadow.c-153-{\n--\nmm/kmsan/shadow.c-158-\tif (!src || !page_has_metadata(src)) {\nmm/kmsan/shadow.c:159:\t\tkmsan_internal_unpoison_memory(page_address(dst), PAGE_SIZE,\nmm/kmsan/shadow.c-160-\t\t\t\t\t       /*checked*/ false);\n--\nmm/kmsan/shadow.c-168-}\nmm/kmsan/shadow.c:169:EXPORT_SYMBOL(kmsan_copy_page_meta);\nmm/kmsan/shadow.c-170-\nmm/kmsan/shadow.c:171:void kmsan_alloc_page(struct page *page, unsigned int order, gfp_t flags)\nmm/kmsan/shadow.c-172-{\n--\nmm/kmsan/shadow.c-205-\nmm/kmsan/shadow.c:206:void kmsan_free_page(struct page *page, unsigned int order)\nmm/kmsan/shadow.c-207-{\n--\nmm/kmsan/shadow.c-210-\tkmsan_enter_runtime();\nmm/kmsan/shadow.c:211:\tkmsan_internal_poison_memory(page_address(page), PAGE_SIZE \u003c\u003c order,\nmm/kmsan/shadow.c-212-\t\t\t\t     GFP_KERNEL \u0026 ~(__GFP_RECLAIM),\n--\nmm/kmsan/shadow.c-216-\nmm/kmsan/shadow.c:217:int kmsan_vmap_pages_range_noflush(unsigned long start, unsigned long end,\nmm/kmsan/shadow.c-218-\t\t\t\t   pgprot_t prot, struct page **pages,\n--\nmm/kmsan/shadow.c=276=void __init kmsan_init_alloc_meta_for_range(void *start, void *end)\n--\nmm/kmsan/shadow.c-298-\nmm/kmsan/shadow.c:299:void kmsan_setup_meta(struct page *page, struct page *shadow,\nmm/kmsan/shadow.c-300-\t\t      struct page *origin, int order)\n--\nmm/memory.c=3985=static vm_fault_t wp_page_copy(struct vm_fault *vmf)\n--\nmm/memory.c-4029-\t\t}\nmm/memory.c:4030:\t\tkmsan_copy_page_meta(\u0026new_folio-\u003epage, vmf-\u003epage);\nmm/memory.c-4031-\t}\n--\nmm/mm_init.c=2409=void __init memblock_free_pages(unsigned long pfn, unsigned int order)\n--\nmm/mm_init.c-2419-\nmm/mm_init.c:2420:\tif (!kmsan_memblock_free_pages(page, order)) {\nmm/mm_init.c-2421-\t\t/* KMSAN will take care of these pages. */\n--\nmm/page_alloc.c=1509=static __always_inline bool __free_pages_prepare(struct page *page,\n--\nmm/page_alloc.c-1536-\ttrace_mm_page_free(page, order);\nmm/page_alloc.c:1537:\tkmsan_free_page(page, order);\nmm/page_alloc.c-1538-\n--\nmm/page_alloc.c=5606=static inline bool alloc_nolock_allowed(void)\n--\nmm/page_alloc.c-5631- *\nmm/page_alloc.c:5632: * Specify __GFP_ZERO to make sure that call to kmsan_alloc_page() below\nmm/page_alloc.c-5633- * is safe in any context. Also zeroing the page is mandatory for\n--\nmm/page_alloc.c=5646=struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order,\n--\nmm/page_alloc.c-5722-\ttrace_mm_page_alloc(page, order, alloc_gfp, ac.migratetype);\nmm/page_alloc.c:5723:\tkmsan_alloc_page(page, order, alloc_gfp);\nmm/page_alloc.c-5724-\n--\nmm/vmalloc.c=355=int vmap_page_range(unsigned long addr, unsigned long end,\n--\nmm/vmalloc.c-363-\tif (!err)\nmm/vmalloc.c:364:\t\terr = kmsan_ioremap_page_range(addr, end, phys_addr, prot,\nmm/vmalloc.c-365-\t\t\t\t\t       ioremap_max_page_shift);\n--\nmm/vmalloc.c=708=int vmap_pages_range_noflush(unsigned long addr, unsigned long end,\n--\nmm/vmalloc.c-711-{\nmm/vmalloc.c:712:\tint ret = kmsan_vmap_pages_range_noflush(addr, end, prot, pages,\nmm/vmalloc.c-713-\t\t\t\t\t\tpage_shift, gfp_mask);\n--\nmm/zsmalloc.c=1841=static int zs_page_migrate(struct page *newpage, struct page *page,\n--\nmm/zsmalloc.c-1910-\tcopy_page(d_addr, s_addr);\nmm/zsmalloc.c:1911:\tkmsan_copy_page_meta(zpdesc_page(newzpdesc), zpdesc_page(zpdesc));\nmm/zsmalloc.c-1912-\tkunmap_local(d_addr);\n"
}

							input: 29566
							output: 350
							thoughts: 2796

**KMSAN Assessment: NeedsKMSAN = false**

Okay, let's break this down. The patch delays the KMSAN poisoning of pages freed via RCU. The key here is the timing: *before* the patch, `kmsan_free_page` was called immediately when a folio was freed, which would be problematic because an RCU reader might still be accessing the page during the RCU grace period. The patch correctly delays the KMSAN poisoning to *after* the grace period to avoid false positives from KMSAN. The core change is about synchronization and ensuring memory validity during RCU grace periods, not introducing uninitialized memory bugs.

Now, let's systematically check for uninitialized memory risks. We have to consider four key criteria:

1. **Kernel structures to user space:** There are no kernel structures directly sent to userspace. File page contents are copied to user space iterators, limited by `i_size` and page boundaries, and that is *not* a kernel structure.
2. **Conditional logic and uninitialized variables:** I've carefully reviewed all the relevant code paths (`filemap_read_rcu_lookup`, `filemap_read`, `rcu_free_pcpu`, `rcu_free_wait`, `rcu_free_defer`, and others). There are no uninitialized variables or struct fields used in conditional logic. All variables are either initialized at declaration, initialized through function calls (e.g., `rcu_free_init`), or explicitly set to an initial value (`false` for `migrated`, `0` for `retval`).
3. **Complex data structure allocation/initialization:** There are no new heap allocations of complex data structures. Per-CPU structures are initialized using standard initialization macros (`INIT_LIST_HEAD`, `spin_lock_init`, etc.).
4. **Bounds checks and uninitialized bytes:** The lockless reads *only* read from up-to-date folios, within file `i_size` and within page bounds. This eliminates the risk of reading uninitialized bytes from existing buffers.

Therefore, the patch doesn't *introduce* any risk of uninitialized memory usage. The patch is about addressing the existing concurrency problems in the page cache subsystem by adding proper synchronization.

The failure modes, however, are important: concurrency or race conditions, folio refcounting issues, deadlocks, use-after-free, and memory leaks. These are *classic* concurrency and memory management bugs. The relevant bug detectors are KASAN (for use-after-free and memory corruption), LOCKDEP (for locking bugs and deadlocks), KMEMLEAK (for memory leaks), and KCSAN (for race conditions and data races). These standard bug detectors cover any introduced risks, not KMSAN.

I've also double-checked all relevant functions and code paths to ensure I have not missed anything: `shmem_file_read_iter`, `filemap_read`, `offline_pages`, `rcu_free_wait`, `rcu_free_defer`, `rcu_free_drain`, and `fuse_try_move_folio` to make sure all variables are properly initialized. In all cases, all variables, local and struct members, are fully initialized before use.

In summary, this patch changes synchronization to prevent use-after-free conditions. Standard bug detectors will catch any issues in the code that are not caught by KMSAN, which only detects uninitialized memory bugs. Therefore, `NeedsKMSAN = false`.

{
  "KMSANReasoning": "The patch series implements lockless page cache reads using RCU (CONFIG_PAGE_RCU_FREE) and defers the physical freeing of hot folios until after an RCU grace period.\n\nFrom a memory safety and sanitizer perspective:\n1. No uninitialized memory is exposed: All local variables and data structures (such as struct rcu_free_pcpu, list heads, and loop variables) are properly initialized before use.\n2. Kernel-to-user copies: The fast path in `filemap_read_rcu_copy()` copies file data from uptodate folios bounded strictly by the file's `i_size` and page boundaries, identically to the existing slow-path page cache read mechanisms; no uninitialized kernel structure fields or padding are leaked to user space.\n3. KMSAN hooks: The patch properly defers `kmsan_free_page()` in `__free_pages_prepare()` until the folio is actually released after the RCU grace period, ensuring KMSAN page shadow tracking remains accurate.\n4. Risk surface: The primary risks associated with this lockless RCU page cache reader patch are concurrency races (e.g. during folio splitting, truncation, migration, or hwpoison), potential use-after-free bugs, reference-counting anomalies, and locking/deadlock issues. These failure modes are covered by KASAN, LOCKDEP, and KCSAN.\n\nBecause the changes do not introduce or expose uninitialized memory reads or kernel information leaks, a dedicated KMSAN fuzzing session is not warranted.",
  "NeedsKMSAN": false
}

{
  "KMSANReasoning": "The patch series implements lockless page cache reads using RCU (CONFIG_PAGE_RCU_FREE) and defers the physical freeing of hot folios until after an RCU grace period.\n\nFrom a memory safety and sanitizer perspective:\n1. No uninitialized memory is exposed: All local variables and data structures (such as struct rcu_free_pcpu, list heads, and loop variables) are properly initialized before use.\n2. Kernel-to-user copies: The fast path in `filemap_read_rcu_copy()` copies file data from uptodate folios bounded strictly by the file's `i_size` and page boundaries, identically to the existing slow-path page cache read mechanisms; no uninitialized kernel structure fields or padding are leaked to user space.\n3. KMSAN hooks: The patch properly defers `kmsan_free_page()` in `__free_pages_prepare()` until the folio is actually released after the RCU grace period, ensuring KMSAN page shadow tracking remains accurate.\n4. Risk surface: The primary risks associated with this lockless RCU page cache reader patch are concurrency races (e.g. during folio splitting, truncation, migration, or hwpoison), potential use-after-free bugs, reference-counting anomalies, and locking/deadlock issues. These failure modes are covered by KASAN, LOCKDEP, and KCSAN.\n\nBecause the changes do not introduce or expose uninitialized memory reads or kernel information leaks, a dedicated KMSAN fuzzing session is not warranted.",
  "NeedsKMSAN": false
}

LLM Calls Summary:
Total Calls	Total Tokens	Avg Tokens	Total Duration (Seconds)	Avg Duration (Seconds)

Tool Calls Summary:
Total Calls	Total Duration (Seconds)	Avg Duration (Seconds)