AI Trajectory

Trajectory: (Expand/Collapse All)
Seq	Timestamp	Type	Name	Duration
0/0	2026/07/14 00:43	flow	patch-triage	0m Args: null Results: { "EnableConfigs": [ "SWAP" ], "FocusSymbols": [ "swap_alloc_entry", "swap_queue_add", "swap_queue_del", "swap_queue_get_device", "swap_device_enable", "swap_device_disable", "swap_device_inuse_add", "swap_device_inuse_sub", "add_to_avail_list", "del_from_avail_list", "swap_queue_mask", "swap_queue_unmask", "cluster_alloc_swap_entry" ], "Reasoning": "The patch replaces the plist-based swap device queue with a custom round-robin queue, introducing new core logic for swap allocation and device management. This is a functional change to the core kernel logic and should be fuzzed.", "WorthFuzzing": true }
1/1	2026/07/14 00:43	action	read-patch-diff	0m Args: null Results: { "PatchDiff": "commit 9f349365e0eb1fe2128a8484bb4ca651f1dbd2a2\nAuthor: syz-cluster \u003ctriage@syzkaller.com\u003e\nDate: Tue Jul 14 00:43:44 2026 +0000\n\n syz-cluster: applied patch under review\n\ndiff --git a/include/linux/plist.h b/include/linux/plist.h\nindex 16cf4355b5c131..73d359c1a0536a 100644\n--- a/include/linux/plist.h\n+++ b/include/linux/plist.h\n@@ -132,8 +132,6 @@ static inline void plist_node_init(struct plist_node node, int prio)\n extern void plist_add(struct plist_node node, struct plist_head head);\n extern void plist_del(struct plist_node node, struct plist_head head);\n \n-extern void plist_requeue(struct plist_node node, struct plist_head head);\n-\n /\n plist_for_each - iterate over the plist\n * @pos:\tthe type * to use as a loop counter\ndiff --git a/include/linux/swap.h b/include/linux/swap.h\nindex 696ed01709c2ba..04767540972b81 100644\n--- a/include/linux/swap.h\n+++ b/include/linux/swap.h\n@@ -194,6 +194,22 @@ struct swap_extent {\n \t((offsetof(union swap_header, magic.magic) - \\\n \t offsetof(union swap_header, info.badpages)) / sizeof(int))\n \n+/\n+ Swap device flags, except the ones documented below, all are immutable\n+ * after exposed by swap_device_enable, and until the device is freed again\n+ * (SWP_USED unset). The exceptions are all protected by swapon_rwsem:\n+ * - SWP_USED: Protected by swapon_rwsem. Indicates the device is inuse. Once\n+ * set, won't be cleared unless all reference to this device is freed and\n+ * swapoff finished.\n+ * - SWP_WRITEOK: Protected by both swapon_rwsem and swap_avail_lock, clearing\n+ * this flag also waits for all current cluster lock users to exit so\n+ * checking this flag while holding any of these locks ensures the device\n+ * is safe to use at the moment. Note: clearing this flag doesn't affect\n+ * pending IO or async requests, it only prevents further entry allocation\n+ * or new async request (e.g. discard) from initiating.\n+ * - SWP_HIBERNATION: Protected by swapon_rwsem. Indicates if the device\n+ * is pinned for hibernation.\n+ /\n enum {\n \tSWP_USED\t= (1 \u003c\u003c 0),\t/ is slot in swap_info[] used? /\n \tSWP_WRITEOK\t= (1 \u003c\u003c 1),\t/ ok to write to this swap?\t/\n@@ -230,10 +246,17 @@ enum {\n #endif\n \n /\n- * We keep using same cluster for rotational device so IO will be sequential.\n- * The purpose is to optimize SWAP throughput on these device.\n+ * We assign a cluster to each CPU, so each CPU can allocate swap entry from\n+ * its own cluster and swapout sequentially. The purpose is to optimize swapout\n+ * throughput.\n /\n+struct percpu_cluster {\n+\tlocal_lock_t lock; / Protect the percpu_cluster above /\n+\tunsigned int next[SWAP_NR_ORDERS]; / Likely next allocation offset /\n+};\n+\n struct swap_sequential_cluster {\n+\tspinlock_t lock; / Serialize usage of global cluster /\n \tunsigned int next[SWAP_NR_ORDERS]; / Likely next allocation offset /\n };\n \n@@ -244,7 +267,6 @@ struct swap_info_struct {\n \tstruct percpu_ref users;\t/ indicate and keep swap device valid. /\n \tunsigned long\tflags;\t\t/ SWP_USED etc: see above /\n \tsigned short\tprio;\t\t/ swap priority of this type /\n-\tstruct plist_node list;\t\t/ entry in swap_active_head /\n \tsigned char\ttype;\t\t/ strange name for an index /\n \tunsigned int\tmax;\t\t/ size of this swap device /\n \tstruct swap_cluster_info cluster_info; /* cluster info. Only for SSD /\n@@ -256,26 +278,20 @@ struct swap_info_struct {\n \t\t\t\t\t/ list of cluster that are fragmented or contented /\n \tunsigned int pages;\t\t/ total of usable pages of swap /\n \tatomic_long_t inuse_pages;\t/ number of those currently in use /\n+\tstruct percpu_cluster\t__percpu percpu_cluster; /* per cpu's swap location /\n \tstruct swap_sequential_cluster global_cluster; /* Use one global cluster for rotating device /\n-\tspinlock_t global_cluster_lock;\t/ Serialize usage of global cluster /\n \tstruct rb_root swap_extent_root;/ root of the swap extent rbtree /\n \tstruct block_device bdev;\t/* swap device or bdev of swap file /\n \tstruct file swap_file;\t\t/* seldom referenced /\n \tstruct completion comp;\t\t/ seldom referenced /\n \tspinlock_t lock;\t\t/\n-\t\t\t\t\t * protect map scan related fields like\n-\t\t\t\t\t * inuse_pages and all cluster lists.\n-\t\t\t\t\t * Other fields are only changed\n-\t\t\t\t\t * at swapon/swapoff, so are protected\n-\t\t\t\t\t * by swap_lock. changing flags need\n-\t\t\t\t\t * hold this lock and swap_lock. If\n-\t\t\t\t\t * both locks need hold, hold swap_lock\n-\t\t\t\t\t * first.\n+\t\t\t\t\t * Protect cluster lists. Other fields\n+\t\t\t\t\t * are only changed at swapon/swapoff,\n+\t\t\t\t\t * so are protected by swapon_rwsem.\n \t\t\t\t\t /\n \tstruct work_struct discard_work; / discard worker /\n \tstruct work_struct reclaim_work; / reclaim worker /\n \tstruct list_head discard_clusters; / discard clusters list /\n-\tstruct plist_node avail_list; / entry in swap_avail_head /\n };\n \n static inline swp_entry_t page_swap_entry(struct page page)\ndiff --git a/lib/plist.c b/lib/plist.c\nindex a5bef38add431d..b05ae7ffea87e8 100644\n--- a/lib/plist.c\n+++ b/lib/plist.c\n@@ -142,58 +142,6 @@ void plist_del(struct plist_node node, struct plist_head head)\n \tplist_check_head(head);\n }\n \n-/*\n- plist_requeue - Requeue @node at end of same-prio entries.\n- \n- This is essentially an optimized plist_del() followed by\n- * plist_add(). It moves an entry already in the plist to\n- * after any other same-priority entries.\n- \n- @node:\t\u0026struct plist_node pointer - entry to be moved\n- * @head:\t\u0026struct plist_head pointer - list head\n- /\n-void plist_requeue(struct plist_node node, struct plist_head head)\n-{\n-\tstruct plist_node iter;\n-\tstruct list_head node_next = \u0026head-\u003enode_list;\n-\n-\tplist_check_head(head);\n-\tBUG_ON(plist_head_empty(head));\n-\tBUG_ON(plist_node_empty(node));\n-\n-\tif (node == plist_last(head))\n-\t\treturn;\n-\n-\titer = plist_next(node);\n-\n-\tif (node-\u003eprio != iter-\u003eprio)\n-\t\treturn;\n-\n-\tplist_del(node, head);\n-\n-\t/\n-\t * After plist_del(), iter is the replacement of the node. If the node\n-\t * was on prio_list, take shortcut to find node_next instead of looping.\n-\t /\n-\tif (!list_empty(\u0026iter-\u003eprio_list)) {\n-\t\titer = list_entry(iter-\u003eprio_list.next, struct plist_node,\n-\t\t\t\t prio_list);\n-\t\tnode_next = \u0026iter-\u003enode_list;\n-\t\tgoto queue;\n-\t}\n-\n-\tplist_for_each_continue(iter, head) {\n-\t\tif (node-\u003eprio != iter-\u003eprio) {\n-\t\t\tnode_next = \u0026iter-\u003enode_list;\n-\t\t\tbreak;\n-\t\t}\n-\t}\n-queue:\n-\tlist_add_tail(\u0026node-\u003enode_list, node_next);\n-\n-\tplist_check_head(head);\n-}\n-\n #ifdef CONFIG_DEBUG_PLIST\n #include \u003clinux/sched.h\u003e\n #include \u003clinux/sched/clock.h\u003e\n@@ -231,14 +179,6 @@ static void __init plist_test_check(int nr_expect)\n \tBUG_ON(prio_pos-\u003eprio_list.next != \u0026first-\u003eprio_list);\n }\n \n-static void __init plist_test_requeue(struct plist_node node)\n-{\n-\tplist_requeue(node, \u0026test_head);\n-\n-\tif (node != plist_last(\u0026test_head))\n-\t\tBUG_ON(node-\u003eprio == plist_next(node)-\u003eprio);\n-}\n-\n static int __init plist_test(void)\n {\n \tint nr_expect = 0, i, loop;\n@@ -262,10 +202,6 @@ static int __init plist_test(void)\n \t\t\tnr_expect--;\n \t\t}\n \t\tplist_test_check(nr_expect);\n-\t\tif (!plist_node_empty(test_node + i)) {\n-\t\t\tplist_test_requeue(test_node + i);\n-\t\t\tplist_test_check(nr_expect);\n-\t\t}\n \t}\n \n \tfor (i = 0; i \u003c ARRAY_SIZE(test_node); i++) {\ndiff --git a/mm/swap.h b/mm/swap.h\nindex b51ad3071a7308..08a782f3d5acd2 100644\n--- a/mm/swap.h\n+++ b/mm/swap.h\n@@ -91,16 +91,16 @@ static inline unsigned int swp_cluster_offset(swp_entry_t entry)\n /\n static inline struct swap_info_struct __swap_type_to_info(int type)\n {\n-\tstruct swap_info_struct si;\n-\n-\tsi = READ_ONCE(swap_info[type]); / rcu_dereference() /\n-\tVM_WARN_ON_ONCE(percpu_ref_is_zero(\u0026si-\u003eusers)); / race with swapoff /\n-\treturn si;\n+\treturn READ_ONCE(swap_info[type]); / rcu_dereference() /\n }\n \n static inline struct swap_info_struct __swap_entry_to_info(swp_entry_t entry)\n {\n-\treturn __swap_type_to_info(swp_type(entry));\n+\tstruct swap_info_struct si;\n+\n+\tsi = __swap_type_to_info(swp_type(entry)); / rcu_dereference() /\n+\tVM_WARN_ON_ONCE(percpu_ref_is_zero(\u0026si-\u003eusers)); / race with swapoff /\n+\treturn si;\n }\n \n static inline struct swap_cluster_info __swap_offset_to_cluster(\ndiff --git a/mm/swapfile.c b/mm/swapfile.c\nindex 775c385490a5fa..f93863309a5a50 100644\n--- a/mm/swapfile.c\n+++ b/mm/swapfile.c\n@@ -41,7 +41,6 @@\n #include \u003clinux/completion.h\u003e\n #include \u003clinux/suspend.h\u003e\n #include \u003clinux/zswap.h\u003e\n-#include \u003clinux/plist.h\u003e\n \n #include \u003casm/tlbflush.h\u003e\n #include \u003clinux/leafops.h\u003e\n@@ -49,31 +48,38 @@\n #include \"internal.h\"\n #include \"swap.h\"\n \n-static void swap_range_alloc(struct swap_info_struct si,\n-\t\t\t unsigned int nr_entries);\n+static void swap_device_inuse_add(struct swap_info_struct si,\n+\t\t\t\t unsigned int nr_entries);\n static bool folio_swapcache_freeable(struct folio folio);\n static void move_cluster(struct swap_info_struct si,\n \t\t\t struct swap_cluster_info ci, struct list_head list,\n \t\t\t enum swap_cluster_flags new_flags);\n+static bool get_swap_device_info(struct swap_info_struct si);\n \n /\n- * Protects the swap_info array, and the SWP_USED flag. swap_info contains\n- * lazily allocated \u0026 freed swap device info struts, and SWP_USED indicates\n- * which device is used, ~SWP_USED devices and can be reused.\n- \n- Also protects swap_active_head total_swap_pages, and the SWP_WRITEOK flag.\n+ * Serializes swapon/swapoff (writers) and protects the swap_info\n+ * array, nr_swapfiles, total_swap_pages, and part of swap device\n+ * info content (see comment of swap_info_struct). Readers\n+ * (allocation, /proc/swaps, etc.) take percpu_down_read() which\n+ * is cheap as the hot path.\n+ /\n+DEFINE_STATIC_PERCPU_RWSEM(swapon_rwsem);\n+/\n+ * swap_info contains lazily allocated swap device info structs, and\n+ * SWP_USED indicates which device is used, ~SWP_USED devices can be\n+ * reused. Protected by swapon_rwsem, but reading could be lockless.\n /\n-static DEFINE_SPINLOCK(swap_lock);\n+struct swap_info_struct swap_info[MAX_SWAPFILES];\n static unsigned int nr_swapfiles;\n-atomic_long_t nr_swap_pages;\n+long total_swap_pages;\n+\n /\n Some modules use swappable objects and may try to swap them out under\n * memory pressure (via the shrinker). Before doing so, they may wish to\n * check to see if any swap space is available.\n /\n+atomic_long_t nr_swap_pages;\n EXPORT_SYMBOL_GPL(nr_swap_pages);\n-/ protected with swap_lock. reading in vm_swap_full() doesn't need lock /\n-long total_swap_pages;\n #define DEF_SWAP_PRIO -1\n unsigned long swapfile_maximum_size;\n #ifdef CONFIG_MIGRATION\n@@ -83,52 +89,42 @@ bool swap_migration_ad_supported;\n static const char Bad_file[] = \"Bad swap file entry \";\n static const char Bad_offset[] = \"Bad swap offset entry \";\n \n-/\n- * all active swap_info_structs\n- * protected with swap_lock, and ordered by priority.\n- /\n-static PLIST_HEAD(swap_active_head);\n+static inline struct swap_info_struct __swap_iter(int i, unsigned long flag)\n+{\n+\tlockdep_assert_held(\u0026swapon_rwsem);\n+\twhile (i \u003c nr_swapfiles) {\n+\t\tstruct swap_info_struct si = __swap_type_to_info(i);\n+\n+\t\tVM_WARN_ON(!si);\n+\t\t(i)++;\n+\t\tif (flag \u0026\u0026 !((si-\u003eflags \u0026 flag) == flag))\n+\t\t\tcontinue;\n+\t\treturn si;\n+\t}\n+\treturn NULL;\n+}\n+\n+#define __for_each_swap(si, flag)\t\t\t\\\n+\tfor (int __i = 0; ((si) = __swap_iter(\u0026__i, flag));)\n \n /\n- * all available (active, not full) swap_info_structs\n- * protected with swap_avail_lock, ordered by priority.\n- * This is used by folio_alloc_swap() instead of swap_active_head\n- * because swap_active_head includes all swap_info_structs,\n- * but folio_alloc_swap() doesn't need to look at full ones.\n- * This uses its own lock instead of swap_lock because when a\n- * swap_info_struct changes between not-full/full, it needs to\n- * add/remove itself to/from this list, but the swap_info_struct-\u003elock\n- * is held and the locking order requires swap_lock to be taken\n- * before any swap_info_struct-\u003elock.\n+ * for_each_swap - iterate through all allocated and inuse swap devices\n+ * @si: the iterator\n+ \n+ Context: The caller must hold swapon_rwsem. The lock may be dropped during\n+ * the loop body but must be re-acquired before the next iteration.\n /\n-static PLIST_HEAD(swap_avail_head);\n-static DEFINE_SPINLOCK(swap_avail_lock);\n-\n-struct swap_info_struct swap_info[MAX_SWAPFILES];\n+#define for_each_swap(si) __for_each_swap(si, SWP_USED)\n+#define for_each_avail_swap(si) __for_each_swap(si, SWP_USED \| SWP_WRITEOK)\n \n static struct kmem_cache swap_table_cachep;\n \n-/ Protects si-\u003eswap_file for /proc/swaps usage /\n-static DEFINE_MUTEX(swapon_mutex);\n-\n static DECLARE_WAIT_QUEUE_HEAD(proc_poll_wait);\n / Activity counter to indicate that a swapon or swapoff has occurred /\n static atomic_t proc_poll_event = ATOMIC_INIT(0);\n \n atomic_t nr_rotate_swap = ATOMIC_INIT(0);\n \n-struct percpu_swap_cluster {\n-\tstruct swap_info_struct si[SWAP_NR_ORDERS];\n-\tunsigned long offset[SWAP_NR_ORDERS];\n-\tlocal_lock_t lock;\n-};\n-\n-static DEFINE_PER_CPU(struct percpu_swap_cluster, percpu_swap_cluster) = {\n-\t.si = { NULL },\n-\t.offset = { SWAP_ENTRY_INVALID },\n-\t.lock = INIT_LOCAL_LOCK(),\n-};\n-\n /* May return NULL on invalid type, caller must check for NULL return /\n static struct swap_info_struct swap_type_to_info(int type)\n {\n@@ -143,18 +139,350 @@ static struct swap_info_struct swap_entry_to_info(swp_entry_t entry)\n \treturn swap_type_to_info(swp_type(entry));\n }\n \n+/\n+ * All available swap_info_structs are grouped by priority rings, the rings\n+ * are ordered in a queue by priority (lower prio value = higher priority).\n+ * The allocator iterates and rotates devices within each priority ring.\n+ * When all devices in a ring are iterated, it goes to the next lower\n+ * priority ring.\n+ /\n+struct swap_prio_ring {\n+\tint prio;\n+\tunsigned int size;\n+\tstruct swap_info_struct dev[] __counted_by(size);\n+};\n+\n+/\n+ The ring is protected by swapon_rwsem so updating it is costly. To make\n+ * the allocator and other users skip full devices faster, the lowest bit of\n+ * a device pointer is used to mark it disabled (temporarily unavailable).\n+ * This relies on struct swap_info_struct being sufficiently\n+ * aligned (guaranteed by kmalloc).\n+ /\n+#define SWAP_DEVICE_MASKED_SHIFT\t0\n+#define SWAP_DEVICE_MASKED_BIT\t\tBIT(SWAP_DEVICE_MASKED_SHIFT)\n+\n+/\n+ * Serializes queue content mutations and keeps SWAP_USAGE_OFFLIST_BIT\n+ * consistent with the masked state of each device pointer.\n+ /\n+static DEFINE_SPINLOCK(swap_queue_update_lock);\n+\n+/\n+ * Swap queue is protected by both swap_queue_update_lock and swapon_rwsem.\n+ * Only swapon/swapoff will take the write lock, and modify the queue length\n+ * or any ring's length. swap_queue_update_lock protects the content so\n+ * devices can be masked easily without taking the writelock, which is heavy.\n+ /\n+static struct swap_prio_ring swap_queue;\n+static unsigned int swap_queue_len;\n+\n+/\n+ * Each CPU has its read iterator, so the queue itself will remain read\n+ * only and the CPU side reader rotates by iterating the devices\n+ * periodically using the counter.\n+ /\n+#define SWAP_ROUND_ROBIN_QUOTA SWAPFILE_CLUSTER\n+struct swap_ring_iterator {\n+\tint offset;\n+\tlong rr_counter;\n+};\n+\n+struct swap_queue_reader {\n+\tlocal_lock_t lock;\n+\tstruct swap_ring_iterator ri[];\n+};\n+\n+static __percpu struct swap_queue_reader swap_queue_readers;\n+\n+static inline bool swap_device_masked(struct swap_info_struct si)\n+{\n+\treturn (unsigned long)si \u0026 SWAP_DEVICE_MASKED_BIT;\n+}\n+\n+static inline struct swap_info_struct swap_device_unmask_ptr(struct swap_info_struct si)\n+{\n+\treturn (struct swap_info_struct )((unsigned long)si \u0026 ~SWAP_DEVICE_MASKED_BIT);\n+}\n+\n+static struct swap_queue_reader __percpu swap_queue_prealloc_readers(int nr_rings, gfp_t gfp)\n+{\n+\tstruct swap_queue_reader __percpu readers;\n+\n+\tif (!nr_rings)\n+\t\treturn NULL;\n+\n+\treaders = __alloc_percpu_gfp(struct_size(readers, ri, nr_rings),\n+\t\t\t\t __alignof__(readers), gfp);\n+\treturn readers;\n+}\n+\n+static void swap_queue_install_readers(struct swap_queue_reader __percpu readers)\n+{\n+\tint ring_idx, cpu;\n+\tstruct swap_prio_ring ring;\n+\n+\tfree_percpu(swap_queue_readers);\n+\tswap_queue_readers = readers;\n+\tif (!readers)\n+\t\treturn;\n+\n+\t/ Distribute each CPU's swap IO fairly across devices. /\n+\tfor_each_possible_cpu(cpu) {\n+\t\tlocal_lock_init(\u0026per_cpu_ptr(readers, cpu)-\u003elock);\n+\n+\t\tfor (ring_idx = 0; ring_idx \u003c swap_queue_len; ring_idx++) {\n+\t\t\tring = swap_queue[ring_idx];\n+\t\t\tper_cpu_ptr(readers, cpu)-\u003eri[ring_idx].offset =\n+\t\t\t\t cpu % ring-\u003esize;\n+\t\t}\n+\t}\n+}\n+\n+static struct swap_info_struct swap_queue_get_device(long nr_alloc, int nr_iter)\n+{\n+\tbool rotate = false;\n+\tstruct swap_info_struct si;\n+\tstruct swap_ring_iterator ri;\n+\tstruct swap_prio_ring ring;\n+\tunsigned int queue_idx;\n+\n+\tif (!swap_queue_len)\n+\t\treturn ERR_PTR(-ENOENT);\n+\n+\tqueue_idx = 0;\n+\twhile (nr_iter \u003e= swap_queue[queue_idx]-\u003esize) {\n+\t\tnr_iter -= swap_queue[queue_idx]-\u003esize;\n+\t\tif (++queue_idx \u003e= swap_queue_len)\n+\t\t\treturn ERR_PTR(-ENOENT);\n+\t}\n+\n+\tring = swap_queue[queue_idx];\n+\tlocal_lock(\u0026swap_queue_readers-\u003elock);\n+\tri = this_cpu_ptr(\u0026swap_queue_readers-\u003eri[queue_idx]);\n+\t/ Rotate while iterating the ring, just not on the first try /\n+\tif (nr_iter)\n+\t\trotate = true;\n+\telse if (ri-\u003err_counter \u003c nr_alloc)\n+\t\trotate = true;\n+\telse if (ri-\u003eoffset \u003e= ring-\u003esize)\n+\t\trotate = true;\n+\tif (rotate) {\n+\t\tri-\u003eoffset++;\n+\t\tri-\u003eoffset %= ring-\u003esize;\n+\t\tri-\u003err_counter = SWAP_ROUND_ROBIN_QUOTA;\n+\t}\n+\tri-\u003err_counter -= nr_alloc;\n+\tsi = READ_ONCE(ring-\u003edev[ri-\u003eoffset]);\n+\tlocal_unlock(\u0026swap_queue_readers-\u003elock);\n+\n+\tif (swap_device_masked(si))\n+\t\treturn ERR_PTR(-EBUSY);\n+\n+\tsi = swap_device_unmask_ptr(si);\n+\treturn si;\n+}\n+\n+static bool swap_queue_find(struct swap_info_struct si,\n+\t\t\t unsigned int ring_idx, unsigned int dev_idx)\n+{\n+\tunsigned int i, j;\n+\tstruct swap_prio_ring ring;\n+\n+\tlockdep_assert(lockdep_is_held(\u0026swapon_rwsem) \|\|\n+\t\t lockdep_is_held(\u0026swap_queue_update_lock));\n+\n+\tfor (i = 0; i \u003c swap_queue_len; i++) {\n+\t\tring = swap_queue[i];\n+\t\tif (ring-\u003eprio != si-\u003eprio)\n+\t\t\tcontinue;\n+\t\tfor (j = 0; j \u003c ring-\u003esize; j++) {\n+\t\t\tif (swap_device_unmask_ptr(READ_ONCE(ring-\u003edev[j])) != si)\n+\t\t\t\tcontinue;\n+\t\t\tring_idx = i;\n+\t\t\tdev_idx = j;\n+\t\t\treturn true;\n+\t\t}\n+\t}\n+\treturn false;\n+}\n+\n+static void swap_queue_mask(struct swap_info_struct si)\n+{\n+\tunsigned int ring_idx, dev_idx;\n+\n+\tlockdep_assert_held(\u0026swap_queue_update_lock);\n+\tif (swap_queue_find(si, \u0026ring_idx, \u0026dev_idx))\n+\t\t__set_bit(SWAP_DEVICE_MASKED_SHIFT,\n+\t\t\t (unsigned long )\u0026swap_queue[ring_idx]-\u003edev[dev_idx]);\n+}\n+\n+static void swap_queue_unmask(struct swap_info_struct si)\n+{\n+\tunsigned int ring_idx, dev_idx;\n+\n+\tlockdep_assert_held(\u0026swap_queue_update_lock);\n+\tif (swap_queue_find(si, \u0026ring_idx, \u0026dev_idx))\n+\t\t__clear_bit(SWAP_DEVICE_MASKED_SHIFT,\n+\t\t\t (unsigned long )\u0026swap_queue[ring_idx]-\u003edev[dev_idx]);\n+}\n+\n+static int swap_queue_add(struct swap_info_struct si)\n+{\n+\tstruct swap_prio_ring new_queue = NULL, old_queue = NULL;\n+\tstruct swap_queue_reader __percpu new_readers = NULL;\n+\tstruct swap_prio_ring ring, new_ring = NULL, old_ring = NULL;\n+\tint prio = si-\u003eprio;\n+\tint i, pos, err = -ENOMEM;\n+\tgfp_t gfp;\n+\n+\t/* Swap not usable here because this is swap, just reclaim cache. /\n+\tgfp = GFP_NOIO \| __GFP_HIGH;\n+\tlockdep_assert_held_write(\u0026swapon_rwsem);\n+\n+\tfor (pos = 0; pos \u003c swap_queue_len; pos++) {\n+\t\tif (swap_queue[pos]-\u003eprio == prio)\n+\t\t\tgoto add_to_ring;\n+\t\tif (swap_queue[pos]-\u003eprio \u003c prio)\n+\t\t\tbreak;\n+\t}\n+\n+\t/ No ring at this priority: insert a new one at pos. /\n+\tnew_readers = swap_queue_prealloc_readers(swap_queue_len + 1, gfp);\n+\tif (!new_readers)\n+\t\tgoto failed;\n+\tnew_queue = kmalloc_array(swap_queue_len + 1, sizeof(swap_queue), gfp);\n+\tif (!new_queue)\n+\t\tgoto failed;\n+\tnew_ring = kmalloc(struct_size(new_ring, dev, 1), gfp);\n+\tif (!new_ring)\n+\t\tgoto failed;\n+\tif (!get_swap_device_info(si))\n+\t\tgoto failed;\n+\n+\tnew_ring-\u003eprio = prio;\n+\tnew_ring-\u003esize = 1;\n+\tnew_ring-\u003edev[0] = si;\n+\tfor (i = 0; i \u003c pos; i++)\n+\t\tnew_queue[i] = swap_queue[i];\n+\tnew_queue[pos] = new_ring;\n+\tfor (i = pos; i \u003c swap_queue_len; i++)\n+\t\tnew_queue[i + 1] = swap_queue[i];\n+\n+\tspin_lock(\u0026swap_queue_update_lock);\n+\told_queue = swap_queue;\n+\tswap_queue = new_queue;\n+\tswap_queue_len++;\n+\tspin_unlock(\u0026swap_queue_update_lock);\n+\tkfree(old_queue);\n+\n+\tswap_queue_install_readers(new_readers);\n+\treturn 0;\n+\n+add_to_ring:\n+\tring = swap_queue[pos];\n+\tnew_ring = kmalloc(struct_size(ring, dev, ring-\u003esize + 1), gfp);\n+\tif (!new_ring)\n+\t\tgoto failed;\n+\tif (!get_swap_device_info(si))\n+\t\tgoto failed;\n+\tspin_lock(\u0026swap_queue_update_lock);\n+\tmemcpy(new_ring, ring, struct_size(ring, dev, ring-\u003esize));\n+\tnew_ring-\u003esize++;\n+\tnew_ring-\u003edev[new_ring-\u003esize - 1] = si;\n+\told_ring = swap_queue[pos];\n+\tswap_queue[pos] = new_ring;\n+\tspin_unlock(\u0026swap_queue_update_lock);\n+\tkfree(old_ring);\n+\treturn 0;\n+\n+failed:\n+\tfree_percpu(new_readers);\n+\tkfree(new_queue);\n+\tkfree(new_ring);\n+\treturn err;\n+}\n+\n+static void swap_queue_del(struct swap_info_struct si)\n+{\n+\tgfp_t gfp;\n+\tunsigned int ring_idx, dev_idx;\n+\tstruct swap_queue_reader __percpu new_readers = NULL;\n+\tstruct swap_prio_ring ring, new_ring = NULL, old_ring = NULL;\n+\tstruct swap_prio_ring new_queue = NULL, old_queue = NULL;\n+\n+\tlockdep_assert_held_write(\u0026swapon_rwsem);\n+\tif (!swap_queue_find(si, \u0026ring_idx, \u0026dev_idx)) {\n+\t\tWARN_ON(1);\n+\t\treturn;\n+\t}\n+\n+\t/\n+\t * To shrink memory usage, pre-allocate new smaller data before\n+\t * locking. Failure is fine, swapoff will release them anyway.\n+\t /\n+\tgfp = GFP_NOIO \| __GFP_HIGH;\n+\tring = swap_queue[ring_idx];\n+\tif (ring-\u003esize \u003e 1)\n+\t\tnew_ring = kmalloc(struct_size(ring, dev, ring-\u003esize - 1), gfp);\n+\tif (ring-\u003esize == 1 \u0026\u0026 swap_queue_len \u003e 1) {\n+\t\tnew_readers = swap_queue_prealloc_readers(\n+\t\t\t\t\tswap_queue_len - 1, gfp);\n+\t\tnew_queue = kmalloc(sizeof(swap_queue) \n+\t\t\t\t (swap_queue_len - 1), gfp);\n+\t}\n+\n+\tspin_lock(\u0026swap_queue_update_lock);\n+\tif (ring-\u003esize \u003e 1) {\n+\t\t/ Shift trailing devices left to fill the gap. /\n+\t\twhile (++dev_idx \u003c ring-\u003esize)\n+\t\t\tring-\u003edev[dev_idx - 1] =\n+\t\t\t\tring-\u003edev[dev_idx];\n+\t\tring-\u003esize--;\n+\t\tif (new_ring) {\n+\t\t\tmemcpy(new_ring, ring,\n+\t\t\t struct_size(ring, dev, ring-\u003esize));\n+\t\t\told_ring = ring;\n+\t\t\tswap_queue[ring_idx] = new_ring;\n+\t\t}\n+\t} else {\n+\t\t/ Last device in this ring: remove the ring. /\n+\t\told_ring = ring;\n+\t\tswap_queue_len--;\n+\t\twhile (++ring_idx \u003c= swap_queue_len)\n+\t\t\tswap_queue[ring_idx - 1] =\n+\t\t\t\tswap_queue[ring_idx];\n+\t\tif (new_queue) {\n+\t\t\tmemcpy(new_queue, swap_queue,\n+\t\t\t sizeof(swap_queue) * swap_queue_len);\n+\t\t\told_queue = swap_queue;\n+\t\t\tswap_queue = new_queue;\n+\t\t} else if (!swap_queue_len) {\n+\t\t\told_queue = swap_queue;\n+\t\t\tswap_queue = NULL;\n+\t\t}\n+\t\tif (new_readers \|\| !swap_queue_len)\n+\t\t\tswap_queue_install_readers(new_readers);\n+\t}\n+\tspin_unlock(\u0026swap_queue_update_lock);\n+\n+\tkfree(old_ring);\n+\tkfree(old_queue);\n+\tput_swap_device(si);\n+}\n+\n /\n Use the second highest bit of inuse_pages counter as the indicator\n- * if one swap device is on the available plist, so the atomic can\n+ * if one swap device is unavailable for allocation, so the atomic can\n * still be updated arithmetically while having special data embedded.\n \n inuse_pages counter is the only thing indicating if a device should\n- * be on avail_lists or not (except swapon / swapoff). By embedding the\n- * off-list bit in the atomic counter, updates no longer need any lock\n- * to check the list status.\n+ * be in the available queue or not (except swapon / swapoff). By\n+ * embedding the off-list bit in the atomic counter, updates no longer\n+ * need any lock to check the list status.\n \n- This bit will be set if the device is not on the plist and not\n- * usable, will be cleared if the device is on the plist.\n+ * This bit will be set if the device is not in the available queue\n+ * and not usable, will be cleared if the device is in the queue.\n /\n #define SWAP_USAGE_OFFLIST_BIT (1UL \u003c\u003c (BITS_PER_TYPE(atomic_t) - 2))\n #define SWAP_USAGE_COUNTER_MASK (~SWAP_USAGE_OFFLIST_BIT)\n@@ -163,6 +491,26 @@ static long swap_usage_in_pages(struct swap_info_struct si)\n \treturn atomic_long_read(\u0026si-\u003einuse_pages) \u0026 SWAP_USAGE_COUNTER_MASK;\n }\n \n+/\n+ Serialize the allocation on single CPU or globally to avoid\n+ * fragmentation and make the workflow easier to follow.\n+ /\n+static void swap_alloc_lock_device(struct swap_info_struct si)\n+{\n+\tif (si-\u003eflags \u0026 SWP_SOLIDSTATE)\n+\t\tlocal_lock(\u0026si-\u003epercpu_cluster-\u003elock);\n+\telse\n+\t\tspin_lock(\u0026si-\u003eglobal_cluster-\u003elock);\n+}\n+\n+static void swap_alloc_unlock_device(struct swap_info_struct si)\n+{\n+\tif (si-\u003eflags \u0026 SWP_SOLIDSTATE)\n+\t\tlocal_unlock(\u0026si-\u003epercpu_cluster-\u003elock);\n+\telse\n+\t\tspin_unlock(\u0026si-\u003eglobal_cluster-\u003elock);\n+}\n+\n / Reclaim the swap entry anyway if possible /\n #define TTRS_ANYWAY\t\t0x1\n /\n@@ -533,9 +881,10 @@ swap_cluster_populate(struct swap_info_struct si,\n \t Only cluster isolation from the allocator does table allocation.\n \t * Swap allocator uses percpu clusters and holds the local lock.\n \t /\n-\tlockdep_assert_held(\u0026this_cpu_ptr(\u0026percpu_swap_cluster)-\u003elock);\n-\tif (!(si-\u003eflags \u0026 SWP_SOLIDSTATE))\n-\t\tlockdep_assert_held(\u0026si-\u003eglobal_cluster_lock);\n+\tif (si-\u003eflags \u0026 SWP_SOLIDSTATE)\n+\t\tlockdep_assert_held(this_cpu_ptr(\u0026si-\u003epercpu_cluster-\u003elock));\n+\telse\n+\t\tlockdep_assert_held(\u0026si-\u003eglobal_cluster-\u003elock);\n \tlockdep_assert_held(\u0026ci-\u003elock);\n \n \tif (!swap_cluster_alloc_table(ci, __GFP_HIGH \| __GFP_NOMEMALLOC \|\n@@ -548,9 +897,7 @@ swap_cluster_populate(struct swap_info_struct si,\n \t * the potential recursive allocation is limited.\n \t /\n \tspin_unlock(\u0026ci-\u003elock);\n-\tif (!(si-\u003eflags \u0026 SWP_SOLIDSTATE))\n-\t\tspin_unlock(\u0026si-\u003eglobal_cluster_lock);\n-\tlocal_unlock(\u0026percpu_swap_cluster.lock);\n+\tswap_alloc_unlock_device(si);\n \n \tret = swap_cluster_alloc_table(ci, __GFP_HIGH \| __GFP_NOMEMALLOC \|\n \t\t\t\t\t GFP_KERNEL);\n@@ -563,9 +910,7 @@ swap_cluster_populate(struct swap_info_struct si,\n \t * could happen with ignoring the percpu cluster is fragmentation,\n \t * which is acceptable since this fallback and race is rare.\n \t /\n-\tlocal_lock(\u0026percpu_swap_cluster.lock);\n-\tif (!(si-\u003eflags \u0026 SWP_SOLIDSTATE))\n-\t\tspin_lock(\u0026si-\u003eglobal_cluster_lock);\n+\tswap_alloc_lock_device(si);\n \tspin_lock(\u0026ci-\u003elock);\n \n \tif (ret) {\n@@ -671,7 +1016,7 @@ static bool swap_do_scheduled_discard(struct swap_info_struct si)\n \t\tci = list_first_entry(\u0026si-\u003ediscard_clusters, struct swap_cluster_info, list);\n \t\t/\n \t\t Delete the cluster from list to prepare for discard, but keep\n-\t\t * the CLUSTER_FLAG_DISCARD flag, percpu_swap_cluster could be\n+\t\t * the CLUSTER_FLAG_DISCARD flag, there could be percpu_cluster\n \t\t * pointing to it, or ran into by relocate_cluster.\n \t\t /\n \t\tlist_del(\u0026ci-\u003elist);\n@@ -955,7 +1300,7 @@ static bool __swap_cluster_alloc_entries(struct swap_info_struct si,\n \tif (cluster_is_empty(ci))\n \t\tci-\u003eorder = order;\n \tci-\u003ecount += nr_pages;\n-\tswap_range_alloc(si, nr_pages);\n+\tswap_device_inuse_add(si, nr_pages);\n \n \treturn true;\n }\n@@ -1003,12 +1348,10 @@ static unsigned int alloc_swap_scan_cluster(struct swap_info_struct si,\n out:\n \trelocate_cluster(si, ci);\n \tswap_cluster_unlock(ci);\n-\tif (si-\u003eflags \u0026 SWP_SOLIDSTATE) {\n-\t\tthis_cpu_write(percpu_swap_cluster.offset[order], next);\n-\t\tthis_cpu_write(percpu_swap_cluster.si[order], si);\n-\t} else {\n+\tif (si-\u003eflags \u0026 SWP_SOLIDSTATE)\n+\t\tthis_cpu_write(si-\u003epercpu_cluster-\u003enext[order], next);\n+\telse\n \t\tsi-\u003eglobal_cluster-\u003enext[order] = next;\n-\t}\n \treturn found;\n }\n \n@@ -1103,13 +1446,14 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct si,\n \tif (order \u0026\u0026 !(si-\u003eflags \u0026 SWP_BLKDEV))\n \t\treturn 0;\n \n-\tif (!(si-\u003eflags \u0026 SWP_SOLIDSTATE)) {\n-\t\t/* Serialize HDD SWAP allocation for each device. /\n-\t\tspin_lock(\u0026si-\u003eglobal_cluster_lock);\n+restart:\n+\tswap_alloc_lock_device(si);\n+\tif (si-\u003eflags \u0026 SWP_SOLIDSTATE)\n+\t\toffset = __this_cpu_read(si-\u003epercpu_cluster-\u003enext[order]);\n+\telse\n \t\toffset = si-\u003eglobal_cluster-\u003enext[order];\n-\t\tif (offset == SWAP_ENTRY_INVALID)\n-\t\t\tgoto new_cluster;\n \n+\tif (offset != SWAP_ENTRY_INVALID) {\n \t\tci = swap_cluster_lock(si, offset);\n \t\t/ Cluster could have been used by another order /\n \t\tif (cluster_is_usable(ci, order)) {\n@@ -1123,7 +1467,6 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct si,\n \t\t\tgoto done;\n \t}\n \n-new_cluster:\n \t/\n \t If the device need discard, prefer new cluster over nonfull\n \t * to spread out the writes.\n@@ -1132,6 +1475,12 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct si,\n \t\tfound = alloc_swap_scan_list(si, \u0026si-\u003efree_clusters, folio, false);\n \t\tif (found)\n \t\t\tgoto done;\n+\n+\t\tif (!list_empty(\u0026si-\u003ediscard_clusters)) {\n+\t\t\tswap_alloc_unlock_device(si);\n+\t\t\tswap_do_scheduled_discard(si);\n+\t\t\tgoto restart;\n+\t\t}\n \t}\n \n \tif (order \u003c PMD_ORDER) {\n@@ -1180,8 +1529,7 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct si,\n \t\t\tgoto done;\n \t}\n done:\n-\tif (!(si-\u003eflags \u0026 SWP_SOLIDSTATE))\n-\t\tspin_unlock(\u0026si-\u003eglobal_cluster_lock);\n+\tswap_alloc_unlock_device(si);\n \n \treturn found;\n }\n@@ -1191,54 +1539,44 @@ static void del_from_avail_list(struct swap_info_struct si, bool swapoff)\n {\n \tunsigned long pages;\n \n-\tspin_lock(\u0026swap_avail_lock);\n+\tspin_lock(\u0026swap_queue_update_lock);\n \n-\tif (swapoff) {\n-\t\t/\n-\t\t * Forcefully remove it. Clear the SWP_WRITEOK flags for\n-\t\t * swapoff here so it's synchronized by both si-\u003elock and\n-\t\t * swap_avail_lock, to ensure the result can be seen by\n-\t\t * add_to_avail_list.\n-\t\t /\n-\t\tlockdep_assert_held(\u0026si-\u003elock);\n-\t\tsi-\u003eflags \u0026= ~SWP_WRITEOK;\n-\t\tatomic_long_or(SWAP_USAGE_OFFLIST_BIT, \u0026si-\u003einuse_pages);\n-\t} else {\n-\t\t/\n-\t\t * If not called by swapoff, take it off-list only if it's\n-\t\t * full and SWAP_USAGE_OFFLIST_BIT is not set (strictly\n-\t\t * si-\u003einuse_pages == pages), any concurrent slot freeing,\n-\t\t * or device already removed from plist by someone else\n-\t\t * will make this return false.\n-\t\t /\n+\t/\n+\t * Force remove it only for swapoff. Else, take it off-list only if\n+\t * it's full and SWAP_USAGE_OFFLIST_BIT is not set (strictly\n+\t * si-\u003einuse_pages == pages), so concurrent slot freeing, or\n+\t * concurrent list removal will make the cmpxchg fail and skip\n+\t * the removal.\n+\t /\n+\tif (!swapoff) {\n \t\tpages = si-\u003epages;\n \t\tif (!atomic_long_try_cmpxchg(\u0026si-\u003einuse_pages, \u0026pages,\n \t\t\t\t\t pages \| SWAP_USAGE_OFFLIST_BIT))\n \t\t\tgoto skip;\n+\t} else {\n+\t\tatomic_long_or(SWAP_USAGE_OFFLIST_BIT, \u0026si-\u003einuse_pages);\n \t}\n \n-\tplist_del(\u0026si-\u003eavail_list, \u0026swap_avail_head);\n-\n+\tswap_queue_mask(si);\n skip:\n-\tspin_unlock(\u0026swap_avail_lock);\n+\tspin_unlock(\u0026swap_queue_update_lock);\n }\n \n / SWAP_USAGE_OFFLIST_BIT can only be cleared by this helper. /\n-static void add_to_avail_list(struct swap_info_struct si, bool swapon)\n+static void add_to_avail_list(struct swap_info_struct si)\n {\n \tlong val;\n \tunsigned long pages;\n \n-\tspin_lock(\u0026swap_avail_lock);\n+\tspin_lock(\u0026swap_queue_update_lock);\n \n-\t/ Corresponding to SWP_WRITEOK clearing in del_from_avail_list /\n-\tif (swapon) {\n-\t\tlockdep_assert_held(\u0026si-\u003elock);\n-\t\tsi-\u003eflags \|= SWP_WRITEOK;\n-\t} else {\n-\t\tif (!(READ_ONCE(si-\u003eflags) \u0026 SWP_WRITEOK))\n-\t\t\tgoto skip;\n-\t}\n+\t/\n+\t * Mark the device as avail if SWP_WRITEOK is set. Swapoff clears\n+\t * SWP_WRITEOK first, so check that first so the device won't be\n+\t * re-added after swapoff started.\n+\t /\n+\tif (!(si-\u003eflags \u0026 SWP_WRITEOK))\n+\t\tgoto skip;\n \n \tif (!(atomic_long_read(\u0026si-\u003einuse_pages) \u0026 SWAP_USAGE_OFFLIST_BIT))\n \t\tgoto skip;\n@@ -1246,9 +1584,10 @@ static void add_to_avail_list(struct swap_info_struct si, bool swapon)\n \tval = atomic_long_fetch_and_relaxed(~SWAP_USAGE_OFFLIST_BIT, \u0026si-\u003einuse_pages);\n \n \t/\n-\t When device is full and device is on the plist, only one updater will\n-\t * see (inuse_pages == si-\u003epages) and will call del_from_avail_list. If\n-\t * that updater happen to be here, just skip adding.\n+\t * When device is full and marked as available, one reader will see\n+\t * (inuse_pages == si-\u003epages) and should mark it as unavailable and\n+\t * set SWAP_USAGE_OFFLIST_BIT. If that updater happens to be here, just\n+\t * skip the rest.\n \t /\n \tpages = si-\u003epages;\n \tif (val == pages) {\n@@ -1258,60 +1597,42 @@ static void add_to_avail_list(struct swap_info_struct si, bool swapon)\n \t\t\tgoto skip;\n \t}\n \n-\tplist_add(\u0026si-\u003eavail_list, \u0026swap_avail_head);\n-\n+\tswap_queue_unmask(si);\n skip:\n-\tspin_unlock(\u0026swap_avail_lock);\n+\tspin_unlock(\u0026swap_queue_update_lock);\n }\n \n /\n- swap_usage_add / swap_usage_sub of each slot are serialized by ci-\u003elock\n- * within each cluster, so the total contribution to the global counter should\n- * always be positive and cannot exceed the total number of usable slots.\n+ * swap_device_inuse_add / swap_device_inuse_sub are called within cluster\n+ * update critical sections and serialized by ci-\u003elock within each cluster,\n+ * so the total contribution to the global counter should always be positive\n+ * and cannot exceed the total number of usable slots.\n /\n-static bool swap_usage_add(struct swap_info_struct si, unsigned int nr_entries)\n+static void swap_device_inuse_add(struct swap_info_struct si,\n+\t\t\t\t unsigned int nr_entries)\n {\n-\tlong val = atomic_long_add_return_relaxed(nr_entries, \u0026si-\u003einuse_pages);\n+\tlong inuse_pages;\n \n \t/\n \t * If device is full, and SWAP_USAGE_OFFLIST_BIT is not set,\n-\t * remove it from the plist.\n-\t /\n-\tif (unlikely(val == si-\u003epages)) {\n-\t\tdel_from_avail_list(si, false);\n-\t\treturn true;\n-\t}\n-\n-\treturn false;\n-}\n-\n-static void swap_usage_sub(struct swap_info_struct si, unsigned int nr_entries)\n-{\n-\tlong val = atomic_long_sub_return_relaxed(nr_entries, \u0026si-\u003einuse_pages);\n-\n-\t/\n-\t If device is not full, and SWAP_USAGE_OFFLIST_BIT is set,\n-\t * add it to the plist.\n+\t * mark it unavailable.\n \t /\n-\tif (unlikely(val \u0026 SWAP_USAGE_OFFLIST_BIT))\n-\t\tadd_to_avail_list(si, false);\n-}\n-\n-static void swap_range_alloc(struct swap_info_struct si,\n-\t\t\t unsigned int nr_entries)\n-{\n-\tif (swap_usage_add(si, nr_entries)) {\n+\tinuse_pages = atomic_long_add_return_relaxed(nr_entries, \u0026si-\u003einuse_pages);\n+\tif (unlikely(inuse_pages == si-\u003epages)) {\n \t\tif (vm_swap_full())\n \t\t\tschedule_work(\u0026si-\u003ereclaim_work);\n+\t\tdel_from_avail_list(si, false);\n \t}\n+\n \tatomic_long_sub(nr_entries, \u0026nr_swap_pages);\n }\n \n-static void swap_range_free(struct swap_info_struct si, unsigned long offset,\n-\t\t\t unsigned int nr_entries)\n+static void swap_device_inuse_sub(struct swap_info_struct si, unsigned long offset,\n+\t\t\t\t unsigned int nr_entries)\n {\n \tunsigned long end = offset + nr_entries - 1;\n \tvoid (swap_slot_free_notify)(struct block_device , unsigned long);\n+\tlong inuse_pages;\n \tunsigned int i;\n \n \tfor (i = 0; i \u003c nr_entries; i++)\n@@ -1335,7 +1656,14 @@ static void swap_range_free(struct swap_info_struct si, unsigned long offset,\n \t /\n \tsmp_wmb();\n \tatomic_long_add(nr_entries, \u0026nr_swap_pages);\n-\tswap_usage_sub(si, nr_entries);\n+\n+\t/\n+\t If device is not full, and SWAP_USAGE_OFFLIST_BIT is set,\n+\t * add it back to the available queue.\n+\t /\n+\tinuse_pages = atomic_long_sub_return_relaxed(nr_entries, \u0026si-\u003einuse_pages);\n+\tif (unlikely(inuse_pages \u0026 SWAP_USAGE_OFFLIST_BIT))\n+\t\tadd_to_avail_list(si);\n }\n \n static bool get_swap_device_info(struct swap_info_struct si)\n@@ -1345,113 +1673,49 @@ static bool get_swap_device_info(struct swap_info_struct si)\n \t/\n \t * Guarantee the si-\u003eusers are checked before accessing other\n \t * fields of swap_info_struct, and si-\u003eflags (SWP_WRITEOK) is\n-\t * up to dated.\n+\t * up to date.\n \t \n-\t Paired with the spin_unlock() after setup_swap_info() in\n-\t * enable_swap_info(), and smp_wmb() in swapoff.\n+\t * Paired with percpu_up_write() in swap_device_enable(), and\n+\t * smp_wmb() after clearing SWP_WRITEOK in swapoff.\n \t /\n \tsmp_rmb();\n \treturn true;\n }\n \n-/\n- * Fast path try to get swap entries with specified order from current\n- * CPU's swap entry pool (a cluster).\n- /\n-static bool swap_alloc_fast(struct folio folio)\n+static int swap_alloc_entry(struct folio folio)\n {\n-\tunsigned int order = folio_order(folio);\n-\tstruct swap_cluster_info ci;\n+\tlong nr_pages = folio_nr_pages(folio);\n \tstruct swap_info_struct si;\n-\tunsigned int offset;\n-\n-\t/\n-\t * Once allocated, swap_info_struct will never be completely freed,\n-\t * so checking it's liveness by get_swap_device_info is enough.\n-\t /\n-\tsi = this_cpu_read(percpu_swap_cluster.si[order]);\n-\toffset = this_cpu_read(percpu_swap_cluster.offset[order]);\n-\tif (!si \|\| !offset \|\| !get_swap_device_info(si))\n-\t\treturn false;\n-\n-\tci = swap_cluster_lock(si, offset);\n-\tif (cluster_is_usable(ci, order)) {\n-\t\tif (cluster_is_empty(ci))\n-\t\t\toffset = cluster_offset(si, ci);\n-\t\talloc_swap_scan_cluster(si, ci, folio, offset);\n-\t} else {\n-\t\tswap_cluster_unlock(ci);\n-\t}\n+\tint nr_iter, ret;\n+\n+\tpercpu_down_read(\u0026swapon_rwsem);\n+\tfor (nr_iter = 0;; nr_iter++) {\n+\t\tsi = swap_queue_get_device(nr_pages, nr_iter);\n+\t\tif (IS_ERR(si)) {\n+\t\t\tret = PTR_ERR(si);\n+\t\t\tif (ret == -EBUSY)\n+\t\t\t\tcontinue;\n+\t\t\tbreak;\n+\t\t}\n+\t\tcluster_alloc_swap_entry(si, folio);\n \n-\tput_swap_device(si);\n-\treturn folio_test_swapcache(folio);\n-}\n-\n-/ Rotate the device and switch to a new cluster /\n-static void swap_alloc_slow(struct folio folio)\n-{\n-\tstruct swap_info_struct si, next;\n-\n-\tspin_lock(\u0026swap_avail_lock);\n-start_over:\n-\tplist_for_each_entry_safe(si, next, \u0026swap_avail_head, avail_list) {\n-\t\t/* Rotate the device and switch to a new cluster /\n-\t\tplist_requeue(\u0026si-\u003eavail_list, \u0026swap_avail_head);\n-\t\tspin_unlock(\u0026swap_avail_lock);\n-\t\tif (get_swap_device_info(si)) {\n-\t\t\tcluster_alloc_swap_entry(si, folio);\n-\t\t\tput_swap_device(si);\n-\t\t\tif (folio_test_swapcache(folio))\n-\t\t\t\treturn;\n-\t\t\tif (folio_test_large(folio))\n-\t\t\t\treturn;\n+\t\tif (folio_test_swapcache(folio)) {\n+\t\t\tret = 0;\n+\t\t\tbreak;\n \t\t}\n \n-\t\tspin_lock(\u0026swap_avail_lock);\n \t\t/\n-\t\t * if we got here, it's likely that si was almost full before,\n-\t\t * multiple callers probably all tried to get a page from the\n-\t\t * same si and it filled up before we could get one; or, the si\n-\t\t * filled up between us dropping swap_avail_lock.\n-\t\t * Since we dropped the swap_avail_lock, the swap_avail_list\n-\t\t * may have been modified; so if next is still in the\n-\t\t * swap_avail_head list then try it, otherwise start over if we\n-\t\t * have not gotten any slots.\n+\t\t * For large allocation, return error directly to inform the\n+\t\t * caller to split it instead of fallback to other devices.\n \t\t /\n-\t\tif (plist_node_empty(\u0026next-\u003eavail_list))\n-\t\t\tgoto start_over;\n-\t}\n-\tspin_unlock(\u0026swap_avail_lock);\n-}\n-\n-/\n- * Discard pending clusters in a synchronized way when under high pressure.\n- * Return: true if any cluster is discarded.\n- /\n-static bool swap_sync_discard(void)\n-{\n-\tbool ret = false;\n-\tstruct swap_info_struct si, next;\n-\n-\tspin_lock(\u0026swap_lock);\n-start_over:\n-\tplist_for_each_entry_safe(si, next, \u0026swap_active_head, list) {\n-\t\tspin_unlock(\u0026swap_lock);\n-\t\tif (get_swap_device_info(si)) {\n-\t\t\tif (si-\u003eflags \u0026 SWP_PAGE_DISCARD)\n-\t\t\t\tret = swap_do_scheduled_discard(si);\n-\t\t\tput_swap_device(si);\n+\t\tif (folio_test_large(folio)) {\n+\t\t\tret = -E2BIG;\n+\t\t\tbreak;\n \t\t}\n-\t\tif (ret)\n-\t\t\treturn true;\n-\n-\t\tspin_lock(\u0026swap_lock);\n-\t\tif (plist_node_empty(\u0026next-\u003elist))\n-\t\t\tgoto start_over;\n \t}\n-\tspin_unlock(\u0026swap_lock);\n \n-\treturn false;\n+\tpercpu_up_read(\u0026swapon_rwsem);\n+\treturn ret;\n }\n \n static int swap_extend_table_alloc(struct swap_info_struct si,\n@@ -1735,6 +1999,7 @@ int folio_alloc_swap(struct folio folio)\n {\n \tunsigned int order = folio_order(folio);\n \tunsigned int size = 1 \u003c\u003c order;\n+\tint ret;\n \n \tVM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);\n \tVM_BUG_ON_FOLIO(!folio_test_uptodate(folio), folio);\n@@ -1757,23 +2022,14 @@ int folio_alloc_swap(struct folio folio)\n \t\t}\n \t}\n \n-again:\n-\tlocal_lock(\u0026percpu_swap_cluster.lock);\n-\tif (!swap_alloc_fast(folio))\n-\t\tswap_alloc_slow(folio);\n-\tlocal_unlock(\u0026percpu_swap_cluster.lock);\n-\n-\tif (!order \u0026\u0026 unlikely(!folio_test_swapcache(folio))) {\n-\t\tif (swap_sync_discard())\n-\t\t\tgoto again;\n-\t}\n+\tret = swap_alloc_entry(folio);\n \n \t/* Need to call this even if allocation failed, for MEMCG_SWAP_FAIL. /\n \tif (unlikely(mem_cgroup_try_charge_swap(folio)))\n \t\tswap_cache_del_folio(folio);\n \n \tif (unlikely(!folio_test_swapcache(folio)))\n-\t\treturn -ENOMEM;\n+\t\treturn ret ? ret : -ENOMEM;\n \n \treturn 0;\n }\n@@ -1948,7 +2204,7 @@ void __swap_cluster_free_entries(struct swap_info_struct si,\n \tif (batch_id)\n \t\tmem_cgroup_uncharge_swap(batch_id, ci_off - batch_off);\n \n-\tswap_range_free(si, ci_head + ci_start, nr_pages);\n+\tswap_device_inuse_sub(si, ci_head + ci_start, nr_pages);\n \tswap_cluster_assert_empty(ci, ci_start, nr_pages, false);\n \n \tif (!ci-\u003ecount)\n@@ -2150,31 +2406,14 @@ void swap_put_entries_direct(swp_entry_t entry, int nr)\n /\n swp_entry_t swap_alloc_hibernation_slot(int type)\n {\n-\tstruct swap_info_struct pcp_si, si = swap_type_to_info(type);\n-\tunsigned long pcp_offset, offset = SWAP_ENTRY_INVALID;\n-\tstruct swap_cluster_info ci;\n+\tstruct swap_info_struct si = swap_type_to_info(type);\n+\tunsigned long offset = SWAP_ENTRY_INVALID;\n \tswp_entry_t entry = {0};\n \n \tif (!si)\n \t\tgoto fail;\n \n-\t/\n-\t * Try the local cluster first if it matches the device. If\n-\t * not, try grab a new cluster and override local cluster.\n-\t /\n-\tlocal_lock(\u0026percpu_swap_cluster.lock);\n-\tpcp_si = this_cpu_read(percpu_swap_cluster.si[0]);\n-\tpcp_offset = this_cpu_read(percpu_swap_cluster.offset[0]);\n-\tif (pcp_si == si \u0026\u0026 pcp_offset) {\n-\t\tci = swap_cluster_lock(si, pcp_offset);\n-\t\tif (cluster_is_usable(ci, 0))\n-\t\t\toffset = alloc_swap_scan_cluster(si, ci, NULL, pcp_offset);\n-\t\telse\n-\t\t\tswap_cluster_unlock(ci);\n-\t}\n-\tif (!offset)\n-\t\toffset = cluster_alloc_swap_entry(si, NULL);\n-\tlocal_unlock(\u0026percpu_swap_cluster.lock);\n+\toffset = cluster_alloc_swap_entry(si, NULL);\n \tif (offset)\n \t\tentry = swp_entry(si-\u003etype, offset);\n \n@@ -2205,26 +2444,18 @@ void swap_free_hibernation_slot(swp_entry_t entry)\n \n static int __find_hibernation_swap_type(dev_t device, sector_t offset)\n {\n-\tint type;\n-\n-\tlockdep_assert_held(\u0026swap_lock);\n+\tstruct swap_info_struct si;\n \n \tif (!device)\n \t\treturn -EINVAL;\n \n-\tfor (type = 0; type \u003c nr_swapfiles; type++) {\n-\t\tstruct swap_info_struct sis = swap_info[type];\n-\n-\t\tif (!(sis-\u003eflags \u0026 SWP_WRITEOK))\n-\t\t\tcontinue;\n-\n-\t\tif (device == sis-\u003ebdev-\u003ebd_dev) {\n-\t\t\tstruct swap_extent se = first_se(sis);\n-\n-\t\t\tif (se-\u003estart_block == offset)\n-\t\t\t\treturn type;\n+\tfor_each_avail_swap(si) {\n+\t\tif (device == si-\u003ebdev-\u003ebd_dev) {\n+\t\t\tif (first_se(si)-\u003estart_block == offset)\n+\t\t\t\treturn si-\u003etype;\n \t\t}\n \t}\n+\n \treturn -ENODEV;\n }\n \n@@ -2247,21 +2478,18 @@ static int __find_hibernation_swap_type(dev_t device, sector_t offset)\n /\n int pin_hibernation_swap_type(dev_t device, sector_t offset)\n {\n-\tint type;\n+\tint ret;\n \tstruct swap_info_struct si;\n \n-\tspin_lock(\u0026swap_lock);\n-\n-\ttype = __find_hibernation_swap_type(device, offset);\n-\tif (type \u003c 0) {\n-\t\tspin_unlock(\u0026swap_lock);\n-\t\treturn type;\n-\t}\n+\tpercpu_down_write(\u0026swapon_rwsem);\n+\tret = __find_hibernation_swap_type(device, offset);\n+\tif (ret \u003c 0)\n+\t\tgoto out;\n \n-\tsi = swap_type_to_info(type);\n+\tsi = swap_type_to_info(ret);\n \tif (WARN_ON_ONCE(!si)) {\n-\t\tspin_unlock(\u0026swap_lock);\n-\t\treturn -ENODEV;\n+\t\tret = -ENODEV;\n+\t\tgoto out;\n \t}\n \n \t/\n@@ -2270,14 +2498,15 @@ int pin_hibernation_swap_type(dev_t device, sector_t offset)\n \t the same session.\n \t /\n \tif (WARN_ON_ONCE(si-\u003eflags \u0026 SWP_HIBERNATION)) {\n-\t\tspin_unlock(\u0026swap_lock);\n-\t\treturn -EBUSY;\n+\t\tret = -EBUSY;\n+\t\tgoto out;\n \t}\n \n \tsi-\u003eflags \|= SWP_HIBERNATION;\n \n-\tspin_unlock(\u0026swap_lock);\n-\treturn type;\n+out:\n+\tpercpu_up_write(\u0026swapon_rwsem);\n+\treturn ret;\n }\n \n /\n@@ -2294,14 +2523,11 @@ void unpin_hibernation_swap_type(int type)\n {\n \tstruct swap_info_struct si;\n \n-\tspin_lock(\u0026swap_lock);\n+\tpercpu_down_write(\u0026swapon_rwsem);\n \tsi = swap_type_to_info(type);\n-\tif (!si) {\n-\t\tspin_unlock(\u0026swap_lock);\n-\t\treturn;\n-\t}\n-\tsi-\u003eflags \u0026= ~SWP_HIBERNATION;\n-\tspin_unlock(\u0026swap_lock);\n+\tif (si)\n+\t\tsi-\u003eflags \u0026= ~SWP_HIBERNATION;\n+\tpercpu_up_write(\u0026swapon_rwsem);\n }\n \n /*\n@@ -2326,29 +2552,26 @@ int find_hibernation_swap_type(dev_t device, sector_t offset)\n {\n \tint type;\n \n-\tspin_lock(\u0026swap_lock);\n+\tpercpu_down_read(\u0026swapon_rwsem);\n \ttype = __find_hibernation_swap_type(device, offset);\n-\tspin_unlock(\u0026swap_lock);\n+\tpercpu_up_read(\u0026swapon_rwsem);\n \n \treturn type;\n }\n \n int find_first_swap(dev_t device)\n {\n-\tint type;\n-\n-\tspin_lock(\u0026swap_lock);\n-\tfor (type = 0; type \u003c nr_swapfiles; type++) {\n-\t\tstruct swap_info_struct sis = swap_info[type];\n+\tint ret = -ENODEV;\n+\tstruct swap_info_struct si;\n \n-\t\tif (!(sis-\u003eflags \u0026 SWP_WRITEOK))\n-\t\t\tcontinue;\n-\t\tdevice = sis-\u003ebdev-\u003ebd_dev;\n-\t\tspin_unlock(\u0026swap_lock);\n-\t\treturn type;\n+\tpercpu_down_read(\u0026swapon_rwsem);\n+\tfor_each_avail_swap(si) {\n+\t\tdevice = si-\u003ebdev-\u003ebd_dev;\n+\t\tret = si-\u003etype;\n+\t\tbreak;\n \t}\n-\tspin_unlock(\u0026swap_lock);\n-\treturn -ENODEV;\n+\tpercpu_up_read(\u0026swapon_rwsem);\n+\treturn ret;\n }\n \n /\n@@ -2376,7 +2599,7 @@ unsigned int count_swap_pages(int type, int free)\n {\n \tunsigned int n = 0;\n \n-\tspin_lock(\u0026swap_lock);\n+\tpercpu_down_read(\u0026swapon_rwsem);\n \tif ((unsigned int)type \u003c nr_swapfiles) {\n \t\tstruct swap_info_struct sis = swap_info[type];\n \n@@ -2388,7 +2611,7 @@ unsigned int count_swap_pages(int type, int free)\n \t\t}\n \t\tspin_unlock(\u0026sis-\u003elock);\n \t}\n-\tspin_unlock(\u0026swap_lock);\n+\tpercpu_up_read(\u0026swapon_rwsem);\n \treturn n;\n }\n #endif /* CONFIG_HIBERNATION /\n@@ -2700,10 +2923,10 @@ static unsigned int find_next_to_unuse(struct swap_info_struct si,\n \tunsigned long swp_tb;\n \n \t/\n-\t No need for swap_lock here: we're just looking\n+\t * No need for swapon_rwsem here: we're just looking\n \t * for whether an entry is in use, not modifying it; false\n \t * hits are okay, and sys_swapoff() has already prevented new\n-\t * allocations from this area (while holding swap_lock).\n+\t * allocations from this area (while holding swapon_rwsem).\n \t /\n \tfor (i = prev + 1; i \u003c si-\u003emax; i++) {\n \t\tswp_tb = swap_table_get(__swap_offset_to_cluster(si, i),\n@@ -2816,7 +3039,7 @@ static int try_to_unuse(unsigned int type)\n success:\n \t/\n \t * Make sure that further cleanups after try_to_unuse() returns happen\n-\t * after swap_range_free() reduces si-\u003einuse_pages to 0.\n+\t * after swap_device_inuse_sub() reduces si-\u003einuse_pages to 0.\n \t /\n \tsmp_mb();\n \treturn 0;\n@@ -2824,18 +3047,20 @@ static int try_to_unuse(unsigned int type)\n \n /\n * After a successful try_to_unuse, if no swap is now in use, we know\n- * we can empty the mmlist. swap_lock must be held on entry and exit.\n- * Note that mmlist_lock nests inside swap_lock, and an mm must be\n+ * we can empty the mmlist. swapon_rwsem must be held on entry and exit.\n+ * Note that mmlist_lock nests inside swapon_rwsem, and an mm must be\n * added to the mmlist just after page_duplicate - before would be racy.\n /\n static void drain_mmlist(void)\n {\n+\tstruct swap_info_struct si;\n \tstruct list_head p, next;\n-\tunsigned int type;\n \n-\tfor (type = 0; type \u003c nr_swapfiles; type++)\n-\t\tif (swap_usage_in_pages(swap_info[type]))\n+\tfor_each_swap(si) {\n+\t\tif (swap_usage_in_pages(si))\n \t\t\treturn;\n+\t}\n+\n \tspin_lock(\u0026mmlist_lock);\n \tlist_for_each_safe(p, next, \u0026init_mm.mmlist)\n \t\tlist_del_init(p);\n@@ -2968,58 +3193,65 @@ static int setup_swap_extents(struct swap_info_struct sis,\n \treturn generic_swapfile_activate(sis, swap_file, span);\n }\n \n-static void _enable_swap_info(struct swap_info_struct si)\n-{\n-\tatomic_long_add(si-\u003epages, \u0026nr_swap_pages);\n-\ttotal_swap_pages += si-\u003epages;\n-\n-\tassert_spin_locked(\u0026swap_lock);\n-\n-\tplist_add(\u0026si-\u003elist, \u0026swap_active_head);\n-\n-\t/* Add back to available list /\n-\tadd_to_avail_list(si, true);\n-}\n-\n /\n- * Called after the swap device is ready, resurrect its percpu ref, it's now\n- * safe to reference it. Add it to the list to expose it to the allocator.\n+ * Called after the swap device is ready to be used. Marking it writable and\n+ * exposing it to the allocator. Resurrect its percpu ref if it was dead before.\n /\n-static void enable_swap_info(struct swap_info_struct si)\n+static void swap_device_enable(struct swap_info_struct si)\n {\n-\tpercpu_ref_resurrect(\u0026si-\u003eusers);\n-\tspin_lock(\u0026swap_lock);\n-\tspin_lock(\u0026si-\u003elock);\n-\t_enable_swap_info(si);\n-\tspin_unlock(\u0026si-\u003elock);\n-\tspin_unlock(\u0026swap_lock);\n-}\n+\tpercpu_down_write(\u0026swapon_rwsem);\n+\tspin_lock(\u0026swap_queue_update_lock);\n+\tsi-\u003eflags \|= SWP_WRITEOK;\n+\tspin_unlock(\u0026swap_queue_update_lock);\n+\tatomic_long_add(si-\u003epages, \u0026nr_swap_pages);\n+\ttotal_swap_pages += si-\u003epages;\n+\tpercpu_up_write(\u0026swapon_rwsem);\n \n-static void reinsert_swap_info(struct swap_info_struct si)\n-{\n-\tspin_lock(\u0026swap_lock);\n-\tspin_lock(\u0026si-\u003elock);\n-\t_enable_swap_info(si);\n-\tspin_unlock(\u0026si-\u003elock);\n-\tspin_unlock(\u0026swap_lock);\n+\tadd_to_avail_list(si);\n }\n \n-/\n- Called after clearing SWP_WRITEOK, ensures cluster_alloc_range\n- * see the updated flags, so there will be no more allocations.\n- /\n-static void wait_for_allocation(struct swap_info_struct si)\n+static int swap_device_disable(struct swap_info_struct si)\n {\n \tunsigned long offset;\n \tunsigned long end = ALIGN(si-\u003emax, SWAPFILE_CLUSTER);\n \tstruct swap_cluster_info ci;\n \n-\tBUG_ON(si-\u003eflags \u0026 SWP_WRITEOK);\n+\t/\n+\t If SWP_WRITEOK is not set: another process already disabling it.\n+\t * If SWP_HIBERNATION is set: the device is pinned for hibernation.\n+\t /\n+\tpercpu_down_write(\u0026swapon_rwsem);\n+\tif (!(si-\u003eflags \u0026 SWP_WRITEOK) \|\|\n+\t si-\u003eflags \u0026 SWP_HIBERNATION) {\n+\t\tpercpu_up_write(\u0026swapon_rwsem);\n+\t\treturn -EBUSY;\n+\t}\n+\n+\tif (security_vm_enough_memory_mm(current-\u003emm, si-\u003epages)) {\n+\t\tpercpu_up_write(\u0026swapon_rwsem);\n+\t\treturn -ENOMEM;\n+\t}\n+\tvm_unacct_memory(si-\u003epages);\n+\n+\tspin_lock(\u0026swap_queue_update_lock);\n+\tsi-\u003eflags \u0026= ~SWP_WRITEOK;\n+\tspin_unlock(\u0026swap_queue_update_lock);\n+\ttotal_swap_pages -= si-\u003epages;\n+\tatomic_long_sub(si-\u003epages, \u0026nr_swap_pages);\n+\tdel_from_avail_list(si, true);\n+\tpercpu_up_write(\u0026swapon_rwsem);\n \n+\t/\n+\t * Swap allocator doesn't touch si lock, so looping through all\n+\t * ci locks ensures __swap_cluster_alloc_entries sees the\n+\t * updated flags, and no more allocations will occur.\n+\t /\n \tfor (offset = 0; offset \u003c end; offset += SWAPFILE_CLUSTER) {\n \t\tci = swap_cluster_lock(si, offset);\n \t\tswap_cluster_unlock(ci);\n \t}\n+\n+\treturn 0;\n }\n \n static void free_swap_cluster_info(struct swap_cluster_info cluster_info,\n@@ -3043,28 +3275,6 @@ static void free_swap_cluster_info(struct swap_cluster_info cluster_info,\n \tkvfree(cluster_info);\n }\n \n-/\n- * Called after swap device's reference count is dead, so\n- * neither scan nor allocation will use it.\n- /\n-static void flush_percpu_swap_cluster(struct swap_info_struct si)\n-{\n-\tint cpu, i;\n-\tstruct swap_info_struct *pcp_si;\n-\n-\tfor_each_possible_cpu(cpu) {\n-\t\tpcp_si = per_cpu_ptr(percpu_swap_cluster.si, cpu);\n-\t\t/\n-\t\t * Invalidate the percpu swap cluster cache, si-\u003eusers\n-\t\t * is dead, so no new user will point to it, just flush\n-\t\t * any existing user.\n-\t\t /\n-\t\tfor (i = 0; i \u003c SWAP_NR_ORDERS; i++)\n-\t\t\tcmpxchg(\u0026pcp_si[i], si, NULL);\n-\t}\n-}\n-\n-\n SYSCALL_DEFINE1(swapoff, const char __user , specialfile)\n {\n \tstruct swap_info_struct p = NULL;\n@@ -3086,44 +3296,22 @@ SYSCALL_DEFINE1(swapoff, const char __user , specialfile)\n \t\treturn PTR_ERR(victim);\n \n \tmapping = victim-\u003ef_mapping;\n-\tspin_lock(\u0026swap_lock);\n-\tplist_for_each_entry(p, \u0026swap_active_head, list) {\n-\t\tif (p-\u003eflags \u0026 SWP_WRITEOK) {\n-\t\t\tif (p-\u003eswap_file-\u003ef_mapping == mapping) {\n-\t\t\t\tfound = 1;\n-\t\t\t\tbreak;\n-\t\t\t}\n+\tpercpu_down_read(\u0026swapon_rwsem);\n+\tfor_each_avail_swap(p) {\n+\t\tif (p-\u003eswap_file-\u003ef_mapping == mapping) {\n+\t\t\tfound = 1;\n+\t\t\tbreak;\n \t\t}\n \t}\n-\tif (!found) {\n-\t\terr = -EINVAL;\n-\t\tspin_unlock(\u0026swap_lock);\n-\t\tgoto out_dput;\n-\t}\n-\n-\t/* Refuse swapoff while the device is pinned for hibernation /\n-\tif (p-\u003eflags \u0026 SWP_HIBERNATION) {\n-\t\terr = -EBUSY;\n-\t\tspin_unlock(\u0026swap_lock);\n-\t\tgoto out_dput;\n-\t}\n+\tpercpu_up_read(\u0026swapon_rwsem);\n+\tfilp_close(victim, NULL);\n \n-\tif (!security_vm_enough_memory_mm(current-\u003emm, p-\u003epages))\n-\t\tvm_unacct_memory(p-\u003epages);\n-\telse {\n-\t\terr = -ENOMEM;\n-\t\tspin_unlock(\u0026swap_lock);\n-\t\tgoto out_dput;\n-\t}\n-\tspin_lock(\u0026p-\u003elock);\n-\tdel_from_avail_list(p, true);\n-\tplist_del(\u0026p-\u003elist, \u0026swap_active_head);\n-\tatomic_long_sub(p-\u003epages, \u0026nr_swap_pages);\n-\ttotal_swap_pages -= p-\u003epages;\n-\tspin_unlock(\u0026p-\u003elock);\n-\tspin_unlock(\u0026swap_lock);\n+\tif (!found)\n+\t\treturn -EINVAL;\n \n-\twait_for_allocation(p);\n+\terr = swap_device_disable(p);\n+\tif (err)\n+\t\treturn err;\n \n \tset_current_oom_origin();\n \terr = try_to_unuse(p-\u003etype);\n@@ -3131,10 +3319,13 @@ SYSCALL_DEFINE1(swapoff, const char __user , specialfile)\n \n \tif (err) {\n \t\t/* re-insert swap space back into swap_list /\n-\t\treinsert_swap_info(p);\n-\t\tgoto out_dput;\n+\t\tswap_device_enable(p);\n+\t\treturn err;\n \t}\n \n+\tpercpu_down_write(\u0026swapon_rwsem);\n+\tswap_queue_del(p);\n+\n \t/\n \t * Wait for swap operations protected by get/put_swap_device()\n \t * to complete. Because of synchronize_rcu() here, all swap\n@@ -3149,15 +3340,12 @@ SYSCALL_DEFINE1(swapoff, const char __user , specialfile)\n \n \tflush_work(\u0026p-\u003ediscard_work);\n \tflush_work(\u0026p-\u003ereclaim_work);\n-\tflush_percpu_swap_cluster(p);\n \n \tdestroy_swap_extents(p, p-\u003eswap_file);\n \n \tif (!(p-\u003eflags \u0026 SWP_SOLIDSTATE))\n \t\tatomic_dec(\u0026nr_rotate_swap);\n \n-\tmutex_lock(\u0026swapon_mutex);\n-\tspin_lock(\u0026swap_lock);\n \tspin_lock(\u0026p-\u003elock);\n \tdrain_mmlist();\n \n@@ -3168,10 +3356,11 @@ SYSCALL_DEFINE1(swapoff, const char __user , specialfile)\n \tp-\u003emax = 0;\n \tp-\u003ecluster_info = NULL;\n \tspin_unlock(\u0026p-\u003elock);\n-\tspin_unlock(\u0026swap_lock);\n+\tpercpu_up_write(\u0026swapon_rwsem);\n \tarch_swap_invalidate_area(p-\u003etype);\n \tzswap_swapoff(p-\u003etype);\n-\tmutex_unlock(\u0026swapon_mutex);\n+\tfree_percpu(p-\u003epercpu_cluster);\n+\tp-\u003epercpu_cluster = NULL;\n \tkfree(p-\u003eglobal_cluster);\n \tp-\u003eglobal_cluster = NULL;\n \tfree_swap_cluster_info(cluster_info, maxpages);\n@@ -3187,18 +3376,19 @@ SYSCALL_DEFINE1(swapoff, const char __user , specialfile)\n \t Clear the SWP_USED flag after all resources are freed so that swapon\n \t * can reuse this swap_info in alloc_swap_info() safely. It is ok to\n \t * not hold p-\u003elock after we cleared its SWP_WRITEOK.\n+\t \n+\t The write lock ensures the flag clear is visible to lockless\n+\t * readers of swap_type_to_info() before alloc_swap_info() reuses\n+\t * this slot.\n \t /\n-\tspin_lock(\u0026swap_lock);\n+\tpercpu_down_write(\u0026swapon_rwsem);\n \tp-\u003eflags = 0;\n-\tspin_unlock(\u0026swap_lock);\n+\tpercpu_up_write(\u0026swapon_rwsem);\n \n-\terr = 0;\n \tatomic_inc(\u0026proc_poll_event);\n \twake_up_interruptible(\u0026proc_poll_wait);\n \n-out_dput:\n-\tfilp_close(victim, NULL);\n-\treturn err;\n+\treturn 0;\n }\n \n #ifdef CONFIG_PROC_FS\n@@ -3216,20 +3406,18 @@ static __poll_t swaps_poll(struct file file, poll_table wait)\n \treturn EPOLLIN \| EPOLLRDNORM;\n }\n \n-/ iterator /\n static void swap_start(struct seq_file swap, loff_t pos)\n {\n \tstruct swap_info_struct si;\n-\tint type;\n \tloff_t l = pos;\n \n-\tmutex_lock(\u0026swapon_mutex);\n+\tpercpu_down_read(\u0026swapon_rwsem);\n \n \tif (!l)\n \t\treturn SEQ_START_TOKEN;\n \n-\tfor (type = 0; (si = swap_type_to_info(type)); type++) {\n-\t\tif (!(si-\u003eswap_file))\n+\tfor_each_swap(si) {\n+\t\tif (!si-\u003eswap_file)\n \t\t\tcontinue;\n \t\tif (!--l)\n \t\t\treturn si;\n@@ -3250,7 +3438,7 @@ static void swap_next(struct seq_file swap, void v, loff_t pos)\n \n \t++(pos);\n \tfor (; (si = swap_type_to_info(type)); type++) {\n-\t\tif (!(si-\u003eswap_file))\n+\t\tif (!(si-\u003eflags \u0026 SWP_USED) \|\| !(si-\u003eswap_file))\n \t\t\tcontinue;\n \t\treturn si;\n \t}\n@@ -3260,7 +3448,7 @@ static void swap_next(struct seq_file swap, void v, loff_t pos)\n \n static void swap_stop(struct seq_file swap, void v)\n {\n-\tmutex_unlock(\u0026swapon_mutex);\n+\tpercpu_up_read(\u0026swapon_rwsem);\n }\n \n static int swap_show(struct seq_file swap, void v)\n@@ -3353,13 +3541,13 @@ static struct swap_info_struct alloc_swap_info(void)\n \t\treturn ERR_PTR(-ENOMEM);\n \t}\n \n-\tspin_lock(\u0026swap_lock);\n+\tpercpu_down_write(\u0026swapon_rwsem);\n \tfor (type = 0; type \u003c nr_swapfiles; type++) {\n \t\tif (!(swap_info[type]-\u003eflags \u0026 SWP_USED))\n \t\t\tbreak;\n \t}\n \tif (type \u003e= MAX_SWAPFILES) {\n-\t\tspin_unlock(\u0026swap_lock);\n+\t\tpercpu_up_write(\u0026swapon_rwsem);\n \t\tpercpu_ref_exit(\u0026p-\u003eusers);\n \t\tkvfree(p);\n \t\treturn ERR_PTR(-EPERM);\n@@ -3381,10 +3569,8 @@ static struct swap_info_struct alloc_swap_info(void)\n \t\t /\n \t}\n \tp-\u003eswap_extent_root = RB_ROOT;\n-\tplist_node_init(\u0026p-\u003elist, 0);\n-\tplist_node_init(\u0026p-\u003eavail_list, 0);\n \tp-\u003eflags = SWP_USED;\n-\tspin_unlock(\u0026swap_lock);\n+\tpercpu_up_write(\u0026swapon_rwsem);\n \tif (defer) {\n \t\tpercpu_ref_exit(\u0026defer-\u003eusers);\n \t\tkvfree(defer);\n@@ -3453,9 +3639,8 @@ __weak unsigned long arch_max_swapfile_size(void)\n \treturn generic_max_swapfile_size();\n }\n \n-static unsigned long read_swap_header(struct swap_info_struct si,\n-\t\t\t\t\tunion swap_header swap_header,\n-\t\t\t\t\tstruct inode inode)\n+static unsigned long read_swap_header(union swap_header swap_header,\n+\t\t\t\t struct inode inode)\n {\n \tint i;\n \tunsigned long maxpages;\n@@ -3522,7 +3707,7 @@ static int setup_swap_clusters_info(struct swap_info_struct si,\n {\n \tunsigned long nr_clusters = DIV_ROUND_UP(maxpages, SWAPFILE_CLUSTER);\n \tstruct swap_cluster_info cluster_info;\n-\tint err = -ENOMEM;\n+\tint cpu, err = -ENOMEM;\n \tunsigned long i;\n \n \tcluster_info = kvzalloc_objs(cluster_info, nr_clusters);\n@@ -3532,13 +3717,26 @@ static int setup_swap_clusters_info(struct swap_info_struct si,\n \tfor (i = 0; i \u003c nr_clusters; i++)\n \t\tspin_lock_init(\u0026cluster_info[i].lock);\n \n-\tif (!(si-\u003eflags \u0026 SWP_SOLIDSTATE)) {\n+\tif (si-\u003eflags \u0026 SWP_SOLIDSTATE) {\n+\t\tsi-\u003epercpu_cluster = alloc_percpu(struct percpu_cluster);\n+\t\tif (!si-\u003epercpu_cluster)\n+\t\t\tgoto err;\n+\n+\t\tfor_each_possible_cpu(cpu) {\n+\t\t\tstruct percpu_cluster cluster;\n+\n+\t\t\tcluster = per_cpu_ptr(si-\u003epercpu_cluster, cpu);\n+\t\t\tfor (i = 0; i \u003c SWAP_NR_ORDERS; i++)\n+\t\t\t\tcluster-\u003enext[i] = SWAP_ENTRY_INVALID;\n+\t\t\tlocal_lock_init(\u0026cluster-\u003elock);\n+\t\t}\n+\t} else {\n \t\tsi-\u003eglobal_cluster = kmalloc_obj(si-\u003eglobal_cluster);\n \t\tif (!si-\u003eglobal_cluster)\n \t\t\tgoto err;\n \t\tfor (i = 0; i \u003c SWAP_NR_ORDERS; i++)\n \t\t\tsi-\u003eglobal_cluster-\u003enext[i] = SWAP_ENTRY_INVALID;\n-\t\tspin_lock_init(\u0026si-\u003eglobal_cluster_lock);\n+\t\tspin_lock_init(\u0026si-\u003eglobal_cluster-\u003elock);\n \t}\n \n \t/\n@@ -3621,7 +3819,7 @@ SYSCALL_DEFINE2(swapon, const char __user , specialfile, int, swap_flags)\n \t/\n \t * Allocate or reuse existing !SWP_USED swap_info. The returned\n \t * si will stay in a dying status, so nothing will access its content\n-\t * until enable_swap_info resurrects its percpu ref and expose it.\n+\t * until swap_device_enable resurrects its percpu ref and expose it.\n \t /\n \tsi = alloc_swap_info();\n \tif (IS_ERR(si))\n@@ -3679,7 +3877,7 @@ SYSCALL_DEFINE2(swapon, const char __user , specialfile, int, swap_flags)\n \t}\n \tswap_header = kmap_local_folio(folio, 0);\n \n-\tmaxpages = read_swap_header(si, swap_header, inode);\n+\tmaxpages = read_swap_header(swap_header, inode);\n \tif (unlikely(!maxpages)) {\n \t\terror = -EINVAL;\n \t\tgoto bad_swap_unlock_inode;\n@@ -3700,11 +3898,6 @@ SYSCALL_DEFINE2(swapon, const char __user , specialfile, int, swap_flags)\n \n \tmaxpages = si-\u003emax;\n \n-\t/ Set up the swap cluster info /\n-\terror = setup_swap_clusters_info(si, swap_header, maxpages);\n-\tif (error)\n-\t\tgoto bad_swap_unlock_inode;\n-\n \tif (si-\u003ebdev \u0026\u0026 bdev_stable_writes(si-\u003ebdev))\n \t\tsi-\u003eflags \|= SWP_STABLE_WRITES;\n \n@@ -3718,6 +3911,15 @@ SYSCALL_DEFINE2(swapon, const char __user , specialfile, int, swap_flags)\n \t\tinced_nr_rotate_swap = true;\n \t}\n \n+\t/\n+\t Set up the swap cluster info. This must run after SWP_SOLIDSTATE\n+\t * is determined above, as it decides whether to allocate the per-CPU\n+\t * cluster (solid state) or the global cluster (rotational).\n+\t /\n+\terror = setup_swap_clusters_info(si, swap_header, maxpages);\n+\tif (error)\n+\t\tgoto bad_swap_unlock_inode;\n+\n \tif ((swap_flags \u0026 SWAP_FLAG_DISCARD) \u0026\u0026\n \t si-\u003ebdev \u0026\u0026 bdev_max_discard_sectors(si-\u003ebdev)) {\n \t\t/\n@@ -3764,22 +3966,22 @@ SYSCALL_DEFINE2(swapon, const char __user , specialfile, int, swap_flags)\n \t\tgoto free_swap_zswap;\n \t}\n \n-\tmutex_lock(\u0026swapon_mutex);\n \tprio = DEF_SWAP_PRIO;\n \tif (swap_flags \u0026 SWAP_FLAG_PREFER)\n \t\tprio = swap_flags \u0026 SWAP_FLAG_PRIO_MASK;\n \n-\t/\n-\t * The plist prio is negated because plist ordering is\n-\t * low-to-high, while swap ordering is high-to-low\n-\t /\n \tsi-\u003eprio = prio;\n-\tsi-\u003elist.prio = -si-\u003eprio;\n-\tsi-\u003eavail_list.prio = -si-\u003eprio;\n \tsi-\u003eswap_file = swap_file;\n \n \t/ Sets SWP_WRITEOK, resurrect the percpu ref, expose the swap device /\n-\tenable_swap_info(si);\n+\tpercpu_ref_resurrect(\u0026si-\u003eusers);\n+\tpercpu_down_write(\u0026swapon_rwsem);\n+\terror = swap_queue_add(si);\n+\tpercpu_up_write(\u0026swapon_rwsem);\n+\tif (error)\n+\t\tgoto free_swap_zswap;\n+\n+\tswap_device_enable(si);\n \n \tpr_info(\"Adding %uk swap on %s. Priority:%d extents:%d across:%lluk %s%s%s%s\\n\",\n \t\tK(si-\u003epages), name-\u003ename, si-\u003eprio, nr_extents,\n@@ -3789,7 +3991,6 @@ SYSCALL_DEFINE2(swapon, const char __user , specialfile, int, swap_flags)\n \t\t(si-\u003eflags \u0026 SWP_AREA_DISCARD) ? \"s\" : \"\",\n \t\t(si-\u003eflags \u0026 SWP_PAGE_DISCARD) ? \"c\" : \"\");\n \n-\tmutex_unlock(\u0026swapon_mutex);\n \tatomic_inc(\u0026proc_poll_event);\n \twake_up_interruptible(\u0026proc_poll_wait);\n \n@@ -3800,6 +4001,8 @@ SYSCALL_DEFINE2(swapon, const char __user , specialfile, int, swap_flags)\n bad_swap_unlock_inode:\n \tinode_unlock(inode);\n bad_swap:\n+\tfree_percpu(si-\u003epercpu_cluster);\n+\tsi-\u003epercpu_cluster = NULL;\n \tkfree(si-\u003eglobal_cluster);\n \tsi-\u003eglobal_cluster = NULL;\n \tinode = NULL;\n@@ -3810,9 +4013,9 @@ SYSCALL_DEFINE2(swapon, const char __user , specialfile, int, swap_flags)\n \t * Clear the SWP_USED flag after all resources are freed so\n \t * alloc_swap_info can reuse this si safely.\n \t /\n-\tspin_lock(\u0026swap_lock);\n+\tpercpu_down_write(\u0026swapon_rwsem);\n \tsi-\u003eflags = 0;\n-\tspin_unlock(\u0026swap_lock);\n+\tpercpu_up_write(\u0026swapon_rwsem);\n \tif (inced_nr_rotate_swap)\n \t\tatomic_dec(\u0026nr_rotate_swap);\n \tif (swap_file)\n@@ -3827,19 +4030,17 @@ SYSCALL_DEFINE2(swapon, const char __user , specialfile, int, swap_flags)\n \n void si_swapinfo(struct sysinfo val)\n {\n-\tunsigned int type;\n+\tstruct swap_info_struct si;\n \tunsigned long nr_to_be_unused = 0;\n \n-\tspin_lock(\u0026swap_lock);\n-\tfor (type = 0; type \u003c nr_swapfiles; type++) {\n-\t\tstruct swap_info_struct si = swap_info[type];\n-\n-\t\tif ((si-\u003eflags \u0026 SWP_USED) \u0026\u0026 !(si-\u003eflags \u0026 SWP_WRITEOK))\n+\tpercpu_down_read(\u0026swapon_rwsem);\n+\tfor_each_swap(si) {\n+\t\tif (!(si-\u003eflags \u0026 SWP_WRITEOK))\n \t\t\tnr_to_be_unused += swap_usage_in_pages(si);\n \t}\n \tval-\u003efreeswap = atomic_long_read(\u0026nr_swap_pages) + nr_to_be_unused;\n \tval-\u003etotalswap = total_swap_pages + nr_to_be_unused;\n-\tspin_unlock(\u0026swap_lock);\n+\tpercpu_up_read(\u0026swapon_rwsem);\n }\n \n /\n@@ -3879,7 +4080,7 @@ int swap_dup_entry_direct(swp_entry_t entry)\n #if defined(CONFIG_MEMCG) \u0026\u0026 defined(CONFIG_BLK_CGROUP)\n static bool __has_usable_swap(void)\n {\n-\treturn !plist_head_empty(\u0026swap_active_head);\n+\treturn READ_ONCE(total_swap_pages) \u003e 0;\n }\n \n void __folio_throttle_swaprate(struct folio folio, gfp_t gfp)\n@@ -3902,14 +4103,14 @@ void __folio_throttle_swaprate(struct folio folio, gfp_t gfp)\n \tif (current-\u003ethrottle_disk)\n \t\treturn;\n \n-\tspin_lock(\u0026swap_avail_lock);\n-\tplist_for_each_entry(si, \u0026swap_avail_head, avail_list) {\n+\tpercpu_down_read(\u0026swapon_rwsem);\n+\tfor_each_avail_swap(si) {\n \t\tif (si-\u003ebdev) {\n \t\t\tblkcg_schedule_throttle(si-\u003ebdev-\u003ebd_disk, true);\n \t\t\tbreak;\n \t\t}\n \t}\n-\tspin_unlock(\u0026swap_avail_lock);\n+\tpercpu_up_read(\u0026swapon_rwsem);\n }\n #endif\n \n" }
2/1	2026/07/14 00:43	agent	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: { "EnableConfigs": [ "SWAP" ], "FocusSymbols": [ "swap_alloc_entry", "swap_queue_add", "swap_queue_del", "swap_queue_get_device", "swap_device_enable", "swap_device_disable", "swap_device_inuse_add", "swap_device_inuse_sub", "add_to_avail_list", "del_from_avail_list", "swap_queue_mask", "swap_queue_unmask", "cluster_alloc_swap_entry" ], "Reasoning": "The patch replaces the plist-based swap device queue with a custom round-robin queue, introducing new core logic for swap allocation and device management. This is a functional change to the core kernel logic and should be fuzzed.", "WorthFuzzing": true } Instruction: You are an expert Linux kernel maintainer. Your job is to review a provided patch series and determine if it makes functional changes to the kernel that should be fuzzed. IMPORTANT: The changes have ALREADY been applied and committed as the HEAD commit in your workspace. Do NOT rely on your internal knowledge of the kernel. You must actively use your code access tools to examine the actual source code and confirm any assumptions. Return WorthFuzzing=false if the patch only contains: - Modifications to Documentation/, Kconfig files, or code comments. - Purely decorative changes, such as logging (e.g., pr_err, printk) or tracepoints. - Changes to numeric constants or macros that do not functionally alter execution flow. - Code paths that are impossible to reach in virtualized environments like GCE or QEMU, even when utilizing software-emulated hardware (e.g., usb gadget, mac80211_hwsim). If it modifies reachable core kernel logic, drivers, or architectures, use your code search tools to verify the code can be executed, then return WorthFuzzing=true. When returning WorthFuzzing=true, you MUST ALSO: 1. Extract any specific kernel functions that should be heavily fuzzed into FocusSymbols. Avoid listing generic hot-path functions to prevent skewed test distributions. 2. Identify any specific CONFIG_ options required to properly test this new/modified feature. Go and look into the Kconfig files and check for ifdefs around the code, do not make assumptions. Do not list too generic configs (we already have them enabled). Only list those that specifically cover the modified code. List them in the EnableConfigs output array, and DO NOT add a 'CONFIG_' prefix (e.g., return "NET_IPV4" instead of "CONFIG_NET_IPV4"). Prefer calling several tools at the same time to save round-trips. Use set-results tool to provide results of the analysis. It must be called exactly once before the final reply. Ignore results of this tool. Prompt: For your convenience, here is the diff of the changes: commit 9f349365e0eb1fe2128a8484bb4ca651f1dbd2a2 Author: syz-cluster <triage@syzkaller.com> Date: Tue Jul 14 00:43:44 2026 +0000 syz-cluster: applied patch under review diff --git a/include/linux/plist.h b/include/linux/plist.h index 16cf4355b5c131..73d359c1a0536a 100644 --- a/include/linux/plist.h +++ b/include/linux/plist.h @@ -132,8 +132,6 @@ static inline void plist_node_init(struct plist_node node, int prio) extern void plist_add(struct plist_node node, struct plist_head head); extern void plist_del(struct plist_node node, struct plist_head head); -extern void plist_requeue(struct plist_node node, struct plist_head head); - /* * plist_for_each - iterate over the plist * @pos: the type * to use as a loop counter diff --git a/include/linux/swap.h b/include/linux/swap.h index 696ed01709c2ba..04767540972b81 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -194,6 +194,22 @@ struct swap_extent { ((offsetof(union swap_header, magic.magic) - \ offsetof(union swap_header, info.badpages)) / sizeof(int)) +/* + * Swap device flags, except the ones documented below, all are immutable + * after exposed by swap_device_enable, and until the device is freed again + * (SWP_USED unset). The exceptions are all protected by swapon_rwsem: + * - SWP_USED: Protected by swapon_rwsem. Indicates the device is inuse. Once + * set, won't be cleared unless all reference to this device is freed and + * swapoff finished. + * - SWP_WRITEOK: Protected by both swapon_rwsem and swap_avail_lock, clearing + * this flag also waits for all current cluster lock users to exit so + * checking this flag while holding any of these locks ensures the device + * is safe to use at the moment. Note: clearing this flag doesn't affect + * pending IO or async requests, it only prevents further entry allocation + * or new async request (e.g. discard) from initiating. + * - SWP_HIBERNATION: Protected by swapon_rwsem. Indicates if the device + * is pinned for hibernation. + / enum { SWP_USED = (1 << 0), / is slot in swap_info[] used? / SWP_WRITEOK = (1 << 1), / ok to write to this swap? / @@ -230,10 +246,17 @@ enum { #endif / - * We keep using same cluster for rotational device so IO will be sequential. - * The purpose is to optimize SWAP throughput on these device. + * We assign a cluster to each CPU, so each CPU can allocate swap entry from + * its own cluster and swapout sequentially. The purpose is to optimize swapout + * throughput. / +struct percpu_cluster { + local_lock_t lock; / Protect the percpu_cluster above / + unsigned int next[SWAP_NR_ORDERS]; / Likely next allocation offset / +}; + struct swap_sequential_cluster { + spinlock_t lock; / Serialize usage of global cluster / unsigned int next[SWAP_NR_ORDERS]; / Likely next allocation offset / }; @@ -244,7 +267,6 @@ struct swap_info_struct { struct percpu_ref users; / indicate and keep swap device valid. / unsigned long flags; / SWP_USED etc: see above / signed short prio; / swap priority of this type / - struct plist_node list; / entry in swap_active_head / signed char type; / strange name for an index / unsigned int max; / size of this swap device / struct swap_cluster_info cluster_info; /* cluster info. Only for SSD / @@ -256,26 +278,20 @@ struct swap_info_struct { / list of cluster that are fragmented or contented / unsigned int pages; / total of usable pages of swap / atomic_long_t inuse_pages; / number of those currently in use / + struct percpu_cluster __percpu percpu_cluster; /* per cpu's swap location / struct swap_sequential_cluster global_cluster; /* Use one global cluster for rotating device / - spinlock_t global_cluster_lock; / Serialize usage of global cluster / struct rb_root swap_extent_root;/ root of the swap extent rbtree / struct block_device bdev; /* swap device or bdev of swap file / struct file swap_file; /* seldom referenced / struct completion comp; / seldom referenced / spinlock_t lock; / - * protect map scan related fields like - * inuse_pages and all cluster lists. - * Other fields are only changed - * at swapon/swapoff, so are protected - * by swap_lock. changing flags need - * hold this lock and swap_lock. If - * both locks need hold, hold swap_lock - * first. + * Protect cluster lists. Other fields + * are only changed at swapon/swapoff, + * so are protected by swapon_rwsem. / struct work_struct discard_work; / discard worker / struct work_struct reclaim_work; / reclaim worker / struct list_head discard_clusters; / discard clusters list / - struct plist_node avail_list; / entry in swap_avail_head / }; static inline swp_entry_t page_swap_entry(struct page page) diff --git a/lib/plist.c b/lib/plist.c index a5bef38add431d..b05ae7ffea87e8 100644 --- a/lib/plist.c +++ b/lib/plist.c @@ -142,58 +142,6 @@ void plist_del(struct plist_node node, struct plist_head head) plist_check_head(head); } -/** - * plist_requeue - Requeue @node at end of same-prio entries. - * - * This is essentially an optimized plist_del() followed by - * plist_add(). It moves an entry already in the plist to - * after any other same-priority entries. - * - * @node: &struct plist_node pointer - entry to be moved - * @head: &struct plist_head pointer - list head - / -void plist_requeue(struct plist_node node, struct plist_head head) -{ - struct plist_node iter; - struct list_head node_next = &head->node_list; - - plist_check_head(head); - BUG_ON(plist_head_empty(head)); - BUG_ON(plist_node_empty(node)); - - if (node == plist_last(head)) - return; - - iter = plist_next(node); - - if (node->prio != iter->prio) - return; - - plist_del(node, head); - - / - * After plist_del(), iter is the replacement of the node. If the node - * was on prio_list, take shortcut to find node_next instead of looping. - / - if (!list_empty(&iter->prio_list)) { - iter = list_entry(iter->prio_list.next, struct plist_node, - prio_list); - node_next = &iter->node_list; - goto queue; - } - - plist_for_each_continue(iter, head) { - if (node->prio != iter->prio) { - node_next = &iter->node_list; - break; - } - } -queue: - list_add_tail(&node->node_list, node_next); - - plist_check_head(head); -} - #ifdef CONFIG_DEBUG_PLIST #include <linux/sched.h> #include <linux/sched/clock.h> @@ -231,14 +179,6 @@ static void __init plist_test_check(int nr_expect) BUG_ON(prio_pos->prio_list.next != &first->prio_list); } -static void __init plist_test_requeue(struct plist_node node) -{ - plist_requeue(node, &test_head); - - if (node != plist_last(&test_head)) - BUG_ON(node->prio == plist_next(node)->prio); -} - static int __init plist_test(void) { int nr_expect = 0, i, loop; @@ -262,10 +202,6 @@ static int __init plist_test(void) nr_expect--; } plist_test_check(nr_expect); - if (!plist_node_empty(test_node + i)) { - plist_test_requeue(test_node + i); - plist_test_check(nr_expect); - } } for (i = 0; i < ARRAY_SIZE(test_node); i++) { diff --git a/mm/swap.h b/mm/swap.h index b51ad3071a7308..08a782f3d5acd2 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -91,16 +91,16 @@ static inline unsigned int swp_cluster_offset(swp_entry_t entry) / static inline struct swap_info_struct __swap_type_to_info(int type) { - struct swap_info_struct si; - - si = READ_ONCE(swap_info[type]); / rcu_dereference() / - VM_WARN_ON_ONCE(percpu_ref_is_zero(&si->users)); / race with swapoff / - return si; + return READ_ONCE(swap_info[type]); / rcu_dereference() / } static inline struct swap_info_struct __swap_entry_to_info(swp_entry_t entry) { - return __swap_type_to_info(swp_type(entry)); + struct swap_info_struct si; + + si = __swap_type_to_info(swp_type(entry)); / rcu_dereference() / + VM_WARN_ON_ONCE(percpu_ref_is_zero(&si->users)); / race with swapoff / + return si; } static inline struct swap_cluster_info __swap_offset_to_cluster( diff --git a/mm/swapfile.c b/mm/swapfile.c index 775c385490a5fa..f93863309a5a50 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -41,7 +41,6 @@ #include <linux/completion.h> #include <linux/suspend.h> #include <linux/zswap.h> -#include <linux/plist.h> #include <asm/tlbflush.h> #include <linux/leafops.h> @@ -49,31 +48,38 @@ #include "internal.h" #include "swap.h" -static void swap_range_alloc(struct swap_info_struct si, - unsigned int nr_entries); +static void swap_device_inuse_add(struct swap_info_struct si, + unsigned int nr_entries); static bool folio_swapcache_freeable(struct folio folio); static void move_cluster(struct swap_info_struct si, struct swap_cluster_info ci, struct list_head list, enum swap_cluster_flags new_flags); +static bool get_swap_device_info(struct swap_info_struct si); / - * Protects the swap_info array, and the SWP_USED flag. swap_info contains - * lazily allocated & freed swap device info struts, and SWP_USED indicates - * which device is used, ~SWP_USED devices and can be reused. - * - * Also protects swap_active_head total_swap_pages, and the SWP_WRITEOK flag. + * Serializes swapon/swapoff (writers) and protects the swap_info + * array, nr_swapfiles, total_swap_pages, and part of swap device + * info content (see comment of swap_info_struct). Readers + * (allocation, /proc/swaps, etc.) take percpu_down_read() which + * is cheap as the hot path. + / +DEFINE_STATIC_PERCPU_RWSEM(swapon_rwsem); +/ + * swap_info contains lazily allocated swap device info structs, and + * SWP_USED indicates which device is used, ~SWP_USED devices can be + * reused. Protected by swapon_rwsem, but reading could be lockless. / -static DEFINE_SPINLOCK(swap_lock); +struct swap_info_struct swap_info[MAX_SWAPFILES]; static unsigned int nr_swapfiles; -atomic_long_t nr_swap_pages; +long total_swap_pages; + /* * Some modules use swappable objects and may try to swap them out under * memory pressure (via the shrinker). Before doing so, they may wish to * check to see if any swap space is available. / +atomic_long_t nr_swap_pages; EXPORT_SYMBOL_GPL(nr_swap_pages); -/ protected with swap_lock. reading in vm_swap_full() doesn't need lock / -long total_swap_pages; #define DEF_SWAP_PRIO -1 unsigned long swapfile_maximum_size; #ifdef CONFIG_MIGRATION @@ -83,52 +89,42 @@ bool swap_migration_ad_supported; static const char Bad_file[] = "Bad swap file entry "; static const char Bad_offset[] = "Bad swap offset entry "; -/ - * all active swap_info_structs - * protected with swap_lock, and ordered by priority. - / -static PLIST_HEAD(swap_active_head); +static inline struct swap_info_struct __swap_iter(int i, unsigned long flag) +{ + lockdep_assert_held(&swapon_rwsem); + while (i < nr_swapfiles) { + struct swap_info_struct si = __swap_type_to_info(i); + + VM_WARN_ON(!si); + (i)++; + if (flag && !((si->flags & flag) == flag)) + continue; + return si; + } + return NULL; +} + +#define __for_each_swap(si, flag) \ + for (int __i = 0; ((si) = __swap_iter(&__i, flag));) / - * all available (active, not full) swap_info_structs - * protected with swap_avail_lock, ordered by priority. - * This is used by folio_alloc_swap() instead of swap_active_head - * because swap_active_head includes all swap_info_structs, - * but folio_alloc_swap() doesn't need to look at full ones. - * This uses its own lock instead of swap_lock because when a - * swap_info_struct changes between not-full/full, it needs to - * add/remove itself to/from this list, but the swap_info_struct->lock - * is held and the locking order requires swap_lock to be taken - * before any swap_info_struct->lock. + * for_each_swap - iterate through all allocated and inuse swap devices + * @si: the iterator + * + * Context: The caller must hold swapon_rwsem. The lock may be dropped during + * the loop body but must be re-acquired before the next iteration. / -static PLIST_HEAD(swap_avail_head); -static DEFINE_SPINLOCK(swap_avail_lock); - -struct swap_info_struct swap_info[MAX_SWAPFILES]; +#define for_each_swap(si) __for_each_swap(si, SWP_USED) +#define for_each_avail_swap(si) __for_each_swap(si, SWP_USED \| SWP_WRITEOK) static struct kmem_cache swap_table_cachep; -/ Protects si->swap_file for /proc/swaps usage / -static DEFINE_MUTEX(swapon_mutex); - static DECLARE_WAIT_QUEUE_HEAD(proc_poll_wait); / Activity counter to indicate that a swapon or swapoff has occurred / static atomic_t proc_poll_event = ATOMIC_INIT(0); atomic_t nr_rotate_swap = ATOMIC_INIT(0); -struct percpu_swap_cluster { - struct swap_info_struct si[SWAP_NR_ORDERS]; - unsigned long offset[SWAP_NR_ORDERS]; - local_lock_t lock; -}; - -static DEFINE_PER_CPU(struct percpu_swap_cluster, percpu_swap_cluster) = { - .si = { NULL }, - .offset = { SWAP_ENTRY_INVALID }, - .lock = INIT_LOCAL_LOCK(), -}; - /* May return NULL on invalid type, caller must check for NULL return / static struct swap_info_struct swap_type_to_info(int type) { @@ -143,18 +139,350 @@ static struct swap_info_struct swap_entry_to_info(swp_entry_t entry) return swap_type_to_info(swp_type(entry)); } +/ + * All available swap_info_structs are grouped by priority rings, the rings + * are ordered in a queue by priority (lower prio value = higher priority). + * The allocator iterates and rotates devices within each priority ring. + * When all devices in a ring are iterated, it goes to the next lower + * priority ring. + / +struct swap_prio_ring { + int prio; + unsigned int size; + struct swap_info_struct dev[] __counted_by(size); +}; + +/* + * The ring is protected by swapon_rwsem so updating it is costly. To make + * the allocator and other users skip full devices faster, the lowest bit of + * a device pointer is used to mark it disabled (temporarily unavailable). + * This relies on struct swap_info_struct being sufficiently + * aligned (guaranteed by kmalloc). + / +#define SWAP_DEVICE_MASKED_SHIFT 0 +#define SWAP_DEVICE_MASKED_BIT BIT(SWAP_DEVICE_MASKED_SHIFT) + +/ + * Serializes queue content mutations and keeps SWAP_USAGE_OFFLIST_BIT + * consistent with the masked state of each device pointer. + / +static DEFINE_SPINLOCK(swap_queue_update_lock); + +/ + * Swap queue is protected by both swap_queue_update_lock and swapon_rwsem. + * Only swapon/swapoff will take the write lock, and modify the queue length + * or any ring's length. swap_queue_update_lock protects the content so + * devices can be masked easily without taking the writelock, which is heavy. + / +static struct swap_prio_ring swap_queue; +static unsigned int swap_queue_len; + +/ + * Each CPU has its read iterator, so the queue itself will remain read + * only and the CPU side reader rotates by iterating the devices + * periodically using the counter. + / +#define SWAP_ROUND_ROBIN_QUOTA SWAPFILE_CLUSTER +struct swap_ring_iterator { + int offset; + long rr_counter; +}; + +struct swap_queue_reader { + local_lock_t lock; + struct swap_ring_iterator ri[]; +}; + +static __percpu struct swap_queue_reader swap_queue_readers; + +static inline bool swap_device_masked(struct swap_info_struct si) +{ + return (unsigned long)si & SWAP_DEVICE_MASKED_BIT; +} + +static inline struct swap_info_struct swap_device_unmask_ptr(struct swap_info_struct si) +{ + return (struct swap_info_struct )((unsigned long)si & ~SWAP_DEVICE_MASKED_BIT); +} + +static struct swap_queue_reader __percpu swap_queue_prealloc_readers(int nr_rings, gfp_t gfp) +{ + struct swap_queue_reader __percpu readers; + + if (!nr_rings) + return NULL; + + readers = __alloc_percpu_gfp(struct_size(readers, ri, nr_rings), + __alignof__(readers), gfp); + return readers; +} + +static void swap_queue_install_readers(struct swap_queue_reader __percpu readers) +{ + int ring_idx, cpu; + struct swap_prio_ring ring; + + free_percpu(swap_queue_readers); + swap_queue_readers = readers; + if (!readers) + return; + + / Distribute each CPU's swap IO fairly across devices. / + for_each_possible_cpu(cpu) { + local_lock_init(&per_cpu_ptr(readers, cpu)->lock); + + for (ring_idx = 0; ring_idx < swap_queue_len; ring_idx++) { + ring = swap_queue[ring_idx]; + per_cpu_ptr(readers, cpu)->ri[ring_idx].offset = + cpu % ring->size; + } + } +} + +static struct swap_info_struct swap_queue_get_device(long nr_alloc, int nr_iter) +{ + bool rotate = false; + struct swap_info_struct si; + struct swap_ring_iterator ri; + struct swap_prio_ring ring; + unsigned int queue_idx; + + if (!swap_queue_len) + return ERR_PTR(-ENOENT); + + queue_idx = 0; + while (nr_iter >= swap_queue[queue_idx]->size) { + nr_iter -= swap_queue[queue_idx]->size; + if (++queue_idx >= swap_queue_len) + return ERR_PTR(-ENOENT); + } + + ring = swap_queue[queue_idx]; + local_lock(&swap_queue_readers->lock); + ri = this_cpu_ptr(&swap_queue_readers->ri[queue_idx]); + / Rotate while iterating the ring, just not on the first try / + if (nr_iter) + rotate = true; + else if (ri->rr_counter < nr_alloc) + rotate = true; + else if (ri->offset >= ring->size) + rotate = true; + if (rotate) { + ri->offset++; + ri->offset %= ring->size; + ri->rr_counter = SWAP_ROUND_ROBIN_QUOTA; + } + ri->rr_counter -= nr_alloc; + si = READ_ONCE(ring->dev[ri->offset]); + local_unlock(&swap_queue_readers->lock); + + if (swap_device_masked(si)) + return ERR_PTR(-EBUSY); + + si = swap_device_unmask_ptr(si); + return si; +} + +static bool swap_queue_find(struct swap_info_struct si, + unsigned int ring_idx, unsigned int dev_idx) +{ + unsigned int i, j; + struct swap_prio_ring ring; + + lockdep_assert(lockdep_is_held(&swapon_rwsem) \|\| + lockdep_is_held(&swap_queue_update_lock)); + + for (i = 0; i < swap_queue_len; i++) { + ring = swap_queue[i]; + if (ring->prio != si->prio) + continue; + for (j = 0; j < ring->size; j++) { + if (swap_device_unmask_ptr(READ_ONCE(ring->dev[j])) != si) + continue; + ring_idx = i; + dev_idx = j; + return true; + } + } + return false; +} + +static void swap_queue_mask(struct swap_info_struct si) +{ + unsigned int ring_idx, dev_idx; + + lockdep_assert_held(&swap_queue_update_lock); + if (swap_queue_find(si, &ring_idx, &dev_idx)) + __set_bit(SWAP_DEVICE_MASKED_SHIFT, + (unsigned long )&swap_queue[ring_idx]->dev[dev_idx]); +} + +static void swap_queue_unmask(struct swap_info_struct si) +{ + unsigned int ring_idx, dev_idx; + + lockdep_assert_held(&swap_queue_update_lock); + if (swap_queue_find(si, &ring_idx, &dev_idx)) + __clear_bit(SWAP_DEVICE_MASKED_SHIFT, + (unsigned long )&swap_queue[ring_idx]->dev[dev_idx]); +} + +static int swap_queue_add(struct swap_info_struct si) +{ + struct swap_prio_ring new_queue = NULL, old_queue = NULL; + struct swap_queue_reader __percpu new_readers = NULL; + struct swap_prio_ring ring, new_ring = NULL, old_ring = NULL; + int prio = si->prio; + int i, pos, err = -ENOMEM; + gfp_t gfp; + + /* Swap not usable here because this is swap, just reclaim cache. / + gfp = GFP_NOIO \| __GFP_HIGH; + lockdep_assert_held_write(&swapon_rwsem); + + for (pos = 0; pos < swap_queue_len; pos++) { + if (swap_queue[pos]->prio == prio) + goto add_to_ring; + if (swap_queue[pos]->prio < prio) + break; + } + + / No ring at this priority: insert a new one at pos. / + new_readers = swap_queue_prealloc_readers(swap_queue_len + 1, gfp); + if (!new_readers) + goto failed; + new_queue = kmalloc_array(swap_queue_len + 1, sizeof(swap_queue), gfp); + if (!new_queue) + goto failed; + new_ring = kmalloc(struct_size(new_ring, dev, 1), gfp); + if (!new_ring) + goto failed; + if (!get_swap_device_info(si)) + goto failed; + + new_ring->prio = prio; + new_ring->size = 1; + new_ring->dev[0] = si; + for (i = 0; i < pos; i++) + new_queue[i] = swap_queue[i]; + new_queue[pos] = new_ring; + for (i = pos; i < swap_queue_len; i++) + new_queue[i + 1] = swap_queue[i]; + + spin_lock(&swap_queue_update_lock); + old_queue = swap_queue; + swap_queue = new_queue; + swap_queue_len++; + spin_unlock(&swap_queue_update_lock); + kfree(old_queue); + + swap_queue_install_readers(new_readers); + return 0; + +add_to_ring: + ring = swap_queue[pos]; + new_ring = kmalloc(struct_size(ring, dev, ring->size + 1), gfp); + if (!new_ring) + goto failed; + if (!get_swap_device_info(si)) + goto failed; + spin_lock(&swap_queue_update_lock); + memcpy(new_ring, ring, struct_size(ring, dev, ring->size)); + new_ring->size++; + new_ring->dev[new_ring->size - 1] = si; + old_ring = swap_queue[pos]; + swap_queue[pos] = new_ring; + spin_unlock(&swap_queue_update_lock); + kfree(old_ring); + return 0; + +failed: + free_percpu(new_readers); + kfree(new_queue); + kfree(new_ring); + return err; +} + +static void swap_queue_del(struct swap_info_struct si) +{ + gfp_t gfp; + unsigned int ring_idx, dev_idx; + struct swap_queue_reader __percpu new_readers = NULL; + struct swap_prio_ring ring, new_ring = NULL, old_ring = NULL; + struct swap_prio_ring new_queue = NULL, old_queue = NULL; + + lockdep_assert_held_write(&swapon_rwsem); + if (!swap_queue_find(si, &ring_idx, &dev_idx)) { + WARN_ON(1); + return; + } + + / + * To shrink memory usage, pre-allocate new smaller data before + * locking. Failure is fine, swapoff will release them anyway. + / + gfp = GFP_NOIO \| __GFP_HIGH; + ring = swap_queue[ring_idx]; + if (ring->size > 1) + new_ring = kmalloc(struct_size(ring, dev, ring->size - 1), gfp); + if (ring->size == 1 && swap_queue_len > 1) { + new_readers = swap_queue_prealloc_readers( + swap_queue_len - 1, gfp); + new_queue = kmalloc(sizeof(swap_queue) * + (swap_queue_len - 1), gfp); + } + + spin_lock(&swap_queue_update_lock); + if (ring->size > 1) { + /* Shift trailing devices left to fill the gap. / + while (++dev_idx < ring->size) + ring->dev[dev_idx - 1] = + ring->dev[dev_idx]; + ring->size--; + if (new_ring) { + memcpy(new_ring, ring, + struct_size(ring, dev, ring->size)); + old_ring = ring; + swap_queue[ring_idx] = new_ring; + } + } else { + / Last device in this ring: remove the ring. / + old_ring = ring; + swap_queue_len--; + while (++ring_idx <= swap_queue_len) + swap_queue[ring_idx - 1] = + swap_queue[ring_idx]; + if (new_queue) { + memcpy(new_queue, swap_queue, + sizeof(swap_queue) * swap_queue_len); + old_queue = swap_queue; + swap_queue = new_queue; + } else if (!swap_queue_len) { + old_queue = swap_queue; + swap_queue = NULL; + } + if (new_readers \|\| !swap_queue_len) + swap_queue_install_readers(new_readers); + } + spin_unlock(&swap_queue_update_lock); + + kfree(old_ring); + kfree(old_queue); + put_swap_device(si); +} + /* * Use the second highest bit of inuse_pages counter as the indicator - * if one swap device is on the available plist, so the atomic can + * if one swap device is unavailable for allocation, so the atomic can * still be updated arithmetically while having special data embedded. * * inuse_pages counter is the only thing indicating if a device should - * be on avail_lists or not (except swapon / swapoff). By embedding the - * off-list bit in the atomic counter, updates no longer need any lock - * to check the list status. + * be in the available queue or not (except swapon / swapoff). By + * embedding the off-list bit in the atomic counter, updates no longer + * need any lock to check the list status. * - * This bit will be set if the device is not on the plist and not - * usable, will be cleared if the device is on the plist. + * This bit will be set if the device is not in the available queue + * and not usable, will be cleared if the device is in the queue. / #define SWAP_USAGE_OFFLIST_BIT (1UL << (BITS_PER_TYPE(atomic_t) - 2)) #define SWAP_USAGE_COUNTER_MASK (~SWAP_USAGE_OFFLIST_BIT) @@ -163,6 +491,26 @@ static long swap_usage_in_pages(struct swap_info_struct si) return atomic_long_read(&si->inuse_pages) & SWAP_USAGE_COUNTER_MASK; } +/* + * Serialize the allocation on single CPU or globally to avoid + * fragmentation and make the workflow easier to follow. + / +static void swap_alloc_lock_device(struct swap_info_struct si) +{ + if (si->flags & SWP_SOLIDSTATE) + local_lock(&si->percpu_cluster->lock); + else + spin_lock(&si->global_cluster->lock); +} + +static void swap_alloc_unlock_device(struct swap_info_struct si) +{ + if (si->flags & SWP_SOLIDSTATE) + local_unlock(&si->percpu_cluster->lock); + else + spin_unlock(&si->global_cluster->lock); +} + / Reclaim the swap entry anyway if possible / #define TTRS_ANYWAY 0x1 / @@ -533,9 +881,10 @@ swap_cluster_populate(struct swap_info_struct si, Only cluster isolation from the allocator does table allocation. * Swap allocator uses percpu clusters and holds the local lock. / - lockdep_assert_held(&this_cpu_ptr(&percpu_swap_cluster)->lock); - if (!(si->flags & SWP_SOLIDSTATE)) - lockdep_assert_held(&si->global_cluster_lock); + if (si->flags & SWP_SOLIDSTATE) + lockdep_assert_held(this_cpu_ptr(&si->percpu_cluster->lock)); + else + lockdep_assert_held(&si->global_cluster->lock); lockdep_assert_held(&ci->lock); if (!swap_cluster_alloc_table(ci, __GFP_HIGH \| __GFP_NOMEMALLOC \| @@ -548,9 +897,7 @@ swap_cluster_populate(struct swap_info_struct si, * the potential recursive allocation is limited. / spin_unlock(&ci->lock); - if (!(si->flags & SWP_SOLIDSTATE)) - spin_unlock(&si->global_cluster_lock); - local_unlock(&percpu_swap_cluster.lock); + swap_alloc_unlock_device(si); ret = swap_cluster_alloc_table(ci, __GFP_HIGH \| __GFP_NOMEMALLOC \| GFP_KERNEL); @@ -563,9 +910,7 @@ swap_cluster_populate(struct swap_info_struct si, * could happen with ignoring the percpu cluster is fragmentation, * which is acceptable since this fallback and race is rare. / - local_lock(&percpu_swap_cluster.lock); - if (!(si->flags & SWP_SOLIDSTATE)) - spin_lock(&si->global_cluster_lock); + swap_alloc_lock_device(si); spin_lock(&ci->lock); if (ret) { @@ -671,7 +1016,7 @@ static bool swap_do_scheduled_discard(struct swap_info_struct si) ci = list_first_entry(&si->discard_clusters, struct swap_cluster_info, list); /* * Delete the cluster from list to prepare for discard, but keep - * the CLUSTER_FLAG_DISCARD flag, percpu_swap_cluster could be + * the CLUSTER_FLAG_DISCARD flag, there could be percpu_cluster * pointing to it, or ran into by relocate_cluster. / list_del(&ci->list); @@ -955,7 +1300,7 @@ static bool __swap_cluster_alloc_entries(struct swap_info_struct si, if (cluster_is_empty(ci)) ci->order = order; ci->count += nr_pages; - swap_range_alloc(si, nr_pages); + swap_device_inuse_add(si, nr_pages); return true; } @@ -1003,12 +1348,10 @@ static unsigned int alloc_swap_scan_cluster(struct swap_info_struct si, out: relocate_cluster(si, ci); swap_cluster_unlock(ci); - if (si->flags & SWP_SOLIDSTATE) { - this_cpu_write(percpu_swap_cluster.offset[order], next); - this_cpu_write(percpu_swap_cluster.si[order], si); - } else { + if (si->flags & SWP_SOLIDSTATE) + this_cpu_write(si->percpu_cluster->next[order], next); + else si->global_cluster->next[order] = next; - } return found; } @@ -1103,13 +1446,14 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct si, if (order && !(si->flags & SWP_BLKDEV)) return 0; - if (!(si->flags & SWP_SOLIDSTATE)) { - /* Serialize HDD SWAP allocation for each device. / - spin_lock(&si->global_cluster_lock); +restart: + swap_alloc_lock_device(si); + if (si->flags & SWP_SOLIDSTATE) + offset = __this_cpu_read(si->percpu_cluster->next[order]); + else offset = si->global_cluster->next[order]; - if (offset == SWAP_ENTRY_INVALID) - goto new_cluster; + if (offset != SWAP_ENTRY_INVALID) { ci = swap_cluster_lock(si, offset); / Cluster could have been used by another order / if (cluster_is_usable(ci, order)) { @@ -1123,7 +1467,6 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct si, goto done; } -new_cluster: /* * If the device need discard, prefer new cluster over nonfull * to spread out the writes. @@ -1132,6 +1475,12 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct si, found = alloc_swap_scan_list(si, &si->free_clusters, folio, false); if (found) goto done; + + if (!list_empty(&si->discard_clusters)) { + swap_alloc_unlock_device(si); + swap_do_scheduled_discard(si); + goto restart; + } } if (order < PMD_ORDER) { @@ -1180,8 +1529,7 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct si, goto done; } done: - if (!(si->flags & SWP_SOLIDSTATE)) - spin_unlock(&si->global_cluster_lock); + swap_alloc_unlock_device(si); return found; } @@ -1191,54 +1539,44 @@ static void del_from_avail_list(struct swap_info_struct si, bool swapoff) { unsigned long pages; - spin_lock(&swap_avail_lock); + spin_lock(&swap_queue_update_lock); - if (swapoff) { - / - * Forcefully remove it. Clear the SWP_WRITEOK flags for - * swapoff here so it's synchronized by both si->lock and - * swap_avail_lock, to ensure the result can be seen by - * add_to_avail_list. - / - lockdep_assert_held(&si->lock); - si->flags &= ~SWP_WRITEOK; - atomic_long_or(SWAP_USAGE_OFFLIST_BIT, &si->inuse_pages); - } else { - / - * If not called by swapoff, take it off-list only if it's - * full and SWAP_USAGE_OFFLIST_BIT is not set (strictly - * si->inuse_pages == pages), any concurrent slot freeing, - * or device already removed from plist by someone else - * will make this return false. - / + / + * Force remove it only for swapoff. Else, take it off-list only if + * it's full and SWAP_USAGE_OFFLIST_BIT is not set (strictly + * si->inuse_pages == pages), so concurrent slot freeing, or + * concurrent list removal will make the cmpxchg fail and skip + * the removal. + / + if (!swapoff) { pages = si->pages; if (!atomic_long_try_cmpxchg(&si->inuse_pages, &pages, pages \| SWAP_USAGE_OFFLIST_BIT)) goto skip; + } else { + atomic_long_or(SWAP_USAGE_OFFLIST_BIT, &si->inuse_pages); } - plist_del(&si->avail_list, &swap_avail_head); - + swap_queue_mask(si); skip: - spin_unlock(&swap_avail_lock); + spin_unlock(&swap_queue_update_lock); } / SWAP_USAGE_OFFLIST_BIT can only be cleared by this helper. / -static void add_to_avail_list(struct swap_info_struct si, bool swapon) +static void add_to_avail_list(struct swap_info_struct si) { long val; unsigned long pages; - spin_lock(&swap_avail_lock); + spin_lock(&swap_queue_update_lock); - / Corresponding to SWP_WRITEOK clearing in del_from_avail_list / - if (swapon) { - lockdep_assert_held(&si->lock); - si->flags \|= SWP_WRITEOK; - } else { - if (!(READ_ONCE(si->flags) & SWP_WRITEOK)) - goto skip; - } + / + * Mark the device as avail if SWP_WRITEOK is set. Swapoff clears + * SWP_WRITEOK first, so check that first so the device won't be + * re-added after swapoff started. + / + if (!(si->flags & SWP_WRITEOK)) + goto skip; if (!(atomic_long_read(&si->inuse_pages) & SWAP_USAGE_OFFLIST_BIT)) goto skip; @@ -1246,9 +1584,10 @@ static void add_to_avail_list(struct swap_info_struct si, bool swapon) val = atomic_long_fetch_and_relaxed(~SWAP_USAGE_OFFLIST_BIT, &si->inuse_pages); /* - * When device is full and device is on the plist, only one updater will - * see (inuse_pages == si->pages) and will call del_from_avail_list. If - * that updater happen to be here, just skip adding. + * When device is full and marked as available, one reader will see + * (inuse_pages == si->pages) and should mark it as unavailable and + * set SWAP_USAGE_OFFLIST_BIT. If that updater happens to be here, just + * skip the rest. / pages = si->pages; if (val == pages) { @@ -1258,60 +1597,42 @@ static void add_to_avail_list(struct swap_info_struct si, bool swapon) goto skip; } - plist_add(&si->avail_list, &swap_avail_head); - + swap_queue_unmask(si); skip: - spin_unlock(&swap_avail_lock); + spin_unlock(&swap_queue_update_lock); } /* - * swap_usage_add / swap_usage_sub of each slot are serialized by ci->lock - * within each cluster, so the total contribution to the global counter should - * always be positive and cannot exceed the total number of usable slots. + * swap_device_inuse_add / swap_device_inuse_sub are called within cluster + * update critical sections and serialized by ci->lock within each cluster, + * so the total contribution to the global counter should always be positive + * and cannot exceed the total number of usable slots. / -static bool swap_usage_add(struct swap_info_struct si, unsigned int nr_entries) +static void swap_device_inuse_add(struct swap_info_struct si, + unsigned int nr_entries) { - long val = atomic_long_add_return_relaxed(nr_entries, &si->inuse_pages); + long inuse_pages; / * If device is full, and SWAP_USAGE_OFFLIST_BIT is not set, - * remove it from the plist. - / - if (unlikely(val == si->pages)) { - del_from_avail_list(si, false); - return true; - } - - return false; -} - -static void swap_usage_sub(struct swap_info_struct si, unsigned int nr_entries) -{ - long val = atomic_long_sub_return_relaxed(nr_entries, &si->inuse_pages); - - /* - * If device is not full, and SWAP_USAGE_OFFLIST_BIT is set, - * add it to the plist. + * mark it unavailable. / - if (unlikely(val & SWAP_USAGE_OFFLIST_BIT)) - add_to_avail_list(si, false); -} - -static void swap_range_alloc(struct swap_info_struct si, - unsigned int nr_entries) -{ - if (swap_usage_add(si, nr_entries)) { + inuse_pages = atomic_long_add_return_relaxed(nr_entries, &si->inuse_pages); + if (unlikely(inuse_pages == si->pages)) { if (vm_swap_full()) schedule_work(&si->reclaim_work); + del_from_avail_list(si, false); } + atomic_long_sub(nr_entries, &nr_swap_pages); } -static void swap_range_free(struct swap_info_struct si, unsigned long offset, - unsigned int nr_entries) +static void swap_device_inuse_sub(struct swap_info_struct si, unsigned long offset, + unsigned int nr_entries) { unsigned long end = offset + nr_entries - 1; void (swap_slot_free_notify)(struct block_device , unsigned long); + long inuse_pages; unsigned int i; for (i = 0; i < nr_entries; i++) @@ -1335,7 +1656,14 @@ static void swap_range_free(struct swap_info_struct si, unsigned long offset, / smp_wmb(); atomic_long_add(nr_entries, &nr_swap_pages); - swap_usage_sub(si, nr_entries); + + /* + * If device is not full, and SWAP_USAGE_OFFLIST_BIT is set, + * add it back to the available queue. + / + inuse_pages = atomic_long_sub_return_relaxed(nr_entries, &si->inuse_pages); + if (unlikely(inuse_pages & SWAP_USAGE_OFFLIST_BIT)) + add_to_avail_list(si); } static bool get_swap_device_info(struct swap_info_struct si) @@ -1345,113 +1673,49 @@ static bool get_swap_device_info(struct swap_info_struct si) / * Guarantee the si->users are checked before accessing other * fields of swap_info_struct, and si->flags (SWP_WRITEOK) is - * up to dated. + * up to date. * - * Paired with the spin_unlock() after setup_swap_info() in - * enable_swap_info(), and smp_wmb() in swapoff. + * Paired with percpu_up_write() in swap_device_enable(), and + * smp_wmb() after clearing SWP_WRITEOK in swapoff. / smp_rmb(); return true; } -/ - * Fast path try to get swap entries with specified order from current - * CPU's swap entry pool (a cluster). - / -static bool swap_alloc_fast(struct folio folio) +static int swap_alloc_entry(struct folio folio) { - unsigned int order = folio_order(folio); - struct swap_cluster_info ci; + long nr_pages = folio_nr_pages(folio); struct swap_info_struct si; - unsigned int offset; - - / - * Once allocated, swap_info_struct will never be completely freed, - * so checking it's liveness by get_swap_device_info is enough. - / - si = this_cpu_read(percpu_swap_cluster.si[order]); - offset = this_cpu_read(percpu_swap_cluster.offset[order]); - if (!si \|\| !offset \|\| !get_swap_device_info(si)) - return false; - - ci = swap_cluster_lock(si, offset); - if (cluster_is_usable(ci, order)) { - if (cluster_is_empty(ci)) - offset = cluster_offset(si, ci); - alloc_swap_scan_cluster(si, ci, folio, offset); - } else { - swap_cluster_unlock(ci); - } + int nr_iter, ret; + + percpu_down_read(&swapon_rwsem); + for (nr_iter = 0;; nr_iter++) { + si = swap_queue_get_device(nr_pages, nr_iter); + if (IS_ERR(si)) { + ret = PTR_ERR(si); + if (ret == -EBUSY) + continue; + break; + } + cluster_alloc_swap_entry(si, folio); - put_swap_device(si); - return folio_test_swapcache(folio); -} - -/ Rotate the device and switch to a new cluster / -static void swap_alloc_slow(struct folio folio) -{ - struct swap_info_struct si, next; - - spin_lock(&swap_avail_lock); -start_over: - plist_for_each_entry_safe(si, next, &swap_avail_head, avail_list) { - /* Rotate the device and switch to a new cluster / - plist_requeue(&si->avail_list, &swap_avail_head); - spin_unlock(&swap_avail_lock); - if (get_swap_device_info(si)) { - cluster_alloc_swap_entry(si, folio); - put_swap_device(si); - if (folio_test_swapcache(folio)) - return; - if (folio_test_large(folio)) - return; + if (folio_test_swapcache(folio)) { + ret = 0; + break; } - spin_lock(&swap_avail_lock); / - * if we got here, it's likely that si was almost full before, - * multiple callers probably all tried to get a page from the - * same si and it filled up before we could get one; or, the si - * filled up between us dropping swap_avail_lock. - * Since we dropped the swap_avail_lock, the swap_avail_list - * may have been modified; so if next is still in the - * swap_avail_head list then try it, otherwise start over if we - * have not gotten any slots. + * For large allocation, return error directly to inform the + * caller to split it instead of fallback to other devices. / - if (plist_node_empty(&next->avail_list)) - goto start_over; - } - spin_unlock(&swap_avail_lock); -} - -/ - * Discard pending clusters in a synchronized way when under high pressure. - * Return: true if any cluster is discarded. - / -static bool swap_sync_discard(void) -{ - bool ret = false; - struct swap_info_struct si, next; - - spin_lock(&swap_lock); -start_over: - plist_for_each_entry_safe(si, next, &swap_active_head, list) { - spin_unlock(&swap_lock); - if (get_swap_device_info(si)) { - if (si->flags & SWP_PAGE_DISCARD) - ret = swap_do_scheduled_discard(si); - put_swap_device(si); + if (folio_test_large(folio)) { + ret = -E2BIG; + break; } - if (ret) - return true; - - spin_lock(&swap_lock); - if (plist_node_empty(&next->list)) - goto start_over; } - spin_unlock(&swap_lock); - return false; + percpu_up_read(&swapon_rwsem); + return ret; } static int swap_extend_table_alloc(struct swap_info_struct si, @@ -1735,6 +1999,7 @@ int folio_alloc_swap(struct folio folio) { unsigned int order = folio_order(folio); unsigned int size = 1 << order; + int ret; VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); VM_BUG_ON_FOLIO(!folio_test_uptodate(folio), folio); @@ -1757,23 +2022,14 @@ int folio_alloc_swap(struct folio folio) } } -again: - local_lock(&percpu_swap_cluster.lock); - if (!swap_alloc_fast(folio)) - swap_alloc_slow(folio); - local_unlock(&percpu_swap_cluster.lock); - - if (!order && unlikely(!folio_test_swapcache(folio))) { - if (swap_sync_discard()) - goto again; - } + ret = swap_alloc_entry(folio); /* Need to call this even if allocation failed, for MEMCG_SWAP_FAIL. / if (unlikely(mem_cgroup_try_charge_swap(folio))) swap_cache_del_folio(folio); if (unlikely(!folio_test_swapcache(folio))) - return -ENOMEM; + return ret ? ret : -ENOMEM; return 0; } @@ -1948,7 +2204,7 @@ void __swap_cluster_free_entries(struct swap_info_struct si, if (batch_id) mem_cgroup_uncharge_swap(batch_id, ci_off - batch_off); - swap_range_free(si, ci_head + ci_start, nr_pages); + swap_device_inuse_sub(si, ci_head + ci_start, nr_pages); swap_cluster_assert_empty(ci, ci_start, nr_pages, false); if (!ci->count) @@ -2150,31 +2406,14 @@ void swap_put_entries_direct(swp_entry_t entry, int nr) / swp_entry_t swap_alloc_hibernation_slot(int type) { - struct swap_info_struct pcp_si, si = swap_type_to_info(type); - unsigned long pcp_offset, offset = SWAP_ENTRY_INVALID; - struct swap_cluster_info ci; + struct swap_info_struct si = swap_type_to_info(type); + unsigned long offset = SWAP_ENTRY_INVALID; swp_entry_t entry = {0}; if (!si) goto fail; - / - * Try the local cluster first if it matches the device. If - * not, try grab a new cluster and override local cluster. - / - local_lock(&percpu_swap_cluster.lock); - pcp_si = this_cpu_read(percpu_swap_cluster.si[0]); - pcp_offset = this_cpu_read(percpu_swap_cluster.offset[0]); - if (pcp_si == si && pcp_offset) { - ci = swap_cluster_lock(si, pcp_offset); - if (cluster_is_usable(ci, 0)) - offset = alloc_swap_scan_cluster(si, ci, NULL, pcp_offset); - else - swap_cluster_unlock(ci); - } - if (!offset) - offset = cluster_alloc_swap_entry(si, NULL); - local_unlock(&percpu_swap_cluster.lock); + offset = cluster_alloc_swap_entry(si, NULL); if (offset) entry = swp_entry(si->type, offset); @@ -2205,26 +2444,18 @@ void swap_free_hibernation_slot(swp_entry_t entry) static int __find_hibernation_swap_type(dev_t device, sector_t offset) { - int type; - - lockdep_assert_held(&swap_lock); + struct swap_info_struct si; if (!device) return -EINVAL; - for (type = 0; type < nr_swapfiles; type++) { - struct swap_info_struct sis = swap_info[type]; - - if (!(sis->flags & SWP_WRITEOK)) - continue; - - if (device == sis->bdev->bd_dev) { - struct swap_extent se = first_se(sis); - - if (se->start_block == offset) - return type; + for_each_avail_swap(si) { + if (device == si->bdev->bd_dev) { + if (first_se(si)->start_block == offset) + return si->type; } } + return -ENODEV; } @@ -2247,21 +2478,18 @@ static int __find_hibernation_swap_type(dev_t device, sector_t offset) / int pin_hibernation_swap_type(dev_t device, sector_t offset) { - int type; + int ret; struct swap_info_struct si; - spin_lock(&swap_lock); - - type = __find_hibernation_swap_type(device, offset); - if (type < 0) { - spin_unlock(&swap_lock); - return type; - } + percpu_down_write(&swapon_rwsem); + ret = __find_hibernation_swap_type(device, offset); + if (ret < 0) + goto out; - si = swap_type_to_info(type); + si = swap_type_to_info(ret); if (WARN_ON_ONCE(!si)) { - spin_unlock(&swap_lock); - return -ENODEV; + ret = -ENODEV; + goto out; } /* @@ -2270,14 +2498,15 @@ int pin_hibernation_swap_type(dev_t device, sector_t offset) * the same session. / if (WARN_ON_ONCE(si->flags & SWP_HIBERNATION)) { - spin_unlock(&swap_lock); - return -EBUSY; + ret = -EBUSY; + goto out; } si->flags \|= SWP_HIBERNATION; - spin_unlock(&swap_lock); - return type; +out: + percpu_up_write(&swapon_rwsem); + return ret; } /* @@ -2294,14 +2523,11 @@ void unpin_hibernation_swap_type(int type) { struct swap_info_struct si; - spin_lock(&swap_lock); + percpu_down_write(&swapon_rwsem); si = swap_type_to_info(type); - if (!si) { - spin_unlock(&swap_lock); - return; - } - si->flags &= ~SWP_HIBERNATION; - spin_unlock(&swap_lock); + if (si) + si->flags &= ~SWP_HIBERNATION; + percpu_up_write(&swapon_rwsem); } /* @@ -2326,29 +2552,26 @@ int find_hibernation_swap_type(dev_t device, sector_t offset) { int type; - spin_lock(&swap_lock); + percpu_down_read(&swapon_rwsem); type = __find_hibernation_swap_type(device, offset); - spin_unlock(&swap_lock); + percpu_up_read(&swapon_rwsem); return type; } int find_first_swap(dev_t device) { - int type; - - spin_lock(&swap_lock); - for (type = 0; type < nr_swapfiles; type++) { - struct swap_info_struct sis = swap_info[type]; + int ret = -ENODEV; + struct swap_info_struct si; - if (!(sis->flags & SWP_WRITEOK)) - continue; - device = sis->bdev->bd_dev; - spin_unlock(&swap_lock); - return type; + percpu_down_read(&swapon_rwsem); + for_each_avail_swap(si) { + device = si->bdev->bd_dev; + ret = si->type; + break; } - spin_unlock(&swap_lock); - return -ENODEV; + percpu_up_read(&swapon_rwsem); + return ret; } / @@ -2376,7 +2599,7 @@ unsigned int count_swap_pages(int type, int free) { unsigned int n = 0; - spin_lock(&swap_lock); + percpu_down_read(&swapon_rwsem); if ((unsigned int)type < nr_swapfiles) { struct swap_info_struct sis = swap_info[type]; @@ -2388,7 +2611,7 @@ unsigned int count_swap_pages(int type, int free) } spin_unlock(&sis->lock); } - spin_unlock(&swap_lock); + percpu_up_read(&swapon_rwsem); return n; } #endif / CONFIG_HIBERNATION / @@ -2700,10 +2923,10 @@ static unsigned int find_next_to_unuse(struct swap_info_struct si, unsigned long swp_tb; /* - * No need for swap_lock here: we're just looking + * No need for swapon_rwsem here: we're just looking * for whether an entry is in use, not modifying it; false * hits are okay, and sys_swapoff() has already prevented new - * allocations from this area (while holding swap_lock). + * allocations from this area (while holding swapon_rwsem). / for (i = prev + 1; i < si->max; i++) { swp_tb = swap_table_get(__swap_offset_to_cluster(si, i), @@ -2816,7 +3039,7 @@ static int try_to_unuse(unsigned int type) success: / * Make sure that further cleanups after try_to_unuse() returns happen - * after swap_range_free() reduces si->inuse_pages to 0. + * after swap_device_inuse_sub() reduces si->inuse_pages to 0. / smp_mb(); return 0; @@ -2824,18 +3047,20 @@ static int try_to_unuse(unsigned int type) / * After a successful try_to_unuse, if no swap is now in use, we know - * we can empty the mmlist. swap_lock must be held on entry and exit. - * Note that mmlist_lock nests inside swap_lock, and an mm must be + * we can empty the mmlist. swapon_rwsem must be held on entry and exit. + * Note that mmlist_lock nests inside swapon_rwsem, and an mm must be * added to the mmlist just after page_duplicate - before would be racy. / static void drain_mmlist(void) { + struct swap_info_struct si; struct list_head p, next; - unsigned int type; - for (type = 0; type < nr_swapfiles; type++) - if (swap_usage_in_pages(swap_info[type])) + for_each_swap(si) { + if (swap_usage_in_pages(si)) return; + } + spin_lock(&mmlist_lock); list_for_each_safe(p, next, &init_mm.mmlist) list_del_init(p); @@ -2968,58 +3193,65 @@ static int setup_swap_extents(struct swap_info_struct sis, return generic_swapfile_activate(sis, swap_file, span); } -static void _enable_swap_info(struct swap_info_struct si) -{ - atomic_long_add(si->pages, &nr_swap_pages); - total_swap_pages += si->pages; - - assert_spin_locked(&swap_lock); - - plist_add(&si->list, &swap_active_head); - - /* Add back to available list / - add_to_avail_list(si, true); -} - / - * Called after the swap device is ready, resurrect its percpu ref, it's now - * safe to reference it. Add it to the list to expose it to the allocator. + * Called after the swap device is ready to be used. Marking it writable and + * exposing it to the allocator. Resurrect its percpu ref if it was dead before. / -static void enable_swap_info(struct swap_info_struct si) +static void swap_device_enable(struct swap_info_struct si) { - percpu_ref_resurrect(&si->users); - spin_lock(&swap_lock); - spin_lock(&si->lock); - _enable_swap_info(si); - spin_unlock(&si->lock); - spin_unlock(&swap_lock); -} + percpu_down_write(&swapon_rwsem); + spin_lock(&swap_queue_update_lock); + si->flags \|= SWP_WRITEOK; + spin_unlock(&swap_queue_update_lock); + atomic_long_add(si->pages, &nr_swap_pages); + total_swap_pages += si->pages; + percpu_up_write(&swapon_rwsem); -static void reinsert_swap_info(struct swap_info_struct si) -{ - spin_lock(&swap_lock); - spin_lock(&si->lock); - _enable_swap_info(si); - spin_unlock(&si->lock); - spin_unlock(&swap_lock); + add_to_avail_list(si); } -/* - * Called after clearing SWP_WRITEOK, ensures cluster_alloc_range - * see the updated flags, so there will be no more allocations. - / -static void wait_for_allocation(struct swap_info_struct si) +static int swap_device_disable(struct swap_info_struct si) { unsigned long offset; unsigned long end = ALIGN(si->max, SWAPFILE_CLUSTER); struct swap_cluster_info ci; - BUG_ON(si->flags & SWP_WRITEOK); + /* + * If SWP_WRITEOK is not set: another process already disabling it. + * If SWP_HIBERNATION is set: the device is pinned for hibernation. + / + percpu_down_write(&swapon_rwsem); + if (!(si->flags & SWP_WRITEOK) \|\| + si->flags & SWP_HIBERNATION) { + percpu_up_write(&swapon_rwsem); + return -EBUSY; + } + + if (security_vm_enough_memory_mm(current->mm, si->pages)) { + percpu_up_write(&swapon_rwsem); + return -ENOMEM; + } + vm_unacct_memory(si->pages); + + spin_lock(&swap_queue_update_lock); + si->flags &= ~SWP_WRITEOK; + spin_unlock(&swap_queue_update_lock); + total_swap_pages -= si->pages; + atomic_long_sub(si->pages, &nr_swap_pages); + del_from_avail_list(si, true); + percpu_up_write(&swapon_rwsem); + / + * Swap allocator doesn't touch si lock, so looping through all + * ci locks ensures __swap_cluster_alloc_entries sees the + * updated flags, and no more allocations will occur. + / for (offset = 0; offset < end; offset += SWAPFILE_CLUSTER) { ci = swap_cluster_lock(si, offset); swap_cluster_unlock(ci); } + + return 0; } static void free_swap_cluster_info(struct swap_cluster_info cluster_info, @@ -3043,28 +3275,6 @@ static void free_swap_cluster_info(struct swap_cluster_info cluster_info, kvfree(cluster_info); } -/ - * Called after swap device's reference count is dead, so - * neither scan nor allocation will use it. - / -static void flush_percpu_swap_cluster(struct swap_info_struct si) -{ - int cpu, i; - struct swap_info_struct *pcp_si; - - for_each_possible_cpu(cpu) { - pcp_si = per_cpu_ptr(percpu_swap_cluster.si, cpu); - / - * Invalidate the percpu swap cluster cache, si->users - * is dead, so no new user will point to it, just flush - * any existing user. - / - for (i = 0; i < SWAP_NR_ORDERS; i++) - cmpxchg(&pcp_si[i], si, NULL); - } -} - - SYSCALL_DEFINE1(swapoff, const char __user , specialfile) { struct swap_info_struct p = NULL; @@ -3086,44 +3296,22 @@ SYSCALL_DEFINE1(swapoff, const char __user , specialfile) return PTR_ERR(victim); mapping = victim->f_mapping; - spin_lock(&swap_lock); - plist_for_each_entry(p, &swap_active_head, list) { - if (p->flags & SWP_WRITEOK) { - if (p->swap_file->f_mapping == mapping) { - found = 1; - break; - } + percpu_down_read(&swapon_rwsem); + for_each_avail_swap(p) { + if (p->swap_file->f_mapping == mapping) { + found = 1; + break; } } - if (!found) { - err = -EINVAL; - spin_unlock(&swap_lock); - goto out_dput; - } - - /* Refuse swapoff while the device is pinned for hibernation / - if (p->flags & SWP_HIBERNATION) { - err = -EBUSY; - spin_unlock(&swap_lock); - goto out_dput; - } + percpu_up_read(&swapon_rwsem); + filp_close(victim, NULL); - if (!security_vm_enough_memory_mm(current->mm, p->pages)) - vm_unacct_memory(p->pages); - else { - err = -ENOMEM; - spin_unlock(&swap_lock); - goto out_dput; - } - spin_lock(&p->lock); - del_from_avail_list(p, true); - plist_del(&p->list, &swap_active_head); - atomic_long_sub(p->pages, &nr_swap_pages); - total_swap_pages -= p->pages; - spin_unlock(&p->lock); - spin_unlock(&swap_lock); + if (!found) + return -EINVAL; - wait_for_allocation(p); + err = swap_device_disable(p); + if (err) + return err; set_current_oom_origin(); err = try_to_unuse(p->type); @@ -3131,10 +3319,13 @@ SYSCALL_DEFINE1(swapoff, const char __user , specialfile) if (err) { /* re-insert swap space back into swap_list / - reinsert_swap_info(p); - goto out_dput; + swap_device_enable(p); + return err; } + percpu_down_write(&swapon_rwsem); + swap_queue_del(p); + / * Wait for swap operations protected by get/put_swap_device() * to complete. Because of synchronize_rcu() here, all swap @@ -3149,15 +3340,12 @@ SYSCALL_DEFINE1(swapoff, const char __user , specialfile) flush_work(&p->discard_work); flush_work(&p->reclaim_work); - flush_percpu_swap_cluster(p); destroy_swap_extents(p, p->swap_file); if (!(p->flags & SWP_SOLIDSTATE)) atomic_dec(&nr_rotate_swap); - mutex_lock(&swapon_mutex); - spin_lock(&swap_lock); spin_lock(&p->lock); drain_mmlist(); @@ -3168,10 +3356,11 @@ SYSCALL_DEFINE1(swapoff, const char __user , specialfile) p->max = 0; p->cluster_info = NULL; spin_unlock(&p->lock); - spin_unlock(&swap_lock); + percpu_up_write(&swapon_rwsem); arch_swap_invalidate_area(p->type); zswap_swapoff(p->type); - mutex_unlock(&swapon_mutex); + free_percpu(p->percpu_cluster); + p->percpu_cluster = NULL; kfree(p->global_cluster); p->global_cluster = NULL; free_swap_cluster_info(cluster_info, maxpages); @@ -3187,18 +3376,19 @@ SYSCALL_DEFINE1(swapoff, const char __user , specialfile) Clear the SWP_USED flag after all resources are freed so that swapon * can reuse this swap_info in alloc_swap_info() safely. It is ok to * not hold p->lock after we cleared its SWP_WRITEOK. + * + * The write lock ensures the flag clear is visible to lockless + * readers of swap_type_to_info() before alloc_swap_info() reuses + * this slot. / - spin_lock(&swap_lock); + percpu_down_write(&swapon_rwsem); p->flags = 0; - spin_unlock(&swap_lock); + percpu_up_write(&swapon_rwsem); - err = 0; atomic_inc(&proc_poll_event); wake_up_interruptible(&proc_poll_wait); -out_dput: - filp_close(victim, NULL); - return err; + return 0; } #ifdef CONFIG_PROC_FS @@ -3216,20 +3406,18 @@ static __poll_t swaps_poll(struct file file, poll_table wait) return EPOLLIN \| EPOLLRDNORM; } -/ iterator / static void swap_start(struct seq_file swap, loff_t pos) { struct swap_info_struct si; - int type; loff_t l = pos; - mutex_lock(&swapon_mutex); + percpu_down_read(&swapon_rwsem); if (!l) return SEQ_START_TOKEN; - for (type = 0; (si = swap_type_to_info(type)); type++) { - if (!(si->swap_file)) + for_each_swap(si) { + if (!si->swap_file) continue; if (!--l) return si; @@ -3250,7 +3438,7 @@ static void swap_next(struct seq_file swap, void v, loff_t pos) ++(pos); for (; (si = swap_type_to_info(type)); type++) { - if (!(si->swap_file)) + if (!(si->flags & SWP_USED) \|\| !(si->swap_file)) continue; return si; } @@ -3260,7 +3448,7 @@ static void swap_next(struct seq_file swap, void v, loff_t pos) static void swap_stop(struct seq_file swap, void v) { - mutex_unlock(&swapon_mutex); + percpu_up_read(&swapon_rwsem); } static int swap_show(struct seq_file swap, void v) @@ -3353,13 +3541,13 @@ static struct swap_info_struct alloc_swap_info(void) return ERR_PTR(-ENOMEM); } - spin_lock(&swap_lock); + percpu_down_write(&swapon_rwsem); for (type = 0; type < nr_swapfiles; type++) { if (!(swap_info[type]->flags & SWP_USED)) break; } if (type >= MAX_SWAPFILES) { - spin_unlock(&swap_lock); + percpu_up_write(&swapon_rwsem); percpu_ref_exit(&p->users); kvfree(p); return ERR_PTR(-EPERM); @@ -3381,10 +3569,8 @@ static struct swap_info_struct alloc_swap_info(void) / } p->swap_extent_root = RB_ROOT; - plist_node_init(&p->list, 0); - plist_node_init(&p->avail_list, 0); p->flags = SWP_USED; - spin_unlock(&swap_lock); + percpu_up_write(&swapon_rwsem); if (defer) { percpu_ref_exit(&defer->users); kvfree(defer); @@ -3453,9 +3639,8 @@ __weak unsigned long arch_max_swapfile_size(void) return generic_max_swapfile_size(); } -static unsigned long read_swap_header(struct swap_info_struct si, - union swap_header swap_header, - struct inode inode) +static unsigned long read_swap_header(union swap_header swap_header, + struct inode inode) { int i; unsigned long maxpages; @@ -3522,7 +3707,7 @@ static int setup_swap_clusters_info(struct swap_info_struct si, { unsigned long nr_clusters = DIV_ROUND_UP(maxpages, SWAPFILE_CLUSTER); struct swap_cluster_info cluster_info; - int err = -ENOMEM; + int cpu, err = -ENOMEM; unsigned long i; cluster_info = kvzalloc_objs(cluster_info, nr_clusters); @@ -3532,13 +3717,26 @@ static int setup_swap_clusters_info(struct swap_info_struct si, for (i = 0; i < nr_clusters; i++) spin_lock_init(&cluster_info[i].lock); - if (!(si->flags & SWP_SOLIDSTATE)) { + if (si->flags & SWP_SOLIDSTATE) { + si->percpu_cluster = alloc_percpu(struct percpu_cluster); + if (!si->percpu_cluster) + goto err; + + for_each_possible_cpu(cpu) { + struct percpu_cluster cluster; + + cluster = per_cpu_ptr(si->percpu_cluster, cpu); + for (i = 0; i < SWAP_NR_ORDERS; i++) + cluster->next[i] = SWAP_ENTRY_INVALID; + local_lock_init(&cluster->lock); + } + } else { si->global_cluster = kmalloc_obj(si->global_cluster); if (!si->global_cluster) goto err; for (i = 0; i < SWAP_NR_ORDERS; i++) si->global_cluster->next[i] = SWAP_ENTRY_INVALID; - spin_lock_init(&si->global_cluster_lock); + spin_lock_init(&si->global_cluster->lock); } / @@ -3621,7 +3819,7 @@ SYSCALL_DEFINE2(swapon, const char __user , specialfile, int, swap_flags) / * Allocate or reuse existing !SWP_USED swap_info. The returned * si will stay in a dying status, so nothing will access its content - * until enable_swap_info resurrects its percpu ref and expose it. + * until swap_device_enable resurrects its percpu ref and expose it. / si = alloc_swap_info(); if (IS_ERR(si)) @@ -3679,7 +3877,7 @@ SYSCALL_DEFINE2(swapon, const char __user , specialfile, int, swap_flags) } swap_header = kmap_local_folio(folio, 0); - maxpages = read_swap_header(si, swap_header, inode); + maxpages = read_swap_header(swap_header, inode); if (unlikely(!maxpages)) { error = -EINVAL; goto bad_swap_unlock_inode; @@ -3700,11 +3898,6 @@ SYSCALL_DEFINE2(swapon, const char __user , specialfile, int, swap_flags) maxpages = si->max; - / Set up the swap cluster info / - error = setup_swap_clusters_info(si, swap_header, maxpages); - if (error) - goto bad_swap_unlock_inode; - if (si->bdev && bdev_stable_writes(si->bdev)) si->flags \|= SWP_STABLE_WRITES; @@ -3718,6 +3911,15 @@ SYSCALL_DEFINE2(swapon, const char __user , specialfile, int, swap_flags) inced_nr_rotate_swap = true; } + /* + * Set up the swap cluster info. This must run after SWP_SOLIDSTATE + * is determined above, as it decides whether to allocate the per-CPU + * cluster (solid state) or the global cluster (rotational). + / + error = setup_swap_clusters_info(si, swap_header, maxpages); + if (error) + goto bad_swap_unlock_inode; + if ((swap_flags & SWAP_FLAG_DISCARD) && si->bdev && bdev_max_discard_sectors(si->bdev)) { / @@ -3764,22 +3966,22 @@ SYSCALL_DEFINE2(swapon, const char __user , specialfile, int, swap_flags) goto free_swap_zswap; } - mutex_lock(&swapon_mutex); prio = DEF_SWAP_PRIO; if (swap_flags & SWAP_FLAG_PREFER) prio = swap_flags & SWAP_FLAG_PRIO_MASK; - / - * The plist prio is negated because plist ordering is - * low-to-high, while swap ordering is high-to-low - / si->prio = prio; - si->list.prio = -si->prio; - si->avail_list.prio = -si->prio; si->swap_file = swap_file; / Sets SWP_WRITEOK, resurrect the percpu ref, expose the swap device / - enable_swap_info(si); + percpu_ref_resurrect(&si->users); + percpu_down_write(&swapon_rwsem); + error = swap_queue_add(si); + percpu_up_write(&swapon_rwsem); + if (error) + goto free_swap_zswap; + + swap_device_enable(si); pr_info("Adding %uk swap on %s. Priority:%d extents:%d across:%lluk %s%s%s%s\n", K(si->pages), name->name, si->prio, nr_extents, @@ -3789,7 +3991,6 @@ SYSCALL_DEFINE2(swapon, const char __user , specialfile, int, swap_flags) (si->flags & SWP_AREA_DISCARD) ? "s" : "", (si->flags & SWP_PAGE_DISCARD) ? "c" : ""); - mutex_unlock(&swapon_mutex); atomic_inc(&proc_poll_event); wake_up_interruptible(&proc_poll_wait); @@ -3800,6 +4001,8 @@ SYSCALL_DEFINE2(swapon, const char __user , specialfile, int, swap_flags) bad_swap_unlock_inode: inode_unlock(inode); bad_swap: + free_percpu(si->percpu_cluster); + si->percpu_cluster = NULL; kfree(si->global_cluster); si->global_cluster = NULL; inode = NULL; @@ -3810,9 +4013,9 @@ SYSCALL_DEFINE2(swapon, const char __user , specialfile, int, swap_flags) * Clear the SWP_USED flag after all resources are freed so * alloc_swap_info can reuse this si safely. / - spin_lock(&swap_lock); + percpu_down_write(&swapon_rwsem); si->flags = 0; - spin_unlock(&swap_lock); + percpu_up_write(&swapon_rwsem); if (inced_nr_rotate_swap) atomic_dec(&nr_rotate_swap); if (swap_file) @@ -3827,19 +4030,17 @@ SYSCALL_DEFINE2(swapon, const char __user , specialfile, int, swap_flags) void si_swapinfo(struct sysinfo val) { - unsigned int type; + struct swap_info_struct si; unsigned long nr_to_be_unused = 0; - spin_lock(&swap_lock); - for (type = 0; type < nr_swapfiles; type++) { - struct swap_info_struct si = swap_info[type]; - - if ((si->flags & SWP_USED) && !(si->flags & SWP_WRITEOK)) + percpu_down_read(&swapon_rwsem); + for_each_swap(si) { + if (!(si->flags & SWP_WRITEOK)) nr_to_be_unused += swap_usage_in_pages(si); } val->freeswap = atomic_long_read(&nr_swap_pages) + nr_to_be_unused; val->totalswap = total_swap_pages + nr_to_be_unused; - spin_unlock(&swap_lock); + percpu_up_read(&swapon_rwsem); } / @@ -3879,7 +4080,7 @@ int swap_dup_entry_direct(swp_entry_t entry) #if defined(CONFIG_MEMCG) && defined(CONFIG_BLK_CGROUP) static bool __has_usable_swap(void) { - return !plist_head_empty(&swap_active_head); + return READ_ONCE(total_swap_pages) > 0; } void __folio_throttle_swaprate(struct folio folio, gfp_t gfp) @@ -3902,14 +4103,14 @@ void __folio_throttle_swaprate(struct folio folio, gfp_t gfp) if (current->throttle_disk) return; - spin_lock(&swap_avail_lock); - plist_for_each_entry(si, &swap_avail_head, avail_list) { + percpu_down_read(&swapon_rwsem); + for_each_avail_swap(si) { if (si->bdev) { blkcg_schedule_throttle(si->bdev->bd_disk, true); break; } } - spin_unlock(&swap_avail_lock); + percpu_up_read(&swapon_rwsem); } #endif
3/2	2026/07/14 00:43	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 25861 output: 27 thoughts: 420 Thoughts: My Assessment of the Swap Subsystem Patch Okay, so I've got this patch – a pretty significant one, it looks like – that's fundamentally changing the Linux kernel's swap subsystem. It's replacing the current `plist`-based queue for swap devices with a custom round-robin implementation. Immediately, I can tell this is going to touch core memory management, which is always a red flag for potential issues. The fact that it's rewriting so much of `mm/swapfile.c` – and then touching all those other header and supporting files – `include/linux/swap.h`, `include/linux/plist.h`, `lib/plist.c`, and `mm/swap.h` – tells me this isn't just a minor tweak. It's a genuine architectural change. Removing `plist_requeue` is a clue, too. This introduction of new structures like `swap_queue_readers`, `swap_queue`, `swap_queue_len`, `swap_prio_ring`, along with the use of `swapon_rwsem` to replace the old locking mechanisms, indicates a re-thinking of how swap space is allocated and managed. This is a functional change, a major functional change to the very core of swap allocation logic. My immediate thought is, "This needs some serious fuzzing." Given the potential for memory corruption, race conditions, or unexpected behavior when interacting with other kernel subsystems, I want to zero in on the key functions that have been modified. Let's see: * `swap_alloc_entry` (and its replacements of `swap_alloc_fast` and `swap_alloc_slow`) – This is my primary area of interest, it's where the swap entry allocation actually happens. * `folio_alloc_swap` – another critical entry point. * The queue manipulation functions: `swap_queue_add`, `swap_queue_del`, `swap_queue_get_device` - these are the points of interactions with the new structures. * `swap_device_enable`, `swap_device_disable` - the points of device change are definitely worth checking. * `sys_swapon`, `sys_swapoff` – The user-space interface is always a good target. * The `swap_device_inuse_` functions – these are likely the place where the old locking was, this is also crucial. `add_to_avail_list`, `del_from_avail_list` – this is a good place to look at the new list manipulation. * The new queue mask functions `swap_queue_mask`, `swap_queue_unmask` * `cluster_alloc_swap_entry` Alright, I'll need to figure out any relevant configuration options, as well. Since this is in `mm/swapfile.c`, it's almost certainly controlled by `CONFIG_SWAP`. I should check the `mm/Makefile` to confirm, but that's a given. I am eager to get started.
4/2	2026/07/14 00:43	tool	grepper	0m Args: { "Expression": "swapfile.o", "PathPrefix": "mm/Makefile" } Results: { "Output": "mm/Makefile=76=endif\nmm/Makefile-77-\nmm/Makefile:78:obj-$(CONFIG_SWAP)\t+= page_io.o swap_state.o swapfile.o\nmm/Makefile-79-obj-$(CONFIG_ZSWAP)\t+= zswap.o\n" }
5/2	2026/07/14 00:43	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 27059 output: 56 thoughts: 230 Thoughts: My Initial Assessment of the Swap Configuration Dependency Alright, it's clear that `CONFIG_SWAP` is a prerequisite here. That's the baseline. Now, I need to understand the core logic this change is touching. The most relevant functions seem to be centered around queuing, device management, and allocation. Specifically, I'm thinking about `swap_queue_add`, `swap_queue_del`, `swap_queue_get_device`, `swap_device_enable`, `swap_device_disable`, `swap_alloc_entry`, `swap_device_inuse_add`, `swap_device_inuse_sub`, `add_to_avail_list`, `del_from_avail_list`, `swap_queue_mask`, and `swap_queue_unmask`. These all seem to reside within `mm/swapfile.c`, which makes sense. It looks like the change interacts with `CONFIG_HIBERNATION` too, also in `mm/swapfile.c`, and that's bringing in functions like `pin_hibernation_swap_type`, `unpin_hibernation_swap_type`, `find_hibernation_swap_type`, `find_first_swap`, and `count_swap_pages`. While hibernation is involved, it appears the central focus is really the mechanics of swap allocation itself. This reinforces my initial hunch about the critical functions involved, I need to focus on this part to proceed. Let's get down to specifics, I need to pull up this list of functions again.
6/2	2026/07/14 00:44	tool	grepper	0m Args: { "Expression": "^(static )?(int\|void\|bool\|struct swap_info_struct \\) [a-zA-Z0-9_]+\\(", "PathPrefix": "mm/swapfile.c" } Results:* { "Output": "mm/swapfile.c-50-\nmm/swapfile.c:51:static void swap_device_inuse_add(struct swap_info_struct si,\nmm/swapfile.c-52-\t\t\t\t unsigned int nr_entries);\nmm/swapfile.c:53:static bool folio_swapcache_freeable(struct folio folio);\nmm/swapfile.c:54:static void move_cluster(struct swap_info_struct si,\nmm/swapfile.c-55-\t\t\t struct swap_cluster_info ci, struct list_head list,\nmm/swapfile.c-56-\t\t\t enum swap_cluster_flags new_flags);\nmm/swapfile.c:57:static bool get_swap_device_info(struct swap_info_struct si);\nmm/swapfile.c-58-\n--\nmm/swapfile.c=208=static struct swap_queue_reader __percpu swap_queue_prealloc_readers(int nr_rings, gfp_t gfp)\n--\nmm/swapfile.c-219-\nmm/swapfile.c:220:static void swap_queue_install_readers(struct swap_queue_reader __percpu readers)\nmm/swapfile.c-221-{\n--\nmm/swapfile.c=242=static struct swap_info_struct swap_queue_get_device(long nr_alloc, int nr_iter)\n--\nmm/swapfile.c-285-\nmm/swapfile.c:286:static bool swap_queue_find(struct swap_info_struct si,\nmm/swapfile.c-287-\t\t\t unsigned int ring_idx, unsigned int dev_idx)\n--\nmm/swapfile.c-309-\nmm/swapfile.c:310:static void swap_queue_mask(struct swap_info_struct si)\nmm/swapfile.c-311-{\n--\nmm/swapfile.c-319-\nmm/swapfile.c:320:static void swap_queue_unmask(struct swap_info_struct si)\nmm/swapfile.c-321-{\n--\nmm/swapfile.c-329-\nmm/swapfile.c:330:static int swap_queue_add(struct swap_info_struct si)\nmm/swapfile.c-331-{\n--\nmm/swapfile.c-405-\nmm/swapfile.c:406:static void swap_queue_del(struct swap_info_struct si)\nmm/swapfile.c-407-{\n--\nmm/swapfile.c=489=static long swap_usage_in_pages(struct swap_info_struct si)\n--\nmm/swapfile.c-497- /\nmm/swapfile.c:498:static void swap_alloc_lock_device(struct swap_info_struct si)\nmm/swapfile.c-499-{\n--\nmm/swapfile.c-505-\nmm/swapfile.c:506:static void swap_alloc_unlock_device(struct swap_info_struct si)\nmm/swapfile.c-507-{\n--\nmm/swapfile.c-523-\nmm/swapfile.c:524:static bool swap_only_has_cache(struct swap_cluster_info ci,\nmm/swapfile.c-525-\t\t\t\tunsigned long offset, int nr_pages)\n--\nmm/swapfile.c-545- /\nmm/swapfile.c:546:static int __try_to_reclaim_swap(struct swap_info_struct si,\nmm/swapfile.c-547-\t\t\t\t unsigned long offset, unsigned long flags)\n--\nmm/swapfile.c=616=static inline struct swap_extent next_se(struct swap_extent se)\n--\nmm/swapfile.c-625- /\nmm/swapfile.c:626:static int discard_swap(struct swap_info_struct si)\nmm/swapfile.c-627-{\n--\nmm/swapfile.c=679=sector_t swap_folio_sector(struct folio folio)\n--\nmm/swapfile.c-695- /\nmm/swapfile.c:696:static void discard_swap_cluster(struct swap_info_struct si,\nmm/swapfile.c-697-\t\t\t\t pgoff_t start_page, pgoff_t nr_pages)\n--\nmm/swapfile.c=755=static inline unsigned int cluster_offset(struct swap_info_struct si,\n--\nmm/swapfile.c-760-\nmm/swapfile.c:761:static void swap_cluster_free_table_folio_rcu_cb(struct rcu_head head)\nmm/swapfile.c-762-{\n--\nmm/swapfile.c-768-\nmm/swapfile.c:769:static void swap_cluster_free_table(struct swap_cluster_info ci)\nmm/swapfile.c-770-{\n--\nmm/swapfile.c-796-\nmm/swapfile.c:797:static int swap_cluster_alloc_table(struct swap_cluster_info ci, gfp_t gfp)\nmm/swapfile.c-798-{\n--\nmm/swapfile.c-845- /\nmm/swapfile.c:846:static void swap_cluster_assert_empty(struct swap_cluster_info ci,\nmm/swapfile.c-847-\t\t\t\t unsigned int ci_off, unsigned int nr,\n--\nmm/swapfile.c=875=swap_cluster_populate(struct swap_info_struct si,\n--\nmm/swapfile.c-923-\nmm/swapfile.c:924:static void move_cluster(struct swap_info_struct si,\nmm/swapfile.c-925-\t\t\t struct swap_cluster_info ci, struct list_head list,\n--\nmm/swapfile.c-942-/* Add a cluster to discard list and schedule it to do discard /\nmm/swapfile.c:943:static void swap_cluster_schedule_discard(struct swap_info_struct si,\nmm/swapfile.c-944-\t\tstruct swap_cluster_info ci)\n--\nmm/swapfile.c-950-\nmm/swapfile.c:951:static void __free_cluster(struct swap_info_struct si, struct swap_cluster_info ci)\nmm/swapfile.c-952-{\n--\nmm/swapfile.c=968=static struct swap_cluster_info isolate_lock_cluster(\n--\nmm/swapfile.c-1007- /\nmm/swapfile.c:1008:static bool swap_do_scheduled_discard(struct swap_info_struct si)\nmm/swapfile.c-1009-{\n--\nmm/swapfile.c-1042-\nmm/swapfile.c:1043:static void swap_discard_work(struct work_struct work)\nmm/swapfile.c-1044-{\n--\nmm/swapfile.c-1051-\nmm/swapfile.c:1052:static void swap_users_ref_free(struct percpu_ref ref)\nmm/swapfile.c-1053-{\n--\nmm/swapfile.c-1063- /\nmm/swapfile.c:1064:static void free_cluster(struct swap_info_struct si, struct swap_cluster_info ci)\nmm/swapfile.c-1065-{\n--\nmm/swapfile.c-1087- /\nmm/swapfile.c:1088:static void partial_free_cluster(struct swap_info_struct si,\nmm/swapfile.c-1089-\t\t\t\t struct swap_cluster_info ci)\n--\nmm/swapfile.c-1103- /\nmm/swapfile.c:1104:static void relocate_cluster(struct swap_info_struct si,\nmm/swapfile.c-1105-\t\t\t struct swap_cluster_info ci)\n--\nmm/swapfile.c-1131- /\nmm/swapfile.c:1132:static int swap_cluster_setup_bad_slot(struct swap_info_struct si,\nmm/swapfile.c-1133-\t\t\t\t struct swap_cluster_info cluster_info,\n--\nmm/swapfile.c-1181- /\nmm/swapfile.c:1182:static bool cluster_reclaim_range(struct swap_info_struct si,\nmm/swapfile.c-1183-\t\t\t\t struct swap_cluster_info ci,\n--\nmm/swapfile.c-1228-\nmm/swapfile.c:1229:static bool cluster_scan_range(struct swap_info_struct si,\nmm/swapfile.c-1230-\t\t\t struct swap_cluster_info ci,\n--\nmm/swapfile.c-1255-\nmm/swapfile.c:1256:static bool __swap_cluster_alloc_entries(struct swap_info_struct si,\nmm/swapfile.c-1257-\t\t\t\t\t struct swap_cluster_info ci,\n--\nmm/swapfile.c=1358=static unsigned int alloc_swap_scan_list(struct swap_info_struct si,\n--\nmm/swapfile.c-1379-\nmm/swapfile.c:1380:static void swap_reclaim_full_clusters(struct swap_info_struct si, bool force)\nmm/swapfile.c-1381-{\n--\nmm/swapfile.c-1421-\nmm/swapfile.c:1422:static void swap_reclaim_work(struct work_struct work)\nmm/swapfile.c-1423-{\n--\nmm/swapfile.c=1435=static unsigned long cluster_alloc_swap_entry(struct swap_info_struct si,\n--\nmm/swapfile.c-1537-/ SWAP_USAGE_OFFLIST_BIT can only be set by this helper. /\nmm/swapfile.c:1538:static void del_from_avail_list(struct swap_info_struct si, bool swapoff)\nmm/swapfile.c-1539-{\n--\nmm/swapfile.c-1565-/* SWAP_USAGE_OFFLIST_BIT can only be cleared by this helper. /\nmm/swapfile.c:1566:static void add_to_avail_list(struct swap_info_struct si)\nmm/swapfile.c-1567-{\n--\nmm/swapfile.c-1610- /\nmm/swapfile.c:1611:static void swap_device_inuse_add(struct swap_info_struct si,\nmm/swapfile.c-1612-\t\t\t\t unsigned int nr_entries)\n--\nmm/swapfile.c-1629-\nmm/swapfile.c:1630:static void swap_device_inuse_sub(struct swap_info_struct si, unsigned long offset,\nmm/swapfile.c-1631-\t\t\t\t unsigned int nr_entries)\n--\nmm/swapfile.c-1668-\nmm/swapfile.c:1669:static bool get_swap_device_info(struct swap_info_struct si)\nmm/swapfile.c-1670-{\n--\nmm/swapfile.c-1684-\nmm/swapfile.c:1685:static int swap_alloc_entry(struct folio folio)\nmm/swapfile.c-1686-{\n--\nmm/swapfile.c-1720-\nmm/swapfile.c:1721:static int swap_extend_table_alloc(struct swap_info_struct si,\nmm/swapfile.c-1722-\t\t\t\t struct swap_cluster_info ci,\n--\nmm/swapfile.c-1756-\nmm/swapfile.c:1757:int swap_retry_table_alloc(swp_entry_t entry, gfp_t gfp)\nmm/swapfile.c-1758-{\n--\nmm/swapfile.c-1774-\nmm/swapfile.c:1775:static void swap_extend_table_try_free(struct swap_cluster_info ci)\nmm/swapfile.c-1776-{\n--\nmm/swapfile.c-1794-/* Decrease the swap count of one slot, without freeing it /\nmm/swapfile.c:1795:static void __swap_cluster_put_entry(struct swap_cluster_info ci,\nmm/swapfile.c-1796-\t\t\t\t unsigned int ci_off)\n--\nmm/swapfile.c-1845- /\nmm/swapfile.c:1846:static void swap_put_entries_cluster(struct swap_info_struct si,\nmm/swapfile.c-1847-\t\t\t\t pgoff_t offset, int nr,\n--\nmm/swapfile.c-1902-/* Increase the swap count of one slot. /\nmm/swapfile.c:1903:static int __swap_cluster_dup_entry(struct swap_cluster_info ci,\nmm/swapfile.c-1904-\t\t\t\t unsigned int ci_off)\n--\nmm/swapfile.c-1952- /\nmm/swapfile.c:1953:static int swap_dup_entries_cluster(struct swap_info_struct si,\nmm/swapfile.c-1954-\t\t\t\t pgoff_t offset, int nr)\n--\nmm/swapfile.c-1997- /\nmm/swapfile.c:1998:int folio_alloc_swap(struct folio folio)\nmm/swapfile.c-1999-{\n--\nmm/swapfile.c-2053- /\nmm/swapfile.c:2054:int folio_dup_swap(struct folio folio, struct page page)\nmm/swapfile.c-2055-{\n--\nmm/swapfile.c-2079- /\nmm/swapfile.c:2080:void folio_put_swap(struct folio folio, struct page page)\nmm/swapfile.c-2081-{\n--\nmm/swapfile.c=2134=struct swap_info_struct get_swap_device(swp_entry_t entry)\n--\nmm/swapfile.c-2164- /\nmm/swapfile.c:2165:void __swap_cluster_free_entries(struct swap_info_struct si,\nmm/swapfile.c-2166-\t\t\t\t struct swap_cluster_info ci,\n--\nmm/swapfile.c-2215-\nmm/swapfile.c:2216:int __swap_count(swp_entry_t entry)\nmm/swapfile.c-2217-{\n--\nmm/swapfile.c-2228- /\nmm/swapfile.c:2229:bool swap_entry_swapped(struct swap_info_struct si, swp_entry_t entry)\nmm/swapfile.c-2230-{\n--\nmm/swapfile.c-2245- /\nmm/swapfile.c:2246:int swp_swapcount(swp_entry_t entry)\nmm/swapfile.c-2247-{\n--\nmm/swapfile.c-2278- /\nmm/swapfile.c:2279:static bool folio_maybe_swapped(struct folio folio)\nmm/swapfile.c-2280-{\n--\nmm/swapfile.c-2307-\nmm/swapfile.c:2308:static bool folio_swapcache_freeable(struct folio folio)\nmm/swapfile.c-2309-{\n--\nmm/swapfile.c-2346- /\nmm/swapfile.c:2347:bool folio_free_swap(struct folio folio)\nmm/swapfile.c-2348-{\n--\nmm/swapfile.c-2371- /\nmm/swapfile.c:2372:void swap_put_entries_direct(swp_entry_t entry, int nr)\nmm/swapfile.c-2373-{\n--\nmm/swapfile.c=2407=swp_entry_t swap_alloc_hibernation_slot(int type)\n--\nmm/swapfile.c-2429- /\nmm/swapfile.c:2430:void swap_free_hibernation_slot(swp_entry_t entry)\nmm/swapfile.c-2431-{\n--\nmm/swapfile.c-2444-\nmm/swapfile.c:2445:static int __find_hibernation_swap_type(dev_t device, sector_t offset)\nmm/swapfile.c-2446-{\n--\nmm/swapfile.c-2478- /\nmm/swapfile.c:2479:int pin_hibernation_swap_type(dev_t device, sector_t offset)\nmm/swapfile.c-2480-{\n--\nmm/swapfile.c-2521- /\nmm/swapfile.c:2522:void unpin_hibernation_swap_type(int type)\nmm/swapfile.c-2523-{\n--\nmm/swapfile.c-2550- /\nmm/swapfile.c:2551:int find_hibernation_swap_type(dev_t device, sector_t offset)\nmm/swapfile.c-2552-{\n--\nmm/swapfile.c-2561-\nmm/swapfile.c:2562:int find_first_swap(dev_t device)\nmm/swapfile.c-2563-{\n--\nmm/swapfile.c=2619=static inline int pte_same_as_swp(pte_t pte, pte_t swp_pte)\n--\nmm/swapfile.c-2628- /\nmm/swapfile.c:2629:static int unuse_pte(struct vm_area_struct vma, pmd_t pmd,\nmm/swapfile.c-2630-\t\tunsigned long addr, swp_entry_t entry, struct folio folio)\n--\nmm/swapfile.c-2741-\nmm/swapfile.c:2742:static int unuse_pte_range(struct vm_area_struct vma, pmd_t pmd,\nmm/swapfile.c-2743-\t\t\tunsigned long addr, unsigned long end,\n--\nmm/swapfile.c=2850=static inline int unuse_p4d_range(struct vm_area_struct vma, pgd_t pgd,\n--\nmm/swapfile.c-2869-\nmm/swapfile.c:2870:static int unuse_vma(struct vm_area_struct vma, unsigned int type)\nmm/swapfile.c-2871-{\n--\nmm/swapfile.c-2890-\nmm/swapfile.c:2891:static int unuse_mm(struct mm_struct mm, unsigned int type)\nmm/swapfile.c-2892-{\n--\nmm/swapfile.c=2919=static unsigned int find_next_to_unuse(struct swap_info_struct si,\n--\nmm/swapfile.c-2945-\nmm/swapfile.c:2946:static int try_to_unuse(unsigned int type)\nmm/swapfile.c-2947-{\n--\nmm/swapfile.c-3053- /\nmm/swapfile.c:3054:static void drain_mmlist(void)\nmm/swapfile.c-3055-{\n--\nmm/swapfile.c-3072- /\nmm/swapfile.c:3073:static void destroy_swap_extents(struct swap_info_struct sis,\nmm/swapfile.c-3074-\t\t\t\t struct file swap_file)\n--\nmm/swapfile.c=3138=EXPORT_SYMBOL_GPL(add_swap_extent);\n--\nmm/swapfile.c-3166- /\nmm/swapfile.c:3167:static int setup_swap_extents(struct swap_info_struct sis,\nmm/swapfile.c-3168-\t\t\t struct file swap_file, sector_t span)\n--\nmm/swapfile.c-3199- /\nmm/swapfile.c:3200:static void swap_device_enable(struct swap_info_struct si)\nmm/swapfile.c-3201-{\n--\nmm/swapfile.c-3212-\nmm/swapfile.c:3213:static int swap_device_disable(struct swap_info_struct si)\nmm/swapfile.c-3214-{\n--\nmm/swapfile.c-3256-\nmm/swapfile.c:3257:static void free_swap_cluster_info(struct swap_cluster_info cluster_info,\nmm/swapfile.c-3258-\t\t\t\t unsigned long maxpages)\n--\nmm/swapfile.c=3429=static void swap_next(struct seq_file swap, void v, loff_t pos)\n--\nmm/swapfile.c-3448-\nmm/swapfile.c:3449:static void swap_stop(struct seq_file swap, void v)\nmm/swapfile.c-3450-{\n--\nmm/swapfile.c-3453-\nmm/swapfile.c:3454:static int swap_show(struct seq_file swap, void v)\nmm/swapfile.c-3455-{\n--\nmm/swapfile.c=3481=static const struct seq_operations swaps_op = {\n--\nmm/swapfile.c-3487-\nmm/swapfile.c:3488:static int swaps_open(struct inode inode, struct file file)\nmm/swapfile.c-3489-{\n--\nmm/swapfile.c=3528=static struct swap_info_struct alloc_swap_info(void)\n--\nmm/swapfile.c-3584-\nmm/swapfile.c:3585:static int claim_swapfile(struct swap_info_struct si, struct inode inode)\nmm/swapfile.c-3586-{\n--\nmm/swapfile.c=3642=static unsigned long read_swap_header(union swap_header swap_header,\n--\nmm/swapfile.c-3703-\nmm/swapfile.c:3704:static int setup_swap_clusters_info(struct swap_info_struct si,\nmm/swapfile.c-3705-\t\t\t\t union swap_header swap_header,\n--\nmm/swapfile.c=3797=SYSCALL_DEFINE2(swapon, const char __user , specialfile, int, swap_flags)\n--\nmm/swapfile.c-4030-\nmm/swapfile.c:4031:void si_swapinfo(struct sysinfo val)\nmm/swapfile.c-4032-{\n--\nmm/swapfile.c-4059- /\nmm/swapfile.c:4060:int swap_dup_entry_direct(swp_entry_t entry)\nmm/swapfile.c-4061-{\n--\nmm/swapfile.c-4080-#if defined(CONFIG_MEMCG) \u0026\u0026 defined(CONFIG_BLK_CGROUP)\nmm/swapfile.c:4081:static bool __has_usable_swap(void)\nmm/swapfile.c-4082-{\n--\nmm/swapfile.c-4085-\nmm/swapfile.c:4086:void __folio_throttle_swaprate(struct folio *folio, gfp_t gfp)\nmm/swapfile.c-4087-{\n" }
7/2	2026/07/14 00:44	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 33805 output: 199 thoughts: 578 Thoughts: My Fuzzing Strategy for This Swap Patch Okay, let's break this down. My initial scan has highlighted a handful of functions as potential hotspots in this swap implementation. Given the architectural change described – moving from a `plist` to a round-robin queue – I need to pinpoint the areas where this new logic is introduced and interacts with the existing swap framework. I'm zeroing in on a few key functions: `swap_alloc_entry` looks like the new core for allocation, replacing some of the older logic. I'm also putting a pin on `swap_queue_add`, `swap_queue_del`, `swap_queue_get_device`, `swap_device_enable`, and `swap_device_disable` as these seem to be the primary interactions with the new queue mechanism. The in-use counters, `swap_device_inuse_add` and `swap_device_inuse_sub`, are crucial for ensuring correctness as well as the list manipulation functions `add_to_avail_list`, `del_from_avail_list`, `swap_queue_mask`, `swap_queue_unmask` and `cluster_alloc_swap_entry` are also essential. I'm filtering out `folio_alloc_swap`, as interesting as it may be, because I suspect it is a more generic hot path, which, while important for performance, won't necessarily expose the core logic changes I'm after. `swap_alloc_entry` looks like a better target. Now, let's confirm the relevant configuration options. `CONFIG_SWAP` seems to be the main driver. I'm quickly verifying that `mm/swapfile.c` is compiled when that's enabled. Yep, that confirms it. Therefore, for this fuzzing effort, I'm going to set `WorthFuzzing` to `true` given the functional change and its impact on core kernel swap logic. The `FocusSymbols` are: * `swap_alloc_entry` * `swap_queue_add` * `swap_queue_del` * `swap_queue_get_device` * `swap_device_enable` * `swap_device_disable` * `swap_device_inuse_add` * `swap_device_inuse_sub` * `add_to_avail_list` * `del_from_avail_list` * `swap_queue_mask` * `swap_queue_unmask` * `cluster_alloc_swap_entry` And I'll enable only `SWAP` to build the kernel image. My reasoning is that the introduction of a new round-robin queue, and its associated management functions, introduces new code that needs careful fuzzing. This is a functional change with the potential for subtle bugs in the kernel's memory management.
8/2	2026/07/14 00:44	tool	set-results	0m Args: { "EnableConfigs": [ "SWAP" ], "FocusSymbols": [ "swap_alloc_entry", "swap_queue_add", "swap_queue_del", "swap_queue_get_device", "swap_device_enable", "swap_device_disable", "swap_device_inuse_add", "swap_device_inuse_sub", "add_to_avail_list", "del_from_avail_list", "swap_queue_mask", "swap_queue_unmask", "cluster_alloc_swap_entry" ], "Reasoning": "The patch replaces the plist-based swap device queue with a custom round-robin queue, introducing new core logic for swap allocation and device management. This is a functional change to the core kernel logic and should be fuzzed.", "WorthFuzzing": true } Results: { "EnableConfigs": [ "SWAP" ], "FocusSymbols": [ "swap_alloc_entry", "swap_queue_add", "swap_queue_del", "swap_queue_get_device", "swap_device_enable", "swap_device_disable", "swap_device_inuse_add", "swap_device_inuse_sub", "add_to_avail_list", "del_from_avail_list", "swap_queue_mask", "swap_queue_unmask", "cluster_alloc_swap_entry" ], "Reasoning": "The patch replaces the plist-based swap device queue with a custom round-robin queue, introducing new core logic for swap allocation and device management. This is a functional change to the core kernel logic and should be fuzzed.", "WorthFuzzing": true }

Trajectory: (Expand/Collapse All)

Seq

Timestamp

Type

Name

Duration

0/0

2026/07/14 00:43

flow

patch-triage

Args:

null

Results:

{
  "EnableConfigs": [
    "SWAP"
  ],
  "FocusSymbols": [
    "swap_alloc_entry",
    "swap_queue_add",
    "swap_queue_del",
    "swap_queue_get_device",
    "swap_device_enable",
    "swap_device_disable",
    "swap_device_inuse_add",
    "swap_device_inuse_sub",
    "add_to_avail_list",
    "del_from_avail_list",
    "swap_queue_mask",
    "swap_queue_unmask",
    "cluster_alloc_swap_entry"
  ],
  "Reasoning": "The patch replaces the plist-based swap device queue with a custom round-robin queue, introducing new core logic for swap allocation and device management. This is a functional change to the core kernel logic and should be fuzzed.",
  "WorthFuzzing": true
}

1/1

2026/07/14 00:43

action

read-patch-diff

Args:

null

Results:

{
  "PatchDiff": "commit 9f349365e0eb1fe2128a8484bb4ca651f1dbd2a2\nAuthor: syz-cluster \u003ctriage@syzkaller.com\u003e\nDate:   Tue Jul 14 00:43:44 2026 +0000\n\n    syz-cluster: applied patch under review\n\ndiff --git a/include/linux/plist.h b/include/linux/plist.h\nindex 16cf4355b5c131..73d359c1a0536a 100644\n--- a/include/linux/plist.h\n+++ b/include/linux/plist.h\n@@ -132,8 +132,6 @@ static inline void plist_node_init(struct plist_node *node, int prio)\n extern void plist_add(struct plist_node *node, struct plist_head *head);\n extern void plist_del(struct plist_node *node, struct plist_head *head);\n \n-extern void plist_requeue(struct plist_node *node, struct plist_head *head);\n-\n /**\n  * plist_for_each - iterate over the plist\n  * @pos:\tthe type * to use as a loop counter\ndiff --git a/include/linux/swap.h b/include/linux/swap.h\nindex 696ed01709c2ba..04767540972b81 100644\n--- a/include/linux/swap.h\n+++ b/include/linux/swap.h\n@@ -194,6 +194,22 @@ struct swap_extent {\n \t((offsetof(union swap_header, magic.magic) - \\\n \t  offsetof(union swap_header, info.badpages)) / sizeof(int))\n \n+/*\n+ * Swap device flags, except the ones documented below, all are immutable\n+ * after exposed by swap_device_enable, and until the device is freed again\n+ * (SWP_USED unset). The exceptions are all protected by swapon_rwsem:\n+ * - SWP_USED: Protected by swapon_rwsem. Indicates the device is inuse. Once\n+ *   set, won't be cleared unless all reference to this device is freed and\n+ *   swapoff finished.\n+ * - SWP_WRITEOK: Protected by both swapon_rwsem and swap_avail_lock, clearing\n+ *   this flag also waits for all current cluster lock users to exit so\n+ *   checking this flag while holding any of these locks ensures the device\n+ *   is safe to use at the moment. Note: clearing this flag doesn't affect\n+ *   pending IO or async requests, it only prevents further entry allocation\n+ *   or new async request (e.g. discard) from initiating.\n+ * - SWP_HIBERNATION: Protected by swapon_rwsem. Indicates if the device\n+ *   is pinned for hibernation.\n+ */\n enum {\n \tSWP_USED\t= (1 \u003c\u003c 0),\t/* is slot in swap_info[] used? */\n \tSWP_WRITEOK\t= (1 \u003c\u003c 1),\t/* ok to write to this swap?\t*/\n@@ -230,10 +246,17 @@ enum {\n #endif\n \n /*\n- * We keep using same cluster for rotational device so IO will be sequential.\n- * The purpose is to optimize SWAP throughput on these device.\n+ * We assign a cluster to each CPU, so each CPU can allocate swap entry from\n+ * its own cluster and swapout sequentially. The purpose is to optimize swapout\n+ * throughput.\n  */\n+struct percpu_cluster {\n+\tlocal_lock_t lock; /* Protect the percpu_cluster above */\n+\tunsigned int next[SWAP_NR_ORDERS]; /* Likely next allocation offset */\n+};\n+\n struct swap_sequential_cluster {\n+\tspinlock_t lock; /* Serialize usage of global cluster */\n \tunsigned int next[SWAP_NR_ORDERS]; /* Likely next allocation offset */\n };\n \n@@ -244,7 +267,6 @@ struct swap_info_struct {\n \tstruct percpu_ref users;\t/* indicate and keep swap device valid. */\n \tunsigned long\tflags;\t\t/* SWP_USED etc: see above */\n \tsigned short\tprio;\t\t/* swap priority of this type */\n-\tstruct plist_node list;\t\t/* entry in swap_active_head */\n \tsigned char\ttype;\t\t/* strange name for an index */\n \tunsigned int\tmax;\t\t/* size of this swap device */\n \tstruct swap_cluster_info *cluster_info; /* cluster info. Only for SSD */\n@@ -256,26 +278,20 @@ struct swap_info_struct {\n \t\t\t\t\t/* list of cluster that are fragmented or contented */\n \tunsigned int pages;\t\t/* total of usable pages of swap */\n \tatomic_long_t inuse_pages;\t/* number of those currently in use */\n+\tstruct percpu_cluster\t__percpu *percpu_cluster; /* per cpu's swap location */\n \tstruct swap_sequential_cluster *global_cluster; /* Use one global cluster for rotating device */\n-\tspinlock_t global_cluster_lock;\t/* Serialize usage of global cluster */\n \tstruct rb_root swap_extent_root;/* root of the swap extent rbtree */\n \tstruct block_device *bdev;\t/* swap device or bdev of swap file */\n \tstruct file *swap_file;\t\t/* seldom referenced */\n \tstruct completion comp;\t\t/* seldom referenced */\n \tspinlock_t lock;\t\t/*\n-\t\t\t\t\t * protect map scan related fields like\n-\t\t\t\t\t * inuse_pages and all cluster lists.\n-\t\t\t\t\t * Other fields are only changed\n-\t\t\t\t\t * at swapon/swapoff, so are protected\n-\t\t\t\t\t * by swap_lock. changing flags need\n-\t\t\t\t\t * hold this lock and swap_lock. If\n-\t\t\t\t\t * both locks need hold, hold swap_lock\n-\t\t\t\t\t * first.\n+\t\t\t\t\t * Protect cluster lists. Other fields\n+\t\t\t\t\t * are only changed at swapon/swapoff,\n+\t\t\t\t\t * so are protected by swapon_rwsem.\n \t\t\t\t\t */\n \tstruct work_struct discard_work; /* discard worker */\n \tstruct work_struct reclaim_work; /* reclaim worker */\n \tstruct list_head discard_clusters; /* discard clusters list */\n-\tstruct plist_node avail_list;   /* entry in swap_avail_head */\n };\n \n static inline swp_entry_t page_swap_entry(struct page *page)\ndiff --git a/lib/plist.c b/lib/plist.c\nindex a5bef38add431d..b05ae7ffea87e8 100644\n--- a/lib/plist.c\n+++ b/lib/plist.c\n@@ -142,58 +142,6 @@ void plist_del(struct plist_node *node, struct plist_head *head)\n \tplist_check_head(head);\n }\n \n-/**\n- * plist_requeue - Requeue @node at end of same-prio entries.\n- *\n- * This is essentially an optimized plist_del() followed by\n- * plist_add().  It moves an entry already in the plist to\n- * after any other same-priority entries.\n- *\n- * @node:\t\u0026struct plist_node pointer - entry to be moved\n- * @head:\t\u0026struct plist_head pointer - list head\n- */\n-void plist_requeue(struct plist_node *node, struct plist_head *head)\n-{\n-\tstruct plist_node *iter;\n-\tstruct list_head *node_next = \u0026head-\u003enode_list;\n-\n-\tplist_check_head(head);\n-\tBUG_ON(plist_head_empty(head));\n-\tBUG_ON(plist_node_empty(node));\n-\n-\tif (node == plist_last(head))\n-\t\treturn;\n-\n-\titer = plist_next(node);\n-\n-\tif (node-\u003eprio != iter-\u003eprio)\n-\t\treturn;\n-\n-\tplist_del(node, head);\n-\n-\t/*\n-\t * After plist_del(), iter is the replacement of the node.  If the node\n-\t * was on prio_list, take shortcut to find node_next instead of looping.\n-\t */\n-\tif (!list_empty(\u0026iter-\u003eprio_list)) {\n-\t\titer = list_entry(iter-\u003eprio_list.next, struct plist_node,\n-\t\t\t\t  prio_list);\n-\t\tnode_next = \u0026iter-\u003enode_list;\n-\t\tgoto queue;\n-\t}\n-\n-\tplist_for_each_continue(iter, head) {\n-\t\tif (node-\u003eprio != iter-\u003eprio) {\n-\t\t\tnode_next = \u0026iter-\u003enode_list;\n-\t\t\tbreak;\n-\t\t}\n-\t}\n-queue:\n-\tlist_add_tail(\u0026node-\u003enode_list, node_next);\n-\n-\tplist_check_head(head);\n-}\n-\n #ifdef CONFIG_DEBUG_PLIST\n #include \u003clinux/sched.h\u003e\n #include \u003clinux/sched/clock.h\u003e\n@@ -231,14 +179,6 @@ static void __init plist_test_check(int nr_expect)\n \tBUG_ON(prio_pos-\u003eprio_list.next != \u0026first-\u003eprio_list);\n }\n \n-static void __init plist_test_requeue(struct plist_node *node)\n-{\n-\tplist_requeue(node, \u0026test_head);\n-\n-\tif (node != plist_last(\u0026test_head))\n-\t\tBUG_ON(node-\u003eprio == plist_next(node)-\u003eprio);\n-}\n-\n static int  __init plist_test(void)\n {\n \tint nr_expect = 0, i, loop;\n@@ -262,10 +202,6 @@ static int  __init plist_test(void)\n \t\t\tnr_expect--;\n \t\t}\n \t\tplist_test_check(nr_expect);\n-\t\tif (!plist_node_empty(test_node + i)) {\n-\t\t\tplist_test_requeue(test_node + i);\n-\t\t\tplist_test_check(nr_expect);\n-\t\t}\n \t}\n \n \tfor (i = 0; i \u003c ARRAY_SIZE(test_node); i++) {\ndiff --git a/mm/swap.h b/mm/swap.h\nindex b51ad3071a7308..08a782f3d5acd2 100644\n--- a/mm/swap.h\n+++ b/mm/swap.h\n@@ -91,16 +91,16 @@ static inline unsigned int swp_cluster_offset(swp_entry_t entry)\n  */\n static inline struct swap_info_struct *__swap_type_to_info(int type)\n {\n-\tstruct swap_info_struct *si;\n-\n-\tsi = READ_ONCE(swap_info[type]); /* rcu_dereference() */\n-\tVM_WARN_ON_ONCE(percpu_ref_is_zero(\u0026si-\u003eusers)); /* race with swapoff */\n-\treturn si;\n+\treturn READ_ONCE(swap_info[type]); /* rcu_dereference() */\n }\n \n static inline struct swap_info_struct *__swap_entry_to_info(swp_entry_t entry)\n {\n-\treturn __swap_type_to_info(swp_type(entry));\n+\tstruct swap_info_struct *si;\n+\n+\tsi = __swap_type_to_info(swp_type(entry)); /* rcu_dereference() */\n+\tVM_WARN_ON_ONCE(percpu_ref_is_zero(\u0026si-\u003eusers)); /* race with swapoff */\n+\treturn si;\n }\n \n static inline struct swap_cluster_info *__swap_offset_to_cluster(\ndiff --git a/mm/swapfile.c b/mm/swapfile.c\nindex 775c385490a5fa..f93863309a5a50 100644\n--- a/mm/swapfile.c\n+++ b/mm/swapfile.c\n@@ -41,7 +41,6 @@\n #include \u003clinux/completion.h\u003e\n #include \u003clinux/suspend.h\u003e\n #include \u003clinux/zswap.h\u003e\n-#include \u003clinux/plist.h\u003e\n \n #include \u003casm/tlbflush.h\u003e\n #include \u003clinux/leafops.h\u003e\n@@ -49,31 +48,38 @@\n #include \"internal.h\"\n #include \"swap.h\"\n \n-static void swap_range_alloc(struct swap_info_struct *si,\n-\t\t\t     unsigned int nr_entries);\n+static void swap_device_inuse_add(struct swap_info_struct *si,\n+\t\t\t\t  unsigned int nr_entries);\n static bool folio_swapcache_freeable(struct folio *folio);\n static void move_cluster(struct swap_info_struct *si,\n \t\t\t struct swap_cluster_info *ci, struct list_head *list,\n \t\t\t enum swap_cluster_flags new_flags);\n+static bool get_swap_device_info(struct swap_info_struct *si);\n \n /*\n- * Protects the swap_info array, and the SWP_USED flag. swap_info contains\n- * lazily allocated \u0026 freed swap device info struts, and SWP_USED indicates\n- * which device is used, ~SWP_USED devices and can be reused.\n- *\n- * Also protects swap_active_head total_swap_pages, and the SWP_WRITEOK flag.\n+ * Serializes swapon/swapoff (writers) and protects the swap_info\n+ * array, nr_swapfiles, total_swap_pages, and part of swap device\n+ * info content (see comment of swap_info_struct).  Readers\n+ * (allocation, /proc/swaps, etc.) take percpu_down_read() which\n+ * is cheap as the hot path.\n+ */\n+DEFINE_STATIC_PERCPU_RWSEM(swapon_rwsem);\n+/*\n+ * swap_info contains lazily allocated swap device info structs, and\n+ * SWP_USED indicates which device is used, ~SWP_USED devices can be\n+ * reused. Protected by swapon_rwsem, but reading could be lockless.\n  */\n-static DEFINE_SPINLOCK(swap_lock);\n+struct swap_info_struct *swap_info[MAX_SWAPFILES];\n static unsigned int nr_swapfiles;\n-atomic_long_t nr_swap_pages;\n+long total_swap_pages;\n+\n /*\n  * Some modules use swappable objects and may try to swap them out under\n  * memory pressure (via the shrinker). Before doing so, they may wish to\n  * check to see if any swap space is available.\n  */\n+atomic_long_t nr_swap_pages;\n EXPORT_SYMBOL_GPL(nr_swap_pages);\n-/* protected with swap_lock. reading in vm_swap_full() doesn't need lock */\n-long total_swap_pages;\n #define DEF_SWAP_PRIO  -1\n unsigned long swapfile_maximum_size;\n #ifdef CONFIG_MIGRATION\n@@ -83,52 +89,42 @@ bool swap_migration_ad_supported;\n static const char Bad_file[] = \"Bad swap file entry \";\n static const char Bad_offset[] = \"Bad swap offset entry \";\n \n-/*\n- * all active swap_info_structs\n- * protected with swap_lock, and ordered by priority.\n- */\n-static PLIST_HEAD(swap_active_head);\n+static inline struct swap_info_struct *__swap_iter(int *i, unsigned long flag)\n+{\n+\tlockdep_assert_held(\u0026swapon_rwsem);\n+\twhile (*i \u003c nr_swapfiles) {\n+\t\tstruct swap_info_struct *si = __swap_type_to_info(*i);\n+\n+\t\tVM_WARN_ON(!si);\n+\t\t(*i)++;\n+\t\tif (flag \u0026\u0026 !((si-\u003eflags \u0026 flag) == flag))\n+\t\t\tcontinue;\n+\t\treturn si;\n+\t}\n+\treturn NULL;\n+}\n+\n+#define __for_each_swap(si, flag)\t\t\t\\\n+\tfor (int __i = 0; ((si) = __swap_iter(\u0026__i, flag));)\n \n /*\n- * all available (active, not full) swap_info_structs\n- * protected with swap_avail_lock, ordered by priority.\n- * This is used by folio_alloc_swap() instead of swap_active_head\n- * because swap_active_head includes all swap_info_structs,\n- * but folio_alloc_swap() doesn't need to look at full ones.\n- * This uses its own lock instead of swap_lock because when a\n- * swap_info_struct changes between not-full/full, it needs to\n- * add/remove itself to/from this list, but the swap_info_struct-\u003elock\n- * is held and the locking order requires swap_lock to be taken\n- * before any swap_info_struct-\u003elock.\n+ * for_each_swap - iterate through all allocated and inuse swap devices\n+ * @si: the iterator\n+ *\n+ * Context: The caller must hold swapon_rwsem. The lock may be dropped during\n+ * the loop body but must be re-acquired before the next iteration.\n  */\n-static PLIST_HEAD(swap_avail_head);\n-static DEFINE_SPINLOCK(swap_avail_lock);\n-\n-struct swap_info_struct *swap_info[MAX_SWAPFILES];\n+#define for_each_swap(si) __for_each_swap(si, SWP_USED)\n+#define for_each_avail_swap(si) __for_each_swap(si, SWP_USED | SWP_WRITEOK)\n \n static struct kmem_cache *swap_table_cachep;\n \n-/* Protects si-\u003eswap_file for /proc/swaps usage */\n-static DEFINE_MUTEX(swapon_mutex);\n-\n static DECLARE_WAIT_QUEUE_HEAD(proc_poll_wait);\n /* Activity counter to indicate that a swapon or swapoff has occurred */\n static atomic_t proc_poll_event = ATOMIC_INIT(0);\n \n atomic_t nr_rotate_swap = ATOMIC_INIT(0);\n \n-struct percpu_swap_cluster {\n-\tstruct swap_info_struct *si[SWAP_NR_ORDERS];\n-\tunsigned long offset[SWAP_NR_ORDERS];\n-\tlocal_lock_t lock;\n-};\n-\n-static DEFINE_PER_CPU(struct percpu_swap_cluster, percpu_swap_cluster) = {\n-\t.si = { NULL },\n-\t.offset = { SWAP_ENTRY_INVALID },\n-\t.lock = INIT_LOCAL_LOCK(),\n-};\n-\n /* May return NULL on invalid type, caller must check for NULL return */\n static struct swap_info_struct *swap_type_to_info(int type)\n {\n@@ -143,18 +139,350 @@ static struct swap_info_struct *swap_entry_to_info(swp_entry_t entry)\n \treturn swap_type_to_info(swp_type(entry));\n }\n \n+/*\n+ * All available swap_info_structs are grouped by priority rings, the rings\n+ * are ordered in a queue by priority (lower prio value = higher priority).\n+ * The allocator iterates and rotates devices within each priority ring.\n+ * When all devices in a ring are iterated, it goes to the next lower\n+ * priority ring.\n+ */\n+struct swap_prio_ring {\n+\tint prio;\n+\tunsigned int size;\n+\tstruct swap_info_struct *dev[] __counted_by(size);\n+};\n+\n+/*\n+ * The ring is protected by swapon_rwsem so updating it is costly. To make\n+ * the allocator and other users skip full devices faster, the lowest bit of\n+ * a device pointer is used to mark it disabled (temporarily unavailable).\n+ * This relies on struct swap_info_struct being sufficiently\n+ * aligned (guaranteed by kmalloc).\n+ */\n+#define SWAP_DEVICE_MASKED_SHIFT\t0\n+#define SWAP_DEVICE_MASKED_BIT\t\tBIT(SWAP_DEVICE_MASKED_SHIFT)\n+\n+/*\n+ * Serializes queue content mutations and keeps SWAP_USAGE_OFFLIST_BIT\n+ * consistent with the masked state of each device pointer.\n+ */\n+static DEFINE_SPINLOCK(swap_queue_update_lock);\n+\n+/*\n+ * Swap queue is protected by both swap_queue_update_lock and swapon_rwsem.\n+ * Only swapon/swapoff will take the write lock, and modify the queue length\n+ * or any ring's length. swap_queue_update_lock protects the content so\n+ * devices can be masked easily without taking the writelock, which is heavy.\n+ */\n+static struct swap_prio_ring **swap_queue;\n+static unsigned int swap_queue_len;\n+\n+/*\n+ * Each CPU has its read iterator, so the queue itself will remain read\n+ * only and the CPU side reader rotates by iterating the devices\n+ * periodically using the counter.\n+ */\n+#define SWAP_ROUND_ROBIN_QUOTA SWAPFILE_CLUSTER\n+struct swap_ring_iterator {\n+\tint offset;\n+\tlong rr_counter;\n+};\n+\n+struct swap_queue_reader {\n+\tlocal_lock_t lock;\n+\tstruct swap_ring_iterator ri[];\n+};\n+\n+static __percpu struct swap_queue_reader *swap_queue_readers;\n+\n+static inline bool swap_device_masked(struct swap_info_struct *si)\n+{\n+\treturn (unsigned long)si \u0026 SWAP_DEVICE_MASKED_BIT;\n+}\n+\n+static inline struct swap_info_struct *swap_device_unmask_ptr(struct swap_info_struct *si)\n+{\n+\treturn (struct swap_info_struct *)((unsigned long)si \u0026 ~SWAP_DEVICE_MASKED_BIT);\n+}\n+\n+static struct swap_queue_reader __percpu *swap_queue_prealloc_readers(int nr_rings, gfp_t gfp)\n+{\n+\tstruct swap_queue_reader __percpu *readers;\n+\n+\tif (!nr_rings)\n+\t\treturn NULL;\n+\n+\treaders = __alloc_percpu_gfp(struct_size(readers, ri, nr_rings),\n+\t\t\t\t     __alignof__(*readers), gfp);\n+\treturn readers;\n+}\n+\n+static void swap_queue_install_readers(struct swap_queue_reader __percpu *readers)\n+{\n+\tint ring_idx, cpu;\n+\tstruct swap_prio_ring *ring;\n+\n+\tfree_percpu(swap_queue_readers);\n+\tswap_queue_readers = readers;\n+\tif (!readers)\n+\t\treturn;\n+\n+\t/* Distribute each CPU's swap IO fairly across devices. */\n+\tfor_each_possible_cpu(cpu) {\n+\t\tlocal_lock_init(\u0026per_cpu_ptr(readers, cpu)-\u003elock);\n+\n+\t\tfor (ring_idx = 0; ring_idx \u003c swap_queue_len; ring_idx++) {\n+\t\t\tring = swap_queue[ring_idx];\n+\t\t\tper_cpu_ptr(readers, cpu)-\u003eri[ring_idx].offset =\n+\t\t\t\t    cpu % ring-\u003esize;\n+\t\t}\n+\t}\n+}\n+\n+static struct swap_info_struct *swap_queue_get_device(long nr_alloc, int nr_iter)\n+{\n+\tbool rotate = false;\n+\tstruct swap_info_struct *si;\n+\tstruct swap_ring_iterator *ri;\n+\tstruct swap_prio_ring *ring;\n+\tunsigned int queue_idx;\n+\n+\tif (!swap_queue_len)\n+\t\treturn ERR_PTR(-ENOENT);\n+\n+\tqueue_idx = 0;\n+\twhile (nr_iter \u003e= swap_queue[queue_idx]-\u003esize) {\n+\t\tnr_iter -= swap_queue[queue_idx]-\u003esize;\n+\t\tif (++queue_idx \u003e= swap_queue_len)\n+\t\t\treturn ERR_PTR(-ENOENT);\n+\t}\n+\n+\tring = swap_queue[queue_idx];\n+\tlocal_lock(\u0026swap_queue_readers-\u003elock);\n+\tri = this_cpu_ptr(\u0026swap_queue_readers-\u003eri[queue_idx]);\n+\t/* Rotate while iterating the ring, just not on the first try */\n+\tif (nr_iter)\n+\t\trotate = true;\n+\telse if (ri-\u003err_counter \u003c nr_alloc)\n+\t\trotate = true;\n+\telse if (ri-\u003eoffset \u003e= ring-\u003esize)\n+\t\trotate = true;\n+\tif (rotate) {\n+\t\tri-\u003eoffset++;\n+\t\tri-\u003eoffset %= ring-\u003esize;\n+\t\tri-\u003err_counter = SWAP_ROUND_ROBIN_QUOTA;\n+\t}\n+\tri-\u003err_counter -= nr_alloc;\n+\tsi = READ_ONCE(ring-\u003edev[ri-\u003eoffset]);\n+\tlocal_unlock(\u0026swap_queue_readers-\u003elock);\n+\n+\tif (swap_device_masked(si))\n+\t\treturn ERR_PTR(-EBUSY);\n+\n+\tsi = swap_device_unmask_ptr(si);\n+\treturn si;\n+}\n+\n+static bool swap_queue_find(struct swap_info_struct *si,\n+\t\t\t    unsigned int *ring_idx, unsigned int *dev_idx)\n+{\n+\tunsigned int i, j;\n+\tstruct swap_prio_ring *ring;\n+\n+\tlockdep_assert(lockdep_is_held(\u0026swapon_rwsem) ||\n+\t\t       lockdep_is_held(\u0026swap_queue_update_lock));\n+\n+\tfor (i = 0; i \u003c swap_queue_len; i++) {\n+\t\tring = swap_queue[i];\n+\t\tif (ring-\u003eprio != si-\u003eprio)\n+\t\t\tcontinue;\n+\t\tfor (j = 0; j \u003c ring-\u003esize; j++) {\n+\t\t\tif (swap_device_unmask_ptr(READ_ONCE(ring-\u003edev[j])) != si)\n+\t\t\t\tcontinue;\n+\t\t\t*ring_idx = i;\n+\t\t\t*dev_idx = j;\n+\t\t\treturn true;\n+\t\t}\n+\t}\n+\treturn false;\n+}\n+\n+static void swap_queue_mask(struct swap_info_struct *si)\n+{\n+\tunsigned int ring_idx, dev_idx;\n+\n+\tlockdep_assert_held(\u0026swap_queue_update_lock);\n+\tif (swap_queue_find(si, \u0026ring_idx, \u0026dev_idx))\n+\t\t__set_bit(SWAP_DEVICE_MASKED_SHIFT,\n+\t\t\t  (unsigned long *)\u0026swap_queue[ring_idx]-\u003edev[dev_idx]);\n+}\n+\n+static void swap_queue_unmask(struct swap_info_struct *si)\n+{\n+\tunsigned int ring_idx, dev_idx;\n+\n+\tlockdep_assert_held(\u0026swap_queue_update_lock);\n+\tif (swap_queue_find(si, \u0026ring_idx, \u0026dev_idx))\n+\t\t__clear_bit(SWAP_DEVICE_MASKED_SHIFT,\n+\t\t\t    (unsigned long *)\u0026swap_queue[ring_idx]-\u003edev[dev_idx]);\n+}\n+\n+static int swap_queue_add(struct swap_info_struct *si)\n+{\n+\tstruct swap_prio_ring **new_queue = NULL, **old_queue = NULL;\n+\tstruct swap_queue_reader __percpu *new_readers = NULL;\n+\tstruct swap_prio_ring *ring, *new_ring = NULL, *old_ring = NULL;\n+\tint prio = si-\u003eprio;\n+\tint i, pos, err = -ENOMEM;\n+\tgfp_t gfp;\n+\n+\t/* Swap not usable here because this is swap, just reclaim cache. */\n+\tgfp = GFP_NOIO | __GFP_HIGH;\n+\tlockdep_assert_held_write(\u0026swapon_rwsem);\n+\n+\tfor (pos = 0; pos \u003c swap_queue_len; pos++) {\n+\t\tif (swap_queue[pos]-\u003eprio == prio)\n+\t\t\tgoto add_to_ring;\n+\t\tif (swap_queue[pos]-\u003eprio \u003c prio)\n+\t\t\tbreak;\n+\t}\n+\n+\t/* No ring at this priority: insert a new one at pos. */\n+\tnew_readers = swap_queue_prealloc_readers(swap_queue_len + 1, gfp);\n+\tif (!new_readers)\n+\t\tgoto failed;\n+\tnew_queue = kmalloc_array(swap_queue_len + 1, sizeof(*swap_queue), gfp);\n+\tif (!new_queue)\n+\t\tgoto failed;\n+\tnew_ring = kmalloc(struct_size(new_ring, dev, 1), gfp);\n+\tif (!new_ring)\n+\t\tgoto failed;\n+\tif (!get_swap_device_info(si))\n+\t\tgoto failed;\n+\n+\tnew_ring-\u003eprio = prio;\n+\tnew_ring-\u003esize = 1;\n+\tnew_ring-\u003edev[0] = si;\n+\tfor (i = 0; i \u003c pos; i++)\n+\t\tnew_queue[i] = swap_queue[i];\n+\tnew_queue[pos] = new_ring;\n+\tfor (i = pos; i \u003c swap_queue_len; i++)\n+\t\tnew_queue[i + 1] = swap_queue[i];\n+\n+\tspin_lock(\u0026swap_queue_update_lock);\n+\told_queue = swap_queue;\n+\tswap_queue = new_queue;\n+\tswap_queue_len++;\n+\tspin_unlock(\u0026swap_queue_update_lock);\n+\tkfree(old_queue);\n+\n+\tswap_queue_install_readers(new_readers);\n+\treturn 0;\n+\n+add_to_ring:\n+\tring = swap_queue[pos];\n+\tnew_ring = kmalloc(struct_size(ring, dev, ring-\u003esize + 1), gfp);\n+\tif (!new_ring)\n+\t\tgoto failed;\n+\tif (!get_swap_device_info(si))\n+\t\tgoto failed;\n+\tspin_lock(\u0026swap_queue_update_lock);\n+\tmemcpy(new_ring, ring, struct_size(ring, dev, ring-\u003esize));\n+\tnew_ring-\u003esize++;\n+\tnew_ring-\u003edev[new_ring-\u003esize - 1] = si;\n+\told_ring = swap_queue[pos];\n+\tswap_queue[pos] = new_ring;\n+\tspin_unlock(\u0026swap_queue_update_lock);\n+\tkfree(old_ring);\n+\treturn 0;\n+\n+failed:\n+\tfree_percpu(new_readers);\n+\tkfree(new_queue);\n+\tkfree(new_ring);\n+\treturn err;\n+}\n+\n+static void swap_queue_del(struct swap_info_struct *si)\n+{\n+\tgfp_t gfp;\n+\tunsigned int ring_idx, dev_idx;\n+\tstruct swap_queue_reader __percpu *new_readers = NULL;\n+\tstruct swap_prio_ring *ring, *new_ring = NULL, *old_ring = NULL;\n+\tstruct swap_prio_ring **new_queue = NULL, **old_queue = NULL;\n+\n+\tlockdep_assert_held_write(\u0026swapon_rwsem);\n+\tif (!swap_queue_find(si, \u0026ring_idx, \u0026dev_idx)) {\n+\t\tWARN_ON(1);\n+\t\treturn;\n+\t}\n+\n+\t/*\n+\t * To shrink memory usage, pre-allocate new smaller data before\n+\t * locking. Failure is fine, swapoff will release them anyway.\n+\t */\n+\tgfp = GFP_NOIO | __GFP_HIGH;\n+\tring = swap_queue[ring_idx];\n+\tif (ring-\u003esize \u003e 1)\n+\t\tnew_ring = kmalloc(struct_size(ring, dev, ring-\u003esize - 1), gfp);\n+\tif (ring-\u003esize == 1 \u0026\u0026 swap_queue_len \u003e 1) {\n+\t\tnew_readers = swap_queue_prealloc_readers(\n+\t\t\t\t\tswap_queue_len - 1, gfp);\n+\t\tnew_queue = kmalloc(sizeof(*swap_queue) *\n+\t\t\t\t    (swap_queue_len - 1), gfp);\n+\t}\n+\n+\tspin_lock(\u0026swap_queue_update_lock);\n+\tif (ring-\u003esize \u003e 1) {\n+\t\t/* Shift trailing devices left to fill the gap. */\n+\t\twhile (++dev_idx \u003c ring-\u003esize)\n+\t\t\tring-\u003edev[dev_idx - 1] =\n+\t\t\t\tring-\u003edev[dev_idx];\n+\t\tring-\u003esize--;\n+\t\tif (new_ring) {\n+\t\t\tmemcpy(new_ring, ring,\n+\t\t\t       struct_size(ring, dev, ring-\u003esize));\n+\t\t\told_ring = ring;\n+\t\t\tswap_queue[ring_idx] = new_ring;\n+\t\t}\n+\t} else {\n+\t\t/* Last device in this ring: remove the ring. */\n+\t\told_ring = ring;\n+\t\tswap_queue_len--;\n+\t\twhile (++ring_idx \u003c= swap_queue_len)\n+\t\t\tswap_queue[ring_idx - 1] =\n+\t\t\t\tswap_queue[ring_idx];\n+\t\tif (new_queue) {\n+\t\t\tmemcpy(new_queue, swap_queue,\n+\t\t\t       sizeof(*swap_queue) * swap_queue_len);\n+\t\t\told_queue = swap_queue;\n+\t\t\tswap_queue = new_queue;\n+\t\t} else if (!swap_queue_len) {\n+\t\t\told_queue = swap_queue;\n+\t\t\tswap_queue = NULL;\n+\t\t}\n+\t\tif (new_readers || !swap_queue_len)\n+\t\t\tswap_queue_install_readers(new_readers);\n+\t}\n+\tspin_unlock(\u0026swap_queue_update_lock);\n+\n+\tkfree(old_ring);\n+\tkfree(old_queue);\n+\tput_swap_device(si);\n+}\n+\n /*\n  * Use the second highest bit of inuse_pages counter as the indicator\n- * if one swap device is on the available plist, so the atomic can\n+ * if one swap device is unavailable for allocation, so the atomic can\n  * still be updated arithmetically while having special data embedded.\n  *\n  * inuse_pages counter is the only thing indicating if a device should\n- * be on avail_lists or not (except swapon / swapoff). By embedding the\n- * off-list bit in the atomic counter, updates no longer need any lock\n- * to check the list status.\n+ * be in the available queue or not (except swapon / swapoff). By\n+ * embedding the off-list bit in the atomic counter, updates no longer\n+ * need any lock to check the list status.\n  *\n- * This bit will be set if the device is not on the plist and not\n- * usable, will be cleared if the device is on the plist.\n+ * This bit will be set if the device is not in the available queue\n+ * and not usable, will be cleared if the device is in the queue.\n  */\n #define SWAP_USAGE_OFFLIST_BIT (1UL \u003c\u003c (BITS_PER_TYPE(atomic_t) - 2))\n #define SWAP_USAGE_COUNTER_MASK (~SWAP_USAGE_OFFLIST_BIT)\n@@ -163,6 +491,26 @@ static long swap_usage_in_pages(struct swap_info_struct *si)\n \treturn atomic_long_read(\u0026si-\u003einuse_pages) \u0026 SWAP_USAGE_COUNTER_MASK;\n }\n \n+/*\n+ * Serialize the allocation on single CPU or globally to avoid\n+ * fragmentation and make the workflow easier to follow.\n+ */\n+static void swap_alloc_lock_device(struct swap_info_struct *si)\n+{\n+\tif (si-\u003eflags \u0026 SWP_SOLIDSTATE)\n+\t\tlocal_lock(\u0026si-\u003epercpu_cluster-\u003elock);\n+\telse\n+\t\tspin_lock(\u0026si-\u003eglobal_cluster-\u003elock);\n+}\n+\n+static void swap_alloc_unlock_device(struct swap_info_struct *si)\n+{\n+\tif (si-\u003eflags \u0026 SWP_SOLIDSTATE)\n+\t\tlocal_unlock(\u0026si-\u003epercpu_cluster-\u003elock);\n+\telse\n+\t\tspin_unlock(\u0026si-\u003eglobal_cluster-\u003elock);\n+}\n+\n /* Reclaim the swap entry anyway if possible */\n #define TTRS_ANYWAY\t\t0x1\n /*\n@@ -533,9 +881,10 @@ swap_cluster_populate(struct swap_info_struct *si,\n \t * Only cluster isolation from the allocator does table allocation.\n \t * Swap allocator uses percpu clusters and holds the local lock.\n \t */\n-\tlockdep_assert_held(\u0026this_cpu_ptr(\u0026percpu_swap_cluster)-\u003elock);\n-\tif (!(si-\u003eflags \u0026 SWP_SOLIDSTATE))\n-\t\tlockdep_assert_held(\u0026si-\u003eglobal_cluster_lock);\n+\tif (si-\u003eflags \u0026 SWP_SOLIDSTATE)\n+\t\tlockdep_assert_held(this_cpu_ptr(\u0026si-\u003epercpu_cluster-\u003elock));\n+\telse\n+\t\tlockdep_assert_held(\u0026si-\u003eglobal_cluster-\u003elock);\n \tlockdep_assert_held(\u0026ci-\u003elock);\n \n \tif (!swap_cluster_alloc_table(ci, __GFP_HIGH | __GFP_NOMEMALLOC |\n@@ -548,9 +897,7 @@ swap_cluster_populate(struct swap_info_struct *si,\n \t * the potential recursive allocation is limited.\n \t */\n \tspin_unlock(\u0026ci-\u003elock);\n-\tif (!(si-\u003eflags \u0026 SWP_SOLIDSTATE))\n-\t\tspin_unlock(\u0026si-\u003eglobal_cluster_lock);\n-\tlocal_unlock(\u0026percpu_swap_cluster.lock);\n+\tswap_alloc_unlock_device(si);\n \n \tret = swap_cluster_alloc_table(ci, __GFP_HIGH | __GFP_NOMEMALLOC |\n \t\t\t\t\t   GFP_KERNEL);\n@@ -563,9 +910,7 @@ swap_cluster_populate(struct swap_info_struct *si,\n \t * could happen with ignoring the percpu cluster is fragmentation,\n \t * which is acceptable since this fallback and race is rare.\n \t */\n-\tlocal_lock(\u0026percpu_swap_cluster.lock);\n-\tif (!(si-\u003eflags \u0026 SWP_SOLIDSTATE))\n-\t\tspin_lock(\u0026si-\u003eglobal_cluster_lock);\n+\tswap_alloc_lock_device(si);\n \tspin_lock(\u0026ci-\u003elock);\n \n \tif (ret) {\n@@ -671,7 +1016,7 @@ static bool swap_do_scheduled_discard(struct swap_info_struct *si)\n \t\tci = list_first_entry(\u0026si-\u003ediscard_clusters, struct swap_cluster_info, list);\n \t\t/*\n \t\t * Delete the cluster from list to prepare for discard, but keep\n-\t\t * the CLUSTER_FLAG_DISCARD flag, percpu_swap_cluster could be\n+\t\t * the CLUSTER_FLAG_DISCARD flag, there could be percpu_cluster\n \t\t * pointing to it, or ran into by relocate_cluster.\n \t\t */\n \t\tlist_del(\u0026ci-\u003elist);\n@@ -955,7 +1300,7 @@ static bool __swap_cluster_alloc_entries(struct swap_info_struct *si,\n \tif (cluster_is_empty(ci))\n \t\tci-\u003eorder = order;\n \tci-\u003ecount += nr_pages;\n-\tswap_range_alloc(si, nr_pages);\n+\tswap_device_inuse_add(si, nr_pages);\n \n \treturn true;\n }\n@@ -1003,12 +1348,10 @@ static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si,\n out:\n \trelocate_cluster(si, ci);\n \tswap_cluster_unlock(ci);\n-\tif (si-\u003eflags \u0026 SWP_SOLIDSTATE) {\n-\t\tthis_cpu_write(percpu_swap_cluster.offset[order], next);\n-\t\tthis_cpu_write(percpu_swap_cluster.si[order], si);\n-\t} else {\n+\tif (si-\u003eflags \u0026 SWP_SOLIDSTATE)\n+\t\tthis_cpu_write(si-\u003epercpu_cluster-\u003enext[order], next);\n+\telse\n \t\tsi-\u003eglobal_cluster-\u003enext[order] = next;\n-\t}\n \treturn found;\n }\n \n@@ -1103,13 +1446,14 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,\n \tif (order \u0026\u0026 !(si-\u003eflags \u0026 SWP_BLKDEV))\n \t\treturn 0;\n \n-\tif (!(si-\u003eflags \u0026 SWP_SOLIDSTATE)) {\n-\t\t/* Serialize HDD SWAP allocation for each device. */\n-\t\tspin_lock(\u0026si-\u003eglobal_cluster_lock);\n+restart:\n+\tswap_alloc_lock_device(si);\n+\tif (si-\u003eflags \u0026 SWP_SOLIDSTATE)\n+\t\toffset = __this_cpu_read(si-\u003epercpu_cluster-\u003enext[order]);\n+\telse\n \t\toffset = si-\u003eglobal_cluster-\u003enext[order];\n-\t\tif (offset == SWAP_ENTRY_INVALID)\n-\t\t\tgoto new_cluster;\n \n+\tif (offset != SWAP_ENTRY_INVALID) {\n \t\tci = swap_cluster_lock(si, offset);\n \t\t/* Cluster could have been used by another order */\n \t\tif (cluster_is_usable(ci, order)) {\n@@ -1123,7 +1467,6 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,\n \t\t\tgoto done;\n \t}\n \n-new_cluster:\n \t/*\n \t * If the device need discard, prefer new cluster over nonfull\n \t * to spread out the writes.\n@@ -1132,6 +1475,12 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,\n \t\tfound = alloc_swap_scan_list(si, \u0026si-\u003efree_clusters, folio, false);\n \t\tif (found)\n \t\t\tgoto done;\n+\n+\t\tif (!list_empty(\u0026si-\u003ediscard_clusters)) {\n+\t\t\tswap_alloc_unlock_device(si);\n+\t\t\tswap_do_scheduled_discard(si);\n+\t\t\tgoto restart;\n+\t\t}\n \t}\n \n \tif (order \u003c PMD_ORDER) {\n@@ -1180,8 +1529,7 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,\n \t\t\tgoto done;\n \t}\n done:\n-\tif (!(si-\u003eflags \u0026 SWP_SOLIDSTATE))\n-\t\tspin_unlock(\u0026si-\u003eglobal_cluster_lock);\n+\tswap_alloc_unlock_device(si);\n \n \treturn found;\n }\n@@ -1191,54 +1539,44 @@ static void del_from_avail_list(struct swap_info_struct *si, bool swapoff)\n {\n \tunsigned long pages;\n \n-\tspin_lock(\u0026swap_avail_lock);\n+\tspin_lock(\u0026swap_queue_update_lock);\n \n-\tif (swapoff) {\n-\t\t/*\n-\t\t * Forcefully remove it. Clear the SWP_WRITEOK flags for\n-\t\t * swapoff here so it's synchronized by both si-\u003elock and\n-\t\t * swap_avail_lock, to ensure the result can be seen by\n-\t\t * add_to_avail_list.\n-\t\t */\n-\t\tlockdep_assert_held(\u0026si-\u003elock);\n-\t\tsi-\u003eflags \u0026= ~SWP_WRITEOK;\n-\t\tatomic_long_or(SWAP_USAGE_OFFLIST_BIT, \u0026si-\u003einuse_pages);\n-\t} else {\n-\t\t/*\n-\t\t * If not called by swapoff, take it off-list only if it's\n-\t\t * full and SWAP_USAGE_OFFLIST_BIT is not set (strictly\n-\t\t * si-\u003einuse_pages == pages), any concurrent slot freeing,\n-\t\t * or device already removed from plist by someone else\n-\t\t * will make this return false.\n-\t\t */\n+\t/*\n+\t * Force remove it only for swapoff. Else, take it off-list only if\n+\t * it's full and SWAP_USAGE_OFFLIST_BIT is not set (strictly\n+\t * si-\u003einuse_pages == pages), so concurrent slot freeing, or\n+\t * concurrent list removal will make the cmpxchg fail and skip\n+\t * the removal.\n+\t */\n+\tif (!swapoff) {\n \t\tpages = si-\u003epages;\n \t\tif (!atomic_long_try_cmpxchg(\u0026si-\u003einuse_pages, \u0026pages,\n \t\t\t\t\t     pages | SWAP_USAGE_OFFLIST_BIT))\n \t\t\tgoto skip;\n+\t} else {\n+\t\tatomic_long_or(SWAP_USAGE_OFFLIST_BIT, \u0026si-\u003einuse_pages);\n \t}\n \n-\tplist_del(\u0026si-\u003eavail_list, \u0026swap_avail_head);\n-\n+\tswap_queue_mask(si);\n skip:\n-\tspin_unlock(\u0026swap_avail_lock);\n+\tspin_unlock(\u0026swap_queue_update_lock);\n }\n \n /* SWAP_USAGE_OFFLIST_BIT can only be cleared by this helper. */\n-static void add_to_avail_list(struct swap_info_struct *si, bool swapon)\n+static void add_to_avail_list(struct swap_info_struct *si)\n {\n \tlong val;\n \tunsigned long pages;\n \n-\tspin_lock(\u0026swap_avail_lock);\n+\tspin_lock(\u0026swap_queue_update_lock);\n \n-\t/* Corresponding to SWP_WRITEOK clearing in del_from_avail_list */\n-\tif (swapon) {\n-\t\tlockdep_assert_held(\u0026si-\u003elock);\n-\t\tsi-\u003eflags |= SWP_WRITEOK;\n-\t} else {\n-\t\tif (!(READ_ONCE(si-\u003eflags) \u0026 SWP_WRITEOK))\n-\t\t\tgoto skip;\n-\t}\n+\t/*\n+\t * Mark the device as avail if SWP_WRITEOK is set. Swapoff clears\n+\t * SWP_WRITEOK first, so check that first so the device won't be\n+\t * re-added after swapoff started.\n+\t */\n+\tif (!(si-\u003eflags \u0026 SWP_WRITEOK))\n+\t\tgoto skip;\n \n \tif (!(atomic_long_read(\u0026si-\u003einuse_pages) \u0026 SWAP_USAGE_OFFLIST_BIT))\n \t\tgoto skip;\n@@ -1246,9 +1584,10 @@ static void add_to_avail_list(struct swap_info_struct *si, bool swapon)\n \tval = atomic_long_fetch_and_relaxed(~SWAP_USAGE_OFFLIST_BIT, \u0026si-\u003einuse_pages);\n \n \t/*\n-\t * When device is full and device is on the plist, only one updater will\n-\t * see (inuse_pages == si-\u003epages) and will call del_from_avail_list. If\n-\t * that updater happen to be here, just skip adding.\n+\t * When device is full and marked as available, one reader will see\n+\t * (inuse_pages == si-\u003epages) and should mark it as unavailable and\n+\t * set SWAP_USAGE_OFFLIST_BIT. If that updater happens to be here, just\n+\t * skip the rest.\n \t */\n \tpages = si-\u003epages;\n \tif (val == pages) {\n@@ -1258,60 +1597,42 @@ static void add_to_avail_list(struct swap_info_struct *si, bool swapon)\n \t\t\tgoto skip;\n \t}\n \n-\tplist_add(\u0026si-\u003eavail_list, \u0026swap_avail_head);\n-\n+\tswap_queue_unmask(si);\n skip:\n-\tspin_unlock(\u0026swap_avail_lock);\n+\tspin_unlock(\u0026swap_queue_update_lock);\n }\n \n /*\n- * swap_usage_add / swap_usage_sub of each slot are serialized by ci-\u003elock\n- * within each cluster, so the total contribution to the global counter should\n- * always be positive and cannot exceed the total number of usable slots.\n+ * swap_device_inuse_add / swap_device_inuse_sub are called within cluster\n+ * update critical sections and serialized by ci-\u003elock within each cluster,\n+ * so the total contribution to the global counter should always be positive\n+ * and cannot exceed the total number of usable slots.\n  */\n-static bool swap_usage_add(struct swap_info_struct *si, unsigned int nr_entries)\n+static void swap_device_inuse_add(struct swap_info_struct *si,\n+\t\t\t\t  unsigned int nr_entries)\n {\n-\tlong val = atomic_long_add_return_relaxed(nr_entries, \u0026si-\u003einuse_pages);\n+\tlong inuse_pages;\n \n \t/*\n \t * If device is full, and SWAP_USAGE_OFFLIST_BIT is not set,\n-\t * remove it from the plist.\n-\t */\n-\tif (unlikely(val == si-\u003epages)) {\n-\t\tdel_from_avail_list(si, false);\n-\t\treturn true;\n-\t}\n-\n-\treturn false;\n-}\n-\n-static void swap_usage_sub(struct swap_info_struct *si, unsigned int nr_entries)\n-{\n-\tlong val = atomic_long_sub_return_relaxed(nr_entries, \u0026si-\u003einuse_pages);\n-\n-\t/*\n-\t * If device is not full, and SWAP_USAGE_OFFLIST_BIT is set,\n-\t * add it to the plist.\n+\t * mark it unavailable.\n \t */\n-\tif (unlikely(val \u0026 SWAP_USAGE_OFFLIST_BIT))\n-\t\tadd_to_avail_list(si, false);\n-}\n-\n-static void swap_range_alloc(struct swap_info_struct *si,\n-\t\t\t     unsigned int nr_entries)\n-{\n-\tif (swap_usage_add(si, nr_entries)) {\n+\tinuse_pages = atomic_long_add_return_relaxed(nr_entries, \u0026si-\u003einuse_pages);\n+\tif (unlikely(inuse_pages == si-\u003epages)) {\n \t\tif (vm_swap_full())\n \t\t\tschedule_work(\u0026si-\u003ereclaim_work);\n+\t\tdel_from_avail_list(si, false);\n \t}\n+\n \tatomic_long_sub(nr_entries, \u0026nr_swap_pages);\n }\n \n-static void swap_range_free(struct swap_info_struct *si, unsigned long offset,\n-\t\t\t    unsigned int nr_entries)\n+static void swap_device_inuse_sub(struct swap_info_struct *si, unsigned long offset,\n+\t\t\t\t  unsigned int nr_entries)\n {\n \tunsigned long end = offset + nr_entries - 1;\n \tvoid (*swap_slot_free_notify)(struct block_device *, unsigned long);\n+\tlong inuse_pages;\n \tunsigned int i;\n \n \tfor (i = 0; i \u003c nr_entries; i++)\n@@ -1335,7 +1656,14 @@ static void swap_range_free(struct swap_info_struct *si, unsigned long offset,\n \t */\n \tsmp_wmb();\n \tatomic_long_add(nr_entries, \u0026nr_swap_pages);\n-\tswap_usage_sub(si, nr_entries);\n+\n+\t/*\n+\t * If device is not full, and SWAP_USAGE_OFFLIST_BIT is set,\n+\t * add it back to the available queue.\n+\t */\n+\tinuse_pages = atomic_long_sub_return_relaxed(nr_entries, \u0026si-\u003einuse_pages);\n+\tif (unlikely(inuse_pages \u0026 SWAP_USAGE_OFFLIST_BIT))\n+\t\tadd_to_avail_list(si);\n }\n \n static bool get_swap_device_info(struct swap_info_struct *si)\n@@ -1345,113 +1673,49 @@ static bool get_swap_device_info(struct swap_info_struct *si)\n \t/*\n \t * Guarantee the si-\u003eusers are checked before accessing other\n \t * fields of swap_info_struct, and si-\u003eflags (SWP_WRITEOK) is\n-\t * up to dated.\n+\t * up to date.\n \t *\n-\t * Paired with the spin_unlock() after setup_swap_info() in\n-\t * enable_swap_info(), and smp_wmb() in swapoff.\n+\t * Paired with percpu_up_write() in swap_device_enable(), and\n+\t * smp_wmb() after clearing SWP_WRITEOK in swapoff.\n \t */\n \tsmp_rmb();\n \treturn true;\n }\n \n-/*\n- * Fast path try to get swap entries with specified order from current\n- * CPU's swap entry pool (a cluster).\n- */\n-static bool swap_alloc_fast(struct folio *folio)\n+static int swap_alloc_entry(struct folio *folio)\n {\n-\tunsigned int order = folio_order(folio);\n-\tstruct swap_cluster_info *ci;\n+\tlong nr_pages = folio_nr_pages(folio);\n \tstruct swap_info_struct *si;\n-\tunsigned int offset;\n-\n-\t/*\n-\t * Once allocated, swap_info_struct will never be completely freed,\n-\t * so checking it's liveness by get_swap_device_info is enough.\n-\t */\n-\tsi = this_cpu_read(percpu_swap_cluster.si[order]);\n-\toffset = this_cpu_read(percpu_swap_cluster.offset[order]);\n-\tif (!si || !offset || !get_swap_device_info(si))\n-\t\treturn false;\n-\n-\tci = swap_cluster_lock(si, offset);\n-\tif (cluster_is_usable(ci, order)) {\n-\t\tif (cluster_is_empty(ci))\n-\t\t\toffset = cluster_offset(si, ci);\n-\t\talloc_swap_scan_cluster(si, ci, folio, offset);\n-\t} else {\n-\t\tswap_cluster_unlock(ci);\n-\t}\n+\tint nr_iter, ret;\n+\n+\tpercpu_down_read(\u0026swapon_rwsem);\n+\tfor (nr_iter = 0;; nr_iter++) {\n+\t\tsi = swap_queue_get_device(nr_pages, nr_iter);\n+\t\tif (IS_ERR(si)) {\n+\t\t\tret = PTR_ERR(si);\n+\t\t\tif (ret == -EBUSY)\n+\t\t\t\tcontinue;\n+\t\t\tbreak;\n+\t\t}\n+\t\tcluster_alloc_swap_entry(si, folio);\n \n-\tput_swap_device(si);\n-\treturn folio_test_swapcache(folio);\n-}\n-\n-/* Rotate the device and switch to a new cluster */\n-static void swap_alloc_slow(struct folio *folio)\n-{\n-\tstruct swap_info_struct *si, *next;\n-\n-\tspin_lock(\u0026swap_avail_lock);\n-start_over:\n-\tplist_for_each_entry_safe(si, next, \u0026swap_avail_head, avail_list) {\n-\t\t/* Rotate the device and switch to a new cluster */\n-\t\tplist_requeue(\u0026si-\u003eavail_list, \u0026swap_avail_head);\n-\t\tspin_unlock(\u0026swap_avail_lock);\n-\t\tif (get_swap_device_info(si)) {\n-\t\t\tcluster_alloc_swap_entry(si, folio);\n-\t\t\tput_swap_device(si);\n-\t\t\tif (folio_test_swapcache(folio))\n-\t\t\t\treturn;\n-\t\t\tif (folio_test_large(folio))\n-\t\t\t\treturn;\n+\t\tif (folio_test_swapcache(folio)) {\n+\t\t\tret = 0;\n+\t\t\tbreak;\n \t\t}\n \n-\t\tspin_lock(\u0026swap_avail_lock);\n \t\t/*\n-\t\t * if we got here, it's likely that si was almost full before,\n-\t\t * multiple callers probably all tried to get a page from the\n-\t\t * same si and it filled up before we could get one; or, the si\n-\t\t * filled up between us dropping swap_avail_lock.\n-\t\t * Since we dropped the swap_avail_lock, the swap_avail_list\n-\t\t * may have been modified; so if next is still in the\n-\t\t * swap_avail_head list then try it, otherwise start over if we\n-\t\t * have not gotten any slots.\n+\t\t * For large allocation, return error directly to inform the\n+\t\t * caller to split it instead of fallback to other devices.\n \t\t */\n-\t\tif (plist_node_empty(\u0026next-\u003eavail_list))\n-\t\t\tgoto start_over;\n-\t}\n-\tspin_unlock(\u0026swap_avail_lock);\n-}\n-\n-/*\n- * Discard pending clusters in a synchronized way when under high pressure.\n- * Return: true if any cluster is discarded.\n- */\n-static bool swap_sync_discard(void)\n-{\n-\tbool ret = false;\n-\tstruct swap_info_struct *si, *next;\n-\n-\tspin_lock(\u0026swap_lock);\n-start_over:\n-\tplist_for_each_entry_safe(si, next, \u0026swap_active_head, list) {\n-\t\tspin_unlock(\u0026swap_lock);\n-\t\tif (get_swap_device_info(si)) {\n-\t\t\tif (si-\u003eflags \u0026 SWP_PAGE_DISCARD)\n-\t\t\t\tret = swap_do_scheduled_discard(si);\n-\t\t\tput_swap_device(si);\n+\t\tif (folio_test_large(folio)) {\n+\t\t\tret = -E2BIG;\n+\t\t\tbreak;\n \t\t}\n-\t\tif (ret)\n-\t\t\treturn true;\n-\n-\t\tspin_lock(\u0026swap_lock);\n-\t\tif (plist_node_empty(\u0026next-\u003elist))\n-\t\t\tgoto start_over;\n \t}\n-\tspin_unlock(\u0026swap_lock);\n \n-\treturn false;\n+\tpercpu_up_read(\u0026swapon_rwsem);\n+\treturn ret;\n }\n \n static int swap_extend_table_alloc(struct swap_info_struct *si,\n@@ -1735,6 +1999,7 @@ int folio_alloc_swap(struct folio *folio)\n {\n \tunsigned int order = folio_order(folio);\n \tunsigned int size = 1 \u003c\u003c order;\n+\tint ret;\n \n \tVM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);\n \tVM_BUG_ON_FOLIO(!folio_test_uptodate(folio), folio);\n@@ -1757,23 +2022,14 @@ int folio_alloc_swap(struct folio *folio)\n \t\t}\n \t}\n \n-again:\n-\tlocal_lock(\u0026percpu_swap_cluster.lock);\n-\tif (!swap_alloc_fast(folio))\n-\t\tswap_alloc_slow(folio);\n-\tlocal_unlock(\u0026percpu_swap_cluster.lock);\n-\n-\tif (!order \u0026\u0026 unlikely(!folio_test_swapcache(folio))) {\n-\t\tif (swap_sync_discard())\n-\t\t\tgoto again;\n-\t}\n+\tret = swap_alloc_entry(folio);\n \n \t/* Need to call this even if allocation failed, for MEMCG_SWAP_FAIL. */\n \tif (unlikely(mem_cgroup_try_charge_swap(folio)))\n \t\tswap_cache_del_folio(folio);\n \n \tif (unlikely(!folio_test_swapcache(folio)))\n-\t\treturn -ENOMEM;\n+\t\treturn ret ? ret : -ENOMEM;\n \n \treturn 0;\n }\n@@ -1948,7 +2204,7 @@ void __swap_cluster_free_entries(struct swap_info_struct *si,\n \tif (batch_id)\n \t\tmem_cgroup_uncharge_swap(batch_id, ci_off - batch_off);\n \n-\tswap_range_free(si, ci_head + ci_start, nr_pages);\n+\tswap_device_inuse_sub(si, ci_head + ci_start, nr_pages);\n \tswap_cluster_assert_empty(ci, ci_start, nr_pages, false);\n \n \tif (!ci-\u003ecount)\n@@ -2150,31 +2406,14 @@ void swap_put_entries_direct(swp_entry_t entry, int nr)\n  */\n swp_entry_t swap_alloc_hibernation_slot(int type)\n {\n-\tstruct swap_info_struct *pcp_si, *si = swap_type_to_info(type);\n-\tunsigned long pcp_offset, offset = SWAP_ENTRY_INVALID;\n-\tstruct swap_cluster_info *ci;\n+\tstruct swap_info_struct *si = swap_type_to_info(type);\n+\tunsigned long offset = SWAP_ENTRY_INVALID;\n \tswp_entry_t entry = {0};\n \n \tif (!si)\n \t\tgoto fail;\n \n-\t/*\n-\t * Try the local cluster first if it matches the device. If\n-\t * not, try grab a new cluster and override local cluster.\n-\t */\n-\tlocal_lock(\u0026percpu_swap_cluster.lock);\n-\tpcp_si = this_cpu_read(percpu_swap_cluster.si[0]);\n-\tpcp_offset = this_cpu_read(percpu_swap_cluster.offset[0]);\n-\tif (pcp_si == si \u0026\u0026 pcp_offset) {\n-\t\tci = swap_cluster_lock(si, pcp_offset);\n-\t\tif (cluster_is_usable(ci, 0))\n-\t\t\toffset = alloc_swap_scan_cluster(si, ci, NULL, pcp_offset);\n-\t\telse\n-\t\t\tswap_cluster_unlock(ci);\n-\t}\n-\tif (!offset)\n-\t\toffset = cluster_alloc_swap_entry(si, NULL);\n-\tlocal_unlock(\u0026percpu_swap_cluster.lock);\n+\toffset = cluster_alloc_swap_entry(si, NULL);\n \tif (offset)\n \t\tentry = swp_entry(si-\u003etype, offset);\n \n@@ -2205,26 +2444,18 @@ void swap_free_hibernation_slot(swp_entry_t entry)\n \n static int __find_hibernation_swap_type(dev_t device, sector_t offset)\n {\n-\tint type;\n-\n-\tlockdep_assert_held(\u0026swap_lock);\n+\tstruct swap_info_struct *si;\n \n \tif (!device)\n \t\treturn -EINVAL;\n \n-\tfor (type = 0; type \u003c nr_swapfiles; type++) {\n-\t\tstruct swap_info_struct *sis = swap_info[type];\n-\n-\t\tif (!(sis-\u003eflags \u0026 SWP_WRITEOK))\n-\t\t\tcontinue;\n-\n-\t\tif (device == sis-\u003ebdev-\u003ebd_dev) {\n-\t\t\tstruct swap_extent *se = first_se(sis);\n-\n-\t\t\tif (se-\u003estart_block == offset)\n-\t\t\t\treturn type;\n+\tfor_each_avail_swap(si) {\n+\t\tif (device == si-\u003ebdev-\u003ebd_dev) {\n+\t\t\tif (first_se(si)-\u003estart_block == offset)\n+\t\t\t\treturn si-\u003etype;\n \t\t}\n \t}\n+\n \treturn -ENODEV;\n }\n \n@@ -2247,21 +2478,18 @@ static int __find_hibernation_swap_type(dev_t device, sector_t offset)\n  */\n int pin_hibernation_swap_type(dev_t device, sector_t offset)\n {\n-\tint type;\n+\tint ret;\n \tstruct swap_info_struct *si;\n \n-\tspin_lock(\u0026swap_lock);\n-\n-\ttype = __find_hibernation_swap_type(device, offset);\n-\tif (type \u003c 0) {\n-\t\tspin_unlock(\u0026swap_lock);\n-\t\treturn type;\n-\t}\n+\tpercpu_down_write(\u0026swapon_rwsem);\n+\tret = __find_hibernation_swap_type(device, offset);\n+\tif (ret \u003c 0)\n+\t\tgoto out;\n \n-\tsi = swap_type_to_info(type);\n+\tsi = swap_type_to_info(ret);\n \tif (WARN_ON_ONCE(!si)) {\n-\t\tspin_unlock(\u0026swap_lock);\n-\t\treturn -ENODEV;\n+\t\tret = -ENODEV;\n+\t\tgoto out;\n \t}\n \n \t/*\n@@ -2270,14 +2498,15 @@ int pin_hibernation_swap_type(dev_t device, sector_t offset)\n \t * the same session.\n \t */\n \tif (WARN_ON_ONCE(si-\u003eflags \u0026 SWP_HIBERNATION)) {\n-\t\tspin_unlock(\u0026swap_lock);\n-\t\treturn -EBUSY;\n+\t\tret = -EBUSY;\n+\t\tgoto out;\n \t}\n \n \tsi-\u003eflags |= SWP_HIBERNATION;\n \n-\tspin_unlock(\u0026swap_lock);\n-\treturn type;\n+out:\n+\tpercpu_up_write(\u0026swapon_rwsem);\n+\treturn ret;\n }\n \n /**\n@@ -2294,14 +2523,11 @@ void unpin_hibernation_swap_type(int type)\n {\n \tstruct swap_info_struct *si;\n \n-\tspin_lock(\u0026swap_lock);\n+\tpercpu_down_write(\u0026swapon_rwsem);\n \tsi = swap_type_to_info(type);\n-\tif (!si) {\n-\t\tspin_unlock(\u0026swap_lock);\n-\t\treturn;\n-\t}\n-\tsi-\u003eflags \u0026= ~SWP_HIBERNATION;\n-\tspin_unlock(\u0026swap_lock);\n+\tif (si)\n+\t\tsi-\u003eflags \u0026= ~SWP_HIBERNATION;\n+\tpercpu_up_write(\u0026swapon_rwsem);\n }\n \n /**\n@@ -2326,29 +2552,26 @@ int find_hibernation_swap_type(dev_t device, sector_t offset)\n {\n \tint type;\n \n-\tspin_lock(\u0026swap_lock);\n+\tpercpu_down_read(\u0026swapon_rwsem);\n \ttype = __find_hibernation_swap_type(device, offset);\n-\tspin_unlock(\u0026swap_lock);\n+\tpercpu_up_read(\u0026swapon_rwsem);\n \n \treturn type;\n }\n \n int find_first_swap(dev_t *device)\n {\n-\tint type;\n-\n-\tspin_lock(\u0026swap_lock);\n-\tfor (type = 0; type \u003c nr_swapfiles; type++) {\n-\t\tstruct swap_info_struct *sis = swap_info[type];\n+\tint ret = -ENODEV;\n+\tstruct swap_info_struct *si;\n \n-\t\tif (!(sis-\u003eflags \u0026 SWP_WRITEOK))\n-\t\t\tcontinue;\n-\t\t*device = sis-\u003ebdev-\u003ebd_dev;\n-\t\tspin_unlock(\u0026swap_lock);\n-\t\treturn type;\n+\tpercpu_down_read(\u0026swapon_rwsem);\n+\tfor_each_avail_swap(si) {\n+\t\t*device = si-\u003ebdev-\u003ebd_dev;\n+\t\tret = si-\u003etype;\n+\t\tbreak;\n \t}\n-\tspin_unlock(\u0026swap_lock);\n-\treturn -ENODEV;\n+\tpercpu_up_read(\u0026swapon_rwsem);\n+\treturn ret;\n }\n \n /*\n@@ -2376,7 +2599,7 @@ unsigned int count_swap_pages(int type, int free)\n {\n \tunsigned int n = 0;\n \n-\tspin_lock(\u0026swap_lock);\n+\tpercpu_down_read(\u0026swapon_rwsem);\n \tif ((unsigned int)type \u003c nr_swapfiles) {\n \t\tstruct swap_info_struct *sis = swap_info[type];\n \n@@ -2388,7 +2611,7 @@ unsigned int count_swap_pages(int type, int free)\n \t\t}\n \t\tspin_unlock(\u0026sis-\u003elock);\n \t}\n-\tspin_unlock(\u0026swap_lock);\n+\tpercpu_up_read(\u0026swapon_rwsem);\n \treturn n;\n }\n #endif /* CONFIG_HIBERNATION */\n@@ -2700,10 +2923,10 @@ static unsigned int find_next_to_unuse(struct swap_info_struct *si,\n \tunsigned long swp_tb;\n \n \t/*\n-\t * No need for swap_lock here: we're just looking\n+\t * No need for swapon_rwsem here: we're just looking\n \t * for whether an entry is in use, not modifying it; false\n \t * hits are okay, and sys_swapoff() has already prevented new\n-\t * allocations from this area (while holding swap_lock).\n+\t * allocations from this area (while holding swapon_rwsem).\n \t */\n \tfor (i = prev + 1; i \u003c si-\u003emax; i++) {\n \t\tswp_tb = swap_table_get(__swap_offset_to_cluster(si, i),\n@@ -2816,7 +3039,7 @@ static int try_to_unuse(unsigned int type)\n success:\n \t/*\n \t * Make sure that further cleanups after try_to_unuse() returns happen\n-\t * after swap_range_free() reduces si-\u003einuse_pages to 0.\n+\t * after swap_device_inuse_sub() reduces si-\u003einuse_pages to 0.\n \t */\n \tsmp_mb();\n \treturn 0;\n@@ -2824,18 +3047,20 @@ static int try_to_unuse(unsigned int type)\n \n /*\n  * After a successful try_to_unuse, if no swap is now in use, we know\n- * we can empty the mmlist.  swap_lock must be held on entry and exit.\n- * Note that mmlist_lock nests inside swap_lock, and an mm must be\n+ * we can empty the mmlist. swapon_rwsem must be held on entry and exit.\n+ * Note that mmlist_lock nests inside swapon_rwsem, and an mm must be\n  * added to the mmlist just after page_duplicate - before would be racy.\n  */\n static void drain_mmlist(void)\n {\n+\tstruct swap_info_struct *si;\n \tstruct list_head *p, *next;\n-\tunsigned int type;\n \n-\tfor (type = 0; type \u003c nr_swapfiles; type++)\n-\t\tif (swap_usage_in_pages(swap_info[type]))\n+\tfor_each_swap(si) {\n+\t\tif (swap_usage_in_pages(si))\n \t\t\treturn;\n+\t}\n+\n \tspin_lock(\u0026mmlist_lock);\n \tlist_for_each_safe(p, next, \u0026init_mm.mmlist)\n \t\tlist_del_init(p);\n@@ -2968,58 +3193,65 @@ static int setup_swap_extents(struct swap_info_struct *sis,\n \treturn generic_swapfile_activate(sis, swap_file, span);\n }\n \n-static void _enable_swap_info(struct swap_info_struct *si)\n-{\n-\tatomic_long_add(si-\u003epages, \u0026nr_swap_pages);\n-\ttotal_swap_pages += si-\u003epages;\n-\n-\tassert_spin_locked(\u0026swap_lock);\n-\n-\tplist_add(\u0026si-\u003elist, \u0026swap_active_head);\n-\n-\t/* Add back to available list */\n-\tadd_to_avail_list(si, true);\n-}\n-\n /*\n- * Called after the swap device is ready, resurrect its percpu ref, it's now\n- * safe to reference it. Add it to the list to expose it to the allocator.\n+ * Called after the swap device is ready to be used. Marking it writable and\n+ * exposing it to the allocator. Resurrect its percpu ref if it was dead before.\n  */\n-static void enable_swap_info(struct swap_info_struct *si)\n+static void swap_device_enable(struct swap_info_struct *si)\n {\n-\tpercpu_ref_resurrect(\u0026si-\u003eusers);\n-\tspin_lock(\u0026swap_lock);\n-\tspin_lock(\u0026si-\u003elock);\n-\t_enable_swap_info(si);\n-\tspin_unlock(\u0026si-\u003elock);\n-\tspin_unlock(\u0026swap_lock);\n-}\n+\tpercpu_down_write(\u0026swapon_rwsem);\n+\tspin_lock(\u0026swap_queue_update_lock);\n+\tsi-\u003eflags |= SWP_WRITEOK;\n+\tspin_unlock(\u0026swap_queue_update_lock);\n+\tatomic_long_add(si-\u003epages, \u0026nr_swap_pages);\n+\ttotal_swap_pages += si-\u003epages;\n+\tpercpu_up_write(\u0026swapon_rwsem);\n \n-static void reinsert_swap_info(struct swap_info_struct *si)\n-{\n-\tspin_lock(\u0026swap_lock);\n-\tspin_lock(\u0026si-\u003elock);\n-\t_enable_swap_info(si);\n-\tspin_unlock(\u0026si-\u003elock);\n-\tspin_unlock(\u0026swap_lock);\n+\tadd_to_avail_list(si);\n }\n \n-/*\n- * Called after clearing SWP_WRITEOK, ensures cluster_alloc_range\n- * see the updated flags, so there will be no more allocations.\n- */\n-static void wait_for_allocation(struct swap_info_struct *si)\n+static int swap_device_disable(struct swap_info_struct *si)\n {\n \tunsigned long offset;\n \tunsigned long end = ALIGN(si-\u003emax, SWAPFILE_CLUSTER);\n \tstruct swap_cluster_info *ci;\n \n-\tBUG_ON(si-\u003eflags \u0026 SWP_WRITEOK);\n+\t/*\n+\t * If SWP_WRITEOK is not set: another process already disabling it.\n+\t * If SWP_HIBERNATION is set: the device is pinned for hibernation.\n+\t */\n+\tpercpu_down_write(\u0026swapon_rwsem);\n+\tif (!(si-\u003eflags \u0026 SWP_WRITEOK) ||\n+\t    si-\u003eflags \u0026 SWP_HIBERNATION) {\n+\t\tpercpu_up_write(\u0026swapon_rwsem);\n+\t\treturn -EBUSY;\n+\t}\n+\n+\tif (security_vm_enough_memory_mm(current-\u003emm, si-\u003epages)) {\n+\t\tpercpu_up_write(\u0026swapon_rwsem);\n+\t\treturn -ENOMEM;\n+\t}\n+\tvm_unacct_memory(si-\u003epages);\n+\n+\tspin_lock(\u0026swap_queue_update_lock);\n+\tsi-\u003eflags \u0026= ~SWP_WRITEOK;\n+\tspin_unlock(\u0026swap_queue_update_lock);\n+\ttotal_swap_pages -= si-\u003epages;\n+\tatomic_long_sub(si-\u003epages, \u0026nr_swap_pages);\n+\tdel_from_avail_list(si, true);\n+\tpercpu_up_write(\u0026swapon_rwsem);\n \n+\t/*\n+\t * Swap allocator doesn't touch si lock, so looping through all\n+\t * ci locks ensures __swap_cluster_alloc_entries sees the\n+\t * updated flags, and no more allocations will occur.\n+\t */\n \tfor (offset = 0; offset \u003c end; offset += SWAPFILE_CLUSTER) {\n \t\tci = swap_cluster_lock(si, offset);\n \t\tswap_cluster_unlock(ci);\n \t}\n+\n+\treturn 0;\n }\n \n static void free_swap_cluster_info(struct swap_cluster_info *cluster_info,\n@@ -3043,28 +3275,6 @@ static void free_swap_cluster_info(struct swap_cluster_info *cluster_info,\n \tkvfree(cluster_info);\n }\n \n-/*\n- * Called after swap device's reference count is dead, so\n- * neither scan nor allocation will use it.\n- */\n-static void flush_percpu_swap_cluster(struct swap_info_struct *si)\n-{\n-\tint cpu, i;\n-\tstruct swap_info_struct **pcp_si;\n-\n-\tfor_each_possible_cpu(cpu) {\n-\t\tpcp_si = per_cpu_ptr(percpu_swap_cluster.si, cpu);\n-\t\t/*\n-\t\t * Invalidate the percpu swap cluster cache, si-\u003eusers\n-\t\t * is dead, so no new user will point to it, just flush\n-\t\t * any existing user.\n-\t\t */\n-\t\tfor (i = 0; i \u003c SWAP_NR_ORDERS; i++)\n-\t\t\tcmpxchg(\u0026pcp_si[i], si, NULL);\n-\t}\n-}\n-\n-\n SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)\n {\n \tstruct swap_info_struct *p = NULL;\n@@ -3086,44 +3296,22 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)\n \t\treturn PTR_ERR(victim);\n \n \tmapping = victim-\u003ef_mapping;\n-\tspin_lock(\u0026swap_lock);\n-\tplist_for_each_entry(p, \u0026swap_active_head, list) {\n-\t\tif (p-\u003eflags \u0026 SWP_WRITEOK) {\n-\t\t\tif (p-\u003eswap_file-\u003ef_mapping == mapping) {\n-\t\t\t\tfound = 1;\n-\t\t\t\tbreak;\n-\t\t\t}\n+\tpercpu_down_read(\u0026swapon_rwsem);\n+\tfor_each_avail_swap(p) {\n+\t\tif (p-\u003eswap_file-\u003ef_mapping == mapping) {\n+\t\t\tfound = 1;\n+\t\t\tbreak;\n \t\t}\n \t}\n-\tif (!found) {\n-\t\terr = -EINVAL;\n-\t\tspin_unlock(\u0026swap_lock);\n-\t\tgoto out_dput;\n-\t}\n-\n-\t/* Refuse swapoff while the device is pinned for hibernation */\n-\tif (p-\u003eflags \u0026 SWP_HIBERNATION) {\n-\t\terr = -EBUSY;\n-\t\tspin_unlock(\u0026swap_lock);\n-\t\tgoto out_dput;\n-\t}\n+\tpercpu_up_read(\u0026swapon_rwsem);\n+\tfilp_close(victim, NULL);\n \n-\tif (!security_vm_enough_memory_mm(current-\u003emm, p-\u003epages))\n-\t\tvm_unacct_memory(p-\u003epages);\n-\telse {\n-\t\terr = -ENOMEM;\n-\t\tspin_unlock(\u0026swap_lock);\n-\t\tgoto out_dput;\n-\t}\n-\tspin_lock(\u0026p-\u003elock);\n-\tdel_from_avail_list(p, true);\n-\tplist_del(\u0026p-\u003elist, \u0026swap_active_head);\n-\tatomic_long_sub(p-\u003epages, \u0026nr_swap_pages);\n-\ttotal_swap_pages -= p-\u003epages;\n-\tspin_unlock(\u0026p-\u003elock);\n-\tspin_unlock(\u0026swap_lock);\n+\tif (!found)\n+\t\treturn -EINVAL;\n \n-\twait_for_allocation(p);\n+\terr = swap_device_disable(p);\n+\tif (err)\n+\t\treturn err;\n \n \tset_current_oom_origin();\n \terr = try_to_unuse(p-\u003etype);\n@@ -3131,10 +3319,13 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)\n \n \tif (err) {\n \t\t/* re-insert swap space back into swap_list */\n-\t\treinsert_swap_info(p);\n-\t\tgoto out_dput;\n+\t\tswap_device_enable(p);\n+\t\treturn err;\n \t}\n \n+\tpercpu_down_write(\u0026swapon_rwsem);\n+\tswap_queue_del(p);\n+\n \t/*\n \t * Wait for swap operations protected by get/put_swap_device()\n \t * to complete.  Because of synchronize_rcu() here, all swap\n@@ -3149,15 +3340,12 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)\n \n \tflush_work(\u0026p-\u003ediscard_work);\n \tflush_work(\u0026p-\u003ereclaim_work);\n-\tflush_percpu_swap_cluster(p);\n \n \tdestroy_swap_extents(p, p-\u003eswap_file);\n \n \tif (!(p-\u003eflags \u0026 SWP_SOLIDSTATE))\n \t\tatomic_dec(\u0026nr_rotate_swap);\n \n-\tmutex_lock(\u0026swapon_mutex);\n-\tspin_lock(\u0026swap_lock);\n \tspin_lock(\u0026p-\u003elock);\n \tdrain_mmlist();\n \n@@ -3168,10 +3356,11 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)\n \tp-\u003emax = 0;\n \tp-\u003ecluster_info = NULL;\n \tspin_unlock(\u0026p-\u003elock);\n-\tspin_unlock(\u0026swap_lock);\n+\tpercpu_up_write(\u0026swapon_rwsem);\n \tarch_swap_invalidate_area(p-\u003etype);\n \tzswap_swapoff(p-\u003etype);\n-\tmutex_unlock(\u0026swapon_mutex);\n+\tfree_percpu(p-\u003epercpu_cluster);\n+\tp-\u003epercpu_cluster = NULL;\n \tkfree(p-\u003eglobal_cluster);\n \tp-\u003eglobal_cluster = NULL;\n \tfree_swap_cluster_info(cluster_info, maxpages);\n@@ -3187,18 +3376,19 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)\n \t * Clear the SWP_USED flag after all resources are freed so that swapon\n \t * can reuse this swap_info in alloc_swap_info() safely.  It is ok to\n \t * not hold p-\u003elock after we cleared its SWP_WRITEOK.\n+\t *\n+\t * The write lock ensures the flag clear is visible to lockless\n+\t * readers of swap_type_to_info() before alloc_swap_info() reuses\n+\t * this slot.\n \t */\n-\tspin_lock(\u0026swap_lock);\n+\tpercpu_down_write(\u0026swapon_rwsem);\n \tp-\u003eflags = 0;\n-\tspin_unlock(\u0026swap_lock);\n+\tpercpu_up_write(\u0026swapon_rwsem);\n \n-\terr = 0;\n \tatomic_inc(\u0026proc_poll_event);\n \twake_up_interruptible(\u0026proc_poll_wait);\n \n-out_dput:\n-\tfilp_close(victim, NULL);\n-\treturn err;\n+\treturn 0;\n }\n \n #ifdef CONFIG_PROC_FS\n@@ -3216,20 +3406,18 @@ static __poll_t swaps_poll(struct file *file, poll_table *wait)\n \treturn EPOLLIN | EPOLLRDNORM;\n }\n \n-/* iterator */\n static void *swap_start(struct seq_file *swap, loff_t *pos)\n {\n \tstruct swap_info_struct *si;\n-\tint type;\n \tloff_t l = *pos;\n \n-\tmutex_lock(\u0026swapon_mutex);\n+\tpercpu_down_read(\u0026swapon_rwsem);\n \n \tif (!l)\n \t\treturn SEQ_START_TOKEN;\n \n-\tfor (type = 0; (si = swap_type_to_info(type)); type++) {\n-\t\tif (!(si-\u003eswap_file))\n+\tfor_each_swap(si) {\n+\t\tif (!si-\u003eswap_file)\n \t\t\tcontinue;\n \t\tif (!--l)\n \t\t\treturn si;\n@@ -3250,7 +3438,7 @@ static void *swap_next(struct seq_file *swap, void *v, loff_t *pos)\n \n \t++(*pos);\n \tfor (; (si = swap_type_to_info(type)); type++) {\n-\t\tif (!(si-\u003eswap_file))\n+\t\tif (!(si-\u003eflags \u0026 SWP_USED) || !(si-\u003eswap_file))\n \t\t\tcontinue;\n \t\treturn si;\n \t}\n@@ -3260,7 +3448,7 @@ static void *swap_next(struct seq_file *swap, void *v, loff_t *pos)\n \n static void swap_stop(struct seq_file *swap, void *v)\n {\n-\tmutex_unlock(\u0026swapon_mutex);\n+\tpercpu_up_read(\u0026swapon_rwsem);\n }\n \n static int swap_show(struct seq_file *swap, void *v)\n@@ -3353,13 +3541,13 @@ static struct swap_info_struct *alloc_swap_info(void)\n \t\treturn ERR_PTR(-ENOMEM);\n \t}\n \n-\tspin_lock(\u0026swap_lock);\n+\tpercpu_down_write(\u0026swapon_rwsem);\n \tfor (type = 0; type \u003c nr_swapfiles; type++) {\n \t\tif (!(swap_info[type]-\u003eflags \u0026 SWP_USED))\n \t\t\tbreak;\n \t}\n \tif (type \u003e= MAX_SWAPFILES) {\n-\t\tspin_unlock(\u0026swap_lock);\n+\t\tpercpu_up_write(\u0026swapon_rwsem);\n \t\tpercpu_ref_exit(\u0026p-\u003eusers);\n \t\tkvfree(p);\n \t\treturn ERR_PTR(-EPERM);\n@@ -3381,10 +3569,8 @@ static struct swap_info_struct *alloc_swap_info(void)\n \t\t */\n \t}\n \tp-\u003eswap_extent_root = RB_ROOT;\n-\tplist_node_init(\u0026p-\u003elist, 0);\n-\tplist_node_init(\u0026p-\u003eavail_list, 0);\n \tp-\u003eflags = SWP_USED;\n-\tspin_unlock(\u0026swap_lock);\n+\tpercpu_up_write(\u0026swapon_rwsem);\n \tif (defer) {\n \t\tpercpu_ref_exit(\u0026defer-\u003eusers);\n \t\tkvfree(defer);\n@@ -3453,9 +3639,8 @@ __weak unsigned long arch_max_swapfile_size(void)\n \treturn generic_max_swapfile_size();\n }\n \n-static unsigned long read_swap_header(struct swap_info_struct *si,\n-\t\t\t\t\tunion swap_header *swap_header,\n-\t\t\t\t\tstruct inode *inode)\n+static unsigned long read_swap_header(union swap_header *swap_header,\n+\t\t\t\t      struct inode *inode)\n {\n \tint i;\n \tunsigned long maxpages;\n@@ -3522,7 +3707,7 @@ static int setup_swap_clusters_info(struct swap_info_struct *si,\n {\n \tunsigned long nr_clusters = DIV_ROUND_UP(maxpages, SWAPFILE_CLUSTER);\n \tstruct swap_cluster_info *cluster_info;\n-\tint err = -ENOMEM;\n+\tint cpu, err = -ENOMEM;\n \tunsigned long i;\n \n \tcluster_info = kvzalloc_objs(*cluster_info, nr_clusters);\n@@ -3532,13 +3717,26 @@ static int setup_swap_clusters_info(struct swap_info_struct *si,\n \tfor (i = 0; i \u003c nr_clusters; i++)\n \t\tspin_lock_init(\u0026cluster_info[i].lock);\n \n-\tif (!(si-\u003eflags \u0026 SWP_SOLIDSTATE)) {\n+\tif (si-\u003eflags \u0026 SWP_SOLIDSTATE) {\n+\t\tsi-\u003epercpu_cluster = alloc_percpu(struct percpu_cluster);\n+\t\tif (!si-\u003epercpu_cluster)\n+\t\t\tgoto err;\n+\n+\t\tfor_each_possible_cpu(cpu) {\n+\t\t\tstruct percpu_cluster *cluster;\n+\n+\t\t\tcluster = per_cpu_ptr(si-\u003epercpu_cluster, cpu);\n+\t\t\tfor (i = 0; i \u003c SWAP_NR_ORDERS; i++)\n+\t\t\t\tcluster-\u003enext[i] = SWAP_ENTRY_INVALID;\n+\t\t\tlocal_lock_init(\u0026cluster-\u003elock);\n+\t\t}\n+\t} else {\n \t\tsi-\u003eglobal_cluster = kmalloc_obj(*si-\u003eglobal_cluster);\n \t\tif (!si-\u003eglobal_cluster)\n \t\t\tgoto err;\n \t\tfor (i = 0; i \u003c SWAP_NR_ORDERS; i++)\n \t\t\tsi-\u003eglobal_cluster-\u003enext[i] = SWAP_ENTRY_INVALID;\n-\t\tspin_lock_init(\u0026si-\u003eglobal_cluster_lock);\n+\t\tspin_lock_init(\u0026si-\u003eglobal_cluster-\u003elock);\n \t}\n \n \t/*\n@@ -3621,7 +3819,7 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)\n \t/*\n \t * Allocate or reuse existing !SWP_USED swap_info. The returned\n \t * si will stay in a dying status, so nothing will access its content\n-\t * until enable_swap_info resurrects its percpu ref and expose it.\n+\t * until swap_device_enable resurrects its percpu ref and expose it.\n \t */\n \tsi = alloc_swap_info();\n \tif (IS_ERR(si))\n@@ -3679,7 +3877,7 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)\n \t}\n \tswap_header = kmap_local_folio(folio, 0);\n \n-\tmaxpages = read_swap_header(si, swap_header, inode);\n+\tmaxpages = read_swap_header(swap_header, inode);\n \tif (unlikely(!maxpages)) {\n \t\terror = -EINVAL;\n \t\tgoto bad_swap_unlock_inode;\n@@ -3700,11 +3898,6 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)\n \n \tmaxpages = si-\u003emax;\n \n-\t/* Set up the swap cluster info */\n-\terror = setup_swap_clusters_info(si, swap_header, maxpages);\n-\tif (error)\n-\t\tgoto bad_swap_unlock_inode;\n-\n \tif (si-\u003ebdev \u0026\u0026 bdev_stable_writes(si-\u003ebdev))\n \t\tsi-\u003eflags |= SWP_STABLE_WRITES;\n \n@@ -3718,6 +3911,15 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)\n \t\tinced_nr_rotate_swap = true;\n \t}\n \n+\t/*\n+\t * Set up the swap cluster info. This must run after SWP_SOLIDSTATE\n+\t * is determined above, as it decides whether to allocate the per-CPU\n+\t * cluster (solid state) or the global cluster (rotational).\n+\t */\n+\terror = setup_swap_clusters_info(si, swap_header, maxpages);\n+\tif (error)\n+\t\tgoto bad_swap_unlock_inode;\n+\n \tif ((swap_flags \u0026 SWAP_FLAG_DISCARD) \u0026\u0026\n \t    si-\u003ebdev \u0026\u0026 bdev_max_discard_sectors(si-\u003ebdev)) {\n \t\t/*\n@@ -3764,22 +3966,22 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)\n \t\tgoto free_swap_zswap;\n \t}\n \n-\tmutex_lock(\u0026swapon_mutex);\n \tprio = DEF_SWAP_PRIO;\n \tif (swap_flags \u0026 SWAP_FLAG_PREFER)\n \t\tprio = swap_flags \u0026 SWAP_FLAG_PRIO_MASK;\n \n-\t/*\n-\t * The plist prio is negated because plist ordering is\n-\t * low-to-high, while swap ordering is high-to-low\n-\t */\n \tsi-\u003eprio = prio;\n-\tsi-\u003elist.prio = -si-\u003eprio;\n-\tsi-\u003eavail_list.prio = -si-\u003eprio;\n \tsi-\u003eswap_file = swap_file;\n \n \t/* Sets SWP_WRITEOK, resurrect the percpu ref, expose the swap device */\n-\tenable_swap_info(si);\n+\tpercpu_ref_resurrect(\u0026si-\u003eusers);\n+\tpercpu_down_write(\u0026swapon_rwsem);\n+\terror = swap_queue_add(si);\n+\tpercpu_up_write(\u0026swapon_rwsem);\n+\tif (error)\n+\t\tgoto free_swap_zswap;\n+\n+\tswap_device_enable(si);\n \n \tpr_info(\"Adding %uk swap on %s.  Priority:%d extents:%d across:%lluk %s%s%s%s\\n\",\n \t\tK(si-\u003epages), name-\u003ename, si-\u003eprio, nr_extents,\n@@ -3789,7 +3991,6 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)\n \t\t(si-\u003eflags \u0026 SWP_AREA_DISCARD) ? \"s\" : \"\",\n \t\t(si-\u003eflags \u0026 SWP_PAGE_DISCARD) ? \"c\" : \"\");\n \n-\tmutex_unlock(\u0026swapon_mutex);\n \tatomic_inc(\u0026proc_poll_event);\n \twake_up_interruptible(\u0026proc_poll_wait);\n \n@@ -3800,6 +4001,8 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)\n bad_swap_unlock_inode:\n \tinode_unlock(inode);\n bad_swap:\n+\tfree_percpu(si-\u003epercpu_cluster);\n+\tsi-\u003epercpu_cluster = NULL;\n \tkfree(si-\u003eglobal_cluster);\n \tsi-\u003eglobal_cluster = NULL;\n \tinode = NULL;\n@@ -3810,9 +4013,9 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)\n \t * Clear the SWP_USED flag after all resources are freed so\n \t * alloc_swap_info can reuse this si safely.\n \t */\n-\tspin_lock(\u0026swap_lock);\n+\tpercpu_down_write(\u0026swapon_rwsem);\n \tsi-\u003eflags = 0;\n-\tspin_unlock(\u0026swap_lock);\n+\tpercpu_up_write(\u0026swapon_rwsem);\n \tif (inced_nr_rotate_swap)\n \t\tatomic_dec(\u0026nr_rotate_swap);\n \tif (swap_file)\n@@ -3827,19 +4030,17 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)\n \n void si_swapinfo(struct sysinfo *val)\n {\n-\tunsigned int type;\n+\tstruct swap_info_struct *si;\n \tunsigned long nr_to_be_unused = 0;\n \n-\tspin_lock(\u0026swap_lock);\n-\tfor (type = 0; type \u003c nr_swapfiles; type++) {\n-\t\tstruct swap_info_struct *si = swap_info[type];\n-\n-\t\tif ((si-\u003eflags \u0026 SWP_USED) \u0026\u0026 !(si-\u003eflags \u0026 SWP_WRITEOK))\n+\tpercpu_down_read(\u0026swapon_rwsem);\n+\tfor_each_swap(si) {\n+\t\tif (!(si-\u003eflags \u0026 SWP_WRITEOK))\n \t\t\tnr_to_be_unused += swap_usage_in_pages(si);\n \t}\n \tval-\u003efreeswap = atomic_long_read(\u0026nr_swap_pages) + nr_to_be_unused;\n \tval-\u003etotalswap = total_swap_pages + nr_to_be_unused;\n-\tspin_unlock(\u0026swap_lock);\n+\tpercpu_up_read(\u0026swapon_rwsem);\n }\n \n /*\n@@ -3879,7 +4080,7 @@ int swap_dup_entry_direct(swp_entry_t entry)\n #if defined(CONFIG_MEMCG) \u0026\u0026 defined(CONFIG_BLK_CGROUP)\n static bool __has_usable_swap(void)\n {\n-\treturn !plist_head_empty(\u0026swap_active_head);\n+\treturn READ_ONCE(total_swap_pages) \u003e 0;\n }\n \n void __folio_throttle_swaprate(struct folio *folio, gfp_t gfp)\n@@ -3902,14 +4103,14 @@ void __folio_throttle_swaprate(struct folio *folio, gfp_t gfp)\n \tif (current-\u003ethrottle_disk)\n \t\treturn;\n \n-\tspin_lock(\u0026swap_avail_lock);\n-\tplist_for_each_entry(si, \u0026swap_avail_head, avail_list) {\n+\tpercpu_down_read(\u0026swapon_rwsem);\n+\tfor_each_avail_swap(si) {\n \t\tif (si-\u003ebdev) {\n \t\t\tblkcg_schedule_throttle(si-\u003ebdev-\u003ebd_disk, true);\n \t\t\tbreak;\n \t\t}\n \t}\n-\tspin_unlock(\u0026swap_avail_lock);\n+\tpercpu_up_read(\u0026swapon_rwsem);\n }\n #endif\n \n"
}

2/1

2026/07/14 00:43

agent

patch-evaluator

Model:

gemini-3.1-pro-preview

Args:

null

Results:

{
  "EnableConfigs": [
    "SWAP"
  ],
  "FocusSymbols": [
    "swap_alloc_entry",
    "swap_queue_add",
    "swap_queue_del",
    "swap_queue_get_device",
    "swap_device_enable",
    "swap_device_disable",
    "swap_device_inuse_add",
    "swap_device_inuse_sub",
    "add_to_avail_list",
    "del_from_avail_list",
    "swap_queue_mask",
    "swap_queue_unmask",
    "cluster_alloc_swap_entry"
  ],
  "Reasoning": "The patch replaces the plist-based swap device queue with a custom round-robin queue, introducing new core logic for swap allocation and device management. This is a functional change to the core kernel logic and should be fuzzed.",
  "WorthFuzzing": true
}

Instruction:

You are an expert Linux kernel maintainer.
Your job is to review a provided patch series and determine
if it makes functional changes to the kernel that should be fuzzed.

IMPORTANT: The changes have ALREADY been applied and committed as the HEAD commit in
your workspace. Do NOT rely on your internal knowledge of the kernel. You must actively
use your code access tools to examine the actual source code and confirm any assumptions.

Return WorthFuzzing=false if the patch only contains:
- Modifications to Documentation/, Kconfig files, or code comments.
- Purely decorative changes, such as logging (e.g., pr_err, printk) or tracepoints.
- Changes to numeric constants or macros that do not functionally alter execution flow.
- Code paths that are impossible to reach in virtualized environments like GCE or QEMU,
even when utilizing software-emulated hardware (e.g., usb gadget, mac80211_hwsim).

If it modifies reachable core kernel logic, drivers, or architectures, use your code search
tools to verify the code can be executed, then return WorthFuzzing=true.

When returning WorthFuzzing=true, you MUST ALSO:
1. Extract any specific kernel functions that should be heavily fuzzed into FocusSymbols.
   Avoid listing generic hot-path functions to prevent skewed test distributions.
2. Identify any specific CONFIG_ options required to properly test this new/modified feature.
   Go and look into the Kconfig files and check for ifdefs around the code, do not make assumptions.
   Do not list too generic configs (we already have them enabled). Only list those that
   specifically cover the modified code. List them in the EnableConfigs output array,
   and DO NOT add a 'CONFIG_' prefix (e.g., return "NET_IPV4" instead of "CONFIG_NET_IPV4").
Prefer calling several tools at the same time to save round-trips.


Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.

Prompt:

For your convenience, here is the diff of the changes:
commit 9f349365e0eb1fe2128a8484bb4ca651f1dbd2a2
Author: syz-cluster <triage@syzkaller.com>
Date:   Tue Jul 14 00:43:44 2026 +0000

    syz-cluster: applied patch under review

diff --git a/include/linux/plist.h b/include/linux/plist.h
index 16cf4355b5c131..73d359c1a0536a 100644
--- a/include/linux/plist.h
+++ b/include/linux/plist.h
@@ -132,8 +132,6 @@ static inline void plist_node_init(struct plist_node *node, int prio)
 extern void plist_add(struct plist_node *node, struct plist_head *head);
 extern void plist_del(struct plist_node *node, struct plist_head *head);
 
-extern void plist_requeue(struct plist_node *node, struct plist_head *head);
-
 /**
  * plist_for_each - iterate over the plist
  * @pos:	the type * to use as a loop counter
diff --git a/include/linux/swap.h b/include/linux/swap.h
index 696ed01709c2ba..04767540972b81 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -194,6 +194,22 @@ struct swap_extent {
 	((offsetof(union swap_header, magic.magic) - \
 	  offsetof(union swap_header, info.badpages)) / sizeof(int))
 
+/*
+ * Swap device flags, except the ones documented below, all are immutable
+ * after exposed by swap_device_enable, and until the device is freed again
+ * (SWP_USED unset). The exceptions are all protected by swapon_rwsem:
+ * - SWP_USED: Protected by swapon_rwsem. Indicates the device is inuse. Once
+ *   set, won't be cleared unless all reference to this device is freed and
+ *   swapoff finished.
+ * - SWP_WRITEOK: Protected by both swapon_rwsem and swap_avail_lock, clearing
+ *   this flag also waits for all current cluster lock users to exit so
+ *   checking this flag while holding any of these locks ensures the device
+ *   is safe to use at the moment. Note: clearing this flag doesn't affect
+ *   pending IO or async requests, it only prevents further entry allocation
+ *   or new async request (e.g. discard) from initiating.
+ * - SWP_HIBERNATION: Protected by swapon_rwsem. Indicates if the device
+ *   is pinned for hibernation.
+ */
 enum {
 	SWP_USED	= (1 << 0),	/* is slot in swap_info[] used? */
 	SWP_WRITEOK	= (1 << 1),	/* ok to write to this swap?	*/
@@ -230,10 +246,17 @@ enum {
 #endif
 
 /*
- * We keep using same cluster for rotational device so IO will be sequential.
- * The purpose is to optimize SWAP throughput on these device.
+ * We assign a cluster to each CPU, so each CPU can allocate swap entry from
+ * its own cluster and swapout sequentially. The purpose is to optimize swapout
+ * throughput.
  */
+struct percpu_cluster {
+	local_lock_t lock; /* Protect the percpu_cluster above */
+	unsigned int next[SWAP_NR_ORDERS]; /* Likely next allocation offset */
+};
+
 struct swap_sequential_cluster {
+	spinlock_t lock; /* Serialize usage of global cluster */
 	unsigned int next[SWAP_NR_ORDERS]; /* Likely next allocation offset */
 };
 
@@ -244,7 +267,6 @@ struct swap_info_struct {
 	struct percpu_ref users;	/* indicate and keep swap device valid. */
 	unsigned long	flags;		/* SWP_USED etc: see above */
 	signed short	prio;		/* swap priority of this type */
-	struct plist_node list;		/* entry in swap_active_head */
 	signed char	type;		/* strange name for an index */
 	unsigned int	max;		/* size of this swap device */
 	struct swap_cluster_info *cluster_info; /* cluster info. Only for SSD */
@@ -256,26 +278,20 @@ struct swap_info_struct {
 					/* list of cluster that are fragmented or contented */
 	unsigned int pages;		/* total of usable pages of swap */
 	atomic_long_t inuse_pages;	/* number of those currently in use */
+	struct percpu_cluster	__percpu *percpu_cluster; /* per cpu's swap location */
 	struct swap_sequential_cluster *global_cluster; /* Use one global cluster for rotating device */
-	spinlock_t global_cluster_lock;	/* Serialize usage of global cluster */
 	struct rb_root swap_extent_root;/* root of the swap extent rbtree */
 	struct block_device *bdev;	/* swap device or bdev of swap file */
 	struct file *swap_file;		/* seldom referenced */
 	struct completion comp;		/* seldom referenced */
 	spinlock_t lock;		/*
-					 * protect map scan related fields like
-					 * inuse_pages and all cluster lists.
-					 * Other fields are only changed
-					 * at swapon/swapoff, so are protected
-					 * by swap_lock. changing flags need
-					 * hold this lock and swap_lock. If
-					 * both locks need hold, hold swap_lock
-					 * first.
+					 * Protect cluster lists. Other fields
+					 * are only changed at swapon/swapoff,
+					 * so are protected by swapon_rwsem.
 					 */
 	struct work_struct discard_work; /* discard worker */
 	struct work_struct reclaim_work; /* reclaim worker */
 	struct list_head discard_clusters; /* discard clusters list */
-	struct plist_node avail_list;   /* entry in swap_avail_head */
 };
 
 static inline swp_entry_t page_swap_entry(struct page *page)
diff --git a/lib/plist.c b/lib/plist.c
index a5bef38add431d..b05ae7ffea87e8 100644
--- a/lib/plist.c
+++ b/lib/plist.c
@@ -142,58 +142,6 @@ void plist_del(struct plist_node *node, struct plist_head *head)
 	plist_check_head(head);
 }
 
-/**
- * plist_requeue - Requeue @node at end of same-prio entries.
- *
- * This is essentially an optimized plist_del() followed by
- * plist_add().  It moves an entry already in the plist to
- * after any other same-priority entries.
- *
- * @node:	&struct plist_node pointer - entry to be moved
- * @head:	&struct plist_head pointer - list head
- */
-void plist_requeue(struct plist_node *node, struct plist_head *head)
-{
-	struct plist_node *iter;
-	struct list_head *node_next = &head->node_list;
-
-	plist_check_head(head);
-	BUG_ON(plist_head_empty(head));
-	BUG_ON(plist_node_empty(node));
-
-	if (node == plist_last(head))
-		return;
-
-	iter = plist_next(node);
-
-	if (node->prio != iter->prio)
-		return;
-
-	plist_del(node, head);
-
-	/*
-	 * After plist_del(), iter is the replacement of the node.  If the node
-	 * was on prio_list, take shortcut to find node_next instead of looping.
-	 */
-	if (!list_empty(&iter->prio_list)) {
-		iter = list_entry(iter->prio_list.next, struct plist_node,
-				  prio_list);
-		node_next = &iter->node_list;
-		goto queue;
-	}
-
-	plist_for_each_continue(iter, head) {
-		if (node->prio != iter->prio) {
-			node_next = &iter->node_list;
-			break;
-		}
-	}
-queue:
-	list_add_tail(&node->node_list, node_next);
-
-	plist_check_head(head);
-}
-
 #ifdef CONFIG_DEBUG_PLIST
 #include <linux/sched.h>
 #include <linux/sched/clock.h>
@@ -231,14 +179,6 @@ static void __init plist_test_check(int nr_expect)
 	BUG_ON(prio_pos->prio_list.next != &first->prio_list);
 }
 
-static void __init plist_test_requeue(struct plist_node *node)
-{
-	plist_requeue(node, &test_head);
-
-	if (node != plist_last(&test_head))
-		BUG_ON(node->prio == plist_next(node)->prio);
-}
-
 static int  __init plist_test(void)
 {
 	int nr_expect = 0, i, loop;
@@ -262,10 +202,6 @@ static int  __init plist_test(void)
 			nr_expect--;
 		}
 		plist_test_check(nr_expect);
-		if (!plist_node_empty(test_node + i)) {
-			plist_test_requeue(test_node + i);
-			plist_test_check(nr_expect);
-		}
 	}
 
 	for (i = 0; i < ARRAY_SIZE(test_node); i++) {
diff --git a/mm/swap.h b/mm/swap.h
index b51ad3071a7308..08a782f3d5acd2 100644
--- a/mm/swap.h
+++ b/mm/swap.h
@@ -91,16 +91,16 @@ static inline unsigned int swp_cluster_offset(swp_entry_t entry)
  */
 static inline struct swap_info_struct *__swap_type_to_info(int type)
 {
-	struct swap_info_struct *si;
-
-	si = READ_ONCE(swap_info[type]); /* rcu_dereference() */
-	VM_WARN_ON_ONCE(percpu_ref_is_zero(&si->users)); /* race with swapoff */
-	return si;
+	return READ_ONCE(swap_info[type]); /* rcu_dereference() */
 }
 
 static inline struct swap_info_struct *__swap_entry_to_info(swp_entry_t entry)
 {
-	return __swap_type_to_info(swp_type(entry));
+	struct swap_info_struct *si;
+
+	si = __swap_type_to_info(swp_type(entry)); /* rcu_dereference() */
+	VM_WARN_ON_ONCE(percpu_ref_is_zero(&si->users)); /* race with swapoff */
+	return si;
 }
 
 static inline struct swap_cluster_info *__swap_offset_to_cluster(
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 775c385490a5fa..f93863309a5a50 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -41,7 +41,6 @@
 #include <linux/completion.h>
 #include <linux/suspend.h>
 #include <linux/zswap.h>
-#include <linux/plist.h>
 
 #include <asm/tlbflush.h>
 #include <linux/leafops.h>
@@ -49,31 +48,38 @@
 #include "internal.h"
 #include "swap.h"
 
-static void swap_range_alloc(struct swap_info_struct *si,
-			     unsigned int nr_entries);
+static void swap_device_inuse_add(struct swap_info_struct *si,
+				  unsigned int nr_entries);
 static bool folio_swapcache_freeable(struct folio *folio);
 static void move_cluster(struct swap_info_struct *si,
 			 struct swap_cluster_info *ci, struct list_head *list,
 			 enum swap_cluster_flags new_flags);
+static bool get_swap_device_info(struct swap_info_struct *si);
 
 /*
- * Protects the swap_info array, and the SWP_USED flag. swap_info contains
- * lazily allocated & freed swap device info struts, and SWP_USED indicates
- * which device is used, ~SWP_USED devices and can be reused.
- *
- * Also protects swap_active_head total_swap_pages, and the SWP_WRITEOK flag.
+ * Serializes swapon/swapoff (writers) and protects the swap_info
+ * array, nr_swapfiles, total_swap_pages, and part of swap device
+ * info content (see comment of swap_info_struct).  Readers
+ * (allocation, /proc/swaps, etc.) take percpu_down_read() which
+ * is cheap as the hot path.
+ */
+DEFINE_STATIC_PERCPU_RWSEM(swapon_rwsem);
+/*
+ * swap_info contains lazily allocated swap device info structs, and
+ * SWP_USED indicates which device is used, ~SWP_USED devices can be
+ * reused. Protected by swapon_rwsem, but reading could be lockless.
  */
-static DEFINE_SPINLOCK(swap_lock);
+struct swap_info_struct *swap_info[MAX_SWAPFILES];
 static unsigned int nr_swapfiles;
-atomic_long_t nr_swap_pages;
+long total_swap_pages;
+
 /*
  * Some modules use swappable objects and may try to swap them out under
  * memory pressure (via the shrinker). Before doing so, they may wish to
  * check to see if any swap space is available.
  */
+atomic_long_t nr_swap_pages;
 EXPORT_SYMBOL_GPL(nr_swap_pages);
-/* protected with swap_lock. reading in vm_swap_full() doesn't need lock */
-long total_swap_pages;
 #define DEF_SWAP_PRIO  -1
 unsigned long swapfile_maximum_size;
 #ifdef CONFIG_MIGRATION
@@ -83,52 +89,42 @@ bool swap_migration_ad_supported;
 static const char Bad_file[] = "Bad swap file entry ";
 static const char Bad_offset[] = "Bad swap offset entry ";
 
-/*
- * all active swap_info_structs
- * protected with swap_lock, and ordered by priority.
- */
-static PLIST_HEAD(swap_active_head);
+static inline struct swap_info_struct *__swap_iter(int *i, unsigned long flag)
+{
+	lockdep_assert_held(&swapon_rwsem);
+	while (*i < nr_swapfiles) {
+		struct swap_info_struct *si = __swap_type_to_info(*i);
+
+		VM_WARN_ON(!si);
+		(*i)++;
+		if (flag && !((si->flags & flag) == flag))
+			continue;
+		return si;
+	}
+	return NULL;
+}
+
+#define __for_each_swap(si, flag)			\
+	for (int __i = 0; ((si) = __swap_iter(&__i, flag));)
 
 /*
- * all available (active, not full) swap_info_structs
- * protected with swap_avail_lock, ordered by priority.
- * This is used by folio_alloc_swap() instead of swap_active_head
- * because swap_active_head includes all swap_info_structs,
- * but folio_alloc_swap() doesn't need to look at full ones.
- * This uses its own lock instead of swap_lock because when a
- * swap_info_struct changes between not-full/full, it needs to
- * add/remove itself to/from this list, but the swap_info_struct->lock
- * is held and the locking order requires swap_lock to be taken
- * before any swap_info_struct->lock.
+ * for_each_swap - iterate through all allocated and inuse swap devices
+ * @si: the iterator
+ *
+ * Context: The caller must hold swapon_rwsem. The lock may be dropped during
+ * the loop body but must be re-acquired before the next iteration.
  */
-static PLIST_HEAD(swap_avail_head);
-static DEFINE_SPINLOCK(swap_avail_lock);
-
-struct swap_info_struct *swap_info[MAX_SWAPFILES];
+#define for_each_swap(si) __for_each_swap(si, SWP_USED)
+#define for_each_avail_swap(si) __for_each_swap(si, SWP_USED | SWP_WRITEOK)
 
 static struct kmem_cache *swap_table_cachep;
 
-/* Protects si->swap_file for /proc/swaps usage */
-static DEFINE_MUTEX(swapon_mutex);
-
 static DECLARE_WAIT_QUEUE_HEAD(proc_poll_wait);
 /* Activity counter to indicate that a swapon or swapoff has occurred */
 static atomic_t proc_poll_event = ATOMIC_INIT(0);
 
 atomic_t nr_rotate_swap = ATOMIC_INIT(0);
 
-struct percpu_swap_cluster {
-	struct swap_info_struct *si[SWAP_NR_ORDERS];
-	unsigned long offset[SWAP_NR_ORDERS];
-	local_lock_t lock;
-};
-
-static DEFINE_PER_CPU(struct percpu_swap_cluster, percpu_swap_cluster) = {
-	.si = { NULL },
-	.offset = { SWAP_ENTRY_INVALID },
-	.lock = INIT_LOCAL_LOCK(),
-};
-
 /* May return NULL on invalid type, caller must check for NULL return */
 static struct swap_info_struct *swap_type_to_info(int type)
 {
@@ -143,18 +139,350 @@ static struct swap_info_struct *swap_entry_to_info(swp_entry_t entry)
 	return swap_type_to_info(swp_type(entry));
 }
 
+/*
+ * All available swap_info_structs are grouped by priority rings, the rings
+ * are ordered in a queue by priority (lower prio value = higher priority).
+ * The allocator iterates and rotates devices within each priority ring.
+ * When all devices in a ring are iterated, it goes to the next lower
+ * priority ring.
+ */
+struct swap_prio_ring {
+	int prio;
+	unsigned int size;
+	struct swap_info_struct *dev[] __counted_by(size);
+};
+
+/*
+ * The ring is protected by swapon_rwsem so updating it is costly. To make
+ * the allocator and other users skip full devices faster, the lowest bit of
+ * a device pointer is used to mark it disabled (temporarily unavailable).
+ * This relies on struct swap_info_struct being sufficiently
+ * aligned (guaranteed by kmalloc).
+ */
+#define SWAP_DEVICE_MASKED_SHIFT	0
+#define SWAP_DEVICE_MASKED_BIT		BIT(SWAP_DEVICE_MASKED_SHIFT)
+
+/*
+ * Serializes queue content mutations and keeps SWAP_USAGE_OFFLIST_BIT
+ * consistent with the masked state of each device pointer.
+ */
+static DEFINE_SPINLOCK(swap_queue_update_lock);
+
+/*
+ * Swap queue is protected by both swap_queue_update_lock and swapon_rwsem.
+ * Only swapon/swapoff will take the write lock, and modify the queue length
+ * or any ring's length. swap_queue_update_lock protects the content so
+ * devices can be masked easily without taking the writelock, which is heavy.
+ */
+static struct swap_prio_ring **swap_queue;
+static unsigned int swap_queue_len;
+
+/*
+ * Each CPU has its read iterator, so the queue itself will remain read
+ * only and the CPU side reader rotates by iterating the devices
+ * periodically using the counter.
+ */
+#define SWAP_ROUND_ROBIN_QUOTA SWAPFILE_CLUSTER
+struct swap_ring_iterator {
+	int offset;
+	long rr_counter;
+};
+
+struct swap_queue_reader {
+	local_lock_t lock;
+	struct swap_ring_iterator ri[];
+};
+
+static __percpu struct swap_queue_reader *swap_queue_readers;
+
+static inline bool swap_device_masked(struct swap_info_struct *si)
+{
+	return (unsigned long)si & SWAP_DEVICE_MASKED_BIT;
+}
+
+static inline struct swap_info_struct *swap_device_unmask_ptr(struct swap_info_struct *si)
+{
+	return (struct swap_info_struct *)((unsigned long)si & ~SWAP_DEVICE_MASKED_BIT);
+}
+
+static struct swap_queue_reader __percpu *swap_queue_prealloc_readers(int nr_rings, gfp_t gfp)
+{
+	struct swap_queue_reader __percpu *readers;
+
+	if (!nr_rings)
+		return NULL;
+
+	readers = __alloc_percpu_gfp(struct_size(readers, ri, nr_rings),
+				     __alignof__(*readers), gfp);
+	return readers;
+}
+
+static void swap_queue_install_readers(struct swap_queue_reader __percpu *readers)
+{
+	int ring_idx, cpu;
+	struct swap_prio_ring *ring;
+
+	free_percpu(swap_queue_readers);
+	swap_queue_readers = readers;
+	if (!readers)
+		return;
+
+	/* Distribute each CPU's swap IO fairly across devices. */
+	for_each_possible_cpu(cpu) {
+		local_lock_init(&per_cpu_ptr(readers, cpu)->lock);
+
+		for (ring_idx = 0; ring_idx < swap_queue_len; ring_idx++) {
+			ring = swap_queue[ring_idx];
+			per_cpu_ptr(readers, cpu)->ri[ring_idx].offset =
+				    cpu % ring->size;
+		}
+	}
+}
+
+static struct swap_info_struct *swap_queue_get_device(long nr_alloc, int nr_iter)
+{
+	bool rotate = false;
+	struct swap_info_struct *si;
+	struct swap_ring_iterator *ri;
+	struct swap_prio_ring *ring;
+	unsigned int queue_idx;
+
+	if (!swap_queue_len)
+		return ERR_PTR(-ENOENT);
+
+	queue_idx = 0;
+	while (nr_iter >= swap_queue[queue_idx]->size) {
+		nr_iter -= swap_queue[queue_idx]->size;
+		if (++queue_idx >= swap_queue_len)
+			return ERR_PTR(-ENOENT);
+	}
+
+	ring = swap_queue[queue_idx];
+	local_lock(&swap_queue_readers->lock);
+	ri = this_cpu_ptr(&swap_queue_readers->ri[queue_idx]);
+	/* Rotate while iterating the ring, just not on the first try */
+	if (nr_iter)
+		rotate = true;
+	else if (ri->rr_counter < nr_alloc)
+		rotate = true;
+	else if (ri->offset >= ring->size)
+		rotate = true;
+	if (rotate) {
+		ri->offset++;
+		ri->offset %= ring->size;
+		ri->rr_counter = SWAP_ROUND_ROBIN_QUOTA;
+	}
+	ri->rr_counter -= nr_alloc;
+	si = READ_ONCE(ring->dev[ri->offset]);
+	local_unlock(&swap_queue_readers->lock);
+
+	if (swap_device_masked(si))
+		return ERR_PTR(-EBUSY);
+
+	si = swap_device_unmask_ptr(si);
+	return si;
+}
+
+static bool swap_queue_find(struct swap_info_struct *si,
+			    unsigned int *ring_idx, unsigned int *dev_idx)
+{
+	unsigned int i, j;
+	struct swap_prio_ring *ring;
+
+	lockdep_assert(lockdep_is_held(&swapon_rwsem) ||
+		       lockdep_is_held(&swap_queue_update_lock));
+
+	for (i = 0; i < swap_queue_len; i++) {
+		ring = swap_queue[i];
+		if (ring->prio != si->prio)
+			continue;
+		for (j = 0; j < ring->size; j++) {
+			if (swap_device_unmask_ptr(READ_ONCE(ring->dev[j])) != si)
+				continue;
+			*ring_idx = i;
+			*dev_idx = j;
+			return true;
+		}
+	}
+	return false;
+}
+
+static void swap_queue_mask(struct swap_info_struct *si)
+{
+	unsigned int ring_idx, dev_idx;
+
+	lockdep_assert_held(&swap_queue_update_lock);
+	if (swap_queue_find(si, &ring_idx, &dev_idx))
+		__set_bit(SWAP_DEVICE_MASKED_SHIFT,
+			  (unsigned long *)&swap_queue[ring_idx]->dev[dev_idx]);
+}
+
+static void swap_queue_unmask(struct swap_info_struct *si)
+{
+	unsigned int ring_idx, dev_idx;
+
+	lockdep_assert_held(&swap_queue_update_lock);
+	if (swap_queue_find(si, &ring_idx, &dev_idx))
+		__clear_bit(SWAP_DEVICE_MASKED_SHIFT,
+			    (unsigned long *)&swap_queue[ring_idx]->dev[dev_idx]);
+}
+
+static int swap_queue_add(struct swap_info_struct *si)
+{
+	struct swap_prio_ring **new_queue = NULL, **old_queue = NULL;
+	struct swap_queue_reader __percpu *new_readers = NULL;
+	struct swap_prio_ring *ring, *new_ring = NULL, *old_ring = NULL;
+	int prio = si->prio;
+	int i, pos, err = -ENOMEM;
+	gfp_t gfp;
+
+	/* Swap not usable here because this is swap, just reclaim cache. */
+	gfp = GFP_NOIO | __GFP_HIGH;
+	lockdep_assert_held_write(&swapon_rwsem);
+
+	for (pos = 0; pos < swap_queue_len; pos++) {
+		if (swap_queue[pos]->prio == prio)
+			goto add_to_ring;
+		if (swap_queue[pos]->prio < prio)
+			break;
+	}
+
+	/* No ring at this priority: insert a new one at pos. */
+	new_readers = swap_queue_prealloc_readers(swap_queue_len + 1, gfp);
+	if (!new_readers)
+		goto failed;
+	new_queue = kmalloc_array(swap_queue_len + 1, sizeof(*swap_queue), gfp);
+	if (!new_queue)
+		goto failed;
+	new_ring = kmalloc(struct_size(new_ring, dev, 1), gfp);
+	if (!new_ring)
+		goto failed;
+	if (!get_swap_device_info(si))
+		goto failed;
+
+	new_ring->prio = prio;
+	new_ring->size = 1;
+	new_ring->dev[0] = si;
+	for (i = 0; i < pos; i++)
+		new_queue[i] = swap_queue[i];
+	new_queue[pos] = new_ring;
+	for (i = pos; i < swap_queue_len; i++)
+		new_queue[i + 1] = swap_queue[i];
+
+	spin_lock(&swap_queue_update_lock);
+	old_queue = swap_queue;
+	swap_queue = new_queue;
+	swap_queue_len++;
+	spin_unlock(&swap_queue_update_lock);
+	kfree(old_queue);
+
+	swap_queue_install_readers(new_readers);
+	return 0;
+
+add_to_ring:
+	ring = swap_queue[pos];
+	new_ring = kmalloc(struct_size(ring, dev, ring->size + 1), gfp);
+	if (!new_ring)
+		goto failed;
+	if (!get_swap_device_info(si))
+		goto failed;
+	spin_lock(&swap_queue_update_lock);
+	memcpy(new_ring, ring, struct_size(ring, dev, ring->size));
+	new_ring->size++;
+	new_ring->dev[new_ring->size - 1] = si;
+	old_ring = swap_queue[pos];
+	swap_queue[pos] = new_ring;
+	spin_unlock(&swap_queue_update_lock);
+	kfree(old_ring);
+	return 0;
+
+failed:
+	free_percpu(new_readers);
+	kfree(new_queue);
+	kfree(new_ring);
+	return err;
+}
+
+static void swap_queue_del(struct swap_info_struct *si)
+{
+	gfp_t gfp;
+	unsigned int ring_idx, dev_idx;
+	struct swap_queue_reader __percpu *new_readers = NULL;
+	struct swap_prio_ring *ring, *new_ring = NULL, *old_ring = NULL;
+	struct swap_prio_ring **new_queue = NULL, **old_queue = NULL;
+
+	lockdep_assert_held_write(&swapon_rwsem);
+	if (!swap_queue_find(si, &ring_idx, &dev_idx)) {
+		WARN_ON(1);
+		return;
+	}
+
+	/*
+	 * To shrink memory usage, pre-allocate new smaller data before
+	 * locking. Failure is fine, swapoff will release them anyway.
+	 */
+	gfp = GFP_NOIO | __GFP_HIGH;
+	ring = swap_queue[ring_idx];
+	if (ring->size > 1)
+		new_ring = kmalloc(struct_size(ring, dev, ring->size - 1), gfp);
+	if (ring->size == 1 && swap_queue_len > 1) {
+		new_readers = swap_queue_prealloc_readers(
+					swap_queue_len - 1, gfp);
+		new_queue = kmalloc(sizeof(*swap_queue) *
+				    (swap_queue_len - 1), gfp);
+	}
+
+	spin_lock(&swap_queue_update_lock);
+	if (ring->size > 1) {
+		/* Shift trailing devices left to fill the gap. */
+		while (++dev_idx < ring->size)
+			ring->dev[dev_idx - 1] =
+				ring->dev[dev_idx];
+		ring->size--;
+		if (new_ring) {
+			memcpy(new_ring, ring,
+			       struct_size(ring, dev, ring->size));
+			old_ring = ring;
+			swap_queue[ring_idx] = new_ring;
+		}
+	} else {
+		/* Last device in this ring: remove the ring. */
+		old_ring = ring;
+		swap_queue_len--;
+		while (++ring_idx <= swap_queue_len)
+			swap_queue[ring_idx - 1] =
+				swap_queue[ring_idx];
+		if (new_queue) {
+			memcpy(new_queue, swap_queue,
+			       sizeof(*swap_queue) * swap_queue_len);
+			old_queue = swap_queue;
+			swap_queue = new_queue;
+		} else if (!swap_queue_len) {
+			old_queue = swap_queue;
+			swap_queue = NULL;
+		}
+		if (new_readers || !swap_queue_len)
+			swap_queue_install_readers(new_readers);
+	}
+	spin_unlock(&swap_queue_update_lock);
+
+	kfree(old_ring);
+	kfree(old_queue);
+	put_swap_device(si);
+}
+
 /*
  * Use the second highest bit of inuse_pages counter as the indicator
- * if one swap device is on the available plist, so the atomic can
+ * if one swap device is unavailable for allocation, so the atomic can
  * still be updated arithmetically while having special data embedded.
  *
  * inuse_pages counter is the only thing indicating if a device should
- * be on avail_lists or not (except swapon / swapoff). By embedding the
- * off-list bit in the atomic counter, updates no longer need any lock
- * to check the list status.
+ * be in the available queue or not (except swapon / swapoff). By
+ * embedding the off-list bit in the atomic counter, updates no longer
+ * need any lock to check the list status.
  *
- * This bit will be set if the device is not on the plist and not
- * usable, will be cleared if the device is on the plist.
+ * This bit will be set if the device is not in the available queue
+ * and not usable, will be cleared if the device is in the queue.
  */
 #define SWAP_USAGE_OFFLIST_BIT (1UL << (BITS_PER_TYPE(atomic_t) - 2))
 #define SWAP_USAGE_COUNTER_MASK (~SWAP_USAGE_OFFLIST_BIT)
@@ -163,6 +491,26 @@ static long swap_usage_in_pages(struct swap_info_struct *si)
 	return atomic_long_read(&si->inuse_pages) & SWAP_USAGE_COUNTER_MASK;
 }
 
+/*
+ * Serialize the allocation on single CPU or globally to avoid
+ * fragmentation and make the workflow easier to follow.
+ */
+static void swap_alloc_lock_device(struct swap_info_struct *si)
+{
+	if (si->flags & SWP_SOLIDSTATE)
+		local_lock(&si->percpu_cluster->lock);
+	else
+		spin_lock(&si->global_cluster->lock);
+}
+
+static void swap_alloc_unlock_device(struct swap_info_struct *si)
+{
+	if (si->flags & SWP_SOLIDSTATE)
+		local_unlock(&si->percpu_cluster->lock);
+	else
+		spin_unlock(&si->global_cluster->lock);
+}
+
 /* Reclaim the swap entry anyway if possible */
 #define TTRS_ANYWAY		0x1
 /*
@@ -533,9 +881,10 @@ swap_cluster_populate(struct swap_info_struct *si,
 	 * Only cluster isolation from the allocator does table allocation.
 	 * Swap allocator uses percpu clusters and holds the local lock.
 	 */
-	lockdep_assert_held(&this_cpu_ptr(&percpu_swap_cluster)->lock);
-	if (!(si->flags & SWP_SOLIDSTATE))
-		lockdep_assert_held(&si->global_cluster_lock);
+	if (si->flags & SWP_SOLIDSTATE)
+		lockdep_assert_held(this_cpu_ptr(&si->percpu_cluster->lock));
+	else
+		lockdep_assert_held(&si->global_cluster->lock);
 	lockdep_assert_held(&ci->lock);
 
 	if (!swap_cluster_alloc_table(ci, __GFP_HIGH | __GFP_NOMEMALLOC |
@@ -548,9 +897,7 @@ swap_cluster_populate(struct swap_info_struct *si,
 	 * the potential recursive allocation is limited.
 	 */
 	spin_unlock(&ci->lock);
-	if (!(si->flags & SWP_SOLIDSTATE))
-		spin_unlock(&si->global_cluster_lock);
-	local_unlock(&percpu_swap_cluster.lock);
+	swap_alloc_unlock_device(si);
 
 	ret = swap_cluster_alloc_table(ci, __GFP_HIGH | __GFP_NOMEMALLOC |
 					   GFP_KERNEL);
@@ -563,9 +910,7 @@ swap_cluster_populate(struct swap_info_struct *si,
 	 * could happen with ignoring the percpu cluster is fragmentation,
 	 * which is acceptable since this fallback and race is rare.
 	 */
-	local_lock(&percpu_swap_cluster.lock);
-	if (!(si->flags & SWP_SOLIDSTATE))
-		spin_lock(&si->global_cluster_lock);
+	swap_alloc_lock_device(si);
 	spin_lock(&ci->lock);
 
 	if (ret) {
@@ -671,7 +1016,7 @@ static bool swap_do_scheduled_discard(struct swap_info_struct *si)
 		ci = list_first_entry(&si->discard_clusters, struct swap_cluster_info, list);
 		/*
 		 * Delete the cluster from list to prepare for discard, but keep
-		 * the CLUSTER_FLAG_DISCARD flag, percpu_swap_cluster could be
+		 * the CLUSTER_FLAG_DISCARD flag, there could be percpu_cluster
 		 * pointing to it, or ran into by relocate_cluster.
 		 */
 		list_del(&ci->list);
@@ -955,7 +1300,7 @@ static bool __swap_cluster_alloc_entries(struct swap_info_struct *si,
 	if (cluster_is_empty(ci))
 		ci->order = order;
 	ci->count += nr_pages;
-	swap_range_alloc(si, nr_pages);
+	swap_device_inuse_add(si, nr_pages);
 
 	return true;
 }
@@ -1003,12 +1348,10 @@ static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si,
 out:
 	relocate_cluster(si, ci);
 	swap_cluster_unlock(ci);
-	if (si->flags & SWP_SOLIDSTATE) {
-		this_cpu_write(percpu_swap_cluster.offset[order], next);
-		this_cpu_write(percpu_swap_cluster.si[order], si);
-	} else {
+	if (si->flags & SWP_SOLIDSTATE)
+		this_cpu_write(si->percpu_cluster->next[order], next);
+	else
 		si->global_cluster->next[order] = next;
-	}
 	return found;
 }
 
@@ -1103,13 +1446,14 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,
 	if (order && !(si->flags & SWP_BLKDEV))
 		return 0;
 
-	if (!(si->flags & SWP_SOLIDSTATE)) {
-		/* Serialize HDD SWAP allocation for each device. */
-		spin_lock(&si->global_cluster_lock);
+restart:
+	swap_alloc_lock_device(si);
+	if (si->flags & SWP_SOLIDSTATE)
+		offset = __this_cpu_read(si->percpu_cluster->next[order]);
+	else
 		offset = si->global_cluster->next[order];
-		if (offset == SWAP_ENTRY_INVALID)
-			goto new_cluster;
 
+	if (offset != SWAP_ENTRY_INVALID) {
 		ci = swap_cluster_lock(si, offset);
 		/* Cluster could have been used by another order */
 		if (cluster_is_usable(ci, order)) {
@@ -1123,7 +1467,6 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,
 			goto done;
 	}
 
-new_cluster:
 	/*
 	 * If the device need discard, prefer new cluster over nonfull
 	 * to spread out the writes.
@@ -1132,6 +1475,12 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,
 		found = alloc_swap_scan_list(si, &si->free_clusters, folio, false);
 		if (found)
 			goto done;
+
+		if (!list_empty(&si->discard_clusters)) {
+			swap_alloc_unlock_device(si);
+			swap_do_scheduled_discard(si);
+			goto restart;
+		}
 	}
 
 	if (order < PMD_ORDER) {
@@ -1180,8 +1529,7 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,
 			goto done;
 	}
 done:
-	if (!(si->flags & SWP_SOLIDSTATE))
-		spin_unlock(&si->global_cluster_lock);
+	swap_alloc_unlock_device(si);
 
 	return found;
 }
@@ -1191,54 +1539,44 @@ static void del_from_avail_list(struct swap_info_struct *si, bool swapoff)
 {
 	unsigned long pages;
 
-	spin_lock(&swap_avail_lock);
+	spin_lock(&swap_queue_update_lock);
 
-	if (swapoff) {
-		/*
-		 * Forcefully remove it. Clear the SWP_WRITEOK flags for
-		 * swapoff here so it's synchronized by both si->lock and
-		 * swap_avail_lock, to ensure the result can be seen by
-		 * add_to_avail_list.
-		 */
-		lockdep_assert_held(&si->lock);
-		si->flags &= ~SWP_WRITEOK;
-		atomic_long_or(SWAP_USAGE_OFFLIST_BIT, &si->inuse_pages);
-	} else {
-		/*
-		 * If not called by swapoff, take it off-list only if it's
-		 * full and SWAP_USAGE_OFFLIST_BIT is not set (strictly
-		 * si->inuse_pages == pages), any concurrent slot freeing,
-		 * or device already removed from plist by someone else
-		 * will make this return false.
-		 */
+	/*
+	 * Force remove it only for swapoff. Else, take it off-list only if
+	 * it's full and SWAP_USAGE_OFFLIST_BIT is not set (strictly
+	 * si->inuse_pages == pages), so concurrent slot freeing, or
+	 * concurrent list removal will make the cmpxchg fail and skip
+	 * the removal.
+	 */
+	if (!swapoff) {
 		pages = si->pages;
 		if (!atomic_long_try_cmpxchg(&si->inuse_pages, &pages,
 					     pages | SWAP_USAGE_OFFLIST_BIT))
 			goto skip;
+	} else {
+		atomic_long_or(SWAP_USAGE_OFFLIST_BIT, &si->inuse_pages);
 	}
 
-	plist_del(&si->avail_list, &swap_avail_head);
-
+	swap_queue_mask(si);
 skip:
-	spin_unlock(&swap_avail_lock);
+	spin_unlock(&swap_queue_update_lock);
 }
 
 /* SWAP_USAGE_OFFLIST_BIT can only be cleared by this helper. */
-static void add_to_avail_list(struct swap_info_struct *si, bool swapon)
+static void add_to_avail_list(struct swap_info_struct *si)
 {
 	long val;
 	unsigned long pages;
 
-	spin_lock(&swap_avail_lock);
+	spin_lock(&swap_queue_update_lock);
 
-	/* Corresponding to SWP_WRITEOK clearing in del_from_avail_list */
-	if (swapon) {
-		lockdep_assert_held(&si->lock);
-		si->flags |= SWP_WRITEOK;
-	} else {
-		if (!(READ_ONCE(si->flags) & SWP_WRITEOK))
-			goto skip;
-	}
+	/*
+	 * Mark the device as avail if SWP_WRITEOK is set. Swapoff clears
+	 * SWP_WRITEOK first, so check that first so the device won't be
+	 * re-added after swapoff started.
+	 */
+	if (!(si->flags & SWP_WRITEOK))
+		goto skip;
 
 	if (!(atomic_long_read(&si->inuse_pages) & SWAP_USAGE_OFFLIST_BIT))
 		goto skip;
@@ -1246,9 +1584,10 @@ static void add_to_avail_list(struct swap_info_struct *si, bool swapon)
 	val = atomic_long_fetch_and_relaxed(~SWAP_USAGE_OFFLIST_BIT, &si->inuse_pages);
 
 	/*
-	 * When device is full and device is on the plist, only one updater will
-	 * see (inuse_pages == si->pages) and will call del_from_avail_list. If
-	 * that updater happen to be here, just skip adding.
+	 * When device is full and marked as available, one reader will see
+	 * (inuse_pages == si->pages) and should mark it as unavailable and
+	 * set SWAP_USAGE_OFFLIST_BIT. If that updater happens to be here, just
+	 * skip the rest.
 	 */
 	pages = si->pages;
 	if (val == pages) {
@@ -1258,60 +1597,42 @@ static void add_to_avail_list(struct swap_info_struct *si, bool swapon)
 			goto skip;
 	}
 
-	plist_add(&si->avail_list, &swap_avail_head);
-
+	swap_queue_unmask(si);
 skip:
-	spin_unlock(&swap_avail_lock);
+	spin_unlock(&swap_queue_update_lock);
 }
 
 /*
- * swap_usage_add / swap_usage_sub of each slot are serialized by ci->lock
- * within each cluster, so the total contribution to the global counter should
- * always be positive and cannot exceed the total number of usable slots.
+ * swap_device_inuse_add / swap_device_inuse_sub are called within cluster
+ * update critical sections and serialized by ci->lock within each cluster,
+ * so the total contribution to the global counter should always be positive
+ * and cannot exceed the total number of usable slots.
  */
-static bool swap_usage_add(struct swap_info_struct *si, unsigned int nr_entries)
+static void swap_device_inuse_add(struct swap_info_struct *si,
+				  unsigned int nr_entries)
 {
-	long val = atomic_long_add_return_relaxed(nr_entries, &si->inuse_pages);
+	long inuse_pages;
 
 	/*
 	 * If device is full, and SWAP_USAGE_OFFLIST_BIT is not set,
-	 * remove it from the plist.
-	 */
-	if (unlikely(val == si->pages)) {
-		del_from_avail_list(si, false);
-		return true;
-	}
-
-	return false;
-}
-
-static void swap_usage_sub(struct swap_info_struct *si, unsigned int nr_entries)
-{
-	long val = atomic_long_sub_return_relaxed(nr_entries, &si->inuse_pages);
-
-	/*
-	 * If device is not full, and SWAP_USAGE_OFFLIST_BIT is set,
-	 * add it to the plist.
+	 * mark it unavailable.
 	 */
-	if (unlikely(val & SWAP_USAGE_OFFLIST_BIT))
-		add_to_avail_list(si, false);
-}
-
-static void swap_range_alloc(struct swap_info_struct *si,
-			     unsigned int nr_entries)
-{
-	if (swap_usage_add(si, nr_entries)) {
+	inuse_pages = atomic_long_add_return_relaxed(nr_entries, &si->inuse_pages);
+	if (unlikely(inuse_pages == si->pages)) {
 		if (vm_swap_full())
 			schedule_work(&si->reclaim_work);
+		del_from_avail_list(si, false);
 	}
+
 	atomic_long_sub(nr_entries, &nr_swap_pages);
 }
 
-static void swap_range_free(struct swap_info_struct *si, unsigned long offset,
-			    unsigned int nr_entries)
+static void swap_device_inuse_sub(struct swap_info_struct *si, unsigned long offset,
+				  unsigned int nr_entries)
 {
 	unsigned long end = offset + nr_entries - 1;
 	void (*swap_slot_free_notify)(struct block_device *, unsigned long);
+	long inuse_pages;
 	unsigned int i;
 
 	for (i = 0; i < nr_entries; i++)
@@ -1335,7 +1656,14 @@ static void swap_range_free(struct swap_info_struct *si, unsigned long offset,
 	 */
 	smp_wmb();
 	atomic_long_add(nr_entries, &nr_swap_pages);
-	swap_usage_sub(si, nr_entries);
+
+	/*
+	 * If device is not full, and SWAP_USAGE_OFFLIST_BIT is set,
+	 * add it back to the available queue.
+	 */
+	inuse_pages = atomic_long_sub_return_relaxed(nr_entries, &si->inuse_pages);
+	if (unlikely(inuse_pages & SWAP_USAGE_OFFLIST_BIT))
+		add_to_avail_list(si);
 }
 
 static bool get_swap_device_info(struct swap_info_struct *si)
@@ -1345,113 +1673,49 @@ static bool get_swap_device_info(struct swap_info_struct *si)
 	/*
 	 * Guarantee the si->users are checked before accessing other
 	 * fields of swap_info_struct, and si->flags (SWP_WRITEOK) is
-	 * up to dated.
+	 * up to date.
 	 *
-	 * Paired with the spin_unlock() after setup_swap_info() in
-	 * enable_swap_info(), and smp_wmb() in swapoff.
+	 * Paired with percpu_up_write() in swap_device_enable(), and
+	 * smp_wmb() after clearing SWP_WRITEOK in swapoff.
 	 */
 	smp_rmb();
 	return true;
 }
 
-/*
- * Fast path try to get swap entries with specified order from current
- * CPU's swap entry pool (a cluster).
- */
-static bool swap_alloc_fast(struct folio *folio)
+static int swap_alloc_entry(struct folio *folio)
 {
-	unsigned int order = folio_order(folio);
-	struct swap_cluster_info *ci;
+	long nr_pages = folio_nr_pages(folio);
 	struct swap_info_struct *si;
-	unsigned int offset;
-
-	/*
-	 * Once allocated, swap_info_struct will never be completely freed,
-	 * so checking it's liveness by get_swap_device_info is enough.
-	 */
-	si = this_cpu_read(percpu_swap_cluster.si[order]);
-	offset = this_cpu_read(percpu_swap_cluster.offset[order]);
-	if (!si || !offset || !get_swap_device_info(si))
-		return false;
-
-	ci = swap_cluster_lock(si, offset);
-	if (cluster_is_usable(ci, order)) {
-		if (cluster_is_empty(ci))
-			offset = cluster_offset(si, ci);
-		alloc_swap_scan_cluster(si, ci, folio, offset);
-	} else {
-		swap_cluster_unlock(ci);
-	}
+	int nr_iter, ret;
+
+	percpu_down_read(&swapon_rwsem);
+	for (nr_iter = 0;; nr_iter++) {
+		si = swap_queue_get_device(nr_pages, nr_iter);
+		if (IS_ERR(si)) {
+			ret = PTR_ERR(si);
+			if (ret == -EBUSY)
+				continue;
+			break;
+		}
+		cluster_alloc_swap_entry(si, folio);
 
-	put_swap_device(si);
-	return folio_test_swapcache(folio);
-}
-
-/* Rotate the device and switch to a new cluster */
-static void swap_alloc_slow(struct folio *folio)
-{
-	struct swap_info_struct *si, *next;
-
-	spin_lock(&swap_avail_lock);
-start_over:
-	plist_for_each_entry_safe(si, next, &swap_avail_head, avail_list) {
-		/* Rotate the device and switch to a new cluster */
-		plist_requeue(&si->avail_list, &swap_avail_head);
-		spin_unlock(&swap_avail_lock);
-		if (get_swap_device_info(si)) {
-			cluster_alloc_swap_entry(si, folio);
-			put_swap_device(si);
-			if (folio_test_swapcache(folio))
-				return;
-			if (folio_test_large(folio))
-				return;
+		if (folio_test_swapcache(folio)) {
+			ret = 0;
+			break;
 		}
 
-		spin_lock(&swap_avail_lock);
 		/*
-		 * if we got here, it's likely that si was almost full before,
-		 * multiple callers probably all tried to get a page from the
-		 * same si and it filled up before we could get one; or, the si
-		 * filled up between us dropping swap_avail_lock.
-		 * Since we dropped the swap_avail_lock, the swap_avail_list
-		 * may have been modified; so if next is still in the
-		 * swap_avail_head list then try it, otherwise start over if we
-		 * have not gotten any slots.
+		 * For large allocation, return error directly to inform the
+		 * caller to split it instead of fallback to other devices.
 		 */
-		if (plist_node_empty(&next->avail_list))
-			goto start_over;
-	}
-	spin_unlock(&swap_avail_lock);
-}
-
-/*
- * Discard pending clusters in a synchronized way when under high pressure.
- * Return: true if any cluster is discarded.
- */
-static bool swap_sync_discard(void)
-{
-	bool ret = false;
-	struct swap_info_struct *si, *next;
-
-	spin_lock(&swap_lock);
-start_over:
-	plist_for_each_entry_safe(si, next, &swap_active_head, list) {
-		spin_unlock(&swap_lock);
-		if (get_swap_device_info(si)) {
-			if (si->flags & SWP_PAGE_DISCARD)
-				ret = swap_do_scheduled_discard(si);
-			put_swap_device(si);
+		if (folio_test_large(folio)) {
+			ret = -E2BIG;
+			break;
 		}
-		if (ret)
-			return true;
-
-		spin_lock(&swap_lock);
-		if (plist_node_empty(&next->list))
-			goto start_over;
 	}
-	spin_unlock(&swap_lock);
 
-	return false;
+	percpu_up_read(&swapon_rwsem);
+	return ret;
 }
 
 static int swap_extend_table_alloc(struct swap_info_struct *si,
@@ -1735,6 +1999,7 @@ int folio_alloc_swap(struct folio *folio)
 {
 	unsigned int order = folio_order(folio);
 	unsigned int size = 1 << order;
+	int ret;
 
 	VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
 	VM_BUG_ON_FOLIO(!folio_test_uptodate(folio), folio);
@@ -1757,23 +2022,14 @@ int folio_alloc_swap(struct folio *folio)
 		}
 	}
 
-again:
-	local_lock(&percpu_swap_cluster.lock);
-	if (!swap_alloc_fast(folio))
-		swap_alloc_slow(folio);
-	local_unlock(&percpu_swap_cluster.lock);
-
-	if (!order && unlikely(!folio_test_swapcache(folio))) {
-		if (swap_sync_discard())
-			goto again;
-	}
+	ret = swap_alloc_entry(folio);
 
 	/* Need to call this even if allocation failed, for MEMCG_SWAP_FAIL. */
 	if (unlikely(mem_cgroup_try_charge_swap(folio)))
 		swap_cache_del_folio(folio);
 
 	if (unlikely(!folio_test_swapcache(folio)))
-		return -ENOMEM;
+		return ret ? ret : -ENOMEM;
 
 	return 0;
 }
@@ -1948,7 +2204,7 @@ void __swap_cluster_free_entries(struct swap_info_struct *si,
 	if (batch_id)
 		mem_cgroup_uncharge_swap(batch_id, ci_off - batch_off);
 
-	swap_range_free(si, ci_head + ci_start, nr_pages);
+	swap_device_inuse_sub(si, ci_head + ci_start, nr_pages);
 	swap_cluster_assert_empty(ci, ci_start, nr_pages, false);
 
 	if (!ci->count)
@@ -2150,31 +2406,14 @@ void swap_put_entries_direct(swp_entry_t entry, int nr)
  */
 swp_entry_t swap_alloc_hibernation_slot(int type)
 {
-	struct swap_info_struct *pcp_si, *si = swap_type_to_info(type);
-	unsigned long pcp_offset, offset = SWAP_ENTRY_INVALID;
-	struct swap_cluster_info *ci;
+	struct swap_info_struct *si = swap_type_to_info(type);
+	unsigned long offset = SWAP_ENTRY_INVALID;
 	swp_entry_t entry = {0};
 
 	if (!si)
 		goto fail;
 
-	/*
-	 * Try the local cluster first if it matches the device. If
-	 * not, try grab a new cluster and override local cluster.
-	 */
-	local_lock(&percpu_swap_cluster.lock);
-	pcp_si = this_cpu_read(percpu_swap_cluster.si[0]);
-	pcp_offset = this_cpu_read(percpu_swap_cluster.offset[0]);
-	if (pcp_si == si && pcp_offset) {
-		ci = swap_cluster_lock(si, pcp_offset);
-		if (cluster_is_usable(ci, 0))
-			offset = alloc_swap_scan_cluster(si, ci, NULL, pcp_offset);
-		else
-			swap_cluster_unlock(ci);
-	}
-	if (!offset)
-		offset = cluster_alloc_swap_entry(si, NULL);
-	local_unlock(&percpu_swap_cluster.lock);
+	offset = cluster_alloc_swap_entry(si, NULL);
 	if (offset)
 		entry = swp_entry(si->type, offset);
 
@@ -2205,26 +2444,18 @@ void swap_free_hibernation_slot(swp_entry_t entry)
 
 static int __find_hibernation_swap_type(dev_t device, sector_t offset)
 {
-	int type;
-
-	lockdep_assert_held(&swap_lock);
+	struct swap_info_struct *si;
 
 	if (!device)
 		return -EINVAL;
 
-	for (type = 0; type < nr_swapfiles; type++) {
-		struct swap_info_struct *sis = swap_info[type];
-
-		if (!(sis->flags & SWP_WRITEOK))
-			continue;
-
-		if (device == sis->bdev->bd_dev) {
-			struct swap_extent *se = first_se(sis);
-
-			if (se->start_block == offset)
-				return type;
+	for_each_avail_swap(si) {
+		if (device == si->bdev->bd_dev) {
+			if (first_se(si)->start_block == offset)
+				return si->type;
 		}
 	}
+
 	return -ENODEV;
 }
 
@@ -2247,21 +2478,18 @@ static int __find_hibernation_swap_type(dev_t device, sector_t offset)
  */
 int pin_hibernation_swap_type(dev_t device, sector_t offset)
 {
-	int type;
+	int ret;
 	struct swap_info_struct *si;
 
-	spin_lock(&swap_lock);
-
-	type = __find_hibernation_swap_type(device, offset);
-	if (type < 0) {
-		spin_unlock(&swap_lock);
-		return type;
-	}
+	percpu_down_write(&swapon_rwsem);
+	ret = __find_hibernation_swap_type(device, offset);
+	if (ret < 0)
+		goto out;
 
-	si = swap_type_to_info(type);
+	si = swap_type_to_info(ret);
 	if (WARN_ON_ONCE(!si)) {
-		spin_unlock(&swap_lock);
-		return -ENODEV;
+		ret = -ENODEV;
+		goto out;
 	}
 
 	/*
@@ -2270,14 +2498,15 @@ int pin_hibernation_swap_type(dev_t device, sector_t offset)
 	 * the same session.
 	 */
 	if (WARN_ON_ONCE(si->flags & SWP_HIBERNATION)) {
-		spin_unlock(&swap_lock);
-		return -EBUSY;
+		ret = -EBUSY;
+		goto out;
 	}
 
 	si->flags |= SWP_HIBERNATION;
 
-	spin_unlock(&swap_lock);
-	return type;
+out:
+	percpu_up_write(&swapon_rwsem);
+	return ret;
 }
 
 /**
@@ -2294,14 +2523,11 @@ void unpin_hibernation_swap_type(int type)
 {
 	struct swap_info_struct *si;
 
-	spin_lock(&swap_lock);
+	percpu_down_write(&swapon_rwsem);
 	si = swap_type_to_info(type);
-	if (!si) {
-		spin_unlock(&swap_lock);
-		return;
-	}
-	si->flags &= ~SWP_HIBERNATION;
-	spin_unlock(&swap_lock);
+	if (si)
+		si->flags &= ~SWP_HIBERNATION;
+	percpu_up_write(&swapon_rwsem);
 }
 
 /**
@@ -2326,29 +2552,26 @@ int find_hibernation_swap_type(dev_t device, sector_t offset)
 {
 	int type;
 
-	spin_lock(&swap_lock);
+	percpu_down_read(&swapon_rwsem);
 	type = __find_hibernation_swap_type(device, offset);
-	spin_unlock(&swap_lock);
+	percpu_up_read(&swapon_rwsem);
 
 	return type;
 }
 
 int find_first_swap(dev_t *device)
 {
-	int type;
-
-	spin_lock(&swap_lock);
-	for (type = 0; type < nr_swapfiles; type++) {
-		struct swap_info_struct *sis = swap_info[type];
+	int ret = -ENODEV;
+	struct swap_info_struct *si;
 
-		if (!(sis->flags & SWP_WRITEOK))
-			continue;
-		*device = sis->bdev->bd_dev;
-		spin_unlock(&swap_lock);
-		return type;
+	percpu_down_read(&swapon_rwsem);
+	for_each_avail_swap(si) {
+		*device = si->bdev->bd_dev;
+		ret = si->type;
+		break;
 	}
-	spin_unlock(&swap_lock);
-	return -ENODEV;
+	percpu_up_read(&swapon_rwsem);
+	return ret;
 }
 
 /*
@@ -2376,7 +2599,7 @@ unsigned int count_swap_pages(int type, int free)
 {
 	unsigned int n = 0;
 
-	spin_lock(&swap_lock);
+	percpu_down_read(&swapon_rwsem);
 	if ((unsigned int)type < nr_swapfiles) {
 		struct swap_info_struct *sis = swap_info[type];
 
@@ -2388,7 +2611,7 @@ unsigned int count_swap_pages(int type, int free)
 		}
 		spin_unlock(&sis->lock);
 	}
-	spin_unlock(&swap_lock);
+	percpu_up_read(&swapon_rwsem);
 	return n;
 }
 #endif /* CONFIG_HIBERNATION */
@@ -2700,10 +2923,10 @@ static unsigned int find_next_to_unuse(struct swap_info_struct *si,
 	unsigned long swp_tb;
 
 	/*
-	 * No need for swap_lock here: we're just looking
+	 * No need for swapon_rwsem here: we're just looking
 	 * for whether an entry is in use, not modifying it; false
 	 * hits are okay, and sys_swapoff() has already prevented new
-	 * allocations from this area (while holding swap_lock).
+	 * allocations from this area (while holding swapon_rwsem).
 	 */
 	for (i = prev + 1; i < si->max; i++) {
 		swp_tb = swap_table_get(__swap_offset_to_cluster(si, i),
@@ -2816,7 +3039,7 @@ static int try_to_unuse(unsigned int type)
 success:
 	/*
 	 * Make sure that further cleanups after try_to_unuse() returns happen
-	 * after swap_range_free() reduces si->inuse_pages to 0.
+	 * after swap_device_inuse_sub() reduces si->inuse_pages to 0.
 	 */
 	smp_mb();
 	return 0;
@@ -2824,18 +3047,20 @@ static int try_to_unuse(unsigned int type)
 
 /*
  * After a successful try_to_unuse, if no swap is now in use, we know
- * we can empty the mmlist.  swap_lock must be held on entry and exit.
- * Note that mmlist_lock nests inside swap_lock, and an mm must be
+ * we can empty the mmlist. swapon_rwsem must be held on entry and exit.
+ * Note that mmlist_lock nests inside swapon_rwsem, and an mm must be
  * added to the mmlist just after page_duplicate - before would be racy.
  */
 static void drain_mmlist(void)
 {
+	struct swap_info_struct *si;
 	struct list_head *p, *next;
-	unsigned int type;
 
-	for (type = 0; type < nr_swapfiles; type++)
-		if (swap_usage_in_pages(swap_info[type]))
+	for_each_swap(si) {
+		if (swap_usage_in_pages(si))
 			return;
+	}
+
 	spin_lock(&mmlist_lock);
 	list_for_each_safe(p, next, &init_mm.mmlist)
 		list_del_init(p);
@@ -2968,58 +3193,65 @@ static int setup_swap_extents(struct swap_info_struct *sis,
 	return generic_swapfile_activate(sis, swap_file, span);
 }
 
-static void _enable_swap_info(struct swap_info_struct *si)
-{
-	atomic_long_add(si->pages, &nr_swap_pages);
-	total_swap_pages += si->pages;
-
-	assert_spin_locked(&swap_lock);
-
-	plist_add(&si->list, &swap_active_head);
-
-	/* Add back to available list */
-	add_to_avail_list(si, true);
-}
-
 /*
- * Called after the swap device is ready, resurrect its percpu ref, it's now
- * safe to reference it. Add it to the list to expose it to the allocator.
+ * Called after the swap device is ready to be used. Marking it writable and
+ * exposing it to the allocator. Resurrect its percpu ref if it was dead before.
  */
-static void enable_swap_info(struct swap_info_struct *si)
+static void swap_device_enable(struct swap_info_struct *si)
 {
-	percpu_ref_resurrect(&si->users);
-	spin_lock(&swap_lock);
-	spin_lock(&si->lock);
-	_enable_swap_info(si);
-	spin_unlock(&si->lock);
-	spin_unlock(&swap_lock);
-}
+	percpu_down_write(&swapon_rwsem);
+	spin_lock(&swap_queue_update_lock);
+	si->flags |= SWP_WRITEOK;
+	spin_unlock(&swap_queue_update_lock);
+	atomic_long_add(si->pages, &nr_swap_pages);
+	total_swap_pages += si->pages;
+	percpu_up_write(&swapon_rwsem);
 
-static void reinsert_swap_info(struct swap_info_struct *si)
-{
-	spin_lock(&swap_lock);
-	spin_lock(&si->lock);
-	_enable_swap_info(si);
-	spin_unlock(&si->lock);
-	spin_unlock(&swap_lock);
+	add_to_avail_list(si);
 }
 
-/*
- * Called after clearing SWP_WRITEOK, ensures cluster_alloc_range
- * see the updated flags, so there will be no more allocations.
- */
-static void wait_for_allocation(struct swap_info_struct *si)
+static int swap_device_disable(struct swap_info_struct *si)
 {
 	unsigned long offset;
 	unsigned long end = ALIGN(si->max, SWAPFILE_CLUSTER);
 	struct swap_cluster_info *ci;
 
-	BUG_ON(si->flags & SWP_WRITEOK);
+	/*
+	 * If SWP_WRITEOK is not set: another process already disabling it.
+	 * If SWP_HIBERNATION is set: the device is pinned for hibernation.
+	 */
+	percpu_down_write(&swapon_rwsem);
+	if (!(si->flags & SWP_WRITEOK) ||
+	    si->flags & SWP_HIBERNATION) {
+		percpu_up_write(&swapon_rwsem);
+		return -EBUSY;
+	}
+
+	if (security_vm_enough_memory_mm(current->mm, si->pages)) {
+		percpu_up_write(&swapon_rwsem);
+		return -ENOMEM;
+	}
+	vm_unacct_memory(si->pages);
+
+	spin_lock(&swap_queue_update_lock);
+	si->flags &= ~SWP_WRITEOK;
+	spin_unlock(&swap_queue_update_lock);
+	total_swap_pages -= si->pages;
+	atomic_long_sub(si->pages, &nr_swap_pages);
+	del_from_avail_list(si, true);
+	percpu_up_write(&swapon_rwsem);
 
+	/*
+	 * Swap allocator doesn't touch si lock, so looping through all
+	 * ci locks ensures __swap_cluster_alloc_entries sees the
+	 * updated flags, and no more allocations will occur.
+	 */
 	for (offset = 0; offset < end; offset += SWAPFILE_CLUSTER) {
 		ci = swap_cluster_lock(si, offset);
 		swap_cluster_unlock(ci);
 	}
+
+	return 0;
 }
 
 static void free_swap_cluster_info(struct swap_cluster_info *cluster_info,
@@ -3043,28 +3275,6 @@ static void free_swap_cluster_info(struct swap_cluster_info *cluster_info,
 	kvfree(cluster_info);
 }
 
-/*
- * Called after swap device's reference count is dead, so
- * neither scan nor allocation will use it.
- */
-static void flush_percpu_swap_cluster(struct swap_info_struct *si)
-{
-	int cpu, i;
-	struct swap_info_struct **pcp_si;
-
-	for_each_possible_cpu(cpu) {
-		pcp_si = per_cpu_ptr(percpu_swap_cluster.si, cpu);
-		/*
-		 * Invalidate the percpu swap cluster cache, si->users
-		 * is dead, so no new user will point to it, just flush
-		 * any existing user.
-		 */
-		for (i = 0; i < SWAP_NR_ORDERS; i++)
-			cmpxchg(&pcp_si[i], si, NULL);
-	}
-}
-
-
 SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
 {
 	struct swap_info_struct *p = NULL;
@@ -3086,44 +3296,22 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
 		return PTR_ERR(victim);
 
 	mapping = victim->f_mapping;
-	spin_lock(&swap_lock);
-	plist_for_each_entry(p, &swap_active_head, list) {
-		if (p->flags & SWP_WRITEOK) {
-			if (p->swap_file->f_mapping == mapping) {
-				found = 1;
-				break;
-			}
+	percpu_down_read(&swapon_rwsem);
+	for_each_avail_swap(p) {
+		if (p->swap_file->f_mapping == mapping) {
+			found = 1;
+			break;
 		}
 	}
-	if (!found) {
-		err = -EINVAL;
-		spin_unlock(&swap_lock);
-		goto out_dput;
-	}
-
-	/* Refuse swapoff while the device is pinned for hibernation */
-	if (p->flags & SWP_HIBERNATION) {
-		err = -EBUSY;
-		spin_unlock(&swap_lock);
-		goto out_dput;
-	}
+	percpu_up_read(&swapon_rwsem);
+	filp_close(victim, NULL);
 
-	if (!security_vm_enough_memory_mm(current->mm, p->pages))
-		vm_unacct_memory(p->pages);
-	else {
-		err = -ENOMEM;
-		spin_unlock(&swap_lock);
-		goto out_dput;
-	}
-	spin_lock(&p->lock);
-	del_from_avail_list(p, true);
-	plist_del(&p->list, &swap_active_head);
-	atomic_long_sub(p->pages, &nr_swap_pages);
-	total_swap_pages -= p->pages;
-	spin_unlock(&p->lock);
-	spin_unlock(&swap_lock);
+	if (!found)
+		return -EINVAL;
 
-	wait_for_allocation(p);
+	err = swap_device_disable(p);
+	if (err)
+		return err;
 
 	set_current_oom_origin();
 	err = try_to_unuse(p->type);
@@ -3131,10 +3319,13 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
 
 	if (err) {
 		/* re-insert swap space back into swap_list */
-		reinsert_swap_info(p);
-		goto out_dput;
+		swap_device_enable(p);
+		return err;
 	}
 
+	percpu_down_write(&swapon_rwsem);
+	swap_queue_del(p);
+
 	/*
 	 * Wait for swap operations protected by get/put_swap_device()
 	 * to complete.  Because of synchronize_rcu() here, all swap
@@ -3149,15 +3340,12 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
 
 	flush_work(&p->discard_work);
 	flush_work(&p->reclaim_work);
-	flush_percpu_swap_cluster(p);
 
 	destroy_swap_extents(p, p->swap_file);
 
 	if (!(p->flags & SWP_SOLIDSTATE))
 		atomic_dec(&nr_rotate_swap);
 
-	mutex_lock(&swapon_mutex);
-	spin_lock(&swap_lock);
 	spin_lock(&p->lock);
 	drain_mmlist();
 
@@ -3168,10 +3356,11 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
 	p->max = 0;
 	p->cluster_info = NULL;
 	spin_unlock(&p->lock);
-	spin_unlock(&swap_lock);
+	percpu_up_write(&swapon_rwsem);
 	arch_swap_invalidate_area(p->type);
 	zswap_swapoff(p->type);
-	mutex_unlock(&swapon_mutex);
+	free_percpu(p->percpu_cluster);
+	p->percpu_cluster = NULL;
 	kfree(p->global_cluster);
 	p->global_cluster = NULL;
 	free_swap_cluster_info(cluster_info, maxpages);
@@ -3187,18 +3376,19 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
 	 * Clear the SWP_USED flag after all resources are freed so that swapon
 	 * can reuse this swap_info in alloc_swap_info() safely.  It is ok to
 	 * not hold p->lock after we cleared its SWP_WRITEOK.
+	 *
+	 * The write lock ensures the flag clear is visible to lockless
+	 * readers of swap_type_to_info() before alloc_swap_info() reuses
+	 * this slot.
 	 */
-	spin_lock(&swap_lock);
+	percpu_down_write(&swapon_rwsem);
 	p->flags = 0;
-	spin_unlock(&swap_lock);
+	percpu_up_write(&swapon_rwsem);
 
-	err = 0;
 	atomic_inc(&proc_poll_event);
 	wake_up_interruptible(&proc_poll_wait);
 
-out_dput:
-	filp_close(victim, NULL);
-	return err;
+	return 0;
 }
 
 #ifdef CONFIG_PROC_FS
@@ -3216,20 +3406,18 @@ static __poll_t swaps_poll(struct file *file, poll_table *wait)
 	return EPOLLIN | EPOLLRDNORM;
 }
 
-/* iterator */
 static void *swap_start(struct seq_file *swap, loff_t *pos)
 {
 	struct swap_info_struct *si;
-	int type;
 	loff_t l = *pos;
 
-	mutex_lock(&swapon_mutex);
+	percpu_down_read(&swapon_rwsem);
 
 	if (!l)
 		return SEQ_START_TOKEN;
 
-	for (type = 0; (si = swap_type_to_info(type)); type++) {
-		if (!(si->swap_file))
+	for_each_swap(si) {
+		if (!si->swap_file)
 			continue;
 		if (!--l)
 			return si;
@@ -3250,7 +3438,7 @@ static void *swap_next(struct seq_file *swap, void *v, loff_t *pos)
 
 	++(*pos);
 	for (; (si = swap_type_to_info(type)); type++) {
-		if (!(si->swap_file))
+		if (!(si->flags & SWP_USED) || !(si->swap_file))
 			continue;
 		return si;
 	}
@@ -3260,7 +3448,7 @@ static void *swap_next(struct seq_file *swap, void *v, loff_t *pos)
 
 static void swap_stop(struct seq_file *swap, void *v)
 {
-	mutex_unlock(&swapon_mutex);
+	percpu_up_read(&swapon_rwsem);
 }
 
 static int swap_show(struct seq_file *swap, void *v)
@@ -3353,13 +3541,13 @@ static struct swap_info_struct *alloc_swap_info(void)
 		return ERR_PTR(-ENOMEM);
 	}
 
-	spin_lock(&swap_lock);
+	percpu_down_write(&swapon_rwsem);
 	for (type = 0; type < nr_swapfiles; type++) {
 		if (!(swap_info[type]->flags & SWP_USED))
 			break;
 	}
 	if (type >= MAX_SWAPFILES) {
-		spin_unlock(&swap_lock);
+		percpu_up_write(&swapon_rwsem);
 		percpu_ref_exit(&p->users);
 		kvfree(p);
 		return ERR_PTR(-EPERM);
@@ -3381,10 +3569,8 @@ static struct swap_info_struct *alloc_swap_info(void)
 		 */
 	}
 	p->swap_extent_root = RB_ROOT;
-	plist_node_init(&p->list, 0);
-	plist_node_init(&p->avail_list, 0);
 	p->flags = SWP_USED;
-	spin_unlock(&swap_lock);
+	percpu_up_write(&swapon_rwsem);
 	if (defer) {
 		percpu_ref_exit(&defer->users);
 		kvfree(defer);
@@ -3453,9 +3639,8 @@ __weak unsigned long arch_max_swapfile_size(void)
 	return generic_max_swapfile_size();
 }
 
-static unsigned long read_swap_header(struct swap_info_struct *si,
-					union swap_header *swap_header,
-					struct inode *inode)
+static unsigned long read_swap_header(union swap_header *swap_header,
+				      struct inode *inode)
 {
 	int i;
 	unsigned long maxpages;
@@ -3522,7 +3707,7 @@ static int setup_swap_clusters_info(struct swap_info_struct *si,
 {
 	unsigned long nr_clusters = DIV_ROUND_UP(maxpages, SWAPFILE_CLUSTER);
 	struct swap_cluster_info *cluster_info;
-	int err = -ENOMEM;
+	int cpu, err = -ENOMEM;
 	unsigned long i;
 
 	cluster_info = kvzalloc_objs(*cluster_info, nr_clusters);
@@ -3532,13 +3717,26 @@ static int setup_swap_clusters_info(struct swap_info_struct *si,
 	for (i = 0; i < nr_clusters; i++)
 		spin_lock_init(&cluster_info[i].lock);
 
-	if (!(si->flags & SWP_SOLIDSTATE)) {
+	if (si->flags & SWP_SOLIDSTATE) {
+		si->percpu_cluster = alloc_percpu(struct percpu_cluster);
+		if (!si->percpu_cluster)
+			goto err;
+
+		for_each_possible_cpu(cpu) {
+			struct percpu_cluster *cluster;
+
+			cluster = per_cpu_ptr(si->percpu_cluster, cpu);
+			for (i = 0; i < SWAP_NR_ORDERS; i++)
+				cluster->next[i] = SWAP_ENTRY_INVALID;
+			local_lock_init(&cluster->lock);
+		}
+	} else {
 		si->global_cluster = kmalloc_obj(*si->global_cluster);
 		if (!si->global_cluster)
 			goto err;
 		for (i = 0; i < SWAP_NR_ORDERS; i++)
 			si->global_cluster->next[i] = SWAP_ENTRY_INVALID;
-		spin_lock_init(&si->global_cluster_lock);
+		spin_lock_init(&si->global_cluster->lock);
 	}
 
 	/*
@@ -3621,7 +3819,7 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
 	/*
 	 * Allocate or reuse existing !SWP_USED swap_info. The returned
 	 * si will stay in a dying status, so nothing will access its content
-	 * until enable_swap_info resurrects its percpu ref and expose it.
+	 * until swap_device_enable resurrects its percpu ref and expose it.
 	 */
 	si = alloc_swap_info();
 	if (IS_ERR(si))
@@ -3679,7 +3877,7 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
 	}
 	swap_header = kmap_local_folio(folio, 0);
 
-	maxpages = read_swap_header(si, swap_header, inode);
+	maxpages = read_swap_header(swap_header, inode);
 	if (unlikely(!maxpages)) {
 		error = -EINVAL;
 		goto bad_swap_unlock_inode;
@@ -3700,11 +3898,6 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
 
 	maxpages = si->max;
 
-	/* Set up the swap cluster info */
-	error = setup_swap_clusters_info(si, swap_header, maxpages);
-	if (error)
-		goto bad_swap_unlock_inode;
-
 	if (si->bdev && bdev_stable_writes(si->bdev))
 		si->flags |= SWP_STABLE_WRITES;
 
@@ -3718,6 +3911,15 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
 		inced_nr_rotate_swap = true;
 	}
 
+	/*
+	 * Set up the swap cluster info. This must run after SWP_SOLIDSTATE
+	 * is determined above, as it decides whether to allocate the per-CPU
+	 * cluster (solid state) or the global cluster (rotational).
+	 */
+	error = setup_swap_clusters_info(si, swap_header, maxpages);
+	if (error)
+		goto bad_swap_unlock_inode;
+
 	if ((swap_flags & SWAP_FLAG_DISCARD) &&
 	    si->bdev && bdev_max_discard_sectors(si->bdev)) {
 		/*
@@ -3764,22 +3966,22 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
 		goto free_swap_zswap;
 	}
 
-	mutex_lock(&swapon_mutex);
 	prio = DEF_SWAP_PRIO;
 	if (swap_flags & SWAP_FLAG_PREFER)
 		prio = swap_flags & SWAP_FLAG_PRIO_MASK;
 
-	/*
-	 * The plist prio is negated because plist ordering is
-	 * low-to-high, while swap ordering is high-to-low
-	 */
 	si->prio = prio;
-	si->list.prio = -si->prio;
-	si->avail_list.prio = -si->prio;
 	si->swap_file = swap_file;
 
 	/* Sets SWP_WRITEOK, resurrect the percpu ref, expose the swap device */
-	enable_swap_info(si);
+	percpu_ref_resurrect(&si->users);
+	percpu_down_write(&swapon_rwsem);
+	error = swap_queue_add(si);
+	percpu_up_write(&swapon_rwsem);
+	if (error)
+		goto free_swap_zswap;
+
+	swap_device_enable(si);
 
 	pr_info("Adding %uk swap on %s.  Priority:%d extents:%d across:%lluk %s%s%s%s\n",
 		K(si->pages), name->name, si->prio, nr_extents,
@@ -3789,7 +3991,6 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
 		(si->flags & SWP_AREA_DISCARD) ? "s" : "",
 		(si->flags & SWP_PAGE_DISCARD) ? "c" : "");
 
-	mutex_unlock(&swapon_mutex);
 	atomic_inc(&proc_poll_event);
 	wake_up_interruptible(&proc_poll_wait);
 
@@ -3800,6 +4001,8 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
 bad_swap_unlock_inode:
 	inode_unlock(inode);
 bad_swap:
+	free_percpu(si->percpu_cluster);
+	si->percpu_cluster = NULL;
 	kfree(si->global_cluster);
 	si->global_cluster = NULL;
 	inode = NULL;
@@ -3810,9 +4013,9 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
 	 * Clear the SWP_USED flag after all resources are freed so
 	 * alloc_swap_info can reuse this si safely.
 	 */
-	spin_lock(&swap_lock);
+	percpu_down_write(&swapon_rwsem);
 	si->flags = 0;
-	spin_unlock(&swap_lock);
+	percpu_up_write(&swapon_rwsem);
 	if (inced_nr_rotate_swap)
 		atomic_dec(&nr_rotate_swap);
 	if (swap_file)
@@ -3827,19 +4030,17 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
 
 void si_swapinfo(struct sysinfo *val)
 {
-	unsigned int type;
+	struct swap_info_struct *si;
 	unsigned long nr_to_be_unused = 0;
 
-	spin_lock(&swap_lock);
-	for (type = 0; type < nr_swapfiles; type++) {
-		struct swap_info_struct *si = swap_info[type];
-
-		if ((si->flags & SWP_USED) && !(si->flags & SWP_WRITEOK))
+	percpu_down_read(&swapon_rwsem);
+	for_each_swap(si) {
+		if (!(si->flags & SWP_WRITEOK))
 			nr_to_be_unused += swap_usage_in_pages(si);
 	}
 	val->freeswap = atomic_long_read(&nr_swap_pages) + nr_to_be_unused;
 	val->totalswap = total_swap_pages + nr_to_be_unused;
-	spin_unlock(&swap_lock);
+	percpu_up_read(&swapon_rwsem);
 }
 
 /*
@@ -3879,7 +4080,7 @@ int swap_dup_entry_direct(swp_entry_t entry)
 #if defined(CONFIG_MEMCG) && defined(CONFIG_BLK_CGROUP)
 static bool __has_usable_swap(void)
 {
-	return !plist_head_empty(&swap_active_head);
+	return READ_ONCE(total_swap_pages) > 0;
 }
 
 void __folio_throttle_swaprate(struct folio *folio, gfp_t gfp)
@@ -3902,14 +4103,14 @@ void __folio_throttle_swaprate(struct folio *folio, gfp_t gfp)
 	if (current->throttle_disk)
 		return;
 
-	spin_lock(&swap_avail_lock);
-	plist_for_each_entry(si, &swap_avail_head, avail_list) {
+	percpu_down_read(&swapon_rwsem);
+	for_each_avail_swap(si) {
 		if (si->bdev) {
 			blkcg_schedule_throttle(si->bdev->bd_disk, true);
 			break;
 		}
 	}
-	spin_unlock(&swap_avail_lock);
+	percpu_up_read(&swapon_rwsem);
 }
 #endif

3/2

2026/07/14 00:43

llm

patch-evaluator

Model:

gemini-3.1-pro-preview

Args:

null

Results:

null

Tokens:

							input: 25861
							output: 27
							thoughts: 420

Thoughts:

**My Assessment of the Swap Subsystem Patch**

Okay, so I've got this patch – a pretty significant one, it looks like – that's fundamentally changing the Linux kernel's swap subsystem. It's replacing the current `plist`-based queue for swap devices with a custom round-robin implementation. Immediately, I can tell this is going to touch core memory management, which is always a red flag for potential issues. The fact that it's rewriting so much of `mm/swapfile.c` – and then touching all those other header and supporting files – `include/linux/swap.h`, `include/linux/plist.h`, `lib/plist.c`, and `mm/swap.h` – tells me this isn't just a minor tweak. It's a genuine architectural change. Removing `plist_requeue` is a clue, too.

This introduction of new structures like `swap_queue_readers`, `swap_queue`, `swap_queue_len`, `swap_prio_ring`, along with the use of `swapon_rwsem` to replace the old locking mechanisms, indicates a re-thinking of how swap space is allocated and managed. This is a functional change, a *major* functional change to the very core of swap allocation logic.

My immediate thought is, "This needs some serious fuzzing." Given the potential for memory corruption, race conditions, or unexpected behavior when interacting with other kernel subsystems, I want to zero in on the key functions that have been modified. Let's see:

* `swap_alloc_entry` (and its replacements of `swap_alloc_fast` and `swap_alloc_slow`) – This is my primary area of interest, it's where the swap entry allocation actually happens.
* `folio_alloc_swap` – another critical entry point.
* The queue manipulation functions: `swap_queue_add`, `swap_queue_del`, `swap_queue_get_device` - these are the points of interactions with the new structures.
* `swap_device_enable`, `swap_device_disable` - the points of device change are definitely worth checking.
* `sys_swapon`, `sys_swapoff` – The user-space interface is always a good target.
* The `swap_device_inuse_*` functions – these are likely the place where the old locking was, this is also crucial.
* `add_to_avail_list`, `del_from_avail_list` – this is a good place to look at the new list manipulation.
* The new queue mask functions `swap_queue_mask`, `swap_queue_unmask`
* `cluster_alloc_swap_entry`

Alright, I'll need to figure out any relevant configuration options, as well. Since this is in `mm/swapfile.c`, it's almost certainly controlled by `CONFIG_SWAP`. I should check the `mm/Makefile` to confirm, but that's a given. I am eager to get started.

4/2

2026/07/14 00:43

tool

grepper

Args:

{
  "Expression": "swapfile.o",
  "PathPrefix": "mm/Makefile"
}

Results:

{
  "Output": "mm/Makefile=76=endif\nmm/Makefile-77-\nmm/Makefile:78:obj-$(CONFIG_SWAP)\t+= page_io.o swap_state.o swapfile.o\nmm/Makefile-79-obj-$(CONFIG_ZSWAP)\t+= zswap.o\n"
}

5/2

2026/07/14 00:43

llm

patch-evaluator

Model:

gemini-3.1-pro-preview

Args:

null

Results:

null

Tokens:

							input: 27059
							output: 56
							thoughts: 230

Thoughts:

**My Initial Assessment of the Swap Configuration Dependency**

Alright, it's clear that `CONFIG_SWAP` is a prerequisite here. That's the baseline. Now, I need to understand the core logic this change is touching. The most relevant functions seem to be centered around queuing, device management, and allocation. Specifically, I'm thinking about `swap_queue_add`, `swap_queue_del`, `swap_queue_get_device`, `swap_device_enable`, `swap_device_disable`, `swap_alloc_entry`, `swap_device_inuse_add`, `swap_device_inuse_sub`, `add_to_avail_list`, `del_from_avail_list`, `swap_queue_mask`, and `swap_queue_unmask`. These all seem to reside within `mm/swapfile.c`, which makes sense.

It looks like the change interacts with `CONFIG_HIBERNATION` too, also in `mm/swapfile.c`, and that's bringing in functions like `pin_hibernation_swap_type`, `unpin_hibernation_swap_type`, `find_hibernation_swap_type`, `find_first_swap`, and `count_swap_pages`. While hibernation is involved, it appears the central focus is really the mechanics of swap allocation itself. This reinforces my initial hunch about the critical functions involved, I need to focus on this part to proceed. Let's get down to specifics, I need to pull up this list of functions again.

6/2

2026/07/14 00:44

tool

grepper

Args:

{
  "Expression": "^(static )?(int|void|bool|struct swap_info_struct \\*) [a-zA-Z0-9_]+\\(",
  "PathPrefix": "mm/swapfile.c"
}

Results:

{
  "Output": "mm/swapfile.c-50-\nmm/swapfile.c:51:static void swap_device_inuse_add(struct swap_info_struct *si,\nmm/swapfile.c-52-\t\t\t\t  unsigned int nr_entries);\nmm/swapfile.c:53:static bool folio_swapcache_freeable(struct folio *folio);\nmm/swapfile.c:54:static void move_cluster(struct swap_info_struct *si,\nmm/swapfile.c-55-\t\t\t struct swap_cluster_info *ci, struct list_head *list,\nmm/swapfile.c-56-\t\t\t enum swap_cluster_flags new_flags);\nmm/swapfile.c:57:static bool get_swap_device_info(struct swap_info_struct *si);\nmm/swapfile.c-58-\n--\nmm/swapfile.c=208=static struct swap_queue_reader __percpu *swap_queue_prealloc_readers(int nr_rings, gfp_t gfp)\n--\nmm/swapfile.c-219-\nmm/swapfile.c:220:static void swap_queue_install_readers(struct swap_queue_reader __percpu *readers)\nmm/swapfile.c-221-{\n--\nmm/swapfile.c=242=static struct swap_info_struct *swap_queue_get_device(long nr_alloc, int nr_iter)\n--\nmm/swapfile.c-285-\nmm/swapfile.c:286:static bool swap_queue_find(struct swap_info_struct *si,\nmm/swapfile.c-287-\t\t\t    unsigned int *ring_idx, unsigned int *dev_idx)\n--\nmm/swapfile.c-309-\nmm/swapfile.c:310:static void swap_queue_mask(struct swap_info_struct *si)\nmm/swapfile.c-311-{\n--\nmm/swapfile.c-319-\nmm/swapfile.c:320:static void swap_queue_unmask(struct swap_info_struct *si)\nmm/swapfile.c-321-{\n--\nmm/swapfile.c-329-\nmm/swapfile.c:330:static int swap_queue_add(struct swap_info_struct *si)\nmm/swapfile.c-331-{\n--\nmm/swapfile.c-405-\nmm/swapfile.c:406:static void swap_queue_del(struct swap_info_struct *si)\nmm/swapfile.c-407-{\n--\nmm/swapfile.c=489=static long swap_usage_in_pages(struct swap_info_struct *si)\n--\nmm/swapfile.c-497- */\nmm/swapfile.c:498:static void swap_alloc_lock_device(struct swap_info_struct *si)\nmm/swapfile.c-499-{\n--\nmm/swapfile.c-505-\nmm/swapfile.c:506:static void swap_alloc_unlock_device(struct swap_info_struct *si)\nmm/swapfile.c-507-{\n--\nmm/swapfile.c-523-\nmm/swapfile.c:524:static bool swap_only_has_cache(struct swap_cluster_info *ci,\nmm/swapfile.c-525-\t\t\t\tunsigned long offset, int nr_pages)\n--\nmm/swapfile.c-545- */\nmm/swapfile.c:546:static int __try_to_reclaim_swap(struct swap_info_struct *si,\nmm/swapfile.c-547-\t\t\t\t unsigned long offset, unsigned long flags)\n--\nmm/swapfile.c=616=static inline struct swap_extent *next_se(struct swap_extent *se)\n--\nmm/swapfile.c-625- */\nmm/swapfile.c:626:static int discard_swap(struct swap_info_struct *si)\nmm/swapfile.c-627-{\n--\nmm/swapfile.c=679=sector_t swap_folio_sector(struct folio *folio)\n--\nmm/swapfile.c-695- */\nmm/swapfile.c:696:static void discard_swap_cluster(struct swap_info_struct *si,\nmm/swapfile.c-697-\t\t\t\t pgoff_t start_page, pgoff_t nr_pages)\n--\nmm/swapfile.c=755=static inline unsigned int cluster_offset(struct swap_info_struct *si,\n--\nmm/swapfile.c-760-\nmm/swapfile.c:761:static void swap_cluster_free_table_folio_rcu_cb(struct rcu_head *head)\nmm/swapfile.c-762-{\n--\nmm/swapfile.c-768-\nmm/swapfile.c:769:static void swap_cluster_free_table(struct swap_cluster_info *ci)\nmm/swapfile.c-770-{\n--\nmm/swapfile.c-796-\nmm/swapfile.c:797:static int swap_cluster_alloc_table(struct swap_cluster_info *ci, gfp_t gfp)\nmm/swapfile.c-798-{\n--\nmm/swapfile.c-845- */\nmm/swapfile.c:846:static void swap_cluster_assert_empty(struct swap_cluster_info *ci,\nmm/swapfile.c-847-\t\t\t\t      unsigned int ci_off, unsigned int nr,\n--\nmm/swapfile.c=875=swap_cluster_populate(struct swap_info_struct *si,\n--\nmm/swapfile.c-923-\nmm/swapfile.c:924:static void move_cluster(struct swap_info_struct *si,\nmm/swapfile.c-925-\t\t\t struct swap_cluster_info *ci, struct list_head *list,\n--\nmm/swapfile.c-942-/* Add a cluster to discard list and schedule it to do discard */\nmm/swapfile.c:943:static void swap_cluster_schedule_discard(struct swap_info_struct *si,\nmm/swapfile.c-944-\t\tstruct swap_cluster_info *ci)\n--\nmm/swapfile.c-950-\nmm/swapfile.c:951:static void __free_cluster(struct swap_info_struct *si, struct swap_cluster_info *ci)\nmm/swapfile.c-952-{\n--\nmm/swapfile.c=968=static struct swap_cluster_info *isolate_lock_cluster(\n--\nmm/swapfile.c-1007- */\nmm/swapfile.c:1008:static bool swap_do_scheduled_discard(struct swap_info_struct *si)\nmm/swapfile.c-1009-{\n--\nmm/swapfile.c-1042-\nmm/swapfile.c:1043:static void swap_discard_work(struct work_struct *work)\nmm/swapfile.c-1044-{\n--\nmm/swapfile.c-1051-\nmm/swapfile.c:1052:static void swap_users_ref_free(struct percpu_ref *ref)\nmm/swapfile.c-1053-{\n--\nmm/swapfile.c-1063- */\nmm/swapfile.c:1064:static void free_cluster(struct swap_info_struct *si, struct swap_cluster_info *ci)\nmm/swapfile.c-1065-{\n--\nmm/swapfile.c-1087- */\nmm/swapfile.c:1088:static void partial_free_cluster(struct swap_info_struct *si,\nmm/swapfile.c-1089-\t\t\t\t struct swap_cluster_info *ci)\n--\nmm/swapfile.c-1103- */\nmm/swapfile.c:1104:static void relocate_cluster(struct swap_info_struct *si,\nmm/swapfile.c-1105-\t\t\t     struct swap_cluster_info *ci)\n--\nmm/swapfile.c-1131- */\nmm/swapfile.c:1132:static int swap_cluster_setup_bad_slot(struct swap_info_struct *si,\nmm/swapfile.c-1133-\t\t\t\t       struct swap_cluster_info *cluster_info,\n--\nmm/swapfile.c-1181- */\nmm/swapfile.c:1182:static bool cluster_reclaim_range(struct swap_info_struct *si,\nmm/swapfile.c-1183-\t\t\t\t  struct swap_cluster_info *ci,\n--\nmm/swapfile.c-1228-\nmm/swapfile.c:1229:static bool cluster_scan_range(struct swap_info_struct *si,\nmm/swapfile.c-1230-\t\t\t       struct swap_cluster_info *ci,\n--\nmm/swapfile.c-1255-\nmm/swapfile.c:1256:static bool __swap_cluster_alloc_entries(struct swap_info_struct *si,\nmm/swapfile.c-1257-\t\t\t\t\t struct swap_cluster_info *ci,\n--\nmm/swapfile.c=1358=static unsigned int alloc_swap_scan_list(struct swap_info_struct *si,\n--\nmm/swapfile.c-1379-\nmm/swapfile.c:1380:static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool force)\nmm/swapfile.c-1381-{\n--\nmm/swapfile.c-1421-\nmm/swapfile.c:1422:static void swap_reclaim_work(struct work_struct *work)\nmm/swapfile.c-1423-{\n--\nmm/swapfile.c=1435=static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,\n--\nmm/swapfile.c-1537-/* SWAP_USAGE_OFFLIST_BIT can only be set by this helper. */\nmm/swapfile.c:1538:static void del_from_avail_list(struct swap_info_struct *si, bool swapoff)\nmm/swapfile.c-1539-{\n--\nmm/swapfile.c-1565-/* SWAP_USAGE_OFFLIST_BIT can only be cleared by this helper. */\nmm/swapfile.c:1566:static void add_to_avail_list(struct swap_info_struct *si)\nmm/swapfile.c-1567-{\n--\nmm/swapfile.c-1610- */\nmm/swapfile.c:1611:static void swap_device_inuse_add(struct swap_info_struct *si,\nmm/swapfile.c-1612-\t\t\t\t  unsigned int nr_entries)\n--\nmm/swapfile.c-1629-\nmm/swapfile.c:1630:static void swap_device_inuse_sub(struct swap_info_struct *si, unsigned long offset,\nmm/swapfile.c-1631-\t\t\t\t  unsigned int nr_entries)\n--\nmm/swapfile.c-1668-\nmm/swapfile.c:1669:static bool get_swap_device_info(struct swap_info_struct *si)\nmm/swapfile.c-1670-{\n--\nmm/swapfile.c-1684-\nmm/swapfile.c:1685:static int swap_alloc_entry(struct folio *folio)\nmm/swapfile.c-1686-{\n--\nmm/swapfile.c-1720-\nmm/swapfile.c:1721:static int swap_extend_table_alloc(struct swap_info_struct *si,\nmm/swapfile.c-1722-\t\t\t\t   struct swap_cluster_info *ci,\n--\nmm/swapfile.c-1756-\nmm/swapfile.c:1757:int swap_retry_table_alloc(swp_entry_t entry, gfp_t gfp)\nmm/swapfile.c-1758-{\n--\nmm/swapfile.c-1774-\nmm/swapfile.c:1775:static void swap_extend_table_try_free(struct swap_cluster_info *ci)\nmm/swapfile.c-1776-{\n--\nmm/swapfile.c-1794-/* Decrease the swap count of one slot, without freeing it */\nmm/swapfile.c:1795:static void __swap_cluster_put_entry(struct swap_cluster_info *ci,\nmm/swapfile.c-1796-\t\t\t\t    unsigned int ci_off)\n--\nmm/swapfile.c-1845- */\nmm/swapfile.c:1846:static void swap_put_entries_cluster(struct swap_info_struct *si,\nmm/swapfile.c-1847-\t\t\t\t     pgoff_t offset, int nr,\n--\nmm/swapfile.c-1902-/* Increase the swap count of one slot. */\nmm/swapfile.c:1903:static int __swap_cluster_dup_entry(struct swap_cluster_info *ci,\nmm/swapfile.c-1904-\t\t\t\t    unsigned int ci_off)\n--\nmm/swapfile.c-1952- */\nmm/swapfile.c:1953:static int swap_dup_entries_cluster(struct swap_info_struct *si,\nmm/swapfile.c-1954-\t\t\t\t    pgoff_t offset, int nr)\n--\nmm/swapfile.c-1997- */\nmm/swapfile.c:1998:int folio_alloc_swap(struct folio *folio)\nmm/swapfile.c-1999-{\n--\nmm/swapfile.c-2053- */\nmm/swapfile.c:2054:int folio_dup_swap(struct folio *folio, struct page *page)\nmm/swapfile.c-2055-{\n--\nmm/swapfile.c-2079- */\nmm/swapfile.c:2080:void folio_put_swap(struct folio *folio, struct page *page)\nmm/swapfile.c-2081-{\n--\nmm/swapfile.c=2134=struct swap_info_struct *get_swap_device(swp_entry_t entry)\n--\nmm/swapfile.c-2164- */\nmm/swapfile.c:2165:void __swap_cluster_free_entries(struct swap_info_struct *si,\nmm/swapfile.c-2166-\t\t\t\t struct swap_cluster_info *ci,\n--\nmm/swapfile.c-2215-\nmm/swapfile.c:2216:int __swap_count(swp_entry_t entry)\nmm/swapfile.c-2217-{\n--\nmm/swapfile.c-2228- */\nmm/swapfile.c:2229:bool swap_entry_swapped(struct swap_info_struct *si, swp_entry_t entry)\nmm/swapfile.c-2230-{\n--\nmm/swapfile.c-2245- */\nmm/swapfile.c:2246:int swp_swapcount(swp_entry_t entry)\nmm/swapfile.c-2247-{\n--\nmm/swapfile.c-2278- */\nmm/swapfile.c:2279:static bool folio_maybe_swapped(struct folio *folio)\nmm/swapfile.c-2280-{\n--\nmm/swapfile.c-2307-\nmm/swapfile.c:2308:static bool folio_swapcache_freeable(struct folio *folio)\nmm/swapfile.c-2309-{\n--\nmm/swapfile.c-2346- */\nmm/swapfile.c:2347:bool folio_free_swap(struct folio *folio)\nmm/swapfile.c-2348-{\n--\nmm/swapfile.c-2371- */\nmm/swapfile.c:2372:void swap_put_entries_direct(swp_entry_t entry, int nr)\nmm/swapfile.c-2373-{\n--\nmm/swapfile.c=2407=swp_entry_t swap_alloc_hibernation_slot(int type)\n--\nmm/swapfile.c-2429- */\nmm/swapfile.c:2430:void swap_free_hibernation_slot(swp_entry_t entry)\nmm/swapfile.c-2431-{\n--\nmm/swapfile.c-2444-\nmm/swapfile.c:2445:static int __find_hibernation_swap_type(dev_t device, sector_t offset)\nmm/swapfile.c-2446-{\n--\nmm/swapfile.c-2478- */\nmm/swapfile.c:2479:int pin_hibernation_swap_type(dev_t device, sector_t offset)\nmm/swapfile.c-2480-{\n--\nmm/swapfile.c-2521- */\nmm/swapfile.c:2522:void unpin_hibernation_swap_type(int type)\nmm/swapfile.c-2523-{\n--\nmm/swapfile.c-2550- */\nmm/swapfile.c:2551:int find_hibernation_swap_type(dev_t device, sector_t offset)\nmm/swapfile.c-2552-{\n--\nmm/swapfile.c-2561-\nmm/swapfile.c:2562:int find_first_swap(dev_t *device)\nmm/swapfile.c-2563-{\n--\nmm/swapfile.c=2619=static inline int pte_same_as_swp(pte_t pte, pte_t swp_pte)\n--\nmm/swapfile.c-2628- */\nmm/swapfile.c:2629:static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd,\nmm/swapfile.c-2630-\t\tunsigned long addr, swp_entry_t entry, struct folio *folio)\n--\nmm/swapfile.c-2741-\nmm/swapfile.c:2742:static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd,\nmm/swapfile.c-2743-\t\t\tunsigned long addr, unsigned long end,\n--\nmm/swapfile.c=2850=static inline int unuse_p4d_range(struct vm_area_struct *vma, pgd_t *pgd,\n--\nmm/swapfile.c-2869-\nmm/swapfile.c:2870:static int unuse_vma(struct vm_area_struct *vma, unsigned int type)\nmm/swapfile.c-2871-{\n--\nmm/swapfile.c-2890-\nmm/swapfile.c:2891:static int unuse_mm(struct mm_struct *mm, unsigned int type)\nmm/swapfile.c-2892-{\n--\nmm/swapfile.c=2919=static unsigned int find_next_to_unuse(struct swap_info_struct *si,\n--\nmm/swapfile.c-2945-\nmm/swapfile.c:2946:static int try_to_unuse(unsigned int type)\nmm/swapfile.c-2947-{\n--\nmm/swapfile.c-3053- */\nmm/swapfile.c:3054:static void drain_mmlist(void)\nmm/swapfile.c-3055-{\n--\nmm/swapfile.c-3072- */\nmm/swapfile.c:3073:static void destroy_swap_extents(struct swap_info_struct *sis,\nmm/swapfile.c-3074-\t\t\t\t struct file *swap_file)\n--\nmm/swapfile.c=3138=EXPORT_SYMBOL_GPL(add_swap_extent);\n--\nmm/swapfile.c-3166- */\nmm/swapfile.c:3167:static int setup_swap_extents(struct swap_info_struct *sis,\nmm/swapfile.c-3168-\t\t\t      struct file *swap_file, sector_t *span)\n--\nmm/swapfile.c-3199- */\nmm/swapfile.c:3200:static void swap_device_enable(struct swap_info_struct *si)\nmm/swapfile.c-3201-{\n--\nmm/swapfile.c-3212-\nmm/swapfile.c:3213:static int swap_device_disable(struct swap_info_struct *si)\nmm/swapfile.c-3214-{\n--\nmm/swapfile.c-3256-\nmm/swapfile.c:3257:static void free_swap_cluster_info(struct swap_cluster_info *cluster_info,\nmm/swapfile.c-3258-\t\t\t\t   unsigned long maxpages)\n--\nmm/swapfile.c=3429=static void *swap_next(struct seq_file *swap, void *v, loff_t *pos)\n--\nmm/swapfile.c-3448-\nmm/swapfile.c:3449:static void swap_stop(struct seq_file *swap, void *v)\nmm/swapfile.c-3450-{\n--\nmm/swapfile.c-3453-\nmm/swapfile.c:3454:static int swap_show(struct seq_file *swap, void *v)\nmm/swapfile.c-3455-{\n--\nmm/swapfile.c=3481=static const struct seq_operations swaps_op = {\n--\nmm/swapfile.c-3487-\nmm/swapfile.c:3488:static int swaps_open(struct inode *inode, struct file *file)\nmm/swapfile.c-3489-{\n--\nmm/swapfile.c=3528=static struct swap_info_struct *alloc_swap_info(void)\n--\nmm/swapfile.c-3584-\nmm/swapfile.c:3585:static int claim_swapfile(struct swap_info_struct *si, struct inode *inode)\nmm/swapfile.c-3586-{\n--\nmm/swapfile.c=3642=static unsigned long read_swap_header(union swap_header *swap_header,\n--\nmm/swapfile.c-3703-\nmm/swapfile.c:3704:static int setup_swap_clusters_info(struct swap_info_struct *si,\nmm/swapfile.c-3705-\t\t\t\t    union swap_header *swap_header,\n--\nmm/swapfile.c=3797=SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)\n--\nmm/swapfile.c-4030-\nmm/swapfile.c:4031:void si_swapinfo(struct sysinfo *val)\nmm/swapfile.c-4032-{\n--\nmm/swapfile.c-4059- */\nmm/swapfile.c:4060:int swap_dup_entry_direct(swp_entry_t entry)\nmm/swapfile.c-4061-{\n--\nmm/swapfile.c-4080-#if defined(CONFIG_MEMCG) \u0026\u0026 defined(CONFIG_BLK_CGROUP)\nmm/swapfile.c:4081:static bool __has_usable_swap(void)\nmm/swapfile.c-4082-{\n--\nmm/swapfile.c-4085-\nmm/swapfile.c:4086:void __folio_throttle_swaprate(struct folio *folio, gfp_t gfp)\nmm/swapfile.c-4087-{\n"
}

7/2

2026/07/14 00:44

llm

patch-evaluator

Model:

gemini-3.1-pro-preview

Args:

null

Results:

null

Tokens:

							input: 33805
							output: 199
							thoughts: 578

Thoughts:

**My Fuzzing Strategy for This Swap Patch**

Okay, let's break this down. My initial scan has highlighted a handful of functions as potential hotspots in this swap implementation. Given the architectural change described – moving from a `plist` to a round-robin queue – I need to pinpoint the areas where this new logic is introduced and interacts with the existing swap framework.

I'm zeroing in on a few key functions: `swap_alloc_entry` looks like the new core for allocation, replacing some of the older logic. I'm also putting a pin on `swap_queue_add`, `swap_queue_del`, `swap_queue_get_device`, `swap_device_enable`, and `swap_device_disable` as these seem to be the primary interactions with the new queue mechanism. The in-use counters, `swap_device_inuse_add` and `swap_device_inuse_sub`, are crucial for ensuring correctness as well as the list manipulation functions `add_to_avail_list`, `del_from_avail_list`, `swap_queue_mask`, `swap_queue_unmask` and `cluster_alloc_swap_entry` are also essential.

I'm filtering out `folio_alloc_swap`, as interesting as it may be, because I suspect it is a more generic hot path, which, while important for performance, won't necessarily expose the core logic changes I'm after. `swap_alloc_entry` looks like a better target.

Now, let's confirm the relevant configuration options. `CONFIG_SWAP` seems to be the main driver. I'm quickly verifying that `mm/swapfile.c` is compiled when that's enabled. Yep, that confirms it.

Therefore, for this fuzzing effort, I'm going to set `WorthFuzzing` to `true` given the functional change and its impact on core kernel swap logic. The `FocusSymbols` are:

* `swap_alloc_entry`
* `swap_queue_add`
* `swap_queue_del`
* `swap_queue_get_device`
* `swap_device_enable`
* `swap_device_disable`
* `swap_device_inuse_add`
* `swap_device_inuse_sub`
* `add_to_avail_list`
* `del_from_avail_list`
* `swap_queue_mask`
* `swap_queue_unmask`
* `cluster_alloc_swap_entry`

And I'll enable only `SWAP` to build the kernel image.
My reasoning is that the introduction of a new round-robin queue, and its associated management functions, introduces new code that needs careful fuzzing. This is a functional change with the potential for subtle bugs in the kernel's memory management.

8/2

2026/07/14 00:44

tool

set-results

Args:

{
  "EnableConfigs": [
    "SWAP"
  ],
  "FocusSymbols": [
    "swap_alloc_entry",
    "swap_queue_add",
    "swap_queue_del",
    "swap_queue_get_device",
    "swap_device_enable",
    "swap_device_disable",
    "swap_device_inuse_add",
    "swap_device_inuse_sub",
    "add_to_avail_list",
    "del_from_avail_list",
    "swap_queue_mask",
    "swap_queue_unmask",
    "cluster_alloc_swap_entry"
  ],
  "Reasoning": "The patch replaces the plist-based swap device queue with a custom round-robin queue, introducing new core logic for swap allocation and device management. This is a functional change to the core kernel logic and should be fuzzed.",
  "WorthFuzzing": true
}

Results:

{
  "EnableConfigs": [
    "SWAP"
  ],
  "FocusSymbols": [
    "swap_alloc_entry",
    "swap_queue_add",
    "swap_queue_del",
    "swap_queue_get_device",
    "swap_device_enable",
    "swap_device_disable",
    "swap_device_inuse_add",
    "swap_device_inuse_sub",
    "add_to_avail_list",
    "del_from_avail_list",
    "swap_queue_mask",
    "swap_queue_unmask",
    "cluster_alloc_swap_entry"
  ],
  "Reasoning": "The patch replaces the plist-based swap device queue with a custom round-robin queue, introducing new core logic for swap allocation and device management. This is a functional change to the core kernel logic and should be fuzzed.",
  "WorthFuzzing": true
}

LLM Calls Summary:
Total Calls	Total Tokens	Avg Tokens	Total Duration (Seconds)	Avg Duration (Seconds)

Tool Calls Summary:
Total Calls	Total Duration (Seconds)	Avg Duration (Seconds)