| Seq | Timestamp | Type | Name | Duration |
|---|---|---|---|---|
| 0/0 | 2026/08/25 15:55 | flow | patch-triage |
1mArgs:null Results: {
"EnableConfigs": [
"VSWAP_DEFAULT_ON",
"ZSWAP",
"MEMCG"
],
"FocusSymbols": [
"folio_alloc_swap",
"folio_realloc_swap",
"folio_release_vswap_backing",
"folio_release_non_phys_swap_backing",
"__vswap_release_backing",
"__mem_cgroup_swap_charge",
"__mem_cgroup_swap_uncharge",
"__mem_cgroup_swap_get",
"__mem_cgroup_swap_record",
"__mem_cgroup_swap_put"
],
"KMSANReasoning": "The patch introduces a virtual swap layer (`vswap`) to decouple swap entries from physical swap slots, allowing better integration with `zswap` and zero-filled pages. The changes involve complex memory management logic, including dynamic allocation of swap clusters, virtual-to-physical mapping tables, and modifications to the swap read/write paths.\n\nAll new data structures (e.g., `struct swap_cluster_info_dynamic`, virtual tables, swap tables) are allocated using zeroing allocators (`kzalloc_obj`, `kcalloc`, `folio_alloc` with `__GFP_ZERO`). The patch does not introduce new kernel-to-user interfaces that copy structures to user space, nor does it add complex parsing of user-provided data where uninitialized fields could be used.\n\nIf a virtual swap entry lacks physical backing during a read fault, the code correctly handles it by returning an error (`-EIO`) without marking the folio as up-to-date, preventing the exposure of uninitialized page memory to user space (as explicitly noted in the `try_to_unuse` comments).\n\nAny potential bugs introduced by this patch (e.g., incorrect reference counting, race conditions in cluster management, or incorrect physical slot resolution) would manifest as use-after-free, out-of-bounds accesses, or memory corruption, which are effectively detected by KASAN, LOCKDEP, and standard kernel testing. Therefore, a dedicated KMSAN fuzzing session is not justified.",
"NeedsKMSAN": false,
"Reasoning": "The patch introduces a major new core memory management feature: the virtual swap layer (vswap). It modifies core swap allocation, zswap interaction, and memcg swap accounting. The changes are highly reachable and complex, making them a prime target for fuzzing.",
"WorthFuzzing": true
} |
| 1/1 | 2026/08/25 15:55 | action | read-patch-diff |
0mArgs:null Results: {
"PatchDiff": "commit 90158fb09c94f90a0fb0c502bca752b81303fd9d\nAuthor: syz-cluster \u003ctriage@syzkaller.com\u003e\nDate: Tue Aug 25 15:55:55 2026 +0000\n\n syz-cluster: applied patch under review\n\ndiff --git a/Documentation/admin-guide/cgroup-v1/memcg_test.rst b/Documentation/admin-guide/cgroup-v1/memcg_test.rst\nindex ebedbc3c3f9cc..13b9ae800b728 100644\n--- a/Documentation/admin-guide/cgroup-v1/memcg_test.rst\n+++ b/Documentation/admin-guide/cgroup-v1/memcg_test.rst\n@@ -43,7 +43,7 @@ Please note that implementation details can be changed.\n \tmem_cgroup_uncharge()\n \t Called when a page's refcount goes down to 0.\n \n-\tmem_cgroup_uncharge_swap()\n+\tmem_cgroup_swap_uncharge()\n \t Called when swp_entry's refcnt goes down to 0. A charge against swap\n \t disappears.\n \ndiff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt\nindex 1af62cd16c9de..6612b5e0a0554 100644\n--- a/Documentation/admin-guide/kernel-parameters.txt\n+++ b/Documentation/admin-guide/kernel-parameters.txt\n@@ -8364,6 +8364,13 @@ Kernel parameters\n \t\t\tforce\t\t- force vulnerability detection even on\n \t\t\t\t\t unaffected processors\n \n+\tvswap=\t\t[MM,EARLY]\n+\t\t\tRoute swapouts through the virtual swap layer, which\n+\t\t\tallows zswap and zero-filled pages to be used without\n+\t\t\ta physical swap device. 64-bit only.\n+\t\t\tFormat: { on | off }\n+\t\t\tDefault: on if CONFIG_VSWAP_DEFAULT_ON=y, else off.\n+\n \tvsyscall=\t[X86-64,EARLY]\n \t\t\tControls the behavior of vsyscalls (i.e. calls to\n \t\t\tfixed addresses of 0xffffffffff600x00 from legacy\ndiff --git a/MAINTAINERS b/MAINTAINERS\nindex 29236523cefb6..8a5827d271778 100644\n--- a/MAINTAINERS\n+++ b/MAINTAINERS\n@@ -17251,6 +17251,7 @@ F:\tmm/swap.h\n F:\tmm/swap_table.h\n F:\tmm/swap_state.c\n F:\tmm/swapfile.c\n+F:\tmm/vswap.h\n \n MEMORY MANAGEMENT - THP (TRANSPARENT HUGE PAGE)\n M:\tAndrew Morton \u003cakpm@linux-foundation.org\u003e\ndiff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h\nindex 215e2e87f42b2..4d89a35f49ff4 100644\n--- a/include/linux/memcontrol.h\n+++ b/include/linux/memcontrol.h\n@@ -1905,6 +1905,7 @@ static inline bool memcg_is_dying(struct mem_cgroup *memcg)\n \n #if defined(CONFIG_MEMCG) \u0026\u0026 defined(CONFIG_ZSWAP)\n bool obj_cgroup_may_zswap(struct obj_cgroup *objcg);\n+bool mem_cgroup_may_zswap(struct mem_cgroup *memcg, bool may_flush);\n void obj_cgroup_charge_zswap(struct obj_cgroup *objcg, size_t size);\n void obj_cgroup_uncharge_zswap(struct obj_cgroup *objcg, size_t size);\n bool mem_cgroup_zswap_writeback_enabled(struct mem_cgroup *memcg);\n@@ -1913,6 +1914,11 @@ static inline bool obj_cgroup_may_zswap(struct obj_cgroup *objcg)\n {\n \treturn true;\n }\n+\n+static inline bool mem_cgroup_may_zswap(struct mem_cgroup *memcg, bool may_flush)\n+{\n+\treturn true;\n+}\n static inline void obj_cgroup_charge_zswap(struct obj_cgroup *objcg,\n \t\t\t\t\t size_t size)\n {\ndiff --git a/include/linux/swap.h b/include/linux/swap.h\nindex 5658a1634b85e..5310ac0f3faae 100644\n--- a/include/linux/swap.h\n+++ b/include/linux/swap.h\n@@ -207,6 +207,7 @@ enum {\n \tSWP_STABLE_WRITES = (1 \u003c\u003c 11),\t/* no overwrite PG_writeback pages */\n \tSWP_SYNCHRONOUS_IO = (1 \u003c\u003c 12),\t/* synchronous IO is efficient */\n \tSWP_HIBERNATION = (1 \u003c\u003c 13),\t/* pinned for hibernation */\n+\tSWP_VSWAP\t= (1 \u003c\u003c 14),\t/* virtual swap device */\n \t\t\t\t\t/* add others here before... */\n };\n \n@@ -245,7 +246,7 @@ struct swap_info_struct {\n \tsigned short\tprio;\t\t/* swap priority of this type */\n \tstruct plist_node list;\t\t/* entry in swap_active_head */\n \tsigned char\ttype;\t\t/* strange name for an index */\n-\tunsigned int\tmax;\t\t/* size of this swap device */\n+\tunsigned long\tmax;\t\t/* size of this swap device */\n \tstruct swap_cluster_info *cluster_info; /* cluster info. Only for SSD */\n \tstruct list_head free_clusters; /* free clusters list */\n \tstruct list_head full_clusters; /* full clusters list */\n@@ -253,7 +254,7 @@ struct swap_info_struct {\n \t\t\t\t\t/* list of cluster that contains at least one free slot */\n \tstruct list_head frag_clusters[SWAP_NR_ORDERS];\n \t\t\t\t\t/* list of cluster that are fragmented or contented */\n-\tunsigned int pages;\t\t/* total of usable pages of swap */\n+\tunsigned long pages;\t\t/* total of usable pages of swap */\n \tatomic_long_t inuse_pages;\t/* number of those currently in use */\n \tstruct swap_sequential_cluster *global_cluster; /* Use one global cluster for rotating device */\n \tspinlock_t global_cluster_lock;\t/* Serialize usage of global cluster */\n@@ -276,8 +277,14 @@ struct swap_info_struct {\n \tstruct list_head discard_clusters; /* discard clusters list */\n \tstruct plist_node avail_list; /* entry in swap_avail_head */\n \tconst struct swap_ops *ops;\n+\tstruct xarray cluster_info_pool; /* Xarray for vswap dynamic cluster info */\n };\n \n+static inline bool swap_is_vswap(struct swap_info_struct *si)\n+{\n+\treturn si-\u003eflags \u0026 SWP_VSWAP;\n+}\n+\n static inline swp_entry_t page_swap_entry(struct page *page)\n {\n \tstruct folio *folio = page_folio(page);\n@@ -381,7 +388,7 @@ extern int __swap_count(swp_entry_t entry);\n extern bool swap_entry_swapped(struct swap_info_struct *si, swp_entry_t entry);\n extern int swp_swapcount(swp_entry_t entry);\n extern struct swap_info_struct *get_swap_device(swp_entry_t entry);\n-sector_t swap_folio_sector(struct folio *folio);\n+sector_t swap_entry_sector(swp_entry_t entry);\n \n /*\n * If there is an existing swap slot reference (swap entry) and the caller\n@@ -408,6 +415,8 @@ void swap_free_hibernation_slot(swp_entry_t entry);\n \n static inline void put_swap_device(struct swap_info_struct *si)\n {\n+\tif (swap_is_vswap(si))\n+\t\treturn;\n \tpercpu_ref_put(\u0026si-\u003eusers);\n }\n \n@@ -492,35 +501,80 @@ static inline void folio_throttle_swaprate(struct folio *folio, gfp_t gfp)\n #endif\n \n #if defined(CONFIG_MEMCG) \u0026\u0026 defined(CONFIG_SWAP)\n-int __mem_cgroup_try_charge_swap(struct folio *folio);\n-static inline int mem_cgroup_try_charge_swap(struct folio *folio)\n+struct mem_cgroup *__mem_cgroup_swap_get(struct folio *folio);\n+static inline struct mem_cgroup *mem_cgroup_swap_get(struct folio *folio)\n+{\n+\tif (mem_cgroup_disabled())\n+\t\treturn NULL;\n+\treturn __mem_cgroup_swap_get(folio);\n+}\n+\n+int __mem_cgroup_swap_charge(struct mem_cgroup *memcg, unsigned int nr_pages);\n+static inline int mem_cgroup_swap_charge(struct mem_cgroup *memcg,\n+\t\t\t\t\t unsigned int nr_pages)\n {\n \tif (mem_cgroup_disabled())\n \t\treturn 0;\n-\treturn __mem_cgroup_try_charge_swap(folio);\n+\treturn __mem_cgroup_swap_charge(memcg, nr_pages);\n }\n \n-extern void __mem_cgroup_uncharge_swap(unsigned short id, unsigned int nr_pages);\n-static inline void mem_cgroup_uncharge_swap(unsigned short id, unsigned int nr_pages)\n+void __mem_cgroup_swap_record(struct folio *folio, struct mem_cgroup *memcg);\n+static inline void mem_cgroup_swap_record(struct folio *folio,\n+\t\t\t\t\t struct mem_cgroup *memcg)\n {\n \tif (mem_cgroup_disabled())\n \t\treturn;\n-\t__mem_cgroup_uncharge_swap(id, nr_pages);\n+\t__mem_cgroup_swap_record(folio, memcg);\n+}\n+\n+void __mem_cgroup_swap_uncharge(struct mem_cgroup *memcg,\n+\t\t\t\tunsigned int nr_pages);\n+static inline void mem_cgroup_swap_uncharge(struct mem_cgroup *memcg,\n+\t\t\t\t\t unsigned int nr_pages)\n+{\n+\tif (mem_cgroup_disabled())\n+\t\treturn;\n+\t__mem_cgroup_swap_uncharge(memcg, nr_pages);\n+}\n+\n+void __mem_cgroup_swap_put(struct mem_cgroup *memcg, unsigned int nr_pages);\n+static inline void mem_cgroup_swap_put(struct mem_cgroup *memcg,\n+\t\t\t\t unsigned int nr_pages)\n+{\n+\tif (mem_cgroup_disabled())\n+\t\treturn;\n+\t__mem_cgroup_swap_put(memcg, nr_pages);\n }\n \n extern long mem_cgroup_get_nr_swap_pages(struct mem_cgroup *memcg);\n extern bool mem_cgroup_swap_full(struct folio *folio);\n #else\n-static inline int mem_cgroup_try_charge_swap(struct folio *folio)\n+static inline struct mem_cgroup *mem_cgroup_swap_get(struct folio *folio)\n+{\n+\treturn NULL;\n+}\n+\n+static inline int mem_cgroup_swap_charge(struct mem_cgroup *memcg,\n+\t\t\t\t\t unsigned int nr_pages)\n {\n \treturn 0;\n }\n \n-static inline void mem_cgroup_uncharge_swap(unsigned short id,\n+static inline void mem_cgroup_swap_record(struct folio *folio,\n+\t\t\t\t\t struct mem_cgroup *memcg)\n+{\n+}\n+\n+static inline void mem_cgroup_swap_uncharge(struct mem_cgroup *memcg,\n \t\t\t\t\t unsigned int nr_pages)\n {\n }\n \n+static inline void mem_cgroup_swap_put(struct mem_cgroup *memcg,\n+\t\t\t\t unsigned int nr_pages)\n+{\n+}\n+\n static inline long mem_cgroup_get_nr_swap_pages(struct mem_cgroup *memcg)\n {\n \treturn get_nr_swap_pages();\ndiff --git a/include/linux/swap_ops.h b/include/linux/swap_ops.h\nindex 57ac6c703f685..223c84548bde6 100644\n--- a/include/linux/swap_ops.h\n+++ b/include/linux/swap_ops.h\n@@ -12,6 +12,7 @@ struct swap_iocb {\n \tstruct bio_vec\t\tbvecs[SWAP_CLUSTER_MAX];\n \tint\t\t\tnr_bvecs;\n \tint\t\t\tlen;\n+\tswp_entry_t\t\tentry;\t/* first slot in the batch; addresses the IO */\n };\n \n struct swap_io_ctx {\n@@ -30,15 +31,15 @@ struct swap_io_ctx {\n struct swap_ops {\n \tunsigned int\t\tflags;\n \n-\tbool (*can_merge)(struct folio *folio, struct folio *prev_folio,\n-\t\t\tsize_t prev_folio_size, int rw);\n+\tbool (*can_merge)(struct folio *folio, swp_entry_t phys,\n+\t\t\tstruct swap_iocb *sio, int rw);\n \tvoid (*submit_write)(struct swap_io_ctx *ctx);\n \tvoid (*submit_read)(struct swap_io_ctx *ctx);\n };\n \n void swap_fs_prepare_rw(struct swap_io_ctx *ctx, int rw, struct iov_iter *iter);\n-bool swap_fs_can_merge(struct folio *folio, struct folio *prev_folio,\n-\t\tsize_t prev_folio_size, int rw);\n+bool swap_fs_can_merge(struct folio *folio, swp_entry_t phys,\n+\t\tstruct swap_iocb *sio, int rw);\n int swap_fs_activate(struct swap_info_struct *sis, const struct swap_ops *ops);\n \n #endif /* _MM_SWAP_OPS_H */\ndiff --git a/include/linux/zswap.h b/include/linux/zswap.h\nindex 30c193a1207e1..4b4f211f33017 100644\n--- a/include/linux/zswap.h\n+++ b/include/linux/zswap.h\n@@ -6,6 +6,7 @@\n #include \u003clinux/mm_types.h\u003e\n \n struct lruvec;\n+struct zswap_entry;\n \n extern atomic_long_t zswap_stored_pages;\n \n@@ -28,6 +29,7 @@ unsigned long zswap_total_pages(void);\n bool zswap_store(struct folio *folio);\n int zswap_load(struct folio *folio);\n void zswap_invalidate(swp_entry_t swp);\n+void zswap_entry_free(struct zswap_entry *entry);\n int zswap_swapon(int type, unsigned long nr_pages);\n void zswap_swapoff(int type);\n void zswap_memcg_offline_cleanup(struct mem_cgroup *memcg);\n@@ -50,6 +52,7 @@ static inline int zswap_load(struct folio *folio)\n }\n \n static inline void zswap_invalidate(swp_entry_t swp) {}\n+static inline void zswap_entry_free(struct zswap_entry *entry) {}\n static inline int zswap_swapon(int type, unsigned long nr_pages)\n {\n \treturn 0;\ndiff --git a/mm/Kconfig b/mm/Kconfig\nindex 604c58199acbf..08fdc7502c1de 100644\n--- a/mm/Kconfig\n+++ b/mm/Kconfig\n@@ -19,6 +19,26 @@ menuconfig SWAP\n \t used to provide more virtual memory than the actual RAM present\n \t in your computer. If unsure say Y.\n \n+config VSWAP_DEFAULT_ON\n+\tbool \"Route swapouts through virtual swap by default\"\n+\tdepends on SWAP \u0026\u0026 64BIT\n+\tdefault n\n+\thelp\n+\t Virtual swap allows zswap and zero-filled pages to be used\n+\t without swapping on a physical device first, and lets a page\n+\t move between zswap and a swapfile without invalidating the page\n+\t table entries that refer to it.\n+\n+\t Swap entries are handed out by a virtual swap device instead of\n+\t naming a slot on a real one, so the backing can be chosen and\n+\t changed after the entry exists.\n+\n+\t Say Y to make \"vswap=on\" the default, routing swapouts through\n+\t the virtual swap layer from boot.\n+\n+\t Say N (default) to leave vswap off unless \"vswap=on\" is passed\n+\t on the kernel command line.\n+\n config ZSWAP\n \tbool \"Compressed cache for swap pages\"\n \tdepends on SWAP\ndiff --git a/mm/memcontrol-v1.c b/mm/memcontrol-v1.c\nindex 05ef55cae4dc6..88016c8f22a26 100644\n--- a/mm/memcontrol-v1.c\n+++ b/mm/memcontrol-v1.c\n@@ -690,6 +690,7 @@ void __memcg1_swapout(struct folio *folio, struct swap_cluster_info *ci)\n void memcg1_swapin(struct folio *folio)\n {\n \tstruct swap_cluster_info *ci;\n+\tstruct mem_cgroup *memcg;\n \tunsigned long nr_pages;\n \tunsigned short id;\n \n@@ -721,7 +722,14 @@ void memcg1_swapin(struct folio *folio)\n \tid = __swap_cgroup_clear(ci, swp_cluster_offset(folio-\u003eswap),\n \t\t\t\t nr_pages);\n \tswap_cluster_unlock(ci);\n-\tmem_cgroup_uncharge_swap(id, nr_pages);\n+\n+\trcu_read_lock();\n+\tmemcg = mem_cgroup_from_private_id(id);\n+\tif (memcg) {\n+\t\tmem_cgroup_swap_uncharge(memcg, nr_pages);\n+\t\tmem_cgroup_swap_put(memcg, nr_pages);\n+\t}\n+\trcu_read_unlock();\n }\n #endif\n \ndiff --git a/mm/memcontrol.c b/mm/memcontrol.c\nindex 11b85f4b6828b..6ae0a4191d887 100644\n--- a/mm/memcontrol.c\n+++ b/mm/memcontrol.c\n@@ -65,6 +65,7 @@\n #include \"internal.h\"\n #include \"swap.h\"\n #include \"swap_table.h\"\n+#include \"vswap.h\"\n #include \u003cnet/sock.h\u003e\n #include \u003cnet/ip.h\u003e\n #include \"slab.h\"\n@@ -5721,86 +5722,129 @@ int __init mem_cgroup_init(void)\n \n #ifdef CONFIG_SWAP\n /**\n- * __mem_cgroup_try_charge_swap - try charging swap space for a folio\n+ * __mem_cgroup_swap_get - pin the memcg to account a folio's swap slots to\n * @folio: folio being added to swap\n *\n- * Try to charge @folio's memcg for the swap space at folio-\u003eswap.\n+ * Pins one private ID ref per page of @folio on its memcg, or on its closest\n+ * online ancestor if it has been offlined. The caller charges and records\n+ * against whichever memcg is returned, so both land on the same one.\n *\n- * Returns 0 on success, -ENOMEM on failure.\n+ * Return: the pinned memcg, or NULL if there is nothing to account. Drop the\n+ * pins with __mem_cgroup_swap_put().\n */\n-int __mem_cgroup_try_charge_swap(struct folio *folio)\n+struct mem_cgroup *__mem_cgroup_swap_get(struct folio *folio)\n {\n \tunsigned int nr_pages = folio_nr_pages(folio);\n-\tstruct swap_cluster_info *ci;\n-\tstruct page_counter *counter;\n \tstruct mem_cgroup *memcg;\n \tstruct obj_cgroup *objcg;\n \n \tif (do_memsw_account())\n-\t\treturn 0;\n+\t\treturn NULL;\n \n \tobjcg = folio_objcg(folio);\n \tVM_WARN_ON_ONCE_FOLIO(!objcg, folio);\n \tif (!objcg)\n-\t\treturn 0;\n+\t\treturn NULL;\n \n \trcu_read_lock();\n \tmemcg = obj_cgroup_memcg(objcg);\n \tif (!folio_test_swapcache(folio)) {\n \t\tmemcg_memory_event(memcg, MEMCG_SWAP_FAIL);\n \t\trcu_read_unlock();\n-\t\treturn 0;\n+\t\treturn NULL;\n \t}\n \n \tmemcg = mem_cgroup_private_id_get_online(memcg, nr_pages);\n \t/* memcg is pined by memcg ID. */\n \trcu_read_unlock();\n \n+\treturn memcg;\n+}\n+\n+/**\n+ * __mem_cgroup_swap_charge - charge physical swap space\n+ * @memcg: the mem_cgroup to charge (may be NULL)\n+ * @nr_pages: the amount of swap space to charge\n+ *\n+ * Return: 0 on success, -ENOMEM if memory.swap.max is exceeded.\n+ */\n+int __mem_cgroup_swap_charge(struct mem_cgroup *memcg, unsigned int nr_pages)\n+{\n+\tstruct page_counter *counter;\n+\n+\tif (do_memsw_account() || !memcg)\n+\t\treturn 0;\n+\n \tif (!mem_cgroup_is_root(memcg) \u0026\u0026\n \t !page_counter_try_charge(\u0026memcg-\u003eswap, nr_pages, \u0026counter)) {\n \t\tmemcg_memory_event(memcg, MEMCG_SWAP_MAX);\n \t\tmemcg_memory_event(memcg, MEMCG_SWAP_FAIL);\n-\t\tmem_cgroup_private_id_put(memcg, nr_pages);\n \t\treturn -ENOMEM;\n \t}\n \tmod_memcg_state(memcg, MEMCG_SWAP, nr_pages);\n+\treturn 0;\n+}\n+\n+/**\n+ * __mem_cgroup_swap_record - record the owner of a folio's swap slots\n+ * @folio: folio being added to swap\n+ * @memcg: the memcg pinned by __mem_cgroup_swap_get()\n+ */\n+void __mem_cgroup_swap_record(struct folio *folio, struct mem_cgroup *memcg)\n+{\n+\tstruct swap_cluster_info *ci;\n \n \tci = swap_cluster_get_and_lock(folio);\n-\t__swap_cgroup_set(ci, swp_cluster_offset(folio-\u003eswap), nr_pages,\n-\t\t\t mem_cgroup_private_id(memcg));\n+\t__swap_cgroup_set(ci, swp_cluster_offset(folio-\u003eswap),\n+\t\t\t folio_nr_pages(folio), mem_cgroup_private_id(memcg));\n \tswap_cluster_unlock(ci);\n-\n-\treturn 0;\n }\n \n /**\n- * __mem_cgroup_uncharge_swap - uncharge swap space\n- * @id: cgroup id to uncharge\n+ * __mem_cgroup_swap_uncharge - uncharge physical swap space\n+ * @memcg: the mem_cgroup to uncharge (may be NULL)\n * @nr_pages: the amount of swap space to uncharge\n */\n-void __mem_cgroup_uncharge_swap(unsigned short id, unsigned int nr_pages)\n+void __mem_cgroup_swap_uncharge(struct mem_cgroup *memcg, unsigned int nr_pages)\n {\n-\tstruct mem_cgroup *memcg;\n+\tif (!memcg)\n+\t\treturn;\n \n-\trcu_read_lock();\n-\tmemcg = mem_cgroup_from_private_id(id);\n-\tif (memcg) {\n-\t\tif (!mem_cgroup_is_root(memcg)) {\n-\t\t\tif (do_memsw_account())\n-\t\t\t\tpage_counter_uncharge(\u0026memcg-\u003ememsw, nr_pages);\n-\t\t\telse\n-\t\t\t\tpage_counter_uncharge(\u0026memcg-\u003eswap, nr_pages);\n-\t\t}\n-\t\tmod_memcg_state(memcg, MEMCG_SWAP, -nr_pages);\n-\t\tmem_cgroup_private_id_put(memcg, nr_pages);\n+\tif (!mem_cgroup_is_root(memcg)) {\n+\t\tif (do_memsw_account())\n+\t\t\tpage_counter_uncharge(\u0026memcg-\u003ememsw, nr_pages);\n+\t\telse\n+\t\t\tpage_counter_uncharge(\u0026memcg-\u003eswap, nr_pages);\n \t}\n-\trcu_read_unlock();\n+\tmod_memcg_state(memcg, MEMCG_SWAP, -nr_pages);\n+}\n+\n+/**\n+ * __mem_cgroup_swap_put - drop the private ID refs taken for swap slots\n+ * @memcg: the pinned mem_cgroup\n+ * @nr_pages: number of refs to drop\n+ */\n+void __mem_cgroup_swap_put(struct mem_cgroup *memcg, unsigned int nr_pages)\n+{\n+\tmem_cgroup_private_id_put(memcg, nr_pages);\n }\n \n long mem_cgroup_get_nr_swap_pages(struct mem_cgroup *memcg)\n {\n-\tlong nr_swap_pages = get_nr_swap_pages();\n+\tlong nr_swap_pages;\n+\n+\t/*\n+\t * vswap charges physical backing, not allocation, so virtual swap is\n+\t * unbounded for a zswap-capable memcg and the swap.max walk below\n+\t * would starve anon reclaim. swap.max is still enforced when the\n+\t * backing is charged.\n+\t */\n+\tif (vswap_is_enabled() \u0026\u0026 zswap_is_enabled() \u0026\u0026\n+\t (mem_cgroup_disabled() || do_memsw_account() ||\n+\t mem_cgroup_may_zswap(memcg, false)))\n+\t\treturn PAGE_COUNTER_MAX;\n \n+\tnr_swap_pages = get_nr_swap_pages();\n \tif (mem_cgroup_disabled() || do_memsw_account())\n \t\treturn nr_swap_pages;\n \tfor (; !mem_cgroup_is_root(memcg); memcg = parent_mem_cgroup(memcg))\n@@ -5972,8 +6016,10 @@ static struct cftype swap_files[] = {\n \n #ifdef CONFIG_ZSWAP\n /**\n- * obj_cgroup_may_zswap - check if this cgroup can zswap\n- * @objcg: the object cgroup\n+ * mem_cgroup_may_zswap - check if this cgroup can zswap\n+ * @memcg: the memcg to query\n+ * @may_flush: force-flush stats for an accurate check (sleeps). Pass false\n+ * from atomic contexts; the check is then best-effort.\n *\n * Check if the hierarchical zswap limit has been reached.\n *\n@@ -5983,36 +6029,38 @@ static struct cftype swap_files[] = {\n * spending cycles on compression when there is already no room left\n * or zswap is disabled altogether somewhere in the hierarchy.\n */\n-bool obj_cgroup_may_zswap(struct obj_cgroup *objcg)\n+bool mem_cgroup_may_zswap(struct mem_cgroup *memcg, bool may_flush)\n {\n-\tstruct mem_cgroup *memcg, *original_memcg;\n-\tbool ret = true;\n-\n \tif (!cgroup_subsys_on_dfl(memory_cgrp_subsys))\n \t\treturn true;\n \n-\toriginal_memcg = get_mem_cgroup_from_objcg(objcg);\n-\tfor (memcg = original_memcg; !mem_cgroup_is_root(memcg);\n-\t memcg = parent_mem_cgroup(memcg)) {\n+\tfor (; !mem_cgroup_is_root(memcg); memcg = parent_mem_cgroup(memcg)) {\n \t\tunsigned long max = READ_ONCE(memcg-\u003ezswap_max);\n \t\tunsigned long pages;\n \n \t\tif (max == PAGE_COUNTER_MAX)\n \t\t\tcontinue;\n-\t\tif (max == 0) {\n-\t\t\tret = false;\n-\t\t\tbreak;\n-\t\t}\n+\t\tif (max == 0)\n+\t\t\treturn false;\n \n \t\t/* Force flush to get accurate stats for charging */\n-\t\t__mem_cgroup_flush_stats(memcg, true);\n+\t\tif (may_flush)\n+\t\t\t__mem_cgroup_flush_stats(memcg, true);\n \t\tpages = memcg_page_state(memcg, MEMCG_ZSWAP_B) / PAGE_SIZE;\n-\t\tif (pages \u003c max)\n-\t\t\tcontinue;\n-\t\tret = false;\n-\t\tbreak;\n+\t\tif (pages \u003e= max)\n+\t\t\treturn false;\n \t}\n-\tmem_cgroup_put(original_memcg);\n+\treturn true;\n+}\n+\n+bool obj_cgroup_may_zswap(struct obj_cgroup *objcg)\n+{\n+\tstruct mem_cgroup *memcg;\n+\tbool ret;\n+\n+\tmemcg = get_mem_cgroup_from_objcg(objcg);\n+\tret = mem_cgroup_may_zswap(memcg, true);\n+\tmem_cgroup_put(memcg);\n \treturn ret;\n }\n \ndiff --git a/mm/memory.c b/mm/memory.c\nindex c549433025532..62f7b82427e21 100644\n--- a/mm/memory.c\n+++ b/mm/memory.c\n@@ -89,6 +89,7 @@\n #include \"pgalloc-track.h\"\n #include \"internal.h\"\n #include \"swap.h\"\n+#include \"vswap.h\"\n \n #if defined(LAST_CPUPID_NOT_IN_PAGE_FLAGS) \u0026\u0026 !defined(CONFIG_COMPILE_TEST)\n #warning Unfortunate NUMA and NUMA Balancing config, growing page-frame for last_cpupid.\n@@ -4659,7 +4660,13 @@ static inline bool should_try_to_free_swap(struct swap_info_struct *si,\n \t * are fast, and meanwhile, swap cache pinning the slot deferring the\n \t * release of metadata or fragmentation is a more critical issue.\n \t */\n-\tif (data_race(si-\u003eflags \u0026 SWP_SYNCHRONOUS_IO))\n+\tif (swap_entry_backend_has_flag(si, folio-\u003eswap, SWP_SYNCHRONOUS_IO))\n+\t\treturn true;\n+\t/*\n+\t * Non-swapfile backends cannot be reused for future swapouts.\n+\t * Free the swap slot unless backed by contiguous physical swap.\n+\t */\n+\tif (!folio_phys_swap_backed(folio))\n \t\treturn true;\n \tif (mem_cgroup_swap_full(folio) || (vma-\u003evm_flags \u0026 VM_LOCKED) ||\n \t folio_test_mlocked(folio))\n@@ -4809,15 +4816,19 @@ static unsigned long thp_swapin_suitable_orders(struct vm_fault *vmf)\n \tif (unlikely(userfaultfd_armed(vma)))\n \t\treturn 0;\n \n+\tentry = softleaf_from_pte(vmf-\u003eorig_pte);\n+\n \t/*\n \t * A large swapped out folio could be partially or fully in zswap. We\n \t * lack handling for such cases, so fallback to swapping in order-0\n \t * folio.\n+\t *\n+\t * Vswap entries are checked later, under the cluster lock in\n+\t * __swap_cache_add_check().\n \t */\n-\tif (!zswap_never_enabled())\n+\tif (!is_vswap_entry(entry) \u0026\u0026 !zswap_never_enabled())\n \t\treturn 0;\n \n-\tentry = softleaf_from_pte(vmf-\u003eorig_pte);\n \t/*\n \t * Get a list of all the (large) orders below PMD_ORDER that are enabled\n \t * and suitable for swapping THP.\n@@ -4963,7 +4974,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)\n \t\tswap_update_readahead(folio, vma, vmf-\u003eaddress);\n \tif (!folio) {\n \t\t/* Swapin bypasses readahead for SWP_SYNCHRONOUS_IO devices */\n-\t\tif (data_race(si-\u003eflags \u0026 SWP_SYNCHRONOUS_IO))\n+\t\tif (swap_entry_backend_has_flag(si, entry, SWP_SYNCHRONOUS_IO))\n \t\t\tfolio = swapin_sync(entry, GFP_HIGHUSER_MOVABLE,\n \t\t\t\t\t thp_swapin_suitable_orders(vmf) | BIT(0),\n \t\t\t\t\t vmf, NULL, 0);\n@@ -5128,7 +5139,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)\n \t\t\t */\n \t\t\texclusive = true;\n \t\t} else if (exclusive \u0026\u0026 folio_test_writeback(folio) \u0026\u0026\n-\t\t\t data_race(si-\u003eflags \u0026 SWP_STABLE_WRITES)) {\n+\t\t\t swap_entry_backend_has_flag(si, entry, SWP_STABLE_WRITES)) {\n \t\t\t/*\n \t\t\t * This is tricky: not all swap backends support\n \t\t\t * concurrent page modifications while under writeback.\ndiff --git a/mm/page_io.c b/mm/page_io.c\nindex 88962571cb931..b36a898358c64 100644\n--- a/mm/page_io.c\n+++ b/mm/page_io.c\n@@ -28,6 +28,7 @@\n #include \u003clinux/swap_ops.h\u003e\n #include \"swap.h\"\n #include \"swap_table.h\"\n+#include \"vswap.h\"\n \n int generic_swapfile_activate(struct swap_info_struct *sis,\n \t\t\t\tstruct file *swap_file,\n@@ -160,14 +161,19 @@ static void swap_zeromap_folio_set(struct folio *folio)\n \tstruct obj_cgroup *objcg = get_obj_cgroup_from_folio(folio);\n \tint nr_pages = folio_nr_pages(folio);\n \tstruct swap_cluster_info *ci;\n+\tunsigned int voff, i;\n \tswp_entry_t entry;\n-\tunsigned int i;\n \n \tVM_WARN_ON_ONCE_FOLIO(!folio_test_swapcache(folio), folio);\n \tVM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);\n \n \tci = swap_cluster_get_and_lock(folio);\n-\tfor (i = 0; i \u003c folio_nr_pages(folio); i++) {\n+\tif (is_vswap_entry(folio-\u003eswap)) {\n+\t\t/* Free any prior backing (e.g. ZSWAP entry from earlier swapout) */\n+\t\tvoff = swp_cluster_offset(folio-\u003eswap);\n+\t\t__vswap_release_backing(ci, voff, nr_pages);\n+\t}\n+\tfor (i = 0; i \u003c nr_pages; i++) {\n \t\tentry = page_swap_entry(folio_page(folio, i));\n \t\t__swap_table_set_zero(ci, swp_cluster_offset(entry));\n \t}\n@@ -203,6 +209,7 @@ static void swap_zeromap_folio_clear(struct folio *folio)\n */\n int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio)\n {\n+\tswp_entry_t phys;\n \tint ret = 0;\n \n \tif (folio_free_swap(folio))\n@@ -235,6 +242,15 @@ int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio)\n \t */\n \tswap_zeromap_folio_clear(folio);\n \n+\t/*\n+\t * For vswap: release stale non-swapfile backings (e.g. ZSWAP from a\n+\t * previous swapout cycle) so zswap_store or folio_realloc_swap\n+\t * starts on clean slots. Contiguous PHYS backing is preserved for\n+\t * reuse by folio_realloc_swap.\n+\t */\n+\tif (is_vswap_entry(folio-\u003eswap))\n+\t\tfolio_release_non_phys_swap_backing(folio);\n+\n \tif (zswap_store(folio)) {\n \t\tcount_mthp_stat(folio_order(folio), MTHP_STAT_ZSWPOUT);\n \t\tgoto out_unlock;\n@@ -248,7 +264,23 @@ int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio)\n \t}\n \trcu_read_unlock();\n \n-\t__swap_writepage(ctx, folio);\n+\t/*\n+\t * A vswap folio with no backend needs a physical slot to write to.\n+\t * zswap_store rolled back any partial vtable state on failure, so\n+\t * PHYS backing from a prior cycle is still there to reuse. If none\n+\t * is free, keep it dirty.\n+\t */\n+\tif (is_vswap_entry(folio-\u003eswap)) {\n+\t\tphys = folio_realloc_swap(folio);\n+\t\tif (!phys.val) {\n+\t\t\tfolio_mark_dirty(folio);\n+\t\t\treturn AOP_WRITEPAGE_ACTIVATE;\n+\t\t}\n+\t\t__swap_writepage(ctx, folio, phys);\n+\t\treturn 0;\n+\t}\n+\n+\t__swap_writepage(ctx, folio, folio-\u003eswap);\n \treturn 0;\n out_unlock:\n \tfolio_unlock(folio);\n@@ -317,24 +349,22 @@ int sio_pool_init(void)\n }\n \n static bool swap_can_merge(struct swap_io_ctx *ctx, struct folio *folio,\n-\t\tint rw)\n+\t\tswp_entry_t phys, int rw)\n {\n-\tstruct swap_info_struct *sis = __swap_entry_to_info(folio-\u003eswap);\n-\tstruct bio_vec *last_bv = \u0026ctx-\u003esio-\u003ebvecs[ctx-\u003esio-\u003enr_bvecs - 1];\n-\tstruct folio *prev_folio = bvec_folio(last_bv);\n-\tsize_t prev_folio_size = folio_size(prev_folio);\n+\tstruct swap_info_struct *sis = __swap_entry_to_info(phys);\n \n \tif (ctx-\u003esis != sis)\n \t\treturn false;\n-\treturn sis-\u003eops-\u003ecan_merge(folio, prev_folio, prev_folio_size, rw);\n+\treturn sis-\u003eops-\u003ecan_merge(folio, phys, ctx-\u003esio, rw);\n }\n \n-static void swap_add_folio(struct swap_io_ctx *ctx, struct folio *folio, int rw)\n+static void swap_add_folio(struct swap_io_ctx *ctx, struct folio *folio,\n+\t\tswp_entry_t phys, int rw)\n {\n-\tstruct swap_info_struct *sis = __swap_entry_to_info(folio-\u003eswap);\n+\tstruct swap_info_struct *sis = __swap_entry_to_info(phys);\n \tstruct swap_iocb *sio = ctx-\u003esio;\n \n-\tif (sio \u0026\u0026 !swap_can_merge(ctx, folio, rw)) {\n+\tif (sio \u0026\u0026 !swap_can_merge(ctx, folio, phys, rw)) {\n \t\tif (rw == WRITE)\n \t\t\tswap_write_submit(ctx);\n \t\telse\n@@ -347,6 +377,7 @@ static void swap_add_folio(struct swap_io_ctx *ctx, struct folio *folio, int rw)\n \t\tctx-\u003esio = sio = mempool_alloc(sio_pool, GFP_NOIO);\n \t\tsio-\u003enr_bvecs = 0;\n \t\tsio-\u003elen = 0;\n+\t\tsio-\u003eentry = phys;\n \t}\n \tbvec_set_folio(\u0026sio-\u003ebvecs[sio-\u003enr_bvecs], folio, folio_size(folio), 0);\n \tsio-\u003elen += folio_size(folio);\n@@ -367,7 +398,8 @@ static void swap_add_folio(struct swap_io_ctx *ctx, struct folio *folio, int rw)\n \t}\n }\n \n-void __swap_writepage(struct swap_io_ctx *ctx, struct folio *folio)\n+void __swap_writepage(struct swap_io_ctx *ctx, struct folio *folio,\n+\t\tswp_entry_t phys)\n {\n \tVM_BUG_ON_FOLIO(!folio_test_swapcache(folio), folio);\n \n@@ -383,7 +415,7 @@ void __swap_writepage(struct swap_io_ctx *ctx, struct folio *folio)\n \n \tfolio_start_writeback(folio);\n \tfolio_unlock(folio);\n-\tswap_add_folio(ctx, folio, WRITE);\n+\tswap_add_folio(ctx, folio, phys, WRITE);\n }\n \n /*\n@@ -456,6 +488,7 @@ void swap_read_folio(struct swap_io_ctx *ctx, struct folio *folio)\n \tbool workingset = folio_test_workingset(folio);\n \tunsigned long pflags;\n \tbool in_thrashing;\n+\tswp_entry_t phys;\n \n \tVM_BUG_ON_FOLIO(!folio_test_swapcache(folio) \u0026\u0026 !synchronous, folio);\n \tVM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);\n@@ -480,9 +513,24 @@ void swap_read_folio(struct swap_io_ctx *ctx, struct folio *folio)\n \tif (zswap_load(folio) != -ENOENT)\n \t\tgoto finish;\n \n+\t/*\n+\t * Resolve the physical slot to read from. A vswap entry keeps\n+\t * folio-\u003eswap virtual, so map it to its physical backing; a folio with\n+\t * no backing has nothing to read.\n+\t */\n+\tif (swap_is_vswap(sis)) {\n+\t\tphys = vswap_to_phys(folio-\u003eswap);\n+\t\tif (!phys.val) {\n+\t\t\tfolio_unlock(folio);\n+\t\t\tgoto finish;\n+\t\t}\n+\t} else {\n+\t\tphys = folio-\u003eswap;\n+\t}\n+\n \t/* We have to read from slower devices. Increase zswap protection. */\n \tzswap_folio_swapin(folio);\n-\tswap_add_folio(ctx, folio, READ);\n+\tswap_add_folio(ctx, folio, phys, READ);\n \n finish:\n \tif (workingset) {\n@@ -514,8 +562,6 @@ static void swap_fs_write_complete(struct kiocb *iocb, long ret)\n \tbool failed = ret != sio-\u003elen;\n \n \tif (failed) {\n-\t\tstruct page *page = sio-\u003ebvecs[0].bv_page;\n-\n \t\t/*\n \t\t * In the case of swap-over-nfs, this can be a temporary failure\n \t\t * if the system has limited memory for allocating transmit\n@@ -523,7 +569,7 @@ static void swap_fs_write_complete(struct kiocb *iocb, long ret)\n \t\t * folio_rotate_reclaimable but rate-limit the messages.\n \t\t */\n \t\tpr_err_ratelimited(\"Write error %ld on dio swapfile (%llu)\\n\",\n-\t\t\t\t ret, swap_dev_pos(page_swap_entry(page)));\n+\t\t\t\t ret, swap_dev_pos(sio-\u003eentry));\n \t}\n \n \tswap_write_end(sio, failed);\n@@ -595,7 +641,7 @@ static void swap_bdev_submit_write(struct swap_io_ctx *ctx)\n \tbio_init(bio, ctx-\u003esis-\u003ebdev, sio-\u003ebvecs, ARRAY_SIZE(sio-\u003ebvecs),\n \t\t\tREQ_OP_WRITE | REQ_SWAP);\n \tbio-\u003ebi_iter.bi_size = sio-\u003elen;\n-\tbio-\u003ebi_iter.bi_sector = swap_folio_sector(bio_first_folio_all(bio));\n+\tbio-\u003ebi_iter.bi_sector = swap_entry_sector(sio-\u003eentry);\n \tbio_associate_blkg_from_page(bio, bio_first_folio_all(bio));\n \n \tif (ctx-\u003esis-\u003eflags \u0026 SWP_SYNCHRONOUS_IO) {\n@@ -615,7 +661,7 @@ static void swap_bdev_submit_read(struct swap_io_ctx *ctx)\n \tbio_init(bio, ctx-\u003esis-\u003ebdev, sio-\u003ebvecs, ARRAY_SIZE(sio-\u003ebvecs),\n \t\t\tREQ_OP_READ);\n \tbio-\u003ebi_iter.bi_size = sio-\u003elen;\n-\tbio-\u003ebi_iter.bi_sector = swap_folio_sector(bio_first_folio_all(bio));\n+\tbio-\u003ebi_iter.bi_sector = swap_entry_sector(sio-\u003eentry);\n \n \tif (ctx-\u003esis-\u003eflags \u0026 SWP_SYNCHRONOUS_IO) {\n \t\t/*\n@@ -633,13 +679,14 @@ static void swap_bdev_submit_read(struct swap_io_ctx *ctx)\n \t}\n }\n \n-static bool swap_bdev_can_merge(struct folio *folio, struct folio *prev_folio,\n-\t\tsize_t prev_folio_size, int rw)\n+static bool swap_bdev_can_merge(struct folio *folio, swp_entry_t phys,\n+\t\tstruct swap_iocb *sio, int rw)\n {\n-\tif (swap_folio_sector(folio) !=\n-\t swap_folio_sector(prev_folio) + (prev_folio_size \u003e\u003e SECTOR_SHIFT))\n+\tif (swap_entry_sector(phys) !=\n+\t swap_entry_sector(sio-\u003eentry) + (sio-\u003elen \u003e\u003e SECTOR_SHIFT))\n \t\treturn false;\n-\tif (rw == WRITE \u0026\u0026 !folio_blkg_can_merge(folio, prev_folio))\n+\tif (rw == WRITE \u0026\u0026 !folio_blkg_can_merge(folio,\n+\t\t\tbvec_folio(\u0026sio-\u003ebvecs[sio-\u003enr_bvecs - 1])))\n \t\treturn false;\n \treturn true;\n }\n@@ -655,7 +702,7 @@ void swap_fs_prepare_rw(struct swap_io_ctx *ctx, int rw, struct iov_iter *iter)\n \tstruct swap_iocb *sio = ctx-\u003esio;\n \n \tinit_sync_kiocb(\u0026sio-\u003eiocb, ctx-\u003esis-\u003eswap_file);\n-\tsio-\u003eiocb.ki_pos = swap_dev_pos(bvec_folio(\u0026sio-\u003ebvecs[0])-\u003eswap);\n+\tsio-\u003eiocb.ki_pos = swap_dev_pos(sio-\u003eentry);\n \tif (rw == WRITE)\n \t\tsio-\u003eiocb.ki_complete = swap_fs_write_complete;\n \telse\n@@ -666,11 +713,10 @@ void swap_fs_prepare_rw(struct swap_io_ctx *ctx, int rw, struct iov_iter *iter)\n }\n EXPORT_SYMBOL_GPL(swap_fs_prepare_rw);\n \n-bool swap_fs_can_merge(struct folio *folio, struct folio *prev_folio,\n-\t\tsize_t prev_folio_size, int rw)\n+bool swap_fs_can_merge(struct folio *folio, swp_entry_t phys,\n+\t\tstruct swap_iocb *sio, int rw)\n {\n-\treturn swap_dev_pos(folio-\u003eswap) ==\n-\t\tswap_dev_pos(prev_folio-\u003eswap) + prev_folio_size;\n+\treturn swap_dev_pos(phys) == swap_dev_pos(sio-\u003eentry) + sio-\u003elen;\n }\n EXPORT_SYMBOL_GPL(swap_fs_can_merge);\n \ndiff --git a/mm/shmem.c b/mm/shmem.c\nindex 599665a3d6e7b..d58f30b06255b 100644\n--- a/mm/shmem.c\n+++ b/mm/shmem.c\n@@ -86,6 +86,7 @@ static struct vfsmount *shm_mnt __ro_after_init;\n #include \u003clinux/uaccess.h\u003e\n \n #include \"internal.h\"\n+#include \"vswap.h\"\n \n #define VM_ACCT(size) (PAGE_ALIGN(size) \u003e\u003e PAGE_SHIFT)\n \n@@ -1618,7 +1619,8 @@ int shmem_writeout(struct swap_io_ctx *ctx, struct folio *folio,\n \tif ((info-\u003eflags \u0026 SHMEM_F_LOCKED) || sbinfo-\u003enoswap)\n \t\tgoto redirty;\n \n-\tif (!total_swap_pages)\n+\t/* vswap doesn't contribute to total_swap_pages */\n+\tif (!total_swap_pages \u0026\u0026 !(vswap_is_enabled() \u0026\u0026 zswap_is_enabled()))\n \t\tgoto redirty;\n \n \t/*\ndiff --git a/mm/swap.h b/mm/swap.h\nindex 90a551a88df63..f2b64920ae5b3 100644\n--- a/mm/swap.h\n+++ b/mm/swap.h\n@@ -67,6 +67,13 @@ struct swap_cluster_info {\n \tstruct list_head list;\n };\n \n+struct swap_cluster_info_dynamic {\n+\tstruct swap_cluster_info ci;\n+\tunsigned int index;\t\t/* for cluster_index() */\n+\tstruct rcu_head rcu;\n+\tatomic_long_t *virtual_table;\t/* Backing pointers for vswap slots */\n+};\n+\n /* All on-list cluster must have a non-zero flag. */\n enum swap_cluster_flags {\n \tCLUSTER_FLAG_NONE = 0, /* For temporary off-list cluster */\n@@ -77,6 +84,7 @@ enum swap_cluster_flags {\n \tCLUSTER_FLAG_USABLE = CLUSTER_FLAG_FRAG,\n \tCLUSTER_FLAG_FULL,\n \tCLUSTER_FLAG_DISCARD,\n+\tCLUSTER_FLAG_DEAD,\t/* Vswap dynamic cluster pending kfree_rcu */\n \tCLUSTER_FLAG_MAX,\n };\n \n@@ -119,12 +127,33 @@ static inline struct swap_info_struct *__swap_entry_to_info(swp_entry_t entry)\n \treturn __swap_type_to_info(swp_type(entry));\n }\n \n+/**\n+ * __swap_offset_to_cluster - look up the cluster holding a swap offset\n+ * @si: the swap device\n+ * @offset: the swap entry offset\n+ *\n+ * Context: A vswap cluster is freed by kfree_rcu(). Callers must hold the\n+ * RCU read lock, or know the cluster is pinned by an in-use entry.\n+ *\n+ * Return: the cluster, or NULL if @si is a vswap device with no cluster\n+ * allocated at @offset.\n+ */\n static inline struct swap_cluster_info *__swap_offset_to_cluster(\n \t\tstruct swap_info_struct *si, pgoff_t offset)\n {\n+\tunsigned int cluster_idx = offset / SWAPFILE_CLUSTER;\n+\n \tVM_WARN_ON_ONCE(percpu_ref_is_zero(\u0026si-\u003eusers)); /* race with swapoff */\n \tVM_WARN_ON_ONCE(offset \u003e= roundup(si-\u003emax, SWAPFILE_CLUSTER));\n-\treturn \u0026si-\u003ecluster_info[offset / SWAPFILE_CLUSTER];\n+\n+\tif (swap_is_vswap(si)) {\n+\t\tstruct swap_cluster_info_dynamic *ci_dyn;\n+\n+\t\tci_dyn = xa_load(\u0026si-\u003ecluster_info_pool, cluster_idx);\n+\t\treturn ci_dyn ? \u0026ci_dyn-\u003eci : NULL;\n+\t}\n+\n+\treturn \u0026si-\u003ecluster_info[cluster_idx];\n }\n \n static inline struct swap_cluster_info *__swap_entry_to_cluster(swp_entry_t entry)\n@@ -133,10 +162,36 @@ static inline struct swap_cluster_info *__swap_entry_to_cluster(swp_entry_t entr\n \t\t\t\t\tswp_offset(entry));\n }\n \n+static inline struct swap_cluster_info *__vswap_cluster_lock(\n+\t\tstruct swap_info_struct *si, unsigned long offset, bool irq)\n+{\n+\tstruct swap_cluster_info *ci;\n+\n+\trcu_read_lock();\n+\tci = __swap_offset_to_cluster(si, offset);\n+\tif (ci) {\n+\t\tif (irq)\n+\t\t\tspin_lock_irq(\u0026ci-\u003elock);\n+\t\telse\n+\t\t\tspin_lock(\u0026ci-\u003elock);\n+\n+\t\t/* The cluster can be torn down while we wait for the lock. */\n+\t\tif (ci-\u003eflags == CLUSTER_FLAG_DEAD) {\n+\t\t\tif (irq)\n+\t\t\t\tspin_unlock_irq(\u0026ci-\u003elock);\n+\t\t\telse\n+\t\t\t\tspin_unlock(\u0026ci-\u003elock);\n+\t\t\tci = NULL;\n+\t\t}\n+\t}\n+\trcu_read_unlock();\n+\treturn ci;\n+}\n+\n static __always_inline struct swap_cluster_info *__swap_cluster_lock(\n \t\tstruct swap_info_struct *si, unsigned long offset, bool irq)\n {\n-\tstruct swap_cluster_info *ci = __swap_offset_to_cluster(si, offset);\n+\tstruct swap_cluster_info *ci;\n \n \t/*\n \t * Nothing modifies swap cache in an IRQ context. All access to\n@@ -149,6 +204,11 @@ static __always_inline struct swap_cluster_info *__swap_cluster_lock(\n \t */\n \tVM_WARN_ON_ONCE(!in_task());\n \tVM_WARN_ON_ONCE(percpu_ref_is_zero(\u0026si-\u003eusers)); /* race with swapoff */\n+\n+\tif (swap_is_vswap(si))\n+\t\treturn __vswap_cluster_lock(si, offset, irq);\n+\n+\tci = __swap_offset_to_cluster(si, offset);\n \tif (irq)\n \t\tspin_lock_irq(\u0026ci-\u003elock);\n \telse\n@@ -159,10 +219,12 @@ static __always_inline struct swap_cluster_info *__swap_cluster_lock(\n /**\n * swap_cluster_lock - Lock and return the swap cluster of given offset.\n * @si: swap device the cluster belongs to.\n- * @offset: the swap entry offset, pointing to a valid slot.\n+ * @offset: the swap entry offset.\n *\n * Context: The caller must ensure the offset is in the valid range and\n * protect the swap device with reference count or locks.\n+ * Return: the locked cluster, or NULL if it is gone. Only a vswap device\n+ * can return NULL, as its clusters are allocated and freed on demand.\n */\n static inline struct swap_cluster_info *swap_cluster_lock(\n \t\tstruct swap_info_struct *si, unsigned long offset)\n@@ -258,7 +320,8 @@ void swap_read_folio(struct swap_io_ctx *ctx, struct folio *folio);\n void swap_read_submit(struct swap_io_ctx *ctx);\n void swap_write_submit(struct swap_io_ctx *ctx);\n int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio);\n-void __swap_writepage(struct swap_io_ctx *ctx, struct folio *folio);\n+void __swap_writepage(struct swap_io_ctx *ctx, struct folio *folio,\n+\t\tswp_entry_t phys);\n \n /* linux/mm/swap_state.c */\n extern struct address_space swap_space __read_mostly;\ndiff --git a/mm/swap_state.c b/mm/swap_state.c\nindex b76eb3d876fd7..5cfddec8633b9 100644\n--- a/mm/swap_state.c\n+++ b/mm/swap_state.c\n@@ -27,6 +27,7 @@\n #include \"internal.h\"\n #include \"swap_table.h\"\n #include \"swap.h\"\n+#include \"vswap.h\"\n \n /* Swap readahead cluster size, as a power of 2 pages. */\n static int page_cluster;\n@@ -96,8 +97,10 @@ struct folio *swap_cache_get_folio(swp_entry_t entry)\n \tstruct folio *folio;\n \n \tfor (;;) {\n+\t\trcu_read_lock();\n \t\tswp_tb = swap_table_get(__swap_entry_to_cluster(entry),\n \t\t\t\t\tswp_cluster_offset(entry));\n+\t\trcu_read_unlock();\n \t\tif (!swp_tb_is_folio(swp_tb))\n \t\t\treturn NULL;\n \t\tfolio = swp_tb_to_folio(swp_tb);\n@@ -119,8 +122,10 @@ bool swap_cache_has_folio(swp_entry_t entry)\n {\n \tunsigned long swp_tb;\n \n+\trcu_read_lock();\n \tswp_tb = swap_table_get(__swap_entry_to_cluster(entry),\n \t\t\t\tswp_cluster_offset(entry));\n+\trcu_read_unlock();\n \treturn swp_tb_is_folio(swp_tb);\n }\n \n@@ -136,8 +141,10 @@ void *swap_cache_get_shadow(swp_entry_t entry)\n {\n \tunsigned long swp_tb;\n \n+\trcu_read_lock();\n \tswp_tb = swap_table_get(__swap_entry_to_cluster(entry),\n \t\t\t\tswp_cluster_offset(entry));\n+\trcu_read_unlock();\n \tif (swp_tb_is_shadow(swp_tb))\n \t\treturn swp_tb_to_shadow(swp_tb);\n \treturn NULL;\n@@ -167,6 +174,9 @@ static int __swap_cache_add_check(struct swap_cluster_info *ci,\n \tunsigned int ci_off, ci_end;\n \tunsigned long old_tb;\n \tbool is_zero;\n+\tstruct swap_cluster_info_dynamic *ci_dyn;\n+\tenum vswap_backing_type type;\n+\tint ret;\n \n \tlockdep_assert_held(\u0026ci-\u003elock);\n \n@@ -179,6 +189,9 @@ static int __swap_cache_add_check(struct swap_cluster_info *ci,\n \t\treturn -ENOENT;\n \tci_off = swp_cluster_offset(targ_entry);\n \told_tb = __swap_table_get(ci, ci_off);\n+\t/* Physical readahead can hit a vswap-backing rmap slot; skip it. */\n+\tif (swp_tb_is_pointer(old_tb))\n+\t\treturn -ENOENT;\n \tif (swp_tb_is_folio(old_tb))\n \t\treturn -EEXIST;\n \tif (!__swp_tb_get_count(old_tb))\n@@ -191,12 +204,26 @@ static int __swap_cache_add_check(struct swap_cluster_info *ci,\n \tif (nr == 1)\n \t\treturn 0;\n \n+\t/*\n+\t * For a vswap entry batch, reject if the backing is not THP-amenable\n+\t * (e.g. uniformly ZSWAP, or mixed). The order-fallback loop in\n+\t * swap_cache_alloc_folio will retry with a smaller order on -EBUSY.\n+\t */\n+\tif (is_vswap_entry(targ_entry)) {\n+\t\tci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);\n+\t\tret = __vswap_check_backing(ci_dyn, round_down(ci_off, nr),\n+\t\t\t\t\t nr, \u0026type);\n+\t\tif (ret != nr || type == VSWAP_ZSWAP)\n+\t\t\treturn -EBUSY;\n+\t}\n+\n \tis_zero = __swap_table_test_zero(ci, ci_off);\n \tci_off = round_down(ci_off, nr);\n \tci_end = ci_off + nr;\n \tdo {\n \t\told_tb = __swap_table_get(ci, ci_off);\n-\t\tif (unlikely(swp_tb_is_folio(old_tb) ||\n+\t\tif (unlikely(swp_tb_is_pointer(old_tb) ||\n+\t\t\t swp_tb_is_folio(old_tb) ||\n \t\t\t !__swp_tb_get_count(old_tb) ||\n \t\t\t is_zero != __swap_table_test_zero(ci, ci_off) ||\n \t\t\t (memcg_id \u0026\u0026 *memcg_id != __swap_cgroup_get(ci, ci_off))))\n@@ -406,14 +433,16 @@ void __swap_cache_replace_folio(struct swap_cluster_info *ci,\n * -ENOENT / -EEXIST: Target swap entry is unavailable or cached, the caller\n * should abort or try to use the cached folio instead\n */\n-static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,\n-\t\t\t\t\tswp_entry_t targ_entry, gfp_t gfp,\n+static struct folio *__swap_cache_alloc(swp_entry_t targ_entry, gfp_t gfp,\n \t\t\t\t\tunsigned int order, struct vm_fault *vmf,\n \t\t\t\t\tstruct mempolicy *mpol, pgoff_t ilx)\n {\n \tint err;\n \tswp_entry_t entry;\n \tstruct folio *folio;\n+\tstruct swap_cluster_info *ci;\n+\tstruct swap_info_struct *si = __swap_entry_to_info(targ_entry);\n+\tunsigned long offset = swp_offset(targ_entry);\n \tvoid *shadow = NULL;\n \tunsigned short memcg_id;\n \tunsigned long address, nr_pages = 1UL \u003c\u003c order;\n@@ -423,9 +452,12 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,\n \tentry.val = round_down(targ_entry.val, nr_pages);\n \n \t/* Check if the slot and range are available, skip allocation if not */\n-\tspin_lock(\u0026ci-\u003elock);\n-\terr = __swap_cache_add_check(ci, targ_entry, nr_pages, NULL, NULL);\n-\tspin_unlock(\u0026ci-\u003elock);\n+\terr = -ENOENT;\n+\tci = swap_cluster_lock(si, offset);\n+\tif (ci) {\n+\t\terr = __swap_cache_add_check(ci, targ_entry, nr_pages, NULL, NULL);\n+\t\tswap_cluster_unlock(ci);\n+\t}\n \tif (unlikely(err))\n \t\treturn ERR_PTR(err);\n \n@@ -446,10 +478,13 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,\n \t\treturn ERR_PTR(-ENOMEM);\n \n \t/* Double check the range is still not in conflict */\n-\tspin_lock(\u0026ci-\u003elock);\n-\terr = __swap_cache_add_check(ci, targ_entry, nr_pages, \u0026shadow, \u0026memcg_id);\n+\terr = -ENOENT;\n+\tci = swap_cluster_lock(si, offset);\n+\tif (ci)\n+\t\terr = __swap_cache_add_check(ci, targ_entry, nr_pages, \u0026shadow, \u0026memcg_id);\n \tif (unlikely(err)) {\n-\t\tspin_unlock(\u0026ci-\u003elock);\n+\t\tif (ci)\n+\t\t\tswap_cluster_unlock(ci);\n \t\tfolio_put(folio);\n \t\treturn ERR_PTR(err);\n \t}\n@@ -457,10 +492,11 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,\n \t__folio_set_locked(folio);\n \t__folio_set_swapbacked(folio);\n \t__swap_cache_do_add_folio(ci, folio, entry);\n-\tspin_unlock(\u0026ci-\u003elock);\n+\tswap_cluster_unlock(ci);\n \n \tif (mem_cgroup_swapin_charge_folio(folio, memcg_id,\n \t\t\t\t\t vmf ? vmf-\u003evma-\u003evm_mm : NULL, gfp)) {\n+\t\t/* The folio pins the cluster */\n \t\tspin_lock(\u0026ci-\u003elock);\n \t\t__swap_cache_do_del_folio(ci, folio, entry, shadow);\n \t\tspin_unlock(\u0026ci-\u003elock);\n@@ -517,9 +553,7 @@ struct folio *swap_cache_alloc_folio(swp_entry_t targ_entry, gfp_t gfp,\n {\n \tint order, err;\n \tstruct folio *ret;\n-\tstruct swap_cluster_info *ci;\n \n-\tci = __swap_entry_to_cluster(targ_entry);\n \torder = highest_order(orders);\n \n \t/* orders must be non-zero, and must not exceed cluster size. */\n@@ -527,7 +561,7 @@ struct folio *swap_cache_alloc_folio(swp_entry_t targ_entry, gfp_t gfp,\n \t\treturn ERR_PTR(-EINVAL);\n \n \tdo {\n-\t\tret = __swap_cache_alloc(ci, targ_entry, gfp, order,\n+\t\tret = __swap_cache_alloc(targ_entry, gfp, order,\n \t\t\t\t\t vmf, mpol, ilx);\n \t\tif (!IS_ERR(ret))\n \t\t\tbreak;\ndiff --git a/mm/swap_table.h b/mm/swap_table.h\nindex e6613e62f8d0f..b614b1989fd9d 100644\n--- a/mm/swap_table.h\n+++ b/mm/swap_table.h\n@@ -4,8 +4,11 @@\n \n #include \u003clinux/rcupdate.h\u003e\n #include \u003clinux/atomic.h\u003e\n+#include \u003clinux/swapops.h\u003e\n #include \"swap.h\"\n \n+extern struct swap_info_struct *vswap_si;\n+\n /* A typical flat array in each cluster as swap table */\n struct swap_table {\n \tatomic_long_t entries[SWAPFILE_CLUSTER];\n@@ -28,7 +31,7 @@ struct swap_memcg_table {\n * NULL: |---------------- 0 ---------------| - Free slot\n * Shadow: |SWAP_COUNT|Z|---- SHADOW_VAL ---|1| - Swapped out slot\n * PFN: |SWAP_COUNT|Z|------ PFN -------|10| - Cached slot\n- * Pointer: |----------- Pointer ----------|100| - (Unused)\n+ * Pointer: |C|------- vswap offset -------|100| - vswap rmap\n * Bad: |------------- 1 -------------|1000| - Bad slot\n *\n * COUNT is `SWP_TB_COUNT_BITS` long, Z is the `SWP_TB_ZERO_FLAG` bit,\n@@ -49,9 +52,8 @@ struct swap_memcg_table {\n * - PFN: Swap slot is in use, and cached. Memcg info is recorded on the page\n * struct.\n *\n- * - Pointer: Unused yet. `0b100` is reserved for potential pointer usage\n- * because only the lower three bits can be used as a marker for 8 bytes\n- * aligned pointers.\n+ * - Pointer: Reverse map from a physical slot to the vswap entry that owns\n+ * it. See the layout below.\n *\n * - Bad: Swap slot is reserved, protects swap header or holes on swap devices.\n */\n@@ -255,6 +257,8 @@ static inline unsigned long swap_table_get(struct swap_cluster_info *ci,\n \tunsigned long swp_tb;\n \n \tVM_WARN_ON_ONCE(off \u003e= SWAPFILE_CLUSTER);\n+\tif (!ci)\n+\t\treturn SWP_TB_NULL;\n \n \trcu_read_lock();\n \ttable = rcu_dereference(ci-\u003etable);\n@@ -366,4 +370,42 @@ static inline unsigned short __swap_cgroup_clear(struct swap_cluster_info *ci,\n }\n #endif\n \n+/*\n+ * Pointer-tagged swap table entry: rmap for vswap-backing physical slots.\n+ *\n+ * On physical clusters, a Pointer-tagged entry stores the offset of the\n+ * vswap entry that owns this physical slot (the reverse map). Only the\n+ * offset is stored; the swap type is implicit (always vswap_si-\u003etype,\n+ * since there is exactly one vswap device). The top bit is reserved as\n+ * a cache-only flag, set when vswap swap_count drops to 0 but the folio\n+ * is still in swap cache.\n+ *\n+ * Pointer: |C|---- vswap offset ----|100|\n+ * C = SWP_RMAP_CACHE_ONLY (bit 63)\n+ */\n+#define SWP_TB_PTR_MARK_BITS\t3\n+#define SWP_TB_PTR_MARK\t\t0b100UL\n+#define SWP_TB_PTR_MARK_MASK\t((1UL \u003c\u003c SWP_TB_PTR_MARK_BITS) - 1)\n+#define SWP_RMAP_CACHE_ONLY\t(1UL \u003c\u003c (BITS_PER_LONG - 1))\n+#define SWP_RMAP_ENTRY_MASK\t(~(SWP_RMAP_CACHE_ONLY | SWP_TB_PTR_MARK_MASK))\n+\n+static inline bool swp_tb_is_pointer(unsigned long swp_tb)\n+{\n+\treturn (swp_tb \u0026 SWP_TB_PTR_MARK_MASK) == SWP_TB_PTR_MARK;\n+}\n+\n+static inline unsigned long swp_entry_to_swp_tb_ptr(swp_entry_t entry)\n+{\n+\treturn (swp_offset(entry) \u003c\u003c SWP_TB_PTR_MARK_BITS) | SWP_TB_PTR_MARK;\n+}\n+\n+static inline swp_entry_t swp_tb_ptr_to_swp_entry(unsigned long swp_tb)\n+{\n+\tunsigned long offset;\n+\n+\tVM_WARN_ON(!swp_tb_is_pointer(swp_tb));\n+\toffset = (swp_tb \u0026 SWP_RMAP_ENTRY_MASK) \u003e\u003e SWP_TB_PTR_MARK_BITS;\n+\treturn swp_entry(vswap_si-\u003etype, offset);\n+}\n+\n #endif\ndiff --git a/mm/swapfile.c b/mm/swapfile.c\nindex 53bf01d5f7f11..9860fb3b079fb 100644\n--- a/mm/swapfile.c\n+++ b/mm/swapfile.c\n@@ -7,6 +7,7 @@\n */\n \n #include \u003clinux/blkdev.h\u003e\n+#include \u003clinux/debugfs.h\u003e\n #include \u003clinux/mm.h\u003e\n #include \u003clinux/sched/mm.h\u003e\n #include \u003clinux/sched/task.h\u003e\n@@ -36,6 +37,7 @@\n #include \u003clinux/poll.h\u003e\n #include \u003clinux/oom.h\u003e\n #include \u003clinux/swapfile.h\u003e\n+#include \u003clinux/swap_ops.h\u003e\n #include \u003clinux/export.h\u003e\n #include \u003clinux/sort.h\u003e\n #include \u003clinux/completion.h\u003e\n@@ -45,7 +47,9 @@\n \n #include \u003casm/tlbflush.h\u003e\n #include \u003clinux/leafops.h\u003e\n+#include \"memcontrol-v1.h\"\n #include \"swap_table.h\"\n+#include \"vswap.h\"\n #include \"internal.h\"\n #include \"swap.h\"\n \n@@ -129,6 +133,24 @@ static DEFINE_PER_CPU(struct percpu_swap_cluster, percpu_swap_cluster) = {\n \t.lock = INIT_LOCAL_LOCK(),\n };\n \n+struct percpu_vswap_cluster {\n+\tunsigned long offset[SWAP_NR_ORDERS];\n+\tlocal_lock_t lock;\n+};\n+\n+static DEFINE_PER_CPU(struct percpu_vswap_cluster, percpu_vswap_cluster) = {\n+\t.offset = { [0 ... SWAP_NR_ORDERS - 1] = SWAP_ENTRY_INVALID },\n+\t.lock = INIT_LOCAL_LOCK(),\n+};\n+\n+static atomic_long_t vswap_alloc_reject = ATOMIC_LONG_INIT(0);\n+\n+static bool vswap_alloc(struct folio *folio);\n+static void vswap_mark_cache_only(struct swap_cluster_info *ci,\n+\t\t\t\t unsigned int ci_off);\n+static void vswap_clear_cache_only(struct swap_cluster_info *ci,\n+\t\t\t\t unsigned int ci_start, int nr);\n+\n /* May return NULL on invalid type, caller must check for NULL return */\n static struct swap_info_struct *swap_type_to_info(int type)\n {\n@@ -234,7 +256,8 @@ static int __try_to_reclaim_swap(struct swap_info_struct *si,\n \n \tneed_reclaim = ((flags \u0026 TTRS_ANYWAY) ||\n \t\t\t((flags \u0026 TTRS_UNMAPPED) \u0026\u0026 !folio_mapped(folio)) ||\n-\t\t\t((flags \u0026 TTRS_FULL) \u0026\u0026 mem_cgroup_swap_full(folio)));\n+\t\t\t((flags \u0026 TTRS_FULL) \u0026\u0026 mem_cgroup_swap_full(folio) \u0026\u0026\n+\t\t\t folio_phys_swap_backed(folio)));\n \tif (!need_reclaim || !folio_swapcache_freeable(folio))\n \t\tgoto out_unlock;\n \n@@ -328,14 +351,14 @@ offset_to_swap_extent(struct swap_info_struct *sis, unsigned long offset)\n \tBUG();\n }\n \n-sector_t swap_folio_sector(struct folio *folio)\n+sector_t swap_entry_sector(swp_entry_t entry)\n {\n-\tstruct swap_info_struct *sis = __swap_entry_to_info(folio-\u003eswap);\n+\tstruct swap_info_struct *sis = __swap_entry_to_info(entry);\n \tstruct swap_extent *se;\n \tsector_t sector;\n \tpgoff_t offset;\n \n-\toffset = swp_offset(folio-\u003eswap);\n+\toffset = swp_offset(entry);\n \tse = offset_to_swap_extent(sis, offset);\n \tsector = se-\u003estart_block + (offset - se-\u003estart_page);\n \treturn sector \u003c\u003c (PAGE_SHIFT - 9);\n@@ -401,13 +424,15 @@ static inline bool cluster_is_usable(struct swap_cluster_info *ci, int order)\n static inline unsigned int cluster_index(struct swap_info_struct *si,\n \t\t\t\t\t struct swap_cluster_info *ci)\n {\n+\tif (swap_is_vswap(si))\n+\t\treturn container_of(ci, struct swap_cluster_info_dynamic, ci)-\u003eindex;\n \treturn ci - si-\u003ecluster_info;\n }\n \n-static inline unsigned int cluster_offset(struct swap_info_struct *si,\n-\t\t\t\t\t struct swap_cluster_info *ci)\n+static inline unsigned long cluster_offset(struct swap_info_struct *si,\n+\t\t\t\t\t struct swap_cluster_info *ci)\n {\n-\treturn cluster_index(si, ci) * SWAPFILE_CLUSTER;\n+\treturn (unsigned long)cluster_index(si, ci) * SWAPFILE_CLUSTER;\n }\n \n static void swap_cluster_free_table_folio_rcu_cb(struct rcu_head *head)\n@@ -446,7 +471,8 @@ static void swap_cluster_free_table(struct swap_cluster_info *ci)\n \t\t swap_cluster_free_table_folio_rcu_cb);\n }\n \n-static int swap_cluster_alloc_table(struct swap_cluster_info *ci, gfp_t gfp)\n+static int swap_cluster_alloc_table(struct swap_info_struct *si,\n+\t\t\t\t struct swap_cluster_info *ci, gfp_t gfp)\n {\n \tstruct swap_table *table = NULL;\n \tstruct folio *folio;\n@@ -469,7 +495,14 @@ static int swap_cluster_alloc_table(struct swap_cluster_info *ci, gfp_t gfp)\n \trcu_assign_pointer(ci-\u003etable, table);\n \n #ifdef CONFIG_MEMCG\n-\tif (!mem_cgroup_disabled()) {\n+\t/*\n+\t * A physical cluster under vswap may hold only vswap backings, which\n+\t * record their memcg on the vswap cluster's table, not this one. Such\n+\t * clusters defer memcg_table allocation until they hand out a slot\n+\t * that maps directly into the PTEs.\n+\t */\n+\tif ((!vswap_is_enabled() || swap_is_vswap(si)) \u0026\u0026\n+\t !mem_cgroup_disabled()) {\n \t\tVM_WARN_ON_ONCE(ci-\u003ememcg_table);\n \t\tci-\u003ememcg_table = kzalloc_obj(*ci-\u003ememcg_table, gfp);\n \t\tif (!ci-\u003ememcg_table) {\n@@ -532,14 +565,16 @@ swap_cluster_populate(struct swap_info_struct *si,\n \t/*\n \t * Only cluster isolation from the allocator does table allocation.\n \t * Swap allocator uses percpu clusters and holds the local lock.\n+\t * vswap clusters are destroyed rather than freed to si-\u003efree_clusters.\n \t */\n+\tVM_WARN_ON_ONCE(swap_is_vswap(si));\n \tlockdep_assert_held(\u0026this_cpu_ptr(\u0026percpu_swap_cluster)-\u003elock);\n \tif (!(si-\u003eflags \u0026 SWP_SOLIDSTATE))\n \t\tlockdep_assert_held(\u0026si-\u003eglobal_cluster_lock);\n \tlockdep_assert_held(\u0026ci-\u003elock);\n \n-\tif (!swap_cluster_alloc_table(ci, __GFP_HIGH | __GFP_NOMEMALLOC |\n-\t\t\t\t\t __GFP_NOWARN))\n+\tif (!swap_cluster_alloc_table(si, ci, __GFP_HIGH | __GFP_NOMEMALLOC |\n+\t\t\t\t\t __GFP_NOWARN))\n \t\treturn ci;\n \n \t/*\n@@ -552,8 +587,8 @@ swap_cluster_populate(struct swap_info_struct *si,\n \t\tspin_unlock(\u0026si-\u003eglobal_cluster_lock);\n \tlocal_unlock(\u0026percpu_swap_cluster.lock);\n \n-\tret = swap_cluster_alloc_table(ci, __GFP_HIGH | __GFP_NOMEMALLOC |\n-\t\t\t\t\t GFP_KERNEL);\n+\tret = swap_cluster_alloc_table(si, ci, __GFP_HIGH | __GFP_NOMEMALLOC |\n+\t\t\t\t\t GFP_KERNEL);\n \n \t/*\n \t * Back to atomic context. We might have migrated to a new CPU with a\n@@ -586,10 +621,15 @@ static void move_cluster(struct swap_info_struct *si,\n \tlockdep_assert_held(\u0026ci-\u003elock);\n \n \tspin_lock(\u0026si-\u003elock);\n-\tif (ci-\u003eflags == CLUSTER_FLAG_NONE)\n+\tif (!list) {\n+\t\t/* Going away. An isolated cluster is already off its list. */\n+\t\tif (ci-\u003eflags != CLUSTER_FLAG_NONE)\n+\t\t\tlist_del(\u0026ci-\u003elist);\n+\t} else if (ci-\u003eflags == CLUSTER_FLAG_NONE) {\n \t\tlist_add_tail(\u0026ci-\u003elist, list);\n-\telse\n+\t} else {\n \t\tlist_move_tail(\u0026ci-\u003elist, list);\n+\t}\n \tspin_unlock(\u0026si-\u003elock);\n \tci-\u003eflags = new_flags;\n }\n@@ -607,6 +647,19 @@ static void __free_cluster(struct swap_info_struct *si, struct swap_cluster_info\n {\n \tswap_cluster_assert_empty(ci, 0, SWAPFILE_CLUSTER, false);\n \tswap_cluster_free_table(ci);\n+\n+\tif (swap_is_vswap(si)) {\n+\t\tstruct swap_cluster_info_dynamic *ci_dyn;\n+\n+\t\t/* vswap clusters are destroyed, not returned to free_clusters. */\n+\t\tci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);\n+\t\txa_erase(\u0026si-\u003ecluster_info_pool, ci_dyn-\u003eindex);\n+\t\tmove_cluster(si, ci, NULL, CLUSTER_FLAG_DEAD);\n+\t\tvswap_cluster_free_vtable(ci);\n+\t\tkfree_rcu(ci_dyn, rcu);\n+\t\treturn;\n+\t}\n+\n \tmove_cluster(si, ci, \u0026si-\u003efree_clusters, CLUSTER_FLAG_FREE);\n \tci-\u003eorder = 0;\n }\n@@ -795,7 +848,7 @@ static int swap_cluster_setup_bad_slot(struct swap_info_struct *si,\n \n \t/* si-\u003emax may got shrunk by swap swap_activate() */\n \tif (offset \u003e= si-\u003emax \u0026\u0026 !mask) {\n-\t\tpr_debug(\"Ignoring bad slot %u (max: %u)\\n\", offset, si-\u003emax);\n+\t\tpr_debug(\"Ignoring bad slot %u (max: %lu)\\n\", offset, si-\u003emax);\n \t\treturn 0;\n \t}\n \t/*\n@@ -812,7 +865,7 @@ static int swap_cluster_setup_bad_slot(struct swap_info_struct *si,\n \n \tci = cluster_info + idx;\n \t/* Need to allocate swap table first for initial bad slot marking. */\n-\tif (!ci-\u003ecount \u0026\u0026 swap_cluster_alloc_table(ci, GFP_KERNEL))\n+\tif (!ci-\u003ecount \u0026\u0026 swap_cluster_alloc_table(si, ci, GFP_KERNEL))\n \t\treturn -ENOMEM;\n \tspin_lock(\u0026ci-\u003elock);\n \t/* Check for duplicated bad swap slots. */\n@@ -830,6 +883,54 @@ static int swap_cluster_setup_bad_slot(struct swap_info_struct *si,\n \treturn ret;\n }\n \n+/*\n+ * Try to reclaim a Pointer-tagged physical slot backing a vswap entry.\n+ * The physical cluster lock must NOT be held. Returns the backing folio's\n+ * page count, negated if the slots could not be reclaimed, or 0 if the\n+ * folio could not be shown to own @offset (i.e. there is a race).\n+ */\n+static int try_to_reclaim_vswap_backing(struct swap_info_struct *si,\n+\t\t\t\t\tunsigned long offset,\n+\t\t\t\t\tswp_entry_t vswap_entry)\n+{\n+\tswp_entry_t phys_base;\n+\tstruct folio *folio;\n+\tunsigned int i;\n+\tint ret;\n+\n+\tfolio = swap_cache_get_folio(vswap_entry);\n+\tif (!folio)\n+\t\treturn 0;\n+\n+\tif (!folio_trylock(folio)) {\n+\t\tfolio_put(folio);\n+\t\treturn 0;\n+\t}\n+\n+\tif (!folio_matches_swap_entry(folio, vswap_entry)) {\n+\t\tfolio_unlock(folio);\n+\t\tfolio_put(folio);\n+\t\treturn 0;\n+\t}\n+\n+\ti = vswap_entry.val - folio-\u003eswap.val;\n+\tphys_base = vswap_to_phys(folio-\u003eswap);\n+\tif (!phys_base.val || swp_type(phys_base) != si-\u003etype ||\n+\t swp_offset(phys_base) + i != offset) {\n+\t\tfolio_unlock(folio);\n+\t\tfolio_put(folio);\n+\t\treturn 0;\n+\t}\n+\n+\t/* The run is ours: skip it all, whether or not the free succeeds. */\n+\tret = folio_nr_pages(folio);\n+\tif (!folio_free_swap(folio))\n+\t\tret = -ret;\n+\tfolio_unlock(folio);\n+\tfolio_put(folio);\n+\treturn ret;\n+}\n+\n /*\n * Reclaim drops the ci lock, so the cluster may become unusable (freed or\n * stolen by a lower order). @usable will be set to false if that happens.\n@@ -843,6 +944,8 @@ static bool cluster_reclaim_range(struct swap_info_struct *si,\n \tunsigned long offset = start, end = start + nr_pages;\n \tunsigned long swp_tb;\n \n+\tVM_WARN_ON_ONCE(swap_is_vswap(si));\n+\n \tspin_unlock(\u0026ci-\u003elock);\n \tdo {\n \t\tswp_tb = swap_table_get(ci, offset % SWAPFILE_CLUSTER);\n@@ -895,7 +998,8 @@ static bool cluster_scan_range(struct swap_info_struct *si,\n \t\tif (swp_tb_is_null(swp_tb))\n \t\t\tcontinue;\n \t\tif (swp_tb_is_folio(swp_tb) \u0026\u0026 !__swp_tb_get_count(swp_tb)) {\n-\t\t\tif (!vm_swap_full())\n+\t\t\t/* vswap slots are abundant; never reclaim to reuse one */\n+\t\t\tif (swap_is_vswap(si) || !vm_swap_full())\n \t\t\t\treturn false;\n \t\t\t*need_reclaim = true;\n \t\t\tcontinue;\n@@ -915,6 +1019,8 @@ static bool __swap_cluster_alloc_entries(struct swap_info_struct *si,\n {\n \tunsigned int order;\n \tunsigned long nr_pages;\n+\tswp_entry_t vswap_entry, v;\n+\tunsigned int i;\n \n \tlockdep_assert_held(\u0026ci-\u003elock);\n \n@@ -934,8 +1040,26 @@ static bool __swap_cluster_alloc_entries(struct swap_info_struct *si,\n \t\torder = folio_order(folio);\n \t\tnr_pages = 1 \u003c\u003c order;\n \t\tswap_cluster_assert_empty(ci, ci_off, nr_pages, false);\n-\t\t__swap_cache_add_folio(ci, folio, swp_entry(si-\u003etype,\n-\t\t\t\t\t\t\t ci_off + cluster_offset(si, ci)));\n+\t\tif (folio_test_swapcache(folio)) {\n+\t\t\t/*\n+\t\t\t * Folio already in the swap cache: we are allocating\n+\t\t\t * physical backing for its vswap entry. Point each\n+\t\t\t * physical slot back at its own vswap entry\n+\t\t\t * (Pointer-tagged rmap).\n+\t\t\t */\n+\t\t\tVM_WARN_ON(!is_vswap_entry(folio-\u003eswap));\n+\t\t\tvswap_entry = folio-\u003eswap;\n+\t\t\tfor (i = 0; i \u003c nr_pages; i++) {\n+\t\t\t\tv = vswap_entry;\n+\t\t\t\tv.val += i;\n+\t\t\t\t__swap_table_set(ci, ci_off + i,\n+\t\t\t\t\t\t swp_entry_to_swp_tb_ptr(v));\n+\t\t\t}\n+\t\t} else {\n+\t\t\t__swap_cache_add_folio(ci, folio,\n+\t\t\t\tswp_entry(si-\u003etype,\n+\t\t\t\t\t ci_off + cluster_offset(si, ci)));\n+\t\t}\n \t} else if (IS_ENABLED(CONFIG_HIBERNATION)) {\n \t\torder = 0;\n \t\tnr_pages = 1;\n@@ -961,11 +1085,13 @@ static bool __swap_cluster_alloc_entries(struct swap_info_struct *si,\n }\n \n /* Try use a new cluster for current CPU and allocate from it. */\n-static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si,\n-\t\t\t\t\t struct swap_cluster_info *ci,\n-\t\t\t\t\t struct folio *folio, unsigned long offset)\n+static unsigned long alloc_swap_scan_cluster(struct swap_info_struct *si,\n+\t\t\t\t\t struct swap_cluster_info *ci,\n+\t\t\t\t\t struct folio *folio,\n+\t\t\t\t\t unsigned long offset,\n+\t\t\t\t\t bool *nomem)\n {\n-\tunsigned int next = SWAP_ENTRY_INVALID, found = SWAP_ENTRY_INVALID;\n+\tunsigned long next = SWAP_ENTRY_INVALID, found = SWAP_ENTRY_INVALID;\n \tunsigned long start = ALIGN_DOWN(offset, SWAPFILE_CLUSTER);\n \tunsigned int order = likely(folio) ? folio_order(folio) : 0;\n \tunsigned long end = start + SWAPFILE_CLUSTER;\n@@ -992,6 +1118,24 @@ static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si,\n \t\t\tif (!ret)\n \t\t\t\tcontinue;\n \t\t}\n+#ifdef CONFIG_MEMCG\n+\t\t/*\n+\t\t * Lazy-allocate memcg_table on the first direct-use slot of a\n+\t\t * physical cluster.\n+\t\t */\n+\t\tif (vswap_is_enabled() \u0026\u0026 folio \u0026\u0026\n+\t\t !folio_test_swapcache(folio) \u0026\u0026 !mem_cgroup_disabled() \u0026\u0026\n+\t\t !ci-\u003ememcg_table) {\n+\t\t\tci-\u003ememcg_table = kzalloc_obj(*ci-\u003ememcg_table,\n+\t\t\t\t\t\t GFP_ATOMIC | __GFP_NOMEMALLOC |\n+\t\t\t\t\t\t __GFP_NOWARN);\n+\t\t\tif (!ci-\u003ememcg_table) {\n+\t\t\t\tif (nomem)\n+\t\t\t\t\t*nomem = true;\n+\t\t\t\tgoto out;\n+\t\t\t}\n+\t\t}\n+#endif\n \t\tif (!__swap_cluster_alloc_entries(si, ci, folio, offset % SWAPFILE_CLUSTER))\n \t\t\tbreak;\n \t\tfound = offset;\n@@ -1001,8 +1145,20 @@ static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si,\n \t\tbreak;\n \t}\n out:\n-\trelocate_cluster(si, ci);\n+\t/*\n+\t * On a discard-capable device, relocating a cluster whose memcg_table\n+\t * allocation failed queues a discard for slots that were never used,\n+\t * which folio_alloc_phys_swap() reads as progress and retries on.\n+\t */\n+\tif (nomem \u0026\u0026 *nomem \u0026\u0026 !ci-\u003ecount)\n+\t\t__free_cluster(si, ci);\n+\telse\n+\t\trelocate_cluster(si, ci);\n \tswap_cluster_unlock(ci);\n+\tif (swap_is_vswap(si)) {\n+\t\tthis_cpu_write(percpu_vswap_cluster.offset[order], next);\n+\t\treturn found;\n+\t}\n \tif (si-\u003eflags \u0026 SWP_SOLIDSTATE) {\n \t\tthis_cpu_write(percpu_swap_cluster.offset[order], next);\n \t\tthis_cpu_write(percpu_swap_cluster.si[order], si);\n@@ -1012,13 +1168,19 @@ static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si,\n \treturn found;\n }\n \n-static unsigned int alloc_swap_scan_list(struct swap_info_struct *si,\n-\t\t\t\t\t struct list_head *list,\n-\t\t\t\t\t struct folio *folio,\n-\t\t\t\t\t bool scan_all)\n+static unsigned long alloc_swap_scan_list(struct swap_info_struct *si,\n+\t\t\t\t\t struct list_head *list,\n+\t\t\t\t\t struct folio *folio,\n+\t\t\t\t\t bool scan_all)\n {\n-\tunsigned int found = SWAP_ENTRY_INVALID;\n+\tunsigned long found = SWAP_ENTRY_INVALID;\n+\tbool nomem = false;\n \n+\t/*\n+\t * In rare cases alloc_swap_scan_cluster() can fail due to\n+\t * memcg_table allocation failure. Short-circuit to avoid looping\n+\t * over the list indefinitely.\n+\t */\n \tdo {\n \t\tstruct swap_cluster_info *ci = isolate_lock_cluster(si, list);\n \t\tunsigned long offset;\n@@ -1026,19 +1188,65 @@ static unsigned int alloc_swap_scan_list(struct swap_info_struct *si,\n \t\tif (!ci)\n \t\t\tbreak;\n \t\toffset = cluster_offset(si, ci);\n-\t\tfound = alloc_swap_scan_cluster(si, ci, folio, offset);\n+\t\tfound = alloc_swap_scan_cluster(si, ci, folio, offset, \u0026nomem);\n \t\tif (found)\n \t\t\tbreak;\n-\t} while (scan_all);\n+\t} while (scan_all \u0026\u0026 !nomem);\n \n \treturn found;\n }\n \n+static unsigned long vswap_alloc_cluster(struct swap_info_struct *si,\n+\t\t\t\t\t struct folio *folio)\n+{\n+\tstruct swap_cluster_info_dynamic *ci_dyn;\n+\tstruct swap_cluster_info *ci;\n+\tunsigned long offset;\n+\n+\tVM_WARN_ON(!swap_is_vswap(si));\n+\n+\tci_dyn = kzalloc_obj(*ci_dyn, GFP_ATOMIC);\n+\tif (!ci_dyn)\n+\t\treturn SWAP_ENTRY_INVALID;\n+\n+\tspin_lock_init(\u0026ci_dyn-\u003eci.lock);\n+\tINIT_LIST_HEAD(\u0026ci_dyn-\u003eci.list);\n+\n+\tif (swap_cluster_alloc_table(si, \u0026ci_dyn-\u003eci, GFP_ATOMIC)) {\n+\t\tkfree(ci_dyn);\n+\t\treturn SWAP_ENTRY_INVALID;\n+\t}\n+\n+\tif (vswap_cluster_alloc_vtable(ci_dyn, GFP_ATOMIC)) {\n+\t\tswap_cluster_free_table(\u0026ci_dyn-\u003eci);\n+\t\tkfree(ci_dyn);\n+\t\treturn SWAP_ENTRY_INVALID;\n+\t}\n+\n+\t/* Lock before publishing: xa_alloc makes the cluster findable by offset. */\n+\tci = \u0026ci_dyn-\u003eci;\n+\tspin_lock(\u0026ci-\u003elock);\n+\n+\tif (xa_alloc(\u0026si-\u003ecluster_info_pool, \u0026ci_dyn-\u003eindex, ci_dyn,\n+\t\t XA_LIMIT(1, DIV_ROUND_UP(si-\u003emax, SWAPFILE_CLUSTER) - 1),\n+\t\t GFP_ATOMIC)) {\n+\t\tspin_unlock(\u0026ci-\u003elock);\n+\t\tswap_cluster_free_table(\u0026ci_dyn-\u003eci);\n+\t\tvswap_cluster_free_vtable(\u0026ci_dyn-\u003eci);\n+\t\tkfree(ci_dyn);\n+\t\treturn SWAP_ENTRY_INVALID;\n+\t}\n+\n+\toffset = cluster_offset(si, ci);\n+\treturn alloc_swap_scan_cluster(si, ci, folio, offset, NULL);\n+}\n+\n static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool force)\n {\n \tlong to_scan = 1;\n \tunsigned long offset, end;\n \tstruct swap_cluster_info *ci;\n+\tswp_entry_t vswap_entry;\n \tunsigned long swp_tb;\n \tint nr_reclaim;\n \n@@ -1056,7 +1264,22 @@ static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool force)\n \t\t\t\tspin_unlock(\u0026ci-\u003elock);\n \t\t\t\tnr_reclaim = __try_to_reclaim_swap(si, offset,\n \t\t\t\t\t\t\t\t TTRS_ANYWAY);\n-\t\t\t\tspin_lock(\u0026ci-\u003elock);\n+\t\t\t\tci = swap_cluster_lock(si, offset);\n+\t\t\t\tif (!ci)\n+\t\t\t\t\tgoto next;\n+\t\t\t\tif (nr_reclaim) {\n+\t\t\t\t\toffset += abs(nr_reclaim);\n+\t\t\t\t\tcontinue;\n+\t\t\t\t}\n+\t\t\t} else if (swp_tb_is_pointer(swp_tb) \u0026\u0026\n+\t\t\t\t (swp_tb \u0026 SWP_RMAP_CACHE_ONLY)) {\n+\t\t\t\tvswap_entry = swp_tb_ptr_to_swp_entry(swp_tb);\n+\t\t\t\tspin_unlock(\u0026ci-\u003elock);\n+\t\t\t\tnr_reclaim = try_to_reclaim_vswap_backing(si, offset,\n+\t\t\t\t\t\t\t\t\t vswap_entry);\n+\t\t\t\tci = swap_cluster_lock(si, offset);\n+\t\t\t\tif (!ci)\n+\t\t\t\t\tgoto next;\n \t\t\t\tif (nr_reclaim) {\n \t\t\t\t\toffset += abs(nr_reclaim);\n \t\t\t\t\tcontinue;\n@@ -1070,6 +1293,7 @@ static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool force)\n \t\t\trelocate_cluster(si, ci);\n \n \t\tswap_cluster_unlock(ci);\n+next:\n \t\tif (to_scan \u003c= 0)\n \t\t\tbreak;\n \n@@ -1100,13 +1324,13 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,\n {\n \tstruct swap_cluster_info *ci;\n \tunsigned int order = likely(folio) ? folio_order(folio) : 0;\n-\tunsigned int offset = SWAP_ENTRY_INVALID, found = SWAP_ENTRY_INVALID;\n+\tunsigned long offset = SWAP_ENTRY_INVALID, found = SWAP_ENTRY_INVALID;\n \n \t/*\n \t * Swapfile is not block device so unable\n \t * to allocate large entries.\n \t */\n-\tif (order \u0026\u0026 !(si-\u003eflags \u0026 SWP_BLKDEV))\n+\tif (order \u0026\u0026 !(si-\u003eflags \u0026 SWP_BLKDEV) \u0026\u0026 !swap_is_vswap(si))\n \t\treturn 0;\n \n \tif (!(si-\u003eflags \u0026 SWP_SOLIDSTATE)) {\n@@ -1121,7 +1345,8 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,\n \t\tif (cluster_is_usable(ci, order)) {\n \t\t\tif (cluster_is_empty(ci))\n \t\t\t\toffset = cluster_offset(si, ci);\n-\t\t\tfound = alloc_swap_scan_cluster(si, ci, folio, offset);\n+\t\t\tfound = alloc_swap_scan_cluster(si, ci, folio, offset,\n+\t\t\t\t\t\t\tNULL);\n \t\t} else {\n \t\t\tswap_cluster_unlock(ci);\n \t\t}\n@@ -1146,6 +1371,12 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,\n \t\t\tgoto done;\n \t}\n \n+\tif (swap_is_vswap(si)) {\n+\t\tfound = vswap_alloc_cluster(si, folio);\n+\t\tif (found)\n+\t\t\tgoto done;\n+\t}\n+\n \tif (!(si-\u003eflags \u0026 SWP_PAGE_DISCARD)) {\n \t\tfound = alloc_swap_scan_list(si, \u0026si-\u003efree_clusters, folio, false);\n \t\tif (found)\n@@ -1153,13 +1384,14 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,\n \t}\n \n \t/* Try reclaim full clusters if free and nonfull lists are drained */\n-\tif (vm_swap_full())\n+\tif (!swap_is_vswap(si) \u0026\u0026 vm_swap_full())\n \t\tswap_reclaim_full_clusters(si, false);\n \n \tif (order \u003c PMD_ORDER) {\n \t\t/*\n \t\t * Scan only one fragment cluster is good enough. Order 0\n-\t\t * allocation will surely success, and large allocation\n+\t\t * allocation will surely success unless the memcg table\n+\t\t * allocation fails, which is rare, and large allocation\n \t\t * failure is not critical. Scanning one cluster still\n \t\t * keeps the list rotated and reclaimed (for clean swap cache).\n \t\t */\n@@ -1175,7 +1407,8 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,\n \tfor (int o = 1; o \u003c SWAP_NR_ORDERS; o++) {\n \t\t/*\n \t\t * Clusters here have at least one usable slots and can't fail order 0\n-\t\t * allocation, but reclaim may drop si-\u003elock and race with another user.\n+\t\t * allocation, but reclaim may drop si-\u003elock and race with another user,\n+\t\t * and the memcg table allocation may fail.\n \t\t */\n \t\tfound = alloc_swap_scan_list(si, \u0026si-\u003efrag_clusters[o], folio, true);\n \t\tif (found)\n@@ -1282,8 +1515,10 @@ static bool swap_usage_add(struct swap_info_struct *si, unsigned int nr_entries)\n \t/*\n \t * If device is full, and SWAP_USAGE_OFFLIST_BIT is not set,\n \t * remove it from the plist.\n+\t *\n+\t * Vswap is never on the avail list, so skip it.\n \t */\n-\tif (unlikely(val == si-\u003epages)) {\n+\tif (unlikely(val == si-\u003epages) \u0026\u0026 !swap_is_vswap(si)) {\n \t\tdel_from_avail_list(si, false);\n \t\treturn true;\n \t}\n@@ -1298,8 +1533,10 @@ static void swap_usage_sub(struct swap_info_struct *si, unsigned int nr_entries)\n \t/*\n \t * If device is not full, and SWAP_USAGE_OFFLIST_BIT is set,\n \t * add it to the plist.\n+\t *\n+\t * Vswap is never on the avail list, so skip it.\n \t */\n-\tif (unlikely(val \u0026 SWAP_USAGE_OFFLIST_BIT))\n+\tif (unlikely(val \u0026 SWAP_USAGE_OFFLIST_BIT) \u0026\u0026 !swap_is_vswap(si))\n \t\tadd_to_avail_list(si, false);\n }\n \n@@ -1310,7 +1547,8 @@ static void swap_range_alloc(struct swap_info_struct *si,\n \t\tif (vm_swap_full())\n \t\t\tschedule_work(\u0026si-\u003ereclaim_work);\n \t}\n-\tatomic_long_sub(nr_entries, \u0026nr_swap_pages);\n+\tif (!swap_is_vswap(si))\n+\t\tatomic_long_sub(nr_entries, \u0026nr_swap_pages);\n }\n \n static void swap_range_free(struct swap_info_struct *si, unsigned long offset,\n@@ -1320,8 +1558,10 @@ static void swap_range_free(struct swap_info_struct *si, unsigned long offset,\n \tvoid (*swap_slot_free_notify)(struct block_device *, unsigned long);\n \tunsigned int i;\n \n-\tfor (i = 0; i \u003c nr_entries; i++)\n-\t\tzswap_invalidate(swp_entry(si-\u003etype, offset + i));\n+\tif (!swap_is_vswap(si)) {\n+\t\tfor (i = 0; i \u003c nr_entries; i++)\n+\t\t\tzswap_invalidate(swp_entry(si-\u003etype, offset + i));\n+\t}\n \n \tif (si-\u003eflags \u0026 SWP_BLKDEV)\n \t\tswap_slot_free_notify =\n@@ -1340,12 +1580,17 @@ static void swap_range_free(struct swap_info_struct *si, unsigned long offset,\n \t * only after the above cleanups are done.\n \t */\n \tsmp_wmb();\n-\tatomic_long_add(nr_entries, \u0026nr_swap_pages);\n+\tif (!swap_is_vswap(si))\n+\t\tatomic_long_add(nr_entries, \u0026nr_swap_pages);\n \tswap_usage_sub(si, nr_entries);\n }\n \n static bool get_swap_device_info(struct swap_info_struct *si)\n {\n+\t/* The vswap device is always alive, so it needs no refcount. */\n+\tif (swap_is_vswap(si))\n+\t\treturn true;\n+\n \tif (!percpu_ref_tryget_live(\u0026si-\u003eusers))\n \t\treturn false;\n \t/*\n@@ -1364,12 +1609,14 @@ static bool get_swap_device_info(struct swap_info_struct *si)\n * Fast path try to get swap entries with specified order from current\n * CPU's swap entry pool (a cluster).\n */\n-static bool swap_alloc_fast(struct folio *folio)\n+static swp_entry_t swap_alloc_fast(struct folio *folio)\n {\n \tunsigned int order = folio_order(folio);\n \tstruct swap_cluster_info *ci;\n \tstruct swap_info_struct *si;\n-\tunsigned int offset;\n+\tunsigned long offset, found = 0;\n+\n+\tlockdep_assert_held(\u0026this_cpu_ptr(\u0026percpu_swap_cluster)-\u003elock);\n \n \t/*\n \t * Once allocated, swap_info_struct will never be completely freed,\n@@ -1378,25 +1625,28 @@ static bool swap_alloc_fast(struct folio *folio)\n \tsi = this_cpu_read(percpu_swap_cluster.si[order]);\n \toffset = this_cpu_read(percpu_swap_cluster.offset[order]);\n \tif (!si || !offset || !get_swap_device_info(si))\n-\t\treturn false;\n+\t\treturn (swp_entry_t){};\n \n \tci = swap_cluster_lock(si, offset);\n-\tif (cluster_is_usable(ci, order)) {\n+\tif (ci \u0026\u0026 cluster_is_usable(ci, order)) {\n \t\tif (cluster_is_empty(ci))\n \t\t\toffset = cluster_offset(si, ci);\n-\t\talloc_swap_scan_cluster(si, ci, folio, offset);\n-\t} else {\n+\t\tfound = alloc_swap_scan_cluster(si, ci, folio, offset, NULL);\n+\t} else if (ci) {\n \t\tswap_cluster_unlock(ci);\n \t}\n \n \tput_swap_device(si);\n-\treturn folio_test_swapcache(folio);\n+\tif (found)\n+\t\treturn swp_entry(si-\u003etype, found);\n+\treturn (swp_entry_t){};\n }\n \n /* Rotate the device and switch to a new cluster */\n-static void swap_alloc_slow(struct folio *folio)\n+static swp_entry_t swap_alloc_slow(struct folio *folio)\n {\n \tstruct swap_info_struct *si, *next;\n+\tunsigned long found;\n \n \tspin_lock(\u0026swap_avail_lock);\n start_over:\n@@ -1405,12 +1655,12 @@ static void swap_alloc_slow(struct folio *folio)\n \t\tplist_requeue(\u0026si-\u003eavail_list, \u0026swap_avail_head);\n \t\tspin_unlock(\u0026swap_avail_lock);\n \t\tif (get_swap_device_info(si)) {\n-\t\t\tcluster_alloc_swap_entry(si, folio);\n+\t\t\tfound = cluster_alloc_swap_entry(si, folio);\n \t\t\tput_swap_device(si);\n-\t\t\tif (folio_test_swapcache(folio))\n-\t\t\t\treturn;\n+\t\t\tif (found)\n+\t\t\t\treturn swp_entry(si-\u003etype, found);\n \t\t\tif (folio_test_large(folio))\n-\t\t\t\treturn;\n+\t\t\t\treturn (swp_entry_t){};\n \t\t}\n \n \t\tspin_lock(\u0026swap_avail_lock);\n@@ -1428,6 +1678,7 @@ static void swap_alloc_slow(struct folio *folio)\n \t\t\tgoto start_over;\n \t}\n \tspin_unlock(\u0026swap_avail_lock);\n+\treturn (swp_entry_t){};\n }\n \n /*\n@@ -1507,6 +1758,7 @@ int swap_retry_table_alloc(swp_entry_t entry, gfp_t gfp)\n \tif (!si)\n \t\treturn 0;\n \n+\t/* The source PTE pins the entry, so its cluster is alive. */\n \tci = __swap_offset_to_cluster(si, offset);\n \tret = swap_extend_table_alloc(si, ci, swp_cluster_offset(entry), gfp);\n \n@@ -1611,6 +1863,8 @@ static void swap_put_entries_cluster(struct swap_info_struct *si,\n \t\t\t}\n \t\t\t/* count will be 0 after put, slot can be reclaimed */\n \t\t\tneed_reclaim = true;\n+\t\t\tif (swap_is_vswap(si))\n+\t\t\t\tvswap_mark_cache_only(ci, ci_off);\n \t\t}\n \t\t/*\n \t\t * A count != 1 or cached slot can't be freed. Put its swap\n@@ -1717,6 +1971,8 @@ static int swap_dup_entries_cluster(struct swap_info_struct *si,\n \t\t\tgoto failed;\n \t\t}\n \t} while (++ci_off \u003c ci_end);\n+\tif (swap_is_vswap(si))\n+\t\tvswap_clear_cache_only(ci, ci_start, nr);\n \tswap_cluster_unlock(ci);\n \treturn 0;\n failed:\n@@ -1727,6 +1983,76 @@ static int swap_dup_entries_cluster(struct swap_info_struct *si,\n \treturn err;\n }\n \n+static bool vswap_alloc(struct folio *folio)\n+{\n+\tunsigned int order = folio_order(folio);\n+\tstruct swap_cluster_info *ci;\n+\tstruct obj_cgroup *objcg;\n+\tunsigned long offset;\n+\tbool may_zswap;\n+\n+\tif (!vswap_is_enabled() || !zswap_is_enabled())\n+\t\treturn false;\n+\n+\t/*\n+\t * If zswap will not take the folio, writeout has to find a physical\n+\t * slot anyway. We are just incurring indirection overhead\n+\t * unnecessarily.\n+\t */\n+\tobjcg = get_obj_cgroup_from_folio(folio);\n+\tmay_zswap = !objcg || obj_cgroup_may_zswap(objcg);\n+\tif (objcg)\n+\t\tobj_cgroup_put(objcg);\n+\tif (!may_zswap)\n+\t\treturn false;\n+\n+\tlocal_lock(\u0026percpu_vswap_cluster.lock);\n+\toffset = this_cpu_read(percpu_vswap_cluster.offset[order]);\n+\n+\tif (offset != SWAP_ENTRY_INVALID) {\n+\t\tci = swap_cluster_lock(vswap_si, offset);\n+\t\tif (ci \u0026\u0026 cluster_is_usable(ci, order)) {\n+\t\t\tif (cluster_is_empty(ci))\n+\t\t\t\toffset = cluster_offset(vswap_si, ci);\n+\t\t\talloc_swap_scan_cluster(vswap_si, ci, folio, offset,\n+\t\t\t\t\t\tNULL);\n+\t\t} else if (ci) {\n+\t\t\tswap_cluster_unlock(ci);\n+\t\t}\n+\t}\n+\n+\tif (!folio_test_swapcache(folio))\n+\t\tcluster_alloc_swap_entry(vswap_si, folio);\n+\n+\tif (folio_test_swapcache(folio)) {\n+\t\t/* alloc_swap_scan_cluster updated percpu offset already */\n+\t\tlocal_unlock(\u0026percpu_vswap_cluster.lock);\n+\t\treturn true;\n+\t}\n+\n+\tthis_cpu_write(percpu_vswap_cluster.offset[order], SWAP_ENTRY_INVALID);\n+\tlocal_unlock(\u0026percpu_vswap_cluster.lock);\n+\tatomic_long_add(folio_nr_pages(folio), \u0026vswap_alloc_reject);\n+\treturn false;\n+}\n+\n+static swp_entry_t folio_alloc_phys_swap(struct folio *folio)\n+{\n+\tswp_entry_t entry;\n+\n+again:\n+\tlocal_lock(\u0026percpu_swap_cluster.lock);\n+\tentry = swap_alloc_fast(folio);\n+\tif (!entry.val)\n+\t\tentry = swap_alloc_slow(folio);\n+\tlocal_unlock(\u0026percpu_swap_cluster.lock);\n+\n+\tif (!entry.val \u0026\u0026 !folio_order(folio) \u0026\u0026 swap_sync_discard())\n+\t\tgoto again;\n+\n+\treturn entry;\n+}\n+\n /**\n * folio_alloc_swap - allocate swap space for a folio\n * @folio: folio we want to move to swap\n@@ -1740,6 +2066,7 @@ static int swap_dup_entries_cluster(struct swap_info_struct *si,\n int folio_alloc_swap(struct folio *folio)\n {\n \tunsigned int order = folio_order(folio);\n+\tstruct mem_cgroup *memcg;\n \tunsigned int size = 1 \u003c\u003c order;\n \n \tVM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);\n@@ -1763,27 +2090,350 @@ int folio_alloc_swap(struct folio *folio)\n \t\t}\n \t}\n \n-again:\n-\tlocal_lock(\u0026percpu_swap_cluster.lock);\n-\tif (!swap_alloc_fast(folio))\n-\t\tswap_alloc_slow(folio);\n-\tlocal_unlock(\u0026percpu_swap_cluster.lock);\n+\tif (!vswap_alloc(folio))\n+\t\tfolio_alloc_phys_swap(folio);\n \n-\tif (!order \u0026\u0026 unlikely(!folio_test_swapcache(folio))) {\n-\t\tif (swap_sync_discard())\n-\t\t\tgoto again;\n+\t/*\n+\t * Need to call this even if allocation failed, for MEMCG_SWAP_FAIL.\n+\t * A vswap entry has no physical swap yet, so only record the memcg.\n+\t * folio_realloc_swap() charges it once backing is allocated.\n+\t */\n+\tmemcg = mem_cgroup_swap_get(folio);\n+\tif (memcg) {\n+\t\tif (!is_vswap_entry(folio-\u003eswap) \u0026\u0026\n+\t\t unlikely(mem_cgroup_swap_charge(memcg, size))) {\n+\t\t\tmem_cgroup_swap_put(memcg, size);\n+\t\t\tswap_cache_del_folio(folio);\n+\t\t} else {\n+\t\t\tmem_cgroup_swap_record(folio, memcg);\n+\t\t}\n \t}\n \n-\t/* Need to call this even if allocation failed, for MEMCG_SWAP_FAIL. */\n-\tif (unlikely(mem_cgroup_try_charge_swap(folio)))\n-\t\tswap_cache_del_folio(folio);\n-\n \tif (unlikely(!folio_test_swapcache(folio)))\n \t\treturn -ENOMEM;\n \n \treturn 0;\n }\n \n+static void vswap_mark_cache_only(struct swap_cluster_info *ci,\n+\t\t\t\t unsigned int ci_off)\n+{\n+\tstruct swap_cluster_info_dynamic *ci_dyn;\n+\tstruct swap_cluster_info *pci;\n+\tswp_entry_t phys;\n+\tunsigned long vt;\n+\n+\tci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);\n+\tvt = __vtable_get(ci_dyn, ci_off);\n+\n+\tif (vtable_type(vt) == VSWAP_SWAPFILE) {\n+\t\tphys = vtable_to_phys(vt);\n+\t\tpci = __swap_entry_to_cluster(phys);\n+\t\tswap_rmap_mark_cache_only(pci, swp_cluster_offset(phys));\n+\t}\n+}\n+\n+/*\n+ * Clear the cache-only rmap hint for entries re-referenced from count 0 to 1\n+ * (no longer reclaimable), so the physical reclaim scanner skips them.\n+ */\n+static void vswap_clear_cache_only(struct swap_cluster_info *ci,\n+\t\t\t\t unsigned int ci_start, int nr)\n+{\n+\tstruct swap_cluster_info_dynamic *ci_dyn;\n+\tstruct swap_cluster_info *pci;\n+\tunsigned long swp_tb, vt;\n+\tswp_entry_t phys;\n+\tunsigned int off;\n+\n+\tci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);\n+\tfor (off = ci_start; off \u003c ci_start + nr; off++) {\n+\t\tswp_tb = __swap_table_get(ci, off);\n+\t\tif (!swp_tb_is_folio(swp_tb) || swp_tb_get_count(swp_tb) != 1)\n+\t\t\tcontinue;\n+\t\tvt = __vtable_get(ci_dyn, off);\n+\t\tif (vtable_type(vt) != VSWAP_SWAPFILE)\n+\t\t\tcontinue;\n+\t\tphys = vtable_to_phys(vt);\n+\t\tpci = __swap_entry_to_cluster(phys);\n+\t\tswap_rmap_clear_cache_only(pci, swp_cluster_offset(phys));\n+\t}\n+}\n+\n+static void __swap_cluster_free_phys_backing(struct swap_info_struct *psi,\n+\t\t\t\t\t struct swap_cluster_info *pci,\n+\t\t\t\t\t unsigned int ci_start,\n+\t\t\t\t\t unsigned int nr_pages);\n+\n+static void vswap_uncharge_cgroup_batch(unsigned short memcg_id,\n+\t\t\t\t\tunsigned int batch_nr,\n+\t\t\t\t\tunsigned int batch_nr_swapfile)\n+{\n+\tstruct mem_cgroup *memcg;\n+\tunsigned int n;\n+\n+\t/*\n+\t * v1 (memsw): entries keep their memsw charge across swapout\n+\t * regardless of backing, so uncharge all of them. v2: only\n+\t * swapfile-backed entries are charged, so uncharge just those.\n+\t *\n+\t * On v1 the id is written by __memcg1_swapout() as the folio leaves the\n+\t * swap cache and cleared by memcg1_swapin() when it comes back, both\n+\t * under the cluster lock. Callers still holding a cached folio are\n+\t * outside that window and see @memcg_id == 0, so only the free path\n+\t * uncharges. On v2 the id is set when swap is allocated, so those\n+\t * callers do uncharge, which balances the charge folio_realloc_swap()\n+\t * took.\n+\t */\n+\tn = do_memsw_account() ? batch_nr : batch_nr_swapfile;\n+\tif (!n)\n+\t\treturn;\n+\n+\trcu_read_lock();\n+\tmemcg = memcg_id ? mem_cgroup_from_private_id(memcg_id) : NULL;\n+\trcu_read_unlock();\n+\tmem_cgroup_swap_uncharge(memcg, n);\n+}\n+\n+/**\n+ * __vswap_release_backing - release the backing of a range of vtable slots\n+ * @ci: the locked vswap cluster\n+ * @ci_start: first slot offset within @ci\n+ * @nr: number of slots\n+ *\n+ * Releases the backing of each slot in [@ci_start, @ci_start + @nr).\n+ * Clears the zero marks if set.\n+ *\n+ * Context: caller must hold @ci-\u003elock.\n+ */\n+void __vswap_release_backing(struct swap_cluster_info *ci,\n+\t\t\t unsigned int ci_start, unsigned int nr)\n+{\n+\tstruct swap_cluster_info_dynamic *ci_dyn;\n+\tstruct swap_info_struct *psi;\n+\tunsigned long phys_start = 0, phys_end = 0;\n+\tunsigned int phys_type = 0;\n+\tunsigned int ci_off;\n+\tunsigned long vt;\n+\tswp_entry_t phys;\n+\tunsigned short batch_id;\n+\tunsigned int batch_nr = 0, batch_nr_swapfile = 0;\n+\n+\tlockdep_assert_held(\u0026ci-\u003elock);\n+\tci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);\n+\tbatch_id = __swap_cgroup_get(ci, ci_start);\n+\n+\tfor (ci_off = ci_start; ci_off \u003c ci_start + nr; ci_off++) {\n+\t\tunsigned short cur_id;\n+\n+\t\tvt = __vtable_get(ci_dyn, ci_off);\n+\t\tcur_id = __swap_cgroup_get(ci, ci_off);\n+\n+\t\tif (cur_id != batch_id) {\n+\t\t\tvswap_uncharge_cgroup_batch(batch_id, batch_nr,\n+\t\t\t\t\t\t batch_nr_swapfile);\n+\t\t\tbatch_id = cur_id;\n+\t\t\tbatch_nr = 0;\n+\t\t\tbatch_nr_swapfile = 0;\n+\t\t}\n+\t\tbatch_nr++;\n+\n+\t\t/* The free helper takes one contiguous run within one cluster. */\n+\t\tif (phys_start != phys_end \u0026\u0026\n+\t\t (vtable_type(vt) != VSWAP_SWAPFILE ||\n+\t\t swp_type(vtable_to_phys(vt)) != phys_type ||\n+\t\t swp_offset(vtable_to_phys(vt)) != phys_end ||\n+\t\t phys_end % SWAPFILE_CLUSTER == 0)) {\n+\t\t\tpsi = __swap_type_to_info(phys_type);\n+\t\t\t__swap_cluster_free_phys_backing(psi,\n+\t\t\t\t__swap_entry_to_cluster(\n+\t\t\t\t\tswp_entry(phys_type, phys_start)),\n+\t\t\t\tphys_start % SWAPFILE_CLUSTER,\n+\t\t\t\tphys_end - phys_start);\n+\t\t\tphys_start = phys_end = 0;\n+\t\t}\n+\n+\t\tswitch (vtable_type(vt)) {\n+\t\tcase VSWAP_SWAPFILE:\n+\t\t\tbatch_nr_swapfile++;\n+\t\t\tif (phys_start == phys_end) {\n+\t\t\t\tphys = vtable_to_phys(vt);\n+\t\t\t\tphys_start = swp_offset(phys);\n+\t\t\t\tphys_end = phys_start + 1;\n+\t\t\t\tphys_type = swp_type(phys);\n+\t\t\t} else {\n+\t\t\t\tphys_end++;\n+\t\t\t}\n+\t\t\tbreak;\n+\t\tcase VSWAP_ZSWAP:\n+\t\t\tzswap_entry_free(vtable_to_zswap(vt));\n+\t\t\tbreak;\n+\t\tcase VSWAP_NONE:\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\t/* VSWAP_ZERO/VSWAP_FOLIO are return-only, not vtable tags */\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\t__vtable_set(ci_dyn, ci_off, VSWAP_NONE);\n+\t\t/* Zero-backed state lives in swap_table; clear it too. */\n+\t\tif (__swap_table_test_zero(ci, ci_off))\n+\t\t\t__swap_table_clear_zero(ci, ci_off);\n+\t}\n+\n+\tif (phys_start != phys_end) {\n+\t\tpsi = __swap_type_to_info(phys_type);\n+\t\t__swap_cluster_free_phys_backing(psi,\n+\t\t\t__swap_entry_to_cluster(\n+\t\t\t\tswp_entry(phys_type, phys_start)),\n+\t\t\tphys_start % SWAPFILE_CLUSTER,\n+\t\t\tphys_end - phys_start);\n+\t}\n+\n+\tvswap_uncharge_cgroup_batch(batch_id, batch_nr, batch_nr_swapfile);\n+}\n+\n+/**\n+ * folio_release_vswap_backing() - Drop all backing for a folio's vswap entry.\n+ * @folio: the folio, occupying a virtual swap entry.\n+ *\n+ * Release whatever backing the folio's virtual swap slots currently hold and\n+ * reset them to empty, so a fresh backing can be installed. Used when a\n+ * folio's swap backend is replaced.\n+ *\n+ * Context: Caller must hold the folio lock; @folio must be in the swap cache\n+ * and occupy a virtual swap entry.\n+ */\n+void folio_release_vswap_backing(struct folio *folio)\n+{\n+\tstruct swap_cluster_info *ci;\n+\tint nr = folio_nr_pages(folio);\n+\tunsigned int voff;\n+\n+\tci = __swap_entry_to_cluster(folio-\u003eswap);\n+\tvoff = swp_cluster_offset(folio-\u003eswap);\n+\n+\tspin_lock(\u0026ci-\u003elock);\n+\t__vswap_release_backing(ci, voff, nr);\n+\tspin_unlock(\u0026ci-\u003elock);\n+}\n+\n+/**\n+ * folio_release_non_phys_swap_backing() - Drop a folio's non-physical vswap backing.\n+ * @folio: the folio, occupying a virtual swap entry.\n+ *\n+ * Release the zswap backing recorded for @folio's virtual swap entry,\n+ * leaving the slots empty so the writeout path can install fresh physical\n+ * backing. Does nothing when the entry is already backed by physical\n+ * swapfile slots, which are kept for reuse, or when it has no backing\n+ * beyond the swap cache folio itself.\n+ *\n+ * Context: Caller must hold the folio lock; @folio must be in the swap cache\n+ * and occupy a virtual swap entry.\n+ */\n+void folio_release_non_phys_swap_backing(struct folio *folio)\n+{\n+\tstruct swap_cluster_info *ci;\n+\tstruct swap_cluster_info_dynamic *ci_dyn;\n+\tint nr = folio_nr_pages(folio);\n+\tunsigned int voff;\n+\tunsigned long vt;\n+\tenum vswap_backing_type type;\n+\n+\tci = __swap_entry_to_cluster(folio-\u003eswap);\n+\tci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);\n+\tvoff = swp_cluster_offset(folio-\u003eswap);\n+\n+\tspin_lock(\u0026ci-\u003elock);\n+\t/* A folio's slots cannot mix swapfile with other backends. */\n+\tvt = __vtable_get(ci_dyn, voff);\n+\ttype = vtable_type(vt);\n+\n+\tif (type == VSWAP_SWAPFILE || type == VSWAP_NONE) {\n+\t\tspin_unlock(\u0026ci-\u003elock);\n+\t\treturn;\n+\t}\n+\n+\t__vswap_release_backing(ci, voff, nr);\n+\tspin_unlock(\u0026ci-\u003elock);\n+}\n+\n+/**\n+ * folio_realloc_swap() - Back a virtual swap folio with a physical swap slot.\n+ * @folio: the folio, occupying a virtual swap entry.\n+ *\n+ * Ensure @folio's virtual swap entry has physical (swapfile) backing,\n+ * allocating a physical slot on demand if it has none. If @folio is\n+ * already physically backed, the existing physical entry is returned\n+ * unchanged.\n+ *\n+ * Context: Caller must hold the folio lock; @folio must be in the swap cache\n+ * and occupy a virtual swap entry.\n+ * Return: The physical swap entry now backing @folio, or an empty entry\n+ * (.val == 0) on failure.\n+ */\n+swp_entry_t folio_realloc_swap(struct folio *folio)\n+{\n+\tswp_entry_t vswap_entry = folio-\u003eswap;\n+\tstruct swap_cluster_info *ci;\n+\tstruct swap_cluster_info_dynamic *ci_dyn;\n+\tstruct mem_cgroup *memcg;\n+\tunsigned int voff;\n+\tunsigned long vt;\n+\tunsigned short memcg_id;\n+\tswp_entry_t phys_entry = {};\n+\tswp_entry_t pe;\n+\tint i, nr = folio_nr_pages(folio);\n+\n+\tVM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);\n+\tVM_BUG_ON_FOLIO(!folio_test_swapcache(folio), folio);\n+\tVM_WARN_ON(!is_vswap_entry(vswap_entry));\n+\n+\tvoff = swp_cluster_offset(vswap_entry);\n+\tci = __swap_entry_to_cluster(vswap_entry);\n+\tci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);\n+\n+\tspin_lock(\u0026ci-\u003elock);\n+\tvt = __vtable_get(ci_dyn, voff);\n+\tif (vtable_type(vt) == VSWAP_SWAPFILE) {\n+\t\tspin_unlock(\u0026ci-\u003elock);\n+\t\treturn vtable_to_phys(vt);\n+\t}\n+\tmemcg_id = __swap_cgroup_get(ci, voff);\n+\tspin_unlock(\u0026ci-\u003elock);\n+\n+\tphys_entry = folio_alloc_phys_swap(folio);\n+\tif (!phys_entry.val)\n+\t\treturn (swp_entry_t){};\n+\n+\trcu_read_lock();\n+\tmemcg = folio_memcg(folio);\n+\tif (!memcg || mem_cgroup_private_id(memcg) != memcg_id)\n+\t\tmemcg = memcg_id ? mem_cgroup_from_private_id(memcg_id) : NULL;\n+\trcu_read_unlock();\n+\n+\tif (mem_cgroup_swap_charge(memcg, nr)) {\n+\t\t__swap_cluster_free_phys_backing(__swap_entry_to_info(phys_entry),\n+\t\t\t\t\t\t __swap_entry_to_cluster(phys_entry),\n+\t\t\t\t\t\t swp_cluster_offset(phys_entry),\n+\t\t\t\t\t\t nr);\n+\t\treturn (swp_entry_t){};\n+\t}\n+\n+\tspin_lock(\u0026ci-\u003elock);\n+\t/*\n+\t * Install PHYS backing without freeing any prior contents of the\n+\t * vtable. Releasing the old backing is the caller's job.\n+\t */\n+\tfor (i = 0; i \u003c nr; i++) {\n+\t\tpe.val = phys_entry.val + i;\n+\t\t__vtable_set(ci_dyn, voff + i, vtable_mk_phys(pe));\n+\t}\n+\tspin_unlock(\u0026ci-\u003elock);\n+\n+\treturn phys_entry;\n+}\n+\n /**\n * folio_dup_swap() - Increase swap count of swap entries of a folio.\n * @folio: folio with swap entries bounded.\n@@ -1904,10 +2554,70 @@ struct swap_info_struct *get_swap_device(swp_entry_t entry)\n \treturn NULL;\n put_out:\n \tpr_err_ratelimited(\"%s: %s%08lx\\n\", __func__, Bad_offset, entry.val);\n-\tpercpu_ref_put(\u0026si-\u003eusers);\n+\tput_swap_device(si);\n \treturn NULL;\n }\n \n+/*\n+ * Common tail for freeing swap slots: device-level accounting\n+ * and cluster list management.\n+ */\n+static void __swap_cluster_finish_free(struct swap_info_struct *si,\n+\t\t\t\t struct swap_cluster_info *ci,\n+\t\t\t\t unsigned int ci_start,\n+\t\t\t\t unsigned int nr_pages)\n+{\n+\tlockdep_assert_held(\u0026ci-\u003elock);\n+\tswap_range_free(si, cluster_offset(si, ci) + ci_start, nr_pages);\n+\tswap_cluster_assert_empty(ci, ci_start, nr_pages, false);\n+\n+\tif (!ci-\u003ecount)\n+\t\tfree_cluster(si, ci);\n+\telse\n+\t\tpartial_free_cluster(si, ci);\n+}\n+\n+/*\n+ * Free physical swap slots that were backing vswap entries (Pointer-tagged).\n+ */\n+static void __swap_cluster_free_phys_backing(struct swap_info_struct *psi,\n+\t\t\t\t\t struct swap_cluster_info *pci,\n+\t\t\t\t\t unsigned int ci_start,\n+\t\t\t\t\t unsigned int nr_pages)\n+{\n+\tunsigned int ci_off;\n+\n+\tspin_lock_nested(\u0026pci-\u003elock, SINGLE_DEPTH_NESTING);\n+\tVM_WARN_ON(pci-\u003ecount \u003c nr_pages);\n+\tpci-\u003ecount -= nr_pages;\n+\tfor (ci_off = ci_start; ci_off \u003c ci_start + nr_pages; ci_off++) {\n+\t\t__swap_table_set(pci, ci_off, null_to_swp_tb());\n+\t\tif (!SWAP_TABLE_HAS_ZEROFLAG)\n+\t\t\t__swap_table_clear_zero(pci, ci_off);\n+\t}\n+\t__swap_cluster_finish_free(psi, pci, ci_start, nr_pages);\n+\tswap_cluster_unlock(pci);\n+}\n+\n+/*\n+ * Release the cgroup accounting of a batch of freed slots. For vswap the\n+ * physical swap was already uncharged by __vswap_release_backing(), so only\n+ * the ID ref is left to drop.\n+ */\n+static void memcg_swap_free(unsigned short id, unsigned int nr, bool is_vswap)\n+{\n+\tstruct mem_cgroup *memcg;\n+\n+\trcu_read_lock();\n+\tmemcg = mem_cgroup_from_private_id(id);\n+\tif (memcg) {\n+\t\tif (!is_vswap)\n+\t\t\tmem_cgroup_swap_uncharge(memcg, nr);\n+\t\tmem_cgroup_swap_put(memcg, nr);\n+\t}\n+\trcu_read_unlock();\n+}\n+\n /*\n * Free a set of swap slots after their swap count dropped to zero, or will be\n * zero after putting the last ref (saves one __swap_cluster_put_entry call).\n@@ -1919,11 +2629,14 @@ void __swap_cluster_free_entries(struct swap_info_struct *si,\n \tunsigned long old_tb;\n \tunsigned short batch_id = 0, id_cur;\n \tunsigned int ci_off = ci_start, ci_end = ci_start + nr_pages;\n-\tunsigned long ci_head = cluster_offset(si, ci);\n \tunsigned int batch_off = ci_off;\n+\tbool is_vswap = swap_is_vswap(si);\n \n \tVM_WARN_ON(ci-\u003ecount \u003c nr_pages);\n \n+\tif (is_vswap)\n+\t\t__vswap_release_backing(ci, ci_start, nr_pages);\n+\n \tci-\u003ecount -= nr_pages;\n \tdo {\n \t\told_tb = __swap_table_get(ci, ci_off);\n@@ -1945,22 +2658,16 @@ void __swap_cluster_free_entries(struct swap_info_struct *si,\n \t\tid_cur = __swap_cgroup_clear(ci, ci_off, 1);\n \t\tif (batch_id != id_cur) {\n \t\t\tif (batch_id)\n-\t\t\t\tmem_cgroup_uncharge_swap(batch_id, ci_off - batch_off);\n+\t\t\t\tmemcg_swap_free(batch_id, ci_off - batch_off, is_vswap);\n \t\t\tbatch_id = id_cur;\n \t\t\tbatch_off = ci_off;\n \t\t}\n \t} while (++ci_off \u003c ci_end);\n \n \tif (batch_id)\n-\t\tmem_cgroup_uncharge_swap(batch_id, ci_off - batch_off);\n-\n-\tswap_range_free(si, ci_head + ci_start, nr_pages);\n-\tswap_cluster_assert_empty(ci, ci_start, nr_pages, false);\n+\t\tmemcg_swap_free(batch_id, ci_off - batch_off, is_vswap);\n \n-\tif (!ci-\u003ecount)\n-\t\tfree_cluster(si, ci);\n-\telse\n-\t\tpartial_free_cluster(si, ci);\n+\t__swap_cluster_finish_free(si, ci, ci_start, nr_pages);\n }\n \n int __swap_count(swp_entry_t entry)\n@@ -2036,6 +2743,7 @@ static bool folio_maybe_swapped(struct folio *folio)\n \tVM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);\n \tVM_WARN_ON_ONCE_FOLIO(!folio_test_swapcache(folio), folio);\n \n+\t/* Folio is locked and in swap cache, so ci-\u003ecount \u003e 0: cluster is alive. */\n \tci = __swap_entry_to_cluster(entry);\n \tci_off = swp_cluster_offset(entry);\n \tci_end = ci_off + folio_nr_pages(folio);\n@@ -2174,7 +2882,8 @@ swp_entry_t swap_alloc_hibernation_slot(int type)\n \tif (pcp_si == si \u0026\u0026 pcp_offset) {\n \t\tci = swap_cluster_lock(si, pcp_offset);\n \t\tif (cluster_is_usable(ci, 0))\n-\t\t\toffset = alloc_swap_scan_cluster(si, ci, NULL, pcp_offset);\n+\t\t\toffset = alloc_swap_scan_cluster(si, ci, NULL,\n+\t\t\t\t\t\t\t pcp_offset, NULL);\n \t\telse\n \t\t\tswap_cluster_unlock(ci);\n \t}\n@@ -2230,6 +2939,9 @@ static int __find_hibernation_swap_type(dev_t device, sector_t offset)\n \n \t\tif (!(sis-\u003eflags \u0026 SWP_WRITEOK))\n \t\t\tcontinue;\n+\t\t/* vswap has no bdev, so it is never a hibernation target. */\n+\t\tif (swap_is_vswap(sis))\n+\t\t\tcontinue;\n \n \t\tif (device == sis-\u003ebdev-\u003ebd_dev) {\n \t\t\tstruct swap_extent *se = first_se(sis);\n@@ -2356,6 +3068,9 @@ int find_first_swap(dev_t *device)\n \n \t\tif (!(sis-\u003eflags \u0026 SWP_WRITEOK))\n \t\t\tcontinue;\n+\t\t/* vswap has no bdev, so it is never a hibernation target. */\n+\t\tif (swap_is_vswap(sis))\n+\t\t\tcontinue;\n \t\t*device = sis-\u003ebdev-\u003ebd_dev;\n \t\tspin_unlock(\u0026swap_lock);\n \t\treturn type;\n@@ -2572,8 +3287,10 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd,\n \t\t\t\t\t\t\u0026vmf);\n \t\t}\n \t\tif (!folio) {\n+\t\t\trcu_read_lock();\n \t\t\tswp_tb = swap_table_get(__swap_entry_to_cluster(entry),\n \t\t\t\t\t\tswp_cluster_offset(entry));\n+\t\t\trcu_read_unlock();\n \t\t\tif (swp_tb_get_count(swp_tb) \u003c= 0)\n \t\t\t\tcontinue;\n \t\t\treturn -ENOMEM;\n@@ -2706,10 +3423,10 @@ static int unuse_mm(struct mm_struct *mm, unsigned int type)\n * Return 0 if there are no inuse entries after prev till end of\n * the map.\n */\n-static unsigned int find_next_to_unuse(struct swap_info_struct *si,\n-\t\t\t\t\tunsigned int prev)\n+static unsigned long find_next_to_unuse(struct swap_info_struct *si,\n+\t\t\t\t\tunsigned long prev)\n {\n-\tunsigned int i;\n+\tunsigned long i;\n \tunsigned long swp_tb;\n \n \t/*\n@@ -2719,8 +3436,10 @@ static unsigned int find_next_to_unuse(struct swap_info_struct *si,\n \t * allocations from this area (while holding swap_lock).\n \t */\n \tfor (i = prev + 1; i \u003c si-\u003emax; i++) {\n+\t\trcu_read_lock();\n \t\tswp_tb = swap_table_get(__swap_offset_to_cluster(si, i),\n \t\t\t\t\ti % SWAPFILE_CLUSTER);\n+\t\trcu_read_unlock();\n \t\tif (!swp_tb_is_null(swp_tb) \u0026\u0026 !swp_tb_is_bad(swp_tb))\n \t\t\tbreak;\n \t\tif ((i % LATENCY_LIMIT) == 0)\n@@ -2735,19 +3454,100 @@ static unsigned int find_next_to_unuse(struct swap_info_struct *si,\n \n static int try_to_unuse(unsigned int type)\n {\n+\tstruct mempolicy *mpol = get_task_policy(current);\n \tstruct mm_struct *prev_mm;\n \tstruct mm_struct *mm;\n \tstruct list_head *p;\n \tint retval = 0;\n \tstruct swap_info_struct *si = swap_info[type];\n \tstruct folio *folio;\n-\tswp_entry_t entry;\n-\tunsigned int i;\n+\tstruct swap_io_ctx ctx;\n+\tswp_entry_t entry, vswap_entry;\n+\tunsigned long swp_tb;\n+\tunsigned long i;\n+\tunsigned int j;\n \n \tif (!swap_usage_in_pages(si))\n \t\tgoto success;\n \n retry:\n+\t/*\n+\t * Free vswap-backing slots (Pointer-tagged) first. Walk physical\n+\t * clusters, read the vswap entry from the rmap, ensure the data\n+\t * is in the swap cache, and transition PHYS to FOLIO. Freeing the\n+\t * physical backing is enough, so no page table walk is needed.\n+\t */\n+\ti = 0;\n+\twhile (vswap_is_enabled() \u0026\u0026\n+\t swap_usage_in_pages(si) \u0026\u0026\n+\t !signal_pending(current) \u0026\u0026\n+\t (i = find_next_to_unuse(si, i)) != 0) {\n+\t\tswp_entry_t phys;\n+\n+\t\tswp_tb = swap_table_get(__swap_offset_to_cluster(si, i),\n+\t\t\t\t\ti % SWAPFILE_CLUSTER);\n+\t\tif (!swp_tb_is_pointer(swp_tb))\n+\t\t\tcontinue;\n+\n+\t\tvswap_entry = swp_tb_ptr_to_swp_entry(swp_tb);\n+\n+\t\tfolio = swap_cache_get_folio(vswap_entry);\n+\t\tif (!folio) {\n+\t\t\tfolio = swap_cache_alloc_folio(vswap_entry,\n+\t\t\t\t\t\t GFP_HIGHUSER_MOVABLE,\n+\t\t\t\t\t\t BIT(0), NULL, mpol,\n+\t\t\t\t\t\t NO_INTERLEAVE_INDEX);\n+\t\t\tif (IS_ERR(folio)) {\n+\t\t\t\tif (PTR_ERR(folio) == -ENOMEM)\n+\t\t\t\t\treturn -ENOMEM;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t\tctx = (struct swap_io_ctx){};\n+\t\t\tswap_read_folio(\u0026ctx, folio);\n+\t\t\tswap_read_submit(\u0026ctx);\n+\t\t\tfolio_lock(folio);\n+\t\t} else {\n+\t\t\tfolio_lock(folio);\n+\t\t}\n+\n+\t\tif (!folio_matches_swap_entry(folio, vswap_entry)) {\n+\t\t\tfolio_unlock(folio);\n+\t\t\tfolio_put(folio);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\t/*\n+\t\t * Re-validate under folio lock: rmap holds folio-\u003eswap + j\n+\t\t * for some j in [0, nr_pages). Check folio-\u003eswap still maps\n+\t\t * to the contiguous physical run that includes our slot i.\n+\t\t */\n+\t\tj = vswap_entry.val - folio-\u003eswap.val;\n+\t\tphys = vswap_to_phys(folio-\u003eswap);\n+\t\tif (!phys.val || swp_type(phys) != type ||\n+\t\t swp_offset(phys) + j != i) {\n+\t\t\tfolio_unlock(folio);\n+\t\t\tfolio_put(folio);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tfolio_wait_writeback(folio);\n+\t\tfolio_release_vswap_backing(folio);\n+\t\t/*\n+\t\t * Drop a folio whose read failed rather than dirtying\n+\t\t * uninitialised memory; the next fault finds no backing and\n+\t\t * gets SIGBUS.\n+\t\t */\n+\t\tif (unlikely(!folio_test_uptodate(folio)))\n+\t\t\tswap_cache_del_folio(folio);\n+\t\telse\n+\t\t\tfolio_mark_dirty(folio);\n+\t\tfolio_unlock(folio);\n+\t\tfolio_put(folio);\n+\t}\n+\n+\tif (!swap_usage_in_pages(si))\n+\t\tgoto success;\n+\n \tretval = shmem_unuse(type);\n \tif (retval)\n \t\treturn retval;\n@@ -2790,6 +3590,8 @@ static int try_to_unuse(unsigned int type)\n \t (i = find_next_to_unuse(si, i)) != 0) {\n \n \t\tentry = swp_entry(type, i);\n+\n+\t\t/* Pointer-tagged rmap slots have no folio; the pre-pass took them. */\n \t\tfolio = swap_cache_get_folio(entry);\n \t\tif (!folio)\n \t\t\tcontinue;\n@@ -2959,6 +3761,11 @@ static int setup_swap_extents(struct swap_info_struct *sis,\n \tstruct inode *inode = mapping-\u003ehost;\n \tint ret;\n \n+\tif (swap_is_vswap(sis)) {\n+\t\t*span = 0;\n+\t\treturn 0;\n+\t}\n+\n \tret = sio_pool_init();\n \tif (ret)\n \t\treturn ret;\n@@ -2984,15 +3791,24 @@ static int setup_swap_extents(struct swap_info_struct *sis,\n \n static void _enable_swap_info(struct swap_info_struct *si)\n {\n-\tatomic_long_add(si-\u003epages, \u0026nr_swap_pages);\n-\ttotal_swap_pages += si-\u003epages;\n+\tif (!swap_is_vswap(si)) {\n+\t\tatomic_long_add(si-\u003epages, \u0026nr_swap_pages);\n+\t\ttotal_swap_pages += si-\u003epages;\n+\t}\n \n \tassert_spin_locked(\u0026swap_lock);\n \n-\tplist_add(\u0026si-\u003elist, \u0026swap_active_head);\n+\t/*\n+\t * Vswap has no backing file and no swapoff support, so keep it\n+\t * off swap_active_head (used by swapoff filename lookup and\n+\t * swap_sync_discard) and swap_avail_head (physical allocator).\n+\t */\n+\tif (!swap_is_vswap(si)) {\n+\t\tplist_add(\u0026si-\u003elist, \u0026swap_active_head);\n \n-\t/* Add back to available list */\n-\tadd_to_avail_list(si, true);\n+\t\t/* Add back to available list */\n+\t\tadd_to_avail_list(si, true);\n+\t}\n }\n \n /*\n@@ -3036,12 +3852,32 @@ static void wait_for_allocation(struct swap_info_struct *si)\n \t}\n }\n \n-static void free_swap_cluster_info(struct swap_cluster_info *cluster_info,\n+static void free_swap_cluster_info(struct swap_info_struct *si,\n+\t\t\t\t struct swap_cluster_info *cluster_info,\n \t\t\t\t unsigned long maxpages)\n {\n+\tstruct swap_cluster_info_dynamic *ci_dyn;\n \tstruct swap_cluster_info *ci;\n+\tunsigned long idx;\n \tint i, nr_clusters = DIV_ROUND_UP(maxpages, SWAPFILE_CLUSTER);\n \n+\tif (swap_is_vswap(si)) {\n+\t\txa_for_each(\u0026si-\u003ecluster_info_pool, idx, ci_dyn) {\n+\t\t\tci = \u0026ci_dyn-\u003eci;\n+\t\t\tspin_lock(\u0026ci-\u003elock);\n+\t\t\tif (cluster_table_is_alloced(ci)) {\n+\t\t\t\tswap_cluster_assert_empty(ci, 0,\n+\t\t\t\t\t\t\t SWAPFILE_CLUSTER, true);\n+\t\t\t\tswap_cluster_free_table(ci);\n+\t\t\t}\n+\t\t\tspin_unlock(\u0026ci-\u003elock);\n+\t\t\tvswap_cluster_free_vtable(ci);\n+\t\t\tkfree(ci_dyn);\n+\t\t}\n+\t\txa_destroy(\u0026si-\u003ecluster_info_pool);\n+\t\treturn;\n+\t}\n+\n \tif (!cluster_info)\n \t\treturn;\n \tfor (i = 0; i \u003c nr_clusters; i++) {\n@@ -3086,7 +3922,7 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)\n \tstruct file *swap_file, *victim;\n \tstruct address_space *mapping;\n \tstruct inode *inode;\n-\tunsigned int maxpages;\n+\tunsigned long maxpages;\n \tint err, found = 0;\n \n \tif (!capable(CAP_SYS_ADMIN))\n@@ -3188,7 +4024,7 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)\n \tmutex_unlock(\u0026swapon_mutex);\n \tkfree(p-\u003eglobal_cluster);\n \tp-\u003eglobal_cluster = NULL;\n-\tfree_swap_cluster_info(cluster_info, maxpages);\n+\tfree_swap_cluster_info(p, cluster_info, maxpages);\n \n \tinode = mapping-\u003ehost;\n \n@@ -3508,12 +4344,8 @@ static unsigned long read_swap_header(struct swap_info_struct *si,\n \t\tpr_warn(\"Truncating oversized swap area, only using %luk out of %luk\\n\",\n \t\t\tK(maxpages), K(last_page));\n \t}\n-\tif (maxpages \u003e last_page) {\n+\tif (maxpages \u003e last_page)\n \t\tmaxpages = last_page + 1;\n-\t\t/* p-\u003emax is an unsigned int: don't overflow it */\n-\t\tif ((unsigned int)maxpages == 0)\n-\t\t\tmaxpages = UINT_MAX;\n-\t}\n \n \tif (!maxpages)\n \t\treturn 0;\n@@ -3535,10 +4367,43 @@ static int setup_swap_clusters_info(struct swap_info_struct *si,\n \t\t\t\t unsigned long maxpages)\n {\n \tunsigned long nr_clusters = DIV_ROUND_UP(maxpages, SWAPFILE_CLUSTER);\n-\tstruct swap_cluster_info *cluster_info;\n+\tstruct swap_cluster_info *cluster_info = NULL;\n+\tstruct swap_cluster_info_dynamic *ci_dyn = NULL;\n \tint err = -ENOMEM;\n \tunsigned long i;\n \n+\t/* A vswap device uses an xarray pool instead of a static array. */\n+\tif (swap_is_vswap(si)) {\n+\t\tnr_clusters = 0;\n+\t\txa_init_flags(\u0026si-\u003ecluster_info_pool, XA_FLAGS_ALLOC);\n+\n+\t\t/*\n+\t\t * Pre-allocate cluster 0 and mark slot 0 (header page)\n+\t\t * as bad so the allocator never hands out page offset 0.\n+\t\t */\n+\t\tci_dyn = kzalloc_obj(*ci_dyn, GFP_KERNEL);\n+\t\tif (!ci_dyn)\n+\t\t\tgoto err;\n+\t\tspin_lock_init(\u0026ci_dyn-\u003eci.lock);\n+\t\tINIT_LIST_HEAD(\u0026ci_dyn-\u003eci.list);\n+\n+\t\terr = xa_insert(\u0026si-\u003ecluster_info_pool, 0, ci_dyn, GFP_KERNEL);\n+\t\tif (err) {\n+\t\t\tkfree(ci_dyn);\n+\t\t\tgoto err;\n+\t\t}\n+\n+\t\terr = swap_cluster_setup_bad_slot(si, \u0026ci_dyn-\u003eci, 0, false);\n+\t\tif (err)\n+\t\t\tgoto err;\n+\n+\t\terr = vswap_cluster_alloc_vtable(ci_dyn, GFP_KERNEL);\n+\t\tif (err)\n+\t\t\tgoto err;\n+\n+\t\tgoto setup_cluster_info;\n+\t}\n+\n \tcluster_info = kvzalloc_objs(*cluster_info, nr_clusters);\n \tif (!cluster_info)\n \t\tgoto err;\n@@ -3582,6 +4447,7 @@ static int setup_swap_clusters_info(struct swap_info_struct *si,\n \t\t\tgoto err;\n \t}\n \n+setup_cluster_info:\n \tINIT_LIST_HEAD(\u0026si-\u003efree_clusters);\n \tINIT_LIST_HEAD(\u0026si-\u003efull_clusters);\n \tINIT_LIST_HEAD(\u0026si-\u003ediscard_clusters);\n@@ -3603,10 +4469,16 @@ static int setup_swap_clusters_info(struct swap_info_struct *si,\n \t\t}\n \t}\n \n+\t/* Slot 0 is bad, so cluster 0 never empties. The rest of it is usable. */\n+\tif (swap_is_vswap(si)) {\n+\t\tci_dyn-\u003eci.flags = CLUSTER_FLAG_NONFULL;\n+\t\tlist_add_tail(\u0026ci_dyn-\u003eci.list, \u0026si-\u003enonfull_clusters[0]);\n+\t}\n+\n \tsi-\u003ecluster_info = cluster_info;\n \treturn 0;\n err:\n-\tfree_swap_cluster_info(cluster_info, maxpages);\n+\tfree_swap_cluster_info(si, cluster_info, maxpages);\n \treturn err;\n }\n \n@@ -3714,7 +4586,7 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)\n \t\tgoto bad_swap_unlock_inode;\n \t}\n \tif (si-\u003epages != si-\u003emax - 1) {\n-\t\tpr_err(\"swap:%u != (max:%u - 1)\\n\", si-\u003epages, si-\u003emax);\n+\t\tpr_err(\"swap:%lu != (max:%lu - 1)\\n\", si-\u003epages, si-\u003emax);\n \t\terror = -EINVAL;\n \t\tgoto bad_swap_unlock_inode;\n \t}\n@@ -3802,7 +4674,7 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)\n \t/* Sets SWP_WRITEOK, resurrect the percpu ref, expose the swap device */\n \tenable_swap_info(si);\n \n-\tpr_info(\"Adding %uk swap on %s. Priority:%d extents:%d across:%lluk %s%s%s%s\\n\",\n+\tpr_info(\"Adding %luk swap on %s. Priority:%d extents:%d across:%lluk %s%s%s%s\\n\",\n \t\tK(si-\u003epages), name-\u003ename, si-\u003eprio, nr_extents,\n \t\tK((unsigned long long)span),\n \t\t(si-\u003eflags \u0026 SWP_SOLIDSTATE) ? \"SS\" : \"\",\n@@ -3825,7 +4697,7 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)\n \tsi-\u003eglobal_cluster = NULL;\n \tinode = NULL;\n \tdestroy_swap_extents(si, swap_file);\n-\tfree_swap_cluster_info(si-\u003ecluster_info, si-\u003emax);\n+\tfree_swap_cluster_info(si, si-\u003ecluster_info, si-\u003emax);\n \tsi-\u003ecluster_info = NULL;\n \t/*\n \t * Clear the SWP_USED flag after all resources are freed so\n@@ -3956,3 +4828,105 @@ static int __init swapfile_init(void)\n \treturn 0;\n }\n subsys_initcall(swapfile_init);\n+\n+struct swap_info_struct *vswap_si;\n+DEFINE_STATIC_KEY_FALSE(vswap_key);\n+\n+static bool vswap_enabled_early __initdata = IS_ENABLED(CONFIG_VSWAP_DEFAULT_ON);\n+\n+static int __init early_vswap(char *buf)\n+{\n+\treturn kstrtobool(buf, \u0026vswap_enabled_early);\n+}\n+early_param(\"vswap\", early_vswap);\n+\n+/* vswap does no IO on its own. */\n+static const struct swap_ops vswap_ops = { };\n+\n+static int vswap_used_get(void *data, u64 *val)\n+{\n+\t*val = swap_usage_in_pages(vswap_si);\n+\treturn 0;\n+}\n+DEFINE_DEBUGFS_ATTRIBUTE(vswap_used_fops, vswap_used_get, NULL, \"%llu\\n\");\n+\n+static int vswap_alloc_reject_get(void *data, u64 *val)\n+{\n+\t*val = atomic_long_read(\u0026vswap_alloc_reject);\n+\treturn 0;\n+}\n+DEFINE_DEBUGFS_ATTRIBUTE(vswap_alloc_reject_fops, vswap_alloc_reject_get, NULL,\n+\t\t\t \"%llu\\n\");\n+\n+static int __init vswap_init(void)\n+{\n+\tstruct swap_info_struct *si;\n+\tstruct dentry *root;\n+\tunsigned long maxpages;\n+\tint err;\n+\n+\tif (!IS_ENABLED(CONFIG_64BIT)) {\n+\t\tif (vswap_enabled_early)\n+\t\t\tpr_warn(\"vswap: requires 64-bit architecture; vswap disabled, swapout falls back to direct physical swap\\n\");\n+\t\treturn 0;\n+\t}\n+\n+\tif (!vswap_enabled_early)\n+\t\treturn 0;\n+\n+\tsi = alloc_swap_info();\n+\tif (IS_ERR(si)) {\n+\t\tpr_warn(\"vswap: alloc_swap_info failed (%ld); vswap disabled, swapout falls back to direct physical swap\\n\",\n+\t\t\tPTR_ERR(si));\n+\t\treturn 0;\n+\t}\n+\n+\t/*\n+\t * One u32 xarray ID per cluster, so the device cannot be larger\n+\t * than UINT_MAX clusters.\n+\t */\n+\tmaxpages = min(swapfile_maximum_size,\n+\t\t (unsigned long)UINT_MAX * SWAPFILE_CLUSTER);\n+\t/*\n+\t * SWP_WRITEOK enables slot allocation. SWP_SOLIDSTATE selects\n+\t * per-CPU cluster allocation; vswap has no si-\u003eglobal_cluster.\n+\t */\n+\tsi-\u003eflags |= SWP_VSWAP | SWP_SOLIDSTATE | SWP_WRITEOK;\n+\tsi-\u003eops = \u0026vswap_ops;\n+\tsi-\u003ebdev = NULL;\n+\tsi-\u003emax = maxpages;\n+\tsi-\u003epages = maxpages - 1;\n+\tsi-\u003eprio = SHRT_MAX;\n+\tsi-\u003elist.prio = -si-\u003eprio;\n+\tsi-\u003eavail_list.prio = -si-\u003eprio;\n+\n+\terr = setup_swap_clusters_info(si, NULL, maxpages);\n+\tif (err)\n+\t\tgoto fail;\n+\n+\tmutex_lock(\u0026swapon_mutex);\n+\tenable_swap_info(si);\n+\tmutex_unlock(\u0026swapon_mutex);\n+\n+\tvswap_si = si;\n+\n+\troot = debugfs_create_dir(\"vswap\", NULL);\n+\tdebugfs_create_file(\"used\", 0444, root, NULL, \u0026vswap_used_fops);\n+\tdebugfs_create_file(\"alloc_reject\", 0444, root, NULL,\n+\t\t\t \u0026vswap_alloc_reject_fops);\n+\n+\tpr_info(\"vswap: created virtual swap device (%lu pages)\\n\", maxpages);\n+\n+\t/* Last: everything above must be visible before routing starts. */\n+\tstatic_branch_enable(\u0026vswap_key);\n+\treturn 0;\n+\n+fail:\n+\tpr_warn(\"vswap: setup_swap_clusters_info failed (%d); vswap disabled, swapout falls back to direct physical swap\\n\",\n+\t\terr);\n+\tspin_lock(\u0026swap_lock);\n+\tsi-\u003eflags = 0;\n+\tspin_unlock(\u0026swap_lock);\n+\treturn 0;\n+}\n+late_initcall(vswap_init);\ndiff --git a/mm/vmscan.c b/mm/vmscan.c\nindex c1404a59523d6..7960cc489ea03 100644\n--- a/mm/vmscan.c\n+++ b/mm/vmscan.c\n@@ -69,6 +69,7 @@\n #include \"internal.h\"\n #include \"page_alloc.h\"\n #include \"swap.h\"\n+#include \"vswap.h\"\n \n #define CREATE_TRACE_POINTS\n #include \u003ctrace/events/vmscan.h\u003e\n@@ -353,6 +354,9 @@ static inline bool can_reclaim_anon_pages(struct mem_cgroup *memcg,\n \t\t */\n \t\tif (get_nr_swap_pages() \u003e 0)\n \t\t\treturn true;\n+\t\t/* vswap doesn't contribute to nr_swap_pages */\n+\t\tif (vswap_is_enabled() \u0026\u0026 zswap_is_enabled())\n+\t\t\treturn true;\n \t} else {\n \t\t/* Is the memcg below its swap limit? */\n \t\tif (mem_cgroup_get_nr_swap_pages(memcg) \u003e 0)\n@@ -1524,7 +1528,8 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,\n activate_locked:\n \t\t/* Not a candidate for swapping, so reclaim swap space. */\n \t\tif (folio_test_swapcache(folio) \u0026\u0026\n-\t\t (mem_cgroup_swap_full(folio) || folio_test_mlocked(folio)))\n+\t\t ((mem_cgroup_swap_full(folio) \u0026\u0026 folio_phys_swap_backed(folio)) ||\n+\t\t folio_test_mlocked(folio)))\n \t\t\tfolio_free_swap(folio);\n \t\tVM_BUG_ON_FOLIO(folio_test_active(folio), folio);\n \t\tif (!folio_test_mlocked(folio)) {\n@@ -2681,7 +2686,7 @@ static bool can_age_anon_pages(struct lruvec *lruvec,\n \t\t\t struct scan_control *sc)\n {\n \t/* Aging the anon LRU is valuable if swap is present: */\n-\tif (total_swap_pages \u003e 0)\n+\tif (total_swap_pages \u003e 0 || (vswap_is_enabled() \u0026\u0026 zswap_is_enabled()))\n \t\treturn true;\n \n \t/* Also valuable if anon pages can be demoted: */\ndiff --git a/mm/vswap.h b/mm/vswap.h\nnew file mode 100644\nindex 0000000000000..c66fa34e2e60c\n--- /dev/null\n+++ b/mm/vswap.h\n@@ -0,0 +1,440 @@\n+/* SPDX-License-Identifier: GPL-2.0 */\n+/*\n+ * Virtual swap space\n+ *\n+ * Copyright (C) 2026 Nhat Pham\n+ */\n+#ifndef _MM_VSWAP_H\n+#define _MM_VSWAP_H\n+\n+#include \u003clinux/jump_label.h\u003e\n+#include \u003clinux/swap.h\u003e\n+#include \"swap.h\"\n+\n+struct zswap_entry;\n+\n+/*\n+ * VSWAP_ZERO and VSWAP_FOLIO are return-only values synthesized from\n+ * swap_table state; the rest are stored in the vtable per slot.\n+ */\n+enum vswap_backing_type {\n+\tVSWAP_NONE\t= 0,\n+\tVSWAP_ZSWAP\t= 1,\n+\tVSWAP_SWAPFILE\t= 2,\n+\tVSWAP_ZERO,\n+\tVSWAP_FOLIO,\n+};\n+\n+#ifdef CONFIG_SWAP\n+\n+#include \"swap_table.h\"\n+DECLARE_STATIC_KEY_FALSE(vswap_key);\n+\n+/*\n+ * Only true once vswap_init() has published vswap_si, so callers never\n+ * see the device half built.\n+ */\n+static inline bool vswap_is_enabled(void)\n+{\n+\treturn static_branch_unlikely(\u0026vswap_key);\n+}\n+\n+static inline bool is_vswap_entry(swp_entry_t entry)\n+{\n+\treturn swap_is_vswap(__swap_entry_to_info(entry));\n+}\n+\n+/*\n+ * Rmap cache-only helpers for physical cluster Pointer-tagged entries.\n+ * SWP_RMAP_CACHE_ONLY records, inline on the physical swap_table entry,\n+ * that the backing vswap entry has swap_count == 0 (swap-cache-only, so\n+ * reclaimable). The physical reclaim scanner reads it directly instead of\n+ * chasing the rmap into the vswap layer and paying the cluster-lookup\n+ * indirection.\n+ *\n+ * Callers hold the vswap cluster lock, not the physical one. The rmap is\n+ * only touched while the vtable holds the slot as SWAPFILE, and that\n+ * window is opened and closed under the vswap cluster lock, so the\n+ * allocator has finished writing the entry by then.\n+ */\n+static inline void swap_rmap_mark_cache_only(struct swap_cluster_info *ci,\n+\t\t\t\t\t unsigned int off)\n+{\n+\tatomic_long_t *table;\n+\n+\ttable = rcu_dereference_check(ci-\u003etable, true);\n+\tatomic_long_or(SWP_RMAP_CACHE_ONLY, \u0026table[off]);\n+}\n+\n+static inline void swap_rmap_clear_cache_only(struct swap_cluster_info *ci,\n+\t\t\t\t\t unsigned int off)\n+{\n+\tatomic_long_t *table;\n+\n+\ttable = rcu_dereference_check(ci-\u003etable, true);\n+\tatomic_long_and(~SWP_RMAP_CACHE_ONLY, \u0026table[off]);\n+}\n+\n+/*\n+ * Virtual table entry encoding for vswap clusters.\n+ *\n+ * Each entry in ci_dyn-\u003evirtual_table stores the backing type and\n+ * pointer for a virtual swap slot. Tag in low 3 bits, payload in\n+ * upper 61 bits.\n+ *\n+ * NONE: |----- 0000 ------|000| - no separate backend pointer\n+ * ZSWAP: |--- zswap_entry* |001| - compressed in zswap (tag in low bits)\n+ * SWAPFILE: |- type:5,off:56 -|010| - on a physical swapfile\n+ *\n+ * SWAPFILE packs swp_type in the top MAX_SWAPFILES_SHIFT bits and swp_offset in\n+ * the middle VTABLE_PHYS_OFF_BITS bits, both above the tag, so the type is\n+ * not shifted off the word. Pointer payloads (ZSWAP) are stored directly with\n+ * the tag OR'd into the low bits (kernel pointers are \u003e= 8-byte aligned, same\n+ * approach as xarray).\n+ *\n+ * vtable[i] = NONE does not by itself mean \"free\". The swap_table entry\n+ * and the per-slot zero flag carry the rest of the state. The full\n+ * per-slot state table is:\n+ *\n+ * vtable[i] | swap_table[i] | zero | meaning\n+ * ----------+---------------+-------+--------------------------------\n+ * NONE | NULL | clear | truly free / unbacked\n+ * NONE | PFN | clear | folio cached, no backing\n+ * NONE | shadow | clear | evicted, no backing: data lost\n+ * NONE | * | set | zero-backed; cached if PFN set\n+ * ZSWAP | PFN | clear | folio cached + zswap entry\n+ * ZSWAP | shadow / NULL | clear | evicted, only in zswap\n+ * SWAPFILE | PFN | clear | folio cached + physical slot\n+ * SWAPFILE | shadow / NULL | clear | evicted, only on the swapfile\n+ *\n+ * Locking: a slot's vtable entry (the vswap entry's backend) is only\n+ * stable while the caller owns and holds the lock on that entry's swap\n+ * cache folio. The cluster lock (ci_dyn-\u003eci.lock) only makes an individual\n+ * vtable read atomic, and by itself does not give the caller the right to\n+ * change the backend. A backend read without the folio lock is\n+ * best-effort and must be re-validated under the folio lock before\n+ * being acted on.\n+ *\n+ * Zero-backed slots use the swap_table per-slot zero flag (same as\n+ * direct-mapped physical swap), via __swap_table_test_zero() and friends,\n+ * which fall back to ci-\u003ezero_bitmap where the flag does not fit. Cached\n+ * folios are read out of the swap_table PFN entry; there is no separate FOLIO\n+ * vtable type because the folio pointer would duplicate that PFN and\n+ * would go stale on folio migration / split.\n+ */\n+\n+#define VTABLE_TAG_BITS\t\t3\n+#define VTABLE_TAG_MASK\t\t((1UL \u003c\u003c VTABLE_TAG_BITS) - 1)\n+\n+static inline enum vswap_backing_type vtable_type(unsigned long vt)\n+{\n+\treturn vt \u0026 VTABLE_TAG_MASK;\n+}\n+\n+/* swp_offset field width in a physical backend slot; layout described above. */\n+#define VTABLE_PHYS_OFF_BITS\t(BITS_PER_LONG - VTABLE_TAG_BITS - MAX_SWAPFILES_SHIFT)\n+\n+static inline unsigned long vtable_mk_phys(swp_entry_t entry)\n+{\n+\tVM_WARN_ON_ONCE(swp_offset(entry) \u003e\u003e VTABLE_PHYS_OFF_BITS);\n+\treturn ((unsigned long)swp_type(entry) \u003c\u003c (VTABLE_TAG_BITS + VTABLE_PHYS_OFF_BITS)) |\n+\t (swp_offset(entry) \u003c\u003c VTABLE_TAG_BITS) | VSWAP_SWAPFILE;\n+}\n+\n+static inline swp_entry_t vtable_to_phys(unsigned long vt)\n+{\n+\tVM_WARN_ON(vtable_type(vt) != VSWAP_SWAPFILE);\n+\treturn swp_entry(vt \u003e\u003e (VTABLE_TAG_BITS + VTABLE_PHYS_OFF_BITS),\n+\t\t\t (vt \u003e\u003e VTABLE_TAG_BITS) \u0026 ((1UL \u003c\u003c VTABLE_PHYS_OFF_BITS) - 1));\n+}\n+\n+static inline struct zswap_entry *vtable_to_zswap(unsigned long vt)\n+{\n+\tVM_WARN_ON(vtable_type(vt) != VSWAP_ZSWAP);\n+\treturn (struct zswap_entry *)(vt \u0026 ~VTABLE_TAG_MASK);\n+}\n+\n+/* Virtual table accessors */\n+\n+static inline unsigned long __vtable_get(struct swap_cluster_info_dynamic *ci_dyn,\n+\t\t\t\t\t unsigned int off)\n+{\n+\tVM_WARN_ON_ONCE(off \u003e= SWAPFILE_CLUSTER);\n+\treturn atomic_long_read(\u0026ci_dyn-\u003evirtual_table[off]);\n+}\n+\n+static inline void __vtable_set(struct swap_cluster_info_dynamic *ci_dyn,\n+\t\t\t\tunsigned int off, unsigned long vt)\n+{\n+\tVM_WARN_ON_ONCE(off \u003e= SWAPFILE_CLUSTER);\n+\tatomic_long_set(\u0026ci_dyn-\u003evirtual_table[off], vt);\n+}\n+\n+/**\n+ * vswap_lock_cluster - look up and lock the vswap cluster for an entry\n+ * @entry: the virtual swap entry\n+ * @voff: out param, receives @entry's slot offset within the cluster\n+ *\n+ * Return: the locked vswap cluster, or NULL if @entry has no live cluster.\n+ */\n+static inline struct swap_cluster_info_dynamic *\n+vswap_lock_cluster(swp_entry_t entry, unsigned int *voff)\n+{\n+\tstruct swap_cluster_info *ci;\n+\n+\tci = swap_cluster_lock(__swap_entry_to_info(entry), swp_offset(entry));\n+\tif (!ci)\n+\t\treturn NULL;\n+\t*voff = swp_cluster_offset(entry);\n+\treturn container_of(ci, struct swap_cluster_info_dynamic, ci);\n+}\n+\n+/**\n+ * vswap_to_phys - resolve a vswap entry's physical swap backing\n+ * @entry: the virtual swap entry\n+ *\n+ * Context: takes and drops the vswap cluster lock internally.\n+ * Return: the backing physical swp_entry_t, or the null entry (.val == 0)\n+ * when @entry has no physical backing (NONE/ZSWAP/ZERO).\n+ */\n+static inline swp_entry_t vswap_to_phys(swp_entry_t entry)\n+{\n+\tstruct swap_cluster_info_dynamic *ci_dyn;\n+\tunsigned int voff;\n+\tunsigned long vt;\n+\n+\tci_dyn = vswap_lock_cluster(entry, \u0026voff);\n+\tif (!ci_dyn)\n+\t\treturn (swp_entry_t){};\n+\n+\tvt = __vtable_get(ci_dyn, voff);\n+\tswap_cluster_unlock(\u0026ci_dyn-\u003eci);\n+\n+\tif (vtable_type(vt) != VSWAP_SWAPFILE)\n+\t\treturn (swp_entry_t){};\n+\n+\treturn vtable_to_phys(vt);\n+}\n+\n+void __vswap_release_backing(struct swap_cluster_info *ci,\n+\t\t\t unsigned int ci_start, unsigned int nr);\n+\n+/**\n+ * vswap_zswap_store - record a zswap entry as the backing for a vswap entry.\n+ * @entry: the vswap entry\n+ * @ze: the zswap entry now holding @entry's compressed data\n+ *\n+ * Releases @entry's previous backing, and sets the zswap entry @ze as the new\n+ * backing.\n+ *\n+ * Context: takes and drops the vswap cluster lock internally.\n+ */\n+static inline void vswap_zswap_store(swp_entry_t entry,\n+\t\t\t\t struct zswap_entry *ze)\n+{\n+\tstruct swap_cluster_info_dynamic *ci_dyn;\n+\tunsigned int voff;\n+\n+\tci_dyn = vswap_lock_cluster(entry, \u0026voff);\n+\t__vswap_release_backing(\u0026ci_dyn-\u003eci, voff, 1);\n+\t__vtable_set(ci_dyn, voff, (unsigned long)ze | VSWAP_ZSWAP);\n+\tswap_cluster_unlock(\u0026ci_dyn-\u003eci);\n+}\n+\n+/**\n+ * vswap_zswap_load - return the zswap entry backing a vswap entry\n+ * @entry: the virtual swap entry\n+ *\n+ * Context: takes and drops the vswap cluster lock internally.\n+ * Return: the backing zswap entry, or NULL if @entry is not zswap-backed.\n+ */\n+static inline struct zswap_entry *vswap_zswap_load(swp_entry_t entry)\n+{\n+\tstruct swap_cluster_info_dynamic *ci_dyn;\n+\tunsigned int voff;\n+\tunsigned long vt;\n+\n+\tci_dyn = vswap_lock_cluster(entry, \u0026voff);\n+\tif (!ci_dyn)\n+\t\treturn NULL;\n+\tvt = __vtable_get(ci_dyn, voff);\n+\tswap_cluster_unlock(\u0026ci_dyn-\u003eci);\n+\n+\tif (vtable_type(vt) != VSWAP_ZSWAP)\n+\t\treturn NULL;\n+\treturn vtable_to_zswap(vt);\n+}\n+\n+void folio_release_vswap_backing(struct folio *folio);\n+swp_entry_t folio_realloc_swap(struct folio *folio);\n+void folio_release_non_phys_swap_backing(struct folio *folio);\n+\n+/*\n+ * Walk nr vtable slots starting at voff in ci_dyn. Returns the prefix\n+ * length of slots sharing one effective backing type. For SWAPFILE,\n+ * the prefix is also restricted to contiguous offsets in the same\n+ * swapfile.\n+ *\n+ * Effective type per slot:\n+ * vtable=NONE + zero flag set -\u003e VSWAP_ZERO\n+ * vtable=NONE + swap_table PFN tag -\u003e VSWAP_FOLIO\n+ * vtable=NONE + neither -\u003e VSWAP_NONE\n+ * vtable=SWAPFILE -\u003e VSWAP_SWAPFILE\n+ * vtable=ZSWAP -\u003e VSWAP_ZSWAP\n+ *\n+ * *typep returns the effective type of slot 0. Caller holds\n+ * ci_dyn-\u003eci.lock.\n+ */\n+static inline int __vswap_check_backing(struct swap_cluster_info_dynamic *ci_dyn,\n+\t\t\t\t\tunsigned int voff, int nr,\n+\t\t\t\t\tenum vswap_backing_type *typep)\n+{\n+\tenum vswap_backing_type first_type = VSWAP_NONE;\n+\tenum vswap_backing_type slot_type;\n+\tswp_entry_t first_phys = {};\n+\tunsigned long vt, swap_tb;\n+\tint i;\n+\n+\tlockdep_assert_held(\u0026ci_dyn-\u003eci.lock);\n+\n+\tfor (i = 0; i \u003c nr; i++) {\n+\t\tvt = __vtable_get(ci_dyn, voff + i);\n+\t\tif (vtable_type(vt) == VSWAP_NONE) {\n+\t\t\tswap_tb = __swap_table_get(\u0026ci_dyn-\u003eci, voff + i);\n+\t\t\tif (__swap_table_test_zero(\u0026ci_dyn-\u003eci, voff + i))\n+\t\t\t\tslot_type = VSWAP_ZERO;\n+\t\t\telse if (swp_tb_is_folio(swap_tb))\n+\t\t\t\tslot_type = VSWAP_FOLIO;\n+\t\t\telse\n+\t\t\t\tslot_type = VSWAP_NONE;\n+\t\t} else {\n+\t\t\tslot_type = vtable_type(vt);\n+\t\t}\n+\n+\t\tif (!i) {\n+\t\t\tfirst_type = slot_type;\n+\t\t\tif (first_type == VSWAP_SWAPFILE)\n+\t\t\t\tfirst_phys = vtable_to_phys(vt);\n+\t\t} else if (slot_type != first_type) {\n+\t\t\tbreak;\n+\t\t} else if (first_type == VSWAP_SWAPFILE \u0026\u0026\n+\t\t\t vtable_to_phys(vt).val != first_phys.val + i) {\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tif (typep)\n+\t\t*typep = first_type;\n+\treturn i;\n+}\n+\n+static inline int vswap_check_backing(swp_entry_t entry, int nr,\n+\t\t\t\t enum vswap_backing_type *typep)\n+{\n+\tstruct swap_cluster_info_dynamic *ci_dyn;\n+\tunsigned int voff;\n+\tint ret;\n+\n+\tci_dyn = vswap_lock_cluster(entry, \u0026voff);\n+\tif (!ci_dyn) {\n+\t\tif (typep)\n+\t\t\t*typep = VSWAP_NONE;\n+\t\treturn 0;\n+\t}\n+\tret = __vswap_check_backing(ci_dyn, voff, nr, typep);\n+\tswap_cluster_unlock(\u0026ci_dyn-\u003eci);\n+\treturn ret;\n+}\n+\n+/**\n+ * folio_phys_swap_backed - test whether a folio is backed by a contiguous\n+ * range of physical swap slots.\n+ * @folio: a swap-cache resident folio\n+ *\n+ * Return: %true if @folio-\u003eswap is not a vswap entry, or if these vswap\n+ * entries are backed by a contiguous range of physical slots.\n+ */\n+static inline bool folio_phys_swap_backed(struct folio *folio)\n+{\n+\tswp_entry_t entry = folio-\u003eswap;\n+\tint nr = folio_nr_pages(folio);\n+\tenum vswap_backing_type type;\n+\n+\treturn !is_vswap_entry(entry) ||\n+\t (vswap_check_backing(entry, nr, \u0026type) == nr \u0026\u0026\n+\t\ttype == VSWAP_SWAPFILE);\n+}\n+\n+static inline int vswap_cluster_alloc_vtable(struct swap_cluster_info_dynamic *ci_dyn,\n+\t\t\t\t\t gfp_t gfp)\n+{\n+\tci_dyn-\u003evirtual_table = kcalloc(SWAPFILE_CLUSTER,\n+\t\t\t\t\tsizeof(*ci_dyn-\u003evirtual_table), gfp);\n+\treturn ci_dyn-\u003evirtual_table ? 0 : -ENOMEM;\n+}\n+\n+static inline void vswap_cluster_free_vtable(struct swap_cluster_info *ci)\n+{\n+\tstruct swap_cluster_info_dynamic *ci_dyn;\n+\n+\tci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);\n+\tkfree(ci_dyn-\u003evirtual_table);\n+\tci_dyn-\u003evirtual_table = NULL;\n+}\n+\n+#else /* !CONFIG_SWAP */\n+\n+static inline bool vswap_is_enabled(void)\n+{\n+\treturn false;\n+}\n+\n+static inline bool is_vswap_entry(swp_entry_t entry)\n+{\n+\treturn false;\n+}\n+\n+static inline swp_entry_t vswap_to_phys(swp_entry_t entry)\n+{\n+\treturn (swp_entry_t){};\n+}\n+\n+static inline bool folio_phys_swap_backed(struct folio *folio)\n+{\n+\treturn true;\n+}\n+\n+#endif /* CONFIG_SWAP */\n+\n+/*\n+ * Test a per-backend swap flag (SWP_SYNCHRONOUS_IO, SWP_STABLE_WRITES, ...)\n+ * for @entry. For a vswap entry the property belongs to the current\n+ * physical backing rather than vswap_si itself; resolve to the backing\n+ * and test there. Returns false for zswap/zero/unbacked vswap entries\n+ * as they don't have a backing bdev.\n+ */\n+static inline bool swap_entry_backend_has_flag(struct swap_info_struct *si,\n+\t\t\t\t\t swp_entry_t entry,\n+\t\t\t\t\t unsigned long flag)\n+{\n+\tstruct swap_info_struct *phys_si;\n+\tswp_entry_t phys;\n+\tbool has_flag;\n+\n+\tif (!swap_is_vswap(si))\n+\t\treturn data_race(si-\u003eflags \u0026 flag);\n+\n+\tphys = vswap_to_phys(entry);\n+\tif (!phys.val)\n+\t\treturn false;\n+\n+\tphys_si = get_swap_device(phys);\n+\tif (!phys_si)\n+\t\treturn false;\n+\n+\thas_flag = data_race(phys_si-\u003eflags \u0026 flag);\n+\tput_swap_device(phys_si);\n+\treturn has_flag;\n+}\n+\n+#endif /* _MM_VSWAP_H */\ndiff --git a/mm/zswap.c b/mm/zswap.c\nindex 37f34e406c8e3..70ad8010f18a1 100644\n--- a/mm/zswap.c\n+++ b/mm/zswap.c\n@@ -38,6 +38,7 @@\n #include \u003clinux/zsmalloc.h\u003e\n \n #include \"swap.h\"\n+#include \"vswap.h\"\n #include \"internal.h\"\n \n /*********************************\n@@ -234,6 +235,25 @@ static inline struct xarray *swap_zswap_tree(swp_entry_t swp)\n \t\t\u003e\u003e ZSWAP_ADDRESS_SPACE_SHIFT];\n }\n \n+static struct zswap_entry *zswap_entry_load(swp_entry_t swp)\n+{\n+\tif (is_vswap_entry(swp))\n+\t\treturn vswap_zswap_load(swp);\n+\treturn xa_load(swap_zswap_tree(swp), swp_offset(swp));\n+}\n+\n+static struct zswap_entry *zswap_entry_store(swp_entry_t swp,\n+\t\t\t\t\t struct zswap_entry *entry)\n+{\n+\tif (is_vswap_entry(swp)) {\n+\t\tvswap_zswap_store(swp, entry);\n+\t\treturn NULL;\n+\t}\n+\n+\treturn xa_store(swap_zswap_tree(swp), swp_offset(swp), entry,\n+\t\t\tGFP_KERNEL);\n+}\n+\n #define zswap_pool_debug(msg, p)\t\t\t\\\n \tpr_debug(\"%s pool %s\\n\", msg, (p)-\u003etfm_name)\n \n@@ -762,7 +782,7 @@ static void zswap_entry_cache_free(struct zswap_entry *entry)\n * Carries out the common pattern of freeing an entry's zsmalloc allocation,\n * freeing the entry itself, and decrementing the number of stored pages.\n */\n-static void zswap_entry_free(struct zswap_entry *entry)\n+void zswap_entry_free(struct zswap_entry *entry)\n {\n \tzswap_lru_del(entry);\n \tzs_free(entry-\u003epool-\u003ezs_pool, entry-\u003ehandle);\n@@ -987,12 +1007,13 @@ static bool zswap_decompress(struct zswap_entry *entry, struct folio *folio)\n static int zswap_writeback_entry(struct zswap_entry *entry,\n \t\t\t\t swp_entry_t swpentry)\n {\n-\tstruct xarray *tree;\n \tpgoff_t offset = swp_offset(swpentry);\n \tstruct folio *folio;\n \tstruct mempolicy *mpol;\n \tstruct swap_info_struct *si;\n \tstruct swap_io_ctx ctx = {};\n+\tswp_entry_t phys = {};\n+\tbool is_vswap;\n \tint ret = 0;\n \n \t/* try to allocate swap cache folio */\n@@ -1000,6 +1021,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry,\n \tif (!si)\n \t\treturn -EEXIST;\n \n+\tis_vswap = swap_is_vswap(si);\n \tmpol = get_task_policy(current);\n \tfolio = swap_cache_alloc_folio(swpentry, GFP_KERNEL, BIT(0), NULL, mpol,\n \t\t\t\t NO_INTERLEAVE_INDEX);\n@@ -1018,24 +1040,44 @@ static int zswap_writeback_entry(struct zswap_entry *entry,\n \t/*\n \t * folio is locked, and the swapcache is now secured against\n \t * concurrent swapping to and from the slot, and concurrent\n-\t * swapoff so we can safely dereference the zswap tree here.\n+\t * swapoff so we can safely dereference the zswap tree (or vswap\n+\t * vtable) here.\n \t * Verify that the swap entry hasn't been invalidated and recycled\n \t * behind our backs, to avoid overwriting a new swap folio with\n \t * old compressed data. Only when this is successful can the entry\n \t * be dereferenced.\n \t */\n-\ttree = swap_zswap_tree(swpentry);\n-\tif (entry != xa_load(tree, offset)) {\n+\tif (entry != zswap_entry_load(swpentry)) {\n \t\tret = -ENOMEM;\n \t\tgoto out;\n \t}\n \n+\tif (is_vswap) {\n+\t\t/*\n+\t\t * Allocate physical backing before decompress so a failure\n+\t\t * wastes no work.\n+\t\t */\n+\t\tphys = folio_realloc_swap(folio);\n+\t\tif (!phys.val) {\n+\t\t\tret = -ENOMEM;\n+\t\t\tgoto out;\n+\t\t}\n+\t}\n+\n \tif (!zswap_decompress(entry, folio)) {\n \t\tret = -EIO;\n+\t\t/*\n+\t\t * The phys allocation above took the entry out of the vtable.\n+\t\t * Restore the zswap entry to the vtable, which also frees the\n+\t\t * allocated physical swap space.\n+\t\t */\n+\t\tif (is_vswap)\n+\t\t\tvswap_zswap_store(swpentry, entry);\n \t\tgoto out;\n \t}\n \n-\txa_erase(tree, offset);\n+\tif (!is_vswap)\n+\t\txa_erase(swap_zswap_tree(swpentry), offset);\n \n \tcount_vm_event(ZSWPWB);\n \tif (entry-\u003eobjcg)\n@@ -1050,7 +1092,10 @@ static int zswap_writeback_entry(struct zswap_entry *entry,\n \tfolio_set_reclaim(folio);\n \n \t/* start writeback */\n-\t__swap_writepage(\u0026ctx, folio);\n+\tif (is_vswap)\n+\t\t__swap_writepage(\u0026ctx, folio, phys);\n+\telse\n+\t\t__swap_writepage(\u0026ctx, folio, folio-\u003eswap);\n \tswap_write_submit(\u0026ctx);\n \n out:\n@@ -1065,6 +1110,15 @@ static int zswap_writeback_entry(struct zswap_entry *entry,\n /*********************************\n * shrinker functions\n **********************************/\n+/*\n+ * vswap zswap entries get a physical slot allocated on demand at writeback\n+ * time. Skip the shrinker when none is available.\n+ */\n+static bool zswap_writeback_possible(void)\n+{\n+\treturn !vswap_is_enabled() || get_nr_swap_pages() \u003e 0;\n+}\n+\n /*\n * The dynamic shrinker is modulated by the following factors:\n *\n@@ -1202,6 +1256,9 @@ static unsigned long zswap_shrinker_count(struct shrinker *shrinker,\n \tif (!zswap_shrinker_enabled || !mem_cgroup_zswap_writeback_enabled(memcg))\n \t\treturn 0;\n \n+\tif (!zswap_writeback_possible())\n+\t\treturn 0;\n+\n \t/*\n \t * The shrinker resumes swap writeback, which will enter block\n \t * and may enter fs. XXX: Harmonize with vmscan.c __GFP_FS\n@@ -1284,6 +1341,9 @@ static struct shrinker *zswap_alloc_shrinker(void)\n * Return: 0 if at least one entry was written back, -EAGAIN if entries\n * were scanned but none could be written back, or -ENOENT if @memcg has\n * writeback disabled, is a zombie cgroup, or has empty zswap LRUs.\n+ *\n+ * Also returns -ENOENT when vswap is enabled and there is no physical\n+ * swap to write back to.\n */\n static int shrink_memcg(struct mem_cgroup *memcg)\n {\n@@ -1292,6 +1352,9 @@ static int shrink_memcg(struct mem_cgroup *memcg)\n \tif (!mem_cgroup_zswap_writeback_enabled(memcg))\n \t\treturn -ENOENT;\n \n+\tif (!zswap_writeback_possible())\n+\t\treturn -ENOENT;\n+\n \t/*\n \t * Skip zombies because their LRUs are reparented and we would be\n \t * reclaiming from the parent instead of the dead memcg.\n@@ -1320,6 +1383,9 @@ static void shrink_worker(struct work_struct *w)\n \tint ret, failures = 0, attempts = 0;\n \tunsigned long thr;\n \n+\tif (!zswap_writeback_possible())\n+\t\treturn;\n+\n \t/* Reclaim down to the accept threshold */\n \tthr = zswap_accept_thr_pages();\n \n@@ -1398,7 +1464,7 @@ static void shrink_worker(struct work_struct *w)\n \t\t\tbreak;\n resched:\n \t\tcond_resched();\n-\t} while (zswap_total_pages() \u003e thr);\n+\t} while (zswap_total_pages() \u003e thr \u0026\u0026 zswap_writeback_possible());\n }\n \n /*********************************\n@@ -1422,9 +1488,7 @@ static bool zswap_store_page(struct page *page,\n \tif (!zswap_compress(page, entry, pool))\n \t\tgoto compress_failed;\n \n-\told = xa_store(swap_zswap_tree(page_swpentry),\n-\t\t swp_offset(page_swpentry),\n-\t\t entry, GFP_KERNEL);\n+\told = zswap_entry_store(page_swpentry, entry);\n \tif (xa_is_err(old)) {\n \t\tint err = xa_err(old);\n \n@@ -1493,7 +1557,7 @@ bool zswap_store(struct folio *folio)\n \tstruct mem_cgroup *memcg = NULL;\n \tstruct zswap_pool *pool;\n \tbool ret = false;\n-\tlong index;\n+\tlong index = 0;\n \n \tVM_WARN_ON_ONCE(!folio_test_locked(folio));\n \tVM_WARN_ON_ONCE(!folio_test_swapcache(folio));\n@@ -1548,13 +1612,19 @@ bool zswap_store(struct folio *folio)\n \tif (!ret \u0026\u0026 zswap_pool_reached_full)\n \t\tqueue_work(shrink_wq, \u0026zswap_shrink_work);\n check_old:\n+\tif (ret)\n+\t\treturn ret;\n+\n \t/*\n \t * If the zswap store fails or zswap is disabled, we must invalidate\n \t * the possibly stale entries which were previously stored at the\n \t * offsets corresponding to each page of the folio. Otherwise,\n \t * writeback could overwrite the new data in the swapfile.\n \t */\n-\tif (!ret) {\n+\tif (is_vswap_entry(swp)) {\n+\t\tif (index \u003e 0)\n+\t\t\tfolio_release_non_phys_swap_backing(folio);\n+\t} else {\n \t\tunsigned type = swp_type(swp);\n \t\tpgoff_t offset = swp_offset(swp);\n \t\tstruct zswap_entry *entry;\n@@ -1584,9 +1654,9 @@ bool zswap_store(struct folio *folio)\n * will SIGBUS).\n *\n * -EINVAL: if the swapped out content was in zswap, but the page belongs\n- * to a large folio, which is not supported by zswap. The folio is unlocked,\n- * but NOT marked up-to-date, so that an IO error is emitted (e.g.\n- * do_swap_page() will SIGBUS).\n+ * to a large non-vswap folio, which is not supported by zswap. The folio\n+ * is unlocked, but NOT marked up-to-date, so that an IO error is emitted\n+ * (e.g. do_swap_page() will SIGBUS).\n *\n * -ENOENT: if the swapped out content was not in zswap. The folio remains\n * locked on return.\n@@ -1594,8 +1664,7 @@ bool zswap_store(struct folio *folio)\n int zswap_load(struct folio *folio)\n {\n \tswp_entry_t swp = folio-\u003eswap;\n-\tpgoff_t offset = swp_offset(swp);\n-\tstruct xarray *tree = swap_zswap_tree(swp);\n+\tstruct swap_info_struct *si = __swap_entry_to_info(swp);\n \tstruct zswap_entry *entry;\n \n \tVM_WARN_ON_ONCE(!folio_test_locked(folio));\n@@ -1608,13 +1677,20 @@ int zswap_load(struct folio *folio)\n \t * Large folios should not be swapped in while zswap is being used, as\n \t * they are not properly handled. Zswap does not properly load large\n \t * folios, and a large folio may only be partially in zswap.\n+\t *\n+\t * A large vswap folio cannot reach here ZSWAP-backed, since\n+\t * __swap_cache_add_check() refuses such a batch, so hand it to the\n+\t * phys path without warning.\n \t */\n-\tif (WARN_ON_ONCE(folio_test_large(folio))) {\n-\t\tfolio_unlock(folio);\n-\t\treturn -EINVAL;\n+\tif (folio_test_large(folio)) {\n+\t\tif (WARN_ON_ONCE(!swap_is_vswap(si))) {\n+\t\t\tfolio_unlock(folio);\n+\t\t\treturn -EINVAL;\n+\t\t}\n+\t\treturn -ENOENT;\n \t}\n \n-\tentry = xa_load(tree, offset);\n+\tentry = zswap_entry_load(swp);\n \tif (!entry)\n \t\treturn -ENOENT;\n \n@@ -1637,8 +1713,13 @@ int zswap_load(struct folio *folio)\n \t * compression work.\n \t */\n \tfolio_mark_dirty(folio);\n-\txa_erase(tree, offset);\n-\tzswap_entry_free(entry);\n+\n+\tif (swap_is_vswap(si)) {\n+\t\tfolio_release_vswap_backing(folio);\n+\t} else {\n+\t\txa_erase(swap_zswap_tree(swp), swp_offset(swp));\n+\t\tzswap_entry_free(entry);\n+\t}\n \n \tfolio_unlock(folio);\n \treturn 0;\n"
} |
| 2/1 | 2026/08/25 15:55 | agent | patch-evaluator |
0mModel:best-expensive Args: null Results: {
"EnableConfigs": [
"VSWAP_DEFAULT_ON",
"ZSWAP",
"MEMCG"
],
"FocusSymbols": [
"folio_alloc_swap",
"folio_realloc_swap",
"folio_release_vswap_backing",
"folio_release_non_phys_swap_backing",
"__vswap_release_backing",
"__mem_cgroup_swap_charge",
"__mem_cgroup_swap_uncharge",
"__mem_cgroup_swap_get",
"__mem_cgroup_swap_record",
"__mem_cgroup_swap_put"
],
"Reasoning": "The patch introduces a major new core memory management feature: the virtual swap layer (vswap). It modifies core swap allocation, zswap interaction, and memcg swap accounting. The changes are highly reachable and complex, making them a prime target for fuzzing.",
"WorthFuzzing": true
}Instruction: You are an expert Linux kernel maintainer. Your job is to review a provided patch series and determine if it makes functional changes to the kernel that should be fuzzed. IMPORTANT: The changes have ALREADY been applied and committed as the HEAD commit in your workspace. Do NOT rely on your internal knowledge of the kernel. You must actively use your code access tools to examine the actual source code and confirm any assumptions. Return WorthFuzzing=false if the patch only contains: - Modifications to Documentation/, Kconfig files, or code comments. - Purely decorative changes, such as logging (e.g., pr_err, printk) or tracepoints. - Changes to numeric constants or macros that do not functionally alter execution flow. - Code paths that are impossible to reach in virtualized environments like GCE or QEMU, even when utilizing software-emulated hardware (e.g., usb gadget, mac80211_hwsim). - Code in vendor-specific PCIe switch, SmartNIC, or GPU drivers (e.g., mlxsw, pds_core, qed, ionic, amdgpu) that require physical PCIe hardware cards not emulated in standard QEMU. - Driver .remove, .shutdown, or pci_unregister_driver teardown callbacks (e.g., igb_remove) that are executed only during PCI hot-unplug or sysfs driver unbind operations. If it modifies reachable core kernel logic, drivers, or architectures, use your code search tools to verify the code can be executed, then return WorthFuzzing=true. When returning WorthFuzzing=true, you MUST ALSO: 1. Extract any specific kernel functions that should be heavily fuzzed into FocusSymbols. Avoid listing generic hot-path functions to prevent skewed test distributions. Prefer non-static, non-inlined API entrypoint functions over internal static helper functions (which are inlined by the compiler and do not have distinct symbol addresses). 2. Identify any specific CONFIG_ options required to properly test this new/modified feature. Go and look into the Kconfig files and check for ifdefs around the code, do not make assumptions. Also check "depends on" lines in Kconfig to include any non-standard parent subsystem configs needed for Kbuild to compile the code statically into vmlinux. List them in the EnableConfigs output array, and DO NOT add a 'CONFIG_' prefix (e.g., return "NET_IPV4" instead of "CONFIG_NET_IPV4"). Prefer calling several tools at the same time to save round-trips. Use set-results tool to provide results of the analysis. It must be called exactly once before the final reply. Ignore results of this tool. Prompt: For your convenience, here is the diff of the changes:
commit 90158fb09c94f90a0fb0c502bca752b81303fd9d
Author: syz-cluster <triage@syzkaller.com>
Date: Tue Aug 25 15:55:55 2026 +0000
syz-cluster: applied patch under review
diff --git a/Documentation/admin-guide/cgroup-v1/memcg_test.rst b/Documentation/admin-guide/cgroup-v1/memcg_test.rst
index ebedbc3c3f9cc..13b9ae800b728 100644
--- a/Documentation/admin-guide/cgroup-v1/memcg_test.rst
+++ b/Documentation/admin-guide/cgroup-v1/memcg_test.rst
@@ -43,7 +43,7 @@ Please note that implementation details can be changed.
mem_cgroup_uncharge()
Called when a page's refcount goes down to 0.
- mem_cgroup_uncharge_swap()
+ mem_cgroup_swap_uncharge()
Called when swp_entry's refcnt goes down to 0. A charge against swap
disappears.
diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 1af62cd16c9de..6612b5e0a0554 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -8364,6 +8364,13 @@ Kernel parameters
force - force vulnerability detection even on
unaffected processors
+ vswap= [MM,EARLY]
+ Route swapouts through the virtual swap layer, which
+ allows zswap and zero-filled pages to be used without
+ a physical swap device. 64-bit only.
+ Format: { on | off }
+ Default: on if CONFIG_VSWAP_DEFAULT_ON=y, else off.
+
vsyscall= [X86-64,EARLY]
Controls the behavior of vsyscalls (i.e. calls to
fixed addresses of 0xffffffffff600x00 from legacy
diff --git a/MAINTAINERS b/MAINTAINERS
index 29236523cefb6..8a5827d271778 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -17251,6 +17251,7 @@ F: mm/swap.h
F: mm/swap_table.h
F: mm/swap_state.c
F: mm/swapfile.c
+F: mm/vswap.h
MEMORY MANAGEMENT - THP (TRANSPARENT HUGE PAGE)
M: Andrew Morton <akpm@linux-foundation.org>
diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index 215e2e87f42b2..4d89a35f49ff4 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -1905,6 +1905,7 @@ static inline bool memcg_is_dying(struct mem_cgroup *memcg)
#if defined(CONFIG_MEMCG) && defined(CONFIG_ZSWAP)
bool obj_cgroup_may_zswap(struct obj_cgroup *objcg);
+bool mem_cgroup_may_zswap(struct mem_cgroup *memcg, bool may_flush);
void obj_cgroup_charge_zswap(struct obj_cgroup *objcg, size_t size);
void obj_cgroup_uncharge_zswap(struct obj_cgroup *objcg, size_t size);
bool mem_cgroup_zswap_writeback_enabled(struct mem_cgroup *memcg);
@@ -1913,6 +1914,11 @@ static inline bool obj_cgroup_may_zswap(struct obj_cgroup *objcg)
{
return true;
}
+
+static inline bool mem_cgroup_may_zswap(struct mem_cgroup *memcg, bool may_flush)
+{
+ return true;
+}
static inline void obj_cgroup_charge_zswap(struct obj_cgroup *objcg,
size_t size)
{
diff --git a/include/linux/swap.h b/include/linux/swap.h
index 5658a1634b85e..5310ac0f3faae 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -207,6 +207,7 @@ enum {
SWP_STABLE_WRITES = (1 << 11), /* no overwrite PG_writeback pages */
SWP_SYNCHRONOUS_IO = (1 << 12), /* synchronous IO is efficient */
SWP_HIBERNATION = (1 << 13), /* pinned for hibernation */
+ SWP_VSWAP = (1 << 14), /* virtual swap device */
/* add others here before... */
};
@@ -245,7 +246,7 @@ struct swap_info_struct {
signed short prio; /* swap priority of this type */
struct plist_node list; /* entry in swap_active_head */
signed char type; /* strange name for an index */
- unsigned int max; /* size of this swap device */
+ unsigned long max; /* size of this swap device */
struct swap_cluster_info *cluster_info; /* cluster info. Only for SSD */
struct list_head free_clusters; /* free clusters list */
struct list_head full_clusters; /* full clusters list */
@@ -253,7 +254,7 @@ struct swap_info_struct {
/* list of cluster that contains at least one free slot */
struct list_head frag_clusters[SWAP_NR_ORDERS];
/* list of cluster that are fragmented or contented */
- unsigned int pages; /* total of usable pages of swap */
+ unsigned long pages; /* total of usable pages of swap */
atomic_long_t inuse_pages; /* number of those currently in use */
struct swap_sequential_cluster *global_cluster; /* Use one global cluster for rotating device */
spinlock_t global_cluster_lock; /* Serialize usage of global cluster */
@@ -276,8 +277,14 @@ struct swap_info_struct {
struct list_head discard_clusters; /* discard clusters list */
struct plist_node avail_list; /* entry in swap_avail_head */
const struct swap_ops *ops;
+ struct xarray cluster_info_pool; /* Xarray for vswap dynamic cluster info */
};
+static inline bool swap_is_vswap(struct swap_info_struct *si)
+{
+ return si->flags & SWP_VSWAP;
+}
+
static inline swp_entry_t page_swap_entry(struct page *page)
{
struct folio *folio = page_folio(page);
@@ -381,7 +388,7 @@ extern int __swap_count(swp_entry_t entry);
extern bool swap_entry_swapped(struct swap_info_struct *si, swp_entry_t entry);
extern int swp_swapcount(swp_entry_t entry);
extern struct swap_info_struct *get_swap_device(swp_entry_t entry);
-sector_t swap_folio_sector(struct folio *folio);
+sector_t swap_entry_sector(swp_entry_t entry);
/*
* If there is an existing swap slot reference (swap entry) and the caller
@@ -408,6 +415,8 @@ void swap_free_hibernation_slot(swp_entry_t entry);
static inline void put_swap_device(struct swap_info_struct *si)
{
+ if (swap_is_vswap(si))
+ return;
percpu_ref_put(&si->users);
}
@@ -492,35 +501,80 @@ static inline void folio_throttle_swaprate(struct folio *folio, gfp_t gfp)
#endif
#if defined(CONFIG_MEMCG) && defined(CONFIG_SWAP)
-int __mem_cgroup_try_charge_swap(struct folio *folio);
-static inline int mem_cgroup_try_charge_swap(struct folio *folio)
+struct mem_cgroup *__mem_cgroup_swap_get(struct folio *folio);
+static inline struct mem_cgroup *mem_cgroup_swap_get(struct folio *folio)
+{
+ if (mem_cgroup_disabled())
+ return NULL;
+ return __mem_cgroup_swap_get(folio);
+}
+
+int __mem_cgroup_swap_charge(struct mem_cgroup *memcg, unsigned int nr_pages);
+static inline int mem_cgroup_swap_charge(struct mem_cgroup *memcg,
+ unsigned int nr_pages)
{
if (mem_cgroup_disabled())
return 0;
- return __mem_cgroup_try_charge_swap(folio);
+ return __mem_cgroup_swap_charge(memcg, nr_pages);
}
-extern void __mem_cgroup_uncharge_swap(unsigned short id, unsigned int nr_pages);
-static inline void mem_cgroup_uncharge_swap(unsigned short id, unsigned int nr_pages)
+void __mem_cgroup_swap_record(struct folio *folio, struct mem_cgroup *memcg);
+static inline void mem_cgroup_swap_record(struct folio *folio,
+ struct mem_cgroup *memcg)
{
if (mem_cgroup_disabled())
return;
- __mem_cgroup_uncharge_swap(id, nr_pages);
+ __mem_cgroup_swap_record(folio, memcg);
+}
+
+void __mem_cgroup_swap_uncharge(struct mem_cgroup *memcg,
+ unsigned int nr_pages);
+static inline void mem_cgroup_swap_uncharge(struct mem_cgroup *memcg,
+ unsigned int nr_pages)
+{
+ if (mem_cgroup_disabled())
+ return;
+ __mem_cgroup_swap_uncharge(memcg, nr_pages);
+}
+
+void __mem_cgroup_swap_put(struct mem_cgroup *memcg, unsigned int nr_pages);
+static inline void mem_cgroup_swap_put(struct mem_cgroup *memcg,
+ unsigned int nr_pages)
+{
+ if (mem_cgroup_disabled())
+ return;
+ __mem_cgroup_swap_put(memcg, nr_pages);
}
extern long mem_cgroup_get_nr_swap_pages(struct mem_cgroup *memcg);
extern bool mem_cgroup_swap_full(struct folio *folio);
#else
-static inline int mem_cgroup_try_charge_swap(struct folio *folio)
+static inline struct mem_cgroup *mem_cgroup_swap_get(struct folio *folio)
+{
+ return NULL;
+}
+
+static inline int mem_cgroup_swap_charge(struct mem_cgroup *memcg,
+ unsigned int nr_pages)
{
return 0;
}
-static inline void mem_cgroup_uncharge_swap(unsigned short id,
+static inline void mem_cgroup_swap_record(struct folio *folio,
+ struct mem_cgroup *memcg)
+{
+}
+
+static inline void mem_cgroup_swap_uncharge(struct mem_cgroup *memcg,
unsigned int nr_pages)
{
}
+static inline void mem_cgroup_swap_put(struct mem_cgroup *memcg,
+ unsigned int nr_pages)
+{
+}
+
static inline long mem_cgroup_get_nr_swap_pages(struct mem_cgroup *memcg)
{
return get_nr_swap_pages();
diff --git a/include/linux/swap_ops.h b/include/linux/swap_ops.h
index 57ac6c703f685..223c84548bde6 100644
--- a/include/linux/swap_ops.h
+++ b/include/linux/swap_ops.h
@@ -12,6 +12,7 @@ struct swap_iocb {
struct bio_vec bvecs[SWAP_CLUSTER_MAX];
int nr_bvecs;
int len;
+ swp_entry_t entry; /* first slot in the batch; addresses the IO */
};
struct swap_io_ctx {
@@ -30,15 +31,15 @@ struct swap_io_ctx {
struct swap_ops {
unsigned int flags;
- bool (*can_merge)(struct folio *folio, struct folio *prev_folio,
- size_t prev_folio_size, int rw);
+ bool (*can_merge)(struct folio *folio, swp_entry_t phys,
+ struct swap_iocb *sio, int rw);
void (*submit_write)(struct swap_io_ctx *ctx);
void (*submit_read)(struct swap_io_ctx *ctx);
};
void swap_fs_prepare_rw(struct swap_io_ctx *ctx, int rw, struct iov_iter *iter);
-bool swap_fs_can_merge(struct folio *folio, struct folio *prev_folio,
- size_t prev_folio_size, int rw);
+bool swap_fs_can_merge(struct folio *folio, swp_entry_t phys,
+ struct swap_iocb *sio, int rw);
int swap_fs_activate(struct swap_info_struct *sis, const struct swap_ops *ops);
#endif /* _MM_SWAP_OPS_H */
diff --git a/include/linux/zswap.h b/include/linux/zswap.h
index 30c193a1207e1..4b4f211f33017 100644
--- a/include/linux/zswap.h
+++ b/include/linux/zswap.h
@@ -6,6 +6,7 @@
#include <linux/mm_types.h>
struct lruvec;
+struct zswap_entry;
extern atomic_long_t zswap_stored_pages;
@@ -28,6 +29,7 @@ unsigned long zswap_total_pages(void);
bool zswap_store(struct folio *folio);
int zswap_load(struct folio *folio);
void zswap_invalidate(swp_entry_t swp);
+void zswap_entry_free(struct zswap_entry *entry);
int zswap_swapon(int type, unsigned long nr_pages);
void zswap_swapoff(int type);
void zswap_memcg_offline_cleanup(struct mem_cgroup *memcg);
@@ -50,6 +52,7 @@ static inline int zswap_load(struct folio *folio)
}
static inline void zswap_invalidate(swp_entry_t swp) {}
+static inline void zswap_entry_free(struct zswap_entry *entry) {}
static inline int zswap_swapon(int type, unsigned long nr_pages)
{
return 0;
diff --git a/mm/Kconfig b/mm/Kconfig
index 604c58199acbf..08fdc7502c1de 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -19,6 +19,26 @@ menuconfig SWAP
used to provide more virtual memory than the actual RAM present
in your computer. If unsure say Y.
+config VSWAP_DEFAULT_ON
+ bool "Route swapouts through virtual swap by default"
+ depends on SWAP && 64BIT
+ default n
+ help
+ Virtual swap allows zswap and zero-filled pages to be used
+ without swapping on a physical device first, and lets a page
+ move between zswap and a swapfile without invalidating the page
+ table entries that refer to it.
+
+ Swap entries are handed out by a virtual swap device instead of
+ naming a slot on a real one, so the backing can be chosen and
+ changed after the entry exists.
+
+ Say Y to make "vswap=on" the default, routing swapouts through
+ the virtual swap layer from boot.
+
+ Say N (default) to leave vswap off unless "vswap=on" is passed
+ on the kernel command line.
+
config ZSWAP
bool "Compressed cache for swap pages"
depends on SWAP
diff --git a/mm/memcontrol-v1.c b/mm/memcontrol-v1.c
index 05ef55cae4dc6..88016c8f22a26 100644
--- a/mm/memcontrol-v1.c
+++ b/mm/memcontrol-v1.c
@@ -690,6 +690,7 @@ void __memcg1_swapout(struct folio *folio, struct swap_cluster_info *ci)
void memcg1_swapin(struct folio *folio)
{
struct swap_cluster_info *ci;
+ struct mem_cgroup *memcg;
unsigned long nr_pages;
unsigned short id;
@@ -721,7 +722,14 @@ void memcg1_swapin(struct folio *folio)
id = __swap_cgroup_clear(ci, swp_cluster_offset(folio->swap),
nr_pages);
swap_cluster_unlock(ci);
- mem_cgroup_uncharge_swap(id, nr_pages);
+
+ rcu_read_lock();
+ memcg = mem_cgroup_from_private_id(id);
+ if (memcg) {
+ mem_cgroup_swap_uncharge(memcg, nr_pages);
+ mem_cgroup_swap_put(memcg, nr_pages);
+ }
+ rcu_read_unlock();
}
#endif
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 11b85f4b6828b..6ae0a4191d887 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -65,6 +65,7 @@
#include "internal.h"
#include "swap.h"
#include "swap_table.h"
+#include "vswap.h"
#include <net/sock.h>
#include <net/ip.h>
#include "slab.h"
@@ -5721,86 +5722,129 @@ int __init mem_cgroup_init(void)
#ifdef CONFIG_SWAP
/**
- * __mem_cgroup_try_charge_swap - try charging swap space for a folio
+ * __mem_cgroup_swap_get - pin the memcg to account a folio's swap slots to
* @folio: folio being added to swap
*
- * Try to charge @folio's memcg for the swap space at folio->swap.
+ * Pins one private ID ref per page of @folio on its memcg, or on its closest
+ * online ancestor if it has been offlined. The caller charges and records
+ * against whichever memcg is returned, so both land on the same one.
*
- * Returns 0 on success, -ENOMEM on failure.
+ * Return: the pinned memcg, or NULL if there is nothing to account. Drop the
+ * pins with __mem_cgroup_swap_put().
*/
-int __mem_cgroup_try_charge_swap(struct folio *folio)
+struct mem_cgroup *__mem_cgroup_swap_get(struct folio *folio)
{
unsigned int nr_pages = folio_nr_pages(folio);
- struct swap_cluster_info *ci;
- struct page_counter *counter;
struct mem_cgroup *memcg;
struct obj_cgroup *objcg;
if (do_memsw_account())
- return 0;
+ return NULL;
objcg = folio_objcg(folio);
VM_WARN_ON_ONCE_FOLIO(!objcg, folio);
if (!objcg)
- return 0;
+ return NULL;
rcu_read_lock();
memcg = obj_cgroup_memcg(objcg);
if (!folio_test_swapcache(folio)) {
memcg_memory_event(memcg, MEMCG_SWAP_FAIL);
rcu_read_unlock();
- return 0;
+ return NULL;
}
memcg = mem_cgroup_private_id_get_online(memcg, nr_pages);
/* memcg is pined by memcg ID. */
rcu_read_unlock();
+ return memcg;
+}
+
+/**
+ * __mem_cgroup_swap_charge - charge physical swap space
+ * @memcg: the mem_cgroup to charge (may be NULL)
+ * @nr_pages: the amount of swap space to charge
+ *
+ * Return: 0 on success, -ENOMEM if memory.swap.max is exceeded.
+ */
+int __mem_cgroup_swap_charge(struct mem_cgroup *memcg, unsigned int nr_pages)
+{
+ struct page_counter *counter;
+
+ if (do_memsw_account() || !memcg)
+ return 0;
+
if (!mem_cgroup_is_root(memcg) &&
!page_counter_try_charge(&memcg->swap, nr_pages, &counter)) {
memcg_memory_event(memcg, MEMCG_SWAP_MAX);
memcg_memory_event(memcg, MEMCG_SWAP_FAIL);
- mem_cgroup_private_id_put(memcg, nr_pages);
return -ENOMEM;
}
mod_memcg_state(memcg, MEMCG_SWAP, nr_pages);
+ return 0;
+}
+
+/**
+ * __mem_cgroup_swap_record - record the owner of a folio's swap slots
+ * @folio: folio being added to swap
+ * @memcg: the memcg pinned by __mem_cgroup_swap_get()
+ */
+void __mem_cgroup_swap_record(struct folio *folio, struct mem_cgroup *memcg)
+{
+ struct swap_cluster_info *ci;
ci = swap_cluster_get_and_lock(folio);
- __swap_cgroup_set(ci, swp_cluster_offset(folio->swap), nr_pages,
- mem_cgroup_private_id(memcg));
+ __swap_cgroup_set(ci, swp_cluster_offset(folio->swap),
+ folio_nr_pages(folio), mem_cgroup_private_id(memcg));
swap_cluster_unlock(ci);
-
- return 0;
}
/**
- * __mem_cgroup_uncharge_swap - uncharge swap space
- * @id: cgroup id to uncharge
+ * __mem_cgroup_swap_uncharge - uncharge physical swap space
+ * @memcg: the mem_cgroup to uncharge (may be NULL)
* @nr_pages: the amount of swap space to uncharge
*/
-void __mem_cgroup_uncharge_swap(unsigned short id, unsigned int nr_pages)
+void __mem_cgroup_swap_uncharge(struct mem_cgroup *memcg, unsigned int nr_pages)
{
- struct mem_cgroup *memcg;
+ if (!memcg)
+ return;
- rcu_read_lock();
- memcg = mem_cgroup_from_private_id(id);
- if (memcg) {
- if (!mem_cgroup_is_root(memcg)) {
- if (do_memsw_account())
- page_counter_uncharge(&memcg->memsw, nr_pages);
- else
- page_counter_uncharge(&memcg->swap, nr_pages);
- }
- mod_memcg_state(memcg, MEMCG_SWAP, -nr_pages);
- mem_cgroup_private_id_put(memcg, nr_pages);
+ if (!mem_cgroup_is_root(memcg)) {
+ if (do_memsw_account())
+ page_counter_uncharge(&memcg->memsw, nr_pages);
+ else
+ page_counter_uncharge(&memcg->swap, nr_pages);
}
- rcu_read_unlock();
+ mod_memcg_state(memcg, MEMCG_SWAP, -nr_pages);
+}
+
+/**
+ * __mem_cgroup_swap_put - drop the private ID refs taken for swap slots
+ * @memcg: the pinned mem_cgroup
+ * @nr_pages: number of refs to drop
+ */
+void __mem_cgroup_swap_put(struct mem_cgroup *memcg, unsigned int nr_pages)
+{
+ mem_cgroup_private_id_put(memcg, nr_pages);
}
long mem_cgroup_get_nr_swap_pages(struct mem_cgroup *memcg)
{
- long nr_swap_pages = get_nr_swap_pages();
+ long nr_swap_pages;
+
+ /*
+ * vswap charges physical backing, not allocation, so virtual swap is
+ * unbounded for a zswap-capable memcg and the swap.max walk below
+ * would starve anon reclaim. swap.max is still enforced when the
+ * backing is charged.
+ */
+ if (vswap_is_enabled() && zswap_is_enabled() &&
+ (mem_cgroup_disabled() || do_memsw_account() ||
+ mem_cgroup_may_zswap(memcg, false)))
+ return PAGE_COUNTER_MAX;
+ nr_swap_pages = get_nr_swap_pages();
if (mem_cgroup_disabled() || do_memsw_account())
return nr_swap_pages;
for (; !mem_cgroup_is_root(memcg); memcg = parent_mem_cgroup(memcg))
@@ -5972,8 +6016,10 @@ static struct cftype swap_files[] = {
#ifdef CONFIG_ZSWAP
/**
- * obj_cgroup_may_zswap - check if this cgroup can zswap
- * @objcg: the object cgroup
+ * mem_cgroup_may_zswap - check if this cgroup can zswap
+ * @memcg: the memcg to query
+ * @may_flush: force-flush stats for an accurate check (sleeps). Pass false
+ * from atomic contexts; the check is then best-effort.
*
* Check if the hierarchical zswap limit has been reached.
*
@@ -5983,36 +6029,38 @@ static struct cftype swap_files[] = {
* spending cycles on compression when there is already no room left
* or zswap is disabled altogether somewhere in the hierarchy.
*/
-bool obj_cgroup_may_zswap(struct obj_cgroup *objcg)
+bool mem_cgroup_may_zswap(struct mem_cgroup *memcg, bool may_flush)
{
- struct mem_cgroup *memcg, *original_memcg;
- bool ret = true;
-
if (!cgroup_subsys_on_dfl(memory_cgrp_subsys))
return true;
- original_memcg = get_mem_cgroup_from_objcg(objcg);
- for (memcg = original_memcg; !mem_cgroup_is_root(memcg);
- memcg = parent_mem_cgroup(memcg)) {
+ for (; !mem_cgroup_is_root(memcg); memcg = parent_mem_cgroup(memcg)) {
unsigned long max = READ_ONCE(memcg->zswap_max);
unsigned long pages;
if (max == PAGE_COUNTER_MAX)
continue;
- if (max == 0) {
- ret = false;
- break;
- }
+ if (max == 0)
+ return false;
/* Force flush to get accurate stats for charging */
- __mem_cgroup_flush_stats(memcg, true);
+ if (may_flush)
+ __mem_cgroup_flush_stats(memcg, true);
pages = memcg_page_state(memcg, MEMCG_ZSWAP_B) / PAGE_SIZE;
- if (pages < max)
- continue;
- ret = false;
- break;
+ if (pages >= max)
+ return false;
}
- mem_cgroup_put(original_memcg);
+ return true;
+}
+
+bool obj_cgroup_may_zswap(struct obj_cgroup *objcg)
+{
+ struct mem_cgroup *memcg;
+ bool ret;
+
+ memcg = get_mem_cgroup_from_objcg(objcg);
+ ret = mem_cgroup_may_zswap(memcg, true);
+ mem_cgroup_put(memcg);
return ret;
}
diff --git a/mm/memory.c b/mm/memory.c
index c549433025532..62f7b82427e21 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -89,6 +89,7 @@
#include "pgalloc-track.h"
#include "internal.h"
#include "swap.h"
+#include "vswap.h"
#if defined(LAST_CPUPID_NOT_IN_PAGE_FLAGS) && !defined(CONFIG_COMPILE_TEST)
#warning Unfortunate NUMA and NUMA Balancing config, growing page-frame for last_cpupid.
@@ -4659,7 +4660,13 @@ static inline bool should_try_to_free_swap(struct swap_info_struct *si,
* are fast, and meanwhile, swap cache pinning the slot deferring the
* release of metadata or fragmentation is a more critical issue.
*/
- if (data_race(si->flags & SWP_SYNCHRONOUS_IO))
+ if (swap_entry_backend_has_flag(si, folio->swap, SWP_SYNCHRONOUS_IO))
+ return true;
+ /*
+ * Non-swapfile backends cannot be reused for future swapouts.
+ * Free the swap slot unless backed by contiguous physical swap.
+ */
+ if (!folio_phys_swap_backed(folio))
return true;
if (mem_cgroup_swap_full(folio) || (vma->vm_flags & VM_LOCKED) ||
folio_test_mlocked(folio))
@@ -4809,15 +4816,19 @@ static unsigned long thp_swapin_suitable_orders(struct vm_fault *vmf)
if (unlikely(userfaultfd_armed(vma)))
return 0;
+ entry = softleaf_from_pte(vmf->orig_pte);
+
/*
* A large swapped out folio could be partially or fully in zswap. We
* lack handling for such cases, so fallback to swapping in order-0
* folio.
+ *
+ * Vswap entries are checked later, under the cluster lock in
+ * __swap_cache_add_check().
*/
- if (!zswap_never_enabled())
+ if (!is_vswap_entry(entry) && !zswap_never_enabled())
return 0;
- entry = softleaf_from_pte(vmf->orig_pte);
/*
* Get a list of all the (large) orders below PMD_ORDER that are enabled
* and suitable for swapping THP.
@@ -4963,7 +4974,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
swap_update_readahead(folio, vma, vmf->address);
if (!folio) {
/* Swapin bypasses readahead for SWP_SYNCHRONOUS_IO devices */
- if (data_race(si->flags & SWP_SYNCHRONOUS_IO))
+ if (swap_entry_backend_has_flag(si, entry, SWP_SYNCHRONOUS_IO))
folio = swapin_sync(entry, GFP_HIGHUSER_MOVABLE,
thp_swapin_suitable_orders(vmf) | BIT(0),
vmf, NULL, 0);
@@ -5128,7 +5139,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
*/
exclusive = true;
} else if (exclusive && folio_test_writeback(folio) &&
- data_race(si->flags & SWP_STABLE_WRITES)) {
+ swap_entry_backend_has_flag(si, entry, SWP_STABLE_WRITES)) {
/*
* This is tricky: not all swap backends support
* concurrent page modifications while under writeback.
diff --git a/mm/page_io.c b/mm/page_io.c
index 88962571cb931..b36a898358c64 100644
--- a/mm/page_io.c
+++ b/mm/page_io.c
@@ -28,6 +28,7 @@
#include <linux/swap_ops.h>
#include "swap.h"
#include "swap_table.h"
+#include "vswap.h"
int generic_swapfile_activate(struct swap_info_struct *sis,
struct file *swap_file,
@@ -160,14 +161,19 @@ static void swap_zeromap_folio_set(struct folio *folio)
struct obj_cgroup *objcg = get_obj_cgroup_from_folio(folio);
int nr_pages = folio_nr_pages(folio);
struct swap_cluster_info *ci;
+ unsigned int voff, i;
swp_entry_t entry;
- unsigned int i;
VM_WARN_ON_ONCE_FOLIO(!folio_test_swapcache(folio), folio);
VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
ci = swap_cluster_get_and_lock(folio);
- for (i = 0; i < folio_nr_pages(folio); i++) {
+ if (is_vswap_entry(folio->swap)) {
+ /* Free any prior backing (e.g. ZSWAP entry from earlier swapout) */
+ voff = swp_cluster_offset(folio->swap);
+ __vswap_release_backing(ci, voff, nr_pages);
+ }
+ for (i = 0; i < nr_pages; i++) {
entry = page_swap_entry(folio_page(folio, i));
__swap_table_set_zero(ci, swp_cluster_offset(entry));
}
@@ -203,6 +209,7 @@ static void swap_zeromap_folio_clear(struct folio *folio)
*/
int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio)
{
+ swp_entry_t phys;
int ret = 0;
if (folio_free_swap(folio))
@@ -235,6 +242,15 @@ int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio)
*/
swap_zeromap_folio_clear(folio);
+ /*
+ * For vswap: release stale non-swapfile backings (e.g. ZSWAP from a
+ * previous swapout cycle) so zswap_store or folio_realloc_swap
+ * starts on clean slots. Contiguous PHYS backing is preserved for
+ * reuse by folio_realloc_swap.
+ */
+ if (is_vswap_entry(folio->swap))
+ folio_release_non_phys_swap_backing(folio);
+
if (zswap_store(folio)) {
count_mthp_stat(folio_order(folio), MTHP_STAT_ZSWPOUT);
goto out_unlock;
@@ -248,7 +264,23 @@ int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio)
}
rcu_read_unlock();
- __swap_writepage(ctx, folio);
+ /*
+ * A vswap folio with no backend needs a physical slot to write to.
+ * zswap_store rolled back any partial vtable state on failure, so
+ * PHYS backing from a prior cycle is still there to reuse. If none
+ * is free, keep it dirty.
+ */
+ if (is_vswap_entry(folio->swap)) {
+ phys = folio_realloc_swap(folio);
+ if (!phys.val) {
+ folio_mark_dirty(folio);
+ return AOP_WRITEPAGE_ACTIVATE;
+ }
+ __swap_writepage(ctx, folio, phys);
+ return 0;
+ }
+
+ __swap_writepage(ctx, folio, folio->swap);
return 0;
out_unlock:
folio_unlock(folio);
@@ -317,24 +349,22 @@ int sio_pool_init(void)
}
static bool swap_can_merge(struct swap_io_ctx *ctx, struct folio *folio,
- int rw)
+ swp_entry_t phys, int rw)
{
- struct swap_info_struct *sis = __swap_entry_to_info(folio->swap);
- struct bio_vec *last_bv = &ctx->sio->bvecs[ctx->sio->nr_bvecs - 1];
- struct folio *prev_folio = bvec_folio(last_bv);
- size_t prev_folio_size = folio_size(prev_folio);
+ struct swap_info_struct *sis = __swap_entry_to_info(phys);
if (ctx->sis != sis)
return false;
- return sis->ops->can_merge(folio, prev_folio, prev_folio_size, rw);
+ return sis->ops->can_merge(folio, phys, ctx->sio, rw);
}
-static void swap_add_folio(struct swap_io_ctx *ctx, struct folio *folio, int rw)
+static void swap_add_folio(struct swap_io_ctx *ctx, struct folio *folio,
+ swp_entry_t phys, int rw)
{
- struct swap_info_struct *sis = __swap_entry_to_info(folio->swap);
+ struct swap_info_struct *sis = __swap_entry_to_info(phys);
struct swap_iocb *sio = ctx->sio;
- if (sio && !swap_can_merge(ctx, folio, rw)) {
+ if (sio && !swap_can_merge(ctx, folio, phys, rw)) {
if (rw == WRITE)
swap_write_submit(ctx);
else
@@ -347,6 +377,7 @@ static void swap_add_folio(struct swap_io_ctx *ctx, struct folio *folio, int rw)
ctx->sio = sio = mempool_alloc(sio_pool, GFP_NOIO);
sio->nr_bvecs = 0;
sio->len = 0;
+ sio->entry = phys;
}
bvec_set_folio(&sio->bvecs[sio->nr_bvecs], folio, folio_size(folio), 0);
sio->len += folio_size(folio);
@@ -367,7 +398,8 @@ static void swap_add_folio(struct swap_io_ctx *ctx, struct folio *folio, int rw)
}
}
-void __swap_writepage(struct swap_io_ctx *ctx, struct folio *folio)
+void __swap_writepage(struct swap_io_ctx *ctx, struct folio *folio,
+ swp_entry_t phys)
{
VM_BUG_ON_FOLIO(!folio_test_swapcache(folio), folio);
@@ -383,7 +415,7 @@ void __swap_writepage(struct swap_io_ctx *ctx, struct folio *folio)
folio_start_writeback(folio);
folio_unlock(folio);
- swap_add_folio(ctx, folio, WRITE);
+ swap_add_folio(ctx, folio, phys, WRITE);
}
/*
@@ -456,6 +488,7 @@ void swap_read_folio(struct swap_io_ctx *ctx, struct folio *folio)
bool workingset = folio_test_workingset(folio);
unsigned long pflags;
bool in_thrashing;
+ swp_entry_t phys;
VM_BUG_ON_FOLIO(!folio_test_swapcache(folio) && !synchronous, folio);
VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
@@ -480,9 +513,24 @@ void swap_read_folio(struct swap_io_ctx *ctx, struct folio *folio)
if (zswap_load(folio) != -ENOENT)
goto finish;
+ /*
+ * Resolve the physical slot to read from. A vswap entry keeps
+ * folio->swap virtual, so map it to its physical backing; a folio with
+ * no backing has nothing to read.
+ */
+ if (swap_is_vswap(sis)) {
+ phys = vswap_to_phys(folio->swap);
+ if (!phys.val) {
+ folio_unlock(folio);
+ goto finish;
+ }
+ } else {
+ phys = folio->swap;
+ }
+
/* We have to read from slower devices. Increase zswap protection. */
zswap_folio_swapin(folio);
- swap_add_folio(ctx, folio, READ);
+ swap_add_folio(ctx, folio, phys, READ);
finish:
if (workingset) {
@@ -514,8 +562,6 @@ static void swap_fs_write_complete(struct kiocb *iocb, long ret)
bool failed = ret != sio->len;
if (failed) {
- struct page *page = sio->bvecs[0].bv_page;
-
/*
* In the case of swap-over-nfs, this can be a temporary failure
* if the system has limited memory for allocating transmit
@@ -523,7 +569,7 @@ static void swap_fs_write_complete(struct kiocb *iocb, long ret)
* folio_rotate_reclaimable but rate-limit the messages.
*/
pr_err_ratelimited("Write error %ld on dio swapfile (%llu)\n",
- ret, swap_dev_pos(page_swap_entry(page)));
+ ret, swap_dev_pos(sio->entry));
}
swap_write_end(sio, failed);
@@ -595,7 +641,7 @@ static void swap_bdev_submit_write(struct swap_io_ctx *ctx)
bio_init(bio, ctx->sis->bdev, sio->bvecs, ARRAY_SIZE(sio->bvecs),
REQ_OP_WRITE | REQ_SWAP);
bio->bi_iter.bi_size = sio->len;
- bio->bi_iter.bi_sector = swap_folio_sector(bio_first_folio_all(bio));
+ bio->bi_iter.bi_sector = swap_entry_sector(sio->entry);
bio_associate_blkg_from_page(bio, bio_first_folio_all(bio));
if (ctx->sis->flags & SWP_SYNCHRONOUS_IO) {
@@ -615,7 +661,7 @@ static void swap_bdev_submit_read(struct swap_io_ctx *ctx)
bio_init(bio, ctx->sis->bdev, sio->bvecs, ARRAY_SIZE(sio->bvecs),
REQ_OP_READ);
bio->bi_iter.bi_size = sio->len;
- bio->bi_iter.bi_sector = swap_folio_sector(bio_first_folio_all(bio));
+ bio->bi_iter.bi_sector = swap_entry_sector(sio->entry);
if (ctx->sis->flags & SWP_SYNCHRONOUS_IO) {
/*
@@ -633,13 +679,14 @@ static void swap_bdev_submit_read(struct swap_io_ctx *ctx)
}
}
-static bool swap_bdev_can_merge(struct folio *folio, struct folio *prev_folio,
- size_t prev_folio_size, int rw)
+static bool swap_bdev_can_merge(struct folio *folio, swp_entry_t phys,
+ struct swap_iocb *sio, int rw)
{
- if (swap_folio_sector(folio) !=
- swap_folio_sector(prev_folio) + (prev_folio_size >> SECTOR_SHIFT))
+ if (swap_entry_sector(phys) !=
+ swap_entry_sector(sio->entry) + (sio->len >> SECTOR_SHIFT))
return false;
- if (rw == WRITE && !folio_blkg_can_merge(folio, prev_folio))
+ if (rw == WRITE && !folio_blkg_can_merge(folio,
+ bvec_folio(&sio->bvecs[sio->nr_bvecs - 1])))
return false;
return true;
}
@@ -655,7 +702,7 @@ void swap_fs_prepare_rw(struct swap_io_ctx *ctx, int rw, struct iov_iter *iter)
struct swap_iocb *sio = ctx->sio;
init_sync_kiocb(&sio->iocb, ctx->sis->swap_file);
- sio->iocb.ki_pos = swap_dev_pos(bvec_folio(&sio->bvecs[0])->swap);
+ sio->iocb.ki_pos = swap_dev_pos(sio->entry);
if (rw == WRITE)
sio->iocb.ki_complete = swap_fs_write_complete;
else
@@ -666,11 +713,10 @@ void swap_fs_prepare_rw(struct swap_io_ctx *ctx, int rw, struct iov_iter *iter)
}
EXPORT_SYMBOL_GPL(swap_fs_prepare_rw);
-bool swap_fs_can_merge(struct folio *folio, struct folio *prev_folio,
- size_t prev_folio_size, int rw)
+bool swap_fs_can_merge(struct folio *folio, swp_entry_t phys,
+ struct swap_iocb *sio, int rw)
{
- return swap_dev_pos(folio->swap) ==
- swap_dev_pos(prev_folio->swap) + prev_folio_size;
+ return swap_dev_pos(phys) == swap_dev_pos(sio->entry) + sio->len;
}
EXPORT_SYMBOL_GPL(swap_fs_can_merge);
diff --git a/mm/shmem.c b/mm/shmem.c
index 599665a3d6e7b..d58f30b06255b 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -86,6 +86,7 @@ static struct vfsmount *shm_mnt __ro_after_init;
#include <linux/uaccess.h>
#include "internal.h"
+#include "vswap.h"
#define VM_ACCT(size) (PAGE_ALIGN(size) >> PAGE_SHIFT)
@@ -1618,7 +1619,8 @@ int shmem_writeout(struct swap_io_ctx *ctx, struct folio *folio,
if ((info->flags & SHMEM_F_LOCKED) || sbinfo->noswap)
goto redirty;
- if (!total_swap_pages)
+ /* vswap doesn't contribute to total_swap_pages */
+ if (!total_swap_pages && !(vswap_is_enabled() && zswap_is_enabled()))
goto redirty;
/*
diff --git a/mm/swap.h b/mm/swap.h
index 90a551a88df63..f2b64920ae5b3 100644
--- a/mm/swap.h
+++ b/mm/swap.h
@@ -67,6 +67,13 @@ struct swap_cluster_info {
struct list_head list;
};
+struct swap_cluster_info_dynamic {
+ struct swap_cluster_info ci;
+ unsigned int index; /* for cluster_index() */
+ struct rcu_head rcu;
+ atomic_long_t *virtual_table; /* Backing pointers for vswap slots */
+};
+
/* All on-list cluster must have a non-zero flag. */
enum swap_cluster_flags {
CLUSTER_FLAG_NONE = 0, /* For temporary off-list cluster */
@@ -77,6 +84,7 @@ enum swap_cluster_flags {
CLUSTER_FLAG_USABLE = CLUSTER_FLAG_FRAG,
CLUSTER_FLAG_FULL,
CLUSTER_FLAG_DISCARD,
+ CLUSTER_FLAG_DEAD, /* Vswap dynamic cluster pending kfree_rcu */
CLUSTER_FLAG_MAX,
};
@@ -119,12 +127,33 @@ static inline struct swap_info_struct *__swap_entry_to_info(swp_entry_t entry)
return __swap_type_to_info(swp_type(entry));
}
+/**
+ * __swap_offset_to_cluster - look up the cluster holding a swap offset
+ * @si: the swap device
+ * @offset: the swap entry offset
+ *
+ * Context: A vswap cluster is freed by kfree_rcu(). Callers must hold the
+ * RCU read lock, or know the cluster is pinned by an in-use entry.
+ *
+ * Return: the cluster, or NULL if @si is a vswap device with no cluster
+ * allocated at @offset.
+ */
static inline struct swap_cluster_info *__swap_offset_to_cluster(
struct swap_info_struct *si, pgoff_t offset)
{
+ unsigned int cluster_idx = offset / SWAPFILE_CLUSTER;
+
VM_WARN_ON_ONCE(percpu_ref_is_zero(&si->users)); /* race with swapoff */
VM_WARN_ON_ONCE(offset >= roundup(si->max, SWAPFILE_CLUSTER));
- return &si->cluster_info[offset / SWAPFILE_CLUSTER];
+
+ if (swap_is_vswap(si)) {
+ struct swap_cluster_info_dynamic *ci_dyn;
+
+ ci_dyn = xa_load(&si->cluster_info_pool, cluster_idx);
+ return ci_dyn ? &ci_dyn->ci : NULL;
+ }
+
+ return &si->cluster_info[cluster_idx];
}
static inline struct swap_cluster_info *__swap_entry_to_cluster(swp_entry_t entry)
@@ -133,10 +162,36 @@ static inline struct swap_cluster_info *__swap_entry_to_cluster(swp_entry_t entr
swp_offset(entry));
}
+static inline struct swap_cluster_info *__vswap_cluster_lock(
+ struct swap_info_struct *si, unsigned long offset, bool irq)
+{
+ struct swap_cluster_info *ci;
+
+ rcu_read_lock();
+ ci = __swap_offset_to_cluster(si, offset);
+ if (ci) {
+ if (irq)
+ spin_lock_irq(&ci->lock);
+ else
+ spin_lock(&ci->lock);
+
+ /* The cluster can be torn down while we wait for the lock. */
+ if (ci->flags == CLUSTER_FLAG_DEAD) {
+ if (irq)
+ spin_unlock_irq(&ci->lock);
+ else
+ spin_unlock(&ci->lock);
+ ci = NULL;
+ }
+ }
+ rcu_read_unlock();
+ return ci;
+}
+
static __always_inline struct swap_cluster_info *__swap_cluster_lock(
struct swap_info_struct *si, unsigned long offset, bool irq)
{
- struct swap_cluster_info *ci = __swap_offset_to_cluster(si, offset);
+ struct swap_cluster_info *ci;
/*
* Nothing modifies swap cache in an IRQ context. All access to
@@ -149,6 +204,11 @@ static __always_inline struct swap_cluster_info *__swap_cluster_lock(
*/
VM_WARN_ON_ONCE(!in_task());
VM_WARN_ON_ONCE(percpu_ref_is_zero(&si->users)); /* race with swapoff */
+
+ if (swap_is_vswap(si))
+ return __vswap_cluster_lock(si, offset, irq);
+
+ ci = __swap_offset_to_cluster(si, offset);
if (irq)
spin_lock_irq(&ci->lock);
else
@@ -159,10 +219,12 @@ static __always_inline struct swap_cluster_info *__swap_cluster_lock(
/**
* swap_cluster_lock - Lock and return the swap cluster of given offset.
* @si: swap device the cluster belongs to.
- * @offset: the swap entry offset, pointing to a valid slot.
+ * @offset: the swap entry offset.
*
* Context: The caller must ensure the offset is in the valid range and
* protect the swap device with reference count or locks.
+ * Return: the locked cluster, or NULL if it is gone. Only a vswap device
+ * can return NULL, as its clusters are allocated and freed on demand.
*/
static inline struct swap_cluster_info *swap_cluster_lock(
struct swap_info_struct *si, unsigned long offset)
@@ -258,7 +320,8 @@ void swap_read_folio(struct swap_io_ctx *ctx, struct folio *folio);
void swap_read_submit(struct swap_io_ctx *ctx);
void swap_write_submit(struct swap_io_ctx *ctx);
int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio);
-void __swap_writepage(struct swap_io_ctx *ctx, struct folio *folio);
+void __swap_writepage(struct swap_io_ctx *ctx, struct folio *folio,
+ swp_entry_t phys);
/* linux/mm/swap_state.c */
extern struct address_space swap_space __read_mostly;
diff --git a/mm/swap_state.c b/mm/swap_state.c
index b76eb3d876fd7..5cfddec8633b9 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -27,6 +27,7 @@
#include "internal.h"
#include "swap_table.h"
#include "swap.h"
+#include "vswap.h"
/* Swap readahead cluster size, as a power of 2 pages. */
static int page_cluster;
@@ -96,8 +97,10 @@ struct folio *swap_cache_get_folio(swp_entry_t entry)
struct folio *folio;
for (;;) {
+ rcu_read_lock();
swp_tb = swap_table_get(__swap_entry_to_cluster(entry),
swp_cluster_offset(entry));
+ rcu_read_unlock();
if (!swp_tb_is_folio(swp_tb))
return NULL;
folio = swp_tb_to_folio(swp_tb);
@@ -119,8 +122,10 @@ bool swap_cache_has_folio(swp_entry_t entry)
{
unsigned long swp_tb;
+ rcu_read_lock();
swp_tb = swap_table_get(__swap_entry_to_cluster(entry),
swp_cluster_offset(entry));
+ rcu_read_unlock();
return swp_tb_is_folio(swp_tb);
}
@@ -136,8 +141,10 @@ void *swap_cache_get_shadow(swp_entry_t entry)
{
unsigned long swp_tb;
+ rcu_read_lock();
swp_tb = swap_table_get(__swap_entry_to_cluster(entry),
swp_cluster_offset(entry));
+ rcu_read_unlock();
if (swp_tb_is_shadow(swp_tb))
return swp_tb_to_shadow(swp_tb);
return NULL;
@@ -167,6 +174,9 @@ static int __swap_cache_add_check(struct swap_cluster_info *ci,
unsigned int ci_off, ci_end;
unsigned long old_tb;
bool is_zero;
+ struct swap_cluster_info_dynamic *ci_dyn;
+ enum vswap_backing_type type;
+ int ret;
lockdep_assert_held(&ci->lock);
@@ -179,6 +189,9 @@ static int __swap_cache_add_check(struct swap_cluster_info *ci,
return -ENOENT;
ci_off = swp_cluster_offset(targ_entry);
old_tb = __swap_table_get(ci, ci_off);
+ /* Physical readahead can hit a vswap-backing rmap slot; skip it. */
+ if (swp_tb_is_pointer(old_tb))
+ return -ENOENT;
if (swp_tb_is_folio(old_tb))
return -EEXIST;
if (!__swp_tb_get_count(old_tb))
@@ -191,12 +204,26 @@ static int __swap_cache_add_check(struct swap_cluster_info *ci,
if (nr == 1)
return 0;
+ /*
+ * For a vswap entry batch, reject if the backing is not THP-amenable
+ * (e.g. uniformly ZSWAP, or mixed). The order-fallback loop in
+ * swap_cache_alloc_folio will retry with a smaller order on -EBUSY.
+ */
+ if (is_vswap_entry(targ_entry)) {
+ ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
+ ret = __vswap_check_backing(ci_dyn, round_down(ci_off, nr),
+ nr, &type);
+ if (ret != nr || type == VSWAP_ZSWAP)
+ return -EBUSY;
+ }
+
is_zero = __swap_table_test_zero(ci, ci_off);
ci_off = round_down(ci_off, nr);
ci_end = ci_off + nr;
do {
old_tb = __swap_table_get(ci, ci_off);
- if (unlikely(swp_tb_is_folio(old_tb) ||
+ if (unlikely(swp_tb_is_pointer(old_tb) ||
+ swp_tb_is_folio(old_tb) ||
!__swp_tb_get_count(old_tb) ||
is_zero != __swap_table_test_zero(ci, ci_off) ||
(memcg_id && *memcg_id != __swap_cgroup_get(ci, ci_off))))
@@ -406,14 +433,16 @@ void __swap_cache_replace_folio(struct swap_cluster_info *ci,
* -ENOENT / -EEXIST: Target swap entry is unavailable or cached, the caller
* should abort or try to use the cached folio instead
*/
-static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,
- swp_entry_t targ_entry, gfp_t gfp,
+static struct folio *__swap_cache_alloc(swp_entry_t targ_entry, gfp_t gfp,
unsigned int order, struct vm_fault *vmf,
struct mempolicy *mpol, pgoff_t ilx)
{
int err;
swp_entry_t entry;
struct folio *folio;
+ struct swap_cluster_info *ci;
+ struct swap_info_struct *si = __swap_entry_to_info(targ_entry);
+ unsigned long offset = swp_offset(targ_entry);
void *shadow = NULL;
unsigned short memcg_id;
unsigned long address, nr_pages = 1UL << order;
@@ -423,9 +452,12 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,
entry.val = round_down(targ_entry.val, nr_pages);
/* Check if the slot and range are available, skip allocation if not */
- spin_lock(&ci->lock);
- err = __swap_cache_add_check(ci, targ_entry, nr_pages, NULL, NULL);
- spin_unlock(&ci->lock);
+ err = -ENOENT;
+ ci = swap_cluster_lock(si, offset);
+ if (ci) {
+ err = __swap_cache_add_check(ci, targ_entry, nr_pages, NULL, NULL);
+ swap_cluster_unlock(ci);
+ }
if (unlikely(err))
return ERR_PTR(err);
@@ -446,10 +478,13 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,
return ERR_PTR(-ENOMEM);
/* Double check the range is still not in conflict */
- spin_lock(&ci->lock);
- err = __swap_cache_add_check(ci, targ_entry, nr_pages, &shadow, &memcg_id);
+ err = -ENOENT;
+ ci = swap_cluster_lock(si, offset);
+ if (ci)
+ err = __swap_cache_add_check(ci, targ_entry, nr_pages, &shadow, &memcg_id);
if (unlikely(err)) {
- spin_unlock(&ci->lock);
+ if (ci)
+ swap_cluster_unlock(ci);
folio_put(folio);
return ERR_PTR(err);
}
@@ -457,10 +492,11 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,
__folio_set_locked(folio);
__folio_set_swapbacked(folio);
__swap_cache_do_add_folio(ci, folio, entry);
- spin_unlock(&ci->lock);
+ swap_cluster_unlock(ci);
if (mem_cgroup_swapin_charge_folio(folio, memcg_id,
vmf ? vmf->vma->vm_mm : NULL, gfp)) {
+ /* The folio pins the cluster */
spin_lock(&ci->lock);
__swap_cache_do_del_folio(ci, folio, entry, shadow);
spin_unlock(&ci->lock);
@@ -517,9 +553,7 @@ struct folio *swap_cache_alloc_folio(swp_entry_t targ_entry, gfp_t gfp,
{
int order, err;
struct folio *ret;
- struct swap_cluster_info *ci;
- ci = __swap_entry_to_cluster(targ_entry);
order = highest_order(orders);
/* orders must be non-zero, and must not exceed cluster size. */
@@ -527,7 +561,7 @@ struct folio *swap_cache_alloc_folio(swp_entry_t targ_entry, gfp_t gfp,
return ERR_PTR(-EINVAL);
do {
- ret = __swap_cache_alloc(ci, targ_entry, gfp, order,
+ ret = __swap_cache_alloc(targ_entry, gfp, order,
vmf, mpol, ilx);
if (!IS_ERR(ret))
break;
diff --git a/mm/swap_table.h b/mm/swap_table.h
index e6613e62f8d0f..b614b1989fd9d 100644
--- a/mm/swap_table.h
+++ b/mm/swap_table.h
@@ -4,8 +4,11 @@
#include <linux/rcupdate.h>
#include <linux/atomic.h>
+#include <linux/swapops.h>
#include "swap.h"
+extern struct swap_info_struct *vswap_si;
+
/* A typical flat array in each cluster as swap table */
struct swap_table {
atomic_long_t entries[SWAPFILE_CLUSTER];
@@ -28,7 +31,7 @@ struct swap_memcg_table {
* NULL: |---------------- 0 ---------------| - Free slot
* Shadow: |SWAP_COUNT|Z|---- SHADOW_VAL ---|1| - Swapped out slot
* PFN: |SWAP_COUNT|Z|------ PFN -------|10| - Cached slot
- * Pointer: |----------- Pointer ----------|100| - (Unused)
+ * Pointer: |C|------- vswap offset -------|100| - vswap rmap
* Bad: |------------- 1 -------------|1000| - Bad slot
*
* COUNT is `SWP_TB_COUNT_BITS` long, Z is the `SWP_TB_ZERO_FLAG` bit,
@@ -49,9 +52,8 @@ struct swap_memcg_table {
* - PFN: Swap slot is in use, and cached. Memcg info is recorded on the page
* struct.
*
- * - Pointer: Unused yet. `0b100` is reserved for potential pointer usage
- * because only the lower three bits can be used as a marker for 8 bytes
- * aligned pointers.
+ * - Pointer: Reverse map from a physical slot to the vswap entry that owns
+ * it. See the layout below.
*
* - Bad: Swap slot is reserved, protects swap header or holes on swap devices.
*/
@@ -255,6 +257,8 @@ static inline unsigned long swap_table_get(struct swap_cluster_info *ci,
unsigned long swp_tb;
VM_WARN_ON_ONCE(off >= SWAPFILE_CLUSTER);
+ if (!ci)
+ return SWP_TB_NULL;
rcu_read_lock();
table = rcu_dereference(ci->table);
@@ -366,4 +370,42 @@ static inline unsigned short __swap_cgroup_clear(struct swap_cluster_info *ci,
}
#endif
+/*
+ * Pointer-tagged swap table entry: rmap for vswap-backing physical slots.
+ *
+ * On physical clusters, a Pointer-tagged entry stores the offset of the
+ * vswap entry that owns this physical slot (the reverse map). Only the
+ * offset is stored; the swap type is implicit (always vswap_si->type,
+ * since there is exactly one vswap device). The top bit is reserved as
+ * a cache-only flag, set when vswap swap_count drops to 0 but the folio
+ * is still in swap cache.
+ *
+ * Pointer: |C|---- vswap offset ----|100|
+ * C = SWP_RMAP_CACHE_ONLY (bit 63)
+ */
+#define SWP_TB_PTR_MARK_BITS 3
+#define SWP_TB_PTR_MARK 0b100UL
+#define SWP_TB_PTR_MARK_MASK ((1UL << SWP_TB_PTR_MARK_BITS) - 1)
+#define SWP_RMAP_CACHE_ONLY (1UL << (BITS_PER_LONG - 1))
+#define SWP_RMAP_ENTRY_MASK (~(SWP_RMAP_CACHE_ONLY | SWP_TB_PTR_MARK_MASK))
+
+static inline bool swp_tb_is_pointer(unsigned long swp_tb)
+{
+ return (swp_tb & SWP_TB_PTR_MARK_MASK) == SWP_TB_PTR_MARK;
+}
+
+static inline unsigned long swp_entry_to_swp_tb_ptr(swp_entry_t entry)
+{
+ return (swp_offset(entry) << SWP_TB_PTR_MARK_BITS) | SWP_TB_PTR_MARK;
+}
+
+static inline swp_entry_t swp_tb_ptr_to_swp_entry(unsigned long swp_tb)
+{
+ unsigned long offset;
+
+ VM_WARN_ON(!swp_tb_is_pointer(swp_tb));
+ offset = (swp_tb & SWP_RMAP_ENTRY_MASK) >> SWP_TB_PTR_MARK_BITS;
+ return swp_entry(vswap_si->type, offset);
+}
+
#endif
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 53bf01d5f7f11..9860fb3b079fb 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -7,6 +7,7 @@
*/
#include <linux/blkdev.h>
+#include <linux/debugfs.h>
#include <linux/mm.h>
#include <linux/sched/mm.h>
#include <linux/sched/task.h>
@@ -36,6 +37,7 @@
#include <linux/poll.h>
#include <linux/oom.h>
#include <linux/swapfile.h>
+#include <linux/swap_ops.h>
#include <linux/export.h>
#include <linux/sort.h>
#include <linux/completion.h>
@@ -45,7 +47,9 @@
#include <asm/tlbflush.h>
#include <linux/leafops.h>
+#include "memcontrol-v1.h"
#include "swap_table.h"
+#include "vswap.h"
#include "internal.h"
#include "swap.h"
@@ -129,6 +133,24 @@ static DEFINE_PER_CPU(struct percpu_swap_cluster, percpu_swap_cluster) = {
.lock = INIT_LOCAL_LOCK(),
};
+struct percpu_vswap_cluster {
+ unsigned long offset[SWAP_NR_ORDERS];
+ local_lock_t lock;
+};
+
+static DEFINE_PER_CPU(struct percpu_vswap_cluster, percpu_vswap_cluster) = {
+ .offset = { [0 ... SWAP_NR_ORDERS - 1] = SWAP_ENTRY_INVALID },
+ .lock = INIT_LOCAL_LOCK(),
+};
+
+static atomic_long_t vswap_alloc_reject = ATOMIC_LONG_INIT(0);
+
+static bool vswap_alloc(struct folio *folio);
+static void vswap_mark_cache_only(struct swap_cluster_info *ci,
+ unsigned int ci_off);
+static void vswap_clear_cache_only(struct swap_cluster_info *ci,
+ unsigned int ci_start, int nr);
+
/* May return NULL on invalid type, caller must check for NULL return */
static struct swap_info_struct *swap_type_to_info(int type)
{
@@ -234,7 +256,8 @@ static int __try_to_reclaim_swap(struct swap_info_struct *si,
need_reclaim = ((flags & TTRS_ANYWAY) ||
((flags & TTRS_UNMAPPED) && !folio_mapped(folio)) ||
- ((flags & TTRS_FULL) && mem_cgroup_swap_full(folio)));
+ ((flags & TTRS_FULL) && mem_cgroup_swap_full(folio) &&
+ folio_phys_swap_backed(folio)));
if (!need_reclaim || !folio_swapcache_freeable(folio))
goto out_unlock;
@@ -328,14 +351,14 @@ offset_to_swap_extent(struct swap_info_struct *sis, unsigned long offset)
BUG();
}
-sector_t swap_folio_sector(struct folio *folio)
+sector_t swap_entry_sector(swp_entry_t entry)
{
- struct swap_info_struct *sis = __swap_entry_to_info(folio->swap);
+ struct swap_info_struct *sis = __swap_entry_to_info(entry);
struct swap_extent *se;
sector_t sector;
pgoff_t offset;
- offset = swp_offset(folio->swap);
+ offset = swp_offset(entry);
se = offset_to_swap_extent(sis, offset);
sector = se->start_block + (offset - se->start_page);
return sector << (PAGE_SHIFT - 9);
@@ -401,13 +424,15 @@ static inline bool cluster_is_usable(struct swap_cluster_info *ci, int order)
static inline unsigned int cluster_index(struct swap_info_struct *si,
struct swap_cluster_info *ci)
{
+ if (swap_is_vswap(si))
+ return container_of(ci, struct swap_cluster_info_dynamic, ci)->index;
return ci - si->cluster_info;
}
-static inline unsigned int cluster_offset(struct swap_info_struct *si,
- struct swap_cluster_info *ci)
+static inline unsigned long cluster_offset(struct swap_info_struct *si,
+ struct swap_cluster_info *ci)
{
- return cluster_index(si, ci) * SWAPFILE_CLUSTER;
+ return (unsigned long)cluster_index(si, ci) * SWAPFILE_CLUSTER;
}
static void swap_cluster_free_table_folio_rcu_cb(struct rcu_head *head)
@@ -446,7 +471,8 @@ static void swap_cluster_free_table(struct swap_cluster_info *ci)
swap_cluster_free_table_folio_rcu_cb);
}
-static int swap_cluster_alloc_table(struct swap_cluster_info *ci, gfp_t gfp)
+static int swap_cluster_alloc_table(struct swap_info_struct *si,
+ struct swap_cluster_info *ci, gfp_t gfp)
{
struct swap_table *table = NULL;
struct folio *folio;
@@ -469,7 +495,14 @@ static int swap_cluster_alloc_table(struct swap_cluster_info *ci, gfp_t gfp)
rcu_assign_pointer(ci->table, table);
#ifdef CONFIG_MEMCG
- if (!mem_cgroup_disabled()) {
+ /*
+ * A physical cluster under vswap may hold only vswap backings, which
+ * record their memcg on the vswap cluster's table, not this one. Such
+ * clusters defer memcg_table allocation until they hand out a slot
+ * that maps directly into the PTEs.
+ */
+ if ((!vswap_is_enabled() || swap_is_vswap(si)) &&
+ !mem_cgroup_disabled()) {
VM_WARN_ON_ONCE(ci->memcg_table);
ci->memcg_table = kzalloc_obj(*ci->memcg_table, gfp);
if (!ci->memcg_table) {
@@ -532,14 +565,16 @@ swap_cluster_populate(struct swap_info_struct *si,
/*
* Only cluster isolation from the allocator does table allocation.
* Swap allocator uses percpu clusters and holds the local lock.
+ * vswap clusters are destroyed rather than freed to si->free_clusters.
*/
+ VM_WARN_ON_ONCE(swap_is_vswap(si));
lockdep_assert_held(&this_cpu_ptr(&percpu_swap_cluster)->lock);
if (!(si->flags & SWP_SOLIDSTATE))
lockdep_assert_held(&si->global_cluster_lock);
lockdep_assert_held(&ci->lock);
- if (!swap_cluster_alloc_table(ci, __GFP_HIGH | __GFP_NOMEMALLOC |
- __GFP_NOWARN))
+ if (!swap_cluster_alloc_table(si, ci, __GFP_HIGH | __GFP_NOMEMALLOC |
+ __GFP_NOWARN))
return ci;
/*
@@ -552,8 +587,8 @@ swap_cluster_populate(struct swap_info_struct *si,
spin_unlock(&si->global_cluster_lock);
local_unlock(&percpu_swap_cluster.lock);
- ret = swap_cluster_alloc_table(ci, __GFP_HIGH | __GFP_NOMEMALLOC |
- GFP_KERNEL);
+ ret = swap_cluster_alloc_table(si, ci, __GFP_HIGH | __GFP_NOMEMALLOC |
+ GFP_KERNEL);
/*
* Back to atomic context. We might have migrated to a new CPU with a
@@ -586,10 +621,15 @@ static void move_cluster(struct swap_info_struct *si,
lockdep_assert_held(&ci->lock);
spin_lock(&si->lock);
- if (ci->flags == CLUSTER_FLAG_NONE)
+ if (!list) {
+ /* Going away. An isolated cluster is already off its list. */
+ if (ci->flags != CLUSTER_FLAG_NONE)
+ list_del(&ci->list);
+ } else if (ci->flags == CLUSTER_FLAG_NONE) {
list_add_tail(&ci->list, list);
- else
+ } else {
list_move_tail(&ci->list, list);
+ }
spin_unlock(&si->lock);
ci->flags = new_flags;
}
@@ -607,6 +647,19 @@ static void __free_cluster(struct swap_info_struct *si, struct swap_cluster_info
{
swap_cluster_assert_empty(ci, 0, SWAPFILE_CLUSTER, false);
swap_cluster_free_table(ci);
+
+ if (swap_is_vswap(si)) {
+ struct swap_cluster_info_dynamic *ci_dyn;
+
+ /* vswap clusters are destroyed, not returned to free_clusters. */
+ ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
+ xa_erase(&si->cluster_info_pool, ci_dyn->index);
+ move_cluster(si, ci, NULL, CLUSTER_FLAG_DEAD);
+ vswap_cluster_free_vtable(ci);
+ kfree_rcu(ci_dyn, rcu);
+ return;
+ }
+
move_cluster(si, ci, &si->free_clusters, CLUSTER_FLAG_FREE);
ci->order = 0;
}
@@ -795,7 +848,7 @@ static int swap_cluster_setup_bad_slot(struct swap_info_struct *si,
/* si->max may got shrunk by swap swap_activate() */
if (offset >= si->max && !mask) {
- pr_debug("Ignoring bad slot %u (max: %u)\n", offset, si->max);
+ pr_debug("Ignoring bad slot %u (max: %lu)\n", offset, si->max);
return 0;
}
/*
@@ -812,7 +865,7 @@ static int swap_cluster_setup_bad_slot(struct swap_info_struct *si,
ci = cluster_info + idx;
/* Need to allocate swap table first for initial bad slot marking. */
- if (!ci->count && swap_cluster_alloc_table(ci, GFP_KERNEL))
+ if (!ci->count && swap_cluster_alloc_table(si, ci, GFP_KERNEL))
return -ENOMEM;
spin_lock(&ci->lock);
/* Check for duplicated bad swap slots. */
@@ -830,6 +883,54 @@ static int swap_cluster_setup_bad_slot(struct swap_info_struct *si,
return ret;
}
+/*
+ * Try to reclaim a Pointer-tagged physical slot backing a vswap entry.
+ * The physical cluster lock must NOT be held. Returns the backing folio's
+ * page count, negated if the slots could not be reclaimed, or 0 if the
+ * folio could not be shown to own @offset (i.e. there is a race).
+ */
+static int try_to_reclaim_vswap_backing(struct swap_info_struct *si,
+ unsigned long offset,
+ swp_entry_t vswap_entry)
+{
+ swp_entry_t phys_base;
+ struct folio *folio;
+ unsigned int i;
+ int ret;
+
+ folio = swap_cache_get_folio(vswap_entry);
+ if (!folio)
+ return 0;
+
+ if (!folio_trylock(folio)) {
+ folio_put(folio);
+ return 0;
+ }
+
+ if (!folio_matches_swap_entry(folio, vswap_entry)) {
+ folio_unlock(folio);
+ folio_put(folio);
+ return 0;
+ }
+
+ i = vswap_entry.val - folio->swap.val;
+ phys_base = vswap_to_phys(folio->swap);
+ if (!phys_base.val || swp_type(phys_base) != si->type ||
+ swp_offset(phys_base) + i != offset) {
+ folio_unlock(folio);
+ folio_put(folio);
+ return 0;
+ }
+
+ /* The run is ours: skip it all, whether or not the free succeeds. */
+ ret = folio_nr_pages(folio);
+ if (!folio_free_swap(folio))
+ ret = -ret;
+ folio_unlock(folio);
+ folio_put(folio);
+ return ret;
+}
+
/*
* Reclaim drops the ci lock, so the cluster may become unusable (freed or
* stolen by a lower order). @usable will be set to false if that happens.
@@ -843,6 +944,8 @@ static bool cluster_reclaim_range(struct swap_info_struct *si,
unsigned long offset = start, end = start + nr_pages;
unsigned long swp_tb;
+ VM_WARN_ON_ONCE(swap_is_vswap(si));
+
spin_unlock(&ci->lock);
do {
swp_tb = swap_table_get(ci, offset % SWAPFILE_CLUSTER);
@@ -895,7 +998,8 @@ static bool cluster_scan_range(struct swap_info_struct *si,
if (swp_tb_is_null(swp_tb))
continue;
if (swp_tb_is_folio(swp_tb) && !__swp_tb_get_count(swp_tb)) {
- if (!vm_swap_full())
+ /* vswap slots are abundant; never reclaim to reuse one */
+ if (swap_is_vswap(si) || !vm_swap_full())
return false;
*need_reclaim = true;
continue;
@@ -915,6 +1019,8 @@ static bool __swap_cluster_alloc_entries(struct swap_info_struct *si,
{
unsigned int order;
unsigned long nr_pages;
+ swp_entry_t vswap_entry, v;
+ unsigned int i;
lockdep_assert_held(&ci->lock);
@@ -934,8 +1040,26 @@ static bool __swap_cluster_alloc_entries(struct swap_info_struct *si,
order = folio_order(folio);
nr_pages = 1 << order;
swap_cluster_assert_empty(ci, ci_off, nr_pages, false);
- __swap_cache_add_folio(ci, folio, swp_entry(si->type,
- ci_off + cluster_offset(si, ci)));
+ if (folio_test_swapcache(folio)) {
+ /*
+ * Folio already in the swap cache: we are allocating
+ * physical backing for its vswap entry. Point each
+ * physical slot back at its own vswap entry
+ * (Pointer-tagged rmap).
+ */
+ VM_WARN_ON(!is_vswap_entry(folio->swap));
+ vswap_entry = folio->swap;
+ for (i = 0; i < nr_pages; i++) {
+ v = vswap_entry;
+ v.val += i;
+ __swap_table_set(ci, ci_off + i,
+ swp_entry_to_swp_tb_ptr(v));
+ }
+ } else {
+ __swap_cache_add_folio(ci, folio,
+ swp_entry(si->type,
+ ci_off + cluster_offset(si, ci)));
+ }
} else if (IS_ENABLED(CONFIG_HIBERNATION)) {
order = 0;
nr_pages = 1;
@@ -961,11 +1085,13 @@ static bool __swap_cluster_alloc_entries(struct swap_info_struct *si,
}
/* Try use a new cluster for current CPU and allocate from it. */
-static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si,
- struct swap_cluster_info *ci,
- struct folio *folio, unsigned long offset)
+static unsigned long alloc_swap_scan_cluster(struct swap_info_struct *si,
+ struct swap_cluster_info *ci,
+ struct folio *folio,
+ unsigned long offset,
+ bool *nomem)
{
- unsigned int next = SWAP_ENTRY_INVALID, found = SWAP_ENTRY_INVALID;
+ unsigned long next = SWAP_ENTRY_INVALID, found = SWAP_ENTRY_INVALID;
unsigned long start = ALIGN_DOWN(offset, SWAPFILE_CLUSTER);
unsigned int order = likely(folio) ? folio_order(folio) : 0;
unsigned long end = start + SWAPFILE_CLUSTER;
@@ -992,6 +1118,24 @@ static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si,
if (!ret)
continue;
}
+#ifdef CONFIG_MEMCG
+ /*
+ * Lazy-allocate memcg_table on the first direct-use slot of a
+ * physical cluster.
+ */
+ if (vswap_is_enabled() && folio &&
+ !folio_test_swapcache(folio) && !mem_cgroup_disabled() &&
+ !ci->memcg_table) {
+ ci->memcg_table = kzalloc_obj(*ci->memcg_table,
+ GFP_ATOMIC | __GFP_NOMEMALLOC |
+ __GFP_NOWARN);
+ if (!ci->memcg_table) {
+ if (nomem)
+ *nomem = true;
+ goto out;
+ }
+ }
+#endif
if (!__swap_cluster_alloc_entries(si, ci, folio, offset % SWAPFILE_CLUSTER))
break;
found = offset;
@@ -1001,8 +1145,20 @@ static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si,
break;
}
out:
- relocate_cluster(si, ci);
+ /*
+ * On a discard-capable device, relocating a cluster whose memcg_table
+ * allocation failed queues a discard for slots that were never used,
+ * which folio_alloc_phys_swap() reads as progress and retries on.
+ */
+ if (nomem && *nomem && !ci->count)
+ __free_cluster(si, ci);
+ else
+ relocate_cluster(si, ci);
swap_cluster_unlock(ci);
+ if (swap_is_vswap(si)) {
+ this_cpu_write(percpu_vswap_cluster.offset[order], next);
+ return found;
+ }
if (si->flags & SWP_SOLIDSTATE) {
this_cpu_write(percpu_swap_cluster.offset[order], next);
this_cpu_write(percpu_swap_cluster.si[order], si);
@@ -1012,13 +1168,19 @@ static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si,
return found;
}
-static unsigned int alloc_swap_scan_list(struct swap_info_struct *si,
- struct list_head *list,
- struct folio *folio,
- bool scan_all)
+static unsigned long alloc_swap_scan_list(struct swap_info_struct *si,
+ struct list_head *list,
+ struct folio *folio,
+ bool scan_all)
{
- unsigned int found = SWAP_ENTRY_INVALID;
+ unsigned long found = SWAP_ENTRY_INVALID;
+ bool nomem = false;
+ /*
+ * In rare cases alloc_swap_scan_cluster() can fail due to
+ * memcg_table allocation failure. Short-circuit to avoid looping
+ * over the list indefinitely.
+ */
do {
struct swap_cluster_info *ci = isolate_lock_cluster(si, list);
unsigned long offset;
@@ -1026,19 +1188,65 @@ static unsigned int alloc_swap_scan_list(struct swap_info_struct *si,
if (!ci)
break;
offset = cluster_offset(si, ci);
- found = alloc_swap_scan_cluster(si, ci, folio, offset);
+ found = alloc_swap_scan_cluster(si, ci, folio, offset, &nomem);
if (found)
break;
- } while (scan_all);
+ } while (scan_all && !nomem);
return found;
}
+static unsigned long vswap_alloc_cluster(struct swap_info_struct *si,
+ struct folio *folio)
+{
+ struct swap_cluster_info_dynamic *ci_dyn;
+ struct swap_cluster_info *ci;
+ unsigned long offset;
+
+ VM_WARN_ON(!swap_is_vswap(si));
+
+ ci_dyn = kzalloc_obj(*ci_dyn, GFP_ATOMIC);
+ if (!ci_dyn)
+ return SWAP_ENTRY_INVALID;
+
+ spin_lock_init(&ci_dyn->ci.lock);
+ INIT_LIST_HEAD(&ci_dyn->ci.list);
+
+ if (swap_cluster_alloc_table(si, &ci_dyn->ci, GFP_ATOMIC)) {
+ kfree(ci_dyn);
+ return SWAP_ENTRY_INVALID;
+ }
+
+ if (vswap_cluster_alloc_vtable(ci_dyn, GFP_ATOMIC)) {
+ swap_cluster_free_table(&ci_dyn->ci);
+ kfree(ci_dyn);
+ return SWAP_ENTRY_INVALID;
+ }
+
+ /* Lock before publishing: xa_alloc makes the cluster findable by offset. */
+ ci = &ci_dyn->ci;
+ spin_lock(&ci->lock);
+
+ if (xa_alloc(&si->cluster_info_pool, &ci_dyn->index, ci_dyn,
+ XA_LIMIT(1, DIV_ROUND_UP(si->max, SWAPFILE_CLUSTER) - 1),
+ GFP_ATOMIC)) {
+ spin_unlock(&ci->lock);
+ swap_cluster_free_table(&ci_dyn->ci);
+ vswap_cluster_free_vtable(&ci_dyn->ci);
+ kfree(ci_dyn);
+ return SWAP_ENTRY_INVALID;
+ }
+
+ offset = cluster_offset(si, ci);
+ return alloc_swap_scan_cluster(si, ci, folio, offset, NULL);
+}
+
static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool force)
{
long to_scan = 1;
unsigned long offset, end;
struct swap_cluster_info *ci;
+ swp_entry_t vswap_entry;
unsigned long swp_tb;
int nr_reclaim;
@@ -1056,7 +1264,22 @@ static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool force)
spin_unlock(&ci->lock);
nr_reclaim = __try_to_reclaim_swap(si, offset,
TTRS_ANYWAY);
- spin_lock(&ci->lock);
+ ci = swap_cluster_lock(si, offset);
+ if (!ci)
+ goto next;
+ if (nr_reclaim) {
+ offset += abs(nr_reclaim);
+ continue;
+ }
+ } else if (swp_tb_is_pointer(swp_tb) &&
+ (swp_tb & SWP_RMAP_CACHE_ONLY)) {
+ vswap_entry = swp_tb_ptr_to_swp_entry(swp_tb);
+ spin_unlock(&ci->lock);
+ nr_reclaim = try_to_reclaim_vswap_backing(si, offset,
+ vswap_entry);
+ ci = swap_cluster_lock(si, offset);
+ if (!ci)
+ goto next;
if (nr_reclaim) {
offset += abs(nr_reclaim);
continue;
@@ -1070,6 +1293,7 @@ static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool force)
relocate_cluster(si, ci);
swap_cluster_unlock(ci);
+next:
if (to_scan <= 0)
break;
@@ -1100,13 +1324,13 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,
{
struct swap_cluster_info *ci;
unsigned int order = likely(folio) ? folio_order(folio) : 0;
- unsigned int offset = SWAP_ENTRY_INVALID, found = SWAP_ENTRY_INVALID;
+ unsigned long offset = SWAP_ENTRY_INVALID, found = SWAP_ENTRY_INVALID;
/*
* Swapfile is not block device so unable
* to allocate large entries.
*/
- if (order && !(si->flags & SWP_BLKDEV))
+ if (order && !(si->flags & SWP_BLKDEV) && !swap_is_vswap(si))
return 0;
if (!(si->flags & SWP_SOLIDSTATE)) {
@@ -1121,7 +1345,8 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,
if (cluster_is_usable(ci, order)) {
if (cluster_is_empty(ci))
offset = cluster_offset(si, ci);
- found = alloc_swap_scan_cluster(si, ci, folio, offset);
+ found = alloc_swap_scan_cluster(si, ci, folio, offset,
+ NULL);
} else {
swap_cluster_unlock(ci);
}
@@ -1146,6 +1371,12 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,
goto done;
}
+ if (swap_is_vswap(si)) {
+ found = vswap_alloc_cluster(si, folio);
+ if (found)
+ goto done;
+ }
+
if (!(si->flags & SWP_PAGE_DISCARD)) {
found = alloc_swap_scan_list(si, &si->free_clusters, folio, false);
if (found)
@@ -1153,13 +1384,14 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,
}
/* Try reclaim full clusters if free and nonfull lists are drained */
- if (vm_swap_full())
+ if (!swap_is_vswap(si) && vm_swap_full())
swap_reclaim_full_clusters(si, false);
if (order < PMD_ORDER) {
/*
* Scan only one fragment cluster is good enough. Order 0
- * allocation will surely success, and large allocation
+ * allocation will surely success unless the memcg table
+ * allocation fails, which is rare, and large allocation
* failure is not critical. Scanning one cluster still
* keeps the list rotated and reclaimed (for clean swap cache).
*/
@@ -1175,7 +1407,8 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,
for (int o = 1; o < SWAP_NR_ORDERS; o++) {
/*
* Clusters here have at least one usable slots and can't fail order 0
- * allocation, but reclaim may drop si->lock and race with another user.
+ * allocation, but reclaim may drop si->lock and race with another user,
+ * and the memcg table allocation may fail.
*/
found = alloc_swap_scan_list(si, &si->frag_clusters[o], folio, true);
if (found)
@@ -1282,8 +1515,10 @@ static bool swap_usage_add(struct swap_info_struct *si, unsigned int nr_entries)
/*
* If device is full, and SWAP_USAGE_OFFLIST_BIT is not set,
* remove it from the plist.
+ *
+ * Vswap is never on the avail list, so skip it.
*/
- if (unlikely(val == si->pages)) {
+ if (unlikely(val == si->pages) && !swap_is_vswap(si)) {
del_from_avail_list(si, false);
return true;
}
@@ -1298,8 +1533,10 @@ static void swap_usage_sub(struct swap_info_struct *si, unsigned int nr_entries)
/*
* If device is not full, and SWAP_USAGE_OFFLIST_BIT is set,
* add it to the plist.
+ *
+ * Vswap is never on the avail list, so skip it.
*/
- if (unlikely(val & SWAP_USAGE_OFFLIST_BIT))
+ if (unlikely(val & SWAP_USAGE_OFFLIST_BIT) && !swap_is_vswap(si))
add_to_avail_list(si, false);
}
@@ -1310,7 +1547,8 @@ static void swap_range_alloc(struct swap_info_struct *si,
if (vm_swap_full())
schedule_work(&si->reclaim_work);
}
- atomic_long_sub(nr_entries, &nr_swap_pages);
+ if (!swap_is_vswap(si))
+ atomic_long_sub(nr_entries, &nr_swap_pages);
}
static void swap_range_free(struct swap_info_struct *si, unsigned long offset,
@@ -1320,8 +1558,10 @@ static void swap_range_free(struct swap_info_struct *si, unsigned long offset,
void (*swap_slot_free_notify)(struct block_device *, unsigned long);
unsigned int i;
- for (i = 0; i < nr_entries; i++)
- zswap_invalidate(swp_entry(si->type, offset + i));
+ if (!swap_is_vswap(si)) {
+ for (i = 0; i < nr_entries; i++)
+ zswap_invalidate(swp_entry(si->type, offset + i));
+ }
if (si->flags & SWP_BLKDEV)
swap_slot_free_notify =
@@ -1340,12 +1580,17 @@ static void swap_range_free(struct swap_info_struct *si, unsigned long offset,
* only after the above cleanups are done.
*/
smp_wmb();
- atomic_long_add(nr_entries, &nr_swap_pages);
+ if (!swap_is_vswap(si))
+ atomic_long_add(nr_entries, &nr_swap_pages);
swap_usage_sub(si, nr_entries);
}
static bool get_swap_device_info(struct swap_info_struct *si)
{
+ /* The vswap device is always alive, so it needs no refcount. */
+ if (swap_is_vswap(si))
+ return true;
+
if (!percpu_ref_tryget_live(&si->users))
return false;
/*
@@ -1364,12 +1609,14 @@ static bool get_swap_device_info(struct swap_info_struct *si)
* Fast path try to get swap entries with specified order from current
* CPU's swap entry pool (a cluster).
*/
-static bool swap_alloc_fast(struct folio *folio)
+static swp_entry_t swap_alloc_fast(struct folio *folio)
{
unsigned int order = folio_order(folio);
struct swap_cluster_info *ci;
struct swap_info_struct *si;
- unsigned int offset;
+ unsigned long offset, found = 0;
+
+ lockdep_assert_held(&this_cpu_ptr(&percpu_swap_cluster)->lock);
/*
* Once allocated, swap_info_struct will never be completely freed,
@@ -1378,25 +1625,28 @@ static bool swap_alloc_fast(struct folio *folio)
si = this_cpu_read(percpu_swap_cluster.si[order]);
offset = this_cpu_read(percpu_swap_cluster.offset[order]);
if (!si || !offset || !get_swap_device_info(si))
- return false;
+ return (swp_entry_t){};
ci = swap_cluster_lock(si, offset);
- if (cluster_is_usable(ci, order)) {
+ if (ci && cluster_is_usable(ci, order)) {
if (cluster_is_empty(ci))
offset = cluster_offset(si, ci);
- alloc_swap_scan_cluster(si, ci, folio, offset);
- } else {
+ found = alloc_swap_scan_cluster(si, ci, folio, offset, NULL);
+ } else if (ci) {
swap_cluster_unlock(ci);
}
put_swap_device(si);
- return folio_test_swapcache(folio);
+ if (found)
+ return swp_entry(si->type, found);
+ return (swp_entry_t){};
}
/* Rotate the device and switch to a new cluster */
-static void swap_alloc_slow(struct folio *folio)
+static swp_entry_t swap_alloc_slow(struct folio *folio)
{
struct swap_info_struct *si, *next;
+ unsigned long found;
spin_lock(&swap_avail_lock);
start_over:
@@ -1405,12 +1655,12 @@ static void swap_alloc_slow(struct folio *folio)
plist_requeue(&si->avail_list, &swap_avail_head);
spin_unlock(&swap_avail_lock);
if (get_swap_device_info(si)) {
- cluster_alloc_swap_entry(si, folio);
+ found = cluster_alloc_swap_entry(si, folio);
put_swap_device(si);
- if (folio_test_swapcache(folio))
- return;
+ if (found)
+ return swp_entry(si->type, found);
if (folio_test_large(folio))
- return;
+ return (swp_entry_t){};
}
spin_lock(&swap_avail_lock);
@@ -1428,6 +1678,7 @@ static void swap_alloc_slow(struct folio *folio)
goto start_over;
}
spin_unlock(&swap_avail_lock);
+ return (swp_entry_t){};
}
/*
@@ -1507,6 +1758,7 @@ int swap_retry_table_alloc(swp_entry_t entry, gfp_t gfp)
if (!si)
return 0;
+ /* The source PTE pins the entry, so its cluster is alive. */
ci = __swap_offset_to_cluster(si, offset);
ret = swap_extend_table_alloc(si, ci, swp_cluster_offset(entry), gfp);
@@ -1611,6 +1863,8 @@ static void swap_put_entries_cluster(struct swap_info_struct *si,
}
/* count will be 0 after put, slot can be reclaimed */
need_reclaim = true;
+ if (swap_is_vswap(si))
+ vswap_mark_cache_only(ci, ci_off);
}
/*
* A count != 1 or cached slot can't be freed. Put its swap
@@ -1717,6 +1971,8 @@ static int swap_dup_entries_cluster(struct swap_info_struct *si,
goto failed;
}
} while (++ci_off < ci_end);
+ if (swap_is_vswap(si))
+ vswap_clear_cache_only(ci, ci_start, nr);
swap_cluster_unlock(ci);
return 0;
failed:
@@ -1727,6 +1983,76 @@ static int swap_dup_entries_cluster(struct swap_info_struct *si,
return err;
}
+static bool vswap_alloc(struct folio *folio)
+{
+ unsigned int order = folio_order(folio);
+ struct swap_cluster_info *ci;
+ struct obj_cgroup *objcg;
+ unsigned long offset;
+ bool may_zswap;
+
+ if (!vswap_is_enabled() || !zswap_is_enabled())
+ return false;
+
+ /*
+ * If zswap will not take the folio, writeout has to find a physical
+ * slot anyway. We are just incurring indirection overhead
+ * unnecessarily.
+ */
+ objcg = get_obj_cgroup_from_folio(folio);
+ may_zswap = !objcg || obj_cgroup_may_zswap(objcg);
+ if (objcg)
+ obj_cgroup_put(objcg);
+ if (!may_zswap)
+ return false;
+
+ local_lock(&percpu_vswap_cluster.lock);
+ offset = this_cpu_read(percpu_vswap_cluster.offset[order]);
+
+ if (offset != SWAP_ENTRY_INVALID) {
+ ci = swap_cluster_lock(vswap_si, offset);
+ if (ci && cluster_is_usable(ci, order)) {
+ if (cluster_is_empty(ci))
+ offset = cluster_offset(vswap_si, ci);
+ alloc_swap_scan_cluster(vswap_si, ci, folio, offset,
+ NULL);
+ } else if (ci) {
+ swap_cluster_unlock(ci);
+ }
+ }
+
+ if (!folio_test_swapcache(folio))
+ cluster_alloc_swap_entry(vswap_si, folio);
+
+ if (folio_test_swapcache(folio)) {
+ /* alloc_swap_scan_cluster updated percpu offset already */
+ local_unlock(&percpu_vswap_cluster.lock);
+ return true;
+ }
+
+ this_cpu_write(percpu_vswap_cluster.offset[order], SWAP_ENTRY_INVALID);
+ local_unlock(&percpu_vswap_cluster.lock);
+ atomic_long_add(folio_nr_pages(folio), &vswap_alloc_reject);
+ return false;
+}
+
+static swp_entry_t folio_alloc_phys_swap(struct folio *folio)
+{
+ swp_entry_t entry;
+
+again:
+ local_lock(&percpu_swap_cluster.lock);
+ entry = swap_alloc_fast(folio);
+ if (!entry.val)
+ entry = swap_alloc_slow(folio);
+ local_unlock(&percpu_swap_cluster.lock);
+
+ if (!entry.val && !folio_order(folio) && swap_sync_discard())
+ goto again;
+
+ return entry;
+}
+
/**
* folio_alloc_swap - allocate swap space for a folio
* @folio: folio we want to move to swap
@@ -1740,6 +2066,7 @@ static int swap_dup_entries_cluster(struct swap_info_struct *si,
int folio_alloc_swap(struct folio *folio)
{
unsigned int order = folio_order(folio);
+ struct mem_cgroup *memcg;
unsigned int size = 1 << order;
VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
@@ -1763,27 +2090,350 @@ int folio_alloc_swap(struct folio *folio)
}
}
-again:
- local_lock(&percpu_swap_cluster.lock);
- if (!swap_alloc_fast(folio))
- swap_alloc_slow(folio);
- local_unlock(&percpu_swap_cluster.lock);
+ if (!vswap_alloc(folio))
+ folio_alloc_phys_swap(folio);
- if (!order && unlikely(!folio_test_swapcache(folio))) {
- if (swap_sync_discard())
- goto again;
+ /*
+ * Need to call this even if allocation failed, for MEMCG_SWAP_FAIL.
+ * A vswap entry has no physical swap yet, so only record the memcg.
+ * folio_realloc_swap() charges it once backing is allocated.
+ */
+ memcg = mem_cgroup_swap_get(folio);
+ if (memcg) {
+ if (!is_vswap_entry(folio->swap) &&
+ unlikely(mem_cgroup_swap_charge(memcg, size))) {
+ mem_cgroup_swap_put(memcg, size);
+ swap_cache_del_folio(folio);
+ } else {
+ mem_cgroup_swap_record(folio, memcg);
+ }
}
- /* Need to call this even if allocation failed, for MEMCG_SWAP_FAIL. */
- if (unlikely(mem_cgroup_try_charge_swap(folio)))
- swap_cache_del_folio(folio);
-
if (unlikely(!folio_test_swapcache(folio)))
return -ENOMEM;
return 0;
}
+static void vswap_mark_cache_only(struct swap_cluster_info *ci,
+ unsigned int ci_off)
+{
+ struct swap_cluster_info_dynamic *ci_dyn;
+ struct swap_cluster_info *pci;
+ swp_entry_t phys;
+ unsigned long vt;
+
+ ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
+ vt = __vtable_get(ci_dyn, ci_off);
+
+ if (vtable_type(vt) == VSWAP_SWAPFILE) {
+ phys = vtable_to_phys(vt);
+ pci = __swap_entry_to_cluster(phys);
+ swap_rmap_mark_cache_only(pci, swp_cluster_offset(phys));
+ }
+}
+
+/*
+ * Clear the cache-only rmap hint for entries re-referenced from count 0 to 1
+ * (no longer reclaimable), so the physical reclaim scanner skips them.
+ */
+static void vswap_clear_cache_only(struct swap_cluster_info *ci,
+ unsigned int ci_start, int nr)
+{
+ struct swap_cluster_info_dynamic *ci_dyn;
+ struct swap_cluster_info *pci;
+ unsigned long swp_tb, vt;
+ swp_entry_t phys;
+ unsigned int off;
+
+ ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
+ for (off = ci_start; off < ci_start + nr; off++) {
+ swp_tb = __swap_table_get(ci, off);
+ if (!swp_tb_is_folio(swp_tb) || swp_tb_get_count(swp_tb) != 1)
+ continue;
+ vt = __vtable_get(ci_dyn, off);
+ if (vtable_type(vt) != VSWAP_SWAPFILE)
+ continue;
+ phys = vtable_to_phys(vt);
+ pci = __swap_entry_to_cluster(phys);
+ swap_rmap_clear_cache_only(pci, swp_cluster_offset(phys));
+ }
+}
+
+static void __swap_cluster_free_phys_backing(struct swap_info_struct *psi,
+ struct swap_cluster_info *pci,
+ unsigned int ci_start,
+ unsigned int nr_pages);
+
+static void vswap_uncharge_cgroup_batch(unsigned short memcg_id,
+ unsigned int batch_nr,
+ unsigned int batch_nr_swapfile)
+{
+ struct mem_cgroup *memcg;
+ unsigned int n;
+
+ /*
+ * v1 (memsw): entries keep their memsw charge across swapout
+ * regardless of backing, so uncharge all of them. v2: only
+ * swapfile-backed entries are charged, so uncharge just those.
+ *
+ * On v1 the id is written by __memcg1_swapout() as the folio leaves the
+ * swap cache and cleared by memcg1_swapin() when it comes back, both
+ * under the cluster lock. Callers still holding a cached folio are
+ * outside that window and see @memcg_id == 0, so only the free path
+ * uncharges. On v2 the id is set when swap is allocated, so those
+ * callers do uncharge, which balances the charge folio_realloc_swap()
+ * took.
+ */
+ n = do_memsw_account() ? batch_nr : batch_nr_swapfile;
+ if (!n)
+ return;
+
+ rcu_read_lock();
+ memcg = memcg_id ? mem_cgroup_from_private_id(memcg_id) : NULL;
+ rcu_read_unlock();
+ mem_cgroup_swap_uncharge(memcg, n);
+}
+
+/**
+ * __vswap_release_backing - release the backing of a range of vtable slots
+ * @ci: the locked vswap cluster
+ * @ci_start: first slot offset within @ci
+ * @nr: number of slots
+ *
+ * Releases the backing of each slot in [@ci_start, @ci_start + @nr).
+ * Clears the zero marks if set.
+ *
+ * Context: caller must hold @ci->lock.
+ */
+void __vswap_release_backing(struct swap_cluster_info *ci,
+ unsigned int ci_start, unsigned int nr)
+{
+ struct swap_cluster_info_dynamic *ci_dyn;
+ struct swap_info_struct *psi;
+ unsigned long phys_start = 0, phys_end = 0;
+ unsigned int phys_type = 0;
+ unsigned int ci_off;
+ unsigned long vt;
+ swp_entry_t phys;
+ unsigned short batch_id;
+ unsigned int batch_nr = 0, batch_nr_swapfile = 0;
+
+ lockdep_assert_held(&ci->lock);
+ ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
+ batch_id = __swap_cgroup_get(ci, ci_start);
+
+ for (ci_off = ci_start; ci_off < ci_start + nr; ci_off++) {
+ unsigned short cur_id;
+
+ vt = __vtable_get(ci_dyn, ci_off);
+ cur_id = __swap_cgroup_get(ci, ci_off);
+
+ if (cur_id != batch_id) {
+ vswap_uncharge_cgroup_batch(batch_id, batch_nr,
+ batch_nr_swapfile);
+ batch_id = cur_id;
+ batch_nr = 0;
+ batch_nr_swapfile = 0;
+ }
+ batch_nr++;
+
+ /* The free helper takes one contiguous run within one cluster. */
+ if (phys_start != phys_end &&
+ (vtable_type(vt) != VSWAP_SWAPFILE ||
+ swp_type(vtable_to_phys(vt)) != phys_type ||
+ swp_offset(vtable_to_phys(vt)) != phys_end ||
+ phys_end % SWAPFILE_CLUSTER == 0)) {
+ psi = __swap_type_to_info(phys_type);
+ __swap_cluster_free_phys_backing(psi,
+ __swap_entry_to_cluster(
+ swp_entry(phys_type, phys_start)),
+ phys_start % SWAPFILE_CLUSTER,
+ phys_end - phys_start);
+ phys_start = phys_end = 0;
+ }
+
+ switch (vtable_type(vt)) {
+ case VSWAP_SWAPFILE:
+ batch_nr_swapfile++;
+ if (phys_start == phys_end) {
+ phys = vtable_to_phys(vt);
+ phys_start = swp_offset(phys);
+ phys_end = phys_start + 1;
+ phys_type = swp_type(phys);
+ } else {
+ phys_end++;
+ }
+ break;
+ case VSWAP_ZSWAP:
+ zswap_entry_free(vtable_to_zswap(vt));
+ break;
+ case VSWAP_NONE:
+ break;
+ default:
+ /* VSWAP_ZERO/VSWAP_FOLIO are return-only, not vtable tags */
+ break;
+ }
+
+ __vtable_set(ci_dyn, ci_off, VSWAP_NONE);
+ /* Zero-backed state lives in swap_table; clear it too. */
+ if (__swap_table_test_zero(ci, ci_off))
+ __swap_table_clear_zero(ci, ci_off);
+ }
+
+ if (phys_start != phys_end) {
+ psi = __swap_type_to_info(phys_type);
+ __swap_cluster_free_phys_backing(psi,
+ __swap_entry_to_cluster(
+ swp_entry(phys_type, phys_start)),
+ phys_start % SWAPFILE_CLUSTER,
+ phys_end - phys_start);
+ }
+
+ vswap_uncharge_cgroup_batch(batch_id, batch_nr, batch_nr_swapfile);
+}
+
+/**
+ * folio_release_vswap_backing() - Drop all backing for a folio's vswap entry.
+ * @folio: the folio, occupying a virtual swap entry.
+ *
+ * Release whatever backing the folio's virtual swap slots currently hold and
+ * reset them to empty, so a fresh backing can be installed. Used when a
+ * folio's swap backend is replaced.
+ *
+ * Context: Caller must hold the folio lock; @folio must be in the swap cache
+ * and occupy a virtual swap entry.
+ */
+void folio_release_vswap_backing(struct folio *folio)
+{
+ struct swap_cluster_info *ci;
+ int nr = folio_nr_pages(folio);
+ unsigned int voff;
+
+ ci = __swap_entry_to_cluster(folio->swap);
+ voff = swp_cluster_offset(folio->swap);
+
+ spin_lock(&ci->lock);
+ __vswap_release_backing(ci, voff, nr);
+ spin_unlock(&ci->lock);
+}
+
+/**
+ * folio_release_non_phys_swap_backing() - Drop a folio's non-physical vswap backing.
+ * @folio: the folio, occupying a virtual swap entry.
+ *
+ * Release the zswap backing recorded for @folio's virtual swap entry,
+ * leaving the slots empty so the writeout path can install fresh physical
+ * backing. Does nothing when the entry is already backed by physical
+ * swapfile slots, which are kept for reuse, or when it has no backing
+ * beyond the swap cache folio itself.
+ *
+ * Context: Caller must hold the folio lock; @folio must be in the swap cache
+ * and occupy a virtual swap entry.
+ */
+void folio_release_non_phys_swap_backing(struct folio *folio)
+{
+ struct swap_cluster_info *ci;
+ struct swap_cluster_info_dynamic *ci_dyn;
+ int nr = folio_nr_pages(folio);
+ unsigned int voff;
+ unsigned long vt;
+ enum vswap_backing_type type;
+
+ ci = __swap_entry_to_cluster(folio->swap);
+ ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
+ voff = swp_cluster_offset(folio->swap);
+
+ spin_lock(&ci->lock);
+ /* A folio's slots cannot mix swapfile with other backends. */
+ vt = __vtable_get(ci_dyn, voff);
+ type = vtable_type(vt);
+
+ if (type == VSWAP_SWAPFILE || type == VSWAP_NONE) {
+ spin_unlock(&ci->lock);
+ return;
+ }
+
+ __vswap_release_backing(ci, voff, nr);
+ spin_unlock(&ci->lock);
+}
+
+/**
+ * folio_realloc_swap() - Back a virtual swap folio with a physical swap slot.
+ * @folio: the folio, occupying a virtual swap entry.
+ *
+ * Ensure @folio's virtual swap entry has physical (swapfile) backing,
+ * allocating a physical slot on demand if it has none. If @folio is
+ * already physically backed, the existing physical entry is returned
+ * unchanged.
+ *
+ * Context: Caller must hold the folio lock; @folio must be in the swap cache
+ * and occupy a virtual swap entry.
+ * Return: The physical swap entry now backing @folio, or an empty entry
+ * (.val == 0) on failure.
+ */
+swp_entry_t folio_realloc_swap(struct folio *folio)
+{
+ swp_entry_t vswap_entry = folio->swap;
+ struct swap_cluster_info *ci;
+ struct swap_cluster_info_dynamic *ci_dyn;
+ struct mem_cgroup *memcg;
+ unsigned int voff;
+ unsigned long vt;
+ unsigned short memcg_id;
+ swp_entry_t phys_entry = {};
+ swp_entry_t pe;
+ int i, nr = folio_nr_pages(folio);
+
+ VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
+ VM_BUG_ON_FOLIO(!folio_test_swapcache(folio), folio);
+ VM_WARN_ON(!is_vswap_entry(vswap_entry));
+
+ voff = swp_cluster_offset(vswap_entry);
+ ci = __swap_entry_to_cluster(vswap_entry);
+ ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
+
+ spin_lock(&ci->lock);
+ vt = __vtable_get(ci_dyn, voff);
+ if (vtable_type(vt) == VSWAP_SWAPFILE) {
+ spin_unlock(&ci->lock);
+ return vtable_to_phys(vt);
+ }
+ memcg_id = __swap_cgroup_get(ci, voff);
+ spin_unlock(&ci->lock);
+
+ phys_entry = folio_alloc_phys_swap(folio);
+ if (!phys_entry.val)
+ return (swp_entry_t){};
+
+ rcu_read_lock();
+ memcg = folio_memcg(folio);
+ if (!memcg || mem_cgroup_private_id(memcg) != memcg_id)
+ memcg = memcg_id ? mem_cgroup_from_private_id(memcg_id) : NULL;
+ rcu_read_unlock();
+
+ if (mem_cgroup_swap_charge(memcg, nr)) {
+ __swap_cluster_free_phys_backing(__swap_entry_to_info(phys_entry),
+ __swap_entry_to_cluster(phys_entry),
+ swp_cluster_offset(phys_entry),
+ nr);
+ return (swp_entry_t){};
+ }
+
+ spin_lock(&ci->lock);
+ /*
+ * Install PHYS backing without freeing any prior contents of the
+ * vtable. Releasing the old backing is the caller's job.
+ */
+ for (i = 0; i < nr; i++) {
+ pe.val = phys_entry.val + i;
+ __vtable_set(ci_dyn, voff + i, vtable_mk_phys(pe));
+ }
+ spin_unlock(&ci->lock);
+
+ return phys_entry;
+}
+
/**
* folio_dup_swap() - Increase swap count of swap entries of a folio.
* @folio: folio with swap entries bounded.
@@ -1904,10 +2554,70 @@ struct swap_info_struct *get_swap_device(swp_entry_t entry)
return NULL;
put_out:
pr_err_ratelimited("%s: %s%08lx\n", __func__, Bad_offset, entry.val);
- percpu_ref_put(&si->users);
+ put_swap_device(si);
return NULL;
}
+/*
+ * Common tail for freeing swap slots: device-level accounting
+ * and cluster list management.
+ */
+static void __swap_cluster_finish_free(struct swap_info_struct *si,
+ struct swap_cluster_info *ci,
+ unsigned int ci_start,
+ unsigned int nr_pages)
+{
+ lockdep_assert_held(&ci->lock);
+ swap_range_free(si, cluster_offset(si, ci) + ci_start, nr_pages);
+ swap_cluster_assert_empty(ci, ci_start, nr_pages, false);
+
+ if (!ci->count)
+ free_cluster(si, ci);
+ else
+ partial_free_cluster(si, ci);
+}
+
+/*
+ * Free physical swap slots that were backing vswap entries (Pointer-tagged).
+ */
+static void __swap_cluster_free_phys_backing(struct swap_info_struct *psi,
+ struct swap_cluster_info *pci,
+ unsigned int ci_start,
+ unsigned int nr_pages)
+{
+ unsigned int ci_off;
+
+ spin_lock_nested(&pci->lock, SINGLE_DEPTH_NESTING);
+ VM_WARN_ON(pci->count < nr_pages);
+ pci->count -= nr_pages;
+ for (ci_off = ci_start; ci_off < ci_start + nr_pages; ci_off++) {
+ __swap_table_set(pci, ci_off, null_to_swp_tb());
+ if (!SWAP_TABLE_HAS_ZEROFLAG)
+ __swap_table_clear_zero(pci, ci_off);
+ }
+ __swap_cluster_finish_free(psi, pci, ci_start, nr_pages);
+ swap_cluster_unlock(pci);
+}
+
+/*
+ * Release the cgroup accounting of a batch of freed slots. For vswap the
+ * physical swap was already uncharged by __vswap_release_backing(), so only
+ * the ID ref is left to drop.
+ */
+static void memcg_swap_free(unsigned short id, unsigned int nr, bool is_vswap)
+{
+ struct mem_cgroup *memcg;
+
+ rcu_read_lock();
+ memcg = mem_cgroup_from_private_id(id);
+ if (memcg) {
+ if (!is_vswap)
+ mem_cgroup_swap_uncharge(memcg, nr);
+ mem_cgroup_swap_put(memcg, nr);
+ }
+ rcu_read_unlock();
+}
+
/*
* Free a set of swap slots after their swap count dropped to zero, or will be
* zero after putting the last ref (saves one __swap_cluster_put_entry call).
@@ -1919,11 +2629,14 @@ void __swap_cluster_free_entries(struct swap_info_struct *si,
unsigned long old_tb;
unsigned short batch_id = 0, id_cur;
unsigned int ci_off = ci_start, ci_end = ci_start + nr_pages;
- unsigned long ci_head = cluster_offset(si, ci);
unsigned int batch_off = ci_off;
+ bool is_vswap = swap_is_vswap(si);
VM_WARN_ON(ci->count < nr_pages);
+ if (is_vswap)
+ __vswap_release_backing(ci, ci_start, nr_pages);
+
ci->count -= nr_pages;
do {
old_tb = __swap_table_get(ci, ci_off);
@@ -1945,22 +2658,16 @@ void __swap_cluster_free_entries(struct swap_info_struct *si,
id_cur = __swap_cgroup_clear(ci, ci_off, 1);
if (batch_id != id_cur) {
if (batch_id)
- mem_cgroup_uncharge_swap(batch_id, ci_off - batch_off);
+ memcg_swap_free(batch_id, ci_off - batch_off, is_vswap);
batch_id = id_cur;
batch_off = ci_off;
}
} while (++ci_off < ci_end);
if (batch_id)
- mem_cgroup_uncharge_swap(batch_id, ci_off - batch_off);
-
- swap_range_free(si, ci_head + ci_start, nr_pages);
- swap_cluster_assert_empty(ci, ci_start, nr_pages, false);
+ memcg_swap_free(batch_id, ci_off - batch_off, is_vswap);
- if (!ci->count)
- free_cluster(si, ci);
- else
- partial_free_cluster(si, ci);
+ __swap_cluster_finish_free(si, ci, ci_start, nr_pages);
}
int __swap_count(swp_entry_t entry)
@@ -2036,6 +2743,7 @@ static bool folio_maybe_swapped(struct folio *folio)
VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
VM_WARN_ON_ONCE_FOLIO(!folio_test_swapcache(folio), folio);
+ /* Folio is locked and in swap cache, so ci->count > 0: cluster is alive. */
ci = __swap_entry_to_cluster(entry);
ci_off = swp_cluster_offset(entry);
ci_end = ci_off + folio_nr_pages(folio);
@@ -2174,7 +2882,8 @@ swp_entry_t swap_alloc_hibernation_slot(int type)
if (pcp_si == si && pcp_offset) {
ci = swap_cluster_lock(si, pcp_offset);
if (cluster_is_usable(ci, 0))
- offset = alloc_swap_scan_cluster(si, ci, NULL, pcp_offset);
+ offset = alloc_swap_scan_cluster(si, ci, NULL,
+ pcp_offset, NULL);
else
swap_cluster_unlock(ci);
}
@@ -2230,6 +2939,9 @@ static int __find_hibernation_swap_type(dev_t device, sector_t offset)
if (!(sis->flags & SWP_WRITEOK))
continue;
+ /* vswap has no bdev, so it is never a hibernation target. */
+ if (swap_is_vswap(sis))
+ continue;
if (device == sis->bdev->bd_dev) {
struct swap_extent *se = first_se(sis);
@@ -2356,6 +3068,9 @@ int find_first_swap(dev_t *device)
if (!(sis->flags & SWP_WRITEOK))
continue;
+ /* vswap has no bdev, so it is never a hibernation target. */
+ if (swap_is_vswap(sis))
+ continue;
*device = sis->bdev->bd_dev;
spin_unlock(&swap_lock);
return type;
@@ -2572,8 +3287,10 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd,
&vmf);
}
if (!folio) {
+ rcu_read_lock();
swp_tb = swap_table_get(__swap_entry_to_cluster(entry),
swp_cluster_offset(entry));
+ rcu_read_unlock();
if (swp_tb_get_count(swp_tb) <= 0)
continue;
return -ENOMEM;
@@ -2706,10 +3423,10 @@ static int unuse_mm(struct mm_struct *mm, unsigned int type)
* Return 0 if there are no inuse entries after prev till end of
* the map.
*/
-static unsigned int find_next_to_unuse(struct swap_info_struct *si,
- unsigned int prev)
+static unsigned long find_next_to_unuse(struct swap_info_struct *si,
+ unsigned long prev)
{
- unsigned int i;
+ unsigned long i;
unsigned long swp_tb;
/*
@@ -2719,8 +3436,10 @@ static unsigned int find_next_to_unuse(struct swap_info_struct *si,
* allocations from this area (while holding swap_lock).
*/
for (i = prev + 1; i < si->max; i++) {
+ rcu_read_lock();
swp_tb = swap_table_get(__swap_offset_to_cluster(si, i),
i % SWAPFILE_CLUSTER);
+ rcu_read_unlock();
if (!swp_tb_is_null(swp_tb) && !swp_tb_is_bad(swp_tb))
break;
if ((i % LATENCY_LIMIT) == 0)
@@ -2735,19 +3454,100 @@ static unsigned int find_next_to_unuse(struct swap_info_struct *si,
static int try_to_unuse(unsigned int type)
{
+ struct mempolicy *mpol = get_task_policy(current);
struct mm_struct *prev_mm;
struct mm_struct *mm;
struct list_head *p;
int retval = 0;
struct swap_info_struct *si = swap_info[type];
struct folio *folio;
- swp_entry_t entry;
- unsigned int i;
+ struct swap_io_ctx ctx;
+ swp_entry_t entry, vswap_entry;
+ unsigned long swp_tb;
+ unsigned long i;
+ unsigned int j;
if (!swap_usage_in_pages(si))
goto success;
retry:
+ /*
+ * Free vswap-backing slots (Pointer-tagged) first. Walk physical
+ * clusters, read the vswap entry from the rmap, ensure the data
+ * is in the swap cache, and transition PHYS to FOLIO. Freeing the
+ * physical backing is enough, so no page table walk is needed.
+ */
+ i = 0;
+ while (vswap_is_enabled() &&
+ swap_usage_in_pages(si) &&
+ !signal_pending(current) &&
+ (i = find_next_to_unuse(si, i)) != 0) {
+ swp_entry_t phys;
+
+ swp_tb = swap_table_get(__swap_offset_to_cluster(si, i),
+ i % SWAPFILE_CLUSTER);
+ if (!swp_tb_is_pointer(swp_tb))
+ continue;
+
+ vswap_entry = swp_tb_ptr_to_swp_entry(swp_tb);
+
+ folio = swap_cache_get_folio(vswap_entry);
+ if (!folio) {
+ folio = swap_cache_alloc_folio(vswap_entry,
+ GFP_HIGHUSER_MOVABLE,
+ BIT(0), NULL, mpol,
+ NO_INTERLEAVE_INDEX);
+ if (IS_ERR(folio)) {
+ if (PTR_ERR(folio) == -ENOMEM)
+ return -ENOMEM;
+ continue;
+ }
+ ctx = (struct swap_io_ctx){};
+ swap_read_folio(&ctx, folio);
+ swap_read_submit(&ctx);
+ folio_lock(folio);
+ } else {
+ folio_lock(folio);
+ }
+
+ if (!folio_matches_swap_entry(folio, vswap_entry)) {
+ folio_unlock(folio);
+ folio_put(folio);
+ continue;
+ }
+
+ /*
+ * Re-validate under folio lock: rmap holds folio->swap + j
+ * for some j in [0, nr_pages). Check folio->swap still maps
+ * to the contiguous physical run that includes our slot i.
+ */
+ j = vswap_entry.val - folio->swap.val;
+ phys = vswap_to_phys(folio->swap);
+ if (!phys.val || swp_type(phys) != type ||
+ swp_offset(phys) + j != i) {
+ folio_unlock(folio);
+ folio_put(folio);
+ continue;
+ }
+
+ folio_wait_writeback(folio);
+ folio_release_vswap_backing(folio);
+ /*
+ * Drop a folio whose read failed rather than dirtying
+ * uninitialised memory; the next fault finds no backing and
+ * gets SIGBUS.
+ */
+ if (unlikely(!folio_test_uptodate(folio)))
+ swap_cache_del_folio(folio);
+ else
+ folio_mark_dirty(folio);
+ folio_unlock(folio);
+ folio_put(folio);
+ }
+
+ if (!swap_usage_in_pages(si))
+ goto success;
+
retval = shmem_unuse(type);
if (retval)
return retval;
@@ -2790,6 +3590,8 @@ static int try_to_unuse(unsigned int type)
(i = find_next_to_unuse(si, i)) != 0) {
entry = swp_entry(type, i);
+
+ /* Pointer-tagged rmap slots have no folio; the pre-pass took them. */
folio = swap_cache_get_folio(entry);
if (!folio)
continue;
@@ -2959,6 +3761,11 @@ static int setup_swap_extents(struct swap_info_struct *sis,
struct inode *inode = mapping->host;
int ret;
+ if (swap_is_vswap(sis)) {
+ *span = 0;
+ return 0;
+ }
+
ret = sio_pool_init();
if (ret)
return ret;
@@ -2984,15 +3791,24 @@ static int setup_swap_extents(struct swap_info_struct *sis,
static void _enable_swap_info(struct swap_info_struct *si)
{
- atomic_long_add(si->pages, &nr_swap_pages);
- total_swap_pages += si->pages;
+ if (!swap_is_vswap(si)) {
+ atomic_long_add(si->pages, &nr_swap_pages);
+ total_swap_pages += si->pages;
+ }
assert_spin_locked(&swap_lock);
- plist_add(&si->list, &swap_active_head);
+ /*
+ * Vswap has no backing file and no swapoff support, so keep it
+ * off swap_active_head (used by swapoff filename lookup and
+ * swap_sync_discard) and swap_avail_head (physical allocator).
+ */
+ if (!swap_is_vswap(si)) {
+ plist_add(&si->list, &swap_active_head);
- /* Add back to available list */
- add_to_avail_list(si, true);
+ /* Add back to available list */
+ add_to_avail_list(si, true);
+ }
}
/*
@@ -3036,12 +3852,32 @@ static void wait_for_allocation(struct swap_info_struct *si)
}
}
-static void free_swap_cluster_info(struct swap_cluster_info *cluster_info,
+static void free_swap_cluster_info(struct swap_info_struct *si,
+ struct swap_cluster_info *cluster_info,
unsigned long maxpages)
{
+ struct swap_cluster_info_dynamic *ci_dyn;
struct swap_cluster_info *ci;
+ unsigned long idx;
int i, nr_clusters = DIV_ROUND_UP(maxpages, SWAPFILE_CLUSTER);
+ if (swap_is_vswap(si)) {
+ xa_for_each(&si->cluster_info_pool, idx, ci_dyn) {
+ ci = &ci_dyn->ci;
+ spin_lock(&ci->lock);
+ if (cluster_table_is_alloced(ci)) {
+ swap_cluster_assert_empty(ci, 0,
+ SWAPFILE_CLUSTER, true);
+ swap_cluster_free_table(ci);
+ }
+ spin_unlock(&ci->lock);
+ vswap_cluster_free_vtable(ci);
+ kfree(ci_dyn);
+ }
+ xa_destroy(&si->cluster_info_pool);
+ return;
+ }
+
if (!cluster_info)
return;
for (i = 0; i < nr_clusters; i++) {
@@ -3086,7 +3922,7 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
struct file *swap_file, *victim;
struct address_space *mapping;
struct inode *inode;
- unsigned int maxpages;
+ unsigned long maxpages;
int err, found = 0;
if (!capable(CAP_SYS_ADMIN))
@@ -3188,7 +4024,7 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
mutex_unlock(&swapon_mutex);
kfree(p->global_cluster);
p->global_cluster = NULL;
- free_swap_cluster_info(cluster_info, maxpages);
+ free_swap_cluster_info(p, cluster_info, maxpages);
inode = mapping->host;
@@ -3508,12 +4344,8 @@ static unsigned long read_swap_header(struct swap_info_struct *si,
pr_warn("Truncating oversized swap area, only using %luk out of %luk\n",
K(maxpages), K(last_page));
}
- if (maxpages > last_page) {
+ if (maxpages > last_page)
maxpages = last_page + 1;
- /* p->max is an unsigned int: don't overflow it */
- if ((unsigned int)maxpages == 0)
- maxpages = UINT_MAX;
- }
if (!maxpages)
return 0;
@@ -3535,10 +4367,43 @@ static int setup_swap_clusters_info(struct swap_info_struct *si,
unsigned long maxpages)
{
unsigned long nr_clusters = DIV_ROUND_UP(maxpages, SWAPFILE_CLUSTER);
- struct swap_cluster_info *cluster_info;
+ struct swap_cluster_info *cluster_info = NULL;
+ struct swap_cluster_info_dynamic *ci_dyn = NULL;
int err = -ENOMEM;
unsigned long i;
+ /* A vswap device uses an xarray pool instead of a static array. */
+ if (swap_is_vswap(si)) {
+ nr_clusters = 0;
+ xa_init_flags(&si->cluster_info_pool, XA_FLAGS_ALLOC);
+
+ /*
+ * Pre-allocate cluster 0 and mark slot 0 (header page)
+ * as bad so the allocator never hands out page offset 0.
+ */
+ ci_dyn = kzalloc_obj(*ci_dyn, GFP_KERNEL);
+ if (!ci_dyn)
+ goto err;
+ spin_lock_init(&ci_dyn->ci.lock);
+ INIT_LIST_HEAD(&ci_dyn->ci.list);
+
+ err = xa_insert(&si->cluster_info_pool, 0, ci_dyn, GFP_KERNEL);
+ if (err) {
+ kfree(ci_dyn);
+ goto err;
+ }
+
+ err = swap_cluster_setup_bad_slot(si, &ci_dyn->ci, 0, false);
+ if (err)
+ goto err;
+
+ err = vswap_cluster_alloc_vtable(ci_dyn, GFP_KERNEL);
+ if (err)
+ goto err;
+
+ goto setup_cluster_info;
+ }
+
cluster_info = kvzalloc_objs(*cluster_info, nr_clusters);
if (!cluster_info)
goto err;
@@ -3582,6 +4447,7 @@ static int setup_swap_clusters_info(struct swap_info_struct *si,
goto err;
}
+setup_cluster_info:
INIT_LIST_HEAD(&si->free_clusters);
INIT_LIST_HEAD(&si->full_clusters);
INIT_LIST_HEAD(&si->discard_clusters);
@@ -3603,10 +4469,16 @@ static int setup_swap_clusters_info(struct swap_info_struct *si,
}
}
+ /* Slot 0 is bad, so cluster 0 never empties. The rest of it is usable. */
+ if (swap_is_vswap(si)) {
+ ci_dyn->ci.flags = CLUSTER_FLAG_NONFULL;
+ list_add_tail(&ci_dyn->ci.list, &si->nonfull_clusters[0]);
+ }
+
si->cluster_info = cluster_info;
return 0;
err:
- free_swap_cluster_info(cluster_info, maxpages);
+ free_swap_cluster_info(si, cluster_info, maxpages);
return err;
}
@@ -3714,7 +4586,7 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
goto bad_swap_unlock_inode;
}
if (si->pages != si->max - 1) {
- pr_err("swap:%u != (max:%u - 1)\n", si->pages, si->max);
+ pr_err("swap:%lu != (max:%lu - 1)\n", si->pages, si->max);
error = -EINVAL;
goto bad_swap_unlock_inode;
}
@@ -3802,7 +4674,7 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
/* Sets SWP_WRITEOK, resurrect the percpu ref, expose the swap device */
enable_swap_info(si);
- pr_info("Adding %uk swap on %s. Priority:%d extents:%d across:%lluk %s%s%s%s\n",
+ pr_info("Adding %luk swap on %s. Priority:%d extents:%d across:%lluk %s%s%s%s\n",
K(si->pages), name->name, si->prio, nr_extents,
K((unsigned long long)span),
(si->flags & SWP_SOLIDSTATE) ? "SS" : "",
@@ -3825,7 +4697,7 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
si->global_cluster = NULL;
inode = NULL;
destroy_swap_extents(si, swap_file);
- free_swap_cluster_info(si->cluster_info, si->max);
+ free_swap_cluster_info(si, si->cluster_info, si->max);
si->cluster_info = NULL;
/*
* Clear the SWP_USED flag after all resources are freed so
@@ -3956,3 +4828,105 @@ static int __init swapfile_init(void)
return 0;
}
subsys_initcall(swapfile_init);
+
+struct swap_info_struct *vswap_si;
+DEFINE_STATIC_KEY_FALSE(vswap_key);
+
+static bool vswap_enabled_early __initdata = IS_ENABLED(CONFIG_VSWAP_DEFAULT_ON);
+
+static int __init early_vswap(char *buf)
+{
+ return kstrtobool(buf, &vswap_enabled_early);
+}
+early_param("vswap", early_vswap);
+
+/* vswap does no IO on its own. */
+static const struct swap_ops vswap_ops = { };
+
+static int vswap_used_get(void *data, u64 *val)
+{
+ *val = swap_usage_in_pages(vswap_si);
+ return 0;
+}
+DEFINE_DEBUGFS_ATTRIBUTE(vswap_used_fops, vswap_used_get, NULL, "%llu\n");
+
+static int vswap_alloc_reject_get(void *data, u64 *val)
+{
+ *val = atomic_long_read(&vswap_alloc_reject);
+ return 0;
+}
+DEFINE_DEBUGFS_ATTRIBUTE(vswap_alloc_reject_fops, vswap_alloc_reject_get, NULL,
+ "%llu\n");
+
+static int __init vswap_init(void)
+{
+ struct swap_info_struct *si;
+ struct dentry *root;
+ unsigned long maxpages;
+ int err;
+
+ if (!IS_ENABLED(CONFIG_64BIT)) {
+ if (vswap_enabled_early)
+ pr_warn("vswap: requires 64-bit architecture; vswap disabled, swapout falls back to direct physical swap\n");
+ return 0;
+ }
+
+ if (!vswap_enabled_early)
+ return 0;
+
+ si = alloc_swap_info();
+ if (IS_ERR(si)) {
+ pr_warn("vswap: alloc_swap_info failed (%ld); vswap disabled, swapout falls back to direct physical swap\n",
+ PTR_ERR(si));
+ return 0;
+ }
+
+ /*
+ * One u32 xarray ID per cluster, so the device cannot be larger
+ * than UINT_MAX clusters.
+ */
+ maxpages = min(swapfile_maximum_size,
+ (unsigned long)UINT_MAX * SWAPFILE_CLUSTER);
+ /*
+ * SWP_WRITEOK enables slot allocation. SWP_SOLIDSTATE selects
+ * per-CPU cluster allocation; vswap has no si->global_cluster.
+ */
+ si->flags |= SWP_VSWAP | SWP_SOLIDSTATE | SWP_WRITEOK;
+ si->ops = &vswap_ops;
+ si->bdev = NULL;
+ si->max = maxpages;
+ si->pages = maxpages - 1;
+ si->prio = SHRT_MAX;
+ si->list.prio = -si->prio;
+ si->avail_list.prio = -si->prio;
+
+ err = setup_swap_clusters_info(si, NULL, maxpages);
+ if (err)
+ goto fail;
+
+ mutex_lock(&swapon_mutex);
+ enable_swap_info(si);
+ mutex_unlock(&swapon_mutex);
+
+ vswap_si = si;
+
+ root = debugfs_create_dir("vswap", NULL);
+ debugfs_create_file("used", 0444, root, NULL, &vswap_used_fops);
+ debugfs_create_file("alloc_reject", 0444, root, NULL,
+ &vswap_alloc_reject_fops);
+
+ pr_info("vswap: created virtual swap device (%lu pages)\n", maxpages);
+
+ /* Last: everything above must be visible before routing starts. */
+ static_branch_enable(&vswap_key);
+ return 0;
+
+fail:
+ pr_warn("vswap: setup_swap_clusters_info failed (%d); vswap disabled, swapout falls back to direct physical swap\n",
+ err);
+ spin_lock(&swap_lock);
+ si->flags = 0;
+ spin_unlock(&swap_lock);
+ return 0;
+}
+late_initcall(vswap_init);
diff --git a/mm/vmscan.c b/mm/vmscan.c
index c1404a59523d6..7960cc489ea03 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -69,6 +69,7 @@
#include "internal.h"
#include "page_alloc.h"
#include "swap.h"
+#include "vswap.h"
#define CREATE_TRACE_POINTS
#include <trace/events/vmscan.h>
@@ -353,6 +354,9 @@ static inline bool can_reclaim_anon_pages(struct mem_cgroup *memcg,
*/
if (get_nr_swap_pages() > 0)
return true;
+ /* vswap doesn't contribute to nr_swap_pages */
+ if (vswap_is_enabled() && zswap_is_enabled())
+ return true;
} else {
/* Is the memcg below its swap limit? */
if (mem_cgroup_get_nr_swap_pages(memcg) > 0)
@@ -1524,7 +1528,8 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
activate_locked:
/* Not a candidate for swapping, so reclaim swap space. */
if (folio_test_swapcache(folio) &&
- (mem_cgroup_swap_full(folio) || folio_test_mlocked(folio)))
+ ((mem_cgroup_swap_full(folio) && folio_phys_swap_backed(folio)) ||
+ folio_test_mlocked(folio)))
folio_free_swap(folio);
VM_BUG_ON_FOLIO(folio_test_active(folio), folio);
if (!folio_test_mlocked(folio)) {
@@ -2681,7 +2686,7 @@ static bool can_age_anon_pages(struct lruvec *lruvec,
struct scan_control *sc)
{
/* Aging the anon LRU is valuable if swap is present: */
- if (total_swap_pages > 0)
+ if (total_swap_pages > 0 || (vswap_is_enabled() && zswap_is_enabled()))
return true;
/* Also valuable if anon pages can be demoted: */
diff --git a/mm/vswap.h b/mm/vswap.h
new file mode 100644
index 0000000000000..c66fa34e2e60c
--- /dev/null
+++ b/mm/vswap.h
@@ -0,0 +1,440 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Virtual swap space
+ *
+ * Copyright (C) 2026 Nhat Pham
+ */
+#ifndef _MM_VSWAP_H
+#define _MM_VSWAP_H
+
+#include <linux/jump_label.h>
+#include <linux/swap.h>
+#include "swap.h"
+
+struct zswap_entry;
+
+/*
+ * VSWAP_ZERO and VSWAP_FOLIO are return-only values synthesized from
+ * swap_table state; the rest are stored in the vtable per slot.
+ */
+enum vswap_backing_type {
+ VSWAP_NONE = 0,
+ VSWAP_ZSWAP = 1,
+ VSWAP_SWAPFILE = 2,
+ VSWAP_ZERO,
+ VSWAP_FOLIO,
+};
+
+#ifdef CONFIG_SWAP
+
+#include "swap_table.h"
+DECLARE_STATIC_KEY_FALSE(vswap_key);
+
+/*
+ * Only true once vswap_init() has published vswap_si, so callers never
+ * see the device half built.
+ */
+static inline bool vswap_is_enabled(void)
+{
+ return static_branch_unlikely(&vswap_key);
+}
+
+static inline bool is_vswap_entry(swp_entry_t entry)
+{
+ return swap_is_vswap(__swap_entry_to_info(entry));
+}
+
+/*
+ * Rmap cache-only helpers for physical cluster Pointer-tagged entries.
+ * SWP_RMAP_CACHE_ONLY records, inline on the physical swap_table entry,
+ * that the backing vswap entry has swap_count == 0 (swap-cache-only, so
+ * reclaimable). The physical reclaim scanner reads it directly instead of
+ * chasing the rmap into the vswap layer and paying the cluster-lookup
+ * indirection.
+ *
+ * Callers hold the vswap cluster lock, not the physical one. The rmap is
+ * only touched while the vtable holds the slot as SWAPFILE, and that
+ * window is opened and closed under the vswap cluster lock, so the
+ * allocator has finished writing the entry by then.
+ */
+static inline void swap_rmap_mark_cache_only(struct swap_cluster_info *ci,
+ unsigned int off)
+{
+ atomic_long_t *table;
+
+ table = rcu_dereference_check(ci->table, true);
+ atomic_long_or(SWP_RMAP_CACHE_ONLY, &table[off]);
+}
+
+static inline void swap_rmap_clear_cache_only(struct swap_cluster_info *ci,
+ unsigned int off)
+{
+ atomic_long_t *table;
+
+ table = rcu_dereference_check(ci->table, true);
+ atomic_long_and(~SWP_RMAP_CACHE_ONLY, &table[off]);
+}
+
+/*
+ * Virtual table entry encoding for vswap clusters.
+ *
+ * Each entry in ci_dyn->virtual_table stores the backing type and
+ * pointer for a virtual swap slot. Tag in low 3 bits, payload in
+ * upper 61 bits.
+ *
+ * NONE: |----- 0000 ------|000| - no separate backend pointer
+ * ZSWAP: |--- zswap_entry* |001| - compressed in zswap (tag in low bits)
+ * SWAPFILE: |- type:5,off:56 -|010| - on a physical swapfile
+ *
+ * SWAPFILE packs swp_type in the top MAX_SWAPFILES_SHIFT bits and swp_offset in
+ * the middle VTABLE_PHYS_OFF_BITS bits, both above the tag, so the type is
+ * not shifted off the word. Pointer payloads (ZSWAP) are stored directly with
+ * the tag OR'd into the low bits (kernel pointers are >= 8-byte aligned, same
+ * approach as xarray).
+ *
+ * vtable[i] = NONE does not by itself mean "free". The swap_table entry
+ * and the per-slot zero flag carry the rest of the state. The full
+ * per-slot state table is:
+ *
+ * vtable[i] | swap_table[i] | zero | meaning
+ * ----------+---------------+-------+--------------------------------
+ * NONE | NULL | clear | truly free / unbacked
+ * NONE | PFN | clear | folio cached, no backing
+ * NONE | shadow | clear | evicted, no backing: data lost
+ * NONE | * | set | zero-backed; cached if PFN set
+ * ZSWAP | PFN | clear | folio cached + zswap entry
+ * ZSWAP | shadow / NULL | clear | evicted, only in zswap
+ * SWAPFILE | PFN | clear | folio cached + physical slot
+ * SWAPFILE | shadow / NULL | clear | evicted, only on the swapfile
+ *
+ * Locking: a slot's vtable entry (the vswap entry's backend) is only
+ * stable while the caller owns and holds the lock on that entry's swap
+ * cache folio. The cluster lock (ci_dyn->ci.lock) only makes an individual
+ * vtable read atomic, and by itself does not give the caller the right to
+ * change the backend. A backend read without the folio lock is
+ * best-effort and must be re-validated under the folio lock before
+ * being acted on.
+ *
+ * Zero-backed slots use the swap_table per-slot zero flag (same as
+ * direct-mapped physical swap), via __swap_table_test_zero() and friends,
+ * which fall back to ci->zero_bitmap where the flag does not fit. Cached
+ * folios are read out of the swap_table PFN entry; there is no separate FOLIO
+ * vtable type because the folio pointer would duplicate that PFN and
+ * would go stale on folio migration / split.
+ */
+
+#define VTABLE_TAG_BITS 3
+#define VTABLE_TAG_MASK ((1UL << VTABLE_TAG_BITS) - 1)
+
+static inline enum vswap_backing_type vtable_type(unsigned long vt)
+{
+ return vt & VTABLE_TAG_MASK;
+}
+
+/* swp_offset field width in a physical backend slot; layout described above. */
+#define VTABLE_PHYS_OFF_BITS (BITS_PER_LONG - VTABLE_TAG_BITS - MAX_SWAPFILES_SHIFT)
+
+static inline unsigned long vtable_mk_phys(swp_entry_t entry)
+{
+ VM_WARN_ON_ONCE(swp_offset(entry) >> VTABLE_PHYS_OFF_BITS);
+ return ((unsigned long)swp_type(entry) << (VTABLE_TAG_BITS + VTABLE_PHYS_OFF_BITS)) |
+ (swp_offset(entry) << VTABLE_TAG_BITS) | VSWAP_SWAPFILE;
+}
+
+static inline swp_entry_t vtable_to_phys(unsigned long vt)
+{
+ VM_WARN_ON(vtable_type(vt) != VSWAP_SWAPFILE);
+ return swp_entry(vt >> (VTABLE_TAG_BITS + VTABLE_PHYS_OFF_BITS),
+ (vt >> VTABLE_TAG_BITS) & ((1UL << VTABLE_PHYS_OFF_BITS) - 1));
+}
+
+static inline struct zswap_entry *vtable_to_zswap(unsigned long vt)
+{
+ VM_WARN_ON(vtable_type(vt) != VSWAP_ZSWAP);
+ return (struct zswap_entry *)(vt & ~VTABLE_TAG_MASK);
+}
+
+/* Virtual table accessors */
+
+static inline unsigned long __vtable_get(struct swap_cluster_info_dynamic *ci_dyn,
+ unsigned int off)
+{
+ VM_WARN_ON_ONCE(off >= SWAPFILE_CLUSTER);
+ return atomic_long_read(&ci_dyn->virtual_table[off]);
+}
+
+static inline void __vtable_set(struct swap_cluster_info_dynamic *ci_dyn,
+ unsigned int off, unsigned long vt)
+{
+ VM_WARN_ON_ONCE(off >= SWAPFILE_CLUSTER);
+ atomic_long_set(&ci_dyn->virtual_table[off], vt);
+}
+
+/**
+ * vswap_lock_cluster - look up and lock the vswap cluster for an entry
+ * @entry: the virtual swap entry
+ * @voff: out param, receives @entry's slot offset within the cluster
+ *
+ * Return: the locked vswap cluster, or NULL if @entry has no live cluster.
+ */
+static inline struct swap_cluster_info_dynamic *
+vswap_lock_cluster(swp_entry_t entry, unsigned int *voff)
+{
+ struct swap_cluster_info *ci;
+
+ ci = swap_cluster_lock(__swap_entry_to_info(entry), swp_offset(entry));
+ if (!ci)
+ return NULL;
+ *voff = swp_cluster_offset(entry);
+ return container_of(ci, struct swap_cluster_info_dynamic, ci);
+}
+
+/**
+ * vswap_to_phys - resolve a vswap entry's physical swap backing
+ * @entry: the virtual swap entry
+ *
+ * Context: takes and drops the vswap cluster lock internally.
+ * Return: the backing physical swp_entry_t, or the null entry (.val == 0)
+ * when @entry has no physical backing (NONE/ZSWAP/ZERO).
+ */
+static inline swp_entry_t vswap_to_phys(swp_entry_t entry)
+{
+ struct swap_cluster_info_dynamic *ci_dyn;
+ unsigned int voff;
+ unsigned long vt;
+
+ ci_dyn = vswap_lock_cluster(entry, &voff);
+ if (!ci_dyn)
+ return (swp_entry_t){};
+
+ vt = __vtable_get(ci_dyn, voff);
+ swap_cluster_unlock(&ci_dyn->ci);
+
+ if (vtable_type(vt) != VSWAP_SWAPFILE)
+ return (swp_entry_t){};
+
+ return vtable_to_phys(vt);
+}
+
+void __vswap_release_backing(struct swap_cluster_info *ci,
+ unsigned int ci_start, unsigned int nr);
+
+/**
+ * vswap_zswap_store - record a zswap entry as the backing for a vswap entry.
+ * @entry: the vswap entry
+ * @ze: the zswap entry now holding @entry's compressed data
+ *
+ * Releases @entry's previous backing, and sets the zswap entry @ze as the new
+ * backing.
+ *
+ * Context: takes and drops the vswap cluster lock internally.
+ */
+static inline void vswap_zswap_store(swp_entry_t entry,
+ struct zswap_entry *ze)
+{
+ struct swap_cluster_info_dynamic *ci_dyn;
+ unsigned int voff;
+
+ ci_dyn = vswap_lock_cluster(entry, &voff);
+ __vswap_release_backing(&ci_dyn->ci, voff, 1);
+ __vtable_set(ci_dyn, voff, (unsigned long)ze | VSWAP_ZSWAP);
+ swap_cluster_unlock(&ci_dyn->ci);
+}
+
+/**
+ * vswap_zswap_load - return the zswap entry backing a vswap entry
+ * @entry: the virtual swap entry
+ *
+ * Context: takes and drops the vswap cluster lock internally.
+ * Return: the backing zswap entry, or NULL if @entry is not zswap-backed.
+ */
+static inline struct zswap_entry *vswap_zswap_load(swp_entry_t entry)
+{
+ struct swap_cluster_info_dynamic *ci_dyn;
+ unsigned int voff;
+ unsigned long vt;
+
+ ci_dyn = vswap_lock_cluster(entry, &voff);
+ if (!ci_dyn)
+ return NULL;
+ vt = __vtable_get(ci_dyn, voff);
+ swap_cluster_unlock(&ci_dyn->ci);
+
+ if (vtable_type(vt) != VSWAP_ZSWAP)
+ return NULL;
+ return vtable_to_zswap(vt);
+}
+
+void folio_release_vswap_backing(struct folio *folio);
+swp_entry_t folio_realloc_swap(struct folio *folio);
+void folio_release_non_phys_swap_backing(struct folio *folio);
+
+/*
+ * Walk nr vtable slots starting at voff in ci_dyn. Returns the prefix
+ * length of slots sharing one effective backing type. For SWAPFILE,
+ * the prefix is also restricted to contiguous offsets in the same
+ * swapfile.
+ *
+ * Effective type per slot:
+ * vtable=NONE + zero flag set -> VSWAP_ZERO
+ * vtable=NONE + swap_table PFN tag -> VSWAP_FOLIO
+ * vtable=NONE + neither -> VSWAP_NONE
+ * vtable=SWAPFILE -> VSWAP_SWAPFILE
+ * vtable=ZSWAP -> VSWAP_ZSWAP
+ *
+ * *typep returns the effective type of slot 0. Caller holds
+ * ci_dyn->ci.lock.
+ */
+static inline int __vswap_check_backing(struct swap_cluster_info_dynamic *ci_dyn,
+ unsigned int voff, int nr,
+ enum vswap_backing_type *typep)
+{
+ enum vswap_backing_type first_type = VSWAP_NONE;
+ enum vswap_backing_type slot_type;
+ swp_entry_t first_phys = {};
+ unsigned long vt, swap_tb;
+ int i;
+
+ lockdep_assert_held(&ci_dyn->ci.lock);
+
+ for (i = 0; i < nr; i++) {
+ vt = __vtable_get(ci_dyn, voff + i);
+ if (vtable_type(vt) == VSWAP_NONE) {
+ swap_tb = __swap_table_get(&ci_dyn->ci, voff + i);
+ if (__swap_table_test_zero(&ci_dyn->ci, voff + i))
+ slot_type = VSWAP_ZERO;
+ else if (swp_tb_is_folio(swap_tb))
+ slot_type = VSWAP_FOLIO;
+ else
+ slot_type = VSWAP_NONE;
+ } else {
+ slot_type = vtable_type(vt);
+ }
+
+ if (!i) {
+ first_type = slot_type;
+ if (first_type == VSWAP_SWAPFILE)
+ first_phys = vtable_to_phys(vt);
+ } else if (slot_type != first_type) {
+ break;
+ } else if (first_type == VSWAP_SWAPFILE &&
+ vtable_to_phys(vt).val != first_phys.val + i) {
+ break;
+ }
+ }
+
+ if (typep)
+ *typep = first_type;
+ return i;
+}
+
+static inline int vswap_check_backing(swp_entry_t entry, int nr,
+ enum vswap_backing_type *typep)
+{
+ struct swap_cluster_info_dynamic *ci_dyn;
+ unsigned int voff;
+ int ret;
+
+ ci_dyn = vswap_lock_cluster(entry, &voff);
+ if (!ci_dyn) {
+ if (typep)
+ *typep = VSWAP_NONE;
+ return 0;
+ }
+ ret = __vswap_check_backing(ci_dyn, voff, nr, typep);
+ swap_cluster_unlock(&ci_dyn->ci);
+ return ret;
+}
+
+/**
+ * folio_phys_swap_backed - test whether a folio is backed by a contiguous
+ * range of physical swap slots.
+ * @folio: a swap-cache resident folio
+ *
+ * Return: %true if @folio->swap is not a vswap entry, or if these vswap
+ * entries are backed by a contiguous range of physical slots.
+ */
+static inline bool folio_phys_swap_backed(struct folio *folio)
+{
+ swp_entry_t entry = folio->swap;
+ int nr = folio_nr_pages(folio);
+ enum vswap_backing_type type;
+
+ return !is_vswap_entry(entry) ||
+ (vswap_check_backing(entry, nr, &type) == nr &&
+ type == VSWAP_SWAPFILE);
+}
+
+static inline int vswap_cluster_alloc_vtable(struct swap_cluster_info_dynamic *ci_dyn,
+ gfp_t gfp)
+{
+ ci_dyn->virtual_table = kcalloc(SWAPFILE_CLUSTER,
+ sizeof(*ci_dyn->virtual_table), gfp);
+ return ci_dyn->virtual_table ? 0 : -ENOMEM;
+}
+
+static inline void vswap_cluster_free_vtable(struct swap_cluster_info *ci)
+{
+ struct swap_cluster_info_dynamic *ci_dyn;
+
+ ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
+ kfree(ci_dyn->virtual_table);
+ ci_dyn->virtual_table = NULL;
+}
+
+#else /* !CONFIG_SWAP */
+
+static inline bool vswap_is_enabled(void)
+{
+ return false;
+}
+
+static inline bool is_vswap_entry(swp_entry_t entry)
+{
+ return false;
+}
+
+static inline swp_entry_t vswap_to_phys(swp_entry_t entry)
+{
+ return (swp_entry_t){};
+}
+
+static inline bool folio_phys_swap_backed(struct folio *folio)
+{
+ return true;
+}
+
+#endif /* CONFIG_SWAP */
+
+/*
+ * Test a per-backend swap flag (SWP_SYNCHRONOUS_IO, SWP_STABLE_WRITES, ...)
+ * for @entry. For a vswap entry the property belongs to the current
+ * physical backing rather than vswap_si itself; resolve to the backing
+ * and test there. Returns false for zswap/zero/unbacked vswap entries
+ * as they don't have a backing bdev.
+ */
+static inline bool swap_entry_backend_has_flag(struct swap_info_struct *si,
+ swp_entry_t entry,
+ unsigned long flag)
+{
+ struct swap_info_struct *phys_si;
+ swp_entry_t phys;
+ bool has_flag;
+
+ if (!swap_is_vswap(si))
+ return data_race(si->flags & flag);
+
+ phys = vswap_to_phys(entry);
+ if (!phys.val)
+ return false;
+
+ phys_si = get_swap_device(phys);
+ if (!phys_si)
+ return false;
+
+ has_flag = data_race(phys_si->flags & flag);
+ put_swap_device(phys_si);
+ return has_flag;
+}
+
+#endif /* _MM_VSWAP_H */
diff --git a/mm/zswap.c b/mm/zswap.c
index 37f34e406c8e3..70ad8010f18a1 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -38,6 +38,7 @@
#include <linux/zsmalloc.h>
#include "swap.h"
+#include "vswap.h"
#include "internal.h"
/*********************************
@@ -234,6 +235,25 @@ static inline struct xarray *swap_zswap_tree(swp_entry_t swp)
>> ZSWAP_ADDRESS_SPACE_SHIFT];
}
+static struct zswap_entry *zswap_entry_load(swp_entry_t swp)
+{
+ if (is_vswap_entry(swp))
+ return vswap_zswap_load(swp);
+ return xa_load(swap_zswap_tree(swp), swp_offset(swp));
+}
+
+static struct zswap_entry *zswap_entry_store(swp_entry_t swp,
+ struct zswap_entry *entry)
+{
+ if (is_vswap_entry(swp)) {
+ vswap_zswap_store(swp, entry);
+ return NULL;
+ }
+
+ return xa_store(swap_zswap_tree(swp), swp_offset(swp), entry,
+ GFP_KERNEL);
+}
+
#define zswap_pool_debug(msg, p) \
pr_debug("%s pool %s\n", msg, (p)->tfm_name)
@@ -762,7 +782,7 @@ static void zswap_entry_cache_free(struct zswap_entry *entry)
* Carries out the common pattern of freeing an entry's zsmalloc allocation,
* freeing the entry itself, and decrementing the number of stored pages.
*/
-static void zswap_entry_free(struct zswap_entry *entry)
+void zswap_entry_free(struct zswap_entry *entry)
{
zswap_lru_del(entry);
zs_free(entry->pool->zs_pool, entry->handle);
@@ -987,12 +1007,13 @@ static bool zswap_decompress(struct zswap_entry *entry, struct folio *folio)
static int zswap_writeback_entry(struct zswap_entry *entry,
swp_entry_t swpentry)
{
- struct xarray *tree;
pgoff_t offset = swp_offset(swpentry);
struct folio *folio;
struct mempolicy *mpol;
struct swap_info_struct *si;
struct swap_io_ctx ctx = {};
+ swp_entry_t phys = {};
+ bool is_vswap;
int ret = 0;
/* try to allocate swap cache folio */
@@ -1000,6 +1021,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry,
if (!si)
return -EEXIST;
+ is_vswap = swap_is_vswap(si);
mpol = get_task_policy(current);
folio = swap_cache_alloc_folio(swpentry, GFP_KERNEL, BIT(0), NULL, mpol,
NO_INTERLEAVE_INDEX);
@@ -1018,24 +1040,44 @@ static int zswap_writeback_entry(struct zswap_entry *entry,
/*
* folio is locked, and the swapcache is now secured against
* concurrent swapping to and from the slot, and concurrent
- * swapoff so we can safely dereference the zswap tree here.
+ * swapoff so we can safely dereference the zswap tree (or vswap
+ * vtable) here.
* Verify that the swap entry hasn't been invalidated and recycled
* behind our backs, to avoid overwriting a new swap folio with
* old compressed data. Only when this is successful can the entry
* be dereferenced.
*/
- tree = swap_zswap_tree(swpentry);
- if (entry != xa_load(tree, offset)) {
+ if (entry != zswap_entry_load(swpentry)) {
ret = -ENOMEM;
goto out;
}
+ if (is_vswap) {
+ /*
+ * Allocate physical backing before decompress so a failure
+ * wastes no work.
+ */
+ phys = folio_realloc_swap(folio);
+ if (!phys.val) {
+ ret = -ENOMEM;
+ goto out;
+ }
+ }
+
if (!zswap_decompress(entry, folio)) {
ret = -EIO;
+ /*
+ * The phys allocation above took the entry out of the vtable.
+ * Restore the zswap entry to the vtable, which also frees the
+ * allocated physical swap space.
+ */
+ if (is_vswap)
+ vswap_zswap_store(swpentry, entry);
goto out;
}
- xa_erase(tree, offset);
+ if (!is_vswap)
+ xa_erase(swap_zswap_tree(swpentry), offset);
count_vm_event(ZSWPWB);
if (entry->objcg)
@@ -1050,7 +1092,10 @@ static int zswap_writeback_entry(struct zswap_entry *entry,
folio_set_reclaim(folio);
/* start writeback */
- __swap_writepage(&ctx, folio);
+ if (is_vswap)
+ __swap_writepage(&ctx, folio, phys);
+ else
+ __swap_writepage(&ctx, folio, folio->swap);
swap_write_submit(&ctx);
out:
@@ -1065,6 +1110,15 @@ static int zswap_writeback_entry(struct zswap_entry *entry,
/*********************************
* shrinker functions
**********************************/
+/*
+ * vswap zswap entries get a physical slot allocated on demand at writeback
+ * time. Skip the shrinker when none is available.
+ */
+static bool zswap_writeback_possible(void)
+{
+ return !vswap_is_enabled() || get_nr_swap_pages() > 0;
+}
+
/*
* The dynamic shrinker is modulated by the following factors:
*
@@ -1202,6 +1256,9 @@ static unsigned long zswap_shrinker_count(struct shrinker *shrinker,
if (!zswap_shrinker_enabled || !mem_cgroup_zswap_writeback_enabled(memcg))
return 0;
+ if (!zswap_writeback_possible())
+ return 0;
+
/*
* The shrinker resumes swap writeback, which will enter block
* and may enter fs. XXX: Harmonize with vmscan.c __GFP_FS
@@ -1284,6 +1341,9 @@ static struct shrinker *zswap_alloc_shrinker(void)
* Return: 0 if at least one entry was written back, -EAGAIN if entries
* were scanned but none could be written back, or -ENOENT if @memcg has
* writeback disabled, is a zombie cgroup, or has empty zswap LRUs.
+ *
+ * Also returns -ENOENT when vswap is enabled and there is no physical
+ * swap to write back to.
*/
static int shrink_memcg(struct mem_cgroup *memcg)
{
@@ -1292,6 +1352,9 @@ static int shrink_memcg(struct mem_cgroup *memcg)
if (!mem_cgroup_zswap_writeback_enabled(memcg))
return -ENOENT;
+ if (!zswap_writeback_possible())
+ return -ENOENT;
+
/*
* Skip zombies because their LRUs are reparented and we would be
* reclaiming from the parent instead of the dead memcg.
@@ -1320,6 +1383,9 @@ static void shrink_worker(struct work_struct *w)
int ret, failures = 0, attempts = 0;
unsigned long thr;
+ if (!zswap_writeback_possible())
+ return;
+
/* Reclaim down to the accept threshold */
thr = zswap_accept_thr_pages();
@@ -1398,7 +1464,7 @@ static void shrink_worker(struct work_struct *w)
break;
resched:
cond_resched();
- } while (zswap_total_pages() > thr);
+ } while (zswap_total_pages() > thr && zswap_writeback_possible());
}
/*********************************
@@ -1422,9 +1488,7 @@ static bool zswap_store_page(struct page *page,
if (!zswap_compress(page, entry, pool))
goto compress_failed;
- old = xa_store(swap_zswap_tree(page_swpentry),
- swp_offset(page_swpentry),
- entry, GFP_KERNEL);
+ old = zswap_entry_store(page_swpentry, entry);
if (xa_is_err(old)) {
int err = xa_err(old);
@@ -1493,7 +1557,7 @@ bool zswap_store(struct folio *folio)
struct mem_cgroup *memcg = NULL;
struct zswap_pool *pool;
bool ret = false;
- long index;
+ long index = 0;
VM_WARN_ON_ONCE(!folio_test_locked(folio));
VM_WARN_ON_ONCE(!folio_test_swapcache(folio));
@@ -1548,13 +1612,19 @@ bool zswap_store(struct folio *folio)
if (!ret && zswap_pool_reached_full)
queue_work(shrink_wq, &zswap_shrink_work);
check_old:
+ if (ret)
+ return ret;
+
/*
* If the zswap store fails or zswap is disabled, we must invalidate
* the possibly stale entries which were previously stored at the
* offsets corresponding to each page of the folio. Otherwise,
* writeback could overwrite the new data in the swapfile.
*/
- if (!ret) {
+ if (is_vswap_entry(swp)) {
+ if (index > 0)
+ folio_release_non_phys_swap_backing(folio);
+ } else {
unsigned type = swp_type(swp);
pgoff_t offset = swp_offset(swp);
struct zswap_entry *entry;
@@ -1584,9 +1654,9 @@ bool zswap_store(struct folio *folio)
* will SIGBUS).
*
* -EINVAL: if the swapped out content was in zswap, but the page belongs
- * to a large folio, which is not supported by zswap. The folio is unlocked,
- * but NOT marked up-to-date, so that an IO error is emitted (e.g.
- * do_swap_page() will SIGBUS).
+ * to a large non-vswap folio, which is not supported by zswap. The folio
+ * is unlocked, but NOT marked up-to-date, so that an IO error is emitted
+ * (e.g. do_swap_page() will SIGBUS).
*
* -ENOENT: if the swapped out content was not in zswap. The folio remains
* locked on return.
@@ -1594,8 +1664,7 @@ bool zswap_store(struct folio *folio)
int zswap_load(struct folio *folio)
{
swp_entry_t swp = folio->swap;
- pgoff_t offset = swp_offset(swp);
- struct xarray *tree = swap_zswap_tree(swp);
+ struct swap_info_struct *si = __swap_entry_to_info(swp);
struct zswap_entry *entry;
VM_WARN_ON_ONCE(!folio_test_locked(folio));
@@ -1608,13 +1677,20 @@ int zswap_load(struct folio *folio)
* Large folios should not be swapped in while zswap is being used, as
* they are not properly handled. Zswap does not properly load large
* folios, and a large folio may only be partially in zswap.
+ *
+ * A large vswap folio cannot reach here ZSWAP-backed, since
+ * __swap_cache_add_check() refuses such a batch, so hand it to the
+ * phys path without warning.
*/
- if (WARN_ON_ONCE(folio_test_large(folio))) {
- folio_unlock(folio);
- return -EINVAL;
+ if (folio_test_large(folio)) {
+ if (WARN_ON_ONCE(!swap_is_vswap(si))) {
+ folio_unlock(folio);
+ return -EINVAL;
+ }
+ return -ENOENT;
}
- entry = xa_load(tree, offset);
+ entry = zswap_entry_load(swp);
if (!entry)
return -ENOENT;
@@ -1637,8 +1713,13 @@ int zswap_load(struct folio *folio)
* compression work.
*/
folio_mark_dirty(folio);
- xa_erase(tree, offset);
- zswap_entry_free(entry);
+
+ if (swap_is_vswap(si)) {
+ folio_release_vswap_backing(folio);
+ } else {
+ xa_erase(swap_zswap_tree(swp), swp_offset(swp));
+ zswap_entry_free(entry);
+ }
folio_unlock(folio);
return 0;
|
| 3/2 | 2026/08/25 15:55 | llm | patch-evaluator |
0mModel:gemini-3.1-pro-preview Args: null Results: null Tokens: input: 51777 output: 215 thoughts: 1532 Thoughts: **Fuzzing Strategy for the New "vswap" Feature** Okay, let's get down to brass tacks. I need to determine if this new "vswap" feature, the virtual swap layer, introduces any functional changes to the kernel that warrant a fuzzing campaign. This looks significant, touching a ton of core memory management code: `mm/swapfile.c`, `mm/zswap.c`, `mm/page_io.c`, `mm/memory.c`, `mm/memcontrol.c`, and so on. This new feature introduces a config, `CONFIG_VSWAP_DEFAULT_ON`, a new file `mm/vswap.h`, and a boot parameter `vswap=`. This all signals a substantial addition to the memory management subsystem. Reachable? Absolutely. This is deep in the heart of MM. Worth fuzzing? Without a doubt – it's a major new feature, practically screaming for attention. Now, I need to pinpoint the key functions within this new "vswap" code that are the most likely to expose vulnerabilities. The aim is not to fuzz everything, it is to focus efforts. These are going to be those that are entry points, or those which we think are likely to be incorrect. Specifically, I'm focusing on functions that interact directly with the swap and memory control. This will include: * `folio_alloc_swap` * `folio_realloc_swap` * `folio_release_vswap_backing` * `folio_release_non_phys_swap_backing` * `__vswap_release_backing` I need to make sure these are non-static. Of course, the memcg integration is important: * `__mem_cgroup_swap_charge` * `__mem_cgroup_swap_uncharge` I'm confident these are the crucial ones to probe. Next, I need to check the required configs. Besides `CONFIG_SWAP` and `64BIT`, which are implicit, I need to enable `CONFIG_ZSWAP` to test the interaction with zswap and `CONFIG_MEMCG` for memcg swap accounting. The `VSWAP_DEFAULT_ON` config in `mm/Kconfig` confirms that's the primary switch for this functionality. Now, let's solidify this for the `set-results` tool: * **EnableConfigs:** `VSWAP_DEFAULT_ON`, `ZSWAP`, `MEMCG` * **FocusSymbols:** `folio_alloc_swap`, `folio_realloc_swap`, `folio_release_vswap_backing`, `folio_release_non_phys_swap_backing`, `__vswap_release_backing`, `__mem_cgroup_swap_charge`, `__mem_cgroup_swap_uncharge` I'm confident this is the right strategy. This should provide a focused and effective fuzzing campaign to identify any vulnerabilities in this new and complex "vswap" feature. |
| 4/2 | 2026/08/25 15:56 | tool | set-results |
0mArgs:{
"EnableConfigs": [
"VSWAP_DEFAULT_ON",
"ZSWAP",
"MEMCG"
],
"FocusSymbols": [
"folio_alloc_swap",
"folio_realloc_swap",
"folio_release_vswap_backing",
"folio_release_non_phys_swap_backing",
"__vswap_release_backing",
"__mem_cgroup_swap_charge",
"__mem_cgroup_swap_uncharge",
"__mem_cgroup_swap_get",
"__mem_cgroup_swap_record",
"__mem_cgroup_swap_put"
],
"Reasoning": "The patch introduces a major new core memory management feature: the virtual swap layer (vswap). It modifies core swap allocation, zswap interaction, and memcg swap accounting. The changes are highly reachable and complex, making them a prime target for fuzzing.",
"WorthFuzzing": true
}Results: {
"EnableConfigs": [
"VSWAP_DEFAULT_ON",
"ZSWAP",
"MEMCG"
],
"FocusSymbols": [
"folio_alloc_swap",
"folio_realloc_swap",
"folio_release_vswap_backing",
"folio_release_non_phys_swap_backing",
"__vswap_release_backing",
"__mem_cgroup_swap_charge",
"__mem_cgroup_swap_uncharge",
"__mem_cgroup_swap_get",
"__mem_cgroup_swap_record",
"__mem_cgroup_swap_put"
],
"Reasoning": "The patch introduces a major new core memory management feature: the virtual swap layer (vswap). It modifies core swap allocation, zswap interaction, and memcg swap accounting. The changes are highly reachable and complex, making them a prime target for fuzzing.",
"WorthFuzzing": true
} |
| 5/1 | 2026/08/25 15:56 | action | If |
1mArgs:{
"WorthFuzzing": true
}Results: null |
| 6/2 | 2026/08/25 15:56 | agent | kmsan-evaluator |
1mModel:best-expensive Args: null Results: {
"KMSANReasoning": "The patch introduces a virtual swap layer (`vswap`) to decouple swap entries from physical swap slots, allowing better integration with `zswap` and zero-filled pages. The changes involve complex memory management logic, including dynamic allocation of swap clusters, virtual-to-physical mapping tables, and modifications to the swap read/write paths.\n\nAll new data structures (e.g., `struct swap_cluster_info_dynamic`, virtual tables, swap tables) are allocated using zeroing allocators (`kzalloc_obj`, `kcalloc`, `folio_alloc` with `__GFP_ZERO`). The patch does not introduce new kernel-to-user interfaces that copy structures to user space, nor does it add complex parsing of user-provided data where uninitialized fields could be used.\n\nIf a virtual swap entry lacks physical backing during a read fault, the code correctly handles it by returning an error (`-EIO`) without marking the folio as up-to-date, preventing the exposure of uninitialized page memory to user space (as explicitly noted in the `try_to_unuse` comments).\n\nAny potential bugs introduced by this patch (e.g., incorrect reference counting, race conditions in cluster management, or incorrect physical slot resolution) would manifest as use-after-free, out-of-bounds accesses, or memory corruption, which are effectively detected by KASAN, LOCKDEP, and standard kernel testing. Therefore, a dedicated KMSAN fuzzing session is not justified.",
"NeedsKMSAN": false
}Instruction: You are an expert Linux kernel security engineer specializing in kernel memory
error detectors (KASAN and KMSAN). Your job is to review the provided patch series and
determine if the code changes justify spawning a dedicated KMSAN (KernelMemorySanitizer)
fuzzing session in addition to standard KASAN fuzzing.
CRITICAL DISTINCTION BETWEEN KASAN AND KMSAN:
- Standard KASAN kernel builds (upstream-apparmor-kasan.config) already enable
a comprehensive suite of debugging tools and sanitizers, including KASAN
(out-of-bounds accesses, use-after-free, double free, invalid free), LOCKDEP
(locking bugs and deadlocks), UB-sanitizers, and memory corruption checks.
- KMSAN (KernelMemorySanitizer) detects reads of UNINITIALIZED memory (stack, heap,
or page allocations) and kernel-to-user memory info-leaks.
Rule: THERE IS NO SENSE IN RUNNING A KMSAN SESSION IF A BUG CAN BE CAUGHT BY KASAN,
LOCKDEP, OR OTHER STANDARD BUG DETECTORS.
A dedicated KMSAN fuzzing session incurs significant resource costs. You must ONLY
set NeedsKMSAN=true if the code changes introduce or expose UNINITIALIZED MEMORY risks
that are detected ONLY by KMSAN.
Look holistically at the patch series and surrounding code. Even if no direct
uninitialized field accesses or new buffer allocations are added in the diff itself,
a patch may alter control flow, bounds checking, or data length calculations in ways
that change how the rest of the code operates on existing buffers (e.g. allowing
uninitialized stack/heap memory to be read, copied to user space, or used in control
flow). Do not hesitate to use your code access tools to inspect the surrounding code,
called functions, and callers.
Set NeedsKMSAN=true ONLY IF the patch introduces or modifies:
1. Kernel structures sent to user space (via copy_to_user, put_user, netlink skb
attributes, ioctl output arguments, socket options, or BPF buffers) where fields
or structure padding might not be fully initialized/zeroed.
2. Conditional logic or branching that depends on potentially uninitialized variables
or struct fields.
3. Allocation or initialization of complex data structures where uninitialized fields
could be read later in reachable code paths.
4. Bounds checks, lengths, or logic in a way that allows surrounding code to access
uninitialized bytes of existing buffers.
Set NeedsKMSAN=false IF:
- The code changes primarily risk out-of-bounds access, array overflows, NULL pointer
dereferences, locking deadlocks, or use-after-free bugs (these are already caught
by KASAN, LOCKDEP, or standard bug detectors).
- All stack/heap structures touched or introduced by the patch are fully zeroed
or initialized (e.g. using = {0}, memset, kzalloc) before being read or copied.
- The patch does not introduce any risk of uninitialized memory usage or info-leaks.
Use your code access tools to inspect the surrounding code if necessary, then provide
detailed KMSANReasoning contrasting KASAN vs KMSAN applicability for this patch.
Prefer calling several tools at the same time to save round-trips.
Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.
Prompt: For your convenience, here is the diff of the changes:
commit 90158fb09c94f90a0fb0c502bca752b81303fd9d
Author: syz-cluster <triage@syzkaller.com>
Date: Tue Aug 25 15:55:55 2026 +0000
syz-cluster: applied patch under review
diff --git a/Documentation/admin-guide/cgroup-v1/memcg_test.rst b/Documentation/admin-guide/cgroup-v1/memcg_test.rst
index ebedbc3c3f9cc..13b9ae800b728 100644
--- a/Documentation/admin-guide/cgroup-v1/memcg_test.rst
+++ b/Documentation/admin-guide/cgroup-v1/memcg_test.rst
@@ -43,7 +43,7 @@ Please note that implementation details can be changed.
mem_cgroup_uncharge()
Called when a page's refcount goes down to 0.
- mem_cgroup_uncharge_swap()
+ mem_cgroup_swap_uncharge()
Called when swp_entry's refcnt goes down to 0. A charge against swap
disappears.
diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 1af62cd16c9de..6612b5e0a0554 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -8364,6 +8364,13 @@ Kernel parameters
force - force vulnerability detection even on
unaffected processors
+ vswap= [MM,EARLY]
+ Route swapouts through the virtual swap layer, which
+ allows zswap and zero-filled pages to be used without
+ a physical swap device. 64-bit only.
+ Format: { on | off }
+ Default: on if CONFIG_VSWAP_DEFAULT_ON=y, else off.
+
vsyscall= [X86-64,EARLY]
Controls the behavior of vsyscalls (i.e. calls to
fixed addresses of 0xffffffffff600x00 from legacy
diff --git a/MAINTAINERS b/MAINTAINERS
index 29236523cefb6..8a5827d271778 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -17251,6 +17251,7 @@ F: mm/swap.h
F: mm/swap_table.h
F: mm/swap_state.c
F: mm/swapfile.c
+F: mm/vswap.h
MEMORY MANAGEMENT - THP (TRANSPARENT HUGE PAGE)
M: Andrew Morton <akpm@linux-foundation.org>
diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index 215e2e87f42b2..4d89a35f49ff4 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -1905,6 +1905,7 @@ static inline bool memcg_is_dying(struct mem_cgroup *memcg)
#if defined(CONFIG_MEMCG) && defined(CONFIG_ZSWAP)
bool obj_cgroup_may_zswap(struct obj_cgroup *objcg);
+bool mem_cgroup_may_zswap(struct mem_cgroup *memcg, bool may_flush);
void obj_cgroup_charge_zswap(struct obj_cgroup *objcg, size_t size);
void obj_cgroup_uncharge_zswap(struct obj_cgroup *objcg, size_t size);
bool mem_cgroup_zswap_writeback_enabled(struct mem_cgroup *memcg);
@@ -1913,6 +1914,11 @@ static inline bool obj_cgroup_may_zswap(struct obj_cgroup *objcg)
{
return true;
}
+
+static inline bool mem_cgroup_may_zswap(struct mem_cgroup *memcg, bool may_flush)
+{
+ return true;
+}
static inline void obj_cgroup_charge_zswap(struct obj_cgroup *objcg,
size_t size)
{
diff --git a/include/linux/swap.h b/include/linux/swap.h
index 5658a1634b85e..5310ac0f3faae 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -207,6 +207,7 @@ enum {
SWP_STABLE_WRITES = (1 << 11), /* no overwrite PG_writeback pages */
SWP_SYNCHRONOUS_IO = (1 << 12), /* synchronous IO is efficient */
SWP_HIBERNATION = (1 << 13), /* pinned for hibernation */
+ SWP_VSWAP = (1 << 14), /* virtual swap device */
/* add others here before... */
};
@@ -245,7 +246,7 @@ struct swap_info_struct {
signed short prio; /* swap priority of this type */
struct plist_node list; /* entry in swap_active_head */
signed char type; /* strange name for an index */
- unsigned int max; /* size of this swap device */
+ unsigned long max; /* size of this swap device */
struct swap_cluster_info *cluster_info; /* cluster info. Only for SSD */
struct list_head free_clusters; /* free clusters list */
struct list_head full_clusters; /* full clusters list */
@@ -253,7 +254,7 @@ struct swap_info_struct {
/* list of cluster that contains at least one free slot */
struct list_head frag_clusters[SWAP_NR_ORDERS];
/* list of cluster that are fragmented or contented */
- unsigned int pages; /* total of usable pages of swap */
+ unsigned long pages; /* total of usable pages of swap */
atomic_long_t inuse_pages; /* number of those currently in use */
struct swap_sequential_cluster *global_cluster; /* Use one global cluster for rotating device */
spinlock_t global_cluster_lock; /* Serialize usage of global cluster */
@@ -276,8 +277,14 @@ struct swap_info_struct {
struct list_head discard_clusters; /* discard clusters list */
struct plist_node avail_list; /* entry in swap_avail_head */
const struct swap_ops *ops;
+ struct xarray cluster_info_pool; /* Xarray for vswap dynamic cluster info */
};
+static inline bool swap_is_vswap(struct swap_info_struct *si)
+{
+ return si->flags & SWP_VSWAP;
+}
+
static inline swp_entry_t page_swap_entry(struct page *page)
{
struct folio *folio = page_folio(page);
@@ -381,7 +388,7 @@ extern int __swap_count(swp_entry_t entry);
extern bool swap_entry_swapped(struct swap_info_struct *si, swp_entry_t entry);
extern int swp_swapcount(swp_entry_t entry);
extern struct swap_info_struct *get_swap_device(swp_entry_t entry);
-sector_t swap_folio_sector(struct folio *folio);
+sector_t swap_entry_sector(swp_entry_t entry);
/*
* If there is an existing swap slot reference (swap entry) and the caller
@@ -408,6 +415,8 @@ void swap_free_hibernation_slot(swp_entry_t entry);
static inline void put_swap_device(struct swap_info_struct *si)
{
+ if (swap_is_vswap(si))
+ return;
percpu_ref_put(&si->users);
}
@@ -492,35 +501,80 @@ static inline void folio_throttle_swaprate(struct folio *folio, gfp_t gfp)
#endif
#if defined(CONFIG_MEMCG) && defined(CONFIG_SWAP)
-int __mem_cgroup_try_charge_swap(struct folio *folio);
-static inline int mem_cgroup_try_charge_swap(struct folio *folio)
+struct mem_cgroup *__mem_cgroup_swap_get(struct folio *folio);
+static inline struct mem_cgroup *mem_cgroup_swap_get(struct folio *folio)
+{
+ if (mem_cgroup_disabled())
+ return NULL;
+ return __mem_cgroup_swap_get(folio);
+}
+
+int __mem_cgroup_swap_charge(struct mem_cgroup *memcg, unsigned int nr_pages);
+static inline int mem_cgroup_swap_charge(struct mem_cgroup *memcg,
+ unsigned int nr_pages)
{
if (mem_cgroup_disabled())
return 0;
- return __mem_cgroup_try_charge_swap(folio);
+ return __mem_cgroup_swap_charge(memcg, nr_pages);
}
-extern void __mem_cgroup_uncharge_swap(unsigned short id, unsigned int nr_pages);
-static inline void mem_cgroup_uncharge_swap(unsigned short id, unsigned int nr_pages)
+void __mem_cgroup_swap_record(struct folio *folio, struct mem_cgroup *memcg);
+static inline void mem_cgroup_swap_record(struct folio *folio,
+ struct mem_cgroup *memcg)
{
if (mem_cgroup_disabled())
return;
- __mem_cgroup_uncharge_swap(id, nr_pages);
+ __mem_cgroup_swap_record(folio, memcg);
+}
+
+void __mem_cgroup_swap_uncharge(struct mem_cgroup *memcg,
+ unsigned int nr_pages);
+static inline void mem_cgroup_swap_uncharge(struct mem_cgroup *memcg,
+ unsigned int nr_pages)
+{
+ if (mem_cgroup_disabled())
+ return;
+ __mem_cgroup_swap_uncharge(memcg, nr_pages);
+}
+
+void __mem_cgroup_swap_put(struct mem_cgroup *memcg, unsigned int nr_pages);
+static inline void mem_cgroup_swap_put(struct mem_cgroup *memcg,
+ unsigned int nr_pages)
+{
+ if (mem_cgroup_disabled())
+ return;
+ __mem_cgroup_swap_put(memcg, nr_pages);
}
extern long mem_cgroup_get_nr_swap_pages(struct mem_cgroup *memcg);
extern bool mem_cgroup_swap_full(struct folio *folio);
#else
-static inline int mem_cgroup_try_charge_swap(struct folio *folio)
+static inline struct mem_cgroup *mem_cgroup_swap_get(struct folio *folio)
+{
+ return NULL;
+}
+
+static inline int mem_cgroup_swap_charge(struct mem_cgroup *memcg,
+ unsigned int nr_pages)
{
return 0;
}
-static inline void mem_cgroup_uncharge_swap(unsigned short id,
+static inline void mem_cgroup_swap_record(struct folio *folio,
+ struct mem_cgroup *memcg)
+{
+}
+
+static inline void mem_cgroup_swap_uncharge(struct mem_cgroup *memcg,
unsigned int nr_pages)
{
}
+static inline void mem_cgroup_swap_put(struct mem_cgroup *memcg,
+ unsigned int nr_pages)
+{
+}
+
static inline long mem_cgroup_get_nr_swap_pages(struct mem_cgroup *memcg)
{
return get_nr_swap_pages();
diff --git a/include/linux/swap_ops.h b/include/linux/swap_ops.h
index 57ac6c703f685..223c84548bde6 100644
--- a/include/linux/swap_ops.h
+++ b/include/linux/swap_ops.h
@@ -12,6 +12,7 @@ struct swap_iocb {
struct bio_vec bvecs[SWAP_CLUSTER_MAX];
int nr_bvecs;
int len;
+ swp_entry_t entry; /* first slot in the batch; addresses the IO */
};
struct swap_io_ctx {
@@ -30,15 +31,15 @@ struct swap_io_ctx {
struct swap_ops {
unsigned int flags;
- bool (*can_merge)(struct folio *folio, struct folio *prev_folio,
- size_t prev_folio_size, int rw);
+ bool (*can_merge)(struct folio *folio, swp_entry_t phys,
+ struct swap_iocb *sio, int rw);
void (*submit_write)(struct swap_io_ctx *ctx);
void (*submit_read)(struct swap_io_ctx *ctx);
};
void swap_fs_prepare_rw(struct swap_io_ctx *ctx, int rw, struct iov_iter *iter);
-bool swap_fs_can_merge(struct folio *folio, struct folio *prev_folio,
- size_t prev_folio_size, int rw);
+bool swap_fs_can_merge(struct folio *folio, swp_entry_t phys,
+ struct swap_iocb *sio, int rw);
int swap_fs_activate(struct swap_info_struct *sis, const struct swap_ops *ops);
#endif /* _MM_SWAP_OPS_H */
diff --git a/include/linux/zswap.h b/include/linux/zswap.h
index 30c193a1207e1..4b4f211f33017 100644
--- a/include/linux/zswap.h
+++ b/include/linux/zswap.h
@@ -6,6 +6,7 @@
#include <linux/mm_types.h>
struct lruvec;
+struct zswap_entry;
extern atomic_long_t zswap_stored_pages;
@@ -28,6 +29,7 @@ unsigned long zswap_total_pages(void);
bool zswap_store(struct folio *folio);
int zswap_load(struct folio *folio);
void zswap_invalidate(swp_entry_t swp);
+void zswap_entry_free(struct zswap_entry *entry);
int zswap_swapon(int type, unsigned long nr_pages);
void zswap_swapoff(int type);
void zswap_memcg_offline_cleanup(struct mem_cgroup *memcg);
@@ -50,6 +52,7 @@ static inline int zswap_load(struct folio *folio)
}
static inline void zswap_invalidate(swp_entry_t swp) {}
+static inline void zswap_entry_free(struct zswap_entry *entry) {}
static inline int zswap_swapon(int type, unsigned long nr_pages)
{
return 0;
diff --git a/mm/Kconfig b/mm/Kconfig
index 604c58199acbf..08fdc7502c1de 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -19,6 +19,26 @@ menuconfig SWAP
used to provide more virtual memory than the actual RAM present
in your computer. If unsure say Y.
+config VSWAP_DEFAULT_ON
+ bool "Route swapouts through virtual swap by default"
+ depends on SWAP && 64BIT
+ default n
+ help
+ Virtual swap allows zswap and zero-filled pages to be used
+ without swapping on a physical device first, and lets a page
+ move between zswap and a swapfile without invalidating the page
+ table entries that refer to it.
+
+ Swap entries are handed out by a virtual swap device instead of
+ naming a slot on a real one, so the backing can be chosen and
+ changed after the entry exists.
+
+ Say Y to make "vswap=on" the default, routing swapouts through
+ the virtual swap layer from boot.
+
+ Say N (default) to leave vswap off unless "vswap=on" is passed
+ on the kernel command line.
+
config ZSWAP
bool "Compressed cache for swap pages"
depends on SWAP
diff --git a/mm/memcontrol-v1.c b/mm/memcontrol-v1.c
index 05ef55cae4dc6..88016c8f22a26 100644
--- a/mm/memcontrol-v1.c
+++ b/mm/memcontrol-v1.c
@@ -690,6 +690,7 @@ void __memcg1_swapout(struct folio *folio, struct swap_cluster_info *ci)
void memcg1_swapin(struct folio *folio)
{
struct swap_cluster_info *ci;
+ struct mem_cgroup *memcg;
unsigned long nr_pages;
unsigned short id;
@@ -721,7 +722,14 @@ void memcg1_swapin(struct folio *folio)
id = __swap_cgroup_clear(ci, swp_cluster_offset(folio->swap),
nr_pages);
swap_cluster_unlock(ci);
- mem_cgroup_uncharge_swap(id, nr_pages);
+
+ rcu_read_lock();
+ memcg = mem_cgroup_from_private_id(id);
+ if (memcg) {
+ mem_cgroup_swap_uncharge(memcg, nr_pages);
+ mem_cgroup_swap_put(memcg, nr_pages);
+ }
+ rcu_read_unlock();
}
#endif
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 11b85f4b6828b..6ae0a4191d887 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -65,6 +65,7 @@
#include "internal.h"
#include "swap.h"
#include "swap_table.h"
+#include "vswap.h"
#include <net/sock.h>
#include <net/ip.h>
#include "slab.h"
@@ -5721,86 +5722,129 @@ int __init mem_cgroup_init(void)
#ifdef CONFIG_SWAP
/**
- * __mem_cgroup_try_charge_swap - try charging swap space for a folio
+ * __mem_cgroup_swap_get - pin the memcg to account a folio's swap slots to
* @folio: folio being added to swap
*
- * Try to charge @folio's memcg for the swap space at folio->swap.
+ * Pins one private ID ref per page of @folio on its memcg, or on its closest
+ * online ancestor if it has been offlined. The caller charges and records
+ * against whichever memcg is returned, so both land on the same one.
*
- * Returns 0 on success, -ENOMEM on failure.
+ * Return: the pinned memcg, or NULL if there is nothing to account. Drop the
+ * pins with __mem_cgroup_swap_put().
*/
-int __mem_cgroup_try_charge_swap(struct folio *folio)
+struct mem_cgroup *__mem_cgroup_swap_get(struct folio *folio)
{
unsigned int nr_pages = folio_nr_pages(folio);
- struct swap_cluster_info *ci;
- struct page_counter *counter;
struct mem_cgroup *memcg;
struct obj_cgroup *objcg;
if (do_memsw_account())
- return 0;
+ return NULL;
objcg = folio_objcg(folio);
VM_WARN_ON_ONCE_FOLIO(!objcg, folio);
if (!objcg)
- return 0;
+ return NULL;
rcu_read_lock();
memcg = obj_cgroup_memcg(objcg);
if (!folio_test_swapcache(folio)) {
memcg_memory_event(memcg, MEMCG_SWAP_FAIL);
rcu_read_unlock();
- return 0;
+ return NULL;
}
memcg = mem_cgroup_private_id_get_online(memcg, nr_pages);
/* memcg is pined by memcg ID. */
rcu_read_unlock();
+ return memcg;
+}
+
+/**
+ * __mem_cgroup_swap_charge - charge physical swap space
+ * @memcg: the mem_cgroup to charge (may be NULL)
+ * @nr_pages: the amount of swap space to charge
+ *
+ * Return: 0 on success, -ENOMEM if memory.swap.max is exceeded.
+ */
+int __mem_cgroup_swap_charge(struct mem_cgroup *memcg, unsigned int nr_pages)
+{
+ struct page_counter *counter;
+
+ if (do_memsw_account() || !memcg)
+ return 0;
+
if (!mem_cgroup_is_root(memcg) &&
!page_counter_try_charge(&memcg->swap, nr_pages, &counter)) {
memcg_memory_event(memcg, MEMCG_SWAP_MAX);
memcg_memory_event(memcg, MEMCG_SWAP_FAIL);
- mem_cgroup_private_id_put(memcg, nr_pages);
return -ENOMEM;
}
mod_memcg_state(memcg, MEMCG_SWAP, nr_pages);
+ return 0;
+}
+
+/**
+ * __mem_cgroup_swap_record - record the owner of a folio's swap slots
+ * @folio: folio being added to swap
+ * @memcg: the memcg pinned by __mem_cgroup_swap_get()
+ */
+void __mem_cgroup_swap_record(struct folio *folio, struct mem_cgroup *memcg)
+{
+ struct swap_cluster_info *ci;
ci = swap_cluster_get_and_lock(folio);
- __swap_cgroup_set(ci, swp_cluster_offset(folio->swap), nr_pages,
- mem_cgroup_private_id(memcg));
+ __swap_cgroup_set(ci, swp_cluster_offset(folio->swap),
+ folio_nr_pages(folio), mem_cgroup_private_id(memcg));
swap_cluster_unlock(ci);
-
- return 0;
}
/**
- * __mem_cgroup_uncharge_swap - uncharge swap space
- * @id: cgroup id to uncharge
+ * __mem_cgroup_swap_uncharge - uncharge physical swap space
+ * @memcg: the mem_cgroup to uncharge (may be NULL)
* @nr_pages: the amount of swap space to uncharge
*/
-void __mem_cgroup_uncharge_swap(unsigned short id, unsigned int nr_pages)
+void __mem_cgroup_swap_uncharge(struct mem_cgroup *memcg, unsigned int nr_pages)
{
- struct mem_cgroup *memcg;
+ if (!memcg)
+ return;
- rcu_read_lock();
- memcg = mem_cgroup_from_private_id(id);
- if (memcg) {
- if (!mem_cgroup_is_root(memcg)) {
- if (do_memsw_account())
- page_counter_uncharge(&memcg->memsw, nr_pages);
- else
- page_counter_uncharge(&memcg->swap, nr_pages);
- }
- mod_memcg_state(memcg, MEMCG_SWAP, -nr_pages);
- mem_cgroup_private_id_put(memcg, nr_pages);
+ if (!mem_cgroup_is_root(memcg)) {
+ if (do_memsw_account())
+ page_counter_uncharge(&memcg->memsw, nr_pages);
+ else
+ page_counter_uncharge(&memcg->swap, nr_pages);
}
- rcu_read_unlock();
+ mod_memcg_state(memcg, MEMCG_SWAP, -nr_pages);
+}
+
+/**
+ * __mem_cgroup_swap_put - drop the private ID refs taken for swap slots
+ * @memcg: the pinned mem_cgroup
+ * @nr_pages: number of refs to drop
+ */
+void __mem_cgroup_swap_put(struct mem_cgroup *memcg, unsigned int nr_pages)
+{
+ mem_cgroup_private_id_put(memcg, nr_pages);
}
long mem_cgroup_get_nr_swap_pages(struct mem_cgroup *memcg)
{
- long nr_swap_pages = get_nr_swap_pages();
+ long nr_swap_pages;
+
+ /*
+ * vswap charges physical backing, not allocation, so virtual swap is
+ * unbounded for a zswap-capable memcg and the swap.max walk below
+ * would starve anon reclaim. swap.max is still enforced when the
+ * backing is charged.
+ */
+ if (vswap_is_enabled() && zswap_is_enabled() &&
+ (mem_cgroup_disabled() || do_memsw_account() ||
+ mem_cgroup_may_zswap(memcg, false)))
+ return PAGE_COUNTER_MAX;
+ nr_swap_pages = get_nr_swap_pages();
if (mem_cgroup_disabled() || do_memsw_account())
return nr_swap_pages;
for (; !mem_cgroup_is_root(memcg); memcg = parent_mem_cgroup(memcg))
@@ -5972,8 +6016,10 @@ static struct cftype swap_files[] = {
#ifdef CONFIG_ZSWAP
/**
- * obj_cgroup_may_zswap - check if this cgroup can zswap
- * @objcg: the object cgroup
+ * mem_cgroup_may_zswap - check if this cgroup can zswap
+ * @memcg: the memcg to query
+ * @may_flush: force-flush stats for an accurate check (sleeps). Pass false
+ * from atomic contexts; the check is then best-effort.
*
* Check if the hierarchical zswap limit has been reached.
*
@@ -5983,36 +6029,38 @@ static struct cftype swap_files[] = {
* spending cycles on compression when there is already no room left
* or zswap is disabled altogether somewhere in the hierarchy.
*/
-bool obj_cgroup_may_zswap(struct obj_cgroup *objcg)
+bool mem_cgroup_may_zswap(struct mem_cgroup *memcg, bool may_flush)
{
- struct mem_cgroup *memcg, *original_memcg;
- bool ret = true;
-
if (!cgroup_subsys_on_dfl(memory_cgrp_subsys))
return true;
- original_memcg = get_mem_cgroup_from_objcg(objcg);
- for (memcg = original_memcg; !mem_cgroup_is_root(memcg);
- memcg = parent_mem_cgroup(memcg)) {
+ for (; !mem_cgroup_is_root(memcg); memcg = parent_mem_cgroup(memcg)) {
unsigned long max = READ_ONCE(memcg->zswap_max);
unsigned long pages;
if (max == PAGE_COUNTER_MAX)
continue;
- if (max == 0) {
- ret = false;
- break;
- }
+ if (max == 0)
+ return false;
/* Force flush to get accurate stats for charging */
- __mem_cgroup_flush_stats(memcg, true);
+ if (may_flush)
+ __mem_cgroup_flush_stats(memcg, true);
pages = memcg_page_state(memcg, MEMCG_ZSWAP_B) / PAGE_SIZE;
- if (pages < max)
- continue;
- ret = false;
- break;
+ if (pages >= max)
+ return false;
}
- mem_cgroup_put(original_memcg);
+ return true;
+}
+
+bool obj_cgroup_may_zswap(struct obj_cgroup *objcg)
+{
+ struct mem_cgroup *memcg;
+ bool ret;
+
+ memcg = get_mem_cgroup_from_objcg(objcg);
+ ret = mem_cgroup_may_zswap(memcg, true);
+ mem_cgroup_put(memcg);
return ret;
}
diff --git a/mm/memory.c b/mm/memory.c
index c549433025532..62f7b82427e21 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -89,6 +89,7 @@
#include "pgalloc-track.h"
#include "internal.h"
#include "swap.h"
+#include "vswap.h"
#if defined(LAST_CPUPID_NOT_IN_PAGE_FLAGS) && !defined(CONFIG_COMPILE_TEST)
#warning Unfortunate NUMA and NUMA Balancing config, growing page-frame for last_cpupid.
@@ -4659,7 +4660,13 @@ static inline bool should_try_to_free_swap(struct swap_info_struct *si,
* are fast, and meanwhile, swap cache pinning the slot deferring the
* release of metadata or fragmentation is a more critical issue.
*/
- if (data_race(si->flags & SWP_SYNCHRONOUS_IO))
+ if (swap_entry_backend_has_flag(si, folio->swap, SWP_SYNCHRONOUS_IO))
+ return true;
+ /*
+ * Non-swapfile backends cannot be reused for future swapouts.
+ * Free the swap slot unless backed by contiguous physical swap.
+ */
+ if (!folio_phys_swap_backed(folio))
return true;
if (mem_cgroup_swap_full(folio) || (vma->vm_flags & VM_LOCKED) ||
folio_test_mlocked(folio))
@@ -4809,15 +4816,19 @@ static unsigned long thp_swapin_suitable_orders(struct vm_fault *vmf)
if (unlikely(userfaultfd_armed(vma)))
return 0;
+ entry = softleaf_from_pte(vmf->orig_pte);
+
/*
* A large swapped out folio could be partially or fully in zswap. We
* lack handling for such cases, so fallback to swapping in order-0
* folio.
+ *
+ * Vswap entries are checked later, under the cluster lock in
+ * __swap_cache_add_check().
*/
- if (!zswap_never_enabled())
+ if (!is_vswap_entry(entry) && !zswap_never_enabled())
return 0;
- entry = softleaf_from_pte(vmf->orig_pte);
/*
* Get a list of all the (large) orders below PMD_ORDER that are enabled
* and suitable for swapping THP.
@@ -4963,7 +4974,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
swap_update_readahead(folio, vma, vmf->address);
if (!folio) {
/* Swapin bypasses readahead for SWP_SYNCHRONOUS_IO devices */
- if (data_race(si->flags & SWP_SYNCHRONOUS_IO))
+ if (swap_entry_backend_has_flag(si, entry, SWP_SYNCHRONOUS_IO))
folio = swapin_sync(entry, GFP_HIGHUSER_MOVABLE,
thp_swapin_suitable_orders(vmf) | BIT(0),
vmf, NULL, 0);
@@ -5128,7 +5139,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
*/
exclusive = true;
} else if (exclusive && folio_test_writeback(folio) &&
- data_race(si->flags & SWP_STABLE_WRITES)) {
+ swap_entry_backend_has_flag(si, entry, SWP_STABLE_WRITES)) {
/*
* This is tricky: not all swap backends support
* concurrent page modifications while under writeback.
diff --git a/mm/page_io.c b/mm/page_io.c
index 88962571cb931..b36a898358c64 100644
--- a/mm/page_io.c
+++ b/mm/page_io.c
@@ -28,6 +28,7 @@
#include <linux/swap_ops.h>
#include "swap.h"
#include "swap_table.h"
+#include "vswap.h"
int generic_swapfile_activate(struct swap_info_struct *sis,
struct file *swap_file,
@@ -160,14 +161,19 @@ static void swap_zeromap_folio_set(struct folio *folio)
struct obj_cgroup *objcg = get_obj_cgroup_from_folio(folio);
int nr_pages = folio_nr_pages(folio);
struct swap_cluster_info *ci;
+ unsigned int voff, i;
swp_entry_t entry;
- unsigned int i;
VM_WARN_ON_ONCE_FOLIO(!folio_test_swapcache(folio), folio);
VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
ci = swap_cluster_get_and_lock(folio);
- for (i = 0; i < folio_nr_pages(folio); i++) {
+ if (is_vswap_entry(folio->swap)) {
+ /* Free any prior backing (e.g. ZSWAP entry from earlier swapout) */
+ voff = swp_cluster_offset(folio->swap);
+ __vswap_release_backing(ci, voff, nr_pages);
+ }
+ for (i = 0; i < nr_pages; i++) {
entry = page_swap_entry(folio_page(folio, i));
__swap_table_set_zero(ci, swp_cluster_offset(entry));
}
@@ -203,6 +209,7 @@ static void swap_zeromap_folio_clear(struct folio *folio)
*/
int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio)
{
+ swp_entry_t phys;
int ret = 0;
if (folio_free_swap(folio))
@@ -235,6 +242,15 @@ int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio)
*/
swap_zeromap_folio_clear(folio);
+ /*
+ * For vswap: release stale non-swapfile backings (e.g. ZSWAP from a
+ * previous swapout cycle) so zswap_store or folio_realloc_swap
+ * starts on clean slots. Contiguous PHYS backing is preserved for
+ * reuse by folio_realloc_swap.
+ */
+ if (is_vswap_entry(folio->swap))
+ folio_release_non_phys_swap_backing(folio);
+
if (zswap_store(folio)) {
count_mthp_stat(folio_order(folio), MTHP_STAT_ZSWPOUT);
goto out_unlock;
@@ -248,7 +264,23 @@ int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio)
}
rcu_read_unlock();
- __swap_writepage(ctx, folio);
+ /*
+ * A vswap folio with no backend needs a physical slot to write to.
+ * zswap_store rolled back any partial vtable state on failure, so
+ * PHYS backing from a prior cycle is still there to reuse. If none
+ * is free, keep it dirty.
+ */
+ if (is_vswap_entry(folio->swap)) {
+ phys = folio_realloc_swap(folio);
+ if (!phys.val) {
+ folio_mark_dirty(folio);
+ return AOP_WRITEPAGE_ACTIVATE;
+ }
+ __swap_writepage(ctx, folio, phys);
+ return 0;
+ }
+
+ __swap_writepage(ctx, folio, folio->swap);
return 0;
out_unlock:
folio_unlock(folio);
@@ -317,24 +349,22 @@ int sio_pool_init(void)
}
static bool swap_can_merge(struct swap_io_ctx *ctx, struct folio *folio,
- int rw)
+ swp_entry_t phys, int rw)
{
- struct swap_info_struct *sis = __swap_entry_to_info(folio->swap);
- struct bio_vec *last_bv = &ctx->sio->bvecs[ctx->sio->nr_bvecs - 1];
- struct folio *prev_folio = bvec_folio(last_bv);
- size_t prev_folio_size = folio_size(prev_folio);
+ struct swap_info_struct *sis = __swap_entry_to_info(phys);
if (ctx->sis != sis)
return false;
- return sis->ops->can_merge(folio, prev_folio, prev_folio_size, rw);
+ return sis->ops->can_merge(folio, phys, ctx->sio, rw);
}
-static void swap_add_folio(struct swap_io_ctx *ctx, struct folio *folio, int rw)
+static void swap_add_folio(struct swap_io_ctx *ctx, struct folio *folio,
+ swp_entry_t phys, int rw)
{
- struct swap_info_struct *sis = __swap_entry_to_info(folio->swap);
+ struct swap_info_struct *sis = __swap_entry_to_info(phys);
struct swap_iocb *sio = ctx->sio;
- if (sio && !swap_can_merge(ctx, folio, rw)) {
+ if (sio && !swap_can_merge(ctx, folio, phys, rw)) {
if (rw == WRITE)
swap_write_submit(ctx);
else
@@ -347,6 +377,7 @@ static void swap_add_folio(struct swap_io_ctx *ctx, struct folio *folio, int rw)
ctx->sio = sio = mempool_alloc(sio_pool, GFP_NOIO);
sio->nr_bvecs = 0;
sio->len = 0;
+ sio->entry = phys;
}
bvec_set_folio(&sio->bvecs[sio->nr_bvecs], folio, folio_size(folio), 0);
sio->len += folio_size(folio);
@@ -367,7 +398,8 @@ static void swap_add_folio(struct swap_io_ctx *ctx, struct folio *folio, int rw)
}
}
-void __swap_writepage(struct swap_io_ctx *ctx, struct folio *folio)
+void __swap_writepage(struct swap_io_ctx *ctx, struct folio *folio,
+ swp_entry_t phys)
{
VM_BUG_ON_FOLIO(!folio_test_swapcache(folio), folio);
@@ -383,7 +415,7 @@ void __swap_writepage(struct swap_io_ctx *ctx, struct folio *folio)
folio_start_writeback(folio);
folio_unlock(folio);
- swap_add_folio(ctx, folio, WRITE);
+ swap_add_folio(ctx, folio, phys, WRITE);
}
/*
@@ -456,6 +488,7 @@ void swap_read_folio(struct swap_io_ctx *ctx, struct folio *folio)
bool workingset = folio_test_workingset(folio);
unsigned long pflags;
bool in_thrashing;
+ swp_entry_t phys;
VM_BUG_ON_FOLIO(!folio_test_swapcache(folio) && !synchronous, folio);
VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
@@ -480,9 +513,24 @@ void swap_read_folio(struct swap_io_ctx *ctx, struct folio *folio)
if (zswap_load(folio) != -ENOENT)
goto finish;
+ /*
+ * Resolve the physical slot to read from. A vswap entry keeps
+ * folio->swap virtual, so map it to its physical backing; a folio with
+ * no backing has nothing to read.
+ */
+ if (swap_is_vswap(sis)) {
+ phys = vswap_to_phys(folio->swap);
+ if (!phys.val) {
+ folio_unlock(folio);
+ goto finish;
+ }
+ } else {
+ phys = folio->swap;
+ }
+
/* We have to read from slower devices. Increase zswap protection. */
zswap_folio_swapin(folio);
- swap_add_folio(ctx, folio, READ);
+ swap_add_folio(ctx, folio, phys, READ);
finish:
if (workingset) {
@@ -514,8 +562,6 @@ static void swap_fs_write_complete(struct kiocb *iocb, long ret)
bool failed = ret != sio->len;
if (failed) {
- struct page *page = sio->bvecs[0].bv_page;
-
/*
* In the case of swap-over-nfs, this can be a temporary failure
* if the system has limited memory for allocating transmit
@@ -523,7 +569,7 @@ static void swap_fs_write_complete(struct kiocb *iocb, long ret)
* folio_rotate_reclaimable but rate-limit the messages.
*/
pr_err_ratelimited("Write error %ld on dio swapfile (%llu)\n",
- ret, swap_dev_pos(page_swap_entry(page)));
+ ret, swap_dev_pos(sio->entry));
}
swap_write_end(sio, failed);
@@ -595,7 +641,7 @@ static void swap_bdev_submit_write(struct swap_io_ctx *ctx)
bio_init(bio, ctx->sis->bdev, sio->bvecs, ARRAY_SIZE(sio->bvecs),
REQ_OP_WRITE | REQ_SWAP);
bio->bi_iter.bi_size = sio->len;
- bio->bi_iter.bi_sector = swap_folio_sector(bio_first_folio_all(bio));
+ bio->bi_iter.bi_sector = swap_entry_sector(sio->entry);
bio_associate_blkg_from_page(bio, bio_first_folio_all(bio));
if (ctx->sis->flags & SWP_SYNCHRONOUS_IO) {
@@ -615,7 +661,7 @@ static void swap_bdev_submit_read(struct swap_io_ctx *ctx)
bio_init(bio, ctx->sis->bdev, sio->bvecs, ARRAY_SIZE(sio->bvecs),
REQ_OP_READ);
bio->bi_iter.bi_size = sio->len;
- bio->bi_iter.bi_sector = swap_folio_sector(bio_first_folio_all(bio));
+ bio->bi_iter.bi_sector = swap_entry_sector(sio->entry);
if (ctx->sis->flags & SWP_SYNCHRONOUS_IO) {
/*
@@ -633,13 +679,14 @@ static void swap_bdev_submit_read(struct swap_io_ctx *ctx)
}
}
-static bool swap_bdev_can_merge(struct folio *folio, struct folio *prev_folio,
- size_t prev_folio_size, int rw)
+static bool swap_bdev_can_merge(struct folio *folio, swp_entry_t phys,
+ struct swap_iocb *sio, int rw)
{
- if (swap_folio_sector(folio) !=
- swap_folio_sector(prev_folio) + (prev_folio_size >> SECTOR_SHIFT))
+ if (swap_entry_sector(phys) !=
+ swap_entry_sector(sio->entry) + (sio->len >> SECTOR_SHIFT))
return false;
- if (rw == WRITE && !folio_blkg_can_merge(folio, prev_folio))
+ if (rw == WRITE && !folio_blkg_can_merge(folio,
+ bvec_folio(&sio->bvecs[sio->nr_bvecs - 1])))
return false;
return true;
}
@@ -655,7 +702,7 @@ void swap_fs_prepare_rw(struct swap_io_ctx *ctx, int rw, struct iov_iter *iter)
struct swap_iocb *sio = ctx->sio;
init_sync_kiocb(&sio->iocb, ctx->sis->swap_file);
- sio->iocb.ki_pos = swap_dev_pos(bvec_folio(&sio->bvecs[0])->swap);
+ sio->iocb.ki_pos = swap_dev_pos(sio->entry);
if (rw == WRITE)
sio->iocb.ki_complete = swap_fs_write_complete;
else
@@ -666,11 +713,10 @@ void swap_fs_prepare_rw(struct swap_io_ctx *ctx, int rw, struct iov_iter *iter)
}
EXPORT_SYMBOL_GPL(swap_fs_prepare_rw);
-bool swap_fs_can_merge(struct folio *folio, struct folio *prev_folio,
- size_t prev_folio_size, int rw)
+bool swap_fs_can_merge(struct folio *folio, swp_entry_t phys,
+ struct swap_iocb *sio, int rw)
{
- return swap_dev_pos(folio->swap) ==
- swap_dev_pos(prev_folio->swap) + prev_folio_size;
+ return swap_dev_pos(phys) == swap_dev_pos(sio->entry) + sio->len;
}
EXPORT_SYMBOL_GPL(swap_fs_can_merge);
diff --git a/mm/shmem.c b/mm/shmem.c
index 599665a3d6e7b..d58f30b06255b 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -86,6 +86,7 @@ static struct vfsmount *shm_mnt __ro_after_init;
#include <linux/uaccess.h>
#include "internal.h"
+#include "vswap.h"
#define VM_ACCT(size) (PAGE_ALIGN(size) >> PAGE_SHIFT)
@@ -1618,7 +1619,8 @@ int shmem_writeout(struct swap_io_ctx *ctx, struct folio *folio,
if ((info->flags & SHMEM_F_LOCKED) || sbinfo->noswap)
goto redirty;
- if (!total_swap_pages)
+ /* vswap doesn't contribute to total_swap_pages */
+ if (!total_swap_pages && !(vswap_is_enabled() && zswap_is_enabled()))
goto redirty;
/*
diff --git a/mm/swap.h b/mm/swap.h
index 90a551a88df63..f2b64920ae5b3 100644
--- a/mm/swap.h
+++ b/mm/swap.h
@@ -67,6 +67,13 @@ struct swap_cluster_info {
struct list_head list;
};
+struct swap_cluster_info_dynamic {
+ struct swap_cluster_info ci;
+ unsigned int index; /* for cluster_index() */
+ struct rcu_head rcu;
+ atomic_long_t *virtual_table; /* Backing pointers for vswap slots */
+};
+
/* All on-list cluster must have a non-zero flag. */
enum swap_cluster_flags {
CLUSTER_FLAG_NONE = 0, /* For temporary off-list cluster */
@@ -77,6 +84,7 @@ enum swap_cluster_flags {
CLUSTER_FLAG_USABLE = CLUSTER_FLAG_FRAG,
CLUSTER_FLAG_FULL,
CLUSTER_FLAG_DISCARD,
+ CLUSTER_FLAG_DEAD, /* Vswap dynamic cluster pending kfree_rcu */
CLUSTER_FLAG_MAX,
};
@@ -119,12 +127,33 @@ static inline struct swap_info_struct *__swap_entry_to_info(swp_entry_t entry)
return __swap_type_to_info(swp_type(entry));
}
+/**
+ * __swap_offset_to_cluster - look up the cluster holding a swap offset
+ * @si: the swap device
+ * @offset: the swap entry offset
+ *
+ * Context: A vswap cluster is freed by kfree_rcu(). Callers must hold the
+ * RCU read lock, or know the cluster is pinned by an in-use entry.
+ *
+ * Return: the cluster, or NULL if @si is a vswap device with no cluster
+ * allocated at @offset.
+ */
static inline struct swap_cluster_info *__swap_offset_to_cluster(
struct swap_info_struct *si, pgoff_t offset)
{
+ unsigned int cluster_idx = offset / SWAPFILE_CLUSTER;
+
VM_WARN_ON_ONCE(percpu_ref_is_zero(&si->users)); /* race with swapoff */
VM_WARN_ON_ONCE(offset >= roundup(si->max, SWAPFILE_CLUSTER));
- return &si->cluster_info[offset / SWAPFILE_CLUSTER];
+
+ if (swap_is_vswap(si)) {
+ struct swap_cluster_info_dynamic *ci_dyn;
+
+ ci_dyn = xa_load(&si->cluster_info_pool, cluster_idx);
+ return ci_dyn ? &ci_dyn->ci : NULL;
+ }
+
+ return &si->cluster_info[cluster_idx];
}
static inline struct swap_cluster_info *__swap_entry_to_cluster(swp_entry_t entry)
@@ -133,10 +162,36 @@ static inline struct swap_cluster_info *__swap_entry_to_cluster(swp_entry_t entr
swp_offset(entry));
}
+static inline struct swap_cluster_info *__vswap_cluster_lock(
+ struct swap_info_struct *si, unsigned long offset, bool irq)
+{
+ struct swap_cluster_info *ci;
+
+ rcu_read_lock();
+ ci = __swap_offset_to_cluster(si, offset);
+ if (ci) {
+ if (irq)
+ spin_lock_irq(&ci->lock);
+ else
+ spin_lock(&ci->lock);
+
+ /* The cluster can be torn down while we wait for the lock. */
+ if (ci->flags == CLUSTER_FLAG_DEAD) {
+ if (irq)
+ spin_unlock_irq(&ci->lock);
+ else
+ spin_unlock(&ci->lock);
+ ci = NULL;
+ }
+ }
+ rcu_read_unlock();
+ return ci;
+}
+
static __always_inline struct swap_cluster_info *__swap_cluster_lock(
struct swap_info_struct *si, unsigned long offset, bool irq)
{
- struct swap_cluster_info *ci = __swap_offset_to_cluster(si, offset);
+ struct swap_cluster_info *ci;
/*
* Nothing modifies swap cache in an IRQ context. All access to
@@ -149,6 +204,11 @@ static __always_inline struct swap_cluster_info *__swap_cluster_lock(
*/
VM_WARN_ON_ONCE(!in_task());
VM_WARN_ON_ONCE(percpu_ref_is_zero(&si->users)); /* race with swapoff */
+
+ if (swap_is_vswap(si))
+ return __vswap_cluster_lock(si, offset, irq);
+
+ ci = __swap_offset_to_cluster(si, offset);
if (irq)
spin_lock_irq(&ci->lock);
else
@@ -159,10 +219,12 @@ static __always_inline struct swap_cluster_info *__swap_cluster_lock(
/**
* swap_cluster_lock - Lock and return the swap cluster of given offset.
* @si: swap device the cluster belongs to.
- * @offset: the swap entry offset, pointing to a valid slot.
+ * @offset: the swap entry offset.
*
* Context: The caller must ensure the offset is in the valid range and
* protect the swap device with reference count or locks.
+ * Return: the locked cluster, or NULL if it is gone. Only a vswap device
+ * can return NULL, as its clusters are allocated and freed on demand.
*/
static inline struct swap_cluster_info *swap_cluster_lock(
struct swap_info_struct *si, unsigned long offset)
@@ -258,7 +320,8 @@ void swap_read_folio(struct swap_io_ctx *ctx, struct folio *folio);
void swap_read_submit(struct swap_io_ctx *ctx);
void swap_write_submit(struct swap_io_ctx *ctx);
int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio);
-void __swap_writepage(struct swap_io_ctx *ctx, struct folio *folio);
+void __swap_writepage(struct swap_io_ctx *ctx, struct folio *folio,
+ swp_entry_t phys);
/* linux/mm/swap_state.c */
extern struct address_space swap_space __read_mostly;
diff --git a/mm/swap_state.c b/mm/swap_state.c
index b76eb3d876fd7..5cfddec8633b9 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -27,6 +27,7 @@
#include "internal.h"
#include "swap_table.h"
#include "swap.h"
+#include "vswap.h"
/* Swap readahead cluster size, as a power of 2 pages. */
static int page_cluster;
@@ -96,8 +97,10 @@ struct folio *swap_cache_get_folio(swp_entry_t entry)
struct folio *folio;
for (;;) {
+ rcu_read_lock();
swp_tb = swap_table_get(__swap_entry_to_cluster(entry),
swp_cluster_offset(entry));
+ rcu_read_unlock();
if (!swp_tb_is_folio(swp_tb))
return NULL;
folio = swp_tb_to_folio(swp_tb);
@@ -119,8 +122,10 @@ bool swap_cache_has_folio(swp_entry_t entry)
{
unsigned long swp_tb;
+ rcu_read_lock();
swp_tb = swap_table_get(__swap_entry_to_cluster(entry),
swp_cluster_offset(entry));
+ rcu_read_unlock();
return swp_tb_is_folio(swp_tb);
}
@@ -136,8 +141,10 @@ void *swap_cache_get_shadow(swp_entry_t entry)
{
unsigned long swp_tb;
+ rcu_read_lock();
swp_tb = swap_table_get(__swap_entry_to_cluster(entry),
swp_cluster_offset(entry));
+ rcu_read_unlock();
if (swp_tb_is_shadow(swp_tb))
return swp_tb_to_shadow(swp_tb);
return NULL;
@@ -167,6 +174,9 @@ static int __swap_cache_add_check(struct swap_cluster_info *ci,
unsigned int ci_off, ci_end;
unsigned long old_tb;
bool is_zero;
+ struct swap_cluster_info_dynamic *ci_dyn;
+ enum vswap_backing_type type;
+ int ret;
lockdep_assert_held(&ci->lock);
@@ -179,6 +189,9 @@ static int __swap_cache_add_check(struct swap_cluster_info *ci,
return -ENOENT;
ci_off = swp_cluster_offset(targ_entry);
old_tb = __swap_table_get(ci, ci_off);
+ /* Physical readahead can hit a vswap-backing rmap slot; skip it. */
+ if (swp_tb_is_pointer(old_tb))
+ return -ENOENT;
if (swp_tb_is_folio(old_tb))
return -EEXIST;
if (!__swp_tb_get_count(old_tb))
@@ -191,12 +204,26 @@ static int __swap_cache_add_check(struct swap_cluster_info *ci,
if (nr == 1)
return 0;
+ /*
+ * For a vswap entry batch, reject if the backing is not THP-amenable
+ * (e.g. uniformly ZSWAP, or mixed). The order-fallback loop in
+ * swap_cache_alloc_folio will retry with a smaller order on -EBUSY.
+ */
+ if (is_vswap_entry(targ_entry)) {
+ ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
+ ret = __vswap_check_backing(ci_dyn, round_down(ci_off, nr),
+ nr, &type);
+ if (ret != nr || type == VSWAP_ZSWAP)
+ return -EBUSY;
+ }
+
is_zero = __swap_table_test_zero(ci, ci_off);
ci_off = round_down(ci_off, nr);
ci_end = ci_off + nr;
do {
old_tb = __swap_table_get(ci, ci_off);
- if (unlikely(swp_tb_is_folio(old_tb) ||
+ if (unlikely(swp_tb_is_pointer(old_tb) ||
+ swp_tb_is_folio(old_tb) ||
!__swp_tb_get_count(old_tb) ||
is_zero != __swap_table_test_zero(ci, ci_off) ||
(memcg_id && *memcg_id != __swap_cgroup_get(ci, ci_off))))
@@ -406,14 +433,16 @@ void __swap_cache_replace_folio(struct swap_cluster_info *ci,
* -ENOENT / -EEXIST: Target swap entry is unavailable or cached, the caller
* should abort or try to use the cached folio instead
*/
-static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,
- swp_entry_t targ_entry, gfp_t gfp,
+static struct folio *__swap_cache_alloc(swp_entry_t targ_entry, gfp_t gfp,
unsigned int order, struct vm_fault *vmf,
struct mempolicy *mpol, pgoff_t ilx)
{
int err;
swp_entry_t entry;
struct folio *folio;
+ struct swap_cluster_info *ci;
+ struct swap_info_struct *si = __swap_entry_to_info(targ_entry);
+ unsigned long offset = swp_offset(targ_entry);
void *shadow = NULL;
unsigned short memcg_id;
unsigned long address, nr_pages = 1UL << order;
@@ -423,9 +452,12 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,
entry.val = round_down(targ_entry.val, nr_pages);
/* Check if the slot and range are available, skip allocation if not */
- spin_lock(&ci->lock);
- err = __swap_cache_add_check(ci, targ_entry, nr_pages, NULL, NULL);
- spin_unlock(&ci->lock);
+ err = -ENOENT;
+ ci = swap_cluster_lock(si, offset);
+ if (ci) {
+ err = __swap_cache_add_check(ci, targ_entry, nr_pages, NULL, NULL);
+ swap_cluster_unlock(ci);
+ }
if (unlikely(err))
return ERR_PTR(err);
@@ -446,10 +478,13 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,
return ERR_PTR(-ENOMEM);
/* Double check the range is still not in conflict */
- spin_lock(&ci->lock);
- err = __swap_cache_add_check(ci, targ_entry, nr_pages, &shadow, &memcg_id);
+ err = -ENOENT;
+ ci = swap_cluster_lock(si, offset);
+ if (ci)
+ err = __swap_cache_add_check(ci, targ_entry, nr_pages, &shadow, &memcg_id);
if (unlikely(err)) {
- spin_unlock(&ci->lock);
+ if (ci)
+ swap_cluster_unlock(ci);
folio_put(folio);
return ERR_PTR(err);
}
@@ -457,10 +492,11 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,
__folio_set_locked(folio);
__folio_set_swapbacked(folio);
__swap_cache_do_add_folio(ci, folio, entry);
- spin_unlock(&ci->lock);
+ swap_cluster_unlock(ci);
if (mem_cgroup_swapin_charge_folio(folio, memcg_id,
vmf ? vmf->vma->vm_mm : NULL, gfp)) {
+ /* The folio pins the cluster */
spin_lock(&ci->lock);
__swap_cache_do_del_folio(ci, folio, entry, shadow);
spin_unlock(&ci->lock);
@@ -517,9 +553,7 @@ struct folio *swap_cache_alloc_folio(swp_entry_t targ_entry, gfp_t gfp,
{
int order, err;
struct folio *ret;
- struct swap_cluster_info *ci;
- ci = __swap_entry_to_cluster(targ_entry);
order = highest_order(orders);
/* orders must be non-zero, and must not exceed cluster size. */
@@ -527,7 +561,7 @@ struct folio *swap_cache_alloc_folio(swp_entry_t targ_entry, gfp_t gfp,
return ERR_PTR(-EINVAL);
do {
- ret = __swap_cache_alloc(ci, targ_entry, gfp, order,
+ ret = __swap_cache_alloc(targ_entry, gfp, order,
vmf, mpol, ilx);
if (!IS_ERR(ret))
break;
diff --git a/mm/swap_table.h b/mm/swap_table.h
index e6613e62f8d0f..b614b1989fd9d 100644
--- a/mm/swap_table.h
+++ b/mm/swap_table.h
@@ -4,8 +4,11 @@
#include <linux/rcupdate.h>
#include <linux/atomic.h>
+#include <linux/swapops.h>
#include "swap.h"
+extern struct swap_info_struct *vswap_si;
+
/* A typical flat array in each cluster as swap table */
struct swap_table {
atomic_long_t entries[SWAPFILE_CLUSTER];
@@ -28,7 +31,7 @@ struct swap_memcg_table {
* NULL: |---------------- 0 ---------------| - Free slot
* Shadow: |SWAP_COUNT|Z|---- SHADOW_VAL ---|1| - Swapped out slot
* PFN: |SWAP_COUNT|Z|------ PFN -------|10| - Cached slot
- * Pointer: |----------- Pointer ----------|100| - (Unused)
+ * Pointer: |C|------- vswap offset -------|100| - vswap rmap
* Bad: |------------- 1 -------------|1000| - Bad slot
*
* COUNT is `SWP_TB_COUNT_BITS` long, Z is the `SWP_TB_ZERO_FLAG` bit,
@@ -49,9 +52,8 @@ struct swap_memcg_table {
* - PFN: Swap slot is in use, and cached. Memcg info is recorded on the page
* struct.
*
- * - Pointer: Unused yet. `0b100` is reserved for potential pointer usage
- * because only the lower three bits can be used as a marker for 8 bytes
- * aligned pointers.
+ * - Pointer: Reverse map from a physical slot to the vswap entry that owns
+ * it. See the layout below.
*
* - Bad: Swap slot is reserved, protects swap header or holes on swap devices.
*/
@@ -255,6 +257,8 @@ static inline unsigned long swap_table_get(struct swap_cluster_info *ci,
unsigned long swp_tb;
VM_WARN_ON_ONCE(off >= SWAPFILE_CLUSTER);
+ if (!ci)
+ return SWP_TB_NULL;
rcu_read_lock();
table = rcu_dereference(ci->table);
@@ -366,4 +370,42 @@ static inline unsigned short __swap_cgroup_clear(struct swap_cluster_info *ci,
}
#endif
+/*
+ * Pointer-tagged swap table entry: rmap for vswap-backing physical slots.
+ *
+ * On physical clusters, a Pointer-tagged entry stores the offset of the
+ * vswap entry that owns this physical slot (the reverse map). Only the
+ * offset is stored; the swap type is implicit (always vswap_si->type,
+ * since there is exactly one vswap device). The top bit is reserved as
+ * a cache-only flag, set when vswap swap_count drops to 0 but the folio
+ * is still in swap cache.
+ *
+ * Pointer: |C|---- vswap offset ----|100|
+ * C = SWP_RMAP_CACHE_ONLY (bit 63)
+ */
+#define SWP_TB_PTR_MARK_BITS 3
+#define SWP_TB_PTR_MARK 0b100UL
+#define SWP_TB_PTR_MARK_MASK ((1UL << SWP_TB_PTR_MARK_BITS) - 1)
+#define SWP_RMAP_CACHE_ONLY (1UL << (BITS_PER_LONG - 1))
+#define SWP_RMAP_ENTRY_MASK (~(SWP_RMAP_CACHE_ONLY | SWP_TB_PTR_MARK_MASK))
+
+static inline bool swp_tb_is_pointer(unsigned long swp_tb)
+{
+ return (swp_tb & SWP_TB_PTR_MARK_MASK) == SWP_TB_PTR_MARK;
+}
+
+static inline unsigned long swp_entry_to_swp_tb_ptr(swp_entry_t entry)
+{
+ return (swp_offset(entry) << SWP_TB_PTR_MARK_BITS) | SWP_TB_PTR_MARK;
+}
+
+static inline swp_entry_t swp_tb_ptr_to_swp_entry(unsigned long swp_tb)
+{
+ unsigned long offset;
+
+ VM_WARN_ON(!swp_tb_is_pointer(swp_tb));
+ offset = (swp_tb & SWP_RMAP_ENTRY_MASK) >> SWP_TB_PTR_MARK_BITS;
+ return swp_entry(vswap_si->type, offset);
+}
+
#endif
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 53bf01d5f7f11..9860fb3b079fb 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -7,6 +7,7 @@
*/
#include <linux/blkdev.h>
+#include <linux/debugfs.h>
#include <linux/mm.h>
#include <linux/sched/mm.h>
#include <linux/sched/task.h>
@@ -36,6 +37,7 @@
#include <linux/poll.h>
#include <linux/oom.h>
#include <linux/swapfile.h>
+#include <linux/swap_ops.h>
#include <linux/export.h>
#include <linux/sort.h>
#include <linux/completion.h>
@@ -45,7 +47,9 @@
#include <asm/tlbflush.h>
#include <linux/leafops.h>
+#include "memcontrol-v1.h"
#include "swap_table.h"
+#include "vswap.h"
#include "internal.h"
#include "swap.h"
@@ -129,6 +133,24 @@ static DEFINE_PER_CPU(struct percpu_swap_cluster, percpu_swap_cluster) = {
.lock = INIT_LOCAL_LOCK(),
};
+struct percpu_vswap_cluster {
+ unsigned long offset[SWAP_NR_ORDERS];
+ local_lock_t lock;
+};
+
+static DEFINE_PER_CPU(struct percpu_vswap_cluster, percpu_vswap_cluster) = {
+ .offset = { [0 ... SWAP_NR_ORDERS - 1] = SWAP_ENTRY_INVALID },
+ .lock = INIT_LOCAL_LOCK(),
+};
+
+static atomic_long_t vswap_alloc_reject = ATOMIC_LONG_INIT(0);
+
+static bool vswap_alloc(struct folio *folio);
+static void vswap_mark_cache_only(struct swap_cluster_info *ci,
+ unsigned int ci_off);
+static void vswap_clear_cache_only(struct swap_cluster_info *ci,
+ unsigned int ci_start, int nr);
+
/* May return NULL on invalid type, caller must check for NULL return */
static struct swap_info_struct *swap_type_to_info(int type)
{
@@ -234,7 +256,8 @@ static int __try_to_reclaim_swap(struct swap_info_struct *si,
need_reclaim = ((flags & TTRS_ANYWAY) ||
((flags & TTRS_UNMAPPED) && !folio_mapped(folio)) ||
- ((flags & TTRS_FULL) && mem_cgroup_swap_full(folio)));
+ ((flags & TTRS_FULL) && mem_cgroup_swap_full(folio) &&
+ folio_phys_swap_backed(folio)));
if (!need_reclaim || !folio_swapcache_freeable(folio))
goto out_unlock;
@@ -328,14 +351,14 @@ offset_to_swap_extent(struct swap_info_struct *sis, unsigned long offset)
BUG();
}
-sector_t swap_folio_sector(struct folio *folio)
+sector_t swap_entry_sector(swp_entry_t entry)
{
- struct swap_info_struct *sis = __swap_entry_to_info(folio->swap);
+ struct swap_info_struct *sis = __swap_entry_to_info(entry);
struct swap_extent *se;
sector_t sector;
pgoff_t offset;
- offset = swp_offset(folio->swap);
+ offset = swp_offset(entry);
se = offset_to_swap_extent(sis, offset);
sector = se->start_block + (offset - se->start_page);
return sector << (PAGE_SHIFT - 9);
@@ -401,13 +424,15 @@ static inline bool cluster_is_usable(struct swap_cluster_info *ci, int order)
static inline unsigned int cluster_index(struct swap_info_struct *si,
struct swap_cluster_info *ci)
{
+ if (swap_is_vswap(si))
+ return container_of(ci, struct swap_cluster_info_dynamic, ci)->index;
return ci - si->cluster_info;
}
-static inline unsigned int cluster_offset(struct swap_info_struct *si,
- struct swap_cluster_info *ci)
+static inline unsigned long cluster_offset(struct swap_info_struct *si,
+ struct swap_cluster_info *ci)
{
- return cluster_index(si, ci) * SWAPFILE_CLUSTER;
+ return (unsigned long)cluster_index(si, ci) * SWAPFILE_CLUSTER;
}
static void swap_cluster_free_table_folio_rcu_cb(struct rcu_head *head)
@@ -446,7 +471,8 @@ static void swap_cluster_free_table(struct swap_cluster_info *ci)
swap_cluster_free_table_folio_rcu_cb);
}
-static int swap_cluster_alloc_table(struct swap_cluster_info *ci, gfp_t gfp)
+static int swap_cluster_alloc_table(struct swap_info_struct *si,
+ struct swap_cluster_info *ci, gfp_t gfp)
{
struct swap_table *table = NULL;
struct folio *folio;
@@ -469,7 +495,14 @@ static int swap_cluster_alloc_table(struct swap_cluster_info *ci, gfp_t gfp)
rcu_assign_pointer(ci->table, table);
#ifdef CONFIG_MEMCG
- if (!mem_cgroup_disabled()) {
+ /*
+ * A physical cluster under vswap may hold only vswap backings, which
+ * record their memcg on the vswap cluster's table, not this one. Such
+ * clusters defer memcg_table allocation until they hand out a slot
+ * that maps directly into the PTEs.
+ */
+ if ((!vswap_is_enabled() || swap_is_vswap(si)) &&
+ !mem_cgroup_disabled()) {
VM_WARN_ON_ONCE(ci->memcg_table);
ci->memcg_table = kzalloc_obj(*ci->memcg_table, gfp);
if (!ci->memcg_table) {
@@ -532,14 +565,16 @@ swap_cluster_populate(struct swap_info_struct *si,
/*
* Only cluster isolation from the allocator does table allocation.
* Swap allocator uses percpu clusters and holds the local lock.
+ * vswap clusters are destroyed rather than freed to si->free_clusters.
*/
+ VM_WARN_ON_ONCE(swap_is_vswap(si));
lockdep_assert_held(&this_cpu_ptr(&percpu_swap_cluster)->lock);
if (!(si->flags & SWP_SOLIDSTATE))
lockdep_assert_held(&si->global_cluster_lock);
lockdep_assert_held(&ci->lock);
- if (!swap_cluster_alloc_table(ci, __GFP_HIGH | __GFP_NOMEMALLOC |
- __GFP_NOWARN))
+ if (!swap_cluster_alloc_table(si, ci, __GFP_HIGH | __GFP_NOMEMALLOC |
+ __GFP_NOWARN))
return ci;
/*
@@ -552,8 +587,8 @@ swap_cluster_populate(struct swap_info_struct *si,
spin_unlock(&si->global_cluster_lock);
local_unlock(&percpu_swap_cluster.lock);
- ret = swap_cluster_alloc_table(ci, __GFP_HIGH | __GFP_NOMEMALLOC |
- GFP_KERNEL);
+ ret = swap_cluster_alloc_table(si, ci, __GFP_HIGH | __GFP_NOMEMALLOC |
+ GFP_KERNEL);
/*
* Back to atomic context. We might have migrated to a new CPU with a
@@ -586,10 +621,15 @@ static void move_cluster(struct swap_info_struct *si,
lockdep_assert_held(&ci->lock);
spin_lock(&si->lock);
- if (ci->flags == CLUSTER_FLAG_NONE)
+ if (!list) {
+ /* Going away. An isolated cluster is already off its list. */
+ if (ci->flags != CLUSTER_FLAG_NONE)
+ list_del(&ci->list);
+ } else if (ci->flags == CLUSTER_FLAG_NONE) {
list_add_tail(&ci->list, list);
- else
+ } else {
list_move_tail(&ci->list, list);
+ }
spin_unlock(&si->lock);
ci->flags = new_flags;
}
@@ -607,6 +647,19 @@ static void __free_cluster(struct swap_info_struct *si, struct swap_cluster_info
{
swap_cluster_assert_empty(ci, 0, SWAPFILE_CLUSTER, false);
swap_cluster_free_table(ci);
+
+ if (swap_is_vswap(si)) {
+ struct swap_cluster_info_dynamic *ci_dyn;
+
+ /* vswap clusters are destroyed, not returned to free_clusters. */
+ ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
+ xa_erase(&si->cluster_info_pool, ci_dyn->index);
+ move_cluster(si, ci, NULL, CLUSTER_FLAG_DEAD);
+ vswap_cluster_free_vtable(ci);
+ kfree_rcu(ci_dyn, rcu);
+ return;
+ }
+
move_cluster(si, ci, &si->free_clusters, CLUSTER_FLAG_FREE);
ci->order = 0;
}
@@ -795,7 +848,7 @@ static int swap_cluster_setup_bad_slot(struct swap_info_struct *si,
/* si->max may got shrunk by swap swap_activate() */
if (offset >= si->max && !mask) {
- pr_debug("Ignoring bad slot %u (max: %u)\n", offset, si->max);
+ pr_debug("Ignoring bad slot %u (max: %lu)\n", offset, si->max);
return 0;
}
/*
@@ -812,7 +865,7 @@ static int swap_cluster_setup_bad_slot(struct swap_info_struct *si,
ci = cluster_info + idx;
/* Need to allocate swap table first for initial bad slot marking. */
- if (!ci->count && swap_cluster_alloc_table(ci, GFP_KERNEL))
+ if (!ci->count && swap_cluster_alloc_table(si, ci, GFP_KERNEL))
return -ENOMEM;
spin_lock(&ci->lock);
/* Check for duplicated bad swap slots. */
@@ -830,6 +883,54 @@ static int swap_cluster_setup_bad_slot(struct swap_info_struct *si,
return ret;
}
+/*
+ * Try to reclaim a Pointer-tagged physical slot backing a vswap entry.
+ * The physical cluster lock must NOT be held. Returns the backing folio's
+ * page count, negated if the slots could not be reclaimed, or 0 if the
+ * folio could not be shown to own @offset (i.e. there is a race).
+ */
+static int try_to_reclaim_vswap_backing(struct swap_info_struct *si,
+ unsigned long offset,
+ swp_entry_t vswap_entry)
+{
+ swp_entry_t phys_base;
+ struct folio *folio;
+ unsigned int i;
+ int ret;
+
+ folio = swap_cache_get_folio(vswap_entry);
+ if (!folio)
+ return 0;
+
+ if (!folio_trylock(folio)) {
+ folio_put(folio);
+ return 0;
+ }
+
+ if (!folio_matches_swap_entry(folio, vswap_entry)) {
+ folio_unlock(folio);
+ folio_put(folio);
+ return 0;
+ }
+
+ i = vswap_entry.val - folio->swap.val;
+ phys_base = vswap_to_phys(folio->swap);
+ if (!phys_base.val || swp_type(phys_base) != si->type ||
+ swp_offset(phys_base) + i != offset) {
+ folio_unlock(folio);
+ folio_put(folio);
+ return 0;
+ }
+
+ /* The run is ours: skip it all, whether or not the free succeeds. */
+ ret = folio_nr_pages(folio);
+ if (!folio_free_swap(folio))
+ ret = -ret;
+ folio_unlock(folio);
+ folio_put(folio);
+ return ret;
+}
+
/*
* Reclaim drops the ci lock, so the cluster may become unusable (freed or
* stolen by a lower order). @usable will be set to false if that happens.
@@ -843,6 +944,8 @@ static bool cluster_reclaim_range(struct swap_info_struct *si,
unsigned long offset = start, end = start + nr_pages;
unsigned long swp_tb;
+ VM_WARN_ON_ONCE(swap_is_vswap(si));
+
spin_unlock(&ci->lock);
do {
swp_tb = swap_table_get(ci, offset % SWAPFILE_CLUSTER);
@@ -895,7 +998,8 @@ static bool cluster_scan_range(struct swap_info_struct *si,
if (swp_tb_is_null(swp_tb))
continue;
if (swp_tb_is_folio(swp_tb) && !__swp_tb_get_count(swp_tb)) {
- if (!vm_swap_full())
+ /* vswap slots are abundant; never reclaim to reuse one */
+ if (swap_is_vswap(si) || !vm_swap_full())
return false;
*need_reclaim = true;
continue;
@@ -915,6 +1019,8 @@ static bool __swap_cluster_alloc_entries(struct swap_info_struct *si,
{
unsigned int order;
unsigned long nr_pages;
+ swp_entry_t vswap_entry, v;
+ unsigned int i;
lockdep_assert_held(&ci->lock);
@@ -934,8 +1040,26 @@ static bool __swap_cluster_alloc_entries(struct swap_info_struct *si,
order = folio_order(folio);
nr_pages = 1 << order;
swap_cluster_assert_empty(ci, ci_off, nr_pages, false);
- __swap_cache_add_folio(ci, folio, swp_entry(si->type,
- ci_off + cluster_offset(si, ci)));
+ if (folio_test_swapcache(folio)) {
+ /*
+ * Folio already in the swap cache: we are allocating
+ * physical backing for its vswap entry. Point each
+ * physical slot back at its own vswap entry
+ * (Pointer-tagged rmap).
+ */
+ VM_WARN_ON(!is_vswap_entry(folio->swap));
+ vswap_entry = folio->swap;
+ for (i = 0; i < nr_pages; i++) {
+ v = vswap_entry;
+ v.val += i;
+ __swap_table_set(ci, ci_off + i,
+ swp_entry_to_swp_tb_ptr(v));
+ }
+ } else {
+ __swap_cache_add_folio(ci, folio,
+ swp_entry(si->type,
+ ci_off + cluster_offset(si, ci)));
+ }
} else if (IS_ENABLED(CONFIG_HIBERNATION)) {
order = 0;
nr_pages = 1;
@@ -961,11 +1085,13 @@ static bool __swap_cluster_alloc_entries(struct swap_info_struct *si,
}
/* Try use a new cluster for current CPU and allocate from it. */
-static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si,
- struct swap_cluster_info *ci,
- struct folio *folio, unsigned long offset)
+static unsigned long alloc_swap_scan_cluster(struct swap_info_struct *si,
+ struct swap_cluster_info *ci,
+ struct folio *folio,
+ unsigned long offset,
+ bool *nomem)
{
- unsigned int next = SWAP_ENTRY_INVALID, found = SWAP_ENTRY_INVALID;
+ unsigned long next = SWAP_ENTRY_INVALID, found = SWAP_ENTRY_INVALID;
unsigned long start = ALIGN_DOWN(offset, SWAPFILE_CLUSTER);
unsigned int order = likely(folio) ? folio_order(folio) : 0;
unsigned long end = start + SWAPFILE_CLUSTER;
@@ -992,6 +1118,24 @@ static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si,
if (!ret)
continue;
}
+#ifdef CONFIG_MEMCG
+ /*
+ * Lazy-allocate memcg_table on the first direct-use slot of a
+ * physical cluster.
+ */
+ if (vswap_is_enabled() && folio &&
+ !folio_test_swapcache(folio) && !mem_cgroup_disabled() &&
+ !ci->memcg_table) {
+ ci->memcg_table = kzalloc_obj(*ci->memcg_table,
+ GFP_ATOMIC | __GFP_NOMEMALLOC |
+ __GFP_NOWARN);
+ if (!ci->memcg_table) {
+ if (nomem)
+ *nomem = true;
+ goto out;
+ }
+ }
+#endif
if (!__swap_cluster_alloc_entries(si, ci, folio, offset % SWAPFILE_CLUSTER))
break;
found = offset;
@@ -1001,8 +1145,20 @@ static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si,
break;
}
out:
- relocate_cluster(si, ci);
+ /*
+ * On a discard-capable device, relocating a cluster whose memcg_table
+ * allocation failed queues a discard for slots that were never used,
+ * which folio_alloc_phys_swap() reads as progress and retries on.
+ */
+ if (nomem && *nomem && !ci->count)
+ __free_cluster(si, ci);
+ else
+ relocate_cluster(si, ci);
swap_cluster_unlock(ci);
+ if (swap_is_vswap(si)) {
+ this_cpu_write(percpu_vswap_cluster.offset[order], next);
+ return found;
+ }
if (si->flags & SWP_SOLIDSTATE) {
this_cpu_write(percpu_swap_cluster.offset[order], next);
this_cpu_write(percpu_swap_cluster.si[order], si);
@@ -1012,13 +1168,19 @@ static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si,
return found;
}
-static unsigned int alloc_swap_scan_list(struct swap_info_struct *si,
- struct list_head *list,
- struct folio *folio,
- bool scan_all)
+static unsigned long alloc_swap_scan_list(struct swap_info_struct *si,
+ struct list_head *list,
+ struct folio *folio,
+ bool scan_all)
{
- unsigned int found = SWAP_ENTRY_INVALID;
+ unsigned long found = SWAP_ENTRY_INVALID;
+ bool nomem = false;
+ /*
+ * In rare cases alloc_swap_scan_cluster() can fail due to
+ * memcg_table allocation failure. Short-circuit to avoid looping
+ * over the list indefinitely.
+ */
do {
struct swap_cluster_info *ci = isolate_lock_cluster(si, list);
unsigned long offset;
@@ -1026,19 +1188,65 @@ static unsigned int alloc_swap_scan_list(struct swap_info_struct *si,
if (!ci)
break;
offset = cluster_offset(si, ci);
- found = alloc_swap_scan_cluster(si, ci, folio, offset);
+ found = alloc_swap_scan_cluster(si, ci, folio, offset, &nomem);
if (found)
break;
- } while (scan_all);
+ } while (scan_all && !nomem);
return found;
}
+static unsigned long vswap_alloc_cluster(struct swap_info_struct *si,
+ struct folio *folio)
+{
+ struct swap_cluster_info_dynamic *ci_dyn;
+ struct swap_cluster_info *ci;
+ unsigned long offset;
+
+ VM_WARN_ON(!swap_is_vswap(si));
+
+ ci_dyn = kzalloc_obj(*ci_dyn, GFP_ATOMIC);
+ if (!ci_dyn)
+ return SWAP_ENTRY_INVALID;
+
+ spin_lock_init(&ci_dyn->ci.lock);
+ INIT_LIST_HEAD(&ci_dyn->ci.list);
+
+ if (swap_cluster_alloc_table(si, &ci_dyn->ci, GFP_ATOMIC)) {
+ kfree(ci_dyn);
+ return SWAP_ENTRY_INVALID;
+ }
+
+ if (vswap_cluster_alloc_vtable(ci_dyn, GFP_ATOMIC)) {
+ swap_cluster_free_table(&ci_dyn->ci);
+ kfree(ci_dyn);
+ return SWAP_ENTRY_INVALID;
+ }
+
+ /* Lock before publishing: xa_alloc makes the cluster findable by offset. */
+ ci = &ci_dyn->ci;
+ spin_lock(&ci->lock);
+
+ if (xa_alloc(&si->cluster_info_pool, &ci_dyn->index, ci_dyn,
+ XA_LIMIT(1, DIV_ROUND_UP(si->max, SWAPFILE_CLUSTER) - 1),
+ GFP_ATOMIC)) {
+ spin_unlock(&ci->lock);
+ swap_cluster_free_table(&ci_dyn->ci);
+ vswap_cluster_free_vtable(&ci_dyn->ci);
+ kfree(ci_dyn);
+ return SWAP_ENTRY_INVALID;
+ }
+
+ offset = cluster_offset(si, ci);
+ return alloc_swap_scan_cluster(si, ci, folio, offset, NULL);
+}
+
static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool force)
{
long to_scan = 1;
unsigned long offset, end;
struct swap_cluster_info *ci;
+ swp_entry_t vswap_entry;
unsigned long swp_tb;
int nr_reclaim;
@@ -1056,7 +1264,22 @@ static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool force)
spin_unlock(&ci->lock);
nr_reclaim = __try_to_reclaim_swap(si, offset,
TTRS_ANYWAY);
- spin_lock(&ci->lock);
+ ci = swap_cluster_lock(si, offset);
+ if (!ci)
+ goto next;
+ if (nr_reclaim) {
+ offset += abs(nr_reclaim);
+ continue;
+ }
+ } else if (swp_tb_is_pointer(swp_tb) &&
+ (swp_tb & SWP_RMAP_CACHE_ONLY)) {
+ vswap_entry = swp_tb_ptr_to_swp_entry(swp_tb);
+ spin_unlock(&ci->lock);
+ nr_reclaim = try_to_reclaim_vswap_backing(si, offset,
+ vswap_entry);
+ ci = swap_cluster_lock(si, offset);
+ if (!ci)
+ goto next;
if (nr_reclaim) {
offset += abs(nr_reclaim);
continue;
@@ -1070,6 +1293,7 @@ static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool force)
relocate_cluster(si, ci);
swap_cluster_unlock(ci);
+next:
if (to_scan <= 0)
break;
@@ -1100,13 +1324,13 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,
{
struct swap_cluster_info *ci;
unsigned int order = likely(folio) ? folio_order(folio) : 0;
- unsigned int offset = SWAP_ENTRY_INVALID, found = SWAP_ENTRY_INVALID;
+ unsigned long offset = SWAP_ENTRY_INVALID, found = SWAP_ENTRY_INVALID;
/*
* Swapfile is not block device so unable
* to allocate large entries.
*/
- if (order && !(si->flags & SWP_BLKDEV))
+ if (order && !(si->flags & SWP_BLKDEV) && !swap_is_vswap(si))
return 0;
if (!(si->flags & SWP_SOLIDSTATE)) {
@@ -1121,7 +1345,8 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,
if (cluster_is_usable(ci, order)) {
if (cluster_is_empty(ci))
offset = cluster_offset(si, ci);
- found = alloc_swap_scan_cluster(si, ci, folio, offset);
+ found = alloc_swap_scan_cluster(si, ci, folio, offset,
+ NULL);
} else {
swap_cluster_unlock(ci);
}
@@ -1146,6 +1371,12 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,
goto done;
}
+ if (swap_is_vswap(si)) {
+ found = vswap_alloc_cluster(si, folio);
+ if (found)
+ goto done;
+ }
+
if (!(si->flags & SWP_PAGE_DISCARD)) {
found = alloc_swap_scan_list(si, &si->free_clusters, folio, false);
if (found)
@@ -1153,13 +1384,14 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,
}
/* Try reclaim full clusters if free and nonfull lists are drained */
- if (vm_swap_full())
+ if (!swap_is_vswap(si) && vm_swap_full())
swap_reclaim_full_clusters(si, false);
if (order < PMD_ORDER) {
/*
* Scan only one fragment cluster is good enough. Order 0
- * allocation will surely success, and large allocation
+ * allocation will surely success unless the memcg table
+ * allocation fails, which is rare, and large allocation
* failure is not critical. Scanning one cluster still
* keeps the list rotated and reclaimed (for clean swap cache).
*/
@@ -1175,7 +1407,8 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si,
for (int o = 1; o < SWAP_NR_ORDERS; o++) {
/*
* Clusters here have at least one usable slots and can't fail order 0
- * allocation, but reclaim may drop si->lock and race with another user.
+ * allocation, but reclaim may drop si->lock and race with another user,
+ * and the memcg table allocation may fail.
*/
found = alloc_swap_scan_list(si, &si->frag_clusters[o], folio, true);
if (found)
@@ -1282,8 +1515,10 @@ static bool swap_usage_add(struct swap_info_struct *si, unsigned int nr_entries)
/*
* If device is full, and SWAP_USAGE_OFFLIST_BIT is not set,
* remove it from the plist.
+ *
+ * Vswap is never on the avail list, so skip it.
*/
- if (unlikely(val == si->pages)) {
+ if (unlikely(val == si->pages) && !swap_is_vswap(si)) {
del_from_avail_list(si, false);
return true;
}
@@ -1298,8 +1533,10 @@ static void swap_usage_sub(struct swap_info_struct *si, unsigned int nr_entries)
/*
* If device is not full, and SWAP_USAGE_OFFLIST_BIT is set,
* add it to the plist.
+ *
+ * Vswap is never on the avail list, so skip it.
*/
- if (unlikely(val & SWAP_USAGE_OFFLIST_BIT))
+ if (unlikely(val & SWAP_USAGE_OFFLIST_BIT) && !swap_is_vswap(si))
add_to_avail_list(si, false);
}
@@ -1310,7 +1547,8 @@ static void swap_range_alloc(struct swap_info_struct *si,
if (vm_swap_full())
schedule_work(&si->reclaim_work);
}
- atomic_long_sub(nr_entries, &nr_swap_pages);
+ if (!swap_is_vswap(si))
+ atomic_long_sub(nr_entries, &nr_swap_pages);
}
static void swap_range_free(struct swap_info_struct *si, unsigned long offset,
@@ -1320,8 +1558,10 @@ static void swap_range_free(struct swap_info_struct *si, unsigned long offset,
void (*swap_slot_free_notify)(struct block_device *, unsigned long);
unsigned int i;
- for (i = 0; i < nr_entries; i++)
- zswap_invalidate(swp_entry(si->type, offset + i));
+ if (!swap_is_vswap(si)) {
+ for (i = 0; i < nr_entries; i++)
+ zswap_invalidate(swp_entry(si->type, offset + i));
+ }
if (si->flags & SWP_BLKDEV)
swap_slot_free_notify =
@@ -1340,12 +1580,17 @@ static void swap_range_free(struct swap_info_struct *si, unsigned long offset,
* only after the above cleanups are done.
*/
smp_wmb();
- atomic_long_add(nr_entries, &nr_swap_pages);
+ if (!swap_is_vswap(si))
+ atomic_long_add(nr_entries, &nr_swap_pages);
swap_usage_sub(si, nr_entries);
}
static bool get_swap_device_info(struct swap_info_struct *si)
{
+ /* The vswap device is always alive, so it needs no refcount. */
+ if (swap_is_vswap(si))
+ return true;
+
if (!percpu_ref_tryget_live(&si->users))
return false;
/*
@@ -1364,12 +1609,14 @@ static bool get_swap_device_info(struct swap_info_struct *si)
* Fast path try to get swap entries with specified order from current
* CPU's swap entry pool (a cluster).
*/
-static bool swap_alloc_fast(struct folio *folio)
+static swp_entry_t swap_alloc_fast(struct folio *folio)
{
unsigned int order = folio_order(folio);
struct swap_cluster_info *ci;
struct swap_info_struct *si;
- unsigned int offset;
+ unsigned long offset, found = 0;
+
+ lockdep_assert_held(&this_cpu_ptr(&percpu_swap_cluster)->lock);
/*
* Once allocated, swap_info_struct will never be completely freed,
@@ -1378,25 +1625,28 @@ static bool swap_alloc_fast(struct folio *folio)
si = this_cpu_read(percpu_swap_cluster.si[order]);
offset = this_cpu_read(percpu_swap_cluster.offset[order]);
if (!si || !offset || !get_swap_device_info(si))
- return false;
+ return (swp_entry_t){};
ci = swap_cluster_lock(si, offset);
- if (cluster_is_usable(ci, order)) {
+ if (ci && cluster_is_usable(ci, order)) {
if (cluster_is_empty(ci))
offset = cluster_offset(si, ci);
- alloc_swap_scan_cluster(si, ci, folio, offset);
- } else {
+ found = alloc_swap_scan_cluster(si, ci, folio, offset, NULL);
+ } else if (ci) {
swap_cluster_unlock(ci);
}
put_swap_device(si);
- return folio_test_swapcache(folio);
+ if (found)
+ return swp_entry(si->type, found);
+ return (swp_entry_t){};
}
/* Rotate the device and switch to a new cluster */
-static void swap_alloc_slow(struct folio *folio)
+static swp_entry_t swap_alloc_slow(struct folio *folio)
{
struct swap_info_struct *si, *next;
+ unsigned long found;
spin_lock(&swap_avail_lock);
start_over:
@@ -1405,12 +1655,12 @@ static void swap_alloc_slow(struct folio *folio)
plist_requeue(&si->avail_list, &swap_avail_head);
spin_unlock(&swap_avail_lock);
if (get_swap_device_info(si)) {
- cluster_alloc_swap_entry(si, folio);
+ found = cluster_alloc_swap_entry(si, folio);
put_swap_device(si);
- if (folio_test_swapcache(folio))
- return;
+ if (found)
+ return swp_entry(si->type, found);
if (folio_test_large(folio))
- return;
+ return (swp_entry_t){};
}
spin_lock(&swap_avail_lock);
@@ -1428,6 +1678,7 @@ static void swap_alloc_slow(struct folio *folio)
goto start_over;
}
spin_unlock(&swap_avail_lock);
+ return (swp_entry_t){};
}
/*
@@ -1507,6 +1758,7 @@ int swap_retry_table_alloc(swp_entry_t entry, gfp_t gfp)
if (!si)
return 0;
+ /* The source PTE pins the entry, so its cluster is alive. */
ci = __swap_offset_to_cluster(si, offset);
ret = swap_extend_table_alloc(si, ci, swp_cluster_offset(entry), gfp);
@@ -1611,6 +1863,8 @@ static void swap_put_entries_cluster(struct swap_info_struct *si,
}
/* count will be 0 after put, slot can be reclaimed */
need_reclaim = true;
+ if (swap_is_vswap(si))
+ vswap_mark_cache_only(ci, ci_off);
}
/*
* A count != 1 or cached slot can't be freed. Put its swap
@@ -1717,6 +1971,8 @@ static int swap_dup_entries_cluster(struct swap_info_struct *si,
goto failed;
}
} while (++ci_off < ci_end);
+ if (swap_is_vswap(si))
+ vswap_clear_cache_only(ci, ci_start, nr);
swap_cluster_unlock(ci);
return 0;
failed:
@@ -1727,6 +1983,76 @@ static int swap_dup_entries_cluster(struct swap_info_struct *si,
return err;
}
+static bool vswap_alloc(struct folio *folio)
+{
+ unsigned int order = folio_order(folio);
+ struct swap_cluster_info *ci;
+ struct obj_cgroup *objcg;
+ unsigned long offset;
+ bool may_zswap;
+
+ if (!vswap_is_enabled() || !zswap_is_enabled())
+ return false;
+
+ /*
+ * If zswap will not take the folio, writeout has to find a physical
+ * slot anyway. We are just incurring indirection overhead
+ * unnecessarily.
+ */
+ objcg = get_obj_cgroup_from_folio(folio);
+ may_zswap = !objcg || obj_cgroup_may_zswap(objcg);
+ if (objcg)
+ obj_cgroup_put(objcg);
+ if (!may_zswap)
+ return false;
+
+ local_lock(&percpu_vswap_cluster.lock);
+ offset = this_cpu_read(percpu_vswap_cluster.offset[order]);
+
+ if (offset != SWAP_ENTRY_INVALID) {
+ ci = swap_cluster_lock(vswap_si, offset);
+ if (ci && cluster_is_usable(ci, order)) {
+ if (cluster_is_empty(ci))
+ offset = cluster_offset(vswap_si, ci);
+ alloc_swap_scan_cluster(vswap_si, ci, folio, offset,
+ NULL);
+ } else if (ci) {
+ swap_cluster_unlock(ci);
+ }
+ }
+
+ if (!folio_test_swapcache(folio))
+ cluster_alloc_swap_entry(vswap_si, folio);
+
+ if (folio_test_swapcache(folio)) {
+ /* alloc_swap_scan_cluster updated percpu offset already */
+ local_unlock(&percpu_vswap_cluster.lock);
+ return true;
+ }
+
+ this_cpu_write(percpu_vswap_cluster.offset[order], SWAP_ENTRY_INVALID);
+ local_unlock(&percpu_vswap_cluster.lock);
+ atomic_long_add(folio_nr_pages(folio), &vswap_alloc_reject);
+ return false;
+}
+
+static swp_entry_t folio_alloc_phys_swap(struct folio *folio)
+{
+ swp_entry_t entry;
+
+again:
+ local_lock(&percpu_swap_cluster.lock);
+ entry = swap_alloc_fast(folio);
+ if (!entry.val)
+ entry = swap_alloc_slow(folio);
+ local_unlock(&percpu_swap_cluster.lock);
+
+ if (!entry.val && !folio_order(folio) && swap_sync_discard())
+ goto again;
+
+ return entry;
+}
+
/**
* folio_alloc_swap - allocate swap space for a folio
* @folio: folio we want to move to swap
@@ -1740,6 +2066,7 @@ static int swap_dup_entries_cluster(struct swap_info_struct *si,
int folio_alloc_swap(struct folio *folio)
{
unsigned int order = folio_order(folio);
+ struct mem_cgroup *memcg;
unsigned int size = 1 << order;
VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
@@ -1763,27 +2090,350 @@ int folio_alloc_swap(struct folio *folio)
}
}
-again:
- local_lock(&percpu_swap_cluster.lock);
- if (!swap_alloc_fast(folio))
- swap_alloc_slow(folio);
- local_unlock(&percpu_swap_cluster.lock);
+ if (!vswap_alloc(folio))
+ folio_alloc_phys_swap(folio);
- if (!order && unlikely(!folio_test_swapcache(folio))) {
- if (swap_sync_discard())
- goto again;
+ /*
+ * Need to call this even if allocation failed, for MEMCG_SWAP_FAIL.
+ * A vswap entry has no physical swap yet, so only record the memcg.
+ * folio_realloc_swap() charges it once backing is allocated.
+ */
+ memcg = mem_cgroup_swap_get(folio);
+ if (memcg) {
+ if (!is_vswap_entry(folio->swap) &&
+ unlikely(mem_cgroup_swap_charge(memcg, size))) {
+ mem_cgroup_swap_put(memcg, size);
+ swap_cache_del_folio(folio);
+ } else {
+ mem_cgroup_swap_record(folio, memcg);
+ }
}
- /* Need to call this even if allocation failed, for MEMCG_SWAP_FAIL. */
- if (unlikely(mem_cgroup_try_charge_swap(folio)))
- swap_cache_del_folio(folio);
-
if (unlikely(!folio_test_swapcache(folio)))
return -ENOMEM;
return 0;
}
+static void vswap_mark_cache_only(struct swap_cluster_info *ci,
+ unsigned int ci_off)
+{
+ struct swap_cluster_info_dynamic *ci_dyn;
+ struct swap_cluster_info *pci;
+ swp_entry_t phys;
+ unsigned long vt;
+
+ ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
+ vt = __vtable_get(ci_dyn, ci_off);
+
+ if (vtable_type(vt) == VSWAP_SWAPFILE) {
+ phys = vtable_to_phys(vt);
+ pci = __swap_entry_to_cluster(phys);
+ swap_rmap_mark_cache_only(pci, swp_cluster_offset(phys));
+ }
+}
+
+/*
+ * Clear the cache-only rmap hint for entries re-referenced from count 0 to 1
+ * (no longer reclaimable), so the physical reclaim scanner skips them.
+ */
+static void vswap_clear_cache_only(struct swap_cluster_info *ci,
+ unsigned int ci_start, int nr)
+{
+ struct swap_cluster_info_dynamic *ci_dyn;
+ struct swap_cluster_info *pci;
+ unsigned long swp_tb, vt;
+ swp_entry_t phys;
+ unsigned int off;
+
+ ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
+ for (off = ci_start; off < ci_start + nr; off++) {
+ swp_tb = __swap_table_get(ci, off);
+ if (!swp_tb_is_folio(swp_tb) || swp_tb_get_count(swp_tb) != 1)
+ continue;
+ vt = __vtable_get(ci_dyn, off);
+ if (vtable_type(vt) != VSWAP_SWAPFILE)
+ continue;
+ phys = vtable_to_phys(vt);
+ pci = __swap_entry_to_cluster(phys);
+ swap_rmap_clear_cache_only(pci, swp_cluster_offset(phys));
+ }
+}
+
+static void __swap_cluster_free_phys_backing(struct swap_info_struct *psi,
+ struct swap_cluster_info *pci,
+ unsigned int ci_start,
+ unsigned int nr_pages);
+
+static void vswap_uncharge_cgroup_batch(unsigned short memcg_id,
+ unsigned int batch_nr,
+ unsigned int batch_nr_swapfile)
+{
+ struct mem_cgroup *memcg;
+ unsigned int n;
+
+ /*
+ * v1 (memsw): entries keep their memsw charge across swapout
+ * regardless of backing, so uncharge all of them. v2: only
+ * swapfile-backed entries are charged, so uncharge just those.
+ *
+ * On v1 the id is written by __memcg1_swapout() as the folio leaves the
+ * swap cache and cleared by memcg1_swapin() when it comes back, both
+ * under the cluster lock. Callers still holding a cached folio are
+ * outside that window and see @memcg_id == 0, so only the free path
+ * uncharges. On v2 the id is set when swap is allocated, so those
+ * callers do uncharge, which balances the charge folio_realloc_swap()
+ * took.
+ */
+ n = do_memsw_account() ? batch_nr : batch_nr_swapfile;
+ if (!n)
+ return;
+
+ rcu_read_lock();
+ memcg = memcg_id ? mem_cgroup_from_private_id(memcg_id) : NULL;
+ rcu_read_unlock();
+ mem_cgroup_swap_uncharge(memcg, n);
+}
+
+/**
+ * __vswap_release_backing - release the backing of a range of vtable slots
+ * @ci: the locked vswap cluster
+ * @ci_start: first slot offset within @ci
+ * @nr: number of slots
+ *
+ * Releases the backing of each slot in [@ci_start, @ci_start + @nr).
+ * Clears the zero marks if set.
+ *
+ * Context: caller must hold @ci->lock.
+ */
+void __vswap_release_backing(struct swap_cluster_info *ci,
+ unsigned int ci_start, unsigned int nr)
+{
+ struct swap_cluster_info_dynamic *ci_dyn;
+ struct swap_info_struct *psi;
+ unsigned long phys_start = 0, phys_end = 0;
+ unsigned int phys_type = 0;
+ unsigned int ci_off;
+ unsigned long vt;
+ swp_entry_t phys;
+ unsigned short batch_id;
+ unsigned int batch_nr = 0, batch_nr_swapfile = 0;
+
+ lockdep_assert_held(&ci->lock);
+ ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
+ batch_id = __swap_cgroup_get(ci, ci_start);
+
+ for (ci_off = ci_start; ci_off < ci_start + nr; ci_off++) {
+ unsigned short cur_id;
+
+ vt = __vtable_get(ci_dyn, ci_off);
+ cur_id = __swap_cgroup_get(ci, ci_off);
+
+ if (cur_id != batch_id) {
+ vswap_uncharge_cgroup_batch(batch_id, batch_nr,
+ batch_nr_swapfile);
+ batch_id = cur_id;
+ batch_nr = 0;
+ batch_nr_swapfile = 0;
+ }
+ batch_nr++;
+
+ /* The free helper takes one contiguous run within one cluster. */
+ if (phys_start != phys_end &&
+ (vtable_type(vt) != VSWAP_SWAPFILE ||
+ swp_type(vtable_to_phys(vt)) != phys_type ||
+ swp_offset(vtable_to_phys(vt)) != phys_end ||
+ phys_end % SWAPFILE_CLUSTER == 0)) {
+ psi = __swap_type_to_info(phys_type);
+ __swap_cluster_free_phys_backing(psi,
+ __swap_entry_to_cluster(
+ swp_entry(phys_type, phys_start)),
+ phys_start % SWAPFILE_CLUSTER,
+ phys_end - phys_start);
+ phys_start = phys_end = 0;
+ }
+
+ switch (vtable_type(vt)) {
+ case VSWAP_SWAPFILE:
+ batch_nr_swapfile++;
+ if (phys_start == phys_end) {
+ phys = vtable_to_phys(vt);
+ phys_start = swp_offset(phys);
+ phys_end = phys_start + 1;
+ phys_type = swp_type(phys);
+ } else {
+ phys_end++;
+ }
+ break;
+ case VSWAP_ZSWAP:
+ zswap_entry_free(vtable_to_zswap(vt));
+ break;
+ case VSWAP_NONE:
+ break;
+ default:
+ /* VSWAP_ZERO/VSWAP_FOLIO are return-only, not vtable tags */
+ break;
+ }
+
+ __vtable_set(ci_dyn, ci_off, VSWAP_NONE);
+ /* Zero-backed state lives in swap_table; clear it too. */
+ if (__swap_table_test_zero(ci, ci_off))
+ __swap_table_clear_zero(ci, ci_off);
+ }
+
+ if (phys_start != phys_end) {
+ psi = __swap_type_to_info(phys_type);
+ __swap_cluster_free_phys_backing(psi,
+ __swap_entry_to_cluster(
+ swp_entry(phys_type, phys_start)),
+ phys_start % SWAPFILE_CLUSTER,
+ phys_end - phys_start);
+ }
+
+ vswap_uncharge_cgroup_batch(batch_id, batch_nr, batch_nr_swapfile);
+}
+
+/**
+ * folio_release_vswap_backing() - Drop all backing for a folio's vswap entry.
+ * @folio: the folio, occupying a virtual swap entry.
+ *
+ * Release whatever backing the folio's virtual swap slots currently hold and
+ * reset them to empty, so a fresh backing can be installed. Used when a
+ * folio's swap backend is replaced.
+ *
+ * Context: Caller must hold the folio lock; @folio must be in the swap cache
+ * and occupy a virtual swap entry.
+ */
+void folio_release_vswap_backing(struct folio *folio)
+{
+ struct swap_cluster_info *ci;
+ int nr = folio_nr_pages(folio);
+ unsigned int voff;
+
+ ci = __swap_entry_to_cluster(folio->swap);
+ voff = swp_cluster_offset(folio->swap);
+
+ spin_lock(&ci->lock);
+ __vswap_release_backing(ci, voff, nr);
+ spin_unlock(&ci->lock);
+}
+
+/**
+ * folio_release_non_phys_swap_backing() - Drop a folio's non-physical vswap backing.
+ * @folio: the folio, occupying a virtual swap entry.
+ *
+ * Release the zswap backing recorded for @folio's virtual swap entry,
+ * leaving the slots empty so the writeout path can install fresh physical
+ * backing. Does nothing when the entry is already backed by physical
+ * swapfile slots, which are kept for reuse, or when it has no backing
+ * beyond the swap cache folio itself.
+ *
+ * Context: Caller must hold the folio lock; @folio must be in the swap cache
+ * and occupy a virtual swap entry.
+ */
+void folio_release_non_phys_swap_backing(struct folio *folio)
+{
+ struct swap_cluster_info *ci;
+ struct swap_cluster_info_dynamic *ci_dyn;
+ int nr = folio_nr_pages(folio);
+ unsigned int voff;
+ unsigned long vt;
+ enum vswap_backing_type type;
+
+ ci = __swap_entry_to_cluster(folio->swap);
+ ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
+ voff = swp_cluster_offset(folio->swap);
+
+ spin_lock(&ci->lock);
+ /* A folio's slots cannot mix swapfile with other backends. */
+ vt = __vtable_get(ci_dyn, voff);
+ type = vtable_type(vt);
+
+ if (type == VSWAP_SWAPFILE || type == VSWAP_NONE) {
+ spin_unlock(&ci->lock);
+ return;
+ }
+
+ __vswap_release_backing(ci, voff, nr);
+ spin_unlock(&ci->lock);
+}
+
+/**
+ * folio_realloc_swap() - Back a virtual swap folio with a physical swap slot.
+ * @folio: the folio, occupying a virtual swap entry.
+ *
+ * Ensure @folio's virtual swap entry has physical (swapfile) backing,
+ * allocating a physical slot on demand if it has none. If @folio is
+ * already physically backed, the existing physical entry is returned
+ * unchanged.
+ *
+ * Context: Caller must hold the folio lock; @folio must be in the swap cache
+ * and occupy a virtual swap entry.
+ * Return: The physical swap entry now backing @folio, or an empty entry
+ * (.val == 0) on failure.
+ */
+swp_entry_t folio_realloc_swap(struct folio *folio)
+{
+ swp_entry_t vswap_entry = folio->swap;
+ struct swap_cluster_info *ci;
+ struct swap_cluster_info_dynamic *ci_dyn;
+ struct mem_cgroup *memcg;
+ unsigned int voff;
+ unsigned long vt;
+ unsigned short memcg_id;
+ swp_entry_t phys_entry = {};
+ swp_entry_t pe;
+ int i, nr = folio_nr_pages(folio);
+
+ VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
+ VM_BUG_ON_FOLIO(!folio_test_swapcache(folio), folio);
+ VM_WARN_ON(!is_vswap_entry(vswap_entry));
+
+ voff = swp_cluster_offset(vswap_entry);
+ ci = __swap_entry_to_cluster(vswap_entry);
+ ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
+
+ spin_lock(&ci->lock);
+ vt = __vtable_get(ci_dyn, voff);
+ if (vtable_type(vt) == VSWAP_SWAPFILE) {
+ spin_unlock(&ci->lock);
+ return vtable_to_phys(vt);
+ }
+ memcg_id = __swap_cgroup_get(ci, voff);
+ spin_unlock(&ci->lock);
+
+ phys_entry = folio_alloc_phys_swap(folio);
+ if (!phys_entry.val)
+ return (swp_entry_t){};
+
+ rcu_read_lock();
+ memcg = folio_memcg(folio);
+ if (!memcg || mem_cgroup_private_id(memcg) != memcg_id)
+ memcg = memcg_id ? mem_cgroup_from_private_id(memcg_id) : NULL;
+ rcu_read_unlock();
+
+ if (mem_cgroup_swap_charge(memcg, nr)) {
+ __swap_cluster_free_phys_backing(__swap_entry_to_info(phys_entry),
+ __swap_entry_to_cluster(phys_entry),
+ swp_cluster_offset(phys_entry),
+ nr);
+ return (swp_entry_t){};
+ }
+
+ spin_lock(&ci->lock);
+ /*
+ * Install PHYS backing without freeing any prior contents of the
+ * vtable. Releasing the old backing is the caller's job.
+ */
+ for (i = 0; i < nr; i++) {
+ pe.val = phys_entry.val + i;
+ __vtable_set(ci_dyn, voff + i, vtable_mk_phys(pe));
+ }
+ spin_unlock(&ci->lock);
+
+ return phys_entry;
+}
+
/**
* folio_dup_swap() - Increase swap count of swap entries of a folio.
* @folio: folio with swap entries bounded.
@@ -1904,10 +2554,70 @@ struct swap_info_struct *get_swap_device(swp_entry_t entry)
return NULL;
put_out:
pr_err_ratelimited("%s: %s%08lx\n", __func__, Bad_offset, entry.val);
- percpu_ref_put(&si->users);
+ put_swap_device(si);
return NULL;
}
+/*
+ * Common tail for freeing swap slots: device-level accounting
+ * and cluster list management.
+ */
+static void __swap_cluster_finish_free(struct swap_info_struct *si,
+ struct swap_cluster_info *ci,
+ unsigned int ci_start,
+ unsigned int nr_pages)
+{
+ lockdep_assert_held(&ci->lock);
+ swap_range_free(si, cluster_offset(si, ci) + ci_start, nr_pages);
+ swap_cluster_assert_empty(ci, ci_start, nr_pages, false);
+
+ if (!ci->count)
+ free_cluster(si, ci);
+ else
+ partial_free_cluster(si, ci);
+}
+
+/*
+ * Free physical swap slots that were backing vswap entries (Pointer-tagged).
+ */
+static void __swap_cluster_free_phys_backing(struct swap_info_struct *psi,
+ struct swap_cluster_info *pci,
+ unsigned int ci_start,
+ unsigned int nr_pages)
+{
+ unsigned int ci_off;
+
+ spin_lock_nested(&pci->lock, SINGLE_DEPTH_NESTING);
+ VM_WARN_ON(pci->count < nr_pages);
+ pci->count -= nr_pages;
+ for (ci_off = ci_start; ci_off < ci_start + nr_pages; ci_off++) {
+ __swap_table_set(pci, ci_off, null_to_swp_tb());
+ if (!SWAP_TABLE_HAS_ZEROFLAG)
+ __swap_table_clear_zero(pci, ci_off);
+ }
+ __swap_cluster_finish_free(psi, pci, ci_start, nr_pages);
+ swap_cluster_unlock(pci);
+}
+
+/*
+ * Release the cgroup accounting of a batch of freed slots. For vswap the
+ * physical swap was already uncharged by __vswap_release_backing(), so only
+ * the ID ref is left to drop.
+ */
+static void memcg_swap_free(unsigned short id, unsigned int nr, bool is_vswap)
+{
+ struct mem_cgroup *memcg;
+
+ rcu_read_lock();
+ memcg = mem_cgroup_from_private_id(id);
+ if (memcg) {
+ if (!is_vswap)
+ mem_cgroup_swap_uncharge(memcg, nr);
+ mem_cgroup_swap_put(memcg, nr);
+ }
+ rcu_read_unlock();
+}
+
/*
* Free a set of swap slots after their swap count dropped to zero, or will be
* zero after putting the last ref (saves one __swap_cluster_put_entry call).
@@ -1919,11 +2629,14 @@ void __swap_cluster_free_entries(struct swap_info_struct *si,
unsigned long old_tb;
unsigned short batch_id = 0, id_cur;
unsigned int ci_off = ci_start, ci_end = ci_start + nr_pages;
- unsigned long ci_head = cluster_offset(si, ci);
unsigned int batch_off = ci_off;
+ bool is_vswap = swap_is_vswap(si);
VM_WARN_ON(ci->count < nr_pages);
+ if (is_vswap)
+ __vswap_release_backing(ci, ci_start, nr_pages);
+
ci->count -= nr_pages;
do {
old_tb = __swap_table_get(ci, ci_off);
@@ -1945,22 +2658,16 @@ void __swap_cluster_free_entries(struct swap_info_struct *si,
id_cur = __swap_cgroup_clear(ci, ci_off, 1);
if (batch_id != id_cur) {
if (batch_id)
- mem_cgroup_uncharge_swap(batch_id, ci_off - batch_off);
+ memcg_swap_free(batch_id, ci_off - batch_off, is_vswap);
batch_id = id_cur;
batch_off = ci_off;
}
} while (++ci_off < ci_end);
if (batch_id)
- mem_cgroup_uncharge_swap(batch_id, ci_off - batch_off);
-
- swap_range_free(si, ci_head + ci_start, nr_pages);
- swap_cluster_assert_empty(ci, ci_start, nr_pages, false);
+ memcg_swap_free(batch_id, ci_off - batch_off, is_vswap);
- if (!ci->count)
- free_cluster(si, ci);
- else
- partial_free_cluster(si, ci);
+ __swap_cluster_finish_free(si, ci, ci_start, nr_pages);
}
int __swap_count(swp_entry_t entry)
@@ -2036,6 +2743,7 @@ static bool folio_maybe_swapped(struct folio *folio)
VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
VM_WARN_ON_ONCE_FOLIO(!folio_test_swapcache(folio), folio);
+ /* Folio is locked and in swap cache, so ci->count > 0: cluster is alive. */
ci = __swap_entry_to_cluster(entry);
ci_off = swp_cluster_offset(entry);
ci_end = ci_off + folio_nr_pages(folio);
@@ -2174,7 +2882,8 @@ swp_entry_t swap_alloc_hibernation_slot(int type)
if (pcp_si == si && pcp_offset) {
ci = swap_cluster_lock(si, pcp_offset);
if (cluster_is_usable(ci, 0))
- offset = alloc_swap_scan_cluster(si, ci, NULL, pcp_offset);
+ offset = alloc_swap_scan_cluster(si, ci, NULL,
+ pcp_offset, NULL);
else
swap_cluster_unlock(ci);
}
@@ -2230,6 +2939,9 @@ static int __find_hibernation_swap_type(dev_t device, sector_t offset)
if (!(sis->flags & SWP_WRITEOK))
continue;
+ /* vswap has no bdev, so it is never a hibernation target. */
+ if (swap_is_vswap(sis))
+ continue;
if (device == sis->bdev->bd_dev) {
struct swap_extent *se = first_se(sis);
@@ -2356,6 +3068,9 @@ int find_first_swap(dev_t *device)
if (!(sis->flags & SWP_WRITEOK))
continue;
+ /* vswap has no bdev, so it is never a hibernation target. */
+ if (swap_is_vswap(sis))
+ continue;
*device = sis->bdev->bd_dev;
spin_unlock(&swap_lock);
return type;
@@ -2572,8 +3287,10 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd,
&vmf);
}
if (!folio) {
+ rcu_read_lock();
swp_tb = swap_table_get(__swap_entry_to_cluster(entry),
swp_cluster_offset(entry));
+ rcu_read_unlock();
if (swp_tb_get_count(swp_tb) <= 0)
continue;
return -ENOMEM;
@@ -2706,10 +3423,10 @@ static int unuse_mm(struct mm_struct *mm, unsigned int type)
* Return 0 if there are no inuse entries after prev till end of
* the map.
*/
-static unsigned int find_next_to_unuse(struct swap_info_struct *si,
- unsigned int prev)
+static unsigned long find_next_to_unuse(struct swap_info_struct *si,
+ unsigned long prev)
{
- unsigned int i;
+ unsigned long i;
unsigned long swp_tb;
/*
@@ -2719,8 +3436,10 @@ static unsigned int find_next_to_unuse(struct swap_info_struct *si,
* allocations from this area (while holding swap_lock).
*/
for (i = prev + 1; i < si->max; i++) {
+ rcu_read_lock();
swp_tb = swap_table_get(__swap_offset_to_cluster(si, i),
i % SWAPFILE_CLUSTER);
+ rcu_read_unlock();
if (!swp_tb_is_null(swp_tb) && !swp_tb_is_bad(swp_tb))
break;
if ((i % LATENCY_LIMIT) == 0)
@@ -2735,19 +3454,100 @@ static unsigned int find_next_to_unuse(struct swap_info_struct *si,
static int try_to_unuse(unsigned int type)
{
+ struct mempolicy *mpol = get_task_policy(current);
struct mm_struct *prev_mm;
struct mm_struct *mm;
struct list_head *p;
int retval = 0;
struct swap_info_struct *si = swap_info[type];
struct folio *folio;
- swp_entry_t entry;
- unsigned int i;
+ struct swap_io_ctx ctx;
+ swp_entry_t entry, vswap_entry;
+ unsigned long swp_tb;
+ unsigned long i;
+ unsigned int j;
if (!swap_usage_in_pages(si))
goto success;
retry:
+ /*
+ * Free vswap-backing slots (Pointer-tagged) first. Walk physical
+ * clusters, read the vswap entry from the rmap, ensure the data
+ * is in the swap cache, and transition PHYS to FOLIO. Freeing the
+ * physical backing is enough, so no page table walk is needed.
+ */
+ i = 0;
+ while (vswap_is_enabled() &&
+ swap_usage_in_pages(si) &&
+ !signal_pending(current) &&
+ (i = find_next_to_unuse(si, i)) != 0) {
+ swp_entry_t phys;
+
+ swp_tb = swap_table_get(__swap_offset_to_cluster(si, i),
+ i % SWAPFILE_CLUSTER);
+ if (!swp_tb_is_pointer(swp_tb))
+ continue;
+
+ vswap_entry = swp_tb_ptr_to_swp_entry(swp_tb);
+
+ folio = swap_cache_get_folio(vswap_entry);
+ if (!folio) {
+ folio = swap_cache_alloc_folio(vswap_entry,
+ GFP_HIGHUSER_MOVABLE,
+ BIT(0), NULL, mpol,
+ NO_INTERLEAVE_INDEX);
+ if (IS_ERR(folio)) {
+ if (PTR_ERR(folio) == -ENOMEM)
+ return -ENOMEM;
+ continue;
+ }
+ ctx = (struct swap_io_ctx){};
+ swap_read_folio(&ctx, folio);
+ swap_read_submit(&ctx);
+ folio_lock(folio);
+ } else {
+ folio_lock(folio);
+ }
+
+ if (!folio_matches_swap_entry(folio, vswap_entry)) {
+ folio_unlock(folio);
+ folio_put(folio);
+ continue;
+ }
+
+ /*
+ * Re-validate under folio lock: rmap holds folio->swap + j
+ * for some j in [0, nr_pages). Check folio->swap still maps
+ * to the contiguous physical run that includes our slot i.
+ */
+ j = vswap_entry.val - folio->swap.val;
+ phys = vswap_to_phys(folio->swap);
+ if (!phys.val || swp_type(phys) != type ||
+ swp_offset(phys) + j != i) {
+ folio_unlock(folio);
+ folio_put(folio);
+ continue;
+ }
+
+ folio_wait_writeback(folio);
+ folio_release_vswap_backing(folio);
+ /*
+ * Drop a folio whose read failed rather than dirtying
+ * uninitialised memory; the next fault finds no backing and
+ * gets SIGBUS.
+ */
+ if (unlikely(!folio_test_uptodate(folio)))
+ swap_cache_del_folio(folio);
+ else
+ folio_mark_dirty(folio);
+ folio_unlock(folio);
+ folio_put(folio);
+ }
+
+ if (!swap_usage_in_pages(si))
+ goto success;
+
retval = shmem_unuse(type);
if (retval)
return retval;
@@ -2790,6 +3590,8 @@ static int try_to_unuse(unsigned int type)
(i = find_next_to_unuse(si, i)) != 0) {
entry = swp_entry(type, i);
+
+ /* Pointer-tagged rmap slots have no folio; the pre-pass took them. */
folio = swap_cache_get_folio(entry);
if (!folio)
continue;
@@ -2959,6 +3761,11 @@ static int setup_swap_extents(struct swap_info_struct *sis,
struct inode *inode = mapping->host;
int ret;
+ if (swap_is_vswap(sis)) {
+ *span = 0;
+ return 0;
+ }
+
ret = sio_pool_init();
if (ret)
return ret;
@@ -2984,15 +3791,24 @@ static int setup_swap_extents(struct swap_info_struct *sis,
static void _enable_swap_info(struct swap_info_struct *si)
{
- atomic_long_add(si->pages, &nr_swap_pages);
- total_swap_pages += si->pages;
+ if (!swap_is_vswap(si)) {
+ atomic_long_add(si->pages, &nr_swap_pages);
+ total_swap_pages += si->pages;
+ }
assert_spin_locked(&swap_lock);
- plist_add(&si->list, &swap_active_head);
+ /*
+ * Vswap has no backing file and no swapoff support, so keep it
+ * off swap_active_head (used by swapoff filename lookup and
+ * swap_sync_discard) and swap_avail_head (physical allocator).
+ */
+ if (!swap_is_vswap(si)) {
+ plist_add(&si->list, &swap_active_head);
- /* Add back to available list */
- add_to_avail_list(si, true);
+ /* Add back to available list */
+ add_to_avail_list(si, true);
+ }
}
/*
@@ -3036,12 +3852,32 @@ static void wait_for_allocation(struct swap_info_struct *si)
}
}
-static void free_swap_cluster_info(struct swap_cluster_info *cluster_info,
+static void free_swap_cluster_info(struct swap_info_struct *si,
+ struct swap_cluster_info *cluster_info,
unsigned long maxpages)
{
+ struct swap_cluster_info_dynamic *ci_dyn;
struct swap_cluster_info *ci;
+ unsigned long idx;
int i, nr_clusters = DIV_ROUND_UP(maxpages, SWAPFILE_CLUSTER);
+ if (swap_is_vswap(si)) {
+ xa_for_each(&si->cluster_info_pool, idx, ci_dyn) {
+ ci = &ci_dyn->ci;
+ spin_lock(&ci->lock);
+ if (cluster_table_is_alloced(ci)) {
+ swap_cluster_assert_empty(ci, 0,
+ SWAPFILE_CLUSTER, true);
+ swap_cluster_free_table(ci);
+ }
+ spin_unlock(&ci->lock);
+ vswap_cluster_free_vtable(ci);
+ kfree(ci_dyn);
+ }
+ xa_destroy(&si->cluster_info_pool);
+ return;
+ }
+
if (!cluster_info)
return;
for (i = 0; i < nr_clusters; i++) {
@@ -3086,7 +3922,7 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
struct file *swap_file, *victim;
struct address_space *mapping;
struct inode *inode;
- unsigned int maxpages;
+ unsigned long maxpages;
int err, found = 0;
if (!capable(CAP_SYS_ADMIN))
@@ -3188,7 +4024,7 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
mutex_unlock(&swapon_mutex);
kfree(p->global_cluster);
p->global_cluster = NULL;
- free_swap_cluster_info(cluster_info, maxpages);
+ free_swap_cluster_info(p, cluster_info, maxpages);
inode = mapping->host;
@@ -3508,12 +4344,8 @@ static unsigned long read_swap_header(struct swap_info_struct *si,
pr_warn("Truncating oversized swap area, only using %luk out of %luk\n",
K(maxpages), K(last_page));
}
- if (maxpages > last_page) {
+ if (maxpages > last_page)
maxpages = last_page + 1;
- /* p->max is an unsigned int: don't overflow it */
- if ((unsigned int)maxpages == 0)
- maxpages = UINT_MAX;
- }
if (!maxpages)
return 0;
@@ -3535,10 +4367,43 @@ static int setup_swap_clusters_info(struct swap_info_struct *si,
unsigned long maxpages)
{
unsigned long nr_clusters = DIV_ROUND_UP(maxpages, SWAPFILE_CLUSTER);
- struct swap_cluster_info *cluster_info;
+ struct swap_cluster_info *cluster_info = NULL;
+ struct swap_cluster_info_dynamic *ci_dyn = NULL;
int err = -ENOMEM;
unsigned long i;
+ /* A vswap device uses an xarray pool instead of a static array. */
+ if (swap_is_vswap(si)) {
+ nr_clusters = 0;
+ xa_init_flags(&si->cluster_info_pool, XA_FLAGS_ALLOC);
+
+ /*
+ * Pre-allocate cluster 0 and mark slot 0 (header page)
+ * as bad so the allocator never hands out page offset 0.
+ */
+ ci_dyn = kzalloc_obj(*ci_dyn, GFP_KERNEL);
+ if (!ci_dyn)
+ goto err;
+ spin_lock_init(&ci_dyn->ci.lock);
+ INIT_LIST_HEAD(&ci_dyn->ci.list);
+
+ err = xa_insert(&si->cluster_info_pool, 0, ci_dyn, GFP_KERNEL);
+ if (err) {
+ kfree(ci_dyn);
+ goto err;
+ }
+
+ err = swap_cluster_setup_bad_slot(si, &ci_dyn->ci, 0, false);
+ if (err)
+ goto err;
+
+ err = vswap_cluster_alloc_vtable(ci_dyn, GFP_KERNEL);
+ if (err)
+ goto err;
+
+ goto setup_cluster_info;
+ }
+
cluster_info = kvzalloc_objs(*cluster_info, nr_clusters);
if (!cluster_info)
goto err;
@@ -3582,6 +4447,7 @@ static int setup_swap_clusters_info(struct swap_info_struct *si,
goto err;
}
+setup_cluster_info:
INIT_LIST_HEAD(&si->free_clusters);
INIT_LIST_HEAD(&si->full_clusters);
INIT_LIST_HEAD(&si->discard_clusters);
@@ -3603,10 +4469,16 @@ static int setup_swap_clusters_info(struct swap_info_struct *si,
}
}
+ /* Slot 0 is bad, so cluster 0 never empties. The rest of it is usable. */
+ if (swap_is_vswap(si)) {
+ ci_dyn->ci.flags = CLUSTER_FLAG_NONFULL;
+ list_add_tail(&ci_dyn->ci.list, &si->nonfull_clusters[0]);
+ }
+
si->cluster_info = cluster_info;
return 0;
err:
- free_swap_cluster_info(cluster_info, maxpages);
+ free_swap_cluster_info(si, cluster_info, maxpages);
return err;
}
@@ -3714,7 +4586,7 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
goto bad_swap_unlock_inode;
}
if (si->pages != si->max - 1) {
- pr_err("swap:%u != (max:%u - 1)\n", si->pages, si->max);
+ pr_err("swap:%lu != (max:%lu - 1)\n", si->pages, si->max);
error = -EINVAL;
goto bad_swap_unlock_inode;
}
@@ -3802,7 +4674,7 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
/* Sets SWP_WRITEOK, resurrect the percpu ref, expose the swap device */
enable_swap_info(si);
- pr_info("Adding %uk swap on %s. Priority:%d extents:%d across:%lluk %s%s%s%s\n",
+ pr_info("Adding %luk swap on %s. Priority:%d extents:%d across:%lluk %s%s%s%s\n",
K(si->pages), name->name, si->prio, nr_extents,
K((unsigned long long)span),
(si->flags & SWP_SOLIDSTATE) ? "SS" : "",
@@ -3825,7 +4697,7 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
si->global_cluster = NULL;
inode = NULL;
destroy_swap_extents(si, swap_file);
- free_swap_cluster_info(si->cluster_info, si->max);
+ free_swap_cluster_info(si, si->cluster_info, si->max);
si->cluster_info = NULL;
/*
* Clear the SWP_USED flag after all resources are freed so
@@ -3956,3 +4828,105 @@ static int __init swapfile_init(void)
return 0;
}
subsys_initcall(swapfile_init);
+
+struct swap_info_struct *vswap_si;
+DEFINE_STATIC_KEY_FALSE(vswap_key);
+
+static bool vswap_enabled_early __initdata = IS_ENABLED(CONFIG_VSWAP_DEFAULT_ON);
+
+static int __init early_vswap(char *buf)
+{
+ return kstrtobool(buf, &vswap_enabled_early);
+}
+early_param("vswap", early_vswap);
+
+/* vswap does no IO on its own. */
+static const struct swap_ops vswap_ops = { };
+
+static int vswap_used_get(void *data, u64 *val)
+{
+ *val = swap_usage_in_pages(vswap_si);
+ return 0;
+}
+DEFINE_DEBUGFS_ATTRIBUTE(vswap_used_fops, vswap_used_get, NULL, "%llu\n");
+
+static int vswap_alloc_reject_get(void *data, u64 *val)
+{
+ *val = atomic_long_read(&vswap_alloc_reject);
+ return 0;
+}
+DEFINE_DEBUGFS_ATTRIBUTE(vswap_alloc_reject_fops, vswap_alloc_reject_get, NULL,
+ "%llu\n");
+
+static int __init vswap_init(void)
+{
+ struct swap_info_struct *si;
+ struct dentry *root;
+ unsigned long maxpages;
+ int err;
+
+ if (!IS_ENABLED(CONFIG_64BIT)) {
+ if (vswap_enabled_early)
+ pr_warn("vswap: requires 64-bit architecture; vswap disabled, swapout falls back to direct physical swap\n");
+ return 0;
+ }
+
+ if (!vswap_enabled_early)
+ return 0;
+
+ si = alloc_swap_info();
+ if (IS_ERR(si)) {
+ pr_warn("vswap: alloc_swap_info failed (%ld); vswap disabled, swapout falls back to direct physical swap\n",
+ PTR_ERR(si));
+ return 0;
+ }
+
+ /*
+ * One u32 xarray ID per cluster, so the device cannot be larger
+ * than UINT_MAX clusters.
+ */
+ maxpages = min(swapfile_maximum_size,
+ (unsigned long)UINT_MAX * SWAPFILE_CLUSTER);
+ /*
+ * SWP_WRITEOK enables slot allocation. SWP_SOLIDSTATE selects
+ * per-CPU cluster allocation; vswap has no si->global_cluster.
+ */
+ si->flags |= SWP_VSWAP | SWP_SOLIDSTATE | SWP_WRITEOK;
+ si->ops = &vswap_ops;
+ si->bdev = NULL;
+ si->max = maxpages;
+ si->pages = maxpages - 1;
+ si->prio = SHRT_MAX;
+ si->list.prio = -si->prio;
+ si->avail_list.prio = -si->prio;
+
+ err = setup_swap_clusters_info(si, NULL, maxpages);
+ if (err)
+ goto fail;
+
+ mutex_lock(&swapon_mutex);
+ enable_swap_info(si);
+ mutex_unlock(&swapon_mutex);
+
+ vswap_si = si;
+
+ root = debugfs_create_dir("vswap", NULL);
+ debugfs_create_file("used", 0444, root, NULL, &vswap_used_fops);
+ debugfs_create_file("alloc_reject", 0444, root, NULL,
+ &vswap_alloc_reject_fops);
+
+ pr_info("vswap: created virtual swap device (%lu pages)\n", maxpages);
+
+ /* Last: everything above must be visible before routing starts. */
+ static_branch_enable(&vswap_key);
+ return 0;
+
+fail:
+ pr_warn("vswap: setup_swap_clusters_info failed (%d); vswap disabled, swapout falls back to direct physical swap\n",
+ err);
+ spin_lock(&swap_lock);
+ si->flags = 0;
+ spin_unlock(&swap_lock);
+ return 0;
+}
+late_initcall(vswap_init);
diff --git a/mm/vmscan.c b/mm/vmscan.c
index c1404a59523d6..7960cc489ea03 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -69,6 +69,7 @@
#include "internal.h"
#include "page_alloc.h"
#include "swap.h"
+#include "vswap.h"
#define CREATE_TRACE_POINTS
#include <trace/events/vmscan.h>
@@ -353,6 +354,9 @@ static inline bool can_reclaim_anon_pages(struct mem_cgroup *memcg,
*/
if (get_nr_swap_pages() > 0)
return true;
+ /* vswap doesn't contribute to nr_swap_pages */
+ if (vswap_is_enabled() && zswap_is_enabled())
+ return true;
} else {
/* Is the memcg below its swap limit? */
if (mem_cgroup_get_nr_swap_pages(memcg) > 0)
@@ -1524,7 +1528,8 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
activate_locked:
/* Not a candidate for swapping, so reclaim swap space. */
if (folio_test_swapcache(folio) &&
- (mem_cgroup_swap_full(folio) || folio_test_mlocked(folio)))
+ ((mem_cgroup_swap_full(folio) && folio_phys_swap_backed(folio)) ||
+ folio_test_mlocked(folio)))
folio_free_swap(folio);
VM_BUG_ON_FOLIO(folio_test_active(folio), folio);
if (!folio_test_mlocked(folio)) {
@@ -2681,7 +2686,7 @@ static bool can_age_anon_pages(struct lruvec *lruvec,
struct scan_control *sc)
{
/* Aging the anon LRU is valuable if swap is present: */
- if (total_swap_pages > 0)
+ if (total_swap_pages > 0 || (vswap_is_enabled() && zswap_is_enabled()))
return true;
/* Also valuable if anon pages can be demoted: */
diff --git a/mm/vswap.h b/mm/vswap.h
new file mode 100644
index 0000000000000..c66fa34e2e60c
--- /dev/null
+++ b/mm/vswap.h
@@ -0,0 +1,440 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Virtual swap space
+ *
+ * Copyright (C) 2026 Nhat Pham
+ */
+#ifndef _MM_VSWAP_H
+#define _MM_VSWAP_H
+
+#include <linux/jump_label.h>
+#include <linux/swap.h>
+#include "swap.h"
+
+struct zswap_entry;
+
+/*
+ * VSWAP_ZERO and VSWAP_FOLIO are return-only values synthesized from
+ * swap_table state; the rest are stored in the vtable per slot.
+ */
+enum vswap_backing_type {
+ VSWAP_NONE = 0,
+ VSWAP_ZSWAP = 1,
+ VSWAP_SWAPFILE = 2,
+ VSWAP_ZERO,
+ VSWAP_FOLIO,
+};
+
+#ifdef CONFIG_SWAP
+
+#include "swap_table.h"
+DECLARE_STATIC_KEY_FALSE(vswap_key);
+
+/*
+ * Only true once vswap_init() has published vswap_si, so callers never
+ * see the device half built.
+ */
+static inline bool vswap_is_enabled(void)
+{
+ return static_branch_unlikely(&vswap_key);
+}
+
+static inline bool is_vswap_entry(swp_entry_t entry)
+{
+ return swap_is_vswap(__swap_entry_to_info(entry));
+}
+
+/*
+ * Rmap cache-only helpers for physical cluster Pointer-tagged entries.
+ * SWP_RMAP_CACHE_ONLY records, inline on the physical swap_table entry,
+ * that the backing vswap entry has swap_count == 0 (swap-cache-only, so
+ * reclaimable). The physical reclaim scanner reads it directly instead of
+ * chasing the rmap into the vswap layer and paying the cluster-lookup
+ * indirection.
+ *
+ * Callers hold the vswap cluster lock, not the physical one. The rmap is
+ * only touched while the vtable holds the slot as SWAPFILE, and that
+ * window is opened and closed under the vswap cluster lock, so the
+ * allocator has finished writing the entry by then.
+ */
+static inline void swap_rmap_mark_cache_only(struct swap_cluster_info *ci,
+ unsigned int off)
+{
+ atomic_long_t *table;
+
+ table = rcu_dereference_check(ci->table, true);
+ atomic_long_or(SWP_RMAP_CACHE_ONLY, &table[off]);
+}
+
+static inline void swap_rmap_clear_cache_only(struct swap_cluster_info *ci,
+ unsigned int off)
+{
+ atomic_long_t *table;
+
+ table = rcu_dereference_check(ci->table, true);
+ atomic_long_and(~SWP_RMAP_CACHE_ONLY, &table[off]);
+}
+
+/*
+ * Virtual table entry encoding for vswap clusters.
+ *
+ * Each entry in ci_dyn->virtual_table stores the backing type and
+ * pointer for a virtual swap slot. Tag in low 3 bits, payload in
+ * upper 61 bits.
+ *
+ * NONE: |----- 0000 ------|000| - no separate backend pointer
+ * ZSWAP: |--- zswap_entry* |001| - compressed in zswap (tag in low bits)
+ * SWAPFILE: |- type:5,off:56 -|010| - on a physical swapfile
+ *
+ * SWAPFILE packs swp_type in the top MAX_SWAPFILES_SHIFT bits and swp_offset in
+ * the middle VTABLE_PHYS_OFF_BITS bits, both above the tag, so the type is
+ * not shifted off the word. Pointer payloads (ZSWAP) are stored directly with
+ * the tag OR'd into the low bits (kernel pointers are >= 8-byte aligned, same
+ * approach as xarray).
+ *
+ * vtable[i] = NONE does not by itself mean "free". The swap_table entry
+ * and the per-slot zero flag carry the rest of the state. The full
+ * per-slot state table is:
+ *
+ * vtable[i] | swap_table[i] | zero | meaning
+ * ----------+---------------+-------+--------------------------------
+ * NONE | NULL | clear | truly free / unbacked
+ * NONE | PFN | clear | folio cached, no backing
+ * NONE | shadow | clear | evicted, no backing: data lost
+ * NONE | * | set | zero-backed; cached if PFN set
+ * ZSWAP | PFN | clear | folio cached + zswap entry
+ * ZSWAP | shadow / NULL | clear | evicted, only in zswap
+ * SWAPFILE | PFN | clear | folio cached + physical slot
+ * SWAPFILE | shadow / NULL | clear | evicted, only on the swapfile
+ *
+ * Locking: a slot's vtable entry (the vswap entry's backend) is only
+ * stable while the caller owns and holds the lock on that entry's swap
+ * cache folio. The cluster lock (ci_dyn->ci.lock) only makes an individual
+ * vtable read atomic, and by itself does not give the caller the right to
+ * change the backend. A backend read without the folio lock is
+ * best-effort and must be re-validated under the folio lock before
+ * being acted on.
+ *
+ * Zero-backed slots use the swap_table per-slot zero flag (same as
+ * direct-mapped physical swap), via __swap_table_test_zero() and friends,
+ * which fall back to ci->zero_bitmap where the flag does not fit. Cached
+ * folios are read out of the swap_table PFN entry; there is no separate FOLIO
+ * vtable type because the folio pointer would duplicate that PFN and
+ * would go stale on folio migration / split.
+ */
+
+#define VTABLE_TAG_BITS 3
+#define VTABLE_TAG_MASK ((1UL << VTABLE_TAG_BITS) - 1)
+
+static inline enum vswap_backing_type vtable_type(unsigned long vt)
+{
+ return vt & VTABLE_TAG_MASK;
+}
+
+/* swp_offset field width in a physical backend slot; layout described above. */
+#define VTABLE_PHYS_OFF_BITS (BITS_PER_LONG - VTABLE_TAG_BITS - MAX_SWAPFILES_SHIFT)
+
+static inline unsigned long vtable_mk_phys(swp_entry_t entry)
+{
+ VM_WARN_ON_ONCE(swp_offset(entry) >> VTABLE_PHYS_OFF_BITS);
+ return ((unsigned long)swp_type(entry) << (VTABLE_TAG_BITS + VTABLE_PHYS_OFF_BITS)) |
+ (swp_offset(entry) << VTABLE_TAG_BITS) | VSWAP_SWAPFILE;
+}
+
+static inline swp_entry_t vtable_to_phys(unsigned long vt)
+{
+ VM_WARN_ON(vtable_type(vt) != VSWAP_SWAPFILE);
+ return swp_entry(vt >> (VTABLE_TAG_BITS + VTABLE_PHYS_OFF_BITS),
+ (vt >> VTABLE_TAG_BITS) & ((1UL << VTABLE_PHYS_OFF_BITS) - 1));
+}
+
+static inline struct zswap_entry *vtable_to_zswap(unsigned long vt)
+{
+ VM_WARN_ON(vtable_type(vt) != VSWAP_ZSWAP);
+ return (struct zswap_entry *)(vt & ~VTABLE_TAG_MASK);
+}
+
+/* Virtual table accessors */
+
+static inline unsigned long __vtable_get(struct swap_cluster_info_dynamic *ci_dyn,
+ unsigned int off)
+{
+ VM_WARN_ON_ONCE(off >= SWAPFILE_CLUSTER);
+ return atomic_long_read(&ci_dyn->virtual_table[off]);
+}
+
+static inline void __vtable_set(struct swap_cluster_info_dynamic *ci_dyn,
+ unsigned int off, unsigned long vt)
+{
+ VM_WARN_ON_ONCE(off >= SWAPFILE_CLUSTER);
+ atomic_long_set(&ci_dyn->virtual_table[off], vt);
+}
+
+/**
+ * vswap_lock_cluster - look up and lock the vswap cluster for an entry
+ * @entry: the virtual swap entry
+ * @voff: out param, receives @entry's slot offset within the cluster
+ *
+ * Return: the locked vswap cluster, or NULL if @entry has no live cluster.
+ */
+static inline struct swap_cluster_info_dynamic *
+vswap_lock_cluster(swp_entry_t entry, unsigned int *voff)
+{
+ struct swap_cluster_info *ci;
+
+ ci = swap_cluster_lock(__swap_entry_to_info(entry), swp_offset(entry));
+ if (!ci)
+ return NULL;
+ *voff = swp_cluster_offset(entry);
+ return container_of(ci, struct swap_cluster_info_dynamic, ci);
+}
+
+/**
+ * vswap_to_phys - resolve a vswap entry's physical swap backing
+ * @entry: the virtual swap entry
+ *
+ * Context: takes and drops the vswap cluster lock internally.
+ * Return: the backing physical swp_entry_t, or the null entry (.val == 0)
+ * when @entry has no physical backing (NONE/ZSWAP/ZERO).
+ */
+static inline swp_entry_t vswap_to_phys(swp_entry_t entry)
+{
+ struct swap_cluster_info_dynamic *ci_dyn;
+ unsigned int voff;
+ unsigned long vt;
+
+ ci_dyn = vswap_lock_cluster(entry, &voff);
+ if (!ci_dyn)
+ return (swp_entry_t){};
+
+ vt = __vtable_get(ci_dyn, voff);
+ swap_cluster_unlock(&ci_dyn->ci);
+
+ if (vtable_type(vt) != VSWAP_SWAPFILE)
+ return (swp_entry_t){};
+
+ return vtable_to_phys(vt);
+}
+
+void __vswap_release_backing(struct swap_cluster_info *ci,
+ unsigned int ci_start, unsigned int nr);
+
+/**
+ * vswap_zswap_store - record a zswap entry as the backing for a vswap entry.
+ * @entry: the vswap entry
+ * @ze: the zswap entry now holding @entry's compressed data
+ *
+ * Releases @entry's previous backing, and sets the zswap entry @ze as the new
+ * backing.
+ *
+ * Context: takes and drops the vswap cluster lock internally.
+ */
+static inline void vswap_zswap_store(swp_entry_t entry,
+ struct zswap_entry *ze)
+{
+ struct swap_cluster_info_dynamic *ci_dyn;
+ unsigned int voff;
+
+ ci_dyn = vswap_lock_cluster(entry, &voff);
+ __vswap_release_backing(&ci_dyn->ci, voff, 1);
+ __vtable_set(ci_dyn, voff, (unsigned long)ze | VSWAP_ZSWAP);
+ swap_cluster_unlock(&ci_dyn->ci);
+}
+
+/**
+ * vswap_zswap_load - return the zswap entry backing a vswap entry
+ * @entry: the virtual swap entry
+ *
+ * Context: takes and drops the vswap cluster lock internally.
+ * Return: the backing zswap entry, or NULL if @entry is not zswap-backed.
+ */
+static inline struct zswap_entry *vswap_zswap_load(swp_entry_t entry)
+{
+ struct swap_cluster_info_dynamic *ci_dyn;
+ unsigned int voff;
+ unsigned long vt;
+
+ ci_dyn = vswap_lock_cluster(entry, &voff);
+ if (!ci_dyn)
+ return NULL;
+ vt = __vtable_get(ci_dyn, voff);
+ swap_cluster_unlock(&ci_dyn->ci);
+
+ if (vtable_type(vt) != VSWAP_ZSWAP)
+ return NULL;
+ return vtable_to_zswap(vt);
+}
+
+void folio_release_vswap_backing(struct folio *folio);
+swp_entry_t folio_realloc_swap(struct folio *folio);
+void folio_release_non_phys_swap_backing(struct folio *folio);
+
+/*
+ * Walk nr vtable slots starting at voff in ci_dyn. Returns the prefix
+ * length of slots sharing one effective backing type. For SWAPFILE,
+ * the prefix is also restricted to contiguous offsets in the same
+ * swapfile.
+ *
+ * Effective type per slot:
+ * vtable=NONE + zero flag set -> VSWAP_ZERO
+ * vtable=NONE + swap_table PFN tag -> VSWAP_FOLIO
+ * vtable=NONE + neither -> VSWAP_NONE
+ * vtable=SWAPFILE -> VSWAP_SWAPFILE
+ * vtable=ZSWAP -> VSWAP_ZSWAP
+ *
+ * *typep returns the effective type of slot 0. Caller holds
+ * ci_dyn->ci.lock.
+ */
+static inline int __vswap_check_backing(struct swap_cluster_info_dynamic *ci_dyn,
+ unsigned int voff, int nr,
+ enum vswap_backing_type *typep)
+{
+ enum vswap_backing_type first_type = VSWAP_NONE;
+ enum vswap_backing_type slot_type;
+ swp_entry_t first_phys = {};
+ unsigned long vt, swap_tb;
+ int i;
+
+ lockdep_assert_held(&ci_dyn->ci.lock);
+
+ for (i = 0; i < nr; i++) {
+ vt = __vtable_get(ci_dyn, voff + i);
+ if (vtable_type(vt) == VSWAP_NONE) {
+ swap_tb = __swap_table_get(&ci_dyn->ci, voff + i);
+ if (__swap_table_test_zero(&ci_dyn->ci, voff + i))
+ slot_type = VSWAP_ZERO;
+ else if (swp_tb_is_folio(swap_tb))
+ slot_type = VSWAP_FOLIO;
+ else
+ slot_type = VSWAP_NONE;
+ } else {
+ slot_type = vtable_type(vt);
+ }
+
+ if (!i) {
+ first_type = slot_type;
+ if (first_type == VSWAP_SWAPFILE)
+ first_phys = vtable_to_phys(vt);
+ } else if (slot_type != first_type) {
+ break;
+ } else if (first_type == VSWAP_SWAPFILE &&
+ vtable_to_phys(vt).val != first_phys.val + i) {
+ break;
+ }
+ }
+
+ if (typep)
+ *typep = first_type;
+ return i;
+}
+
+static inline int vswap_check_backing(swp_entry_t entry, int nr,
+ enum vswap_backing_type *typep)
+{
+ struct swap_cluster_info_dynamic *ci_dyn;
+ unsigned int voff;
+ int ret;
+
+ ci_dyn = vswap_lock_cluster(entry, &voff);
+ if (!ci_dyn) {
+ if (typep)
+ *typep = VSWAP_NONE;
+ return 0;
+ }
+ ret = __vswap_check_backing(ci_dyn, voff, nr, typep);
+ swap_cluster_unlock(&ci_dyn->ci);
+ return ret;
+}
+
+/**
+ * folio_phys_swap_backed - test whether a folio is backed by a contiguous
+ * range of physical swap slots.
+ * @folio: a swap-cache resident folio
+ *
+ * Return: %true if @folio->swap is not a vswap entry, or if these vswap
+ * entries are backed by a contiguous range of physical slots.
+ */
+static inline bool folio_phys_swap_backed(struct folio *folio)
+{
+ swp_entry_t entry = folio->swap;
+ int nr = folio_nr_pages(folio);
+ enum vswap_backing_type type;
+
+ return !is_vswap_entry(entry) ||
+ (vswap_check_backing(entry, nr, &type) == nr &&
+ type == VSWAP_SWAPFILE);
+}
+
+static inline int vswap_cluster_alloc_vtable(struct swap_cluster_info_dynamic *ci_dyn,
+ gfp_t gfp)
+{
+ ci_dyn->virtual_table = kcalloc(SWAPFILE_CLUSTER,
+ sizeof(*ci_dyn->virtual_table), gfp);
+ return ci_dyn->virtual_table ? 0 : -ENOMEM;
+}
+
+static inline void vswap_cluster_free_vtable(struct swap_cluster_info *ci)
+{
+ struct swap_cluster_info_dynamic *ci_dyn;
+
+ ci_dyn = container_of(ci, struct swap_cluster_info_dynamic, ci);
+ kfree(ci_dyn->virtual_table);
+ ci_dyn->virtual_table = NULL;
+}
+
+#else /* !CONFIG_SWAP */
+
+static inline bool vswap_is_enabled(void)
+{
+ return false;
+}
+
+static inline bool is_vswap_entry(swp_entry_t entry)
+{
+ return false;
+}
+
+static inline swp_entry_t vswap_to_phys(swp_entry_t entry)
+{
+ return (swp_entry_t){};
+}
+
+static inline bool folio_phys_swap_backed(struct folio *folio)
+{
+ return true;
+}
+
+#endif /* CONFIG_SWAP */
+
+/*
+ * Test a per-backend swap flag (SWP_SYNCHRONOUS_IO, SWP_STABLE_WRITES, ...)
+ * for @entry. For a vswap entry the property belongs to the current
+ * physical backing rather than vswap_si itself; resolve to the backing
+ * and test there. Returns false for zswap/zero/unbacked vswap entries
+ * as they don't have a backing bdev.
+ */
+static inline bool swap_entry_backend_has_flag(struct swap_info_struct *si,
+ swp_entry_t entry,
+ unsigned long flag)
+{
+ struct swap_info_struct *phys_si;
+ swp_entry_t phys;
+ bool has_flag;
+
+ if (!swap_is_vswap(si))
+ return data_race(si->flags & flag);
+
+ phys = vswap_to_phys(entry);
+ if (!phys.val)
+ return false;
+
+ phys_si = get_swap_device(phys);
+ if (!phys_si)
+ return false;
+
+ has_flag = data_race(phys_si->flags & flag);
+ put_swap_device(phys_si);
+ return has_flag;
+}
+
+#endif /* _MM_VSWAP_H */
diff --git a/mm/zswap.c b/mm/zswap.c
index 37f34e406c8e3..70ad8010f18a1 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -38,6 +38,7 @@
#include <linux/zsmalloc.h>
#include "swap.h"
+#include "vswap.h"
#include "internal.h"
/*********************************
@@ -234,6 +235,25 @@ static inline struct xarray *swap_zswap_tree(swp_entry_t swp)
>> ZSWAP_ADDRESS_SPACE_SHIFT];
}
+static struct zswap_entry *zswap_entry_load(swp_entry_t swp)
+{
+ if (is_vswap_entry(swp))
+ return vswap_zswap_load(swp);
+ return xa_load(swap_zswap_tree(swp), swp_offset(swp));
+}
+
+static struct zswap_entry *zswap_entry_store(swp_entry_t swp,
+ struct zswap_entry *entry)
+{
+ if (is_vswap_entry(swp)) {
+ vswap_zswap_store(swp, entry);
+ return NULL;
+ }
+
+ return xa_store(swap_zswap_tree(swp), swp_offset(swp), entry,
+ GFP_KERNEL);
+}
+
#define zswap_pool_debug(msg, p) \
pr_debug("%s pool %s\n", msg, (p)->tfm_name)
@@ -762,7 +782,7 @@ static void zswap_entry_cache_free(struct zswap_entry *entry)
* Carries out the common pattern of freeing an entry's zsmalloc allocation,
* freeing the entry itself, and decrementing the number of stored pages.
*/
-static void zswap_entry_free(struct zswap_entry *entry)
+void zswap_entry_free(struct zswap_entry *entry)
{
zswap_lru_del(entry);
zs_free(entry->pool->zs_pool, entry->handle);
@@ -987,12 +1007,13 @@ static bool zswap_decompress(struct zswap_entry *entry, struct folio *folio)
static int zswap_writeback_entry(struct zswap_entry *entry,
swp_entry_t swpentry)
{
- struct xarray *tree;
pgoff_t offset = swp_offset(swpentry);
struct folio *folio;
struct mempolicy *mpol;
struct swap_info_struct *si;
struct swap_io_ctx ctx = {};
+ swp_entry_t phys = {};
+ bool is_vswap;
int ret = 0;
/* try to allocate swap cache folio */
@@ -1000,6 +1021,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry,
if (!si)
return -EEXIST;
+ is_vswap = swap_is_vswap(si);
mpol = get_task_policy(current);
folio = swap_cache_alloc_folio(swpentry, GFP_KERNEL, BIT(0), NULL, mpol,
NO_INTERLEAVE_INDEX);
@@ -1018,24 +1040,44 @@ static int zswap_writeback_entry(struct zswap_entry *entry,
/*
* folio is locked, and the swapcache is now secured against
* concurrent swapping to and from the slot, and concurrent
- * swapoff so we can safely dereference the zswap tree here.
+ * swapoff so we can safely dereference the zswap tree (or vswap
+ * vtable) here.
* Verify that the swap entry hasn't been invalidated and recycled
* behind our backs, to avoid overwriting a new swap folio with
* old compressed data. Only when this is successful can the entry
* be dereferenced.
*/
- tree = swap_zswap_tree(swpentry);
- if (entry != xa_load(tree, offset)) {
+ if (entry != zswap_entry_load(swpentry)) {
ret = -ENOMEM;
goto out;
}
+ if (is_vswap) {
+ /*
+ * Allocate physical backing before decompress so a failure
+ * wastes no work.
+ */
+ phys = folio_realloc_swap(folio);
+ if (!phys.val) {
+ ret = -ENOMEM;
+ goto out;
+ }
+ }
+
if (!zswap_decompress(entry, folio)) {
ret = -EIO;
+ /*
+ * The phys allocation above took the entry out of the vtable.
+ * Restore the zswap entry to the vtable, which also frees the
+ * allocated physical swap space.
+ */
+ if (is_vswap)
+ vswap_zswap_store(swpentry, entry);
goto out;
}
- xa_erase(tree, offset);
+ if (!is_vswap)
+ xa_erase(swap_zswap_tree(swpentry), offset);
count_vm_event(ZSWPWB);
if (entry->objcg)
@@ -1050,7 +1092,10 @@ static int zswap_writeback_entry(struct zswap_entry *entry,
folio_set_reclaim(folio);
/* start writeback */
- __swap_writepage(&ctx, folio);
+ if (is_vswap)
+ __swap_writepage(&ctx, folio, phys);
+ else
+ __swap_writepage(&ctx, folio, folio->swap);
swap_write_submit(&ctx);
out:
@@ -1065,6 +1110,15 @@ static int zswap_writeback_entry(struct zswap_entry *entry,
/*********************************
* shrinker functions
**********************************/
+/*
+ * vswap zswap entries get a physical slot allocated on demand at writeback
+ * time. Skip the shrinker when none is available.
+ */
+static bool zswap_writeback_possible(void)
+{
+ return !vswap_is_enabled() || get_nr_swap_pages() > 0;
+}
+
/*
* The dynamic shrinker is modulated by the following factors:
*
@@ -1202,6 +1256,9 @@ static unsigned long zswap_shrinker_count(struct shrinker *shrinker,
if (!zswap_shrinker_enabled || !mem_cgroup_zswap_writeback_enabled(memcg))
return 0;
+ if (!zswap_writeback_possible())
+ return 0;
+
/*
* The shrinker resumes swap writeback, which will enter block
* and may enter fs. XXX: Harmonize with vmscan.c __GFP_FS
@@ -1284,6 +1341,9 @@ static struct shrinker *zswap_alloc_shrinker(void)
* Return: 0 if at least one entry was written back, -EAGAIN if entries
* were scanned but none could be written back, or -ENOENT if @memcg has
* writeback disabled, is a zombie cgroup, or has empty zswap LRUs.
+ *
+ * Also returns -ENOENT when vswap is enabled and there is no physical
+ * swap to write back to.
*/
static int shrink_memcg(struct mem_cgroup *memcg)
{
@@ -1292,6 +1352,9 @@ static int shrink_memcg(struct mem_cgroup *memcg)
if (!mem_cgroup_zswap_writeback_enabled(memcg))
return -ENOENT;
+ if (!zswap_writeback_possible())
+ return -ENOENT;
+
/*
* Skip zombies because their LRUs are reparented and we would be
* reclaiming from the parent instead of the dead memcg.
@@ -1320,6 +1383,9 @@ static void shrink_worker(struct work_struct *w)
int ret, failures = 0, attempts = 0;
unsigned long thr;
+ if (!zswap_writeback_possible())
+ return;
+
/* Reclaim down to the accept threshold */
thr = zswap_accept_thr_pages();
@@ -1398,7 +1464,7 @@ static void shrink_worker(struct work_struct *w)
break;
resched:
cond_resched();
- } while (zswap_total_pages() > thr);
+ } while (zswap_total_pages() > thr && zswap_writeback_possible());
}
/*********************************
@@ -1422,9 +1488,7 @@ static bool zswap_store_page(struct page *page,
if (!zswap_compress(page, entry, pool))
goto compress_failed;
- old = xa_store(swap_zswap_tree(page_swpentry),
- swp_offset(page_swpentry),
- entry, GFP_KERNEL);
+ old = zswap_entry_store(page_swpentry, entry);
if (xa_is_err(old)) {
int err = xa_err(old);
@@ -1493,7 +1557,7 @@ bool zswap_store(struct folio *folio)
struct mem_cgroup *memcg = NULL;
struct zswap_pool *pool;
bool ret = false;
- long index;
+ long index = 0;
VM_WARN_ON_ONCE(!folio_test_locked(folio));
VM_WARN_ON_ONCE(!folio_test_swapcache(folio));
@@ -1548,13 +1612,19 @@ bool zswap_store(struct folio *folio)
if (!ret && zswap_pool_reached_full)
queue_work(shrink_wq, &zswap_shrink_work);
check_old:
+ if (ret)
+ return ret;
+
/*
* If the zswap store fails or zswap is disabled, we must invalidate
* the possibly stale entries which were previously stored at the
* offsets corresponding to each page of the folio. Otherwise,
* writeback could overwrite the new data in the swapfile.
*/
- if (!ret) {
+ if (is_vswap_entry(swp)) {
+ if (index > 0)
+ folio_release_non_phys_swap_backing(folio);
+ } else {
unsigned type = swp_type(swp);
pgoff_t offset = swp_offset(swp);
struct zswap_entry *entry;
@@ -1584,9 +1654,9 @@ bool zswap_store(struct folio *folio)
* will SIGBUS).
*
* -EINVAL: if the swapped out content was in zswap, but the page belongs
- * to a large folio, which is not supported by zswap. The folio is unlocked,
- * but NOT marked up-to-date, so that an IO error is emitted (e.g.
- * do_swap_page() will SIGBUS).
+ * to a large non-vswap folio, which is not supported by zswap. The folio
+ * is unlocked, but NOT marked up-to-date, so that an IO error is emitted
+ * (e.g. do_swap_page() will SIGBUS).
*
* -ENOENT: if the swapped out content was not in zswap. The folio remains
* locked on return.
@@ -1594,8 +1664,7 @@ bool zswap_store(struct folio *folio)
int zswap_load(struct folio *folio)
{
swp_entry_t swp = folio->swap;
- pgoff_t offset = swp_offset(swp);
- struct xarray *tree = swap_zswap_tree(swp);
+ struct swap_info_struct *si = __swap_entry_to_info(swp);
struct zswap_entry *entry;
VM_WARN_ON_ONCE(!folio_test_locked(folio));
@@ -1608,13 +1677,20 @@ int zswap_load(struct folio *folio)
* Large folios should not be swapped in while zswap is being used, as
* they are not properly handled. Zswap does not properly load large
* folios, and a large folio may only be partially in zswap.
+ *
+ * A large vswap folio cannot reach here ZSWAP-backed, since
+ * __swap_cache_add_check() refuses such a batch, so hand it to the
+ * phys path without warning.
*/
- if (WARN_ON_ONCE(folio_test_large(folio))) {
- folio_unlock(folio);
- return -EINVAL;
+ if (folio_test_large(folio)) {
+ if (WARN_ON_ONCE(!swap_is_vswap(si))) {
+ folio_unlock(folio);
+ return -EINVAL;
+ }
+ return -ENOENT;
}
- entry = xa_load(tree, offset);
+ entry = zswap_entry_load(swp);
if (!entry)
return -ENOENT;
@@ -1637,8 +1713,13 @@ int zswap_load(struct folio *folio)
* compression work.
*/
folio_mark_dirty(folio);
- xa_erase(tree, offset);
- zswap_entry_free(entry);
+
+ if (swap_is_vswap(si)) {
+ folio_release_vswap_backing(folio);
+ } else {
+ xa_erase(swap_zswap_tree(swp), swp_offset(swp));
+ zswap_entry_free(entry);
+ }
folio_unlock(folio);
return 0;
|
| 7/3 | 2026/08/25 15:56 | llm | kmsan-evaluator |
1mModel:gemini-3.1-pro-preview Args: null Results: null Tokens: input: 51774 output: 332 thoughts: 3104 Thoughts: **KMSAN Risk Assessment: Virtual Swap Patch** Okay, let's break this down. This patch is introducing "virtual swap" (vswap), which essentially layers a virtual indirection on top of the existing swap mechanism. The core idea is to allow zswap and zero-filled pages to be used seamlessly without requiring a physical swap device. My primary concern here is always uninitialized memory, which KMSAN is designed to catch. First, I'm going to look at allocation sites. I see quite a few new allocations: `vswap_alloc_cluster`, `vswap_alloc`, `folio_alloc_phys_swap`, `folio_realloc_swap`, `vswap_zswap_store`, and `vswap_zswap_load`, among others. I'm focusing on where kernel memory could be exposed to userspace. A quick check of the relevant code blocks shows that new structures like `swap_cluster_info_dynamic`, and the virtual tables, and the folio structs are all allocated using zeroing allocators like `kzalloc_obj`, `kcalloc`, and `folio_alloc` with `__GFP_ZERO`. This means that newly allocated memory is initialized to zero, mitigating the immediate risk of uninitialized memory being read. The use of these allocators is a good sign. Next, I'm checking for places where the patch modifies swap read or write paths since these involve user-space pages. I'm watching for situations where potentially uninitialized memory could be read from swap and copied back to user space. It modifies `swap_writeout`, `swap_read_folio`, `do_swap_page`, and other swap-related functions. If a vswap entry has no physical backing, what happens? In the `swap_read_folio` case, if the "physical" address `phys.val` is zero, there's a check, folio is unlocked and control goes to finish where it returns -EIO. If a `zswap_load` is called, and that fails it goes to the check for the physical backing. In `do_swap_page` it checks if the folio is up to date, if not it returns -EIO, and the user-space process will get a SIGBUS. This mechanism is sound. The code handles this situation explicitly, avoiding the potential leak of uninitialized kernel memory. The comment in `try_to_unuse` explicitly acknowledges and guards against "dirtying uninitialized memory". Are there any kernel structures sent to user space? No, this patch deals with internal memory management. I'm making sure `vswap_alloc` and `folio_alloc_phys_swap` are safe. If they fail and return 0, there is a check in the calling function. So it looks clean. So, I'm looking carefully at these `swap_read_folio` and `do_swap_page` paths. The code explicitly checks for the absence of backing and, crucially, doesn't mark the folio as uptodate. This is key. The subsequent check for `folio_test_uptodate` ensures that if a backing entry is missing it reports `-EIO`. This is the safeguard to prevent leaking kernel memory to user space. I'm thinking, this is a complex patch, and it makes fundamental changes to the swap subsystem. But it does seem that these risks are properly handled with existing kernel checks and safeguards. Any bugs here would likely manifest as the kernel not finding a swapped-out page and returning an error to the user space process. This is something that standard testing, and tools like KASAN, will easily find. Therefore, because this patch *explicitly* addresses and guards against uninitialized memory exposure and relies on zeroing allocators, and doesn't introduce any new user-facing APIs, I don't see any KMSAN-specific vulnerabilities introduced by this change. The more likely bugs are the memory corruption and use-after-free bugs that KASAN will catch. NeedsKMSAN = false. |
| 8/3 | 2026/08/25 15:57 | tool | set-results |
0mArgs:{
"KMSANReasoning": "The patch introduces a virtual swap layer (`vswap`) to decouple swap entries from physical swap slots, allowing better integration with `zswap` and zero-filled pages. The changes involve complex memory management logic, including dynamic allocation of swap clusters, virtual-to-physical mapping tables, and modifications to the swap read/write paths.\n\nAll new data structures (e.g., `struct swap_cluster_info_dynamic`, virtual tables, swap tables) are allocated using zeroing allocators (`kzalloc_obj`, `kcalloc`, `folio_alloc` with `__GFP_ZERO`). The patch does not introduce new kernel-to-user interfaces that copy structures to user space, nor does it add complex parsing of user-provided data where uninitialized fields could be used.\n\nIf a virtual swap entry lacks physical backing during a read fault, the code correctly handles it by returning an error (`-EIO`) without marking the folio as up-to-date, preventing the exposure of uninitialized page memory to user space (as explicitly noted in the `try_to_unuse` comments).\n\nAny potential bugs introduced by this patch (e.g., incorrect reference counting, race conditions in cluster management, or incorrect physical slot resolution) would manifest as use-after-free, out-of-bounds accesses, or memory corruption, which are effectively detected by KASAN, LOCKDEP, and standard kernel testing. Therefore, a dedicated KMSAN fuzzing session is not justified.",
"NeedsKMSAN": false
}Results: {
"KMSANReasoning": "The patch introduces a virtual swap layer (`vswap`) to decouple swap entries from physical swap slots, allowing better integration with `zswap` and zero-filled pages. The changes involve complex memory management logic, including dynamic allocation of swap clusters, virtual-to-physical mapping tables, and modifications to the swap read/write paths.\n\nAll new data structures (e.g., `struct swap_cluster_info_dynamic`, virtual tables, swap tables) are allocated using zeroing allocators (`kzalloc_obj`, `kcalloc`, `folio_alloc` with `__GFP_ZERO`). The patch does not introduce new kernel-to-user interfaces that copy structures to user space, nor does it add complex parsing of user-provided data where uninitialized fields could be used.\n\nIf a virtual swap entry lacks physical backing during a read fault, the code correctly handles it by returning an error (`-EIO`) without marking the folio as up-to-date, preventing the exposure of uninitialized page memory to user space (as explicitly noted in the `try_to_unuse` comments).\n\nAny potential bugs introduced by this patch (e.g., incorrect reference counting, race conditions in cluster management, or incorrect physical slot resolution) would manifest as use-after-free, out-of-bounds accesses, or memory corruption, which are effectively detected by KASAN, LOCKDEP, and standard kernel testing. Therefore, a dedicated KMSAN fuzzing session is not justified.",
"NeedsKMSAN": false
} |
| Total Calls | Total Tokens | Avg Tokens | Total Duration (Seconds) | Avg Duration (Seconds) |
|---|
| Total Calls | Total Duration (Seconds) | Avg Duration (Seconds) |
|---|