| Seq | Timestamp | Type | Name | Duration |
|---|---|---|---|---|
| 0/0 | 2026/06/29 12:30 | flow | patch-triage |
0mArgs:null Results: {
"EnableConfigs": [
"USERFAULTFD_RWP"
],
"FocusSymbols": [
"userfaultfd_rwprotect",
"mrwprotect_range",
"userfaultfd_set_mode",
"do_uffd_rwp",
"do_huge_pmd_uffd_rwp"
],
"Reasoning": "The patch introduces a new userfaultfd feature, Read-Write Protection (RWP), which adds new ioctls (UFFDIO_RWPROTECT, UFFDIO_SET_MODE) and modifies page fault handling to support RWP. This is a significant functional change to core memory management and userfaultfd that should be fuzzed.",
"WorthFuzzing": true
} |
| 1/1 | 2026/06/29 12:30 | action | read-patch-diff |
0mArgs:null Results: {
"PatchDiff": "commit 09db1b5e82f6242632e83646e40234035529896b\nAuthor: syz-cluster \u003ctriage@syzkaller.com\u003e\nDate: Mon Jun 29 12:30:51 2026 +0000\n\n syz-cluster: applied patch under review\n\ndiff --git a/Documentation/admin-guide/mm/pagemap.rst b/Documentation/admin-guide/mm/pagemap.rst\nindex c57e61b5d8aa81..ffa690a171c862 100644\n--- a/Documentation/admin-guide/mm/pagemap.rst\n+++ b/Documentation/admin-guide/mm/pagemap.rst\n@@ -19,8 +19,11 @@ There are four components to pagemap:\n * Bit 55 pte is soft-dirty (see\n Documentation/admin-guide/mm/soft-dirty.rst)\n * Bit 56 page exclusively mapped (since 4.2)\n- * Bit 57 pte is uffd-wp write-protected (since 5.13) (see\n- Documentation/admin-guide/mm/userfaultfd.rst)\n+ * Bit 57 pte is tracked by userfaultfd (since 5.13) — in a\n+ ``VM_UFFD_WP`` VMA this indicates a write-protected PTE; in a\n+ ``VM_UFFD_RWP`` VMA it indicates an RWP-protected PTE. WP and\n+ RWP are mutually exclusive per VMA, so the meaning is\n+ unambiguous. See Documentation/admin-guide/mm/userfaultfd.rst.\n * Bit 58 pte is a guard region (since 6.15) (see madvise (2) man page)\n * Bits 59-60 zero\n * Bit 61 page is file-page or shared-anon (since 3.5)\n@@ -244,7 +247,8 @@ in this IOCTL:\n Following flags about pages are currently supported:\n \n - ``PAGE_IS_WPALLOWED`` - Page has async-write-protection enabled\n-- ``PAGE_IS_WRITTEN`` - Page has been written to from the time it was write protected\n+- ``PAGE_IS_WRITTEN`` - Page in a ``UFFDIO_REGISTER_MODE_WP`` VMA has been\n+ written to since it was write-protected. Only reported inside such VMAs.\n - ``PAGE_IS_FILE`` - Page is file backed\n - ``PAGE_IS_PRESENT`` - Page is present in the memory\n - ``PAGE_IS_SWAPPED`` - Page is in swapped\n@@ -252,6 +256,9 @@ Following flags about pages are currently supported:\n - ``PAGE_IS_HUGE`` - Page is PMD-mapped THP or Hugetlb backed\n - ``PAGE_IS_SOFT_DIRTY`` - Page is soft-dirty\n - ``PAGE_IS_GUARD`` - Page is a part of a guard region\n+- ``PAGE_IS_ACCESSED`` - Page in a ``UFFDIO_REGISTER_MODE_RWP`` VMA has been\n+ accessed since RWP was applied. Only reported inside such VMAs. See\n+ Documentation/admin-guide/mm/userfaultfd.rst for the RWP workflow.\n \n The ``struct pm_scan_arg`` is used as the argument of the IOCTL.\n \ndiff --git a/Documentation/admin-guide/mm/userfaultfd.rst b/Documentation/admin-guide/mm/userfaultfd.rst\nindex e5cc8848dcb39f..69836e39aa0b64 100644\n--- a/Documentation/admin-guide/mm/userfaultfd.rst\n+++ b/Documentation/admin-guide/mm/userfaultfd.rst\n@@ -131,6 +131,16 @@ userfaults on the range registered. Not all ioctls will necessarily be\n supported for all memory types (e.g. anonymous memory vs. shmem vs.\n hugetlbfs), or all types of intercepted faults.\n \n+.. note::\n+\n+ Re-registering an already-registered range must not drop any of the\n+ modes that install per-PTE markers — currently\n+ ``UFFDIO_REGISTER_MODE_WP`` and ``UFFDIO_REGISTER_MODE_RWP``. Doing\n+ so would strand markers with no flag to describe them, so the call\n+ is rejected with ``-EBUSY``; userspace must issue\n+ ``UFFDIO_UNREGISTER`` first. This differs from older kernels, which\n+ silently replaced the mode bits on re-registration.\n+\n Userland can use the ``uffdio_register.ioctls`` to manage the virtual\n address space in the background (to add or potentially also remove\n memory from the ``userfaultfd`` registered range). This means a userfault\n@@ -265,16 +275,16 @@ tracking and it can be different in a few ways:\n - Dirty information will not get lost if the pte was zapped due to\n various reasons (e.g. during split of a shmem transparent huge page).\n \n- - Due to a reverted meaning of soft-dirty (page clean when uffd-wp bit\n- set; dirty when uffd-wp bit cleared), it has different semantics on\n- some of the memory operations. For example: ``MADV_DONTNEED`` on\n+ - Due to a reverted meaning of soft-dirty (page clean when the uffd bit\n+ is set; dirty when the uffd bit is cleared), it has different semantics\n+ on some of the memory operations. For example: ``MADV_DONTNEED`` on\n anonymous (or ``MADV_REMOVE`` on a file mapping) will be treated as\n- dirtying of memory by dropping uffd-wp bit during the procedure.\n+ dirtying of memory by dropping the uffd bit during the procedure.\n \n The user app can collect the \"written/dirty\" status by looking up the\n-uffd-wp bit for the pages being interested in /proc/pagemap.\n+uffd bit for the pages being interested in /proc/pagemap.\n \n-The page will not be under track of uffd-wp async mode until the page is\n+The page will not be under track of userfaultfd-wp async mode until the page is\n explicitly write-protected by ``ioctl(UFFDIO_WRITEPROTECT)`` with the mode\n flag ``UFFDIO_WRITEPROTECT_MODE_WP`` set. Trying to resolve a page fault\n that was tracked by async mode userfaultfd-wp is invalid.\n@@ -297,6 +307,253 @@ transparent to the guest, we want that same address range to act as if it was\n still poisoned, even though it's on a new physical host which ostensibly\n doesn't have a memory error in the exact same spot.\n \n+Read-Write Protection\n+---------------------\n+\n+``UFFDIO_REGISTER_MODE_RWP`` enables read-write protection tracking on a\n+memory range. It is similar to (but faster than) ``mprotect(PROT_NONE)``\n+combined with a signal handler; unlike ``mprotect(PROT_NONE)``, RWP only\n+traps accesses to *present* PTEs, so accesses to unpopulated addresses in a\n+protected range fall through to the normal missing-page path. It uses the\n+PROT_NONE hinting mechanism (same as NUMA balancing) to make pages\n+inaccessible while keeping them resident in memory. Works on anonymous,\n+shmem, and hugetlbfs memory.\n+\n+RWP is designed for VM memory managers that need to track the working set\n+of guest memory for cold page eviction to tiered or remote storage.\n+\n+**Setup:**\n+\n+1. Open a userfaultfd and enable ``UFFD_FEATURE_RWP`` via ``UFFDIO_API``.\n+ Optionally request ``UFFD_FEATURE_RWP_ASYNC`` as well — it requires\n+ ``UFFD_FEATURE_RWP`` to be set in the same ``UFFDIO_API`` call.\n+\n+2. Register the guest memory range with ``UFFDIO_REGISTER_MODE_RWP``\n+ (and ``UFFDIO_REGISTER_MODE_MISSING`` if evicted pages will need to be\n+ fetched back from storage).\n+\n+**Feature availability:**\n+\n+RWP is built on top of two kernel primitives: a spare PTE bit owned by\n+userfaultfd (``CONFIG_HAVE_ARCH_USERFAULTFD_WP``) and architecture support\n+for present-but-inaccessible PTEs (``CONFIG_ARCH_HAS_PTE_PROTNONE``). When both\n+are available on a 64-bit kernel, the build selects\n+``CONFIG_USERFAULTFD_RWP=y`` and the ``VM_UFFD_RWP`` VMA flag becomes\n+available.\n+\n+``UFFD_FEATURE_RWP`` and ``UFFD_FEATURE_RWP_ASYNC`` are unavailable when\n+the running kernel or architecture does not support them — for example\n+32-bit kernels (where ``VM_UFFD_RWP`` is unavailable), kernels built\n+without ``CONFIG_USERFAULTFD_RWP``, and architectures whose ptes cannot\n+carry the uffd bit at runtime (e.g. riscv without the ``SVRSW60T59B``\n+extension). Requesting an unsupported feature in\n+``uffdio_api.features`` makes ``UFFDIO_API`` fail with ``EINVAL`` and\n+leaves the userfaultfd context uninitialized; the bitmask returned in\n+``uffdio_api.features`` then advertises the features the kernel does\n+support. The recommended probe sequence is therefore to open a\n+throwaway userfaultfd, call ``UFFDIO_API`` once with ``features = 0``,\n+inspect the returned bitmask, close that fd, then open the real one\n+and call ``UFFDIO_API`` again with only the supported features set.\n+\n+**Protecting and Unprotecting:**\n+\n+Use ``UFFDIO_RWPROTECT`` to protect or unprotect a range, mirroring the\n+``UFFDIO_WRITEPROTECT`` interface::\n+\n+ struct uffdio_rwprotect rwp = {\n+ .range = { .start = addr, .len = len },\n+ .mode = UFFDIO_RWPROTECT_MODE_RWP, /* protect */\n+ };\n+ ioctl(uffd, UFFDIO_RWPROTECT, \u0026rwp);\n+\n+Setting ``UFFDIO_RWPROTECT_MODE_RWP`` sets PROT_NONE on present PTEs in the\n+range. Pages stay resident and their physical frames are preserved — only\n+access permissions are removed.\n+\n+Clearing ``UFFDIO_RWPROTECT_MODE_RWP`` restores normal VMA permissions and\n+wakes any faulting threads (unless ``UFFDIO_RWPROTECT_MODE_DONTWAKE`` is set).\n+\n+**Scope of protection:**\n+\n+RWP protection is a property of *present* PTEs. ``UFFDIO_RWPROTECT`` only\n+affects entries that are already populated. Unpopulated addresses within\n+the range remain unpopulated; when first accessed they fault through the\n+normal missing path (``do_anonymous_page()``, ``do_swap_page()``,\n+``finish_fault()``) and the resulting PTE is not RWP-protected. To observe\n+the population itself, co-register the range with\n+``UFFDIO_REGISTER_MODE_MISSING``.\n+\n+Protection is preserved across page reclaim: a page swapped out while\n+RWP-protected carries the marker on its swap entry, and swap-in restores\n+the PROT_NONE state so the first access after swap-in still faults. The\n+same applies to pages temporarily replaced by migration entries.\n+\n+Operations that drop the PTE entirely — ``MADV_DONTNEED`` on anonymous\n+memory, hole-punch on shmem, truncation of a file mapping — also drop the\n+RWP marker: the next access re-populates the range without protection.\n+Unlike WP (which persists via ``PTE_MARKER_UFFD_WP``), there is no\n+persistent RWP marker today. The user needs to re-arm the range with\n+``UFFDIO_RWPROTECT`` after any operation that explicitly frees PTEs.\n+\n+**Fault Handling:**\n+\n+When a protected page is accessed:\n+\n+- **Sync mode** (default): The faulting thread blocks and a\n+ ``UFFD_PAGEFAULT_FLAG_RWP`` message is delivered to the userfaultfd\n+ handler. The handler resolves the fault with ``UFFDIO_RWPROTECT``\n+ (clearing ``MODE_RWP``), which restores the PTE permissions and wakes\n+ the faulting thread.\n+\n+- **Async mode** (``UFFD_FEATURE_RWP_ASYNC``): The kernel automatically\n+ restores PTE permissions and the thread continues without blocking. No\n+ message is delivered to the handler.\n+\n+**Runtime Mode Switching:**\n+\n+``UFFDIO_SET_MODE`` toggles ``UFFD_FEATURE_RWP_ASYNC`` at runtime, allowing\n+the VMM to switch between lightweight async detection and safe sync\n+eviction without re-registering. The toggle takes ``mmap_write_lock()``\n+and calls ``vma_start_write()`` on each UFFD-armed VMA, draining\n+in-flight per-VMA-locked faults before the new mode takes effect.\n+\n+**Working-set detection with PAGEMAP_SCAN:**\n+\n+RWP-protected PTEs carry the uffd PTE bit; an access (and, in async mode, its\n+auto-resolution) clears it. ``PAGEMAP_SCAN`` reports ``PAGE_IS_ACCESSED`` once\n+the bit is clear on a ``VM_UFFD_RWP`` VMA, so a *non-inverted* scan reports the\n+pages that were touched during the interval -- the hot set::\n+\n+ struct pm_scan_arg arg = {\n+ .size = sizeof(arg),\n+ .start = guest_mem_start,\n+ .end = guest_mem_end,\n+ .vec = (uint64_t)regions,\n+ .vec_len = regions_len,\n+ .category_mask = PAGE_IS_ACCESSED,\n+ .return_mask = PAGE_IS_ACCESSED,\n+ };\n+ long n = ioctl(pagemap_fd, PAGEMAP_SCAN, \u0026arg);\n+\n+The returned ``page_region`` array lists the hot ranges. ``PAGE_IS_ACCESSED``\n+is set on an accessed page whether it is still present or has since been\n+swapped out, so the hot scan needs no ``PAGE_IS_PRESENT`` filter -- unpopulated\n+holes carry neither bit and are excluded on their own.\n+\n+Track the hot set and reclaim everything else from the backing file (see the\n+workflow below). Do **not** invert the scan to enumerate \"cold\" pages\n+directly: an inverted scan reports only the ``VM_UFFD_RWP`` PTEs that are still\n+protected, i.e. the resident portion of *this* VMA. For a file mapping the\n+working set spans the whole file -- pages that live in the page cache but are\n+not mapped into this VMA (a pre-populated tmpfs file, or memory populated\n+through another mapping) are ``pte_none`` here, never appear in the scan, and\n+would never be considered for eviction even though they occupy memory. Driving\n+eviction from \"file offsets minus the hot set\" avoids that blind spot; a cold\n+PTE scan cannot. To additionally record the *first* access to a cached but\n+unmapped page (e.g. pre-populated content) as hot, co-register the range with\n+``UFFDIO_REGISTER_MODE_MINOR``: such accesses then fault as minor faults\n+instead of mapping the page silently.\n+\n+**Cleanup:**\n+\n+When the userfaultfd is closed or the range is unregistered, all PROT_NONE\n+PTEs are automatically restored to their normal VMA permissions. This\n+prevents pages from becoming permanently inaccessible.\n+\n+**VMM Working Set Tracking Workflow:**\n+\n+A typical VMM lifecycle for cold page eviction to tiered storage. Two\n+mappings of the same shmem (or hugetlbfs) file are used: ``guest_mem`` is\n+the RWP-registered mapping that vCPUs access through, and ``io_mem`` is a\n+private mapping for VMM-side I/O. Reading ``io_mem`` does not go through\n+the RWP-protected PTEs of ``guest_mem``, so the VMM's own ``pwrite()``\n+never traps on its own ::\n+\n+ /* One-time setup */\n+ fd = memfd_create(\"guest\", MFD_CLOEXEC);\n+ ftruncate(fd, guest_size);\n+ guest_mem = mmap(NULL, guest_size, PROT_READ | PROT_WRITE,\n+ MAP_SHARED, fd, 0); /* vCPU view, RWP-registered */\n+ io_mem = mmap(NULL, guest_size, PROT_READ | PROT_WRITE,\n+ MAP_SHARED, fd, 0); /* VMM I/O view, unprotected */\n+\n+ uffd = userfaultfd(O_CLOEXEC | O_NONBLOCK);\n+ struct uffdio_api api = {\n+ .api = UFFD_API,\n+ .features = UFFD_FEATURE_RWP | UFFD_FEATURE_RWP_ASYNC,\n+ };\n+ ioctl(uffd, UFFDIO_API, \u0026api);\n+ if (!(api.features \u0026 UFFD_FEATURE_RWP))\n+ /* RWP unavailable on this kernel/arch -- fall back. */\n+ ioctl(uffd, UFFDIO_REGISTER, \u0026(struct uffdio_register){\n+ .range = { guest_mem, guest_size },\n+ .mode = UFFDIO_REGISTER_MODE_RWP |\n+ UFFDIO_REGISTER_MODE_MISSING,\n+ });\n+\n+ /* Tracking loop */\n+ while (vm_running) {\n+ /* 1. Detection phase (async -- no vCPU stalls) */\n+ ioctl(uffd, UFFDIO_RWPROTECT, \u0026(struct uffdio_rwprotect){\n+ .range = full_range,\n+ .mode = UFFDIO_RWPROTECT_MODE_RWP });\n+ sleep(tracking_interval);\n+\n+ /*\n+ * 2. Switch to sync BEFORE scanning. In async mode a vCPU\n+ * access races eviction: it would auto-resolve and mark the\n+ * page hot just as the VMM writes it out and punches it,\n+ * losing the update. Sync mode makes such accesses block and\n+ * be delivered, freezing the hot snapshot for the rest of the\n+ * iteration.\n+ */\n+ ioctl(uffd, UFFDIO_SET_MODE,\n+ \u0026(struct uffdio_set_mode){\n+ .disable = UFFD_FEATURE_RWP_ASYNC });\n+\n+ /* 3. Read the hot set: pages touched this interval. */\n+ ioctl(pagemap_fd, PAGEMAP_SCAN, \u0026(struct pm_scan_arg){\n+ .category_mask = PAGE_IS_ACCESSED,\n+ .return_mask = PAGE_IS_ACCESSED,\n+ ...\n+ });\n+\n+ /*\n+ * 4. Reclaim the file offsets that are NOT in the hot set.\n+ * Driving this from the file's offset space (rather than from a\n+ * cold PTE scan) also reclaims pages that are cached but not\n+ * mapped into guest_mem, e.g. pre-populated content.\n+ */\n+ for each non-hot offset range:\n+ /* Read from io_mem -- bypasses RWP, no fault. */\n+ pwrite(storage_fd, (char *)io_mem + off, len, off);\n+ /* Drop the page from the shared file. */\n+ fallocate(fd, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE,\n+ off, len);\n+ /*\n+ * Wake any vCPU blocked on the RWP fault for this range:\n+ * fallocate() does not iterate ctx-\u003efault_pending_wqh.\n+ */\n+ ioctl(uffd, UFFDIO_WAKE, \u0026(struct uffdio_range){\n+ .start = (uintptr_t)guest_mem + off, .len = len });\n+\n+ /* 5. Resume async tracking */\n+ ioctl(uffd, UFFDIO_SET_MODE,\n+ \u0026(struct uffdio_set_mode){\n+ .enable = UFFD_FEATURE_RWP_ASYNC });\n+ }\n+\n+During step 4, a vCPU that accesses a ``guest_mem`` offset being evicted\n+blocks with a ``UFFD_PAGEFAULT_FLAG_RWP`` fault while the eviction is in\n+progress. After ``fallocate()`` punches the page out and ``UFFDIO_WAKE``\n+fires, the vCPU retries the access, faults as ``MISSING``, and the\n+handler resolves it with ``UFFDIO_COPY`` from storage.\n+\n+This workflow targets shmem and hugetlbfs (both support a private\n+``io_mem`` mapping over the same fd). Anonymous-memory backings need a\n+different inner-loop strategy because the VMM has no way to read the\n+page without going through the RWP-protected mapping.\n+\n QEMU/KVM\n ========\n \ndiff --git a/Documentation/filesystems/proc.rst b/Documentation/filesystems/proc.rst\nindex 2ccd5b2dfdd33a..27189f6e004eed 100644\n--- a/Documentation/filesystems/proc.rst\n+++ b/Documentation/filesystems/proc.rst\n@@ -608,6 +608,7 @@ encoded manner. The codes are the following:\n um userfaultfd missing tracking\n uw userfaultfd wr-protect tracking\n ui userfaultfd minor fault\n+ ur userfaultfd read-write-protect tracking\n ss shadow/guarded control stack page\n sl sealed\n lf lock on fault pages\ndiff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig\nindex b3afe0688919bd..6e123adca29b92 100644\n--- a/arch/arm64/Kconfig\n+++ b/arch/arm64/Kconfig\n@@ -78,6 +78,7 @@ config ARM64\n \tselect ARCH_SUPPORTS_CFI\n \tselect ARCH_SUPPORTS_ATOMIC_RMW\n \tselect ARCH_SUPPORTS_INT128 if CC_HAS_INT128\n+\tselect ARCH_HAS_PTE_PROTNONE\n \tselect ARCH_SUPPORTS_NUMA_BALANCING\n \tselect ARCH_SUPPORTS_PAGE_TABLE_CHECK\n \tselect ARCH_SUPPORTS_PER_VMA_LOCK\ndiff --git a/arch/arm64/include/asm/pgtable-prot.h b/arch/arm64/include/asm/pgtable-prot.h\nindex 212ce1b02e15ed..09d7c00cf40541 100644\n--- a/arch/arm64/include/asm/pgtable-prot.h\n+++ b/arch/arm64/include/asm/pgtable-prot.h\n@@ -28,11 +28,11 @@\n #define PTE_PRESENT_VALID_KERNEL (PTE_VALID | PTE_MAYBE_NG)\n \n #ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP\n-#define PTE_UFFD_WP\t\t(_AT(pteval_t, 1) \u003c\u003c 58) /* uffd-wp tracking */\n-#define PTE_SWP_UFFD_WP\t\t(_AT(pteval_t, 1) \u003c\u003c 3)\t /* only for swp ptes */\n+#define PTE_UFFD\t\t(_AT(pteval_t, 1) \u003c\u003c 58) /* userfaultfd tracking */\n+#define PTE_SWP_UFFD\t\t(_AT(pteval_t, 1) \u003c\u003c 3)\t /* only for swp ptes */\n #else\n-#define PTE_UFFD_WP\t\t(_AT(pteval_t, 0))\n-#define PTE_SWP_UFFD_WP\t\t(_AT(pteval_t, 0))\n+#define PTE_UFFD\t\t(_AT(pteval_t, 0))\n+#define PTE_SWP_UFFD\t\t(_AT(pteval_t, 0))\n #endif /* CONFIG_HAVE_ARCH_USERFAULTFD_WP */\n \n #define _PROT_DEFAULT\t\t(PTE_TYPE_PAGE | PTE_AF | PTE_SHARED)\ndiff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h\nindex 27689c62bd25dd..c49a553ad12fbe 100644\n--- a/arch/arm64/include/asm/pgtable.h\n+++ b/arch/arm64/include/asm/pgtable.h\n@@ -341,19 +341,19 @@ static inline pmd_t pmd_mknoncont(pmd_t pmd)\n }\n \n #ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP\n-static inline int pte_uffd_wp(pte_t pte)\n+static inline int pte_uffd(pte_t pte)\n {\n-\treturn !!(pte_val(pte) \u0026 PTE_UFFD_WP);\n+\treturn !!(pte_val(pte) \u0026 PTE_UFFD);\n }\n \n-static inline pte_t pte_mkuffd_wp(pte_t pte)\n+static inline pte_t pte_mkuffd(pte_t pte)\n {\n-\treturn pte_wrprotect(set_pte_bit(pte, __pgprot(PTE_UFFD_WP)));\n+\treturn pte_wrprotect(set_pte_bit(pte, __pgprot(PTE_UFFD)));\n }\n \n-static inline pte_t pte_clear_uffd_wp(pte_t pte)\n+static inline pte_t pte_clear_uffd(pte_t pte)\n {\n-\treturn clear_pte_bit(pte, __pgprot(PTE_UFFD_WP));\n+\treturn clear_pte_bit(pte, __pgprot(PTE_UFFD));\n }\n #endif /* CONFIG_HAVE_ARCH_USERFAULTFD_WP */\n \n@@ -537,26 +537,23 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pte)\n }\n \n #ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP\n-static inline pte_t pte_swp_mkuffd_wp(pte_t pte)\n+static inline pte_t pte_swp_mkuffd(pte_t pte)\n {\n-\treturn set_pte_bit(pte, __pgprot(PTE_SWP_UFFD_WP));\n+\treturn set_pte_bit(pte, __pgprot(PTE_SWP_UFFD));\n }\n \n-static inline int pte_swp_uffd_wp(pte_t pte)\n+static inline int pte_swp_uffd(pte_t pte)\n {\n-\treturn !!(pte_val(pte) \u0026 PTE_SWP_UFFD_WP);\n+\treturn !!(pte_val(pte) \u0026 PTE_SWP_UFFD);\n }\n \n-static inline pte_t pte_swp_clear_uffd_wp(pte_t pte)\n+static inline pte_t pte_swp_clear_uffd(pte_t pte)\n {\n-\treturn clear_pte_bit(pte, __pgprot(PTE_SWP_UFFD_WP));\n+\treturn clear_pte_bit(pte, __pgprot(PTE_SWP_UFFD));\n }\n #endif /* CONFIG_HAVE_ARCH_USERFAULTFD_WP */\n \n-#ifdef CONFIG_NUMA_BALANCING\n-/*\n- * See the comment in include/linux/pgtable.h\n- */\n+#ifdef CONFIG_ARCH_HAS_PTE_PROTNONE\n static inline int pte_protnone(pte_t pte)\n {\n \t/*\n@@ -575,7 +572,7 @@ static inline int pmd_protnone(pmd_t pmd)\n {\n \treturn pte_protnone(pmd_pte(pmd));\n }\n-#endif\n+#endif /* CONFIG_ARCH_HAS_PTE_PROTNONE */\n \n #define pmd_present(pmd)\tpte_present(pmd_pte(pmd))\n #define pmd_dirty(pmd)\t\tpte_dirty(pmd_pte(pmd))\n@@ -593,13 +590,13 @@ static inline int pmd_protnone(pmd_t pmd)\n #define pmd_mkvalid_k(pmd)\tpte_pmd(pte_mkvalid_k(pmd_pte(pmd)))\n #define pmd_mkinvalid(pmd)\tpte_pmd(pte_mkinvalid(pmd_pte(pmd)))\n #ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP\n-#define pmd_uffd_wp(pmd)\tpte_uffd_wp(pmd_pte(pmd))\n-#define pmd_mkuffd_wp(pmd)\tpte_pmd(pte_mkuffd_wp(pmd_pte(pmd)))\n-#define pmd_clear_uffd_wp(pmd)\tpte_pmd(pte_clear_uffd_wp(pmd_pte(pmd)))\n-#define pmd_swp_uffd_wp(pmd)\tpte_swp_uffd_wp(pmd_pte(pmd))\n-#define pmd_swp_mkuffd_wp(pmd)\tpte_pmd(pte_swp_mkuffd_wp(pmd_pte(pmd)))\n-#define pmd_swp_clear_uffd_wp(pmd) \\\n-\t\t\t\tpte_pmd(pte_swp_clear_uffd_wp(pmd_pte(pmd)))\n+#define pmd_uffd(pmd)\tpte_uffd(pmd_pte(pmd))\n+#define pmd_mkuffd(pmd)\tpte_pmd(pte_mkuffd(pmd_pte(pmd)))\n+#define pmd_clear_uffd(pmd)\tpte_pmd(pte_clear_uffd(pmd_pte(pmd)))\n+#define pmd_swp_uffd(pmd)\tpte_swp_uffd(pmd_pte(pmd))\n+#define pmd_swp_mkuffd(pmd)\tpte_pmd(pte_swp_mkuffd(pmd_pte(pmd)))\n+#define pmd_swp_clear_uffd(pmd) \\\n+\t\t\t\tpte_pmd(pte_swp_clear_uffd(pmd_pte(pmd)))\n #endif /* CONFIG_HAVE_ARCH_USERFAULTFD_WP */\n \n #define pmd_write(pmd)\t\tpte_write(pmd_pte(pmd))\n@@ -1515,7 +1512,7 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma,\n * Encode and decode a swap entry:\n *\tbits 0-1:\tpresent (must be zero)\n *\tbits 2:\t\tremember PG_anon_exclusive\n- *\tbit 3:\t\tremember uffd-wp state\n+ *\tbit 3:\t\tremember uffd state\n *\tbits 6-10:\tswap type\n *\tbit 11:\tPTE_PRESENT_INVALID (must be zero)\n *\tbits 12-61:\tswap offset\ndiff --git a/arch/loongarch/Kconfig b/arch/loongarch/Kconfig\nindex d8d25232501791..177fb9dc64320c 100644\n--- a/arch/loongarch/Kconfig\n+++ b/arch/loongarch/Kconfig\n@@ -67,6 +67,7 @@ config LOONGARCH\n \tselect ARCH_SUPPORTS_LTO_CLANG\n \tselect ARCH_SUPPORTS_LTO_CLANG_THIN\n \tselect ARCH_SUPPORTS_MSEAL_SYSTEM_MAPPINGS\n+\tselect ARCH_HAS_PTE_PROTNONE if 64BIT\n \tselect ARCH_SUPPORTS_NUMA_BALANCING if NUMA\n \tselect ARCH_SUPPORTS_PER_VMA_LOCK\n \tselect ARCH_SUPPORTS_RT\ndiff --git a/arch/loongarch/include/asm/pgtable.h b/arch/loongarch/include/asm/pgtable.h\nindex 223528c04d7392..1952e34bc8ee00 100644\n--- a/arch/loongarch/include/asm/pgtable.h\n+++ b/arch/loongarch/include/asm/pgtable.h\n@@ -623,7 +623,7 @@ static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm,\n \n #endif /* CONFIG_TRANSPARENT_HUGEPAGE */\n \n-#ifdef CONFIG_NUMA_BALANCING\n+#ifdef CONFIG_ARCH_HAS_PTE_PROTNONE\n static inline long pte_protnone(pte_t pte)\n {\n \treturn (pte_val(pte) \u0026 _PAGE_PROTNONE);\n@@ -633,7 +633,7 @@ static inline long pmd_protnone(pmd_t pmd)\n {\n \treturn (pmd_val(pmd) \u0026 _PAGE_PROTNONE);\n }\n-#endif /* CONFIG_NUMA_BALANCING */\n+#endif /* CONFIG_ARCH_HAS_PTE_PROTNONE */\n \n #define pmd_leaf(pmd)\t\t((pmd_val(pmd) \u0026 _PAGE_HUGE) != 0)\n #define pud_leaf(pud)\t\t((pud_val(pud) \u0026 _PAGE_HUGE) != 0)\ndiff --git a/arch/powerpc/include/asm/book3s/64/pgtable.h b/arch/powerpc/include/asm/book3s/64/pgtable.h\nindex e67e64ac6e8cfc..53a0c5892548fc 100644\n--- a/arch/powerpc/include/asm/book3s/64/pgtable.h\n+++ b/arch/powerpc/include/asm/book3s/64/pgtable.h\n@@ -490,13 +490,13 @@ static inline pte_t pte_clear_soft_dirty(pte_t pte)\n }\n #endif /* CONFIG_HAVE_ARCH_SOFT_DIRTY */\n \n-#ifdef CONFIG_NUMA_BALANCING\n+#ifdef CONFIG_ARCH_HAS_PTE_PROTNONE\n static inline int pte_protnone(pte_t pte)\n {\n \treturn (pte_raw(pte) \u0026 cpu_to_be64(_PAGE_PRESENT | _PAGE_PTE | _PAGE_RWX)) ==\n \t\tcpu_to_be64(_PAGE_PRESENT | _PAGE_PTE);\n }\n-#endif /* CONFIG_NUMA_BALANCING */\n+#endif /* CONFIG_ARCH_HAS_PTE_PROTNONE */\n \n static inline bool pte_hw_valid(pte_t pte)\n {\n@@ -1067,12 +1067,12 @@ static inline pte_t *pmdp_ptep(pmd_t *pmd)\n #endif\n #endif /* CONFIG_HAVE_ARCH_SOFT_DIRTY */\n \n-#ifdef CONFIG_NUMA_BALANCING\n+#ifdef CONFIG_ARCH_HAS_PTE_PROTNONE\n static inline int pmd_protnone(pmd_t pmd)\n {\n \treturn pte_protnone(pmd_pte(pmd));\n }\n-#endif /* CONFIG_NUMA_BALANCING */\n+#endif /* CONFIG_ARCH_HAS_PTE_PROTNONE */\n \n #define pmd_write(pmd)\t\tpte_write(pmd_pte(pmd))\n \ndiff --git a/arch/powerpc/platforms/Kconfig.cputype b/arch/powerpc/platforms/Kconfig.cputype\nindex bac02c83bb3e4f..36b64a24cf3003 100644\n--- a/arch/powerpc/platforms/Kconfig.cputype\n+++ b/arch/powerpc/platforms/Kconfig.cputype\n@@ -87,6 +87,7 @@ config PPC_BOOK3S_64\n \tselect ARCH_ENABLE_HUGEPAGE_MIGRATION if HUGETLB_PAGE \u0026\u0026 MIGRATION\n \tselect ARCH_ENABLE_SPLIT_PMD_PTLOCK\n \tselect ARCH_SUPPORTS_HUGETLBFS\n+\tselect ARCH_HAS_PTE_PROTNONE\n \tselect ARCH_SUPPORTS_NUMA_BALANCING\n \tselect HAVE_MOVE_PMD\n \tselect HAVE_MOVE_PUD\ndiff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig\nindex 3f0a647218e407..58c12f4fac6681 100644\n--- a/arch/riscv/Kconfig\n+++ b/arch/riscv/Kconfig\n@@ -71,6 +71,7 @@ config RISCV\n \tselect ARCH_SUPPORTS_MSEAL_SYSTEM_MAPPINGS if 64BIT \u0026\u0026 MMU\n \tselect ARCH_SUPPORTS_PAGE_TABLE_CHECK if MMU\n \tselect ARCH_SUPPORTS_PER_VMA_LOCK if MMU\n+\tselect ARCH_HAS_PTE_PROTNONE if MMU\n \tselect ARCH_SUPPORTS_RT\n \tselect ARCH_SUPPORTS_SHADOW_CALL_STACK if HAVE_SHADOW_CALL_STACK\n \tselect ARCH_SUPPORTS_SCHED_MC if SMP\ndiff --git a/arch/riscv/include/asm/pgtable-bits.h b/arch/riscv/include/asm/pgtable-bits.h\nindex b422d9691e60d8..d5a86b4df3ce69 100644\n--- a/arch/riscv/include/asm/pgtable-bits.h\n+++ b/arch/riscv/include/asm/pgtable-bits.h\n@@ -40,20 +40,20 @@\n \n #ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP\n \n-/* ext_svrsw60t59b: Bit(60) for uffd-wp tracking */\n-#define _PAGE_UFFD_WP\t\t\t\t\t\t\t\\\n+/* ext_svrsw60t59b: Bit(60) for userfaultfd tracking */\n+#define _PAGE_UFFD\t\t\t\t\t\t\t\\\n \t((riscv_has_extension_unlikely(RISCV_ISA_EXT_SVRSW60T59B)) ?\t\\\n \t (1UL \u003c\u003c 60) : 0)\n /*\n * Bit 4 is not involved into swap entry computation, so we\n- * can borrow it for swap page uffd-wp tracking.\n+ * can borrow it for swap page userfaultfd tracking.\n */\n-#define _PAGE_SWP_UFFD_WP\t\t\t\t\t\t\\\n+#define _PAGE_SWP_UFFD\t\t\t\t\t\t\\\n \t((riscv_has_extension_unlikely(RISCV_ISA_EXT_SVRSW60T59B)) ?\t\\\n \t _PAGE_USER : 0)\n #else\n-#define _PAGE_UFFD_WP\t\t0\n-#define _PAGE_SWP_UFFD_WP\t0\n+#define _PAGE_UFFD\t\t0\n+#define _PAGE_SWP_UFFD\t\t0\n #endif\n \n #define _PAGE_TABLE _PAGE_PRESENT\ndiff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h\nindex 5d5756bda82e3d..8194a2060f82bc 100644\n--- a/arch/riscv/include/asm/pgtable.h\n+++ b/arch/riscv/include/asm/pgtable.h\n@@ -410,37 +410,37 @@ static inline pte_t pte_wrprotect(pte_t pte)\n }\n \n #ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP\n-#define pgtable_supports_uffd_wp()\t\\\n+#define pgtable_supports_uffd()\t\\\n \triscv_has_extension_unlikely(RISCV_ISA_EXT_SVRSW60T59B)\n \n-static inline bool pte_uffd_wp(pte_t pte)\n+static inline bool pte_uffd(pte_t pte)\n {\n-\treturn !!(pte_val(pte) \u0026 _PAGE_UFFD_WP);\n+\treturn !!(pte_val(pte) \u0026 _PAGE_UFFD);\n }\n \n-static inline pte_t pte_mkuffd_wp(pte_t pte)\n+static inline pte_t pte_mkuffd(pte_t pte)\n {\n-\treturn pte_wrprotect(__pte(pte_val(pte) | _PAGE_UFFD_WP));\n+\treturn pte_wrprotect(__pte(pte_val(pte) | _PAGE_UFFD));\n }\n \n-static inline pte_t pte_clear_uffd_wp(pte_t pte)\n+static inline pte_t pte_clear_uffd(pte_t pte)\n {\n-\treturn __pte(pte_val(pte) \u0026 ~(_PAGE_UFFD_WP));\n+\treturn __pte(pte_val(pte) \u0026 ~(_PAGE_UFFD));\n }\n \n-static inline bool pte_swp_uffd_wp(pte_t pte)\n+static inline bool pte_swp_uffd(pte_t pte)\n {\n-\treturn !!(pte_val(pte) \u0026 _PAGE_SWP_UFFD_WP);\n+\treturn !!(pte_val(pte) \u0026 _PAGE_SWP_UFFD);\n }\n \n-static inline pte_t pte_swp_mkuffd_wp(pte_t pte)\n+static inline pte_t pte_swp_mkuffd(pte_t pte)\n {\n-\treturn __pte(pte_val(pte) | _PAGE_SWP_UFFD_WP);\n+\treturn __pte(pte_val(pte) | _PAGE_SWP_UFFD);\n }\n \n-static inline pte_t pte_swp_clear_uffd_wp(pte_t pte)\n+static inline pte_t pte_swp_clear_uffd(pte_t pte)\n {\n-\treturn __pte(pte_val(pte) \u0026 ~(_PAGE_SWP_UFFD_WP));\n+\treturn __pte(pte_val(pte) \u0026 ~(_PAGE_SWP_UFFD));\n }\n #endif /* CONFIG_HAVE_ARCH_USERFAULTFD_WP */\n \n@@ -534,10 +534,7 @@ static inline pte_t pte_swp_clear_soft_dirty(pte_t pte)\n \t\t\t\t\tPAGE_SIZE)\n #endif\n \n-#ifdef CONFIG_NUMA_BALANCING\n-/*\n- * See the comment in include/asm-generic/pgtable.h\n- */\n+#ifdef CONFIG_ARCH_HAS_PTE_PROTNONE\n static inline int pte_protnone(pte_t pte)\n {\n \treturn (pte_val(pte) \u0026 (_PAGE_PRESENT | _PAGE_PROT_NONE)) == _PAGE_PROT_NONE;\n@@ -547,7 +544,7 @@ static inline int pmd_protnone(pmd_t pmd)\n {\n \treturn pte_protnone(pmd_pte(pmd));\n }\n-#endif\n+#endif /* CONFIG_ARCH_HAS_PTE_PROTNONE */\n \n /* Modify page protection bits */\n static inline pte_t pte_modify(pte_t pte, pgprot_t newprot)\n@@ -899,34 +896,34 @@ static inline pud_t pud_mkspecial(pud_t pud)\n #endif\n \n #ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP\n-static inline bool pmd_uffd_wp(pmd_t pmd)\n+static inline bool pmd_uffd(pmd_t pmd)\n {\n-\treturn pte_uffd_wp(pmd_pte(pmd));\n+\treturn pte_uffd(pmd_pte(pmd));\n }\n \n-static inline pmd_t pmd_mkuffd_wp(pmd_t pmd)\n+static inline pmd_t pmd_mkuffd(pmd_t pmd)\n {\n-\treturn pte_pmd(pte_mkuffd_wp(pmd_pte(pmd)));\n+\treturn pte_pmd(pte_mkuffd(pmd_pte(pmd)));\n }\n \n-static inline pmd_t pmd_clear_uffd_wp(pmd_t pmd)\n+static inline pmd_t pmd_clear_uffd(pmd_t pmd)\n {\n-\treturn pte_pmd(pte_clear_uffd_wp(pmd_pte(pmd)));\n+\treturn pte_pmd(pte_clear_uffd(pmd_pte(pmd)));\n }\n \n-static inline bool pmd_swp_uffd_wp(pmd_t pmd)\n+static inline bool pmd_swp_uffd(pmd_t pmd)\n {\n-\treturn pte_swp_uffd_wp(pmd_pte(pmd));\n+\treturn pte_swp_uffd(pmd_pte(pmd));\n }\n \n-static inline pmd_t pmd_swp_mkuffd_wp(pmd_t pmd)\n+static inline pmd_t pmd_swp_mkuffd(pmd_t pmd)\n {\n-\treturn pte_pmd(pte_swp_mkuffd_wp(pmd_pte(pmd)));\n+\treturn pte_pmd(pte_swp_mkuffd(pmd_pte(pmd)));\n }\n \n-static inline pmd_t pmd_swp_clear_uffd_wp(pmd_t pmd)\n+static inline pmd_t pmd_swp_clear_uffd(pmd_t pmd)\n {\n-\treturn pte_pmd(pte_swp_clear_uffd_wp(pmd_pte(pmd)));\n+\treturn pte_pmd(pte_swp_clear_uffd(pmd_pte(pmd)));\n }\n #endif /* CONFIG_HAVE_ARCH_USERFAULTFD_WP */\n \n@@ -1170,7 +1167,7 @@ static inline pud_t pud_modify(pud_t pud, pgprot_t newprot)\n *\tbit 0:\t_PAGE_PRESENT (zero)\n *\tbit 1 to 2:\t(zero)\n *\tbit 3:\t_PAGE_SWP_SOFT_DIRTY\n- *\tbit 4:\t_PAGE_SWP_UFFD_WP\n+ *\tbit 4:\t_PAGE_SWP_UFFD\n *\tbit 5:\t_PAGE_PROT_NONE (zero)\n *\tbit 6:\texclusive marker\n *\tbits 7 to 11:\tswap type\ndiff --git a/arch/s390/Kconfig b/arch/s390/Kconfig\nindex 84404e6778d504..ad56daa6f874cd 100644\n--- a/arch/s390/Kconfig\n+++ b/arch/s390/Kconfig\n@@ -148,6 +148,7 @@ config S390\n \tselect ARCH_SUPPORTS_HUGETLBFS\n \tselect ARCH_SUPPORTS_INT128 if CC_HAS_INT128 \u0026\u0026 CC_IS_CLANG\n \tselect ARCH_SUPPORTS_MSEAL_SYSTEM_MAPPINGS\n+\tselect ARCH_HAS_PTE_PROTNONE\n \tselect ARCH_SUPPORTS_NUMA_BALANCING\n \tselect ARCH_SUPPORTS_PAGE_TABLE_CHECK\n \tselect ARCH_SUPPORTS_PER_VMA_LOCK\ndiff --git a/arch/s390/include/asm/hugetlb.h b/arch/s390/include/asm/hugetlb.h\nindex e33a5b587ee4e6..02821a7d9d9138 100644\n--- a/arch/s390/include/asm/hugetlb.h\n+++ b/arch/s390/include/asm/hugetlb.h\n@@ -77,20 +77,20 @@ static inline void huge_ptep_set_wrprotect(struct mm_struct *mm,\n \t__set_huge_pte_at(mm, addr, ptep, pte_wrprotect(pte));\n }\n \n-#define __HAVE_ARCH_HUGE_PTE_MKUFFD_WP\n-static inline pte_t huge_pte_mkuffd_wp(pte_t pte)\n+#define __HAVE_ARCH_HUGE_PTE_MKUFFD\n+static inline pte_t huge_pte_mkuffd(pte_t pte)\n {\n \treturn pte;\n }\n \n-#define __HAVE_ARCH_HUGE_PTE_CLEAR_UFFD_WP\n-static inline pte_t huge_pte_clear_uffd_wp(pte_t pte)\n+#define __HAVE_ARCH_HUGE_PTE_CLEAR_UFFD\n+static inline pte_t huge_pte_clear_uffd(pte_t pte)\n {\n \treturn pte;\n }\n \n-#define __HAVE_ARCH_HUGE_PTE_UFFD_WP\n-static inline int huge_pte_uffd_wp(pte_t pte)\n+#define __HAVE_ARCH_HUGE_PTE_UFFD\n+static inline int huge_pte_uffd(pte_t pte)\n {\n \treturn 0;\n }\ndiff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h\nindex 859ce7c7d45448..0689dfdba40772 100644\n--- a/arch/s390/include/asm/pgtable.h\n+++ b/arch/s390/include/asm/pgtable.h\n@@ -842,7 +842,7 @@ static inline int pte_same(pte_t a, pte_t b)\n \treturn pte_val(a) == pte_val(b);\n }\n \n-#ifdef CONFIG_NUMA_BALANCING\n+#ifdef CONFIG_ARCH_HAS_PTE_PROTNONE\n static inline int pte_protnone(pte_t pte)\n {\n \treturn pte_present(pte) \u0026\u0026 !(pte_val(pte) \u0026 _PAGE_READ);\n@@ -853,7 +853,7 @@ static inline int pmd_protnone(pmd_t pmd)\n \t/* pmd_leaf(pmd) implies pmd_present(pmd) */\n \treturn pmd_leaf(pmd) \u0026\u0026 !(pmd_val(pmd) \u0026 _SEGMENT_ENTRY_READ);\n }\n-#endif\n+#endif /* CONFIG_ARCH_HAS_PTE_PROTNONE */\n \n static inline bool pte_swp_exclusive(pte_t pte)\n {\ndiff --git a/arch/x86/Kconfig b/arch/x86/Kconfig\nindex bdad90f210e4bf..380c4aacd42a1c 100644\n--- a/arch/x86/Kconfig\n+++ b/arch/x86/Kconfig\n@@ -124,6 +124,7 @@ config X86\n \tselect ARCH_SUPPORTS_DEBUG_PAGEALLOC\n \tselect ARCH_SUPPORTS_HUGETLBFS\n \tselect ARCH_SUPPORTS_PAGE_TABLE_CHECK\tif X86_64\n+\tselect ARCH_HAS_PTE_PROTNONE\t\tif X86_64\n \tselect ARCH_SUPPORTS_NUMA_BALANCING\tif X86_64\n \tselect ARCH_SUPPORTS_KMAP_LOCAL_FORCE_MAP\tif NR_CPUS \u003c= 4096\n \tselect ARCH_SUPPORTS_CFI\t\tif X86_64\ndiff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h\nindex ac295ca6c92f99..cfb66e5f380be5 100644\n--- a/arch/x86/include/asm/pgtable.h\n+++ b/arch/x86/include/asm/pgtable.h\n@@ -411,19 +411,19 @@ static inline pte_t pte_wrprotect(pte_t pte)\n }\n \n #ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP\n-static inline int pte_uffd_wp(pte_t pte)\n+static inline int pte_uffd(pte_t pte)\n {\n-\treturn pte_flags(pte) \u0026 _PAGE_UFFD_WP;\n+\treturn pte_flags(pte) \u0026 _PAGE_UFFD;\n }\n \n-static inline pte_t pte_mkuffd_wp(pte_t pte)\n+static inline pte_t pte_mkuffd(pte_t pte)\n {\n-\treturn pte_wrprotect(pte_set_flags(pte, _PAGE_UFFD_WP));\n+\treturn pte_wrprotect(pte_set_flags(pte, _PAGE_UFFD));\n }\n \n-static inline pte_t pte_clear_uffd_wp(pte_t pte)\n+static inline pte_t pte_clear_uffd(pte_t pte)\n {\n-\treturn pte_clear_flags(pte, _PAGE_UFFD_WP);\n+\treturn pte_clear_flags(pte, _PAGE_UFFD);\n }\n #endif /* CONFIG_HAVE_ARCH_USERFAULTFD_WP */\n \n@@ -526,19 +526,19 @@ static inline pmd_t pmd_wrprotect(pmd_t pmd)\n }\n \n #ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP\n-static inline int pmd_uffd_wp(pmd_t pmd)\n+static inline int pmd_uffd(pmd_t pmd)\n {\n-\treturn pmd_flags(pmd) \u0026 _PAGE_UFFD_WP;\n+\treturn pmd_flags(pmd) \u0026 _PAGE_UFFD;\n }\n \n-static inline pmd_t pmd_mkuffd_wp(pmd_t pmd)\n+static inline pmd_t pmd_mkuffd(pmd_t pmd)\n {\n-\treturn pmd_wrprotect(pmd_set_flags(pmd, _PAGE_UFFD_WP));\n+\treturn pmd_wrprotect(pmd_set_flags(pmd, _PAGE_UFFD));\n }\n \n-static inline pmd_t pmd_clear_uffd_wp(pmd_t pmd)\n+static inline pmd_t pmd_clear_uffd(pmd_t pmd)\n {\n-\treturn pmd_clear_flags(pmd, _PAGE_UFFD_WP);\n+\treturn pmd_clear_flags(pmd, _PAGE_UFFD);\n }\n #endif /* CONFIG_HAVE_ARCH_USERFAULTFD_WP */\n \n@@ -985,11 +985,7 @@ static inline int pmd_present(pmd_t pmd)\n \treturn pmd_flags(pmd) \u0026 (_PAGE_PRESENT | _PAGE_PROTNONE | _PAGE_PSE);\n }\n \n-#ifdef CONFIG_NUMA_BALANCING\n-/*\n- * These work without NUMA balancing but the kernel does not care. See the\n- * comment in include/linux/pgtable.h\n- */\n+#ifdef CONFIG_ARCH_HAS_PTE_PROTNONE\n static inline int pte_protnone(pte_t pte)\n {\n \treturn (pte_flags(pte) \u0026 (_PAGE_PROTNONE | _PAGE_PRESENT))\n@@ -1001,7 +997,7 @@ static inline int pmd_protnone(pmd_t pmd)\n \treturn (pmd_flags(pmd) \u0026 (_PAGE_PROTNONE | _PAGE_PRESENT))\n \t\t== _PAGE_PROTNONE;\n }\n-#endif /* CONFIG_NUMA_BALANCING */\n+#endif /* CONFIG_ARCH_HAS_PTE_PROTNONE */\n \n static inline int pmd_none(pmd_t pmd)\n {\n@@ -1564,34 +1560,34 @@ static inline pmd_t pmd_swp_clear_soft_dirty(pmd_t pmd)\n #endif\n \n #ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP\n-static inline pte_t pte_swp_mkuffd_wp(pte_t pte)\n+static inline pte_t pte_swp_mkuffd(pte_t pte)\n {\n-\treturn pte_set_flags(pte, _PAGE_SWP_UFFD_WP);\n+\treturn pte_set_flags(pte, _PAGE_SWP_UFFD);\n }\n \n-static inline int pte_swp_uffd_wp(pte_t pte)\n+static inline int pte_swp_uffd(pte_t pte)\n {\n-\treturn pte_flags(pte) \u0026 _PAGE_SWP_UFFD_WP;\n+\treturn pte_flags(pte) \u0026 _PAGE_SWP_UFFD;\n }\n \n-static inline pte_t pte_swp_clear_uffd_wp(pte_t pte)\n+static inline pte_t pte_swp_clear_uffd(pte_t pte)\n {\n-\treturn pte_clear_flags(pte, _PAGE_SWP_UFFD_WP);\n+\treturn pte_clear_flags(pte, _PAGE_SWP_UFFD);\n }\n \n-static inline pmd_t pmd_swp_mkuffd_wp(pmd_t pmd)\n+static inline pmd_t pmd_swp_mkuffd(pmd_t pmd)\n {\n-\treturn pmd_set_flags(pmd, _PAGE_SWP_UFFD_WP);\n+\treturn pmd_set_flags(pmd, _PAGE_SWP_UFFD);\n }\n \n-static inline int pmd_swp_uffd_wp(pmd_t pmd)\n+static inline int pmd_swp_uffd(pmd_t pmd)\n {\n-\treturn pmd_flags(pmd) \u0026 _PAGE_SWP_UFFD_WP;\n+\treturn pmd_flags(pmd) \u0026 _PAGE_SWP_UFFD;\n }\n \n-static inline pmd_t pmd_swp_clear_uffd_wp(pmd_t pmd)\n+static inline pmd_t pmd_swp_clear_uffd(pmd_t pmd)\n {\n-\treturn pmd_clear_flags(pmd, _PAGE_SWP_UFFD_WP);\n+\treturn pmd_clear_flags(pmd, _PAGE_SWP_UFFD);\n }\n #endif /* CONFIG_HAVE_ARCH_USERFAULTFD_WP */\n \ndiff --git a/arch/x86/include/asm/pgtable_types.h b/arch/x86/include/asm/pgtable_types.h\nindex 2ec250ba467e2d..af08d98be9309c 100644\n--- a/arch/x86/include/asm/pgtable_types.h\n+++ b/arch/x86/include/asm/pgtable_types.h\n@@ -31,7 +31,7 @@\n \n #define _PAGE_BIT_SPECIAL\t_PAGE_BIT_SOFTW1\n #define _PAGE_BIT_CPA_TEST\t_PAGE_BIT_SOFTW1\n-#define _PAGE_BIT_UFFD_WP\t_PAGE_BIT_SOFTW2 /* userfaultfd wrprotected */\n+#define _PAGE_BIT_UFFD\t\t_PAGE_BIT_SOFTW2 /* userfaultfd tracking */\n #define _PAGE_BIT_SOFT_DIRTY\t_PAGE_BIT_SOFTW3 /* software dirty tracking */\n #define _PAGE_BIT_KERNEL_4K\t_PAGE_BIT_SOFTW3 /* page must not be converted to large */\n \n@@ -39,7 +39,7 @@\n #define _PAGE_BIT_SAVED_DIRTY\t_PAGE_BIT_SOFTW5 /* Saved Dirty bit (leaf) */\n #define _PAGE_BIT_NOPTISHADOW\t_PAGE_BIT_SOFTW5 /* No PTI shadow (root PGD) */\n #else\n-/* Shared with _PAGE_BIT_UFFD_WP which is not supported on 32 bit */\n+/* Shared with _PAGE_BIT_UFFD which is not supported on 32 bit */\n #define _PAGE_BIT_SAVED_DIRTY\t_PAGE_BIT_SOFTW2 /* Saved Dirty bit (leaf) */\n #define _PAGE_BIT_NOPTISHADOW\t_PAGE_BIT_SOFTW2 /* No PTI shadow (root PGD) */\n #endif\n@@ -111,11 +111,11 @@\n #endif\n \n #ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP\n-#define _PAGE_UFFD_WP\t\t(_AT(pteval_t, 1) \u003c\u003c _PAGE_BIT_UFFD_WP)\n-#define _PAGE_SWP_UFFD_WP\t_PAGE_USER\n+#define _PAGE_UFFD\t\t(_AT(pteval_t, 1) \u003c\u003c _PAGE_BIT_UFFD)\n+#define _PAGE_SWP_UFFD\t\t_PAGE_USER\n #else\n-#define _PAGE_UFFD_WP\t\t(_AT(pteval_t, 0))\n-#define _PAGE_SWP_UFFD_WP\t(_AT(pteval_t, 0))\n+#define _PAGE_UFFD\t\t(_AT(pteval_t, 0))\n+#define _PAGE_SWP_UFFD\t\t(_AT(pteval_t, 0))\n #endif\n \n #if defined(CONFIG_X86_64) || defined(CONFIG_X86_PAE)\n@@ -129,7 +129,7 @@\n /*\n * The hardware requires shadow stack to be Write=0,Dirty=1. However,\n * there are valid cases where the kernel might create read-only PTEs that\n- * are dirty (e.g., fork(), mprotect(), uffd-wp(), soft-dirty tracking). In\n+ * are dirty (e.g., fork(), mprotect(), userfaultfd, soft-dirty tracking). In\n * this case, the _PAGE_SAVED_DIRTY bit is used instead of the HW-dirty bit,\n * to avoid creating a wrong \"shadow stack\" PTEs. Such PTEs have\n * (Write=0,SavedDirty=1,Dirty=0) set.\n@@ -151,7 +151,7 @@\n #define _COMMON_PAGE_CHG_MASK\t(PTE_PFN_MASK | _PAGE_PCD | _PAGE_PWT |\t\\\n \t\t\t\t _PAGE_SPECIAL | _PAGE_ACCESSED |\t\\\n \t\t\t\t _PAGE_DIRTY_BITS | _PAGE_SOFT_DIRTY |\t\\\n-\t\t\t\t _PAGE_CC | _PAGE_UFFD_WP)\n+\t\t\t\t _PAGE_CC | _PAGE_UFFD)\n #define _PAGE_CHG_MASK\t(_COMMON_PAGE_CHG_MASK | _PAGE_PAT)\n #define _HPAGE_CHG_MASK (_COMMON_PAGE_CHG_MASK | _PAGE_PSE | _PAGE_PAT_LARGE)\n \ndiff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c\nindex d32408f7cd5ed2..2d724156763d2c 100644\n--- a/fs/proc/task_mmu.c\n+++ b/fs/proc/task_mmu.c\n@@ -1237,6 +1237,9 @@ static void show_smap_vma_flags(struct seq_file *m, struct vm_area_struct *vma)\n #ifdef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR\n \t\t[ilog2(VM_UFFD_MINOR)]\t= \"ui\",\n #endif /* CONFIG_HAVE_ARCH_USERFAULTFD_MINOR */\n+#ifdef CONFIG_USERFAULTFD_RWP\n+\t\t[ilog2(VM_UFFD_RWP)]\t= \"ur\",\n+#endif\n #ifdef CONFIG_ARCH_HAS_USER_SHADOW_STACK\n \t\t[ilog2(VM_SHADOW_STACK)] = \"ss\",\n #endif\n@@ -2035,14 +2038,14 @@ static pagemap_entry_t pte_to_pagemap_entry(struct pagemapread *pm,\n \t\tpage = vm_normal_page(vma, addr, pte);\n \t\tif (pte_soft_dirty(pte))\n \t\t\tflags |= PM_SOFT_DIRTY;\n-\t\tif (pte_uffd_wp(pte))\n+\t\tif (pte_uffd(pte))\n \t\t\tflags |= PM_UFFD_WP;\n \t} else {\n \t\tsoftleaf_t entry;\n \n \t\tif (pte_swp_soft_dirty(pte))\n \t\t\tflags |= PM_SOFT_DIRTY;\n-\t\tif (pte_swp_uffd_wp(pte))\n+\t\tif (pte_swp_uffd(pte))\n \t\t\tflags |= PM_UFFD_WP;\n \t\tentry = softleaf_from_pte(pte);\n \t\tif (pm-\u003eshow_pfn) {\n@@ -2108,7 +2111,7 @@ static int pagemap_pmd_range_thp(pmd_t *pmdp, unsigned long addr,\n \t\tflags |= PM_PRESENT;\n \t\tif (pmd_soft_dirty(pmd))\n \t\t\tflags |= PM_SOFT_DIRTY;\n-\t\tif (pmd_uffd_wp(pmd))\n+\t\tif (pmd_uffd(pmd))\n \t\t\tflags |= PM_UFFD_WP;\n \t\tif (pm-\u003eshow_pfn)\n \t\t\tframe = pmd_pfn(pmd) + idx;\n@@ -2127,7 +2130,7 @@ static int pagemap_pmd_range_thp(pmd_t *pmdp, unsigned long addr,\n \t\tflags |= PM_SWAP;\n \t\tif (pmd_swp_soft_dirty(pmd))\n \t\t\tflags |= PM_SOFT_DIRTY;\n-\t\tif (pmd_swp_uffd_wp(pmd))\n+\t\tif (pmd_swp_uffd(pmd))\n \t\t\tflags |= PM_UFFD_WP;\n \t\tpage = softleaf_to_page(entry);\n \t}\n@@ -2232,14 +2235,14 @@ static int pagemap_hugetlb_range(pte_t *ptep, unsigned long hmask,\n \t\t !hugetlb_pmd_shared(ptep))\n \t\t\tflags |= PM_MMAP_EXCLUSIVE;\n \n-\t\tif (huge_pte_uffd_wp(pte))\n+\t\tif (huge_pte_uffd(pte))\n \t\t\tflags |= PM_UFFD_WP;\n \n \t\tflags |= PM_PRESENT;\n \t\tif (pm-\u003eshow_pfn)\n \t\t\tframe = pte_pfn(pte) +\n \t\t\t\t((addr \u0026 ~hmask) \u003e\u003e PAGE_SHIFT);\n-\t} else if (pte_swp_uffd_wp_any(pte)) {\n+\t} else if (pte_swp_uffd_any(pte)) {\n \t\tflags |= PM_UFFD_WP;\n \t}\n \n@@ -2280,7 +2283,7 @@ static const struct mm_walk_ops pagemap_ops = {\n * Bits 5-54 swap offset if swapped\n * Bit 55 pte is soft-dirty (see Documentation/admin-guide/mm/soft-dirty.rst)\n * Bit 56 page exclusively mapped\n- * Bit 57 pte is uffd-wp write-protected\n+ * Bit 57 pte is tracked by userfaultfd (uffd-wp or RWP)\n * Bit 58 pte is a guard region\n * Bits 59-60 zero\n * Bit 61 page is file-page or shared-anon\n@@ -2415,7 +2418,7 @@ static int pagemap_release(struct inode *inode, struct file *file)\n \t\t\t\t PAGE_IS_FILE |\tPAGE_IS_PRESENT |\t\\\n \t\t\t\t PAGE_IS_SWAPPED | PAGE_IS_PFNZERO |\t\\\n \t\t\t\t PAGE_IS_HUGE | PAGE_IS_SOFT_DIRTY |\t\\\n-\t\t\t\t PAGE_IS_GUARD)\n+\t\t\t\t PAGE_IS_GUARD | PAGE_IS_ACCESSED)\n #define PM_SCAN_FLAGS\t\t(PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC)\n \n struct pagemap_scan_private {\n@@ -2440,8 +2443,12 @@ static unsigned long pagemap_page_category(struct pagemap_scan_private *p,\n \n \t\tcategories = PAGE_IS_PRESENT;\n \n-\t\tif (!pte_uffd_wp(pte))\n-\t\t\tcategories |= PAGE_IS_WRITTEN;\n+\t\tif (!pte_uffd(pte)) {\n+\t\t\tif (userfaultfd_wp(vma))\n+\t\t\t\tcategories |= PAGE_IS_WRITTEN;\n+\t\t\tif (userfaultfd_rwp(vma))\n+\t\t\t\tcategories |= PAGE_IS_ACCESSED;\n+\t\t}\n \n \t\tif (p-\u003emasks_of_interest \u0026 PAGE_IS_FILE) {\n \t\t\tpage = vm_normal_page(vma, addr, pte);\n@@ -2458,8 +2465,12 @@ static unsigned long pagemap_page_category(struct pagemap_scan_private *p,\n \n \t\tcategories = PAGE_IS_SWAPPED;\n \n-\t\tif (!pte_swp_uffd_wp_any(pte))\n-\t\t\tcategories |= PAGE_IS_WRITTEN;\n+\t\tif (!pte_swp_uffd_any(pte)) {\n+\t\t\tif (userfaultfd_wp(vma))\n+\t\t\t\tcategories |= PAGE_IS_WRITTEN;\n+\t\t\tif (userfaultfd_rwp(vma))\n+\t\t\t\tcategories |= PAGE_IS_ACCESSED;\n+\t\t}\n \n \t\tentry = softleaf_from_pte(pte);\n \t\tif (softleaf_is_guard_marker(entry))\n@@ -2483,13 +2494,13 @@ static void make_uffd_wp_pte(struct vm_area_struct *vma,\n \t\tpte_t old_pte;\n \n \t\told_pte = ptep_modify_prot_start(vma, addr, pte);\n-\t\tptent = pte_mkuffd_wp(old_pte);\n+\t\tptent = pte_mkuffd(old_pte);\n \t\tptep_modify_prot_commit(vma, addr, pte, old_pte, ptent);\n \t} else if (pte_none(ptent)) {\n \t\tset_pte_at(vma-\u003evm_mm, addr, pte,\n \t\t\t make_pte_marker(PTE_MARKER_UFFD_WP));\n \t} else {\n-\t\tptent = pte_swp_mkuffd_wp(ptent);\n+\t\tptent = pte_swp_mkuffd(ptent);\n \t\tset_pte_at(vma-\u003evm_mm, addr, pte, ptent);\n \t}\n }\n@@ -2508,8 +2519,12 @@ static unsigned long pagemap_thp_category(struct pagemap_scan_private *p,\n \t\tstruct page *page;\n \n \t\tcategories |= PAGE_IS_PRESENT;\n-\t\tif (!pmd_uffd_wp(pmd))\n-\t\t\tcategories |= PAGE_IS_WRITTEN;\n+\t\tif (!pmd_uffd(pmd)) {\n+\t\t\tif (userfaultfd_wp(vma))\n+\t\t\t\tcategories |= PAGE_IS_WRITTEN;\n+\t\t\tif (userfaultfd_rwp(vma))\n+\t\t\t\tcategories |= PAGE_IS_ACCESSED;\n+\t\t}\n \n \t\tif (p-\u003emasks_of_interest \u0026 PAGE_IS_FILE) {\n \t\t\tpage = vm_normal_page_pmd(vma, addr, pmd);\n@@ -2523,8 +2538,12 @@ static unsigned long pagemap_thp_category(struct pagemap_scan_private *p,\n \t\t\tcategories |= PAGE_IS_SOFT_DIRTY;\n \t} else {\n \t\tcategories |= PAGE_IS_SWAPPED;\n-\t\tif (!pmd_swp_uffd_wp(pmd))\n-\t\t\tcategories |= PAGE_IS_WRITTEN;\n+\t\tif (!pmd_swp_uffd(pmd)) {\n+\t\t\tif (userfaultfd_wp(vma))\n+\t\t\t\tcategories |= PAGE_IS_WRITTEN;\n+\t\t\tif (userfaultfd_rwp(vma))\n+\t\t\t\tcategories |= PAGE_IS_ACCESSED;\n+\t\t}\n \t\tif (pmd_swp_soft_dirty(pmd))\n \t\t\tcategories |= PAGE_IS_SOFT_DIRTY;\n \n@@ -2547,17 +2566,18 @@ static void make_uffd_wp_pmd(struct vm_area_struct *vma,\n \n \tif (pmd_present(pmd)) {\n \t\told = pmdp_invalidate_ad(vma, addr, pmdp);\n-\t\tpmd = pmd_mkuffd_wp(old);\n+\t\tpmd = pmd_mkuffd(old);\n \t\tset_pmd_at(vma-\u003evm_mm, addr, pmdp, pmd);\n \t} else if (pmd_is_migration_entry(pmd)) {\n-\t\tpmd = pmd_swp_mkuffd_wp(pmd);\n+\t\tpmd = pmd_swp_mkuffd(pmd);\n \t\tset_pmd_at(vma-\u003evm_mm, addr, pmdp, pmd);\n \t}\n }\n #endif /* CONFIG_TRANSPARENT_HUGEPAGE */\n \n #ifdef CONFIG_HUGETLB_PAGE\n-static unsigned long pagemap_hugetlb_category(pte_t pte)\n+static unsigned long pagemap_hugetlb_category(struct vm_area_struct *vma,\n+\t\t\t\t\t pte_t pte)\n {\n \tunsigned long categories = PAGE_IS_HUGE;\n \n@@ -2572,8 +2592,12 @@ static unsigned long pagemap_hugetlb_category(pte_t pte)\n \tif (pte_present(pte)) {\n \t\tcategories |= PAGE_IS_PRESENT;\n \n-\t\tif (!huge_pte_uffd_wp(pte))\n-\t\t\tcategories |= PAGE_IS_WRITTEN;\n+\t\tif (!huge_pte_uffd(pte)) {\n+\t\t\tif (userfaultfd_wp(vma))\n+\t\t\t\tcategories |= PAGE_IS_WRITTEN;\n+\t\t\tif (userfaultfd_rwp(vma))\n+\t\t\t\tcategories |= PAGE_IS_ACCESSED;\n+\t\t}\n \t\tif (!PageAnon(pte_page(pte)))\n \t\t\tcategories |= PAGE_IS_FILE;\n \t\tif (is_zero_pfn(pte_pfn(pte)))\n@@ -2583,8 +2607,12 @@ static unsigned long pagemap_hugetlb_category(pte_t pte)\n \t} else {\n \t\tcategories |= PAGE_IS_SWAPPED;\n \n-\t\tif (!pte_swp_uffd_wp_any(pte))\n-\t\t\tcategories |= PAGE_IS_WRITTEN;\n+\t\tif (!pte_swp_uffd_any(pte)) {\n+\t\t\tif (userfaultfd_wp(vma))\n+\t\t\t\tcategories |= PAGE_IS_WRITTEN;\n+\t\t\tif (userfaultfd_rwp(vma))\n+\t\t\t\tcategories |= PAGE_IS_ACCESSED;\n+\t\t}\n \t\tif (pte_swp_soft_dirty(pte))\n \t\t\tcategories |= PAGE_IS_SOFT_DIRTY;\n \t}\n@@ -2611,12 +2639,12 @@ static void make_uffd_wp_huge_pte(struct vm_area_struct *vma,\n \n \tif (softleaf_is_migration(entry)) {\n \t\tset_huge_pte_at(vma-\u003evm_mm, addr, ptep,\n-\t\t\t\tpte_swp_mkuffd_wp(ptent), psize);\n+\t\t\t\tpte_swp_mkuffd(ptent), psize);\n \t} else {\n \t\tpte_t old_pte, new_pte;\n \n \t\told_pte = huge_ptep_modify_prot_start(vma, addr, ptep);\n-\t\tnew_pte = huge_pte_mkuffd_wp(old_pte);\n+\t\tnew_pte = huge_pte_mkuffd(old_pte);\n \t\thuge_ptep_modify_prot_commit(vma, addr, ptep, old_pte, new_pte);\n \t}\n }\n@@ -2849,8 +2877,8 @@ static int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start,\n \t\tfor (addr = start; addr != end; pte++, addr += PAGE_SIZE) {\n \t\t\tpte_t ptent = ptep_get(pte);\n \n-\t\t\tif ((pte_present(ptent) \u0026\u0026 pte_uffd_wp(ptent)) ||\n-\t\t\t pte_swp_uffd_wp_any(ptent))\n+\t\t\tif ((pte_present(ptent) \u0026\u0026 pte_uffd(ptent)) ||\n+\t\t\t pte_swp_uffd_any(ptent))\n \t\t\t\tcontinue;\n \t\t\tmake_uffd_wp_pte(vma, addr, pte, ptent);\n \t\t\tif (!flush_end)\n@@ -2860,15 +2888,16 @@ static int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start,\n \t\tgoto flush_and_return;\n \t}\n \n-\tif (!p-\u003earg.category_anyof_mask \u0026\u0026 !p-\u003earg.category_inverted \u0026\u0026\n+\tif (userfaultfd_wp(vma) \u0026\u0026 !p-\u003earg.category_anyof_mask \u0026\u0026\n+\t !p-\u003earg.category_inverted \u0026\u0026\n \t p-\u003earg.category_mask == PAGE_IS_WRITTEN \u0026\u0026\n \t p-\u003earg.return_mask == PAGE_IS_WRITTEN) {\n \t\tfor (addr = start; addr \u003c end; pte++, addr += PAGE_SIZE) {\n \t\t\tunsigned long next = addr + PAGE_SIZE;\n \t\t\tpte_t ptent = ptep_get(pte);\n \n-\t\t\tif ((pte_present(ptent) \u0026\u0026 pte_uffd_wp(ptent)) ||\n-\t\t\t pte_swp_uffd_wp_any(ptent))\n+\t\t\tif ((pte_present(ptent) \u0026\u0026 pte_uffd(ptent)) ||\n+\t\t\t pte_swp_uffd_any(ptent))\n \t\t\t\tcontinue;\n \t\t\tret = pagemap_scan_output(p-\u003ecur_vma_category | PAGE_IS_WRITTEN,\n \t\t\t\t\t\t p, addr, \u0026next);\n@@ -2935,7 +2964,8 @@ static int pagemap_scan_hugetlb_entry(pte_t *ptep, unsigned long hmask,\n \t\t/* Go the short route when not write-protecting pages. */\n \n \t\tpte = huge_ptep_get(walk-\u003emm, start, ptep);\n-\t\tcategories = p-\u003ecur_vma_category | pagemap_hugetlb_category(pte);\n+\t\tcategories = p-\u003ecur_vma_category |\n+\t\t\t pagemap_hugetlb_category(vma, pte);\n \n \t\tif (!pagemap_scan_is_interesting_page(categories, p))\n \t\t\treturn 0;\n@@ -2947,7 +2977,7 @@ static int pagemap_scan_hugetlb_entry(pte_t *ptep, unsigned long hmask,\n \tptl = huge_pte_lock(hstate_vma(vma), vma-\u003evm_mm, ptep);\n \n \tpte = huge_ptep_get(walk-\u003emm, start, ptep);\n-\tcategories = p-\u003ecur_vma_category | pagemap_hugetlb_category(pte);\n+\tcategories = p-\u003ecur_vma_category | pagemap_hugetlb_category(vma, pte);\n \n \tif (!pagemap_scan_is_interesting_page(categories, p))\n \t\tgoto out_unlock;\ndiff --git a/include/asm-generic/hugetlb.h b/include/asm-generic/hugetlb.h\nindex e1a2e1b7c8e7b8..635c41cc34797b 100644\n--- a/include/asm-generic/hugetlb.h\n+++ b/include/asm-generic/hugetlb.h\n@@ -37,24 +37,24 @@ static inline pte_t huge_pte_modify(pte_t pte, pgprot_t newprot)\n \treturn pte_modify(pte, newprot);\n }\n \n-#ifndef __HAVE_ARCH_HUGE_PTE_MKUFFD_WP\n-static inline pte_t huge_pte_mkuffd_wp(pte_t pte)\n+#ifndef __HAVE_ARCH_HUGE_PTE_MKUFFD\n+static inline pte_t huge_pte_mkuffd(pte_t pte)\n {\n-\treturn huge_pte_wrprotect(pte_mkuffd_wp(pte));\n+\treturn huge_pte_wrprotect(pte_mkuffd(pte));\n }\n #endif\n \n-#ifndef __HAVE_ARCH_HUGE_PTE_CLEAR_UFFD_WP\n-static inline pte_t huge_pte_clear_uffd_wp(pte_t pte)\n+#ifndef __HAVE_ARCH_HUGE_PTE_CLEAR_UFFD\n+static inline pte_t huge_pte_clear_uffd(pte_t pte)\n {\n-\treturn pte_clear_uffd_wp(pte);\n+\treturn pte_clear_uffd(pte);\n }\n #endif\n \n-#ifndef __HAVE_ARCH_HUGE_PTE_UFFD_WP\n-static inline int huge_pte_uffd_wp(pte_t pte)\n+#ifndef __HAVE_ARCH_HUGE_PTE_UFFD\n+static inline int huge_pte_uffd(pte_t pte)\n {\n-\treturn pte_uffd_wp(pte);\n+\treturn pte_uffd(pte);\n }\n #endif\n \ndiff --git a/include/asm-generic/pgtable_uffd.h b/include/asm-generic/pgtable_uffd.h\nindex 0d85791efdf778..30e88fc1de2f02 100644\n--- a/include/asm-generic/pgtable_uffd.h\n+++ b/include/asm-generic/pgtable_uffd.h\n@@ -2,79 +2,79 @@\n #define _ASM_GENERIC_PGTABLE_UFFD_H\n \n /*\n- * Some platforms can customize the uffd-wp bit, making it unavailable\n+ * Some platforms can customize the uffd PTE bit, making it unavailable\n * even if the architecture provides the resource.\n * Adding this API allows architectures to add their own checks for the\n * devices on which the kernel is running.\n * Note: When overriding it, please make sure the\n * CONFIG_HAVE_ARCH_USERFAULTFD_WP is part of this macro.\n */\n-#ifndef pgtable_supports_uffd_wp\n-#define pgtable_supports_uffd_wp()\tIS_ENABLED(CONFIG_HAVE_ARCH_USERFAULTFD_WP)\n+#ifndef pgtable_supports_uffd\n+#define pgtable_supports_uffd()\tIS_ENABLED(CONFIG_HAVE_ARCH_USERFAULTFD_WP)\n #endif\n \n static inline bool uffd_supports_wp_marker(void)\n {\n-\treturn pgtable_supports_uffd_wp() \u0026\u0026 IS_ENABLED(CONFIG_PTE_MARKER_UFFD_WP);\n+\treturn pgtable_supports_uffd() \u0026\u0026 IS_ENABLED(CONFIG_PTE_MARKER_UFFD_WP);\n }\n \n #ifndef CONFIG_HAVE_ARCH_USERFAULTFD_WP\n-static __always_inline int pte_uffd_wp(pte_t pte)\n+static __always_inline int pte_uffd(pte_t pte)\n {\n \treturn 0;\n }\n \n-static __always_inline int pmd_uffd_wp(pmd_t pmd)\n+static __always_inline int pmd_uffd(pmd_t pmd)\n {\n \treturn 0;\n }\n \n-static __always_inline pte_t pte_mkuffd_wp(pte_t pte)\n+static __always_inline pte_t pte_mkuffd(pte_t pte)\n {\n \treturn pte;\n }\n \n-static __always_inline pmd_t pmd_mkuffd_wp(pmd_t pmd)\n+static __always_inline pmd_t pmd_mkuffd(pmd_t pmd)\n {\n \treturn pmd;\n }\n \n-static __always_inline pte_t pte_clear_uffd_wp(pte_t pte)\n+static __always_inline pte_t pte_clear_uffd(pte_t pte)\n {\n \treturn pte;\n }\n \n-static __always_inline pmd_t pmd_clear_uffd_wp(pmd_t pmd)\n+static __always_inline pmd_t pmd_clear_uffd(pmd_t pmd)\n {\n \treturn pmd;\n }\n \n-static __always_inline pte_t pte_swp_mkuffd_wp(pte_t pte)\n+static __always_inline pte_t pte_swp_mkuffd(pte_t pte)\n {\n \treturn pte;\n }\n \n-static __always_inline int pte_swp_uffd_wp(pte_t pte)\n+static __always_inline int pte_swp_uffd(pte_t pte)\n {\n \treturn 0;\n }\n \n-static __always_inline pte_t pte_swp_clear_uffd_wp(pte_t pte)\n+static __always_inline pte_t pte_swp_clear_uffd(pte_t pte)\n {\n \treturn pte;\n }\n \n-static inline pmd_t pmd_swp_mkuffd_wp(pmd_t pmd)\n+static inline pmd_t pmd_swp_mkuffd(pmd_t pmd)\n {\n \treturn pmd;\n }\n \n-static inline int pmd_swp_uffd_wp(pmd_t pmd)\n+static inline int pmd_swp_uffd(pmd_t pmd)\n {\n \treturn 0;\n }\n \n-static inline pmd_t pmd_swp_clear_uffd_wp(pmd_t pmd)\n+static inline pmd_t pmd_swp_clear_uffd(pmd_t pmd)\n {\n \treturn pmd;\n }\ndiff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h\nindex ad20f7f8c17944..9c4ebf0fc227a8 100644\n--- a/include/linux/huge_mm.h\n+++ b/include/linux/huge_mm.h\n@@ -529,6 +529,8 @@ static inline bool folio_test_pmd_mappable(struct folio *folio)\n \n vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf);\n \n+vm_fault_t do_huge_pmd_uffd_rwp(struct vm_fault *vmf);\n+\n vm_fault_t do_huge_pmd_device_private(struct vm_fault *vmf);\n \n extern struct folio *huge_zero_folio;\n@@ -723,6 +725,11 @@ static inline spinlock_t *pud_trans_huge_lock(pud_t *pud,\n \treturn NULL;\n }\n \n+static inline vm_fault_t do_huge_pmd_uffd_rwp(struct vm_fault *vmf)\n+{\n+\treturn 0;\n+}\n+\n static inline vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)\n {\n \treturn 0;\ndiff --git a/include/linux/leafops.h b/include/linux/leafops.h\nindex 992cd8bd8ed0e9..2ce2f37ac883bc 100644\n--- a/include/linux/leafops.h\n+++ b/include/linux/leafops.h\n@@ -100,8 +100,8 @@ static inline softleaf_t softleaf_from_pmd(pmd_t pmd)\n \n \tif (pmd_swp_soft_dirty(pmd))\n \t\tpmd = pmd_swp_clear_soft_dirty(pmd);\n-\tif (pmd_swp_uffd_wp(pmd))\n-\t\tpmd = pmd_swp_clear_uffd_wp(pmd);\n+\tif (pmd_swp_uffd(pmd))\n+\t\tpmd = pmd_swp_clear_uffd(pmd);\n \tarch_entry = __pmd_to_swp_entry(pmd);\n \n \t/* Temporary until swp_entry_t eliminated. */\ndiff --git a/include/linux/mm.h b/include/linux/mm.h\nindex 485df9c2dbddb3..da8a07cd49a148 100644\n--- a/include/linux/mm.h\n+++ b/include/linux/mm.h\n@@ -353,6 +353,7 @@ enum {\n #endif\n \tDECLARE_VMA_BIT(UFFD_MINOR, 41),\n \tDECLARE_VMA_BIT(SEALED, 42),\n+\tDECLARE_VMA_BIT(UFFD_RWP, 43),\n \t/* Flags that reuse flags above. */\n \tDECLARE_VMA_BIT_ALIAS(PKEY_BIT0, HIGH_ARCH_0),\n \tDECLARE_VMA_BIT_ALIAS(PKEY_BIT1, HIGH_ARCH_1),\n@@ -496,12 +497,17 @@ enum {\n #else\n #define VM_UFFD_MINOR\tVM_NONE\n #endif\n+#ifdef CONFIG_USERFAULTFD_RWP\n+#define VM_UFFD_RWP\t\tINIT_VM_FLAG(UFFD_RWP)\n+#else\n+#define VM_UFFD_RWP\t\tVM_NONE\n+#endif\n \n /*\n- * vma_flags_t masks for the userfaultfd VMA flags. VMA_UFFD_MINOR is gated on\n- * the same config as VM_UFFD_MINOR -- which implies 64BIT, where the bit fits\n- * -- so an out-of-range bit is never fed to mk_vma_flags() on a build whose\n- * bitmap cannot hold it.\n+ * vma_flags_t masks for the userfaultfd VMA flags. The two high-bit modes are\n+ * gated on the same configs as their VM_* flags above -- both of which imply\n+ * 64BIT -- so an out-of-range bit is never fed to mk_vma_flags() on a build\n+ * whose bitmap cannot hold it.\n */\n #define VMA_UFFD_MISSING\tmk_vma_flags(VMA_UFFD_MISSING_BIT)\n #define VMA_UFFD_WP\t\tmk_vma_flags(VMA_UFFD_WP_BIT)\n@@ -510,6 +516,11 @@ enum {\n #else\n #define VMA_UFFD_MINOR\t\tEMPTY_VMA_FLAGS\n #endif\n+#ifdef CONFIG_USERFAULTFD_RWP\n+#define VMA_UFFD_RWP\t\tmk_vma_flags(VMA_UFFD_RWP_BIT)\n+#else\n+#define VMA_UFFD_RWP\t\tEMPTY_VMA_FLAGS\n+#endif\n \n #ifdef CONFIG_64BIT\n #define VM_ALLOW_ANY_UNCACHED\tINIT_VM_FLAG(ALLOW_ANY_UNCACHED)\n@@ -648,29 +659,32 @@ enum {\n * reconsistuted upon page fault, so necessitate page table copying upon fork.\n *\n * Note that these flags should be compared with the DESTINATION VMA not the\n- * source, as VM_UFFD_WP may not be propagated to destination, while all other\n- * flags will be.\n+ * source: VM_UFFD_WP and VM_UFFD_RWP may be cleared on the destination\n+ * (dup_userfaultfd() -\u003e userfaultfd_reset_ctx() when the parent context did\n+ * not negotiate UFFD_FEATURE_EVENT_FORK), while all other flags propagate.\n *\n * VM_PFNMAP / VM_MIXEDMAP - These contain kernel-mapped data which cannot be\n * reasonably reconstructed on page fault.\n *\n * VM_UFFD_WP - Encodes metadata about an installed uffd\n- * write protect handler, which cannot be\n- * reconstructed on page fault.\n+ * VM_UFFD_RWP write- or read-write-protect handler, which\n+ * cannot be reconstructed on page fault.\n *\n- * We always copy pgtables when dst_vma has uffd-wp\n- * enabled even if it's file-backed\n- * (e.g. shmem). Because when uffd-wp is enabled,\n- * pgtable contains uffd-wp protection information,\n- * that's something we can't retrieve from page cache,\n- * and skip copying will lose those info.\n+ * We always copy pgtables when dst_vma has the\n+ * uffd PTE bit in use even if it's file-backed\n+ * (e.g. shmem). Because when the uffd bit is\n+ * in use, the pgtable contains the protection\n+ * information, that's something we can't\n+ * retrieve from page cache, and skip copying\n+ * will lose those info.\n *\n * VM_MAYBE_GUARD - Could contain page guard region markers which\n * by design are a property of the page tables\n * only and thus cannot be reconstructed on page\n * fault.\n */\n-#define VM_COPY_ON_FORK (VM_PFNMAP | VM_MIXEDMAP | VM_UFFD_WP | VM_MAYBE_GUARD)\n+#define VM_COPY_ON_FORK (VM_PFNMAP | VM_MIXEDMAP | VM_UFFD_WP | VM_UFFD_RWP | \\\n+\t\t\t VM_MAYBE_GUARD)\n \n /*\n * mapping from the currently active vm_flags protection bits (the\n@@ -3317,6 +3331,11 @@ int get_cmdline(struct task_struct *task, char *buffer, int buflen);\n #define MM_CP_UFFD_WP_RESOLVE (1UL \u003c\u003c 3) /* Resolve wp */\n #define MM_CP_UFFD_WP_ALL (MM_CP_UFFD_WP | \\\n \t\t\t\t\t MM_CP_UFFD_WP_RESOLVE)\n+/* Whether this change is for uffd RWP */\n+#define MM_CP_UFFD_RWP (1UL \u003c\u003c 4) /* do rwp */\n+#define MM_CP_UFFD_RWP_RESOLVE (1UL \u003c\u003c 5) /* resolve rwp */\n+#define MM_CP_UFFD_RWP_ALL (MM_CP_UFFD_RWP | \\\n+\t\t\t\t\t MM_CP_UFFD_RWP_RESOLVE)\n \n bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr,\n \t\t\t pte_t pte);\n@@ -4625,11 +4644,25 @@ static inline int vm_fault_to_errno(vm_fault_t vm_fault, int foll_flags)\n \n /*\n * Indicates whether GUP can follow a PROT_NONE mapped page, or whether\n- * a (NUMA hinting) fault is required.\n+ * a (NUMA hinting or userfaultfd RWP) fault is required.\n */\n static inline bool gup_can_follow_protnone(const struct vm_area_struct *vma,\n \t\t\t\t\t unsigned int flags)\n {\n+\t/*\n+\t * VM_UFFD_RWP uses protnone as an access-tracking marker, not for\n+\t * NUMA hinting. GUP must always take a fault so the access is\n+\t * delivered to userfaultfd, regardless of FOLL_HONOR_NUMA_FAULT.\n+\t *\n+\t * Only do so while the VMA is accessible. If it has been made\n+\t * inaccessible (e.g. mprotect(PROT_NONE)), fall through to the guard\n+\t * below: forcing a fault there would loop, as handle_mm_fault() makes\n+\t * no progress on protnone in an inaccessible VMA, and the access is\n+\t * denied regardless of RWP anyway.\n+\t */\n+\tif (vma_test_single_mask(vma, VMA_UFFD_RWP) \u0026\u0026 vma_is_accessible(vma))\n+\t\treturn false;\n+\n \t/*\n \t * If callers don't want to honor NUMA hinting faults, no need to\n \t * determine if we would actually have to trigger a NUMA hinting fault.\ndiff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h\nindex a8430a7ae05447..d839654db1d265 100644\n--- a/include/linux/mm_inline.h\n+++ b/include/linux/mm_inline.h\n@@ -600,14 +600,14 @@ pte_install_uffd_wp_if_needed(struct vm_area_struct *vma, unsigned long addr,\n \t\treturn false;\n \n \t/* A uffd-wp wr-protected normal pte */\n-\tif (unlikely(pte_present(pteval) \u0026\u0026 pte_uffd_wp(pteval)))\n+\tif (unlikely(pte_present(pteval) \u0026\u0026 pte_uffd(pteval)))\n \t\tarm_uffd_pte = true;\n \n \t/*\n \t * A uffd-wp wr-protected swap pte. Note: this should even cover an\n \t * existing pte marker with uffd-wp bit set.\n \t */\n-\tif (unlikely(pte_swp_uffd_wp_any(pteval)))\n+\tif (unlikely(pte_swp_uffd_any(pteval)))\n \t\tarm_uffd_pte = true;\n \n \tif (unlikely(arm_uffd_pte)) {\ndiff --git a/include/linux/pgtable.h b/include/linux/pgtable.h\nindex 2981e386da7b3c..7d3dc8abc6c878 100644\n--- a/include/linux/pgtable.h\n+++ b/include/linux/pgtable.h\n@@ -2129,18 +2129,26 @@ static inline int pud_trans_unstable(pud_t *pud)\n \treturn 0;\n }\n \n-#ifndef CONFIG_NUMA_BALANCING\n+#ifndef CONFIG_ARCH_HAS_PTE_PROTNONE\n /*\n- * In an inaccessible (PROT_NONE) VMA, pte_protnone() may indicate \"yes\". It is\n- * perfectly valid to indicate \"no\" in that case, which is why our default\n- * implementation defaults to \"always no\".\n+ * In an inaccessible (PROT_NONE) VMA, pte_protnone() may indicate \"yes\". It\n+ * is perfectly valid to indicate \"no\" in that case, which is why our\n+ * default implementation defaults to \"always no\".\n *\n- * In an accessible VMA, however, pte_protnone() reliably indicates PROT_NONE\n- * page protection due to NUMA hinting. NUMA hinting faults only apply in\n- * accessible VMAs.\n+ * In an accessible VMA, pte_protnone() reliably indicates a present\n+ * PROT_NONE page protection. Today the kernel uses such PTEs for two\n+ * purposes: NUMA hinting faults, and userfaultfd RWP tracking on\n+ * VM_UFFD_RWP VMAs. The two are distinguished by the uffd PTE bit and\n+ * the VMA flag; see include/linux/userfaultfd_k.h.\n *\n- * So, to reliably identify PROT_NONE PTEs that require a NUMA hinting fault,\n- * looking at the VMA accessibility is sufficient.\n+ * So, to reliably identify PROT_NONE PTEs that require kernel handling,\n+ * looking at the VMA accessibility (and the uffd bit on RWP VMAs) is\n+ * sufficient.\n+ *\n+ * Architectures without CONFIG_ARCH_HAS_PTE_PROTNONE get the always-zero\n+ * stubs below; PAGE_NONE references that survive to runtime fire the\n+ * BUILD_BUG() fallback, since callers should have folded such paths to\n+ * dead code via IS_ENABLED(CONFIG_ARCH_HAS_PTE_PROTNONE).\n */\n static inline int pte_protnone(pte_t pte)\n {\n@@ -2151,7 +2159,11 @@ static inline int pmd_protnone(pmd_t pmd)\n {\n \treturn 0;\n }\n-#endif /* CONFIG_NUMA_BALANCING */\n+\n+#ifndef PAGE_NONE\n+#define PAGE_NONE ({ BUILD_BUG(); (pgprot_t){0}; })\n+#endif\n+#endif /* CONFIG_ARCH_HAS_PTE_PROTNONE */\n \n #endif /* CONFIG_MMU */\n \ndiff --git a/include/linux/swapops.h b/include/linux/swapops.h\nindex 8cfc966eae48eb..15c6440e38dd8c 100644\n--- a/include/linux/swapops.h\n+++ b/include/linux/swapops.h\n@@ -73,8 +73,8 @@ static inline pte_t pte_swp_clear_flags(pte_t pte)\n \t\tpte = pte_swp_clear_exclusive(pte);\n \tif (pte_swp_soft_dirty(pte))\n \t\tpte = pte_swp_clear_soft_dirty(pte);\n-\tif (pte_swp_uffd_wp(pte))\n-\t\tpte = pte_swp_clear_uffd_wp(pte);\n+\tif (pte_swp_uffd(pte))\n+\t\tpte = pte_swp_clear_uffd(pte);\n \treturn pte;\n }\n \ndiff --git a/include/linux/userfaultfd_k.h b/include/linux/userfaultfd_k.h\nindex 68edac4dcd78b3..ca718e8590c5cb 100644\n--- a/include/linux/userfaultfd_k.h\n+++ b/include/linux/userfaultfd_k.h\n@@ -21,10 +21,11 @@\n #include \u003clinux/hugetlb_inline.h\u003e\n \n /* The set of all possible UFFD-related VM flags. */\n-#define __VM_UFFD_FLAGS (VM_UFFD_MISSING | VM_UFFD_WP | VM_UFFD_MINOR)\n+#define __VM_UFFD_FLAGS (VM_UFFD_MISSING | VM_UFFD_MINOR | \\\n+\t\t\t VM_UFFD_WP | VM_UFFD_RWP)\n \n #define __VMA_UFFD_FLAGS mk_vma_flags_from_masks(VMA_UFFD_MISSING, VMA_UFFD_WP, \\\n-\t\t\t\t\t\t VMA_UFFD_MINOR)\n+\t\t\t\t\t\t VMA_UFFD_MINOR, VMA_UFFD_RWP)\n \n /*\n * CAREFUL: Check include/uapi/asm-generic/fcntl.h when defining\n@@ -149,6 +150,8 @@ static inline uffd_flags_t uffd_flags_set_mode(uffd_flags_t flags, enum mfill_at\n \n extern long uffd_wp_range(struct vm_area_struct *vma,\n \t\t\t unsigned long start, unsigned long len, bool enable_wp);\n+extern int mrwprotect_range(struct userfaultfd_ctx *ctx, unsigned long start,\n+\t\t\t unsigned long len, bool enable_rwp);\n \n /* move_pages */\n void double_pt_lock(spinlock_t *ptl1, spinlock_t *ptl2);\n@@ -178,7 +181,9 @@ static inline bool is_mergeable_vm_userfaultfd_ctx(struct vm_area_struct *vma,\n */\n static inline bool uffd_disable_huge_pmd_share(struct vm_area_struct *vma)\n {\n-\treturn vma-\u003evm_flags \u0026 (VM_UFFD_WP | VM_UFFD_MINOR);\n+\treturn vma_test_any_mask(vma,\n+\t\tmk_vma_flags_from_masks(VMA_UFFD_WP, VMA_UFFD_MINOR,\n+\t\t\t\t\tVMA_UFFD_RWP));\n }\n \n /*\n@@ -190,39 +195,62 @@ static inline bool uffd_disable_huge_pmd_share(struct vm_area_struct *vma)\n */\n static inline bool uffd_disable_fault_around(struct vm_area_struct *vma)\n {\n-\treturn vma-\u003evm_flags \u0026 (VM_UFFD_WP | VM_UFFD_MINOR);\n+\treturn vma_test_any_mask(vma,\n+\t\tmk_vma_flags_from_masks(VMA_UFFD_WP, VMA_UFFD_MINOR));\n }\n \n static inline bool userfaultfd_missing(struct vm_area_struct *vma)\n {\n-\treturn vma-\u003evm_flags \u0026 VM_UFFD_MISSING;\n+\treturn vma_test_any_mask(vma, VMA_UFFD_MISSING);\n }\n \n static inline bool userfaultfd_wp(struct vm_area_struct *vma)\n {\n-\treturn vma-\u003evm_flags \u0026 VM_UFFD_WP;\n+\treturn vma_test_any_mask(vma, VMA_UFFD_WP);\n }\n \n static inline bool userfaultfd_minor(struct vm_area_struct *vma)\n {\n-\treturn vma-\u003evm_flags \u0026 VM_UFFD_MINOR;\n+\treturn vma_test_any_mask(vma, VMA_UFFD_MINOR);\n+}\n+\n+static inline bool userfaultfd_rwp(struct vm_area_struct *vma)\n+{\n+\treturn vma_test_single_mask(vma, VMA_UFFD_RWP);\n+}\n+\n+static inline bool userfaultfd_protected(struct vm_area_struct *vma)\n+{\n+\treturn userfaultfd_wp(vma) || userfaultfd_rwp(vma);\n }\n \n static inline bool userfaultfd_pte_wp(struct vm_area_struct *vma,\n \t\t\t\t pte_t pte)\n {\n-\treturn userfaultfd_wp(vma) \u0026\u0026 pte_uffd_wp(pte);\n+\treturn userfaultfd_wp(vma) \u0026\u0026 pte_uffd(pte);\n }\n \n static inline bool userfaultfd_huge_pmd_wp(struct vm_area_struct *vma,\n \t\t\t\t\t pmd_t pmd)\n {\n-\treturn userfaultfd_wp(vma) \u0026\u0026 pmd_uffd_wp(pmd);\n+\treturn userfaultfd_wp(vma) \u0026\u0026 pmd_uffd(pmd);\n+}\n+\n+static inline bool userfaultfd_pte_rwp(struct vm_area_struct *vma,\n+\t\t\t\t pte_t pte)\n+{\n+\treturn userfaultfd_rwp(vma) \u0026\u0026 pte_uffd(pte);\n+}\n+\n+static inline bool userfaultfd_huge_pmd_rwp(struct vm_area_struct *vma,\n+\t\t\t\t\t pmd_t pmd)\n+{\n+\treturn userfaultfd_rwp(vma) \u0026\u0026 pmd_uffd(pmd);\n }\n \n static inline bool userfaultfd_armed(struct vm_area_struct *vma)\n {\n-\treturn vma-\u003evm_flags \u0026 __VM_UFFD_FLAGS;\n+\treturn vma_test_any_mask(vma, __VMA_UFFD_FLAGS);\n }\n \n static inline bool vma_has_uffd_without_event_remap(struct vm_area_struct *vma)\n@@ -253,6 +281,7 @@ extern void userfaultfd_unmap_complete(struct mm_struct *mm,\n \t\t\t\t struct list_head *uf);\n extern bool userfaultfd_wp_unpopulated(struct vm_area_struct *vma);\n extern bool userfaultfd_wp_async(struct vm_area_struct *vma);\n+extern bool userfaultfd_rwp_async(struct vm_area_struct *vma);\n \n static inline bool userfaultfd_wp_use_markers(struct vm_area_struct *vma)\n {\n@@ -272,10 +301,10 @@ static inline bool userfaultfd_wp_use_markers(struct vm_area_struct *vma)\n }\n \n /*\n- * Returns true if this is a swap pte and was uffd-wp wr-protected in either\n- * forms (pte marker or a normal swap pte), false otherwise.\n+ * Returns true if this swap pte carries uffd-tracked state in either\n+ * form (pte marker or a normal swap pte), false otherwise.\n */\n-static inline bool pte_swp_uffd_wp_any(pte_t pte)\n+static inline bool pte_swp_uffd_any(pte_t pte)\n {\n \tif (!uffd_supports_wp_marker())\n \t\treturn false;\n@@ -283,7 +312,7 @@ static inline bool pte_swp_uffd_wp_any(pte_t pte)\n \tif (pte_present(pte))\n \t\treturn false;\n \n-\tif (pte_swp_uffd_wp(pte))\n+\tif (pte_swp_uffd(pte))\n \t\treturn true;\n \n \tif (pte_is_uffd_wp_marker(pte))\n@@ -328,6 +357,16 @@ static inline bool userfaultfd_minor(struct vm_area_struct *vma)\n \treturn false;\n }\n \n+static inline bool userfaultfd_rwp(struct vm_area_struct *vma)\n+{\n+\treturn false;\n+}\n+\n+static inline bool userfaultfd_protected(struct vm_area_struct *vma)\n+{\n+\treturn false;\n+}\n+\n static inline bool userfaultfd_pte_wp(struct vm_area_struct *vma,\n \t\t\t\t pte_t pte)\n {\n@@ -340,6 +379,17 @@ static inline bool userfaultfd_huge_pmd_wp(struct vm_area_struct *vma,\n \treturn false;\n }\n \n+static inline bool userfaultfd_pte_rwp(struct vm_area_struct *vma,\n+\t\t\t\t pte_t pte)\n+{\n+\treturn false;\n+}\n+\n+static inline bool userfaultfd_huge_pmd_rwp(struct vm_area_struct *vma,\n+\t\t\t\t\t pmd_t pmd)\n+{\n+\treturn false;\n+}\n \n static inline bool userfaultfd_armed(struct vm_area_struct *vma)\n {\n@@ -410,6 +460,11 @@ static inline bool userfaultfd_wp_async(struct vm_area_struct *vma)\n \treturn false;\n }\n \n+static inline bool userfaultfd_rwp_async(struct vm_area_struct *vma)\n+{\n+\treturn false;\n+}\n+\n static inline bool vma_has_uffd_without_event_remap(struct vm_area_struct *vma)\n {\n \treturn false;\n@@ -421,10 +476,10 @@ static inline bool userfaultfd_wp_use_markers(struct vm_area_struct *vma)\n }\n \n /*\n- * Returns true if this is a swap pte and was uffd-wp wr-protected in either\n- * forms (pte marker or a normal swap pte), false otherwise.\n+ * Returns true if this swap pte carries uffd-tracked state in either\n+ * form (pte marker or a normal swap pte), false otherwise.\n */\n-static inline bool pte_swp_uffd_wp_any(pte_t pte)\n+static inline bool pte_swp_uffd_any(pte_t pte)\n {\n \treturn false;\n }\ndiff --git a/include/trace/events/huge_memory.h b/include/trace/events/huge_memory.h\nindex 291fae364c62d2..5a48c5406cce49 100644\n--- a/include/trace/events/huge_memory.h\n+++ b/include/trace/events/huge_memory.h\n@@ -16,7 +16,7 @@\n \tEM( SCAN_EXCEED_SWAP_PTE,\t\"exceed_swap_pte\")\t\t\\\n \tEM( SCAN_EXCEED_SHARED_PTE,\t\"exceed_shared_pte\")\t\t\\\n \tEM( SCAN_PTE_NON_PRESENT,\t\"pte_non_present\")\t\t\\\n-\tEM( SCAN_PTE_UFFD_WP,\t\t\"pte_uffd_wp\")\t\t\t\\\n+\tEM( SCAN_PTE_UFFD,\t\t\"pte_uffd_wp\")\t\t\t\\\n \tEM( SCAN_PTE_MAPPED_HUGEPAGE,\t\"pte_mapped_hugepage\")\t\t\\\n \tEM( SCAN_LACK_REFERENCED_PAGE,\t\"lack_referenced_page\")\t\t\\\n \tEM( SCAN_PAGE_NULL,\t\t\"page_null\")\t\t\t\\\ndiff --git a/include/trace/events/mmflags.h b/include/trace/events/mmflags.h\nindex a6e5a44c9b4299..bfface3d0203fa 100644\n--- a/include/trace/events/mmflags.h\n+++ b/include/trace/events/mmflags.h\n@@ -194,6 +194,12 @@ IF_HAVE_PG_ARCH_3(arch_3)\n # define IF_HAVE_UFFD_MINOR(flag, name)\n #endif\n \n+#ifdef CONFIG_USERFAULTFD_RWP\n+# define IF_HAVE_UFFD_RWP(flag, name) {flag, name},\n+#else\n+# define IF_HAVE_UFFD_RWP(flag, name)\n+#endif\n+\n #if defined(CONFIG_64BIT) || defined(CONFIG_PPC32)\n # define IF_HAVE_VM_DROPPABLE(flag, name) {flag, name},\n #else\n@@ -215,6 +221,7 @@ IF_HAVE_UFFD_MINOR(VM_UFFD_MINOR,\t\"uffd_minor\"\t)\t\t\\\n \t{VM_PFNMAP,\t\t\t\"pfnmap\"\t},\t\t\\\n \t{VM_MAYBE_GUARD,\t\t\"maybe_guard\"\t},\t\t\\\n \t{VM_UFFD_WP,\t\t\t\"uffd_wp\"\t},\t\t\\\n+IF_HAVE_UFFD_RWP(VM_UFFD_RWP,\t\t\"uffd_rwp\"\t)\t\t\\\n \t{VM_LOCKED,\t\t\t\"locked\"\t},\t\t\\\n \t{VM_IO,\t\t\t\t\"io\"\t\t},\t\t\\\n \t{VM_SEQ_READ,\t\t\t\"seqread\"\t},\t\t\\\ndiff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h\nindex bd87262f2e3494..34c6f219462a59 100644\n--- a/include/uapi/linux/fs.h\n+++ b/include/uapi/linux/fs.h\n@@ -471,6 +471,7 @@ typedef int __bitwise __kernel_rwf_t;\n #define PAGE_IS_HUGE\t\t(1 \u003c\u003c 6)\n #define PAGE_IS_SOFT_DIRTY\t(1 \u003c\u003c 7)\n #define PAGE_IS_GUARD\t\t(1 \u003c\u003c 8)\n+#define PAGE_IS_ACCESSED\t(1 \u003c\u003c 9)\n \n /*\n * struct page_region - Page region with flags\ndiff --git a/include/uapi/linux/userfaultfd.h b/include/uapi/linux/userfaultfd.h\nindex 2841e4ea8f2ce3..cea11aad6b5445 100644\n--- a/include/uapi/linux/userfaultfd.h\n+++ b/include/uapi/linux/userfaultfd.h\n@@ -25,7 +25,8 @@\n #define UFFD_API ((__u64)0xAA)\n #define UFFD_API_REGISTER_MODES (UFFDIO_REGISTER_MODE_MISSING |\t\\\n \t\t\t\t UFFDIO_REGISTER_MODE_WP |\t\\\n-\t\t\t\t UFFDIO_REGISTER_MODE_MINOR)\n+\t\t\t\t UFFDIO_REGISTER_MODE_MINOR |\t\\\n+\t\t\t\t UFFDIO_REGISTER_MODE_RWP)\n #define UFFD_API_FEATURES (UFFD_FEATURE_PAGEFAULT_FLAG_WP |\t\\\n \t\t\t UFFD_FEATURE_EVENT_FORK |\t\t\\\n \t\t\t UFFD_FEATURE_EVENT_REMAP |\t\t\\\n@@ -42,10 +43,13 @@\n \t\t\t UFFD_FEATURE_WP_UNPOPULATED |\t\\\n \t\t\t UFFD_FEATURE_POISON |\t\t\\\n \t\t\t UFFD_FEATURE_WP_ASYNC |\t\t\\\n-\t\t\t UFFD_FEATURE_MOVE)\n+\t\t\t UFFD_FEATURE_MOVE |\t\t\t\\\n+\t\t\t UFFD_FEATURE_RWP |\t\t\t\\\n+\t\t\t UFFD_FEATURE_RWP_ASYNC)\n #define UFFD_API_IOCTLS\t\t\t\t\\\n \t((__u64)1 \u003c\u003c _UFFDIO_REGISTER |\t\t\\\n \t (__u64)1 \u003c\u003c _UFFDIO_UNREGISTER |\t\\\n+\t (__u64)1 \u003c\u003c _UFFDIO_SET_MODE |\t\t\\\n \t (__u64)1 \u003c\u003c _UFFDIO_API)\n #define UFFD_API_RANGE_IOCTLS\t\t\t\\\n \t((__u64)1 \u003c\u003c _UFFDIO_WAKE |\t\t\\\n@@ -54,13 +58,15 @@\n \t (__u64)1 \u003c\u003c _UFFDIO_MOVE |\t\t\\\n \t (__u64)1 \u003c\u003c _UFFDIO_WRITEPROTECT |\t\\\n \t (__u64)1 \u003c\u003c _UFFDIO_CONTINUE |\t\t\\\n-\t (__u64)1 \u003c\u003c _UFFDIO_POISON)\n+\t (__u64)1 \u003c\u003c _UFFDIO_POISON |\t\t\\\n+\t (__u64)1 \u003c\u003c _UFFDIO_RWPROTECT)\n #define UFFD_API_RANGE_IOCTLS_BASIC\t\t\\\n \t((__u64)1 \u003c\u003c _UFFDIO_WAKE |\t\t\\\n \t (__u64)1 \u003c\u003c _UFFDIO_COPY |\t\t\\\n \t (__u64)1 \u003c\u003c _UFFDIO_WRITEPROTECT |\t\\\n \t (__u64)1 \u003c\u003c _UFFDIO_CONTINUE |\t\t\\\n-\t (__u64)1 \u003c\u003c _UFFDIO_POISON)\n+\t (__u64)1 \u003c\u003c _UFFDIO_POISON |\t\t\\\n+\t (__u64)1 \u003c\u003c _UFFDIO_RWPROTECT)\n \n /*\n * Valid ioctl command number range with this API is from 0x00 to\n@@ -79,6 +85,8 @@\n #define _UFFDIO_WRITEPROTECT\t\t(0x06)\n #define _UFFDIO_CONTINUE\t\t(0x07)\n #define _UFFDIO_POISON\t\t\t(0x08)\n+#define _UFFDIO_RWPROTECT\t\t(0x09)\n+#define _UFFDIO_SET_MODE\t\t(0x0A)\n #define _UFFDIO_API\t\t\t(0x3F)\n \n /* userfaultfd ioctl ids */\n@@ -103,6 +111,10 @@\n \t\t\t\t struct uffdio_continue)\n #define UFFDIO_POISON\t\t_IOWR(UFFDIO, _UFFDIO_POISON, \\\n \t\t\t\t struct uffdio_poison)\n+#define UFFDIO_RWPROTECT\t_IOWR(UFFDIO, _UFFDIO_RWPROTECT,\t\\\n+\t\t\t\t struct uffdio_rwprotect)\n+#define UFFDIO_SET_MODE\t\t_IOW(UFFDIO, _UFFDIO_SET_MODE,\t\\\n+\t\t\t\t struct uffdio_set_mode)\n \n /* read() structure */\n struct uffd_msg {\n@@ -158,6 +170,7 @@ struct uffd_msg {\n #define UFFD_PAGEFAULT_FLAG_WRITE\t(1\u003c\u003c0)\t/* If this was a write fault */\n #define UFFD_PAGEFAULT_FLAG_WP\t\t(1\u003c\u003c1)\t/* If reason is VM_UFFD_WP */\n #define UFFD_PAGEFAULT_FLAG_MINOR\t(1\u003c\u003c2)\t/* If reason is VM_UFFD_MINOR */\n+#define UFFD_PAGEFAULT_FLAG_RWP\t\t(1\u003c\u003c3)\t/* If reason is VM_UFFD_RWP */\n \n struct uffdio_api {\n \t/* userland asks for an API number and the features to enable */\n@@ -230,6 +243,18 @@ struct uffdio_api {\n \t *\n \t * UFFD_FEATURE_MOVE indicates that the kernel supports moving an\n \t * existing page contents from userspace.\n+\t *\n+\t * UFFD_FEATURE_RWP indicates that the kernel supports\n+\t * UFFDIO_REGISTER_MODE_RWP for read-write protection tracking.\n+\t * Pages are made inaccessible via UFFDIO_RWPROTECT and faults\n+\t * are delivered when the pages are re-accessed.\n+\t *\n+\t * UFFD_FEATURE_RWP_ASYNC indicates asynchronous mode for\n+\t * UFFDIO_REGISTER_MODE_RWP. When set, faults on read-write\n+\t * protected pages are auto-resolved by the kernel (PTE\n+\t * permissions restored immediately) without delivering a message\n+\t * to the userfaultfd handler. Use PAGEMAP_SCAN with inverted\n+\t * PAGE_IS_ACCESSED to find pages that were not re-accessed.\n \t */\n #define UFFD_FEATURE_PAGEFAULT_FLAG_WP\t\t(1\u003c\u003c0)\n #define UFFD_FEATURE_EVENT_FORK\t\t\t(1\u003c\u003c1)\n@@ -248,6 +273,8 @@ struct uffdio_api {\n #define UFFD_FEATURE_POISON\t\t\t(1\u003c\u003c14)\n #define UFFD_FEATURE_WP_ASYNC\t\t\t(1\u003c\u003c15)\n #define UFFD_FEATURE_MOVE\t\t\t(1\u003c\u003c16)\n+#define UFFD_FEATURE_RWP\t\t\t(1\u003c\u003c17)\n+#define UFFD_FEATURE_RWP_ASYNC\t\t\t(1\u003c\u003c18)\n \t__u64 features;\n \n \t__u64 ioctls;\n@@ -263,6 +290,7 @@ struct uffdio_register {\n #define UFFDIO_REGISTER_MODE_MISSING\t((__u64)1\u003c\u003c0)\n #define UFFDIO_REGISTER_MODE_WP\t\t((__u64)1\u003c\u003c1)\n #define UFFDIO_REGISTER_MODE_MINOR\t((__u64)1\u003c\u003c2)\n+#define UFFDIO_REGISTER_MODE_RWP\t((__u64)1\u003c\u003c3)\n \t__u64 mode;\n \n \t/*\n@@ -356,6 +384,14 @@ struct uffdio_poison {\n \t__s64 updated;\n };\n \n+struct uffdio_rwprotect {\n+\tstruct uffdio_range range;\n+\t/* !RWP means undo RWP-protection */\n+#define UFFDIO_RWPROTECT_MODE_RWP\t\t((__u64)1\u003c\u003c0)\n+#define UFFDIO_RWPROTECT_MODE_DONTWAKE\t\t((__u64)1\u003c\u003c1)\n+\t__u64 mode;\n+};\n+\n struct uffdio_move {\n \t__u64 dst;\n \t__u64 src;\n@@ -374,6 +410,16 @@ struct uffdio_move {\n \t__s64 move;\n };\n \n+struct uffdio_set_mode {\n+\t/*\n+\t * Toggle async mode for features at runtime.\n+\t * Supported: UFFD_FEATURE_RWP_ASYNC.\n+\t * Setting a bit in both enable and disable is invalid.\n+\t */\n+\t__u64 enable;\n+\t__u64 disable;\n+};\n+\n /*\n * Flags for the userfaultfd(2) system call itself.\n */\ndiff --git a/init/Kconfig b/init/Kconfig\nindex 5230d4879b1c84..9d91074c5d0a30 100644\n--- a/init/Kconfig\n+++ b/init/Kconfig\n@@ -944,6 +944,13 @@ config SCHED_PROXY_EXEC\n \n endmenu\n \n+#\n+# For architectures that support present-but-inaccessible (PROT_NONE) page\n+# table entries detectable via pte_protnone() / pmd_protnone():\n+#\n+config ARCH_HAS_PTE_PROTNONE\n+\tbool\n+\n #\n # For architectures that want to enable the support for NUMA-affine scheduler\n # balancing logic:\n@@ -1010,6 +1017,7 @@ config ARCH_WANT_NUMA_VARIABLE_LOCALITY\n config NUMA_BALANCING\n \tbool \"Memory placement aware NUMA scheduler\"\n \tdepends on ARCH_SUPPORTS_NUMA_BALANCING\n+\tdepends on ARCH_HAS_PTE_PROTNONE\n \tdepends on !ARCH_WANT_NUMA_VARIABLE_LOCALITY\n \tdepends on SMP \u0026\u0026 NUMA_MIGRATION \u0026\u0026 !PREEMPT_RT\n \thelp\ndiff --git a/mm/Kconfig b/mm/Kconfig\nindex 9e0ca48249054f..ccb71653515515 100644\n--- a/mm/Kconfig\n+++ b/mm/Kconfig\n@@ -1386,6 +1386,15 @@ config HAVE_ARCH_USERFAULTFD_MINOR\n \thelp\n \t Arch has userfaultfd minor fault support\n \n+config USERFAULTFD_RWP\n+\tdef_bool y\n+\tdepends on 64BIT \u0026\u0026 ARCH_HAS_PTE_PROTNONE \u0026\u0026 HAVE_ARCH_USERFAULTFD_WP\n+\thelp\n+\t Userfaultfd read-write protection (UFFDIO_RWPROTECT) delivers a\n+\t userfaultfd notification on every access -- read or write -- to a\n+\t protected range, letting userspace observe the working set of a\n+\t process.\n+\n menuconfig USERFAULTFD\n \tbool \"Enable userfaultfd() system call\"\n \tdepends on MMU\ndiff --git a/mm/debug_vm_pgtable.c b/mm/debug_vm_pgtable.c\nindex 23dc3ee0956192..5e9f3a35f924d9 100644\n--- a/mm/debug_vm_pgtable.c\n+++ b/mm/debug_vm_pgtable.c\n@@ -672,7 +672,7 @@ static void __init pte_protnone_tests(struct pgtable_debug_args *args)\n {\n \tpte_t pte = pfn_pte(args-\u003efixed_pte_pfn, args-\u003epage_prot_none);\n \n-\tif (!IS_ENABLED(CONFIG_NUMA_BALANCING))\n+\tif (!IS_ENABLED(CONFIG_ARCH_HAS_PTE_PROTNONE))\n \t\treturn;\n \n \tpr_debug(\"Validating PTE protnone\\n\");\n@@ -685,7 +685,7 @@ static void __init pmd_protnone_tests(struct pgtable_debug_args *args)\n {\n \tpmd_t pmd;\n \n-\tif (!IS_ENABLED(CONFIG_NUMA_BALANCING))\n+\tif (!IS_ENABLED(CONFIG_ARCH_HAS_PTE_PROTNONE))\n \t\treturn;\n \n \tif (!has_transparent_hugepage())\ndiff --git a/mm/huge_memory.c b/mm/huge_memory.c\nindex 2bccb0a53a0a60..400a35ee536fb0 100644\n--- a/mm/huge_memory.c\n+++ b/mm/huge_memory.c\n@@ -1822,8 +1822,8 @@ static void copy_huge_non_present_pmd(\n \t\tpmd = swp_entry_to_pmd(entry);\n \t\tif (pmd_swp_soft_dirty(*src_pmd))\n \t\t\tpmd = pmd_swp_mksoft_dirty(pmd);\n-\t\tif (pmd_swp_uffd_wp(*src_pmd))\n-\t\t\tpmd = pmd_swp_mkuffd_wp(pmd);\n+\t\tif (pmd_swp_uffd(*src_pmd))\n+\t\t\tpmd = pmd_swp_mkuffd(pmd);\n \t\tset_pmd_at(src_mm, addr, src_pmd, pmd);\n \t} else if (softleaf_is_device_private(entry)) {\n \t\t/*\n@@ -1836,8 +1836,8 @@ static void copy_huge_non_present_pmd(\n \n \t\t\tif (pmd_swp_soft_dirty(*src_pmd))\n \t\t\t\tpmd = pmd_swp_mksoft_dirty(pmd);\n-\t\t\tif (pmd_swp_uffd_wp(*src_pmd))\n-\t\t\t\tpmd = pmd_swp_mkuffd_wp(pmd);\n+\t\t\tif (pmd_swp_uffd(*src_pmd))\n+\t\t\t\tpmd = pmd_swp_mkuffd(pmd);\n \t\t\tset_pmd_at(src_mm, addr, src_pmd, pmd);\n \t\t}\n \n@@ -1856,8 +1856,8 @@ static void copy_huge_non_present_pmd(\n \tadd_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR);\n \tmm_inc_nr_ptes(dst_mm);\n \tpgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable);\n-\tif (!userfaultfd_wp(dst_vma))\n-\t\tpmd = pmd_swp_clear_uffd_wp(pmd);\n+\tif (!userfaultfd_protected(dst_vma))\n+\t\tpmd = pmd_swp_clear_uffd(pmd);\n \tset_pmd_at(dst_mm, addr, dst_pmd, pmd);\n }\n \n@@ -1951,9 +1951,15 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,\n out_zero_page:\n \tmm_inc_nr_ptes(dst_mm);\n \tpgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable);\n+\n+\t/* See __copy_present_ptes(): restore accessible protection. */\n+\tif (!userfaultfd_protected(dst_vma)) {\n+\t\tif (userfaultfd_rwp(src_vma) \u0026\u0026 pmd_uffd(pmd))\n+\t\t\tpmd = pmd_modify(pmd, dst_vma-\u003evm_page_prot);\n+\t\tpmd = pmd_clear_uffd(pmd);\n+\t}\n+\n \tpmdp_set_wrprotect(src_mm, addr, src_pmd);\n-\tif (!userfaultfd_wp(dst_vma))\n-\t\tpmd = pmd_clear_uffd_wp(pmd);\n \tpmd = pmd_wrprotect(pmd);\n set_pmd:\n \tpmd = pmd_mkold(pmd);\n@@ -2196,6 +2202,34 @@ static inline bool can_change_pmd_writable(struct vm_area_struct *vma,\n \treturn pmd_dirty(pmd);\n }\n \n+vm_fault_t do_huge_pmd_uffd_rwp(struct vm_fault *vmf)\n+{\n+\tstruct vm_area_struct *vma = vmf-\u003evma;\n+\tpmd_t pmd;\n+\n+\tif (!userfaultfd_rwp_async(vma))\n+\t\treturn handle_userfault(vmf, VM_UFFD_RWP);\n+\n+\tvmf-\u003eptl = pmd_lock(vma-\u003evm_mm, vmf-\u003epmd);\n+\tif (unlikely(!pmd_same(pmdp_get(vmf-\u003epmd), vmf-\u003eorig_pmd))) {\n+\t\tspin_unlock(vmf-\u003eptl);\n+\t\treturn 0;\n+\t}\n+\tpmd = pmd_modify(vmf-\u003eorig_pmd, vma-\u003evm_page_prot);\n+\t/* pmd_modify() preserves _PAGE_UFFD; drop it on resolution */\n+\tpmd = pmd_clear_uffd(pmd);\n+\tpmd = pmd_mkyoung(pmd);\n+\tif (!pmd_write(pmd) \u0026\u0026\n+\t vma_wants_manual_pte_write_upgrade(vma) \u0026\u0026\n+\t can_change_pmd_writable(vma, vmf-\u003eaddress, pmd))\n+\t\tpmd = pmd_mkwrite(pmd, vma);\n+\tset_pmd_at(vma-\u003evm_mm, vmf-\u003eaddress \u0026 HPAGE_PMD_MASK,\n+\t\t vmf-\u003epmd, pmd);\n+\tupdate_mmu_cache_pmd(vma, vmf-\u003eaddress, vmf-\u003epmd);\n+\tspin_unlock(vmf-\u003eptl);\n+\treturn 0;\n+}\n+\n /* NUMA hinting page fault entry point for trans huge pmds */\n vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)\n {\n@@ -2494,9 +2528,9 @@ static pmd_t clear_uffd_wp_pmd(pmd_t pmd)\n \tif (pmd_none(pmd))\n \t\treturn pmd;\n \tif (pmd_present(pmd))\n-\t\tpmd = pmd_clear_uffd_wp(pmd);\n+\t\tpmd = pmd_clear_uffd(pmd);\n \telse\n-\t\tpmd = pmd_swp_clear_uffd_wp(pmd);\n+\t\tpmd = pmd_swp_clear_uffd(pmd);\n \n \treturn pmd;\n }\n@@ -2539,8 +2573,16 @@ bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr,\n \t\t\tpgtable_trans_huge_deposit(mm, new_pmd, pgtable);\n \t\t}\n \t\tpmd = move_soft_dirty_pmd(pmd);\n-\t\tif (vma_has_uffd_without_event_remap(vma))\n+\t\tif (vma_has_uffd_without_event_remap(vma)) {\n+\t\t\t/*\n+\t\t\t * See __copy_present_ptes(): normalise RWP PMDs so\n+\t\t\t * the destination starts accessible instead of taking\n+\t\t\t * a numa-hinting fault on first access.\n+\t\t\t */\n+\t\t\tif (pmd_present(pmd) \u0026\u0026 userfaultfd_rwp(vma))\n+\t\t\t\tpmd = pmd_modify(pmd, vma-\u003evm_page_prot);\n \t\t\tpmd = clear_uffd_wp_pmd(pmd);\n+\t\t}\n \t\tset_pmd_at(mm, new_addr, new_pmd, pmd);\n \t\tif (force_flush)\n \t\t\tflush_pmd_tlb_range(vma, old_addr, old_addr + PMD_SIZE);\n@@ -2553,8 +2595,8 @@ bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr,\n }\n \n static void change_non_present_huge_pmd(struct mm_struct *mm,\n-\t\tunsigned long addr, pmd_t *pmd, bool uffd_wp,\n-\t\tbool uffd_wp_resolve)\n+\t\tunsigned long addr, pmd_t *pmd, bool uffd_prot,\n+\t\tbool uffd_prot_resolve)\n {\n \tsoftleaf_t entry = softleaf_from_pmd(*pmd);\n \tconst struct folio *folio = softleaf_to_folio(entry);\n@@ -2576,16 +2618,16 @@ static void change_non_present_huge_pmd(struct mm_struct *mm,\n \t} else if (softleaf_is_device_private_write(entry)) {\n \t\tentry = make_readable_device_private_entry(swp_offset(entry));\n \t\tnewpmd = swp_entry_to_pmd(entry);\n-\t\tif (pmd_swp_uffd_wp(*pmd))\n-\t\t\tnewpmd = pmd_swp_mkuffd_wp(newpmd);\n+\t\tif (pmd_swp_uffd(*pmd))\n+\t\t\tnewpmd = pmd_swp_mkuffd(newpmd);\n \t} else {\n \t\tnewpmd = *pmd;\n \t}\n \n-\tif (uffd_wp)\n-\t\tnewpmd = pmd_swp_mkuffd_wp(newpmd);\n-\telse if (uffd_wp_resolve)\n-\t\tnewpmd = pmd_swp_clear_uffd_wp(newpmd);\n+\tif (uffd_prot)\n+\t\tnewpmd = pmd_swp_mkuffd(newpmd);\n+\telse if (uffd_prot_resolve)\n+\t\tnewpmd = pmd_swp_clear_uffd(newpmd);\n \tif (!pmd_same(*pmd, newpmd))\n \t\tset_pmd_at(mm, addr, pmd, newpmd);\n }\n@@ -2605,8 +2647,9 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,\n \tspinlock_t *ptl;\n \tpmd_t oldpmd, entry;\n \tbool prot_numa = cp_flags \u0026 MM_CP_PROT_NUMA;\n-\tbool uffd_wp = cp_flags \u0026 MM_CP_UFFD_WP;\n-\tbool uffd_wp_resolve = cp_flags \u0026 MM_CP_UFFD_WP_RESOLVE;\n+\tbool uffd_prot = cp_flags \u0026 (MM_CP_UFFD_WP | MM_CP_UFFD_RWP);\n+\tbool uffd_prot_resolve = cp_flags \u0026\n+\t\t(MM_CP_UFFD_WP_RESOLVE | MM_CP_UFFD_RWP_RESOLVE);\n \tint ret = 1;\n \n \ttlb_change_page_size(tlb, HPAGE_PMD_SIZE);\n@@ -2619,11 +2662,17 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,\n \t\treturn 0;\n \n \tif (thp_migration_supported() \u0026\u0026 pmd_is_valid_softleaf(*pmd)) {\n-\t\tchange_non_present_huge_pmd(mm, addr, pmd, uffd_wp,\n-\t\t\t\t\t uffd_wp_resolve);\n+\t\tchange_non_present_huge_pmd(mm, addr, pmd, uffd_prot,\n+\t\t\t\t\t uffd_prot_resolve);\n \t\tgoto unlock;\n \t}\n \n+\t/* Already in the desired state */\n+\tif (prot_numa \u0026\u0026 pmd_protnone(*pmd))\n+\t\tgoto unlock;\n+\tif ((cp_flags \u0026 MM_CP_UFFD_RWP) \u0026\u0026 pmd_protnone(*pmd) \u0026\u0026 pmd_uffd(*pmd))\n+\t\tgoto unlock;\n+\n \tif (prot_numa) {\n \n \t\t/*\n@@ -2634,9 +2683,6 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,\n \t\tif (is_huge_zero_pmd(*pmd))\n \t\t\tgoto unlock;\n \n-\t\tif (pmd_protnone(*pmd))\n-\t\t\tgoto unlock;\n-\n \t\tif (!folio_can_map_prot_numa(pmd_folio(*pmd), vma,\n \t\t\t\t\t vma_is_single_threaded_private(vma)))\n \t\t\tgoto unlock;\n@@ -2665,15 +2711,19 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,\n \toldpmd = pmdp_invalidate_ad(vma, addr, pmd);\n \n \tentry = pmd_modify(oldpmd, newprot);\n-\tif (uffd_wp)\n-\t\tentry = pmd_mkuffd_wp(entry);\n-\telse if (uffd_wp_resolve)\n+\tif (uffd_prot)\n+\t\tentry = pmd_mkuffd(entry);\n+\telse if (uffd_prot_resolve)\n \t\t/*\n \t\t * Leave the write bit to be handled by PF interrupt\n \t\t * handler, then things like COW could be properly\n \t\t * handled.\n \t\t */\n-\t\tentry = pmd_clear_uffd_wp(entry);\n+\t\tentry = pmd_clear_uffd(entry);\n+\n+\t/* See change_pte_range(): preserve RWP protection across mprotect() */\n+\tif (userfaultfd_rwp(vma) \u0026\u0026 pmd_uffd(entry))\n+\t\tentry = pmd_modify(entry, PAGE_NONE);\n \n \t/* See change_pte_range(). */\n \tif ((cp_flags \u0026 MM_CP_TRY_CHANGE_WRITABLE) \u0026\u0026 !pmd_write(entry) \u0026\u0026\n@@ -2842,6 +2892,13 @@ int move_pages_huge_pmd(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, pm\n \t\t_dst_pmd = move_soft_dirty_pmd(src_pmdval);\n \t\t_dst_pmd = clear_uffd_wp_pmd(_dst_pmd);\n \t}\n+\n+\t/* Re-arm RWP on the moved PMD if dst_vma is RWP-registered. */\n+\tif (userfaultfd_rwp(dst_vma)) {\n+\t\t_dst_pmd = pmd_modify(_dst_pmd, PAGE_NONE);\n+\t\t_dst_pmd = pmd_mkuffd(_dst_pmd);\n+\t}\n+\n \tset_pmd_at(mm, dst_addr, dst_pmd, _dst_pmd);\n \n \tsrc_pgtable = pgtable_trans_huge_withdraw(mm, src_pmd);\n@@ -3016,8 +3073,13 @@ static void __split_huge_zero_page_pmd(struct vm_area_struct *vma,\n \n \t\tentry = pfn_pte(zero_pfn(addr), vma-\u003evm_page_prot);\n \t\tentry = pte_mkspecial(entry);\n-\t\tif (pmd_uffd_wp(old_pmd))\n-\t\t\tentry = pte_mkuffd_wp(entry);\n+\t\tif (pmd_uffd(old_pmd))\n+\t\t\tentry = pte_mkuffd(entry);\n+\n+\t\t/* Restore PAGE_NONE so an RWP marker keeps trapping */\n+\t\tif (userfaultfd_rwp(vma) \u0026\u0026 pmd_uffd(old_pmd))\n+\t\t\tentry = pte_modify(entry, PAGE_NONE);\n+\n \t\tVM_BUG_ON(!pte_none(ptep_get(pte)));\n \t\tset_pte_at(mm, addr, pte, entry);\n \t\tpte++;\n@@ -3103,7 +3165,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,\n \t\tfolio = page_folio(page);\n \n \t\tsoft_dirty = pmd_swp_soft_dirty(old_pmd);\n-\t\tuffd_wp = pmd_swp_uffd_wp(old_pmd);\n+\t\tuffd_wp = pmd_swp_uffd(old_pmd);\n \n \t\twrite = softleaf_is_migration_write(entry);\n \t\tif (PageAnon(page))\n@@ -3119,7 +3181,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,\n \t\tfolio = page_folio(page);\n \n \t\tsoft_dirty = pmd_swp_soft_dirty(old_pmd);\n-\t\tuffd_wp = pmd_swp_uffd_wp(old_pmd);\n+\t\tuffd_wp = pmd_swp_uffd(old_pmd);\n \n \t\twrite = softleaf_is_device_private_write(entry);\n \t\tanon_exclusive = PageAnonExclusive(page);\n@@ -3176,7 +3238,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,\n \t\twrite = pmd_write(old_pmd);\n \t\tyoung = pmd_young(old_pmd);\n \t\tsoft_dirty = pmd_soft_dirty(old_pmd);\n-\t\tuffd_wp = pmd_uffd_wp(old_pmd);\n+\t\tuffd_wp = pmd_uffd(old_pmd);\n \n \t\tVM_WARN_ON_FOLIO(!folio_ref_count(folio), folio);\n \t\tVM_WARN_ON_FOLIO(!folio_test_anon(folio), folio);\n@@ -3247,7 +3309,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,\n \t\t\tif (soft_dirty)\n \t\t\t\tentry = pte_swp_mksoft_dirty(entry);\n \t\t\tif (uffd_wp)\n-\t\t\t\tentry = pte_swp_mkuffd_wp(entry);\n+\t\t\t\tentry = pte_swp_mkuffd(entry);\n \t\t\tVM_WARN_ON(!pte_none(ptep_get(pte + i)));\n \t\t\tset_pte_at(mm, addr, pte + i, entry);\n \t\t}\n@@ -3274,7 +3336,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,\n \t\t\tif (soft_dirty)\n \t\t\t\tentry = pte_swp_mksoft_dirty(entry);\n \t\t\tif (uffd_wp)\n-\t\t\t\tentry = pte_swp_mkuffd_wp(entry);\n+\t\t\t\tentry = pte_swp_mkuffd(entry);\n \t\t\tVM_WARN_ON(!pte_none(ptep_get(pte + i)));\n \t\t\tset_pte_at(mm, addr, pte + i, entry);\n \t\t}\n@@ -3292,7 +3354,11 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,\n \t\tif (soft_dirty)\n \t\t\tentry = pte_mksoft_dirty(entry);\n \t\tif (uffd_wp)\n-\t\t\tentry = pte_mkuffd_wp(entry);\n+\t\t\tentry = pte_mkuffd(entry);\n+\n+\t\t/* Restore PAGE_NONE so an RWP marker keeps trapping */\n+\t\tif (userfaultfd_rwp(vma) \u0026\u0026 uffd_wp)\n+\t\t\tentry = pte_modify(entry, PAGE_NONE);\n \n \t\tfor (i = 0; i \u003c HPAGE_PMD_NR; i++)\n \t\t\tVM_WARN_ON(!pte_none(ptep_get(pte + i)));\n@@ -4887,7 +4953,7 @@ int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw,\n \n \t\twritable = pmd_write(pmdval);\n \t\tsoftdirty = pmd_soft_dirty(pmdval);\n-\t\tuffd_wp = pmd_uffd_wp(pmdval);\n+\t\tuffd_wp = pmd_uffd(pmdval);\n \t} else {\n \t\tsoftleaf_t old_entry;\n \n@@ -4896,7 +4962,7 @@ int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw,\n \n \t\twritable = softleaf_is_device_private_write(old_entry);\n \t\tsoftdirty = pmd_swp_soft_dirty(pmdval);\n-\t\tuffd_wp = pmd_swp_uffd_wp(pmdval);\n+\t\tuffd_wp = pmd_swp_uffd(pmdval);\n \t}\n \n \t/* See folio_try_share_anon_rmap_pmd(): invalidate PMD first. */\n@@ -4927,7 +4993,7 @@ int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw,\n \tif (softdirty)\n \t\tpmdswp = pmd_swp_mksoft_dirty(pmdswp);\n \tif (uffd_wp)\n-\t\tpmdswp = pmd_swp_mkuffd_wp(pmdswp);\n+\t\tpmdswp = pmd_swp_mkuffd(pmdswp);\n \tset_pmd_at(mm, address, pvmw-\u003epmd, pmdswp);\n \n \t/* Migration entry installed: cleanup rmap, folio. */\n@@ -4959,8 +5025,13 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new)\n \t\tpmde = pmd_mksoft_dirty(pmde);\n \tif (softleaf_is_migration_write(entry))\n \t\tpmde = pmd_mkwrite(pmde, vma);\n-\tif (pmd_swp_uffd_wp(*pvmw-\u003epmd))\n-\t\tpmde = pmd_mkuffd_wp(pmde);\n+\tif (pmd_swp_uffd(*pvmw-\u003epmd))\n+\t\tpmde = pmd_mkuffd(pmde);\n+\n+\t/* See do_swap_page(): restore PAGE_NONE for RWP */\n+\tif (pmd_swp_uffd(*pvmw-\u003epmd) \u0026\u0026 userfaultfd_rwp(vma))\n+\t\tpmde = pmd_modify(pmde, PAGE_NONE);\n+\n \tif (!softleaf_is_migration_young(entry))\n \t\tpmde = pmd_mkold(pmde);\n \t/* NOTE: this may contain setting soft-dirty on some archs */\n@@ -4980,8 +5051,8 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new)\n \n \t\tif (pmd_swp_soft_dirty(*pvmw-\u003epmd))\n \t\t\tpmde = pmd_swp_mksoft_dirty(pmde);\n-\t\tif (pmd_swp_uffd_wp(*pvmw-\u003epmd))\n-\t\t\tpmde = pmd_swp_mkuffd_wp(pmde);\n+\t\tif (pmd_swp_uffd(*pvmw-\u003epmd))\n+\t\t\tpmde = pmd_swp_mkuffd(pmde);\n \t}\n \n \tif (folio_test_anon(folio)) {\ndiff --git a/mm/hugetlb.c b/mm/hugetlb.c\nindex 571212b80835ec..9da52d95b3fb76 100644\n--- a/mm/hugetlb.c\n+++ b/mm/hugetlb.c\n@@ -4843,8 +4843,16 @@ hugetlb_install_folio(struct vm_area_struct *vma, pte_t *ptep, unsigned long add\n \n \t__folio_mark_uptodate(new_folio);\n \thugetlb_add_new_anon_rmap(new_folio, vma, addr);\n-\tif (userfaultfd_wp(vma) \u0026\u0026 huge_pte_uffd_wp(old))\n-\t\tnewpte = huge_pte_mkuffd_wp(newpte);\n+\tif (userfaultfd_protected(vma) \u0026\u0026 huge_pte_uffd(old)) {\n+\t\tnewpte = huge_pte_mkuffd(newpte);\n+\t\t/* Restore PAGE_NONE so the RWP marker keeps trapping. */\n+\t\tif (userfaultfd_rwp(vma)) {\n+\t\t\tunsigned int shift = huge_page_shift(hstate_vma(vma));\n+\n+\t\t\tnewpte = huge_pte_modify(newpte, PAGE_NONE);\n+\t\t\tnewpte = arch_make_huge_pte(newpte, shift, vma-\u003evm_flags);\n+\t\t}\n+\t}\n \tset_huge_pte_at(vma-\u003evm_mm, addr, ptep, newpte, sz);\n \thugetlb_count_add(pages_per_huge_page(hstate_vma(vma)), vma-\u003evm_mm);\n \tfolio_set_hugetlb_migratable(new_folio);\n@@ -4917,11 +4925,11 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,\n \n \t\tsoftleaf = softleaf_from_pte(entry);\n \t\tif (unlikely(softleaf_is_hwpoison(softleaf))) {\n-\t\t\tif (!userfaultfd_wp(dst_vma))\n-\t\t\t\tentry = huge_pte_clear_uffd_wp(entry);\n+\t\t\tif (!userfaultfd_protected(dst_vma))\n+\t\t\t\tentry = huge_pte_clear_uffd(entry);\n \t\t\tset_huge_pte_at(dst, addr, dst_pte, entry, sz);\n \t\t} else if (unlikely(softleaf_is_migration(softleaf))) {\n-\t\t\tbool uffd_wp = pte_swp_uffd_wp(entry);\n+\t\t\tbool uffd = pte_swp_uffd(entry);\n \n \t\t\tif (!softleaf_is_migration_read(softleaf) \u0026\u0026 cow) {\n \t\t\t\t/*\n@@ -4931,12 +4939,12 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,\n \t\t\t\tsoftleaf = make_readable_migration_entry(\n \t\t\t\t\t\t\tswp_offset(softleaf));\n \t\t\t\tentry = swp_entry_to_pte(softleaf);\n-\t\t\t\tif (userfaultfd_wp(src_vma) \u0026\u0026 uffd_wp)\n-\t\t\t\t\tentry = pte_swp_mkuffd_wp(entry);\n+\t\t\t\tif (userfaultfd_protected(src_vma) \u0026\u0026 uffd)\n+\t\t\t\t\tentry = pte_swp_mkuffd(entry);\n \t\t\t\tset_huge_pte_at(src, addr, src_pte, entry, sz);\n \t\t\t}\n-\t\t\tif (!userfaultfd_wp(dst_vma))\n-\t\t\t\tentry = huge_pte_clear_uffd_wp(entry);\n+\t\t\tif (!userfaultfd_protected(dst_vma))\n+\t\t\t\tentry = huge_pte_clear_uffd(entry);\n \t\t\tset_huge_pte_at(dst, addr, dst_pte, entry, sz);\n \t\t} else if (unlikely(pte_is_marker(entry))) {\n \t\t\tconst pte_marker marker = copy_pte_marker(softleaf, dst_vma);\n@@ -5000,6 +5008,16 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,\n \t\t\t\tgoto next;\n \t\t\t}\n \n+\t\t\t/* See __copy_present_ptes(): restore accessible protection. */\n+\t\t\tif (!userfaultfd_protected(dst_vma)) {\n+\t\t\t\tif (userfaultfd_rwp(src_vma) \u0026\u0026 huge_pte_uffd(entry)) {\n+\t\t\t\t\tentry = huge_pte_modify(entry, dst_vma-\u003evm_page_prot);\n+\t\t\t\t\tentry = arch_make_huge_pte(entry, huge_page_shift(h),\n+\t\t\t\t\t\t\t\t dst_vma-\u003evm_flags);\n+\t\t\t\t}\n+\t\t\t\tentry = huge_pte_clear_uffd(entry);\n+\t\t\t}\n+\n \t\t\tif (cow) {\n \t\t\t\t/*\n \t\t\t\t * No need to notify as we are downgrading page\n@@ -5012,9 +5030,6 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,\n \t\t\t\tentry = huge_pte_wrprotect(entry);\n \t\t\t}\n \n-\t\t\tif (!userfaultfd_wp(dst_vma))\n-\t\t\t\tentry = huge_pte_clear_uffd_wp(entry);\n-\n \t\t\tset_huge_pte_at(dst, addr, dst_pte, entry, sz);\n \t\t\thugetlb_count_add(npages, dst);\n \t\t}\n@@ -5060,10 +5075,22 @@ static void move_huge_pte(struct vm_area_struct *vma, unsigned long old_addr,\n \t\thuge_pte_clear(mm, new_addr, dst_pte, sz);\n \t} else {\n \t\tif (need_clear_uffd_wp) {\n-\t\t\tif (pte_present(pte))\n-\t\t\t\tpte = huge_pte_clear_uffd_wp(pte);\n-\t\t\telse\n-\t\t\t\tpte = pte_swp_clear_uffd_wp(pte);\n+\t\t\tif (pte_present(pte)) {\n+\t\t\t\t/*\n+\t\t\t\t * See __copy_present_ptes(): normalise RWP\n+\t\t\t\t * PTEs so the destination starts accessible\n+\t\t\t\t * instead of taking a numa-hinting fault on\n+\t\t\t\t * first access.\n+\t\t\t\t */\n+\t\t\t\tif (userfaultfd_rwp(vma)) {\n+\t\t\t\t\tpte = huge_pte_modify(pte, vma-\u003evm_page_prot);\n+\t\t\t\t\tpte = arch_make_huge_pte(pte, huge_page_shift(h),\n+\t\t\t\t\t\t\t\t vma-\u003evm_flags);\n+\t\t\t\t}\n+\t\t\t\tpte = huge_pte_clear_uffd(pte);\n+\t\t\t} else {\n+\t\t\t\tpte = pte_swp_clear_uffd(pte);\n+\t\t\t}\n \t\t}\n \t\tset_huge_pte_at(mm, new_addr, dst_pte, pte, sz);\n \t}\n@@ -5197,7 +5224,7 @@ void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma,\n \t\t\t * drop the uffd-wp bit in this zap, then replace the\n \t\t\t * pte with a marker.\n \t\t\t */\n-\t\t\tif (pte_swp_uffd_wp_any(pte) \u0026\u0026\n+\t\t\tif (pte_swp_uffd_any(pte) \u0026\u0026\n \t\t\t !(zap_flags \u0026 ZAP_FLAG_DROP_MARKER))\n \t\t\t\tset_huge_pte_at(mm, address, ptep,\n \t\t\t\t\t\tmake_pte_marker(PTE_MARKER_UFFD_WP),\n@@ -5233,7 +5260,7 @@ void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma,\n \t\tif (huge_pte_dirty(pte))\n \t\t\tfolio_mark_dirty(folio);\n \t\t/* Leave a uffd-wp pte marker if needed */\n-\t\tif (huge_pte_uffd_wp(pte) \u0026\u0026\n+\t\tif (huge_pte_uffd(pte) \u0026\u0026\n \t\t !(zap_flags \u0026 ZAP_FLAG_DROP_MARKER))\n \t\t\tset_huge_pte_at(mm, address, ptep,\n \t\t\t\t\tmake_pte_marker(PTE_MARKER_UFFD_WP),\n@@ -5437,7 +5464,7 @@ static vm_fault_t hugetlb_wp(struct vm_fault *vmf)\n \t * can trigger this, because hugetlb_fault() will always resolve\n \t * uffd-wp bit first.\n \t */\n-\tif (!unshare \u0026\u0026 huge_pte_uffd_wp(pte))\n+\tif (!unshare \u0026\u0026 huge_pte_uffd(pte))\n \t\treturn 0;\n \n \t/* Let's take out MAP_SHARED mappings first. */\n@@ -5581,8 +5608,8 @@ static vm_fault_t hugetlb_wp(struct vm_fault *vmf)\n \t\thuge_ptep_clear_flush(vma, vmf-\u003eaddress, vmf-\u003epte);\n \t\thugetlb_remove_rmap(old_folio);\n \t\thugetlb_add_new_anon_rmap(new_folio, vma, vmf-\u003eaddress);\n-\t\tif (huge_pte_uffd_wp(pte))\n-\t\t\tnewpte = huge_pte_mkuffd_wp(newpte);\n+\t\tif (huge_pte_uffd(pte))\n+\t\t\tnewpte = huge_pte_mkuffd(newpte);\n \t\tset_huge_pte_at(mm, vmf-\u003eaddress, vmf-\u003epte, newpte,\n \t\t\t\thuge_page_size(h));\n \t\tfolio_set_hugetlb_migratable(new_folio);\n@@ -5860,7 +5887,7 @@ static vm_fault_t hugetlb_no_page(struct address_space *mapping,\n \t * if populated.\n \t */\n \tif (unlikely(pte_is_uffd_wp_marker(vmf-\u003eorig_pte)))\n-\t\tnew_pte = huge_pte_mkuffd_wp(new_pte);\n+\t\tnew_pte = huge_pte_mkuffd(new_pte);\n \tset_huge_pte_at(mm, vmf-\u003eaddress, vmf-\u003epte, new_pte, huge_page_size(h));\n \n \thugetlb_count_add(pages_per_huge_page(h), mm);\n@@ -6035,6 +6062,47 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,\n \t\tgoto out_mutex;\n \t}\n \n+\t/*\n+\t * Protnone hugetlb PTEs with the uffd bit are used by\n+\t * userfaultfd RWP for access tracking. Plain PROT_NONE (without the\n+\t * marker) is not an RWP fault and is not expected on hugetlb (no\n+\t * NUMA hinting), so let normal hugetlb fault handling proceed.\n+\t */\n+\tif (pte_protnone(vmf.orig_pte) \u0026\u0026 vma_is_accessible(vma) \u0026\u0026\n+\t userfaultfd_rwp(vma) \u0026\u0026 huge_pte_uffd(vmf.orig_pte)) {\n+\t\tspinlock_t *ptl;\n+\t\tpte_t pte;\n+\n+\t\t/* Sync: drop hugetlb locks before blocking in handle_userfault() */\n+\t\tif (!userfaultfd_rwp_async(vma))\n+\t\t\treturn hugetlb_handle_userfault(\u0026vmf, mapping, VM_UFFD_RWP);\n+\n+\t\tptl = huge_pte_lock(h, mm, vmf.pte);\n+\t\tpte = huge_ptep_get(mm, vmf.address, vmf.pte);\n+\t\tif (pte_protnone(pte) \u0026\u0026 huge_pte_uffd(pte)) {\n+\t\t\tunsigned int shift = huge_page_shift(h);\n+\n+\t\t\tpte = huge_pte_modify(pte, vma-\u003evm_page_prot);\n+\t\t\tpte = arch_make_huge_pte(pte, shift, vma-\u003evm_flags);\n+\t\t\t/* huge_pte_modify() preserves _PAGE_UFFD; drop it on resolution */\n+\t\t\tpte = huge_pte_clear_uffd(pte);\n+\t\t\tpte = pte_mkyoung(pte);\n+\t\t\t/*\n+\t\t\t * Unlike do_uffd_rwp(), do not upgrade to writable\n+\t\t\t * here. Hugetlb lacks a can_change_huge_pte_writable()\n+\t\t\t * equivalent, so a write access will take a separate\n+\t\t\t * COW fault — acceptable for the rare private hugetlb\n+\t\t\t * case.\n+\t\t\t */\n+\t\t\tset_huge_pte_at(mm, vmf.address, vmf.pte, pte,\n+\t\t\t\t\thuge_page_size(h));\n+\t\t\tupdate_mmu_cache(vma, vmf.address, vmf.pte);\n+\t\t}\n+\t\tspin_unlock(ptl);\n+\t\tret = 0;\n+\t\tgoto out_mutex;\n+\t}\n+\n \t/*\n \t * If we are going to COW/unshare the mapping later, we examine the\n \t * pending reservations for this page now. This will ensure that any\n@@ -6058,7 +6126,7 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,\n \t\tgoto out_ptl;\n \n \t/* Handle userfault-wp first, before trying to lock more pages */\n-\tif (userfaultfd_wp(vma) \u0026\u0026 huge_pte_uffd_wp(huge_ptep_get(mm, vmf.address, vmf.pte)) \u0026\u0026\n+\tif (userfaultfd_wp(vma) \u0026\u0026 huge_pte_uffd(huge_ptep_get(mm, vmf.address, vmf.pte)) \u0026\u0026\n \t (flags \u0026 FAULT_FLAG_WRITE) \u0026\u0026 !huge_pte_write(vmf.orig_pte)) {\n \t\tif (!userfaultfd_wp_async(vma)) {\n \t\t\tspin_unlock(vmf.ptl);\n@@ -6067,7 +6135,7 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,\n \t\t\treturn handle_userfault(\u0026vmf, VM_UFFD_WP);\n \t\t}\n \n-\t\tvmf.orig_pte = huge_pte_clear_uffd_wp(vmf.orig_pte);\n+\t\tvmf.orig_pte = huge_pte_clear_uffd(vmf.orig_pte);\n \t\tset_huge_pte_at(mm, vmf.address, vmf.pte, vmf.orig_pte,\n \t\t\t\thuge_page_size(hstate_vma(vma)));\n \t\t/* Fallthrough to CoW */\n@@ -6352,7 +6420,7 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte,\n \t_dst_pte = pte_mkyoung(_dst_pte);\n \n \tif (wp_enabled)\n-\t\t_dst_pte = huge_pte_mkuffd_wp(_dst_pte);\n+\t\t_dst_pte = huge_pte_mkuffd(_dst_pte);\n \n \tset_huge_pte_at(dst_mm, dst_addr, dst_pte, _dst_pte, size);\n \n@@ -6395,6 +6463,8 @@ long hugetlb_change_protection(struct vm_area_struct *vma,\n \tunsigned long last_addr_mask;\n \tbool uffd_wp = cp_flags \u0026 MM_CP_UFFD_WP;\n \tbool uffd_wp_resolve = cp_flags \u0026 MM_CP_UFFD_WP_RESOLVE;\n+\tbool uffd_rwp = cp_flags \u0026 MM_CP_UFFD_RWP;\n+\tbool uffd_rwp_resolve = cp_flags \u0026 MM_CP_UFFD_RWP_RESOLVE;\n \tstruct mmu_gather tlb;\n \n \t/*\n@@ -6420,6 +6490,11 @@ long hugetlb_change_protection(struct vm_area_struct *vma,\n \n \t\tptep = hugetlb_walk(vma, address, psize);\n \t\tif (!ptep) {\n+\t\t\t/*\n+\t\t\t * uffd_wp installs a pte marker on the unpopulated\n+\t\t\t * entry; uffd_rwp does not install markers so the\n+\t\t\t * allocation is unnecessary for it.\n+\t\t\t */\n \t\t\tif (!uffd_wp) {\n \t\t\t\taddress |= last_addr_mask;\n \t\t\t\tcontinue;\n@@ -6441,7 +6516,8 @@ long hugetlb_change_protection(struct vm_area_struct *vma,\n \t\t\t * shouldn't happen at all. Warn about it if it\n \t\t\t * happened due to some reason.\n \t\t\t */\n-\t\t\tWARN_ON_ONCE(uffd_wp || uffd_wp_resolve);\n+\t\t\tWARN_ON_ONCE(uffd_wp || uffd_wp_resolve ||\n+\t\t\t\t uffd_rwp || uffd_rwp_resolve);\n \t\t\tpages++;\n \t\t\tspin_unlock(ptl);\n \t\t\taddress |= last_addr_mask;\n@@ -6475,10 +6551,10 @@ long hugetlb_change_protection(struct vm_area_struct *vma,\n \t\t\t\tpages++;\n \t\t\t}\n \n-\t\t\tif (uffd_wp)\n-\t\t\t\tnewpte = pte_swp_mkuffd_wp(newpte);\n-\t\t\telse if (uffd_wp_resolve)\n-\t\t\t\tnewpte = pte_swp_clear_uffd_wp(newpte);\n+\t\t\tif (uffd_wp || uffd_rwp)\n+\t\t\t\tnewpte = pte_swp_mkuffd(newpte);\n+\t\t\telse if (uffd_wp_resolve || uffd_rwp_resolve)\n+\t\t\t\tnewpte = pte_swp_clear_uffd(newpte);\n \t\t\tif (!pte_same(pte, newpte))\n \t\t\t\tset_huge_pte_at(mm, address, ptep, newpte, psize);\n \t\t} else if (unlikely(pte_is_marker(pte))) {\n@@ -6488,20 +6564,32 @@ long hugetlb_change_protection(struct vm_area_struct *vma,\n \t\t\t * pte_marker_uffd_wp()==true implies !poison\n \t\t\t * because they're mutual exclusive.\n \t\t\t */\n-\t\t\tif (pte_is_uffd_wp_marker(pte) \u0026\u0026 uffd_wp_resolve)\n+\t\t\tif (pte_is_uffd_wp_marker(pte) \u0026\u0026\n+\t\t\t (uffd_wp_resolve || uffd_rwp_resolve))\n \t\t\t\t/* Safe to modify directly (non-present-\u003enone). */\n \t\t\t\thuge_pte_clear(mm, address, ptep, psize);\n \t\t} else {\n \t\t\tpte_t old_pte;\n \t\t\tunsigned int shift = huge_page_shift(hstate_vma(vma));\n \n+\t\t\t/* Already protnone with uffd bit set? Nothing to do. */\n+\t\t\tif (uffd_rwp \u0026\u0026 pte_protnone(pte) \u0026\u0026 huge_pte_uffd(pte))\n+\t\t\t\tgoto next;\n+\n \t\t\told_pte = huge_ptep_modify_prot_start(vma, address, ptep);\n \t\t\tpte = huge_pte_modify(old_pte, newprot);\n \t\t\tpte = arch_make_huge_pte(pte, shift, vma-\u003evm_flags);\n-\t\t\tif (uffd_wp)\n-\t\t\t\tpte = huge_pte_mkuffd_wp(pte);\n-\t\t\telse if (uffd_wp_resolve)\n-\t\t\t\tpte = huge_pte_clear_uffd_wp(pte);\n+\t\t\tif (uffd_wp || uffd_rwp)\n+\t\t\t\tpte = huge_pte_mkuffd(pte);\n+\t\t\telse if (uffd_wp_resolve || uffd_rwp_resolve)\n+\t\t\t\tpte = huge_pte_clear_uffd(pte);\n+\n+\t\t\t/* Preserve RWP protection across mprotect() */\n+\t\t\tif (userfaultfd_rwp(vma) \u0026\u0026 huge_pte_uffd(pte)) {\n+\t\t\t\tpte = huge_pte_modify(pte, PAGE_NONE);\n+\t\t\t\tpte = arch_make_huge_pte(pte, shift, vma-\u003evm_flags);\n+\t\t\t}\n+\n \t\t\thuge_ptep_modify_prot_commit(vma, address, ptep, old_pte, pte);\n \t\t\tpages++;\n \t\t\ttlb_remove_huge_tlb_entry(h, \u0026tlb, ptep, address);\ndiff --git a/mm/internal.h b/mm/internal.h\nindex 181e79f1d6a207..87bf7fbd0e9034 100644\n--- a/mm/internal.h\n+++ b/mm/internal.h\n@@ -412,8 +412,8 @@ static inline pte_t pte_move_swp_offset(pte_t pte, long delta)\n \t\tnew = pte_swp_mksoft_dirty(new);\n \tif (pte_swp_exclusive(pte))\n \t\tnew = pte_swp_mkexclusive(new);\n-\tif (pte_swp_uffd_wp(pte))\n-\t\tnew = pte_swp_mkuffd_wp(new);\n+\tif (pte_swp_uffd(pte))\n+\t\tnew = pte_swp_mkuffd(new);\n \n \treturn new;\n }\ndiff --git a/mm/khugepaged.c b/mm/khugepaged.c\nindex 617bca76db49b6..483bcebd6977d0 100644\n--- a/mm/khugepaged.c\n+++ b/mm/khugepaged.c\n@@ -37,7 +37,7 @@ enum scan_result {\n \tSCAN_EXCEED_SWAP_PTE,\n \tSCAN_EXCEED_SHARED_PTE,\n \tSCAN_PTE_NON_PRESENT,\n-\tSCAN_PTE_UFFD_WP,\n+\tSCAN_PTE_UFFD,\n \tSCAN_PTE_MAPPED_HUGEPAGE,\n \tSCAN_LACK_REFERENCED_PAGE,\n \tSCAN_PAGE_NULL,\n@@ -695,8 +695,8 @@ static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma,\n \t\t\tresult = SCAN_PTE_NON_PRESENT;\n \t\t\tgoto out;\n \t\t}\n-\t\tif (pte_uffd_wp(pteval)) {\n-\t\t\tresult = SCAN_PTE_UFFD_WP;\n+\t\tif (pte_uffd(pteval)) {\n+\t\t\tresult = SCAN_PTE_UFFD;\n \t\t\tgoto out;\n \t\t}\n \t\tpage = vm_normal_page(vma, addr, pteval);\n@@ -1543,7 +1543,7 @@ static enum scan_result mthp_collapse(struct mm_struct *mm,\n \t\t\tcase SCAN_PAGE_NULL:\n \t\t\tcase SCAN_DEL_PAGE_LRU:\n \t\t\tcase SCAN_PTE_NON_PRESENT:\n-\t\t\tcase SCAN_PTE_UFFD_WP:\n+\t\t\tcase SCAN_PTE_UFFD:\n \t\t\tcase SCAN_PAGE_LAZYFREE:\n \t\t\t\tlast_result = ret;\n \t\t\t\tgoto next_order;\n@@ -1664,15 +1664,15 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,\n \t\t\t/*\n \t\t\t * Always be strict with uffd-wp\n \t\t\t * enabled swap entries. Please see\n-\t\t\t * comment below for pte_uffd_wp().\n+\t\t\t * comment below for pte_uffd().\n \t\t\t */\n-\t\t\tif (pte_swp_uffd_wp_any(pteval)) {\n-\t\t\t\tresult = SCAN_PTE_UFFD_WP;\n+\t\t\tif (pte_swp_uffd_any(pteval)) {\n+\t\t\t\tresult = SCAN_PTE_UFFD;\n \t\t\t\tgoto out_unmap;\n \t\t\t}\n \t\t\tcontinue;\n \t\t}\n-\t\tif (pte_uffd_wp(pteval)) {\n+\t\tif (pte_uffd(pteval)) {\n \t\t\t/*\n \t\t\t * Don't collapse the page if any of the small\n \t\t\t * PTEs are armed with uffd write protection.\n@@ -1682,7 +1682,7 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,\n \t\t\t * userfault messages that falls outside of\n \t\t\t * the registered range. So, just be simple.\n \t\t\t */\n-\t\t\tresult = SCAN_PTE_UFFD_WP;\n+\t\t\tresult = SCAN_PTE_UFFD;\n \t\t\tgoto out_unmap;\n \t\t}\n \n@@ -1892,9 +1892,12 @@ static enum scan_result try_collapse_pte_mapped_thp(struct mm_struct *mm, unsign\n \tif (!thp_vma_allowable_order(vma, vma-\u003evm_flags, TVA_FORCED_COLLAPSE, PMD_ORDER))\n \t\treturn SCAN_VMA_CHECK;\n \n-\t/* Keep pmd pgtable for uffd-wp; see comment in retract_page_tables() */\n-\tif (userfaultfd_wp(vma))\n-\t\treturn SCAN_PTE_UFFD_WP;\n+\t/*\n+\t * Keep pmd pgtable while the uffd bit is in use; see comment in\n+\t * retract_page_tables().\n+\t */\n+\tif (userfaultfd_protected(vma))\n+\t\treturn SCAN_PTE_UFFD;\n \n \tfolio = filemap_lock_folio(vma-\u003evm_file-\u003ef_mapping,\n \t\t\t linear_page_index(vma, haddr));\n@@ -2106,13 +2109,14 @@ static bool file_backed_vma_is_retractable(struct vm_area_struct *vma)\n \t\treturn false;\n \n \t/*\n-\t * When a vma is registered with uffd-wp, we cannot recycle\n+\t * When a vma is registered with uffd-wp or RWP, we cannot recycle\n \t * the page table because there may be pte markers installed.\n-\t * Other vmas can still have the same file mapped hugely, but\n-\t * skip this one: it will always be mapped in small page size\n-\t * for uffd-wp registered ranges.\n+\t * VM_UFFD_RWP ranges similarly rely on per-PTE uffd state\n+\t * and cannot be recycled to a shared PMD. Other vmas can still\n+\t * have the same file mapped hugely, but skip this one: it will\n+\t * always be mapped in small page size for these registrations.\n \t */\n-\tif (userfaultfd_wp(vma))\n+\tif (userfaultfd_protected(vma))\n \t\treturn false;\n \n \t/*\n@@ -3241,7 +3245,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,\n \t\t/* Whitelisted set of results where continuing OK */\n \t\tcase SCAN_NO_PTE_TABLE:\n \t\tcase SCAN_PTE_NON_PRESENT:\n-\t\tcase SCAN_PTE_UFFD_WP:\n+\t\tcase SCAN_PTE_UFFD:\n \t\tcase SCAN_LACK_REFERENCED_PAGE:\n \t\tcase SCAN_PAGE_NULL:\n \t\tcase SCAN_PAGE_COUNT:\ndiff --git a/mm/memory.c b/mm/memory.c\nindex ff338c2abe9231..20abe12da481cf 100644\n--- a/mm/memory.c\n+++ b/mm/memory.c\n@@ -893,8 +893,12 @@ static void restore_exclusive_pte(struct vm_area_struct *vma,\n \tif (pte_swp_soft_dirty(orig_pte))\n \t\tpte = pte_mksoft_dirty(pte);\n \n-\tif (pte_swp_uffd_wp(orig_pte))\n-\t\tpte = pte_mkuffd_wp(pte);\n+\tif (pte_swp_uffd(orig_pte))\n+\t\tpte = pte_mkuffd(pte);\n+\n+\t/* See do_swap_page(): restore PAGE_NONE for RWP */\n+\tif (pte_swp_uffd(orig_pte) \u0026\u0026 userfaultfd_rwp(vma))\n+\t\tpte = pte_modify(pte, PAGE_NONE);\n \n \tif ((vma-\u003evm_flags \u0026 VM_WRITE) \u0026\u0026\n \t can_change_pte_writable(vma, address, pte)) {\n@@ -984,8 +988,8 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,\n \t\t\tpte = softleaf_to_pte(entry);\n \t\t\tif (pte_swp_soft_dirty(orig_pte))\n \t\t\t\tpte = pte_swp_mksoft_dirty(pte);\n-\t\t\tif (pte_swp_uffd_wp(orig_pte))\n-\t\t\t\tpte = pte_swp_mkuffd_wp(pte);\n+\t\t\tif (pte_swp_uffd(orig_pte))\n+\t\t\t\tpte = pte_swp_mkuffd(pte);\n \t\t\tset_pte_at(src_mm, addr, src_pte, pte);\n \t\t}\n \t} else if (softleaf_is_device_private(entry)) {\n@@ -1018,8 +1022,8 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,\n \t\t\tentry = make_readable_device_private_entry(\n \t\t\t\t\t\t\tswp_offset(entry));\n \t\t\tpte = swp_entry_to_pte(entry);\n-\t\t\tif (pte_swp_uffd_wp(orig_pte))\n-\t\t\t\tpte = pte_swp_mkuffd_wp(pte);\n+\t\t\tif (pte_swp_uffd(orig_pte))\n+\t\t\t\tpte = pte_swp_mkuffd(pte);\n \t\t\tset_pte_at(src_mm, addr, src_pte, pte);\n \t\t}\n \t} else if (softleaf_is_device_exclusive(entry)) {\n@@ -1041,8 +1045,8 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,\n \t\t\t\t make_pte_marker(marker));\n \t\treturn 0;\n \t}\n-\tif (!userfaultfd_wp(dst_vma))\n-\t\tpte = pte_swp_clear_uffd_wp(pte);\n+\tif (!userfaultfd_protected(dst_vma))\n+\t\tpte = pte_swp_clear_uffd(pte);\n \tset_pte_at(dst_mm, addr, dst_pte, pte);\n \treturn 0;\n }\n@@ -1088,9 +1092,13 @@ copy_present_page(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma\n \t/* All done, just insert the new page copy in the child */\n \tpte = folio_mk_pte(new_folio, dst_vma-\u003evm_page_prot);\n \tpte = maybe_mkwrite(pte_mkdirty(pte), dst_vma);\n-\tif (userfaultfd_pte_wp(dst_vma, ptep_get(src_pte)))\n-\t\t/* Uffd-wp needs to be delivered to dest pte as well */\n-\t\tpte = pte_mkuffd_wp(pte);\n+\tif (userfaultfd_protected(dst_vma) \u0026\u0026 pte_uffd(ptep_get(src_pte))) {\n+\t\t/* The uffd bit needs to be delivered to the dest pte as well */\n+\t\tpte = pte_mkuffd(pte);\n+\t\t/* Restore PAGE_NONE so the RWP marker keeps trapping */\n+\t\tif (userfaultfd_rwp(dst_vma))\n+\t\t\tpte = pte_modify(pte, PAGE_NONE);\n+\t}\n \tset_pte_at(dst_vma-\u003evm_mm, addr, dst_pte, pte);\n \treturn 0;\n }\n@@ -1100,9 +1108,31 @@ static __always_inline void __copy_present_ptes(struct vm_area_struct *dst_vma,\n \t\tpte_t pte, unsigned long addr, int nr)\n {\n \tstruct mm_struct *src_mm = src_vma-\u003evm_mm;\n+\tbool writable;\n+\n+\t/*\n+\t * Snapshot writability before the RWP-disarm rewrite below: when the\n+\t * child is not RWP-armed, pte_modify(pte, dst_vma-\u003evm_page_prot) can\n+\t * silently drop _PAGE_RW from a resolved (no-marker) writable PTE,\n+\t * so a later pte_write(pte) check would skip the COW wrprotect and\n+\t * leave the parent writable over a folio shared with the child.\n+\t */\n+\twritable = pte_write(pte);\n+\n+\t/*\n+\t * Child is not RWP-armed: restore accessible protection so the\n+\t * inherited PAGE_NONE does not cost a fault on first read. Gate on\n+\t * pte_uffd(pte) so unrelated PAGE_NONE markers (e.g. NUMA balancing)\n+\t * are not normalised away.\n+\t */\n+\tif (!userfaultfd_protected(dst_vma)) {\n+\t\tif (userfaultfd_rwp(src_vma) \u0026\u0026 pte_uffd(pte))\n+\t\t\tpte = pte_modify(pte, dst_vma-\u003evm_page_prot);\n+\t\tpte = pte_clear_uffd(pte);\n+\t}\n \n \t/* If it's a COW mapping, write protect it both processes. */\n-\tif (is_cow_mapping(src_vma-\u003evm_flags) \u0026\u0026 pte_write(pte)) {\n+\tif (is_cow_mapping(src_vma-\u003evm_flags) \u0026\u0026 writable) {\n \t\twrprotect_ptes(src_mm, addr, src_pte, nr);\n \t\tpte = pte_wrprotect(pte);\n \t}\n@@ -1112,9 +1142,6 @@ static __always_inline void __copy_present_ptes(struct vm_area_struct *dst_vma,\n \t\tpte = pte_mkclean(pte);\n \tpte = pte_mkold(pte);\n \n-\tif (!userfaultfd_wp(dst_vma))\n-\t\tpte = pte_clear_uffd_wp(pte);\n-\n \tset_ptes(dst_vma-\u003evm_mm, addr, dst_pte, pte, nr);\n }\n \n@@ -3925,8 +3952,8 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf)\n \t\tif (unlikely(unshare)) {\n \t\t\tif (pte_soft_dirty(vmf-\u003eorig_pte))\n \t\t\t\tentry = pte_mksoft_dirty(entry);\n-\t\t\tif (pte_uffd_wp(vmf-\u003eorig_pte))\n-\t\t\t\tentry = pte_mkuffd_wp(entry);\n+\t\t\tif (pte_uffd(vmf-\u003eorig_pte))\n+\t\t\t\tentry = pte_mkuffd(entry);\n \t\t} else {\n \t\t\tentry = maybe_mkwrite(pte_mkdirty(entry), vma);\n \t\t}\n@@ -4261,7 +4288,7 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf)\n \t\t\t * etc.) because we're only removing the uffd-wp bit,\n \t\t\t * which is completely invisible to the user.\n \t\t\t */\n-\t\t\tpte = pte_clear_uffd_wp(ptep_get(vmf-\u003epte));\n+\t\t\tpte = pte_clear_uffd(ptep_get(vmf-\u003epte));\n \n \t\t\tset_pte_at(vma-\u003evm_mm, vmf-\u003eaddress, vmf-\u003epte, pte);\n \t\t\t/*\n@@ -5038,8 +5065,16 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)\n \tpte = mk_pte(page, vma-\u003evm_page_prot);\n \tif (pte_swp_soft_dirty(vmf-\u003eorig_pte))\n \t\tpte = pte_mksoft_dirty(pte);\n-\tif (pte_swp_uffd_wp(vmf-\u003eorig_pte))\n-\t\tpte = pte_mkuffd_wp(pte);\n+\tif (pte_swp_uffd(vmf-\u003eorig_pte))\n+\t\tpte = pte_mkuffd(pte);\n+\n+\t/*\n+\t * A page reclaimed while RWP-protected carries the uffd bit on\n+\t * its swap entry. Re-apply PAGE_NONE on swap-in so the first access\n+\t * still traps as an RWP fault. pte_modify() preserves _PAGE_UFFD.\n+\t */\n+\tif (pte_swp_uffd(vmf-\u003eorig_pte) \u0026\u0026 userfaultfd_rwp(vma))\n+\t\tpte = pte_modify(pte, PAGE_NONE);\n \n \t/*\n \t * Same logic as in do_wp_page(); however, optimize for pages that are\n@@ -5259,7 +5294,7 @@ void map_anon_folio_pte_nopf(struct folio *folio, pte_t *pte,\n \tif (vma-\u003evm_flags \u0026 VM_WRITE)\n \t\tentry = pte_mkwrite(pte_mkdirty(entry), vma);\n \tif (uffd_wp)\n-\t\tentry = pte_mkuffd_wp(entry);\n+\t\tentry = pte_mkuffd(entry);\n \n \tfolio_ref_add(folio, nr_pages - 1);\n \tfolio_add_new_anon_rmap(folio, vma, addr, RMAP_EXCLUSIVE);\n@@ -5326,7 +5361,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)\n \t\t\treturn handle_userfault(vmf, VM_UFFD_MISSING);\n \t\t}\n \t\tif (vmf_orig_pte_uffd_wp(vmf))\n-\t\t\tentry = pte_mkuffd_wp(entry);\n+\t\t\tentry = pte_mkuffd(entry);\n \t\tset_pte_at(vma-\u003evm_mm, addr, vmf-\u003epte, entry);\n \n \t\t/* No need to invalidate - it was non-present before */\n@@ -5576,7 +5611,7 @@ void set_pte_range(struct vm_fault *vmf, struct folio *folio,\n \telse if (pte_write(entry) \u0026\u0026 folio_test_dirty(folio))\n \t\tentry = pte_mkdirty(entry);\n \tif (unlikely(vmf_orig_pte_uffd_wp(vmf)))\n-\t\tentry = pte_mkuffd_wp(entry);\n+\t\tentry = pte_mkuffd(entry);\n \t/* copy-on-write page */\n \tif (write \u0026\u0026 !(vma-\u003evm_flags \u0026 VM_SHARED)) {\n \t\tVM_BUG_ON_FOLIO(nr != 1, folio);\n@@ -6091,6 +6126,16 @@ static void numa_rebuild_large_mapping(struct vm_fault *vmf, struct vm_area_stru\n \t\tif (!pte_present(ptent) || !pte_protnone(ptent))\n \t\t\tcontinue;\n \n+\t\t/*\n+\t\t * RWP-armed PTEs are also protnone but carry _PAGE_UFFD as a\n+\t\t * marker. Leave them alone -- rewriting to vm_page_prot would\n+\t\t * stop the RWP trap. Gate on userfaultfd_rwp(vma) too:\n+\t\t * NUMA balancing preserves _PAGE_UFFD on UFFD_WP-marked PTEs\n+\t\t * when applying PROT_NONE, and those still need rebuilding.\n+\t\t */\n+\t\tif (userfaultfd_rwp(vma) \u0026\u0026 pte_uffd(ptent))\n+\t\t\tcontinue;\n+\n \t\tif (pfn_folio(pte_pfn(ptent)) != folio)\n \t\t\tcontinue;\n \n@@ -6106,6 +6151,35 @@ static void numa_rebuild_large_mapping(struct vm_fault *vmf, struct vm_area_stru\n \t}\n }\n \n+static vm_fault_t do_uffd_rwp(struct vm_fault *vmf)\n+{\n+\tpte_t pte;\n+\n+\tif (!userfaultfd_rwp_async(vmf-\u003evma)) {\n+\t\t/* Sync mode: unmap PTE and deliver to userfaultfd handler */\n+\t\tpte_unmap(vmf-\u003epte);\n+\t\treturn handle_userfault(vmf, VM_UFFD_RWP);\n+\t}\n+\n+\tspin_lock(vmf-\u003eptl);\n+\tif (unlikely(!pte_same(ptep_get(vmf-\u003epte), vmf-\u003eorig_pte))) {\n+\t\tpte_unmap_unlock(vmf-\u003epte, vmf-\u003eptl);\n+\t\treturn 0;\n+\t}\n+\tpte = pte_modify(vmf-\u003eorig_pte, vmf-\u003evma-\u003evm_page_prot);\n+\t/* pte_modify() preserves _PAGE_UFFD; drop it on resolution */\n+\tpte = pte_clear_uffd(pte);\n+\tpte = pte_mkyoung(pte);\n+\tif (!pte_write(pte) \u0026\u0026\n+\t vma_wants_manual_pte_write_upgrade(vmf-\u003evma) \u0026\u0026\n+\t can_change_pte_writable(vmf-\u003evma, vmf-\u003eaddress, pte))\n+\t\tpte = pte_mkwrite(pte, vmf-\u003evma);\n+\tset_pte_at(vmf-\u003evma-\u003evm_mm, vmf-\u003eaddress, vmf-\u003epte, pte);\n+\tupdate_mmu_cache(vmf-\u003evma, vmf-\u003eaddress, vmf-\u003epte);\n+\tpte_unmap_unlock(vmf-\u003epte, vmf-\u003eptl);\n+\treturn 0;\n+}\n+\n static vm_fault_t do_numa_page(struct vm_fault *vmf)\n {\n \tstruct vm_area_struct *vma = vmf-\u003evma;\n@@ -6381,8 +6455,16 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf)\n \tif (!pte_present(vmf-\u003eorig_pte))\n \t\treturn do_swap_page(vmf);\n \n-\tif (pte_protnone(vmf-\u003eorig_pte) \u0026\u0026 vma_is_accessible(vmf-\u003evma))\n+\tif (pte_protnone(vmf-\u003eorig_pte) \u0026\u0026 vma_is_accessible(vmf-\u003evma)) {\n+\t\t/*\n+\t\t * RWP-protected PTEs are protnone plus the uffd bit. On a\n+\t\t * VM_UFFD_RWP VMA, a protnone PTE without the uffd bit is\n+\t\t * NUMA hinting and must still fall through to do_numa_page().\n+\t\t */\n+\t\tif (userfaultfd_pte_rwp(vmf-\u003evma, vmf-\u003eorig_pte))\n+\t\t\treturn do_uffd_rwp(vmf);\n \t\treturn do_numa_page(vmf);\n+\t}\n \n \tspin_lock(vmf-\u003eptl);\n \tentry = vmf-\u003eorig_pte;\n@@ -6496,8 +6578,11 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma,\n \t\treturn 0;\n \t}\n \tif (pmd_trans_huge(vmf.orig_pmd)) {\n-\t\tif (pmd_protnone(vmf.orig_pmd) \u0026\u0026 vma_is_accessible(vma))\n+\t\tif (pmd_protnone(vmf.orig_pmd) \u0026\u0026 vma_is_accessible(vma)) {\n+\t\t\tif (userfaultfd_huge_pmd_rwp(vma, vmf.orig_pmd))\n+\t\t\t\treturn do_huge_pmd_uffd_rwp(\u0026vmf);\n \t\t\treturn do_huge_pmd_numa_page(\u0026vmf);\n+\t\t}\n \n \t\tif ((flags \u0026 (FAULT_FLAG_WRITE|FAULT_FLAG_UNSHARE)) \u0026\u0026\n \t\t !pmd_write(vmf.orig_pmd)) {\ndiff --git a/mm/migrate.c b/mm/migrate.c\nindex d9b23909d716c6..9516cda35e341c 100644\n--- a/mm/migrate.c\n+++ b/mm/migrate.c\n@@ -326,8 +326,12 @@ static bool try_to_map_unused_to_zeropage(struct page_vma_mapped_walk *pvmw,\n \n \tif (pte_swp_soft_dirty(old_pte))\n \t\tnewpte = pte_mksoft_dirty(newpte);\n-\tif (pte_swp_uffd_wp(old_pte))\n-\t\tnewpte = pte_mkuffd_wp(newpte);\n+\tif (pte_swp_uffd(old_pte))\n+\t\tnewpte = pte_mkuffd(newpte);\n+\n+\t/* See remove_migration_pte(): restore PAGE_NONE for RWP */\n+\tif (pte_swp_uffd(old_pte) \u0026\u0026 userfaultfd_rwp(pvmw-\u003evma))\n+\t\tnewpte = pte_modify(newpte, PAGE_NONE);\n \n \tset_pte_at(pvmw-\u003evma-\u003evm_mm, pvmw-\u003eaddress, pvmw-\u003epte, newpte);\n \n@@ -391,8 +395,12 @@ static bool remove_migration_pte(struct folio *folio,\n \n \t\tif (softleaf_is_migration_write(entry))\n \t\t\tpte = pte_mkwrite(pte, vma);\n-\t\telse if (pte_swp_uffd_wp(old_pte))\n-\t\t\tpte = pte_mkuffd_wp(pte);\n+\t\telse if (pte_swp_uffd(old_pte))\n+\t\t\tpte = pte_mkuffd(pte);\n+\n+\t\t/* See do_swap_page(): restore PAGE_NONE for RWP */\n+\t\tif (pte_swp_uffd(old_pte) \u0026\u0026 userfaultfd_rwp(vma))\n+\t\t\tpte = pte_modify(pte, PAGE_NONE);\n \n \t\tif (folio_test_anon(folio) \u0026\u0026 !softleaf_is_migration_read(entry))\n \t\t\trmap_flags |= RMAP_EXCLUSIVE;\n@@ -407,8 +415,8 @@ static bool remove_migration_pte(struct folio *folio,\n \t\t\tpte = softleaf_to_pte(entry);\n \t\t\tif (pte_swp_soft_dirty(old_pte))\n \t\t\t\tpte = pte_swp_mksoft_dirty(pte);\n-\t\t\tif (pte_swp_uffd_wp(old_pte))\n-\t\t\t\tpte = pte_swp_mkuffd_wp(pte);\n+\t\t\tif (pte_swp_uffd(old_pte))\n+\t\t\t\tpte = pte_swp_mkuffd(pte);\n \t\t}\n \n #ifdef CONFIG_HUGETLB_PAGE\ndiff --git a/mm/migrate_device.c b/mm/migrate_device.c\nindex 554754eb26ff2f..17da1bab02484e 100644\n--- a/mm/migrate_device.c\n+++ b/mm/migrate_device.c\n@@ -445,13 +445,13 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,\n \t\t\tif (pte_present(pte)) {\n \t\t\t\tif (pte_soft_dirty(pte))\n \t\t\t\t\tswp_pte = pte_swp_mksoft_dirty(swp_pte);\n-\t\t\t\tif (pte_uffd_wp(pte))\n-\t\t\t\t\tswp_pte = pte_swp_mkuffd_wp(swp_pte);\n+\t\t\t\tif (pte_uffd(pte))\n+\t\t\t\t\tswp_pte = pte_swp_mkuffd(swp_pte);\n \t\t\t} else {\n \t\t\t\tif (pte_swp_soft_dirty(pte))\n \t\t\t\t\tswp_pte = pte_swp_mksoft_dirty(swp_pte);\n-\t\t\t\tif (pte_swp_uffd_wp(pte))\n-\t\t\t\t\tswp_pte = pte_swp_mkuffd_wp(swp_pte);\n+\t\t\t\tif (pte_swp_uffd(pte))\n+\t\t\t\t\tswp_pte = pte_swp_mkuffd(swp_pte);\n \t\t\t}\n \t\t\tset_pte_at(mm, addr, ptep, swp_pte);\n \ndiff --git a/mm/mprotect.c b/mm/mprotect.c\nindex 9cbf932b028cf9..cc85a8862c2818 100644\n--- a/mm/mprotect.c\n+++ b/mm/mprotect.c\n@@ -214,8 +214,9 @@ static __always_inline void set_write_prot_commit_flush_ptes(struct vm_area_stru\n static long change_softleaf_pte(struct vm_area_struct *vma,\n \tunsigned long addr, pte_t *pte, pte_t oldpte, unsigned long cp_flags)\n {\n-\tconst bool uffd_wp = cp_flags \u0026 MM_CP_UFFD_WP;\n-\tconst bool uffd_wp_resolve = cp_flags \u0026 MM_CP_UFFD_WP_RESOLVE;\n+\tconst bool uffd_prot = cp_flags \u0026 (MM_CP_UFFD_WP | MM_CP_UFFD_RWP);\n+\tconst bool uffd_prot_resolve = cp_flags \u0026\n+\t\t(MM_CP_UFFD_WP_RESOLVE | MM_CP_UFFD_RWP_RESOLVE);\n \tsoftleaf_t entry = softleaf_from_pte(oldpte);\n \tpte_t newpte;\n \n@@ -240,8 +241,8 @@ static long change_softleaf_pte(struct vm_area_struct *vma,\n \t\t */\n \t\tentry = make_readable_device_private_entry(swp_offset(entry));\n \t\tnewpte = swp_entry_to_pte(entry);\n-\t\tif (pte_swp_uffd_wp(oldpte))\n-\t\t\tnewpte = pte_swp_mkuffd_wp(newpte);\n+\t\tif (pte_swp_uffd(oldpte))\n+\t\t\tnewpte = pte_swp_mkuffd(newpte);\n \t} else if (softleaf_is_marker(entry)) {\n \t\t/*\n \t\t * Ignore error swap entries unconditionally,\n@@ -256,7 +257,7 @@ static long change_softleaf_pte(struct vm_area_struct *vma,\n \t\t * to unprotect it, drop it; the next page\n \t\t * fault will trigger without uffd trapping.\n \t\t */\n-\t\tif (uffd_wp_resolve) {\n+\t\tif (uffd_prot_resolve) {\n \t\t\tpte_clear(vma-\u003evm_mm, addr, pte);\n \t\t\treturn 1;\n \t\t}\n@@ -265,10 +266,10 @@ static long change_softleaf_pte(struct vm_area_struct *vma,\n \t\tnewpte = oldpte;\n \t}\n \n-\tif (uffd_wp)\n-\t\tnewpte = pte_swp_mkuffd_wp(newpte);\n-\telse if (uffd_wp_resolve)\n-\t\tnewpte = pte_swp_clear_uffd_wp(newpte);\n+\tif (uffd_prot)\n+\t\tnewpte = pte_swp_mkuffd(newpte);\n+\telse if (uffd_prot_resolve)\n+\t\tnewpte = pte_swp_clear_uffd(newpte);\n \n \tif (!pte_same(oldpte, newpte)) {\n \t\tset_pte_at(vma-\u003evm_mm, addr, pte, newpte);\n@@ -282,17 +283,28 @@ static __always_inline void change_present_ptes(struct mmu_gather *tlb,\n \t\tint nr_ptes, unsigned long end, pgprot_t newprot,\n \t\tstruct folio *folio, struct page *page, unsigned long cp_flags)\n {\n-\tconst bool uffd_wp_resolve = cp_flags \u0026 MM_CP_UFFD_WP_RESOLVE;\n-\tconst bool uffd_wp = cp_flags \u0026 MM_CP_UFFD_WP;\n+\tconst bool uffd_prot = cp_flags \u0026 (MM_CP_UFFD_WP | MM_CP_UFFD_RWP);\n+\tconst bool uffd_prot_resolve = cp_flags \u0026\n+\t\t(MM_CP_UFFD_WP_RESOLVE | MM_CP_UFFD_RWP_RESOLVE);\n \tpte_t ptent, oldpte;\n \n \toldpte = modify_prot_start_ptes(vma, addr, ptep, nr_ptes);\n \tptent = pte_modify(oldpte, newprot);\n \n-\tif (uffd_wp)\n-\t\tptent = pte_mkuffd_wp(ptent);\n-\telse if (uffd_wp_resolve)\n-\t\tptent = pte_clear_uffd_wp(ptent);\n+\tif (uffd_prot)\n+\t\tptent = pte_mkuffd(ptent);\n+\telse if (uffd_prot_resolve)\n+\t\tptent = pte_clear_uffd(ptent);\n+\n+\t/*\n+\t * The uffd bit on a VM_UFFD_RWP VMA carries PROT_NONE\n+\t * semantics. If mprotect() or NUMA hinting changed the\n+\t * base protection, restore PAGE_NONE so the PTE still\n+\t * traps on any access. pte_modify() preserves\n+\t * _PAGE_UFFD.\n+\t */\n+\tif (userfaultfd_rwp(vma) \u0026\u0026 pte_uffd(ptent))\n+\t\tptent = pte_modify(ptent, PAGE_NONE);\n \n \t/*\n \t * In some writable, shared mappings, we might want\n@@ -325,6 +337,7 @@ static long change_pte_range(struct mmu_gather *tlb,\n \tlong pages = 0;\n \tbool is_private_single_threaded;\n \tbool prot_numa = cp_flags \u0026 MM_CP_PROT_NUMA;\n+\tbool uffd_rwp = cp_flags \u0026 MM_CP_UFFD_RWP;\n \tbool uffd_wp = cp_flags \u0026 MM_CP_UFFD_WP;\n \tint nr_ptes;\n \n@@ -350,6 +363,14 @@ static long change_pte_range(struct mmu_gather *tlb,\n \t\t\t/* Already in the desired state. */\n \t\t\tif (prot_numa \u0026\u0026 pte_protnone(oldpte))\n \t\t\t\tcontinue;\n+\t\t\t/*\n+\t\t\t * RWP-protected PTEs carry _PAGE_UFFD as a marker on\n+\t\t\t * top of PROT_NONE. Skip only entries already in that\n+\t\t\t * exact state; plain PROT_NONE from mprotect() still needs\n+\t\t\t * to be promoted so future faults can be distinguished.\n+\t\t\t */\n+\t\t\tif (uffd_rwp \u0026\u0026 pte_protnone(oldpte) \u0026\u0026 pte_uffd(oldpte))\n+\t\t\t\tcontinue;\n \n \t\t\tpage = vm_normal_page(vma, addr, oldpte);\n \t\t\tif (page)\n@@ -358,6 +379,8 @@ static long change_pte_range(struct mmu_gather *tlb,\n \t\t\t/*\n \t\t\t * Avoid trapping faults against the zero or KSM\n \t\t\t * pages. See similar comment in change_huge_pmd.\n+\t\t\t * Skip this filter for uffd RWP which\n+\t\t\t * must set protnone regardless of NUMA placement.\n \t\t\t */\n \t\t\tif (prot_numa \u0026\u0026\n \t\t\t !folio_can_map_prot_numa(folio, vma,\n@@ -428,7 +451,7 @@ pgtable_split_needed(struct vm_area_struct *vma, unsigned long cp_flags)\n \t * (e.g. 2M shmem) because file thp is handled differently when\n \t * split by erasing the pmd so far.\n \t */\n-\treturn (cp_flags \u0026 MM_CP_UFFD_WP) \u0026\u0026 !vma_is_anonymous(vma);\n+\treturn (cp_flags \u0026 (MM_CP_UFFD_WP | MM_CP_UFFD_RWP)) \u0026\u0026 !vma_is_anonymous(vma);\n }\n \n /*\n@@ -667,7 +690,16 @@ long change_protection(struct mmu_gather *tlb,\n \tpgprot_t newprot = vma-\u003evm_page_prot;\n \tlong pages;\n \n-\tBUG_ON((cp_flags \u0026 MM_CP_UFFD_WP_ALL) == MM_CP_UFFD_WP_ALL);\n+\t/*\n+\t * MM_CP_UFFD_{WP,RWP} and _RESOLVE are mutually exclusive within one\n+\t * change, and WP and RWP cannot mix. Miswired callers get a warn and\n+\t * a no-op; userspace cannot reach this state.\n+\t */\n+\tif (WARN_ON_ONCE((cp_flags \u0026 MM_CP_UFFD_WP_ALL) == MM_CP_UFFD_WP_ALL ||\n+\t\t\t (cp_flags \u0026 MM_CP_UFFD_RWP_ALL) == MM_CP_UFFD_RWP_ALL ||\n+\t\t\t ((cp_flags \u0026 MM_CP_UFFD_WP_ALL) \u0026\u0026\n+\t\t\t (cp_flags \u0026 MM_CP_UFFD_RWP_ALL))))\n+\t\treturn 0;\n \n #ifdef CONFIG_NUMA_BALANCING\n \t/*\n@@ -681,6 +713,10 @@ long change_protection(struct mmu_gather *tlb,\n \tWARN_ON_ONCE(cp_flags \u0026 MM_CP_PROT_NUMA);\n #endif\n \n+\tif (IS_ENABLED(CONFIG_ARCH_HAS_PTE_PROTNONE) \u0026\u0026\n+\t (cp_flags \u0026 MM_CP_UFFD_RWP))\n+\t\tnewprot = PAGE_NONE;\n+\n \tif (is_vm_hugetlb_page(vma))\n \t\tpages = hugetlb_change_protection(vma, start, end, newprot,\n \t\t\t\t\t\t cp_flags);\ndiff --git a/mm/mremap.c b/mm/mremap.c\nindex e9c8b1d05832be..8a46ec5831c84e 100644\n--- a/mm/mremap.c\n+++ b/mm/mremap.c\n@@ -296,10 +296,19 @@ static int move_ptes(struct pagetable_move_control *pmc,\n \t\t\tpte_clear(mm, new_addr, new_ptep);\n \t\telse {\n \t\t\tif (need_clear_uffd_wp) {\n-\t\t\t\tif (pte_present(pte))\n-\t\t\t\t\tpte = pte_clear_uffd_wp(pte);\n-\t\t\t\telse\n-\t\t\t\t\tpte = pte_swp_clear_uffd_wp(pte);\n+\t\t\t\tif (pte_present(pte)) {\n+\t\t\t\t\t/*\n+\t\t\t\t\t * See __copy_present_ptes(): normalise\n+\t\t\t\t\t * RWP PTEs so the destination starts\n+\t\t\t\t\t * accessible instead of taking a\n+\t\t\t\t\t * numa-hinting fault on first access.\n+\t\t\t\t\t */\n+\t\t\t\t\tif (userfaultfd_rwp(vma) \u0026\u0026 pte_uffd(pte))\n+\t\t\t\t\t\tpte = pte_modify(pte, vma-\u003evm_page_prot);\n+\t\t\t\t\tpte = pte_clear_uffd(pte);\n+\t\t\t\t} else {\n+\t\t\t\t\tpte = pte_swp_clear_uffd(pte);\n+\t\t\t\t}\n \t\t\t}\n \t\t\tset_ptes(mm, new_addr, new_ptep, pte, nr_ptes);\n \t\t}\ndiff --git a/mm/page_table_check.c b/mm/page_table_check.c\nindex 53a8997ec043ea..3fb995e5d40d60 100644\n--- a/mm/page_table_check.c\n+++ b/mm/page_table_check.c\n@@ -188,8 +188,8 @@ static inline bool softleaf_cached_writable(softleaf_t entry)\n static void page_table_check_pte_flags(pte_t pte)\n {\n \tif (pte_present(pte)) {\n-\t\tWARN_ON_ONCE(pte_uffd_wp(pte) \u0026\u0026 pte_write(pte));\n-\t} else if (pte_swp_uffd_wp(pte)) {\n+\t\tWARN_ON_ONCE(pte_uffd(pte) \u0026\u0026 pte_write(pte));\n+\t} else if (pte_swp_uffd(pte)) {\n \t\tconst softleaf_t entry = softleaf_from_pte(pte);\n \n \t\tWARN_ON_ONCE(softleaf_cached_writable(entry));\n@@ -216,9 +216,9 @@ EXPORT_SYMBOL(__page_table_check_ptes_set);\n static inline void page_table_check_pmd_flags(pmd_t pmd)\n {\n \tif (pmd_present(pmd)) {\n-\t\tif (pmd_uffd_wp(pmd))\n+\t\tif (pmd_uffd(pmd))\n \t\t\tWARN_ON_ONCE(pmd_write(pmd));\n-\t} else if (pmd_swp_uffd_wp(pmd)) {\n+\t} else if (pmd_swp_uffd(pmd)) {\n \t\tconst softleaf_t entry = softleaf_from_pmd(pmd);\n \n \t\tWARN_ON_ONCE(softleaf_cached_writable(entry));\ndiff --git a/mm/rmap.c b/mm/rmap.c\nindex 1c77d5dc06e9fa..9fb733489898de 100644\n--- a/mm/rmap.c\n+++ b/mm/rmap.c\n@@ -1965,7 +1965,7 @@ static inline unsigned int folio_unmap_pte_batch(struct folio *folio,\n \tif (pte_unused(pte))\n \t\treturn 1;\n \n-\tif (userfaultfd_wp(vma))\n+\tif (userfaultfd_protected(vma))\n \t\treturn 1;\n \n \t/*\n@@ -2318,13 +2318,13 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,\n \t\t\tif (likely(pte_present(pteval))) {\n \t\t\t\tif (pte_soft_dirty(pteval))\n \t\t\t\t\tswp_pte = pte_swp_mksoft_dirty(swp_pte);\n-\t\t\t\tif (pte_uffd_wp(pteval))\n-\t\t\t\t\tswp_pte = pte_swp_mkuffd_wp(swp_pte);\n+\t\t\t\tif (pte_uffd(pteval))\n+\t\t\t\t\tswp_pte = pte_swp_mkuffd(swp_pte);\n \t\t\t} else {\n \t\t\t\tif (pte_swp_soft_dirty(pteval))\n \t\t\t\t\tswp_pte = pte_swp_mksoft_dirty(swp_pte);\n-\t\t\t\tif (pte_swp_uffd_wp(pteval))\n-\t\t\t\t\tswp_pte = pte_swp_mkuffd_wp(swp_pte);\n+\t\t\t\tif (pte_swp_uffd(pteval))\n+\t\t\t\t\tswp_pte = pte_swp_mkuffd(swp_pte);\n \t\t\t}\n \t\t\tset_pte_at(mm, address, pvmw.pte, swp_pte);\n \t\t} else {\n@@ -2692,14 +2692,14 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,\n \t\t\t\tswp_pte = swp_entry_to_pte(entry);\n \t\t\t\tif (pte_soft_dirty(pteval))\n \t\t\t\t\tswp_pte = pte_swp_mksoft_dirty(swp_pte);\n-\t\t\t\tif (pte_uffd_wp(pteval))\n-\t\t\t\t\tswp_pte = pte_swp_mkuffd_wp(swp_pte);\n+\t\t\t\tif (pte_uffd(pteval))\n+\t\t\t\t\tswp_pte = pte_swp_mkuffd(swp_pte);\n \t\t\t} else {\n \t\t\t\tswp_pte = swp_entry_to_pte(entry);\n \t\t\t\tif (pte_swp_soft_dirty(pteval))\n \t\t\t\t\tswp_pte = pte_swp_mksoft_dirty(swp_pte);\n-\t\t\t\tif (pte_swp_uffd_wp(pteval))\n-\t\t\t\t\tswp_pte = pte_swp_mkuffd_wp(swp_pte);\n+\t\t\t\tif (pte_swp_uffd(pteval))\n+\t\t\t\t\tswp_pte = pte_swp_mkuffd(swp_pte);\n \t\t\t}\n \t\t\tif (folio_test_hugetlb(folio))\n \t\t\t\tset_huge_pte_at(mm, address, pvmw.pte, swp_pte,\ndiff --git a/mm/swapfile.c b/mm/swapfile.c\nindex 78b49b0658adde..a6f19759e8e302 100644\n--- a/mm/swapfile.c\n+++ b/mm/swapfile.c\n@@ -2496,8 +2496,13 @@ static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd,\n \tnew_pte = pte_mkold(mk_pte(page, vma-\u003evm_page_prot));\n \tif (pte_swp_soft_dirty(old_pte))\n \t\tnew_pte = pte_mksoft_dirty(new_pte);\n-\tif (pte_swp_uffd_wp(old_pte))\n-\t\tnew_pte = pte_mkuffd_wp(new_pte);\n+\tif (pte_swp_uffd(old_pte))\n+\t\tnew_pte = pte_mkuffd(new_pte);\n+\n+\t/* See do_swap_page(): restore PAGE_NONE for RWP */\n+\tif (pte_swp_uffd(old_pte) \u0026\u0026 userfaultfd_rwp(vma))\n+\t\tnew_pte = pte_modify(new_pte, PAGE_NONE);\n+\n setpte:\n \tset_pte_at(vma-\u003evm_mm, addr, pte, new_pte);\n \tfolio_put_swap(swapcache, folio_file_page(swapcache, swp_offset(entry)));\ndiff --git a/mm/userfaultfd.c b/mm/userfaultfd.c\nindex 246af12bf80142..a7659b0a1147ed 100644\n--- a/mm/userfaultfd.c\n+++ b/mm/userfaultfd.c\n@@ -376,7 +376,7 @@ static int mfill_atomic_install_pte(pmd_t *dst_pmd,\n \tif (writable)\n \t\t_dst_pte = pte_mkwrite(_dst_pte, dst_vma);\n \tif (flags \u0026 MFILL_ATOMIC_WP)\n-\t\t_dst_pte = pte_mkuffd_wp(_dst_pte);\n+\t\t_dst_pte = pte_mkuffd(_dst_pte);\n \n \tret = -EAGAIN;\n \tdst_pte = pte_offset_map_lock(dst_mm, dst_pmd, dst_addr, \u0026ptl);\n@@ -1164,6 +1164,75 @@ static int mwriteprotect_range(struct userfaultfd_ctx *ctx, unsigned long start,\n \treturn err;\n }\n \n+int mrwprotect_range(struct userfaultfd_ctx *ctx, unsigned long start,\n+\t\t unsigned long len, bool enable_rwp)\n+{\n+\tstruct mm_struct *dst_mm = ctx-\u003emm;\n+\tunsigned long end = start + len;\n+\tstruct vm_area_struct *dst_vma;\n+\tunsigned int mm_cp_flags;\n+\tstruct mmu_gather tlb;\n+\tbool found = false;\n+\tVMA_ITERATOR(vmi, dst_mm, start);\n+\n+\tVM_WARN_ON_ONCE(start \u0026 ~PAGE_MASK);\n+\tVM_WARN_ON_ONCE(len \u0026 ~PAGE_MASK);\n+\tVM_WARN_ON_ONCE(start + len \u003c= start);\n+\n+\tguard(mmap_read_lock)(dst_mm);\n+\tguard(rwsem_read)(\u0026ctx-\u003emap_changing_lock);\n+\n+\tif (atomic_read(\u0026ctx-\u003emmap_changing))\n+\t\treturn -EAGAIN;\n+\n+\tif (enable_rwp)\n+\t\tmm_cp_flags = MM_CP_UFFD_RWP;\n+\telse\n+\t\tmm_cp_flags = MM_CP_UFFD_RWP_RESOLVE;\n+\n+\t/*\n+\t * Pre-scan the range: validate every spanned VMA before applying\n+\t * any change_protection() so a partial failure cannot leave the\n+\t * process with only a prefix of the range re-protected.\n+\t */\n+\tfor_each_vma_range(vmi, dst_vma, end) {\n+\t\tif (!userfaultfd_rwp(dst_vma))\n+\t\t\treturn -ENOENT;\n+\n+\t\tif (is_vm_hugetlb_page(dst_vma)) {\n+\t\t\tunsigned long page_mask;\n+\n+\t\t\tpage_mask = vma_kernel_pagesize(dst_vma) - 1;\n+\t\t\tif ((start \u0026 page_mask) || (len \u0026 page_mask))\n+\t\t\t\treturn -EINVAL;\n+\t\t}\n+\t\tfound = true;\n+\t}\n+\tif (!found)\n+\t\treturn -ENOENT;\n+\n+\tvma_iter_set(\u0026vmi, start);\n+\ttlb_gather_mmu(\u0026tlb, dst_mm);\n+\tfor_each_vma_range(vmi, dst_vma, end) {\n+\t\tunsigned long vma_start = max(dst_vma-\u003evm_start, start);\n+\t\tunsigned long vma_end = min(dst_vma-\u003evm_end, end);\n+\t\tunsigned int flags = mm_cp_flags;\n+\n+\t\t/*\n+\t\t * On resolve, try to upgrade writability per-VMA --\n+\t\t * MM_CP_TRY_CHANGE_WRITABLE WARNs in\n+\t\t * maybe_change_pte_writable() if the VMA is not VM_WRITE,\n+\t\t * and RWP can be registered on PROT_READ-only mappings.\n+\t\t */\n+\t\tif (!enable_rwp \u0026\u0026 vma_wants_manual_pte_write_upgrade(dst_vma))\n+\t\t\tflags |= MM_CP_TRY_CHANGE_WRITABLE;\n+\n+\t\tchange_protection(\u0026tlb, dst_vma, vma_start, vma_end, flags);\n+\t}\n+\ttlb_finish_mmu(\u0026tlb);\n+\n+\treturn 0;\n+}\n \n void double_pt_lock(spinlock_t *ptl1,\n \t\t spinlock_t *ptl2)\n@@ -1292,6 +1361,13 @@ static long move_present_ptes(struct mm_struct *mm,\n \t\tif (pte_dirty(orig_src_pte))\n \t\t\torig_dst_pte = pte_mkdirty(orig_dst_pte);\n \t\torig_dst_pte = pte_mkwrite(orig_dst_pte, dst_vma);\n+\n+\t\t/* Re-arm RWP on the moved PTE if dst_vma is RWP-registered. */\n+\t\tif (userfaultfd_rwp(dst_vma)) {\n+\t\t\torig_dst_pte = pte_modify(orig_dst_pte, PAGE_NONE);\n+\t\t\torig_dst_pte = pte_mkuffd(orig_dst_pte);\n+\t\t}\n+\n \t\tset_pte_at(mm, dst_addr, dst_pte, orig_dst_pte);\n \n \t\tsrc_addr += PAGE_SIZE;\n@@ -1373,6 +1449,9 @@ static int move_swap_pte(struct mm_struct *mm, struct vm_area_struct *dst_vma,\n \torig_src_pte = ptep_get_and_clear(mm, src_addr, src_pte);\n \tif (pgtable_supports_soft_dirty())\n \t\torig_src_pte = pte_swp_mksoft_dirty(orig_src_pte);\n+\t/* Re-arm RWP on the moved swap entry if dst_vma is RWP-registered. */\n+\tif (userfaultfd_rwp(dst_vma))\n+\t\torig_src_pte = pte_swp_mkuffd(orig_src_pte);\n \tset_pte_at(mm, dst_addr, dst_pte, orig_src_pte);\n \tdouble_pt_unlock(dst_ptl, src_ptl);\n \n@@ -1399,6 +1478,13 @@ static int move_zeropage_pte(struct mm_struct *mm,\n \n \tzero_pte = pte_mkspecial(pfn_pte(zero_pfn(dst_addr),\n \t\t\t\t\t dst_vma-\u003evm_page_prot));\n+\n+\t/* Re-arm RWP on the moved PTE if dst_vma is RWP-registered. */\n+\tif (userfaultfd_rwp(dst_vma)) {\n+\t\tzero_pte = pte_modify(zero_pte, PAGE_NONE);\n+\t\tzero_pte = pte_mkuffd(zero_pte);\n+\t}\n+\n \tptep_clear_flush(src_vma, src_addr, src_pte);\n \tset_pte_at(mm, dst_addr, dst_pte, zero_pte);\n \tdouble_pt_unlock(dst_ptl, src_ptl);\n@@ -2135,6 +2221,15 @@ static bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags,\n \t !vma_is_anonymous(vma))\n \t\treturn false;\n \n+\t/*\n+\t * RWP uses protnone as an access-tracking marker. PROT_NONE VMAs\n+\t * have vm_page_prot == PAGE_NONE, so RWP resolution can't make a\n+\t * page accessible -- the next access would fault again. Reject up\n+\t * front instead of letting FOLL_FORCE loop on protnone+uffd PTEs.\n+\t */\n+\tif ((vm_flags \u0026 VM_UFFD_RWP) \u0026\u0026 !vma_is_accessible(vma))\n+\t\treturn false;\n+\n \treturn ops-\u003ecan_userfault(vma, vm_flags);\n }\n \n@@ -2187,9 +2282,22 @@ static struct vm_area_struct *userfaultfd_clear_vma(struct vma_iterator *vmi,\n \tif (start == vma-\u003evm_start \u0026\u0026 end == vma-\u003evm_end)\n \t\tgive_up_on_oom = true;\n \n-\t/* Reset ptes for the whole vma range if wr-protected */\n-\tif (userfaultfd_wp(vma))\n-\t\tuffd_wp_range(vma, start, end - start, false);\n+\t/* Clear the uffd bit and/or restore protnone PTEs */\n+\tif (userfaultfd_protected(vma)) {\n+\t\tunsigned int mm_cp_flags = 0;\n+\t\tstruct mmu_gather tlb;\n+\n+\t\tif (userfaultfd_wp(vma))\n+\t\t\tmm_cp_flags |= MM_CP_UFFD_WP_RESOLVE;\n+\t\tif (userfaultfd_rwp(vma))\n+\t\t\tmm_cp_flags |= MM_CP_UFFD_RWP_RESOLVE;\n+\t\tif (vma_wants_manual_pte_write_upgrade(vma))\n+\t\t\tmm_cp_flags |= MM_CP_TRY_CHANGE_WRITABLE;\n+\n+\t\ttlb_gather_mmu(\u0026tlb, vma-\u003evm_mm);\n+\t\tchange_protection(\u0026tlb, vma, start, end, mm_cp_flags);\n+\t\ttlb_finish_mmu(\u0026tlb);\n+\t}\n \n \tret = vma_modify_flags_uffd(vmi, prev, vma, start, end,\n \t\t\t\t \u0026new_vma_flags, NULL_VM_UFFD_CTX,\n@@ -2238,6 +2346,14 @@ static int userfaultfd_register_range(struct userfaultfd_ctx *ctx,\n \t\t vma_test_all_mask(vma, vma_flags))\n \t\t\tgoto skip;\n \n+\t\t/*\n+\t\t * Pre-scan in userfaultfd_register() already rejected mode\n+\t\t * switches that would drop VM_UFFD_WP or VM_UFFD_RWP, so a\n+\t\t * stray bit here is a bug.\n+\t\t */\n+\t\tVM_WARN_ON_ONCE(vma-\u003evm_userfaultfd_ctx.ctx == ctx \u0026\u0026\n+\t\t\t\tvma-\u003evm_flags \u0026 (VM_UFFD_WP | VM_UFFD_RWP) \u0026 ~vm_flags);\n+\n \t\tif (vma-\u003evm_start \u003e start)\n \t\t\tstart = vma-\u003evm_start;\n \t\tvma_end = min(end, vma-\u003evm_end);\n@@ -2368,14 +2484,29 @@ struct userfaultfd_wake_range {\n /* internal indication that UFFD_API ioctl was successfully executed */\n #define UFFD_FEATURE_INITIALIZED\t\t(1u \u003c\u003c 31)\n \n+/*\n+ * UFFDIO_SET_MODE updates ctx-\u003efeatures under mmap_write_lock with\n+ * WRITE_ONCE; readers that run outside mmap_read_lock or the per-VMA\n+ * lock (poll/read_iter/ioctl, fdinfo) must pair with READ_ONCE.\n+ */\n+static unsigned int userfaultfd_features(struct userfaultfd_ctx *ctx)\n+{\n+\treturn READ_ONCE(ctx-\u003efeatures);\n+}\n+\n static bool userfaultfd_is_initialized(struct userfaultfd_ctx *ctx)\n {\n-\treturn ctx-\u003efeatures \u0026 UFFD_FEATURE_INITIALIZED;\n+\treturn userfaultfd_features(ctx) \u0026 UFFD_FEATURE_INITIALIZED;\n }\n \n static bool userfaultfd_wp_async_ctx(struct userfaultfd_ctx *ctx)\n {\n-\treturn ctx \u0026\u0026 (ctx-\u003efeatures \u0026 UFFD_FEATURE_WP_ASYNC);\n+\treturn ctx \u0026\u0026 (userfaultfd_features(ctx) \u0026 UFFD_FEATURE_WP_ASYNC);\n+}\n+\n+static bool userfaultfd_rwp_async_ctx(struct userfaultfd_ctx *ctx)\n+{\n+\treturn ctx \u0026\u0026 (userfaultfd_features(ctx) \u0026 UFFD_FEATURE_RWP_ASYNC);\n }\n \n /*\n@@ -2390,7 +2521,7 @@ bool userfaultfd_wp_unpopulated(struct vm_area_struct *vma)\n \tif (!ctx)\n \t\treturn false;\n \n-\treturn ctx-\u003efeatures \u0026 UFFD_FEATURE_WP_UNPOPULATED;\n+\treturn userfaultfd_features(ctx) \u0026 UFFD_FEATURE_WP_UNPOPULATED;\n }\n \n static int userfaultfd_wake_function(wait_queue_entry_t *wq, unsigned mode,\n@@ -2504,6 +2635,8 @@ static inline struct uffd_msg userfault_msg(unsigned long address,\n \t\tmsg.arg.pagefault.flags |= UFFD_PAGEFAULT_FLAG_WRITE;\n \tif (reason \u0026 VM_UFFD_WP)\n \t\tmsg.arg.pagefault.flags |= UFFD_PAGEFAULT_FLAG_WP;\n+\tif (reason \u0026 VM_UFFD_RWP)\n+\t\tmsg.arg.pagefault.flags |= UFFD_PAGEFAULT_FLAG_RWP;\n \tif (reason \u0026 VM_UFFD_MINOR)\n \t\tmsg.arg.pagefault.flags |= UFFD_PAGEFAULT_FLAG_MINOR;\n \tif (features \u0026 UFFD_FEATURE_THREAD_ID)\n@@ -2557,6 +2690,12 @@ static inline bool userfaultfd_huge_must_wait(struct userfaultfd_ctx *ctx,\n \t */\n \tif (!huge_pte_write(pte) \u0026\u0026 (reason \u0026 VM_UFFD_WP))\n \t\treturn true;\n+\t/*\n+\t * PTE is still RW-protected (protnone with uffd bit), wait for\n+\t * resolution. Plain PROT_NONE without the marker is not an RWP fault.\n+\t */\n+\tif (pte_protnone(pte) \u0026\u0026 huge_pte_uffd(pte) \u0026\u0026 (reason \u0026 VM_UFFD_RWP))\n+\t\treturn true;\n \n \treturn false;\n }\n@@ -2617,8 +2756,14 @@ static inline bool userfaultfd_must_wait(struct userfaultfd_ctx *ctx,\n \tif (!pmd_present(_pmd))\n \t\treturn false;\n \n-\tif (pmd_trans_huge(_pmd))\n-\t\treturn !pmd_write(_pmd) \u0026\u0026 (reason \u0026 VM_UFFD_WP);\n+\tif (pmd_trans_huge(_pmd)) {\n+\t\tif (!pmd_write(_pmd) \u0026\u0026 (reason \u0026 VM_UFFD_WP))\n+\t\t\treturn true;\n+\t\tif (pmd_protnone(_pmd) \u0026\u0026 pmd_uffd(_pmd) \u0026\u0026\n+\t\t (reason \u0026 VM_UFFD_RWP))\n+\t\t\treturn true;\n+\t\treturn false;\n+\t}\n \n \tpte = pte_offset_map(pmd, address);\n \tif (!pte)\n@@ -2654,6 +2799,13 @@ static inline bool userfaultfd_must_wait(struct userfaultfd_ctx *ctx,\n \t */\n \tif (!pte_write(ptent) \u0026\u0026 (reason \u0026 VM_UFFD_WP))\n \t\tgoto out;\n+\t/*\n+\t * PTE is still RW-protected (protnone with uffd bit), wait for\n+\t * userspace to resolve. Plain PROT_NONE without the marker is not\n+\t * an RWP fault.\n+\t */\n+\tif (pte_protnone(ptent) \u0026\u0026 pte_uffd(ptent) \u0026\u0026 (reason \u0026 VM_UFFD_RWP))\n+\t\tgoto out;\n \n \tret = false;\n out:\n@@ -3598,11 +3750,27 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx,\n \tif (uffdio_register.mode \u0026 UFFDIO_REGISTER_MODE_MISSING)\n \t\tvm_flags |= VM_UFFD_MISSING;\n \tif (uffdio_register.mode \u0026 UFFDIO_REGISTER_MODE_WP) {\n-\t\tif (!pgtable_supports_uffd_wp())\n+\t\tif (!pgtable_supports_uffd())\n \t\t\tgoto out;\n \n \t\tvm_flags |= VM_UFFD_WP;\n \t}\n+\tif (uffdio_register.mode \u0026 UFFDIO_REGISTER_MODE_RWP) {\n+\t\tif (!pgtable_supports_uffd() || VM_UFFD_RWP == VM_NONE)\n+\t\t\tgoto out;\n+\t\tif (!(ctx-\u003efeatures \u0026 UFFD_FEATURE_RWP))\n+\t\t\tgoto out;\n+\t\tvm_flags |= VM_UFFD_RWP;\n+\t}\n+\n+\t/*\n+\t * WP and RWP share the uffd PTE bit and\n+\t * cannot coexist in the same VMA — the bit would carry ambiguous\n+\t * semantics. Reject the combination up front.\n+\t */\n+\tif ((vm_flags \u0026 VM_UFFD_WP) \u0026\u0026 (vm_flags \u0026 VM_UFFD_RWP))\n+\t\tgoto out;\n+\n \tif (uffdio_register.mode \u0026 UFFDIO_REGISTER_MODE_MINOR) {\n #ifndef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR\n \t\tgoto out;\n@@ -3696,6 +3864,16 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx,\n \t\t cur-\u003evm_userfaultfd_ctx.ctx != ctx)\n \t\t\tgoto out_unlock;\n \n+\t\t/*\n+\t\t * Mode switches that drop VM_UFFD_WP or VM_UFFD_RWP would\n+\t\t * leave PTE markers without the flag that describes them;\n+\t\t * subsequent mprotect() would then promote stale markers\n+\t\t * into the other mode. Require an unregister first.\n+\t\t */\n+\t\tif (cur-\u003evm_userfaultfd_ctx.ctx == ctx \u0026\u0026\n+\t\t cur-\u003evm_flags \u0026 (VM_UFFD_WP | VM_UFFD_RWP) \u0026 ~vm_flags)\n+\t\t\tgoto out_unlock;\n+\n \t\t/*\n \t\t * Note vmas containing huge pages\n \t\t */\n@@ -3729,6 +3907,10 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx,\n \t\tif (!(uffdio_register.mode \u0026 UFFDIO_REGISTER_MODE_MINOR))\n \t\t\tioctls_out \u0026= ~((__u64)1 \u003c\u003c _UFFDIO_CONTINUE);\n \n+\t\t/* RWPROTECT is only supported for RWP ranges */\n+\t\tif (!(uffdio_register.mode \u0026 UFFDIO_REGISTER_MODE_RWP))\n+\t\t\tioctls_out \u0026= ~((__u64)1 \u003c\u003c _UFFDIO_RWPROTECT);\n+\n \t\t/*\n \t\t * Now that we scanned all vmas we can already tell\n \t\t * userland which ioctls methods are guaranteed to\n@@ -4076,6 +4258,158 @@ static int userfaultfd_writeprotect(struct userfaultfd_ctx *ctx,\n \treturn ret;\n }\n \n+static int userfaultfd_rwprotect(struct userfaultfd_ctx *ctx,\n+\t\t\t\t unsigned long arg)\n+{\n+\tint ret;\n+\tstruct uffdio_rwprotect uffdio_rwp;\n+\tstruct userfaultfd_wake_range range;\n+\tbool mode_rwp, mode_dontwake;\n+\n+\tif (atomic_read(\u0026ctx-\u003emmap_changing))\n+\t\treturn -EAGAIN;\n+\n+\tif (copy_from_user(\u0026uffdio_rwp, (void __user *)arg,\n+\t\t\t sizeof(uffdio_rwp)))\n+\t\treturn -EFAULT;\n+\n+\tret = validate_range(ctx-\u003emm, uffdio_rwp.range.start,\n+\t\t\t uffdio_rwp.range.len);\n+\tif (ret)\n+\t\treturn ret;\n+\n+\tif (uffdio_rwp.mode \u0026 ~(UFFDIO_RWPROTECT_MODE_DONTWAKE |\n+\t\t\t\tUFFDIO_RWPROTECT_MODE_RWP))\n+\t\treturn -EINVAL;\n+\n+\tmode_rwp = uffdio_rwp.mode \u0026 UFFDIO_RWPROTECT_MODE_RWP;\n+\tmode_dontwake = uffdio_rwp.mode \u0026 UFFDIO_RWPROTECT_MODE_DONTWAKE;\n+\n+\tif (mode_rwp \u0026\u0026 mode_dontwake)\n+\t\treturn -EINVAL;\n+\n+\tif (mmget_not_zero(ctx-\u003emm)) {\n+\t\tret = mrwprotect_range(ctx, uffdio_rwp.range.start,\n+\t\t\t\t uffdio_rwp.range.len, mode_rwp);\n+\t\tmmput(ctx-\u003emm);\n+\t} else {\n+\t\treturn -ESRCH;\n+\t}\n+\n+\tif (ret)\n+\t\treturn ret;\n+\n+\tif (!mode_rwp \u0026\u0026 !mode_dontwake) {\n+\t\trange.start = uffdio_rwp.range.start;\n+\t\trange.len = uffdio_rwp.range.len;\n+\t\twake_userfault(ctx, \u0026range);\n+\t}\n+\treturn ret;\n+}\n+\n+/* Subset of UFFD_API_FEATURES actually supported by this kernel/arch */\n+static __u64 uffd_api_available_features(void)\n+{\n+\t__u64 f = UFFD_API_FEATURES;\n+\n+\tif (!IS_ENABLED(CONFIG_HAVE_ARCH_USERFAULTFD_MINOR))\n+\t\tf \u0026= ~(UFFD_FEATURE_MINOR_HUGETLBFS | UFFD_FEATURE_MINOR_SHMEM);\n+\tif (!pgtable_supports_uffd())\n+\t\tf \u0026= ~UFFD_FEATURE_PAGEFAULT_FLAG_WP;\n+\tif (!uffd_supports_wp_marker())\n+\t\tf \u0026= ~(UFFD_FEATURE_WP_HUGETLBFS_SHMEM |\n+\t\t UFFD_FEATURE_WP_UNPOPULATED |\n+\t\t UFFD_FEATURE_WP_ASYNC);\n+\t/*\n+\t * RWP needs both PROT_NONE support and the uffd PTE bit. The\n+\t * VM_UFFD_RWP check covers compile-time unavailability; the\n+\t * pgtable_supports_uffd() check covers runtime (e.g. riscv\n+\t * without the SVRSW60T59B extension) where the PTE bit is declared\n+\t * but not actually usable.\n+\t */\n+\tif (VM_UFFD_RWP == VM_NONE || !pgtable_supports_uffd())\n+\t\tf \u0026= ~(UFFD_FEATURE_RWP | UFFD_FEATURE_RWP_ASYNC);\n+\treturn f;\n+}\n+\n+/* Async features that can be toggled at runtime via UFFDIO_SET_MODE */\n+#define UFFD_FEATURE_TOGGLEABLE\tUFFD_FEATURE_RWP_ASYNC\n+\n+static int userfaultfd_set_mode(struct userfaultfd_ctx *ctx,\n+\t\t\t\tunsigned long arg)\n+{\n+\tstruct uffdio_set_mode mode;\n+\tstruct mm_struct *mm = ctx-\u003emm;\n+\n+\tif (copy_from_user(\u0026mode, (void __user *)arg, sizeof(mode)))\n+\t\treturn -EFAULT;\n+\n+\t/* enable and disable must not overlap */\n+\tif (mode.enable \u0026 mode.disable)\n+\t\treturn -EINVAL;\n+\n+\t/* only toggleable features that this kernel/arch actually supports */\n+\tif ((mode.enable | mode.disable) \u0026\n+\t ~(uffd_api_available_features() \u0026 UFFD_FEATURE_TOGGLEABLE))\n+\t\treturn -EINVAL;\n+\n+\t/* RWP_ASYNC can only be enabled on contexts that negotiated RWP */\n+\tif ((mode.enable \u0026 UFFD_FEATURE_RWP_ASYNC) \u0026\u0026\n+\t !(userfaultfd_features(ctx) \u0026 UFFD_FEATURE_RWP))\n+\t\treturn -EINVAL;\n+\n+\tif (!mmget_not_zero(mm))\n+\t\treturn -ESRCH;\n+\n+\t/*\n+\t * Drain in-flight faults before flipping features. mmap_write_lock()\n+\t * blocks new mmap_read_lock() callers, but per-VMA locked faults\n+\t * (lock_vma_under_rcu() + FAULT_FLAG_VMA_LOCK) that acquired before\n+\t * this point keep running. Calling vma_start_write() on each UFFD-\n+\t * armed VMA waits for those readers to drop, so no in-flight fault\n+\t * can observe the old features after mmap_write_unlock().\n+\t */\n+\tmmap_write_lock(mm);\n+\t{\n+\t\tstruct vm_area_struct *vma;\n+\t\tVMA_ITERATOR(vmi, mm, 0);\n+\n+\t\tfor_each_vma(vmi, vma) {\n+\t\t\tif (vma-\u003evm_userfaultfd_ctx.ctx == ctx)\n+\t\t\t\tvma_start_write(vma);\n+\t\t}\n+\t}\n+\t/*\n+\t * Single WRITE_ONCE so lockless readers (fdinfo, poll/read_iter\n+\t * via userfaultfd_is_initialized(), and the userfaultfd_features()\n+\t * helper used elsewhere) can't observe a mid-RMW intermediate\n+\t * value. Hot-path readers already serialise through the mmap lock\n+\t * + vma_start_write() drain above, so their load doesn't need an\n+\t * annotation.\n+\t */\n+\tWRITE_ONCE(ctx-\u003efeatures,\n+\t\t (ctx-\u003efeatures | mode.enable) \u0026 ~mode.disable);\n+\tmmap_write_unlock(mm);\n+\n+\t/*\n+\t * If switching to async, wake threads blocked in handle_userfault().\n+\t * They will retry the fault and auto-resolve under the new mode.\n+\t * len=0 means wake all pending faults on this context.\n+\t */\n+\tif (mode.enable \u0026 UFFD_FEATURE_RWP_ASYNC) {\n+\t\tstruct userfaultfd_wake_range range = { .len = 0 };\n+\n+\t\tspin_lock_irq(\u0026ctx-\u003efault_pending_wqh.lock);\n+\t\t__wake_up_locked_key(\u0026ctx-\u003efault_pending_wqh, TASK_NORMAL,\n+\t\t\t\t \u0026range);\n+\t\t__wake_up(\u0026ctx-\u003efault_wqh, TASK_NORMAL, 1, \u0026range);\n+\t\tspin_unlock_irq(\u0026ctx-\u003efault_pending_wqh.lock);\n+\t}\n+\n+\tmmput(mm);\n+\treturn 0;\n+}\n+\n static int userfaultfd_continue(struct userfaultfd_ctx *ctx, unsigned long arg)\n {\n \t__s64 ret;\n@@ -4199,6 +4533,11 @@ bool userfaultfd_wp_async(struct vm_area_struct *vma)\n \treturn userfaultfd_wp_async_ctx(vma-\u003evm_userfaultfd_ctx.ctx);\n }\n \n+bool userfaultfd_rwp_async(struct vm_area_struct *vma)\n+{\n+\treturn userfaultfd_rwp_async_ctx(vma-\u003evm_userfaultfd_ctx.ctx);\n+}\n+\n static inline unsigned int uffd_ctx_features(__u64 user_features)\n {\n \t/*\n@@ -4302,20 +4641,14 @@ static int userfaultfd_api(struct userfaultfd_ctx *ctx,\n \tif (features \u0026 UFFD_FEATURE_WP_ASYNC)\n \t\tfeatures |= UFFD_FEATURE_WP_UNPOPULATED;\n \n-\t/* report all available features and ioctls to userland */\n-\tuffdio_api.features = UFFD_API_FEATURES;\n-#ifndef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR\n-\tuffdio_api.features \u0026=\n-\t\t~(UFFD_FEATURE_MINOR_HUGETLBFS | UFFD_FEATURE_MINOR_SHMEM);\n-#endif\n-\tif (!pgtable_supports_uffd_wp())\n-\t\tuffdio_api.features \u0026= ~UFFD_FEATURE_PAGEFAULT_FLAG_WP;\n+\tret = -EINVAL;\n+\t/* RWP_ASYNC requires RWP */\n+\tif ((features \u0026 UFFD_FEATURE_RWP_ASYNC) \u0026\u0026\n+\t !(features \u0026 UFFD_FEATURE_RWP))\n+\t\tgoto err_out;\n \n-\tif (!uffd_supports_wp_marker()) {\n-\t\tuffdio_api.features \u0026= ~UFFD_FEATURE_WP_HUGETLBFS_SHMEM;\n-\t\tuffdio_api.features \u0026= ~UFFD_FEATURE_WP_UNPOPULATED;\n-\t\tuffdio_api.features \u0026= ~UFFD_FEATURE_WP_ASYNC;\n-\t}\n+\t/* report all available features and ioctls to userland */\n+\tuffdio_api.features = uffd_api_available_features();\n \n \tret = -EINVAL;\n \tif (features \u0026 ~uffdio_api.features)\n@@ -4382,6 +4715,12 @@ static long userfaultfd_ioctl(struct file *file, unsigned cmd,\n \tcase UFFDIO_POISON:\n \t\tret = userfaultfd_poison(ctx, arg);\n \t\tbreak;\n+\tcase UFFDIO_RWPROTECT:\n+\t\tret = userfaultfd_rwprotect(ctx, arg);\n+\t\tbreak;\n+\tcase UFFDIO_SET_MODE:\n+\t\tret = userfaultfd_set_mode(ctx, arg);\n+\t\tbreak;\n \t}\n \treturn ret;\n }\n@@ -4409,7 +4748,7 @@ static void userfaultfd_show_fdinfo(struct seq_file *m, struct file *f)\n \t *\tprotocols: aa:... bb:...\n \t */\n \tseq_printf(m, \"pending:\\t%lu\\ntotal:\\t%lu\\nAPI:\\t%Lx:%x:%Lx\\n\",\n-\t\t pending, total, UFFD_API, ctx-\u003efeatures,\n+\t\t pending, total, UFFD_API, userfaultfd_features(ctx),\n \t\t UFFD_API_IOCTLS|UFFD_API_RANGE_IOCTLS);\n }\n #endif\ndiff --git a/tools/include/uapi/linux/fs.h b/tools/include/uapi/linux/fs.h\nindex 24ddf7bc4f25f0..f0a26309b6d547 100644\n--- a/tools/include/uapi/linux/fs.h\n+++ b/tools/include/uapi/linux/fs.h\n@@ -364,6 +364,7 @@ typedef int __bitwise __kernel_rwf_t;\n #define PAGE_IS_HUGE\t\t(1 \u003c\u003c 6)\n #define PAGE_IS_SOFT_DIRTY\t(1 \u003c\u003c 7)\n #define PAGE_IS_GUARD\t\t(1 \u003c\u003c 8)\n+#define PAGE_IS_ACCESSED\t(1 \u003c\u003c 9)\n \n /*\n * struct page_region - Page region with flags\ndiff --git a/tools/testing/selftests/mm/uffd-unit-tests.c b/tools/testing/selftests/mm/uffd-unit-tests.c\nindex a6c14109e81887..8c49191835ea1d 100644\n--- a/tools/testing/selftests/mm/uffd-unit-tests.c\n+++ b/tools/testing/selftests/mm/uffd-unit-tests.c\n@@ -7,6 +7,8 @@\n \n #include \"uffd-common.h\"\n \n+#include \u003clinux/fs.h\u003e\n+#include \u003csys/uio.h\u003e\n #include \"../../../../mm/gup_test.h\"\n \n #ifdef __NR_userfaultfd\n@@ -109,6 +111,10 @@ static void uffd_test_skip(const char *message)\n \n static void test_uffd_api(bool use_dev)\n {\n+\tconst uint64_t expected_ioctls =\n+\t\tBIT_ULL(_UFFDIO_REGISTER) |\n+\t\tBIT_ULL(_UFFDIO_UNREGISTER) |\n+\t\tBIT_ULL(_UFFDIO_API);\n \tstruct uffdio_api uffdio_api;\n \tint uffd;\n \n@@ -148,6 +154,15 @@ static void test_uffd_api(bool use_dev)\n \t\tgoto out;\n \t}\n \n+\t/* Verify returned fd-level ioctls bitmask */\n+\tif ((uffdio_api.ioctls \u0026 expected_ioctls) != expected_ioctls) {\n+\t\tuffd_test_fail(\"UFFDIO_API missing expected ioctls: \"\n+\t\t\t \"got=0x%\"PRIx64\", expected=0x%\"PRIx64,\n+\t\t\t (uint64_t)uffdio_api.ioctls,\n+\t\t\t expected_ioctls);\n+\t\tgoto out;\n+\t}\n+\n \t/* Test double requests of UFFDIO_API with a random feature set */\n \tuffdio_api.features = BIT_ULL(0);\n \tif (ioctl(uffd, UFFDIO_API, \u0026uffdio_api) == 0) {\n@@ -602,6 +617,687 @@ void uffd_minor_collapse_test(uffd_global_test_opts_t *gopts, uffd_test_args_t *\n \tuffd_minor_test_common(gopts, true, false);\n }\n \n+static int uffd_register_rwp(int uffd, void *addr, uint64_t len)\n+{\n+\tstruct uffdio_register reg = {\n+\t\t.range = { .start = (unsigned long)addr, .len = len },\n+\t\t.mode = UFFDIO_REGISTER_MODE_RWP,\n+\t};\n+\n+\tif (ioctl(uffd, UFFDIO_REGISTER, \u0026reg) == -1)\n+\t\treturn -errno;\n+\treturn 0;\n+}\n+\n+static void rwprotect_range(int uffd, __u64 start, __u64 len, bool protect)\n+{\n+\tstruct uffdio_rwprotect rwp = {\n+\t\t.range = { .start = start, .len = len },\n+\t\t.mode = protect ? UFFDIO_RWPROTECT_MODE_RWP : 0,\n+\t};\n+\n+\tif (ioctl(uffd, UFFDIO_RWPROTECT, \u0026rwp))\n+\t\terr(\"UFFDIO_RWPROTECT failed\");\n+}\n+\n+static void set_async_mode(int uffd, bool enable)\n+{\n+\tstruct uffdio_set_mode mode = { };\n+\n+\tif (enable)\n+\t\tmode.enable = UFFD_FEATURE_RWP_ASYNC;\n+\telse\n+\t\tmode.disable = UFFD_FEATURE_RWP_ASYNC;\n+\n+\tif (ioctl(uffd, UFFDIO_SET_MODE, \u0026mode))\n+\t\terr(\"UFFDIO_SET_MODE failed\");\n+}\n+\n+/*\n+ * Test async RWP faults on anonymous memory.\n+ * Populate pages, register MODE_RWP with RWP_ASYNC,\n+ * RW-protect, re-access, verify content preserved and no faults delivered.\n+ */\n+static void uffd_rwp_async_test(uffd_global_test_opts_t *gopts,\n+\t\t\t\t uffd_test_args_t *args)\n+{\n+\tunsigned long nr_pages = gopts-\u003enr_pages;\n+\tunsigned long page_size = gopts-\u003epage_size;\n+\tunsigned long p;\n+\n+\t/* Populate all pages with known content */\n+\tfor (p = 0; p \u003c nr_pages; p++)\n+\t\tmemset(gopts-\u003earea_dst + p * page_size, p % 255 + 1, page_size);\n+\n+\t/* Register MODE_RWP */\n+\tif (uffd_register_rwp(gopts-\u003euffd, gopts-\u003earea_dst,\n+\t\t\t nr_pages * page_size))\n+\t\terr(\"register failure\");\n+\n+\t/* RW-protect all pages (sets protnone) */\n+\trwprotect_range(gopts-\u003euffd, (uint64_t)gopts-\u003earea_dst,\n+\t\t\t nr_pages * page_size, true);\n+\n+\t/* Access all pages — should auto-resolve, no faults */\n+\tfor (p = 0; p \u003c nr_pages; p++) {\n+\t\tunsigned char *page = (unsigned char *)gopts-\u003earea_dst +\n+\t\t\t\t p * page_size;\n+\t\tunsigned char expected = p % 255 + 1;\n+\n+\t\tif (page[0] != expected) {\n+\t\t\tuffd_test_fail(\"page %lu content mismatch: %u != %u\",\n+\t\t\t\t p, page[0], expected);\n+\t\t\treturn;\n+\t\t}\n+\t}\n+\n+\tuffd_test_pass();\n+}\n+\n+/*\n+ * Fault handler for RWP — unprotect the page via UFFDIO_RWPROTECT.\n+ */\n+static void uffd_handle_rwp_fault(uffd_global_test_opts_t *gopts,\n+\t\t\t\t struct uffd_msg *msg,\n+\t\t\t\t struct uffd_args *uargs)\n+{\n+\tif (!(msg-\u003earg.pagefault.flags \u0026 UFFD_PAGEFAULT_FLAG_RWP))\n+\t\terr(\"expected RWP fault, got 0x%llx\",\n+\t\t msg-\u003earg.pagefault.flags);\n+\n+\trwprotect_range(gopts-\u003euffd, msg-\u003earg.pagefault.address,\n+\t\t\tgopts-\u003epage_size, false);\n+\tuargs-\u003eminor_faults++;\n+}\n+\n+/*\n+ * Test sync RWP faults on anonymous memory.\n+ * Populate pages, register MODE_RWP (sync), RW-protect,\n+ * access from worker thread, verify fault delivered, UFFDIO_RWPROTECT resolves.\n+ */\n+static void uffd_rwp_sync_test(uffd_global_test_opts_t *gopts,\n+\t\t\t\t uffd_test_args_t *args)\n+{\n+\tunsigned long nr_pages = gopts-\u003enr_pages;\n+\tunsigned long page_size = gopts-\u003epage_size;\n+\tpthread_t uffd_mon;\n+\tstruct uffd_args uargs = { };\n+\tbool failed = false;\n+\tchar c = '\\0';\n+\tunsigned long p;\n+\n+\tuargs.gopts = gopts;\n+\tuargs.handle_fault = uffd_handle_rwp_fault;\n+\n+\t/* Populate all pages */\n+\tfor (p = 0; p \u003c nr_pages; p++)\n+\t\tmemset(gopts-\u003earea_dst + p * page_size, p % 255 + 1, page_size);\n+\n+\t/* Register MODE_RWP */\n+\tif (uffd_register_rwp(gopts-\u003euffd, gopts-\u003earea_dst,\n+\t\t\t nr_pages * page_size))\n+\t\terr(\"register failure\");\n+\n+\t/* RW-protect all pages */\n+\trwprotect_range(gopts-\u003euffd, (uint64_t)gopts-\u003earea_dst,\n+\t\t\t nr_pages * page_size, true);\n+\n+\t/* Start fault handler thread */\n+\tif (pthread_create(\u0026uffd_mon, NULL, uffd_poll_thread, \u0026uargs))\n+\t\terr(\"uffd_poll_thread create\");\n+\n+\t/* Access all pages — triggers sync RWP faults, handler unprotects */\n+\tfor (p = 0; p \u003c nr_pages; p++) {\n+\t\tunsigned char *page = (unsigned char *)gopts-\u003earea_dst +\n+\t\t\t\t p * page_size;\n+\n+\t\tif (page[0] != (p % 255 + 1)) {\n+\t\t\tuffd_test_fail(\"page %lu content mismatch\", p);\n+\t\t\tfailed = true;\n+\t\t\tgoto out;\n+\t\t}\n+\t}\n+\n+out:\n+\t/*\n+\t * Stop the handler before reading minor_faults: the last fault\n+\t * resolution rwprotect_range()s before incrementing the counter,\n+\t * so the main thread can race ahead of the increment.\n+\t */\n+\tif (write(gopts-\u003epipefd[1], \u0026c, sizeof(c)) != sizeof(c))\n+\t\terr(\"pipe write\");\n+\tif (pthread_join(uffd_mon, NULL))\n+\t\terr(\"join() failed\");\n+\n+\tif (failed)\n+\t\treturn;\n+\tif (uargs.minor_faults == 0)\n+\t\tuffd_test_fail(\"expected RWP faults, got 0\");\n+\telse\n+\t\tuffd_test_pass();\n+}\n+\n+/*\n+ * Test PAGEMAP_SCAN working-set discovery via the \"hot\" (accessed) scan.\n+ *\n+ * The working-set primitive is to find pages that were accessed: scan for\n+ * PAGE_IS_ACCESSED, which is set once an access clears the protnone+uffd\n+ * marker. A VMM treats every access fault as \"hot\" (RWP here; MINOR/MISSING\n+ * for non-resident pages) and reclaims the rest from the backing file.\n+ *\n+ * We deliberately do NOT use an inverted \"cold\" scan: that only sees\n+ * VMA-resident ptes, so for a file mapping it misses cached-but-unmapped (and\n+ * never-faulted, pre-populated) pages, which are pte_none and thus invisible.\n+ * Hot tracking + file-level reclaim covers them; a cold pte scan cannot.\n+ */\n+static void uffd_rwp_pagemap_test(uffd_global_test_opts_t *gopts,\n+\t\t\t\t\t uffd_test_args_t *args)\n+{\n+\tunsigned long nr_pages = gopts-\u003enr_pages;\n+\tunsigned long page_size = gopts-\u003epage_size;\n+\tunsigned long p;\n+\tstruct page_region regions[16];\n+\tstruct pm_scan_arg pm_arg;\n+\tint pagemap_fd;\n+\tlong ret;\n+\n+\t/* Need at least 4 pages */\n+\tif (nr_pages \u003c 4) {\n+\t\tuffd_test_skip(\"need at least 4 pages\");\n+\t\treturn;\n+\t}\n+\n+\t/* Populate all pages */\n+\tfor (p = 0; p \u003c nr_pages; p++)\n+\t\tmemset(gopts-\u003earea_dst + p * page_size, 0xab, page_size);\n+\n+\t/* Register and RW-protect */\n+\tif (uffd_register_rwp(gopts-\u003euffd, gopts-\u003earea_dst,\n+\t\t\t nr_pages * page_size))\n+\t\terr(\"register failure\");\n+\n+\trwprotect_range(gopts-\u003euffd, (uint64_t)gopts-\u003earea_dst,\n+\t\t\t nr_pages * page_size, true);\n+\n+\t/* Touch first half of pages to re-activate them (async auto-resolve) */\n+\tfor (p = 0; p \u003c nr_pages / 2; p++) {\n+\t\tvolatile char *page = gopts-\u003earea_dst + p * page_size;\n+\t\t(void)*page;\n+\t}\n+\n+\tuint64_t start = (uint64_t)gopts-\u003earea_dst;\n+\tuint64_t boundary = start + (nr_pages / 2) * page_size;\n+\tuint64_t end = start + nr_pages * page_size;\n+\n+\tpagemap_fd = open(\"/proc/self/pagemap\", O_RDONLY);\n+\tif (pagemap_fd \u003c 0)\n+\t\terr(\"open pagemap\");\n+\n+\t/*\n+\t * Hot scan: report the pages that were accessed. PAGE_IS_ACCESSED is\n+\t * set once the protnone+uffd marker is cleared (by the access, async\n+\t * auto-resolve here). The touched first half must come back as exactly\n+\t * one hot region [start, boundary); the untouched second half must not\n+\t * appear.\n+\t */\n+\tmemset(\u0026pm_arg, 0, sizeof(pm_arg));\n+\tpm_arg.size = sizeof(pm_arg);\n+\tpm_arg.start = start;\n+\tpm_arg.end = end;\n+\tpm_arg.vec = (uint64_t)regions;\n+\tpm_arg.vec_len = ARRAY_SIZE(regions);\n+\tpm_arg.category_mask = PAGE_IS_ACCESSED;\n+\tpm_arg.return_mask = PAGE_IS_ACCESSED;\n+\n+\tret = ioctl(pagemap_fd, PAGEMAP_SCAN, \u0026pm_arg);\n+\tclose(pagemap_fd);\n+\n+\tif (ret \u003c 0) {\n+\t\tuffd_test_fail(\"PAGEMAP_SCAN failed: %s\", strerror(errno));\n+\t\treturn;\n+\t}\n+\n+\tif (ret != 1 || regions[0].start != start ||\n+\t regions[0].end != boundary) {\n+\t\tuffd_test_fail(\"hot set wrong: got %ld regions [0x%lx,0x%lx), expected 1 [0x%lx,0x%lx)\",\n+\t\t\t ret, (unsigned long)regions[0].start,\n+\t\t\t (unsigned long)regions[0].end,\n+\t\t\t (unsigned long)start, (unsigned long)boundary);\n+\t\treturn;\n+\t}\n+\n+\tuffd_test_pass();\n+}\n+\n+/*\n+ * Test that RWP protection survives a mprotect(PROT_NONE) -\u003e\n+ * mprotect(PROT_READ|PROT_WRITE) round-trip. The uffd-wp bit on a\n+ * VM_UFFD_RWP VMA must continue to carry PROT_NONE semantics after\n+ * mprotect() changes the base protection; otherwise accesses would\n+ * silently succeed and the pagemap bit would stick without a fault\n+ * ever clearing it.\n+ */\n+static void uffd_rwp_mprotect_test(uffd_global_test_opts_t *gopts,\n+\t\t\t\t uffd_test_args_t *args)\n+{\n+\tunsigned long nr_pages = gopts-\u003enr_pages;\n+\tunsigned long page_size = gopts-\u003epage_size;\n+\tunsigned long p;\n+\tstruct page_region regions[16];\n+\tstruct pm_scan_arg pm_arg;\n+\tint pagemap_fd;\n+\tlong ret;\n+\n+\t/* Populate all pages */\n+\tfor (p = 0; p \u003c nr_pages; p++)\n+\t\tmemset(gopts-\u003earea_dst + p * page_size, 0xab, page_size);\n+\n+\t/* Register and RW-protect the whole range */\n+\tif (uffd_register_rwp(gopts-\u003euffd, gopts-\u003earea_dst,\n+\t\t\t nr_pages * page_size))\n+\t\terr(\"register failure\");\n+\trwprotect_range(gopts-\u003euffd, (uint64_t)gopts-\u003earea_dst,\n+\t\t\tnr_pages * page_size, true);\n+\n+\t/* Round-trip mprotect(): PROT_NONE -\u003e PROT_READ|PROT_WRITE */\n+\tif (mprotect(gopts-\u003earea_dst, nr_pages * page_size, PROT_NONE))\n+\t\terr(\"mprotect() PROT_NONE\");\n+\tif (mprotect(gopts-\u003earea_dst, nr_pages * page_size,\n+\t\t PROT_READ | PROT_WRITE))\n+\t\terr(\"mprotect() PROT_READ|PROT_WRITE\");\n+\n+\t/* Touch every page. Async RWP must auto-resolve each fault. */\n+\tfor (p = 0; p \u003c nr_pages; p++) {\n+\t\tvolatile char *page = gopts-\u003earea_dst + p * page_size;\n+\t\t(void)*page;\n+\t}\n+\n+\t/*\n+\t * After touching, no page should remain RW-protected. A stuck\n+\t * uffd-wp bit would mean mprotect() silently dropped PROT_NONE and\n+\t * the access never faulted.\n+\t */\n+\tpagemap_fd = open(\"/proc/self/pagemap\", O_RDONLY);\n+\tif (pagemap_fd \u003c 0)\n+\t\terr(\"open pagemap\");\n+\n+\tmemset(\u0026pm_arg, 0, sizeof(pm_arg));\n+\tpm_arg.size = sizeof(pm_arg);\n+\tpm_arg.start = (uint64_t)gopts-\u003earea_dst;\n+\tpm_arg.end = (uint64_t)gopts-\u003earea_dst + nr_pages * page_size;\n+\tpm_arg.vec = (uint64_t)regions;\n+\tpm_arg.vec_len = ARRAY_SIZE(regions);\n+\tpm_arg.category_mask = PAGE_IS_ACCESSED;\n+\tpm_arg.category_inverted = PAGE_IS_ACCESSED;\n+\tpm_arg.return_mask = PAGE_IS_ACCESSED;\n+\n+\tret = ioctl(pagemap_fd, PAGEMAP_SCAN, \u0026pm_arg);\n+\tclose(pagemap_fd);\n+\n+\tif (ret \u003c 0) {\n+\t\tuffd_test_fail(\"PAGEMAP_SCAN failed: %s\", strerror(errno));\n+\t\treturn;\n+\t}\n+\tif (ret != 0) {\n+\t\tuffd_test_fail(\"expected no cold pages after mprotect()+touch, got %ld regions\",\n+\t\t\t ret);\n+\t\treturn;\n+\t}\n+\n+\tuffd_test_pass();\n+}\n+\n+/*\n+ * Test that GUP resolves through protnone PTEs (async mode).\n+ * vmsplice() into a pipe pins user pages via get_user_pages_fast() --\n+ * unlike write(), which goes through copy_from_user() and ordinary\n+ * hardware page faults -- so it exercises gup_can_follow_protnone() on\n+ * the RW-protected PTE. In async mode the kernel auto-restores\n+ * permissions and GUP returns the page.\n+ */\n+static void uffd_rwp_gup_test(uffd_global_test_opts_t *gopts,\n+\t\t\t\t uffd_test_args_t *args)\n+{\n+\tstruct iovec iov;\n+\tchar buf;\n+\tint pipefd[2];\n+\n+\t/* Populate first page with known content */\n+\tmemset(gopts-\u003earea_dst, 0xCD, gopts-\u003epage_size);\n+\n+\tif (uffd_register_rwp(gopts-\u003euffd, gopts-\u003earea_dst, gopts-\u003epage_size))\n+\t\terr(\"register failure\");\n+\n+\trwprotect_range(gopts-\u003euffd, (uint64_t)gopts-\u003earea_dst,\n+\t\t\tgopts-\u003epage_size, true);\n+\n+\tif (pipe(pipefd))\n+\t\terr(\"pipe\");\n+\n+\t/*\n+\t * One byte's worth of iov is enough to GUP the containing page and\n+\t * keeps the pipe transfer well under any pipe-capacity limit even on\n+\t * hugetlb-backed runs.\n+\t */\n+\tiov.iov_base = gopts-\u003earea_dst;\n+\tiov.iov_len = 1;\n+\tif (vmsplice(pipefd[1], \u0026iov, 1, 0) != 1) {\n+\t\tuffd_test_fail(\"vmsplice from RW-protected page failed: %s\",\n+\t\t\t strerror(errno));\n+\t\tgoto out;\n+\t}\n+\n+\tif (read(pipefd[0], \u0026buf, 1) != 1) {\n+\t\tuffd_test_fail(\"read from pipe failed\");\n+\t\tgoto out;\n+\t}\n+\n+\tif (buf != (char)0xCD) {\n+\t\tuffd_test_fail(\"content mismatch: got 0x%02x, expected 0xCD\",\n+\t\t\t (unsigned char)buf);\n+\t\tgoto out;\n+\t}\n+\n+\tuffd_test_pass();\n+out:\n+\tclose(pipefd[0]);\n+\tclose(pipefd[1]);\n+}\n+\n+/*\n+ * Test runtime toggle between async and sync modes.\n+ * Start in async mode (detection), flip to sync (eviction), verify faults\n+ * block, resolve them, flip back to async.\n+ */\n+static void uffd_rwp_async_toggle_test(uffd_global_test_opts_t *gopts,\n+\t\t\t\t\t uffd_test_args_t *args)\n+{\n+\tunsigned long nr_pages = gopts-\u003enr_pages;\n+\tunsigned long page_size = gopts-\u003epage_size;\n+\tstruct uffd_args uargs = { };\n+\tpthread_t uffd_mon;\n+\tchar c = '\\0';\n+\tunsigned long p;\n+\n+\tuargs.gopts = gopts;\n+\tuargs.handle_fault = uffd_handle_rwp_fault;\n+\n+\t/* Populate */\n+\tfor (p = 0; p \u003c nr_pages; p++)\n+\t\tmemset(gopts-\u003earea_dst + p * page_size, p % 255 + 1, page_size);\n+\n+\tif (uffd_register_rwp(gopts-\u003euffd, gopts-\u003earea_dst,\n+\t\t\t nr_pages * page_size))\n+\t\terr(\"register failure\");\n+\n+\t/* Phase 1: async detection — RW-protect, access first half */\n+\trwprotect_range(gopts-\u003euffd, (uint64_t)gopts-\u003earea_dst,\n+\t\t\t nr_pages * page_size, true);\n+\n+\tfor (p = 0; p \u003c nr_pages / 2; p++) {\n+\t\tvolatile char *page = gopts-\u003earea_dst + p * page_size;\n+\t\t(void)*page; /* auto-resolves in async mode */\n+\t}\n+\n+\t/* Phase 2: flip to sync for eviction */\n+\tset_async_mode(gopts-\u003euffd, false);\n+\n+\t/* Start handler — will receive faults for cold pages */\n+\tif (pthread_create(\u0026uffd_mon, NULL, uffd_poll_thread, \u0026uargs))\n+\t\terr(\"uffd_poll_thread create\");\n+\n+\t/* Access second half (cold pages) — should trigger sync faults */\n+\tfor (p = nr_pages / 2; p \u003c nr_pages; p++) {\n+\t\tunsigned char *page = (unsigned char *)gopts-\u003earea_dst +\n+\t\t\t\t p * page_size;\n+\t\tif (page[0] != (p % 255 + 1)) {\n+\t\t\tuffd_test_fail(\"page %lu content mismatch\", p);\n+\t\t\tgoto out;\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * Stop the handler before reading minor_faults: the last fault\n+\t * resolution rwprotect_range()s before incrementing the counter,\n+\t * so the main thread can race ahead of the increment. Stopping\n+\t * here also makes Phase 3 a clean async-only test -- with the\n+\t * handler still running it would silently resolve any sync fault\n+\t * the kernel erroneously delivers, masking a regression.\n+\t */\n+\tif (write(gopts-\u003epipefd[1], \u0026c, sizeof(c)) != sizeof(c))\n+\t\terr(\"pipe write\");\n+\tif (pthread_join(uffd_mon, NULL))\n+\t\terr(\"join() failed\");\n+\n+\tif (uargs.minor_faults == 0) {\n+\t\tuffd_test_fail(\"expected sync faults, got 0\");\n+\t\treturn;\n+\t}\n+\n+\t/* Phase 3: flip back to async */\n+\tset_async_mode(gopts-\u003euffd, true);\n+\n+\t/* RW-protect and access again — should auto-resolve */\n+\trwprotect_range(gopts-\u003euffd, (uint64_t)gopts-\u003earea_dst,\n+\t\t\t nr_pages * page_size, true);\n+\n+\tfor (p = 0; p \u003c nr_pages; p++) {\n+\t\tvolatile char *page = gopts-\u003earea_dst + p * page_size;\n+\t\t(void)*page;\n+\t}\n+\n+\tuffd_test_pass();\n+\treturn;\n+out:\n+\tif (write(gopts-\u003epipefd[1], \u0026c, sizeof(c)) != sizeof(c))\n+\t\terr(\"pipe write\");\n+\tif (pthread_join(uffd_mon, NULL))\n+\t\terr(\"join() failed\");\n+}\n+\n+/*\n+ * Test that RW-protected pages become accessible after closing uffd.\n+ */\n+static void uffd_rwp_close_test(uffd_global_test_opts_t *gopts,\n+\t\t\t\t uffd_test_args_t *args)\n+{\n+\tunsigned long nr_pages = gopts-\u003enr_pages;\n+\tunsigned long page_size = gopts-\u003epage_size;\n+\tunsigned long p;\n+\n+\t/* Populate */\n+\tfor (p = 0; p \u003c nr_pages; p++)\n+\t\tmemset(gopts-\u003earea_dst + p * page_size, p % 255 + 1, page_size);\n+\n+\tif (uffd_register_rwp(gopts-\u003euffd, gopts-\u003earea_dst,\n+\t\t\t nr_pages * page_size))\n+\t\terr(\"register failure\");\n+\n+\trwprotect_range(gopts-\u003euffd, (uint64_t)gopts-\u003earea_dst,\n+\t\t\t nr_pages * page_size, true);\n+\n+\t/* Close uffd — should restore protnone PTEs */\n+\tclose(gopts-\u003euffd);\n+\tgopts-\u003euffd = -1;\n+\n+\t/* All pages should be accessible with original content */\n+\tfor (p = 0; p \u003c nr_pages; p++) {\n+\t\tunsigned char *page = (unsigned char *)gopts-\u003earea_dst +\n+\t\t\t\t p * page_size;\n+\t\tunsigned char expected = p % 255 + 1;\n+\n+\t\tif (page[0] != expected) {\n+\t\t\tuffd_test_fail(\"page %lu not accessible after close\", p);\n+\t\t\treturn;\n+\t\t}\n+\t}\n+\n+\tuffd_test_pass();\n+}\n+\n+/*\n+ * Test that RWP protection is preserved across fork() when\n+ * UFFD_FEATURE_EVENT_FORK is enabled. Without preservation, the child's\n+ * PTEs would lose the uffd-wp marker and RWP-protected accesses would\n+ * silently fall through to do_numa_page().\n+ */\n+static void uffd_rwp_fork_test(uffd_global_test_opts_t *gopts,\n+\t\t\t uffd_test_args_t *args)\n+{\n+\tunsigned long nr_pages = gopts-\u003enr_pages;\n+\tunsigned long page_size = gopts-\u003epage_size;\n+\tint pagemap_fd;\n+\tuint64_t value;\n+\n+\tif (uffd_register_rwp(gopts-\u003euffd, gopts-\u003earea_dst,\n+\t\t\t nr_pages * page_size))\n+\t\terr(\"register failed\");\n+\n+\t/* Populate + RWP-protect */\n+\t*gopts-\u003earea_dst = 1;\n+\trwprotect_range(gopts-\u003euffd, (uint64_t)gopts-\u003earea_dst,\n+\t\t\tpage_size, true);\n+\n+\t/* Parent: verify uffd-wp bit is set before fork */\n+\tpagemap_fd = pagemap_open();\n+\tvalue = pagemap_get_entry(pagemap_fd, gopts-\u003earea_dst);\n+\tpagemap_check_wp(value, true);\n+\n+\t/*\n+\t * Fork with EVENT_FORK: child inherits VM_UFFD_RWP. Child reads\n+\t * its own pagemap and must still see the uffd-wp bit set.\n+\t */\n+\tif (pagemap_test_fork(gopts, true, false)) {\n+\t\tuffd_test_fail(\"RWP marker lost in child after fork\");\n+\t\tgoto out;\n+\t}\n+\n+\tuffd_test_pass();\n+out:\n+\tclose(pagemap_fd);\n+}\n+\n+/*\n+ * Test that RWP protection on a pinned anon page is preserved across fork().\n+ * Pinning forces copy_present_page() in the child path, which must restore\n+ * PAGE_NONE on top of the uffd bit. Using async mode, a read in the child\n+ * auto-resolves if — and only if — the PTE was actually protnone+uffd; the\n+ * cleared uffd bit afterward proves the fault path ran.\n+ */\n+static void uffd_rwp_fork_pin_test(uffd_global_test_opts_t *gopts,\n+\t\t\t\t uffd_test_args_t *args)\n+{\n+\tunsigned long page_size = gopts-\u003epage_size;\n+\tfork_event_args fevent_args = { .gopts = gopts, .child_uffd = -1 };\n+\tpin_args pin_args = {};\n+\tint pagemap_fd, status;\n+\tpthread_t fevent_thread;\n+\tuint64_t value;\n+\tpid_t child;\n+\n+\tif (uffd_register_rwp(gopts-\u003euffd, gopts-\u003earea_dst, page_size))\n+\t\terr(\"register failed\");\n+\n+\t/* Populate. */\n+\t*gopts-\u003earea_dst = 1;\n+\n+\t/* RO-longterm pin so fork() takes copy_present_page() for this PTE. */\n+\tif (pin_pages(\u0026pin_args, gopts-\u003earea_dst, page_size)) {\n+\t\tuffd_test_skip(\"Possibly CONFIG_GUP_TEST missing or unprivileged\");\n+\t\tuffd_unregister(gopts-\u003euffd, gopts-\u003earea_dst, page_size);\n+\t\treturn;\n+\t}\n+\n+\t/* RWP-protect: PTE is now PAGE_NONE + uffd bit. */\n+\trwprotect_range(gopts-\u003euffd, (uint64_t)gopts-\u003earea_dst, page_size, true);\n+\n+\tpagemap_fd = pagemap_open();\n+\tvalue = pagemap_get_entry(pagemap_fd, gopts-\u003earea_dst);\n+\tpagemap_check_wp(value, true);\n+\n+\t/*\n+\t * UFFD_FEATURE_EVENT_FORK is required so the child inherits\n+\t * VM_UFFD_RWP and the marker; without it dup_userfaultfd() resets\n+\t * the child VMA and the test would pass for the wrong reason.\n+\t * dup_userfaultfd() blocks until the EVENT_FORK message is consumed,\n+\t * so spawn a reader before the fork().\n+\t */\n+\tgopts-\u003eready_for_fork = false;\n+\tif (pthread_create(\u0026fevent_thread, NULL, fork_event_consumer,\n+\t\t\t \u0026fevent_args))\n+\t\terr(\"pthread_create() for fork event consumer\");\n+\twhile (!gopts-\u003eready_for_fork)\n+\t\t; /* Wait for consumer to start polling. */\n+\n+\tchild = fork();\n+\tif (child \u003c 0)\n+\t\terr(\"fork\");\n+\tif (child == 0) {\n+\t\tvolatile char c;\n+\t\tint cfd;\n+\n+\t\t/*\n+\t\t * Read the pinned page. Only reaches the fault path if the\n+\t\t * child PTE is protnone + uffd; async mode auto-resolves and\n+\t\t * clears the uffd bit. If copy_present_page() dropped\n+\t\t * PAGE_NONE, the read would silently succeed and the bit\n+\t\t * would still be set.\n+\t\t */\n+\t\tc = *(volatile char *)gopts-\u003earea_dst;\n+\t\t(void)c;\n+\n+\t\tcfd = pagemap_open();\n+\t\tvalue = pagemap_get_entry(cfd, gopts-\u003earea_dst);\n+\t\tclose(cfd);\n+\t\t_exit((value \u0026 PM_UFFD_WP) ? 1 : 0);\n+\t}\n+\tif (waitpid(child, \u0026status, 0) \u003c 0)\n+\t\terr(\"waitpid\");\n+\tif (pthread_join(fevent_thread, NULL))\n+\t\terr(\"pthread_join() for fork event consumer\");\n+\tif (fevent_args.child_uffd \u003e= 0)\n+\t\tclose(fevent_args.child_uffd);\n+\n+\tunpin_pages(\u0026pin_args);\n+\tclose(pagemap_fd);\n+\tif (uffd_unregister(gopts-\u003euffd, gopts-\u003earea_dst, page_size))\n+\t\terr(\"unregister failed\");\n+\n+\tif (!WIFEXITED(status) || WEXITSTATUS(status) != 0) {\n+\t\tuffd_test_fail(\"RWP not enforced in child after pinned fork\");\n+\t\treturn;\n+\t}\n+\n+\tuffd_test_pass();\n+}\n+\n+/*\n+ * WP and RWP share the uffd-wp PTE bit and cannot coexist in the same VMA.\n+ * Registration requesting both modes must be rejected.\n+ */\n+static void uffd_rwp_wp_exclusive_test(uffd_global_test_opts_t *gopts,\n+\t\t\t\t uffd_test_args_t *args)\n+{\n+\tunsigned long nr_pages = gopts-\u003enr_pages;\n+\tunsigned long page_size = gopts-\u003epage_size;\n+\tstruct uffdio_register reg = { };\n+\n+\treg.range.start = (unsigned long)gopts-\u003earea_dst;\n+\treg.range.len = nr_pages * page_size;\n+\treg.mode = UFFDIO_REGISTER_MODE_WP | UFFDIO_REGISTER_MODE_RWP;\n+\n+\tif (ioctl(gopts-\u003euffd, UFFDIO_REGISTER, \u0026reg) == 0) {\n+\t\tuffd_test_fail(\"register with WP|RWP unexpectedly succeeded\");\n+\t\treturn;\n+\t}\n+\tif (errno != EINVAL) {\n+\t\tuffd_test_fail(\"register with WP|RWP: expected EINVAL, got %d\",\n+\t\t\t errno);\n+\t\treturn;\n+\t}\n+\tuffd_test_pass();\n+}\n+\n static sigjmp_buf jbuf, *sigbuf;\n \n static void sighndl(int sig, siginfo_t *siginfo, void *ptr)\n@@ -1604,6 +2300,77 @@ uffd_test_case_t uffd_tests[] = {\n \t\t/* We can't test MADV_COLLAPSE, so try our luck */\n \t\t.uffd_feature_required = UFFD_FEATURE_MINOR_SHMEM,\n \t},\n+\t{\n+\t\t.name = \"rwp-async\",\n+\t\t.uffd_fn = uffd_rwp_async_test,\n+\t\t.mem_targets = MEM_ALL,\n+\t\t.uffd_feature_required =\n+\t\tUFFD_FEATURE_RWP | UFFD_FEATURE_RWP_ASYNC,\n+\t},\n+\t{\n+\t\t.name = \"rwp-sync\",\n+\t\t.uffd_fn = uffd_rwp_sync_test,\n+\t\t.mem_targets = MEM_ALL,\n+\t\t.uffd_feature_required = UFFD_FEATURE_RWP,\n+\t},\n+\t{\n+\t\t.name = \"rwp-pagemap\",\n+\t\t.uffd_fn = uffd_rwp_pagemap_test,\n+\t\t.mem_targets = MEM_ALL,\n+\t\t.uffd_feature_required =\n+\t\tUFFD_FEATURE_RWP | UFFD_FEATURE_RWP_ASYNC,\n+\t},\n+\t{\n+\t\t.name = \"rwp-mprotect\",\n+\t\t.uffd_fn = uffd_rwp_mprotect_test,\n+\t\t.mem_targets = MEM_ALL,\n+\t\t.uffd_feature_required =\n+\t\tUFFD_FEATURE_RWP | UFFD_FEATURE_RWP_ASYNC,\n+\t},\n+\t{\n+\t\t.name = \"rwp-gup\",\n+\t\t.uffd_fn = uffd_rwp_gup_test,\n+\t\t.mem_targets = MEM_ALL,\n+\t\t.uffd_feature_required =\n+\t\tUFFD_FEATURE_RWP | UFFD_FEATURE_RWP_ASYNC,\n+\t},\n+\t{\n+\t\t.name = \"rwp-async-toggle\",\n+\t\t.uffd_fn = uffd_rwp_async_toggle_test,\n+\t\t.mem_targets = MEM_ALL,\n+\t\t.uffd_feature_required =\n+\t\tUFFD_FEATURE_RWP | UFFD_FEATURE_RWP_ASYNC,\n+\t},\n+\t{\n+\t\t.name = \"rwp-close\",\n+\t\t.uffd_fn = uffd_rwp_close_test,\n+\t\t.mem_targets = MEM_ALL,\n+\t\t.uffd_feature_required = UFFD_FEATURE_RWP,\n+\t},\n+\t{\n+\t\t.name = \"rwp-fork\",\n+\t\t.uffd_fn = uffd_rwp_fork_test,\n+\t\t.mem_targets = MEM_ALL,\n+\t\t.uffd_feature_required =\n+\t\tUFFD_FEATURE_RWP | UFFD_FEATURE_EVENT_FORK,\n+\t},\n+\t{\n+\t\t.name = \"rwp-fork-pin\",\n+\t\t.uffd_fn = uffd_rwp_fork_pin_test,\n+\t\t.mem_targets = MEM_ANON,\n+\t\t.uffd_feature_required =\n+\t\tUFFD_FEATURE_RWP | UFFD_FEATURE_RWP_ASYNC |\n+\t\tUFFD_FEATURE_EVENT_FORK,\n+\t},\n+\t{\n+\t\t.name = \"rwp-wp-exclusive\",\n+\t\t.uffd_fn = uffd_rwp_wp_exclusive_test,\n+\t\t.mem_targets = MEM_ALL,\n+\t\t.uffd_feature_required =\n+\t\tUFFD_FEATURE_RWP |\n+\t\tUFFD_FEATURE_PAGEFAULT_FLAG_WP |\n+\t\tUFFD_FEATURE_WP_HUGETLBFS_SHMEM,\n+\t},\n \t{\n \t\t.name = \"sigbus\",\n \t\t.uffd_fn = uffd_sigbus_test,\n@@ -1814,6 +2581,20 @@ int main(int argc, char *argv[])\n \t\t\t\tuffd_test_skip(errmsg);\n \t\t\t\tcontinue;\n \t\t\t}\n+\t\t\t/*\n+\t\t\t * RWP tracks protection on ptes; a THP-backed shmem/anon\n+\t\t\t * range (e.g. shmem_enabled=always) would split on\n+\t\t\t * rwprotect and change behaviour under the test. Keep\n+\t\t\t * such ranges off THP. hugetlb is huge by definition and\n+\t\t\t * rejects MADV_NOHUGEPAGE, so skip it.\n+\t\t\t */\n+\t\t\tif ((test-\u003euffd_feature_required \u0026 UFFD_FEATURE_RWP) \u0026\u0026\n+\t\t\t !(mem_type-\u003emem_flag \u0026 (MEM_HUGETLB | MEM_HUGETLB_PRIVATE))) {\n+\t\t\t\tunsigned long len = gopts.nr_pages * gopts.page_size;\n+\n+\t\t\t\tif (madvise(gopts.area_dst, len, MADV_NOHUGEPAGE))\n+\t\t\t\t\terr(\"madvise(MADV_NOHUGEPAGE)\");\n+\t\t\t}\n \t\t\ttest-\u003euffd_fn(\u0026gopts, \u0026args);\n \t\t\tuffd_test_ctx_clear(\u0026gopts);\n \t\t}\n"
} |
| 2/1 | 2026/06/29 12:30 | agent | patch-evaluator |
0mModel:gemini-3.1-pro-preview Args: null Results: {
"EnableConfigs": [
"USERFAULTFD_RWP"
],
"FocusSymbols": [
"userfaultfd_rwprotect",
"mrwprotect_range",
"userfaultfd_set_mode",
"do_uffd_rwp",
"do_huge_pmd_uffd_rwp"
],
"Reasoning": "The patch introduces a new userfaultfd feature, Read-Write Protection (RWP), which adds new ioctls (UFFDIO_RWPROTECT, UFFDIO_SET_MODE) and modifies page fault handling to support RWP. This is a significant functional change to core memory management and userfaultfd that should be fuzzed.",
"WorthFuzzing": true
}Instruction: You are an expert Linux kernel maintainer. Your job is to review a provided patch series and determine if it makes functional changes to the kernel that should be fuzzed. IMPORTANT: The changes have ALREADY been applied and committed as the HEAD commit in your workspace. Do NOT rely on your internal knowledge of the kernel. You must actively use your code access tools to examine the actual source code and confirm any assumptions. Return WorthFuzzing=false if the patch only contains: - Modifications to Documentation/, Kconfig files, or code comments. - Purely decorative changes, such as logging (e.g., pr_err, printk) or tracepoints. - Changes to numeric constants or macros that do not functionally alter execution flow. - Code paths that are impossible to reach in virtualized environments like GCE or QEMU, even when utilizing software-emulated hardware (e.g., usb gadget, mac80211_hwsim). If it modifies reachable core kernel logic, drivers, or architectures, use your code search tools to verify the code can be executed, then return WorthFuzzing=true. When returning WorthFuzzing=true, you MUST ALSO: 1. Extract any specific kernel functions that should be heavily fuzzed into FocusSymbols. Avoid listing generic hot-path functions to prevent skewed test distributions. 2. Identify any specific CONFIG_ options required to properly test this new/modified feature. Go and look into the Kconfig files and check for ifdefs around the code, do not make assumptions. Do not list too generic configs (we already have them enabled). Only list those that specifically cover the modified code. List them in the EnableConfigs output array, and DO NOT add a 'CONFIG_' prefix (e.g., return "NET_IPV4" instead of "CONFIG_NET_IPV4"). Prefer calling several tools at the same time to save round-trips. Use set-results tool to provide results of the analysis. It must be called exactly once before the final reply. Ignore results of this tool. Prompt: For your convenience, here is the diff of the changes:
commit 09db1b5e82f6242632e83646e40234035529896b
Author: syz-cluster <triage@syzkaller.com>
Date: Mon Jun 29 12:30:51 2026 +0000
syz-cluster: applied patch under review
diff --git a/Documentation/admin-guide/mm/pagemap.rst b/Documentation/admin-guide/mm/pagemap.rst
index c57e61b5d8aa81..ffa690a171c862 100644
--- a/Documentation/admin-guide/mm/pagemap.rst
+++ b/Documentation/admin-guide/mm/pagemap.rst
@@ -19,8 +19,11 @@ There are four components to pagemap:
* Bit 55 pte is soft-dirty (see
Documentation/admin-guide/mm/soft-dirty.rst)
* Bit 56 page exclusively mapped (since 4.2)
- * Bit 57 pte is uffd-wp write-protected (since 5.13) (see
- Documentation/admin-guide/mm/userfaultfd.rst)
+ * Bit 57 pte is tracked by userfaultfd (since 5.13) — in a
+ ``VM_UFFD_WP`` VMA this indicates a write-protected PTE; in a
+ ``VM_UFFD_RWP`` VMA it indicates an RWP-protected PTE. WP and
+ RWP are mutually exclusive per VMA, so the meaning is
+ unambiguous. See Documentation/admin-guide/mm/userfaultfd.rst.
* Bit 58 pte is a guard region (since 6.15) (see madvise (2) man page)
* Bits 59-60 zero
* Bit 61 page is file-page or shared-anon (since 3.5)
@@ -244,7 +247,8 @@ in this IOCTL:
Following flags about pages are currently supported:
- ``PAGE_IS_WPALLOWED`` - Page has async-write-protection enabled
-- ``PAGE_IS_WRITTEN`` - Page has been written to from the time it was write protected
+- ``PAGE_IS_WRITTEN`` - Page in a ``UFFDIO_REGISTER_MODE_WP`` VMA has been
+ written to since it was write-protected. Only reported inside such VMAs.
- ``PAGE_IS_FILE`` - Page is file backed
- ``PAGE_IS_PRESENT`` - Page is present in the memory
- ``PAGE_IS_SWAPPED`` - Page is in swapped
@@ -252,6 +256,9 @@ Following flags about pages are currently supported:
- ``PAGE_IS_HUGE`` - Page is PMD-mapped THP or Hugetlb backed
- ``PAGE_IS_SOFT_DIRTY`` - Page is soft-dirty
- ``PAGE_IS_GUARD`` - Page is a part of a guard region
+- ``PAGE_IS_ACCESSED`` - Page in a ``UFFDIO_REGISTER_MODE_RWP`` VMA has been
+ accessed since RWP was applied. Only reported inside such VMAs. See
+ Documentation/admin-guide/mm/userfaultfd.rst for the RWP workflow.
The ``struct pm_scan_arg`` is used as the argument of the IOCTL.
diff --git a/Documentation/admin-guide/mm/userfaultfd.rst b/Documentation/admin-guide/mm/userfaultfd.rst
index e5cc8848dcb39f..69836e39aa0b64 100644
--- a/Documentation/admin-guide/mm/userfaultfd.rst
+++ b/Documentation/admin-guide/mm/userfaultfd.rst
@@ -131,6 +131,16 @@ userfaults on the range registered. Not all ioctls will necessarily be
supported for all memory types (e.g. anonymous memory vs. shmem vs.
hugetlbfs), or all types of intercepted faults.
+.. note::
+
+ Re-registering an already-registered range must not drop any of the
+ modes that install per-PTE markers — currently
+ ``UFFDIO_REGISTER_MODE_WP`` and ``UFFDIO_REGISTER_MODE_RWP``. Doing
+ so would strand markers with no flag to describe them, so the call
+ is rejected with ``-EBUSY``; userspace must issue
+ ``UFFDIO_UNREGISTER`` first. This differs from older kernels, which
+ silently replaced the mode bits on re-registration.
+
Userland can use the ``uffdio_register.ioctls`` to manage the virtual
address space in the background (to add or potentially also remove
memory from the ``userfaultfd`` registered range). This means a userfault
@@ -265,16 +275,16 @@ tracking and it can be different in a few ways:
- Dirty information will not get lost if the pte was zapped due to
various reasons (e.g. during split of a shmem transparent huge page).
- - Due to a reverted meaning of soft-dirty (page clean when uffd-wp bit
- set; dirty when uffd-wp bit cleared), it has different semantics on
- some of the memory operations. For example: ``MADV_DONTNEED`` on
+ - Due to a reverted meaning of soft-dirty (page clean when the uffd bit
+ is set; dirty when the uffd bit is cleared), it has different semantics
+ on some of the memory operations. For example: ``MADV_DONTNEED`` on
anonymous (or ``MADV_REMOVE`` on a file mapping) will be treated as
- dirtying of memory by dropping uffd-wp bit during the procedure.
+ dirtying of memory by dropping the uffd bit during the procedure.
The user app can collect the "written/dirty" status by looking up the
-uffd-wp bit for the pages being interested in /proc/pagemap.
+uffd bit for the pages being interested in /proc/pagemap.
-The page will not be under track of uffd-wp async mode until the page is
+The page will not be under track of userfaultfd-wp async mode until the page is
explicitly write-protected by ``ioctl(UFFDIO_WRITEPROTECT)`` with the mode
flag ``UFFDIO_WRITEPROTECT_MODE_WP`` set. Trying to resolve a page fault
that was tracked by async mode userfaultfd-wp is invalid.
@@ -297,6 +307,253 @@ transparent to the guest, we want that same address range to act as if it was
still poisoned, even though it's on a new physical host which ostensibly
doesn't have a memory error in the exact same spot.
+Read-Write Protection
+---------------------
+
+``UFFDIO_REGISTER_MODE_RWP`` enables read-write protection tracking on a
+memory range. It is similar to (but faster than) ``mprotect(PROT_NONE)``
+combined with a signal handler; unlike ``mprotect(PROT_NONE)``, RWP only
+traps accesses to *present* PTEs, so accesses to unpopulated addresses in a
+protected range fall through to the normal missing-page path. It uses the
+PROT_NONE hinting mechanism (same as NUMA balancing) to make pages
+inaccessible while keeping them resident in memory. Works on anonymous,
+shmem, and hugetlbfs memory.
+
+RWP is designed for VM memory managers that need to track the working set
+of guest memory for cold page eviction to tiered or remote storage.
+
+**Setup:**
+
+1. Open a userfaultfd and enable ``UFFD_FEATURE_RWP`` via ``UFFDIO_API``.
+ Optionally request ``UFFD_FEATURE_RWP_ASYNC`` as well — it requires
+ ``UFFD_FEATURE_RWP`` to be set in the same ``UFFDIO_API`` call.
+
+2. Register the guest memory range with ``UFFDIO_REGISTER_MODE_RWP``
+ (and ``UFFDIO_REGISTER_MODE_MISSING`` if evicted pages will need to be
+ fetched back from storage).
+
+**Feature availability:**
+
+RWP is built on top of two kernel primitives: a spare PTE bit owned by
+userfaultfd (``CONFIG_HAVE_ARCH_USERFAULTFD_WP``) and architecture support
+for present-but-inaccessible PTEs (``CONFIG_ARCH_HAS_PTE_PROTNONE``). When both
+are available on a 64-bit kernel, the build selects
+``CONFIG_USERFAULTFD_RWP=y`` and the ``VM_UFFD_RWP`` VMA flag becomes
+available.
+
+``UFFD_FEATURE_RWP`` and ``UFFD_FEATURE_RWP_ASYNC`` are unavailable when
+the running kernel or architecture does not support them — for example
+32-bit kernels (where ``VM_UFFD_RWP`` is unavailable), kernels built
+without ``CONFIG_USERFAULTFD_RWP``, and architectures whose ptes cannot
+carry the uffd bit at runtime (e.g. riscv without the ``SVRSW60T59B``
+extension). Requesting an unsupported feature in
+``uffdio_api.features`` makes ``UFFDIO_API`` fail with ``EINVAL`` and
+leaves the userfaultfd context uninitialized; the bitmask returned in
+``uffdio_api.features`` then advertises the features the kernel does
+support. The recommended probe sequence is therefore to open a
+throwaway userfaultfd, call ``UFFDIO_API`` once with ``features = 0``,
+inspect the returned bitmask, close that fd, then open the real one
+and call ``UFFDIO_API`` again with only the supported features set.
+
+**Protecting and Unprotecting:**
+
+Use ``UFFDIO_RWPROTECT`` to protect or unprotect a range, mirroring the
+``UFFDIO_WRITEPROTECT`` interface::
+
+ struct uffdio_rwprotect rwp = {
+ .range = { .start = addr, .len = len },
+ .mode = UFFDIO_RWPROTECT_MODE_RWP, /* protect */
+ };
+ ioctl(uffd, UFFDIO_RWPROTECT, &rwp);
+
+Setting ``UFFDIO_RWPROTECT_MODE_RWP`` sets PROT_NONE on present PTEs in the
+range. Pages stay resident and their physical frames are preserved — only
+access permissions are removed.
+
+Clearing ``UFFDIO_RWPROTECT_MODE_RWP`` restores normal VMA permissions and
+wakes any faulting threads (unless ``UFFDIO_RWPROTECT_MODE_DONTWAKE`` is set).
+
+**Scope of protection:**
+
+RWP protection is a property of *present* PTEs. ``UFFDIO_RWPROTECT`` only
+affects entries that are already populated. Unpopulated addresses within
+the range remain unpopulated; when first accessed they fault through the
+normal missing path (``do_anonymous_page()``, ``do_swap_page()``,
+``finish_fault()``) and the resulting PTE is not RWP-protected. To observe
+the population itself, co-register the range with
+``UFFDIO_REGISTER_MODE_MISSING``.
+
+Protection is preserved across page reclaim: a page swapped out while
+RWP-protected carries the marker on its swap entry, and swap-in restores
+the PROT_NONE state so the first access after swap-in still faults. The
+same applies to pages temporarily replaced by migration entries.
+
+Operations that drop the PTE entirely — ``MADV_DONTNEED`` on anonymous
+memory, hole-punch on shmem, truncation of a file mapping — also drop the
+RWP marker: the next access re-populates the range without protection.
+Unlike WP (which persists via ``PTE_MARKER_UFFD_WP``), there is no
+persistent RWP marker today. The user needs to re-arm the range with
+``UFFDIO_RWPROTECT`` after any operation that explicitly frees PTEs.
+
+**Fault Handling:**
+
+When a protected page is accessed:
+
+- **Sync mode** (default): The faulting thread blocks and a
+ ``UFFD_PAGEFAULT_FLAG_RWP`` message is delivered to the userfaultfd
+ handler. The handler resolves the fault with ``UFFDIO_RWPROTECT``
+ (clearing ``MODE_RWP``), which restores the PTE permissions and wakes
+ the faulting thread.
+
+- **Async mode** (``UFFD_FEATURE_RWP_ASYNC``): The kernel automatically
+ restores PTE permissions and the thread continues without blocking. No
+ message is delivered to the handler.
+
+**Runtime Mode Switching:**
+
+``UFFDIO_SET_MODE`` toggles ``UFFD_FEATURE_RWP_ASYNC`` at runtime, allowing
+the VMM to switch between lightweight async detection and safe sync
+eviction without re-registering. The toggle takes ``mmap_write_lock()``
+and calls ``vma_start_write()`` on each UFFD-armed VMA, draining
+in-flight per-VMA-locked faults before the new mode takes effect.
+
+**Working-set detection with PAGEMAP_SCAN:**
+
+RWP-protected PTEs carry the uffd PTE bit; an access (and, in async mode, its
+auto-resolution) clears it. ``PAGEMAP_SCAN`` reports ``PAGE_IS_ACCESSED`` once
+the bit is clear on a ``VM_UFFD_RWP`` VMA, so a *non-inverted* scan reports the
+pages that were touched during the interval -- the hot set::
+
+ struct pm_scan_arg arg = {
+ .size = sizeof(arg),
+ .start = guest_mem_start,
+ .end = guest_mem_end,
+ .vec = (uint64_t)regions,
+ .vec_len = regions_len,
+ .category_mask = PAGE_IS_ACCESSED,
+ .return_mask = PAGE_IS_ACCESSED,
+ };
+ long n = ioctl(pagemap_fd, PAGEMAP_SCAN, &arg);
+
+The returned ``page_region`` array lists the hot ranges. ``PAGE_IS_ACCESSED``
+is set on an accessed page whether it is still present or has since been
+swapped out, so the hot scan needs no ``PAGE_IS_PRESENT`` filter -- unpopulated
+holes carry neither bit and are excluded on their own.
+
+Track the hot set and reclaim everything else from the backing file (see the
+workflow below). Do **not** invert the scan to enumerate "cold" pages
+directly: an inverted scan reports only the ``VM_UFFD_RWP`` PTEs that are still
+protected, i.e. the resident portion of *this* VMA. For a file mapping the
+working set spans the whole file -- pages that live in the page cache but are
+not mapped into this VMA (a pre-populated tmpfs file, or memory populated
+through another mapping) are ``pte_none`` here, never appear in the scan, and
+would never be considered for eviction even though they occupy memory. Driving
+eviction from "file offsets minus the hot set" avoids that blind spot; a cold
+PTE scan cannot. To additionally record the *first* access to a cached but
+unmapped page (e.g. pre-populated content) as hot, co-register the range with
+``UFFDIO_REGISTER_MODE_MINOR``: such accesses then fault as minor faults
+instead of mapping the page silently.
+
+**Cleanup:**
+
+When the userfaultfd is closed or the range is unregistered, all PROT_NONE
+PTEs are automatically restored to their normal VMA permissions. This
+prevents pages from becoming permanently inaccessible.
+
+**VMM Working Set Tracking Workflow:**
+
+A typical VMM lifecycle for cold page eviction to tiered storage. Two
+mappings of the same shmem (or hugetlbfs) file are used: ``guest_mem`` is
+the RWP-registered mapping that vCPUs access through, and ``io_mem`` is a
+private mapping for VMM-side I/O. Reading ``io_mem`` does not go through
+the RWP-protected PTEs of ``guest_mem``, so the VMM's own ``pwrite()``
+never traps on its own ::
+
+ /* One-time setup */
+ fd = memfd_create("guest", MFD_CLOEXEC);
+ ftruncate(fd, guest_size);
+ guest_mem = mmap(NULL, guest_size, PROT_READ | PROT_WRITE,
+ MAP_SHARED, fd, 0); /* vCPU view, RWP-registered */
+ io_mem = mmap(NULL, guest_size, PROT_READ | PROT_WRITE,
+ MAP_SHARED, fd, 0); /* VMM I/O view, unprotected */
+
+ uffd = userfaultfd(O_CLOEXEC | O_NONBLOCK);
+ struct uffdio_api api = {
+ .api = UFFD_API,
+ .features = UFFD_FEATURE_RWP | UFFD_FEATURE_RWP_ASYNC,
+ };
+ ioctl(uffd, UFFDIO_API, &api);
+ if (!(api.features & UFFD_FEATURE_RWP))
+ /* RWP unavailable on this kernel/arch -- fall back. */
+ ioctl(uffd, UFFDIO_REGISTER, &(struct uffdio_register){
+ .range = { guest_mem, guest_size },
+ .mode = UFFDIO_REGISTER_MODE_RWP |
+ UFFDIO_REGISTER_MODE_MISSING,
+ });
+
+ /* Tracking loop */
+ while (vm_running) {
+ /* 1. Detection phase (async -- no vCPU stalls) */
+ ioctl(uffd, UFFDIO_RWPROTECT, &(struct uffdio_rwprotect){
+ .range = full_range,
+ .mode = UFFDIO_RWPROTECT_MODE_RWP });
+ sleep(tracking_interval);
+
+ /*
+ * 2. Switch to sync BEFORE scanning. In async mode a vCPU
+ * access races eviction: it would auto-resolve and mark the
+ * page hot just as the VMM writes it out and punches it,
+ * losing the update. Sync mode makes such accesses block and
+ * be delivered, freezing the hot snapshot for the rest of the
+ * iteration.
+ */
+ ioctl(uffd, UFFDIO_SET_MODE,
+ &(struct uffdio_set_mode){
+ .disable = UFFD_FEATURE_RWP_ASYNC });
+
+ /* 3. Read the hot set: pages touched this interval. */
+ ioctl(pagemap_fd, PAGEMAP_SCAN, &(struct pm_scan_arg){
+ .category_mask = PAGE_IS_ACCESSED,
+ .return_mask = PAGE_IS_ACCESSED,
+ ...
+ });
+
+ /*
+ * 4. Reclaim the file offsets that are NOT in the hot set.
+ * Driving this from the file's offset space (rather than from a
+ * cold PTE scan) also reclaims pages that are cached but not
+ * mapped into guest_mem, e.g. pre-populated content.
+ */
+ for each non-hot offset range:
+ /* Read from io_mem -- bypasses RWP, no fault. */
+ pwrite(storage_fd, (char *)io_mem + off, len, off);
+ /* Drop the page from the shared file. */
+ fallocate(fd, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE,
+ off, len);
+ /*
+ * Wake any vCPU blocked on the RWP fault for this range:
+ * fallocate() does not iterate ctx->fault_pending_wqh.
+ */
+ ioctl(uffd, UFFDIO_WAKE, &(struct uffdio_range){
+ .start = (uintptr_t)guest_mem + off, .len = len });
+
+ /* 5. Resume async tracking */
+ ioctl(uffd, UFFDIO_SET_MODE,
+ &(struct uffdio_set_mode){
+ .enable = UFFD_FEATURE_RWP_ASYNC });
+ }
+
+During step 4, a vCPU that accesses a ``guest_mem`` offset being evicted
+blocks with a ``UFFD_PAGEFAULT_FLAG_RWP`` fault while the eviction is in
+progress. After ``fallocate()`` punches the page out and ``UFFDIO_WAKE``
+fires, the vCPU retries the access, faults as ``MISSING``, and the
+handler resolves it with ``UFFDIO_COPY`` from storage.
+
+This workflow targets shmem and hugetlbfs (both support a private
+``io_mem`` mapping over the same fd). Anonymous-memory backings need a
+different inner-loop strategy because the VMM has no way to read the
+page without going through the RWP-protected mapping.
+
QEMU/KVM
========
diff --git a/Documentation/filesystems/proc.rst b/Documentation/filesystems/proc.rst
index 2ccd5b2dfdd33a..27189f6e004eed 100644
--- a/Documentation/filesystems/proc.rst
+++ b/Documentation/filesystems/proc.rst
@@ -608,6 +608,7 @@ encoded manner. The codes are the following:
um userfaultfd missing tracking
uw userfaultfd wr-protect tracking
ui userfaultfd minor fault
+ ur userfaultfd read-write-protect tracking
ss shadow/guarded control stack page
sl sealed
lf lock on fault pages
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index b3afe0688919bd..6e123adca29b92 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -78,6 +78,7 @@ config ARM64
select ARCH_SUPPORTS_CFI
select ARCH_SUPPORTS_ATOMIC_RMW
select ARCH_SUPPORTS_INT128 if CC_HAS_INT128
+ select ARCH_HAS_PTE_PROTNONE
select ARCH_SUPPORTS_NUMA_BALANCING
select ARCH_SUPPORTS_PAGE_TABLE_CHECK
select ARCH_SUPPORTS_PER_VMA_LOCK
diff --git a/arch/arm64/include/asm/pgtable-prot.h b/arch/arm64/include/asm/pgtable-prot.h
index 212ce1b02e15ed..09d7c00cf40541 100644
--- a/arch/arm64/include/asm/pgtable-prot.h
+++ b/arch/arm64/include/asm/pgtable-prot.h
@@ -28,11 +28,11 @@
#define PTE_PRESENT_VALID_KERNEL (PTE_VALID | PTE_MAYBE_NG)
#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP
-#define PTE_UFFD_WP (_AT(pteval_t, 1) << 58) /* uffd-wp tracking */
-#define PTE_SWP_UFFD_WP (_AT(pteval_t, 1) << 3) /* only for swp ptes */
+#define PTE_UFFD (_AT(pteval_t, 1) << 58) /* userfaultfd tracking */
+#define PTE_SWP_UFFD (_AT(pteval_t, 1) << 3) /* only for swp ptes */
#else
-#define PTE_UFFD_WP (_AT(pteval_t, 0))
-#define PTE_SWP_UFFD_WP (_AT(pteval_t, 0))
+#define PTE_UFFD (_AT(pteval_t, 0))
+#define PTE_SWP_UFFD (_AT(pteval_t, 0))
#endif /* CONFIG_HAVE_ARCH_USERFAULTFD_WP */
#define _PROT_DEFAULT (PTE_TYPE_PAGE | PTE_AF | PTE_SHARED)
diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index 27689c62bd25dd..c49a553ad12fbe 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -341,19 +341,19 @@ static inline pmd_t pmd_mknoncont(pmd_t pmd)
}
#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP
-static inline int pte_uffd_wp(pte_t pte)
+static inline int pte_uffd(pte_t pte)
{
- return !!(pte_val(pte) & PTE_UFFD_WP);
+ return !!(pte_val(pte) & PTE_UFFD);
}
-static inline pte_t pte_mkuffd_wp(pte_t pte)
+static inline pte_t pte_mkuffd(pte_t pte)
{
- return pte_wrprotect(set_pte_bit(pte, __pgprot(PTE_UFFD_WP)));
+ return pte_wrprotect(set_pte_bit(pte, __pgprot(PTE_UFFD)));
}
-static inline pte_t pte_clear_uffd_wp(pte_t pte)
+static inline pte_t pte_clear_uffd(pte_t pte)
{
- return clear_pte_bit(pte, __pgprot(PTE_UFFD_WP));
+ return clear_pte_bit(pte, __pgprot(PTE_UFFD));
}
#endif /* CONFIG_HAVE_ARCH_USERFAULTFD_WP */
@@ -537,26 +537,23 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pte)
}
#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP
-static inline pte_t pte_swp_mkuffd_wp(pte_t pte)
+static inline pte_t pte_swp_mkuffd(pte_t pte)
{
- return set_pte_bit(pte, __pgprot(PTE_SWP_UFFD_WP));
+ return set_pte_bit(pte, __pgprot(PTE_SWP_UFFD));
}
-static inline int pte_swp_uffd_wp(pte_t pte)
+static inline int pte_swp_uffd(pte_t pte)
{
- return !!(pte_val(pte) & PTE_SWP_UFFD_WP);
+ return !!(pte_val(pte) & PTE_SWP_UFFD);
}
-static inline pte_t pte_swp_clear_uffd_wp(pte_t pte)
+static inline pte_t pte_swp_clear_uffd(pte_t pte)
{
- return clear_pte_bit(pte, __pgprot(PTE_SWP_UFFD_WP));
+ return clear_pte_bit(pte, __pgprot(PTE_SWP_UFFD));
}
#endif /* CONFIG_HAVE_ARCH_USERFAULTFD_WP */
-#ifdef CONFIG_NUMA_BALANCING
-/*
- * See the comment in include/linux/pgtable.h
- */
+#ifdef CONFIG_ARCH_HAS_PTE_PROTNONE
static inline int pte_protnone(pte_t pte)
{
/*
@@ -575,7 +572,7 @@ static inline int pmd_protnone(pmd_t pmd)
{
return pte_protnone(pmd_pte(pmd));
}
-#endif
+#endif /* CONFIG_ARCH_HAS_PTE_PROTNONE */
#define pmd_present(pmd) pte_present(pmd_pte(pmd))
#define pmd_dirty(pmd) pte_dirty(pmd_pte(pmd))
@@ -593,13 +590,13 @@ static inline int pmd_protnone(pmd_t pmd)
#define pmd_mkvalid_k(pmd) pte_pmd(pte_mkvalid_k(pmd_pte(pmd)))
#define pmd_mkinvalid(pmd) pte_pmd(pte_mkinvalid(pmd_pte(pmd)))
#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP
-#define pmd_uffd_wp(pmd) pte_uffd_wp(pmd_pte(pmd))
-#define pmd_mkuffd_wp(pmd) pte_pmd(pte_mkuffd_wp(pmd_pte(pmd)))
-#define pmd_clear_uffd_wp(pmd) pte_pmd(pte_clear_uffd_wp(pmd_pte(pmd)))
-#define pmd_swp_uffd_wp(pmd) pte_swp_uffd_wp(pmd_pte(pmd))
-#define pmd_swp_mkuffd_wp(pmd) pte_pmd(pte_swp_mkuffd_wp(pmd_pte(pmd)))
-#define pmd_swp_clear_uffd_wp(pmd) \
- pte_pmd(pte_swp_clear_uffd_wp(pmd_pte(pmd)))
+#define pmd_uffd(pmd) pte_uffd(pmd_pte(pmd))
+#define pmd_mkuffd(pmd) pte_pmd(pte_mkuffd(pmd_pte(pmd)))
+#define pmd_clear_uffd(pmd) pte_pmd(pte_clear_uffd(pmd_pte(pmd)))
+#define pmd_swp_uffd(pmd) pte_swp_uffd(pmd_pte(pmd))
+#define pmd_swp_mkuffd(pmd) pte_pmd(pte_swp_mkuffd(pmd_pte(pmd)))
+#define pmd_swp_clear_uffd(pmd) \
+ pte_pmd(pte_swp_clear_uffd(pmd_pte(pmd)))
#endif /* CONFIG_HAVE_ARCH_USERFAULTFD_WP */
#define pmd_write(pmd) pte_write(pmd_pte(pmd))
@@ -1515,7 +1512,7 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma,
* Encode and decode a swap entry:
* bits 0-1: present (must be zero)
* bits 2: remember PG_anon_exclusive
- * bit 3: remember uffd-wp state
+ * bit 3: remember uffd state
* bits 6-10: swap type
* bit 11: PTE_PRESENT_INVALID (must be zero)
* bits 12-61: swap offset
diff --git a/arch/loongarch/Kconfig b/arch/loongarch/Kconfig
index d8d25232501791..177fb9dc64320c 100644
--- a/arch/loongarch/Kconfig
+++ b/arch/loongarch/Kconfig
@@ -67,6 +67,7 @@ config LOONGARCH
select ARCH_SUPPORTS_LTO_CLANG
select ARCH_SUPPORTS_LTO_CLANG_THIN
select ARCH_SUPPORTS_MSEAL_SYSTEM_MAPPINGS
+ select ARCH_HAS_PTE_PROTNONE if 64BIT
select ARCH_SUPPORTS_NUMA_BALANCING if NUMA
select ARCH_SUPPORTS_PER_VMA_LOCK
select ARCH_SUPPORTS_RT
diff --git a/arch/loongarch/include/asm/pgtable.h b/arch/loongarch/include/asm/pgtable.h
index 223528c04d7392..1952e34bc8ee00 100644
--- a/arch/loongarch/include/asm/pgtable.h
+++ b/arch/loongarch/include/asm/pgtable.h
@@ -623,7 +623,7 @@ static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm,
#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
-#ifdef CONFIG_NUMA_BALANCING
+#ifdef CONFIG_ARCH_HAS_PTE_PROTNONE
static inline long pte_protnone(pte_t pte)
{
return (pte_val(pte) & _PAGE_PROTNONE);
@@ -633,7 +633,7 @@ static inline long pmd_protnone(pmd_t pmd)
{
return (pmd_val(pmd) & _PAGE_PROTNONE);
}
-#endif /* CONFIG_NUMA_BALANCING */
+#endif /* CONFIG_ARCH_HAS_PTE_PROTNONE */
#define pmd_leaf(pmd) ((pmd_val(pmd) & _PAGE_HUGE) != 0)
#define pud_leaf(pud) ((pud_val(pud) & _PAGE_HUGE) != 0)
diff --git a/arch/powerpc/include/asm/book3s/64/pgtable.h b/arch/powerpc/include/asm/book3s/64/pgtable.h
index e67e64ac6e8cfc..53a0c5892548fc 100644
--- a/arch/powerpc/include/asm/book3s/64/pgtable.h
+++ b/arch/powerpc/include/asm/book3s/64/pgtable.h
@@ -490,13 +490,13 @@ static inline pte_t pte_clear_soft_dirty(pte_t pte)
}
#endif /* CONFIG_HAVE_ARCH_SOFT_DIRTY */
-#ifdef CONFIG_NUMA_BALANCING
+#ifdef CONFIG_ARCH_HAS_PTE_PROTNONE
static inline int pte_protnone(pte_t pte)
{
return (pte_raw(pte) & cpu_to_be64(_PAGE_PRESENT | _PAGE_PTE | _PAGE_RWX)) ==
cpu_to_be64(_PAGE_PRESENT | _PAGE_PTE);
}
-#endif /* CONFIG_NUMA_BALANCING */
+#endif /* CONFIG_ARCH_HAS_PTE_PROTNONE */
static inline bool pte_hw_valid(pte_t pte)
{
@@ -1067,12 +1067,12 @@ static inline pte_t *pmdp_ptep(pmd_t *pmd)
#endif
#endif /* CONFIG_HAVE_ARCH_SOFT_DIRTY */
-#ifdef CONFIG_NUMA_BALANCING
+#ifdef CONFIG_ARCH_HAS_PTE_PROTNONE
static inline int pmd_protnone(pmd_t pmd)
{
return pte_protnone(pmd_pte(pmd));
}
-#endif /* CONFIG_NUMA_BALANCING */
+#endif /* CONFIG_ARCH_HAS_PTE_PROTNONE */
#define pmd_write(pmd) pte_write(pmd_pte(pmd))
diff --git a/arch/powerpc/platforms/Kconfig.cputype b/arch/powerpc/platforms/Kconfig.cputype
index bac02c83bb3e4f..36b64a24cf3003 100644
--- a/arch/powerpc/platforms/Kconfig.cputype
+++ b/arch/powerpc/platforms/Kconfig.cputype
@@ -87,6 +87,7 @@ config PPC_BOOK3S_64
select ARCH_ENABLE_HUGEPAGE_MIGRATION if HUGETLB_PAGE && MIGRATION
select ARCH_ENABLE_SPLIT_PMD_PTLOCK
select ARCH_SUPPORTS_HUGETLBFS
+ select ARCH_HAS_PTE_PROTNONE
select ARCH_SUPPORTS_NUMA_BALANCING
select HAVE_MOVE_PMD
select HAVE_MOVE_PUD
diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
index 3f0a647218e407..58c12f4fac6681 100644
--- a/arch/riscv/Kconfig
+++ b/arch/riscv/Kconfig
@@ -71,6 +71,7 @@ config RISCV
select ARCH_SUPPORTS_MSEAL_SYSTEM_MAPPINGS if 64BIT && MMU
select ARCH_SUPPORTS_PAGE_TABLE_CHECK if MMU
select ARCH_SUPPORTS_PER_VMA_LOCK if MMU
+ select ARCH_HAS_PTE_PROTNONE if MMU
select ARCH_SUPPORTS_RT
select ARCH_SUPPORTS_SHADOW_CALL_STACK if HAVE_SHADOW_CALL_STACK
select ARCH_SUPPORTS_SCHED_MC if SMP
diff --git a/arch/riscv/include/asm/pgtable-bits.h b/arch/riscv/include/asm/pgtable-bits.h
index b422d9691e60d8..d5a86b4df3ce69 100644
--- a/arch/riscv/include/asm/pgtable-bits.h
+++ b/arch/riscv/include/asm/pgtable-bits.h
@@ -40,20 +40,20 @@
#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP
-/* ext_svrsw60t59b: Bit(60) for uffd-wp tracking */
-#define _PAGE_UFFD_WP \
+/* ext_svrsw60t59b: Bit(60) for userfaultfd tracking */
+#define _PAGE_UFFD \
((riscv_has_extension_unlikely(RISCV_ISA_EXT_SVRSW60T59B)) ? \
(1UL << 60) : 0)
/*
* Bit 4 is not involved into swap entry computation, so we
- * can borrow it for swap page uffd-wp tracking.
+ * can borrow it for swap page userfaultfd tracking.
*/
-#define _PAGE_SWP_UFFD_WP \
+#define _PAGE_SWP_UFFD \
((riscv_has_extension_unlikely(RISCV_ISA_EXT_SVRSW60T59B)) ? \
_PAGE_USER : 0)
#else
-#define _PAGE_UFFD_WP 0
-#define _PAGE_SWP_UFFD_WP 0
+#define _PAGE_UFFD 0
+#define _PAGE_SWP_UFFD 0
#endif
#define _PAGE_TABLE _PAGE_PRESENT
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index 5d5756bda82e3d..8194a2060f82bc 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -410,37 +410,37 @@ static inline pte_t pte_wrprotect(pte_t pte)
}
#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP
-#define pgtable_supports_uffd_wp() \
+#define pgtable_supports_uffd() \
riscv_has_extension_unlikely(RISCV_ISA_EXT_SVRSW60T59B)
-static inline bool pte_uffd_wp(pte_t pte)
+static inline bool pte_uffd(pte_t pte)
{
- return !!(pte_val(pte) & _PAGE_UFFD_WP);
+ return !!(pte_val(pte) & _PAGE_UFFD);
}
-static inline pte_t pte_mkuffd_wp(pte_t pte)
+static inline pte_t pte_mkuffd(pte_t pte)
{
- return pte_wrprotect(__pte(pte_val(pte) | _PAGE_UFFD_WP));
+ return pte_wrprotect(__pte(pte_val(pte) | _PAGE_UFFD));
}
-static inline pte_t pte_clear_uffd_wp(pte_t pte)
+static inline pte_t pte_clear_uffd(pte_t pte)
{
- return __pte(pte_val(pte) & ~(_PAGE_UFFD_WP));
+ return __pte(pte_val(pte) & ~(_PAGE_UFFD));
}
-static inline bool pte_swp_uffd_wp(pte_t pte)
+static inline bool pte_swp_uffd(pte_t pte)
{
- return !!(pte_val(pte) & _PAGE_SWP_UFFD_WP);
+ return !!(pte_val(pte) & _PAGE_SWP_UFFD);
}
-static inline pte_t pte_swp_mkuffd_wp(pte_t pte)
+static inline pte_t pte_swp_mkuffd(pte_t pte)
{
- return __pte(pte_val(pte) | _PAGE_SWP_UFFD_WP);
+ return __pte(pte_val(pte) | _PAGE_SWP_UFFD);
}
-static inline pte_t pte_swp_clear_uffd_wp(pte_t pte)
+static inline pte_t pte_swp_clear_uffd(pte_t pte)
{
- return __pte(pte_val(pte) & ~(_PAGE_SWP_UFFD_WP));
+ return __pte(pte_val(pte) & ~(_PAGE_SWP_UFFD));
}
#endif /* CONFIG_HAVE_ARCH_USERFAULTFD_WP */
@@ -534,10 +534,7 @@ static inline pte_t pte_swp_clear_soft_dirty(pte_t pte)
PAGE_SIZE)
#endif
-#ifdef CONFIG_NUMA_BALANCING
-/*
- * See the comment in include/asm-generic/pgtable.h
- */
+#ifdef CONFIG_ARCH_HAS_PTE_PROTNONE
static inline int pte_protnone(pte_t pte)
{
return (pte_val(pte) & (_PAGE_PRESENT | _PAGE_PROT_NONE)) == _PAGE_PROT_NONE;
@@ -547,7 +544,7 @@ static inline int pmd_protnone(pmd_t pmd)
{
return pte_protnone(pmd_pte(pmd));
}
-#endif
+#endif /* CONFIG_ARCH_HAS_PTE_PROTNONE */
/* Modify page protection bits */
static inline pte_t pte_modify(pte_t pte, pgprot_t newprot)
@@ -899,34 +896,34 @@ static inline pud_t pud_mkspecial(pud_t pud)
#endif
#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP
-static inline bool pmd_uffd_wp(pmd_t pmd)
+static inline bool pmd_uffd(pmd_t pmd)
{
- return pte_uffd_wp(pmd_pte(pmd));
+ return pte_uffd(pmd_pte(pmd));
}
-static inline pmd_t pmd_mkuffd_wp(pmd_t pmd)
+static inline pmd_t pmd_mkuffd(pmd_t pmd)
{
- return pte_pmd(pte_mkuffd_wp(pmd_pte(pmd)));
+ return pte_pmd(pte_mkuffd(pmd_pte(pmd)));
}
-static inline pmd_t pmd_clear_uffd_wp(pmd_t pmd)
+static inline pmd_t pmd_clear_uffd(pmd_t pmd)
{
- return pte_pmd(pte_clear_uffd_wp(pmd_pte(pmd)));
+ return pte_pmd(pte_clear_uffd(pmd_pte(pmd)));
}
-static inline bool pmd_swp_uffd_wp(pmd_t pmd)
+static inline bool pmd_swp_uffd(pmd_t pmd)
{
- return pte_swp_uffd_wp(pmd_pte(pmd));
+ return pte_swp_uffd(pmd_pte(pmd));
}
-static inline pmd_t pmd_swp_mkuffd_wp(pmd_t pmd)
+static inline pmd_t pmd_swp_mkuffd(pmd_t pmd)
{
- return pte_pmd(pte_swp_mkuffd_wp(pmd_pte(pmd)));
+ return pte_pmd(pte_swp_mkuffd(pmd_pte(pmd)));
}
-static inline pmd_t pmd_swp_clear_uffd_wp(pmd_t pmd)
+static inline pmd_t pmd_swp_clear_uffd(pmd_t pmd)
{
- return pte_pmd(pte_swp_clear_uffd_wp(pmd_pte(pmd)));
+ return pte_pmd(pte_swp_clear_uffd(pmd_pte(pmd)));
}
#endif /* CONFIG_HAVE_ARCH_USERFAULTFD_WP */
@@ -1170,7 +1167,7 @@ static inline pud_t pud_modify(pud_t pud, pgprot_t newprot)
* bit 0: _PAGE_PRESENT (zero)
* bit 1 to 2: (zero)
* bit 3: _PAGE_SWP_SOFT_DIRTY
- * bit 4: _PAGE_SWP_UFFD_WP
+ * bit 4: _PAGE_SWP_UFFD
* bit 5: _PAGE_PROT_NONE (zero)
* bit 6: exclusive marker
* bits 7 to 11: swap type
diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig
index 84404e6778d504..ad56daa6f874cd 100644
--- a/arch/s390/Kconfig
+++ b/arch/s390/Kconfig
@@ -148,6 +148,7 @@ config S390
select ARCH_SUPPORTS_HUGETLBFS
select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 && CC_IS_CLANG
select ARCH_SUPPORTS_MSEAL_SYSTEM_MAPPINGS
+ select ARCH_HAS_PTE_PROTNONE
select ARCH_SUPPORTS_NUMA_BALANCING
select ARCH_SUPPORTS_PAGE_TABLE_CHECK
select ARCH_SUPPORTS_PER_VMA_LOCK
diff --git a/arch/s390/include/asm/hugetlb.h b/arch/s390/include/asm/hugetlb.h
index e33a5b587ee4e6..02821a7d9d9138 100644
--- a/arch/s390/include/asm/hugetlb.h
+++ b/arch/s390/include/asm/hugetlb.h
@@ -77,20 +77,20 @@ static inline void huge_ptep_set_wrprotect(struct mm_struct *mm,
__set_huge_pte_at(mm, addr, ptep, pte_wrprotect(pte));
}
-#define __HAVE_ARCH_HUGE_PTE_MKUFFD_WP
-static inline pte_t huge_pte_mkuffd_wp(pte_t pte)
+#define __HAVE_ARCH_HUGE_PTE_MKUFFD
+static inline pte_t huge_pte_mkuffd(pte_t pte)
{
return pte;
}
-#define __HAVE_ARCH_HUGE_PTE_CLEAR_UFFD_WP
-static inline pte_t huge_pte_clear_uffd_wp(pte_t pte)
+#define __HAVE_ARCH_HUGE_PTE_CLEAR_UFFD
+static inline pte_t huge_pte_clear_uffd(pte_t pte)
{
return pte;
}
-#define __HAVE_ARCH_HUGE_PTE_UFFD_WP
-static inline int huge_pte_uffd_wp(pte_t pte)
+#define __HAVE_ARCH_HUGE_PTE_UFFD
+static inline int huge_pte_uffd(pte_t pte)
{
return 0;
}
diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h
index 859ce7c7d45448..0689dfdba40772 100644
--- a/arch/s390/include/asm/pgtable.h
+++ b/arch/s390/include/asm/pgtable.h
@@ -842,7 +842,7 @@ static inline int pte_same(pte_t a, pte_t b)
return pte_val(a) == pte_val(b);
}
-#ifdef CONFIG_NUMA_BALANCING
+#ifdef CONFIG_ARCH_HAS_PTE_PROTNONE
static inline int pte_protnone(pte_t pte)
{
return pte_present(pte) && !(pte_val(pte) & _PAGE_READ);
@@ -853,7 +853,7 @@ static inline int pmd_protnone(pmd_t pmd)
/* pmd_leaf(pmd) implies pmd_present(pmd) */
return pmd_leaf(pmd) && !(pmd_val(pmd) & _SEGMENT_ENTRY_READ);
}
-#endif
+#endif /* CONFIG_ARCH_HAS_PTE_PROTNONE */
static inline bool pte_swp_exclusive(pte_t pte)
{
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index bdad90f210e4bf..380c4aacd42a1c 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -124,6 +124,7 @@ config X86
select ARCH_SUPPORTS_DEBUG_PAGEALLOC
select ARCH_SUPPORTS_HUGETLBFS
select ARCH_SUPPORTS_PAGE_TABLE_CHECK if X86_64
+ select ARCH_HAS_PTE_PROTNONE if X86_64
select ARCH_SUPPORTS_NUMA_BALANCING if X86_64
select ARCH_SUPPORTS_KMAP_LOCAL_FORCE_MAP if NR_CPUS <= 4096
select ARCH_SUPPORTS_CFI if X86_64
diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h
index ac295ca6c92f99..cfb66e5f380be5 100644
--- a/arch/x86/include/asm/pgtable.h
+++ b/arch/x86/include/asm/pgtable.h
@@ -411,19 +411,19 @@ static inline pte_t pte_wrprotect(pte_t pte)
}
#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP
-static inline int pte_uffd_wp(pte_t pte)
+static inline int pte_uffd(pte_t pte)
{
- return pte_flags(pte) & _PAGE_UFFD_WP;
+ return pte_flags(pte) & _PAGE_UFFD;
}
-static inline pte_t pte_mkuffd_wp(pte_t pte)
+static inline pte_t pte_mkuffd(pte_t pte)
{
- return pte_wrprotect(pte_set_flags(pte, _PAGE_UFFD_WP));
+ return pte_wrprotect(pte_set_flags(pte, _PAGE_UFFD));
}
-static inline pte_t pte_clear_uffd_wp(pte_t pte)
+static inline pte_t pte_clear_uffd(pte_t pte)
{
- return pte_clear_flags(pte, _PAGE_UFFD_WP);
+ return pte_clear_flags(pte, _PAGE_UFFD);
}
#endif /* CONFIG_HAVE_ARCH_USERFAULTFD_WP */
@@ -526,19 +526,19 @@ static inline pmd_t pmd_wrprotect(pmd_t pmd)
}
#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP
-static inline int pmd_uffd_wp(pmd_t pmd)
+static inline int pmd_uffd(pmd_t pmd)
{
- return pmd_flags(pmd) & _PAGE_UFFD_WP;
+ return pmd_flags(pmd) & _PAGE_UFFD;
}
-static inline pmd_t pmd_mkuffd_wp(pmd_t pmd)
+static inline pmd_t pmd_mkuffd(pmd_t pmd)
{
- return pmd_wrprotect(pmd_set_flags(pmd, _PAGE_UFFD_WP));
+ return pmd_wrprotect(pmd_set_flags(pmd, _PAGE_UFFD));
}
-static inline pmd_t pmd_clear_uffd_wp(pmd_t pmd)
+static inline pmd_t pmd_clear_uffd(pmd_t pmd)
{
- return pmd_clear_flags(pmd, _PAGE_UFFD_WP);
+ return pmd_clear_flags(pmd, _PAGE_UFFD);
}
#endif /* CONFIG_HAVE_ARCH_USERFAULTFD_WP */
@@ -985,11 +985,7 @@ static inline int pmd_present(pmd_t pmd)
return pmd_flags(pmd) & (_PAGE_PRESENT | _PAGE_PROTNONE | _PAGE_PSE);
}
-#ifdef CONFIG_NUMA_BALANCING
-/*
- * These work without NUMA balancing but the kernel does not care. See the
- * comment in include/linux/pgtable.h
- */
+#ifdef CONFIG_ARCH_HAS_PTE_PROTNONE
static inline int pte_protnone(pte_t pte)
{
return (pte_flags(pte) & (_PAGE_PROTNONE | _PAGE_PRESENT))
@@ -1001,7 +997,7 @@ static inline int pmd_protnone(pmd_t pmd)
return (pmd_flags(pmd) & (_PAGE_PROTNONE | _PAGE_PRESENT))
== _PAGE_PROTNONE;
}
-#endif /* CONFIG_NUMA_BALANCING */
+#endif /* CONFIG_ARCH_HAS_PTE_PROTNONE */
static inline int pmd_none(pmd_t pmd)
{
@@ -1564,34 +1560,34 @@ static inline pmd_t pmd_swp_clear_soft_dirty(pmd_t pmd)
#endif
#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP
-static inline pte_t pte_swp_mkuffd_wp(pte_t pte)
+static inline pte_t pte_swp_mkuffd(pte_t pte)
{
- return pte_set_flags(pte, _PAGE_SWP_UFFD_WP);
+ return pte_set_flags(pte, _PAGE_SWP_UFFD);
}
-static inline int pte_swp_uffd_wp(pte_t pte)
+static inline int pte_swp_uffd(pte_t pte)
{
- return pte_flags(pte) & _PAGE_SWP_UFFD_WP;
+ return pte_flags(pte) & _PAGE_SWP_UFFD;
}
-static inline pte_t pte_swp_clear_uffd_wp(pte_t pte)
+static inline pte_t pte_swp_clear_uffd(pte_t pte)
{
- return pte_clear_flags(pte, _PAGE_SWP_UFFD_WP);
+ return pte_clear_flags(pte, _PAGE_SWP_UFFD);
}
-static inline pmd_t pmd_swp_mkuffd_wp(pmd_t pmd)
+static inline pmd_t pmd_swp_mkuffd(pmd_t pmd)
{
- return pmd_set_flags(pmd, _PAGE_SWP_UFFD_WP);
+ return pmd_set_flags(pmd, _PAGE_SWP_UFFD);
}
-static inline int pmd_swp_uffd_wp(pmd_t pmd)
+static inline int pmd_swp_uffd(pmd_t pmd)
{
- return pmd_flags(pmd) & _PAGE_SWP_UFFD_WP;
+ return pmd_flags(pmd) & _PAGE_SWP_UFFD;
}
-static inline pmd_t pmd_swp_clear_uffd_wp(pmd_t pmd)
+static inline pmd_t pmd_swp_clear_uffd(pmd_t pmd)
{
- return pmd_clear_flags(pmd, _PAGE_SWP_UFFD_WP);
+ return pmd_clear_flags(pmd, _PAGE_SWP_UFFD);
}
#endif /* CONFIG_HAVE_ARCH_USERFAULTFD_WP */
diff --git a/arch/x86/include/asm/pgtable_types.h b/arch/x86/include/asm/pgtable_types.h
index 2ec250ba467e2d..af08d98be9309c 100644
--- a/arch/x86/include/asm/pgtable_types.h
+++ b/arch/x86/include/asm/pgtable_types.h
@@ -31,7 +31,7 @@
#define _PAGE_BIT_SPECIAL _PAGE_BIT_SOFTW1
#define _PAGE_BIT_CPA_TEST _PAGE_BIT_SOFTW1
-#define _PAGE_BIT_UFFD_WP _PAGE_BIT_SOFTW2 /* userfaultfd wrprotected */
+#define _PAGE_BIT_UFFD _PAGE_BIT_SOFTW2 /* userfaultfd tracking */
#define _PAGE_BIT_SOFT_DIRTY _PAGE_BIT_SOFTW3 /* software dirty tracking */
#define _PAGE_BIT_KERNEL_4K _PAGE_BIT_SOFTW3 /* page must not be converted to large */
@@ -39,7 +39,7 @@
#define _PAGE_BIT_SAVED_DIRTY _PAGE_BIT_SOFTW5 /* Saved Dirty bit (leaf) */
#define _PAGE_BIT_NOPTISHADOW _PAGE_BIT_SOFTW5 /* No PTI shadow (root PGD) */
#else
-/* Shared with _PAGE_BIT_UFFD_WP which is not supported on 32 bit */
+/* Shared with _PAGE_BIT_UFFD which is not supported on 32 bit */
#define _PAGE_BIT_SAVED_DIRTY _PAGE_BIT_SOFTW2 /* Saved Dirty bit (leaf) */
#define _PAGE_BIT_NOPTISHADOW _PAGE_BIT_SOFTW2 /* No PTI shadow (root PGD) */
#endif
@@ -111,11 +111,11 @@
#endif
#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP
-#define _PAGE_UFFD_WP (_AT(pteval_t, 1) << _PAGE_BIT_UFFD_WP)
-#define _PAGE_SWP_UFFD_WP _PAGE_USER
+#define _PAGE_UFFD (_AT(pteval_t, 1) << _PAGE_BIT_UFFD)
+#define _PAGE_SWP_UFFD _PAGE_USER
#else
-#define _PAGE_UFFD_WP (_AT(pteval_t, 0))
-#define _PAGE_SWP_UFFD_WP (_AT(pteval_t, 0))
+#define _PAGE_UFFD (_AT(pteval_t, 0))
+#define _PAGE_SWP_UFFD (_AT(pteval_t, 0))
#endif
#if defined(CONFIG_X86_64) || defined(CONFIG_X86_PAE)
@@ -129,7 +129,7 @@
/*
* The hardware requires shadow stack to be Write=0,Dirty=1. However,
* there are valid cases where the kernel might create read-only PTEs that
- * are dirty (e.g., fork(), mprotect(), uffd-wp(), soft-dirty tracking). In
+ * are dirty (e.g., fork(), mprotect(), userfaultfd, soft-dirty tracking). In
* this case, the _PAGE_SAVED_DIRTY bit is used instead of the HW-dirty bit,
* to avoid creating a wrong "shadow stack" PTEs. Such PTEs have
* (Write=0,SavedDirty=1,Dirty=0) set.
@@ -151,7 +151,7 @@
#define _COMMON_PAGE_CHG_MASK (PTE_PFN_MASK | _PAGE_PCD | _PAGE_PWT | \
_PAGE_SPECIAL | _PAGE_ACCESSED | \
_PAGE_DIRTY_BITS | _PAGE_SOFT_DIRTY | \
- _PAGE_CC | _PAGE_UFFD_WP)
+ _PAGE_CC | _PAGE_UFFD)
#define _PAGE_CHG_MASK (_COMMON_PAGE_CHG_MASK | _PAGE_PAT)
#define _HPAGE_CHG_MASK (_COMMON_PAGE_CHG_MASK | _PAGE_PSE | _PAGE_PAT_LARGE)
diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
index d32408f7cd5ed2..2d724156763d2c 100644
--- a/fs/proc/task_mmu.c
+++ b/fs/proc/task_mmu.c
@@ -1237,6 +1237,9 @@ static void show_smap_vma_flags(struct seq_file *m, struct vm_area_struct *vma)
#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR
[ilog2(VM_UFFD_MINOR)] = "ui",
#endif /* CONFIG_HAVE_ARCH_USERFAULTFD_MINOR */
+#ifdef CONFIG_USERFAULTFD_RWP
+ [ilog2(VM_UFFD_RWP)] = "ur",
+#endif
#ifdef CONFIG_ARCH_HAS_USER_SHADOW_STACK
[ilog2(VM_SHADOW_STACK)] = "ss",
#endif
@@ -2035,14 +2038,14 @@ static pagemap_entry_t pte_to_pagemap_entry(struct pagemapread *pm,
page = vm_normal_page(vma, addr, pte);
if (pte_soft_dirty(pte))
flags |= PM_SOFT_DIRTY;
- if (pte_uffd_wp(pte))
+ if (pte_uffd(pte))
flags |= PM_UFFD_WP;
} else {
softleaf_t entry;
if (pte_swp_soft_dirty(pte))
flags |= PM_SOFT_DIRTY;
- if (pte_swp_uffd_wp(pte))
+ if (pte_swp_uffd(pte))
flags |= PM_UFFD_WP;
entry = softleaf_from_pte(pte);
if (pm->show_pfn) {
@@ -2108,7 +2111,7 @@ static int pagemap_pmd_range_thp(pmd_t *pmdp, unsigned long addr,
flags |= PM_PRESENT;
if (pmd_soft_dirty(pmd))
flags |= PM_SOFT_DIRTY;
- if (pmd_uffd_wp(pmd))
+ if (pmd_uffd(pmd))
flags |= PM_UFFD_WP;
if (pm->show_pfn)
frame = pmd_pfn(pmd) + idx;
@@ -2127,7 +2130,7 @@ static int pagemap_pmd_range_thp(pmd_t *pmdp, unsigned long addr,
flags |= PM_SWAP;
if (pmd_swp_soft_dirty(pmd))
flags |= PM_SOFT_DIRTY;
- if (pmd_swp_uffd_wp(pmd))
+ if (pmd_swp_uffd(pmd))
flags |= PM_UFFD_WP;
page = softleaf_to_page(entry);
}
@@ -2232,14 +2235,14 @@ static int pagemap_hugetlb_range(pte_t *ptep, unsigned long hmask,
!hugetlb_pmd_shared(ptep))
flags |= PM_MMAP_EXCLUSIVE;
- if (huge_pte_uffd_wp(pte))
+ if (huge_pte_uffd(pte))
flags |= PM_UFFD_WP;
flags |= PM_PRESENT;
if (pm->show_pfn)
frame = pte_pfn(pte) +
((addr & ~hmask) >> PAGE_SHIFT);
- } else if (pte_swp_uffd_wp_any(pte)) {
+ } else if (pte_swp_uffd_any(pte)) {
flags |= PM_UFFD_WP;
}
@@ -2280,7 +2283,7 @@ static const struct mm_walk_ops pagemap_ops = {
* Bits 5-54 swap offset if swapped
* Bit 55 pte is soft-dirty (see Documentation/admin-guide/mm/soft-dirty.rst)
* Bit 56 page exclusively mapped
- * Bit 57 pte is uffd-wp write-protected
+ * Bit 57 pte is tracked by userfaultfd (uffd-wp or RWP)
* Bit 58 pte is a guard region
* Bits 59-60 zero
* Bit 61 page is file-page or shared-anon
@@ -2415,7 +2418,7 @@ static int pagemap_release(struct inode *inode, struct file *file)
PAGE_IS_FILE | PAGE_IS_PRESENT | \
PAGE_IS_SWAPPED | PAGE_IS_PFNZERO | \
PAGE_IS_HUGE | PAGE_IS_SOFT_DIRTY | \
- PAGE_IS_GUARD)
+ PAGE_IS_GUARD | PAGE_IS_ACCESSED)
#define PM_SCAN_FLAGS (PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC)
struct pagemap_scan_private {
@@ -2440,8 +2443,12 @@ static unsigned long pagemap_page_category(struct pagemap_scan_private *p,
categories = PAGE_IS_PRESENT;
- if (!pte_uffd_wp(pte))
- categories |= PAGE_IS_WRITTEN;
+ if (!pte_uffd(pte)) {
+ if (userfaultfd_wp(vma))
+ categories |= PAGE_IS_WRITTEN;
+ if (userfaultfd_rwp(vma))
+ categories |= PAGE_IS_ACCESSED;
+ }
if (p->masks_of_interest & PAGE_IS_FILE) {
page = vm_normal_page(vma, addr, pte);
@@ -2458,8 +2465,12 @@ static unsigned long pagemap_page_category(struct pagemap_scan_private *p,
categories = PAGE_IS_SWAPPED;
- if (!pte_swp_uffd_wp_any(pte))
- categories |= PAGE_IS_WRITTEN;
+ if (!pte_swp_uffd_any(pte)) {
+ if (userfaultfd_wp(vma))
+ categories |= PAGE_IS_WRITTEN;
+ if (userfaultfd_rwp(vma))
+ categories |= PAGE_IS_ACCESSED;
+ }
entry = softleaf_from_pte(pte);
if (softleaf_is_guard_marker(entry))
@@ -2483,13 +2494,13 @@ static void make_uffd_wp_pte(struct vm_area_struct *vma,
pte_t old_pte;
old_pte = ptep_modify_prot_start(vma, addr, pte);
- ptent = pte_mkuffd_wp(old_pte);
+ ptent = pte_mkuffd(old_pte);
ptep_modify_prot_commit(vma, addr, pte, old_pte, ptent);
} else if (pte_none(ptent)) {
set_pte_at(vma->vm_mm, addr, pte,
make_pte_marker(PTE_MARKER_UFFD_WP));
} else {
- ptent = pte_swp_mkuffd_wp(ptent);
+ ptent = pte_swp_mkuffd(ptent);
set_pte_at(vma->vm_mm, addr, pte, ptent);
}
}
@@ -2508,8 +2519,12 @@ static unsigned long pagemap_thp_category(struct pagemap_scan_private *p,
struct page *page;
categories |= PAGE_IS_PRESENT;
- if (!pmd_uffd_wp(pmd))
- categories |= PAGE_IS_WRITTEN;
+ if (!pmd_uffd(pmd)) {
+ if (userfaultfd_wp(vma))
+ categories |= PAGE_IS_WRITTEN;
+ if (userfaultfd_rwp(vma))
+ categories |= PAGE_IS_ACCESSED;
+ }
if (p->masks_of_interest & PAGE_IS_FILE) {
page = vm_normal_page_pmd(vma, addr, pmd);
@@ -2523,8 +2538,12 @@ static unsigned long pagemap_thp_category(struct pagemap_scan_private *p,
categories |= PAGE_IS_SOFT_DIRTY;
} else {
categories |= PAGE_IS_SWAPPED;
- if (!pmd_swp_uffd_wp(pmd))
- categories |= PAGE_IS_WRITTEN;
+ if (!pmd_swp_uffd(pmd)) {
+ if (userfaultfd_wp(vma))
+ categories |= PAGE_IS_WRITTEN;
+ if (userfaultfd_rwp(vma))
+ categories |= PAGE_IS_ACCESSED;
+ }
if (pmd_swp_soft_dirty(pmd))
categories |= PAGE_IS_SOFT_DIRTY;
@@ -2547,17 +2566,18 @@ static void make_uffd_wp_pmd(struct vm_area_struct *vma,
if (pmd_present(pmd)) {
old = pmdp_invalidate_ad(vma, addr, pmdp);
- pmd = pmd_mkuffd_wp(old);
+ pmd = pmd_mkuffd(old);
set_pmd_at(vma->vm_mm, addr, pmdp, pmd);
} else if (pmd_is_migration_entry(pmd)) {
- pmd = pmd_swp_mkuffd_wp(pmd);
+ pmd = pmd_swp_mkuffd(pmd);
set_pmd_at(vma->vm_mm, addr, pmdp, pmd);
}
}
#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
#ifdef CONFIG_HUGETLB_PAGE
-static unsigned long pagemap_hugetlb_category(pte_t pte)
+static unsigned long pagemap_hugetlb_category(struct vm_area_struct *vma,
+ pte_t pte)
{
unsigned long categories = PAGE_IS_HUGE;
@@ -2572,8 +2592,12 @@ static unsigned long pagemap_hugetlb_category(pte_t pte)
if (pte_present(pte)) {
categories |= PAGE_IS_PRESENT;
- if (!huge_pte_uffd_wp(pte))
- categories |= PAGE_IS_WRITTEN;
+ if (!huge_pte_uffd(pte)) {
+ if (userfaultfd_wp(vma))
+ categories |= PAGE_IS_WRITTEN;
+ if (userfaultfd_rwp(vma))
+ categories |= PAGE_IS_ACCESSED;
+ }
if (!PageAnon(pte_page(pte)))
categories |= PAGE_IS_FILE;
if (is_zero_pfn(pte_pfn(pte)))
@@ -2583,8 +2607,12 @@ static unsigned long pagemap_hugetlb_category(pte_t pte)
} else {
categories |= PAGE_IS_SWAPPED;
- if (!pte_swp_uffd_wp_any(pte))
- categories |= PAGE_IS_WRITTEN;
+ if (!pte_swp_uffd_any(pte)) {
+ if (userfaultfd_wp(vma))
+ categories |= PAGE_IS_WRITTEN;
+ if (userfaultfd_rwp(vma))
+ categories |= PAGE_IS_ACCESSED;
+ }
if (pte_swp_soft_dirty(pte))
categories |= PAGE_IS_SOFT_DIRTY;
}
@@ -2611,12 +2639,12 @@ static void make_uffd_wp_huge_pte(struct vm_area_struct *vma,
if (softleaf_is_migration(entry)) {
set_huge_pte_at(vma->vm_mm, addr, ptep,
- pte_swp_mkuffd_wp(ptent), psize);
+ pte_swp_mkuffd(ptent), psize);
} else {
pte_t old_pte, new_pte;
old_pte = huge_ptep_modify_prot_start(vma, addr, ptep);
- new_pte = huge_pte_mkuffd_wp(old_pte);
+ new_pte = huge_pte_mkuffd(old_pte);
huge_ptep_modify_prot_commit(vma, addr, ptep, old_pte, new_pte);
}
}
@@ -2849,8 +2877,8 @@ static int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start,
for (addr = start; addr != end; pte++, addr += PAGE_SIZE) {
pte_t ptent = ptep_get(pte);
- if ((pte_present(ptent) && pte_uffd_wp(ptent)) ||
- pte_swp_uffd_wp_any(ptent))
+ if ((pte_present(ptent) && pte_uffd(ptent)) ||
+ pte_swp_uffd_any(ptent))
continue;
make_uffd_wp_pte(vma, addr, pte, ptent);
if (!flush_end)
@@ -2860,15 +2888,16 @@ static int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start,
goto flush_and_return;
}
- if (!p->arg.category_anyof_mask && !p->arg.category_inverted &&
+ if (userfaultfd_wp(vma) && !p->arg.category_anyof_mask &&
+ !p->arg.category_inverted &&
p->arg.category_mask == PAGE_IS_WRITTEN &&
p->arg.return_mask == PAGE_IS_WRITTEN) {
for (addr = start; addr < end; pte++, addr += PAGE_SIZE) {
unsigned long next = addr + PAGE_SIZE;
pte_t ptent = ptep_get(pte);
- if ((pte_present(ptent) && pte_uffd_wp(ptent)) ||
- pte_swp_uffd_wp_any(ptent))
+ if ((pte_present(ptent) && pte_uffd(ptent)) ||
+ pte_swp_uffd_any(ptent))
continue;
ret = pagemap_scan_output(p->cur_vma_category | PAGE_IS_WRITTEN,
p, addr, &next);
@@ -2935,7 +2964,8 @@ static int pagemap_scan_hugetlb_entry(pte_t *ptep, unsigned long hmask,
/* Go the short route when not write-protecting pages. */
pte = huge_ptep_get(walk->mm, start, ptep);
- categories = p->cur_vma_category | pagemap_hugetlb_category(pte);
+ categories = p->cur_vma_category |
+ pagemap_hugetlb_category(vma, pte);
if (!pagemap_scan_is_interesting_page(categories, p))
return 0;
@@ -2947,7 +2977,7 @@ static int pagemap_scan_hugetlb_entry(pte_t *ptep, unsigned long hmask,
ptl = huge_pte_lock(hstate_vma(vma), vma->vm_mm, ptep);
pte = huge_ptep_get(walk->mm, start, ptep);
- categories = p->cur_vma_category | pagemap_hugetlb_category(pte);
+ categories = p->cur_vma_category | pagemap_hugetlb_category(vma, pte);
if (!pagemap_scan_is_interesting_page(categories, p))
goto out_unlock;
diff --git a/include/asm-generic/hugetlb.h b/include/asm-generic/hugetlb.h
index e1a2e1b7c8e7b8..635c41cc34797b 100644
--- a/include/asm-generic/hugetlb.h
+++ b/include/asm-generic/hugetlb.h
@@ -37,24 +37,24 @@ static inline pte_t huge_pte_modify(pte_t pte, pgprot_t newprot)
return pte_modify(pte, newprot);
}
-#ifndef __HAVE_ARCH_HUGE_PTE_MKUFFD_WP
-static inline pte_t huge_pte_mkuffd_wp(pte_t pte)
+#ifndef __HAVE_ARCH_HUGE_PTE_MKUFFD
+static inline pte_t huge_pte_mkuffd(pte_t pte)
{
- return huge_pte_wrprotect(pte_mkuffd_wp(pte));
+ return huge_pte_wrprotect(pte_mkuffd(pte));
}
#endif
-#ifndef __HAVE_ARCH_HUGE_PTE_CLEAR_UFFD_WP
-static inline pte_t huge_pte_clear_uffd_wp(pte_t pte)
+#ifndef __HAVE_ARCH_HUGE_PTE_CLEAR_UFFD
+static inline pte_t huge_pte_clear_uffd(pte_t pte)
{
- return pte_clear_uffd_wp(pte);
+ return pte_clear_uffd(pte);
}
#endif
-#ifndef __HAVE_ARCH_HUGE_PTE_UFFD_WP
-static inline int huge_pte_uffd_wp(pte_t pte)
+#ifndef __HAVE_ARCH_HUGE_PTE_UFFD
+static inline int huge_pte_uffd(pte_t pte)
{
- return pte_uffd_wp(pte);
+ return pte_uffd(pte);
}
#endif
diff --git a/include/asm-generic/pgtable_uffd.h b/include/asm-generic/pgtable_uffd.h
index 0d85791efdf778..30e88fc1de2f02 100644
--- a/include/asm-generic/pgtable_uffd.h
+++ b/include/asm-generic/pgtable_uffd.h
@@ -2,79 +2,79 @@
#define _ASM_GENERIC_PGTABLE_UFFD_H
/*
- * Some platforms can customize the uffd-wp bit, making it unavailable
+ * Some platforms can customize the uffd PTE bit, making it unavailable
* even if the architecture provides the resource.
* Adding this API allows architectures to add their own checks for the
* devices on which the kernel is running.
* Note: When overriding it, please make sure the
* CONFIG_HAVE_ARCH_USERFAULTFD_WP is part of this macro.
*/
-#ifndef pgtable_supports_uffd_wp
-#define pgtable_supports_uffd_wp() IS_ENABLED(CONFIG_HAVE_ARCH_USERFAULTFD_WP)
+#ifndef pgtable_supports_uffd
+#define pgtable_supports_uffd() IS_ENABLED(CONFIG_HAVE_ARCH_USERFAULTFD_WP)
#endif
static inline bool uffd_supports_wp_marker(void)
{
- return pgtable_supports_uffd_wp() && IS_ENABLED(CONFIG_PTE_MARKER_UFFD_WP);
+ return pgtable_supports_uffd() && IS_ENABLED(CONFIG_PTE_MARKER_UFFD_WP);
}
#ifndef CONFIG_HAVE_ARCH_USERFAULTFD_WP
-static __always_inline int pte_uffd_wp(pte_t pte)
+static __always_inline int pte_uffd(pte_t pte)
{
return 0;
}
-static __always_inline int pmd_uffd_wp(pmd_t pmd)
+static __always_inline int pmd_uffd(pmd_t pmd)
{
return 0;
}
-static __always_inline pte_t pte_mkuffd_wp(pte_t pte)
+static __always_inline pte_t pte_mkuffd(pte_t pte)
{
return pte;
}
-static __always_inline pmd_t pmd_mkuffd_wp(pmd_t pmd)
+static __always_inline pmd_t pmd_mkuffd(pmd_t pmd)
{
return pmd;
}
-static __always_inline pte_t pte_clear_uffd_wp(pte_t pte)
+static __always_inline pte_t pte_clear_uffd(pte_t pte)
{
return pte;
}
-static __always_inline pmd_t pmd_clear_uffd_wp(pmd_t pmd)
+static __always_inline pmd_t pmd_clear_uffd(pmd_t pmd)
{
return pmd;
}
-static __always_inline pte_t pte_swp_mkuffd_wp(pte_t pte)
+static __always_inline pte_t pte_swp_mkuffd(pte_t pte)
{
return pte;
}
-static __always_inline int pte_swp_uffd_wp(pte_t pte)
+static __always_inline int pte_swp_uffd(pte_t pte)
{
return 0;
}
-static __always_inline pte_t pte_swp_clear_uffd_wp(pte_t pte)
+static __always_inline pte_t pte_swp_clear_uffd(pte_t pte)
{
return pte;
}
-static inline pmd_t pmd_swp_mkuffd_wp(pmd_t pmd)
+static inline pmd_t pmd_swp_mkuffd(pmd_t pmd)
{
return pmd;
}
-static inline int pmd_swp_uffd_wp(pmd_t pmd)
+static inline int pmd_swp_uffd(pmd_t pmd)
{
return 0;
}
-static inline pmd_t pmd_swp_clear_uffd_wp(pmd_t pmd)
+static inline pmd_t pmd_swp_clear_uffd(pmd_t pmd)
{
return pmd;
}
diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h
index ad20f7f8c17944..9c4ebf0fc227a8 100644
--- a/include/linux/huge_mm.h
+++ b/include/linux/huge_mm.h
@@ -529,6 +529,8 @@ static inline bool folio_test_pmd_mappable(struct folio *folio)
vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf);
+vm_fault_t do_huge_pmd_uffd_rwp(struct vm_fault *vmf);
+
vm_fault_t do_huge_pmd_device_private(struct vm_fault *vmf);
extern struct folio *huge_zero_folio;
@@ -723,6 +725,11 @@ static inline spinlock_t *pud_trans_huge_lock(pud_t *pud,
return NULL;
}
+static inline vm_fault_t do_huge_pmd_uffd_rwp(struct vm_fault *vmf)
+{
+ return 0;
+}
+
static inline vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
{
return 0;
diff --git a/include/linux/leafops.h b/include/linux/leafops.h
index 992cd8bd8ed0e9..2ce2f37ac883bc 100644
--- a/include/linux/leafops.h
+++ b/include/linux/leafops.h
@@ -100,8 +100,8 @@ static inline softleaf_t softleaf_from_pmd(pmd_t pmd)
if (pmd_swp_soft_dirty(pmd))
pmd = pmd_swp_clear_soft_dirty(pmd);
- if (pmd_swp_uffd_wp(pmd))
- pmd = pmd_swp_clear_uffd_wp(pmd);
+ if (pmd_swp_uffd(pmd))
+ pmd = pmd_swp_clear_uffd(pmd);
arch_entry = __pmd_to_swp_entry(pmd);
/* Temporary until swp_entry_t eliminated. */
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 485df9c2dbddb3..da8a07cd49a148 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -353,6 +353,7 @@ enum {
#endif
DECLARE_VMA_BIT(UFFD_MINOR, 41),
DECLARE_VMA_BIT(SEALED, 42),
+ DECLARE_VMA_BIT(UFFD_RWP, 43),
/* Flags that reuse flags above. */
DECLARE_VMA_BIT_ALIAS(PKEY_BIT0, HIGH_ARCH_0),
DECLARE_VMA_BIT_ALIAS(PKEY_BIT1, HIGH_ARCH_1),
@@ -496,12 +497,17 @@ enum {
#else
#define VM_UFFD_MINOR VM_NONE
#endif
+#ifdef CONFIG_USERFAULTFD_RWP
+#define VM_UFFD_RWP INIT_VM_FLAG(UFFD_RWP)
+#else
+#define VM_UFFD_RWP VM_NONE
+#endif
/*
- * vma_flags_t masks for the userfaultfd VMA flags. VMA_UFFD_MINOR is gated on
- * the same config as VM_UFFD_MINOR -- which implies 64BIT, where the bit fits
- * -- so an out-of-range bit is never fed to mk_vma_flags() on a build whose
- * bitmap cannot hold it.
+ * vma_flags_t masks for the userfaultfd VMA flags. The two high-bit modes are
+ * gated on the same configs as their VM_* flags above -- both of which imply
+ * 64BIT -- so an out-of-range bit is never fed to mk_vma_flags() on a build
+ * whose bitmap cannot hold it.
*/
#define VMA_UFFD_MISSING mk_vma_flags(VMA_UFFD_MISSING_BIT)
#define VMA_UFFD_WP mk_vma_flags(VMA_UFFD_WP_BIT)
@@ -510,6 +516,11 @@ enum {
#else
#define VMA_UFFD_MINOR EMPTY_VMA_FLAGS
#endif
+#ifdef CONFIG_USERFAULTFD_RWP
+#define VMA_UFFD_RWP mk_vma_flags(VMA_UFFD_RWP_BIT)
+#else
+#define VMA_UFFD_RWP EMPTY_VMA_FLAGS
+#endif
#ifdef CONFIG_64BIT
#define VM_ALLOW_ANY_UNCACHED INIT_VM_FLAG(ALLOW_ANY_UNCACHED)
@@ -648,29 +659,32 @@ enum {
* reconsistuted upon page fault, so necessitate page table copying upon fork.
*
* Note that these flags should be compared with the DESTINATION VMA not the
- * source, as VM_UFFD_WP may not be propagated to destination, while all other
- * flags will be.
+ * source: VM_UFFD_WP and VM_UFFD_RWP may be cleared on the destination
+ * (dup_userfaultfd() -> userfaultfd_reset_ctx() when the parent context did
+ * not negotiate UFFD_FEATURE_EVENT_FORK), while all other flags propagate.
*
* VM_PFNMAP / VM_MIXEDMAP - These contain kernel-mapped data which cannot be
* reasonably reconstructed on page fault.
*
* VM_UFFD_WP - Encodes metadata about an installed uffd
- * write protect handler, which cannot be
- * reconstructed on page fault.
+ * VM_UFFD_RWP write- or read-write-protect handler, which
+ * cannot be reconstructed on page fault.
*
- * We always copy pgtables when dst_vma has uffd-wp
- * enabled even if it's file-backed
- * (e.g. shmem). Because when uffd-wp is enabled,
- * pgtable contains uffd-wp protection information,
- * that's something we can't retrieve from page cache,
- * and skip copying will lose those info.
+ * We always copy pgtables when dst_vma has the
+ * uffd PTE bit in use even if it's file-backed
+ * (e.g. shmem). Because when the uffd bit is
+ * in use, the pgtable contains the protection
+ * information, that's something we can't
+ * retrieve from page cache, and skip copying
+ * will lose those info.
*
* VM_MAYBE_GUARD - Could contain page guard region markers which
* by design are a property of the page tables
* only and thus cannot be reconstructed on page
* fault.
*/
-#define VM_COPY_ON_FORK (VM_PFNMAP | VM_MIXEDMAP | VM_UFFD_WP | VM_MAYBE_GUARD)
+#define VM_COPY_ON_FORK (VM_PFNMAP | VM_MIXEDMAP | VM_UFFD_WP | VM_UFFD_RWP | \
+ VM_MAYBE_GUARD)
/*
* mapping from the currently active vm_flags protection bits (the
@@ -3317,6 +3331,11 @@ int get_cmdline(struct task_struct *task, char *buffer, int buflen);
#define MM_CP_UFFD_WP_RESOLVE (1UL << 3) /* Resolve wp */
#define MM_CP_UFFD_WP_ALL (MM_CP_UFFD_WP | \
MM_CP_UFFD_WP_RESOLVE)
+/* Whether this change is for uffd RWP */
+#define MM_CP_UFFD_RWP (1UL << 4) /* do rwp */
+#define MM_CP_UFFD_RWP_RESOLVE (1UL << 5) /* resolve rwp */
+#define MM_CP_UFFD_RWP_ALL (MM_CP_UFFD_RWP | \
+ MM_CP_UFFD_RWP_RESOLVE)
bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr,
pte_t pte);
@@ -4625,11 +4644,25 @@ static inline int vm_fault_to_errno(vm_fault_t vm_fault, int foll_flags)
/*
* Indicates whether GUP can follow a PROT_NONE mapped page, or whether
- * a (NUMA hinting) fault is required.
+ * a (NUMA hinting or userfaultfd RWP) fault is required.
*/
static inline bool gup_can_follow_protnone(const struct vm_area_struct *vma,
unsigned int flags)
{
+ /*
+ * VM_UFFD_RWP uses protnone as an access-tracking marker, not for
+ * NUMA hinting. GUP must always take a fault so the access is
+ * delivered to userfaultfd, regardless of FOLL_HONOR_NUMA_FAULT.
+ *
+ * Only do so while the VMA is accessible. If it has been made
+ * inaccessible (e.g. mprotect(PROT_NONE)), fall through to the guard
+ * below: forcing a fault there would loop, as handle_mm_fault() makes
+ * no progress on protnone in an inaccessible VMA, and the access is
+ * denied regardless of RWP anyway.
+ */
+ if (vma_test_single_mask(vma, VMA_UFFD_RWP) && vma_is_accessible(vma))
+ return false;
+
/*
* If callers don't want to honor NUMA hinting faults, no need to
* determine if we would actually have to trigger a NUMA hinting fault.
diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
index a8430a7ae05447..d839654db1d265 100644
--- a/include/linux/mm_inline.h
+++ b/include/linux/mm_inline.h
@@ -600,14 +600,14 @@ pte_install_uffd_wp_if_needed(struct vm_area_struct *vma, unsigned long addr,
return false;
/* A uffd-wp wr-protected normal pte */
- if (unlikely(pte_present(pteval) && pte_uffd_wp(pteval)))
+ if (unlikely(pte_present(pteval) && pte_uffd(pteval)))
arm_uffd_pte = true;
/*
* A uffd-wp wr-protected swap pte. Note: this should even cover an
* existing pte marker with uffd-wp bit set.
*/
- if (unlikely(pte_swp_uffd_wp_any(pteval)))
+ if (unlikely(pte_swp_uffd_any(pteval)))
arm_uffd_pte = true;
if (unlikely(arm_uffd_pte)) {
diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h
index 2981e386da7b3c..7d3dc8abc6c878 100644
--- a/include/linux/pgtable.h
+++ b/include/linux/pgtable.h
@@ -2129,18 +2129,26 @@ static inline int pud_trans_unstable(pud_t *pud)
return 0;
}
-#ifndef CONFIG_NUMA_BALANCING
+#ifndef CONFIG_ARCH_HAS_PTE_PROTNONE
/*
- * In an inaccessible (PROT_NONE) VMA, pte_protnone() may indicate "yes". It is
- * perfectly valid to indicate "no" in that case, which is why our default
- * implementation defaults to "always no".
+ * In an inaccessible (PROT_NONE) VMA, pte_protnone() may indicate "yes". It
+ * is perfectly valid to indicate "no" in that case, which is why our
+ * default implementation defaults to "always no".
*
- * In an accessible VMA, however, pte_protnone() reliably indicates PROT_NONE
- * page protection due to NUMA hinting. NUMA hinting faults only apply in
- * accessible VMAs.
+ * In an accessible VMA, pte_protnone() reliably indicates a present
+ * PROT_NONE page protection. Today the kernel uses such PTEs for two
+ * purposes: NUMA hinting faults, and userfaultfd RWP tracking on
+ * VM_UFFD_RWP VMAs. The two are distinguished by the uffd PTE bit and
+ * the VMA flag; see include/linux/userfaultfd_k.h.
*
- * So, to reliably identify PROT_NONE PTEs that require a NUMA hinting fault,
- * looking at the VMA accessibility is sufficient.
+ * So, to reliably identify PROT_NONE PTEs that require kernel handling,
+ * looking at the VMA accessibility (and the uffd bit on RWP VMAs) is
+ * sufficient.
+ *
+ * Architectures without CONFIG_ARCH_HAS_PTE_PROTNONE get the always-zero
+ * stubs below; PAGE_NONE references that survive to runtime fire the
+ * BUILD_BUG() fallback, since callers should have folded such paths to
+ * dead code via IS_ENABLED(CONFIG_ARCH_HAS_PTE_PROTNONE).
*/
static inline int pte_protnone(pte_t pte)
{
@@ -2151,7 +2159,11 @@ static inline int pmd_protnone(pmd_t pmd)
{
return 0;
}
-#endif /* CONFIG_NUMA_BALANCING */
+
+#ifndef PAGE_NONE
+#define PAGE_NONE ({ BUILD_BUG(); (pgprot_t){0}; })
+#endif
+#endif /* CONFIG_ARCH_HAS_PTE_PROTNONE */
#endif /* CONFIG_MMU */
diff --git a/include/linux/swapops.h b/include/linux/swapops.h
index 8cfc966eae48eb..15c6440e38dd8c 100644
--- a/include/linux/swapops.h
+++ b/include/linux/swapops.h
@@ -73,8 +73,8 @@ static inline pte_t pte_swp_clear_flags(pte_t pte)
pte = pte_swp_clear_exclusive(pte);
if (pte_swp_soft_dirty(pte))
pte = pte_swp_clear_soft_dirty(pte);
- if (pte_swp_uffd_wp(pte))
- pte = pte_swp_clear_uffd_wp(pte);
+ if (pte_swp_uffd(pte))
+ pte = pte_swp_clear_uffd(pte);
return pte;
}
diff --git a/include/linux/userfaultfd_k.h b/include/linux/userfaultfd_k.h
index 68edac4dcd78b3..ca718e8590c5cb 100644
--- a/include/linux/userfaultfd_k.h
+++ b/include/linux/userfaultfd_k.h
@@ -21,10 +21,11 @@
#include <linux/hugetlb_inline.h>
/* The set of all possible UFFD-related VM flags. */
-#define __VM_UFFD_FLAGS (VM_UFFD_MISSING | VM_UFFD_WP | VM_UFFD_MINOR)
+#define __VM_UFFD_FLAGS (VM_UFFD_MISSING | VM_UFFD_MINOR | \
+ VM_UFFD_WP | VM_UFFD_RWP)
#define __VMA_UFFD_FLAGS mk_vma_flags_from_masks(VMA_UFFD_MISSING, VMA_UFFD_WP, \
- VMA_UFFD_MINOR)
+ VMA_UFFD_MINOR, VMA_UFFD_RWP)
/*
* CAREFUL: Check include/uapi/asm-generic/fcntl.h when defining
@@ -149,6 +150,8 @@ static inline uffd_flags_t uffd_flags_set_mode(uffd_flags_t flags, enum mfill_at
extern long uffd_wp_range(struct vm_area_struct *vma,
unsigned long start, unsigned long len, bool enable_wp);
+extern int mrwprotect_range(struct userfaultfd_ctx *ctx, unsigned long start,
+ unsigned long len, bool enable_rwp);
/* move_pages */
void double_pt_lock(spinlock_t *ptl1, spinlock_t *ptl2);
@@ -178,7 +181,9 @@ static inline bool is_mergeable_vm_userfaultfd_ctx(struct vm_area_struct *vma,
*/
static inline bool uffd_disable_huge_pmd_share(struct vm_area_struct *vma)
{
- return vma->vm_flags & (VM_UFFD_WP | VM_UFFD_MINOR);
+ return vma_test_any_mask(vma,
+ mk_vma_flags_from_masks(VMA_UFFD_WP, VMA_UFFD_MINOR,
+ VMA_UFFD_RWP));
}
/*
@@ -190,39 +195,62 @@ static inline bool uffd_disable_huge_pmd_share(struct vm_area_struct *vma)
*/
static inline bool uffd_disable_fault_around(struct vm_area_struct *vma)
{
- return vma->vm_flags & (VM_UFFD_WP | VM_UFFD_MINOR);
+ return vma_test_any_mask(vma,
+ mk_vma_flags_from_masks(VMA_UFFD_WP, VMA_UFFD_MINOR));
}
static inline bool userfaultfd_missing(struct vm_area_struct *vma)
{
- return vma->vm_flags & VM_UFFD_MISSING;
+ return vma_test_any_mask(vma, VMA_UFFD_MISSING);
}
static inline bool userfaultfd_wp(struct vm_area_struct *vma)
{
- return vma->vm_flags & VM_UFFD_WP;
+ return vma_test_any_mask(vma, VMA_UFFD_WP);
}
static inline bool userfaultfd_minor(struct vm_area_struct *vma)
{
- return vma->vm_flags & VM_UFFD_MINOR;
+ return vma_test_any_mask(vma, VMA_UFFD_MINOR);
+}
+
+static inline bool userfaultfd_rwp(struct vm_area_struct *vma)
+{
+ return vma_test_single_mask(vma, VMA_UFFD_RWP);
+}
+
+static inline bool userfaultfd_protected(struct vm_area_struct *vma)
+{
+ return userfaultfd_wp(vma) || userfaultfd_rwp(vma);
}
static inline bool userfaultfd_pte_wp(struct vm_area_struct *vma,
pte_t pte)
{
- return userfaultfd_wp(vma) && pte_uffd_wp(pte);
+ return userfaultfd_wp(vma) && pte_uffd(pte);
}
static inline bool userfaultfd_huge_pmd_wp(struct vm_area_struct *vma,
pmd_t pmd)
{
- return userfaultfd_wp(vma) && pmd_uffd_wp(pmd);
+ return userfaultfd_wp(vma) && pmd_uffd(pmd);
+}
+
+static inline bool userfaultfd_pte_rwp(struct vm_area_struct *vma,
+ pte_t pte)
+{
+ return userfaultfd_rwp(vma) && pte_uffd(pte);
+}
+
+static inline bool userfaultfd_huge_pmd_rwp(struct vm_area_struct *vma,
+ pmd_t pmd)
+{
+ return userfaultfd_rwp(vma) && pmd_uffd(pmd);
}
static inline bool userfaultfd_armed(struct vm_area_struct *vma)
{
- return vma->vm_flags & __VM_UFFD_FLAGS;
+ return vma_test_any_mask(vma, __VMA_UFFD_FLAGS);
}
static inline bool vma_has_uffd_without_event_remap(struct vm_area_struct *vma)
@@ -253,6 +281,7 @@ extern void userfaultfd_unmap_complete(struct mm_struct *mm,
struct list_head *uf);
extern bool userfaultfd_wp_unpopulated(struct vm_area_struct *vma);
extern bool userfaultfd_wp_async(struct vm_area_struct *vma);
+extern bool userfaultfd_rwp_async(struct vm_area_struct *vma);
static inline bool userfaultfd_wp_use_markers(struct vm_area_struct *vma)
{
@@ -272,10 +301,10 @@ static inline bool userfaultfd_wp_use_markers(struct vm_area_struct *vma)
}
/*
- * Returns true if this is a swap pte and was uffd-wp wr-protected in either
- * forms (pte marker or a normal swap pte), false otherwise.
+ * Returns true if this swap pte carries uffd-tracked state in either
+ * form (pte marker or a normal swap pte), false otherwise.
*/
-static inline bool pte_swp_uffd_wp_any(pte_t pte)
+static inline bool pte_swp_uffd_any(pte_t pte)
{
if (!uffd_supports_wp_marker())
return false;
@@ -283,7 +312,7 @@ static inline bool pte_swp_uffd_wp_any(pte_t pte)
if (pte_present(pte))
return false;
- if (pte_swp_uffd_wp(pte))
+ if (pte_swp_uffd(pte))
return true;
if (pte_is_uffd_wp_marker(pte))
@@ -328,6 +357,16 @@ static inline bool userfaultfd_minor(struct vm_area_struct *vma)
return false;
}
+static inline bool userfaultfd_rwp(struct vm_area_struct *vma)
+{
+ return false;
+}
+
+static inline bool userfaultfd_protected(struct vm_area_struct *vma)
+{
+ return false;
+}
+
static inline bool userfaultfd_pte_wp(struct vm_area_struct *vma,
pte_t pte)
{
@@ -340,6 +379,17 @@ static inline bool userfaultfd_huge_pmd_wp(struct vm_area_struct *vma,
return false;
}
+static inline bool userfaultfd_pte_rwp(struct vm_area_struct *vma,
+ pte_t pte)
+{
+ return false;
+}
+
+static inline bool userfaultfd_huge_pmd_rwp(struct vm_area_struct *vma,
+ pmd_t pmd)
+{
+ return false;
+}
static inline bool userfaultfd_armed(struct vm_area_struct *vma)
{
@@ -410,6 +460,11 @@ static inline bool userfaultfd_wp_async(struct vm_area_struct *vma)
return false;
}
+static inline bool userfaultfd_rwp_async(struct vm_area_struct *vma)
+{
+ return false;
+}
+
static inline bool vma_has_uffd_without_event_remap(struct vm_area_struct *vma)
{
return false;
@@ -421,10 +476,10 @@ static inline bool userfaultfd_wp_use_markers(struct vm_area_struct *vma)
}
/*
- * Returns true if this is a swap pte and was uffd-wp wr-protected in either
- * forms (pte marker or a normal swap pte), false otherwise.
+ * Returns true if this swap pte carries uffd-tracked state in either
+ * form (pte marker or a normal swap pte), false otherwise.
*/
-static inline bool pte_swp_uffd_wp_any(pte_t pte)
+static inline bool pte_swp_uffd_any(pte_t pte)
{
return false;
}
diff --git a/include/trace/events/huge_memory.h b/include/trace/events/huge_memory.h
index 291fae364c62d2..5a48c5406cce49 100644
--- a/include/trace/events/huge_memory.h
+++ b/include/trace/events/huge_memory.h
@@ -16,7 +16,7 @@
EM( SCAN_EXCEED_SWAP_PTE, "exceed_swap_pte") \
EM( SCAN_EXCEED_SHARED_PTE, "exceed_shared_pte") \
EM( SCAN_PTE_NON_PRESENT, "pte_non_present") \
- EM( SCAN_PTE_UFFD_WP, "pte_uffd_wp") \
+ EM( SCAN_PTE_UFFD, "pte_uffd_wp") \
EM( SCAN_PTE_MAPPED_HUGEPAGE, "pte_mapped_hugepage") \
EM( SCAN_LACK_REFERENCED_PAGE, "lack_referenced_page") \
EM( SCAN_PAGE_NULL, "page_null") \
diff --git a/include/trace/events/mmflags.h b/include/trace/events/mmflags.h
index a6e5a44c9b4299..bfface3d0203fa 100644
--- a/include/trace/events/mmflags.h
+++ b/include/trace/events/mmflags.h
@@ -194,6 +194,12 @@ IF_HAVE_PG_ARCH_3(arch_3)
# define IF_HAVE_UFFD_MINOR(flag, name)
#endif
+#ifdef CONFIG_USERFAULTFD_RWP
+# define IF_HAVE_UFFD_RWP(flag, name) {flag, name},
+#else
+# define IF_HAVE_UFFD_RWP(flag, name)
+#endif
+
#if defined(CONFIG_64BIT) || defined(CONFIG_PPC32)
# define IF_HAVE_VM_DROPPABLE(flag, name) {flag, name},
#else
@@ -215,6 +221,7 @@ IF_HAVE_UFFD_MINOR(VM_UFFD_MINOR, "uffd_minor" ) \
{VM_PFNMAP, "pfnmap" }, \
{VM_MAYBE_GUARD, "maybe_guard" }, \
{VM_UFFD_WP, "uffd_wp" }, \
+IF_HAVE_UFFD_RWP(VM_UFFD_RWP, "uffd_rwp" ) \
{VM_LOCKED, "locked" }, \
{VM_IO, "io" }, \
{VM_SEQ_READ, "seqread" }, \
diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h
index bd87262f2e3494..34c6f219462a59 100644
--- a/include/uapi/linux/fs.h
+++ b/include/uapi/linux/fs.h
@@ -471,6 +471,7 @@ typedef int __bitwise __kernel_rwf_t;
#define PAGE_IS_HUGE (1 << 6)
#define PAGE_IS_SOFT_DIRTY (1 << 7)
#define PAGE_IS_GUARD (1 << 8)
+#define PAGE_IS_ACCESSED (1 << 9)
/*
* struct page_region - Page region with flags
diff --git a/include/uapi/linux/userfaultfd.h b/include/uapi/linux/userfaultfd.h
index 2841e4ea8f2ce3..cea11aad6b5445 100644
--- a/include/uapi/linux/userfaultfd.h
+++ b/include/uapi/linux/userfaultfd.h
@@ -25,7 +25,8 @@
#define UFFD_API ((__u64)0xAA)
#define UFFD_API_REGISTER_MODES (UFFDIO_REGISTER_MODE_MISSING | \
UFFDIO_REGISTER_MODE_WP | \
- UFFDIO_REGISTER_MODE_MINOR)
+ UFFDIO_REGISTER_MODE_MINOR | \
+ UFFDIO_REGISTER_MODE_RWP)
#define UFFD_API_FEATURES (UFFD_FEATURE_PAGEFAULT_FLAG_WP | \
UFFD_FEATURE_EVENT_FORK | \
UFFD_FEATURE_EVENT_REMAP | \
@@ -42,10 +43,13 @@
UFFD_FEATURE_WP_UNPOPULATED | \
UFFD_FEATURE_POISON | \
UFFD_FEATURE_WP_ASYNC | \
- UFFD_FEATURE_MOVE)
+ UFFD_FEATURE_MOVE | \
+ UFFD_FEATURE_RWP | \
+ UFFD_FEATURE_RWP_ASYNC)
#define UFFD_API_IOCTLS \
((__u64)1 << _UFFDIO_REGISTER | \
(__u64)1 << _UFFDIO_UNREGISTER | \
+ (__u64)1 << _UFFDIO_SET_MODE | \
(__u64)1 << _UFFDIO_API)
#define UFFD_API_RANGE_IOCTLS \
((__u64)1 << _UFFDIO_WAKE | \
@@ -54,13 +58,15 @@
(__u64)1 << _UFFDIO_MOVE | \
(__u64)1 << _UFFDIO_WRITEPROTECT | \
(__u64)1 << _UFFDIO_CONTINUE | \
- (__u64)1 << _UFFDIO_POISON)
+ (__u64)1 << _UFFDIO_POISON | \
+ (__u64)1 << _UFFDIO_RWPROTECT)
#define UFFD_API_RANGE_IOCTLS_BASIC \
((__u64)1 << _UFFDIO_WAKE | \
(__u64)1 << _UFFDIO_COPY | \
(__u64)1 << _UFFDIO_WRITEPROTECT | \
(__u64)1 << _UFFDIO_CONTINUE | \
- (__u64)1 << _UFFDIO_POISON)
+ (__u64)1 << _UFFDIO_POISON | \
+ (__u64)1 << _UFFDIO_RWPROTECT)
/*
* Valid ioctl command number range with this API is from 0x00 to
@@ -79,6 +85,8 @@
#define _UFFDIO_WRITEPROTECT (0x06)
#define _UFFDIO_CONTINUE (0x07)
#define _UFFDIO_POISON (0x08)
+#define _UFFDIO_RWPROTECT (0x09)
+#define _UFFDIO_SET_MODE (0x0A)
#define _UFFDIO_API (0x3F)
/* userfaultfd ioctl ids */
@@ -103,6 +111,10 @@
struct uffdio_continue)
#define UFFDIO_POISON _IOWR(UFFDIO, _UFFDIO_POISON, \
struct uffdio_poison)
+#define UFFDIO_RWPROTECT _IOWR(UFFDIO, _UFFDIO_RWPROTECT, \
+ struct uffdio_rwprotect)
+#define UFFDIO_SET_MODE _IOW(UFFDIO, _UFFDIO_SET_MODE, \
+ struct uffdio_set_mode)
/* read() structure */
struct uffd_msg {
@@ -158,6 +170,7 @@ struct uffd_msg {
#define UFFD_PAGEFAULT_FLAG_WRITE (1<<0) /* If this was a write fault */
#define UFFD_PAGEFAULT_FLAG_WP (1<<1) /* If reason is VM_UFFD_WP */
#define UFFD_PAGEFAULT_FLAG_MINOR (1<<2) /* If reason is VM_UFFD_MINOR */
+#define UFFD_PAGEFAULT_FLAG_RWP (1<<3) /* If reason is VM_UFFD_RWP */
struct uffdio_api {
/* userland asks for an API number and the features to enable */
@@ -230,6 +243,18 @@ struct uffdio_api {
*
* UFFD_FEATURE_MOVE indicates that the kernel supports moving an
* existing page contents from userspace.
+ *
+ * UFFD_FEATURE_RWP indicates that the kernel supports
+ * UFFDIO_REGISTER_MODE_RWP for read-write protection tracking.
+ * Pages are made inaccessible via UFFDIO_RWPROTECT and faults
+ * are delivered when the pages are re-accessed.
+ *
+ * UFFD_FEATURE_RWP_ASYNC indicates asynchronous mode for
+ * UFFDIO_REGISTER_MODE_RWP. When set, faults on read-write
+ * protected pages are auto-resolved by the kernel (PTE
+ * permissions restored immediately) without delivering a message
+ * to the userfaultfd handler. Use PAGEMAP_SCAN with inverted
+ * PAGE_IS_ACCESSED to find pages that were not re-accessed.
*/
#define UFFD_FEATURE_PAGEFAULT_FLAG_WP (1<<0)
#define UFFD_FEATURE_EVENT_FORK (1<<1)
@@ -248,6 +273,8 @@ struct uffdio_api {
#define UFFD_FEATURE_POISON (1<<14)
#define UFFD_FEATURE_WP_ASYNC (1<<15)
#define UFFD_FEATURE_MOVE (1<<16)
+#define UFFD_FEATURE_RWP (1<<17)
+#define UFFD_FEATURE_RWP_ASYNC (1<<18)
__u64 features;
__u64 ioctls;
@@ -263,6 +290,7 @@ struct uffdio_register {
#define UFFDIO_REGISTER_MODE_MISSING ((__u64)1<<0)
#define UFFDIO_REGISTER_MODE_WP ((__u64)1<<1)
#define UFFDIO_REGISTER_MODE_MINOR ((__u64)1<<2)
+#define UFFDIO_REGISTER_MODE_RWP ((__u64)1<<3)
__u64 mode;
/*
@@ -356,6 +384,14 @@ struct uffdio_poison {
__s64 updated;
};
+struct uffdio_rwprotect {
+ struct uffdio_range range;
+ /* !RWP means undo RWP-protection */
+#define UFFDIO_RWPROTECT_MODE_RWP ((__u64)1<<0)
+#define UFFDIO_RWPROTECT_MODE_DONTWAKE ((__u64)1<<1)
+ __u64 mode;
+};
+
struct uffdio_move {
__u64 dst;
__u64 src;
@@ -374,6 +410,16 @@ struct uffdio_move {
__s64 move;
};
+struct uffdio_set_mode {
+ /*
+ * Toggle async mode for features at runtime.
+ * Supported: UFFD_FEATURE_RWP_ASYNC.
+ * Setting a bit in both enable and disable is invalid.
+ */
+ __u64 enable;
+ __u64 disable;
+};
+
/*
* Flags for the userfaultfd(2) system call itself.
*/
diff --git a/init/Kconfig b/init/Kconfig
index 5230d4879b1c84..9d91074c5d0a30 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -944,6 +944,13 @@ config SCHED_PROXY_EXEC
endmenu
+#
+# For architectures that support present-but-inaccessible (PROT_NONE) page
+# table entries detectable via pte_protnone() / pmd_protnone():
+#
+config ARCH_HAS_PTE_PROTNONE
+ bool
+
#
# For architectures that want to enable the support for NUMA-affine scheduler
# balancing logic:
@@ -1010,6 +1017,7 @@ config ARCH_WANT_NUMA_VARIABLE_LOCALITY
config NUMA_BALANCING
bool "Memory placement aware NUMA scheduler"
depends on ARCH_SUPPORTS_NUMA_BALANCING
+ depends on ARCH_HAS_PTE_PROTNONE
depends on !ARCH_WANT_NUMA_VARIABLE_LOCALITY
depends on SMP && NUMA_MIGRATION && !PREEMPT_RT
help
diff --git a/mm/Kconfig b/mm/Kconfig
index 9e0ca48249054f..ccb71653515515 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -1386,6 +1386,15 @@ config HAVE_ARCH_USERFAULTFD_MINOR
help
Arch has userfaultfd minor fault support
+config USERFAULTFD_RWP
+ def_bool y
+ depends on 64BIT && ARCH_HAS_PTE_PROTNONE && HAVE_ARCH_USERFAULTFD_WP
+ help
+ Userfaultfd read-write protection (UFFDIO_RWPROTECT) delivers a
+ userfaultfd notification on every access -- read or write -- to a
+ protected range, letting userspace observe the working set of a
+ process.
+
menuconfig USERFAULTFD
bool "Enable userfaultfd() system call"
depends on MMU
diff --git a/mm/debug_vm_pgtable.c b/mm/debug_vm_pgtable.c
index 23dc3ee0956192..5e9f3a35f924d9 100644
--- a/mm/debug_vm_pgtable.c
+++ b/mm/debug_vm_pgtable.c
@@ -672,7 +672,7 @@ static void __init pte_protnone_tests(struct pgtable_debug_args *args)
{
pte_t pte = pfn_pte(args->fixed_pte_pfn, args->page_prot_none);
- if (!IS_ENABLED(CONFIG_NUMA_BALANCING))
+ if (!IS_ENABLED(CONFIG_ARCH_HAS_PTE_PROTNONE))
return;
pr_debug("Validating PTE protnone\n");
@@ -685,7 +685,7 @@ static void __init pmd_protnone_tests(struct pgtable_debug_args *args)
{
pmd_t pmd;
- if (!IS_ENABLED(CONFIG_NUMA_BALANCING))
+ if (!IS_ENABLED(CONFIG_ARCH_HAS_PTE_PROTNONE))
return;
if (!has_transparent_hugepage())
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 2bccb0a53a0a60..400a35ee536fb0 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -1822,8 +1822,8 @@ static void copy_huge_non_present_pmd(
pmd = swp_entry_to_pmd(entry);
if (pmd_swp_soft_dirty(*src_pmd))
pmd = pmd_swp_mksoft_dirty(pmd);
- if (pmd_swp_uffd_wp(*src_pmd))
- pmd = pmd_swp_mkuffd_wp(pmd);
+ if (pmd_swp_uffd(*src_pmd))
+ pmd = pmd_swp_mkuffd(pmd);
set_pmd_at(src_mm, addr, src_pmd, pmd);
} else if (softleaf_is_device_private(entry)) {
/*
@@ -1836,8 +1836,8 @@ static void copy_huge_non_present_pmd(
if (pmd_swp_soft_dirty(*src_pmd))
pmd = pmd_swp_mksoft_dirty(pmd);
- if (pmd_swp_uffd_wp(*src_pmd))
- pmd = pmd_swp_mkuffd_wp(pmd);
+ if (pmd_swp_uffd(*src_pmd))
+ pmd = pmd_swp_mkuffd(pmd);
set_pmd_at(src_mm, addr, src_pmd, pmd);
}
@@ -1856,8 +1856,8 @@ static void copy_huge_non_present_pmd(
add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR);
mm_inc_nr_ptes(dst_mm);
pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable);
- if (!userfaultfd_wp(dst_vma))
- pmd = pmd_swp_clear_uffd_wp(pmd);
+ if (!userfaultfd_protected(dst_vma))
+ pmd = pmd_swp_clear_uffd(pmd);
set_pmd_at(dst_mm, addr, dst_pmd, pmd);
}
@@ -1951,9 +1951,15 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
out_zero_page:
mm_inc_nr_ptes(dst_mm);
pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable);
+
+ /* See __copy_present_ptes(): restore accessible protection. */
+ if (!userfaultfd_protected(dst_vma)) {
+ if (userfaultfd_rwp(src_vma) && pmd_uffd(pmd))
+ pmd = pmd_modify(pmd, dst_vma->vm_page_prot);
+ pmd = pmd_clear_uffd(pmd);
+ }
+
pmdp_set_wrprotect(src_mm, addr, src_pmd);
- if (!userfaultfd_wp(dst_vma))
- pmd = pmd_clear_uffd_wp(pmd);
pmd = pmd_wrprotect(pmd);
set_pmd:
pmd = pmd_mkold(pmd);
@@ -2196,6 +2202,34 @@ static inline bool can_change_pmd_writable(struct vm_area_struct *vma,
return pmd_dirty(pmd);
}
+vm_fault_t do_huge_pmd_uffd_rwp(struct vm_fault *vmf)
+{
+ struct vm_area_struct *vma = vmf->vma;
+ pmd_t pmd;
+
+ if (!userfaultfd_rwp_async(vma))
+ return handle_userfault(vmf, VM_UFFD_RWP);
+
+ vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);
+ if (unlikely(!pmd_same(pmdp_get(vmf->pmd), vmf->orig_pmd))) {
+ spin_unlock(vmf->ptl);
+ return 0;
+ }
+ pmd = pmd_modify(vmf->orig_pmd, vma->vm_page_prot);
+ /* pmd_modify() preserves _PAGE_UFFD; drop it on resolution */
+ pmd = pmd_clear_uffd(pmd);
+ pmd = pmd_mkyoung(pmd);
+ if (!pmd_write(pmd) &&
+ vma_wants_manual_pte_write_upgrade(vma) &&
+ can_change_pmd_writable(vma, vmf->address, pmd))
+ pmd = pmd_mkwrite(pmd, vma);
+ set_pmd_at(vma->vm_mm, vmf->address & HPAGE_PMD_MASK,
+ vmf->pmd, pmd);
+ update_mmu_cache_pmd(vma, vmf->address, vmf->pmd);
+ spin_unlock(vmf->ptl);
+ return 0;
+}
+
/* NUMA hinting page fault entry point for trans huge pmds */
vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
{
@@ -2494,9 +2528,9 @@ static pmd_t clear_uffd_wp_pmd(pmd_t pmd)
if (pmd_none(pmd))
return pmd;
if (pmd_present(pmd))
- pmd = pmd_clear_uffd_wp(pmd);
+ pmd = pmd_clear_uffd(pmd);
else
- pmd = pmd_swp_clear_uffd_wp(pmd);
+ pmd = pmd_swp_clear_uffd(pmd);
return pmd;
}
@@ -2539,8 +2573,16 @@ bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr,
pgtable_trans_huge_deposit(mm, new_pmd, pgtable);
}
pmd = move_soft_dirty_pmd(pmd);
- if (vma_has_uffd_without_event_remap(vma))
+ if (vma_has_uffd_without_event_remap(vma)) {
+ /*
+ * See __copy_present_ptes(): normalise RWP PMDs so
+ * the destination starts accessible instead of taking
+ * a numa-hinting fault on first access.
+ */
+ if (pmd_present(pmd) && userfaultfd_rwp(vma))
+ pmd = pmd_modify(pmd, vma->vm_page_prot);
pmd = clear_uffd_wp_pmd(pmd);
+ }
set_pmd_at(mm, new_addr, new_pmd, pmd);
if (force_flush)
flush_pmd_tlb_range(vma, old_addr, old_addr + PMD_SIZE);
@@ -2553,8 +2595,8 @@ bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr,
}
static void change_non_present_huge_pmd(struct mm_struct *mm,
- unsigned long addr, pmd_t *pmd, bool uffd_wp,
- bool uffd_wp_resolve)
+ unsigned long addr, pmd_t *pmd, bool uffd_prot,
+ bool uffd_prot_resolve)
{
softleaf_t entry = softleaf_from_pmd(*pmd);
const struct folio *folio = softleaf_to_folio(entry);
@@ -2576,16 +2618,16 @@ static void change_non_present_huge_pmd(struct mm_struct *mm,
} else if (softleaf_is_device_private_write(entry)) {
entry = make_readable_device_private_entry(swp_offset(entry));
newpmd = swp_entry_to_pmd(entry);
- if (pmd_swp_uffd_wp(*pmd))
- newpmd = pmd_swp_mkuffd_wp(newpmd);
+ if (pmd_swp_uffd(*pmd))
+ newpmd = pmd_swp_mkuffd(newpmd);
} else {
newpmd = *pmd;
}
- if (uffd_wp)
- newpmd = pmd_swp_mkuffd_wp(newpmd);
- else if (uffd_wp_resolve)
- newpmd = pmd_swp_clear_uffd_wp(newpmd);
+ if (uffd_prot)
+ newpmd = pmd_swp_mkuffd(newpmd);
+ else if (uffd_prot_resolve)
+ newpmd = pmd_swp_clear_uffd(newpmd);
if (!pmd_same(*pmd, newpmd))
set_pmd_at(mm, addr, pmd, newpmd);
}
@@ -2605,8 +2647,9 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
spinlock_t *ptl;
pmd_t oldpmd, entry;
bool prot_numa = cp_flags & MM_CP_PROT_NUMA;
- bool uffd_wp = cp_flags & MM_CP_UFFD_WP;
- bool uffd_wp_resolve = cp_flags & MM_CP_UFFD_WP_RESOLVE;
+ bool uffd_prot = cp_flags & (MM_CP_UFFD_WP | MM_CP_UFFD_RWP);
+ bool uffd_prot_resolve = cp_flags &
+ (MM_CP_UFFD_WP_RESOLVE | MM_CP_UFFD_RWP_RESOLVE);
int ret = 1;
tlb_change_page_size(tlb, HPAGE_PMD_SIZE);
@@ -2619,11 +2662,17 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
return 0;
if (thp_migration_supported() && pmd_is_valid_softleaf(*pmd)) {
- change_non_present_huge_pmd(mm, addr, pmd, uffd_wp,
- uffd_wp_resolve);
+ change_non_present_huge_pmd(mm, addr, pmd, uffd_prot,
+ uffd_prot_resolve);
goto unlock;
}
+ /* Already in the desired state */
+ if (prot_numa && pmd_protnone(*pmd))
+ goto unlock;
+ if ((cp_flags & MM_CP_UFFD_RWP) && pmd_protnone(*pmd) && pmd_uffd(*pmd))
+ goto unlock;
+
if (prot_numa) {
/*
@@ -2634,9 +2683,6 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
if (is_huge_zero_pmd(*pmd))
goto unlock;
- if (pmd_protnone(*pmd))
- goto unlock;
-
if (!folio_can_map_prot_numa(pmd_folio(*pmd), vma,
vma_is_single_threaded_private(vma)))
goto unlock;
@@ -2665,15 +2711,19 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
oldpmd = pmdp_invalidate_ad(vma, addr, pmd);
entry = pmd_modify(oldpmd, newprot);
- if (uffd_wp)
- entry = pmd_mkuffd_wp(entry);
- else if (uffd_wp_resolve)
+ if (uffd_prot)
+ entry = pmd_mkuffd(entry);
+ else if (uffd_prot_resolve)
/*
* Leave the write bit to be handled by PF interrupt
* handler, then things like COW could be properly
* handled.
*/
- entry = pmd_clear_uffd_wp(entry);
+ entry = pmd_clear_uffd(entry);
+
+ /* See change_pte_range(): preserve RWP protection across mprotect() */
+ if (userfaultfd_rwp(vma) && pmd_uffd(entry))
+ entry = pmd_modify(entry, PAGE_NONE);
/* See change_pte_range(). */
if ((cp_flags & MM_CP_TRY_CHANGE_WRITABLE) && !pmd_write(entry) &&
@@ -2842,6 +2892,13 @@ int move_pages_huge_pmd(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, pm
_dst_pmd = move_soft_dirty_pmd(src_pmdval);
_dst_pmd = clear_uffd_wp_pmd(_dst_pmd);
}
+
+ /* Re-arm RWP on the moved PMD if dst_vma is RWP-registered. */
+ if (userfaultfd_rwp(dst_vma)) {
+ _dst_pmd = pmd_modify(_dst_pmd, PAGE_NONE);
+ _dst_pmd = pmd_mkuffd(_dst_pmd);
+ }
+
set_pmd_at(mm, dst_addr, dst_pmd, _dst_pmd);
src_pgtable = pgtable_trans_huge_withdraw(mm, src_pmd);
@@ -3016,8 +3073,13 @@ static void __split_huge_zero_page_pmd(struct vm_area_struct *vma,
entry = pfn_pte(zero_pfn(addr), vma->vm_page_prot);
entry = pte_mkspecial(entry);
- if (pmd_uffd_wp(old_pmd))
- entry = pte_mkuffd_wp(entry);
+ if (pmd_uffd(old_pmd))
+ entry = pte_mkuffd(entry);
+
+ /* Restore PAGE_NONE so an RWP marker keeps trapping */
+ if (userfaultfd_rwp(vma) && pmd_uffd(old_pmd))
+ entry = pte_modify(entry, PAGE_NONE);
+
VM_BUG_ON(!pte_none(ptep_get(pte)));
set_pte_at(mm, addr, pte, entry);
pte++;
@@ -3103,7 +3165,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,
folio = page_folio(page);
soft_dirty = pmd_swp_soft_dirty(old_pmd);
- uffd_wp = pmd_swp_uffd_wp(old_pmd);
+ uffd_wp = pmd_swp_uffd(old_pmd);
write = softleaf_is_migration_write(entry);
if (PageAnon(page))
@@ -3119,7 +3181,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,
folio = page_folio(page);
soft_dirty = pmd_swp_soft_dirty(old_pmd);
- uffd_wp = pmd_swp_uffd_wp(old_pmd);
+ uffd_wp = pmd_swp_uffd(old_pmd);
write = softleaf_is_device_private_write(entry);
anon_exclusive = PageAnonExclusive(page);
@@ -3176,7 +3238,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,
write = pmd_write(old_pmd);
young = pmd_young(old_pmd);
soft_dirty = pmd_soft_dirty(old_pmd);
- uffd_wp = pmd_uffd_wp(old_pmd);
+ uffd_wp = pmd_uffd(old_pmd);
VM_WARN_ON_FOLIO(!folio_ref_count(folio), folio);
VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio);
@@ -3247,7 +3309,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,
if (soft_dirty)
entry = pte_swp_mksoft_dirty(entry);
if (uffd_wp)
- entry = pte_swp_mkuffd_wp(entry);
+ entry = pte_swp_mkuffd(entry);
VM_WARN_ON(!pte_none(ptep_get(pte + i)));
set_pte_at(mm, addr, pte + i, entry);
}
@@ -3274,7 +3336,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,
if (soft_dirty)
entry = pte_swp_mksoft_dirty(entry);
if (uffd_wp)
- entry = pte_swp_mkuffd_wp(entry);
+ entry = pte_swp_mkuffd(entry);
VM_WARN_ON(!pte_none(ptep_get(pte + i)));
set_pte_at(mm, addr, pte + i, entry);
}
@@ -3292,7 +3354,11 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,
if (soft_dirty)
entry = pte_mksoft_dirty(entry);
if (uffd_wp)
- entry = pte_mkuffd_wp(entry);
+ entry = pte_mkuffd(entry);
+
+ /* Restore PAGE_NONE so an RWP marker keeps trapping */
+ if (userfaultfd_rwp(vma) && uffd_wp)
+ entry = pte_modify(entry, PAGE_NONE);
for (i = 0; i < HPAGE_PMD_NR; i++)
VM_WARN_ON(!pte_none(ptep_get(pte + i)));
@@ -4887,7 +4953,7 @@ int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw,
writable = pmd_write(pmdval);
softdirty = pmd_soft_dirty(pmdval);
- uffd_wp = pmd_uffd_wp(pmdval);
+ uffd_wp = pmd_uffd(pmdval);
} else {
softleaf_t old_entry;
@@ -4896,7 +4962,7 @@ int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw,
writable = softleaf_is_device_private_write(old_entry);
softdirty = pmd_swp_soft_dirty(pmdval);
- uffd_wp = pmd_swp_uffd_wp(pmdval);
+ uffd_wp = pmd_swp_uffd(pmdval);
}
/* See folio_try_share_anon_rmap_pmd(): invalidate PMD first. */
@@ -4927,7 +4993,7 @@ int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw,
if (softdirty)
pmdswp = pmd_swp_mksoft_dirty(pmdswp);
if (uffd_wp)
- pmdswp = pmd_swp_mkuffd_wp(pmdswp);
+ pmdswp = pmd_swp_mkuffd(pmdswp);
set_pmd_at(mm, address, pvmw->pmd, pmdswp);
/* Migration entry installed: cleanup rmap, folio. */
@@ -4959,8 +5025,13 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new)
pmde = pmd_mksoft_dirty(pmde);
if (softleaf_is_migration_write(entry))
pmde = pmd_mkwrite(pmde, vma);
- if (pmd_swp_uffd_wp(*pvmw->pmd))
- pmde = pmd_mkuffd_wp(pmde);
+ if (pmd_swp_uffd(*pvmw->pmd))
+ pmde = pmd_mkuffd(pmde);
+
+ /* See do_swap_page(): restore PAGE_NONE for RWP */
+ if (pmd_swp_uffd(*pvmw->pmd) && userfaultfd_rwp(vma))
+ pmde = pmd_modify(pmde, PAGE_NONE);
+
if (!softleaf_is_migration_young(entry))
pmde = pmd_mkold(pmde);
/* NOTE: this may contain setting soft-dirty on some archs */
@@ -4980,8 +5051,8 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new)
if (pmd_swp_soft_dirty(*pvmw->pmd))
pmde = pmd_swp_mksoft_dirty(pmde);
- if (pmd_swp_uffd_wp(*pvmw->pmd))
- pmde = pmd_swp_mkuffd_wp(pmde);
+ if (pmd_swp_uffd(*pvmw->pmd))
+ pmde = pmd_swp_mkuffd(pmde);
}
if (folio_test_anon(folio)) {
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 571212b80835ec..9da52d95b3fb76 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -4843,8 +4843,16 @@ hugetlb_install_folio(struct vm_area_struct *vma, pte_t *ptep, unsigned long add
__folio_mark_uptodate(new_folio);
hugetlb_add_new_anon_rmap(new_folio, vma, addr);
- if (userfaultfd_wp(vma) && huge_pte_uffd_wp(old))
- newpte = huge_pte_mkuffd_wp(newpte);
+ if (userfaultfd_protected(vma) && huge_pte_uffd(old)) {
+ newpte = huge_pte_mkuffd(newpte);
+ /* Restore PAGE_NONE so the RWP marker keeps trapping. */
+ if (userfaultfd_rwp(vma)) {
+ unsigned int shift = huge_page_shift(hstate_vma(vma));
+
+ newpte = huge_pte_modify(newpte, PAGE_NONE);
+ newpte = arch_make_huge_pte(newpte, shift, vma->vm_flags);
+ }
+ }
set_huge_pte_at(vma->vm_mm, addr, ptep, newpte, sz);
hugetlb_count_add(pages_per_huge_page(hstate_vma(vma)), vma->vm_mm);
folio_set_hugetlb_migratable(new_folio);
@@ -4917,11 +4925,11 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,
softleaf = softleaf_from_pte(entry);
if (unlikely(softleaf_is_hwpoison(softleaf))) {
- if (!userfaultfd_wp(dst_vma))
- entry = huge_pte_clear_uffd_wp(entry);
+ if (!userfaultfd_protected(dst_vma))
+ entry = huge_pte_clear_uffd(entry);
set_huge_pte_at(dst, addr, dst_pte, entry, sz);
} else if (unlikely(softleaf_is_migration(softleaf))) {
- bool uffd_wp = pte_swp_uffd_wp(entry);
+ bool uffd = pte_swp_uffd(entry);
if (!softleaf_is_migration_read(softleaf) && cow) {
/*
@@ -4931,12 +4939,12 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,
softleaf = make_readable_migration_entry(
swp_offset(softleaf));
entry = swp_entry_to_pte(softleaf);
- if (userfaultfd_wp(src_vma) && uffd_wp)
- entry = pte_swp_mkuffd_wp(entry);
+ if (userfaultfd_protected(src_vma) && uffd)
+ entry = pte_swp_mkuffd(entry);
set_huge_pte_at(src, addr, src_pte, entry, sz);
}
- if (!userfaultfd_wp(dst_vma))
- entry = huge_pte_clear_uffd_wp(entry);
+ if (!userfaultfd_protected(dst_vma))
+ entry = huge_pte_clear_uffd(entry);
set_huge_pte_at(dst, addr, dst_pte, entry, sz);
} else if (unlikely(pte_is_marker(entry))) {
const pte_marker marker = copy_pte_marker(softleaf, dst_vma);
@@ -5000,6 +5008,16 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,
goto next;
}
+ /* See __copy_present_ptes(): restore accessible protection. */
+ if (!userfaultfd_protected(dst_vma)) {
+ if (userfaultfd_rwp(src_vma) && huge_pte_uffd(entry)) {
+ entry = huge_pte_modify(entry, dst_vma->vm_page_prot);
+ entry = arch_make_huge_pte(entry, huge_page_shift(h),
+ dst_vma->vm_flags);
+ }
+ entry = huge_pte_clear_uffd(entry);
+ }
+
if (cow) {
/*
* No need to notify as we are downgrading page
@@ -5012,9 +5030,6 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,
entry = huge_pte_wrprotect(entry);
}
- if (!userfaultfd_wp(dst_vma))
- entry = huge_pte_clear_uffd_wp(entry);
-
set_huge_pte_at(dst, addr, dst_pte, entry, sz);
hugetlb_count_add(npages, dst);
}
@@ -5060,10 +5075,22 @@ static void move_huge_pte(struct vm_area_struct *vma, unsigned long old_addr,
huge_pte_clear(mm, new_addr, dst_pte, sz);
} else {
if (need_clear_uffd_wp) {
- if (pte_present(pte))
- pte = huge_pte_clear_uffd_wp(pte);
- else
- pte = pte_swp_clear_uffd_wp(pte);
+ if (pte_present(pte)) {
+ /*
+ * See __copy_present_ptes(): normalise RWP
+ * PTEs so the destination starts accessible
+ * instead of taking a numa-hinting fault on
+ * first access.
+ */
+ if (userfaultfd_rwp(vma)) {
+ pte = huge_pte_modify(pte, vma->vm_page_prot);
+ pte = arch_make_huge_pte(pte, huge_page_shift(h),
+ vma->vm_flags);
+ }
+ pte = huge_pte_clear_uffd(pte);
+ } else {
+ pte = pte_swp_clear_uffd(pte);
+ }
}
set_huge_pte_at(mm, new_addr, dst_pte, pte, sz);
}
@@ -5197,7 +5224,7 @@ void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma,
* drop the uffd-wp bit in this zap, then replace the
* pte with a marker.
*/
- if (pte_swp_uffd_wp_any(pte) &&
+ if (pte_swp_uffd_any(pte) &&
!(zap_flags & ZAP_FLAG_DROP_MARKER))
set_huge_pte_at(mm, address, ptep,
make_pte_marker(PTE_MARKER_UFFD_WP),
@@ -5233,7 +5260,7 @@ void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma,
if (huge_pte_dirty(pte))
folio_mark_dirty(folio);
/* Leave a uffd-wp pte marker if needed */
- if (huge_pte_uffd_wp(pte) &&
+ if (huge_pte_uffd(pte) &&
!(zap_flags & ZAP_FLAG_DROP_MARKER))
set_huge_pte_at(mm, address, ptep,
make_pte_marker(PTE_MARKER_UFFD_WP),
@@ -5437,7 +5464,7 @@ static vm_fault_t hugetlb_wp(struct vm_fault *vmf)
* can trigger this, because hugetlb_fault() will always resolve
* uffd-wp bit first.
*/
- if (!unshare && huge_pte_uffd_wp(pte))
+ if (!unshare && huge_pte_uffd(pte))
return 0;
/* Let's take out MAP_SHARED mappings first. */
@@ -5581,8 +5608,8 @@ static vm_fault_t hugetlb_wp(struct vm_fault *vmf)
huge_ptep_clear_flush(vma, vmf->address, vmf->pte);
hugetlb_remove_rmap(old_folio);
hugetlb_add_new_anon_rmap(new_folio, vma, vmf->address);
- if (huge_pte_uffd_wp(pte))
- newpte = huge_pte_mkuffd_wp(newpte);
+ if (huge_pte_uffd(pte))
+ newpte = huge_pte_mkuffd(newpte);
set_huge_pte_at(mm, vmf->address, vmf->pte, newpte,
huge_page_size(h));
folio_set_hugetlb_migratable(new_folio);
@@ -5860,7 +5887,7 @@ static vm_fault_t hugetlb_no_page(struct address_space *mapping,
* if populated.
*/
if (unlikely(pte_is_uffd_wp_marker(vmf->orig_pte)))
- new_pte = huge_pte_mkuffd_wp(new_pte);
+ new_pte = huge_pte_mkuffd(new_pte);
set_huge_pte_at(mm, vmf->address, vmf->pte, new_pte, huge_page_size(h));
hugetlb_count_add(pages_per_huge_page(h), mm);
@@ -6035,6 +6062,47 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,
goto out_mutex;
}
+ /*
+ * Protnone hugetlb PTEs with the uffd bit are used by
+ * userfaultfd RWP for access tracking. Plain PROT_NONE (without the
+ * marker) is not an RWP fault and is not expected on hugetlb (no
+ * NUMA hinting), so let normal hugetlb fault handling proceed.
+ */
+ if (pte_protnone(vmf.orig_pte) && vma_is_accessible(vma) &&
+ userfaultfd_rwp(vma) && huge_pte_uffd(vmf.orig_pte)) {
+ spinlock_t *ptl;
+ pte_t pte;
+
+ /* Sync: drop hugetlb locks before blocking in handle_userfault() */
+ if (!userfaultfd_rwp_async(vma))
+ return hugetlb_handle_userfault(&vmf, mapping, VM_UFFD_RWP);
+
+ ptl = huge_pte_lock(h, mm, vmf.pte);
+ pte = huge_ptep_get(mm, vmf.address, vmf.pte);
+ if (pte_protnone(pte) && huge_pte_uffd(pte)) {
+ unsigned int shift = huge_page_shift(h);
+
+ pte = huge_pte_modify(pte, vma->vm_page_prot);
+ pte = arch_make_huge_pte(pte, shift, vma->vm_flags);
+ /* huge_pte_modify() preserves _PAGE_UFFD; drop it on resolution */
+ pte = huge_pte_clear_uffd(pte);
+ pte = pte_mkyoung(pte);
+ /*
+ * Unlike do_uffd_rwp(), do not upgrade to writable
+ * here. Hugetlb lacks a can_change_huge_pte_writable()
+ * equivalent, so a write access will take a separate
+ * COW fault — acceptable for the rare private hugetlb
+ * case.
+ */
+ set_huge_pte_at(mm, vmf.address, vmf.pte, pte,
+ huge_page_size(h));
+ update_mmu_cache(vma, vmf.address, vmf.pte);
+ }
+ spin_unlock(ptl);
+ ret = 0;
+ goto out_mutex;
+ }
+
/*
* If we are going to COW/unshare the mapping later, we examine the
* pending reservations for this page now. This will ensure that any
@@ -6058,7 +6126,7 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,
goto out_ptl;
/* Handle userfault-wp first, before trying to lock more pages */
- if (userfaultfd_wp(vma) && huge_pte_uffd_wp(huge_ptep_get(mm, vmf.address, vmf.pte)) &&
+ if (userfaultfd_wp(vma) && huge_pte_uffd(huge_ptep_get(mm, vmf.address, vmf.pte)) &&
(flags & FAULT_FLAG_WRITE) && !huge_pte_write(vmf.orig_pte)) {
if (!userfaultfd_wp_async(vma)) {
spin_unlock(vmf.ptl);
@@ -6067,7 +6135,7 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,
return handle_userfault(&vmf, VM_UFFD_WP);
}
- vmf.orig_pte = huge_pte_clear_uffd_wp(vmf.orig_pte);
+ vmf.orig_pte = huge_pte_clear_uffd(vmf.orig_pte);
set_huge_pte_at(mm, vmf.address, vmf.pte, vmf.orig_pte,
huge_page_size(hstate_vma(vma)));
/* Fallthrough to CoW */
@@ -6352,7 +6420,7 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte,
_dst_pte = pte_mkyoung(_dst_pte);
if (wp_enabled)
- _dst_pte = huge_pte_mkuffd_wp(_dst_pte);
+ _dst_pte = huge_pte_mkuffd(_dst_pte);
set_huge_pte_at(dst_mm, dst_addr, dst_pte, _dst_pte, size);
@@ -6395,6 +6463,8 @@ long hugetlb_change_protection(struct vm_area_struct *vma,
unsigned long last_addr_mask;
bool uffd_wp = cp_flags & MM_CP_UFFD_WP;
bool uffd_wp_resolve = cp_flags & MM_CP_UFFD_WP_RESOLVE;
+ bool uffd_rwp = cp_flags & MM_CP_UFFD_RWP;
+ bool uffd_rwp_resolve = cp_flags & MM_CP_UFFD_RWP_RESOLVE;
struct mmu_gather tlb;
/*
@@ -6420,6 +6490,11 @@ long hugetlb_change_protection(struct vm_area_struct *vma,
ptep = hugetlb_walk(vma, address, psize);
if (!ptep) {
+ /*
+ * uffd_wp installs a pte marker on the unpopulated
+ * entry; uffd_rwp does not install markers so the
+ * allocation is unnecessary for it.
+ */
if (!uffd_wp) {
address |= last_addr_mask;
continue;
@@ -6441,7 +6516,8 @@ long hugetlb_change_protection(struct vm_area_struct *vma,
* shouldn't happen at all. Warn about it if it
* happened due to some reason.
*/
- WARN_ON_ONCE(uffd_wp || uffd_wp_resolve);
+ WARN_ON_ONCE(uffd_wp || uffd_wp_resolve ||
+ uffd_rwp || uffd_rwp_resolve);
pages++;
spin_unlock(ptl);
address |= last_addr_mask;
@@ -6475,10 +6551,10 @@ long hugetlb_change_protection(struct vm_area_struct *vma,
pages++;
}
- if (uffd_wp)
- newpte = pte_swp_mkuffd_wp(newpte);
- else if (uffd_wp_resolve)
- newpte = pte_swp_clear_uffd_wp(newpte);
+ if (uffd_wp || uffd_rwp)
+ newpte = pte_swp_mkuffd(newpte);
+ else if (uffd_wp_resolve || uffd_rwp_resolve)
+ newpte = pte_swp_clear_uffd(newpte);
if (!pte_same(pte, newpte))
set_huge_pte_at(mm, address, ptep, newpte, psize);
} else if (unlikely(pte_is_marker(pte))) {
@@ -6488,20 +6564,32 @@ long hugetlb_change_protection(struct vm_area_struct *vma,
* pte_marker_uffd_wp()==true implies !poison
* because they're mutual exclusive.
*/
- if (pte_is_uffd_wp_marker(pte) && uffd_wp_resolve)
+ if (pte_is_uffd_wp_marker(pte) &&
+ (uffd_wp_resolve || uffd_rwp_resolve))
/* Safe to modify directly (non-present->none). */
huge_pte_clear(mm, address, ptep, psize);
} else {
pte_t old_pte;
unsigned int shift = huge_page_shift(hstate_vma(vma));
+ /* Already protnone with uffd bit set? Nothing to do. */
+ if (uffd_rwp && pte_protnone(pte) && huge_pte_uffd(pte))
+ goto next;
+
old_pte = huge_ptep_modify_prot_start(vma, address, ptep);
pte = huge_pte_modify(old_pte, newprot);
pte = arch_make_huge_pte(pte, shift, vma->vm_flags);
- if (uffd_wp)
- pte = huge_pte_mkuffd_wp(pte);
- else if (uffd_wp_resolve)
- pte = huge_pte_clear_uffd_wp(pte);
+ if (uffd_wp || uffd_rwp)
+ pte = huge_pte_mkuffd(pte);
+ else if (uffd_wp_resolve || uffd_rwp_resolve)
+ pte = huge_pte_clear_uffd(pte);
+
+ /* Preserve RWP protection across mprotect() */
+ if (userfaultfd_rwp(vma) && huge_pte_uffd(pte)) {
+ pte = huge_pte_modify(pte, PAGE_NONE);
+ pte = arch_make_huge_pte(pte, shift, vma->vm_flags);
+ }
+
huge_ptep_modify_prot_commit(vma, address, ptep, old_pte, pte);
pages++;
tlb_remove_huge_tlb_entry(h, &tlb, ptep, address);
diff --git a/mm/internal.h b/mm/internal.h
index 181e79f1d6a207..87bf7fbd0e9034 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -412,8 +412,8 @@ static inline pte_t pte_move_swp_offset(pte_t pte, long delta)
new = pte_swp_mksoft_dirty(new);
if (pte_swp_exclusive(pte))
new = pte_swp_mkexclusive(new);
- if (pte_swp_uffd_wp(pte))
- new = pte_swp_mkuffd_wp(new);
+ if (pte_swp_uffd(pte))
+ new = pte_swp_mkuffd(new);
return new;
}
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 617bca76db49b6..483bcebd6977d0 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -37,7 +37,7 @@ enum scan_result {
SCAN_EXCEED_SWAP_PTE,
SCAN_EXCEED_SHARED_PTE,
SCAN_PTE_NON_PRESENT,
- SCAN_PTE_UFFD_WP,
+ SCAN_PTE_UFFD,
SCAN_PTE_MAPPED_HUGEPAGE,
SCAN_LACK_REFERENCED_PAGE,
SCAN_PAGE_NULL,
@@ -695,8 +695,8 @@ static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma,
result = SCAN_PTE_NON_PRESENT;
goto out;
}
- if (pte_uffd_wp(pteval)) {
- result = SCAN_PTE_UFFD_WP;
+ if (pte_uffd(pteval)) {
+ result = SCAN_PTE_UFFD;
goto out;
}
page = vm_normal_page(vma, addr, pteval);
@@ -1543,7 +1543,7 @@ static enum scan_result mthp_collapse(struct mm_struct *mm,
case SCAN_PAGE_NULL:
case SCAN_DEL_PAGE_LRU:
case SCAN_PTE_NON_PRESENT:
- case SCAN_PTE_UFFD_WP:
+ case SCAN_PTE_UFFD:
case SCAN_PAGE_LAZYFREE:
last_result = ret;
goto next_order;
@@ -1664,15 +1664,15 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
/*
* Always be strict with uffd-wp
* enabled swap entries. Please see
- * comment below for pte_uffd_wp().
+ * comment below for pte_uffd().
*/
- if (pte_swp_uffd_wp_any(pteval)) {
- result = SCAN_PTE_UFFD_WP;
+ if (pte_swp_uffd_any(pteval)) {
+ result = SCAN_PTE_UFFD;
goto out_unmap;
}
continue;
}
- if (pte_uffd_wp(pteval)) {
+ if (pte_uffd(pteval)) {
/*
* Don't collapse the page if any of the small
* PTEs are armed with uffd write protection.
@@ -1682,7 +1682,7 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
* userfault messages that falls outside of
* the registered range. So, just be simple.
*/
- result = SCAN_PTE_UFFD_WP;
+ result = SCAN_PTE_UFFD;
goto out_unmap;
}
@@ -1892,9 +1892,12 @@ static enum scan_result try_collapse_pte_mapped_thp(struct mm_struct *mm, unsign
if (!thp_vma_allowable_order(vma, vma->vm_flags, TVA_FORCED_COLLAPSE, PMD_ORDER))
return SCAN_VMA_CHECK;
- /* Keep pmd pgtable for uffd-wp; see comment in retract_page_tables() */
- if (userfaultfd_wp(vma))
- return SCAN_PTE_UFFD_WP;
+ /*
+ * Keep pmd pgtable while the uffd bit is in use; see comment in
+ * retract_page_tables().
+ */
+ if (userfaultfd_protected(vma))
+ return SCAN_PTE_UFFD;
folio = filemap_lock_folio(vma->vm_file->f_mapping,
linear_page_index(vma, haddr));
@@ -2106,13 +2109,14 @@ static bool file_backed_vma_is_retractable(struct vm_area_struct *vma)
return false;
/*
- * When a vma is registered with uffd-wp, we cannot recycle
+ * When a vma is registered with uffd-wp or RWP, we cannot recycle
* the page table because there may be pte markers installed.
- * Other vmas can still have the same file mapped hugely, but
- * skip this one: it will always be mapped in small page size
- * for uffd-wp registered ranges.
+ * VM_UFFD_RWP ranges similarly rely on per-PTE uffd state
+ * and cannot be recycled to a shared PMD. Other vmas can still
+ * have the same file mapped hugely, but skip this one: it will
+ * always be mapped in small page size for these registrations.
*/
- if (userfaultfd_wp(vma))
+ if (userfaultfd_protected(vma))
return false;
/*
@@ -3241,7 +3245,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
/* Whitelisted set of results where continuing OK */
case SCAN_NO_PTE_TABLE:
case SCAN_PTE_NON_PRESENT:
- case SCAN_PTE_UFFD_WP:
+ case SCAN_PTE_UFFD:
case SCAN_LACK_REFERENCED_PAGE:
case SCAN_PAGE_NULL:
case SCAN_PAGE_COUNT:
diff --git a/mm/memory.c b/mm/memory.c
index ff338c2abe9231..20abe12da481cf 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -893,8 +893,12 @@ static void restore_exclusive_pte(struct vm_area_struct *vma,
if (pte_swp_soft_dirty(orig_pte))
pte = pte_mksoft_dirty(pte);
- if (pte_swp_uffd_wp(orig_pte))
- pte = pte_mkuffd_wp(pte);
+ if (pte_swp_uffd(orig_pte))
+ pte = pte_mkuffd(pte);
+
+ /* See do_swap_page(): restore PAGE_NONE for RWP */
+ if (pte_swp_uffd(orig_pte) && userfaultfd_rwp(vma))
+ pte = pte_modify(pte, PAGE_NONE);
if ((vma->vm_flags & VM_WRITE) &&
can_change_pte_writable(vma, address, pte)) {
@@ -984,8 +988,8 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,
pte = softleaf_to_pte(entry);
if (pte_swp_soft_dirty(orig_pte))
pte = pte_swp_mksoft_dirty(pte);
- if (pte_swp_uffd_wp(orig_pte))
- pte = pte_swp_mkuffd_wp(pte);
+ if (pte_swp_uffd(orig_pte))
+ pte = pte_swp_mkuffd(pte);
set_pte_at(src_mm, addr, src_pte, pte);
}
} else if (softleaf_is_device_private(entry)) {
@@ -1018,8 +1022,8 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,
entry = make_readable_device_private_entry(
swp_offset(entry));
pte = swp_entry_to_pte(entry);
- if (pte_swp_uffd_wp(orig_pte))
- pte = pte_swp_mkuffd_wp(pte);
+ if (pte_swp_uffd(orig_pte))
+ pte = pte_swp_mkuffd(pte);
set_pte_at(src_mm, addr, src_pte, pte);
}
} else if (softleaf_is_device_exclusive(entry)) {
@@ -1041,8 +1045,8 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,
make_pte_marker(marker));
return 0;
}
- if (!userfaultfd_wp(dst_vma))
- pte = pte_swp_clear_uffd_wp(pte);
+ if (!userfaultfd_protected(dst_vma))
+ pte = pte_swp_clear_uffd(pte);
set_pte_at(dst_mm, addr, dst_pte, pte);
return 0;
}
@@ -1088,9 +1092,13 @@ copy_present_page(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma
/* All done, just insert the new page copy in the child */
pte = folio_mk_pte(new_folio, dst_vma->vm_page_prot);
pte = maybe_mkwrite(pte_mkdirty(pte), dst_vma);
- if (userfaultfd_pte_wp(dst_vma, ptep_get(src_pte)))
- /* Uffd-wp needs to be delivered to dest pte as well */
- pte = pte_mkuffd_wp(pte);
+ if (userfaultfd_protected(dst_vma) && pte_uffd(ptep_get(src_pte))) {
+ /* The uffd bit needs to be delivered to the dest pte as well */
+ pte = pte_mkuffd(pte);
+ /* Restore PAGE_NONE so the RWP marker keeps trapping */
+ if (userfaultfd_rwp(dst_vma))
+ pte = pte_modify(pte, PAGE_NONE);
+ }
set_pte_at(dst_vma->vm_mm, addr, dst_pte, pte);
return 0;
}
@@ -1100,9 +1108,31 @@ static __always_inline void __copy_present_ptes(struct vm_area_struct *dst_vma,
pte_t pte, unsigned long addr, int nr)
{
struct mm_struct *src_mm = src_vma->vm_mm;
+ bool writable;
+
+ /*
+ * Snapshot writability before the RWP-disarm rewrite below: when the
+ * child is not RWP-armed, pte_modify(pte, dst_vma->vm_page_prot) can
+ * silently drop _PAGE_RW from a resolved (no-marker) writable PTE,
+ * so a later pte_write(pte) check would skip the COW wrprotect and
+ * leave the parent writable over a folio shared with the child.
+ */
+ writable = pte_write(pte);
+
+ /*
+ * Child is not RWP-armed: restore accessible protection so the
+ * inherited PAGE_NONE does not cost a fault on first read. Gate on
+ * pte_uffd(pte) so unrelated PAGE_NONE markers (e.g. NUMA balancing)
+ * are not normalised away.
+ */
+ if (!userfaultfd_protected(dst_vma)) {
+ if (userfaultfd_rwp(src_vma) && pte_uffd(pte))
+ pte = pte_modify(pte, dst_vma->vm_page_prot);
+ pte = pte_clear_uffd(pte);
+ }
/* If it's a COW mapping, write protect it both processes. */
- if (is_cow_mapping(src_vma->vm_flags) && pte_write(pte)) {
+ if (is_cow_mapping(src_vma->vm_flags) && writable) {
wrprotect_ptes(src_mm, addr, src_pte, nr);
pte = pte_wrprotect(pte);
}
@@ -1112,9 +1142,6 @@ static __always_inline void __copy_present_ptes(struct vm_area_struct *dst_vma,
pte = pte_mkclean(pte);
pte = pte_mkold(pte);
- if (!userfaultfd_wp(dst_vma))
- pte = pte_clear_uffd_wp(pte);
-
set_ptes(dst_vma->vm_mm, addr, dst_pte, pte, nr);
}
@@ -3925,8 +3952,8 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf)
if (unlikely(unshare)) {
if (pte_soft_dirty(vmf->orig_pte))
entry = pte_mksoft_dirty(entry);
- if (pte_uffd_wp(vmf->orig_pte))
- entry = pte_mkuffd_wp(entry);
+ if (pte_uffd(vmf->orig_pte))
+ entry = pte_mkuffd(entry);
} else {
entry = maybe_mkwrite(pte_mkdirty(entry), vma);
}
@@ -4261,7 +4288,7 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf)
* etc.) because we're only removing the uffd-wp bit,
* which is completely invisible to the user.
*/
- pte = pte_clear_uffd_wp(ptep_get(vmf->pte));
+ pte = pte_clear_uffd(ptep_get(vmf->pte));
set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte);
/*
@@ -5038,8 +5065,16 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
pte = mk_pte(page, vma->vm_page_prot);
if (pte_swp_soft_dirty(vmf->orig_pte))
pte = pte_mksoft_dirty(pte);
- if (pte_swp_uffd_wp(vmf->orig_pte))
- pte = pte_mkuffd_wp(pte);
+ if (pte_swp_uffd(vmf->orig_pte))
+ pte = pte_mkuffd(pte);
+
+ /*
+ * A page reclaimed while RWP-protected carries the uffd bit on
+ * its swap entry. Re-apply PAGE_NONE on swap-in so the first access
+ * still traps as an RWP fault. pte_modify() preserves _PAGE_UFFD.
+ */
+ if (pte_swp_uffd(vmf->orig_pte) && userfaultfd_rwp(vma))
+ pte = pte_modify(pte, PAGE_NONE);
/*
* Same logic as in do_wp_page(); however, optimize for pages that are
@@ -5259,7 +5294,7 @@ void map_anon_folio_pte_nopf(struct folio *folio, pte_t *pte,
if (vma->vm_flags & VM_WRITE)
entry = pte_mkwrite(pte_mkdirty(entry), vma);
if (uffd_wp)
- entry = pte_mkuffd_wp(entry);
+ entry = pte_mkuffd(entry);
folio_ref_add(folio, nr_pages - 1);
folio_add_new_anon_rmap(folio, vma, addr, RMAP_EXCLUSIVE);
@@ -5326,7 +5361,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
return handle_userfault(vmf, VM_UFFD_MISSING);
}
if (vmf_orig_pte_uffd_wp(vmf))
- entry = pte_mkuffd_wp(entry);
+ entry = pte_mkuffd(entry);
set_pte_at(vma->vm_mm, addr, vmf->pte, entry);
/* No need to invalidate - it was non-present before */
@@ -5576,7 +5611,7 @@ void set_pte_range(struct vm_fault *vmf, struct folio *folio,
else if (pte_write(entry) && folio_test_dirty(folio))
entry = pte_mkdirty(entry);
if (unlikely(vmf_orig_pte_uffd_wp(vmf)))
- entry = pte_mkuffd_wp(entry);
+ entry = pte_mkuffd(entry);
/* copy-on-write page */
if (write && !(vma->vm_flags & VM_SHARED)) {
VM_BUG_ON_FOLIO(nr != 1, folio);
@@ -6091,6 +6126,16 @@ static void numa_rebuild_large_mapping(struct vm_fault *vmf, struct vm_area_stru
if (!pte_present(ptent) || !pte_protnone(ptent))
continue;
+ /*
+ * RWP-armed PTEs are also protnone but carry _PAGE_UFFD as a
+ * marker. Leave them alone -- rewriting to vm_page_prot would
+ * stop the RWP trap. Gate on userfaultfd_rwp(vma) too:
+ * NUMA balancing preserves _PAGE_UFFD on UFFD_WP-marked PTEs
+ * when applying PROT_NONE, and those still need rebuilding.
+ */
+ if (userfaultfd_rwp(vma) && pte_uffd(ptent))
+ continue;
+
if (pfn_folio(pte_pfn(ptent)) != folio)
continue;
@@ -6106,6 +6151,35 @@ static void numa_rebuild_large_mapping(struct vm_fault *vmf, struct vm_area_stru
}
}
+static vm_fault_t do_uffd_rwp(struct vm_fault *vmf)
+{
+ pte_t pte;
+
+ if (!userfaultfd_rwp_async(vmf->vma)) {
+ /* Sync mode: unmap PTE and deliver to userfaultfd handler */
+ pte_unmap(vmf->pte);
+ return handle_userfault(vmf, VM_UFFD_RWP);
+ }
+
+ spin_lock(vmf->ptl);
+ if (unlikely(!pte_same(ptep_get(vmf->pte), vmf->orig_pte))) {
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
+ return 0;
+ }
+ pte = pte_modify(vmf->orig_pte, vmf->vma->vm_page_prot);
+ /* pte_modify() preserves _PAGE_UFFD; drop it on resolution */
+ pte = pte_clear_uffd(pte);
+ pte = pte_mkyoung(pte);
+ if (!pte_write(pte) &&
+ vma_wants_manual_pte_write_upgrade(vmf->vma) &&
+ can_change_pte_writable(vmf->vma, vmf->address, pte))
+ pte = pte_mkwrite(pte, vmf->vma);
+ set_pte_at(vmf->vma->vm_mm, vmf->address, vmf->pte, pte);
+ update_mmu_cache(vmf->vma, vmf->address, vmf->pte);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
+ return 0;
+}
+
static vm_fault_t do_numa_page(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
@@ -6381,8 +6455,16 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf)
if (!pte_present(vmf->orig_pte))
return do_swap_page(vmf);
- if (pte_protnone(vmf->orig_pte) && vma_is_accessible(vmf->vma))
+ if (pte_protnone(vmf->orig_pte) && vma_is_accessible(vmf->vma)) {
+ /*
+ * RWP-protected PTEs are protnone plus the uffd bit. On a
+ * VM_UFFD_RWP VMA, a protnone PTE without the uffd bit is
+ * NUMA hinting and must still fall through to do_numa_page().
+ */
+ if (userfaultfd_pte_rwp(vmf->vma, vmf->orig_pte))
+ return do_uffd_rwp(vmf);
return do_numa_page(vmf);
+ }
spin_lock(vmf->ptl);
entry = vmf->orig_pte;
@@ -6496,8 +6578,11 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma,
return 0;
}
if (pmd_trans_huge(vmf.orig_pmd)) {
- if (pmd_protnone(vmf.orig_pmd) && vma_is_accessible(vma))
+ if (pmd_protnone(vmf.orig_pmd) && vma_is_accessible(vma)) {
+ if (userfaultfd_huge_pmd_rwp(vma, vmf.orig_pmd))
+ return do_huge_pmd_uffd_rwp(&vmf);
return do_huge_pmd_numa_page(&vmf);
+ }
if ((flags & (FAULT_FLAG_WRITE|FAULT_FLAG_UNSHARE)) &&
!pmd_write(vmf.orig_pmd)) {
diff --git a/mm/migrate.c b/mm/migrate.c
index d9b23909d716c6..9516cda35e341c 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -326,8 +326,12 @@ static bool try_to_map_unused_to_zeropage(struct page_vma_mapped_walk *pvmw,
if (pte_swp_soft_dirty(old_pte))
newpte = pte_mksoft_dirty(newpte);
- if (pte_swp_uffd_wp(old_pte))
- newpte = pte_mkuffd_wp(newpte);
+ if (pte_swp_uffd(old_pte))
+ newpte = pte_mkuffd(newpte);
+
+ /* See remove_migration_pte(): restore PAGE_NONE for RWP */
+ if (pte_swp_uffd(old_pte) && userfaultfd_rwp(pvmw->vma))
+ newpte = pte_modify(newpte, PAGE_NONE);
set_pte_at(pvmw->vma->vm_mm, pvmw->address, pvmw->pte, newpte);
@@ -391,8 +395,12 @@ static bool remove_migration_pte(struct folio *folio,
if (softleaf_is_migration_write(entry))
pte = pte_mkwrite(pte, vma);
- else if (pte_swp_uffd_wp(old_pte))
- pte = pte_mkuffd_wp(pte);
+ else if (pte_swp_uffd(old_pte))
+ pte = pte_mkuffd(pte);
+
+ /* See do_swap_page(): restore PAGE_NONE for RWP */
+ if (pte_swp_uffd(old_pte) && userfaultfd_rwp(vma))
+ pte = pte_modify(pte, PAGE_NONE);
if (folio_test_anon(folio) && !softleaf_is_migration_read(entry))
rmap_flags |= RMAP_EXCLUSIVE;
@@ -407,8 +415,8 @@ static bool remove_migration_pte(struct folio *folio,
pte = softleaf_to_pte(entry);
if (pte_swp_soft_dirty(old_pte))
pte = pte_swp_mksoft_dirty(pte);
- if (pte_swp_uffd_wp(old_pte))
- pte = pte_swp_mkuffd_wp(pte);
+ if (pte_swp_uffd(old_pte))
+ pte = pte_swp_mkuffd(pte);
}
#ifdef CONFIG_HUGETLB_PAGE
diff --git a/mm/migrate_device.c b/mm/migrate_device.c
index 554754eb26ff2f..17da1bab02484e 100644
--- a/mm/migrate_device.c
+++ b/mm/migrate_device.c
@@ -445,13 +445,13 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
if (pte_present(pte)) {
if (pte_soft_dirty(pte))
swp_pte = pte_swp_mksoft_dirty(swp_pte);
- if (pte_uffd_wp(pte))
- swp_pte = pte_swp_mkuffd_wp(swp_pte);
+ if (pte_uffd(pte))
+ swp_pte = pte_swp_mkuffd(swp_pte);
} else {
if (pte_swp_soft_dirty(pte))
swp_pte = pte_swp_mksoft_dirty(swp_pte);
- if (pte_swp_uffd_wp(pte))
- swp_pte = pte_swp_mkuffd_wp(swp_pte);
+ if (pte_swp_uffd(pte))
+ swp_pte = pte_swp_mkuffd(swp_pte);
}
set_pte_at(mm, addr, ptep, swp_pte);
diff --git a/mm/mprotect.c b/mm/mprotect.c
index 9cbf932b028cf9..cc85a8862c2818 100644
--- a/mm/mprotect.c
+++ b/mm/mprotect.c
@@ -214,8 +214,9 @@ static __always_inline void set_write_prot_commit_flush_ptes(struct vm_area_stru
static long change_softleaf_pte(struct vm_area_struct *vma,
unsigned long addr, pte_t *pte, pte_t oldpte, unsigned long cp_flags)
{
- const bool uffd_wp = cp_flags & MM_CP_UFFD_WP;
- const bool uffd_wp_resolve = cp_flags & MM_CP_UFFD_WP_RESOLVE;
+ const bool uffd_prot = cp_flags & (MM_CP_UFFD_WP | MM_CP_UFFD_RWP);
+ const bool uffd_prot_resolve = cp_flags &
+ (MM_CP_UFFD_WP_RESOLVE | MM_CP_UFFD_RWP_RESOLVE);
softleaf_t entry = softleaf_from_pte(oldpte);
pte_t newpte;
@@ -240,8 +241,8 @@ static long change_softleaf_pte(struct vm_area_struct *vma,
*/
entry = make_readable_device_private_entry(swp_offset(entry));
newpte = swp_entry_to_pte(entry);
- if (pte_swp_uffd_wp(oldpte))
- newpte = pte_swp_mkuffd_wp(newpte);
+ if (pte_swp_uffd(oldpte))
+ newpte = pte_swp_mkuffd(newpte);
} else if (softleaf_is_marker(entry)) {
/*
* Ignore error swap entries unconditionally,
@@ -256,7 +257,7 @@ static long change_softleaf_pte(struct vm_area_struct *vma,
* to unprotect it, drop it; the next page
* fault will trigger without uffd trapping.
*/
- if (uffd_wp_resolve) {
+ if (uffd_prot_resolve) {
pte_clear(vma->vm_mm, addr, pte);
return 1;
}
@@ -265,10 +266,10 @@ static long change_softleaf_pte(struct vm_area_struct *vma,
newpte = oldpte;
}
- if (uffd_wp)
- newpte = pte_swp_mkuffd_wp(newpte);
- else if (uffd_wp_resolve)
- newpte = pte_swp_clear_uffd_wp(newpte);
+ if (uffd_prot)
+ newpte = pte_swp_mkuffd(newpte);
+ else if (uffd_prot_resolve)
+ newpte = pte_swp_clear_uffd(newpte);
if (!pte_same(oldpte, newpte)) {
set_pte_at(vma->vm_mm, addr, pte, newpte);
@@ -282,17 +283,28 @@ static __always_inline void change_present_ptes(struct mmu_gather *tlb,
int nr_ptes, unsigned long end, pgprot_t newprot,
struct folio *folio, struct page *page, unsigned long cp_flags)
{
- const bool uffd_wp_resolve = cp_flags & MM_CP_UFFD_WP_RESOLVE;
- const bool uffd_wp = cp_flags & MM_CP_UFFD_WP;
+ const bool uffd_prot = cp_flags & (MM_CP_UFFD_WP | MM_CP_UFFD_RWP);
+ const bool uffd_prot_resolve = cp_flags &
+ (MM_CP_UFFD_WP_RESOLVE | MM_CP_UFFD_RWP_RESOLVE);
pte_t ptent, oldpte;
oldpte = modify_prot_start_ptes(vma, addr, ptep, nr_ptes);
ptent = pte_modify(oldpte, newprot);
- if (uffd_wp)
- ptent = pte_mkuffd_wp(ptent);
- else if (uffd_wp_resolve)
- ptent = pte_clear_uffd_wp(ptent);
+ if (uffd_prot)
+ ptent = pte_mkuffd(ptent);
+ else if (uffd_prot_resolve)
+ ptent = pte_clear_uffd(ptent);
+
+ /*
+ * The uffd bit on a VM_UFFD_RWP VMA carries PROT_NONE
+ * semantics. If mprotect() or NUMA hinting changed the
+ * base protection, restore PAGE_NONE so the PTE still
+ * traps on any access. pte_modify() preserves
+ * _PAGE_UFFD.
+ */
+ if (userfaultfd_rwp(vma) && pte_uffd(ptent))
+ ptent = pte_modify(ptent, PAGE_NONE);
/*
* In some writable, shared mappings, we might want
@@ -325,6 +337,7 @@ static long change_pte_range(struct mmu_gather *tlb,
long pages = 0;
bool is_private_single_threaded;
bool prot_numa = cp_flags & MM_CP_PROT_NUMA;
+ bool uffd_rwp = cp_flags & MM_CP_UFFD_RWP;
bool uffd_wp = cp_flags & MM_CP_UFFD_WP;
int nr_ptes;
@@ -350,6 +363,14 @@ static long change_pte_range(struct mmu_gather *tlb,
/* Already in the desired state. */
if (prot_numa && pte_protnone(oldpte))
continue;
+ /*
+ * RWP-protected PTEs carry _PAGE_UFFD as a marker on
+ * top of PROT_NONE. Skip only entries already in that
+ * exact state; plain PROT_NONE from mprotect() still needs
+ * to be promoted so future faults can be distinguished.
+ */
+ if (uffd_rwp && pte_protnone(oldpte) && pte_uffd(oldpte))
+ continue;
page = vm_normal_page(vma, addr, oldpte);
if (page)
@@ -358,6 +379,8 @@ static long change_pte_range(struct mmu_gather *tlb,
/*
* Avoid trapping faults against the zero or KSM
* pages. See similar comment in change_huge_pmd.
+ * Skip this filter for uffd RWP which
+ * must set protnone regardless of NUMA placement.
*/
if (prot_numa &&
!folio_can_map_prot_numa(folio, vma,
@@ -428,7 +451,7 @@ pgtable_split_needed(struct vm_area_struct *vma, unsigned long cp_flags)
* (e.g. 2M shmem) because file thp is handled differently when
* split by erasing the pmd so far.
*/
- return (cp_flags & MM_CP_UFFD_WP) && !vma_is_anonymous(vma);
+ return (cp_flags & (MM_CP_UFFD_WP | MM_CP_UFFD_RWP)) && !vma_is_anonymous(vma);
}
/*
@@ -667,7 +690,16 @@ long change_protection(struct mmu_gather *tlb,
pgprot_t newprot = vma->vm_page_prot;
long pages;
- BUG_ON((cp_flags & MM_CP_UFFD_WP_ALL) == MM_CP_UFFD_WP_ALL);
+ /*
+ * MM_CP_UFFD_{WP,RWP} and _RESOLVE are mutually exclusive within one
+ * change, and WP and RWP cannot mix. Miswired callers get a warn and
+ * a no-op; userspace cannot reach this state.
+ */
+ if (WARN_ON_ONCE((cp_flags & MM_CP_UFFD_WP_ALL) == MM_CP_UFFD_WP_ALL ||
+ (cp_flags & MM_CP_UFFD_RWP_ALL) == MM_CP_UFFD_RWP_ALL ||
+ ((cp_flags & MM_CP_UFFD_WP_ALL) &&
+ (cp_flags & MM_CP_UFFD_RWP_ALL))))
+ return 0;
#ifdef CONFIG_NUMA_BALANCING
/*
@@ -681,6 +713,10 @@ long change_protection(struct mmu_gather *tlb,
WARN_ON_ONCE(cp_flags & MM_CP_PROT_NUMA);
#endif
+ if (IS_ENABLED(CONFIG_ARCH_HAS_PTE_PROTNONE) &&
+ (cp_flags & MM_CP_UFFD_RWP))
+ newprot = PAGE_NONE;
+
if (is_vm_hugetlb_page(vma))
pages = hugetlb_change_protection(vma, start, end, newprot,
cp_flags);
diff --git a/mm/mremap.c b/mm/mremap.c
index e9c8b1d05832be..8a46ec5831c84e 100644
--- a/mm/mremap.c
+++ b/mm/mremap.c
@@ -296,10 +296,19 @@ static int move_ptes(struct pagetable_move_control *pmc,
pte_clear(mm, new_addr, new_ptep);
else {
if (need_clear_uffd_wp) {
- if (pte_present(pte))
- pte = pte_clear_uffd_wp(pte);
- else
- pte = pte_swp_clear_uffd_wp(pte);
+ if (pte_present(pte)) {
+ /*
+ * See __copy_present_ptes(): normalise
+ * RWP PTEs so the destination starts
+ * accessible instead of taking a
+ * numa-hinting fault on first access.
+ */
+ if (userfaultfd_rwp(vma) && pte_uffd(pte))
+ pte = pte_modify(pte, vma->vm_page_prot);
+ pte = pte_clear_uffd(pte);
+ } else {
+ pte = pte_swp_clear_uffd(pte);
+ }
}
set_ptes(mm, new_addr, new_ptep, pte, nr_ptes);
}
diff --git a/mm/page_table_check.c b/mm/page_table_check.c
index 53a8997ec043ea..3fb995e5d40d60 100644
--- a/mm/page_table_check.c
+++ b/mm/page_table_check.c
@@ -188,8 +188,8 @@ static inline bool softleaf_cached_writable(softleaf_t entry)
static void page_table_check_pte_flags(pte_t pte)
{
if (pte_present(pte)) {
- WARN_ON_ONCE(pte_uffd_wp(pte) && pte_write(pte));
- } else if (pte_swp_uffd_wp(pte)) {
+ WARN_ON_ONCE(pte_uffd(pte) && pte_write(pte));
+ } else if (pte_swp_uffd(pte)) {
const softleaf_t entry = softleaf_from_pte(pte);
WARN_ON_ONCE(softleaf_cached_writable(entry));
@@ -216,9 +216,9 @@ EXPORT_SYMBOL(__page_table_check_ptes_set);
static inline void page_table_check_pmd_flags(pmd_t pmd)
{
if (pmd_present(pmd)) {
- if (pmd_uffd_wp(pmd))
+ if (pmd_uffd(pmd))
WARN_ON_ONCE(pmd_write(pmd));
- } else if (pmd_swp_uffd_wp(pmd)) {
+ } else if (pmd_swp_uffd(pmd)) {
const softleaf_t entry = softleaf_from_pmd(pmd);
WARN_ON_ONCE(softleaf_cached_writable(entry));
diff --git a/mm/rmap.c b/mm/rmap.c
index 1c77d5dc06e9fa..9fb733489898de 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -1965,7 +1965,7 @@ static inline unsigned int folio_unmap_pte_batch(struct folio *folio,
if (pte_unused(pte))
return 1;
- if (userfaultfd_wp(vma))
+ if (userfaultfd_protected(vma))
return 1;
/*
@@ -2318,13 +2318,13 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
if (likely(pte_present(pteval))) {
if (pte_soft_dirty(pteval))
swp_pte = pte_swp_mksoft_dirty(swp_pte);
- if (pte_uffd_wp(pteval))
- swp_pte = pte_swp_mkuffd_wp(swp_pte);
+ if (pte_uffd(pteval))
+ swp_pte = pte_swp_mkuffd(swp_pte);
} else {
if (pte_swp_soft_dirty(pteval))
swp_pte = pte_swp_mksoft_dirty(swp_pte);
- if (pte_swp_uffd_wp(pteval))
- swp_pte = pte_swp_mkuffd_wp(swp_pte);
+ if (pte_swp_uffd(pteval))
+ swp_pte = pte_swp_mkuffd(swp_pte);
}
set_pte_at(mm, address, pvmw.pte, swp_pte);
} else {
@@ -2692,14 +2692,14 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,
swp_pte = swp_entry_to_pte(entry);
if (pte_soft_dirty(pteval))
swp_pte = pte_swp_mksoft_dirty(swp_pte);
- if (pte_uffd_wp(pteval))
- swp_pte = pte_swp_mkuffd_wp(swp_pte);
+ if (pte_uffd(pteval))
+ swp_pte = pte_swp_mkuffd(swp_pte);
} else {
swp_pte = swp_entry_to_pte(entry);
if (pte_swp_soft_dirty(pteval))
swp_pte = pte_swp_mksoft_dirty(swp_pte);
- if (pte_swp_uffd_wp(pteval))
- swp_pte = pte_swp_mkuffd_wp(swp_pte);
+ if (pte_swp_uffd(pteval))
+ swp_pte = pte_swp_mkuffd(swp_pte);
}
if (folio_test_hugetlb(folio))
set_huge_pte_at(mm, address, pvmw.pte, swp_pte,
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 78b49b0658adde..a6f19759e8e302 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -2496,8 +2496,13 @@ static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd,
new_pte = pte_mkold(mk_pte(page, vma->vm_page_prot));
if (pte_swp_soft_dirty(old_pte))
new_pte = pte_mksoft_dirty(new_pte);
- if (pte_swp_uffd_wp(old_pte))
- new_pte = pte_mkuffd_wp(new_pte);
+ if (pte_swp_uffd(old_pte))
+ new_pte = pte_mkuffd(new_pte);
+
+ /* See do_swap_page(): restore PAGE_NONE for RWP */
+ if (pte_swp_uffd(old_pte) && userfaultfd_rwp(vma))
+ new_pte = pte_modify(new_pte, PAGE_NONE);
+
setpte:
set_pte_at(vma->vm_mm, addr, pte, new_pte);
folio_put_swap(swapcache, folio_file_page(swapcache, swp_offset(entry)));
diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c
index 246af12bf80142..a7659b0a1147ed 100644
--- a/mm/userfaultfd.c
+++ b/mm/userfaultfd.c
@@ -376,7 +376,7 @@ static int mfill_atomic_install_pte(pmd_t *dst_pmd,
if (writable)
_dst_pte = pte_mkwrite(_dst_pte, dst_vma);
if (flags & MFILL_ATOMIC_WP)
- _dst_pte = pte_mkuffd_wp(_dst_pte);
+ _dst_pte = pte_mkuffd(_dst_pte);
ret = -EAGAIN;
dst_pte = pte_offset_map_lock(dst_mm, dst_pmd, dst_addr, &ptl);
@@ -1164,6 +1164,75 @@ static int mwriteprotect_range(struct userfaultfd_ctx *ctx, unsigned long start,
return err;
}
+int mrwprotect_range(struct userfaultfd_ctx *ctx, unsigned long start,
+ unsigned long len, bool enable_rwp)
+{
+ struct mm_struct *dst_mm = ctx->mm;
+ unsigned long end = start + len;
+ struct vm_area_struct *dst_vma;
+ unsigned int mm_cp_flags;
+ struct mmu_gather tlb;
+ bool found = false;
+ VMA_ITERATOR(vmi, dst_mm, start);
+
+ VM_WARN_ON_ONCE(start & ~PAGE_MASK);
+ VM_WARN_ON_ONCE(len & ~PAGE_MASK);
+ VM_WARN_ON_ONCE(start + len <= start);
+
+ guard(mmap_read_lock)(dst_mm);
+ guard(rwsem_read)(&ctx->map_changing_lock);
+
+ if (atomic_read(&ctx->mmap_changing))
+ return -EAGAIN;
+
+ if (enable_rwp)
+ mm_cp_flags = MM_CP_UFFD_RWP;
+ else
+ mm_cp_flags = MM_CP_UFFD_RWP_RESOLVE;
+
+ /*
+ * Pre-scan the range: validate every spanned VMA before applying
+ * any change_protection() so a partial failure cannot leave the
+ * process with only a prefix of the range re-protected.
+ */
+ for_each_vma_range(vmi, dst_vma, end) {
+ if (!userfaultfd_rwp(dst_vma))
+ return -ENOENT;
+
+ if (is_vm_hugetlb_page(dst_vma)) {
+ unsigned long page_mask;
+
+ page_mask = vma_kernel_pagesize(dst_vma) - 1;
+ if ((start & page_mask) || (len & page_mask))
+ return -EINVAL;
+ }
+ found = true;
+ }
+ if (!found)
+ return -ENOENT;
+
+ vma_iter_set(&vmi, start);
+ tlb_gather_mmu(&tlb, dst_mm);
+ for_each_vma_range(vmi, dst_vma, end) {
+ unsigned long vma_start = max(dst_vma->vm_start, start);
+ unsigned long vma_end = min(dst_vma->vm_end, end);
+ unsigned int flags = mm_cp_flags;
+
+ /*
+ * On resolve, try to upgrade writability per-VMA --
+ * MM_CP_TRY_CHANGE_WRITABLE WARNs in
+ * maybe_change_pte_writable() if the VMA is not VM_WRITE,
+ * and RWP can be registered on PROT_READ-only mappings.
+ */
+ if (!enable_rwp && vma_wants_manual_pte_write_upgrade(dst_vma))
+ flags |= MM_CP_TRY_CHANGE_WRITABLE;
+
+ change_protection(&tlb, dst_vma, vma_start, vma_end, flags);
+ }
+ tlb_finish_mmu(&tlb);
+
+ return 0;
+}
void double_pt_lock(spinlock_t *ptl1,
spinlock_t *ptl2)
@@ -1292,6 +1361,13 @@ static long move_present_ptes(struct mm_struct *mm,
if (pte_dirty(orig_src_pte))
orig_dst_pte = pte_mkdirty(orig_dst_pte);
orig_dst_pte = pte_mkwrite(orig_dst_pte, dst_vma);
+
+ /* Re-arm RWP on the moved PTE if dst_vma is RWP-registered. */
+ if (userfaultfd_rwp(dst_vma)) {
+ orig_dst_pte = pte_modify(orig_dst_pte, PAGE_NONE);
+ orig_dst_pte = pte_mkuffd(orig_dst_pte);
+ }
+
set_pte_at(mm, dst_addr, dst_pte, orig_dst_pte);
src_addr += PAGE_SIZE;
@@ -1373,6 +1449,9 @@ static int move_swap_pte(struct mm_struct *mm, struct vm_area_struct *dst_vma,
orig_src_pte = ptep_get_and_clear(mm, src_addr, src_pte);
if (pgtable_supports_soft_dirty())
orig_src_pte = pte_swp_mksoft_dirty(orig_src_pte);
+ /* Re-arm RWP on the moved swap entry if dst_vma is RWP-registered. */
+ if (userfaultfd_rwp(dst_vma))
+ orig_src_pte = pte_swp_mkuffd(orig_src_pte);
set_pte_at(mm, dst_addr, dst_pte, orig_src_pte);
double_pt_unlock(dst_ptl, src_ptl);
@@ -1399,6 +1478,13 @@ static int move_zeropage_pte(struct mm_struct *mm,
zero_pte = pte_mkspecial(pfn_pte(zero_pfn(dst_addr),
dst_vma->vm_page_prot));
+
+ /* Re-arm RWP on the moved PTE if dst_vma is RWP-registered. */
+ if (userfaultfd_rwp(dst_vma)) {
+ zero_pte = pte_modify(zero_pte, PAGE_NONE);
+ zero_pte = pte_mkuffd(zero_pte);
+ }
+
ptep_clear_flush(src_vma, src_addr, src_pte);
set_pte_at(mm, dst_addr, dst_pte, zero_pte);
double_pt_unlock(dst_ptl, src_ptl);
@@ -2135,6 +2221,15 @@ static bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags,
!vma_is_anonymous(vma))
return false;
+ /*
+ * RWP uses protnone as an access-tracking marker. PROT_NONE VMAs
+ * have vm_page_prot == PAGE_NONE, so RWP resolution can't make a
+ * page accessible -- the next access would fault again. Reject up
+ * front instead of letting FOLL_FORCE loop on protnone+uffd PTEs.
+ */
+ if ((vm_flags & VM_UFFD_RWP) && !vma_is_accessible(vma))
+ return false;
+
return ops->can_userfault(vma, vm_flags);
}
@@ -2187,9 +2282,22 @@ static struct vm_area_struct *userfaultfd_clear_vma(struct vma_iterator *vmi,
if (start == vma->vm_start && end == vma->vm_end)
give_up_on_oom = true;
- /* Reset ptes for the whole vma range if wr-protected */
- if (userfaultfd_wp(vma))
- uffd_wp_range(vma, start, end - start, false);
+ /* Clear the uffd bit and/or restore protnone PTEs */
+ if (userfaultfd_protected(vma)) {
+ unsigned int mm_cp_flags = 0;
+ struct mmu_gather tlb;
+
+ if (userfaultfd_wp(vma))
+ mm_cp_flags |= MM_CP_UFFD_WP_RESOLVE;
+ if (userfaultfd_rwp(vma))
+ mm_cp_flags |= MM_CP_UFFD_RWP_RESOLVE;
+ if (vma_wants_manual_pte_write_upgrade(vma))
+ mm_cp_flags |= MM_CP_TRY_CHANGE_WRITABLE;
+
+ tlb_gather_mmu(&tlb, vma->vm_mm);
+ change_protection(&tlb, vma, start, end, mm_cp_flags);
+ tlb_finish_mmu(&tlb);
+ }
ret = vma_modify_flags_uffd(vmi, prev, vma, start, end,
&new_vma_flags, NULL_VM_UFFD_CTX,
@@ -2238,6 +2346,14 @@ static int userfaultfd_register_range(struct userfaultfd_ctx *ctx,
vma_test_all_mask(vma, vma_flags))
goto skip;
+ /*
+ * Pre-scan in userfaultfd_register() already rejected mode
+ * switches that would drop VM_UFFD_WP or VM_UFFD_RWP, so a
+ * stray bit here is a bug.
+ */
+ VM_WARN_ON_ONCE(vma->vm_userfaultfd_ctx.ctx == ctx &&
+ vma->vm_flags & (VM_UFFD_WP | VM_UFFD_RWP) & ~vm_flags);
+
if (vma->vm_start > start)
start = vma->vm_start;
vma_end = min(end, vma->vm_end);
@@ -2368,14 +2484,29 @@ struct userfaultfd_wake_range {
/* internal indication that UFFD_API ioctl was successfully executed */
#define UFFD_FEATURE_INITIALIZED (1u << 31)
+/*
+ * UFFDIO_SET_MODE updates ctx->features under mmap_write_lock with
+ * WRITE_ONCE; readers that run outside mmap_read_lock or the per-VMA
+ * lock (poll/read_iter/ioctl, fdinfo) must pair with READ_ONCE.
+ */
+static unsigned int userfaultfd_features(struct userfaultfd_ctx *ctx)
+{
+ return READ_ONCE(ctx->features);
+}
+
static bool userfaultfd_is_initialized(struct userfaultfd_ctx *ctx)
{
- return ctx->features & UFFD_FEATURE_INITIALIZED;
+ return userfaultfd_features(ctx) & UFFD_FEATURE_INITIALIZED;
}
static bool userfaultfd_wp_async_ctx(struct userfaultfd_ctx *ctx)
{
- return ctx && (ctx->features & UFFD_FEATURE_WP_ASYNC);
+ return ctx && (userfaultfd_features(ctx) & UFFD_FEATURE_WP_ASYNC);
+}
+
+static bool userfaultfd_rwp_async_ctx(struct userfaultfd_ctx *ctx)
+{
+ return ctx && (userfaultfd_features(ctx) & UFFD_FEATURE_RWP_ASYNC);
}
/*
@@ -2390,7 +2521,7 @@ bool userfaultfd_wp_unpopulated(struct vm_area_struct *vma)
if (!ctx)
return false;
- return ctx->features & UFFD_FEATURE_WP_UNPOPULATED;
+ return userfaultfd_features(ctx) & UFFD_FEATURE_WP_UNPOPULATED;
}
static int userfaultfd_wake_function(wait_queue_entry_t *wq, unsigned mode,
@@ -2504,6 +2635,8 @@ static inline struct uffd_msg userfault_msg(unsigned long address,
msg.arg.pagefault.flags |= UFFD_PAGEFAULT_FLAG_WRITE;
if (reason & VM_UFFD_WP)
msg.arg.pagefault.flags |= UFFD_PAGEFAULT_FLAG_WP;
+ if (reason & VM_UFFD_RWP)
+ msg.arg.pagefault.flags |= UFFD_PAGEFAULT_FLAG_RWP;
if (reason & VM_UFFD_MINOR)
msg.arg.pagefault.flags |= UFFD_PAGEFAULT_FLAG_MINOR;
if (features & UFFD_FEATURE_THREAD_ID)
@@ -2557,6 +2690,12 @@ static inline bool userfaultfd_huge_must_wait(struct userfaultfd_ctx *ctx,
*/
if (!huge_pte_write(pte) && (reason & VM_UFFD_WP))
return true;
+ /*
+ * PTE is still RW-protected (protnone with uffd bit), wait for
+ * resolution. Plain PROT_NONE without the marker is not an RWP fault.
+ */
+ if (pte_protnone(pte) && huge_pte_uffd(pte) && (reason & VM_UFFD_RWP))
+ return true;
return false;
}
@@ -2617,8 +2756,14 @@ static inline bool userfaultfd_must_wait(struct userfaultfd_ctx *ctx,
if (!pmd_present(_pmd))
return false;
- if (pmd_trans_huge(_pmd))
- return !pmd_write(_pmd) && (reason & VM_UFFD_WP);
+ if (pmd_trans_huge(_pmd)) {
+ if (!pmd_write(_pmd) && (reason & VM_UFFD_WP))
+ return true;
+ if (pmd_protnone(_pmd) && pmd_uffd(_pmd) &&
+ (reason & VM_UFFD_RWP))
+ return true;
+ return false;
+ }
pte = pte_offset_map(pmd, address);
if (!pte)
@@ -2654,6 +2799,13 @@ static inline bool userfaultfd_must_wait(struct userfaultfd_ctx *ctx,
*/
if (!pte_write(ptent) && (reason & VM_UFFD_WP))
goto out;
+ /*
+ * PTE is still RW-protected (protnone with uffd bit), wait for
+ * userspace to resolve. Plain PROT_NONE without the marker is not
+ * an RWP fault.
+ */
+ if (pte_protnone(ptent) && pte_uffd(ptent) && (reason & VM_UFFD_RWP))
+ goto out;
ret = false;
out:
@@ -3598,11 +3750,27 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx,
if (uffdio_register.mode & UFFDIO_REGISTER_MODE_MISSING)
vm_flags |= VM_UFFD_MISSING;
if (uffdio_register.mode & UFFDIO_REGISTER_MODE_WP) {
- if (!pgtable_supports_uffd_wp())
+ if (!pgtable_supports_uffd())
goto out;
vm_flags |= VM_UFFD_WP;
}
+ if (uffdio_register.mode & UFFDIO_REGISTER_MODE_RWP) {
+ if (!pgtable_supports_uffd() || VM_UFFD_RWP == VM_NONE)
+ goto out;
+ if (!(ctx->features & UFFD_FEATURE_RWP))
+ goto out;
+ vm_flags |= VM_UFFD_RWP;
+ }
+
+ /*
+ * WP and RWP share the uffd PTE bit and
+ * cannot coexist in the same VMA — the bit would carry ambiguous
+ * semantics. Reject the combination up front.
+ */
+ if ((vm_flags & VM_UFFD_WP) && (vm_flags & VM_UFFD_RWP))
+ goto out;
+
if (uffdio_register.mode & UFFDIO_REGISTER_MODE_MINOR) {
#ifndef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR
goto out;
@@ -3696,6 +3864,16 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx,
cur->vm_userfaultfd_ctx.ctx != ctx)
goto out_unlock;
+ /*
+ * Mode switches that drop VM_UFFD_WP or VM_UFFD_RWP would
+ * leave PTE markers without the flag that describes them;
+ * subsequent mprotect() would then promote stale markers
+ * into the other mode. Require an unregister first.
+ */
+ if (cur->vm_userfaultfd_ctx.ctx == ctx &&
+ cur->vm_flags & (VM_UFFD_WP | VM_UFFD_RWP) & ~vm_flags)
+ goto out_unlock;
+
/*
* Note vmas containing huge pages
*/
@@ -3729,6 +3907,10 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx,
if (!(uffdio_register.mode & UFFDIO_REGISTER_MODE_MINOR))
ioctls_out &= ~((__u64)1 << _UFFDIO_CONTINUE);
+ /* RWPROTECT is only supported for RWP ranges */
+ if (!(uffdio_register.mode & UFFDIO_REGISTER_MODE_RWP))
+ ioctls_out &= ~((__u64)1 << _UFFDIO_RWPROTECT);
+
/*
* Now that we scanned all vmas we can already tell
* userland which ioctls methods are guaranteed to
@@ -4076,6 +4258,158 @@ static int userfaultfd_writeprotect(struct userfaultfd_ctx *ctx,
return ret;
}
+static int userfaultfd_rwprotect(struct userfaultfd_ctx *ctx,
+ unsigned long arg)
+{
+ int ret;
+ struct uffdio_rwprotect uffdio_rwp;
+ struct userfaultfd_wake_range range;
+ bool mode_rwp, mode_dontwake;
+
+ if (atomic_read(&ctx->mmap_changing))
+ return -EAGAIN;
+
+ if (copy_from_user(&uffdio_rwp, (void __user *)arg,
+ sizeof(uffdio_rwp)))
+ return -EFAULT;
+
+ ret = validate_range(ctx->mm, uffdio_rwp.range.start,
+ uffdio_rwp.range.len);
+ if (ret)
+ return ret;
+
+ if (uffdio_rwp.mode & ~(UFFDIO_RWPROTECT_MODE_DONTWAKE |
+ UFFDIO_RWPROTECT_MODE_RWP))
+ return -EINVAL;
+
+ mode_rwp = uffdio_rwp.mode & UFFDIO_RWPROTECT_MODE_RWP;
+ mode_dontwake = uffdio_rwp.mode & UFFDIO_RWPROTECT_MODE_DONTWAKE;
+
+ if (mode_rwp && mode_dontwake)
+ return -EINVAL;
+
+ if (mmget_not_zero(ctx->mm)) {
+ ret = mrwprotect_range(ctx, uffdio_rwp.range.start,
+ uffdio_rwp.range.len, mode_rwp);
+ mmput(ctx->mm);
+ } else {
+ return -ESRCH;
+ }
+
+ if (ret)
+ return ret;
+
+ if (!mode_rwp && !mode_dontwake) {
+ range.start = uffdio_rwp.range.start;
+ range.len = uffdio_rwp.range.len;
+ wake_userfault(ctx, &range);
+ }
+ return ret;
+}
+
+/* Subset of UFFD_API_FEATURES actually supported by this kernel/arch */
+static __u64 uffd_api_available_features(void)
+{
+ __u64 f = UFFD_API_FEATURES;
+
+ if (!IS_ENABLED(CONFIG_HAVE_ARCH_USERFAULTFD_MINOR))
+ f &= ~(UFFD_FEATURE_MINOR_HUGETLBFS | UFFD_FEATURE_MINOR_SHMEM);
+ if (!pgtable_supports_uffd())
+ f &= ~UFFD_FEATURE_PAGEFAULT_FLAG_WP;
+ if (!uffd_supports_wp_marker())
+ f &= ~(UFFD_FEATURE_WP_HUGETLBFS_SHMEM |
+ UFFD_FEATURE_WP_UNPOPULATED |
+ UFFD_FEATURE_WP_ASYNC);
+ /*
+ * RWP needs both PROT_NONE support and the uffd PTE bit. The
+ * VM_UFFD_RWP check covers compile-time unavailability; the
+ * pgtable_supports_uffd() check covers runtime (e.g. riscv
+ * without the SVRSW60T59B extension) where the PTE bit is declared
+ * but not actually usable.
+ */
+ if (VM_UFFD_RWP == VM_NONE || !pgtable_supports_uffd())
+ f &= ~(UFFD_FEATURE_RWP | UFFD_FEATURE_RWP_ASYNC);
+ return f;
+}
+
+/* Async features that can be toggled at runtime via UFFDIO_SET_MODE */
+#define UFFD_FEATURE_TOGGLEABLE UFFD_FEATURE_RWP_ASYNC
+
+static int userfaultfd_set_mode(struct userfaultfd_ctx *ctx,
+ unsigned long arg)
+{
+ struct uffdio_set_mode mode;
+ struct mm_struct *mm = ctx->mm;
+
+ if (copy_from_user(&mode, (void __user *)arg, sizeof(mode)))
+ return -EFAULT;
+
+ /* enable and disable must not overlap */
+ if (mode.enable & mode.disable)
+ return -EINVAL;
+
+ /* only toggleable features that this kernel/arch actually supports */
+ if ((mode.enable | mode.disable) &
+ ~(uffd_api_available_features() & UFFD_FEATURE_TOGGLEABLE))
+ return -EINVAL;
+
+ /* RWP_ASYNC can only be enabled on contexts that negotiated RWP */
+ if ((mode.enable & UFFD_FEATURE_RWP_ASYNC) &&
+ !(userfaultfd_features(ctx) & UFFD_FEATURE_RWP))
+ return -EINVAL;
+
+ if (!mmget_not_zero(mm))
+ return -ESRCH;
+
+ /*
+ * Drain in-flight faults before flipping features. mmap_write_lock()
+ * blocks new mmap_read_lock() callers, but per-VMA locked faults
+ * (lock_vma_under_rcu() + FAULT_FLAG_VMA_LOCK) that acquired before
+ * this point keep running. Calling vma_start_write() on each UFFD-
+ * armed VMA waits for those readers to drop, so no in-flight fault
+ * can observe the old features after mmap_write_unlock().
+ */
+ mmap_write_lock(mm);
+ {
+ struct vm_area_struct *vma;
+ VMA_ITERATOR(vmi, mm, 0);
+
+ for_each_vma(vmi, vma) {
+ if (vma->vm_userfaultfd_ctx.ctx == ctx)
+ vma_start_write(vma);
+ }
+ }
+ /*
+ * Single WRITE_ONCE so lockless readers (fdinfo, poll/read_iter
+ * via userfaultfd_is_initialized(), and the userfaultfd_features()
+ * helper used elsewhere) can't observe a mid-RMW intermediate
+ * value. Hot-path readers already serialise through the mmap lock
+ * + vma_start_write() drain above, so their load doesn't need an
+ * annotation.
+ */
+ WRITE_ONCE(ctx->features,
+ (ctx->features | mode.enable) & ~mode.disable);
+ mmap_write_unlock(mm);
+
+ /*
+ * If switching to async, wake threads blocked in handle_userfault().
+ * They will retry the fault and auto-resolve under the new mode.
+ * len=0 means wake all pending faults on this context.
+ */
+ if (mode.enable & UFFD_FEATURE_RWP_ASYNC) {
+ struct userfaultfd_wake_range range = { .len = 0 };
+
+ spin_lock_irq(&ctx->fault_pending_wqh.lock);
+ __wake_up_locked_key(&ctx->fault_pending_wqh, TASK_NORMAL,
+ &range);
+ __wake_up(&ctx->fault_wqh, TASK_NORMAL, 1, &range);
+ spin_unlock_irq(&ctx->fault_pending_wqh.lock);
+ }
+
+ mmput(mm);
+ return 0;
+}
+
static int userfaultfd_continue(struct userfaultfd_ctx *ctx, unsigned long arg)
{
__s64 ret;
@@ -4199,6 +4533,11 @@ bool userfaultfd_wp_async(struct vm_area_struct *vma)
return userfaultfd_wp_async_ctx(vma->vm_userfaultfd_ctx.ctx);
}
+bool userfaultfd_rwp_async(struct vm_area_struct *vma)
+{
+ return userfaultfd_rwp_async_ctx(vma->vm_userfaultfd_ctx.ctx);
+}
+
static inline unsigned int uffd_ctx_features(__u64 user_features)
{
/*
@@ -4302,20 +4641,14 @@ static int userfaultfd_api(struct userfaultfd_ctx *ctx,
if (features & UFFD_FEATURE_WP_ASYNC)
features |= UFFD_FEATURE_WP_UNPOPULATED;
- /* report all available features and ioctls to userland */
- uffdio_api.features = UFFD_API_FEATURES;
-#ifndef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR
- uffdio_api.features &=
- ~(UFFD_FEATURE_MINOR_HUGETLBFS | UFFD_FEATURE_MINOR_SHMEM);
-#endif
- if (!pgtable_supports_uffd_wp())
- uffdio_api.features &= ~UFFD_FEATURE_PAGEFAULT_FLAG_WP;
+ ret = -EINVAL;
+ /* RWP_ASYNC requires RWP */
+ if ((features & UFFD_FEATURE_RWP_ASYNC) &&
+ !(features & UFFD_FEATURE_RWP))
+ goto err_out;
- if (!uffd_supports_wp_marker()) {
- uffdio_api.features &= ~UFFD_FEATURE_WP_HUGETLBFS_SHMEM;
- uffdio_api.features &= ~UFFD_FEATURE_WP_UNPOPULATED;
- uffdio_api.features &= ~UFFD_FEATURE_WP_ASYNC;
- }
+ /* report all available features and ioctls to userland */
+ uffdio_api.features = uffd_api_available_features();
ret = -EINVAL;
if (features & ~uffdio_api.features)
@@ -4382,6 +4715,12 @@ static long userfaultfd_ioctl(struct file *file, unsigned cmd,
case UFFDIO_POISON:
ret = userfaultfd_poison(ctx, arg);
break;
+ case UFFDIO_RWPROTECT:
+ ret = userfaultfd_rwprotect(ctx, arg);
+ break;
+ case UFFDIO_SET_MODE:
+ ret = userfaultfd_set_mode(ctx, arg);
+ break;
}
return ret;
}
@@ -4409,7 +4748,7 @@ static void userfaultfd_show_fdinfo(struct seq_file *m, struct file *f)
* protocols: aa:... bb:...
*/
seq_printf(m, "pending:\t%lu\ntotal:\t%lu\nAPI:\t%Lx:%x:%Lx\n",
- pending, total, UFFD_API, ctx->features,
+ pending, total, UFFD_API, userfaultfd_features(ctx),
UFFD_API_IOCTLS|UFFD_API_RANGE_IOCTLS);
}
#endif
diff --git a/tools/include/uapi/linux/fs.h b/tools/include/uapi/linux/fs.h
index 24ddf7bc4f25f0..f0a26309b6d547 100644
--- a/tools/include/uapi/linux/fs.h
+++ b/tools/include/uapi/linux/fs.h
@@ -364,6 +364,7 @@ typedef int __bitwise __kernel_rwf_t;
#define PAGE_IS_HUGE (1 << 6)
#define PAGE_IS_SOFT_DIRTY (1 << 7)
#define PAGE_IS_GUARD (1 << 8)
+#define PAGE_IS_ACCESSED (1 << 9)
/*
* struct page_region - Page region with flags
diff --git a/tools/testing/selftests/mm/uffd-unit-tests.c b/tools/testing/selftests/mm/uffd-unit-tests.c
index a6c14109e81887..8c49191835ea1d 100644
--- a/tools/testing/selftests/mm/uffd-unit-tests.c
+++ b/tools/testing/selftests/mm/uffd-unit-tests.c
@@ -7,6 +7,8 @@
#include "uffd-common.h"
+#include <linux/fs.h>
+#include <sys/uio.h>
#include "../../../../mm/gup_test.h"
#ifdef __NR_userfaultfd
@@ -109,6 +111,10 @@ static void uffd_test_skip(const char *message)
static void test_uffd_api(bool use_dev)
{
+ const uint64_t expected_ioctls =
+ BIT_ULL(_UFFDIO_REGISTER) |
+ BIT_ULL(_UFFDIO_UNREGISTER) |
+ BIT_ULL(_UFFDIO_API);
struct uffdio_api uffdio_api;
int uffd;
@@ -148,6 +154,15 @@ static void test_uffd_api(bool use_dev)
goto out;
}
+ /* Verify returned fd-level ioctls bitmask */
+ if ((uffdio_api.ioctls & expected_ioctls) != expected_ioctls) {
+ uffd_test_fail("UFFDIO_API missing expected ioctls: "
+ "got=0x%"PRIx64", expected=0x%"PRIx64,
+ (uint64_t)uffdio_api.ioctls,
+ expected_ioctls);
+ goto out;
+ }
+
/* Test double requests of UFFDIO_API with a random feature set */
uffdio_api.features = BIT_ULL(0);
if (ioctl(uffd, UFFDIO_API, &uffdio_api) == 0) {
@@ -602,6 +617,687 @@ void uffd_minor_collapse_test(uffd_global_test_opts_t *gopts, uffd_test_args_t *
uffd_minor_test_common(gopts, true, false);
}
+static int uffd_register_rwp(int uffd, void *addr, uint64_t len)
+{
+ struct uffdio_register reg = {
+ .range = { .start = (unsigned long)addr, .len = len },
+ .mode = UFFDIO_REGISTER_MODE_RWP,
+ };
+
+ if (ioctl(uffd, UFFDIO_REGISTER, ®) == -1)
+ return -errno;
+ return 0;
+}
+
+static void rwprotect_range(int uffd, __u64 start, __u64 len, bool protect)
+{
+ struct uffdio_rwprotect rwp = {
+ .range = { .start = start, .len = len },
+ .mode = protect ? UFFDIO_RWPROTECT_MODE_RWP : 0,
+ };
+
+ if (ioctl(uffd, UFFDIO_RWPROTECT, &rwp))
+ err("UFFDIO_RWPROTECT failed");
+}
+
+static void set_async_mode(int uffd, bool enable)
+{
+ struct uffdio_set_mode mode = { };
+
+ if (enable)
+ mode.enable = UFFD_FEATURE_RWP_ASYNC;
+ else
+ mode.disable = UFFD_FEATURE_RWP_ASYNC;
+
+ if (ioctl(uffd, UFFDIO_SET_MODE, &mode))
+ err("UFFDIO_SET_MODE failed");
+}
+
+/*
+ * Test async RWP faults on anonymous memory.
+ * Populate pages, register MODE_RWP with RWP_ASYNC,
+ * RW-protect, re-access, verify content preserved and no faults delivered.
+ */
+static void uffd_rwp_async_test(uffd_global_test_opts_t *gopts,
+ uffd_test_args_t *args)
+{
+ unsigned long nr_pages = gopts->nr_pages;
+ unsigned long page_size = gopts->page_size;
+ unsigned long p;
+
+ /* Populate all pages with known content */
+ for (p = 0; p < nr_pages; p++)
+ memset(gopts->area_dst + p * page_size, p % 255 + 1, page_size);
+
+ /* Register MODE_RWP */
+ if (uffd_register_rwp(gopts->uffd, gopts->area_dst,
+ nr_pages * page_size))
+ err("register failure");
+
+ /* RW-protect all pages (sets protnone) */
+ rwprotect_range(gopts->uffd, (uint64_t)gopts->area_dst,
+ nr_pages * page_size, true);
+
+ /* Access all pages — should auto-resolve, no faults */
+ for (p = 0; p < nr_pages; p++) {
+ unsigned char *page = (unsigned char *)gopts->area_dst +
+ p * page_size;
+ unsigned char expected = p % 255 + 1;
+
+ if (page[0] != expected) {
+ uffd_test_fail("page %lu content mismatch: %u != %u",
+ p, page[0], expected);
+ return;
+ }
+ }
+
+ uffd_test_pass();
+}
+
+/*
+ * Fault handler for RWP — unprotect the page via UFFDIO_RWPROTECT.
+ */
+static void uffd_handle_rwp_fault(uffd_global_test_opts_t *gopts,
+ struct uffd_msg *msg,
+ struct uffd_args *uargs)
+{
+ if (!(msg->arg.pagefault.flags & UFFD_PAGEFAULT_FLAG_RWP))
+ err("expected RWP fault, got 0x%llx",
+ msg->arg.pagefault.flags);
+
+ rwprotect_range(gopts->uffd, msg->arg.pagefault.address,
+ gopts->page_size, false);
+ uargs->minor_faults++;
+}
+
+/*
+ * Test sync RWP faults on anonymous memory.
+ * Populate pages, register MODE_RWP (sync), RW-protect,
+ * access from worker thread, verify fault delivered, UFFDIO_RWPROTECT resolves.
+ */
+static void uffd_rwp_sync_test(uffd_global_test_opts_t *gopts,
+ uffd_test_args_t *args)
+{
+ unsigned long nr_pages = gopts->nr_pages;
+ unsigned long page_size = gopts->page_size;
+ pthread_t uffd_mon;
+ struct uffd_args uargs = { };
+ bool failed = false;
+ char c = '\0';
+ unsigned long p;
+
+ uargs.gopts = gopts;
+ uargs.handle_fault = uffd_handle_rwp_fault;
+
+ /* Populate all pages */
+ for (p = 0; p < nr_pages; p++)
+ memset(gopts->area_dst + p * page_size, p % 255 + 1, page_size);
+
+ /* Register MODE_RWP */
+ if (uffd_register_rwp(gopts->uffd, gopts->area_dst,
+ nr_pages * page_size))
+ err("register failure");
+
+ /* RW-protect all pages */
+ rwprotect_range(gopts->uffd, (uint64_t)gopts->area_dst,
+ nr_pages * page_size, true);
+
+ /* Start fault handler thread */
+ if (pthread_create(&uffd_mon, NULL, uffd_poll_thread, &uargs))
+ err("uffd_poll_thread create");
+
+ /* Access all pages — triggers sync RWP faults, handler unprotects */
+ for (p = 0; p < nr_pages; p++) {
+ unsigned char *page = (unsigned char *)gopts->area_dst +
+ p * page_size;
+
+ if (page[0] != (p % 255 + 1)) {
+ uffd_test_fail("page %lu content mismatch", p);
+ failed = true;
+ goto out;
+ }
+ }
+
+out:
+ /*
+ * Stop the handler before reading minor_faults: the last fault
+ * resolution rwprotect_range()s before incrementing the counter,
+ * so the main thread can race ahead of the increment.
+ */
+ if (write(gopts->pipefd[1], &c, sizeof(c)) != sizeof(c))
+ err("pipe write");
+ if (pthread_join(uffd_mon, NULL))
+ err("join() failed");
+
+ if (failed)
+ return;
+ if (uargs.minor_faults == 0)
+ uffd_test_fail("expected RWP faults, got 0");
+ else
+ uffd_test_pass();
+}
+
+/*
+ * Test PAGEMAP_SCAN working-set discovery via the "hot" (accessed) scan.
+ *
+ * The working-set primitive is to find pages that were accessed: scan for
+ * PAGE_IS_ACCESSED, which is set once an access clears the protnone+uffd
+ * marker. A VMM treats every access fault as "hot" (RWP here; MINOR/MISSING
+ * for non-resident pages) and reclaims the rest from the backing file.
+ *
+ * We deliberately do NOT use an inverted "cold" scan: that only sees
+ * VMA-resident ptes, so for a file mapping it misses cached-but-unmapped (and
+ * never-faulted, pre-populated) pages, which are pte_none and thus invisible.
+ * Hot tracking + file-level reclaim covers them; a cold pte scan cannot.
+ */
+static void uffd_rwp_pagemap_test(uffd_global_test_opts_t *gopts,
+ uffd_test_args_t *args)
+{
+ unsigned long nr_pages = gopts->nr_pages;
+ unsigned long page_size = gopts->page_size;
+ unsigned long p;
+ struct page_region regions[16];
+ struct pm_scan_arg pm_arg;
+ int pagemap_fd;
+ long ret;
+
+ /* Need at least 4 pages */
+ if (nr_pages < 4) {
+ uffd_test_skip("need at least 4 pages");
+ return;
+ }
+
+ /* Populate all pages */
+ for (p = 0; p < nr_pages; p++)
+ memset(gopts->area_dst + p * page_size, 0xab, page_size);
+
+ /* Register and RW-protect */
+ if (uffd_register_rwp(gopts->uffd, gopts->area_dst,
+ nr_pages * page_size))
+ err("register failure");
+
+ rwprotect_range(gopts->uffd, (uint64_t)gopts->area_dst,
+ nr_pages * page_size, true);
+
+ /* Touch first half of pages to re-activate them (async auto-resolve) */
+ for (p = 0; p < nr_pages / 2; p++) {
+ volatile char *page = gopts->area_dst + p * page_size;
+ (void)*page;
+ }
+
+ uint64_t start = (uint64_t)gopts->area_dst;
+ uint64_t boundary = start + (nr_pages / 2) * page_size;
+ uint64_t end = start + nr_pages * page_size;
+
+ pagemap_fd = open("/proc/self/pagemap", O_RDONLY);
+ if (pagemap_fd < 0)
+ err("open pagemap");
+
+ /*
+ * Hot scan: report the pages that were accessed. PAGE_IS_ACCESSED is
+ * set once the protnone+uffd marker is cleared (by the access, async
+ * auto-resolve here). The touched first half must come back as exactly
+ * one hot region [start, boundary); the untouched second half must not
+ * appear.
+ */
+ memset(&pm_arg, 0, sizeof(pm_arg));
+ pm_arg.size = sizeof(pm_arg);
+ pm_arg.start = start;
+ pm_arg.end = end;
+ pm_arg.vec = (uint64_t)regions;
+ pm_arg.vec_len = ARRAY_SIZE(regions);
+ pm_arg.category_mask = PAGE_IS_ACCESSED;
+ pm_arg.return_mask = PAGE_IS_ACCESSED;
+
+ ret = ioctl(pagemap_fd, PAGEMAP_SCAN, &pm_arg);
+ close(pagemap_fd);
+
+ if (ret < 0) {
+ uffd_test_fail("PAGEMAP_SCAN failed: %s", strerror(errno));
+ return;
+ }
+
+ if (ret != 1 || regions[0].start != start ||
+ regions[0].end != boundary) {
+ uffd_test_fail("hot set wrong: got %ld regions [0x%lx,0x%lx), expected 1 [0x%lx,0x%lx)",
+ ret, (unsigned long)regions[0].start,
+ (unsigned long)regions[0].end,
+ (unsigned long)start, (unsigned long)boundary);
+ return;
+ }
+
+ uffd_test_pass();
+}
+
+/*
+ * Test that RWP protection survives a mprotect(PROT_NONE) ->
+ * mprotect(PROT_READ|PROT_WRITE) round-trip. The uffd-wp bit on a
+ * VM_UFFD_RWP VMA must continue to carry PROT_NONE semantics after
+ * mprotect() changes the base protection; otherwise accesses would
+ * silently succeed and the pagemap bit would stick without a fault
+ * ever clearing it.
+ */
+static void uffd_rwp_mprotect_test(uffd_global_test_opts_t *gopts,
+ uffd_test_args_t *args)
+{
+ unsigned long nr_pages = gopts->nr_pages;
+ unsigned long page_size = gopts->page_size;
+ unsigned long p;
+ struct page_region regions[16];
+ struct pm_scan_arg pm_arg;
+ int pagemap_fd;
+ long ret;
+
+ /* Populate all pages */
+ for (p = 0; p < nr_pages; p++)
+ memset(gopts->area_dst + p * page_size, 0xab, page_size);
+
+ /* Register and RW-protect the whole range */
+ if (uffd_register_rwp(gopts->uffd, gopts->area_dst,
+ nr_pages * page_size))
+ err("register failure");
+ rwprotect_range(gopts->uffd, (uint64_t)gopts->area_dst,
+ nr_pages * page_size, true);
+
+ /* Round-trip mprotect(): PROT_NONE -> PROT_READ|PROT_WRITE */
+ if (mprotect(gopts->area_dst, nr_pages * page_size, PROT_NONE))
+ err("mprotect() PROT_NONE");
+ if (mprotect(gopts->area_dst, nr_pages * page_size,
+ PROT_READ | PROT_WRITE))
+ err("mprotect() PROT_READ|PROT_WRITE");
+
+ /* Touch every page. Async RWP must auto-resolve each fault. */
+ for (p = 0; p < nr_pages; p++) {
+ volatile char *page = gopts->area_dst + p * page_size;
+ (void)*page;
+ }
+
+ /*
+ * After touching, no page should remain RW-protected. A stuck
+ * uffd-wp bit would mean mprotect() silently dropped PROT_NONE and
+ * the access never faulted.
+ */
+ pagemap_fd = open("/proc/self/pagemap", O_RDONLY);
+ if (pagemap_fd < 0)
+ err("open pagemap");
+
+ memset(&pm_arg, 0, sizeof(pm_arg));
+ pm_arg.size = sizeof(pm_arg);
+ pm_arg.start = (uint64_t)gopts->area_dst;
+ pm_arg.end = (uint64_t)gopts->area_dst + nr_pages * page_size;
+ pm_arg.vec = (uint64_t)regions;
+ pm_arg.vec_len = ARRAY_SIZE(regions);
+ pm_arg.category_mask = PAGE_IS_ACCESSED;
+ pm_arg.category_inverted = PAGE_IS_ACCESSED;
+ pm_arg.return_mask = PAGE_IS_ACCESSED;
+
+ ret = ioctl(pagemap_fd, PAGEMAP_SCAN, &pm_arg);
+ close(pagemap_fd);
+
+ if (ret < 0) {
+ uffd_test_fail("PAGEMAP_SCAN failed: %s", strerror(errno));
+ return;
+ }
+ if (ret != 0) {
+ uffd_test_fail("expected no cold pages after mprotect()+touch, got %ld regions",
+ ret);
+ return;
+ }
+
+ uffd_test_pass();
+}
+
+/*
+ * Test that GUP resolves through protnone PTEs (async mode).
+ * vmsplice() into a pipe pins user pages via get_user_pages_fast() --
+ * unlike write(), which goes through copy_from_user() and ordinary
+ * hardware page faults -- so it exercises gup_can_follow_protnone() on
+ * the RW-protected PTE. In async mode the kernel auto-restores
+ * permissions and GUP returns the page.
+ */
+static void uffd_rwp_gup_test(uffd_global_test_opts_t *gopts,
+ uffd_test_args_t *args)
+{
+ struct iovec iov;
+ char buf;
+ int pipefd[2];
+
+ /* Populate first page with known content */
+ memset(gopts->area_dst, 0xCD, gopts->page_size);
+
+ if (uffd_register_rwp(gopts->uffd, gopts->area_dst, gopts->page_size))
+ err("register failure");
+
+ rwprotect_range(gopts->uffd, (uint64_t)gopts->area_dst,
+ gopts->page_size, true);
+
+ if (pipe(pipefd))
+ err("pipe");
+
+ /*
+ * One byte's worth of iov is enough to GUP the containing page and
+ * keeps the pipe transfer well under any pipe-capacity limit even on
+ * hugetlb-backed runs.
+ */
+ iov.iov_base = gopts->area_dst;
+ iov.iov_len = 1;
+ if (vmsplice(pipefd[1], &iov, 1, 0) != 1) {
+ uffd_test_fail("vmsplice from RW-protected page failed: %s",
+ strerror(errno));
+ goto out;
+ }
+
+ if (read(pipefd[0], &buf, 1) != 1) {
+ uffd_test_fail("read from pipe failed");
+ goto out;
+ }
+
+ if (buf != (char)0xCD) {
+ uffd_test_fail("content mismatch: got 0x%02x, expected 0xCD",
+ (unsigned char)buf);
+ goto out;
+ }
+
+ uffd_test_pass();
+out:
+ close(pipefd[0]);
+ close(pipefd[1]);
+}
+
+/*
+ * Test runtime toggle between async and sync modes.
+ * Start in async mode (detection), flip to sync (eviction), verify faults
+ * block, resolve them, flip back to async.
+ */
+static void uffd_rwp_async_toggle_test(uffd_global_test_opts_t *gopts,
+ uffd_test_args_t *args)
+{
+ unsigned long nr_pages = gopts->nr_pages;
+ unsigned long page_size = gopts->page_size;
+ struct uffd_args uargs = { };
+ pthread_t uffd_mon;
+ char c = '\0';
+ unsigned long p;
+
+ uargs.gopts = gopts;
+ uargs.handle_fault = uffd_handle_rwp_fault;
+
+ /* Populate */
+ for (p = 0; p < nr_pages; p++)
+ memset(gopts->area_dst + p * page_size, p % 255 + 1, page_size);
+
+ if (uffd_register_rwp(gopts->uffd, gopts->area_dst,
+ nr_pages * page_size))
+ err("register failure");
+
+ /* Phase 1: async detection — RW-protect, access first half */
+ rwprotect_range(gopts->uffd, (uint64_t)gopts->area_dst,
+ nr_pages * page_size, true);
+
+ for (p = 0; p < nr_pages / 2; p++) {
+ volatile char *page = gopts->area_dst + p * page_size;
+ (void)*page; /* auto-resolves in async mode */
+ }
+
+ /* Phase 2: flip to sync for eviction */
+ set_async_mode(gopts->uffd, false);
+
+ /* Start handler — will receive faults for cold pages */
+ if (pthread_create(&uffd_mon, NULL, uffd_poll_thread, &uargs))
+ err("uffd_poll_thread create");
+
+ /* Access second half (cold pages) — should trigger sync faults */
+ for (p = nr_pages / 2; p < nr_pages; p++) {
+ unsigned char *page = (unsigned char *)gopts->area_dst +
+ p * page_size;
+ if (page[0] != (p % 255 + 1)) {
+ uffd_test_fail("page %lu content mismatch", p);
+ goto out;
+ }
+ }
+
+ /*
+ * Stop the handler before reading minor_faults: the last fault
+ * resolution rwprotect_range()s before incrementing the counter,
+ * so the main thread can race ahead of the increment. Stopping
+ * here also makes Phase 3 a clean async-only test -- with the
+ * handler still running it would silently resolve any sync fault
+ * the kernel erroneously delivers, masking a regression.
+ */
+ if (write(gopts->pipefd[1], &c, sizeof(c)) != sizeof(c))
+ err("pipe write");
+ if (pthread_join(uffd_mon, NULL))
+ err("join() failed");
+
+ if (uargs.minor_faults == 0) {
+ uffd_test_fail("expected sync faults, got 0");
+ return;
+ }
+
+ /* Phase 3: flip back to async */
+ set_async_mode(gopts->uffd, true);
+
+ /* RW-protect and access again — should auto-resolve */
+ rwprotect_range(gopts->uffd, (uint64_t)gopts->area_dst,
+ nr_pages * page_size, true);
+
+ for (p = 0; p < nr_pages; p++) {
+ volatile char *page = gopts->area_dst + p * page_size;
+ (void)*page;
+ }
+
+ uffd_test_pass();
+ return;
+out:
+ if (write(gopts->pipefd[1], &c, sizeof(c)) != sizeof(c))
+ err("pipe write");
+ if (pthread_join(uffd_mon, NULL))
+ err("join() failed");
+}
+
+/*
+ * Test that RW-protected pages become accessible after closing uffd.
+ */
+static void uffd_rwp_close_test(uffd_global_test_opts_t *gopts,
+ uffd_test_args_t *args)
+{
+ unsigned long nr_pages = gopts->nr_pages;
+ unsigned long page_size = gopts->page_size;
+ unsigned long p;
+
+ /* Populate */
+ for (p = 0; p < nr_pages; p++)
+ memset(gopts->area_dst + p * page_size, p % 255 + 1, page_size);
+
+ if (uffd_register_rwp(gopts->uffd, gopts->area_dst,
+ nr_pages * page_size))
+ err("register failure");
+
+ rwprotect_range(gopts->uffd, (uint64_t)gopts->area_dst,
+ nr_pages * page_size, true);
+
+ /* Close uffd — should restore protnone PTEs */
+ close(gopts->uffd);
+ gopts->uffd = -1;
+
+ /* All pages should be accessible with original content */
+ for (p = 0; p < nr_pages; p++) {
+ unsigned char *page = (unsigned char *)gopts->area_dst +
+ p * page_size;
+ unsigned char expected = p % 255 + 1;
+
+ if (page[0] != expected) {
+ uffd_test_fail("page %lu not accessible after close", p);
+ return;
+ }
+ }
+
+ uffd_test_pass();
+}
+
+/*
+ * Test that RWP protection is preserved across fork() when
+ * UFFD_FEATURE_EVENT_FORK is enabled. Without preservation, the child's
+ * PTEs would lose the uffd-wp marker and RWP-protected accesses would
+ * silently fall through to do_numa_page().
+ */
+static void uffd_rwp_fork_test(uffd_global_test_opts_t *gopts,
+ uffd_test_args_t *args)
+{
+ unsigned long nr_pages = gopts->nr_pages;
+ unsigned long page_size = gopts->page_size;
+ int pagemap_fd;
+ uint64_t value;
+
+ if (uffd_register_rwp(gopts->uffd, gopts->area_dst,
+ nr_pages * page_size))
+ err("register failed");
+
+ /* Populate + RWP-protect */
+ *gopts->area_dst = 1;
+ rwprotect_range(gopts->uffd, (uint64_t)gopts->area_dst,
+ page_size, true);
+
+ /* Parent: verify uffd-wp bit is set before fork */
+ pagemap_fd = pagemap_open();
+ value = pagemap_get_entry(pagemap_fd, gopts->area_dst);
+ pagemap_check_wp(value, true);
+
+ /*
+ * Fork with EVENT_FORK: child inherits VM_UFFD_RWP. Child reads
+ * its own pagemap and must still see the uffd-wp bit set.
+ */
+ if (pagemap_test_fork(gopts, true, false)) {
+ uffd_test_fail("RWP marker lost in child after fork");
+ goto out;
+ }
+
+ uffd_test_pass();
+out:
+ close(pagemap_fd);
+}
+
+/*
+ * Test that RWP protection on a pinned anon page is preserved across fork().
+ * Pinning forces copy_present_page() in the child path, which must restore
+ * PAGE_NONE on top of the uffd bit. Using async mode, a read in the child
+ * auto-resolves if — and only if — the PTE was actually protnone+uffd; the
+ * cleared uffd bit afterward proves the fault path ran.
+ */
+static void uffd_rwp_fork_pin_test(uffd_global_test_opts_t *gopts,
+ uffd_test_args_t *args)
+{
+ unsigned long page_size = gopts->page_size;
+ fork_event_args fevent_args = { .gopts = gopts, .child_uffd = -1 };
+ pin_args pin_args = {};
+ int pagemap_fd, status;
+ pthread_t fevent_thread;
+ uint64_t value;
+ pid_t child;
+
+ if (uffd_register_rwp(gopts->uffd, gopts->area_dst, page_size))
+ err("register failed");
+
+ /* Populate. */
+ *gopts->area_dst = 1;
+
+ /* RO-longterm pin so fork() takes copy_present_page() for this PTE. */
+ if (pin_pages(&pin_args, gopts->area_dst, page_size)) {
+ uffd_test_skip("Possibly CONFIG_GUP_TEST missing or unprivileged");
+ uffd_unregister(gopts->uffd, gopts->area_dst, page_size);
+ return;
+ }
+
+ /* RWP-protect: PTE is now PAGE_NONE + uffd bit. */
+ rwprotect_range(gopts->uffd, (uint64_t)gopts->area_dst, page_size, true);
+
+ pagemap_fd = pagemap_open();
+ value = pagemap_get_entry(pagemap_fd, gopts->area_dst);
+ pagemap_check_wp(value, true);
+
+ /*
+ * UFFD_FEATURE_EVENT_FORK is required so the child inherits
+ * VM_UFFD_RWP and the marker; without it dup_userfaultfd() resets
+ * the child VMA and the test would pass for the wrong reason.
+ * dup_userfaultfd() blocks until the EVENT_FORK message is consumed,
+ * so spawn a reader before the fork().
+ */
+ gopts->ready_for_fork = false;
+ if (pthread_create(&fevent_thread, NULL, fork_event_consumer,
+ &fevent_args))
+ err("pthread_create() for fork event consumer");
+ while (!gopts->ready_for_fork)
+ ; /* Wait for consumer to start polling. */
+
+ child = fork();
+ if (child < 0)
+ err("fork");
+ if (child == 0) {
+ volatile char c;
+ int cfd;
+
+ /*
+ * Read the pinned page. Only reaches the fault path if the
+ * child PTE is protnone + uffd; async mode auto-resolves and
+ * clears the uffd bit. If copy_present_page() dropped
+ * PAGE_NONE, the read would silently succeed and the bit
+ * would still be set.
+ */
+ c = *(volatile char *)gopts->area_dst;
+ (void)c;
+
+ cfd = pagemap_open();
+ value = pagemap_get_entry(cfd, gopts->area_dst);
+ close(cfd);
+ _exit((value & PM_UFFD_WP) ? 1 : 0);
+ }
+ if (waitpid(child, &status, 0) < 0)
+ err("waitpid");
+ if (pthread_join(fevent_thread, NULL))
+ err("pthread_join() for fork event consumer");
+ if (fevent_args.child_uffd >= 0)
+ close(fevent_args.child_uffd);
+
+ unpin_pages(&pin_args);
+ close(pagemap_fd);
+ if (uffd_unregister(gopts->uffd, gopts->area_dst, page_size))
+ err("unregister failed");
+
+ if (!WIFEXITED(status) || WEXITSTATUS(status) != 0) {
+ uffd_test_fail("RWP not enforced in child after pinned fork");
+ return;
+ }
+
+ uffd_test_pass();
+}
+
+/*
+ * WP and RWP share the uffd-wp PTE bit and cannot coexist in the same VMA.
+ * Registration requesting both modes must be rejected.
+ */
+static void uffd_rwp_wp_exclusive_test(uffd_global_test_opts_t *gopts,
+ uffd_test_args_t *args)
+{
+ unsigned long nr_pages = gopts->nr_pages;
+ unsigned long page_size = gopts->page_size;
+ struct uffdio_register reg = { };
+
+ reg.range.start = (unsigned long)gopts->area_dst;
+ reg.range.len = nr_pages * page_size;
+ reg.mode = UFFDIO_REGISTER_MODE_WP | UFFDIO_REGISTER_MODE_RWP;
+
+ if (ioctl(gopts->uffd, UFFDIO_REGISTER, ®) == 0) {
+ uffd_test_fail("register with WP|RWP unexpectedly succeeded");
+ return;
+ }
+ if (errno != EINVAL) {
+ uffd_test_fail("register with WP|RWP: expected EINVAL, got %d",
+ errno);
+ return;
+ }
+ uffd_test_pass();
+}
+
static sigjmp_buf jbuf, *sigbuf;
static void sighndl(int sig, siginfo_t *siginfo, void *ptr)
@@ -1604,6 +2300,77 @@ uffd_test_case_t uffd_tests[] = {
/* We can't test MADV_COLLAPSE, so try our luck */
.uffd_feature_required = UFFD_FEATURE_MINOR_SHMEM,
},
+ {
+ .name = "rwp-async",
+ .uffd_fn = uffd_rwp_async_test,
+ .mem_targets = MEM_ALL,
+ .uffd_feature_required =
+ UFFD_FEATURE_RWP | UFFD_FEATURE_RWP_ASYNC,
+ },
+ {
+ .name = "rwp-sync",
+ .uffd_fn = uffd_rwp_sync_test,
+ .mem_targets = MEM_ALL,
+ .uffd_feature_required = UFFD_FEATURE_RWP,
+ },
+ {
+ .name = "rwp-pagemap",
+ .uffd_fn = uffd_rwp_pagemap_test,
+ .mem_targets = MEM_ALL,
+ .uffd_feature_required =
+ UFFD_FEATURE_RWP | UFFD_FEATURE_RWP_ASYNC,
+ },
+ {
+ .name = "rwp-mprotect",
+ .uffd_fn = uffd_rwp_mprotect_test,
+ .mem_targets = MEM_ALL,
+ .uffd_feature_required =
+ UFFD_FEATURE_RWP | UFFD_FEATURE_RWP_ASYNC,
+ },
+ {
+ .name = "rwp-gup",
+ .uffd_fn = uffd_rwp_gup_test,
+ .mem_targets = MEM_ALL,
+ .uffd_feature_required =
+ UFFD_FEATURE_RWP | UFFD_FEATURE_RWP_ASYNC,
+ },
+ {
+ .name = "rwp-async-toggle",
+ .uffd_fn = uffd_rwp_async_toggle_test,
+ .mem_targets = MEM_ALL,
+ .uffd_feature_required =
+ UFFD_FEATURE_RWP | UFFD_FEATURE_RWP_ASYNC,
+ },
+ {
+ .name = "rwp-close",
+ .uffd_fn = uffd_rwp_close_test,
+ .mem_targets = MEM_ALL,
+ .uffd_feature_required = UFFD_FEATURE_RWP,
+ },
+ {
+ .name = "rwp-fork",
+ .uffd_fn = uffd_rwp_fork_test,
+ .mem_targets = MEM_ALL,
+ .uffd_feature_required =
+ UFFD_FEATURE_RWP | UFFD_FEATURE_EVENT_FORK,
+ },
+ {
+ .name = "rwp-fork-pin",
+ .uffd_fn = uffd_rwp_fork_pin_test,
+ .mem_targets = MEM_ANON,
+ .uffd_feature_required =
+ UFFD_FEATURE_RWP | UFFD_FEATURE_RWP_ASYNC |
+ UFFD_FEATURE_EVENT_FORK,
+ },
+ {
+ .name = "rwp-wp-exclusive",
+ .uffd_fn = uffd_rwp_wp_exclusive_test,
+ .mem_targets = MEM_ALL,
+ .uffd_feature_required =
+ UFFD_FEATURE_RWP |
+ UFFD_FEATURE_PAGEFAULT_FLAG_WP |
+ UFFD_FEATURE_WP_HUGETLBFS_SHMEM,
+ },
{
.name = "sigbus",
.uffd_fn = uffd_sigbus_test,
@@ -1814,6 +2581,20 @@ int main(int argc, char *argv[])
uffd_test_skip(errmsg);
continue;
}
+ /*
+ * RWP tracks protection on ptes; a THP-backed shmem/anon
+ * range (e.g. shmem_enabled=always) would split on
+ * rwprotect and change behaviour under the test. Keep
+ * such ranges off THP. hugetlb is huge by definition and
+ * rejects MADV_NOHUGEPAGE, so skip it.
+ */
+ if ((test->uffd_feature_required & UFFD_FEATURE_RWP) &&
+ !(mem_type->mem_flag & (MEM_HUGETLB | MEM_HUGETLB_PRIVATE))) {
+ unsigned long len = gopts.nr_pages * gopts.page_size;
+
+ if (madvise(gopts.area_dst, len, MADV_NOHUGEPAGE))
+ err("madvise(MADV_NOHUGEPAGE)");
+ }
test->uffd_fn(&gopts, &args);
uffd_test_ctx_clear(&gopts);
}
|
| 3/2 | 2026/06/29 12:30 | llm | patch-evaluator |
0mModel:gemini-3.1-pro-preview Args: null Results: null Tokens: input: 75053 output: 161 thoughts: 1195 Thoughts: **RWP Userfaultfd Patch Analysis: Fuzzing Strategy** Okay, so we have this patch introducing Read-Write Protection (RWP) for userfaultfd. My initial assessment is that it's adding a whole new mode of operation, which, inherently, screams "fuzz me!" We're looking at a completely new code path, triggered by userfaultfd syscalls and subsequent memory accesses. Definitely reachable, and therefore, definitely worth fuzzing. Now, let's get down to the brass tacks. First, the configuration. It looks like `CONFIG_USERFAULTFD_RWP` is the key. It depends on `64BIT`, `ARCH_HAS_PTE_PROTNONE`, and `HAVE_ARCH_USERFAULTFD_WP`. That dependency is important to note and we’ll have to make sure our fuzzing setup satisfies it. It also uses `CONFIG_USERFAULTFD`, which is usually on by default, but it's good to call it out for the record. I'm going to explicitly include `USERFAULTFD_RWP` in `EnableConfigs`. Checked `mm/Kconfig` to confirm. Next, the functions. We need to pinpoint the critical parts. The patch modifies core memory management, which means we have to check both `mm` and `fs` related code. The obvious candidates are the added and modified functions: `userfaultfd_rwprotect`, `mrwprotect_range`, `userfaultfd_set_mode`, `do_uffd_rwp`, and `do_huge_pmd_uffd_rwp`. I took a look through the files: `mm/userfaultfd.c`, `mm/memory.c`, `mm/huge_memory.c`, `fs/proc/task_mmu.c`, `mm/mprotect.c` and `mm/hugetlb.c`. Besides the ones mentioned above, `userfaultfd_register` is modified, which is a key part of the hot path, and we'll need to pay attention to any fuzzing results there. `userfaultfd_ioctl` is called by `userfaultfd_rwprotect` and `userfaultfd_set_mode` which makes it interesting as well. So, to summarize, the key focus points for fuzzing are: `userfaultfd_rwprotect`, `mrwprotect_range`, `userfaultfd_set_mode`, `do_uffd_rwp`, and `do_huge_pmd_uffd_rwp`. And, of course, we'll want to enable `USERFAULTFD_RWP` for our fuzzing runs. I believe the patch introduces a functional change to core memory management and userfaultfd that definitely needs fuzzing. |
| 4/2 | 2026/06/29 12:31 | tool | set-results |
0mArgs:{
"EnableConfigs": [
"USERFAULTFD_RWP"
],
"FocusSymbols": [
"userfaultfd_rwprotect",
"mrwprotect_range",
"userfaultfd_set_mode",
"do_uffd_rwp",
"do_huge_pmd_uffd_rwp"
],
"Reasoning": "The patch introduces a new userfaultfd feature, Read-Write Protection (RWP), which adds new ioctls (UFFDIO_RWPROTECT, UFFDIO_SET_MODE) and modifies page fault handling to support RWP. This is a significant functional change to core memory management and userfaultfd that should be fuzzed.",
"WorthFuzzing": true
}Results: {
"EnableConfigs": [
"USERFAULTFD_RWP"
],
"FocusSymbols": [
"userfaultfd_rwprotect",
"mrwprotect_range",
"userfaultfd_set_mode",
"do_uffd_rwp",
"do_huge_pmd_uffd_rwp"
],
"Reasoning": "The patch introduces a new userfaultfd feature, Read-Write Protection (RWP), which adds new ioctls (UFFDIO_RWPROTECT, UFFDIO_SET_MODE) and modifies page fault handling to support RWP. This is a significant functional change to core memory management and userfaultfd that should be fuzzed.",
"WorthFuzzing": true
} |
| Total Calls | Total Tokens | Avg Tokens | Total Duration (Seconds) | Avg Duration (Seconds) |
|---|
| Total Calls | Total Duration (Seconds) | Avg Duration (Seconds) |
|---|