UFFDIO_COPY fills MAP_PRIVATE file-backed VMAs with anonymous memory. mfill_atomic_pte_copy() implements that by overriding the VMA's uffd ops with anon_uffd_ops when VM_SHARED is not set. mfill_copy_folio_retry() can drop the destination VMA lock after an initial copy_from_user() failure and reacquire the destination VMA. It currently checks whether vma_uffd_ops() changed while the lock was dropped, but that is not the same as checking whether the effective UFFDIO_COPY ops changed. Private and shared shmem VMAs both expose shmem_uffd_ops through vm_ops. If a private shmem destination is replaced with a shared shmem destination while the retry has dropped the lock, vma_uffd_ops() still compares equal even though the effective copy ops changed from anon_uffd_ops to shmem_uffd_ops. The stale anon folio can then be installed into the new shared shmem VMA. mfill_atomic_install_pte() sees a folio without page-cache mapping and calls folio_add_new_anon_rmap(), which reaches BUG_ON(!anon_vma) because the new shared shmem VMA has no anon_vma. Compare both the raw VMA uffd ops and the effective UFFDIO_COPY ops across the retry. The raw comparison preserves the existing VMA-type replacement guard, while the effective comparison also catches replacements where the raw ops stay equal but the MAP_PRIVATE override result changes. If either comparison changes, return -EAGAIN and let the ioctl retry instead of installing the stale folio through the wrong path. Fixes: 292411fda25b ("mm/userfaultfd: detect VMA type change after copy retry in mfill_copy_folio_retry()") Assisted-by: Codex:gpt-5-5-xhigh Assisted-by: Claude:opus-4-7 Signed-off-by: Michael Bommarito --- mm/userfaultfd.c | 40 ++++++++++++++++++++++++---------------- 1 file changed, 24 insertions(+), 16 deletions(-) diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 180bad42fc79..5af13953c29a 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -69,6 +69,24 @@ static const struct vm_uffd_ops *vma_uffd_ops(struct vm_area_struct *vma) return vma->vm_ops ? vma->vm_ops->uffd_ops : NULL; } +static const struct vm_uffd_ops *vma_uffd_copy_ops(struct vm_area_struct *vma) +{ + const struct vm_uffd_ops *ops = vma_uffd_ops(vma); + + if (!ops) + return NULL; + + /* + * UFFDIO_COPY fills MAP_PRIVATE file-backed mappings as anonymous + * memory. This is an effective ops override, so retry validation must + * compare the override result, not just vma->vm_ops->uffd_ops. + */ + if (!(vma->vm_flags & VM_SHARED)) + return &anon_uffd_ops; + + return ops; +} + static __always_inline bool validate_dst_vma(struct vm_area_struct *dst_vma, unsigned long dst_end) { @@ -447,6 +465,7 @@ static int mfill_copy_folio_retry(struct mfill_state *state, struct folio *folio) { const struct vm_uffd_ops *orig_ops = vma_uffd_ops(state->vma); + const struct vm_uffd_ops *orig_copy_ops = vma_uffd_copy_ops(state->vma); unsigned long src_addr = state->src_addr; void *kaddr; int err; @@ -469,10 +488,11 @@ static int mfill_copy_folio_retry(struct mfill_state *state, /* * The VMA type may have changed while the lock was dropped - * (e.g. replaced with a hugetlb mapping), making the caller's - * ops pointer stale. + * (e.g. replaced with a hugetlb mapping). Also catch changes where + * the raw ops stay equal but the effective UFFDIO_COPY ops differ. */ - if (vma_uffd_ops(state->vma) != orig_ops) + if (vma_uffd_ops(state->vma) != orig_ops || + vma_uffd_copy_ops(state->vma) != orig_copy_ops) return -EAGAIN; err = mfill_establish_pmd(state); @@ -545,19 +565,7 @@ static int __mfill_atomic_pte(struct mfill_state *state, static int mfill_atomic_pte_copy(struct mfill_state *state) { - const struct vm_uffd_ops *ops = vma_uffd_ops(state->vma); - - /* - * The normal page fault path for a MAP_PRIVATE mapping in a - * file-backed VMA will invoke the fault, fill the hole in the file and - * COW it right away. The result generates plain anonymous memory. - * So when we are asked to fill a hole in a MAP_PRIVATE mapping, we'll - * generate anonymous memory directly without actually filling the - * hole. For the MAP_PRIVATE case the robustness check only happens in - * the pagetable (to verify it's still none) and not in the page cache. - */ - if (!(state->vma->vm_flags & VM_SHARED)) - ops = &anon_uffd_ops; + const struct vm_uffd_ops *ops = vma_uffd_copy_ops(state->vma); return __mfill_atomic_pte(state, ops); } -- 2.46.0