From: Hongfu Li If hugetlb_add_to_page_cache() in memfd_alloc_folio() fails with -EEXIST, a concurrent fault has already instantiated the folio in the page cache, and the reservation now belongs to that folio. Calling hugetlb_unreserve_pages() in that case incorrectly removes the region backing the cached folio. A later truncate or inode eviction then passes a negative (chg - freed) into hugepage_subpool_put_pages(), corrupting subpool and resv_huge_pages accounting. Hold the hugetlb fault mutex from hugetlb_reserve_pages() until the error-path unreserve completes to make the reserve, allocate and instantiate steps atomic against concurrent faults. With the mutex held from the start, a concurrent fault can no longer consume the reservation between reserve and allocate/instantiate. If a fault completed before the mutex was taken, it has already added the region for that index, so hugetlb_reserve_pages() returns 0 and the error path leaves the region in place. Fixes: 717cf9357325 ("mm/memfd: reserve hugetlb folios before allocation") Signed-off-by: Hongfu Li --- v2: - Take the hugetlb fault mutex before hugetlb_reserve_pages() and hold it until the error-path unreserve completes. - Update commit message - Link to v1: https://lore.kernel.org/all/20260831090631.29227-1-hongfu.li@linux.dev/ --- mm/memfd.c | 31 ++++++++++++++++--------------- 1 file changed, 16 insertions(+), 15 deletions(-) diff --git a/mm/memfd.c b/mm/memfd.c index c708d92533f4..0f6fff004f5e 100644 --- a/mm/memfd.c +++ b/mm/memfd.c @@ -82,22 +82,31 @@ struct folio *memfd_alloc_folio(struct file *memfd, pgoff_t idx) struct hstate *h = hstate_file(memfd); int err = -ENOMEM; long nr_resv; + u32 hash; gfp_mask = htlb_alloc_mask(h); gfp_mask &= ~(__GFP_HIGHMEM | __GFP_MOVABLE); idx >>= huge_page_order(h); + /* + * Serialize hugepage allocation and instantiation to prevent + * races with concurrent allocations, as required by all other + * callers of hugetlb_add_to_page_cache(). + */ + hash = hugetlb_fault_mutex_hash(memfd->f_mapping, idx); + mutex_lock(&hugetlb_fault_mutex_table[hash]); + nr_resv = hugetlb_reserve_pages(inode, idx, idx + 1, NULL, EMPTY_VMA_FLAGS); - if (nr_resv < 0) - return ERR_PTR(nr_resv); + if (nr_resv < 0) { + err = nr_resv; + goto out_unlock; + } folio = alloc_hugetlb_folio_reserve(h, numa_node_id(), NULL, gfp_mask); if (folio) { - u32 hash; - /* * Zero the folio to prevent information leaks to userspace. * Use folio_zero_user() which is optimized for huge/gigantic @@ -112,20 +121,9 @@ struct folio *memfd_alloc_folio(struct file *memfd, pgoff_t idx) */ __folio_mark_uptodate(folio); - /* - * Serialize hugepage allocation and instantiation to prevent - * races with concurrent allocations, as required by all other - * callers of hugetlb_add_to_page_cache(). - */ - hash = hugetlb_fault_mutex_hash(memfd->f_mapping, idx); - mutex_lock(&hugetlb_fault_mutex_table[hash]); - err = hugetlb_add_to_page_cache(folio, memfd->f_mapping, idx); - - mutex_unlock(&hugetlb_fault_mutex_table[hash]); - if (err) { folio_put(folio); goto err_unresv; @@ -133,11 +131,14 @@ struct folio *memfd_alloc_folio(struct file *memfd, pgoff_t idx) hugetlb_set_folio_subpool(folio, subpool_inode(inode)); folio_unlock(folio); + mutex_unlock(&hugetlb_fault_mutex_table[hash]); return folio; } err_unresv: if (nr_resv > 0) hugetlb_unreserve_pages(inode, idx, idx + 1, 0); +out_unlock: + mutex_unlock(&hugetlb_fault_mutex_table[hash]); return ERR_PTR(err); } #endif -- 2.54.0