From: Hongfu Li alloc_hugetlb_folio_reserve() hands out a folio from the hugetlb pool and consumes a global reservation, but unlike the fault path it never charged the folio to the hugetlb cgroup or to the memcg. Folios allocated for memfd_pin_folios() through memfd_alloc_folio() were therefore silently bypassing the accounting: they are never charged to the hugetlb cgroup usage counter (hugetlb..current or hugetlb..usage_in_bytes), so the limits (hugetlb..max or hugetlb..limit_in_bytes) are bypassed. In cgroup v2, with memory_hugetlb_accounting enabled, these folios are also not accounted in memory.current and are not bounded by memory.max. Charge the folios to both the hugetlb cgroup and the memcg, just like hugetlb_alloc_folio() does. Test with a process that creates a hugetlb memfd and pins pages through UDMABUF_CREATE. The counters were read while the folio was still pinned: Before this patch: $ echo $$ > /sys/fs/cgroup/test/cgroup.procs $ ./memfd-alloc-folio 1 & $ cat /sys/fs/cgroup/test/hugetlb.2MB.current 0 $ cat /sys/fs/cgroup/test/memory.current 1351680 $ After this patch: $ echo $$ > /sys/fs/cgroup/test/cgroup.procs $ ./memfd-alloc-folio 1 & $ cat /sys/fs/cgroup/test/hugetlb.2MB.current 2097152 $ cat /sys/fs/cgroup/test/memory.current 3313664 $ memfd_pin_folios() can now fail because the memcg charge fails. Two tasks allocating concurrently against the same memory.max may see a spurious failure. That failure frees the folio, returns the global reservation, and undoes the hugetlb cgroup charge, so no accounting is left behind. Fixes: 89c1905d9c14 ("mm/gup: introduce memfd_pin_folios() for pinning memfd folios") Cc: stable@vger.kernel.org Signed-off-by: Hongfu Li Reviewed-by: Gregory Price (Meta) Tested-by: Gregory Price (Meta) --- Changes in v3: - Updated the commit message to describe the new -ENOMEM condition on memfd_pin_folios() when the memcg charge fails, as suggested by Gregory Price. - Added Gregory Price's Reviewed-by and Tested-by tags. Changes in v2: - Rebased on a tree that includes Daehyeon's "mm/hugetlb_cgroup: prevent use-after-free during cgroup offlining", as suggested by Andrew Morton. No functional change. - Since 268d64fabbf1 ("mm/hugetlb: fix NR_HUGETLB accounting for surplus folios") reworked the NR_HUGETLB accounting, use mod_hugetlb_memcg_state() instead of lruvec_stat_mod_folio(), called after a successful memcg charge. - Return early once a folio has been dequeued, instead of falling through to a shared return. - Corrected the Fixes tag to 89c1905d9c14. - Updated the commit message to qualify the memory.max claim: only in cgroup v2 with memory_hugetlb_accounting enabled. --- mm/hugetlb.c | 33 ++++++++++++++++++++++++++++++--- 1 file changed, 30 insertions(+), 3 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index da1c17f14a62..256bb2095b59 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -2223,12 +2223,19 @@ struct folio *alloc_buddy_hugetlb_folio(struct hstate *h, struct folio *alloc_hugetlb_folio_reserve(struct hstate *h, int preferred_nid, nodemask_t *nmask, gfp_t gfp_mask) { + struct hugetlb_cgroup *h_cg = NULL; + size_t nr_pages = pages_per_huge_page(h); struct folio *folio; + int idx = hstate_index(h); + int ret; + + if (hugetlb_cgroup_charge_cgroup(idx, nr_pages, &h_cg)) + return NULL; spin_lock_irq(&hugetlb_lock); if (!h->resv_huge_pages) { spin_unlock_irq(&hugetlb_lock); - return NULL; + goto err_uncharge_hugetlb_cgroup; } folio = dequeue_hugetlb_folio_nodemask(h, gfp_mask, preferred_nid, @@ -2236,11 +2243,31 @@ struct folio *alloc_hugetlb_folio_reserve(struct hstate *h, int preferred_nid, if (folio) { folio_set_hugetlb_restore_reserve(folio); h->resv_huge_pages--; - } + hugetlb_cgroup_commit_charge(idx, nr_pages, h_cg, folio); + spin_unlock_irq(&hugetlb_lock); + + ret = mem_cgroup_charge_hugetlb(folio, current->mm, + gfp_mask | __GFP_RETRY_MAYFAIL); + if (ret == -ENOMEM) { + folio_put(folio); + /* + * Skip uncharging hugetlb_cgroup since the charges + * were committed to the folio and freeing the folio + * would have cleared those up. + */ + return NULL; + } + + mod_hugetlb_memcg_state(folio, nr_pages); + + return folio; + } spin_unlock_irq(&hugetlb_lock); - return folio; +err_uncharge_hugetlb_cgroup: + hugetlb_cgroup_uncharge_cgroup(idx, nr_pages, h_cg); + return NULL; } /* folio migration callback function */ --- base-commit: 2c5dafa7f289d760f4492260b0988029aab478b0 change-id: 20261109-for-alloc_hugetlb-1109-0ccac58e69d1 Best regards, -- Hongfu Li