From: Ackerley Tng When mem_cgroup_charge_hugetlb() fails with -ENOMEM, alloc_hugetlb_folio() currently propagates this error. This results in the page fault handler returning VM_FAULT_OOM. Because HugeTLB allocations are high-order and use __GFP_RETRY_MAYFAIL, they bypass the OOM killer. Returning VM_FAULT_OOM to the #PF handler without triggering the OOM killer (or having it make progress) leads to an infinite loop of retrying the fault. Avoid this loop by returning -ENOSPC when charging fails, which maps to VM_FAULT_SIGBUS, terminating the process cleanly. Make mem_cgroup_charge_hugetlb() fault handling use a common error handling path, the same handling used for hugetlb_cgroup_uncharge_cgroup{,_rsvd}(), which also don't trigger the OOM killer and hence opt to terminate the process with a SIGBUS. Fixes: 991135774c0e0 ("memcg/hugetlb: introduce mem_cgroup_charge_hugetlb") Signed-off-by: Ackerley Tng --- mm/hugetlb.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 46e8dfffcfa5d..e42049e2c038b 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -2997,7 +2997,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, if (ret == -ENOMEM) { folio_put(folio); - return ERR_PTR(-ENOMEM); + goto err; } return folio; @@ -3020,6 +3020,17 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, out_end_reservation: if (map_chg != MAP_CHG_ENFORCED) vma_end_reservation(h, vma, addr); +err: + /* + * Return -ENOSPC when this function fails to allocate or + * charge a huge page. If a standard (PAGE_SIZE) page + * allocation fails, the OOM killer is given a chance to run, + * which may resolve the failure on retry. However, for + * HugeTLB allocations, the OOM killer is not triggered. + * Returning -ENOMEM (or anything resulting in VM_FAULT_OOM) + * would leak to the #PF handler, causing it to loop + * indefinitely retrying the fault. + */ return ERR_PTR(-ENOSPC); } -- 2.55.0.795.g602f6c329a-goog