From: Qi Zheng Allow hugetlb_cma=@ to select the architecture's PMD size in addition to its existing CMA size. PMD pages are 2M on x86-64 and 2M, 32M or 512M on arm64 with 4K, 16K or 64K base pages. Give PMD pages their own pool while preserving the existing sharing among gigantic sizes. Unsuffixed arguments and an explicit default size continue to configure the default pool. Gigantic PMD sizes retain access to that pool when no PMD pool is reserved. Apply this mapping consistently to allocation, CMA-only policy and boot-time accounting. Try buddy allocation first for non-gigantic PMD pages, with CMA as a fallback, to reduce CMA migration and leave CMA capacity for later allocations. Preserve CMA-only policy and CMA-first allocation for gigantic pages. Align reservations to both the selected page size and CMA's minimum alignment, and use a bitmap granularity no larger than the selected order. Keep the original CMA area names and give PMD areas distinct names. Document the syntax and pool selection. Assisted-by: LLM Signed-off-by: Qi Zheng --- .../admin-guide/kernel-parameters.txt | 40 ++++++++++++------- mm/hugetlb.c | 18 ++++----- mm/hugetlb_cma.c | 22 +++++----- 3 files changed, 45 insertions(+), 35 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index c06cb660a3ce4..0cc77e648f396 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -2065,31 +2065,43 @@ Kernel parameters Note that this parameter only applies to non-gigantic huge pages. hugetlb_cma= [HW,CMA,EARLY] The size of a CMA area used for allocation - of gigantic hugepages. Or using node format, the size - of a CMA area per node can be specified. + of hugepages. Or using node format, the size of a CMA area + per node can be specified. The size can be an absolute value (e.g., 2G) or a percentage of the total memory or node memory (e.g., 20%). Percentage-derived sizes are rounded down to a multiple of - the architecture's gigantic hugepage size and may become - zero. - Format: nn[KMGTPE] or nn% or (node format) - :nn[KMGTPE][,:nn[KMGTPE]] or - :nn%[,:nn%] - - The size must be a multiple of the gigantic page size. + the larger of the selected hugepage size and CMA's minimum + alignment, and may become zero. + Format: nn[KMGTPE][@] or nn%[@] + or (node format) + :nn[KMGTPE][,:nn[KMGTPE]][@] or + :nn%[,:nn%][@] + + An optional @ suffix selects the architecture's + default CMA hugepage size or a supported PMD size. If + omitted, the default is used. For node format, append + the suffix after the entire node list. Repeat the + parameter to configure the two pools independently. + + For CMA allocation, PMD pages use the PMD pool if reserved. + Otherwise, gigantic PMD pages use the default pool. Other + gigantic page sizes always share the default pool. + + The size must be aligned to both the selected hugepage + size and CMA's minimum alignment. When using node format, this applies to each per-node size. Missaligned values are dropped with a warning. - Reserve a CMA area of given size and allocate gigantic - hugepages using the CMA allocator. If enabled, the - boot-time allocation of gigantic hugepages is skipped. + For gigantic page sizes with a reserved CMA pool, normal + boot-time allocation is skipped. With hugetlb_cma_only=1, + early CMA allocation is used where supported. hugetlb_cma_only= [HW,CMA,EARLY] When allocating new HugeTLB pages, only try to allocate from the CMA areas. - This option does nothing if hugetlb_cma= is not also - specified. + This option only affects page sizes with a CMA pool + successfully reserved by hugetlb_cma=. hugetlb_free_vmemmap= [KNL] Requires CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP diff --git a/mm/hugetlb.c b/mm/hugetlb.c index c1a8d00be11ac..bad5cd4978c5e 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1835,22 +1835,20 @@ static struct folio *only_alloc_fresh_hugetlb_folio(struct hstate *h, { struct folio *folio = NULL; int order = huge_page_order(h); + bool cma_only = hugetlb_cma_exclusive_alloc(order); if (nid == NUMA_NO_NODE) nid = numa_mem_id(); - if (allow_cma && order_is_gigantic(order)) + if (!order_is_gigantic(order) && !cma_only) + folio = alloc_buddy_frozen_folio(order, gfp_mask, nid, nmask, + node_alloc_noretry); + + if (!folio && allow_cma) folio = hugetlb_cma_alloc_frozen_folio(order, gfp_mask, nid, nmask); - if (!folio) { - if (hugetlb_cma_exclusive_alloc(order)) - return NULL; - if (order_is_gigantic(order)) - folio = alloc_gigantic_frozen_folio(order, gfp_mask, nid, nmask); - else - folio = alloc_buddy_frozen_folio(order, gfp_mask, nid, nmask, - node_alloc_noretry); - } + if (!folio && order_is_gigantic(order) && !cma_only) + folio = alloc_gigantic_frozen_folio(order, gfp_mask, nid, nmask); if (folio) init_new_hugetlb_folio(folio); diff --git a/mm/hugetlb_cma.c b/mm/hugetlb_cma.c index c279c3f548705..5071f0ae012bd 100644 --- a/mm/hugetlb_cma.c +++ b/mm/hugetlb_cma.c @@ -180,11 +180,13 @@ static int __init hugetlb_cma_parse(char *p) pr_warn("hugetlb_cma: invalid parameter %s\n", p); return -EINVAL; } - if (!order || page_size != (PAGE_SIZE << order) || + if ((page_size != PMD_SIZE && + page_size != (PAGE_SIZE << order)) || !arch_hugetlb_valid_size(page_size)) { pr_warn("hugetlb_cma: unsupported page size in %s\n", p); return -EINVAL; } + order = ilog2(page_size) - PAGE_SHIFT; } if (!order) { @@ -339,14 +341,7 @@ static void __init hugetlb_cma_reserve_pool(struct hugetlb_cma_pool *pool, if (!pool->size) return; - /* - * HugeTLB CMA reservation is required for gigantic - * huge pages which could not be allocated via the - * page allocator. Just warn if there is any change - * breaking this assumption. - */ - VM_WARN_ON(order <= MAX_PAGE_ORDER); - hugepage_size = PAGE_SIZE << order; + hugepage_size = max(PAGE_SIZE << order, CMA_MIN_ALIGNMENT_BYTES); if (config->percent) { unsigned long orig_size = pool->size; @@ -437,14 +432,19 @@ static void __init hugetlb_cma_reserve_pool(struct hugetlb_cma_pool *pool, size = min(per_node, pool->size - reserved); } - snprintf(name, sizeof(name), "hugetlb%d", nid); + if (pool == &hugetlb_cma[HUGETLB_CMA_DEFAULT]) + snprintf(name, sizeof(name), "hugetlb%d", nid); + else + snprintf(name, sizeof(name), "hugetlb-%lukB-%d", + (PAGE_SIZE << order) / SZ_1K, nid); + /* * Note that 'order per bit' is based on smallest size that * may be returned to CMA allocator in the case of * huge page demotion. */ res = cma_declare_contiguous_multi(size, hugepage_size, - HUGETLB_PAGE_ORDER, name, + min_t(unsigned int, order, HUGETLB_PAGE_ORDER), name, &pool->cma[nid], nid); if (res || !cma_validate_zones(pool->cma[nid])) { pr_warn("hugetlb_cma: %s: err %d, node %d\n", -- 2.54.0