Sometimes, vm_area_alloc_pages() will want many pages from the buddy allocator. Rather than making requests to the buddy allocator for at most 100 pages at a time, we can eagerly request large order pages a smaller number of times. We still split the large order pages down to order-0 as the rest of the vmalloc code (and some callers) depend on it. We still defer to the bulk allocator and fallback path in case of order-0 pages or failure. Running 1000 iterations of allocations on a small 4GB system finds: 1000 2mb allocations: [Baseline] [This patch] real 46.310s real 34.380s user 0.001s user 0.008s sys 46.058s sys 34.152s 10000 200kb allocations: [Baseline] [This patch] real 56.104s real 43.946s user 0.001s user 0.003s sys 55.375s sys 43.259s 10000 20kb allocations: [Baseline] [This patch] real 0m8.438s real 0m9.160s user 0m0.001s user 0m0.002s sys 0m7.936s sys 0m8.671s This is an RFC, comments and thoughts are welcomed. There is a clear benefit to be had for large allocations, but there is some regression for smaller allocations. Signed-off-by: Vishal Moola (Oracle) --- mm/vmalloc.c | 34 +++++++++++++++++++++++++++++++++- 1 file changed, 33 insertions(+), 1 deletion(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 97cef2cc14d3..0a25e5cf841c 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -3621,6 +3621,38 @@ vm_area_alloc_pages(gfp_t gfp, int nid, unsigned int nr_allocated = 0; struct page *page; int i; + gfp_t large_gfp = (gfp & ~__GFP_DIRECT_RECLAIM) | __GFP_NOWARN; + unsigned int large_order = ilog2(nr_pages - nr_allocated); + + /* + * Initially, attempt to have the page allocator give us large order + * pages. Do not attempt allocating smaller than order chunks since + * __vmap_pages_range() expects physically contigous pages of exactly + * order long chunks. + */ + while (large_order > order && nr_allocated < nr_pages) { + /* + * High-order nofail allocations are really expensive and + * potentially dangerous (pre-mature OOM, disruptive reclaim + * and compaction etc. + */ + if (gfp & __GFP_NOFAIL) + break; + if (nid == NUMA_NO_NODE) + page = alloc_pages_noprof(large_gfp, large_order); + else + page = alloc_pages_node_noprof(nid, large_gfp, large_order); + + if (unlikely(!page)) + break; + + split_page(page, large_order); + for (i = 0; i < (1U << large_order); i++) + pages[nr_allocated + i] = page + i; + + nr_allocated += 1U << large_order; + large_order = ilog2(nr_pages - nr_allocated); + } /* * For order-0 pages we make use of bulk allocator, if @@ -3665,7 +3697,7 @@ vm_area_alloc_pages(gfp_t gfp, int nid, } } - /* High-order pages or fallback path if "bulk" fails. */ + /* High-order arch pages or fallback path if "bulk" fails. */ while (nr_allocated < nr_pages) { if (!(gfp & __GFP_NOFAIL) && fatal_signal_pending(current)) break; -- 2.51.0