With debug_guardpage_minorder set, expand() turns the unused halves of a split into guard pages, which page_del_and_expand() leaves out of the free counts. Freeing the allocated half credits only that half, and __free_one_page() then merges the guard buddy with clear_page_guard(), which no longer touches the counters. The merged block goes on a free list with the guard pages never counted, so NR_FREE_PAGES falls behind the free lists by every guard that merges. Before commit e0932b6c1f94 ("mm: page_alloc: consolidate free page accounting"), __set_page_guard() and __clear_page_guard() adjusted the counts themselves; that commit dropped both adjustments but kept the subtraction implicit in expand(). Count the guard pages under the merged block's migratetype when the buddy is cleared. account_freepages() skips an isolated block as it does for every other free page, and moving the block off the isolated list counts it then. Booting a 16GB VM with debug_pagealloc=on debug_guardpage_minorder=1, check the difference in free pages reported between /proc/vmstat and /proc/buddyinfo. At three points (idle, after a read and file-creation load, after a second load), check the difference between nr_free_pages and the buddyinfo numbers in the normal zone, as a number of pages: idle after load after second load unpatched -130 79205 99121 patched -16 45 0 The small remaining difference seems to be due to the numbers not being read at exactly the same time, and is also seen without guard pages. Fixes: e0932b6c1f94 ("mm: page_alloc: consolidate free page accounting") Cc: stable@vger.kernel.org Assisted-by: LLM Signed-off-by: Rik van Riel --- mm/page_alloc.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 12fac9084c483..4658af97be010 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -988,10 +988,12 @@ static inline void __free_one_page(struct page *page, * Our buddy is free or it is CONFIG_DEBUG_PAGEALLOC guard page, * merge with it and move up one order. */ - if (page_is_guard(buddy)) + if (page_is_guard(buddy)) { clear_page_guard(zone, buddy, order); - else + account_freepages(zone, 1 << order, migratetype); + } else { __del_page_from_free_list(buddy, zone, order, buddy_mt); + } if (unlikely(buddy_mt != migratetype)) { /* -- 2.53.0-Meta __free_one_page() merges a freed page with a free buddy of another mergeable type at pageblock order and above, and retypes the buddy to match. The buddy's pages stay counted under the old type: nothing moves them to the merged type in the free counts. That is unobservable today. NR_FREE_PAGES does not separate types, so the missing move nets to zero there. Any future per-type free count would undercount on a merge from the other side, and allocating the merged run later could wrap it. Move the buddy's pages to the merged type in the count too. Assisted-by: LLM Signed-off-by: Rik van Riel --- mm/page_alloc.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 4658af97be010..ae38e25971559 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -993,6 +993,11 @@ static inline void __free_one_page(struct page *page, account_freepages(zone, 1 << order, migratetype); } else { __del_page_from_free_list(buddy, zone, order, buddy_mt); + /* The buddy's free pages join the merged block's type. */ + if (unlikely(buddy_mt != migratetype)) { + account_freepages(zone, -(1 << order), buddy_mt); + account_freepages(zone, 1 << order, migratetype); + } } if (unlikely(buddy_mt != migratetype)) { -- 2.53.0-Meta try_to_claim_block() converts a movable pageblock for a non-movable allocation only when half its pages are free or compatible. Otherwise the steal path takes pages without converting, and non-movable content sits in a block still typed movable. Convert a movable block on any non-movable claim, however few of its pages are free, so the block's type matches the allocation it serves. When other pages in the block are freed later, they end up on the non-movable free lists, directing more non-movable allocations to the already non-movable page blocks. Converting a non-movable pageblock to movable is made stricter: a movable claim takes a non-movable block only when every page is free or movable. Blocks straddling a zone edge are never claimed; zone_spans_pageblock() tests for a block wholly inside its zone. Assisted-by: LLM Signed-off-by: Rik van Riel --- include/linux/mmzone.h | 13 ++++++++ mm/page_alloc.c | 71 ++++++++++++++++++++---------------------- 2 files changed, 46 insertions(+), 38 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 94f9c3ff54160..81ad83ff69073 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -1227,6 +1227,19 @@ static inline bool zone_spans_pfn(const struct zone *zone, unsigned long pfn) return zone->zone_start_pfn <= pfn && pfn < zone_end_pfn(zone); } +/* + * Whether the pageblock holding @pfn lies wholly inside @zone. Zone + * spans are not pageblock-aligned, so the edge pageblocks of a zone + * can straddle into the next zone; those never change type. + */ +static inline bool zone_spans_pageblock(const struct zone *zone, unsigned long pfn) +{ + unsigned long start = pageblock_start_pfn(pfn); + + return zone->zone_start_pfn <= start && + start + pageblock_nr_pages <= zone_end_pfn(zone); +} + static inline bool zone_is_initialized(const struct zone *zone) { return zone->initialized; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index ae38e25971559..9f1520506ae1d 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2295,18 +2295,39 @@ find_suitable_fallback(struct free_area *area, unsigned int order, } /* - * This function implements actual block claiming behaviour. If order is large - * enough, we can claim the whole pageblock for the requested migratetype. If - * not, we check the pageblock for constituent pages; if at least half of the - * pages are free or compatible, we can still claim the whole block, so pages - * freed in the future will be put on the correct free list. + * Whether to retype a partly used block. A block wholly inside its zone + * takes the allocation's type, so no non-movable page sits in a + * movable-typed block; a straddling block is never retyped. */ +static bool should_claim_used_block(struct zone *zone, unsigned long start_pfn, + int start_type, int block_type, + int free_pages, int movable_pages) +{ + if (!zone_spans_pageblock(zone, start_pfn)) + return false; + + if (page_group_by_mobility_disabled) + return true; + + /* Convert to movable only if no non-movable pages are present. */ + if (is_migrate_movable(start_type)) + return free_pages + movable_pages == pageblock_nr_pages; + + /* Non-movable pages break compaction; claim as non-movable */ + if (is_migrate_movable(block_type)) + return true; + + /* Unmovable or reclaimable claiming from the other. */ + return free_pages >= (1 << (pageblock_order - 1)); +} + +/* Convert the block so later frees use the allocation's migratetype. */ static struct page * try_to_claim_block(struct zone *zone, struct page *page, int current_order, int order, int start_type, int block_type, unsigned int alloc_flags) { - int free_pages, movable_pages, alike_pages; + int free_pages, movable_pages; unsigned long start_pfn; /* Take ownership for orders >= pageblock_order */ @@ -2333,39 +2354,13 @@ try_to_claim_block(struct zone *zone, struct page *page, &movable_pages)) return NULL; - /* - * Determine how many pages are compatible with our allocation. - * For movable allocation, it's the number of movable pages which - * we just obtained. For other types it's a bit more tricky. - */ - if (start_type == MIGRATE_MOVABLE) { - alike_pages = movable_pages; - } else { - /* - * If we are falling back a RECLAIMABLE or UNMOVABLE allocation - * to MOVABLE pageblock, consider all non-movable pages as - * compatible. If it's UNMOVABLE falling back to RECLAIMABLE or - * vice versa, be conservative since we can't distinguish the - * exact migratetype of non-movable pages. - */ - if (block_type == MIGRATE_MOVABLE) - alike_pages = pageblock_nr_pages - - (free_pages + movable_pages); - else - alike_pages = 0; - } - /* - * If a sufficient number of pages in the block are either free or of - * compatible migratability as our allocation, claim the whole block. - */ - if (free_pages + alike_pages >= (1 << (pageblock_order-1)) || - page_group_by_mobility_disabled) { - __move_freepages_block(zone, start_pfn, block_type, start_type); - set_pageblock_migratetype(pfn_to_page(start_pfn), start_type); - return __rmqueue_smallest(zone, order, start_type); - } + if (!should_claim_used_block(zone, start_pfn, start_type, block_type, + free_pages, movable_pages)) + return NULL; - return NULL; + __move_freepages_block(zone, start_pfn, block_type, start_type); + set_pageblock_migratetype(pfn_to_page(start_pfn), start_type); + return __rmqueue_smallest(zone, order, start_type); } /* -- 2.53.0-Meta __rmqueue_steal() hands a non-movable allocation pages from a movable pageblock and leaves the block's type alone. Those pages can never migrate, so the block reads movable to compaction while holding content compaction cannot move. On a 3 GB guest running ls -lR / and a 2.4 GB block device read against a fragmenter, a temporary counter found 2654, 5341 and 5272 such allocations in three boots. __rmqueue_claim() runs first and would convert the block, but rmqueue_bulk() remembers the mode across its batch. Prevent those allocations from placing non-movable pages inside movable pageblocks by refusing __rmqueue_steal() for non-movable allocations in movable pageblocks. Movable is last in fallbacks[] for both non-movable types, so only a larger order is left to try; failing that, the steal returns NULL and the allocation will loop around to a claim, another zone, or reclaim. Keep the steal where mobility grouping is disabled, or where the block straddles a zone edge. A block that straddles zones cannot be claimed, because the allocator only holds the lock for one zone, which leaves stealing as the alloc path there. Movable allocations can steal, because kcompactd can always move those pages out of non-movable blocks later. Over three boots allocstall_normal runs 48 to 129 against 32 to 134 on the base, and compact_stall 324 to 415 against 288 to 387. Neither direct reclaim nor compaction stalls rise beyond run-to-run spread. Assisted-by: LLM Signed-off-by: Rik van Riel --- mm/page_alloc.c | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 9f1520506ae1d..16d3594d913a0 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2423,8 +2423,10 @@ __rmqueue_claim(struct zone *zone, int order, int start_migratetype, } /* - * Try to steal a single page from some fallback migratetype. Leave the rest of - * the block as its current migratetype, potentially causing fragmentation. + * Try to steal one page from a fallback type, leaving the rest of the block + * unchanged and possibly fragmented. A non-movable allocation must not steal + * from a movable block: without converting its type, the steal would leave + * non-movable content under a movable type. */ static __always_inline struct page * __rmqueue_steal(struct zone *zone, int order, int start_migratetype) @@ -2444,6 +2446,19 @@ __rmqueue_steal(struct zone *zone, int order, int start_migratetype) continue; page = get_page_from_free_area(area, fallback_mt); + + /* + * Do not allow non-movable allocations in movable + * pageblocks; that could break compaction. + * Non-movable allocations should claim pageblocks, instead. + */ + if (!is_migrate_movable(start_migratetype) && + is_migrate_movable(fallback_mt) && + !page_group_by_mobility_disabled && + zone_spans_pageblock(zone, page_to_pfn(page))) { + continue; + } + page_del_and_expand(zone, page, order, current_order, fallback_mt); trace_mm_page_alloc_extfrag(page, order, current_order, start_migratetype, fallback_mt); -- 2.53.0-Meta compaction_capture() hands any merged block to the compacting task with no type change, except it refuses a movable block to a non-movable request below pageblock order. A non-movable request capturing a whole movable block leaves non-movable content in a movable-typed block. Make a non-movable capture claim a movable block: convert it to the request. A partial block cannot be retyped, so the refusal below pageblock order stays; reaching the conversion means a whole block. A movable capture steals: it takes the pages and leaves the type alone. Assisted-by: LLM Signed-off-by: Rik van Riel --- mm/page_alloc.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 16d3594d913a0..edb69d8a74c73 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -728,6 +728,9 @@ static inline struct capture_control *task_capc(struct zone *zone) capc->zone == zone ? capc : NULL; } +static void change_pageblock_range(struct page *pageblock_page, + int start_order, int migratetype); + static inline bool compaction_capture(struct capture_control *capc, struct page *page, int order, int migratetype) @@ -751,6 +754,19 @@ compaction_capture(struct capture_control *capc, struct page *page, capc->migratetype != MIGRATE_MOVABLE) return false; + /* + * A non-movable capture claims a movable block: convert it to + * the request. A movable capture steals: take the pages and + * leave the type alone. Partial movable blocks returned above, + * so reaching here means a whole block. + */ + if (capc->migratetype != MIGRATE_MOVABLE && + migratetype == MIGRATE_MOVABLE) { + change_pageblock_range(page, order, capc->migratetype); + /* Converted whole, so no fragmentation to report below. */ + migratetype = capc->migratetype; + } + if (migratetype != capc->migratetype) trace_mm_page_alloc_extfrag(page, capc->order, order, capc->migratetype, migratetype); -- 2.53.0-Meta