A customer has reported excessive reclaim activity despite memory above high watermarks. Analyzing support data (including sar log) revealed the following scenario: - the system has almost all memory reserved as 1G hugepages for kvm guests, and operates normally - due to a spike of memory pressure from userspace, highatomic reservations occur (probably due to e.g. concurrent network receive activity) - on one of the 8 NUMA nodes, this creates a situation where free pages are above high watermark, but when __zone_watermark_unusable_free() considers highatomic pageblocks, the result appears to be below the low watermark - pgdat_balanced() is false, kswapd hopelessly keeps trying to balance, spending cpu time and disrupting the system. It has nothing to reclaim because the node is filled with hugetlbfs hugepages. - since the overal memory pressure is under control again and there are no allocations restricted to the affected node, they are satisfied from other nodes without going through direct reclaim - since direct reclaim is the only code that calls unreserve_highatomic_pageblock(), this never happens and nothing fixes the situation Note this was observed on a 6.4-based kernel and there have been recent fixes upstream limiting the highatomic reservation side, but I don't think they can prevent the same happening in mainline, only perhaps lower the chances. Nothing seems to prevent the issue completely. For example, while we limit the amount of highatomic pageblocks to 1% of the zone, this calculation does not subtract the amount of hugepages or otherwise permanent allocations, so the highatomic pageblocks can end up taking much higher share of actually reclaimable memory. The most targeted fix thus appears to be to allow kswapd perform the highatomic pageblock unreserve to match direct reclaim. It's also somewhat similar in principle to how kswapd reduces watermark boosting. Hook this highatomic unreserve to the moment we are increasing kswapd reclaim priority, as that is when kswapd is struggling to make progress and it should roughly match the case of direct reclaim failure. The only caveat is that direct reclaim knows which migratetype it wants to allocate and thus changes the highatomic pageblock to. Kswapd doesn't track that, so use MIGRATE_UNMOVABLE. Highatomic allocations usually are not movable, and this will avoid creating nominally movable pageblocks polluted with unmovable pages, so it should be the least problematic default. Signed-off-by: Vlastimil Babka (SUSE) --- We interrupt the regular stream of AI fixes by something a bit different. RFC to get agreement on the approach. Haven't tried reproducing the issue in mainline, but it seems quite clear to me from the data I've seen. --- mm/page_alloc.c | 126 +++++++++++++++++++++++++++++--------------------------- mm/page_alloc.h | 3 ++ mm/vmscan.c | 16 ++++++- 3 files changed, 83 insertions(+), 62 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index a8cac5e01066..785a13570788 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3476,6 +3476,68 @@ static void reserve_highatomic_pageblock(struct page *page, int order, } } +bool unreserve_highatomic_pageblock_zone(struct zone *zone, int migratetype, + bool force) +{ + /* + * Preserve at least one pageblock unless memory pressure + * is really high. + */ + if (!force && zone->nr_reserved_highatomic <= pageblock_nr_pages) + return false; + + guard(spinlock_irqsave)(&zone->lock); + for (int order = 0; order < NR_PAGE_ORDERS; order++) { + struct free_area *area = &(zone->free_area[order]); + unsigned long size; + struct page *page; + int ret; + + page = get_page_from_free_area(area, MIGRATE_HIGHATOMIC); + if (!page) + continue; + + size = max(pageblock_nr_pages, 1UL << order); + /* + * It should never happen but changes to locking could + * inadvertently allow a per-cpu drain to add pages to + * MIGRATE_HIGHATOMIC while unreserving so be safe and watch for + * underflows. + */ + if (WARN_ON_ONCE(size > zone->nr_reserved_highatomic)) + size = zone->nr_reserved_highatomic; + zone->nr_reserved_highatomic -= size; + + /* + * Convert to migratetype and avoid the normal pageblock + * stealing heuristics. Minimally, the caller is doing the work + * and needs the pages. More importantly, if the block was + * always converted to MIGRATE_UNMOVABLE or another type then + * the number of pageblocks that cannot be completely freed may + * increase. + */ + if (order < pageblock_order) + ret = move_freepages_block(zone, page, + MIGRATE_HIGHATOMIC, migratetype); + else { + move_to_free_list(page, zone, order, + MIGRATE_HIGHATOMIC, + migratetype); + change_pageblock_range(page, order, migratetype); + ret = 1; + } + /* + * Reserving the block(s) already succeeded, + * so this should not fail on zone boundaries. + */ + WARN_ON_ONCE(ret == -1); + if (ret > 0) + return true; + } + + return false; +} + /* * Used when an allocation is about to fail under memory pressure. This * potentially hurts the reliability of high-order allocations when under @@ -3491,70 +3553,12 @@ static bool unreserve_highatomic_pageblock(const struct alloc_context *ac, struct zonelist *zonelist = ac->zonelist; struct zoneref *z; struct zone *zone; - struct page *page; - int order; - int ret; for_each_zone_zonelist_nodemask(zone, z, zonelist, ac->highest_zoneidx, ac->nodemask) { - /* - * Preserve at least one pageblock unless memory pressure - * is really high. - */ - if (!force && zone->nr_reserved_highatomic <= - pageblock_nr_pages) - continue; - - guard(spinlock_irqsave)(&zone->lock); - for (order = 0; order < NR_PAGE_ORDERS; order++) { - struct free_area *area = &(zone->free_area[order]); - unsigned long size; - - page = get_page_from_free_area(area, MIGRATE_HIGHATOMIC); - if (!page) - continue; - - size = max(pageblock_nr_pages, 1UL << order); - /* - * It should never happen but changes to - * locking could inadvertently allow a per-cpu - * drain to add pages to MIGRATE_HIGHATOMIC - * while unreserving so be safe and watch for - * underflows. - */ - if (WARN_ON_ONCE(size > zone->nr_reserved_highatomic)) - size = zone->nr_reserved_highatomic; - zone->nr_reserved_highatomic -= size; - - /* - * Convert to ac->migratetype and avoid the normal - * pageblock stealing heuristics. Minimally, the caller - * is doing the work and needs the pages. More - * importantly, if the block was always converted to - * MIGRATE_UNMOVABLE or another type then the number - * of pageblocks that cannot be completely freed - * may increase. - */ - if (order < pageblock_order) - ret = move_freepages_block(zone, page, - MIGRATE_HIGHATOMIC, - ac->migratetype); - else { - move_to_free_list(page, zone, order, - MIGRATE_HIGHATOMIC, - ac->migratetype); - change_pageblock_range(page, order, - ac->migratetype); - ret = 1; - } - /* - * Reserving the block(s) already succeeded, - * so this should not fail on zone boundaries. - */ - WARN_ON_ONCE(ret == -1); - if (ret > 0) - return ret; - } + if (unreserve_highatomic_pageblock_zone(zone, + ac->migratetype, force)) + return true; } return false; diff --git a/mm/page_alloc.h b/mm/page_alloc.h index e238ee91ed24..9fba6480b18d 100644 --- a/mm/page_alloc.h +++ b/mm/page_alloc.h @@ -282,6 +282,9 @@ static inline bool free_area_empty(struct free_area *area, int migratetype) return list_empty(&area->free_list[migratetype]); } +bool unreserve_highatomic_pageblock_zone(struct zone *zone, int migratetype, + bool force); + /* Convert GFP flags to their corresponding migrate type */ #define GFP_MOVABLE_MASK (__GFP_RECLAIMABLE|__GFP_MOVABLE) #define GFP_MOVABLE_SHIFT 3 diff --git a/mm/vmscan.c b/mm/vmscan.c index 9959ca158ce8..91c622068d99 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -7460,8 +7460,22 @@ static int balance_pgdat(pg_data_t *pgdat, int order, int highest_zoneidx) if (nr_boost_reclaim && !nr_reclaimed) break; - if (raise_priority || !nr_reclaimed) + if (raise_priority || !nr_reclaimed) { + /* + * Similarly to when direct reclaim is retrying, try to + * unreserve highatomic pageblocks on the node. Convert + * them to movable ones as kswapd doesn't have a target + * migratetype, and passing unmovable should lead to + * least permanent fragmentation. + */ + for (int i = 0; i <= highest_zoneidx; i++) { + struct zone *zone = pgdat->node_zones + i; + + unreserve_highatomic_pageblock_zone(zone, + MIGRATE_UNMOVABLE, false); + } sc.priority--; + } } while (sc.priority >= 1); /* --- base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4 change-id: 20261030-kswapd-highatomic-unreserve-7e6629ec65d0