When onlining memory using the "auto-movable" online policy with nid == NUMA_NO_NODE, auto_movable_can_online_movable() walks all populated zones across all nodes on each invocation to collect the MOVABLE vs. KERNEL_EARLY stats. This walk happens twice for every memory block onlining decision (global check and, with NUMA awareness enabled, per-node check) and is repeated for every single memory block that gets onlined, although the underlying zone counters rarely change. Cache the stats for nid == NUMA_NO_NODE and only recalculate them when a zone counter actually changes. All modifications of the relevant zone counters (present_pages, present_early_pages) funnel through adjust_present_page_count(), which is only called while holding the mem_hotplug_lock in write mode and the device_lock() of the memory block device. Invalidate the cache there, such that the next auto-movable onlining decision recalculates the stats from scratch. CMA adjustments (zone->cma_pages) only happen during boot (via init_cma_reserved_pageblock()/init_cma_pageblock(), both __init) and cannot race with memory onlining. Boot-time zone initialization happens before any memory block can be onlined. The per-node path (nid != NUMA_NO_NODE) is left uncached; it only walks a single node's zones. Resolve a TODO that was left when the "auto-movable" online policy was introduced. Tested on QEMU (x86_64, 512M boot + 256M hotplugged pc-dimm, online_policy=auto-movable): hotplugged memory blocks get onlined to ZONE_MOVABLE as expected, and an offline/online cycle of a hotplugged block shows the correct zone counters after cache invalidation, with no kernel warnings. Link: https://lore.kernel.org/r/20210806124715.17090-3-david@redhat.com Signed-off-by: Zhijian Han --- mm/memory_hotplug.c | 26 +++++++++++++++++++++++--- 1 file changed, 23 insertions(+), 3 deletions(-) diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index d7a59167bec4..dfac59739a6d 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -789,6 +789,19 @@ struct auto_movable_stats { unsigned long movable_pages; }; +/* + * Cached stats for all populated zones across all nodes. Modified (incl. + * invalidation) only while holding the mem_hotplug_lock in write mode and + * the device_lock() of the memory block device. + */ +static struct auto_movable_stats auto_movable_zone_stats; +static bool auto_movable_zone_stats_valid; + +static void auto_movable_zone_stats_invalidate(void) +{ + auto_movable_zone_stats_valid = false; +} + static void auto_movable_stats_account_zone(struct auto_movable_stats *stats, struct zone *zone) { @@ -849,9 +862,14 @@ static bool auto_movable_can_online_movable(int nid, struct memory_group *group, /* Walk all relevant zones and collect MOVABLE vs. KERNEL stats. */ if (nid == NUMA_NO_NODE) { - /* TODO: cache values */ - for_each_populated_zone(zone) - auto_movable_stats_account_zone(&stats, zone); + if (auto_movable_zone_stats_valid) { + stats = auto_movable_zone_stats; + } else { + for_each_populated_zone(zone) + auto_movable_stats_account_zone(&stats, zone); + auto_movable_zone_stats = stats; + auto_movable_zone_stats_valid = true; + } } else { for (i = 0; i < MAX_NR_ZONES; i++) { pg_data_t *pgdat = NODE_DATA(nid); @@ -1079,6 +1097,8 @@ void adjust_present_page_count(struct page *page, struct memory_group *group, zone->present_pages += nr_pages; zone->zone_pgdat->node_present_pages += nr_pages; + auto_movable_zone_stats_invalidate(); + if (group && movable) group->present_movable_pages += nr_pages; else if (group && !movable) -- 2.43.0