During CPU hotplug, vmstat_cpu_online() and vmstat_cpu_dead() invoke refresh_zone_stat_thresholds() on every CPU transition. Currently, refresh_zone_stat_thresholds() first zeroes per_cpu_nodestats->stat_threshold across all online CPUs and all online nodes, and then unconditionally writes per_cpu_zonestats->stat_threshold and per_cpu_nodestats->stat_threshold inside the zone loop for every populated zone and every online CPU. However, calculate_normal_threshold(zone) only changes when fls(num_online_cpus()) changes (at powers of 2) and is clamped at 125 for large normal zones (>= 16 GB) once num_online_cpus() >= 64. Unconditionally zeroing and rewriting every CPU's per-CPU stat_threshold on every hotplug step repeatedly dirties remote per-CPU cachelines with identical values (and briefly exposes stat_threshold == 0 on live CPUs). Compute pgdat_threshold across each node's populated zones first, hoist the per_cpu_nodestats->stat_threshold update out of the zone loop, and only write stat_threshold when the value actually changes. 2-socket, 480-CPU Intel Xeon 6985P-C (N=479, mean of 10 runs): Offline: 3.37 s -> 3.09 s (-8.3%) Online: 2.06 s -> 1.98 s (-3.9%) Total: 5.43 s -> 5.07 s (-6.6%) refresh_zone_stat_thresholds() (cpu479 offline / online): 1,158.9 us -> 135.3 us (-88.3%) / 1,468.2 us -> 128.6 us (-91.2%) 2-socket, 512-CPU AMD EPYC 9B45 (N=511, mean of 10 runs): Offline: 2.34 s -> 2.30 s (-1.7%) Online: 1.12 s -> 1.08 s (-3.6%) Total: 3.46 s -> 3.38 s (-2.3%) refresh_zone_stat_thresholds() (cpu511 offline / online): 104.6 us -> 50.7 us (-51.6%) / 109.3 us -> 53.5 us (-51.1%) Signed-off-by: Pasha Tatashin --- mm/vmstat.c | 58 ++++++++++++++++++++++++++++------------------------- 1 file changed, 31 insertions(+), 27 deletions(-) diff --git a/mm/vmstat.c b/mm/vmstat.c index cb57714539fb5..c671e9669b43b 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -280,41 +280,45 @@ void refresh_zone_stat_thresholds(void) int cpu; int threshold; - /* Zero current pgdat thresholds */ for_each_online_pgdat(pgdat) { - for_each_online_cpu(cpu) { - per_cpu_ptr(pgdat->per_cpu_nodestats, cpu)->stat_threshold = 0; - } - } + int pgdat_threshold = 0; - for_each_populated_zone(zone) { - struct pglist_data *pgdat = zone->zone_pgdat; - unsigned long max_drift, tolerate_drift; + for (int z = 0; z < MAX_NR_ZONES; z++) { + unsigned long max_drift, tolerate_drift; - threshold = calculate_normal_threshold(zone); + zone = &pgdat->node_zones[z]; + if (!populated_zone(zone)) + continue; - for_each_online_cpu(cpu) { - int pgdat_threshold; + threshold = calculate_normal_threshold(zone); + /* Base nodestat threshold on the largest populated zone. */ + pgdat_threshold = max(threshold, pgdat_threshold); - per_cpu_ptr(zone->per_cpu_zonestats, cpu)->stat_threshold - = threshold; + for_each_online_cpu(cpu) { + s8 *pz = &per_cpu_ptr(zone->per_cpu_zonestats, cpu)->stat_threshold; - /* Base nodestat threshold on the largest populated zone. */ - pgdat_threshold = per_cpu_ptr(pgdat->per_cpu_nodestats, cpu)->stat_threshold; - per_cpu_ptr(pgdat->per_cpu_nodestats, cpu)->stat_threshold - = max(threshold, pgdat_threshold); + if (*pz != threshold) + *pz = threshold; + } + + /* + * Only set percpu_drift_mark if there is a danger that + * NR_FREE_PAGES reports the low watermark is ok when in fact + * the min watermark could be breached by an allocation + */ + tolerate_drift = low_wmark_pages(zone) - min_wmark_pages(zone); + max_drift = num_online_cpus() * threshold; + if (max_drift > tolerate_drift) + zone->percpu_drift_mark = high_wmark_pages(zone) + + max_drift; } - /* - * Only set percpu_drift_mark if there is a danger that - * NR_FREE_PAGES reports the low watermark is ok when in fact - * the min watermark could be breached by an allocation - */ - tolerate_drift = low_wmark_pages(zone) - min_wmark_pages(zone); - max_drift = num_online_cpus() * threshold; - if (max_drift > tolerate_drift) - zone->percpu_drift_mark = high_wmark_pages(zone) + - max_drift; + for_each_online_cpu(cpu) { + s8 *pn = &per_cpu_ptr(pgdat->per_cpu_nodestats, cpu)->stat_threshold; + + if (*pn != pgdat_threshold) + *pn = pgdat_threshold; + } } } -- 2.56.0.385.gd3acb90ef8-goog