We can skip the rstat flush in workingset_test_recent() when the eviction memcg is the root memcg. lruvec_page_state() now reads the LRU sizes of the root memcg from the node counters, which are always up to date, and workingset_test_recent() only reads the eviction memcg's own lruvec. This avoids an expensive flush on the refault path, as flushing the root memcg flushes every memcg in the system. For reclaim however, we still need to keep the flush (for now) since global and root reclaim also read the rstat stats of the child memcgs they scan, which the flush is needed for in order to keep up to date. On an 80-CPU machine, a root flush took 69us on average with little cgroup activity and ~0.9ms with 128 memcgs actively allocating (wall time, including waiting for the rstat lock). Reading a node counter doesn't add any cost. It took the same amount of time as reading the rstat counter (~150ns, which includes bpftrace probe overhead). Signed-off-by: Joanne Koong --- mm/workingset.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/mm/workingset.c b/mm/workingset.c index 1504f91cdca5..383f951eaad4 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -483,10 +483,13 @@ bool workingset_test_recent(void *shadow, bool file, bool *workingset, * section (for e.g, in cachestat) - these callers need to skip flushing * stats (via the flush argument). * + * For the root memcg, the LRU sizes are read directly from the node + * counters (see lruvec_page_state()). + * * XXX: With per-memcg flushing and thresholding, is ratelimiting * still needed here? */ - if (flush) + if (flush && !mem_cgroup_is_root(eviction_memcg)) mem_cgroup_flush_stats_ratelimited(eviction_memcg); eviction_lruvec = mem_cgroup_lruvec(eviction_memcg, pgdat); -- 2.52.1