From: Hui Zhu Expose memcg proactive reclaim to sleepable BPF programs: unsigned long bpf_proactive_reclaim(memcg, size); unsigned long bpf_proactive_reclaim_swappiness(memcg, size, swappiness); They perform one reclaim pass on @memcg, like a write to memory.reclaim: swap is allowed, and the anon/file balance follows the cgroup's swappiness or an explicit override in [MIN_SWAPPINESS, MAX_SWAPPINESS] plus SWAPPINESS_ANON_ONLY. Both go through a shared helper, bpf_proactive_reclaim_pages(), which guards against reclaim recursion and calls try_to_free_mem_cgroup_pages() with GFP_KERNEL and MEMCG_RECLAIM_MAY_SWAP | MEMCG_RECLAIM_PROACTIVE, the same parameters user_proactive_reclaim() uses, and unlike memory.reclaim they do not retry until @size is reached. Reclaim must not recurse: try_to_free_mem_cgroup_pages() overwrites current->reclaim_state on entry and NULLs it on exit, so a nested call from an in-flight reclaim would corrupt the outer reclaim state (e.g. MGLRU dereferences current->reclaim_state->mm_walk). Both kfuncs therefore refuse to reclaim when PF_MEMALLOC is set, mirroring the guards in the memcg charging path and node_reclaim(). Signed-off-by: Hui Zhu --- mm/bpf_memcontrol.c | 101 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 101 insertions(+) diff --git a/mm/bpf_memcontrol.c b/mm/bpf_memcontrol.c index 716df49d7647..b0a0d4c55dc4 100644 --- a/mm/bpf_memcontrol.c +++ b/mm/bpf_memcontrol.c @@ -6,6 +6,7 @@ */ #include +#include #include __bpf_kfunc_start_defs(); @@ -159,6 +160,103 @@ __bpf_kfunc void bpf_mem_cgroup_flush_stats(struct mem_cgroup *memcg) mem_cgroup_flush_stats(memcg); } +/* + * Reclaim must not recurse. try_to_free_mem_cgroup_pages() unconditionally + * overwrites current->reclaim_state on entry and resets it to NULL on exit. + * So invoking it from an in-flight reclaim would clobber the outer reclaim + * state and corrupt its accounting. + * + * The guard is PF_MEMALLOC. Every reclaim entry point marks the current + * task with it for the whole reclaim window: try_to_free_mem_cgroup_pages() + * and __perform_reclaim() do so via memalloc_noreclaim_save(), and kswapd + * keeps it set for its entire lifetime. A hook inside the reclaim path + * (shrink_node, shrink_slab, ...) executes in the context of the + * reclaiming task, where current->flags already carries the flag. The page + * allocator, the memcg charging path and node_reclaim() rely on the same + * flag to avoid reclaim recursion. + * + * In try_to_free_mem_cgroup_pages(), reclaim_state is set slightly before + * PF_MEMALLOC, with only a tracepoint in between, which a sleepable BPF + * program cannot attach to. + * Also, PF_MEMALLOC is set in some non-reclaim contexts (e.g. direct compaction + * and vmalloc), where the kfunc conservatively refuses to reclaim as well. + */ +static bool bpf_in_reclaim_context(void) +{ + return current->flags & PF_MEMALLOC; +} + +/* + * Shared implementation of the proactive reclaim kfuncs: performs one + * reclaim pass on @memcg with @nr_pages as the goal, allowing swap, and + * @swappiness as the anon/file balance override (NULL to follow the + * cgroup's own swappiness setting). + */ +static unsigned long +bpf_proactive_reclaim_pages(struct mem_cgroup *memcg, unsigned long nr_pages, + int *swappiness) +{ + if (!nr_pages || unlikely(bpf_in_reclaim_context())) + return 0; + + return try_to_free_mem_cgroup_pages(memcg, nr_pages, GFP_KERNEL, + MEMCG_RECLAIM_MAY_SWAP | + MEMCG_RECLAIM_PROACTIVE, + swappiness); +} + +/** + * bpf_proactive_reclaim - proactively reclaim memory from a memory + * cgroup + * @memcg: the target memory cgroup to reclaim from + * @size: the amount of memory to reclaim, in bytes + * + * Trigger one proactive reclaim pass on @memcg, similar to a write to + * the memory.reclaim cgroup file: pages are reclaimed according to the + * cgroup's own swappiness setting and swap is allowed. Note that, + * unlike memory.reclaim, this does not retry until @size is reached; + * callers can invoke it again if needed. + * + * Return: + * The number of pages actually reclaimed, or 0 if @size is smaller + * than a page or the calling task is already in a reclaim/freeing + * context (PF_MEMALLOC). + */ +__bpf_kfunc unsigned long bpf_proactive_reclaim(struct mem_cgroup *memcg, + unsigned long size) +{ + return bpf_proactive_reclaim_pages(memcg, size / PAGE_SIZE, NULL); +} + +/** + * bpf_proactive_reclaim_swappiness - proactively reclaim memory from a + * memory cgroup with an explicit + * swappiness + * @memcg: the target memory cgroup to reclaim from + * @size: the amount of memory to reclaim, in bytes + * @swappiness: swappiness override for this reclaim pass + * + * Same as bpf_proactive_reclaim(), except that the anon/file reclaim + * balance is controlled by @swappiness instead of the cgroup's + * swappiness setting. Valid values are [MIN_SWAPPINESS, MAX_SWAPPINESS] + * and SWAPPINESS_ANON_ONLY, which restricts reclaim to anon folios. + * + * Return: + * The number of pages actually reclaimed, or 0 if @size is smaller + * than a page, @swappiness is out of range, or the calling task is + * already in a reclaim/freeing context (PF_MEMALLOC). + */ +__bpf_kfunc unsigned long +bpf_proactive_reclaim_swappiness(struct mem_cgroup *memcg, unsigned long size, + int swappiness) +{ + if (swappiness < MIN_SWAPPINESS || swappiness > SWAPPINESS_ANON_ONLY) + return 0; + + return bpf_proactive_reclaim_pages(memcg, size / PAGE_SIZE, + &swappiness); +} + __bpf_kfunc_end_defs(); BTF_KFUNCS_START(bpf_memcontrol_kfuncs) @@ -172,6 +270,9 @@ BTF_ID_FLAGS(func, bpf_mem_cgroup_usage) BTF_ID_FLAGS(func, bpf_mem_cgroup_page_state) BTF_ID_FLAGS(func, bpf_mem_cgroup_flush_stats, KF_SLEEPABLE) +BTF_ID_FLAGS(func, bpf_proactive_reclaim, KF_SLEEPABLE) +BTF_ID_FLAGS(func, bpf_proactive_reclaim_swappiness, KF_SLEEPABLE) + BTF_KFUNCS_END(bpf_memcontrol_kfuncs) static const struct btf_kfunc_id_set bpf_memcontrol_kfunc_set = { -- 2.53.0