A task stuck in memcg limit enforcement in the charge path keeps its locks, and tasks in other cgroups that need them wait for a limit that has nothing to do with them. memory.max is a hard limit, so the charge path has to enforce it. memory.high doesn't: since commit b23afb93d317 ("memcg: punt high overage reclaim to return-to-userland path"), the charge only records the overage, and the task reclaims and is throttled on return to userspace, where it holds no locks. Avoiding this priority inversion was one of the reasons for that commit. But return to userspace is too late when one system call charges a lot, like mlock() of a large range. So commit c9afe31ec443 ("memcg: synchronously enforce memory.high for large overcharges") made a charge that can block handle the overage right away once more than a batch is pending. That step reclaims and sleeps, up to 2s per call, under whatever locks the caller holds: mmap_lock while mlock() faults in a MADV_HUGEPAGE range, or invalidate_lock and locked folios in readahead. It also misses THP charges in other ranges, which can't block under the default defrag setting. The previous patches handle the overage where the task holds no locks, or only the locks of its file, in the operations that charge the most in one call: populating memory, shmem fallocate, buffered writes, reads into user memory and MADV_COLLAPSE. Remove the synchronous step. Other paths now handle the overage only on return to userspace, and memory.max still limits them. Many can get a hook later. For example, reading /dev/urandom into an untouched buffer reaches 513M under memory.high=64M. With THP on, it did before too, but with small pages the synchronous step used to throttle it. Another is mprotect() making an mlocked private range writable, which copies its pages with mmap_lock held for writing. Update the memory.high documentation to say when tasks are throttled. Suggested-by: Tejun Heo Signed-off-by: Shakeel Butt --- Documentation/admin-guide/cgroup-v2.rst | 12 ++++++++++-- mm/memcontrol.c | 20 ++++++-------------- 2 files changed, 16 insertions(+), 16 deletions(-) diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst index 9e3c67b02ae2..215710a44a88 100644 --- a/Documentation/admin-guide/cgroup-v2.rst +++ b/Documentation/admin-guide/cgroup-v2.rst @@ -1386,6 +1386,14 @@ write "max"; to set it to zero explicitly, write "0". over the high boundary, the processes of the cgroup are throttled and put under heavy reclaim pressure. + A task is throttled when it returns to userspace after + charging, and at some points in long operations, such as + populating memory or large buffered reads and writes. A + system call that allocates a lot without reaching such a + point can take the usage over the high boundary by up to + the amount it allocates. Kernel threads are not throttled: + a worker reclaims for them instead. + Going over the high limit never invokes the OOM killer and under extreme conditions the limit may be breached. The high limit should be used in scenarios where an external process @@ -1396,8 +1404,8 @@ write "max"; to set it to zero explicitly, write "0". reclaim is bypassed. This is useful for admin processes that need to dynamically adjust the job's memory limits without expending their own CPU resources on memory reclamation. The - job will trigger the reclaim and/or get throttled on its - next charge request. + job will reclaim and/or get throttled after it charges + memory again. Please note that with O_NONBLOCK, there is a chance that the target memory cgroup may take indefinite amount of time to diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 6d147153bd06..4771730e017a 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2615,8 +2615,9 @@ static unsigned long calculate_high_delay(unsigned int nr_pages, } /* - * Handle pending high overage during a blocking charge or on return - * to userspace. + * Handle pending high overage on return to userspace, or at a point in a + * long-running operation where the task can sleep (see + * mem_cgroup_over_high_throttle()). */ void __mem_cgroup_handle_over_high(gfp_t gfp_mask) { @@ -2725,9 +2726,9 @@ static void memcg_enforce_high(struct mem_cgroup *memcg, unsigned int nr_pages, !current->mm || !mm_match_cgroup(current->mm, memcg); /* - * Check the charged cgroup and its parents. Local charges normally defer - * reclaim until return to userspace so it can use GFP_KERNEL. Large - * pending overages may be handled synchronously below. The handler looks + * Check the charged cgroup and its parents. Local charges defer reclaim + * until return to userspace, or until a long-running operation reaches + * a point where it can sleep and calls the handler. The handler looks * up current->mm and rechecks the limits. If the task moved to another * cgroup in the meantime, the handler works on the new cgroup instead. * That is rare and costs little. @@ -2765,15 +2766,6 @@ static void memcg_enforce_high(struct mem_cgroup *memcg, unsigned int nr_pages, current->memcg_nr_pages_over_high += nr_pages; set_notify_resume(current); } - - /* - * Earlier charges may have left high overage pending even if this charge - * found no new overage. Handle it now if this charge can block, to limit - * overrun while the task remains in the kernel. - */ - if (current->memcg_nr_pages_over_high > MEMCG_CHARGE_BATCH && - gfpflags_allow_blocking(gfp_mask)) - __mem_cgroup_handle_over_high(gfp_mask); } static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask, -- 2.53.1-Meta