mlock(), mlockall(), mmap() with MAP_POPULATE or MAP_LOCKED, and MADV_POPULATE_READ/WRITE can fault in a lot of memory in one call, with mmap_lock held. When their charges can block, the charge path throttles them for memory.high with mmap_lock still held. In a VM with THP on, mlock() of an 80M MADV_HUGEPAGE range under memory.high=64M made another thread wait up to 17.2s to mmap(), and a task in another cgroup wait up to 15.3s to read the process's /proc//cmdline. THP charges in VMAs without MADV_HUGEPAGE can't block under the default defrag setting, so they are not throttled at all: mlock(512M) reached 513M. Throttle these calls with mmap_lock dropped instead. The populate loops, __mm_populate() and madvise_populate(), now fault in at most 2MiB per GUP call. Between calls, if more than a batch of overage is pending, they drop mmap_lock, handle it, and go on. GUP itself doesn't change. One fault can still charge a whole folio. With this patch, mlock(512M) stays at 84M, and populating 2G with no limit takes as long as before. The mmap_lock stall goes away when a later patch removes the synchronous step. Signed-off-by: Shakeel Butt --- mm/gup.c | 9 +++++++-- mm/internal.h | 27 +++++++++++++++++++++++++++ mm/madvise.c | 11 +++++++++-- 3 files changed, 43 insertions(+), 4 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 7605da0e06d8..d93d89f6aad0 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1951,6 +1951,7 @@ int __mm_populate(unsigned long start, unsigned long len, int ignore_errors) end = start + len; for (nstart = start; nstart < end; nstart = nend) { + populate_handle_over_high(mm, &locked); /* * We want to fault in pages for [nstart; end) address range. * Find first corresponding VMA. @@ -1976,9 +1977,13 @@ int __mm_populate(unsigned long start, unsigned long len, int ignore_errors) /* * Now fault in a range of pages. populate_vma_page_range() * double checks the vma flags, so that it won't mlock pages - * if the vma was already munlocked. + * if the vma was already munlocked. Do one chunk at a time, + * see POPULATE_CHUNK_SIZE. On error, nend skips the rest of + * the VMA. */ - ret = populate_vma_page_range(vma, nstart, nend, &locked); + ret = populate_vma_page_range(vma, nstart, + populate_chunk_end(nstart, nend), + &locked); if (ret < 0) { if (ignore_errors) { ret = 0; diff --git a/mm/internal.h b/mm/internal.h index de4593b3e3b6..12f7b4b1254f 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -948,6 +948,33 @@ extern long populate_vma_page_range(struct vm_area_struct *vma, unsigned long start, unsigned long end, int *locked); extern long faultin_page_range(struct mm_struct *mm, unsigned long start, unsigned long end, bool write, int *locked); + +/* + * __mm_populate() and madvise_populate() fault in at most + * POPULATE_CHUNK_SIZE at a time, and handle memory.high overage between + * chunks with mmap_lock dropped, so other tasks don't wait for them. + */ +#define POPULATE_CHUNK_SIZE SZ_2M + +static inline unsigned long populate_chunk_end(unsigned long start, + unsigned long end) +{ + return min(end, round_down(start, POPULATE_CHUNK_SIZE) + + POPULATE_CHUNK_SIZE); +} + +static inline void populate_handle_over_high(struct mm_struct *mm, + int *locked) +{ + if (!mem_cgroup_over_high_pending()) + return; + if (*locked) { + mmap_read_unlock(mm); + *locked = 0; + } + mem_cgroup_handle_over_high(GFP_KERNEL); +} + bool mlock_future_ok(const struct mm_struct *mm, bool is_vma_locked, unsigned long bytes); diff --git a/mm/madvise.c b/mm/madvise.c index 2d54cb9863ea..c77924e52d79 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -992,8 +992,15 @@ static long madvise_populate(struct madvise_behavior *madv_behavior) long pages; while (start < end) { - /* Populate (prefault) page tables readable/writable. */ - pages = faultin_page_range(mm, start, end, write, &locked); + /* + * Populate (prefault) page tables readable/writable, at most a + * chunk at a time, see POPULATE_CHUNK_SIZE. + */ + pages = faultin_page_range(mm, start, + populate_chunk_end(start, end), + write, &locked); + if (pages >= 0) + populate_handle_over_high(mm, &locked); if (!locked) { mmap_read_lock(mm); locked = 1; -- 2.53.1-Meta