__vm_anon_pgoff_hi sits right after vm_refcnt in the vm_refcnt cacheline. Since commit 93c0c8dc87f6 ("mm/rmap: use anon pgoff to track MAP_PRIVATE file-backed anon folios"), which made anon rmap use it, __folio_set_anon() reads it on every new anonymous folio, i.e. once per fault that maps a new anonymous page, between vma_start_read() and vma_end_read(). When several threads fault on the same VMA, the other CPUs keep writing vm_refcnt from vma_start_read()/vma_end_read(), so this read often misses and pulls the line over. The extra load can force a coherence transfer and makes the following vm_refcnt updates more expensive. For anonymous VMAs the anonymous page offset is always equal to vm_pgoff (linear_anon_page_index() already asserts this under CONFIG_DEBUG_VM; it holds for MAP_PRIVATE /dev/zero too since commit e59c9a971a9f ("mm/vma: make MAP_PRIVATE-mapped /dev/zero mappings truly anonymous")), so compute the index from vm_pgoff as before. That brings back the pre-series behaviour for the hot path; MAP_PRIVATE file-backed VMAs still read the split field. Keep the assertion, the other way round. Moving __vm_anon_pgoff_hi out of the line does not work without growing vm_area_struct. With e.g. CONFIG_ANON_VMA_NAME=y the struct is exactly 192 bytes, cachelines 0 and 1 have no hole, and the only 4-byte fields that could fill the slot next to vm_refcnt are vm_lock_seq and the lo/hi halves. vm_lock_seq is read before vm_refcnt is taken in vma_start_read(), so it would just move the miss. Measured on an AMD EPYC 9634 (84 cores, 12 CCDs, CONFIG_DEBUG_VM=n), on mainline after v7.4-rc1 (the base of this series). Threads share one anonymous VMA, each faults its own slice and zaps it with MADV_DONTNEED; faults/s, median of 3: CPUs base patched 1 CCD, 7 threads 2.360M 2.369M +0.4% 7 CCDs, 1 thread each 1.738M 1.982M +14.0% 12 CCDs, 1 thread each 2.288M 2.548M +11.4% On 7.3.1 (with the 7.3 variant of this change) the same runs gave +1.9%, +16.0% and +29.4%. There, perf c2c shows the folio_add_new_anon_rmap() load at offset 132 disappearing from the vm_refcnt line and the average load latency of the vm_refcnt inc/dec on that line dropping by about 45%. With one VMA per thread, or with processes, this patch alone changes throughput by -2.0% to -0.1% on that base (less than 1% on 7.3.1); the next patch more than recovers that. will-it-scale page_fault1 threads does not show the difference on this machine: each thread mmap()s and munmap()s its own region, which merges VMAs and is dominated by VMA write locking, and its results vary by up to 10% between boots of the same kernel. This relies on the /dev/zero commit above, which is not in v7.3, so this is not tagged for stable: without it, MAP_PRIVATE /dev/zero VMAs are anonymous but keep the file offset in vm_pgoff, and must keep using the anon pgoff field. The tools/testing/vma copy of the helper is updated to match. Signed-off-by: Seunghyun An --- include/linux/pagemap.h | 19 ++++++++++++++----- tools/testing/vma/include/dup.h | 13 ++++++++----- 2 files changed, 22 insertions(+), 10 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 73af18a373..300ee83fc1 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -1122,13 +1122,22 @@ static inline pgoff_t __linear_anon_page_index(const struct vm_area_struct *vma, static inline pgoff_t linear_anon_page_index(const struct vm_area_struct *vma, const unsigned long address) { - const pgoff_t pgoff = __linear_anon_page_index(vma, address); - VM_WARN_ON_ONCE(!vma_is_cow_mapping(vma)); - if (vma_is_anonymous(vma)) - VM_WARN_ON_ONCE(pgoff != linear_page_index(vma, address)); - return pgoff; + /* + * For anonymous VMAs the anonymous page offset always equals vm_pgoff, + * so use it. This keeps the anonymous fault path (__folio_set_anon()) + * off __vm_anon_pgoff_hi, which shares a cacheline with vm_refcnt and + * is therefore contended by concurrent faults on the same VMA. + */ + if (vma_is_anonymous(vma)) { + const pgoff_t pgoff = linear_page_index(vma, address); + + VM_WARN_ON_ONCE(pgoff != __linear_anon_page_index(vma, address)); + return pgoff; + } + + return __linear_anon_page_index(vma, address); } struct wait_page_key { diff --git a/tools/testing/vma/include/dup.h b/tools/testing/vma/include/dup.h index dc4927d262..f3d69b0c74 100644 --- a/tools/testing/vma/include/dup.h +++ b/tools/testing/vma/include/dup.h @@ -1655,13 +1655,16 @@ static inline pgoff_t __linear_anon_page_index(const struct vm_area_struct *vma, static inline pgoff_t linear_anon_page_index(const struct vm_area_struct *vma, const unsigned long address) { - const pgoff_t pgoff = __linear_anon_page_index(vma, address); - VM_WARN_ON_ONCE(!vma_is_cow_mapping(vma)); - if (vma_is_anonymous(vma)) - VM_WARN_ON_ONCE(pgoff != linear_page_index(vma, address)); - return pgoff; + if (vma_is_anonymous(vma)) { + const pgoff_t pgoff = linear_page_index(vma, address); + + VM_WARN_ON_ONCE(pgoff != __linear_anon_page_index(vma, address)); + return pgoff; + } + + return __linear_anon_page_index(vma, address); } extern const struct file_operations zero_fops; -- 2.53.0