On a VM_HUGEPAGE file mapping, a synchronous fault reads the PMD-sized folio that holds the faulting address and the one above it, and maps the faulting folio with a PMD. do_sync_mmap_readahead() counts each such fault in mmap_miss and stops reading ahead once it passes MMAP_LOTSAMISS, while filemap_map_pages() lowers the count for faults served from the page cache, but only when it maps them with PTEs. A reader that moves towards lower offsets, such as memmove() copying backwards out of a mapping, never finds its next folio cached: the folio above is the one it just read. Every folio is a miss and none is a hit, so after about a hundred folios mmap_miss passes the limit and every page after that is read on its own. Since the PTEs mapped around a fault no longer count as hits, nothing lowers the count for it. When the folio above the faulting one is cached and the one below is not, read the folio below instead, without a readahead mark, and count a PMD mapping of the faulting address as a hit the way a PTE mapping is. Such a reader then misses on every other folio and hits on the rest. Reading one byte from each page of a 2 GiB file, last page first, in a MADV_HUGEPAGE mapping with a cold page cache, read_ahead_kb 8192: before after time 42.83 s 0.57 s major faults 473189 514 Fixes: 0b9c0aeba938 ("mm/filemap: count only the faulting address as a mmap hit") Cc: stable@vger.kernel.org # 7.2.x: needs the previous patch Signed-off-by: Qiliang Yuan --- mm/filemap.c | 48 +++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 41 insertions(+), 7 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index c38f0e4cf6123..e445279cc561f 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3336,6 +3336,27 @@ static int lock_folio_maybe_drop_mmap(struct vm_fault *vmf, struct folio *folio, return 1; } +/* + * Whether a miss at @index, right below @nr cached pages and right above + * @nr uncached ones, comes from a stream that runs towards lower offsets. + */ +static bool filemap_descending(struct address_space *mapping, pgoff_t index, + unsigned long nr) +{ + void *entry; + + if (index < nr) + return false; + rcu_read_lock(); + entry = xa_load(&mapping->i_pages, index + nr); + rcu_read_unlock(); + if (!entry || xa_is_value(entry)) + return false; + return !filemap_range_has_page(mapping, + (loff_t)(index - nr) << PAGE_SHIFT, + ((loff_t)index << PAGE_SHIFT) - 1); +} + /* * Synchronous readahead happens when we don't even find a page in the page * cache at all. We don't want to perform IO under the mmap sem, so if we have @@ -3409,13 +3430,20 @@ static struct file *do_sync_mmap_readahead(struct vm_fault *vmf) fpin = maybe_unlock_mmap_for_io(vmf, fpin); ractl._index &= ~(folio_nr_pages - 1); ra->size = folio_nr_pages; + ra->async_size = folio_nr_pages; /* * Fetch two folios so we get the chance to actually - * readahead, unless we've been told not to. + * readahead, unless we've been told not to. A stream that + * runs towards lower offsets gets the folio below. */ - if (!(vm_flags & VM_RAND_READ)) + if (!(vm_flags & VM_RAND_READ)) { ra->size *= 2; - ra->async_size = folio_nr_pages; + if (filemap_descending(mapping, ractl._index, + folio_nr_pages)) { + ractl._index -= folio_nr_pages; + ra->async_size = 0; + } + } ra->order = thp_order; page_cache_ra_order(&ractl, ra); return fpin; @@ -3981,10 +4009,16 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, * Make an exception for shmem/tmpfs that for long time * intentionally mapped with PMDs across i_size. */ - if ((file_end >= folio_next_index(folio) || shmem_mapping(mapping)) && - filemap_map_pmd(vmf, folio, start_pgoff)) { - ret = VM_FAULT_NOPAGE; - goto out; + if (file_end >= folio_next_index(folio) || shmem_mapping(mapping)) { + /* Check before filemap_map_pmd(), which may drop the last ref */ + bool hit = filemap_mmap_hit(vmf, folio); + + if (filemap_map_pmd(vmf, folio, start_pgoff)) { + if (hit) + filemap_count_mmap_hit(file); + ret = VM_FAULT_NOPAGE; + goto out; + } } addr = vma->vm_start + -- 2.43.0