follow_page_pte() now walks a whole page table in one call, but still resolves and commits each PTE on its own, so a PTE-mapped large folio (mTHP) pays one try_grab_folio() per subpage. Add gup_pte_batch_len() and give gup_grab_folio_pages() a run length. Each run starts with a full per-PTE resolve, then gup_pte_batch_len() finds how many more PTEs extend it with a pte_same() scan rather than a re-derivation of each page, and the run is committed with one refcount grab. Runs stop at folio boundaries, so plain base pages are unaffected. Gather the dirty bits from all the PTEs in a batch, in order to mark the folio dirty for FOLL_WRITE. Same benchmark as the previous change, with before again taken at the base of the series: gup_test -L -m 256 -n 65536 -r 16 -t before after 64 kB mTHP 2946 us 231 us (12.8x) 4 kB base (control) 2753 us 1198 us (2.3x) 2 MB THP (control) 72 us 71 us The 4 kB column is the previous change's 2.3x, unchanged: base pages share no folio, so there is nothing to batch. 64 kB mTHP goes from that change's 2.2x to 12.8x, a further 5.9x from committing a run with one refcount grab instead of one per subpage. With FOLL_TOUCH, folio_mark_accessed() runs once per run instead of once per page: 512 calls before and 32 after over 2 MB of 64 kB folios, leaving them referenced but inactive after the first call. That matches the huge-page and fast paths, which mark once per folio. (Measured by Yuan-Hao Hsu.) Suggested-by: David Hildenbrand Signed-off-by: Rik van Riel --- mm/gup.c | 49 ++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 46 insertions(+), 3 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 67e20b326be6e..47f204e21a3a7 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -937,6 +937,35 @@ static long gup_lookup_pte(struct vm_area_struct *vma, unsigned long address, return 0; } +/* + * Return how many PTEs map consecutive pages of the same folio and can be + * committed as one run. Always at least 1. + * + * The write-fault and unshare checks in gup_lookup_pte() are per PTE, but a + * writable run needs no repeat: a writable anon page is exclusive. A read-only + * run under FOLL_WRITE or FOLL_PIN does need the per-page check, so it stays + * one page at a time. + */ +static unsigned long gup_pte_batch_len(struct vm_area_struct *vma, + unsigned long address, unsigned long walk_end, + struct folio *folio, pte_t *ptep, pte_t *batch_pte, unsigned int flags) +{ + unsigned long max; + + if (!folio_test_large(folio)) + return 1; + if (!pte_write(*batch_pte) && (flags & (FOLL_WRITE | FOLL_PIN))) + return 1; + + max = (walk_end - address) >> PAGE_SHIFT; + if (max <= 1) + return 1; + + /* Merge young/dirty across batch so folio_mark_dirty sees any dirty. */ + return folio_pte_batch_flags(folio, vma, ptep, batch_pte, max, + FPB_RESPECT_WRITE | FPB_MERGE_YOUNG_DIRTY); +} + /* * Walk the PTEs from the start address to the end of this page table or VMA, * whichever comes first, and commit every page found. @@ -968,7 +997,10 @@ static long follow_page_pte(struct vm_area_struct *vma, for (; address < walk_end; address += PAGE_SIZE, ptep++) { pte_t pte = ptep_get(ptep); + struct folio *folio; struct page *page; + unsigned long batch; + pte_t batch_pte; ret = gup_lookup_pte(vma, address, ptep, pte, flags, &page); if (ret || !page) { @@ -983,12 +1015,23 @@ static long follow_page_pte(struct vm_area_struct *vma, need_no_page_table = true; break; } - ret = gup_grab_folio_pages(page_folio(page), page, - 1, pte_dirty(pte), flags, + folio = page_folio(page); + batch_pte = pte; + + batch = gup_pte_batch_len(vma, address, walk_end, folio, + ptep, &batch_pte, flags); + ret = gup_grab_folio_pages(folio, page, batch, + pte_dirty(batch_pte), flags, pages ? pages + nr : NULL); if (ret) break; - nr++; + nr += batch; + /* + * The loop's own increment covers one PTE; skip + * the rest of the batch. + */ + ptep += batch - 1; + address += (batch - 1) * PAGE_SIZE; } pte_unmap_unlock(orig_ptep, ptl); -- 2.53.0-Meta