truncate_inode_pages_range() takes the folios it finds out of the page cache with delete_from_page_cache_batch(), which takes the inode's i_lock and the i_pages lock once for each batch of up to FOLIO_BATCH_SIZE folios. shmem_undo_range() calls truncate_inode_folio() on each folio instead, so filemap_remove_folio() takes and drops both locks once per folio. Truncating, punching or freeing a tmpfs file or memfd made of small folios, such as a large /dev/shm segment, pays for that on every page. Clean each folio up as truncate_inode_folio() does, collect the ones to remove and take them out with delete_from_page_cache_batch(), which needs truncate_cleanup_folio() outside mm/truncate.c. Swap entries, and the folios kept when undoing a failed fallocate(), stay out of the batch. Dropping an 8 GiB file of 4K folios, median of five runs, averaged over two boots of each kernel, in a 32 vCPU VM: before after ftruncate() to 0 1163 ms 913 ms hole punch 1155 ms 914 ms close() of a memfd 1161 ms 914 ms unlink() 1162 ms 916 ms Files of PMD-sized folios take 20-25 ms with or without this change. Signed-off-by: Qiliang Yuan --- To see it, fill and remove a large /dev/shm file: fallocate -l 8G /dev/shm/f && time rm /dev/shm/f --- mm/internal.h | 1 + mm/shmem.c | 16 +++++++++++++--- mm/truncate.c | 2 +- 3 files changed, 15 insertions(+), 4 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 566cbd2dce485..abe64e49a4170 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -628,6 +628,7 @@ unsigned find_lock_entries(struct address_space *mapping, pgoff_t *start, pgoff_t end, struct folio_batch *fbatch, pgoff_t *indices); unsigned find_get_entries(struct address_space *mapping, pgoff_t *start, pgoff_t end, struct folio_batch *fbatch, pgoff_t *indices); +void truncate_cleanup_folio(struct folio *folio); int truncate_inode_folio(struct address_space *mapping, struct folio *folio); bool truncate_inode_partial_folio(struct folio *folio, loff_t lstart, loff_t lend, pgoff_t *pstart, pgoff_t *pend); diff --git a/mm/shmem.c b/mm/shmem.c index af21a89abb0f6..16b6e38503bc3 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1352,6 +1352,9 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, uoff_t lend, index = start; while (index < end && find_lock_entries(mapping, &index, end - 1, &fbatch, indices)) { + struct folio_batch remove; + + folio_batch_init(&remove); for (i = 0; i < folio_batch_count(&fbatch); i++) { folio = fbatch.folios[i]; @@ -1363,9 +1366,16 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, uoff_t lend, continue; } - if (!unfalloc || !folio_test_uptodate(folio)) - truncate_inode_folio(mapping, folio); - folio_unlock(folio); + if (!unfalloc || !folio_test_uptodate(folio)) { + truncate_cleanup_folio(folio); + folio_batch_add(&remove, folio); + } + } + delete_from_page_cache_batch(mapping, &remove); + for (i = 0; i < folio_batch_count(&fbatch); i++) { + folio = fbatch.folios[i]; + if (!xa_is_value(folio)) + folio_unlock(folio); } folio_batch_remove_exceptionals(&fbatch); folio_batch_release(&fbatch); diff --git a/mm/truncate.c b/mm/truncate.c index 733d2713a875a..c6bf4aa0b91e8 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -151,7 +151,7 @@ EXPORT_SYMBOL_GPL(folio_invalidate); * its lock, b) when a concurrent invalidate_mapping_pages got there first and * c) when tmpfs swizzles a page between a tmpfs inode and swapper_space. */ -static void truncate_cleanup_folio(struct folio *folio) +void truncate_cleanup_folio(struct folio *folio) { if (folio_mapped(folio)) unmap_mapping_folio(folio); --- base-commit: d89e1d186ec9271bf78e62771779e9fc03bf0fc3 change-id: 20261029-bug-mm-shmem-undo-batch-de001d9c6f1e Best regards, -- Qiliang Yuan