A folio allocated by an RWF_DONTCACHE write is marked dropbehind before it is dirtied, so folio_account_dirtied() adds it to WB_DONTCACHE_DIRTY and folio_clear_dirty_for_io() or folio_account_cleaned() subtracts it when it is cleaned. Nothing marks a folio that already exists. A caller that sets PG_dropbehind on a folio that is already dirty unbalances the counter: the clean subtracts what was never added, and once the counter goes negative, wb_stat_sum() reads it as zero and the DONTCACHE writeback kick stops writing anything back for every writer on that bdi_writeback. Add folio_mark_dropbehind(), the inverse of the clear in __filemap_get_folio_mpol(): with the folio locked, mark it and, if it has been counted as dirty, add it to WB_DONTCACHE_DIRTY. PG_dirty is not the test for that: filemap_dirty_folio() sets it before folio_account_dirtied() counts the folio, under the i_pages lock and without the folio lock, so a folio marked between the two would be counted twice. folio_account_dirtied() sets the folio's dirty tag in the same i_pages section that counts it, and every path that uncounts a dirty folio holds the folio lock, so set the flag and test the dirty tag under the i_pages lock: a dirtier not yet counting the folio finds the flag set and counts it itself. Under that lock the wb is inode_to_wb(), as in folio_account_dirtied(). A concurrent lookup without FGP_DONTCACHE that clears the flag again subtracts what this added. The name follows folio_set_dirty() and folio_mark_dirty(): the first only sets PG_dirty, the second also accounts the dirty folio. folio_set_dropbehind() likewise only sets PG_dropbehind, and folio_mark_dropbehind() also accounts it in WB_DONTCACHE_DIRTY. A later commit uses it in NFSD to drop a page shared by two misaligned WRITEs once both have written it. Signed-off-by: Mike Snitzer --- include/linux/pagemap.h | 1 + mm/filemap.c | 39 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 40 insertions(+) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 0adfa6605653..3012bd95812a 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -1310,6 +1310,7 @@ void end_page_writeback(struct page *page); void folio_end_writeback(struct folio *folio); void folio_end_writeback_no_dropbehind(struct folio *folio); void folio_end_dropbehind(struct folio *folio); +void folio_mark_dropbehind(struct folio *folio); void folio_wait_stable(struct folio *folio); void __folio_mark_dirty(struct folio *folio, struct address_space *, int warn); void folio_account_cleaned(struct folio *folio, struct bdi_writeback *wb); diff --git a/mm/filemap.c b/mm/filemap.c index 1cc28224d547..0c76ee5750b3 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1644,6 +1644,45 @@ void folio_end_dropbehind(struct folio *folio) } EXPORT_SYMBOL_GPL(folio_end_dropbehind); +/** + * folio_mark_dropbehind - mark an existing folio dropbehind + * @folio: the folio, locked by the caller + * + * Mark @folio so that the writeback that cleans it drops it. A folio + * already counted as dirty is added to WB_DONTCACHE_DIRTY, as + * folio_account_dirtied() would have added it had it been marked when + * dirtied, so that the clean that subtracts it leaves the counter balanced. + * + * folio_account_dirtied() counts a folio under the i_pages lock, in the + * section that sets its dirty tag, after PG_dirty has been set without the + * folio lock, and every path that uncounts a dirty folio holds the folio + * lock. So with the folio locked and the i_pages lock held, the dirty tag, + * not PG_dirty, says whether the folio has been counted: a dirtier past + * folio_test_set_dirty() but not yet counting finds the flag set and adds + * the folio itself. + */ +void folio_mark_dropbehind(struct folio *folio) +{ + struct address_space *mapping = folio->mapping; + unsigned long flags; + + VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio); + + if (folio_test_dropbehind(folio)) + return; + if (!mapping || !mapping_can_writeback(mapping)) { + folio_set_dropbehind(folio); + return; + } + xa_lock_irqsave(&mapping->i_pages, flags); + folio_set_dropbehind(folio); + if (xa_get_mark(&mapping->i_pages, folio->index, PAGECACHE_TAG_DIRTY)) + wb_stat_mod(inode_to_wb(mapping->host), WB_DONTCACHE_DIRTY, + folio_nr_pages(folio)); + xa_unlock_irqrestore(&mapping->i_pages, flags); +} +EXPORT_SYMBOL_GPL(folio_mark_dropbehind); + /** * folio_end_writeback_no_dropbehind - End writeback against a folio. * @folio: The folio. -- 2.52.0