drm_pagemap_migrate_populate_ram_pfn() had two issues when populating RAM PFNs with higher-order folios: 1. The higher-order vma_alloc_folio()/folio_alloc() calls did not pass __GFP_NOWARN, so a THP allocation failure under memory pressure would spam the kernel log, and there was no fallback path despite a TODO comment stating one was needed. Add __GFP_NOWARN to the higher-order allocation and, on failure, fall back to order-0 allocations for the entire range originally covered by the failed higher-order allocation, leaving MIGRATE_PFN_COMPOUND unset for those PFNs. 2. In the free_pages error path, order was computed via folio_order(page_folio(page)) *after* put_page(page) had already dropped the reference, resulting in a use-after-free/put when that was the last reference on the page. Compute order before releasing the page. Introducing the fallback in 1. also requires the source page array handed to ->copy_to_ram() to be built differently. Both callers only populated the entry at the head of each source folio, relying on the copy callback to derive the rest of the folio from the order recorded in the matching drm_pagemap_addr. Once the destination has been demoted to order-0 folios the drm_pagemap_addr entries are per-page, so a source page is needed for every one of them; leaving them NULL makes the copy callback stop after the first page and the remainder of the range is never copied. The source folio is only split later, by migrate_vma_pages() / migrate_device_pages(), so its order cannot be used to detect the demotion - test the destination for MIGRATE_PFN_COMPOUND instead. Factor the array population out into drm_pagemap_migrate_populate_src_pages() and use it from both drm_pagemap_evict_to_ram() and __drm_pagemap_migrate_to_ram(). Fixes: ddeda6136038 ("drm/pagemap: Allocate folios when possible") Cc: stable@vger.kernel.org Cc: Andrew Morton Cc: David Hildenbrand Cc: Lorenzo Stoakes Cc: Zi Yan Cc: Baolin Wang Cc: Liam R. Howlett Cc: Nico Pache Cc: Ryan Roberts Cc: Dev Jain Cc: Barry Song Cc: Lance Yang Cc: Usama Arif Cc: Joshua Hahn Cc: Rakie Kim Cc: Byungchul Park Cc: Gregory Price Cc: Ying Huang Cc: Alistair Popple Cc: Balbir Singh Cc: Maarten Lankhorst Cc: Maxime Ripard Cc: Thomas Zimmermann Cc: David Airlie Cc: Simona Vetter Cc: Thomas Hellström Cc: Francois Dugast Cc: dri-devel@lists.freedesktop.org Cc: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org Cc: stable@vger.kernel.org Assisted-by: GitHub Copilot:claude-opus-5 Signed-off-by: Matthew Brost --- drivers/gpu/drm/drm_pagemap.c | 114 ++++++++++++++++++++++++++-------- 1 file changed, 89 insertions(+), 25 deletions(-) diff --git a/drivers/gpu/drm/drm_pagemap.c b/drivers/gpu/drm/drm_pagemap.c index 892b325fa99b..7610e233d238 100644 --- a/drivers/gpu/drm/drm_pagemap.c +++ b/drivers/gpu/drm/drm_pagemap.c @@ -383,6 +383,58 @@ drm_pagemap_migrate_map_system_pages(struct device *dev, return 0; } +/** + * drm_pagemap_migrate_populate_src_pages() - Populate the source page array + * @pages: Array of source pages to populate + * @src_mpfn: Source array of migrate PFNs + * @dst_mpfn: Destination array of migrate PFNs + * @npages: Number of pages in the arrays + * + * Populate @pages with the device pages the copy callback is to read from. + * + * Entries are normally only populated at the head of each source folio, with + * the copy callback deriving the rest of the folio from the order recorded in + * the corresponding drm_pagemap_addr. That does not work where + * drm_pagemap_migrate_populate_ram_pfn() had to demote a higher-order source + * folio to order-0 destination folios: the drm_pagemap_addr entries are then + * per-page, and the copy callback needs a source page for each of them. + * Populate every entry for those ranges. + * + * Note that the source folio itself is only split later, by + * migrate_vma_pages() / migrate_device_pages(), so its order cannot be used to + * detect the demotion - the destination has to be inspected instead. + */ +static void drm_pagemap_migrate_populate_src_pages(struct page **pages, + unsigned long *src_mpfn, + unsigned long *dst_mpfn, + unsigned long npages) +{ + unsigned long i; + + for (i = 0; i < npages;) { + struct page *page = migrate_pfn_to_page(src_mpfn[i]); + unsigned int order = 0; + unsigned long j, nr; + + if (!page) { + i++; + continue; + } + + order = folio_order(page_folio(page)); + nr = NR_PAGES(order); + + if (order && !(dst_mpfn[i] & MIGRATE_PFN_COMPOUND)) { + for (j = 0; j < nr && i + j < npages; j++) + pages[i + j] = folio_page(page_folio(page), j); + } else { + pages[i] = page; + } + + i += nr; + } +} + /** * drm_pagemap_migrate_unmap_pages() - Unmap pages previously mapped for GPU SVM migration * @dev: The device for which the pages were mapped @@ -875,6 +927,7 @@ static int drm_pagemap_migrate_populate_ram_pfn(struct vm_area_struct *vas, struct page *page = NULL, *src_page; struct folio *folio; unsigned int order = 0; + gfp_t gfp = GFP_HIGHUSER; if (!(src_mpfn[i] & MIGRATE_PFN_MIGRATE)) goto next; @@ -890,12 +943,38 @@ static int drm_pagemap_migrate_populate_ram_pfn(struct vm_area_struct *vas, } order = folio_order(page_folio(src_page)); + if (order) + gfp |= __GFP_NOWARN; - /* TODO: Support fallback to single pages if THP allocation fails */ if (vas) - folio = vma_alloc_folio(GFP_HIGHUSER, order, vas, addr); + folio = vma_alloc_folio(gfp, order, vas, addr); else - folio = folio_alloc(GFP_HIGHUSER, order); + folio = folio_alloc(gfp, order); + + if (!folio && order) { + /* + * Higher-order allocation failed, fall back to + * order-0 allocations for the entire range covered + * by the original higher-order allocation, without + * setting MIGRATE_PFN_COMPOUND, until we move past + * that range. + */ + unsigned long nr = NR_PAGES(order); + unsigned long j; + + gfp &= ~__GFP_NOWARN; + for (j = 0; j < nr && i < npages; j++, i++, addr += PAGE_SIZE) { + folio = vas ? + vma_alloc_folio(gfp, 0, vas, addr) : + folio_alloc(gfp, 0); + if (!folio) + goto free_pages; + + page = folio_page(folio, 0); + mpfn[i] = migrate_pfn(page_to_pfn(page)); + } + continue; + } if (!folio) goto free_pages; @@ -940,11 +1019,11 @@ static int drm_pagemap_migrate_populate_ram_pfn(struct vm_area_struct *vas, if (!page) goto next_put; + order = folio_order(page_folio(page)); + put_page(page); mpfn[i] = 0; - order = folio_order(page_folio(page)); - next_put: i += NR_PAGES(order); } @@ -1120,7 +1199,7 @@ int drm_pagemap_evict_to_ram(struct drm_pagemap_devmem *devmem_allocation) unsigned long *src, *dst; struct drm_pagemap_addr *pagemap_addr; void *buf; - int i, err = 0; + int err = 0; unsigned int retry_count = 2; npages = devmem_allocation->size >> PAGE_SHIFT; @@ -1160,15 +1239,7 @@ int drm_pagemap_evict_to_ram(struct drm_pagemap_devmem *devmem_allocation) if (err) goto err_finalize; - for (i = 0; i < npages;) { - unsigned int order = 0; - - pages[i] = migrate_pfn_to_page(src[i]); - if (pages[i]) - order = folio_order(page_folio(pages[i])); - - i += NR_PAGES(order); - } + drm_pagemap_migrate_populate_src_pages(pages, src, dst, npages); err = ops->copy_to_ram(pages, pagemap_addr, npages, NULL); if (err) @@ -1235,7 +1306,7 @@ static int __drm_pagemap_migrate_to_ram(struct vm_area_struct *vas, struct drm_pagemap_addr *pagemap_addr; unsigned long start, end; void *buf; - int i, err = 0; + int err = 0; zdd = drm_pagemap_page_zone_device_data(page); if (time_before64(get_jiffies_64(), zdd->devmem_allocation->timeslice_expiration)) @@ -1290,15 +1361,8 @@ static int __drm_pagemap_migrate_to_ram(struct vm_area_struct *vas, if (err) goto err_finalize; - for (i = 0; i < npages;) { - unsigned int order = 0; - - pages[i] = migrate_pfn_to_page(migrate.src[i]); - if (pages[i]) - order = folio_order(page_folio(pages[i])); - - i += NR_PAGES(order); - } + drm_pagemap_migrate_populate_src_pages(pages, migrate.src, migrate.dst, + npages); err = ops->copy_to_ram(pages, pagemap_addr, npages, NULL); if (err) -- 2.34.1