x86 and arm64 invoke ptdump_walk_pgd() with non-init_mm mm whilst still walking kernel page table ranges. For x86 this is done in ptdump_curknl_show() and ptdump_efi_show(), the first passing current->mm, and the second passing efi_mm (we reach kernel mappings that init_mm protects for current->mm due to x86 cloning shared kernel page tables for arbitrary mm's). arm64 does so via ptdump_debugfs_register(), configured by efi_ptdump_info for efi ranges against efi_mm. The init_mm mmap lock is used to stabilise page table freeing against ptdump, so take a nested lock on init_mm to ensure that we are correctly stabilised. We take this after mmap write locking the non-init_mm mm. Nothing acquires the init_mm lock first before locking an arbitrary mm, so no deadlock is possible. The preceding patches in this series updated the two cases which can cause races with ptdump in init_mm ranges - vmap and x86 CPA huge page promotion. For arm64, commit fa93b45fd397 ("arm64: Enable vmalloc-huge with ptdump") already provides exclusion against init_mm for the vmap case, which this patch also pairs with. The first point at which ptdump can race kernel page table freeing is commit b6bdb7517c3d ("mm/vmalloc: add interfaces to free unmapped page table"), so we target this in the Fixes tag. Also update walk_page_range_debug() to assert that init_mm is write locked, add a comment explaining why and remove some redundant code. We do not need to check for walk.mm being non-NULL, as the mmap lock asserts would NULL pointer deref if it was (and of course no callers do this). The rest of this code is exactly as it would be if invoking walk_kernel_page_table_range(), only now it's far clearer that that's the case. Fixes: b6bdb7517c3d ("mm/vmalloc: add interfaces to free unmapped page table") Cc: stable@vger.kernel.org Reviewed-by: Mike Rapoport (Microsoft) Signed-off-by: Lorenzo Stoakes --- mm/pagewalk.c | 14 +++++++++----- mm/ptdump.c | 7 +++++++ 2 files changed, 16 insertions(+), 5 deletions(-) diff --git a/mm/pagewalk.c b/mm/pagewalk.c index bbcfd68d0907..5d87c632a255 100644 --- a/mm/pagewalk.c +++ b/mm/pagewalk.c @@ -702,12 +702,16 @@ int walk_page_range_debug(struct mm_struct *mm, unsigned long start, * to account for page table freeing on vmap huge page mapping. */ mmap_assert_write_locked(mm); + /* + * x86, arm64 ptdump allow walks of efi mm's and x86 ptdump allows walks + * of arbitrary mm's. + * + * However, they both must also hold the init_mm lock to account for + * concurrent kernel page table freeing. + */ + mmap_assert_write_locked(&init_mm); - /* For convenience, we allow traversal of kernel mappings. */ - if (mm == &init_mm) - return walk_kernel_page_table_range(start, end, ops, - pgd, private); - if (start >= end || !walk.mm) + if (start >= end) return -EINVAL; if (!check_ops_safe(ops)) return -EINVAL; diff --git a/mm/ptdump.c b/mm/ptdump.c index 973020000096..5851096e6f65 100644 --- a/mm/ptdump.c +++ b/mm/ptdump.c @@ -178,11 +178,18 @@ void ptdump_walk_pgd(struct ptdump_state *st, struct mm_struct *mm, pgd_t *pgd) get_online_mems(); mmap_write_lock(mm); + /* To stabilise kernel page tables we must hold the init_mm lock too. */ + if (mm != &init_mm) + mmap_write_lock_nested(&init_mm, SINGLE_DEPTH_NESTING); + while (range->start != range->end) { walk_page_range_debug(mm, range->start, range->end, &ptdump_ops, pgd, st); range++; } + + if (mm != &init_mm) + mmap_write_unlock(&init_mm); mmap_write_unlock(mm); put_online_mems(); -- 2.55.0