The kpkeys_hardened_pgtables feature automatically protect early vmemmap page tables (covering standard boot memory) when it is enabled. In order to protect late vmemmap page tables as well, allocate them like regular page tables using pagetable_alloc(). pagetable_alloc() does the right thing to protect the backing pages using kpkeys, but isn't NUMA-aware, so the pages may be allocated on any node. This could potentially incur some overhead on large NUMA systems that make heavy use of memory hotplug. The arm64 hotplug code is also amended to use a matching pagetable_free(), ensuring that the pkey is reset when the page tables are freed. x86 already uses pagetable_free() on that path. Unlike in vmemmap_alloc_block(), __GFP_RETRY_MAYFAIL is not used as it isn't justified for allocating page tables - this disables the OOM and we do not have a fallback if we fail to allocate page tables. See previous discussion linked below. Link: https://lore.kernel.org/all/38d2a358-4146-bfc9-2a4f-68ce02f75c94@suse.cz/ Signed-off-by: Kevin Brodsky --- This is a minimal patch to protect vmemmmap page tables. More work may be needed here: * Restoring NUMA awareness * Moving the arm64 change to a separate commit? * General refactoring of how these page tables are allocated: since we are not using the standard per-level functions (e.g. pmd_alloc()), we are not calling pagetable_*_ctor() or ptdesc_set_kernel(). I sent a separate series [1] that partially addresses this, but more work is still needed. * x86 allows boot memory to be hot-removed in some circumstances. free_pagetable() should handle this situation by resetting the direct map pkey of reserved pages. [1] https://lore.kernel.org/all/20260714-remove_pgtable_cdtor-v1-0-44be8a7685d7@arm.com/ --- arch/arm64/mm/mmu.c | 2 +- mm/sparse-vmemmap.c | 23 ++++++++++++++++------- 2 files changed, 17 insertions(+), 8 deletions(-) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 6a2155db3d05..a61ea65bd496 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -1496,7 +1496,7 @@ static void free_hotplug_page_range(struct page *page, size_t size, static void free_hotplug_pgtable_page(struct page *page) { pagetable_dtor(page_ptdesc(page)); - free_hotplug_page_range(page, PAGE_SIZE, NULL); + pagetable_free(page_ptdesc(page)); } static bool pgtable_range_aligned(unsigned long start, unsigned long end, diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 5d7e982f94d0..c47d3a016891 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -150,13 +150,22 @@ int __meminit section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages return 0; } -static void * __meminit vmemmap_alloc_block_zero(unsigned long size, int node) +static void * __meminit vmemmap_alloc_pgtable(int node) { - void *p = vmemmap_alloc_block(size, node); + void *p; + + if (slab_is_available()) { + gfp_t gfp = GFP_KERNEL | __GFP_ZERO; + struct ptdesc *ptdesc = pagetable_alloc(gfp, 0); + + return ptdesc ? ptdesc_address(ptdesc) : NULL; + } + + p = vmemmap_alloc_block(PAGE_SIZE, node); if (!p) return NULL; - memset(p, 0, size); + memset(p, 0, PAGE_SIZE); return p; } @@ -305,7 +314,7 @@ static pmd_t * __meminit vmemmap_pmd_populate(pud_t *pud, unsigned long addr, in { pmd_t *pmd = pmd_offset(pud, addr); if (pmd_none(*pmd)) { - void *p = vmemmap_alloc_block_zero(PAGE_SIZE, node); + void *p = vmemmap_alloc_pgtable(node); if (!p) return NULL; kernel_pte_init(p); @@ -318,7 +327,7 @@ static pud_t * __meminit vmemmap_pud_populate(p4d_t *p4d, unsigned long addr, in { pud_t *pud = pud_offset(p4d, addr); if (pud_none(*pud)) { - void *p = vmemmap_alloc_block_zero(PAGE_SIZE, node); + void *p = vmemmap_alloc_pgtable(node); if (!p) return NULL; pmd_init(p); @@ -331,7 +340,7 @@ static p4d_t * __meminit vmemmap_p4d_populate(pgd_t *pgd, unsigned long addr, in { p4d_t *p4d = p4d_offset(pgd, addr); if (p4d_none(*p4d)) { - void *p = vmemmap_alloc_block_zero(PAGE_SIZE, node); + void *p = vmemmap_alloc_pgtable(node); if (!p) return NULL; pud_init(p); @@ -344,7 +353,7 @@ static pgd_t * __meminit vmemmap_pgd_populate(unsigned long addr, int node) { pgd_t *pgd = pgd_offset_k(addr); if (pgd_none(*pgd)) { - void *p = vmemmap_alloc_block_zero(PAGE_SIZE, node); + void *p = vmemmap_alloc_pgtable(node); if (!p) return NULL; pgd_populate_kernel(addr, pgd, p); -- 2.56.0