AI Trajectory

Trajectory: (Expand/Collapse All)
Seq	Timestamp	Type	Name	Duration
0/0	2026/08/18 17:01	flow	patch-triage	7m Args: null Results: { "EnableConfigs": [ "KPKEYS_HARDENED_PGTABLES" ], "FocusSymbols": [ "kpkeys_pgtable_alloc", "kpkeys_pgtable_free", "kpkeys_physmem_pgtable_alloc", "set_memory_pkey", "arch_set_user_pkey_access" ], "KMSANReasoning": "The patch series introduces `KPKEYS_HARDENED_PGTABLES` to harden page tables using kernel pkeys (KPKEYS) on ARM64. It modifies page table allocation to set a non-default protection key and uses `guard(kpkeys_hardened_pgtables)()` to temporarily enable write access when modifying page table entries.\n\nAfter a thorough review of the changes:\n1. No new structures sent to user space: The patch adds `por_el1` and `kpkeys_state_lazy_mmu` to `struct thread_struct`, but these are internal kernel state and are not exposed to user space via ptrace, signals, or any other mechanism.\n2. Page table allocation zeroing: The patch modifies page table allocation to use `kpkeys_pgtable_alloc` and `kpkeys_physmem_pgtable_alloc`. These functions wrap standard allocators (`alloc_pages_noprof` and `memblock_phys_alloc_range`). The zeroing behavior remains identical to the original code: `early_pgtable_alloc` allocations are zeroed by `init_clear_pgtable()`, `pagetable_alloc_noprof` allocations are zeroed by `alloc_pages_noprof` if `__GFP_ZERO` is passed, and `vmemmap_alloc_pgtable` explicitly calls `memset(p, 0, PAGE_SIZE)`.\n3. No uninitialized memory risks: The `struct kpkeys_state` used in `scoped_guard` is a local variable and is not copied to user space. The conditional initialization in `arch_kpkeys_enter_context` is safe because the uninitialized fields are never read if `entered_context` is false.\n\nThe changes primarily involve memory protection and control flow around page table modifications. Any bugs introduced would likely be out-of-bounds accesses, use-after-free, or invalid memory accesses, which are well-covered by KASAN and standard bug detectors. There is no risk of uninitialized memory usage or info-leaks that would require KMSAN.", "NeedsKMSAN": false, "Reasoning": "The patch introduces a new feature to harden page tables using kernel pkeys (kpkeys) on ARM64. It adds a new file `mm/kpkeys_hardened_pgtables.c` and modifies ARM64 memory management code to allocate page tables with a specific pkey and restrict write access to them. The changes include new page table allocation and freeing functions, as well as modifications to `arch_set_user_pkey_access`. This is a significant functional change to the core memory management subsystem that should be fuzzed to ensure memory safety and correctness.", "WorthFuzzing": true }
1/1	2026/08/18 17:01	action	read-patch-diff	0m Args: null Results: { "PatchDiff": "commit bd271568204405923f3b37fcb4ded599be77ced1\nAuthor: syz-cluster \u003ctriage@syzkaller.com\u003e\nDate: Tue Aug 18 17:01:35 2026 +0000\n\n syz-cluster: applied patch under review\n\ndiff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig\nindex b3afe0688919b..ac9faec1a0184 100644\n--- a/arch/arm64/Kconfig\n+++ b/arch/arm64/Kconfig\n@@ -2202,6 +2202,8 @@ config ARM64_POE\n \tdef_bool y\n \tselect ARCH_USES_HIGH_VMA_FLAGS\n \tselect ARCH_HAS_PKEYS\n+\tselect ARCH_HAS_KPKEYS\n+\tselect ARCH_HAS_KPKEYS_HARDENED_PGTABLES\n \thelp\n \t The Permission Overlay Extension is used to implement Memory\n \t Protection Keys. Memory Protection Keys provides a mechanism for\ndiff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h\nindex a57870fa96db5..37aae60c2d7f0 100644\n--- a/arch/arm64/include/asm/cpufeature.h\n+++ b/arch/arm64/include/asm/cpufeature.h\n@@ -1085,6 +1085,18 @@ static inline bool cpu_has_lpa2(void)\n #endif\n }\n \n+static inline bool cpu_has_poe(void)\n+{\n+\tu64 mmfr3;\n+\n+\tif (!IS_ENABLED(CONFIG_ARM64_POE))\n+\t\treturn false;\n+\n+\tmmfr3 = read_sysreg_s(SYS_ID_AA64MMFR3_EL1);\n+\treturn cpuid_feature_extract_unsigned_field(mmfr3,\n+\t\t\t\t\t\t ID_AA64MMFR3_EL1_S1POE_SHIFT);\n+}\n+\n #endif /* __ASSEMBLER__ /\n \n #endif\ndiff --git a/arch/arm64/include/asm/kpkeys.h b/arch/arm64/include/asm/kpkeys.h\nnew file mode 100644\nindex 0000000000000..7747af54fd386\n--- /dev/null\n+++ b/arch/arm64/include/asm/kpkeys.h\n@@ -0,0 +1,84 @@\n+/ SPDX-License-Identifier: GPL-2.0-only /\n+#ifndef __ASM_KPKEYS_H\n+#define __ASM_KPKEYS_H\n+\n+#include \u003clinux/kpkeys_types.h\u003e\n+\n+#include \u003casm/barrier.h\u003e\n+#include \u003casm/cpufeature.h\u003e\n+#include \u003casm/por.h\u003e\n+\n+/\n+ * Equivalent to por_set_kpkeys_context(0, KPKEYS_CTX_DEFAULT), but can also be\n+ * used in assembly.\n+ /\n+#define POR_EL1_INIT\t(POR_ELx_PERM_PREP(0, POE_RWX) \| \\\n+\t\t\t POR_ELx_PERM_PREP(KPKEYS_PKEY_PGTABLES, POE_R))\n+\n+#ifndef __ASSEMBLY__\n+\n+static inline bool arch_supports_kpkeys(void)\n+{\n+\treturn system_supports_poe();\n+}\n+\n+static inline bool arch_supports_kpkeys_early(void)\n+{\n+\t/ POE is a boot feature /\n+\treturn boot_capabilities_finalized() ?\n+\t\tsystem_supports_poe() : cpu_has_poe();\n+}\n+\n+#ifdef CONFIG_ARM64_POE\n+\n+static inline u64 por_set_kpkeys_context(u64 por, enum kpkeys_ctx ctx)\n+{\n+\tpor = por_elx_set_pkey_perms(por, 0, POE_RWX);\n+\tpor = por_elx_set_pkey_perms(por, KPKEYS_PKEY_PGTABLES,\n+\t\t\t\t ctx == KPKEYS_CTX_PGTABLES ? POE_RW : POE_R);\n+\n+\treturn por;\n+}\n+\n+static __always_inline void __kpkeys_set_pkey_reg_nosync(u64 pkey_reg)\n+{\n+\twrite_sysreg_s(pkey_reg, SYS_POR_EL1);\n+}\n+\n+static __always_inline\n+struct kpkeys_state arch_kpkeys_enter_context(enum kpkeys_ctx ctx)\n+{\n+\tconst u64 prev_por = read_sysreg_s(SYS_POR_EL1);\n+\tconst u64 new_por = por_set_kpkeys_context(prev_por, ctx);\n+\n+\tif (prev_por == new_por)\n+\t\treturn (struct kpkeys_state) { .entered_context = false };\n+\n+\t__kpkeys_set_pkey_reg_nosync(new_por);\n+\tisb();\n+\n+\treturn (struct kpkeys_state) {\n+\t\t.entered_context = true,\n+\t\t.arch_state.por = prev_por,\n+\t};\n+}\n+\n+static __always_inline\n+void arch_kpkeys_leave_context(const struct kpkeys_state state)\n+{\n+\t__kpkeys_set_pkey_reg_nosync(state-\u003earch_state.por);\n+\tisb();\n+}\n+\n+#endif /* CONFIG_ARM64_POE /\n+\n+#ifdef CONFIG_KPKEYS_HARDENED_PGTABLES\n+\n+#define arch_kpkeys_protect_static_pgtables arch_kpkeys_protect_static_pgtables\n+void arch_kpkeys_protect_static_pgtables(void);\n+\n+#endif / CONFIG_KPKEYS_HARDENED_PGTABLES /\n+\n+#endif\t/ __ASSEMBLY__ /\n+\n+#endif\t/ __ASM_KPKEYS_H /\ndiff --git a/arch/arm64/include/asm/kpkeys_types.h b/arch/arm64/include/asm/kpkeys_types.h\nnew file mode 100644\nindex 0000000000000..e9c59a26ee7a9\n--- /dev/null\n+++ b/arch/arm64/include/asm/kpkeys_types.h\n@@ -0,0 +1,15 @@\n+/ SPDX-License-Identifier: GPL-2.0-only /\n+#ifndef __ASM_KPKEYS_TYPES_H\n+#define __ASM_KPKEYS_TYPES_H\n+\n+#include \u003clinux/types.h\u003e\n+\n+#ifndef __ASSEMBLY__\n+\n+struct arch_kpkeys_state {\n+\tu64 por;\n+};\n+\n+#endif\t/ __ASSEMBLY__ /\n+\n+#endif\t/ __ASM_KPKEYS_TYPES_H /\ndiff --git a/arch/arm64/include/asm/pgtable-prot.h b/arch/arm64/include/asm/pgtable-prot.h\nindex 212ce1b02e15e..d4d45ab86a5a8 100644\n--- a/arch/arm64/include/asm/pgtable-prot.h\n+++ b/arch/arm64/include/asm/pgtable-prot.h\n@@ -183,9 +183,9 @@ static inline bool __pure lpa2_is_enabled(void)\n \tPIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_SHARED_EXEC), PIE_RW) \| \\\n \tPIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_READONLY), PIE_R) \| \\\n \tPIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_SHARED), PIE_RW) \| \\\n-\tPIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL_ROX), PIE_RX) \| \\\n-\tPIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL_EXEC), PIE_RWX) \| \\\n-\tPIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL_RO), PIE_R) \| \\\n-\tPIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL), PIE_RW))\n+\tPIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL_ROX), PIE_RX_O) \| \\\n+\tPIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL_EXEC), PIE_RWX_O) \| \\\n+\tPIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL_RO), PIE_R_O) \| \\\n+\tPIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL), PIE_RW_O))\n \n #endif / __ASM_PGTABLE_PROT_H /\ndiff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h\nindex 27689c62bd25d..6e5d14c7e8b8f 100644\n--- a/arch/arm64/include/asm/pgtable.h\n+++ b/arch/arm64/include/asm/pgtable.h\n@@ -39,6 +39,48 @@\n #include \u003clinux/mm_types.h\u003e\n #include \u003clinux/sched.h\u003e\n #include \u003clinux/page_table_check.h\u003e\n+#include \u003clinux/kpkeys.h\u003e\n+\n+#ifdef CONFIG_KPKEYS_HARDENED_PGTABLES\n+KPKEYS_GUARD_COND(kpkeys_hardened_pgtables, KPKEYS_CTX_PGTABLES,\n+\t\t kpkeys_hardened_pgtables_enabled() \u0026\u0026\n+\t\t !is_lazy_mmu_mode_active())\n+\n+static inline void kpkeys_lazy_mmu_enter(void)\n+{\n+\tif (!kpkeys_hardened_pgtables_enabled())\n+\t\treturn;\n+\n+\tcurrent-\u003ethread.kpkeys_state_lazy_mmu =\n+\t\tkpkeys_enter_context(KPKEYS_CTX_PGTABLES);\n+}\n+\n+static inline void kpkeys_lazy_mmu_exit(void)\n+{\n+\tconst struct kpkeys_state state =\n+\t\t\u0026current-\u003ethread.kpkeys_state_lazy_mmu;\n+\n+\tif (!kpkeys_hardened_pgtables_enabled())\n+\t\treturn;\n+\n+\t/\n+\t We skip any barrier if TIF_LAZY_MMU_PENDING is set:\n+\t * emit_pte_barriers() will issue an ISB just after this function\n+\t * returns.\n+\t /\n+\tif (test_thread_flag(TIF_LAZY_MMU_PENDING) \u0026\u0026 state-\u003eentered_context)\n+\t\t__kpkeys_set_pkey_reg_nosync(state-\u003earch_state.por);\n+\telse\n+\t\tkpkeys_leave_context(state);\n+}\n+#else / CONFIG_KPKEYS_HARDENED_PGTABLES /\n+KPKEYS_GUARD_NOOP(kpkeys_hardened_pgtables)\n+\n+static inline void kpkeys_lazy_mmu_enter(void) {}\n+static inline void kpkeys_lazy_mmu_exit(void) {}\n+#endif / CONFIG_KPKEYS_HARDENED_PGTABLES /\n+\n+\n \n static inline void emit_pte_barriers(void)\n {\n@@ -71,7 +113,10 @@ static inline void queue_pte_barriers(void)\n \t}\n }\n \n-static inline void arch_enter_lazy_mmu_mode(void) {}\n+static inline void arch_enter_lazy_mmu_mode(void)\n+{\n+\tkpkeys_lazy_mmu_enter();\n+}\n \n static inline void arch_flush_lazy_mmu_mode(void)\n {\n@@ -81,6 +126,11 @@ static inline void arch_flush_lazy_mmu_mode(void)\n \n static inline void arch_leave_lazy_mmu_mode(void)\n {\n+\t/\n+\t * The ordering should be preserved to allow kpkeys_lazy_mmu_exit()\n+\t * to skip any barrier when TIF_LAZY_MMU_PENDING is set.\n+\t /\n+\tkpkeys_lazy_mmu_exit();\n \tarch_flush_lazy_mmu_mode();\n }\n \n@@ -359,6 +409,7 @@ static inline pte_t pte_clear_uffd_wp(pte_t pte)\n \n static inline void __set_pte_nosync(pte_t ptep, pte_t pte)\n {\n+\tguard(kpkeys_hardened_pgtables)();\n \tWRITE_ONCE(ptep, pte);\n }\n \n@@ -830,6 +881,7 @@ static inline void set_pmd(pmd_t pmdp, pmd_t pmd)\n \t}\n #endif /* __PAGETABLE_PMD_FOLDED /\n \n+\tguard(kpkeys_hardened_pgtables)();\n \tWRITE_ONCE(pmdp, pmd);\n \n \tif (pmd_valid(pmd))\n@@ -894,6 +946,7 @@ static inline void set_pud(pud_t pudp, pud_t pud)\n \t\treturn;\n \t}\n \n+\tguard(kpkeys_hardened_pgtables)();\n \tWRITE_ONCE(pudp, pud);\n \n \tif (pud_valid(pud))\n@@ -975,6 +1028,7 @@ static inline void set_p4d(p4d_t p4dp, p4d_t p4d)\n \t\treturn;\n \t}\n \n+\tguard(kpkeys_hardened_pgtables)();\n \tWRITE_ONCE(p4dp, p4d);\n \tqueue_pte_barriers();\n }\n@@ -1103,6 +1157,7 @@ static inline void set_pgd(pgd_t pgdp, pgd_t pgd)\n \t\treturn;\n \t}\n \n+\tguard(kpkeys_hardened_pgtables)();\n \tWRITE_ONCE(pgdp, pgd);\n \tqueue_pte_barriers();\n }\n@@ -1307,6 +1362,7 @@ static inline bool __ptep_test_and_clear_young(struct vm_area_struct vma,\n {\n \tpte_t old_pte, pte;\n \n+\tguard(kpkeys_hardened_pgtables)();\n \tpte = __ptep_get(ptep);\n \tdo {\n \t\told_pte = pte;\n@@ -1354,7 +1410,10 @@ static inline pte_t __ptep_get_and_clear_anysz(struct mm_struct mm,\n \t\t\t\t\t pte_t ptep,\n \t\t\t\t\t unsigned long pgsize)\n {\n-\tpte_t pte = __pte(xchg_relaxed(\u0026pte_val(ptep), 0));\n+\tpte_t pte;\n+\n+\tscoped_guard(kpkeys_hardened_pgtables)\n+\t\tpte = __pte(xchg_relaxed(\u0026pte_val(ptep), 0));\n \n \tswitch (pgsize) {\n \tcase PAGE_SIZE:\n@@ -1427,6 +1486,7 @@ static inline void ___ptep_set_wrprotect(struct mm_struct mm,\n {\n \tpte_t old_pte;\n \n+\tguard(kpkeys_hardened_pgtables)();\n \tdo {\n \t\told_pte = pte;\n \t\tpte = pte_wrprotect(pte);\n@@ -1460,6 +1520,7 @@ static inline void __clear_young_dirty_pte(struct vm_area_struct vma,\n {\n \tpte_t old_pte;\n \n+\tguard(kpkeys_hardened_pgtables)();\n \tdo {\n \t\told_pte = pte;\n \n@@ -1507,6 +1568,7 @@ static inline pmd_t pmdp_establish(struct vm_area_struct vma,\n \t\tunsigned long address, pmd_t pmdp, pmd_t pmd)\n {\n \tpage_table_check_pmd_set(vma-\u003evm_mm, address, pmdp, pmd);\n+\tguard(kpkeys_hardened_pgtables)();\n \treturn __pmd(xchg_relaxed(\u0026pmd_val(pmdp), pmd_val(pmd)));\n }\n #endif\ndiff --git a/arch/arm64/include/asm/por.h b/arch/arm64/include/asm/por.h\nindex d913d5b529e49..dddabf6ffae6d 100644\n--- a/arch/arm64/include/asm/por.h\n+++ b/arch/arm64/include/asm/por.h\n@@ -10,6 +10,8 @@\n \n #define POR_EL0_INIT\tPOR_ELx_PERM_PREP(0, POE_RWX)\n \n+#ifndef __ASSEMBLY__\n+\n static inline bool por_elx_allows_read(u64 por, u8 pkey)\n {\n \tu8 perm = POR_ELx_PERM_GET(pkey, por);\n@@ -31,4 +33,13 @@ static inline bool por_elx_allows_exec(u64 por, u8 pkey)\n \treturn perm \u0026 POE_X;\n }\n \n+static inline u64 por_elx_set_pkey_perms(u64 por, u8 pkey, u8 perms)\n+{\n+\tu64 shift = POR_ELx_PERM_SHIFT(pkey);\n+\n+\treturn (por \u0026 ~(POE_MASK \u003c\u003c shift)) \| ((u64)perms \u003c\u003c shift);\n+}\n+\n+#endif\t/* __ASSEMBLY__ /\n+\n #endif / _ASM_ARM64_POR_H /\ndiff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h\nindex c2a627f393144..23660a233da28 100644\n--- a/arch/arm64/include/asm/processor.h\n+++ b/arch/arm64/include/asm/processor.h\n@@ -30,6 +30,7 @@\n #include \u003clinux/build_bug.h\u003e\n #include \u003clinux/cache.h\u003e\n #include \u003clinux/init.h\u003e\n+#include \u003clinux/kpkeys_types.h\u003e\n #include \u003clinux/stddef.h\u003e\n #include \u003clinux/string.h\u003e\n #include \u003clinux/thread_info.h\u003e\n@@ -195,6 +196,10 @@ struct thread_struct {\n \tu64\t\t\tsvcr;\n \tu64\t\t\ttpidr2_el0;\n \tu64\t\t\tpor_el0;\n+\tu64\t\t\tpor_el1;\n+#ifdef CONFIG_KPKEYS_HARDENED_PGTABLES\n+\tstruct kpkeys_state\tkpkeys_state_lazy_mmu;\n+#endif\n #ifdef CONFIG_ARM64_GCS\n \tunsigned int\t\tgcs_el0_mode;\n \tunsigned int\t\tgcs_el0_locked;\ndiff --git a/arch/arm64/include/asm/sections.h b/arch/arm64/include/asm/sections.h\nindex 51b0d594239eb..dc961cf67168c 100644\n--- a/arch/arm64/include/asm/sections.h\n+++ b/arch/arm64/include/asm/sections.h\n@@ -23,6 +23,7 @@ extern char __irqentry_text_start[], __irqentry_text_end[];\n extern char __mmuoff_data_start[], __mmuoff_data_end[];\n extern char __entry_tramp_text_start[], __entry_tramp_text_end[];\n extern char __relocate_new_kernel_start[], __relocate_new_kernel_end[];\n+extern char __pi_init_pg_dir[], __pi_init_pg_end[];\n \n static inline size_t entry_tramp_text_size(void)\n {\ndiff --git a/arch/arm64/include/asm/set_memory.h b/arch/arm64/include/asm/set_memory.h\nindex 90f61b17275e1..b6cd6de34abf1 100644\n--- a/arch/arm64/include/asm/set_memory.h\n+++ b/arch/arm64/include/asm/set_memory.h\n@@ -19,4 +19,8 @@ bool kernel_page_present(struct page page);\n int set_memory_encrypted(unsigned long addr, int numpages);\n int set_memory_decrypted(unsigned long addr, int numpages);\n \n+#ifdef CONFIG_ARCH_HAS_KPKEYS\n+int set_memory_pkey(unsigned long addr, int numpages, int pkey);\n+#endif\n+\n #endif /* _ASM_ARM64_SET_MEMORY_H /\ndiff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c\nindex 9a22df0c5120f..4f2f54007ed6f 100644\n--- a/arch/arm64/kernel/cpufeature.c\n+++ b/arch/arm64/kernel/cpufeature.c\n@@ -76,6 +76,7 @@\n #include \u003clinux/kasan.h\u003e\n #include \u003clinux/percpu.h\u003e\n #include \u003clinux/sched/isolation.h\u003e\n+#include \u003clinux/kpkeys.h\u003e\n \n #include \u003casm/arm_pmuv3.h\u003e\n #include \u003casm/cpu.h\u003e\n@@ -2463,8 +2464,10 @@ static void cpu_enable_mops(const struct arm64_cpu_capabilities __unused)\n #ifdef CONFIG_ARM64_POE\n static void cpu_enable_poe(const struct arm64_cpu_capabilities __unused)\n {\n-\tsysreg_clear_set(REG_TCR2_EL1, 0, TCR2_EL1_E0POE);\n+\twrite_sysreg_s(POR_EL1_INIT, SYS_POR_EL1);\n+\tsysreg_clear_set(REG_TCR2_EL1, 0, TCR2_EL1_E0POE \| TCR2_EL1_POE);\n \tsysreg_clear_set(CPACR_EL1, 0, CPACR_EL1_E0POE);\n+\tisb();\n }\n #endif\n \ndiff --git a/arch/arm64/kernel/pi/map_kernel.c b/arch/arm64/kernel/pi/map_kernel.c\nindex fb44cbdd2f292..81ca3e98f0f7b 100644\n--- a/arch/arm64/kernel/pi/map_kernel.c\n+++ b/arch/arm64/kernel/pi/map_kernel.c\n@@ -92,8 +92,13 @@ static void __init map_kernel(u64 kaslr_offset, u64 va_offset, int root_level)\n \t\t __inittext_end, prot, false, root_level);\n \tmap_segment(init_pg_dir, \u0026pgdp, va_offset, __initdata_begin,\n \t\t __initdata_end, data_prot, false, root_level);\n-\tmap_segment(init_pg_dir, \u0026pgdp, va_offset, _data, _end, data_prot,\n-\t\t true, root_level);\n+\tmap_segment(init_pg_dir, \u0026pgdp, va_offset, _data, init_pg_dir,\n+\t\t data_prot, true, root_level);\n+\t/ Map init_pg_dir separately as we will remap it read-only later on /\n+\tmap_segment(init_pg_dir, \u0026pgdp, va_offset, init_pg_dir, init_pg_end,\n+\t\t data_prot, true, root_level);\n+\tmap_segment(init_pg_dir, \u0026pgdp, va_offset, init_pg_end, _end,\n+\t\t data_prot, true, root_level);\n \tdsb(ishst);\n \n \tidmap_cpu_replace_ttbr1((phys_addr_t)init_pg_dir);\ndiff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c\nindex 581f80e9b9b73..1dec143decf15 100644\n--- a/arch/arm64/kernel/process.c\n+++ b/arch/arm64/kernel/process.c\n@@ -51,6 +51,7 @@\n #include \u003casm/exec.h\u003e\n #include \u003casm/fpsimd.h\u003e\n #include \u003casm/gcs.h\u003e\n+#include \u003casm/kpkeys.h\u003e\n #include \u003casm/mmu_context.h\u003e\n #include \u003casm/mpam.h\u003e\n #include \u003casm/mte.h\u003e\n@@ -431,6 +432,9 @@ int copy_thread(struct task_struct p, const struct kernel_clone_args args)\n \n \tptrauth_thread_init_kernel(p);\n \n+\tif (system_supports_poe())\n+\t\tp-\u003ethread.por_el1 = POR_EL1_INIT;\n+\n \tif (likely(!args-\u003efn)) {\n \t\tchildregs = current_pt_regs();\n \t\tchildregs-\u003eregs[0] = 0;\n@@ -681,6 +685,12 @@ static void permission_overlay_switch(struct task_struct next)\n \t\t * of POR_EL0.\n \t\t /\n \t}\n+\n+\tcurrent-\u003ethread.por_el1 = read_sysreg_s(SYS_POR_EL1);\n+\tif (current-\u003ethread.por_el1 != next-\u003ethread.por_el1) {\n+\t\twrite_sysreg_s(next-\u003ethread.por_el1, SYS_POR_EL1);\n+\t\tisb();\n+\t}\n }\n \n /\ndiff --git a/arch/arm64/kernel/sleep.S b/arch/arm64/kernel/sleep.S\nindex f093cdf71be11..e0a6ad85cd242 100644\n--- a/arch/arm64/kernel/sleep.S\n+++ b/arch/arm64/kernel/sleep.S\n@@ -3,6 +3,7 @@\n #include \u003clinux/linkage.h\u003e\n #include \u003casm/asm-offsets.h\u003e\n #include \u003casm/assembler.h\u003e\n+#include \u003casm/kpkeys.h\u003e\n #include \u003casm/smp.h\u003e\n \n \t.text\n@@ -134,6 +135,17 @@ SYM_FUNC_START(_cpu_resume)\n \t/* load sp from context /\n \tldr\tx2, [x0, #CPU_CTX_SP]\n \tmov\tsp, x2\n+\n+#ifdef CONFIG_ARM64_POE\n+alternative_if_not ARM64_HAS_S1POE\n+\tb\t.Lskip_por_set\n+alternative_else_nop_endif\n+\tmov_q\tx2, POR_EL1_INIT\n+\tmsr_s\tSYS_POR_EL1, x2\n+\t/ isb can be skipped since cpu_do_resume() will do it. /\n+.Lskip_por_set:\n+#endif / CONFIG_ARM64_POE /\n+\n \t/\n \t * cpu_do_resume expects x0 to contain context address pointer\n \t /\ndiff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c\nindex 0b52557652be6..a853a8d4e61af 100644\n--- a/arch/arm64/mm/fault.c\n+++ b/arch/arm64/mm/fault.c\n@@ -229,13 +229,16 @@ int __ptep_set_access_flags_anysz(struct vm_area_struct vma,\n \t /\n \tpte_val(entry) ^= PTE_RDONLY;\n \tpteval = pte_val(pte);\n-\tdo {\n-\t\told_pteval = pteval;\n-\t\tpteval ^= PTE_RDONLY;\n-\t\tpteval \|= pte_val(entry);\n-\t\tpteval ^= PTE_RDONLY;\n-\t\tpteval = cmpxchg_relaxed(\u0026pte_val(ptep), old_pteval, pteval);\n-\t} while (pteval != old_pteval);\n+\tscoped_guard(kpkeys_hardened_pgtables) {\n+\t\tdo {\n+\t\t\told_pteval = pteval;\n+\t\t\tpteval ^= PTE_RDONLY;\n+\t\t\tpteval \|= pte_val(entry);\n+\t\t\tpteval ^= PTE_RDONLY;\n+\t\t\tpteval = cmpxchg_relaxed(\u0026pte_val(ptep), old_pteval,\n+\t\t\t\t\t\t pteval);\n+\t\t} while (pteval != old_pteval);\n+\t}\n \n \t/\n \t * Invalidate the local stale read-only entry. Remote stale entries\ndiff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c\nindex 97987f850a33c..7985045875cf6 100644\n--- a/arch/arm64/mm/init.c\n+++ b/arch/arm64/mm/init.c\n@@ -386,6 +386,7 @@ bool page_alloc_available __ro_after_init;\n void __init mem_init(void)\n {\n \tpage_alloc_available = true;\n+\tkpkeys_hardened_pgtables_init();\n \tswiotlb_update_mem_attributes();\n }\n \ndiff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c\nindex d4de88770ecfe..472ad0f8f002c 100644\n--- a/arch/arm64/mm/mmu.c\n+++ b/arch/arm64/mm/mmu.c\n@@ -107,8 +107,11 @@ static phys_addr_t __init early_pgtable_alloc(enum pgtable_level pgtable_level)\n {\n \tphys_addr_t phys;\n \n-\tphys = memblock_phys_alloc_range(PAGE_SIZE, PAGE_SIZE, 0,\n-\t\t\t\t\t MEMBLOCK_ALLOC_NOLEAKTRACE);\n+\tif (kpkeys_hardened_pgtables_early_enabled())\n+\t\tphys = kpkeys_physmem_pgtable_alloc();\n+\telse\n+\t\tphys = memblock_phys_alloc_range(PAGE_SIZE, PAGE_SIZE, 0,\n+\t\t\t\t\t\t MEMBLOCK_ALLOC_NOLEAKTRACE);\n \tif (!phys)\n \t\tpanic(\"Failed to allocate page table page\\n\");\n \n@@ -780,7 +783,7 @@ static inline bool force_pte_mapping(void)\n \tconst bool bbml2 = system_capabilities_finalized() ?\n \t\tsystem_supports_bbml2_noabort() : cpu_supports_bbml2_noabort();\n \n-\tif (debug_pagealloc_enabled())\n+\tif (debug_pagealloc_enabled() \|\| kpkeys_hardened_pgtables_early_enabled())\n \t\treturn true;\n \tif (bbml2)\n \t\treturn false;\n@@ -1070,6 +1073,18 @@ void __init mark_linear_text_alias_ro(void)\n \t\t\t PAGE_KERNEL_RO);\n }\n \n+#ifdef CONFIG_KPKEYS_HARDENED_PGTABLES\n+void __init arch_kpkeys_protect_static_pgtables(void)\n+{\n+\tunsigned long addr = (unsigned long)lm_alias(__pi_init_pg_dir);\n+\tunsigned long size = __pi_init_pg_end - __pi_init_pg_dir;\n+\tint ret;\n+\n+\tret = set_memory_pkey(addr, size / PAGE_SIZE, KPKEYS_PKEY_PGTABLES);\n+\tWARN_ON(ret);\n+}\n+#endif /* CONFIG_KPKEYS_HARDENED_PGTABLES /\n+\n #ifdef CONFIG_KFENCE\n \n bool __ro_after_init kfence_early_init = !!CONFIG_KFENCE_SAMPLE_INTERVAL;\n@@ -1214,6 +1229,14 @@ void mark_rodata_ro(void)\n \tupdate_mapping_prot(__pa_symbol(_text), (unsigned long)_text,\n \t\t\t (unsigned long)_stext - (unsigned long)_text,\n \t\t\t PAGE_KERNEL_RO);\n+\t/\n+\t * Map the kernel image mapping of init_pg_dir read-only; it should\n+\t * only be written via the linear map.\n+\t /\n+\tsection_size = (unsigned long)__pi_init_pg_end - (unsigned long)__pi_init_pg_dir;\n+\tupdate_mapping_prot(__pa_symbol(__pi_init_pg_dir),\n+\t\t\t (unsigned long)__pi_init_pg_dir,\n+\t\t\t section_size, PAGE_KERNEL_RO);\n \n \t/ Map the kernel data/bss read-only in the linear map /\n \tupdate_mapping_prot(__pa_symbol(__init_end),\n@@ -1446,7 +1469,7 @@ static void free_hotplug_page_range(struct page page, size_t size,\n static void free_hotplug_pgtable_page(struct page page)\n {\n \tpagetable_dtor(page_ptdesc(page));\n-\tfree_hotplug_page_range(page, PAGE_SIZE, NULL);\n+\tpagetable_free(page_ptdesc(page));\n }\n \n static bool pgtable_range_aligned(unsigned long start, unsigned long end,\n@@ -2305,8 +2328,8 @@ void __cpu_replace_ttbr1(pgd_t pgdp, bool cnp)\n #ifdef CONFIG_ARCH_HAS_PKEYS\n int arch_set_user_pkey_access(int pkey, unsigned long init_val)\n {\n-\tu64 new_por;\n-\tu64 old_por;\n+\tu8 new_perms;\n+\tu64 por;\n \n \tif (!system_supports_poe())\n \t\treturn -ENOSPC;\n@@ -2320,25 +2343,19 @@ int arch_set_user_pkey_access(int pkey, unsigned long init_val)\n \t\treturn -EINVAL;\n \n \t/* Set the bits we need in POR: /\n-\tnew_por = POE_RWX;\n+\tnew_perms = POE_RWX;\n \tif (init_val \u0026 PKEY_DISABLE_WRITE)\n-\t\tnew_por \u0026= ~POE_W;\n+\t\tnew_perms \u0026= ~POE_W;\n \tif (init_val \u0026 PKEY_DISABLE_ACCESS)\n-\t\tnew_por \u0026= ~POE_RW;\n+\t\tnew_perms \u0026= ~POE_RW;\n \tif (init_val \u0026 PKEY_DISABLE_READ)\n-\t\tnew_por \u0026= ~POE_R;\n+\t\tnew_perms \u0026= ~POE_R;\n \tif (init_val \u0026 PKEY_DISABLE_EXECUTE)\n-\t\tnew_por \u0026= ~POE_X;\n-\n-\t/ Shift the bits in to the correct place in POR for pkey: /\n-\tnew_por = POR_ELx_PERM_PREP(pkey, new_por);\n-\n-\t/ Get old POR and mask off any old bits in place: /\n-\told_por = read_sysreg_s(SYS_POR_EL0);\n-\told_por \u0026= ~(POE_MASK \u003c\u003c POR_ELx_PERM_SHIFT(pkey));\n+\t\tnew_perms \u0026= ~POE_X;\n \n-\t/ Write old part along with new part: /\n-\twrite_sysreg_s(old_por \| new_por, SYS_POR_EL0);\n+\tpor = read_sysreg_s(SYS_POR_EL0);\n+\tpor = por_elx_set_pkey_perms(por, pkey, new_perms);\n+\twrite_sysreg_s(por, SYS_POR_EL0);\n \n \treturn 0;\n }\ndiff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c\nindex bbe98ac9ad8c6..46fbebfc51003 100644\n--- a/arch/arm64/mm/pageattr.c\n+++ b/arch/arm64/mm/pageattr.c\n@@ -9,6 +9,8 @@\n #include \u003clinux/sched.h\u003e\n #include \u003clinux/vmalloc.h\u003e\n #include \u003clinux/pagewalk.h\u003e\n+#include \u003clinux/pkeys.h\u003e\n+#include \u003clinux/kpkeys.h\u003e\n \n #include \u003casm/cacheflush.h\u003e\n #include \u003casm/pgtable-prot.h\u003e\n@@ -99,7 +101,8 @@ bool can_set_direct_map(void)\n \t Realms need to make pages shared/protected at page granularity.\n \t /\n \treturn rodata_full \|\| debug_pagealloc_enabled() \|\|\n-\t\tarm64_kfence_can_set_direct_map() \|\| is_realm_world();\n+\t\tarm64_kfence_can_set_direct_map() \|\| is_realm_world() \|\|\n+\t\tkpkeys_hardened_pgtables_early_enabled();\n }\n \n static int update_range_prot(unsigned long start, unsigned long size,\n@@ -365,6 +368,30 @@ int set_direct_map_valid_noflush(struct page page, unsigned nr, bool valid)\n \treturn set_memory_valid(addr, nr, valid);\n }\n \n+#ifdef CONFIG_ARCH_HAS_KPKEYS\n+int set_memory_pkey(unsigned long addr, int numpages, int pkey)\n+{\n+\tunsigned long set_prot = 0;\n+\n+\tif (!kpkeys_supported())\n+\t\treturn 0;\n+\n+\tif (!__is_lm_address(kasan_reset_tag((void )addr)))\n+\t\treturn -EINVAL;\n+\n+\tif (pkey \u003c 0 \|\| pkey \u003e= arch_max_pkey())\n+\t\treturn -EINVAL;\n+\n+\tset_prot \|= pkey \u0026 BIT(0) ? PTE_PO_IDX_0 : 0;\n+\tset_prot \|= pkey \u0026 BIT(1) ? PTE_PO_IDX_1 : 0;\n+\tset_prot \|= pkey \u0026 BIT(2) ? PTE_PO_IDX_2 : 0;\n+\n+\treturn __change_memory_common(addr, PAGE_SIZE numpages,\n+\t\t\t\t __pgprot(set_prot),\n+\t\t\t\t __pgprot(PTE_PO_IDX_MASK));\n+}\n+#endif\n+\n #ifdef CONFIG_DEBUG_PAGEALLOC\n /\n This is - apart from the return value - doing the same\ndiff --git a/include/linux/kpkeys.h b/include/linux/kpkeys.h\nnew file mode 100644\nindex 0000000000000..df55ec854586e\n--- /dev/null\n+++ b/include/linux/kpkeys.h\n@@ -0,0 +1,162 @@\n+/* SPDX-License-Identifier: GPL-2.0-only /\n+#ifndef _LINUX_KPKEYS_H\n+#define _LINUX_KPKEYS_H\n+\n+#include \u003clinux/bug.h\u003e\n+#include \u003clinux/cleanup.h\u003e\n+#include \u003clinux/jump_label.h\u003e\n+#include \u003clinux/kpkeys_types.h\u003e\n+\n+/\n+ KPKEYS_GUARD_NOOP() - define a guard type that does nothing\n+ * @name: the name of the guard type\n+ \n+ Define a guard type that does nothing, useful to match a real guard type\n+ * that is defined under an #ifdef.\n+ /\n+#define KPKEYS_GUARD_NOOP(name)\t\t\t\t\t\t\\\n+\t__DEFINE_CLASS_IS_CONDITIONAL(name, false);\t\t\t\\\n+\tDEFINE_CLASS(name, bool, (void)_T, false, void);\t\t\\\n+\tstatic inline void class_##name##_lock_ptr(bool _T)\t\t\\\n+\t{ return _T; }\n+\n+#ifdef CONFIG_ARCH_HAS_KPKEYS\n+\n+#include \u003casm/kpkeys.h\u003e\n+\n+/\n+ KPKEYS_GUARD_COND() - define a guard type that conditionally switches to\n+ * a given kpkeys context\n+ * @name: the name of the guard type\n+ * @ctx: the kpkeys context to switch to\n+ * @cond: an expression that is evaluated as condition\n+ \n+ Define a guard type that switches to @ctx if @cond evaluates to true,\n+ * and does nothing otherwise.\n+ /\n+#define KPKEYS_GUARD_COND(name, ctx, cond)\t\t\t\t\\\n+\t__DEFINE_CLASS_IS_CONDITIONAL(name, false);\t\t\t\\\n+\tDEFINE_CLASS(name, struct kpkeys_state,\t\t\t\t\\\n+\t\t kpkeys_leave_context(\u0026_T),\t\t\t\t\\\n+\t\t (cond) ? kpkeys_enter_context(ctx) :\t\t\\\n+\t\t\t (struct kpkeys_state) {}, void);\t\t\\\n+\tstatic inline\t\t\t\t\t\t\t\\\n+\tvoid class_##name##_lock_ptr(struct kpkeys_state _T)\t\t\\\n+\t{ return _T; }\n+\n+/\n+ KPKEYS_GUARD() - define a guard type that switches to a given kpkeys context\n+ * if kpkeys are supported\n+ * @name: the name of the guard type\n+ * @ctx: the kpkeys context to switch to\n+ \n+ Define a guard type that switches to @ctx if the system supports kpkeys.\n+ /\n+#define KPKEYS_GUARD(name, ctx)\t\t\t\t\t\t\\\n+\tKPKEYS_GUARD_COND(name, ctx, kpkeys_supported())\n+\n+/\n+ kpkeys_enter_context() - enter a kpkeys context\n+ * @ctx: the context to switch to\n+ \n+ Enters the specified kpkeys context. @ctx must be a compile-time constant.\n+ \n+ Return: state to be passed to kpkeys_leave_context().\n+ /\n+static __always_inline\n+struct kpkeys_state kpkeys_enter_context(enum kpkeys_ctx ctx)\n+{\n+\tBUILD_BUG_ON_MSG(!__builtin_constant_p(ctx),\n+\t\t\t \"kpkeys_enter_context() only takes constant values\");\n+\tBUILD_BUG_ON_MSG(ctx \u003c 0 \|\| ctx \u003e= KPKEYS_CTX_COUNT,\n+\t\t\t \"Invalid value passed to kpkeys_enter_context()\");\n+\n+\treturn arch_kpkeys_enter_context(ctx);\n+}\n+\n+/\n+ kpkeys_leave_context() - leave a kpkeys context\n+ * @state: state returned by kpkeys_enter_context()\n+ \n+ Restores the state saved when entering a kpkeys context. If no context was\n+ * entered, this function does nothing.\n+ /\n+static __always_inline\n+void kpkeys_leave_context(const struct kpkeys_state state)\n+{\n+\tif (state-\u003eentered_context)\n+\t\tarch_kpkeys_leave_context(state);\n+}\n+\n+static inline bool kpkeys_supported(void)\n+{\n+\treturn arch_supports_kpkeys();\n+}\n+\n+#else /* CONFIG_ARCH_HAS_KPKEYS /\n+\n+static inline bool kpkeys_supported(void)\n+{\n+\treturn false;\n+}\n+\n+#endif / CONFIG_ARCH_HAS_KPKEYS /\n+\n+#ifdef CONFIG_KPKEYS_HARDENED_PGTABLES\n+\n+DECLARE_STATIC_KEY_FALSE(kpkeys_hardened_pgtables_key);\n+\n+static inline bool kpkeys_hardened_pgtables_enabled(void)\n+{\n+\treturn static_branch_unlikely(\u0026kpkeys_hardened_pgtables_key);\n+}\n+\n+static inline bool kpkeys_hardened_pgtables_early_enabled(void)\n+{\n+\treturn arch_supports_kpkeys_early();\n+}\n+\n+struct page kpkeys_pgtable_alloc(gfp_t gfp, unsigned int order);\n+void kpkeys_pgtable_free(struct page page, unsigned int order);\n+\n+/\n+ * Should be called from mem_init(): as soon as the buddy allocator becomes\n+ * available and before any call to pagetable_alloc().\n+ /\n+void kpkeys_hardened_pgtables_init(void);\n+\n+phys_addr_t kpkeys_physmem_pgtable_alloc(void);\n+\n+#ifndef arch_kpkeys_protect_static_pgtables\n+static inline void arch_kpkeys_protect_static_pgtables(void) {}\n+#endif\n+\n+#else / CONFIG_KPKEYS_HARDENED_PGTABLES /\n+\n+static inline bool kpkeys_hardened_pgtables_enabled(void)\n+{\n+\treturn false;\n+}\n+\n+static inline bool kpkeys_hardened_pgtables_early_enabled(void)\n+{\n+\treturn false;\n+}\n+\n+static inline struct page kpkeys_pgtable_alloc(gfp_t gfp, unsigned int order)\n+{\n+\treturn NULL;\n+}\n+\n+static inline void kpkeys_pgtable_free(struct page page, unsigned int order) {}\n+\n+static inline void kpkeys_hardened_pgtables_init(void) {}\n+\n+static inline phys_addr_t kpkeys_physmem_pgtable_alloc(void)\n+{\n+\treturn 0;\n+}\n+\n+#endif / CONFIG_KPKEYS_HARDENED_PGTABLES /\n+\n+#endif / _LINUX_KPKEYS_H /\ndiff --git a/include/linux/kpkeys_types.h b/include/linux/kpkeys_types.h\nnew file mode 100644\nindex 0000000000000..09f15a9565cf3\n--- /dev/null\n+++ b/include/linux/kpkeys_types.h\n@@ -0,0 +1,30 @@\n+/ SPDX-License-Identifier: GPL-2.0-only /\n+#ifndef _LINUX_KPKEYS_TYPES_H\n+#define _LINUX_KPKEYS_TYPES_H\n+\n+#include \u003clinux/types.h\u003e\n+\n+#ifdef CONFIG_ARCH_HAS_KPKEYS\n+#include \u003casm/kpkeys_types.h\u003e\n+#endif\n+\n+#ifndef KPKEYS_PKEY_PGTABLES\n+#define KPKEYS_PKEY_PGTABLES\t1\n+#endif\n+\n+#if defined(CONFIG_ARCH_HAS_KPKEYS) \u0026\u0026 !defined(__ASSEMBLY__)\n+\n+enum kpkeys_ctx {\n+\tKPKEYS_CTX_DEFAULT = 0,\n+\tKPKEYS_CTX_PGTABLES,\n+\tKPKEYS_CTX_COUNT,\n+};\n+\n+struct kpkeys_state {\n+\tbool entered_context;\n+\tstruct arch_kpkeys_state arch_state;\n+};\n+\n+#endif / CONFIG_ARCH_HAS_KPKEYS \u0026\u0026 !__ASSEMBLY__ /\n+\n+#endif / _LINUX_KPKEYS_TYPES_H /\ndiff --git a/include/linux/memblock.h b/include/linux/memblock.h\nindex 5afcd99aa8c14..c61f80e31cc10 100644\n--- a/include/linux/memblock.h\n+++ b/include/linux/memblock.h\n@@ -24,6 +24,17 @@ extern unsigned long max_pfn;\n /\n extern unsigned long long max_possible_pfn;\n \n+#define INIT_MEMBLOCK_REGIONS\t\t\t128\n+#define INIT_PHYSMEM_REGIONS\t\t\t4\n+\n+#ifndef INIT_MEMBLOCK_RESERVED_REGIONS\n+#define INIT_MEMBLOCK_RESERVED_REGIONS\t\tINIT_MEMBLOCK_REGIONS\n+#endif\n+\n+#ifndef INIT_MEMBLOCK_MEMORY_REGIONS\n+#define INIT_MEMBLOCK_MEMORY_REGIONS\t\tINIT_MEMBLOCK_REGIONS\n+#endif\n+\n /*\n enum memblock_flags - definition of memory region attributes\n * @MEMBLOCK_NONE: no special request\ndiff --git a/include/linux/mm.h b/include/linux/mm.h\nindex 485df9c2dbddb..eea3131786b35 100644\n--- a/include/linux/mm.h\n+++ b/include/linux/mm.h\n@@ -37,6 +37,7 @@\n #include \u003clinux/bitmap.h\u003e\n #include \u003clinux/bitops.h\u003e\n #include \u003clinux/iommu-debug-pagealloc.h\u003e\n+#include \u003clinux/kpkeys.h\u003e\n \n struct mempolicy;\n struct anon_vma;\n@@ -3682,7 +3683,12 @@ static inline bool ptdesc_test_kernel(const struct ptdesc ptdesc)\n /\n static inline struct ptdesc pagetable_alloc_noprof(gfp_t gfp, unsigned int order)\n {\n-\tstruct page page = alloc_pages_noprof(gfp \| __GFP_COMP, order);\n+\tstruct page page;\n+\n+\tif (kpkeys_hardened_pgtables_enabled())\n+\t\tpage = kpkeys_pgtable_alloc(gfp \| __GFP_COMP, order);\n+\telse\n+\t\tpage = alloc_pages_noprof(gfp \| __GFP_COMP, order);\n \n \treturn page_ptdesc(page);\n }\n@@ -3691,8 +3697,12 @@ static inline struct ptdesc pagetable_alloc_noprof(gfp_t gfp, unsigned int orde\n static inline void __pagetable_free(struct ptdesc pt)\n {\n \tstruct page page = ptdesc_page(pt);\n+\tunsigned int order = compound_order(page);\n \n-\t__free_pages(page, compound_order(page));\n+\tif (kpkeys_hardened_pgtables_enabled())\n+\t\tkpkeys_pgtable_free(page, order);\n+\telse\n+\t\t__free_pages(page, order);\n }\n \n #ifdef CONFIG_ASYNC_KERNEL_PGTABLE_FREE\ndiff --git a/include/linux/set_memory.h b/include/linux/set_memory.h\nindex 3030d9245f5ac..7b3a8bfde3c64 100644\n--- a/include/linux/set_memory.h\n+++ b/include/linux/set_memory.h\n@@ -84,4 +84,11 @@ static inline int set_memory_decrypted(unsigned long addr, int numpages)\n }\n #endif /* CONFIG_ARCH_HAS_MEM_ENCRYPT /\n \n+#ifndef CONFIG_ARCH_HAS_KPKEYS\n+static inline int set_memory_pkey(unsigned long addr, int numpages, int pkey)\n+{\n+\treturn 0;\n+}\n+#endif\n+\n #endif / _LINUX_SET_MEMORY_H_ /\ndiff --git a/mm/Kconfig b/mm/Kconfig\nindex 9e0ca48249054..21bd3e550de83 100644\n--- a/mm/Kconfig\n+++ b/mm/Kconfig\n@@ -1281,6 +1281,11 @@ config ARCH_USES_HIGH_VMA_FLAGS\n \tbool\n config ARCH_HAS_PKEYS\n \tbool\n+config ARCH_HAS_KPKEYS\n+\tbool\n+config ARCH_HAS_KPKEYS_HARDENED_PGTABLES\n+\tbool\n+\tdepends on ARCH_HAS_KPKEYS\n \n config ARCH_USES_PG_ARCH_2\n \tbool\ndiff --git a/mm/Makefile b/mm/Makefile\nindex eff9f9e7e061c..e31e582948451 100644\n--- a/mm/Makefile\n+++ b/mm/Makefile\n@@ -147,3 +147,5 @@ obj-$(CONFIG_SHRINKER_DEBUG) += shrinker_debug.o\n obj-$(CONFIG_EXECMEM) += execmem.o\n obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o\n obj-$(CONFIG_LAZY_MMU_MODE_KUNIT_TEST) += tests/lazy_mmu_mode_kunit.o\n+obj-$(CONFIG_KPKEYS_HARDENED_PGTABLES) += kpkeys_hardened_pgtables.o\n+obj-$(CONFIG_KPKEYS_HARDENED_PGTABLES_KUNIT_TEST) += tests/kpkeys_hardened_pgtables_kunit.o\ndiff --git a/mm/kpkeys_hardened_pgtables.c b/mm/kpkeys_hardened_pgtables.c\nnew file mode 100644\nindex 0000000000000..c133e5896fccc\n--- /dev/null\n+++ b/mm/kpkeys_hardened_pgtables.c\n@@ -0,0 +1,179 @@\n+// SPDX-License-Identifier: GPL-2.0-only\n+#include \u003clinux/kpkeys.h\u003e\n+#include \u003clinux/memblock.h\u003e\n+#include \u003clinux/mm.h\u003e\n+#include \u003clinux/set_memory.h\u003e\n+\n+#include \u003ckunit/visibility.h\u003e\n+\n+__ro_after_init DEFINE_STATIC_KEY_FALSE(kpkeys_hardened_pgtables_key);\n+EXPORT_SYMBOL_IF_KUNIT(kpkeys_hardened_pgtables_key);\n+\n+static int set_pkey_pgtable(struct page page, unsigned int nr_pages)\n+{\n+\tunsigned long addr = (unsigned long)page_address(page);\n+\tint ret;\n+\n+\tret = set_memory_pkey(addr, nr_pages, KPKEYS_PKEY_PGTABLES);\n+\n+\tWARN_ON(ret);\n+\treturn ret;\n+}\n+\n+static int set_pkey_default(struct page page, unsigned int nr_pages)\n+{\n+\tunsigned long addr = (unsigned long)page_address(page);\n+\tint ret;\n+\n+\tret = set_memory_pkey(addr, nr_pages, 0);\n+\n+\tWARN_ON(ret);\n+\treturn ret;\n+}\n+\n+/ pkeys physmem allocator (PPA) - implemented below /\n+static void ppa_finalize(void);\n+\n+struct page kpkeys_pgtable_alloc(gfp_t gfp, unsigned int order)\n+{\n+\tstruct page page;\n+\tint ret;\n+\n+\tpage = alloc_pages_noprof(gfp, order);\n+\tif (!page)\n+\t\treturn page;\n+\n+\tret = set_pkey_pgtable(page, 1 \u003c\u003c order);\n+\tif (ret) {\n+\t\t__free_pages(page, order);\n+\t\treturn NULL;\n+\t}\n+\n+\treturn page;\n+}\n+\n+void kpkeys_pgtable_free(struct page page, unsigned int order)\n+{\n+\tset_pkey_default(page, 1 \u003c\u003c order);\n+\t__free_pages(page, order);\n+}\n+\n+void __init kpkeys_hardened_pgtables_init(void)\n+{\n+\tif (!kpkeys_supported())\n+\t\treturn;\n+\n+\tstatic_branch_enable(\u0026kpkeys_hardened_pgtables_key);\n+\n+\tppa_finalize();\n+\tarch_kpkeys_protect_static_pgtables();\n+}\n+\n+/\n+ pkeys physmem allocator (PPA): allocator for very early page tables\n+ * (especially for creating the linear map), based on memblock. Allocated\n+ * ranges are tracked so that their pkey can be set once it is safe to do so.\n+ /\n+\n+/\n+ * We may have to track many ranges when allocating page tables for the linear\n+ * map, as their number grows with the amount of available memory. Assuming that\n+ * memblock returns contiguous blocks whenever possible, the number of ranges\n+ * to track cannot however exceed the number of regions that memblock itself\n+ * tracks. memblock_allow_resize() hasn't been called yet at that point, so\n+ * that limit is the size of the statically allocated array.\n+ /\n+#define PHYSMEM_MAX_RANGES\tINIT_MEMBLOCK_MEMORY_REGIONS\n+\n+struct physmem_range {\n+\tphys_addr_t addr;\n+\tphys_addr_t size;\n+};\n+\n+struct pkeys_physmem_allocator {\n+\tstruct physmem_range allocated_ranges[PHYSMEM_MAX_RANGES];\n+\tunsigned int nr_allocated_ranges;\n+};\n+\n+static struct pkeys_physmem_allocator pkeys_physmem_allocator __initdata;\n+\n+static int __init set_pkey_pgtable_phys(phys_addr_t pa, phys_addr_t size)\n+{\n+\tunsigned long addr = (unsigned long)__va(pa);\n+\tint ret;\n+\n+\tret = set_memory_pkey(addr, size / PAGE_SIZE, KPKEYS_PKEY_PGTABLES);\n+\n+\tWARN_ON(ret);\n+\treturn ret;\n+}\n+\n+static bool __init ppa_try_extend_last_range(phys_addr_t addr, phys_addr_t size)\n+{\n+\tstruct pkeys_physmem_allocator ppa = \u0026pkeys_physmem_allocator;\n+\tstruct physmem_range range;\n+\n+\tif (!ppa-\u003enr_allocated_ranges)\n+\t\treturn false;\n+\n+\trange = \u0026ppa-\u003eallocated_ranges[ppa-\u003enr_allocated_ranges - 1];\n+\n+\t/ Merge the new range into the last range if they are contiguous /\n+\tif (addr == range-\u003eaddr + range-\u003esize) {\n+\t\trange-\u003esize += size;\n+\t\treturn true;\n+\t} else if (addr + size == range-\u003eaddr) {\n+\t\trange-\u003eaddr -= size;\n+\t\trange-\u003esize += size;\n+\t\treturn true;\n+\t}\n+\n+\treturn false;\n+}\n+\n+static void __init ppa_register_allocated_range(phys_addr_t addr,\n+\t\t\t\t\t\tphys_addr_t size)\n+{\n+\tstruct pkeys_physmem_allocator ppa = \u0026pkeys_physmem_allocator;\n+\tstruct physmem_range range;\n+\n+\tif (!addr)\n+\t\treturn;\n+\n+\tif (ppa_try_extend_last_range(addr, size))\n+\t\treturn;\n+\n+\t/ Could not extend the last range, create a new one /\n+\tif (WARN_ON(ppa-\u003enr_allocated_ranges \u003e= PHYSMEM_MAX_RANGES))\n+\t\treturn;\n+\n+\trange = \u0026ppa-\u003eallocated_ranges[ppa-\u003enr_allocated_ranges++];\n+\trange-\u003eaddr = addr;\n+\trange-\u003esize = size;\n+}\n+\n+static void __init ppa_finalize(void)\n+{\n+\tstruct pkeys_physmem_allocator ppa = \u0026pkeys_physmem_allocator;\n+\n+\tfor (unsigned int i = 0; i \u003c ppa-\u003enr_allocated_ranges; i++) {\n+\t\tstruct physmem_range range = \u0026ppa-\u003eallocated_ranges[i];\n+\n+\t\tset_pkey_pgtable_phys(range-\u003eaddr, range-\u003esize);\n+\t}\n+}\n+\n+phys_addr_t __ref kpkeys_physmem_pgtable_alloc(void)\n+{\n+\tsize_t size = PAGE_SIZE;\n+\tphys_addr_t addr;\n+\n+\taddr = memblock_phys_alloc_range(size, size, 0,\n+\t\t\t\t\t MEMBLOCK_ALLOC_NOLEAKTRACE);\n+\tif (!addr)\n+\t\treturn addr;\n+\n+\tppa_register_allocated_range(addr, size);\n+\n+\treturn addr;\n+}\ndiff --git a/mm/memblock.c b/mm/memblock.c\nindex 6349c48154f4b..3b497d3556bb0 100644\n--- a/mm/memblock.c\n+++ b/mm/memblock.c\n@@ -30,17 +30,6 @@\n \n #include \"internal.h\"\n \n-#define INIT_MEMBLOCK_REGIONS\t\t\t128\n-#define INIT_PHYSMEM_REGIONS\t\t\t4\n-\n-#ifndef INIT_MEMBLOCK_RESERVED_REGIONS\n-# define INIT_MEMBLOCK_RESERVED_REGIONS\t\tINIT_MEMBLOCK_REGIONS\n-#endif\n-\n-#ifndef INIT_MEMBLOCK_MEMORY_REGIONS\n-#define INIT_MEMBLOCK_MEMORY_REGIONS\t\tINIT_MEMBLOCK_REGIONS\n-#endif\n-\n /\n DOC: memblock overview\n \ndiff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c\nindex ebd3ac997f645..1fec89168ff10 100644\n--- a/mm/sparse-vmemmap.c\n+++ b/mm/sparse-vmemmap.c\n@@ -179,13 +179,29 @@ static pte_t __meminit vmemmap_pte_populate(pmd_t pmd, unsigned long addr, in\n \treturn pte;\n }\n \n-static void __meminit vmemmap_alloc_block_zero(unsigned long size, int node)\n+static void * __meminit vmemmap_alloc_pgtable(int node)\n {\n-\tvoid p = vmemmap_alloc_block(size, node);\n+\tvoid p;\n+\n+\tif (slab_is_available()) {\n+\t\tgfp_t gfp = GFP_KERNEL \| __GFP_ZERO;\n+\t\tstruct ptdesc ptdesc = pagetable_alloc(gfp, 0);\n+\n+\t\treturn ptdesc ? ptdesc_address(ptdesc) : NULL;\n+\t}\n+\n+\tif (kpkeys_hardened_pgtables_early_enabled()) {\n+\t\tphys_addr_t phys = kpkeys_physmem_pgtable_alloc();\n+\n+\t\tp = phys ? phys_to_virt(phys) : NULL;\n+\t} else {\n+\t\tp = __earlyonly_bootmem_alloc(node, PAGE_SIZE, PAGE_SIZE,\n+\t\t\t\t\t __pa(MAX_DMA_ADDRESS));\n+\t}\n \n \tif (!p)\n \t\treturn NULL;\n-\tmemset(p, 0, size);\n+\tmemset(p, 0, PAGE_SIZE);\n \n \treturn p;\n }\n@@ -194,7 +210,7 @@ static pmd_t __meminit vmemmap_pmd_populate(pud_t pud, unsigned long addr, in\n {\n \tpmd_t pmd = pmd_offset(pud, addr);\n \tif (pmd_none(pmd)) {\n-\t\tvoid p = vmemmap_alloc_block_zero(PAGE_SIZE, node);\n+\t\tvoid p = vmemmap_alloc_pgtable(node);\n \t\tif (!p)\n \t\t\treturn NULL;\n \t\tkernel_pte_init(p);\n@@ -207,7 +223,7 @@ static pud_t __meminit vmemmap_pud_populate(p4d_t p4d, unsigned long addr, in\n {\n \tpud_t pud = pud_offset(p4d, addr);\n \tif (pud_none(pud)) {\n-\t\tvoid p = vmemmap_alloc_block_zero(PAGE_SIZE, node);\n+\t\tvoid p = vmemmap_alloc_pgtable(node);\n \t\tif (!p)\n \t\t\treturn NULL;\n \t\tpmd_init(p);\n@@ -220,7 +236,7 @@ static p4d_t __meminit vmemmap_p4d_populate(pgd_t pgd, unsigned long addr, in\n {\n \tp4d_t p4d = p4d_offset(pgd, addr);\n \tif (p4d_none(p4d)) {\n-\t\tvoid p = vmemmap_alloc_block_zero(PAGE_SIZE, node);\n+\t\tvoid p = vmemmap_alloc_pgtable(node);\n \t\tif (!p)\n \t\t\treturn NULL;\n \t\tpud_init(p);\n@@ -233,7 +249,7 @@ static pgd_t __meminit vmemmap_pgd_populate(unsigned long addr, int node)\n {\n \tpgd_t pgd = pgd_offset_k(addr);\n \tif (pgd_none(pgd)) {\n-\t\tvoid p = vmemmap_alloc_block_zero(PAGE_SIZE, node);\n+\t\tvoid p = vmemmap_alloc_pgtable(node);\n \t\tif (!p)\n \t\t\treturn NULL;\n \t\tpgd_populate_kernel(addr, pgd, p);\n@@ -346,10 +362,11 @@ static __meminit struct page vmemmap_get_tail(unsigned int order, struct zone \n \t * memmap_init().\n \t /\n \n-\tp = vmemmap_alloc_block_zero(PAGE_SIZE, node);\n+\tp = vmemmap_alloc_block(PAGE_SIZE, node);\n \tif (!p)\n \t\treturn NULL;\n \n+\tmemset(p, 0, PAGE_SIZE);\n \ttail = virt_to_page(p);\n \tzone-\u003evmemmap_tails[idx] = tail;\n \ndiff --git a/mm/tests/kpkeys_hardened_pgtables_kunit.c b/mm/tests/kpkeys_hardened_pgtables_kunit.c\nnew file mode 100644\nindex 0000000000000..80d4f5144e48d\n--- /dev/null\n+++ b/mm/tests/kpkeys_hardened_pgtables_kunit.c\n@@ -0,0 +1,200 @@\n+// SPDX-License-Identifier: GPL-2.0-only\n+#include \u003ckunit/test.h\u003e\n+#include \u003clinux/mman.h\u003e\n+#include \u003clinux/pgtable.h\u003e\n+#include \u003clinux/set_memory.h\u003e\n+#include \u003clinux/vmalloc.h\u003e\n+\n+static void free_page_wrapper(void ctx)\n+{\n+\t__free_page((struct page )ctx);\n+}\n+\n+KUNIT_DEFINE_ACTION_WRAPPER(vfree_wrapper, vfree, const void );\n+\n+static pud_t pud_off_k(unsigned long va)\n+{\n+\treturn pud_offset(p4d_offset(pgd_offset_k(va), va), va);\n+}\n+\n+static pte_t get_kernel_pte(unsigned long addr)\n+{\n+\tpmd_t pmdp = pmd_off_k(addr);\n+\n+\tif (!pmdp \|\| pmd_leaf(pmdp))\n+\t\treturn NULL;\n+\n+\treturn pte_offset_kernel(pmdp, addr);\n+}\n+\n+#define write_pgtable(type, ptr) do {\t\t\t\t\t\\\n+\ttype##_t val;\t\t\t\t\t\t\t\\\n+\tint ret;\t\t\t\t\t\t\t\\\n+\t\t\t\t\t\t\t\t\t\\\n+\tpr_debug(\"%s: writing to \"#type\" at %px\\n\", __func__, (ptr));\t\\\n+\t\t\t\t\t\t\t\t\t\\\n+\tval = type##p_get(ptr);\t\t\t\t\t\t\\\n+\tret = copy_to_kernel_nofault(ptr, \u0026val, sizeof(val));\t\t\\\n+\tKUNIT_EXPECT_EQ_MSG(test, ret, -EFAULT,\t\t\t\t\\\n+\t\t\t \"Direct \"#type\" write wasn't prevented\");\t\\\n+} while (0)\n+\n+/\n+ Try to write linear map page tables, at every level. This is worthwhile\n+ * because those page table pages are obtained from different allocators:\n+ \n+ - Static memory (part of the kernel image) for PGD\n+ * - memblock for PUD and possibly PMD/PTE\n+ * - pagetable_alloc() (buddy allocator) for PMD/PTE if large block mappings are\n+ * used and the linear map is split after being created\n+ /\n+static void write_direct_map_pgtables(struct kunit test)\n+{\n+\tstruct page page;\n+\tunsigned long addr;\n+\tpgd_t pgdp;\n+\tp4d_t p4dp;\n+\tpud_t pudp;\n+\tpmd_t pmdp;\n+\tpte_t ptep;\n+\tint ret;\n+\n+\tif (!kpkeys_supported())\n+\t\tkunit_skip(test, \"kpkeys are not supported\");\n+\n+\tpage = alloc_page(GFP_KERNEL);\n+\tKUNIT_ASSERT_NOT_NULL(test, page);\n+\tret = kunit_add_action_or_reset(test, free_page_wrapper, page);\n+\tKUNIT_ASSERT_EQ(test, ret, 0);\n+\n+\t/* Ensure page is PTE-mapped (splitting the linear map if necessary) /\n+\tret = set_direct_map_invalid_noflush(page);\n+\tKUNIT_ASSERT_EQ(test, ret, 0);\n+\tret = set_direct_map_default_noflush(page);\n+\tKUNIT_ASSERT_EQ(test, ret, 0);\n+\n+\taddr = (unsigned long)page_address(page);\n+\n+\tpgdp = pgd_offset_k(addr);\n+\tKUNIT_ASSERT_NOT_NULL_MSG(test, pgdp, \"Failed to get PGD\");\n+\t/\n+\t * swapper_pg_dir is still writable at this stage, so don't check it.\n+\t * It is not protected by kpkeys_hardened_pgtables because it should be\n+\t * made read-only by mark_rodata_ro(). However since these\n+\t * KUnit tests are builtin, they are run before mark_rodata_ro() is\n+\t * called.\n+\t /\n+\n+\tp4dp = p4d_offset(pgdp, addr);\n+\tKUNIT_ASSERT_NOT_NULL_MSG(test, p4dp, \"Failed to get P4D\");\n+\tif (!mm_p4d_folded(\u0026init_mm))\n+\t\twrite_pgtable(p4d, p4dp);\n+\n+\tpudp = pud_offset(p4dp, addr);\n+\tKUNIT_ASSERT_NOT_NULL_MSG(test, pudp, \"Failed to get PUD\");\n+\tif (!mm_pud_folded(\u0026init_mm))\n+\t\twrite_pgtable(pud, pudp);\n+\n+\tpmdp = pmd_offset(pudp, addr);\n+\tKUNIT_ASSERT_NOT_NULL_MSG(test, pmdp, \"Failed to get PMD\");\n+\twrite_pgtable(pmd, pmdp);\n+\n+\tptep = pte_offset_kernel(pmdp, addr);\n+\tKUNIT_ASSERT_NOT_NULL_MSG(test, ptep, \"Failed to get PTE\");\n+\twrite_pgtable(pte, ptep);\n+}\n+\n+/ Worth checking since the kernel image is mapped with static page tables /\n+static void write_kernel_image_pud(struct kunit test)\n+{\n+\tpud_t pudp;\n+\n+\tif (!kpkeys_supported())\n+\t\tkunit_skip(test, \"kpkeys are not supported\");\n+\n+\t/ The kernel is probably block-mapped, check the PUD to be safe /\n+\tpudp = pud_off_k((unsigned long)\u0026init_mm);\n+\tKUNIT_ASSERT_NOT_NULL_MSG(test, pudp, \"Failed to get PUD\");\n+\n+\twrite_pgtable(pud, pudp);\n+}\n+\n+static void write_kernel_vmalloc_pte(struct kunit test)\n+{\n+\tvoid mem;\n+\tpte_t ptep;\n+\tint ret;\n+\n+\tif (!kpkeys_supported())\n+\t\tkunit_skip(test, \"kpkeys are not supported\");\n+\n+\tmem = vmalloc(PAGE_SIZE);\n+\tKUNIT_ASSERT_NOT_NULL(test, mem);\n+\tret = kunit_add_action_or_reset(test, vfree_wrapper, mem);\n+\tKUNIT_ASSERT_EQ(test, ret, 0);\n+\n+\t/* vmalloc() without VM_ALLOW_HUGE_VMAP is PTE-mapped /\n+\tptep = get_kernel_pte((unsigned long)mem);\n+\tKUNIT_ASSERT_NOT_NULL_MSG(test, ptep, \"Failed to get PTE\");\n+\n+\twrite_pgtable(pte, ptep);\n+}\n+\n+static void write_vmemmap_pmd(struct kunit test)\n+{\n+\tstruct page page;\n+\tpmd_t pmdp;\n+\n+\tif (!kpkeys_supported())\n+\t\tkunit_skip(test, \"kpkeys are not supported\");\n+\n+\t/\n+\t We just need the address of some struct page, so we can free the\n+\t * page right away.\n+\t /\n+\tpage = alloc_page(GFP_KERNEL);\n+\tKUNIT_ASSERT_NOT_NULL(test, page);\n+\t__free_page(page);\n+\n+\t/ vmemmap may use PMD block mappings /\n+\tpmdp = pmd_off_k((unsigned long)page);\n+\tKUNIT_ASSERT_NOT_NULL_MSG(test, pmdp, \"Failed to get PMD\");\n+\twrite_pgtable(pmd, pmdp);\n+}\n+\n+static void write_user_pmd(struct kunit test)\n+{\n+\tpmd_t pmdp;\n+\tunsigned long uaddr;\n+\n+\tif (!kpkeys_supported())\n+\t\tkunit_skip(test, \"kpkeys are not supported\");\n+\n+\tuaddr = kunit_vm_mmap(test, NULL, 0, PAGE_SIZE, PROT_READ,\n+\t\t\t MAP_ANONYMOUS \| MAP_PRIVATE \| MAP_POPULATE, 0);\n+\tKUNIT_ASSERT_NE_MSG(test, uaddr, 0, \"Could not create userspace mm\");\n+\n+\t/ We passed MAP_POPULATE so a PMD should already be allocated */\n+\tpmdp = pmd_off(current-\u003emm, uaddr);\n+\tKUNIT_ASSERT_NOT_NULL_MSG(test, pmdp, \"Failed to get PMD\");\n+\n+\twrite_pgtable(pmd, pmdp);\n+}\n+\n+static struct kunit_case kpkeys_hardened_pgtables_test_cases[] = {\n+\tKUNIT_CASE(write_direct_map_pgtables),\n+\tKUNIT_CASE(write_kernel_image_pud),\n+\tKUNIT_CASE(write_kernel_vmalloc_pte),\n+\tKUNIT_CASE(write_vmemmap_pmd),\n+\tKUNIT_CASE(write_user_pmd),\n+\t{}\n+};\n+\n+static struct kunit_suite kpkeys_hardened_pgtables_test_suite = {\n+\t.name = \"kpkeys_hardened_pgtables\",\n+\t.test_cases = kpkeys_hardened_pgtables_test_cases,\n+};\n+kunit_test_suite(kpkeys_hardened_pgtables_test_suite);\n+\n+MODULE_DESCRIPTION(\"Tests for the kpkeys_hardened_pgtables feature\");\n+MODULE_LICENSE(\"GPL\");\ndiff --git a/security/Kconfig.hardening b/security/Kconfig.hardening\nindex 6923036e1a2f5..ad6152961f8ee 100644\n--- a/security/Kconfig.hardening\n+++ b/security/Kconfig.hardening\n@@ -271,6 +271,30 @@ config BUG_ON_DATA_CORRUPTION\n \n \t If unsure, say N.\n \n+config KPKEYS_HARDENED_PGTABLES\n+\tbool \"Harden page tables using kernel pkeys\"\n+\tdepends on ARCH_HAS_KPKEYS_HARDENED_PGTABLES\n+\thelp\n+\t This option makes all page tables mostly read-only by\n+\t allocating them with a non-default protection key (pkey) and\n+\t only enabling write access to that pkey in routines that are\n+\t expected to write to page table entries.\n+\n+\t This option has no effect if the system does not support\n+\t kernel pkeys.\n+\n+config KPKEYS_HARDENED_PGTABLES_KUNIT_TEST\n+\tbool \"KUnit tests for kpkeys_hardened_pgtables\" if !KUNIT_ALL_TESTS\n+\tdepends on KPKEYS_HARDENED_PGTABLES\n+\tdepends on KUNIT=y\n+\tdefault KUNIT_ALL_TESTS\n+\thelp\n+\t Enable this option to check that the kpkeys_hardened_pgtables feature\n+\t functions as intended, i.e. prevents arbitrary writes to user and\n+\t kernel page tables.\n+\n+\t If unsure, say N.\n+\n endmenu\n \n config CC_HAS_RANDSTRUCT\n" }
2/1	2026/08/18 17:01	agent	patch-evaluator	4m Model: best-expensive Args: null Results: { "EnableConfigs": [ "KPKEYS_HARDENED_PGTABLES" ], "FocusSymbols": [ "kpkeys_pgtable_alloc", "kpkeys_pgtable_free", "kpkeys_physmem_pgtable_alloc", "set_memory_pkey", "arch_set_user_pkey_access" ], "Reasoning": "The patch introduces a new feature to harden page tables using kernel pkeys (kpkeys) on ARM64. It adds a new file `mm/kpkeys_hardened_pgtables.c` and modifies ARM64 memory management code to allocate page tables with a specific pkey and restrict write access to them. The changes include new page table allocation and freeing functions, as well as modifications to `arch_set_user_pkey_access`. This is a significant functional change to the core memory management subsystem that should be fuzzed to ensure memory safety and correctness.", "WorthFuzzing": true } Instruction: You are an expert Linux kernel maintainer. Your job is to review a provided patch series and determine if it makes functional changes to the kernel that should be fuzzed. IMPORTANT: The changes have ALREADY been applied and committed as the HEAD commit in your workspace. Do NOT rely on your internal knowledge of the kernel. You must actively use your code access tools to examine the actual source code and confirm any assumptions. Return WorthFuzzing=false if the patch only contains: - Modifications to Documentation/, Kconfig files, or code comments. - Purely decorative changes, such as logging (e.g., pr_err, printk) or tracepoints. - Changes to numeric constants or macros that do not functionally alter execution flow. - Code paths that are impossible to reach in virtualized environments like GCE or QEMU, even when utilizing software-emulated hardware (e.g., usb gadget, mac80211_hwsim). - Code in vendor-specific PCIe switch, SmartNIC, or GPU drivers (e.g., mlxsw, pds_core, qed, ionic, amdgpu) that require physical PCIe hardware cards not emulated in standard QEMU. - Driver .remove, .shutdown, or pci_unregister_driver teardown callbacks (e.g., igb_remove) that are executed only during PCI hot-unplug or sysfs driver unbind operations. If it modifies reachable core kernel logic, drivers, or architectures, use your code search tools to verify the code can be executed, then return WorthFuzzing=true. When returning WorthFuzzing=true, you MUST ALSO: 1. Extract any specific kernel functions that should be heavily fuzzed into FocusSymbols. Avoid listing generic hot-path functions to prevent skewed test distributions. Prefer non-static, non-inlined API entrypoint functions over internal static helper functions (which are inlined by the compiler and do not have distinct symbol addresses). 2. Identify any specific CONFIG_ options required to properly test this new/modified feature. Go and look into the Kconfig files and check for ifdefs around the code, do not make assumptions. Also check "depends on" lines in Kconfig to include any non-standard parent subsystem configs needed for Kbuild to compile the code statically into vmlinux. List them in the EnableConfigs output array, and DO NOT add a 'CONFIG_' prefix (e.g., return "NET_IPV4" instead of "CONFIG_NET_IPV4"). Prefer calling several tools at the same time to save round-trips. Use set-results tool to provide results of the analysis. It must be called exactly once before the final reply. Ignore results of this tool. Prompt: For your convenience, here is the diff of the changes: commit bd271568204405923f3b37fcb4ded599be77ced1 Author: syz-cluster <triage@syzkaller.com> Date: Tue Aug 18 17:01:35 2026 +0000 syz-cluster: applied patch under review diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index b3afe0688919b..ac9faec1a0184 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -2202,6 +2202,8 @@ config ARM64_POE def_bool y select ARCH_USES_HIGH_VMA_FLAGS select ARCH_HAS_PKEYS + select ARCH_HAS_KPKEYS + select ARCH_HAS_KPKEYS_HARDENED_PGTABLES help The Permission Overlay Extension is used to implement Memory Protection Keys. Memory Protection Keys provides a mechanism for diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index a57870fa96db5..37aae60c2d7f0 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -1085,6 +1085,18 @@ static inline bool cpu_has_lpa2(void) #endif } +static inline bool cpu_has_poe(void) +{ + u64 mmfr3; + + if (!IS_ENABLED(CONFIG_ARM64_POE)) + return false; + + mmfr3 = read_sysreg_s(SYS_ID_AA64MMFR3_EL1); + return cpuid_feature_extract_unsigned_field(mmfr3, + ID_AA64MMFR3_EL1_S1POE_SHIFT); +} + #endif /* __ASSEMBLER__ / #endif diff --git a/arch/arm64/include/asm/kpkeys.h b/arch/arm64/include/asm/kpkeys.h new file mode 100644 index 0000000000000..7747af54fd386 --- /dev/null +++ b/arch/arm64/include/asm/kpkeys.h @@ -0,0 +1,84 @@ +/ SPDX-License-Identifier: GPL-2.0-only / +#ifndef __ASM_KPKEYS_H +#define __ASM_KPKEYS_H + +#include <linux/kpkeys_types.h> + +#include <asm/barrier.h> +#include <asm/cpufeature.h> +#include <asm/por.h> + +/ + * Equivalent to por_set_kpkeys_context(0, KPKEYS_CTX_DEFAULT), but can also be + * used in assembly. + / +#define POR_EL1_INIT (POR_ELx_PERM_PREP(0, POE_RWX) \| \ + POR_ELx_PERM_PREP(KPKEYS_PKEY_PGTABLES, POE_R)) + +#ifndef __ASSEMBLY__ + +static inline bool arch_supports_kpkeys(void) +{ + return system_supports_poe(); +} + +static inline bool arch_supports_kpkeys_early(void) +{ + / POE is a boot feature / + return boot_capabilities_finalized() ? + system_supports_poe() : cpu_has_poe(); +} + +#ifdef CONFIG_ARM64_POE + +static inline u64 por_set_kpkeys_context(u64 por, enum kpkeys_ctx ctx) +{ + por = por_elx_set_pkey_perms(por, 0, POE_RWX); + por = por_elx_set_pkey_perms(por, KPKEYS_PKEY_PGTABLES, + ctx == KPKEYS_CTX_PGTABLES ? POE_RW : POE_R); + + return por; +} + +static __always_inline void __kpkeys_set_pkey_reg_nosync(u64 pkey_reg) +{ + write_sysreg_s(pkey_reg, SYS_POR_EL1); +} + +static __always_inline +struct kpkeys_state arch_kpkeys_enter_context(enum kpkeys_ctx ctx) +{ + const u64 prev_por = read_sysreg_s(SYS_POR_EL1); + const u64 new_por = por_set_kpkeys_context(prev_por, ctx); + + if (prev_por == new_por) + return (struct kpkeys_state) { .entered_context = false }; + + __kpkeys_set_pkey_reg_nosync(new_por); + isb(); + + return (struct kpkeys_state) { + .entered_context = true, + .arch_state.por = prev_por, + }; +} + +static __always_inline +void arch_kpkeys_leave_context(const struct kpkeys_state state) +{ + __kpkeys_set_pkey_reg_nosync(state->arch_state.por); + isb(); +} + +#endif /* CONFIG_ARM64_POE / + +#ifdef CONFIG_KPKEYS_HARDENED_PGTABLES + +#define arch_kpkeys_protect_static_pgtables arch_kpkeys_protect_static_pgtables +void arch_kpkeys_protect_static_pgtables(void); + +#endif / CONFIG_KPKEYS_HARDENED_PGTABLES / + +#endif / __ASSEMBLY__ / + +#endif / __ASM_KPKEYS_H / diff --git a/arch/arm64/include/asm/kpkeys_types.h b/arch/arm64/include/asm/kpkeys_types.h new file mode 100644 index 0000000000000..e9c59a26ee7a9 --- /dev/null +++ b/arch/arm64/include/asm/kpkeys_types.h @@ -0,0 +1,15 @@ +/ SPDX-License-Identifier: GPL-2.0-only / +#ifndef __ASM_KPKEYS_TYPES_H +#define __ASM_KPKEYS_TYPES_H + +#include <linux/types.h> + +#ifndef __ASSEMBLY__ + +struct arch_kpkeys_state { + u64 por; +}; + +#endif / __ASSEMBLY__ / + +#endif / __ASM_KPKEYS_TYPES_H / diff --git a/arch/arm64/include/asm/pgtable-prot.h b/arch/arm64/include/asm/pgtable-prot.h index 212ce1b02e15e..d4d45ab86a5a8 100644 --- a/arch/arm64/include/asm/pgtable-prot.h +++ b/arch/arm64/include/asm/pgtable-prot.h @@ -183,9 +183,9 @@ static inline bool __pure lpa2_is_enabled(void) PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_SHARED_EXEC), PIE_RW) \| \ PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_READONLY), PIE_R) \| \ PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_SHARED), PIE_RW) \| \ - PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL_ROX), PIE_RX) \| \ - PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL_EXEC), PIE_RWX) \| \ - PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL_RO), PIE_R) \| \ - PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL), PIE_RW)) + PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL_ROX), PIE_RX_O) \| \ + PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL_EXEC), PIE_RWX_O) \| \ + PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL_RO), PIE_R_O) \| \ + PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL), PIE_RW_O)) #endif / __ASM_PGTABLE_PROT_H / diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 27689c62bd25d..6e5d14c7e8b8f 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -39,6 +39,48 @@ #include <linux/mm_types.h> #include <linux/sched.h> #include <linux/page_table_check.h> +#include <linux/kpkeys.h> + +#ifdef CONFIG_KPKEYS_HARDENED_PGTABLES +KPKEYS_GUARD_COND(kpkeys_hardened_pgtables, KPKEYS_CTX_PGTABLES, + kpkeys_hardened_pgtables_enabled() && + !is_lazy_mmu_mode_active()) + +static inline void kpkeys_lazy_mmu_enter(void) +{ + if (!kpkeys_hardened_pgtables_enabled()) + return; + + current->thread.kpkeys_state_lazy_mmu = + kpkeys_enter_context(KPKEYS_CTX_PGTABLES); +} + +static inline void kpkeys_lazy_mmu_exit(void) +{ + const struct kpkeys_state state = + &current->thread.kpkeys_state_lazy_mmu; + + if (!kpkeys_hardened_pgtables_enabled()) + return; + + /* + * We skip any barrier if TIF_LAZY_MMU_PENDING is set: + * emit_pte_barriers() will issue an ISB just after this function + * returns. + / + if (test_thread_flag(TIF_LAZY_MMU_PENDING) && state->entered_context) + __kpkeys_set_pkey_reg_nosync(state->arch_state.por); + else + kpkeys_leave_context(state); +} +#else / CONFIG_KPKEYS_HARDENED_PGTABLES / +KPKEYS_GUARD_NOOP(kpkeys_hardened_pgtables) + +static inline void kpkeys_lazy_mmu_enter(void) {} +static inline void kpkeys_lazy_mmu_exit(void) {} +#endif / CONFIG_KPKEYS_HARDENED_PGTABLES / + + static inline void emit_pte_barriers(void) { @@ -71,7 +113,10 @@ static inline void queue_pte_barriers(void) } } -static inline void arch_enter_lazy_mmu_mode(void) {} +static inline void arch_enter_lazy_mmu_mode(void) +{ + kpkeys_lazy_mmu_enter(); +} static inline void arch_flush_lazy_mmu_mode(void) { @@ -81,6 +126,11 @@ static inline void arch_flush_lazy_mmu_mode(void) static inline void arch_leave_lazy_mmu_mode(void) { + / + * The ordering should be preserved to allow kpkeys_lazy_mmu_exit() + * to skip any barrier when TIF_LAZY_MMU_PENDING is set. + / + kpkeys_lazy_mmu_exit(); arch_flush_lazy_mmu_mode(); } @@ -359,6 +409,7 @@ static inline pte_t pte_clear_uffd_wp(pte_t pte) static inline void __set_pte_nosync(pte_t ptep, pte_t pte) { + guard(kpkeys_hardened_pgtables)(); WRITE_ONCE(ptep, pte); } @@ -830,6 +881,7 @@ static inline void set_pmd(pmd_t pmdp, pmd_t pmd) } #endif /* __PAGETABLE_PMD_FOLDED / + guard(kpkeys_hardened_pgtables)(); WRITE_ONCE(pmdp, pmd); if (pmd_valid(pmd)) @@ -894,6 +946,7 @@ static inline void set_pud(pud_t pudp, pud_t pud) return; } + guard(kpkeys_hardened_pgtables)(); WRITE_ONCE(pudp, pud); if (pud_valid(pud)) @@ -975,6 +1028,7 @@ static inline void set_p4d(p4d_t p4dp, p4d_t p4d) return; } + guard(kpkeys_hardened_pgtables)(); WRITE_ONCE(p4dp, p4d); queue_pte_barriers(); } @@ -1103,6 +1157,7 @@ static inline void set_pgd(pgd_t pgdp, pgd_t pgd) return; } + guard(kpkeys_hardened_pgtables)(); WRITE_ONCE(pgdp, pgd); queue_pte_barriers(); } @@ -1307,6 +1362,7 @@ static inline bool __ptep_test_and_clear_young(struct vm_area_struct vma, { pte_t old_pte, pte; + guard(kpkeys_hardened_pgtables)(); pte = __ptep_get(ptep); do { old_pte = pte; @@ -1354,7 +1410,10 @@ static inline pte_t __ptep_get_and_clear_anysz(struct mm_struct mm, pte_t ptep, unsigned long pgsize) { - pte_t pte = __pte(xchg_relaxed(&pte_val(ptep), 0)); + pte_t pte; + + scoped_guard(kpkeys_hardened_pgtables) + pte = __pte(xchg_relaxed(&pte_val(ptep), 0)); switch (pgsize) { case PAGE_SIZE: @@ -1427,6 +1486,7 @@ static inline void ___ptep_set_wrprotect(struct mm_struct mm, { pte_t old_pte; + guard(kpkeys_hardened_pgtables)(); do { old_pte = pte; pte = pte_wrprotect(pte); @@ -1460,6 +1520,7 @@ static inline void __clear_young_dirty_pte(struct vm_area_struct vma, { pte_t old_pte; + guard(kpkeys_hardened_pgtables)(); do { old_pte = pte; @@ -1507,6 +1568,7 @@ static inline pmd_t pmdp_establish(struct vm_area_struct vma, unsigned long address, pmd_t pmdp, pmd_t pmd) { page_table_check_pmd_set(vma->vm_mm, address, pmdp, pmd); + guard(kpkeys_hardened_pgtables)(); return __pmd(xchg_relaxed(&pmd_val(pmdp), pmd_val(pmd))); } #endif diff --git a/arch/arm64/include/asm/por.h b/arch/arm64/include/asm/por.h index d913d5b529e49..dddabf6ffae6d 100644 --- a/arch/arm64/include/asm/por.h +++ b/arch/arm64/include/asm/por.h @@ -10,6 +10,8 @@ #define POR_EL0_INIT POR_ELx_PERM_PREP(0, POE_RWX) +#ifndef __ASSEMBLY__ + static inline bool por_elx_allows_read(u64 por, u8 pkey) { u8 perm = POR_ELx_PERM_GET(pkey, por); @@ -31,4 +33,13 @@ static inline bool por_elx_allows_exec(u64 por, u8 pkey) return perm & POE_X; } +static inline u64 por_elx_set_pkey_perms(u64 por, u8 pkey, u8 perms) +{ + u64 shift = POR_ELx_PERM_SHIFT(pkey); + + return (por & ~(POE_MASK << shift)) \| ((u64)perms << shift); +} + +#endif /* __ASSEMBLY__ / + #endif / _ASM_ARM64_POR_H / diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h index c2a627f393144..23660a233da28 100644 --- a/arch/arm64/include/asm/processor.h +++ b/arch/arm64/include/asm/processor.h @@ -30,6 +30,7 @@ #include <linux/build_bug.h> #include <linux/cache.h> #include <linux/init.h> +#include <linux/kpkeys_types.h> #include <linux/stddef.h> #include <linux/string.h> #include <linux/thread_info.h> @@ -195,6 +196,10 @@ struct thread_struct { u64 svcr; u64 tpidr2_el0; u64 por_el0; + u64 por_el1; +#ifdef CONFIG_KPKEYS_HARDENED_PGTABLES + struct kpkeys_state kpkeys_state_lazy_mmu; +#endif #ifdef CONFIG_ARM64_GCS unsigned int gcs_el0_mode; unsigned int gcs_el0_locked; diff --git a/arch/arm64/include/asm/sections.h b/arch/arm64/include/asm/sections.h index 51b0d594239eb..dc961cf67168c 100644 --- a/arch/arm64/include/asm/sections.h +++ b/arch/arm64/include/asm/sections.h @@ -23,6 +23,7 @@ extern char __irqentry_text_start[], __irqentry_text_end[]; extern char __mmuoff_data_start[], __mmuoff_data_end[]; extern char __entry_tramp_text_start[], __entry_tramp_text_end[]; extern char __relocate_new_kernel_start[], __relocate_new_kernel_end[]; +extern char __pi_init_pg_dir[], __pi_init_pg_end[]; static inline size_t entry_tramp_text_size(void) { diff --git a/arch/arm64/include/asm/set_memory.h b/arch/arm64/include/asm/set_memory.h index 90f61b17275e1..b6cd6de34abf1 100644 --- a/arch/arm64/include/asm/set_memory.h +++ b/arch/arm64/include/asm/set_memory.h @@ -19,4 +19,8 @@ bool kernel_page_present(struct page page); int set_memory_encrypted(unsigned long addr, int numpages); int set_memory_decrypted(unsigned long addr, int numpages); +#ifdef CONFIG_ARCH_HAS_KPKEYS +int set_memory_pkey(unsigned long addr, int numpages, int pkey); +#endif + #endif /* _ASM_ARM64_SET_MEMORY_H / diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 9a22df0c5120f..4f2f54007ed6f 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -76,6 +76,7 @@ #include <linux/kasan.h> #include <linux/percpu.h> #include <linux/sched/isolation.h> +#include <linux/kpkeys.h> #include <asm/arm_pmuv3.h> #include <asm/cpu.h> @@ -2463,8 +2464,10 @@ static void cpu_enable_mops(const struct arm64_cpu_capabilities __unused) #ifdef CONFIG_ARM64_POE static void cpu_enable_poe(const struct arm64_cpu_capabilities __unused) { - sysreg_clear_set(REG_TCR2_EL1, 0, TCR2_EL1_E0POE); + write_sysreg_s(POR_EL1_INIT, SYS_POR_EL1); + sysreg_clear_set(REG_TCR2_EL1, 0, TCR2_EL1_E0POE \| TCR2_EL1_POE); sysreg_clear_set(CPACR_EL1, 0, CPACR_EL1_E0POE); + isb(); } #endif diff --git a/arch/arm64/kernel/pi/map_kernel.c b/arch/arm64/kernel/pi/map_kernel.c index fb44cbdd2f292..81ca3e98f0f7b 100644 --- a/arch/arm64/kernel/pi/map_kernel.c +++ b/arch/arm64/kernel/pi/map_kernel.c @@ -92,8 +92,13 @@ static void __init map_kernel(u64 kaslr_offset, u64 va_offset, int root_level) __inittext_end, prot, false, root_level); map_segment(init_pg_dir, &pgdp, va_offset, __initdata_begin, __initdata_end, data_prot, false, root_level); - map_segment(init_pg_dir, &pgdp, va_offset, _data, _end, data_prot, - true, root_level); + map_segment(init_pg_dir, &pgdp, va_offset, _data, init_pg_dir, + data_prot, true, root_level); + / Map init_pg_dir separately as we will remap it read-only later on / + map_segment(init_pg_dir, &pgdp, va_offset, init_pg_dir, init_pg_end, + data_prot, true, root_level); + map_segment(init_pg_dir, &pgdp, va_offset, init_pg_end, _end, + data_prot, true, root_level); dsb(ishst); idmap_cpu_replace_ttbr1((phys_addr_t)init_pg_dir); diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c index 581f80e9b9b73..1dec143decf15 100644 --- a/arch/arm64/kernel/process.c +++ b/arch/arm64/kernel/process.c @@ -51,6 +51,7 @@ #include <asm/exec.h> #include <asm/fpsimd.h> #include <asm/gcs.h> +#include <asm/kpkeys.h> #include <asm/mmu_context.h> #include <asm/mpam.h> #include <asm/mte.h> @@ -431,6 +432,9 @@ int copy_thread(struct task_struct p, const struct kernel_clone_args args) ptrauth_thread_init_kernel(p); + if (system_supports_poe()) + p->thread.por_el1 = POR_EL1_INIT; + if (likely(!args->fn)) { childregs = current_pt_regs(); childregs->regs[0] = 0; @@ -681,6 +685,12 @@ static void permission_overlay_switch(struct task_struct next) * of POR_EL0. / } + + current->thread.por_el1 = read_sysreg_s(SYS_POR_EL1); + if (current->thread.por_el1 != next->thread.por_el1) { + write_sysreg_s(next->thread.por_el1, SYS_POR_EL1); + isb(); + } } / diff --git a/arch/arm64/kernel/sleep.S b/arch/arm64/kernel/sleep.S index f093cdf71be11..e0a6ad85cd242 100644 --- a/arch/arm64/kernel/sleep.S +++ b/arch/arm64/kernel/sleep.S @@ -3,6 +3,7 @@ #include <linux/linkage.h> #include <asm/asm-offsets.h> #include <asm/assembler.h> +#include <asm/kpkeys.h> #include <asm/smp.h> .text @@ -134,6 +135,17 @@ SYM_FUNC_START(_cpu_resume) /* load sp from context / ldr x2, [x0, #CPU_CTX_SP] mov sp, x2 + +#ifdef CONFIG_ARM64_POE +alternative_if_not ARM64_HAS_S1POE + b .Lskip_por_set +alternative_else_nop_endif + mov_q x2, POR_EL1_INIT + msr_s SYS_POR_EL1, x2 + / isb can be skipped since cpu_do_resume() will do it. / +.Lskip_por_set: +#endif / CONFIG_ARM64_POE / + / * cpu_do_resume expects x0 to contain context address pointer / diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 0b52557652be6..a853a8d4e61af 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -229,13 +229,16 @@ int __ptep_set_access_flags_anysz(struct vm_area_struct vma, / pte_val(entry) ^= PTE_RDONLY; pteval = pte_val(pte); - do { - old_pteval = pteval; - pteval ^= PTE_RDONLY; - pteval \|= pte_val(entry); - pteval ^= PTE_RDONLY; - pteval = cmpxchg_relaxed(&pte_val(ptep), old_pteval, pteval); - } while (pteval != old_pteval); + scoped_guard(kpkeys_hardened_pgtables) { + do { + old_pteval = pteval; + pteval ^= PTE_RDONLY; + pteval \|= pte_val(entry); + pteval ^= PTE_RDONLY; + pteval = cmpxchg_relaxed(&pte_val(ptep), old_pteval, + pteval); + } while (pteval != old_pteval); + } / * Invalidate the local stale read-only entry. Remote stale entries diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c index 97987f850a33c..7985045875cf6 100644 --- a/arch/arm64/mm/init.c +++ b/arch/arm64/mm/init.c @@ -386,6 +386,7 @@ bool page_alloc_available __ro_after_init; void __init mem_init(void) { page_alloc_available = true; + kpkeys_hardened_pgtables_init(); swiotlb_update_mem_attributes(); } diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index d4de88770ecfe..472ad0f8f002c 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -107,8 +107,11 @@ static phys_addr_t __init early_pgtable_alloc(enum pgtable_level pgtable_level) { phys_addr_t phys; - phys = memblock_phys_alloc_range(PAGE_SIZE, PAGE_SIZE, 0, - MEMBLOCK_ALLOC_NOLEAKTRACE); + if (kpkeys_hardened_pgtables_early_enabled()) + phys = kpkeys_physmem_pgtable_alloc(); + else + phys = memblock_phys_alloc_range(PAGE_SIZE, PAGE_SIZE, 0, + MEMBLOCK_ALLOC_NOLEAKTRACE); if (!phys) panic("Failed to allocate page table page\n"); @@ -780,7 +783,7 @@ static inline bool force_pte_mapping(void) const bool bbml2 = system_capabilities_finalized() ? system_supports_bbml2_noabort() : cpu_supports_bbml2_noabort(); - if (debug_pagealloc_enabled()) + if (debug_pagealloc_enabled() \|\| kpkeys_hardened_pgtables_early_enabled()) return true; if (bbml2) return false; @@ -1070,6 +1073,18 @@ void __init mark_linear_text_alias_ro(void) PAGE_KERNEL_RO); } +#ifdef CONFIG_KPKEYS_HARDENED_PGTABLES +void __init arch_kpkeys_protect_static_pgtables(void) +{ + unsigned long addr = (unsigned long)lm_alias(__pi_init_pg_dir); + unsigned long size = __pi_init_pg_end - __pi_init_pg_dir; + int ret; + + ret = set_memory_pkey(addr, size / PAGE_SIZE, KPKEYS_PKEY_PGTABLES); + WARN_ON(ret); +} +#endif /* CONFIG_KPKEYS_HARDENED_PGTABLES / + #ifdef CONFIG_KFENCE bool __ro_after_init kfence_early_init = !!CONFIG_KFENCE_SAMPLE_INTERVAL; @@ -1214,6 +1229,14 @@ void mark_rodata_ro(void) update_mapping_prot(__pa_symbol(_text), (unsigned long)_text, (unsigned long)_stext - (unsigned long)_text, PAGE_KERNEL_RO); + / + * Map the kernel image mapping of init_pg_dir read-only; it should + * only be written via the linear map. + / + section_size = (unsigned long)__pi_init_pg_end - (unsigned long)__pi_init_pg_dir; + update_mapping_prot(__pa_symbol(__pi_init_pg_dir), + (unsigned long)__pi_init_pg_dir, + section_size, PAGE_KERNEL_RO); / Map the kernel data/bss read-only in the linear map / update_mapping_prot(__pa_symbol(__init_end), @@ -1446,7 +1469,7 @@ static void free_hotplug_page_range(struct page page, size_t size, static void free_hotplug_pgtable_page(struct page page) { pagetable_dtor(page_ptdesc(page)); - free_hotplug_page_range(page, PAGE_SIZE, NULL); + pagetable_free(page_ptdesc(page)); } static bool pgtable_range_aligned(unsigned long start, unsigned long end, @@ -2305,8 +2328,8 @@ void __cpu_replace_ttbr1(pgd_t pgdp, bool cnp) #ifdef CONFIG_ARCH_HAS_PKEYS int arch_set_user_pkey_access(int pkey, unsigned long init_val) { - u64 new_por; - u64 old_por; + u8 new_perms; + u64 por; if (!system_supports_poe()) return -ENOSPC; @@ -2320,25 +2343,19 @@ int arch_set_user_pkey_access(int pkey, unsigned long init_val) return -EINVAL; /* Set the bits we need in POR: / - new_por = POE_RWX; + new_perms = POE_RWX; if (init_val & PKEY_DISABLE_WRITE) - new_por &= ~POE_W; + new_perms &= ~POE_W; if (init_val & PKEY_DISABLE_ACCESS) - new_por &= ~POE_RW; + new_perms &= ~POE_RW; if (init_val & PKEY_DISABLE_READ) - new_por &= ~POE_R; + new_perms &= ~POE_R; if (init_val & PKEY_DISABLE_EXECUTE) - new_por &= ~POE_X; - - / Shift the bits in to the correct place in POR for pkey: / - new_por = POR_ELx_PERM_PREP(pkey, new_por); - - / Get old POR and mask off any old bits in place: / - old_por = read_sysreg_s(SYS_POR_EL0); - old_por &= ~(POE_MASK << POR_ELx_PERM_SHIFT(pkey)); + new_perms &= ~POE_X; - / Write old part along with new part: / - write_sysreg_s(old_por \| new_por, SYS_POR_EL0); + por = read_sysreg_s(SYS_POR_EL0); + por = por_elx_set_pkey_perms(por, pkey, new_perms); + write_sysreg_s(por, SYS_POR_EL0); return 0; } diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c index bbe98ac9ad8c6..46fbebfc51003 100644 --- a/arch/arm64/mm/pageattr.c +++ b/arch/arm64/mm/pageattr.c @@ -9,6 +9,8 @@ #include <linux/sched.h> #include <linux/vmalloc.h> #include <linux/pagewalk.h> +#include <linux/pkeys.h> +#include <linux/kpkeys.h> #include <asm/cacheflush.h> #include <asm/pgtable-prot.h> @@ -99,7 +101,8 @@ bool can_set_direct_map(void) Realms need to make pages shared/protected at page granularity. / return rodata_full \|\| debug_pagealloc_enabled() \|\| - arm64_kfence_can_set_direct_map() \|\| is_realm_world(); + arm64_kfence_can_set_direct_map() \|\| is_realm_world() \|\| + kpkeys_hardened_pgtables_early_enabled(); } static int update_range_prot(unsigned long start, unsigned long size, @@ -365,6 +368,30 @@ int set_direct_map_valid_noflush(struct page page, unsigned nr, bool valid) return set_memory_valid(addr, nr, valid); } +#ifdef CONFIG_ARCH_HAS_KPKEYS +int set_memory_pkey(unsigned long addr, int numpages, int pkey) +{ + unsigned long set_prot = 0; + + if (!kpkeys_supported()) + return 0; + + if (!__is_lm_address(kasan_reset_tag((void )addr))) + return -EINVAL; + + if (pkey < 0 \|\| pkey >= arch_max_pkey()) + return -EINVAL; + + set_prot \|= pkey & BIT(0) ? PTE_PO_IDX_0 : 0; + set_prot \|= pkey & BIT(1) ? PTE_PO_IDX_1 : 0; + set_prot \|= pkey & BIT(2) ? PTE_PO_IDX_2 : 0; + + return __change_memory_common(addr, PAGE_SIZE numpages, + __pgprot(set_prot), + __pgprot(PTE_PO_IDX_MASK)); +} +#endif + #ifdef CONFIG_DEBUG_PAGEALLOC /* * This is - apart from the return value - doing the same diff --git a/include/linux/kpkeys.h b/include/linux/kpkeys.h new file mode 100644 index 0000000000000..df55ec854586e --- /dev/null +++ b/include/linux/kpkeys.h @@ -0,0 +1,162 @@ +/* SPDX-License-Identifier: GPL-2.0-only / +#ifndef _LINUX_KPKEYS_H +#define _LINUX_KPKEYS_H + +#include <linux/bug.h> +#include <linux/cleanup.h> +#include <linux/jump_label.h> +#include <linux/kpkeys_types.h> + +/* + * KPKEYS_GUARD_NOOP() - define a guard type that does nothing + * @name: the name of the guard type + * + * Define a guard type that does nothing, useful to match a real guard type + * that is defined under an #ifdef. + / +#define KPKEYS_GUARD_NOOP(name) \ + __DEFINE_CLASS_IS_CONDITIONAL(name, false); \ + DEFINE_CLASS(name, bool, (void)_T, false, void); \ + static inline void class_##name##_lock_ptr(bool _T) \ + { return _T; } + +#ifdef CONFIG_ARCH_HAS_KPKEYS + +#include <asm/kpkeys.h> + +/* + * KPKEYS_GUARD_COND() - define a guard type that conditionally switches to + * a given kpkeys context + * @name: the name of the guard type + * @ctx: the kpkeys context to switch to + * @cond: an expression that is evaluated as condition + * + * Define a guard type that switches to @ctx if @cond evaluates to true, + * and does nothing otherwise. + / +#define KPKEYS_GUARD_COND(name, ctx, cond) \ + __DEFINE_CLASS_IS_CONDITIONAL(name, false); \ + DEFINE_CLASS(name, struct kpkeys_state, \ + kpkeys_leave_context(&_T), \ + (cond) ? kpkeys_enter_context(ctx) : \ + (struct kpkeys_state) {}, void); \ + static inline \ + void class_##name##_lock_ptr(struct kpkeys_state _T) \ + { return _T; } + +/* + * KPKEYS_GUARD() - define a guard type that switches to a given kpkeys context + * if kpkeys are supported + * @name: the name of the guard type + * @ctx: the kpkeys context to switch to + * + * Define a guard type that switches to @ctx if the system supports kpkeys. + / +#define KPKEYS_GUARD(name, ctx) \ + KPKEYS_GUARD_COND(name, ctx, kpkeys_supported()) + +/* + * kpkeys_enter_context() - enter a kpkeys context + * @ctx: the context to switch to + * + * Enters the specified kpkeys context. @ctx must be a compile-time constant. + * + * Return: state to be passed to kpkeys_leave_context(). + / +static __always_inline +struct kpkeys_state kpkeys_enter_context(enum kpkeys_ctx ctx) +{ + BUILD_BUG_ON_MSG(!__builtin_constant_p(ctx), + "kpkeys_enter_context() only takes constant values"); + BUILD_BUG_ON_MSG(ctx < 0 \|\| ctx >= KPKEYS_CTX_COUNT, + "Invalid value passed to kpkeys_enter_context()"); + + return arch_kpkeys_enter_context(ctx); +} + +/* + * kpkeys_leave_context() - leave a kpkeys context + * @state: state returned by kpkeys_enter_context() + * + * Restores the state saved when entering a kpkeys context. If no context was + * entered, this function does nothing. + / +static __always_inline +void kpkeys_leave_context(const struct kpkeys_state state) +{ + if (state->entered_context) + arch_kpkeys_leave_context(state); +} + +static inline bool kpkeys_supported(void) +{ + return arch_supports_kpkeys(); +} + +#else /* CONFIG_ARCH_HAS_KPKEYS / + +static inline bool kpkeys_supported(void) +{ + return false; +} + +#endif / CONFIG_ARCH_HAS_KPKEYS / + +#ifdef CONFIG_KPKEYS_HARDENED_PGTABLES + +DECLARE_STATIC_KEY_FALSE(kpkeys_hardened_pgtables_key); + +static inline bool kpkeys_hardened_pgtables_enabled(void) +{ + return static_branch_unlikely(&kpkeys_hardened_pgtables_key); +} + +static inline bool kpkeys_hardened_pgtables_early_enabled(void) +{ + return arch_supports_kpkeys_early(); +} + +struct page kpkeys_pgtable_alloc(gfp_t gfp, unsigned int order); +void kpkeys_pgtable_free(struct page page, unsigned int order); + +/ + * Should be called from mem_init(): as soon as the buddy allocator becomes + * available and before any call to pagetable_alloc(). + / +void kpkeys_hardened_pgtables_init(void); + +phys_addr_t kpkeys_physmem_pgtable_alloc(void); + +#ifndef arch_kpkeys_protect_static_pgtables +static inline void arch_kpkeys_protect_static_pgtables(void) {} +#endif + +#else / CONFIG_KPKEYS_HARDENED_PGTABLES / + +static inline bool kpkeys_hardened_pgtables_enabled(void) +{ + return false; +} + +static inline bool kpkeys_hardened_pgtables_early_enabled(void) +{ + return false; +} + +static inline struct page kpkeys_pgtable_alloc(gfp_t gfp, unsigned int order) +{ + return NULL; +} + +static inline void kpkeys_pgtable_free(struct page page, unsigned int order) {} + +static inline void kpkeys_hardened_pgtables_init(void) {} + +static inline phys_addr_t kpkeys_physmem_pgtable_alloc(void) +{ + return 0; +} + +#endif / CONFIG_KPKEYS_HARDENED_PGTABLES / + +#endif / _LINUX_KPKEYS_H / diff --git a/include/linux/kpkeys_types.h b/include/linux/kpkeys_types.h new file mode 100644 index 0000000000000..09f15a9565cf3 --- /dev/null +++ b/include/linux/kpkeys_types.h @@ -0,0 +1,30 @@ +/ SPDX-License-Identifier: GPL-2.0-only / +#ifndef _LINUX_KPKEYS_TYPES_H +#define _LINUX_KPKEYS_TYPES_H + +#include <linux/types.h> + +#ifdef CONFIG_ARCH_HAS_KPKEYS +#include <asm/kpkeys_types.h> +#endif + +#ifndef KPKEYS_PKEY_PGTABLES +#define KPKEYS_PKEY_PGTABLES 1 +#endif + +#if defined(CONFIG_ARCH_HAS_KPKEYS) && !defined(__ASSEMBLY__) + +enum kpkeys_ctx { + KPKEYS_CTX_DEFAULT = 0, + KPKEYS_CTX_PGTABLES, + KPKEYS_CTX_COUNT, +}; + +struct kpkeys_state { + bool entered_context; + struct arch_kpkeys_state arch_state; +}; + +#endif / CONFIG_ARCH_HAS_KPKEYS && !__ASSEMBLY__ / + +#endif / _LINUX_KPKEYS_TYPES_H / diff --git a/include/linux/memblock.h b/include/linux/memblock.h index 5afcd99aa8c14..c61f80e31cc10 100644 --- a/include/linux/memblock.h +++ b/include/linux/memblock.h @@ -24,6 +24,17 @@ extern unsigned long max_pfn; / extern unsigned long long max_possible_pfn; +#define INIT_MEMBLOCK_REGIONS 128 +#define INIT_PHYSMEM_REGIONS 4 + +#ifndef INIT_MEMBLOCK_RESERVED_REGIONS +#define INIT_MEMBLOCK_RESERVED_REGIONS INIT_MEMBLOCK_REGIONS +#endif + +#ifndef INIT_MEMBLOCK_MEMORY_REGIONS +#define INIT_MEMBLOCK_MEMORY_REGIONS INIT_MEMBLOCK_REGIONS +#endif + /** * enum memblock_flags - definition of memory region attributes * @MEMBLOCK_NONE: no special request diff --git a/include/linux/mm.h b/include/linux/mm.h index 485df9c2dbddb..eea3131786b35 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -37,6 +37,7 @@ #include <linux/bitmap.h> #include <linux/bitops.h> #include <linux/iommu-debug-pagealloc.h> +#include <linux/kpkeys.h> struct mempolicy; struct anon_vma; @@ -3682,7 +3683,12 @@ static inline bool ptdesc_test_kernel(const struct ptdesc ptdesc) / static inline struct ptdesc pagetable_alloc_noprof(gfp_t gfp, unsigned int order) { - struct page page = alloc_pages_noprof(gfp \| __GFP_COMP, order); + struct page page; + + if (kpkeys_hardened_pgtables_enabled()) + page = kpkeys_pgtable_alloc(gfp \| __GFP_COMP, order); + else + page = alloc_pages_noprof(gfp \| __GFP_COMP, order); return page_ptdesc(page); } @@ -3691,8 +3697,12 @@ static inline struct ptdesc pagetable_alloc_noprof(gfp_t gfp, unsigned int orde static inline void __pagetable_free(struct ptdesc pt) { struct page page = ptdesc_page(pt); + unsigned int order = compound_order(page); - __free_pages(page, compound_order(page)); + if (kpkeys_hardened_pgtables_enabled()) + kpkeys_pgtable_free(page, order); + else + __free_pages(page, order); } #ifdef CONFIG_ASYNC_KERNEL_PGTABLE_FREE diff --git a/include/linux/set_memory.h b/include/linux/set_memory.h index 3030d9245f5ac..7b3a8bfde3c64 100644 --- a/include/linux/set_memory.h +++ b/include/linux/set_memory.h @@ -84,4 +84,11 @@ static inline int set_memory_decrypted(unsigned long addr, int numpages) } #endif /* CONFIG_ARCH_HAS_MEM_ENCRYPT / +#ifndef CONFIG_ARCH_HAS_KPKEYS +static inline int set_memory_pkey(unsigned long addr, int numpages, int pkey) +{ + return 0; +} +#endif + #endif / _LINUX_SET_MEMORY_H_ / diff --git a/mm/Kconfig b/mm/Kconfig index 9e0ca48249054..21bd3e550de83 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1281,6 +1281,11 @@ config ARCH_USES_HIGH_VMA_FLAGS bool config ARCH_HAS_PKEYS bool +config ARCH_HAS_KPKEYS + bool +config ARCH_HAS_KPKEYS_HARDENED_PGTABLES + bool + depends on ARCH_HAS_KPKEYS config ARCH_USES_PG_ARCH_2 bool diff --git a/mm/Makefile b/mm/Makefile index eff9f9e7e061c..e31e582948451 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -147,3 +147,5 @@ obj-$(CONFIG_SHRINKER_DEBUG) += shrinker_debug.o obj-$(CONFIG_EXECMEM) += execmem.o obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o obj-$(CONFIG_LAZY_MMU_MODE_KUNIT_TEST) += tests/lazy_mmu_mode_kunit.o +obj-$(CONFIG_KPKEYS_HARDENED_PGTABLES) += kpkeys_hardened_pgtables.o +obj-$(CONFIG_KPKEYS_HARDENED_PGTABLES_KUNIT_TEST) += tests/kpkeys_hardened_pgtables_kunit.o diff --git a/mm/kpkeys_hardened_pgtables.c b/mm/kpkeys_hardened_pgtables.c new file mode 100644 index 0000000000000..c133e5896fccc --- /dev/null +++ b/mm/kpkeys_hardened_pgtables.c @@ -0,0 +1,179 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include <linux/kpkeys.h> +#include <linux/memblock.h> +#include <linux/mm.h> +#include <linux/set_memory.h> + +#include <kunit/visibility.h> + +__ro_after_init DEFINE_STATIC_KEY_FALSE(kpkeys_hardened_pgtables_key); +EXPORT_SYMBOL_IF_KUNIT(kpkeys_hardened_pgtables_key); + +static int set_pkey_pgtable(struct page page, unsigned int nr_pages) +{ + unsigned long addr = (unsigned long)page_address(page); + int ret; + + ret = set_memory_pkey(addr, nr_pages, KPKEYS_PKEY_PGTABLES); + + WARN_ON(ret); + return ret; +} + +static int set_pkey_default(struct page page, unsigned int nr_pages) +{ + unsigned long addr = (unsigned long)page_address(page); + int ret; + + ret = set_memory_pkey(addr, nr_pages, 0); + + WARN_ON(ret); + return ret; +} + +/ pkeys physmem allocator (PPA) - implemented below / +static void ppa_finalize(void); + +struct page kpkeys_pgtable_alloc(gfp_t gfp, unsigned int order) +{ + struct page page; + int ret; + + page = alloc_pages_noprof(gfp, order); + if (!page) + return page; + + ret = set_pkey_pgtable(page, 1 << order); + if (ret) { + __free_pages(page, order); + return NULL; + } + + return page; +} + +void kpkeys_pgtable_free(struct page page, unsigned int order) +{ + set_pkey_default(page, 1 << order); + __free_pages(page, order); +} + +void __init kpkeys_hardened_pgtables_init(void) +{ + if (!kpkeys_supported()) + return; + + static_branch_enable(&kpkeys_hardened_pgtables_key); + + ppa_finalize(); + arch_kpkeys_protect_static_pgtables(); +} + +/* + * pkeys physmem allocator (PPA): allocator for very early page tables + * (especially for creating the linear map), based on memblock. Allocated + * ranges are tracked so that their pkey can be set once it is safe to do so. + / + +/ + * We may have to track many ranges when allocating page tables for the linear + * map, as their number grows with the amount of available memory. Assuming that + * memblock returns contiguous blocks whenever possible, the number of ranges + * to track cannot however exceed the number of regions that memblock itself + * tracks. memblock_allow_resize() hasn't been called yet at that point, so + * that limit is the size of the statically allocated array. + / +#define PHYSMEM_MAX_RANGES INIT_MEMBLOCK_MEMORY_REGIONS + +struct physmem_range { + phys_addr_t addr; + phys_addr_t size; +}; + +struct pkeys_physmem_allocator { + struct physmem_range allocated_ranges[PHYSMEM_MAX_RANGES]; + unsigned int nr_allocated_ranges; +}; + +static struct pkeys_physmem_allocator pkeys_physmem_allocator __initdata; + +static int __init set_pkey_pgtable_phys(phys_addr_t pa, phys_addr_t size) +{ + unsigned long addr = (unsigned long)__va(pa); + int ret; + + ret = set_memory_pkey(addr, size / PAGE_SIZE, KPKEYS_PKEY_PGTABLES); + + WARN_ON(ret); + return ret; +} + +static bool __init ppa_try_extend_last_range(phys_addr_t addr, phys_addr_t size) +{ + struct pkeys_physmem_allocator ppa = &pkeys_physmem_allocator; + struct physmem_range range; + + if (!ppa->nr_allocated_ranges) + return false; + + range = &ppa->allocated_ranges[ppa->nr_allocated_ranges - 1]; + + / Merge the new range into the last range if they are contiguous / + if (addr == range->addr + range->size) { + range->size += size; + return true; + } else if (addr + size == range->addr) { + range->addr -= size; + range->size += size; + return true; + } + + return false; +} + +static void __init ppa_register_allocated_range(phys_addr_t addr, + phys_addr_t size) +{ + struct pkeys_physmem_allocator ppa = &pkeys_physmem_allocator; + struct physmem_range range; + + if (!addr) + return; + + if (ppa_try_extend_last_range(addr, size)) + return; + + / Could not extend the last range, create a new one / + if (WARN_ON(ppa->nr_allocated_ranges >= PHYSMEM_MAX_RANGES)) + return; + + range = &ppa->allocated_ranges[ppa->nr_allocated_ranges++]; + range->addr = addr; + range->size = size; +} + +static void __init ppa_finalize(void) +{ + struct pkeys_physmem_allocator ppa = &pkeys_physmem_allocator; + + for (unsigned int i = 0; i < ppa->nr_allocated_ranges; i++) { + struct physmem_range range = &ppa->allocated_ranges[i]; + + set_pkey_pgtable_phys(range->addr, range->size); + } +} + +phys_addr_t __ref kpkeys_physmem_pgtable_alloc(void) +{ + size_t size = PAGE_SIZE; + phys_addr_t addr; + + addr = memblock_phys_alloc_range(size, size, 0, + MEMBLOCK_ALLOC_NOLEAKTRACE); + if (!addr) + return addr; + + ppa_register_allocated_range(addr, size); + + return addr; +} diff --git a/mm/memblock.c b/mm/memblock.c index 6349c48154f4b..3b497d3556bb0 100644 --- a/mm/memblock.c +++ b/mm/memblock.c @@ -30,17 +30,6 @@ #include "internal.h" -#define INIT_MEMBLOCK_REGIONS 128 -#define INIT_PHYSMEM_REGIONS 4 - -#ifndef INIT_MEMBLOCK_RESERVED_REGIONS -# define INIT_MEMBLOCK_RESERVED_REGIONS INIT_MEMBLOCK_REGIONS -#endif - -#ifndef INIT_MEMBLOCK_MEMORY_REGIONS -#define INIT_MEMBLOCK_MEMORY_REGIONS INIT_MEMBLOCK_REGIONS -#endif - /* * DOC: memblock overview * diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index ebd3ac997f645..1fec89168ff10 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -179,13 +179,29 @@ static pte_t * __meminit vmemmap_pte_populate(pmd_t pmd, unsigned long addr, in return pte; } -static void __meminit vmemmap_alloc_block_zero(unsigned long size, int node) +static void * __meminit vmemmap_alloc_pgtable(int node) { - void p = vmemmap_alloc_block(size, node); + void p; + + if (slab_is_available()) { + gfp_t gfp = GFP_KERNEL \| __GFP_ZERO; + struct ptdesc ptdesc = pagetable_alloc(gfp, 0); + + return ptdesc ? ptdesc_address(ptdesc) : NULL; + } + + if (kpkeys_hardened_pgtables_early_enabled()) { + phys_addr_t phys = kpkeys_physmem_pgtable_alloc(); + + p = phys ? phys_to_virt(phys) : NULL; + } else { + p = __earlyonly_bootmem_alloc(node, PAGE_SIZE, PAGE_SIZE, + __pa(MAX_DMA_ADDRESS)); + } if (!p) return NULL; - memset(p, 0, size); + memset(p, 0, PAGE_SIZE); return p; } @@ -194,7 +210,7 @@ static pmd_t __meminit vmemmap_pmd_populate(pud_t pud, unsigned long addr, in { pmd_t pmd = pmd_offset(pud, addr); if (pmd_none(pmd)) { - void p = vmemmap_alloc_block_zero(PAGE_SIZE, node); + void p = vmemmap_alloc_pgtable(node); if (!p) return NULL; kernel_pte_init(p); @@ -207,7 +223,7 @@ static pud_t __meminit vmemmap_pud_populate(p4d_t p4d, unsigned long addr, in { pud_t pud = pud_offset(p4d, addr); if (pud_none(pud)) { - void p = vmemmap_alloc_block_zero(PAGE_SIZE, node); + void p = vmemmap_alloc_pgtable(node); if (!p) return NULL; pmd_init(p); @@ -220,7 +236,7 @@ static p4d_t __meminit vmemmap_p4d_populate(pgd_t pgd, unsigned long addr, in { p4d_t p4d = p4d_offset(pgd, addr); if (p4d_none(p4d)) { - void p = vmemmap_alloc_block_zero(PAGE_SIZE, node); + void p = vmemmap_alloc_pgtable(node); if (!p) return NULL; pud_init(p); @@ -233,7 +249,7 @@ static pgd_t __meminit vmemmap_pgd_populate(unsigned long addr, int node) { pgd_t pgd = pgd_offset_k(addr); if (pgd_none(pgd)) { - void p = vmemmap_alloc_block_zero(PAGE_SIZE, node); + void p = vmemmap_alloc_pgtable(node); if (!p) return NULL; pgd_populate_kernel(addr, pgd, p); @@ -346,10 +362,11 @@ static __meminit struct page vmemmap_get_tail(unsigned int order, struct zone * memmap_init(). / - p = vmemmap_alloc_block_zero(PAGE_SIZE, node); + p = vmemmap_alloc_block(PAGE_SIZE, node); if (!p) return NULL; + memset(p, 0, PAGE_SIZE); tail = virt_to_page(p); zone->vmemmap_tails[idx] = tail; diff --git a/mm/tests/kpkeys_hardened_pgtables_kunit.c b/mm/tests/kpkeys_hardened_pgtables_kunit.c new file mode 100644 index 0000000000000..80d4f5144e48d --- /dev/null +++ b/mm/tests/kpkeys_hardened_pgtables_kunit.c @@ -0,0 +1,200 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include <kunit/test.h> +#include <linux/mman.h> +#include <linux/pgtable.h> +#include <linux/set_memory.h> +#include <linux/vmalloc.h> + +static void free_page_wrapper(void ctx) +{ + __free_page((struct page )ctx); +} + +KUNIT_DEFINE_ACTION_WRAPPER(vfree_wrapper, vfree, const void ); + +static pud_t pud_off_k(unsigned long va) +{ + return pud_offset(p4d_offset(pgd_offset_k(va), va), va); +} + +static pte_t get_kernel_pte(unsigned long addr) +{ + pmd_t pmdp = pmd_off_k(addr); + + if (!pmdp \|\| pmd_leaf(pmdp)) + return NULL; + + return pte_offset_kernel(pmdp, addr); +} + +#define write_pgtable(type, ptr) do { \ + type##_t val; \ + int ret; \ + \ + pr_debug("%s: writing to "#type" at %px\n", __func__, (ptr)); \ + \ + val = type##p_get(ptr); \ + ret = copy_to_kernel_nofault(ptr, &val, sizeof(val)); \ + KUNIT_EXPECT_EQ_MSG(test, ret, -EFAULT, \ + "Direct "#type" write wasn't prevented"); \ +} while (0) + +/* + * Try to write linear map page tables, at every level. This is worthwhile + * because those page table pages are obtained from different allocators: + * + * - Static memory (part of the kernel image) for PGD + * - memblock for PUD and possibly PMD/PTE + * - pagetable_alloc() (buddy allocator) for PMD/PTE if large block mappings are + * used and the linear map is split after being created + / +static void write_direct_map_pgtables(struct kunit test) +{ + struct page page; + unsigned long addr; + pgd_t pgdp; + p4d_t p4dp; + pud_t pudp; + pmd_t pmdp; + pte_t ptep; + int ret; + + if (!kpkeys_supported()) + kunit_skip(test, "kpkeys are not supported"); + + page = alloc_page(GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, page); + ret = kunit_add_action_or_reset(test, free_page_wrapper, page); + KUNIT_ASSERT_EQ(test, ret, 0); + + /* Ensure page is PTE-mapped (splitting the linear map if necessary) / + ret = set_direct_map_invalid_noflush(page); + KUNIT_ASSERT_EQ(test, ret, 0); + ret = set_direct_map_default_noflush(page); + KUNIT_ASSERT_EQ(test, ret, 0); + + addr = (unsigned long)page_address(page); + + pgdp = pgd_offset_k(addr); + KUNIT_ASSERT_NOT_NULL_MSG(test, pgdp, "Failed to get PGD"); + / + * swapper_pg_dir is still writable at this stage, so don't check it. + * It is not protected by kpkeys_hardened_pgtables because it should be + * made read-only by mark_rodata_ro(). However since these + * KUnit tests are builtin, they are run before mark_rodata_ro() is + * called. + / + + p4dp = p4d_offset(pgdp, addr); + KUNIT_ASSERT_NOT_NULL_MSG(test, p4dp, "Failed to get P4D"); + if (!mm_p4d_folded(&init_mm)) + write_pgtable(p4d, p4dp); + + pudp = pud_offset(p4dp, addr); + KUNIT_ASSERT_NOT_NULL_MSG(test, pudp, "Failed to get PUD"); + if (!mm_pud_folded(&init_mm)) + write_pgtable(pud, pudp); + + pmdp = pmd_offset(pudp, addr); + KUNIT_ASSERT_NOT_NULL_MSG(test, pmdp, "Failed to get PMD"); + write_pgtable(pmd, pmdp); + + ptep = pte_offset_kernel(pmdp, addr); + KUNIT_ASSERT_NOT_NULL_MSG(test, ptep, "Failed to get PTE"); + write_pgtable(pte, ptep); +} + +/ Worth checking since the kernel image is mapped with static page tables / +static void write_kernel_image_pud(struct kunit test) +{ + pud_t pudp; + + if (!kpkeys_supported()) + kunit_skip(test, "kpkeys are not supported"); + + / The kernel is probably block-mapped, check the PUD to be safe / + pudp = pud_off_k((unsigned long)&init_mm); + KUNIT_ASSERT_NOT_NULL_MSG(test, pudp, "Failed to get PUD"); + + write_pgtable(pud, pudp); +} + +static void write_kernel_vmalloc_pte(struct kunit test) +{ + void mem; + pte_t ptep; + int ret; + + if (!kpkeys_supported()) + kunit_skip(test, "kpkeys are not supported"); + + mem = vmalloc(PAGE_SIZE); + KUNIT_ASSERT_NOT_NULL(test, mem); + ret = kunit_add_action_or_reset(test, vfree_wrapper, mem); + KUNIT_ASSERT_EQ(test, ret, 0); + + /* vmalloc() without VM_ALLOW_HUGE_VMAP is PTE-mapped / + ptep = get_kernel_pte((unsigned long)mem); + KUNIT_ASSERT_NOT_NULL_MSG(test, ptep, "Failed to get PTE"); + + write_pgtable(pte, ptep); +} + +static void write_vmemmap_pmd(struct kunit test) +{ + struct page page; + pmd_t pmdp; + + if (!kpkeys_supported()) + kunit_skip(test, "kpkeys are not supported"); + + /* + * We just need the address of some struct page, so we can free the + * page right away. + / + page = alloc_page(GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, page); + __free_page(page); + + / vmemmap may use PMD block mappings / + pmdp = pmd_off_k((unsigned long)page); + KUNIT_ASSERT_NOT_NULL_MSG(test, pmdp, "Failed to get PMD"); + write_pgtable(pmd, pmdp); +} + +static void write_user_pmd(struct kunit test) +{ + pmd_t pmdp; + unsigned long uaddr; + + if (!kpkeys_supported()) + kunit_skip(test, "kpkeys are not supported"); + + uaddr = kunit_vm_mmap(test, NULL, 0, PAGE_SIZE, PROT_READ, + MAP_ANONYMOUS \| MAP_PRIVATE \| MAP_POPULATE, 0); + KUNIT_ASSERT_NE_MSG(test, uaddr, 0, "Could not create userspace mm"); + + / We passed MAP_POPULATE so a PMD should already be allocated */ + pmdp = pmd_off(current->mm, uaddr); + KUNIT_ASSERT_NOT_NULL_MSG(test, pmdp, "Failed to get PMD"); + + write_pgtable(pmd, pmdp); +} + +static struct kunit_case kpkeys_hardened_pgtables_test_cases[] = { + KUNIT_CASE(write_direct_map_pgtables), + KUNIT_CASE(write_kernel_image_pud), + KUNIT_CASE(write_kernel_vmalloc_pte), + KUNIT_CASE(write_vmemmap_pmd), + KUNIT_CASE(write_user_pmd), + {} +}; + +static struct kunit_suite kpkeys_hardened_pgtables_test_suite = { + .name = "kpkeys_hardened_pgtables", + .test_cases = kpkeys_hardened_pgtables_test_cases, +}; +kunit_test_suite(kpkeys_hardened_pgtables_test_suite); + +MODULE_DESCRIPTION("Tests for the kpkeys_hardened_pgtables feature"); +MODULE_LICENSE("GPL"); diff --git a/security/Kconfig.hardening b/security/Kconfig.hardening index 6923036e1a2f5..ad6152961f8ee 100644 --- a/security/Kconfig.hardening +++ b/security/Kconfig.hardening @@ -271,6 +271,30 @@ config BUG_ON_DATA_CORRUPTION If unsure, say N. +config KPKEYS_HARDENED_PGTABLES + bool "Harden page tables using kernel pkeys" + depends on ARCH_HAS_KPKEYS_HARDENED_PGTABLES + help + This option makes all page tables mostly read-only by + allocating them with a non-default protection key (pkey) and + only enabling write access to that pkey in routines that are + expected to write to page table entries. + + This option has no effect if the system does not support + kernel pkeys. + +config KPKEYS_HARDENED_PGTABLES_KUNIT_TEST + bool "KUnit tests for kpkeys_hardened_pgtables" if !KUNIT_ALL_TESTS + depends on KPKEYS_HARDENED_PGTABLES + depends on KUNIT=y + default KUNIT_ALL_TESTS + help + Enable this option to check that the kpkeys_hardened_pgtables feature + functions as intended, i.e. prevents arbitrary writes to user and + kernel page tables. + + If unsure, say N. + endmenu config CC_HAS_RANDSTRUCT
3/2	2026/08/18 17:01	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 21257 output: 34 thoughts: 1736 Thoughts: Fuzzing Target Identification: KPKEYS Hardened Page Tables on ARM64 Okay, so I need to assess this patch's impact for fuzzing purposes. It looks like it introduces a significant functional change – hardening page tables on ARM64 using kernel pkeys (KPKEYS). Specifically, `KPKEYS_HARDENED_PGTABLES` is the key, and it seems to modify memory management, particularly page table handling. Given the nature of this change, memory management is always a good target. First, let's verify the relevant configurations: `ARM64_POE` and `KPKEYS_HARDENED_PGTABLES`. Good, they exist and `ARM64_POE` appears to always be enabled, or has been in the current environment. Now, where are the functions of interest? I've spotted `kpkeys_pgtable_alloc`, `kpkeys_pgtable_free`, `kpkeys_physmem_pgtable_alloc` within `mm/kpkeys_hardened_pgtables.c`. I'm also seeing modifications in `arch/arm64/mm/mmu.c`, specifically `arch_set_user_pkey_access` and `arch/arm64/mm/pageattr.c` with `set_memory_pkey`. I need to check how reachable `kpkeys_pgtable_alloc` is, which is called from `pagetable_alloc_noprof` in `include/linux/mm.h`. `kpkeys_pgtable_free` is called from `__pagetable_free`. These are good. Let's prioritize those as potential fuzzing targets. After a deeper dive, I've confirmed that these are global, non-static functions and thus, are more easily accessible for continuous fuzzing. My list of functions to focus on is now: * `kpkeys_pgtable_alloc` * `kpkeys_pgtable_free` * `kpkeys_physmem_pgtable_alloc` * `set_memory_pkey` * `arch_set_user_pkey_access` `arch_set_user_pkey_access` seems to be modified, so definitely worth a closer look. I'll keep `KPKEYS_HARDENED_PGTABLES_KUNIT_TEST` in mind, but it isn't required for fuzzing this. So, the configurations to ensure are: * `ARM64_POE` (always enabled) * `KPKEYS_HARDENED_PGTABLES` Okay, this is a solid starting point. I'll focus my fuzzing efforts on those specific functions and make sure the appropriate configurations are enabled.
4/2	2026/08/18 17:01	tool	grepper	0m Args: { "Expression": "config ARM64_POE", "PathPrefix": "arch/arm64/Kconfig" } Results: { "Output": "arch/arm64/Kconfig=2198=menu \"ARMv8.9 architectural features\"\narch/arm64/Kconfig-2199-\narch/arm64/Kconfig:2200:config ARM64_POE\narch/arm64/Kconfig-2201-\tprompt \"Permission Overlay Extension\"\n" }
5/2	2026/08/18 17:01	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 23715 output: 37 thoughts: 15 Thoughts: Here's my attempt at summarizing the thought process, tailored for an expert audience: Analyzing the `ARM64_POE` Definition Okay, so the initial prompt is pretty straightforward - let's dive right into the full definition of `ARM64_POE`. I'm immediately thinking, "why the interest in this particular prefetch optimization?". It's a common area, but understanding the specific context is key here. Am I looking at it from a microarchitecture perspective, perhaps analyzing instruction-level details and pipeline stalls, or is this related to memory management and coherence on a multi-core ARM system? Is the context embedded systems, or high-performance computing? My immediate goal is to establish the scope of this definition. What aspects are being emphasized? Are we talking about the core architecture (e.g., specific instructions, register usage), the memory hierarchy interaction, or perhaps some software-level implications with compiler optimization or runtime libraries? Understanding the relevant abstraction level is important, since the definition will almost certainly have implications for memory access and instruction fetching. I'm mentally preparing to parse the definition for keywords related to prefetch algorithms, cache management policies (e.g., speculative loading, demand prefetching), and the impact on the memory subsystem (latency, bandwidth, coherency). I'm also anticipating potential links to power consumption and performance trade-offs, depending on the particular implementation. A formal definition will hopefully include the specific architectural features and their potential constraints. In short, let's understand the problem that `ARM64_POE` is trying to solve, and the specific mechanisms used.
6/2	2026/08/18 17:01	tool	read-file	0m Args: { "File": "arch/arm64/Kconfig", "FirstLine": 2198, "LineCount": 20 } Results: { "Contents": "2198:\tmenu \"ARMv8.9 architectural features\"\n2199:\t\n2200:\tconfig ARM64_POE\n2201:\t\tprompt \"Permission Overlay Extension\"\n2202:\t\tdef_bool y\n2203:\t\tselect ARCH_USES_HIGH_VMA_FLAGS\n2204:\t\tselect ARCH_HAS_PKEYS\n2205:\t\tselect ARCH_HAS_KPKEYS\n2206:\t\tselect ARCH_HAS_KPKEYS_HARDENED_PGTABLES\n2207:\t\thelp\n2208:\t\t The Permission Overlay Extension is used to implement Memory\n2209:\t\t Protection Keys. Memory Protection Keys provides a mechanism for\n2210:\t\t enforcing page-based protections, but without requiring modification\n2211:\t\t of the page tables when an application changes protection domains.\n2212:\t\n2213:\t\t For details, see Documentation/core-api/protection-keys.rst\n2214:\t\n2215:\t\t If unsure, say y.\n2216:\t\n2217:\tconfig ARCH_PKEY_BITS\n" }
7/2	2026/08/18 17:01	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 24398 output: 34 thoughts: 40 Thoughts: My Initial Thoughts on ARM64_POE and KPKEYS_HARDENED_PGTABLES Okay, so I'm looking at these configurations, and immediately my mind jumps to the implications. `ARM64_POE` is being enabled by default – the prompt suggests a default of "y". That's interesting, suggesting this is a potentially important or recommended feature for the ARM64 architecture, and that it's probably going to be part of most builds. I'll need to dig into what "POE" actually represents in this context on ARM64 and the default setting. Then, there's `KPKEYS_HARDENED_PGTABLES`, which lives in `security/Kconfig.hardening`. Right away, that location is telling me this is security-related. The "KPKEYS" part suggests key protection for the kernel, which is a significant signal; it's likely related to protecting page tables from unauthorized modification. The fact that it’s hardened is good. I need to investigate how it is hardened, as well as the performance and compatibility implications. Security hardening is an area I'm always keen to understand the details of.
8/2	2026/08/18 17:02	tool	read-file	0m Args: { "File": "security/Kconfig.hardening", "FirstLine": 270, "LineCount": 20 } Results: { "Contents": " 270:\t\t for validity.\n 271:\t\n 272:\t\t If unsure, say N.\n 273:\t\n 274:\tconfig KPKEYS_HARDENED_PGTABLES\n 275:\t\tbool \"Harden page tables using kernel pkeys\"\n 276:\t\tdepends on ARCH_HAS_KPKEYS_HARDENED_PGTABLES\n 277:\t\thelp\n 278:\t\t This option makes all page tables mostly read-only by\n 279:\t\t allocating them with a non-default protection key (pkey) and\n 280:\t\t only enabling write access to that pkey in routines that are\n 281:\t\t expected to write to page table entries.\n 282:\t\n 283:\t\t This option has no effect if the system does not support\n 284:\t\t kernel pkeys.\n 285:\t\n 286:\tconfig KPKEYS_HARDENED_PGTABLES_KUNIT_TEST\n 287:\t\tbool \"KUnit tests for kpkeys_hardened_pgtables\" if !KUNIT_ALL_TESTS\n 288:\t\tdepends on KPKEYS_HARDENED_PGTABLES\n 289:\t\tdepends on KUNIT=y\n" }
9/2	2026/08/18 17:02	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 25044 output: 41 thoughts: 231 Thoughts: My Configuration Considerations for Enhanced ARM64 Memory Protection Okay, so I'm looking at enabling some key configurations here. `KPKEYS_HARDENED_PGTABLES` seems to be the crucial one, and it's contingent on `ARCH_HAS_KPKEYS_HARDENED_PGTABLES`. Given the goal, I think enabling `ARM64_POE` is the logical first step, and that should, in turn, pull in `KPKEYS_HARDENED_PGTABLES`. Now, before I commit to anything, let's just do a quick scan of related configurations to make sure I haven't missed anything obvious. I need to keep `ARCH_HAS_KPKEYS` and `ARCH_HAS_KPKEYS_HARDENED_PGTABLES` in mind. Ah, right, those are selected by `ARM64_POE`. So, it looks like `ARM64_POE` is definitely the parent, and `KPKEYS_HARDENED_PGTABLES` is the dependent. No surprises there. Alright, so the configs to enable, in addition to `ARM64_POE`, would then be `KPKEYS_HARDENED_PGTABLES`. I've got a good handle on that part. Next, I need to focus on the key functions involved in this. I'll need to pay close attention to `kpkeys_pgtable_alloc`, `kpkeys_pgtable_free`, `kpkeys_physmem_pgtable_alloc`, `set_memory_pkey`, and `arch_set_user_pkey_access`. I'll need to review these and make sure they are operating as intended. Let's make sure that `arch_set_user_pkey_access` is in `arch/arm64/mm/mmu.c`, as I'm pretty sure that's where I need to look.
10/2	2026/08/18 17:02	tool	grepper	0m Args: { "Expression": "arch_set_user_pkey_access", "PathPrefix": "arch/arm64/mm/mmu.c" } Results: { "Output": "arch/arm64/mm/mmu.c=2300=void __cpu_replace_ttbr1(pgd_t *pgdp, bool cnp)\n--\narch/arm64/mm/mmu.c-2328-#ifdef CONFIG_ARCH_HAS_PKEYS\narch/arm64/mm/mmu.c:2329:int arch_set_user_pkey_access(int pkey, unsigned long init_val)\narch/arm64/mm/mmu.c-2330-{\n" }
11/2	2026/08/18 17:02	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 25866 output: 37 thoughts: 32 Thoughts: Okay, here's my take on those thoughts, tailored for an expert audience: Analyzing `set_memory_pkey` on ARM64 Yes, that's correct. I'm on the right track here. Now, I need to confirm something specific about how the page protection keys are managed, so let's dig into `set_memory_pkey` within `arch/arm64/mm/pageattr.c`. I need to refresh my memory on the interactions there, particularly how the physical address translation is updated when page keys change and what the implications are regarding TLB invalidation strategies and potential races. I'm primarily interested in looking into the code related to memory protection keys and how they impact security.
12/2	2026/08/18 17:02	tool	grepper	0m Args: { "Expression": "set_memory_pkey", "PathPrefix": "arch/arm64/mm/pageattr.c" } Results: { "Output": "arch/arm64/mm/pageattr.c=361=int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid)\n--\narch/arm64/mm/pageattr.c-371-#ifdef CONFIG_ARCH_HAS_KPKEYS\narch/arm64/mm/pageattr.c:372:int set_memory_pkey(unsigned long addr, int numpages, int pkey)\narch/arm64/mm/pageattr.c-373-{\n" }
13/2	2026/08/18 17:02	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 26226 output: 39 thoughts: 34 Thoughts: Okay, here's my take on those thoughts, framed as an expert's internal monologue: Quick Confirmation and a Code Dive Right, that seems straightforward enough. Yep, it is. Let's not waste any time, I'm going to quickly check `kpkeys_pgtable_alloc` in `mm/kpkeys_hardened_pgtables.c`. I need to ensure the allocation behavior aligns with the current security design. I should probably pay particular attention to its interaction with the hardened page table mechanisms. I need a quick sanity check before moving on to the more complex bits, you know, to confirm my assumptions are accurate.
14/2	2026/08/18 17:03	tool	grepper	0m Args: { "Expression": "kpkeys_pgtable_alloc", "PathPrefix": "mm/kpkeys_hardened_pgtables.c" } Results: { "Output": "mm/kpkeys_hardened_pgtables.c=35=static void ppa_finalize(void);\nmm/kpkeys_hardened_pgtables.c-36-\nmm/kpkeys_hardened_pgtables.c:37:struct page *kpkeys_pgtable_alloc(gfp_t gfp, unsigned int order)\nmm/kpkeys_hardened_pgtables.c-38-{\n" }
15/2	2026/08/18 17:03	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 26547 output: 39 thoughts: 31 Thoughts: Okay, here's my summary, taking on the persona you requested: Assessing the Landscape: pgtable_free Yes. I've got it. Time to dive into the specifics of `kpkeys_pgtable_free`. I need to zero in on this function within `mm/kpkeys_hardened_pgtables.c`. I need to ensure a complete understanding of its role and how it interacts with the broader context of hardened page tables. My expertise allows me to analyze this with precision, identifying potential strengths and vulnerabilities.
16/2	2026/08/18 17:03	tool	grepper	0m Args: { "Expression": "kpkeys_pgtable_free", "PathPrefix": "mm/kpkeys_hardened_pgtables.c" } Results: { "Output": "mm/kpkeys_hardened_pgtables.c=37=struct page kpkeys_pgtable_alloc(gfp_t gfp, unsigned int order)\n--\nmm/kpkeys_hardened_pgtables.c-54-\nmm/kpkeys_hardened_pgtables.c:55:void kpkeys_pgtable_free(struct page page, unsigned int order)\nmm/kpkeys_hardened_pgtables.c-56-{\n" }
17/2	2026/08/18 17:03	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 26854 output: 42 thoughts: 34 Thoughts: Okay, here's my interpretation of those thoughts, tailored for an expert audience, presented in the first person: Kernel Memory Protection - Deep Dive Right. I need to take a look at the memory allocation function for the `kpkeys_physmem_pgtable` specifically in relation to the hardened page table implementation. Let's dig into `mm/kpkeys_hardened_pgtables.c` and focus on `kpkeys_physmem_pgtable_alloc`. That's where the critical aspects of this are going to be defined; need to figure out how memory is allocated, initialized, and protected from there.
18/2	2026/08/18 17:03	tool	grepper	0m Args: { "Expression": "kpkeys_physmem_pgtable_alloc", "PathPrefix": "mm/kpkeys_hardened_pgtables.c" } Results: { "Output": "mm/kpkeys_hardened_pgtables.c=155=static void __init ppa_finalize(void)\n--\nmm/kpkeys_hardened_pgtables.c-165-\nmm/kpkeys_hardened_pgtables.c:166:phys_addr_t __ref kpkeys_physmem_pgtable_alloc(void)\nmm/kpkeys_hardened_pgtables.c-167-{\n" }
19/2	2026/08/18 17:03	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 27181 output: 221 thoughts: 802 Thoughts: My Analysis of the Kernel PKey Hardened Paging Table Implementation Okay, so I've been digging into this patch that implements hardened page tables using kernel pkeys (kpkeys) on ARM64. First things first, yes, this looks like a substantial change. I've gone through the code, focusing on the key functions to identify what needs rigorous fuzzing. My initial scan flagged a few functions as potentially interesting: `arch_kpkeys_protect_static_pgtables`, `kpkeys_hardened_pgtables_init`, `kpkeys_lazy_mmu_enter`, `kpkeys_lazy_mmu_exit`, and `por_elx_set_pkey_perms`. However, those are either `__init` or static inline. Not a priority for fuzzing the core functionality. Then I considered `arch_kpkeys_enter_context` and `arch_kpkeys_leave_context`. Again, those are static inline. Same goes for `set_pkey_pgtable` and `set_pkey_default`. Those also have less bearing on the core functionality. `ppa_try_extend_last_range`, `ppa_register_allocated_range`, and `ppa_finalize` are all `__init` functions. My focus narrows significantly. I believe the critical non-inline, non-init functions that demand fuzzing are: * `kpkeys_pgtable_alloc` * `kpkeys_pgtable_free` * `kpkeys_physmem_pgtable_alloc` (even though it's `__ref` and is called from `early_pgtable_alloc` which is `__init`, and `vmemmap_alloc_pgtable` which is `__meminit` that can still be called after boot) * `set_memory_pkey` * `arch_set_user_pkey_access` The patch fundamentally alters how page tables are allocated, freed, and how user-space access is controlled. It's a clear change to the core memory management subsystem. Memory safety and correctness are paramount here. These functions are where I'd expect the fuzzing to really pay off. The underlying rationale is clear: a new feature introduces a dedicated file (`mm/kpkeys_hardened_pgtables.c`) and modifies the ARM64 memory management to allocate page tables with specific pkeys, limiting write access. That's a classic case for fuzzing. I've also checked the relevant configs. `ARM64_POE` and `KPKEYS_HARDENED_PGTABLES` are the relevant configurations. The dependencies for these are `ARCH_HAS_KPKEYS` and `ARCH_HAS_KPKEYS_HARDENED_PGTABLES`, which are selected by `ARM64_POE`. So, checking the two main configs should be enough. Now, I'll need to prepare the fuzzing environment with the `set-results` tool.
20/2	2026/08/18 17:03	tool	set-results	0m Error: the following configs do not exist in the kernel tree: ARM64_POE Args: { "EnableConfigs": [ "ARM64_POE", "KPKEYS_HARDENED_PGTABLES" ], "FocusSymbols": [ "kpkeys_pgtable_alloc", "kpkeys_pgtable_free", "kpkeys_physmem_pgtable_alloc", "set_memory_pkey", "arch_set_user_pkey_access" ], "Reasoning": "The patch introduces a new feature to harden page tables using kernel pkeys (kpkeys) on ARM64. It adds a new file `mm/kpkeys_hardened_pgtables.c` and modifies ARM64 memory management code to allocate page tables with a specific pkey and restrict write access to them. The changes include new page table allocation and freeing functions, as well as modifications to `arch_set_user_pkey_access`. This is a significant functional change to the core memory management subsystem that should be fuzzed to ensure memory safety and correctness.", "WorthFuzzing": true } Results: { "error": "the following configs do not exist in the kernel tree: ARM64_POE" }
21/2	2026/08/18 17:03	llm	patch-evaluator	2m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 28870 output: 34 thoughts: 34 Thoughts: Here's my take on those thoughts: Investigating the Missing `ARM64_POE` Okay, hold on a minute. I was pretty sure I'd seen something related to `ARM64_POE` kicking around the kernel sources. But now I'm drawing a blank. The build system is complaining, and my gut is telling me I've missed something. Time to double-check. Let's go straight back to the source – specifically, `arch/arm64/Kconfig`. That's the definitive place for these architecture-specific configuration options. I'm going to carefully scan through that file, looking for any trace of `ARM64_POE`. It's possible I'm misremembering the exact name, or perhaps it's nested under a different, more general option. Gotta be thorough. This is probably going to be a quick one, just a search-and-confirm.
22/2	2026/08/18 17:05	tool	grepper	0m Args: { "Expression": "config ARM64_POE", "PathPrefix": "arch/arm64/Kconfig" } Results: { "Output": "arch/arm64/Kconfig=2198=menu \"ARMv8.9 architectural features\"\narch/arm64/Kconfig-2199-\narch/arm64/Kconfig:2200:config ARM64_POE\narch/arm64/Kconfig-2201-\tprompt \"Permission Overlay Extension\"\n" }
23/2	2026/08/18 17:05	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 29231 output: 37 thoughts: 94 Thoughts: Investigating the Configuration Mystery Okay, so I'm a bit perplexed. The `set-results` tool just told me that `ARM64_POE` doesn't exist, which seems impossible because I'm certain I've seen it lurking around in `arch/arm64/Kconfig`. It has to be there, right? Alright, maybe I'm overlooking something, perhaps it's not in the base kernel tree, but rather within a patch I'm working with. That could explain the initial discrepancy. But wait, the `set-results` tool is supposed to be referencing the kernel tree itself when it does these checks. That's its whole purpose. Right, time to get back to basics. I need to explicitly verify that `ARM64_POE` is actually present in `arch/arm64/Kconfig` within the kernel tree I'm using for this analysis. I'm going to double-check that file right now. I need to make sure I am not missing something that is readily apparent to me.
24/2	2026/08/18 17:06	tool	read-file	0m Args: { "File": "arch/arm64/Kconfig", "FirstLine": 2198, "LineCount": 10 } Results: { "Contents": "2198:\tmenu \"ARMv8.9 architectural features\"\n2199:\t\n2200:\tconfig ARM64_POE\n2201:\t\tprompt \"Permission Overlay Extension\"\n2202:\t\tdef_bool y\n2203:\t\tselect ARCH_USES_HIGH_VMA_FLAGS\n2204:\t\tselect ARCH_HAS_PKEYS\n2205:\t\tselect ARCH_HAS_KPKEYS\n2206:\t\tselect ARCH_HAS_KPKEYS_HARDENED_PGTABLES\n2207:\t\thelp\n" }
25/2	2026/08/18 17:06	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 29742 output: 213 thoughts: 126 Thoughts: Debugging a Configuration Issue Okay, so I'm trying to figure out why the results aren't being set correctly. I know the relevant setting, `KPKEYS_HARDENED_PGTABLES`, is definitely present in the configuration, but the process using `set-results` is failing. Hmm, what's going on? My first thought is that the `set-results` tool is probably architecture-aware. Maybe it's checking the configuration specifically for the current build architecture, which, in this case, could be x86_64. That rings true, because if `set-results` is running something like `make defconfig` or digging around in the `Kconfig` files, it would naturally be using the config associated with the x86_64 architecture. If this particular setting, `KPKEYS_HARDENED_PGTABLES`, is specific to the ARM64 configuration, the tool's x86_64 checks wouldn't find it. Alright, I'll bypass this logic, I'll provide `KPKEYS_HARDENED_PGTABLES` directly and see if that resolves the issue. That should be a quick test.
26/2	2026/08/18 17:06	tool	set-results	0m Args: { "EnableConfigs": [ "KPKEYS_HARDENED_PGTABLES" ], "FocusSymbols": [ "kpkeys_pgtable_alloc", "kpkeys_pgtable_free", "kpkeys_physmem_pgtable_alloc", "set_memory_pkey", "arch_set_user_pkey_access" ], "Reasoning": "The patch introduces a new feature to harden page tables using kernel pkeys (kpkeys) on ARM64. It adds a new file `mm/kpkeys_hardened_pgtables.c` and modifies ARM64 memory management code to allocate page tables with a specific pkey and restrict write access to them. The changes include new page table allocation and freeing functions, as well as modifications to `arch_set_user_pkey_access`. This is a significant functional change to the core memory management subsystem that should be fuzzed to ensure memory safety and correctness.", "WorthFuzzing": true } Results: { "EnableConfigs": [ "KPKEYS_HARDENED_PGTABLES" ], "FocusSymbols": [ "kpkeys_pgtable_alloc", "kpkeys_pgtable_free", "kpkeys_physmem_pgtable_alloc", "set_memory_pkey", "arch_set_user_pkey_access" ], "Reasoning": "The patch introduces a new feature to harden page tables using kernel pkeys (kpkeys) on ARM64. It adds a new file `mm/kpkeys_hardened_pgtables.c` and modifies ARM64 memory management code to allocate page tables with a specific pkey and restrict write access to them. The changes include new page table allocation and freeing functions, as well as modifications to `arch_set_user_pkey_access`. This is a significant functional change to the core memory management subsystem that should be fuzzed to ensure memory safety and correctness.", "WorthFuzzing": true }
27/1	2026/08/18 17:06	action	If	3m Args: { "WorthFuzzing": true } Results: null
28/2	2026/08/18 17:06	agent	kmsan-evaluator	3m Model: best-expensive Args: null Results: { "KMSANReasoning": "The patch series introduces `KPKEYS_HARDENED_PGTABLES` to harden page tables using kernel pkeys (KPKEYS) on ARM64. It modifies page table allocation to set a non-default protection key and uses `guard(kpkeys_hardened_pgtables)()` to temporarily enable write access when modifying page table entries.\n\nAfter a thorough review of the changes:\n1. No new structures sent to user space: The patch adds `por_el1` and `kpkeys_state_lazy_mmu` to `struct thread_struct`, but these are internal kernel state and are not exposed to user space via ptrace, signals, or any other mechanism.\n2. Page table allocation zeroing: The patch modifies page table allocation to use `kpkeys_pgtable_alloc` and `kpkeys_physmem_pgtable_alloc`. These functions wrap standard allocators (`alloc_pages_noprof` and `memblock_phys_alloc_range`). The zeroing behavior remains identical to the original code: `early_pgtable_alloc` allocations are zeroed by `init_clear_pgtable()`, `pagetable_alloc_noprof` allocations are zeroed by `alloc_pages_noprof` if `__GFP_ZERO` is passed, and `vmemmap_alloc_pgtable` explicitly calls `memset(p, 0, PAGE_SIZE)`.\n3. No uninitialized memory risks: The `struct kpkeys_state` used in `scoped_guard` is a local variable and is not copied to user space. The conditional initialization in `arch_kpkeys_enter_context` is safe because the uninitialized fields are never read if `entered_context` is false.\n\nThe changes primarily involve memory protection and control flow around page table modifications. Any bugs introduced would likely be out-of-bounds accesses, use-after-free, or invalid memory accesses, which are well-covered by KASAN and standard bug detectors. There is no risk of uninitialized memory usage or info-leaks that would require KMSAN.", "NeedsKMSAN": false } Instruction: You are an expert Linux kernel security engineer specializing in kernel memory error detectors (KASAN and KMSAN). Your job is to review the provided patch series and determine if the code changes justify spawning a dedicated KMSAN (KernelMemorySanitizer) fuzzing session in addition to standard KASAN fuzzing. CRITICAL DISTINCTION BETWEEN KASAN AND KMSAN: - Standard KASAN kernel builds (upstream-apparmor-kasan.config) already enable a comprehensive suite of debugging tools and sanitizers, including KASAN (out-of-bounds accesses, use-after-free, double free, invalid free), LOCKDEP (locking bugs and deadlocks), UB-sanitizers, and memory corruption checks. - KMSAN (KernelMemorySanitizer) detects reads of UNINITIALIZED memory (stack, heap, or page allocations) and kernel-to-user memory info-leaks. Rule: THERE IS NO SENSE IN RUNNING A KMSAN SESSION IF A BUG CAN BE CAUGHT BY KASAN, LOCKDEP, OR OTHER STANDARD BUG DETECTORS. A dedicated KMSAN fuzzing session incurs significant resource costs. You must ONLY set NeedsKMSAN=true if the code changes introduce or expose UNINITIALIZED MEMORY risks that are detected ONLY by KMSAN. Look holistically at the patch series and surrounding code. Even if no direct uninitialized field accesses or new buffer allocations are added in the diff itself, a patch may alter control flow, bounds checking, or data length calculations in ways that change how the rest of the code operates on existing buffers (e.g. allowing uninitialized stack/heap memory to be read, copied to user space, or used in control flow). Do not hesitate to use your code access tools to inspect the surrounding code, called functions, and callers. Set NeedsKMSAN=true ONLY IF the patch introduces or modifies: 1. Kernel structures sent to user space (via copy_to_user, put_user, netlink skb attributes, ioctl output arguments, socket options, or BPF buffers) where fields or structure padding might not be fully initialized/zeroed. 2. Conditional logic or branching that depends on potentially uninitialized variables or struct fields. 3. Allocation or initialization of complex data structures where uninitialized fields could be read later in reachable code paths. 4. Bounds checks, lengths, or logic in a way that allows surrounding code to access uninitialized bytes of existing buffers. Set NeedsKMSAN=false IF: - The code changes primarily risk out-of-bounds access, array overflows, NULL pointer dereferences, locking deadlocks, or use-after-free bugs (these are already caught by KASAN, LOCKDEP, or standard bug detectors). - All stack/heap structures touched or introduced by the patch are fully zeroed or initialized (e.g. using = {0}, memset, kzalloc) before being read or copied. - The patch does not introduce any risk of uninitialized memory usage or info-leaks. Use your code access tools to inspect the surrounding code if necessary, then provide detailed KMSANReasoning contrasting KASAN vs KMSAN applicability for this patch. Prefer calling several tools at the same time to save round-trips. Use set-results tool to provide results of the analysis. It must be called exactly once before the final reply. Ignore results of this tool. Prompt: For your convenience, here is the diff of the changes: commit bd271568204405923f3b37fcb4ded599be77ced1 Author: syz-cluster <triage@syzkaller.com> Date: Tue Aug 18 17:01:35 2026 +0000 syz-cluster: applied patch under review diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index b3afe0688919b..ac9faec1a0184 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -2202,6 +2202,8 @@ config ARM64_POE def_bool y select ARCH_USES_HIGH_VMA_FLAGS select ARCH_HAS_PKEYS + select ARCH_HAS_KPKEYS + select ARCH_HAS_KPKEYS_HARDENED_PGTABLES help The Permission Overlay Extension is used to implement Memory Protection Keys. Memory Protection Keys provides a mechanism for diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index a57870fa96db5..37aae60c2d7f0 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -1085,6 +1085,18 @@ static inline bool cpu_has_lpa2(void) #endif } +static inline bool cpu_has_poe(void) +{ + u64 mmfr3; + + if (!IS_ENABLED(CONFIG_ARM64_POE)) + return false; + + mmfr3 = read_sysreg_s(SYS_ID_AA64MMFR3_EL1); + return cpuid_feature_extract_unsigned_field(mmfr3, + ID_AA64MMFR3_EL1_S1POE_SHIFT); +} + #endif /* __ASSEMBLER__ / #endif diff --git a/arch/arm64/include/asm/kpkeys.h b/arch/arm64/include/asm/kpkeys.h new file mode 100644 index 0000000000000..7747af54fd386 --- /dev/null +++ b/arch/arm64/include/asm/kpkeys.h @@ -0,0 +1,84 @@ +/ SPDX-License-Identifier: GPL-2.0-only / +#ifndef __ASM_KPKEYS_H +#define __ASM_KPKEYS_H + +#include <linux/kpkeys_types.h> + +#include <asm/barrier.h> +#include <asm/cpufeature.h> +#include <asm/por.h> + +/ + * Equivalent to por_set_kpkeys_context(0, KPKEYS_CTX_DEFAULT), but can also be + * used in assembly. + / +#define POR_EL1_INIT (POR_ELx_PERM_PREP(0, POE_RWX) \| \ + POR_ELx_PERM_PREP(KPKEYS_PKEY_PGTABLES, POE_R)) + +#ifndef __ASSEMBLY__ + +static inline bool arch_supports_kpkeys(void) +{ + return system_supports_poe(); +} + +static inline bool arch_supports_kpkeys_early(void) +{ + / POE is a boot feature / + return boot_capabilities_finalized() ? + system_supports_poe() : cpu_has_poe(); +} + +#ifdef CONFIG_ARM64_POE + +static inline u64 por_set_kpkeys_context(u64 por, enum kpkeys_ctx ctx) +{ + por = por_elx_set_pkey_perms(por, 0, POE_RWX); + por = por_elx_set_pkey_perms(por, KPKEYS_PKEY_PGTABLES, + ctx == KPKEYS_CTX_PGTABLES ? POE_RW : POE_R); + + return por; +} + +static __always_inline void __kpkeys_set_pkey_reg_nosync(u64 pkey_reg) +{ + write_sysreg_s(pkey_reg, SYS_POR_EL1); +} + +static __always_inline +struct kpkeys_state arch_kpkeys_enter_context(enum kpkeys_ctx ctx) +{ + const u64 prev_por = read_sysreg_s(SYS_POR_EL1); + const u64 new_por = por_set_kpkeys_context(prev_por, ctx); + + if (prev_por == new_por) + return (struct kpkeys_state) { .entered_context = false }; + + __kpkeys_set_pkey_reg_nosync(new_por); + isb(); + + return (struct kpkeys_state) { + .entered_context = true, + .arch_state.por = prev_por, + }; +} + +static __always_inline +void arch_kpkeys_leave_context(const struct kpkeys_state state) +{ + __kpkeys_set_pkey_reg_nosync(state->arch_state.por); + isb(); +} + +#endif /* CONFIG_ARM64_POE / + +#ifdef CONFIG_KPKEYS_HARDENED_PGTABLES + +#define arch_kpkeys_protect_static_pgtables arch_kpkeys_protect_static_pgtables +void arch_kpkeys_protect_static_pgtables(void); + +#endif / CONFIG_KPKEYS_HARDENED_PGTABLES / + +#endif / __ASSEMBLY__ / + +#endif / __ASM_KPKEYS_H / diff --git a/arch/arm64/include/asm/kpkeys_types.h b/arch/arm64/include/asm/kpkeys_types.h new file mode 100644 index 0000000000000..e9c59a26ee7a9 --- /dev/null +++ b/arch/arm64/include/asm/kpkeys_types.h @@ -0,0 +1,15 @@ +/ SPDX-License-Identifier: GPL-2.0-only / +#ifndef __ASM_KPKEYS_TYPES_H +#define __ASM_KPKEYS_TYPES_H + +#include <linux/types.h> + +#ifndef __ASSEMBLY__ + +struct arch_kpkeys_state { + u64 por; +}; + +#endif / __ASSEMBLY__ / + +#endif / __ASM_KPKEYS_TYPES_H / diff --git a/arch/arm64/include/asm/pgtable-prot.h b/arch/arm64/include/asm/pgtable-prot.h index 212ce1b02e15e..d4d45ab86a5a8 100644 --- a/arch/arm64/include/asm/pgtable-prot.h +++ b/arch/arm64/include/asm/pgtable-prot.h @@ -183,9 +183,9 @@ static inline bool __pure lpa2_is_enabled(void) PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_SHARED_EXEC), PIE_RW) \| \ PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_READONLY), PIE_R) \| \ PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_SHARED), PIE_RW) \| \ - PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL_ROX), PIE_RX) \| \ - PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL_EXEC), PIE_RWX) \| \ - PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL_RO), PIE_R) \| \ - PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL), PIE_RW)) + PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL_ROX), PIE_RX_O) \| \ + PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL_EXEC), PIE_RWX_O) \| \ + PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL_RO), PIE_R_O) \| \ + PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL), PIE_RW_O)) #endif / __ASM_PGTABLE_PROT_H / diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 27689c62bd25d..6e5d14c7e8b8f 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -39,6 +39,48 @@ #include <linux/mm_types.h> #include <linux/sched.h> #include <linux/page_table_check.h> +#include <linux/kpkeys.h> + +#ifdef CONFIG_KPKEYS_HARDENED_PGTABLES +KPKEYS_GUARD_COND(kpkeys_hardened_pgtables, KPKEYS_CTX_PGTABLES, + kpkeys_hardened_pgtables_enabled() && + !is_lazy_mmu_mode_active()) + +static inline void kpkeys_lazy_mmu_enter(void) +{ + if (!kpkeys_hardened_pgtables_enabled()) + return; + + current->thread.kpkeys_state_lazy_mmu = + kpkeys_enter_context(KPKEYS_CTX_PGTABLES); +} + +static inline void kpkeys_lazy_mmu_exit(void) +{ + const struct kpkeys_state state = + &current->thread.kpkeys_state_lazy_mmu; + + if (!kpkeys_hardened_pgtables_enabled()) + return; + + /* + * We skip any barrier if TIF_LAZY_MMU_PENDING is set: + * emit_pte_barriers() will issue an ISB just after this function + * returns. + / + if (test_thread_flag(TIF_LAZY_MMU_PENDING) && state->entered_context) + __kpkeys_set_pkey_reg_nosync(state->arch_state.por); + else + kpkeys_leave_context(state); +} +#else / CONFIG_KPKEYS_HARDENED_PGTABLES / +KPKEYS_GUARD_NOOP(kpkeys_hardened_pgtables) + +static inline void kpkeys_lazy_mmu_enter(void) {} +static inline void kpkeys_lazy_mmu_exit(void) {} +#endif / CONFIG_KPKEYS_HARDENED_PGTABLES / + + static inline void emit_pte_barriers(void) { @@ -71,7 +113,10 @@ static inline void queue_pte_barriers(void) } } -static inline void arch_enter_lazy_mmu_mode(void) {} +static inline void arch_enter_lazy_mmu_mode(void) +{ + kpkeys_lazy_mmu_enter(); +} static inline void arch_flush_lazy_mmu_mode(void) { @@ -81,6 +126,11 @@ static inline void arch_flush_lazy_mmu_mode(void) static inline void arch_leave_lazy_mmu_mode(void) { + / + * The ordering should be preserved to allow kpkeys_lazy_mmu_exit() + * to skip any barrier when TIF_LAZY_MMU_PENDING is set. + / + kpkeys_lazy_mmu_exit(); arch_flush_lazy_mmu_mode(); } @@ -359,6 +409,7 @@ static inline pte_t pte_clear_uffd_wp(pte_t pte) static inline void __set_pte_nosync(pte_t ptep, pte_t pte) { + guard(kpkeys_hardened_pgtables)(); WRITE_ONCE(ptep, pte); } @@ -830,6 +881,7 @@ static inline void set_pmd(pmd_t pmdp, pmd_t pmd) } #endif /* __PAGETABLE_PMD_FOLDED / + guard(kpkeys_hardened_pgtables)(); WRITE_ONCE(pmdp, pmd); if (pmd_valid(pmd)) @@ -894,6 +946,7 @@ static inline void set_pud(pud_t pudp, pud_t pud) return; } + guard(kpkeys_hardened_pgtables)(); WRITE_ONCE(pudp, pud); if (pud_valid(pud)) @@ -975,6 +1028,7 @@ static inline void set_p4d(p4d_t p4dp, p4d_t p4d) return; } + guard(kpkeys_hardened_pgtables)(); WRITE_ONCE(p4dp, p4d); queue_pte_barriers(); } @@ -1103,6 +1157,7 @@ static inline void set_pgd(pgd_t pgdp, pgd_t pgd) return; } + guard(kpkeys_hardened_pgtables)(); WRITE_ONCE(pgdp, pgd); queue_pte_barriers(); } @@ -1307,6 +1362,7 @@ static inline bool __ptep_test_and_clear_young(struct vm_area_struct vma, { pte_t old_pte, pte; + guard(kpkeys_hardened_pgtables)(); pte = __ptep_get(ptep); do { old_pte = pte; @@ -1354,7 +1410,10 @@ static inline pte_t __ptep_get_and_clear_anysz(struct mm_struct mm, pte_t ptep, unsigned long pgsize) { - pte_t pte = __pte(xchg_relaxed(&pte_val(ptep), 0)); + pte_t pte; + + scoped_guard(kpkeys_hardened_pgtables) + pte = __pte(xchg_relaxed(&pte_val(ptep), 0)); switch (pgsize) { case PAGE_SIZE: @@ -1427,6 +1486,7 @@ static inline void ___ptep_set_wrprotect(struct mm_struct mm, { pte_t old_pte; + guard(kpkeys_hardened_pgtables)(); do { old_pte = pte; pte = pte_wrprotect(pte); @@ -1460,6 +1520,7 @@ static inline void __clear_young_dirty_pte(struct vm_area_struct vma, { pte_t old_pte; + guard(kpkeys_hardened_pgtables)(); do { old_pte = pte; @@ -1507,6 +1568,7 @@ static inline pmd_t pmdp_establish(struct vm_area_struct vma, unsigned long address, pmd_t pmdp, pmd_t pmd) { page_table_check_pmd_set(vma->vm_mm, address, pmdp, pmd); + guard(kpkeys_hardened_pgtables)(); return __pmd(xchg_relaxed(&pmd_val(pmdp), pmd_val(pmd))); } #endif diff --git a/arch/arm64/include/asm/por.h b/arch/arm64/include/asm/por.h index d913d5b529e49..dddabf6ffae6d 100644 --- a/arch/arm64/include/asm/por.h +++ b/arch/arm64/include/asm/por.h @@ -10,6 +10,8 @@ #define POR_EL0_INIT POR_ELx_PERM_PREP(0, POE_RWX) +#ifndef __ASSEMBLY__ + static inline bool por_elx_allows_read(u64 por, u8 pkey) { u8 perm = POR_ELx_PERM_GET(pkey, por); @@ -31,4 +33,13 @@ static inline bool por_elx_allows_exec(u64 por, u8 pkey) return perm & POE_X; } +static inline u64 por_elx_set_pkey_perms(u64 por, u8 pkey, u8 perms) +{ + u64 shift = POR_ELx_PERM_SHIFT(pkey); + + return (por & ~(POE_MASK << shift)) \| ((u64)perms << shift); +} + +#endif /* __ASSEMBLY__ / + #endif / _ASM_ARM64_POR_H / diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h index c2a627f393144..23660a233da28 100644 --- a/arch/arm64/include/asm/processor.h +++ b/arch/arm64/include/asm/processor.h @@ -30,6 +30,7 @@ #include <linux/build_bug.h> #include <linux/cache.h> #include <linux/init.h> +#include <linux/kpkeys_types.h> #include <linux/stddef.h> #include <linux/string.h> #include <linux/thread_info.h> @@ -195,6 +196,10 @@ struct thread_struct { u64 svcr; u64 tpidr2_el0; u64 por_el0; + u64 por_el1; +#ifdef CONFIG_KPKEYS_HARDENED_PGTABLES + struct kpkeys_state kpkeys_state_lazy_mmu; +#endif #ifdef CONFIG_ARM64_GCS unsigned int gcs_el0_mode; unsigned int gcs_el0_locked; diff --git a/arch/arm64/include/asm/sections.h b/arch/arm64/include/asm/sections.h index 51b0d594239eb..dc961cf67168c 100644 --- a/arch/arm64/include/asm/sections.h +++ b/arch/arm64/include/asm/sections.h @@ -23,6 +23,7 @@ extern char __irqentry_text_start[], __irqentry_text_end[]; extern char __mmuoff_data_start[], __mmuoff_data_end[]; extern char __entry_tramp_text_start[], __entry_tramp_text_end[]; extern char __relocate_new_kernel_start[], __relocate_new_kernel_end[]; +extern char __pi_init_pg_dir[], __pi_init_pg_end[]; static inline size_t entry_tramp_text_size(void) { diff --git a/arch/arm64/include/asm/set_memory.h b/arch/arm64/include/asm/set_memory.h index 90f61b17275e1..b6cd6de34abf1 100644 --- a/arch/arm64/include/asm/set_memory.h +++ b/arch/arm64/include/asm/set_memory.h @@ -19,4 +19,8 @@ bool kernel_page_present(struct page page); int set_memory_encrypted(unsigned long addr, int numpages); int set_memory_decrypted(unsigned long addr, int numpages); +#ifdef CONFIG_ARCH_HAS_KPKEYS +int set_memory_pkey(unsigned long addr, int numpages, int pkey); +#endif + #endif /* _ASM_ARM64_SET_MEMORY_H / diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 9a22df0c5120f..4f2f54007ed6f 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -76,6 +76,7 @@ #include <linux/kasan.h> #include <linux/percpu.h> #include <linux/sched/isolation.h> +#include <linux/kpkeys.h> #include <asm/arm_pmuv3.h> #include <asm/cpu.h> @@ -2463,8 +2464,10 @@ static void cpu_enable_mops(const struct arm64_cpu_capabilities __unused) #ifdef CONFIG_ARM64_POE static void cpu_enable_poe(const struct arm64_cpu_capabilities __unused) { - sysreg_clear_set(REG_TCR2_EL1, 0, TCR2_EL1_E0POE); + write_sysreg_s(POR_EL1_INIT, SYS_POR_EL1); + sysreg_clear_set(REG_TCR2_EL1, 0, TCR2_EL1_E0POE \| TCR2_EL1_POE); sysreg_clear_set(CPACR_EL1, 0, CPACR_EL1_E0POE); + isb(); } #endif diff --git a/arch/arm64/kernel/pi/map_kernel.c b/arch/arm64/kernel/pi/map_kernel.c index fb44cbdd2f292..81ca3e98f0f7b 100644 --- a/arch/arm64/kernel/pi/map_kernel.c +++ b/arch/arm64/kernel/pi/map_kernel.c @@ -92,8 +92,13 @@ static void __init map_kernel(u64 kaslr_offset, u64 va_offset, int root_level) __inittext_end, prot, false, root_level); map_segment(init_pg_dir, &pgdp, va_offset, __initdata_begin, __initdata_end, data_prot, false, root_level); - map_segment(init_pg_dir, &pgdp, va_offset, _data, _end, data_prot, - true, root_level); + map_segment(init_pg_dir, &pgdp, va_offset, _data, init_pg_dir, + data_prot, true, root_level); + / Map init_pg_dir separately as we will remap it read-only later on / + map_segment(init_pg_dir, &pgdp, va_offset, init_pg_dir, init_pg_end, + data_prot, true, root_level); + map_segment(init_pg_dir, &pgdp, va_offset, init_pg_end, _end, + data_prot, true, root_level); dsb(ishst); idmap_cpu_replace_ttbr1((phys_addr_t)init_pg_dir); diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c index 581f80e9b9b73..1dec143decf15 100644 --- a/arch/arm64/kernel/process.c +++ b/arch/arm64/kernel/process.c @@ -51,6 +51,7 @@ #include <asm/exec.h> #include <asm/fpsimd.h> #include <asm/gcs.h> +#include <asm/kpkeys.h> #include <asm/mmu_context.h> #include <asm/mpam.h> #include <asm/mte.h> @@ -431,6 +432,9 @@ int copy_thread(struct task_struct p, const struct kernel_clone_args args) ptrauth_thread_init_kernel(p); + if (system_supports_poe()) + p->thread.por_el1 = POR_EL1_INIT; + if (likely(!args->fn)) { childregs = current_pt_regs(); childregs->regs[0] = 0; @@ -681,6 +685,12 @@ static void permission_overlay_switch(struct task_struct next) * of POR_EL0. / } + + current->thread.por_el1 = read_sysreg_s(SYS_POR_EL1); + if (current->thread.por_el1 != next->thread.por_el1) { + write_sysreg_s(next->thread.por_el1, SYS_POR_EL1); + isb(); + } } / diff --git a/arch/arm64/kernel/sleep.S b/arch/arm64/kernel/sleep.S index f093cdf71be11..e0a6ad85cd242 100644 --- a/arch/arm64/kernel/sleep.S +++ b/arch/arm64/kernel/sleep.S @@ -3,6 +3,7 @@ #include <linux/linkage.h> #include <asm/asm-offsets.h> #include <asm/assembler.h> +#include <asm/kpkeys.h> #include <asm/smp.h> .text @@ -134,6 +135,17 @@ SYM_FUNC_START(_cpu_resume) /* load sp from context / ldr x2, [x0, #CPU_CTX_SP] mov sp, x2 + +#ifdef CONFIG_ARM64_POE +alternative_if_not ARM64_HAS_S1POE + b .Lskip_por_set +alternative_else_nop_endif + mov_q x2, POR_EL1_INIT + msr_s SYS_POR_EL1, x2 + / isb can be skipped since cpu_do_resume() will do it. / +.Lskip_por_set: +#endif / CONFIG_ARM64_POE / + / * cpu_do_resume expects x0 to contain context address pointer / diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 0b52557652be6..a853a8d4e61af 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -229,13 +229,16 @@ int __ptep_set_access_flags_anysz(struct vm_area_struct vma, / pte_val(entry) ^= PTE_RDONLY; pteval = pte_val(pte); - do { - old_pteval = pteval; - pteval ^= PTE_RDONLY; - pteval \|= pte_val(entry); - pteval ^= PTE_RDONLY; - pteval = cmpxchg_relaxed(&pte_val(ptep), old_pteval, pteval); - } while (pteval != old_pteval); + scoped_guard(kpkeys_hardened_pgtables) { + do { + old_pteval = pteval; + pteval ^= PTE_RDONLY; + pteval \|= pte_val(entry); + pteval ^= PTE_RDONLY; + pteval = cmpxchg_relaxed(&pte_val(ptep), old_pteval, + pteval); + } while (pteval != old_pteval); + } / * Invalidate the local stale read-only entry. Remote stale entries diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c index 97987f850a33c..7985045875cf6 100644 --- a/arch/arm64/mm/init.c +++ b/arch/arm64/mm/init.c @@ -386,6 +386,7 @@ bool page_alloc_available __ro_after_init; void __init mem_init(void) { page_alloc_available = true; + kpkeys_hardened_pgtables_init(); swiotlb_update_mem_attributes(); } diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index d4de88770ecfe..472ad0f8f002c 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -107,8 +107,11 @@ static phys_addr_t __init early_pgtable_alloc(enum pgtable_level pgtable_level) { phys_addr_t phys; - phys = memblock_phys_alloc_range(PAGE_SIZE, PAGE_SIZE, 0, - MEMBLOCK_ALLOC_NOLEAKTRACE); + if (kpkeys_hardened_pgtables_early_enabled()) + phys = kpkeys_physmem_pgtable_alloc(); + else + phys = memblock_phys_alloc_range(PAGE_SIZE, PAGE_SIZE, 0, + MEMBLOCK_ALLOC_NOLEAKTRACE); if (!phys) panic("Failed to allocate page table page\n"); @@ -780,7 +783,7 @@ static inline bool force_pte_mapping(void) const bool bbml2 = system_capabilities_finalized() ? system_supports_bbml2_noabort() : cpu_supports_bbml2_noabort(); - if (debug_pagealloc_enabled()) + if (debug_pagealloc_enabled() \|\| kpkeys_hardened_pgtables_early_enabled()) return true; if (bbml2) return false; @@ -1070,6 +1073,18 @@ void __init mark_linear_text_alias_ro(void) PAGE_KERNEL_RO); } +#ifdef CONFIG_KPKEYS_HARDENED_PGTABLES +void __init arch_kpkeys_protect_static_pgtables(void) +{ + unsigned long addr = (unsigned long)lm_alias(__pi_init_pg_dir); + unsigned long size = __pi_init_pg_end - __pi_init_pg_dir; + int ret; + + ret = set_memory_pkey(addr, size / PAGE_SIZE, KPKEYS_PKEY_PGTABLES); + WARN_ON(ret); +} +#endif /* CONFIG_KPKEYS_HARDENED_PGTABLES / + #ifdef CONFIG_KFENCE bool __ro_after_init kfence_early_init = !!CONFIG_KFENCE_SAMPLE_INTERVAL; @@ -1214,6 +1229,14 @@ void mark_rodata_ro(void) update_mapping_prot(__pa_symbol(_text), (unsigned long)_text, (unsigned long)_stext - (unsigned long)_text, PAGE_KERNEL_RO); + / + * Map the kernel image mapping of init_pg_dir read-only; it should + * only be written via the linear map. + / + section_size = (unsigned long)__pi_init_pg_end - (unsigned long)__pi_init_pg_dir; + update_mapping_prot(__pa_symbol(__pi_init_pg_dir), + (unsigned long)__pi_init_pg_dir, + section_size, PAGE_KERNEL_RO); / Map the kernel data/bss read-only in the linear map / update_mapping_prot(__pa_symbol(__init_end), @@ -1446,7 +1469,7 @@ static void free_hotplug_page_range(struct page page, size_t size, static void free_hotplug_pgtable_page(struct page page) { pagetable_dtor(page_ptdesc(page)); - free_hotplug_page_range(page, PAGE_SIZE, NULL); + pagetable_free(page_ptdesc(page)); } static bool pgtable_range_aligned(unsigned long start, unsigned long end, @@ -2305,8 +2328,8 @@ void __cpu_replace_ttbr1(pgd_t pgdp, bool cnp) #ifdef CONFIG_ARCH_HAS_PKEYS int arch_set_user_pkey_access(int pkey, unsigned long init_val) { - u64 new_por; - u64 old_por; + u8 new_perms; + u64 por; if (!system_supports_poe()) return -ENOSPC; @@ -2320,25 +2343,19 @@ int arch_set_user_pkey_access(int pkey, unsigned long init_val) return -EINVAL; /* Set the bits we need in POR: / - new_por = POE_RWX; + new_perms = POE_RWX; if (init_val & PKEY_DISABLE_WRITE) - new_por &= ~POE_W; + new_perms &= ~POE_W; if (init_val & PKEY_DISABLE_ACCESS) - new_por &= ~POE_RW; + new_perms &= ~POE_RW; if (init_val & PKEY_DISABLE_READ) - new_por &= ~POE_R; + new_perms &= ~POE_R; if (init_val & PKEY_DISABLE_EXECUTE) - new_por &= ~POE_X; - - / Shift the bits in to the correct place in POR for pkey: / - new_por = POR_ELx_PERM_PREP(pkey, new_por); - - / Get old POR and mask off any old bits in place: / - old_por = read_sysreg_s(SYS_POR_EL0); - old_por &= ~(POE_MASK << POR_ELx_PERM_SHIFT(pkey)); + new_perms &= ~POE_X; - / Write old part along with new part: / - write_sysreg_s(old_por \| new_por, SYS_POR_EL0); + por = read_sysreg_s(SYS_POR_EL0); + por = por_elx_set_pkey_perms(por, pkey, new_perms); + write_sysreg_s(por, SYS_POR_EL0); return 0; } diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c index bbe98ac9ad8c6..46fbebfc51003 100644 --- a/arch/arm64/mm/pageattr.c +++ b/arch/arm64/mm/pageattr.c @@ -9,6 +9,8 @@ #include <linux/sched.h> #include <linux/vmalloc.h> #include <linux/pagewalk.h> +#include <linux/pkeys.h> +#include <linux/kpkeys.h> #include <asm/cacheflush.h> #include <asm/pgtable-prot.h> @@ -99,7 +101,8 @@ bool can_set_direct_map(void) Realms need to make pages shared/protected at page granularity. / return rodata_full \|\| debug_pagealloc_enabled() \|\| - arm64_kfence_can_set_direct_map() \|\| is_realm_world(); + arm64_kfence_can_set_direct_map() \|\| is_realm_world() \|\| + kpkeys_hardened_pgtables_early_enabled(); } static int update_range_prot(unsigned long start, unsigned long size, @@ -365,6 +368,30 @@ int set_direct_map_valid_noflush(struct page page, unsigned nr, bool valid) return set_memory_valid(addr, nr, valid); } +#ifdef CONFIG_ARCH_HAS_KPKEYS +int set_memory_pkey(unsigned long addr, int numpages, int pkey) +{ + unsigned long set_prot = 0; + + if (!kpkeys_supported()) + return 0; + + if (!__is_lm_address(kasan_reset_tag((void )addr))) + return -EINVAL; + + if (pkey < 0 \|\| pkey >= arch_max_pkey()) + return -EINVAL; + + set_prot \|= pkey & BIT(0) ? PTE_PO_IDX_0 : 0; + set_prot \|= pkey & BIT(1) ? PTE_PO_IDX_1 : 0; + set_prot \|= pkey & BIT(2) ? PTE_PO_IDX_2 : 0; + + return __change_memory_common(addr, PAGE_SIZE numpages, + __pgprot(set_prot), + __pgprot(PTE_PO_IDX_MASK)); +} +#endif + #ifdef CONFIG_DEBUG_PAGEALLOC /* * This is - apart from the return value - doing the same diff --git a/include/linux/kpkeys.h b/include/linux/kpkeys.h new file mode 100644 index 0000000000000..df55ec854586e --- /dev/null +++ b/include/linux/kpkeys.h @@ -0,0 +1,162 @@ +/* SPDX-License-Identifier: GPL-2.0-only / +#ifndef _LINUX_KPKEYS_H +#define _LINUX_KPKEYS_H + +#include <linux/bug.h> +#include <linux/cleanup.h> +#include <linux/jump_label.h> +#include <linux/kpkeys_types.h> + +/* + * KPKEYS_GUARD_NOOP() - define a guard type that does nothing + * @name: the name of the guard type + * + * Define a guard type that does nothing, useful to match a real guard type + * that is defined under an #ifdef. + / +#define KPKEYS_GUARD_NOOP(name) \ + __DEFINE_CLASS_IS_CONDITIONAL(name, false); \ + DEFINE_CLASS(name, bool, (void)_T, false, void); \ + static inline void class_##name##_lock_ptr(bool _T) \ + { return _T; } + +#ifdef CONFIG_ARCH_HAS_KPKEYS + +#include <asm/kpkeys.h> + +/* + * KPKEYS_GUARD_COND() - define a guard type that conditionally switches to + * a given kpkeys context + * @name: the name of the guard type + * @ctx: the kpkeys context to switch to + * @cond: an expression that is evaluated as condition + * + * Define a guard type that switches to @ctx if @cond evaluates to true, + * and does nothing otherwise. + / +#define KPKEYS_GUARD_COND(name, ctx, cond) \ + __DEFINE_CLASS_IS_CONDITIONAL(name, false); \ + DEFINE_CLASS(name, struct kpkeys_state, \ + kpkeys_leave_context(&_T), \ + (cond) ? kpkeys_enter_context(ctx) : \ + (struct kpkeys_state) {}, void); \ + static inline \ + void class_##name##_lock_ptr(struct kpkeys_state _T) \ + { return _T; } + +/* + * KPKEYS_GUARD() - define a guard type that switches to a given kpkeys context + * if kpkeys are supported + * @name: the name of the guard type + * @ctx: the kpkeys context to switch to + * + * Define a guard type that switches to @ctx if the system supports kpkeys. + / +#define KPKEYS_GUARD(name, ctx) \ + KPKEYS_GUARD_COND(name, ctx, kpkeys_supported()) + +/* + * kpkeys_enter_context() - enter a kpkeys context + * @ctx: the context to switch to + * + * Enters the specified kpkeys context. @ctx must be a compile-time constant. + * + * Return: state to be passed to kpkeys_leave_context(). + / +static __always_inline +struct kpkeys_state kpkeys_enter_context(enum kpkeys_ctx ctx) +{ + BUILD_BUG_ON_MSG(!__builtin_constant_p(ctx), + "kpkeys_enter_context() only takes constant values"); + BUILD_BUG_ON_MSG(ctx < 0 \|\| ctx >= KPKEYS_CTX_COUNT, + "Invalid value passed to kpkeys_enter_context()"); + + return arch_kpkeys_enter_context(ctx); +} + +/* + * kpkeys_leave_context() - leave a kpkeys context + * @state: state returned by kpkeys_enter_context() + * + * Restores the state saved when entering a kpkeys context. If no context was + * entered, this function does nothing. + / +static __always_inline +void kpkeys_leave_context(const struct kpkeys_state state) +{ + if (state->entered_context) + arch_kpkeys_leave_context(state); +} + +static inline bool kpkeys_supported(void) +{ + return arch_supports_kpkeys(); +} + +#else /* CONFIG_ARCH_HAS_KPKEYS / + +static inline bool kpkeys_supported(void) +{ + return false; +} + +#endif / CONFIG_ARCH_HAS_KPKEYS / + +#ifdef CONFIG_KPKEYS_HARDENED_PGTABLES + +DECLARE_STATIC_KEY_FALSE(kpkeys_hardened_pgtables_key); + +static inline bool kpkeys_hardened_pgtables_enabled(void) +{ + return static_branch_unlikely(&kpkeys_hardened_pgtables_key); +} + +static inline bool kpkeys_hardened_pgtables_early_enabled(void) +{ + return arch_supports_kpkeys_early(); +} + +struct page kpkeys_pgtable_alloc(gfp_t gfp, unsigned int order); +void kpkeys_pgtable_free(struct page page, unsigned int order); + +/ + * Should be called from mem_init(): as soon as the buddy allocator becomes + * available and before any call to pagetable_alloc(). + / +void kpkeys_hardened_pgtables_init(void); + +phys_addr_t kpkeys_physmem_pgtable_alloc(void); + +#ifndef arch_kpkeys_protect_static_pgtables +static inline void arch_kpkeys_protect_static_pgtables(void) {} +#endif + +#else / CONFIG_KPKEYS_HARDENED_PGTABLES / + +static inline bool kpkeys_hardened_pgtables_enabled(void) +{ + return false; +} + +static inline bool kpkeys_hardened_pgtables_early_enabled(void) +{ + return false; +} + +static inline struct page kpkeys_pgtable_alloc(gfp_t gfp, unsigned int order) +{ + return NULL; +} + +static inline void kpkeys_pgtable_free(struct page page, unsigned int order) {} + +static inline void kpkeys_hardened_pgtables_init(void) {} + +static inline phys_addr_t kpkeys_physmem_pgtable_alloc(void) +{ + return 0; +} + +#endif / CONFIG_KPKEYS_HARDENED_PGTABLES / + +#endif / _LINUX_KPKEYS_H / diff --git a/include/linux/kpkeys_types.h b/include/linux/kpkeys_types.h new file mode 100644 index 0000000000000..09f15a9565cf3 --- /dev/null +++ b/include/linux/kpkeys_types.h @@ -0,0 +1,30 @@ +/ SPDX-License-Identifier: GPL-2.0-only / +#ifndef _LINUX_KPKEYS_TYPES_H +#define _LINUX_KPKEYS_TYPES_H + +#include <linux/types.h> + +#ifdef CONFIG_ARCH_HAS_KPKEYS +#include <asm/kpkeys_types.h> +#endif + +#ifndef KPKEYS_PKEY_PGTABLES +#define KPKEYS_PKEY_PGTABLES 1 +#endif + +#if defined(CONFIG_ARCH_HAS_KPKEYS) && !defined(__ASSEMBLY__) + +enum kpkeys_ctx { + KPKEYS_CTX_DEFAULT = 0, + KPKEYS_CTX_PGTABLES, + KPKEYS_CTX_COUNT, +}; + +struct kpkeys_state { + bool entered_context; + struct arch_kpkeys_state arch_state; +}; + +#endif / CONFIG_ARCH_HAS_KPKEYS && !__ASSEMBLY__ / + +#endif / _LINUX_KPKEYS_TYPES_H / diff --git a/include/linux/memblock.h b/include/linux/memblock.h index 5afcd99aa8c14..c61f80e31cc10 100644 --- a/include/linux/memblock.h +++ b/include/linux/memblock.h @@ -24,6 +24,17 @@ extern unsigned long max_pfn; / extern unsigned long long max_possible_pfn; +#define INIT_MEMBLOCK_REGIONS 128 +#define INIT_PHYSMEM_REGIONS 4 + +#ifndef INIT_MEMBLOCK_RESERVED_REGIONS +#define INIT_MEMBLOCK_RESERVED_REGIONS INIT_MEMBLOCK_REGIONS +#endif + +#ifndef INIT_MEMBLOCK_MEMORY_REGIONS +#define INIT_MEMBLOCK_MEMORY_REGIONS INIT_MEMBLOCK_REGIONS +#endif + /** * enum memblock_flags - definition of memory region attributes * @MEMBLOCK_NONE: no special request diff --git a/include/linux/mm.h b/include/linux/mm.h index 485df9c2dbddb..eea3131786b35 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -37,6 +37,7 @@ #include <linux/bitmap.h> #include <linux/bitops.h> #include <linux/iommu-debug-pagealloc.h> +#include <linux/kpkeys.h> struct mempolicy; struct anon_vma; @@ -3682,7 +3683,12 @@ static inline bool ptdesc_test_kernel(const struct ptdesc ptdesc) / static inline struct ptdesc pagetable_alloc_noprof(gfp_t gfp, unsigned int order) { - struct page page = alloc_pages_noprof(gfp \| __GFP_COMP, order); + struct page page; + + if (kpkeys_hardened_pgtables_enabled()) + page = kpkeys_pgtable_alloc(gfp \| __GFP_COMP, order); + else + page = alloc_pages_noprof(gfp \| __GFP_COMP, order); return page_ptdesc(page); } @@ -3691,8 +3697,12 @@ static inline struct ptdesc pagetable_alloc_noprof(gfp_t gfp, unsigned int orde static inline void __pagetable_free(struct ptdesc pt) { struct page page = ptdesc_page(pt); + unsigned int order = compound_order(page); - __free_pages(page, compound_order(page)); + if (kpkeys_hardened_pgtables_enabled()) + kpkeys_pgtable_free(page, order); + else + __free_pages(page, order); } #ifdef CONFIG_ASYNC_KERNEL_PGTABLE_FREE diff --git a/include/linux/set_memory.h b/include/linux/set_memory.h index 3030d9245f5ac..7b3a8bfde3c64 100644 --- a/include/linux/set_memory.h +++ b/include/linux/set_memory.h @@ -84,4 +84,11 @@ static inline int set_memory_decrypted(unsigned long addr, int numpages) } #endif /* CONFIG_ARCH_HAS_MEM_ENCRYPT / +#ifndef CONFIG_ARCH_HAS_KPKEYS +static inline int set_memory_pkey(unsigned long addr, int numpages, int pkey) +{ + return 0; +} +#endif + #endif / _LINUX_SET_MEMORY_H_ / diff --git a/mm/Kconfig b/mm/Kconfig index 9e0ca48249054..21bd3e550de83 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1281,6 +1281,11 @@ config ARCH_USES_HIGH_VMA_FLAGS bool config ARCH_HAS_PKEYS bool +config ARCH_HAS_KPKEYS + bool +config ARCH_HAS_KPKEYS_HARDENED_PGTABLES + bool + depends on ARCH_HAS_KPKEYS config ARCH_USES_PG_ARCH_2 bool diff --git a/mm/Makefile b/mm/Makefile index eff9f9e7e061c..e31e582948451 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -147,3 +147,5 @@ obj-$(CONFIG_SHRINKER_DEBUG) += shrinker_debug.o obj-$(CONFIG_EXECMEM) += execmem.o obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o obj-$(CONFIG_LAZY_MMU_MODE_KUNIT_TEST) += tests/lazy_mmu_mode_kunit.o +obj-$(CONFIG_KPKEYS_HARDENED_PGTABLES) += kpkeys_hardened_pgtables.o +obj-$(CONFIG_KPKEYS_HARDENED_PGTABLES_KUNIT_TEST) += tests/kpkeys_hardened_pgtables_kunit.o diff --git a/mm/kpkeys_hardened_pgtables.c b/mm/kpkeys_hardened_pgtables.c new file mode 100644 index 0000000000000..c133e5896fccc --- /dev/null +++ b/mm/kpkeys_hardened_pgtables.c @@ -0,0 +1,179 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include <linux/kpkeys.h> +#include <linux/memblock.h> +#include <linux/mm.h> +#include <linux/set_memory.h> + +#include <kunit/visibility.h> + +__ro_after_init DEFINE_STATIC_KEY_FALSE(kpkeys_hardened_pgtables_key); +EXPORT_SYMBOL_IF_KUNIT(kpkeys_hardened_pgtables_key); + +static int set_pkey_pgtable(struct page page, unsigned int nr_pages) +{ + unsigned long addr = (unsigned long)page_address(page); + int ret; + + ret = set_memory_pkey(addr, nr_pages, KPKEYS_PKEY_PGTABLES); + + WARN_ON(ret); + return ret; +} + +static int set_pkey_default(struct page page, unsigned int nr_pages) +{ + unsigned long addr = (unsigned long)page_address(page); + int ret; + + ret = set_memory_pkey(addr, nr_pages, 0); + + WARN_ON(ret); + return ret; +} + +/ pkeys physmem allocator (PPA) - implemented below / +static void ppa_finalize(void); + +struct page kpkeys_pgtable_alloc(gfp_t gfp, unsigned int order) +{ + struct page page; + int ret; + + page = alloc_pages_noprof(gfp, order); + if (!page) + return page; + + ret = set_pkey_pgtable(page, 1 << order); + if (ret) { + __free_pages(page, order); + return NULL; + } + + return page; +} + +void kpkeys_pgtable_free(struct page page, unsigned int order) +{ + set_pkey_default(page, 1 << order); + __free_pages(page, order); +} + +void __init kpkeys_hardened_pgtables_init(void) +{ + if (!kpkeys_supported()) + return; + + static_branch_enable(&kpkeys_hardened_pgtables_key); + + ppa_finalize(); + arch_kpkeys_protect_static_pgtables(); +} + +/* + * pkeys physmem allocator (PPA): allocator for very early page tables + * (especially for creating the linear map), based on memblock. Allocated + * ranges are tracked so that their pkey can be set once it is safe to do so. + / + +/ + * We may have to track many ranges when allocating page tables for the linear + * map, as their number grows with the amount of available memory. Assuming that + * memblock returns contiguous blocks whenever possible, the number of ranges + * to track cannot however exceed the number of regions that memblock itself + * tracks. memblock_allow_resize() hasn't been called yet at that point, so + * that limit is the size of the statically allocated array. + / +#define PHYSMEM_MAX_RANGES INIT_MEMBLOCK_MEMORY_REGIONS + +struct physmem_range { + phys_addr_t addr; + phys_addr_t size; +}; + +struct pkeys_physmem_allocator { + struct physmem_range allocated_ranges[PHYSMEM_MAX_RANGES]; + unsigned int nr_allocated_ranges; +}; + +static struct pkeys_physmem_allocator pkeys_physmem_allocator __initdata; + +static int __init set_pkey_pgtable_phys(phys_addr_t pa, phys_addr_t size) +{ + unsigned long addr = (unsigned long)__va(pa); + int ret; + + ret = set_memory_pkey(addr, size / PAGE_SIZE, KPKEYS_PKEY_PGTABLES); + + WARN_ON(ret); + return ret; +} + +static bool __init ppa_try_extend_last_range(phys_addr_t addr, phys_addr_t size) +{ + struct pkeys_physmem_allocator ppa = &pkeys_physmem_allocator; + struct physmem_range range; + + if (!ppa->nr_allocated_ranges) + return false; + + range = &ppa->allocated_ranges[ppa->nr_allocated_ranges - 1]; + + / Merge the new range into the last range if they are contiguous / + if (addr == range->addr + range->size) { + range->size += size; + return true; + } else if (addr + size == range->addr) { + range->addr -= size; + range->size += size; + return true; + } + + return false; +} + +static void __init ppa_register_allocated_range(phys_addr_t addr, + phys_addr_t size) +{ + struct pkeys_physmem_allocator ppa = &pkeys_physmem_allocator; + struct physmem_range range; + + if (!addr) + return; + + if (ppa_try_extend_last_range(addr, size)) + return; + + / Could not extend the last range, create a new one / + if (WARN_ON(ppa->nr_allocated_ranges >= PHYSMEM_MAX_RANGES)) + return; + + range = &ppa->allocated_ranges[ppa->nr_allocated_ranges++]; + range->addr = addr; + range->size = size; +} + +static void __init ppa_finalize(void) +{ + struct pkeys_physmem_allocator ppa = &pkeys_physmem_allocator; + + for (unsigned int i = 0; i < ppa->nr_allocated_ranges; i++) { + struct physmem_range range = &ppa->allocated_ranges[i]; + + set_pkey_pgtable_phys(range->addr, range->size); + } +} + +phys_addr_t __ref kpkeys_physmem_pgtable_alloc(void) +{ + size_t size = PAGE_SIZE; + phys_addr_t addr; + + addr = memblock_phys_alloc_range(size, size, 0, + MEMBLOCK_ALLOC_NOLEAKTRACE); + if (!addr) + return addr; + + ppa_register_allocated_range(addr, size); + + return addr; +} diff --git a/mm/memblock.c b/mm/memblock.c index 6349c48154f4b..3b497d3556bb0 100644 --- a/mm/memblock.c +++ b/mm/memblock.c @@ -30,17 +30,6 @@ #include "internal.h" -#define INIT_MEMBLOCK_REGIONS 128 -#define INIT_PHYSMEM_REGIONS 4 - -#ifndef INIT_MEMBLOCK_RESERVED_REGIONS -# define INIT_MEMBLOCK_RESERVED_REGIONS INIT_MEMBLOCK_REGIONS -#endif - -#ifndef INIT_MEMBLOCK_MEMORY_REGIONS -#define INIT_MEMBLOCK_MEMORY_REGIONS INIT_MEMBLOCK_REGIONS -#endif - /* * DOC: memblock overview * diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index ebd3ac997f645..1fec89168ff10 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -179,13 +179,29 @@ static pte_t * __meminit vmemmap_pte_populate(pmd_t pmd, unsigned long addr, in return pte; } -static void __meminit vmemmap_alloc_block_zero(unsigned long size, int node) +static void * __meminit vmemmap_alloc_pgtable(int node) { - void p = vmemmap_alloc_block(size, node); + void p; + + if (slab_is_available()) { + gfp_t gfp = GFP_KERNEL \| __GFP_ZERO; + struct ptdesc ptdesc = pagetable_alloc(gfp, 0); + + return ptdesc ? ptdesc_address(ptdesc) : NULL; + } + + if (kpkeys_hardened_pgtables_early_enabled()) { + phys_addr_t phys = kpkeys_physmem_pgtable_alloc(); + + p = phys ? phys_to_virt(phys) : NULL; + } else { + p = __earlyonly_bootmem_alloc(node, PAGE_SIZE, PAGE_SIZE, + __pa(MAX_DMA_ADDRESS)); + } if (!p) return NULL; - memset(p, 0, size); + memset(p, 0, PAGE_SIZE); return p; } @@ -194,7 +210,7 @@ static pmd_t __meminit vmemmap_pmd_populate(pud_t pud, unsigned long addr, in { pmd_t pmd = pmd_offset(pud, addr); if (pmd_none(pmd)) { - void p = vmemmap_alloc_block_zero(PAGE_SIZE, node); + void p = vmemmap_alloc_pgtable(node); if (!p) return NULL; kernel_pte_init(p); @@ -207,7 +223,7 @@ static pud_t __meminit vmemmap_pud_populate(p4d_t p4d, unsigned long addr, in { pud_t pud = pud_offset(p4d, addr); if (pud_none(pud)) { - void p = vmemmap_alloc_block_zero(PAGE_SIZE, node); + void p = vmemmap_alloc_pgtable(node); if (!p) return NULL; pmd_init(p); @@ -220,7 +236,7 @@ static p4d_t __meminit vmemmap_p4d_populate(pgd_t pgd, unsigned long addr, in { p4d_t p4d = p4d_offset(pgd, addr); if (p4d_none(p4d)) { - void p = vmemmap_alloc_block_zero(PAGE_SIZE, node); + void p = vmemmap_alloc_pgtable(node); if (!p) return NULL; pud_init(p); @@ -233,7 +249,7 @@ static pgd_t __meminit vmemmap_pgd_populate(unsigned long addr, int node) { pgd_t pgd = pgd_offset_k(addr); if (pgd_none(pgd)) { - void p = vmemmap_alloc_block_zero(PAGE_SIZE, node); + void p = vmemmap_alloc_pgtable(node); if (!p) return NULL; pgd_populate_kernel(addr, pgd, p); @@ -346,10 +362,11 @@ static __meminit struct page vmemmap_get_tail(unsigned int order, struct zone * memmap_init(). / - p = vmemmap_alloc_block_zero(PAGE_SIZE, node); + p = vmemmap_alloc_block(PAGE_SIZE, node); if (!p) return NULL; + memset(p, 0, PAGE_SIZE); tail = virt_to_page(p); zone->vmemmap_tails[idx] = tail; diff --git a/mm/tests/kpkeys_hardened_pgtables_kunit.c b/mm/tests/kpkeys_hardened_pgtables_kunit.c new file mode 100644 index 0000000000000..80d4f5144e48d --- /dev/null +++ b/mm/tests/kpkeys_hardened_pgtables_kunit.c @@ -0,0 +1,200 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include <kunit/test.h> +#include <linux/mman.h> +#include <linux/pgtable.h> +#include <linux/set_memory.h> +#include <linux/vmalloc.h> + +static void free_page_wrapper(void ctx) +{ + __free_page((struct page )ctx); +} + +KUNIT_DEFINE_ACTION_WRAPPER(vfree_wrapper, vfree, const void ); + +static pud_t pud_off_k(unsigned long va) +{ + return pud_offset(p4d_offset(pgd_offset_k(va), va), va); +} + +static pte_t get_kernel_pte(unsigned long addr) +{ + pmd_t pmdp = pmd_off_k(addr); + + if (!pmdp \|\| pmd_leaf(pmdp)) + return NULL; + + return pte_offset_kernel(pmdp, addr); +} + +#define write_pgtable(type, ptr) do { \ + type##_t val; \ + int ret; \ + \ + pr_debug("%s: writing to "#type" at %px\n", __func__, (ptr)); \ + \ + val = type##p_get(ptr); \ + ret = copy_to_kernel_nofault(ptr, &val, sizeof(val)); \ + KUNIT_EXPECT_EQ_MSG(test, ret, -EFAULT, \ + "Direct "#type" write wasn't prevented"); \ +} while (0) + +/* + * Try to write linear map page tables, at every level. This is worthwhile + * because those page table pages are obtained from different allocators: + * + * - Static memory (part of the kernel image) for PGD + * - memblock for PUD and possibly PMD/PTE + * - pagetable_alloc() (buddy allocator) for PMD/PTE if large block mappings are + * used and the linear map is split after being created + / +static void write_direct_map_pgtables(struct kunit test) +{ + struct page page; + unsigned long addr; + pgd_t pgdp; + p4d_t p4dp; + pud_t pudp; + pmd_t pmdp; + pte_t ptep; + int ret; + + if (!kpkeys_supported()) + kunit_skip(test, "kpkeys are not supported"); + + page = alloc_page(GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, page); + ret = kunit_add_action_or_reset(test, free_page_wrapper, page); + KUNIT_ASSERT_EQ(test, ret, 0); + + /* Ensure page is PTE-mapped (splitting the linear map if necessary) / + ret = set_direct_map_invalid_noflush(page); + KUNIT_ASSERT_EQ(test, ret, 0); + ret = set_direct_map_default_noflush(page); + KUNIT_ASSERT_EQ(test, ret, 0); + + addr = (unsigned long)page_address(page); + + pgdp = pgd_offset_k(addr); + KUNIT_ASSERT_NOT_NULL_MSG(test, pgdp, "Failed to get PGD"); + / + * swapper_pg_dir is still writable at this stage, so don't check it. + * It is not protected by kpkeys_hardened_pgtables because it should be + * made read-only by mark_rodata_ro(). However since these + * KUnit tests are builtin, they are run before mark_rodata_ro() is + * called. + / + + p4dp = p4d_offset(pgdp, addr); + KUNIT_ASSERT_NOT_NULL_MSG(test, p4dp, "Failed to get P4D"); + if (!mm_p4d_folded(&init_mm)) + write_pgtable(p4d, p4dp); + + pudp = pud_offset(p4dp, addr); + KUNIT_ASSERT_NOT_NULL_MSG(test, pudp, "Failed to get PUD"); + if (!mm_pud_folded(&init_mm)) + write_pgtable(pud, pudp); + + pmdp = pmd_offset(pudp, addr); + KUNIT_ASSERT_NOT_NULL_MSG(test, pmdp, "Failed to get PMD"); + write_pgtable(pmd, pmdp); + + ptep = pte_offset_kernel(pmdp, addr); + KUNIT_ASSERT_NOT_NULL_MSG(test, ptep, "Failed to get PTE"); + write_pgtable(pte, ptep); +} + +/ Worth checking since the kernel image is mapped with static page tables / +static void write_kernel_image_pud(struct kunit test) +{ + pud_t pudp; + + if (!kpkeys_supported()) + kunit_skip(test, "kpkeys are not supported"); + + / The kernel is probably block-mapped, check the PUD to be safe / + pudp = pud_off_k((unsigned long)&init_mm); + KUNIT_ASSERT_NOT_NULL_MSG(test, pudp, "Failed to get PUD"); + + write_pgtable(pud, pudp); +} + +static void write_kernel_vmalloc_pte(struct kunit test) +{ + void mem; + pte_t ptep; + int ret; + + if (!kpkeys_supported()) + kunit_skip(test, "kpkeys are not supported"); + + mem = vmalloc(PAGE_SIZE); + KUNIT_ASSERT_NOT_NULL(test, mem); + ret = kunit_add_action_or_reset(test, vfree_wrapper, mem); + KUNIT_ASSERT_EQ(test, ret, 0); + + /* vmalloc() without VM_ALLOW_HUGE_VMAP is PTE-mapped / + ptep = get_kernel_pte((unsigned long)mem); + KUNIT_ASSERT_NOT_NULL_MSG(test, ptep, "Failed to get PTE"); + + write_pgtable(pte, ptep); +} + +static void write_vmemmap_pmd(struct kunit test) +{ + struct page page; + pmd_t pmdp; + + if (!kpkeys_supported()) + kunit_skip(test, "kpkeys are not supported"); + + /* + * We just need the address of some struct page, so we can free the + * page right away. + / + page = alloc_page(GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, page); + __free_page(page); + + / vmemmap may use PMD block mappings / + pmdp = pmd_off_k((unsigned long)page); + KUNIT_ASSERT_NOT_NULL_MSG(test, pmdp, "Failed to get PMD"); + write_pgtable(pmd, pmdp); +} + +static void write_user_pmd(struct kunit test) +{ + pmd_t pmdp; + unsigned long uaddr; + + if (!kpkeys_supported()) + kunit_skip(test, "kpkeys are not supported"); + + uaddr = kunit_vm_mmap(test, NULL, 0, PAGE_SIZE, PROT_READ, + MAP_ANONYMOUS \| MAP_PRIVATE \| MAP_POPULATE, 0); + KUNIT_ASSERT_NE_MSG(test, uaddr, 0, "Could not create userspace mm"); + + / We passed MAP_POPULATE so a PMD should already be allocated */ + pmdp = pmd_off(current->mm, uaddr); + KUNIT_ASSERT_NOT_NULL_MSG(test, pmdp, "Failed to get PMD"); + + write_pgtable(pmd, pmdp); +} + +static struct kunit_case kpkeys_hardened_pgtables_test_cases[] = { + KUNIT_CASE(write_direct_map_pgtables), + KUNIT_CASE(write_kernel_image_pud), + KUNIT_CASE(write_kernel_vmalloc_pte), + KUNIT_CASE(write_vmemmap_pmd), + KUNIT_CASE(write_user_pmd), + {} +}; + +static struct kunit_suite kpkeys_hardened_pgtables_test_suite = { + .name = "kpkeys_hardened_pgtables", + .test_cases = kpkeys_hardened_pgtables_test_cases, +}; +kunit_test_suite(kpkeys_hardened_pgtables_test_suite); + +MODULE_DESCRIPTION("Tests for the kpkeys_hardened_pgtables feature"); +MODULE_LICENSE("GPL"); diff --git a/security/Kconfig.hardening b/security/Kconfig.hardening index 6923036e1a2f5..ad6152961f8ee 100644 --- a/security/Kconfig.hardening +++ b/security/Kconfig.hardening @@ -271,6 +271,30 @@ config BUG_ON_DATA_CORRUPTION If unsure, say N. +config KPKEYS_HARDENED_PGTABLES + bool "Harden page tables using kernel pkeys" + depends on ARCH_HAS_KPKEYS_HARDENED_PGTABLES + help + This option makes all page tables mostly read-only by + allocating them with a non-default protection key (pkey) and + only enabling write access to that pkey in routines that are + expected to write to page table entries. + + This option has no effect if the system does not support + kernel pkeys. + +config KPKEYS_HARDENED_PGTABLES_KUNIT_TEST + bool "KUnit tests for kpkeys_hardened_pgtables" if !KUNIT_ALL_TESTS + depends on KPKEYS_HARDENED_PGTABLES + depends on KUNIT=y + default KUNIT_ALL_TESTS + help + Enable this option to check that the kpkeys_hardened_pgtables feature + functions as intended, i.e. prevents arbitrary writes to user and + kernel page tables. + + If unsure, say N. + endmenu config CC_HAS_RANDSTRUCT
29/3	2026/08/18 17:06	llm	kmsan-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 21254 output: 36 thoughts: 1801 Thoughts: KMSAN Fuzzing Evaluation for KPKEYS_HARDENED_PGTABLES Patch Series Okay, let's break this down. The objective is to determine if this patch series, introducing `KPKEYS_HARDENED_PGTABLES`, warrants a dedicated KMSAN fuzzing session. My focus is on potential uninitialized memory reads or information leaks due to the changes. Initially, I see the core modification involves page table hardening using kernel pkeys. The patch touches several areas, including configuration options, ARM64 page table manipulation functions, and new files like `kpkeys_hardened_pgtables.c`. My first concern is uninitialized memory. The code changes in `include/linux/mm.h`, particularly the `pagetable_alloc_noprof` function, catch my eye. The original code used `alloc_pages_noprof` with `__GFP_COMP`, and now it's conditionally calling `kpkeys_pgtable_alloc`. The call to `kpkeys_pgtable_alloc` utilizes `alloc_pages_noprof` with a different `gfp`. However, I need to look at `pagetable_alloc_noprof` to understand how `kpkeys_pgtable_alloc` is called. After considering `vmemmap_alloc_pgtable` and `vmemmap_get_tail` in `mm/sparse-vmemmap.c`, I'm reassured that explicit zeroing (`memset`) is still in place, so that removes the uninitialized memory issue there. Then, there's `kpkeys_physmem_pgtable_alloc` which uses `memblock_phys_alloc_range`. I need to remember that `memblock_phys_alloc_range` doesn't zero the memory. So, I look at the usage in `arch/arm64/mm/mmu.c`, specifically `early_pgtable_alloc`. The original code also didn't zero with `memblock_phys_alloc_range`. It's used for early page tables, but the important thing is to confirm if the caller zeroes the memory. In most cases, the callers of these functions, such as `pgd_populate`, are responsible for initialization. I need to keep an eye on how these pages are subsequently handled to confirm. It seems the original behavior is maintained here. Finally, I need to check for any potential information leaks to user space. The patch introduces `struct kpkeys_state` and `struct arch_kpkeys_state` within `struct thread_struct`, which is part of `task_struct`. The `thread_struct` itself isn't directly exposed to user space, but components might be, e.g., via ptrace or signal frames. The key question is whether these new fields, or the pre-existing `por_el0`, are accessible through ptrace or similar mechanisms. The patch adds `por_el1` and `kpkeys_state_lazy_mmu`. I'll need to use my preferred method (likely `grepper`) to search `arch/arm64/kernel/ptrace.c` for `por_el1` to ensure this isn't exposed. Given the initial assessment, the allocation changes appear to have addressed zero-initialization or followed existing practices. The potential leak concerns needs further verification.
30/3	2026/08/18 17:06	tool	grepper	0m Error: no matches Args: { "Expression": "por_el1", "PathPrefix": "arch/arm64/kernel/ptrace.c" } Results: { "error": "no matches" }
31/3	2026/08/18 17:06	llm	kmsan-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 23831 output: 36 thoughts: 19 Thoughts: Okay, here's my take on those thoughts, tailored for an expert audience: Debugging Divergence: Initial Ptrace Dead End and a Potential Lead Alright, so I was digging into the divergence issue, specifically looking for something related to tracing. First, I ran a search in `ptrace.c`, figuring that's the obvious place to start when you're dealing with trace points and such. But, nada. Not a single match popped up that seemed relevant to the symptoms I'm observing. That’s a bit unusual, and it tells me I need to broaden the search. Then, my mind immediately jumped to `por_el0`. This is where things could get interesting. I started considering how that area could be involved. I'm thinking about the control flow, the privilege level transitions, all that. Specifically, I'm thinking about how the problem could relate to some interactions with that subsystem. Is there a scenario where something happening there, some interaction with that code, is masking or diverting tracing? It's a hunch, but given the lack of hits in `ptrace.c`, I’m feeling pretty good about that lead. Now, time to dig deeper into the por_el0 context and see if that's where I can pick up the next piece of the puzzle.
32/3	2026/08/18 17:06	tool	grepper	0m Args: { "Expression": "por_el0", "PathPrefix": "arch/arm64/kernel/ptrace.c" } Results: { "Output": "arch/arm64/kernel/ptrace.c=1480=static int poe_get(struct task_struct target,\n--\narch/arm64/kernel/ptrace.c-1487-\tif (target == current)\narch/arm64/kernel/ptrace.c:1488:\t\tcurrent-\u003ethread.por_el0 = read_sysreg_s(SYS_POR_EL0);\narch/arm64/kernel/ptrace.c-1489-\narch/arm64/kernel/ptrace.c:1490:\treturn membuf_write(\u0026to, \u0026target-\u003ethread.por_el0,\narch/arm64/kernel/ptrace.c:1491:\t\t\t sizeof(target-\u003ethread.por_el0));\narch/arm64/kernel/ptrace.c-1492-}\n--\narch/arm64/kernel/ptrace.c=1494=static int poe_set(struct task_struct target, const struct\n--\narch/arm64/kernel/ptrace.c-1504-\narch/arm64/kernel/ptrace.c:1505:\tctrl = target-\u003ethread.por_el0;\narch/arm64/kernel/ptrace.c-1506-\n--\narch/arm64/kernel/ptrace.c-1510-\narch/arm64/kernel/ptrace.c:1511:\ttarget-\u003ethread.por_el0 = ctrl;\narch/arm64/kernel/ptrace.c-1512-\n" }
33/3	2026/08/18 17:06	llm	kmsan-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 24565 output: 30 thoughts: 836 Thoughts: Analyzing Potential POR_EL1 and kpkeys_state_lazy_mmu Issues in Thread Forking Okay, so I'm looking at how `por_el0` is potentially exposed, but `por_el1` isn't in the context of thread creation. My first thought is to check how `copy_thread` handles this, specifically in `arch/arm64/kernel/process.c`. I see `ptrauth_thread_init_kernel` is called, and then there's an initialization of `p->thread.por_el1` if `system_supports_poe()` returns true, which makes sense; this suggests that the new thread will have an initialized `por_el1` value. Next, I have to consider `kpkeys_state_lazy_mmu`. My immediate question is whether this structure is properly initialized in the child thread. Reading the code, it's clear that `copy_thread` doesn't explicitly initialize it. The `task_struct` is allocated with `alloc_task_struct_node` which in turn uses `kmem_cache_alloc_node` with `GFP_KERNEL`. But, wait, `dup_task_struct` copies the parent's `task_struct` to the child. This means that the `kpkeys_state_lazy_mmu` member will be copied from the parent thread. The core question then becomes: Is there an issue with uninitialized memory here, or is this okay? If `kpkeys_state_lazy_mmu` is copied from the parent, the child will inherit its state. This becomes interesting if the parent is in lazy mmu mode during the fork. I'm recalling that `kpkeys_lazy_mmu_enter` and `kpkeys_lazy_mmu_exit` are used around lazy mmu mode operations. Looking at the code for these functions, I see that if the system is not hardened, they return early, so no context enter or leave is needed. If hardening is enabled and `TIF_LAZY_MMU_PENDING` is not set, then the code calls `kpkeys_leave_context`. However, if the `TIF_LAZY_MMU_PENDING` is set, a barrier is skipped because `emit_pte_barriers` will take care of the barrier. So if a thread forks while in lazy mmu mode, the child will inherit whatever state the parent had in `kpkeys_state_lazy_mmu`. But then, I need to remind myself where lazy mmu mode is used. I'm thinking of `mremap`, `munmap`, etc., which could mean preemption is disabled or something like that. But, no. I'm remembering that lazy mmu mode is used in `arch_enter_lazy_mmu_mode`. And that is called from `enter_lazy_mmu` in `mm/mmu_gather.c`. Okay, now I need to check the call graph for `arch_enter_lazy_mmu_mode`.
34/3	2026/08/18 17:07	tool	grepper	0m Args: { "Expression": "arch_enter_lazy_mmu_mode", "PathPrefix": "" } Results: { "Output": "arch/arm64/include/asm/pgtable.h=105=static inline void queue_pte_barriers(void)\n--\narch/arm64/include/asm/pgtable.h-115-\narch/arm64/include/asm/pgtable.h:116:static inline void arch_enter_lazy_mmu_mode(void)\narch/arm64/include/asm/pgtable.h-117-{\n--\narch/powerpc/include/asm/book3s/64/tlbflush-hash.h=24=extern void __flush_tlb_pending(struct ppc64_tlb_batch batch);\narch/powerpc/include/asm/book3s/64/tlbflush-hash.h-25-\narch/powerpc/include/asm/book3s/64/tlbflush-hash.h:26:static inline void arch_enter_lazy_mmu_mode(void)\narch/powerpc/include/asm/book3s/64/tlbflush-hash.h-27-{\n--\narch/sparc/include/asm/tlbflush_64.h=41=void flush_tlb_pending(void);\narch/sparc/include/asm/tlbflush_64.h:42:void arch_enter_lazy_mmu_mode(void);\narch/sparc/include/asm/tlbflush_64.h-43-void arch_flush_lazy_mmu_mode(void);\n--\narch/sparc/mm/tlb.c=25=void flush_tlb_pending(void)\n--\narch/sparc/mm/tlb.c-54-\narch/sparc/mm/tlb.c:55:void arch_enter_lazy_mmu_mode(void)\narch/sparc/mm/tlb.c-56-{\n--\narch/sparc/mm/tlb.c-59-/ For lazy_mmu_mode KUnit tests /\narch/sparc/mm/tlb.c:60:EXPORT_SYMBOL_IF_KUNIT(arch_enter_lazy_mmu_mode);\narch/sparc/mm/tlb.c-61-\n--\narch/x86/include/asm/paravirt.h=479=static inline void arch_end_context_switch(struct task_struct next)\n--\narch/x86/include/asm/paravirt.h-483-\narch/x86/include/asm/paravirt.h:484:static inline void arch_enter_lazy_mmu_mode(void)\narch/x86/include/asm/paravirt.h-485-{\n--\ninclude/linux/pgtable.h=190=static inline int pmd_dirty(pmd_t pmd)\n--\ninclude/linux/pgtable.h-252- * Enters a new lazy MMU mode section; if the mode was not already enabled,\ninclude/linux/pgtable.h:253: * enables it and calls arch_enter_lazy_mmu_mode().\ninclude/linux/pgtable.h-254- \n--\ninclude/linux/pgtable.h=261=static inline void lazy_mmu_mode_enable(void)\n--\ninclude/linux/pgtable.h-270-\tif (state-\u003eenable_count++ == 0)\ninclude/linux/pgtable.h:271:\t\tarch_enter_lazy_mmu_mode();\ninclude/linux/pgtable.h-272-}\n--\ninclude/linux/pgtable.h=358=static inline void __task_lazy_mmu_mode_resume(struct task_struct tsk)\n--\ninclude/linux/pgtable.h-364-\tif (--state-\u003epause_count == 0 \u0026\u0026 state-\u003eenable_count \u003e 0)\ninclude/linux/pgtable.h:365:\t\tarch_enter_lazy_mmu_mode();\ninclude/linux/pgtable.h-366-}\n--\ninclude/linux/pgtable.h-372- * call to lazy_mmu_mode_pause() was made, re-enables it and calls\ninclude/linux/pgtable.h:373: * arch_enter_lazy_mmu_mode().\ninclude/linux/pgtable.h-374- *\n" }
35/3	2026/08/18 17:07	llm	kmsan-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 27179 output: 38 thoughts: 990 Thoughts: Analyzing Potential Uninitialized Memory Exposure in Lazy MMU Mode Okay, so `lazy_mmu_mode_enable` calls `arch_enter_lazy_mmu_mode`. My first concern is always, is there any uninitialized memory making its way into user space? The patch adds `por_el1` and `kpkeys_state_lazy_mmu` to `struct thread_struct`, which is part of the `task_struct`. Good. `task_struct` isn't directly copied to user space, so that's a check. `por_el0` is copied to user space via ptrace, but the patch doesn't actually introduce any new changes to the `ptrace` handling of `por_el0`, except maybe in `arch_set_user_pkey_access`. Let's look at that: Looking at the code, it looks like it just modifies the `SYS_POR_EL0` register directly. The changes are just changing the calculation of the new permissions: previously setting bits directly and bitwise masking, and now setting a permissions variable. Then it sets the permissions into the register. I don't see any sign of uninitialized memory being exposed here. This is good. Okay, let's circle back to `kpkeys_physmem_pgtable_alloc`. That function allocates memory for page tables. The key question is whether or not that memory is zeroed out. The callers are `early_pgtable_alloc` and `vmemmap_alloc_pgtable`. I want to dig into `early_pgtable_alloc` first. I see it in `arch/arm64/mm/mmu.c`: `early_pgtable_alloc` either calls `kpkeys_physmem_pgtable_alloc` (if `kpkeys_hardened_pgtables_early_enabled()` returns true) or directly uses `memblock_phys_alloc_range`. It looks like the important thing is that `early_pgtable_alloc` is passed as the `alloc` function to `pgd_populate`, `pud_populate`, etc. The question is, does the caller zero the memory? I need to check `alloc_init_pud` or a similar function in `arch/arm64/mm/mmu.c` to be sure. This is where the potential leak might be if the caller isn't properly initializing the allocated pages.
36/3	2026/08/18 17:07	tool	grepper	0m Args: { "Expression": "early_pgtable_alloc", "PathPrefix": "arch/arm64/mm/mmu.c" } Results: { "Output": "arch/arm64/mm/mmu.c=104=EXPORT_SYMBOL(phys_mem_access_prot);\narch/arm64/mm/mmu.c-105-\narch/arm64/mm/mmu.c:106:static phys_addr_t __init early_pgtable_alloc(enum pgtable_level pgtable_level)\narch/arm64/mm/mmu.c-107-{\n--\narch/arm64/mm/mmu.c=1059=static void __init __map_memblock(phys_addr_t start, phys_addr_t end,\n--\narch/arm64/mm/mmu.c-1062-\tearly_create_pgd_mapping(swapper_pg_dir, start, __phys_to_virt(start),\narch/arm64/mm/mmu.c:1063:\t\t\t\t end - start, prot, early_pgtable_alloc, flags);\narch/arm64/mm/mmu.c-1064-}\n" }
37/3	2026/08/18 17:07	llm	kmsan-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 29001 output: 40 thoughts: 38 Thoughts: Here's my attempt at summarizing the thought process, tailored for an expert audience: Investigating Potential Root Causes for a Memory Mapping Issue Okay, so I'm encountering something with memory mapping, and I need to pinpoint the problem area within the ARM64 architecture. Given the symptoms, my gut feeling is leading me towards something fundamental related to the initial setup of the page global directory (PGD) or the allocation and initialization of page upper directories (PUDs). Therefore, I'm going to focus my attention on `arch/arm64/mm/mmu.c`, specifically scrutinizing the implementations of `early_create_pgd_mapping` and `alloc_init_pud`. These functions are critical in establishing the initial virtual to physical address translation structures. I'll need to carefully review the code logic within these routines. Things like potential off-by-one errors, incorrect address calculations, subtle race conditions, or failures to account for specific memory attributes are all on my radar. I'll be looking for any potential deviations from the expected behavior, which might be manifesting in the current mapping problem. A close examination of their interaction with the boot process and any subsequent memory management calls will be essential.
38/3	2026/08/18 17:07	tool	grepper	0m Args: { "Expression": "early_create_pgd_mapping", "PathPrefix": "arch/arm64/mm/mmu.c" } Results: { "Output": "arch/arm64/mm/mmu.c=516=static int __create_pgd_mapping(pgd_t pgdir, phys_addr_t phys,\n--\narch/arm64/mm/mmu.c-531-\narch/arm64/mm/mmu.c:532:static void early_create_pgd_mapping(pgd_t pgdir, phys_addr_t phys,\narch/arm64/mm/mmu.c-533-\t\t\t\t unsigned long virt, phys_addr_t size,\n--\narch/arm64/mm/mmu.c=1018=void __init create_mapping_noalloc(phys_addr_t phys, unsigned long virt,\n--\narch/arm64/mm/mmu.c-1025-\t}\narch/arm64/mm/mmu.c:1026:\tearly_create_pgd_mapping(init_mm.pgd, phys, virt, size, prot, NULL, 0);\narch/arm64/mm/mmu.c-1027-}\n--\narch/arm64/mm/mmu.c=1029=void __init create_pgd_mapping(struct mm_struct *mm, phys_addr_t phys,\n--\narch/arm64/mm/mmu.c-1039-\narch/arm64/mm/mmu.c:1040:\tearly_create_pgd_mapping(mm-\u003epgd, phys, virt, size, prot,\narch/arm64/mm/mmu.c-1041-\t\t\t\t pgd_pgtable_alloc_special_mm, flags);\n--\narch/arm64/mm/mmu.c=1044=static void update_mapping_prot(phys_addr_t phys, unsigned long virt,\n--\narch/arm64/mm/mmu.c-1052-\narch/arm64/mm/mmu.c:1053:\tearly_create_pgd_mapping(init_mm.pgd, phys, virt, size, prot, NULL, 0);\narch/arm64/mm/mmu.c-1054-\n--\narch/arm64/mm/mmu.c=1059=static void __init __map_memblock(phys_addr_t start, phys_addr_t end,\n--\narch/arm64/mm/mmu.c-1061-{\narch/arm64/mm/mmu.c:1062:\tearly_create_pgd_mapping(swapper_pg_dir, start, __phys_to_virt(start),\narch/arm64/mm/mmu.c-1063-\t\t\t\t end - start, prot, early_pgtable_alloc, flags);\n--\narch/arm64/mm/mmu.c=1366=static int __init map_entry_trampoline(void)\n--\narch/arm64/mm/mmu.c-1380-\tmemset(tramp_pg_dir, 0, PGD_SIZE);\narch/arm64/mm/mmu.c:1381:\tearly_create_pgd_mapping(tramp_pg_dir, pa_start, TRAMP_VALIAS,\narch/arm64/mm/mmu.c-1382-\t\t\t\t entry_tramp_text_size(), prot,\n" }
39/3	2026/08/18 17:07	llm	kmsan-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 30188 output: 38 thoughts: 20 Thoughts: Here's my thought process, as if I were considering this issue: Delving into the Allocation Landscape Alright, I'm thinking we need to trace down where this memory allocation problem is originating. Given the context, my initial hunch is that it might be connected to the page table initialization. Let's start with a focused investigation of `alloc_init_pud`, or perhaps related functions. Anything that's responsible for setting up those crucial page directory entries and uses the memory allocator, is a prime suspect. I'll need to carefully examine those allocation routines, looking for any potential issues. Things like unchecked return values from the allocator, or maybe a subtle off-by-one error in the calculations of the size or alignment. I should look at any places that might be allocating metadata for the page table structures.
40/3	2026/08/18 17:07	tool	grepper	0m Args: { "Expression": "alloc_init_pud", "PathPrefix": "arch/arm64/mm/mmu.c" } Results: { "Output": "arch/arm64/mm/mmu.c=306=static int alloc_init_cont_pmd(pud_t pudp, unsigned long addr,\n--\narch/arm64/mm/mmu.c-364-\narch/arm64/mm/mmu.c:365:static int alloc_init_pud(p4d_t p4dp, unsigned long addr, unsigned long end,\narch/arm64/mm/mmu.c-366-\t\t\t phys_addr_t phys, pgprot_t prot,\n--\narch/arm64/mm/mmu.c=432=static int alloc_init_p4d(pgd_t *pgdp, unsigned long addr, unsigned long end,\n--\narch/arm64/mm/mmu.c-465-\narch/arm64/mm/mmu.c:466:\t\tret = alloc_init_pud(p4dp, addr, next, phys, prot,\narch/arm64/mm/mmu.c-467-\t\t\t\t pgtable_alloc, flags);\n" }
41/3	2026/08/18 17:07	llm	kmsan-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 30702 output: 38 thoughts: 14 Thoughts: Here's my attempt at summarizing the thought process, tailored for an expert audience: Diving into `alloc_init_pud` Okay, time to dig into `alloc_init_pud`. I need to refresh my memory on exactly how this thing works. I'm focusing specifically on how the PUD is initialized. I'm guessing this function is central to the setup of a new page directory entry, which of course is fundamental to the memory management of a process. I expect to see the low-level details, perhaps the direct manipulation of hardware page table entries, or calls to lower-level allocation routines. I wonder if it directly zeroes out the PUD entry, or if there's a more nuanced initialization strategy, perhaps involving a bitmask or some early configuration of the present bit. Let's get to the code and see. My focus here is going to be about how it sets up a page directory entry at the PUD level.
42/3	2026/08/18 17:07	tool	grepper	0m Args: { "Expression": "alloc_init_pud", "PathPrefix": "arch/arm64/mm/mmu.c" } Results: { "Output": "arch/arm64/mm/mmu.c=306=static int alloc_init_cont_pmd(pud_t pudp, unsigned long addr,\n--\narch/arm64/mm/mmu.c-364-\narch/arm64/mm/mmu.c:365:static int alloc_init_pud(p4d_t p4dp, unsigned long addr, unsigned long end,\narch/arm64/mm/mmu.c-366-\t\t\t phys_addr_t phys, pgprot_t prot,\n--\narch/arm64/mm/mmu.c=432=static int alloc_init_p4d(pgd_t *pgdp, unsigned long addr, unsigned long end,\n--\narch/arm64/mm/mmu.c-465-\narch/arm64/mm/mmu.c:466:\t\tret = alloc_init_pud(p4dp, addr, next, phys, prot,\narch/arm64/mm/mmu.c-467-\t\t\t\t pgtable_alloc, flags);\n" }
43/3	2026/08/18 17:07	llm	kmsan-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 31240 output: 40 thoughts: 25 Thoughts: Here's a summary of my thought process, as requested: Diving into the ARM64 MMU Implementation Okay, time to get my hands dirty. I need to understand the specifics of how the ARM64 MMU is handling memory management, particularly the translation process. Let's start with a focused look at `arch/arm64/mm/mmu.c`. Based on my experience, line 365 is a good place to begin. It's likely involved in the core translation logic, perhaps dealing with page table walks or the initial setup. I expect it'll give me a solid entry point to trace the control flow and unravel the specific implementation details, especially if I can discern how it is dealing with virtual and physical address spaces.
44/3	2026/08/18 17:07	tool	read-file	0m Args: { "File": "arch/arm64/mm/mmu.c", "FirstLine": 365, "LineCount": 50 } Results: { "Contents": " 365:\tstatic int alloc_init_pud(p4d_t p4dp, unsigned long addr, unsigned long end,\n 366:\t\t\t\t phys_addr_t phys, pgprot_t prot,\n 367:\t\t\t\t phys_addr_t (pgtable_alloc)(enum pgtable_level),\n 368:\t\t\t\t int flags)\n 369:\t{\n 370:\t\tint ret = 0;\n 371:\t\tunsigned long next;\n 372:\t\tp4d_t p4d = READ_ONCE(p4dp);\n 373:\t\tpud_t pudp;\n 374:\t\n 375:\t\tif (p4d_none(p4d)) {\n 376:\t\t\tp4dval_t p4dval = P4D_TYPE_TABLE \| P4D_TABLE_UXN \| P4D_TABLE_AF;\n 377:\t\t\tphys_addr_t pud_phys;\n 378:\t\n 379:\t\t\tif (flags \u0026 NO_EXEC_MAPPINGS)\n 380:\t\t\t\tp4dval \|= P4D_TABLE_PXN;\n 381:\t\t\tBUG_ON(!pgtable_alloc);\n 382:\t\t\tpud_phys = pgtable_alloc(PGTABLE_LEVEL_PUD);\n 383:\t\t\tif (pud_phys == INVALID_PHYS_ADDR)\n 384:\t\t\t\treturn -ENOMEM;\n 385:\t\t\tpudp = pud_set_fixmap(pud_phys);\n 386:\t\t\tinit_clear_pgtable(pudp);\n 387:\t\t\tpudp += pud_index(addr);\n 388:\t\t\t__p4d_populate(p4dp, pud_phys, p4dval);\n 389:\t\t} else {\n 390:\t\t\tBUG_ON(p4d_bad(p4d));\n 391:\t\t\tpudp = pud_set_fixmap_offset(p4dp, addr);\n 392:\t\t}\n 393:\t\n 394:\t\tdo {\n 395:\t\t\tpud_t old_pud = READ_ONCE(pudp);\n 396:\t\n 397:\t\t\tnext = pud_addr_end(addr, end);\n 398:\t\n 399:\t\t\t/\n 400:\t\t\t * For 4K granule only, attempt to put down a 1GB block\n 401:\t\t\t /\n 402:\t\t\tif (pud_sect_supported() \u0026\u0026\n 403:\t\t\t ((addr \| next \| phys) \u0026 ~PUD_MASK) == 0 \u0026\u0026\n 404:\t\t\t (flags \u0026 NO_BLOCK_MAPPINGS) == 0 \u0026\u0026\n 405:\t\t\t !pud_table(old_pud)) {\n 406:\t\t\t\tWARN_ON(!pud_set_huge(pudp, phys, prot));\n 407:\t\n 408:\t\t\t\t/\n 409:\t\t\t\t * After the PUD entry has been populated once, we\n 410:\t\t\t\t * only allow updates to the permission attributes.\n 411:\t\t\t\t /\n 412:\t\t\t\tBUG_ON(!pgattr_change_is_safe(pud_val(old_pud),\n 413:\t\t\t\t\t\t\t READ_ONCE(pud_val(pudp))));\n 414:\t\t\t} else {\n" }
45/3	2026/08/18 17:07	llm	kmsan-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 32339 output: 38 thoughts: 30 Thoughts: Here's my take on the thought process, as requested: Deconstructing the Page Table Initialization Okay, so I see `init_clear_pgtable(pudp)` called. That's the signal, isn't it? Time to dig into how the page tables are being initialized and cleared. The `pudp` likely points to the pagetable being initialized. I'm immediately interested in the implementation of that function. Let's start with a deep dive into `init_clear_pgtable` itself. I need to understand what's happening at the lowest levels here – the actual bit manipulation or memory writes. What's the strategy? Are we zeroing out entries? Setting specific flags? Is there any optimization going on, like using vector instructions or memory copies if appropriate? Let's figure out the full context.
46/3	2026/08/18 17:07	tool	grepper	0m Args: { "Expression": "init_clear_pgtable", "PathPrefix": "arch/arm64/mm/mmu.c" } Results: { "Output": "arch/arm64/mm/mmu.c=121=bool pgattr_change_is_safe(pteval_t old, pteval_t new)\n--\narch/arm64/mm/mmu.c-155-\narch/arm64/mm/mmu.c:156:static void init_clear_pgtable(void table)\narch/arm64/mm/mmu.c-157-{\n--\narch/arm64/mm/mmu.c=198=static int alloc_init_cont_pte(pmd_t pmdp, unsigned long addr,\n--\narch/arm64/mm/mmu.c-219-\t\tptep = pte_set_fixmap(pte_phys);\narch/arm64/mm/mmu.c:220:\t\tinit_clear_pgtable(ptep);\narch/arm64/mm/mmu.c-221-\t\tptep += pte_index(addr);\n--\narch/arm64/mm/mmu.c=306=static int alloc_init_cont_pmd(pud_t pudp, unsigned long addr,\n--\narch/arm64/mm/mmu.c-331-\t\tpmdp = pmd_set_fixmap(pmd_phys);\narch/arm64/mm/mmu.c:332:\t\tinit_clear_pgtable(pmdp);\narch/arm64/mm/mmu.c-333-\t\tpmdp += pmd_index(addr);\n--\narch/arm64/mm/mmu.c=365=static int alloc_init_pud(p4d_t p4dp, unsigned long addr, unsigned long end,\n--\narch/arm64/mm/mmu.c-385-\t\tpudp = pud_set_fixmap(pud_phys);\narch/arm64/mm/mmu.c:386:\t\tinit_clear_pgtable(pudp);\narch/arm64/mm/mmu.c-387-\t\tpudp += pud_index(addr);\n--\narch/arm64/mm/mmu.c=432=static int alloc_init_p4d(pgd_t pgdp, unsigned long addr, unsigned long end,\n--\narch/arm64/mm/mmu.c-452-\t\tp4dp = p4d_set_fixmap(p4d_phys);\narch/arm64/mm/mmu.c:453:\t\tinit_clear_pgtable(p4dp);\narch/arm64/mm/mmu.c-454-\t\tp4dp += p4d_index(addr);\n--\narch/arm64/mm/mmu.c=546=static phys_addr_t __pgd_pgtable_alloc(struct mm_struct mm, gfp_t gfp,\n--\narch/arm64/mm/mmu.c-548-{\narch/arm64/mm/mmu.c:549:\t/* Page is zeroed by init_clear_pgtable() so don't duplicate effort. /\narch/arm64/mm/mmu.c-550-\tstruct ptdesc ptdesc = pagetable_alloc(gfp \u0026 ~__GFP_ZERO, 0);\n" }
47/3	2026/08/18 17:07	llm	kmsan-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 33420 output: 38 thoughts: 691 Thoughts: Analyzing Page Table Hardening Okay, so I'm digging into this page table hardening patch. My initial thought was to make sure no new uninitialized memory vulnerabilities are introduced. The `init_clear_pgtable` function is clearly zeroing the page table from the start, which gives me a starting point of known state. Now, let's circle back to `kpkeys_pgtable_alloc`. Looking at the code for `kpkeys_pgtable_alloc`, it's using `alloc_pages_noprof` and then `set_pkey_pgtable`. The caller, `pagetable_alloc_noprof`, is passing `gfp \| __GFP_COMP`. Now, if `gfp` already includes `__GFP_ZERO`, `alloc_pages_noprof` does a zero-fill. The crucial bit here is to make sure we're still zeroing; the original code used `alloc_pages_noprof(gfp \| __GFP_COMP, order)`, so the behavior appears identical regarding zeroing. Now, let's step back and look at the bigger picture. The patch itself introduces `KPKEYS_HARDENED_PGTABLES`, which leverages kernel pkeys to harden the page tables. It modifies those fundamental page table accessors – `set_pte_nosync`, `set_pmd`, the whole lot – to utilize `guard(kpkeys_hardened_pgtables)()` or `scoped_guard(kpkeys_hardened_pgtables)`. Essentially, these guards temporarily allow write access to otherwise read-only page tables during the modification. My primary concern is around uninitialized memory. Does the patch introduce any new structures sent to user space? No, not that I can see. It modifies `struct thread_struct` with `por_el1` and `kpkeys_state_lazy_mmu`, but these are internal to the kernel, so no exposure there. The page table allocation routines, using `kpkeys_pgtable_alloc` and `kpkeys_physmem_pgtable_alloc`, appear to retain the same zeroing behavior as before. Excellent. Now, a small but important detail. It looks like `vmemmap_alloc_pgtable` now calls `kpkeys_physmem_pgtable_alloc` and explicitly zeroes the allocated page. Same for `vmemmap_get_tail` now calls `vmemmap_alloc_block` and zeroing is explicitly handled. Wait a second, the `vmemmap_alloc_block_zero` function was removed and replaced with `vmemmap_alloc_block` followed by `memset(p, 0, PAGE_SIZE)`. This is worth a second look, let's take a look at the code for that replacement in `mm/sparse-vmemmap.c` and make sure it is correct.
48/3	2026/08/18 17:07	tool	grepper	0m Error: no matches Args: { "Expression": "vmemmap_alloc_block_zero", "PathPrefix": "mm/sparse-vmemmap.c" } Results: { "error": "no matches" }
49/3	2026/08/18 17:07	llm	kmsan-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 34781 output: 29 thoughts: 492 Thoughts: Analyzing the Removal of `vmemmap_alloc_block_zero` Okay, so the patch I'm looking at removed `vmemmap_alloc_block_zero` from `mm/sparse-vmemmap.c`. Let's dig into the details. Examining the code, I see the old function was replaced with `vmemmap_alloc_pgtable`. The original function allocated a block, and then zeroed it out. Now, with the replacement, the code seems to check if the slab allocator is available and use it with the `__GFP_ZERO` flag. If not, it falls back to either `kpkeys_physmem_pgtable_alloc` if `kpkeys_hardened_pgtables_early_enabled()` is true, otherwise it uses `__earlyonly_bootmem_alloc`. Crucially, it now zero's out each page size after allocation regardless of which route is taken. Interesting. It looks like `vmemmap_alloc_block_zero` was also used in `vmemmap_get_tail`, where it was used to allocate a page and zero it. This has been replaced by `vmemmap_alloc_block`, followed by a `memset` to zero the memory. It seems like the core functionality is still there, but handled differently. So, the next question is: Did they miss any other uses of the old function? I need to make sure this change is complete. I'll need to run a quick search through the code to make sure there are no other places where `vmemmap_alloc_block_zero` was used that might have been overlooked.
50/3	2026/08/18 17:08	tool	grepper	0m Error: no matches Args: { "Expression": "vmemmap_alloc_block_zero", "PathPrefix": "" } Results: { "error": "no matches" }
51/3	2026/08/18 17:08	llm	kmsan-evaluator	1m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 35665 output: 473 thoughts: 11719 Thoughts: Analysis of Kernel Page Table Hardening Implementation Okay, so I've meticulously reviewed this patch series, and it all boils down to enhancing the security of our page tables using kernel Protection Key (PKey) functionality, primarily on ARM64. It’s a fairly focused refactoring to protect page tables with a specific PKey, to prevent unauthorized modification. I’ve gone through all the relevant code and I can confirm that there are no apparent uninitialized memory access vulnerabilities introduced by this implementation, nor any new paths for information leakage to user space. The process primarily involves initializing and setting the PKey appropriately, and then enabling write access to page tables conditionally using `guard(kpkeys_hardened_pgtables)()` where modifications are required. This approach seems robust and correct. The new code allocates memory for page tables using standard methods, like `alloc_pages_noprof` and `memblock_phys_alloc_range`, which has been carefully examined to ensure it's not a source of any uninitialized memory access. I've confirmed they are either zeroed or re-initialized before use, as they were prior to these changes. Moreover, the core data structures, such as the `struct kpkeys_state` and the `por_el1` field within the `thread_struct`, are either local to the kernel or carefully handled during context switches, never directly exposed to user space. There's no scenario where uninitialized memory would leak out. Even when there's an instance where a local struct, such as within the `KPKEYS_GUARD_COND` macro, might have uninitialized fields, this is managed within the scope of the code and isn't something that can become an information leak. Importantly, I’ve checked the KUnit tests included within the series. It's safe, since these KUnit tests are just that, they test, and write values and checks. Essentially, this patch series seems to have been carefully implemented. I do not see any reason to run this specific code through a KMSAN session. KASAN and other standard bug detectors should be able to catch potential memory corruption issues. Therefore, I'll set `NeedsKMSAN=false` and provide this detailed reasoning.
52/3	2026/08/18 17:09	tool	set-results	0m Args: { "KMSANReasoning": "The patch series introduces `KPKEYS_HARDENED_PGTABLES` to harden page tables using kernel pkeys (KPKEYS) on ARM64. It modifies page table allocation to set a non-default protection key and uses `guard(kpkeys_hardened_pgtables)()` to temporarily enable write access when modifying page table entries.\n\nAfter a thorough review of the changes:\n1. No new structures sent to user space: The patch adds `por_el1` and `kpkeys_state_lazy_mmu` to `struct thread_struct`, but these are internal kernel state and are not exposed to user space via ptrace, signals, or any other mechanism.\n2. Page table allocation zeroing: The patch modifies page table allocation to use `kpkeys_pgtable_alloc` and `kpkeys_physmem_pgtable_alloc`. These functions wrap standard allocators (`alloc_pages_noprof` and `memblock_phys_alloc_range`). The zeroing behavior remains identical to the original code: `early_pgtable_alloc` allocations are zeroed by `init_clear_pgtable()`, `pagetable_alloc_noprof` allocations are zeroed by `alloc_pages_noprof` if `__GFP_ZERO` is passed, and `vmemmap_alloc_pgtable` explicitly calls `memset(p, 0, PAGE_SIZE)`.\n3. No uninitialized memory risks: The `struct kpkeys_state` used in `scoped_guard` is a local variable and is not copied to user space. The conditional initialization in `arch_kpkeys_enter_context` is safe because the uninitialized fields are never read if `entered_context` is false.\n\nThe changes primarily involve memory protection and control flow around page table modifications. Any bugs introduced would likely be out-of-bounds accesses, use-after-free, or invalid memory accesses, which are well-covered by KASAN and standard bug detectors. There is no risk of uninitialized memory usage or info-leaks that would require KMSAN.", "NeedsKMSAN": false } Results: { "KMSANReasoning": "The patch series introduces `KPKEYS_HARDENED_PGTABLES` to harden page tables using kernel pkeys (KPKEYS) on ARM64. It modifies page table allocation to set a non-default protection key and uses `guard(kpkeys_hardened_pgtables)()` to temporarily enable write access when modifying page table entries.\n\nAfter a thorough review of the changes:\n1. No new structures sent to user space: The patch adds `por_el1` and `kpkeys_state_lazy_mmu` to `struct thread_struct`, but these are internal kernel state and are not exposed to user space via ptrace, signals, or any other mechanism.\n2. Page table allocation zeroing: The patch modifies page table allocation to use `kpkeys_pgtable_alloc` and `kpkeys_physmem_pgtable_alloc`. These functions wrap standard allocators (`alloc_pages_noprof` and `memblock_phys_alloc_range`). The zeroing behavior remains identical to the original code: `early_pgtable_alloc` allocations are zeroed by `init_clear_pgtable()`, `pagetable_alloc_noprof` allocations are zeroed by `alloc_pages_noprof` if `__GFP_ZERO` is passed, and `vmemmap_alloc_pgtable` explicitly calls `memset(p, 0, PAGE_SIZE)`.\n3. No uninitialized memory risks: The `struct kpkeys_state` used in `scoped_guard` is a local variable and is not copied to user space. The conditional initialization in `arch_kpkeys_enter_context` is safe because the uninitialized fields are never read if `entered_context` is false.\n\nThe changes primarily involve memory protection and control flow around page table modifications. Any bugs introduced would likely be out-of-bounds accesses, use-after-free, or invalid memory accesses, which are well-covered by KASAN and standard bug detectors. There is no risk of uninitialized memory usage or info-leaks that would require KMSAN.", "NeedsKMSAN": false }

{
  "EnableConfigs": [
    "KPKEYS_HARDENED_PGTABLES"
  ],
  "FocusSymbols": [
    "kpkeys_pgtable_alloc",
    "kpkeys_pgtable_free",
    "kpkeys_physmem_pgtable_alloc",
    "set_memory_pkey",
    "arch_set_user_pkey_access"
  ],
  "KMSANReasoning": "The patch series introduces `KPKEYS_HARDENED_PGTABLES` to harden page tables using kernel pkeys (KPKEYS) on ARM64. It modifies page table allocation to set a non-default protection key and uses `guard(kpkeys_hardened_pgtables)()` to temporarily enable write access when modifying page table entries.\n\nAfter a thorough review of the changes:\n1. **No new structures sent to user space:** The patch adds `por_el1` and `kpkeys_state_lazy_mmu` to `struct thread_struct`, but these are internal kernel state and are not exposed to user space via ptrace, signals, or any other mechanism.\n2. **Page table allocation zeroing:** The patch modifies page table allocation to use `kpkeys_pgtable_alloc` and `kpkeys_physmem_pgtable_alloc`. These functions wrap standard allocators (`alloc_pages_noprof` and `memblock_phys_alloc_range`). The zeroing behavior remains identical to the original code: `early_pgtable_alloc` allocations are zeroed by `init_clear_pgtable()`, `pagetable_alloc_noprof` allocations are zeroed by `alloc_pages_noprof` if `__GFP_ZERO` is passed, and `vmemmap_alloc_pgtable` explicitly calls `memset(p, 0, PAGE_SIZE)`.\n3. **No uninitialized memory risks:** The `struct kpkeys_state` used in `scoped_guard` is a local variable and is not copied to user space. The conditional initialization in `arch_kpkeys_enter_context` is safe because the uninitialized fields are never read if `entered_context` is false.\n\nThe changes primarily involve memory protection and control flow around page table modifications. Any bugs introduced would likely be out-of-bounds accesses, use-after-free, or invalid memory accesses, which are well-covered by KASAN and standard bug detectors. There is no risk of uninitialized memory usage or info-leaks that would require KMSAN.",
  "NeedsKMSAN": false,
  "Reasoning": "The patch introduces a new feature to harden page tables using kernel pkeys (kpkeys) on ARM64. It adds a new file `mm/kpkeys_hardened_pgtables.c` and modifies ARM64 memory management code to allocate page tables with a specific pkey and restrict write access to them. The changes include new page table allocation and freeing functions, as well as modifications to `arch_set_user_pkey_access`. This is a significant functional change to the core memory management subsystem that should be fuzzed to ensure memory safety and correctness.",
  "WorthFuzzing": true
}

{
  "PatchDiff": "commit bd271568204405923f3b37fcb4ded599be77ced1\nAuthor: syz-cluster \u003ctriage@syzkaller.com\u003e\nDate:   Tue Aug 18 17:01:35 2026 +0000\n\n    syz-cluster: applied patch under review\n\ndiff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig\nindex b3afe0688919b..ac9faec1a0184 100644\n--- a/arch/arm64/Kconfig\n+++ b/arch/arm64/Kconfig\n@@ -2202,6 +2202,8 @@ config ARM64_POE\n \tdef_bool y\n \tselect ARCH_USES_HIGH_VMA_FLAGS\n \tselect ARCH_HAS_PKEYS\n+\tselect ARCH_HAS_KPKEYS\n+\tselect ARCH_HAS_KPKEYS_HARDENED_PGTABLES\n \thelp\n \t  The Permission Overlay Extension is used to implement Memory\n \t  Protection Keys. Memory Protection Keys provides a mechanism for\ndiff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h\nindex a57870fa96db5..37aae60c2d7f0 100644\n--- a/arch/arm64/include/asm/cpufeature.h\n+++ b/arch/arm64/include/asm/cpufeature.h\n@@ -1085,6 +1085,18 @@ static inline bool cpu_has_lpa2(void)\n #endif\n }\n \n+static inline bool cpu_has_poe(void)\n+{\n+\tu64 mmfr3;\n+\n+\tif (!IS_ENABLED(CONFIG_ARM64_POE))\n+\t\treturn false;\n+\n+\tmmfr3 = read_sysreg_s(SYS_ID_AA64MMFR3_EL1);\n+\treturn cpuid_feature_extract_unsigned_field(mmfr3,\n+\t\t\t\t\t\t    ID_AA64MMFR3_EL1_S1POE_SHIFT);\n+}\n+\n #endif /* __ASSEMBLER__ */\n \n #endif\ndiff --git a/arch/arm64/include/asm/kpkeys.h b/arch/arm64/include/asm/kpkeys.h\nnew file mode 100644\nindex 0000000000000..7747af54fd386\n--- /dev/null\n+++ b/arch/arm64/include/asm/kpkeys.h\n@@ -0,0 +1,84 @@\n+/* SPDX-License-Identifier: GPL-2.0-only */\n+#ifndef __ASM_KPKEYS_H\n+#define __ASM_KPKEYS_H\n+\n+#include \u003clinux/kpkeys_types.h\u003e\n+\n+#include \u003casm/barrier.h\u003e\n+#include \u003casm/cpufeature.h\u003e\n+#include \u003casm/por.h\u003e\n+\n+/*\n+ * Equivalent to por_set_kpkeys_context(0, KPKEYS_CTX_DEFAULT), but can also be\n+ * used in assembly.\n+ */\n+#define POR_EL1_INIT\t(POR_ELx_PERM_PREP(0, POE_RWX) | \\\n+\t\t\t POR_ELx_PERM_PREP(KPKEYS_PKEY_PGTABLES, POE_R))\n+\n+#ifndef __ASSEMBLY__\n+\n+static inline bool arch_supports_kpkeys(void)\n+{\n+\treturn system_supports_poe();\n+}\n+\n+static inline bool arch_supports_kpkeys_early(void)\n+{\n+\t/* POE is a boot feature */\n+\treturn boot_capabilities_finalized() ?\n+\t\tsystem_supports_poe() : cpu_has_poe();\n+}\n+\n+#ifdef CONFIG_ARM64_POE\n+\n+static inline u64 por_set_kpkeys_context(u64 por, enum kpkeys_ctx ctx)\n+{\n+\tpor = por_elx_set_pkey_perms(por, 0, POE_RWX);\n+\tpor = por_elx_set_pkey_perms(por, KPKEYS_PKEY_PGTABLES,\n+\t\t\t\t     ctx == KPKEYS_CTX_PGTABLES ? POE_RW : POE_R);\n+\n+\treturn por;\n+}\n+\n+static __always_inline void __kpkeys_set_pkey_reg_nosync(u64 pkey_reg)\n+{\n+\twrite_sysreg_s(pkey_reg, SYS_POR_EL1);\n+}\n+\n+static __always_inline\n+struct kpkeys_state arch_kpkeys_enter_context(enum kpkeys_ctx ctx)\n+{\n+\tconst u64 prev_por = read_sysreg_s(SYS_POR_EL1);\n+\tconst u64 new_por = por_set_kpkeys_context(prev_por, ctx);\n+\n+\tif (prev_por == new_por)\n+\t\treturn (struct kpkeys_state) { .entered_context = false };\n+\n+\t__kpkeys_set_pkey_reg_nosync(new_por);\n+\tisb();\n+\n+\treturn (struct kpkeys_state) {\n+\t\t.entered_context = true,\n+\t\t.arch_state.por = prev_por,\n+\t};\n+}\n+\n+static __always_inline\n+void arch_kpkeys_leave_context(const struct kpkeys_state *state)\n+{\n+\t__kpkeys_set_pkey_reg_nosync(state-\u003earch_state.por);\n+\tisb();\n+}\n+\n+#endif /* CONFIG_ARM64_POE */\n+\n+#ifdef CONFIG_KPKEYS_HARDENED_PGTABLES\n+\n+#define arch_kpkeys_protect_static_pgtables arch_kpkeys_protect_static_pgtables\n+void arch_kpkeys_protect_static_pgtables(void);\n+\n+#endif /* CONFIG_KPKEYS_HARDENED_PGTABLES */\n+\n+#endif\t/* __ASSEMBLY__ */\n+\n+#endif\t/* __ASM_KPKEYS_H */\ndiff --git a/arch/arm64/include/asm/kpkeys_types.h b/arch/arm64/include/asm/kpkeys_types.h\nnew file mode 100644\nindex 0000000000000..e9c59a26ee7a9\n--- /dev/null\n+++ b/arch/arm64/include/asm/kpkeys_types.h\n@@ -0,0 +1,15 @@\n+/* SPDX-License-Identifier: GPL-2.0-only */\n+#ifndef __ASM_KPKEYS_TYPES_H\n+#define __ASM_KPKEYS_TYPES_H\n+\n+#include \u003clinux/types.h\u003e\n+\n+#ifndef __ASSEMBLY__\n+\n+struct arch_kpkeys_state {\n+\tu64 por;\n+};\n+\n+#endif\t/* __ASSEMBLY__ */\n+\n+#endif\t/* __ASM_KPKEYS_TYPES_H */\ndiff --git a/arch/arm64/include/asm/pgtable-prot.h b/arch/arm64/include/asm/pgtable-prot.h\nindex 212ce1b02e15e..d4d45ab86a5a8 100644\n--- a/arch/arm64/include/asm/pgtable-prot.h\n+++ b/arch/arm64/include/asm/pgtable-prot.h\n@@ -183,9 +183,9 @@ static inline bool __pure lpa2_is_enabled(void)\n \tPIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_SHARED_EXEC),   PIE_RW)     | \\\n \tPIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_READONLY),      PIE_R)      | \\\n \tPIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_SHARED),        PIE_RW)     | \\\n-\tPIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL_ROX),    PIE_RX)     | \\\n-\tPIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL_EXEC),   PIE_RWX)    | \\\n-\tPIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL_RO),     PIE_R)      | \\\n-\tPIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL),        PIE_RW))\n+\tPIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL_ROX),    PIE_RX_O)   | \\\n+\tPIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL_EXEC),   PIE_RWX_O)  | \\\n+\tPIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL_RO),     PIE_R_O)    | \\\n+\tPIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL),        PIE_RW_O))\n \n #endif /* __ASM_PGTABLE_PROT_H */\ndiff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h\nindex 27689c62bd25d..6e5d14c7e8b8f 100644\n--- a/arch/arm64/include/asm/pgtable.h\n+++ b/arch/arm64/include/asm/pgtable.h\n@@ -39,6 +39,48 @@\n #include \u003clinux/mm_types.h\u003e\n #include \u003clinux/sched.h\u003e\n #include \u003clinux/page_table_check.h\u003e\n+#include \u003clinux/kpkeys.h\u003e\n+\n+#ifdef CONFIG_KPKEYS_HARDENED_PGTABLES\n+KPKEYS_GUARD_COND(kpkeys_hardened_pgtables, KPKEYS_CTX_PGTABLES,\n+\t\t  kpkeys_hardened_pgtables_enabled() \u0026\u0026\n+\t\t  !is_lazy_mmu_mode_active())\n+\n+static inline void kpkeys_lazy_mmu_enter(void)\n+{\n+\tif (!kpkeys_hardened_pgtables_enabled())\n+\t\treturn;\n+\n+\tcurrent-\u003ethread.kpkeys_state_lazy_mmu =\n+\t\tkpkeys_enter_context(KPKEYS_CTX_PGTABLES);\n+}\n+\n+static inline void kpkeys_lazy_mmu_exit(void)\n+{\n+\tconst struct kpkeys_state *state =\n+\t\t\u0026current-\u003ethread.kpkeys_state_lazy_mmu;\n+\n+\tif (!kpkeys_hardened_pgtables_enabled())\n+\t\treturn;\n+\n+\t/*\n+\t * We skip any barrier if TIF_LAZY_MMU_PENDING is set:\n+\t * emit_pte_barriers() will issue an ISB just after this function\n+\t * returns.\n+\t */\n+\tif (test_thread_flag(TIF_LAZY_MMU_PENDING) \u0026\u0026 state-\u003eentered_context)\n+\t\t__kpkeys_set_pkey_reg_nosync(state-\u003earch_state.por);\n+\telse\n+\t\tkpkeys_leave_context(state);\n+}\n+#else /* CONFIG_KPKEYS_HARDENED_PGTABLES */\n+KPKEYS_GUARD_NOOP(kpkeys_hardened_pgtables)\n+\n+static inline void kpkeys_lazy_mmu_enter(void) {}\n+static inline void kpkeys_lazy_mmu_exit(void) {}\n+#endif /* CONFIG_KPKEYS_HARDENED_PGTABLES */\n+\n+\n \n static inline void emit_pte_barriers(void)\n {\n@@ -71,7 +113,10 @@ static inline void queue_pte_barriers(void)\n \t}\n }\n \n-static inline void arch_enter_lazy_mmu_mode(void) {}\n+static inline void arch_enter_lazy_mmu_mode(void)\n+{\n+\tkpkeys_lazy_mmu_enter();\n+}\n \n static inline void arch_flush_lazy_mmu_mode(void)\n {\n@@ -81,6 +126,11 @@ static inline void arch_flush_lazy_mmu_mode(void)\n \n static inline void arch_leave_lazy_mmu_mode(void)\n {\n+\t/*\n+\t * The ordering should be preserved to allow kpkeys_lazy_mmu_exit()\n+\t * to skip any barrier when TIF_LAZY_MMU_PENDING is set.\n+\t */\n+\tkpkeys_lazy_mmu_exit();\n \tarch_flush_lazy_mmu_mode();\n }\n \n@@ -359,6 +409,7 @@ static inline pte_t pte_clear_uffd_wp(pte_t pte)\n \n static inline void __set_pte_nosync(pte_t *ptep, pte_t pte)\n {\n+\tguard(kpkeys_hardened_pgtables)();\n \tWRITE_ONCE(*ptep, pte);\n }\n \n@@ -830,6 +881,7 @@ static inline void set_pmd(pmd_t *pmdp, pmd_t pmd)\n \t}\n #endif /* __PAGETABLE_PMD_FOLDED */\n \n+\tguard(kpkeys_hardened_pgtables)();\n \tWRITE_ONCE(*pmdp, pmd);\n \n \tif (pmd_valid(pmd))\n@@ -894,6 +946,7 @@ static inline void set_pud(pud_t *pudp, pud_t pud)\n \t\treturn;\n \t}\n \n+\tguard(kpkeys_hardened_pgtables)();\n \tWRITE_ONCE(*pudp, pud);\n \n \tif (pud_valid(pud))\n@@ -975,6 +1028,7 @@ static inline void set_p4d(p4d_t *p4dp, p4d_t p4d)\n \t\treturn;\n \t}\n \n+\tguard(kpkeys_hardened_pgtables)();\n \tWRITE_ONCE(*p4dp, p4d);\n \tqueue_pte_barriers();\n }\n@@ -1103,6 +1157,7 @@ static inline void set_pgd(pgd_t *pgdp, pgd_t pgd)\n \t\treturn;\n \t}\n \n+\tguard(kpkeys_hardened_pgtables)();\n \tWRITE_ONCE(*pgdp, pgd);\n \tqueue_pte_barriers();\n }\n@@ -1307,6 +1362,7 @@ static inline bool __ptep_test_and_clear_young(struct vm_area_struct *vma,\n {\n \tpte_t old_pte, pte;\n \n+\tguard(kpkeys_hardened_pgtables)();\n \tpte = __ptep_get(ptep);\n \tdo {\n \t\told_pte = pte;\n@@ -1354,7 +1410,10 @@ static inline pte_t __ptep_get_and_clear_anysz(struct mm_struct *mm,\n \t\t\t\t\t       pte_t *ptep,\n \t\t\t\t\t       unsigned long pgsize)\n {\n-\tpte_t pte = __pte(xchg_relaxed(\u0026pte_val(*ptep), 0));\n+\tpte_t pte;\n+\n+\tscoped_guard(kpkeys_hardened_pgtables)\n+\t\tpte = __pte(xchg_relaxed(\u0026pte_val(*ptep), 0));\n \n \tswitch (pgsize) {\n \tcase PAGE_SIZE:\n@@ -1427,6 +1486,7 @@ static inline void ___ptep_set_wrprotect(struct mm_struct *mm,\n {\n \tpte_t old_pte;\n \n+\tguard(kpkeys_hardened_pgtables)();\n \tdo {\n \t\told_pte = pte;\n \t\tpte = pte_wrprotect(pte);\n@@ -1460,6 +1520,7 @@ static inline void __clear_young_dirty_pte(struct vm_area_struct *vma,\n {\n \tpte_t old_pte;\n \n+\tguard(kpkeys_hardened_pgtables)();\n \tdo {\n \t\told_pte = pte;\n \n@@ -1507,6 +1568,7 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma,\n \t\tunsigned long address, pmd_t *pmdp, pmd_t pmd)\n {\n \tpage_table_check_pmd_set(vma-\u003evm_mm, address, pmdp, pmd);\n+\tguard(kpkeys_hardened_pgtables)();\n \treturn __pmd(xchg_relaxed(\u0026pmd_val(*pmdp), pmd_val(pmd)));\n }\n #endif\ndiff --git a/arch/arm64/include/asm/por.h b/arch/arm64/include/asm/por.h\nindex d913d5b529e49..dddabf6ffae6d 100644\n--- a/arch/arm64/include/asm/por.h\n+++ b/arch/arm64/include/asm/por.h\n@@ -10,6 +10,8 @@\n \n #define POR_EL0_INIT\tPOR_ELx_PERM_PREP(0, POE_RWX)\n \n+#ifndef __ASSEMBLY__\n+\n static inline bool por_elx_allows_read(u64 por, u8 pkey)\n {\n \tu8 perm = POR_ELx_PERM_GET(pkey, por);\n@@ -31,4 +33,13 @@ static inline bool por_elx_allows_exec(u64 por, u8 pkey)\n \treturn perm \u0026 POE_X;\n }\n \n+static inline u64 por_elx_set_pkey_perms(u64 por, u8 pkey, u8 perms)\n+{\n+\tu64 shift = POR_ELx_PERM_SHIFT(pkey);\n+\n+\treturn (por \u0026 ~(POE_MASK \u003c\u003c shift)) | ((u64)perms \u003c\u003c shift);\n+}\n+\n+#endif\t/* __ASSEMBLY__ */\n+\n #endif /* _ASM_ARM64_POR_H */\ndiff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h\nindex c2a627f393144..23660a233da28 100644\n--- a/arch/arm64/include/asm/processor.h\n+++ b/arch/arm64/include/asm/processor.h\n@@ -30,6 +30,7 @@\n #include \u003clinux/build_bug.h\u003e\n #include \u003clinux/cache.h\u003e\n #include \u003clinux/init.h\u003e\n+#include \u003clinux/kpkeys_types.h\u003e\n #include \u003clinux/stddef.h\u003e\n #include \u003clinux/string.h\u003e\n #include \u003clinux/thread_info.h\u003e\n@@ -195,6 +196,10 @@ struct thread_struct {\n \tu64\t\t\tsvcr;\n \tu64\t\t\ttpidr2_el0;\n \tu64\t\t\tpor_el0;\n+\tu64\t\t\tpor_el1;\n+#ifdef CONFIG_KPKEYS_HARDENED_PGTABLES\n+\tstruct kpkeys_state\tkpkeys_state_lazy_mmu;\n+#endif\n #ifdef CONFIG_ARM64_GCS\n \tunsigned int\t\tgcs_el0_mode;\n \tunsigned int\t\tgcs_el0_locked;\ndiff --git a/arch/arm64/include/asm/sections.h b/arch/arm64/include/asm/sections.h\nindex 51b0d594239eb..dc961cf67168c 100644\n--- a/arch/arm64/include/asm/sections.h\n+++ b/arch/arm64/include/asm/sections.h\n@@ -23,6 +23,7 @@ extern char __irqentry_text_start[], __irqentry_text_end[];\n extern char __mmuoff_data_start[], __mmuoff_data_end[];\n extern char __entry_tramp_text_start[], __entry_tramp_text_end[];\n extern char __relocate_new_kernel_start[], __relocate_new_kernel_end[];\n+extern char __pi_init_pg_dir[], __pi_init_pg_end[];\n \n static inline size_t entry_tramp_text_size(void)\n {\ndiff --git a/arch/arm64/include/asm/set_memory.h b/arch/arm64/include/asm/set_memory.h\nindex 90f61b17275e1..b6cd6de34abf1 100644\n--- a/arch/arm64/include/asm/set_memory.h\n+++ b/arch/arm64/include/asm/set_memory.h\n@@ -19,4 +19,8 @@ bool kernel_page_present(struct page *page);\n int set_memory_encrypted(unsigned long addr, int numpages);\n int set_memory_decrypted(unsigned long addr, int numpages);\n \n+#ifdef CONFIG_ARCH_HAS_KPKEYS\n+int set_memory_pkey(unsigned long addr, int numpages, int pkey);\n+#endif\n+\n #endif /* _ASM_ARM64_SET_MEMORY_H */\ndiff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c\nindex 9a22df0c5120f..4f2f54007ed6f 100644\n--- a/arch/arm64/kernel/cpufeature.c\n+++ b/arch/arm64/kernel/cpufeature.c\n@@ -76,6 +76,7 @@\n #include \u003clinux/kasan.h\u003e\n #include \u003clinux/percpu.h\u003e\n #include \u003clinux/sched/isolation.h\u003e\n+#include \u003clinux/kpkeys.h\u003e\n \n #include \u003casm/arm_pmuv3.h\u003e\n #include \u003casm/cpu.h\u003e\n@@ -2463,8 +2464,10 @@ static void cpu_enable_mops(const struct arm64_cpu_capabilities *__unused)\n #ifdef CONFIG_ARM64_POE\n static void cpu_enable_poe(const struct arm64_cpu_capabilities *__unused)\n {\n-\tsysreg_clear_set(REG_TCR2_EL1, 0, TCR2_EL1_E0POE);\n+\twrite_sysreg_s(POR_EL1_INIT, SYS_POR_EL1);\n+\tsysreg_clear_set(REG_TCR2_EL1, 0, TCR2_EL1_E0POE | TCR2_EL1_POE);\n \tsysreg_clear_set(CPACR_EL1, 0, CPACR_EL1_E0POE);\n+\tisb();\n }\n #endif\n \ndiff --git a/arch/arm64/kernel/pi/map_kernel.c b/arch/arm64/kernel/pi/map_kernel.c\nindex fb44cbdd2f292..81ca3e98f0f7b 100644\n--- a/arch/arm64/kernel/pi/map_kernel.c\n+++ b/arch/arm64/kernel/pi/map_kernel.c\n@@ -92,8 +92,13 @@ static void __init map_kernel(u64 kaslr_offset, u64 va_offset, int root_level)\n \t\t    __inittext_end, prot, false, root_level);\n \tmap_segment(init_pg_dir, \u0026pgdp, va_offset, __initdata_begin,\n \t\t    __initdata_end, data_prot, false, root_level);\n-\tmap_segment(init_pg_dir, \u0026pgdp, va_offset, _data, _end, data_prot,\n-\t\t    true, root_level);\n+\tmap_segment(init_pg_dir, \u0026pgdp, va_offset, _data, init_pg_dir,\n+\t\t    data_prot, true, root_level);\n+\t/* Map init_pg_dir separately as we will remap it read-only later on */\n+\tmap_segment(init_pg_dir, \u0026pgdp, va_offset, init_pg_dir, init_pg_end,\n+\t\t    data_prot, true, root_level);\n+\tmap_segment(init_pg_dir, \u0026pgdp, va_offset, init_pg_end, _end,\n+\t\t    data_prot, true, root_level);\n \tdsb(ishst);\n \n \tidmap_cpu_replace_ttbr1((phys_addr_t)init_pg_dir);\ndiff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c\nindex 581f80e9b9b73..1dec143decf15 100644\n--- a/arch/arm64/kernel/process.c\n+++ b/arch/arm64/kernel/process.c\n@@ -51,6 +51,7 @@\n #include \u003casm/exec.h\u003e\n #include \u003casm/fpsimd.h\u003e\n #include \u003casm/gcs.h\u003e\n+#include \u003casm/kpkeys.h\u003e\n #include \u003casm/mmu_context.h\u003e\n #include \u003casm/mpam.h\u003e\n #include \u003casm/mte.h\u003e\n@@ -431,6 +432,9 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args)\n \n \tptrauth_thread_init_kernel(p);\n \n+\tif (system_supports_poe())\n+\t\tp-\u003ethread.por_el1 = POR_EL1_INIT;\n+\n \tif (likely(!args-\u003efn)) {\n \t\t*childregs = *current_pt_regs();\n \t\tchildregs-\u003eregs[0] = 0;\n@@ -681,6 +685,12 @@ static void permission_overlay_switch(struct task_struct *next)\n \t\t * of POR_EL0.\n \t\t */\n \t}\n+\n+\tcurrent-\u003ethread.por_el1 = read_sysreg_s(SYS_POR_EL1);\n+\tif (current-\u003ethread.por_el1 != next-\u003ethread.por_el1) {\n+\t\twrite_sysreg_s(next-\u003ethread.por_el1, SYS_POR_EL1);\n+\t\tisb();\n+\t}\n }\n \n /*\ndiff --git a/arch/arm64/kernel/sleep.S b/arch/arm64/kernel/sleep.S\nindex f093cdf71be11..e0a6ad85cd242 100644\n--- a/arch/arm64/kernel/sleep.S\n+++ b/arch/arm64/kernel/sleep.S\n@@ -3,6 +3,7 @@\n #include \u003clinux/linkage.h\u003e\n #include \u003casm/asm-offsets.h\u003e\n #include \u003casm/assembler.h\u003e\n+#include \u003casm/kpkeys.h\u003e\n #include \u003casm/smp.h\u003e\n \n \t.text\n@@ -134,6 +135,17 @@ SYM_FUNC_START(_cpu_resume)\n \t/* load sp from context */\n \tldr\tx2, [x0, #CPU_CTX_SP]\n \tmov\tsp, x2\n+\n+#ifdef CONFIG_ARM64_POE\n+alternative_if_not ARM64_HAS_S1POE\n+\tb\t.Lskip_por_set\n+alternative_else_nop_endif\n+\tmov_q\tx2, POR_EL1_INIT\n+\tmsr_s\tSYS_POR_EL1, x2\n+\t/* isb can be skipped since cpu_do_resume() will do it. */\n+.Lskip_por_set:\n+#endif /* CONFIG_ARM64_POE */\n+\n \t/*\n \t * cpu_do_resume expects x0 to contain context address pointer\n \t */\ndiff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c\nindex 0b52557652be6..a853a8d4e61af 100644\n--- a/arch/arm64/mm/fault.c\n+++ b/arch/arm64/mm/fault.c\n@@ -229,13 +229,16 @@ int __ptep_set_access_flags_anysz(struct vm_area_struct *vma,\n \t */\n \tpte_val(entry) ^= PTE_RDONLY;\n \tpteval = pte_val(pte);\n-\tdo {\n-\t\told_pteval = pteval;\n-\t\tpteval ^= PTE_RDONLY;\n-\t\tpteval |= pte_val(entry);\n-\t\tpteval ^= PTE_RDONLY;\n-\t\tpteval = cmpxchg_relaxed(\u0026pte_val(*ptep), old_pteval, pteval);\n-\t} while (pteval != old_pteval);\n+\tscoped_guard(kpkeys_hardened_pgtables) {\n+\t\tdo {\n+\t\t\told_pteval = pteval;\n+\t\t\tpteval ^= PTE_RDONLY;\n+\t\t\tpteval |= pte_val(entry);\n+\t\t\tpteval ^= PTE_RDONLY;\n+\t\t\tpteval = cmpxchg_relaxed(\u0026pte_val(*ptep), old_pteval,\n+\t\t\t\t\t\t pteval);\n+\t\t} while (pteval != old_pteval);\n+\t}\n \n \t/*\n \t * Invalidate the local stale read-only entry.  Remote stale entries\ndiff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c\nindex 97987f850a33c..7985045875cf6 100644\n--- a/arch/arm64/mm/init.c\n+++ b/arch/arm64/mm/init.c\n@@ -386,6 +386,7 @@ bool page_alloc_available __ro_after_init;\n void __init mem_init(void)\n {\n \tpage_alloc_available = true;\n+\tkpkeys_hardened_pgtables_init();\n \tswiotlb_update_mem_attributes();\n }\n \ndiff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c\nindex d4de88770ecfe..472ad0f8f002c 100644\n--- a/arch/arm64/mm/mmu.c\n+++ b/arch/arm64/mm/mmu.c\n@@ -107,8 +107,11 @@ static phys_addr_t __init early_pgtable_alloc(enum pgtable_level pgtable_level)\n {\n \tphys_addr_t phys;\n \n-\tphys = memblock_phys_alloc_range(PAGE_SIZE, PAGE_SIZE, 0,\n-\t\t\t\t\t MEMBLOCK_ALLOC_NOLEAKTRACE);\n+\tif (kpkeys_hardened_pgtables_early_enabled())\n+\t\tphys = kpkeys_physmem_pgtable_alloc();\n+\telse\n+\t\tphys = memblock_phys_alloc_range(PAGE_SIZE, PAGE_SIZE, 0,\n+\t\t\t\t\t\t MEMBLOCK_ALLOC_NOLEAKTRACE);\n \tif (!phys)\n \t\tpanic(\"Failed to allocate page table page\\n\");\n \n@@ -780,7 +783,7 @@ static inline bool force_pte_mapping(void)\n \tconst bool bbml2 = system_capabilities_finalized() ?\n \t\tsystem_supports_bbml2_noabort() : cpu_supports_bbml2_noabort();\n \n-\tif (debug_pagealloc_enabled())\n+\tif (debug_pagealloc_enabled() || kpkeys_hardened_pgtables_early_enabled())\n \t\treturn true;\n \tif (bbml2)\n \t\treturn false;\n@@ -1070,6 +1073,18 @@ void __init mark_linear_text_alias_ro(void)\n \t\t\t    PAGE_KERNEL_RO);\n }\n \n+#ifdef CONFIG_KPKEYS_HARDENED_PGTABLES\n+void __init arch_kpkeys_protect_static_pgtables(void)\n+{\n+\tunsigned long addr = (unsigned long)lm_alias(__pi_init_pg_dir);\n+\tunsigned long size = __pi_init_pg_end - __pi_init_pg_dir;\n+\tint ret;\n+\n+\tret = set_memory_pkey(addr, size / PAGE_SIZE, KPKEYS_PKEY_PGTABLES);\n+\tWARN_ON(ret);\n+}\n+#endif /* CONFIG_KPKEYS_HARDENED_PGTABLES */\n+\n #ifdef CONFIG_KFENCE\n \n bool __ro_after_init kfence_early_init = !!CONFIG_KFENCE_SAMPLE_INTERVAL;\n@@ -1214,6 +1229,14 @@ void mark_rodata_ro(void)\n \tupdate_mapping_prot(__pa_symbol(_text), (unsigned long)_text,\n \t\t\t    (unsigned long)_stext - (unsigned long)_text,\n \t\t\t    PAGE_KERNEL_RO);\n+\t/*\n+\t * Map the kernel image mapping of init_pg_dir read-only; it should\n+\t * only be written via the linear map.\n+\t */\n+\tsection_size = (unsigned long)__pi_init_pg_end - (unsigned long)__pi_init_pg_dir;\n+\tupdate_mapping_prot(__pa_symbol(__pi_init_pg_dir),\n+\t\t\t    (unsigned long)__pi_init_pg_dir,\n+\t\t\t    section_size, PAGE_KERNEL_RO);\n \n \t/* Map the kernel data/bss read-only in the linear map */\n \tupdate_mapping_prot(__pa_symbol(__init_end),\n@@ -1446,7 +1469,7 @@ static void free_hotplug_page_range(struct page *page, size_t size,\n static void free_hotplug_pgtable_page(struct page *page)\n {\n \tpagetable_dtor(page_ptdesc(page));\n-\tfree_hotplug_page_range(page, PAGE_SIZE, NULL);\n+\tpagetable_free(page_ptdesc(page));\n }\n \n static bool pgtable_range_aligned(unsigned long start, unsigned long end,\n@@ -2305,8 +2328,8 @@ void __cpu_replace_ttbr1(pgd_t *pgdp, bool cnp)\n #ifdef CONFIG_ARCH_HAS_PKEYS\n int arch_set_user_pkey_access(int pkey, unsigned long init_val)\n {\n-\tu64 new_por;\n-\tu64 old_por;\n+\tu8 new_perms;\n+\tu64 por;\n \n \tif (!system_supports_poe())\n \t\treturn -ENOSPC;\n@@ -2320,25 +2343,19 @@ int arch_set_user_pkey_access(int pkey, unsigned long init_val)\n \t\treturn -EINVAL;\n \n \t/* Set the bits we need in POR:  */\n-\tnew_por = POE_RWX;\n+\tnew_perms = POE_RWX;\n \tif (init_val \u0026 PKEY_DISABLE_WRITE)\n-\t\tnew_por \u0026= ~POE_W;\n+\t\tnew_perms \u0026= ~POE_W;\n \tif (init_val \u0026 PKEY_DISABLE_ACCESS)\n-\t\tnew_por \u0026= ~POE_RW;\n+\t\tnew_perms \u0026= ~POE_RW;\n \tif (init_val \u0026 PKEY_DISABLE_READ)\n-\t\tnew_por \u0026= ~POE_R;\n+\t\tnew_perms \u0026= ~POE_R;\n \tif (init_val \u0026 PKEY_DISABLE_EXECUTE)\n-\t\tnew_por \u0026= ~POE_X;\n-\n-\t/* Shift the bits in to the correct place in POR for pkey: */\n-\tnew_por = POR_ELx_PERM_PREP(pkey, new_por);\n-\n-\t/* Get old POR and mask off any old bits in place: */\n-\told_por = read_sysreg_s(SYS_POR_EL0);\n-\told_por \u0026= ~(POE_MASK \u003c\u003c POR_ELx_PERM_SHIFT(pkey));\n+\t\tnew_perms \u0026= ~POE_X;\n \n-\t/* Write old part along with new part: */\n-\twrite_sysreg_s(old_por | new_por, SYS_POR_EL0);\n+\tpor = read_sysreg_s(SYS_POR_EL0);\n+\tpor = por_elx_set_pkey_perms(por, pkey, new_perms);\n+\twrite_sysreg_s(por, SYS_POR_EL0);\n \n \treturn 0;\n }\ndiff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c\nindex bbe98ac9ad8c6..46fbebfc51003 100644\n--- a/arch/arm64/mm/pageattr.c\n+++ b/arch/arm64/mm/pageattr.c\n@@ -9,6 +9,8 @@\n #include \u003clinux/sched.h\u003e\n #include \u003clinux/vmalloc.h\u003e\n #include \u003clinux/pagewalk.h\u003e\n+#include \u003clinux/pkeys.h\u003e\n+#include \u003clinux/kpkeys.h\u003e\n \n #include \u003casm/cacheflush.h\u003e\n #include \u003casm/pgtable-prot.h\u003e\n@@ -99,7 +101,8 @@ bool can_set_direct_map(void)\n \t * Realms need to make pages shared/protected at page granularity.\n \t */\n \treturn rodata_full || debug_pagealloc_enabled() ||\n-\t\tarm64_kfence_can_set_direct_map() || is_realm_world();\n+\t\tarm64_kfence_can_set_direct_map() || is_realm_world() ||\n+\t\tkpkeys_hardened_pgtables_early_enabled();\n }\n \n static int update_range_prot(unsigned long start, unsigned long size,\n@@ -365,6 +368,30 @@ int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid)\n \treturn set_memory_valid(addr, nr, valid);\n }\n \n+#ifdef CONFIG_ARCH_HAS_KPKEYS\n+int set_memory_pkey(unsigned long addr, int numpages, int pkey)\n+{\n+\tunsigned long set_prot = 0;\n+\n+\tif (!kpkeys_supported())\n+\t\treturn 0;\n+\n+\tif (!__is_lm_address(kasan_reset_tag((void *)addr)))\n+\t\treturn -EINVAL;\n+\n+\tif (pkey \u003c 0 || pkey \u003e= arch_max_pkey())\n+\t\treturn -EINVAL;\n+\n+\tset_prot |= pkey \u0026 BIT(0) ? PTE_PO_IDX_0 : 0;\n+\tset_prot |= pkey \u0026 BIT(1) ? PTE_PO_IDX_1 : 0;\n+\tset_prot |= pkey \u0026 BIT(2) ? PTE_PO_IDX_2 : 0;\n+\n+\treturn __change_memory_common(addr, PAGE_SIZE * numpages,\n+\t\t\t\t      __pgprot(set_prot),\n+\t\t\t\t      __pgprot(PTE_PO_IDX_MASK));\n+}\n+#endif\n+\n #ifdef CONFIG_DEBUG_PAGEALLOC\n /*\n  * This is - apart from the return value - doing the same\ndiff --git a/include/linux/kpkeys.h b/include/linux/kpkeys.h\nnew file mode 100644\nindex 0000000000000..df55ec854586e\n--- /dev/null\n+++ b/include/linux/kpkeys.h\n@@ -0,0 +1,162 @@\n+/* SPDX-License-Identifier: GPL-2.0-only */\n+#ifndef _LINUX_KPKEYS_H\n+#define _LINUX_KPKEYS_H\n+\n+#include \u003clinux/bug.h\u003e\n+#include \u003clinux/cleanup.h\u003e\n+#include \u003clinux/jump_label.h\u003e\n+#include \u003clinux/kpkeys_types.h\u003e\n+\n+/**\n+ * KPKEYS_GUARD_NOOP() - define a guard type that does nothing\n+ * @name: the name of the guard type\n+ *\n+ * Define a guard type that does nothing, useful to match a real guard type\n+ * that is defined under an #ifdef.\n+ */\n+#define KPKEYS_GUARD_NOOP(name)\t\t\t\t\t\t\\\n+\t__DEFINE_CLASS_IS_CONDITIONAL(name, false);\t\t\t\\\n+\tDEFINE_CLASS(name, bool, (void)_T, false, void);\t\t\\\n+\tstatic inline void *class_##name##_lock_ptr(bool *_T)\t\t\\\n+\t{ return _T; }\n+\n+#ifdef CONFIG_ARCH_HAS_KPKEYS\n+\n+#include \u003casm/kpkeys.h\u003e\n+\n+/**\n+ * KPKEYS_GUARD_COND() - define a guard type that conditionally switches to\n+ *                       a given kpkeys context\n+ * @name: the name of the guard type\n+ * @ctx: the kpkeys context to switch to\n+ * @cond: an expression that is evaluated as condition\n+ *\n+ * Define a guard type that switches to @ctx if @cond evaluates to true,\n+ * and does nothing otherwise.\n+ */\n+#define KPKEYS_GUARD_COND(name, ctx, cond)\t\t\t\t\\\n+\t__DEFINE_CLASS_IS_CONDITIONAL(name, false);\t\t\t\\\n+\tDEFINE_CLASS(name, struct kpkeys_state,\t\t\t\t\\\n+\t\t     kpkeys_leave_context(\u0026_T),\t\t\t\t\\\n+\t\t     (cond) ? kpkeys_enter_context(ctx) :\t\t\\\n+\t\t\t      (struct kpkeys_state) {}, void);\t\t\\\n+\tstatic inline\t\t\t\t\t\t\t\\\n+\tvoid *class_##name##_lock_ptr(struct kpkeys_state *_T)\t\t\\\n+\t{ return _T; }\n+\n+/**\n+ * KPKEYS_GUARD() - define a guard type that switches to a given kpkeys context\n+ *                  if kpkeys are supported\n+ * @name: the name of the guard type\n+ * @ctx: the kpkeys context to switch to\n+ *\n+ * Define a guard type that switches to @ctx if the system supports kpkeys.\n+ */\n+#define KPKEYS_GUARD(name, ctx)\t\t\t\t\t\t\\\n+\tKPKEYS_GUARD_COND(name, ctx, kpkeys_supported())\n+\n+/**\n+ * kpkeys_enter_context() - enter a kpkeys context\n+ * @ctx: the context to switch to\n+ *\n+ * Enters the specified kpkeys context. @ctx must be a compile-time constant.\n+ *\n+ * Return: state to be passed to kpkeys_leave_context().\n+ */\n+static __always_inline\n+struct kpkeys_state kpkeys_enter_context(enum kpkeys_ctx ctx)\n+{\n+\tBUILD_BUG_ON_MSG(!__builtin_constant_p(ctx),\n+\t\t\t \"kpkeys_enter_context() only takes constant values\");\n+\tBUILD_BUG_ON_MSG(ctx \u003c 0 || ctx \u003e= KPKEYS_CTX_COUNT,\n+\t\t\t \"Invalid value passed to kpkeys_enter_context()\");\n+\n+\treturn arch_kpkeys_enter_context(ctx);\n+}\n+\n+/**\n+ * kpkeys_leave_context() - leave a kpkeys context\n+ * @state: state returned by kpkeys_enter_context()\n+ *\n+ * Restores the state saved when entering a kpkeys context. If no context was\n+ * entered, this function does nothing.\n+ */\n+static __always_inline\n+void kpkeys_leave_context(const struct kpkeys_state *state)\n+{\n+\tif (state-\u003eentered_context)\n+\t\tarch_kpkeys_leave_context(state);\n+}\n+\n+static inline bool kpkeys_supported(void)\n+{\n+\treturn arch_supports_kpkeys();\n+}\n+\n+#else /* CONFIG_ARCH_HAS_KPKEYS */\n+\n+static inline bool kpkeys_supported(void)\n+{\n+\treturn false;\n+}\n+\n+#endif /* CONFIG_ARCH_HAS_KPKEYS */\n+\n+#ifdef CONFIG_KPKEYS_HARDENED_PGTABLES\n+\n+DECLARE_STATIC_KEY_FALSE(kpkeys_hardened_pgtables_key);\n+\n+static inline bool kpkeys_hardened_pgtables_enabled(void)\n+{\n+\treturn static_branch_unlikely(\u0026kpkeys_hardened_pgtables_key);\n+}\n+\n+static inline bool kpkeys_hardened_pgtables_early_enabled(void)\n+{\n+\treturn arch_supports_kpkeys_early();\n+}\n+\n+struct page *kpkeys_pgtable_alloc(gfp_t gfp, unsigned int order);\n+void kpkeys_pgtable_free(struct page *page, unsigned int order);\n+\n+/*\n+ * Should be called from mem_init(): as soon as the buddy allocator becomes\n+ * available and before any call to pagetable_alloc().\n+ */\n+void kpkeys_hardened_pgtables_init(void);\n+\n+phys_addr_t kpkeys_physmem_pgtable_alloc(void);\n+\n+#ifndef arch_kpkeys_protect_static_pgtables\n+static inline void arch_kpkeys_protect_static_pgtables(void) {}\n+#endif\n+\n+#else /* CONFIG_KPKEYS_HARDENED_PGTABLES */\n+\n+static inline bool kpkeys_hardened_pgtables_enabled(void)\n+{\n+\treturn false;\n+}\n+\n+static inline bool kpkeys_hardened_pgtables_early_enabled(void)\n+{\n+\treturn false;\n+}\n+\n+static inline struct page *kpkeys_pgtable_alloc(gfp_t gfp, unsigned int order)\n+{\n+\treturn NULL;\n+}\n+\n+static inline void kpkeys_pgtable_free(struct page *page, unsigned int order) {}\n+\n+static inline void kpkeys_hardened_pgtables_init(void) {}\n+\n+static inline phys_addr_t kpkeys_physmem_pgtable_alloc(void)\n+{\n+\treturn 0;\n+}\n+\n+#endif /* CONFIG_KPKEYS_HARDENED_PGTABLES */\n+\n+#endif /* _LINUX_KPKEYS_H */\ndiff --git a/include/linux/kpkeys_types.h b/include/linux/kpkeys_types.h\nnew file mode 100644\nindex 0000000000000..09f15a9565cf3\n--- /dev/null\n+++ b/include/linux/kpkeys_types.h\n@@ -0,0 +1,30 @@\n+/* SPDX-License-Identifier: GPL-2.0-only */\n+#ifndef _LINUX_KPKEYS_TYPES_H\n+#define _LINUX_KPKEYS_TYPES_H\n+\n+#include \u003clinux/types.h\u003e\n+\n+#ifdef CONFIG_ARCH_HAS_KPKEYS\n+#include \u003casm/kpkeys_types.h\u003e\n+#endif\n+\n+#ifndef KPKEYS_PKEY_PGTABLES\n+#define KPKEYS_PKEY_PGTABLES\t1\n+#endif\n+\n+#if defined(CONFIG_ARCH_HAS_KPKEYS) \u0026\u0026 !defined(__ASSEMBLY__)\n+\n+enum kpkeys_ctx {\n+\tKPKEYS_CTX_DEFAULT = 0,\n+\tKPKEYS_CTX_PGTABLES,\n+\tKPKEYS_CTX_COUNT,\n+};\n+\n+struct kpkeys_state {\n+\tbool entered_context;\n+\tstruct arch_kpkeys_state arch_state;\n+};\n+\n+#endif /* CONFIG_ARCH_HAS_KPKEYS \u0026\u0026 !__ASSEMBLY__ */\n+\n+#endif /* _LINUX_KPKEYS_TYPES_H */\ndiff --git a/include/linux/memblock.h b/include/linux/memblock.h\nindex 5afcd99aa8c14..c61f80e31cc10 100644\n--- a/include/linux/memblock.h\n+++ b/include/linux/memblock.h\n@@ -24,6 +24,17 @@ extern unsigned long max_pfn;\n  */\n extern unsigned long long max_possible_pfn;\n \n+#define INIT_MEMBLOCK_REGIONS\t\t\t128\n+#define INIT_PHYSMEM_REGIONS\t\t\t4\n+\n+#ifndef INIT_MEMBLOCK_RESERVED_REGIONS\n+#define INIT_MEMBLOCK_RESERVED_REGIONS\t\tINIT_MEMBLOCK_REGIONS\n+#endif\n+\n+#ifndef INIT_MEMBLOCK_MEMORY_REGIONS\n+#define INIT_MEMBLOCK_MEMORY_REGIONS\t\tINIT_MEMBLOCK_REGIONS\n+#endif\n+\n /**\n  * enum memblock_flags - definition of memory region attributes\n  * @MEMBLOCK_NONE: no special request\ndiff --git a/include/linux/mm.h b/include/linux/mm.h\nindex 485df9c2dbddb..eea3131786b35 100644\n--- a/include/linux/mm.h\n+++ b/include/linux/mm.h\n@@ -37,6 +37,7 @@\n #include \u003clinux/bitmap.h\u003e\n #include \u003clinux/bitops.h\u003e\n #include \u003clinux/iommu-debug-pagealloc.h\u003e\n+#include \u003clinux/kpkeys.h\u003e\n \n struct mempolicy;\n struct anon_vma;\n@@ -3682,7 +3683,12 @@ static inline bool ptdesc_test_kernel(const struct ptdesc *ptdesc)\n  */\n static inline struct ptdesc *pagetable_alloc_noprof(gfp_t gfp, unsigned int order)\n {\n-\tstruct page *page = alloc_pages_noprof(gfp | __GFP_COMP, order);\n+\tstruct page *page;\n+\n+\tif (kpkeys_hardened_pgtables_enabled())\n+\t\tpage = kpkeys_pgtable_alloc(gfp | __GFP_COMP, order);\n+\telse\n+\t\tpage = alloc_pages_noprof(gfp | __GFP_COMP, order);\n \n \treturn page_ptdesc(page);\n }\n@@ -3691,8 +3697,12 @@ static inline struct ptdesc *pagetable_alloc_noprof(gfp_t gfp, unsigned int orde\n static inline void __pagetable_free(struct ptdesc *pt)\n {\n \tstruct page *page = ptdesc_page(pt);\n+\tunsigned int order = compound_order(page);\n \n-\t__free_pages(page, compound_order(page));\n+\tif (kpkeys_hardened_pgtables_enabled())\n+\t\tkpkeys_pgtable_free(page, order);\n+\telse\n+\t\t__free_pages(page, order);\n }\n \n #ifdef CONFIG_ASYNC_KERNEL_PGTABLE_FREE\ndiff --git a/include/linux/set_memory.h b/include/linux/set_memory.h\nindex 3030d9245f5ac..7b3a8bfde3c64 100644\n--- a/include/linux/set_memory.h\n+++ b/include/linux/set_memory.h\n@@ -84,4 +84,11 @@ static inline int set_memory_decrypted(unsigned long addr, int numpages)\n }\n #endif /* CONFIG_ARCH_HAS_MEM_ENCRYPT */\n \n+#ifndef CONFIG_ARCH_HAS_KPKEYS\n+static inline int set_memory_pkey(unsigned long addr, int numpages, int pkey)\n+{\n+\treturn 0;\n+}\n+#endif\n+\n #endif /* _LINUX_SET_MEMORY_H_ */\ndiff --git a/mm/Kconfig b/mm/Kconfig\nindex 9e0ca48249054..21bd3e550de83 100644\n--- a/mm/Kconfig\n+++ b/mm/Kconfig\n@@ -1281,6 +1281,11 @@ config ARCH_USES_HIGH_VMA_FLAGS\n \tbool\n config ARCH_HAS_PKEYS\n \tbool\n+config ARCH_HAS_KPKEYS\n+\tbool\n+config ARCH_HAS_KPKEYS_HARDENED_PGTABLES\n+\tbool\n+\tdepends on ARCH_HAS_KPKEYS\n \n config ARCH_USES_PG_ARCH_2\n \tbool\ndiff --git a/mm/Makefile b/mm/Makefile\nindex eff9f9e7e061c..e31e582948451 100644\n--- a/mm/Makefile\n+++ b/mm/Makefile\n@@ -147,3 +147,5 @@ obj-$(CONFIG_SHRINKER_DEBUG) += shrinker_debug.o\n obj-$(CONFIG_EXECMEM) += execmem.o\n obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o\n obj-$(CONFIG_LAZY_MMU_MODE_KUNIT_TEST) += tests/lazy_mmu_mode_kunit.o\n+obj-$(CONFIG_KPKEYS_HARDENED_PGTABLES) += kpkeys_hardened_pgtables.o\n+obj-$(CONFIG_KPKEYS_HARDENED_PGTABLES_KUNIT_TEST) += tests/kpkeys_hardened_pgtables_kunit.o\ndiff --git a/mm/kpkeys_hardened_pgtables.c b/mm/kpkeys_hardened_pgtables.c\nnew file mode 100644\nindex 0000000000000..c133e5896fccc\n--- /dev/null\n+++ b/mm/kpkeys_hardened_pgtables.c\n@@ -0,0 +1,179 @@\n+// SPDX-License-Identifier: GPL-2.0-only\n+#include \u003clinux/kpkeys.h\u003e\n+#include \u003clinux/memblock.h\u003e\n+#include \u003clinux/mm.h\u003e\n+#include \u003clinux/set_memory.h\u003e\n+\n+#include \u003ckunit/visibility.h\u003e\n+\n+__ro_after_init DEFINE_STATIC_KEY_FALSE(kpkeys_hardened_pgtables_key);\n+EXPORT_SYMBOL_IF_KUNIT(kpkeys_hardened_pgtables_key);\n+\n+static int set_pkey_pgtable(struct page *page, unsigned int nr_pages)\n+{\n+\tunsigned long addr = (unsigned long)page_address(page);\n+\tint ret;\n+\n+\tret = set_memory_pkey(addr, nr_pages, KPKEYS_PKEY_PGTABLES);\n+\n+\tWARN_ON(ret);\n+\treturn ret;\n+}\n+\n+static int set_pkey_default(struct page *page, unsigned int nr_pages)\n+{\n+\tunsigned long addr = (unsigned long)page_address(page);\n+\tint ret;\n+\n+\tret = set_memory_pkey(addr, nr_pages, 0);\n+\n+\tWARN_ON(ret);\n+\treturn ret;\n+}\n+\n+/* pkeys physmem allocator (PPA) - implemented below */\n+static void ppa_finalize(void);\n+\n+struct page *kpkeys_pgtable_alloc(gfp_t gfp, unsigned int order)\n+{\n+\tstruct page *page;\n+\tint ret;\n+\n+\tpage = alloc_pages_noprof(gfp, order);\n+\tif (!page)\n+\t\treturn page;\n+\n+\tret = set_pkey_pgtable(page, 1 \u003c\u003c order);\n+\tif (ret) {\n+\t\t__free_pages(page, order);\n+\t\treturn NULL;\n+\t}\n+\n+\treturn page;\n+}\n+\n+void kpkeys_pgtable_free(struct page *page, unsigned int order)\n+{\n+\tset_pkey_default(page, 1 \u003c\u003c order);\n+\t__free_pages(page, order);\n+}\n+\n+void __init kpkeys_hardened_pgtables_init(void)\n+{\n+\tif (!kpkeys_supported())\n+\t\treturn;\n+\n+\tstatic_branch_enable(\u0026kpkeys_hardened_pgtables_key);\n+\n+\tppa_finalize();\n+\tarch_kpkeys_protect_static_pgtables();\n+}\n+\n+/*\n+ * pkeys physmem allocator (PPA): allocator for very early page tables\n+ * (especially for creating the linear map), based on memblock. Allocated\n+ * ranges are tracked so that their pkey can be set once it is safe to do so.\n+ */\n+\n+/*\n+ * We may have to track many ranges when allocating page tables for the linear\n+ * map, as their number grows with the amount of available memory. Assuming that\n+ * memblock returns contiguous blocks whenever possible, the number of ranges\n+ * to track cannot however exceed the number of regions that memblock itself\n+ * tracks. memblock_allow_resize() hasn't been called yet at that point, so\n+ * that limit is the size of the statically allocated array.\n+ */\n+#define PHYSMEM_MAX_RANGES\tINIT_MEMBLOCK_MEMORY_REGIONS\n+\n+struct physmem_range {\n+\tphys_addr_t addr;\n+\tphys_addr_t size;\n+};\n+\n+struct pkeys_physmem_allocator {\n+\tstruct physmem_range allocated_ranges[PHYSMEM_MAX_RANGES];\n+\tunsigned int nr_allocated_ranges;\n+};\n+\n+static struct pkeys_physmem_allocator pkeys_physmem_allocator __initdata;\n+\n+static int __init set_pkey_pgtable_phys(phys_addr_t pa, phys_addr_t size)\n+{\n+\tunsigned long addr = (unsigned long)__va(pa);\n+\tint ret;\n+\n+\tret = set_memory_pkey(addr, size / PAGE_SIZE, KPKEYS_PKEY_PGTABLES);\n+\n+\tWARN_ON(ret);\n+\treturn ret;\n+}\n+\n+static bool __init ppa_try_extend_last_range(phys_addr_t addr, phys_addr_t size)\n+{\n+\tstruct pkeys_physmem_allocator *ppa = \u0026pkeys_physmem_allocator;\n+\tstruct physmem_range *range;\n+\n+\tif (!ppa-\u003enr_allocated_ranges)\n+\t\treturn false;\n+\n+\trange = \u0026ppa-\u003eallocated_ranges[ppa-\u003enr_allocated_ranges - 1];\n+\n+\t/* Merge the new range into the last range if they are contiguous */\n+\tif (addr == range-\u003eaddr + range-\u003esize) {\n+\t\trange-\u003esize += size;\n+\t\treturn true;\n+\t} else if (addr + size == range-\u003eaddr) {\n+\t\trange-\u003eaddr -= size;\n+\t\trange-\u003esize += size;\n+\t\treturn true;\n+\t}\n+\n+\treturn false;\n+}\n+\n+static void __init ppa_register_allocated_range(phys_addr_t addr,\n+\t\t\t\t\t\tphys_addr_t size)\n+{\n+\tstruct pkeys_physmem_allocator *ppa = \u0026pkeys_physmem_allocator;\n+\tstruct physmem_range *range;\n+\n+\tif (!addr)\n+\t\treturn;\n+\n+\tif (ppa_try_extend_last_range(addr, size))\n+\t\treturn;\n+\n+\t/* Could not extend the last range, create a new one */\n+\tif (WARN_ON(ppa-\u003enr_allocated_ranges \u003e= PHYSMEM_MAX_RANGES))\n+\t\treturn;\n+\n+\trange = \u0026ppa-\u003eallocated_ranges[ppa-\u003enr_allocated_ranges++];\n+\trange-\u003eaddr = addr;\n+\trange-\u003esize = size;\n+}\n+\n+static void __init ppa_finalize(void)\n+{\n+\tstruct pkeys_physmem_allocator *ppa = \u0026pkeys_physmem_allocator;\n+\n+\tfor (unsigned int i = 0; i \u003c ppa-\u003enr_allocated_ranges; i++) {\n+\t\tstruct physmem_range *range = \u0026ppa-\u003eallocated_ranges[i];\n+\n+\t\tset_pkey_pgtable_phys(range-\u003eaddr, range-\u003esize);\n+\t}\n+}\n+\n+phys_addr_t __ref kpkeys_physmem_pgtable_alloc(void)\n+{\n+\tsize_t size = PAGE_SIZE;\n+\tphys_addr_t addr;\n+\n+\taddr = memblock_phys_alloc_range(size, size, 0,\n+\t\t\t\t\t MEMBLOCK_ALLOC_NOLEAKTRACE);\n+\tif (!addr)\n+\t\treturn addr;\n+\n+\tppa_register_allocated_range(addr, size);\n+\n+\treturn addr;\n+}\ndiff --git a/mm/memblock.c b/mm/memblock.c\nindex 6349c48154f4b..3b497d3556bb0 100644\n--- a/mm/memblock.c\n+++ b/mm/memblock.c\n@@ -30,17 +30,6 @@\n \n #include \"internal.h\"\n \n-#define INIT_MEMBLOCK_REGIONS\t\t\t128\n-#define INIT_PHYSMEM_REGIONS\t\t\t4\n-\n-#ifndef INIT_MEMBLOCK_RESERVED_REGIONS\n-# define INIT_MEMBLOCK_RESERVED_REGIONS\t\tINIT_MEMBLOCK_REGIONS\n-#endif\n-\n-#ifndef INIT_MEMBLOCK_MEMORY_REGIONS\n-#define INIT_MEMBLOCK_MEMORY_REGIONS\t\tINIT_MEMBLOCK_REGIONS\n-#endif\n-\n /**\n  * DOC: memblock overview\n  *\ndiff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c\nindex ebd3ac997f645..1fec89168ff10 100644\n--- a/mm/sparse-vmemmap.c\n+++ b/mm/sparse-vmemmap.c\n@@ -179,13 +179,29 @@ static pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, in\n \treturn pte;\n }\n \n-static void * __meminit vmemmap_alloc_block_zero(unsigned long size, int node)\n+static void * __meminit vmemmap_alloc_pgtable(int node)\n {\n-\tvoid *p = vmemmap_alloc_block(size, node);\n+\tvoid *p;\n+\n+\tif (slab_is_available()) {\n+\t\tgfp_t gfp = GFP_KERNEL | __GFP_ZERO;\n+\t\tstruct ptdesc *ptdesc = pagetable_alloc(gfp, 0);\n+\n+\t\treturn ptdesc ? ptdesc_address(ptdesc) : NULL;\n+\t}\n+\n+\tif (kpkeys_hardened_pgtables_early_enabled()) {\n+\t\tphys_addr_t phys = kpkeys_physmem_pgtable_alloc();\n+\n+\t\tp = phys ? phys_to_virt(phys) : NULL;\n+\t} else {\n+\t\tp = __earlyonly_bootmem_alloc(node, PAGE_SIZE, PAGE_SIZE,\n+\t\t\t\t\t      __pa(MAX_DMA_ADDRESS));\n+\t}\n \n \tif (!p)\n \t\treturn NULL;\n-\tmemset(p, 0, size);\n+\tmemset(p, 0, PAGE_SIZE);\n \n \treturn p;\n }\n@@ -194,7 +210,7 @@ static pmd_t * __meminit vmemmap_pmd_populate(pud_t *pud, unsigned long addr, in\n {\n \tpmd_t *pmd = pmd_offset(pud, addr);\n \tif (pmd_none(*pmd)) {\n-\t\tvoid *p = vmemmap_alloc_block_zero(PAGE_SIZE, node);\n+\t\tvoid *p = vmemmap_alloc_pgtable(node);\n \t\tif (!p)\n \t\t\treturn NULL;\n \t\tkernel_pte_init(p);\n@@ -207,7 +223,7 @@ static pud_t * __meminit vmemmap_pud_populate(p4d_t *p4d, unsigned long addr, in\n {\n \tpud_t *pud = pud_offset(p4d, addr);\n \tif (pud_none(*pud)) {\n-\t\tvoid *p = vmemmap_alloc_block_zero(PAGE_SIZE, node);\n+\t\tvoid *p = vmemmap_alloc_pgtable(node);\n \t\tif (!p)\n \t\t\treturn NULL;\n \t\tpmd_init(p);\n@@ -220,7 +236,7 @@ static p4d_t * __meminit vmemmap_p4d_populate(pgd_t *pgd, unsigned long addr, in\n {\n \tp4d_t *p4d = p4d_offset(pgd, addr);\n \tif (p4d_none(*p4d)) {\n-\t\tvoid *p = vmemmap_alloc_block_zero(PAGE_SIZE, node);\n+\t\tvoid *p = vmemmap_alloc_pgtable(node);\n \t\tif (!p)\n \t\t\treturn NULL;\n \t\tpud_init(p);\n@@ -233,7 +249,7 @@ static pgd_t * __meminit vmemmap_pgd_populate(unsigned long addr, int node)\n {\n \tpgd_t *pgd = pgd_offset_k(addr);\n \tif (pgd_none(*pgd)) {\n-\t\tvoid *p = vmemmap_alloc_block_zero(PAGE_SIZE, node);\n+\t\tvoid *p = vmemmap_alloc_pgtable(node);\n \t\tif (!p)\n \t\t\treturn NULL;\n \t\tpgd_populate_kernel(addr, pgd, p);\n@@ -346,10 +362,11 @@ static __meminit struct page *vmemmap_get_tail(unsigned int order, struct zone *\n \t * memmap_init().\n \t */\n \n-\tp = vmemmap_alloc_block_zero(PAGE_SIZE, node);\n+\tp = vmemmap_alloc_block(PAGE_SIZE, node);\n \tif (!p)\n \t\treturn NULL;\n \n+\tmemset(p, 0, PAGE_SIZE);\n \ttail = virt_to_page(p);\n \tzone-\u003evmemmap_tails[idx] = tail;\n \ndiff --git a/mm/tests/kpkeys_hardened_pgtables_kunit.c b/mm/tests/kpkeys_hardened_pgtables_kunit.c\nnew file mode 100644\nindex 0000000000000..80d4f5144e48d\n--- /dev/null\n+++ b/mm/tests/kpkeys_hardened_pgtables_kunit.c\n@@ -0,0 +1,200 @@\n+// SPDX-License-Identifier: GPL-2.0-only\n+#include \u003ckunit/test.h\u003e\n+#include \u003clinux/mman.h\u003e\n+#include \u003clinux/pgtable.h\u003e\n+#include \u003clinux/set_memory.h\u003e\n+#include \u003clinux/vmalloc.h\u003e\n+\n+static void free_page_wrapper(void *ctx)\n+{\n+\t__free_page((struct page *)ctx);\n+}\n+\n+KUNIT_DEFINE_ACTION_WRAPPER(vfree_wrapper, vfree, const void *);\n+\n+static pud_t *pud_off_k(unsigned long va)\n+{\n+\treturn pud_offset(p4d_offset(pgd_offset_k(va), va), va);\n+}\n+\n+static pte_t *get_kernel_pte(unsigned long addr)\n+{\n+\tpmd_t *pmdp = pmd_off_k(addr);\n+\n+\tif (!pmdp || pmd_leaf(*pmdp))\n+\t\treturn NULL;\n+\n+\treturn pte_offset_kernel(pmdp, addr);\n+}\n+\n+#define write_pgtable(type, ptr) do {\t\t\t\t\t\\\n+\ttype##_t val;\t\t\t\t\t\t\t\\\n+\tint ret;\t\t\t\t\t\t\t\\\n+\t\t\t\t\t\t\t\t\t\\\n+\tpr_debug(\"%s: writing to \"#type\" at %px\\n\", __func__, (ptr));\t\\\n+\t\t\t\t\t\t\t\t\t\\\n+\tval = type##p_get(ptr);\t\t\t\t\t\t\\\n+\tret = copy_to_kernel_nofault(ptr, \u0026val, sizeof(val));\t\t\\\n+\tKUNIT_EXPECT_EQ_MSG(test, ret, -EFAULT,\t\t\t\t\\\n+\t\t\t    \"Direct \"#type\" write wasn't prevented\");\t\\\n+} while (0)\n+\n+/*\n+ * Try to write linear map page tables, at every level. This is worthwhile\n+ * because those page table pages are obtained from different allocators:\n+ *\n+ * - Static memory (part of the kernel image) for PGD\n+ * - memblock for PUD and possibly PMD/PTE\n+ * - pagetable_alloc() (buddy allocator) for PMD/PTE if large block mappings are\n+ *   used and the linear map is split after being created\n+ */\n+static void write_direct_map_pgtables(struct kunit *test)\n+{\n+\tstruct page *page;\n+\tunsigned long addr;\n+\tpgd_t *pgdp;\n+\tp4d_t *p4dp;\n+\tpud_t *pudp;\n+\tpmd_t *pmdp;\n+\tpte_t *ptep;\n+\tint ret;\n+\n+\tif (!kpkeys_supported())\n+\t\tkunit_skip(test, \"kpkeys are not supported\");\n+\n+\tpage = alloc_page(GFP_KERNEL);\n+\tKUNIT_ASSERT_NOT_NULL(test, page);\n+\tret = kunit_add_action_or_reset(test, free_page_wrapper, page);\n+\tKUNIT_ASSERT_EQ(test, ret, 0);\n+\n+\t/* Ensure page is PTE-mapped (splitting the linear map if necessary) */\n+\tret = set_direct_map_invalid_noflush(page);\n+\tKUNIT_ASSERT_EQ(test, ret, 0);\n+\tret = set_direct_map_default_noflush(page);\n+\tKUNIT_ASSERT_EQ(test, ret, 0);\n+\n+\taddr = (unsigned long)page_address(page);\n+\n+\tpgdp = pgd_offset_k(addr);\n+\tKUNIT_ASSERT_NOT_NULL_MSG(test, pgdp, \"Failed to get PGD\");\n+\t/*\n+\t * swapper_pg_dir is still writable at this stage, so don't check it.\n+\t * It is not protected by kpkeys_hardened_pgtables because it should be\n+\t * made read-only by mark_rodata_ro(). However since these\n+\t * KUnit tests are builtin, they are run before mark_rodata_ro() is\n+\t * called.\n+\t */\n+\n+\tp4dp = p4d_offset(pgdp, addr);\n+\tKUNIT_ASSERT_NOT_NULL_MSG(test, p4dp, \"Failed to get P4D\");\n+\tif (!mm_p4d_folded(\u0026init_mm))\n+\t\twrite_pgtable(p4d, p4dp);\n+\n+\tpudp = pud_offset(p4dp, addr);\n+\tKUNIT_ASSERT_NOT_NULL_MSG(test, pudp, \"Failed to get PUD\");\n+\tif (!mm_pud_folded(\u0026init_mm))\n+\t\twrite_pgtable(pud, pudp);\n+\n+\tpmdp = pmd_offset(pudp, addr);\n+\tKUNIT_ASSERT_NOT_NULL_MSG(test, pmdp, \"Failed to get PMD\");\n+\twrite_pgtable(pmd, pmdp);\n+\n+\tptep = pte_offset_kernel(pmdp, addr);\n+\tKUNIT_ASSERT_NOT_NULL_MSG(test, ptep, \"Failed to get PTE\");\n+\twrite_pgtable(pte, ptep);\n+}\n+\n+/* Worth checking since the kernel image is mapped with static page tables */\n+static void write_kernel_image_pud(struct kunit *test)\n+{\n+\tpud_t *pudp;\n+\n+\tif (!kpkeys_supported())\n+\t\tkunit_skip(test, \"kpkeys are not supported\");\n+\n+\t/* The kernel is probably block-mapped, check the PUD to be safe */\n+\tpudp = pud_off_k((unsigned long)\u0026init_mm);\n+\tKUNIT_ASSERT_NOT_NULL_MSG(test, pudp, \"Failed to get PUD\");\n+\n+\twrite_pgtable(pud, pudp);\n+}\n+\n+static void write_kernel_vmalloc_pte(struct kunit *test)\n+{\n+\tvoid *mem;\n+\tpte_t *ptep;\n+\tint ret;\n+\n+\tif (!kpkeys_supported())\n+\t\tkunit_skip(test, \"kpkeys are not supported\");\n+\n+\tmem = vmalloc(PAGE_SIZE);\n+\tKUNIT_ASSERT_NOT_NULL(test, mem);\n+\tret = kunit_add_action_or_reset(test, vfree_wrapper, mem);\n+\tKUNIT_ASSERT_EQ(test, ret, 0);\n+\n+\t/* vmalloc() without VM_ALLOW_HUGE_VMAP is PTE-mapped */\n+\tptep = get_kernel_pte((unsigned long)mem);\n+\tKUNIT_ASSERT_NOT_NULL_MSG(test, ptep, \"Failed to get PTE\");\n+\n+\twrite_pgtable(pte, ptep);\n+}\n+\n+static void write_vmemmap_pmd(struct kunit *test)\n+{\n+\tstruct page *page;\n+\tpmd_t *pmdp;\n+\n+\tif (!kpkeys_supported())\n+\t\tkunit_skip(test, \"kpkeys are not supported\");\n+\n+\t/*\n+\t * We just need the address of some struct page, so we can free the\n+\t * page right away.\n+\t */\n+\tpage = alloc_page(GFP_KERNEL);\n+\tKUNIT_ASSERT_NOT_NULL(test, page);\n+\t__free_page(page);\n+\n+\t/* vmemmap may use PMD block mappings */\n+\tpmdp = pmd_off_k((unsigned long)page);\n+\tKUNIT_ASSERT_NOT_NULL_MSG(test, pmdp, \"Failed to get PMD\");\n+\twrite_pgtable(pmd, pmdp);\n+}\n+\n+static void write_user_pmd(struct kunit *test)\n+{\n+\tpmd_t *pmdp;\n+\tunsigned long uaddr;\n+\n+\tif (!kpkeys_supported())\n+\t\tkunit_skip(test, \"kpkeys are not supported\");\n+\n+\tuaddr = kunit_vm_mmap(test, NULL, 0, PAGE_SIZE, PROT_READ,\n+\t\t\t      MAP_ANONYMOUS | MAP_PRIVATE | MAP_POPULATE, 0);\n+\tKUNIT_ASSERT_NE_MSG(test, uaddr, 0, \"Could not create userspace mm\");\n+\n+\t/* We passed MAP_POPULATE so a PMD should already be allocated */\n+\tpmdp = pmd_off(current-\u003emm, uaddr);\n+\tKUNIT_ASSERT_NOT_NULL_MSG(test, pmdp, \"Failed to get PMD\");\n+\n+\twrite_pgtable(pmd, pmdp);\n+}\n+\n+static struct kunit_case kpkeys_hardened_pgtables_test_cases[] = {\n+\tKUNIT_CASE(write_direct_map_pgtables),\n+\tKUNIT_CASE(write_kernel_image_pud),\n+\tKUNIT_CASE(write_kernel_vmalloc_pte),\n+\tKUNIT_CASE(write_vmemmap_pmd),\n+\tKUNIT_CASE(write_user_pmd),\n+\t{}\n+};\n+\n+static struct kunit_suite kpkeys_hardened_pgtables_test_suite = {\n+\t.name = \"kpkeys_hardened_pgtables\",\n+\t.test_cases = kpkeys_hardened_pgtables_test_cases,\n+};\n+kunit_test_suite(kpkeys_hardened_pgtables_test_suite);\n+\n+MODULE_DESCRIPTION(\"Tests for the kpkeys_hardened_pgtables feature\");\n+MODULE_LICENSE(\"GPL\");\ndiff --git a/security/Kconfig.hardening b/security/Kconfig.hardening\nindex 6923036e1a2f5..ad6152961f8ee 100644\n--- a/security/Kconfig.hardening\n+++ b/security/Kconfig.hardening\n@@ -271,6 +271,30 @@ config BUG_ON_DATA_CORRUPTION\n \n \t  If unsure, say N.\n \n+config KPKEYS_HARDENED_PGTABLES\n+\tbool \"Harden page tables using kernel pkeys\"\n+\tdepends on ARCH_HAS_KPKEYS_HARDENED_PGTABLES\n+\thelp\n+\t  This option makes all page tables mostly read-only by\n+\t  allocating them with a non-default protection key (pkey) and\n+\t  only enabling write access to that pkey in routines that are\n+\t  expected to write to page table entries.\n+\n+\t  This option has no effect if the system does not support\n+\t  kernel pkeys.\n+\n+config KPKEYS_HARDENED_PGTABLES_KUNIT_TEST\n+\tbool \"KUnit tests for kpkeys_hardened_pgtables\" if !KUNIT_ALL_TESTS\n+\tdepends on KPKEYS_HARDENED_PGTABLES\n+\tdepends on KUNIT=y\n+\tdefault KUNIT_ALL_TESTS\n+\thelp\n+\t  Enable this option to check that the kpkeys_hardened_pgtables feature\n+\t  functions as intended, i.e. prevents arbitrary writes to user and\n+\t  kernel page tables.\n+\n+\t  If unsure, say N.\n+\n endmenu\n \n config CC_HAS_RANDSTRUCT\n"
}

{
  "EnableConfigs": [
    "KPKEYS_HARDENED_PGTABLES"
  ],
  "FocusSymbols": [
    "kpkeys_pgtable_alloc",
    "kpkeys_pgtable_free",
    "kpkeys_physmem_pgtable_alloc",
    "set_memory_pkey",
    "arch_set_user_pkey_access"
  ],
  "Reasoning": "The patch introduces a new feature to harden page tables using kernel pkeys (kpkeys) on ARM64. It adds a new file `mm/kpkeys_hardened_pgtables.c` and modifies ARM64 memory management code to allocate page tables with a specific pkey and restrict write access to them. The changes include new page table allocation and freeing functions, as well as modifications to `arch_set_user_pkey_access`. This is a significant functional change to the core memory management subsystem that should be fuzzed to ensure memory safety and correctness.",
  "WorthFuzzing": true
}

You are an expert Linux kernel maintainer.
Your job is to review a provided patch series and determine
if it makes functional changes to the kernel that should be fuzzed.

IMPORTANT: The changes have ALREADY been applied and committed as the HEAD commit in
your workspace. Do NOT rely on your internal knowledge of the kernel. You must actively
use your code access tools to examine the actual source code and confirm any assumptions.

Return WorthFuzzing=false if the patch only contains:
- Modifications to Documentation/, Kconfig files, or code comments.
- Purely decorative changes, such as logging (e.g., pr_err, printk) or tracepoints.
- Changes to numeric constants or macros that do not functionally alter execution flow.
- Code paths that are impossible to reach in virtualized environments like GCE or QEMU,
  even when utilizing software-emulated hardware (e.g., usb gadget, mac80211_hwsim).
- Code in vendor-specific PCIe switch, SmartNIC, or GPU drivers (e.g., mlxsw, pds_core, qed,
  ionic, amdgpu) that require physical PCIe hardware cards not emulated in standard QEMU.
- Driver .remove, .shutdown, or pci_unregister_driver teardown callbacks (e.g., igb_remove)
  that are executed only during PCI hot-unplug or sysfs driver unbind operations.

If it modifies reachable core kernel logic, drivers, or architectures, use your code search
tools to verify the code can be executed, then return WorthFuzzing=true.

When returning WorthFuzzing=true, you MUST ALSO:
1. Extract any specific kernel functions that should be heavily fuzzed into FocusSymbols.
   Avoid listing generic hot-path functions to prevent skewed test distributions.
   Prefer non-static, non-inlined API entrypoint functions over internal static helper functions
   (which are inlined by the compiler and do not have distinct symbol addresses).
2. Identify any specific CONFIG_ options required to properly test this new/modified feature.
   Go and look into the Kconfig files and check for ifdefs around the code, do not make assumptions.
   Also check "depends on" lines in Kconfig to include any non-standard parent subsystem configs
   needed for Kbuild to compile the code statically into vmlinux. List them in the EnableConfigs
   output array, and DO NOT add a 'CONFIG_' prefix (e.g., return "NET_IPV4" instead of "CONFIG_NET_IPV4").
Prefer calling several tools at the same time to save round-trips.


Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.

For your convenience, here is the diff of the changes:
commit bd271568204405923f3b37fcb4ded599be77ced1
Author: syz-cluster <triage@syzkaller.com>
Date:   Tue Aug 18 17:01:35 2026 +0000

    syz-cluster: applied patch under review

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index b3afe0688919b..ac9faec1a0184 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -2202,6 +2202,8 @@ config ARM64_POE
 	def_bool y
 	select ARCH_USES_HIGH_VMA_FLAGS
 	select ARCH_HAS_PKEYS
+	select ARCH_HAS_KPKEYS
+	select ARCH_HAS_KPKEYS_HARDENED_PGTABLES
 	help
 	  The Permission Overlay Extension is used to implement Memory
 	  Protection Keys. Memory Protection Keys provides a mechanism for
diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h
index a57870fa96db5..37aae60c2d7f0 100644
--- a/arch/arm64/include/asm/cpufeature.h
+++ b/arch/arm64/include/asm/cpufeature.h
@@ -1085,6 +1085,18 @@ static inline bool cpu_has_lpa2(void)
 #endif
 }
 
+static inline bool cpu_has_poe(void)
+{
+	u64 mmfr3;
+
+	if (!IS_ENABLED(CONFIG_ARM64_POE))
+		return false;
+
+	mmfr3 = read_sysreg_s(SYS_ID_AA64MMFR3_EL1);
+	return cpuid_feature_extract_unsigned_field(mmfr3,
+						    ID_AA64MMFR3_EL1_S1POE_SHIFT);
+}
+
 #endif /* __ASSEMBLER__ */
 
 #endif
diff --git a/arch/arm64/include/asm/kpkeys.h b/arch/arm64/include/asm/kpkeys.h
new file mode 100644
index 0000000000000..7747af54fd386
--- /dev/null
+++ b/arch/arm64/include/asm/kpkeys.h
@@ -0,0 +1,84 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef __ASM_KPKEYS_H
+#define __ASM_KPKEYS_H
+
+#include <linux/kpkeys_types.h>
+
+#include <asm/barrier.h>
+#include <asm/cpufeature.h>
+#include <asm/por.h>
+
+/*
+ * Equivalent to por_set_kpkeys_context(0, KPKEYS_CTX_DEFAULT), but can also be
+ * used in assembly.
+ */
+#define POR_EL1_INIT	(POR_ELx_PERM_PREP(0, POE_RWX) | \
+			 POR_ELx_PERM_PREP(KPKEYS_PKEY_PGTABLES, POE_R))
+
+#ifndef __ASSEMBLY__
+
+static inline bool arch_supports_kpkeys(void)
+{
+	return system_supports_poe();
+}
+
+static inline bool arch_supports_kpkeys_early(void)
+{
+	/* POE is a boot feature */
+	return boot_capabilities_finalized() ?
+		system_supports_poe() : cpu_has_poe();
+}
+
+#ifdef CONFIG_ARM64_POE
+
+static inline u64 por_set_kpkeys_context(u64 por, enum kpkeys_ctx ctx)
+{
+	por = por_elx_set_pkey_perms(por, 0, POE_RWX);
+	por = por_elx_set_pkey_perms(por, KPKEYS_PKEY_PGTABLES,
+				     ctx == KPKEYS_CTX_PGTABLES ? POE_RW : POE_R);
+
+	return por;
+}
+
+static __always_inline void __kpkeys_set_pkey_reg_nosync(u64 pkey_reg)
+{
+	write_sysreg_s(pkey_reg, SYS_POR_EL1);
+}
+
+static __always_inline
+struct kpkeys_state arch_kpkeys_enter_context(enum kpkeys_ctx ctx)
+{
+	const u64 prev_por = read_sysreg_s(SYS_POR_EL1);
+	const u64 new_por = por_set_kpkeys_context(prev_por, ctx);
+
+	if (prev_por == new_por)
+		return (struct kpkeys_state) { .entered_context = false };
+
+	__kpkeys_set_pkey_reg_nosync(new_por);
+	isb();
+
+	return (struct kpkeys_state) {
+		.entered_context = true,
+		.arch_state.por = prev_por,
+	};
+}
+
+static __always_inline
+void arch_kpkeys_leave_context(const struct kpkeys_state *state)
+{
+	__kpkeys_set_pkey_reg_nosync(state->arch_state.por);
+	isb();
+}
+
+#endif /* CONFIG_ARM64_POE */
+
+#ifdef CONFIG_KPKEYS_HARDENED_PGTABLES
+
+#define arch_kpkeys_protect_static_pgtables arch_kpkeys_protect_static_pgtables
+void arch_kpkeys_protect_static_pgtables(void);
+
+#endif /* CONFIG_KPKEYS_HARDENED_PGTABLES */
+
+#endif	/* __ASSEMBLY__ */
+
+#endif	/* __ASM_KPKEYS_H */
diff --git a/arch/arm64/include/asm/kpkeys_types.h b/arch/arm64/include/asm/kpkeys_types.h
new file mode 100644
index 0000000000000..e9c59a26ee7a9
--- /dev/null
+++ b/arch/arm64/include/asm/kpkeys_types.h
@@ -0,0 +1,15 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef __ASM_KPKEYS_TYPES_H
+#define __ASM_KPKEYS_TYPES_H
+
+#include <linux/types.h>
+
+#ifndef __ASSEMBLY__
+
+struct arch_kpkeys_state {
+	u64 por;
+};
+
+#endif	/* __ASSEMBLY__ */
+
+#endif	/* __ASM_KPKEYS_TYPES_H */
diff --git a/arch/arm64/include/asm/pgtable-prot.h b/arch/arm64/include/asm/pgtable-prot.h
index 212ce1b02e15e..d4d45ab86a5a8 100644
--- a/arch/arm64/include/asm/pgtable-prot.h
+++ b/arch/arm64/include/asm/pgtable-prot.h
@@ -183,9 +183,9 @@ static inline bool __pure lpa2_is_enabled(void)
 	PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_SHARED_EXEC),   PIE_RW)     | \
 	PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_READONLY),      PIE_R)      | \
 	PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_SHARED),        PIE_RW)     | \
-	PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL_ROX),    PIE_RX)     | \
-	PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL_EXEC),   PIE_RWX)    | \
-	PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL_RO),     PIE_R)      | \
-	PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL),        PIE_RW))
+	PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL_ROX),    PIE_RX_O)   | \
+	PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL_EXEC),   PIE_RWX_O)  | \
+	PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL_RO),     PIE_R_O)    | \
+	PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL),        PIE_RW_O))
 
 #endif /* __ASM_PGTABLE_PROT_H */
diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index 27689c62bd25d..6e5d14c7e8b8f 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -39,6 +39,48 @@
 #include <linux/mm_types.h>
 #include <linux/sched.h>
 #include <linux/page_table_check.h>
+#include <linux/kpkeys.h>
+
+#ifdef CONFIG_KPKEYS_HARDENED_PGTABLES
+KPKEYS_GUARD_COND(kpkeys_hardened_pgtables, KPKEYS_CTX_PGTABLES,
+		  kpkeys_hardened_pgtables_enabled() &&
+		  !is_lazy_mmu_mode_active())
+
+static inline void kpkeys_lazy_mmu_enter(void)
+{
+	if (!kpkeys_hardened_pgtables_enabled())
+		return;
+
+	current->thread.kpkeys_state_lazy_mmu =
+		kpkeys_enter_context(KPKEYS_CTX_PGTABLES);
+}
+
+static inline void kpkeys_lazy_mmu_exit(void)
+{
+	const struct kpkeys_state *state =
+		&current->thread.kpkeys_state_lazy_mmu;
+
+	if (!kpkeys_hardened_pgtables_enabled())
+		return;
+
+	/*
+	 * We skip any barrier if TIF_LAZY_MMU_PENDING is set:
+	 * emit_pte_barriers() will issue an ISB just after this function
+	 * returns.
+	 */
+	if (test_thread_flag(TIF_LAZY_MMU_PENDING) && state->entered_context)
+		__kpkeys_set_pkey_reg_nosync(state->arch_state.por);
+	else
+		kpkeys_leave_context(state);
+}
+#else /* CONFIG_KPKEYS_HARDENED_PGTABLES */
+KPKEYS_GUARD_NOOP(kpkeys_hardened_pgtables)
+
+static inline void kpkeys_lazy_mmu_enter(void) {}
+static inline void kpkeys_lazy_mmu_exit(void) {}
+#endif /* CONFIG_KPKEYS_HARDENED_PGTABLES */
+
+
 
 static inline void emit_pte_barriers(void)
 {
@@ -71,7 +113,10 @@ static inline void queue_pte_barriers(void)
 	}
 }
 
-static inline void arch_enter_lazy_mmu_mode(void) {}
+static inline void arch_enter_lazy_mmu_mode(void)
+{
+	kpkeys_lazy_mmu_enter();
+}
 
 static inline void arch_flush_lazy_mmu_mode(void)
 {
@@ -81,6 +126,11 @@ static inline void arch_flush_lazy_mmu_mode(void)
 
 static inline void arch_leave_lazy_mmu_mode(void)
 {
+	/*
+	 * The ordering should be preserved to allow kpkeys_lazy_mmu_exit()
+	 * to skip any barrier when TIF_LAZY_MMU_PENDING is set.
+	 */
+	kpkeys_lazy_mmu_exit();
 	arch_flush_lazy_mmu_mode();
 }
 
@@ -359,6 +409,7 @@ static inline pte_t pte_clear_uffd_wp(pte_t pte)
 
 static inline void __set_pte_nosync(pte_t *ptep, pte_t pte)
 {
+	guard(kpkeys_hardened_pgtables)();
 	WRITE_ONCE(*ptep, pte);
 }
 
@@ -830,6 +881,7 @@ static inline void set_pmd(pmd_t *pmdp, pmd_t pmd)
 	}
 #endif /* __PAGETABLE_PMD_FOLDED */
 
+	guard(kpkeys_hardened_pgtables)();
 	WRITE_ONCE(*pmdp, pmd);
 
 	if (pmd_valid(pmd))
@@ -894,6 +946,7 @@ static inline void set_pud(pud_t *pudp, pud_t pud)
 		return;
 	}
 
+	guard(kpkeys_hardened_pgtables)();
 	WRITE_ONCE(*pudp, pud);
 
 	if (pud_valid(pud))
@@ -975,6 +1028,7 @@ static inline void set_p4d(p4d_t *p4dp, p4d_t p4d)
 		return;
 	}
 
+	guard(kpkeys_hardened_pgtables)();
 	WRITE_ONCE(*p4dp, p4d);
 	queue_pte_barriers();
 }
@@ -1103,6 +1157,7 @@ static inline void set_pgd(pgd_t *pgdp, pgd_t pgd)
 		return;
 	}
 
+	guard(kpkeys_hardened_pgtables)();
 	WRITE_ONCE(*pgdp, pgd);
 	queue_pte_barriers();
 }
@@ -1307,6 +1362,7 @@ static inline bool __ptep_test_and_clear_young(struct vm_area_struct *vma,
 {
 	pte_t old_pte, pte;
 
+	guard(kpkeys_hardened_pgtables)();
 	pte = __ptep_get(ptep);
 	do {
 		old_pte = pte;
@@ -1354,7 +1410,10 @@ static inline pte_t __ptep_get_and_clear_anysz(struct mm_struct *mm,
 					       pte_t *ptep,
 					       unsigned long pgsize)
 {
-	pte_t pte = __pte(xchg_relaxed(&pte_val(*ptep), 0));
+	pte_t pte;
+
+	scoped_guard(kpkeys_hardened_pgtables)
+		pte = __pte(xchg_relaxed(&pte_val(*ptep), 0));
 
 	switch (pgsize) {
 	case PAGE_SIZE:
@@ -1427,6 +1486,7 @@ static inline void ___ptep_set_wrprotect(struct mm_struct *mm,
 {
 	pte_t old_pte;
 
+	guard(kpkeys_hardened_pgtables)();
 	do {
 		old_pte = pte;
 		pte = pte_wrprotect(pte);
@@ -1460,6 +1520,7 @@ static inline void __clear_young_dirty_pte(struct vm_area_struct *vma,
 {
 	pte_t old_pte;
 
+	guard(kpkeys_hardened_pgtables)();
 	do {
 		old_pte = pte;
 
@@ -1507,6 +1568,7 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma,
 		unsigned long address, pmd_t *pmdp, pmd_t pmd)
 {
 	page_table_check_pmd_set(vma->vm_mm, address, pmdp, pmd);
+	guard(kpkeys_hardened_pgtables)();
 	return __pmd(xchg_relaxed(&pmd_val(*pmdp), pmd_val(pmd)));
 }
 #endif
diff --git a/arch/arm64/include/asm/por.h b/arch/arm64/include/asm/por.h
index d913d5b529e49..dddabf6ffae6d 100644
--- a/arch/arm64/include/asm/por.h
+++ b/arch/arm64/include/asm/por.h
@@ -10,6 +10,8 @@
 
 #define POR_EL0_INIT	POR_ELx_PERM_PREP(0, POE_RWX)
 
+#ifndef __ASSEMBLY__
+
 static inline bool por_elx_allows_read(u64 por, u8 pkey)
 {
 	u8 perm = POR_ELx_PERM_GET(pkey, por);
@@ -31,4 +33,13 @@ static inline bool por_elx_allows_exec(u64 por, u8 pkey)
 	return perm & POE_X;
 }
 
+static inline u64 por_elx_set_pkey_perms(u64 por, u8 pkey, u8 perms)
+{
+	u64 shift = POR_ELx_PERM_SHIFT(pkey);
+
+	return (por & ~(POE_MASK << shift)) | ((u64)perms << shift);
+}
+
+#endif	/* __ASSEMBLY__ */
+
 #endif /* _ASM_ARM64_POR_H */
diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h
index c2a627f393144..23660a233da28 100644
--- a/arch/arm64/include/asm/processor.h
+++ b/arch/arm64/include/asm/processor.h
@@ -30,6 +30,7 @@
 #include <linux/build_bug.h>
 #include <linux/cache.h>
 #include <linux/init.h>
+#include <linux/kpkeys_types.h>
 #include <linux/stddef.h>
 #include <linux/string.h>
 #include <linux/thread_info.h>
@@ -195,6 +196,10 @@ struct thread_struct {
 	u64			svcr;
 	u64			tpidr2_el0;
 	u64			por_el0;
+	u64			por_el1;
+#ifdef CONFIG_KPKEYS_HARDENED_PGTABLES
+	struct kpkeys_state	kpkeys_state_lazy_mmu;
+#endif
 #ifdef CONFIG_ARM64_GCS
 	unsigned int		gcs_el0_mode;
 	unsigned int		gcs_el0_locked;
diff --git a/arch/arm64/include/asm/sections.h b/arch/arm64/include/asm/sections.h
index 51b0d594239eb..dc961cf67168c 100644
--- a/arch/arm64/include/asm/sections.h
+++ b/arch/arm64/include/asm/sections.h
@@ -23,6 +23,7 @@ extern char __irqentry_text_start[], __irqentry_text_end[];
 extern char __mmuoff_data_start[], __mmuoff_data_end[];
 extern char __entry_tramp_text_start[], __entry_tramp_text_end[];
 extern char __relocate_new_kernel_start[], __relocate_new_kernel_end[];
+extern char __pi_init_pg_dir[], __pi_init_pg_end[];
 
 static inline size_t entry_tramp_text_size(void)
 {
diff --git a/arch/arm64/include/asm/set_memory.h b/arch/arm64/include/asm/set_memory.h
index 90f61b17275e1..b6cd6de34abf1 100644
--- a/arch/arm64/include/asm/set_memory.h
+++ b/arch/arm64/include/asm/set_memory.h
@@ -19,4 +19,8 @@ bool kernel_page_present(struct page *page);
 int set_memory_encrypted(unsigned long addr, int numpages);
 int set_memory_decrypted(unsigned long addr, int numpages);
 
+#ifdef CONFIG_ARCH_HAS_KPKEYS
+int set_memory_pkey(unsigned long addr, int numpages, int pkey);
+#endif
+
 #endif /* _ASM_ARM64_SET_MEMORY_H */
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index 9a22df0c5120f..4f2f54007ed6f 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -76,6 +76,7 @@
 #include <linux/kasan.h>
 #include <linux/percpu.h>
 #include <linux/sched/isolation.h>
+#include <linux/kpkeys.h>
 
 #include <asm/arm_pmuv3.h>
 #include <asm/cpu.h>
@@ -2463,8 +2464,10 @@ static void cpu_enable_mops(const struct arm64_cpu_capabilities *__unused)
 #ifdef CONFIG_ARM64_POE
 static void cpu_enable_poe(const struct arm64_cpu_capabilities *__unused)
 {
-	sysreg_clear_set(REG_TCR2_EL1, 0, TCR2_EL1_E0POE);
+	write_sysreg_s(POR_EL1_INIT, SYS_POR_EL1);
+	sysreg_clear_set(REG_TCR2_EL1, 0, TCR2_EL1_E0POE | TCR2_EL1_POE);
 	sysreg_clear_set(CPACR_EL1, 0, CPACR_EL1_E0POE);
+	isb();
 }
 #endif
 
diff --git a/arch/arm64/kernel/pi/map_kernel.c b/arch/arm64/kernel/pi/map_kernel.c
index fb44cbdd2f292..81ca3e98f0f7b 100644
--- a/arch/arm64/kernel/pi/map_kernel.c
+++ b/arch/arm64/kernel/pi/map_kernel.c
@@ -92,8 +92,13 @@ static void __init map_kernel(u64 kaslr_offset, u64 va_offset, int root_level)
 		    __inittext_end, prot, false, root_level);
 	map_segment(init_pg_dir, &pgdp, va_offset, __initdata_begin,
 		    __initdata_end, data_prot, false, root_level);
-	map_segment(init_pg_dir, &pgdp, va_offset, _data, _end, data_prot,
-		    true, root_level);
+	map_segment(init_pg_dir, &pgdp, va_offset, _data, init_pg_dir,
+		    data_prot, true, root_level);
+	/* Map init_pg_dir separately as we will remap it read-only later on */
+	map_segment(init_pg_dir, &pgdp, va_offset, init_pg_dir, init_pg_end,
+		    data_prot, true, root_level);
+	map_segment(init_pg_dir, &pgdp, va_offset, init_pg_end, _end,
+		    data_prot, true, root_level);
 	dsb(ishst);
 
 	idmap_cpu_replace_ttbr1((phys_addr_t)init_pg_dir);
diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c
index 581f80e9b9b73..1dec143decf15 100644
--- a/arch/arm64/kernel/process.c
+++ b/arch/arm64/kernel/process.c
@@ -51,6 +51,7 @@
 #include <asm/exec.h>
 #include <asm/fpsimd.h>
 #include <asm/gcs.h>
+#include <asm/kpkeys.h>
 #include <asm/mmu_context.h>
 #include <asm/mpam.h>
 #include <asm/mte.h>
@@ -431,6 +432,9 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args)
 
 	ptrauth_thread_init_kernel(p);
 
+	if (system_supports_poe())
+		p->thread.por_el1 = POR_EL1_INIT;
+
 	if (likely(!args->fn)) {
 		*childregs = *current_pt_regs();
 		childregs->regs[0] = 0;
@@ -681,6 +685,12 @@ static void permission_overlay_switch(struct task_struct *next)
 		 * of POR_EL0.
 		 */
 	}
+
+	current->thread.por_el1 = read_sysreg_s(SYS_POR_EL1);
+	if (current->thread.por_el1 != next->thread.por_el1) {
+		write_sysreg_s(next->thread.por_el1, SYS_POR_EL1);
+		isb();
+	}
 }
 
 /*
diff --git a/arch/arm64/kernel/sleep.S b/arch/arm64/kernel/sleep.S
index f093cdf71be11..e0a6ad85cd242 100644
--- a/arch/arm64/kernel/sleep.S
+++ b/arch/arm64/kernel/sleep.S
@@ -3,6 +3,7 @@
 #include <linux/linkage.h>
 #include <asm/asm-offsets.h>
 #include <asm/assembler.h>
+#include <asm/kpkeys.h>
 #include <asm/smp.h>
 
 	.text
@@ -134,6 +135,17 @@ SYM_FUNC_START(_cpu_resume)
 	/* load sp from context */
 	ldr	x2, [x0, #CPU_CTX_SP]
 	mov	sp, x2
+
+#ifdef CONFIG_ARM64_POE
+alternative_if_not ARM64_HAS_S1POE
+	b	.Lskip_por_set
+alternative_else_nop_endif
+	mov_q	x2, POR_EL1_INIT
+	msr_s	SYS_POR_EL1, x2
+	/* isb can be skipped since cpu_do_resume() will do it. */
+.Lskip_por_set:
+#endif /* CONFIG_ARM64_POE */
+
 	/*
 	 * cpu_do_resume expects x0 to contain context address pointer
 	 */
diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
index 0b52557652be6..a853a8d4e61af 100644
--- a/arch/arm64/mm/fault.c
+++ b/arch/arm64/mm/fault.c
@@ -229,13 +229,16 @@ int __ptep_set_access_flags_anysz(struct vm_area_struct *vma,
 	 */
 	pte_val(entry) ^= PTE_RDONLY;
 	pteval = pte_val(pte);
-	do {
-		old_pteval = pteval;
-		pteval ^= PTE_RDONLY;
-		pteval |= pte_val(entry);
-		pteval ^= PTE_RDONLY;
-		pteval = cmpxchg_relaxed(&pte_val(*ptep), old_pteval, pteval);
-	} while (pteval != old_pteval);
+	scoped_guard(kpkeys_hardened_pgtables) {
+		do {
+			old_pteval = pteval;
+			pteval ^= PTE_RDONLY;
+			pteval |= pte_val(entry);
+			pteval ^= PTE_RDONLY;
+			pteval = cmpxchg_relaxed(&pte_val(*ptep), old_pteval,
+						 pteval);
+		} while (pteval != old_pteval);
+	}
 
 	/*
 	 * Invalidate the local stale read-only entry.  Remote stale entries
diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
index 97987f850a33c..7985045875cf6 100644
--- a/arch/arm64/mm/init.c
+++ b/arch/arm64/mm/init.c
@@ -386,6 +386,7 @@ bool page_alloc_available __ro_after_init;
 void __init mem_init(void)
 {
 	page_alloc_available = true;
+	kpkeys_hardened_pgtables_init();
 	swiotlb_update_mem_attributes();
 }
 
diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index d4de88770ecfe..472ad0f8f002c 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -107,8 +107,11 @@ static phys_addr_t __init early_pgtable_alloc(enum pgtable_level pgtable_level)
 {
 	phys_addr_t phys;
 
-	phys = memblock_phys_alloc_range(PAGE_SIZE, PAGE_SIZE, 0,
-					 MEMBLOCK_ALLOC_NOLEAKTRACE);
+	if (kpkeys_hardened_pgtables_early_enabled())
+		phys = kpkeys_physmem_pgtable_alloc();
+	else
+		phys = memblock_phys_alloc_range(PAGE_SIZE, PAGE_SIZE, 0,
+						 MEMBLOCK_ALLOC_NOLEAKTRACE);
 	if (!phys)
 		panic("Failed to allocate page table page\n");
 
@@ -780,7 +783,7 @@ static inline bool force_pte_mapping(void)
 	const bool bbml2 = system_capabilities_finalized() ?
 		system_supports_bbml2_noabort() : cpu_supports_bbml2_noabort();
 
-	if (debug_pagealloc_enabled())
+	if (debug_pagealloc_enabled() || kpkeys_hardened_pgtables_early_enabled())
 		return true;
 	if (bbml2)
 		return false;
@@ -1070,6 +1073,18 @@ void __init mark_linear_text_alias_ro(void)
 			    PAGE_KERNEL_RO);
 }
 
+#ifdef CONFIG_KPKEYS_HARDENED_PGTABLES
+void __init arch_kpkeys_protect_static_pgtables(void)
+{
+	unsigned long addr = (unsigned long)lm_alias(__pi_init_pg_dir);
+	unsigned long size = __pi_init_pg_end - __pi_init_pg_dir;
+	int ret;
+
+	ret = set_memory_pkey(addr, size / PAGE_SIZE, KPKEYS_PKEY_PGTABLES);
+	WARN_ON(ret);
+}
+#endif /* CONFIG_KPKEYS_HARDENED_PGTABLES */
+
 #ifdef CONFIG_KFENCE
 
 bool __ro_after_init kfence_early_init = !!CONFIG_KFENCE_SAMPLE_INTERVAL;
@@ -1214,6 +1229,14 @@ void mark_rodata_ro(void)
 	update_mapping_prot(__pa_symbol(_text), (unsigned long)_text,
 			    (unsigned long)_stext - (unsigned long)_text,
 			    PAGE_KERNEL_RO);
+	/*
+	 * Map the kernel image mapping of init_pg_dir read-only; it should
+	 * only be written via the linear map.
+	 */
+	section_size = (unsigned long)__pi_init_pg_end - (unsigned long)__pi_init_pg_dir;
+	update_mapping_prot(__pa_symbol(__pi_init_pg_dir),
+			    (unsigned long)__pi_init_pg_dir,
+			    section_size, PAGE_KERNEL_RO);
 
 	/* Map the kernel data/bss read-only in the linear map */
 	update_mapping_prot(__pa_symbol(__init_end),
@@ -1446,7 +1469,7 @@ static void free_hotplug_page_range(struct page *page, size_t size,
 static void free_hotplug_pgtable_page(struct page *page)
 {
 	pagetable_dtor(page_ptdesc(page));
-	free_hotplug_page_range(page, PAGE_SIZE, NULL);
+	pagetable_free(page_ptdesc(page));
 }
 
 static bool pgtable_range_aligned(unsigned long start, unsigned long end,
@@ -2305,8 +2328,8 @@ void __cpu_replace_ttbr1(pgd_t *pgdp, bool cnp)
 #ifdef CONFIG_ARCH_HAS_PKEYS
 int arch_set_user_pkey_access(int pkey, unsigned long init_val)
 {
-	u64 new_por;
-	u64 old_por;
+	u8 new_perms;
+	u64 por;
 
 	if (!system_supports_poe())
 		return -ENOSPC;
@@ -2320,25 +2343,19 @@ int arch_set_user_pkey_access(int pkey, unsigned long init_val)
 		return -EINVAL;
 
 	/* Set the bits we need in POR:  */
-	new_por = POE_RWX;
+	new_perms = POE_RWX;
 	if (init_val & PKEY_DISABLE_WRITE)
-		new_por &= ~POE_W;
+		new_perms &= ~POE_W;
 	if (init_val & PKEY_DISABLE_ACCESS)
-		new_por &= ~POE_RW;
+		new_perms &= ~POE_RW;
 	if (init_val & PKEY_DISABLE_READ)
-		new_por &= ~POE_R;
+		new_perms &= ~POE_R;
 	if (init_val & PKEY_DISABLE_EXECUTE)
-		new_por &= ~POE_X;
-
-	/* Shift the bits in to the correct place in POR for pkey: */
-	new_por = POR_ELx_PERM_PREP(pkey, new_por);
-
-	/* Get old POR and mask off any old bits in place: */
-	old_por = read_sysreg_s(SYS_POR_EL0);
-	old_por &= ~(POE_MASK << POR_ELx_PERM_SHIFT(pkey));
+		new_perms &= ~POE_X;
 
-	/* Write old part along with new part: */
-	write_sysreg_s(old_por | new_por, SYS_POR_EL0);
+	por = read_sysreg_s(SYS_POR_EL0);
+	por = por_elx_set_pkey_perms(por, pkey, new_perms);
+	write_sysreg_s(por, SYS_POR_EL0);
 
 	return 0;
 }
diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c
index bbe98ac9ad8c6..46fbebfc51003 100644
--- a/arch/arm64/mm/pageattr.c
+++ b/arch/arm64/mm/pageattr.c
@@ -9,6 +9,8 @@
 #include <linux/sched.h>
 #include <linux/vmalloc.h>
 #include <linux/pagewalk.h>
+#include <linux/pkeys.h>
+#include <linux/kpkeys.h>
 
 #include <asm/cacheflush.h>
 #include <asm/pgtable-prot.h>
@@ -99,7 +101,8 @@ bool can_set_direct_map(void)
 	 * Realms need to make pages shared/protected at page granularity.
 	 */
 	return rodata_full || debug_pagealloc_enabled() ||
-		arm64_kfence_can_set_direct_map() || is_realm_world();
+		arm64_kfence_can_set_direct_map() || is_realm_world() ||
+		kpkeys_hardened_pgtables_early_enabled();
 }
 
 static int update_range_prot(unsigned long start, unsigned long size,
@@ -365,6 +368,30 @@ int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid)
 	return set_memory_valid(addr, nr, valid);
 }
 
+#ifdef CONFIG_ARCH_HAS_KPKEYS
+int set_memory_pkey(unsigned long addr, int numpages, int pkey)
+{
+	unsigned long set_prot = 0;
+
+	if (!kpkeys_supported())
+		return 0;
+
+	if (!__is_lm_address(kasan_reset_tag((void *)addr)))
+		return -EINVAL;
+
+	if (pkey < 0 || pkey >= arch_max_pkey())
+		return -EINVAL;
+
+	set_prot |= pkey & BIT(0) ? PTE_PO_IDX_0 : 0;
+	set_prot |= pkey & BIT(1) ? PTE_PO_IDX_1 : 0;
+	set_prot |= pkey & BIT(2) ? PTE_PO_IDX_2 : 0;
+
+	return __change_memory_common(addr, PAGE_SIZE * numpages,
+				      __pgprot(set_prot),
+				      __pgprot(PTE_PO_IDX_MASK));
+}
+#endif
+
 #ifdef CONFIG_DEBUG_PAGEALLOC
 /*
  * This is - apart from the return value - doing the same
diff --git a/include/linux/kpkeys.h b/include/linux/kpkeys.h
new file mode 100644
index 0000000000000..df55ec854586e
--- /dev/null
+++ b/include/linux/kpkeys.h
@@ -0,0 +1,162 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef _LINUX_KPKEYS_H
+#define _LINUX_KPKEYS_H
+
+#include <linux/bug.h>
+#include <linux/cleanup.h>
+#include <linux/jump_label.h>
+#include <linux/kpkeys_types.h>
+
+/**
+ * KPKEYS_GUARD_NOOP() - define a guard type that does nothing
+ * @name: the name of the guard type
+ *
+ * Define a guard type that does nothing, useful to match a real guard type
+ * that is defined under an #ifdef.
+ */
+#define KPKEYS_GUARD_NOOP(name)						\
+	__DEFINE_CLASS_IS_CONDITIONAL(name, false);			\
+	DEFINE_CLASS(name, bool, (void)_T, false, void);		\
+	static inline void *class_##name##_lock_ptr(bool *_T)		\
+	{ return _T; }
+
+#ifdef CONFIG_ARCH_HAS_KPKEYS
+
+#include <asm/kpkeys.h>
+
+/**
+ * KPKEYS_GUARD_COND() - define a guard type that conditionally switches to
+ *                       a given kpkeys context
+ * @name: the name of the guard type
+ * @ctx: the kpkeys context to switch to
+ * @cond: an expression that is evaluated as condition
+ *
+ * Define a guard type that switches to @ctx if @cond evaluates to true,
+ * and does nothing otherwise.
+ */
+#define KPKEYS_GUARD_COND(name, ctx, cond)				\
+	__DEFINE_CLASS_IS_CONDITIONAL(name, false);			\
+	DEFINE_CLASS(name, struct kpkeys_state,				\
+		     kpkeys_leave_context(&_T),				\
+		     (cond) ? kpkeys_enter_context(ctx) :		\
+			      (struct kpkeys_state) {}, void);		\
+	static inline							\
+	void *class_##name##_lock_ptr(struct kpkeys_state *_T)		\
+	{ return _T; }
+
+/**
+ * KPKEYS_GUARD() - define a guard type that switches to a given kpkeys context
+ *                  if kpkeys are supported
+ * @name: the name of the guard type
+ * @ctx: the kpkeys context to switch to
+ *
+ * Define a guard type that switches to @ctx if the system supports kpkeys.
+ */
+#define KPKEYS_GUARD(name, ctx)						\
+	KPKEYS_GUARD_COND(name, ctx, kpkeys_supported())
+
+/**
+ * kpkeys_enter_context() - enter a kpkeys context
+ * @ctx: the context to switch to
+ *
+ * Enters the specified kpkeys context. @ctx must be a compile-time constant.
+ *
+ * Return: state to be passed to kpkeys_leave_context().
+ */
+static __always_inline
+struct kpkeys_state kpkeys_enter_context(enum kpkeys_ctx ctx)
+{
+	BUILD_BUG_ON_MSG(!__builtin_constant_p(ctx),
+			 "kpkeys_enter_context() only takes constant values");
+	BUILD_BUG_ON_MSG(ctx < 0 || ctx >= KPKEYS_CTX_COUNT,
+			 "Invalid value passed to kpkeys_enter_context()");
+
+	return arch_kpkeys_enter_context(ctx);
+}
+
+/**
+ * kpkeys_leave_context() - leave a kpkeys context
+ * @state: state returned by kpkeys_enter_context()
+ *
+ * Restores the state saved when entering a kpkeys context. If no context was
+ * entered, this function does nothing.
+ */
+static __always_inline
+void kpkeys_leave_context(const struct kpkeys_state *state)
+{
+	if (state->entered_context)
+		arch_kpkeys_leave_context(state);
+}
+
+static inline bool kpkeys_supported(void)
+{
+	return arch_supports_kpkeys();
+}
+
+#else /* CONFIG_ARCH_HAS_KPKEYS */
+
+static inline bool kpkeys_supported(void)
+{
+	return false;
+}
+
+#endif /* CONFIG_ARCH_HAS_KPKEYS */
+
+#ifdef CONFIG_KPKEYS_HARDENED_PGTABLES
+
+DECLARE_STATIC_KEY_FALSE(kpkeys_hardened_pgtables_key);
+
+static inline bool kpkeys_hardened_pgtables_enabled(void)
+{
+	return static_branch_unlikely(&kpkeys_hardened_pgtables_key);
+}
+
+static inline bool kpkeys_hardened_pgtables_early_enabled(void)
+{
+	return arch_supports_kpkeys_early();
+}
+
+struct page *kpkeys_pgtable_alloc(gfp_t gfp, unsigned int order);
+void kpkeys_pgtable_free(struct page *page, unsigned int order);
+
+/*
+ * Should be called from mem_init(): as soon as the buddy allocator becomes
+ * available and before any call to pagetable_alloc().
+ */
+void kpkeys_hardened_pgtables_init(void);
+
+phys_addr_t kpkeys_physmem_pgtable_alloc(void);
+
+#ifndef arch_kpkeys_protect_static_pgtables
+static inline void arch_kpkeys_protect_static_pgtables(void) {}
+#endif
+
+#else /* CONFIG_KPKEYS_HARDENED_PGTABLES */
+
+static inline bool kpkeys_hardened_pgtables_enabled(void)
+{
+	return false;
+}
+
+static inline bool kpkeys_hardened_pgtables_early_enabled(void)
+{
+	return false;
+}
+
+static inline struct page *kpkeys_pgtable_alloc(gfp_t gfp, unsigned int order)
+{
+	return NULL;
+}
+
+static inline void kpkeys_pgtable_free(struct page *page, unsigned int order) {}
+
+static inline void kpkeys_hardened_pgtables_init(void) {}
+
+static inline phys_addr_t kpkeys_physmem_pgtable_alloc(void)
+{
+	return 0;
+}
+
+#endif /* CONFIG_KPKEYS_HARDENED_PGTABLES */
+
+#endif /* _LINUX_KPKEYS_H */
diff --git a/include/linux/kpkeys_types.h b/include/linux/kpkeys_types.h
new file mode 100644
index 0000000000000..09f15a9565cf3
--- /dev/null
+++ b/include/linux/kpkeys_types.h
@@ -0,0 +1,30 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef _LINUX_KPKEYS_TYPES_H
+#define _LINUX_KPKEYS_TYPES_H
+
+#include <linux/types.h>
+
+#ifdef CONFIG_ARCH_HAS_KPKEYS
+#include <asm/kpkeys_types.h>
+#endif
+
+#ifndef KPKEYS_PKEY_PGTABLES
+#define KPKEYS_PKEY_PGTABLES	1
+#endif
+
+#if defined(CONFIG_ARCH_HAS_KPKEYS) && !defined(__ASSEMBLY__)
+
+enum kpkeys_ctx {
+	KPKEYS_CTX_DEFAULT = 0,
+	KPKEYS_CTX_PGTABLES,
+	KPKEYS_CTX_COUNT,
+};
+
+struct kpkeys_state {
+	bool entered_context;
+	struct arch_kpkeys_state arch_state;
+};
+
+#endif /* CONFIG_ARCH_HAS_KPKEYS && !__ASSEMBLY__ */
+
+#endif /* _LINUX_KPKEYS_TYPES_H */
diff --git a/include/linux/memblock.h b/include/linux/memblock.h
index 5afcd99aa8c14..c61f80e31cc10 100644
--- a/include/linux/memblock.h
+++ b/include/linux/memblock.h
@@ -24,6 +24,17 @@ extern unsigned long max_pfn;
  */
 extern unsigned long long max_possible_pfn;
 
+#define INIT_MEMBLOCK_REGIONS			128
+#define INIT_PHYSMEM_REGIONS			4
+
+#ifndef INIT_MEMBLOCK_RESERVED_REGIONS
+#define INIT_MEMBLOCK_RESERVED_REGIONS		INIT_MEMBLOCK_REGIONS
+#endif
+
+#ifndef INIT_MEMBLOCK_MEMORY_REGIONS
+#define INIT_MEMBLOCK_MEMORY_REGIONS		INIT_MEMBLOCK_REGIONS
+#endif
+
 /**
  * enum memblock_flags - definition of memory region attributes
  * @MEMBLOCK_NONE: no special request
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 485df9c2dbddb..eea3131786b35 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -37,6 +37,7 @@
 #include <linux/bitmap.h>
 #include <linux/bitops.h>
 #include <linux/iommu-debug-pagealloc.h>
+#include <linux/kpkeys.h>
 
 struct mempolicy;
 struct anon_vma;
@@ -3682,7 +3683,12 @@ static inline bool ptdesc_test_kernel(const struct ptdesc *ptdesc)
  */
 static inline struct ptdesc *pagetable_alloc_noprof(gfp_t gfp, unsigned int order)
 {
-	struct page *page = alloc_pages_noprof(gfp | __GFP_COMP, order);
+	struct page *page;
+
+	if (kpkeys_hardened_pgtables_enabled())
+		page = kpkeys_pgtable_alloc(gfp | __GFP_COMP, order);
+	else
+		page = alloc_pages_noprof(gfp | __GFP_COMP, order);
 
 	return page_ptdesc(page);
 }
@@ -3691,8 +3697,12 @@ static inline struct ptdesc *pagetable_alloc_noprof(gfp_t gfp, unsigned int orde
 static inline void __pagetable_free(struct ptdesc *pt)
 {
 	struct page *page = ptdesc_page(pt);
+	unsigned int order = compound_order(page);
 
-	__free_pages(page, compound_order(page));
+	if (kpkeys_hardened_pgtables_enabled())
+		kpkeys_pgtable_free(page, order);
+	else
+		__free_pages(page, order);
 }
 
 #ifdef CONFIG_ASYNC_KERNEL_PGTABLE_FREE
diff --git a/include/linux/set_memory.h b/include/linux/set_memory.h
index 3030d9245f5ac..7b3a8bfde3c64 100644
--- a/include/linux/set_memory.h
+++ b/include/linux/set_memory.h
@@ -84,4 +84,11 @@ static inline int set_memory_decrypted(unsigned long addr, int numpages)
 }
 #endif /* CONFIG_ARCH_HAS_MEM_ENCRYPT */
 
+#ifndef CONFIG_ARCH_HAS_KPKEYS
+static inline int set_memory_pkey(unsigned long addr, int numpages, int pkey)
+{
+	return 0;
+}
+#endif
+
 #endif /* _LINUX_SET_MEMORY_H_ */
diff --git a/mm/Kconfig b/mm/Kconfig
index 9e0ca48249054..21bd3e550de83 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -1281,6 +1281,11 @@ config ARCH_USES_HIGH_VMA_FLAGS
 	bool
 config ARCH_HAS_PKEYS
 	bool
+config ARCH_HAS_KPKEYS
+	bool
+config ARCH_HAS_KPKEYS_HARDENED_PGTABLES
+	bool
+	depends on ARCH_HAS_KPKEYS
 
 config ARCH_USES_PG_ARCH_2
 	bool
diff --git a/mm/Makefile b/mm/Makefile
index eff9f9e7e061c..e31e582948451 100644
--- a/mm/Makefile
+++ b/mm/Makefile
@@ -147,3 +147,5 @@ obj-$(CONFIG_SHRINKER_DEBUG) += shrinker_debug.o
 obj-$(CONFIG_EXECMEM) += execmem.o
 obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o
 obj-$(CONFIG_LAZY_MMU_MODE_KUNIT_TEST) += tests/lazy_mmu_mode_kunit.o
+obj-$(CONFIG_KPKEYS_HARDENED_PGTABLES) += kpkeys_hardened_pgtables.o
+obj-$(CONFIG_KPKEYS_HARDENED_PGTABLES_KUNIT_TEST) += tests/kpkeys_hardened_pgtables_kunit.o
diff --git a/mm/kpkeys_hardened_pgtables.c b/mm/kpkeys_hardened_pgtables.c
new file mode 100644
index 0000000000000..c133e5896fccc
--- /dev/null
+++ b/mm/kpkeys_hardened_pgtables.c
@@ -0,0 +1,179 @@
+// SPDX-License-Identifier: GPL-2.0-only
+#include <linux/kpkeys.h>
+#include <linux/memblock.h>
+#include <linux/mm.h>
+#include <linux/set_memory.h>
+
+#include <kunit/visibility.h>
+
+__ro_after_init DEFINE_STATIC_KEY_FALSE(kpkeys_hardened_pgtables_key);
+EXPORT_SYMBOL_IF_KUNIT(kpkeys_hardened_pgtables_key);
+
+static int set_pkey_pgtable(struct page *page, unsigned int nr_pages)
+{
+	unsigned long addr = (unsigned long)page_address(page);
+	int ret;
+
+	ret = set_memory_pkey(addr, nr_pages, KPKEYS_PKEY_PGTABLES);
+
+	WARN_ON(ret);
+	return ret;
+}
+
+static int set_pkey_default(struct page *page, unsigned int nr_pages)
+{
+	unsigned long addr = (unsigned long)page_address(page);
+	int ret;
+
+	ret = set_memory_pkey(addr, nr_pages, 0);
+
+	WARN_ON(ret);
+	return ret;
+}
+
+/* pkeys physmem allocator (PPA) - implemented below */
+static void ppa_finalize(void);
+
+struct page *kpkeys_pgtable_alloc(gfp_t gfp, unsigned int order)
+{
+	struct page *page;
+	int ret;
+
+	page = alloc_pages_noprof(gfp, order);
+	if (!page)
+		return page;
+
+	ret = set_pkey_pgtable(page, 1 << order);
+	if (ret) {
+		__free_pages(page, order);
+		return NULL;
+	}
+
+	return page;
+}
+
+void kpkeys_pgtable_free(struct page *page, unsigned int order)
+{
+	set_pkey_default(page, 1 << order);
+	__free_pages(page, order);
+}
+
+void __init kpkeys_hardened_pgtables_init(void)
+{
+	if (!kpkeys_supported())
+		return;
+
+	static_branch_enable(&kpkeys_hardened_pgtables_key);
+
+	ppa_finalize();
+	arch_kpkeys_protect_static_pgtables();
+}
+
+/*
+ * pkeys physmem allocator (PPA): allocator for very early page tables
+ * (especially for creating the linear map), based on memblock. Allocated
+ * ranges are tracked so that their pkey can be set once it is safe to do so.
+ */
+
+/*
+ * We may have to track many ranges when allocating page tables for the linear
+ * map, as their number grows with the amount of available memory. Assuming that
+ * memblock returns contiguous blocks whenever possible, the number of ranges
+ * to track cannot however exceed the number of regions that memblock itself
+ * tracks. memblock_allow_resize() hasn't been called yet at that point, so
+ * that limit is the size of the statically allocated array.
+ */
+#define PHYSMEM_MAX_RANGES	INIT_MEMBLOCK_MEMORY_REGIONS
+
+struct physmem_range {
+	phys_addr_t addr;
+	phys_addr_t size;
+};
+
+struct pkeys_physmem_allocator {
+	struct physmem_range allocated_ranges[PHYSMEM_MAX_RANGES];
+	unsigned int nr_allocated_ranges;
+};
+
+static struct pkeys_physmem_allocator pkeys_physmem_allocator __initdata;
+
+static int __init set_pkey_pgtable_phys(phys_addr_t pa, phys_addr_t size)
+{
+	unsigned long addr = (unsigned long)__va(pa);
+	int ret;
+
+	ret = set_memory_pkey(addr, size / PAGE_SIZE, KPKEYS_PKEY_PGTABLES);
+
+	WARN_ON(ret);
+	return ret;
+}
+
+static bool __init ppa_try_extend_last_range(phys_addr_t addr, phys_addr_t size)
+{
+	struct pkeys_physmem_allocator *ppa = &pkeys_physmem_allocator;
+	struct physmem_range *range;
+
+	if (!ppa->nr_allocated_ranges)
+		return false;
+
+	range = &ppa->allocated_ranges[ppa->nr_allocated_ranges - 1];
+
+	/* Merge the new range into the last range if they are contiguous */
+	if (addr == range->addr + range->size) {
+		range->size += size;
+		return true;
+	} else if (addr + size == range->addr) {
+		range->addr -= size;
+		range->size += size;
+		return true;
+	}
+
+	return false;
+}
+
+static void __init ppa_register_allocated_range(phys_addr_t addr,
+						phys_addr_t size)
+{
+	struct pkeys_physmem_allocator *ppa = &pkeys_physmem_allocator;
+	struct physmem_range *range;
+
+	if (!addr)
+		return;
+
+	if (ppa_try_extend_last_range(addr, size))
+		return;
+
+	/* Could not extend the last range, create a new one */
+	if (WARN_ON(ppa->nr_allocated_ranges >= PHYSMEM_MAX_RANGES))
+		return;
+
+	range = &ppa->allocated_ranges[ppa->nr_allocated_ranges++];
+	range->addr = addr;
+	range->size = size;
+}
+
+static void __init ppa_finalize(void)
+{
+	struct pkeys_physmem_allocator *ppa = &pkeys_physmem_allocator;
+
+	for (unsigned int i = 0; i < ppa->nr_allocated_ranges; i++) {
+		struct physmem_range *range = &ppa->allocated_ranges[i];
+
+		set_pkey_pgtable_phys(range->addr, range->size);
+	}
+}
+
+phys_addr_t __ref kpkeys_physmem_pgtable_alloc(void)
+{
+	size_t size = PAGE_SIZE;
+	phys_addr_t addr;
+
+	addr = memblock_phys_alloc_range(size, size, 0,
+					 MEMBLOCK_ALLOC_NOLEAKTRACE);
+	if (!addr)
+		return addr;
+
+	ppa_register_allocated_range(addr, size);
+
+	return addr;
+}
diff --git a/mm/memblock.c b/mm/memblock.c
index 6349c48154f4b..3b497d3556bb0 100644
--- a/mm/memblock.c
+++ b/mm/memblock.c
@@ -30,17 +30,6 @@
 
 #include "internal.h"
 
-#define INIT_MEMBLOCK_REGIONS			128
-#define INIT_PHYSMEM_REGIONS			4
-
-#ifndef INIT_MEMBLOCK_RESERVED_REGIONS
-# define INIT_MEMBLOCK_RESERVED_REGIONS		INIT_MEMBLOCK_REGIONS
-#endif
-
-#ifndef INIT_MEMBLOCK_MEMORY_REGIONS
-#define INIT_MEMBLOCK_MEMORY_REGIONS		INIT_MEMBLOCK_REGIONS
-#endif
-
 /**
  * DOC: memblock overview
  *
diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c
index ebd3ac997f645..1fec89168ff10 100644
--- a/mm/sparse-vmemmap.c
+++ b/mm/sparse-vmemmap.c
@@ -179,13 +179,29 @@ static pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, in
 	return pte;
 }
 
-static void * __meminit vmemmap_alloc_block_zero(unsigned long size, int node)
+static void * __meminit vmemmap_alloc_pgtable(int node)
 {
-	void *p = vmemmap_alloc_block(size, node);
+	void *p;
+
+	if (slab_is_available()) {
+		gfp_t gfp = GFP_KERNEL | __GFP_ZERO;
+		struct ptdesc *ptdesc = pagetable_alloc(gfp, 0);
+
+		return ptdesc ? ptdesc_address(ptdesc) : NULL;
+	}
+
+	if (kpkeys_hardened_pgtables_early_enabled()) {
+		phys_addr_t phys = kpkeys_physmem_pgtable_alloc();
+
+		p = phys ? phys_to_virt(phys) : NULL;
+	} else {
+		p = __earlyonly_bootmem_alloc(node, PAGE_SIZE, PAGE_SIZE,
+					      __pa(MAX_DMA_ADDRESS));
+	}
 
 	if (!p)
 		return NULL;
-	memset(p, 0, size);
+	memset(p, 0, PAGE_SIZE);
 
 	return p;
 }
@@ -194,7 +210,7 @@ static pmd_t * __meminit vmemmap_pmd_populate(pud_t *pud, unsigned long addr, in
 {
 	pmd_t *pmd = pmd_offset(pud, addr);
 	if (pmd_none(*pmd)) {
-		void *p = vmemmap_alloc_block_zero(PAGE_SIZE, node);
+		void *p = vmemmap_alloc_pgtable(node);
 		if (!p)
 			return NULL;
 		kernel_pte_init(p);
@@ -207,7 +223,7 @@ static pud_t * __meminit vmemmap_pud_populate(p4d_t *p4d, unsigned long addr, in
 {
 	pud_t *pud = pud_offset(p4d, addr);
 	if (pud_none(*pud)) {
-		void *p = vmemmap_alloc_block_zero(PAGE_SIZE, node);
+		void *p = vmemmap_alloc_pgtable(node);
 		if (!p)
 			return NULL;
 		pmd_init(p);
@@ -220,7 +236,7 @@ static p4d_t * __meminit vmemmap_p4d_populate(pgd_t *pgd, unsigned long addr, in
 {
 	p4d_t *p4d = p4d_offset(pgd, addr);
 	if (p4d_none(*p4d)) {
-		void *p = vmemmap_alloc_block_zero(PAGE_SIZE, node);
+		void *p = vmemmap_alloc_pgtable(node);
 		if (!p)
 			return NULL;
 		pud_init(p);
@@ -233,7 +249,7 @@ static pgd_t * __meminit vmemmap_pgd_populate(unsigned long addr, int node)
 {
 	pgd_t *pgd = pgd_offset_k(addr);
 	if (pgd_none(*pgd)) {
-		void *p = vmemmap_alloc_block_zero(PAGE_SIZE, node);
+		void *p = vmemmap_alloc_pgtable(node);
 		if (!p)
 			return NULL;
 		pgd_populate_kernel(addr, pgd, p);
@@ -346,10 +362,11 @@ static __meminit struct page *vmemmap_get_tail(unsigned int order, struct zone *
 	 * memmap_init().
 	 */
 
-	p = vmemmap_alloc_block_zero(PAGE_SIZE, node);
+	p = vmemmap_alloc_block(PAGE_SIZE, node);
 	if (!p)
 		return NULL;
 
+	memset(p, 0, PAGE_SIZE);
 	tail = virt_to_page(p);
 	zone->vmemmap_tails[idx] = tail;
 
diff --git a/mm/tests/kpkeys_hardened_pgtables_kunit.c b/mm/tests/kpkeys_hardened_pgtables_kunit.c
new file mode 100644
index 0000000000000..80d4f5144e48d
--- /dev/null
+++ b/mm/tests/kpkeys_hardened_pgtables_kunit.c
@@ -0,0 +1,200 @@
+// SPDX-License-Identifier: GPL-2.0-only
+#include <kunit/test.h>
+#include <linux/mman.h>
+#include <linux/pgtable.h>
+#include <linux/set_memory.h>
+#include <linux/vmalloc.h>
+
+static void free_page_wrapper(void *ctx)
+{
+	__free_page((struct page *)ctx);
+}
+
+KUNIT_DEFINE_ACTION_WRAPPER(vfree_wrapper, vfree, const void *);
+
+static pud_t *pud_off_k(unsigned long va)
+{
+	return pud_offset(p4d_offset(pgd_offset_k(va), va), va);
+}
+
+static pte_t *get_kernel_pte(unsigned long addr)
+{
+	pmd_t *pmdp = pmd_off_k(addr);
+
+	if (!pmdp || pmd_leaf(*pmdp))
+		return NULL;
+
+	return pte_offset_kernel(pmdp, addr);
+}
+
+#define write_pgtable(type, ptr) do {					\
+	type##_t val;							\
+	int ret;							\
+									\
+	pr_debug("%s: writing to "#type" at %px\n", __func__, (ptr));	\
+									\
+	val = type##p_get(ptr);						\
+	ret = copy_to_kernel_nofault(ptr, &val, sizeof(val));		\
+	KUNIT_EXPECT_EQ_MSG(test, ret, -EFAULT,				\
+			    "Direct "#type" write wasn't prevented");	\
+} while (0)
+
+/*
+ * Try to write linear map page tables, at every level. This is worthwhile
+ * because those page table pages are obtained from different allocators:
+ *
+ * - Static memory (part of the kernel image) for PGD
+ * - memblock for PUD and possibly PMD/PTE
+ * - pagetable_alloc() (buddy allocator) for PMD/PTE if large block mappings are
+ *   used and the linear map is split after being created
+ */
+static void write_direct_map_pgtables(struct kunit *test)
+{
+	struct page *page;
+	unsigned long addr;
+	pgd_t *pgdp;
+	p4d_t *p4dp;
+	pud_t *pudp;
+	pmd_t *pmdp;
+	pte_t *ptep;
+	int ret;
+
+	if (!kpkeys_supported())
+		kunit_skip(test, "kpkeys are not supported");
+
+	page = alloc_page(GFP_KERNEL);
+	KUNIT_ASSERT_NOT_NULL(test, page);
+	ret = kunit_add_action_or_reset(test, free_page_wrapper, page);
+	KUNIT_ASSERT_EQ(test, ret, 0);
+
+	/* Ensure page is PTE-mapped (splitting the linear map if necessary) */
+	ret = set_direct_map_invalid_noflush(page);
+	KUNIT_ASSERT_EQ(test, ret, 0);
+	ret = set_direct_map_default_noflush(page);
+	KUNIT_ASSERT_EQ(test, ret, 0);
+
+	addr = (unsigned long)page_address(page);
+
+	pgdp = pgd_offset_k(addr);
+	KUNIT_ASSERT_NOT_NULL_MSG(test, pgdp, "Failed to get PGD");
+	/*
+	 * swapper_pg_dir is still writable at this stage, so don't check it.
+	 * It is not protected by kpkeys_hardened_pgtables because it should be
+	 * made read-only by mark_rodata_ro(). However since these
+	 * KUnit tests are builtin, they are run before mark_rodata_ro() is
+	 * called.
+	 */
+
+	p4dp = p4d_offset(pgdp, addr);
+	KUNIT_ASSERT_NOT_NULL_MSG(test, p4dp, "Failed to get P4D");
+	if (!mm_p4d_folded(&init_mm))
+		write_pgtable(p4d, p4dp);
+
+	pudp = pud_offset(p4dp, addr);
+	KUNIT_ASSERT_NOT_NULL_MSG(test, pudp, "Failed to get PUD");
+	if (!mm_pud_folded(&init_mm))
+		write_pgtable(pud, pudp);
+
+	pmdp = pmd_offset(pudp, addr);
+	KUNIT_ASSERT_NOT_NULL_MSG(test, pmdp, "Failed to get PMD");
+	write_pgtable(pmd, pmdp);
+
+	ptep = pte_offset_kernel(pmdp, addr);
+	KUNIT_ASSERT_NOT_NULL_MSG(test, ptep, "Failed to get PTE");
+	write_pgtable(pte, ptep);
+}
+
+/* Worth checking since the kernel image is mapped with static page tables */
+static void write_kernel_image_pud(struct kunit *test)
+{
+	pud_t *pudp;
+
+	if (!kpkeys_supported())
+		kunit_skip(test, "kpkeys are not supported");
+
+	/* The kernel is probably block-mapped, check the PUD to be safe */
+	pudp = pud_off_k((unsigned long)&init_mm);
+	KUNIT_ASSERT_NOT_NULL_MSG(test, pudp, "Failed to get PUD");
+
+	write_pgtable(pud, pudp);
+}
+
+static void write_kernel_vmalloc_pte(struct kunit *test)
+{
+	void *mem;
+	pte_t *ptep;
+	int ret;
+
+	if (!kpkeys_supported())
+		kunit_skip(test, "kpkeys are not supported");
+
+	mem = vmalloc(PAGE_SIZE);
+	KUNIT_ASSERT_NOT_NULL(test, mem);
+	ret = kunit_add_action_or_reset(test, vfree_wrapper, mem);
+	KUNIT_ASSERT_EQ(test, ret, 0);
+
+	/* vmalloc() without VM_ALLOW_HUGE_VMAP is PTE-mapped */
+	ptep = get_kernel_pte((unsigned long)mem);
+	KUNIT_ASSERT_NOT_NULL_MSG(test, ptep, "Failed to get PTE");
+
+	write_pgtable(pte, ptep);
+}
+
+static void write_vmemmap_pmd(struct kunit *test)
+{
+	struct page *page;
+	pmd_t *pmdp;
+
+	if (!kpkeys_supported())
+		kunit_skip(test, "kpkeys are not supported");
+
+	/*
+	 * We just need the address of some struct page, so we can free the
+	 * page right away.
+	 */
+	page = alloc_page(GFP_KERNEL);
+	KUNIT_ASSERT_NOT_NULL(test, page);
+	__free_page(page);
+
+	/* vmemmap may use PMD block mappings */
+	pmdp = pmd_off_k((unsigned long)page);
+	KUNIT_ASSERT_NOT_NULL_MSG(test, pmdp, "Failed to get PMD");
+	write_pgtable(pmd, pmdp);
+}
+
+static void write_user_pmd(struct kunit *test)
+{
+	pmd_t *pmdp;
+	unsigned long uaddr;
+
+	if (!kpkeys_supported())
+		kunit_skip(test, "kpkeys are not supported");
+
+	uaddr = kunit_vm_mmap(test, NULL, 0, PAGE_SIZE, PROT_READ,
+			      MAP_ANONYMOUS | MAP_PRIVATE | MAP_POPULATE, 0);
+	KUNIT_ASSERT_NE_MSG(test, uaddr, 0, "Could not create userspace mm");
+
+	/* We passed MAP_POPULATE so a PMD should already be allocated */
+	pmdp = pmd_off(current->mm, uaddr);
+	KUNIT_ASSERT_NOT_NULL_MSG(test, pmdp, "Failed to get PMD");
+
+	write_pgtable(pmd, pmdp);
+}
+
+static struct kunit_case kpkeys_hardened_pgtables_test_cases[] = {
+	KUNIT_CASE(write_direct_map_pgtables),
+	KUNIT_CASE(write_kernel_image_pud),
+	KUNIT_CASE(write_kernel_vmalloc_pte),
+	KUNIT_CASE(write_vmemmap_pmd),
+	KUNIT_CASE(write_user_pmd),
+	{}
+};
+
+static struct kunit_suite kpkeys_hardened_pgtables_test_suite = {
+	.name = "kpkeys_hardened_pgtables",
+	.test_cases = kpkeys_hardened_pgtables_test_cases,
+};
+kunit_test_suite(kpkeys_hardened_pgtables_test_suite);
+
+MODULE_DESCRIPTION("Tests for the kpkeys_hardened_pgtables feature");
+MODULE_LICENSE("GPL");
diff --git a/security/Kconfig.hardening b/security/Kconfig.hardening
index 6923036e1a2f5..ad6152961f8ee 100644
--- a/security/Kconfig.hardening
+++ b/security/Kconfig.hardening
@@ -271,6 +271,30 @@ config BUG_ON_DATA_CORRUPTION
 
 	  If unsure, say N.
 
+config KPKEYS_HARDENED_PGTABLES
+	bool "Harden page tables using kernel pkeys"
+	depends on ARCH_HAS_KPKEYS_HARDENED_PGTABLES
+	help
+	  This option makes all page tables mostly read-only by
+	  allocating them with a non-default protection key (pkey) and
+	  only enabling write access to that pkey in routines that are
+	  expected to write to page table entries.
+
+	  This option has no effect if the system does not support
+	  kernel pkeys.
+
+config KPKEYS_HARDENED_PGTABLES_KUNIT_TEST
+	bool "KUnit tests for kpkeys_hardened_pgtables" if !KUNIT_ALL_TESTS
+	depends on KPKEYS_HARDENED_PGTABLES
+	depends on KUNIT=y
+	default KUNIT_ALL_TESTS
+	help
+	  Enable this option to check that the kpkeys_hardened_pgtables feature
+	  functions as intended, i.e. prevents arbitrary writes to user and
+	  kernel page tables.
+
+	  If unsure, say N.
+
 endmenu
 
 config CC_HAS_RANDSTRUCT

gemini-3.1-pro-preview

							input: 21257
							output: 34
							thoughts: 1736

**Fuzzing Target Identification: KPKEYS Hardened Page Tables on ARM64**

Okay, so I need to assess this patch's impact for fuzzing purposes. It looks like it introduces a significant functional change – hardening page tables on ARM64 using kernel pkeys (KPKEYS). Specifically, `KPKEYS_HARDENED_PGTABLES` is the key, and it seems to modify memory management, particularly page table handling.  Given the nature of this change, memory management is always a good target.

First, let's verify the relevant configurations: `ARM64_POE` and `KPKEYS_HARDENED_PGTABLES`. Good, they exist and `ARM64_POE` appears to always be enabled, or has been in the current environment.

Now, where are the functions of interest? I've spotted `kpkeys_pgtable_alloc`, `kpkeys_pgtable_free`, `kpkeys_physmem_pgtable_alloc` within `mm/kpkeys_hardened_pgtables.c`.  I'm also seeing modifications in `arch/arm64/mm/mmu.c`, specifically `arch_set_user_pkey_access` and `arch/arm64/mm/pageattr.c` with `set_memory_pkey`.  I need to check how reachable `kpkeys_pgtable_alloc` is, which is called from `pagetable_alloc_noprof` in `include/linux/mm.h`. `kpkeys_pgtable_free` is called from `__pagetable_free`. These are good.

Let's prioritize those as potential fuzzing targets.

After a deeper dive, I've confirmed that these are global, non-static functions and thus, are more easily accessible for continuous fuzzing.

My list of functions to focus on is now:

*   `kpkeys_pgtable_alloc`
*   `kpkeys_pgtable_free`
*   `kpkeys_physmem_pgtable_alloc`
*   `set_memory_pkey`
*   `arch_set_user_pkey_access`

`arch_set_user_pkey_access` seems to be modified, so definitely worth a closer look.

I'll keep `KPKEYS_HARDENED_PGTABLES_KUNIT_TEST` in mind, but it isn't required for fuzzing this.

So, the configurations to ensure are:

*   `ARM64_POE` (always enabled)
*   `KPKEYS_HARDENED_PGTABLES`

Okay, this is a solid starting point. I'll focus my fuzzing efforts on those specific functions and make sure the appropriate configurations are enabled.

{
  "Expression": "config ARM64_POE",
  "PathPrefix": "arch/arm64/Kconfig"
}

{
  "Output": "arch/arm64/Kconfig=2198=menu \"ARMv8.9 architectural features\"\narch/arm64/Kconfig-2199-\narch/arm64/Kconfig:2200:config ARM64_POE\narch/arm64/Kconfig-2201-\tprompt \"Permission Overlay Extension\"\n"
}

gemini-3.1-pro-preview

							input: 23715
							output: 37
							thoughts: 15

Here's my attempt at summarizing the thought process, tailored for an expert audience:

**Analyzing the `ARM64_POE` Definition**

Okay, so the initial prompt is pretty straightforward - let's dive right into the full definition of `ARM64_POE`. I'm immediately thinking, "why the interest in *this* particular prefetch optimization?". It's a common area, but understanding the specific context is key here. Am I looking at it from a microarchitecture perspective, perhaps analyzing instruction-level details and pipeline stalls, or is this related to memory management and coherence on a multi-core ARM system? Is the context embedded systems, or high-performance computing?

My immediate goal is to establish the scope of this definition. What aspects are being emphasized? Are we talking about the core architecture (e.g., specific instructions, register usage), the memory hierarchy interaction, or perhaps some software-level implications with compiler optimization or runtime libraries? Understanding the relevant abstraction level is important, since the definition will almost certainly have implications for memory access and instruction fetching.

I'm mentally preparing to parse the definition for keywords related to prefetch algorithms, cache management policies (e.g., speculative loading, demand prefetching), and the impact on the memory subsystem (latency, bandwidth, coherency). I'm also anticipating potential links to power consumption and performance trade-offs, depending on the particular implementation. A formal definition will hopefully include the specific architectural features and their potential constraints. In short, let's understand the problem that `ARM64_POE` is trying to solve, and the specific mechanisms used.

{
  "File": "arch/arm64/Kconfig",
  "FirstLine": 2198,
  "LineCount": 20
}

{
  "Contents": "2198:\tmenu \"ARMv8.9 architectural features\"\n2199:\t\n2200:\tconfig ARM64_POE\n2201:\t\tprompt \"Permission Overlay Extension\"\n2202:\t\tdef_bool y\n2203:\t\tselect ARCH_USES_HIGH_VMA_FLAGS\n2204:\t\tselect ARCH_HAS_PKEYS\n2205:\t\tselect ARCH_HAS_KPKEYS\n2206:\t\tselect ARCH_HAS_KPKEYS_HARDENED_PGTABLES\n2207:\t\thelp\n2208:\t\t  The Permission Overlay Extension is used to implement Memory\n2209:\t\t  Protection Keys. Memory Protection Keys provides a mechanism for\n2210:\t\t  enforcing page-based protections, but without requiring modification\n2211:\t\t  of the page tables when an application changes protection domains.\n2212:\t\n2213:\t\t  For details, see Documentation/core-api/protection-keys.rst\n2214:\t\n2215:\t\t  If unsure, say y.\n2216:\t\n2217:\tconfig ARCH_PKEY_BITS\n"
}

gemini-3.1-pro-preview

							input: 24398
							output: 34
							thoughts: 40

**My Initial Thoughts on ARM64_POE and KPKEYS_HARDENED_PGTABLES**

Okay, so I'm looking at these configurations, and immediately my mind jumps to the implications. `ARM64_POE` is being enabled by default – the prompt suggests a default of "y". That's interesting, suggesting this is a potentially important or recommended feature for the ARM64 architecture, and that it's probably going to be part of most builds. I'll need to dig into what "POE" actually represents in this context on ARM64 and the default setting.

Then, there's `KPKEYS_HARDENED_PGTABLES`, which lives in `security/Kconfig.hardening`. Right away, that location is telling me this is security-related. The "KPKEYS" part suggests key protection for the kernel, which is a significant signal; it's likely related to protecting page tables from unauthorized modification. The fact that it’s hardened is good. I need to investigate how it is hardened, as well as the performance and compatibility implications. Security hardening is an area I'm always keen to understand the details of.

{
  "File": "security/Kconfig.hardening",
  "FirstLine": 270,
  "LineCount": 20
}

{
  "Contents": " 270:\t\t  for validity.\n 271:\t\n 272:\t\t  If unsure, say N.\n 273:\t\n 274:\tconfig KPKEYS_HARDENED_PGTABLES\n 275:\t\tbool \"Harden page tables using kernel pkeys\"\n 276:\t\tdepends on ARCH_HAS_KPKEYS_HARDENED_PGTABLES\n 277:\t\thelp\n 278:\t\t  This option makes all page tables mostly read-only by\n 279:\t\t  allocating them with a non-default protection key (pkey) and\n 280:\t\t  only enabling write access to that pkey in routines that are\n 281:\t\t  expected to write to page table entries.\n 282:\t\n 283:\t\t  This option has no effect if the system does not support\n 284:\t\t  kernel pkeys.\n 285:\t\n 286:\tconfig KPKEYS_HARDENED_PGTABLES_KUNIT_TEST\n 287:\t\tbool \"KUnit tests for kpkeys_hardened_pgtables\" if !KUNIT_ALL_TESTS\n 288:\t\tdepends on KPKEYS_HARDENED_PGTABLES\n 289:\t\tdepends on KUNIT=y\n"
}

gemini-3.1-pro-preview

							input: 25044
							output: 41
							thoughts: 231

**My Configuration Considerations for Enhanced ARM64 Memory Protection**

Okay, so I'm looking at enabling some key configurations here. `KPKEYS_HARDENED_PGTABLES` seems to be the crucial one, and it's contingent on `ARCH_HAS_KPKEYS_HARDENED_PGTABLES`. Given the goal, I think enabling `ARM64_POE` is the logical first step, and that should, in turn, pull in `KPKEYS_HARDENED_PGTABLES`.

Now, before I commit to anything, let's just do a quick scan of related configurations to make sure I haven't missed anything obvious. I need to keep `ARCH_HAS_KPKEYS` and `ARCH_HAS_KPKEYS_HARDENED_PGTABLES` in mind. Ah, right, those are selected by `ARM64_POE`. So, it looks like `ARM64_POE` is definitely the parent, and `KPKEYS_HARDENED_PGTABLES` is the dependent. No surprises there.

Alright, so the configs to enable, in addition to `ARM64_POE`, would then be `KPKEYS_HARDENED_PGTABLES`. I've got a good handle on that part.

Next, I need to focus on the key functions involved in this. I'll need to pay close attention to `kpkeys_pgtable_alloc`, `kpkeys_pgtable_free`, `kpkeys_physmem_pgtable_alloc`, `set_memory_pkey`, and `arch_set_user_pkey_access`. I'll need to review these and make sure they are operating as intended. Let's make sure that `arch_set_user_pkey_access` is in `arch/arm64/mm/mmu.c`, as I'm pretty sure that's where I need to look.

{
  "Expression": "arch_set_user_pkey_access",
  "PathPrefix": "arch/arm64/mm/mmu.c"
}

{
  "Output": "arch/arm64/mm/mmu.c=2300=void __cpu_replace_ttbr1(pgd_t *pgdp, bool cnp)\n--\narch/arm64/mm/mmu.c-2328-#ifdef CONFIG_ARCH_HAS_PKEYS\narch/arm64/mm/mmu.c:2329:int arch_set_user_pkey_access(int pkey, unsigned long init_val)\narch/arm64/mm/mmu.c-2330-{\n"
}

gemini-3.1-pro-preview

							input: 25866
							output: 37
							thoughts: 32

Okay, here's my take on those thoughts, tailored for an expert audience:

**Analyzing `set_memory_pkey` on ARM64**

Yes, that's correct. I'm on the right track here. Now, I need to confirm something specific about how the page protection keys are managed, so let's dig into `set_memory_pkey` within `arch/arm64/mm/pageattr.c`. I need to refresh my memory on the interactions there, particularly how the physical address translation is updated when page keys change and what the implications are regarding TLB invalidation strategies and potential races. I'm primarily interested in looking into the code related to memory protection keys and how they impact security.

{
  "Expression": "set_memory_pkey",
  "PathPrefix": "arch/arm64/mm/pageattr.c"
}

{
  "Output": "arch/arm64/mm/pageattr.c=361=int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid)\n--\narch/arm64/mm/pageattr.c-371-#ifdef CONFIG_ARCH_HAS_KPKEYS\narch/arm64/mm/pageattr.c:372:int set_memory_pkey(unsigned long addr, int numpages, int pkey)\narch/arm64/mm/pageattr.c-373-{\n"
}

gemini-3.1-pro-preview

							input: 26226
							output: 39
							thoughts: 34

Okay, here's my take on those thoughts, framed as an expert's internal monologue:

**Quick Confirmation and a Code Dive**

Right, that seems straightforward enough. Yep, it is. Let's not waste any time, I'm going to quickly check `kpkeys_pgtable_alloc` in `mm/kpkeys_hardened_pgtables.c`. I need to ensure the allocation behavior aligns with the current security design. I should probably pay particular attention to its interaction with the hardened page table mechanisms. I need a quick sanity check before moving on to the more complex bits, you know, to confirm my assumptions are accurate.

{
  "Expression": "kpkeys_pgtable_alloc",
  "PathPrefix": "mm/kpkeys_hardened_pgtables.c"
}

{
  "Output": "mm/kpkeys_hardened_pgtables.c=35=static void ppa_finalize(void);\nmm/kpkeys_hardened_pgtables.c-36-\nmm/kpkeys_hardened_pgtables.c:37:struct page *kpkeys_pgtable_alloc(gfp_t gfp, unsigned int order)\nmm/kpkeys_hardened_pgtables.c-38-{\n"
}

gemini-3.1-pro-preview

							input: 26547
							output: 39
							thoughts: 31

Okay, here's my summary, taking on the persona you requested:

**Assessing the Landscape: pgtable_free**

Yes. I've got it. Time to dive into the specifics of `kpkeys_pgtable_free`. I need to zero in on this function within `mm/kpkeys_hardened_pgtables.c`. I need to ensure a complete understanding of its role and how it interacts with the broader context of hardened page tables. My expertise allows me to analyze this with precision, identifying potential strengths and vulnerabilities.

{
  "Expression": "kpkeys_pgtable_free",
  "PathPrefix": "mm/kpkeys_hardened_pgtables.c"
}

{
  "Output": "mm/kpkeys_hardened_pgtables.c=37=struct page *kpkeys_pgtable_alloc(gfp_t gfp, unsigned int order)\n--\nmm/kpkeys_hardened_pgtables.c-54-\nmm/kpkeys_hardened_pgtables.c:55:void kpkeys_pgtable_free(struct page *page, unsigned int order)\nmm/kpkeys_hardened_pgtables.c-56-{\n"
}

gemini-3.1-pro-preview

							input: 26854
							output: 42
							thoughts: 34

Okay, here's my interpretation of those thoughts, tailored for an expert audience, presented in the first person:

**Kernel Memory Protection - Deep Dive**

Right. I need to take a look at the memory allocation function for the `kpkeys_physmem_pgtable` specifically in relation to the hardened page table implementation. Let's dig into `mm/kpkeys_hardened_pgtables.c` and focus on `kpkeys_physmem_pgtable_alloc`. That's where the critical aspects of this are going to be defined; need to figure out how memory is allocated, initialized, and protected from there.

{
  "Expression": "kpkeys_physmem_pgtable_alloc",
  "PathPrefix": "mm/kpkeys_hardened_pgtables.c"
}

{
  "Output": "mm/kpkeys_hardened_pgtables.c=155=static void __init ppa_finalize(void)\n--\nmm/kpkeys_hardened_pgtables.c-165-\nmm/kpkeys_hardened_pgtables.c:166:phys_addr_t __ref kpkeys_physmem_pgtable_alloc(void)\nmm/kpkeys_hardened_pgtables.c-167-{\n"
}

gemini-3.1-pro-preview

							input: 27181
							output: 221
							thoughts: 802

**My Analysis of the Kernel PKey Hardened Paging Table Implementation**

Okay, so I've been digging into this patch that implements hardened page tables using kernel pkeys (kpkeys) on ARM64. First things first, yes, this looks like a substantial change. I've gone through the code, focusing on the key functions to identify what needs rigorous fuzzing.

My initial scan flagged a few functions as potentially interesting: `arch_kpkeys_protect_static_pgtables`, `kpkeys_hardened_pgtables_init`, `kpkeys_lazy_mmu_enter`, `kpkeys_lazy_mmu_exit`, and `por_elx_set_pkey_perms`. However, those are either `__init` or static inline. Not a priority for fuzzing the core functionality.

Then I considered `arch_kpkeys_enter_context` and `arch_kpkeys_leave_context`. Again, those are static inline. Same goes for `set_pkey_pgtable` and `set_pkey_default`. Those also have less bearing on the core functionality. `ppa_try_extend_last_range`, `ppa_register_allocated_range`, and `ppa_finalize` are all `__init` functions.

My focus narrows significantly. I believe the critical non-inline, non-init functions that demand fuzzing are:

* `kpkeys_pgtable_alloc`
* `kpkeys_pgtable_free`
* `kpkeys_physmem_pgtable_alloc` (even though it's `__ref` and is called from `early_pgtable_alloc` which is `__init`, and `vmemmap_alloc_pgtable` which is `__meminit` that can still be called after boot)
* `set_memory_pkey`
* `arch_set_user_pkey_access`

The patch fundamentally alters how page tables are allocated, freed, and how user-space access is controlled. It's a clear change to the core memory management subsystem. Memory safety and correctness are paramount here. These functions are where I'd expect the fuzzing to really pay off.

The underlying rationale is clear: a new feature introduces a dedicated file (`mm/kpkeys_hardened_pgtables.c`) and modifies the ARM64 memory management to allocate page tables with specific pkeys, limiting write access. That's a classic case for fuzzing.

I've also checked the relevant configs. `ARM64_POE` and `KPKEYS_HARDENED_PGTABLES` are the relevant configurations. The dependencies for these are `ARCH_HAS_KPKEYS` and `ARCH_HAS_KPKEYS_HARDENED_PGTABLES`, which are selected by `ARM64_POE`. So, checking the two main configs should be enough. Now, I'll need to prepare the fuzzing environment with the `set-results` tool.

the following configs do not exist in the kernel tree: ARM64_POE

{
  "EnableConfigs": [
    "ARM64_POE",
    "KPKEYS_HARDENED_PGTABLES"
  ],
  "FocusSymbols": [
    "kpkeys_pgtable_alloc",
    "kpkeys_pgtable_free",
    "kpkeys_physmem_pgtable_alloc",
    "set_memory_pkey",
    "arch_set_user_pkey_access"
  ],
  "Reasoning": "The patch introduces a new feature to harden page tables using kernel pkeys (kpkeys) on ARM64. It adds a new file `mm/kpkeys_hardened_pgtables.c` and modifies ARM64 memory management code to allocate page tables with a specific pkey and restrict write access to them. The changes include new page table allocation and freeing functions, as well as modifications to `arch_set_user_pkey_access`. This is a significant functional change to the core memory management subsystem that should be fuzzed to ensure memory safety and correctness.",
  "WorthFuzzing": true
}

{
  "error": "the following configs do not exist in the kernel tree: ARM64_POE"
}

gemini-3.1-pro-preview

							input: 28870
							output: 34
							thoughts: 34

Here's my take on those thoughts:

**Investigating the Missing `ARM64_POE`**

Okay, hold on a minute. I was pretty sure I'd seen something related to `ARM64_POE` kicking around the kernel sources. But now I'm drawing a blank. The build system is complaining, and my gut is telling me I've missed something. Time to double-check. Let's go straight back to the source – specifically, `arch/arm64/Kconfig`. That's the definitive place for these architecture-specific configuration options. I'm going to carefully scan through that file, looking for any trace of `ARM64_POE`. It's possible I'm misremembering the exact name, or perhaps it's nested under a different, more general option. Gotta be thorough. This is probably going to be a quick one, just a search-and-confirm.

{
  "Expression": "config ARM64_POE",
  "PathPrefix": "arch/arm64/Kconfig"
}

{
  "Output": "arch/arm64/Kconfig=2198=menu \"ARMv8.9 architectural features\"\narch/arm64/Kconfig-2199-\narch/arm64/Kconfig:2200:config ARM64_POE\narch/arm64/Kconfig-2201-\tprompt \"Permission Overlay Extension\"\n"
}

gemini-3.1-pro-preview

							input: 29231
							output: 37
							thoughts: 94

**Investigating the Configuration Mystery**

Okay, so I'm a bit perplexed. The `set-results` tool just told me that `ARM64_POE` doesn't exist, which seems impossible because I'm *certain* I've seen it lurking around in `arch/arm64/Kconfig`. It has to be there, right?

Alright, maybe I'm overlooking something, perhaps it's not in the base kernel tree, but rather within a patch I'm working with. That could explain the initial discrepancy. But wait, the `set-results` tool is supposed to be referencing the kernel tree itself when it does these checks. That's its whole purpose.

Right, time to get back to basics. I need to explicitly verify that `ARM64_POE` is actually present in `arch/arm64/Kconfig` *within* the kernel tree I'm using for this analysis. I'm going to double-check that file right now. I need to make sure I am not missing something that is readily apparent to me.

{
  "File": "arch/arm64/Kconfig",
  "FirstLine": 2198,
  "LineCount": 10
}

{
  "Contents": "2198:\tmenu \"ARMv8.9 architectural features\"\n2199:\t\n2200:\tconfig ARM64_POE\n2201:\t\tprompt \"Permission Overlay Extension\"\n2202:\t\tdef_bool y\n2203:\t\tselect ARCH_USES_HIGH_VMA_FLAGS\n2204:\t\tselect ARCH_HAS_PKEYS\n2205:\t\tselect ARCH_HAS_KPKEYS\n2206:\t\tselect ARCH_HAS_KPKEYS_HARDENED_PGTABLES\n2207:\t\thelp\n"
}

gemini-3.1-pro-preview

							input: 29742
							output: 213
							thoughts: 126

**Debugging a Configuration Issue**

Okay, so I'm trying to figure out why the results aren't being set correctly. I know the relevant setting, `KPKEYS_HARDENED_PGTABLES`, is definitely present in the configuration, but the process using `set-results` is failing. Hmm, what's going on? My first thought is that the `set-results` tool is probably architecture-aware. Maybe it's checking the configuration specifically for the current build architecture, which, in this case, could be x86_64. 

That rings true, because if `set-results` is running something like `make defconfig` or digging around in the `Kconfig` files, it would naturally be using the config associated with the x86_64 architecture. If this particular setting, `KPKEYS_HARDENED_PGTABLES`, is specific to the ARM64 configuration, the tool's x86_64 checks wouldn't find it. Alright, I'll bypass this logic, I'll provide `KPKEYS_HARDENED_PGTABLES` directly and see if that resolves the issue. That should be a quick test.

{
  "EnableConfigs": [
    "KPKEYS_HARDENED_PGTABLES"
  ],
  "FocusSymbols": [
    "kpkeys_pgtable_alloc",
    "kpkeys_pgtable_free",
    "kpkeys_physmem_pgtable_alloc",
    "set_memory_pkey",
    "arch_set_user_pkey_access"
  ],
  "Reasoning": "The patch introduces a new feature to harden page tables using kernel pkeys (kpkeys) on ARM64. It adds a new file `mm/kpkeys_hardened_pgtables.c` and modifies ARM64 memory management code to allocate page tables with a specific pkey and restrict write access to them. The changes include new page table allocation and freeing functions, as well as modifications to `arch_set_user_pkey_access`. This is a significant functional change to the core memory management subsystem that should be fuzzed to ensure memory safety and correctness.",
  "WorthFuzzing": true
}

{
  "EnableConfigs": [
    "KPKEYS_HARDENED_PGTABLES"
  ],
  "FocusSymbols": [
    "kpkeys_pgtable_alloc",
    "kpkeys_pgtable_free",
    "kpkeys_physmem_pgtable_alloc",
    "set_memory_pkey",
    "arch_set_user_pkey_access"
  ],
  "Reasoning": "The patch introduces a new feature to harden page tables using kernel pkeys (kpkeys) on ARM64. It adds a new file `mm/kpkeys_hardened_pgtables.c` and modifies ARM64 memory management code to allocate page tables with a specific pkey and restrict write access to them. The changes include new page table allocation and freeing functions, as well as modifications to `arch_set_user_pkey_access`. This is a significant functional change to the core memory management subsystem that should be fuzzed to ensure memory safety and correctness.",
  "WorthFuzzing": true
}

{
  "WorthFuzzing": true
}

{
  "KMSANReasoning": "The patch series introduces `KPKEYS_HARDENED_PGTABLES` to harden page tables using kernel pkeys (KPKEYS) on ARM64. It modifies page table allocation to set a non-default protection key and uses `guard(kpkeys_hardened_pgtables)()` to temporarily enable write access when modifying page table entries.\n\nAfter a thorough review of the changes:\n1. **No new structures sent to user space:** The patch adds `por_el1` and `kpkeys_state_lazy_mmu` to `struct thread_struct`, but these are internal kernel state and are not exposed to user space via ptrace, signals, or any other mechanism.\n2. **Page table allocation zeroing:** The patch modifies page table allocation to use `kpkeys_pgtable_alloc` and `kpkeys_physmem_pgtable_alloc`. These functions wrap standard allocators (`alloc_pages_noprof` and `memblock_phys_alloc_range`). The zeroing behavior remains identical to the original code: `early_pgtable_alloc` allocations are zeroed by `init_clear_pgtable()`, `pagetable_alloc_noprof` allocations are zeroed by `alloc_pages_noprof` if `__GFP_ZERO` is passed, and `vmemmap_alloc_pgtable` explicitly calls `memset(p, 0, PAGE_SIZE)`.\n3. **No uninitialized memory risks:** The `struct kpkeys_state` used in `scoped_guard` is a local variable and is not copied to user space. The conditional initialization in `arch_kpkeys_enter_context` is safe because the uninitialized fields are never read if `entered_context` is false.\n\nThe changes primarily involve memory protection and control flow around page table modifications. Any bugs introduced would likely be out-of-bounds accesses, use-after-free, or invalid memory accesses, which are well-covered by KASAN and standard bug detectors. There is no risk of uninitialized memory usage or info-leaks that would require KMSAN.",
  "NeedsKMSAN": false
}

You are an expert Linux kernel security engineer specializing in kernel memory
error detectors (KASAN and KMSAN). Your job is to review the provided patch series and
determine if the code changes justify spawning a dedicated KMSAN (KernelMemorySanitizer)
fuzzing session in addition to standard KASAN fuzzing.

CRITICAL DISTINCTION BETWEEN KASAN AND KMSAN:
- Standard KASAN kernel builds (upstream-apparmor-kasan.config) already enable
  a comprehensive suite of debugging tools and sanitizers, including KASAN
  (out-of-bounds accesses, use-after-free, double free, invalid free), LOCKDEP
  (locking bugs and deadlocks), UB-sanitizers, and memory corruption checks.
- KMSAN (KernelMemorySanitizer) detects reads of UNINITIALIZED memory (stack, heap,
  or page allocations) and kernel-to-user memory info-leaks.

Rule: THERE IS NO SENSE IN RUNNING A KMSAN SESSION IF A BUG CAN BE CAUGHT BY KASAN,
LOCKDEP, OR OTHER STANDARD BUG DETECTORS.
A dedicated KMSAN fuzzing session incurs significant resource costs. You must ONLY
set NeedsKMSAN=true if the code changes introduce or expose UNINITIALIZED MEMORY risks
that are detected ONLY by KMSAN.

Look holistically at the patch series and surrounding code. Even if no direct
uninitialized field accesses or new buffer allocations are added in the diff itself,
a patch may alter control flow, bounds checking, or data length calculations in ways
that change how the rest of the code operates on existing buffers (e.g. allowing
uninitialized stack/heap memory to be read, copied to user space, or used in control
flow). Do not hesitate to use your code access tools to inspect the surrounding code,
called functions, and callers.

Set NeedsKMSAN=true ONLY IF the patch introduces or modifies:
1. Kernel structures sent to user space (via copy_to_user, put_user, netlink skb
   attributes, ioctl output arguments, socket options, or BPF buffers) where fields
   or structure padding might not be fully initialized/zeroed.
2. Conditional logic or branching that depends on potentially uninitialized variables
   or struct fields.
3. Allocation or initialization of complex data structures where uninitialized fields
   could be read later in reachable code paths.
4. Bounds checks, lengths, or logic in a way that allows surrounding code to access
   uninitialized bytes of existing buffers.

Set NeedsKMSAN=false IF:
- The code changes primarily risk out-of-bounds access, array overflows, NULL pointer
  dereferences, locking deadlocks, or use-after-free bugs (these are already caught
  by KASAN, LOCKDEP, or standard bug detectors).
- All stack/heap structures touched or introduced by the patch are fully zeroed
  or initialized (e.g. using = {0}, memset, kzalloc) before being read or copied.
- The patch does not introduce any risk of uninitialized memory usage or info-leaks.

Use your code access tools to inspect the surrounding code if necessary, then provide
detailed KMSANReasoning contrasting KASAN vs KMSAN applicability for this patch.
Prefer calling several tools at the same time to save round-trips.


Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.

For your convenience, here is the diff of the changes:
commit bd271568204405923f3b37fcb4ded599be77ced1
Author: syz-cluster <triage@syzkaller.com>
Date:   Tue Aug 18 17:01:35 2026 +0000

    syz-cluster: applied patch under review

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index b3afe0688919b..ac9faec1a0184 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -2202,6 +2202,8 @@ config ARM64_POE
 	def_bool y
 	select ARCH_USES_HIGH_VMA_FLAGS
 	select ARCH_HAS_PKEYS
+	select ARCH_HAS_KPKEYS
+	select ARCH_HAS_KPKEYS_HARDENED_PGTABLES
 	help
 	  The Permission Overlay Extension is used to implement Memory
 	  Protection Keys. Memory Protection Keys provides a mechanism for
diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h
index a57870fa96db5..37aae60c2d7f0 100644
--- a/arch/arm64/include/asm/cpufeature.h
+++ b/arch/arm64/include/asm/cpufeature.h
@@ -1085,6 +1085,18 @@ static inline bool cpu_has_lpa2(void)
 #endif
 }
 
+static inline bool cpu_has_poe(void)
+{
+	u64 mmfr3;
+
+	if (!IS_ENABLED(CONFIG_ARM64_POE))
+		return false;
+
+	mmfr3 = read_sysreg_s(SYS_ID_AA64MMFR3_EL1);
+	return cpuid_feature_extract_unsigned_field(mmfr3,
+						    ID_AA64MMFR3_EL1_S1POE_SHIFT);
+}
+
 #endif /* __ASSEMBLER__ */
 
 #endif
diff --git a/arch/arm64/include/asm/kpkeys.h b/arch/arm64/include/asm/kpkeys.h
new file mode 100644
index 0000000000000..7747af54fd386
--- /dev/null
+++ b/arch/arm64/include/asm/kpkeys.h
@@ -0,0 +1,84 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef __ASM_KPKEYS_H
+#define __ASM_KPKEYS_H
+
+#include <linux/kpkeys_types.h>
+
+#include <asm/barrier.h>
+#include <asm/cpufeature.h>
+#include <asm/por.h>
+
+/*
+ * Equivalent to por_set_kpkeys_context(0, KPKEYS_CTX_DEFAULT), but can also be
+ * used in assembly.
+ */
+#define POR_EL1_INIT	(POR_ELx_PERM_PREP(0, POE_RWX) | \
+			 POR_ELx_PERM_PREP(KPKEYS_PKEY_PGTABLES, POE_R))
+
+#ifndef __ASSEMBLY__
+
+static inline bool arch_supports_kpkeys(void)
+{
+	return system_supports_poe();
+}
+
+static inline bool arch_supports_kpkeys_early(void)
+{
+	/* POE is a boot feature */
+	return boot_capabilities_finalized() ?
+		system_supports_poe() : cpu_has_poe();
+}
+
+#ifdef CONFIG_ARM64_POE
+
+static inline u64 por_set_kpkeys_context(u64 por, enum kpkeys_ctx ctx)
+{
+	por = por_elx_set_pkey_perms(por, 0, POE_RWX);
+	por = por_elx_set_pkey_perms(por, KPKEYS_PKEY_PGTABLES,
+				     ctx == KPKEYS_CTX_PGTABLES ? POE_RW : POE_R);
+
+	return por;
+}
+
+static __always_inline void __kpkeys_set_pkey_reg_nosync(u64 pkey_reg)
+{
+	write_sysreg_s(pkey_reg, SYS_POR_EL1);
+}
+
+static __always_inline
+struct kpkeys_state arch_kpkeys_enter_context(enum kpkeys_ctx ctx)
+{
+	const u64 prev_por = read_sysreg_s(SYS_POR_EL1);
+	const u64 new_por = por_set_kpkeys_context(prev_por, ctx);
+
+	if (prev_por == new_por)
+		return (struct kpkeys_state) { .entered_context = false };
+
+	__kpkeys_set_pkey_reg_nosync(new_por);
+	isb();
+
+	return (struct kpkeys_state) {
+		.entered_context = true,
+		.arch_state.por = prev_por,
+	};
+}
+
+static __always_inline
+void arch_kpkeys_leave_context(const struct kpkeys_state *state)
+{
+	__kpkeys_set_pkey_reg_nosync(state->arch_state.por);
+	isb();
+}
+
+#endif /* CONFIG_ARM64_POE */
+
+#ifdef CONFIG_KPKEYS_HARDENED_PGTABLES
+
+#define arch_kpkeys_protect_static_pgtables arch_kpkeys_protect_static_pgtables
+void arch_kpkeys_protect_static_pgtables(void);
+
+#endif /* CONFIG_KPKEYS_HARDENED_PGTABLES */
+
+#endif	/* __ASSEMBLY__ */
+
+#endif	/* __ASM_KPKEYS_H */
diff --git a/arch/arm64/include/asm/kpkeys_types.h b/arch/arm64/include/asm/kpkeys_types.h
new file mode 100644
index 0000000000000..e9c59a26ee7a9
--- /dev/null
+++ b/arch/arm64/include/asm/kpkeys_types.h
@@ -0,0 +1,15 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef __ASM_KPKEYS_TYPES_H
+#define __ASM_KPKEYS_TYPES_H
+
+#include <linux/types.h>
+
+#ifndef __ASSEMBLY__
+
+struct arch_kpkeys_state {
+	u64 por;
+};
+
+#endif	/* __ASSEMBLY__ */
+
+#endif	/* __ASM_KPKEYS_TYPES_H */
diff --git a/arch/arm64/include/asm/pgtable-prot.h b/arch/arm64/include/asm/pgtable-prot.h
index 212ce1b02e15e..d4d45ab86a5a8 100644
--- a/arch/arm64/include/asm/pgtable-prot.h
+++ b/arch/arm64/include/asm/pgtable-prot.h
@@ -183,9 +183,9 @@ static inline bool __pure lpa2_is_enabled(void)
 	PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_SHARED_EXEC),   PIE_RW)     | \
 	PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_READONLY),      PIE_R)      | \
 	PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_SHARED),        PIE_RW)     | \
-	PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL_ROX),    PIE_RX)     | \
-	PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL_EXEC),   PIE_RWX)    | \
-	PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL_RO),     PIE_R)      | \
-	PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL),        PIE_RW))
+	PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL_ROX),    PIE_RX_O)   | \
+	PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL_EXEC),   PIE_RWX_O)  | \
+	PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL_RO),     PIE_R_O)    | \
+	PIRx_ELx_PERM_PREP(pte_pi_index(_PAGE_KERNEL),        PIE_RW_O))
 
 #endif /* __ASM_PGTABLE_PROT_H */
diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index 27689c62bd25d..6e5d14c7e8b8f 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -39,6 +39,48 @@
 #include <linux/mm_types.h>
 #include <linux/sched.h>
 #include <linux/page_table_check.h>
+#include <linux/kpkeys.h>
+
+#ifdef CONFIG_KPKEYS_HARDENED_PGTABLES
+KPKEYS_GUARD_COND(kpkeys_hardened_pgtables, KPKEYS_CTX_PGTABLES,
+		  kpkeys_hardened_pgtables_enabled() &&
+		  !is_lazy_mmu_mode_active())
+
+static inline void kpkeys_lazy_mmu_enter(void)
+{
+	if (!kpkeys_hardened_pgtables_enabled())
+		return;
+
+	current->thread.kpkeys_state_lazy_mmu =
+		kpkeys_enter_context(KPKEYS_CTX_PGTABLES);
+}
+
+static inline void kpkeys_lazy_mmu_exit(void)
+{
+	const struct kpkeys_state *state =
+		&current->thread.kpkeys_state_lazy_mmu;
+
+	if (!kpkeys_hardened_pgtables_enabled())
+		return;
+
+	/*
+	 * We skip any barrier if TIF_LAZY_MMU_PENDING is set:
+	 * emit_pte_barriers() will issue an ISB just after this function
+	 * returns.
+	 */
+	if (test_thread_flag(TIF_LAZY_MMU_PENDING) && state->entered_context)
+		__kpkeys_set_pkey_reg_nosync(state->arch_state.por);
+	else
+		kpkeys_leave_context(state);
+}
+#else /* CONFIG_KPKEYS_HARDENED_PGTABLES */
+KPKEYS_GUARD_NOOP(kpkeys_hardened_pgtables)
+
+static inline void kpkeys_lazy_mmu_enter(void) {}
+static inline void kpkeys_lazy_mmu_exit(void) {}
+#endif /* CONFIG_KPKEYS_HARDENED_PGTABLES */
+
+
 
 static inline void emit_pte_barriers(void)
 {
@@ -71,7 +113,10 @@ static inline void queue_pte_barriers(void)
 	}
 }
 
-static inline void arch_enter_lazy_mmu_mode(void) {}
+static inline void arch_enter_lazy_mmu_mode(void)
+{
+	kpkeys_lazy_mmu_enter();
+}
 
 static inline void arch_flush_lazy_mmu_mode(void)
 {
@@ -81,6 +126,11 @@ static inline void arch_flush_lazy_mmu_mode(void)
 
 static inline void arch_leave_lazy_mmu_mode(void)
 {
+	/*
+	 * The ordering should be preserved to allow kpkeys_lazy_mmu_exit()
+	 * to skip any barrier when TIF_LAZY_MMU_PENDING is set.
+	 */
+	kpkeys_lazy_mmu_exit();
 	arch_flush_lazy_mmu_mode();
 }
 
@@ -359,6 +409,7 @@ static inline pte_t pte_clear_uffd_wp(pte_t pte)
 
 static inline void __set_pte_nosync(pte_t *ptep, pte_t pte)
 {
+	guard(kpkeys_hardened_pgtables)();
 	WRITE_ONCE(*ptep, pte);
 }
 
@@ -830,6 +881,7 @@ static inline void set_pmd(pmd_t *pmdp, pmd_t pmd)
 	}
 #endif /* __PAGETABLE_PMD_FOLDED */
 
+	guard(kpkeys_hardened_pgtables)();
 	WRITE_ONCE(*pmdp, pmd);
 
 	if (pmd_valid(pmd))
@@ -894,6 +946,7 @@ static inline void set_pud(pud_t *pudp, pud_t pud)
 		return;
 	}
 
+	guard(kpkeys_hardened_pgtables)();
 	WRITE_ONCE(*pudp, pud);
 
 	if (pud_valid(pud))
@@ -975,6 +1028,7 @@ static inline void set_p4d(p4d_t *p4dp, p4d_t p4d)
 		return;
 	}
 
+	guard(kpkeys_hardened_pgtables)();
 	WRITE_ONCE(*p4dp, p4d);
 	queue_pte_barriers();
 }
@@ -1103,6 +1157,7 @@ static inline void set_pgd(pgd_t *pgdp, pgd_t pgd)
 		return;
 	}
 
+	guard(kpkeys_hardened_pgtables)();
 	WRITE_ONCE(*pgdp, pgd);
 	queue_pte_barriers();
 }
@@ -1307,6 +1362,7 @@ static inline bool __ptep_test_and_clear_young(struct vm_area_struct *vma,
 {
 	pte_t old_pte, pte;
 
+	guard(kpkeys_hardened_pgtables)();
 	pte = __ptep_get(ptep);
 	do {
 		old_pte = pte;
@@ -1354,7 +1410,10 @@ static inline pte_t __ptep_get_and_clear_anysz(struct mm_struct *mm,
 					       pte_t *ptep,
 					       unsigned long pgsize)
 {
-	pte_t pte = __pte(xchg_relaxed(&pte_val(*ptep), 0));
+	pte_t pte;
+
+	scoped_guard(kpkeys_hardened_pgtables)
+		pte = __pte(xchg_relaxed(&pte_val(*ptep), 0));
 
 	switch (pgsize) {
 	case PAGE_SIZE:
@@ -1427,6 +1486,7 @@ static inline void ___ptep_set_wrprotect(struct mm_struct *mm,
 {
 	pte_t old_pte;
 
+	guard(kpkeys_hardened_pgtables)();
 	do {
 		old_pte = pte;
 		pte = pte_wrprotect(pte);
@@ -1460,6 +1520,7 @@ static inline void __clear_young_dirty_pte(struct vm_area_struct *vma,
 {
 	pte_t old_pte;
 
+	guard(kpkeys_hardened_pgtables)();
 	do {
 		old_pte = pte;
 
@@ -1507,6 +1568,7 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma,
 		unsigned long address, pmd_t *pmdp, pmd_t pmd)
 {
 	page_table_check_pmd_set(vma->vm_mm, address, pmdp, pmd);
+	guard(kpkeys_hardened_pgtables)();
 	return __pmd(xchg_relaxed(&pmd_val(*pmdp), pmd_val(pmd)));
 }
 #endif
diff --git a/arch/arm64/include/asm/por.h b/arch/arm64/include/asm/por.h
index d913d5b529e49..dddabf6ffae6d 100644
--- a/arch/arm64/include/asm/por.h
+++ b/arch/arm64/include/asm/por.h
@@ -10,6 +10,8 @@
 
 #define POR_EL0_INIT	POR_ELx_PERM_PREP(0, POE_RWX)
 
+#ifndef __ASSEMBLY__
+
 static inline bool por_elx_allows_read(u64 por, u8 pkey)
 {
 	u8 perm = POR_ELx_PERM_GET(pkey, por);
@@ -31,4 +33,13 @@ static inline bool por_elx_allows_exec(u64 por, u8 pkey)
 	return perm & POE_X;
 }
 
+static inline u64 por_elx_set_pkey_perms(u64 por, u8 pkey, u8 perms)
+{
+	u64 shift = POR_ELx_PERM_SHIFT(pkey);
+
+	return (por & ~(POE_MASK << shift)) | ((u64)perms << shift);
+}
+
+#endif	/* __ASSEMBLY__ */
+
 #endif /* _ASM_ARM64_POR_H */
diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h
index c2a627f393144..23660a233da28 100644
--- a/arch/arm64/include/asm/processor.h
+++ b/arch/arm64/include/asm/processor.h
@@ -30,6 +30,7 @@
 #include <linux/build_bug.h>
 #include <linux/cache.h>
 #include <linux/init.h>
+#include <linux/kpkeys_types.h>
 #include <linux/stddef.h>
 #include <linux/string.h>
 #include <linux/thread_info.h>
@@ -195,6 +196,10 @@ struct thread_struct {
 	u64			svcr;
 	u64			tpidr2_el0;
 	u64			por_el0;
+	u64			por_el1;
+#ifdef CONFIG_KPKEYS_HARDENED_PGTABLES
+	struct kpkeys_state	kpkeys_state_lazy_mmu;
+#endif
 #ifdef CONFIG_ARM64_GCS
 	unsigned int		gcs_el0_mode;
 	unsigned int		gcs_el0_locked;
diff --git a/arch/arm64/include/asm/sections.h b/arch/arm64/include/asm/sections.h
index 51b0d594239eb..dc961cf67168c 100644
--- a/arch/arm64/include/asm/sections.h
+++ b/arch/arm64/include/asm/sections.h
@@ -23,6 +23,7 @@ extern char __irqentry_text_start[], __irqentry_text_end[];
 extern char __mmuoff_data_start[], __mmuoff_data_end[];
 extern char __entry_tramp_text_start[], __entry_tramp_text_end[];
 extern char __relocate_new_kernel_start[], __relocate_new_kernel_end[];
+extern char __pi_init_pg_dir[], __pi_init_pg_end[];
 
 static inline size_t entry_tramp_text_size(void)
 {
diff --git a/arch/arm64/include/asm/set_memory.h b/arch/arm64/include/asm/set_memory.h
index 90f61b17275e1..b6cd6de34abf1 100644
--- a/arch/arm64/include/asm/set_memory.h
+++ b/arch/arm64/include/asm/set_memory.h
@@ -19,4 +19,8 @@ bool kernel_page_present(struct page *page);
 int set_memory_encrypted(unsigned long addr, int numpages);
 int set_memory_decrypted(unsigned long addr, int numpages);
 
+#ifdef CONFIG_ARCH_HAS_KPKEYS
+int set_memory_pkey(unsigned long addr, int numpages, int pkey);
+#endif
+
 #endif /* _ASM_ARM64_SET_MEMORY_H */
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index 9a22df0c5120f..4f2f54007ed6f 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -76,6 +76,7 @@
 #include <linux/kasan.h>
 #include <linux/percpu.h>
 #include <linux/sched/isolation.h>
+#include <linux/kpkeys.h>
 
 #include <asm/arm_pmuv3.h>
 #include <asm/cpu.h>
@@ -2463,8 +2464,10 @@ static void cpu_enable_mops(const struct arm64_cpu_capabilities *__unused)
 #ifdef CONFIG_ARM64_POE
 static void cpu_enable_poe(const struct arm64_cpu_capabilities *__unused)
 {
-	sysreg_clear_set(REG_TCR2_EL1, 0, TCR2_EL1_E0POE);
+	write_sysreg_s(POR_EL1_INIT, SYS_POR_EL1);
+	sysreg_clear_set(REG_TCR2_EL1, 0, TCR2_EL1_E0POE | TCR2_EL1_POE);
 	sysreg_clear_set(CPACR_EL1, 0, CPACR_EL1_E0POE);
+	isb();
 }
 #endif
 
diff --git a/arch/arm64/kernel/pi/map_kernel.c b/arch/arm64/kernel/pi/map_kernel.c
index fb44cbdd2f292..81ca3e98f0f7b 100644
--- a/arch/arm64/kernel/pi/map_kernel.c
+++ b/arch/arm64/kernel/pi/map_kernel.c
@@ -92,8 +92,13 @@ static void __init map_kernel(u64 kaslr_offset, u64 va_offset, int root_level)
 		    __inittext_end, prot, false, root_level);
 	map_segment(init_pg_dir, &pgdp, va_offset, __initdata_begin,
 		    __initdata_end, data_prot, false, root_level);
-	map_segment(init_pg_dir, &pgdp, va_offset, _data, _end, data_prot,
-		    true, root_level);
+	map_segment(init_pg_dir, &pgdp, va_offset, _data, init_pg_dir,
+		    data_prot, true, root_level);
+	/* Map init_pg_dir separately as we will remap it read-only later on */
+	map_segment(init_pg_dir, &pgdp, va_offset, init_pg_dir, init_pg_end,
+		    data_prot, true, root_level);
+	map_segment(init_pg_dir, &pgdp, va_offset, init_pg_end, _end,
+		    data_prot, true, root_level);
 	dsb(ishst);
 
 	idmap_cpu_replace_ttbr1((phys_addr_t)init_pg_dir);
diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c
index 581f80e9b9b73..1dec143decf15 100644
--- a/arch/arm64/kernel/process.c
+++ b/arch/arm64/kernel/process.c
@@ -51,6 +51,7 @@
 #include <asm/exec.h>
 #include <asm/fpsimd.h>
 #include <asm/gcs.h>
+#include <asm/kpkeys.h>
 #include <asm/mmu_context.h>
 #include <asm/mpam.h>
 #include <asm/mte.h>
@@ -431,6 +432,9 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args)
 
 	ptrauth_thread_init_kernel(p);
 
+	if (system_supports_poe())
+		p->thread.por_el1 = POR_EL1_INIT;
+
 	if (likely(!args->fn)) {
 		*childregs = *current_pt_regs();
 		childregs->regs[0] = 0;
@@ -681,6 +685,12 @@ static void permission_overlay_switch(struct task_struct *next)
 		 * of POR_EL0.
 		 */
 	}
+
+	current->thread.por_el1 = read_sysreg_s(SYS_POR_EL1);
+	if (current->thread.por_el1 != next->thread.por_el1) {
+		write_sysreg_s(next->thread.por_el1, SYS_POR_EL1);
+		isb();
+	}
 }
 
 /*
diff --git a/arch/arm64/kernel/sleep.S b/arch/arm64/kernel/sleep.S
index f093cdf71be11..e0a6ad85cd242 100644
--- a/arch/arm64/kernel/sleep.S
+++ b/arch/arm64/kernel/sleep.S
@@ -3,6 +3,7 @@
 #include <linux/linkage.h>
 #include <asm/asm-offsets.h>
 #include <asm/assembler.h>
+#include <asm/kpkeys.h>
 #include <asm/smp.h>
 
 	.text
@@ -134,6 +135,17 @@ SYM_FUNC_START(_cpu_resume)
 	/* load sp from context */
 	ldr	x2, [x0, #CPU_CTX_SP]
 	mov	sp, x2
+
+#ifdef CONFIG_ARM64_POE
+alternative_if_not ARM64_HAS_S1POE
+	b	.Lskip_por_set
+alternative_else_nop_endif
+	mov_q	x2, POR_EL1_INIT
+	msr_s	SYS_POR_EL1, x2
+	/* isb can be skipped since cpu_do_resume() will do it. */
+.Lskip_por_set:
+#endif /* CONFIG_ARM64_POE */
+
 	/*
 	 * cpu_do_resume expects x0 to contain context address pointer
 	 */
diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
index 0b52557652be6..a853a8d4e61af 100644
--- a/arch/arm64/mm/fault.c
+++ b/arch/arm64/mm/fault.c
@@ -229,13 +229,16 @@ int __ptep_set_access_flags_anysz(struct vm_area_struct *vma,
 	 */
 	pte_val(entry) ^= PTE_RDONLY;
 	pteval = pte_val(pte);
-	do {
-		old_pteval = pteval;
-		pteval ^= PTE_RDONLY;
-		pteval |= pte_val(entry);
-		pteval ^= PTE_RDONLY;
-		pteval = cmpxchg_relaxed(&pte_val(*ptep), old_pteval, pteval);
-	} while (pteval != old_pteval);
+	scoped_guard(kpkeys_hardened_pgtables) {
+		do {
+			old_pteval = pteval;
+			pteval ^= PTE_RDONLY;
+			pteval |= pte_val(entry);
+			pteval ^= PTE_RDONLY;
+			pteval = cmpxchg_relaxed(&pte_val(*ptep), old_pteval,
+						 pteval);
+		} while (pteval != old_pteval);
+	}
 
 	/*
 	 * Invalidate the local stale read-only entry.  Remote stale entries
diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
index 97987f850a33c..7985045875cf6 100644
--- a/arch/arm64/mm/init.c
+++ b/arch/arm64/mm/init.c
@@ -386,6 +386,7 @@ bool page_alloc_available __ro_after_init;
 void __init mem_init(void)
 {
 	page_alloc_available = true;
+	kpkeys_hardened_pgtables_init();
 	swiotlb_update_mem_attributes();
 }
 
diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index d4de88770ecfe..472ad0f8f002c 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -107,8 +107,11 @@ static phys_addr_t __init early_pgtable_alloc(enum pgtable_level pgtable_level)
 {
 	phys_addr_t phys;
 
-	phys = memblock_phys_alloc_range(PAGE_SIZE, PAGE_SIZE, 0,
-					 MEMBLOCK_ALLOC_NOLEAKTRACE);
+	if (kpkeys_hardened_pgtables_early_enabled())
+		phys = kpkeys_physmem_pgtable_alloc();
+	else
+		phys = memblock_phys_alloc_range(PAGE_SIZE, PAGE_SIZE, 0,
+						 MEMBLOCK_ALLOC_NOLEAKTRACE);
 	if (!phys)
 		panic("Failed to allocate page table page\n");
 
@@ -780,7 +783,7 @@ static inline bool force_pte_mapping(void)
 	const bool bbml2 = system_capabilities_finalized() ?
 		system_supports_bbml2_noabort() : cpu_supports_bbml2_noabort();
 
-	if (debug_pagealloc_enabled())
+	if (debug_pagealloc_enabled() || kpkeys_hardened_pgtables_early_enabled())
 		return true;
 	if (bbml2)
 		return false;
@@ -1070,6 +1073,18 @@ void __init mark_linear_text_alias_ro(void)
 			    PAGE_KERNEL_RO);
 }
 
+#ifdef CONFIG_KPKEYS_HARDENED_PGTABLES
+void __init arch_kpkeys_protect_static_pgtables(void)
+{
+	unsigned long addr = (unsigned long)lm_alias(__pi_init_pg_dir);
+	unsigned long size = __pi_init_pg_end - __pi_init_pg_dir;
+	int ret;
+
+	ret = set_memory_pkey(addr, size / PAGE_SIZE, KPKEYS_PKEY_PGTABLES);
+	WARN_ON(ret);
+}
+#endif /* CONFIG_KPKEYS_HARDENED_PGTABLES */
+
 #ifdef CONFIG_KFENCE
 
 bool __ro_after_init kfence_early_init = !!CONFIG_KFENCE_SAMPLE_INTERVAL;
@@ -1214,6 +1229,14 @@ void mark_rodata_ro(void)
 	update_mapping_prot(__pa_symbol(_text), (unsigned long)_text,
 			    (unsigned long)_stext - (unsigned long)_text,
 			    PAGE_KERNEL_RO);
+	/*
+	 * Map the kernel image mapping of init_pg_dir read-only; it should
+	 * only be written via the linear map.
+	 */
+	section_size = (unsigned long)__pi_init_pg_end - (unsigned long)__pi_init_pg_dir;
+	update_mapping_prot(__pa_symbol(__pi_init_pg_dir),
+			    (unsigned long)__pi_init_pg_dir,
+			    section_size, PAGE_KERNEL_RO);
 
 	/* Map the kernel data/bss read-only in the linear map */
 	update_mapping_prot(__pa_symbol(__init_end),
@@ -1446,7 +1469,7 @@ static void free_hotplug_page_range(struct page *page, size_t size,
 static void free_hotplug_pgtable_page(struct page *page)
 {
 	pagetable_dtor(page_ptdesc(page));
-	free_hotplug_page_range(page, PAGE_SIZE, NULL);
+	pagetable_free(page_ptdesc(page));
 }
 
 static bool pgtable_range_aligned(unsigned long start, unsigned long end,
@@ -2305,8 +2328,8 @@ void __cpu_replace_ttbr1(pgd_t *pgdp, bool cnp)
 #ifdef CONFIG_ARCH_HAS_PKEYS
 int arch_set_user_pkey_access(int pkey, unsigned long init_val)
 {
-	u64 new_por;
-	u64 old_por;
+	u8 new_perms;
+	u64 por;
 
 	if (!system_supports_poe())
 		return -ENOSPC;
@@ -2320,25 +2343,19 @@ int arch_set_user_pkey_access(int pkey, unsigned long init_val)
 		return -EINVAL;
 
 	/* Set the bits we need in POR:  */
-	new_por = POE_RWX;
+	new_perms = POE_RWX;
 	if (init_val & PKEY_DISABLE_WRITE)
-		new_por &= ~POE_W;
+		new_perms &= ~POE_W;
 	if (init_val & PKEY_DISABLE_ACCESS)
-		new_por &= ~POE_RW;
+		new_perms &= ~POE_RW;
 	if (init_val & PKEY_DISABLE_READ)
-		new_por &= ~POE_R;
+		new_perms &= ~POE_R;
 	if (init_val & PKEY_DISABLE_EXECUTE)
-		new_por &= ~POE_X;
-
-	/* Shift the bits in to the correct place in POR for pkey: */
-	new_por = POR_ELx_PERM_PREP(pkey, new_por);
-
-	/* Get old POR and mask off any old bits in place: */
-	old_por = read_sysreg_s(SYS_POR_EL0);
-	old_por &= ~(POE_MASK << POR_ELx_PERM_SHIFT(pkey));
+		new_perms &= ~POE_X;
 
-	/* Write old part along with new part: */
-	write_sysreg_s(old_por | new_por, SYS_POR_EL0);
+	por = read_sysreg_s(SYS_POR_EL0);
+	por = por_elx_set_pkey_perms(por, pkey, new_perms);
+	write_sysreg_s(por, SYS_POR_EL0);
 
 	return 0;
 }
diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c
index bbe98ac9ad8c6..46fbebfc51003 100644
--- a/arch/arm64/mm/pageattr.c
+++ b/arch/arm64/mm/pageattr.c
@@ -9,6 +9,8 @@
 #include <linux/sched.h>
 #include <linux/vmalloc.h>
 #include <linux/pagewalk.h>
+#include <linux/pkeys.h>
+#include <linux/kpkeys.h>
 
 #include <asm/cacheflush.h>
 #include <asm/pgtable-prot.h>
@@ -99,7 +101,8 @@ bool can_set_direct_map(void)
 	 * Realms need to make pages shared/protected at page granularity.
 	 */
 	return rodata_full || debug_pagealloc_enabled() ||
-		arm64_kfence_can_set_direct_map() || is_realm_world();
+		arm64_kfence_can_set_direct_map() || is_realm_world() ||
+		kpkeys_hardened_pgtables_early_enabled();
 }
 
 static int update_range_prot(unsigned long start, unsigned long size,
@@ -365,6 +368,30 @@ int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool valid)
 	return set_memory_valid(addr, nr, valid);
 }
 
+#ifdef CONFIG_ARCH_HAS_KPKEYS
+int set_memory_pkey(unsigned long addr, int numpages, int pkey)
+{
+	unsigned long set_prot = 0;
+
+	if (!kpkeys_supported())
+		return 0;
+
+	if (!__is_lm_address(kasan_reset_tag((void *)addr)))
+		return -EINVAL;
+
+	if (pkey < 0 || pkey >= arch_max_pkey())
+		return -EINVAL;
+
+	set_prot |= pkey & BIT(0) ? PTE_PO_IDX_0 : 0;
+	set_prot |= pkey & BIT(1) ? PTE_PO_IDX_1 : 0;
+	set_prot |= pkey & BIT(2) ? PTE_PO_IDX_2 : 0;
+
+	return __change_memory_common(addr, PAGE_SIZE * numpages,
+				      __pgprot(set_prot),
+				      __pgprot(PTE_PO_IDX_MASK));
+}
+#endif
+
 #ifdef CONFIG_DEBUG_PAGEALLOC
 /*
  * This is - apart from the return value - doing the same
diff --git a/include/linux/kpkeys.h b/include/linux/kpkeys.h
new file mode 100644
index 0000000000000..df55ec854586e
--- /dev/null
+++ b/include/linux/kpkeys.h
@@ -0,0 +1,162 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef _LINUX_KPKEYS_H
+#define _LINUX_KPKEYS_H
+
+#include <linux/bug.h>
+#include <linux/cleanup.h>
+#include <linux/jump_label.h>
+#include <linux/kpkeys_types.h>
+
+/**
+ * KPKEYS_GUARD_NOOP() - define a guard type that does nothing
+ * @name: the name of the guard type
+ *
+ * Define a guard type that does nothing, useful to match a real guard type
+ * that is defined under an #ifdef.
+ */
+#define KPKEYS_GUARD_NOOP(name)						\
+	__DEFINE_CLASS_IS_CONDITIONAL(name, false);			\
+	DEFINE_CLASS(name, bool, (void)_T, false, void);		\
+	static inline void *class_##name##_lock_ptr(bool *_T)		\
+	{ return _T; }
+
+#ifdef CONFIG_ARCH_HAS_KPKEYS
+
+#include <asm/kpkeys.h>
+
+/**
+ * KPKEYS_GUARD_COND() - define a guard type that conditionally switches to
+ *                       a given kpkeys context
+ * @name: the name of the guard type
+ * @ctx: the kpkeys context to switch to
+ * @cond: an expression that is evaluated as condition
+ *
+ * Define a guard type that switches to @ctx if @cond evaluates to true,
+ * and does nothing otherwise.
+ */
+#define KPKEYS_GUARD_COND(name, ctx, cond)				\
+	__DEFINE_CLASS_IS_CONDITIONAL(name, false);			\
+	DEFINE_CLASS(name, struct kpkeys_state,				\
+		     kpkeys_leave_context(&_T),				\
+		     (cond) ? kpkeys_enter_context(ctx) :		\
+			      (struct kpkeys_state) {}, void);		\
+	static inline							\
+	void *class_##name##_lock_ptr(struct kpkeys_state *_T)		\
+	{ return _T; }
+
+/**
+ * KPKEYS_GUARD() - define a guard type that switches to a given kpkeys context
+ *                  if kpkeys are supported
+ * @name: the name of the guard type
+ * @ctx: the kpkeys context to switch to
+ *
+ * Define a guard type that switches to @ctx if the system supports kpkeys.
+ */
+#define KPKEYS_GUARD(name, ctx)						\
+	KPKEYS_GUARD_COND(name, ctx, kpkeys_supported())
+
+/**
+ * kpkeys_enter_context() - enter a kpkeys context
+ * @ctx: the context to switch to
+ *
+ * Enters the specified kpkeys context. @ctx must be a compile-time constant.
+ *
+ * Return: state to be passed to kpkeys_leave_context().
+ */
+static __always_inline
+struct kpkeys_state kpkeys_enter_context(enum kpkeys_ctx ctx)
+{
+	BUILD_BUG_ON_MSG(!__builtin_constant_p(ctx),
+			 "kpkeys_enter_context() only takes constant values");
+	BUILD_BUG_ON_MSG(ctx < 0 || ctx >= KPKEYS_CTX_COUNT,
+			 "Invalid value passed to kpkeys_enter_context()");
+
+	return arch_kpkeys_enter_context(ctx);
+}
+
+/**
+ * kpkeys_leave_context() - leave a kpkeys context
+ * @state: state returned by kpkeys_enter_context()
+ *
+ * Restores the state saved when entering a kpkeys context. If no context was
+ * entered, this function does nothing.
+ */
+static __always_inline
+void kpkeys_leave_context(const struct kpkeys_state *state)
+{
+	if (state->entered_context)
+		arch_kpkeys_leave_context(state);
+}
+
+static inline bool kpkeys_supported(void)
+{
+	return arch_supports_kpkeys();
+}
+
+#else /* CONFIG_ARCH_HAS_KPKEYS */
+
+static inline bool kpkeys_supported(void)
+{
+	return false;
+}
+
+#endif /* CONFIG_ARCH_HAS_KPKEYS */
+
+#ifdef CONFIG_KPKEYS_HARDENED_PGTABLES
+
+DECLARE_STATIC_KEY_FALSE(kpkeys_hardened_pgtables_key);
+
+static inline bool kpkeys_hardened_pgtables_enabled(void)
+{
+	return static_branch_unlikely(&kpkeys_hardened_pgtables_key);
+}
+
+static inline bool kpkeys_hardened_pgtables_early_enabled(void)
+{
+	return arch_supports_kpkeys_early();
+}
+
+struct page *kpkeys_pgtable_alloc(gfp_t gfp, unsigned int order);
+void kpkeys_pgtable_free(struct page *page, unsigned int order);
+
+/*
+ * Should be called from mem_init(): as soon as the buddy allocator becomes
+ * available and before any call to pagetable_alloc().
+ */
+void kpkeys_hardened_pgtables_init(void);
+
+phys_addr_t kpkeys_physmem_pgtable_alloc(void);
+
+#ifndef arch_kpkeys_protect_static_pgtables
+static inline void arch_kpkeys_protect_static_pgtables(void) {}
+#endif
+
+#else /* CONFIG_KPKEYS_HARDENED_PGTABLES */
+
+static inline bool kpkeys_hardened_pgtables_enabled(void)
+{
+	return false;
+}
+
+static inline bool kpkeys_hardened_pgtables_early_enabled(void)
+{
+	return false;
+}
+
+static inline struct page *kpkeys_pgtable_alloc(gfp_t gfp, unsigned int order)
+{
+	return NULL;
+}
+
+static inline void kpkeys_pgtable_free(struct page *page, unsigned int order) {}
+
+static inline void kpkeys_hardened_pgtables_init(void) {}
+
+static inline phys_addr_t kpkeys_physmem_pgtable_alloc(void)
+{
+	return 0;
+}
+
+#endif /* CONFIG_KPKEYS_HARDENED_PGTABLES */
+
+#endif /* _LINUX_KPKEYS_H */
diff --git a/include/linux/kpkeys_types.h b/include/linux/kpkeys_types.h
new file mode 100644
index 0000000000000..09f15a9565cf3
--- /dev/null
+++ b/include/linux/kpkeys_types.h
@@ -0,0 +1,30 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef _LINUX_KPKEYS_TYPES_H
+#define _LINUX_KPKEYS_TYPES_H
+
+#include <linux/types.h>
+
+#ifdef CONFIG_ARCH_HAS_KPKEYS
+#include <asm/kpkeys_types.h>
+#endif
+
+#ifndef KPKEYS_PKEY_PGTABLES
+#define KPKEYS_PKEY_PGTABLES	1
+#endif
+
+#if defined(CONFIG_ARCH_HAS_KPKEYS) && !defined(__ASSEMBLY__)
+
+enum kpkeys_ctx {
+	KPKEYS_CTX_DEFAULT = 0,
+	KPKEYS_CTX_PGTABLES,
+	KPKEYS_CTX_COUNT,
+};
+
+struct kpkeys_state {
+	bool entered_context;
+	struct arch_kpkeys_state arch_state;
+};
+
+#endif /* CONFIG_ARCH_HAS_KPKEYS && !__ASSEMBLY__ */
+
+#endif /* _LINUX_KPKEYS_TYPES_H */
diff --git a/include/linux/memblock.h b/include/linux/memblock.h
index 5afcd99aa8c14..c61f80e31cc10 100644
--- a/include/linux/memblock.h
+++ b/include/linux/memblock.h
@@ -24,6 +24,17 @@ extern unsigned long max_pfn;
  */
 extern unsigned long long max_possible_pfn;
 
+#define INIT_MEMBLOCK_REGIONS			128
+#define INIT_PHYSMEM_REGIONS			4
+
+#ifndef INIT_MEMBLOCK_RESERVED_REGIONS
+#define INIT_MEMBLOCK_RESERVED_REGIONS		INIT_MEMBLOCK_REGIONS
+#endif
+
+#ifndef INIT_MEMBLOCK_MEMORY_REGIONS
+#define INIT_MEMBLOCK_MEMORY_REGIONS		INIT_MEMBLOCK_REGIONS
+#endif
+
 /**
  * enum memblock_flags - definition of memory region attributes
  * @MEMBLOCK_NONE: no special request
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 485df9c2dbddb..eea3131786b35 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -37,6 +37,7 @@
 #include <linux/bitmap.h>
 #include <linux/bitops.h>
 #include <linux/iommu-debug-pagealloc.h>
+#include <linux/kpkeys.h>
 
 struct mempolicy;
 struct anon_vma;
@@ -3682,7 +3683,12 @@ static inline bool ptdesc_test_kernel(const struct ptdesc *ptdesc)
  */
 static inline struct ptdesc *pagetable_alloc_noprof(gfp_t gfp, unsigned int order)
 {
-	struct page *page = alloc_pages_noprof(gfp | __GFP_COMP, order);
+	struct page *page;
+
+	if (kpkeys_hardened_pgtables_enabled())
+		page = kpkeys_pgtable_alloc(gfp | __GFP_COMP, order);
+	else
+		page = alloc_pages_noprof(gfp | __GFP_COMP, order);
 
 	return page_ptdesc(page);
 }
@@ -3691,8 +3697,12 @@ static inline struct ptdesc *pagetable_alloc_noprof(gfp_t gfp, unsigned int orde
 static inline void __pagetable_free(struct ptdesc *pt)
 {
 	struct page *page = ptdesc_page(pt);
+	unsigned int order = compound_order(page);
 
-	__free_pages(page, compound_order(page));
+	if (kpkeys_hardened_pgtables_enabled())
+		kpkeys_pgtable_free(page, order);
+	else
+		__free_pages(page, order);
 }
 
 #ifdef CONFIG_ASYNC_KERNEL_PGTABLE_FREE
diff --git a/include/linux/set_memory.h b/include/linux/set_memory.h
index 3030d9245f5ac..7b3a8bfde3c64 100644
--- a/include/linux/set_memory.h
+++ b/include/linux/set_memory.h
@@ -84,4 +84,11 @@ static inline int set_memory_decrypted(unsigned long addr, int numpages)
 }
 #endif /* CONFIG_ARCH_HAS_MEM_ENCRYPT */
 
+#ifndef CONFIG_ARCH_HAS_KPKEYS
+static inline int set_memory_pkey(unsigned long addr, int numpages, int pkey)
+{
+	return 0;
+}
+#endif
+
 #endif /* _LINUX_SET_MEMORY_H_ */
diff --git a/mm/Kconfig b/mm/Kconfig
index 9e0ca48249054..21bd3e550de83 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -1281,6 +1281,11 @@ config ARCH_USES_HIGH_VMA_FLAGS
 	bool
 config ARCH_HAS_PKEYS
 	bool
+config ARCH_HAS_KPKEYS
+	bool
+config ARCH_HAS_KPKEYS_HARDENED_PGTABLES
+	bool
+	depends on ARCH_HAS_KPKEYS
 
 config ARCH_USES_PG_ARCH_2
 	bool
diff --git a/mm/Makefile b/mm/Makefile
index eff9f9e7e061c..e31e582948451 100644
--- a/mm/Makefile
+++ b/mm/Makefile
@@ -147,3 +147,5 @@ obj-$(CONFIG_SHRINKER_DEBUG) += shrinker_debug.o
 obj-$(CONFIG_EXECMEM) += execmem.o
 obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o
 obj-$(CONFIG_LAZY_MMU_MODE_KUNIT_TEST) += tests/lazy_mmu_mode_kunit.o
+obj-$(CONFIG_KPKEYS_HARDENED_PGTABLES) += kpkeys_hardened_pgtables.o
+obj-$(CONFIG_KPKEYS_HARDENED_PGTABLES_KUNIT_TEST) += tests/kpkeys_hardened_pgtables_kunit.o
diff --git a/mm/kpkeys_hardened_pgtables.c b/mm/kpkeys_hardened_pgtables.c
new file mode 100644
index 0000000000000..c133e5896fccc
--- /dev/null
+++ b/mm/kpkeys_hardened_pgtables.c
@@ -0,0 +1,179 @@
+// SPDX-License-Identifier: GPL-2.0-only
+#include <linux/kpkeys.h>
+#include <linux/memblock.h>
+#include <linux/mm.h>
+#include <linux/set_memory.h>
+
+#include <kunit/visibility.h>
+
+__ro_after_init DEFINE_STATIC_KEY_FALSE(kpkeys_hardened_pgtables_key);
+EXPORT_SYMBOL_IF_KUNIT(kpkeys_hardened_pgtables_key);
+
+static int set_pkey_pgtable(struct page *page, unsigned int nr_pages)
+{
+	unsigned long addr = (unsigned long)page_address(page);
+	int ret;
+
+	ret = set_memory_pkey(addr, nr_pages, KPKEYS_PKEY_PGTABLES);
+
+	WARN_ON(ret);
+	return ret;
+}
+
+static int set_pkey_default(struct page *page, unsigned int nr_pages)
+{
+	unsigned long addr = (unsigned long)page_address(page);
+	int ret;
+
+	ret = set_memory_pkey(addr, nr_pages, 0);
+
+	WARN_ON(ret);
+	return ret;
+}
+
+/* pkeys physmem allocator (PPA) - implemented below */
+static void ppa_finalize(void);
+
+struct page *kpkeys_pgtable_alloc(gfp_t gfp, unsigned int order)
+{
+	struct page *page;
+	int ret;
+
+	page = alloc_pages_noprof(gfp, order);
+	if (!page)
+		return page;
+
+	ret = set_pkey_pgtable(page, 1 << order);
+	if (ret) {
+		__free_pages(page, order);
+		return NULL;
+	}
+
+	return page;
+}
+
+void kpkeys_pgtable_free(struct page *page, unsigned int order)
+{
+	set_pkey_default(page, 1 << order);
+	__free_pages(page, order);
+}
+
+void __init kpkeys_hardened_pgtables_init(void)
+{
+	if (!kpkeys_supported())
+		return;
+
+	static_branch_enable(&kpkeys_hardened_pgtables_key);
+
+	ppa_finalize();
+	arch_kpkeys_protect_static_pgtables();
+}
+
+/*
+ * pkeys physmem allocator (PPA): allocator for very early page tables
+ * (especially for creating the linear map), based on memblock. Allocated
+ * ranges are tracked so that their pkey can be set once it is safe to do so.
+ */
+
+/*
+ * We may have to track many ranges when allocating page tables for the linear
+ * map, as their number grows with the amount of available memory. Assuming that
+ * memblock returns contiguous blocks whenever possible, the number of ranges
+ * to track cannot however exceed the number of regions that memblock itself
+ * tracks. memblock_allow_resize() hasn't been called yet at that point, so
+ * that limit is the size of the statically allocated array.
+ */
+#define PHYSMEM_MAX_RANGES	INIT_MEMBLOCK_MEMORY_REGIONS
+
+struct physmem_range {
+	phys_addr_t addr;
+	phys_addr_t size;
+};
+
+struct pkeys_physmem_allocator {
+	struct physmem_range allocated_ranges[PHYSMEM_MAX_RANGES];
+	unsigned int nr_allocated_ranges;
+};
+
+static struct pkeys_physmem_allocator pkeys_physmem_allocator __initdata;
+
+static int __init set_pkey_pgtable_phys(phys_addr_t pa, phys_addr_t size)
+{
+	unsigned long addr = (unsigned long)__va(pa);
+	int ret;
+
+	ret = set_memory_pkey(addr, size / PAGE_SIZE, KPKEYS_PKEY_PGTABLES);
+
+	WARN_ON(ret);
+	return ret;
+}
+
+static bool __init ppa_try_extend_last_range(phys_addr_t addr, phys_addr_t size)
+{
+	struct pkeys_physmem_allocator *ppa = &pkeys_physmem_allocator;
+	struct physmem_range *range;
+
+	if (!ppa->nr_allocated_ranges)
+		return false;
+
+	range = &ppa->allocated_ranges[ppa->nr_allocated_ranges - 1];
+
+	/* Merge the new range into the last range if they are contiguous */
+	if (addr == range->addr + range->size) {
+		range->size += size;
+		return true;
+	} else if (addr + size == range->addr) {
+		range->addr -= size;
+		range->size += size;
+		return true;
+	}
+
+	return false;
+}
+
+static void __init ppa_register_allocated_range(phys_addr_t addr,
+						phys_addr_t size)
+{
+	struct pkeys_physmem_allocator *ppa = &pkeys_physmem_allocator;
+	struct physmem_range *range;
+
+	if (!addr)
+		return;
+
+	if (ppa_try_extend_last_range(addr, size))
+		return;
+
+	/* Could not extend the last range, create a new one */
+	if (WARN_ON(ppa->nr_allocated_ranges >= PHYSMEM_MAX_RANGES))
+		return;
+
+	range = &ppa->allocated_ranges[ppa->nr_allocated_ranges++];
+	range->addr = addr;
+	range->size = size;
+}
+
+static void __init ppa_finalize(void)
+{
+	struct pkeys_physmem_allocator *ppa = &pkeys_physmem_allocator;
+
+	for (unsigned int i = 0; i < ppa->nr_allocated_ranges; i++) {
+		struct physmem_range *range = &ppa->allocated_ranges[i];
+
+		set_pkey_pgtable_phys(range->addr, range->size);
+	}
+}
+
+phys_addr_t __ref kpkeys_physmem_pgtable_alloc(void)
+{
+	size_t size = PAGE_SIZE;
+	phys_addr_t addr;
+
+	addr = memblock_phys_alloc_range(size, size, 0,
+					 MEMBLOCK_ALLOC_NOLEAKTRACE);
+	if (!addr)
+		return addr;
+
+	ppa_register_allocated_range(addr, size);
+
+	return addr;
+}
diff --git a/mm/memblock.c b/mm/memblock.c
index 6349c48154f4b..3b497d3556bb0 100644
--- a/mm/memblock.c
+++ b/mm/memblock.c
@@ -30,17 +30,6 @@
 
 #include "internal.h"
 
-#define INIT_MEMBLOCK_REGIONS			128
-#define INIT_PHYSMEM_REGIONS			4
-
-#ifndef INIT_MEMBLOCK_RESERVED_REGIONS
-# define INIT_MEMBLOCK_RESERVED_REGIONS		INIT_MEMBLOCK_REGIONS
-#endif
-
-#ifndef INIT_MEMBLOCK_MEMORY_REGIONS
-#define INIT_MEMBLOCK_MEMORY_REGIONS		INIT_MEMBLOCK_REGIONS
-#endif
-
 /**
  * DOC: memblock overview
  *
diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c
index ebd3ac997f645..1fec89168ff10 100644
--- a/mm/sparse-vmemmap.c
+++ b/mm/sparse-vmemmap.c
@@ -179,13 +179,29 @@ static pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, in
 	return pte;
 }
 
-static void * __meminit vmemmap_alloc_block_zero(unsigned long size, int node)
+static void * __meminit vmemmap_alloc_pgtable(int node)
 {
-	void *p = vmemmap_alloc_block(size, node);
+	void *p;
+
+	if (slab_is_available()) {
+		gfp_t gfp = GFP_KERNEL | __GFP_ZERO;
+		struct ptdesc *ptdesc = pagetable_alloc(gfp, 0);
+
+		return ptdesc ? ptdesc_address(ptdesc) : NULL;
+	}
+
+	if (kpkeys_hardened_pgtables_early_enabled()) {
+		phys_addr_t phys = kpkeys_physmem_pgtable_alloc();
+
+		p = phys ? phys_to_virt(phys) : NULL;
+	} else {
+		p = __earlyonly_bootmem_alloc(node, PAGE_SIZE, PAGE_SIZE,
+					      __pa(MAX_DMA_ADDRESS));
+	}
 
 	if (!p)
 		return NULL;
-	memset(p, 0, size);
+	memset(p, 0, PAGE_SIZE);
 
 	return p;
 }
@@ -194,7 +210,7 @@ static pmd_t * __meminit vmemmap_pmd_populate(pud_t *pud, unsigned long addr, in
 {
 	pmd_t *pmd = pmd_offset(pud, addr);
 	if (pmd_none(*pmd)) {
-		void *p = vmemmap_alloc_block_zero(PAGE_SIZE, node);
+		void *p = vmemmap_alloc_pgtable(node);
 		if (!p)
 			return NULL;
 		kernel_pte_init(p);
@@ -207,7 +223,7 @@ static pud_t * __meminit vmemmap_pud_populate(p4d_t *p4d, unsigned long addr, in
 {
 	pud_t *pud = pud_offset(p4d, addr);
 	if (pud_none(*pud)) {
-		void *p = vmemmap_alloc_block_zero(PAGE_SIZE, node);
+		void *p = vmemmap_alloc_pgtable(node);
 		if (!p)
 			return NULL;
 		pmd_init(p);
@@ -220,7 +236,7 @@ static p4d_t * __meminit vmemmap_p4d_populate(pgd_t *pgd, unsigned long addr, in
 {
 	p4d_t *p4d = p4d_offset(pgd, addr);
 	if (p4d_none(*p4d)) {
-		void *p = vmemmap_alloc_block_zero(PAGE_SIZE, node);
+		void *p = vmemmap_alloc_pgtable(node);
 		if (!p)
 			return NULL;
 		pud_init(p);
@@ -233,7 +249,7 @@ static pgd_t * __meminit vmemmap_pgd_populate(unsigned long addr, int node)
 {
 	pgd_t *pgd = pgd_offset_k(addr);
 	if (pgd_none(*pgd)) {
-		void *p = vmemmap_alloc_block_zero(PAGE_SIZE, node);
+		void *p = vmemmap_alloc_pgtable(node);
 		if (!p)
 			return NULL;
 		pgd_populate_kernel(addr, pgd, p);
@@ -346,10 +362,11 @@ static __meminit struct page *vmemmap_get_tail(unsigned int order, struct zone *
 	 * memmap_init().
 	 */
 
-	p = vmemmap_alloc_block_zero(PAGE_SIZE, node);
+	p = vmemmap_alloc_block(PAGE_SIZE, node);
 	if (!p)
 		return NULL;
 
+	memset(p, 0, PAGE_SIZE);
 	tail = virt_to_page(p);
 	zone->vmemmap_tails[idx] = tail;
 
diff --git a/mm/tests/kpkeys_hardened_pgtables_kunit.c b/mm/tests/kpkeys_hardened_pgtables_kunit.c
new file mode 100644
index 0000000000000..80d4f5144e48d
--- /dev/null
+++ b/mm/tests/kpkeys_hardened_pgtables_kunit.c
@@ -0,0 +1,200 @@
+// SPDX-License-Identifier: GPL-2.0-only
+#include <kunit/test.h>
+#include <linux/mman.h>
+#include <linux/pgtable.h>
+#include <linux/set_memory.h>
+#include <linux/vmalloc.h>
+
+static void free_page_wrapper(void *ctx)
+{
+	__free_page((struct page *)ctx);
+}
+
+KUNIT_DEFINE_ACTION_WRAPPER(vfree_wrapper, vfree, const void *);
+
+static pud_t *pud_off_k(unsigned long va)
+{
+	return pud_offset(p4d_offset(pgd_offset_k(va), va), va);
+}
+
+static pte_t *get_kernel_pte(unsigned long addr)
+{
+	pmd_t *pmdp = pmd_off_k(addr);
+
+	if (!pmdp || pmd_leaf(*pmdp))
+		return NULL;
+
+	return pte_offset_kernel(pmdp, addr);
+}
+
+#define write_pgtable(type, ptr) do {					\
+	type##_t val;							\
+	int ret;							\
+									\
+	pr_debug("%s: writing to "#type" at %px\n", __func__, (ptr));	\
+									\
+	val = type##p_get(ptr);						\
+	ret = copy_to_kernel_nofault(ptr, &val, sizeof(val));		\
+	KUNIT_EXPECT_EQ_MSG(test, ret, -EFAULT,				\
+			    "Direct "#type" write wasn't prevented");	\
+} while (0)
+
+/*
+ * Try to write linear map page tables, at every level. This is worthwhile
+ * because those page table pages are obtained from different allocators:
+ *
+ * - Static memory (part of the kernel image) for PGD
+ * - memblock for PUD and possibly PMD/PTE
+ * - pagetable_alloc() (buddy allocator) for PMD/PTE if large block mappings are
+ *   used and the linear map is split after being created
+ */
+static void write_direct_map_pgtables(struct kunit *test)
+{
+	struct page *page;
+	unsigned long addr;
+	pgd_t *pgdp;
+	p4d_t *p4dp;
+	pud_t *pudp;
+	pmd_t *pmdp;
+	pte_t *ptep;
+	int ret;
+
+	if (!kpkeys_supported())
+		kunit_skip(test, "kpkeys are not supported");
+
+	page = alloc_page(GFP_KERNEL);
+	KUNIT_ASSERT_NOT_NULL(test, page);
+	ret = kunit_add_action_or_reset(test, free_page_wrapper, page);
+	KUNIT_ASSERT_EQ(test, ret, 0);
+
+	/* Ensure page is PTE-mapped (splitting the linear map if necessary) */
+	ret = set_direct_map_invalid_noflush(page);
+	KUNIT_ASSERT_EQ(test, ret, 0);
+	ret = set_direct_map_default_noflush(page);
+	KUNIT_ASSERT_EQ(test, ret, 0);
+
+	addr = (unsigned long)page_address(page);
+
+	pgdp = pgd_offset_k(addr);
+	KUNIT_ASSERT_NOT_NULL_MSG(test, pgdp, "Failed to get PGD");
+	/*
+	 * swapper_pg_dir is still writable at this stage, so don't check it.
+	 * It is not protected by kpkeys_hardened_pgtables because it should be
+	 * made read-only by mark_rodata_ro(). However since these
+	 * KUnit tests are builtin, they are run before mark_rodata_ro() is
+	 * called.
+	 */
+
+	p4dp = p4d_offset(pgdp, addr);
+	KUNIT_ASSERT_NOT_NULL_MSG(test, p4dp, "Failed to get P4D");
+	if (!mm_p4d_folded(&init_mm))
+		write_pgtable(p4d, p4dp);
+
+	pudp = pud_offset(p4dp, addr);
+	KUNIT_ASSERT_NOT_NULL_MSG(test, pudp, "Failed to get PUD");
+	if (!mm_pud_folded(&init_mm))
+		write_pgtable(pud, pudp);
+
+	pmdp = pmd_offset(pudp, addr);
+	KUNIT_ASSERT_NOT_NULL_MSG(test, pmdp, "Failed to get PMD");
+	write_pgtable(pmd, pmdp);
+
+	ptep = pte_offset_kernel(pmdp, addr);
+	KUNIT_ASSERT_NOT_NULL_MSG(test, ptep, "Failed to get PTE");
+	write_pgtable(pte, ptep);
+}
+
+/* Worth checking since the kernel image is mapped with static page tables */
+static void write_kernel_image_pud(struct kunit *test)
+{
+	pud_t *pudp;
+
+	if (!kpkeys_supported())
+		kunit_skip(test, "kpkeys are not supported");
+
+	/* The kernel is probably block-mapped, check the PUD to be safe */
+	pudp = pud_off_k((unsigned long)&init_mm);
+	KUNIT_ASSERT_NOT_NULL_MSG(test, pudp, "Failed to get PUD");
+
+	write_pgtable(pud, pudp);
+}
+
+static void write_kernel_vmalloc_pte(struct kunit *test)
+{
+	void *mem;
+	pte_t *ptep;
+	int ret;
+
+	if (!kpkeys_supported())
+		kunit_skip(test, "kpkeys are not supported");
+
+	mem = vmalloc(PAGE_SIZE);
+	KUNIT_ASSERT_NOT_NULL(test, mem);
+	ret = kunit_add_action_or_reset(test, vfree_wrapper, mem);
+	KUNIT_ASSERT_EQ(test, ret, 0);
+
+	/* vmalloc() without VM_ALLOW_HUGE_VMAP is PTE-mapped */
+	ptep = get_kernel_pte((unsigned long)mem);
+	KUNIT_ASSERT_NOT_NULL_MSG(test, ptep, "Failed to get PTE");
+
+	write_pgtable(pte, ptep);
+}
+
+static void write_vmemmap_pmd(struct kunit *test)
+{
+	struct page *page;
+	pmd_t *pmdp;
+
+	if (!kpkeys_supported())
+		kunit_skip(test, "kpkeys are not supported");
+
+	/*
+	 * We just need the address of some struct page, so we can free the
+	 * page right away.
+	 */
+	page = alloc_page(GFP_KERNEL);
+	KUNIT_ASSERT_NOT_NULL(test, page);
+	__free_page(page);
+
+	/* vmemmap may use PMD block mappings */
+	pmdp = pmd_off_k((unsigned long)page);
+	KUNIT_ASSERT_NOT_NULL_MSG(test, pmdp, "Failed to get PMD");
+	write_pgtable(pmd, pmdp);
+}
+
+static void write_user_pmd(struct kunit *test)
+{
+	pmd_t *pmdp;
+	unsigned long uaddr;
+
+	if (!kpkeys_supported())
+		kunit_skip(test, "kpkeys are not supported");
+
+	uaddr = kunit_vm_mmap(test, NULL, 0, PAGE_SIZE, PROT_READ,
+			      MAP_ANONYMOUS | MAP_PRIVATE | MAP_POPULATE, 0);
+	KUNIT_ASSERT_NE_MSG(test, uaddr, 0, "Could not create userspace mm");
+
+	/* We passed MAP_POPULATE so a PMD should already be allocated */
+	pmdp = pmd_off(current->mm, uaddr);
+	KUNIT_ASSERT_NOT_NULL_MSG(test, pmdp, "Failed to get PMD");
+
+	write_pgtable(pmd, pmdp);
+}
+
+static struct kunit_case kpkeys_hardened_pgtables_test_cases[] = {
+	KUNIT_CASE(write_direct_map_pgtables),
+	KUNIT_CASE(write_kernel_image_pud),
+	KUNIT_CASE(write_kernel_vmalloc_pte),
+	KUNIT_CASE(write_vmemmap_pmd),
+	KUNIT_CASE(write_user_pmd),
+	{}
+};
+
+static struct kunit_suite kpkeys_hardened_pgtables_test_suite = {
+	.name = "kpkeys_hardened_pgtables",
+	.test_cases = kpkeys_hardened_pgtables_test_cases,
+};
+kunit_test_suite(kpkeys_hardened_pgtables_test_suite);
+
+MODULE_DESCRIPTION("Tests for the kpkeys_hardened_pgtables feature");
+MODULE_LICENSE("GPL");
diff --git a/security/Kconfig.hardening b/security/Kconfig.hardening
index 6923036e1a2f5..ad6152961f8ee 100644
--- a/security/Kconfig.hardening
+++ b/security/Kconfig.hardening
@@ -271,6 +271,30 @@ config BUG_ON_DATA_CORRUPTION
 
 	  If unsure, say N.
 
+config KPKEYS_HARDENED_PGTABLES
+	bool "Harden page tables using kernel pkeys"
+	depends on ARCH_HAS_KPKEYS_HARDENED_PGTABLES
+	help
+	  This option makes all page tables mostly read-only by
+	  allocating them with a non-default protection key (pkey) and
+	  only enabling write access to that pkey in routines that are
+	  expected to write to page table entries.
+
+	  This option has no effect if the system does not support
+	  kernel pkeys.
+
+config KPKEYS_HARDENED_PGTABLES_KUNIT_TEST
+	bool "KUnit tests for kpkeys_hardened_pgtables" if !KUNIT_ALL_TESTS
+	depends on KPKEYS_HARDENED_PGTABLES
+	depends on KUNIT=y
+	default KUNIT_ALL_TESTS
+	help
+	  Enable this option to check that the kpkeys_hardened_pgtables feature
+	  functions as intended, i.e. prevents arbitrary writes to user and
+	  kernel page tables.
+
+	  If unsure, say N.
+
 endmenu
 
 config CC_HAS_RANDSTRUCT

gemini-3.1-pro-preview

							input: 21254
							output: 36
							thoughts: 1801

**KMSAN Fuzzing Evaluation for KPKEYS_HARDENED_PGTABLES Patch Series**

Okay, let's break this down. The objective is to determine if this patch series, introducing `KPKEYS_HARDENED_PGTABLES`, warrants a dedicated KMSAN fuzzing session. My focus is on potential uninitialized memory reads or information leaks due to the changes.

Initially, I see the core modification involves page table hardening using kernel pkeys. The patch touches several areas, including configuration options, ARM64 page table manipulation functions, and new files like `kpkeys_hardened_pgtables.c`.

My first concern is uninitialized memory. The code changes in `include/linux/mm.h`, particularly the `pagetable_alloc_noprof` function, catch my eye.  The original code used `alloc_pages_noprof` with `__GFP_COMP`, and now it's conditionally calling `kpkeys_pgtable_alloc`. The call to `kpkeys_pgtable_alloc` utilizes `alloc_pages_noprof` with a different `gfp`. However, I need to look at `pagetable_alloc_noprof` to understand how `kpkeys_pgtable_alloc` is called.

After considering `vmemmap_alloc_pgtable` and `vmemmap_get_tail` in `mm/sparse-vmemmap.c`, I'm reassured that explicit zeroing (`memset`) is still in place, so that removes the uninitialized memory issue there.

Then, there's `kpkeys_physmem_pgtable_alloc` which uses `memblock_phys_alloc_range`.  I need to remember that `memblock_phys_alloc_range` doesn't zero the memory. So, I look at the usage in `arch/arm64/mm/mmu.c`, specifically `early_pgtable_alloc`. The original code *also* didn't zero with `memblock_phys_alloc_range`. It's used for early page tables, but the important thing is to confirm if the *caller* zeroes the memory. In most cases, the callers of these functions, such as `pgd_populate`, are responsible for initialization. I need to keep an eye on *how* these pages are subsequently handled to confirm.  It seems the original behavior is maintained here.

Finally, I need to check for any potential information leaks to user space. The patch introduces `struct kpkeys_state` and `struct arch_kpkeys_state` within `struct thread_struct`, which is part of `task_struct`.  The `thread_struct` itself isn't directly exposed to user space, but components might be, e.g., via ptrace or signal frames. The key question is whether these new fields, or the pre-existing `por_el0`, are accessible through ptrace or similar mechanisms. The patch *adds* `por_el1` and `kpkeys_state_lazy_mmu`. I'll need to use my preferred method (likely `grepper`) to search `arch/arm64/kernel/ptrace.c` for `por_el1` to ensure this isn't exposed.

Given the initial assessment, the allocation changes appear to have addressed zero-initialization or followed existing practices. The potential leak concerns needs further verification.

{
  "Expression": "por_el1",
  "PathPrefix": "arch/arm64/kernel/ptrace.c"
}

{
  "error": "no matches"
}

gemini-3.1-pro-preview

							input: 23831
							output: 36
							thoughts: 19

Okay, here's my take on those thoughts, tailored for an expert audience:

**Debugging Divergence: Initial Ptrace Dead End and a Potential Lead**

Alright, so I was digging into the divergence issue, specifically looking for something related to tracing. First, I ran a search in `ptrace.c`, figuring that's the obvious place to start when you're dealing with trace points and such. But, *nada*. Not a single match popped up that seemed relevant to the symptoms I'm observing. That’s a bit unusual, and it tells me I need to broaden the search.

Then, my mind immediately jumped to `por_el0`. This is where things could get interesting. I started considering how that area could be involved. I'm thinking about the control flow, the privilege level transitions, all that. Specifically, I'm thinking about how the problem could relate to some interactions with that subsystem. Is there a scenario where something happening there, some interaction with that code, is masking or diverting tracing? It's a hunch, but given the lack of hits in `ptrace.c`, I’m feeling pretty good about that lead. Now, time to dig deeper into the por_el0 context and see if that's where I can pick up the next piece of the puzzle.

{
  "Expression": "por_el0",
  "PathPrefix": "arch/arm64/kernel/ptrace.c"
}

{
  "Output": "arch/arm64/kernel/ptrace.c=1480=static int poe_get(struct task_struct *target,\n--\narch/arm64/kernel/ptrace.c-1487-\tif (target == current)\narch/arm64/kernel/ptrace.c:1488:\t\tcurrent-\u003ethread.por_el0 = read_sysreg_s(SYS_POR_EL0);\narch/arm64/kernel/ptrace.c-1489-\narch/arm64/kernel/ptrace.c:1490:\treturn membuf_write(\u0026to, \u0026target-\u003ethread.por_el0,\narch/arm64/kernel/ptrace.c:1491:\t\t\t    sizeof(target-\u003ethread.por_el0));\narch/arm64/kernel/ptrace.c-1492-}\n--\narch/arm64/kernel/ptrace.c=1494=static int poe_set(struct task_struct *target, const struct\n--\narch/arm64/kernel/ptrace.c-1504-\narch/arm64/kernel/ptrace.c:1505:\tctrl = target-\u003ethread.por_el0;\narch/arm64/kernel/ptrace.c-1506-\n--\narch/arm64/kernel/ptrace.c-1510-\narch/arm64/kernel/ptrace.c:1511:\ttarget-\u003ethread.por_el0 = ctrl;\narch/arm64/kernel/ptrace.c-1512-\n"
}

gemini-3.1-pro-preview

							input: 24565
							output: 30
							thoughts: 836

**Analyzing Potential POR_EL1 and kpkeys_state_lazy_mmu Issues in Thread Forking**

Okay, so I'm looking at how `por_el0` is potentially exposed, but `por_el1` isn't in the context of thread creation. My first thought is to check how `copy_thread` handles this, specifically in `arch/arm64/kernel/process.c`. I see `ptrauth_thread_init_kernel` is called, and then there's an initialization of `p->thread.por_el1` if `system_supports_poe()` returns true, which makes sense; this suggests that the new thread *will* have an initialized `por_el1` value.

Next, I have to consider `kpkeys_state_lazy_mmu`. My immediate question is whether this structure is properly initialized in the child thread. Reading the code, it's clear that `copy_thread` doesn't explicitly initialize it. The `task_struct` is allocated with `alloc_task_struct_node` which in turn uses `kmem_cache_alloc_node` with `GFP_KERNEL`. But, wait, `dup_task_struct` *copies* the parent's `task_struct` to the child. This means that the `kpkeys_state_lazy_mmu` member will be copied from the parent thread.

The core question then becomes: Is there an issue with uninitialized memory here, or is this okay? If `kpkeys_state_lazy_mmu` is copied from the parent, the child will inherit its state. This becomes interesting if the parent is in lazy mmu mode during the fork. I'm recalling that `kpkeys_lazy_mmu_enter` and `kpkeys_lazy_mmu_exit` are used around lazy mmu mode operations.

Looking at the code for these functions, I see that if the system is not hardened, they return early, so no context enter or leave is needed. If hardening is enabled and `TIF_LAZY_MMU_PENDING` is not set, then the code calls `kpkeys_leave_context`. However, if the `TIF_LAZY_MMU_PENDING` is set, a barrier is skipped because `emit_pte_barriers` will take care of the barrier.

So if a thread *forks* while in lazy mmu mode, the child will inherit whatever state the parent had in `kpkeys_state_lazy_mmu`. But then, I need to remind myself where lazy mmu mode is *used*. I'm thinking of `mremap`, `munmap`, etc., which could mean preemption is disabled or something like that. But, no. I'm remembering that lazy mmu mode is used in `arch_enter_lazy_mmu_mode`. And that is called from `enter_lazy_mmu` in `mm/mmu_gather.c`. Okay, now I need to check the call graph for `arch_enter_lazy_mmu_mode`.

{
  "Expression": "arch_enter_lazy_mmu_mode",
  "PathPrefix": ""
}

{
  "Output": "arch/arm64/include/asm/pgtable.h=105=static inline void queue_pte_barriers(void)\n--\narch/arm64/include/asm/pgtable.h-115-\narch/arm64/include/asm/pgtable.h:116:static inline void arch_enter_lazy_mmu_mode(void)\narch/arm64/include/asm/pgtable.h-117-{\n--\narch/powerpc/include/asm/book3s/64/tlbflush-hash.h=24=extern void __flush_tlb_pending(struct ppc64_tlb_batch *batch);\narch/powerpc/include/asm/book3s/64/tlbflush-hash.h-25-\narch/powerpc/include/asm/book3s/64/tlbflush-hash.h:26:static inline void arch_enter_lazy_mmu_mode(void)\narch/powerpc/include/asm/book3s/64/tlbflush-hash.h-27-{\n--\narch/sparc/include/asm/tlbflush_64.h=41=void flush_tlb_pending(void);\narch/sparc/include/asm/tlbflush_64.h:42:void arch_enter_lazy_mmu_mode(void);\narch/sparc/include/asm/tlbflush_64.h-43-void arch_flush_lazy_mmu_mode(void);\n--\narch/sparc/mm/tlb.c=25=void flush_tlb_pending(void)\n--\narch/sparc/mm/tlb.c-54-\narch/sparc/mm/tlb.c:55:void arch_enter_lazy_mmu_mode(void)\narch/sparc/mm/tlb.c-56-{\n--\narch/sparc/mm/tlb.c-59-/* For lazy_mmu_mode KUnit tests */\narch/sparc/mm/tlb.c:60:EXPORT_SYMBOL_IF_KUNIT(arch_enter_lazy_mmu_mode);\narch/sparc/mm/tlb.c-61-\n--\narch/x86/include/asm/paravirt.h=479=static inline void arch_end_context_switch(struct task_struct *next)\n--\narch/x86/include/asm/paravirt.h-483-\narch/x86/include/asm/paravirt.h:484:static inline void arch_enter_lazy_mmu_mode(void)\narch/x86/include/asm/paravirt.h-485-{\n--\ninclude/linux/pgtable.h=190=static inline int pmd_dirty(pmd_t pmd)\n--\ninclude/linux/pgtable.h-252- * Enters a new lazy MMU mode section; if the mode was not already enabled,\ninclude/linux/pgtable.h:253: * enables it and calls arch_enter_lazy_mmu_mode().\ninclude/linux/pgtable.h-254- *\n--\ninclude/linux/pgtable.h=261=static inline void lazy_mmu_mode_enable(void)\n--\ninclude/linux/pgtable.h-270-\tif (state-\u003eenable_count++ == 0)\ninclude/linux/pgtable.h:271:\t\tarch_enter_lazy_mmu_mode();\ninclude/linux/pgtable.h-272-}\n--\ninclude/linux/pgtable.h=358=static inline void __task_lazy_mmu_mode_resume(struct task_struct *tsk)\n--\ninclude/linux/pgtable.h-364-\tif (--state-\u003epause_count == 0 \u0026\u0026 state-\u003eenable_count \u003e 0)\ninclude/linux/pgtable.h:365:\t\tarch_enter_lazy_mmu_mode();\ninclude/linux/pgtable.h-366-}\n--\ninclude/linux/pgtable.h-372- * call to lazy_mmu_mode_pause() was made, re-enables it and calls\ninclude/linux/pgtable.h:373: * arch_enter_lazy_mmu_mode().\ninclude/linux/pgtable.h-374- *\n"
}

gemini-3.1-pro-preview

							input: 27179
							output: 38
							thoughts: 990

**Analyzing Potential Uninitialized Memory Exposure in Lazy MMU Mode**

Okay, so `lazy_mmu_mode_enable` calls `arch_enter_lazy_mmu_mode`. My first concern is always, is there any uninitialized memory making its way into user space?

The patch adds `por_el1` and `kpkeys_state_lazy_mmu` to `struct thread_struct`, which is part of the `task_struct`. Good. `task_struct` isn't directly copied to user space, so that's a check. `por_el0` *is* copied to user space via ptrace, but the patch doesn't actually introduce any new changes to the `ptrace` handling of `por_el0`, except maybe in `arch_set_user_pkey_access`. Let's look at that:

Looking at the code, it looks like it just modifies the `SYS_POR_EL0` register directly. The changes are just changing the calculation of the new permissions: previously setting bits directly and bitwise masking, and now setting a permissions variable. Then it sets the permissions into the register. I don't see any sign of uninitialized memory being exposed here. This is good.

Okay, let's circle back to `kpkeys_physmem_pgtable_alloc`. That function allocates memory for page tables. The key question is whether or not that memory is zeroed out. The callers are `early_pgtable_alloc` and `vmemmap_alloc_pgtable`. I want to dig into `early_pgtable_alloc` first. I see it in `arch/arm64/mm/mmu.c`:

`early_pgtable_alloc` either calls `kpkeys_physmem_pgtable_alloc` (if `kpkeys_hardened_pgtables_early_enabled()` returns true) or directly uses `memblock_phys_alloc_range`. It looks like the important thing is that `early_pgtable_alloc` is passed as the `alloc` function to `pgd_populate`, `pud_populate`, etc. The question is, does the *caller* zero the memory? I need to check `alloc_init_pud` or a similar function in `arch/arm64/mm/mmu.c` to be sure. This is where the potential leak might be if the caller isn't properly initializing the allocated pages.

{
  "Expression": "early_pgtable_alloc",
  "PathPrefix": "arch/arm64/mm/mmu.c"
}

{
  "Output": "arch/arm64/mm/mmu.c=104=EXPORT_SYMBOL(phys_mem_access_prot);\narch/arm64/mm/mmu.c-105-\narch/arm64/mm/mmu.c:106:static phys_addr_t __init early_pgtable_alloc(enum pgtable_level pgtable_level)\narch/arm64/mm/mmu.c-107-{\n--\narch/arm64/mm/mmu.c=1059=static void __init __map_memblock(phys_addr_t start, phys_addr_t end,\n--\narch/arm64/mm/mmu.c-1062-\tearly_create_pgd_mapping(swapper_pg_dir, start, __phys_to_virt(start),\narch/arm64/mm/mmu.c:1063:\t\t\t\t end - start, prot, early_pgtable_alloc, flags);\narch/arm64/mm/mmu.c-1064-}\n"
}

gemini-3.1-pro-preview

							input: 29001
							output: 40
							thoughts: 38

Here's my attempt at summarizing the thought process, tailored for an expert audience:

**Investigating Potential Root Causes for a Memory Mapping Issue**

Okay, so I'm encountering something with memory mapping, and I need to pinpoint the problem area within the ARM64 architecture. Given the symptoms, my gut feeling is leading me towards something fundamental related to the initial setup of the page global directory (PGD) or the allocation and initialization of page upper directories (PUDs).

Therefore, I'm going to focus my attention on `arch/arm64/mm/mmu.c`, specifically scrutinizing the implementations of `early_create_pgd_mapping` and `alloc_init_pud`. These functions are critical in establishing the initial virtual to physical address translation structures. I'll need to carefully review the code logic within these routines. Things like potential off-by-one errors, incorrect address calculations, subtle race conditions, or failures to account for specific memory attributes are all on my radar. I'll be looking for any potential deviations from the expected behavior, which might be manifesting in the current mapping problem. A close examination of their interaction with the boot process and any subsequent memory management calls will be essential.

{
  "Expression": "early_create_pgd_mapping",
  "PathPrefix": "arch/arm64/mm/mmu.c"
}

{
  "Output": "arch/arm64/mm/mmu.c=516=static int __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys,\n--\narch/arm64/mm/mmu.c-531-\narch/arm64/mm/mmu.c:532:static void early_create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys,\narch/arm64/mm/mmu.c-533-\t\t\t\t     unsigned long virt, phys_addr_t size,\n--\narch/arm64/mm/mmu.c=1018=void __init create_mapping_noalloc(phys_addr_t phys, unsigned long virt,\n--\narch/arm64/mm/mmu.c-1025-\t}\narch/arm64/mm/mmu.c:1026:\tearly_create_pgd_mapping(init_mm.pgd, phys, virt, size, prot, NULL, 0);\narch/arm64/mm/mmu.c-1027-}\n--\narch/arm64/mm/mmu.c=1029=void __init create_pgd_mapping(struct mm_struct *mm, phys_addr_t phys,\n--\narch/arm64/mm/mmu.c-1039-\narch/arm64/mm/mmu.c:1040:\tearly_create_pgd_mapping(mm-\u003epgd, phys, virt, size, prot,\narch/arm64/mm/mmu.c-1041-\t\t\t\t pgd_pgtable_alloc_special_mm, flags);\n--\narch/arm64/mm/mmu.c=1044=static void update_mapping_prot(phys_addr_t phys, unsigned long virt,\n--\narch/arm64/mm/mmu.c-1052-\narch/arm64/mm/mmu.c:1053:\tearly_create_pgd_mapping(init_mm.pgd, phys, virt, size, prot, NULL, 0);\narch/arm64/mm/mmu.c-1054-\n--\narch/arm64/mm/mmu.c=1059=static void __init __map_memblock(phys_addr_t start, phys_addr_t end,\n--\narch/arm64/mm/mmu.c-1061-{\narch/arm64/mm/mmu.c:1062:\tearly_create_pgd_mapping(swapper_pg_dir, start, __phys_to_virt(start),\narch/arm64/mm/mmu.c-1063-\t\t\t\t end - start, prot, early_pgtable_alloc, flags);\n--\narch/arm64/mm/mmu.c=1366=static int __init map_entry_trampoline(void)\n--\narch/arm64/mm/mmu.c-1380-\tmemset(tramp_pg_dir, 0, PGD_SIZE);\narch/arm64/mm/mmu.c:1381:\tearly_create_pgd_mapping(tramp_pg_dir, pa_start, TRAMP_VALIAS,\narch/arm64/mm/mmu.c-1382-\t\t\t\t entry_tramp_text_size(), prot,\n"
}

gemini-3.1-pro-preview

							input: 30188
							output: 38
							thoughts: 20

Here's my thought process, as if I were considering this issue:

**Delving into the Allocation Landscape**

Alright, I'm thinking we need to trace down where this memory allocation problem is originating. Given the context, my initial hunch is that it might be connected to the page table initialization. Let's start with a focused investigation of `alloc_init_pud`, or perhaps related functions. Anything that's responsible for setting up those crucial page directory entries and uses the memory allocator, is a prime suspect. I'll need to carefully examine those allocation routines, looking for any potential issues. Things like unchecked return values from the allocator, or maybe a subtle off-by-one error in the calculations of the size or alignment. I should look at any places that might be allocating metadata for the page table structures.

{
  "Expression": "alloc_init_pud",
  "PathPrefix": "arch/arm64/mm/mmu.c"
}

{
  "Output": "arch/arm64/mm/mmu.c=306=static int alloc_init_cont_pmd(pud_t *pudp, unsigned long addr,\n--\narch/arm64/mm/mmu.c-364-\narch/arm64/mm/mmu.c:365:static int alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end,\narch/arm64/mm/mmu.c-366-\t\t\t  phys_addr_t phys, pgprot_t prot,\n--\narch/arm64/mm/mmu.c=432=static int alloc_init_p4d(pgd_t *pgdp, unsigned long addr, unsigned long end,\n--\narch/arm64/mm/mmu.c-465-\narch/arm64/mm/mmu.c:466:\t\tret = alloc_init_pud(p4dp, addr, next, phys, prot,\narch/arm64/mm/mmu.c-467-\t\t\t\t     pgtable_alloc, flags);\n"
}

gemini-3.1-pro-preview

							input: 30702
							output: 38
							thoughts: 14

Here's my attempt at summarizing the thought process, tailored for an expert audience:

**Diving into `alloc_init_pud`**

Okay, time to dig into `alloc_init_pud`. I need to refresh my memory on exactly how this thing works. I'm focusing specifically on how the PUD is initialized. I'm guessing this function is central to the setup of a new page directory entry, which of course is fundamental to the memory management of a process. I expect to see the low-level details, perhaps the direct manipulation of hardware page table entries, or calls to lower-level allocation routines. I wonder if it directly zeroes out the PUD entry, or if there's a more nuanced initialization strategy, perhaps involving a bitmask or some early configuration of the present bit. Let's get to the code and see. My focus here is going to be about how it sets up a page directory entry at the PUD level.

{
  "Expression": "alloc_init_pud",
  "PathPrefix": "arch/arm64/mm/mmu.c"
}

{
  "Output": "arch/arm64/mm/mmu.c=306=static int alloc_init_cont_pmd(pud_t *pudp, unsigned long addr,\n--\narch/arm64/mm/mmu.c-364-\narch/arm64/mm/mmu.c:365:static int alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end,\narch/arm64/mm/mmu.c-366-\t\t\t  phys_addr_t phys, pgprot_t prot,\n--\narch/arm64/mm/mmu.c=432=static int alloc_init_p4d(pgd_t *pgdp, unsigned long addr, unsigned long end,\n--\narch/arm64/mm/mmu.c-465-\narch/arm64/mm/mmu.c:466:\t\tret = alloc_init_pud(p4dp, addr, next, phys, prot,\narch/arm64/mm/mmu.c-467-\t\t\t\t     pgtable_alloc, flags);\n"
}

gemini-3.1-pro-preview

							input: 31240
							output: 40
							thoughts: 25

Here's a summary of my thought process, as requested:

**Diving into the ARM64 MMU Implementation**

Okay, time to get my hands dirty. I need to understand the specifics of how the ARM64 MMU is handling memory management, particularly the translation process. Let's start with a focused look at `arch/arm64/mm/mmu.c`. Based on my experience, line 365 is a good place to begin. It's likely involved in the core translation logic, perhaps dealing with page table walks or the initial setup. I expect it'll give me a solid entry point to trace the control flow and unravel the specific implementation details, especially if I can discern how it is dealing with virtual and physical address spaces.

{
  "File": "arch/arm64/mm/mmu.c",
  "FirstLine": 365,
  "LineCount": 50
}

{
  "Contents": " 365:\tstatic int alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end,\n 366:\t\t\t\t  phys_addr_t phys, pgprot_t prot,\n 367:\t\t\t\t  phys_addr_t (*pgtable_alloc)(enum pgtable_level),\n 368:\t\t\t\t  int flags)\n 369:\t{\n 370:\t\tint ret = 0;\n 371:\t\tunsigned long next;\n 372:\t\tp4d_t p4d = READ_ONCE(*p4dp);\n 373:\t\tpud_t *pudp;\n 374:\t\n 375:\t\tif (p4d_none(p4d)) {\n 376:\t\t\tp4dval_t p4dval = P4D_TYPE_TABLE | P4D_TABLE_UXN | P4D_TABLE_AF;\n 377:\t\t\tphys_addr_t pud_phys;\n 378:\t\n 379:\t\t\tif (flags \u0026 NO_EXEC_MAPPINGS)\n 380:\t\t\t\tp4dval |= P4D_TABLE_PXN;\n 381:\t\t\tBUG_ON(!pgtable_alloc);\n 382:\t\t\tpud_phys = pgtable_alloc(PGTABLE_LEVEL_PUD);\n 383:\t\t\tif (pud_phys == INVALID_PHYS_ADDR)\n 384:\t\t\t\treturn -ENOMEM;\n 385:\t\t\tpudp = pud_set_fixmap(pud_phys);\n 386:\t\t\tinit_clear_pgtable(pudp);\n 387:\t\t\tpudp += pud_index(addr);\n 388:\t\t\t__p4d_populate(p4dp, pud_phys, p4dval);\n 389:\t\t} else {\n 390:\t\t\tBUG_ON(p4d_bad(p4d));\n 391:\t\t\tpudp = pud_set_fixmap_offset(p4dp, addr);\n 392:\t\t}\n 393:\t\n 394:\t\tdo {\n 395:\t\t\tpud_t old_pud = READ_ONCE(*pudp);\n 396:\t\n 397:\t\t\tnext = pud_addr_end(addr, end);\n 398:\t\n 399:\t\t\t/*\n 400:\t\t\t * For 4K granule only, attempt to put down a 1GB block\n 401:\t\t\t */\n 402:\t\t\tif (pud_sect_supported() \u0026\u0026\n 403:\t\t\t   ((addr | next | phys) \u0026 ~PUD_MASK) == 0 \u0026\u0026\n 404:\t\t\t    (flags \u0026 NO_BLOCK_MAPPINGS) == 0 \u0026\u0026\n 405:\t\t\t    !pud_table(old_pud)) {\n 406:\t\t\t\tWARN_ON(!pud_set_huge(pudp, phys, prot));\n 407:\t\n 408:\t\t\t\t/*\n 409:\t\t\t\t * After the PUD entry has been populated once, we\n 410:\t\t\t\t * only allow updates to the permission attributes.\n 411:\t\t\t\t */\n 412:\t\t\t\tBUG_ON(!pgattr_change_is_safe(pud_val(old_pud),\n 413:\t\t\t\t\t\t\t      READ_ONCE(pud_val(*pudp))));\n 414:\t\t\t} else {\n"
}

gemini-3.1-pro-preview

							input: 32339
							output: 38
							thoughts: 30

Here's my take on the thought process, as requested:

**Deconstructing the Page Table Initialization**

Okay, so I see `init_clear_pgtable(pudp)` called. That's the signal, isn't it? Time to dig into how the page tables are being initialized and cleared. The `pudp` likely points to the pagetable being initialized. I'm immediately interested in the implementation of that function. Let's start with a deep dive into `init_clear_pgtable` itself. I need to understand what's happening at the lowest levels here – the actual bit manipulation or memory writes. What's the strategy? Are we zeroing out entries? Setting specific flags? Is there any optimization going on, like using vector instructions or memory copies if appropriate? Let's figure out the full context.

{
  "Expression": "init_clear_pgtable",
  "PathPrefix": "arch/arm64/mm/mmu.c"
}

{
  "Output": "arch/arm64/mm/mmu.c=121=bool pgattr_change_is_safe(pteval_t old, pteval_t new)\n--\narch/arm64/mm/mmu.c-155-\narch/arm64/mm/mmu.c:156:static void init_clear_pgtable(void *table)\narch/arm64/mm/mmu.c-157-{\n--\narch/arm64/mm/mmu.c=198=static int alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr,\n--\narch/arm64/mm/mmu.c-219-\t\tptep = pte_set_fixmap(pte_phys);\narch/arm64/mm/mmu.c:220:\t\tinit_clear_pgtable(ptep);\narch/arm64/mm/mmu.c-221-\t\tptep += pte_index(addr);\n--\narch/arm64/mm/mmu.c=306=static int alloc_init_cont_pmd(pud_t *pudp, unsigned long addr,\n--\narch/arm64/mm/mmu.c-331-\t\tpmdp = pmd_set_fixmap(pmd_phys);\narch/arm64/mm/mmu.c:332:\t\tinit_clear_pgtable(pmdp);\narch/arm64/mm/mmu.c-333-\t\tpmdp += pmd_index(addr);\n--\narch/arm64/mm/mmu.c=365=static int alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end,\n--\narch/arm64/mm/mmu.c-385-\t\tpudp = pud_set_fixmap(pud_phys);\narch/arm64/mm/mmu.c:386:\t\tinit_clear_pgtable(pudp);\narch/arm64/mm/mmu.c-387-\t\tpudp += pud_index(addr);\n--\narch/arm64/mm/mmu.c=432=static int alloc_init_p4d(pgd_t *pgdp, unsigned long addr, unsigned long end,\n--\narch/arm64/mm/mmu.c-452-\t\tp4dp = p4d_set_fixmap(p4d_phys);\narch/arm64/mm/mmu.c:453:\t\tinit_clear_pgtable(p4dp);\narch/arm64/mm/mmu.c-454-\t\tp4dp += p4d_index(addr);\n--\narch/arm64/mm/mmu.c=546=static phys_addr_t __pgd_pgtable_alloc(struct mm_struct *mm, gfp_t gfp,\n--\narch/arm64/mm/mmu.c-548-{\narch/arm64/mm/mmu.c:549:\t/* Page is zeroed by init_clear_pgtable() so don't duplicate effort. */\narch/arm64/mm/mmu.c-550-\tstruct ptdesc *ptdesc = pagetable_alloc(gfp \u0026 ~__GFP_ZERO, 0);\n"
}

gemini-3.1-pro-preview

							input: 33420
							output: 38
							thoughts: 691

**Analyzing Page Table Hardening**

Okay, so I'm digging into this page table hardening patch. My initial thought was to make sure no new uninitialized memory vulnerabilities are introduced. The `init_clear_pgtable` function is clearly zeroing the page table from the start, which gives me a starting point of known state. Now, let's circle back to `kpkeys_pgtable_alloc`.

Looking at the code for `kpkeys_pgtable_alloc`, it's using `alloc_pages_noprof` and then `set_pkey_pgtable`. The caller, `pagetable_alloc_noprof`, is passing `gfp | __GFP_COMP`. Now, if `gfp` already includes `__GFP_ZERO`, `alloc_pages_noprof` does a zero-fill. The crucial bit here is to make sure we're *still* zeroing; the original code used `alloc_pages_noprof(gfp | __GFP_COMP, order)`, so the behavior appears identical regarding zeroing.

Now, let's step back and look at the bigger picture. The patch itself introduces `KPKEYS_HARDENED_PGTABLES`, which leverages kernel pkeys to harden the page tables. It modifies those fundamental page table accessors – `set_pte_nosync`, `set_pmd`, the whole lot – to utilize `guard(kpkeys_hardened_pgtables)()` or `scoped_guard(kpkeys_hardened_pgtables)`. Essentially, these guards temporarily allow write access to otherwise read-only page tables during the modification.

My primary concern is around uninitialized memory. Does the patch introduce any new structures sent to user space? No, not that I can see. It modifies `struct thread_struct` with `por_el1` and `kpkeys_state_lazy_mmu`, but these are internal to the kernel, so no exposure there. The page table allocation routines, using `kpkeys_pgtable_alloc` and `kpkeys_physmem_pgtable_alloc`, appear to retain the same zeroing behavior as before. Excellent.

Now, a small but important detail. It looks like `vmemmap_alloc_pgtable` now calls `kpkeys_physmem_pgtable_alloc` and *explicitly* zeroes the allocated page. Same for `vmemmap_get_tail` now calls `vmemmap_alloc_block` and zeroing is explicitly handled. Wait a second, the `vmemmap_alloc_block_zero` function was removed and replaced with `vmemmap_alloc_block` followed by `memset(p, 0, PAGE_SIZE)`. This is worth a second look, let's take a look at the code for that replacement in `mm/sparse-vmemmap.c` and make sure it is correct.

{
  "Expression": "vmemmap_alloc_block_zero",
  "PathPrefix": "mm/sparse-vmemmap.c"
}

{
  "error": "no matches"
}

gemini-3.1-pro-preview

							input: 34781
							output: 29
							thoughts: 492

**Analyzing the Removal of `vmemmap_alloc_block_zero`**

Okay, so the patch I'm looking at removed `vmemmap_alloc_block_zero` from `mm/sparse-vmemmap.c`. Let's dig into the details. Examining the code, I see the old function was replaced with `vmemmap_alloc_pgtable`. The original function allocated a block, and then zeroed it out. Now, with the replacement, the code seems to check if the slab allocator is available and use it with the `__GFP_ZERO` flag. If not, it falls back to either `kpkeys_physmem_pgtable_alloc` if `kpkeys_hardened_pgtables_early_enabled()` is true, otherwise it uses `__earlyonly_bootmem_alloc`. Crucially, it now zero's out each page size after allocation regardless of which route is taken.

Interesting. It looks like `vmemmap_alloc_block_zero` was also used in `vmemmap_get_tail`, where it was used to allocate a page and zero it. This has been replaced by `vmemmap_alloc_block`, followed by a `memset` to zero the memory. It seems like the core functionality is still there, but handled differently.

So, the next question is: Did they miss any other uses of the old function? I need to make sure this change is complete. I'll need to run a quick search through the code to make sure there are no other places where `vmemmap_alloc_block_zero` was used that might have been overlooked.

{
  "Expression": "vmemmap_alloc_block_zero",
  "PathPrefix": ""
}

{
  "error": "no matches"
}

gemini-3.1-pro-preview

							input: 35665
							output: 473
							thoughts: 11719

**Analysis of Kernel Page Table Hardening Implementation**

Okay, so I've meticulously reviewed this patch series, and it all boils down to enhancing the security of our page tables using kernel Protection Key (PKey) functionality, primarily on ARM64. It’s a fairly focused refactoring to protect page tables with a specific PKey, to prevent unauthorized modification. I’ve gone through all the relevant code and I can confirm that there are **no apparent uninitialized memory access vulnerabilities** introduced by this implementation, nor any new paths for information leakage to user space.

The process primarily involves initializing and setting the PKey appropriately, and then enabling write access to page tables conditionally using `guard(kpkeys_hardened_pgtables)()` where modifications are required. This approach seems robust and correct. The new code allocates memory for page tables using standard methods, like `alloc_pages_noprof` and `memblock_phys_alloc_range`, which has been carefully examined to ensure it's not a source of any uninitialized memory access. I've confirmed they are either zeroed or re-initialized before use, as they were prior to these changes.

Moreover, the core data structures, such as the `struct kpkeys_state` and the `por_el1` field within the `thread_struct`, are either local to the kernel or carefully handled during context switches, never directly exposed to user space. There's no scenario where uninitialized memory would leak out. Even when there's an instance where a local struct, such as within the `KPKEYS_GUARD_COND` macro, might have uninitialized fields, this is managed within the scope of the code and isn't something that can become an information leak.

Importantly, I’ve checked the KUnit tests included within the series. It's safe, since these KUnit tests are just that, they *test*, and write values and checks.

Essentially, this patch series seems to have been carefully implemented. I do not see any reason to run this specific code through a KMSAN session. KASAN and other standard bug detectors should be able to catch potential memory corruption issues. Therefore, I'll set `NeedsKMSAN=false` and provide this detailed reasoning.

{
  "KMSANReasoning": "The patch series introduces `KPKEYS_HARDENED_PGTABLES` to harden page tables using kernel pkeys (KPKEYS) on ARM64. It modifies page table allocation to set a non-default protection key and uses `guard(kpkeys_hardened_pgtables)()` to temporarily enable write access when modifying page table entries.\n\nAfter a thorough review of the changes:\n1. **No new structures sent to user space:** The patch adds `por_el1` and `kpkeys_state_lazy_mmu` to `struct thread_struct`, but these are internal kernel state and are not exposed to user space via ptrace, signals, or any other mechanism.\n2. **Page table allocation zeroing:** The patch modifies page table allocation to use `kpkeys_pgtable_alloc` and `kpkeys_physmem_pgtable_alloc`. These functions wrap standard allocators (`alloc_pages_noprof` and `memblock_phys_alloc_range`). The zeroing behavior remains identical to the original code: `early_pgtable_alloc` allocations are zeroed by `init_clear_pgtable()`, `pagetable_alloc_noprof` allocations are zeroed by `alloc_pages_noprof` if `__GFP_ZERO` is passed, and `vmemmap_alloc_pgtable` explicitly calls `memset(p, 0, PAGE_SIZE)`.\n3. **No uninitialized memory risks:** The `struct kpkeys_state` used in `scoped_guard` is a local variable and is not copied to user space. The conditional initialization in `arch_kpkeys_enter_context` is safe because the uninitialized fields are never read if `entered_context` is false.\n\nThe changes primarily involve memory protection and control flow around page table modifications. Any bugs introduced would likely be out-of-bounds accesses, use-after-free, or invalid memory accesses, which are well-covered by KASAN and standard bug detectors. There is no risk of uninitialized memory usage or info-leaks that would require KMSAN.",
  "NeedsKMSAN": false
}

{
  "KMSANReasoning": "The patch series introduces `KPKEYS_HARDENED_PGTABLES` to harden page tables using kernel pkeys (KPKEYS) on ARM64. It modifies page table allocation to set a non-default protection key and uses `guard(kpkeys_hardened_pgtables)()` to temporarily enable write access when modifying page table entries.\n\nAfter a thorough review of the changes:\n1. **No new structures sent to user space:** The patch adds `por_el1` and `kpkeys_state_lazy_mmu` to `struct thread_struct`, but these are internal kernel state and are not exposed to user space via ptrace, signals, or any other mechanism.\n2. **Page table allocation zeroing:** The patch modifies page table allocation to use `kpkeys_pgtable_alloc` and `kpkeys_physmem_pgtable_alloc`. These functions wrap standard allocators (`alloc_pages_noprof` and `memblock_phys_alloc_range`). The zeroing behavior remains identical to the original code: `early_pgtable_alloc` allocations are zeroed by `init_clear_pgtable()`, `pagetable_alloc_noprof` allocations are zeroed by `alloc_pages_noprof` if `__GFP_ZERO` is passed, and `vmemmap_alloc_pgtable` explicitly calls `memset(p, 0, PAGE_SIZE)`.\n3. **No uninitialized memory risks:** The `struct kpkeys_state` used in `scoped_guard` is a local variable and is not copied to user space. The conditional initialization in `arch_kpkeys_enter_context` is safe because the uninitialized fields are never read if `entered_context` is false.\n\nThe changes primarily involve memory protection and control flow around page table modifications. Any bugs introduced would likely be out-of-bounds accesses, use-after-free, or invalid memory accesses, which are well-covered by KASAN and standard bug detectors. There is no risk of uninitialized memory usage or info-leaks that would require KMSAN.",
  "NeedsKMSAN": false
}

LLM Calls Summary:
Total Calls	Total Tokens	Avg Tokens	Total Duration (Seconds)	Avg Duration (Seconds)

Tool Calls Summary:
Total Calls	Total Duration (Seconds)	Avg Duration (Seconds)