The tmpfs.rst doc references the has_transparent_hugepage() helper, which is an implementation detail in the kernel and not relevant for users wishing to properly configure THP support for tmpfs. Remove it. Acked-by: David Hildenbrand (Red Hat) Reviewed-by: Baolin Wang Reviewed-by: Lance Yang Reviewed-by: Zi Yan Reviewed-by: Dev Jain Signed-off-by: Luiz Capitulino --- Documentation/filesystems/tmpfs.rst | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/Documentation/filesystems/tmpfs.rst b/Documentation/filesystems/tmpfs.rst index d677e0428c3f..46fc986c3388 100644 --- a/Documentation/filesystems/tmpfs.rst +++ b/Documentation/filesystems/tmpfs.rst @@ -109,9 +109,8 @@ noswap Disables swap. Remounts must respect the original settings. ====== =========================================================== tmpfs also supports Transparent Huge Pages which requires a kernel -configured with CONFIG_TRANSPARENT_HUGEPAGE and with huge supported for -your system (has_transparent_hugepage(), which is architecture specific). -The mount options for this are: +configured with CONFIG_TRANSPARENT_HUGEPAGE and with huge pages +supported for your system. The mount options for this are: ================ ============================================================== huge=never Do not allocate huge pages. This is the default. -- 2.55.0 Today, shmem_getattr() sets stat->blksize to PMD size whenever shmem_huge_global_enabled() returns non-zero. While this works fine for the normal THP-enabled case as explained by Baolin in [1], this has two problems: 1. Theoretically, when shmem is configured for within_size, this could set blksize to PMD size even though the allocation may be a smaller mTHP order 2. A future commit will allow shmem THP support to be enabled even when the CPU doesn't support PMD-sized pages. We should not allow blksize to be set to PMD size in this case In order to fix #1 and prepare for #2, this commit sets blksize to the size of the highest supported order returned by shmem_huge_global_enabled(). [1] https://lore.kernel.org/linux-mm/6591a74c-7ef9-4614-9ae9-cb2fbed86ebf@linux.alibaba.com/ Suggested-by: Baolin Wang Acked-by: Zi Yan Signed-off-by: Luiz Capitulino --- mm/shmem.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index b572c60f2af8..776dff8a848e 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1506,6 +1506,7 @@ static int shmem_getattr(struct mnt_idmap *idmap, { struct inode *inode = path->dentry->d_inode; struct shmem_inode_info *info = SHMEM_I(inode); + unsigned int orders; /* Fast-path hint; recalc under info->lock corrects any stale read. */ if (data_race(info->alloced - info->swapped != inode->i_mapping->nrpages)) @@ -1522,8 +1523,9 @@ static int shmem_getattr(struct mnt_idmap *idmap, STATX_ATTR_NODUMP); generic_fillattr(idmap, request_mask, inode, stat); - if (shmem_huge_global_enabled(inode, 0, 0, false, NULL, 0)) - stat->blksize = HPAGE_PMD_SIZE; + orders = shmem_huge_global_enabled(inode, 0, 0, false, NULL, 0); + if (orders) + stat->blksize = PAGE_SIZE << highest_order(orders); if (request_mask & STATX_BTIME) { stat->result_mask |= STATX_BTIME; -- 2.55.0 Currently, we have two helpers that check for PMD-sized pages but have different names and slightly different semantics: - has_transparent_hugepage(): the name suggests it checks if THP is enabled, but when CONFIG_TRANSPARENT_HUGEPAGE=y and the architecture implements this helper, it actually checks if the CPU supports PMD-sized pages - thp_disabled_by_hw(): the name suggests it checks if THP is disabled by the hardware, but it just returns a cached value acquired with has_transparent_hugepage() during boot. This helper is used in fast paths A better design would be to separate CONFIG_TRANSPARENT_HUGEPAGE checking, which can be done with the IS_ENABLED() macro, from checking if a CPU supports PMD-sized pages. To this end, this commit introduces a new helper called pgtable_has_pmd_leaves() which offers the following advantages: 1. Well defined and clear semantics: it returns true if the CPU supports PMD-sized pages and false otherwise 2. It always returns a cached value, so it can be used in fast paths 3. It's implemented with a static key: it's a no-op for archs not implemeting it and for archs implementing it the static key is changed only during boot The new helper requires an initialization step which is performed by pgtable_leaf_support_init(). We call pgtable_leaf_support_init() early during boot from mm_core_init(). The next commits will convert users of both has_transparent_hugepage() and thp_disabled_by_hw() to pgtable_has_pmd_leaves() and/or IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE). Signed-off-by: Luiz Capitulino --- include/linux/pgtable.h | 17 +++++++++++++++++ mm/memory.c | 9 +++++++++ mm/mm_init.c | 1 + 3 files changed, 27 insertions(+) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index e3c8ab96941c..ad852a415114 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -14,6 +14,7 @@ #include #include #include +#include #include #include @@ -2327,6 +2328,22 @@ void ptval_bytes_to_hex_str(char *buf, size_t buf_size, const void *entry, size_ #else #define PTVAL_STR_MAX (16 + 1) /* Max 64-bit value in hex + NUL */ #endif + +#ifdef CONFIG_MMU +DECLARE_STATIC_KEY_TRUE(__arch_has_pmd_leaves_key); +static inline bool pgtable_has_pmd_leaves(void) +{ + return static_branch_likely(&__arch_has_pmd_leaves_key); +} +void __init pgtable_leaf_support_init(void); +#else +static inline bool pgtable_has_pmd_leaves(void) +{ + return false; +} +static inline void __init pgtable_leaf_support_init(void) { } +#endif + #endif /* !__ASSEMBLER__ */ #if !defined(MAX_POSSIBLE_PHYSMEM_BITS) && !defined(CONFIG_64BIT) diff --git a/mm/memory.c b/mm/memory.c index 9e4a70421a6b..900539acb7d9 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -164,6 +164,15 @@ __setup("norandmaps", disable_randmaps); unsigned long highest_memmap_pfn __read_mostly; +DEFINE_STATIC_KEY_TRUE(__arch_has_pmd_leaves_key); +EXPORT_SYMBOL(__arch_has_pmd_leaves_key); + +void __init pgtable_leaf_support_init(void) +{ + if (!has_transparent_hugepage()) + static_branch_disable(&__arch_has_pmd_leaves_key); +} + void mm_trace_rss_stat(struct mm_struct *mm, int member) { trace_rss_stat(mm, member); diff --git a/mm/mm_init.c b/mm/mm_init.c index 56bb4567a494..28685f766234 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -2693,6 +2693,7 @@ void __init mm_core_init(void) { arch_mm_preinit(); init_zero_page_pfn(); + pgtable_leaf_support_init(); /* Initializations relying on SMP setup */ BUILD_BUG_ON(MAX_ZONELISTS > 2); -- 2.55.0 dax_align_valid() uses has_transparent_hugepage() to check if PMD-sized pages are supported, use pgtable_has_pmd_leaves() instead. Acked-by: David Hildenbrand (Red Hat) Signed-off-by: Luiz Capitulino --- drivers/dax/dax-private.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/dax/dax-private.h b/drivers/dax/dax-private.h index 607a53a91f58..18a6295aa766 100644 --- a/drivers/dax/dax-private.h +++ b/drivers/dax/dax-private.h @@ -121,7 +121,7 @@ static inline bool dax_align_valid(unsigned long align) { if (align == PUD_SIZE && IS_ENABLED(CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD)) return true; - if (align == PMD_SIZE && has_transparent_hugepage()) + if (align == PMD_SIZE && pgtable_has_pmd_leaves()) return true; if (align == PAGE_SIZE) return true; -- 2.55.0 nd_pfn_supported_alignments() and nd_pfn_default_alignment() use has_transparent_hugepage() to check if THP is supported with PMD-sized pages. Use pgtable_has_pmd_leaves() instead. Also, check for IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) to preserve the current implementation semantics. Acked-by: Zi Yan Signed-off-by: Luiz Capitulino --- drivers/nvdimm/pfn_devs.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/drivers/nvdimm/pfn_devs.c b/drivers/nvdimm/pfn_devs.c index 8fa9c16aba7e..457eb54e7ab6 100644 --- a/drivers/nvdimm/pfn_devs.c +++ b/drivers/nvdimm/pfn_devs.c @@ -94,7 +94,8 @@ static unsigned long *nd_pfn_supported_alignments(unsigned long *alignments) alignments[0] = PAGE_SIZE; - if (has_transparent_hugepage()) { + if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && + pgtable_has_pmd_leaves()) { alignments[1] = HPAGE_PMD_SIZE; if (has_transparent_pud_hugepage()) alignments[2] = HPAGE_PUD_SIZE; @@ -109,7 +110,8 @@ static unsigned long *nd_pfn_supported_alignments(unsigned long *alignments) static unsigned long nd_pfn_default_alignment(void) { - if (has_transparent_hugepage()) + if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && + pgtable_has_pmd_leaves()) return HPAGE_PMD_SIZE; return PAGE_SIZE; } -- 2.55.0 debug_vm_pgtable calls has_transparent_hugepage() in multiple places to check if PMD-sized pages are supported, use pgtable_has_pmd_leaves() instead. Reviewed-by: Lance Yang Reviewed-by: Zi Yan Signed-off-by: Luiz Capitulino --- mm/debug_vm_pgtable.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/mm/debug_vm_pgtable.c b/mm/debug_vm_pgtable.c index 2875fd22d7bb..25038b74c7bc 100644 --- a/mm/debug_vm_pgtable.c +++ b/mm/debug_vm_pgtable.c @@ -177,7 +177,7 @@ static void __init pmd_basic_tests(struct pgtable_debug_args *args, int idx) unsigned long val = idx, *ptr = &val; pmd_t pmd; - if (!has_transparent_hugepage()) + if (!pgtable_has_pmd_leaves()) return; pr_debug("Validating PMD basic (%pGv)\n", ptr); @@ -222,7 +222,7 @@ static void __init pmd_advanced_tests(struct pgtable_debug_args *args) pmd_t pmd; unsigned long vaddr = args->vaddr; - if (!has_transparent_hugepage()) + if (!pgtable_has_pmd_leaves()) return; page = (args->pmd_pfn != ULONG_MAX) ? pfn_to_page(args->pmd_pfn) : NULL; @@ -283,7 +283,7 @@ static void __init pmd_leaf_tests(struct pgtable_debug_args *args) { pmd_t pmd; - if (!has_transparent_hugepage()) + if (!pgtable_has_pmd_leaves()) return; pr_debug("Validating PMD leaf\n"); @@ -688,7 +688,7 @@ static void __init pmd_protnone_tests(struct pgtable_debug_args *args) if (!IS_ENABLED(CONFIG_ARCH_HAS_PTE_PROTNONE)) return; - if (!has_transparent_hugepage()) + if (!pgtable_has_pmd_leaves()) return; pr_debug("Validating PMD protnone\n"); @@ -737,7 +737,7 @@ static void __init pmd_soft_dirty_tests(struct pgtable_debug_args *args) if (!pgtable_supports_soft_dirty()) return; - if (!has_transparent_hugepage()) + if (!pgtable_has_pmd_leaves()) return; pr_debug("Validating PMD soft dirty\n"); @@ -754,7 +754,7 @@ static void __init pmd_leaf_soft_dirty_tests(struct pgtable_debug_args *args) !IS_ENABLED(CONFIG_ARCH_HAS_PMD_SOFTLEAVES)) return; - if (!has_transparent_hugepage()) + if (!pgtable_has_pmd_leaves()) return; pr_debug("Validating PMD swap soft dirty\n"); @@ -825,7 +825,7 @@ static void __init pmd_softleaf_tests(struct pgtable_debug_args *args) swp_entry_t arch_entry; pmd_t pmd1, pmd2; - if (!has_transparent_hugepage()) + if (!pgtable_has_pmd_leaves()) return; pr_debug("Validating PMD swap\n"); @@ -906,7 +906,7 @@ static void __init pmd_thp_tests(struct pgtable_debug_args *args) { pmd_t pmd; - if (!has_transparent_hugepage()) + if (!pgtable_has_pmd_leaves()) return; pr_debug("Validating PMD based THP\n"); @@ -997,7 +997,7 @@ static void __init destroy_args(struct pgtable_debug_args *args) } if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && - has_transparent_hugepage() && + pgtable_has_pmd_leaves() && args->pmd_pfn != ULONG_MAX) { debug_vm_pgtable_free_huge_page(args, args->pmd_pfn, HPAGE_PMD_ORDER); args->pmd_pfn = ULONG_MAX; @@ -1249,7 +1249,7 @@ static int __init init_args(struct pgtable_debug_args *args) } if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && - has_transparent_hugepage()) { + pgtable_has_pmd_leaves()) { page = debug_vm_pgtable_alloc_huge_page(args, HPAGE_PMD_ORDER); if (page) { args->pmd_pfn = page_to_pfn(page); -- 2.55.0 In order to enable THP support in shmem today, besides the user configuration required, the CPU must support PMD-sized pages. This is the case because of the following has_transparent_hugepage() usage: - shmem_parse_one() and shmem_parse_huge(): Check if THP is built-in and if the CPU supports PMD-sized pages - shmem_init(): Since the CONFIG_TRANSPARENT_HUGEPAGE guard is outside the code block calling has_transparent_hugepage(), the has_transparent_hugepage() call is exclusively checking if the CPU supports PMD-sized pages While it's necessary to check if CONFIG_TRANSPARENT_HUGEPAGE is enabled in all cases, shmem can determine THP size support at folio allocation time. Therefore, drop the has_transparent_hugepage() usage listed above while keeping the CONFIG_TRANSPARENT_HUGEPAGE checks. Additionally, we need to check if PMD size order is supported in shmem_getattr(). Use pgtable_has_pmd_leaves() for that. Reviewed-by: Baolin Wang Signed-off-by: Luiz Capitulino --- mm/shmem.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index 776dff8a848e..930657d05375 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -690,7 +690,7 @@ static int shmem_parse_huge(const char *str) else return -EINVAL; - if (!has_transparent_hugepage() && + if (!IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && huge != SHMEM_HUGE_NEVER && huge != SHMEM_HUGE_DENY) return -EINVAL; @@ -1524,6 +1524,8 @@ static int shmem_getattr(struct mnt_idmap *idmap, generic_fillattr(idmap, request_mask, inode, stat); orders = shmem_huge_global_enabled(inode, 0, 0, false, NULL, 0); + if (!pgtable_has_pmd_leaves()) + orders &= ~BIT(PMD_ORDER); if (orders) stat->blksize = PAGE_SIZE << highest_order(orders); @@ -4814,8 +4816,7 @@ static int shmem_parse_one(struct fs_context *fc, struct fs_parameter *param) case Opt_huge: ctx->huge = result.uint_32; if (ctx->huge != SHMEM_HUGE_NEVER && - !(IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && - has_transparent_hugepage())) + !IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) goto unsupported_parameter; ctx->seen |= SHMEM_SEEN_HUGE; break; @@ -5620,7 +5621,7 @@ void __init shmem_init(void) #endif #ifdef CONFIG_TRANSPARENT_HUGEPAGE - if (has_transparent_hugepage() && shmem_huge > SHMEM_HUGE_DENY) + if (shmem_huge > SHMEM_HUGE_DENY) SHMEM_SB(shm_mnt->mnt_sb)->huge = shmem_huge; else shmem_huge = SHMEM_HUGE_NEVER; /* just in case it was patched */ -- 2.55.0 A future commit will introduce a kernel API to allow for checking if the CPU supports PMD-sized pages. This API will be based on the has_transparent_hugepage() implementation but will be orthogonal to THP and therefore must work when CONFIG_TRANSPARENT_HUGEPAGE=n. Move its definition out of the THP guard. Signed-off-by: Luiz Capitulino --- arch/s390/include/asm/pgtable.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h index 2d5c2ab06de9..e7a2bf9636b7 100644 --- a/arch/s390/include/asm/pgtable.h +++ b/arch/s390/include/asm/pgtable.h @@ -1814,13 +1814,13 @@ static inline int pmd_trans_huge(pmd_t pmd) { return pmd_leaf(pmd); } +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ #define has_transparent_hugepage has_transparent_hugepage static inline int has_transparent_hugepage(void) { return cpu_has_edat1() ? 1 : 0; } -#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ #ifdef CONFIG_PAGE_TABLE_CHECK static inline bool pte_user_accessible_page(struct mm_struct *mm, unsigned long addr, pte_t pte) -- 2.55.0 A future commit will introduce a kernel API to allow for checking if the CPU supports PMD-sized pages. This API will be based on the has_transparent_hugepage() implementation but will be orthogonal to THP and therefore must work when CONFIG_TRANSPARENT_HUGEPAGE=n. Move its definition out of the THP guard. Signed-off-by: Luiz Capitulino --- arch/powerpc/include/asm/book3s/64/hash-4k.h | 2 +- arch/powerpc/include/asm/book3s/64/hash-64k.h | 2 +- arch/powerpc/include/asm/book3s/64/pgtable.h | 18 +++++++++--------- arch/powerpc/include/asm/book3s/64/radix.h | 14 +++++++------- arch/powerpc/mm/book3s64/hash_pgtable.c | 4 ++-- 5 files changed, 20 insertions(+), 20 deletions(-) diff --git a/arch/powerpc/include/asm/book3s/64/hash-4k.h b/arch/powerpc/include/asm/book3s/64/hash-4k.h index 8e5bd9902bed..79511e6abfca 100644 --- a/arch/powerpc/include/asm/book3s/64/hash-4k.h +++ b/arch/powerpc/include/asm/book3s/64/hash-4k.h @@ -165,9 +165,9 @@ extern void hash__pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp, extern pgtable_t hash__pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp); extern pmd_t hash__pmdp_huge_get_and_clear(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp); -extern int hash__has_transparent_hugepage(void); #endif +extern int hash__has_transparent_hugepage(void); #endif /* !__ASSEMBLER__ */ #endif /* _ASM_POWERPC_BOOK3S_64_HASH_4K_H */ diff --git a/arch/powerpc/include/asm/book3s/64/hash-64k.h b/arch/powerpc/include/asm/book3s/64/hash-64k.h index 7deb3a66890b..a4a44a112ff9 100644 --- a/arch/powerpc/include/asm/book3s/64/hash-64k.h +++ b/arch/powerpc/include/asm/book3s/64/hash-64k.h @@ -278,9 +278,9 @@ extern void hash__pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp, extern pgtable_t hash__pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp); extern pmd_t hash__pmdp_huge_get_and_clear(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp); -extern int hash__has_transparent_hugepage(void); #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ +extern int hash__has_transparent_hugepage(void); #endif /* __ASSEMBLER__ */ #endif /* _ASM_POWERPC_BOOK3S_64_HASH_64K_H */ diff --git a/arch/powerpc/include/asm/book3s/64/pgtable.h b/arch/powerpc/include/asm/book3s/64/pgtable.h index dff8790a047d..522bb58b6241 100644 --- a/arch/powerpc/include/asm/book3s/64/pgtable.h +++ b/arch/powerpc/include/asm/book3s/64/pgtable.h @@ -1114,15 +1114,6 @@ static inline void update_mmu_cache_pud(struct vm_area_struct *vma, { } -extern int hash__has_transparent_hugepage(void); -static inline int has_transparent_hugepage(void) -{ - if (radix_enabled()) - return radix__has_transparent_hugepage(); - return hash__has_transparent_hugepage(); -} -#define has_transparent_hugepage has_transparent_hugepage - static inline int has_transparent_pud_hugepage(void) { if (radix_enabled()) @@ -1434,6 +1425,15 @@ static inline bool arch_needs_pgtable_deposit(void) #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ +extern int hash__has_transparent_hugepage(void); +static inline int has_transparent_hugepage(void) +{ + if (radix_enabled()) + return radix__has_transparent_hugepage(); + return hash__has_transparent_hugepage(); +} +#define has_transparent_hugepage has_transparent_hugepage + #define __HAVE_ARCH_PTEP_MODIFY_PROT_TRANSACTION pte_t ptep_modify_prot_start(struct vm_area_struct *, unsigned long, pte_t *); void ptep_modify_prot_commit(struct vm_area_struct *, unsigned long, diff --git a/arch/powerpc/include/asm/book3s/64/radix.h b/arch/powerpc/include/asm/book3s/64/radix.h index da954e779744..50545cf519bd 100644 --- a/arch/powerpc/include/asm/book3s/64/radix.h +++ b/arch/powerpc/include/asm/book3s/64/radix.h @@ -298,22 +298,22 @@ extern pmd_t radix__pmdp_huge_get_and_clear(struct mm_struct *mm, pud_t radix__pudp_huge_get_and_clear(struct mm_struct *mm, unsigned long addr, pud_t *pudp); -static inline int radix__has_transparent_hugepage(void) +static inline int radix__has_transparent_pud_hugepage(void) { - /* For radix 2M at PMD level means thp */ - if (mmu_psize_defs[MMU_PAGE_2M].shift == PMD_SHIFT) + /* For radix 1G at PUD level means pud hugepage support */ + if (mmu_psize_defs[MMU_PAGE_1G].shift == PUD_SHIFT) return 1; return 0; } +#endif -static inline int radix__has_transparent_pud_hugepage(void) +static inline int radix__has_transparent_hugepage(void) { - /* For radix 1G at PUD level means pud hugepage support */ - if (mmu_psize_defs[MMU_PAGE_1G].shift == PUD_SHIFT) + /* For radix 2M at PMD level means thp */ + if (mmu_psize_defs[MMU_PAGE_2M].shift == PMD_SHIFT) return 1; return 0; } -#endif struct vmem_altmap; struct dev_pagemap; diff --git a/arch/powerpc/mm/book3s64/hash_pgtable.c b/arch/powerpc/mm/book3s64/hash_pgtable.c index d9b5b751d7b7..50316f3fc5d3 100644 --- a/arch/powerpc/mm/book3s64/hash_pgtable.c +++ b/arch/powerpc/mm/book3s64/hash_pgtable.c @@ -391,6 +391,8 @@ pmd_t hash__pmdp_huge_get_and_clear(struct mm_struct *mm, return old_pmd; } +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ + int hash__has_transparent_hugepage(void) { @@ -422,8 +424,6 @@ int hash__has_transparent_hugepage(void) } EXPORT_SYMBOL_GPL(hash__has_transparent_hugepage); -#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ - #ifdef CONFIG_STRICT_KERNEL_RWX struct change_memory_parms { -- 2.55.0 A future commit will introduce a kernel API to allow for checking if the CPU supports PMD-sized pages. This API will be based on the has_transparent_hugepage() implementation but will be orthogonal to THP and therefore must work when CONFIG_TRANSPARENT_HUGEPAGE=n. This commit moves has_transparent_hugepage() definition to arch/mips/mm/pgtable.c under CONFIG_CPU_SUPPORTS_HUGEPAGES guard in order to ensure that it's only used in CPUs that support the register operations performed. Signed-off-by: Luiz Capitulino --- arch/mips/include/asm/pgtable.h | 6 +++--- arch/mips/mm/pgtable.c | 25 +++++++++++++++++++++++++ arch/mips/mm/tlb-r4k.c | 22 ---------------------- 3 files changed, 28 insertions(+), 25 deletions(-) diff --git a/arch/mips/include/asm/pgtable.h b/arch/mips/include/asm/pgtable.h index fa7b935f947c..b038da872ec6 100644 --- a/arch/mips/include/asm/pgtable.h +++ b/arch/mips/include/asm/pgtable.h @@ -615,9 +615,6 @@ unsigned long io_remap_pfn_range_pfn(unsigned long pfn, unsigned long size); /* We don't have hardware dirty/accessed bits, generic_pmdp_establish is fine.*/ #define pmdp_establish generic_pmdp_establish -#define has_transparent_hugepage has_transparent_hugepage -extern int has_transparent_hugepage(void); - static inline int pmd_trans_huge(pmd_t pmd) { return !!(pmd_val(pmd) & _PAGE_HUGE); @@ -743,6 +740,9 @@ static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ +#define has_transparent_hugepage has_transparent_hugepage +extern int has_transparent_hugepage(void); + #ifdef _PAGE_HUGE #define pmd_leaf(pmd) ((pmd_val(pmd) & _PAGE_HUGE) != 0) #define pud_leaf(pud) ((pud_val(pud) & _PAGE_HUGE) != 0) diff --git a/arch/mips/mm/pgtable.c b/arch/mips/mm/pgtable.c index 10835414819f..4515fa42f0b3 100644 --- a/arch/mips/mm/pgtable.c +++ b/arch/mips/mm/pgtable.c @@ -23,3 +23,28 @@ pgd_t *pgd_alloc(struct mm_struct *mm) return ret; } EXPORT_SYMBOL_GPL(pgd_alloc); + +#ifdef CONFIG_CPU_SUPPORTS_HUGEPAGES +int has_transparent_hugepage(void) +{ + static unsigned int mask = -1; + + if (mask == -1) { /* first call comes during __init */ + unsigned long flags; + + local_irq_save(flags); + write_c0_pagemask(PM_HUGE_MASK); + back_to_back_c0_hazard(); + mask = read_c0_pagemask(); + write_c0_pagemask(PM_DEFAULT_MASK); + local_irq_restore(flags); + } + return mask == PM_HUGE_MASK; +} +#else +int has_transparent_hugepage(void) +{ + return 0; +} +#endif +EXPORT_SYMBOL(has_transparent_hugepage); diff --git a/arch/mips/mm/tlb-r4k.c b/arch/mips/mm/tlb-r4k.c index 24fe85fa169d..625f1c0dd71e 100644 --- a/arch/mips/mm/tlb-r4k.c +++ b/arch/mips/mm/tlb-r4k.c @@ -432,28 +432,6 @@ void add_wired_entry(unsigned long entrylo0, unsigned long entrylo1, #endif } -#ifdef CONFIG_TRANSPARENT_HUGEPAGE - -int has_transparent_hugepage(void) -{ - static unsigned int mask = -1; - - if (mask == -1) { /* first call comes during __init */ - unsigned long flags; - - local_irq_save(flags); - write_c0_pagemask(PM_HUGE_MASK); - back_to_back_c0_hazard(); - mask = read_c0_pagemask(); - write_c0_pagemask(PM_DEFAULT_MASK); - local_irq_restore(flags); - } - return mask == PM_HUGE_MASK; -} -EXPORT_SYMBOL(has_transparent_hugepage); - -#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ - /* * Used for loading TLB entries before trap_init() has started, when we * don't actually want to add a wired entry which remains throughout the -- 2.55.0 A future commit will introduce a kernel API to allow for checking if the CPU supports PMD-sized pages. This API will be based on the has_transparent_hugepage() implementation but will be orthogonal to THP and therefore must work when CONFIG_TRANSPARENT_HUGEPAGE=n. Move its definition out of the THP guard. Signed-off-by: Luiz Capitulino --- arch/x86/include/asm/pgtable.h | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index d551120a7c88..f46c1b502eb1 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -306,12 +306,6 @@ static inline int pud_trans_huge(pud_t pud) } #endif -#define has_transparent_hugepage has_transparent_hugepage -static inline int has_transparent_hugepage(void) -{ - return boot_cpu_has(X86_FEATURE_PSE); -} - #ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP static inline bool pmd_special(pmd_t pmd) { @@ -337,6 +331,12 @@ static inline pud_t pud_mkspecial(pud_t pud) #endif /* CONFIG_ARCH_SUPPORTS_PUD_PFNMAP */ #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ +#define has_transparent_hugepage has_transparent_hugepage +static inline int has_transparent_hugepage(void) +{ + return boot_cpu_has(X86_FEATURE_PSE); +} + static inline pte_t pte_set_flags(pte_t pte, pteval_t set) { pteval_t v = native_pte_val(pte); -- 2.55.0 Now that all the has_transparent_hugepage() callers have been converted to pgtable_has_pmd_leaves(), this commit does two things: 1. Rename has_transparent_hugepage() arch implementations to arch_has_pmd_leaves(), since that's what the helper checks for 2. Introduce the default implementation of arch_has_pmd_leaves() as IS_ENABLED(CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE). This means that if the arch doesn't implement arch_has_pmd_leaves() we default to checking CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE as a way to determine if PMD-sized pages are supported Note that arch_has_pmd_leaves() is supposed to be called only by pgtable_leaf_support_init(). The remaining exception is hugepage_init() which will be converted in a future commit. Signed-off-by: Luiz Capitulino --- arch/mips/include/asm/pgtable.h | 4 ++-- arch/mips/mm/pgtable.c | 6 +++--- arch/powerpc/include/asm/book3s/64/hash-4k.h | 2 +- arch/powerpc/include/asm/book3s/64/hash-64k.h | 2 +- arch/powerpc/include/asm/book3s/64/pgtable.h | 10 +++++----- arch/powerpc/include/asm/book3s/64/radix.h | 2 +- arch/powerpc/mm/book3s64/hash_pgtable.c | 4 ++-- arch/s390/include/asm/pgtable.h | 4 ++-- arch/x86/include/asm/pgtable.h | 4 ++-- include/linux/pgtable.h | 4 ++-- mm/huge_memory.c | 2 +- mm/memory.c | 2 +- 12 files changed, 23 insertions(+), 23 deletions(-) diff --git a/arch/mips/include/asm/pgtable.h b/arch/mips/include/asm/pgtable.h index b038da872ec6..2a0a7ddf8bee 100644 --- a/arch/mips/include/asm/pgtable.h +++ b/arch/mips/include/asm/pgtable.h @@ -740,8 +740,8 @@ static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ -#define has_transparent_hugepage has_transparent_hugepage -extern int has_transparent_hugepage(void); +#define arch_has_pmd_leaves arch_has_pmd_leaves +extern int arch_has_pmd_leaves(void); #ifdef _PAGE_HUGE #define pmd_leaf(pmd) ((pmd_val(pmd) & _PAGE_HUGE) != 0) diff --git a/arch/mips/mm/pgtable.c b/arch/mips/mm/pgtable.c index 4515fa42f0b3..f498fc58bd7a 100644 --- a/arch/mips/mm/pgtable.c +++ b/arch/mips/mm/pgtable.c @@ -25,7 +25,7 @@ pgd_t *pgd_alloc(struct mm_struct *mm) EXPORT_SYMBOL_GPL(pgd_alloc); #ifdef CONFIG_CPU_SUPPORTS_HUGEPAGES -int has_transparent_hugepage(void) +int arch_has_pmd_leaves(void) { static unsigned int mask = -1; @@ -42,9 +42,9 @@ int has_transparent_hugepage(void) return mask == PM_HUGE_MASK; } #else -int has_transparent_hugepage(void) +int arch_has_pmd_leaves(void) { return 0; } #endif -EXPORT_SYMBOL(has_transparent_hugepage); +EXPORT_SYMBOL(arch_has_pmd_leaves); diff --git a/arch/powerpc/include/asm/book3s/64/hash-4k.h b/arch/powerpc/include/asm/book3s/64/hash-4k.h index 79511e6abfca..d532e3f0dfc3 100644 --- a/arch/powerpc/include/asm/book3s/64/hash-4k.h +++ b/arch/powerpc/include/asm/book3s/64/hash-4k.h @@ -167,7 +167,7 @@ extern pmd_t hash__pmdp_huge_get_and_clear(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp); #endif -extern int hash__has_transparent_hugepage(void); +extern int hash__arch_has_pmd_leaves(void); #endif /* !__ASSEMBLER__ */ #endif /* _ASM_POWERPC_BOOK3S_64_HASH_4K_H */ diff --git a/arch/powerpc/include/asm/book3s/64/hash-64k.h b/arch/powerpc/include/asm/book3s/64/hash-64k.h index a4a44a112ff9..d523c80f44f7 100644 --- a/arch/powerpc/include/asm/book3s/64/hash-64k.h +++ b/arch/powerpc/include/asm/book3s/64/hash-64k.h @@ -280,7 +280,7 @@ extern pmd_t hash__pmdp_huge_get_and_clear(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp); #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ -extern int hash__has_transparent_hugepage(void); +extern int hash__arch_has_pmd_leaves(void); #endif /* __ASSEMBLER__ */ #endif /* _ASM_POWERPC_BOOK3S_64_HASH_64K_H */ diff --git a/arch/powerpc/include/asm/book3s/64/pgtable.h b/arch/powerpc/include/asm/book3s/64/pgtable.h index 522bb58b6241..88c417e3f016 100644 --- a/arch/powerpc/include/asm/book3s/64/pgtable.h +++ b/arch/powerpc/include/asm/book3s/64/pgtable.h @@ -1425,14 +1425,14 @@ static inline bool arch_needs_pgtable_deposit(void) #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ -extern int hash__has_transparent_hugepage(void); -static inline int has_transparent_hugepage(void) +extern int hash__arch_has_pmd_leaves(void); +static inline int arch_has_pmd_leaves(void) { if (radix_enabled()) - return radix__has_transparent_hugepage(); - return hash__has_transparent_hugepage(); + return radix__arch_has_pmd_leaves(); + return hash__arch_has_pmd_leaves(); } -#define has_transparent_hugepage has_transparent_hugepage +#define arch_has_pmd_leaves arch_has_pmd_leaves #define __HAVE_ARCH_PTEP_MODIFY_PROT_TRANSACTION pte_t ptep_modify_prot_start(struct vm_area_struct *, unsigned long, pte_t *); diff --git a/arch/powerpc/include/asm/book3s/64/radix.h b/arch/powerpc/include/asm/book3s/64/radix.h index 50545cf519bd..e8b74528953f 100644 --- a/arch/powerpc/include/asm/book3s/64/radix.h +++ b/arch/powerpc/include/asm/book3s/64/radix.h @@ -307,7 +307,7 @@ static inline int radix__has_transparent_pud_hugepage(void) } #endif -static inline int radix__has_transparent_hugepage(void) +static inline int radix__arch_has_pmd_leaves(void) { /* For radix 2M at PMD level means thp */ if (mmu_psize_defs[MMU_PAGE_2M].shift == PMD_SHIFT) diff --git a/arch/powerpc/mm/book3s64/hash_pgtable.c b/arch/powerpc/mm/book3s64/hash_pgtable.c index 50316f3fc5d3..a67bc1723404 100644 --- a/arch/powerpc/mm/book3s64/hash_pgtable.c +++ b/arch/powerpc/mm/book3s64/hash_pgtable.c @@ -393,7 +393,7 @@ pmd_t hash__pmdp_huge_get_and_clear(struct mm_struct *mm, #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ -int hash__has_transparent_hugepage(void) +int hash__arch_has_pmd_leaves(void) { if (!mmu_has_feature(MMU_FTR_16M_PAGE)) @@ -422,7 +422,7 @@ int hash__has_transparent_hugepage(void) return 1; } -EXPORT_SYMBOL_GPL(hash__has_transparent_hugepage); +EXPORT_SYMBOL_GPL(hash__arch_has_pmd_leaves); #ifdef CONFIG_STRICT_KERNEL_RWX diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h index e7a2bf9636b7..d4a7c938e2e4 100644 --- a/arch/s390/include/asm/pgtable.h +++ b/arch/s390/include/asm/pgtable.h @@ -1816,8 +1816,8 @@ static inline int pmd_trans_huge(pmd_t pmd) } #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ -#define has_transparent_hugepage has_transparent_hugepage -static inline int has_transparent_hugepage(void) +#define arch_has_pmd_leaves arch_has_pmd_leaves +static inline int arch_has_pmd_leaves(void) { return cpu_has_edat1() ? 1 : 0; } diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index f46c1b502eb1..b4de9068d461 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -331,8 +331,8 @@ static inline pud_t pud_mkspecial(pud_t pud) #endif /* CONFIG_ARCH_SUPPORTS_PUD_PFNMAP */ #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ -#define has_transparent_hugepage has_transparent_hugepage -static inline int has_transparent_hugepage(void) +#define arch_has_pmd_leaves arch_has_pmd_leaves +static inline int arch_has_pmd_leaves(void) { return boot_cpu_has(X86_FEATURE_PSE); } diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index ad852a415114..fd2c73335604 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -2359,8 +2359,8 @@ static inline void __init pgtable_leaf_support_init(void) { } #endif #endif -#ifndef has_transparent_hugepage -#define has_transparent_hugepage() IS_BUILTIN(CONFIG_TRANSPARENT_HUGEPAGE) +#ifndef arch_has_pmd_leaves +#define arch_has_pmd_leaves() IS_ENABLED(CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE) #endif #ifndef has_transparent_pud_hugepage diff --git a/mm/huge_memory.c b/mm/huge_memory.c index b49bffe36d22..c9ee387a59f9 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1091,7 +1091,7 @@ static int __init hugepage_init(void) int err; struct kobject *hugepage_kobj; - if (!has_transparent_hugepage()) { + if (!arch_has_pmd_leaves()) { transparent_hugepage_flags = 1 << TRANSPARENT_HUGEPAGE_UNSUPPORTED; return -EINVAL; } diff --git a/mm/memory.c b/mm/memory.c index 900539acb7d9..cbd9082d33a0 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -169,7 +169,7 @@ EXPORT_SYMBOL(__arch_has_pmd_leaves_key); void __init pgtable_leaf_support_init(void) { - if (!has_transparent_hugepage()) + if (!arch_has_pmd_leaves()) static_branch_disable(&__arch_has_pmd_leaves_key); } -- 2.55.0 Despite its name, thp_disabled_by_hw() just checks whether the architecture supports PMD-sized pages. It returns true when TRANSPARENT_HUGEPAGE_UNSUPPORTED is set in transparent_hugepage_flags, this only occurs if the architecture implements arch_has_pmd_leaves() and that function returns false. Since pgtable_has_pmd_leaves() provides the same semantics, use it instead. Reviewed-by: Lance Yang Reviewed-by: Baolin Wang Acked-by: Zi Yan Acked-by: David Hildenbrand (Arm) Signed-off-by: Luiz Capitulino --- include/linux/huge_mm.h | 7 ------- mm/huge_memory.c | 6 ++---- mm/memory.c | 2 +- mm/shmem.c | 2 +- 4 files changed, 4 insertions(+), 13 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 8ca0fa3be2ac..196f17afcb5d 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -47,7 +47,6 @@ vm_fault_t vmf_insert_folio_pud(struct vm_fault *vmf, struct folio *folio, bool write); enum transparent_hugepage_flag { - TRANSPARENT_HUGEPAGE_UNSUPPORTED, TRANSPARENT_HUGEPAGE_FLAG, TRANSPARENT_HUGEPAGE_REQ_MADV_FLAG, TRANSPARENT_HUGEPAGE_DEFRAG_DIRECT_FLAG, @@ -383,12 +382,6 @@ static inline bool vma_thp_disabled(struct vm_area_struct *vma, return mm_flags_test(MMF_DISABLE_THP_EXCEPT_ADVISED, vma->vm_mm); } -static inline bool thp_disabled_by_hw(void) -{ - /* If the hardware/firmware marked hugepage support disabled. */ - return transparent_hugepage_flags & (1 << TRANSPARENT_HUGEPAGE_UNSUPPORTED); -} - unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr, unsigned long len, unsigned long pgoff, unsigned long flags); unsigned long thp_get_unmapped_area_vmaflags(struct file *filp, unsigned long addr, diff --git a/mm/huge_memory.c b/mm/huge_memory.c index c9ee387a59f9..a06025b87e7c 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -196,7 +196,7 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma, if (!vma->vm_mm) /* vdso */ return 0; - if (thp_disabled_by_hw() || vma_thp_disabled(vma, vm_flags, forced_collapse)) + if (!pgtable_has_pmd_leaves() || vma_thp_disabled(vma, vm_flags, forced_collapse)) return 0; /* khugepaged doesn't collapse DAX vma, but page fault is fine. */ @@ -1091,10 +1091,8 @@ static int __init hugepage_init(void) int err; struct kobject *hugepage_kobj; - if (!arch_has_pmd_leaves()) { - transparent_hugepage_flags = 1 << TRANSPARENT_HUGEPAGE_UNSUPPORTED; + if (!pgtable_has_pmd_leaves()) return -EINVAL; - } /* * hugepages can't be allocated by the buddy allocator diff --git a/mm/memory.c b/mm/memory.c index cbd9082d33a0..9e7d2d73fa5b 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5694,7 +5694,7 @@ vm_fault_t do_set_pmd(struct vm_fault *vmf, struct folio *folio, struct page *pa * PMD mappings if THPs are disabled. As we already have a THP, * behave as if we are forcing a collapse. */ - if (thp_disabled_by_hw() || vma_thp_disabled(vma, vma->vm_flags, + if (!pgtable_has_pmd_leaves() || vma_thp_disabled(vma, vma->vm_flags, /* forced_collapse=*/ true)) return ret; diff --git a/mm/shmem.c b/mm/shmem.c index 930657d05375..bc2de3a7c1ea 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -2048,7 +2048,7 @@ unsigned long shmem_allowable_huge_orders(struct inode *inode, vm_flags_t vm_flags = vma ? vma->vm_flags : 0; unsigned int global_orders; - if (thp_disabled_by_hw() || (vma && vma_thp_disabled(vma, vm_flags, shmem_huge_force))) + if (!pgtable_has_pmd_leaves() || (vma && vma_thp_disabled(vma, vm_flags, shmem_huge_force))) return 0; global_orders = shmem_huge_global_enabled(inode, index, write_end, -- 2.55.0 If PMD-sized pages are not supported on an architecture (ie. the arch implements arch_has_pmd_leaves() and it returns false) then the current code disables all THP, including mTHP. This commit fixes this by allowing mTHP to be always enabled for all archs. When PMD-sized pages are not supported, its sysfs entry won't be created and their mapping will be disallowed at page-fault time. Similarly, this commit implements the following changes for shmem in shmem_allowable_huge_orders(): - Drop the pgtable_has_pmd_leaves() check so that mTHP sizes are considered - Filter out PMD and PUD orders from allowable orders when PMD-sized pages are not supported by the CPU Signed-off-by: Luiz Capitulino --- mm/huge_memory.c | 25 ++++++++++++++++++++----- mm/shmem.c | 14 +++++++++----- 2 files changed, 29 insertions(+), 10 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index a06025b87e7c..a2d6de3ea988 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -189,6 +189,15 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma, else supported_orders = THP_ORDERS_ALL_FILE_DEFAULT; + if (!pgtable_has_pmd_leaves()) { + /* + * If the CPU does not support PMD leaves, assume for + * now that it does not support PUD leaves and disable + * both folio orders. + */ + supported_orders &= ~(BIT(PMD_ORDER) | BIT(PUD_ORDER)); + } + orders &= supported_orders; if (!orders) return 0; @@ -196,7 +205,7 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma, if (!vma->vm_mm) /* vdso */ return 0; - if (!pgtable_has_pmd_leaves() || vma_thp_disabled(vma, vm_flags, forced_collapse)) + if (vma_thp_disabled(vma, vm_flags, forced_collapse)) return 0; /* khugepaged doesn't collapse DAX vma, but page fault is fine. */ @@ -979,7 +988,7 @@ static int __init hugepage_init_sysfs(struct kobject **hugepage_kobj) * disable all other sizes. powerpc's PMD_ORDER isn't a compile-time * constant so we have to do this here. */ - if (!anon_orders_configured) + if (!anon_orders_configured && pgtable_has_pmd_leaves()) huge_anon_orders_inherit = BIT(PMD_ORDER); *hugepage_kobj = kobject_create_and_add("transparent_hugepage", mm_kobj); @@ -1001,6 +1010,15 @@ static int __init hugepage_init_sysfs(struct kobject **hugepage_kobj) } orders = THP_ORDERS_ALL_ANON | THP_ORDERS_ALL_FILE_DEFAULT; + if (!pgtable_has_pmd_leaves()) { + /* + * If the CPU does not support PMD leaves, assume for + * now that it does not support PUD leaves and disable + * both folio orders. + */ + orders &= ~(BIT(PMD_ORDER) | BIT(PUD_ORDER)); + } + order = highest_order(orders); while (orders) { thpsize = thpsize_create(order, *hugepage_kobj); @@ -1091,9 +1109,6 @@ static int __init hugepage_init(void) int err; struct kobject *hugepage_kobj; - if (!pgtable_has_pmd_leaves()) - return -EINVAL; - /* * hugepages can't be allocated by the buddy allocator */ diff --git a/mm/shmem.c b/mm/shmem.c index bc2de3a7c1ea..8c0f7e3efeeb 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -2046,11 +2046,14 @@ unsigned long shmem_allowable_huge_orders(struct inode *inode, unsigned long mask = READ_ONCE(huge_shmem_orders_always); unsigned long within_size_orders = READ_ONCE(huge_shmem_orders_within_size); vm_flags_t vm_flags = vma ? vma->vm_flags : 0; - unsigned int global_orders; + unsigned int global_orders, disabled_orders = 0; - if (!pgtable_has_pmd_leaves() || (vma && vma_thp_disabled(vma, vm_flags, shmem_huge_force))) + if (vma && vma_thp_disabled(vma, vm_flags, shmem_huge_force)) return 0; + if (!pgtable_has_pmd_leaves()) + disabled_orders = BIT(PMD_ORDER); + global_orders = shmem_huge_global_enabled(inode, index, write_end, shmem_huge_force, vma, vm_flags); /* @@ -2058,7 +2061,7 @@ unsigned long shmem_allowable_huge_orders(struct inode *inode, * sysfs configs. */ if (!vma || !vma_is_anon_shmem(vma) || shmem_huge_force) - return global_orders; + return global_orders & ~disabled_orders; /* * Following the 'deny' semantics of the top level, force the huge @@ -2072,7 +2075,7 @@ unsigned long shmem_allowable_huge_orders(struct inode *inode, * means non-PMD sized THP can not override 'huge' mount option now. */ if (shmem_huge == SHMEM_HUGE_FORCE) - return READ_ONCE(huge_shmem_orders_inherit); + return READ_ONCE(huge_shmem_orders_inherit) & ~disabled_orders; /* Allow mTHP that will be fully within i_size. */ mask |= shmem_get_orders_within_size(inode, within_size_orders, index, 0); @@ -2083,6 +2086,7 @@ unsigned long shmem_allowable_huge_orders(struct inode *inode, if (global_orders > 0) mask |= READ_ONCE(huge_shmem_orders_inherit); + mask &= ~disabled_orders; return THP_ORDERS_ALL_FILE_DEFAULT & mask; } @@ -5630,7 +5634,7 @@ void __init shmem_init(void) * Default to setting PMD-sized THP to inherit the global setting and * disable all other multi-size THPs. */ - if (!shmem_orders_configured) + if (!shmem_orders_configured && pgtable_has_pmd_leaves()) huge_shmem_orders_inherit = BIT(HPAGE_PMD_ORDER); #endif return; -- 2.55.0