From: Lance Yang When unsharing hugetlb PMD page tables, we currently send two IPIs: one for TLB invalidation, and another to synchronize with concurrent GUP-fast walkers. However, if the TLB flush already reaches all CPUs, the second IPI is redundant. GUP-fast runs with IRQs disabled, so when the TLB flush IPI completes, any concurrent GUP-fast must have finished. Add tlb_table_flush_implies_ipi_broadcast() to let architectures indicate their TLB flush provides full synchronization, enabling the redundant IPI to be skipped. The default implementation returns false to maintain current behavior. Suggested-by: David Hildenbrand (Red Hat) Signed-off-by: Lance Yang --- include/asm-generic/tlb.h | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 324a21f53b64..3f0add95604f 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -248,6 +248,21 @@ static inline void tlb_remove_table(struct mmu_gather *tlb, void *table) #define tlb_needs_table_invalidate() (true) #endif +/* + * Architectures can override if their TLB flush already broadcasts IPIs to all + * CPUs when freeing or unsharing page tables. + * + * Return true only when the flush guarantees: + * - IPIs reach all CPUs with potentially stale paging-structure cache entries + * - Synchronization with IRQ-disabled code like GUP-fast + */ +#ifndef tlb_table_flush_implies_ipi_broadcast +static inline bool tlb_table_flush_implies_ipi_broadcast(void) +{ + return false; +} +#endif + void tlb_remove_table_sync_one(void); #else @@ -829,12 +844,17 @@ static inline void tlb_flush_unshared_tables(struct mmu_gather *tlb) * We only perform this when we are the last sharer of a page table, * as the IPI will reach all CPUs: any GUP-fast. * + * However, if the TLB flush already synchronized with other CPUs + * (indicated by tlb_table_flush_implies_ipi_broadcast()), we can skip + * the additional IPI. + * * Note that on configs where tlb_remove_table_sync_one() is a NOP, * the expectation is that the tlb_flush_mmu_tlbonly() would have issued * required IPIs already for us. */ if (tlb->fully_unshared_tables) { - tlb_remove_table_sync_one(); + if (!tlb_table_flush_implies_ipi_broadcast()) + tlb_remove_table_sync_one(); tlb->fully_unshared_tables = false; } } -- 2.49.0