The expedited membarrier commands need a full memory barrier after the scheduler stores to rq->curr, before the CPU returns to user-space. membarrier() skips the IPI to a CPU which it sees running another mm. This barrier orders the user-space accesses of the incoming thread after that check. See scenario (A) in kernel/sched/membarrier.c. LoongArch has no dedicated barrier. It used to get it from spin_unlock() in finish_lock_switch(), which was a full barrier ("dbar 0"). This is no longer true. Commit e031a5f3f1ed ("LoongArch: Support dbar with different hints") made smp_store_release() an ordering barrier ("dbar 0b10010"). It does not order a previous store with a following load. That commit kept a full barrier in queued_spin_unlock(). Commit 9c68ece8b2a5 ("LoongArch: Define the __io_aw() hook as mmiowb()") then removed it. Nothing else is a full barrier on a switch between two user processes. switch_mm_irqs_off() writes CSRs and sets a bit in the mm cpumask with a plain "amor". finish_task() uses smp_store_release(). The LoongArch manual lists the instructions which order memory accesses: DBAR, IBAR, the AM*_DB instructions and LL/SC pairs. A CSR write and ERTN are not among them. This was found by reading the code and the architecture manual. On Loongson-3A6000 or newer, a thread which is scheduled in would run user-space code without the barrier which membarrier promised to the caller. User-space would see a memory ordering failure: the caller of an expedited membarrier command would miss a recent store of that thread. Older processors treat every dbar hint as "dbar 0" and are not affected. Do as powerpc and riscv: add membarrier_arch_switch_mm() and call it from switch_mm_irqs_off() when the mm changes. It issues smp_mb() only when the next mm registered for an expedited command. Compile-tested only. Fixes: 9c68ece8b2a5 ("LoongArch: Define the __io_aw() hook as mmiowb()") Cc: stable@vger.kernel.org Signed-off-by: Mathieu Desnoyers --- arch/loongarch/Kconfig | 1 + arch/loongarch/include/asm/membarrier.h | 33 ++++++++++++++++++++++++ arch/loongarch/include/asm/mmu_context.h | 4 +++ 3 files changed, 38 insertions(+) create mode 100644 arch/loongarch/include/asm/membarrier.h diff --git a/arch/loongarch/Kconfig b/arch/loongarch/Kconfig index b5ed4881d3c0..be6e7545d646 100644 --- a/arch/loongarch/Kconfig +++ b/arch/loongarch/Kconfig @@ -22,6 +22,7 @@ config LOONGARCH select ARCH_HAS_FORTIFY_SOURCE select ARCH_HAS_KCOV select ARCH_HAS_KERNEL_FPU_SUPPORT if 64BIT && CPU_HAS_FPU + select ARCH_HAS_MEMBARRIER_CALLBACKS select ARCH_HAS_NMI_SAFE_THIS_CPU_OPS select ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE select ARCH_HAS_PREEMPT_LAZY diff --git a/arch/loongarch/include/asm/membarrier.h b/arch/loongarch/include/asm/membarrier.h new file mode 100644 index 000000000000..1789f878821f --- /dev/null +++ b/arch/loongarch/include/asm/membarrier.h @@ -0,0 +1,33 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef _ASM_LOONGARCH_MEMBARRIER_H +#define _ASM_LOONGARCH_MEMBARRIER_H + +static inline void membarrier_arch_switch_mm(struct mm_struct *prev, + struct mm_struct *next, + struct task_struct *tsk) +{ + /* + * Only need the full barrier when switching between processes. + * Barrier when switching from kernel to userspace is not + * required here, given that it is implied by mmdrop(). Barrier + * when switching from userspace to kernel is not needed after + * store to rq->curr. + */ + if (IS_ENABLED(CONFIG_SMP) && + likely(!(atomic_read(&next->membarrier_state) & + (MEMBARRIER_STATE_PRIVATE_EXPEDITED | + MEMBARRIER_STATE_GLOBAL_EXPEDITED)))) + return; + + /* + * The membarrier system call requires a full memory barrier + * after storing to rq->curr, before going back to user-space. + * + * Nothing else provides it on this path. spin_unlock() and + * smp_store_release() are ordering barriers ("dbar 0b10010"): + * they do not order a previous store with a following load. + */ + smp_mb(); +} + +#endif /* _ASM_LOONGARCH_MEMBARRIER_H */ diff --git a/arch/loongarch/include/asm/mmu_context.h b/arch/loongarch/include/asm/mmu_context.h index 304363bd3935..dba62dd7a131 100644 --- a/arch/loongarch/include/asm/mmu_context.h +++ b/arch/loongarch/include/asm/mmu_context.h @@ -9,6 +9,7 @@ #include #include +#include #include #include #include @@ -91,6 +92,9 @@ static inline void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct * bool need_flush = false; unsigned int cpu = smp_processor_id(); + if (prev != next) + membarrier_arch_switch_mm(prev, next, tsk); + /* Check if our ASID is of an older version and thus invalid */ if (!asid_valid(next, cpu)) get_new_mmu_context(next, cpu, &need_flush); -- 2.47.3