MEMBARRIER_CMD_GLOBAL_EXPEDITED sends an IPI to each CPU which runs a registered process. It finds those CPUs with rq->membarrier_state, a per-runqueue copy of the membarrier state of the mm. Commit 227a4aadc75b ("sched/membarrier: Fix p->mm->membarrier_state racy load") added the copy. The scheduler updates it when it switches between two mm. A registration sends an IPI to update the runqueues which keep running the mm. membarrier_switch_mm() compares the previous active_mm with the next mm. So it also skips the update when the previous task only borrowed the mm. This is a kernel thread in lazy TLB mode, or a task past exit_mm(). The copy can be stale in two such cases. Case 1: a process registers while a lazy TLB kernel thread borrows its mm on another CPU. T is a thread of a process with mm M, which is not registered yet. S is a thread of another process. CPU 0 CPU 1 T runs in user-space rq->membarrier_state == 0 T migrates to CPU 1 context_switch(T -> idle) idle borrows M as its active_mm context_switch(-> T) T: MEMBARRIER_CMD_REGISTER_GLOBAL_EXPEDITED sets the state of M sets rq->membarrier_state of CPU 1 CPU 0 runs idle: no IPI T migrates to CPU 0 context_switch(idle -> T) membarrier_switch_mm(rq, idle->active_mm, T->mm) prev_mm == next_mm: no update T runs in user-space rq->membarrier_state == 0 (stale) S: MEMBARRIER_CMD_GLOBAL_EXPEDITED CPU 0: state not set, no IPI Case 2: a thread exits and the CPU then runs another thread of the same process. A and B are threads of a registered process with mm M. S is a thread of another process. CPU 0 CPU 1 B exits exit_mm() B->mm = NULL membarrier_update_current_mm(NULL) rq->membarrier_state = 0 context_switch(B -> A) membarrier_switch_mm(rq, B->active_mm, A->mm) prev_mm == next_mm: no update A runs in user-space rq->membarrier_state == 0 (stale) S: MEMBARRIER_CMD_GLOBAL_EXPEDITED CPU 0: state not set, no IPI kthread_unuse_mm() clears the copy in the same way as exit_mm(). In both cases, the registered process runs without the promised memory barrier. This lasts until the CPU switches to the process from another mm. User-space sees this as a memory ordering failure. The registered processes omit memory barriers from their fast paths. The caller issues MEMBARRIER_CMD_GLOBAL_EXPEDITED where it needs those barriers. An example is RCU-like synchronization over shared memory. Without the IPI, the caller would miss a recent store of a registered process. It would then, for example, free memory which that process still uses. A thread exit in a registered multi-threaded process is enough to trigger case 2. Both cases were reproduced on v7.4-rc2: 1000 commands send no IPI to the registered process. Fix this by passing prev->mm rather than prev->active_mm. It is NULL for a kernel thread and for an exited task, which forces the update. The update is still skipped between two user tasks of the same mm. Fixes: 227a4aadc75b ("sched/membarrier: Fix p->mm->membarrier_state racy load") Cc: stable@vger.kernel.org Signed-off-by: Mathieu Desnoyers --- kernel/sched/core.c | 6 +++++- kernel/sched/sched.h | 15 +++++++++++++++ 2 files changed, 20 insertions(+), 1 deletion(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 1a088d4576f5..df6d15cf7109 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -5551,7 +5551,11 @@ context_switch(struct rq *rq, struct task_struct *prev, else prev->active_mm = NULL; } else { // to user - membarrier_switch_mm(rq, prev->active_mm, next->mm); + /* + * Pass prev->mm, not prev->active_mm: see + * membarrier_switch_mm(). + */ + membarrier_switch_mm(rq, prev->mm, next->mm); /* * sys_membarrier() requires an smp_mb() between setting * rq->curr / membarrier_switch_mm() and returning to userspace. diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 1d76950dbb0a..68353796a7f9 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -3907,6 +3907,21 @@ static inline bool sched_energy_enabled(void) { return false; } * - prior user-space memory accesses and store to rq->membarrier_state, * - store to rq->membarrier_state and following user-space memory accesses. * In the same way it provides those guarantees around store to rq->curr. + * + * @prev_mm is the mm of the previous task. It is NULL for a kernel + * thread and for a task past exit_mm(). + * + * The update is skipped only between two user tasks of the same mm. + * rq->membarrier_state is up to date in that case. It was set when + * the first of those tasks was scheduled in, or later by the + * registration IPI. + * + * It must not be skipped when the previous task only borrowed the mm + * as its active_mm. rq->membarrier_state can be stale then: + * + * - exit_mm() and kthread_unuse_mm() clear it, + * - a registration does not send an IPI to a CPU which runs a lazy + * TLB kernel thread. */ static inline void membarrier_switch_mm(struct rq *rq, struct mm_struct *prev_mm, -- 2.47.3