Add the acquire variant of smp_cond_load_relaxed_timeout(). smp_cond_load_acquire_timeout() reuses the relaxed variant for the actual wait. This has two paths out: C1. "if (cond_expr)": loop condition evaluates to true C2. "if (__scl_time_now <= 0 || __scl_timeout <= 0)": timeout case a. cond_expr evaluates to false b. cond_expr evaluates to true C1 already provides LOAD->STORE order via the control-dependency. C2b does not. So re-evaluate the "if (cond_expr)" branch in smp_cond_load_acquire_timeout() to provide that, and follow that with smp_acquire__after_ctrl_dep() for the additional LOAD->LOAD order, together providing the full load-acquire order. For the pure timeout case (C2a), we have neither, so just go with a straight smp_load_acquire(). Cc: Kumar Kartikeya Dwivedi Cc: Alexei Starovoitov Cc: Arnd Bergmann Cc: Will Deacon Cc: Catalin Marinas Cc: Peter Zijlstra Cc: linux-arch@vger.kernel.org Cc: bpf@vger.kernel.org Reviewed-by: Catalin Marinas Reviewed-by: Haris Okanovic Tested-by: Haris Okanovic Signed-off-by: Ankur Arora --- Notes: In earlier revisions sashiko comments pointed out: "Does this standalone if block fail to extend a control dependency to the macro caller's subsequent code?" This was a problem for the timeout case where the "if (cond_expr())" was missing. My solution in earlier versions was to not provide acquire ordering on timeout -- that looked okay for BPF and we have similar semantics in other places as well (ex. try_page_mte_tagging()). On second thoughts, those semantics were unnecessary special for a non-performance path. So this revision just provides a full smp_load_acquire() in the failure path. Also update the comment in smp_cond_load_acquire_timeout() to describe the barrier semantics for the three cases. Catalin, Haris: I've retained both your R-bys. Hope that's okay. include/asm-generic/barrier.h | 48 +++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/include/asm-generic/barrier.h b/include/asm-generic/barrier.h index 4437d27c46b9..81db4da12f5b 100644 --- a/include/asm-generic/barrier.h +++ b/include/asm-generic/barrier.h @@ -379,6 +379,54 @@ do { \ }) #endif +/** + * smp_cond_load_acquire_timeout() - (Spin) wait for cond with ACQUIRE ordering + * until a timeout expires. + * @ptr: pointer to the variable to wait on. + * @cond_expr: boolean expression to wait for. + * @time_expr_ns: monotonic expression that evaluates to time in ns or, + * on failure, returns a negative value. + * @timeout_ns: timeout value in ns + * (Both of the above are assumed to be compatible with s64.) + * + * Equivalent to using smp_cond_load_acquire() on the condition variable with + * a timeout. + */ +#ifndef smp_cond_load_acquire_timeout +#define smp_cond_load_acquire_timeout(ptr, cond_expr, \ + time_expr_ns, timeout_ns) \ +({ \ + __unqual_scalar_typeof(*(ptr)) VAL; \ + VAL = smp_cond_load_relaxed_timeout(ptr, cond_expr, \ + time_expr_ns, \ + timeout_ns); \ + /* \ + * We arrive here once the loop condition is hit, on timeout, \ + * or, if we hit both the timeout and the loop condition. \ + * \ + * For the first case, we come here having already evaluated \ + * the control dependency. \ + * In the last case -- low probability, possible in the last \ + * iteration, especially on architectures with waiting \ + * cpu_poll_relax() -- the control dependency has not been \ + * evaluated. \ + * \ + * So, force it to be re-evaluated before \ + * smp_acquire__after_ctrl_dep() to provide ACQUIRE ordering \ + * for both. \ + * \ + * The other case is of pure timeout, where again we don't have \ + * the advantage of having the control dependency. Given that \ + * this is the slowpath, we go with a full smp_load_acquire(). \ + */ \ + if (cond_expr) \ + smp_acquire__after_ctrl_dep(); \ + else \ + VAL = smp_load_acquire(ptr); \ + (typeof(*(ptr)))VAL; \ +}) +#endif + /* * pmem_wmb() ensures that all stores for which the modification * are written to persistent storage by preceding instructions have -- 2.43.7