Add the acquire variant of smp_cond_load_relaxed_timeout(). This reuses the relaxed variant, with additional LOAD->LOAD ordering via smp_acquire__after_ctrl_dep(). To ensure that the necessary control dependency on the dereference of @ptr exists (which does not in the timeout path), re-evaluate the cond_expr branch. Cc: Kumar Kartikeya Dwivedi Cc: Alexei Starovoitov Cc: Arnd Bergmann Cc: Will Deacon Cc: Catalin Marinas Cc: Peter Zijlstra Cc: linux-arch@vger.kernel.org Cc: bpf@vger.kernel.org Reviewed-by: Catalin Marinas Reviewed-by: Haris Okanovic Tested-by: Haris Okanovic Signed-off-by: Ankur Arora --- include/asm-generic/barrier.h | 40 +++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/include/asm-generic/barrier.h b/include/asm-generic/barrier.h index d1e9ed15bbfc..b2b260a41122 100644 --- a/include/asm-generic/barrier.h +++ b/include/asm-generic/barrier.h @@ -352,6 +352,46 @@ do { \ }) #endif +/** + * smp_cond_load_acquire_timeout() - (Spin) wait for cond until a timeout + * expires. ACQUIRE ordering when @cond_expr is satisfied. + * @ptr: pointer to the variable to wait on. + * @cond_expr: boolean expression to wait for. + * @time_expr_ns: monotonic expression that evaluates to time in ns or, + * on failure, returns a negative value. + * @timeout_ns: timeout value in ns + * (Both of the above are assumed to be compatible with s64.) + * + * Equivalent to using smp_cond_load_acquire() on the condition variable with + * a timeout. + */ +#ifndef smp_cond_load_acquire_timeout +#define smp_cond_load_acquire_timeout(ptr, cond_expr, \ + time_expr_ns, timeout_ns) \ +({ \ + __unqual_scalar_typeof(*(ptr)) VAL; \ + VAL = smp_cond_load_relaxed_timeout(ptr, cond_expr, \ + time_expr_ns, \ + timeout_ns); \ + /* \ + * We arrive here once the loop condition is hit, on timeout, \ + * or, if we hit both the timeout and the loop condition. \ + * \ + * The last case is low probability, but possible in the last \ + * iteration, especially on architectures with waiting \ + * cpu_poll_relax() implementations (ex. arm64). \ + * Now since the loop condition is not evaluated on timeout, \ + * we have a missed control dependency. \ + * \ + * So, force a re-evaluation of the control dependency to \ + * provide an ACQUIRE ordering for that case as well. \ + */ \ + if (cond_expr) \ + smp_acquire__after_ctrl_dep(); \ + (typeof(*(ptr)))VAL; \ +}) +#endif + /* * pmem_wmb() ensures that all stores for which the modification * are written to persistent storage by preceding instructions have -- 2.43.7