Add smp_cond_load_relaxed_timeout(), which extends smp_cond_load_relaxed() to allow waiting for a duration. The interface loops around doing a periodic time-check, waiting for the condition variable to change. Loop iterations use cpu_poll_relax() to slow down the busy-wait and these, unless overridden by the architecture code, amount to a cpu_relax(). There are two ways for the time-check to fail: the timeout case or, @time_expr_ns returning an invalid value (negative or zero). The second failure mode allows for clocks attached to the clock-domain of @cond_expr -- clocks which might cease to operate meaningfully once some state internal to @cond_expr has changed -- to fail. Evaluation of @time_expr_ns: in the fastpath we want to keep the performance close to smp_cond_load_relaxed(). So defer evaluation of the potentially costly @time_expr_ns to the slowpath. This also means that there will always be some hardware dependent duration that has passed in cpu_poll_relax() iterations at the time of first evaluation. Additionally cpu_poll_relax() is not guaranteed to return at timeout boundary. In sum, expect timeout overshoot when we exit due to expiration of the timeout. The number of spin iterations before time-check, SMP_TIMEOUT_POLL_COUNT is chosen to be 200 by default. With a cpu_poll_relax() iteration taking ~20-30 cycles (measured on a variety of x86 platforms), we expect a time-check every ~4000-6000 cycles. If a architecture provides a waiting implementation for cpu_poll_relax() (and signifies that by defining CPU_POLL_RELAX_WAITS) we define SMP_TIMEOUT_POLL_COUNT to 1. Lastly, config option ARCH_HAS_CPU_RELAX indicates availability of a cpu_poll_relax() that is cheaper than polling. Long timeout values might not make sense for architectures not having ARCH_HAS_CPU_RELAX. Cc: Arnd Bergmann Cc: Will Deacon Cc: Catalin Marinas Cc: Peter Zijlstra Cc: linux-arch@vger.kernel.org Tested-by: Haris Okanovic Tested-by: Sahil Chandna Signed-off-by: Ankur Arora --- Notes: - Expose the timeout state in __smp_cond_load_relaxed_timeout(). We use it later to provide acquire ordering for the timeout case in smp_cond_load_acquire_timeout(). - Associate the kernel-doc comment with the actual barrier, not the helper. include/asm-generic/barrier.h | 112 ++++++++++++++++++++++++++++++++++ 1 file changed, 112 insertions(+) diff --git a/include/asm-generic/barrier.h b/include/asm-generic/barrier.h index b99cb57dfccc..bafc8eeff132 100644 --- a/include/asm-generic/barrier.h +++ b/include/asm-generic/barrier.h @@ -15,6 +15,7 @@ #include #include +#include #include #ifndef nop @@ -273,6 +274,117 @@ do { \ }) #endif +/* + * Number of times we iterate in the loop before doing the time check. + */ +#ifndef SMP_TIMEOUT_POLL_COUNT +#ifdef CPU_POLL_RELAX_WAITS +#define SMP_TIMEOUT_POLL_COUNT 1 /* Wait mode. No need to poll. */ +#else +/* + * Assume that cpu_poll_relax() provides a small blip in the pipeline. + * Combine a reasonable number of cpu_poll_relax() instances before + * doing anything substantial like a time-check. + * Note that this assumes that the rest of the loop (largely evaluation + * of the loop condition is relatively cheap.) + */ +#define SMP_TIMEOUT_POLL_COUNT 200 +#endif +#endif + +/* + * cpu_poll_relax() stitches up two kinds of primitives: ones that provide + * a momentary blip in the pipeline (ex. cpu_relax() on x86), or ones that + * support waiting for @ptr value to change, coupled with a precise (or not) + * timeout. + * + * We keep both together because the objective is to minimize expensive + * operations while polling on @ptr waiting for it to change. Either + * version allows for that. + * The arguments (@ptr, @val, @timeout_ns) are only needed for waiting + * implementations. + * + * Note that platforms with a suitable cpu_poll_relax() implementation are + * expected to define ARCH_HAS_CPU_RELAX. + */ +#ifndef cpu_poll_relax +#define cpu_poll_relax(ptr, val, timeout_ns) cpu_relax() +#endif + +#ifndef smp_cond_load_relaxed_timeout +#define __smp_cond_load_relaxed_timeout(ptr, cond_expr, \ + time_expr_ns, timeout_ns, \ + timed_out) \ +({ \ + typeof(ptr) __PTR = (ptr); \ + __unqual_scalar_typeof(*(ptr)) VAL; \ + u32 __scl_count = 0, __scl_spin = SMP_TIMEOUT_POLL_COUNT; \ + u64 __scl_timeout = NSEC_PER_USEC, __scl_time_end = 0; \ + s64 __scl_time_now; \ + \ + for (;;) { \ + VAL = READ_ONCE(*__PTR); \ + if (cond_expr || timed_out) \ + break; \ + cpu_poll_relax(__PTR, VAL, (u64)__scl_timeout); \ + if (++__scl_count < __scl_spin) \ + continue; \ + __scl_time_now = (s64)(time_expr_ns); \ + if (unlikely(__scl_time_end == 0)) { \ + __scl_timeout = (timeout_ns); \ + __scl_time_end = __scl_time_now + __scl_timeout;\ + } \ + if (__scl_time_now <= 0 || \ + __scl_time_end <= __scl_time_now) { \ + timed_out = true; \ + continue; \ + } \ + __scl_count = 0; \ + __scl_timeout = __scl_time_end - __scl_time_now; \ + } \ + (typeof(*(ptr)))VAL; \ +}) + +/** + * smp_cond_load_relaxed_timeout() - (Spin) wait for cond with no ordering + * until a timeout expires. + * @ptr: pointer to the variable to wait on. + * @cond_expr: boolean expression to wait for. + * @time_expr_ns: expression that evaluates to monotonic time (in ns) or, + * on failure, returns zero or a negative value. + * @timeout_ns: timeout value in ns. A timeout value of zero is treated + * as the minimum wait so we evaluate cond_expr at least once. + * Both of the above are expected to be compatible with s64; the signed + * value is used to handle the failure case in @time_expr_ns. + * + * Equivalent to using READ_ONCE() on the condition variable. + * + * Callers that expect to wait for prolonged durations might want + * to take into account the availability of ARCH_HAS_CPU_RELAX. + * + * Note that @ptr is expected to point to a memory address. Using this + * interface with MMIO will be slower (since SMP_TIMEOUT_POLL_COUNT is + * tuned for memory) and might also break in interesting architecture + * dependent ways. + */ +#define smp_cond_load_relaxed_timeout(ptr, cond_expr, \ + time_expr_ns, timeout_ns) \ +({ \ + s64 __scl_timeout_ns = (s64)(timeout_ns); \ + __unqual_scalar_typeof(*(ptr)) VAL; \ + bool __maybe_unused __scl_timed_out = false; \ + \ + if (unlikely(__scl_timeout_ns < 0)) \ + VAL = READ_ONCE(*(ptr)); \ + else \ + VAL = __smp_cond_load_relaxed_timeout(ptr, cond_expr, \ + time_expr_ns, \ + __scl_timeout_ns, \ + __scl_timed_out); \ + (typeof(*(ptr)))VAL; \ +}) +#endif + /* * pmem_wmb() ensures that all stores for which the modification * are written to persistent storage by preceding instructions have -- 2.43.7