The ll/sc versions of the fetching atomics load the old value into the source register, or into r0 for cmpxchg, and then store through the address register. With a zero offset the address register is dst_reg itself, so for src_reg == dst_reg of a fetch or xchg, or dst_reg == r0 of a cmpxchg, the ldxr overwrites the address and the stlxr stores to whatever the old value was. The verifier keeps such programs away from the JIT today, but the JIT should not depend on it. The LSE versions are not affected, LD, SWP and CAS have no register overlap constraint. Take a copy of the address when the register taking the old value is the address register. Before/after for r1 = atomic_fetch_add((u64 *)(r1 + 0), r1) without LSE: aa0003e9 mov x9, x0 c85f7c00 ldxr x0, [x0] 8b09000b add x11, x0, x9 c80cfc0b stlxr w12, x11, [x0] 35ffffac cbnz w12, ... d5033bbf dmb ish aa0003ea mov x10, x0 aa0003e9 mov x9, x0 c85f7d40 ldxr x0, [x10] 8b09000b add x11, x0, x9 c80cfd4b stlxr w12, x11, [x10] 35ffffac cbnz w12, ... d5033bbf dmb ish Signed-off-by: Daniel Borkmann --- arch/arm64/net/bpf_jit_comp.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index fe6c8bf3dc02..99f1d9bd13ea 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -881,6 +881,7 @@ static int emit_ll_sc_atomic(const struct bpf_insn *insn, struct jit_ctx *ctx) const s32 imm = insn->imm; const s16 off = insn->off; const bool isdw = BPF_SIZE(code) == BPF_DW; + const int load_reg = bpf_atomic_load_reg(insn); u8 reg = dst; s32 jmp_offset; @@ -894,6 +895,11 @@ static int emit_ll_sc_atomic(const struct bpf_insn *insn, struct jit_ctx *ctx) emit_a64_add_i(1, tmp, reg, tmp, off, ctx); reg = tmp; } + /* The load of the old value must not clobber the address */ + if (load_reg >= 0 && bpf2a64[load_reg] == reg) { + emit(A64_MOV(1, tmp, reg), ctx); + reg = tmp; + } if (imm == BPF_ADD || imm == BPF_AND || imm == BPF_OR || imm == BPF_XOR) { -- 2.43.0