Same missing register clear as on riscv64. A RMW atomic on an arena pointer is converted to BPF_PROBE_ATOMIC and gets an exception table entry, but bpf_jit_probe_atomic_pre() only fills in the arena base and the probe offset, leaving probe->reg at the -1 that bpf_jit_probe_init() set, which bpf_jit_probe_post() writes into the entry and ex_handler_bpf() then reads back as "there is nothing to clear". That is right for a plain BPF_{ADD,AND,OR,XOR}, which only writes memory, but an RMW carrying BPF_FETCH also reads the old value into a register: src_reg for BPF_{ADD,AND,OR,XOR} | BPF_FETCH and BPF_XCHG, and r0 for BPF_CMPXCHG. So on a fault over an unmapped arena page the program resumes at the landing pad with whatever that register held before the atomic instead of the 0 that every other BPF_PROBE_* access delivers. Fill probe->reg in from bpf_atomic_load_reg(). Unlike x86-64 and arm64, s390x does not report arena violations from its exception handler, so there is no access direction to correct here, only the missing register clear. Fixes: 2f9469484a3b ("s390/bpf: Support arena atomics") Signed-off-by: Daniel Borkmann Cc: Ilya Leoshkevich --- arch/s390/net/bpf_jit_comp.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/arch/s390/net/bpf_jit_comp.c b/arch/s390/net/bpf_jit_comp.c index b60877478b45..c46872b071ce 100644 --- a/arch/s390/net/bpf_jit_comp.c +++ b/arch/s390/net/bpf_jit_comp.c @@ -774,6 +774,8 @@ static void bpf_jit_probe_atomic_pre(struct bpf_jit *jit, struct bpf_insn *insn, struct bpf_jit_probe *probe) { + int load_reg; + if (BPF_MODE(insn->code) != BPF_PROBE_ATOMIC) return; @@ -783,6 +785,14 @@ static void bpf_jit_probe_atomic_pre(struct bpf_jit *jit, EMIT4(0xb9080000, REG_W1, insn->dst_reg); probe->arena_reg = REG_W1; probe->prg = jit->prg; + /* + * A read-modify-write carrying BPF_FETCH reads the old value into + * src_reg, or into r0 for a BPF_CMPXCHG. Clear that register on + * fault, the remaining atomics only write memory. + */ + load_reg = bpf_atomic_load_reg(insn); + if (load_reg >= 0) + probe->reg = reg2hex[load_reg]; } static int bpf_jit_probe_post(struct bpf_jit *jit, struct bpf_prog *fp, @@ -1684,6 +1694,7 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp, if (load_probe.prg != -1) { probe.prg = jit->prg; probe.arena_reg = load_probe.arena_reg; + probe.reg = load_probe.reg; } loop_start = jit->prg; /* 0: {csy|csg} %w0,%src,off(%arena) */ -- 2.43.0