Below __LINUX_ARM_ARCH__ 7, where MOVW/MOVT cannot be assumed, emit_mov_i_no8m() put every constant that is not an imm8m into a literal pool behind the epilogue and loaded it with LDR rX, [pc, #off]. That offset field is 12 bits and the pool sits at the end of the image, so from roughly 4 KB of JITed code on, the load no longer reaches it. Commit 0b59d8806a31 ("ARM: net: delegate filter to kernel interpreter when imm_offset() return value can't fit into 12bits.") handles that by setting FLAG_IMM_OVERFLOW and dropping the whole program to the interpreter, and already noted that multiple pools "would better be done as a feature instead of fix". That check does not catch every case: imm is a u16 in imm_offset(), so imm = offset - (8 + ctx->idx * 4); if (imm & ~0xfff) sees the truncated value. Take a classic filter that starts with 'div #3', whose divisor helper is reached through such a load, and that is long enough to push the pool out of reach: at a displacement of 0x10008, imm becomes 0x8 and the check passes. The emitted LDR ip, [pc, #8] then loads an instruction instead of the address of jit_udiv32, and BLX ip jumps to whatever that instruction happens to be. Build the constant in the register instead: a single MVN when the inverted value is an imm8m, otherwise one MOV/ORR per non-zero byte of val, or MVN/BIC per non-zero byte of ~val once the top half is all ones. A byte at a byte-aligned position is always an imm8m, so the sequence always exists. Inverting pays for small negative values: -264 is 0xfffffef8, which costs an instruction per 0xff byte, while ~val is 0x107 and takes MVN #0x7 plus BIC #0x100. Without a pool, constant loads no longer put a ceiling on program size, and FLAG_IMM_OVERFLOW with ctx->flags, which held nothing else, ctx->imms, ctx->imm_count and ctx->epilogue_bytes go away with it. It costs up to four instructions per hard constant instead of one. The pool was not much cheaper: imm_offset() counts duplicates on the sizing pass, so every use already took 4 bytes of LDR plus 4 bytes of slot. The 4 KB ceiling is low enough to be hit by the kernel itself. An ARMv5 kernel fails to boot when CONFIG_BPF_JIT_ALWAYS_ON=y and CONFIG_NET_PTP_CLASSIFY=y are set together: the first leaves no interpreter to fall back to, the second makes sock_init() build the PTP classifier, which needs 5124 bytes of ARM code. bpf_prog_create() then returns -ENOTSUPP and ptp_classifier_init() turns that into a BUG_ON. Neither option is unusual - NET_PTP_CLASSIFY is selected by PTP_1588_CLOCK, which defaults to ETHERNET, which defaults to y: kernel BUG at net/core/ptp_classifier.c:223! PC is at ptp_classifier_init+0x3c/0x70 ptp_classifier_init from sock_init+0xac/0xf0 seccomp reaches the JIT the same way, so filters of that size are likely rejected there as well; that was not measured. This is not confined to ARMv5 and ARMv6 hardware. __LINUX_ARM_ARCH__ is 6 whenever CPU_32v6 is set, even alongside CPU_32v7, because the v6 line wins in arch/arm/Makefile; imx_v6_v7_defconfig and omap2plus_defconfig enable both. With the JIT on, an i.MX6 runs this path. Tested on versatile_defconfig (ARM926EJ-S) under qemu with both options set: the kernel boots again, and SO_ATTACH_FILTER accepts cBPF filters up to the 4096-instruction maximum. Fixes: 0b59d8806a31 ("ARM: net: delegate filter to kernel interpreter when imm_offset() return value can't fit into 12bits.") Suggested-by: Alexei Starovoitov Signed-off-by: Ramin Moussavi --- v3: - emit_mov_i_no8m(): take the MVN/BIC path for val >= 0xffff0000, so a small negative value costs two instructions instead of four, and drop the unreachable 'if (first)' fallback (Alexei) - describe the u16 truncation in imm_offset() that lets the 12-bit check pass on a displacement it cannot encode, and add the Fixes tag for it (Alexei) - correct the commit message: __LINUX_ARM_ARCH__ is 6 on a kernel built for both v6 and v7, so this path is not limited to ARMv5 hardware, and drop the arm64 comparison, which does not hold for small negative values (Alexei) - Alexei's hunk was untested as posted: the emitted sequences are now decoded and simulated for all 2^32 values, and the filters below are attached and then run, not just attached Tested under qemu with cBPF filters of 16, 64, 256, 1024, 2048, 4000 and 4096 instructions, each starting with a 'div #3' whose helper address is one of these constants. With CONFIG_BPF_JIT_ALWAYS_ON=y there is no interpreter to fall back to, so an accepted filter means the JIT translated it. 4096 is the classic cBPF maximum; eBPF programs loaded through bpf() can be far larger and were not part of the test. versatilepb / ARM926EJ-S ARMv5 versatile_defconfig 7/7 raspi1ap / ARM1176 ARMv6K bcm2835_defconfig 7/7 virt / Cortex-A15 ARMv7 multi_v7+ARCH_MULTI_V6 7/7 That a v6+v7 kernel never emits MOVW/MOVT, even on the v7 part, is worth fixing on its own; I will send that separately, since it is an optimisation rather than part of this fix. This patch was written and reviewed with LLM assistance. The simulation mentioned above came out of that review; it was written three times independently, and the three agreed on every value. v2: https://lore.kernel.org/all/20261120140147.166934-1-ramin.moussavi@yacoub.de/ - rebased onto bpf-next, which had taken 0af298e62e5c in the ctx.imms allocation this patch removes - drop ctx->flags, which holds nothing once FLAG_IMM_OVERFLOW is gone - comment style in emit_mov_i_no8m() v1: https://lore.kernel.org/all/20261111121621.665533-1-ramin.moussavi@yacoub.de/ Applies cleanly to both bpf and bpf-next; arch/arm/net/bpf_jit_32.{c,h} are identical in the two trees. arch/arm/net/bpf_jit_32.c | 119 ++++++++++---------------------------- arch/arm/net/bpf_jit_32.h | 3 + 2 files changed, 33 insertions(+), 89 deletions(-) diff --git a/arch/arm/net/bpf_jit_32.c b/arch/arm/net/bpf_jit_32.c index aa240e3de6e70..16ae9c435c619 100644 --- a/arch/arm/net/bpf_jit_32.c +++ b/arch/arm/net/bpf_jit_32.c @@ -130,8 +130,6 @@ enum { #define TMP_REG_2 (MAX_BPF_JIT_REG + 1) /* TEMP Register 2 */ #define TCALL_CNT (MAX_BPF_JIT_REG + 2) /* Tail Call Count */ -#define FLAG_IMM_OVERFLOW (1 << 0) - /* * Map eBPF registers to ARM 32bit registers or stack scratch space. * @@ -193,10 +191,6 @@ static const s8 bpf2a32[][2] = { * offsets : array of eBPF instruction offsets in * JITed code. * target : final JITed code. - * epilogue_bytes : no of bytes used in epilogue. - * imm_count : no of immediate counts used for global - * variables. - * imms : array of global variable addresses. */ struct jit_ctx { @@ -205,15 +199,9 @@ struct jit_ctx { unsigned int prologue_bytes; unsigned int epilogue_offset; unsigned int cpu_architecture; - u32 flags; u32 *offsets; u32 *target; u32 stack_size; -#if __LINUX_ARM_ARCH__ < 7 - u16 epilogue_bytes; - u16 imm_count; - u32 *imms; -#endif }; /* @@ -404,53 +392,6 @@ static void jit_fill_hole(void *area, unsigned int size) #define _STACK_SIZE (ctx->prog->aux->stack_depth + SCRATCH_SIZE) #define STACK_SIZE ALIGN(_STACK_SIZE, STACK_ALIGNMENT) -#if __LINUX_ARM_ARCH__ < 7 - -static u16 imm_offset(u32 k, struct jit_ctx *ctx) -{ - unsigned int i = 0, offset; - u16 imm; - - /* on the "fake" run we just count them (duplicates included) */ - if (ctx->target == NULL) { - ctx->imm_count++; - return 0; - } - - while ((i < ctx->imm_count) && ctx->imms[i]) { - if (ctx->imms[i] == k) - break; - i++; - } - - if (ctx->imms[i] == 0) - ctx->imms[i] = k; - - /* constants go just after the epilogue */ - offset = ctx->offsets[ctx->prog->len - 1] * 4; - offset += ctx->prologue_bytes; - offset += ctx->epilogue_bytes; - offset += i * 4; - - ctx->target[offset / 4] = k; - - /* PC in ARM mode == address of the instruction + 8 */ - imm = offset - (8 + ctx->idx * 4); - - if (imm & ~0xfff) { - /* - * literal pool is too far, signal it into flags. we - * can only detect it on the second pass unfortunately. - */ - ctx->flags |= FLAG_IMM_OVERFLOW; - return 0; - } - - return imm; -} - -#endif /* __LINUX_ARM_ARCH__ */ - static inline int bpf2a32_offset(int bpf_to, int bpf_from, const struct jit_ctx *ctx) { int to, from; @@ -469,7 +410,34 @@ static inline int bpf2a32_offset(int bpf_to, int bpf_from, static inline void emit_mov_i_no8m(const u8 rd, u32 val, struct jit_ctx *ctx) { #if __LINUX_ARM_ARCH__ < 7 - emit(ARM_LDR_I(rd, ARM_PC, imm_offset(val, ctx)), ctx); + /* + * A single MVN if the inverted value is an imm8m. Else MOV + ORR set + * the bytes of val, or MVN + BIC clear the bytes of ~val -- a byte is + * always an imm8m. Inverting pays once the top half is all ones, as + * it is for a small negative value: those two bytes cost an insn each + * the other way round. + */ + u32 op = ARM_MOV_I(rd, 0), next = ARM_ORR_I(rd, rd, 0); + int imm12 = imm8m(~val); + unsigned int shift; + + if (imm12 >= 0) { + emit(ARM_MVN_I(rd, imm12), ctx); + return; + } + if (val >= 0xffff0000) { + op = ARM_MVN_I(rd, 0); + next = ARM_BIC_I(rd, rd, 0); + val = ~val; + } + for (shift = 0; shift < 32; shift += 8) { + u32 part = val & (0xffU << shift); + + if (!part) + continue; + emit(op | imm8m(part), ctx); + op = next; + } #else emit(ARM_MOVW(rd, val & 0xffff), ctx); if (val > 0xffff) @@ -2091,14 +2059,6 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx) pr_err_once("unknown opcode %02x\n", code); return -EINVAL; } - - if (ctx->flags & FLAG_IMM_OVERFLOW) - /* - * this instruction generated an overflow when - * trying to access the literal pool, so - * delegate this filter to the kernel interpreter. - */ - return -1; return 0; } @@ -2192,21 +2152,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr ctx.epilogue_offset = ctx.idx; -#if __LINUX_ARM_ARCH__ < 7 - tmp_idx = ctx.idx; - build_epilogue(&ctx); - ctx.epilogue_bytes = (ctx.idx - tmp_idx) * 4; - - ctx.idx += ctx.imm_count; - if (ctx.imm_count) { - ctx.imms = kzalloc_objs(u32, ctx.imm_count); - if (ctx.imms == NULL) - goto out_off; - } -#else - /* there's nothing about the epilogue on ARMv7 */ build_epilogue(&ctx); -#endif /* Now we can get the actual image size of the JITed arm code. * Currently, we are not considering the THUMB-2 instructions * for jit, although it can decrease the size of the image. @@ -2224,7 +2170,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr * we must fall back to the interpretation */ if (header == NULL) - goto out_imms; + goto out_off; /* 2.) Actual pass to generate final JIT code */ ctx.target = (u32 *) image_ptr; @@ -2254,11 +2200,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr prog->jited = 1; prog->jited_len = image_size; -out_imms: -#if __LINUX_ARM_ARCH__ < 7 - if (ctx.imm_count) - kfree(ctx.imms); -#endif out_off: kfree(ctx.offsets); @@ -2267,6 +2208,6 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr out_free: image_ptr = NULL; bpf_jit_binary_free(header); - goto out_imms; + goto out_off; } diff --git a/arch/arm/net/bpf_jit_32.h b/arch/arm/net/bpf_jit_32.h index 438f0e1f91a00..76d83b486deda 100644 --- a/arch/arm/net/bpf_jit_32.h +++ b/arch/arm/net/bpf_jit_32.h @@ -104,6 +104,7 @@ #define ARM_INST_MOV_I 0x03a00000 #define ARM_INST_MOVW 0x03000000 #define ARM_INST_MOVT 0x03400000 +#define ARM_INST_MVN_I 0x03e00000 #define ARM_INST_MUL 0x00000090 @@ -231,6 +232,8 @@ #define ARM_MOVT(rd, imm) \ (ARM_INST_MOVT | ((imm) >> 12) << 16 | (rd) << 12 | ((imm) & 0x0fff)) +#define ARM_MVN_I(rd, imm) _AL3_I(ARM_INST_MVN, rd, 0, imm) + #define ARM_MUL(rd, rm, rn) (ARM_INST_MUL | (rd) << 16 | (rm) << 8 | (rn)) #define ARM_POP(regs) (ARM_INST_POP | (regs)) -- 2.53.0