AI Trajectory

Trajectory: (Expand/Collapse All)
Seq	Timestamp	Type	Name	Duration
0/0	2026/08/31 14:02	flow	patch-triage	19m Args: null Results: { "EnableConfigs": [ "BPF_SYSCALL", "BPF_JIT", "CALL_THUNKS", "MITIGATION_CALL_DEPTH_TRACKING", "X86_64" ], "FocusSymbols": [ "arch_prepare_bpf_trampoline", "arch_bpf_trampoline_size", "bpf_int_jit_compile", "arch_prepare_bpf_dispatcher", "x86_call_depth_emit_accounting" ], "KMSANReasoning": "The patch refactors the x86 BPF JIT compiler and callthunk accounting code to introduce a dry-run mode for trampoline size calculation via `struct jit_emit_context`, replacing dummy allocation in `arch_bpf_trampoline_size()`. All new context and buffer variables are properly initialized (`= {}`), and code generation buffers are accurately tracked and populated. The patch does not introduce any uninitialized memory reads, uninitialized structure padding leaks to user space, or uninitialized branch conditions. Any potential bugs in JIT generation or pointer calculations would be out-of-bounds or logic errors caught by KASAN or standard kernel assertions, not uninitialized memory bugs detectable only by KMSAN. Therefore, dedicated KMSAN fuzzing is not required.", "NeedsKMSAN": false, "Reasoning": "The patch refactors the x86 BPF JIT compiler and trampoline generator by introducing a struct jit_emit_context with dry-run support. This changes how trampoline sizing is computed (dry-run without allocation), updates instruction emission logic, and alters x86 call depth accounting integration. These changes affect BPF program compilation, trampoline generation, and execution reachable via the bpf() syscall.", "WorthFuzzing": true }
1/1	2026/08/31 14:02	action	read-patch-diff	0m Args: null Results: { "PatchDiff": "commit e09e4ad11f933fa9ab21debd310aeb220f3f52c0\nAuthor: syz-cluster \u003ctriage@syzkaller.com\u003e\nDate: Mon Aug 31 14:02:28 2026 +0000\n\n syz-cluster: applied patch under review\n\ndiff --git a/arch/x86/include/asm/alternative.h b/arch/x86/include/asm/alternative.h\nindex 08af86ef090a8..2ef4937411860 100644\n--- a/arch/x86/include/asm/alternative.h\n+++ b/arch/x86/include/asm/alternative.h\n@@ -83,7 +83,7 @@ extern void callthunks_patch_builtin_calls(void);\n extern void callthunks_patch_module_calls(struct callthunk_sites sites,\n \t\t\t\t\t struct module mod);\n extern void callthunks_translate_call_dest(void dest);\n-extern int x86_call_depth_emit_accounting(u8 *pprog, void func, void ip);\n+extern int x86_call_depth_emit_accounting(u8 insn_buff, void func, void ip);\n #else\n static __always_inline void callthunks_patch_builtin_calls(void) {}\n static __always_inline void\n@@ -93,7 +93,7 @@ static __always_inline void callthunks_translate_call_dest(void dest)\n {\n \treturn dest;\n }\n-static __always_inline int x86_call_depth_emit_accounting(u8 *pprog,\n+static __always_inline int x86_call_depth_emit_accounting(u8 insn_buff,\n \t\t\t\t\t\t\t void func, void ip)\n {\n \treturn 0;\ndiff --git a/arch/x86/include/asm/text-patching.h b/arch/x86/include/asm/text-patching.h\nindex f2d142a0a862e..a0a7c778b0a2a 100644\n--- a/arch/x86/include/asm/text-patching.h\n+++ b/arch/x86/include/asm/text-patching.h\n@@ -13,6 +13,8 @@\n /\n #define TEXT_POKE_MAX_OPCODE_SIZE\t5\n \n+#define MAX_PATCH_LEN\t(255-1)\n+\n extern void text_poke_early(void addr, const void opcode, size_t len);\n \n extern void text_poke_apply_relocation(u8 buf, const u8 * const instr, size_t instrlen, u8 repl, size_t repl_len);\ndiff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c\nindex 91b1cdd165699..5f9989c8559ea 100644\n--- a/arch/x86/kernel/alternative.c\n+++ b/arch/x86/kernel/alternative.c\n@@ -18,8 +18,6 @@ int __read_mostly alternatives_patched;\n \n EXPORT_SYMBOL_GPL(alternatives_patched);\n \n-#define MAX_PATCH_LEN (255-1)\n-\n #define DA_ALL\t\t(~0)\n #define DA_ALT\t\t0x01\n #define DA_RET\t\t0x02\ndiff --git a/arch/x86/kernel/callthunks.c b/arch/x86/kernel/callthunks.c\nindex e37728f703227..15b221d06901c 100644\n--- a/arch/x86/kernel/callthunks.c\n+++ b/arch/x86/kernel/callthunks.c\n@@ -23,8 +23,6 @@\n \n static int __initdata_or_module debug_callthunks;\n \n-#define MAX_PATCH_LEN (255-1)\n-\n #define prdbg(fmt, args...)\t\t\t\t\t\\\n do {\t\t\t\t\t\t\t\t\\\n \tif (debug_callthunks)\t\t\t\t\t\\\n@@ -298,10 +296,9 @@ static bool is_callthunk(void addr)\n \treturn !bcmp(pad, insn_buff, tmpl_size);\n }\n \n-int x86_call_depth_emit_accounting(u8 *pprog, void func, void ip)\n+int x86_call_depth_emit_accounting(u8 insn_buff, void func, void ip)\n {\n \tunsigned int tmpl_size = SKL_TMPL_SIZE;\n-\tu8 insn_buff[MAX_PATCH_LEN];\n \n \tif (!thunks_initialized)\n \t\treturn 0;\n@@ -313,8 +310,6 @@ int x86_call_depth_emit_accounting(u8 *pprog, void func, void ip)\n \tmemcpy(insn_buff, skl_call_thunk_template, tmpl_size);\n \ttext_poke_apply_relocation(insn_buff, ip, tmpl_size, skl_call_thunk_template, tmpl_size);\n \n-\tmemcpy(pprog, insn_buff, tmpl_size);\n-\tpprog += tmpl_size;\n \treturn tmpl_size;\n }\n #endif\ndiff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c\nindex 48429fae06410..2ec734c69b693 100644\n--- a/arch/x86/net/bpf_jit_comp.c\n+++ b/arch/x86/net/bpf_jit_comp.c\n@@ -23,6 +23,11 @@\n \n static bool all_callee_regs_used[4] = {true, true, true, true};\n \n+struct jit_emit_context {\n+\tu8 prog;\n+\tbool dry_run;\n+};\n+\n static u8 emit_code(u8 ptr, u32 bytes, unsigned int len)\n {\n \tif (len == 1)\n@@ -36,8 +41,16 @@ static u8 emit_code(u8 ptr, u32 bytes, unsigned int len)\n \treturn ptr + len;\n }\n \n+static void emit_code_jit(struct jit_emit_context jit, u32 bytes, unsigned int len)\n+{\n+\tif (jit-\u003edry_run)\n+\t\tjit-\u003eprog += len;\n+\telse\n+\t\tjit-\u003eprog = emit_code(jit-\u003eprog, bytes, len);\n+}\n+\n #define EMIT(bytes, len) \\\n-\tdo { prog = emit_code(prog, bytes, len); } while (0)\n+\temit_code_jit(jit, bytes, len)\n \n #define EMIT1(b1)\t\tEMIT(b1, 1)\n #define EMIT2(b1, b2)\t\tEMIT((b1) + ((b2) \u003c\u003c 8), 2)\n@@ -149,6 +162,19 @@ static int bpf_size_to_x86_bytes(int bpf_size)\n \t\treturn 0;\n }\n \n+static int bpf_call_depth_emit_accounting(struct jit_emit_context jit, void func, void ip)\n+{\n+\tu8 insn_buff[MAX_PATCH_LEN];\n+\tint size;\n+\n+\tsize = x86_call_depth_emit_accounting(insn_buff, func, ip);\n+\tif (!jit-\u003edry_run)\n+\t\tmemcpy(jit-\u003eprog, insn_buff, size);\n+\n+\tjit-\u003eprog += size;\n+\treturn size;\n+}\n+\n /\n List of x86 cond jumps opcodes (. + s8)\n * Add 0x10 (and an extra 0x0f) to generate far jumps (. + s32)\n@@ -309,6 +335,8 @@ int bpf_arch_text_invalidate(void dst, size_t len)\n }\n \n struct jit_context {\n+\tstruct jit_emit_context jit;\n+\n \tint cleanup_addr; / Epilogue code offset /\n \n \t/\n@@ -329,34 +357,23 @@ struct jit_context {\n /* Number of bytes that will be skipped on tailcall /\n #define X86_TAIL_CALL_OFFSET\t(12 + ENDBR_INSN_SIZE)\n \n-static void push_r9(u8 pprog)\n+static void push_r9(struct jit_emit_context jit)\n {\n-\tu8 prog = pprog;\n-\n \tEMIT2(0x41, 0x51); /* push r9 /\n-\tpprog = prog;\n }\n \n-static void pop_r9(u8 *pprog)\n+static void pop_r9(struct jit_emit_context jit)\n {\n-\tu8 prog = pprog;\n-\n \tEMIT2(0x41, 0x59); /* pop r9 /\n-\tpprog = prog;\n }\n \n-static void push_r12(u8 *pprog)\n+static void push_r12(struct jit_emit_context jit)\n {\n-\tu8 prog = pprog;\n-\n \tEMIT2(0x41, 0x54); /* push r12 /\n-\tpprog = prog;\n }\n \n-static void push_callee_regs(u8 *pprog, bool callee_regs_used)\n+static void push_callee_regs(struct jit_emit_context jit, bool callee_regs_used)\n {\n-\tu8 prog = pprog;\n-\n \tif (callee_regs_used[0])\n \t\tEMIT1(0x53); /* push rbx /\n \tif (callee_regs_used[1])\n@@ -365,21 +382,15 @@ static void push_callee_regs(u8 pprog, bool callee_regs_used)\n \t\tEMIT2(0x41, 0x56); /* push r14 /\n \tif (callee_regs_used[3])\n \t\tEMIT2(0x41, 0x57); / push r15 /\n-\tpprog = prog;\n }\n \n-static void pop_r12(u8 *pprog)\n+static void pop_r12(struct jit_emit_context jit)\n {\n-\tu8 prog = pprog;\n-\n \tEMIT2(0x41, 0x5C); /* pop r12 /\n-\tpprog = prog;\n }\n \n-static void pop_callee_regs(u8 *pprog, bool callee_regs_used)\n+static void pop_callee_regs(struct jit_emit_context jit, bool callee_regs_used)\n {\n-\tu8 prog = pprog;\n-\n \tif (callee_regs_used[3])\n \t\tEMIT2(0x41, 0x5F); /* pop r15 /\n \tif (callee_regs_used[2])\n@@ -388,40 +399,32 @@ static void pop_callee_regs(u8 pprog, bool callee_regs_used)\n \t\tEMIT2(0x41, 0x5D); /* pop r13 /\n \tif (callee_regs_used[0])\n \t\tEMIT1(0x5B); / pop rbx /\n-\tpprog = prog;\n }\n \n /* add rsp, depth /\n-static void emit_add_rsp(u8 pprog, u16 depth)\n+static void emit_add_rsp(struct jit_emit_context jit, u16 depth)\n {\n-\tu8 prog = pprog;\n-\n \tif (!depth)\n \t\treturn;\n \tif (is_imm8(depth))\n \t\tEMIT4(0x48, 0x83, 0xC4, depth); /* add rsp, imm8 /\n \telse\n \t\tEMIT3_off32(0x48, 0x81, 0xC4, depth); / add rsp, imm32 /\n-\tpprog = prog;\n }\n \n /* sub rsp, depth /\n-static void emit_sub_rsp(u8 pprog, u16 depth)\n+static void emit_sub_rsp(struct jit_emit_context jit, u16 depth)\n {\n-\tu8 prog = pprog;\n-\n \tif (!depth)\n \t\treturn;\n \tif (is_imm8(depth))\n \t\tEMIT4(0x48, 0x83, 0xEC, depth); /* sub rsp, imm8 /\n \telse\n \t\tEMIT3_off32(0x48, 0x81, 0xEC, depth); / sub rsp, imm32 /\n-\tpprog = prog;\n }\n \n-static void emit_nops(u8 *pprog, int len)\n+static void emit_nops(struct jit_emit_context jit, int len)\n {\n-\tu8 prog = pprog;\n \tint i, noplen;\n \n \twhile (len \u003e 0) {\n@@ -434,72 +437,56 @@ static void emit_nops(u8 *pprog, int len)\n \t\t\tEMIT1(x86_nops[noplen][i]);\n \t\tlen -= noplen;\n \t}\n-\n-\tpprog = prog;\n }\n \n /\n Emit the various CFI preambles, see asm/cfi.h and the comments about FineIBT\n * in arch/x86/kernel/alternative.c\n /\n-static int emit_call(u8 prog, void func, void ip);\n+static int emit_call(struct jit_emit_context jit, void func, void ip);\n \n-static void emit_fineibt(u8 *pprog, u8 ip, u32 hash, int arity)\n+static void emit_fineibt(struct jit_emit_context jit, u8 ip, u32 hash, int arity)\n {\n-\tu8 prog = pprog;\n-\n \tEMIT_ENDBR();\n \tEMIT1_off32(0x2d, hash);\t\t\t/* subl $hash, %eax\t/\n \tif (cfi_bhi) {\n \t\tEMIT2(0x2e, 0x2e);\t\t\t/ cs cs /\n-\t\temit_call(\u0026prog, __bhi_args[arity], ip + 11);\n+\t\temit_call(jit, __bhi_args[arity], ip + 11);\n \t} else {\n \t\tEMIT3_off32(0x2e, 0x0f, 0x85, 3);\t/ jne.d32,pn 3\t\t/\n \t}\n \tEMIT_ENDBR_POISON();\n-\n-\tpprog = prog;\n }\n \n-static void emit_kcfi(u8 *pprog, u32 hash)\n+static void emit_kcfi(struct jit_emit_context jit, u32 hash)\n {\n-\tu8 prog = pprog;\n-\n \tEMIT1_off32(0xb8, hash);\t\t\t/* movl $hash, %eax\t/\n #ifdef CONFIG_CALL_PADDING\n \tfor (int i = 0; i \u003c CONFIG_FUNCTION_PADDING_CFI; i++)\n \t\tEMIT1(0x90);\n #endif\n \tEMIT_ENDBR();\n-\n-\tpprog = prog;\n }\n \n-static void emit_cfi(u8 *pprog, u8 ip, u32 hash, int arity)\n+static void emit_cfi(struct jit_emit_context jit, u8 ip, u32 hash, int arity)\n {\n-\tu8 prog = pprog;\n-\n \tswitch (cfi_mode) {\n \tcase CFI_FINEIBT:\n-\t\temit_fineibt(\u0026prog, ip, hash, arity);\n+\t\temit_fineibt(jit, ip, hash, arity);\n \t\tbreak;\n \n \tcase CFI_KCFI:\n-\t\temit_kcfi(\u0026prog, hash);\n+\t\temit_kcfi(jit, hash);\n \t\tbreak;\n \n \tdefault:\n \t\tEMIT_ENDBR();\n \t\tbreak;\n \t}\n-\n-\tpprog = prog;\n }\n \n-static void emit_prologue_tail_call(u8 pprog, bool is_subprog)\n+static void emit_prologue_tail_call(struct jit_emit_context jit, bool is_subprog)\n {\n-\tu8 prog = pprog;\n-\n \tif (!is_subprog) {\n \t\t/* cmp rax, MAX_TAIL_CALL_CNT /\n \t\tEMIT4(0x48, 0x83, 0xF8, MAX_TAIL_CALL_CNT);\n@@ -523,8 +510,6 @@ static void emit_prologue_tail_call(u8 pprog, bool is_subprog)\n \t\tEMIT1(0x50); / push rax /\n \t\tEMIT1(0x50); / push rax /\n \t}\n-\n-\tpprog = prog;\n }\n \n /\n@@ -532,21 +517,19 @@ static void emit_prologue_tail_call(u8 pprog, bool is_subprog)\n bpf_tail_call helper will skip the first X86_TAIL_CALL_OFFSET bytes\n * while jumping to another program\n /\n-static void emit_prologue(u8 pprog, u8 ip, u32 stack_depth, bool ebpf_from_cbpf,\n-\t\t\t bool tail_call_reachable, bool is_subprog,\n+static void emit_prologue(struct jit_emit_context jit, u8 ip, u32 stack_depth,\n+\t\t\t bool ebpf_from_cbpf, bool tail_call_reachable, bool is_subprog,\n \t\t\t bool is_exception_cb)\n {\n-\tu8 prog = pprog;\n-\n \tif (is_subprog) {\n-\t\temit_cfi(\u0026prog, ip, cfi_bpf_subprog_hash, 5);\n+\t\temit_cfi(jit, ip, cfi_bpf_subprog_hash, 5);\n \t} else {\n-\t\temit_cfi(\u0026prog, ip, cfi_bpf_hash, 1);\n+\t\temit_cfi(jit, ip, cfi_bpf_hash, 1);\n \t}\n \t/* BPF trampoline can be made to work without these nops,\n \t * but let's waste 5 bytes for now and optimize later\n \t /\n-\temit_nops(\u0026prog, X86_PATCH_SIZE);\n+\temit_nops(jit, X86_PATCH_SIZE);\n \tif (!ebpf_from_cbpf) {\n \t\tif (tail_call_reachable \u0026\u0026 !is_subprog)\n \t\t\t/ When it's the entry of the whole tailcall context,\n@@ -555,7 +538,7 @@ static void emit_prologue(u8 *pprog, u8 ip, u32 stack_depth, bool ebpf_from_cb\n \t\t\tEMIT3(0x48, 0x31, 0xC0); /* xor rax, rax /\n \t\telse\n \t\t\t/ Keep the same instruction layout. /\n-\t\t\temit_nops(\u0026prog, 3); / nop3 /\n+\t\t\temit_nops(jit, 3); / nop3 /\n \t}\n \t/ Exception callback receives FP as third parameter /\n \tif (is_exception_cb) {\n@@ -565,8 +548,8 @@ static void emit_prologue(u8 pprog, u8 ip, u32 stack_depth, bool ebpf_from_cb\n \t\t * first restore those callee-saved regs from stack, before\n \t\t * reusing the stack frame.\n \t\t /\n-\t\tpop_callee_regs(\u0026prog, all_callee_regs_used);\n-\t\tpop_r12(\u0026prog);\n+\t\tpop_callee_regs(jit, all_callee_regs_used);\n+\t\tpop_r12(jit);\n \t\t/ Reset the stack frame. /\n \t\tEMIT3(0x48, 0x89, 0xEC); / mov rsp, rbp /\n \t} else {\n@@ -581,40 +564,38 @@ static void emit_prologue(u8 pprog, u8 ip, u32 stack_depth, bool ebpf_from_cb\n \tif (stack_depth)\n \t\tEMIT3_off32(0x48, 0x81, 0xEC, round_up(stack_depth, 8));\n \tif (tail_call_reachable)\n-\t\temit_prologue_tail_call(\u0026prog, is_subprog);\n-\tpprog = prog;\n+\t\temit_prologue_tail_call(jit, is_subprog);\n }\n \n-static int emit_patch(u8 pprog, void func, void ip, u8 opcode)\n+static int emit_patch(struct jit_emit_context jit, void func, void ip, u8 opcode)\n {\n-\tu8 prog = pprog;\n \ts64 offset;\n \n \toffset = func - (ip + X86_PATCH_SIZE);\n-\tif (!is_simm32(offset)) {\n+\t/* We do not have meaningful ip value in the dry run, skip the check. /\n+\tif (!jit-\u003edry_run \u0026\u0026 !is_simm32(offset)) {\n \t\tpr_err(\"Target call %p is out of range\\n\", func);\n \t\treturn -ERANGE;\n \t}\n \tEMIT1_off32(opcode, offset);\n-\tpprog = prog;\n \treturn 0;\n }\n \n-static int emit_call(u8 *pprog, void func, void ip)\n+static int emit_call(struct jit_emit_context jit, void func, void ip)\n {\n-\treturn emit_patch(pprog, func, ip, 0xE8);\n+\treturn emit_patch(jit, func, ip, 0xE8);\n }\n \n-static int emit_rsb_call(u8 *pprog, void func, void ip)\n+static int emit_rsb_call(struct jit_emit_context jit, void func, void ip)\n {\n \tOPTIMIZER_HIDE_VAR(func);\n-\tip += x86_call_depth_emit_accounting(pprog, func, ip);\n-\treturn emit_patch(pprog, func, ip, 0xE8);\n+\tip += bpf_call_depth_emit_accounting(jit, func, ip);\n+\treturn emit_patch(jit, func, ip, 0xE8);\n }\n \n-static int emit_jump(u8 *pprog, void func, void ip)\n+static int emit_jump(struct jit_emit_context jit, void func, void ip)\n {\n-\treturn emit_patch(pprog, func, ip, 0xE9);\n+\treturn emit_patch(jit, func, ip, 0xE9);\n }\n \n static int __bpf_arch_text_poke(void ip, enum bpf_text_poke_type old_t,\n@@ -622,27 +603,27 @@ static int __bpf_arch_text_poke(void ip, enum bpf_text_poke_type old_t,\n \t\t\t\tvoid old_addr, void new_addr)\n {\n \tconst u8 nop_insn = x86_nops[5];\n+\tstruct jit_emit_context jit = {};\n \tu8 old_insn[X86_PATCH_SIZE];\n \tu8 new_insn[X86_PATCH_SIZE];\n-\tu8 prog;\n \tint ret;\n \n \tmemcpy(old_insn, nop_insn, X86_PATCH_SIZE);\n \tif (old_t != BPF_MOD_NOP \u0026\u0026 old_addr) {\n-\t\tprog = old_insn;\n+\t\tjit.prog = old_insn;\n \t\tret = old_t == BPF_MOD_CALL ?\n-\t\t emit_call(\u0026prog, old_addr, ip) :\n-\t\t emit_jump(\u0026prog, old_addr, ip);\n+\t\t emit_call(\u0026jit, old_addr, ip) :\n+\t\t emit_jump(\u0026jit, old_addr, ip);\n \t\tif (ret)\n \t\t\treturn ret;\n \t}\n \n \tmemcpy(new_insn, nop_insn, X86_PATCH_SIZE);\n \tif (new_t != BPF_MOD_NOP \u0026\u0026 new_addr) {\n-\t\tprog = new_insn;\n+\t\tjit.prog = new_insn;\n \t\tret = new_t == BPF_MOD_CALL ?\n-\t\t emit_call(\u0026prog, new_addr, ip) :\n-\t\t emit_jump(\u0026prog, new_addr, ip);\n+\t\t emit_call(\u0026jit, new_addr, ip) :\n+\t\t emit_jump(\u0026jit, new_addr, ip);\n \t\tif (ret)\n \t\t\treturn ret;\n \t}\n@@ -682,58 +663,47 @@ int bpf_arch_text_poke(void ip, enum bpf_text_poke_type old_t,\n \n #define EMIT_LFENCE()\tEMIT3(0x0F, 0xAE, 0xE8)\n \n-static void __emit_indirect_jump(u8 pprog, int reg, bool ereg)\n+static void __emit_indirect_jump(struct jit_emit_context jit, int reg, bool ereg)\n {\n-\tu8 prog = pprog;\n-\n \tif (ereg)\n \t\tEMIT1(0x41);\n \n \tEMIT2(0xFF, 0xE0 + reg);\n-\n-\tpprog = prog;\n }\n \n-static void emit_indirect_jump(u8 pprog, int bpf_reg, u8 ip)\n+static void emit_indirect_jump(struct jit_emit_context jit, int bpf_reg, u8 ip)\n {\n-\tu8 prog = pprog;\n \tint reg = reg2hex[bpf_reg];\n \tbool ereg = is_ereg(bpf_reg);\n \n \tif (cpu_feature_enabled(X86_FEATURE_INDIRECT_THUNK_ITS)) {\n \t\tOPTIMIZER_HIDE_VAR(reg);\n-\t\temit_jump(\u0026prog, its_static_thunk(reg + 8ereg), ip);\n+\t\temit_jump(jit, its_static_thunk(reg + 8ereg), ip);\n \t} else if (cpu_feature_enabled(X86_FEATURE_RETPOLINE_LFENCE)) {\n \t\tEMIT_LFENCE();\n-\t\t__emit_indirect_jump(\u0026prog, reg, ereg);\n+\t\t__emit_indirect_jump(jit, reg, ereg);\n \t} else if (cpu_feature_enabled(X86_FEATURE_RETPOLINE)) {\n \t\tOPTIMIZER_HIDE_VAR(reg);\n \t\tif (cpu_feature_enabled(X86_FEATURE_CALL_DEPTH))\n-\t\t\temit_jump(\u0026prog, \u0026__x86_indirect_jump_thunk_array[reg + 8ereg], ip);\n+\t\t\temit_jump(jit, \u0026__x86_indirect_jump_thunk_array[reg + 8ereg], ip);\n \t\telse\n-\t\t\temit_jump(\u0026prog, \u0026__x86_indirect_thunk_array[reg + 8ereg], ip);\n+\t\t\temit_jump(jit, \u0026__x86_indirect_thunk_array[reg + 8ereg], ip);\n \t} else {\n-\t\t__emit_indirect_jump(\u0026prog, reg, ereg);\n+\t\t__emit_indirect_jump(jit, reg, ereg);\n \t\tif (IS_ENABLED(CONFIG_MITIGATION_RETPOLINE) \|\| IS_ENABLED(CONFIG_MITIGATION_SLS))\n \t\t\tEMIT1(0xCC);\t\t/* int3 /\n \t}\n-\n-\tpprog = prog;\n }\n \n-static void emit_return(u8 *pprog, u8 ip)\n+static void emit_return(struct jit_emit_context jit, u8 ip)\n {\n-\tu8 prog = pprog;\n-\n \tif (cpu_wants_rethunk()) {\n-\t\temit_jump(\u0026prog, x86_return_thunk, ip);\n+\t\temit_jump(jit, x86_return_thunk, ip);\n \t} else {\n \t\tEMIT1(0xC3);\t\t/* ret /\n \t\tif (IS_ENABLED(CONFIG_MITIGATION_SLS))\n \t\t\tEMIT1(0xCC);\t/ int3 /\n \t}\n-\n-\tpprog = prog;\n }\n \n #define BPF_TAIL_CALL_CNT_PTR_STACK_OFF(stack)\t(-16 - round_up(stack, 8))\n@@ -753,12 +723,13 @@ static void emit_return(u8 *pprog, u8 ip)\n * out:\n /\n static void emit_bpf_tail_call_indirect(struct bpf_prog bpf_prog,\n-\t\t\t\t\tu8 *pprog, bool callee_regs_used,\n+\t\t\t\t\tbool callee_regs_used,\n \t\t\t\t\tu32 stack_depth, u8 ip,\n \t\t\t\t\tstruct jit_context ctx)\n {\n \tint tcc_ptr_off = BPF_TAIL_CALL_CNT_PTR_STACK_OFF(stack_depth);\n-\tu8 prog = pprog, start = pprog;\n+\tstruct jit_emit_context jit = \u0026ctx-\u003ejit;\n+\tu8 start = jit-\u003eprog;\n \tint offset;\n \n \t/\n@@ -775,7 +746,7 @@ static void emit_bpf_tail_call_indirect(struct bpf_prog bpf_prog,\n \tEMIT3(0x39, 0x56, / cmp dword ptr [rsi + 16], edx /\n \t offsetof(struct bpf_array, map.max_entries));\n \n-\toffset = ctx-\u003etail_call_indirect_label - (prog + 2 - start);\n+\toffset = ctx-\u003etail_call_indirect_label - (jit-\u003eprog + 2 - start);\n \tEMIT2(X86_JBE, offset); / jbe out /\n \n \t/\n@@ -785,7 +756,7 @@ static void emit_bpf_tail_call_indirect(struct bpf_prog bpf_prog,\n \tEMIT3_off32(0x48, 0x8B, 0x85, tcc_ptr_off); / mov rax, qword ptr [rbp - tcc_ptr_off] /\n \tEMIT4(0x48, 0x83, 0x38, MAX_TAIL_CALL_CNT); / cmp qword ptr [rax], MAX_TAIL_CALL_CNT /\n \n-\toffset = ctx-\u003etail_call_indirect_label - (prog + 2 - start);\n+\toffset = ctx-\u003etail_call_indirect_label - (jit-\u003eprog + 2 - start);\n \tEMIT2(X86_JAE, offset); / jae out /\n \n \t/ prog = array-\u003eptrs[index]; /\n@@ -798,19 +769,19 @@ static void emit_bpf_tail_call_indirect(struct bpf_prog bpf_prog,\n \t /\n \tEMIT3(0x48, 0x85, 0xC9); / test rcx,rcx /\n \n-\toffset = ctx-\u003etail_call_indirect_label - (prog + 2 - start);\n+\toffset = ctx-\u003etail_call_indirect_label - (jit-\u003eprog + 2 - start);\n \tEMIT2(X86_JE, offset); / je out /\n \n \t/ Inc tail_call_cnt if the slot is populated. /\n \tEMIT4(0x48, 0x83, 0x00, 0x01); / add qword ptr [rax], 1 /\n \n \tif (bpf_prog-\u003eaux-\u003eexception_boundary) {\n-\t\tpop_callee_regs(\u0026prog, all_callee_regs_used);\n-\t\tpop_r12(\u0026prog);\n+\t\tpop_callee_regs(jit, all_callee_regs_used);\n+\t\tpop_r12(jit);\n \t} else {\n-\t\tpop_callee_regs(\u0026prog, callee_regs_used);\n+\t\tpop_callee_regs(jit, callee_regs_used);\n \t\tif (bpf_arena_get_kern_vm_start(bpf_prog-\u003eaux-\u003earena))\n-\t\t\tpop_r12(\u0026prog);\n+\t\t\tpop_r12(jit);\n \t}\n \n \t/ Pop tail_call_cnt_ptr. /\n@@ -833,21 +804,21 @@ static void emit_bpf_tail_call_indirect(struct bpf_prog bpf_prog,\n \t * rdi == ctx (1st arg)\n \t * rcx == prog-\u003ebpf_func + X86_TAIL_CALL_OFFSET\n \t /\n-\temit_indirect_jump(\u0026prog, BPF_REG_4 / R4 -\u003e rcx /, ip + (prog - start));\n+\temit_indirect_jump(jit, BPF_REG_4 / R4 -\u003e rcx /, ip + (jit-\u003eprog - start));\n \n \t/ out: /\n-\tctx-\u003etail_call_indirect_label = prog - start;\n-\tpprog = prog;\n+\tctx-\u003etail_call_indirect_label = jit-\u003eprog - start;\n }\n \n static void emit_bpf_tail_call_direct(struct bpf_prog bpf_prog,\n \t\t\t\t struct bpf_jit_poke_descriptor poke,\n-\t\t\t\t u8 *pprog, u8 ip,\n+\t\t\t\t u8 ip,\n \t\t\t\t bool callee_regs_used, u32 stack_depth,\n \t\t\t\t struct jit_context ctx)\n {\n+\tstruct jit_emit_context jit = \u0026ctx-\u003ejit;\n \tint tcc_ptr_off = BPF_TAIL_CALL_CNT_PTR_STACK_OFF(stack_depth);\n-\tu8 prog = pprog, start = pprog;\n+\tu8 start = jit-\u003eprog;\n \tint offset;\n \n \t/\n@@ -857,27 +828,27 @@ static void emit_bpf_tail_call_direct(struct bpf_prog bpf_prog,\n \tEMIT3_off32(0x48, 0x8B, 0x85, tcc_ptr_off); / mov rax, qword ptr [rbp - tcc_ptr_off] /\n \tEMIT4(0x48, 0x83, 0x38, MAX_TAIL_CALL_CNT); / cmp qword ptr [rax], MAX_TAIL_CALL_CNT /\n \n-\toffset = ctx-\u003etail_call_direct_label - (prog + 2 - start);\n+\toffset = ctx-\u003etail_call_direct_label - (jit-\u003eprog + 2 - start);\n \tEMIT2(X86_JAE, offset); / jae out /\n \n-\tpoke-\u003etailcall_bypass = ip + (prog - start);\n+\tpoke-\u003etailcall_bypass = ip + (jit-\u003eprog - start);\n \tpoke-\u003eadj_off = X86_TAIL_CALL_OFFSET;\n \tpoke-\u003etailcall_target = ip + ctx-\u003etail_call_direct_label - X86_PATCH_SIZE;\n \tpoke-\u003ebypass_addr = (u8 )poke-\u003etailcall_target + X86_PATCH_SIZE;\n \n-\temit_jump(\u0026prog, (u8 )poke-\u003etailcall_target + X86_PATCH_SIZE,\n+\temit_jump(jit, (u8 )poke-\u003etailcall_target + X86_PATCH_SIZE,\n \t\t poke-\u003etailcall_bypass);\n \n \t/* Inc tail_call_cnt if the slot is populated. /\n \tEMIT4(0x48, 0x83, 0x00, 0x01); / add qword ptr [rax], 1 /\n \n \tif (bpf_prog-\u003eaux-\u003eexception_boundary) {\n-\t\tpop_callee_regs(\u0026prog, all_callee_regs_used);\n-\t\tpop_r12(\u0026prog);\n+\t\tpop_callee_regs(jit, all_callee_regs_used);\n+\t\tpop_r12(jit);\n \t} else {\n-\t\tpop_callee_regs(\u0026prog, callee_regs_used);\n+\t\tpop_callee_regs(jit, callee_regs_used);\n \t\tif (bpf_arena_get_kern_vm_start(bpf_prog-\u003eaux-\u003earena))\n-\t\t\tpop_r12(\u0026prog);\n+\t\t\tpop_r12(jit);\n \t}\n \n \t/ Pop tail_call_cnt_ptr. /\n@@ -889,12 +860,10 @@ static void emit_bpf_tail_call_direct(struct bpf_prog bpf_prog,\n \tif (stack_depth)\n \t\tEMIT3_off32(0x48, 0x81, 0xC4, round_up(stack_depth, 8));\n \n-\temit_nops(\u0026prog, X86_PATCH_SIZE);\n+\temit_nops(jit, X86_PATCH_SIZE);\n \n \t/* out: /\n-\tctx-\u003etail_call_direct_label = prog - start;\n-\n-\tpprog = prog;\n+\tctx-\u003etail_call_direct_label = jit-\u003eprog - start;\n }\n \n static void bpf_tail_call_direct_fixup(struct bpf_prog prog)\n@@ -935,10 +904,9 @@ static void bpf_tail_call_direct_fixup(struct bpf_prog prog)\n \t}\n }\n \n-static void emit_mov_imm32(u8 *pprog, bool sign_propagate,\n+static void emit_mov_imm32(struct jit_emit_context jit, bool sign_propagate,\n \t\t\t u32 dst_reg, const u32 imm32)\n {\n-\tu8 prog = pprog;\n \tu8 b1, b2, b3;\n \n \t/\n@@ -972,14 +940,12 @@ static void emit_mov_imm32(u8 pprog, bool sign_propagate,\n \t\tEMIT1(add_1mod(0x40, dst_reg));\n \tEMIT1_off32(add_1reg(0xB8, dst_reg), imm32);\n done:\n-\tpprog = prog;\n }\n \n-static void emit_mov_imm64(u8 *pprog, u32 dst_reg,\n+static void emit_mov_imm64(struct jit_emit_context jit, u32 dst_reg,\n \t\t\t const u32 imm32_hi, const u32 imm32_lo)\n {\n \tu64 imm64 = ((u64)imm32_hi \u003c\u003c 32) \| (u32)imm32_lo;\n-\tu8 prog = pprog;\n \n \tif (is_uimm32(imm64)) {\n \t\t/\n@@ -988,23 +954,19 @@ static void emit_mov_imm64(u8 pprog, u32 dst_reg,\n \t\t directly, so save couple of bytes by just doing\n \t\t * 'mov %eax, imm32' instead.\n \t\t /\n-\t\temit_mov_imm32(\u0026prog, false, dst_reg, imm32_lo);\n+\t\temit_mov_imm32(jit, false, dst_reg, imm32_lo);\n \t} else if (is_simm32(imm64)) {\n-\t\temit_mov_imm32(\u0026prog, true, dst_reg, imm32_lo);\n+\t\temit_mov_imm32(jit, true, dst_reg, imm32_lo);\n \t} else {\n \t\t/ movabsq rax, imm64 /\n \t\tEMIT2(add_1mod(0x48, dst_reg), add_1reg(0xB8, dst_reg));\n \t\tEMIT(imm32_lo, 4);\n \t\tEMIT(imm32_hi, 4);\n \t}\n-\n-\tpprog = prog;\n }\n \n-static void emit_mov_reg(u8 *pprog, bool is64, u32 dst_reg, u32 src_reg)\n+static void emit_mov_reg(struct jit_emit_context jit, bool is64, u32 dst_reg, u32 src_reg)\n {\n-\tu8 prog = pprog;\n-\n \tif (is64) {\n \t\t/* mov dst, src /\n \t\tEMIT_mov(dst_reg, src_reg);\n@@ -1014,15 +976,11 @@ static void emit_mov_reg(u8 pprog, bool is64, u32 dst_reg, u32 src_reg)\n \t\t\tEMIT1(add_2mod(0x40, dst_reg, src_reg));\n \t\tEMIT2(0x89, add_2reg(0xC0, dst_reg, src_reg));\n \t}\n-\n-\tpprog = prog;\n }\n \n-static void emit_movsx_reg(u8 *pprog, int num_bits, bool is64, u32 dst_reg,\n+static void emit_movsx_reg(struct jit_emit_context jit, int num_bits, bool is64, u32 dst_reg,\n \t\t\t u32 src_reg)\n {\n-\tu8 prog = pprog;\n-\n \tif (is64) {\n \t\t/* movs[b,w,l]q dst, src /\n \t\tif (num_bits == 8)\n@@ -1046,15 +1004,11 @@ static void emit_movsx_reg(u8 pprog, int num_bits, bool is64, u32 dst_reg,\n \t\t\t add_2reg(0xC0, src_reg, dst_reg));\n \t\t}\n \t}\n-\n-\tpprog = prog;\n }\n \n /* Emit the suffix (ModR/M etc) for addressing (ptr_reg + off) and val_reg /\n-static void emit_insn_suffix(u8 *pprog, u32 ptr_reg, u32 val_reg, int off)\n+static void emit_insn_suffix(struct jit_emit_context jit, u32 ptr_reg, u32 val_reg, int off)\n {\n-\tu8 prog = pprog;\n-\n \tif (is_imm8(off)) {\n \t\t/* 1-byte signed displacement.\n \t\t \n@@ -1067,54 +1021,43 @@ static void emit_insn_suffix(u8 pprog, u32 ptr_reg, u32 val_reg, int off)\n \t\t/ 4-byte signed displacement /\n \t\tEMIT1_off32(add_2reg(0x80, ptr_reg, val_reg), off);\n \t}\n-\tpprog = prog;\n }\n \n-static void emit_insn_suffix_SIB(u8 *pprog, u32 ptr_reg, u32 val_reg, u32 index_reg, int off)\n+static void emit_insn_suffix_SIB(struct jit_emit_context jit, u32 ptr_reg, u32 val_reg,\n+\t\t\t\t u32 index_reg, int off)\n {\n-\tu8 prog = pprog;\n-\n \tif (is_imm8(off)) {\n \t\tEMIT3(add_2reg(0x44, BPF_REG_0, val_reg), add_2reg(0, ptr_reg, index_reg) /* SIB /, off);\n \t} else {\n \t\tEMIT2_off32(add_2reg(0x84, BPF_REG_0, val_reg), add_2reg(0, ptr_reg, index_reg) / SIB /, off);\n \t}\n-\tpprog = prog;\n }\n \n /\n Emit a REX byte if it will be necessary to address these registers\n /\n-static void maybe_emit_mod(u8 pprog, u32 dst_reg, u32 src_reg, bool is64)\n+static void maybe_emit_mod(struct jit_emit_context jit, u32 dst_reg, u32 src_reg, bool is64)\n {\n-\tu8 prog = pprog;\n-\n \tif (is64)\n \t\tEMIT1(add_2mod(0x48, dst_reg, src_reg));\n \telse if (is_ereg(dst_reg) \|\| is_ereg(src_reg))\n \t\tEMIT1(add_2mod(0x40, dst_reg, src_reg));\n-\tpprog = prog;\n }\n \n /\n * Similar version of maybe_emit_mod() for a single register\n /\n-static void maybe_emit_1mod(u8 pprog, u32 reg, bool is64)\n+static void maybe_emit_1mod(struct jit_emit_context jit, u32 reg, bool is64)\n {\n-\tu8 prog = pprog;\n-\n \tif (is64)\n \t\tEMIT1(add_1mod(0x48, reg));\n \telse if (is_ereg(reg))\n \t\tEMIT1(add_1mod(0x40, reg));\n-\tpprog = prog;\n }\n \n / LDX: dst_reg = (u8)(src_reg + off) /\n-static void emit_ldx(u8 pprog, u32 size, u32 dst_reg, u32 src_reg, int off)\n+static void emit_ldx(struct jit_emit_context jit, u32 size, u32 dst_reg, u32 src_reg, int off)\n {\n-\tu8 prog = pprog;\n-\n \tswitch (size) {\n \tcase BPF_B:\n \t\t/* Emit 'movzx rax, byte ptr [rax + off]' /\n@@ -1136,15 +1079,12 @@ static void emit_ldx(u8 pprog, u32 size, u32 dst_reg, u32 src_reg, int off)\n \t\tEMIT2(add_2mod(0x48, src_reg, dst_reg), 0x8B);\n \t\tbreak;\n \t}\n-\temit_insn_suffix(\u0026prog, src_reg, dst_reg, off);\n-\tpprog = prog;\n+\temit_insn_suffix(jit, src_reg, dst_reg, off);\n }\n \n /* LDSX: dst_reg = (s8)(src_reg + off) /\n-static void emit_ldsx(u8 pprog, u32 size, u32 dst_reg, u32 src_reg, int off)\n+static void emit_ldsx(struct jit_emit_context jit, u32 size, u32 dst_reg, u32 src_reg, int off)\n {\n-\tu8 prog = pprog;\n-\n \tswitch (size) {\n \tcase BPF_B:\n \t\t/* Emit 'movsx rax, byte ptr [rax + off]' /\n@@ -1159,14 +1099,12 @@ static void emit_ldsx(u8 pprog, u32 size, u32 dst_reg, u32 src_reg, int off)\n \t\tEMIT2(add_2mod(0x48, src_reg, dst_reg), 0x63);\n \t\tbreak;\n \t}\n-\temit_insn_suffix(\u0026prog, src_reg, dst_reg, off);\n-\tpprog = prog;\n+\temit_insn_suffix(jit, src_reg, dst_reg, off);\n }\n \n-static void emit_ldx_index(u8 *pprog, u32 size, u32 dst_reg, u32 src_reg, u32 index_reg, int off)\n+static void emit_ldx_index(struct jit_emit_context jit, u32 size, u32 dst_reg, u32 src_reg,\n+\t\t\t u32 index_reg, int off)\n {\n-\tu8 prog = pprog;\n-\n \tswitch (size) {\n \tcase BPF_B:\n \t\t/* movzx rax, byte ptr [rax + r12 + off] /\n@@ -1185,14 +1123,12 @@ static void emit_ldx_index(u8 pprog, u32 size, u32 dst_reg, u32 src_reg, u32 i\n \t\tEMIT2(add_3mod(0x48, src_reg, dst_reg, index_reg), 0x8B);\n \t\tbreak;\n \t}\n-\temit_insn_suffix_SIB(\u0026prog, src_reg, dst_reg, index_reg, off);\n-\tpprog = prog;\n+\temit_insn_suffix_SIB(jit, src_reg, dst_reg, index_reg, off);\n }\n \n-static void emit_ldsx_index(u8 *pprog, u32 size, u32 dst_reg, u32 src_reg, u32 index_reg, int off)\n+static void emit_ldsx_index(struct jit_emit_context jit, u32 size, u32 dst_reg, u32 src_reg,\n+\t\t\t u32 index_reg, int off)\n {\n-\tu8 prog = pprog;\n-\n \tswitch (size) {\n \tcase BPF_B:\n \t\t/* movsx rax, byte ptr [rax + r12 + off] /\n@@ -1207,25 +1143,22 @@ static void emit_ldsx_index(u8 pprog, u32 size, u32 dst_reg, u32 src_reg, u32\n \t\tEMIT2(add_3mod(0x48, src_reg, dst_reg, index_reg), 0x63);\n \t\tbreak;\n \t}\n-\temit_insn_suffix_SIB(\u0026prog, src_reg, dst_reg, index_reg, off);\n-\tpprog = prog;\n+\temit_insn_suffix_SIB(jit, src_reg, dst_reg, index_reg, off);\n }\n \n-static void emit_ldx_r12(u8 *pprog, u32 size, u32 dst_reg, u32 src_reg, int off)\n+static void emit_ldx_r12(struct jit_emit_context jit, u32 size, u32 dst_reg, u32 src_reg, int off)\n {\n-\temit_ldx_index(pprog, size, dst_reg, src_reg, X86_REG_R12, off);\n+\temit_ldx_index(jit, size, dst_reg, src_reg, X86_REG_R12, off);\n }\n \n-static void emit_ldsx_r12(u8 *prog, u32 size, u32 dst_reg, u32 src_reg, int off)\n+static void emit_ldsx_r12(struct jit_emit_context jit, u32 size, u32 dst_reg, u32 src_reg, int off)\n {\n-\temit_ldsx_index(prog, size, dst_reg, src_reg, X86_REG_R12, off);\n+\temit_ldsx_index(jit, size, dst_reg, src_reg, X86_REG_R12, off);\n }\n \n /* STX: (u8)(dst_reg + off) = src_reg /\n-static void emit_stx(u8 pprog, u32 size, u32 dst_reg, u32 src_reg, int off)\n+static void emit_stx(struct jit_emit_context jit, u32 size, u32 dst_reg, u32 src_reg, int off)\n {\n-\tu8 prog = pprog;\n-\n \tswitch (size) {\n \tcase BPF_B:\n \t\t/* Emit 'mov byte ptr [rax + off], al' /\n@@ -1251,15 +1184,13 @@ static void emit_stx(u8 pprog, u32 size, u32 dst_reg, u32 src_reg, int off)\n \t\tEMIT2(add_2mod(0x48, dst_reg, src_reg), 0x89);\n \t\tbreak;\n \t}\n-\temit_insn_suffix(\u0026prog, dst_reg, src_reg, off);\n-\tpprog = prog;\n+\temit_insn_suffix(jit, dst_reg, src_reg, off);\n }\n \n /* STX: (u8)(dst_reg + index_reg + off) = src_reg /\n-static void emit_stx_index(u8 pprog, u32 size, u32 dst_reg, u32 src_reg, u32 index_reg, int off)\n+static void emit_stx_index(struct jit_emit_context jit, u32 size, u32 dst_reg, u32 src_reg,\n+\t\t\t u32 index_reg, int off)\n {\n-\tu8 prog = pprog;\n-\n \tswitch (size) {\n \tcase BPF_B:\n \t\t/* mov byte ptr [rax + r12 + off], al /\n@@ -1278,20 +1209,18 @@ static void emit_stx_index(u8 pprog, u32 size, u32 dst_reg, u32 src_reg, u32 i\n \t\tEMIT2(add_3mod(0x48, dst_reg, src_reg, index_reg), 0x89);\n \t\tbreak;\n \t}\n-\temit_insn_suffix_SIB(\u0026prog, dst_reg, src_reg, index_reg, off);\n-\tpprog = prog;\n+\temit_insn_suffix_SIB(jit, dst_reg, src_reg, index_reg, off);\n }\n \n-static void emit_stx_r12(u8 *pprog, u32 size, u32 dst_reg, u32 src_reg, int off)\n+static void emit_stx_r12(struct jit_emit_context jit, u32 size, u32 dst_reg, u32 src_reg, int off)\n {\n-\temit_stx_index(pprog, size, dst_reg, src_reg, X86_REG_R12, off);\n+\temit_stx_index(jit, size, dst_reg, src_reg, X86_REG_R12, off);\n }\n \n /* ST: (u8)(dst_reg + index_reg + off) = imm32 /\n-static void emit_st_index(u8 pprog, u32 size, u32 dst_reg, u32 index_reg, int off, int imm)\n+static void emit_st_index(struct jit_emit_context jit, u32 size, u32 dst_reg, u32 index_reg,\n+\t\t\t int off, int imm)\n {\n-\tu8 prog = pprog;\n-\n \tswitch (size) {\n \tcase BPF_B:\n \t\t/* mov byte ptr [rax + r12 + off], imm8 /\n@@ -1310,35 +1239,32 @@ static void emit_st_index(u8 pprog, u32 size, u32 dst_reg, u32 index_reg, int\n \t\tEMIT2(add_3mod(0x48, dst_reg, 0, index_reg), 0xC7);\n \t\tbreak;\n \t}\n-\temit_insn_suffix_SIB(\u0026prog, dst_reg, 0, index_reg, off);\n+\temit_insn_suffix_SIB(jit, dst_reg, 0, index_reg, off);\n \tEMIT(imm, bpf_size_to_x86_bytes(size));\n-\tpprog = prog;\n }\n \n-static void emit_st_r12(u8 *pprog, u32 size, u32 dst_reg, int off, int imm)\n+static void emit_st_r12(struct jit_emit_context jit, u32 size, u32 dst_reg, int off, int imm)\n {\n-\temit_st_index(pprog, size, dst_reg, X86_REG_R12, off, imm);\n+\temit_st_index(jit, size, dst_reg, X86_REG_R12, off, imm);\n }\n \n-static void emit_store_stack_imm64(u8 *pprog, int reg, int stack_off, u64 imm64)\n+static void emit_store_stack_imm64(struct jit_emit_context jit, int reg, int stack_off, u64 imm64)\n {\n \t/\n \t mov reg, imm64\n \t * mov QWORD PTR [rbp + stack_off], reg\n \t /\n-\temit_mov_imm64(pprog, reg, imm64 \u003e\u003e 32, (u32) imm64);\n-\temit_stx(pprog, BPF_DW, BPF_REG_FP, reg, stack_off);\n+\temit_mov_imm64(jit, reg, imm64 \u003e\u003e 32, (u32) imm64);\n+\temit_stx(jit, BPF_DW, BPF_REG_FP, reg, stack_off);\n }\n \n-static int emit_atomic_rmw(u8 pprog, u32 atomic_op,\n+static int emit_atomic_rmw(struct jit_emit_context jit, u32 atomic_op,\n \t\t\t u32 dst_reg, u32 src_reg, s16 off, u8 bpf_size)\n {\n-\tu8 prog = pprog;\n-\n \tif (atomic_op != BPF_XCHG)\n \t\tEMIT1(0xF0); /* lock prefix /\n \n-\tmaybe_emit_mod(\u0026prog, dst_reg, src_reg, bpf_size == BPF_DW);\n+\tmaybe_emit_mod(jit, dst_reg, src_reg, bpf_size == BPF_DW);\n \n \t/ emit opcode /\n \tswitch (atomic_op) {\n@@ -1366,18 +1292,15 @@ static int emit_atomic_rmw(u8 pprog, u32 atomic_op,\n \t\treturn -EFAULT;\n \t}\n \n-\temit_insn_suffix(\u0026prog, dst_reg, src_reg, off);\n+\temit_insn_suffix(jit, dst_reg, src_reg, off);\n \n-\tpprog = prog;\n \treturn 0;\n }\n \n-static int emit_atomic_rmw_index(u8 *pprog, u32 atomic_op, u32 size,\n+static int emit_atomic_rmw_index(struct jit_emit_context jit, u32 atomic_op, u32 size,\n \t\t\t\t u32 dst_reg, u32 src_reg, u32 index_reg,\n \t\t\t\t int off)\n {\n-\tu8 prog = pprog;\n-\n \tif (atomic_op != BPF_XCHG)\n \t\tEMIT1(0xF0); /* lock prefix /\n \n@@ -1418,22 +1341,21 @@ static int emit_atomic_rmw_index(u8 pprog, u32 atomic_op, u32 size,\n \t\tpr_err(\"bpf_jit: unknown atomic opcode %02x\\n\", atomic_op);\n \t\treturn -EFAULT;\n \t}\n-\temit_insn_suffix_SIB(\u0026prog, dst_reg, src_reg, index_reg, off);\n-\tpprog = prog;\n+\temit_insn_suffix_SIB(jit, dst_reg, src_reg, index_reg, off);\n \treturn 0;\n }\n \n-static int emit_atomic_ld_st(u8 *pprog, u32 atomic_op, u32 dst_reg,\n+static int emit_atomic_ld_st(struct jit_emit_context jit, u32 atomic_op, u32 dst_reg,\n \t\t\t u32 src_reg, s16 off, u8 bpf_size)\n {\n \tswitch (atomic_op) {\n \tcase BPF_LOAD_ACQ:\n \t\t/* dst_reg = smp_load_acquire(src_reg + off16) /\n-\t\temit_ldx(pprog, bpf_size, dst_reg, src_reg, off);\n+\t\temit_ldx(jit, bpf_size, dst_reg, src_reg, off);\n \t\tbreak;\n \tcase BPF_STORE_REL:\n \t\t/ smp_store_release(dst_reg + off16, src_reg) /\n-\t\temit_stx(pprog, bpf_size, dst_reg, src_reg, off);\n+\t\temit_stx(jit, bpf_size, dst_reg, src_reg, off);\n \t\tbreak;\n \tdefault:\n \t\tpr_err(\"bpf_jit: unknown atomic load/store opcode %02x\\n\",\n@@ -1444,18 +1366,18 @@ static int emit_atomic_ld_st(u8 pprog, u32 atomic_op, u32 dst_reg,\n \treturn 0;\n }\n \n-static int emit_atomic_ld_st_index(u8 pprog, u32 atomic_op, u32 size,\n+static int emit_atomic_ld_st_index(struct jit_emit_context jit, u32 atomic_op, u32 size,\n \t\t\t\t u32 dst_reg, u32 src_reg, u32 index_reg,\n \t\t\t\t int off)\n {\n \tswitch (atomic_op) {\n \tcase BPF_LOAD_ACQ:\n \t\t/* dst_reg = smp_load_acquire(src_reg + idx_reg + off16) /\n-\t\temit_ldx_index(pprog, size, dst_reg, src_reg, index_reg, off);\n+\t\temit_ldx_index(jit, size, dst_reg, src_reg, index_reg, off);\n \t\tbreak;\n \tcase BPF_STORE_REL:\n \t\t/ smp_store_release(dst_reg + idx_reg + off16, src_reg) /\n-\t\temit_stx_index(pprog, size, dst_reg, src_reg, index_reg, off);\n+\t\temit_stx_index(jit, size, dst_reg, src_reg, index_reg, off);\n \t\tbreak;\n \tdefault:\n \t\tpr_err(\"bpf_jit: unknown atomic load/store opcode %02x\\n\",\n@@ -1562,10 +1484,9 @@ static void detect_reg_usage(struct bpf_insn insn, int insn_cnt,\n * l: vector length (128 bit or 256 bit) or reserved\n * pp: opcode prefix (none, 0x66, 0xf2 or 0xf3)\n /\n-static void emit_3vex(u8 pprog, bool r, bool x, bool b, u8 m,\n+static void emit_3vex(struct jit_emit_context jit, bool r, bool x, bool b, u8 m,\n \t\t bool w, u8 src_reg2, bool l, u8 pp)\n {\n-\tu8 prog = pprog;\n \tconst u8 b0 = 0xc4; /* first byte of 3-byte VEX prefix /\n \tu8 b1, b2;\n \tu8 vvvv = reg2hex[src_reg2];\n@@ -1595,27 +1516,22 @@ static void emit_3vex(u8 pprog, bool r, bool x, bool b, u8 m,\n \tb2 = (w \u003c\u003c 7) \| ((~vvvv \u0026 0xf) \u003c\u003c 3) \| (l \u003c\u003c 2) \| (pp \u0026 3);\n \n \tEMIT3(b0, b1, b2);\n-\tpprog = prog;\n }\n \n /* emit BMI2 shift instruction /\n-static void emit_shiftx(u8 pprog, u32 dst_reg, u8 src_reg, bool is64, u8 op)\n+static void emit_shiftx(struct jit_emit_context jit, u32 dst_reg, u8 src_reg, bool is64, u8 op)\n {\n-\tu8 prog = pprog;\n \tbool r = is_ereg(dst_reg);\n \tu8 m = 2; /* escape code 0f38 /\n \n-\temit_3vex(\u0026prog, r, false, r, m, is64, src_reg, false, op);\n+\temit_3vex(jit, r, false, r, m, is64, src_reg, false, op);\n \tEMIT2(0xf7, add_2reg(0xC0, dst_reg, dst_reg));\n-\tpprog = prog;\n }\n \n-static void emit_priv_frame_ptr(u8 *pprog, void __percpu priv_frame_ptr)\n+static void emit_priv_frame_ptr(struct jit_emit_context jit, void __percpu priv_frame_ptr)\n {\n-\tu8 prog = pprog;\n-\n \t/* movabs r9, priv_frame_ptr /\n-\temit_mov_imm64(\u0026prog, X86_REG_R9, (__force long) priv_frame_ptr \u003e\u003e 32,\n+\temit_mov_imm64(jit, X86_REG_R9, (__force long) priv_frame_ptr \u003e\u003e 32,\n \t\t (u32) (__force long) priv_frame_ptr);\n \n #ifdef CONFIG_SMP\n@@ -1624,11 +1540,9 @@ static void emit_priv_frame_ptr(u8 pprog, void __percpu priv_frame_ptr)\n \tEMIT3(0x03, 0x0c, 0x25);\n \tEMIT((u32)(unsigned long)\u0026this_cpu_off, 4);\n #endif\n-\n-\tpprog = prog;\n }\n \n-#define INSN_SZ_DIFF (((addrs[i] - addrs[i - 1]) - (prog - temp)))\n+#define INSN_SZ_DIFF (((addrs[i] - addrs[i - 1]) - (jit-\u003eprog - temp)))\n \n #define __LOAD_TCC_PTR(off)\t\t\t\\\n \tEMIT3_off32(0x48, 0x8B, 0x85, off)\n@@ -1640,10 +1554,9 @@ static void emit_priv_frame_ptr(u8 pprog, void __percpu priv_frame_ptr)\n #define PRIV_STACK_GUARD_SZ 8\n #define PRIV_STACK_GUARD_VAL 0xEB9F12345678eb9fULL\n \n-static int emit_spectre_bhb_barrier(u8 *pprog, u8 ip,\n+static int emit_spectre_bhb_barrier(struct jit_emit_context jit, u8 ip,\n \t\t\t\t struct bpf_prog bpf_prog)\n {\n-\tu8 prog = pprog;\n \tu8 func;\n \n \tif (cpu_feature_enabled(X86_FEATURE_CLEAR_BHB_LOOP)) {\n@@ -1653,9 +1566,9 @@ static int emit_spectre_bhb_barrier(u8 *pprog, u8 ip,\n \t\tip += 2;\n \n \t\tfunc = (u8 )clear_bhb_loop;\n-\t\tip += x86_call_depth_emit_accounting(\u0026prog, func, ip);\n+\t\tip += bpf_call_depth_emit_accounting(jit, func, ip);\n \n-\t\tif (emit_call(\u0026prog, func, ip))\n+\t\tif (emit_call(jit, func, ip))\n \t\t\treturn -EINVAL;\n \t\tEMIT1(0x59); / pop rcx /\n \t\tEMIT1(0x58); / pop rax /\n@@ -1678,7 +1591,6 @@ static int emit_spectre_bhb_barrier(u8 pprog, u8 ip,\n \t\t /\n \t\tEMIT5(0xF3, 0x48, 0x0F, 0x1E, 0xF8); / ibhf /\n \t}\n-\tpprog = prog;\n \treturn 0;\n }\n \n@@ -1689,10 +1601,9 @@ static int emit_spectre_bhb_barrier(u8 *pprog, u8 ip,\n * arena NULL is offset 0. Return the number of emitted bytes.\n /\n static int emit_kfunc_arena_args(struct bpf_prog bpf_prog,\n-\t\t\t\t const struct btf_func_model fm, u8 pprog)\n+\t\t\t\t const struct btf_func_model fm, struct jit_emit_context jit)\n {\n-\tu8 prog = pprog;\n-\tu8 start = prog;\n+\tu8 start = jit-\u003eprog;\n \tint i;\n \n \tfor (i = 0; i \u003c min_t(int, fm-\u003enr_args, MAX_BPF_FUNC_REG_ARGS); i++) {\n@@ -1705,20 +1616,19 @@ static int emit_kfunc_arena_args(struct bpf_prog bpf_prog,\n \t\t\treturn -EINVAL;\n \n \t\t/* mov eN, eN: truncate and clear the upper 32 bits /\n-\t\temit_mov_reg(\u0026prog, false, reg, reg);\n+\t\temit_mov_reg(jit, false, reg, reg);\n \t\tif (flags \u0026 BTF_FMODEL_NULLABLE_ARG) {\n \t\t\t/ test eN, eN; jz over the 3-byte add /\n-\t\t\tmaybe_emit_mod(\u0026prog, reg, reg, false);\n+\t\t\tmaybe_emit_mod(jit, reg, reg, false);\n \t\t\tEMIT2(0x85, add_2reg(0xC0, reg, reg));\n \t\t\tEMIT2(X86_JE, 3);\n \t\t}\n \t\t/ add rN, r12 /\n-\t\tmaybe_emit_mod(\u0026prog, reg, X86_REG_R12, true);\n+\t\tmaybe_emit_mod(jit, reg, X86_REG_R12, true);\n \t\tEMIT2(0x01, add_2reg(0xC0, reg, X86_REG_R12));\n \t}\n \n-\tpprog = prog;\n-\treturn prog - start;\n+\treturn jit-\u003eprog - start;\n }\n \n static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int addrs, u8 image,\n@@ -1726,6 +1636,7 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int \n {\n \tbool tail_call_reachable = bpf_prog-\u003eaux-\u003etail_call_reachable;\n \tstruct bpf_insn insn = bpf_prog-\u003einsnsi;\n+\tstruct jit_emit_context jit = \u0026ctx-\u003ejit;\n \tbool callee_regs_used[4] = {};\n \tint insn_cnt = bpf_prog-\u003elen;\n \tbool seen_exit = false;\n@@ -1736,12 +1647,15 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int \n \tvoid __percpu priv_stack_ptr;\n \tint i, excnt = 0;\n \tint ilen, proglen = 0;\n-\tu8 ip, prog = temp;\n+\tu8 ip;\n \tu32 stack_depth;\n \tint callee_saved_size;\n \ts32 outgoing_arg_base;\n \tint err;\n \n+\tjit-\u003eprog = temp;\n+\tjit-\u003edry_run = false;\n+\n \tstack_depth = bpf_prog-\u003eaux-\u003estack_depth;\n \tout_stack_arg_cnt = bpf_out_stack_arg_cnt(env, bpf_prog);\n \tpriv_stack_ptr = bpf_prog-\u003eaux-\u003epriv_stack_ptr;\n@@ -1777,11 +1691,11 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int \n \n \tdetect_reg_usage(insn, insn_cnt, callee_regs_used);\n \n-\temit_prologue(\u0026prog, image, stack_depth,\n+\temit_prologue(jit, image, stack_depth,\n \t\t bpf_prog_was_classic(bpf_prog), tail_call_reachable,\n \t\t bpf_is_subprog(bpf_prog), bpf_prog-\u003eaux-\u003eexception_cb);\n \n-\tbpf_prog-\u003eaux-\u003eksym.fp_start = prog - temp;\n+\tbpf_prog-\u003eaux-\u003eksym.fp_start = jit-\u003eprog - temp;\n \n \t/ Exception callback will clobber callee regs for its own use, and\n \t * restore the original callee regs from main prog's stack frame.\n@@ -1791,12 +1705,12 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int \n \t\t register, as we throw after entry into the kernel, which may\n \t\t * overwrite r12.\n \t\t /\n-\t\tpush_r12(\u0026prog);\n-\t\tpush_callee_regs(\u0026prog, all_callee_regs_used);\n+\t\tpush_r12(jit);\n+\t\tpush_callee_regs(jit, all_callee_regs_used);\n \t} else {\n \t\tif (arena_vm_start)\n-\t\t\tpush_r12(\u0026prog);\n-\t\tpush_callee_regs(\u0026prog, callee_regs_used);\n+\t\t\tpush_r12(jit);\n+\t\tpush_callee_regs(jit, callee_regs_used);\n \t}\n \n \t/ Compute callee-saved register area size. /\n@@ -1834,21 +1748,21 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int \n \toutgoing_rsp = out_stack_arg_cnt \u003e 1 ? (out_stack_arg_cnt - 1) * 8 : 0;\n \tif (bpf_prog-\u003eaux-\u003eexception_boundary)\n \t\tbpf_prog-\u003eaux-\u003estack_arg_sp_adjust = outgoing_rsp;\n-\temit_sub_rsp(\u0026prog, outgoing_rsp);\n+\temit_sub_rsp(jit, outgoing_rsp);\n \n \tif (arena_vm_start)\n-\t\temit_mov_imm64(\u0026prog, X86_REG_R12,\n+\t\temit_mov_imm64(jit, X86_REG_R12,\n \t\t\t arena_vm_start \u003e\u003e 32, (u32) arena_vm_start);\n \n \tif (priv_frame_ptr)\n-\t\temit_priv_frame_ptr(\u0026prog, priv_frame_ptr);\n+\t\temit_priv_frame_ptr(jit, priv_frame_ptr);\n \n-\tilen = prog - temp;\n+\tilen = jit-\u003eprog - temp;\n \tif (rw_image)\n \t\tmemcpy(rw_image + proglen, temp, ilen);\n \tproglen += ilen;\n \taddrs[0] = proglen;\n-\tprog = temp;\n+\tjit-\u003eprog = temp;\n \n \tfor (i = 1; i \u003c= insn_cnt; i++, insn++) {\n \t\tconst s32 imm32 = insn-\u003eimm;\n@@ -1873,7 +1787,7 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int \n \t\tif (bpf_insn_is_indirect_target(env, bpf_prog, i - 1))\n \t\t\tEMIT_ENDBR();\n \n-\t\tip = image + addrs[i - 1] + (prog - temp);\n+\t\tip = image + addrs[i - 1] + (jit-\u003eprog - temp);\n \n \t\tswitch (insn-\u003ecode) {\n \t\t\t/ ALU /\n@@ -1887,7 +1801,7 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int \n \t\tcase BPF_ALU64 \| BPF_AND \| BPF_X:\n \t\tcase BPF_ALU64 \| BPF_OR \| BPF_X:\n \t\tcase BPF_ALU64 \| BPF_XOR \| BPF_X:\n-\t\t\tmaybe_emit_mod(\u0026prog, dst_reg, src_reg,\n+\t\t\tmaybe_emit_mod(jit, dst_reg, src_reg,\n \t\t\t\t BPF_CLASS(insn-\u003ecode) == BPF_ALU64);\n \t\t\tb2 = simple_alu_opcodes[BPF_OP(insn-\u003ecode)];\n \t\t\tEMIT2(b2, add_2reg(0xC0, dst_reg, src_reg));\n@@ -1897,32 +1811,32 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int \n \t\t\tif (insn_is_cast_user(insn)) {\n \t\t\t\tif (dst_reg != src_reg)\n \t\t\t\t\t/ 32-bit mov /\n-\t\t\t\t\temit_mov_reg(\u0026prog, false, dst_reg, src_reg);\n+\t\t\t\t\temit_mov_reg(jit, false, dst_reg, src_reg);\n \t\t\t\t/ shl dst_reg, 32 /\n-\t\t\t\tmaybe_emit_1mod(\u0026prog, dst_reg, true);\n+\t\t\t\tmaybe_emit_1mod(jit, dst_reg, true);\n \t\t\t\tEMIT3(0xC1, add_1reg(0xE0, dst_reg), 32);\n \n \t\t\t\t/ or dst_reg, user_vm_start /\n-\t\t\t\tmaybe_emit_1mod(\u0026prog, dst_reg, true);\n+\t\t\t\tmaybe_emit_1mod(jit, dst_reg, true);\n \t\t\t\tif (is_axreg(dst_reg))\n \t\t\t\t\tEMIT1_off32(0x0D, user_vm_start \u003e\u003e 32);\n \t\t\t\telse\n \t\t\t\t\tEMIT2_off32(0x81, add_1reg(0xC8, dst_reg), user_vm_start \u003e\u003e 32);\n \n \t\t\t\t/ rol dst_reg, 32 /\n-\t\t\t\tmaybe_emit_1mod(\u0026prog, dst_reg, true);\n+\t\t\t\tmaybe_emit_1mod(jit, dst_reg, true);\n \t\t\t\tEMIT3(0xC1, add_1reg(0xC0, dst_reg), 32);\n \n \t\t\t\t/ xor r11, r11 /\n \t\t\t\tEMIT3(0x4D, 0x31, 0xDB);\n \n \t\t\t\t/ test dst_reg32, dst_reg32; check if lower 32-bit are zero /\n-\t\t\t\tmaybe_emit_mod(\u0026prog, dst_reg, dst_reg, false);\n+\t\t\t\tmaybe_emit_mod(jit, dst_reg, dst_reg, false);\n \t\t\t\tEMIT2(0x85, add_2reg(0xC0, dst_reg, dst_reg));\n \n \t\t\t\t/ cmove r11, dst_reg; if so, set dst_reg to zero /\n \t\t\t\t/ WARNING: Intel swapped src/dst register encoding in CMOVcc !!! /\n-\t\t\t\tmaybe_emit_mod(\u0026prog, AUX_REG, dst_reg, true);\n+\t\t\t\tmaybe_emit_mod(jit, AUX_REG, dst_reg, true);\n \t\t\t\tEMIT3(0x0F, 0x44, add_2reg(0xC0, AUX_REG, dst_reg));\n \t\t\t\tbreak;\n \t\t\t} else if (insn_is_mov_percpu_addr(insn)) {\n@@ -1939,11 +1853,11 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int \n \t\t\tfallthrough;\n \t\tcase BPF_ALU \| BPF_MOV \| BPF_X:\n \t\t\tif (insn-\u003eoff == 0)\n-\t\t\t\temit_mov_reg(\u0026prog,\n+\t\t\t\temit_mov_reg(jit,\n \t\t\t\t\t BPF_CLASS(insn-\u003ecode) == BPF_ALU64,\n \t\t\t\t\t dst_reg, src_reg);\n \t\t\telse\n-\t\t\t\temit_movsx_reg(\u0026prog, insn-\u003eoff,\n+\t\t\t\temit_movsx_reg(jit, insn-\u003eoff,\n \t\t\t\t\t BPF_CLASS(insn-\u003ecode) == BPF_ALU64,\n \t\t\t\t\t dst_reg, src_reg);\n \t\t\tbreak;\n@@ -1951,7 +1865,7 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int \n \t\t\t/ neg dst /\n \t\tcase BPF_ALU \| BPF_NEG:\n \t\tcase BPF_ALU64 \| BPF_NEG:\n-\t\t\tmaybe_emit_1mod(\u0026prog, dst_reg,\n+\t\t\tmaybe_emit_1mod(jit, dst_reg,\n \t\t\t\t\tBPF_CLASS(insn-\u003ecode) == BPF_ALU64);\n \t\t\tEMIT2(0xF7, add_1reg(0xD8, dst_reg));\n \t\t\tbreak;\n@@ -1966,7 +1880,7 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int \n \t\tcase BPF_ALU64 \| BPF_AND \| BPF_K:\n \t\tcase BPF_ALU64 \| BPF_OR \| BPF_K:\n \t\tcase BPF_ALU64 \| BPF_XOR \| BPF_K:\n-\t\t\tmaybe_emit_1mod(\u0026prog, dst_reg,\n+\t\t\tmaybe_emit_1mod(jit, dst_reg,\n \t\t\t\t\tBPF_CLASS(insn-\u003ecode) == BPF_ALU64);\n \n \t\t\t/\n@@ -2006,12 +1920,12 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int \n \n \t\tcase BPF_ALU64 \| BPF_MOV \| BPF_K:\n \t\tcase BPF_ALU \| BPF_MOV \| BPF_K:\n-\t\t\temit_mov_imm32(\u0026prog, BPF_CLASS(insn-\u003ecode) == BPF_ALU64,\n+\t\t\temit_mov_imm32(jit, BPF_CLASS(insn-\u003ecode) == BPF_ALU64,\n \t\t\t\t dst_reg, imm32);\n \t\t\tbreak;\n \n \t\tcase BPF_LD \| BPF_IMM \| BPF_DW:\n-\t\t\temit_mov_imm64(\u0026prog, dst_reg, insn[1].imm, insn[0].imm);\n+\t\t\temit_mov_imm64(jit, dst_reg, insn[1].imm, insn[0].imm);\n \t\t\tinsn++;\n \t\t\ti++;\n \t\t\tbreak;\n@@ -2047,7 +1961,7 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int \n \n \t\t\tif (dst_reg != BPF_REG_0)\n \t\t\t\t/ mov rax, dst_reg /\n-\t\t\t\temit_mov_reg(\u0026prog, is64, BPF_REG_0, dst_reg);\n+\t\t\t\temit_mov_reg(jit, is64, BPF_REG_0, dst_reg);\n \n \t\t\tif (insn-\u003eoff == 0) {\n \t\t\t\t/\n@@ -2057,7 +1971,7 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int \n \t\t\t\tEMIT2(0x31, 0xd2);\n \n \t\t\t\t/ div src_reg /\n-\t\t\t\tmaybe_emit_1mod(\u0026prog, src_reg, is64);\n+\t\t\t\tmaybe_emit_1mod(jit, src_reg, is64);\n \t\t\t\tEMIT2(0xF7, add_1reg(0xF0, src_reg));\n \t\t\t} else {\n \t\t\t\tif (BPF_CLASS(insn-\u003ecode) == BPF_ALU)\n@@ -2066,18 +1980,18 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int \n \t\t\t\t\tEMIT2(0x48, 0x99); /* cqo /\n \n \t\t\t\t/ idiv src_reg /\n-\t\t\t\tmaybe_emit_1mod(\u0026prog, src_reg, is64);\n+\t\t\t\tmaybe_emit_1mod(jit, src_reg, is64);\n \t\t\t\tEMIT2(0xF7, add_1reg(0xF8, src_reg));\n \t\t\t}\n \n \t\t\tif (BPF_OP(insn-\u003ecode) == BPF_MOD \u0026\u0026\n \t\t\t dst_reg != BPF_REG_3)\n \t\t\t\t/ mov dst_reg, rdx /\n-\t\t\t\temit_mov_reg(\u0026prog, is64, dst_reg, BPF_REG_3);\n+\t\t\t\temit_mov_reg(jit, is64, dst_reg, BPF_REG_3);\n \t\t\telse if (BPF_OP(insn-\u003ecode) == BPF_DIV \u0026\u0026\n \t\t\t\t dst_reg != BPF_REG_0)\n \t\t\t\t/ mov dst_reg, rax /\n-\t\t\t\temit_mov_reg(\u0026prog, is64, dst_reg, BPF_REG_0);\n+\t\t\t\temit_mov_reg(jit, is64, dst_reg, BPF_REG_0);\n \n \t\t\tif (dst_reg != BPF_REG_3)\n \t\t\t\tEMIT1(0x5A); / pop rdx /\n@@ -2088,7 +2002,7 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int \n \n \t\tcase BPF_ALU \| BPF_MUL \| BPF_K:\n \t\tcase BPF_ALU64 \| BPF_MUL \| BPF_K:\n-\t\t\tmaybe_emit_mod(\u0026prog, dst_reg, dst_reg,\n+\t\t\tmaybe_emit_mod(jit, dst_reg, dst_reg,\n \t\t\t\t BPF_CLASS(insn-\u003ecode) == BPF_ALU64);\n \n \t\t\tif (is_imm8(imm32))\n@@ -2104,7 +2018,7 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int \n \n \t\tcase BPF_ALU \| BPF_MUL \| BPF_X:\n \t\tcase BPF_ALU64 \| BPF_MUL \| BPF_X:\n-\t\t\tmaybe_emit_mod(\u0026prog, src_reg, dst_reg,\n+\t\t\tmaybe_emit_mod(jit, src_reg, dst_reg,\n \t\t\t\t BPF_CLASS(insn-\u003ecode) == BPF_ALU64);\n \n \t\t\t/ imul dst_reg, src_reg /\n@@ -2118,7 +2032,7 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int \n \t\tcase BPF_ALU64 \| BPF_LSH \| BPF_K:\n \t\tcase BPF_ALU64 \| BPF_RSH \| BPF_K:\n \t\tcase BPF_ALU64 \| BPF_ARSH \| BPF_K:\n-\t\t\tmaybe_emit_1mod(\u0026prog, dst_reg,\n+\t\t\tmaybe_emit_1mod(jit, dst_reg,\n \t\t\t\t\tBPF_CLASS(insn-\u003ecode) == BPF_ALU64);\n \n \t\t\tb3 = simple_alu_opcodes[BPF_OP(insn-\u003ecode)];\n@@ -2152,7 +2066,7 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int \n \t\t\t\t\tbreak;\n \t\t\t\t}\n \n-\t\t\t\temit_shiftx(\u0026prog, dst_reg, src_reg, w, op);\n+\t\t\t\temit_shiftx(jit, dst_reg, src_reg, w, op);\n \n \t\t\t\tbreak;\n \t\t\t}\n@@ -2171,7 +2085,7 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int \n \t\t\t}\n \n \t\t\t/* shl %rax, %cl \| shr %rax, %cl \| sar %rax, %cl /\n-\t\t\tmaybe_emit_1mod(\u0026prog, dst_reg,\n+\t\t\tmaybe_emit_1mod(jit, dst_reg,\n \t\t\t\t\tBPF_CLASS(insn-\u003ecode) == BPF_ALU64);\n \n \t\t\tb3 = simple_alu_opcodes[BPF_OP(insn-\u003ecode)];\n@@ -2272,7 +2186,7 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int \n \t\tcase BPF_ST \| BPF_MEM \| BPF_DW:\n \t\t\tif (dst_reg == BPF_REG_PARAMS \u0026\u0026 insn-\u003eoff == -8) {\n \t\t\t\t/* Arg 6: store immediate in r9 register /\n-\t\t\t\temit_mov_imm64(\u0026prog, X86_REG_R9, imm32 \u003e\u003e 31, (u32)imm32);\n+\t\t\t\temit_mov_imm64(jit, X86_REG_R9, imm32 \u003e\u003e 31, (u32)imm32);\n \t\t\t\tbreak;\n \t\t\t}\n \t\t\tEMIT2(add_1mod(0x48, dst_reg), 0xC7);\n@@ -2310,15 +2224,15 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\t\t\tinsn_off = outgoing_arg_base - outgoing_rsp - insn_off - 16;\n \t\t\t\tdst_reg = BPF_REG_FP;\n \t\t\t}\n-\t\t\temit_stx(\u0026prog, BPF_SIZE(insn-\u003ecode), dst_reg, src_reg, insn_off);\n+\t\t\temit_stx(jit, BPF_SIZE(insn-\u003ecode), dst_reg, src_reg, insn_off);\n \t\t\tbreak;\n \n \t\tcase BPF_ST \| BPF_PROBE_MEM32 \| BPF_B:\n \t\tcase BPF_ST \| BPF_PROBE_MEM32 \| BPF_H:\n \t\tcase BPF_ST \| BPF_PROBE_MEM32 \| BPF_W:\n \t\tcase BPF_ST \| BPF_PROBE_MEM32 \| BPF_DW:\n-\t\t\tstart_of_ldx = prog;\n-\t\t\temit_st_r12(\u0026prog, BPF_SIZE(insn-\u003ecode), dst_reg, insn-\u003eoff, insn-\u003eimm);\n+\t\t\tstart_of_ldx = jit-\u003eprog;\n+\t\t\temit_st_r12(jit, BPF_SIZE(insn-\u003ecode), dst_reg, insn-\u003eoff, insn-\u003eimm);\n \t\t\tgoto populate_extable;\n \n \t\t\t/ LDX: dst_reg = (u8)(src_reg + r12 + off) /\n@@ -2333,14 +2247,14 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\tcase BPF_STX \| BPF_PROBE_MEM32 \| BPF_H:\n \t\tcase BPF_STX \| BPF_PROBE_MEM32 \| BPF_W:\n \t\tcase BPF_STX \| BPF_PROBE_MEM32 \| BPF_DW:\n-\t\t\tstart_of_ldx = prog;\n+\t\t\tstart_of_ldx = jit-\u003eprog;\n \t\t\tif (BPF_CLASS(insn-\u003ecode) == BPF_LDX) {\n \t\t\t\tif (BPF_MODE(insn-\u003ecode) == BPF_PROBE_MEM32SX)\n-\t\t\t\t\temit_ldsx_r12(\u0026prog, BPF_SIZE(insn-\u003ecode), dst_reg, src_reg, insn-\u003eoff);\n+\t\t\t\t\temit_ldsx_r12(jit, BPF_SIZE(insn-\u003ecode), dst_reg, src_reg, insn-\u003eoff);\n \t\t\t\telse\n-\t\t\t\t\temit_ldx_r12(\u0026prog, BPF_SIZE(insn-\u003ecode), dst_reg, src_reg, insn-\u003eoff);\n+\t\t\t\t\temit_ldx_r12(jit, BPF_SIZE(insn-\u003ecode), dst_reg, src_reg, insn-\u003eoff);\n \t\t\t} else {\n-\t\t\t\temit_stx_r12(\u0026prog, BPF_SIZE(insn-\u003ecode), dst_reg, src_reg, insn-\u003eoff);\n+\t\t\t\temit_stx_r12(jit, BPF_SIZE(insn-\u003ecode), dst_reg, src_reg, insn-\u003eoff);\n \t\t\t}\n populate_extable:\n \t\t\t{\n@@ -2401,7 +2315,7 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\t\t\t\tis_write = true;\n \t\t\t\t}\n \n-\t\t\t\tex-\u003efixup = FIELD_PREP(FIXUP_INSN_LEN_MASK, prog - start_of_ldx) \|\n+\t\t\t\tex-\u003efixup = FIELD_PREP(FIXUP_INSN_LEN_MASK, jit-\u003eprog - start_of_ldx) \|\n \t\t\t\t\t FIELD_PREP(FIXUP_ARENA_REG_MASK, arena_reg) \|\n \t\t\t\t\t FIELD_PREP(FIXUP_REG_MASK, fixup_reg);\n \t\t\t\tex-\u003efixup \|= FIXUP_ARENA_ACCESS;\n@@ -2455,7 +2369,7 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\t\t\tu8 end_of_jmp;\n \n \t\t\t\t/* movabsq r10, VSYSCALL_ADDR /\n-\t\t\t\temit_mov_imm64(\u0026prog, BPF_REG_AX, (long)VSYSCALL_ADDR \u003e\u003e 32,\n+\t\t\t\temit_mov_imm64(jit, BPF_REG_AX, (long)VSYSCALL_ADDR \u003e\u003e 32,\n \t\t\t\t\t (u32)(long)VSYSCALL_ADDR);\n \n \t\t\t\t/ mov src_reg, r11 /\n@@ -2463,40 +2377,40 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \n \t\t\t\tif (insn-\u003eoff) {\n \t\t\t\t\t/ add r11, insn-\u003eoff /\n-\t\t\t\t\tmaybe_emit_1mod(\u0026prog, AUX_REG, true);\n+\t\t\t\t\tmaybe_emit_1mod(jit, AUX_REG, true);\n \t\t\t\t\tEMIT2_off32(0x81, add_1reg(0xC0, AUX_REG), insn-\u003eoff);\n \t\t\t\t}\n \n \t\t\t\t/ sub r11, r10 /\n-\t\t\t\tmaybe_emit_mod(\u0026prog, AUX_REG, BPF_REG_AX, true);\n+\t\t\t\tmaybe_emit_mod(jit, AUX_REG, BPF_REG_AX, true);\n \t\t\t\tEMIT2(0x29, add_2reg(0xC0, AUX_REG, BPF_REG_AX));\n \n \t\t\t\t/ movabsq r10, limit /\n-\t\t\t\temit_mov_imm64(\u0026prog, BPF_REG_AX, (long)limit \u003e\u003e 32,\n+\t\t\t\temit_mov_imm64(jit, BPF_REG_AX, (long)limit \u003e\u003e 32,\n \t\t\t\t\t (u32)(long)limit);\n \n \t\t\t\t/ cmp r10, r11 /\n-\t\t\t\tmaybe_emit_mod(\u0026prog, AUX_REG, BPF_REG_AX, true);\n+\t\t\t\tmaybe_emit_mod(jit, AUX_REG, BPF_REG_AX, true);\n \t\t\t\tEMIT2(0x39, add_2reg(0xC0, AUX_REG, BPF_REG_AX));\n \n \t\t\t\t/ if unsigned '\u003e', goto load /\n \t\t\t\tEMIT2(X86_JA, 0);\n-\t\t\t\tend_of_jmp = prog;\n+\t\t\t\tend_of_jmp = jit-\u003eprog;\n \n \t\t\t\t/ xor dst_reg, dst_reg /\n-\t\t\t\temit_mov_imm32(\u0026prog, false, dst_reg, 0);\n+\t\t\t\temit_mov_imm32(jit, false, dst_reg, 0);\n \t\t\t\t/ jmp byte_after_ldx /\n \t\t\t\tEMIT2(0xEB, 0);\n \n \t\t\t\t/ populate jmp_offset for JAE above to jump to start_of_ldx /\n-\t\t\t\tstart_of_ldx = prog;\n+\t\t\t\tstart_of_ldx = jit-\u003eprog;\n \t\t\t\tend_of_jmp[-1] = start_of_ldx - end_of_jmp;\n \t\t\t}\n \t\t\tif (BPF_MODE(insn-\u003ecode) == BPF_PROBE_MEMSX \|\|\n \t\t\t BPF_MODE(insn-\u003ecode) == BPF_MEMSX)\n-\t\t\t\temit_ldsx(\u0026prog, BPF_SIZE(insn-\u003ecode), dst_reg, src_reg, insn_off);\n+\t\t\t\temit_ldsx(jit, BPF_SIZE(insn-\u003ecode), dst_reg, src_reg, insn_off);\n \t\t\telse\n-\t\t\t\temit_ldx(\u0026prog, BPF_SIZE(insn-\u003ecode), dst_reg, src_reg, insn_off);\n+\t\t\t\temit_ldx(jit, BPF_SIZE(insn-\u003ecode), dst_reg, src_reg, insn_off);\n \t\t\tif (BPF_MODE(insn-\u003ecode) == BPF_PROBE_MEM \|\|\n \t\t\t BPF_MODE(insn-\u003ecode) == BPF_PROBE_MEMSX) {\n \t\t\t\tstruct exception_table_entry ex;\n@@ -2504,7 +2418,7 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\t\t\ts64 delta;\n \n \t\t\t\t/* populate jmp_offset for JMP above /\n-\t\t\t\tstart_of_ldx[-1] = prog - start_of_ldx;\n+\t\t\t\tstart_of_ldx[-1] = jit-\u003eprog - start_of_ldx;\n \n \t\t\t\tif (!bpf_prog-\u003eaux-\u003eextable)\n \t\t\t\t\tbreak;\n@@ -2539,7 +2453,7 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\t\t\t End result: x86 insn \"mov rbx, qword ptr [rax+0x14]\"\n \t\t\t\t * of 4 bytes will be ignored and rbx will be zero inited.\n \t\t\t\t /\n-\t\t\t\tex-\u003efixup = FIELD_PREP(FIXUP_INSN_LEN_MASK, prog - start_of_ldx) \|\n+\t\t\t\tex-\u003efixup = FIELD_PREP(FIXUP_INSN_LEN_MASK, jit-\u003eprog - start_of_ldx) \|\n \t\t\t\t\t FIELD_PREP(FIXUP_REG_MASK, reg2pt_regs[dst_reg]);\n \t\t\t}\n \t\t\tbreak;\n@@ -2567,26 +2481,26 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\t\t\t /\n \n \t\t\t\t/* Will need RAX as a CMPXCHG operand so save R0 /\n-\t\t\t\temit_mov_reg(\u0026prog, true, BPF_REG_AX, BPF_REG_0);\n+\t\t\t\temit_mov_reg(jit, true, BPF_REG_AX, BPF_REG_0);\n \t\t\t\tif (src_reg == BPF_REG_0)\n \t\t\t\t\treal_src_reg = BPF_REG_AX;\n \t\t\t\tif (dst_reg == BPF_REG_0)\n \t\t\t\t\treal_dst_reg = BPF_REG_AX;\n \n-\t\t\t\tbranch_target = prog;\n+\t\t\t\tbranch_target = jit-\u003eprog;\n \t\t\t\t/ Load old value /\n-\t\t\t\temit_ldx(\u0026prog, BPF_SIZE(insn-\u003ecode),\n+\t\t\t\temit_ldx(jit, BPF_SIZE(insn-\u003ecode),\n \t\t\t\t\t BPF_REG_0, real_dst_reg, insn-\u003eoff);\n \t\t\t\t/\n \t\t\t\t * Perform the (commutative) operation locally,\n \t\t\t\t * put the result in the AUX_REG.\n \t\t\t\t /\n-\t\t\t\temit_mov_reg(\u0026prog, is64, AUX_REG, BPF_REG_0);\n-\t\t\t\tmaybe_emit_mod(\u0026prog, AUX_REG, real_src_reg, is64);\n+\t\t\t\temit_mov_reg(jit, is64, AUX_REG, BPF_REG_0);\n+\t\t\t\tmaybe_emit_mod(jit, AUX_REG, real_src_reg, is64);\n \t\t\t\tEMIT2(simple_alu_opcodes[BPF_OP(insn-\u003eimm)],\n \t\t\t\t add_2reg(0xC0, AUX_REG, real_src_reg));\n \t\t\t\t/ Attempt to swap in new value /\n-\t\t\t\terr = emit_atomic_rmw(\u0026prog, BPF_CMPXCHG,\n+\t\t\t\terr = emit_atomic_rmw(jit, BPF_CMPXCHG,\n \t\t\t\t\t\t real_dst_reg, AUX_REG,\n \t\t\t\t\t\t insn-\u003eoff,\n \t\t\t\t\t\t BPF_SIZE(insn-\u003ecode));\n@@ -2596,19 +2510,19 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\t\t\t ZF tells us whether we won the race. If it's\n \t\t\t\t * cleared we need to try again.\n \t\t\t\t /\n-\t\t\t\tEMIT2(X86_JNE, -(prog - branch_target) - 2);\n+\t\t\t\tEMIT2(X86_JNE, -(jit-\u003eprog - branch_target) - 2);\n \t\t\t\t/ Return the pre-modification value /\n-\t\t\t\temit_mov_reg(\u0026prog, is64, real_src_reg, BPF_REG_0);\n+\t\t\t\temit_mov_reg(jit, is64, real_src_reg, BPF_REG_0);\n \t\t\t\t/ Restore R0 after clobbering RAX /\n-\t\t\t\temit_mov_reg(\u0026prog, true, BPF_REG_0, BPF_REG_AX);\n+\t\t\t\temit_mov_reg(jit, true, BPF_REG_0, BPF_REG_AX);\n \t\t\t\tbreak;\n \t\t\t}\n \n \t\t\tif (bpf_atomic_is_load_store(insn))\n-\t\t\t\terr = emit_atomic_ld_st(\u0026prog, insn-\u003eimm, dst_reg, src_reg,\n+\t\t\t\terr = emit_atomic_ld_st(jit, insn-\u003eimm, dst_reg, src_reg,\n \t\t\t\t\t\t\tinsn-\u003eoff, BPF_SIZE(insn-\u003ecode));\n \t\t\telse\n-\t\t\t\terr = emit_atomic_rmw(\u0026prog, insn-\u003eimm, dst_reg, src_reg,\n+\t\t\t\terr = emit_atomic_rmw(jit, insn-\u003eimm, dst_reg, src_reg,\n \t\t\t\t\t\t insn-\u003eoff, BPF_SIZE(insn-\u003ecode));\n \t\t\tif (err)\n \t\t\t\treturn err;\n@@ -2623,14 +2537,14 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\t\tfallthrough;\n \t\tcase BPF_STX \| BPF_PROBE_ATOMIC \| BPF_W:\n \t\tcase BPF_STX \| BPF_PROBE_ATOMIC \| BPF_DW:\n-\t\t\tstart_of_ldx = prog;\n+\t\t\tstart_of_ldx = jit-\u003eprog;\n \n \t\t\tif (bpf_atomic_is_load_store(insn))\n-\t\t\t\terr = emit_atomic_ld_st_index(\u0026prog, insn-\u003eimm,\n+\t\t\t\terr = emit_atomic_ld_st_index(jit, insn-\u003eimm,\n \t\t\t\t\t\t\t BPF_SIZE(insn-\u003ecode), dst_reg,\n \t\t\t\t\t\t\t src_reg, X86_REG_R12, insn-\u003eoff);\n \t\t\telse\n-\t\t\t\terr = emit_atomic_rmw_index(\u0026prog, insn-\u003eimm, BPF_SIZE(insn-\u003ecode),\n+\t\t\t\terr = emit_atomic_rmw_index(jit, insn-\u003eimm, BPF_SIZE(insn-\u003ecode),\n \t\t\t\t\t\t\t dst_reg, src_reg, X86_REG_R12,\n \t\t\t\t\t\t\t insn-\u003eoff);\n \t\t\tif (err)\n@@ -2652,20 +2566,20 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\t\t\tfm = bpf_jit_find_kfunc_model(bpf_prog, insn);\n \t\t\t\tif (!fm)\n \t\t\t\t\treturn -EINVAL;\n-\t\t\t\terr = emit_kfunc_arena_args(bpf_prog, fm, \u0026prog);\n+\t\t\t\terr = emit_kfunc_arena_args(bpf_prog, fm, jit);\n \t\t\t\tif (err \u003c 0)\n \t\t\t\t\treturn err;\n \t\t\t\tip += err;\n \t\t\t}\n \t\t\tif (priv_frame_ptr) {\n-\t\t\t\tpush_r9(\u0026prog);\n+\t\t\t\tpush_r9(jit);\n \t\t\t\tip += 2;\n \t\t\t}\n-\t\t\tip += x86_call_depth_emit_accounting(\u0026prog, func, ip);\n-\t\t\tif (emit_call(\u0026prog, func, ip))\n+\t\t\tip += bpf_call_depth_emit_accounting(jit, func, ip);\n+\t\t\tif (emit_call(jit, func, ip))\n \t\t\t\treturn -EINVAL;\n \t\t\tif (priv_frame_ptr)\n-\t\t\t\tpop_r9(\u0026prog);\n+\t\t\t\tpop_r9(jit);\n \t\t\t/\n \t\t\t * A kfunc returning more than 8 bytes hands the second\n \t\t\t * half back in RDX (the native ABI's second return reg),\n@@ -2673,7 +2587,7 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\t\t * needed), while BPF R2 is RSI, so copy RDX into RSI.\n \t\t\t /\n \t\t\tif (fm \u0026\u0026 fm-\u003eret_size \u003e 8)\n-\t\t\t\temit_mov_reg(\u0026prog, true, BPF_REG_2, BPF_REG_3);\n+\t\t\t\temit_mov_reg(jit, true, BPF_REG_2, BPF_REG_3);\n \t\t\tbreak;\n \t\t}\n \n@@ -2681,14 +2595,12 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\t\tif (imm32)\n \t\t\t\temit_bpf_tail_call_direct(bpf_prog,\n \t\t\t\t\t\t\t \u0026bpf_prog-\u003eaux-\u003epoke_tab[imm32 - 1],\n-\t\t\t\t\t\t\t \u0026prog,\n \t\t\t\t\t\t\t ip,\n \t\t\t\t\t\t\t callee_regs_used,\n \t\t\t\t\t\t\t stack_depth,\n \t\t\t\t\t\t\t ctx);\n \t\t\telse\n \t\t\t\temit_bpf_tail_call_indirect(bpf_prog,\n-\t\t\t\t\t\t\t \u0026prog,\n \t\t\t\t\t\t\t callee_regs_used,\n \t\t\t\t\t\t\t stack_depth,\n \t\t\t\t\t\t\t ip,\n@@ -2717,7 +2629,7 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\tcase BPF_JMP32 \| BPF_JSGE \| BPF_X:\n \t\tcase BPF_JMP32 \| BPF_JSLE \| BPF_X:\n \t\t\t/ cmp dst_reg, src_reg /\n-\t\t\tmaybe_emit_mod(\u0026prog, dst_reg, src_reg,\n+\t\t\tmaybe_emit_mod(jit, dst_reg, src_reg,\n \t\t\t\t BPF_CLASS(insn-\u003ecode) == BPF_JMP);\n \t\t\tEMIT2(0x39, add_2reg(0xC0, dst_reg, src_reg));\n \t\t\tgoto emit_cond_jmp;\n@@ -2725,7 +2637,7 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\tcase BPF_JMP \| BPF_JSET \| BPF_X:\n \t\tcase BPF_JMP32 \| BPF_JSET \| BPF_X:\n \t\t\t/ test dst_reg, src_reg /\n-\t\t\tmaybe_emit_mod(\u0026prog, dst_reg, src_reg,\n+\t\t\tmaybe_emit_mod(jit, dst_reg, src_reg,\n \t\t\t\t BPF_CLASS(insn-\u003ecode) == BPF_JMP);\n \t\t\tEMIT2(0x85, add_2reg(0xC0, dst_reg, src_reg));\n \t\t\tgoto emit_cond_jmp;\n@@ -2733,7 +2645,7 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\tcase BPF_JMP \| BPF_JSET \| BPF_K:\n \t\tcase BPF_JMP32 \| BPF_JSET \| BPF_K:\n \t\t\t/ test dst_reg, imm32 /\n-\t\t\tmaybe_emit_1mod(\u0026prog, dst_reg,\n+\t\t\tmaybe_emit_1mod(jit, dst_reg,\n \t\t\t\t\tBPF_CLASS(insn-\u003ecode) == BPF_JMP);\n \t\t\tEMIT2_off32(0xF7, add_1reg(0xC0, dst_reg), imm32);\n \t\t\tgoto emit_cond_jmp;\n@@ -2760,14 +2672,14 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\tcase BPF_JMP32 \| BPF_JSLE \| BPF_K:\n \t\t\t/ test dst_reg, dst_reg to save one extra byte /\n \t\t\tif (imm32 == 0) {\n-\t\t\t\tmaybe_emit_mod(\u0026prog, dst_reg, dst_reg,\n+\t\t\t\tmaybe_emit_mod(jit, dst_reg, dst_reg,\n \t\t\t\t\t BPF_CLASS(insn-\u003ecode) == BPF_JMP);\n \t\t\t\tEMIT2(0x85, add_2reg(0xC0, dst_reg, dst_reg));\n \t\t\t\tgoto emit_cond_jmp;\n \t\t\t}\n \n \t\t\t/ cmp dst_reg, imm8/32 /\n-\t\t\tmaybe_emit_1mod(\u0026prog, dst_reg,\n+\t\t\tmaybe_emit_1mod(jit, dst_reg,\n \t\t\t\t\tBPF_CLASS(insn-\u003ecode) == BPF_JMP);\n \n \t\t\tif (is_imm8(imm32))\n@@ -2843,7 +2755,7 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\t\t\t\t\t nops);\n \t\t\t\t\t\treturn -EFAULT;\n \t\t\t\t\t}\n-\t\t\t\t\temit_nops(\u0026prog, nops);\n+\t\t\t\t\temit_nops(jit, nops);\n \t\t\t\t}\n \t\t\t\tEMIT2(jmp_cond, jmp_offset);\n \t\t\t} else if (is_simm32(jmp_offset)) {\n@@ -2856,7 +2768,7 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\t\tbreak;\n \n \t\tcase BPF_JMP \| BPF_JA \| BPF_X:\n-\t\t\temit_indirect_jump(\u0026prog, insn-\u003edst_reg, ip);\n+\t\t\temit_indirect_jump(jit, insn-\u003edst_reg, ip);\n \t\t\tbreak;\n \t\tcase BPF_JMP \| BPF_JA:\n \t\tcase BPF_JMP32 \| BPF_JA:\n@@ -2900,7 +2812,7 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\t\t\t\t\t nops);\n \t\t\t\t\t\treturn -EFAULT;\n \t\t\t\t\t}\n-\t\t\t\t\temit_nops(\u0026prog, nops);\n+\t\t\t\t\temit_nops(jit, nops);\n \t\t\t\t}\n \t\t\t\tbreak;\n \t\t\t}\n@@ -2925,7 +2837,7 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\t\t\t\t\t nops);\n \t\t\t\t\t\treturn -EFAULT;\n \t\t\t\t\t}\n-\t\t\t\t\temit_nops(\u0026prog, INSN_SZ_DIFF - 2);\n+\t\t\t\t\temit_nops(jit, INSN_SZ_DIFF - 2);\n \t\t\t\t}\n \t\t\t\tEMIT2(0xEB, jmp_offset);\n \t\t\t} else if (is_simm32(jmp_offset)) {\n@@ -2946,23 +2858,23 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\t\tctx-\u003ecleanup_addr = proglen;\n \t\t\tif (bpf_prog_was_classic(bpf_prog) \u0026\u0026\n \t\t\t !ns_capable_noaudit(\u0026init_user_ns, CAP_SYS_ADMIN)) {\n-\t\t\t\tif (emit_spectre_bhb_barrier(\u0026prog, ip, bpf_prog))\n+\t\t\t\tif (emit_spectre_bhb_barrier(jit, ip, bpf_prog))\n \t\t\t\t\treturn -EINVAL;\n \t\t\t}\n \t\t\t/ Deallocate outgoing args 7+ area. /\n-\t\t\temit_add_rsp(\u0026prog, outgoing_rsp);\n+\t\t\temit_add_rsp(jit, outgoing_rsp);\n \t\t\tif (bpf_prog-\u003eaux-\u003eexception_boundary) {\n-\t\t\t\tpop_callee_regs(\u0026prog, all_callee_regs_used);\n-\t\t\t\tpop_r12(\u0026prog);\n+\t\t\t\tpop_callee_regs(jit, all_callee_regs_used);\n+\t\t\t\tpop_r12(jit);\n \t\t\t} else {\n-\t\t\t\tpop_callee_regs(\u0026prog, callee_regs_used);\n+\t\t\t\tpop_callee_regs(jit, callee_regs_used);\n \t\t\t\tif (arena_vm_start)\n-\t\t\t\t\tpop_r12(\u0026prog);\n+\t\t\t\t\tpop_r12(jit);\n \t\t\t}\n \t\t\tEMIT1(0xC9); / leave /\n-\t\t\tbpf_prog-\u003eaux-\u003eksym.fp_end = prog - temp;\n+\t\t\tbpf_prog-\u003eaux-\u003eksym.fp_end = jit-\u003eprog - temp;\n \n-\t\t\temit_return(\u0026prog, image + addrs[i - 1] + (prog - temp));\n+\t\t\temit_return(jit, image + addrs[i - 1] + (jit-\u003eprog - temp));\n \t\t\tbreak;\n \n \t\tdefault:\n@@ -2976,7 +2888,7 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\t\treturn -EINVAL;\n \t\t}\n \n-\t\tilen = prog - temp;\n+\t\tilen = jit-\u003eprog - temp;\n \t\tif (ilen \u003e BPF_MAX_INSN_SIZE) {\n \t\t\tpr_err(\"bpf_jit: fatal insn size error\\n\");\n \t\t\treturn -EFAULT;\n@@ -3000,7 +2912,7 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\t}\n \t\tproglen += ilen;\n \t\taddrs[i] = proglen;\n-\t\tprog = temp;\n+\t\tjit-\u003eprog = temp;\n \t}\n \n \tif (image \u0026\u0026 excnt != bpf_prog-\u003eaux-\u003enum_exentries) {\n@@ -3011,11 +2923,10 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n }\n \n static void clean_stack_garbage(const struct btf_func_model m,\n-\t\t\t\tu8 *pprog, int nr_stack_slots,\n+\t\t\t\tstruct jit_emit_context jit, int nr_stack_slots,\n \t\t\t\tint stack_size)\n {\n \tint arg_size, off;\n-\tu8 prog;\n \n \t/ Generally speaking, the compiler will pass the arguments\n \t * on-stack with \"push\" instruction, which will take 8-byte\n@@ -3047,14 +2958,12 @@ static void clean_stack_garbage(const struct btf_func_model m,\n \targ_size = m-\u003earg_size[m-\u003enr_args - 1];\n \tif (arg_size \u003c= 4) {\n \t\toff = -(stack_size - 4);\n-\t\tprog = pprog;\n \t\t/* mov DWORD PTR [rbp + off], 0 /\n \t\tif (!is_imm8(off))\n \t\t\tEMIT2_off32(0xC7, 0x85, off);\n \t\telse\n \t\t\tEMIT3(0xC7, 0x45, off);\n \t\tEMIT(0, 4);\n-\t\tpprog = prog;\n \t}\n }\n \n@@ -3082,26 +2991,23 @@ static int get_nr_used_regs(const struct btf_func_model m)\n subtraction both truncates and clears the upper half, so the stored\n * value satisfies the JIT invariant for arena pointer registers.\n /\n-static void emit_arena_arg_conv(u8 pprog, u32 src_reg, bool nullable, u32 base_lo)\n+static void emit_arena_arg_conv(struct jit_emit_context jit, u32 src_reg, bool nullable,\n+\t\t\t\tu32 base_lo)\n {\n-\tu8 prog = pprog;\n-\n \tif (nullable) {\n \t\tif (src_reg != BPF_REG_0)\n-\t\t\temit_mov_reg(\u0026prog, true, BPF_REG_0, src_reg);\n+\t\t\temit_mov_reg(jit, true, BPF_REG_0, src_reg);\n \t\t/* test rax, rax; jz over the 5-byte sub /\n \t\tEMIT3(0x48, 0x85, 0xC0);\n \t\tEMIT2(X86_JE, 5);\n \t} else if (src_reg != BPF_REG_0) {\n-\t\temit_mov_reg(\u0026prog, false, BPF_REG_0, src_reg);\n+\t\temit_mov_reg(jit, false, BPF_REG_0, src_reg);\n \t}\n \t/ sub eax, base_lo /\n \tEMIT1_off32(0x2D, base_lo);\n-\n-\tpprog = prog;\n }\n \n-static void save_args(const struct btf_func_model m, u8 prog,\n+static void save_args(const struct btf_func_model m, struct jit_emit_context jit,\n \t\t int stack_size, bool for_call_origin, u32 flags,\n \t\t u64 arena_base)\n {\n@@ -3150,12 +3056,12 @@ static void save_args(const struct btf_func_model m, u8 *prog,\n \t\t\t called indirectly, so rbp + 16.\n \t\t\t /\n \t\t\tfor (j = 0; j \u003c arg_regs; j++) {\n-\t\t\t\temit_ldx(prog, BPF_DW, BPF_REG_0, BPF_REG_FP,\n+\t\t\t\temit_ldx(jit, BPF_DW, BPF_REG_0, BPF_REG_FP,\n \t\t\t\t\t nr_stack_slots 8 + stack_args_off);\n \t\t\t\tif (arena_arg)\n-\t\t\t\t\temit_arena_arg_conv(prog, BPF_REG_0, nullable,\n+\t\t\t\t\temit_arena_arg_conv(jit, BPF_REG_0, nullable,\n \t\t\t\t\t\t\t (u32)arena_base);\n-\t\t\t\temit_stx(prog, BPF_DW, BPF_REG_FP, BPF_REG_0,\n+\t\t\t\temit_stx(jit, BPF_DW, BPF_REG_FP, BPF_REG_0,\n \t\t\t\t\t -stack_size);\n \n \t\t\t\tif (!nr_stack_slots)\n@@ -3178,20 +3084,20 @@ static void save_args(const struct btf_func_model m, u8 prog,\n \t\t\t\tu32 src = nr_regs == 5 ? X86_REG_R9 : BPF_REG_1 + nr_regs;\n \n \t\t\t\tif (arena_arg) {\n-\t\t\t\t\temit_arena_arg_conv(prog, src, nullable, (u32)arena_base);\n+\t\t\t\t\temit_arena_arg_conv(jit, src, nullable, (u32)arena_base);\n \t\t\t\t\tsrc = BPF_REG_0;\n \t\t\t\t}\n-\t\t\t\temit_stx(prog, BPF_DW, BPF_REG_FP, src, -stack_size);\n+\t\t\t\temit_stx(jit, BPF_DW, BPF_REG_FP, src, -stack_size);\n \t\t\t\tstack_size -= 8;\n \t\t\t\tnr_regs++;\n \t\t\t}\n \t\t}\n \t}\n \n-\tclean_stack_garbage(m, prog, nr_stack_slots, first_off);\n+\tclean_stack_garbage(m, jit, nr_stack_slots, first_off);\n }\n \n-static void restore_regs(const struct btf_func_model m, u8 *prog,\n+static void restore_regs(const struct btf_func_model m, struct jit_emit_context jit,\n \t\t\t int stack_size)\n {\n \tint i, j, arg_regs, nr_regs = 0;\n@@ -3207,7 +3113,7 @@ static void restore_regs(const struct btf_func_model m, u8 *prog,\n \t\targ_regs = (m-\u003earg_size[i] + 7) / 8;\n \t\tif (nr_regs + arg_regs \u003c= 6) {\n \t\t\tfor (j = 0; j \u003c arg_regs; j++) {\n-\t\t\t\temit_ldx(prog, BPF_DW,\n+\t\t\t\temit_ldx(jit, BPF_DW,\n \t\t\t\t\t nr_regs == 5 ? X86_REG_R9 : BPF_REG_1 + nr_regs,\n \t\t\t\t\t BPF_REG_FP,\n \t\t\t\t\t -stack_size);\n@@ -3223,19 +3129,18 @@ static void restore_regs(const struct btf_func_model m, u8 *prog,\n \t}\n }\n \n-static int invoke_bpf_prog(const struct btf_func_model m, u8 *pprog,\n+static int invoke_bpf_prog(const struct btf_func_model m, struct jit_emit_context jit,\n \t\t\t struct bpf_tramp_node node, int stack_size,\n \t\t\t int run_ctx_off, bool save_ret,\n \t\t\t void image, void rw_image)\n {\n-\tu8 prog = pprog;\n \tu8 jmp_insn;\n \tint ctx_cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie);\n \tstruct bpf_prog p = node-\u003elink-\u003eprog;\n \tu64 cookie = node-\u003ecookie;\n \n \t/* mov rdi, cookie /\n-\temit_mov_imm64(\u0026prog, BPF_REG_1, (long) cookie \u003e\u003e 32, (u32) (long) cookie);\n+\temit_mov_imm64(jit, BPF_REG_1, (long) cookie \u003e\u003e 32, (u32) (long) cookie);\n \n \t/ Prepare struct bpf_tramp_run_ctx.\n \t \n@@ -3244,28 +3149,28 @@ static int invoke_bpf_prog(const struct btf_func_model m, u8 *pprog,\n \t \n \t * mov QWORD PTR [rbp - run_ctx_off + ctx_cookie_off], rdi\n \t /\n-\temit_stx(\u0026prog, BPF_DW, BPF_REG_FP, BPF_REG_1, -run_ctx_off + ctx_cookie_off);\n+\temit_stx(jit, BPF_DW, BPF_REG_FP, BPF_REG_1, -run_ctx_off + ctx_cookie_off);\n \n \t/ arg1: mov rdi, progs[i] /\n-\temit_mov_imm64(\u0026prog, BPF_REG_1, (long) p \u003e\u003e 32, (u32) (long) p);\n+\temit_mov_imm64(jit, BPF_REG_1, (long) p \u003e\u003e 32, (u32) (long) p);\n \t/ arg2: lea rsi, [rbp - ctx_cookie_off] /\n \tif (!is_imm8(-run_ctx_off))\n \t\tEMIT3_off32(0x48, 0x8D, 0xB5, -run_ctx_off);\n \telse\n \t\tEMIT4(0x48, 0x8D, 0x75, -run_ctx_off);\n \n-\tif (emit_rsb_call(\u0026prog, bpf_trampoline_enter(p), image + (prog - (u8 )rw_image)))\n+\tif (emit_rsb_call(jit, bpf_trampoline_enter(p), image + (jit-\u003eprog - (u8 )rw_image)))\n \t\treturn -EINVAL;\n \t/ remember prog start time returned by __bpf_prog_enter /\n-\temit_mov_reg(\u0026prog, true, BPF_REG_6, BPF_REG_0);\n+\temit_mov_reg(jit, true, BPF_REG_6, BPF_REG_0);\n \n \t/ if (__bpf_prog_enter(prog) == 0)\n \t \tgoto skip_exec_of_prog;\n \t /\n \tEMIT3(0x48, 0x85, 0xC0); / test rax,rax /\n \t/ emit 2 nops that will be replaced with JE insn /\n-\tjmp_insn = prog;\n-\temit_nops(\u0026prog, 2);\n+\tjmp_insn = jit-\u003eprog;\n+\temit_nops(jit, 2);\n \n \t/ arg1: lea rdi, [rbp - stack_size] /\n \tif (!is_imm8(-stack_size))\n@@ -3274,11 +3179,11 @@ static int invoke_bpf_prog(const struct btf_func_model m, u8 *pprog,\n \t\tEMIT4(0x48, 0x8D, 0x7D, -stack_size);\n \t/ arg2: progs[i]-\u003einsnsi for interpreter /\n \tif (!p-\u003ejited)\n-\t\temit_mov_imm64(\u0026prog, BPF_REG_2,\n+\t\temit_mov_imm64(jit, BPF_REG_2,\n \t\t\t (long) p-\u003einsnsi \u003e\u003e 32,\n \t\t\t (u32) (long) p-\u003einsnsi);\n \t/ call JITed bpf program or interpreter /\n-\tif (emit_rsb_call(\u0026prog, p-\u003ebpf_func, image + (prog - (u8 )rw_image)))\n+\tif (emit_rsb_call(jit, p-\u003ebpf_func, image + (jit-\u003eprog - (u8 )rw_image)))\n \t\treturn -EINVAL;\n \n \t/\n@@ -3290,42 +3195,39 @@ static int invoke_bpf_prog(const struct btf_func_model m, u8 pprog,\n \t value of BPF_PROG_TYPE_STRUCT_OPS prog.\n \t /\n \tif (save_ret)\n-\t\temit_stx(\u0026prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8);\n+\t\temit_stx(jit, BPF_DW, BPF_REG_FP, BPF_REG_0, -8);\n \n \t/ replace 2 nops with JE insn, since jmp target is known /\n-\tjmp_insn[0] = X86_JE;\n-\tjmp_insn[1] = prog - jmp_insn - 2;\n+\tif (!jit-\u003edry_run) {\n+\t\tjmp_insn[0] = X86_JE;\n+\t\tjmp_insn[1] = jit-\u003eprog - jmp_insn - 2;\n+\t}\n \n \t/ arg1: mov rdi, progs[i] /\n-\temit_mov_imm64(\u0026prog, BPF_REG_1, (long) p \u003e\u003e 32, (u32) (long) p);\n+\temit_mov_imm64(jit, BPF_REG_1, (long) p \u003e\u003e 32, (u32) (long) p);\n \t/ arg2: mov rsi, rbx \u003c- start time in nsec /\n-\temit_mov_reg(\u0026prog, true, BPF_REG_2, BPF_REG_6);\n+\temit_mov_reg(jit, true, BPF_REG_2, BPF_REG_6);\n \t/ arg3: lea rdx, [rbp - run_ctx_off] /\n \tif (!is_imm8(-run_ctx_off))\n \t\tEMIT3_off32(0x48, 0x8D, 0x95, -run_ctx_off);\n \telse\n \t\tEMIT4(0x48, 0x8D, 0x55, -run_ctx_off);\n-\tif (emit_rsb_call(\u0026prog, bpf_trampoline_exit(p), image + (prog - (u8 )rw_image)))\n+\tif (emit_rsb_call(jit, bpf_trampoline_exit(p), image + (jit-\u003eprog - (u8 )rw_image)))\n \t\treturn -EINVAL;\n \n-\tpprog = prog;\n \treturn 0;\n }\n \n-static void emit_align(u8 *pprog, u32 align)\n+static void emit_align(struct jit_emit_context jit, u32 align)\n {\n-\tu8 target, prog = pprog;\n-\n-\ttarget = PTR_ALIGN(prog, align);\n-\tif (target != prog)\n-\t\temit_nops(\u0026prog, target - prog);\n+\tu8 target = PTR_ALIGN(jit-\u003eprog, align);\n \n-\tpprog = prog;\n+\tif (target != jit-\u003eprog)\n+\t\temit_nops(jit, target - jit-\u003eprog);\n }\n \n-static int emit_cond_near_jump(u8 pprog, void func, void ip, u8 jmp_cond)\n+static int emit_cond_near_jump(struct jit_emit_context jit, void func, void ip, u8 jmp_cond)\n {\n-\tu8 prog = pprog;\n \ts64 offset;\n \n \toffset = func - (ip + 2 + 4);\n@@ -3334,48 +3236,44 @@ static int emit_cond_near_jump(u8 *pprog, void func, void ip, u8 jmp_cond)\n \t\treturn -EINVAL;\n \t}\n \tEMIT2_off32(0x0F, jmp_cond + 0x10, offset);\n-\tpprog = prog;\n \treturn 0;\n }\n \n-static int invoke_bpf(const struct btf_func_model m, u8 pprog,\n+static int invoke_bpf(const struct btf_func_model m, struct jit_emit_context jit,\n \t\t struct bpf_tramp_nodes tl, int stack_size,\n \t\t int run_ctx_off, int func_meta_off, bool save_ret,\n \t\t void image, void rw_image, u64 func_meta,\n \t\t int cookie_off)\n {\n \tint i, cur_cookie = (cookie_off - stack_size) / 8;\n-\tu8 prog = pprog;\n \n \tfor (i = 0; i \u003c tl-\u003enr_nodes; i++) {\n \t\tif (tl-\u003enodes[i]-\u003elink-\u003eprog-\u003ecall_session_cookie) {\n-\t\t\temit_store_stack_imm64(\u0026prog, BPF_REG_0, -func_meta_off,\n+\t\t\temit_store_stack_imm64(jit, BPF_REG_0, -func_meta_off,\n \t\t\t\tfunc_meta \| (cur_cookie \u003c\u003c BPF_TRAMP_COOKIE_INDEX_SHIFT));\n \t\t\tcur_cookie--;\n \t\t}\n-\t\tif (invoke_bpf_prog(m, \u0026prog, tl-\u003enodes[i], stack_size,\n+\t\tif (invoke_bpf_prog(m, jit, tl-\u003enodes[i], stack_size,\n \t\t\t\t run_ctx_off, save_ret, image, rw_image))\n \t\t\treturn -EINVAL;\n \t}\n-\tpprog = prog;\n \treturn 0;\n }\n \n-static int invoke_bpf_mod_ret(const struct btf_func_model m, u8 *pprog,\n+static int invoke_bpf_mod_ret(const struct btf_func_model m, struct jit_emit_context jit,\n \t\t\t struct bpf_tramp_nodes tl, int stack_size,\n \t\t\t int run_ctx_off, u8 *branches,\n \t\t\t void image, void rw_image)\n {\n-\tu8 prog = pprog;\n \tint i;\n \n \t/ The first fmod_ret program will receive a garbage return value.\n \t * Set this to 0 to avoid confusing the program.\n \t /\n-\temit_mov_imm32(\u0026prog, false, BPF_REG_0, 0);\n-\temit_stx(\u0026prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8);\n+\temit_mov_imm32(jit, false, BPF_REG_0, 0);\n+\temit_stx(jit, BPF_DW, BPF_REG_FP, BPF_REG_0, -8);\n \tfor (i = 0; i \u003c tl-\u003enr_nodes; i++) {\n-\t\tif (invoke_bpf_prog(m, \u0026prog, tl-\u003enodes[i], stack_size, run_ctx_off, true,\n+\t\tif (invoke_bpf_prog(m, jit, tl-\u003enodes[i], stack_size, run_ctx_off, true,\n \t\t\t\t image, rw_image))\n \t\t\treturn -EINVAL;\n \n@@ -3391,11 +3289,10 @@ static int invoke_bpf_mod_ret(const struct btf_func_model m, u8 *pprog,\n \t\t are replaced with a conditional jump once do_fexit (i.e. the\n \t\t * start of the fexit invocation) is finalized.\n \t\t /\n-\t\tbranches[i] = prog;\n-\t\temit_nops(\u0026prog, 4 + 2);\n+\t\tbranches[i] = jit-\u003eprog;\n+\t\temit_nops(jit, 4 + 2);\n \t}\n \n-\tpprog = prog;\n \treturn 0;\n }\n \n@@ -3474,12 +3371,12 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image im, void rw_im\n \tstruct bpf_tramp_nodes fentry = \u0026tnodes[BPF_TRAMP_FENTRY];\n \tstruct bpf_tramp_nodes fexit = \u0026tnodes[BPF_TRAMP_FEXIT];\n \tstruct bpf_tramp_nodes fmod_ret = \u0026tnodes[BPF_TRAMP_MODIFY_RETURN];\n+\tstruct jit_emit_context jit_ctx = {}, jit = \u0026jit_ctx;\n \tvoid orig_call = func_addr;\n \tint cookie_off, cookie_cnt;\n \tu8 branches = NULL;\n \tu64 arena_base;\n \tu64 func_meta;\n-\tu8 prog;\n \tbool save_ret;\n \n \t/\n@@ -3584,13 +3481,14 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image im, void rw_im\n \t\torig_call += X86_PATCH_SIZE;\n \t}\n \n-\tprog = rw_image;\n+\tjit-\u003eprog = rw_image;\n+\tjit-\u003edry_run = !rw_image;\n \n \tif (flags \u0026 BPF_TRAMP_F_INDIRECT) {\n \t\t/\n \t\t * Indirect call for bpf_struct_ops\n \t\t /\n-\t\temit_cfi(\u0026prog, image,\n+\t\temit_cfi(jit, image,\n \t\t\t cfi_get_func_hash(func_addr),\n \t\t\t cfi_get_func_arity(func_addr));\n \t} else {\n@@ -3598,12 +3496,12 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image im, void rw_im\n \t\t Direct-call fentry stub, as such it needs accounting for the\n \t\t * __fentry__ call.\n \t\t /\n-\t\tx86_call_depth_emit_accounting(\u0026prog, NULL, image);\n+\t\tbpf_call_depth_emit_accounting(jit, NULL, image);\n \t}\n \tEMIT1(0x55);\t\t / push rbp /\n \tEMIT3(0x48, 0x89, 0xE5); / mov rbp, rsp /\n \tif (im)\n-\t\tim-\u003eksym.fp_start = prog - (u8 )rw_image;\n+\t\tim-\u003eksym.fp_start = jit-\u003eprog - (u8 )rw_image;\n \n \tif (!is_imm8(stack_size)) {\n \t\t/ sub rsp, stack_size /\n@@ -3615,24 +3513,24 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image im, void rw_im\n \tif (flags \u0026 BPF_TRAMP_F_TAIL_CALL_CTX)\n \t\tEMIT1(0x50);\t\t/ push rax /\n \t/ mov QWORD PTR [rbp - rbx_off], rbx /\n-\temit_stx(\u0026prog, BPF_DW, BPF_REG_FP, BPF_REG_6, -rbx_off);\n+\temit_stx(jit, BPF_DW, BPF_REG_FP, BPF_REG_6, -rbx_off);\n \n \tfunc_meta = nr_regs;\n \t/ Store number of argument registers of the traced function /\n-\temit_store_stack_imm64(\u0026prog, BPF_REG_0, -func_meta_off, func_meta);\n+\temit_store_stack_imm64(jit, BPF_REG_0, -func_meta_off, func_meta);\n \n \tif (flags \u0026 BPF_TRAMP_F_IP_ARG) {\n \t\t/ Store IP address of the traced function /\n-\t\temit_store_stack_imm64(\u0026prog, BPF_REG_0, -ip_off, (long)func_addr);\n+\t\temit_store_stack_imm64(jit, BPF_REG_0, -ip_off, (long)func_addr);\n \t}\n \n-\tsave_args(m, \u0026prog, regs_off, false, flags, arena_base);\n+\tsave_args(m, jit, regs_off, false, flags, arena_base);\n \n \tif (flags \u0026 BPF_TRAMP_F_CALL_ORIG) {\n \t\t/ arg1: mov rdi, im /\n-\t\temit_mov_imm64(\u0026prog, BPF_REG_1, (long) im \u003e\u003e 32, (u32) (long) im);\n-\t\tif (emit_rsb_call(\u0026prog, __bpf_tramp_enter,\n-\t\t\t\t image + (prog - (u8 )rw_image))) {\n+\t\temit_mov_imm64(jit, BPF_REG_1, (long) im \u003e\u003e 32, (u32) (long) im);\n+\t\tif (emit_rsb_call(jit, __bpf_tramp_enter,\n+\t\t\t\t image + (jit-\u003eprog - (u8 )rw_image))) {\n \t\t\tret = -EINVAL;\n \t\t\tgoto cleanup;\n \t\t}\n@@ -3641,13 +3539,13 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image im, void rw_im\n \tif (bpf_fsession_cnt(tnodes)) {\n \t\t/ clear all the session cookies' value /\n \t\tfor (int i = 0; i \u003c cookie_cnt; i++)\n-\t\t\temit_store_stack_imm64(\u0026prog, BPF_REG_0, -cookie_off + 8 i, 0);\n+\t\t\temit_store_stack_imm64(jit, BPF_REG_0, -cookie_off + 8 * i, 0);\n \t\t/* clear the return value to make sure fentry always get 0 /\n-\t\temit_store_stack_imm64(\u0026prog, BPF_REG_0, -8, 0);\n+\t\temit_store_stack_imm64(jit, BPF_REG_0, -8, 0);\n \t}\n \n \tif (fentry-\u003enr_nodes) {\n-\t\tif (invoke_bpf(m, \u0026prog, fentry, regs_off, run_ctx_off, func_meta_off,\n+\t\tif (invoke_bpf(m, jit, fentry, regs_off, run_ctx_off, func_meta_off,\n \t\t\t flags \u0026 BPF_TRAMP_F_RET_FENTRY_RET, image, rw_image,\n \t\t\t func_meta, cookie_off))\n \t\t\treturn -EINVAL;\n@@ -3659,7 +3557,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image im, void rw_im\n \t\tif (!branches)\n \t\t\treturn -ENOMEM;\n \n-\t\tif (invoke_bpf_mod_ret(m, \u0026prog, fmod_ret, regs_off,\n+\t\tif (invoke_bpf_mod_ret(m, jit, fmod_ret, regs_off,\n \t\t\t\t run_ctx_off, branches, image, rw_image)) {\n \t\t\tret = -EINVAL;\n \t\t\tgoto cleanup;\n@@ -3667,8 +3565,8 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image im, void rw_im\n \t}\n \n \tif (flags \u0026 BPF_TRAMP_F_CALL_ORIG) {\n-\t\trestore_regs(m, \u0026prog, regs_off);\n-\t\tsave_args(m, \u0026prog, arg_stack_off, true, flags, 0);\n+\t\trestore_regs(m, jit, regs_off);\n+\t\tsave_args(m, jit, arg_stack_off, true, flags, 0);\n \n \t\tif (flags \u0026 BPF_TRAMP_F_TAIL_CALL_CTX) {\n \t\t\t/ Before calling the original function, load the\n@@ -3678,19 +3576,19 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image im, void rw_im\n \t\t}\n \n \t\tif (flags \u0026 BPF_TRAMP_F_ORIG_STACK) {\n-\t\t\temit_ldx(\u0026prog, BPF_DW, BPF_REG_6, BPF_REG_FP, 8);\n+\t\t\temit_ldx(jit, BPF_DW, BPF_REG_6, BPF_REG_FP, 8);\n \t\t\tEMIT2(0xff, 0xd3); /* call rbx /\n \t\t} else {\n \t\t\t/* call original function /\n-\t\t\tif (emit_rsb_call(\u0026prog, orig_call, image + (prog - (u8 )rw_image))) {\n+\t\t\tif (emit_rsb_call(jit, orig_call, image + (jit-\u003eprog - (u8 )rw_image))) {\n \t\t\t\tret = -EINVAL;\n \t\t\t\tgoto cleanup;\n \t\t\t}\n \t\t}\n \t\t/ remember return value in a stack for bpf prog to access /\n-\t\temit_stx(\u0026prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8);\n-\t\tim-\u003eip_after_call = image + (prog - (u8 )rw_image);\n-\t\temit_nops(\u0026prog, X86_PATCH_SIZE);\n+\t\temit_stx(jit, BPF_DW, BPF_REG_FP, BPF_REG_0, -8);\n+\t\tim-\u003eip_after_call = image + (jit-\u003eprog - (u8 )rw_image);\n+\t\temit_nops(jit, X86_PATCH_SIZE);\n \t}\n \n \tif (fmod_ret-\u003enr_nodes) {\n@@ -3699,12 +3597,17 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image im, void rw_im\n \t\t Coding Rule 11: All branch targets should be 16-byte\n \t\t * aligned.\n \t\t /\n-\t\temit_align(\u0026prog, 16);\n+\t\temit_align(jit, 16);\n \t\t/ Update the branches saved in invoke_bpf_mod_ret with the\n \t\t * aligned address of do_fexit.\n \t\t /\n \t\tfor (i = 0; i \u003c fmod_ret-\u003enr_nodes; i++) {\n-\t\t\temit_cond_near_jump(\u0026branches[i], image + (prog - (u8 )rw_image),\n+\t\t\tstruct jit_emit_context branch_jit = {\n+\t\t\t\t.prog = branches[i],\n+\t\t\t\t.dry_run = jit-\u003edry_run,\n+\t\t\t};\n+\n+\t\t\temit_cond_near_jump(\u0026branch_jit, image + (jit-\u003eprog - (u8 )rw_image),\n \t\t\t\t\t image + (branches[i] - (u8 )rw_image), X86_JNE);\n \t\t}\n \t}\n@@ -3712,10 +3615,10 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image im, void rw_im\n \t/* set the \"is_return\" flag for fsession /\n \tfunc_meta \|= (1ULL \u003c\u003c BPF_TRAMP_IS_RETURN_SHIFT);\n \tif (bpf_fsession_cnt(tnodes))\n-\t\temit_store_stack_imm64(\u0026prog, BPF_REG_0, -func_meta_off, func_meta);\n+\t\temit_store_stack_imm64(jit, BPF_REG_0, -func_meta_off, func_meta);\n \n \tif (fexit-\u003enr_nodes) {\n-\t\tif (invoke_bpf(m, \u0026prog, fexit, regs_off, run_ctx_off, func_meta_off,\n+\t\tif (invoke_bpf(m, jit, fexit, regs_off, run_ctx_off, func_meta_off,\n \t\t\t false, image, rw_image, func_meta, cookie_off)) {\n \t\t\tret = -EINVAL;\n \t\t\tgoto cleanup;\n@@ -3723,17 +3626,17 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image im, void rw_im\n \t}\n \n \tif (flags \u0026 BPF_TRAMP_F_RESTORE_REGS)\n-\t\trestore_regs(m, \u0026prog, regs_off);\n+\t\trestore_regs(m, jit, regs_off);\n \n \t/ This needs to be done regardless. If there were fmod_ret programs,\n \t * the return value is only updated on the stack and still needs to be\n \t * restored to R0.\n \t /\n \tif (flags \u0026 BPF_TRAMP_F_CALL_ORIG) {\n-\t\tim-\u003eip_epilogue = image + (prog - (u8 )rw_image);\n+\t\tim-\u003eip_epilogue = image + (jit-\u003eprog - (u8 )rw_image);\n \t\t/ arg1: mov rdi, im /\n-\t\temit_mov_imm64(\u0026prog, BPF_REG_1, (long) im \u003e\u003e 32, (u32) (long) im);\n-\t\tif (emit_rsb_call(\u0026prog, __bpf_tramp_exit, image + (prog - (u8 )rw_image))) {\n+\t\temit_mov_imm64(jit, BPF_REG_1, (long) im \u003e\u003e 32, (u32) (long) im);\n+\t\tif (emit_rsb_call(jit, __bpf_tramp_exit, image + (jit-\u003eprog - (u8 )rw_image))) {\n \t\t\tret = -EINVAL;\n \t\t\tgoto cleanup;\n \t\t}\n@@ -3746,25 +3649,26 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image im, void rw_im\n \n \t/ restore return value of orig_call or fentry prog back into RAX /\n \tif (save_ret)\n-\t\temit_ldx(\u0026prog, BPF_DW, BPF_REG_0, BPF_REG_FP, -8);\n+\t\temit_ldx(jit, BPF_DW, BPF_REG_0, BPF_REG_FP, -8);\n \n-\temit_ldx(\u0026prog, BPF_DW, BPF_REG_6, BPF_REG_FP, -rbx_off);\n+\temit_ldx(jit, BPF_DW, BPF_REG_6, BPF_REG_FP, -rbx_off);\n \n \tEMIT1(0xC9); / leave /\n \tif (im)\n-\t\tim-\u003eksym.fp_end = prog - (u8 )rw_image;\n+\t\tim-\u003eksym.fp_end = jit-\u003eprog - (u8 )rw_image;\n \n \tif (flags \u0026 BPF_TRAMP_F_SKIP_FRAME) {\n \t\t/ skip our return address and return to parent /\n \t\tEMIT4(0x48, 0x83, 0xC4, 8); / add rsp, 8 /\n \t}\n-\temit_return(\u0026prog, image + (prog - (u8 )rw_image));\n+\temit_return(jit, image + (jit-\u003eprog - (u8 )rw_image));\n \t/ Make sure the trampoline generation logic doesn't overflow /\n-\tif (WARN_ON_ONCE(prog \u003e (u8 )rw_image_end - BPF_INSN_SAFETY)) {\n+\tif (!jit-\u003edry_run \u0026\u0026\n+\t WARN_ON_ONCE(jit-\u003eprog \u003e (u8 )rw_image_end - BPF_INSN_SAFETY)) {\n \t\tret = -EFAULT;\n \t\tgoto cleanup;\n \t}\n-\tret = prog - (u8 )rw_image + BPF_INSN_SAFETY;\n+\tret = jit-\u003eprog - (u8 )rw_image + BPF_INSN_SAFETY;\n \n cleanup:\n \tkfree(branches);\n@@ -3819,28 +3723,15 @@ int arch_bpf_trampoline_size(const struct btf_func_model m, u32 flags,\n \t\t\t struct bpf_tramp_nodes tnodes, void func_addr)\n {\n \tstruct bpf_tramp_image im;\n-\tvoid image;\n-\tint ret;\n \n-\t/ Allocate a temporary buffer for __arch_prepare_bpf_trampoline().\n-\t \n-\t We cannot use kvmalloc here, because we need image to be in\n-\t * module memory range.\n-\t * Since it must be writable use bpf_jit_alloc_exec_rw().\n-\t /\n-\timage = bpf_jit_alloc_exec_rw(PAGE_SIZE);\n-\tif (!image)\n-\t\treturn -ENOMEM;\n-\n-\tret = __arch_prepare_bpf_trampoline(\u0026im, image, image + PAGE_SIZE, image,\n-\t\t\t\t\t m, flags, tnodes, func_addr);\n-\tbpf_jit_free_exec(image);\n-\treturn ret;\n+\treturn __arch_prepare_bpf_trampoline(\u0026im, NULL, NULL, NULL, m, flags,\n+\t\t\t\t\t tnodes, func_addr);\n }\n \n-static int emit_bpf_dispatcher(u8 pprog, int a, int b, s64 progs, u8 image, u8 buf)\n+static int emit_bpf_dispatcher(struct jit_emit_context jit, int a, int b, s64 progs, u8 image,\n+\t\t\t u8 buf)\n {\n-\tu8 jg_reloc, prog = pprog;\n+\tu8 jg_reloc;\n \tint pivot, err, jg_bytes = 1;\n \ts64 jg_offset;\n \n@@ -3853,15 +3744,13 @@ static int emit_bpf_dispatcher(u8 *pprog, int a, int b, s64 progs, u8 image,\n \t\t\treturn -1;\n \t\tEMIT2_off32(0x81, add_1reg(0xF8, BPF_REG_3),\n \t\t\t progs[a]);\n-\t\terr = emit_cond_near_jump(\u0026prog,\t/ je func /\n-\t\t\t\t\t (void )progs[a], image + (prog - buf),\n+\t\terr = emit_cond_near_jump(jit,\t/* je func /\n+\t\t\t\t\t (void )progs[a], image + (jit-\u003eprog - buf),\n \t\t\t\t\t X86_JE);\n \t\tif (err)\n \t\t\treturn err;\n \n-\t\temit_indirect_jump(\u0026prog, BPF_REG_3 /* R3 -\u003e rdx /, image + (prog - buf));\n-\n-\t\tpprog = prog;\n+\t\temit_indirect_jump(jit, BPF_REG_3 /* R3 -\u003e rdx /, image + (jit-\u003eprog - buf));\n \t\treturn 0;\n \t}\n \n@@ -3881,9 +3770,9 @@ static int emit_bpf_dispatcher(u8 pprog, int a, int b, s64 progs, u8 image,\n \t} else {\n \t\tEMIT2(X86_JG, 0);\n \t}\n-\tjg_reloc = prog;\n+\tjg_reloc = jit-\u003eprog;\n \n-\terr = emit_bpf_dispatcher(\u0026prog, a, a + pivot,\t/ emit lower_part /\n+\terr = emit_bpf_dispatcher(jit, a, a + pivot,\t/ emit lower_part /\n \t\t\t\t progs, image, buf);\n \tif (err)\n \t\treturn err;\n@@ -3893,16 +3782,15 @@ static int emit_bpf_dispatcher(u8 pprog, int a, int b, s64 progs, u8 image,\n \t Coding Rule 11: All branch targets should be 16-byte\n \t * aligned.\n \t /\n-\temit_align(\u0026prog, 16);\n-\tjg_offset = prog - jg_reloc;\n+\temit_align(jit, 16);\n+\tjg_offset = jit-\u003eprog - jg_reloc;\n \temit_code(jg_reloc - jg_bytes, jg_offset, jg_bytes);\n \n-\terr = emit_bpf_dispatcher(\u0026prog, a + pivot + 1,\t/ emit upper_part /\n+\terr = emit_bpf_dispatcher(jit, a + pivot + 1,\t/ emit upper_part /\n \t\t\t\t b, progs, image, buf);\n \tif (err)\n \t\treturn err;\n \n-\tpprog = prog;\n \treturn 0;\n }\n \n@@ -3920,10 +3808,10 @@ static int cmp_ips(const void a, const void b)\n \n int arch_prepare_bpf_dispatcher(void image, void buf, s64 funcs, int num_funcs)\n {\n-\tu8 prog = buf;\n+\tstruct jit_emit_context jit = { .prog = buf };\n \n \tsort(funcs, num_funcs, sizeof(funcs[0]), cmp_ips, NULL);\n-\treturn emit_bpf_dispatcher(\u0026prog, 0, num_funcs - 1, funcs, image, buf);\n+\treturn emit_bpf_dispatcher(\u0026jit, 0, num_funcs - 1, funcs, image, buf);\n }\n \n static void priv_stack_init_guard(void __percpu *priv_stack_ptr, int alloc_size)\n" }
2/1	2026/08/31 14:02	agent	patch-evaluator	16m Model: core Args: null Results: { "EnableConfigs": [ "BPF_SYSCALL", "BPF_JIT", "CALL_THUNKS", "MITIGATION_CALL_DEPTH_TRACKING", "X86_64" ], "FocusSymbols": [ "arch_prepare_bpf_trampoline", "arch_bpf_trampoline_size", "bpf_int_jit_compile", "arch_prepare_bpf_dispatcher", "x86_call_depth_emit_accounting" ], "Reasoning": "The patch refactors the x86 BPF JIT compiler and trampoline generator by introducing a struct jit_emit_context with dry-run support. This changes how trampoline sizing is computed (dry-run without allocation), updates instruction emission logic, and alters x86 call depth accounting integration. These changes affect BPF program compilation, trampoline generation, and execution reachable via the bpf() syscall.", "WorthFuzzing": true } Instruction: You are an expert Linux kernel maintainer. Your job is to review a provided patch series and determine if it makes functional changes to the kernel that should be fuzzed. IMPORTANT: The changes have ALREADY been applied and committed as the HEAD commit in your workspace. Do NOT rely on your internal knowledge of the kernel. You must actively use your code access tools to examine the actual source code and confirm any assumptions. Return WorthFuzzing=false if the patch only contains: - Modifications to Documentation/, Kconfig files, or code comments. - Purely decorative changes, such as logging (e.g., pr_err, printk) or tracepoints. - Changes to numeric constants or macros that do not functionally alter execution flow. - Code paths that are impossible to reach in virtualized environments like GCE or QEMU, even when utilizing software-emulated hardware (e.g., usb gadget, mac80211_hwsim). - Code in vendor-specific PCIe switch, SmartNIC, or GPU drivers (e.g., mlxsw, pds_core, qed, ionic, amdgpu) that require physical PCIe hardware cards not emulated in standard QEMU. - Driver .remove, .shutdown, or pci_unregister_driver teardown callbacks (e.g., igb_remove) that are executed only during PCI hot-unplug or sysfs driver unbind operations. If it modifies reachable core kernel logic, drivers, or architectures, use your code search tools to verify the code can be executed, then return WorthFuzzing=true. When returning WorthFuzzing=true, you MUST ALSO: 1. Extract any specific kernel functions that should be heavily fuzzed into FocusSymbols. Avoid listing generic hot-path functions to prevent skewed test distributions. Prefer non-static, non-inlined API entrypoint functions over internal static helper functions (which are inlined by the compiler and do not have distinct symbol addresses). 2. Identify any specific CONFIG_ options required to properly test this new/modified feature. Go and look into the Kconfig files and check for ifdefs around the code, do not make assumptions. Also check "depends on" lines in Kconfig to include any non-standard parent subsystem configs needed for Kbuild to compile the code statically into vmlinux. List them in the EnableConfigs output array, and DO NOT add a 'CONFIG_' prefix (e.g., return "NET_IPV4" instead of "CONFIG_NET_IPV4"). Prefer calling several tools at the same time to save round-trips. Use set-results tool to provide results of the analysis. It must be called exactly once before the final reply. Ignore results of this tool. Prompt: For your convenience, here is the diff of the changes: commit e09e4ad11f933fa9ab21debd310aeb220f3f52c0 Author: syz-cluster <triage@syzkaller.com> Date: Mon Aug 31 14:02:28 2026 +0000 syz-cluster: applied patch under review diff --git a/arch/x86/include/asm/alternative.h b/arch/x86/include/asm/alternative.h index 08af86ef090a8..2ef4937411860 100644 --- a/arch/x86/include/asm/alternative.h +++ b/arch/x86/include/asm/alternative.h @@ -83,7 +83,7 @@ extern void callthunks_patch_builtin_calls(void); extern void callthunks_patch_module_calls(struct callthunk_sites sites, struct module mod); extern void callthunks_translate_call_dest(void dest); -extern int x86_call_depth_emit_accounting(u8 *pprog, void func, void ip); +extern int x86_call_depth_emit_accounting(u8 insn_buff, void func, void ip); #else static __always_inline void callthunks_patch_builtin_calls(void) {} static __always_inline void @@ -93,7 +93,7 @@ static __always_inline void callthunks_translate_call_dest(void dest) { return dest; } -static __always_inline int x86_call_depth_emit_accounting(u8 *pprog, +static __always_inline int x86_call_depth_emit_accounting(u8 insn_buff, void func, void ip) { return 0; diff --git a/arch/x86/include/asm/text-patching.h b/arch/x86/include/asm/text-patching.h index f2d142a0a862e..a0a7c778b0a2a 100644 --- a/arch/x86/include/asm/text-patching.h +++ b/arch/x86/include/asm/text-patching.h @@ -13,6 +13,8 @@ / #define TEXT_POKE_MAX_OPCODE_SIZE 5 +#define MAX_PATCH_LEN (255-1) + extern void text_poke_early(void addr, const void opcode, size_t len); extern void text_poke_apply_relocation(u8 buf, const u8 * const instr, size_t instrlen, u8 repl, size_t repl_len); diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index 91b1cdd165699..5f9989c8559ea 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -18,8 +18,6 @@ int __read_mostly alternatives_patched; EXPORT_SYMBOL_GPL(alternatives_patched); -#define MAX_PATCH_LEN (255-1) - #define DA_ALL (~0) #define DA_ALT 0x01 #define DA_RET 0x02 diff --git a/arch/x86/kernel/callthunks.c b/arch/x86/kernel/callthunks.c index e37728f703227..15b221d06901c 100644 --- a/arch/x86/kernel/callthunks.c +++ b/arch/x86/kernel/callthunks.c @@ -23,8 +23,6 @@ static int __initdata_or_module debug_callthunks; -#define MAX_PATCH_LEN (255-1) - #define prdbg(fmt, args...) \ do { \ if (debug_callthunks) \ @@ -298,10 +296,9 @@ static bool is_callthunk(void addr) return !bcmp(pad, insn_buff, tmpl_size); } -int x86_call_depth_emit_accounting(u8 *pprog, void func, void ip) +int x86_call_depth_emit_accounting(u8 insn_buff, void func, void ip) { unsigned int tmpl_size = SKL_TMPL_SIZE; - u8 insn_buff[MAX_PATCH_LEN]; if (!thunks_initialized) return 0; @@ -313,8 +310,6 @@ int x86_call_depth_emit_accounting(u8 *pprog, void func, void ip) memcpy(insn_buff, skl_call_thunk_template, tmpl_size); text_poke_apply_relocation(insn_buff, ip, tmpl_size, skl_call_thunk_template, tmpl_size); - memcpy(pprog, insn_buff, tmpl_size); - pprog += tmpl_size; return tmpl_size; } #endif diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 48429fae06410..2ec734c69b693 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -23,6 +23,11 @@ static bool all_callee_regs_used[4] = {true, true, true, true}; +struct jit_emit_context { + u8 prog; + bool dry_run; +}; + static u8 emit_code(u8 ptr, u32 bytes, unsigned int len) { if (len == 1) @@ -36,8 +41,16 @@ static u8 emit_code(u8 ptr, u32 bytes, unsigned int len) return ptr + len; } +static void emit_code_jit(struct jit_emit_context jit, u32 bytes, unsigned int len) +{ + if (jit->dry_run) + jit->prog += len; + else + jit->prog = emit_code(jit->prog, bytes, len); +} + #define EMIT(bytes, len) \ - do { prog = emit_code(prog, bytes, len); } while (0) + emit_code_jit(jit, bytes, len) #define EMIT1(b1) EMIT(b1, 1) #define EMIT2(b1, b2) EMIT((b1) + ((b2) << 8), 2) @@ -149,6 +162,19 @@ static int bpf_size_to_x86_bytes(int bpf_size) return 0; } +static int bpf_call_depth_emit_accounting(struct jit_emit_context jit, void func, void ip) +{ + u8 insn_buff[MAX_PATCH_LEN]; + int size; + + size = x86_call_depth_emit_accounting(insn_buff, func, ip); + if (!jit->dry_run) + memcpy(jit->prog, insn_buff, size); + + jit->prog += size; + return size; +} + /* * List of x86 cond jumps opcodes (. + s8) * Add 0x10 (and an extra 0x0f) to generate far jumps (. + s32) @@ -309,6 +335,8 @@ int bpf_arch_text_invalidate(void dst, size_t len) } struct jit_context { + struct jit_emit_context jit; + int cleanup_addr; / Epilogue code offset / / @@ -329,34 +357,23 @@ struct jit_context { /* Number of bytes that will be skipped on tailcall / #define X86_TAIL_CALL_OFFSET (12 + ENDBR_INSN_SIZE) -static void push_r9(u8 pprog) +static void push_r9(struct jit_emit_context jit) { - u8 prog = pprog; - EMIT2(0x41, 0x51); /* push r9 / - pprog = prog; } -static void pop_r9(u8 *pprog) +static void pop_r9(struct jit_emit_context jit) { - u8 prog = pprog; - EMIT2(0x41, 0x59); /* pop r9 / - pprog = prog; } -static void push_r12(u8 *pprog) +static void push_r12(struct jit_emit_context jit) { - u8 prog = pprog; - EMIT2(0x41, 0x54); /* push r12 / - pprog = prog; } -static void push_callee_regs(u8 *pprog, bool callee_regs_used) +static void push_callee_regs(struct jit_emit_context jit, bool callee_regs_used) { - u8 prog = pprog; - if (callee_regs_used[0]) EMIT1(0x53); /* push rbx / if (callee_regs_used[1]) @@ -365,21 +382,15 @@ static void push_callee_regs(u8 pprog, bool callee_regs_used) EMIT2(0x41, 0x56); /* push r14 / if (callee_regs_used[3]) EMIT2(0x41, 0x57); / push r15 / - pprog = prog; } -static void pop_r12(u8 *pprog) +static void pop_r12(struct jit_emit_context jit) { - u8 prog = pprog; - EMIT2(0x41, 0x5C); /* pop r12 / - pprog = prog; } -static void pop_callee_regs(u8 *pprog, bool callee_regs_used) +static void pop_callee_regs(struct jit_emit_context jit, bool callee_regs_used) { - u8 prog = pprog; - if (callee_regs_used[3]) EMIT2(0x41, 0x5F); /* pop r15 / if (callee_regs_used[2]) @@ -388,40 +399,32 @@ static void pop_callee_regs(u8 pprog, bool callee_regs_used) EMIT2(0x41, 0x5D); /* pop r13 / if (callee_regs_used[0]) EMIT1(0x5B); / pop rbx / - pprog = prog; } /* add rsp, depth / -static void emit_add_rsp(u8 pprog, u16 depth) +static void emit_add_rsp(struct jit_emit_context jit, u16 depth) { - u8 prog = pprog; - if (!depth) return; if (is_imm8(depth)) EMIT4(0x48, 0x83, 0xC4, depth); /* add rsp, imm8 / else EMIT3_off32(0x48, 0x81, 0xC4, depth); / add rsp, imm32 / - pprog = prog; } /* sub rsp, depth / -static void emit_sub_rsp(u8 pprog, u16 depth) +static void emit_sub_rsp(struct jit_emit_context jit, u16 depth) { - u8 prog = pprog; - if (!depth) return; if (is_imm8(depth)) EMIT4(0x48, 0x83, 0xEC, depth); /* sub rsp, imm8 / else EMIT3_off32(0x48, 0x81, 0xEC, depth); / sub rsp, imm32 / - pprog = prog; } -static void emit_nops(u8 *pprog, int len) +static void emit_nops(struct jit_emit_context jit, int len) { - u8 prog = pprog; int i, noplen; while (len > 0) { @@ -434,72 +437,56 @@ static void emit_nops(u8 *pprog, int len) EMIT1(x86_nops[noplen][i]); len -= noplen; } - - pprog = prog; } /* * Emit the various CFI preambles, see asm/cfi.h and the comments about FineIBT * in arch/x86/kernel/alternative.c / -static int emit_call(u8 prog, void func, void ip); +static int emit_call(struct jit_emit_context jit, void func, void ip); -static void emit_fineibt(u8 *pprog, u8 ip, u32 hash, int arity) +static void emit_fineibt(struct jit_emit_context jit, u8 ip, u32 hash, int arity) { - u8 prog = pprog; - EMIT_ENDBR(); EMIT1_off32(0x2d, hash); /* subl $hash, %eax / if (cfi_bhi) { EMIT2(0x2e, 0x2e); / cs cs / - emit_call(&prog, __bhi_args[arity], ip + 11); + emit_call(jit, __bhi_args[arity], ip + 11); } else { EMIT3_off32(0x2e, 0x0f, 0x85, 3); / jne.d32,pn 3 / } EMIT_ENDBR_POISON(); - - pprog = prog; } -static void emit_kcfi(u8 *pprog, u32 hash) +static void emit_kcfi(struct jit_emit_context jit, u32 hash) { - u8 prog = pprog; - EMIT1_off32(0xb8, hash); /* movl $hash, %eax / #ifdef CONFIG_CALL_PADDING for (int i = 0; i < CONFIG_FUNCTION_PADDING_CFI; i++) EMIT1(0x90); #endif EMIT_ENDBR(); - - pprog = prog; } -static void emit_cfi(u8 *pprog, u8 ip, u32 hash, int arity) +static void emit_cfi(struct jit_emit_context jit, u8 ip, u32 hash, int arity) { - u8 prog = pprog; - switch (cfi_mode) { case CFI_FINEIBT: - emit_fineibt(&prog, ip, hash, arity); + emit_fineibt(jit, ip, hash, arity); break; case CFI_KCFI: - emit_kcfi(&prog, hash); + emit_kcfi(jit, hash); break; default: EMIT_ENDBR(); break; } - - pprog = prog; } -static void emit_prologue_tail_call(u8 pprog, bool is_subprog) +static void emit_prologue_tail_call(struct jit_emit_context jit, bool is_subprog) { - u8 prog = pprog; - if (!is_subprog) { /* cmp rax, MAX_TAIL_CALL_CNT / EMIT4(0x48, 0x83, 0xF8, MAX_TAIL_CALL_CNT); @@ -523,8 +510,6 @@ static void emit_prologue_tail_call(u8 pprog, bool is_subprog) EMIT1(0x50); / push rax / EMIT1(0x50); / push rax / } - - pprog = prog; } /* @@ -532,21 +517,19 @@ static void emit_prologue_tail_call(u8 *pprog, bool is_subprog) bpf_tail_call helper will skip the first X86_TAIL_CALL_OFFSET bytes * while jumping to another program / -static void emit_prologue(u8 pprog, u8 ip, u32 stack_depth, bool ebpf_from_cbpf, - bool tail_call_reachable, bool is_subprog, +static void emit_prologue(struct jit_emit_context jit, u8 ip, u32 stack_depth, + bool ebpf_from_cbpf, bool tail_call_reachable, bool is_subprog, bool is_exception_cb) { - u8 prog = pprog; - if (is_subprog) { - emit_cfi(&prog, ip, cfi_bpf_subprog_hash, 5); + emit_cfi(jit, ip, cfi_bpf_subprog_hash, 5); } else { - emit_cfi(&prog, ip, cfi_bpf_hash, 1); + emit_cfi(jit, ip, cfi_bpf_hash, 1); } /* BPF trampoline can be made to work without these nops, * but let's waste 5 bytes for now and optimize later / - emit_nops(&prog, X86_PATCH_SIZE); + emit_nops(jit, X86_PATCH_SIZE); if (!ebpf_from_cbpf) { if (tail_call_reachable && !is_subprog) / When it's the entry of the whole tailcall context, @@ -555,7 +538,7 @@ static void emit_prologue(u8 *pprog, u8 ip, u32 stack_depth, bool ebpf_from_cb EMIT3(0x48, 0x31, 0xC0); /* xor rax, rax / else / Keep the same instruction layout. / - emit_nops(&prog, 3); / nop3 / + emit_nops(jit, 3); / nop3 / } / Exception callback receives FP as third parameter / if (is_exception_cb) { @@ -565,8 +548,8 @@ static void emit_prologue(u8 pprog, u8 ip, u32 stack_depth, bool ebpf_from_cb * first restore those callee-saved regs from stack, before * reusing the stack frame. / - pop_callee_regs(&prog, all_callee_regs_used); - pop_r12(&prog); + pop_callee_regs(jit, all_callee_regs_used); + pop_r12(jit); / Reset the stack frame. / EMIT3(0x48, 0x89, 0xEC); / mov rsp, rbp / } else { @@ -581,40 +564,38 @@ static void emit_prologue(u8 pprog, u8 ip, u32 stack_depth, bool ebpf_from_cb if (stack_depth) EMIT3_off32(0x48, 0x81, 0xEC, round_up(stack_depth, 8)); if (tail_call_reachable) - emit_prologue_tail_call(&prog, is_subprog); - pprog = prog; + emit_prologue_tail_call(jit, is_subprog); } -static int emit_patch(u8 pprog, void func, void ip, u8 opcode) +static int emit_patch(struct jit_emit_context jit, void func, void ip, u8 opcode) { - u8 prog = pprog; s64 offset; offset = func - (ip + X86_PATCH_SIZE); - if (!is_simm32(offset)) { + /* We do not have meaningful ip value in the dry run, skip the check. / + if (!jit->dry_run && !is_simm32(offset)) { pr_err("Target call %p is out of range\n", func); return -ERANGE; } EMIT1_off32(opcode, offset); - pprog = prog; return 0; } -static int emit_call(u8 *pprog, void func, void ip) +static int emit_call(struct jit_emit_context jit, void func, void ip) { - return emit_patch(pprog, func, ip, 0xE8); + return emit_patch(jit, func, ip, 0xE8); } -static int emit_rsb_call(u8 *pprog, void func, void ip) +static int emit_rsb_call(struct jit_emit_context jit, void func, void ip) { OPTIMIZER_HIDE_VAR(func); - ip += x86_call_depth_emit_accounting(pprog, func, ip); - return emit_patch(pprog, func, ip, 0xE8); + ip += bpf_call_depth_emit_accounting(jit, func, ip); + return emit_patch(jit, func, ip, 0xE8); } -static int emit_jump(u8 *pprog, void func, void ip) +static int emit_jump(struct jit_emit_context jit, void func, void ip) { - return emit_patch(pprog, func, ip, 0xE9); + return emit_patch(jit, func, ip, 0xE9); } static int __bpf_arch_text_poke(void ip, enum bpf_text_poke_type old_t, @@ -622,27 +603,27 @@ static int __bpf_arch_text_poke(void ip, enum bpf_text_poke_type old_t, void old_addr, void new_addr) { const u8 nop_insn = x86_nops[5]; + struct jit_emit_context jit = {}; u8 old_insn[X86_PATCH_SIZE]; u8 new_insn[X86_PATCH_SIZE]; - u8 prog; int ret; memcpy(old_insn, nop_insn, X86_PATCH_SIZE); if (old_t != BPF_MOD_NOP && old_addr) { - prog = old_insn; + jit.prog = old_insn; ret = old_t == BPF_MOD_CALL ? - emit_call(&prog, old_addr, ip) : - emit_jump(&prog, old_addr, ip); + emit_call(&jit, old_addr, ip) : + emit_jump(&jit, old_addr, ip); if (ret) return ret; } memcpy(new_insn, nop_insn, X86_PATCH_SIZE); if (new_t != BPF_MOD_NOP && new_addr) { - prog = new_insn; + jit.prog = new_insn; ret = new_t == BPF_MOD_CALL ? - emit_call(&prog, new_addr, ip) : - emit_jump(&prog, new_addr, ip); + emit_call(&jit, new_addr, ip) : + emit_jump(&jit, new_addr, ip); if (ret) return ret; } @@ -682,58 +663,47 @@ int bpf_arch_text_poke(void ip, enum bpf_text_poke_type old_t, #define EMIT_LFENCE() EMIT3(0x0F, 0xAE, 0xE8) -static void __emit_indirect_jump(u8 pprog, int reg, bool ereg) +static void __emit_indirect_jump(struct jit_emit_context jit, int reg, bool ereg) { - u8 prog = pprog; - if (ereg) EMIT1(0x41); EMIT2(0xFF, 0xE0 + reg); - - pprog = prog; } -static void emit_indirect_jump(u8 pprog, int bpf_reg, u8 ip) +static void emit_indirect_jump(struct jit_emit_context jit, int bpf_reg, u8 ip) { - u8 prog = pprog; int reg = reg2hex[bpf_reg]; bool ereg = is_ereg(bpf_reg); if (cpu_feature_enabled(X86_FEATURE_INDIRECT_THUNK_ITS)) { OPTIMIZER_HIDE_VAR(reg); - emit_jump(&prog, its_static_thunk(reg + 8ereg), ip); + emit_jump(jit, its_static_thunk(reg + 8ereg), ip); } else if (cpu_feature_enabled(X86_FEATURE_RETPOLINE_LFENCE)) { EMIT_LFENCE(); - __emit_indirect_jump(&prog, reg, ereg); + __emit_indirect_jump(jit, reg, ereg); } else if (cpu_feature_enabled(X86_FEATURE_RETPOLINE)) { OPTIMIZER_HIDE_VAR(reg); if (cpu_feature_enabled(X86_FEATURE_CALL_DEPTH)) - emit_jump(&prog, &__x86_indirect_jump_thunk_array[reg + 8ereg], ip); + emit_jump(jit, &__x86_indirect_jump_thunk_array[reg + 8ereg], ip); else - emit_jump(&prog, &__x86_indirect_thunk_array[reg + 8ereg], ip); + emit_jump(jit, &__x86_indirect_thunk_array[reg + 8ereg], ip); } else { - __emit_indirect_jump(&prog, reg, ereg); + __emit_indirect_jump(jit, reg, ereg); if (IS_ENABLED(CONFIG_MITIGATION_RETPOLINE) \|\| IS_ENABLED(CONFIG_MITIGATION_SLS)) EMIT1(0xCC); /* int3 / } - - pprog = prog; } -static void emit_return(u8 *pprog, u8 ip) +static void emit_return(struct jit_emit_context jit, u8 ip) { - u8 prog = pprog; - if (cpu_wants_rethunk()) { - emit_jump(&prog, x86_return_thunk, ip); + emit_jump(jit, x86_return_thunk, ip); } else { EMIT1(0xC3); /* ret / if (IS_ENABLED(CONFIG_MITIGATION_SLS)) EMIT1(0xCC); / int3 / } - - pprog = prog; } #define BPF_TAIL_CALL_CNT_PTR_STACK_OFF(stack) (-16 - round_up(stack, 8)) @@ -753,12 +723,13 @@ static void emit_return(u8 *pprog, u8 ip) * out: / static void emit_bpf_tail_call_indirect(struct bpf_prog bpf_prog, - u8 *pprog, bool callee_regs_used, + bool callee_regs_used, u32 stack_depth, u8 ip, struct jit_context ctx) { int tcc_ptr_off = BPF_TAIL_CALL_CNT_PTR_STACK_OFF(stack_depth); - u8 prog = pprog, start = pprog; + struct jit_emit_context jit = &ctx->jit; + u8 start = jit->prog; int offset; / @@ -775,7 +746,7 @@ static void emit_bpf_tail_call_indirect(struct bpf_prog bpf_prog, EMIT3(0x39, 0x56, / cmp dword ptr [rsi + 16], edx / offsetof(struct bpf_array, map.max_entries)); - offset = ctx->tail_call_indirect_label - (prog + 2 - start); + offset = ctx->tail_call_indirect_label - (jit->prog + 2 - start); EMIT2(X86_JBE, offset); / jbe out / / @@ -785,7 +756,7 @@ static void emit_bpf_tail_call_indirect(struct bpf_prog bpf_prog, EMIT3_off32(0x48, 0x8B, 0x85, tcc_ptr_off); / mov rax, qword ptr [rbp - tcc_ptr_off] / EMIT4(0x48, 0x83, 0x38, MAX_TAIL_CALL_CNT); / cmp qword ptr [rax], MAX_TAIL_CALL_CNT / - offset = ctx->tail_call_indirect_label - (prog + 2 - start); + offset = ctx->tail_call_indirect_label - (jit->prog + 2 - start); EMIT2(X86_JAE, offset); / jae out / / prog = array->ptrs[index]; / @@ -798,19 +769,19 @@ static void emit_bpf_tail_call_indirect(struct bpf_prog bpf_prog, / EMIT3(0x48, 0x85, 0xC9); / test rcx,rcx / - offset = ctx->tail_call_indirect_label - (prog + 2 - start); + offset = ctx->tail_call_indirect_label - (jit->prog + 2 - start); EMIT2(X86_JE, offset); / je out / / Inc tail_call_cnt if the slot is populated. / EMIT4(0x48, 0x83, 0x00, 0x01); / add qword ptr [rax], 1 / if (bpf_prog->aux->exception_boundary) { - pop_callee_regs(&prog, all_callee_regs_used); - pop_r12(&prog); + pop_callee_regs(jit, all_callee_regs_used); + pop_r12(jit); } else { - pop_callee_regs(&prog, callee_regs_used); + pop_callee_regs(jit, callee_regs_used); if (bpf_arena_get_kern_vm_start(bpf_prog->aux->arena)) - pop_r12(&prog); + pop_r12(jit); } / Pop tail_call_cnt_ptr. / @@ -833,21 +804,21 @@ static void emit_bpf_tail_call_indirect(struct bpf_prog bpf_prog, * rdi == ctx (1st arg) * rcx == prog->bpf_func + X86_TAIL_CALL_OFFSET / - emit_indirect_jump(&prog, BPF_REG_4 / R4 -> rcx /, ip + (prog - start)); + emit_indirect_jump(jit, BPF_REG_4 / R4 -> rcx /, ip + (jit->prog - start)); / out: / - ctx->tail_call_indirect_label = prog - start; - pprog = prog; + ctx->tail_call_indirect_label = jit->prog - start; } static void emit_bpf_tail_call_direct(struct bpf_prog bpf_prog, struct bpf_jit_poke_descriptor poke, - u8 *pprog, u8 ip, + u8 ip, bool callee_regs_used, u32 stack_depth, struct jit_context ctx) { + struct jit_emit_context jit = &ctx->jit; int tcc_ptr_off = BPF_TAIL_CALL_CNT_PTR_STACK_OFF(stack_depth); - u8 prog = pprog, start = pprog; + u8 start = jit->prog; int offset; / @@ -857,27 +828,27 @@ static void emit_bpf_tail_call_direct(struct bpf_prog bpf_prog, EMIT3_off32(0x48, 0x8B, 0x85, tcc_ptr_off); / mov rax, qword ptr [rbp - tcc_ptr_off] / EMIT4(0x48, 0x83, 0x38, MAX_TAIL_CALL_CNT); / cmp qword ptr [rax], MAX_TAIL_CALL_CNT / - offset = ctx->tail_call_direct_label - (prog + 2 - start); + offset = ctx->tail_call_direct_label - (jit->prog + 2 - start); EMIT2(X86_JAE, offset); / jae out / - poke->tailcall_bypass = ip + (prog - start); + poke->tailcall_bypass = ip + (jit->prog - start); poke->adj_off = X86_TAIL_CALL_OFFSET; poke->tailcall_target = ip + ctx->tail_call_direct_label - X86_PATCH_SIZE; poke->bypass_addr = (u8 )poke->tailcall_target + X86_PATCH_SIZE; - emit_jump(&prog, (u8 )poke->tailcall_target + X86_PATCH_SIZE, + emit_jump(jit, (u8 )poke->tailcall_target + X86_PATCH_SIZE, poke->tailcall_bypass); /* Inc tail_call_cnt if the slot is populated. / EMIT4(0x48, 0x83, 0x00, 0x01); / add qword ptr [rax], 1 / if (bpf_prog->aux->exception_boundary) { - pop_callee_regs(&prog, all_callee_regs_used); - pop_r12(&prog); + pop_callee_regs(jit, all_callee_regs_used); + pop_r12(jit); } else { - pop_callee_regs(&prog, callee_regs_used); + pop_callee_regs(jit, callee_regs_used); if (bpf_arena_get_kern_vm_start(bpf_prog->aux->arena)) - pop_r12(&prog); + pop_r12(jit); } / Pop tail_call_cnt_ptr. / @@ -889,12 +860,10 @@ static void emit_bpf_tail_call_direct(struct bpf_prog bpf_prog, if (stack_depth) EMIT3_off32(0x48, 0x81, 0xC4, round_up(stack_depth, 8)); - emit_nops(&prog, X86_PATCH_SIZE); + emit_nops(jit, X86_PATCH_SIZE); /* out: / - ctx->tail_call_direct_label = prog - start; - - pprog = prog; + ctx->tail_call_direct_label = jit->prog - start; } static void bpf_tail_call_direct_fixup(struct bpf_prog prog) @@ -935,10 +904,9 @@ static void bpf_tail_call_direct_fixup(struct bpf_prog prog) } } -static void emit_mov_imm32(u8 *pprog, bool sign_propagate, +static void emit_mov_imm32(struct jit_emit_context jit, bool sign_propagate, u32 dst_reg, const u32 imm32) { - u8 prog = pprog; u8 b1, b2, b3; /* @@ -972,14 +940,12 @@ static void emit_mov_imm32(u8 *pprog, bool sign_propagate, EMIT1(add_1mod(0x40, dst_reg)); EMIT1_off32(add_1reg(0xB8, dst_reg), imm32); done: - pprog = prog; } -static void emit_mov_imm64(u8 *pprog, u32 dst_reg, +static void emit_mov_imm64(struct jit_emit_context jit, u32 dst_reg, const u32 imm32_hi, const u32 imm32_lo) { u64 imm64 = ((u64)imm32_hi << 32) \| (u32)imm32_lo; - u8 prog = pprog; if (is_uimm32(imm64)) { /* @@ -988,23 +954,19 @@ static void emit_mov_imm64(u8 *pprog, u32 dst_reg, directly, so save couple of bytes by just doing * 'mov %eax, imm32' instead. / - emit_mov_imm32(&prog, false, dst_reg, imm32_lo); + emit_mov_imm32(jit, false, dst_reg, imm32_lo); } else if (is_simm32(imm64)) { - emit_mov_imm32(&prog, true, dst_reg, imm32_lo); + emit_mov_imm32(jit, true, dst_reg, imm32_lo); } else { / movabsq rax, imm64 / EMIT2(add_1mod(0x48, dst_reg), add_1reg(0xB8, dst_reg)); EMIT(imm32_lo, 4); EMIT(imm32_hi, 4); } - - pprog = prog; } -static void emit_mov_reg(u8 *pprog, bool is64, u32 dst_reg, u32 src_reg) +static void emit_mov_reg(struct jit_emit_context jit, bool is64, u32 dst_reg, u32 src_reg) { - u8 prog = pprog; - if (is64) { /* mov dst, src / EMIT_mov(dst_reg, src_reg); @@ -1014,15 +976,11 @@ static void emit_mov_reg(u8 pprog, bool is64, u32 dst_reg, u32 src_reg) EMIT1(add_2mod(0x40, dst_reg, src_reg)); EMIT2(0x89, add_2reg(0xC0, dst_reg, src_reg)); } - - pprog = prog; } -static void emit_movsx_reg(u8 *pprog, int num_bits, bool is64, u32 dst_reg, +static void emit_movsx_reg(struct jit_emit_context jit, int num_bits, bool is64, u32 dst_reg, u32 src_reg) { - u8 prog = pprog; - if (is64) { /* movs[b,w,l]q dst, src / if (num_bits == 8) @@ -1046,15 +1004,11 @@ static void emit_movsx_reg(u8 pprog, int num_bits, bool is64, u32 dst_reg, add_2reg(0xC0, src_reg, dst_reg)); } } - - pprog = prog; } /* Emit the suffix (ModR/M etc) for addressing (ptr_reg + off) and val_reg / -static void emit_insn_suffix(u8 *pprog, u32 ptr_reg, u32 val_reg, int off) +static void emit_insn_suffix(struct jit_emit_context jit, u32 ptr_reg, u32 val_reg, int off) { - u8 prog = pprog; - if (is_imm8(off)) { /* 1-byte signed displacement. * @@ -1067,54 +1021,43 @@ static void emit_insn_suffix(u8 *pprog, u32 ptr_reg, u32 val_reg, int off) / 4-byte signed displacement / EMIT1_off32(add_2reg(0x80, ptr_reg, val_reg), off); } - pprog = prog; } -static void emit_insn_suffix_SIB(u8 *pprog, u32 ptr_reg, u32 val_reg, u32 index_reg, int off) +static void emit_insn_suffix_SIB(struct jit_emit_context jit, u32 ptr_reg, u32 val_reg, + u32 index_reg, int off) { - u8 prog = pprog; - if (is_imm8(off)) { EMIT3(add_2reg(0x44, BPF_REG_0, val_reg), add_2reg(0, ptr_reg, index_reg) /* SIB /, off); } else { EMIT2_off32(add_2reg(0x84, BPF_REG_0, val_reg), add_2reg(0, ptr_reg, index_reg) / SIB /, off); } - pprog = prog; } /* * Emit a REX byte if it will be necessary to address these registers / -static void maybe_emit_mod(u8 pprog, u32 dst_reg, u32 src_reg, bool is64) +static void maybe_emit_mod(struct jit_emit_context jit, u32 dst_reg, u32 src_reg, bool is64) { - u8 prog = pprog; - if (is64) EMIT1(add_2mod(0x48, dst_reg, src_reg)); else if (is_ereg(dst_reg) \|\| is_ereg(src_reg)) EMIT1(add_2mod(0x40, dst_reg, src_reg)); - pprog = prog; } / * Similar version of maybe_emit_mod() for a single register / -static void maybe_emit_1mod(u8 pprog, u32 reg, bool is64) +static void maybe_emit_1mod(struct jit_emit_context jit, u32 reg, bool is64) { - u8 prog = pprog; - if (is64) EMIT1(add_1mod(0x48, reg)); else if (is_ereg(reg)) EMIT1(add_1mod(0x40, reg)); - pprog = prog; } / LDX: dst_reg = (u8)(src_reg + off) / -static void emit_ldx(u8 pprog, u32 size, u32 dst_reg, u32 src_reg, int off) +static void emit_ldx(struct jit_emit_context jit, u32 size, u32 dst_reg, u32 src_reg, int off) { - u8 prog = pprog; - switch (size) { case BPF_B: /* Emit 'movzx rax, byte ptr [rax + off]' / @@ -1136,15 +1079,12 @@ static void emit_ldx(u8 pprog, u32 size, u32 dst_reg, u32 src_reg, int off) EMIT2(add_2mod(0x48, src_reg, dst_reg), 0x8B); break; } - emit_insn_suffix(&prog, src_reg, dst_reg, off); - pprog = prog; + emit_insn_suffix(jit, src_reg, dst_reg, off); } /* LDSX: dst_reg = (s8)(src_reg + off) / -static void emit_ldsx(u8 pprog, u32 size, u32 dst_reg, u32 src_reg, int off) +static void emit_ldsx(struct jit_emit_context jit, u32 size, u32 dst_reg, u32 src_reg, int off) { - u8 prog = pprog; - switch (size) { case BPF_B: /* Emit 'movsx rax, byte ptr [rax + off]' / @@ -1159,14 +1099,12 @@ static void emit_ldsx(u8 pprog, u32 size, u32 dst_reg, u32 src_reg, int off) EMIT2(add_2mod(0x48, src_reg, dst_reg), 0x63); break; } - emit_insn_suffix(&prog, src_reg, dst_reg, off); - pprog = prog; + emit_insn_suffix(jit, src_reg, dst_reg, off); } -static void emit_ldx_index(u8 *pprog, u32 size, u32 dst_reg, u32 src_reg, u32 index_reg, int off) +static void emit_ldx_index(struct jit_emit_context jit, u32 size, u32 dst_reg, u32 src_reg, + u32 index_reg, int off) { - u8 prog = pprog; - switch (size) { case BPF_B: /* movzx rax, byte ptr [rax + r12 + off] / @@ -1185,14 +1123,12 @@ static void emit_ldx_index(u8 pprog, u32 size, u32 dst_reg, u32 src_reg, u32 i EMIT2(add_3mod(0x48, src_reg, dst_reg, index_reg), 0x8B); break; } - emit_insn_suffix_SIB(&prog, src_reg, dst_reg, index_reg, off); - pprog = prog; + emit_insn_suffix_SIB(jit, src_reg, dst_reg, index_reg, off); } -static void emit_ldsx_index(u8 *pprog, u32 size, u32 dst_reg, u32 src_reg, u32 index_reg, int off) +static void emit_ldsx_index(struct jit_emit_context jit, u32 size, u32 dst_reg, u32 src_reg, + u32 index_reg, int off) { - u8 prog = pprog; - switch (size) { case BPF_B: /* movsx rax, byte ptr [rax + r12 + off] / @@ -1207,25 +1143,22 @@ static void emit_ldsx_index(u8 pprog, u32 size, u32 dst_reg, u32 src_reg, u32 EMIT2(add_3mod(0x48, src_reg, dst_reg, index_reg), 0x63); break; } - emit_insn_suffix_SIB(&prog, src_reg, dst_reg, index_reg, off); - pprog = prog; + emit_insn_suffix_SIB(jit, src_reg, dst_reg, index_reg, off); } -static void emit_ldx_r12(u8 *pprog, u32 size, u32 dst_reg, u32 src_reg, int off) +static void emit_ldx_r12(struct jit_emit_context jit, u32 size, u32 dst_reg, u32 src_reg, int off) { - emit_ldx_index(pprog, size, dst_reg, src_reg, X86_REG_R12, off); + emit_ldx_index(jit, size, dst_reg, src_reg, X86_REG_R12, off); } -static void emit_ldsx_r12(u8 *prog, u32 size, u32 dst_reg, u32 src_reg, int off) +static void emit_ldsx_r12(struct jit_emit_context jit, u32 size, u32 dst_reg, u32 src_reg, int off) { - emit_ldsx_index(prog, size, dst_reg, src_reg, X86_REG_R12, off); + emit_ldsx_index(jit, size, dst_reg, src_reg, X86_REG_R12, off); } /* STX: (u8)(dst_reg + off) = src_reg / -static void emit_stx(u8 pprog, u32 size, u32 dst_reg, u32 src_reg, int off) +static void emit_stx(struct jit_emit_context jit, u32 size, u32 dst_reg, u32 src_reg, int off) { - u8 prog = pprog; - switch (size) { case BPF_B: /* Emit 'mov byte ptr [rax + off], al' / @@ -1251,15 +1184,13 @@ static void emit_stx(u8 pprog, u32 size, u32 dst_reg, u32 src_reg, int off) EMIT2(add_2mod(0x48, dst_reg, src_reg), 0x89); break; } - emit_insn_suffix(&prog, dst_reg, src_reg, off); - pprog = prog; + emit_insn_suffix(jit, dst_reg, src_reg, off); } /* STX: (u8)(dst_reg + index_reg + off) = src_reg / -static void emit_stx_index(u8 pprog, u32 size, u32 dst_reg, u32 src_reg, u32 index_reg, int off) +static void emit_stx_index(struct jit_emit_context jit, u32 size, u32 dst_reg, u32 src_reg, + u32 index_reg, int off) { - u8 prog = pprog; - switch (size) { case BPF_B: /* mov byte ptr [rax + r12 + off], al / @@ -1278,20 +1209,18 @@ static void emit_stx_index(u8 pprog, u32 size, u32 dst_reg, u32 src_reg, u32 i EMIT2(add_3mod(0x48, dst_reg, src_reg, index_reg), 0x89); break; } - emit_insn_suffix_SIB(&prog, dst_reg, src_reg, index_reg, off); - pprog = prog; + emit_insn_suffix_SIB(jit, dst_reg, src_reg, index_reg, off); } -static void emit_stx_r12(u8 *pprog, u32 size, u32 dst_reg, u32 src_reg, int off) +static void emit_stx_r12(struct jit_emit_context jit, u32 size, u32 dst_reg, u32 src_reg, int off) { - emit_stx_index(pprog, size, dst_reg, src_reg, X86_REG_R12, off); + emit_stx_index(jit, size, dst_reg, src_reg, X86_REG_R12, off); } /* ST: (u8)(dst_reg + index_reg + off) = imm32 / -static void emit_st_index(u8 pprog, u32 size, u32 dst_reg, u32 index_reg, int off, int imm) +static void emit_st_index(struct jit_emit_context jit, u32 size, u32 dst_reg, u32 index_reg, + int off, int imm) { - u8 prog = pprog; - switch (size) { case BPF_B: /* mov byte ptr [rax + r12 + off], imm8 / @@ -1310,35 +1239,32 @@ static void emit_st_index(u8 pprog, u32 size, u32 dst_reg, u32 index_reg, int EMIT2(add_3mod(0x48, dst_reg, 0, index_reg), 0xC7); break; } - emit_insn_suffix_SIB(&prog, dst_reg, 0, index_reg, off); + emit_insn_suffix_SIB(jit, dst_reg, 0, index_reg, off); EMIT(imm, bpf_size_to_x86_bytes(size)); - pprog = prog; } -static void emit_st_r12(u8 *pprog, u32 size, u32 dst_reg, int off, int imm) +static void emit_st_r12(struct jit_emit_context jit, u32 size, u32 dst_reg, int off, int imm) { - emit_st_index(pprog, size, dst_reg, X86_REG_R12, off, imm); + emit_st_index(jit, size, dst_reg, X86_REG_R12, off, imm); } -static void emit_store_stack_imm64(u8 *pprog, int reg, int stack_off, u64 imm64) +static void emit_store_stack_imm64(struct jit_emit_context jit, int reg, int stack_off, u64 imm64) { /* * mov reg, imm64 * mov QWORD PTR [rbp + stack_off], reg / - emit_mov_imm64(pprog, reg, imm64 >> 32, (u32) imm64); - emit_stx(pprog, BPF_DW, BPF_REG_FP, reg, stack_off); + emit_mov_imm64(jit, reg, imm64 >> 32, (u32) imm64); + emit_stx(jit, BPF_DW, BPF_REG_FP, reg, stack_off); } -static int emit_atomic_rmw(u8 pprog, u32 atomic_op, +static int emit_atomic_rmw(struct jit_emit_context jit, u32 atomic_op, u32 dst_reg, u32 src_reg, s16 off, u8 bpf_size) { - u8 prog = pprog; - if (atomic_op != BPF_XCHG) EMIT1(0xF0); /* lock prefix / - maybe_emit_mod(&prog, dst_reg, src_reg, bpf_size == BPF_DW); + maybe_emit_mod(jit, dst_reg, src_reg, bpf_size == BPF_DW); / emit opcode / switch (atomic_op) { @@ -1366,18 +1292,15 @@ static int emit_atomic_rmw(u8 pprog, u32 atomic_op, return -EFAULT; } - emit_insn_suffix(&prog, dst_reg, src_reg, off); + emit_insn_suffix(jit, dst_reg, src_reg, off); - pprog = prog; return 0; } -static int emit_atomic_rmw_index(u8 *pprog, u32 atomic_op, u32 size, +static int emit_atomic_rmw_index(struct jit_emit_context jit, u32 atomic_op, u32 size, u32 dst_reg, u32 src_reg, u32 index_reg, int off) { - u8 prog = pprog; - if (atomic_op != BPF_XCHG) EMIT1(0xF0); /* lock prefix / @@ -1418,22 +1341,21 @@ static int emit_atomic_rmw_index(u8 pprog, u32 atomic_op, u32 size, pr_err("bpf_jit: unknown atomic opcode %02x\n", atomic_op); return -EFAULT; } - emit_insn_suffix_SIB(&prog, dst_reg, src_reg, index_reg, off); - pprog = prog; + emit_insn_suffix_SIB(jit, dst_reg, src_reg, index_reg, off); return 0; } -static int emit_atomic_ld_st(u8 *pprog, u32 atomic_op, u32 dst_reg, +static int emit_atomic_ld_st(struct jit_emit_context jit, u32 atomic_op, u32 dst_reg, u32 src_reg, s16 off, u8 bpf_size) { switch (atomic_op) { case BPF_LOAD_ACQ: /* dst_reg = smp_load_acquire(src_reg + off16) / - emit_ldx(pprog, bpf_size, dst_reg, src_reg, off); + emit_ldx(jit, bpf_size, dst_reg, src_reg, off); break; case BPF_STORE_REL: / smp_store_release(dst_reg + off16, src_reg) / - emit_stx(pprog, bpf_size, dst_reg, src_reg, off); + emit_stx(jit, bpf_size, dst_reg, src_reg, off); break; default: pr_err("bpf_jit: unknown atomic load/store opcode %02x\n", @@ -1444,18 +1366,18 @@ static int emit_atomic_ld_st(u8 pprog, u32 atomic_op, u32 dst_reg, return 0; } -static int emit_atomic_ld_st_index(u8 pprog, u32 atomic_op, u32 size, +static int emit_atomic_ld_st_index(struct jit_emit_context jit, u32 atomic_op, u32 size, u32 dst_reg, u32 src_reg, u32 index_reg, int off) { switch (atomic_op) { case BPF_LOAD_ACQ: /* dst_reg = smp_load_acquire(src_reg + idx_reg + off16) / - emit_ldx_index(pprog, size, dst_reg, src_reg, index_reg, off); + emit_ldx_index(jit, size, dst_reg, src_reg, index_reg, off); break; case BPF_STORE_REL: / smp_store_release(dst_reg + idx_reg + off16, src_reg) / - emit_stx_index(pprog, size, dst_reg, src_reg, index_reg, off); + emit_stx_index(jit, size, dst_reg, src_reg, index_reg, off); break; default: pr_err("bpf_jit: unknown atomic load/store opcode %02x\n", @@ -1562,10 +1484,9 @@ static void detect_reg_usage(struct bpf_insn insn, int insn_cnt, * l: vector length (128 bit or 256 bit) or reserved * pp: opcode prefix (none, 0x66, 0xf2 or 0xf3) / -static void emit_3vex(u8 pprog, bool r, bool x, bool b, u8 m, +static void emit_3vex(struct jit_emit_context jit, bool r, bool x, bool b, u8 m, bool w, u8 src_reg2, bool l, u8 pp) { - u8 prog = pprog; const u8 b0 = 0xc4; /* first byte of 3-byte VEX prefix / u8 b1, b2; u8 vvvv = reg2hex[src_reg2]; @@ -1595,27 +1516,22 @@ static void emit_3vex(u8 pprog, bool r, bool x, bool b, u8 m, b2 = (w << 7) \| ((~vvvv & 0xf) << 3) \| (l << 2) \| (pp & 3); EMIT3(b0, b1, b2); - pprog = prog; } /* emit BMI2 shift instruction / -static void emit_shiftx(u8 pprog, u32 dst_reg, u8 src_reg, bool is64, u8 op) +static void emit_shiftx(struct jit_emit_context jit, u32 dst_reg, u8 src_reg, bool is64, u8 op) { - u8 prog = pprog; bool r = is_ereg(dst_reg); u8 m = 2; /* escape code 0f38 / - emit_3vex(&prog, r, false, r, m, is64, src_reg, false, op); + emit_3vex(jit, r, false, r, m, is64, src_reg, false, op); EMIT2(0xf7, add_2reg(0xC0, dst_reg, dst_reg)); - pprog = prog; } -static void emit_priv_frame_ptr(u8 *pprog, void __percpu priv_frame_ptr) +static void emit_priv_frame_ptr(struct jit_emit_context jit, void __percpu priv_frame_ptr) { - u8 prog = pprog; - /* movabs r9, priv_frame_ptr / - emit_mov_imm64(&prog, X86_REG_R9, (__force long) priv_frame_ptr >> 32, + emit_mov_imm64(jit, X86_REG_R9, (__force long) priv_frame_ptr >> 32, (u32) (__force long) priv_frame_ptr); #ifdef CONFIG_SMP @@ -1624,11 +1540,9 @@ static void emit_priv_frame_ptr(u8 pprog, void __percpu priv_frame_ptr) EMIT3(0x03, 0x0c, 0x25); EMIT((u32)(unsigned long)&this_cpu_off, 4); #endif - - pprog = prog; } -#define INSN_SZ_DIFF (((addrs[i] - addrs[i - 1]) - (prog - temp))) +#define INSN_SZ_DIFF (((addrs[i] - addrs[i - 1]) - (jit->prog - temp))) #define __LOAD_TCC_PTR(off) \ EMIT3_off32(0x48, 0x8B, 0x85, off) @@ -1640,10 +1554,9 @@ static void emit_priv_frame_ptr(u8 pprog, void __percpu priv_frame_ptr) #define PRIV_STACK_GUARD_SZ 8 #define PRIV_STACK_GUARD_VAL 0xEB9F12345678eb9fULL -static int emit_spectre_bhb_barrier(u8 *pprog, u8 ip, +static int emit_spectre_bhb_barrier(struct jit_emit_context jit, u8 ip, struct bpf_prog bpf_prog) { - u8 prog = pprog; u8 func; if (cpu_feature_enabled(X86_FEATURE_CLEAR_BHB_LOOP)) { @@ -1653,9 +1566,9 @@ static int emit_spectre_bhb_barrier(u8 *pprog, u8 ip, ip += 2; func = (u8 )clear_bhb_loop; - ip += x86_call_depth_emit_accounting(&prog, func, ip); + ip += bpf_call_depth_emit_accounting(jit, func, ip); - if (emit_call(&prog, func, ip)) + if (emit_call(jit, func, ip)) return -EINVAL; EMIT1(0x59); / pop rcx / EMIT1(0x58); / pop rax / @@ -1678,7 +1591,6 @@ static int emit_spectre_bhb_barrier(u8 pprog, u8 ip, / EMIT5(0xF3, 0x48, 0x0F, 0x1E, 0xF8); / ibhf / } - pprog = prog; return 0; } @@ -1689,10 +1601,9 @@ static int emit_spectre_bhb_barrier(u8 *pprog, u8 ip, * arena NULL is offset 0. Return the number of emitted bytes. / static int emit_kfunc_arena_args(struct bpf_prog bpf_prog, - const struct btf_func_model fm, u8 pprog) + const struct btf_func_model fm, struct jit_emit_context jit) { - u8 prog = pprog; - u8 start = prog; + u8 start = jit->prog; int i; for (i = 0; i < min_t(int, fm->nr_args, MAX_BPF_FUNC_REG_ARGS); i++) { @@ -1705,20 +1616,19 @@ static int emit_kfunc_arena_args(struct bpf_prog bpf_prog, return -EINVAL; /* mov eN, eN: truncate and clear the upper 32 bits / - emit_mov_reg(&prog, false, reg, reg); + emit_mov_reg(jit, false, reg, reg); if (flags & BTF_FMODEL_NULLABLE_ARG) { / test eN, eN; jz over the 3-byte add / - maybe_emit_mod(&prog, reg, reg, false); + maybe_emit_mod(jit, reg, reg, false); EMIT2(0x85, add_2reg(0xC0, reg, reg)); EMIT2(X86_JE, 3); } / add rN, r12 / - maybe_emit_mod(&prog, reg, X86_REG_R12, true); + maybe_emit_mod(jit, reg, X86_REG_R12, true); EMIT2(0x01, add_2reg(0xC0, reg, X86_REG_R12)); } - pprog = prog; - return prog - start; + return jit->prog - start; } static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int addrs, u8 image, @@ -1726,6 +1636,7 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int * { bool tail_call_reachable = bpf_prog->aux->tail_call_reachable; struct bpf_insn insn = bpf_prog->insnsi; + struct jit_emit_context jit = &ctx->jit; bool callee_regs_used[4] = {}; int insn_cnt = bpf_prog->len; bool seen_exit = false; @@ -1736,12 +1647,15 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int * void __percpu priv_stack_ptr; int i, excnt = 0; int ilen, proglen = 0; - u8 ip, prog = temp; + u8 ip; u32 stack_depth; int callee_saved_size; s32 outgoing_arg_base; int err; + jit->prog = temp; + jit->dry_run = false; + stack_depth = bpf_prog->aux->stack_depth; out_stack_arg_cnt = bpf_out_stack_arg_cnt(env, bpf_prog); priv_stack_ptr = bpf_prog->aux->priv_stack_ptr; @@ -1777,11 +1691,11 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int * detect_reg_usage(insn, insn_cnt, callee_regs_used); - emit_prologue(&prog, image, stack_depth, + emit_prologue(jit, image, stack_depth, bpf_prog_was_classic(bpf_prog), tail_call_reachable, bpf_is_subprog(bpf_prog), bpf_prog->aux->exception_cb); - bpf_prog->aux->ksym.fp_start = prog - temp; + bpf_prog->aux->ksym.fp_start = jit->prog - temp; /* Exception callback will clobber callee regs for its own use, and * restore the original callee regs from main prog's stack frame. @@ -1791,12 +1705,12 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int * * register, as we throw after entry into the kernel, which may * overwrite r12. / - push_r12(&prog); - push_callee_regs(&prog, all_callee_regs_used); + push_r12(jit); + push_callee_regs(jit, all_callee_regs_used); } else { if (arena_vm_start) - push_r12(&prog); - push_callee_regs(&prog, callee_regs_used); + push_r12(jit); + push_callee_regs(jit, callee_regs_used); } / Compute callee-saved register area size. / @@ -1834,21 +1748,21 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int outgoing_rsp = out_stack_arg_cnt > 1 ? (out_stack_arg_cnt - 1) * 8 : 0; if (bpf_prog->aux->exception_boundary) bpf_prog->aux->stack_arg_sp_adjust = outgoing_rsp; - emit_sub_rsp(&prog, outgoing_rsp); + emit_sub_rsp(jit, outgoing_rsp); if (arena_vm_start) - emit_mov_imm64(&prog, X86_REG_R12, + emit_mov_imm64(jit, X86_REG_R12, arena_vm_start >> 32, (u32) arena_vm_start); if (priv_frame_ptr) - emit_priv_frame_ptr(&prog, priv_frame_ptr); + emit_priv_frame_ptr(jit, priv_frame_ptr); - ilen = prog - temp; + ilen = jit->prog - temp; if (rw_image) memcpy(rw_image + proglen, temp, ilen); proglen += ilen; addrs[0] = proglen; - prog = temp; + jit->prog = temp; for (i = 1; i <= insn_cnt; i++, insn++) { const s32 imm32 = insn->imm; @@ -1873,7 +1787,7 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int * if (bpf_insn_is_indirect_target(env, bpf_prog, i - 1)) EMIT_ENDBR(); - ip = image + addrs[i - 1] + (prog - temp); + ip = image + addrs[i - 1] + (jit->prog - temp); switch (insn->code) { /* ALU / @@ -1887,7 +1801,7 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int case BPF_ALU64 \| BPF_AND \| BPF_X: case BPF_ALU64 \| BPF_OR \| BPF_X: case BPF_ALU64 \| BPF_XOR \| BPF_X: - maybe_emit_mod(&prog, dst_reg, src_reg, + maybe_emit_mod(jit, dst_reg, src_reg, BPF_CLASS(insn->code) == BPF_ALU64); b2 = simple_alu_opcodes[BPF_OP(insn->code)]; EMIT2(b2, add_2reg(0xC0, dst_reg, src_reg)); @@ -1897,32 +1811,32 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int * if (insn_is_cast_user(insn)) { if (dst_reg != src_reg) /* 32-bit mov / - emit_mov_reg(&prog, false, dst_reg, src_reg); + emit_mov_reg(jit, false, dst_reg, src_reg); / shl dst_reg, 32 / - maybe_emit_1mod(&prog, dst_reg, true); + maybe_emit_1mod(jit, dst_reg, true); EMIT3(0xC1, add_1reg(0xE0, dst_reg), 32); / or dst_reg, user_vm_start / - maybe_emit_1mod(&prog, dst_reg, true); + maybe_emit_1mod(jit, dst_reg, true); if (is_axreg(dst_reg)) EMIT1_off32(0x0D, user_vm_start >> 32); else EMIT2_off32(0x81, add_1reg(0xC8, dst_reg), user_vm_start >> 32); / rol dst_reg, 32 / - maybe_emit_1mod(&prog, dst_reg, true); + maybe_emit_1mod(jit, dst_reg, true); EMIT3(0xC1, add_1reg(0xC0, dst_reg), 32); / xor r11, r11 / EMIT3(0x4D, 0x31, 0xDB); / test dst_reg32, dst_reg32; check if lower 32-bit are zero / - maybe_emit_mod(&prog, dst_reg, dst_reg, false); + maybe_emit_mod(jit, dst_reg, dst_reg, false); EMIT2(0x85, add_2reg(0xC0, dst_reg, dst_reg)); / cmove r11, dst_reg; if so, set dst_reg to zero / / WARNING: Intel swapped src/dst register encoding in CMOVcc !!! / - maybe_emit_mod(&prog, AUX_REG, dst_reg, true); + maybe_emit_mod(jit, AUX_REG, dst_reg, true); EMIT3(0x0F, 0x44, add_2reg(0xC0, AUX_REG, dst_reg)); break; } else if (insn_is_mov_percpu_addr(insn)) { @@ -1939,11 +1853,11 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int fallthrough; case BPF_ALU \| BPF_MOV \| BPF_X: if (insn->off == 0) - emit_mov_reg(&prog, + emit_mov_reg(jit, BPF_CLASS(insn->code) == BPF_ALU64, dst_reg, src_reg); else - emit_movsx_reg(&prog, insn->off, + emit_movsx_reg(jit, insn->off, BPF_CLASS(insn->code) == BPF_ALU64, dst_reg, src_reg); break; @@ -1951,7 +1865,7 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int * /* neg dst / case BPF_ALU \| BPF_NEG: case BPF_ALU64 \| BPF_NEG: - maybe_emit_1mod(&prog, dst_reg, + maybe_emit_1mod(jit, dst_reg, BPF_CLASS(insn->code) == BPF_ALU64); EMIT2(0xF7, add_1reg(0xD8, dst_reg)); break; @@ -1966,7 +1880,7 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int case BPF_ALU64 \| BPF_AND \| BPF_K: case BPF_ALU64 \| BPF_OR \| BPF_K: case BPF_ALU64 \| BPF_XOR \| BPF_K: - maybe_emit_1mod(&prog, dst_reg, + maybe_emit_1mod(jit, dst_reg, BPF_CLASS(insn->code) == BPF_ALU64); /* @@ -2006,12 +1920,12 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int * case BPF_ALU64 \| BPF_MOV \| BPF_K: case BPF_ALU \| BPF_MOV \| BPF_K: - emit_mov_imm32(&prog, BPF_CLASS(insn->code) == BPF_ALU64, + emit_mov_imm32(jit, BPF_CLASS(insn->code) == BPF_ALU64, dst_reg, imm32); break; case BPF_LD \| BPF_IMM \| BPF_DW: - emit_mov_imm64(&prog, dst_reg, insn[1].imm, insn[0].imm); + emit_mov_imm64(jit, dst_reg, insn[1].imm, insn[0].imm); insn++; i++; break; @@ -2047,7 +1961,7 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int * if (dst_reg != BPF_REG_0) /* mov rax, dst_reg / - emit_mov_reg(&prog, is64, BPF_REG_0, dst_reg); + emit_mov_reg(jit, is64, BPF_REG_0, dst_reg); if (insn->off == 0) { / @@ -2057,7 +1971,7 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int * EMIT2(0x31, 0xd2); /* div src_reg / - maybe_emit_1mod(&prog, src_reg, is64); + maybe_emit_1mod(jit, src_reg, is64); EMIT2(0xF7, add_1reg(0xF0, src_reg)); } else { if (BPF_CLASS(insn->code) == BPF_ALU) @@ -2066,18 +1980,18 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int EMIT2(0x48, 0x99); /* cqo / / idiv src_reg / - maybe_emit_1mod(&prog, src_reg, is64); + maybe_emit_1mod(jit, src_reg, is64); EMIT2(0xF7, add_1reg(0xF8, src_reg)); } if (BPF_OP(insn->code) == BPF_MOD && dst_reg != BPF_REG_3) / mov dst_reg, rdx / - emit_mov_reg(&prog, is64, dst_reg, BPF_REG_3); + emit_mov_reg(jit, is64, dst_reg, BPF_REG_3); else if (BPF_OP(insn->code) == BPF_DIV && dst_reg != BPF_REG_0) / mov dst_reg, rax / - emit_mov_reg(&prog, is64, dst_reg, BPF_REG_0); + emit_mov_reg(jit, is64, dst_reg, BPF_REG_0); if (dst_reg != BPF_REG_3) EMIT1(0x5A); / pop rdx / @@ -2088,7 +2002,7 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int case BPF_ALU \| BPF_MUL \| BPF_K: case BPF_ALU64 \| BPF_MUL \| BPF_K: - maybe_emit_mod(&prog, dst_reg, dst_reg, + maybe_emit_mod(jit, dst_reg, dst_reg, BPF_CLASS(insn->code) == BPF_ALU64); if (is_imm8(imm32)) @@ -2104,7 +2018,7 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int * case BPF_ALU \| BPF_MUL \| BPF_X: case BPF_ALU64 \| BPF_MUL \| BPF_X: - maybe_emit_mod(&prog, src_reg, dst_reg, + maybe_emit_mod(jit, src_reg, dst_reg, BPF_CLASS(insn->code) == BPF_ALU64); /* imul dst_reg, src_reg / @@ -2118,7 +2032,7 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int case BPF_ALU64 \| BPF_LSH \| BPF_K: case BPF_ALU64 \| BPF_RSH \| BPF_K: case BPF_ALU64 \| BPF_ARSH \| BPF_K: - maybe_emit_1mod(&prog, dst_reg, + maybe_emit_1mod(jit, dst_reg, BPF_CLASS(insn->code) == BPF_ALU64); b3 = simple_alu_opcodes[BPF_OP(insn->code)]; @@ -2152,7 +2066,7 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int * break; } - emit_shiftx(&prog, dst_reg, src_reg, w, op); + emit_shiftx(jit, dst_reg, src_reg, w, op); break; } @@ -2171,7 +2085,7 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int * } /* shl %rax, %cl \| shr %rax, %cl \| sar %rax, %cl / - maybe_emit_1mod(&prog, dst_reg, + maybe_emit_1mod(jit, dst_reg, BPF_CLASS(insn->code) == BPF_ALU64); b3 = simple_alu_opcodes[BPF_OP(insn->code)]; @@ -2272,7 +2186,7 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int case BPF_ST \| BPF_MEM \| BPF_DW: if (dst_reg == BPF_REG_PARAMS && insn->off == -8) { /* Arg 6: store immediate in r9 register / - emit_mov_imm64(&prog, X86_REG_R9, imm32 >> 31, (u32)imm32); + emit_mov_imm64(jit, X86_REG_R9, imm32 >> 31, (u32)imm32); break; } EMIT2(add_1mod(0x48, dst_reg), 0xC7); @@ -2310,15 +2224,15 @@ st: insn_off = insn->off; insn_off = outgoing_arg_base - outgoing_rsp - insn_off - 16; dst_reg = BPF_REG_FP; } - emit_stx(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn_off); + emit_stx(jit, BPF_SIZE(insn->code), dst_reg, src_reg, insn_off); break; case BPF_ST \| BPF_PROBE_MEM32 \| BPF_B: case BPF_ST \| BPF_PROBE_MEM32 \| BPF_H: case BPF_ST \| BPF_PROBE_MEM32 \| BPF_W: case BPF_ST \| BPF_PROBE_MEM32 \| BPF_DW: - start_of_ldx = prog; - emit_st_r12(&prog, BPF_SIZE(insn->code), dst_reg, insn->off, insn->imm); + start_of_ldx = jit->prog; + emit_st_r12(jit, BPF_SIZE(insn->code), dst_reg, insn->off, insn->imm); goto populate_extable; / LDX: dst_reg = (u8)(src_reg + r12 + off) / @@ -2333,14 +2247,14 @@ st: insn_off = insn->off; case BPF_STX \| BPF_PROBE_MEM32 \| BPF_H: case BPF_STX \| BPF_PROBE_MEM32 \| BPF_W: case BPF_STX \| BPF_PROBE_MEM32 \| BPF_DW: - start_of_ldx = prog; + start_of_ldx = jit->prog; if (BPF_CLASS(insn->code) == BPF_LDX) { if (BPF_MODE(insn->code) == BPF_PROBE_MEM32SX) - emit_ldsx_r12(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn->off); + emit_ldsx_r12(jit, BPF_SIZE(insn->code), dst_reg, src_reg, insn->off); else - emit_ldx_r12(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn->off); + emit_ldx_r12(jit, BPF_SIZE(insn->code), dst_reg, src_reg, insn->off); } else { - emit_stx_r12(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn->off); + emit_stx_r12(jit, BPF_SIZE(insn->code), dst_reg, src_reg, insn->off); } populate_extable: { @@ -2401,7 +2315,7 @@ st: insn_off = insn->off; is_write = true; } - ex->fixup = FIELD_PREP(FIXUP_INSN_LEN_MASK, prog - start_of_ldx) \| + ex->fixup = FIELD_PREP(FIXUP_INSN_LEN_MASK, jit->prog - start_of_ldx) \| FIELD_PREP(FIXUP_ARENA_REG_MASK, arena_reg) \| FIELD_PREP(FIXUP_REG_MASK, fixup_reg); ex->fixup \|= FIXUP_ARENA_ACCESS; @@ -2455,7 +2369,7 @@ st: insn_off = insn->off; u8 end_of_jmp; /* movabsq r10, VSYSCALL_ADDR / - emit_mov_imm64(&prog, BPF_REG_AX, (long)VSYSCALL_ADDR >> 32, + emit_mov_imm64(jit, BPF_REG_AX, (long)VSYSCALL_ADDR >> 32, (u32)(long)VSYSCALL_ADDR); / mov src_reg, r11 / @@ -2463,40 +2377,40 @@ st: insn_off = insn->off; if (insn->off) { / add r11, insn->off / - maybe_emit_1mod(&prog, AUX_REG, true); + maybe_emit_1mod(jit, AUX_REG, true); EMIT2_off32(0x81, add_1reg(0xC0, AUX_REG), insn->off); } / sub r11, r10 / - maybe_emit_mod(&prog, AUX_REG, BPF_REG_AX, true); + maybe_emit_mod(jit, AUX_REG, BPF_REG_AX, true); EMIT2(0x29, add_2reg(0xC0, AUX_REG, BPF_REG_AX)); / movabsq r10, limit / - emit_mov_imm64(&prog, BPF_REG_AX, (long)limit >> 32, + emit_mov_imm64(jit, BPF_REG_AX, (long)limit >> 32, (u32)(long)limit); / cmp r10, r11 / - maybe_emit_mod(&prog, AUX_REG, BPF_REG_AX, true); + maybe_emit_mod(jit, AUX_REG, BPF_REG_AX, true); EMIT2(0x39, add_2reg(0xC0, AUX_REG, BPF_REG_AX)); / if unsigned '>', goto load / EMIT2(X86_JA, 0); - end_of_jmp = prog; + end_of_jmp = jit->prog; / xor dst_reg, dst_reg / - emit_mov_imm32(&prog, false, dst_reg, 0); + emit_mov_imm32(jit, false, dst_reg, 0); / jmp byte_after_ldx / EMIT2(0xEB, 0); / populate jmp_offset for JAE above to jump to start_of_ldx / - start_of_ldx = prog; + start_of_ldx = jit->prog; end_of_jmp[-1] = start_of_ldx - end_of_jmp; } if (BPF_MODE(insn->code) == BPF_PROBE_MEMSX \|\| BPF_MODE(insn->code) == BPF_MEMSX) - emit_ldsx(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn_off); + emit_ldsx(jit, BPF_SIZE(insn->code), dst_reg, src_reg, insn_off); else - emit_ldx(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn_off); + emit_ldx(jit, BPF_SIZE(insn->code), dst_reg, src_reg, insn_off); if (BPF_MODE(insn->code) == BPF_PROBE_MEM \|\| BPF_MODE(insn->code) == BPF_PROBE_MEMSX) { struct exception_table_entry ex; @@ -2504,7 +2418,7 @@ st: insn_off = insn->off; s64 delta; /* populate jmp_offset for JMP above / - start_of_ldx[-1] = prog - start_of_ldx; + start_of_ldx[-1] = jit->prog - start_of_ldx; if (!bpf_prog->aux->extable) break; @@ -2539,7 +2453,7 @@ st: insn_off = insn->off; End result: x86 insn "mov rbx, qword ptr [rax+0x14]" * of 4 bytes will be ignored and rbx will be zero inited. / - ex->fixup = FIELD_PREP(FIXUP_INSN_LEN_MASK, prog - start_of_ldx) \| + ex->fixup = FIELD_PREP(FIXUP_INSN_LEN_MASK, jit->prog - start_of_ldx) \| FIELD_PREP(FIXUP_REG_MASK, reg2pt_regs[dst_reg]); } break; @@ -2567,26 +2481,26 @@ st: insn_off = insn->off; / /* Will need RAX as a CMPXCHG operand so save R0 / - emit_mov_reg(&prog, true, BPF_REG_AX, BPF_REG_0); + emit_mov_reg(jit, true, BPF_REG_AX, BPF_REG_0); if (src_reg == BPF_REG_0) real_src_reg = BPF_REG_AX; if (dst_reg == BPF_REG_0) real_dst_reg = BPF_REG_AX; - branch_target = prog; + branch_target = jit->prog; / Load old value / - emit_ldx(&prog, BPF_SIZE(insn->code), + emit_ldx(jit, BPF_SIZE(insn->code), BPF_REG_0, real_dst_reg, insn->off); / * Perform the (commutative) operation locally, * put the result in the AUX_REG. / - emit_mov_reg(&prog, is64, AUX_REG, BPF_REG_0); - maybe_emit_mod(&prog, AUX_REG, real_src_reg, is64); + emit_mov_reg(jit, is64, AUX_REG, BPF_REG_0); + maybe_emit_mod(jit, AUX_REG, real_src_reg, is64); EMIT2(simple_alu_opcodes[BPF_OP(insn->imm)], add_2reg(0xC0, AUX_REG, real_src_reg)); / Attempt to swap in new value / - err = emit_atomic_rmw(&prog, BPF_CMPXCHG, + err = emit_atomic_rmw(jit, BPF_CMPXCHG, real_dst_reg, AUX_REG, insn->off, BPF_SIZE(insn->code)); @@ -2596,19 +2510,19 @@ st: insn_off = insn->off; ZF tells us whether we won the race. If it's * cleared we need to try again. / - EMIT2(X86_JNE, -(prog - branch_target) - 2); + EMIT2(X86_JNE, -(jit->prog - branch_target) - 2); / Return the pre-modification value / - emit_mov_reg(&prog, is64, real_src_reg, BPF_REG_0); + emit_mov_reg(jit, is64, real_src_reg, BPF_REG_0); / Restore R0 after clobbering RAX / - emit_mov_reg(&prog, true, BPF_REG_0, BPF_REG_AX); + emit_mov_reg(jit, true, BPF_REG_0, BPF_REG_AX); break; } if (bpf_atomic_is_load_store(insn)) - err = emit_atomic_ld_st(&prog, insn->imm, dst_reg, src_reg, + err = emit_atomic_ld_st(jit, insn->imm, dst_reg, src_reg, insn->off, BPF_SIZE(insn->code)); else - err = emit_atomic_rmw(&prog, insn->imm, dst_reg, src_reg, + err = emit_atomic_rmw(jit, insn->imm, dst_reg, src_reg, insn->off, BPF_SIZE(insn->code)); if (err) return err; @@ -2623,14 +2537,14 @@ st: insn_off = insn->off; fallthrough; case BPF_STX \| BPF_PROBE_ATOMIC \| BPF_W: case BPF_STX \| BPF_PROBE_ATOMIC \| BPF_DW: - start_of_ldx = prog; + start_of_ldx = jit->prog; if (bpf_atomic_is_load_store(insn)) - err = emit_atomic_ld_st_index(&prog, insn->imm, + err = emit_atomic_ld_st_index(jit, insn->imm, BPF_SIZE(insn->code), dst_reg, src_reg, X86_REG_R12, insn->off); else - err = emit_atomic_rmw_index(&prog, insn->imm, BPF_SIZE(insn->code), + err = emit_atomic_rmw_index(jit, insn->imm, BPF_SIZE(insn->code), dst_reg, src_reg, X86_REG_R12, insn->off); if (err) @@ -2652,20 +2566,20 @@ st: insn_off = insn->off; fm = bpf_jit_find_kfunc_model(bpf_prog, insn); if (!fm) return -EINVAL; - err = emit_kfunc_arena_args(bpf_prog, fm, &prog); + err = emit_kfunc_arena_args(bpf_prog, fm, jit); if (err < 0) return err; ip += err; } if (priv_frame_ptr) { - push_r9(&prog); + push_r9(jit); ip += 2; } - ip += x86_call_depth_emit_accounting(&prog, func, ip); - if (emit_call(&prog, func, ip)) + ip += bpf_call_depth_emit_accounting(jit, func, ip); + if (emit_call(jit, func, ip)) return -EINVAL; if (priv_frame_ptr) - pop_r9(&prog); + pop_r9(jit); / * A kfunc returning more than 8 bytes hands the second * half back in RDX (the native ABI's second return reg), @@ -2673,7 +2587,7 @@ st: insn_off = insn->off; * needed), while BPF R2 is RSI, so copy RDX into RSI. / if (fm && fm->ret_size > 8) - emit_mov_reg(&prog, true, BPF_REG_2, BPF_REG_3); + emit_mov_reg(jit, true, BPF_REG_2, BPF_REG_3); break; } @@ -2681,14 +2595,12 @@ st: insn_off = insn->off; if (imm32) emit_bpf_tail_call_direct(bpf_prog, &bpf_prog->aux->poke_tab[imm32 - 1], - &prog, ip, callee_regs_used, stack_depth, ctx); else emit_bpf_tail_call_indirect(bpf_prog, - &prog, callee_regs_used, stack_depth, ip, @@ -2717,7 +2629,7 @@ st: insn_off = insn->off; case BPF_JMP32 \| BPF_JSGE \| BPF_X: case BPF_JMP32 \| BPF_JSLE \| BPF_X: / cmp dst_reg, src_reg / - maybe_emit_mod(&prog, dst_reg, src_reg, + maybe_emit_mod(jit, dst_reg, src_reg, BPF_CLASS(insn->code) == BPF_JMP); EMIT2(0x39, add_2reg(0xC0, dst_reg, src_reg)); goto emit_cond_jmp; @@ -2725,7 +2637,7 @@ st: insn_off = insn->off; case BPF_JMP \| BPF_JSET \| BPF_X: case BPF_JMP32 \| BPF_JSET \| BPF_X: / test dst_reg, src_reg / - maybe_emit_mod(&prog, dst_reg, src_reg, + maybe_emit_mod(jit, dst_reg, src_reg, BPF_CLASS(insn->code) == BPF_JMP); EMIT2(0x85, add_2reg(0xC0, dst_reg, src_reg)); goto emit_cond_jmp; @@ -2733,7 +2645,7 @@ st: insn_off = insn->off; case BPF_JMP \| BPF_JSET \| BPF_K: case BPF_JMP32 \| BPF_JSET \| BPF_K: / test dst_reg, imm32 / - maybe_emit_1mod(&prog, dst_reg, + maybe_emit_1mod(jit, dst_reg, BPF_CLASS(insn->code) == BPF_JMP); EMIT2_off32(0xF7, add_1reg(0xC0, dst_reg), imm32); goto emit_cond_jmp; @@ -2760,14 +2672,14 @@ st: insn_off = insn->off; case BPF_JMP32 \| BPF_JSLE \| BPF_K: / test dst_reg, dst_reg to save one extra byte / if (imm32 == 0) { - maybe_emit_mod(&prog, dst_reg, dst_reg, + maybe_emit_mod(jit, dst_reg, dst_reg, BPF_CLASS(insn->code) == BPF_JMP); EMIT2(0x85, add_2reg(0xC0, dst_reg, dst_reg)); goto emit_cond_jmp; } / cmp dst_reg, imm8/32 / - maybe_emit_1mod(&prog, dst_reg, + maybe_emit_1mod(jit, dst_reg, BPF_CLASS(insn->code) == BPF_JMP); if (is_imm8(imm32)) @@ -2843,7 +2755,7 @@ st: insn_off = insn->off; nops); return -EFAULT; } - emit_nops(&prog, nops); + emit_nops(jit, nops); } EMIT2(jmp_cond, jmp_offset); } else if (is_simm32(jmp_offset)) { @@ -2856,7 +2768,7 @@ st: insn_off = insn->off; break; case BPF_JMP \| BPF_JA \| BPF_X: - emit_indirect_jump(&prog, insn->dst_reg, ip); + emit_indirect_jump(jit, insn->dst_reg, ip); break; case BPF_JMP \| BPF_JA: case BPF_JMP32 \| BPF_JA: @@ -2900,7 +2812,7 @@ st: insn_off = insn->off; nops); return -EFAULT; } - emit_nops(&prog, nops); + emit_nops(jit, nops); } break; } @@ -2925,7 +2837,7 @@ st: insn_off = insn->off; nops); return -EFAULT; } - emit_nops(&prog, INSN_SZ_DIFF - 2); + emit_nops(jit, INSN_SZ_DIFF - 2); } EMIT2(0xEB, jmp_offset); } else if (is_simm32(jmp_offset)) { @@ -2946,23 +2858,23 @@ st: insn_off = insn->off; ctx->cleanup_addr = proglen; if (bpf_prog_was_classic(bpf_prog) && !ns_capable_noaudit(&init_user_ns, CAP_SYS_ADMIN)) { - if (emit_spectre_bhb_barrier(&prog, ip, bpf_prog)) + if (emit_spectre_bhb_barrier(jit, ip, bpf_prog)) return -EINVAL; } / Deallocate outgoing args 7+ area. / - emit_add_rsp(&prog, outgoing_rsp); + emit_add_rsp(jit, outgoing_rsp); if (bpf_prog->aux->exception_boundary) { - pop_callee_regs(&prog, all_callee_regs_used); - pop_r12(&prog); + pop_callee_regs(jit, all_callee_regs_used); + pop_r12(jit); } else { - pop_callee_regs(&prog, callee_regs_used); + pop_callee_regs(jit, callee_regs_used); if (arena_vm_start) - pop_r12(&prog); + pop_r12(jit); } EMIT1(0xC9); / leave / - bpf_prog->aux->ksym.fp_end = prog - temp; + bpf_prog->aux->ksym.fp_end = jit->prog - temp; - emit_return(&prog, image + addrs[i - 1] + (prog - temp)); + emit_return(jit, image + addrs[i - 1] + (jit->prog - temp)); break; default: @@ -2976,7 +2888,7 @@ st: insn_off = insn->off; return -EINVAL; } - ilen = prog - temp; + ilen = jit->prog - temp; if (ilen > BPF_MAX_INSN_SIZE) { pr_err("bpf_jit: fatal insn size error\n"); return -EFAULT; @@ -3000,7 +2912,7 @@ st: insn_off = insn->off; } proglen += ilen; addrs[i] = proglen; - prog = temp; + jit->prog = temp; } if (image && excnt != bpf_prog->aux->num_exentries) { @@ -3011,11 +2923,10 @@ st: insn_off = insn->off; } static void clean_stack_garbage(const struct btf_func_model m, - u8 *pprog, int nr_stack_slots, + struct jit_emit_context jit, int nr_stack_slots, int stack_size) { int arg_size, off; - u8 prog; / Generally speaking, the compiler will pass the arguments * on-stack with "push" instruction, which will take 8-byte @@ -3047,14 +2958,12 @@ static void clean_stack_garbage(const struct btf_func_model m, arg_size = m->arg_size[m->nr_args - 1]; if (arg_size <= 4) { off = -(stack_size - 4); - prog = pprog; /* mov DWORD PTR [rbp + off], 0 / if (!is_imm8(off)) EMIT2_off32(0xC7, 0x85, off); else EMIT3(0xC7, 0x45, off); EMIT(0, 4); - pprog = prog; } } @@ -3082,26 +2991,23 @@ static int get_nr_used_regs(const struct btf_func_model m) subtraction both truncates and clears the upper half, so the stored * value satisfies the JIT invariant for arena pointer registers. / -static void emit_arena_arg_conv(u8 pprog, u32 src_reg, bool nullable, u32 base_lo) +static void emit_arena_arg_conv(struct jit_emit_context jit, u32 src_reg, bool nullable, + u32 base_lo) { - u8 prog = pprog; - if (nullable) { if (src_reg != BPF_REG_0) - emit_mov_reg(&prog, true, BPF_REG_0, src_reg); + emit_mov_reg(jit, true, BPF_REG_0, src_reg); /* test rax, rax; jz over the 5-byte sub / EMIT3(0x48, 0x85, 0xC0); EMIT2(X86_JE, 5); } else if (src_reg != BPF_REG_0) { - emit_mov_reg(&prog, false, BPF_REG_0, src_reg); + emit_mov_reg(jit, false, BPF_REG_0, src_reg); } / sub eax, base_lo / EMIT1_off32(0x2D, base_lo); - - pprog = prog; } -static void save_args(const struct btf_func_model m, u8 prog, +static void save_args(const struct btf_func_model m, struct jit_emit_context jit, int stack_size, bool for_call_origin, u32 flags, u64 arena_base) { @@ -3150,12 +3056,12 @@ static void save_args(const struct btf_func_model m, u8 *prog, called indirectly, so rbp + 16. / for (j = 0; j < arg_regs; j++) { - emit_ldx(prog, BPF_DW, BPF_REG_0, BPF_REG_FP, + emit_ldx(jit, BPF_DW, BPF_REG_0, BPF_REG_FP, nr_stack_slots 8 + stack_args_off); if (arena_arg) - emit_arena_arg_conv(prog, BPF_REG_0, nullable, + emit_arena_arg_conv(jit, BPF_REG_0, nullable, (u32)arena_base); - emit_stx(prog, BPF_DW, BPF_REG_FP, BPF_REG_0, + emit_stx(jit, BPF_DW, BPF_REG_FP, BPF_REG_0, -stack_size); if (!nr_stack_slots) @@ -3178,20 +3084,20 @@ static void save_args(const struct btf_func_model m, u8 prog, u32 src = nr_regs == 5 ? X86_REG_R9 : BPF_REG_1 + nr_regs; if (arena_arg) { - emit_arena_arg_conv(prog, src, nullable, (u32)arena_base); + emit_arena_arg_conv(jit, src, nullable, (u32)arena_base); src = BPF_REG_0; } - emit_stx(prog, BPF_DW, BPF_REG_FP, src, -stack_size); + emit_stx(jit, BPF_DW, BPF_REG_FP, src, -stack_size); stack_size -= 8; nr_regs++; } } } - clean_stack_garbage(m, prog, nr_stack_slots, first_off); + clean_stack_garbage(m, jit, nr_stack_slots, first_off); } -static void restore_regs(const struct btf_func_model m, u8 *prog, +static void restore_regs(const struct btf_func_model m, struct jit_emit_context jit, int stack_size) { int i, j, arg_regs, nr_regs = 0; @@ -3207,7 +3113,7 @@ static void restore_regs(const struct btf_func_model m, u8 *prog, arg_regs = (m->arg_size[i] + 7) / 8; if (nr_regs + arg_regs <= 6) { for (j = 0; j < arg_regs; j++) { - emit_ldx(prog, BPF_DW, + emit_ldx(jit, BPF_DW, nr_regs == 5 ? X86_REG_R9 : BPF_REG_1 + nr_regs, BPF_REG_FP, -stack_size); @@ -3223,19 +3129,18 @@ static void restore_regs(const struct btf_func_model m, u8 *prog, } } -static int invoke_bpf_prog(const struct btf_func_model m, u8 *pprog, +static int invoke_bpf_prog(const struct btf_func_model m, struct jit_emit_context jit, struct bpf_tramp_node node, int stack_size, int run_ctx_off, bool save_ret, void image, void rw_image) { - u8 prog = pprog; u8 jmp_insn; int ctx_cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie); struct bpf_prog p = node->link->prog; u64 cookie = node->cookie; /* mov rdi, cookie / - emit_mov_imm64(&prog, BPF_REG_1, (long) cookie >> 32, (u32) (long) cookie); + emit_mov_imm64(jit, BPF_REG_1, (long) cookie >> 32, (u32) (long) cookie); / Prepare struct bpf_tramp_run_ctx. * @@ -3244,28 +3149,28 @@ static int invoke_bpf_prog(const struct btf_func_model m, u8 pprog, * mov QWORD PTR [rbp - run_ctx_off + ctx_cookie_off], rdi / - emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_1, -run_ctx_off + ctx_cookie_off); + emit_stx(jit, BPF_DW, BPF_REG_FP, BPF_REG_1, -run_ctx_off + ctx_cookie_off); / arg1: mov rdi, progs[i] / - emit_mov_imm64(&prog, BPF_REG_1, (long) p >> 32, (u32) (long) p); + emit_mov_imm64(jit, BPF_REG_1, (long) p >> 32, (u32) (long) p); / arg2: lea rsi, [rbp - ctx_cookie_off] / if (!is_imm8(-run_ctx_off)) EMIT3_off32(0x48, 0x8D, 0xB5, -run_ctx_off); else EMIT4(0x48, 0x8D, 0x75, -run_ctx_off); - if (emit_rsb_call(&prog, bpf_trampoline_enter(p), image + (prog - (u8 )rw_image))) + if (emit_rsb_call(jit, bpf_trampoline_enter(p), image + (jit->prog - (u8 )rw_image))) return -EINVAL; / remember prog start time returned by __bpf_prog_enter / - emit_mov_reg(&prog, true, BPF_REG_6, BPF_REG_0); + emit_mov_reg(jit, true, BPF_REG_6, BPF_REG_0); / if (__bpf_prog_enter(prog) == 0) goto skip_exec_of_prog; / EMIT3(0x48, 0x85, 0xC0); / test rax,rax / / emit 2 nops that will be replaced with JE insn / - jmp_insn = prog; - emit_nops(&prog, 2); + jmp_insn = jit->prog; + emit_nops(jit, 2); / arg1: lea rdi, [rbp - stack_size] / if (!is_imm8(-stack_size)) @@ -3274,11 +3179,11 @@ static int invoke_bpf_prog(const struct btf_func_model m, u8 *pprog, EMIT4(0x48, 0x8D, 0x7D, -stack_size); / arg2: progs[i]->insnsi for interpreter / if (!p->jited) - emit_mov_imm64(&prog, BPF_REG_2, + emit_mov_imm64(jit, BPF_REG_2, (long) p->insnsi >> 32, (u32) (long) p->insnsi); / call JITed bpf program or interpreter / - if (emit_rsb_call(&prog, p->bpf_func, image + (prog - (u8 )rw_image))) + if (emit_rsb_call(jit, p->bpf_func, image + (jit->prog - (u8 )rw_image))) return -EINVAL; / @@ -3290,42 +3195,39 @@ static int invoke_bpf_prog(const struct btf_func_model m, u8 pprog, value of BPF_PROG_TYPE_STRUCT_OPS prog. / if (save_ret) - emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8); + emit_stx(jit, BPF_DW, BPF_REG_FP, BPF_REG_0, -8); / replace 2 nops with JE insn, since jmp target is known / - jmp_insn[0] = X86_JE; - jmp_insn[1] = prog - jmp_insn - 2; + if (!jit->dry_run) { + jmp_insn[0] = X86_JE; + jmp_insn[1] = jit->prog - jmp_insn - 2; + } / arg1: mov rdi, progs[i] / - emit_mov_imm64(&prog, BPF_REG_1, (long) p >> 32, (u32) (long) p); + emit_mov_imm64(jit, BPF_REG_1, (long) p >> 32, (u32) (long) p); / arg2: mov rsi, rbx <- start time in nsec / - emit_mov_reg(&prog, true, BPF_REG_2, BPF_REG_6); + emit_mov_reg(jit, true, BPF_REG_2, BPF_REG_6); / arg3: lea rdx, [rbp - run_ctx_off] / if (!is_imm8(-run_ctx_off)) EMIT3_off32(0x48, 0x8D, 0x95, -run_ctx_off); else EMIT4(0x48, 0x8D, 0x55, -run_ctx_off); - if (emit_rsb_call(&prog, bpf_trampoline_exit(p), image + (prog - (u8 )rw_image))) + if (emit_rsb_call(jit, bpf_trampoline_exit(p), image + (jit->prog - (u8 )rw_image))) return -EINVAL; - pprog = prog; return 0; } -static void emit_align(u8 *pprog, u32 align) +static void emit_align(struct jit_emit_context jit, u32 align) { - u8 target, prog = pprog; - - target = PTR_ALIGN(prog, align); - if (target != prog) - emit_nops(&prog, target - prog); + u8 target = PTR_ALIGN(jit->prog, align); - pprog = prog; + if (target != jit->prog) + emit_nops(jit, target - jit->prog); } -static int emit_cond_near_jump(u8 pprog, void func, void ip, u8 jmp_cond) +static int emit_cond_near_jump(struct jit_emit_context jit, void func, void ip, u8 jmp_cond) { - u8 prog = pprog; s64 offset; offset = func - (ip + 2 + 4); @@ -3334,48 +3236,44 @@ static int emit_cond_near_jump(u8 *pprog, void func, void ip, u8 jmp_cond) return -EINVAL; } EMIT2_off32(0x0F, jmp_cond + 0x10, offset); - pprog = prog; return 0; } -static int invoke_bpf(const struct btf_func_model m, u8 pprog, +static int invoke_bpf(const struct btf_func_model m, struct jit_emit_context jit, struct bpf_tramp_nodes tl, int stack_size, int run_ctx_off, int func_meta_off, bool save_ret, void image, void rw_image, u64 func_meta, int cookie_off) { int i, cur_cookie = (cookie_off - stack_size) / 8; - u8 prog = pprog; for (i = 0; i < tl->nr_nodes; i++) { if (tl->nodes[i]->link->prog->call_session_cookie) { - emit_store_stack_imm64(&prog, BPF_REG_0, -func_meta_off, + emit_store_stack_imm64(jit, BPF_REG_0, -func_meta_off, func_meta \| (cur_cookie << BPF_TRAMP_COOKIE_INDEX_SHIFT)); cur_cookie--; } - if (invoke_bpf_prog(m, &prog, tl->nodes[i], stack_size, + if (invoke_bpf_prog(m, jit, tl->nodes[i], stack_size, run_ctx_off, save_ret, image, rw_image)) return -EINVAL; } - pprog = prog; return 0; } -static int invoke_bpf_mod_ret(const struct btf_func_model m, u8 *pprog, +static int invoke_bpf_mod_ret(const struct btf_func_model m, struct jit_emit_context jit, struct bpf_tramp_nodes tl, int stack_size, int run_ctx_off, u8 *branches, void image, void rw_image) { - u8 prog = pprog; int i; / The first fmod_ret program will receive a garbage return value. * Set this to 0 to avoid confusing the program. / - emit_mov_imm32(&prog, false, BPF_REG_0, 0); - emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8); + emit_mov_imm32(jit, false, BPF_REG_0, 0); + emit_stx(jit, BPF_DW, BPF_REG_FP, BPF_REG_0, -8); for (i = 0; i < tl->nr_nodes; i++) { - if (invoke_bpf_prog(m, &prog, tl->nodes[i], stack_size, run_ctx_off, true, + if (invoke_bpf_prog(m, jit, tl->nodes[i], stack_size, run_ctx_off, true, image, rw_image)) return -EINVAL; @@ -3391,11 +3289,10 @@ static int invoke_bpf_mod_ret(const struct btf_func_model m, u8 *pprog, are replaced with a conditional jump once do_fexit (i.e. the * start of the fexit invocation) is finalized. / - branches[i] = prog; - emit_nops(&prog, 4 + 2); + branches[i] = jit->prog; + emit_nops(jit, 4 + 2); } - pprog = prog; return 0; } @@ -3474,12 +3371,12 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image im, void rw_im struct bpf_tramp_nodes fentry = &tnodes[BPF_TRAMP_FENTRY]; struct bpf_tramp_nodes fexit = &tnodes[BPF_TRAMP_FEXIT]; struct bpf_tramp_nodes fmod_ret = &tnodes[BPF_TRAMP_MODIFY_RETURN]; + struct jit_emit_context jit_ctx = {}, jit = &jit_ctx; void orig_call = func_addr; int cookie_off, cookie_cnt; u8 branches = NULL; u64 arena_base; u64 func_meta; - u8 prog; bool save_ret; /* @@ -3584,13 +3481,14 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image im, void rw_im orig_call += X86_PATCH_SIZE; } - prog = rw_image; + jit->prog = rw_image; + jit->dry_run = !rw_image; if (flags & BPF_TRAMP_F_INDIRECT) { /* * Indirect call for bpf_struct_ops / - emit_cfi(&prog, image, + emit_cfi(jit, image, cfi_get_func_hash(func_addr), cfi_get_func_arity(func_addr)); } else { @@ -3598,12 +3496,12 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image im, void rw_im Direct-call fentry stub, as such it needs accounting for the * __fentry__ call. / - x86_call_depth_emit_accounting(&prog, NULL, image); + bpf_call_depth_emit_accounting(jit, NULL, image); } EMIT1(0x55); / push rbp / EMIT3(0x48, 0x89, 0xE5); / mov rbp, rsp / if (im) - im->ksym.fp_start = prog - (u8 )rw_image; + im->ksym.fp_start = jit->prog - (u8 )rw_image; if (!is_imm8(stack_size)) { / sub rsp, stack_size / @@ -3615,24 +3513,24 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image im, void rw_im if (flags & BPF_TRAMP_F_TAIL_CALL_CTX) EMIT1(0x50); / push rax / / mov QWORD PTR [rbp - rbx_off], rbx / - emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_6, -rbx_off); + emit_stx(jit, BPF_DW, BPF_REG_FP, BPF_REG_6, -rbx_off); func_meta = nr_regs; / Store number of argument registers of the traced function / - emit_store_stack_imm64(&prog, BPF_REG_0, -func_meta_off, func_meta); + emit_store_stack_imm64(jit, BPF_REG_0, -func_meta_off, func_meta); if (flags & BPF_TRAMP_F_IP_ARG) { / Store IP address of the traced function / - emit_store_stack_imm64(&prog, BPF_REG_0, -ip_off, (long)func_addr); + emit_store_stack_imm64(jit, BPF_REG_0, -ip_off, (long)func_addr); } - save_args(m, &prog, regs_off, false, flags, arena_base); + save_args(m, jit, regs_off, false, flags, arena_base); if (flags & BPF_TRAMP_F_CALL_ORIG) { / arg1: mov rdi, im / - emit_mov_imm64(&prog, BPF_REG_1, (long) im >> 32, (u32) (long) im); - if (emit_rsb_call(&prog, __bpf_tramp_enter, - image + (prog - (u8 )rw_image))) { + emit_mov_imm64(jit, BPF_REG_1, (long) im >> 32, (u32) (long) im); + if (emit_rsb_call(jit, __bpf_tramp_enter, + image + (jit->prog - (u8 )rw_image))) { ret = -EINVAL; goto cleanup; } @@ -3641,13 +3539,13 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image im, void rw_im if (bpf_fsession_cnt(tnodes)) { / clear all the session cookies' value / for (int i = 0; i < cookie_cnt; i++) - emit_store_stack_imm64(&prog, BPF_REG_0, -cookie_off + 8 i, 0); + emit_store_stack_imm64(jit, BPF_REG_0, -cookie_off + 8 * i, 0); /* clear the return value to make sure fentry always get 0 / - emit_store_stack_imm64(&prog, BPF_REG_0, -8, 0); + emit_store_stack_imm64(jit, BPF_REG_0, -8, 0); } if (fentry->nr_nodes) { - if (invoke_bpf(m, &prog, fentry, regs_off, run_ctx_off, func_meta_off, + if (invoke_bpf(m, jit, fentry, regs_off, run_ctx_off, func_meta_off, flags & BPF_TRAMP_F_RET_FENTRY_RET, image, rw_image, func_meta, cookie_off)) return -EINVAL; @@ -3659,7 +3557,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image im, void rw_im if (!branches) return -ENOMEM; - if (invoke_bpf_mod_ret(m, &prog, fmod_ret, regs_off, + if (invoke_bpf_mod_ret(m, jit, fmod_ret, regs_off, run_ctx_off, branches, image, rw_image)) { ret = -EINVAL; goto cleanup; @@ -3667,8 +3565,8 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image im, void rw_im } if (flags & BPF_TRAMP_F_CALL_ORIG) { - restore_regs(m, &prog, regs_off); - save_args(m, &prog, arg_stack_off, true, flags, 0); + restore_regs(m, jit, regs_off); + save_args(m, jit, arg_stack_off, true, flags, 0); if (flags & BPF_TRAMP_F_TAIL_CALL_CTX) { / Before calling the original function, load the @@ -3678,19 +3576,19 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image im, void rw_im } if (flags & BPF_TRAMP_F_ORIG_STACK) { - emit_ldx(&prog, BPF_DW, BPF_REG_6, BPF_REG_FP, 8); + emit_ldx(jit, BPF_DW, BPF_REG_6, BPF_REG_FP, 8); EMIT2(0xff, 0xd3); /* call rbx / } else { /* call original function / - if (emit_rsb_call(&prog, orig_call, image + (prog - (u8 )rw_image))) { + if (emit_rsb_call(jit, orig_call, image + (jit->prog - (u8 )rw_image))) { ret = -EINVAL; goto cleanup; } } / remember return value in a stack for bpf prog to access / - emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8); - im->ip_after_call = image + (prog - (u8 )rw_image); - emit_nops(&prog, X86_PATCH_SIZE); + emit_stx(jit, BPF_DW, BPF_REG_FP, BPF_REG_0, -8); + im->ip_after_call = image + (jit->prog - (u8 )rw_image); + emit_nops(jit, X86_PATCH_SIZE); } if (fmod_ret->nr_nodes) { @@ -3699,12 +3597,17 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image im, void rw_im Coding Rule 11: All branch targets should be 16-byte * aligned. / - emit_align(&prog, 16); + emit_align(jit, 16); / Update the branches saved in invoke_bpf_mod_ret with the * aligned address of do_fexit. / for (i = 0; i < fmod_ret->nr_nodes; i++) { - emit_cond_near_jump(&branches[i], image + (prog - (u8 )rw_image), + struct jit_emit_context branch_jit = { + .prog = branches[i], + .dry_run = jit->dry_run, + }; + + emit_cond_near_jump(&branch_jit, image + (jit->prog - (u8 )rw_image), image + (branches[i] - (u8 )rw_image), X86_JNE); } } @@ -3712,10 +3615,10 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image im, void rw_im /* set the "is_return" flag for fsession / func_meta \|= (1ULL << BPF_TRAMP_IS_RETURN_SHIFT); if (bpf_fsession_cnt(tnodes)) - emit_store_stack_imm64(&prog, BPF_REG_0, -func_meta_off, func_meta); + emit_store_stack_imm64(jit, BPF_REG_0, -func_meta_off, func_meta); if (fexit->nr_nodes) { - if (invoke_bpf(m, &prog, fexit, regs_off, run_ctx_off, func_meta_off, + if (invoke_bpf(m, jit, fexit, regs_off, run_ctx_off, func_meta_off, false, image, rw_image, func_meta, cookie_off)) { ret = -EINVAL; goto cleanup; @@ -3723,17 +3626,17 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image im, void rw_im } if (flags & BPF_TRAMP_F_RESTORE_REGS) - restore_regs(m, &prog, regs_off); + restore_regs(m, jit, regs_off); / This needs to be done regardless. If there were fmod_ret programs, * the return value is only updated on the stack and still needs to be * restored to R0. / if (flags & BPF_TRAMP_F_CALL_ORIG) { - im->ip_epilogue = image + (prog - (u8 )rw_image); + im->ip_epilogue = image + (jit->prog - (u8 )rw_image); / arg1: mov rdi, im / - emit_mov_imm64(&prog, BPF_REG_1, (long) im >> 32, (u32) (long) im); - if (emit_rsb_call(&prog, __bpf_tramp_exit, image + (prog - (u8 )rw_image))) { + emit_mov_imm64(jit, BPF_REG_1, (long) im >> 32, (u32) (long) im); + if (emit_rsb_call(jit, __bpf_tramp_exit, image + (jit->prog - (u8 )rw_image))) { ret = -EINVAL; goto cleanup; } @@ -3746,25 +3649,26 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image im, void rw_im / restore return value of orig_call or fentry prog back into RAX / if (save_ret) - emit_ldx(&prog, BPF_DW, BPF_REG_0, BPF_REG_FP, -8); + emit_ldx(jit, BPF_DW, BPF_REG_0, BPF_REG_FP, -8); - emit_ldx(&prog, BPF_DW, BPF_REG_6, BPF_REG_FP, -rbx_off); + emit_ldx(jit, BPF_DW, BPF_REG_6, BPF_REG_FP, -rbx_off); EMIT1(0xC9); / leave / if (im) - im->ksym.fp_end = prog - (u8 )rw_image; + im->ksym.fp_end = jit->prog - (u8 )rw_image; if (flags & BPF_TRAMP_F_SKIP_FRAME) { / skip our return address and return to parent / EMIT4(0x48, 0x83, 0xC4, 8); / add rsp, 8 / } - emit_return(&prog, image + (prog - (u8 )rw_image)); + emit_return(jit, image + (jit->prog - (u8 )rw_image)); / Make sure the trampoline generation logic doesn't overflow / - if (WARN_ON_ONCE(prog > (u8 )rw_image_end - BPF_INSN_SAFETY)) { + if (!jit->dry_run && + WARN_ON_ONCE(jit->prog > (u8 )rw_image_end - BPF_INSN_SAFETY)) { ret = -EFAULT; goto cleanup; } - ret = prog - (u8 )rw_image + BPF_INSN_SAFETY; + ret = jit->prog - (u8 )rw_image + BPF_INSN_SAFETY; cleanup: kfree(branches); @@ -3819,28 +3723,15 @@ int arch_bpf_trampoline_size(const struct btf_func_model m, u32 flags, struct bpf_tramp_nodes tnodes, void func_addr) { struct bpf_tramp_image im; - void image; - int ret; - / Allocate a temporary buffer for __arch_prepare_bpf_trampoline(). - * - * We cannot use kvmalloc here, because we need image to be in - * module memory range. - * Since it must be writable use bpf_jit_alloc_exec_rw(). - / - image = bpf_jit_alloc_exec_rw(PAGE_SIZE); - if (!image) - return -ENOMEM; - - ret = __arch_prepare_bpf_trampoline(&im, image, image + PAGE_SIZE, image, - m, flags, tnodes, func_addr); - bpf_jit_free_exec(image); - return ret; + return __arch_prepare_bpf_trampoline(&im, NULL, NULL, NULL, m, flags, + tnodes, func_addr); } -static int emit_bpf_dispatcher(u8 pprog, int a, int b, s64 progs, u8 image, u8 buf) +static int emit_bpf_dispatcher(struct jit_emit_context jit, int a, int b, s64 progs, u8 image, + u8 buf) { - u8 jg_reloc, prog = pprog; + u8 jg_reloc; int pivot, err, jg_bytes = 1; s64 jg_offset; @@ -3853,15 +3744,13 @@ static int emit_bpf_dispatcher(u8 *pprog, int a, int b, s64 progs, u8 image, return -1; EMIT2_off32(0x81, add_1reg(0xF8, BPF_REG_3), progs[a]); - err = emit_cond_near_jump(&prog, / je func / - (void )progs[a], image + (prog - buf), + err = emit_cond_near_jump(jit, /* je func / + (void )progs[a], image + (jit->prog - buf), X86_JE); if (err) return err; - emit_indirect_jump(&prog, BPF_REG_3 /* R3 -> rdx /, image + (prog - buf)); - - pprog = prog; + emit_indirect_jump(jit, BPF_REG_3 /* R3 -> rdx /, image + (jit->prog - buf)); return 0; } @@ -3881,9 +3770,9 @@ static int emit_bpf_dispatcher(u8 pprog, int a, int b, s64 progs, u8 image, } else { EMIT2(X86_JG, 0); } - jg_reloc = prog; + jg_reloc = jit->prog; - err = emit_bpf_dispatcher(&prog, a, a + pivot, / emit lower_part / + err = emit_bpf_dispatcher(jit, a, a + pivot, / emit lower_part / progs, image, buf); if (err) return err; @@ -3893,16 +3782,15 @@ static int emit_bpf_dispatcher(u8 pprog, int a, int b, s64 progs, u8 image, Coding Rule 11: All branch targets should be 16-byte * aligned. / - emit_align(&prog, 16); - jg_offset = prog - jg_reloc; + emit_align(jit, 16); + jg_offset = jit->prog - jg_reloc; emit_code(jg_reloc - jg_bytes, jg_offset, jg_bytes); - err = emit_bpf_dispatcher(&prog, a + pivot + 1, / emit upper_part / + err = emit_bpf_dispatcher(jit, a + pivot + 1, / emit upper_part / b, progs, image, buf); if (err) return err; - pprog = prog; return 0; } @@ -3920,10 +3808,10 @@ static int cmp_ips(const void a, const void b) int arch_prepare_bpf_dispatcher(void image, void buf, s64 funcs, int num_funcs) { - u8 prog = buf; + struct jit_emit_context jit = { .prog = buf }; sort(funcs, num_funcs, sizeof(funcs[0]), cmp_ips, NULL); - return emit_bpf_dispatcher(&prog, 0, num_funcs - 1, funcs, image, buf); + return emit_bpf_dispatcher(&jit, 0, num_funcs - 1, funcs, image, buf); } static void priv_stack_init_guard(void __percpu *priv_stack_ptr, int alloc_size)
3/2	2026/08/31 14:02	llm	patch-evaluator	4m Model: gemini-3.7-flash Args: null Results: null Tokens: input: 41934 output: 34 thoughts: 993 Thoughts: Analyzing the BPF JIT Compiler Patch Okay, let's break this down. My initial assessment is that this patch is primarily focused on optimizing the BPF JIT compiler for the x86 architecture. The key change is the introduction of a `struct jit_emit_context`, which seems to be the central mechanism for the refactoring. It has `prog` and `dry_run` members, indicating a strategy of separating the code generation from the actual emission. Specifically, the `dry_run` flag is crucial. The `arch_bpf_trampoline_size` function is significantly altered to calculate the trampoline size without actually allocating an executable page using this flag. The old approach allocated a page, prepared a trampoline, and then freed the page just to get the size. This is clearly a performance win. This "dry run" concept is propagated throughout the code via the `jit_emit_context` struct. Functions like `bpf_call_depth_emit_accounting`, `emit_code_jit`, and `emit_cond_near_jump` are all updated to behave differently based on the `dry_run` flag. I can see the `bpf_call_depth_emit_accounting` function being refactored to use this new context and `insn_buff`. The `emit_code_jit` and `emit_cond_near_jump` functions now have a clear separation of concerns, and the actual emission is conditionally executed based on the `dry_run` flag. The code changes include `arch/x86/include/asm/alternative.h`, `arch/x86/include/asm/text-patching.h`, `arch/x86/kernel/alternative.c`, `arch/x86/kernel/callthunks.c`, and `arch/x86/net/bpf_jit_comp.c`, so I'll need to check the relevant configs and focus symbols using the search tools. This seems to be a functional change and performance optimization to BPF JIT, particularly trampoline sizing and code generation, which makes it an interesting target for deeper analysis and testing. This all appears to revolve around the BPF JIT and the core x86 architecture. I'll need to delve deeper and see if any vulnerabilities are potentially exposed by these architectural changes and optimizations. Now, let's explore the BPF JIT code and relevant configurations and potential focus symbols to assess this patch's fuzzing worthiness.
4/2	2026/08/31 14:07	tool	grepper	0m Args: { "Expression": "arch_prepare_bpf_trampoline", "PathPrefix": "arch/x86" } Results: { "Output": "arch/x86/net/bpf_jit_comp.c=3132=static int invoke_bpf_prog(const struct btf_func_model m, struct jit_emit_context jit,\n--\narch/x86/net/bpf_jit_comp.c-3147-\t * bpf_tramp_run_ctx is already preserved by\narch/x86/net/bpf_jit_comp.c:3148:\t * arch_prepare_bpf_trampoline().\narch/x86/net/bpf_jit_comp.c-3149-\t \n--\narch/x86/net/bpf_jit_comp.c=3263=static int invoke_bpf_mod_ret(const struct btf_func_model m, struct jit_emit_context jit,\n--\narch/x86/net/bpf_jit_comp.c-3362- /\narch/x86/net/bpf_jit_comp.c:3363:static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image im, void rw_image,\narch/x86/net/bpf_jit_comp.c-3364-\t\t\t\t\t void rw_image_end, void image,\n--\narch/x86/net/bpf_jit_comp.c=3688=int arch_protect_bpf_trampoline(void image, unsigned int size)\n--\narch/x86/net/bpf_jit_comp.c-3692-\narch/x86/net/bpf_jit_comp.c:3693:int arch_prepare_bpf_trampoline(struct bpf_tramp_image im, void image, void image_end,\narch/x86/net/bpf_jit_comp.c-3694-\t\t\t\tconst struct btf_func_model m, u32 flags,\n--\narch/x86/net/bpf_jit_comp.c-3708-\narch/x86/net/bpf_jit_comp.c:3709:\tret = __arch_prepare_bpf_trampoline(im, rw_image, rw_image + size, image, m,\narch/x86/net/bpf_jit_comp.c-3710-\t\t\t\t\t flags, tnodes, func_addr);\n--\narch/x86/net/bpf_jit_comp.c=3722=int arch_bpf_trampoline_size(const struct btf_func_model m, u32 flags,\n--\narch/x86/net/bpf_jit_comp.c-3726-\narch/x86/net/bpf_jit_comp.c:3727:\treturn __arch_prepare_bpf_trampoline(\u0026im, NULL, NULL, NULL, m, flags,\narch/x86/net/bpf_jit_comp.c-3728-\t\t\t\t\t tnodes, func_addr);\n" }
5/2	2026/08/31 14:07	llm	patch-evaluator	0m Model: gemini-3.7-flash Args: null Results: null Tokens: input: 44264 output: 31 thoughts: 65 Thoughts: Here's my thinking process: Investigating Kconfig Dependencies and Build System Integration Alright, I need to understand the configuration dependencies and build process surrounding `arch/x86/net/bpf_jit_comp.c` and `arch/x86/kernel/callthunks.c`. I'll start by checking which Kconfig options are directly related to these files. This will give me a better understanding of how the features they implement are enabled and disabled. Next, I need to figure out how these source files are compiled and linked into the kernel. The best way to do that is to look at the Makefiles. I'll search the relevant directories, `arch/x86/net/` and `arch/x86/kernel/`, to find the Makefiles that build these files. That should give me the overall build configuration.
6/2	2026/08/31 14:08	tool	grepper	0m Args: { "Expression": "bpf_jit_comp", "PathPrefix": "arch/x86" } Results: { "Output": "arch/x86/net/Makefile=6=ifeq ($(CONFIG_X86_32),y)\narch/x86/net/Makefile:7: obj-$(CONFIG_BPF_JIT) += bpf_jit_comp32.o\narch/x86/net/Makefile-8-else\narch/x86/net/Makefile:9: obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o bpf_timed_may_goto.o\narch/x86/net/Makefile-10-endif\n" }
7/2	2026/08/31 14:08	llm	patch-evaluator	1m Model: gemini-3.7-flash Args: null Results: null Tokens: input: 44693 output: 28 thoughts: 13 Thoughts: Here's my attempt at summarizing the thought process, tailored for an expert audience: Investigating Potential Configuration Issues Okay, so we've been looking at this module's behavior, and I'm starting to think we might be missing something on the configuration side. My immediate thought is to double-check the `callthunks` build system integration. I need to dive into the `Makefile` and the associated `Kconfig` files for that component. There might be a subtle dependency, a missing flag, or an incorrect setting within those files that's throwing us off. It's a classic debugging scenario: look beyond the immediate code and trace the build environment. This is probably a good place to start, given the symptoms.
8/2	2026/08/31 14:09	tool	grepper	0m Args: { "Expression": "callthunks", "PathPrefix": "arch/x86" } Results: { "Output": "arch/x86/Kconfig=2474=config CALL_THUNKS_DEBUG\n--\narch/x86/Kconfig-2480-\t Enable call/ret counters for imbalance detection and build in\narch/x86/Kconfig:2481:\t a noisy dmesg about callthunks generation and call patching for\narch/x86/Kconfig-2482-\t trouble shooting. The debug prints need to be enabled on the\narch/x86/Kconfig:2483:\t kernel command line with 'debug-callthunks'.\narch/x86/Kconfig-2484-\t Only enable this when you are debugging call thunks as this\n--\narch/x86/include/asm/alternative.h=77=struct callthunk_sites {\n--\narch/x86/include/asm/alternative.h-81-#ifdef CONFIG_CALL_THUNKS\narch/x86/include/asm/alternative.h:82:extern void callthunks_patch_builtin_calls(void);\narch/x86/include/asm/alternative.h:83:extern void callthunks_patch_module_calls(struct callthunk_sites sites,\narch/x86/include/asm/alternative.h-84-\t\t\t\t\t struct module mod);\narch/x86/include/asm/alternative.h:85:extern void callthunks_translate_call_dest(void dest);\narch/x86/include/asm/alternative.h-86-extern int x86_call_depth_emit_accounting(u8 insn_buff, void func, void ip);\narch/x86/include/asm/alternative.h-87-#else\narch/x86/include/asm/alternative.h:88:static __always_inline void callthunks_patch_builtin_calls(void) {}\narch/x86/include/asm/alternative.h-89-static __always_inline void\narch/x86/include/asm/alternative.h:90:callthunks_patch_module_calls(struct callthunk_sites sites,\narch/x86/include/asm/alternative.h-91-\t\t\t struct module mod) {}\narch/x86/include/asm/alternative.h:92:static __always_inline void callthunks_translate_call_dest(void dest)\narch/x86/include/asm/alternative.h-93-{\n--\narch/x86/kernel/Makefile=165=obj-$(CONFIG_CFI)\t\t\t+= cfi.o\narch/x86/kernel/Makefile-166-\narch/x86/kernel/Makefile:167:obj-$(CONFIG_CALL_THUNKS)\t\t+= callthunks.o\narch/x86/kernel/Makefile-168-\n--\narch/x86/kernel/alternative.c=2237=void __init alternative_instructions(void)\n--\narch/x86/kernel/alternative.c-2286-\t /\narch/x86/kernel/alternative.c:2287:\tcallthunks_patch_builtin_calls();\narch/x86/kernel/alternative.c-2288-\n--\narch/x86/kernel/callthunks.c-2-\narch/x86/kernel/callthunks.c:3:#define pr_fmt(fmt) \"callthunks: \" fmt\narch/x86/kernel/callthunks.c-4-\n--\narch/x86/kernel/callthunks.c-23-\narch/x86/kernel/callthunks.c:24:static int __initdata_or_module debug_callthunks;\narch/x86/kernel/callthunks.c-25-\n--\narch/x86/kernel/callthunks.c=27=do {\t\t\t\t\t\t\t\t\\\narch/x86/kernel/callthunks.c:28:\tif (debug_callthunks)\t\t\t\t\t\\\narch/x86/kernel/callthunks.c-29-\t\tprintk(KERN_DEBUG pr_fmt(fmt), ##args);\t\t\\\n--\narch/x86/kernel/callthunks.c=32=static int __init debug_thunks(char str)\narch/x86/kernel/callthunks.c-33-{\narch/x86/kernel/callthunks.c:34:\tdebug_callthunks = 1;\narch/x86/kernel/callthunks.c-35-\treturn 1;\narch/x86/kernel/callthunks.c-36-}\narch/x86/kernel/callthunks.c:37:__setup(\"debug-callthunks\", debug_thunks);\narch/x86/kernel/callthunks.c-38-\n--\narch/x86/kernel/callthunks.c=238=static __init_or_module void\narch/x86/kernel/callthunks.c:239:callthunks_setup(struct callthunk_sites cs, const struct core_text ct)\narch/x86/kernel/callthunks.c-240-{\n--\narch/x86/kernel/callthunks.c-245-\narch/x86/kernel/callthunks.c:246:void __init callthunks_patch_builtin_calls(void)\narch/x86/kernel/callthunks.c-247-{\n--\narch/x86/kernel/callthunks.c-257-\tmutex_lock(\u0026text_mutex);\narch/x86/kernel/callthunks.c:258:\tcallthunks_setup(\u0026cs, \u0026builtin_coretext);\narch/x86/kernel/callthunks.c-259-\tthunks_initialized = true;\n--\narch/x86/kernel/callthunks.c-262-\narch/x86/kernel/callthunks.c:263:void callthunks_translate_call_dest(void dest)\narch/x86/kernel/callthunks.c-264-{\n--\narch/x86/kernel/callthunks.c=299=int x86_call_depth_emit_accounting(u8 insn_buff, void func, void ip)\n--\narch/x86/kernel/callthunks.c-317-#ifdef CONFIG_MODULES\narch/x86/kernel/callthunks.c:318:void noinline callthunks_patch_module_calls(struct callthunk_sites cs,\narch/x86/kernel/callthunks.c-319-\t\t\t\t\t struct module mod)\n--\narch/x86/kernel/callthunks.c-330-\tmutex_lock(\u0026text_mutex);\narch/x86/kernel/callthunks.c:331:\tcallthunks_setup(cs, \u0026ct);\narch/x86/kernel/callthunks.c-332-\tmutex_unlock(\u0026text_mutex);\n--\narch/x86/kernel/callthunks.c-336-#if defined(CONFIG_CALL_THUNKS_DEBUG) \u0026\u0026 defined(CONFIG_DEBUG_FS)\narch/x86/kernel/callthunks.c:337:static int callthunks_debug_show(struct seq_file m, void p)\narch/x86/kernel/callthunks.c-338-{\n--\narch/x86/kernel/callthunks.c-348-\narch/x86/kernel/callthunks.c:349:static int callthunks_debug_open(struct inode inode, struct file file)\narch/x86/kernel/callthunks.c-350-{\narch/x86/kernel/callthunks.c:351:\treturn single_open(file, callthunks_debug_show, inode-\u003ei_private);\narch/x86/kernel/callthunks.c-352-}\n--\narch/x86/kernel/callthunks.c=354=static const struct file_operations dfs_ops = {\narch/x86/kernel/callthunks.c:355:\t.open\t\t= callthunks_debug_open,\narch/x86/kernel/callthunks.c-356-\t.read\t\t= seq_read,\n--\narch/x86/kernel/callthunks.c-360-\narch/x86/kernel/callthunks.c:361:static int __init callthunks_debugfs_init(void)\narch/x86/kernel/callthunks.c-362-{\n--\narch/x86/kernel/callthunks.c-365-\narch/x86/kernel/callthunks.c:366:\tdir = debugfs_create_dir(\"callthunks\", NULL);\narch/x86/kernel/callthunks.c-367-\tfor_each_possible_cpu(cpu) {\n--\narch/x86/kernel/callthunks.c-375-}\narch/x86/kernel/callthunks.c:376:__initcall(callthunks_debugfs_init);\narch/x86/kernel/callthunks.c-377-#endif\n--\narch/x86/kernel/module.c=226=int module_finalize(const Elf_Ehdr hdr,\n--\narch/x86/kernel/module.c-289-\narch/x86/kernel/module.c:290:\t\tcallthunks_patch_module_calls(\u0026cs, me);\narch/x86/kernel/module.c-291-\t}\n--\narch/x86/kernel/static_call.c=75=static void __ref __static_call_transform(void insn, enum insn_type type,\n--\narch/x86/kernel/static_call.c-87-\tcase CALL:\narch/x86/kernel/static_call.c:88:\t\tfunc = callthunks_translate_call_dest(func);\narch/x86/kernel/static_call.c-89-\t\tcode = text_gen_insn(CALL_INSN_OPCODE, insn, func);\n" }
9/2	2026/08/31 14:09	llm	patch-evaluator	1m Model: gemini-3.7-flash Args: null Results: null Tokens: input: 47466 output: 31 thoughts: 14 Thoughts: Here's my attempt at summarizing the thought process, tailored for an expert audience and written in the first person: Investigating the Kconfig for CALL_THUNKS Okay, let's dive in. I need to understand how `CALL_THUNKS` is being defined within the Kconfig system. Specifically, I want to see what options and dependencies are involved with this configuration symbol. This will help me grasp the context and purpose of `CALL_THUNKS` within the kernel build configuration. I'll need to use my experience with the kernel configuration system to quickly navigate the relevant files and options to get the information I need, specifically looking for dependencies, default values, and any relevant documentation or comments within the config files themselves. Knowing its definition will be key to understanding the deeper architectural implications and how it impacts the performance of the system.
10/2	2026/08/31 14:10	tool	grepper	0m Args: { "Expression": "config CALL_THUNKS", "PathPrefix": "arch/x86" } Results: { "Output": "arch/x86/Kconfig=2394=config HAVE_CALL_THUNKS\n--\narch/x86/Kconfig-2397-\narch/x86/Kconfig:2398:config CALL_THUNKS\narch/x86/Kconfig-2399-\tdef_bool n\n--\narch/x86/Kconfig=2457=config MITIGATION_CALL_DEPTH_TRACKING\n--\narch/x86/Kconfig-2473-\narch/x86/Kconfig:2474:config CALL_THUNKS_DEBUG\narch/x86/Kconfig-2475-\tbool \"Enable call thunks and call depth tracking debugging\"\n" }
11/2	2026/08/31 14:10	llm	patch-evaluator	1m Model: gemini-3.7-flash Args: null Results: null Tokens: input: 47877 output: 37 thoughts: 27 Thoughts: Here's a summary of my mental process: Diving into the x86 Architecture Configuration Okay, time to get my hands dirty. I need to understand some specific configuration options for the x86 architecture, specifically around lines 2390 to 2480 in the `arch/x86/Kconfig` file. I'm assuming this section likely deals with something relatively specific or potentially complex; otherwise, why would I be zeroing in on this particular range? I'm anticipating this will require a closer look at the dependencies, descriptions, and potentially the default values of those kernel configuration variables. I'll be looking for nuances in how these options interact and affect the x86 system's behavior. I will be looking out for any potentially controversial configurations or areas that could be impacted by an emerging trend. Right, let's start reading.
12/2	2026/08/31 14:12	tool	read-file	0m Args: { "File": "arch/x86/Kconfig", "FirstLine": 2390, "LineCount": 80 } Results: { "Contents": "2390:\tconfig FINEIBT_BHI\n2391:\t\tdef_bool y\n2392:\t\tdepends on FINEIBT \u0026\u0026 CC_HAS_KCFI_ARITY\n2393:\t\n2394:\tconfig HAVE_CALL_THUNKS\n2395:\t\tdef_bool y\n2396:\t\tdepends on CC_HAS_ENTRY_PADDING \u0026\u0026 MITIGATION_RETHUNK \u0026\u0026 OBJTOOL\n2397:\t\n2398:\tconfig CALL_THUNKS\n2399:\t\tdef_bool n\n2400:\t\tselect CALL_PADDING\n2401:\t\n2402:\tmenuconfig CPU_MITIGATIONS\n2403:\t\tbool \"Mitigations for CPU vulnerabilities\"\n2404:\t\tdefault y\n2405:\t\thelp\n2406:\t\t Say Y here to enable options which enable mitigations for hardware\n2407:\t\t vulnerabilities (usually related to speculative execution).\n2408:\t\t Mitigations can be disabled or restricted to SMT systems at runtime\n2409:\t\t via the \"mitigations\" kernel parameter.\n2410:\t\n2411:\t\t If you say N, all mitigations will be disabled. This CANNOT be\n2412:\t\t overridden at runtime.\n2413:\t\n2414:\t\t Say 'Y', unless you really know what you are doing.\n2415:\t\n2416:\tif CPU_MITIGATIONS\n2417:\t\n2418:\tconfig MITIGATION_PAGE_TABLE_ISOLATION\n2419:\t\tbool \"Remove the kernel mapping in user mode\"\n2420:\t\tdefault y\n2421:\t\tdepends on (X86_64 \|\| X86_PAE)\n2422:\t\thelp\n2423:\t\t This feature reduces the number of hardware side channels by\n2424:\t\t ensuring that the majority of kernel addresses are not mapped\n2425:\t\t into userspace.\n2426:\t\n2427:\t\t See Documentation/arch/x86/pti.rst for more details.\n2428:\t\n2429:\tconfig MITIGATION_RETPOLINE\n2430:\t\tbool \"Avoid speculative indirect branches in kernel\"\n2431:\t\tselect OBJTOOL if HAVE_OBJTOOL\n2432:\t\tdefault y\n2433:\t\thelp\n2434:\t\t Compile kernel with the retpoline compiler options to guard against\n2435:\t\t kernel-to-user data leaks by avoiding speculative indirect\n2436:\t\t branches. Requires a compiler with -mindirect-branch=thunk-extern\n2437:\t\t support for full protection. The kernel may run slower.\n2438:\t\n2439:\tconfig MITIGATION_RETHUNK\n2440:\t\tbool \"Enable return-thunks\"\n2441:\t\tdepends on MITIGATION_RETPOLINE \u0026\u0026 CC_HAS_RETURN_THUNK\n2442:\t\tselect OBJTOOL if HAVE_OBJTOOL\n2443:\t\tdefault y if X86_64\n2444:\t\thelp\n2445:\t\t Compile the kernel with the return-thunks compiler option to guard\n2446:\t\t against kernel-to-user data leaks by avoiding return speculation.\n2447:\t\t Requires a compiler with -mfunction-return=thunk-extern\n2448:\t\t support for full protection. The kernel may run slower.\n2449:\t\n2450:\tconfig MITIGATION_UNRET_ENTRY\n2451:\t\tbool \"Enable UNRET on kernel entry\"\n2452:\t\tdepends on CPU_SUP_AMD \u0026\u0026 MITIGATION_RETHUNK \u0026\u0026 X86_64\n2453:\t\tdefault y\n2454:\t\thelp\n2455:\t\t Compile the kernel with support for the retbleed=unret mitigation.\n2456:\t\n2457:\tconfig MITIGATION_CALL_DEPTH_TRACKING\n2458:\t\tbool \"Mitigate RSB underflow with call depth tracking\"\n2459:\t\tdepends on CPU_SUP_INTEL \u0026\u0026 HAVE_CALL_THUNKS\n2460:\t\tselect HAVE_DYNAMIC_FTRACE_NO_PATCHABLE\n2461:\t\tselect CALL_THUNKS\n2462:\t\tdefault y\n2463:\t\thelp\n2464:\t\t Compile the kernel with call depth tracking to mitigate the Intel\n2465:\t\t SKL Return-Stack-Buffer (RSB) underflow issue. The mitigation is off\n2466:\t\t by default and needs to be enabled on the kernel command line via the\n2467:\t\t retbleed=stuff option. For non-affected systems the overhead of this\n2468:\t\t option is marginal as the call depth tracking is using run-time\n2469:\t\t generated call thunks in a compiler generated padding area and call\n" }
13/2	2026/08/31 14:12	llm	patch-evaluator	3m Model: gemini-3.7-flash Args: null Results: null Tokens: input: 49359 output: 31 thoughts: 61 Thoughts: Here's my thought process, as if I were analyzing this code: Locating the BPF_JIT Configuration Okay, so I'm diving into this BPF setup, and I need to figure out where `BPF_JIT` is actually defined. That's the key to understanding the JIT compiler configuration, and its going to be important to get to the bottom of the config variables. Given my familiarity with kernel structure, my initial thought is that it's probably hanging out in one of a few places. I'd lean towards `net/bpf/`, because that's the obvious home for BPF-related code. However, `init/Kconfig` is also a possibility since it sets up a lot of the initial kernel configs. It could also potentially be in `kernel/bpf/Kconfig`, if it's considered more core BPF functionality. Let's start the search.
14/2	2026/08/31 14:15	tool	grepper	0m Error: no matches Args: { "Expression": "config BPF_JIT", "PathPrefix": "init\|kernel\|net" } Results: { "error": "no matches" }
15/2	2026/08/31 14:15	llm	patch-evaluator	1m Model: gemini-3.7-flash Args: null Results: null Tokens: input: 49662 output: 22 thoughts: 17 Thoughts: Here's my attempt at summarizing the thought process, tailored for an expert audience: Investigating BPF JIT Configuration Okay, so I need to get a handle on the current configuration of the BPF JIT, or specifically, `CONFIG_BPF_JIT`. My immediate instinct is to do a global search across the codebase. `grep` is the obvious tool here, because it's fast and allows me to search for a specific string. I'll need to specify the string `config BPF_JIT`, because I want to look for the configuration option definition specifically, rather than just every time that BPF and JIT appear near each other. I'm going to run that grep command across the entire source tree, from the root, because I want a comprehensive view of how it's defined and used. This should give me a solid baseline, revealing where the option is defined and, likely, in what configuration files. This should be a quick initial sanity check, and from there I can delve deeper into the dependencies or specific implementations I find.
16/2	2026/08/31 14:16	tool	grepper	0m Args: { "Expression": "config BPF_JIT\\b" } Results: { "Output": "kernel/bpf/Kconfig=27=config BPF_SYSCALL\n--\nkernel/bpf/Kconfig-41-\nkernel/bpf/Kconfig:42:config BPF_JIT\nkernel/bpf/Kconfig-43-\tbool \"Enable BPF Just In Time compiler\"\n" }
17/2	2026/08/31 14:16	llm	patch-evaluator	0m Model: gemini-3.7-flash Args: null Results: null Tokens: input: 50012 output: 34 thoughts: 22 Thoughts: Examining Kernel BPF Configuration Okay, let's dive in. I need to understand how the BPF subsystem is configured within this kernel build. My initial thought is to navigate to the `kernel/bpf/Kconfig` file. I'll need to focus my attention around lines 20 through 60. That's usually where the top-level options for the BPF feature set are defined – things like whether it's enabled, any architectural dependencies, and perhaps some of the fundamental knobs that control its behavior. I expect to find things like the default state of BPF, any relevant flags, and maybe even some conditional compilation logic depending on the target architecture or kernel configuration. This section should give me the crucial high-level view of how BPF is integrated into this kernel. I anticipate this will be the starting point to understand the broader implications of BPF configuration.
18/2	2026/08/31 14:16	tool	read-file	0m Args: { "File": "kernel/bpf/Kconfig", "FirstLine": 20, "LineCount": 60 } Results: { "Contents": " 20:\t# Used by archs to tell that they want the BPF JIT compiler enabled by\n 21:\t# default for kernels that were compiled with BPF JIT support.\n 22:\tconfig ARCH_WANT_DEFAULT_BPF_JIT\n 23:\t\tbool\n 24:\t\n 25:\tmenu \"BPF subsystem\"\n 26:\t\n 27:\tconfig BPF_SYSCALL\n 28:\t\tbool \"Enable bpf() system call\"\n 29:\t\tselect BPF\n 30:\t\tselect IRQ_WORK\n 31:\t\tselect NEED_TASKS_RCU\n 32:\t\tselect TASKS_TRACE_RCU\n 33:\t\tselect BINARY_PRINTF\n 34:\t\tselect NET_SOCK_MSG if NET\n 35:\t\tselect NET_XGRESS if NET\n 36:\t\tselect PAGE_POOL if NET\n 37:\t\tdefault n\n 38:\t\thelp\n 39:\t\t Enable the bpf() system call that allows to manipulate BPF programs\n 40:\t\t and maps via file descriptors.\n 41:\t\n 42:\tconfig BPF_JIT\n 43:\t\tbool \"Enable BPF Just In Time compiler\"\n 44:\t\tdepends on BPF\n 45:\t\tdepends on HAVE_CBPF_JIT \|\| HAVE_EBPF_JIT\n 46:\t\tselect EXECMEM\n 47:\t\thelp\n 48:\t\t BPF programs are normally handled by a BPF interpreter. This option\n 49:\t\t allows the kernel to generate native code when a program is loaded\n 50:\t\t into the kernel. This will significantly speed-up processing of BPF\n 51:\t\t programs.\n 52:\t\n 53:\t\t Note, an admin should enable this feature changing:\n 54:\t\t /proc/sys/net/core/bpf_jit_enable\n 55:\t\t /proc/sys/net/core/bpf_jit_harden (optional)\n 56:\t\t /proc/sys/net/core/bpf_jit_kallsyms (optional)\n 57:\t\n 58:\tconfig BPF_JIT_ALWAYS_ON\n 59:\t\tbool \"Permanently enable BPF JIT and remove BPF interpreter\"\n 60:\t\tdepends on BPF_SYSCALL \u0026\u0026 HAVE_EBPF_JIT \u0026\u0026 BPF_JIT\n 61:\t\thelp\n 62:\t\t Enables BPF JIT and removes BPF interpreter to avoid speculative\n 63:\t\t execution of BPF instructions by the interpreter.\n 64:\t\n 65:\t\t When CONFIG_BPF_JIT_ALWAYS_ON is enabled, /proc/sys/net/core/bpf_jit_enable\n 66:\t\t is permanently set to 1 and setting any other value than that will\n 67:\t\t return failure.\n 68:\t\n 69:\tconfig BPF_JIT_DEFAULT_ON\n 70:\t\tdef_bool ARCH_WANT_DEFAULT_BPF_JIT \|\| BPF_JIT_ALWAYS_ON\n 71:\t\tdepends on HAVE_EBPF_JIT \u0026\u0026 BPF_JIT\n 72:\t\n 73:\tconfig BPF_UNPRIV_DEFAULT_OFF\n 74:\t\tbool \"Disable unprivileged BPF by default\"\n 75:\t\tdefault y\n 76:\t\tdepends on BPF_SYSCALL\n 77:\t\thelp\n 78:\t\t Disables unprivileged BPF by default by setting the corresponding\n 79:\t\t /proc/sys/kernel/unprivileged_bpf_disabled knob to 2. An admin can\n" }
19/2	2026/08/31 14:16	llm	patch-evaluator	1m Model: gemini-3.7-flash Args: null Results: null Tokens: input: 51172 output: 49 thoughts: 252 Thoughts: Analyzing BPF JIT Compilation on x86_64 Okay, so I'm diving into the x86_64 BPF JIT compilation flow, specifically looking at potential impacts or interactions with the mitigation strategies involving call depth tracking, or those call thunks. First, I need to get a handle on the key functions involved, especially in `arch/x86/net/bpf_jit_comp.c`. Let's see... I've got `BPF_JIT`, `BPF_SYSCALL`, and `X86_64` in my head as important context. Right, so that file should contain: `bpf_int_jit_compile` which calls `do_jit`, and those related to trampolines: `arch_prepare_bpf_trampoline`, `arch_bpf_trampoline_size`, `arch_prepare_bpf_dispatcher`. And it looks like `x86_call_depth_emit_accounting` is in `arch/x86/kernel/callthunks.c`. Now, the main question is: are any of these functions exposed externally? Meaning, are `arch_bpf_trampoline_size`, `arch_prepare_bpf_trampoline` and `bpf_int_jit_compile` non-static? I need to check if they're exported symbols. That'll let me know if something outside of `bpf_jit_comp.c` could be directly calling them. I'll just grep for non-static functions in `bpf_jit_comp.c`.
20/2	2026/08/31 14:18	tool	grepper	0m Args: { "Expression": "^[a-z_]+ [a-z_]+\\(", "PathPrefix": "arch/x86/net/bpf_jit_comp.c" } Results: { "Output": "arch/x86/net/bpf_jit_comp.c=326=static void jit_fill_hole(void area, unsigned int size)\n--\narch/x86/net/bpf_jit_comp.c-331-\narch/x86/net/bpf_jit_comp.c:332:int bpf_arch_text_invalidate(void dst, size_t len)\narch/x86/net/bpf_jit_comp.c-333-{\n--\narch/x86/net/bpf_jit_comp.c=601=static int __bpf_arch_text_poke(void ip, enum bpf_text_poke_type old_t,\n--\narch/x86/net/bpf_jit_comp.c-644-\narch/x86/net/bpf_jit_comp.c:645:int bpf_arch_text_poke(void ip, enum bpf_text_poke_type old_t,\narch/x86/net/bpf_jit_comp.c-646-\t\t enum bpf_text_poke_type new_t, void old_addr,\n--\narch/x86/net/bpf_jit_comp.c=1369=static int emit_atomic_ld_st_index(struct jit_emit_context jit, u32 atomic_op, u32 size,\n--\narch/x86/net/bpf_jit_comp.c-1433-\narch/x86/net/bpf_jit_comp.c:1434:bool ex_handler_bpf(const struct exception_table_entry x, struct pt_regs regs)\narch/x86/net/bpf_jit_comp.c-1435-{\n--\narch/x86/net/bpf_jit_comp.c=3678=void arch_alloc_bpf_trampoline(unsigned int size)\n--\narch/x86/net/bpf_jit_comp.c-3682-\narch/x86/net/bpf_jit_comp.c:3683:void arch_free_bpf_trampoline(void image, unsigned int size)\narch/x86/net/bpf_jit_comp.c-3684-{\n--\narch/x86/net/bpf_jit_comp.c-3687-\narch/x86/net/bpf_jit_comp.c:3688:int arch_protect_bpf_trampoline(void image, unsigned int size)\narch/x86/net/bpf_jit_comp.c-3689-{\n--\narch/x86/net/bpf_jit_comp.c-3692-\narch/x86/net/bpf_jit_comp.c:3693:int arch_prepare_bpf_trampoline(struct bpf_tramp_image im, void image, void image_end,\narch/x86/net/bpf_jit_comp.c-3694-\t\t\t\tconst struct btf_func_model m, u32 flags,\n--\narch/x86/net/bpf_jit_comp.c-3721-\narch/x86/net/bpf_jit_comp.c:3722:int arch_bpf_trampoline_size(const struct btf_func_model m, u32 flags,\narch/x86/net/bpf_jit_comp.c-3723-\t\t\t struct bpf_tramp_nodes tnodes, void func_addr)\n--\narch/x86/net/bpf_jit_comp.c=3797=static int cmp_ips(const void a, const void b)\n--\narch/x86/net/bpf_jit_comp.c-3808-\narch/x86/net/bpf_jit_comp.c:3809:int arch_prepare_bpf_dispatcher(void image, void buf, s64 funcs, int num_funcs)\narch/x86/net/bpf_jit_comp.c-3810-{\n--\narch/x86/net/bpf_jit_comp.c=3858=struct bpf_prog bpf_int_jit_compile(struct bpf_verifier_env env, struct bpf_prog prog)\n--\narch/x86/net/bpf_jit_comp.c-4049-\narch/x86/net/bpf_jit_comp.c:4050:bool bpf_jit_supports_kfunc_call(void)\narch/x86/net/bpf_jit_comp.c-4051-{\n--\narch/x86/net/bpf_jit_comp.c-4054-\narch/x86/net/bpf_jit_comp.c:4055:bool bpf_jit_supports_kfunc_ret_reg_pair(void)\narch/x86/net/bpf_jit_comp.c-4056-{\n--\narch/x86/net/bpf_jit_comp.c-4059-\narch/x86/net/bpf_jit_comp.c:4060:bool bpf_jit_supports_stack_args(void)\narch/x86/net/bpf_jit_comp.c-4061-{\n--\narch/x86/net/bpf_jit_comp.c-4064-\narch/x86/net/bpf_jit_comp.c:4065:bool bpf_jit_supports_arena_args(void)\narch/x86/net/bpf_jit_comp.c-4066-{\n--\narch/x86/net/bpf_jit_comp.c=4070=void bpf_arch_text_copy(void dst, void src, size_t len)\n--\narch/x86/net/bpf_jit_comp.c-4077-/ Indicate the JIT backend supports mixing bpf2bpf and tailcalls. /\narch/x86/net/bpf_jit_comp.c:4078:bool bpf_jit_supports_subprog_tailcalls(void)\narch/x86/net/bpf_jit_comp.c-4079-{\n--\narch/x86/net/bpf_jit_comp.c-4082-\narch/x86/net/bpf_jit_comp.c:4083:bool bpf_jit_supports_percpu_insn(void)\narch/x86/net/bpf_jit_comp.c-4084-{\n--\narch/x86/net/bpf_jit_comp.c-4087-\narch/x86/net/bpf_jit_comp.c:4088:void bpf_jit_free(struct bpf_prog prog)\narch/x86/net/bpf_jit_comp.c-4089-{\n--\narch/x86/net/bpf_jit_comp.c-4122-\narch/x86/net/bpf_jit_comp.c:4123:bool bpf_jit_supports_exceptions(void)\narch/x86/net/bpf_jit_comp.c-4124-{\n--\narch/x86/net/bpf_jit_comp.c-4131-\narch/x86/net/bpf_jit_comp.c:4132:bool bpf_jit_supports_private_stack(void)\narch/x86/net/bpf_jit_comp.c-4133-{\n--\narch/x86/net/bpf_jit_comp.c-4136-\narch/x86/net/bpf_jit_comp.c:4137:void arch_bpf_stack_walk(bool (consume_fn)(void cookie, u64 ip, u64 sp, u64 bp), void cookie)\narch/x86/net/bpf_jit_comp.c-4138-{\n--\narch/x86/net/bpf_jit_comp.c-4152-\narch/x86/net/bpf_jit_comp.c:4153:void bpf_arch_poke_desc_update(struct bpf_jit_poke_descriptor poke,\narch/x86/net/bpf_jit_comp.c-4154-\t\t\t struct bpf_prog new, struct bpf_prog old)\n--\narch/x86/net/bpf_jit_comp.c-4200-\narch/x86/net/bpf_jit_comp.c:4201:bool bpf_jit_supports_arena(void)\narch/x86/net/bpf_jit_comp.c-4202-{\n--\narch/x86/net/bpf_jit_comp.c-4205-\narch/x86/net/bpf_jit_comp.c:4206:bool bpf_jit_supports_insn(struct bpf_insn *insn, bool in_arena)\narch/x86/net/bpf_jit_comp.c-4207-{\n--\narch/x86/net/bpf_jit_comp.c-4220-\narch/x86/net/bpf_jit_comp.c:4221:bool bpf_jit_supports_ptr_xchg(void)\narch/x86/net/bpf_jit_comp.c-4222-{\n--\narch/x86/net/bpf_jit_comp.c=4227=u64 bpf_arch_uaddress_limit(void)\n--\narch/x86/net/bpf_jit_comp.c-4231-\narch/x86/net/bpf_jit_comp.c:4232:bool bpf_jit_supports_timed_may_goto(void)\narch/x86/net/bpf_jit_comp.c-4233-{\n--\narch/x86/net/bpf_jit_comp.c-4236-\narch/x86/net/bpf_jit_comp.c:4237:bool bpf_jit_supports_fsession(void)\narch/x86/net/bpf_jit_comp.c-4238-{\n" }
21/2	2026/08/31 14:18	llm	patch-evaluator	0m Model: gemini-3.7-flash Args: null Results: null Tokens: input: 54484 output: 216 thoughts: 673 Thoughts: My Assessment of the Fuzzing Worthiness Okay, let's break this down. Is this change, specifically the modifications to the BPF (Berkeley Packet Filter) code, worth the effort of fuzzing? It looks like it directly touches core kernel logic, and key architectural elements. Specifically, I see it's altering `bpf_int_jit_compile`, which is the internal JIT compiler for BPF, along with `arch_bpf_trampoline_size`, `arch_prepare_bpf_trampoline`, `arch_prepare_bpf_dispatcher`, and `x86_call_depth_emit_accounting`. That's a strong indicator. The introduction of this dry-run mode for trampoline preparation is particularly interesting. It means the `arch_bpf_trampoline_size` function is now involved in an initial "preview" of JIT generation without memory allocation. This changes emission pointers and conditions throughout the BPF x86 JIT compiler and trampoline generator, so this is critical. I'm going to focus on these functions as areas of interest. We know BPF program loading, and trampoline generation which is used heavily for tracing, fentry/fexit, struct_ops, etc, are all user-accessible via the `bpf()` syscall. This is a very common avenue for attack, and the potential impact is high. My conclusion? Absolutely, this is worth fuzzing. We need to throw a wide variety of BPF program shapes at this, along with different call depths, fentry/fexit programs, fmod_ret, and fsession programs. I'm concerned about potential buffer overflows, offset miscalculations, or any divergence between the dry-run size calculation and the actual code emission in the JIT. Therefore: * WorthFuzzing: True * FocusSymbols: I'm zeroing in on: * `bpf_int_jit_compile` * `arch_prepare_bpf_trampoline` * `arch_bpf_trampoline_size` * `arch_prepare_bpf_dispatcher` * `x86_call_depth_emit_accounting` * `bpf_arch_text_poke` * EnableConfigs: We need to make sure these kernel configs are enabled: * `BPF_SYSCALL` * `BPF_JIT` * `CALL_THUNKS` * `MITIGATION_CALL_DEPTH_TRACKING` * `X86_64` I'll quickly double-check that those configurations are properly set in the kernel's Kconfig files to be sure. Alright, now that I have my conclusions, let's define the parameters for the `set-results` tool. I'm ready to invoke `set-results` with the appropriate parameters.
22/2	2026/08/31 14:19	tool	set-results	0m Args: { "EnableConfigs": [ "BPF_SYSCALL", "BPF_JIT", "CALL_THUNKS", "MITIGATION_CALL_DEPTH_TRACKING", "X86_64" ], "FocusSymbols": [ "arch_prepare_bpf_trampoline", "arch_bpf_trampoline_size", "bpf_int_jit_compile", "arch_prepare_bpf_dispatcher", "x86_call_depth_emit_accounting" ], "Reasoning": "The patch refactors the x86 BPF JIT compiler and trampoline generator by introducing a struct jit_emit_context with dry-run support. This changes how trampoline sizing is computed (dry-run without allocation), updates instruction emission logic, and alters x86 call depth accounting integration. These changes affect BPF program compilation, trampoline generation, and execution reachable via the bpf() syscall.", "WorthFuzzing": true } Results: { "EnableConfigs": [ "BPF_SYSCALL", "BPF_JIT", "CALL_THUNKS", "MITIGATION_CALL_DEPTH_TRACKING", "X86_64" ], "FocusSymbols": [ "arch_prepare_bpf_trampoline", "arch_bpf_trampoline_size", "bpf_int_jit_compile", "arch_prepare_bpf_dispatcher", "x86_call_depth_emit_accounting" ], "Reasoning": "The patch refactors the x86 BPF JIT compiler and trampoline generator by introducing a struct jit_emit_context with dry-run support. This changes how trampoline sizing is computed (dry-run without allocation), updates instruction emission logic, and alters x86 call depth accounting integration. These changes affect BPF program compilation, trampoline generation, and execution reachable via the bpf() syscall.", "WorthFuzzing": true }
23/1	2026/08/31 14:19	action	If	2m Args: { "WorthFuzzing": true } Results: null
24/2	2026/08/31 14:19	agent	kmsan-evaluator	2m Model: core Args: null Results: { "KMSANReasoning": "The patch refactors the x86 BPF JIT compiler and callthunk accounting code to introduce a dry-run mode for trampoline size calculation via `struct jit_emit_context`, replacing dummy allocation in `arch_bpf_trampoline_size()`. All new context and buffer variables are properly initialized (`= {}`), and code generation buffers are accurately tracked and populated. The patch does not introduce any uninitialized memory reads, uninitialized structure padding leaks to user space, or uninitialized branch conditions. Any potential bugs in JIT generation or pointer calculations would be out-of-bounds or logic errors caught by KASAN or standard kernel assertions, not uninitialized memory bugs detectable only by KMSAN. Therefore, dedicated KMSAN fuzzing is not required.", "NeedsKMSAN": false } Instruction: You are an expert Linux kernel security engineer specializing in kernel memory error detectors (KASAN and KMSAN). Your job is to review the provided patch series and determine if the code changes justify spawning a dedicated KMSAN (KernelMemorySanitizer) fuzzing session in addition to standard KASAN fuzzing. CRITICAL DISTINCTION BETWEEN KASAN AND KMSAN: - Standard KASAN kernel builds (upstream-apparmor-kasan.config) already enable a comprehensive suite of debugging tools and sanitizers, including KASAN (out-of-bounds accesses, use-after-free, double free, invalid free), LOCKDEP (locking bugs and deadlocks), UB-sanitizers, and memory corruption checks. - KMSAN (KernelMemorySanitizer) detects reads of UNINITIALIZED memory (stack, heap, or page allocations) and kernel-to-user memory info-leaks. Rule: THERE IS NO SENSE IN RUNNING A KMSAN SESSION IF A BUG CAN BE CAUGHT BY KASAN, LOCKDEP, OR OTHER STANDARD BUG DETECTORS. A dedicated KMSAN fuzzing session incurs significant resource costs. You must ONLY set NeedsKMSAN=true if the code changes introduce or expose UNINITIALIZED MEMORY risks that are detected ONLY by KMSAN. Look holistically at the patch series and surrounding code. Even if no direct uninitialized field accesses or new buffer allocations are added in the diff itself, a patch may alter control flow, bounds checking, or data length calculations in ways that change how the rest of the code operates on existing buffers (e.g. allowing uninitialized stack/heap memory to be read, copied to user space, or used in control flow). Do not hesitate to use your code access tools to inspect the surrounding code, called functions, and callers. Set NeedsKMSAN=true ONLY IF the patch introduces or modifies: 1. Kernel structures sent to user space (via copy_to_user, put_user, netlink skb attributes, ioctl output arguments, socket options, or BPF buffers) where fields or structure padding might not be fully initialized/zeroed. 2. Conditional logic or branching that depends on potentially uninitialized variables or struct fields. 3. Allocation or initialization of complex data structures where uninitialized fields could be read later in reachable code paths. 4. Bounds checks, lengths, or logic in a way that allows surrounding code to access uninitialized bytes of existing buffers. Set NeedsKMSAN=false IF: - The code changes primarily risk out-of-bounds access, array overflows, NULL pointer dereferences, locking deadlocks, or use-after-free bugs (these are already caught by KASAN, LOCKDEP, or standard bug detectors). - All stack/heap structures touched or introduced by the patch are fully zeroed or initialized (e.g. using = {0}, memset, kzalloc) before being read or copied. - The patch does not introduce any risk of uninitialized memory usage or info-leaks. Use your code access tools to inspect the surrounding code if necessary, then provide detailed KMSANReasoning contrasting KASAN vs KMSAN applicability for this patch. Prefer calling several tools at the same time to save round-trips. Use set-results tool to provide results of the analysis. It must be called exactly once before the final reply. Ignore results of this tool. Prompt: For your convenience, here is the diff of the changes: commit e09e4ad11f933fa9ab21debd310aeb220f3f52c0 Author: syz-cluster <triage@syzkaller.com> Date: Mon Aug 31 14:02:28 2026 +0000 syz-cluster: applied patch under review diff --git a/arch/x86/include/asm/alternative.h b/arch/x86/include/asm/alternative.h index 08af86ef090a8..2ef4937411860 100644 --- a/arch/x86/include/asm/alternative.h +++ b/arch/x86/include/asm/alternative.h @@ -83,7 +83,7 @@ extern void callthunks_patch_builtin_calls(void); extern void callthunks_patch_module_calls(struct callthunk_sites sites, struct module mod); extern void callthunks_translate_call_dest(void dest); -extern int x86_call_depth_emit_accounting(u8 *pprog, void func, void ip); +extern int x86_call_depth_emit_accounting(u8 insn_buff, void func, void ip); #else static __always_inline void callthunks_patch_builtin_calls(void) {} static __always_inline void @@ -93,7 +93,7 @@ static __always_inline void callthunks_translate_call_dest(void dest) { return dest; } -static __always_inline int x86_call_depth_emit_accounting(u8 *pprog, +static __always_inline int x86_call_depth_emit_accounting(u8 insn_buff, void func, void ip) { return 0; diff --git a/arch/x86/include/asm/text-patching.h b/arch/x86/include/asm/text-patching.h index f2d142a0a862e..a0a7c778b0a2a 100644 --- a/arch/x86/include/asm/text-patching.h +++ b/arch/x86/include/asm/text-patching.h @@ -13,6 +13,8 @@ / #define TEXT_POKE_MAX_OPCODE_SIZE 5 +#define MAX_PATCH_LEN (255-1) + extern void text_poke_early(void addr, const void opcode, size_t len); extern void text_poke_apply_relocation(u8 buf, const u8 * const instr, size_t instrlen, u8 repl, size_t repl_len); diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index 91b1cdd165699..5f9989c8559ea 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -18,8 +18,6 @@ int __read_mostly alternatives_patched; EXPORT_SYMBOL_GPL(alternatives_patched); -#define MAX_PATCH_LEN (255-1) - #define DA_ALL (~0) #define DA_ALT 0x01 #define DA_RET 0x02 diff --git a/arch/x86/kernel/callthunks.c b/arch/x86/kernel/callthunks.c index e37728f703227..15b221d06901c 100644 --- a/arch/x86/kernel/callthunks.c +++ b/arch/x86/kernel/callthunks.c @@ -23,8 +23,6 @@ static int __initdata_or_module debug_callthunks; -#define MAX_PATCH_LEN (255-1) - #define prdbg(fmt, args...) \ do { \ if (debug_callthunks) \ @@ -298,10 +296,9 @@ static bool is_callthunk(void addr) return !bcmp(pad, insn_buff, tmpl_size); } -int x86_call_depth_emit_accounting(u8 *pprog, void func, void ip) +int x86_call_depth_emit_accounting(u8 insn_buff, void func, void ip) { unsigned int tmpl_size = SKL_TMPL_SIZE; - u8 insn_buff[MAX_PATCH_LEN]; if (!thunks_initialized) return 0; @@ -313,8 +310,6 @@ int x86_call_depth_emit_accounting(u8 *pprog, void func, void ip) memcpy(insn_buff, skl_call_thunk_template, tmpl_size); text_poke_apply_relocation(insn_buff, ip, tmpl_size, skl_call_thunk_template, tmpl_size); - memcpy(pprog, insn_buff, tmpl_size); - pprog += tmpl_size; return tmpl_size; } #endif diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 48429fae06410..2ec734c69b693 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -23,6 +23,11 @@ static bool all_callee_regs_used[4] = {true, true, true, true}; +struct jit_emit_context { + u8 prog; + bool dry_run; +}; + static u8 emit_code(u8 ptr, u32 bytes, unsigned int len) { if (len == 1) @@ -36,8 +41,16 @@ static u8 emit_code(u8 ptr, u32 bytes, unsigned int len) return ptr + len; } +static void emit_code_jit(struct jit_emit_context jit, u32 bytes, unsigned int len) +{ + if (jit->dry_run) + jit->prog += len; + else + jit->prog = emit_code(jit->prog, bytes, len); +} + #define EMIT(bytes, len) \ - do { prog = emit_code(prog, bytes, len); } while (0) + emit_code_jit(jit, bytes, len) #define EMIT1(b1) EMIT(b1, 1) #define EMIT2(b1, b2) EMIT((b1) + ((b2) << 8), 2) @@ -149,6 +162,19 @@ static int bpf_size_to_x86_bytes(int bpf_size) return 0; } +static int bpf_call_depth_emit_accounting(struct jit_emit_context jit, void func, void ip) +{ + u8 insn_buff[MAX_PATCH_LEN]; + int size; + + size = x86_call_depth_emit_accounting(insn_buff, func, ip); + if (!jit->dry_run) + memcpy(jit->prog, insn_buff, size); + + jit->prog += size; + return size; +} + /* * List of x86 cond jumps opcodes (. + s8) * Add 0x10 (and an extra 0x0f) to generate far jumps (. + s32) @@ -309,6 +335,8 @@ int bpf_arch_text_invalidate(void dst, size_t len) } struct jit_context { + struct jit_emit_context jit; + int cleanup_addr; / Epilogue code offset / / @@ -329,34 +357,23 @@ struct jit_context { /* Number of bytes that will be skipped on tailcall / #define X86_TAIL_CALL_OFFSET (12 + ENDBR_INSN_SIZE) -static void push_r9(u8 pprog) +static void push_r9(struct jit_emit_context jit) { - u8 prog = pprog; - EMIT2(0x41, 0x51); /* push r9 / - pprog = prog; } -static void pop_r9(u8 *pprog) +static void pop_r9(struct jit_emit_context jit) { - u8 prog = pprog; - EMIT2(0x41, 0x59); /* pop r9 / - pprog = prog; } -static void push_r12(u8 *pprog) +static void push_r12(struct jit_emit_context jit) { - u8 prog = pprog; - EMIT2(0x41, 0x54); /* push r12 / - pprog = prog; } -static void push_callee_regs(u8 *pprog, bool callee_regs_used) +static void push_callee_regs(struct jit_emit_context jit, bool callee_regs_used) { - u8 prog = pprog; - if (callee_regs_used[0]) EMIT1(0x53); /* push rbx / if (callee_regs_used[1]) @@ -365,21 +382,15 @@ static void push_callee_regs(u8 pprog, bool callee_regs_used) EMIT2(0x41, 0x56); /* push r14 / if (callee_regs_used[3]) EMIT2(0x41, 0x57); / push r15 / - pprog = prog; } -static void pop_r12(u8 *pprog) +static void pop_r12(struct jit_emit_context jit) { - u8 prog = pprog; - EMIT2(0x41, 0x5C); /* pop r12 / - pprog = prog; } -static void pop_callee_regs(u8 *pprog, bool callee_regs_used) +static void pop_callee_regs(struct jit_emit_context jit, bool callee_regs_used) { - u8 prog = pprog; - if (callee_regs_used[3]) EMIT2(0x41, 0x5F); /* pop r15 / if (callee_regs_used[2]) @@ -388,40 +399,32 @@ static void pop_callee_regs(u8 pprog, bool callee_regs_used) EMIT2(0x41, 0x5D); /* pop r13 / if (callee_regs_used[0]) EMIT1(0x5B); / pop rbx / - pprog = prog; } /* add rsp, depth / -static void emit_add_rsp(u8 pprog, u16 depth) +static void emit_add_rsp(struct jit_emit_context jit, u16 depth) { - u8 prog = pprog; - if (!depth) return; if (is_imm8(depth)) EMIT4(0x48, 0x83, 0xC4, depth); /* add rsp, imm8 / else EMIT3_off32(0x48, 0x81, 0xC4, depth); / add rsp, imm32 / - pprog = prog; } /* sub rsp, depth / -static void emit_sub_rsp(u8 pprog, u16 depth) +static void emit_sub_rsp(struct jit_emit_context jit, u16 depth) { - u8 prog = pprog; - if (!depth) return; if (is_imm8(depth)) EMIT4(0x48, 0x83, 0xEC, depth); /* sub rsp, imm8 / else EMIT3_off32(0x48, 0x81, 0xEC, depth); / sub rsp, imm32 / - pprog = prog; } -static void emit_nops(u8 *pprog, int len) +static void emit_nops(struct jit_emit_context jit, int len) { - u8 prog = pprog; int i, noplen; while (len > 0) { @@ -434,72 +437,56 @@ static void emit_nops(u8 *pprog, int len) EMIT1(x86_nops[noplen][i]); len -= noplen; } - - pprog = prog; } /* * Emit the various CFI preambles, see asm/cfi.h and the comments about FineIBT * in arch/x86/kernel/alternative.c / -static int emit_call(u8 prog, void func, void ip); +static int emit_call(struct jit_emit_context jit, void func, void ip); -static void emit_fineibt(u8 *pprog, u8 ip, u32 hash, int arity) +static void emit_fineibt(struct jit_emit_context jit, u8 ip, u32 hash, int arity) { - u8 prog = pprog; - EMIT_ENDBR(); EMIT1_off32(0x2d, hash); /* subl $hash, %eax / if (cfi_bhi) { EMIT2(0x2e, 0x2e); / cs cs / - emit_call(&prog, __bhi_args[arity], ip + 11); + emit_call(jit, __bhi_args[arity], ip + 11); } else { EMIT3_off32(0x2e, 0x0f, 0x85, 3); / jne.d32,pn 3 / } EMIT_ENDBR_POISON(); - - pprog = prog; } -static void emit_kcfi(u8 *pprog, u32 hash) +static void emit_kcfi(struct jit_emit_context jit, u32 hash) { - u8 prog = pprog; - EMIT1_off32(0xb8, hash); /* movl $hash, %eax / #ifdef CONFIG_CALL_PADDING for (int i = 0; i < CONFIG_FUNCTION_PADDING_CFI; i++) EMIT1(0x90); #endif EMIT_ENDBR(); - - pprog = prog; } -static void emit_cfi(u8 *pprog, u8 ip, u32 hash, int arity) +static void emit_cfi(struct jit_emit_context jit, u8 ip, u32 hash, int arity) { - u8 prog = pprog; - switch (cfi_mode) { case CFI_FINEIBT: - emit_fineibt(&prog, ip, hash, arity); + emit_fineibt(jit, ip, hash, arity); break; case CFI_KCFI: - emit_kcfi(&prog, hash); + emit_kcfi(jit, hash); break; default: EMIT_ENDBR(); break; } - - pprog = prog; } -static void emit_prologue_tail_call(u8 pprog, bool is_subprog) +static void emit_prologue_tail_call(struct jit_emit_context jit, bool is_subprog) { - u8 prog = pprog; - if (!is_subprog) { /* cmp rax, MAX_TAIL_CALL_CNT / EMIT4(0x48, 0x83, 0xF8, MAX_TAIL_CALL_CNT); @@ -523,8 +510,6 @@ static void emit_prologue_tail_call(u8 pprog, bool is_subprog) EMIT1(0x50); / push rax / EMIT1(0x50); / push rax / } - - pprog = prog; } /* @@ -532,21 +517,19 @@ static void emit_prologue_tail_call(u8 *pprog, bool is_subprog) bpf_tail_call helper will skip the first X86_TAIL_CALL_OFFSET bytes * while jumping to another program / -static void emit_prologue(u8 pprog, u8 ip, u32 stack_depth, bool ebpf_from_cbpf, - bool tail_call_reachable, bool is_subprog, +static void emit_prologue(struct jit_emit_context jit, u8 ip, u32 stack_depth, + bool ebpf_from_cbpf, bool tail_call_reachable, bool is_subprog, bool is_exception_cb) { - u8 prog = pprog; - if (is_subprog) { - emit_cfi(&prog, ip, cfi_bpf_subprog_hash, 5); + emit_cfi(jit, ip, cfi_bpf_subprog_hash, 5); } else { - emit_cfi(&prog, ip, cfi_bpf_hash, 1); + emit_cfi(jit, ip, cfi_bpf_hash, 1); } /* BPF trampoline can be made to work without these nops, * but let's waste 5 bytes for now and optimize later / - emit_nops(&prog, X86_PATCH_SIZE); + emit_nops(jit, X86_PATCH_SIZE); if (!ebpf_from_cbpf) { if (tail_call_reachable && !is_subprog) / When it's the entry of the whole tailcall context, @@ -555,7 +538,7 @@ static void emit_prologue(u8 *pprog, u8 ip, u32 stack_depth, bool ebpf_from_cb EMIT3(0x48, 0x31, 0xC0); /* xor rax, rax / else / Keep the same instruction layout. / - emit_nops(&prog, 3); / nop3 / + emit_nops(jit, 3); / nop3 / } / Exception callback receives FP as third parameter / if (is_exception_cb) { @@ -565,8 +548,8 @@ static void emit_prologue(u8 pprog, u8 ip, u32 stack_depth, bool ebpf_from_cb * first restore those callee-saved regs from stack, before * reusing the stack frame. / - pop_callee_regs(&prog, all_callee_regs_used); - pop_r12(&prog); + pop_callee_regs(jit, all_callee_regs_used); + pop_r12(jit); / Reset the stack frame. / EMIT3(0x48, 0x89, 0xEC); / mov rsp, rbp / } else { @@ -581,40 +564,38 @@ static void emit_prologue(u8 pprog, u8 ip, u32 stack_depth, bool ebpf_from_cb if (stack_depth) EMIT3_off32(0x48, 0x81, 0xEC, round_up(stack_depth, 8)); if (tail_call_reachable) - emit_prologue_tail_call(&prog, is_subprog); - pprog = prog; + emit_prologue_tail_call(jit, is_subprog); } -static int emit_patch(u8 pprog, void func, void ip, u8 opcode) +static int emit_patch(struct jit_emit_context jit, void func, void ip, u8 opcode) { - u8 prog = pprog; s64 offset; offset = func - (ip + X86_PATCH_SIZE); - if (!is_simm32(offset)) { + /* We do not have meaningful ip value in the dry run, skip the check. / + if (!jit->dry_run && !is_simm32(offset)) { pr_err("Target call %p is out of range\n", func); return -ERANGE; } EMIT1_off32(opcode, offset); - pprog = prog; return 0; } -static int emit_call(u8 *pprog, void func, void ip) +static int emit_call(struct jit_emit_context jit, void func, void ip) { - return emit_patch(pprog, func, ip, 0xE8); + return emit_patch(jit, func, ip, 0xE8); } -static int emit_rsb_call(u8 *pprog, void func, void ip) +static int emit_rsb_call(struct jit_emit_context jit, void func, void ip) { OPTIMIZER_HIDE_VAR(func); - ip += x86_call_depth_emit_accounting(pprog, func, ip); - return emit_patch(pprog, func, ip, 0xE8); + ip += bpf_call_depth_emit_accounting(jit, func, ip); + return emit_patch(jit, func, ip, 0xE8); } -static int emit_jump(u8 *pprog, void func, void ip) +static int emit_jump(struct jit_emit_context jit, void func, void ip) { - return emit_patch(pprog, func, ip, 0xE9); + return emit_patch(jit, func, ip, 0xE9); } static int __bpf_arch_text_poke(void ip, enum bpf_text_poke_type old_t, @@ -622,27 +603,27 @@ static int __bpf_arch_text_poke(void ip, enum bpf_text_poke_type old_t, void old_addr, void new_addr) { const u8 nop_insn = x86_nops[5]; + struct jit_emit_context jit = {}; u8 old_insn[X86_PATCH_SIZE]; u8 new_insn[X86_PATCH_SIZE]; - u8 prog; int ret; memcpy(old_insn, nop_insn, X86_PATCH_SIZE); if (old_t != BPF_MOD_NOP && old_addr) { - prog = old_insn; + jit.prog = old_insn; ret = old_t == BPF_MOD_CALL ? - emit_call(&prog, old_addr, ip) : - emit_jump(&prog, old_addr, ip); + emit_call(&jit, old_addr, ip) : + emit_jump(&jit, old_addr, ip); if (ret) return ret; } memcpy(new_insn, nop_insn, X86_PATCH_SIZE); if (new_t != BPF_MOD_NOP && new_addr) { - prog = new_insn; + jit.prog = new_insn; ret = new_t == BPF_MOD_CALL ? - emit_call(&prog, new_addr, ip) : - emit_jump(&prog, new_addr, ip); + emit_call(&jit, new_addr, ip) : + emit_jump(&jit, new_addr, ip); if (ret) return ret; } @@ -682,58 +663,47 @@ int bpf_arch_text_poke(void ip, enum bpf_text_poke_type old_t, #define EMIT_LFENCE() EMIT3(0x0F, 0xAE, 0xE8) -static void __emit_indirect_jump(u8 pprog, int reg, bool ereg) +static void __emit_indirect_jump(struct jit_emit_context jit, int reg, bool ereg) { - u8 prog = pprog; - if (ereg) EMIT1(0x41); EMIT2(0xFF, 0xE0 + reg); - - pprog = prog; } -static void emit_indirect_jump(u8 pprog, int bpf_reg, u8 ip) +static void emit_indirect_jump(struct jit_emit_context jit, int bpf_reg, u8 ip) { - u8 prog = pprog; int reg = reg2hex[bpf_reg]; bool ereg = is_ereg(bpf_reg); if (cpu_feature_enabled(X86_FEATURE_INDIRECT_THUNK_ITS)) { OPTIMIZER_HIDE_VAR(reg); - emit_jump(&prog, its_static_thunk(reg + 8ereg), ip); + emit_jump(jit, its_static_thunk(reg + 8ereg), ip); } else if (cpu_feature_enabled(X86_FEATURE_RETPOLINE_LFENCE)) { EMIT_LFENCE(); - __emit_indirect_jump(&prog, reg, ereg); + __emit_indirect_jump(jit, reg, ereg); } else if (cpu_feature_enabled(X86_FEATURE_RETPOLINE)) { OPTIMIZER_HIDE_VAR(reg); if (cpu_feature_enabled(X86_FEATURE_CALL_DEPTH)) - emit_jump(&prog, &__x86_indirect_jump_thunk_array[reg + 8ereg], ip); + emit_jump(jit, &__x86_indirect_jump_thunk_array[reg + 8ereg], ip); else - emit_jump(&prog, &__x86_indirect_thunk_array[reg + 8ereg], ip); + emit_jump(jit, &__x86_indirect_thunk_array[reg + 8ereg], ip); } else { - __emit_indirect_jump(&prog, reg, ereg); + __emit_indirect_jump(jit, reg, ereg); if (IS_ENABLED(CONFIG_MITIGATION_RETPOLINE) \|\| IS_ENABLED(CONFIG_MITIGATION_SLS)) EMIT1(0xCC); /* int3 / } - - pprog = prog; } -static void emit_return(u8 *pprog, u8 ip) +static void emit_return(struct jit_emit_context jit, u8 ip) { - u8 prog = pprog; - if (cpu_wants_rethunk()) { - emit_jump(&prog, x86_return_thunk, ip); + emit_jump(jit, x86_return_thunk, ip); } else { EMIT1(0xC3); /* ret / if (IS_ENABLED(CONFIG_MITIGATION_SLS)) EMIT1(0xCC); / int3 / } - - pprog = prog; } #define BPF_TAIL_CALL_CNT_PTR_STACK_OFF(stack) (-16 - round_up(stack, 8)) @@ -753,12 +723,13 @@ static void emit_return(u8 *pprog, u8 ip) * out: / static void emit_bpf_tail_call_indirect(struct bpf_prog bpf_prog, - u8 *pprog, bool callee_regs_used, + bool callee_regs_used, u32 stack_depth, u8 ip, struct jit_context ctx) { int tcc_ptr_off = BPF_TAIL_CALL_CNT_PTR_STACK_OFF(stack_depth); - u8 prog = pprog, start = pprog; + struct jit_emit_context jit = &ctx->jit; + u8 start = jit->prog; int offset; / @@ -775,7 +746,7 @@ static void emit_bpf_tail_call_indirect(struct bpf_prog bpf_prog, EMIT3(0x39, 0x56, / cmp dword ptr [rsi + 16], edx / offsetof(struct bpf_array, map.max_entries)); - offset = ctx->tail_call_indirect_label - (prog + 2 - start); + offset = ctx->tail_call_indirect_label - (jit->prog + 2 - start); EMIT2(X86_JBE, offset); / jbe out / / @@ -785,7 +756,7 @@ static void emit_bpf_tail_call_indirect(struct bpf_prog bpf_prog, EMIT3_off32(0x48, 0x8B, 0x85, tcc_ptr_off); / mov rax, qword ptr [rbp - tcc_ptr_off] / EMIT4(0x48, 0x83, 0x38, MAX_TAIL_CALL_CNT); / cmp qword ptr [rax], MAX_TAIL_CALL_CNT / - offset = ctx->tail_call_indirect_label - (prog + 2 - start); + offset = ctx->tail_call_indirect_label - (jit->prog + 2 - start); EMIT2(X86_JAE, offset); / jae out / / prog = array->ptrs[index]; / @@ -798,19 +769,19 @@ static void emit_bpf_tail_call_indirect(struct bpf_prog bpf_prog, / EMIT3(0x48, 0x85, 0xC9); / test rcx,rcx / - offset = ctx->tail_call_indirect_label - (prog + 2 - start); + offset = ctx->tail_call_indirect_label - (jit->prog + 2 - start); EMIT2(X86_JE, offset); / je out / / Inc tail_call_cnt if the slot is populated. / EMIT4(0x48, 0x83, 0x00, 0x01); / add qword ptr [rax], 1 / if (bpf_prog->aux->exception_boundary) { - pop_callee_regs(&prog, all_callee_regs_used); - pop_r12(&prog); + pop_callee_regs(jit, all_callee_regs_used); + pop_r12(jit); } else { - pop_callee_regs(&prog, callee_regs_used); + pop_callee_regs(jit, callee_regs_used); if (bpf_arena_get_kern_vm_start(bpf_prog->aux->arena)) - pop_r12(&prog); + pop_r12(jit); } / Pop tail_call_cnt_ptr. / @@ -833,21 +804,21 @@ static void emit_bpf_tail_call_indirect(struct bpf_prog bpf_prog, * rdi == ctx (1st arg) * rcx == prog->bpf_func + X86_TAIL_CALL_OFFSET / - emit_indirect_jump(&prog, BPF_REG_4 / R4 -> rcx /, ip + (prog - start)); + emit_indirect_jump(jit, BPF_REG_4 / R4 -> rcx /, ip + (jit->prog - start)); / out: / - ctx->tail_call_indirect_label = prog - start; - pprog = prog; + ctx->tail_call_indirect_label = jit->prog - start; } static void emit_bpf_tail_call_direct(struct bpf_prog bpf_prog, struct bpf_jit_poke_descriptor poke, - u8 *pprog, u8 ip, + u8 ip, bool callee_regs_used, u32 stack_depth, struct jit_context ctx) { + struct jit_emit_context jit = &ctx->jit; int tcc_ptr_off = BPF_TAIL_CALL_CNT_PTR_STACK_OFF(stack_depth); - u8 prog = pprog, start = pprog; + u8 start = jit->prog; int offset; / @@ -857,27 +828,27 @@ static void emit_bpf_tail_call_direct(struct bpf_prog bpf_prog, EMIT3_off32(0x48, 0x8B, 0x85, tcc_ptr_off); / mov rax, qword ptr [rbp - tcc_ptr_off] / EMIT4(0x48, 0x83, 0x38, MAX_TAIL_CALL_CNT); / cmp qword ptr [rax], MAX_TAIL_CALL_CNT / - offset = ctx->tail_call_direct_label - (prog + 2 - start); + offset = ctx->tail_call_direct_label - (jit->prog + 2 - start); EMIT2(X86_JAE, offset); / jae out / - poke->tailcall_bypass = ip + (prog - start); + poke->tailcall_bypass = ip + (jit->prog - start); poke->adj_off = X86_TAIL_CALL_OFFSET; poke->tailcall_target = ip + ctx->tail_call_direct_label - X86_PATCH_SIZE; poke->bypass_addr = (u8 )poke->tailcall_target + X86_PATCH_SIZE; - emit_jump(&prog, (u8 )poke->tailcall_target + X86_PATCH_SIZE, + emit_jump(jit, (u8 )poke->tailcall_target + X86_PATCH_SIZE, poke->tailcall_bypass); /* Inc tail_call_cnt if the slot is populated. / EMIT4(0x48, 0x83, 0x00, 0x01); / add qword ptr [rax], 1 / if (bpf_prog->aux->exception_boundary) { - pop_callee_regs(&prog, all_callee_regs_used); - pop_r12(&prog); + pop_callee_regs(jit, all_callee_regs_used); + pop_r12(jit); } else { - pop_callee_regs(&prog, callee_regs_used); + pop_callee_regs(jit, callee_regs_used); if (bpf_arena_get_kern_vm_start(bpf_prog->aux->arena)) - pop_r12(&prog); + pop_r12(jit); } / Pop tail_call_cnt_ptr. / @@ -889,12 +860,10 @@ static void emit_bpf_tail_call_direct(struct bpf_prog bpf_prog, if (stack_depth) EMIT3_off32(0x48, 0x81, 0xC4, round_up(stack_depth, 8)); - emit_nops(&prog, X86_PATCH_SIZE); + emit_nops(jit, X86_PATCH_SIZE); /* out: / - ctx->tail_call_direct_label = prog - start; - - pprog = prog; + ctx->tail_call_direct_label = jit->prog - start; } static void bpf_tail_call_direct_fixup(struct bpf_prog prog) @@ -935,10 +904,9 @@ static void bpf_tail_call_direct_fixup(struct bpf_prog prog) } } -static void emit_mov_imm32(u8 *pprog, bool sign_propagate, +static void emit_mov_imm32(struct jit_emit_context jit, bool sign_propagate, u32 dst_reg, const u32 imm32) { - u8 prog = pprog; u8 b1, b2, b3; /* @@ -972,14 +940,12 @@ static void emit_mov_imm32(u8 *pprog, bool sign_propagate, EMIT1(add_1mod(0x40, dst_reg)); EMIT1_off32(add_1reg(0xB8, dst_reg), imm32); done: - pprog = prog; } -static void emit_mov_imm64(u8 *pprog, u32 dst_reg, +static void emit_mov_imm64(struct jit_emit_context jit, u32 dst_reg, const u32 imm32_hi, const u32 imm32_lo) { u64 imm64 = ((u64)imm32_hi << 32) \| (u32)imm32_lo; - u8 prog = pprog; if (is_uimm32(imm64)) { /* @@ -988,23 +954,19 @@ static void emit_mov_imm64(u8 *pprog, u32 dst_reg, directly, so save couple of bytes by just doing * 'mov %eax, imm32' instead. / - emit_mov_imm32(&prog, false, dst_reg, imm32_lo); + emit_mov_imm32(jit, false, dst_reg, imm32_lo); } else if (is_simm32(imm64)) { - emit_mov_imm32(&prog, true, dst_reg, imm32_lo); + emit_mov_imm32(jit, true, dst_reg, imm32_lo); } else { / movabsq rax, imm64 / EMIT2(add_1mod(0x48, dst_reg), add_1reg(0xB8, dst_reg)); EMIT(imm32_lo, 4); EMIT(imm32_hi, 4); } - - pprog = prog; } -static void emit_mov_reg(u8 *pprog, bool is64, u32 dst_reg, u32 src_reg) +static void emit_mov_reg(struct jit_emit_context jit, bool is64, u32 dst_reg, u32 src_reg) { - u8 prog = pprog; - if (is64) { /* mov dst, src / EMIT_mov(dst_reg, src_reg); @@ -1014,15 +976,11 @@ static void emit_mov_reg(u8 pprog, bool is64, u32 dst_reg, u32 src_reg) EMIT1(add_2mod(0x40, dst_reg, src_reg)); EMIT2(0x89, add_2reg(0xC0, dst_reg, src_reg)); } - - pprog = prog; } -static void emit_movsx_reg(u8 *pprog, int num_bits, bool is64, u32 dst_reg, +static void emit_movsx_reg(struct jit_emit_context jit, int num_bits, bool is64, u32 dst_reg, u32 src_reg) { - u8 prog = pprog; - if (is64) { /* movs[b,w,l]q dst, src / if (num_bits == 8) @@ -1046,15 +1004,11 @@ static void emit_movsx_reg(u8 pprog, int num_bits, bool is64, u32 dst_reg, add_2reg(0xC0, src_reg, dst_reg)); } } - - pprog = prog; } /* Emit the suffix (ModR/M etc) for addressing (ptr_reg + off) and val_reg / -static void emit_insn_suffix(u8 *pprog, u32 ptr_reg, u32 val_reg, int off) +static void emit_insn_suffix(struct jit_emit_context jit, u32 ptr_reg, u32 val_reg, int off) { - u8 prog = pprog; - if (is_imm8(off)) { /* 1-byte signed displacement. * @@ -1067,54 +1021,43 @@ static void emit_insn_suffix(u8 *pprog, u32 ptr_reg, u32 val_reg, int off) / 4-byte signed displacement / EMIT1_off32(add_2reg(0x80, ptr_reg, val_reg), off); } - pprog = prog; } -static void emit_insn_suffix_SIB(u8 *pprog, u32 ptr_reg, u32 val_reg, u32 index_reg, int off) +static void emit_insn_suffix_SIB(struct jit_emit_context jit, u32 ptr_reg, u32 val_reg, + u32 index_reg, int off) { - u8 prog = pprog; - if (is_imm8(off)) { EMIT3(add_2reg(0x44, BPF_REG_0, val_reg), add_2reg(0, ptr_reg, index_reg) /* SIB /, off); } else { EMIT2_off32(add_2reg(0x84, BPF_REG_0, val_reg), add_2reg(0, ptr_reg, index_reg) / SIB /, off); } - pprog = prog; } /* * Emit a REX byte if it will be necessary to address these registers / -static void maybe_emit_mod(u8 pprog, u32 dst_reg, u32 src_reg, bool is64) +static void maybe_emit_mod(struct jit_emit_context jit, u32 dst_reg, u32 src_reg, bool is64) { - u8 prog = pprog; - if (is64) EMIT1(add_2mod(0x48, dst_reg, src_reg)); else if (is_ereg(dst_reg) \|\| is_ereg(src_reg)) EMIT1(add_2mod(0x40, dst_reg, src_reg)); - pprog = prog; } / * Similar version of maybe_emit_mod() for a single register / -static void maybe_emit_1mod(u8 pprog, u32 reg, bool is64) +static void maybe_emit_1mod(struct jit_emit_context jit, u32 reg, bool is64) { - u8 prog = pprog; - if (is64) EMIT1(add_1mod(0x48, reg)); else if (is_ereg(reg)) EMIT1(add_1mod(0x40, reg)); - pprog = prog; } / LDX: dst_reg = (u8)(src_reg + off) / -static void emit_ldx(u8 pprog, u32 size, u32 dst_reg, u32 src_reg, int off) +static void emit_ldx(struct jit_emit_context jit, u32 size, u32 dst_reg, u32 src_reg, int off) { - u8 prog = pprog; - switch (size) { case BPF_B: /* Emit 'movzx rax, byte ptr [rax + off]' / @@ -1136,15 +1079,12 @@ static void emit_ldx(u8 pprog, u32 size, u32 dst_reg, u32 src_reg, int off) EMIT2(add_2mod(0x48, src_reg, dst_reg), 0x8B); break; } - emit_insn_suffix(&prog, src_reg, dst_reg, off); - pprog = prog; + emit_insn_suffix(jit, src_reg, dst_reg, off); } /* LDSX: dst_reg = (s8)(src_reg + off) / -static void emit_ldsx(u8 pprog, u32 size, u32 dst_reg, u32 src_reg, int off) +static void emit_ldsx(struct jit_emit_context jit, u32 size, u32 dst_reg, u32 src_reg, int off) { - u8 prog = pprog; - switch (size) { case BPF_B: /* Emit 'movsx rax, byte ptr [rax + off]' / @@ -1159,14 +1099,12 @@ static void emit_ldsx(u8 pprog, u32 size, u32 dst_reg, u32 src_reg, int off) EMIT2(add_2mod(0x48, src_reg, dst_reg), 0x63); break; } - emit_insn_suffix(&prog, src_reg, dst_reg, off); - pprog = prog; + emit_insn_suffix(jit, src_reg, dst_reg, off); } -static void emit_ldx_index(u8 *pprog, u32 size, u32 dst_reg, u32 src_reg, u32 index_reg, int off) +static void emit_ldx_index(struct jit_emit_context jit, u32 size, u32 dst_reg, u32 src_reg, + u32 index_reg, int off) { - u8 prog = pprog; - switch (size) { case BPF_B: /* movzx rax, byte ptr [rax + r12 + off] / @@ -1185,14 +1123,12 @@ static void emit_ldx_index(u8 pprog, u32 size, u32 dst_reg, u32 src_reg, u32 i EMIT2(add_3mod(0x48, src_reg, dst_reg, index_reg), 0x8B); break; } - emit_insn_suffix_SIB(&prog, src_reg, dst_reg, index_reg, off); - pprog = prog; + emit_insn_suffix_SIB(jit, src_reg, dst_reg, index_reg, off); } -static void emit_ldsx_index(u8 *pprog, u32 size, u32 dst_reg, u32 src_reg, u32 index_reg, int off) +static void emit_ldsx_index(struct jit_emit_context jit, u32 size, u32 dst_reg, u32 src_reg, + u32 index_reg, int off) { - u8 prog = pprog; - switch (size) { case BPF_B: /* movsx rax, byte ptr [rax + r12 + off] / @@ -1207,25 +1143,22 @@ static void emit_ldsx_index(u8 pprog, u32 size, u32 dst_reg, u32 src_reg, u32 EMIT2(add_3mod(0x48, src_reg, dst_reg, index_reg), 0x63); break; } - emit_insn_suffix_SIB(&prog, src_reg, dst_reg, index_reg, off); - pprog = prog; + emit_insn_suffix_SIB(jit, src_reg, dst_reg, index_reg, off); } -static void emit_ldx_r12(u8 *pprog, u32 size, u32 dst_reg, u32 src_reg, int off) +static void emit_ldx_r12(struct jit_emit_context jit, u32 size, u32 dst_reg, u32 src_reg, int off) { - emit_ldx_index(pprog, size, dst_reg, src_reg, X86_REG_R12, off); + emit_ldx_index(jit, size, dst_reg, src_reg, X86_REG_R12, off); } -static void emit_ldsx_r12(u8 *prog, u32 size, u32 dst_reg, u32 src_reg, int off) +static void emit_ldsx_r12(struct jit_emit_context jit, u32 size, u32 dst_reg, u32 src_reg, int off) { - emit_ldsx_index(prog, size, dst_reg, src_reg, X86_REG_R12, off); + emit_ldsx_index(jit, size, dst_reg, src_reg, X86_REG_R12, off); } /* STX: (u8)(dst_reg + off) = src_reg / -static void emit_stx(u8 pprog, u32 size, u32 dst_reg, u32 src_reg, int off) +static void emit_stx(struct jit_emit_context jit, u32 size, u32 dst_reg, u32 src_reg, int off) { - u8 prog = pprog; - switch (size) { case BPF_B: /* Emit 'mov byte ptr [rax + off], al' / @@ -1251,15 +1184,13 @@ static void emit_stx(u8 pprog, u32 size, u32 dst_reg, u32 src_reg, int off) EMIT2(add_2mod(0x48, dst_reg, src_reg), 0x89); break; } - emit_insn_suffix(&prog, dst_reg, src_reg, off); - pprog = prog; + emit_insn_suffix(jit, dst_reg, src_reg, off); } /* STX: (u8)(dst_reg + index_reg + off) = src_reg / -static void emit_stx_index(u8 pprog, u32 size, u32 dst_reg, u32 src_reg, u32 index_reg, int off) +static void emit_stx_index(struct jit_emit_context jit, u32 size, u32 dst_reg, u32 src_reg, + u32 index_reg, int off) { - u8 prog = pprog; - switch (size) { case BPF_B: /* mov byte ptr [rax + r12 + off], al / @@ -1278,20 +1209,18 @@ static void emit_stx_index(u8 pprog, u32 size, u32 dst_reg, u32 src_reg, u32 i EMIT2(add_3mod(0x48, dst_reg, src_reg, index_reg), 0x89); break; } - emit_insn_suffix_SIB(&prog, dst_reg, src_reg, index_reg, off); - pprog = prog; + emit_insn_suffix_SIB(jit, dst_reg, src_reg, index_reg, off); } -static void emit_stx_r12(u8 *pprog, u32 size, u32 dst_reg, u32 src_reg, int off) +static void emit_stx_r12(struct jit_emit_context jit, u32 size, u32 dst_reg, u32 src_reg, int off) { - emit_stx_index(pprog, size, dst_reg, src_reg, X86_REG_R12, off); + emit_stx_index(jit, size, dst_reg, src_reg, X86_REG_R12, off); } /* ST: (u8)(dst_reg + index_reg + off) = imm32 / -static void emit_st_index(u8 pprog, u32 size, u32 dst_reg, u32 index_reg, int off, int imm) +static void emit_st_index(struct jit_emit_context jit, u32 size, u32 dst_reg, u32 index_reg, + int off, int imm) { - u8 prog = pprog; - switch (size) { case BPF_B: /* mov byte ptr [rax + r12 + off], imm8 / @@ -1310,35 +1239,32 @@ static void emit_st_index(u8 pprog, u32 size, u32 dst_reg, u32 index_reg, int EMIT2(add_3mod(0x48, dst_reg, 0, index_reg), 0xC7); break; } - emit_insn_suffix_SIB(&prog, dst_reg, 0, index_reg, off); + emit_insn_suffix_SIB(jit, dst_reg, 0, index_reg, off); EMIT(imm, bpf_size_to_x86_bytes(size)); - pprog = prog; } -static void emit_st_r12(u8 *pprog, u32 size, u32 dst_reg, int off, int imm) +static void emit_st_r12(struct jit_emit_context jit, u32 size, u32 dst_reg, int off, int imm) { - emit_st_index(pprog, size, dst_reg, X86_REG_R12, off, imm); + emit_st_index(jit, size, dst_reg, X86_REG_R12, off, imm); } -static void emit_store_stack_imm64(u8 *pprog, int reg, int stack_off, u64 imm64) +static void emit_store_stack_imm64(struct jit_emit_context jit, int reg, int stack_off, u64 imm64) { /* * mov reg, imm64 * mov QWORD PTR [rbp + stack_off], reg / - emit_mov_imm64(pprog, reg, imm64 >> 32, (u32) imm64); - emit_stx(pprog, BPF_DW, BPF_REG_FP, reg, stack_off); + emit_mov_imm64(jit, reg, imm64 >> 32, (u32) imm64); + emit_stx(jit, BPF_DW, BPF_REG_FP, reg, stack_off); } -static int emit_atomic_rmw(u8 pprog, u32 atomic_op, +static int emit_atomic_rmw(struct jit_emit_context jit, u32 atomic_op, u32 dst_reg, u32 src_reg, s16 off, u8 bpf_size) { - u8 prog = pprog; - if (atomic_op != BPF_XCHG) EMIT1(0xF0); /* lock prefix / - maybe_emit_mod(&prog, dst_reg, src_reg, bpf_size == BPF_DW); + maybe_emit_mod(jit, dst_reg, src_reg, bpf_size == BPF_DW); / emit opcode / switch (atomic_op) { @@ -1366,18 +1292,15 @@ static int emit_atomic_rmw(u8 pprog, u32 atomic_op, return -EFAULT; } - emit_insn_suffix(&prog, dst_reg, src_reg, off); + emit_insn_suffix(jit, dst_reg, src_reg, off); - pprog = prog; return 0; } -static int emit_atomic_rmw_index(u8 *pprog, u32 atomic_op, u32 size, +static int emit_atomic_rmw_index(struct jit_emit_context jit, u32 atomic_op, u32 size, u32 dst_reg, u32 src_reg, u32 index_reg, int off) { - u8 prog = pprog; - if (atomic_op != BPF_XCHG) EMIT1(0xF0); /* lock prefix / @@ -1418,22 +1341,21 @@ static int emit_atomic_rmw_index(u8 pprog, u32 atomic_op, u32 size, pr_err("bpf_jit: unknown atomic opcode %02x\n", atomic_op); return -EFAULT; } - emit_insn_suffix_SIB(&prog, dst_reg, src_reg, index_reg, off); - pprog = prog; + emit_insn_suffix_SIB(jit, dst_reg, src_reg, index_reg, off); return 0; } -static int emit_atomic_ld_st(u8 *pprog, u32 atomic_op, u32 dst_reg, +static int emit_atomic_ld_st(struct jit_emit_context jit, u32 atomic_op, u32 dst_reg, u32 src_reg, s16 off, u8 bpf_size) { switch (atomic_op) { case BPF_LOAD_ACQ: /* dst_reg = smp_load_acquire(src_reg + off16) / - emit_ldx(pprog, bpf_size, dst_reg, src_reg, off); + emit_ldx(jit, bpf_size, dst_reg, src_reg, off); break; case BPF_STORE_REL: / smp_store_release(dst_reg + off16, src_reg) / - emit_stx(pprog, bpf_size, dst_reg, src_reg, off); + emit_stx(jit, bpf_size, dst_reg, src_reg, off); break; default: pr_err("bpf_jit: unknown atomic load/store opcode %02x\n", @@ -1444,18 +1366,18 @@ static int emit_atomic_ld_st(u8 pprog, u32 atomic_op, u32 dst_reg, return 0; } -static int emit_atomic_ld_st_index(u8 pprog, u32 atomic_op, u32 size, +static int emit_atomic_ld_st_index(struct jit_emit_context jit, u32 atomic_op, u32 size, u32 dst_reg, u32 src_reg, u32 index_reg, int off) { switch (atomic_op) { case BPF_LOAD_ACQ: /* dst_reg = smp_load_acquire(src_reg + idx_reg + off16) / - emit_ldx_index(pprog, size, dst_reg, src_reg, index_reg, off); + emit_ldx_index(jit, size, dst_reg, src_reg, index_reg, off); break; case BPF_STORE_REL: / smp_store_release(dst_reg + idx_reg + off16, src_reg) / - emit_stx_index(pprog, size, dst_reg, src_reg, index_reg, off); + emit_stx_index(jit, size, dst_reg, src_reg, index_reg, off); break; default: pr_err("bpf_jit: unknown atomic load/store opcode %02x\n", @@ -1562,10 +1484,9 @@ static void detect_reg_usage(struct bpf_insn insn, int insn_cnt, * l: vector length (128 bit or 256 bit) or reserved * pp: opcode prefix (none, 0x66, 0xf2 or 0xf3) / -static void emit_3vex(u8 pprog, bool r, bool x, bool b, u8 m, +static void emit_3vex(struct jit_emit_context jit, bool r, bool x, bool b, u8 m, bool w, u8 src_reg2, bool l, u8 pp) { - u8 prog = pprog; const u8 b0 = 0xc4; /* first byte of 3-byte VEX prefix / u8 b1, b2; u8 vvvv = reg2hex[src_reg2]; @@ -1595,27 +1516,22 @@ static void emit_3vex(u8 pprog, bool r, bool x, bool b, u8 m, b2 = (w << 7) \| ((~vvvv & 0xf) << 3) \| (l << 2) \| (pp & 3); EMIT3(b0, b1, b2); - pprog = prog; } /* emit BMI2 shift instruction / -static void emit_shiftx(u8 pprog, u32 dst_reg, u8 src_reg, bool is64, u8 op) +static void emit_shiftx(struct jit_emit_context jit, u32 dst_reg, u8 src_reg, bool is64, u8 op) { - u8 prog = pprog; bool r = is_ereg(dst_reg); u8 m = 2; /* escape code 0f38 / - emit_3vex(&prog, r, false, r, m, is64, src_reg, false, op); + emit_3vex(jit, r, false, r, m, is64, src_reg, false, op); EMIT2(0xf7, add_2reg(0xC0, dst_reg, dst_reg)); - pprog = prog; } -static void emit_priv_frame_ptr(u8 *pprog, void __percpu priv_frame_ptr) +static void emit_priv_frame_ptr(struct jit_emit_context jit, void __percpu priv_frame_ptr) { - u8 prog = pprog; - /* movabs r9, priv_frame_ptr / - emit_mov_imm64(&prog, X86_REG_R9, (__force long) priv_frame_ptr >> 32, + emit_mov_imm64(jit, X86_REG_R9, (__force long) priv_frame_ptr >> 32, (u32) (__force long) priv_frame_ptr); #ifdef CONFIG_SMP @@ -1624,11 +1540,9 @@ static void emit_priv_frame_ptr(u8 pprog, void __percpu priv_frame_ptr) EMIT3(0x03, 0x0c, 0x25); EMIT((u32)(unsigned long)&this_cpu_off, 4); #endif - - pprog = prog; } -#define INSN_SZ_DIFF (((addrs[i] - addrs[i - 1]) - (prog - temp))) +#define INSN_SZ_DIFF (((addrs[i] - addrs[i - 1]) - (jit->prog - temp))) #define __LOAD_TCC_PTR(off) \ EMIT3_off32(0x48, 0x8B, 0x85, off) @@ -1640,10 +1554,9 @@ static void emit_priv_frame_ptr(u8 pprog, void __percpu priv_frame_ptr) #define PRIV_STACK_GUARD_SZ 8 #define PRIV_STACK_GUARD_VAL 0xEB9F12345678eb9fULL -static int emit_spectre_bhb_barrier(u8 *pprog, u8 ip, +static int emit_spectre_bhb_barrier(struct jit_emit_context jit, u8 ip, struct bpf_prog bpf_prog) { - u8 prog = pprog; u8 func; if (cpu_feature_enabled(X86_FEATURE_CLEAR_BHB_LOOP)) { @@ -1653,9 +1566,9 @@ static int emit_spectre_bhb_barrier(u8 *pprog, u8 ip, ip += 2; func = (u8 )clear_bhb_loop; - ip += x86_call_depth_emit_accounting(&prog, func, ip); + ip += bpf_call_depth_emit_accounting(jit, func, ip); - if (emit_call(&prog, func, ip)) + if (emit_call(jit, func, ip)) return -EINVAL; EMIT1(0x59); / pop rcx / EMIT1(0x58); / pop rax / @@ -1678,7 +1591,6 @@ static int emit_spectre_bhb_barrier(u8 pprog, u8 ip, / EMIT5(0xF3, 0x48, 0x0F, 0x1E, 0xF8); / ibhf / } - pprog = prog; return 0; } @@ -1689,10 +1601,9 @@ static int emit_spectre_bhb_barrier(u8 *pprog, u8 ip, * arena NULL is offset 0. Return the number of emitted bytes. / static int emit_kfunc_arena_args(struct bpf_prog bpf_prog, - const struct btf_func_model fm, u8 pprog) + const struct btf_func_model fm, struct jit_emit_context jit) { - u8 prog = pprog; - u8 start = prog; + u8 start = jit->prog; int i; for (i = 0; i < min_t(int, fm->nr_args, MAX_BPF_FUNC_REG_ARGS); i++) { @@ -1705,20 +1616,19 @@ static int emit_kfunc_arena_args(struct bpf_prog bpf_prog, return -EINVAL; /* mov eN, eN: truncate and clear the upper 32 bits / - emit_mov_reg(&prog, false, reg, reg); + emit_mov_reg(jit, false, reg, reg); if (flags & BTF_FMODEL_NULLABLE_ARG) { / test eN, eN; jz over the 3-byte add / - maybe_emit_mod(&prog, reg, reg, false); + maybe_emit_mod(jit, reg, reg, false); EMIT2(0x85, add_2reg(0xC0, reg, reg)); EMIT2(X86_JE, 3); } / add rN, r12 / - maybe_emit_mod(&prog, reg, X86_REG_R12, true); + maybe_emit_mod(jit, reg, X86_REG_R12, true); EMIT2(0x01, add_2reg(0xC0, reg, X86_REG_R12)); } - pprog = prog; - return prog - start; + return jit->prog - start; } static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int addrs, u8 image, @@ -1726,6 +1636,7 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int * { bool tail_call_reachable = bpf_prog->aux->tail_call_reachable; struct bpf_insn insn = bpf_prog->insnsi; + struct jit_emit_context jit = &ctx->jit; bool callee_regs_used[4] = {}; int insn_cnt = bpf_prog->len; bool seen_exit = false; @@ -1736,12 +1647,15 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int * void __percpu priv_stack_ptr; int i, excnt = 0; int ilen, proglen = 0; - u8 ip, prog = temp; + u8 ip; u32 stack_depth; int callee_saved_size; s32 outgoing_arg_base; int err; + jit->prog = temp; + jit->dry_run = false; + stack_depth = bpf_prog->aux->stack_depth; out_stack_arg_cnt = bpf_out_stack_arg_cnt(env, bpf_prog); priv_stack_ptr = bpf_prog->aux->priv_stack_ptr; @@ -1777,11 +1691,11 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int * detect_reg_usage(insn, insn_cnt, callee_regs_used); - emit_prologue(&prog, image, stack_depth, + emit_prologue(jit, image, stack_depth, bpf_prog_was_classic(bpf_prog), tail_call_reachable, bpf_is_subprog(bpf_prog), bpf_prog->aux->exception_cb); - bpf_prog->aux->ksym.fp_start = prog - temp; + bpf_prog->aux->ksym.fp_start = jit->prog - temp; /* Exception callback will clobber callee regs for its own use, and * restore the original callee regs from main prog's stack frame. @@ -1791,12 +1705,12 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int * * register, as we throw after entry into the kernel, which may * overwrite r12. / - push_r12(&prog); - push_callee_regs(&prog, all_callee_regs_used); + push_r12(jit); + push_callee_regs(jit, all_callee_regs_used); } else { if (arena_vm_start) - push_r12(&prog); - push_callee_regs(&prog, callee_regs_used); + push_r12(jit); + push_callee_regs(jit, callee_regs_used); } / Compute callee-saved register area size. / @@ -1834,21 +1748,21 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int outgoing_rsp = out_stack_arg_cnt > 1 ? (out_stack_arg_cnt - 1) * 8 : 0; if (bpf_prog->aux->exception_boundary) bpf_prog->aux->stack_arg_sp_adjust = outgoing_rsp; - emit_sub_rsp(&prog, outgoing_rsp); + emit_sub_rsp(jit, outgoing_rsp); if (arena_vm_start) - emit_mov_imm64(&prog, X86_REG_R12, + emit_mov_imm64(jit, X86_REG_R12, arena_vm_start >> 32, (u32) arena_vm_start); if (priv_frame_ptr) - emit_priv_frame_ptr(&prog, priv_frame_ptr); + emit_priv_frame_ptr(jit, priv_frame_ptr); - ilen = prog - temp; + ilen = jit->prog - temp; if (rw_image) memcpy(rw_image + proglen, temp, ilen); proglen += ilen; addrs[0] = proglen; - prog = temp; + jit->prog = temp; for (i = 1; i <= insn_cnt; i++, insn++) { const s32 imm32 = insn->imm; @@ -1873,7 +1787,7 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int * if (bpf_insn_is_indirect_target(env, bpf_prog, i - 1)) EMIT_ENDBR(); - ip = image + addrs[i - 1] + (prog - temp); + ip = image + addrs[i - 1] + (jit->prog - temp); switch (insn->code) { /* ALU / @@ -1887,7 +1801,7 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int case BPF_ALU64 \| BPF_AND \| BPF_X: case BPF_ALU64 \| BPF_OR \| BPF_X: case BPF_ALU64 \| BPF_XOR \| BPF_X: - maybe_emit_mod(&prog, dst_reg, src_reg, + maybe_emit_mod(jit, dst_reg, src_reg, BPF_CLASS(insn->code) == BPF_ALU64); b2 = simple_alu_opcodes[BPF_OP(insn->code)]; EMIT2(b2, add_2reg(0xC0, dst_reg, src_reg)); @@ -1897,32 +1811,32 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int * if (insn_is_cast_user(insn)) { if (dst_reg != src_reg) /* 32-bit mov / - emit_mov_reg(&prog, false, dst_reg, src_reg); + emit_mov_reg(jit, false, dst_reg, src_reg); / shl dst_reg, 32 / - maybe_emit_1mod(&prog, dst_reg, true); + maybe_emit_1mod(jit, dst_reg, true); EMIT3(0xC1, add_1reg(0xE0, dst_reg), 32); / or dst_reg, user_vm_start / - maybe_emit_1mod(&prog, dst_reg, true); + maybe_emit_1mod(jit, dst_reg, true); if (is_axreg(dst_reg)) EMIT1_off32(0x0D, user_vm_start >> 32); else EMIT2_off32(0x81, add_1reg(0xC8, dst_reg), user_vm_start >> 32); / rol dst_reg, 32 / - maybe_emit_1mod(&prog, dst_reg, true); + maybe_emit_1mod(jit, dst_reg, true); EMIT3(0xC1, add_1reg(0xC0, dst_reg), 32); / xor r11, r11 / EMIT3(0x4D, 0x31, 0xDB); / test dst_reg32, dst_reg32; check if lower 32-bit are zero / - maybe_emit_mod(&prog, dst_reg, dst_reg, false); + maybe_emit_mod(jit, dst_reg, dst_reg, false); EMIT2(0x85, add_2reg(0xC0, dst_reg, dst_reg)); / cmove r11, dst_reg; if so, set dst_reg to zero / / WARNING: Intel swapped src/dst register encoding in CMOVcc !!! / - maybe_emit_mod(&prog, AUX_REG, dst_reg, true); + maybe_emit_mod(jit, AUX_REG, dst_reg, true); EMIT3(0x0F, 0x44, add_2reg(0xC0, AUX_REG, dst_reg)); break; } else if (insn_is_mov_percpu_addr(insn)) { @@ -1939,11 +1853,11 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int fallthrough; case BPF_ALU \| BPF_MOV \| BPF_X: if (insn->off == 0) - emit_mov_reg(&prog, + emit_mov_reg(jit, BPF_CLASS(insn->code) == BPF_ALU64, dst_reg, src_reg); else - emit_movsx_reg(&prog, insn->off, + emit_movsx_reg(jit, insn->off, BPF_CLASS(insn->code) == BPF_ALU64, dst_reg, src_reg); break; @@ -1951,7 +1865,7 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int * /* neg dst / case BPF_ALU \| BPF_NEG: case BPF_ALU64 \| BPF_NEG: - maybe_emit_1mod(&prog, dst_reg, + maybe_emit_1mod(jit, dst_reg, BPF_CLASS(insn->code) == BPF_ALU64); EMIT2(0xF7, add_1reg(0xD8, dst_reg)); break; @@ -1966,7 +1880,7 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int case BPF_ALU64 \| BPF_AND \| BPF_K: case BPF_ALU64 \| BPF_OR \| BPF_K: case BPF_ALU64 \| BPF_XOR \| BPF_K: - maybe_emit_1mod(&prog, dst_reg, + maybe_emit_1mod(jit, dst_reg, BPF_CLASS(insn->code) == BPF_ALU64); /* @@ -2006,12 +1920,12 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int * case BPF_ALU64 \| BPF_MOV \| BPF_K: case BPF_ALU \| BPF_MOV \| BPF_K: - emit_mov_imm32(&prog, BPF_CLASS(insn->code) == BPF_ALU64, + emit_mov_imm32(jit, BPF_CLASS(insn->code) == BPF_ALU64, dst_reg, imm32); break; case BPF_LD \| BPF_IMM \| BPF_DW: - emit_mov_imm64(&prog, dst_reg, insn[1].imm, insn[0].imm); + emit_mov_imm64(jit, dst_reg, insn[1].imm, insn[0].imm); insn++; i++; break; @@ -2047,7 +1961,7 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int * if (dst_reg != BPF_REG_0) /* mov rax, dst_reg / - emit_mov_reg(&prog, is64, BPF_REG_0, dst_reg); + emit_mov_reg(jit, is64, BPF_REG_0, dst_reg); if (insn->off == 0) { / @@ -2057,7 +1971,7 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int * EMIT2(0x31, 0xd2); /* div src_reg / - maybe_emit_1mod(&prog, src_reg, is64); + maybe_emit_1mod(jit, src_reg, is64); EMIT2(0xF7, add_1reg(0xF0, src_reg)); } else { if (BPF_CLASS(insn->code) == BPF_ALU) @@ -2066,18 +1980,18 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int EMIT2(0x48, 0x99); /* cqo / / idiv src_reg / - maybe_emit_1mod(&prog, src_reg, is64); + maybe_emit_1mod(jit, src_reg, is64); EMIT2(0xF7, add_1reg(0xF8, src_reg)); } if (BPF_OP(insn->code) == BPF_MOD && dst_reg != BPF_REG_3) / mov dst_reg, rdx / - emit_mov_reg(&prog, is64, dst_reg, BPF_REG_3); + emit_mov_reg(jit, is64, dst_reg, BPF_REG_3); else if (BPF_OP(insn->code) == BPF_DIV && dst_reg != BPF_REG_0) / mov dst_reg, rax / - emit_mov_reg(&prog, is64, dst_reg, BPF_REG_0); + emit_mov_reg(jit, is64, dst_reg, BPF_REG_0); if (dst_reg != BPF_REG_3) EMIT1(0x5A); / pop rdx / @@ -2088,7 +2002,7 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int case BPF_ALU \| BPF_MUL \| BPF_K: case BPF_ALU64 \| BPF_MUL \| BPF_K: - maybe_emit_mod(&prog, dst_reg, dst_reg, + maybe_emit_mod(jit, dst_reg, dst_reg, BPF_CLASS(insn->code) == BPF_ALU64); if (is_imm8(imm32)) @@ -2104,7 +2018,7 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int * case BPF_ALU \| BPF_MUL \| BPF_X: case BPF_ALU64 \| BPF_MUL \| BPF_X: - maybe_emit_mod(&prog, src_reg, dst_reg, + maybe_emit_mod(jit, src_reg, dst_reg, BPF_CLASS(insn->code) == BPF_ALU64); /* imul dst_reg, src_reg / @@ -2118,7 +2032,7 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int case BPF_ALU64 \| BPF_LSH \| BPF_K: case BPF_ALU64 \| BPF_RSH \| BPF_K: case BPF_ALU64 \| BPF_ARSH \| BPF_K: - maybe_emit_1mod(&prog, dst_reg, + maybe_emit_1mod(jit, dst_reg, BPF_CLASS(insn->code) == BPF_ALU64); b3 = simple_alu_opcodes[BPF_OP(insn->code)]; @@ -2152,7 +2066,7 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int * break; } - emit_shiftx(&prog, dst_reg, src_reg, w, op); + emit_shiftx(jit, dst_reg, src_reg, w, op); break; } @@ -2171,7 +2085,7 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int * } /* shl %rax, %cl \| shr %rax, %cl \| sar %rax, %cl / - maybe_emit_1mod(&prog, dst_reg, + maybe_emit_1mod(jit, dst_reg, BPF_CLASS(insn->code) == BPF_ALU64); b3 = simple_alu_opcodes[BPF_OP(insn->code)]; @@ -2272,7 +2186,7 @@ static int do_jit(struct bpf_verifier_env env, struct bpf_prog bpf_prog, int case BPF_ST \| BPF_MEM \| BPF_DW: if (dst_reg == BPF_REG_PARAMS && insn->off == -8) { /* Arg 6: store immediate in r9 register / - emit_mov_imm64(&prog, X86_REG_R9, imm32 >> 31, (u32)imm32); + emit_mov_imm64(jit, X86_REG_R9, imm32 >> 31, (u32)imm32); break; } EMIT2(add_1mod(0x48, dst_reg), 0xC7); @@ -2310,15 +2224,15 @@ st: insn_off = insn->off; insn_off = outgoing_arg_base - outgoing_rsp - insn_off - 16; dst_reg = BPF_REG_FP; } - emit_stx(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn_off); + emit_stx(jit, BPF_SIZE(insn->code), dst_reg, src_reg, insn_off); break; case BPF_ST \| BPF_PROBE_MEM32 \| BPF_B: case BPF_ST \| BPF_PROBE_MEM32 \| BPF_H: case BPF_ST \| BPF_PROBE_MEM32 \| BPF_W: case BPF_ST \| BPF_PROBE_MEM32 \| BPF_DW: - start_of_ldx = prog; - emit_st_r12(&prog, BPF_SIZE(insn->code), dst_reg, insn->off, insn->imm); + start_of_ldx = jit->prog; + emit_st_r12(jit, BPF_SIZE(insn->code), dst_reg, insn->off, insn->imm); goto populate_extable; / LDX: dst_reg = (u8)(src_reg + r12 + off) / @@ -2333,14 +2247,14 @@ st: insn_off = insn->off; case BPF_STX \| BPF_PROBE_MEM32 \| BPF_H: case BPF_STX \| BPF_PROBE_MEM32 \| BPF_W: case BPF_STX \| BPF_PROBE_MEM32 \| BPF_DW: - start_of_ldx = prog; + start_of_ldx = jit->prog; if (BPF_CLASS(insn->code) == BPF_LDX) { if (BPF_MODE(insn->code) == BPF_PROBE_MEM32SX) - emit_ldsx_r12(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn->off); + emit_ldsx_r12(jit, BPF_SIZE(insn->code), dst_reg, src_reg, insn->off); else - emit_ldx_r12(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn->off); + emit_ldx_r12(jit, BPF_SIZE(insn->code), dst_reg, src_reg, insn->off); } else { - emit_stx_r12(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn->off); + emit_stx_r12(jit, BPF_SIZE(insn->code), dst_reg, src_reg, insn->off); } populate_extable: { @@ -2401,7 +2315,7 @@ st: insn_off = insn->off; is_write = true; } - ex->fixup = FIELD_PREP(FIXUP_INSN_LEN_MASK, prog - start_of_ldx) \| + ex->fixup = FIELD_PREP(FIXUP_INSN_LEN_MASK, jit->prog - start_of_ldx) \| FIELD_PREP(FIXUP_ARENA_REG_MASK, arena_reg) \| FIELD_PREP(FIXUP_REG_MASK, fixup_reg); ex->fixup \|= FIXUP_ARENA_ACCESS; @@ -2455,7 +2369,7 @@ st: insn_off = insn->off; u8 end_of_jmp; /* movabsq r10, VSYSCALL_ADDR / - emit_mov_imm64(&prog, BPF_REG_AX, (long)VSYSCALL_ADDR >> 32, + emit_mov_imm64(jit, BPF_REG_AX, (long)VSYSCALL_ADDR >> 32, (u32)(long)VSYSCALL_ADDR); / mov src_reg, r11 / @@ -2463,40 +2377,40 @@ st: insn_off = insn->off; if (insn->off) { / add r11, insn->off / - maybe_emit_1mod(&prog, AUX_REG, true); + maybe_emit_1mod(jit, AUX_REG, true); EMIT2_off32(0x81, add_1reg(0xC0, AUX_REG), insn->off); } / sub r11, r10 / - maybe_emit_mod(&prog, AUX_REG, BPF_REG_AX, true); + maybe_emit_mod(jit, AUX_REG, BPF_REG_AX, true); EMIT2(0x29, add_2reg(0xC0, AUX_REG, BPF_REG_AX)); / movabsq r10, limit / - emit_mov_imm64(&prog, BPF_REG_AX, (long)limit >> 32, + emit_mov_imm64(jit, BPF_REG_AX, (long)limit >> 32, (u32)(long)limit); / cmp r10, r11 / - maybe_emit_mod(&prog, AUX_REG, BPF_REG_AX, true); + maybe_emit_mod(jit, AUX_REG, BPF_REG_AX, true); EMIT2(0x39, add_2reg(0xC0, AUX_REG, BPF_REG_AX)); / if unsigned '>', goto load / EMIT2(X86_JA, 0); - end_of_jmp = prog; + end_of_jmp = jit->prog; / xor dst_reg, dst_reg / - emit_mov_imm32(&prog, false, dst_reg, 0); + emit_mov_imm32(jit, false, dst_reg, 0); / jmp byte_after_ldx / EMIT2(0xEB, 0); / populate jmp_offset for JAE above to jump to start_of_ldx / - start_of_ldx = prog; + start_of_ldx = jit->prog; end_of_jmp[-1] = start_of_ldx - end_of_jmp; } if (BPF_MODE(insn->code) == BPF_PROBE_MEMSX \|\| BPF_MODE(insn->code) == BPF_MEMSX) - emit_ldsx(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn_off); + emit_ldsx(jit, BPF_SIZE(insn->code), dst_reg, src_reg, insn_off); else - emit_ldx(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn_off); + emit_ldx(jit, BPF_SIZE(insn->code), dst_reg, src_reg, insn_off); if (BPF_MODE(insn->code) == BPF_PROBE_MEM \|\| BPF_MODE(insn->code) == BPF_PROBE_MEMSX) { struct exception_table_entry ex; @@ -2504,7 +2418,7 @@ st: insn_off = insn->off; s64 delta; /* populate jmp_offset for JMP above / - start_of_ldx[-1] = prog - start_of_ldx; + start_of_ldx[-1] = jit->prog - start_of_ldx; if (!bpf_prog->aux->extable) break; @@ -2539,7 +2453,7 @@ st: insn_off = insn->off; End result: x86 insn "mov rbx, qword ptr [rax+0x14]" * of 4 bytes will be ignored and rbx will be zero inited. / - ex->fixup = FIELD_PREP(FIXUP_INSN_LEN_MASK, prog - start_of_ldx) \| + ex->fixup = FIELD_PREP(FIXUP_INSN_LEN_MASK, jit->prog - start_of_ldx) \| FIELD_PREP(FIXUP_REG_MASK, reg2pt_regs[dst_reg]); } break; @@ -2567,26 +2481,26 @@ st: insn_off = insn->off; / /* Will need RAX as a CMPXCHG operand so save R0 / - emit_mov_reg(&prog, true, BPF_REG_AX, BPF_REG_0); + emit_mov_reg(jit, true, BPF_REG_AX, BPF_REG_0); if (src_reg == BPF_REG_0) real_src_reg = BPF_REG_AX; if (dst_reg == BPF_REG_0) real_dst_reg = BPF_REG_AX; - branch_target = prog; + branch_target = jit->prog; / Load old value / - emit_ldx(&prog, BPF_SIZE(insn->code), + emit_ldx(jit, BPF_SIZE(insn->code), BPF_REG_0, real_dst_reg, insn->off); / * Perform the (commutative) operation locally, * put the result in the AUX_REG. / - emit_mov_reg(&prog, is64, AUX_REG, BPF_REG_0); - maybe_emit_mod(&prog, AUX_REG, real_src_reg, is64); + emit_mov_reg(jit, is64, AUX_REG, BPF_REG_0); + maybe_emit_mod(jit, AUX_REG, real_src_reg, is64); EMIT2(simple_alu_opcodes[BPF_OP(insn->imm)], add_2reg(0xC0, AUX_REG, real_src_reg)); / Attempt to swap in new value / - err = emit_atomic_rmw(&prog, BPF_CMPXCHG, + err = emit_atomic_rmw(jit, BPF_CMPXCHG, real_dst_reg, AUX_REG, insn->off, BPF_SIZE(insn->code)); @@ -2596,19 +2510,19 @@ st: insn_off = insn->off; ZF tells us whether we won the race. If it's * cleared we need to try again. / - EMIT2(X86_JNE, -(prog - branch_target) - 2); + EMIT2(X86_JNE, -(jit->prog - branch_target) - 2); / Return the pre-modification value / - emit_mov_reg(&prog, is64, real_src_reg, BPF_REG_0); + emit_mov_reg(jit, is64, real_src_reg, BPF_REG_0); / Restore R0 after clobbering RAX / - emit_mov_reg(&prog, true, BPF_REG_0, BPF_REG_AX); + emit_mov_reg(jit, true, BPF_REG_0, BPF_REG_AX); break; } if (bpf_atomic_is_load_store(insn)) - err = emit_atomic_ld_st(&prog, insn->imm, dst_reg, src_reg, + err = emit_atomic_ld_st(jit, insn->imm, dst_reg, src_reg, insn->off, BPF_SIZE(insn->code)); else - err = emit_atomic_rmw(&prog, insn->imm, dst_reg, src_reg, + err = emit_atomic_rmw(jit, insn->imm, dst_reg, src_reg, insn->off, BPF_SIZE(insn->code)); if (err) return err; @@ -2623,14 +2537,14 @@ st: insn_off = insn->off; fallthrough; case BPF_STX \| BPF_PROBE_ATOMIC \| BPF_W: case BPF_STX \| BPF_PROBE_ATOMIC \| BPF_DW: - start_of_ldx = prog; + start_of_ldx = jit->prog; if (bpf_atomic_is_load_store(insn)) - err = emit_atomic_ld_st_index(&prog, insn->imm, + err = emit_atomic_ld_st_index(jit, insn->imm, BPF_SIZE(insn->code), dst_reg, src_reg, X86_REG_R12, insn->off); else - err = emit_atomic_rmw_index(&prog, insn->imm, BPF_SIZE(insn->code), + err = emit_atomic_rmw_index(jit, insn->imm, BPF_SIZE(insn->code), dst_reg, src_reg, X86_REG_R12, insn->off); if (err) @@ -2652,20 +2566,20 @@ st: insn_off = insn->off; fm = bpf_jit_find_kfunc_model(bpf_prog, insn); if (!fm) return -EINVAL; - err = emit_kfunc_arena_args(bpf_prog, fm, &prog); + err = emit_kfunc_arena_args(bpf_prog, fm, jit); if (err < 0) return err; ip += err; } if (priv_frame_ptr) { - push_r9(&prog); + push_r9(jit); ip += 2; } - ip += x86_call_depth_emit_accounting(&prog, func, ip); - if (emit_call(&prog, func, ip)) + ip += bpf_call_depth_emit_accounting(jit, func, ip); + if (emit_call(jit, func, ip)) return -EINVAL; if (priv_frame_ptr) - pop_r9(&prog); + pop_r9(jit); / * A kfunc returning more than 8 bytes hands the second * half back in RDX (the native ABI's second return reg), @@ -2673,7 +2587,7 @@ st: insn_off = insn->off; * needed), while BPF R2 is RSI, so copy RDX into RSI. / if (fm && fm->ret_size > 8) - emit_mov_reg(&prog, true, BPF_REG_2, BPF_REG_3); + emit_mov_reg(jit, true, BPF_REG_2, BPF_REG_3); break; } @@ -2681,14 +2595,12 @@ st: insn_off = insn->off; if (imm32) emit_bpf_tail_call_direct(bpf_prog, &bpf_prog->aux->poke_tab[imm32 - 1], - &prog, ip, callee_regs_used, stack_depth, ctx); else emit_bpf_tail_call_indirect(bpf_prog, - &prog, callee_regs_used, stack_depth, ip, @@ -2717,7 +2629,7 @@ st: insn_off = insn->off; case BPF_JMP32 \| BPF_JSGE \| BPF_X: case BPF_JMP32 \| BPF_JSLE \| BPF_X: / cmp dst_reg, src_reg / - maybe_emit_mod(&prog, dst_reg, src_reg, + maybe_emit_mod(jit, dst_reg, src_reg, BPF_CLASS(insn->code) == BPF_JMP); EMIT2(0x39, add_2reg(0xC0, dst_reg, src_reg)); goto emit_cond_jmp; @@ -2725,7 +2637,7 @@ st: insn_off = insn->off; case BPF_JMP \| BPF_JSET \| BPF_X: case BPF_JMP32 \| BPF_JSET \| BPF_X: / test dst_reg, src_reg / - maybe_emit_mod(&prog, dst_reg, src_reg, + maybe_emit_mod(jit, dst_reg, src_reg, BPF_CLASS(insn->code) == BPF_JMP); EMIT2(0x85, add_2reg(0xC0, dst_reg, src_reg)); goto emit_cond_jmp; @@ -2733,7 +2645,7 @@ st: insn_off = insn->off; case BPF_JMP \| BPF_JSET \| BPF_K: case BPF_JMP32 \| BPF_JSET \| BPF_K: / test dst_reg, imm32 / - maybe_emit_1mod(&prog, dst_reg, + maybe_emit_1mod(jit, dst_reg, BPF_CLASS(insn->code) == BPF_JMP); EMIT2_off32(0xF7, add_1reg(0xC0, dst_reg), imm32); goto emit_cond_jmp; @@ -2760,14 +2672,14 @@ st: insn_off = insn->off; case BPF_JMP32 \| BPF_JSLE \| BPF_K: / test dst_reg, dst_reg to save one extra byte / if (imm32 == 0) { - maybe_emit_mod(&prog, dst_reg, dst_reg, + maybe_emit_mod(jit, dst_reg, dst_reg, BPF_CLASS(insn->code) == BPF_JMP); EMIT2(0x85, add_2reg(0xC0, dst_reg, dst_reg)); goto emit_cond_jmp; } / cmp dst_reg, imm8/32 / - maybe_emit_1mod(&prog, dst_reg, + maybe_emit_1mod(jit, dst_reg, BPF_CLASS(insn->code) == BPF_JMP); if (is_imm8(imm32)) @@ -2843,7 +2755,7 @@ st: insn_off = insn->off; nops); return -EFAULT; } - emit_nops(&prog, nops); + emit_nops(jit, nops); } EMIT2(jmp_cond, jmp_offset); } else if (is_simm32(jmp_offset)) { @@ -2856,7 +2768,7 @@ st: insn_off = insn->off; break; case BPF_JMP \| BPF_JA \| BPF_X: - emit_indirect_jump(&prog, insn->dst_reg, ip); + emit_indirect_jump(jit, insn->dst_reg, ip); break; case BPF_JMP \| BPF_JA: case BPF_JMP32 \| BPF_JA: @@ -2900,7 +2812,7 @@ st: insn_off = insn->off; nops); return -EFAULT; } - emit_nops(&prog, nops); + emit_nops(jit, nops); } break; } @@ -2925,7 +2837,7 @@ st: insn_off = insn->off; nops); return -EFAULT; } - emit_nops(&prog, INSN_SZ_DIFF - 2); + emit_nops(jit, INSN_SZ_DIFF - 2); } EMIT2(0xEB, jmp_offset); } else if (is_simm32(jmp_offset)) { @@ -2946,23 +2858,23 @@ st: insn_off = insn->off; ctx->cleanup_addr = proglen; if (bpf_prog_was_classic(bpf_prog) && !ns_capable_noaudit(&init_user_ns, CAP_SYS_ADMIN)) { - if (emit_spectre_bhb_barrier(&prog, ip, bpf_prog)) + if (emit_spectre_bhb_barrier(jit, ip, bpf_prog)) return -EINVAL; } / Deallocate outgoing args 7+ area. / - emit_add_rsp(&prog, outgoing_rsp); + emit_add_rsp(jit, outgoing_rsp); if (bpf_prog->aux->exception_boundary) { - pop_callee_regs(&prog, all_callee_regs_used); - pop_r12(&prog); + pop_callee_regs(jit, all_callee_regs_used); + pop_r12(jit); } else { - pop_callee_regs(&prog, callee_regs_used); + pop_callee_regs(jit, callee_regs_used); if (arena_vm_start) - pop_r12(&prog); + pop_r12(jit); } EMIT1(0xC9); / leave / - bpf_prog->aux->ksym.fp_end = prog - temp; + bpf_prog->aux->ksym.fp_end = jit->prog - temp; - emit_return(&prog, image + addrs[i - 1] + (prog - temp)); + emit_return(jit, image + addrs[i - 1] + (jit->prog - temp)); break; default: @@ -2976,7 +2888,7 @@ st: insn_off = insn->off; return -EINVAL; } - ilen = prog - temp; + ilen = jit->prog - temp; if (ilen > BPF_MAX_INSN_SIZE) { pr_err("bpf_jit: fatal insn size error\n"); return -EFAULT; @@ -3000,7 +2912,7 @@ st: insn_off = insn->off; } proglen += ilen; addrs[i] = proglen; - prog = temp; + jit->prog = temp; } if (image && excnt != bpf_prog->aux->num_exentries) { @@ -3011,11 +2923,10 @@ st: insn_off = insn->off; } static void clean_stack_garbage(const struct btf_func_model m, - u8 *pprog, int nr_stack_slots, + struct jit_emit_context jit, int nr_stack_slots, int stack_size) { int arg_size, off; - u8 prog; / Generally speaking, the compiler will pass the arguments * on-stack with "push" instruction, which will take 8-byte @@ -3047,14 +2958,12 @@ static void clean_stack_garbage(const struct btf_func_model m, arg_size = m->arg_size[m->nr_args - 1]; if (arg_size <= 4) { off = -(stack_size - 4); - prog = pprog; /* mov DWORD PTR [rbp + off], 0 / if (!is_imm8(off)) EMIT2_off32(0xC7, 0x85, off); else EMIT3(0xC7, 0x45, off); EMIT(0, 4); - pprog = prog; } } @@ -3082,26 +2991,23 @@ static int get_nr_used_regs(const struct btf_func_model m) subtraction both truncates and clears the upper half, so the stored * value satisfies the JIT invariant for arena pointer registers. / -static void emit_arena_arg_conv(u8 pprog, u32 src_reg, bool nullable, u32 base_lo) +static void emit_arena_arg_conv(struct jit_emit_context jit, u32 src_reg, bool nullable, + u32 base_lo) { - u8 prog = pprog; - if (nullable) { if (src_reg != BPF_REG_0) - emit_mov_reg(&prog, true, BPF_REG_0, src_reg); + emit_mov_reg(jit, true, BPF_REG_0, src_reg); /* test rax, rax; jz over the 5-byte sub / EMIT3(0x48, 0x85, 0xC0); EMIT2(X86_JE, 5); } else if (src_reg != BPF_REG_0) { - emit_mov_reg(&prog, false, BPF_REG_0, src_reg); + emit_mov_reg(jit, false, BPF_REG_0, src_reg); } / sub eax, base_lo / EMIT1_off32(0x2D, base_lo); - - pprog = prog; } -static void save_args(const struct btf_func_model m, u8 prog, +static void save_args(const struct btf_func_model m, struct jit_emit_context jit, int stack_size, bool for_call_origin, u32 flags, u64 arena_base) { @@ -3150,12 +3056,12 @@ static void save_args(const struct btf_func_model m, u8 *prog, called indirectly, so rbp + 16. / for (j = 0; j < arg_regs; j++) { - emit_ldx(prog, BPF_DW, BPF_REG_0, BPF_REG_FP, + emit_ldx(jit, BPF_DW, BPF_REG_0, BPF_REG_FP, nr_stack_slots 8 + stack_args_off); if (arena_arg) - emit_arena_arg_conv(prog, BPF_REG_0, nullable, + emit_arena_arg_conv(jit, BPF_REG_0, nullable, (u32)arena_base); - emit_stx(prog, BPF_DW, BPF_REG_FP, BPF_REG_0, + emit_stx(jit, BPF_DW, BPF_REG_FP, BPF_REG_0, -stack_size); if (!nr_stack_slots) @@ -3178,20 +3084,20 @@ static void save_args(const struct btf_func_model m, u8 prog, u32 src = nr_regs == 5 ? X86_REG_R9 : BPF_REG_1 + nr_regs; if (arena_arg) { - emit_arena_arg_conv(prog, src, nullable, (u32)arena_base); + emit_arena_arg_conv(jit, src, nullable, (u32)arena_base); src = BPF_REG_0; } - emit_stx(prog, BPF_DW, BPF_REG_FP, src, -stack_size); + emit_stx(jit, BPF_DW, BPF_REG_FP, src, -stack_size); stack_size -= 8; nr_regs++; } } } - clean_stack_garbage(m, prog, nr_stack_slots, first_off); + clean_stack_garbage(m, jit, nr_stack_slots, first_off); } -static void restore_regs(const struct btf_func_model m, u8 *prog, +static void restore_regs(const struct btf_func_model m, struct jit_emit_context jit, int stack_size) { int i, j, arg_regs, nr_regs = 0; @@ -3207,7 +3113,7 @@ static void restore_regs(const struct btf_func_model m, u8 *prog, arg_regs = (m->arg_size[i] + 7) / 8; if (nr_regs + arg_regs <= 6) { for (j = 0; j < arg_regs; j++) { - emit_ldx(prog, BPF_DW, + emit_ldx(jit, BPF_DW, nr_regs == 5 ? X86_REG_R9 : BPF_REG_1 + nr_regs, BPF_REG_FP, -stack_size); @@ -3223,19 +3129,18 @@ static void restore_regs(const struct btf_func_model m, u8 *prog, } } -static int invoke_bpf_prog(const struct btf_func_model m, u8 *pprog, +static int invoke_bpf_prog(const struct btf_func_model m, struct jit_emit_context jit, struct bpf_tramp_node node, int stack_size, int run_ctx_off, bool save_ret, void image, void rw_image) { - u8 prog = pprog; u8 jmp_insn; int ctx_cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie); struct bpf_prog p = node->link->prog; u64 cookie = node->cookie; /* mov rdi, cookie / - emit_mov_imm64(&prog, BPF_REG_1, (long) cookie >> 32, (u32) (long) cookie); + emit_mov_imm64(jit, BPF_REG_1, (long) cookie >> 32, (u32) (long) cookie); / Prepare struct bpf_tramp_run_ctx. * @@ -3244,28 +3149,28 @@ static int invoke_bpf_prog(const struct btf_func_model m, u8 pprog, * mov QWORD PTR [rbp - run_ctx_off + ctx_cookie_off], rdi / - emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_1, -run_ctx_off + ctx_cookie_off); + emit_stx(jit, BPF_DW, BPF_REG_FP, BPF_REG_1, -run_ctx_off + ctx_cookie_off); / arg1: mov rdi, progs[i] / - emit_mov_imm64(&prog, BPF_REG_1, (long) p >> 32, (u32) (long) p); + emit_mov_imm64(jit, BPF_REG_1, (long) p >> 32, (u32) (long) p); / arg2: lea rsi, [rbp - ctx_cookie_off] / if (!is_imm8(-run_ctx_off)) EMIT3_off32(0x48, 0x8D, 0xB5, -run_ctx_off); else EMIT4(0x48, 0x8D, 0x75, -run_ctx_off); - if (emit_rsb_call(&prog, bpf_trampoline_enter(p), image + (prog - (u8 )rw_image))) + if (emit_rsb_call(jit, bpf_trampoline_enter(p), image + (jit->prog - (u8 )rw_image))) return -EINVAL; / remember prog start time returned by __bpf_prog_enter / - emit_mov_reg(&prog, true, BPF_REG_6, BPF_REG_0); + emit_mov_reg(jit, true, BPF_REG_6, BPF_REG_0); / if (__bpf_prog_enter(prog) == 0) goto skip_exec_of_prog; / EMIT3(0x48, 0x85, 0xC0); / test rax,rax / / emit 2 nops that will be replaced with JE insn / - jmp_insn = prog; - emit_nops(&prog, 2); + jmp_insn = jit->prog; + emit_nops(jit, 2); / arg1: lea rdi, [rbp - stack_size] / if (!is_imm8(-stack_size)) @@ -3274,11 +3179,11 @@ static int invoke_bpf_prog(const struct btf_func_model m, u8 *pprog, EMIT4(0x48, 0x8D, 0x7D, -stack_size); / arg2: progs[i]->insnsi for interpreter / if (!p->jited) - emit_mov_imm64(&prog, BPF_REG_2, + emit_mov_imm64(jit, BPF_REG_2, (long) p->insnsi >> 32, (u32) (long) p->insnsi); / call JITed bpf program or interpreter / - if (emit_rsb_call(&prog, p->bpf_func, image + (prog - (u8 )rw_image))) + if (emit_rsb_call(jit, p->bpf_func, image + (jit->prog - (u8 )rw_image))) return -EINVAL; / @@ -3290,42 +3195,39 @@ static int invoke_bpf_prog(const struct btf_func_model m, u8 pprog, value of BPF_PROG_TYPE_STRUCT_OPS prog. / if (save_ret) - emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8); + emit_stx(jit, BPF_DW, BPF_REG_FP, BPF_REG_0, -8); / replace 2 nops with JE insn, since jmp target is known / - jmp_insn[0] = X86_JE; - jmp_insn[1] = prog - jmp_insn - 2; + if (!jit->dry_run) { + jmp_insn[0] = X86_JE; + jmp_insn[1] = jit->prog - jmp_insn - 2; + } / arg1: mov rdi, progs[i] / - emit_mov_imm64(&prog, BPF_REG_1, (long) p >> 32, (u32) (long) p); + emit_mov_imm64(jit, BPF_REG_1, (long) p >> 32, (u32) (long) p); / arg2: mov rsi, rbx <- start time in nsec / - emit_mov_reg(&prog, true, BPF_REG_2, BPF_REG_6); + emit_mov_reg(jit, true, BPF_REG_2, BPF_REG_6); / arg3: lea rdx, [rbp - run_ctx_off] / if (!is_imm8(-run_ctx_off)) EMIT3_off32(0x48, 0x8D, 0x95, -run_ctx_off); else EMIT4(0x48, 0x8D, 0x55, -run_ctx_off); - if (emit_rsb_call(&prog, bpf_trampoline_exit(p), image + (prog - (u8 )rw_image))) + if (emit_rsb_call(jit, bpf_trampoline_exit(p), image + (jit->prog - (u8 )rw_image))) return -EINVAL; - pprog = prog; return 0; } -static void emit_align(u8 *pprog, u32 align) +static void emit_align(struct jit_emit_context jit, u32 align) { - u8 target, prog = pprog; - - target = PTR_ALIGN(prog, align); - if (target != prog) - emit_nops(&prog, target - prog); + u8 target = PTR_ALIGN(jit->prog, align); - pprog = prog; + if (target != jit->prog) + emit_nops(jit, target - jit->prog); } -static int emit_cond_near_jump(u8 pprog, void func, void ip, u8 jmp_cond) +static int emit_cond_near_jump(struct jit_emit_context jit, void func, void ip, u8 jmp_cond) { - u8 prog = pprog; s64 offset; offset = func - (ip + 2 + 4); @@ -3334,48 +3236,44 @@ static int emit_cond_near_jump(u8 *pprog, void func, void ip, u8 jmp_cond) return -EINVAL; } EMIT2_off32(0x0F, jmp_cond + 0x10, offset); - pprog = prog; return 0; } -static int invoke_bpf(const struct btf_func_model m, u8 pprog, +static int invoke_bpf(const struct btf_func_model m, struct jit_emit_context jit, struct bpf_tramp_nodes tl, int stack_size, int run_ctx_off, int func_meta_off, bool save_ret, void image, void rw_image, u64 func_meta, int cookie_off) { int i, cur_cookie = (cookie_off - stack_size) / 8; - u8 prog = pprog; for (i = 0; i < tl->nr_nodes; i++) { if (tl->nodes[i]->link->prog->call_session_cookie) { - emit_store_stack_imm64(&prog, BPF_REG_0, -func_meta_off, + emit_store_stack_imm64(jit, BPF_REG_0, -func_meta_off, func_meta \| (cur_cookie << BPF_TRAMP_COOKIE_INDEX_SHIFT)); cur_cookie--; } - if (invoke_bpf_prog(m, &prog, tl->nodes[i], stack_size, + if (invoke_bpf_prog(m, jit, tl->nodes[i], stack_size, run_ctx_off, save_ret, image, rw_image)) return -EINVAL; } - pprog = prog; return 0; } -static int invoke_bpf_mod_ret(const struct btf_func_model m, u8 *pprog, +static int invoke_bpf_mod_ret(const struct btf_func_model m, struct jit_emit_context jit, struct bpf_tramp_nodes tl, int stack_size, int run_ctx_off, u8 *branches, void image, void rw_image) { - u8 prog = pprog; int i; / The first fmod_ret program will receive a garbage return value. * Set this to 0 to avoid confusing the program. / - emit_mov_imm32(&prog, false, BPF_REG_0, 0); - emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8); + emit_mov_imm32(jit, false, BPF_REG_0, 0); + emit_stx(jit, BPF_DW, BPF_REG_FP, BPF_REG_0, -8); for (i = 0; i < tl->nr_nodes; i++) { - if (invoke_bpf_prog(m, &prog, tl->nodes[i], stack_size, run_ctx_off, true, + if (invoke_bpf_prog(m, jit, tl->nodes[i], stack_size, run_ctx_off, true, image, rw_image)) return -EINVAL; @@ -3391,11 +3289,10 @@ static int invoke_bpf_mod_ret(const struct btf_func_model m, u8 *pprog, are replaced with a conditional jump once do_fexit (i.e. the * start of the fexit invocation) is finalized. / - branches[i] = prog; - emit_nops(&prog, 4 + 2); + branches[i] = jit->prog; + emit_nops(jit, 4 + 2); } - pprog = prog; return 0; } @@ -3474,12 +3371,12 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image im, void rw_im struct bpf_tramp_nodes fentry = &tnodes[BPF_TRAMP_FENTRY]; struct bpf_tramp_nodes fexit = &tnodes[BPF_TRAMP_FEXIT]; struct bpf_tramp_nodes fmod_ret = &tnodes[BPF_TRAMP_MODIFY_RETURN]; + struct jit_emit_context jit_ctx = {}, jit = &jit_ctx; void orig_call = func_addr; int cookie_off, cookie_cnt; u8 branches = NULL; u64 arena_base; u64 func_meta; - u8 prog; bool save_ret; /* @@ -3584,13 +3481,14 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image im, void rw_im orig_call += X86_PATCH_SIZE; } - prog = rw_image; + jit->prog = rw_image; + jit->dry_run = !rw_image; if (flags & BPF_TRAMP_F_INDIRECT) { /* * Indirect call for bpf_struct_ops / - emit_cfi(&prog, image, + emit_cfi(jit, image, cfi_get_func_hash(func_addr), cfi_get_func_arity(func_addr)); } else { @@ -3598,12 +3496,12 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image im, void rw_im Direct-call fentry stub, as such it needs accounting for the * __fentry__ call. / - x86_call_depth_emit_accounting(&prog, NULL, image); + bpf_call_depth_emit_accounting(jit, NULL, image); } EMIT1(0x55); / push rbp / EMIT3(0x48, 0x89, 0xE5); / mov rbp, rsp / if (im) - im->ksym.fp_start = prog - (u8 )rw_image; + im->ksym.fp_start = jit->prog - (u8 )rw_image; if (!is_imm8(stack_size)) { / sub rsp, stack_size / @@ -3615,24 +3513,24 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image im, void rw_im if (flags & BPF_TRAMP_F_TAIL_CALL_CTX) EMIT1(0x50); / push rax / / mov QWORD PTR [rbp - rbx_off], rbx / - emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_6, -rbx_off); + emit_stx(jit, BPF_DW, BPF_REG_FP, BPF_REG_6, -rbx_off); func_meta = nr_regs; / Store number of argument registers of the traced function / - emit_store_stack_imm64(&prog, BPF_REG_0, -func_meta_off, func_meta); + emit_store_stack_imm64(jit, BPF_REG_0, -func_meta_off, func_meta); if (flags & BPF_TRAMP_F_IP_ARG) { / Store IP address of the traced function / - emit_store_stack_imm64(&prog, BPF_REG_0, -ip_off, (long)func_addr); + emit_store_stack_imm64(jit, BPF_REG_0, -ip_off, (long)func_addr); } - save_args(m, &prog, regs_off, false, flags, arena_base); + save_args(m, jit, regs_off, false, flags, arena_base); if (flags & BPF_TRAMP_F_CALL_ORIG) { / arg1: mov rdi, im / - emit_mov_imm64(&prog, BPF_REG_1, (long) im >> 32, (u32) (long) im); - if (emit_rsb_call(&prog, __bpf_tramp_enter, - image + (prog - (u8 )rw_image))) { + emit_mov_imm64(jit, BPF_REG_1, (long) im >> 32, (u32) (long) im); + if (emit_rsb_call(jit, __bpf_tramp_enter, + image + (jit->prog - (u8 )rw_image))) { ret = -EINVAL; goto cleanup; } @@ -3641,13 +3539,13 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image im, void rw_im if (bpf_fsession_cnt(tnodes)) { / clear all the session cookies' value / for (int i = 0; i < cookie_cnt; i++) - emit_store_stack_imm64(&prog, BPF_REG_0, -cookie_off + 8 i, 0); + emit_store_stack_imm64(jit, BPF_REG_0, -cookie_off + 8 * i, 0); /* clear the return value to make sure fentry always get 0 / - emit_store_stack_imm64(&prog, BPF_REG_0, -8, 0); + emit_store_stack_imm64(jit, BPF_REG_0, -8, 0); } if (fentry->nr_nodes) { - if (invoke_bpf(m, &prog, fentry, regs_off, run_ctx_off, func_meta_off, + if (invoke_bpf(m, jit, fentry, regs_off, run_ctx_off, func_meta_off, flags & BPF_TRAMP_F_RET_FENTRY_RET, image, rw_image, func_meta, cookie_off)) return -EINVAL; @@ -3659,7 +3557,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image im, void rw_im if (!branches) return -ENOMEM; - if (invoke_bpf_mod_ret(m, &prog, fmod_ret, regs_off, + if (invoke_bpf_mod_ret(m, jit, fmod_ret, regs_off, run_ctx_off, branches, image, rw_image)) { ret = -EINVAL; goto cleanup; @@ -3667,8 +3565,8 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image im, void rw_im } if (flags & BPF_TRAMP_F_CALL_ORIG) { - restore_regs(m, &prog, regs_off); - save_args(m, &prog, arg_stack_off, true, flags, 0); + restore_regs(m, jit, regs_off); + save_args(m, jit, arg_stack_off, true, flags, 0); if (flags & BPF_TRAMP_F_TAIL_CALL_CTX) { / Before calling the original function, load the @@ -3678,19 +3576,19 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image im, void rw_im } if (flags & BPF_TRAMP_F_ORIG_STACK) { - emit_ldx(&prog, BPF_DW, BPF_REG_6, BPF_REG_FP, 8); + emit_ldx(jit, BPF_DW, BPF_REG_6, BPF_REG_FP, 8); EMIT2(0xff, 0xd3); /* call rbx / } else { /* call original function / - if (emit_rsb_call(&prog, orig_call, image + (prog - (u8 )rw_image))) { + if (emit_rsb_call(jit, orig_call, image + (jit->prog - (u8 )rw_image))) { ret = -EINVAL; goto cleanup; } } / remember return value in a stack for bpf prog to access / - emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8); - im->ip_after_call = image + (prog - (u8 )rw_image); - emit_nops(&prog, X86_PATCH_SIZE); + emit_stx(jit, BPF_DW, BPF_REG_FP, BPF_REG_0, -8); + im->ip_after_call = image + (jit->prog - (u8 )rw_image); + emit_nops(jit, X86_PATCH_SIZE); } if (fmod_ret->nr_nodes) { @@ -3699,12 +3597,17 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image im, void rw_im Coding Rule 11: All branch targets should be 16-byte * aligned. / - emit_align(&prog, 16); + emit_align(jit, 16); / Update the branches saved in invoke_bpf_mod_ret with the * aligned address of do_fexit. / for (i = 0; i < fmod_ret->nr_nodes; i++) { - emit_cond_near_jump(&branches[i], image + (prog - (u8 )rw_image), + struct jit_emit_context branch_jit = { + .prog = branches[i], + .dry_run = jit->dry_run, + }; + + emit_cond_near_jump(&branch_jit, image + (jit->prog - (u8 )rw_image), image + (branches[i] - (u8 )rw_image), X86_JNE); } } @@ -3712,10 +3615,10 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image im, void rw_im /* set the "is_return" flag for fsession / func_meta \|= (1ULL << BPF_TRAMP_IS_RETURN_SHIFT); if (bpf_fsession_cnt(tnodes)) - emit_store_stack_imm64(&prog, BPF_REG_0, -func_meta_off, func_meta); + emit_store_stack_imm64(jit, BPF_REG_0, -func_meta_off, func_meta); if (fexit->nr_nodes) { - if (invoke_bpf(m, &prog, fexit, regs_off, run_ctx_off, func_meta_off, + if (invoke_bpf(m, jit, fexit, regs_off, run_ctx_off, func_meta_off, false, image, rw_image, func_meta, cookie_off)) { ret = -EINVAL; goto cleanup; @@ -3723,17 +3626,17 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image im, void rw_im } if (flags & BPF_TRAMP_F_RESTORE_REGS) - restore_regs(m, &prog, regs_off); + restore_regs(m, jit, regs_off); / This needs to be done regardless. If there were fmod_ret programs, * the return value is only updated on the stack and still needs to be * restored to R0. / if (flags & BPF_TRAMP_F_CALL_ORIG) { - im->ip_epilogue = image + (prog - (u8 )rw_image); + im->ip_epilogue = image + (jit->prog - (u8 )rw_image); / arg1: mov rdi, im / - emit_mov_imm64(&prog, BPF_REG_1, (long) im >> 32, (u32) (long) im); - if (emit_rsb_call(&prog, __bpf_tramp_exit, image + (prog - (u8 )rw_image))) { + emit_mov_imm64(jit, BPF_REG_1, (long) im >> 32, (u32) (long) im); + if (emit_rsb_call(jit, __bpf_tramp_exit, image + (jit->prog - (u8 )rw_image))) { ret = -EINVAL; goto cleanup; } @@ -3746,25 +3649,26 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image im, void rw_im / restore return value of orig_call or fentry prog back into RAX / if (save_ret) - emit_ldx(&prog, BPF_DW, BPF_REG_0, BPF_REG_FP, -8); + emit_ldx(jit, BPF_DW, BPF_REG_0, BPF_REG_FP, -8); - emit_ldx(&prog, BPF_DW, BPF_REG_6, BPF_REG_FP, -rbx_off); + emit_ldx(jit, BPF_DW, BPF_REG_6, BPF_REG_FP, -rbx_off); EMIT1(0xC9); / leave / if (im) - im->ksym.fp_end = prog - (u8 )rw_image; + im->ksym.fp_end = jit->prog - (u8 )rw_image; if (flags & BPF_TRAMP_F_SKIP_FRAME) { / skip our return address and return to parent / EMIT4(0x48, 0x83, 0xC4, 8); / add rsp, 8 / } - emit_return(&prog, image + (prog - (u8 )rw_image)); + emit_return(jit, image + (jit->prog - (u8 )rw_image)); / Make sure the trampoline generation logic doesn't overflow / - if (WARN_ON_ONCE(prog > (u8 )rw_image_end - BPF_INSN_SAFETY)) { + if (!jit->dry_run && + WARN_ON_ONCE(jit->prog > (u8 )rw_image_end - BPF_INSN_SAFETY)) { ret = -EFAULT; goto cleanup; } - ret = prog - (u8 )rw_image + BPF_INSN_SAFETY; + ret = jit->prog - (u8 )rw_image + BPF_INSN_SAFETY; cleanup: kfree(branches); @@ -3819,28 +3723,15 @@ int arch_bpf_trampoline_size(const struct btf_func_model m, u32 flags, struct bpf_tramp_nodes tnodes, void func_addr) { struct bpf_tramp_image im; - void image; - int ret; - / Allocate a temporary buffer for __arch_prepare_bpf_trampoline(). - * - * We cannot use kvmalloc here, because we need image to be in - * module memory range. - * Since it must be writable use bpf_jit_alloc_exec_rw(). - / - image = bpf_jit_alloc_exec_rw(PAGE_SIZE); - if (!image) - return -ENOMEM; - - ret = __arch_prepare_bpf_trampoline(&im, image, image + PAGE_SIZE, image, - m, flags, tnodes, func_addr); - bpf_jit_free_exec(image); - return ret; + return __arch_prepare_bpf_trampoline(&im, NULL, NULL, NULL, m, flags, + tnodes, func_addr); } -static int emit_bpf_dispatcher(u8 pprog, int a, int b, s64 progs, u8 image, u8 buf) +static int emit_bpf_dispatcher(struct jit_emit_context jit, int a, int b, s64 progs, u8 image, + u8 buf) { - u8 jg_reloc, prog = pprog; + u8 jg_reloc; int pivot, err, jg_bytes = 1; s64 jg_offset; @@ -3853,15 +3744,13 @@ static int emit_bpf_dispatcher(u8 *pprog, int a, int b, s64 progs, u8 image, return -1; EMIT2_off32(0x81, add_1reg(0xF8, BPF_REG_3), progs[a]); - err = emit_cond_near_jump(&prog, / je func / - (void )progs[a], image + (prog - buf), + err = emit_cond_near_jump(jit, /* je func / + (void )progs[a], image + (jit->prog - buf), X86_JE); if (err) return err; - emit_indirect_jump(&prog, BPF_REG_3 /* R3 -> rdx /, image + (prog - buf)); - - pprog = prog; + emit_indirect_jump(jit, BPF_REG_3 /* R3 -> rdx /, image + (jit->prog - buf)); return 0; } @@ -3881,9 +3770,9 @@ static int emit_bpf_dispatcher(u8 pprog, int a, int b, s64 progs, u8 image, } else { EMIT2(X86_JG, 0); } - jg_reloc = prog; + jg_reloc = jit->prog; - err = emit_bpf_dispatcher(&prog, a, a + pivot, / emit lower_part / + err = emit_bpf_dispatcher(jit, a, a + pivot, / emit lower_part / progs, image, buf); if (err) return err; @@ -3893,16 +3782,15 @@ static int emit_bpf_dispatcher(u8 pprog, int a, int b, s64 progs, u8 image, Coding Rule 11: All branch targets should be 16-byte * aligned. / - emit_align(&prog, 16); - jg_offset = prog - jg_reloc; + emit_align(jit, 16); + jg_offset = jit->prog - jg_reloc; emit_code(jg_reloc - jg_bytes, jg_offset, jg_bytes); - err = emit_bpf_dispatcher(&prog, a + pivot + 1, / emit upper_part / + err = emit_bpf_dispatcher(jit, a + pivot + 1, / emit upper_part / b, progs, image, buf); if (err) return err; - pprog = prog; return 0; } @@ -3920,10 +3808,10 @@ static int cmp_ips(const void a, const void b) int arch_prepare_bpf_dispatcher(void image, void buf, s64 funcs, int num_funcs) { - u8 prog = buf; + struct jit_emit_context jit = { .prog = buf }; sort(funcs, num_funcs, sizeof(funcs[0]), cmp_ips, NULL); - return emit_bpf_dispatcher(&prog, 0, num_funcs - 1, funcs, image, buf); + return emit_bpf_dispatcher(&jit, 0, num_funcs - 1, funcs, image, buf); } static void priv_stack_init_guard(void __percpu *priv_stack_ptr, int alloc_size)
25/3	2026/08/31 14:19	llm	kmsan-evaluator	2m Model: gemini-3.7-flash Args: null Results: null Tokens: input: 41931 output: 182 thoughts: 2024 Thoughts: Analysis of the x86 BPF JIT Patch for Uninitialized Memory Risks Alright, let's break down this patch and assess potential KMSAN implications. The core change revolves around a refactoring of the BPF JIT code emitter on x86, introducing a `struct jit_emit_context` to handle both the normal execution path and a "dry-run" mode for size calculations. The `jit_emit_context` struct contains a pointer to the current program buffer (`prog`) and a boolean flag `dry_run`. The primary concern is always uninitialized memory reads, leaks to userspace, or usage in branching conditions or copies. I meticulously examined each code snippet affected by this patch, starting with `bpf_call_depth_emit_accounting`, which now utilizes a caller-provided buffer. The `insn_buff` is initialized within `x86_call_depth_emit_accounting` before being copied. I then traced the `jit_emit_context` usage throughout the BPF JIT code, specifically within the `emit_code_jit` and related helper functions. The dry-run logic increments the `jit->prog` pointer to calculate the trampoline size without writing actual data, avoiding memory allocation. In each case that `jit` is initialized, the `dry_run` is set to false except when calculating size. The dry-run logic relies on pointer arithmetic (incrementing the `jit->prog` pointer) and conditional execution based on the `dry_run` flag. I verified the implications of the dry-run, focusing on scenarios where the `jit->prog` pointer might be NULL during size calculation, which is the only time `dry_run == true`. However, the pointer arithmetic is handled correctly, with the result used solely for size calculation, without dereferencing or other memory access. The same is true for `invoke_bpf_prog` and other such situations. There were no instances of uninitialized data being used in branches or copied to user space. All structures (`jit_emit_context`, `old_insn`, etc.) are either zero-initialized or initialized with valid values. Memory is always written to before it is read. Therefore, after this detailed analysis, there is no evidence that this patch introduces any new uninitialized memory risks. Consequently, it doesn't warrant a KMSAN session. The changes are primarily code refactoring, improving the BPF JIT emitter, and they do not introduce any new kernel-to-user copy or conditional branching on uninitialized data. There are no exposed data structures or new data flows, or issues that would require KMSAN.
26/3	2026/08/31 14:22	tool	set-results	0m Args: { "KMSANReasoning": "The patch refactors the x86 BPF JIT compiler and callthunk accounting code to introduce a dry-run mode for trampoline size calculation via `struct jit_emit_context`, replacing dummy allocation in `arch_bpf_trampoline_size()`. All new context and buffer variables are properly initialized (`= {}`), and code generation buffers are accurately tracked and populated. The patch does not introduce any uninitialized memory reads, uninitialized structure padding leaks to user space, or uninitialized branch conditions. Any potential bugs in JIT generation or pointer calculations would be out-of-bounds or logic errors caught by KASAN or standard kernel assertions, not uninitialized memory bugs detectable only by KMSAN. Therefore, dedicated KMSAN fuzzing is not required.", "NeedsKMSAN": false } Results: { "KMSANReasoning": "The patch refactors the x86 BPF JIT compiler and callthunk accounting code to introduce a dry-run mode for trampoline size calculation via `struct jit_emit_context`, replacing dummy allocation in `arch_bpf_trampoline_size()`. All new context and buffer variables are properly initialized (`= {}`), and code generation buffers are accurately tracked and populated. The patch does not introduce any uninitialized memory reads, uninitialized structure padding leaks to user space, or uninitialized branch conditions. Any potential bugs in JIT generation or pointer calculations would be out-of-bounds or logic errors caught by KASAN or standard kernel assertions, not uninitialized memory bugs detectable only by KMSAN. Therefore, dedicated KMSAN fuzzing is not required.", "NeedsKMSAN": false }

{
  "EnableConfigs": [
    "BPF_SYSCALL",
    "BPF_JIT",
    "CALL_THUNKS",
    "MITIGATION_CALL_DEPTH_TRACKING",
    "X86_64"
  ],
  "FocusSymbols": [
    "arch_prepare_bpf_trampoline",
    "arch_bpf_trampoline_size",
    "bpf_int_jit_compile",
    "arch_prepare_bpf_dispatcher",
    "x86_call_depth_emit_accounting"
  ],
  "KMSANReasoning": "The patch refactors the x86 BPF JIT compiler and callthunk accounting code to introduce a dry-run mode for trampoline size calculation via `struct jit_emit_context`, replacing dummy allocation in `arch_bpf_trampoline_size()`. All new context and buffer variables are properly initialized (`= {}`), and code generation buffers are accurately tracked and populated. The patch does not introduce any uninitialized memory reads, uninitialized structure padding leaks to user space, or uninitialized branch conditions. Any potential bugs in JIT generation or pointer calculations would be out-of-bounds or logic errors caught by KASAN or standard kernel assertions, not uninitialized memory bugs detectable only by KMSAN. Therefore, dedicated KMSAN fuzzing is not required.",
  "NeedsKMSAN": false,
  "Reasoning": "The patch refactors the x86 BPF JIT compiler and trampoline generator by introducing a struct jit_emit_context with dry-run support. This changes how trampoline sizing is computed (dry-run without allocation), updates instruction emission logic, and alters x86 call depth accounting integration. These changes affect BPF program compilation, trampoline generation, and execution reachable via the bpf() syscall.",
  "WorthFuzzing": true
}

{
  "PatchDiff": "commit e09e4ad11f933fa9ab21debd310aeb220f3f52c0\nAuthor: syz-cluster \u003ctriage@syzkaller.com\u003e\nDate:   Mon Aug 31 14:02:28 2026 +0000\n\n    syz-cluster: applied patch under review\n\ndiff --git a/arch/x86/include/asm/alternative.h b/arch/x86/include/asm/alternative.h\nindex 08af86ef090a8..2ef4937411860 100644\n--- a/arch/x86/include/asm/alternative.h\n+++ b/arch/x86/include/asm/alternative.h\n@@ -83,7 +83,7 @@ extern void callthunks_patch_builtin_calls(void);\n extern void callthunks_patch_module_calls(struct callthunk_sites *sites,\n \t\t\t\t\t  struct module *mod);\n extern void *callthunks_translate_call_dest(void *dest);\n-extern int x86_call_depth_emit_accounting(u8 **pprog, void *func, void *ip);\n+extern int x86_call_depth_emit_accounting(u8 *insn_buff, void *func, void *ip);\n #else\n static __always_inline void callthunks_patch_builtin_calls(void) {}\n static __always_inline void\n@@ -93,7 +93,7 @@ static __always_inline void *callthunks_translate_call_dest(void *dest)\n {\n \treturn dest;\n }\n-static __always_inline int x86_call_depth_emit_accounting(u8 **pprog,\n+static __always_inline int x86_call_depth_emit_accounting(u8 *insn_buff,\n \t\t\t\t\t\t\t  void *func, void *ip)\n {\n \treturn 0;\ndiff --git a/arch/x86/include/asm/text-patching.h b/arch/x86/include/asm/text-patching.h\nindex f2d142a0a862e..a0a7c778b0a2a 100644\n--- a/arch/x86/include/asm/text-patching.h\n+++ b/arch/x86/include/asm/text-patching.h\n@@ -13,6 +13,8 @@\n  */\n #define TEXT_POKE_MAX_OPCODE_SIZE\t5\n \n+#define MAX_PATCH_LEN\t(255-1)\n+\n extern void text_poke_early(void *addr, const void *opcode, size_t len);\n \n extern void text_poke_apply_relocation(u8 *buf, const u8 * const instr, size_t instrlen, u8 *repl, size_t repl_len);\ndiff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c\nindex 91b1cdd165699..5f9989c8559ea 100644\n--- a/arch/x86/kernel/alternative.c\n+++ b/arch/x86/kernel/alternative.c\n@@ -18,8 +18,6 @@ int __read_mostly alternatives_patched;\n \n EXPORT_SYMBOL_GPL(alternatives_patched);\n \n-#define MAX_PATCH_LEN (255-1)\n-\n #define DA_ALL\t\t(~0)\n #define DA_ALT\t\t0x01\n #define DA_RET\t\t0x02\ndiff --git a/arch/x86/kernel/callthunks.c b/arch/x86/kernel/callthunks.c\nindex e37728f703227..15b221d06901c 100644\n--- a/arch/x86/kernel/callthunks.c\n+++ b/arch/x86/kernel/callthunks.c\n@@ -23,8 +23,6 @@\n \n static int __initdata_or_module debug_callthunks;\n \n-#define MAX_PATCH_LEN (255-1)\n-\n #define prdbg(fmt, args...)\t\t\t\t\t\\\n do {\t\t\t\t\t\t\t\t\\\n \tif (debug_callthunks)\t\t\t\t\t\\\n@@ -298,10 +296,9 @@ static bool is_callthunk(void *addr)\n \treturn !bcmp(pad, insn_buff, tmpl_size);\n }\n \n-int x86_call_depth_emit_accounting(u8 **pprog, void *func, void *ip)\n+int x86_call_depth_emit_accounting(u8 *insn_buff, void *func, void *ip)\n {\n \tunsigned int tmpl_size = SKL_TMPL_SIZE;\n-\tu8 insn_buff[MAX_PATCH_LEN];\n \n \tif (!thunks_initialized)\n \t\treturn 0;\n@@ -313,8 +310,6 @@ int x86_call_depth_emit_accounting(u8 **pprog, void *func, void *ip)\n \tmemcpy(insn_buff, skl_call_thunk_template, tmpl_size);\n \ttext_poke_apply_relocation(insn_buff, ip, tmpl_size, skl_call_thunk_template, tmpl_size);\n \n-\tmemcpy(*pprog, insn_buff, tmpl_size);\n-\t*pprog += tmpl_size;\n \treturn tmpl_size;\n }\n #endif\ndiff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c\nindex 48429fae06410..2ec734c69b693 100644\n--- a/arch/x86/net/bpf_jit_comp.c\n+++ b/arch/x86/net/bpf_jit_comp.c\n@@ -23,6 +23,11 @@\n \n static bool all_callee_regs_used[4] = {true, true, true, true};\n \n+struct jit_emit_context {\n+\tu8 *prog;\n+\tbool dry_run;\n+};\n+\n static u8 *emit_code(u8 *ptr, u32 bytes, unsigned int len)\n {\n \tif (len == 1)\n@@ -36,8 +41,16 @@ static u8 *emit_code(u8 *ptr, u32 bytes, unsigned int len)\n \treturn ptr + len;\n }\n \n+static void emit_code_jit(struct jit_emit_context *jit, u32 bytes, unsigned int len)\n+{\n+\tif (jit-\u003edry_run)\n+\t\tjit-\u003eprog += len;\n+\telse\n+\t\tjit-\u003eprog = emit_code(jit-\u003eprog, bytes, len);\n+}\n+\n #define EMIT(bytes, len) \\\n-\tdo { prog = emit_code(prog, bytes, len); } while (0)\n+\temit_code_jit(jit, bytes, len)\n \n #define EMIT1(b1)\t\tEMIT(b1, 1)\n #define EMIT2(b1, b2)\t\tEMIT((b1) + ((b2) \u003c\u003c 8), 2)\n@@ -149,6 +162,19 @@ static int bpf_size_to_x86_bytes(int bpf_size)\n \t\treturn 0;\n }\n \n+static int bpf_call_depth_emit_accounting(struct jit_emit_context *jit, void *func, void *ip)\n+{\n+\tu8 insn_buff[MAX_PATCH_LEN];\n+\tint size;\n+\n+\tsize = x86_call_depth_emit_accounting(insn_buff, func, ip);\n+\tif (!jit-\u003edry_run)\n+\t\tmemcpy(jit-\u003eprog, insn_buff, size);\n+\n+\tjit-\u003eprog += size;\n+\treturn size;\n+}\n+\n /*\n  * List of x86 cond jumps opcodes (. + s8)\n  * Add 0x10 (and an extra 0x0f) to generate far jumps (. + s32)\n@@ -309,6 +335,8 @@ int bpf_arch_text_invalidate(void *dst, size_t len)\n }\n \n struct jit_context {\n+\tstruct jit_emit_context jit;\n+\n \tint cleanup_addr; /* Epilogue code offset */\n \n \t/*\n@@ -329,34 +357,23 @@ struct jit_context {\n /* Number of bytes that will be skipped on tailcall */\n #define X86_TAIL_CALL_OFFSET\t(12 + ENDBR_INSN_SIZE)\n \n-static void push_r9(u8 **pprog)\n+static void push_r9(struct jit_emit_context *jit)\n {\n-\tu8 *prog = *pprog;\n-\n \tEMIT2(0x41, 0x51);   /* push r9 */\n-\t*pprog = prog;\n }\n \n-static void pop_r9(u8 **pprog)\n+static void pop_r9(struct jit_emit_context *jit)\n {\n-\tu8 *prog = *pprog;\n-\n \tEMIT2(0x41, 0x59);   /* pop r9 */\n-\t*pprog = prog;\n }\n \n-static void push_r12(u8 **pprog)\n+static void push_r12(struct jit_emit_context *jit)\n {\n-\tu8 *prog = *pprog;\n-\n \tEMIT2(0x41, 0x54);   /* push r12 */\n-\t*pprog = prog;\n }\n \n-static void push_callee_regs(u8 **pprog, bool *callee_regs_used)\n+static void push_callee_regs(struct jit_emit_context *jit, bool *callee_regs_used)\n {\n-\tu8 *prog = *pprog;\n-\n \tif (callee_regs_used[0])\n \t\tEMIT1(0x53);         /* push rbx */\n \tif (callee_regs_used[1])\n@@ -365,21 +382,15 @@ static void push_callee_regs(u8 **pprog, bool *callee_regs_used)\n \t\tEMIT2(0x41, 0x56);   /* push r14 */\n \tif (callee_regs_used[3])\n \t\tEMIT2(0x41, 0x57);   /* push r15 */\n-\t*pprog = prog;\n }\n \n-static void pop_r12(u8 **pprog)\n+static void pop_r12(struct jit_emit_context *jit)\n {\n-\tu8 *prog = *pprog;\n-\n \tEMIT2(0x41, 0x5C);   /* pop r12 */\n-\t*pprog = prog;\n }\n \n-static void pop_callee_regs(u8 **pprog, bool *callee_regs_used)\n+static void pop_callee_regs(struct jit_emit_context *jit, bool *callee_regs_used)\n {\n-\tu8 *prog = *pprog;\n-\n \tif (callee_regs_used[3])\n \t\tEMIT2(0x41, 0x5F);   /* pop r15 */\n \tif (callee_regs_used[2])\n@@ -388,40 +399,32 @@ static void pop_callee_regs(u8 **pprog, bool *callee_regs_used)\n \t\tEMIT2(0x41, 0x5D);   /* pop r13 */\n \tif (callee_regs_used[0])\n \t\tEMIT1(0x5B);         /* pop rbx */\n-\t*pprog = prog;\n }\n \n /* add rsp, depth */\n-static void emit_add_rsp(u8 **pprog, u16 depth)\n+static void emit_add_rsp(struct jit_emit_context *jit, u16 depth)\n {\n-\tu8 *prog = *pprog;\n-\n \tif (!depth)\n \t\treturn;\n \tif (is_imm8(depth))\n \t\tEMIT4(0x48, 0x83, 0xC4, depth); /* add rsp, imm8 */\n \telse\n \t\tEMIT3_off32(0x48, 0x81, 0xC4, depth); /* add rsp, imm32 */\n-\t*pprog = prog;\n }\n \n /* sub rsp, depth */\n-static void emit_sub_rsp(u8 **pprog, u16 depth)\n+static void emit_sub_rsp(struct jit_emit_context *jit, u16 depth)\n {\n-\tu8 *prog = *pprog;\n-\n \tif (!depth)\n \t\treturn;\n \tif (is_imm8(depth))\n \t\tEMIT4(0x48, 0x83, 0xEC, depth); /* sub rsp, imm8 */\n \telse\n \t\tEMIT3_off32(0x48, 0x81, 0xEC, depth); /* sub rsp, imm32 */\n-\t*pprog = prog;\n }\n \n-static void emit_nops(u8 **pprog, int len)\n+static void emit_nops(struct jit_emit_context *jit, int len)\n {\n-\tu8 *prog = *pprog;\n \tint i, noplen;\n \n \twhile (len \u003e 0) {\n@@ -434,72 +437,56 @@ static void emit_nops(u8 **pprog, int len)\n \t\t\tEMIT1(x86_nops[noplen][i]);\n \t\tlen -= noplen;\n \t}\n-\n-\t*pprog = prog;\n }\n \n /*\n  * Emit the various CFI preambles, see asm/cfi.h and the comments about FineIBT\n  * in arch/x86/kernel/alternative.c\n  */\n-static int emit_call(u8 **prog, void *func, void *ip);\n+static int emit_call(struct jit_emit_context *jit, void *func, void *ip);\n \n-static void emit_fineibt(u8 **pprog, u8 *ip, u32 hash, int arity)\n+static void emit_fineibt(struct jit_emit_context *jit, u8 *ip, u32 hash, int arity)\n {\n-\tu8 *prog = *pprog;\n-\n \tEMIT_ENDBR();\n \tEMIT1_off32(0x2d, hash);\t\t\t/* subl $hash, %eax\t*/\n \tif (cfi_bhi) {\n \t\tEMIT2(0x2e, 0x2e);\t\t\t/* cs cs */\n-\t\temit_call(\u0026prog, __bhi_args[arity], ip + 11);\n+\t\temit_call(jit, __bhi_args[arity], ip + 11);\n \t} else {\n \t\tEMIT3_off32(0x2e, 0x0f, 0x85, 3);\t/* jne.d32,pn 3\t\t*/\n \t}\n \tEMIT_ENDBR_POISON();\n-\n-\t*pprog = prog;\n }\n \n-static void emit_kcfi(u8 **pprog, u32 hash)\n+static void emit_kcfi(struct jit_emit_context *jit, u32 hash)\n {\n-\tu8 *prog = *pprog;\n-\n \tEMIT1_off32(0xb8, hash);\t\t\t/* movl $hash, %eax\t*/\n #ifdef CONFIG_CALL_PADDING\n \tfor (int i = 0; i \u003c CONFIG_FUNCTION_PADDING_CFI; i++)\n \t\tEMIT1(0x90);\n #endif\n \tEMIT_ENDBR();\n-\n-\t*pprog = prog;\n }\n \n-static void emit_cfi(u8 **pprog, u8 *ip, u32 hash, int arity)\n+static void emit_cfi(struct jit_emit_context *jit, u8 *ip, u32 hash, int arity)\n {\n-\tu8 *prog = *pprog;\n-\n \tswitch (cfi_mode) {\n \tcase CFI_FINEIBT:\n-\t\temit_fineibt(\u0026prog, ip, hash, arity);\n+\t\temit_fineibt(jit, ip, hash, arity);\n \t\tbreak;\n \n \tcase CFI_KCFI:\n-\t\temit_kcfi(\u0026prog, hash);\n+\t\temit_kcfi(jit, hash);\n \t\tbreak;\n \n \tdefault:\n \t\tEMIT_ENDBR();\n \t\tbreak;\n \t}\n-\n-\t*pprog = prog;\n }\n \n-static void emit_prologue_tail_call(u8 **pprog, bool is_subprog)\n+static void emit_prologue_tail_call(struct jit_emit_context *jit, bool is_subprog)\n {\n-\tu8 *prog = *pprog;\n-\n \tif (!is_subprog) {\n \t\t/* cmp rax, MAX_TAIL_CALL_CNT */\n \t\tEMIT4(0x48, 0x83, 0xF8, MAX_TAIL_CALL_CNT);\n@@ -523,8 +510,6 @@ static void emit_prologue_tail_call(u8 **pprog, bool is_subprog)\n \t\tEMIT1(0x50);             /* push rax */\n \t\tEMIT1(0x50);             /* push rax */\n \t}\n-\n-\t*pprog = prog;\n }\n \n /*\n@@ -532,21 +517,19 @@ static void emit_prologue_tail_call(u8 **pprog, bool is_subprog)\n  * bpf_tail_call helper will skip the first X86_TAIL_CALL_OFFSET bytes\n  * while jumping to another program\n  */\n-static void emit_prologue(u8 **pprog, u8 *ip, u32 stack_depth, bool ebpf_from_cbpf,\n-\t\t\t  bool tail_call_reachable, bool is_subprog,\n+static void emit_prologue(struct jit_emit_context *jit, u8 *ip, u32 stack_depth,\n+\t\t\t  bool ebpf_from_cbpf, bool tail_call_reachable, bool is_subprog,\n \t\t\t  bool is_exception_cb)\n {\n-\tu8 *prog = *pprog;\n-\n \tif (is_subprog) {\n-\t\temit_cfi(\u0026prog, ip, cfi_bpf_subprog_hash, 5);\n+\t\temit_cfi(jit, ip, cfi_bpf_subprog_hash, 5);\n \t} else {\n-\t\temit_cfi(\u0026prog, ip, cfi_bpf_hash, 1);\n+\t\temit_cfi(jit, ip, cfi_bpf_hash, 1);\n \t}\n \t/* BPF trampoline can be made to work without these nops,\n \t * but let's waste 5 bytes for now and optimize later\n \t */\n-\temit_nops(\u0026prog, X86_PATCH_SIZE);\n+\temit_nops(jit, X86_PATCH_SIZE);\n \tif (!ebpf_from_cbpf) {\n \t\tif (tail_call_reachable \u0026\u0026 !is_subprog)\n \t\t\t/* When it's the entry of the whole tailcall context,\n@@ -555,7 +538,7 @@ static void emit_prologue(u8 **pprog, u8 *ip, u32 stack_depth, bool ebpf_from_cb\n \t\t\tEMIT3(0x48, 0x31, 0xC0); /* xor rax, rax */\n \t\telse\n \t\t\t/* Keep the same instruction layout. */\n-\t\t\temit_nops(\u0026prog, 3);     /* nop3 */\n+\t\t\temit_nops(jit, 3);     /* nop3 */\n \t}\n \t/* Exception callback receives FP as third parameter */\n \tif (is_exception_cb) {\n@@ -565,8 +548,8 @@ static void emit_prologue(u8 **pprog, u8 *ip, u32 stack_depth, bool ebpf_from_cb\n \t\t * first restore those callee-saved regs from stack, before\n \t\t * reusing the stack frame.\n \t\t */\n-\t\tpop_callee_regs(\u0026prog, all_callee_regs_used);\n-\t\tpop_r12(\u0026prog);\n+\t\tpop_callee_regs(jit, all_callee_regs_used);\n+\t\tpop_r12(jit);\n \t\t/* Reset the stack frame. */\n \t\tEMIT3(0x48, 0x89, 0xEC); /* mov rsp, rbp */\n \t} else {\n@@ -581,40 +564,38 @@ static void emit_prologue(u8 **pprog, u8 *ip, u32 stack_depth, bool ebpf_from_cb\n \tif (stack_depth)\n \t\tEMIT3_off32(0x48, 0x81, 0xEC, round_up(stack_depth, 8));\n \tif (tail_call_reachable)\n-\t\temit_prologue_tail_call(\u0026prog, is_subprog);\n-\t*pprog = prog;\n+\t\temit_prologue_tail_call(jit, is_subprog);\n }\n \n-static int emit_patch(u8 **pprog, void *func, void *ip, u8 opcode)\n+static int emit_patch(struct jit_emit_context *jit, void *func, void *ip, u8 opcode)\n {\n-\tu8 *prog = *pprog;\n \ts64 offset;\n \n \toffset = func - (ip + X86_PATCH_SIZE);\n-\tif (!is_simm32(offset)) {\n+\t/* We do not have meaningful ip value in the dry run, skip the check. */\n+\tif (!jit-\u003edry_run \u0026\u0026 !is_simm32(offset)) {\n \t\tpr_err(\"Target call %p is out of range\\n\", func);\n \t\treturn -ERANGE;\n \t}\n \tEMIT1_off32(opcode, offset);\n-\t*pprog = prog;\n \treturn 0;\n }\n \n-static int emit_call(u8 **pprog, void *func, void *ip)\n+static int emit_call(struct jit_emit_context *jit, void *func, void *ip)\n {\n-\treturn emit_patch(pprog, func, ip, 0xE8);\n+\treturn emit_patch(jit, func, ip, 0xE8);\n }\n \n-static int emit_rsb_call(u8 **pprog, void *func, void *ip)\n+static int emit_rsb_call(struct jit_emit_context *jit, void *func, void *ip)\n {\n \tOPTIMIZER_HIDE_VAR(func);\n-\tip += x86_call_depth_emit_accounting(pprog, func, ip);\n-\treturn emit_patch(pprog, func, ip, 0xE8);\n+\tip += bpf_call_depth_emit_accounting(jit, func, ip);\n+\treturn emit_patch(jit, func, ip, 0xE8);\n }\n \n-static int emit_jump(u8 **pprog, void *func, void *ip)\n+static int emit_jump(struct jit_emit_context *jit, void *func, void *ip)\n {\n-\treturn emit_patch(pprog, func, ip, 0xE9);\n+\treturn emit_patch(jit, func, ip, 0xE9);\n }\n \n static int __bpf_arch_text_poke(void *ip, enum bpf_text_poke_type old_t,\n@@ -622,27 +603,27 @@ static int __bpf_arch_text_poke(void *ip, enum bpf_text_poke_type old_t,\n \t\t\t\tvoid *old_addr, void *new_addr)\n {\n \tconst u8 *nop_insn = x86_nops[5];\n+\tstruct jit_emit_context jit = {};\n \tu8 old_insn[X86_PATCH_SIZE];\n \tu8 new_insn[X86_PATCH_SIZE];\n-\tu8 *prog;\n \tint ret;\n \n \tmemcpy(old_insn, nop_insn, X86_PATCH_SIZE);\n \tif (old_t != BPF_MOD_NOP \u0026\u0026 old_addr) {\n-\t\tprog = old_insn;\n+\t\tjit.prog = old_insn;\n \t\tret = old_t == BPF_MOD_CALL ?\n-\t\t      emit_call(\u0026prog, old_addr, ip) :\n-\t\t      emit_jump(\u0026prog, old_addr, ip);\n+\t\t      emit_call(\u0026jit, old_addr, ip) :\n+\t\t      emit_jump(\u0026jit, old_addr, ip);\n \t\tif (ret)\n \t\t\treturn ret;\n \t}\n \n \tmemcpy(new_insn, nop_insn, X86_PATCH_SIZE);\n \tif (new_t != BPF_MOD_NOP \u0026\u0026 new_addr) {\n-\t\tprog = new_insn;\n+\t\tjit.prog = new_insn;\n \t\tret = new_t == BPF_MOD_CALL ?\n-\t\t      emit_call(\u0026prog, new_addr, ip) :\n-\t\t      emit_jump(\u0026prog, new_addr, ip);\n+\t\t      emit_call(\u0026jit, new_addr, ip) :\n+\t\t      emit_jump(\u0026jit, new_addr, ip);\n \t\tif (ret)\n \t\t\treturn ret;\n \t}\n@@ -682,58 +663,47 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type old_t,\n \n #define EMIT_LFENCE()\tEMIT3(0x0F, 0xAE, 0xE8)\n \n-static void __emit_indirect_jump(u8 **pprog, int reg, bool ereg)\n+static void __emit_indirect_jump(struct jit_emit_context *jit, int reg, bool ereg)\n {\n-\tu8 *prog = *pprog;\n-\n \tif (ereg)\n \t\tEMIT1(0x41);\n \n \tEMIT2(0xFF, 0xE0 + reg);\n-\n-\t*pprog = prog;\n }\n \n-static void emit_indirect_jump(u8 **pprog, int bpf_reg, u8 *ip)\n+static void emit_indirect_jump(struct jit_emit_context *jit, int bpf_reg, u8 *ip)\n {\n-\tu8 *prog = *pprog;\n \tint reg = reg2hex[bpf_reg];\n \tbool ereg = is_ereg(bpf_reg);\n \n \tif (cpu_feature_enabled(X86_FEATURE_INDIRECT_THUNK_ITS)) {\n \t\tOPTIMIZER_HIDE_VAR(reg);\n-\t\temit_jump(\u0026prog, its_static_thunk(reg + 8*ereg), ip);\n+\t\temit_jump(jit, its_static_thunk(reg + 8*ereg), ip);\n \t} else if (cpu_feature_enabled(X86_FEATURE_RETPOLINE_LFENCE)) {\n \t\tEMIT_LFENCE();\n-\t\t__emit_indirect_jump(\u0026prog, reg, ereg);\n+\t\t__emit_indirect_jump(jit, reg, ereg);\n \t} else if (cpu_feature_enabled(X86_FEATURE_RETPOLINE)) {\n \t\tOPTIMIZER_HIDE_VAR(reg);\n \t\tif (cpu_feature_enabled(X86_FEATURE_CALL_DEPTH))\n-\t\t\temit_jump(\u0026prog, \u0026__x86_indirect_jump_thunk_array[reg + 8*ereg], ip);\n+\t\t\temit_jump(jit, \u0026__x86_indirect_jump_thunk_array[reg + 8*ereg], ip);\n \t\telse\n-\t\t\temit_jump(\u0026prog, \u0026__x86_indirect_thunk_array[reg + 8*ereg], ip);\n+\t\t\temit_jump(jit, \u0026__x86_indirect_thunk_array[reg + 8*ereg], ip);\n \t} else {\n-\t\t__emit_indirect_jump(\u0026prog, reg, ereg);\n+\t\t__emit_indirect_jump(jit, reg, ereg);\n \t\tif (IS_ENABLED(CONFIG_MITIGATION_RETPOLINE) || IS_ENABLED(CONFIG_MITIGATION_SLS))\n \t\t\tEMIT1(0xCC);\t\t/* int3 */\n \t}\n-\n-\t*pprog = prog;\n }\n \n-static void emit_return(u8 **pprog, u8 *ip)\n+static void emit_return(struct jit_emit_context *jit, u8 *ip)\n {\n-\tu8 *prog = *pprog;\n-\n \tif (cpu_wants_rethunk()) {\n-\t\temit_jump(\u0026prog, x86_return_thunk, ip);\n+\t\temit_jump(jit, x86_return_thunk, ip);\n \t} else {\n \t\tEMIT1(0xC3);\t\t/* ret */\n \t\tif (IS_ENABLED(CONFIG_MITIGATION_SLS))\n \t\t\tEMIT1(0xCC);\t/* int3 */\n \t}\n-\n-\t*pprog = prog;\n }\n \n #define BPF_TAIL_CALL_CNT_PTR_STACK_OFF(stack)\t(-16 - round_up(stack, 8))\n@@ -753,12 +723,13 @@ static void emit_return(u8 **pprog, u8 *ip)\n  * out:\n  */\n static void emit_bpf_tail_call_indirect(struct bpf_prog *bpf_prog,\n-\t\t\t\t\tu8 **pprog, bool *callee_regs_used,\n+\t\t\t\t\tbool *callee_regs_used,\n \t\t\t\t\tu32 stack_depth, u8 *ip,\n \t\t\t\t\tstruct jit_context *ctx)\n {\n \tint tcc_ptr_off = BPF_TAIL_CALL_CNT_PTR_STACK_OFF(stack_depth);\n-\tu8 *prog = *pprog, *start = *pprog;\n+\tstruct jit_emit_context *jit = \u0026ctx-\u003ejit;\n+\tu8 *start = jit-\u003eprog;\n \tint offset;\n \n \t/*\n@@ -775,7 +746,7 @@ static void emit_bpf_tail_call_indirect(struct bpf_prog *bpf_prog,\n \tEMIT3(0x39, 0x56,                         /* cmp dword ptr [rsi + 16], edx */\n \t      offsetof(struct bpf_array, map.max_entries));\n \n-\toffset = ctx-\u003etail_call_indirect_label - (prog + 2 - start);\n+\toffset = ctx-\u003etail_call_indirect_label - (jit-\u003eprog + 2 - start);\n \tEMIT2(X86_JBE, offset);                   /* jbe out */\n \n \t/*\n@@ -785,7 +756,7 @@ static void emit_bpf_tail_call_indirect(struct bpf_prog *bpf_prog,\n \tEMIT3_off32(0x48, 0x8B, 0x85, tcc_ptr_off); /* mov rax, qword ptr [rbp - tcc_ptr_off] */\n \tEMIT4(0x48, 0x83, 0x38, MAX_TAIL_CALL_CNT); /* cmp qword ptr [rax], MAX_TAIL_CALL_CNT */\n \n-\toffset = ctx-\u003etail_call_indirect_label - (prog + 2 - start);\n+\toffset = ctx-\u003etail_call_indirect_label - (jit-\u003eprog + 2 - start);\n \tEMIT2(X86_JAE, offset);                   /* jae out */\n \n \t/* prog = array-\u003eptrs[index]; */\n@@ -798,19 +769,19 @@ static void emit_bpf_tail_call_indirect(struct bpf_prog *bpf_prog,\n \t */\n \tEMIT3(0x48, 0x85, 0xC9);                  /* test rcx,rcx */\n \n-\toffset = ctx-\u003etail_call_indirect_label - (prog + 2 - start);\n+\toffset = ctx-\u003etail_call_indirect_label - (jit-\u003eprog + 2 - start);\n \tEMIT2(X86_JE, offset);                    /* je out */\n \n \t/* Inc tail_call_cnt if the slot is populated. */\n \tEMIT4(0x48, 0x83, 0x00, 0x01);            /* add qword ptr [rax], 1 */\n \n \tif (bpf_prog-\u003eaux-\u003eexception_boundary) {\n-\t\tpop_callee_regs(\u0026prog, all_callee_regs_used);\n-\t\tpop_r12(\u0026prog);\n+\t\tpop_callee_regs(jit, all_callee_regs_used);\n+\t\tpop_r12(jit);\n \t} else {\n-\t\tpop_callee_regs(\u0026prog, callee_regs_used);\n+\t\tpop_callee_regs(jit, callee_regs_used);\n \t\tif (bpf_arena_get_kern_vm_start(bpf_prog-\u003eaux-\u003earena))\n-\t\t\tpop_r12(\u0026prog);\n+\t\t\tpop_r12(jit);\n \t}\n \n \t/* Pop tail_call_cnt_ptr. */\n@@ -833,21 +804,21 @@ static void emit_bpf_tail_call_indirect(struct bpf_prog *bpf_prog,\n \t * rdi == ctx (1st arg)\n \t * rcx == prog-\u003ebpf_func + X86_TAIL_CALL_OFFSET\n \t */\n-\temit_indirect_jump(\u0026prog, BPF_REG_4 /* R4 -\u003e rcx */, ip + (prog - start));\n+\temit_indirect_jump(jit, BPF_REG_4 /* R4 -\u003e rcx */, ip + (jit-\u003eprog - start));\n \n \t/* out: */\n-\tctx-\u003etail_call_indirect_label = prog - start;\n-\t*pprog = prog;\n+\tctx-\u003etail_call_indirect_label = jit-\u003eprog - start;\n }\n \n static void emit_bpf_tail_call_direct(struct bpf_prog *bpf_prog,\n \t\t\t\t      struct bpf_jit_poke_descriptor *poke,\n-\t\t\t\t      u8 **pprog, u8 *ip,\n+\t\t\t\t      u8 *ip,\n \t\t\t\t      bool *callee_regs_used, u32 stack_depth,\n \t\t\t\t      struct jit_context *ctx)\n {\n+\tstruct jit_emit_context *jit = \u0026ctx-\u003ejit;\n \tint tcc_ptr_off = BPF_TAIL_CALL_CNT_PTR_STACK_OFF(stack_depth);\n-\tu8 *prog = *pprog, *start = *pprog;\n+\tu8 *start = jit-\u003eprog;\n \tint offset;\n \n \t/*\n@@ -857,27 +828,27 @@ static void emit_bpf_tail_call_direct(struct bpf_prog *bpf_prog,\n \tEMIT3_off32(0x48, 0x8B, 0x85, tcc_ptr_off);   /* mov rax, qword ptr [rbp - tcc_ptr_off] */\n \tEMIT4(0x48, 0x83, 0x38, MAX_TAIL_CALL_CNT);   /* cmp qword ptr [rax], MAX_TAIL_CALL_CNT */\n \n-\toffset = ctx-\u003etail_call_direct_label - (prog + 2 - start);\n+\toffset = ctx-\u003etail_call_direct_label - (jit-\u003eprog + 2 - start);\n \tEMIT2(X86_JAE, offset);                       /* jae out */\n \n-\tpoke-\u003etailcall_bypass = ip + (prog - start);\n+\tpoke-\u003etailcall_bypass = ip + (jit-\u003eprog - start);\n \tpoke-\u003eadj_off = X86_TAIL_CALL_OFFSET;\n \tpoke-\u003etailcall_target = ip + ctx-\u003etail_call_direct_label - X86_PATCH_SIZE;\n \tpoke-\u003ebypass_addr = (u8 *)poke-\u003etailcall_target + X86_PATCH_SIZE;\n \n-\temit_jump(\u0026prog, (u8 *)poke-\u003etailcall_target + X86_PATCH_SIZE,\n+\temit_jump(jit, (u8 *)poke-\u003etailcall_target + X86_PATCH_SIZE,\n \t\t  poke-\u003etailcall_bypass);\n \n \t/* Inc tail_call_cnt if the slot is populated. */\n \tEMIT4(0x48, 0x83, 0x00, 0x01);                /* add qword ptr [rax], 1 */\n \n \tif (bpf_prog-\u003eaux-\u003eexception_boundary) {\n-\t\tpop_callee_regs(\u0026prog, all_callee_regs_used);\n-\t\tpop_r12(\u0026prog);\n+\t\tpop_callee_regs(jit, all_callee_regs_used);\n+\t\tpop_r12(jit);\n \t} else {\n-\t\tpop_callee_regs(\u0026prog, callee_regs_used);\n+\t\tpop_callee_regs(jit, callee_regs_used);\n \t\tif (bpf_arena_get_kern_vm_start(bpf_prog-\u003eaux-\u003earena))\n-\t\t\tpop_r12(\u0026prog);\n+\t\t\tpop_r12(jit);\n \t}\n \n \t/* Pop tail_call_cnt_ptr. */\n@@ -889,12 +860,10 @@ static void emit_bpf_tail_call_direct(struct bpf_prog *bpf_prog,\n \tif (stack_depth)\n \t\tEMIT3_off32(0x48, 0x81, 0xC4, round_up(stack_depth, 8));\n \n-\temit_nops(\u0026prog, X86_PATCH_SIZE);\n+\temit_nops(jit, X86_PATCH_SIZE);\n \n \t/* out: */\n-\tctx-\u003etail_call_direct_label = prog - start;\n-\n-\t*pprog = prog;\n+\tctx-\u003etail_call_direct_label = jit-\u003eprog - start;\n }\n \n static void bpf_tail_call_direct_fixup(struct bpf_prog *prog)\n@@ -935,10 +904,9 @@ static void bpf_tail_call_direct_fixup(struct bpf_prog *prog)\n \t}\n }\n \n-static void emit_mov_imm32(u8 **pprog, bool sign_propagate,\n+static void emit_mov_imm32(struct jit_emit_context *jit, bool sign_propagate,\n \t\t\t   u32 dst_reg, const u32 imm32)\n {\n-\tu8 *prog = *pprog;\n \tu8 b1, b2, b3;\n \n \t/*\n@@ -972,14 +940,12 @@ static void emit_mov_imm32(u8 **pprog, bool sign_propagate,\n \t\tEMIT1(add_1mod(0x40, dst_reg));\n \tEMIT1_off32(add_1reg(0xB8, dst_reg), imm32);\n done:\n-\t*pprog = prog;\n }\n \n-static void emit_mov_imm64(u8 **pprog, u32 dst_reg,\n+static void emit_mov_imm64(struct jit_emit_context *jit, u32 dst_reg,\n \t\t\t   const u32 imm32_hi, const u32 imm32_lo)\n {\n \tu64 imm64 = ((u64)imm32_hi \u003c\u003c 32) | (u32)imm32_lo;\n-\tu8 *prog = *pprog;\n \n \tif (is_uimm32(imm64)) {\n \t\t/*\n@@ -988,23 +954,19 @@ static void emit_mov_imm64(u8 **pprog, u32 dst_reg,\n \t\t * directly, so save couple of bytes by just doing\n \t\t * 'mov %eax, imm32' instead.\n \t\t */\n-\t\temit_mov_imm32(\u0026prog, false, dst_reg, imm32_lo);\n+\t\temit_mov_imm32(jit, false, dst_reg, imm32_lo);\n \t} else if (is_simm32(imm64)) {\n-\t\temit_mov_imm32(\u0026prog, true, dst_reg, imm32_lo);\n+\t\temit_mov_imm32(jit, true, dst_reg, imm32_lo);\n \t} else {\n \t\t/* movabsq rax, imm64 */\n \t\tEMIT2(add_1mod(0x48, dst_reg), add_1reg(0xB8, dst_reg));\n \t\tEMIT(imm32_lo, 4);\n \t\tEMIT(imm32_hi, 4);\n \t}\n-\n-\t*pprog = prog;\n }\n \n-static void emit_mov_reg(u8 **pprog, bool is64, u32 dst_reg, u32 src_reg)\n+static void emit_mov_reg(struct jit_emit_context *jit, bool is64, u32 dst_reg, u32 src_reg)\n {\n-\tu8 *prog = *pprog;\n-\n \tif (is64) {\n \t\t/* mov dst, src */\n \t\tEMIT_mov(dst_reg, src_reg);\n@@ -1014,15 +976,11 @@ static void emit_mov_reg(u8 **pprog, bool is64, u32 dst_reg, u32 src_reg)\n \t\t\tEMIT1(add_2mod(0x40, dst_reg, src_reg));\n \t\tEMIT2(0x89, add_2reg(0xC0, dst_reg, src_reg));\n \t}\n-\n-\t*pprog = prog;\n }\n \n-static void emit_movsx_reg(u8 **pprog, int num_bits, bool is64, u32 dst_reg,\n+static void emit_movsx_reg(struct jit_emit_context *jit, int num_bits, bool is64, u32 dst_reg,\n \t\t\t   u32 src_reg)\n {\n-\tu8 *prog = *pprog;\n-\n \tif (is64) {\n \t\t/* movs[b,w,l]q dst, src */\n \t\tif (num_bits == 8)\n@@ -1046,15 +1004,11 @@ static void emit_movsx_reg(u8 **pprog, int num_bits, bool is64, u32 dst_reg,\n \t\t\t      add_2reg(0xC0, src_reg, dst_reg));\n \t\t}\n \t}\n-\n-\t*pprog = prog;\n }\n \n /* Emit the suffix (ModR/M etc) for addressing *(ptr_reg + off) and val_reg */\n-static void emit_insn_suffix(u8 **pprog, u32 ptr_reg, u32 val_reg, int off)\n+static void emit_insn_suffix(struct jit_emit_context *jit, u32 ptr_reg, u32 val_reg, int off)\n {\n-\tu8 *prog = *pprog;\n-\n \tif (is_imm8(off)) {\n \t\t/* 1-byte signed displacement.\n \t\t *\n@@ -1067,54 +1021,43 @@ static void emit_insn_suffix(u8 **pprog, u32 ptr_reg, u32 val_reg, int off)\n \t\t/* 4-byte signed displacement */\n \t\tEMIT1_off32(add_2reg(0x80, ptr_reg, val_reg), off);\n \t}\n-\t*pprog = prog;\n }\n \n-static void emit_insn_suffix_SIB(u8 **pprog, u32 ptr_reg, u32 val_reg, u32 index_reg, int off)\n+static void emit_insn_suffix_SIB(struct jit_emit_context *jit, u32 ptr_reg, u32 val_reg,\n+\t\t\t\t u32 index_reg, int off)\n {\n-\tu8 *prog = *pprog;\n-\n \tif (is_imm8(off)) {\n \t\tEMIT3(add_2reg(0x44, BPF_REG_0, val_reg), add_2reg(0, ptr_reg, index_reg) /* SIB */, off);\n \t} else {\n \t\tEMIT2_off32(add_2reg(0x84, BPF_REG_0, val_reg), add_2reg(0, ptr_reg, index_reg) /* SIB */, off);\n \t}\n-\t*pprog = prog;\n }\n \n /*\n  * Emit a REX byte if it will be necessary to address these registers\n  */\n-static void maybe_emit_mod(u8 **pprog, u32 dst_reg, u32 src_reg, bool is64)\n+static void maybe_emit_mod(struct jit_emit_context *jit, u32 dst_reg, u32 src_reg, bool is64)\n {\n-\tu8 *prog = *pprog;\n-\n \tif (is64)\n \t\tEMIT1(add_2mod(0x48, dst_reg, src_reg));\n \telse if (is_ereg(dst_reg) || is_ereg(src_reg))\n \t\tEMIT1(add_2mod(0x40, dst_reg, src_reg));\n-\t*pprog = prog;\n }\n \n /*\n  * Similar version of maybe_emit_mod() for a single register\n  */\n-static void maybe_emit_1mod(u8 **pprog, u32 reg, bool is64)\n+static void maybe_emit_1mod(struct jit_emit_context *jit, u32 reg, bool is64)\n {\n-\tu8 *prog = *pprog;\n-\n \tif (is64)\n \t\tEMIT1(add_1mod(0x48, reg));\n \telse if (is_ereg(reg))\n \t\tEMIT1(add_1mod(0x40, reg));\n-\t*pprog = prog;\n }\n \n /* LDX: dst_reg = *(u8*)(src_reg + off) */\n-static void emit_ldx(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, int off)\n+static void emit_ldx(struct jit_emit_context *jit, u32 size, u32 dst_reg, u32 src_reg, int off)\n {\n-\tu8 *prog = *pprog;\n-\n \tswitch (size) {\n \tcase BPF_B:\n \t\t/* Emit 'movzx rax, byte ptr [rax + off]' */\n@@ -1136,15 +1079,12 @@ static void emit_ldx(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, int off)\n \t\tEMIT2(add_2mod(0x48, src_reg, dst_reg), 0x8B);\n \t\tbreak;\n \t}\n-\temit_insn_suffix(\u0026prog, src_reg, dst_reg, off);\n-\t*pprog = prog;\n+\temit_insn_suffix(jit, src_reg, dst_reg, off);\n }\n \n /* LDSX: dst_reg = *(s8*)(src_reg + off) */\n-static void emit_ldsx(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, int off)\n+static void emit_ldsx(struct jit_emit_context *jit, u32 size, u32 dst_reg, u32 src_reg, int off)\n {\n-\tu8 *prog = *pprog;\n-\n \tswitch (size) {\n \tcase BPF_B:\n \t\t/* Emit 'movsx rax, byte ptr [rax + off]' */\n@@ -1159,14 +1099,12 @@ static void emit_ldsx(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, int off)\n \t\tEMIT2(add_2mod(0x48, src_reg, dst_reg), 0x63);\n \t\tbreak;\n \t}\n-\temit_insn_suffix(\u0026prog, src_reg, dst_reg, off);\n-\t*pprog = prog;\n+\temit_insn_suffix(jit, src_reg, dst_reg, off);\n }\n \n-static void emit_ldx_index(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, u32 index_reg, int off)\n+static void emit_ldx_index(struct jit_emit_context *jit, u32 size, u32 dst_reg, u32 src_reg,\n+\t\t\t   u32 index_reg, int off)\n {\n-\tu8 *prog = *pprog;\n-\n \tswitch (size) {\n \tcase BPF_B:\n \t\t/* movzx rax, byte ptr [rax + r12 + off] */\n@@ -1185,14 +1123,12 @@ static void emit_ldx_index(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, u32 i\n \t\tEMIT2(add_3mod(0x48, src_reg, dst_reg, index_reg), 0x8B);\n \t\tbreak;\n \t}\n-\temit_insn_suffix_SIB(\u0026prog, src_reg, dst_reg, index_reg, off);\n-\t*pprog = prog;\n+\temit_insn_suffix_SIB(jit, src_reg, dst_reg, index_reg, off);\n }\n \n-static void emit_ldsx_index(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, u32 index_reg, int off)\n+static void emit_ldsx_index(struct jit_emit_context *jit, u32 size, u32 dst_reg, u32 src_reg,\n+\t\t\t    u32 index_reg, int off)\n {\n-\tu8 *prog = *pprog;\n-\n \tswitch (size) {\n \tcase BPF_B:\n \t\t/* movsx rax, byte ptr [rax + r12 + off] */\n@@ -1207,25 +1143,22 @@ static void emit_ldsx_index(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, u32\n \t\tEMIT2(add_3mod(0x48, src_reg, dst_reg, index_reg), 0x63);\n \t\tbreak;\n \t}\n-\temit_insn_suffix_SIB(\u0026prog, src_reg, dst_reg, index_reg, off);\n-\t*pprog = prog;\n+\temit_insn_suffix_SIB(jit, src_reg, dst_reg, index_reg, off);\n }\n \n-static void emit_ldx_r12(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, int off)\n+static void emit_ldx_r12(struct jit_emit_context *jit, u32 size, u32 dst_reg, u32 src_reg, int off)\n {\n-\temit_ldx_index(pprog, size, dst_reg, src_reg, X86_REG_R12, off);\n+\temit_ldx_index(jit, size, dst_reg, src_reg, X86_REG_R12, off);\n }\n \n-static void emit_ldsx_r12(u8 **prog, u32 size, u32 dst_reg, u32 src_reg, int off)\n+static void emit_ldsx_r12(struct jit_emit_context *jit, u32 size, u32 dst_reg, u32 src_reg, int off)\n {\n-\temit_ldsx_index(prog, size, dst_reg, src_reg, X86_REG_R12, off);\n+\temit_ldsx_index(jit, size, dst_reg, src_reg, X86_REG_R12, off);\n }\n \n /* STX: *(u8*)(dst_reg + off) = src_reg */\n-static void emit_stx(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, int off)\n+static void emit_stx(struct jit_emit_context *jit, u32 size, u32 dst_reg, u32 src_reg, int off)\n {\n-\tu8 *prog = *pprog;\n-\n \tswitch (size) {\n \tcase BPF_B:\n \t\t/* Emit 'mov byte ptr [rax + off], al' */\n@@ -1251,15 +1184,13 @@ static void emit_stx(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, int off)\n \t\tEMIT2(add_2mod(0x48, dst_reg, src_reg), 0x89);\n \t\tbreak;\n \t}\n-\temit_insn_suffix(\u0026prog, dst_reg, src_reg, off);\n-\t*pprog = prog;\n+\temit_insn_suffix(jit, dst_reg, src_reg, off);\n }\n \n /* STX: *(u8*)(dst_reg + index_reg + off) = src_reg */\n-static void emit_stx_index(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, u32 index_reg, int off)\n+static void emit_stx_index(struct jit_emit_context *jit, u32 size, u32 dst_reg, u32 src_reg,\n+\t\t\t   u32 index_reg, int off)\n {\n-\tu8 *prog = *pprog;\n-\n \tswitch (size) {\n \tcase BPF_B:\n \t\t/* mov byte ptr [rax + r12 + off], al */\n@@ -1278,20 +1209,18 @@ static void emit_stx_index(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, u32 i\n \t\tEMIT2(add_3mod(0x48, dst_reg, src_reg, index_reg), 0x89);\n \t\tbreak;\n \t}\n-\temit_insn_suffix_SIB(\u0026prog, dst_reg, src_reg, index_reg, off);\n-\t*pprog = prog;\n+\temit_insn_suffix_SIB(jit, dst_reg, src_reg, index_reg, off);\n }\n \n-static void emit_stx_r12(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, int off)\n+static void emit_stx_r12(struct jit_emit_context *jit, u32 size, u32 dst_reg, u32 src_reg, int off)\n {\n-\temit_stx_index(pprog, size, dst_reg, src_reg, X86_REG_R12, off);\n+\temit_stx_index(jit, size, dst_reg, src_reg, X86_REG_R12, off);\n }\n \n /* ST: *(u8*)(dst_reg + index_reg + off) = imm32 */\n-static void emit_st_index(u8 **pprog, u32 size, u32 dst_reg, u32 index_reg, int off, int imm)\n+static void emit_st_index(struct jit_emit_context *jit, u32 size, u32 dst_reg, u32 index_reg,\n+\t\t\t  int off, int imm)\n {\n-\tu8 *prog = *pprog;\n-\n \tswitch (size) {\n \tcase BPF_B:\n \t\t/* mov byte ptr [rax + r12 + off], imm8 */\n@@ -1310,35 +1239,32 @@ static void emit_st_index(u8 **pprog, u32 size, u32 dst_reg, u32 index_reg, int\n \t\tEMIT2(add_3mod(0x48, dst_reg, 0, index_reg), 0xC7);\n \t\tbreak;\n \t}\n-\temit_insn_suffix_SIB(\u0026prog, dst_reg, 0, index_reg, off);\n+\temit_insn_suffix_SIB(jit, dst_reg, 0, index_reg, off);\n \tEMIT(imm, bpf_size_to_x86_bytes(size));\n-\t*pprog = prog;\n }\n \n-static void emit_st_r12(u8 **pprog, u32 size, u32 dst_reg, int off, int imm)\n+static void emit_st_r12(struct jit_emit_context *jit, u32 size, u32 dst_reg, int off, int imm)\n {\n-\temit_st_index(pprog, size, dst_reg, X86_REG_R12, off, imm);\n+\temit_st_index(jit, size, dst_reg, X86_REG_R12, off, imm);\n }\n \n-static void emit_store_stack_imm64(u8 **pprog, int reg, int stack_off, u64 imm64)\n+static void emit_store_stack_imm64(struct jit_emit_context *jit, int reg, int stack_off, u64 imm64)\n {\n \t/*\n \t * mov reg, imm64\n \t * mov QWORD PTR [rbp + stack_off], reg\n \t */\n-\temit_mov_imm64(pprog, reg, imm64 \u003e\u003e 32, (u32) imm64);\n-\temit_stx(pprog, BPF_DW, BPF_REG_FP, reg, stack_off);\n+\temit_mov_imm64(jit, reg, imm64 \u003e\u003e 32, (u32) imm64);\n+\temit_stx(jit, BPF_DW, BPF_REG_FP, reg, stack_off);\n }\n \n-static int emit_atomic_rmw(u8 **pprog, u32 atomic_op,\n+static int emit_atomic_rmw(struct jit_emit_context *jit, u32 atomic_op,\n \t\t\t   u32 dst_reg, u32 src_reg, s16 off, u8 bpf_size)\n {\n-\tu8 *prog = *pprog;\n-\n \tif (atomic_op != BPF_XCHG)\n \t\tEMIT1(0xF0); /* lock prefix */\n \n-\tmaybe_emit_mod(\u0026prog, dst_reg, src_reg, bpf_size == BPF_DW);\n+\tmaybe_emit_mod(jit, dst_reg, src_reg, bpf_size == BPF_DW);\n \n \t/* emit opcode */\n \tswitch (atomic_op) {\n@@ -1366,18 +1292,15 @@ static int emit_atomic_rmw(u8 **pprog, u32 atomic_op,\n \t\treturn -EFAULT;\n \t}\n \n-\temit_insn_suffix(\u0026prog, dst_reg, src_reg, off);\n+\temit_insn_suffix(jit, dst_reg, src_reg, off);\n \n-\t*pprog = prog;\n \treturn 0;\n }\n \n-static int emit_atomic_rmw_index(u8 **pprog, u32 atomic_op, u32 size,\n+static int emit_atomic_rmw_index(struct jit_emit_context *jit, u32 atomic_op, u32 size,\n \t\t\t\t u32 dst_reg, u32 src_reg, u32 index_reg,\n \t\t\t\t int off)\n {\n-\tu8 *prog = *pprog;\n-\n \tif (atomic_op != BPF_XCHG)\n \t\tEMIT1(0xF0); /* lock prefix */\n \n@@ -1418,22 +1341,21 @@ static int emit_atomic_rmw_index(u8 **pprog, u32 atomic_op, u32 size,\n \t\tpr_err(\"bpf_jit: unknown atomic opcode %02x\\n\", atomic_op);\n \t\treturn -EFAULT;\n \t}\n-\temit_insn_suffix_SIB(\u0026prog, dst_reg, src_reg, index_reg, off);\n-\t*pprog = prog;\n+\temit_insn_suffix_SIB(jit, dst_reg, src_reg, index_reg, off);\n \treturn 0;\n }\n \n-static int emit_atomic_ld_st(u8 **pprog, u32 atomic_op, u32 dst_reg,\n+static int emit_atomic_ld_st(struct jit_emit_context *jit, u32 atomic_op, u32 dst_reg,\n \t\t\t     u32 src_reg, s16 off, u8 bpf_size)\n {\n \tswitch (atomic_op) {\n \tcase BPF_LOAD_ACQ:\n \t\t/* dst_reg = smp_load_acquire(src_reg + off16) */\n-\t\temit_ldx(pprog, bpf_size, dst_reg, src_reg, off);\n+\t\temit_ldx(jit, bpf_size, dst_reg, src_reg, off);\n \t\tbreak;\n \tcase BPF_STORE_REL:\n \t\t/* smp_store_release(dst_reg + off16, src_reg) */\n-\t\temit_stx(pprog, bpf_size, dst_reg, src_reg, off);\n+\t\temit_stx(jit, bpf_size, dst_reg, src_reg, off);\n \t\tbreak;\n \tdefault:\n \t\tpr_err(\"bpf_jit: unknown atomic load/store opcode %02x\\n\",\n@@ -1444,18 +1366,18 @@ static int emit_atomic_ld_st(u8 **pprog, u32 atomic_op, u32 dst_reg,\n \treturn 0;\n }\n \n-static int emit_atomic_ld_st_index(u8 **pprog, u32 atomic_op, u32 size,\n+static int emit_atomic_ld_st_index(struct jit_emit_context *jit, u32 atomic_op, u32 size,\n \t\t\t\t   u32 dst_reg, u32 src_reg, u32 index_reg,\n \t\t\t\t   int off)\n {\n \tswitch (atomic_op) {\n \tcase BPF_LOAD_ACQ:\n \t\t/* dst_reg = smp_load_acquire(src_reg + idx_reg + off16) */\n-\t\temit_ldx_index(pprog, size, dst_reg, src_reg, index_reg, off);\n+\t\temit_ldx_index(jit, size, dst_reg, src_reg, index_reg, off);\n \t\tbreak;\n \tcase BPF_STORE_REL:\n \t\t/* smp_store_release(dst_reg + idx_reg + off16, src_reg) */\n-\t\temit_stx_index(pprog, size, dst_reg, src_reg, index_reg, off);\n+\t\temit_stx_index(jit, size, dst_reg, src_reg, index_reg, off);\n \t\tbreak;\n \tdefault:\n \t\tpr_err(\"bpf_jit: unknown atomic load/store opcode %02x\\n\",\n@@ -1562,10 +1484,9 @@ static void detect_reg_usage(struct bpf_insn *insn, int insn_cnt,\n  * l: vector length (128 bit or 256 bit) or reserved\n  * pp: opcode prefix (none, 0x66, 0xf2 or 0xf3)\n  */\n-static void emit_3vex(u8 **pprog, bool r, bool x, bool b, u8 m,\n+static void emit_3vex(struct jit_emit_context *jit, bool r, bool x, bool b, u8 m,\n \t\t      bool w, u8 src_reg2, bool l, u8 pp)\n {\n-\tu8 *prog = *pprog;\n \tconst u8 b0 = 0xc4; /* first byte of 3-byte VEX prefix */\n \tu8 b1, b2;\n \tu8 vvvv = reg2hex[src_reg2];\n@@ -1595,27 +1516,22 @@ static void emit_3vex(u8 **pprog, bool r, bool x, bool b, u8 m,\n \tb2 = (w \u003c\u003c 7) | ((~vvvv \u0026 0xf) \u003c\u003c 3) | (l \u003c\u003c 2) | (pp \u0026 3);\n \n \tEMIT3(b0, b1, b2);\n-\t*pprog = prog;\n }\n \n /* emit BMI2 shift instruction */\n-static void emit_shiftx(u8 **pprog, u32 dst_reg, u8 src_reg, bool is64, u8 op)\n+static void emit_shiftx(struct jit_emit_context *jit, u32 dst_reg, u8 src_reg, bool is64, u8 op)\n {\n-\tu8 *prog = *pprog;\n \tbool r = is_ereg(dst_reg);\n \tu8 m = 2; /* escape code 0f38 */\n \n-\temit_3vex(\u0026prog, r, false, r, m, is64, src_reg, false, op);\n+\temit_3vex(jit, r, false, r, m, is64, src_reg, false, op);\n \tEMIT2(0xf7, add_2reg(0xC0, dst_reg, dst_reg));\n-\t*pprog = prog;\n }\n \n-static void emit_priv_frame_ptr(u8 **pprog, void __percpu *priv_frame_ptr)\n+static void emit_priv_frame_ptr(struct jit_emit_context *jit, void __percpu *priv_frame_ptr)\n {\n-\tu8 *prog = *pprog;\n-\n \t/* movabs r9, priv_frame_ptr */\n-\temit_mov_imm64(\u0026prog, X86_REG_R9, (__force long) priv_frame_ptr \u003e\u003e 32,\n+\temit_mov_imm64(jit, X86_REG_R9, (__force long) priv_frame_ptr \u003e\u003e 32,\n \t\t       (u32) (__force long) priv_frame_ptr);\n \n #ifdef CONFIG_SMP\n@@ -1624,11 +1540,9 @@ static void emit_priv_frame_ptr(u8 **pprog, void __percpu *priv_frame_ptr)\n \tEMIT3(0x03, 0x0c, 0x25);\n \tEMIT((u32)(unsigned long)\u0026this_cpu_off, 4);\n #endif\n-\n-\t*pprog = prog;\n }\n \n-#define INSN_SZ_DIFF (((addrs[i] - addrs[i - 1]) - (prog - temp)))\n+#define INSN_SZ_DIFF (((addrs[i] - addrs[i - 1]) - (jit-\u003eprog - temp)))\n \n #define __LOAD_TCC_PTR(off)\t\t\t\\\n \tEMIT3_off32(0x48, 0x8B, 0x85, off)\n@@ -1640,10 +1554,9 @@ static void emit_priv_frame_ptr(u8 **pprog, void __percpu *priv_frame_ptr)\n #define PRIV_STACK_GUARD_SZ    8\n #define PRIV_STACK_GUARD_VAL   0xEB9F12345678eb9fULL\n \n-static int emit_spectre_bhb_barrier(u8 **pprog, u8 *ip,\n+static int emit_spectre_bhb_barrier(struct jit_emit_context *jit, u8 *ip,\n \t\t\t\t    struct bpf_prog *bpf_prog)\n {\n-\tu8 *prog = *pprog;\n \tu8 *func;\n \n \tif (cpu_feature_enabled(X86_FEATURE_CLEAR_BHB_LOOP)) {\n@@ -1653,9 +1566,9 @@ static int emit_spectre_bhb_barrier(u8 **pprog, u8 *ip,\n \t\tip += 2;\n \n \t\tfunc = (u8 *)clear_bhb_loop;\n-\t\tip += x86_call_depth_emit_accounting(\u0026prog, func, ip);\n+\t\tip += bpf_call_depth_emit_accounting(jit, func, ip);\n \n-\t\tif (emit_call(\u0026prog, func, ip))\n+\t\tif (emit_call(jit, func, ip))\n \t\t\treturn -EINVAL;\n \t\tEMIT1(0x59); /* pop rcx */\n \t\tEMIT1(0x58); /* pop rax */\n@@ -1678,7 +1591,6 @@ static int emit_spectre_bhb_barrier(u8 **pprog, u8 *ip,\n \t\t */\n \t\tEMIT5(0xF3, 0x48, 0x0F, 0x1E, 0xF8); /* ibhf */\n \t}\n-\t*pprog = prog;\n \treturn 0;\n }\n \n@@ -1689,10 +1601,9 @@ static int emit_spectre_bhb_barrier(u8 **pprog, u8 *ip,\n  * arena NULL is offset 0. Return the number of emitted bytes.\n  */\n static int emit_kfunc_arena_args(struct bpf_prog *bpf_prog,\n-\t\t\t\t const struct btf_func_model *fm, u8 **pprog)\n+\t\t\t\t const struct btf_func_model *fm, struct jit_emit_context *jit)\n {\n-\tu8 *prog = *pprog;\n-\tu8 *start = prog;\n+\tu8 *start = jit-\u003eprog;\n \tint i;\n \n \tfor (i = 0; i \u003c min_t(int, fm-\u003enr_args, MAX_BPF_FUNC_REG_ARGS); i++) {\n@@ -1705,20 +1616,19 @@ static int emit_kfunc_arena_args(struct bpf_prog *bpf_prog,\n \t\t\treturn -EINVAL;\n \n \t\t/* mov eN, eN: truncate and clear the upper 32 bits */\n-\t\temit_mov_reg(\u0026prog, false, reg, reg);\n+\t\temit_mov_reg(jit, false, reg, reg);\n \t\tif (flags \u0026 BTF_FMODEL_NULLABLE_ARG) {\n \t\t\t/* test eN, eN; jz over the 3-byte add */\n-\t\t\tmaybe_emit_mod(\u0026prog, reg, reg, false);\n+\t\t\tmaybe_emit_mod(jit, reg, reg, false);\n \t\t\tEMIT2(0x85, add_2reg(0xC0, reg, reg));\n \t\t\tEMIT2(X86_JE, 3);\n \t\t}\n \t\t/* add rN, r12 */\n-\t\tmaybe_emit_mod(\u0026prog, reg, X86_REG_R12, true);\n+\t\tmaybe_emit_mod(jit, reg, X86_REG_R12, true);\n \t\tEMIT2(0x01, add_2reg(0xC0, reg, X86_REG_R12));\n \t}\n \n-\t*pprog = prog;\n-\treturn prog - start;\n+\treturn jit-\u003eprog - start;\n }\n \n static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *addrs, u8 *image,\n@@ -1726,6 +1636,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *\n {\n \tbool tail_call_reachable = bpf_prog-\u003eaux-\u003etail_call_reachable;\n \tstruct bpf_insn *insn = bpf_prog-\u003einsnsi;\n+\tstruct jit_emit_context *jit = \u0026ctx-\u003ejit;\n \tbool callee_regs_used[4] = {};\n \tint insn_cnt = bpf_prog-\u003elen;\n \tbool seen_exit = false;\n@@ -1736,12 +1647,15 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *\n \tvoid __percpu *priv_stack_ptr;\n \tint i, excnt = 0;\n \tint ilen, proglen = 0;\n-\tu8 *ip, *prog = temp;\n+\tu8 *ip;\n \tu32 stack_depth;\n \tint callee_saved_size;\n \ts32 outgoing_arg_base;\n \tint err;\n \n+\tjit-\u003eprog = temp;\n+\tjit-\u003edry_run = false;\n+\n \tstack_depth = bpf_prog-\u003eaux-\u003estack_depth;\n \tout_stack_arg_cnt = bpf_out_stack_arg_cnt(env, bpf_prog);\n \tpriv_stack_ptr = bpf_prog-\u003eaux-\u003epriv_stack_ptr;\n@@ -1777,11 +1691,11 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *\n \n \tdetect_reg_usage(insn, insn_cnt, callee_regs_used);\n \n-\temit_prologue(\u0026prog, image, stack_depth,\n+\temit_prologue(jit, image, stack_depth,\n \t\t      bpf_prog_was_classic(bpf_prog), tail_call_reachable,\n \t\t      bpf_is_subprog(bpf_prog), bpf_prog-\u003eaux-\u003eexception_cb);\n \n-\tbpf_prog-\u003eaux-\u003eksym.fp_start = prog - temp;\n+\tbpf_prog-\u003eaux-\u003eksym.fp_start = jit-\u003eprog - temp;\n \n \t/* Exception callback will clobber callee regs for its own use, and\n \t * restore the original callee regs from main prog's stack frame.\n@@ -1791,12 +1705,12 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *\n \t\t * register, as we throw after entry into the kernel, which may\n \t\t * overwrite r12.\n \t\t */\n-\t\tpush_r12(\u0026prog);\n-\t\tpush_callee_regs(\u0026prog, all_callee_regs_used);\n+\t\tpush_r12(jit);\n+\t\tpush_callee_regs(jit, all_callee_regs_used);\n \t} else {\n \t\tif (arena_vm_start)\n-\t\t\tpush_r12(\u0026prog);\n-\t\tpush_callee_regs(\u0026prog, callee_regs_used);\n+\t\t\tpush_r12(jit);\n+\t\tpush_callee_regs(jit, callee_regs_used);\n \t}\n \n \t/* Compute callee-saved register area size. */\n@@ -1834,21 +1748,21 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *\n \toutgoing_rsp = out_stack_arg_cnt \u003e 1 ? (out_stack_arg_cnt - 1) * 8 : 0;\n \tif (bpf_prog-\u003eaux-\u003eexception_boundary)\n \t\tbpf_prog-\u003eaux-\u003estack_arg_sp_adjust = outgoing_rsp;\n-\temit_sub_rsp(\u0026prog, outgoing_rsp);\n+\temit_sub_rsp(jit, outgoing_rsp);\n \n \tif (arena_vm_start)\n-\t\temit_mov_imm64(\u0026prog, X86_REG_R12,\n+\t\temit_mov_imm64(jit, X86_REG_R12,\n \t\t\t       arena_vm_start \u003e\u003e 32, (u32) arena_vm_start);\n \n \tif (priv_frame_ptr)\n-\t\temit_priv_frame_ptr(\u0026prog, priv_frame_ptr);\n+\t\temit_priv_frame_ptr(jit, priv_frame_ptr);\n \n-\tilen = prog - temp;\n+\tilen = jit-\u003eprog - temp;\n \tif (rw_image)\n \t\tmemcpy(rw_image + proglen, temp, ilen);\n \tproglen += ilen;\n \taddrs[0] = proglen;\n-\tprog = temp;\n+\tjit-\u003eprog = temp;\n \n \tfor (i = 1; i \u003c= insn_cnt; i++, insn++) {\n \t\tconst s32 imm32 = insn-\u003eimm;\n@@ -1873,7 +1787,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *\n \t\tif (bpf_insn_is_indirect_target(env, bpf_prog, i - 1))\n \t\t\tEMIT_ENDBR();\n \n-\t\tip = image + addrs[i - 1] + (prog - temp);\n+\t\tip = image + addrs[i - 1] + (jit-\u003eprog - temp);\n \n \t\tswitch (insn-\u003ecode) {\n \t\t\t/* ALU */\n@@ -1887,7 +1801,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *\n \t\tcase BPF_ALU64 | BPF_AND | BPF_X:\n \t\tcase BPF_ALU64 | BPF_OR | BPF_X:\n \t\tcase BPF_ALU64 | BPF_XOR | BPF_X:\n-\t\t\tmaybe_emit_mod(\u0026prog, dst_reg, src_reg,\n+\t\t\tmaybe_emit_mod(jit, dst_reg, src_reg,\n \t\t\t\t       BPF_CLASS(insn-\u003ecode) == BPF_ALU64);\n \t\t\tb2 = simple_alu_opcodes[BPF_OP(insn-\u003ecode)];\n \t\t\tEMIT2(b2, add_2reg(0xC0, dst_reg, src_reg));\n@@ -1897,32 +1811,32 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *\n \t\t\tif (insn_is_cast_user(insn)) {\n \t\t\t\tif (dst_reg != src_reg)\n \t\t\t\t\t/* 32-bit mov */\n-\t\t\t\t\temit_mov_reg(\u0026prog, false, dst_reg, src_reg);\n+\t\t\t\t\temit_mov_reg(jit, false, dst_reg, src_reg);\n \t\t\t\t/* shl dst_reg, 32 */\n-\t\t\t\tmaybe_emit_1mod(\u0026prog, dst_reg, true);\n+\t\t\t\tmaybe_emit_1mod(jit, dst_reg, true);\n \t\t\t\tEMIT3(0xC1, add_1reg(0xE0, dst_reg), 32);\n \n \t\t\t\t/* or dst_reg, user_vm_start */\n-\t\t\t\tmaybe_emit_1mod(\u0026prog, dst_reg, true);\n+\t\t\t\tmaybe_emit_1mod(jit, dst_reg, true);\n \t\t\t\tif (is_axreg(dst_reg))\n \t\t\t\t\tEMIT1_off32(0x0D,  user_vm_start \u003e\u003e 32);\n \t\t\t\telse\n \t\t\t\t\tEMIT2_off32(0x81, add_1reg(0xC8, dst_reg),  user_vm_start \u003e\u003e 32);\n \n \t\t\t\t/* rol dst_reg, 32 */\n-\t\t\t\tmaybe_emit_1mod(\u0026prog, dst_reg, true);\n+\t\t\t\tmaybe_emit_1mod(jit, dst_reg, true);\n \t\t\t\tEMIT3(0xC1, add_1reg(0xC0, dst_reg), 32);\n \n \t\t\t\t/* xor r11, r11 */\n \t\t\t\tEMIT3(0x4D, 0x31, 0xDB);\n \n \t\t\t\t/* test dst_reg32, dst_reg32; check if lower 32-bit are zero */\n-\t\t\t\tmaybe_emit_mod(\u0026prog, dst_reg, dst_reg, false);\n+\t\t\t\tmaybe_emit_mod(jit, dst_reg, dst_reg, false);\n \t\t\t\tEMIT2(0x85, add_2reg(0xC0, dst_reg, dst_reg));\n \n \t\t\t\t/* cmove r11, dst_reg; if so, set dst_reg to zero */\n \t\t\t\t/* WARNING: Intel swapped src/dst register encoding in CMOVcc !!! */\n-\t\t\t\tmaybe_emit_mod(\u0026prog, AUX_REG, dst_reg, true);\n+\t\t\t\tmaybe_emit_mod(jit, AUX_REG, dst_reg, true);\n \t\t\t\tEMIT3(0x0F, 0x44, add_2reg(0xC0, AUX_REG, dst_reg));\n \t\t\t\tbreak;\n \t\t\t} else if (insn_is_mov_percpu_addr(insn)) {\n@@ -1939,11 +1853,11 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *\n \t\t\tfallthrough;\n \t\tcase BPF_ALU | BPF_MOV | BPF_X:\n \t\t\tif (insn-\u003eoff == 0)\n-\t\t\t\temit_mov_reg(\u0026prog,\n+\t\t\t\temit_mov_reg(jit,\n \t\t\t\t\t     BPF_CLASS(insn-\u003ecode) == BPF_ALU64,\n \t\t\t\t\t     dst_reg, src_reg);\n \t\t\telse\n-\t\t\t\temit_movsx_reg(\u0026prog, insn-\u003eoff,\n+\t\t\t\temit_movsx_reg(jit, insn-\u003eoff,\n \t\t\t\t\t       BPF_CLASS(insn-\u003ecode) == BPF_ALU64,\n \t\t\t\t\t       dst_reg, src_reg);\n \t\t\tbreak;\n@@ -1951,7 +1865,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *\n \t\t\t/* neg dst */\n \t\tcase BPF_ALU | BPF_NEG:\n \t\tcase BPF_ALU64 | BPF_NEG:\n-\t\t\tmaybe_emit_1mod(\u0026prog, dst_reg,\n+\t\t\tmaybe_emit_1mod(jit, dst_reg,\n \t\t\t\t\tBPF_CLASS(insn-\u003ecode) == BPF_ALU64);\n \t\t\tEMIT2(0xF7, add_1reg(0xD8, dst_reg));\n \t\t\tbreak;\n@@ -1966,7 +1880,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *\n \t\tcase BPF_ALU64 | BPF_AND | BPF_K:\n \t\tcase BPF_ALU64 | BPF_OR | BPF_K:\n \t\tcase BPF_ALU64 | BPF_XOR | BPF_K:\n-\t\t\tmaybe_emit_1mod(\u0026prog, dst_reg,\n+\t\t\tmaybe_emit_1mod(jit, dst_reg,\n \t\t\t\t\tBPF_CLASS(insn-\u003ecode) == BPF_ALU64);\n \n \t\t\t/*\n@@ -2006,12 +1920,12 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *\n \n \t\tcase BPF_ALU64 | BPF_MOV | BPF_K:\n \t\tcase BPF_ALU | BPF_MOV | BPF_K:\n-\t\t\temit_mov_imm32(\u0026prog, BPF_CLASS(insn-\u003ecode) == BPF_ALU64,\n+\t\t\temit_mov_imm32(jit, BPF_CLASS(insn-\u003ecode) == BPF_ALU64,\n \t\t\t\t       dst_reg, imm32);\n \t\t\tbreak;\n \n \t\tcase BPF_LD | BPF_IMM | BPF_DW:\n-\t\t\temit_mov_imm64(\u0026prog, dst_reg, insn[1].imm, insn[0].imm);\n+\t\t\temit_mov_imm64(jit, dst_reg, insn[1].imm, insn[0].imm);\n \t\t\tinsn++;\n \t\t\ti++;\n \t\t\tbreak;\n@@ -2047,7 +1961,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *\n \n \t\t\tif (dst_reg != BPF_REG_0)\n \t\t\t\t/* mov rax, dst_reg */\n-\t\t\t\temit_mov_reg(\u0026prog, is64, BPF_REG_0, dst_reg);\n+\t\t\t\temit_mov_reg(jit, is64, BPF_REG_0, dst_reg);\n \n \t\t\tif (insn-\u003eoff == 0) {\n \t\t\t\t/*\n@@ -2057,7 +1971,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *\n \t\t\t\tEMIT2(0x31, 0xd2);\n \n \t\t\t\t/* div src_reg */\n-\t\t\t\tmaybe_emit_1mod(\u0026prog, src_reg, is64);\n+\t\t\t\tmaybe_emit_1mod(jit, src_reg, is64);\n \t\t\t\tEMIT2(0xF7, add_1reg(0xF0, src_reg));\n \t\t\t} else {\n \t\t\t\tif (BPF_CLASS(insn-\u003ecode) == BPF_ALU)\n@@ -2066,18 +1980,18 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *\n \t\t\t\t\tEMIT2(0x48, 0x99); /* cqo */\n \n \t\t\t\t/* idiv src_reg */\n-\t\t\t\tmaybe_emit_1mod(\u0026prog, src_reg, is64);\n+\t\t\t\tmaybe_emit_1mod(jit, src_reg, is64);\n \t\t\t\tEMIT2(0xF7, add_1reg(0xF8, src_reg));\n \t\t\t}\n \n \t\t\tif (BPF_OP(insn-\u003ecode) == BPF_MOD \u0026\u0026\n \t\t\t    dst_reg != BPF_REG_3)\n \t\t\t\t/* mov dst_reg, rdx */\n-\t\t\t\temit_mov_reg(\u0026prog, is64, dst_reg, BPF_REG_3);\n+\t\t\t\temit_mov_reg(jit, is64, dst_reg, BPF_REG_3);\n \t\t\telse if (BPF_OP(insn-\u003ecode) == BPF_DIV \u0026\u0026\n \t\t\t\t dst_reg != BPF_REG_0)\n \t\t\t\t/* mov dst_reg, rax */\n-\t\t\t\temit_mov_reg(\u0026prog, is64, dst_reg, BPF_REG_0);\n+\t\t\t\temit_mov_reg(jit, is64, dst_reg, BPF_REG_0);\n \n \t\t\tif (dst_reg != BPF_REG_3)\n \t\t\t\tEMIT1(0x5A); /* pop rdx */\n@@ -2088,7 +2002,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *\n \n \t\tcase BPF_ALU | BPF_MUL | BPF_K:\n \t\tcase BPF_ALU64 | BPF_MUL | BPF_K:\n-\t\t\tmaybe_emit_mod(\u0026prog, dst_reg, dst_reg,\n+\t\t\tmaybe_emit_mod(jit, dst_reg, dst_reg,\n \t\t\t\t       BPF_CLASS(insn-\u003ecode) == BPF_ALU64);\n \n \t\t\tif (is_imm8(imm32))\n@@ -2104,7 +2018,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *\n \n \t\tcase BPF_ALU | BPF_MUL | BPF_X:\n \t\tcase BPF_ALU64 | BPF_MUL | BPF_X:\n-\t\t\tmaybe_emit_mod(\u0026prog, src_reg, dst_reg,\n+\t\t\tmaybe_emit_mod(jit, src_reg, dst_reg,\n \t\t\t\t       BPF_CLASS(insn-\u003ecode) == BPF_ALU64);\n \n \t\t\t/* imul dst_reg, src_reg */\n@@ -2118,7 +2032,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *\n \t\tcase BPF_ALU64 | BPF_LSH | BPF_K:\n \t\tcase BPF_ALU64 | BPF_RSH | BPF_K:\n \t\tcase BPF_ALU64 | BPF_ARSH | BPF_K:\n-\t\t\tmaybe_emit_1mod(\u0026prog, dst_reg,\n+\t\t\tmaybe_emit_1mod(jit, dst_reg,\n \t\t\t\t\tBPF_CLASS(insn-\u003ecode) == BPF_ALU64);\n \n \t\t\tb3 = simple_alu_opcodes[BPF_OP(insn-\u003ecode)];\n@@ -2152,7 +2066,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *\n \t\t\t\t\tbreak;\n \t\t\t\t}\n \n-\t\t\t\temit_shiftx(\u0026prog, dst_reg, src_reg, w, op);\n+\t\t\t\temit_shiftx(jit, dst_reg, src_reg, w, op);\n \n \t\t\t\tbreak;\n \t\t\t}\n@@ -2171,7 +2085,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *\n \t\t\t}\n \n \t\t\t/* shl %rax, %cl | shr %rax, %cl | sar %rax, %cl */\n-\t\t\tmaybe_emit_1mod(\u0026prog, dst_reg,\n+\t\t\tmaybe_emit_1mod(jit, dst_reg,\n \t\t\t\t\tBPF_CLASS(insn-\u003ecode) == BPF_ALU64);\n \n \t\t\tb3 = simple_alu_opcodes[BPF_OP(insn-\u003ecode)];\n@@ -2272,7 +2186,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *\n \t\tcase BPF_ST | BPF_MEM | BPF_DW:\n \t\t\tif (dst_reg == BPF_REG_PARAMS \u0026\u0026 insn-\u003eoff == -8) {\n \t\t\t\t/* Arg 6: store immediate in r9 register */\n-\t\t\t\temit_mov_imm64(\u0026prog, X86_REG_R9, imm32 \u003e\u003e 31, (u32)imm32);\n+\t\t\t\temit_mov_imm64(jit, X86_REG_R9, imm32 \u003e\u003e 31, (u32)imm32);\n \t\t\t\tbreak;\n \t\t\t}\n \t\t\tEMIT2(add_1mod(0x48, dst_reg), 0xC7);\n@@ -2310,15 +2224,15 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\t\t\tinsn_off = outgoing_arg_base - outgoing_rsp - insn_off - 16;\n \t\t\t\tdst_reg = BPF_REG_FP;\n \t\t\t}\n-\t\t\temit_stx(\u0026prog, BPF_SIZE(insn-\u003ecode), dst_reg, src_reg, insn_off);\n+\t\t\temit_stx(jit, BPF_SIZE(insn-\u003ecode), dst_reg, src_reg, insn_off);\n \t\t\tbreak;\n \n \t\tcase BPF_ST | BPF_PROBE_MEM32 | BPF_B:\n \t\tcase BPF_ST | BPF_PROBE_MEM32 | BPF_H:\n \t\tcase BPF_ST | BPF_PROBE_MEM32 | BPF_W:\n \t\tcase BPF_ST | BPF_PROBE_MEM32 | BPF_DW:\n-\t\t\tstart_of_ldx = prog;\n-\t\t\temit_st_r12(\u0026prog, BPF_SIZE(insn-\u003ecode), dst_reg, insn-\u003eoff, insn-\u003eimm);\n+\t\t\tstart_of_ldx = jit-\u003eprog;\n+\t\t\temit_st_r12(jit, BPF_SIZE(insn-\u003ecode), dst_reg, insn-\u003eoff, insn-\u003eimm);\n \t\t\tgoto populate_extable;\n \n \t\t\t/* LDX: dst_reg = *(u8*)(src_reg + r12 + off) */\n@@ -2333,14 +2247,14 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\tcase BPF_STX | BPF_PROBE_MEM32 | BPF_H:\n \t\tcase BPF_STX | BPF_PROBE_MEM32 | BPF_W:\n \t\tcase BPF_STX | BPF_PROBE_MEM32 | BPF_DW:\n-\t\t\tstart_of_ldx = prog;\n+\t\t\tstart_of_ldx = jit-\u003eprog;\n \t\t\tif (BPF_CLASS(insn-\u003ecode) == BPF_LDX) {\n \t\t\t\tif (BPF_MODE(insn-\u003ecode) == BPF_PROBE_MEM32SX)\n-\t\t\t\t\temit_ldsx_r12(\u0026prog, BPF_SIZE(insn-\u003ecode), dst_reg, src_reg, insn-\u003eoff);\n+\t\t\t\t\temit_ldsx_r12(jit, BPF_SIZE(insn-\u003ecode), dst_reg, src_reg, insn-\u003eoff);\n \t\t\t\telse\n-\t\t\t\t\temit_ldx_r12(\u0026prog, BPF_SIZE(insn-\u003ecode), dst_reg, src_reg, insn-\u003eoff);\n+\t\t\t\t\temit_ldx_r12(jit, BPF_SIZE(insn-\u003ecode), dst_reg, src_reg, insn-\u003eoff);\n \t\t\t} else {\n-\t\t\t\temit_stx_r12(\u0026prog, BPF_SIZE(insn-\u003ecode), dst_reg, src_reg, insn-\u003eoff);\n+\t\t\t\temit_stx_r12(jit, BPF_SIZE(insn-\u003ecode), dst_reg, src_reg, insn-\u003eoff);\n \t\t\t}\n populate_extable:\n \t\t\t{\n@@ -2401,7 +2315,7 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\t\t\t\tis_write = true;\n \t\t\t\t}\n \n-\t\t\t\tex-\u003efixup = FIELD_PREP(FIXUP_INSN_LEN_MASK, prog - start_of_ldx) |\n+\t\t\t\tex-\u003efixup = FIELD_PREP(FIXUP_INSN_LEN_MASK, jit-\u003eprog - start_of_ldx) |\n \t\t\t\t\t    FIELD_PREP(FIXUP_ARENA_REG_MASK, arena_reg) |\n \t\t\t\t\t    FIELD_PREP(FIXUP_REG_MASK, fixup_reg);\n \t\t\t\tex-\u003efixup |= FIXUP_ARENA_ACCESS;\n@@ -2455,7 +2369,7 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\t\t\tu8 *end_of_jmp;\n \n \t\t\t\t/* movabsq r10, VSYSCALL_ADDR */\n-\t\t\t\temit_mov_imm64(\u0026prog, BPF_REG_AX, (long)VSYSCALL_ADDR \u003e\u003e 32,\n+\t\t\t\temit_mov_imm64(jit, BPF_REG_AX, (long)VSYSCALL_ADDR \u003e\u003e 32,\n \t\t\t\t\t       (u32)(long)VSYSCALL_ADDR);\n \n \t\t\t\t/* mov src_reg, r11 */\n@@ -2463,40 +2377,40 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \n \t\t\t\tif (insn-\u003eoff) {\n \t\t\t\t\t/* add r11, insn-\u003eoff */\n-\t\t\t\t\tmaybe_emit_1mod(\u0026prog, AUX_REG, true);\n+\t\t\t\t\tmaybe_emit_1mod(jit, AUX_REG, true);\n \t\t\t\t\tEMIT2_off32(0x81, add_1reg(0xC0, AUX_REG), insn-\u003eoff);\n \t\t\t\t}\n \n \t\t\t\t/* sub r11, r10 */\n-\t\t\t\tmaybe_emit_mod(\u0026prog, AUX_REG, BPF_REG_AX, true);\n+\t\t\t\tmaybe_emit_mod(jit, AUX_REG, BPF_REG_AX, true);\n \t\t\t\tEMIT2(0x29, add_2reg(0xC0, AUX_REG, BPF_REG_AX));\n \n \t\t\t\t/* movabsq r10, limit */\n-\t\t\t\temit_mov_imm64(\u0026prog, BPF_REG_AX, (long)limit \u003e\u003e 32,\n+\t\t\t\temit_mov_imm64(jit, BPF_REG_AX, (long)limit \u003e\u003e 32,\n \t\t\t\t\t       (u32)(long)limit);\n \n \t\t\t\t/* cmp r10, r11 */\n-\t\t\t\tmaybe_emit_mod(\u0026prog, AUX_REG, BPF_REG_AX, true);\n+\t\t\t\tmaybe_emit_mod(jit, AUX_REG, BPF_REG_AX, true);\n \t\t\t\tEMIT2(0x39, add_2reg(0xC0, AUX_REG, BPF_REG_AX));\n \n \t\t\t\t/* if unsigned '\u003e', goto load */\n \t\t\t\tEMIT2(X86_JA, 0);\n-\t\t\t\tend_of_jmp = prog;\n+\t\t\t\tend_of_jmp = jit-\u003eprog;\n \n \t\t\t\t/* xor dst_reg, dst_reg */\n-\t\t\t\temit_mov_imm32(\u0026prog, false, dst_reg, 0);\n+\t\t\t\temit_mov_imm32(jit, false, dst_reg, 0);\n \t\t\t\t/* jmp byte_after_ldx */\n \t\t\t\tEMIT2(0xEB, 0);\n \n \t\t\t\t/* populate jmp_offset for JAE above to jump to start_of_ldx */\n-\t\t\t\tstart_of_ldx = prog;\n+\t\t\t\tstart_of_ldx = jit-\u003eprog;\n \t\t\t\tend_of_jmp[-1] = start_of_ldx - end_of_jmp;\n \t\t\t}\n \t\t\tif (BPF_MODE(insn-\u003ecode) == BPF_PROBE_MEMSX ||\n \t\t\t    BPF_MODE(insn-\u003ecode) == BPF_MEMSX)\n-\t\t\t\temit_ldsx(\u0026prog, BPF_SIZE(insn-\u003ecode), dst_reg, src_reg, insn_off);\n+\t\t\t\temit_ldsx(jit, BPF_SIZE(insn-\u003ecode), dst_reg, src_reg, insn_off);\n \t\t\telse\n-\t\t\t\temit_ldx(\u0026prog, BPF_SIZE(insn-\u003ecode), dst_reg, src_reg, insn_off);\n+\t\t\t\temit_ldx(jit, BPF_SIZE(insn-\u003ecode), dst_reg, src_reg, insn_off);\n \t\t\tif (BPF_MODE(insn-\u003ecode) == BPF_PROBE_MEM ||\n \t\t\t    BPF_MODE(insn-\u003ecode) == BPF_PROBE_MEMSX) {\n \t\t\t\tstruct exception_table_entry *ex;\n@@ -2504,7 +2418,7 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\t\t\ts64 delta;\n \n \t\t\t\t/* populate jmp_offset for JMP above */\n-\t\t\t\tstart_of_ldx[-1] = prog - start_of_ldx;\n+\t\t\t\tstart_of_ldx[-1] = jit-\u003eprog - start_of_ldx;\n \n \t\t\t\tif (!bpf_prog-\u003eaux-\u003eextable)\n \t\t\t\t\tbreak;\n@@ -2539,7 +2453,7 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\t\t\t * End result: x86 insn \"mov rbx, qword ptr [rax+0x14]\"\n \t\t\t\t * of 4 bytes will be ignored and rbx will be zero inited.\n \t\t\t\t */\n-\t\t\t\tex-\u003efixup = FIELD_PREP(FIXUP_INSN_LEN_MASK, prog - start_of_ldx) |\n+\t\t\t\tex-\u003efixup = FIELD_PREP(FIXUP_INSN_LEN_MASK, jit-\u003eprog - start_of_ldx) |\n \t\t\t\t\t    FIELD_PREP(FIXUP_REG_MASK, reg2pt_regs[dst_reg]);\n \t\t\t}\n \t\t\tbreak;\n@@ -2567,26 +2481,26 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\t\t\t */\n \n \t\t\t\t/* Will need RAX as a CMPXCHG operand so save R0 */\n-\t\t\t\temit_mov_reg(\u0026prog, true, BPF_REG_AX, BPF_REG_0);\n+\t\t\t\temit_mov_reg(jit, true, BPF_REG_AX, BPF_REG_0);\n \t\t\t\tif (src_reg == BPF_REG_0)\n \t\t\t\t\treal_src_reg = BPF_REG_AX;\n \t\t\t\tif (dst_reg == BPF_REG_0)\n \t\t\t\t\treal_dst_reg = BPF_REG_AX;\n \n-\t\t\t\tbranch_target = prog;\n+\t\t\t\tbranch_target = jit-\u003eprog;\n \t\t\t\t/* Load old value */\n-\t\t\t\temit_ldx(\u0026prog, BPF_SIZE(insn-\u003ecode),\n+\t\t\t\temit_ldx(jit, BPF_SIZE(insn-\u003ecode),\n \t\t\t\t\t BPF_REG_0, real_dst_reg, insn-\u003eoff);\n \t\t\t\t/*\n \t\t\t\t * Perform the (commutative) operation locally,\n \t\t\t\t * put the result in the AUX_REG.\n \t\t\t\t */\n-\t\t\t\temit_mov_reg(\u0026prog, is64, AUX_REG, BPF_REG_0);\n-\t\t\t\tmaybe_emit_mod(\u0026prog, AUX_REG, real_src_reg, is64);\n+\t\t\t\temit_mov_reg(jit, is64, AUX_REG, BPF_REG_0);\n+\t\t\t\tmaybe_emit_mod(jit, AUX_REG, real_src_reg, is64);\n \t\t\t\tEMIT2(simple_alu_opcodes[BPF_OP(insn-\u003eimm)],\n \t\t\t\t      add_2reg(0xC0, AUX_REG, real_src_reg));\n \t\t\t\t/* Attempt to swap in new value */\n-\t\t\t\terr = emit_atomic_rmw(\u0026prog, BPF_CMPXCHG,\n+\t\t\t\terr = emit_atomic_rmw(jit, BPF_CMPXCHG,\n \t\t\t\t\t\t      real_dst_reg, AUX_REG,\n \t\t\t\t\t\t      insn-\u003eoff,\n \t\t\t\t\t\t      BPF_SIZE(insn-\u003ecode));\n@@ -2596,19 +2510,19 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\t\t\t * ZF tells us whether we won the race. If it's\n \t\t\t\t * cleared we need to try again.\n \t\t\t\t */\n-\t\t\t\tEMIT2(X86_JNE, -(prog - branch_target) - 2);\n+\t\t\t\tEMIT2(X86_JNE, -(jit-\u003eprog - branch_target) - 2);\n \t\t\t\t/* Return the pre-modification value */\n-\t\t\t\temit_mov_reg(\u0026prog, is64, real_src_reg, BPF_REG_0);\n+\t\t\t\temit_mov_reg(jit, is64, real_src_reg, BPF_REG_0);\n \t\t\t\t/* Restore R0 after clobbering RAX */\n-\t\t\t\temit_mov_reg(\u0026prog, true, BPF_REG_0, BPF_REG_AX);\n+\t\t\t\temit_mov_reg(jit, true, BPF_REG_0, BPF_REG_AX);\n \t\t\t\tbreak;\n \t\t\t}\n \n \t\t\tif (bpf_atomic_is_load_store(insn))\n-\t\t\t\terr = emit_atomic_ld_st(\u0026prog, insn-\u003eimm, dst_reg, src_reg,\n+\t\t\t\terr = emit_atomic_ld_st(jit, insn-\u003eimm, dst_reg, src_reg,\n \t\t\t\t\t\t\tinsn-\u003eoff, BPF_SIZE(insn-\u003ecode));\n \t\t\telse\n-\t\t\t\terr = emit_atomic_rmw(\u0026prog, insn-\u003eimm, dst_reg, src_reg,\n+\t\t\t\terr = emit_atomic_rmw(jit, insn-\u003eimm, dst_reg, src_reg,\n \t\t\t\t\t\t      insn-\u003eoff, BPF_SIZE(insn-\u003ecode));\n \t\t\tif (err)\n \t\t\t\treturn err;\n@@ -2623,14 +2537,14 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\t\tfallthrough;\n \t\tcase BPF_STX | BPF_PROBE_ATOMIC | BPF_W:\n \t\tcase BPF_STX | BPF_PROBE_ATOMIC | BPF_DW:\n-\t\t\tstart_of_ldx = prog;\n+\t\t\tstart_of_ldx = jit-\u003eprog;\n \n \t\t\tif (bpf_atomic_is_load_store(insn))\n-\t\t\t\terr = emit_atomic_ld_st_index(\u0026prog, insn-\u003eimm,\n+\t\t\t\terr = emit_atomic_ld_st_index(jit, insn-\u003eimm,\n \t\t\t\t\t\t\t      BPF_SIZE(insn-\u003ecode), dst_reg,\n \t\t\t\t\t\t\t      src_reg, X86_REG_R12, insn-\u003eoff);\n \t\t\telse\n-\t\t\t\terr = emit_atomic_rmw_index(\u0026prog, insn-\u003eimm, BPF_SIZE(insn-\u003ecode),\n+\t\t\t\terr = emit_atomic_rmw_index(jit, insn-\u003eimm, BPF_SIZE(insn-\u003ecode),\n \t\t\t\t\t\t\t    dst_reg, src_reg, X86_REG_R12,\n \t\t\t\t\t\t\t    insn-\u003eoff);\n \t\t\tif (err)\n@@ -2652,20 +2566,20 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\t\t\tfm = bpf_jit_find_kfunc_model(bpf_prog, insn);\n \t\t\t\tif (!fm)\n \t\t\t\t\treturn -EINVAL;\n-\t\t\t\terr = emit_kfunc_arena_args(bpf_prog, fm, \u0026prog);\n+\t\t\t\terr = emit_kfunc_arena_args(bpf_prog, fm, jit);\n \t\t\t\tif (err \u003c 0)\n \t\t\t\t\treturn err;\n \t\t\t\tip += err;\n \t\t\t}\n \t\t\tif (priv_frame_ptr) {\n-\t\t\t\tpush_r9(\u0026prog);\n+\t\t\t\tpush_r9(jit);\n \t\t\t\tip += 2;\n \t\t\t}\n-\t\t\tip += x86_call_depth_emit_accounting(\u0026prog, func, ip);\n-\t\t\tif (emit_call(\u0026prog, func, ip))\n+\t\t\tip += bpf_call_depth_emit_accounting(jit, func, ip);\n+\t\t\tif (emit_call(jit, func, ip))\n \t\t\t\treturn -EINVAL;\n \t\t\tif (priv_frame_ptr)\n-\t\t\t\tpop_r9(\u0026prog);\n+\t\t\t\tpop_r9(jit);\n \t\t\t/*\n \t\t\t * A kfunc returning more than 8 bytes hands the second\n \t\t\t * half back in RDX (the native ABI's second return reg),\n@@ -2673,7 +2587,7 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\t\t * needed), while BPF R2 is RSI, so copy RDX into RSI.\n \t\t\t */\n \t\t\tif (fm \u0026\u0026 fm-\u003eret_size \u003e 8)\n-\t\t\t\temit_mov_reg(\u0026prog, true, BPF_REG_2, BPF_REG_3);\n+\t\t\t\temit_mov_reg(jit, true, BPF_REG_2, BPF_REG_3);\n \t\t\tbreak;\n \t\t}\n \n@@ -2681,14 +2595,12 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\t\tif (imm32)\n \t\t\t\temit_bpf_tail_call_direct(bpf_prog,\n \t\t\t\t\t\t\t  \u0026bpf_prog-\u003eaux-\u003epoke_tab[imm32 - 1],\n-\t\t\t\t\t\t\t  \u0026prog,\n \t\t\t\t\t\t\t  ip,\n \t\t\t\t\t\t\t  callee_regs_used,\n \t\t\t\t\t\t\t  stack_depth,\n \t\t\t\t\t\t\t  ctx);\n \t\t\telse\n \t\t\t\temit_bpf_tail_call_indirect(bpf_prog,\n-\t\t\t\t\t\t\t    \u0026prog,\n \t\t\t\t\t\t\t    callee_regs_used,\n \t\t\t\t\t\t\t    stack_depth,\n \t\t\t\t\t\t\t    ip,\n@@ -2717,7 +2629,7 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\tcase BPF_JMP32 | BPF_JSGE | BPF_X:\n \t\tcase BPF_JMP32 | BPF_JSLE | BPF_X:\n \t\t\t/* cmp dst_reg, src_reg */\n-\t\t\tmaybe_emit_mod(\u0026prog, dst_reg, src_reg,\n+\t\t\tmaybe_emit_mod(jit, dst_reg, src_reg,\n \t\t\t\t       BPF_CLASS(insn-\u003ecode) == BPF_JMP);\n \t\t\tEMIT2(0x39, add_2reg(0xC0, dst_reg, src_reg));\n \t\t\tgoto emit_cond_jmp;\n@@ -2725,7 +2637,7 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\tcase BPF_JMP | BPF_JSET | BPF_X:\n \t\tcase BPF_JMP32 | BPF_JSET | BPF_X:\n \t\t\t/* test dst_reg, src_reg */\n-\t\t\tmaybe_emit_mod(\u0026prog, dst_reg, src_reg,\n+\t\t\tmaybe_emit_mod(jit, dst_reg, src_reg,\n \t\t\t\t       BPF_CLASS(insn-\u003ecode) == BPF_JMP);\n \t\t\tEMIT2(0x85, add_2reg(0xC0, dst_reg, src_reg));\n \t\t\tgoto emit_cond_jmp;\n@@ -2733,7 +2645,7 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\tcase BPF_JMP | BPF_JSET | BPF_K:\n \t\tcase BPF_JMP32 | BPF_JSET | BPF_K:\n \t\t\t/* test dst_reg, imm32 */\n-\t\t\tmaybe_emit_1mod(\u0026prog, dst_reg,\n+\t\t\tmaybe_emit_1mod(jit, dst_reg,\n \t\t\t\t\tBPF_CLASS(insn-\u003ecode) == BPF_JMP);\n \t\t\tEMIT2_off32(0xF7, add_1reg(0xC0, dst_reg), imm32);\n \t\t\tgoto emit_cond_jmp;\n@@ -2760,14 +2672,14 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\tcase BPF_JMP32 | BPF_JSLE | BPF_K:\n \t\t\t/* test dst_reg, dst_reg to save one extra byte */\n \t\t\tif (imm32 == 0) {\n-\t\t\t\tmaybe_emit_mod(\u0026prog, dst_reg, dst_reg,\n+\t\t\t\tmaybe_emit_mod(jit, dst_reg, dst_reg,\n \t\t\t\t\t       BPF_CLASS(insn-\u003ecode) == BPF_JMP);\n \t\t\t\tEMIT2(0x85, add_2reg(0xC0, dst_reg, dst_reg));\n \t\t\t\tgoto emit_cond_jmp;\n \t\t\t}\n \n \t\t\t/* cmp dst_reg, imm8/32 */\n-\t\t\tmaybe_emit_1mod(\u0026prog, dst_reg,\n+\t\t\tmaybe_emit_1mod(jit, dst_reg,\n \t\t\t\t\tBPF_CLASS(insn-\u003ecode) == BPF_JMP);\n \n \t\t\tif (is_imm8(imm32))\n@@ -2843,7 +2755,7 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\t\t\t\t\t       nops);\n \t\t\t\t\t\treturn -EFAULT;\n \t\t\t\t\t}\n-\t\t\t\t\temit_nops(\u0026prog, nops);\n+\t\t\t\t\temit_nops(jit, nops);\n \t\t\t\t}\n \t\t\t\tEMIT2(jmp_cond, jmp_offset);\n \t\t\t} else if (is_simm32(jmp_offset)) {\n@@ -2856,7 +2768,7 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\t\tbreak;\n \n \t\tcase BPF_JMP | BPF_JA | BPF_X:\n-\t\t\temit_indirect_jump(\u0026prog, insn-\u003edst_reg, ip);\n+\t\t\temit_indirect_jump(jit, insn-\u003edst_reg, ip);\n \t\t\tbreak;\n \t\tcase BPF_JMP | BPF_JA:\n \t\tcase BPF_JMP32 | BPF_JA:\n@@ -2900,7 +2812,7 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\t\t\t\t\t       nops);\n \t\t\t\t\t\treturn -EFAULT;\n \t\t\t\t\t}\n-\t\t\t\t\temit_nops(\u0026prog, nops);\n+\t\t\t\t\temit_nops(jit, nops);\n \t\t\t\t}\n \t\t\t\tbreak;\n \t\t\t}\n@@ -2925,7 +2837,7 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\t\t\t\t\t       nops);\n \t\t\t\t\t\treturn -EFAULT;\n \t\t\t\t\t}\n-\t\t\t\t\temit_nops(\u0026prog, INSN_SZ_DIFF - 2);\n+\t\t\t\t\temit_nops(jit, INSN_SZ_DIFF - 2);\n \t\t\t\t}\n \t\t\t\tEMIT2(0xEB, jmp_offset);\n \t\t\t} else if (is_simm32(jmp_offset)) {\n@@ -2946,23 +2858,23 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\t\tctx-\u003ecleanup_addr = proglen;\n \t\t\tif (bpf_prog_was_classic(bpf_prog) \u0026\u0026\n \t\t\t    !ns_capable_noaudit(\u0026init_user_ns, CAP_SYS_ADMIN)) {\n-\t\t\t\tif (emit_spectre_bhb_barrier(\u0026prog, ip, bpf_prog))\n+\t\t\t\tif (emit_spectre_bhb_barrier(jit, ip, bpf_prog))\n \t\t\t\t\treturn -EINVAL;\n \t\t\t}\n \t\t\t/* Deallocate outgoing args 7+ area. */\n-\t\t\temit_add_rsp(\u0026prog, outgoing_rsp);\n+\t\t\temit_add_rsp(jit, outgoing_rsp);\n \t\t\tif (bpf_prog-\u003eaux-\u003eexception_boundary) {\n-\t\t\t\tpop_callee_regs(\u0026prog, all_callee_regs_used);\n-\t\t\t\tpop_r12(\u0026prog);\n+\t\t\t\tpop_callee_regs(jit, all_callee_regs_used);\n+\t\t\t\tpop_r12(jit);\n \t\t\t} else {\n-\t\t\t\tpop_callee_regs(\u0026prog, callee_regs_used);\n+\t\t\t\tpop_callee_regs(jit, callee_regs_used);\n \t\t\t\tif (arena_vm_start)\n-\t\t\t\t\tpop_r12(\u0026prog);\n+\t\t\t\t\tpop_r12(jit);\n \t\t\t}\n \t\t\tEMIT1(0xC9);         /* leave */\n-\t\t\tbpf_prog-\u003eaux-\u003eksym.fp_end = prog - temp;\n+\t\t\tbpf_prog-\u003eaux-\u003eksym.fp_end = jit-\u003eprog - temp;\n \n-\t\t\temit_return(\u0026prog, image + addrs[i - 1] + (prog - temp));\n+\t\t\temit_return(jit, image + addrs[i - 1] + (jit-\u003eprog - temp));\n \t\t\tbreak;\n \n \t\tdefault:\n@@ -2976,7 +2888,7 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\t\treturn -EINVAL;\n \t\t}\n \n-\t\tilen = prog - temp;\n+\t\tilen = jit-\u003eprog - temp;\n \t\tif (ilen \u003e BPF_MAX_INSN_SIZE) {\n \t\t\tpr_err(\"bpf_jit: fatal insn size error\\n\");\n \t\t\treturn -EFAULT;\n@@ -3000,7 +2912,7 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n \t\t}\n \t\tproglen += ilen;\n \t\taddrs[i] = proglen;\n-\t\tprog = temp;\n+\t\tjit-\u003eprog = temp;\n \t}\n \n \tif (image \u0026\u0026 excnt != bpf_prog-\u003eaux-\u003enum_exentries) {\n@@ -3011,11 +2923,10 @@ st:\t\t\tinsn_off = insn-\u003eoff;\n }\n \n static void clean_stack_garbage(const struct btf_func_model *m,\n-\t\t\t\tu8 **pprog, int nr_stack_slots,\n+\t\t\t\tstruct jit_emit_context *jit, int nr_stack_slots,\n \t\t\t\tint stack_size)\n {\n \tint arg_size, off;\n-\tu8 *prog;\n \n \t/* Generally speaking, the compiler will pass the arguments\n \t * on-stack with \"push\" instruction, which will take 8-byte\n@@ -3047,14 +2958,12 @@ static void clean_stack_garbage(const struct btf_func_model *m,\n \targ_size = m-\u003earg_size[m-\u003enr_args - 1];\n \tif (arg_size \u003c= 4) {\n \t\toff = -(stack_size - 4);\n-\t\tprog = *pprog;\n \t\t/* mov DWORD PTR [rbp + off], 0 */\n \t\tif (!is_imm8(off))\n \t\t\tEMIT2_off32(0xC7, 0x85, off);\n \t\telse\n \t\t\tEMIT3(0xC7, 0x45, off);\n \t\tEMIT(0, 4);\n-\t\t*pprog = prog;\n \t}\n }\n \n@@ -3082,26 +2991,23 @@ static int get_nr_used_regs(const struct btf_func_model *m)\n  * subtraction both truncates and clears the upper half, so the stored\n  * value satisfies the JIT invariant for arena pointer registers.\n  */\n-static void emit_arena_arg_conv(u8 **pprog, u32 src_reg, bool nullable, u32 base_lo)\n+static void emit_arena_arg_conv(struct jit_emit_context *jit, u32 src_reg, bool nullable,\n+\t\t\t\tu32 base_lo)\n {\n-\tu8 *prog = *pprog;\n-\n \tif (nullable) {\n \t\tif (src_reg != BPF_REG_0)\n-\t\t\temit_mov_reg(\u0026prog, true, BPF_REG_0, src_reg);\n+\t\t\temit_mov_reg(jit, true, BPF_REG_0, src_reg);\n \t\t/* test rax, rax; jz over the 5-byte sub */\n \t\tEMIT3(0x48, 0x85, 0xC0);\n \t\tEMIT2(X86_JE, 5);\n \t} else if (src_reg != BPF_REG_0) {\n-\t\temit_mov_reg(\u0026prog, false, BPF_REG_0, src_reg);\n+\t\temit_mov_reg(jit, false, BPF_REG_0, src_reg);\n \t}\n \t/* sub eax, base_lo */\n \tEMIT1_off32(0x2D, base_lo);\n-\n-\t*pprog = prog;\n }\n \n-static void save_args(const struct btf_func_model *m, u8 **prog,\n+static void save_args(const struct btf_func_model *m, struct jit_emit_context *jit,\n \t\t      int stack_size, bool for_call_origin, u32 flags,\n \t\t      u64 arena_base)\n {\n@@ -3150,12 +3056,12 @@ static void save_args(const struct btf_func_model *m, u8 **prog,\n \t\t\t * called indirectly, so rbp + 16.\n \t\t\t */\n \t\t\tfor (j = 0; j \u003c arg_regs; j++) {\n-\t\t\t\temit_ldx(prog, BPF_DW, BPF_REG_0, BPF_REG_FP,\n+\t\t\t\temit_ldx(jit, BPF_DW, BPF_REG_0, BPF_REG_FP,\n \t\t\t\t\t nr_stack_slots * 8 + stack_args_off);\n \t\t\t\tif (arena_arg)\n-\t\t\t\t\temit_arena_arg_conv(prog, BPF_REG_0, nullable,\n+\t\t\t\t\temit_arena_arg_conv(jit, BPF_REG_0, nullable,\n \t\t\t\t\t\t\t    (u32)arena_base);\n-\t\t\t\temit_stx(prog, BPF_DW, BPF_REG_FP, BPF_REG_0,\n+\t\t\t\temit_stx(jit, BPF_DW, BPF_REG_FP, BPF_REG_0,\n \t\t\t\t\t -stack_size);\n \n \t\t\t\tif (!nr_stack_slots)\n@@ -3178,20 +3084,20 @@ static void save_args(const struct btf_func_model *m, u8 **prog,\n \t\t\t\tu32 src = nr_regs == 5 ? X86_REG_R9 : BPF_REG_1 + nr_regs;\n \n \t\t\t\tif (arena_arg) {\n-\t\t\t\t\temit_arena_arg_conv(prog, src, nullable, (u32)arena_base);\n+\t\t\t\t\temit_arena_arg_conv(jit, src, nullable, (u32)arena_base);\n \t\t\t\t\tsrc = BPF_REG_0;\n \t\t\t\t}\n-\t\t\t\temit_stx(prog, BPF_DW, BPF_REG_FP, src, -stack_size);\n+\t\t\t\temit_stx(jit, BPF_DW, BPF_REG_FP, src, -stack_size);\n \t\t\t\tstack_size -= 8;\n \t\t\t\tnr_regs++;\n \t\t\t}\n \t\t}\n \t}\n \n-\tclean_stack_garbage(m, prog, nr_stack_slots, first_off);\n+\tclean_stack_garbage(m, jit, nr_stack_slots, first_off);\n }\n \n-static void restore_regs(const struct btf_func_model *m, u8 **prog,\n+static void restore_regs(const struct btf_func_model *m, struct jit_emit_context *jit,\n \t\t\t int stack_size)\n {\n \tint i, j, arg_regs, nr_regs = 0;\n@@ -3207,7 +3113,7 @@ static void restore_regs(const struct btf_func_model *m, u8 **prog,\n \t\targ_regs = (m-\u003earg_size[i] + 7) / 8;\n \t\tif (nr_regs + arg_regs \u003c= 6) {\n \t\t\tfor (j = 0; j \u003c arg_regs; j++) {\n-\t\t\t\temit_ldx(prog, BPF_DW,\n+\t\t\t\temit_ldx(jit, BPF_DW,\n \t\t\t\t\t nr_regs == 5 ? X86_REG_R9 : BPF_REG_1 + nr_regs,\n \t\t\t\t\t BPF_REG_FP,\n \t\t\t\t\t -stack_size);\n@@ -3223,19 +3129,18 @@ static void restore_regs(const struct btf_func_model *m, u8 **prog,\n \t}\n }\n \n-static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog,\n+static int invoke_bpf_prog(const struct btf_func_model *m, struct jit_emit_context *jit,\n \t\t\t   struct bpf_tramp_node *node, int stack_size,\n \t\t\t   int run_ctx_off, bool save_ret,\n \t\t\t   void *image, void *rw_image)\n {\n-\tu8 *prog = *pprog;\n \tu8 *jmp_insn;\n \tint ctx_cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie);\n \tstruct bpf_prog *p = node-\u003elink-\u003eprog;\n \tu64 cookie = node-\u003ecookie;\n \n \t/* mov rdi, cookie */\n-\temit_mov_imm64(\u0026prog, BPF_REG_1, (long) cookie \u003e\u003e 32, (u32) (long) cookie);\n+\temit_mov_imm64(jit, BPF_REG_1, (long) cookie \u003e\u003e 32, (u32) (long) cookie);\n \n \t/* Prepare struct bpf_tramp_run_ctx.\n \t *\n@@ -3244,28 +3149,28 @@ static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog,\n \t *\n \t * mov QWORD PTR [rbp - run_ctx_off + ctx_cookie_off], rdi\n \t */\n-\temit_stx(\u0026prog, BPF_DW, BPF_REG_FP, BPF_REG_1, -run_ctx_off + ctx_cookie_off);\n+\temit_stx(jit, BPF_DW, BPF_REG_FP, BPF_REG_1, -run_ctx_off + ctx_cookie_off);\n \n \t/* arg1: mov rdi, progs[i] */\n-\temit_mov_imm64(\u0026prog, BPF_REG_1, (long) p \u003e\u003e 32, (u32) (long) p);\n+\temit_mov_imm64(jit, BPF_REG_1, (long) p \u003e\u003e 32, (u32) (long) p);\n \t/* arg2: lea rsi, [rbp - ctx_cookie_off] */\n \tif (!is_imm8(-run_ctx_off))\n \t\tEMIT3_off32(0x48, 0x8D, 0xB5, -run_ctx_off);\n \telse\n \t\tEMIT4(0x48, 0x8D, 0x75, -run_ctx_off);\n \n-\tif (emit_rsb_call(\u0026prog, bpf_trampoline_enter(p), image + (prog - (u8 *)rw_image)))\n+\tif (emit_rsb_call(jit, bpf_trampoline_enter(p), image + (jit-\u003eprog - (u8 *)rw_image)))\n \t\treturn -EINVAL;\n \t/* remember prog start time returned by __bpf_prog_enter */\n-\temit_mov_reg(\u0026prog, true, BPF_REG_6, BPF_REG_0);\n+\temit_mov_reg(jit, true, BPF_REG_6, BPF_REG_0);\n \n \t/* if (__bpf_prog_enter*(prog) == 0)\n \t *\tgoto skip_exec_of_prog;\n \t */\n \tEMIT3(0x48, 0x85, 0xC0);  /* test rax,rax */\n \t/* emit 2 nops that will be replaced with JE insn */\n-\tjmp_insn = prog;\n-\temit_nops(\u0026prog, 2);\n+\tjmp_insn = jit-\u003eprog;\n+\temit_nops(jit, 2);\n \n \t/* arg1: lea rdi, [rbp - stack_size] */\n \tif (!is_imm8(-stack_size))\n@@ -3274,11 +3179,11 @@ static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog,\n \t\tEMIT4(0x48, 0x8D, 0x7D, -stack_size);\n \t/* arg2: progs[i]-\u003einsnsi for interpreter */\n \tif (!p-\u003ejited)\n-\t\temit_mov_imm64(\u0026prog, BPF_REG_2,\n+\t\temit_mov_imm64(jit, BPF_REG_2,\n \t\t\t       (long) p-\u003einsnsi \u003e\u003e 32,\n \t\t\t       (u32) (long) p-\u003einsnsi);\n \t/* call JITed bpf program or interpreter */\n-\tif (emit_rsb_call(\u0026prog, p-\u003ebpf_func, image + (prog - (u8 *)rw_image)))\n+\tif (emit_rsb_call(jit, p-\u003ebpf_func, image + (jit-\u003eprog - (u8 *)rw_image)))\n \t\treturn -EINVAL;\n \n \t/*\n@@ -3290,42 +3195,39 @@ static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog,\n \t * value of BPF_PROG_TYPE_STRUCT_OPS prog.\n \t */\n \tif (save_ret)\n-\t\temit_stx(\u0026prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8);\n+\t\temit_stx(jit, BPF_DW, BPF_REG_FP, BPF_REG_0, -8);\n \n \t/* replace 2 nops with JE insn, since jmp target is known */\n-\tjmp_insn[0] = X86_JE;\n-\tjmp_insn[1] = prog - jmp_insn - 2;\n+\tif (!jit-\u003edry_run) {\n+\t\tjmp_insn[0] = X86_JE;\n+\t\tjmp_insn[1] = jit-\u003eprog - jmp_insn - 2;\n+\t}\n \n \t/* arg1: mov rdi, progs[i] */\n-\temit_mov_imm64(\u0026prog, BPF_REG_1, (long) p \u003e\u003e 32, (u32) (long) p);\n+\temit_mov_imm64(jit, BPF_REG_1, (long) p \u003e\u003e 32, (u32) (long) p);\n \t/* arg2: mov rsi, rbx \u003c- start time in nsec */\n-\temit_mov_reg(\u0026prog, true, BPF_REG_2, BPF_REG_6);\n+\temit_mov_reg(jit, true, BPF_REG_2, BPF_REG_6);\n \t/* arg3: lea rdx, [rbp - run_ctx_off] */\n \tif (!is_imm8(-run_ctx_off))\n \t\tEMIT3_off32(0x48, 0x8D, 0x95, -run_ctx_off);\n \telse\n \t\tEMIT4(0x48, 0x8D, 0x55, -run_ctx_off);\n-\tif (emit_rsb_call(\u0026prog, bpf_trampoline_exit(p), image + (prog - (u8 *)rw_image)))\n+\tif (emit_rsb_call(jit, bpf_trampoline_exit(p), image + (jit-\u003eprog - (u8 *)rw_image)))\n \t\treturn -EINVAL;\n \n-\t*pprog = prog;\n \treturn 0;\n }\n \n-static void emit_align(u8 **pprog, u32 align)\n+static void emit_align(struct jit_emit_context *jit, u32 align)\n {\n-\tu8 *target, *prog = *pprog;\n-\n-\ttarget = PTR_ALIGN(prog, align);\n-\tif (target != prog)\n-\t\temit_nops(\u0026prog, target - prog);\n+\tu8 *target = PTR_ALIGN(jit-\u003eprog, align);\n \n-\t*pprog = prog;\n+\tif (target != jit-\u003eprog)\n+\t\temit_nops(jit, target - jit-\u003eprog);\n }\n \n-static int emit_cond_near_jump(u8 **pprog, void *func, void *ip, u8 jmp_cond)\n+static int emit_cond_near_jump(struct jit_emit_context *jit, void *func, void *ip, u8 jmp_cond)\n {\n-\tu8 *prog = *pprog;\n \ts64 offset;\n \n \toffset = func - (ip + 2 + 4);\n@@ -3334,48 +3236,44 @@ static int emit_cond_near_jump(u8 **pprog, void *func, void *ip, u8 jmp_cond)\n \t\treturn -EINVAL;\n \t}\n \tEMIT2_off32(0x0F, jmp_cond + 0x10, offset);\n-\t*pprog = prog;\n \treturn 0;\n }\n \n-static int invoke_bpf(const struct btf_func_model *m, u8 **pprog,\n+static int invoke_bpf(const struct btf_func_model *m, struct jit_emit_context *jit,\n \t\t      struct bpf_tramp_nodes *tl, int stack_size,\n \t\t      int run_ctx_off, int func_meta_off, bool save_ret,\n \t\t      void *image, void *rw_image, u64 func_meta,\n \t\t      int cookie_off)\n {\n \tint i, cur_cookie = (cookie_off - stack_size) / 8;\n-\tu8 *prog = *pprog;\n \n \tfor (i = 0; i \u003c tl-\u003enr_nodes; i++) {\n \t\tif (tl-\u003enodes[i]-\u003elink-\u003eprog-\u003ecall_session_cookie) {\n-\t\t\temit_store_stack_imm64(\u0026prog, BPF_REG_0, -func_meta_off,\n+\t\t\temit_store_stack_imm64(jit, BPF_REG_0, -func_meta_off,\n \t\t\t\tfunc_meta | (cur_cookie \u003c\u003c BPF_TRAMP_COOKIE_INDEX_SHIFT));\n \t\t\tcur_cookie--;\n \t\t}\n-\t\tif (invoke_bpf_prog(m, \u0026prog, tl-\u003enodes[i], stack_size,\n+\t\tif (invoke_bpf_prog(m, jit, tl-\u003enodes[i], stack_size,\n \t\t\t\t    run_ctx_off, save_ret, image, rw_image))\n \t\t\treturn -EINVAL;\n \t}\n-\t*pprog = prog;\n \treturn 0;\n }\n \n-static int invoke_bpf_mod_ret(const struct btf_func_model *m, u8 **pprog,\n+static int invoke_bpf_mod_ret(const struct btf_func_model *m, struct jit_emit_context *jit,\n \t\t\t      struct bpf_tramp_nodes *tl, int stack_size,\n \t\t\t      int run_ctx_off, u8 **branches,\n \t\t\t      void *image, void *rw_image)\n {\n-\tu8 *prog = *pprog;\n \tint i;\n \n \t/* The first fmod_ret program will receive a garbage return value.\n \t * Set this to 0 to avoid confusing the program.\n \t */\n-\temit_mov_imm32(\u0026prog, false, BPF_REG_0, 0);\n-\temit_stx(\u0026prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8);\n+\temit_mov_imm32(jit, false, BPF_REG_0, 0);\n+\temit_stx(jit, BPF_DW, BPF_REG_FP, BPF_REG_0, -8);\n \tfor (i = 0; i \u003c tl-\u003enr_nodes; i++) {\n-\t\tif (invoke_bpf_prog(m, \u0026prog, tl-\u003enodes[i], stack_size, run_ctx_off, true,\n+\t\tif (invoke_bpf_prog(m, jit, tl-\u003enodes[i], stack_size, run_ctx_off, true,\n \t\t\t\t    image, rw_image))\n \t\t\treturn -EINVAL;\n \n@@ -3391,11 +3289,10 @@ static int invoke_bpf_mod_ret(const struct btf_func_model *m, u8 **pprog,\n \t\t * are replaced with a conditional jump once do_fexit (i.e. the\n \t\t * start of the fexit invocation) is finalized.\n \t\t */\n-\t\tbranches[i] = prog;\n-\t\temit_nops(\u0026prog, 4 + 2);\n+\t\tbranches[i] = jit-\u003eprog;\n+\t\temit_nops(jit, 4 + 2);\n \t}\n \n-\t*pprog = prog;\n \treturn 0;\n }\n \n@@ -3474,12 +3371,12 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im\n \tstruct bpf_tramp_nodes *fentry = \u0026tnodes[BPF_TRAMP_FENTRY];\n \tstruct bpf_tramp_nodes *fexit = \u0026tnodes[BPF_TRAMP_FEXIT];\n \tstruct bpf_tramp_nodes *fmod_ret = \u0026tnodes[BPF_TRAMP_MODIFY_RETURN];\n+\tstruct jit_emit_context jit_ctx = {}, *jit = \u0026jit_ctx;\n \tvoid *orig_call = func_addr;\n \tint cookie_off, cookie_cnt;\n \tu8 **branches = NULL;\n \tu64 arena_base;\n \tu64 func_meta;\n-\tu8 *prog;\n \tbool save_ret;\n \n \t/*\n@@ -3584,13 +3481,14 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im\n \t\torig_call += X86_PATCH_SIZE;\n \t}\n \n-\tprog = rw_image;\n+\tjit-\u003eprog = rw_image;\n+\tjit-\u003edry_run = !rw_image;\n \n \tif (flags \u0026 BPF_TRAMP_F_INDIRECT) {\n \t\t/*\n \t\t * Indirect call for bpf_struct_ops\n \t\t */\n-\t\temit_cfi(\u0026prog, image,\n+\t\temit_cfi(jit, image,\n \t\t\t cfi_get_func_hash(func_addr),\n \t\t\t cfi_get_func_arity(func_addr));\n \t} else {\n@@ -3598,12 +3496,12 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im\n \t\t * Direct-call fentry stub, as such it needs accounting for the\n \t\t * __fentry__ call.\n \t\t */\n-\t\tx86_call_depth_emit_accounting(\u0026prog, NULL, image);\n+\t\tbpf_call_depth_emit_accounting(jit, NULL, image);\n \t}\n \tEMIT1(0x55);\t\t /* push rbp */\n \tEMIT3(0x48, 0x89, 0xE5); /* mov rbp, rsp */\n \tif (im)\n-\t\tim-\u003eksym.fp_start = prog - (u8 *)rw_image;\n+\t\tim-\u003eksym.fp_start = jit-\u003eprog - (u8 *)rw_image;\n \n \tif (!is_imm8(stack_size)) {\n \t\t/* sub rsp, stack_size */\n@@ -3615,24 +3513,24 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im\n \tif (flags \u0026 BPF_TRAMP_F_TAIL_CALL_CTX)\n \t\tEMIT1(0x50);\t\t/* push rax */\n \t/* mov QWORD PTR [rbp - rbx_off], rbx */\n-\temit_stx(\u0026prog, BPF_DW, BPF_REG_FP, BPF_REG_6, -rbx_off);\n+\temit_stx(jit, BPF_DW, BPF_REG_FP, BPF_REG_6, -rbx_off);\n \n \tfunc_meta = nr_regs;\n \t/* Store number of argument registers of the traced function */\n-\temit_store_stack_imm64(\u0026prog, BPF_REG_0, -func_meta_off, func_meta);\n+\temit_store_stack_imm64(jit, BPF_REG_0, -func_meta_off, func_meta);\n \n \tif (flags \u0026 BPF_TRAMP_F_IP_ARG) {\n \t\t/* Store IP address of the traced function */\n-\t\temit_store_stack_imm64(\u0026prog, BPF_REG_0, -ip_off, (long)func_addr);\n+\t\temit_store_stack_imm64(jit, BPF_REG_0, -ip_off, (long)func_addr);\n \t}\n \n-\tsave_args(m, \u0026prog, regs_off, false, flags, arena_base);\n+\tsave_args(m, jit, regs_off, false, flags, arena_base);\n \n \tif (flags \u0026 BPF_TRAMP_F_CALL_ORIG) {\n \t\t/* arg1: mov rdi, im */\n-\t\temit_mov_imm64(\u0026prog, BPF_REG_1, (long) im \u003e\u003e 32, (u32) (long) im);\n-\t\tif (emit_rsb_call(\u0026prog, __bpf_tramp_enter,\n-\t\t\t\t  image + (prog - (u8 *)rw_image))) {\n+\t\temit_mov_imm64(jit, BPF_REG_1, (long) im \u003e\u003e 32, (u32) (long) im);\n+\t\tif (emit_rsb_call(jit, __bpf_tramp_enter,\n+\t\t\t\t  image + (jit-\u003eprog - (u8 *)rw_image))) {\n \t\t\tret = -EINVAL;\n \t\t\tgoto cleanup;\n \t\t}\n@@ -3641,13 +3539,13 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im\n \tif (bpf_fsession_cnt(tnodes)) {\n \t\t/* clear all the session cookies' value */\n \t\tfor (int i = 0; i \u003c cookie_cnt; i++)\n-\t\t\temit_store_stack_imm64(\u0026prog, BPF_REG_0, -cookie_off + 8 * i, 0);\n+\t\t\temit_store_stack_imm64(jit, BPF_REG_0, -cookie_off + 8 * i, 0);\n \t\t/* clear the return value to make sure fentry always get 0 */\n-\t\temit_store_stack_imm64(\u0026prog, BPF_REG_0, -8, 0);\n+\t\temit_store_stack_imm64(jit, BPF_REG_0, -8, 0);\n \t}\n \n \tif (fentry-\u003enr_nodes) {\n-\t\tif (invoke_bpf(m, \u0026prog, fentry, regs_off, run_ctx_off, func_meta_off,\n+\t\tif (invoke_bpf(m, jit, fentry, regs_off, run_ctx_off, func_meta_off,\n \t\t\t       flags \u0026 BPF_TRAMP_F_RET_FENTRY_RET, image, rw_image,\n \t\t\t       func_meta, cookie_off))\n \t\t\treturn -EINVAL;\n@@ -3659,7 +3557,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im\n \t\tif (!branches)\n \t\t\treturn -ENOMEM;\n \n-\t\tif (invoke_bpf_mod_ret(m, \u0026prog, fmod_ret, regs_off,\n+\t\tif (invoke_bpf_mod_ret(m, jit, fmod_ret, regs_off,\n \t\t\t\t       run_ctx_off, branches, image, rw_image)) {\n \t\t\tret = -EINVAL;\n \t\t\tgoto cleanup;\n@@ -3667,8 +3565,8 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im\n \t}\n \n \tif (flags \u0026 BPF_TRAMP_F_CALL_ORIG) {\n-\t\trestore_regs(m, \u0026prog, regs_off);\n-\t\tsave_args(m, \u0026prog, arg_stack_off, true, flags, 0);\n+\t\trestore_regs(m, jit, regs_off);\n+\t\tsave_args(m, jit, arg_stack_off, true, flags, 0);\n \n \t\tif (flags \u0026 BPF_TRAMP_F_TAIL_CALL_CTX) {\n \t\t\t/* Before calling the original function, load the\n@@ -3678,19 +3576,19 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im\n \t\t}\n \n \t\tif (flags \u0026 BPF_TRAMP_F_ORIG_STACK) {\n-\t\t\temit_ldx(\u0026prog, BPF_DW, BPF_REG_6, BPF_REG_FP, 8);\n+\t\t\temit_ldx(jit, BPF_DW, BPF_REG_6, BPF_REG_FP, 8);\n \t\t\tEMIT2(0xff, 0xd3); /* call *rbx */\n \t\t} else {\n \t\t\t/* call original function */\n-\t\t\tif (emit_rsb_call(\u0026prog, orig_call, image + (prog - (u8 *)rw_image))) {\n+\t\t\tif (emit_rsb_call(jit, orig_call, image + (jit-\u003eprog - (u8 *)rw_image))) {\n \t\t\t\tret = -EINVAL;\n \t\t\t\tgoto cleanup;\n \t\t\t}\n \t\t}\n \t\t/* remember return value in a stack for bpf prog to access */\n-\t\temit_stx(\u0026prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8);\n-\t\tim-\u003eip_after_call = image + (prog - (u8 *)rw_image);\n-\t\temit_nops(\u0026prog, X86_PATCH_SIZE);\n+\t\temit_stx(jit, BPF_DW, BPF_REG_FP, BPF_REG_0, -8);\n+\t\tim-\u003eip_after_call = image + (jit-\u003eprog - (u8 *)rw_image);\n+\t\temit_nops(jit, X86_PATCH_SIZE);\n \t}\n \n \tif (fmod_ret-\u003enr_nodes) {\n@@ -3699,12 +3597,17 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im\n \t\t * Coding Rule 11: All branch targets should be 16-byte\n \t\t * aligned.\n \t\t */\n-\t\temit_align(\u0026prog, 16);\n+\t\temit_align(jit, 16);\n \t\t/* Update the branches saved in invoke_bpf_mod_ret with the\n \t\t * aligned address of do_fexit.\n \t\t */\n \t\tfor (i = 0; i \u003c fmod_ret-\u003enr_nodes; i++) {\n-\t\t\temit_cond_near_jump(\u0026branches[i], image + (prog - (u8 *)rw_image),\n+\t\t\tstruct jit_emit_context branch_jit = {\n+\t\t\t\t.prog = branches[i],\n+\t\t\t\t.dry_run = jit-\u003edry_run,\n+\t\t\t};\n+\n+\t\t\temit_cond_near_jump(\u0026branch_jit, image + (jit-\u003eprog - (u8 *)rw_image),\n \t\t\t\t\t    image + (branches[i] - (u8 *)rw_image), X86_JNE);\n \t\t}\n \t}\n@@ -3712,10 +3615,10 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im\n \t/* set the \"is_return\" flag for fsession */\n \tfunc_meta |= (1ULL \u003c\u003c BPF_TRAMP_IS_RETURN_SHIFT);\n \tif (bpf_fsession_cnt(tnodes))\n-\t\temit_store_stack_imm64(\u0026prog, BPF_REG_0, -func_meta_off, func_meta);\n+\t\temit_store_stack_imm64(jit, BPF_REG_0, -func_meta_off, func_meta);\n \n \tif (fexit-\u003enr_nodes) {\n-\t\tif (invoke_bpf(m, \u0026prog, fexit, regs_off, run_ctx_off, func_meta_off,\n+\t\tif (invoke_bpf(m, jit, fexit, regs_off, run_ctx_off, func_meta_off,\n \t\t\t       false, image, rw_image, func_meta, cookie_off)) {\n \t\t\tret = -EINVAL;\n \t\t\tgoto cleanup;\n@@ -3723,17 +3626,17 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im\n \t}\n \n \tif (flags \u0026 BPF_TRAMP_F_RESTORE_REGS)\n-\t\trestore_regs(m, \u0026prog, regs_off);\n+\t\trestore_regs(m, jit, regs_off);\n \n \t/* This needs to be done regardless. If there were fmod_ret programs,\n \t * the return value is only updated on the stack and still needs to be\n \t * restored to R0.\n \t */\n \tif (flags \u0026 BPF_TRAMP_F_CALL_ORIG) {\n-\t\tim-\u003eip_epilogue = image + (prog - (u8 *)rw_image);\n+\t\tim-\u003eip_epilogue = image + (jit-\u003eprog - (u8 *)rw_image);\n \t\t/* arg1: mov rdi, im */\n-\t\temit_mov_imm64(\u0026prog, BPF_REG_1, (long) im \u003e\u003e 32, (u32) (long) im);\n-\t\tif (emit_rsb_call(\u0026prog, __bpf_tramp_exit, image + (prog - (u8 *)rw_image))) {\n+\t\temit_mov_imm64(jit, BPF_REG_1, (long) im \u003e\u003e 32, (u32) (long) im);\n+\t\tif (emit_rsb_call(jit, __bpf_tramp_exit, image + (jit-\u003eprog - (u8 *)rw_image))) {\n \t\t\tret = -EINVAL;\n \t\t\tgoto cleanup;\n \t\t}\n@@ -3746,25 +3649,26 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im\n \n \t/* restore return value of orig_call or fentry prog back into RAX */\n \tif (save_ret)\n-\t\temit_ldx(\u0026prog, BPF_DW, BPF_REG_0, BPF_REG_FP, -8);\n+\t\temit_ldx(jit, BPF_DW, BPF_REG_0, BPF_REG_FP, -8);\n \n-\temit_ldx(\u0026prog, BPF_DW, BPF_REG_6, BPF_REG_FP, -rbx_off);\n+\temit_ldx(jit, BPF_DW, BPF_REG_6, BPF_REG_FP, -rbx_off);\n \n \tEMIT1(0xC9); /* leave */\n \tif (im)\n-\t\tim-\u003eksym.fp_end = prog - (u8 *)rw_image;\n+\t\tim-\u003eksym.fp_end = jit-\u003eprog - (u8 *)rw_image;\n \n \tif (flags \u0026 BPF_TRAMP_F_SKIP_FRAME) {\n \t\t/* skip our return address and return to parent */\n \t\tEMIT4(0x48, 0x83, 0xC4, 8); /* add rsp, 8 */\n \t}\n-\temit_return(\u0026prog, image + (prog - (u8 *)rw_image));\n+\temit_return(jit, image + (jit-\u003eprog - (u8 *)rw_image));\n \t/* Make sure the trampoline generation logic doesn't overflow */\n-\tif (WARN_ON_ONCE(prog \u003e (u8 *)rw_image_end - BPF_INSN_SAFETY)) {\n+\tif (!jit-\u003edry_run \u0026\u0026\n+\t    WARN_ON_ONCE(jit-\u003eprog \u003e (u8 *)rw_image_end - BPF_INSN_SAFETY)) {\n \t\tret = -EFAULT;\n \t\tgoto cleanup;\n \t}\n-\tret = prog - (u8 *)rw_image + BPF_INSN_SAFETY;\n+\tret = jit-\u003eprog - (u8 *)rw_image + BPF_INSN_SAFETY;\n \n cleanup:\n \tkfree(branches);\n@@ -3819,28 +3723,15 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,\n \t\t\t     struct bpf_tramp_nodes *tnodes, void *func_addr)\n {\n \tstruct bpf_tramp_image im;\n-\tvoid *image;\n-\tint ret;\n \n-\t/* Allocate a temporary buffer for __arch_prepare_bpf_trampoline().\n-\t *\n-\t * We cannot use kvmalloc here, because we need image to be in\n-\t * module memory range.\n-\t * Since it must be writable use bpf_jit_alloc_exec_rw().\n-\t */\n-\timage = bpf_jit_alloc_exec_rw(PAGE_SIZE);\n-\tif (!image)\n-\t\treturn -ENOMEM;\n-\n-\tret = __arch_prepare_bpf_trampoline(\u0026im, image, image + PAGE_SIZE, image,\n-\t\t\t\t\t    m, flags, tnodes, func_addr);\n-\tbpf_jit_free_exec(image);\n-\treturn ret;\n+\treturn __arch_prepare_bpf_trampoline(\u0026im, NULL, NULL, NULL, m, flags,\n+\t\t\t\t\t     tnodes, func_addr);\n }\n \n-static int emit_bpf_dispatcher(u8 **pprog, int a, int b, s64 *progs, u8 *image, u8 *buf)\n+static int emit_bpf_dispatcher(struct jit_emit_context *jit, int a, int b, s64 *progs, u8 *image,\n+\t\t\t       u8 *buf)\n {\n-\tu8 *jg_reloc, *prog = *pprog;\n+\tu8 *jg_reloc;\n \tint pivot, err, jg_bytes = 1;\n \ts64 jg_offset;\n \n@@ -3853,15 +3744,13 @@ static int emit_bpf_dispatcher(u8 **pprog, int a, int b, s64 *progs, u8 *image,\n \t\t\treturn -1;\n \t\tEMIT2_off32(0x81, add_1reg(0xF8, BPF_REG_3),\n \t\t\t    progs[a]);\n-\t\terr = emit_cond_near_jump(\u0026prog,\t/* je func */\n-\t\t\t\t\t  (void *)progs[a], image + (prog - buf),\n+\t\terr = emit_cond_near_jump(jit,\t/* je func */\n+\t\t\t\t\t  (void *)progs[a], image + (jit-\u003eprog - buf),\n \t\t\t\t\t  X86_JE);\n \t\tif (err)\n \t\t\treturn err;\n \n-\t\temit_indirect_jump(\u0026prog, BPF_REG_3 /* R3 -\u003e rdx */, image + (prog - buf));\n-\n-\t\t*pprog = prog;\n+\t\temit_indirect_jump(jit, BPF_REG_3 /* R3 -\u003e rdx */, image + (jit-\u003eprog - buf));\n \t\treturn 0;\n \t}\n \n@@ -3881,9 +3770,9 @@ static int emit_bpf_dispatcher(u8 **pprog, int a, int b, s64 *progs, u8 *image,\n \t} else {\n \t\tEMIT2(X86_JG, 0);\n \t}\n-\tjg_reloc = prog;\n+\tjg_reloc = jit-\u003eprog;\n \n-\terr = emit_bpf_dispatcher(\u0026prog, a, a + pivot,\t/* emit lower_part */\n+\terr = emit_bpf_dispatcher(jit, a, a + pivot,\t/* emit lower_part */\n \t\t\t\t  progs, image, buf);\n \tif (err)\n \t\treturn err;\n@@ -3893,16 +3782,15 @@ static int emit_bpf_dispatcher(u8 **pprog, int a, int b, s64 *progs, u8 *image,\n \t * Coding Rule 11: All branch targets should be 16-byte\n \t * aligned.\n \t */\n-\temit_align(\u0026prog, 16);\n-\tjg_offset = prog - jg_reloc;\n+\temit_align(jit, 16);\n+\tjg_offset = jit-\u003eprog - jg_reloc;\n \temit_code(jg_reloc - jg_bytes, jg_offset, jg_bytes);\n \n-\terr = emit_bpf_dispatcher(\u0026prog, a + pivot + 1,\t/* emit upper_part */\n+\terr = emit_bpf_dispatcher(jit, a + pivot + 1,\t/* emit upper_part */\n \t\t\t\t  b, progs, image, buf);\n \tif (err)\n \t\treturn err;\n \n-\t*pprog = prog;\n \treturn 0;\n }\n \n@@ -3920,10 +3808,10 @@ static int cmp_ips(const void *a, const void *b)\n \n int arch_prepare_bpf_dispatcher(void *image, void *buf, s64 *funcs, int num_funcs)\n {\n-\tu8 *prog = buf;\n+\tstruct jit_emit_context jit = { .prog = buf };\n \n \tsort(funcs, num_funcs, sizeof(funcs[0]), cmp_ips, NULL);\n-\treturn emit_bpf_dispatcher(\u0026prog, 0, num_funcs - 1, funcs, image, buf);\n+\treturn emit_bpf_dispatcher(\u0026jit, 0, num_funcs - 1, funcs, image, buf);\n }\n \n static void priv_stack_init_guard(void __percpu *priv_stack_ptr, int alloc_size)\n"
}

{
  "EnableConfigs": [
    "BPF_SYSCALL",
    "BPF_JIT",
    "CALL_THUNKS",
    "MITIGATION_CALL_DEPTH_TRACKING",
    "X86_64"
  ],
  "FocusSymbols": [
    "arch_prepare_bpf_trampoline",
    "arch_bpf_trampoline_size",
    "bpf_int_jit_compile",
    "arch_prepare_bpf_dispatcher",
    "x86_call_depth_emit_accounting"
  ],
  "Reasoning": "The patch refactors the x86 BPF JIT compiler and trampoline generator by introducing a struct jit_emit_context with dry-run support. This changes how trampoline sizing is computed (dry-run without allocation), updates instruction emission logic, and alters x86 call depth accounting integration. These changes affect BPF program compilation, trampoline generation, and execution reachable via the bpf() syscall.",
  "WorthFuzzing": true
}

You are an expert Linux kernel maintainer.
Your job is to review a provided patch series and determine
if it makes functional changes to the kernel that should be fuzzed.

IMPORTANT: The changes have ALREADY been applied and committed as the HEAD commit in
your workspace. Do NOT rely on your internal knowledge of the kernel. You must actively
use your code access tools to examine the actual source code and confirm any assumptions.

Return WorthFuzzing=false if the patch only contains:
- Modifications to Documentation/, Kconfig files, or code comments.
- Purely decorative changes, such as logging (e.g., pr_err, printk) or tracepoints.
- Changes to numeric constants or macros that do not functionally alter execution flow.
- Code paths that are impossible to reach in virtualized environments like GCE or QEMU,
  even when utilizing software-emulated hardware (e.g., usb gadget, mac80211_hwsim).
- Code in vendor-specific PCIe switch, SmartNIC, or GPU drivers (e.g., mlxsw, pds_core, qed,
  ionic, amdgpu) that require physical PCIe hardware cards not emulated in standard QEMU.
- Driver .remove, .shutdown, or pci_unregister_driver teardown callbacks (e.g., igb_remove)
  that are executed only during PCI hot-unplug or sysfs driver unbind operations.

If it modifies reachable core kernel logic, drivers, or architectures, use your code search
tools to verify the code can be executed, then return WorthFuzzing=true.

When returning WorthFuzzing=true, you MUST ALSO:
1. Extract any specific kernel functions that should be heavily fuzzed into FocusSymbols.
   Avoid listing generic hot-path functions to prevent skewed test distributions.
   Prefer non-static, non-inlined API entrypoint functions over internal static helper functions
   (which are inlined by the compiler and do not have distinct symbol addresses).
2. Identify any specific CONFIG_ options required to properly test this new/modified feature.
   Go and look into the Kconfig files and check for ifdefs around the code, do not make assumptions.
   Also check "depends on" lines in Kconfig to include any non-standard parent subsystem configs
   needed for Kbuild to compile the code statically into vmlinux. List them in the EnableConfigs
   output array, and DO NOT add a 'CONFIG_' prefix (e.g., return "NET_IPV4" instead of "CONFIG_NET_IPV4").
Prefer calling several tools at the same time to save round-trips.


Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.

For your convenience, here is the diff of the changes:
commit e09e4ad11f933fa9ab21debd310aeb220f3f52c0
Author: syz-cluster <triage@syzkaller.com>
Date:   Mon Aug 31 14:02:28 2026 +0000

    syz-cluster: applied patch under review

diff --git a/arch/x86/include/asm/alternative.h b/arch/x86/include/asm/alternative.h
index 08af86ef090a8..2ef4937411860 100644
--- a/arch/x86/include/asm/alternative.h
+++ b/arch/x86/include/asm/alternative.h
@@ -83,7 +83,7 @@ extern void callthunks_patch_builtin_calls(void);
 extern void callthunks_patch_module_calls(struct callthunk_sites *sites,
 					  struct module *mod);
 extern void *callthunks_translate_call_dest(void *dest);
-extern int x86_call_depth_emit_accounting(u8 **pprog, void *func, void *ip);
+extern int x86_call_depth_emit_accounting(u8 *insn_buff, void *func, void *ip);
 #else
 static __always_inline void callthunks_patch_builtin_calls(void) {}
 static __always_inline void
@@ -93,7 +93,7 @@ static __always_inline void *callthunks_translate_call_dest(void *dest)
 {
 	return dest;
 }
-static __always_inline int x86_call_depth_emit_accounting(u8 **pprog,
+static __always_inline int x86_call_depth_emit_accounting(u8 *insn_buff,
 							  void *func, void *ip)
 {
 	return 0;
diff --git a/arch/x86/include/asm/text-patching.h b/arch/x86/include/asm/text-patching.h
index f2d142a0a862e..a0a7c778b0a2a 100644
--- a/arch/x86/include/asm/text-patching.h
+++ b/arch/x86/include/asm/text-patching.h
@@ -13,6 +13,8 @@
  */
 #define TEXT_POKE_MAX_OPCODE_SIZE	5
 
+#define MAX_PATCH_LEN	(255-1)
+
 extern void text_poke_early(void *addr, const void *opcode, size_t len);
 
 extern void text_poke_apply_relocation(u8 *buf, const u8 * const instr, size_t instrlen, u8 *repl, size_t repl_len);
diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c
index 91b1cdd165699..5f9989c8559ea 100644
--- a/arch/x86/kernel/alternative.c
+++ b/arch/x86/kernel/alternative.c
@@ -18,8 +18,6 @@ int __read_mostly alternatives_patched;
 
 EXPORT_SYMBOL_GPL(alternatives_patched);
 
-#define MAX_PATCH_LEN (255-1)
-
 #define DA_ALL		(~0)
 #define DA_ALT		0x01
 #define DA_RET		0x02
diff --git a/arch/x86/kernel/callthunks.c b/arch/x86/kernel/callthunks.c
index e37728f703227..15b221d06901c 100644
--- a/arch/x86/kernel/callthunks.c
+++ b/arch/x86/kernel/callthunks.c
@@ -23,8 +23,6 @@
 
 static int __initdata_or_module debug_callthunks;
 
-#define MAX_PATCH_LEN (255-1)
-
 #define prdbg(fmt, args...)					\
 do {								\
 	if (debug_callthunks)					\
@@ -298,10 +296,9 @@ static bool is_callthunk(void *addr)
 	return !bcmp(pad, insn_buff, tmpl_size);
 }
 
-int x86_call_depth_emit_accounting(u8 **pprog, void *func, void *ip)
+int x86_call_depth_emit_accounting(u8 *insn_buff, void *func, void *ip)
 {
 	unsigned int tmpl_size = SKL_TMPL_SIZE;
-	u8 insn_buff[MAX_PATCH_LEN];
 
 	if (!thunks_initialized)
 		return 0;
@@ -313,8 +310,6 @@ int x86_call_depth_emit_accounting(u8 **pprog, void *func, void *ip)
 	memcpy(insn_buff, skl_call_thunk_template, tmpl_size);
 	text_poke_apply_relocation(insn_buff, ip, tmpl_size, skl_call_thunk_template, tmpl_size);
 
-	memcpy(*pprog, insn_buff, tmpl_size);
-	*pprog += tmpl_size;
 	return tmpl_size;
 }
 #endif
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index 48429fae06410..2ec734c69b693 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -23,6 +23,11 @@
 
 static bool all_callee_regs_used[4] = {true, true, true, true};
 
+struct jit_emit_context {
+	u8 *prog;
+	bool dry_run;
+};
+
 static u8 *emit_code(u8 *ptr, u32 bytes, unsigned int len)
 {
 	if (len == 1)
@@ -36,8 +41,16 @@ static u8 *emit_code(u8 *ptr, u32 bytes, unsigned int len)
 	return ptr + len;
 }
 
+static void emit_code_jit(struct jit_emit_context *jit, u32 bytes, unsigned int len)
+{
+	if (jit->dry_run)
+		jit->prog += len;
+	else
+		jit->prog = emit_code(jit->prog, bytes, len);
+}
+
 #define EMIT(bytes, len) \
-	do { prog = emit_code(prog, bytes, len); } while (0)
+	emit_code_jit(jit, bytes, len)
 
 #define EMIT1(b1)		EMIT(b1, 1)
 #define EMIT2(b1, b2)		EMIT((b1) + ((b2) << 8), 2)
@@ -149,6 +162,19 @@ static int bpf_size_to_x86_bytes(int bpf_size)
 		return 0;
 }
 
+static int bpf_call_depth_emit_accounting(struct jit_emit_context *jit, void *func, void *ip)
+{
+	u8 insn_buff[MAX_PATCH_LEN];
+	int size;
+
+	size = x86_call_depth_emit_accounting(insn_buff, func, ip);
+	if (!jit->dry_run)
+		memcpy(jit->prog, insn_buff, size);
+
+	jit->prog += size;
+	return size;
+}
+
 /*
  * List of x86 cond jumps opcodes (. + s8)
  * Add 0x10 (and an extra 0x0f) to generate far jumps (. + s32)
@@ -309,6 +335,8 @@ int bpf_arch_text_invalidate(void *dst, size_t len)
 }
 
 struct jit_context {
+	struct jit_emit_context jit;
+
 	int cleanup_addr; /* Epilogue code offset */
 
 	/*
@@ -329,34 +357,23 @@ struct jit_context {
 /* Number of bytes that will be skipped on tailcall */
 #define X86_TAIL_CALL_OFFSET	(12 + ENDBR_INSN_SIZE)
 
-static void push_r9(u8 **pprog)
+static void push_r9(struct jit_emit_context *jit)
 {
-	u8 *prog = *pprog;
-
 	EMIT2(0x41, 0x51);   /* push r9 */
-	*pprog = prog;
 }
 
-static void pop_r9(u8 **pprog)
+static void pop_r9(struct jit_emit_context *jit)
 {
-	u8 *prog = *pprog;
-
 	EMIT2(0x41, 0x59);   /* pop r9 */
-	*pprog = prog;
 }
 
-static void push_r12(u8 **pprog)
+static void push_r12(struct jit_emit_context *jit)
 {
-	u8 *prog = *pprog;
-
 	EMIT2(0x41, 0x54);   /* push r12 */
-	*pprog = prog;
 }
 
-static void push_callee_regs(u8 **pprog, bool *callee_regs_used)
+static void push_callee_regs(struct jit_emit_context *jit, bool *callee_regs_used)
 {
-	u8 *prog = *pprog;
-
 	if (callee_regs_used[0])
 		EMIT1(0x53);         /* push rbx */
 	if (callee_regs_used[1])
@@ -365,21 +382,15 @@ static void push_callee_regs(u8 **pprog, bool *callee_regs_used)
 		EMIT2(0x41, 0x56);   /* push r14 */
 	if (callee_regs_used[3])
 		EMIT2(0x41, 0x57);   /* push r15 */
-	*pprog = prog;
 }
 
-static void pop_r12(u8 **pprog)
+static void pop_r12(struct jit_emit_context *jit)
 {
-	u8 *prog = *pprog;
-
 	EMIT2(0x41, 0x5C);   /* pop r12 */
-	*pprog = prog;
 }
 
-static void pop_callee_regs(u8 **pprog, bool *callee_regs_used)
+static void pop_callee_regs(struct jit_emit_context *jit, bool *callee_regs_used)
 {
-	u8 *prog = *pprog;
-
 	if (callee_regs_used[3])
 		EMIT2(0x41, 0x5F);   /* pop r15 */
 	if (callee_regs_used[2])
@@ -388,40 +399,32 @@ static void pop_callee_regs(u8 **pprog, bool *callee_regs_used)
 		EMIT2(0x41, 0x5D);   /* pop r13 */
 	if (callee_regs_used[0])
 		EMIT1(0x5B);         /* pop rbx */
-	*pprog = prog;
 }
 
 /* add rsp, depth */
-static void emit_add_rsp(u8 **pprog, u16 depth)
+static void emit_add_rsp(struct jit_emit_context *jit, u16 depth)
 {
-	u8 *prog = *pprog;
-
 	if (!depth)
 		return;
 	if (is_imm8(depth))
 		EMIT4(0x48, 0x83, 0xC4, depth); /* add rsp, imm8 */
 	else
 		EMIT3_off32(0x48, 0x81, 0xC4, depth); /* add rsp, imm32 */
-	*pprog = prog;
 }
 
 /* sub rsp, depth */
-static void emit_sub_rsp(u8 **pprog, u16 depth)
+static void emit_sub_rsp(struct jit_emit_context *jit, u16 depth)
 {
-	u8 *prog = *pprog;
-
 	if (!depth)
 		return;
 	if (is_imm8(depth))
 		EMIT4(0x48, 0x83, 0xEC, depth); /* sub rsp, imm8 */
 	else
 		EMIT3_off32(0x48, 0x81, 0xEC, depth); /* sub rsp, imm32 */
-	*pprog = prog;
 }
 
-static void emit_nops(u8 **pprog, int len)
+static void emit_nops(struct jit_emit_context *jit, int len)
 {
-	u8 *prog = *pprog;
 	int i, noplen;
 
 	while (len > 0) {
@@ -434,72 +437,56 @@ static void emit_nops(u8 **pprog, int len)
 			EMIT1(x86_nops[noplen][i]);
 		len -= noplen;
 	}
-
-	*pprog = prog;
 }
 
 /*
  * Emit the various CFI preambles, see asm/cfi.h and the comments about FineIBT
  * in arch/x86/kernel/alternative.c
  */
-static int emit_call(u8 **prog, void *func, void *ip);
+static int emit_call(struct jit_emit_context *jit, void *func, void *ip);
 
-static void emit_fineibt(u8 **pprog, u8 *ip, u32 hash, int arity)
+static void emit_fineibt(struct jit_emit_context *jit, u8 *ip, u32 hash, int arity)
 {
-	u8 *prog = *pprog;
-
 	EMIT_ENDBR();
 	EMIT1_off32(0x2d, hash);			/* subl $hash, %eax	*/
 	if (cfi_bhi) {
 		EMIT2(0x2e, 0x2e);			/* cs cs */
-		emit_call(&prog, __bhi_args[arity], ip + 11);
+		emit_call(jit, __bhi_args[arity], ip + 11);
 	} else {
 		EMIT3_off32(0x2e, 0x0f, 0x85, 3);	/* jne.d32,pn 3		*/
 	}
 	EMIT_ENDBR_POISON();
-
-	*pprog = prog;
 }
 
-static void emit_kcfi(u8 **pprog, u32 hash)
+static void emit_kcfi(struct jit_emit_context *jit, u32 hash)
 {
-	u8 *prog = *pprog;
-
 	EMIT1_off32(0xb8, hash);			/* movl $hash, %eax	*/
 #ifdef CONFIG_CALL_PADDING
 	for (int i = 0; i < CONFIG_FUNCTION_PADDING_CFI; i++)
 		EMIT1(0x90);
 #endif
 	EMIT_ENDBR();
-
-	*pprog = prog;
 }
 
-static void emit_cfi(u8 **pprog, u8 *ip, u32 hash, int arity)
+static void emit_cfi(struct jit_emit_context *jit, u8 *ip, u32 hash, int arity)
 {
-	u8 *prog = *pprog;
-
 	switch (cfi_mode) {
 	case CFI_FINEIBT:
-		emit_fineibt(&prog, ip, hash, arity);
+		emit_fineibt(jit, ip, hash, arity);
 		break;
 
 	case CFI_KCFI:
-		emit_kcfi(&prog, hash);
+		emit_kcfi(jit, hash);
 		break;
 
 	default:
 		EMIT_ENDBR();
 		break;
 	}
-
-	*pprog = prog;
 }
 
-static void emit_prologue_tail_call(u8 **pprog, bool is_subprog)
+static void emit_prologue_tail_call(struct jit_emit_context *jit, bool is_subprog)
 {
-	u8 *prog = *pprog;
-
 	if (!is_subprog) {
 		/* cmp rax, MAX_TAIL_CALL_CNT */
 		EMIT4(0x48, 0x83, 0xF8, MAX_TAIL_CALL_CNT);
@@ -523,8 +510,6 @@ static void emit_prologue_tail_call(u8 **pprog, bool is_subprog)
 		EMIT1(0x50);             /* push rax */
 		EMIT1(0x50);             /* push rax */
 	}
-
-	*pprog = prog;
 }
 
 /*
@@ -532,21 +517,19 @@ static void emit_prologue_tail_call(u8 **pprog, bool is_subprog)
  * bpf_tail_call helper will skip the first X86_TAIL_CALL_OFFSET bytes
  * while jumping to another program
  */
-static void emit_prologue(u8 **pprog, u8 *ip, u32 stack_depth, bool ebpf_from_cbpf,
-			  bool tail_call_reachable, bool is_subprog,
+static void emit_prologue(struct jit_emit_context *jit, u8 *ip, u32 stack_depth,
+			  bool ebpf_from_cbpf, bool tail_call_reachable, bool is_subprog,
 			  bool is_exception_cb)
 {
-	u8 *prog = *pprog;
-
 	if (is_subprog) {
-		emit_cfi(&prog, ip, cfi_bpf_subprog_hash, 5);
+		emit_cfi(jit, ip, cfi_bpf_subprog_hash, 5);
 	} else {
-		emit_cfi(&prog, ip, cfi_bpf_hash, 1);
+		emit_cfi(jit, ip, cfi_bpf_hash, 1);
 	}
 	/* BPF trampoline can be made to work without these nops,
 	 * but let's waste 5 bytes for now and optimize later
 	 */
-	emit_nops(&prog, X86_PATCH_SIZE);
+	emit_nops(jit, X86_PATCH_SIZE);
 	if (!ebpf_from_cbpf) {
 		if (tail_call_reachable && !is_subprog)
 			/* When it's the entry of the whole tailcall context,
@@ -555,7 +538,7 @@ static void emit_prologue(u8 **pprog, u8 *ip, u32 stack_depth, bool ebpf_from_cb
 			EMIT3(0x48, 0x31, 0xC0); /* xor rax, rax */
 		else
 			/* Keep the same instruction layout. */
-			emit_nops(&prog, 3);     /* nop3 */
+			emit_nops(jit, 3);     /* nop3 */
 	}
 	/* Exception callback receives FP as third parameter */
 	if (is_exception_cb) {
@@ -565,8 +548,8 @@ static void emit_prologue(u8 **pprog, u8 *ip, u32 stack_depth, bool ebpf_from_cb
 		 * first restore those callee-saved regs from stack, before
 		 * reusing the stack frame.
 		 */
-		pop_callee_regs(&prog, all_callee_regs_used);
-		pop_r12(&prog);
+		pop_callee_regs(jit, all_callee_regs_used);
+		pop_r12(jit);
 		/* Reset the stack frame. */
 		EMIT3(0x48, 0x89, 0xEC); /* mov rsp, rbp */
 	} else {
@@ -581,40 +564,38 @@ static void emit_prologue(u8 **pprog, u8 *ip, u32 stack_depth, bool ebpf_from_cb
 	if (stack_depth)
 		EMIT3_off32(0x48, 0x81, 0xEC, round_up(stack_depth, 8));
 	if (tail_call_reachable)
-		emit_prologue_tail_call(&prog, is_subprog);
-	*pprog = prog;
+		emit_prologue_tail_call(jit, is_subprog);
 }
 
-static int emit_patch(u8 **pprog, void *func, void *ip, u8 opcode)
+static int emit_patch(struct jit_emit_context *jit, void *func, void *ip, u8 opcode)
 {
-	u8 *prog = *pprog;
 	s64 offset;
 
 	offset = func - (ip + X86_PATCH_SIZE);
-	if (!is_simm32(offset)) {
+	/* We do not have meaningful ip value in the dry run, skip the check. */
+	if (!jit->dry_run && !is_simm32(offset)) {
 		pr_err("Target call %p is out of range\n", func);
 		return -ERANGE;
 	}
 	EMIT1_off32(opcode, offset);
-	*pprog = prog;
 	return 0;
 }
 
-static int emit_call(u8 **pprog, void *func, void *ip)
+static int emit_call(struct jit_emit_context *jit, void *func, void *ip)
 {
-	return emit_patch(pprog, func, ip, 0xE8);
+	return emit_patch(jit, func, ip, 0xE8);
 }
 
-static int emit_rsb_call(u8 **pprog, void *func, void *ip)
+static int emit_rsb_call(struct jit_emit_context *jit, void *func, void *ip)
 {
 	OPTIMIZER_HIDE_VAR(func);
-	ip += x86_call_depth_emit_accounting(pprog, func, ip);
-	return emit_patch(pprog, func, ip, 0xE8);
+	ip += bpf_call_depth_emit_accounting(jit, func, ip);
+	return emit_patch(jit, func, ip, 0xE8);
 }
 
-static int emit_jump(u8 **pprog, void *func, void *ip)
+static int emit_jump(struct jit_emit_context *jit, void *func, void *ip)
 {
-	return emit_patch(pprog, func, ip, 0xE9);
+	return emit_patch(jit, func, ip, 0xE9);
 }
 
 static int __bpf_arch_text_poke(void *ip, enum bpf_text_poke_type old_t,
@@ -622,27 +603,27 @@ static int __bpf_arch_text_poke(void *ip, enum bpf_text_poke_type old_t,
 				void *old_addr, void *new_addr)
 {
 	const u8 *nop_insn = x86_nops[5];
+	struct jit_emit_context jit = {};
 	u8 old_insn[X86_PATCH_SIZE];
 	u8 new_insn[X86_PATCH_SIZE];
-	u8 *prog;
 	int ret;
 
 	memcpy(old_insn, nop_insn, X86_PATCH_SIZE);
 	if (old_t != BPF_MOD_NOP && old_addr) {
-		prog = old_insn;
+		jit.prog = old_insn;
 		ret = old_t == BPF_MOD_CALL ?
-		      emit_call(&prog, old_addr, ip) :
-		      emit_jump(&prog, old_addr, ip);
+		      emit_call(&jit, old_addr, ip) :
+		      emit_jump(&jit, old_addr, ip);
 		if (ret)
 			return ret;
 	}
 
 	memcpy(new_insn, nop_insn, X86_PATCH_SIZE);
 	if (new_t != BPF_MOD_NOP && new_addr) {
-		prog = new_insn;
+		jit.prog = new_insn;
 		ret = new_t == BPF_MOD_CALL ?
-		      emit_call(&prog, new_addr, ip) :
-		      emit_jump(&prog, new_addr, ip);
+		      emit_call(&jit, new_addr, ip) :
+		      emit_jump(&jit, new_addr, ip);
 		if (ret)
 			return ret;
 	}
@@ -682,58 +663,47 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type old_t,
 
 #define EMIT_LFENCE()	EMIT3(0x0F, 0xAE, 0xE8)
 
-static void __emit_indirect_jump(u8 **pprog, int reg, bool ereg)
+static void __emit_indirect_jump(struct jit_emit_context *jit, int reg, bool ereg)
 {
-	u8 *prog = *pprog;
-
 	if (ereg)
 		EMIT1(0x41);
 
 	EMIT2(0xFF, 0xE0 + reg);
-
-	*pprog = prog;
 }
 
-static void emit_indirect_jump(u8 **pprog, int bpf_reg, u8 *ip)
+static void emit_indirect_jump(struct jit_emit_context *jit, int bpf_reg, u8 *ip)
 {
-	u8 *prog = *pprog;
 	int reg = reg2hex[bpf_reg];
 	bool ereg = is_ereg(bpf_reg);
 
 	if (cpu_feature_enabled(X86_FEATURE_INDIRECT_THUNK_ITS)) {
 		OPTIMIZER_HIDE_VAR(reg);
-		emit_jump(&prog, its_static_thunk(reg + 8*ereg), ip);
+		emit_jump(jit, its_static_thunk(reg + 8*ereg), ip);
 	} else if (cpu_feature_enabled(X86_FEATURE_RETPOLINE_LFENCE)) {
 		EMIT_LFENCE();
-		__emit_indirect_jump(&prog, reg, ereg);
+		__emit_indirect_jump(jit, reg, ereg);
 	} else if (cpu_feature_enabled(X86_FEATURE_RETPOLINE)) {
 		OPTIMIZER_HIDE_VAR(reg);
 		if (cpu_feature_enabled(X86_FEATURE_CALL_DEPTH))
-			emit_jump(&prog, &__x86_indirect_jump_thunk_array[reg + 8*ereg], ip);
+			emit_jump(jit, &__x86_indirect_jump_thunk_array[reg + 8*ereg], ip);
 		else
-			emit_jump(&prog, &__x86_indirect_thunk_array[reg + 8*ereg], ip);
+			emit_jump(jit, &__x86_indirect_thunk_array[reg + 8*ereg], ip);
 	} else {
-		__emit_indirect_jump(&prog, reg, ereg);
+		__emit_indirect_jump(jit, reg, ereg);
 		if (IS_ENABLED(CONFIG_MITIGATION_RETPOLINE) || IS_ENABLED(CONFIG_MITIGATION_SLS))
 			EMIT1(0xCC);		/* int3 */
 	}
-
-	*pprog = prog;
 }
 
-static void emit_return(u8 **pprog, u8 *ip)
+static void emit_return(struct jit_emit_context *jit, u8 *ip)
 {
-	u8 *prog = *pprog;
-
 	if (cpu_wants_rethunk()) {
-		emit_jump(&prog, x86_return_thunk, ip);
+		emit_jump(jit, x86_return_thunk, ip);
 	} else {
 		EMIT1(0xC3);		/* ret */
 		if (IS_ENABLED(CONFIG_MITIGATION_SLS))
 			EMIT1(0xCC);	/* int3 */
 	}
-
-	*pprog = prog;
 }
 
 #define BPF_TAIL_CALL_CNT_PTR_STACK_OFF(stack)	(-16 - round_up(stack, 8))
@@ -753,12 +723,13 @@ static void emit_return(u8 **pprog, u8 *ip)
  * out:
  */
 static void emit_bpf_tail_call_indirect(struct bpf_prog *bpf_prog,
-					u8 **pprog, bool *callee_regs_used,
+					bool *callee_regs_used,
 					u32 stack_depth, u8 *ip,
 					struct jit_context *ctx)
 {
 	int tcc_ptr_off = BPF_TAIL_CALL_CNT_PTR_STACK_OFF(stack_depth);
-	u8 *prog = *pprog, *start = *pprog;
+	struct jit_emit_context *jit = &ctx->jit;
+	u8 *start = jit->prog;
 	int offset;
 
 	/*
@@ -775,7 +746,7 @@ static void emit_bpf_tail_call_indirect(struct bpf_prog *bpf_prog,
 	EMIT3(0x39, 0x56,                         /* cmp dword ptr [rsi + 16], edx */
 	      offsetof(struct bpf_array, map.max_entries));
 
-	offset = ctx->tail_call_indirect_label - (prog + 2 - start);
+	offset = ctx->tail_call_indirect_label - (jit->prog + 2 - start);
 	EMIT2(X86_JBE, offset);                   /* jbe out */
 
 	/*
@@ -785,7 +756,7 @@ static void emit_bpf_tail_call_indirect(struct bpf_prog *bpf_prog,
 	EMIT3_off32(0x48, 0x8B, 0x85, tcc_ptr_off); /* mov rax, qword ptr [rbp - tcc_ptr_off] */
 	EMIT4(0x48, 0x83, 0x38, MAX_TAIL_CALL_CNT); /* cmp qword ptr [rax], MAX_TAIL_CALL_CNT */
 
-	offset = ctx->tail_call_indirect_label - (prog + 2 - start);
+	offset = ctx->tail_call_indirect_label - (jit->prog + 2 - start);
 	EMIT2(X86_JAE, offset);                   /* jae out */
 
 	/* prog = array->ptrs[index]; */
@@ -798,19 +769,19 @@ static void emit_bpf_tail_call_indirect(struct bpf_prog *bpf_prog,
 	 */
 	EMIT3(0x48, 0x85, 0xC9);                  /* test rcx,rcx */
 
-	offset = ctx->tail_call_indirect_label - (prog + 2 - start);
+	offset = ctx->tail_call_indirect_label - (jit->prog + 2 - start);
 	EMIT2(X86_JE, offset);                    /* je out */
 
 	/* Inc tail_call_cnt if the slot is populated. */
 	EMIT4(0x48, 0x83, 0x00, 0x01);            /* add qword ptr [rax], 1 */
 
 	if (bpf_prog->aux->exception_boundary) {
-		pop_callee_regs(&prog, all_callee_regs_used);
-		pop_r12(&prog);
+		pop_callee_regs(jit, all_callee_regs_used);
+		pop_r12(jit);
 	} else {
-		pop_callee_regs(&prog, callee_regs_used);
+		pop_callee_regs(jit, callee_regs_used);
 		if (bpf_arena_get_kern_vm_start(bpf_prog->aux->arena))
-			pop_r12(&prog);
+			pop_r12(jit);
 	}
 
 	/* Pop tail_call_cnt_ptr. */
@@ -833,21 +804,21 @@ static void emit_bpf_tail_call_indirect(struct bpf_prog *bpf_prog,
 	 * rdi == ctx (1st arg)
 	 * rcx == prog->bpf_func + X86_TAIL_CALL_OFFSET
 	 */
-	emit_indirect_jump(&prog, BPF_REG_4 /* R4 -> rcx */, ip + (prog - start));
+	emit_indirect_jump(jit, BPF_REG_4 /* R4 -> rcx */, ip + (jit->prog - start));
 
 	/* out: */
-	ctx->tail_call_indirect_label = prog - start;
-	*pprog = prog;
+	ctx->tail_call_indirect_label = jit->prog - start;
 }
 
 static void emit_bpf_tail_call_direct(struct bpf_prog *bpf_prog,
 				      struct bpf_jit_poke_descriptor *poke,
-				      u8 **pprog, u8 *ip,
+				      u8 *ip,
 				      bool *callee_regs_used, u32 stack_depth,
 				      struct jit_context *ctx)
 {
+	struct jit_emit_context *jit = &ctx->jit;
 	int tcc_ptr_off = BPF_TAIL_CALL_CNT_PTR_STACK_OFF(stack_depth);
-	u8 *prog = *pprog, *start = *pprog;
+	u8 *start = jit->prog;
 	int offset;
 
 	/*
@@ -857,27 +828,27 @@ static void emit_bpf_tail_call_direct(struct bpf_prog *bpf_prog,
 	EMIT3_off32(0x48, 0x8B, 0x85, tcc_ptr_off);   /* mov rax, qword ptr [rbp - tcc_ptr_off] */
 	EMIT4(0x48, 0x83, 0x38, MAX_TAIL_CALL_CNT);   /* cmp qword ptr [rax], MAX_TAIL_CALL_CNT */
 
-	offset = ctx->tail_call_direct_label - (prog + 2 - start);
+	offset = ctx->tail_call_direct_label - (jit->prog + 2 - start);
 	EMIT2(X86_JAE, offset);                       /* jae out */
 
-	poke->tailcall_bypass = ip + (prog - start);
+	poke->tailcall_bypass = ip + (jit->prog - start);
 	poke->adj_off = X86_TAIL_CALL_OFFSET;
 	poke->tailcall_target = ip + ctx->tail_call_direct_label - X86_PATCH_SIZE;
 	poke->bypass_addr = (u8 *)poke->tailcall_target + X86_PATCH_SIZE;
 
-	emit_jump(&prog, (u8 *)poke->tailcall_target + X86_PATCH_SIZE,
+	emit_jump(jit, (u8 *)poke->tailcall_target + X86_PATCH_SIZE,
 		  poke->tailcall_bypass);
 
 	/* Inc tail_call_cnt if the slot is populated. */
 	EMIT4(0x48, 0x83, 0x00, 0x01);                /* add qword ptr [rax], 1 */
 
 	if (bpf_prog->aux->exception_boundary) {
-		pop_callee_regs(&prog, all_callee_regs_used);
-		pop_r12(&prog);
+		pop_callee_regs(jit, all_callee_regs_used);
+		pop_r12(jit);
 	} else {
-		pop_callee_regs(&prog, callee_regs_used);
+		pop_callee_regs(jit, callee_regs_used);
 		if (bpf_arena_get_kern_vm_start(bpf_prog->aux->arena))
-			pop_r12(&prog);
+			pop_r12(jit);
 	}
 
 	/* Pop tail_call_cnt_ptr. */
@@ -889,12 +860,10 @@ static void emit_bpf_tail_call_direct(struct bpf_prog *bpf_prog,
 	if (stack_depth)
 		EMIT3_off32(0x48, 0x81, 0xC4, round_up(stack_depth, 8));
 
-	emit_nops(&prog, X86_PATCH_SIZE);
+	emit_nops(jit, X86_PATCH_SIZE);
 
 	/* out: */
-	ctx->tail_call_direct_label = prog - start;
-
-	*pprog = prog;
+	ctx->tail_call_direct_label = jit->prog - start;
 }
 
 static void bpf_tail_call_direct_fixup(struct bpf_prog *prog)
@@ -935,10 +904,9 @@ static void bpf_tail_call_direct_fixup(struct bpf_prog *prog)
 	}
 }
 
-static void emit_mov_imm32(u8 **pprog, bool sign_propagate,
+static void emit_mov_imm32(struct jit_emit_context *jit, bool sign_propagate,
 			   u32 dst_reg, const u32 imm32)
 {
-	u8 *prog = *pprog;
 	u8 b1, b2, b3;
 
 	/*
@@ -972,14 +940,12 @@ static void emit_mov_imm32(u8 **pprog, bool sign_propagate,
 		EMIT1(add_1mod(0x40, dst_reg));
 	EMIT1_off32(add_1reg(0xB8, dst_reg), imm32);
 done:
-	*pprog = prog;
 }
 
-static void emit_mov_imm64(u8 **pprog, u32 dst_reg,
+static void emit_mov_imm64(struct jit_emit_context *jit, u32 dst_reg,
 			   const u32 imm32_hi, const u32 imm32_lo)
 {
 	u64 imm64 = ((u64)imm32_hi << 32) | (u32)imm32_lo;
-	u8 *prog = *pprog;
 
 	if (is_uimm32(imm64)) {
 		/*
@@ -988,23 +954,19 @@ static void emit_mov_imm64(u8 **pprog, u32 dst_reg,
 		 * directly, so save couple of bytes by just doing
 		 * 'mov %eax, imm32' instead.
 		 */
-		emit_mov_imm32(&prog, false, dst_reg, imm32_lo);
+		emit_mov_imm32(jit, false, dst_reg, imm32_lo);
 	} else if (is_simm32(imm64)) {
-		emit_mov_imm32(&prog, true, dst_reg, imm32_lo);
+		emit_mov_imm32(jit, true, dst_reg, imm32_lo);
 	} else {
 		/* movabsq rax, imm64 */
 		EMIT2(add_1mod(0x48, dst_reg), add_1reg(0xB8, dst_reg));
 		EMIT(imm32_lo, 4);
 		EMIT(imm32_hi, 4);
 	}
-
-	*pprog = prog;
 }
 
-static void emit_mov_reg(u8 **pprog, bool is64, u32 dst_reg, u32 src_reg)
+static void emit_mov_reg(struct jit_emit_context *jit, bool is64, u32 dst_reg, u32 src_reg)
 {
-	u8 *prog = *pprog;
-
 	if (is64) {
 		/* mov dst, src */
 		EMIT_mov(dst_reg, src_reg);
@@ -1014,15 +976,11 @@ static void emit_mov_reg(u8 **pprog, bool is64, u32 dst_reg, u32 src_reg)
 			EMIT1(add_2mod(0x40, dst_reg, src_reg));
 		EMIT2(0x89, add_2reg(0xC0, dst_reg, src_reg));
 	}
-
-	*pprog = prog;
 }
 
-static void emit_movsx_reg(u8 **pprog, int num_bits, bool is64, u32 dst_reg,
+static void emit_movsx_reg(struct jit_emit_context *jit, int num_bits, bool is64, u32 dst_reg,
 			   u32 src_reg)
 {
-	u8 *prog = *pprog;
-
 	if (is64) {
 		/* movs[b,w,l]q dst, src */
 		if (num_bits == 8)
@@ -1046,15 +1004,11 @@ static void emit_movsx_reg(u8 **pprog, int num_bits, bool is64, u32 dst_reg,
 			      add_2reg(0xC0, src_reg, dst_reg));
 		}
 	}
-
-	*pprog = prog;
 }
 
 /* Emit the suffix (ModR/M etc) for addressing *(ptr_reg + off) and val_reg */
-static void emit_insn_suffix(u8 **pprog, u32 ptr_reg, u32 val_reg, int off)
+static void emit_insn_suffix(struct jit_emit_context *jit, u32 ptr_reg, u32 val_reg, int off)
 {
-	u8 *prog = *pprog;
-
 	if (is_imm8(off)) {
 		/* 1-byte signed displacement.
 		 *
@@ -1067,54 +1021,43 @@ static void emit_insn_suffix(u8 **pprog, u32 ptr_reg, u32 val_reg, int off)
 		/* 4-byte signed displacement */
 		EMIT1_off32(add_2reg(0x80, ptr_reg, val_reg), off);
 	}
-	*pprog = prog;
 }
 
-static void emit_insn_suffix_SIB(u8 **pprog, u32 ptr_reg, u32 val_reg, u32 index_reg, int off)
+static void emit_insn_suffix_SIB(struct jit_emit_context *jit, u32 ptr_reg, u32 val_reg,
+				 u32 index_reg, int off)
 {
-	u8 *prog = *pprog;
-
 	if (is_imm8(off)) {
 		EMIT3(add_2reg(0x44, BPF_REG_0, val_reg), add_2reg(0, ptr_reg, index_reg) /* SIB */, off);
 	} else {
 		EMIT2_off32(add_2reg(0x84, BPF_REG_0, val_reg), add_2reg(0, ptr_reg, index_reg) /* SIB */, off);
 	}
-	*pprog = prog;
 }
 
 /*
  * Emit a REX byte if it will be necessary to address these registers
  */
-static void maybe_emit_mod(u8 **pprog, u32 dst_reg, u32 src_reg, bool is64)
+static void maybe_emit_mod(struct jit_emit_context *jit, u32 dst_reg, u32 src_reg, bool is64)
 {
-	u8 *prog = *pprog;
-
 	if (is64)
 		EMIT1(add_2mod(0x48, dst_reg, src_reg));
 	else if (is_ereg(dst_reg) || is_ereg(src_reg))
 		EMIT1(add_2mod(0x40, dst_reg, src_reg));
-	*pprog = prog;
 }
 
 /*
  * Similar version of maybe_emit_mod() for a single register
  */
-static void maybe_emit_1mod(u8 **pprog, u32 reg, bool is64)
+static void maybe_emit_1mod(struct jit_emit_context *jit, u32 reg, bool is64)
 {
-	u8 *prog = *pprog;
-
 	if (is64)
 		EMIT1(add_1mod(0x48, reg));
 	else if (is_ereg(reg))
 		EMIT1(add_1mod(0x40, reg));
-	*pprog = prog;
 }
 
 /* LDX: dst_reg = *(u8*)(src_reg + off) */
-static void emit_ldx(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, int off)
+static void emit_ldx(struct jit_emit_context *jit, u32 size, u32 dst_reg, u32 src_reg, int off)
 {
-	u8 *prog = *pprog;
-
 	switch (size) {
 	case BPF_B:
 		/* Emit 'movzx rax, byte ptr [rax + off]' */
@@ -1136,15 +1079,12 @@ static void emit_ldx(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, int off)
 		EMIT2(add_2mod(0x48, src_reg, dst_reg), 0x8B);
 		break;
 	}
-	emit_insn_suffix(&prog, src_reg, dst_reg, off);
-	*pprog = prog;
+	emit_insn_suffix(jit, src_reg, dst_reg, off);
 }
 
 /* LDSX: dst_reg = *(s8*)(src_reg + off) */
-static void emit_ldsx(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, int off)
+static void emit_ldsx(struct jit_emit_context *jit, u32 size, u32 dst_reg, u32 src_reg, int off)
 {
-	u8 *prog = *pprog;
-
 	switch (size) {
 	case BPF_B:
 		/* Emit 'movsx rax, byte ptr [rax + off]' */
@@ -1159,14 +1099,12 @@ static void emit_ldsx(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, int off)
 		EMIT2(add_2mod(0x48, src_reg, dst_reg), 0x63);
 		break;
 	}
-	emit_insn_suffix(&prog, src_reg, dst_reg, off);
-	*pprog = prog;
+	emit_insn_suffix(jit, src_reg, dst_reg, off);
 }
 
-static void emit_ldx_index(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, u32 index_reg, int off)
+static void emit_ldx_index(struct jit_emit_context *jit, u32 size, u32 dst_reg, u32 src_reg,
+			   u32 index_reg, int off)
 {
-	u8 *prog = *pprog;
-
 	switch (size) {
 	case BPF_B:
 		/* movzx rax, byte ptr [rax + r12 + off] */
@@ -1185,14 +1123,12 @@ static void emit_ldx_index(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, u32 i
 		EMIT2(add_3mod(0x48, src_reg, dst_reg, index_reg), 0x8B);
 		break;
 	}
-	emit_insn_suffix_SIB(&prog, src_reg, dst_reg, index_reg, off);
-	*pprog = prog;
+	emit_insn_suffix_SIB(jit, src_reg, dst_reg, index_reg, off);
 }
 
-static void emit_ldsx_index(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, u32 index_reg, int off)
+static void emit_ldsx_index(struct jit_emit_context *jit, u32 size, u32 dst_reg, u32 src_reg,
+			    u32 index_reg, int off)
 {
-	u8 *prog = *pprog;
-
 	switch (size) {
 	case BPF_B:
 		/* movsx rax, byte ptr [rax + r12 + off] */
@@ -1207,25 +1143,22 @@ static void emit_ldsx_index(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, u32
 		EMIT2(add_3mod(0x48, src_reg, dst_reg, index_reg), 0x63);
 		break;
 	}
-	emit_insn_suffix_SIB(&prog, src_reg, dst_reg, index_reg, off);
-	*pprog = prog;
+	emit_insn_suffix_SIB(jit, src_reg, dst_reg, index_reg, off);
 }
 
-static void emit_ldx_r12(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, int off)
+static void emit_ldx_r12(struct jit_emit_context *jit, u32 size, u32 dst_reg, u32 src_reg, int off)
 {
-	emit_ldx_index(pprog, size, dst_reg, src_reg, X86_REG_R12, off);
+	emit_ldx_index(jit, size, dst_reg, src_reg, X86_REG_R12, off);
 }
 
-static void emit_ldsx_r12(u8 **prog, u32 size, u32 dst_reg, u32 src_reg, int off)
+static void emit_ldsx_r12(struct jit_emit_context *jit, u32 size, u32 dst_reg, u32 src_reg, int off)
 {
-	emit_ldsx_index(prog, size, dst_reg, src_reg, X86_REG_R12, off);
+	emit_ldsx_index(jit, size, dst_reg, src_reg, X86_REG_R12, off);
 }
 
 /* STX: *(u8*)(dst_reg + off) = src_reg */
-static void emit_stx(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, int off)
+static void emit_stx(struct jit_emit_context *jit, u32 size, u32 dst_reg, u32 src_reg, int off)
 {
-	u8 *prog = *pprog;
-
 	switch (size) {
 	case BPF_B:
 		/* Emit 'mov byte ptr [rax + off], al' */
@@ -1251,15 +1184,13 @@ static void emit_stx(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, int off)
 		EMIT2(add_2mod(0x48, dst_reg, src_reg), 0x89);
 		break;
 	}
-	emit_insn_suffix(&prog, dst_reg, src_reg, off);
-	*pprog = prog;
+	emit_insn_suffix(jit, dst_reg, src_reg, off);
 }
 
 /* STX: *(u8*)(dst_reg + index_reg + off) = src_reg */
-static void emit_stx_index(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, u32 index_reg, int off)
+static void emit_stx_index(struct jit_emit_context *jit, u32 size, u32 dst_reg, u32 src_reg,
+			   u32 index_reg, int off)
 {
-	u8 *prog = *pprog;
-
 	switch (size) {
 	case BPF_B:
 		/* mov byte ptr [rax + r12 + off], al */
@@ -1278,20 +1209,18 @@ static void emit_stx_index(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, u32 i
 		EMIT2(add_3mod(0x48, dst_reg, src_reg, index_reg), 0x89);
 		break;
 	}
-	emit_insn_suffix_SIB(&prog, dst_reg, src_reg, index_reg, off);
-	*pprog = prog;
+	emit_insn_suffix_SIB(jit, dst_reg, src_reg, index_reg, off);
 }
 
-static void emit_stx_r12(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, int off)
+static void emit_stx_r12(struct jit_emit_context *jit, u32 size, u32 dst_reg, u32 src_reg, int off)
 {
-	emit_stx_index(pprog, size, dst_reg, src_reg, X86_REG_R12, off);
+	emit_stx_index(jit, size, dst_reg, src_reg, X86_REG_R12, off);
 }
 
 /* ST: *(u8*)(dst_reg + index_reg + off) = imm32 */
-static void emit_st_index(u8 **pprog, u32 size, u32 dst_reg, u32 index_reg, int off, int imm)
+static void emit_st_index(struct jit_emit_context *jit, u32 size, u32 dst_reg, u32 index_reg,
+			  int off, int imm)
 {
-	u8 *prog = *pprog;
-
 	switch (size) {
 	case BPF_B:
 		/* mov byte ptr [rax + r12 + off], imm8 */
@@ -1310,35 +1239,32 @@ static void emit_st_index(u8 **pprog, u32 size, u32 dst_reg, u32 index_reg, int
 		EMIT2(add_3mod(0x48, dst_reg, 0, index_reg), 0xC7);
 		break;
 	}
-	emit_insn_suffix_SIB(&prog, dst_reg, 0, index_reg, off);
+	emit_insn_suffix_SIB(jit, dst_reg, 0, index_reg, off);
 	EMIT(imm, bpf_size_to_x86_bytes(size));
-	*pprog = prog;
 }
 
-static void emit_st_r12(u8 **pprog, u32 size, u32 dst_reg, int off, int imm)
+static void emit_st_r12(struct jit_emit_context *jit, u32 size, u32 dst_reg, int off, int imm)
 {
-	emit_st_index(pprog, size, dst_reg, X86_REG_R12, off, imm);
+	emit_st_index(jit, size, dst_reg, X86_REG_R12, off, imm);
 }
 
-static void emit_store_stack_imm64(u8 **pprog, int reg, int stack_off, u64 imm64)
+static void emit_store_stack_imm64(struct jit_emit_context *jit, int reg, int stack_off, u64 imm64)
 {
 	/*
 	 * mov reg, imm64
 	 * mov QWORD PTR [rbp + stack_off], reg
 	 */
-	emit_mov_imm64(pprog, reg, imm64 >> 32, (u32) imm64);
-	emit_stx(pprog, BPF_DW, BPF_REG_FP, reg, stack_off);
+	emit_mov_imm64(jit, reg, imm64 >> 32, (u32) imm64);
+	emit_stx(jit, BPF_DW, BPF_REG_FP, reg, stack_off);
 }
 
-static int emit_atomic_rmw(u8 **pprog, u32 atomic_op,
+static int emit_atomic_rmw(struct jit_emit_context *jit, u32 atomic_op,
 			   u32 dst_reg, u32 src_reg, s16 off, u8 bpf_size)
 {
-	u8 *prog = *pprog;
-
 	if (atomic_op != BPF_XCHG)
 		EMIT1(0xF0); /* lock prefix */
 
-	maybe_emit_mod(&prog, dst_reg, src_reg, bpf_size == BPF_DW);
+	maybe_emit_mod(jit, dst_reg, src_reg, bpf_size == BPF_DW);
 
 	/* emit opcode */
 	switch (atomic_op) {
@@ -1366,18 +1292,15 @@ static int emit_atomic_rmw(u8 **pprog, u32 atomic_op,
 		return -EFAULT;
 	}
 
-	emit_insn_suffix(&prog, dst_reg, src_reg, off);
+	emit_insn_suffix(jit, dst_reg, src_reg, off);
 
-	*pprog = prog;
 	return 0;
 }
 
-static int emit_atomic_rmw_index(u8 **pprog, u32 atomic_op, u32 size,
+static int emit_atomic_rmw_index(struct jit_emit_context *jit, u32 atomic_op, u32 size,
 				 u32 dst_reg, u32 src_reg, u32 index_reg,
 				 int off)
 {
-	u8 *prog = *pprog;
-
 	if (atomic_op != BPF_XCHG)
 		EMIT1(0xF0); /* lock prefix */
 
@@ -1418,22 +1341,21 @@ static int emit_atomic_rmw_index(u8 **pprog, u32 atomic_op, u32 size,
 		pr_err("bpf_jit: unknown atomic opcode %02x\n", atomic_op);
 		return -EFAULT;
 	}
-	emit_insn_suffix_SIB(&prog, dst_reg, src_reg, index_reg, off);
-	*pprog = prog;
+	emit_insn_suffix_SIB(jit, dst_reg, src_reg, index_reg, off);
 	return 0;
 }
 
-static int emit_atomic_ld_st(u8 **pprog, u32 atomic_op, u32 dst_reg,
+static int emit_atomic_ld_st(struct jit_emit_context *jit, u32 atomic_op, u32 dst_reg,
 			     u32 src_reg, s16 off, u8 bpf_size)
 {
 	switch (atomic_op) {
 	case BPF_LOAD_ACQ:
 		/* dst_reg = smp_load_acquire(src_reg + off16) */
-		emit_ldx(pprog, bpf_size, dst_reg, src_reg, off);
+		emit_ldx(jit, bpf_size, dst_reg, src_reg, off);
 		break;
 	case BPF_STORE_REL:
 		/* smp_store_release(dst_reg + off16, src_reg) */
-		emit_stx(pprog, bpf_size, dst_reg, src_reg, off);
+		emit_stx(jit, bpf_size, dst_reg, src_reg, off);
 		break;
 	default:
 		pr_err("bpf_jit: unknown atomic load/store opcode %02x\n",
@@ -1444,18 +1366,18 @@ static int emit_atomic_ld_st(u8 **pprog, u32 atomic_op, u32 dst_reg,
 	return 0;
 }
 
-static int emit_atomic_ld_st_index(u8 **pprog, u32 atomic_op, u32 size,
+static int emit_atomic_ld_st_index(struct jit_emit_context *jit, u32 atomic_op, u32 size,
 				   u32 dst_reg, u32 src_reg, u32 index_reg,
 				   int off)
 {
 	switch (atomic_op) {
 	case BPF_LOAD_ACQ:
 		/* dst_reg = smp_load_acquire(src_reg + idx_reg + off16) */
-		emit_ldx_index(pprog, size, dst_reg, src_reg, index_reg, off);
+		emit_ldx_index(jit, size, dst_reg, src_reg, index_reg, off);
 		break;
 	case BPF_STORE_REL:
 		/* smp_store_release(dst_reg + idx_reg + off16, src_reg) */
-		emit_stx_index(pprog, size, dst_reg, src_reg, index_reg, off);
+		emit_stx_index(jit, size, dst_reg, src_reg, index_reg, off);
 		break;
 	default:
 		pr_err("bpf_jit: unknown atomic load/store opcode %02x\n",
@@ -1562,10 +1484,9 @@ static void detect_reg_usage(struct bpf_insn *insn, int insn_cnt,
  * l: vector length (128 bit or 256 bit) or reserved
  * pp: opcode prefix (none, 0x66, 0xf2 or 0xf3)
  */
-static void emit_3vex(u8 **pprog, bool r, bool x, bool b, u8 m,
+static void emit_3vex(struct jit_emit_context *jit, bool r, bool x, bool b, u8 m,
 		      bool w, u8 src_reg2, bool l, u8 pp)
 {
-	u8 *prog = *pprog;
 	const u8 b0 = 0xc4; /* first byte of 3-byte VEX prefix */
 	u8 b1, b2;
 	u8 vvvv = reg2hex[src_reg2];
@@ -1595,27 +1516,22 @@ static void emit_3vex(u8 **pprog, bool r, bool x, bool b, u8 m,
 	b2 = (w << 7) | ((~vvvv & 0xf) << 3) | (l << 2) | (pp & 3);
 
 	EMIT3(b0, b1, b2);
-	*pprog = prog;
 }
 
 /* emit BMI2 shift instruction */
-static void emit_shiftx(u8 **pprog, u32 dst_reg, u8 src_reg, bool is64, u8 op)
+static void emit_shiftx(struct jit_emit_context *jit, u32 dst_reg, u8 src_reg, bool is64, u8 op)
 {
-	u8 *prog = *pprog;
 	bool r = is_ereg(dst_reg);
 	u8 m = 2; /* escape code 0f38 */
 
-	emit_3vex(&prog, r, false, r, m, is64, src_reg, false, op);
+	emit_3vex(jit, r, false, r, m, is64, src_reg, false, op);
 	EMIT2(0xf7, add_2reg(0xC0, dst_reg, dst_reg));
-	*pprog = prog;
 }
 
-static void emit_priv_frame_ptr(u8 **pprog, void __percpu *priv_frame_ptr)
+static void emit_priv_frame_ptr(struct jit_emit_context *jit, void __percpu *priv_frame_ptr)
 {
-	u8 *prog = *pprog;
-
 	/* movabs r9, priv_frame_ptr */
-	emit_mov_imm64(&prog, X86_REG_R9, (__force long) priv_frame_ptr >> 32,
+	emit_mov_imm64(jit, X86_REG_R9, (__force long) priv_frame_ptr >> 32,
 		       (u32) (__force long) priv_frame_ptr);
 
 #ifdef CONFIG_SMP
@@ -1624,11 +1540,9 @@ static void emit_priv_frame_ptr(u8 **pprog, void __percpu *priv_frame_ptr)
 	EMIT3(0x03, 0x0c, 0x25);
 	EMIT((u32)(unsigned long)&this_cpu_off, 4);
 #endif
-
-	*pprog = prog;
 }
 
-#define INSN_SZ_DIFF (((addrs[i] - addrs[i - 1]) - (prog - temp)))
+#define INSN_SZ_DIFF (((addrs[i] - addrs[i - 1]) - (jit->prog - temp)))
 
 #define __LOAD_TCC_PTR(off)			\
 	EMIT3_off32(0x48, 0x8B, 0x85, off)
@@ -1640,10 +1554,9 @@ static void emit_priv_frame_ptr(u8 **pprog, void __percpu *priv_frame_ptr)
 #define PRIV_STACK_GUARD_SZ    8
 #define PRIV_STACK_GUARD_VAL   0xEB9F12345678eb9fULL
 
-static int emit_spectre_bhb_barrier(u8 **pprog, u8 *ip,
+static int emit_spectre_bhb_barrier(struct jit_emit_context *jit, u8 *ip,
 				    struct bpf_prog *bpf_prog)
 {
-	u8 *prog = *pprog;
 	u8 *func;
 
 	if (cpu_feature_enabled(X86_FEATURE_CLEAR_BHB_LOOP)) {
@@ -1653,9 +1566,9 @@ static int emit_spectre_bhb_barrier(u8 **pprog, u8 *ip,
 		ip += 2;
 
 		func = (u8 *)clear_bhb_loop;
-		ip += x86_call_depth_emit_accounting(&prog, func, ip);
+		ip += bpf_call_depth_emit_accounting(jit, func, ip);
 
-		if (emit_call(&prog, func, ip))
+		if (emit_call(jit, func, ip))
 			return -EINVAL;
 		EMIT1(0x59); /* pop rcx */
 		EMIT1(0x58); /* pop rax */
@@ -1678,7 +1591,6 @@ static int emit_spectre_bhb_barrier(u8 **pprog, u8 *ip,
 		 */
 		EMIT5(0xF3, 0x48, 0x0F, 0x1E, 0xF8); /* ibhf */
 	}
-	*pprog = prog;
 	return 0;
 }
 
@@ -1689,10 +1601,9 @@ static int emit_spectre_bhb_barrier(u8 **pprog, u8 *ip,
  * arena NULL is offset 0. Return the number of emitted bytes.
  */
 static int emit_kfunc_arena_args(struct bpf_prog *bpf_prog,
-				 const struct btf_func_model *fm, u8 **pprog)
+				 const struct btf_func_model *fm, struct jit_emit_context *jit)
 {
-	u8 *prog = *pprog;
-	u8 *start = prog;
+	u8 *start = jit->prog;
 	int i;
 
 	for (i = 0; i < min_t(int, fm->nr_args, MAX_BPF_FUNC_REG_ARGS); i++) {
@@ -1705,20 +1616,19 @@ static int emit_kfunc_arena_args(struct bpf_prog *bpf_prog,
 			return -EINVAL;
 
 		/* mov eN, eN: truncate and clear the upper 32 bits */
-		emit_mov_reg(&prog, false, reg, reg);
+		emit_mov_reg(jit, false, reg, reg);
 		if (flags & BTF_FMODEL_NULLABLE_ARG) {
 			/* test eN, eN; jz over the 3-byte add */
-			maybe_emit_mod(&prog, reg, reg, false);
+			maybe_emit_mod(jit, reg, reg, false);
 			EMIT2(0x85, add_2reg(0xC0, reg, reg));
 			EMIT2(X86_JE, 3);
 		}
 		/* add rN, r12 */
-		maybe_emit_mod(&prog, reg, X86_REG_R12, true);
+		maybe_emit_mod(jit, reg, X86_REG_R12, true);
 		EMIT2(0x01, add_2reg(0xC0, reg, X86_REG_R12));
 	}
 
-	*pprog = prog;
-	return prog - start;
+	return jit->prog - start;
 }
 
 static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *addrs, u8 *image,
@@ -1726,6 +1636,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 {
 	bool tail_call_reachable = bpf_prog->aux->tail_call_reachable;
 	struct bpf_insn *insn = bpf_prog->insnsi;
+	struct jit_emit_context *jit = &ctx->jit;
 	bool callee_regs_used[4] = {};
 	int insn_cnt = bpf_prog->len;
 	bool seen_exit = false;
@@ -1736,12 +1647,15 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 	void __percpu *priv_stack_ptr;
 	int i, excnt = 0;
 	int ilen, proglen = 0;
-	u8 *ip, *prog = temp;
+	u8 *ip;
 	u32 stack_depth;
 	int callee_saved_size;
 	s32 outgoing_arg_base;
 	int err;
 
+	jit->prog = temp;
+	jit->dry_run = false;
+
 	stack_depth = bpf_prog->aux->stack_depth;
 	out_stack_arg_cnt = bpf_out_stack_arg_cnt(env, bpf_prog);
 	priv_stack_ptr = bpf_prog->aux->priv_stack_ptr;
@@ -1777,11 +1691,11 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 
 	detect_reg_usage(insn, insn_cnt, callee_regs_used);
 
-	emit_prologue(&prog, image, stack_depth,
+	emit_prologue(jit, image, stack_depth,
 		      bpf_prog_was_classic(bpf_prog), tail_call_reachable,
 		      bpf_is_subprog(bpf_prog), bpf_prog->aux->exception_cb);
 
-	bpf_prog->aux->ksym.fp_start = prog - temp;
+	bpf_prog->aux->ksym.fp_start = jit->prog - temp;
 
 	/* Exception callback will clobber callee regs for its own use, and
 	 * restore the original callee regs from main prog's stack frame.
@@ -1791,12 +1705,12 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 		 * register, as we throw after entry into the kernel, which may
 		 * overwrite r12.
 		 */
-		push_r12(&prog);
-		push_callee_regs(&prog, all_callee_regs_used);
+		push_r12(jit);
+		push_callee_regs(jit, all_callee_regs_used);
 	} else {
 		if (arena_vm_start)
-			push_r12(&prog);
-		push_callee_regs(&prog, callee_regs_used);
+			push_r12(jit);
+		push_callee_regs(jit, callee_regs_used);
 	}
 
 	/* Compute callee-saved register area size. */
@@ -1834,21 +1748,21 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 	outgoing_rsp = out_stack_arg_cnt > 1 ? (out_stack_arg_cnt - 1) * 8 : 0;
 	if (bpf_prog->aux->exception_boundary)
 		bpf_prog->aux->stack_arg_sp_adjust = outgoing_rsp;
-	emit_sub_rsp(&prog, outgoing_rsp);
+	emit_sub_rsp(jit, outgoing_rsp);
 
 	if (arena_vm_start)
-		emit_mov_imm64(&prog, X86_REG_R12,
+		emit_mov_imm64(jit, X86_REG_R12,
 			       arena_vm_start >> 32, (u32) arena_vm_start);
 
 	if (priv_frame_ptr)
-		emit_priv_frame_ptr(&prog, priv_frame_ptr);
+		emit_priv_frame_ptr(jit, priv_frame_ptr);
 
-	ilen = prog - temp;
+	ilen = jit->prog - temp;
 	if (rw_image)
 		memcpy(rw_image + proglen, temp, ilen);
 	proglen += ilen;
 	addrs[0] = proglen;
-	prog = temp;
+	jit->prog = temp;
 
 	for (i = 1; i <= insn_cnt; i++, insn++) {
 		const s32 imm32 = insn->imm;
@@ -1873,7 +1787,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 		if (bpf_insn_is_indirect_target(env, bpf_prog, i - 1))
 			EMIT_ENDBR();
 
-		ip = image + addrs[i - 1] + (prog - temp);
+		ip = image + addrs[i - 1] + (jit->prog - temp);
 
 		switch (insn->code) {
 			/* ALU */
@@ -1887,7 +1801,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 		case BPF_ALU64 | BPF_AND | BPF_X:
 		case BPF_ALU64 | BPF_OR | BPF_X:
 		case BPF_ALU64 | BPF_XOR | BPF_X:
-			maybe_emit_mod(&prog, dst_reg, src_reg,
+			maybe_emit_mod(jit, dst_reg, src_reg,
 				       BPF_CLASS(insn->code) == BPF_ALU64);
 			b2 = simple_alu_opcodes[BPF_OP(insn->code)];
 			EMIT2(b2, add_2reg(0xC0, dst_reg, src_reg));
@@ -1897,32 +1811,32 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 			if (insn_is_cast_user(insn)) {
 				if (dst_reg != src_reg)
 					/* 32-bit mov */
-					emit_mov_reg(&prog, false, dst_reg, src_reg);
+					emit_mov_reg(jit, false, dst_reg, src_reg);
 				/* shl dst_reg, 32 */
-				maybe_emit_1mod(&prog, dst_reg, true);
+				maybe_emit_1mod(jit, dst_reg, true);
 				EMIT3(0xC1, add_1reg(0xE0, dst_reg), 32);
 
 				/* or dst_reg, user_vm_start */
-				maybe_emit_1mod(&prog, dst_reg, true);
+				maybe_emit_1mod(jit, dst_reg, true);
 				if (is_axreg(dst_reg))
 					EMIT1_off32(0x0D,  user_vm_start >> 32);
 				else
 					EMIT2_off32(0x81, add_1reg(0xC8, dst_reg),  user_vm_start >> 32);
 
 				/* rol dst_reg, 32 */
-				maybe_emit_1mod(&prog, dst_reg, true);
+				maybe_emit_1mod(jit, dst_reg, true);
 				EMIT3(0xC1, add_1reg(0xC0, dst_reg), 32);
 
 				/* xor r11, r11 */
 				EMIT3(0x4D, 0x31, 0xDB);
 
 				/* test dst_reg32, dst_reg32; check if lower 32-bit are zero */
-				maybe_emit_mod(&prog, dst_reg, dst_reg, false);
+				maybe_emit_mod(jit, dst_reg, dst_reg, false);
 				EMIT2(0x85, add_2reg(0xC0, dst_reg, dst_reg));
 
 				/* cmove r11, dst_reg; if so, set dst_reg to zero */
 				/* WARNING: Intel swapped src/dst register encoding in CMOVcc !!! */
-				maybe_emit_mod(&prog, AUX_REG, dst_reg, true);
+				maybe_emit_mod(jit, AUX_REG, dst_reg, true);
 				EMIT3(0x0F, 0x44, add_2reg(0xC0, AUX_REG, dst_reg));
 				break;
 			} else if (insn_is_mov_percpu_addr(insn)) {
@@ -1939,11 +1853,11 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 			fallthrough;
 		case BPF_ALU | BPF_MOV | BPF_X:
 			if (insn->off == 0)
-				emit_mov_reg(&prog,
+				emit_mov_reg(jit,
 					     BPF_CLASS(insn->code) == BPF_ALU64,
 					     dst_reg, src_reg);
 			else
-				emit_movsx_reg(&prog, insn->off,
+				emit_movsx_reg(jit, insn->off,
 					       BPF_CLASS(insn->code) == BPF_ALU64,
 					       dst_reg, src_reg);
 			break;
@@ -1951,7 +1865,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 			/* neg dst */
 		case BPF_ALU | BPF_NEG:
 		case BPF_ALU64 | BPF_NEG:
-			maybe_emit_1mod(&prog, dst_reg,
+			maybe_emit_1mod(jit, dst_reg,
 					BPF_CLASS(insn->code) == BPF_ALU64);
 			EMIT2(0xF7, add_1reg(0xD8, dst_reg));
 			break;
@@ -1966,7 +1880,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 		case BPF_ALU64 | BPF_AND | BPF_K:
 		case BPF_ALU64 | BPF_OR | BPF_K:
 		case BPF_ALU64 | BPF_XOR | BPF_K:
-			maybe_emit_1mod(&prog, dst_reg,
+			maybe_emit_1mod(jit, dst_reg,
 					BPF_CLASS(insn->code) == BPF_ALU64);
 
 			/*
@@ -2006,12 +1920,12 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 
 		case BPF_ALU64 | BPF_MOV | BPF_K:
 		case BPF_ALU | BPF_MOV | BPF_K:
-			emit_mov_imm32(&prog, BPF_CLASS(insn->code) == BPF_ALU64,
+			emit_mov_imm32(jit, BPF_CLASS(insn->code) == BPF_ALU64,
 				       dst_reg, imm32);
 			break;
 
 		case BPF_LD | BPF_IMM | BPF_DW:
-			emit_mov_imm64(&prog, dst_reg, insn[1].imm, insn[0].imm);
+			emit_mov_imm64(jit, dst_reg, insn[1].imm, insn[0].imm);
 			insn++;
 			i++;
 			break;
@@ -2047,7 +1961,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 
 			if (dst_reg != BPF_REG_0)
 				/* mov rax, dst_reg */
-				emit_mov_reg(&prog, is64, BPF_REG_0, dst_reg);
+				emit_mov_reg(jit, is64, BPF_REG_0, dst_reg);
 
 			if (insn->off == 0) {
 				/*
@@ -2057,7 +1971,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 				EMIT2(0x31, 0xd2);
 
 				/* div src_reg */
-				maybe_emit_1mod(&prog, src_reg, is64);
+				maybe_emit_1mod(jit, src_reg, is64);
 				EMIT2(0xF7, add_1reg(0xF0, src_reg));
 			} else {
 				if (BPF_CLASS(insn->code) == BPF_ALU)
@@ -2066,18 +1980,18 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 					EMIT2(0x48, 0x99); /* cqo */
 
 				/* idiv src_reg */
-				maybe_emit_1mod(&prog, src_reg, is64);
+				maybe_emit_1mod(jit, src_reg, is64);
 				EMIT2(0xF7, add_1reg(0xF8, src_reg));
 			}
 
 			if (BPF_OP(insn->code) == BPF_MOD &&
 			    dst_reg != BPF_REG_3)
 				/* mov dst_reg, rdx */
-				emit_mov_reg(&prog, is64, dst_reg, BPF_REG_3);
+				emit_mov_reg(jit, is64, dst_reg, BPF_REG_3);
 			else if (BPF_OP(insn->code) == BPF_DIV &&
 				 dst_reg != BPF_REG_0)
 				/* mov dst_reg, rax */
-				emit_mov_reg(&prog, is64, dst_reg, BPF_REG_0);
+				emit_mov_reg(jit, is64, dst_reg, BPF_REG_0);
 
 			if (dst_reg != BPF_REG_3)
 				EMIT1(0x5A); /* pop rdx */
@@ -2088,7 +2002,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 
 		case BPF_ALU | BPF_MUL | BPF_K:
 		case BPF_ALU64 | BPF_MUL | BPF_K:
-			maybe_emit_mod(&prog, dst_reg, dst_reg,
+			maybe_emit_mod(jit, dst_reg, dst_reg,
 				       BPF_CLASS(insn->code) == BPF_ALU64);
 
 			if (is_imm8(imm32))
@@ -2104,7 +2018,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 
 		case BPF_ALU | BPF_MUL | BPF_X:
 		case BPF_ALU64 | BPF_MUL | BPF_X:
-			maybe_emit_mod(&prog, src_reg, dst_reg,
+			maybe_emit_mod(jit, src_reg, dst_reg,
 				       BPF_CLASS(insn->code) == BPF_ALU64);
 
 			/* imul dst_reg, src_reg */
@@ -2118,7 +2032,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 		case BPF_ALU64 | BPF_LSH | BPF_K:
 		case BPF_ALU64 | BPF_RSH | BPF_K:
 		case BPF_ALU64 | BPF_ARSH | BPF_K:
-			maybe_emit_1mod(&prog, dst_reg,
+			maybe_emit_1mod(jit, dst_reg,
 					BPF_CLASS(insn->code) == BPF_ALU64);
 
 			b3 = simple_alu_opcodes[BPF_OP(insn->code)];
@@ -2152,7 +2066,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 					break;
 				}
 
-				emit_shiftx(&prog, dst_reg, src_reg, w, op);
+				emit_shiftx(jit, dst_reg, src_reg, w, op);
 
 				break;
 			}
@@ -2171,7 +2085,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 			}
 
 			/* shl %rax, %cl | shr %rax, %cl | sar %rax, %cl */
-			maybe_emit_1mod(&prog, dst_reg,
+			maybe_emit_1mod(jit, dst_reg,
 					BPF_CLASS(insn->code) == BPF_ALU64);
 
 			b3 = simple_alu_opcodes[BPF_OP(insn->code)];
@@ -2272,7 +2186,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 		case BPF_ST | BPF_MEM | BPF_DW:
 			if (dst_reg == BPF_REG_PARAMS && insn->off == -8) {
 				/* Arg 6: store immediate in r9 register */
-				emit_mov_imm64(&prog, X86_REG_R9, imm32 >> 31, (u32)imm32);
+				emit_mov_imm64(jit, X86_REG_R9, imm32 >> 31, (u32)imm32);
 				break;
 			}
 			EMIT2(add_1mod(0x48, dst_reg), 0xC7);
@@ -2310,15 +2224,15 @@ st:			insn_off = insn->off;
 				insn_off = outgoing_arg_base - outgoing_rsp - insn_off - 16;
 				dst_reg = BPF_REG_FP;
 			}
-			emit_stx(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn_off);
+			emit_stx(jit, BPF_SIZE(insn->code), dst_reg, src_reg, insn_off);
 			break;
 
 		case BPF_ST | BPF_PROBE_MEM32 | BPF_B:
 		case BPF_ST | BPF_PROBE_MEM32 | BPF_H:
 		case BPF_ST | BPF_PROBE_MEM32 | BPF_W:
 		case BPF_ST | BPF_PROBE_MEM32 | BPF_DW:
-			start_of_ldx = prog;
-			emit_st_r12(&prog, BPF_SIZE(insn->code), dst_reg, insn->off, insn->imm);
+			start_of_ldx = jit->prog;
+			emit_st_r12(jit, BPF_SIZE(insn->code), dst_reg, insn->off, insn->imm);
 			goto populate_extable;
 
 			/* LDX: dst_reg = *(u8*)(src_reg + r12 + off) */
@@ -2333,14 +2247,14 @@ st:			insn_off = insn->off;
 		case BPF_STX | BPF_PROBE_MEM32 | BPF_H:
 		case BPF_STX | BPF_PROBE_MEM32 | BPF_W:
 		case BPF_STX | BPF_PROBE_MEM32 | BPF_DW:
-			start_of_ldx = prog;
+			start_of_ldx = jit->prog;
 			if (BPF_CLASS(insn->code) == BPF_LDX) {
 				if (BPF_MODE(insn->code) == BPF_PROBE_MEM32SX)
-					emit_ldsx_r12(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn->off);
+					emit_ldsx_r12(jit, BPF_SIZE(insn->code), dst_reg, src_reg, insn->off);
 				else
-					emit_ldx_r12(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn->off);
+					emit_ldx_r12(jit, BPF_SIZE(insn->code), dst_reg, src_reg, insn->off);
 			} else {
-				emit_stx_r12(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn->off);
+				emit_stx_r12(jit, BPF_SIZE(insn->code), dst_reg, src_reg, insn->off);
 			}
 populate_extable:
 			{
@@ -2401,7 +2315,7 @@ st:			insn_off = insn->off;
 					is_write = true;
 				}
 
-				ex->fixup = FIELD_PREP(FIXUP_INSN_LEN_MASK, prog - start_of_ldx) |
+				ex->fixup = FIELD_PREP(FIXUP_INSN_LEN_MASK, jit->prog - start_of_ldx) |
 					    FIELD_PREP(FIXUP_ARENA_REG_MASK, arena_reg) |
 					    FIELD_PREP(FIXUP_REG_MASK, fixup_reg);
 				ex->fixup |= FIXUP_ARENA_ACCESS;
@@ -2455,7 +2369,7 @@ st:			insn_off = insn->off;
 				u8 *end_of_jmp;
 
 				/* movabsq r10, VSYSCALL_ADDR */
-				emit_mov_imm64(&prog, BPF_REG_AX, (long)VSYSCALL_ADDR >> 32,
+				emit_mov_imm64(jit, BPF_REG_AX, (long)VSYSCALL_ADDR >> 32,
 					       (u32)(long)VSYSCALL_ADDR);
 
 				/* mov src_reg, r11 */
@@ -2463,40 +2377,40 @@ st:			insn_off = insn->off;
 
 				if (insn->off) {
 					/* add r11, insn->off */
-					maybe_emit_1mod(&prog, AUX_REG, true);
+					maybe_emit_1mod(jit, AUX_REG, true);
 					EMIT2_off32(0x81, add_1reg(0xC0, AUX_REG), insn->off);
 				}
 
 				/* sub r11, r10 */
-				maybe_emit_mod(&prog, AUX_REG, BPF_REG_AX, true);
+				maybe_emit_mod(jit, AUX_REG, BPF_REG_AX, true);
 				EMIT2(0x29, add_2reg(0xC0, AUX_REG, BPF_REG_AX));
 
 				/* movabsq r10, limit */
-				emit_mov_imm64(&prog, BPF_REG_AX, (long)limit >> 32,
+				emit_mov_imm64(jit, BPF_REG_AX, (long)limit >> 32,
 					       (u32)(long)limit);
 
 				/* cmp r10, r11 */
-				maybe_emit_mod(&prog, AUX_REG, BPF_REG_AX, true);
+				maybe_emit_mod(jit, AUX_REG, BPF_REG_AX, true);
 				EMIT2(0x39, add_2reg(0xC0, AUX_REG, BPF_REG_AX));
 
 				/* if unsigned '>', goto load */
 				EMIT2(X86_JA, 0);
-				end_of_jmp = prog;
+				end_of_jmp = jit->prog;
 
 				/* xor dst_reg, dst_reg */
-				emit_mov_imm32(&prog, false, dst_reg, 0);
+				emit_mov_imm32(jit, false, dst_reg, 0);
 				/* jmp byte_after_ldx */
 				EMIT2(0xEB, 0);
 
 				/* populate jmp_offset for JAE above to jump to start_of_ldx */
-				start_of_ldx = prog;
+				start_of_ldx = jit->prog;
 				end_of_jmp[-1] = start_of_ldx - end_of_jmp;
 			}
 			if (BPF_MODE(insn->code) == BPF_PROBE_MEMSX ||
 			    BPF_MODE(insn->code) == BPF_MEMSX)
-				emit_ldsx(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn_off);
+				emit_ldsx(jit, BPF_SIZE(insn->code), dst_reg, src_reg, insn_off);
 			else
-				emit_ldx(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn_off);
+				emit_ldx(jit, BPF_SIZE(insn->code), dst_reg, src_reg, insn_off);
 			if (BPF_MODE(insn->code) == BPF_PROBE_MEM ||
 			    BPF_MODE(insn->code) == BPF_PROBE_MEMSX) {
 				struct exception_table_entry *ex;
@@ -2504,7 +2418,7 @@ st:			insn_off = insn->off;
 				s64 delta;
 
 				/* populate jmp_offset for JMP above */
-				start_of_ldx[-1] = prog - start_of_ldx;
+				start_of_ldx[-1] = jit->prog - start_of_ldx;
 
 				if (!bpf_prog->aux->extable)
 					break;
@@ -2539,7 +2453,7 @@ st:			insn_off = insn->off;
 				 * End result: x86 insn "mov rbx, qword ptr [rax+0x14]"
 				 * of 4 bytes will be ignored and rbx will be zero inited.
 				 */
-				ex->fixup = FIELD_PREP(FIXUP_INSN_LEN_MASK, prog - start_of_ldx) |
+				ex->fixup = FIELD_PREP(FIXUP_INSN_LEN_MASK, jit->prog - start_of_ldx) |
 					    FIELD_PREP(FIXUP_REG_MASK, reg2pt_regs[dst_reg]);
 			}
 			break;
@@ -2567,26 +2481,26 @@ st:			insn_off = insn->off;
 				 */
 
 				/* Will need RAX as a CMPXCHG operand so save R0 */
-				emit_mov_reg(&prog, true, BPF_REG_AX, BPF_REG_0);
+				emit_mov_reg(jit, true, BPF_REG_AX, BPF_REG_0);
 				if (src_reg == BPF_REG_0)
 					real_src_reg = BPF_REG_AX;
 				if (dst_reg == BPF_REG_0)
 					real_dst_reg = BPF_REG_AX;
 
-				branch_target = prog;
+				branch_target = jit->prog;
 				/* Load old value */
-				emit_ldx(&prog, BPF_SIZE(insn->code),
+				emit_ldx(jit, BPF_SIZE(insn->code),
 					 BPF_REG_0, real_dst_reg, insn->off);
 				/*
 				 * Perform the (commutative) operation locally,
 				 * put the result in the AUX_REG.
 				 */
-				emit_mov_reg(&prog, is64, AUX_REG, BPF_REG_0);
-				maybe_emit_mod(&prog, AUX_REG, real_src_reg, is64);
+				emit_mov_reg(jit, is64, AUX_REG, BPF_REG_0);
+				maybe_emit_mod(jit, AUX_REG, real_src_reg, is64);
 				EMIT2(simple_alu_opcodes[BPF_OP(insn->imm)],
 				      add_2reg(0xC0, AUX_REG, real_src_reg));
 				/* Attempt to swap in new value */
-				err = emit_atomic_rmw(&prog, BPF_CMPXCHG,
+				err = emit_atomic_rmw(jit, BPF_CMPXCHG,
 						      real_dst_reg, AUX_REG,
 						      insn->off,
 						      BPF_SIZE(insn->code));
@@ -2596,19 +2510,19 @@ st:			insn_off = insn->off;
 				 * ZF tells us whether we won the race. If it's
 				 * cleared we need to try again.
 				 */
-				EMIT2(X86_JNE, -(prog - branch_target) - 2);
+				EMIT2(X86_JNE, -(jit->prog - branch_target) - 2);
 				/* Return the pre-modification value */
-				emit_mov_reg(&prog, is64, real_src_reg, BPF_REG_0);
+				emit_mov_reg(jit, is64, real_src_reg, BPF_REG_0);
 				/* Restore R0 after clobbering RAX */
-				emit_mov_reg(&prog, true, BPF_REG_0, BPF_REG_AX);
+				emit_mov_reg(jit, true, BPF_REG_0, BPF_REG_AX);
 				break;
 			}
 
 			if (bpf_atomic_is_load_store(insn))
-				err = emit_atomic_ld_st(&prog, insn->imm, dst_reg, src_reg,
+				err = emit_atomic_ld_st(jit, insn->imm, dst_reg, src_reg,
 							insn->off, BPF_SIZE(insn->code));
 			else
-				err = emit_atomic_rmw(&prog, insn->imm, dst_reg, src_reg,
+				err = emit_atomic_rmw(jit, insn->imm, dst_reg, src_reg,
 						      insn->off, BPF_SIZE(insn->code));
 			if (err)
 				return err;
@@ -2623,14 +2537,14 @@ st:			insn_off = insn->off;
 			fallthrough;
 		case BPF_STX | BPF_PROBE_ATOMIC | BPF_W:
 		case BPF_STX | BPF_PROBE_ATOMIC | BPF_DW:
-			start_of_ldx = prog;
+			start_of_ldx = jit->prog;
 
 			if (bpf_atomic_is_load_store(insn))
-				err = emit_atomic_ld_st_index(&prog, insn->imm,
+				err = emit_atomic_ld_st_index(jit, insn->imm,
 							      BPF_SIZE(insn->code), dst_reg,
 							      src_reg, X86_REG_R12, insn->off);
 			else
-				err = emit_atomic_rmw_index(&prog, insn->imm, BPF_SIZE(insn->code),
+				err = emit_atomic_rmw_index(jit, insn->imm, BPF_SIZE(insn->code),
 							    dst_reg, src_reg, X86_REG_R12,
 							    insn->off);
 			if (err)
@@ -2652,20 +2566,20 @@ st:			insn_off = insn->off;
 				fm = bpf_jit_find_kfunc_model(bpf_prog, insn);
 				if (!fm)
 					return -EINVAL;
-				err = emit_kfunc_arena_args(bpf_prog, fm, &prog);
+				err = emit_kfunc_arena_args(bpf_prog, fm, jit);
 				if (err < 0)
 					return err;
 				ip += err;
 			}
 			if (priv_frame_ptr) {
-				push_r9(&prog);
+				push_r9(jit);
 				ip += 2;
 			}
-			ip += x86_call_depth_emit_accounting(&prog, func, ip);
-			if (emit_call(&prog, func, ip))
+			ip += bpf_call_depth_emit_accounting(jit, func, ip);
+			if (emit_call(jit, func, ip))
 				return -EINVAL;
 			if (priv_frame_ptr)
-				pop_r9(&prog);
+				pop_r9(jit);
 			/*
 			 * A kfunc returning more than 8 bytes hands the second
 			 * half back in RDX (the native ABI's second return reg),
@@ -2673,7 +2587,7 @@ st:			insn_off = insn->off;
 			 * needed), while BPF R2 is RSI, so copy RDX into RSI.
 			 */
 			if (fm && fm->ret_size > 8)
-				emit_mov_reg(&prog, true, BPF_REG_2, BPF_REG_3);
+				emit_mov_reg(jit, true, BPF_REG_2, BPF_REG_3);
 			break;
 		}
 
@@ -2681,14 +2595,12 @@ st:			insn_off = insn->off;
 			if (imm32)
 				emit_bpf_tail_call_direct(bpf_prog,
 							  &bpf_prog->aux->poke_tab[imm32 - 1],
-							  &prog,
 							  ip,
 							  callee_regs_used,
 							  stack_depth,
 							  ctx);
 			else
 				emit_bpf_tail_call_indirect(bpf_prog,
-							    &prog,
 							    callee_regs_used,
 							    stack_depth,
 							    ip,
@@ -2717,7 +2629,7 @@ st:			insn_off = insn->off;
 		case BPF_JMP32 | BPF_JSGE | BPF_X:
 		case BPF_JMP32 | BPF_JSLE | BPF_X:
 			/* cmp dst_reg, src_reg */
-			maybe_emit_mod(&prog, dst_reg, src_reg,
+			maybe_emit_mod(jit, dst_reg, src_reg,
 				       BPF_CLASS(insn->code) == BPF_JMP);
 			EMIT2(0x39, add_2reg(0xC0, dst_reg, src_reg));
 			goto emit_cond_jmp;
@@ -2725,7 +2637,7 @@ st:			insn_off = insn->off;
 		case BPF_JMP | BPF_JSET | BPF_X:
 		case BPF_JMP32 | BPF_JSET | BPF_X:
 			/* test dst_reg, src_reg */
-			maybe_emit_mod(&prog, dst_reg, src_reg,
+			maybe_emit_mod(jit, dst_reg, src_reg,
 				       BPF_CLASS(insn->code) == BPF_JMP);
 			EMIT2(0x85, add_2reg(0xC0, dst_reg, src_reg));
 			goto emit_cond_jmp;
@@ -2733,7 +2645,7 @@ st:			insn_off = insn->off;
 		case BPF_JMP | BPF_JSET | BPF_K:
 		case BPF_JMP32 | BPF_JSET | BPF_K:
 			/* test dst_reg, imm32 */
-			maybe_emit_1mod(&prog, dst_reg,
+			maybe_emit_1mod(jit, dst_reg,
 					BPF_CLASS(insn->code) == BPF_JMP);
 			EMIT2_off32(0xF7, add_1reg(0xC0, dst_reg), imm32);
 			goto emit_cond_jmp;
@@ -2760,14 +2672,14 @@ st:			insn_off = insn->off;
 		case BPF_JMP32 | BPF_JSLE | BPF_K:
 			/* test dst_reg, dst_reg to save one extra byte */
 			if (imm32 == 0) {
-				maybe_emit_mod(&prog, dst_reg, dst_reg,
+				maybe_emit_mod(jit, dst_reg, dst_reg,
 					       BPF_CLASS(insn->code) == BPF_JMP);
 				EMIT2(0x85, add_2reg(0xC0, dst_reg, dst_reg));
 				goto emit_cond_jmp;
 			}
 
 			/* cmp dst_reg, imm8/32 */
-			maybe_emit_1mod(&prog, dst_reg,
+			maybe_emit_1mod(jit, dst_reg,
 					BPF_CLASS(insn->code) == BPF_JMP);
 
 			if (is_imm8(imm32))
@@ -2843,7 +2755,7 @@ st:			insn_off = insn->off;
 						       nops);
 						return -EFAULT;
 					}
-					emit_nops(&prog, nops);
+					emit_nops(jit, nops);
 				}
 				EMIT2(jmp_cond, jmp_offset);
 			} else if (is_simm32(jmp_offset)) {
@@ -2856,7 +2768,7 @@ st:			insn_off = insn->off;
 			break;
 
 		case BPF_JMP | BPF_JA | BPF_X:
-			emit_indirect_jump(&prog, insn->dst_reg, ip);
+			emit_indirect_jump(jit, insn->dst_reg, ip);
 			break;
 		case BPF_JMP | BPF_JA:
 		case BPF_JMP32 | BPF_JA:
@@ -2900,7 +2812,7 @@ st:			insn_off = insn->off;
 						       nops);
 						return -EFAULT;
 					}
-					emit_nops(&prog, nops);
+					emit_nops(jit, nops);
 				}
 				break;
 			}
@@ -2925,7 +2837,7 @@ st:			insn_off = insn->off;
 						       nops);
 						return -EFAULT;
 					}
-					emit_nops(&prog, INSN_SZ_DIFF - 2);
+					emit_nops(jit, INSN_SZ_DIFF - 2);
 				}
 				EMIT2(0xEB, jmp_offset);
 			} else if (is_simm32(jmp_offset)) {
@@ -2946,23 +2858,23 @@ st:			insn_off = insn->off;
 			ctx->cleanup_addr = proglen;
 			if (bpf_prog_was_classic(bpf_prog) &&
 			    !ns_capable_noaudit(&init_user_ns, CAP_SYS_ADMIN)) {
-				if (emit_spectre_bhb_barrier(&prog, ip, bpf_prog))
+				if (emit_spectre_bhb_barrier(jit, ip, bpf_prog))
 					return -EINVAL;
 			}
 			/* Deallocate outgoing args 7+ area. */
-			emit_add_rsp(&prog, outgoing_rsp);
+			emit_add_rsp(jit, outgoing_rsp);
 			if (bpf_prog->aux->exception_boundary) {
-				pop_callee_regs(&prog, all_callee_regs_used);
-				pop_r12(&prog);
+				pop_callee_regs(jit, all_callee_regs_used);
+				pop_r12(jit);
 			} else {
-				pop_callee_regs(&prog, callee_regs_used);
+				pop_callee_regs(jit, callee_regs_used);
 				if (arena_vm_start)
-					pop_r12(&prog);
+					pop_r12(jit);
 			}
 			EMIT1(0xC9);         /* leave */
-			bpf_prog->aux->ksym.fp_end = prog - temp;
+			bpf_prog->aux->ksym.fp_end = jit->prog - temp;
 
-			emit_return(&prog, image + addrs[i - 1] + (prog - temp));
+			emit_return(jit, image + addrs[i - 1] + (jit->prog - temp));
 			break;
 
 		default:
@@ -2976,7 +2888,7 @@ st:			insn_off = insn->off;
 			return -EINVAL;
 		}
 
-		ilen = prog - temp;
+		ilen = jit->prog - temp;
 		if (ilen > BPF_MAX_INSN_SIZE) {
 			pr_err("bpf_jit: fatal insn size error\n");
 			return -EFAULT;
@@ -3000,7 +2912,7 @@ st:			insn_off = insn->off;
 		}
 		proglen += ilen;
 		addrs[i] = proglen;
-		prog = temp;
+		jit->prog = temp;
 	}
 
 	if (image && excnt != bpf_prog->aux->num_exentries) {
@@ -3011,11 +2923,10 @@ st:			insn_off = insn->off;
 }
 
 static void clean_stack_garbage(const struct btf_func_model *m,
-				u8 **pprog, int nr_stack_slots,
+				struct jit_emit_context *jit, int nr_stack_slots,
 				int stack_size)
 {
 	int arg_size, off;
-	u8 *prog;
 
 	/* Generally speaking, the compiler will pass the arguments
 	 * on-stack with "push" instruction, which will take 8-byte
@@ -3047,14 +2958,12 @@ static void clean_stack_garbage(const struct btf_func_model *m,
 	arg_size = m->arg_size[m->nr_args - 1];
 	if (arg_size <= 4) {
 		off = -(stack_size - 4);
-		prog = *pprog;
 		/* mov DWORD PTR [rbp + off], 0 */
 		if (!is_imm8(off))
 			EMIT2_off32(0xC7, 0x85, off);
 		else
 			EMIT3(0xC7, 0x45, off);
 		EMIT(0, 4);
-		*pprog = prog;
 	}
 }
 
@@ -3082,26 +2991,23 @@ static int get_nr_used_regs(const struct btf_func_model *m)
  * subtraction both truncates and clears the upper half, so the stored
  * value satisfies the JIT invariant for arena pointer registers.
  */
-static void emit_arena_arg_conv(u8 **pprog, u32 src_reg, bool nullable, u32 base_lo)
+static void emit_arena_arg_conv(struct jit_emit_context *jit, u32 src_reg, bool nullable,
+				u32 base_lo)
 {
-	u8 *prog = *pprog;
-
 	if (nullable) {
 		if (src_reg != BPF_REG_0)
-			emit_mov_reg(&prog, true, BPF_REG_0, src_reg);
+			emit_mov_reg(jit, true, BPF_REG_0, src_reg);
 		/* test rax, rax; jz over the 5-byte sub */
 		EMIT3(0x48, 0x85, 0xC0);
 		EMIT2(X86_JE, 5);
 	} else if (src_reg != BPF_REG_0) {
-		emit_mov_reg(&prog, false, BPF_REG_0, src_reg);
+		emit_mov_reg(jit, false, BPF_REG_0, src_reg);
 	}
 	/* sub eax, base_lo */
 	EMIT1_off32(0x2D, base_lo);
-
-	*pprog = prog;
 }
 
-static void save_args(const struct btf_func_model *m, u8 **prog,
+static void save_args(const struct btf_func_model *m, struct jit_emit_context *jit,
 		      int stack_size, bool for_call_origin, u32 flags,
 		      u64 arena_base)
 {
@@ -3150,12 +3056,12 @@ static void save_args(const struct btf_func_model *m, u8 **prog,
 			 * called indirectly, so rbp + 16.
 			 */
 			for (j = 0; j < arg_regs; j++) {
-				emit_ldx(prog, BPF_DW, BPF_REG_0, BPF_REG_FP,
+				emit_ldx(jit, BPF_DW, BPF_REG_0, BPF_REG_FP,
 					 nr_stack_slots * 8 + stack_args_off);
 				if (arena_arg)
-					emit_arena_arg_conv(prog, BPF_REG_0, nullable,
+					emit_arena_arg_conv(jit, BPF_REG_0, nullable,
 							    (u32)arena_base);
-				emit_stx(prog, BPF_DW, BPF_REG_FP, BPF_REG_0,
+				emit_stx(jit, BPF_DW, BPF_REG_FP, BPF_REG_0,
 					 -stack_size);
 
 				if (!nr_stack_slots)
@@ -3178,20 +3084,20 @@ static void save_args(const struct btf_func_model *m, u8 **prog,
 				u32 src = nr_regs == 5 ? X86_REG_R9 : BPF_REG_1 + nr_regs;
 
 				if (arena_arg) {
-					emit_arena_arg_conv(prog, src, nullable, (u32)arena_base);
+					emit_arena_arg_conv(jit, src, nullable, (u32)arena_base);
 					src = BPF_REG_0;
 				}
-				emit_stx(prog, BPF_DW, BPF_REG_FP, src, -stack_size);
+				emit_stx(jit, BPF_DW, BPF_REG_FP, src, -stack_size);
 				stack_size -= 8;
 				nr_regs++;
 			}
 		}
 	}
 
-	clean_stack_garbage(m, prog, nr_stack_slots, first_off);
+	clean_stack_garbage(m, jit, nr_stack_slots, first_off);
 }
 
-static void restore_regs(const struct btf_func_model *m, u8 **prog,
+static void restore_regs(const struct btf_func_model *m, struct jit_emit_context *jit,
 			 int stack_size)
 {
 	int i, j, arg_regs, nr_regs = 0;
@@ -3207,7 +3113,7 @@ static void restore_regs(const struct btf_func_model *m, u8 **prog,
 		arg_regs = (m->arg_size[i] + 7) / 8;
 		if (nr_regs + arg_regs <= 6) {
 			for (j = 0; j < arg_regs; j++) {
-				emit_ldx(prog, BPF_DW,
+				emit_ldx(jit, BPF_DW,
 					 nr_regs == 5 ? X86_REG_R9 : BPF_REG_1 + nr_regs,
 					 BPF_REG_FP,
 					 -stack_size);
@@ -3223,19 +3129,18 @@ static void restore_regs(const struct btf_func_model *m, u8 **prog,
 	}
 }
 
-static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog,
+static int invoke_bpf_prog(const struct btf_func_model *m, struct jit_emit_context *jit,
 			   struct bpf_tramp_node *node, int stack_size,
 			   int run_ctx_off, bool save_ret,
 			   void *image, void *rw_image)
 {
-	u8 *prog = *pprog;
 	u8 *jmp_insn;
 	int ctx_cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie);
 	struct bpf_prog *p = node->link->prog;
 	u64 cookie = node->cookie;
 
 	/* mov rdi, cookie */
-	emit_mov_imm64(&prog, BPF_REG_1, (long) cookie >> 32, (u32) (long) cookie);
+	emit_mov_imm64(jit, BPF_REG_1, (long) cookie >> 32, (u32) (long) cookie);
 
 	/* Prepare struct bpf_tramp_run_ctx.
 	 *
@@ -3244,28 +3149,28 @@ static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog,
 	 *
 	 * mov QWORD PTR [rbp - run_ctx_off + ctx_cookie_off], rdi
 	 */
-	emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_1, -run_ctx_off + ctx_cookie_off);
+	emit_stx(jit, BPF_DW, BPF_REG_FP, BPF_REG_1, -run_ctx_off + ctx_cookie_off);
 
 	/* arg1: mov rdi, progs[i] */
-	emit_mov_imm64(&prog, BPF_REG_1, (long) p >> 32, (u32) (long) p);
+	emit_mov_imm64(jit, BPF_REG_1, (long) p >> 32, (u32) (long) p);
 	/* arg2: lea rsi, [rbp - ctx_cookie_off] */
 	if (!is_imm8(-run_ctx_off))
 		EMIT3_off32(0x48, 0x8D, 0xB5, -run_ctx_off);
 	else
 		EMIT4(0x48, 0x8D, 0x75, -run_ctx_off);
 
-	if (emit_rsb_call(&prog, bpf_trampoline_enter(p), image + (prog - (u8 *)rw_image)))
+	if (emit_rsb_call(jit, bpf_trampoline_enter(p), image + (jit->prog - (u8 *)rw_image)))
 		return -EINVAL;
 	/* remember prog start time returned by __bpf_prog_enter */
-	emit_mov_reg(&prog, true, BPF_REG_6, BPF_REG_0);
+	emit_mov_reg(jit, true, BPF_REG_6, BPF_REG_0);
 
 	/* if (__bpf_prog_enter*(prog) == 0)
 	 *	goto skip_exec_of_prog;
 	 */
 	EMIT3(0x48, 0x85, 0xC0);  /* test rax,rax */
 	/* emit 2 nops that will be replaced with JE insn */
-	jmp_insn = prog;
-	emit_nops(&prog, 2);
+	jmp_insn = jit->prog;
+	emit_nops(jit, 2);
 
 	/* arg1: lea rdi, [rbp - stack_size] */
 	if (!is_imm8(-stack_size))
@@ -3274,11 +3179,11 @@ static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog,
 		EMIT4(0x48, 0x8D, 0x7D, -stack_size);
 	/* arg2: progs[i]->insnsi for interpreter */
 	if (!p->jited)
-		emit_mov_imm64(&prog, BPF_REG_2,
+		emit_mov_imm64(jit, BPF_REG_2,
 			       (long) p->insnsi >> 32,
 			       (u32) (long) p->insnsi);
 	/* call JITed bpf program or interpreter */
-	if (emit_rsb_call(&prog, p->bpf_func, image + (prog - (u8 *)rw_image)))
+	if (emit_rsb_call(jit, p->bpf_func, image + (jit->prog - (u8 *)rw_image)))
 		return -EINVAL;
 
 	/*
@@ -3290,42 +3195,39 @@ static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog,
 	 * value of BPF_PROG_TYPE_STRUCT_OPS prog.
 	 */
 	if (save_ret)
-		emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8);
+		emit_stx(jit, BPF_DW, BPF_REG_FP, BPF_REG_0, -8);
 
 	/* replace 2 nops with JE insn, since jmp target is known */
-	jmp_insn[0] = X86_JE;
-	jmp_insn[1] = prog - jmp_insn - 2;
+	if (!jit->dry_run) {
+		jmp_insn[0] = X86_JE;
+		jmp_insn[1] = jit->prog - jmp_insn - 2;
+	}
 
 	/* arg1: mov rdi, progs[i] */
-	emit_mov_imm64(&prog, BPF_REG_1, (long) p >> 32, (u32) (long) p);
+	emit_mov_imm64(jit, BPF_REG_1, (long) p >> 32, (u32) (long) p);
 	/* arg2: mov rsi, rbx <- start time in nsec */
-	emit_mov_reg(&prog, true, BPF_REG_2, BPF_REG_6);
+	emit_mov_reg(jit, true, BPF_REG_2, BPF_REG_6);
 	/* arg3: lea rdx, [rbp - run_ctx_off] */
 	if (!is_imm8(-run_ctx_off))
 		EMIT3_off32(0x48, 0x8D, 0x95, -run_ctx_off);
 	else
 		EMIT4(0x48, 0x8D, 0x55, -run_ctx_off);
-	if (emit_rsb_call(&prog, bpf_trampoline_exit(p), image + (prog - (u8 *)rw_image)))
+	if (emit_rsb_call(jit, bpf_trampoline_exit(p), image + (jit->prog - (u8 *)rw_image)))
 		return -EINVAL;
 
-	*pprog = prog;
 	return 0;
 }
 
-static void emit_align(u8 **pprog, u32 align)
+static void emit_align(struct jit_emit_context *jit, u32 align)
 {
-	u8 *target, *prog = *pprog;
-
-	target = PTR_ALIGN(prog, align);
-	if (target != prog)
-		emit_nops(&prog, target - prog);
+	u8 *target = PTR_ALIGN(jit->prog, align);
 
-	*pprog = prog;
+	if (target != jit->prog)
+		emit_nops(jit, target - jit->prog);
 }
 
-static int emit_cond_near_jump(u8 **pprog, void *func, void *ip, u8 jmp_cond)
+static int emit_cond_near_jump(struct jit_emit_context *jit, void *func, void *ip, u8 jmp_cond)
 {
-	u8 *prog = *pprog;
 	s64 offset;
 
 	offset = func - (ip + 2 + 4);
@@ -3334,48 +3236,44 @@ static int emit_cond_near_jump(u8 **pprog, void *func, void *ip, u8 jmp_cond)
 		return -EINVAL;
 	}
 	EMIT2_off32(0x0F, jmp_cond + 0x10, offset);
-	*pprog = prog;
 	return 0;
 }
 
-static int invoke_bpf(const struct btf_func_model *m, u8 **pprog,
+static int invoke_bpf(const struct btf_func_model *m, struct jit_emit_context *jit,
 		      struct bpf_tramp_nodes *tl, int stack_size,
 		      int run_ctx_off, int func_meta_off, bool save_ret,
 		      void *image, void *rw_image, u64 func_meta,
 		      int cookie_off)
 {
 	int i, cur_cookie = (cookie_off - stack_size) / 8;
-	u8 *prog = *pprog;
 
 	for (i = 0; i < tl->nr_nodes; i++) {
 		if (tl->nodes[i]->link->prog->call_session_cookie) {
-			emit_store_stack_imm64(&prog, BPF_REG_0, -func_meta_off,
+			emit_store_stack_imm64(jit, BPF_REG_0, -func_meta_off,
 				func_meta | (cur_cookie << BPF_TRAMP_COOKIE_INDEX_SHIFT));
 			cur_cookie--;
 		}
-		if (invoke_bpf_prog(m, &prog, tl->nodes[i], stack_size,
+		if (invoke_bpf_prog(m, jit, tl->nodes[i], stack_size,
 				    run_ctx_off, save_ret, image, rw_image))
 			return -EINVAL;
 	}
-	*pprog = prog;
 	return 0;
 }
 
-static int invoke_bpf_mod_ret(const struct btf_func_model *m, u8 **pprog,
+static int invoke_bpf_mod_ret(const struct btf_func_model *m, struct jit_emit_context *jit,
 			      struct bpf_tramp_nodes *tl, int stack_size,
 			      int run_ctx_off, u8 **branches,
 			      void *image, void *rw_image)
 {
-	u8 *prog = *pprog;
 	int i;
 
 	/* The first fmod_ret program will receive a garbage return value.
 	 * Set this to 0 to avoid confusing the program.
 	 */
-	emit_mov_imm32(&prog, false, BPF_REG_0, 0);
-	emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8);
+	emit_mov_imm32(jit, false, BPF_REG_0, 0);
+	emit_stx(jit, BPF_DW, BPF_REG_FP, BPF_REG_0, -8);
 	for (i = 0; i < tl->nr_nodes; i++) {
-		if (invoke_bpf_prog(m, &prog, tl->nodes[i], stack_size, run_ctx_off, true,
+		if (invoke_bpf_prog(m, jit, tl->nodes[i], stack_size, run_ctx_off, true,
 				    image, rw_image))
 			return -EINVAL;
 
@@ -3391,11 +3289,10 @@ static int invoke_bpf_mod_ret(const struct btf_func_model *m, u8 **pprog,
 		 * are replaced with a conditional jump once do_fexit (i.e. the
 		 * start of the fexit invocation) is finalized.
 		 */
-		branches[i] = prog;
-		emit_nops(&prog, 4 + 2);
+		branches[i] = jit->prog;
+		emit_nops(jit, 4 + 2);
 	}
 
-	*pprog = prog;
 	return 0;
 }
 
@@ -3474,12 +3371,12 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
 	struct bpf_tramp_nodes *fentry = &tnodes[BPF_TRAMP_FENTRY];
 	struct bpf_tramp_nodes *fexit = &tnodes[BPF_TRAMP_FEXIT];
 	struct bpf_tramp_nodes *fmod_ret = &tnodes[BPF_TRAMP_MODIFY_RETURN];
+	struct jit_emit_context jit_ctx = {}, *jit = &jit_ctx;
 	void *orig_call = func_addr;
 	int cookie_off, cookie_cnt;
 	u8 **branches = NULL;
 	u64 arena_base;
 	u64 func_meta;
-	u8 *prog;
 	bool save_ret;
 
 	/*
@@ -3584,13 +3481,14 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
 		orig_call += X86_PATCH_SIZE;
 	}
 
-	prog = rw_image;
+	jit->prog = rw_image;
+	jit->dry_run = !rw_image;
 
 	if (flags & BPF_TRAMP_F_INDIRECT) {
 		/*
 		 * Indirect call for bpf_struct_ops
 		 */
-		emit_cfi(&prog, image,
+		emit_cfi(jit, image,
 			 cfi_get_func_hash(func_addr),
 			 cfi_get_func_arity(func_addr));
 	} else {
@@ -3598,12 +3496,12 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
 		 * Direct-call fentry stub, as such it needs accounting for the
 		 * __fentry__ call.
 		 */
-		x86_call_depth_emit_accounting(&prog, NULL, image);
+		bpf_call_depth_emit_accounting(jit, NULL, image);
 	}
 	EMIT1(0x55);		 /* push rbp */
 	EMIT3(0x48, 0x89, 0xE5); /* mov rbp, rsp */
 	if (im)
-		im->ksym.fp_start = prog - (u8 *)rw_image;
+		im->ksym.fp_start = jit->prog - (u8 *)rw_image;
 
 	if (!is_imm8(stack_size)) {
 		/* sub rsp, stack_size */
@@ -3615,24 +3513,24 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
 	if (flags & BPF_TRAMP_F_TAIL_CALL_CTX)
 		EMIT1(0x50);		/* push rax */
 	/* mov QWORD PTR [rbp - rbx_off], rbx */
-	emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_6, -rbx_off);
+	emit_stx(jit, BPF_DW, BPF_REG_FP, BPF_REG_6, -rbx_off);
 
 	func_meta = nr_regs;
 	/* Store number of argument registers of the traced function */
-	emit_store_stack_imm64(&prog, BPF_REG_0, -func_meta_off, func_meta);
+	emit_store_stack_imm64(jit, BPF_REG_0, -func_meta_off, func_meta);
 
 	if (flags & BPF_TRAMP_F_IP_ARG) {
 		/* Store IP address of the traced function */
-		emit_store_stack_imm64(&prog, BPF_REG_0, -ip_off, (long)func_addr);
+		emit_store_stack_imm64(jit, BPF_REG_0, -ip_off, (long)func_addr);
 	}
 
-	save_args(m, &prog, regs_off, false, flags, arena_base);
+	save_args(m, jit, regs_off, false, flags, arena_base);
 
 	if (flags & BPF_TRAMP_F_CALL_ORIG) {
 		/* arg1: mov rdi, im */
-		emit_mov_imm64(&prog, BPF_REG_1, (long) im >> 32, (u32) (long) im);
-		if (emit_rsb_call(&prog, __bpf_tramp_enter,
-				  image + (prog - (u8 *)rw_image))) {
+		emit_mov_imm64(jit, BPF_REG_1, (long) im >> 32, (u32) (long) im);
+		if (emit_rsb_call(jit, __bpf_tramp_enter,
+				  image + (jit->prog - (u8 *)rw_image))) {
 			ret = -EINVAL;
 			goto cleanup;
 		}
@@ -3641,13 +3539,13 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
 	if (bpf_fsession_cnt(tnodes)) {
 		/* clear all the session cookies' value */
 		for (int i = 0; i < cookie_cnt; i++)
-			emit_store_stack_imm64(&prog, BPF_REG_0, -cookie_off + 8 * i, 0);
+			emit_store_stack_imm64(jit, BPF_REG_0, -cookie_off + 8 * i, 0);
 		/* clear the return value to make sure fentry always get 0 */
-		emit_store_stack_imm64(&prog, BPF_REG_0, -8, 0);
+		emit_store_stack_imm64(jit, BPF_REG_0, -8, 0);
 	}
 
 	if (fentry->nr_nodes) {
-		if (invoke_bpf(m, &prog, fentry, regs_off, run_ctx_off, func_meta_off,
+		if (invoke_bpf(m, jit, fentry, regs_off, run_ctx_off, func_meta_off,
 			       flags & BPF_TRAMP_F_RET_FENTRY_RET, image, rw_image,
 			       func_meta, cookie_off))
 			return -EINVAL;
@@ -3659,7 +3557,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
 		if (!branches)
 			return -ENOMEM;
 
-		if (invoke_bpf_mod_ret(m, &prog, fmod_ret, regs_off,
+		if (invoke_bpf_mod_ret(m, jit, fmod_ret, regs_off,
 				       run_ctx_off, branches, image, rw_image)) {
 			ret = -EINVAL;
 			goto cleanup;
@@ -3667,8 +3565,8 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
 	}
 
 	if (flags & BPF_TRAMP_F_CALL_ORIG) {
-		restore_regs(m, &prog, regs_off);
-		save_args(m, &prog, arg_stack_off, true, flags, 0);
+		restore_regs(m, jit, regs_off);
+		save_args(m, jit, arg_stack_off, true, flags, 0);
 
 		if (flags & BPF_TRAMP_F_TAIL_CALL_CTX) {
 			/* Before calling the original function, load the
@@ -3678,19 +3576,19 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
 		}
 
 		if (flags & BPF_TRAMP_F_ORIG_STACK) {
-			emit_ldx(&prog, BPF_DW, BPF_REG_6, BPF_REG_FP, 8);
+			emit_ldx(jit, BPF_DW, BPF_REG_6, BPF_REG_FP, 8);
 			EMIT2(0xff, 0xd3); /* call *rbx */
 		} else {
 			/* call original function */
-			if (emit_rsb_call(&prog, orig_call, image + (prog - (u8 *)rw_image))) {
+			if (emit_rsb_call(jit, orig_call, image + (jit->prog - (u8 *)rw_image))) {
 				ret = -EINVAL;
 				goto cleanup;
 			}
 		}
 		/* remember return value in a stack for bpf prog to access */
-		emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8);
-		im->ip_after_call = image + (prog - (u8 *)rw_image);
-		emit_nops(&prog, X86_PATCH_SIZE);
+		emit_stx(jit, BPF_DW, BPF_REG_FP, BPF_REG_0, -8);
+		im->ip_after_call = image + (jit->prog - (u8 *)rw_image);
+		emit_nops(jit, X86_PATCH_SIZE);
 	}
 
 	if (fmod_ret->nr_nodes) {
@@ -3699,12 +3597,17 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
 		 * Coding Rule 11: All branch targets should be 16-byte
 		 * aligned.
 		 */
-		emit_align(&prog, 16);
+		emit_align(jit, 16);
 		/* Update the branches saved in invoke_bpf_mod_ret with the
 		 * aligned address of do_fexit.
 		 */
 		for (i = 0; i < fmod_ret->nr_nodes; i++) {
-			emit_cond_near_jump(&branches[i], image + (prog - (u8 *)rw_image),
+			struct jit_emit_context branch_jit = {
+				.prog = branches[i],
+				.dry_run = jit->dry_run,
+			};
+
+			emit_cond_near_jump(&branch_jit, image + (jit->prog - (u8 *)rw_image),
 					    image + (branches[i] - (u8 *)rw_image), X86_JNE);
 		}
 	}
@@ -3712,10 +3615,10 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
 	/* set the "is_return" flag for fsession */
 	func_meta |= (1ULL << BPF_TRAMP_IS_RETURN_SHIFT);
 	if (bpf_fsession_cnt(tnodes))
-		emit_store_stack_imm64(&prog, BPF_REG_0, -func_meta_off, func_meta);
+		emit_store_stack_imm64(jit, BPF_REG_0, -func_meta_off, func_meta);
 
 	if (fexit->nr_nodes) {
-		if (invoke_bpf(m, &prog, fexit, regs_off, run_ctx_off, func_meta_off,
+		if (invoke_bpf(m, jit, fexit, regs_off, run_ctx_off, func_meta_off,
 			       false, image, rw_image, func_meta, cookie_off)) {
 			ret = -EINVAL;
 			goto cleanup;
@@ -3723,17 +3626,17 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
 	}
 
 	if (flags & BPF_TRAMP_F_RESTORE_REGS)
-		restore_regs(m, &prog, regs_off);
+		restore_regs(m, jit, regs_off);
 
 	/* This needs to be done regardless. If there were fmod_ret programs,
 	 * the return value is only updated on the stack and still needs to be
 	 * restored to R0.
 	 */
 	if (flags & BPF_TRAMP_F_CALL_ORIG) {
-		im->ip_epilogue = image + (prog - (u8 *)rw_image);
+		im->ip_epilogue = image + (jit->prog - (u8 *)rw_image);
 		/* arg1: mov rdi, im */
-		emit_mov_imm64(&prog, BPF_REG_1, (long) im >> 32, (u32) (long) im);
-		if (emit_rsb_call(&prog, __bpf_tramp_exit, image + (prog - (u8 *)rw_image))) {
+		emit_mov_imm64(jit, BPF_REG_1, (long) im >> 32, (u32) (long) im);
+		if (emit_rsb_call(jit, __bpf_tramp_exit, image + (jit->prog - (u8 *)rw_image))) {
 			ret = -EINVAL;
 			goto cleanup;
 		}
@@ -3746,25 +3649,26 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
 
 	/* restore return value of orig_call or fentry prog back into RAX */
 	if (save_ret)
-		emit_ldx(&prog, BPF_DW, BPF_REG_0, BPF_REG_FP, -8);
+		emit_ldx(jit, BPF_DW, BPF_REG_0, BPF_REG_FP, -8);
 
-	emit_ldx(&prog, BPF_DW, BPF_REG_6, BPF_REG_FP, -rbx_off);
+	emit_ldx(jit, BPF_DW, BPF_REG_6, BPF_REG_FP, -rbx_off);
 
 	EMIT1(0xC9); /* leave */
 	if (im)
-		im->ksym.fp_end = prog - (u8 *)rw_image;
+		im->ksym.fp_end = jit->prog - (u8 *)rw_image;
 
 	if (flags & BPF_TRAMP_F_SKIP_FRAME) {
 		/* skip our return address and return to parent */
 		EMIT4(0x48, 0x83, 0xC4, 8); /* add rsp, 8 */
 	}
-	emit_return(&prog, image + (prog - (u8 *)rw_image));
+	emit_return(jit, image + (jit->prog - (u8 *)rw_image));
 	/* Make sure the trampoline generation logic doesn't overflow */
-	if (WARN_ON_ONCE(prog > (u8 *)rw_image_end - BPF_INSN_SAFETY)) {
+	if (!jit->dry_run &&
+	    WARN_ON_ONCE(jit->prog > (u8 *)rw_image_end - BPF_INSN_SAFETY)) {
 		ret = -EFAULT;
 		goto cleanup;
 	}
-	ret = prog - (u8 *)rw_image + BPF_INSN_SAFETY;
+	ret = jit->prog - (u8 *)rw_image + BPF_INSN_SAFETY;
 
 cleanup:
 	kfree(branches);
@@ -3819,28 +3723,15 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
 			     struct bpf_tramp_nodes *tnodes, void *func_addr)
 {
 	struct bpf_tramp_image im;
-	void *image;
-	int ret;
 
-	/* Allocate a temporary buffer for __arch_prepare_bpf_trampoline().
-	 *
-	 * We cannot use kvmalloc here, because we need image to be in
-	 * module memory range.
-	 * Since it must be writable use bpf_jit_alloc_exec_rw().
-	 */
-	image = bpf_jit_alloc_exec_rw(PAGE_SIZE);
-	if (!image)
-		return -ENOMEM;
-
-	ret = __arch_prepare_bpf_trampoline(&im, image, image + PAGE_SIZE, image,
-					    m, flags, tnodes, func_addr);
-	bpf_jit_free_exec(image);
-	return ret;
+	return __arch_prepare_bpf_trampoline(&im, NULL, NULL, NULL, m, flags,
+					     tnodes, func_addr);
 }
 
-static int emit_bpf_dispatcher(u8 **pprog, int a, int b, s64 *progs, u8 *image, u8 *buf)
+static int emit_bpf_dispatcher(struct jit_emit_context *jit, int a, int b, s64 *progs, u8 *image,
+			       u8 *buf)
 {
-	u8 *jg_reloc, *prog = *pprog;
+	u8 *jg_reloc;
 	int pivot, err, jg_bytes = 1;
 	s64 jg_offset;
 
@@ -3853,15 +3744,13 @@ static int emit_bpf_dispatcher(u8 **pprog, int a, int b, s64 *progs, u8 *image,
 			return -1;
 		EMIT2_off32(0x81, add_1reg(0xF8, BPF_REG_3),
 			    progs[a]);
-		err = emit_cond_near_jump(&prog,	/* je func */
-					  (void *)progs[a], image + (prog - buf),
+		err = emit_cond_near_jump(jit,	/* je func */
+					  (void *)progs[a], image + (jit->prog - buf),
 					  X86_JE);
 		if (err)
 			return err;
 
-		emit_indirect_jump(&prog, BPF_REG_3 /* R3 -> rdx */, image + (prog - buf));
-
-		*pprog = prog;
+		emit_indirect_jump(jit, BPF_REG_3 /* R3 -> rdx */, image + (jit->prog - buf));
 		return 0;
 	}
 
@@ -3881,9 +3770,9 @@ static int emit_bpf_dispatcher(u8 **pprog, int a, int b, s64 *progs, u8 *image,
 	} else {
 		EMIT2(X86_JG, 0);
 	}
-	jg_reloc = prog;
+	jg_reloc = jit->prog;
 
-	err = emit_bpf_dispatcher(&prog, a, a + pivot,	/* emit lower_part */
+	err = emit_bpf_dispatcher(jit, a, a + pivot,	/* emit lower_part */
 				  progs, image, buf);
 	if (err)
 		return err;
@@ -3893,16 +3782,15 @@ static int emit_bpf_dispatcher(u8 **pprog, int a, int b, s64 *progs, u8 *image,
 	 * Coding Rule 11: All branch targets should be 16-byte
 	 * aligned.
 	 */
-	emit_align(&prog, 16);
-	jg_offset = prog - jg_reloc;
+	emit_align(jit, 16);
+	jg_offset = jit->prog - jg_reloc;
 	emit_code(jg_reloc - jg_bytes, jg_offset, jg_bytes);
 
-	err = emit_bpf_dispatcher(&prog, a + pivot + 1,	/* emit upper_part */
+	err = emit_bpf_dispatcher(jit, a + pivot + 1,	/* emit upper_part */
 				  b, progs, image, buf);
 	if (err)
 		return err;
 
-	*pprog = prog;
 	return 0;
 }
 
@@ -3920,10 +3808,10 @@ static int cmp_ips(const void *a, const void *b)
 
 int arch_prepare_bpf_dispatcher(void *image, void *buf, s64 *funcs, int num_funcs)
 {
-	u8 *prog = buf;
+	struct jit_emit_context jit = { .prog = buf };
 
 	sort(funcs, num_funcs, sizeof(funcs[0]), cmp_ips, NULL);
-	return emit_bpf_dispatcher(&prog, 0, num_funcs - 1, funcs, image, buf);
+	return emit_bpf_dispatcher(&jit, 0, num_funcs - 1, funcs, image, buf);
 }
 
 static void priv_stack_init_guard(void __percpu *priv_stack_ptr, int alloc_size)

							input: 41934
							output: 34
							thoughts: 993

**Analyzing the BPF JIT Compiler Patch**

Okay, let's break this down. My initial assessment is that this patch is primarily focused on optimizing the BPF JIT compiler for the x86 architecture. The key change is the introduction of a `struct jit_emit_context`, which seems to be the central mechanism for the refactoring.  It has `prog` and `dry_run` members, indicating a strategy of separating the code generation from the actual emission.

Specifically, the `dry_run` flag is crucial.  The `arch_bpf_trampoline_size` function is significantly altered to calculate the trampoline size *without* actually allocating an executable page using this flag.  The old approach allocated a page, prepared a trampoline, and then freed the page just to get the size. This is clearly a performance win.  This "dry run" concept is propagated throughout the code via the `jit_emit_context` struct. Functions like `bpf_call_depth_emit_accounting`, `emit_code_jit`, and `emit_cond_near_jump` are all updated to behave differently based on the `dry_run` flag.

I can see the `bpf_call_depth_emit_accounting` function being refactored to use this new context and `insn_buff`. The `emit_code_jit` and `emit_cond_near_jump` functions now have a clear separation of concerns, and the actual emission is conditionally executed based on the `dry_run` flag.

The code changes include `arch/x86/include/asm/alternative.h`, `arch/x86/include/asm/text-patching.h`, `arch/x86/kernel/alternative.c`, `arch/x86/kernel/callthunks.c`, and `arch/x86/net/bpf_jit_comp.c`, so I'll need to check the relevant configs and focus symbols using the search tools. This seems to be a functional change and performance optimization to BPF JIT, particularly trampoline sizing and code generation, which makes it an interesting target for deeper analysis and testing. This all appears to revolve around the BPF JIT and the core x86 architecture. I'll need to delve deeper and see if any vulnerabilities are potentially exposed by these architectural changes and optimizations. Now, let's explore the BPF JIT code and relevant configurations and potential focus symbols to assess this patch's fuzzing worthiness.

{
  "Expression": "arch_prepare_bpf_trampoline",
  "PathPrefix": "arch/x86"
}

{
  "Output": "arch/x86/net/bpf_jit_comp.c=3132=static int invoke_bpf_prog(const struct btf_func_model *m, struct jit_emit_context *jit,\n--\narch/x86/net/bpf_jit_comp.c-3147-\t * bpf_tramp_run_ctx is already preserved by\narch/x86/net/bpf_jit_comp.c:3148:\t * arch_prepare_bpf_trampoline().\narch/x86/net/bpf_jit_comp.c-3149-\t *\n--\narch/x86/net/bpf_jit_comp.c=3263=static int invoke_bpf_mod_ret(const struct btf_func_model *m, struct jit_emit_context *jit,\n--\narch/x86/net/bpf_jit_comp.c-3362- */\narch/x86/net/bpf_jit_comp.c:3363:static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_image,\narch/x86/net/bpf_jit_comp.c-3364-\t\t\t\t\t void *rw_image_end, void *image,\n--\narch/x86/net/bpf_jit_comp.c=3688=int arch_protect_bpf_trampoline(void *image, unsigned int size)\n--\narch/x86/net/bpf_jit_comp.c-3692-\narch/x86/net/bpf_jit_comp.c:3693:int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *image_end,\narch/x86/net/bpf_jit_comp.c-3694-\t\t\t\tconst struct btf_func_model *m, u32 flags,\n--\narch/x86/net/bpf_jit_comp.c-3708-\narch/x86/net/bpf_jit_comp.c:3709:\tret = __arch_prepare_bpf_trampoline(im, rw_image, rw_image + size, image, m,\narch/x86/net/bpf_jit_comp.c-3710-\t\t\t\t\t    flags, tnodes, func_addr);\n--\narch/x86/net/bpf_jit_comp.c=3722=int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,\n--\narch/x86/net/bpf_jit_comp.c-3726-\narch/x86/net/bpf_jit_comp.c:3727:\treturn __arch_prepare_bpf_trampoline(\u0026im, NULL, NULL, NULL, m, flags,\narch/x86/net/bpf_jit_comp.c-3728-\t\t\t\t\t     tnodes, func_addr);\n"
}

							input: 44264
							output: 31
							thoughts: 65

Here's my thinking process:

**Investigating Kconfig Dependencies and Build System Integration**

Alright, I need to understand the configuration dependencies and build process surrounding `arch/x86/net/bpf_jit_comp.c` and `arch/x86/kernel/callthunks.c`. I'll start by checking which Kconfig options are directly related to these files. This will give me a better understanding of how the features they implement are enabled and disabled.

Next, I need to figure out how these source files are compiled and linked into the kernel. The best way to do that is to look at the Makefiles. I'll search the relevant directories, `arch/x86/net/` and `arch/x86/kernel/`, to find the Makefiles that build these files. That should give me the overall build configuration.

{
  "Expression": "bpf_jit_comp",
  "PathPrefix": "arch/x86"
}

{
  "Output": "arch/x86/net/Makefile=6=ifeq ($(CONFIG_X86_32),y)\narch/x86/net/Makefile:7:        obj-$(CONFIG_BPF_JIT) += bpf_jit_comp32.o\narch/x86/net/Makefile-8-else\narch/x86/net/Makefile:9:        obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o bpf_timed_may_goto.o\narch/x86/net/Makefile-10-endif\n"
}

							input: 44693
							output: 28
							thoughts: 13

Here's my attempt at summarizing the thought process, tailored for an expert audience:

**Investigating Potential Configuration Issues**

Okay, so we've been looking at this module's behavior, and I'm starting to think we might be missing something on the configuration side. My immediate thought is to double-check the `callthunks` build system integration. I need to dive into the `Makefile` and the associated `Kconfig` files for that component. There might be a subtle dependency, a missing flag, or an incorrect setting within those files that's throwing us off. It's a classic debugging scenario: look beyond the immediate code and trace the build environment. This is probably a good place to start, given the symptoms.

{
  "Expression": "callthunks",
  "PathPrefix": "arch/x86"
}

{
  "Output": "arch/x86/Kconfig=2474=config CALL_THUNKS_DEBUG\n--\narch/x86/Kconfig-2480-\t  Enable call/ret counters for imbalance detection and build in\narch/x86/Kconfig:2481:\t  a noisy dmesg about callthunks generation and call patching for\narch/x86/Kconfig-2482-\t  trouble shooting. The debug prints need to be enabled on the\narch/x86/Kconfig:2483:\t  kernel command line with 'debug-callthunks'.\narch/x86/Kconfig-2484-\t  Only enable this when you are debugging call thunks as this\n--\narch/x86/include/asm/alternative.h=77=struct callthunk_sites {\n--\narch/x86/include/asm/alternative.h-81-#ifdef CONFIG_CALL_THUNKS\narch/x86/include/asm/alternative.h:82:extern void callthunks_patch_builtin_calls(void);\narch/x86/include/asm/alternative.h:83:extern void callthunks_patch_module_calls(struct callthunk_sites *sites,\narch/x86/include/asm/alternative.h-84-\t\t\t\t\t  struct module *mod);\narch/x86/include/asm/alternative.h:85:extern void *callthunks_translate_call_dest(void *dest);\narch/x86/include/asm/alternative.h-86-extern int x86_call_depth_emit_accounting(u8 *insn_buff, void *func, void *ip);\narch/x86/include/asm/alternative.h-87-#else\narch/x86/include/asm/alternative.h:88:static __always_inline void callthunks_patch_builtin_calls(void) {}\narch/x86/include/asm/alternative.h-89-static __always_inline void\narch/x86/include/asm/alternative.h:90:callthunks_patch_module_calls(struct callthunk_sites *sites,\narch/x86/include/asm/alternative.h-91-\t\t\t      struct module *mod) {}\narch/x86/include/asm/alternative.h:92:static __always_inline void *callthunks_translate_call_dest(void *dest)\narch/x86/include/asm/alternative.h-93-{\n--\narch/x86/kernel/Makefile=165=obj-$(CONFIG_CFI)\t\t\t+= cfi.o\narch/x86/kernel/Makefile-166-\narch/x86/kernel/Makefile:167:obj-$(CONFIG_CALL_THUNKS)\t\t+= callthunks.o\narch/x86/kernel/Makefile-168-\n--\narch/x86/kernel/alternative.c=2237=void __init alternative_instructions(void)\n--\narch/x86/kernel/alternative.c-2286-\t */\narch/x86/kernel/alternative.c:2287:\tcallthunks_patch_builtin_calls();\narch/x86/kernel/alternative.c-2288-\n--\narch/x86/kernel/callthunks.c-2-\narch/x86/kernel/callthunks.c:3:#define pr_fmt(fmt) \"callthunks: \" fmt\narch/x86/kernel/callthunks.c-4-\n--\narch/x86/kernel/callthunks.c-23-\narch/x86/kernel/callthunks.c:24:static int __initdata_or_module debug_callthunks;\narch/x86/kernel/callthunks.c-25-\n--\narch/x86/kernel/callthunks.c=27=do {\t\t\t\t\t\t\t\t\\\narch/x86/kernel/callthunks.c:28:\tif (debug_callthunks)\t\t\t\t\t\\\narch/x86/kernel/callthunks.c-29-\t\tprintk(KERN_DEBUG pr_fmt(fmt), ##args);\t\t\\\n--\narch/x86/kernel/callthunks.c=32=static int __init debug_thunks(char *str)\narch/x86/kernel/callthunks.c-33-{\narch/x86/kernel/callthunks.c:34:\tdebug_callthunks = 1;\narch/x86/kernel/callthunks.c-35-\treturn 1;\narch/x86/kernel/callthunks.c-36-}\narch/x86/kernel/callthunks.c:37:__setup(\"debug-callthunks\", debug_thunks);\narch/x86/kernel/callthunks.c-38-\n--\narch/x86/kernel/callthunks.c=238=static __init_or_module void\narch/x86/kernel/callthunks.c:239:callthunks_setup(struct callthunk_sites *cs, const struct core_text *ct)\narch/x86/kernel/callthunks.c-240-{\n--\narch/x86/kernel/callthunks.c-245-\narch/x86/kernel/callthunks.c:246:void __init callthunks_patch_builtin_calls(void)\narch/x86/kernel/callthunks.c-247-{\n--\narch/x86/kernel/callthunks.c-257-\tmutex_lock(\u0026text_mutex);\narch/x86/kernel/callthunks.c:258:\tcallthunks_setup(\u0026cs, \u0026builtin_coretext);\narch/x86/kernel/callthunks.c-259-\tthunks_initialized = true;\n--\narch/x86/kernel/callthunks.c-262-\narch/x86/kernel/callthunks.c:263:void *callthunks_translate_call_dest(void *dest)\narch/x86/kernel/callthunks.c-264-{\n--\narch/x86/kernel/callthunks.c=299=int x86_call_depth_emit_accounting(u8 *insn_buff, void *func, void *ip)\n--\narch/x86/kernel/callthunks.c-317-#ifdef CONFIG_MODULES\narch/x86/kernel/callthunks.c:318:void noinline callthunks_patch_module_calls(struct callthunk_sites *cs,\narch/x86/kernel/callthunks.c-319-\t\t\t\t\t    struct module *mod)\n--\narch/x86/kernel/callthunks.c-330-\tmutex_lock(\u0026text_mutex);\narch/x86/kernel/callthunks.c:331:\tcallthunks_setup(cs, \u0026ct);\narch/x86/kernel/callthunks.c-332-\tmutex_unlock(\u0026text_mutex);\n--\narch/x86/kernel/callthunks.c-336-#if defined(CONFIG_CALL_THUNKS_DEBUG) \u0026\u0026 defined(CONFIG_DEBUG_FS)\narch/x86/kernel/callthunks.c:337:static int callthunks_debug_show(struct seq_file *m, void *p)\narch/x86/kernel/callthunks.c-338-{\n--\narch/x86/kernel/callthunks.c-348-\narch/x86/kernel/callthunks.c:349:static int callthunks_debug_open(struct inode *inode, struct file *file)\narch/x86/kernel/callthunks.c-350-{\narch/x86/kernel/callthunks.c:351:\treturn single_open(file, callthunks_debug_show, inode-\u003ei_private);\narch/x86/kernel/callthunks.c-352-}\n--\narch/x86/kernel/callthunks.c=354=static const struct file_operations dfs_ops = {\narch/x86/kernel/callthunks.c:355:\t.open\t\t= callthunks_debug_open,\narch/x86/kernel/callthunks.c-356-\t.read\t\t= seq_read,\n--\narch/x86/kernel/callthunks.c-360-\narch/x86/kernel/callthunks.c:361:static int __init callthunks_debugfs_init(void)\narch/x86/kernel/callthunks.c-362-{\n--\narch/x86/kernel/callthunks.c-365-\narch/x86/kernel/callthunks.c:366:\tdir = debugfs_create_dir(\"callthunks\", NULL);\narch/x86/kernel/callthunks.c-367-\tfor_each_possible_cpu(cpu) {\n--\narch/x86/kernel/callthunks.c-375-}\narch/x86/kernel/callthunks.c:376:__initcall(callthunks_debugfs_init);\narch/x86/kernel/callthunks.c-377-#endif\n--\narch/x86/kernel/module.c=226=int module_finalize(const Elf_Ehdr *hdr,\n--\narch/x86/kernel/module.c-289-\narch/x86/kernel/module.c:290:\t\tcallthunks_patch_module_calls(\u0026cs, me);\narch/x86/kernel/module.c-291-\t}\n--\narch/x86/kernel/static_call.c=75=static void __ref __static_call_transform(void *insn, enum insn_type type,\n--\narch/x86/kernel/static_call.c-87-\tcase CALL:\narch/x86/kernel/static_call.c:88:\t\tfunc = callthunks_translate_call_dest(func);\narch/x86/kernel/static_call.c-89-\t\tcode = text_gen_insn(CALL_INSN_OPCODE, insn, func);\n"
}

							input: 47466
							output: 31
							thoughts: 14

Here's my attempt at summarizing the thought process, tailored for an expert audience and written in the first person:

**Investigating the Kconfig for CALL_THUNKS**

Okay, let's dive in. I need to understand how `CALL_THUNKS` is being defined within the Kconfig system. Specifically, I want to see what options and dependencies are involved with this configuration symbol. This will help me grasp the context and purpose of `CALL_THUNKS` within the kernel build configuration. I'll need to use my experience with the kernel configuration system to quickly navigate the relevant files and options to get the information I need, specifically looking for dependencies, default values, and any relevant documentation or comments within the config files themselves. Knowing its definition will be key to understanding the deeper architectural implications and how it impacts the performance of the system.

{
  "Expression": "config CALL_THUNKS",
  "PathPrefix": "arch/x86"
}

{
  "Output": "arch/x86/Kconfig=2394=config HAVE_CALL_THUNKS\n--\narch/x86/Kconfig-2397-\narch/x86/Kconfig:2398:config CALL_THUNKS\narch/x86/Kconfig-2399-\tdef_bool n\n--\narch/x86/Kconfig=2457=config MITIGATION_CALL_DEPTH_TRACKING\n--\narch/x86/Kconfig-2473-\narch/x86/Kconfig:2474:config CALL_THUNKS_DEBUG\narch/x86/Kconfig-2475-\tbool \"Enable call thunks and call depth tracking debugging\"\n"
}

							input: 47877
							output: 37
							thoughts: 27

Here's a summary of my mental process:

**Diving into the x86 Architecture Configuration**

Okay, time to get my hands dirty. I need to understand some specific configuration options for the x86 architecture, specifically around lines 2390 to 2480 in the `arch/x86/Kconfig` file. I'm assuming this section likely deals with something relatively specific or potentially complex; otherwise, why would I be zeroing in on this particular range? I'm anticipating this will require a closer look at the dependencies, descriptions, and potentially the default values of those kernel configuration variables. I'll be looking for nuances in how these options interact and affect the x86 system's behavior. I will be looking out for any potentially controversial configurations or areas that could be impacted by an emerging trend. Right, let's start reading.

{
  "File": "arch/x86/Kconfig",
  "FirstLine": 2390,
  "LineCount": 80
}

{
  "Contents": "2390:\tconfig FINEIBT_BHI\n2391:\t\tdef_bool y\n2392:\t\tdepends on FINEIBT \u0026\u0026 CC_HAS_KCFI_ARITY\n2393:\t\n2394:\tconfig HAVE_CALL_THUNKS\n2395:\t\tdef_bool y\n2396:\t\tdepends on CC_HAS_ENTRY_PADDING \u0026\u0026 MITIGATION_RETHUNK \u0026\u0026 OBJTOOL\n2397:\t\n2398:\tconfig CALL_THUNKS\n2399:\t\tdef_bool n\n2400:\t\tselect CALL_PADDING\n2401:\t\n2402:\tmenuconfig CPU_MITIGATIONS\n2403:\t\tbool \"Mitigations for CPU vulnerabilities\"\n2404:\t\tdefault y\n2405:\t\thelp\n2406:\t\t  Say Y here to enable options which enable mitigations for hardware\n2407:\t\t  vulnerabilities (usually related to speculative execution).\n2408:\t\t  Mitigations can be disabled or restricted to SMT systems at runtime\n2409:\t\t  via the \"mitigations\" kernel parameter.\n2410:\t\n2411:\t\t  If you say N, all mitigations will be disabled.  This CANNOT be\n2412:\t\t  overridden at runtime.\n2413:\t\n2414:\t\t  Say 'Y', unless you really know what you are doing.\n2415:\t\n2416:\tif CPU_MITIGATIONS\n2417:\t\n2418:\tconfig MITIGATION_PAGE_TABLE_ISOLATION\n2419:\t\tbool \"Remove the kernel mapping in user mode\"\n2420:\t\tdefault y\n2421:\t\tdepends on (X86_64 || X86_PAE)\n2422:\t\thelp\n2423:\t\t  This feature reduces the number of hardware side channels by\n2424:\t\t  ensuring that the majority of kernel addresses are not mapped\n2425:\t\t  into userspace.\n2426:\t\n2427:\t\t  See Documentation/arch/x86/pti.rst for more details.\n2428:\t\n2429:\tconfig MITIGATION_RETPOLINE\n2430:\t\tbool \"Avoid speculative indirect branches in kernel\"\n2431:\t\tselect OBJTOOL if HAVE_OBJTOOL\n2432:\t\tdefault y\n2433:\t\thelp\n2434:\t\t  Compile kernel with the retpoline compiler options to guard against\n2435:\t\t  kernel-to-user data leaks by avoiding speculative indirect\n2436:\t\t  branches. Requires a compiler with -mindirect-branch=thunk-extern\n2437:\t\t  support for full protection. The kernel may run slower.\n2438:\t\n2439:\tconfig MITIGATION_RETHUNK\n2440:\t\tbool \"Enable return-thunks\"\n2441:\t\tdepends on MITIGATION_RETPOLINE \u0026\u0026 CC_HAS_RETURN_THUNK\n2442:\t\tselect OBJTOOL if HAVE_OBJTOOL\n2443:\t\tdefault y if X86_64\n2444:\t\thelp\n2445:\t\t  Compile the kernel with the return-thunks compiler option to guard\n2446:\t\t  against kernel-to-user data leaks by avoiding return speculation.\n2447:\t\t  Requires a compiler with -mfunction-return=thunk-extern\n2448:\t\t  support for full protection. The kernel may run slower.\n2449:\t\n2450:\tconfig MITIGATION_UNRET_ENTRY\n2451:\t\tbool \"Enable UNRET on kernel entry\"\n2452:\t\tdepends on CPU_SUP_AMD \u0026\u0026 MITIGATION_RETHUNK \u0026\u0026 X86_64\n2453:\t\tdefault y\n2454:\t\thelp\n2455:\t\t  Compile the kernel with support for the retbleed=unret mitigation.\n2456:\t\n2457:\tconfig MITIGATION_CALL_DEPTH_TRACKING\n2458:\t\tbool \"Mitigate RSB underflow with call depth tracking\"\n2459:\t\tdepends on CPU_SUP_INTEL \u0026\u0026 HAVE_CALL_THUNKS\n2460:\t\tselect HAVE_DYNAMIC_FTRACE_NO_PATCHABLE\n2461:\t\tselect CALL_THUNKS\n2462:\t\tdefault y\n2463:\t\thelp\n2464:\t\t  Compile the kernel with call depth tracking to mitigate the Intel\n2465:\t\t  SKL Return-Stack-Buffer (RSB) underflow issue. The mitigation is off\n2466:\t\t  by default and needs to be enabled on the kernel command line via the\n2467:\t\t  retbleed=stuff option. For non-affected systems the overhead of this\n2468:\t\t  option is marginal as the call depth tracking is using run-time\n2469:\t\t  generated call thunks in a compiler generated padding area and call\n"
}

							input: 49359
							output: 31
							thoughts: 61

Here's my thought process, as if I were analyzing this code:

**Locating the BPF_JIT Configuration**

Okay, so I'm diving into this BPF setup, and I need to figure out where `BPF_JIT` is actually defined. That's the key to understanding the JIT compiler configuration, and its going to be important to get to the bottom of the config variables. Given my familiarity with kernel structure, my initial thought is that it's probably hanging out in one of a few places. I'd lean towards `net/bpf/`, because that's the obvious home for BPF-related code. However, `init/Kconfig` is also a possibility since it sets up a lot of the initial kernel configs. It could also potentially be in `kernel/bpf/Kconfig`, if it's considered more core BPF functionality. Let's start the search.

{
  "Expression": "config BPF_JIT",
  "PathPrefix": "init|kernel|net"
}

{
  "error": "no matches"
}

							input: 49662
							output: 22
							thoughts: 17

Here's my attempt at summarizing the thought process, tailored for an expert audience:

**Investigating BPF JIT Configuration**

Okay, so I need to get a handle on the current configuration of the BPF JIT, or specifically, `CONFIG_BPF_JIT`.  My immediate instinct is to do a global search across the codebase. `grep` is the obvious tool here, because it's fast and allows me to search for a specific string. I'll need to specify the string `config BPF_JIT`, because I want to look for the configuration option definition specifically, rather than just every time that BPF and JIT appear near each other. I'm going to run that grep command across the entire source tree, from the root, because I want a comprehensive view of how it's defined and used. This should give me a solid baseline, revealing where the option is defined and, likely, in what configuration files. This should be a quick initial sanity check, and from there I can delve deeper into the dependencies or specific implementations I find.

{
  "Expression": "config BPF_JIT\\b"
}

{
  "Output": "kernel/bpf/Kconfig=27=config BPF_SYSCALL\n--\nkernel/bpf/Kconfig-41-\nkernel/bpf/Kconfig:42:config BPF_JIT\nkernel/bpf/Kconfig-43-\tbool \"Enable BPF Just In Time compiler\"\n"
}

							input: 50012
							output: 34
							thoughts: 22

**Examining Kernel BPF Configuration**

Okay, let's dive in. I need to understand how the BPF subsystem is configured within this kernel build. My initial thought is to navigate to the `kernel/bpf/Kconfig` file. I'll need to focus my attention around lines 20 through 60. That's usually where the top-level options for the BPF feature set are defined – things like whether it's enabled, any architectural dependencies, and perhaps some of the fundamental knobs that control its behavior. I expect to find things like the default state of BPF, any relevant flags, and maybe even some conditional compilation logic depending on the target architecture or kernel configuration. This section should give me the crucial high-level view of how BPF is integrated into this kernel. I anticipate this will be the starting point to understand the broader implications of BPF configuration.

{
  "File": "kernel/bpf/Kconfig",
  "FirstLine": 20,
  "LineCount": 60
}

{
  "Contents": "  20:\t# Used by archs to tell that they want the BPF JIT compiler enabled by\n  21:\t# default for kernels that were compiled with BPF JIT support.\n  22:\tconfig ARCH_WANT_DEFAULT_BPF_JIT\n  23:\t\tbool\n  24:\t\n  25:\tmenu \"BPF subsystem\"\n  26:\t\n  27:\tconfig BPF_SYSCALL\n  28:\t\tbool \"Enable bpf() system call\"\n  29:\t\tselect BPF\n  30:\t\tselect IRQ_WORK\n  31:\t\tselect NEED_TASKS_RCU\n  32:\t\tselect TASKS_TRACE_RCU\n  33:\t\tselect BINARY_PRINTF\n  34:\t\tselect NET_SOCK_MSG if NET\n  35:\t\tselect NET_XGRESS if NET\n  36:\t\tselect PAGE_POOL if NET\n  37:\t\tdefault n\n  38:\t\thelp\n  39:\t\t  Enable the bpf() system call that allows to manipulate BPF programs\n  40:\t\t  and maps via file descriptors.\n  41:\t\n  42:\tconfig BPF_JIT\n  43:\t\tbool \"Enable BPF Just In Time compiler\"\n  44:\t\tdepends on BPF\n  45:\t\tdepends on HAVE_CBPF_JIT || HAVE_EBPF_JIT\n  46:\t\tselect EXECMEM\n  47:\t\thelp\n  48:\t\t  BPF programs are normally handled by a BPF interpreter. This option\n  49:\t\t  allows the kernel to generate native code when a program is loaded\n  50:\t\t  into the kernel. This will significantly speed-up processing of BPF\n  51:\t\t  programs.\n  52:\t\n  53:\t\t  Note, an admin should enable this feature changing:\n  54:\t\t  /proc/sys/net/core/bpf_jit_enable\n  55:\t\t  /proc/sys/net/core/bpf_jit_harden   (optional)\n  56:\t\t  /proc/sys/net/core/bpf_jit_kallsyms (optional)\n  57:\t\n  58:\tconfig BPF_JIT_ALWAYS_ON\n  59:\t\tbool \"Permanently enable BPF JIT and remove BPF interpreter\"\n  60:\t\tdepends on BPF_SYSCALL \u0026\u0026 HAVE_EBPF_JIT \u0026\u0026 BPF_JIT\n  61:\t\thelp\n  62:\t\t  Enables BPF JIT and removes BPF interpreter to avoid speculative\n  63:\t\t  execution of BPF instructions by the interpreter.\n  64:\t\n  65:\t\t  When CONFIG_BPF_JIT_ALWAYS_ON is enabled, /proc/sys/net/core/bpf_jit_enable\n  66:\t\t  is permanently set to 1 and setting any other value than that will\n  67:\t\t  return failure.\n  68:\t\n  69:\tconfig BPF_JIT_DEFAULT_ON\n  70:\t\tdef_bool ARCH_WANT_DEFAULT_BPF_JIT || BPF_JIT_ALWAYS_ON\n  71:\t\tdepends on HAVE_EBPF_JIT \u0026\u0026 BPF_JIT\n  72:\t\n  73:\tconfig BPF_UNPRIV_DEFAULT_OFF\n  74:\t\tbool \"Disable unprivileged BPF by default\"\n  75:\t\tdefault y\n  76:\t\tdepends on BPF_SYSCALL\n  77:\t\thelp\n  78:\t\t  Disables unprivileged BPF by default by setting the corresponding\n  79:\t\t  /proc/sys/kernel/unprivileged_bpf_disabled knob to 2. An admin can\n"
}

							input: 51172
							output: 49
							thoughts: 252

**Analyzing BPF JIT Compilation on x86_64**

Okay, so I'm diving into the x86_64 BPF JIT compilation flow, specifically looking at potential impacts or interactions with the mitigation strategies involving call depth tracking, or those call thunks. First, I need to get a handle on the key functions involved, especially in `arch/x86/net/bpf_jit_comp.c`. Let's see... I've got `BPF_JIT`, `BPF_SYSCALL`, and `X86_64` in my head as important context.

Right, so that file should contain: `bpf_int_jit_compile` which calls `do_jit`, and those related to trampolines: `arch_prepare_bpf_trampoline`, `arch_bpf_trampoline_size`, `arch_prepare_bpf_dispatcher`. And it looks like `x86_call_depth_emit_accounting` is in `arch/x86/kernel/callthunks.c`.

Now, the main question is: are any of these functions *exposed* externally? Meaning, are `arch_bpf_trampoline_size`, `arch_prepare_bpf_trampoline` and `bpf_int_jit_compile` non-static? I need to check if they're exported symbols. That'll let me know if something outside of `bpf_jit_comp.c` could be directly calling them. I'll just grep for non-static functions in `bpf_jit_comp.c`.

{
  "Expression": "^[a-z_]+ [a-z_]+\\(",
  "PathPrefix": "arch/x86/net/bpf_jit_comp.c"
}

{
  "Output": "arch/x86/net/bpf_jit_comp.c=326=static void jit_fill_hole(void *area, unsigned int size)\n--\narch/x86/net/bpf_jit_comp.c-331-\narch/x86/net/bpf_jit_comp.c:332:int bpf_arch_text_invalidate(void *dst, size_t len)\narch/x86/net/bpf_jit_comp.c-333-{\n--\narch/x86/net/bpf_jit_comp.c=601=static int __bpf_arch_text_poke(void *ip, enum bpf_text_poke_type old_t,\n--\narch/x86/net/bpf_jit_comp.c-644-\narch/x86/net/bpf_jit_comp.c:645:int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type old_t,\narch/x86/net/bpf_jit_comp.c-646-\t\t       enum bpf_text_poke_type new_t, void *old_addr,\n--\narch/x86/net/bpf_jit_comp.c=1369=static int emit_atomic_ld_st_index(struct jit_emit_context *jit, u32 atomic_op, u32 size,\n--\narch/x86/net/bpf_jit_comp.c-1433-\narch/x86/net/bpf_jit_comp.c:1434:bool ex_handler_bpf(const struct exception_table_entry *x, struct pt_regs *regs)\narch/x86/net/bpf_jit_comp.c-1435-{\n--\narch/x86/net/bpf_jit_comp.c=3678=void *arch_alloc_bpf_trampoline(unsigned int size)\n--\narch/x86/net/bpf_jit_comp.c-3682-\narch/x86/net/bpf_jit_comp.c:3683:void arch_free_bpf_trampoline(void *image, unsigned int size)\narch/x86/net/bpf_jit_comp.c-3684-{\n--\narch/x86/net/bpf_jit_comp.c-3687-\narch/x86/net/bpf_jit_comp.c:3688:int arch_protect_bpf_trampoline(void *image, unsigned int size)\narch/x86/net/bpf_jit_comp.c-3689-{\n--\narch/x86/net/bpf_jit_comp.c-3692-\narch/x86/net/bpf_jit_comp.c:3693:int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *image_end,\narch/x86/net/bpf_jit_comp.c-3694-\t\t\t\tconst struct btf_func_model *m, u32 flags,\n--\narch/x86/net/bpf_jit_comp.c-3721-\narch/x86/net/bpf_jit_comp.c:3722:int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,\narch/x86/net/bpf_jit_comp.c-3723-\t\t\t     struct bpf_tramp_nodes *tnodes, void *func_addr)\n--\narch/x86/net/bpf_jit_comp.c=3797=static int cmp_ips(const void *a, const void *b)\n--\narch/x86/net/bpf_jit_comp.c-3808-\narch/x86/net/bpf_jit_comp.c:3809:int arch_prepare_bpf_dispatcher(void *image, void *buf, s64 *funcs, int num_funcs)\narch/x86/net/bpf_jit_comp.c-3810-{\n--\narch/x86/net/bpf_jit_comp.c=3858=struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog)\n--\narch/x86/net/bpf_jit_comp.c-4049-\narch/x86/net/bpf_jit_comp.c:4050:bool bpf_jit_supports_kfunc_call(void)\narch/x86/net/bpf_jit_comp.c-4051-{\n--\narch/x86/net/bpf_jit_comp.c-4054-\narch/x86/net/bpf_jit_comp.c:4055:bool bpf_jit_supports_kfunc_ret_reg_pair(void)\narch/x86/net/bpf_jit_comp.c-4056-{\n--\narch/x86/net/bpf_jit_comp.c-4059-\narch/x86/net/bpf_jit_comp.c:4060:bool bpf_jit_supports_stack_args(void)\narch/x86/net/bpf_jit_comp.c-4061-{\n--\narch/x86/net/bpf_jit_comp.c-4064-\narch/x86/net/bpf_jit_comp.c:4065:bool bpf_jit_supports_arena_args(void)\narch/x86/net/bpf_jit_comp.c-4066-{\n--\narch/x86/net/bpf_jit_comp.c=4070=void *bpf_arch_text_copy(void *dst, void *src, size_t len)\n--\narch/x86/net/bpf_jit_comp.c-4077-/* Indicate the JIT backend supports mixing bpf2bpf and tailcalls. */\narch/x86/net/bpf_jit_comp.c:4078:bool bpf_jit_supports_subprog_tailcalls(void)\narch/x86/net/bpf_jit_comp.c-4079-{\n--\narch/x86/net/bpf_jit_comp.c-4082-\narch/x86/net/bpf_jit_comp.c:4083:bool bpf_jit_supports_percpu_insn(void)\narch/x86/net/bpf_jit_comp.c-4084-{\n--\narch/x86/net/bpf_jit_comp.c-4087-\narch/x86/net/bpf_jit_comp.c:4088:void bpf_jit_free(struct bpf_prog *prog)\narch/x86/net/bpf_jit_comp.c-4089-{\n--\narch/x86/net/bpf_jit_comp.c-4122-\narch/x86/net/bpf_jit_comp.c:4123:bool bpf_jit_supports_exceptions(void)\narch/x86/net/bpf_jit_comp.c-4124-{\n--\narch/x86/net/bpf_jit_comp.c-4131-\narch/x86/net/bpf_jit_comp.c:4132:bool bpf_jit_supports_private_stack(void)\narch/x86/net/bpf_jit_comp.c-4133-{\n--\narch/x86/net/bpf_jit_comp.c-4136-\narch/x86/net/bpf_jit_comp.c:4137:void arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, u64 bp), void *cookie)\narch/x86/net/bpf_jit_comp.c-4138-{\n--\narch/x86/net/bpf_jit_comp.c-4152-\narch/x86/net/bpf_jit_comp.c:4153:void bpf_arch_poke_desc_update(struct bpf_jit_poke_descriptor *poke,\narch/x86/net/bpf_jit_comp.c-4154-\t\t\t       struct bpf_prog *new, struct bpf_prog *old)\n--\narch/x86/net/bpf_jit_comp.c-4200-\narch/x86/net/bpf_jit_comp.c:4201:bool bpf_jit_supports_arena(void)\narch/x86/net/bpf_jit_comp.c-4202-{\n--\narch/x86/net/bpf_jit_comp.c-4205-\narch/x86/net/bpf_jit_comp.c:4206:bool bpf_jit_supports_insn(struct bpf_insn *insn, bool in_arena)\narch/x86/net/bpf_jit_comp.c-4207-{\n--\narch/x86/net/bpf_jit_comp.c-4220-\narch/x86/net/bpf_jit_comp.c:4221:bool bpf_jit_supports_ptr_xchg(void)\narch/x86/net/bpf_jit_comp.c-4222-{\n--\narch/x86/net/bpf_jit_comp.c=4227=u64 bpf_arch_uaddress_limit(void)\n--\narch/x86/net/bpf_jit_comp.c-4231-\narch/x86/net/bpf_jit_comp.c:4232:bool bpf_jit_supports_timed_may_goto(void)\narch/x86/net/bpf_jit_comp.c-4233-{\n--\narch/x86/net/bpf_jit_comp.c-4236-\narch/x86/net/bpf_jit_comp.c:4237:bool bpf_jit_supports_fsession(void)\narch/x86/net/bpf_jit_comp.c-4238-{\n"
}

							input: 54484
							output: 216
							thoughts: 673

**My Assessment of the Fuzzing Worthiness**

Okay, let's break this down. Is this change, specifically the modifications to the BPF (Berkeley Packet Filter) code, worth the effort of fuzzing? It looks like it directly touches core kernel logic, and key architectural elements. Specifically, I see it's altering `bpf_int_jit_compile`, which is the internal JIT compiler for BPF, along with `arch_bpf_trampoline_size`, `arch_prepare_bpf_trampoline`, `arch_prepare_bpf_dispatcher`, and `x86_call_depth_emit_accounting`. That's a strong indicator.

The introduction of this dry-run mode for trampoline preparation is particularly interesting. It means the `arch_bpf_trampoline_size` function is now involved in an initial "preview" of JIT generation without memory allocation. This changes emission pointers and conditions throughout the BPF x86 JIT compiler and trampoline generator, so this is critical. I'm going to focus on these functions as areas of interest.

We know BPF program loading, and trampoline generation which is used heavily for tracing, fentry/fexit, struct_ops, etc, are all user-accessible via the `bpf()` syscall. This is a very common avenue for attack, and the potential impact is high.

My conclusion? Absolutely, this is worth fuzzing. We need to throw a wide variety of BPF program shapes at this, along with different call depths, fentry/fexit programs, fmod_ret, and fsession programs. I'm concerned about potential buffer overflows, offset miscalculations, or any divergence between the dry-run size calculation and the actual code emission in the JIT.

Therefore:

* **WorthFuzzing: True**
* **FocusSymbols:** I'm zeroing in on:
* `bpf_int_jit_compile`
* `arch_prepare_bpf_trampoline`
* `arch_bpf_trampoline_size`
* `arch_prepare_bpf_dispatcher`
* `x86_call_depth_emit_accounting`
* `bpf_arch_text_poke`
* **EnableConfigs:** We need to make sure these kernel configs are enabled:
* `BPF_SYSCALL`
* `BPF_JIT`
* `CALL_THUNKS`
* `MITIGATION_CALL_DEPTH_TRACKING`
* `X86_64`

I'll quickly double-check that those configurations are properly set in the kernel's Kconfig files to be sure. Alright, now that I have my conclusions, let's define the parameters for the `set-results` tool.

I'm ready to invoke `set-results` with the appropriate parameters.

{
  "EnableConfigs": [
    "BPF_SYSCALL",
    "BPF_JIT",
    "CALL_THUNKS",
    "MITIGATION_CALL_DEPTH_TRACKING",
    "X86_64"
  ],
  "FocusSymbols": [
    "arch_prepare_bpf_trampoline",
    "arch_bpf_trampoline_size",
    "bpf_int_jit_compile",
    "arch_prepare_bpf_dispatcher",
    "x86_call_depth_emit_accounting"
  ],
  "Reasoning": "The patch refactors the x86 BPF JIT compiler and trampoline generator by introducing a struct jit_emit_context with dry-run support. This changes how trampoline sizing is computed (dry-run without allocation), updates instruction emission logic, and alters x86 call depth accounting integration. These changes affect BPF program compilation, trampoline generation, and execution reachable via the bpf() syscall.",
  "WorthFuzzing": true
}

{
  "EnableConfigs": [
    "BPF_SYSCALL",
    "BPF_JIT",
    "CALL_THUNKS",
    "MITIGATION_CALL_DEPTH_TRACKING",
    "X86_64"
  ],
  "FocusSymbols": [
    "arch_prepare_bpf_trampoline",
    "arch_bpf_trampoline_size",
    "bpf_int_jit_compile",
    "arch_prepare_bpf_dispatcher",
    "x86_call_depth_emit_accounting"
  ],
  "Reasoning": "The patch refactors the x86 BPF JIT compiler and trampoline generator by introducing a struct jit_emit_context with dry-run support. This changes how trampoline sizing is computed (dry-run without allocation), updates instruction emission logic, and alters x86 call depth accounting integration. These changes affect BPF program compilation, trampoline generation, and execution reachable via the bpf() syscall.",
  "WorthFuzzing": true
}

{
  "WorthFuzzing": true
}

{
  "KMSANReasoning": "The patch refactors the x86 BPF JIT compiler and callthunk accounting code to introduce a dry-run mode for trampoline size calculation via `struct jit_emit_context`, replacing dummy allocation in `arch_bpf_trampoline_size()`. All new context and buffer variables are properly initialized (`= {}`), and code generation buffers are accurately tracked and populated. The patch does not introduce any uninitialized memory reads, uninitialized structure padding leaks to user space, or uninitialized branch conditions. Any potential bugs in JIT generation or pointer calculations would be out-of-bounds or logic errors caught by KASAN or standard kernel assertions, not uninitialized memory bugs detectable only by KMSAN. Therefore, dedicated KMSAN fuzzing is not required.",
  "NeedsKMSAN": false
}

You are an expert Linux kernel security engineer specializing in kernel memory
error detectors (KASAN and KMSAN). Your job is to review the provided patch series and
determine if the code changes justify spawning a dedicated KMSAN (KernelMemorySanitizer)
fuzzing session in addition to standard KASAN fuzzing.

CRITICAL DISTINCTION BETWEEN KASAN AND KMSAN:
- Standard KASAN kernel builds (upstream-apparmor-kasan.config) already enable
  a comprehensive suite of debugging tools and sanitizers, including KASAN
  (out-of-bounds accesses, use-after-free, double free, invalid free), LOCKDEP
  (locking bugs and deadlocks), UB-sanitizers, and memory corruption checks.
- KMSAN (KernelMemorySanitizer) detects reads of UNINITIALIZED memory (stack, heap,
  or page allocations) and kernel-to-user memory info-leaks.

Rule: THERE IS NO SENSE IN RUNNING A KMSAN SESSION IF A BUG CAN BE CAUGHT BY KASAN,
LOCKDEP, OR OTHER STANDARD BUG DETECTORS.
A dedicated KMSAN fuzzing session incurs significant resource costs. You must ONLY
set NeedsKMSAN=true if the code changes introduce or expose UNINITIALIZED MEMORY risks
that are detected ONLY by KMSAN.

Look holistically at the patch series and surrounding code. Even if no direct
uninitialized field accesses or new buffer allocations are added in the diff itself,
a patch may alter control flow, bounds checking, or data length calculations in ways
that change how the rest of the code operates on existing buffers (e.g. allowing
uninitialized stack/heap memory to be read, copied to user space, or used in control
flow). Do not hesitate to use your code access tools to inspect the surrounding code,
called functions, and callers.

Set NeedsKMSAN=true ONLY IF the patch introduces or modifies:
1. Kernel structures sent to user space (via copy_to_user, put_user, netlink skb
   attributes, ioctl output arguments, socket options, or BPF buffers) where fields
   or structure padding might not be fully initialized/zeroed.
2. Conditional logic or branching that depends on potentially uninitialized variables
   or struct fields.
3. Allocation or initialization of complex data structures where uninitialized fields
   could be read later in reachable code paths.
4. Bounds checks, lengths, or logic in a way that allows surrounding code to access
   uninitialized bytes of existing buffers.

Set NeedsKMSAN=false IF:
- The code changes primarily risk out-of-bounds access, array overflows, NULL pointer
  dereferences, locking deadlocks, or use-after-free bugs (these are already caught
  by KASAN, LOCKDEP, or standard bug detectors).
- All stack/heap structures touched or introduced by the patch are fully zeroed
  or initialized (e.g. using = {0}, memset, kzalloc) before being read or copied.
- The patch does not introduce any risk of uninitialized memory usage or info-leaks.

Use your code access tools to inspect the surrounding code if necessary, then provide
detailed KMSANReasoning contrasting KASAN vs KMSAN applicability for this patch.
Prefer calling several tools at the same time to save round-trips.


Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.

For your convenience, here is the diff of the changes:
commit e09e4ad11f933fa9ab21debd310aeb220f3f52c0
Author: syz-cluster <triage@syzkaller.com>
Date:   Mon Aug 31 14:02:28 2026 +0000

    syz-cluster: applied patch under review

diff --git a/arch/x86/include/asm/alternative.h b/arch/x86/include/asm/alternative.h
index 08af86ef090a8..2ef4937411860 100644
--- a/arch/x86/include/asm/alternative.h
+++ b/arch/x86/include/asm/alternative.h
@@ -83,7 +83,7 @@ extern void callthunks_patch_builtin_calls(void);
 extern void callthunks_patch_module_calls(struct callthunk_sites *sites,
 					  struct module *mod);
 extern void *callthunks_translate_call_dest(void *dest);
-extern int x86_call_depth_emit_accounting(u8 **pprog, void *func, void *ip);
+extern int x86_call_depth_emit_accounting(u8 *insn_buff, void *func, void *ip);
 #else
 static __always_inline void callthunks_patch_builtin_calls(void) {}
 static __always_inline void
@@ -93,7 +93,7 @@ static __always_inline void *callthunks_translate_call_dest(void *dest)
 {
 	return dest;
 }
-static __always_inline int x86_call_depth_emit_accounting(u8 **pprog,
+static __always_inline int x86_call_depth_emit_accounting(u8 *insn_buff,
 							  void *func, void *ip)
 {
 	return 0;
diff --git a/arch/x86/include/asm/text-patching.h b/arch/x86/include/asm/text-patching.h
index f2d142a0a862e..a0a7c778b0a2a 100644
--- a/arch/x86/include/asm/text-patching.h
+++ b/arch/x86/include/asm/text-patching.h
@@ -13,6 +13,8 @@
  */
 #define TEXT_POKE_MAX_OPCODE_SIZE	5
 
+#define MAX_PATCH_LEN	(255-1)
+
 extern void text_poke_early(void *addr, const void *opcode, size_t len);
 
 extern void text_poke_apply_relocation(u8 *buf, const u8 * const instr, size_t instrlen, u8 *repl, size_t repl_len);
diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c
index 91b1cdd165699..5f9989c8559ea 100644
--- a/arch/x86/kernel/alternative.c
+++ b/arch/x86/kernel/alternative.c
@@ -18,8 +18,6 @@ int __read_mostly alternatives_patched;
 
 EXPORT_SYMBOL_GPL(alternatives_patched);
 
-#define MAX_PATCH_LEN (255-1)
-
 #define DA_ALL		(~0)
 #define DA_ALT		0x01
 #define DA_RET		0x02
diff --git a/arch/x86/kernel/callthunks.c b/arch/x86/kernel/callthunks.c
index e37728f703227..15b221d06901c 100644
--- a/arch/x86/kernel/callthunks.c
+++ b/arch/x86/kernel/callthunks.c
@@ -23,8 +23,6 @@
 
 static int __initdata_or_module debug_callthunks;
 
-#define MAX_PATCH_LEN (255-1)
-
 #define prdbg(fmt, args...)					\
 do {								\
 	if (debug_callthunks)					\
@@ -298,10 +296,9 @@ static bool is_callthunk(void *addr)
 	return !bcmp(pad, insn_buff, tmpl_size);
 }
 
-int x86_call_depth_emit_accounting(u8 **pprog, void *func, void *ip)
+int x86_call_depth_emit_accounting(u8 *insn_buff, void *func, void *ip)
 {
 	unsigned int tmpl_size = SKL_TMPL_SIZE;
-	u8 insn_buff[MAX_PATCH_LEN];
 
 	if (!thunks_initialized)
 		return 0;
@@ -313,8 +310,6 @@ int x86_call_depth_emit_accounting(u8 **pprog, void *func, void *ip)
 	memcpy(insn_buff, skl_call_thunk_template, tmpl_size);
 	text_poke_apply_relocation(insn_buff, ip, tmpl_size, skl_call_thunk_template, tmpl_size);
 
-	memcpy(*pprog, insn_buff, tmpl_size);
-	*pprog += tmpl_size;
 	return tmpl_size;
 }
 #endif
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index 48429fae06410..2ec734c69b693 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -23,6 +23,11 @@
 
 static bool all_callee_regs_used[4] = {true, true, true, true};
 
+struct jit_emit_context {
+	u8 *prog;
+	bool dry_run;
+};
+
 static u8 *emit_code(u8 *ptr, u32 bytes, unsigned int len)
 {
 	if (len == 1)
@@ -36,8 +41,16 @@ static u8 *emit_code(u8 *ptr, u32 bytes, unsigned int len)
 	return ptr + len;
 }
 
+static void emit_code_jit(struct jit_emit_context *jit, u32 bytes, unsigned int len)
+{
+	if (jit->dry_run)
+		jit->prog += len;
+	else
+		jit->prog = emit_code(jit->prog, bytes, len);
+}
+
 #define EMIT(bytes, len) \
-	do { prog = emit_code(prog, bytes, len); } while (0)
+	emit_code_jit(jit, bytes, len)
 
 #define EMIT1(b1)		EMIT(b1, 1)
 #define EMIT2(b1, b2)		EMIT((b1) + ((b2) << 8), 2)
@@ -149,6 +162,19 @@ static int bpf_size_to_x86_bytes(int bpf_size)
 		return 0;
 }
 
+static int bpf_call_depth_emit_accounting(struct jit_emit_context *jit, void *func, void *ip)
+{
+	u8 insn_buff[MAX_PATCH_LEN];
+	int size;
+
+	size = x86_call_depth_emit_accounting(insn_buff, func, ip);
+	if (!jit->dry_run)
+		memcpy(jit->prog, insn_buff, size);
+
+	jit->prog += size;
+	return size;
+}
+
 /*
  * List of x86 cond jumps opcodes (. + s8)
  * Add 0x10 (and an extra 0x0f) to generate far jumps (. + s32)
@@ -309,6 +335,8 @@ int bpf_arch_text_invalidate(void *dst, size_t len)
 }
 
 struct jit_context {
+	struct jit_emit_context jit;
+
 	int cleanup_addr; /* Epilogue code offset */
 
 	/*
@@ -329,34 +357,23 @@ struct jit_context {
 /* Number of bytes that will be skipped on tailcall */
 #define X86_TAIL_CALL_OFFSET	(12 + ENDBR_INSN_SIZE)
 
-static void push_r9(u8 **pprog)
+static void push_r9(struct jit_emit_context *jit)
 {
-	u8 *prog = *pprog;
-
 	EMIT2(0x41, 0x51);   /* push r9 */
-	*pprog = prog;
 }
 
-static void pop_r9(u8 **pprog)
+static void pop_r9(struct jit_emit_context *jit)
 {
-	u8 *prog = *pprog;
-
 	EMIT2(0x41, 0x59);   /* pop r9 */
-	*pprog = prog;
 }
 
-static void push_r12(u8 **pprog)
+static void push_r12(struct jit_emit_context *jit)
 {
-	u8 *prog = *pprog;
-
 	EMIT2(0x41, 0x54);   /* push r12 */
-	*pprog = prog;
 }
 
-static void push_callee_regs(u8 **pprog, bool *callee_regs_used)
+static void push_callee_regs(struct jit_emit_context *jit, bool *callee_regs_used)
 {
-	u8 *prog = *pprog;
-
 	if (callee_regs_used[0])
 		EMIT1(0x53);         /* push rbx */
 	if (callee_regs_used[1])
@@ -365,21 +382,15 @@ static void push_callee_regs(u8 **pprog, bool *callee_regs_used)
 		EMIT2(0x41, 0x56);   /* push r14 */
 	if (callee_regs_used[3])
 		EMIT2(0x41, 0x57);   /* push r15 */
-	*pprog = prog;
 }
 
-static void pop_r12(u8 **pprog)
+static void pop_r12(struct jit_emit_context *jit)
 {
-	u8 *prog = *pprog;
-
 	EMIT2(0x41, 0x5C);   /* pop r12 */
-	*pprog = prog;
 }
 
-static void pop_callee_regs(u8 **pprog, bool *callee_regs_used)
+static void pop_callee_regs(struct jit_emit_context *jit, bool *callee_regs_used)
 {
-	u8 *prog = *pprog;
-
 	if (callee_regs_used[3])
 		EMIT2(0x41, 0x5F);   /* pop r15 */
 	if (callee_regs_used[2])
@@ -388,40 +399,32 @@ static void pop_callee_regs(u8 **pprog, bool *callee_regs_used)
 		EMIT2(0x41, 0x5D);   /* pop r13 */
 	if (callee_regs_used[0])
 		EMIT1(0x5B);         /* pop rbx */
-	*pprog = prog;
 }
 
 /* add rsp, depth */
-static void emit_add_rsp(u8 **pprog, u16 depth)
+static void emit_add_rsp(struct jit_emit_context *jit, u16 depth)
 {
-	u8 *prog = *pprog;
-
 	if (!depth)
 		return;
 	if (is_imm8(depth))
 		EMIT4(0x48, 0x83, 0xC4, depth); /* add rsp, imm8 */
 	else
 		EMIT3_off32(0x48, 0x81, 0xC4, depth); /* add rsp, imm32 */
-	*pprog = prog;
 }
 
 /* sub rsp, depth */
-static void emit_sub_rsp(u8 **pprog, u16 depth)
+static void emit_sub_rsp(struct jit_emit_context *jit, u16 depth)
 {
-	u8 *prog = *pprog;
-
 	if (!depth)
 		return;
 	if (is_imm8(depth))
 		EMIT4(0x48, 0x83, 0xEC, depth); /* sub rsp, imm8 */
 	else
 		EMIT3_off32(0x48, 0x81, 0xEC, depth); /* sub rsp, imm32 */
-	*pprog = prog;
 }
 
-static void emit_nops(u8 **pprog, int len)
+static void emit_nops(struct jit_emit_context *jit, int len)
 {
-	u8 *prog = *pprog;
 	int i, noplen;
 
 	while (len > 0) {
@@ -434,72 +437,56 @@ static void emit_nops(u8 **pprog, int len)
 			EMIT1(x86_nops[noplen][i]);
 		len -= noplen;
 	}
-
-	*pprog = prog;
 }
 
 /*
  * Emit the various CFI preambles, see asm/cfi.h and the comments about FineIBT
  * in arch/x86/kernel/alternative.c
  */
-static int emit_call(u8 **prog, void *func, void *ip);
+static int emit_call(struct jit_emit_context *jit, void *func, void *ip);
 
-static void emit_fineibt(u8 **pprog, u8 *ip, u32 hash, int arity)
+static void emit_fineibt(struct jit_emit_context *jit, u8 *ip, u32 hash, int arity)
 {
-	u8 *prog = *pprog;
-
 	EMIT_ENDBR();
 	EMIT1_off32(0x2d, hash);			/* subl $hash, %eax	*/
 	if (cfi_bhi) {
 		EMIT2(0x2e, 0x2e);			/* cs cs */
-		emit_call(&prog, __bhi_args[arity], ip + 11);
+		emit_call(jit, __bhi_args[arity], ip + 11);
 	} else {
 		EMIT3_off32(0x2e, 0x0f, 0x85, 3);	/* jne.d32,pn 3		*/
 	}
 	EMIT_ENDBR_POISON();
-
-	*pprog = prog;
 }
 
-static void emit_kcfi(u8 **pprog, u32 hash)
+static void emit_kcfi(struct jit_emit_context *jit, u32 hash)
 {
-	u8 *prog = *pprog;
-
 	EMIT1_off32(0xb8, hash);			/* movl $hash, %eax	*/
 #ifdef CONFIG_CALL_PADDING
 	for (int i = 0; i < CONFIG_FUNCTION_PADDING_CFI; i++)
 		EMIT1(0x90);
 #endif
 	EMIT_ENDBR();
-
-	*pprog = prog;
 }
 
-static void emit_cfi(u8 **pprog, u8 *ip, u32 hash, int arity)
+static void emit_cfi(struct jit_emit_context *jit, u8 *ip, u32 hash, int arity)
 {
-	u8 *prog = *pprog;
-
 	switch (cfi_mode) {
 	case CFI_FINEIBT:
-		emit_fineibt(&prog, ip, hash, arity);
+		emit_fineibt(jit, ip, hash, arity);
 		break;
 
 	case CFI_KCFI:
-		emit_kcfi(&prog, hash);
+		emit_kcfi(jit, hash);
 		break;
 
 	default:
 		EMIT_ENDBR();
 		break;
 	}
-
-	*pprog = prog;
 }
 
-static void emit_prologue_tail_call(u8 **pprog, bool is_subprog)
+static void emit_prologue_tail_call(struct jit_emit_context *jit, bool is_subprog)
 {
-	u8 *prog = *pprog;
-
 	if (!is_subprog) {
 		/* cmp rax, MAX_TAIL_CALL_CNT */
 		EMIT4(0x48, 0x83, 0xF8, MAX_TAIL_CALL_CNT);
@@ -523,8 +510,6 @@ static void emit_prologue_tail_call(u8 **pprog, bool is_subprog)
 		EMIT1(0x50);             /* push rax */
 		EMIT1(0x50);             /* push rax */
 	}
-
-	*pprog = prog;
 }
 
 /*
@@ -532,21 +517,19 @@ static void emit_prologue_tail_call(u8 **pprog, bool is_subprog)
  * bpf_tail_call helper will skip the first X86_TAIL_CALL_OFFSET bytes
  * while jumping to another program
  */
-static void emit_prologue(u8 **pprog, u8 *ip, u32 stack_depth, bool ebpf_from_cbpf,
-			  bool tail_call_reachable, bool is_subprog,
+static void emit_prologue(struct jit_emit_context *jit, u8 *ip, u32 stack_depth,
+			  bool ebpf_from_cbpf, bool tail_call_reachable, bool is_subprog,
 			  bool is_exception_cb)
 {
-	u8 *prog = *pprog;
-
 	if (is_subprog) {
-		emit_cfi(&prog, ip, cfi_bpf_subprog_hash, 5);
+		emit_cfi(jit, ip, cfi_bpf_subprog_hash, 5);
 	} else {
-		emit_cfi(&prog, ip, cfi_bpf_hash, 1);
+		emit_cfi(jit, ip, cfi_bpf_hash, 1);
 	}
 	/* BPF trampoline can be made to work without these nops,
 	 * but let's waste 5 bytes for now and optimize later
 	 */
-	emit_nops(&prog, X86_PATCH_SIZE);
+	emit_nops(jit, X86_PATCH_SIZE);
 	if (!ebpf_from_cbpf) {
 		if (tail_call_reachable && !is_subprog)
 			/* When it's the entry of the whole tailcall context,
@@ -555,7 +538,7 @@ static void emit_prologue(u8 **pprog, u8 *ip, u32 stack_depth, bool ebpf_from_cb
 			EMIT3(0x48, 0x31, 0xC0); /* xor rax, rax */
 		else
 			/* Keep the same instruction layout. */
-			emit_nops(&prog, 3);     /* nop3 */
+			emit_nops(jit, 3);     /* nop3 */
 	}
 	/* Exception callback receives FP as third parameter */
 	if (is_exception_cb) {
@@ -565,8 +548,8 @@ static void emit_prologue(u8 **pprog, u8 *ip, u32 stack_depth, bool ebpf_from_cb
 		 * first restore those callee-saved regs from stack, before
 		 * reusing the stack frame.
 		 */
-		pop_callee_regs(&prog, all_callee_regs_used);
-		pop_r12(&prog);
+		pop_callee_regs(jit, all_callee_regs_used);
+		pop_r12(jit);
 		/* Reset the stack frame. */
 		EMIT3(0x48, 0x89, 0xEC); /* mov rsp, rbp */
 	} else {
@@ -581,40 +564,38 @@ static void emit_prologue(u8 **pprog, u8 *ip, u32 stack_depth, bool ebpf_from_cb
 	if (stack_depth)
 		EMIT3_off32(0x48, 0x81, 0xEC, round_up(stack_depth, 8));
 	if (tail_call_reachable)
-		emit_prologue_tail_call(&prog, is_subprog);
-	*pprog = prog;
+		emit_prologue_tail_call(jit, is_subprog);
 }
 
-static int emit_patch(u8 **pprog, void *func, void *ip, u8 opcode)
+static int emit_patch(struct jit_emit_context *jit, void *func, void *ip, u8 opcode)
 {
-	u8 *prog = *pprog;
 	s64 offset;
 
 	offset = func - (ip + X86_PATCH_SIZE);
-	if (!is_simm32(offset)) {
+	/* We do not have meaningful ip value in the dry run, skip the check. */
+	if (!jit->dry_run && !is_simm32(offset)) {
 		pr_err("Target call %p is out of range\n", func);
 		return -ERANGE;
 	}
 	EMIT1_off32(opcode, offset);
-	*pprog = prog;
 	return 0;
 }
 
-static int emit_call(u8 **pprog, void *func, void *ip)
+static int emit_call(struct jit_emit_context *jit, void *func, void *ip)
 {
-	return emit_patch(pprog, func, ip, 0xE8);
+	return emit_patch(jit, func, ip, 0xE8);
 }
 
-static int emit_rsb_call(u8 **pprog, void *func, void *ip)
+static int emit_rsb_call(struct jit_emit_context *jit, void *func, void *ip)
 {
 	OPTIMIZER_HIDE_VAR(func);
-	ip += x86_call_depth_emit_accounting(pprog, func, ip);
-	return emit_patch(pprog, func, ip, 0xE8);
+	ip += bpf_call_depth_emit_accounting(jit, func, ip);
+	return emit_patch(jit, func, ip, 0xE8);
 }
 
-static int emit_jump(u8 **pprog, void *func, void *ip)
+static int emit_jump(struct jit_emit_context *jit, void *func, void *ip)
 {
-	return emit_patch(pprog, func, ip, 0xE9);
+	return emit_patch(jit, func, ip, 0xE9);
 }
 
 static int __bpf_arch_text_poke(void *ip, enum bpf_text_poke_type old_t,
@@ -622,27 +603,27 @@ static int __bpf_arch_text_poke(void *ip, enum bpf_text_poke_type old_t,
 				void *old_addr, void *new_addr)
 {
 	const u8 *nop_insn = x86_nops[5];
+	struct jit_emit_context jit = {};
 	u8 old_insn[X86_PATCH_SIZE];
 	u8 new_insn[X86_PATCH_SIZE];
-	u8 *prog;
 	int ret;
 
 	memcpy(old_insn, nop_insn, X86_PATCH_SIZE);
 	if (old_t != BPF_MOD_NOP && old_addr) {
-		prog = old_insn;
+		jit.prog = old_insn;
 		ret = old_t == BPF_MOD_CALL ?
-		      emit_call(&prog, old_addr, ip) :
-		      emit_jump(&prog, old_addr, ip);
+		      emit_call(&jit, old_addr, ip) :
+		      emit_jump(&jit, old_addr, ip);
 		if (ret)
 			return ret;
 	}
 
 	memcpy(new_insn, nop_insn, X86_PATCH_SIZE);
 	if (new_t != BPF_MOD_NOP && new_addr) {
-		prog = new_insn;
+		jit.prog = new_insn;
 		ret = new_t == BPF_MOD_CALL ?
-		      emit_call(&prog, new_addr, ip) :
-		      emit_jump(&prog, new_addr, ip);
+		      emit_call(&jit, new_addr, ip) :
+		      emit_jump(&jit, new_addr, ip);
 		if (ret)
 			return ret;
 	}
@@ -682,58 +663,47 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type old_t,
 
 #define EMIT_LFENCE()	EMIT3(0x0F, 0xAE, 0xE8)
 
-static void __emit_indirect_jump(u8 **pprog, int reg, bool ereg)
+static void __emit_indirect_jump(struct jit_emit_context *jit, int reg, bool ereg)
 {
-	u8 *prog = *pprog;
-
 	if (ereg)
 		EMIT1(0x41);
 
 	EMIT2(0xFF, 0xE0 + reg);
-
-	*pprog = prog;
 }
 
-static void emit_indirect_jump(u8 **pprog, int bpf_reg, u8 *ip)
+static void emit_indirect_jump(struct jit_emit_context *jit, int bpf_reg, u8 *ip)
 {
-	u8 *prog = *pprog;
 	int reg = reg2hex[bpf_reg];
 	bool ereg = is_ereg(bpf_reg);
 
 	if (cpu_feature_enabled(X86_FEATURE_INDIRECT_THUNK_ITS)) {
 		OPTIMIZER_HIDE_VAR(reg);
-		emit_jump(&prog, its_static_thunk(reg + 8*ereg), ip);
+		emit_jump(jit, its_static_thunk(reg + 8*ereg), ip);
 	} else if (cpu_feature_enabled(X86_FEATURE_RETPOLINE_LFENCE)) {
 		EMIT_LFENCE();
-		__emit_indirect_jump(&prog, reg, ereg);
+		__emit_indirect_jump(jit, reg, ereg);
 	} else if (cpu_feature_enabled(X86_FEATURE_RETPOLINE)) {
 		OPTIMIZER_HIDE_VAR(reg);
 		if (cpu_feature_enabled(X86_FEATURE_CALL_DEPTH))
-			emit_jump(&prog, &__x86_indirect_jump_thunk_array[reg + 8*ereg], ip);
+			emit_jump(jit, &__x86_indirect_jump_thunk_array[reg + 8*ereg], ip);
 		else
-			emit_jump(&prog, &__x86_indirect_thunk_array[reg + 8*ereg], ip);
+			emit_jump(jit, &__x86_indirect_thunk_array[reg + 8*ereg], ip);
 	} else {
-		__emit_indirect_jump(&prog, reg, ereg);
+		__emit_indirect_jump(jit, reg, ereg);
 		if (IS_ENABLED(CONFIG_MITIGATION_RETPOLINE) || IS_ENABLED(CONFIG_MITIGATION_SLS))
 			EMIT1(0xCC);		/* int3 */
 	}
-
-	*pprog = prog;
 }
 
-static void emit_return(u8 **pprog, u8 *ip)
+static void emit_return(struct jit_emit_context *jit, u8 *ip)
 {
-	u8 *prog = *pprog;
-
 	if (cpu_wants_rethunk()) {
-		emit_jump(&prog, x86_return_thunk, ip);
+		emit_jump(jit, x86_return_thunk, ip);
 	} else {
 		EMIT1(0xC3);		/* ret */
 		if (IS_ENABLED(CONFIG_MITIGATION_SLS))
 			EMIT1(0xCC);	/* int3 */
 	}
-
-	*pprog = prog;
 }
 
 #define BPF_TAIL_CALL_CNT_PTR_STACK_OFF(stack)	(-16 - round_up(stack, 8))
@@ -753,12 +723,13 @@ static void emit_return(u8 **pprog, u8 *ip)
  * out:
  */
 static void emit_bpf_tail_call_indirect(struct bpf_prog *bpf_prog,
-					u8 **pprog, bool *callee_regs_used,
+					bool *callee_regs_used,
 					u32 stack_depth, u8 *ip,
 					struct jit_context *ctx)
 {
 	int tcc_ptr_off = BPF_TAIL_CALL_CNT_PTR_STACK_OFF(stack_depth);
-	u8 *prog = *pprog, *start = *pprog;
+	struct jit_emit_context *jit = &ctx->jit;
+	u8 *start = jit->prog;
 	int offset;
 
 	/*
@@ -775,7 +746,7 @@ static void emit_bpf_tail_call_indirect(struct bpf_prog *bpf_prog,
 	EMIT3(0x39, 0x56,                         /* cmp dword ptr [rsi + 16], edx */
 	      offsetof(struct bpf_array, map.max_entries));
 
-	offset = ctx->tail_call_indirect_label - (prog + 2 - start);
+	offset = ctx->tail_call_indirect_label - (jit->prog + 2 - start);
 	EMIT2(X86_JBE, offset);                   /* jbe out */
 
 	/*
@@ -785,7 +756,7 @@ static void emit_bpf_tail_call_indirect(struct bpf_prog *bpf_prog,
 	EMIT3_off32(0x48, 0x8B, 0x85, tcc_ptr_off); /* mov rax, qword ptr [rbp - tcc_ptr_off] */
 	EMIT4(0x48, 0x83, 0x38, MAX_TAIL_CALL_CNT); /* cmp qword ptr [rax], MAX_TAIL_CALL_CNT */
 
-	offset = ctx->tail_call_indirect_label - (prog + 2 - start);
+	offset = ctx->tail_call_indirect_label - (jit->prog + 2 - start);
 	EMIT2(X86_JAE, offset);                   /* jae out */
 
 	/* prog = array->ptrs[index]; */
@@ -798,19 +769,19 @@ static void emit_bpf_tail_call_indirect(struct bpf_prog *bpf_prog,
 	 */
 	EMIT3(0x48, 0x85, 0xC9);                  /* test rcx,rcx */
 
-	offset = ctx->tail_call_indirect_label - (prog + 2 - start);
+	offset = ctx->tail_call_indirect_label - (jit->prog + 2 - start);
 	EMIT2(X86_JE, offset);                    /* je out */
 
 	/* Inc tail_call_cnt if the slot is populated. */
 	EMIT4(0x48, 0x83, 0x00, 0x01);            /* add qword ptr [rax], 1 */
 
 	if (bpf_prog->aux->exception_boundary) {
-		pop_callee_regs(&prog, all_callee_regs_used);
-		pop_r12(&prog);
+		pop_callee_regs(jit, all_callee_regs_used);
+		pop_r12(jit);
 	} else {
-		pop_callee_regs(&prog, callee_regs_used);
+		pop_callee_regs(jit, callee_regs_used);
 		if (bpf_arena_get_kern_vm_start(bpf_prog->aux->arena))
-			pop_r12(&prog);
+			pop_r12(jit);
 	}
 
 	/* Pop tail_call_cnt_ptr. */
@@ -833,21 +804,21 @@ static void emit_bpf_tail_call_indirect(struct bpf_prog *bpf_prog,
 	 * rdi == ctx (1st arg)
 	 * rcx == prog->bpf_func + X86_TAIL_CALL_OFFSET
 	 */
-	emit_indirect_jump(&prog, BPF_REG_4 /* R4 -> rcx */, ip + (prog - start));
+	emit_indirect_jump(jit, BPF_REG_4 /* R4 -> rcx */, ip + (jit->prog - start));
 
 	/* out: */
-	ctx->tail_call_indirect_label = prog - start;
-	*pprog = prog;
+	ctx->tail_call_indirect_label = jit->prog - start;
 }
 
 static void emit_bpf_tail_call_direct(struct bpf_prog *bpf_prog,
 				      struct bpf_jit_poke_descriptor *poke,
-				      u8 **pprog, u8 *ip,
+				      u8 *ip,
 				      bool *callee_regs_used, u32 stack_depth,
 				      struct jit_context *ctx)
 {
+	struct jit_emit_context *jit = &ctx->jit;
 	int tcc_ptr_off = BPF_TAIL_CALL_CNT_PTR_STACK_OFF(stack_depth);
-	u8 *prog = *pprog, *start = *pprog;
+	u8 *start = jit->prog;
 	int offset;
 
 	/*
@@ -857,27 +828,27 @@ static void emit_bpf_tail_call_direct(struct bpf_prog *bpf_prog,
 	EMIT3_off32(0x48, 0x8B, 0x85, tcc_ptr_off);   /* mov rax, qword ptr [rbp - tcc_ptr_off] */
 	EMIT4(0x48, 0x83, 0x38, MAX_TAIL_CALL_CNT);   /* cmp qword ptr [rax], MAX_TAIL_CALL_CNT */
 
-	offset = ctx->tail_call_direct_label - (prog + 2 - start);
+	offset = ctx->tail_call_direct_label - (jit->prog + 2 - start);
 	EMIT2(X86_JAE, offset);                       /* jae out */
 
-	poke->tailcall_bypass = ip + (prog - start);
+	poke->tailcall_bypass = ip + (jit->prog - start);
 	poke->adj_off = X86_TAIL_CALL_OFFSET;
 	poke->tailcall_target = ip + ctx->tail_call_direct_label - X86_PATCH_SIZE;
 	poke->bypass_addr = (u8 *)poke->tailcall_target + X86_PATCH_SIZE;
 
-	emit_jump(&prog, (u8 *)poke->tailcall_target + X86_PATCH_SIZE,
+	emit_jump(jit, (u8 *)poke->tailcall_target + X86_PATCH_SIZE,
 		  poke->tailcall_bypass);
 
 	/* Inc tail_call_cnt if the slot is populated. */
 	EMIT4(0x48, 0x83, 0x00, 0x01);                /* add qword ptr [rax], 1 */
 
 	if (bpf_prog->aux->exception_boundary) {
-		pop_callee_regs(&prog, all_callee_regs_used);
-		pop_r12(&prog);
+		pop_callee_regs(jit, all_callee_regs_used);
+		pop_r12(jit);
 	} else {
-		pop_callee_regs(&prog, callee_regs_used);
+		pop_callee_regs(jit, callee_regs_used);
 		if (bpf_arena_get_kern_vm_start(bpf_prog->aux->arena))
-			pop_r12(&prog);
+			pop_r12(jit);
 	}
 
 	/* Pop tail_call_cnt_ptr. */
@@ -889,12 +860,10 @@ static void emit_bpf_tail_call_direct(struct bpf_prog *bpf_prog,
 	if (stack_depth)
 		EMIT3_off32(0x48, 0x81, 0xC4, round_up(stack_depth, 8));
 
-	emit_nops(&prog, X86_PATCH_SIZE);
+	emit_nops(jit, X86_PATCH_SIZE);
 
 	/* out: */
-	ctx->tail_call_direct_label = prog - start;
-
-	*pprog = prog;
+	ctx->tail_call_direct_label = jit->prog - start;
 }
 
 static void bpf_tail_call_direct_fixup(struct bpf_prog *prog)
@@ -935,10 +904,9 @@ static void bpf_tail_call_direct_fixup(struct bpf_prog *prog)
 	}
 }
 
-static void emit_mov_imm32(u8 **pprog, bool sign_propagate,
+static void emit_mov_imm32(struct jit_emit_context *jit, bool sign_propagate,
 			   u32 dst_reg, const u32 imm32)
 {
-	u8 *prog = *pprog;
 	u8 b1, b2, b3;
 
 	/*
@@ -972,14 +940,12 @@ static void emit_mov_imm32(u8 **pprog, bool sign_propagate,
 		EMIT1(add_1mod(0x40, dst_reg));
 	EMIT1_off32(add_1reg(0xB8, dst_reg), imm32);
 done:
-	*pprog = prog;
 }
 
-static void emit_mov_imm64(u8 **pprog, u32 dst_reg,
+static void emit_mov_imm64(struct jit_emit_context *jit, u32 dst_reg,
 			   const u32 imm32_hi, const u32 imm32_lo)
 {
 	u64 imm64 = ((u64)imm32_hi << 32) | (u32)imm32_lo;
-	u8 *prog = *pprog;
 
 	if (is_uimm32(imm64)) {
 		/*
@@ -988,23 +954,19 @@ static void emit_mov_imm64(u8 **pprog, u32 dst_reg,
 		 * directly, so save couple of bytes by just doing
 		 * 'mov %eax, imm32' instead.
 		 */
-		emit_mov_imm32(&prog, false, dst_reg, imm32_lo);
+		emit_mov_imm32(jit, false, dst_reg, imm32_lo);
 	} else if (is_simm32(imm64)) {
-		emit_mov_imm32(&prog, true, dst_reg, imm32_lo);
+		emit_mov_imm32(jit, true, dst_reg, imm32_lo);
 	} else {
 		/* movabsq rax, imm64 */
 		EMIT2(add_1mod(0x48, dst_reg), add_1reg(0xB8, dst_reg));
 		EMIT(imm32_lo, 4);
 		EMIT(imm32_hi, 4);
 	}
-
-	*pprog = prog;
 }
 
-static void emit_mov_reg(u8 **pprog, bool is64, u32 dst_reg, u32 src_reg)
+static void emit_mov_reg(struct jit_emit_context *jit, bool is64, u32 dst_reg, u32 src_reg)
 {
-	u8 *prog = *pprog;
-
 	if (is64) {
 		/* mov dst, src */
 		EMIT_mov(dst_reg, src_reg);
@@ -1014,15 +976,11 @@ static void emit_mov_reg(u8 **pprog, bool is64, u32 dst_reg, u32 src_reg)
 			EMIT1(add_2mod(0x40, dst_reg, src_reg));
 		EMIT2(0x89, add_2reg(0xC0, dst_reg, src_reg));
 	}
-
-	*pprog = prog;
 }
 
-static void emit_movsx_reg(u8 **pprog, int num_bits, bool is64, u32 dst_reg,
+static void emit_movsx_reg(struct jit_emit_context *jit, int num_bits, bool is64, u32 dst_reg,
 			   u32 src_reg)
 {
-	u8 *prog = *pprog;
-
 	if (is64) {
 		/* movs[b,w,l]q dst, src */
 		if (num_bits == 8)
@@ -1046,15 +1004,11 @@ static void emit_movsx_reg(u8 **pprog, int num_bits, bool is64, u32 dst_reg,
 			      add_2reg(0xC0, src_reg, dst_reg));
 		}
 	}
-
-	*pprog = prog;
 }
 
 /* Emit the suffix (ModR/M etc) for addressing *(ptr_reg + off) and val_reg */
-static void emit_insn_suffix(u8 **pprog, u32 ptr_reg, u32 val_reg, int off)
+static void emit_insn_suffix(struct jit_emit_context *jit, u32 ptr_reg, u32 val_reg, int off)
 {
-	u8 *prog = *pprog;
-
 	if (is_imm8(off)) {
 		/* 1-byte signed displacement.
 		 *
@@ -1067,54 +1021,43 @@ static void emit_insn_suffix(u8 **pprog, u32 ptr_reg, u32 val_reg, int off)
 		/* 4-byte signed displacement */
 		EMIT1_off32(add_2reg(0x80, ptr_reg, val_reg), off);
 	}
-	*pprog = prog;
 }
 
-static void emit_insn_suffix_SIB(u8 **pprog, u32 ptr_reg, u32 val_reg, u32 index_reg, int off)
+static void emit_insn_suffix_SIB(struct jit_emit_context *jit, u32 ptr_reg, u32 val_reg,
+				 u32 index_reg, int off)
 {
-	u8 *prog = *pprog;
-
 	if (is_imm8(off)) {
 		EMIT3(add_2reg(0x44, BPF_REG_0, val_reg), add_2reg(0, ptr_reg, index_reg) /* SIB */, off);
 	} else {
 		EMIT2_off32(add_2reg(0x84, BPF_REG_0, val_reg), add_2reg(0, ptr_reg, index_reg) /* SIB */, off);
 	}
-	*pprog = prog;
 }
 
 /*
  * Emit a REX byte if it will be necessary to address these registers
  */
-static void maybe_emit_mod(u8 **pprog, u32 dst_reg, u32 src_reg, bool is64)
+static void maybe_emit_mod(struct jit_emit_context *jit, u32 dst_reg, u32 src_reg, bool is64)
 {
-	u8 *prog = *pprog;
-
 	if (is64)
 		EMIT1(add_2mod(0x48, dst_reg, src_reg));
 	else if (is_ereg(dst_reg) || is_ereg(src_reg))
 		EMIT1(add_2mod(0x40, dst_reg, src_reg));
-	*pprog = prog;
 }
 
 /*
  * Similar version of maybe_emit_mod() for a single register
  */
-static void maybe_emit_1mod(u8 **pprog, u32 reg, bool is64)
+static void maybe_emit_1mod(struct jit_emit_context *jit, u32 reg, bool is64)
 {
-	u8 *prog = *pprog;
-
 	if (is64)
 		EMIT1(add_1mod(0x48, reg));
 	else if (is_ereg(reg))
 		EMIT1(add_1mod(0x40, reg));
-	*pprog = prog;
 }
 
 /* LDX: dst_reg = *(u8*)(src_reg + off) */
-static void emit_ldx(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, int off)
+static void emit_ldx(struct jit_emit_context *jit, u32 size, u32 dst_reg, u32 src_reg, int off)
 {
-	u8 *prog = *pprog;
-
 	switch (size) {
 	case BPF_B:
 		/* Emit 'movzx rax, byte ptr [rax + off]' */
@@ -1136,15 +1079,12 @@ static void emit_ldx(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, int off)
 		EMIT2(add_2mod(0x48, src_reg, dst_reg), 0x8B);
 		break;
 	}
-	emit_insn_suffix(&prog, src_reg, dst_reg, off);
-	*pprog = prog;
+	emit_insn_suffix(jit, src_reg, dst_reg, off);
 }
 
 /* LDSX: dst_reg = *(s8*)(src_reg + off) */
-static void emit_ldsx(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, int off)
+static void emit_ldsx(struct jit_emit_context *jit, u32 size, u32 dst_reg, u32 src_reg, int off)
 {
-	u8 *prog = *pprog;
-
 	switch (size) {
 	case BPF_B:
 		/* Emit 'movsx rax, byte ptr [rax + off]' */
@@ -1159,14 +1099,12 @@ static void emit_ldsx(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, int off)
 		EMIT2(add_2mod(0x48, src_reg, dst_reg), 0x63);
 		break;
 	}
-	emit_insn_suffix(&prog, src_reg, dst_reg, off);
-	*pprog = prog;
+	emit_insn_suffix(jit, src_reg, dst_reg, off);
 }
 
-static void emit_ldx_index(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, u32 index_reg, int off)
+static void emit_ldx_index(struct jit_emit_context *jit, u32 size, u32 dst_reg, u32 src_reg,
+			   u32 index_reg, int off)
 {
-	u8 *prog = *pprog;
-
 	switch (size) {
 	case BPF_B:
 		/* movzx rax, byte ptr [rax + r12 + off] */
@@ -1185,14 +1123,12 @@ static void emit_ldx_index(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, u32 i
 		EMIT2(add_3mod(0x48, src_reg, dst_reg, index_reg), 0x8B);
 		break;
 	}
-	emit_insn_suffix_SIB(&prog, src_reg, dst_reg, index_reg, off);
-	*pprog = prog;
+	emit_insn_suffix_SIB(jit, src_reg, dst_reg, index_reg, off);
 }
 
-static void emit_ldsx_index(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, u32 index_reg, int off)
+static void emit_ldsx_index(struct jit_emit_context *jit, u32 size, u32 dst_reg, u32 src_reg,
+			    u32 index_reg, int off)
 {
-	u8 *prog = *pprog;
-
 	switch (size) {
 	case BPF_B:
 		/* movsx rax, byte ptr [rax + r12 + off] */
@@ -1207,25 +1143,22 @@ static void emit_ldsx_index(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, u32
 		EMIT2(add_3mod(0x48, src_reg, dst_reg, index_reg), 0x63);
 		break;
 	}
-	emit_insn_suffix_SIB(&prog, src_reg, dst_reg, index_reg, off);
-	*pprog = prog;
+	emit_insn_suffix_SIB(jit, src_reg, dst_reg, index_reg, off);
 }
 
-static void emit_ldx_r12(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, int off)
+static void emit_ldx_r12(struct jit_emit_context *jit, u32 size, u32 dst_reg, u32 src_reg, int off)
 {
-	emit_ldx_index(pprog, size, dst_reg, src_reg, X86_REG_R12, off);
+	emit_ldx_index(jit, size, dst_reg, src_reg, X86_REG_R12, off);
 }
 
-static void emit_ldsx_r12(u8 **prog, u32 size, u32 dst_reg, u32 src_reg, int off)
+static void emit_ldsx_r12(struct jit_emit_context *jit, u32 size, u32 dst_reg, u32 src_reg, int off)
 {
-	emit_ldsx_index(prog, size, dst_reg, src_reg, X86_REG_R12, off);
+	emit_ldsx_index(jit, size, dst_reg, src_reg, X86_REG_R12, off);
 }
 
 /* STX: *(u8*)(dst_reg + off) = src_reg */
-static void emit_stx(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, int off)
+static void emit_stx(struct jit_emit_context *jit, u32 size, u32 dst_reg, u32 src_reg, int off)
 {
-	u8 *prog = *pprog;
-
 	switch (size) {
 	case BPF_B:
 		/* Emit 'mov byte ptr [rax + off], al' */
@@ -1251,15 +1184,13 @@ static void emit_stx(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, int off)
 		EMIT2(add_2mod(0x48, dst_reg, src_reg), 0x89);
 		break;
 	}
-	emit_insn_suffix(&prog, dst_reg, src_reg, off);
-	*pprog = prog;
+	emit_insn_suffix(jit, dst_reg, src_reg, off);
 }
 
 /* STX: *(u8*)(dst_reg + index_reg + off) = src_reg */
-static void emit_stx_index(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, u32 index_reg, int off)
+static void emit_stx_index(struct jit_emit_context *jit, u32 size, u32 dst_reg, u32 src_reg,
+			   u32 index_reg, int off)
 {
-	u8 *prog = *pprog;
-
 	switch (size) {
 	case BPF_B:
 		/* mov byte ptr [rax + r12 + off], al */
@@ -1278,20 +1209,18 @@ static void emit_stx_index(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, u32 i
 		EMIT2(add_3mod(0x48, dst_reg, src_reg, index_reg), 0x89);
 		break;
 	}
-	emit_insn_suffix_SIB(&prog, dst_reg, src_reg, index_reg, off);
-	*pprog = prog;
+	emit_insn_suffix_SIB(jit, dst_reg, src_reg, index_reg, off);
 }
 
-static void emit_stx_r12(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, int off)
+static void emit_stx_r12(struct jit_emit_context *jit, u32 size, u32 dst_reg, u32 src_reg, int off)
 {
-	emit_stx_index(pprog, size, dst_reg, src_reg, X86_REG_R12, off);
+	emit_stx_index(jit, size, dst_reg, src_reg, X86_REG_R12, off);
 }
 
 /* ST: *(u8*)(dst_reg + index_reg + off) = imm32 */
-static void emit_st_index(u8 **pprog, u32 size, u32 dst_reg, u32 index_reg, int off, int imm)
+static void emit_st_index(struct jit_emit_context *jit, u32 size, u32 dst_reg, u32 index_reg,
+			  int off, int imm)
 {
-	u8 *prog = *pprog;
-
 	switch (size) {
 	case BPF_B:
 		/* mov byte ptr [rax + r12 + off], imm8 */
@@ -1310,35 +1239,32 @@ static void emit_st_index(u8 **pprog, u32 size, u32 dst_reg, u32 index_reg, int
 		EMIT2(add_3mod(0x48, dst_reg, 0, index_reg), 0xC7);
 		break;
 	}
-	emit_insn_suffix_SIB(&prog, dst_reg, 0, index_reg, off);
+	emit_insn_suffix_SIB(jit, dst_reg, 0, index_reg, off);
 	EMIT(imm, bpf_size_to_x86_bytes(size));
-	*pprog = prog;
 }
 
-static void emit_st_r12(u8 **pprog, u32 size, u32 dst_reg, int off, int imm)
+static void emit_st_r12(struct jit_emit_context *jit, u32 size, u32 dst_reg, int off, int imm)
 {
-	emit_st_index(pprog, size, dst_reg, X86_REG_R12, off, imm);
+	emit_st_index(jit, size, dst_reg, X86_REG_R12, off, imm);
 }
 
-static void emit_store_stack_imm64(u8 **pprog, int reg, int stack_off, u64 imm64)
+static void emit_store_stack_imm64(struct jit_emit_context *jit, int reg, int stack_off, u64 imm64)
 {
 	/*
 	 * mov reg, imm64
 	 * mov QWORD PTR [rbp + stack_off], reg
 	 */
-	emit_mov_imm64(pprog, reg, imm64 >> 32, (u32) imm64);
-	emit_stx(pprog, BPF_DW, BPF_REG_FP, reg, stack_off);
+	emit_mov_imm64(jit, reg, imm64 >> 32, (u32) imm64);
+	emit_stx(jit, BPF_DW, BPF_REG_FP, reg, stack_off);
 }
 
-static int emit_atomic_rmw(u8 **pprog, u32 atomic_op,
+static int emit_atomic_rmw(struct jit_emit_context *jit, u32 atomic_op,
 			   u32 dst_reg, u32 src_reg, s16 off, u8 bpf_size)
 {
-	u8 *prog = *pprog;
-
 	if (atomic_op != BPF_XCHG)
 		EMIT1(0xF0); /* lock prefix */
 
-	maybe_emit_mod(&prog, dst_reg, src_reg, bpf_size == BPF_DW);
+	maybe_emit_mod(jit, dst_reg, src_reg, bpf_size == BPF_DW);
 
 	/* emit opcode */
 	switch (atomic_op) {
@@ -1366,18 +1292,15 @@ static int emit_atomic_rmw(u8 **pprog, u32 atomic_op,
 		return -EFAULT;
 	}
 
-	emit_insn_suffix(&prog, dst_reg, src_reg, off);
+	emit_insn_suffix(jit, dst_reg, src_reg, off);
 
-	*pprog = prog;
 	return 0;
 }
 
-static int emit_atomic_rmw_index(u8 **pprog, u32 atomic_op, u32 size,
+static int emit_atomic_rmw_index(struct jit_emit_context *jit, u32 atomic_op, u32 size,
 				 u32 dst_reg, u32 src_reg, u32 index_reg,
 				 int off)
 {
-	u8 *prog = *pprog;
-
 	if (atomic_op != BPF_XCHG)
 		EMIT1(0xF0); /* lock prefix */
 
@@ -1418,22 +1341,21 @@ static int emit_atomic_rmw_index(u8 **pprog, u32 atomic_op, u32 size,
 		pr_err("bpf_jit: unknown atomic opcode %02x\n", atomic_op);
 		return -EFAULT;
 	}
-	emit_insn_suffix_SIB(&prog, dst_reg, src_reg, index_reg, off);
-	*pprog = prog;
+	emit_insn_suffix_SIB(jit, dst_reg, src_reg, index_reg, off);
 	return 0;
 }
 
-static int emit_atomic_ld_st(u8 **pprog, u32 atomic_op, u32 dst_reg,
+static int emit_atomic_ld_st(struct jit_emit_context *jit, u32 atomic_op, u32 dst_reg,
 			     u32 src_reg, s16 off, u8 bpf_size)
 {
 	switch (atomic_op) {
 	case BPF_LOAD_ACQ:
 		/* dst_reg = smp_load_acquire(src_reg + off16) */
-		emit_ldx(pprog, bpf_size, dst_reg, src_reg, off);
+		emit_ldx(jit, bpf_size, dst_reg, src_reg, off);
 		break;
 	case BPF_STORE_REL:
 		/* smp_store_release(dst_reg + off16, src_reg) */
-		emit_stx(pprog, bpf_size, dst_reg, src_reg, off);
+		emit_stx(jit, bpf_size, dst_reg, src_reg, off);
 		break;
 	default:
 		pr_err("bpf_jit: unknown atomic load/store opcode %02x\n",
@@ -1444,18 +1366,18 @@ static int emit_atomic_ld_st(u8 **pprog, u32 atomic_op, u32 dst_reg,
 	return 0;
 }
 
-static int emit_atomic_ld_st_index(u8 **pprog, u32 atomic_op, u32 size,
+static int emit_atomic_ld_st_index(struct jit_emit_context *jit, u32 atomic_op, u32 size,
 				   u32 dst_reg, u32 src_reg, u32 index_reg,
 				   int off)
 {
 	switch (atomic_op) {
 	case BPF_LOAD_ACQ:
 		/* dst_reg = smp_load_acquire(src_reg + idx_reg + off16) */
-		emit_ldx_index(pprog, size, dst_reg, src_reg, index_reg, off);
+		emit_ldx_index(jit, size, dst_reg, src_reg, index_reg, off);
 		break;
 	case BPF_STORE_REL:
 		/* smp_store_release(dst_reg + idx_reg + off16, src_reg) */
-		emit_stx_index(pprog, size, dst_reg, src_reg, index_reg, off);
+		emit_stx_index(jit, size, dst_reg, src_reg, index_reg, off);
 		break;
 	default:
 		pr_err("bpf_jit: unknown atomic load/store opcode %02x\n",
@@ -1562,10 +1484,9 @@ static void detect_reg_usage(struct bpf_insn *insn, int insn_cnt,
  * l: vector length (128 bit or 256 bit) or reserved
  * pp: opcode prefix (none, 0x66, 0xf2 or 0xf3)
  */
-static void emit_3vex(u8 **pprog, bool r, bool x, bool b, u8 m,
+static void emit_3vex(struct jit_emit_context *jit, bool r, bool x, bool b, u8 m,
 		      bool w, u8 src_reg2, bool l, u8 pp)
 {
-	u8 *prog = *pprog;
 	const u8 b0 = 0xc4; /* first byte of 3-byte VEX prefix */
 	u8 b1, b2;
 	u8 vvvv = reg2hex[src_reg2];
@@ -1595,27 +1516,22 @@ static void emit_3vex(u8 **pprog, bool r, bool x, bool b, u8 m,
 	b2 = (w << 7) | ((~vvvv & 0xf) << 3) | (l << 2) | (pp & 3);
 
 	EMIT3(b0, b1, b2);
-	*pprog = prog;
 }
 
 /* emit BMI2 shift instruction */
-static void emit_shiftx(u8 **pprog, u32 dst_reg, u8 src_reg, bool is64, u8 op)
+static void emit_shiftx(struct jit_emit_context *jit, u32 dst_reg, u8 src_reg, bool is64, u8 op)
 {
-	u8 *prog = *pprog;
 	bool r = is_ereg(dst_reg);
 	u8 m = 2; /* escape code 0f38 */
 
-	emit_3vex(&prog, r, false, r, m, is64, src_reg, false, op);
+	emit_3vex(jit, r, false, r, m, is64, src_reg, false, op);
 	EMIT2(0xf7, add_2reg(0xC0, dst_reg, dst_reg));
-	*pprog = prog;
 }
 
-static void emit_priv_frame_ptr(u8 **pprog, void __percpu *priv_frame_ptr)
+static void emit_priv_frame_ptr(struct jit_emit_context *jit, void __percpu *priv_frame_ptr)
 {
-	u8 *prog = *pprog;
-
 	/* movabs r9, priv_frame_ptr */
-	emit_mov_imm64(&prog, X86_REG_R9, (__force long) priv_frame_ptr >> 32,
+	emit_mov_imm64(jit, X86_REG_R9, (__force long) priv_frame_ptr >> 32,
 		       (u32) (__force long) priv_frame_ptr);
 
 #ifdef CONFIG_SMP
@@ -1624,11 +1540,9 @@ static void emit_priv_frame_ptr(u8 **pprog, void __percpu *priv_frame_ptr)
 	EMIT3(0x03, 0x0c, 0x25);
 	EMIT((u32)(unsigned long)&this_cpu_off, 4);
 #endif
-
-	*pprog = prog;
 }
 
-#define INSN_SZ_DIFF (((addrs[i] - addrs[i - 1]) - (prog - temp)))
+#define INSN_SZ_DIFF (((addrs[i] - addrs[i - 1]) - (jit->prog - temp)))
 
 #define __LOAD_TCC_PTR(off)			\
 	EMIT3_off32(0x48, 0x8B, 0x85, off)
@@ -1640,10 +1554,9 @@ static void emit_priv_frame_ptr(u8 **pprog, void __percpu *priv_frame_ptr)
 #define PRIV_STACK_GUARD_SZ    8
 #define PRIV_STACK_GUARD_VAL   0xEB9F12345678eb9fULL
 
-static int emit_spectre_bhb_barrier(u8 **pprog, u8 *ip,
+static int emit_spectre_bhb_barrier(struct jit_emit_context *jit, u8 *ip,
 				    struct bpf_prog *bpf_prog)
 {
-	u8 *prog = *pprog;
 	u8 *func;
 
 	if (cpu_feature_enabled(X86_FEATURE_CLEAR_BHB_LOOP)) {
@@ -1653,9 +1566,9 @@ static int emit_spectre_bhb_barrier(u8 **pprog, u8 *ip,
 		ip += 2;
 
 		func = (u8 *)clear_bhb_loop;
-		ip += x86_call_depth_emit_accounting(&prog, func, ip);
+		ip += bpf_call_depth_emit_accounting(jit, func, ip);
 
-		if (emit_call(&prog, func, ip))
+		if (emit_call(jit, func, ip))
 			return -EINVAL;
 		EMIT1(0x59); /* pop rcx */
 		EMIT1(0x58); /* pop rax */
@@ -1678,7 +1591,6 @@ static int emit_spectre_bhb_barrier(u8 **pprog, u8 *ip,
 		 */
 		EMIT5(0xF3, 0x48, 0x0F, 0x1E, 0xF8); /* ibhf */
 	}
-	*pprog = prog;
 	return 0;
 }
 
@@ -1689,10 +1601,9 @@ static int emit_spectre_bhb_barrier(u8 **pprog, u8 *ip,
  * arena NULL is offset 0. Return the number of emitted bytes.
  */
 static int emit_kfunc_arena_args(struct bpf_prog *bpf_prog,
-				 const struct btf_func_model *fm, u8 **pprog)
+				 const struct btf_func_model *fm, struct jit_emit_context *jit)
 {
-	u8 *prog = *pprog;
-	u8 *start = prog;
+	u8 *start = jit->prog;
 	int i;
 
 	for (i = 0; i < min_t(int, fm->nr_args, MAX_BPF_FUNC_REG_ARGS); i++) {
@@ -1705,20 +1616,19 @@ static int emit_kfunc_arena_args(struct bpf_prog *bpf_prog,
 			return -EINVAL;
 
 		/* mov eN, eN: truncate and clear the upper 32 bits */
-		emit_mov_reg(&prog, false, reg, reg);
+		emit_mov_reg(jit, false, reg, reg);
 		if (flags & BTF_FMODEL_NULLABLE_ARG) {
 			/* test eN, eN; jz over the 3-byte add */
-			maybe_emit_mod(&prog, reg, reg, false);
+			maybe_emit_mod(jit, reg, reg, false);
 			EMIT2(0x85, add_2reg(0xC0, reg, reg));
 			EMIT2(X86_JE, 3);
 		}
 		/* add rN, r12 */
-		maybe_emit_mod(&prog, reg, X86_REG_R12, true);
+		maybe_emit_mod(jit, reg, X86_REG_R12, true);
 		EMIT2(0x01, add_2reg(0xC0, reg, X86_REG_R12));
 	}
 
-	*pprog = prog;
-	return prog - start;
+	return jit->prog - start;
 }
 
 static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *addrs, u8 *image,
@@ -1726,6 +1636,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 {
 	bool tail_call_reachable = bpf_prog->aux->tail_call_reachable;
 	struct bpf_insn *insn = bpf_prog->insnsi;
+	struct jit_emit_context *jit = &ctx->jit;
 	bool callee_regs_used[4] = {};
 	int insn_cnt = bpf_prog->len;
 	bool seen_exit = false;
@@ -1736,12 +1647,15 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 	void __percpu *priv_stack_ptr;
 	int i, excnt = 0;
 	int ilen, proglen = 0;
-	u8 *ip, *prog = temp;
+	u8 *ip;
 	u32 stack_depth;
 	int callee_saved_size;
 	s32 outgoing_arg_base;
 	int err;
 
+	jit->prog = temp;
+	jit->dry_run = false;
+
 	stack_depth = bpf_prog->aux->stack_depth;
 	out_stack_arg_cnt = bpf_out_stack_arg_cnt(env, bpf_prog);
 	priv_stack_ptr = bpf_prog->aux->priv_stack_ptr;
@@ -1777,11 +1691,11 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 
 	detect_reg_usage(insn, insn_cnt, callee_regs_used);
 
-	emit_prologue(&prog, image, stack_depth,
+	emit_prologue(jit, image, stack_depth,
 		      bpf_prog_was_classic(bpf_prog), tail_call_reachable,
 		      bpf_is_subprog(bpf_prog), bpf_prog->aux->exception_cb);
 
-	bpf_prog->aux->ksym.fp_start = prog - temp;
+	bpf_prog->aux->ksym.fp_start = jit->prog - temp;
 
 	/* Exception callback will clobber callee regs for its own use, and
 	 * restore the original callee regs from main prog's stack frame.
@@ -1791,12 +1705,12 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 		 * register, as we throw after entry into the kernel, which may
 		 * overwrite r12.
 		 */
-		push_r12(&prog);
-		push_callee_regs(&prog, all_callee_regs_used);
+		push_r12(jit);
+		push_callee_regs(jit, all_callee_regs_used);
 	} else {
 		if (arena_vm_start)
-			push_r12(&prog);
-		push_callee_regs(&prog, callee_regs_used);
+			push_r12(jit);
+		push_callee_regs(jit, callee_regs_used);
 	}
 
 	/* Compute callee-saved register area size. */
@@ -1834,21 +1748,21 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 	outgoing_rsp = out_stack_arg_cnt > 1 ? (out_stack_arg_cnt - 1) * 8 : 0;
 	if (bpf_prog->aux->exception_boundary)
 		bpf_prog->aux->stack_arg_sp_adjust = outgoing_rsp;
-	emit_sub_rsp(&prog, outgoing_rsp);
+	emit_sub_rsp(jit, outgoing_rsp);
 
 	if (arena_vm_start)
-		emit_mov_imm64(&prog, X86_REG_R12,
+		emit_mov_imm64(jit, X86_REG_R12,
 			       arena_vm_start >> 32, (u32) arena_vm_start);
 
 	if (priv_frame_ptr)
-		emit_priv_frame_ptr(&prog, priv_frame_ptr);
+		emit_priv_frame_ptr(jit, priv_frame_ptr);
 
-	ilen = prog - temp;
+	ilen = jit->prog - temp;
 	if (rw_image)
 		memcpy(rw_image + proglen, temp, ilen);
 	proglen += ilen;
 	addrs[0] = proglen;
-	prog = temp;
+	jit->prog = temp;
 
 	for (i = 1; i <= insn_cnt; i++, insn++) {
 		const s32 imm32 = insn->imm;
@@ -1873,7 +1787,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 		if (bpf_insn_is_indirect_target(env, bpf_prog, i - 1))
 			EMIT_ENDBR();
 
-		ip = image + addrs[i - 1] + (prog - temp);
+		ip = image + addrs[i - 1] + (jit->prog - temp);
 
 		switch (insn->code) {
 			/* ALU */
@@ -1887,7 +1801,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 		case BPF_ALU64 | BPF_AND | BPF_X:
 		case BPF_ALU64 | BPF_OR | BPF_X:
 		case BPF_ALU64 | BPF_XOR | BPF_X:
-			maybe_emit_mod(&prog, dst_reg, src_reg,
+			maybe_emit_mod(jit, dst_reg, src_reg,
 				       BPF_CLASS(insn->code) == BPF_ALU64);
 			b2 = simple_alu_opcodes[BPF_OP(insn->code)];
 			EMIT2(b2, add_2reg(0xC0, dst_reg, src_reg));
@@ -1897,32 +1811,32 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 			if (insn_is_cast_user(insn)) {
 				if (dst_reg != src_reg)
 					/* 32-bit mov */
-					emit_mov_reg(&prog, false, dst_reg, src_reg);
+					emit_mov_reg(jit, false, dst_reg, src_reg);
 				/* shl dst_reg, 32 */
-				maybe_emit_1mod(&prog, dst_reg, true);
+				maybe_emit_1mod(jit, dst_reg, true);
 				EMIT3(0xC1, add_1reg(0xE0, dst_reg), 32);
 
 				/* or dst_reg, user_vm_start */
-				maybe_emit_1mod(&prog, dst_reg, true);
+				maybe_emit_1mod(jit, dst_reg, true);
 				if (is_axreg(dst_reg))
 					EMIT1_off32(0x0D,  user_vm_start >> 32);
 				else
 					EMIT2_off32(0x81, add_1reg(0xC8, dst_reg),  user_vm_start >> 32);
 
 				/* rol dst_reg, 32 */
-				maybe_emit_1mod(&prog, dst_reg, true);
+				maybe_emit_1mod(jit, dst_reg, true);
 				EMIT3(0xC1, add_1reg(0xC0, dst_reg), 32);
 
 				/* xor r11, r11 */
 				EMIT3(0x4D, 0x31, 0xDB);
 
 				/* test dst_reg32, dst_reg32; check if lower 32-bit are zero */
-				maybe_emit_mod(&prog, dst_reg, dst_reg, false);
+				maybe_emit_mod(jit, dst_reg, dst_reg, false);
 				EMIT2(0x85, add_2reg(0xC0, dst_reg, dst_reg));
 
 				/* cmove r11, dst_reg; if so, set dst_reg to zero */
 				/* WARNING: Intel swapped src/dst register encoding in CMOVcc !!! */
-				maybe_emit_mod(&prog, AUX_REG, dst_reg, true);
+				maybe_emit_mod(jit, AUX_REG, dst_reg, true);
 				EMIT3(0x0F, 0x44, add_2reg(0xC0, AUX_REG, dst_reg));
 				break;
 			} else if (insn_is_mov_percpu_addr(insn)) {
@@ -1939,11 +1853,11 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 			fallthrough;
 		case BPF_ALU | BPF_MOV | BPF_X:
 			if (insn->off == 0)
-				emit_mov_reg(&prog,
+				emit_mov_reg(jit,
 					     BPF_CLASS(insn->code) == BPF_ALU64,
 					     dst_reg, src_reg);
 			else
-				emit_movsx_reg(&prog, insn->off,
+				emit_movsx_reg(jit, insn->off,
 					       BPF_CLASS(insn->code) == BPF_ALU64,
 					       dst_reg, src_reg);
 			break;
@@ -1951,7 +1865,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 			/* neg dst */
 		case BPF_ALU | BPF_NEG:
 		case BPF_ALU64 | BPF_NEG:
-			maybe_emit_1mod(&prog, dst_reg,
+			maybe_emit_1mod(jit, dst_reg,
 					BPF_CLASS(insn->code) == BPF_ALU64);
 			EMIT2(0xF7, add_1reg(0xD8, dst_reg));
 			break;
@@ -1966,7 +1880,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 		case BPF_ALU64 | BPF_AND | BPF_K:
 		case BPF_ALU64 | BPF_OR | BPF_K:
 		case BPF_ALU64 | BPF_XOR | BPF_K:
-			maybe_emit_1mod(&prog, dst_reg,
+			maybe_emit_1mod(jit, dst_reg,
 					BPF_CLASS(insn->code) == BPF_ALU64);
 
 			/*
@@ -2006,12 +1920,12 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 
 		case BPF_ALU64 | BPF_MOV | BPF_K:
 		case BPF_ALU | BPF_MOV | BPF_K:
-			emit_mov_imm32(&prog, BPF_CLASS(insn->code) == BPF_ALU64,
+			emit_mov_imm32(jit, BPF_CLASS(insn->code) == BPF_ALU64,
 				       dst_reg, imm32);
 			break;
 
 		case BPF_LD | BPF_IMM | BPF_DW:
-			emit_mov_imm64(&prog, dst_reg, insn[1].imm, insn[0].imm);
+			emit_mov_imm64(jit, dst_reg, insn[1].imm, insn[0].imm);
 			insn++;
 			i++;
 			break;
@@ -2047,7 +1961,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 
 			if (dst_reg != BPF_REG_0)
 				/* mov rax, dst_reg */
-				emit_mov_reg(&prog, is64, BPF_REG_0, dst_reg);
+				emit_mov_reg(jit, is64, BPF_REG_0, dst_reg);
 
 			if (insn->off == 0) {
 				/*
@@ -2057,7 +1971,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 				EMIT2(0x31, 0xd2);
 
 				/* div src_reg */
-				maybe_emit_1mod(&prog, src_reg, is64);
+				maybe_emit_1mod(jit, src_reg, is64);
 				EMIT2(0xF7, add_1reg(0xF0, src_reg));
 			} else {
 				if (BPF_CLASS(insn->code) == BPF_ALU)
@@ -2066,18 +1980,18 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 					EMIT2(0x48, 0x99); /* cqo */
 
 				/* idiv src_reg */
-				maybe_emit_1mod(&prog, src_reg, is64);
+				maybe_emit_1mod(jit, src_reg, is64);
 				EMIT2(0xF7, add_1reg(0xF8, src_reg));
 			}
 
 			if (BPF_OP(insn->code) == BPF_MOD &&
 			    dst_reg != BPF_REG_3)
 				/* mov dst_reg, rdx */
-				emit_mov_reg(&prog, is64, dst_reg, BPF_REG_3);
+				emit_mov_reg(jit, is64, dst_reg, BPF_REG_3);
 			else if (BPF_OP(insn->code) == BPF_DIV &&
 				 dst_reg != BPF_REG_0)
 				/* mov dst_reg, rax */
-				emit_mov_reg(&prog, is64, dst_reg, BPF_REG_0);
+				emit_mov_reg(jit, is64, dst_reg, BPF_REG_0);
 
 			if (dst_reg != BPF_REG_3)
 				EMIT1(0x5A); /* pop rdx */
@@ -2088,7 +2002,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 
 		case BPF_ALU | BPF_MUL | BPF_K:
 		case BPF_ALU64 | BPF_MUL | BPF_K:
-			maybe_emit_mod(&prog, dst_reg, dst_reg,
+			maybe_emit_mod(jit, dst_reg, dst_reg,
 				       BPF_CLASS(insn->code) == BPF_ALU64);
 
 			if (is_imm8(imm32))
@@ -2104,7 +2018,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 
 		case BPF_ALU | BPF_MUL | BPF_X:
 		case BPF_ALU64 | BPF_MUL | BPF_X:
-			maybe_emit_mod(&prog, src_reg, dst_reg,
+			maybe_emit_mod(jit, src_reg, dst_reg,
 				       BPF_CLASS(insn->code) == BPF_ALU64);
 
 			/* imul dst_reg, src_reg */
@@ -2118,7 +2032,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 		case BPF_ALU64 | BPF_LSH | BPF_K:
 		case BPF_ALU64 | BPF_RSH | BPF_K:
 		case BPF_ALU64 | BPF_ARSH | BPF_K:
-			maybe_emit_1mod(&prog, dst_reg,
+			maybe_emit_1mod(jit, dst_reg,
 					BPF_CLASS(insn->code) == BPF_ALU64);
 
 			b3 = simple_alu_opcodes[BPF_OP(insn->code)];
@@ -2152,7 +2066,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 					break;
 				}
 
-				emit_shiftx(&prog, dst_reg, src_reg, w, op);
+				emit_shiftx(jit, dst_reg, src_reg, w, op);
 
 				break;
 			}
@@ -2171,7 +2085,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 			}
 
 			/* shl %rax, %cl | shr %rax, %cl | sar %rax, %cl */
-			maybe_emit_1mod(&prog, dst_reg,
+			maybe_emit_1mod(jit, dst_reg,
 					BPF_CLASS(insn->code) == BPF_ALU64);
 
 			b3 = simple_alu_opcodes[BPF_OP(insn->code)];
@@ -2272,7 +2186,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
 		case BPF_ST | BPF_MEM | BPF_DW:
 			if (dst_reg == BPF_REG_PARAMS && insn->off == -8) {
 				/* Arg 6: store immediate in r9 register */
-				emit_mov_imm64(&prog, X86_REG_R9, imm32 >> 31, (u32)imm32);
+				emit_mov_imm64(jit, X86_REG_R9, imm32 >> 31, (u32)imm32);
 				break;
 			}
 			EMIT2(add_1mod(0x48, dst_reg), 0xC7);
@@ -2310,15 +2224,15 @@ st:			insn_off = insn->off;
 				insn_off = outgoing_arg_base - outgoing_rsp - insn_off - 16;
 				dst_reg = BPF_REG_FP;
 			}
-			emit_stx(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn_off);
+			emit_stx(jit, BPF_SIZE(insn->code), dst_reg, src_reg, insn_off);
 			break;
 
 		case BPF_ST | BPF_PROBE_MEM32 | BPF_B:
 		case BPF_ST | BPF_PROBE_MEM32 | BPF_H:
 		case BPF_ST | BPF_PROBE_MEM32 | BPF_W:
 		case BPF_ST | BPF_PROBE_MEM32 | BPF_DW:
-			start_of_ldx = prog;
-			emit_st_r12(&prog, BPF_SIZE(insn->code), dst_reg, insn->off, insn->imm);
+			start_of_ldx = jit->prog;
+			emit_st_r12(jit, BPF_SIZE(insn->code), dst_reg, insn->off, insn->imm);
 			goto populate_extable;
 
 			/* LDX: dst_reg = *(u8*)(src_reg + r12 + off) */
@@ -2333,14 +2247,14 @@ st:			insn_off = insn->off;
 		case BPF_STX | BPF_PROBE_MEM32 | BPF_H:
 		case BPF_STX | BPF_PROBE_MEM32 | BPF_W:
 		case BPF_STX | BPF_PROBE_MEM32 | BPF_DW:
-			start_of_ldx = prog;
+			start_of_ldx = jit->prog;
 			if (BPF_CLASS(insn->code) == BPF_LDX) {
 				if (BPF_MODE(insn->code) == BPF_PROBE_MEM32SX)
-					emit_ldsx_r12(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn->off);
+					emit_ldsx_r12(jit, BPF_SIZE(insn->code), dst_reg, src_reg, insn->off);
 				else
-					emit_ldx_r12(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn->off);
+					emit_ldx_r12(jit, BPF_SIZE(insn->code), dst_reg, src_reg, insn->off);
 			} else {
-				emit_stx_r12(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn->off);
+				emit_stx_r12(jit, BPF_SIZE(insn->code), dst_reg, src_reg, insn->off);
 			}
 populate_extable:
 			{
@@ -2401,7 +2315,7 @@ st:			insn_off = insn->off;
 					is_write = true;
 				}
 
-				ex->fixup = FIELD_PREP(FIXUP_INSN_LEN_MASK, prog - start_of_ldx) |
+				ex->fixup = FIELD_PREP(FIXUP_INSN_LEN_MASK, jit->prog - start_of_ldx) |
 					    FIELD_PREP(FIXUP_ARENA_REG_MASK, arena_reg) |
 					    FIELD_PREP(FIXUP_REG_MASK, fixup_reg);
 				ex->fixup |= FIXUP_ARENA_ACCESS;
@@ -2455,7 +2369,7 @@ st:			insn_off = insn->off;
 				u8 *end_of_jmp;
 
 				/* movabsq r10, VSYSCALL_ADDR */
-				emit_mov_imm64(&prog, BPF_REG_AX, (long)VSYSCALL_ADDR >> 32,
+				emit_mov_imm64(jit, BPF_REG_AX, (long)VSYSCALL_ADDR >> 32,
 					       (u32)(long)VSYSCALL_ADDR);
 
 				/* mov src_reg, r11 */
@@ -2463,40 +2377,40 @@ st:			insn_off = insn->off;
 
 				if (insn->off) {
 					/* add r11, insn->off */
-					maybe_emit_1mod(&prog, AUX_REG, true);
+					maybe_emit_1mod(jit, AUX_REG, true);
 					EMIT2_off32(0x81, add_1reg(0xC0, AUX_REG), insn->off);
 				}
 
 				/* sub r11, r10 */
-				maybe_emit_mod(&prog, AUX_REG, BPF_REG_AX, true);
+				maybe_emit_mod(jit, AUX_REG, BPF_REG_AX, true);
 				EMIT2(0x29, add_2reg(0xC0, AUX_REG, BPF_REG_AX));
 
 				/* movabsq r10, limit */
-				emit_mov_imm64(&prog, BPF_REG_AX, (long)limit >> 32,
+				emit_mov_imm64(jit, BPF_REG_AX, (long)limit >> 32,
 					       (u32)(long)limit);
 
 				/* cmp r10, r11 */
-				maybe_emit_mod(&prog, AUX_REG, BPF_REG_AX, true);
+				maybe_emit_mod(jit, AUX_REG, BPF_REG_AX, true);
 				EMIT2(0x39, add_2reg(0xC0, AUX_REG, BPF_REG_AX));
 
 				/* if unsigned '>', goto load */
 				EMIT2(X86_JA, 0);
-				end_of_jmp = prog;
+				end_of_jmp = jit->prog;
 
 				/* xor dst_reg, dst_reg */
-				emit_mov_imm32(&prog, false, dst_reg, 0);
+				emit_mov_imm32(jit, false, dst_reg, 0);
 				/* jmp byte_after_ldx */
 				EMIT2(0xEB, 0);
 
 				/* populate jmp_offset for JAE above to jump to start_of_ldx */
-				start_of_ldx = prog;
+				start_of_ldx = jit->prog;
 				end_of_jmp[-1] = start_of_ldx - end_of_jmp;
 			}
 			if (BPF_MODE(insn->code) == BPF_PROBE_MEMSX ||
 			    BPF_MODE(insn->code) == BPF_MEMSX)
-				emit_ldsx(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn_off);
+				emit_ldsx(jit, BPF_SIZE(insn->code), dst_reg, src_reg, insn_off);
 			else
-				emit_ldx(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn_off);
+				emit_ldx(jit, BPF_SIZE(insn->code), dst_reg, src_reg, insn_off);
 			if (BPF_MODE(insn->code) == BPF_PROBE_MEM ||
 			    BPF_MODE(insn->code) == BPF_PROBE_MEMSX) {
 				struct exception_table_entry *ex;
@@ -2504,7 +2418,7 @@ st:			insn_off = insn->off;
 				s64 delta;
 
 				/* populate jmp_offset for JMP above */
-				start_of_ldx[-1] = prog - start_of_ldx;
+				start_of_ldx[-1] = jit->prog - start_of_ldx;
 
 				if (!bpf_prog->aux->extable)
 					break;
@@ -2539,7 +2453,7 @@ st:			insn_off = insn->off;
 				 * End result: x86 insn "mov rbx, qword ptr [rax+0x14]"
 				 * of 4 bytes will be ignored and rbx will be zero inited.
 				 */
-				ex->fixup = FIELD_PREP(FIXUP_INSN_LEN_MASK, prog - start_of_ldx) |
+				ex->fixup = FIELD_PREP(FIXUP_INSN_LEN_MASK, jit->prog - start_of_ldx) |
 					    FIELD_PREP(FIXUP_REG_MASK, reg2pt_regs[dst_reg]);
 			}
 			break;
@@ -2567,26 +2481,26 @@ st:			insn_off = insn->off;
 				 */
 
 				/* Will need RAX as a CMPXCHG operand so save R0 */
-				emit_mov_reg(&prog, true, BPF_REG_AX, BPF_REG_0);
+				emit_mov_reg(jit, true, BPF_REG_AX, BPF_REG_0);
 				if (src_reg == BPF_REG_0)
 					real_src_reg = BPF_REG_AX;
 				if (dst_reg == BPF_REG_0)
 					real_dst_reg = BPF_REG_AX;
 
-				branch_target = prog;
+				branch_target = jit->prog;
 				/* Load old value */
-				emit_ldx(&prog, BPF_SIZE(insn->code),
+				emit_ldx(jit, BPF_SIZE(insn->code),
 					 BPF_REG_0, real_dst_reg, insn->off);
 				/*
 				 * Perform the (commutative) operation locally,
 				 * put the result in the AUX_REG.
 				 */
-				emit_mov_reg(&prog, is64, AUX_REG, BPF_REG_0);
-				maybe_emit_mod(&prog, AUX_REG, real_src_reg, is64);
+				emit_mov_reg(jit, is64, AUX_REG, BPF_REG_0);
+				maybe_emit_mod(jit, AUX_REG, real_src_reg, is64);
 				EMIT2(simple_alu_opcodes[BPF_OP(insn->imm)],
 				      add_2reg(0xC0, AUX_REG, real_src_reg));
 				/* Attempt to swap in new value */
-				err = emit_atomic_rmw(&prog, BPF_CMPXCHG,
+				err = emit_atomic_rmw(jit, BPF_CMPXCHG,
 						      real_dst_reg, AUX_REG,
 						      insn->off,
 						      BPF_SIZE(insn->code));
@@ -2596,19 +2510,19 @@ st:			insn_off = insn->off;
 				 * ZF tells us whether we won the race. If it's
 				 * cleared we need to try again.
 				 */
-				EMIT2(X86_JNE, -(prog - branch_target) - 2);
+				EMIT2(X86_JNE, -(jit->prog - branch_target) - 2);
 				/* Return the pre-modification value */
-				emit_mov_reg(&prog, is64, real_src_reg, BPF_REG_0);
+				emit_mov_reg(jit, is64, real_src_reg, BPF_REG_0);
 				/* Restore R0 after clobbering RAX */
-				emit_mov_reg(&prog, true, BPF_REG_0, BPF_REG_AX);
+				emit_mov_reg(jit, true, BPF_REG_0, BPF_REG_AX);
 				break;
 			}
 
 			if (bpf_atomic_is_load_store(insn))
-				err = emit_atomic_ld_st(&prog, insn->imm, dst_reg, src_reg,
+				err = emit_atomic_ld_st(jit, insn->imm, dst_reg, src_reg,
 							insn->off, BPF_SIZE(insn->code));
 			else
-				err = emit_atomic_rmw(&prog, insn->imm, dst_reg, src_reg,
+				err = emit_atomic_rmw(jit, insn->imm, dst_reg, src_reg,
 						      insn->off, BPF_SIZE(insn->code));
 			if (err)
 				return err;
@@ -2623,14 +2537,14 @@ st:			insn_off = insn->off;
 			fallthrough;
 		case BPF_STX | BPF_PROBE_ATOMIC | BPF_W:
 		case BPF_STX | BPF_PROBE_ATOMIC | BPF_DW:
-			start_of_ldx = prog;
+			start_of_ldx = jit->prog;
 
 			if (bpf_atomic_is_load_store(insn))
-				err = emit_atomic_ld_st_index(&prog, insn->imm,
+				err = emit_atomic_ld_st_index(jit, insn->imm,
 							      BPF_SIZE(insn->code), dst_reg,
 							      src_reg, X86_REG_R12, insn->off);
 			else
-				err = emit_atomic_rmw_index(&prog, insn->imm, BPF_SIZE(insn->code),
+				err = emit_atomic_rmw_index(jit, insn->imm, BPF_SIZE(insn->code),
 							    dst_reg, src_reg, X86_REG_R12,
 							    insn->off);
 			if (err)
@@ -2652,20 +2566,20 @@ st:			insn_off = insn->off;
 				fm = bpf_jit_find_kfunc_model(bpf_prog, insn);
 				if (!fm)
 					return -EINVAL;
-				err = emit_kfunc_arena_args(bpf_prog, fm, &prog);
+				err = emit_kfunc_arena_args(bpf_prog, fm, jit);
 				if (err < 0)
 					return err;
 				ip += err;
 			}
 			if (priv_frame_ptr) {
-				push_r9(&prog);
+				push_r9(jit);
 				ip += 2;
 			}
-			ip += x86_call_depth_emit_accounting(&prog, func, ip);
-			if (emit_call(&prog, func, ip))
+			ip += bpf_call_depth_emit_accounting(jit, func, ip);
+			if (emit_call(jit, func, ip))
 				return -EINVAL;
 			if (priv_frame_ptr)
-				pop_r9(&prog);
+				pop_r9(jit);
 			/*
 			 * A kfunc returning more than 8 bytes hands the second
 			 * half back in RDX (the native ABI's second return reg),
@@ -2673,7 +2587,7 @@ st:			insn_off = insn->off;
 			 * needed), while BPF R2 is RSI, so copy RDX into RSI.
 			 */
 			if (fm && fm->ret_size > 8)
-				emit_mov_reg(&prog, true, BPF_REG_2, BPF_REG_3);
+				emit_mov_reg(jit, true, BPF_REG_2, BPF_REG_3);
 			break;
 		}
 
@@ -2681,14 +2595,12 @@ st:			insn_off = insn->off;
 			if (imm32)
 				emit_bpf_tail_call_direct(bpf_prog,
 							  &bpf_prog->aux->poke_tab[imm32 - 1],
-							  &prog,
 							  ip,
 							  callee_regs_used,
 							  stack_depth,
 							  ctx);
 			else
 				emit_bpf_tail_call_indirect(bpf_prog,
-							    &prog,
 							    callee_regs_used,
 							    stack_depth,
 							    ip,
@@ -2717,7 +2629,7 @@ st:			insn_off = insn->off;
 		case BPF_JMP32 | BPF_JSGE | BPF_X:
 		case BPF_JMP32 | BPF_JSLE | BPF_X:
 			/* cmp dst_reg, src_reg */
-			maybe_emit_mod(&prog, dst_reg, src_reg,
+			maybe_emit_mod(jit, dst_reg, src_reg,
 				       BPF_CLASS(insn->code) == BPF_JMP);
 			EMIT2(0x39, add_2reg(0xC0, dst_reg, src_reg));
 			goto emit_cond_jmp;
@@ -2725,7 +2637,7 @@ st:			insn_off = insn->off;
 		case BPF_JMP | BPF_JSET | BPF_X:
 		case BPF_JMP32 | BPF_JSET | BPF_X:
 			/* test dst_reg, src_reg */
-			maybe_emit_mod(&prog, dst_reg, src_reg,
+			maybe_emit_mod(jit, dst_reg, src_reg,
 				       BPF_CLASS(insn->code) == BPF_JMP);
 			EMIT2(0x85, add_2reg(0xC0, dst_reg, src_reg));
 			goto emit_cond_jmp;
@@ -2733,7 +2645,7 @@ st:			insn_off = insn->off;
 		case BPF_JMP | BPF_JSET | BPF_K:
 		case BPF_JMP32 | BPF_JSET | BPF_K:
 			/* test dst_reg, imm32 */
-			maybe_emit_1mod(&prog, dst_reg,
+			maybe_emit_1mod(jit, dst_reg,
 					BPF_CLASS(insn->code) == BPF_JMP);
 			EMIT2_off32(0xF7, add_1reg(0xC0, dst_reg), imm32);
 			goto emit_cond_jmp;
@@ -2760,14 +2672,14 @@ st:			insn_off = insn->off;
 		case BPF_JMP32 | BPF_JSLE | BPF_K:
 			/* test dst_reg, dst_reg to save one extra byte */
 			if (imm32 == 0) {
-				maybe_emit_mod(&prog, dst_reg, dst_reg,
+				maybe_emit_mod(jit, dst_reg, dst_reg,
 					       BPF_CLASS(insn->code) == BPF_JMP);
 				EMIT2(0x85, add_2reg(0xC0, dst_reg, dst_reg));
 				goto emit_cond_jmp;
 			}
 
 			/* cmp dst_reg, imm8/32 */
-			maybe_emit_1mod(&prog, dst_reg,
+			maybe_emit_1mod(jit, dst_reg,
 					BPF_CLASS(insn->code) == BPF_JMP);
 
 			if (is_imm8(imm32))
@@ -2843,7 +2755,7 @@ st:			insn_off = insn->off;
 						       nops);
 						return -EFAULT;
 					}
-					emit_nops(&prog, nops);
+					emit_nops(jit, nops);
 				}
 				EMIT2(jmp_cond, jmp_offset);
 			} else if (is_simm32(jmp_offset)) {
@@ -2856,7 +2768,7 @@ st:			insn_off = insn->off;
 			break;
 
 		case BPF_JMP | BPF_JA | BPF_X:
-			emit_indirect_jump(&prog, insn->dst_reg, ip);
+			emit_indirect_jump(jit, insn->dst_reg, ip);
 			break;
 		case BPF_JMP | BPF_JA:
 		case BPF_JMP32 | BPF_JA:
@@ -2900,7 +2812,7 @@ st:			insn_off = insn->off;
 						       nops);
 						return -EFAULT;
 					}
-					emit_nops(&prog, nops);
+					emit_nops(jit, nops);
 				}
 				break;
 			}
@@ -2925,7 +2837,7 @@ st:			insn_off = insn->off;
 						       nops);
 						return -EFAULT;
 					}
-					emit_nops(&prog, INSN_SZ_DIFF - 2);
+					emit_nops(jit, INSN_SZ_DIFF - 2);
 				}
 				EMIT2(0xEB, jmp_offset);
 			} else if (is_simm32(jmp_offset)) {
@@ -2946,23 +2858,23 @@ st:			insn_off = insn->off;
 			ctx->cleanup_addr = proglen;
 			if (bpf_prog_was_classic(bpf_prog) &&
 			    !ns_capable_noaudit(&init_user_ns, CAP_SYS_ADMIN)) {
-				if (emit_spectre_bhb_barrier(&prog, ip, bpf_prog))
+				if (emit_spectre_bhb_barrier(jit, ip, bpf_prog))
 					return -EINVAL;
 			}
 			/* Deallocate outgoing args 7+ area. */
-			emit_add_rsp(&prog, outgoing_rsp);
+			emit_add_rsp(jit, outgoing_rsp);
 			if (bpf_prog->aux->exception_boundary) {
-				pop_callee_regs(&prog, all_callee_regs_used);
-				pop_r12(&prog);
+				pop_callee_regs(jit, all_callee_regs_used);
+				pop_r12(jit);
 			} else {
-				pop_callee_regs(&prog, callee_regs_used);
+				pop_callee_regs(jit, callee_regs_used);
 				if (arena_vm_start)
-					pop_r12(&prog);
+					pop_r12(jit);
 			}
 			EMIT1(0xC9);         /* leave */
-			bpf_prog->aux->ksym.fp_end = prog - temp;
+			bpf_prog->aux->ksym.fp_end = jit->prog - temp;
 
-			emit_return(&prog, image + addrs[i - 1] + (prog - temp));
+			emit_return(jit, image + addrs[i - 1] + (jit->prog - temp));
 			break;
 
 		default:
@@ -2976,7 +2888,7 @@ st:			insn_off = insn->off;
 			return -EINVAL;
 		}
 
-		ilen = prog - temp;
+		ilen = jit->prog - temp;
 		if (ilen > BPF_MAX_INSN_SIZE) {
 			pr_err("bpf_jit: fatal insn size error\n");
 			return -EFAULT;
@@ -3000,7 +2912,7 @@ st:			insn_off = insn->off;
 		}
 		proglen += ilen;
 		addrs[i] = proglen;
-		prog = temp;
+		jit->prog = temp;
 	}
 
 	if (image && excnt != bpf_prog->aux->num_exentries) {
@@ -3011,11 +2923,10 @@ st:			insn_off = insn->off;
 }
 
 static void clean_stack_garbage(const struct btf_func_model *m,
-				u8 **pprog, int nr_stack_slots,
+				struct jit_emit_context *jit, int nr_stack_slots,
 				int stack_size)
 {
 	int arg_size, off;
-	u8 *prog;
 
 	/* Generally speaking, the compiler will pass the arguments
 	 * on-stack with "push" instruction, which will take 8-byte
@@ -3047,14 +2958,12 @@ static void clean_stack_garbage(const struct btf_func_model *m,
 	arg_size = m->arg_size[m->nr_args - 1];
 	if (arg_size <= 4) {
 		off = -(stack_size - 4);
-		prog = *pprog;
 		/* mov DWORD PTR [rbp + off], 0 */
 		if (!is_imm8(off))
 			EMIT2_off32(0xC7, 0x85, off);
 		else
 			EMIT3(0xC7, 0x45, off);
 		EMIT(0, 4);
-		*pprog = prog;
 	}
 }
 
@@ -3082,26 +2991,23 @@ static int get_nr_used_regs(const struct btf_func_model *m)
  * subtraction both truncates and clears the upper half, so the stored
  * value satisfies the JIT invariant for arena pointer registers.
  */
-static void emit_arena_arg_conv(u8 **pprog, u32 src_reg, bool nullable, u32 base_lo)
+static void emit_arena_arg_conv(struct jit_emit_context *jit, u32 src_reg, bool nullable,
+				u32 base_lo)
 {
-	u8 *prog = *pprog;
-
 	if (nullable) {
 		if (src_reg != BPF_REG_0)
-			emit_mov_reg(&prog, true, BPF_REG_0, src_reg);
+			emit_mov_reg(jit, true, BPF_REG_0, src_reg);
 		/* test rax, rax; jz over the 5-byte sub */
 		EMIT3(0x48, 0x85, 0xC0);
 		EMIT2(X86_JE, 5);
 	} else if (src_reg != BPF_REG_0) {
-		emit_mov_reg(&prog, false, BPF_REG_0, src_reg);
+		emit_mov_reg(jit, false, BPF_REG_0, src_reg);
 	}
 	/* sub eax, base_lo */
 	EMIT1_off32(0x2D, base_lo);
-
-	*pprog = prog;
 }
 
-static void save_args(const struct btf_func_model *m, u8 **prog,
+static void save_args(const struct btf_func_model *m, struct jit_emit_context *jit,
 		      int stack_size, bool for_call_origin, u32 flags,
 		      u64 arena_base)
 {
@@ -3150,12 +3056,12 @@ static void save_args(const struct btf_func_model *m, u8 **prog,
 			 * called indirectly, so rbp + 16.
 			 */
 			for (j = 0; j < arg_regs; j++) {
-				emit_ldx(prog, BPF_DW, BPF_REG_0, BPF_REG_FP,
+				emit_ldx(jit, BPF_DW, BPF_REG_0, BPF_REG_FP,
 					 nr_stack_slots * 8 + stack_args_off);
 				if (arena_arg)
-					emit_arena_arg_conv(prog, BPF_REG_0, nullable,
+					emit_arena_arg_conv(jit, BPF_REG_0, nullable,
 							    (u32)arena_base);
-				emit_stx(prog, BPF_DW, BPF_REG_FP, BPF_REG_0,
+				emit_stx(jit, BPF_DW, BPF_REG_FP, BPF_REG_0,
 					 -stack_size);
 
 				if (!nr_stack_slots)
@@ -3178,20 +3084,20 @@ static void save_args(const struct btf_func_model *m, u8 **prog,
 				u32 src = nr_regs == 5 ? X86_REG_R9 : BPF_REG_1 + nr_regs;
 
 				if (arena_arg) {
-					emit_arena_arg_conv(prog, src, nullable, (u32)arena_base);
+					emit_arena_arg_conv(jit, src, nullable, (u32)arena_base);
 					src = BPF_REG_0;
 				}
-				emit_stx(prog, BPF_DW, BPF_REG_FP, src, -stack_size);
+				emit_stx(jit, BPF_DW, BPF_REG_FP, src, -stack_size);
 				stack_size -= 8;
 				nr_regs++;
 			}
 		}
 	}
 
-	clean_stack_garbage(m, prog, nr_stack_slots, first_off);
+	clean_stack_garbage(m, jit, nr_stack_slots, first_off);
 }
 
-static void restore_regs(const struct btf_func_model *m, u8 **prog,
+static void restore_regs(const struct btf_func_model *m, struct jit_emit_context *jit,
 			 int stack_size)
 {
 	int i, j, arg_regs, nr_regs = 0;
@@ -3207,7 +3113,7 @@ static void restore_regs(const struct btf_func_model *m, u8 **prog,
 		arg_regs = (m->arg_size[i] + 7) / 8;
 		if (nr_regs + arg_regs <= 6) {
 			for (j = 0; j < arg_regs; j++) {
-				emit_ldx(prog, BPF_DW,
+				emit_ldx(jit, BPF_DW,
 					 nr_regs == 5 ? X86_REG_R9 : BPF_REG_1 + nr_regs,
 					 BPF_REG_FP,
 					 -stack_size);
@@ -3223,19 +3129,18 @@ static void restore_regs(const struct btf_func_model *m, u8 **prog,
 	}
 }
 
-static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog,
+static int invoke_bpf_prog(const struct btf_func_model *m, struct jit_emit_context *jit,
 			   struct bpf_tramp_node *node, int stack_size,
 			   int run_ctx_off, bool save_ret,
 			   void *image, void *rw_image)
 {
-	u8 *prog = *pprog;
 	u8 *jmp_insn;
 	int ctx_cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie);
 	struct bpf_prog *p = node->link->prog;
 	u64 cookie = node->cookie;
 
 	/* mov rdi, cookie */
-	emit_mov_imm64(&prog, BPF_REG_1, (long) cookie >> 32, (u32) (long) cookie);
+	emit_mov_imm64(jit, BPF_REG_1, (long) cookie >> 32, (u32) (long) cookie);
 
 	/* Prepare struct bpf_tramp_run_ctx.
 	 *
@@ -3244,28 +3149,28 @@ static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog,
 	 *
 	 * mov QWORD PTR [rbp - run_ctx_off + ctx_cookie_off], rdi
 	 */
-	emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_1, -run_ctx_off + ctx_cookie_off);
+	emit_stx(jit, BPF_DW, BPF_REG_FP, BPF_REG_1, -run_ctx_off + ctx_cookie_off);
 
 	/* arg1: mov rdi, progs[i] */
-	emit_mov_imm64(&prog, BPF_REG_1, (long) p >> 32, (u32) (long) p);
+	emit_mov_imm64(jit, BPF_REG_1, (long) p >> 32, (u32) (long) p);
 	/* arg2: lea rsi, [rbp - ctx_cookie_off] */
 	if (!is_imm8(-run_ctx_off))
 		EMIT3_off32(0x48, 0x8D, 0xB5, -run_ctx_off);
 	else
 		EMIT4(0x48, 0x8D, 0x75, -run_ctx_off);
 
-	if (emit_rsb_call(&prog, bpf_trampoline_enter(p), image + (prog - (u8 *)rw_image)))
+	if (emit_rsb_call(jit, bpf_trampoline_enter(p), image + (jit->prog - (u8 *)rw_image)))
 		return -EINVAL;
 	/* remember prog start time returned by __bpf_prog_enter */
-	emit_mov_reg(&prog, true, BPF_REG_6, BPF_REG_0);
+	emit_mov_reg(jit, true, BPF_REG_6, BPF_REG_0);
 
 	/* if (__bpf_prog_enter*(prog) == 0)
 	 *	goto skip_exec_of_prog;
 	 */
 	EMIT3(0x48, 0x85, 0xC0);  /* test rax,rax */
 	/* emit 2 nops that will be replaced with JE insn */
-	jmp_insn = prog;
-	emit_nops(&prog, 2);
+	jmp_insn = jit->prog;
+	emit_nops(jit, 2);
 
 	/* arg1: lea rdi, [rbp - stack_size] */
 	if (!is_imm8(-stack_size))
@@ -3274,11 +3179,11 @@ static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog,
 		EMIT4(0x48, 0x8D, 0x7D, -stack_size);
 	/* arg2: progs[i]->insnsi for interpreter */
 	if (!p->jited)
-		emit_mov_imm64(&prog, BPF_REG_2,
+		emit_mov_imm64(jit, BPF_REG_2,
 			       (long) p->insnsi >> 32,
 			       (u32) (long) p->insnsi);
 	/* call JITed bpf program or interpreter */
-	if (emit_rsb_call(&prog, p->bpf_func, image + (prog - (u8 *)rw_image)))
+	if (emit_rsb_call(jit, p->bpf_func, image + (jit->prog - (u8 *)rw_image)))
 		return -EINVAL;
 
 	/*
@@ -3290,42 +3195,39 @@ static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog,
 	 * value of BPF_PROG_TYPE_STRUCT_OPS prog.
 	 */
 	if (save_ret)
-		emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8);
+		emit_stx(jit, BPF_DW, BPF_REG_FP, BPF_REG_0, -8);
 
 	/* replace 2 nops with JE insn, since jmp target is known */
-	jmp_insn[0] = X86_JE;
-	jmp_insn[1] = prog - jmp_insn - 2;
+	if (!jit->dry_run) {
+		jmp_insn[0] = X86_JE;
+		jmp_insn[1] = jit->prog - jmp_insn - 2;
+	}
 
 	/* arg1: mov rdi, progs[i] */
-	emit_mov_imm64(&prog, BPF_REG_1, (long) p >> 32, (u32) (long) p);
+	emit_mov_imm64(jit, BPF_REG_1, (long) p >> 32, (u32) (long) p);
 	/* arg2: mov rsi, rbx <- start time in nsec */
-	emit_mov_reg(&prog, true, BPF_REG_2, BPF_REG_6);
+	emit_mov_reg(jit, true, BPF_REG_2, BPF_REG_6);
 	/* arg3: lea rdx, [rbp - run_ctx_off] */
 	if (!is_imm8(-run_ctx_off))
 		EMIT3_off32(0x48, 0x8D, 0x95, -run_ctx_off);
 	else
 		EMIT4(0x48, 0x8D, 0x55, -run_ctx_off);
-	if (emit_rsb_call(&prog, bpf_trampoline_exit(p), image + (prog - (u8 *)rw_image)))
+	if (emit_rsb_call(jit, bpf_trampoline_exit(p), image + (jit->prog - (u8 *)rw_image)))
 		return -EINVAL;
 
-	*pprog = prog;
 	return 0;
 }
 
-static void emit_align(u8 **pprog, u32 align)
+static void emit_align(struct jit_emit_context *jit, u32 align)
 {
-	u8 *target, *prog = *pprog;
-
-	target = PTR_ALIGN(prog, align);
-	if (target != prog)
-		emit_nops(&prog, target - prog);
+	u8 *target = PTR_ALIGN(jit->prog, align);
 
-	*pprog = prog;
+	if (target != jit->prog)
+		emit_nops(jit, target - jit->prog);
 }
 
-static int emit_cond_near_jump(u8 **pprog, void *func, void *ip, u8 jmp_cond)
+static int emit_cond_near_jump(struct jit_emit_context *jit, void *func, void *ip, u8 jmp_cond)
 {
-	u8 *prog = *pprog;
 	s64 offset;
 
 	offset = func - (ip + 2 + 4);
@@ -3334,48 +3236,44 @@ static int emit_cond_near_jump(u8 **pprog, void *func, void *ip, u8 jmp_cond)
 		return -EINVAL;
 	}
 	EMIT2_off32(0x0F, jmp_cond + 0x10, offset);
-	*pprog = prog;
 	return 0;
 }
 
-static int invoke_bpf(const struct btf_func_model *m, u8 **pprog,
+static int invoke_bpf(const struct btf_func_model *m, struct jit_emit_context *jit,
 		      struct bpf_tramp_nodes *tl, int stack_size,
 		      int run_ctx_off, int func_meta_off, bool save_ret,
 		      void *image, void *rw_image, u64 func_meta,
 		      int cookie_off)
 {
 	int i, cur_cookie = (cookie_off - stack_size) / 8;
-	u8 *prog = *pprog;
 
 	for (i = 0; i < tl->nr_nodes; i++) {
 		if (tl->nodes[i]->link->prog->call_session_cookie) {
-			emit_store_stack_imm64(&prog, BPF_REG_0, -func_meta_off,
+			emit_store_stack_imm64(jit, BPF_REG_0, -func_meta_off,
 				func_meta | (cur_cookie << BPF_TRAMP_COOKIE_INDEX_SHIFT));
 			cur_cookie--;
 		}
-		if (invoke_bpf_prog(m, &prog, tl->nodes[i], stack_size,
+		if (invoke_bpf_prog(m, jit, tl->nodes[i], stack_size,
 				    run_ctx_off, save_ret, image, rw_image))
 			return -EINVAL;
 	}
-	*pprog = prog;
 	return 0;
 }
 
-static int invoke_bpf_mod_ret(const struct btf_func_model *m, u8 **pprog,
+static int invoke_bpf_mod_ret(const struct btf_func_model *m, struct jit_emit_context *jit,
 			      struct bpf_tramp_nodes *tl, int stack_size,
 			      int run_ctx_off, u8 **branches,
 			      void *image, void *rw_image)
 {
-	u8 *prog = *pprog;
 	int i;
 
 	/* The first fmod_ret program will receive a garbage return value.
 	 * Set this to 0 to avoid confusing the program.
 	 */
-	emit_mov_imm32(&prog, false, BPF_REG_0, 0);
-	emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8);
+	emit_mov_imm32(jit, false, BPF_REG_0, 0);
+	emit_stx(jit, BPF_DW, BPF_REG_FP, BPF_REG_0, -8);
 	for (i = 0; i < tl->nr_nodes; i++) {
-		if (invoke_bpf_prog(m, &prog, tl->nodes[i], stack_size, run_ctx_off, true,
+		if (invoke_bpf_prog(m, jit, tl->nodes[i], stack_size, run_ctx_off, true,
 				    image, rw_image))
 			return -EINVAL;
 
@@ -3391,11 +3289,10 @@ static int invoke_bpf_mod_ret(const struct btf_func_model *m, u8 **pprog,
 		 * are replaced with a conditional jump once do_fexit (i.e. the
 		 * start of the fexit invocation) is finalized.
 		 */
-		branches[i] = prog;
-		emit_nops(&prog, 4 + 2);
+		branches[i] = jit->prog;
+		emit_nops(jit, 4 + 2);
 	}
 
-	*pprog = prog;
 	return 0;
 }
 
@@ -3474,12 +3371,12 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
 	struct bpf_tramp_nodes *fentry = &tnodes[BPF_TRAMP_FENTRY];
 	struct bpf_tramp_nodes *fexit = &tnodes[BPF_TRAMP_FEXIT];
 	struct bpf_tramp_nodes *fmod_ret = &tnodes[BPF_TRAMP_MODIFY_RETURN];
+	struct jit_emit_context jit_ctx = {}, *jit = &jit_ctx;
 	void *orig_call = func_addr;
 	int cookie_off, cookie_cnt;
 	u8 **branches = NULL;
 	u64 arena_base;
 	u64 func_meta;
-	u8 *prog;
 	bool save_ret;
 
 	/*
@@ -3584,13 +3481,14 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
 		orig_call += X86_PATCH_SIZE;
 	}
 
-	prog = rw_image;
+	jit->prog = rw_image;
+	jit->dry_run = !rw_image;
 
 	if (flags & BPF_TRAMP_F_INDIRECT) {
 		/*
 		 * Indirect call for bpf_struct_ops
 		 */
-		emit_cfi(&prog, image,
+		emit_cfi(jit, image,
 			 cfi_get_func_hash(func_addr),
 			 cfi_get_func_arity(func_addr));
 	} else {
@@ -3598,12 +3496,12 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
 		 * Direct-call fentry stub, as such it needs accounting for the
 		 * __fentry__ call.
 		 */
-		x86_call_depth_emit_accounting(&prog, NULL, image);
+		bpf_call_depth_emit_accounting(jit, NULL, image);
 	}
 	EMIT1(0x55);		 /* push rbp */
 	EMIT3(0x48, 0x89, 0xE5); /* mov rbp, rsp */
 	if (im)
-		im->ksym.fp_start = prog - (u8 *)rw_image;
+		im->ksym.fp_start = jit->prog - (u8 *)rw_image;
 
 	if (!is_imm8(stack_size)) {
 		/* sub rsp, stack_size */
@@ -3615,24 +3513,24 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
 	if (flags & BPF_TRAMP_F_TAIL_CALL_CTX)
 		EMIT1(0x50);		/* push rax */
 	/* mov QWORD PTR [rbp - rbx_off], rbx */
-	emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_6, -rbx_off);
+	emit_stx(jit, BPF_DW, BPF_REG_FP, BPF_REG_6, -rbx_off);
 
 	func_meta = nr_regs;
 	/* Store number of argument registers of the traced function */
-	emit_store_stack_imm64(&prog, BPF_REG_0, -func_meta_off, func_meta);
+	emit_store_stack_imm64(jit, BPF_REG_0, -func_meta_off, func_meta);
 
 	if (flags & BPF_TRAMP_F_IP_ARG) {
 		/* Store IP address of the traced function */
-		emit_store_stack_imm64(&prog, BPF_REG_0, -ip_off, (long)func_addr);
+		emit_store_stack_imm64(jit, BPF_REG_0, -ip_off, (long)func_addr);
 	}
 
-	save_args(m, &prog, regs_off, false, flags, arena_base);
+	save_args(m, jit, regs_off, false, flags, arena_base);
 
 	if (flags & BPF_TRAMP_F_CALL_ORIG) {
 		/* arg1: mov rdi, im */
-		emit_mov_imm64(&prog, BPF_REG_1, (long) im >> 32, (u32) (long) im);
-		if (emit_rsb_call(&prog, __bpf_tramp_enter,
-				  image + (prog - (u8 *)rw_image))) {
+		emit_mov_imm64(jit, BPF_REG_1, (long) im >> 32, (u32) (long) im);
+		if (emit_rsb_call(jit, __bpf_tramp_enter,
+				  image + (jit->prog - (u8 *)rw_image))) {
 			ret = -EINVAL;
 			goto cleanup;
 		}
@@ -3641,13 +3539,13 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
 	if (bpf_fsession_cnt(tnodes)) {
 		/* clear all the session cookies' value */
 		for (int i = 0; i < cookie_cnt; i++)
-			emit_store_stack_imm64(&prog, BPF_REG_0, -cookie_off + 8 * i, 0);
+			emit_store_stack_imm64(jit, BPF_REG_0, -cookie_off + 8 * i, 0);
 		/* clear the return value to make sure fentry always get 0 */
-		emit_store_stack_imm64(&prog, BPF_REG_0, -8, 0);
+		emit_store_stack_imm64(jit, BPF_REG_0, -8, 0);
 	}
 
 	if (fentry->nr_nodes) {
-		if (invoke_bpf(m, &prog, fentry, regs_off, run_ctx_off, func_meta_off,
+		if (invoke_bpf(m, jit, fentry, regs_off, run_ctx_off, func_meta_off,
 			       flags & BPF_TRAMP_F_RET_FENTRY_RET, image, rw_image,
 			       func_meta, cookie_off))
 			return -EINVAL;
@@ -3659,7 +3557,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
 		if (!branches)
 			return -ENOMEM;
 
-		if (invoke_bpf_mod_ret(m, &prog, fmod_ret, regs_off,
+		if (invoke_bpf_mod_ret(m, jit, fmod_ret, regs_off,
 				       run_ctx_off, branches, image, rw_image)) {
 			ret = -EINVAL;
 			goto cleanup;
@@ -3667,8 +3565,8 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
 	}
 
 	if (flags & BPF_TRAMP_F_CALL_ORIG) {
-		restore_regs(m, &prog, regs_off);
-		save_args(m, &prog, arg_stack_off, true, flags, 0);
+		restore_regs(m, jit, regs_off);
+		save_args(m, jit, arg_stack_off, true, flags, 0);
 
 		if (flags & BPF_TRAMP_F_TAIL_CALL_CTX) {
 			/* Before calling the original function, load the
@@ -3678,19 +3576,19 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
 		}
 
 		if (flags & BPF_TRAMP_F_ORIG_STACK) {
-			emit_ldx(&prog, BPF_DW, BPF_REG_6, BPF_REG_FP, 8);
+			emit_ldx(jit, BPF_DW, BPF_REG_6, BPF_REG_FP, 8);
 			EMIT2(0xff, 0xd3); /* call *rbx */
 		} else {
 			/* call original function */
-			if (emit_rsb_call(&prog, orig_call, image + (prog - (u8 *)rw_image))) {
+			if (emit_rsb_call(jit, orig_call, image + (jit->prog - (u8 *)rw_image))) {
 				ret = -EINVAL;
 				goto cleanup;
 			}
 		}
 		/* remember return value in a stack for bpf prog to access */
-		emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8);
-		im->ip_after_call = image + (prog - (u8 *)rw_image);
-		emit_nops(&prog, X86_PATCH_SIZE);
+		emit_stx(jit, BPF_DW, BPF_REG_FP, BPF_REG_0, -8);
+		im->ip_after_call = image + (jit->prog - (u8 *)rw_image);
+		emit_nops(jit, X86_PATCH_SIZE);
 	}
 
 	if (fmod_ret->nr_nodes) {
@@ -3699,12 +3597,17 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
 		 * Coding Rule 11: All branch targets should be 16-byte
 		 * aligned.
 		 */
-		emit_align(&prog, 16);
+		emit_align(jit, 16);
 		/* Update the branches saved in invoke_bpf_mod_ret with the
 		 * aligned address of do_fexit.
 		 */
 		for (i = 0; i < fmod_ret->nr_nodes; i++) {
-			emit_cond_near_jump(&branches[i], image + (prog - (u8 *)rw_image),
+			struct jit_emit_context branch_jit = {
+				.prog = branches[i],
+				.dry_run = jit->dry_run,
+			};
+
+			emit_cond_near_jump(&branch_jit, image + (jit->prog - (u8 *)rw_image),
 					    image + (branches[i] - (u8 *)rw_image), X86_JNE);
 		}
 	}
@@ -3712,10 +3615,10 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
 	/* set the "is_return" flag for fsession */
 	func_meta |= (1ULL << BPF_TRAMP_IS_RETURN_SHIFT);
 	if (bpf_fsession_cnt(tnodes))
-		emit_store_stack_imm64(&prog, BPF_REG_0, -func_meta_off, func_meta);
+		emit_store_stack_imm64(jit, BPF_REG_0, -func_meta_off, func_meta);
 
 	if (fexit->nr_nodes) {
-		if (invoke_bpf(m, &prog, fexit, regs_off, run_ctx_off, func_meta_off,
+		if (invoke_bpf(m, jit, fexit, regs_off, run_ctx_off, func_meta_off,
 			       false, image, rw_image, func_meta, cookie_off)) {
 			ret = -EINVAL;
 			goto cleanup;
@@ -3723,17 +3626,17 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
 	}
 
 	if (flags & BPF_TRAMP_F_RESTORE_REGS)
-		restore_regs(m, &prog, regs_off);
+		restore_regs(m, jit, regs_off);
 
 	/* This needs to be done regardless. If there were fmod_ret programs,
 	 * the return value is only updated on the stack and still needs to be
 	 * restored to R0.
 	 */
 	if (flags & BPF_TRAMP_F_CALL_ORIG) {
-		im->ip_epilogue = image + (prog - (u8 *)rw_image);
+		im->ip_epilogue = image + (jit->prog - (u8 *)rw_image);
 		/* arg1: mov rdi, im */
-		emit_mov_imm64(&prog, BPF_REG_1, (long) im >> 32, (u32) (long) im);
-		if (emit_rsb_call(&prog, __bpf_tramp_exit, image + (prog - (u8 *)rw_image))) {
+		emit_mov_imm64(jit, BPF_REG_1, (long) im >> 32, (u32) (long) im);
+		if (emit_rsb_call(jit, __bpf_tramp_exit, image + (jit->prog - (u8 *)rw_image))) {
 			ret = -EINVAL;
 			goto cleanup;
 		}
@@ -3746,25 +3649,26 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
 
 	/* restore return value of orig_call or fentry prog back into RAX */
 	if (save_ret)
-		emit_ldx(&prog, BPF_DW, BPF_REG_0, BPF_REG_FP, -8);
+		emit_ldx(jit, BPF_DW, BPF_REG_0, BPF_REG_FP, -8);
 
-	emit_ldx(&prog, BPF_DW, BPF_REG_6, BPF_REG_FP, -rbx_off);
+	emit_ldx(jit, BPF_DW, BPF_REG_6, BPF_REG_FP, -rbx_off);
 
 	EMIT1(0xC9); /* leave */
 	if (im)
-		im->ksym.fp_end = prog - (u8 *)rw_image;
+		im->ksym.fp_end = jit->prog - (u8 *)rw_image;
 
 	if (flags & BPF_TRAMP_F_SKIP_FRAME) {
 		/* skip our return address and return to parent */
 		EMIT4(0x48, 0x83, 0xC4, 8); /* add rsp, 8 */
 	}
-	emit_return(&prog, image + (prog - (u8 *)rw_image));
+	emit_return(jit, image + (jit->prog - (u8 *)rw_image));
 	/* Make sure the trampoline generation logic doesn't overflow */
-	if (WARN_ON_ONCE(prog > (u8 *)rw_image_end - BPF_INSN_SAFETY)) {
+	if (!jit->dry_run &&
+	    WARN_ON_ONCE(jit->prog > (u8 *)rw_image_end - BPF_INSN_SAFETY)) {
 		ret = -EFAULT;
 		goto cleanup;
 	}
-	ret = prog - (u8 *)rw_image + BPF_INSN_SAFETY;
+	ret = jit->prog - (u8 *)rw_image + BPF_INSN_SAFETY;
 
 cleanup:
 	kfree(branches);
@@ -3819,28 +3723,15 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
 			     struct bpf_tramp_nodes *tnodes, void *func_addr)
 {
 	struct bpf_tramp_image im;
-	void *image;
-	int ret;
 
-	/* Allocate a temporary buffer for __arch_prepare_bpf_trampoline().
-	 *
-	 * We cannot use kvmalloc here, because we need image to be in
-	 * module memory range.
-	 * Since it must be writable use bpf_jit_alloc_exec_rw().
-	 */
-	image = bpf_jit_alloc_exec_rw(PAGE_SIZE);
-	if (!image)
-		return -ENOMEM;
-
-	ret = __arch_prepare_bpf_trampoline(&im, image, image + PAGE_SIZE, image,
-					    m, flags, tnodes, func_addr);
-	bpf_jit_free_exec(image);
-	return ret;
+	return __arch_prepare_bpf_trampoline(&im, NULL, NULL, NULL, m, flags,
+					     tnodes, func_addr);
 }
 
-static int emit_bpf_dispatcher(u8 **pprog, int a, int b, s64 *progs, u8 *image, u8 *buf)
+static int emit_bpf_dispatcher(struct jit_emit_context *jit, int a, int b, s64 *progs, u8 *image,
+			       u8 *buf)
 {
-	u8 *jg_reloc, *prog = *pprog;
+	u8 *jg_reloc;
 	int pivot, err, jg_bytes = 1;
 	s64 jg_offset;
 
@@ -3853,15 +3744,13 @@ static int emit_bpf_dispatcher(u8 **pprog, int a, int b, s64 *progs, u8 *image,
 			return -1;
 		EMIT2_off32(0x81, add_1reg(0xF8, BPF_REG_3),
 			    progs[a]);
-		err = emit_cond_near_jump(&prog,	/* je func */
-					  (void *)progs[a], image + (prog - buf),
+		err = emit_cond_near_jump(jit,	/* je func */
+					  (void *)progs[a], image + (jit->prog - buf),
 					  X86_JE);
 		if (err)
 			return err;
 
-		emit_indirect_jump(&prog, BPF_REG_3 /* R3 -> rdx */, image + (prog - buf));
-
-		*pprog = prog;
+		emit_indirect_jump(jit, BPF_REG_3 /* R3 -> rdx */, image + (jit->prog - buf));
 		return 0;
 	}
 
@@ -3881,9 +3770,9 @@ static int emit_bpf_dispatcher(u8 **pprog, int a, int b, s64 *progs, u8 *image,
 	} else {
 		EMIT2(X86_JG, 0);
 	}
-	jg_reloc = prog;
+	jg_reloc = jit->prog;
 
-	err = emit_bpf_dispatcher(&prog, a, a + pivot,	/* emit lower_part */
+	err = emit_bpf_dispatcher(jit, a, a + pivot,	/* emit lower_part */
 				  progs, image, buf);
 	if (err)
 		return err;
@@ -3893,16 +3782,15 @@ static int emit_bpf_dispatcher(u8 **pprog, int a, int b, s64 *progs, u8 *image,
 	 * Coding Rule 11: All branch targets should be 16-byte
 	 * aligned.
 	 */
-	emit_align(&prog, 16);
-	jg_offset = prog - jg_reloc;
+	emit_align(jit, 16);
+	jg_offset = jit->prog - jg_reloc;
 	emit_code(jg_reloc - jg_bytes, jg_offset, jg_bytes);
 
-	err = emit_bpf_dispatcher(&prog, a + pivot + 1,	/* emit upper_part */
+	err = emit_bpf_dispatcher(jit, a + pivot + 1,	/* emit upper_part */
 				  b, progs, image, buf);
 	if (err)
 		return err;
 
-	*pprog = prog;
 	return 0;
 }
 
@@ -3920,10 +3808,10 @@ static int cmp_ips(const void *a, const void *b)
 
 int arch_prepare_bpf_dispatcher(void *image, void *buf, s64 *funcs, int num_funcs)
 {
-	u8 *prog = buf;
+	struct jit_emit_context jit = { .prog = buf };
 
 	sort(funcs, num_funcs, sizeof(funcs[0]), cmp_ips, NULL);
-	return emit_bpf_dispatcher(&prog, 0, num_funcs - 1, funcs, image, buf);
+	return emit_bpf_dispatcher(&jit, 0, num_funcs - 1, funcs, image, buf);
 }
 
 static void priv_stack_init_guard(void __percpu *priv_stack_ptr, int alloc_size)

							input: 41931
							output: 182
							thoughts: 2024

**Analysis of the x86 BPF JIT Patch for Uninitialized Memory Risks**

Alright, let's break down this patch and assess potential KMSAN implications. The core change revolves around a refactoring of the BPF JIT code emitter on x86, introducing a `struct jit_emit_context` to handle both the normal execution path and a "dry-run" mode for size calculations. The `jit_emit_context` struct contains a pointer to the current program buffer (`prog`) and a boolean flag `dry_run`.

The primary concern is always uninitialized memory reads, leaks to userspace, or usage in branching conditions or copies. I meticulously examined each code snippet affected by this patch, starting with `bpf_call_depth_emit_accounting`, which now utilizes a caller-provided buffer. The `insn_buff` is initialized within `x86_call_depth_emit_accounting` before being copied.

I then traced the `jit_emit_context` usage throughout the BPF JIT code, specifically within the `emit_code_jit` and related helper functions. The dry-run logic increments the `jit->prog` pointer to calculate the trampoline size without writing actual data, avoiding memory allocation. In each case that `jit` is initialized, the `dry_run` is set to false except when calculating size. The dry-run logic relies on pointer arithmetic (incrementing the `jit->prog` pointer) and conditional execution based on the `dry_run` flag.

I verified the implications of the dry-run, focusing on scenarios where the `jit->prog` pointer might be NULL during size calculation, which is the only time `dry_run == true`. However, the pointer arithmetic is handled correctly, with the result used solely for size calculation, without dereferencing or other memory access. The same is true for `invoke_bpf_prog` and other such situations.

There were no instances of uninitialized data being used in branches or copied to user space. All structures (`jit_emit_context`, `old_insn`, etc.) are either zero-initialized or initialized with valid values. Memory is always written to before it is read.

Therefore, after this detailed analysis, there is no evidence that this patch introduces any new uninitialized memory risks. Consequently, it doesn't warrant a KMSAN session. The changes are primarily code refactoring, improving the BPF JIT emitter, and they do not introduce any new kernel-to-user copy or conditional branching on uninitialized data. There are no exposed data structures or new data flows, or issues that would require KMSAN.

{
  "KMSANReasoning": "The patch refactors the x86 BPF JIT compiler and callthunk accounting code to introduce a dry-run mode for trampoline size calculation via `struct jit_emit_context`, replacing dummy allocation in `arch_bpf_trampoline_size()`. All new context and buffer variables are properly initialized (`= {}`), and code generation buffers are accurately tracked and populated. The patch does not introduce any uninitialized memory reads, uninitialized structure padding leaks to user space, or uninitialized branch conditions. Any potential bugs in JIT generation or pointer calculations would be out-of-bounds or logic errors caught by KASAN or standard kernel assertions, not uninitialized memory bugs detectable only by KMSAN. Therefore, dedicated KMSAN fuzzing is not required.",
  "NeedsKMSAN": false
}

{
  "KMSANReasoning": "The patch refactors the x86 BPF JIT compiler and callthunk accounting code to introduce a dry-run mode for trampoline size calculation via `struct jit_emit_context`, replacing dummy allocation in `arch_bpf_trampoline_size()`. All new context and buffer variables are properly initialized (`= {}`), and code generation buffers are accurately tracked and populated. The patch does not introduce any uninitialized memory reads, uninitialized structure padding leaks to user space, or uninitialized branch conditions. Any potential bugs in JIT generation or pointer calculations would be out-of-bounds or logic errors caught by KASAN or standard kernel assertions, not uninitialized memory bugs detectable only by KMSAN. Therefore, dedicated KMSAN fuzzing is not required.",
  "NeedsKMSAN": false
}

LLM Calls Summary:
Total Calls	Total Tokens	Avg Tokens	Total Duration (Seconds)	Avg Duration (Seconds)

Tool Calls Summary:
Total Calls	Total Duration (Seconds)	Avg Duration (Seconds)