| Seq | Timestamp | Type | Name | Duration |
|---|---|---|---|---|
| 0/0 | 2026/09/22 09:05 | flow | patch-triage |
7mArgs:null Results: {
"EnableConfigs": [
"BPF_SYSCALL",
"BPF_JIT",
"UNWINDER_ORC",
"DEBUG_INFO_BTF"
],
"FocusSymbols": [
"bpf_prepare_cleanup_exceptions",
"bpf_check_cleanup_exceptions",
"bpf_cleanup_attach_main_prog",
"bpf_cleanup_fill_native_ranges",
"bpf_cleanup_pad_for_ip",
"bpf_unwind_resume",
"arch_bpf_run_cleanup_pad",
"bpf_throw"
],
"KMSANReasoning": "The patch series implements support for BPF cleanup landing pads and exception unwinding across x86-64 and arm64 architectures, adding verifier checks, CFG analysis, JIT spills/epilogues, and the bpf_unwind_resume kfunc.\n\nA dedicated KMSAN session is not required based on the following analysis:\n1. Heap and Stack Memory Initialization:\n - All newly allocated kernel data structures (such as `struct bpf_exception_info`, `struct bpf_cleanup_range`, `struct bpf_cleanup_info`, and DFS scratch arrays in `cleanup_ctx`) are allocated using zeroing allocators (`kzalloc_obj`, `kvcalloc`, `kvzalloc`, `bitmap_zalloc`).\n - Verifier state tracking structures (`struct bpf_verifier_state`, `struct bpf_insn_aux_data`) are zero-initialized via `kzalloc_obj` and `__vmalloc(..., __GFP_ZERO)`. Added fields (`unwinding`, `unwind_frameno`, `unwind_frames`) are explicitly initialized and reset.\n - The stack context used during unwinding (`struct bpf_throw_ctx ctx = {}`) is zero-initialized on the stack.\n\n2. Kernel-to-User Space Copies / Info Leaks:\n - The only user copy introduced is writing `min_size` (a 4-byte scalar `u32`) to user space via `copy_to_bpfptr_offset` on `-E2BIG` in `check_cleanup_info()`. It is fully initialized with `min_t(u32, krec_size, urec_size)`.\n - No other kernel structures, struct padding, or uninitialized buffers are copied to user space.\n\n3. JIT and Runtime Register Spills:\n - On x86-64, JIT prologue and throw-site register spills write all 5 callee-saved / context registers before `arch_bpf_run_cleanup_pad()` reads them.\n - The BPF verifier enforces that landing pad entry clears caller-saved registers (`clear_caller_saved_regs`), initializes `BPF_REG_0` as known, and prevents reading uninitialized registers or stack slots within landing pads.\n\nPotential failure modes in this patch involve CFG exploration, jump history backtracking, verifier prune points, or stack offset calculations—bugs that fall under control-flow logic, out-of-bounds access, or memory corruption detectable by KASAN and standard kernel assertions rather than KMSAN uninitialized memory detectors.",
"NeedsKMSAN": false,
"Reasoning": "The patch series implements BPF exception cleanup landing pad support across core BPF (verifier CFG, reachability, liveness, backtracking, dead-code elimination) and architecture-specific JIT/unwind dispatch on x86-64 and arm64. This adds new user-facing fields in union bpf_attr for BPF_PROG_LOAD, verifier checks, kfuncs (bpf_unwind_resume), runtime stack unwinding, and assembly dispatchers. The changes touch core subsystems (kernel/bpf and arch/x86/net) and are fully reachable in virtualized environments via the bpf() syscall. Therefore, this warrants fuzzing.",
"WorthFuzzing": true
} |
| 1/1 | 2026/09/22 09:05 | action | read-patch-diff |
0mArgs:null Results: {
"PatchDiff": "commit 68c10631eef7b88304d24e3ffb2d009f8db416fc\nAuthor: syz-cluster \u003ctriage@syzkaller.com\u003e\nDate: Tue Sep 22 09:05:55 2026 +0000\n\n syz-cluster: applied patch under review\n\ndiff --git a/arch/arm64/net/Makefile b/arch/arm64/net/Makefile\nindex 3ae382bfca879..ebec2a44a52b6 100644\n--- a/arch/arm64/net/Makefile\n+++ b/arch/arm64/net/Makefile\n@@ -2,4 +2,4 @@\n #\n # ARM64 networking code\n #\n-obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o bpf_timed_may_goto.o\n+obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o bpf_timed_may_goto.o bpf_cleanup_pad.o\ndiff --git a/arch/arm64/net/bpf_cleanup_pad.S b/arch/arm64/net/bpf_cleanup_pad.S\nnew file mode 100644\nindex 0000000000000..ef441241949ec\n--- /dev/null\n+++ b/arch/arm64/net/bpf_cleanup_pad.S\n@@ -0,0 +1,95 @@\n+/* SPDX-License-Identifier: GPL-2.0 */\n+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */\n+\n+#include \u003clinux/bpf_cleanup_abi.h\u003e\n+#include \u003clinux/linkage.h\u003e\n+\n+/*\n+ * A frame's prologue pushes the tail call counter pair, and then -- once\n+ * bpf_cleanup_force_spill() says so -- x19/x20, x21/x22, x23/x24, x25/x26 and\n+ * x27/x28. Each A64_PUSH pre-decrements, so the lowest address of the spill\n+ * area holds x27 and the highest x20:\n+ *\n+ *\tspill_base + 0\t\tx27 (private stack pointer)\n+ *\tspill_base + 8\t\tx28 (arena base)\n+ *\tspill_base + 16\t\tx25 (BPF r10, the frame pointer)\n+ *\tspill_base + 24\t\tx26 (tail call counter pointer)\n+ *\tspill_base + 32\t\tx23 -- the pad's own, see below\n+ *\tspill_base + 40\t\tx24 -- likewise\n+ *\tspill_base + 48\t\tx21 (BPF r8)\n+ *\tspill_base + 56\t\tx22 (BPF r9)\n+ *\tspill_base + 64\t\tx19 (BPF r6)\n+ *\tspill_base + 72\t\tx20 (BPF r7)\n+ *\n+ * Neither x23 nor x24 is restored from that spill: the pad has its own use for\n+ * both. bpf2a64[] maps nothing to either -- they are pushed only to keep the\n+ * frame shape the exception callback expects -- so nothing else in generated\n+ * code touches them, and being callee-saved they survive every call the pad\n+ * makes.\n+ *\n+ * x23 is the pad's return address. Unlike x86-64 a pad cannot simply return:\n+ * every call it makes clobbers x30, so nothing is left to return through by\n+ * the time it reaches its resume. The JIT emits \"br x23\" for the pad's\n+ * bpf_unwind_resume() and this routine puts .Lcleanup_pad_done there.\n+ *\n+ * x24 is where the pad's frame is anchored. Generated code addresses the BPF\n+ * frame through the stack pointer, which here is this routine's rather than\n+ * the unwinding frame's, so the JIT has the pad recompute the equivalent from\n+ * BPF r10 on entry and address its frame off x24 for as long as it runs.\n+ *\n+ * Both of those branches are indirect, so both targets carry a BTI landing\n+ * marker: the JIT emits one at each pad, and .Lcleanup_pad_done below has one\n+ * of its own.\n+ */\n+\n+\t.text\n+\n+/*\n+ * void arch_bpf_run_cleanup_pad(u64 pad, u64 frame_fp, u64 spill_base)\n+ *\n+ * x0 = native address of the landing pad\n+ * x1 = frame pointer of the frame the pad belongs to (unused here: BPF r10 is\n+ * x25, which the spill area already holds)\n+ * x2 = spill area holding that frame's BPF callee-saved registers\n+ *\n+ * Give the pad the register state of its own frame and call it. It runs on\n+ * this stack, far below the frame it is cleaning up after, so nothing it\n+ * calls can reach into that frame.\n+ */\n+SYM_FUNC_START(arch_bpf_run_cleanup_pad)\n+\t/* Save the kernel's callee-saved registers; the pad owns them next. */\n+\tstp\tx29, x30, [sp, #-96]!\n+\tmov\tx29, sp\n+\tstp\tx19, x20, [sp, #16]\n+\tstp\tx21, x22, [sp, #32]\n+\tstp\tx23, x24, [sp, #48]\n+\tstp\tx25, x26, [sp, #64]\n+\tstp\tx27, x28, [sp, #80]\n+\n+\t/* x9 is BPF_REG_AX, so the pad's address does not stay in BPF r1. */\n+\tmov\tx9, x0\n+\n+\tldp\tx27, x28, [x2, #0]\n+\tldp\tx25, x26, [x2, #16]\n+\tldp\tx21, x22, [x2, #48]\n+\tldp\tx19, x20, [x2, #64]\n+\n+\t/* BPF r0 (x8) on the way into a pad, not whatever the kernel left. */\n+\tmov\tx8, #BPF_PAD_ENTRY_R0\n+\n+\t/* Where the pad's resume branches back to. */\n+\tadr\tx23, .Lcleanup_pad_done\n+\n+\tbr\tx9\n+\n+.Lcleanup_pad_done:\n+\t/* Reached by the pad's \"br x23\", so it is an indirect branch target. */\n+\tbti\tj\n+\tldp\tx19, x20, [sp, #16]\n+\tldp\tx21, x22, [sp, #32]\n+\tldp\tx23, x24, [sp, #48]\n+\tldp\tx25, x26, [sp, #64]\n+\tldp\tx27, x28, [sp, #80]\n+\tldp\tx29, x30, [sp], #96\n+\tret\n+SYM_FUNC_END(arch_bpf_run_cleanup_pad)\ndiff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c\nindex 6c04fee468766..7d26e43e29c3a 100644\n--- a/arch/arm64/net/bpf_jit_comp.c\n+++ b/arch/arm64/net/bpf_jit_comp.c\n@@ -11,6 +11,7 @@\n #include \u003clinux/bitfield.h\u003e\n #include \u003clinux/bpf.h\u003e\n #include \u003clinux/cfi.h\u003e\n+#include \u003clinux/bpf_verifier.h\u003e\n #include \u003clinux/filter.h\u003e\n #include \u003clinux/memory.h\u003e\n #include \u003clinux/printk.h\u003e\n@@ -75,6 +76,19 @@ static const int bpf2a64[] = {\n \t[ARENA_VM_START] = A64_R(28),\n };\n \n+/*\n+ * Throw-site spill: the five pairs push_callee_regs() forces on, same size and\n+ * slot order, so arch_bpf_run_cleanup_pad() reads both alike.\n+ */\n+#define A64_CLEANUP_SPILL_SZ\t(5 * 16)\n+\n+/*\n+ * Where a landing pad's frame is anchored, since the stack pointer generated\n+ * code normally addresses it through is the walker's inside a pad. bpf2a64[]\n+ * maps nothing to x24, so nothing else in generated code touches it.\n+ */\n+#define A64_CLEANUP_FP\t\tA64_R(24)\n+\n struct jit_ctx {\n \tconst struct bpf_prog *prog;\n \tint idx;\n@@ -87,6 +101,8 @@ struct jit_ctx {\n \t__le32 *ro_image;\n \tu32 stack_size;\n \tu16 stack_arg_size;\n+\t/* Bytes reserved for the throw-site spill; see bpf_cleanup_force_spill(). */\n+\tu32 throw_spill;\n \tu64 user_vm_start;\n \tu64 arena_vm_start;\n \tbool fp_used;\n@@ -432,7 +448,7 @@ static void push_callee_regs(struct jit_ctx *ctx)\n \t * Callee-saved registers as the exception callback needs to recover\n \t * all ARM64 Callee-saved registers in its epilogue.\n \t */\n-\tif (ctx-\u003eprog-\u003eaux-\u003eexception_boundary) {\n+\tif (ctx-\u003eprog-\u003eaux-\u003eexception_boundary || bpf_cleanup_force_spill(ctx-\u003eprog)) {\n \t\temit(A64_PUSH(A64_R(19), A64_R(20), A64_SP), ctx);\n \t\temit(A64_PUSH(A64_R(21), A64_R(22), A64_SP), ctx);\n \t\temit(A64_PUSH(A64_R(23), A64_R(24), A64_SP), ctx);\n@@ -466,7 +482,8 @@ static void pop_callee_regs(struct jit_ctx *ctx)\n \t * program's stack frame, so recover these extra registers in the above\n \t * two cases.\n \t */\n-\tif (aux-\u003eexception_boundary || aux-\u003eexception_cb) {\n+\tif (aux-\u003eexception_boundary || aux-\u003eexception_cb ||\n+\t bpf_cleanup_force_spill(ctx-\u003eprog)) {\n \t\temit(A64_POP(A64_R(27), A64_R(28), A64_SP), ctx);\n \t\temit(A64_POP(A64_R(25), A64_R(26), A64_SP), ctx);\n \t\temit(A64_POP(A64_R(23), A64_R(24), A64_SP), ctx);\n@@ -602,6 +619,20 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)\n \t\temit(A64_SUB_I(1, A64_SP, A64_FP, 96), ctx);\n \t}\n \n+\t/*\n+\t * Lowest address of each spill area, as an offset from A64_FP; see\n+\t * bpf_cleanup_pad.S for the layout. The 16 is the tail call counter\n+\t * pair pushed just below the frame record, and the throw-site area\n+\t * sits below the callee-saved one rather than in the program stack.\n+\t */\n+\tif (bpf_cleanup_force_spill(prog)) {\n+\t\tprog-\u003eaux-\u003eexc-\u003espill_off = -(16 + A64_CLEANUP_SPILL_SZ);\n+\t\tctx-\u003ethrow_spill = A64_CLEANUP_SPILL_SZ;\n+\t\temit(A64_SUB_I(1, A64_SP, A64_SP, ctx-\u003ethrow_spill), ctx);\n+\t\tprog-\u003eaux-\u003eexc-\u003ethrow_spill_off =\n+\t\t\t-(16 + A64_CLEANUP_SPILL_SZ) - ctx-\u003ethrow_spill;\n+\t}\n+\n \t/* Stack must be multiples of 16B */\n \tctx-\u003estack_size = round_up(prog-\u003eaux-\u003estack_depth, 16);\n \n@@ -691,6 +722,10 @@ static int emit_bpf_tail_call(struct jit_ctx *ctx)\n \tif (ctx-\u003estack_size \u0026\u0026 !ctx-\u003epriv_sp_used)\n \t\temit(A64_ADD_I(1, A64_SP, A64_SP, ctx-\u003estack_size), ctx);\n \n+\t/* Release it for the same reason build_epilogue() does. */\n+\tif (ctx-\u003ethrow_spill)\n+\t\temit(A64_ADD_I(1, A64_SP, A64_SP, ctx-\u003ethrow_spill), ctx);\n+\n \tpop_callee_regs(ctx);\n \n \t/* goto *(prog-\u003ebpf_func + prologue_offset); */\n@@ -1055,6 +1090,9 @@ static void build_epilogue(struct jit_ctx *ctx, bool was_classic)\n \tif (ctx-\u003estack_size \u0026\u0026 !ctx-\u003epriv_sp_used)\n \t\temit(A64_ADD_I(1, A64_SP, A64_SP, ctx-\u003estack_size), ctx);\n \n+\tif (ctx-\u003ethrow_spill)\n+\t\temit(A64_ADD_I(1, A64_SP, A64_SP, ctx-\u003ethrow_spill), ctx);\n+\n \tpop_callee_regs(ctx);\n \n \temit(A64_POP(A64_ZR, ptr, A64_SP), ctx);\n@@ -1367,6 +1405,8 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn\n \tconst s16 off = insn-\u003eoff;\n \tconst s32 imm = insn-\u003eimm;\n \tconst int i = insn - ctx-\u003eprog-\u003einsnsi;\n+\tconst bool in_pad = bpf_cleanup_insn_in_pad(ctx-\u003eprog, i);\n+\tconst bool pad_head = bpf_cleanup_insn_is_pad(ctx-\u003eprog, i);\n \tconst bool is64 = BPF_CLASS(code) == BPF_ALU64 ||\n \t\t\t BPF_CLASS(code) == BPF_JMP;\n \tu8 jmp_cond;\n@@ -1378,9 +1418,13 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn\n \tint ret;\n \tbool sign_extend;\n \n-\tif (bpf_insn_is_indirect_target(env, ctx-\u003eprog, i))\n+\tif (bpf_insn_is_indirect_target(env, ctx-\u003eprog, i) || pad_head)\n \t\temit_bti(A64_BTI_J, ctx);\n \n+\tif (pad_head)\n+\t\temit(A64_SUB_I(1, A64_CLEANUP_FP, fp,\n+\t\t\t ctx-\u003estack_size + ctx-\u003estack_arg_size), ctx);\n+\n \tswitch (code) {\n \t/* dst = src */\n \tcase BPF_ALU | BPF_MOV | BPF_X:\n@@ -1743,6 +1787,26 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn\n \t\tu64 func_addr;\n \t\tu32 cpu_offset;\n \n+\t\tif (bpf_cleanup_insn_is_throw(ctx-\u003eprog, i)) {\n+\t\t\t/* Spill where the bpf_throw() walker looks. */\n+\t\t\tconst s32 off = ctx-\u003eprog-\u003eaux-\u003eexc-\u003ethrow_spill_off;\n+\n+\t\t\temit(A64_SUB_I(1, tmp, A64_FP, -off), ctx);\n+\t\t\temit(A64_STR64I(bpf2a64[PRIVATE_SP], tmp, 0), ctx);\n+\t\t\temit(A64_STR64I(bpf2a64[ARENA_VM_START], tmp, 8), ctx);\n+\t\t\temit(A64_STR64I(bpf2a64[BPF_REG_FP], tmp, 16), ctx);\n+\t\t\temit(A64_STR64I(bpf2a64[TCCNT_PTR], tmp, 24), ctx);\n+\t\t\temit(A64_STR64I(bpf2a64[BPF_REG_8], tmp, 48), ctx);\n+\t\t\temit(A64_STR64I(bpf2a64[BPF_REG_9], tmp, 56), ctx);\n+\t\t\temit(A64_STR64I(bpf2a64[BPF_REG_6], tmp, 64), ctx);\n+\t\t\temit(A64_STR64I(bpf2a64[BPF_REG_7], tmp, 72), ctx);\n+\t\t}\n+\n+\t\tif (bpf_cleanup_insn_is_resume(ctx-\u003eprog, i)) {\n+\t\t\temit(A64_BR(A64_R(23)), ctx);\n+\t\t\tbreak;\n+\t\t}\n+\n \t\t/* Implement helper call to bpf_get_smp_processor_id() inline */\n \t\tif (insn-\u003esrc_reg == 0 \u0026\u0026 insn-\u003eimm == BPF_FUNC_get_smp_processor_id) {\n \t\t\tcpu_offset = offsetof(struct thread_info, cpu);\n@@ -1854,7 +1918,8 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn\n \t\t\tsrc = tmp2;\n \t\t}\n \t\tif (src == fp) {\n-\t\t\tsrc_adj = ctx-\u003epriv_sp_used ? priv_sp : A64_SP;\n+\t\t\tsrc_adj = ctx-\u003epriv_sp_used ? priv_sp :\n+\t\t\t\t in_pad ? A64_CLEANUP_FP : A64_SP;\n \t\t\toff_adj = off + ctx-\u003estack_size;\n \t\t\tif (!ctx-\u003epriv_sp_used)\n \t\t\t\toff_adj += ctx-\u003estack_arg_size;\n@@ -1952,7 +2017,8 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn\n \t\t\tdst = tmp3;\n \t\t}\n \t\tif (dst == fp) {\n-\t\t\tdst_adj = ctx-\u003epriv_sp_used ? priv_sp : A64_SP;\n+\t\t\tdst_adj = ctx-\u003epriv_sp_used ? priv_sp :\n+\t\t\t\t in_pad ? A64_CLEANUP_FP : A64_SP;\n \t\t\toff_adj = off + ctx-\u003estack_size;\n \t\t\tif (!ctx-\u003epriv_sp_used)\n \t\t\t\toff_adj += ctx-\u003estack_arg_size;\n@@ -2021,7 +2087,8 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn\n \t\t\tdst = tmp2;\n \t\t}\n \t\tif (dst == fp) {\n-\t\t\tdst_adj = ctx-\u003epriv_sp_used ? priv_sp : A64_SP;\n+\t\t\tdst_adj = ctx-\u003epriv_sp_used ? priv_sp :\n+\t\t\t\t in_pad ? A64_CLEANUP_FP : A64_SP;\n \t\t\toff_adj = off + ctx-\u003estack_size;\n \t\t\tif (!ctx-\u003epriv_sp_used)\n \t\t\t\toff_adj += ctx-\u003estack_arg_size;\n@@ -2410,6 +2477,13 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr\n \t\t * reasons, expects to point to the next instruction)\n \t\t */\n \t\tbpf_prog_update_insn_ptrs(prog, ctx.offset, ctx.ro_image);\n+\n+\t\t/*\n+\t\t * Same byte offsets, consumed by the bpf_throw() frame walker:\n+\t\t * turn the cleanup records into native address ranges now that\n+\t\t * the image is final.\n+\t\t */\n+\t\tbpf_cleanup_fill_native_ranges(prog, ctx.offset, ctx.ro_image);\n out_off:\n \t\tif (!ro_header \u0026\u0026 priv_stack_ptr) {\n \t\t\tfree_percpu(priv_stack_ptr);\n@@ -3385,6 +3459,11 @@ bool bpf_jit_supports_exceptions(void)\n \treturn true;\n }\n \n+bool bpf_jit_supports_cleanup_pads(void)\n+{\n+\treturn true;\n+}\n+\n bool bpf_jit_supports_arena(void)\n {\n \treturn true;\ndiff --git a/arch/x86/net/Makefile b/arch/x86/net/Makefile\nindex dddbefc0f4398..8bb22644ba10b 100644\n--- a/arch/x86/net/Makefile\n+++ b/arch/x86/net/Makefile\n@@ -7,4 +7,7 @@ ifeq ($(CONFIG_X86_32),y)\n obj-$(CONFIG_BPF_JIT) += bpf_jit_comp32.o\n else\n obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o bpf_timed_may_goto.o\n+ ifdef CONFIG_UNWINDER_ORC\n+ obj-$(CONFIG_BPF_JIT) += bpf_cleanup_pad.o\n+ endif\n endif\ndiff --git a/arch/x86/net/bpf_cleanup_pad.S b/arch/x86/net/bpf_cleanup_pad.S\nnew file mode 100644\nindex 0000000000000..025a54e718a40\n--- /dev/null\n+++ b/arch/x86/net/bpf_cleanup_pad.S\n@@ -0,0 +1,80 @@\n+/* SPDX-License-Identifier: GPL-2.0 */\n+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */\n+\n+#include \u003clinux/bpf_cleanup_abi.h\u003e\n+#include \u003clinux/linkage.h\u003e\n+#include \u003casm/nospec-branch.h\u003e\n+\n+/*\n+ * The x86-64 BPF JIT prologue spills, once bpf_cleanup_force_spill() makes it\n+ * unconditional, r12, rbx, r13, r14 and r15 in that order -- so within the\n+ * spill area the lowest address holds r15 and the highest r12. The throw-site\n+ * spill the JIT emits uses the same layout, so the routine below reads both\n+ * the same way:\n+ *\n+ *\tspill_base + 0\t\tBPF r9 (r15)\n+ *\tspill_base + 8\t\tBPF r8 (r14)\n+ *\tspill_base + 16\t\tBPF r7 (r13)\n+ *\tspill_base + 24\t\tBPF r6 (rbx)\n+ *\tspill_base + 32\t\tr12 (arena base, not a BPF register)\n+ */\n+\n+\t.code64\n+\t.section .text, \"ax\"\n+\n+/*\n+ * void arch_bpf_run_cleanup_pad(u64 pad, u64 frame_fp, u64 spill_base)\n+ *\n+ * rdi = native address of the landing pad\n+ * rsi = frame pointer of the frame the pad belongs to\n+ * rdx = spill area holding that frame's BPF callee-saved registers\n+ *\n+ * Give the pad the register state of its own frame and call it. The pad ends\n+ * in the bare return the JIT emits for its bpf_unwind_resume(), so it comes\n+ * back here rather than returning to its frame's caller. It runs on this\n+ * stack, far below the frame it is cleaning up after, so nothing it calls can\n+ * reach into that frame.\n+ *\n+ * rbp addresses the pad's frame and rsp this one, so the two are not the\n+ * neighbours a JITed frame expects them to be. That is why\n+ * cleanup_check_pad_insn() refuses a call in a pad that passes an argument on\n+ * the stack: the JIT stages those relative to rbp, and the callee reads them\n+ * relative to rsp.\n+ */\n+SYM_FUNC_START(arch_bpf_run_cleanup_pad)\n+\tANNOTATE_NOENDBR\n+\n+\tpushq %rbp\n+\tpushq %rbx\n+\tpushq %r12\n+\tpushq %r13\n+\tpushq %r14\n+\tpushq %r15\n+\t/* Keep the pad's entry rsp congruent to a normal call's. */\n+\tsubq $8, %rsp\n+\n+\tmovq 0(%rdx), %r15\n+\tmovq 8(%rdx), %r14\n+\tmovq 16(%rdx), %r13\n+\tmovq 24(%rdx), %rbx\n+\tmovq 32(%rdx), %r12\n+\t/* rbp is BPF r10, so this is the whole of the pad's frame setup. */\n+\tmovq %rsi, %rbp\n+\n+\t/* CALL_NOSPEC needs the target in a register; rcx is BPF r4, dead. */\n+\tmovq %rdi, %rcx\n+\n+\t/* BPF r0 on the way into a pad, not whatever the kernel left in rax. */\n+\tmovl $BPF_PAD_ENTRY_R0, %eax\n+\n+\tCALL_NOSPEC rcx\n+\n+\taddq $8, %rsp\n+\tpopq %r15\n+\tpopq %r14\n+\tpopq %r13\n+\tpopq %r12\n+\tpopq %rbx\n+\tpopq %rbp\n+\tRET\n+SYM_FUNC_END(arch_bpf_run_cleanup_pad)\ndiff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c\nindex d4a980140b48d..04db7d7fc397e 100644\n--- a/arch/x86/net/bpf_jit_comp.c\n+++ b/arch/x86/net/bpf_jit_comp.c\n@@ -357,6 +357,12 @@ struct jit_context {\n /* Number of bytes that will be skipped on tailcall */\n #define X86_TAIL_CALL_OFFSET\t(12 + ENDBR_INSN_SIZE)\n \n+/*\n+ * Throw-site spill: r15, r14, r13, rbx, r12 low to high, the layout the\n+ * prologue's pushes leave, so arch_bpf_run_cleanup_pad() reads both alike.\n+ */\n+#define X86_CLEANUP_SPILL_SZ\t(5 * 8)\n+\n static void push_r9(u8 **pprog)\n {\n \tu8 *prog = *pprog;\n@@ -832,7 +838,7 @@ static void emit_bpf_tail_call_indirect(struct bpf_prog *bpf_prog,\n \t/* Inc tail_call_cnt if the slot is populated. */\n \tEMIT4(0x48, 0x83, 0x00, 0x01); /* add qword ptr [rax], 1 */\n \n-\tif (bpf_prog-\u003eaux-\u003eexception_boundary) {\n+\tif (bpf_prog-\u003eaux-\u003eexception_boundary || bpf_cleanup_force_spill(bpf_prog)) {\n \t\tpop_callee_regs(\u0026prog, all_callee_regs_used);\n \t\tpop_r12(\u0026prog);\n \t} else {\n@@ -899,7 +905,7 @@ static void emit_bpf_tail_call_direct(struct bpf_prog *bpf_prog,\n \t/* Inc tail_call_cnt if the slot is populated. */\n \tEMIT4(0x48, 0x83, 0x00, 0x01); /* add qword ptr [rax], 1 */\n \n-\tif (bpf_prog-\u003eaux-\u003eexception_boundary) {\n+\tif (bpf_prog-\u003eaux-\u003eexception_boundary || bpf_cleanup_force_spill(bpf_prog)) {\n \t\tpop_callee_regs(\u0026prog, all_callee_regs_used);\n \t\tpop_r12(\u0026prog);\n \t} else {\n@@ -1977,6 +1983,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *\n \tu8 *ip, *prog = temp;\n \tu32 stack_depth;\n \tint callee_saved_size;\n+\tu32 throw_spill, prologue_depth;\n \ts32 outgoing_arg_base;\n \tint err;\n \n@@ -2015,7 +2022,10 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *\n \n \tdetect_reg_usage(insn, insn_cnt, callee_regs_used);\n \n-\temit_prologue(\u0026prog, image, stack_depth,\n+\tthrow_spill = bpf_cleanup_force_spill(bpf_prog) ? X86_CLEANUP_SPILL_SZ : 0;\n+\tprologue_depth = stack_depth + throw_spill;\n+\n+\temit_prologue(\u0026prog, image, prologue_depth,\n \t\t bpf_prog_was_classic(bpf_prog), tail_call_reachable,\n \t\t bpf_is_subprog(bpf_prog), bpf_prog-\u003eaux-\u003eexception_cb);\n \n@@ -2024,7 +2034,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *\n \t/* Exception callback will clobber callee regs for its own use, and\n \t * restore the original callee regs from main prog's stack frame.\n \t */\n-\tif (bpf_prog-\u003eaux-\u003eexception_boundary) {\n+\tif (bpf_prog-\u003eaux-\u003eexception_boundary || bpf_cleanup_force_spill(bpf_prog)) {\n \t\t/* We also need to save r12, which is not mapped to any BPF\n \t\t * register, as we throw after entry into the kernel, which may\n \t\t * overwrite r12.\n@@ -2039,9 +2049,10 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *\n \n \t/* Compute callee-saved register area size. */\n \tcallee_saved_size = 0;\n-\tif (bpf_prog-\u003eaux-\u003eexception_boundary || arena_vm_start)\n+\tif (bpf_prog-\u003eaux-\u003eexception_boundary || bpf_cleanup_force_spill(bpf_prog) ||\n+\t arena_vm_start)\n \t\tcallee_saved_size += 8; /* r12 */\n-\tif (bpf_prog-\u003eaux-\u003eexception_boundary) {\n+\tif (bpf_prog-\u003eaux-\u003eexception_boundary || bpf_cleanup_force_spill(bpf_prog)) {\n \t\tcallee_saved_size += 4 * 8; /* rbx, r13, r14, r15 */\n \t} else {\n \t\tint j;\n@@ -2063,7 +2074,19 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *\n \t * Note that tail_call_reachable is guaranteed to be false when\n \t * stack args exist, so tcc pushes need not be accounted for.\n \t */\n-\toutgoing_arg_base = -(round_up(stack_depth, 8) + callee_saved_size);\n+\toutgoing_arg_base = -(round_up(stack_depth, 8) + throw_spill + callee_saved_size);\n+\n+\t/*\n+\t * Lowest address of each spill area, as an offset from rbp; see\n+\t * bpf_cleanup_pad.S for the layout. The 16 is the tail call counter\n+\t * pair emit_prologue_tail_call() pushes above the callee-saved one.\n+\t */\n+\tif (bpf_cleanup_force_spill(bpf_prog)) {\n+\t\tbpf_prog-\u003eaux-\u003eexc-\u003espill_off = -(round_up(stack_depth, 8) + throw_spill +\n+\t\t\t\t\t\t (tail_call_reachable ? 16 : 0) +\n+\t\t\t\t\t\t callee_saved_size);\n+\t\tbpf_prog-\u003eaux-\u003eexc-\u003ethrow_spill_off = -(round_up(stack_depth, 8) + throw_spill);\n+\t}\n \n \t/*\n \t * Allocate outgoing stack arg area for args 7+ only.\n@@ -2110,7 +2133,8 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *\n \t\t\t\tdst_reg = X86_REG_R9;\n \t\t}\n \n-\t\tif (bpf_insn_is_indirect_target(env, bpf_prog, i - 1))\n+\t\tif (bpf_insn_is_indirect_target(env, bpf_prog, i - 1) ||\n+\t\t bpf_cleanup_insn_is_pad(bpf_prog, i - 1))\n \t\t\tEMIT_ENDBR();\n \n \t\tip = image + addrs[i - 1] + (prog - temp);\n@@ -2903,9 +2927,27 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *\n \t\tcase BPF_JMP | BPF_CALL: {\n \t\t\tconst struct btf_func_model *fm = NULL;\n \n+\t\t\tif (bpf_cleanup_insn_is_throw(bpf_prog, i - 1)) {\n+\t\t\t\t/* Spill r6-r9 and r12 where the bpf_throw() walker looks. */\n+\t\t\t\ts32 off = bpf_prog-\u003eaux-\u003eexc-\u003ethrow_spill_off;\n+\t\t\t\tu8 *spill = prog;\n+\n+\t\t\t\temit_stx(\u0026prog, BPF_DW, BPF_REG_FP, BPF_REG_9, off + 0);\n+\t\t\t\temit_stx(\u0026prog, BPF_DW, BPF_REG_FP, BPF_REG_8, off + 8);\n+\t\t\t\temit_stx(\u0026prog, BPF_DW, BPF_REG_FP, BPF_REG_7, off + 16);\n+\t\t\t\temit_stx(\u0026prog, BPF_DW, BPF_REG_FP, BPF_REG_6, off + 24);\n+\t\t\t\temit_stx(\u0026prog, BPF_DW, BPF_REG_FP, X86_REG_R12, off + 32);\n+\t\t\t\tip += prog - spill;\n+\t\t\t}\n+\n+\t\t\tif (bpf_cleanup_insn_is_resume(bpf_prog, i - 1)) {\n+\t\t\t\temit_return(\u0026prog, image + addrs[i - 1] + (prog - temp));\n+\t\t\t\tbreak;\n+\t\t\t}\n+\n \t\t\tfunc = (u8 *) __bpf_call_base + imm32;\n \t\t\tif (src_reg == BPF_PSEUDO_CALL \u0026\u0026 tail_call_reachable) {\n-\t\t\t\tLOAD_TAIL_CALL_CNT_PTR(stack_depth);\n+\t\t\t\tLOAD_TAIL_CALL_CNT_PTR(prologue_depth);\n \t\t\t\tip += 7;\n \t\t\t}\n \t\t\tif (!imm32)\n@@ -2948,13 +2990,13 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *\n \t\t\t\t\t\t\t \u0026prog,\n \t\t\t\t\t\t\t ip,\n \t\t\t\t\t\t\t callee_regs_used,\n-\t\t\t\t\t\t\t stack_depth,\n+\t\t\t\t\t\t\t prologue_depth,\n \t\t\t\t\t\t\t ctx);\n \t\t\telse\n \t\t\t\temit_bpf_tail_call_indirect(bpf_prog,\n \t\t\t\t\t\t\t \u0026prog,\n \t\t\t\t\t\t\t callee_regs_used,\n-\t\t\t\t\t\t\t stack_depth,\n+\t\t\t\t\t\t\t prologue_depth,\n \t\t\t\t\t\t\t ip,\n \t\t\t\t\t\t\t ctx);\n \t\t\tbreak;\n@@ -3215,7 +3257,8 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *\n \t\t\t}\n \t\t\t/* Deallocate outgoing args 7+ area. */\n \t\t\temit_add_rsp(\u0026prog, outgoing_rsp);\n-\t\t\tif (bpf_prog-\u003eaux-\u003eexception_boundary) {\n+\t\t\tif (bpf_prog-\u003eaux-\u003eexception_boundary ||\n+\t\t\t bpf_cleanup_force_spill(bpf_prog)) {\n \t\t\t\tpop_callee_regs(\u0026prog, all_callee_regs_used);\n \t\t\t\tpop_r12(\u0026prog);\n \t\t\t} else {\n@@ -4385,6 +4428,13 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr\n \t\t */\n \t\tbpf_prog_update_insn_ptrs(prog, addrs, image);\n \n+\t\t/*\n+\t\t * Same mapping, consumed by the bpf_throw() frame walker:\n+\t\t * turn the cleanup records into native address ranges now\n+\t\t * that the image is final.\n+\t\t */\n+\t\tbpf_cleanup_fill_native_ranges(prog, addrs, image);\n+\n \t\t/*\n \t\t * ctx.prog_offset is used when CFI preambles put code *before*\n \t\t * the function. See emit_cfi(). For FineIBT specifically this code\n@@ -4501,6 +4551,11 @@ bool bpf_jit_supports_exceptions(void)\n \treturn IS_ENABLED(CONFIG_UNWINDER_ORC);\n }\n \n+bool bpf_jit_supports_cleanup_pads(void)\n+{\n+\treturn IS_ENABLED(CONFIG_UNWINDER_ORC);\n+}\n+\n bool bpf_jit_supports_private_stack(void)\n {\n \treturn true;\ndiff --git a/include/linux/bpf.h b/include/linux/bpf.h\nindex 5033b934ffd90..0502b9fc4a6de 100644\n--- a/include/linux/bpf.h\n+++ b/include/linux/bpf.h\n@@ -1774,6 +1774,89 @@ enum bpf_sig_keyring {\n \tBPF_SIG_KEYRING_BPF,\n };\n \n+/*\n+ * One cleanup region of a JITed (sub)program: @pad is the landing pad to run\n+ * for a return address in (begin, end], the native code of its call sites.\n+ */\n+struct bpf_cleanup_range {\n+\tu64 begin;\n+\tu64 end;\n+\tu64 pad;\n+};\n+\n+struct bpf_exception_info {\n+\tstruct bpf_cleanup_info *info;\n+\tstruct bpf_cleanup_range *ranges;\n+\t/* Landing pad instruction indices, sorted and deduplicated. */\n+\tu32 *pad_at;\n+\t/* bpf_throw() call instruction indices, sorted. */\n+\tu32 *throw_at;\n+\t/* Likewise for bpf_unwind_resume(), the way back out of a pad. */\n+\tu32 *resume_at;\n+\t/* One bit per instruction that only runs while unwinding. */\n+\tunsigned long *pad_body;\n+\tu32 nr_info;\n+\tu32 nr_ranges;\n+\tu32 nr_pad_at;\n+\tu32 nr_throw_at;\n+\tu32 nr_resume_at;\n+\tu32 nr_pad_body;\n+\t/* Offset from a frame's FP to the caller's spilled r6-r9. */\n+\ts32 spill_off;\n+\t/* Likewise, to the registers a frame spills before calling bpf_throw(). */\n+\ts32 throw_spill_off;\n+};\n+\n+#ifdef CONFIG_BPF_SYSCALL\n+bool bpf_cleanup_force_spill(const struct bpf_prog *prog);\n+bool bpf_cleanup_insn_is_pad(const struct bpf_prog *prog, u32 idx);\n+bool bpf_cleanup_insn_in_pad(const struct bpf_prog *prog, u32 idx);\n+bool bpf_cleanup_insn_is_throw(const struct bpf_prog *prog, u32 idx);\n+bool bpf_cleanup_insn_is_resume(const struct bpf_prog *prog, u32 idx);\n+int bpf_cleanup_attach_main_prog(struct bpf_verifier_env *env, struct bpf_prog *prog);\n+void bpf_cleanup_fill_native_ranges(struct bpf_prog *prog, u32 *addrs, void *image);\n+void bpf_cleanup_free_info(struct bpf_prog_aux *aux);\n+#else\n+static inline bool bpf_cleanup_force_spill(const struct bpf_prog *prog)\n+{\n+\treturn false;\n+}\n+\n+static inline bool bpf_cleanup_insn_is_pad(const struct bpf_prog *prog, u32 idx)\n+{\n+\treturn false;\n+}\n+\n+static inline bool bpf_cleanup_insn_in_pad(const struct bpf_prog *prog, u32 idx)\n+{\n+\treturn false;\n+}\n+\n+static inline bool bpf_cleanup_insn_is_throw(const struct bpf_prog *prog, u32 idx)\n+{\n+\treturn false;\n+}\n+\n+static inline bool bpf_cleanup_insn_is_resume(const struct bpf_prog *prog, u32 idx)\n+{\n+\treturn false;\n+}\n+\n+static inline int bpf_cleanup_attach_main_prog(struct bpf_verifier_env *env,\n+\t\t\t\t\t struct bpf_prog *prog)\n+{\n+\treturn 0;\n+}\n+\n+static inline void bpf_cleanup_fill_native_ranges(struct bpf_prog *prog, u32 *addrs, void *image)\n+{\n+}\n+\n+static inline void bpf_cleanup_free_info(struct bpf_prog_aux *aux)\n+{\n+}\n+#endif\n+\n struct bpf_prog_aux {\n \tatomic64_t refcnt;\n \tu32 used_map_cnt;\n@@ -1854,6 +1937,7 @@ struct bpf_prog_aux {\n \tchar name[BPF_OBJ_NAME_LEN];\n \tu64 (*bpf_exception_cb)(u64 cookie, u64 sp, u64 bp, u64, u64);\n \tu16 stack_arg_sp_adjust;\n+\tstruct bpf_exception_info *exc;\n #ifdef CONFIG_SECURITY\n \tvoid *security;\n #endif\ndiff --git a/include/linux/bpf_cleanup_abi.h b/include/linux/bpf_cleanup_abi.h\nnew file mode 100644\nindex 0000000000000..b6c1d589abda7\n--- /dev/null\n+++ b/include/linux/bpf_cleanup_abi.h\n@@ -0,0 +1,16 @@\n+/* SPDX-License-Identifier: GPL-2.0-only */\n+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */\n+#ifndef _LINUX_BPF_CLEANUP_ABI_H\n+#define _LINUX_BPF_CLEANUP_ABI_H\n+\n+/*\n+ * Value arch_bpf_run_cleanup_pad() leaves in r0 on the way into a landing pad.\n+ * It has to be a constant the verifier knows: LLVM names r0 as both the\n+ * exception pointer and the exception selector register, so every pad reads it\n+ * before anything else and is free to store what it read. Kept on its own\n+ * because the verifier and the arch dispatchers, which are assembly, have to\n+ * agree on it.\n+ */\n+#define BPF_PAD_ENTRY_R0\t1\n+\n+#endif /* _LINUX_BPF_CLEANUP_ABI_H */\ndiff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h\nindex cf85141ea1674..0721461aaabbf 100644\n--- a/include/linux/bpf_verifier.h\n+++ b/include/linux/bpf_verifier.h\n@@ -429,7 +429,8 @@ struct bpf_jmp_history_entry {\n \tu32 prev_idx : 20;\n \t/* special INSN_F_xxx flags */\n \tu32 flags : 4;\n-\tu32 : 8;\n+\tu32 unwind_frames : 4;\t/* frames the unwind popped to get here */\n+\tu32 : 4;\n \t/*\n \t * additional registers that need precision tracking when this\n \t * jump is backtracked, vector of five 11-bit records\n@@ -509,6 +510,8 @@ struct bpf_verifier_state {\n \n \tbool speculative;\n \tbool in_sleepable;\n+\tbool unwinding; /* an exception is in flight */\n+\tu8 unwind_frameno; /* the frame whose landing pad it entered */\n \n \t/* first and last insn idx of this verifier state */\n \tu32 first_insn_idx;\n@@ -681,6 +684,14 @@ struct bpf_insn_aux_data {\n \tbool needs_zext; /* alu op needs to clear upper bits */\n \tbool non_sleepable; /* helper/kfunc may be called from non-sleepable context */\n \tbool is_iter_next; /* bpf_iter_\u003ctype\u003e_next() kfunc call */\n+\tbool cleanup_throw_site; /* call to bpf_throw() */\n+\tbool cleanup_resume_site; /* call to bpf_unwind_resume() */\n+\tbool in_cleanup_pad; /* only runs with an exception in flight */\n+\t/*\n+\t * 1 + the instruction index of the exception cleanup landing pad this\n+\t * call site unwinds to, or 0 for none.\n+\t */\n+\tu32 cleanup_pad;\n \tbool call_with_percpu_alloc_ptr; /* {this,per}_cpu_ptr() with prog percpu alloc */\n \tu8 alu_state; /* used in combination with alu_limit */\n \t/* true if STX or LDX instruction is a part of a spill/fill\n@@ -983,10 +994,13 @@ struct bpf_verifier_env {\n \t} cfg;\n \tstruct backtrack_state bt;\n \tstruct bpf_jmp_history_entry *cur_hist_ent;\n+\tu8 unwind_frames; /* scratch: pops for the insn about to be recorded */\n \t/* Per-callsite copy of parent's converged at_stack_in for cross-frame fills. */\n \tstruct arg_track **callsite_at_stack;\n \tu32 pass_cnt; /* number of times do_check() was called */\n \tu32 subprog_cnt;\n+\tstruct bpf_cleanup_info *cleanup_info;\n+\tu32 cleanup_info_cnt;\n \t/* number of instructions analyzed by the verifier */\n \tu32 prev_insn_processed, insn_processed;\n \t/* number of jmps, calls, exits analyzed so far */\n@@ -1516,6 +1530,7 @@ u32 btf_func_arg_align(const struct btf *btf, const struct btf_type *t);\n \n int bpf_find_subprog(struct bpf_verifier_env *env, int off);\n bool bpf_is_throw_kfunc(struct bpf_insn *insn);\n+bool bpf_is_unwind_resume_kfunc(const struct bpf_insn *insn);\n int bpf_compute_const_regs(struct bpf_verifier_env *env);\n int bpf_prune_dead_branches(struct bpf_verifier_env *env);\n int bpf_check_cfg(struct bpf_verifier_env *env);\ndiff --git a/include/linux/filter.h b/include/linux/filter.h\nindex 422284b4fa96f..b0c495e94f740 100644\n--- a/include/linux/filter.h\n+++ b/include/linux/filter.h\n@@ -1242,6 +1242,8 @@ bool bpf_jit_supports_stack_args(void);\n bool bpf_jit_supports_arena_args(void);\n bool bpf_jit_supports_far_kfunc_call(void);\n bool bpf_jit_supports_exceptions(void);\n+bool bpf_jit_supports_cleanup_pads(void);\n+void arch_bpf_run_cleanup_pad(u64 pad, u64 frame_fp, u64 spill_base);\n bool bpf_jit_supports_ptr_xchg(void);\n bool bpf_jit_supports_arena(void);\n bool bpf_jit_supports_insn(struct bpf_insn *insn, bool in_arena);\ndiff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h\nindex 6330b7d745c57..a69cc04275751 100644\n--- a/include/uapi/linux/bpf.h\n+++ b/include/uapi/linux/bpf.h\n@@ -1669,6 +1669,9 @@ union bpf_attr {\n \t\t * verification.\n \t\t */\n \t\t__s32\t\tkeyring_id;\n+\t\t__aligned_u64\tcleanup_info;\t/* exception cleanup table */\n+\t\t__u32\t\tcleanup_info_rec_size; /* userspace bpf_cleanup_info size */\n+\t\t__u32\t\tcleanup_info_cnt; /* number of bpf_cleanup_info records */\n \t};\n \n \tstruct { /* anonymous struct used by BPF_OBJ_* commands */\n@@ -7599,6 +7602,12 @@ struct bpf_line_info {\n \t__u32\tline_col;\n };\n \n+struct bpf_cleanup_info {\n+\t__u32\tbegin_off;\n+\t__u32\tend_off;\n+\t__u32\tlanding_pad_off;\n+};\n+\n struct bpf_spin_lock {\n \t__u32\tval;\n };\ndiff --git a/kernel/bpf/Makefile b/kernel/bpf/Makefile\nindex 9a92c348bbda6..af9bc60428ad1 100644\n--- a/kernel/bpf/Makefile\n+++ b/kernel/bpf/Makefile\n@@ -11,7 +11,7 @@ obj-$(CONFIG_BPF_SYSCALL) += bpf_iter.o map_iter.o task_iter.o prog_iter.o link_\n obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o bloom_filter.o\n obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o bpf_insn_array.o\n obj-$(CONFIG_BPF_SYSCALL) += bpf_local_storage.o bpf_task_storage.o\n-obj-$(CONFIG_BPF_SYSCALL) += fixups.o cfg.o states.o backtrack.o check_btf.o\n+obj-$(CONFIG_BPF_SYSCALL) += fixups.o cfg.o states.o backtrack.o check_btf.o exception.o\n obj-${CONFIG_BPF_LSM}\t += bpf_inode_storage.o\n obj-$(CONFIG_BPF_SYSCALL) += disasm.o mprog.o\n obj-$(CONFIG_BPF_JIT) += trampoline.o\ndiff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c\nindex 507a366dffa47..c19e4fe247ce5 100644\n--- a/kernel/bpf/backtrack.c\n+++ b/kernel/bpf/backtrack.c\n@@ -4,6 +4,7 @@\n #include \u003clinux/bpf_verifier.h\u003e\n #include \u003clinux/filter.h\u003e\n #include \u003clinux/bitmap.h\u003e\n+#include \"exception.h\"\n \n #define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)\n \n@@ -47,6 +48,7 @@ int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state\n \tp-\u003eflags = insn_flags;\n \tp-\u003espi = spi;\n \tp-\u003eframe = frame;\n+\tp-\u003eunwind_frames = 0;\n \tp-\u003elinked_regs = linked_regs;\n \tcur-\u003ejmp_history_cnt = cnt;\n \tenv-\u003ecur_hist_ent = p;\n@@ -419,10 +421,12 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,\n \t\t\t\t * extra instructions from subprog; the next\n \t\t\t\t * instruction after call to global subprog\n \t\t\t\t * should be literally next instruction in\n-\t\t\t\t * caller program\n+\t\t\t\t * caller program -- or, if the callee threw,\n+\t\t\t\t * the landing pad of this call site\n \t\t\t\t */\n-\t\t\t\tverifier_bug_if(idx + 1 != subseq_idx, env,\n-\t\t\t\t\t\t\"extra insn from subprog\");\n+\t\t\t\tverifier_bug_if(idx + 1 != subseq_idx \u0026\u0026\n+\t\t\t\t\t\tbpf_cleanup_pad_of_call(env, idx) != subseq_idx,\n+\t\t\t\t\t\tenv, \"extra insn from subprog\");\n \t\t\t\t/* global subprog always sets R0 */\n \t\t\t\tbt_clear_reg(bt, BPF_REG_0);\n \t\t\t\t/* and if it does not set R2, main pass would catch it */\n@@ -888,11 +892,11 @@ int bpf_mark_chain_precision(struct bpf_verifier_env *env,\n \t\t}\n \n \t\tfor (i = last_idx;;) {\n+\t\t\thist = get_jmp_hist_entry(st, history, i);\n \t\t\tif (skip_first) {\n \t\t\t\terr = 0;\n \t\t\t\tskip_first = false;\n \t\t\t} else {\n-\t\t\t\thist = get_jmp_hist_entry(st, history, i);\n \t\t\t\terr = backtrack_insn(env, i, subseq_idx, hist, bt);\n \t\t\t}\n \t\t\tif (err == -ENOTSUPP) {\n@@ -909,6 +913,15 @@ int bpf_mark_chain_precision(struct bpf_verifier_env *env,\n \t\t\t\t */\n \t\t\t\treturn 0;\n \t\t\tsubseq_idx = i;\n+\t\t\t/* An exception unwind reached this insn, a landing\n+\t\t\t * pad, from a throw or a resume that many frames\n+\t\t\t * deeper. There is no insn in between to backtrack\n+\t\t\t * over, so enter those frames here, the way BPF_EXIT\n+\t\t\t * does one at a time.\n+\t\t\t */\n+\t\t\tfor (fr = 0; hist \u0026\u0026 fr \u003c hist-\u003eunwind_frames; fr++)\n+\t\t\t\tif (bt_subprog_enter(bt))\n+\t\t\t\t\treturn -EFAULT;\n \t\t\ti = get_prev_insn_idx(st, i, \u0026history);\n \t\t\tif (i == -ENOENT)\n \t\t\t\tbreak;\ndiff --git a/kernel/bpf/cfg.c b/kernel/bpf/cfg.c\nindex 842c7d1eabccc..3fd98dcadce89 100644\n--- a/kernel/bpf/cfg.c\n+++ b/kernel/bpf/cfg.c\n@@ -6,6 +6,7 @@\n #include \u003clinux/sort.h\u003e\n \n #include \"diagnostics.h\"\n+#include \"exception.h\"\n \n #define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)\n \n@@ -158,17 +159,64 @@ static int push_insn(int t, int w, int e, struct bpf_verifier_env *env)\n \treturn DONE_EXPLORING;\n }\n \n+static int visit_cleanup_pad_edge(int t, struct bpf_verifier_env *env)\n+{\n+\tint *insn_stack = env-\u003ecfg.insn_stack;\n+\tint *insn_state = env-\u003ecfg.insn_state;\n+\tint w;\n+\n+\tif (!env-\u003ecleanup_info_cnt)\n+\t\treturn DONE_EXPLORING;\n+\tw = bpf_cleanup_pad_of_call(env, t);\n+\tif (w \u003c 0)\n+\t\treturn DONE_EXPLORING;\n+\n+\t/*\n+\t * @t is a call that may branch here, and @w is the target of that\n+\t * branch, so both are prune points. @w especially: every covered call\n+\t * site in a region unwinds to the same pad, and without a prune point\n+\t * at its head the verifier walks the pad again for each of them.\n+\t */\n+\tmark_prune_point(env, t);\n+\tmark_prune_point(env, w);\n+\tmark_jmp_point(env, w);\n+\tmark_jump_target(env, w);\n+\n+\tif (insn_state[w])\n+\t\treturn DONE_EXPLORING;\n+\tif (env-\u003ecfg.cur_stack \u003e= env-\u003eprog-\u003elen)\n+\t\treturn -E2BIG;\n+\tinsn_stack[env-\u003ecfg.cur_stack++] = w;\n+\tinsn_state[w] |= DISCOVERED;\n+\treturn KEEP_EXPLORING;\n+}\n+\n+static int merge_visit_ret(int a, int b)\n+{\n+\tif (a \u003c 0)\n+\t\treturn a;\n+\tif (b \u003c 0)\n+\t\treturn b;\n+\tif (a == KEEP_EXPLORING || b == KEEP_EXPLORING)\n+\t\treturn KEEP_EXPLORING;\n+\treturn DONE_EXPLORING;\n+}\n+\n static int visit_func_call_insn(int t, struct bpf_insn *insns,\n \t\t\t\tstruct bpf_verifier_env *env,\n \t\t\t\tbool visit_callee)\n {\n-\tint ret, insn_sz;\n+\tint ret, insn_sz, pad_ret;\n \tint w;\n \n+\tpad_ret = visit_cleanup_pad_edge(t, env);\n+\tif (pad_ret \u003c 0)\n+\t\treturn pad_ret;\n+\n \tinsn_sz = bpf_is_ldimm64(\u0026insns[t]) ? 2 : 1;\n \tret = push_insn(t, t + insn_sz, FALLTHROUGH, env);\n \tif (ret)\n-\t\treturn ret;\n+\t\treturn merge_visit_ret(pad_ret, ret);\n \n \tmark_prune_point(env, t + insn_sz);\n \t/* when we exit from subprog, we need to record non-linear history */\n@@ -180,7 +228,7 @@ static int visit_func_call_insn(int t, struct bpf_insn *insns,\n \t\tmerge_callee_effects(env, t, w);\n \t\tret = push_insn(t, w, BRANCH, env);\n \t}\n-\treturn ret;\n+\treturn merge_visit_ret(pad_ret, ret);\n }\n \n struct bpf_iarray *bpf_iarray_realloc(struct bpf_iarray *old, size_t n_elem)\n@@ -592,6 +640,7 @@ int bpf_check_cfg(struct bpf_verifier_env *env)\n \tint insn_cnt = env-\u003eprog-\u003elen;\n \tint *insn_stack, *insn_state;\n \tint ex_insn_beg, i, ret = 0;\n+\tu32 pad_idx = 0;\n \n \tinsn_state = env-\u003ecfg.insn_state = kvzalloc_objs(int, insn_cnt,\n \t\t\t\t\t\t\t GFP_KERNEL_ACCOUNT);\n@@ -647,6 +696,22 @@ int bpf_check_cfg(struct bpf_verifier_env *env)\n \t\tgoto walk_cfg;\n \t}\n \n+\t/*\n+\t * A landing pad no call site was marked with -- a record whose range\n+\t * holds no call an exception can unwind out of -- is reached by\n+\t * nothing. Walk it from here, and let the dead code sweep remove it.\n+\t */\n+\twhile (pad_idx \u003c env-\u003ecleanup_info_cnt) {\n+\t\tu32 pad = env-\u003ecleanup_info[pad_idx++].landing_pad_off;\n+\n+\t\tif (insn_state[pad] != EXPLORED) {\n+\t\t\tinsn_state[pad] = DISCOVERED;\n+\t\t\tinsn_stack[0] = pad;\n+\t\t\tenv-\u003ecfg.cur_stack = 1;\n+\t\t\tgoto walk_cfg;\n+\t\t}\n+\t}\n+\n \tfor (i = 0; i \u003c insn_cnt; i++) {\n \t\tstruct bpf_insn *insn = \u0026env-\u003eprog-\u003einsnsi[i];\n \ndiff --git a/kernel/bpf/check_btf.c b/kernel/bpf/check_btf.c\nindex 0e8b3ccc7a5b9..d03dc791042a2 100644\n--- a/kernel/bpf/check_btf.c\n+++ b/kernel/bpf/check_btf.c\n@@ -407,6 +407,149 @@ static int check_core_relo(struct bpf_verifier_env *env,\n \treturn err;\n }\n \n+static int cleanup_insn_subprog(struct bpf_verifier_env *env, u32 off)\n+{\n+\tstruct bpf_subprog_info *info;\n+\n+\tif (off \u003e= env-\u003eprog-\u003elen)\n+\t\treturn -1;\n+\tinfo = bpf_find_containing_subprog(env, off);\n+\treturn info ? info - env-\u003esubprog_info : -1;\n+}\n+\n+#define MIN_BPF_CLEANUP_INFO_SIZE\t12\n+#define MAX_CLEANUP_INFO_REC_SIZE\tMAX_FUNCINFO_REC_SIZE\n+\n+static int check_cleanup_info(struct bpf_verifier_env *env,\n+\t\t\t const union bpf_attr *attr,\n+\t\t\t bpfptr_t uattr)\n+{\n+\tu32 krec_size = sizeof(struct bpf_cleanup_info);\n+\tu32 i, nrec, urec_size, min_size, prev_end = 0;\n+\tstruct bpf_cleanup_info *krecord;\n+\tbpfptr_t urecord;\n+\tint ret = -EINVAL;\n+\n+\tnrec = attr-\u003ecleanup_info_cnt;\n+\tif (!nrec)\n+\t\treturn 0;\n+\tif (nrec \u003e INT_MAX / krec_size)\n+\t\treturn -EINVAL;\n+\n+\turec_size = attr-\u003ecleanup_info_rec_size;\n+\tif (urec_size \u003c MIN_BPF_CLEANUP_INFO_SIZE ||\n+\t urec_size \u003e MAX_CLEANUP_INFO_REC_SIZE ||\n+\t urec_size % sizeof(u32)) {\n+\t\tverbose(env, \"invalid cleanup info rec size %u\\n\", urec_size);\n+\t\treturn -EINVAL;\n+\t}\n+\n+\tkrecord = kvcalloc(nrec, krec_size, GFP_KERNEL_ACCOUNT | __GFP_NOWARN);\n+\tif (!krecord)\n+\t\treturn -ENOMEM;\n+\n+\tmin_size = min_t(u32, krec_size, urec_size);\n+\turecord = make_bpfptr(attr-\u003ecleanup_info, uattr.is_kernel);\n+\tfor (i = 0; i \u003c nrec; i++) {\n+\t\tstruct bpf_cleanup_info *rec = \u0026krecord[i];\n+\t\tint sb, se, sl;\n+\n+\t\tret = bpf_check_uarg_tail_zero(urecord, krec_size, urec_size);\n+\t\tif (ret) {\n+\t\t\tif (ret == -E2BIG) {\n+\t\t\t\tverbose(env, \"nonzero tailing record in cleanup info\\n\");\n+\t\t\t\tif (copy_to_bpfptr_offset(uattr,\n+\t\t\t\t\t\t\t offsetof(union bpf_attr,\n+\t\t\t\t\t\t\t\t cleanup_info_rec_size),\n+\t\t\t\t\t\t\t \u0026min_size, sizeof(min_size)))\n+\t\t\t\t\tret = -EFAULT;\n+\t\t\t}\n+\t\t\tgoto err_free;\n+\t\t}\n+\n+\t\tif (copy_from_bpfptr(rec, urecord, min_size)) {\n+\t\t\tret = -EFAULT;\n+\t\t\tgoto err_free;\n+\t\t}\n+\t\tbpfptr_add(\u0026urecord, urec_size);\n+\n+\t\tret = -EINVAL;\n+\t\tif (rec-\u003ebegin_off \u003e= rec-\u003eend_off) {\n+\t\t\tverbose(env, \"cleanup_info[%u]: begin %u \u003e= end %u\\n\",\n+\t\t\t\ti, rec-\u003ebegin_off, rec-\u003eend_off);\n+\t\t\tgoto err_free;\n+\t\t}\n+\t\tif (i \u0026\u0026 rec-\u003ebegin_off \u003c prev_end) {\n+\t\t\tverbose(env,\n+\t\t\t\t\"cleanup_info[%u]: range [%u,%u) is unsorted or overlaps the previous record\\n\",\n+\t\t\t\ti, rec-\u003ebegin_off, rec-\u003eend_off);\n+\t\t\tgoto err_free;\n+\t\t}\n+\t\tprev_end = rec-\u003eend_off;\n+\n+\t\tsb = cleanup_insn_subprog(env, rec-\u003ebegin_off);\n+\t\tse = cleanup_insn_subprog(env, rec-\u003eend_off - 1);\n+\t\tsl = cleanup_insn_subprog(env, rec-\u003elanding_pad_off);\n+\t\tif (sb \u003c 0 || se \u003c 0 || sl \u003c 0) {\n+\t\t\tverbose(env, \"cleanup_info[%u]: offset out of range\\n\", i);\n+\t\t\tgoto err_free;\n+\t\t}\n+\t\tif (sb != se || sb != sl) {\n+\t\t\tverbose(env,\n+\t\t\t\t\"cleanup_info[%u]: range/landing pad span multiple subprogs\\n\",\n+\t\t\t\ti);\n+\t\t\tgoto err_free;\n+\t\t}\n+\t\t/*\n+\t\t * The second half of a 16-byte instruction carries a zero\n+\t\t * opcode and is not an instruction of its own, so no offset\n+\t\t * may name one. end_off is exclusive, so it may also be one\n+\t\t * past the last instruction of the program.\n+\t\t */\n+\t\tif (!env-\u003eprog-\u003einsnsi[rec-\u003ebegin_off].code ||\n+\t\t !env-\u003eprog-\u003einsnsi[rec-\u003elanding_pad_off].code ||\n+\t\t (rec-\u003eend_off \u003c env-\u003eprog-\u003elen \u0026\u0026\n+\t\t !env-\u003eprog-\u003einsnsi[rec-\u003eend_off].code)) {\n+\t\t\tverbose(env, \"cleanup_info[%u]: points at invalid insn\\n\", i);\n+\t\t\tgoto err_free;\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * Reject a landing pad that lies inside a call-site range, its own\n+\t * included: it would be both a pad and a call that unwinds to one, and\n+\t * an exception out of it would have nowhere to go.\n+\t */\n+\tret = -EINVAL;\n+\tfor (i = 0; i \u003c nrec; i++) {\n+\t\tu32 pad = krecord[i].landing_pad_off;\n+\t\tu32 l = 0, r = nrec;\n+\n+\t\twhile (l \u003c r) {\n+\t\t\tu32 m = l + (r - l) / 2;\n+\n+\t\t\tif (pad \u003c krecord[m].begin_off) {\n+\t\t\t\tr = m;\n+\t\t\t} else if (pad \u003e= krecord[m].end_off) {\n+\t\t\t\tl = m + 1;\n+\t\t\t} else {\n+\t\t\t\tverbose(env,\n+\t\t\t\t\t\"cleanup_info[%u]: landing pad %u is inside the call-site range of cleanup_info[%u]\\n\",\n+\t\t\t\t\ti, pad, m);\n+\t\t\t\tgoto err_free;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tenv-\u003ecleanup_info = krecord;\n+\tenv-\u003ecleanup_info_cnt = nrec;\n+\treturn 0;\n+\n+err_free:\n+\tkvfree(krecord);\n+\treturn ret;\n+}\n+\n int bpf_prepare_btf_info(struct bpf_verifier_env *env,\n \t\t\t const union bpf_attr *attr,\n \t\t\t bpfptr_t uattr)\n@@ -441,6 +584,10 @@ int bpf_check_btf_info(struct bpf_verifier_env *env,\n {\n \tint err;\n \n+\terr = check_cleanup_info(env, attr, uattr);\n+\tif (err)\n+\t\treturn err;\n+\n \tif (!attr-\u003efunc_info_cnt \u0026\u0026 !attr-\u003eline_info_cnt) {\n \t\tif (check_abnormal_return(env))\n \t\t\treturn -EINVAL;\ndiff --git a/kernel/bpf/core.c b/kernel/bpf/core.c\nindex 227211166dccf..f19712be12a7d 100644\n--- a/kernel/bpf/core.c\n+++ b/kernel/bpf/core.c\n@@ -292,6 +292,7 @@ void __bpf_prog_free(struct bpf_prog *fp)\n \t\tmutex_destroy(\u0026fp-\u003eaux-\u003edst_mutex);\n \t\tmutex_destroy(\u0026fp-\u003eaux-\u003est_ops_assoc_mutex);\n \t\tkfree(fp-\u003eaux-\u003epoke_tab);\n+\t\tbpf_cleanup_free_info(fp-\u003eaux);\n \t\tkfree(fp-\u003eaux);\n \t}\n \tfree_percpu(fp-\u003estats);\n@@ -2625,13 +2626,21 @@ static bool bpf_prog_select_interpreter(struct bpf_prog *fp)\n \treturn select_interpreter;\n }\n \n-static struct bpf_prog *bpf_prog_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog)\n+static struct bpf_prog *bpf_prog_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog,\n+\t\t\t\t\t int *err)\n {\n #ifdef CONFIG_BPF_JIT\n \tstruct bpf_prog *orig_prog;\n+\tint ret;\n \n-\tif (!bpf_prog_need_blind(prog))\n+\tif (!bpf_prog_need_blind(prog)) {\n+\t\tret = bpf_cleanup_attach_main_prog(env, prog);\n+\t\tif (ret) {\n+\t\t\t*err = ret;\n+\t\t\treturn prog;\n+\t\t}\n \t\treturn bpf_int_jit_compile(env, prog);\n+\t}\n \n \torig_prog = prog;\n \tprog = bpf_jit_blind_constants(env, prog);\n@@ -2642,6 +2651,13 @@ static struct bpf_prog *bpf_prog_jit_compile(struct bpf_verifier_env *env, struc\n \tif (IS_ERR(prog))\n \t\tgoto out_restore;\n \n+\tret = bpf_cleanup_attach_main_prog(env, prog);\n+\tif (ret) {\n+\t\t*err = ret;\n+\t\tbpf_jit_prog_release_other(orig_prog, prog);\n+\t\tgoto out_restore;\n+\t}\n+\n \tprog = bpf_int_jit_compile(env, prog);\n \tif (prog-\u003ejited) {\n \t\tbpf_jit_prog_release_other(prog, orig_prog);\n@@ -2681,8 +2697,10 @@ struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct\n \t\tif (*err)\n \t\t\treturn fp;\n \n-\t\tfp = bpf_prog_jit_compile(env, fp);\n+\t\tfp = bpf_prog_jit_compile(env, fp, err);\n \t\tbpf_prog_jit_attempt_done(fp);\n+\t\tif (*err)\n+\t\t\treturn fp;\n \t\tif (!fp-\u003ejited \u0026\u0026 jit_needed) {\n \t\t\t*err = -ENOTSUPP;\n \t\t\treturn fp;\n@@ -3475,6 +3493,17 @@ void __weak arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp,\n {\n }\n \n+bool __weak bpf_jit_supports_cleanup_pads(void)\n+{\n+\treturn false;\n+}\n+\n+/* Call @pad with the frame pointer @frame_fp and r6-r9 spilled at @spill_base. */\n+void __weak arch_bpf_run_cleanup_pad(u64 pad, u64 frame_fp, u64 spill_base)\n+{\n+\tWARN_ON_ONCE(1);\n+}\n+\n bool __weak bpf_jit_supports_timed_may_goto(void)\n {\n \treturn false;\ndiff --git a/kernel/bpf/exception.c b/kernel/bpf/exception.c\nnew file mode 100644\nindex 0000000000000..d4dcbcf4fb257\n--- /dev/null\n+++ b/kernel/bpf/exception.c\n@@ -0,0 +1,664 @@\n+// SPDX-License-Identifier: GPL-2.0-only\n+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */\n+#include \u003clinux/bitmap.h\u003e\n+#include \u003clinux/bpf.h\u003e\n+#include \u003clinux/bpf_verifier.h\u003e\n+#include \u003clinux/bsearch.h\u003e\n+#include \u003clinux/btf.h\u003e\n+#include \u003clinux/btf_ids.h\u003e\n+#include \u003clinux/filter.h\u003e\n+#include \u003clinux/slab.h\u003e\n+#include \u003clinux/sort.h\u003e\n+#include \"exception.h\"\n+\n+#define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)\n+\n+enum exc_kfunc {\n+\tEXC_KF_bpf_unwind_resume,\n+};\n+\n+BTF_ID_LIST(exc_kfunc_list)\n+BTF_ID(func, bpf_unwind_resume)\n+\n+static bool insn_is_exc_kfunc(const struct bpf_insn *insn, int kf)\n+{\n+\treturn bpf_pseudo_kfunc_call(insn) \u0026\u0026 insn-\u003eoff == 0 \u0026\u0026\n+\t insn-\u003eimm == exc_kfunc_list[kf];\n+}\n+\n+/* What an instruction does to intra-subprog control flow. */\n+enum cleanup_insn_kind {\n+\tCLEANUP_INSN_PLAIN,\t/* the next insn runs */\n+\tCLEANUP_INSN_JUMP,\t/* unconditional jump */\n+\tCLEANUP_INSN_COND,\t/* the next insn runs, or the branch target */\n+\tCLEANUP_INSN_EXIT,\n+\tCLEANUP_INSN_THROW,\t/* call bpf_throw: nothing after it runs */\n+\tCLEANUP_INSN_RESUME,\t/* call bpf_unwind_resume: likewise */\n+\tCLEANUP_INSN_CALL,\t/* call to another subprog */\n+\tCLEANUP_INSN_GOTOX,\t/* indirect jump: successors not known here */\n+};\n+\n+/* What each instruction can reach, computed once by cleanup_reachability(). */\n+#define CLEANUP_REACH_RESUME\tBIT(0)\t/* a bpf_unwind_resume() call */\n+#define CLEANUP_REACH_EXIT\tBIT(1)\t/* a plain BPF_EXIT */\n+#define CLEANUP_REACH_UNKNOWN\tBIT(2)\t/* an indirect jump */\n+#define CLEANUP_REACH_THROW\tBIT(3)\t/* a bpf_throw() call */\n+\n+/* Scratch shared by the analyses, sized once so no walker has to allocate. */\n+struct cleanup_ctx {\n+\tstruct bpf_verifier_env *env;\n+\tu8 *reach;\t\t/* per insn: CLEANUP_REACH_* mask */\n+\tu32 *stack;\t\t/* per insn: DFS stack */\n+\tvoid *scratch;\t\t/* the one allocation all of the above live in */\n+};\n+\n+static bool in_pad(struct bpf_verifier_env *env, u32 i)\n+{\n+\treturn env-\u003einsn_aux_data[i].in_cleanup_pad;\n+}\n+\n+/* One scratch array for cleanup_alloc() to hand out. */\n+struct cleanup_alloc_req {\n+\tvoid **dst;\n+\tsize_t n, sz;\n+};\n+\n+static void *cleanup_alloc(const struct cleanup_alloc_req *tab, u32 cnt)\n+{\n+\tsize_t total = 0;\n+\tchar *block, *p;\n+\tu32 i;\n+\n+\tfor (i = 0; i \u003c cnt; i++)\n+\t\ttotal += round_up(tab[i].n * tab[i].sz, 8);\n+\n+\tblock = kvzalloc(total, GFP_KERNEL_ACCOUNT | __GFP_NOWARN);\n+\tif (!block)\n+\t\treturn NULL;\n+\n+\tfor (i = 0, p = block; i \u003c cnt; i++) {\n+\t\t*tab[i].dst = p;\n+\t\tp += round_up(tab[i].n * tab[i].sz, 8);\n+\t}\n+\treturn block;\n+}\n+\n+static int cleanup_subprog_of(struct bpf_verifier_env *env, u32 off)\n+{\n+\tstruct bpf_subprog_info *info = bpf_find_containing_subprog(env, off);\n+\n+\treturn info ? info - env-\u003esubprog_info : -1;\n+}\n+\n+/* The subprogram a linear pass is currently in. */\n+struct cleanup_cursor {\n+\tu32 start, end;\t\t/* [start, end) of the current subprogram */\n+\tint sub;\t\t/* its index */\n+};\n+\n+#define CLEANUP_CURSOR_INIT { .sub = -1 }\n+\n+static void cleanup_cursor_to(struct bpf_verifier_env *env, struct cleanup_cursor *c, u32 i)\n+{\n+\twhile (i \u003e= c-\u003eend) {\n+\t\tc-\u003esub++;\n+\t\tc-\u003estart = env-\u003esubprog_info[c-\u003esub].start;\n+\t\tc-\u003eend = env-\u003esubprog_info[c-\u003esub + 1].start;\n+\t}\n+}\n+\n+static enum cleanup_insn_kind cleanup_classify(struct bpf_verifier_env *env, u32 i,\n+\t\t\t\t\t int *next, int *target)\n+{\n+\tstruct bpf_insn *insn = \u0026env-\u003eprog-\u003einsnsi[i];\n+\tu8 class = BPF_CLASS(insn-\u003ecode);\n+\n+\t*next = i + 1;\n+\t*target = -1;\n+\n+\tif (insn-\u003ecode == (BPF_LD | BPF_IMM | BPF_DW)) {\n+\t\t*next = i + 2;\n+\t\treturn CLEANUP_INSN_PLAIN;\n+\t}\n+\tif (class != BPF_JMP \u0026\u0026 class != BPF_JMP32)\n+\t\treturn CLEANUP_INSN_PLAIN;\n+\n+\tswitch (BPF_OP(insn-\u003ecode)) {\n+\tcase BPF_EXIT:\n+\t\t*next = -1;\n+\t\treturn CLEANUP_INSN_EXIT;\n+\tcase BPF_JA:\n+\t\t*next = -1;\n+\t\tif (BPF_SRC(insn-\u003ecode) == BPF_X)\n+\t\t\treturn CLEANUP_INSN_GOTOX;\n+\t\t*target = class == BPF_JMP32 ? i + insn-\u003eimm + 1 : i + insn-\u003eoff + 1;\n+\t\treturn CLEANUP_INSN_JUMP;\n+\tcase BPF_CALL:\n+\t\tif (bpf_is_throw_kfunc(insn)) {\n+\t\t\t*next = -1;\n+\t\t\treturn CLEANUP_INSN_THROW;\n+\t\t}\n+\t\tif (insn_is_exc_kfunc(insn, EXC_KF_bpf_unwind_resume)) {\n+\t\t\t*next = -1;\n+\t\t\treturn CLEANUP_INSN_RESUME;\n+\t\t}\n+\t\treturn bpf_pseudo_call(insn) ? CLEANUP_INSN_CALL : CLEANUP_INSN_PLAIN;\n+\tdefault:\n+\t\t/* Conditional jump, including BPF_JCOND. */\n+\t\t*target = i + insn-\u003eoff + 1;\n+\t\treturn CLEANUP_INSN_COND;\n+\t}\n+}\n+\n+static void cleanup_mark_kfunc_sites(struct bpf_verifier_env *env)\n+{\n+\tu32 i;\n+\n+\tfor (i = 0; i \u003c env-\u003eprog-\u003elen; i++) {\n+\t\tstruct bpf_insn *insn = \u0026env-\u003eprog-\u003einsnsi[i];\n+\n+\t\tif (bpf_is_throw_kfunc(insn))\n+\t\t\tenv-\u003einsn_aux_data[i].cleanup_throw_site = true;\n+\t\telse if (insn_is_exc_kfunc(insn, EXC_KF_bpf_unwind_resume))\n+\t\t\tenv-\u003einsn_aux_data[i].cleanup_resume_site = true;\n+\t}\n+}\n+\n+int bpf_cleanup_check_callback(struct bpf_verifier_env *env, int subprog)\n+{\n+\tif (!env-\u003ecleanup_info_cnt || !env-\u003esubprog_info[subprog].might_throw)\n+\t\treturn 0;\n+\n+\tverbose(env, \"subprog %d may unwind and is used as a callback\\n\", subprog);\n+\treturn -EINVAL;\n+}\n+\n+/* Intra-subprog successors of @i, or -1 each when absent. */\n+static enum cleanup_insn_kind cleanup_succ(struct bpf_verifier_env *env, u32 i,\n+\t\t\t\t\t u32 start, u32 end, int *next, int *target)\n+{\n+\tenum cleanup_insn_kind kind = cleanup_classify(env, i, next, target);\n+\n+\tif (*next \u003c (int)start || *next \u003e= (int)end)\n+\t\t*next = -1;\n+\tif (*target \u003c (int)start || *target \u003e= (int)end)\n+\t\t*target = -1;\n+\treturn kind;\n+}\n+\n+static void cleanup_add_pred(u32 *head, u32 *link, u32 to, u32 e)\n+{\n+\tlink[e] = head[to];\n+\thead[to] = e + 1;\n+}\n+\n+/* What every instruction can reach along intra-subprog edges, for\n+ * cleanup_pad_is_catch(). One backward walk over a predecessor index, rather\n+ * than a forward walk from each landing pad, which would be quadratic.\n+ */\n+static int cleanup_reachability(struct cleanup_ctx *ctx)\n+{\n+\tstruct bpf_verifier_env *env = ctx-\u003eenv;\n+\tu32 len = env-\u003eprog-\u003elen;\n+\tstruct cleanup_cursor c = CLEANUP_CURSOR_INIT;\n+\tu32 *head = NULL, *link = NULL;\n+\tbool *queued = NULL;\n+\tu32 i, sp = 0;\n+\tvoid *scratch;\n+\tconst struct cleanup_alloc_req tab[] = {\n+\t\t{ (void **)\u0026head, len, sizeof(*head) },\n+\t\t{ (void **)\u0026link, 2 * (size_t)len, sizeof(*link) },\n+\t\t{ (void **)\u0026queued, len, sizeof(*queued) },\n+\t};\n+\n+\tscratch = cleanup_alloc(tab, ARRAY_SIZE(tab));\n+\tif (!scratch)\n+\t\treturn -ENOMEM;\n+\n+\t/* Index the predecessors, and seed the walk at the terminators. */\n+\tfor (i = 0; i \u003c len; i++) {\n+\t\tenum cleanup_insn_kind kind;\n+\t\tint next, target;\n+\n+\t\tcleanup_cursor_to(env, \u0026c, i);\n+\t\tkind = cleanup_succ(env, i, c.start, c.end, \u0026next, \u0026target);\n+\n+\t\tif (kind == CLEANUP_INSN_RESUME)\n+\t\t\tctx-\u003ereach[i] |= CLEANUP_REACH_RESUME;\n+\t\telse if (kind == CLEANUP_INSN_EXIT)\n+\t\t\tctx-\u003ereach[i] |= CLEANUP_REACH_EXIT;\n+\t\telse if (kind == CLEANUP_INSN_GOTOX)\n+\t\t\tctx-\u003ereach[i] |= CLEANUP_REACH_UNKNOWN;\n+\t\telse if (kind == CLEANUP_INSN_THROW)\n+\t\t\tctx-\u003ereach[i] |= CLEANUP_REACH_THROW;\n+\n+\t\tif (next \u003e= 0)\n+\t\t\tcleanup_add_pred(head, link, next, 2 * i);\n+\t\tif (target \u003e= 0)\n+\t\t\tcleanup_add_pred(head, link, target, 2 * i + 1);\n+\n+\t\tif (ctx-\u003ereach[i]) {\n+\t\t\tqueued[i] = true;\n+\t\t\tctx-\u003estack[sp++] = i;\n+\t\t}\n+\t}\n+\n+\t/* Each instruction re-enters the worklist at most once per bit it\n+\t * gains, so this is linear in the number of edges.\n+\t */\n+\twhile (sp) {\n+\t\tu32 j = ctx-\u003estack[--sp];\n+\t\tu8 flags = ctx-\u003ereach[j];\n+\t\tu32 e;\n+\n+\t\tqueued[j] = false;\n+\t\tfor (e = head[j]; e; e = link[e - 1]) {\n+\t\t\tu32 p = (e - 1) / 2;\n+\n+\t\t\tif ((ctx-\u003ereach[p] | flags) == ctx-\u003ereach[p])\n+\t\t\t\tcontinue;\n+\t\t\tctx-\u003ereach[p] |= flags;\n+\t\t\tif (!queued[p]) {\n+\t\t\t\tqueued[p] = true;\n+\t\t\t\tctx-\u003estack[sp++] = p;\n+\t\t\t}\n+\t\t}\n+\t}\n+\tkvfree(scratch);\n+\treturn 0;\n+}\n+\n+static int cleanup_pad_is_catch(struct cleanup_ctx *ctx, u32 pad)\n+{\n+\tu8 reach = ctx-\u003ereach[pad];\n+\n+\tif (reach \u0026 CLEANUP_REACH_UNKNOWN) {\n+\t\tverbose(ctx-\u003eenv, \"cleanup landing pad %u reaches an indirect jump\\n\", pad);\n+\t\treturn -EINVAL;\n+\t}\n+\tif (reach \u0026 CLEANUP_REACH_THROW) {\n+\t\tverbose(ctx-\u003eenv,\n+\t\t\t\"cleanup landing pad %u can throw while an exception is in flight\\n\",\n+\t\t\tpad);\n+\t\treturn -EINVAL;\n+\t}\n+\tif (!(reach \u0026 CLEANUP_REACH_RESUME) == !(reach \u0026 CLEANUP_REACH_EXIT)) {\n+\t\tverbose(ctx-\u003eenv, \"cleanup landing pad %u %s\\n\", pad,\n+\t\t\t(reach \u0026 CLEANUP_REACH_RESUME) ?\n+\t\t\t\"reaches both bpf_unwind_resume() and a plain exit\" :\n+\t\t\t\"reaches neither bpf_unwind_resume() nor an exit\");\n+\t\treturn -EINVAL;\n+\t}\n+\treturn !!(reach \u0026 CLEANUP_REACH_EXIT);\n+}\n+\n+static int cleanup_check_pad_insn(struct bpf_verifier_env *env, u32 i)\n+{\n+\tstruct bpf_insn *insn = \u0026env-\u003eprog-\u003einsnsi[i];\n+\n+\tif (bpf_helper_call(insn) \u0026\u0026 insn-\u003eimm == BPF_FUNC_tail_call) {\n+\t\tverbose(env,\n+\t\t\t\"bpf_tail_call() at insn %u is in an exception cleanup landing pad\\n\",\n+\t\t\ti);\n+\t\treturn -EINVAL;\n+\t}\n+\t/* A failed BPF_LD_[ABS|IND] leaves the frame through its epilogue. */\n+\tif (BPF_CLASS(insn-\u003ecode) == BPF_LD \u0026\u0026\n+\t (BPF_MODE(insn-\u003ecode) == BPF_ABS || BPF_MODE(insn-\u003ecode) == BPF_IND)) {\n+\t\tverbose(env,\n+\t\t\t\"BPF_LD_[ABS|IND] at insn %u is in an exception cleanup landing pad\\n\",\n+\t\t\ti);\n+\t\treturn -EINVAL;\n+\t}\n+\t/* Stack arguments are not supported. */\n+\tif (is_stack_arg_st(insn) || is_stack_arg_stx(insn)) {\n+\t\tverbose(env,\n+\t\t\t\"insn %u passes an on-stack call argument in an exception cleanup landing pad\\n\",\n+\t\t\ti);\n+\t\treturn -EINVAL;\n+\t}\n+\t/* Likewise, stack arguments are not supported. */\n+\tif (bpf_pseudo_kfunc_call(insn)) {\n+\t\tstruct bpf_call_summary cs;\n+\n+\t\tif (bpf_get_call_summary(env, insn, \u0026cs) \u0026\u0026\n+\t\t cs.arg_slot_cnt \u003e MAX_BPF_FUNC_REG_ARGS) {\n+\t\t\tverbose(env,\n+\t\t\t\t\"insn %u passes an on-stack call argument in an exception cleanup landing pad\\n\",\n+\t\t\t\ti);\n+\t\t\treturn -EINVAL;\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n+static int cleanup_mark_pad_bodies(struct cleanup_ctx *ctx)\n+{\n+\tstruct bpf_verifier_env *env = ctx-\u003eenv;\n+\tu32 i, sp = 0;\n+\tint ret;\n+\n+\tfor (i = 0; i \u003c env-\u003ecleanup_info_cnt; i++) {\n+\t\tu32 pad = env-\u003ecleanup_info[i].landing_pad_off;\n+\n+\t\tif (in_pad(env, pad))\n+\t\t\tcontinue;\n+\n+\t\tret = cleanup_pad_is_catch(ctx, pad);\n+\t\tif (ret \u003c 0)\n+\t\t\treturn ret;\n+\t\tif (ret) {\n+\t\t\tverbose(env,\n+\t\t\t\t\"catch landing pad %u is not supported yet, only cleanup pads that resume\\n\",\n+\t\t\t\tpad);\n+\t\t\treturn -EOPNOTSUPP;\n+\t\t}\n+\t\tenv-\u003einsn_aux_data[pad].in_cleanup_pad = true;\n+\t\tctx-\u003estack[sp++] = pad;\n+\t}\n+\n+\twhile (sp) {\n+\t\tu32 j = ctx-\u003estack[--sp];\n+\t\tenum cleanup_insn_kind kind;\n+\t\tint next, target, sub;\n+\t\tu32 start, end;\n+\n+\t\tret = cleanup_check_pad_insn(env, j);\n+\t\tif (ret)\n+\t\t\treturn ret;\n+\n+\t\tsub = cleanup_subprog_of(env, j);\n+\t\tstart = env-\u003esubprog_info[sub].start;\n+\t\tend = env-\u003esubprog_info[sub + 1].start;\n+\t\tkind = cleanup_succ(env, j, start, end, \u0026next, \u0026target);\n+\n+\t\tif (kind == CLEANUP_INSN_CALL) {\n+\t\t\tint callee = cleanup_subprog_of(env, j + env-\u003eprog-\u003einsnsi[j].imm + 1);\n+\n+\t\t\tif (env-\u003esubprog_info[callee].might_throw) {\n+\t\t\t\tverbose(env,\n+\t\t\t\t\t\"cleanup landing pad calls subprog %d at insn %u, which can throw while an exception is in flight\\n\",\n+\t\t\t\t\tcallee, j);\n+\t\t\t\treturn -EINVAL;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (next \u003e= 0 \u0026\u0026 !in_pad(env, next)) {\n+\t\t\tenv-\u003einsn_aux_data[next].in_cleanup_pad = true;\n+\t\t\tctx-\u003estack[sp++] = next;\n+\t\t}\n+\t\tif (target \u003e= 0 \u0026\u0026 !in_pad(env, target)) {\n+\t\t\tenv-\u003einsn_aux_data[target].in_cleanup_pad = true;\n+\t\t\tctx-\u003estack[sp++] = target;\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n+static int cleanup_check_resumes(struct cleanup_ctx *ctx)\n+{\n+\tstruct bpf_verifier_env *env = ctx-\u003eenv;\n+\tu32 i;\n+\n+\tfor (i = 0; i \u003c env-\u003eprog-\u003elen; i++) {\n+\t\tif (!insn_is_exc_kfunc(\u0026env-\u003eprog-\u003einsnsi[i], EXC_KF_bpf_unwind_resume))\n+\t\t\tcontinue;\n+\t\tif (in_pad(env, i))\n+\t\t\tcontinue;\n+\t\tverbose(env,\n+\t\t\t\"bpf_unwind_resume() at insn %u is not in an exception cleanup landing pad\\n\",\n+\t\t\ti);\n+\t\treturn -EINVAL;\n+\t}\n+\treturn 0;\n+}\n+\n+static void cleanup_mark_call_sites(struct bpf_verifier_env *env)\n+{\n+\tu32 i, j;\n+\n+\tfor (i = 0; i \u003c env-\u003ecleanup_info_cnt; i++) {\n+\t\tstruct bpf_cleanup_info *rec = \u0026env-\u003ecleanup_info[i];\n+\n+\t\tfor (j = rec-\u003ebegin_off; j \u003c rec-\u003eend_off; j++) {\n+\t\t\tstruct bpf_insn *insn = \u0026env-\u003eprog-\u003einsnsi[j];\n+\n+\t\t\tif (!bpf_pseudo_call(insn) \u0026\u0026 !bpf_is_throw_kfunc(insn))\n+\t\t\t\tcontinue;\n+\t\t\tenv-\u003einsn_aux_data[j].cleanup_pad = rec-\u003elanding_pad_off + 1;\n+\t\t}\n+\t}\n+}\n+\n+int bpf_prepare_cleanup_exceptions(struct bpf_verifier_env *env)\n+{\n+\tif (!env-\u003ecleanup_info_cnt)\n+\t\treturn 0;\n+\n+\tif (bpf_prog_is_offloaded(env-\u003eprog-\u003eaux)) {\n+\t\tverbose(env,\n+\t\t\t\"exception cleanup is not supported for offloaded programs\\n\");\n+\t\treturn -EINVAL;\n+\t}\n+\n+\tif (!bpf_jit_supports_cleanup_pads() || !env-\u003eprog-\u003ejit_requested) {\n+\t\tverbose(env,\n+\t\t\t\"exception cleanup needs a JIT that can dispatch landing pads\\n\");\n+\t\treturn -EOPNOTSUPP;\n+\t}\n+\tenv-\u003eprog-\u003ejit_required = 1;\n+\n+\tif (env-\u003eexception_callback_subprog) {\n+\t\tverbose(env,\n+\t\t\t\"exception cleanup table cannot be combined with an exception callback\\n\");\n+\t\treturn -EINVAL;\n+\t}\n+\n+\tcleanup_mark_kfunc_sites(env);\n+\tcleanup_mark_call_sites(env);\n+\treturn 0;\n+}\n+\n+int bpf_check_cleanup_exceptions(struct bpf_verifier_env *env)\n+{\n+\tu32 len = env-\u003eprog-\u003elen;\n+\tstruct cleanup_ctx ctx = { .env = env };\n+\tconst struct cleanup_alloc_req tab[] = {\n+\t\t{ (void **)\u0026ctx.reach, len, sizeof(*ctx.reach) },\n+\t\t{ (void **)\u0026ctx.stack, len, sizeof(*ctx.stack) },\n+\t};\n+\tint ret;\n+\n+\tif (!env-\u003ecleanup_info_cnt)\n+\t\treturn 0;\n+\n+\tctx.scratch = cleanup_alloc(tab, ARRAY_SIZE(tab));\n+\tif (!ctx.scratch)\n+\t\treturn -ENOMEM;\n+\n+\tret = cleanup_reachability(\u0026ctx);\n+\tif (ret)\n+\t\tgoto out;\n+\n+\tret = cleanup_mark_pad_bodies(\u0026ctx);\n+\tif (ret)\n+\t\tgoto out;\n+\n+\tret = cleanup_check_resumes(\u0026ctx);\n+out:\n+\tkvfree(ctx.scratch);\n+\treturn ret;\n+}\n+\n+bool bpf_is_unwind_resume_kfunc(const struct bpf_insn *insn)\n+{\n+\treturn insn_is_exc_kfunc(insn, EXC_KF_bpf_unwind_resume);\n+}\n+\n+int bpf_cleanup_pad_of_call(struct bpf_verifier_env *env, u32 idx)\n+{\n+\tu32 pad = env-\u003einsn_aux_data[idx].cleanup_pad;\n+\n+\treturn pad ? (int)pad - 1 : -1;\n+}\n+\n+/*\n+ * Every subprogram of a cleanup-carrying program spills the BPF callee-saved\n+ * registers, even one that never throws: a frame's spill holds its caller's\n+ * registers, and that is what the walker restores before running the caller's\n+ * pad. The exception callback does not, because it reuses the boundary frame\n+ * rather than building one of its own.\n+ */\n+bool bpf_cleanup_force_spill(const struct bpf_prog *prog)\n+{\n+\treturn prog-\u003eaux-\u003eexc \u0026\u0026 !prog-\u003eaux-\u003eexception_cb;\n+}\n+\n+const struct bpf_cleanup_range *bpf_cleanup_pad_for_ip(const struct bpf_prog *prog, u64 ip)\n+{\n+\tconst struct bpf_exception_info *exc = prog-\u003eaux-\u003eexc;\n+\tu32 l = 0, r = exc ? exc-\u003enr_ranges : 0;\n+\n+\twhile (l \u003c r) {\n+\t\tu32 m = l + (r - l) / 2;\n+\t\tconst struct bpf_cleanup_range *rec = \u0026exc-\u003eranges[m];\n+\n+\t\tif (ip \u003c= rec-\u003ebegin)\n+\t\t\tr = m;\n+\t\telse if (ip \u003e rec-\u003eend)\n+\t\t\tl = m + 1;\n+\t\telse\n+\t\t\treturn rec;\n+\t}\n+\treturn NULL;\n+}\n+\n+static int cmp_u32(const void *a, const void *b)\n+{\n+\tu32 x = *(const u32 *)a, y = *(const u32 *)b;\n+\n+\treturn x \u003c y ? -1 : x \u003e y;\n+}\n+\n+int bpf_cleanup_alloc_info(struct bpf_prog_aux *aux)\n+{\n+\tif (aux-\u003eexc)\n+\t\treturn 0;\n+\taux-\u003eexc = kzalloc_obj(struct bpf_exception_info, GFP_KERNEL_ACCOUNT | __GFP_NOWARN);\n+\treturn aux-\u003eexc ? 0 : -ENOMEM;\n+}\n+\n+int bpf_cleanup_attach_info(struct bpf_prog_aux *aux, struct bpf_cleanup_info *recs, u32 cnt)\n+{\n+\tstruct bpf_exception_info *exc = aux-\u003eexc;\n+\tstruct bpf_cleanup_range *ranges;\n+\tu32 i, n_at, *at;\n+\n+\tif (!cnt) {\n+\t\tkvfree(recs);\n+\t\treturn 0;\n+\t}\n+\n+\tranges = kvcalloc(cnt, sizeof(*ranges), GFP_KERNEL_ACCOUNT | __GFP_NOWARN);\n+\tif (!ranges) {\n+\t\tkvfree(recs);\n+\t\treturn -ENOMEM;\n+\t}\n+\n+\t/* The pads on their own, sorted and deduplicated. */\n+\tat = kvmalloc_array(cnt, sizeof(*at), GFP_KERNEL_ACCOUNT | __GFP_NOWARN);\n+\tif (!at) {\n+\t\tkvfree(ranges);\n+\t\tkvfree(recs);\n+\t\treturn -ENOMEM;\n+\t}\n+\tfor (i = 0; i \u003c cnt; i++)\n+\t\tat[i] = recs[i].landing_pad_off;\n+\tsort(at, cnt, sizeof(*at), cmp_u32, NULL);\n+\tfor (i = 0, n_at = 0; i \u003c cnt; i++)\n+\t\tif (!n_at || at[n_at - 1] != at[i])\n+\t\t\tat[n_at++] = at[i];\n+\n+\texc-\u003epad_at = at;\n+\texc-\u003enr_pad_at = n_at;\n+\texc-\u003einfo = recs;\n+\texc-\u003enr_info = cnt;\n+\texc-\u003eranges = ranges;\n+\t/* Withheld until the JIT has filled the table in. */\n+\texc-\u003enr_ranges = 0;\n+\treturn 0;\n+}\n+\n+void bpf_cleanup_fill_native_ranges(struct bpf_prog *prog, u32 *addrs, void *image)\n+{\n+\tstruct bpf_exception_info *exc = prog-\u003eaux-\u003eexc;\n+\tu32 i, n;\n+\n+\tif (!exc || !exc-\u003enr_info || !exc-\u003eranges)\n+\t\treturn;\n+\n+\tn = exc-\u003enr_info;\n+\tfor (i = 0; i \u003c n; i++) {\n+\t\tconst struct bpf_cleanup_info *rec = \u0026exc-\u003einfo[i];\n+\n+\t\tif (WARN_ON_ONCE(rec-\u003ebegin_off \u003e= prog-\u003elen ||\n+\t\t\t\t rec-\u003eend_off \u003e prog-\u003elen ||\n+\t\t\t\t rec-\u003elanding_pad_off \u003e= prog-\u003elen))\n+\t\t\treturn;\n+\t\texc-\u003eranges[i].begin = (u64)(long)image + addrs[rec-\u003ebegin_off];\n+\t\texc-\u003eranges[i].end = (u64)(long)image + addrs[rec-\u003eend_off];\n+\t\texc-\u003eranges[i].pad = (u64)(long)image + addrs[rec-\u003elanding_pad_off];\n+\t}\n+\texc-\u003enr_ranges = n;\n+}\n+\n+void bpf_cleanup_free_info(struct bpf_prog_aux *aux)\n+{\n+\tstruct bpf_exception_info *exc = aux-\u003eexc;\n+\n+\tif (!exc)\n+\t\treturn;\n+\tkvfree(exc-\u003eranges);\n+\tkvfree(exc-\u003einfo);\n+\tkvfree(exc-\u003epad_at);\n+\tkvfree(exc-\u003ethrow_at);\n+\tkvfree(exc-\u003eresume_at);\n+\tbitmap_free(exc-\u003epad_body);\n+\tkfree(exc);\n+\taux-\u003eexc = NULL;\n+}\n+\n+/* Is @idx in the sorted array @at of @n instruction indices? */\n+static bool insn_idx_in(const u32 *at, u32 n, u32 idx)\n+{\n+\treturn bsearch(\u0026idx, at, n, sizeof(*at), cmp_u32);\n+}\n+\n+bool bpf_cleanup_insn_is_pad(const struct bpf_prog *prog, u32 idx)\n+{\n+\tconst struct bpf_exception_info *exc = prog-\u003eaux-\u003eexc;\n+\n+\treturn exc \u0026\u0026 insn_idx_in(exc-\u003epad_at, exc-\u003enr_pad_at, idx);\n+}\n+\n+bool bpf_cleanup_insn_is_throw(const struct bpf_prog *prog, u32 idx)\n+{\n+\tconst struct bpf_exception_info *exc = prog-\u003eaux-\u003eexc;\n+\n+\treturn exc \u0026\u0026 insn_idx_in(exc-\u003ethrow_at, exc-\u003enr_throw_at, idx);\n+}\n+\n+bool bpf_cleanup_insn_is_resume(const struct bpf_prog *prog, u32 idx)\n+{\n+\tconst struct bpf_exception_info *exc = prog-\u003eaux-\u003eexc;\n+\n+\treturn exc \u0026\u0026 insn_idx_in(exc-\u003eresume_at, exc-\u003enr_resume_at, idx);\n+}\n+\n+bool bpf_cleanup_insn_in_pad(const struct bpf_prog *prog, u32 idx)\n+{\n+\tconst struct bpf_exception_info *exc = prog-\u003eaux-\u003eexc;\n+\n+\treturn exc \u0026\u0026 exc-\u003epad_body \u0026\u0026 idx \u003c exc-\u003enr_pad_body \u0026\u0026\n+\t test_bit(idx, exc-\u003epad_body);\n+}\ndiff --git a/kernel/bpf/exception.h b/kernel/bpf/exception.h\nnew file mode 100644\nindex 0000000000000..c0e68ce227c83\n--- /dev/null\n+++ b/kernel/bpf/exception.h\n@@ -0,0 +1,22 @@\n+/* SPDX-License-Identifier: GPL-2.0-only */\n+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */\n+#ifndef _LINUX_BPF_EXCEPTION_H\n+#define _LINUX_BPF_EXCEPTION_H\n+\n+#include \u003clinux/types.h\u003e\n+\n+struct bpf_cleanup_info;\n+struct bpf_cleanup_range;\n+struct bpf_prog;\n+struct bpf_prog_aux;\n+struct bpf_verifier_env;\n+\n+int bpf_prepare_cleanup_exceptions(struct bpf_verifier_env *env);\n+int bpf_check_cleanup_exceptions(struct bpf_verifier_env *env);\n+int bpf_cleanup_check_callback(struct bpf_verifier_env *env, int subprog);\n+int bpf_cleanup_pad_of_call(struct bpf_verifier_env *env, u32 idx);\n+int bpf_cleanup_alloc_info(struct bpf_prog_aux *aux);\n+int bpf_cleanup_attach_info(struct bpf_prog_aux *aux, struct bpf_cleanup_info *recs, u32 cnt);\n+const struct bpf_cleanup_range *bpf_cleanup_pad_for_ip(const struct bpf_prog *prog, u64 ip);\n+\n+#endif /* _LINUX_BPF_EXCEPTION_H */\ndiff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c\nindex 2add8001c3ec3..169cbf9f44159 100644\n--- a/kernel/bpf/fixups.c\n+++ b/kernel/bpf/fixups.c\n@@ -1,5 +1,6 @@\n // SPDX-License-Identifier: GPL-2.0-only\n /* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */\n+#include \u003clinux/bitmap.h\u003e\n #include \u003clinux/bpf.h\u003e\n #include \u003clinux/btf.h\u003e\n #include \u003clinux/bpf_verifier.h\u003e\n@@ -10,6 +11,7 @@\n #include \u003clinux/perf_event.h\u003e\n #include \u003cnet/xdp.h\u003e\n #include \"disasm.h\"\n+#include \"exception.h\"\n \n #define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)\n \n@@ -252,15 +254,29 @@ static void adjust_insn_aux_data(struct bpf_verifier_env *env,\n \t\t/* Expand insni[off]'s seen count to the patched range. */\n \t\tdata[i].seen = old_seen;\n \t\tdata[i].zext_dst = bpf_insn_def32(new_prog, insn + i) \u003e= 0;\n+\t\tdata[i].in_cleanup_pad = data[off + cnt - 1].in_cleanup_pad;\n \t\tif (!memcmp(insn + i, original_insn, sizeof(struct bpf_insn))) {\n \t\t\tdata[i].non_stack_access =\n \t\t\t\tdata[off + cnt - 1].non_stack_access;\n \t\t\tdata[off + cnt - 1].non_stack_access = false;\n+\t\t\tdata[i].cleanup_throw_site =\n+\t\t\t\tdata[off + cnt - 1].cleanup_throw_site;\n+\t\t\tdata[off + cnt - 1].cleanup_throw_site = false;\n+\t\t\tdata[i].cleanup_resume_site =\n+\t\t\t\tdata[off + cnt - 1].cleanup_resume_site;\n+\t\t\tdata[off + cnt - 1].cleanup_resume_site = false;\n+\t\t\tdata[i].cleanup_pad = data[off + cnt - 1].cleanup_pad;\n+\t\t\tdata[off + cnt - 1].cleanup_pad = 0;\n \t\t} else if (bpf_is_mem_insn(insn + i)) {\n \t\t\tdata[i].non_stack_access = true;\n \t\t}\n \t}\n \n+\tif (env-\u003ecleanup_info_cnt)\n+\t\tfor (i = 0; i \u003c prog_len; i++)\n+\t\t\tif (data[i].cleanup_pad \u003e off + 1)\n+\t\t\t\tdata[i].cleanup_pad += cnt - 1;\n+\n \t/*\n \t * Last slot instruction could be a newly generated\n \t * BPF_ST/BPF_LDX/BPF_STX, systematically mark it for non-stack access\n@@ -544,11 +560,13 @@ void bpf_clear_insn_aux_data(struct bpf_verifier_env *env, int start, int len)\n \t}\n }\n \n-static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt)\n+static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt,\n+\t\t\t\t bool falls_through)\n {\n \tstruct bpf_insn_aux_data *aux_data = env-\u003einsn_aux_data;\n \tunsigned int orig_prog_len = env-\u003eprog-\u003elen;\n \tint err;\n+\tu32 i;\n \n \tif (bpf_prog_is_offloaded(env-\u003eprog-\u003eaux))\n \t\tbpf_prog_offload_remove_insns(env, off, cnt);\n@@ -573,6 +591,17 @@ static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt)\n \t\tsizeof(*aux_data) * (orig_prog_len - off - cnt));\n \tenv-\u003einsn_aux_data_len -= cnt;\n \n+\tif (env-\u003ecleanup_info_cnt) {\n+\t\tfor (i = 0; i \u003c env-\u003einsn_aux_data_len; i++) {\n+\t\t\tu32 pad = aux_data[i].cleanup_pad;\n+\n+\t\t\tif (pad \u003e off + cnt)\n+\t\t\t\taux_data[i].cleanup_pad = pad - cnt;\n+\t\t\telse if (pad \u003e off)\n+\t\t\t\taux_data[i].cleanup_pad = falls_through ? off + 1 : 0;\n+\t\t}\n+\t}\n+\n \treturn 0;\n }\n \n@@ -634,7 +663,7 @@ int bpf_opt_remove_dead_code(struct bpf_verifier_env *env)\n \t\tif (!j)\n \t\t\tcontinue;\n \n-\t\terr = verifier_remove_insns(env, i, j);\n+\t\terr = verifier_remove_insns(env, i, j, false);\n \t\tif (err)\n \t\t\treturn err;\n \t\tinsn_cnt = env-\u003eprog-\u003elen;\n@@ -657,7 +686,7 @@ int bpf_opt_remove_nops(struct bpf_verifier_env *env)\n \t\tif (!is_may_goto_0 \u0026\u0026 !is_ja)\n \t\t\tcontinue;\n \n-\t\terr = verifier_remove_insns(env, i, 1);\n+\t\terr = verifier_remove_insns(env, i, 1, true);\n \t\tif (err)\n \t\t\treturn err;\n \t\tinsn_cnt--;\n@@ -1095,6 +1124,129 @@ static void bpf_restore_subprog_starts(struct bpf_verifier_env *env, u32 *orig_s\n \tenv-\u003esubprog_info[env-\u003esubprog_cnt].start = env-\u003eprog-\u003elen;\n }\n \n+static bool cleanup_kfunc_site(const struct bpf_insn_aux_data *aux, bool resume)\n+{\n+\treturn resume ? aux-\u003ecleanup_resume_site : aux-\u003ecleanup_throw_site;\n+}\n+\n+static int cleanup_kfunc_sites_for_subprog(struct bpf_verifier_env *env, u32 start, u32 end,\n+\t\t\t\t\t bool resume, u32 **at_p, u32 *nr_p)\n+{\n+\tu32 i, cnt = 0, *at;\n+\n+\tfor (i = start; i \u003c end; i++)\n+\t\tif (cleanup_kfunc_site(\u0026env-\u003einsn_aux_data[i], resume))\n+\t\t\tcnt++;\n+\tif (!cnt)\n+\t\treturn 0;\n+\n+\tat = kvmalloc_array(cnt, sizeof(*at), GFP_KERNEL_ACCOUNT | __GFP_NOWARN);\n+\tif (!at)\n+\t\treturn -ENOMEM;\n+\n+\tfor (i = start, cnt = 0; i \u003c end; i++) {\n+\t\tif (!cleanup_kfunc_site(\u0026env-\u003einsn_aux_data[i], resume))\n+\t\t\tcontinue;\n+\t\tat[cnt++] = i - start;\n+\t}\n+\n+\t*at_p = at;\n+\t*nr_p = cnt;\n+\treturn 0;\n+}\n+\n+static int cleanup_pad_body_for_subprog(struct bpf_verifier_env *env, struct bpf_prog *sub,\n+\t\t\t\t\tu32 start, u32 end)\n+{\n+\tunsigned long *bits;\n+\tu32 i, cnt = 0;\n+\n+\tfor (i = start; i \u003c end; i++)\n+\t\tif (env-\u003einsn_aux_data[i].in_cleanup_pad)\n+\t\t\tcnt++;\n+\tif (!cnt)\n+\t\treturn 0;\n+\n+\tbits = bitmap_zalloc(end - start, GFP_KERNEL_ACCOUNT | __GFP_NOWARN);\n+\tif (!bits)\n+\t\treturn -ENOMEM;\n+\n+\tfor (i = start; i \u003c end; i++)\n+\t\tif (env-\u003einsn_aux_data[i].in_cleanup_pad)\n+\t\t\t__set_bit(i - start, bits);\n+\n+\tsub-\u003eaux-\u003eexc-\u003epad_body = bits;\n+\tsub-\u003eaux-\u003eexc-\u003enr_pad_body = end - start;\n+\treturn 0;\n+}\n+\n+static int cleanup_info_for_subprog(struct bpf_verifier_env *env, struct bpf_prog *sub,\n+\t\t\t\t u32 start, u32 end)\n+{\n+\tstruct bpf_cleanup_info *recs;\n+\tu32 i, cnt = 0;\n+\tint err;\n+\n+\tif (!env-\u003ecleanup_info_cnt)\n+\t\treturn 0;\n+\n+\terr = bpf_cleanup_alloc_info(sub-\u003eaux);\n+\tif (err)\n+\t\treturn err;\n+\n+\terr = cleanup_kfunc_sites_for_subprog(env, start, end, false,\n+\t\t\t\t\t \u0026sub-\u003eaux-\u003eexc-\u003ethrow_at,\n+\t\t\t\t\t \u0026sub-\u003eaux-\u003eexc-\u003enr_throw_at);\n+\tif (err)\n+\t\treturn err;\n+\n+\terr = cleanup_kfunc_sites_for_subprog(env, start, end, true,\n+\t\t\t\t\t \u0026sub-\u003eaux-\u003eexc-\u003eresume_at,\n+\t\t\t\t\t \u0026sub-\u003eaux-\u003eexc-\u003enr_resume_at);\n+\tif (err)\n+\t\treturn err;\n+\n+\terr = cleanup_pad_body_for_subprog(env, sub, start, end);\n+\tif (err)\n+\t\treturn err;\n+\n+\tfor (i = start; i \u003c end; i++)\n+\t\tif (env-\u003einsn_aux_data[i].cleanup_pad)\n+\t\t\tcnt++;\n+\tif (!cnt)\n+\t\treturn 0;\n+\n+\trecs = kvmalloc_array(cnt, sizeof(*recs), GFP_KERNEL_ACCOUNT | __GFP_NOWARN);\n+\tif (!recs)\n+\t\treturn -ENOMEM;\n+\n+\tfor (i = start, cnt = 0; i \u003c end; i++) {\n+\t\tu32 pad = env-\u003einsn_aux_data[i].cleanup_pad;\n+\n+\t\tif (!pad)\n+\t\t\tcontinue;\n+\t\tpad--;\n+\t\tif (verifier_bug_if(pad \u003c start || pad \u003e= end, env,\n+\t\t\t\t \"insn %u is covered by a landing pad at %u outside its subprog [%u, %u)\",\n+\t\t\t\t i, pad, start, end)) {\n+\t\t\tkvfree(recs);\n+\t\t\treturn -EFAULT;\n+\t\t}\n+\t\trecs[cnt].begin_off = i - start;\n+\t\trecs[cnt].end_off = i - start + 1;\n+\t\trecs[cnt].landing_pad_off = pad - start;\n+\t\tcnt++;\n+\t}\n+\treturn bpf_cleanup_attach_info(sub-\u003eaux, recs, cnt);\n+}\n+\n+int bpf_cleanup_attach_main_prog(struct bpf_verifier_env *env, struct bpf_prog *prog)\n+{\n+\tif (!env || env-\u003esubprog_cnt \u003e 1)\n+\t\treturn 0;\n+\treturn cleanup_info_for_subprog(env, prog, 0, prog-\u003elen);\n+}\n+\n static int jit_subprogs(struct bpf_verifier_env *env)\n {\n \tstruct bpf_prog *prog = env-\u003eprog, **func, *tmp;\n@@ -1232,6 +1384,10 @@ static int jit_subprogs(struct bpf_verifier_env *env)\n \t\tfunc[i]-\u003eaux-\u003etoken = prog-\u003eaux-\u003etoken;\n \t\tif (!i)\n \t\t\tfunc[i]-\u003eaux-\u003eexception_boundary = env-\u003eseen_exception;\n+\t\terr = cleanup_info_for_subprog(env, func[i], subprog_start,\n+\t\t\t\t\t subprog_end);\n+\t\tif (err)\n+\t\t\tgoto out_free;\n \t\tfunc[i] = bpf_int_jit_compile(env, func[i]);\n \t\tif (!func[i]-\u003ejited) {\n \t\t\terr = -ENOTSUPP;\n@@ -1336,6 +1492,8 @@ static int jit_subprogs(struct bpf_verifier_env *env)\n \tprog-\u003eaux-\u003ebpf_exception_cb = (void *)func[env-\u003eexception_callback_subprog]-\u003ebpf_func;\n \tprog-\u003eaux-\u003eexception_boundary = func[0]-\u003eaux-\u003eexception_boundary;\n \tprog-\u003eaux-\u003estack_arg_sp_adjust = func[0]-\u003eaux-\u003estack_arg_sp_adjust;\n+\tprog-\u003eaux-\u003eexc = func[0]-\u003eaux-\u003eexc;\n+\tfunc[0]-\u003eaux-\u003eexc = NULL;\n \tbpf_prog_jit_attempt_done(prog);\n \treturn 0;\n out_free:\n@@ -1916,6 +2074,8 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)\n \t\t\tgoto next_insn;\n \t\tif (insn-\u003esrc_reg == BPF_PSEUDO_CALL)\n \t\t\tgoto next_insn;\n+\t\tif (env-\u003einsn_aux_data[i + delta].cleanup_resume_site)\n+\t\t\tgoto next_insn;\n \t\tif (insn-\u003esrc_reg == BPF_PSEUDO_KFUNC_CALL) {\n \t\t\tret = bpf_fixup_kfunc_call(env, insn, insn_buf, i + delta, \u0026cnt);\n \t\t\tif (ret)\ndiff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c\nindex 82402d97ce67c..d639dfbc3cbbf 100644\n--- a/kernel/bpf/helpers.c\n+++ b/kernel/bpf/helpers.c\n@@ -31,6 +31,7 @@\n #include \u003clinux/buildid.h\u003e\n \n #include \"../../lib/kstrtox.h\"\n+#include \"exception.h\"\n \n /* If kernel subsystem is allowing eBPF programs to call this function,\n * inside its own verifier_ops-\u003eget_func_proto() callback it should return\n@@ -3398,8 +3399,36 @@ struct bpf_throw_ctx {\n \tu64 sp;\n \tu64 bp;\n \tint cnt;\n+\tconst struct bpf_prog *callee;\n+\tu64 callee_fp;\n };\n \n+static void bpf_run_cleanup_pad(struct bpf_throw_ctx *ctx, const struct bpf_prog *prog,\n+\t\t\t\tu64 ip, u64 fp)\n+{\n+\tconst struct bpf_exception_info *exc = prog-\u003eaux-\u003eexc;\n+\tconst struct bpf_cleanup_range *rec;\n+\tu64 spill_base;\n+\n+\tif (!exc || !exc-\u003enr_ranges)\n+\t\treturn;\n+\trec = bpf_cleanup_pad_for_ip(prog, ip);\n+\tif (!rec)\n+\t\treturn;\n+\n+\t/*\n+\t * The callee is always another subprogram of this program -- the walk\n+\t * ends at any frame that is not one -- so its prologue spilled these\n+\t * registers and its exc is there to say where.\n+\t */\n+\tif (ctx-\u003ecallee)\n+\t\tspill_base = ctx-\u003ecallee_fp + ctx-\u003ecallee-\u003eaux-\u003eexc-\u003espill_off;\n+\telse\n+\t\tspill_base = fp + exc-\u003ethrow_spill_off;\n+\n+\tarch_bpf_run_cleanup_pad(rec-\u003epad, fp, spill_base);\n+}\n+\n static bool bpf_stack_walker(void *cookie, u64 ip, u64 sp, u64 bp)\n {\n \tstruct bpf_throw_ctx *ctx = cookie;\n@@ -3416,6 +3445,11 @@ static bool bpf_stack_walker(void *cookie, u64 ip, u64 sp, u64 bp)\n \tif (!prog)\n \t\treturn !ctx-\u003ecnt;\n \tctx-\u003ecnt++;\n+\n+\tbpf_run_cleanup_pad(ctx, prog, ip, bp);\n+\tctx-\u003ecallee = prog;\n+\tctx-\u003ecallee_fp = bp;\n+\n \tif (bpf_is_subprog(prog))\n \t\treturn true;\n \tctx-\u003eaux = prog-\u003eaux;\n@@ -3445,6 +3479,17 @@ __bpf_kfunc void bpf_throw(u64 cookie)\n \tWARN(1, \"A call to BPF exception callback should never return\\n\");\n }\n \n+/*\n+ * Terminator of a compiler-emitted cleanup landing pad. The compiler names\n+ * this _Unwind_Resume, the base unwind ABI's entry point for carrying an\n+ * unwind on once a frame's cleanups have run. To match kernel kfunc\n+ * convention, the kernel calls it bpf_unwind_resume and libbpf maps the\n+ * compiler's name onto it.\n+ */\n+__bpf_kfunc void bpf_unwind_resume(void)\n+{\n+}\n+\n __bpf_kfunc int bpf_wq_init(struct bpf_wq *wq, void *p__const_map, unsigned int flags)\n {\n \tstruct bpf_async_kern *async = (struct bpf_async_kern *)wq;\n@@ -4981,6 +5026,7 @@ BTF_ID_FLAGS(func, bpf_task_get_cgroup1, KF_ACQUIRE | KF_RCU | KF_RET_NULL)\n BTF_ID_FLAGS(func, bpf_task_from_pid, KF_ACQUIRE | KF_RET_NULL)\n BTF_ID_FLAGS(func, bpf_task_from_vpid, KF_ACQUIRE | KF_RET_NULL)\n BTF_ID_FLAGS(func, bpf_throw)\n+BTF_ID_FLAGS(func, bpf_unwind_resume)\n #ifdef CONFIG_BPF_EVENTS\n BTF_ID_FLAGS(func, bpf_send_signal_task)\n #endif\ndiff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c\nindex 44ecdc5b4ec2d..9cfd05f970bc9 100644\n--- a/kernel/bpf/liveness.c\n+++ b/kernel/bpf/liveness.c\n@@ -8,6 +8,8 @@\n #include \u003clinux/slab.h\u003e\n #include \u003clinux/sort.h\u003e\n \n+#include \"exception.h\"\n+\n #define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)\n \n struct per_frame_masks {\n@@ -256,6 +258,9 @@ bpf_insn_successors(struct bpf_verifier_env *env, u32 idx)\n \tsucc = env-\u003esucc;\n \tsucc-\u003ecnt = 0;\n \n+\tif (unlikely(bpf_is_unwind_resume_kfunc(insn)))\n+\t\treturn succ;\n+\n \topcode_info = \u0026opcode_info_tbl[BPF_CLASS(insn-\u003ecode) | BPF_OP(insn-\u003ecode)];\n \tinsn_sz = bpf_is_ldimm64(insn) ? 2 : 1;\n \tif (opcode_info-\u003ecan_fallthrough)\n@@ -264,6 +269,13 @@ bpf_insn_successors(struct bpf_verifier_env *env, u32 idx)\n \tif (opcode_info-\u003ecan_jump)\n \t\tsucc-\u003eitems[succ-\u003ecnt++] = idx + bpf_jmp_offset(insn) + 1;\n \n+\tif (unlikely(env-\u003ecleanup_info_cnt)) {\n+\t\tint pad = bpf_cleanup_pad_of_call(env, idx);\n+\n+\t\tif (pad \u003e= 0)\n+\t\t\tsucc-\u003eitems[succ-\u003ecnt++] = pad;\n+\t}\n+\n \treturn succ;\n }\n \n@@ -397,6 +409,14 @@ bool bpf_stack_slot_alive(struct bpf_verifier_env *env, u32 frameno, u32 half_sp\n \t\talive = bpf_calls_callback(env, callsite)\n \t\t\t? is_live_before(instance, callsite, rel, half_spi)\n \t\t\t: is_live_before(instance, callsite + 1, rel, half_spi);\n+\n+\t\t/* Control may also go to the landing pad. */\n+\t\tif (!alive \u0026\u0026 unlikely(env-\u003ecleanup_info_cnt)) {\n+\t\t\tint pad = bpf_cleanup_pad_of_call(env, callsite);\n+\n+\t\t\tif (pad \u003e= 0)\n+\t\t\t\talive = is_live_before(instance, pad, rel, half_spi);\n+\t\t}\n \t\tif (alive)\n \t\t\treturn true;\n \t}\ndiff --git a/kernel/bpf/states.c b/kernel/bpf/states.c\nindex 66fb11b6c6a76..b101baa43171c 100644\n--- a/kernel/bpf/states.c\n+++ b/kernel/bpf/states.c\n@@ -996,6 +996,12 @@ static bool states_equal(struct bpf_verifier_env *env,\n \tif (old-\u003ein_sleepable != cur-\u003ein_sleepable)\n \t\treturn false;\n \n+\tif (old-\u003eunwinding != cur-\u003eunwinding)\n+\t\treturn false;\n+\n+\tif (old-\u003eunwinding \u0026\u0026 old-\u003eunwind_frameno != cur-\u003eunwind_frameno)\n+\t\treturn false;\n+\n \tif (!refsafe(old, cur, \u0026env-\u003eidmap_scratch))\n \t\treturn false;\n \ndiff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c\nindex 113486b15d29f..049717dc3b7d7 100644\n--- a/kernel/bpf/syscall.c\n+++ b/kernel/bpf/syscall.c\n@@ -2915,7 +2915,7 @@ int __init __used bpf_multi_func(void) { return 0; }\n BTF_ID_LIST_GLOBAL_SINGLE(bpf_multi_func_btf_id, func, bpf_multi_func)\n \n /* last field in 'union bpf_attr' used by this command */\n-#define BPF_PROG_LOAD_LAST_FIELD keyring_id\n+#define BPF_PROG_LOAD_LAST_FIELD cleanup_info_cnt\n \n static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, struct bpf_log_attr *attr_log)\n {\ndiff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c\nindex 18aad4886f9cf..b978e710d7b66 100644\n--- a/kernel/bpf/verifier.c\n+++ b/kernel/bpf/verifier.c\n@@ -10,6 +10,7 @@\n #include \u003clinux/slab.h\u003e\n #include \u003clinux/bpf.h\u003e\n #include \u003clinux/btf.h\u003e\n+#include \u003clinux/bpf_cleanup_abi.h\u003e\n #include \u003clinux/bpf_verifier.h\u003e\n #include \u003clinux/filter.h\u003e\n #include \u003cnet/netlink.h\u003e\n@@ -37,6 +38,7 @@\n \n #include \"diagnostics.h\"\n #include \"disasm.h\"\n+#include \"exception.h\"\n \n static const struct bpf_verifier_ops * const bpf_verifier_ops[] = {\n #define BPF_PROG_TYPE(_id, _name, prog_ctx_type, kern_ctx_type) \\\n@@ -1714,6 +1716,8 @@ int bpf_copy_verifier_state(struct bpf_verifier_state *dst_state,\n \t\treturn err;\n \tdst_state-\u003especulative = src-\u003especulative;\n \tdst_state-\u003ein_sleepable = src-\u003ein_sleepable;\n+\tdst_state-\u003eunwinding = src-\u003eunwinding;\n+\tdst_state-\u003eunwind_frameno = src-\u003eunwind_frameno;\n \tdst_state-\u003ecurframe = src-\u003ecurframe;\n \tdst_state-\u003ebranches = src-\u003ebranches;\n \tdst_state-\u003eparent = src-\u003eparent;\n@@ -5588,6 +5592,14 @@ static int check_max_stack_depth(struct bpf_verifier_env *env)\n \t\t}\n \t}\n \n+\t/*\n+\t * A pad rebuilds its frame from a spill area, and on x86-64 a private\n+\t * stack's frame pointer lives in r9, which no spill area holds.\n+\t * Refused on every arch rather than just that one.\n+\t */\n+\tif (env-\u003ecleanup_info_cnt)\n+\t\tpriv_stack_mode = NO_PRIV_STACK;\n+\n \tif (priv_stack_mode == PRIV_STACK_UNKNOWN)\n \t\tpriv_stack_mode = bpf_enable_priv_stack(env-\u003eprog);\n \n@@ -10502,6 +10514,10 @@ static int push_callback_call(struct bpf_verifier_env *env, struct bpf_insn *ins\n \t * callbacks\n \t */\n \tenv-\u003esubprog_info[subprog].is_cb = true;\n+\terr = bpf_cleanup_check_callback(env, subprog);\n+\tif (err)\n+\t\treturn err;\n+\n \tif (bpf_pseudo_kfunc_call(insn) \u0026\u0026\n \t !is_callback_calling_kfunc(insn-\u003eimm)) {\n \t\tverifier_bug(env, \"kfunc %s#%d not marked as callback-calling\",\n@@ -10553,8 +10569,8 @@ static int push_callback_call(struct bpf_verifier_env *env, struct bpf_insn *ins\n \treturn 0;\n }\n \n-static int process_bpf_exit_full(struct bpf_verifier_env *env,\n-\t\t\t\t bool *do_print_state, bool exception_exit);\n+static int process_bpf_exit_full(struct bpf_verifier_env *env, bool *do_print_state);\n+static int unwind_step(struct bpf_verifier_env *env, u32 callsite, int *insn_idx);\n \n static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,\n \t\t\t int *insn_idx)\n@@ -10644,7 +10660,7 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,\n \t\t\t\tverbose(env, \"failed to push state for global subprog exception path\\n\");\n \t\t\t\treturn PTR_ERR(branch);\n \t\t\t}\n-\t\t\treturn process_bpf_exit_full(env, NULL, true);\n+\t\t\treturn unwind_step(env, *insn_idx, insn_idx);\n \t\t}\n \n \t\t/* continue with next insn after call */\n@@ -14545,7 +14561,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,\n \t\tenv-\u003eprog-\u003ecall_session_cookie = true;\n \n \tif (bpf_is_throw_kfunc(insn))\n-\t\treturn process_bpf_exit_full(env, NULL, true);\n+\t\treturn unwind_step(env, insn_idx, \u0026env-\u003einsn_idx);\n \n \treturn 0;\n }\n@@ -18470,9 +18486,103 @@ enum {\n \tINSN_IDX_UPDATED = 2,\n };\n \n-static int process_bpf_exit_full(struct bpf_verifier_env *env,\n-\t\t\t\t bool *do_print_state,\n-\t\t\t\t bool exception_exit)\n+static u32 unwind_pop_frame(struct bpf_verifier_env *env)\n+{\n+\tstruct bpf_verifier_state *state = env-\u003ecur_state;\n+\tstruct bpf_func_state *callee = state-\u003eframe[state-\u003ecurframe];\n+\tu32 callsite = callee-\u003ecallsite;\n+\tstruct bpf_func_state *caller;\n+\n+\tcaller = state-\u003eframe[state-\u003ecurframe - 1];\n+\taccount_processed_insns(env, callee, caller);\n+\tfree_func_state(callee);\n+\tstate-\u003eframe[state-\u003ecurframe--] = NULL;\n+\tinvalidate_outgoing_stack_args(env, caller);\n+\treturn callsite;\n+}\n+\n+static void unwind_enter_pad(struct bpf_verifier_env *env)\n+{\n+\tstruct bpf_verifier_state *state = env-\u003ecur_state;\n+\tstruct bpf_func_state *frame = cur_func(env);\n+\n+\tstate-\u003eunwind_frameno = state-\u003ecurframe;\n+\tclear_caller_saved_regs(env, frame-\u003eregs);\n+\tmark_reg_unknown(env, frame-\u003eregs, BPF_REG_0);\n+\t__mark_reg_known(\u0026frame-\u003eregs[BPF_REG_0], BPF_PAD_ENTRY_R0);\n+}\n+\n+static int unwind_finish(struct bpf_verifier_env *env)\n+{\n+\tint err = check_resource_leak(env, true, true, \"bpf_throw\");\n+\n+\tif (err)\n+\t\treturn err;\n+\treturn PROCESS_BPF_EXIT;\n+}\n+\n+static int unwind_step(struct bpf_verifier_env *env, u32 callsite, int *insn_idx)\n+{\n+\tstruct bpf_verifier_state *state = env-\u003ecur_state;\n+\tu8 popped = 0;\n+\n+\tstate-\u003eunwinding = true;\n+\tfor (;;) {\n+\t\tint pad = bpf_cleanup_pad_of_call(env, callsite);\n+\n+\t\tif (pad \u003e= 0) {\n+\t\t\tunwind_enter_pad(env);\n+\t\t\t/*\n+\t\t\t * The edge from @callsite to the pad crosses @popped\n+\t\t\t * frames, and nothing in the instruction stream says\n+\t\t\t * so. Record it for mark_chain_precision(), which has\n+\t\t\t * to walk back through the same frames.\n+\t\t\t */\n+\t\t\tenv-\u003eunwind_frames = popped;\n+\t\t\t*insn_idx = pad;\n+\t\t\treturn INSN_IDX_UPDATED;\n+\t\t}\n+\t\tif (!state-\u003ecurframe)\n+\t\t\treturn unwind_finish(env);\n+\t\tcallsite = unwind_pop_frame(env);\n+\t\tpopped++;\n+\t}\n+}\n+\n+static int process_cleanup_resume(struct bpf_verifier_env *env, int *insn_idx)\n+{\n+\tstruct bpf_verifier_state *state = env-\u003ecur_state;\n+\tint err;\n+\n+\t/* A pad entered by ordinary control flow. */\n+\tif (!state-\u003eunwinding) {\n+\t\tverbose(env,\n+\t\t\t\"bpf_unwind_resume() at insn %d reached without an exception in flight\\n\",\n+\t\t\t*insn_idx);\n+\t\treturn -EINVAL;\n+\t}\n+\t/*\n+\t * A resume in a subprogram the pad called, which has a landing pad of\n+\t * its own and reached it by ordinary control flow. A JIT lowers a\n+\t * resume as the way back out of a pad, which reaches the walker only\n+\t * from the frame whose pad it called.\n+\t */\n+\tif (state-\u003ecurframe != state-\u003eunwind_frameno) {\n+\t\tverbose(env,\n+\t\t\t\"bpf_unwind_resume() at insn %d is in frame %d, not frame %d whose landing pad the exception entered\\n\",\n+\t\t\t*insn_idx, state-\u003ecurframe, state-\u003eunwind_frameno);\n+\t\treturn -EINVAL;\n+\t}\n+\tif (!state-\u003ecurframe)\n+\t\treturn unwind_finish(env);\n+\terr = unwind_step(env, unwind_pop_frame(env), insn_idx);\n+\t/* unwind_step() counted the frames it popped, not this one. */\n+\tif (err == INSN_IDX_UPDATED)\n+\t\tenv-\u003eunwind_frames++;\n+\treturn err;\n+}\n+\n+static int process_bpf_exit_full(struct bpf_verifier_env *env, bool *do_print_state)\n {\n \tstruct bpf_func_state *cur_frame = cur_func(env);\n \n@@ -18482,25 +18592,11 @@ static int process_bpf_exit_full(struct bpf_verifier_env *env,\n \t * for which reference_state must match caller reference\n \t * state when it exits.\n \t */\n-\tint err = check_resource_leak(env, exception_exit,\n-\t\t\t\t exception_exit || !env-\u003ecur_state-\u003ecurframe,\n-\t\t\t\t exception_exit ? \"bpf_throw\" :\n+\tint err = check_resource_leak(env, false, !env-\u003ecur_state-\u003ecurframe,\n \t\t\t\t \"BPF_EXIT instruction in main prog\");\n \tif (err)\n \t\treturn err;\n \n-\t/* The side effect of the prepare_func_exit which is\n-\t * being skipped is that it frees bpf_func_state.\n-\t * Typically, process_bpf_exit will only be hit with\n-\t * outermost exit. copy_verifier_state in pop_stack will\n-\t * handle freeing of any extra bpf_func_state left over\n-\t * from not processing all nested function exits. We\n-\t * also skip return code checks as they are not needed\n-\t * for exceptional exits.\n-\t */\n-\tif (exception_exit)\n-\t\treturn PROCESS_BPF_EXIT;\n-\n \tif (env-\u003ecur_state-\u003ecurframe) {\n \t\t/* exit from nested function */\n \t\terr = prepare_func_exit(env, \u0026env-\u003einsn_idx);\n@@ -18674,6 +18770,8 @@ static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state)\n \n \t\tenv-\u003ejmps_processed++;\n \t\tif (opcode == BPF_CALL) {\n+\t\t\tif (bpf_is_unwind_resume_kfunc(insn))\n+\t\t\t\treturn process_cleanup_resume(env, \u0026env-\u003einsn_idx);\n \t\t\tif (env-\u003ecur_state-\u003eactive_locks) {\n \t\t\t\tif ((insn-\u003esrc_reg == BPF_REG_0 \u0026\u0026\n \t\t\t\t insn-\u003eimm != BPF_FUNC_spin_unlock \u0026\u0026\n@@ -18707,7 +18805,7 @@ static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state)\n \t\t\t\tenv-\u003einsn_idx += insn-\u003eimm + 1;\n \t\t\treturn INSN_IDX_UPDATED;\n \t\t} else if (opcode == BPF_EXIT) {\n-\t\t\treturn process_bpf_exit_full(env, do_print_state, false);\n+\t\t\treturn process_bpf_exit_full(env, do_print_state);\n \t\t}\n \t\treturn check_cond_jmp_op(env, insn, \u0026env-\u003einsn_idx);\n \t}\n@@ -18744,10 +18842,14 @@ static int do_check(struct bpf_verifier_env *env)\n \tfor (;;) {\n \t\tstruct bpf_insn *insn;\n \t\tstruct bpf_insn_aux_data *insn_aux;\n+\t\tu8 unwind_frames;\n \t\tint err;\n \n \t\t/* reset current history entry on each new instruction */\n \t\tenv-\u003ecur_hist_ent = NULL;\n+\t\t/* frames the unwind popped to reach this insn, if it is a pad */\n+\t\tunwind_frames = env-\u003eunwind_frames;\n+\t\tenv-\u003eunwind_frames = 0;\n \n \t\tenv-\u003eprev_insn_idx = prev_insn_idx;\n \t\tif (env-\u003einsn_idx \u003e= insn_cnt) {\n@@ -18811,10 +18913,16 @@ static int do_check(struct bpf_verifier_env *env)\n \t\t\t}\n \t\t}\n \n-\t\tif (bpf_is_jmp_point(env, env-\u003einsn_idx)) {\n+\t\t/*\n+\t\t * A landing pad is a jump point, but record the edge even if\n+\t\t * that ever stops being true: it is the only place the frames\n+\t\t * the unwind popped are written down.\n+\t\t */\n+\t\tif (bpf_is_jmp_point(env, env-\u003einsn_idx) || unwind_frames) {\n \t\t\terr = bpf_push_jmp_history(env, state, 0, 0, 0, 0);\n \t\t\tif (err)\n \t\t\t\treturn err;\n+\t\t\tenv-\u003ecur_hist_ent-\u003eunwind_frames = unwind_frames;\n \t\t}\n \n \t\tif (signal_pending(current))\n@@ -21677,6 +21785,11 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,\n \tif (ret \u003c 0)\n \t\tgoto skip_full_check;\n \n+\t/* The CFG needs an edge from a call in a cleanup range to its pad. */\n+\tret = bpf_prepare_cleanup_exceptions(env);\n+\tif (ret \u003c 0)\n+\t\tgoto skip_full_check;\n+\n \t/* Validate instructions and resolve the program's referenced resources. */\n \tret = check_and_resolve_insns(env);\n \tif (ret \u003c 0)\n@@ -21697,6 +21810,10 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,\n \tif (ret \u003c 0)\n \t\tgoto skip_full_check;\n \n+\tret = bpf_check_cleanup_exceptions(env);\n+\tif (ret \u003c 0)\n+\t\tgoto skip_full_check;\n+\n \tret = bpf_compute_postorder(env);\n \tif (ret \u003c 0)\n \t\tgoto skip_full_check;\n@@ -21884,6 +22001,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,\n \tkvfree(env-\u003esucc);\n \tkvfree(env-\u003egotox_tmp_buf);\n \tbpf_diag_free(env);\n+\tkvfree(env-\u003ecleanup_info);\n \tkvfree(env);\n \treturn ret;\n }\ndiff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h\nindex 6330b7d745c57..a69cc04275751 100644\n--- a/tools/include/uapi/linux/bpf.h\n+++ b/tools/include/uapi/linux/bpf.h\n@@ -1669,6 +1669,9 @@ union bpf_attr {\n \t\t * verification.\n \t\t */\n \t\t__s32\t\tkeyring_id;\n+\t\t__aligned_u64\tcleanup_info;\t/* exception cleanup table */\n+\t\t__u32\t\tcleanup_info_rec_size; /* userspace bpf_cleanup_info size */\n+\t\t__u32\t\tcleanup_info_cnt; /* number of bpf_cleanup_info records */\n \t};\n \n \tstruct { /* anonymous struct used by BPF_OBJ_* commands */\n@@ -7599,6 +7602,12 @@ struct bpf_line_info {\n \t__u32\tline_col;\n };\n \n+struct bpf_cleanup_info {\n+\t__u32\tbegin_off;\n+\t__u32\tend_off;\n+\t__u32\tlanding_pad_off;\n+};\n+\n struct bpf_spin_lock {\n \t__u32\tval;\n };\ndiff --git a/tools/lib/bpf/bpf.c b/tools/lib/bpf/bpf.c\nindex a9de7f107cf7b..f11d042a28504 100644\n--- a/tools/lib/bpf/bpf.c\n+++ b/tools/lib/bpf/bpf.c\n@@ -295,7 +295,7 @@ int bpf_prog_load(enum bpf_prog_type prog_type,\n \t\t const struct bpf_insn *insns, size_t insn_cnt,\n \t\t struct bpf_prog_load_opts *opts)\n {\n-\tconst size_t attr_sz = offsetofend(union bpf_attr, keyring_id);\n+\tconst size_t attr_sz = offsetofend(union bpf_attr, cleanup_info_cnt);\n \tvoid *finfo = NULL, *linfo = NULL;\n \tconst char *func_info, *line_info;\n \t__u32 log_size, log_level, attach_prog_fd, attach_btf_obj_fd;\n@@ -370,6 +370,10 @@ int bpf_prog_load(enum bpf_prog_type prog_type,\n \tattr.fd_array = ptr_to_u64(OPTS_GET(opts, fd_array, NULL));\n \tattr.fd_array_cnt = OPTS_GET(opts, fd_array_cnt, 0);\n \n+\tattr.cleanup_info = ptr_to_u64(OPTS_GET(opts, cleanup_info, NULL));\n+\tattr.cleanup_info_cnt = OPTS_GET(opts, cleanup_info_cnt, 0);\n+\tattr.cleanup_info_rec_size = OPTS_GET(opts, cleanup_info_rec_size, 0);\n+\n \tif (log_level) {\n \t\tattr.log_buf = ptr_to_u64(log_buf);\n \t\tattr.log_size = log_size;\ndiff --git a/tools/lib/bpf/bpf.h b/tools/lib/bpf/bpf.h\nindex 490e8cb4ba537..ec542fd68f631 100644\n--- a/tools/lib/bpf/bpf.h\n+++ b/tools/lib/bpf/bpf.h\n@@ -128,9 +128,14 @@ struct bpf_prog_load_opts {\n \n \t/* if set, provides the length of fd_array */\n \t__u32 fd_array_cnt;\n+\n+\t/* exception cleanup table, from the .bpf_cleanup section */\n+\tconst void *cleanup_info;\n+\t__u32 cleanup_info_cnt;\n+\t__u32 cleanup_info_rec_size;\n \tsize_t :0;\n };\n-#define bpf_prog_load_opts__last_field fd_array_cnt\n+#define bpf_prog_load_opts__last_field cleanup_info_rec_size\n \n LIBBPF_API int bpf_prog_load(enum bpf_prog_type prog_type,\n \t\t\t const char *prog_name, const char *license,\ndiff --git a/tools/lib/bpf/gen_loader.c b/tools/lib/bpf/gen_loader.c\nindex af3a04f161ac1..2345fbdd46f56 100644\n--- a/tools/lib/bpf/gen_loader.c\n+++ b/tools/lib/bpf/gen_loader.c\n@@ -981,13 +981,15 @@ static void cleanup_relos(struct bpf_gen *gen, int insns)\n \tcleanup_core_relo(gen);\n }\n \n-/* Convert func, line, and core relo info blobs to target endianness */\n+/* Convert func, line, core relo and cleanup info blobs to target endianness */\n static void info_blob_bswap(struct bpf_gen *gen, int func_info, int line_info,\n-\t\t\t int core_relos, struct bpf_prog_load_opts *load_attr)\n+\t\t\t int core_relos, int cleanup_info,\n+\t\t\t struct bpf_prog_load_opts *load_attr)\n {\n \tstruct bpf_func_info *fi = gen-\u003edata_start + func_info;\n \tstruct bpf_line_info *li = gen-\u003edata_start + line_info;\n \tstruct bpf_core_relo *cr = gen-\u003edata_start + core_relos;\n+\tstruct bpf_cleanup_info *ci = gen-\u003edata_start + cleanup_info;\n \tint i;\n \n \tfor (i = 0; i \u003c load_attr-\u003efunc_info_cnt; i++)\n@@ -998,6 +1000,9 @@ static void info_blob_bswap(struct bpf_gen *gen, int func_info, int line_info,\n \n \tfor (i = 0; i \u003c gen-\u003ecore_relo_cnt; i++)\n \t\tbpf_core_relo_bswap(cr++);\n+\n+\tfor (i = 0; i \u003c load_attr-\u003ecleanup_info_cnt; i++)\n+\t\tbpf_cleanup_info_bswap(ci++);\n }\n \n void bpf_gen__prog_load(struct bpf_gen *gen,\n@@ -1011,8 +1016,11 @@ void bpf_gen__prog_load(struct bpf_gen *gen,\n \t\t\t load_attr-\u003eline_info_rec_size;\n \tint core_relo_tot_sz = gen-\u003ecore_relo_cnt *\n \t\t\t sizeof(struct bpf_core_relo);\n+\tint cleanup_info_tot_sz = load_attr-\u003ecleanup_info_cnt *\n+\t\t\t\t load_attr-\u003ecleanup_info_rec_size;\n \tint prog_load_attr, license_off, insns_off, func_info, line_info, core_relos;\n-\tint attr_size = offsetofend(union bpf_attr, core_relo_rec_size);\n+\tint attr_size = offsetofend(union bpf_attr, cleanup_info_cnt);\n+\tint cleanup_info;\n \tunion bpf_attr attr;\n \n \tmemset(\u0026attr, 0, attr_size);\n@@ -1061,9 +1069,17 @@ void bpf_gen__prog_load(struct bpf_gen *gen,\n \t\t core_relos, gen-\u003ecore_relo_cnt,\n \t\t sizeof(struct bpf_core_relo));\n \n+\tattr.cleanup_info_rec_size = tgt_endian(load_attr-\u003ecleanup_info_rec_size);\n+\tattr.cleanup_info_cnt = tgt_endian(load_attr-\u003ecleanup_info_cnt);\n+\tcleanup_info = add_data(gen, load_attr-\u003ecleanup_info, cleanup_info_tot_sz);\n+\tpr_debug(\"gen: prog_load: cleanup_info: off %d cnt %u rec size %u\\n\",\n+\t\t cleanup_info, load_attr-\u003ecleanup_info_cnt,\n+\t\t load_attr-\u003ecleanup_info_rec_size);\n+\n \t/* convert all info blobs to target endianness */\n \tif (gen-\u003eswapped_endian \u0026\u0026 !gen-\u003eerror)\n-\t\tinfo_blob_bswap(gen, func_info, line_info, core_relos, load_attr);\n+\t\tinfo_blob_bswap(gen, func_info, line_info, core_relos, cleanup_info,\n+\t\t\t\tload_attr);\n \n \tlibbpf_strlcpy(attr.prog_name, prog_name, sizeof(attr.prog_name));\n \tprog_load_attr = add_data(gen, \u0026attr, attr_size);\n@@ -1085,6 +1101,11 @@ void bpf_gen__prog_load(struct bpf_gen *gen,\n \t/* populate union bpf_attr with a pointer to core_relos */\n \temit_rel_store(gen, attr_field(prog_load_attr, core_relos), core_relos);\n \n+\t/* populate union bpf_attr with a pointer to cleanup_info, if there is one */\n+\tif (load_attr-\u003ecleanup_info_cnt)\n+\t\temit_rel_store(gen, attr_field(prog_load_attr, cleanup_info),\n+\t\t\t cleanup_info);\n+\n \t/* populate union bpf_attr fd_array with a pointer to data where map_fds are saved */\n \temit_rel_store(gen, attr_field(prog_load_attr, fd_array), gen-\u003efd_array);\n \ndiff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c\nindex 2f53afc985cfb..a8da2b680685c 100644\n--- a/tools/lib/bpf/libbpf.c\n+++ b/tools/lib/bpf/libbpf.c\n@@ -514,6 +514,11 @@ struct bpf_program {\n \tvoid *line_info;\n \t__u32 line_info_rec_size;\n \t__u32 line_info_cnt;\n+\n+\tstruct bpf_cleanup_info *cleanup_info;\n+\t__u32 cleanup_info_rec_size;\n+\t__u32 cleanup_info_cnt;\n+\n \t__u32 prog_flags;\n \t__u8 hash[SHA256_DIGEST_LENGTH];\n \n@@ -549,6 +554,7 @@ struct bpf_struct_ops {\n #define STRUCT_OPS_SEC \".struct_ops\"\n #define STRUCT_OPS_LINK_SEC \".struct_ops.link\"\n #define ARENA_SEC \".addr_space.1\"\n+#define CLEANUP_SEC \".bpf_cleanup\"\n \n enum libbpf_map_type {\n \tLIBBPF_MAP_UNSPEC,\n@@ -677,6 +683,25 @@ struct elf_sec_desc {\n \tElf_Data *data;\n };\n \n+#define CLEANUP_REC_FIELDS\t(sizeof(struct bpf_cleanup_info) / sizeof(__u32))\n+\n+/* Index of each field of struct bpf_cleanup_info, read as an array of __u32. */\n+enum {\n+\tCLEANUP_REC_BEGIN,\n+\tCLEANUP_REC_END,\n+\tCLEANUP_REC_PAD,\n+};\n+\n+/* One (begin, end, landing_pad) triple from .bpf_cleanup, with each field\n+ * resolved from its relocation to an ELF section plus a section-relative\n+ * instruction index. The mapping to final program instruction indices can only\n+ * happen after subprogram placement, which differs per main program.\n+ */\n+struct cleanup_raw_rec {\n+\tint sec_idx[CLEANUP_REC_FIELDS];\n+\tsize_t insn_idx[CLEANUP_REC_FIELDS];\n+};\n+\n struct elf_state {\n \tint fd;\n \tconst void *obj_buf;\n@@ -696,6 +721,8 @@ struct elf_state {\n \tbool has_st_ops;\n \tint arena_data_shndx;\n \tint jumptables_data_shndx;\n+\tElf_Data *cleanup_data;\n+\tint cleanup_shndx;\n };\n \n struct usdt_manager;\n@@ -771,6 +798,9 @@ struct bpf_object {\n \tvoid *jumptables_data;\n \tsize_t jumptables_data_sz;\n \n+\tstruct cleanup_raw_rec *cleanup_recs;\n+\tsize_t cleanup_rec_cnt;\n+\n \tstruct {\n \t\tstruct bpf_program *prog;\n \t\tunsigned int sym_off;\n@@ -817,7 +847,10 @@ static void bpf_program__exit(struct bpf_program *prog)\n \tzfree(\u0026prog-\u003esec_name);\n \tzfree(\u0026prog-\u003einsns);\n \tzfree(\u0026prog-\u003ereloc_desc);\n+\tzfree(\u0026prog-\u003ecleanup_info);\n \n+\tprog-\u003ecleanup_info_rec_size = 0;\n+\tprog-\u003ecleanup_info_cnt = 0;\n \tprog-\u003enr_reloc = 0;\n \tprog-\u003einsns_cnt = 0;\n \tprog-\u003esec_idx = -1;\n@@ -1559,6 +1592,7 @@ static struct bpf_object *bpf_object__new(const char *path,\n \tobj-\u003eefile.obj_buf = obj_buf;\n \tobj-\u003eefile.obj_buf_sz = obj_buf_sz;\n \tobj-\u003eefile.btf_maps_shndx = -1;\n+\tobj-\u003eefile.cleanup_shndx = -1;\n \tobj-\u003ekconfig_map_idx = -1;\n \tobj-\u003earena_map_idx = -1;\n \n@@ -4045,6 +4079,9 @@ static int bpf_object__elf_collect(struct bpf_object *obj)\n \t\t\t\tsec_desc-\u003eshdr = sh;\n \t\t\t\tsec_desc-\u003edata = data;\n \t\t\t\tobj-\u003eefile.has_st_ops = true;\n+\t\t\t} else if (strcmp(name, CLEANUP_SEC) == 0) {\n+\t\t\t\tobj-\u003eefile.cleanup_data = data;\n+\t\t\t\tobj-\u003eefile.cleanup_shndx = idx;\n \t\t\t} else if (strcmp(name, ARENA_SEC) == 0) {\n \t\t\t\tobj-\u003eefile.arena_data = data;\n \t\t\t\tobj-\u003eefile.arena_data_shndx = idx;\n@@ -4072,6 +4109,7 @@ static int bpf_object__elf_collect(struct bpf_object *obj)\n \t\t\t strcmp(name, \".rel\" STRUCT_OPS_LINK_SEC) \u0026\u0026\n \t\t\t strcmp(name, \".rel?\" STRUCT_OPS_SEC) \u0026\u0026\n \t\t\t strcmp(name, \".rel?\" STRUCT_OPS_LINK_SEC) \u0026\u0026\n+\t\t\t strcmp(name, \".rel\" CLEANUP_SEC) \u0026\u0026\n \t\t\t strcmp(name, \".rel\" MAPS_ELF_SEC)) {\n \t\t\t\tpr_info(\"elf: skipping relo section(%d) %s for section(%d) %s\\n\",\n \t\t\t\t\tidx, name, targ_sec_idx,\n@@ -4852,6 +4890,229 @@ static struct bpf_program *find_prog_by_sec_insn(const struct bpf_object *obj,\n \treturn NULL;\n }\n \n+static int bpf_object__init_cleanup_info(struct bpf_object *obj)\n+{\n+\tElf_Data *data = obj-\u003eefile.cleanup_data;\n+\tElf_Data *relo = NULL;\n+\tsize_t i, nrels, nslots, nrecs;\n+\tstruct cleanup_raw_rec *recs;\n+\tint *slot_sec, ret = 0;\n+\tsize_t *slot_val;\n+\tconst __u32 *vals;\n+\tbool native;\n+\n+\tif (!data || obj-\u003eefile.cleanup_shndx \u003c 0)\n+\t\treturn 0;\n+\n+\tnative = is_native_endianness(obj);\n+\n+\tfor (i = 0; i \u003c obj-\u003eefile.sec_cnt; i++) {\n+\t\tstruct elf_sec_desc *sd = \u0026obj-\u003eefile.secs[i];\n+\n+\t\tif (sd-\u003esec_type == SEC_RELO \u0026\u0026 sd-\u003eshdr \u0026\u0026\n+\t\t sd-\u003eshdr-\u003esh_info == (Elf64_Word)obj-\u003eefile.cleanup_shndx) {\n+\t\t\trelo = sd-\u003edata;\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\tif (!relo) {\n+\t\tpr_warn(\"%s present without relocations\\n\", CLEANUP_SEC);\n+\t\treturn -LIBBPF_ERRNO__FORMAT;\n+\t}\n+\tif (data-\u003ed_size % sizeof(struct bpf_cleanup_info)) {\n+\t\tpr_warn(\"%s size %zu is not a multiple of the record size %zu\\n\",\n+\t\t\tCLEANUP_SEC, data-\u003ed_size, sizeof(struct bpf_cleanup_info));\n+\t\treturn -LIBBPF_ERRNO__FORMAT;\n+\t}\n+\n+\tvals = data-\u003ed_buf;\n+\tnslots = data-\u003ed_size / sizeof(__u32);\n+\tnrecs = data-\u003ed_size / sizeof(struct bpf_cleanup_info);\n+\n+\tslot_sec = calloc(nslots, sizeof(*slot_sec));\n+\tslot_val = calloc(nslots, sizeof(*slot_val));\n+\trecs = calloc(nrecs ?: 1, sizeof(*recs));\n+\tif (!slot_sec || !slot_val || !recs) {\n+\t\tret = -ENOMEM;\n+\t\tgoto out;\n+\t}\n+\tfor (i = 0; i \u003c nslots; i++)\n+\t\tslot_sec[i] = -1;\n+\n+\t/* One relocation per 4-byte field, naming the section it points into. */\n+\tnrels = relo-\u003ed_size / sizeof(Elf64_Rel);\n+\tfor (i = 0; i \u003c nrels; i++) {\n+\t\tElf64_Rel *rel = elf_rel_by_idx(relo, i);\n+\t\tElf64_Sym *sym = elf_sym_by_idx(obj, ELF64_R_SYM(rel-\u003er_info));\n+\t\tsize_t type = ELF64_R_TYPE(rel-\u003er_info);\n+\t\tsize_t slot = rel-\u003er_offset / sizeof(__u32);\n+\n+\t\tif (type != R_BPF_64_NODYLD32 \u0026\u0026 type != R_BPF_64_ABS32) {\n+\t\t\tpr_warn(\"%s: relocation %zu has unexpected type %zu\\n\",\n+\t\t\t\tCLEANUP_SEC, i, type);\n+\t\t\tret = -LIBBPF_ERRNO__FORMAT;\n+\t\t\tgoto out;\n+\t\t}\n+\t\tif (!sym || slot \u003e= nslots || rel-\u003er_offset % sizeof(__u32)) {\n+\t\t\tpr_warn(\"%s: bad relocation %zu\\n\", CLEANUP_SEC, i);\n+\t\t\tret = -LIBBPF_ERRNO__FORMAT;\n+\t\t\tgoto out;\n+\t\t}\n+\t\tslot_sec[slot] = sym-\u003est_shndx;\n+\t\t/* The addend lives in the section data, which libelf leaves in\n+\t\t * the object's byte order; a non-section symbol additionally\n+\t\t * contributes its own value.\n+\t\t */\n+\t\tslot_val[slot] = (native ? vals[slot] : bswap_32(vals[slot])) +\n+\t\t\t\t sym-\u003est_value;\n+\t}\n+\n+\tfor (i = 0; i \u003c nslots; i++) {\n+\t\tstruct cleanup_raw_rec *rec = \u0026recs[i / CLEANUP_REC_FIELDS];\n+\t\tsize_t field = i % CLEANUP_REC_FIELDS;\n+\n+\t\tif (slot_sec[i] \u003c 0) {\n+\t\t\tpr_warn(\"%s: field %zu has no relocation\\n\", CLEANUP_SEC, i);\n+\t\t\tret = -LIBBPF_ERRNO__FORMAT;\n+\t\t\tgoto out;\n+\t\t}\n+\t\tif (slot_val[i] % BPF_INSN_SZ) {\n+\t\t\tpr_warn(\"%s: field %zu offset %zu is not instruction aligned\\n\",\n+\t\t\t\tCLEANUP_SEC, i, slot_val[i]);\n+\t\t\tret = -LIBBPF_ERRNO__FORMAT;\n+\t\t\tgoto out;\n+\t\t}\n+\t\trec-\u003esec_idx[field] = slot_sec[i];\n+\t\trec-\u003einsn_idx[field] = slot_val[i] / BPF_INSN_SZ;\n+\t}\n+\n+\tobj-\u003ecleanup_recs = recs;\n+\tobj-\u003ecleanup_rec_cnt = nrecs;\n+\trecs = NULL;\n+out:\n+\tfree(recs);\n+\tfree(slot_val);\n+\tfree(slot_sec);\n+\treturn ret;\n+}\n+\n+static int cmp_cleanup_info(const void *a, const void *b)\n+{\n+\tconst struct bpf_cleanup_info *x = a, *y = b;\n+\n+\tif (x-\u003ebegin_off == y-\u003ebegin_off)\n+\t\treturn 0;\n+\treturn x-\u003ebegin_off \u003c y-\u003ebegin_off ? -1 : 1;\n+}\n+\n+static int bpf_prog_collect_cleanup_info(struct bpf_object *obj,\n+\t\t\t\t\t struct bpf_program *prog)\n+{\n+\tsize_t i;\n+\tint j;\n+\n+\tfor (i = 0; i \u003c obj-\u003ecleanup_rec_cnt; i++) {\n+\t\tstruct cleanup_raw_rec *raw = \u0026obj-\u003ecleanup_recs[i];\n+\t\tstruct bpf_program *owner = NULL;\n+\t\tstruct bpf_cleanup_info ci = {};\n+\t\t__u32 fields[CLEANUP_REC_FIELDS];\n+\t\tvoid *tmp;\n+\n+\t\tif (raw-\u003esec_idx[CLEANUP_REC_BEGIN] == raw-\u003esec_idx[CLEANUP_REC_END] \u0026\u0026\n+\t\t raw-\u003einsn_idx[CLEANUP_REC_BEGIN] \u003e= raw-\u003einsn_idx[CLEANUP_REC_END]) {\n+\t\t\tpr_warn(\"%s: record %zu is an empty range [%zu,%zu)\\n\",\n+\t\t\t\tCLEANUP_SEC, i, raw-\u003einsn_idx[CLEANUP_REC_BEGIN],\n+\t\t\t\traw-\u003einsn_idx[CLEANUP_REC_END]);\n+\t\t\treturn -LIBBPF_ERRNO__FORMAT;\n+\t\t}\n+\n+\t\tfor (j = 0; j \u003c CLEANUP_REC_FIELDS; j++) {\n+\t\t\tsize_t idx = raw-\u003einsn_idx[j], final;\n+\t\t\tstruct bpf_program *p;\n+\n+\t\t\t/* The end of a range is exclusive, so it may name the\n+\t\t\t * instruction just past the last one of a function,\n+\t\t\t * which belongs to the next function or to nothing at\n+\t\t\t * all. Ask about the last instruction the range covers,\n+\t\t\t * the way the kernel does.\n+\t\t\t */\n+\t\t\tif (j == CLEANUP_REC_END) {\n+\t\t\t\tif (!idx) {\n+\t\t\t\t\tpr_warn(\"%s: record %zu ends at instruction 0\\n\",\n+\t\t\t\t\t\tCLEANUP_SEC, i);\n+\t\t\t\t\treturn -LIBBPF_ERRNO__FORMAT;\n+\t\t\t\t}\n+\t\t\t\tidx--;\n+\t\t\t}\n+\n+\t\t\tp = find_prog_by_sec_insn(obj, raw-\u003esec_idx[j], idx);\n+\t\t\tif (!p) {\n+\t\t\t\tpr_warn(\"%s: record %zu field %d is not inside a function\\n\",\n+\t\t\t\t\tCLEANUP_SEC, i, j);\n+\t\t\t\treturn -LIBBPF_ERRNO__FORMAT;\n+\t\t\t}\n+\t\t\tif (!owner) {\n+\t\t\t\towner = p;\n+\t\t\t} else if (owner != p) {\n+\t\t\t\tpr_warn(\"%s: record %zu spans functions '%s' and '%s'\\n\",\n+\t\t\t\t\tCLEANUP_SEC, i, owner-\u003ename, p-\u003ename);\n+\t\t\t\treturn -LIBBPF_ERRNO__FORMAT;\n+\t\t\t}\n+\n+\t\t\tif (owner == prog) {\n+\t\t\t\tfinal = raw-\u003einsn_idx[j] - prog-\u003esec_insn_off;\n+\t\t\t} else if (prog_is_subprog(obj, owner) \u0026\u0026 owner-\u003esub_insn_off) {\n+\t\t\t\t/* sub_insn_off is where this subprogram was\n+\t\t\t\t * appended to the main program being relocated;\n+\t\t\t\t * zero means it is not part of it.\n+\t\t\t\t */\n+\t\t\t\tfinal = owner-\u003esub_insn_off +\n+\t\t\t\t\traw-\u003einsn_idx[j] - owner-\u003esec_insn_off;\n+\t\t\t} else {\n+\t\t\t\towner = NULL;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\tfields[j] = final;\n+\t\t}\n+\t\tif (!owner)\n+\t\t\tcontinue;\n+\n+\t\tci.begin_off = fields[CLEANUP_REC_BEGIN];\n+\t\tci.end_off = fields[CLEANUP_REC_END];\n+\t\tci.landing_pad_off = fields[CLEANUP_REC_PAD];\n+\n+\t\ttmp = libbpf_reallocarray(prog-\u003ecleanup_info, prog-\u003ecleanup_info_cnt + 1,\n+\t\t\t\t\t sizeof(*prog-\u003ecleanup_info));\n+\t\tif (!tmp)\n+\t\t\treturn -ENOMEM;\n+\t\tprog-\u003ecleanup_info = tmp;\n+\t\tprog-\u003ecleanup_info_rec_size = sizeof(struct bpf_cleanup_info);\n+\t\tprog-\u003ecleanup_info[prog-\u003ecleanup_info_cnt++] = ci;\n+\n+\t\tpr_debug(\"prog '%s': cleanup region [%u,%u) -\u003e landing pad %u\\n\",\n+\t\t\t prog-\u003ename, ci.begin_off, ci.end_off, ci.landing_pad_off);\n+\t}\n+\n+\tif (!prog-\u003ecleanup_info_cnt)\n+\t\treturn 0;\n+\n+\tqsort(prog-\u003ecleanup_info, prog-\u003ecleanup_info_cnt,\n+\t sizeof(*prog-\u003ecleanup_info), cmp_cleanup_info);\n+\tfor (i = 1; i \u003c prog-\u003ecleanup_info_cnt; i++) {\n+\t\tstruct bpf_cleanup_info *prev = \u0026prog-\u003ecleanup_info[i - 1];\n+\t\tstruct bpf_cleanup_info *cur = \u0026prog-\u003ecleanup_info[i];\n+\n+\t\tif (cur-\u003ebegin_off \u003c prev-\u003eend_off) {\n+\t\t\tpr_warn(\"prog '%s': overlapping cleanup regions [%u,%u) and [%u,%u)\\n\",\n+\t\t\t\tprog-\u003ename, prev-\u003ebegin_off, prev-\u003eend_off,\n+\t\t\t\tcur-\u003ebegin_off, cur-\u003eend_off);\n+\t\t\treturn -LIBBPF_ERRNO__FORMAT;\n+\t\t}\n+\t}\n+\n+\treturn 0;\n+}\n+\n static int\n bpf_object__collect_prog_relos(struct bpf_object *obj, Elf64_Shdr *shdr, Elf_Data *data)\n {\n@@ -7561,6 +7822,13 @@ static int bpf_object__relocate(struct bpf_object *obj, const char *targ_btf_pat\n \t\t\t\t\treturn err;\n \t\t\t}\n \t\t}\n+\n+\t\terr = bpf_prog_collect_cleanup_info(obj, prog);\n+\t\tif (err) {\n+\t\t\tpr_warn(\"prog '%s': failed to collect cleanup info: %s\\n\",\n+\t\t\t\tprog-\u003ename, errstr(err));\n+\t\t\treturn err;\n+\t\t}\n \t}\n \tfor (i = 0; i \u003c obj-\u003enr_programs; i++) {\n \t\tprog = \u0026obj-\u003eprograms[i];\n@@ -7751,6 +8019,9 @@ static int bpf_object__collect_relos(struct bpf_object *obj)\n \t\t\treturn -LIBBPF_ERRNO__INTERNAL;\n \t\t}\n \n+\t\tif (idx == obj-\u003eefile.cleanup_shndx)\n+\t\t\tcontinue;\n+\n \t\tif (obj-\u003eefile.secs[idx].sec_type == SEC_ST_OPS)\n \t\t\terr = bpf_object__collect_st_ops_relos(obj, shdr, data);\n \t\telse if (idx == obj-\u003eefile.btf_maps_shndx)\n@@ -8023,6 +8294,11 @@ static int bpf_object_load_prog(struct bpf_object *obj, struct bpf_program *prog\n \t\tload_attr.line_info_rec_size = prog-\u003eline_info_rec_size;\n \t\tload_attr.line_info_cnt = prog-\u003eline_info_cnt;\n \t}\n+\tif (prog-\u003ecleanup_info_cnt) {\n+\t\tload_attr.cleanup_info = prog-\u003ecleanup_info;\n+\t\tload_attr.cleanup_info_cnt = prog-\u003ecleanup_info_cnt;\n+\t\tload_attr.cleanup_info_rec_size = prog-\u003ecleanup_info_rec_size;\n+\t}\n \tload_attr.log_level = log_level;\n \tload_attr.prog_flags = prog-\u003eprog_flags;\n \tload_attr.fd_array = obj-\u003efd_array;\n@@ -8351,6 +8627,15 @@ static void fixup_verifier_log(struct bpf_program *prog, char *buf, size_t buf_s\n \t}\n }\n \n+/* LLVM terminates a cleanup landing pad with a call to _Unwind_Resume, the\n+ * base unwind ABI's entry point for carrying an unwind on once a frame's\n+ * cleanups have run. The kernel knows it as bpf_unwind_resume.\n+ */\n+static const char *kern_extern_name(const char *name)\n+{\n+\treturn strcmp(name, \"_Unwind_Resume\") ? name : \"bpf_unwind_resume\";\n+}\n+\n static int bpf_program_record_relos(struct bpf_program *prog)\n {\n \tstruct bpf_object *obj = prog-\u003eobj;\n@@ -8367,12 +8652,12 @@ static int bpf_program_record_relos(struct bpf_program *prog)\n \t\t\t\tcontinue;\n \t\t\tkind = btf_is_var(btf__type_by_id(obj-\u003ebtf, ext-\u003ebtf_id)) ?\n \t\t\t\tBTF_KIND_VAR : BTF_KIND_FUNC;\n-\t\t\tbpf_gen__record_extern(obj-\u003egen_loader, ext-\u003ename,\n+\t\t\tbpf_gen__record_extern(obj-\u003egen_loader, kern_extern_name(ext-\u003ename),\n \t\t\t\t\t ext-\u003eis_weak, !ext-\u003eksym.type_id,\n \t\t\t\t\t true, kind, relo-\u003einsn_idx);\n \t\t\tbreak;\n \t\tcase RELO_EXTERN_CALL:\n-\t\t\tbpf_gen__record_extern(obj-\u003egen_loader, ext-\u003ename,\n+\t\t\tbpf_gen__record_extern(obj-\u003egen_loader, kern_extern_name(ext-\u003ename),\n \t\t\t\t\t ext-\u003eis_weak, false, false, BTF_KIND_FUNC,\n \t\t\t\t\t relo-\u003einsn_idx);\n \t\t\tbreak;\n@@ -8570,6 +8855,7 @@ static struct bpf_object *bpf_object_open(const char *path, const void *obj_buf,\n \terr = err ? : bpf_object__init_maps(obj, opts);\n \terr = err ? : bpf_object_init_progs(obj, opts);\n \terr = err ? : bpf_object__collect_relos(obj);\n+\terr = err ? : bpf_object__init_cleanup_info(obj);\n \tif (err)\n \t\tgoto out;\n \n@@ -8812,17 +9098,19 @@ static int bpf_object__resolve_ksym_func_btf_id(struct bpf_object *obj,\n \tstruct module_btf *mod_btf = NULL;\n \tconst struct btf_type *kern_func;\n \tstruct btf *kern_btf = NULL;\n+\tconst char *kern_name;\n \tint ret;\n \n \tlocal_func_proto_id = ext-\u003eksym.type_id;\n \n-\tkfunc_id = find_ksym_btf_id(obj, ext-\u003eessent_name ?: ext-\u003ename, BTF_KIND_FUNC, \u0026kern_btf,\n-\t\t\t\t \u0026mod_btf);\n+\tkern_name = kern_extern_name(ext-\u003eessent_name ?: ext-\u003ename);\n+\n+\tkfunc_id = find_ksym_btf_id(obj, kern_name, BTF_KIND_FUNC, \u0026kern_btf, \u0026mod_btf);\n \tif (kfunc_id \u003c 0) {\n \t\tif (kfunc_id == -ESRCH \u0026\u0026 ext-\u003eis_weak)\n \t\t\treturn 0;\n \t\tpr_warn(\"extern (func ksym) '%s': not found in kernel or module BTFs\\n\",\n-\t\t\text-\u003ename);\n+\t\t\tstrcmp(kern_name, \"bpf_unwind_resume\") ? ext-\u003ename : kern_name);\n \t\treturn kfunc_id;\n \t}\n \n@@ -9681,6 +9969,9 @@ void bpf_object__close(struct bpf_object *obj)\n \tzfree(\u0026obj-\u003ejumptables_data);\n \tobj-\u003ejumptables_data_sz = 0;\n \n+\tzfree(\u0026obj-\u003ecleanup_recs);\n+\tobj-\u003ecleanup_rec_cnt = 0;\n+\n \tfor (i = 0; i \u003c obj-\u003ejumptable_map_cnt; i++)\n \t\tclose(obj-\u003ejumptable_maps[i].fd);\n \tzfree(\u0026obj-\u003ejumptable_maps);\n@@ -10073,6 +10364,25 @@ int bpf_program__clone(struct bpf_program *prog, const struct bpf_prog_load_opts\n \t\tattr.line_info_rec_size = info ? info_rec_size : prog-\u003eline_info_rec_size;\n \t}\n \n+\t/* exception cleanup table */\n+\tinfo = OPTS_GET(opts, cleanup_info, NULL);\n+\tinfo_cnt = OPTS_GET(opts, cleanup_info_cnt, 0);\n+\tinfo_rec_size = OPTS_GET(opts, cleanup_info_rec_size, 0);\n+\tif (!!info != !!info_cnt || !!info != !!info_rec_size) {\n+\t\tpr_warn(\"prog '%s': cleanup_info, cleanup_info_cnt, and cleanup_info_rec_size must all be specified or all omitted\\n\",\n+\t\t\tprog-\u003ename);\n+\t\treturn libbpf_err(-EINVAL);\n+\t}\n+\tif (info) {\n+\t\tattr.cleanup_info = info;\n+\t\tattr.cleanup_info_cnt = info_cnt;\n+\t\tattr.cleanup_info_rec_size = info_rec_size;\n+\t} else if (prog-\u003ecleanup_info_cnt) {\n+\t\tattr.cleanup_info = prog-\u003ecleanup_info;\n+\t\tattr.cleanup_info_cnt = prog-\u003ecleanup_info_cnt;\n+\t\tattr.cleanup_info_rec_size = prog-\u003ecleanup_info_rec_size;\n+\t}\n+\n \t/* Logging is caller-controlled; no fallback to prog/obj log settings */\n \tattr.log_buf = OPTS_GET(opts, log_buf, NULL);\n \tattr.log_size = OPTS_GET(opts, log_size, 0);\ndiff --git a/tools/lib/bpf/libbpf_internal.h b/tools/lib/bpf/libbpf_internal.h\nindex cb4d962338441..78519f24fb404 100644\n--- a/tools/lib/bpf/libbpf_internal.h\n+++ b/tools/lib/bpf/libbpf_internal.h\n@@ -56,6 +56,9 @@\n #ifndef R_BPF_64_ABS32\n #define R_BPF_64_ABS32 3\n #endif\n+#ifndef R_BPF_64_NODYLD32\n+#define R_BPF_64_NODYLD32 4\n+#endif\n #ifndef R_BPF_64_32\n #define R_BPF_64_32 10\n #endif\n@@ -569,6 +572,13 @@ static inline void bpf_core_relo_bswap(struct bpf_core_relo *i)\n \ti-\u003ekind = bswap_32(i-\u003ekind);\n }\n \n+static inline void bpf_cleanup_info_bswap(struct bpf_cleanup_info *i)\n+{\n+\ti-\u003ebegin_off = bswap_32(i-\u003ebegin_off);\n+\ti-\u003eend_off = bswap_32(i-\u003eend_off);\n+\ti-\u003elanding_pad_off = bswap_32(i-\u003elanding_pad_off);\n+}\n+\n enum btf_field_iter_kind {\n \tBTF_FIELD_ITER_IDS,\n \tBTF_FIELD_ITER_STRS,\ndiff --git a/tools/lib/bpf/linker.c b/tools/lib/bpf/linker.c\nindex 78f92c39290af..fac7390bd109d 100644\n--- a/tools/lib/bpf/linker.c\n+++ b/tools/lib/bpf/linker.c\n@@ -1036,7 +1036,8 @@ static int linker_sanity_check_elf_relos(struct src_obj *obj, struct src_sec *se\n \t\tsize_t sym_type = ELF64_R_TYPE(relo-\u003er_info);\n \n \t\tif (sym_type != R_BPF_64_64 \u0026\u0026 sym_type != R_BPF_64_32 \u0026\u0026\n-\t\t sym_type != R_BPF_64_ABS64 \u0026\u0026 sym_type != R_BPF_64_ABS32) {\n+\t\t sym_type != R_BPF_64_ABS64 \u0026\u0026 sym_type != R_BPF_64_ABS32 \u0026\u0026\n+\t\t sym_type != R_BPF_64_NODYLD32) {\n \t\t\tpr_warn(\"ELF relo #%d in section #%zu has unexpected type %zu in %s\\n\",\n \t\t\t\ti, sec-\u003esec_idx, sym_type, obj-\u003efilename);\n \t\t\treturn -EINVAL;\n@@ -2274,6 +2275,39 @@ static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob\n \t\t\t\t\t\tinsn-\u003eimm += sec-\u003edst_off / sizeof(struct bpf_insn);\n \t\t\t\t\telse\n \t\t\t\t\t\tinsn-\u003eimm += sec-\u003edst_off;\n+\t\t\t\t} else if (sym_type == R_BPF_64_NODYLD32 ||\n+\t\t\t\t\t sym_type == R_BPF_64_ABS32) {\n+\t\t\t\t\t__u32 *val;\n+\n+\t\t\t\t\t/* Two spellings of the one thing: LLVM\n+\t\t\t\t\t * emits NODYLD32 for a .long against a\n+\t\t\t\t\t * section symbol, GNU as emits ABS32\n+\t\t\t\t\t * (bpf_reloc_type_lookup() maps\n+\t\t\t\t\t * BFD_RELOC_32 to it), and the value\n+\t\t\t\t\t * they describe is the same.\n+\t\t\t\t\t */\n+\n+\t\t\t\t\t/* An SHT_NOBITS section has no\n+\t\t\t\t\t * raw_data, and r_offset is sanity\n+\t\t\t\t\t * checked only for sections holding\n+\t\t\t\t\t * instructions. Check both here\n+\t\t\t\t\t * before writing.\n+\t\t\t\t\t */\n+\t\t\t\t\tif (!dst_linked_sec-\u003eraw_data ||\n+\t\t\t\t\t dst_rel-\u003er_offset % sizeof(*val) ||\n+\t\t\t\t\t dst_rel-\u003er_offset + sizeof(*val) \u003e\n+\t\t\t\t\t (size_t)dst_linked_sec-\u003esec_sz) {\n+\t\t\t\t\t\tpr_warn(\"ELF relo #%d in section #%zu points outside the data of section '%s' in %s\\n\",\n+\t\t\t\t\t\t\tj, src_sec-\u003esec_idx,\n+\t\t\t\t\t\t\tdst_linked_sec-\u003esec_name,\n+\t\t\t\t\t\t\tobj-\u003efilename);\n+\t\t\t\t\t\treturn -EINVAL;\n+\t\t\t\t\t}\n+\t\t\t\t\tval = dst_linked_sec-\u003eraw_data + dst_rel-\u003er_offset;\n+\t\t\t\t\tif (linker-\u003eswapped_endian)\n+\t\t\t\t\t\t*val = bswap_32(bswap_32(*val) + sec-\u003edst_off);\n+\t\t\t\t\telse\n+\t\t\t\t\t\t*val += sec-\u003edst_off;\n \t\t\t\t} else {\n \t\t\t\t\tpr_warn(\"relocation against STT_SECTION in non-exec section is not supported!\\n\");\n \t\t\t\t\treturn -EINVAL;\ndiff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile\nindex 7ea5ba1df29e3..981cc26430495 100644\n--- a/tools/testing/selftests/bpf/Makefile\n+++ b/tools/testing/selftests/bpf/Makefile\n@@ -523,7 +523,7 @@ LINKED_SKELS := test_static_linked.skel.h linked_funcs.skel.h\t\t\\\n LSKELS := fexit_sleep.c trace_printk.c trace_vprintk.c map_ptr_kern.c \t\\\n \tcore_kern.c core_kern_overflow.c test_ringbuf.c\t\t\t\\\n \ttest_ringbuf_n.c test_ringbuf_map_key.c test_ringbuf_write.c \\\n-\ttest_ringbuf_overwrite.c\n+\ttest_ringbuf_overwrite.c exceptions_cleanup_light.c\n \n LSKELS_SIGNED := fentry_test.c fexit_test.c atomics.c\n \ndiff --git a/tools/testing/selftests/bpf/exceptions_cleanup.h b/tools/testing/selftests/bpf/exceptions_cleanup.h\nnew file mode 100644\nindex 0000000000000..e14b651263b88\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/exceptions_cleanup.h\n@@ -0,0 +1,55 @@\n+/* SPDX-License-Identifier: GPL-2.0 */\n+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */\n+#ifndef __EXCEPTIONS_CLEANUP_H__\n+#define __EXCEPTIONS_CLEANUP_H__\n+\n+#define THROW_COOKIE\t\t0x100\n+#define INNER_COOKIE\t\t0x200\n+\n+/* progs/exceptions_cleanup.c: one bit per frame that reports it ran. */\n+#define RAN_FOO3_PREEMPT\t0x1\n+#define RAN_FOO2_RCU\t\t0x2\n+#define RAN_FOO1V_PREEMPT\t0x4\n+#define RAN_FOO2_DROP\t\t0x8\n+#define RAN_BUMP\t\t0x10\n+\n+/* progs/exceptions_cleanup_shapes.c: one bit per shape, numbered its own way. */\n+#define RAN_SWEEP\t\t0x1\n+#define RAN_SHARED\t\t0x2\n+#define RAN_REGS\t\t0x4\n+#define RAN_TAIL_CALL\t\t0x8\n+#define RAN_MAIN_PAD\t\t0x10\n+#define RAN_TC_TAKEN\t\t0x20\n+#define RAN_FREPLACE\t\t0x40\n+#define RAN_ADDR_TAKEN\t\t0x80\n+#define RAN_NO_SUBPROG\t\t0x100\n+#define RAN_PAD_CALLS\t\t0x200\n+#define RAN_PAD_FIRST\t\t0x400\n+#define RAN_WIDE_REC\t\t0x800\n+#define RAN_PAD_STACK\t\t0x1000\n+#define RAN_DEEP_PAD\t\t0x2000\n+#define RAN_NOUNWIND_REC\t0x4000\n+#define RAN_RESUME_ALIAS\t0x8000\n+#define RAN_PAD_TAIL_CALL\t0x10000\n+#define RAN_NOP_PAD\t\t0x20000\n+#define RAN_VAR_STACK\t\t0x40000\n+#define RAN_GLOBAL_PAD\t\t0x80000\n+\n+/* progs/exceptions_cleanup_light.c: the one pad it has. */\n+#define RAN_LIGHT\t\t0x1\n+\n+#define CLEANUP_REC(begin, end, landing_pad)\t\t\t\\\n+\t\".pushsection .bpf_cleanup,\\\"a\\\",@progbits;\"\t\t\\\n+\t\".long \" begin \";\"\t\t\t\t\t\\\n+\t\".long \" end \";\"\t\t\t\t\t\\\n+\t\".long \" landing_pad \";\"\t\t\t\t\\\n+\t\".popsection;\"\n+\n+/* Set a bit in @pads_ran. */\n+#define PAD_RAN(bit)\t\t\t\t\t\t\\\n+\t\"r1 = %[pads_ran] ll;\"\t\t\t\t\t\\\n+\t\"r2 = *(u64 *)(r1 + 0);\"\t\t\t\t\\\n+\t\"r2 |= \" bit \";\"\t\t\t\t\t\\\n+\t\"*(u64 *)(r1 + 0) = r2;\"\n+\n+#endif /* __EXCEPTIONS_CLEANUP_H__ */\ndiff --git a/tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c b/tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c\nnew file mode 100644\nindex 0000000000000..b69be77b2e26c\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c\n@@ -0,0 +1,454 @@\n+// SPDX-License-Identifier: GPL-2.0\n+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */\n+#include \u003ctest_progs.h\u003e\n+#include \"exceptions_cleanup.h\"\n+#include \"exceptions_cleanup.skel.h\"\n+#include \"exceptions_cleanup_fail.skel.h\"\n+#include \"exceptions_cleanup_shapes.skel.h\"\n+#include \"exceptions_cleanup_freplace.skel.h\"\n+#include \"exceptions_cleanup_pad_freplace.skel.h\"\n+#include \"exceptions_cleanup_ext_table.skel.h\"\n+#include \"exceptions_cleanup_light.lskel.h\"\n+\n+/* foo3 threw: every frame that has a pad ran it. */\n+#define PADS_FOO3_THREW \\\n+\t(RAN_FOO3_PREEMPT | RAN_FOO2_RCU | RAN_FOO1V_PREEMPT | RAN_FOO2_DROP)\n+\n+/* foo2 threw after foo3 returned normally: foo3's pad must not run. */\n+#define PADS_FOO2_THREW \\\n+\t(RAN_FOO2_RCU | RAN_FOO1V_PREEMPT | RAN_FOO2_DROP)\n+\n+static void run(struct exceptions_cleanup *skel, __u64 input, __u32 retval,\n+\t\t__u64 pads)\n+{\n+\t__u64 ctx = 0;\n+\tint err;\n+\n+\tLIBBPF_OPTS(bpf_test_run_opts, topts,\n+\t\t .ctx_in = \u0026ctx,\n+\t\t .ctx_size_in = sizeof(ctx),\n+\t);\n+\n+\tskel-\u003ebss-\u003einput = input;\n+\tskel-\u003ebss-\u003epads_ran = 0;\n+\tskel-\u003ebss-\u003eresult = 0;\n+\n+\terr = bpf_prog_test_run_opts(bpf_program__fd(skel-\u003eprogs.entry), \u0026topts);\n+\tif (!ASSERT_OK(err, \"run\"))\n+\t\treturn;\n+\tASSERT_EQ(topts.retval, retval, \"retval\");\n+\tASSERT_EQ(skel-\u003ebss-\u003epads_ran, pads | RAN_BUMP, \"pads_ran\");\n+}\n+\n+static void run_shape(struct exceptions_cleanup_shapes *skel, struct bpf_program *prog,\n+\t\t __u64 input, __u32 retval, __u64 pads)\n+{\n+\t__u64 ctx = 0;\n+\tint err;\n+\n+\tLIBBPF_OPTS(bpf_test_run_opts, topts,\n+\t\t .ctx_in = \u0026ctx,\n+\t\t .ctx_size_in = sizeof(ctx),\n+\t);\n+\n+\tskel-\u003ebss-\u003einput = input;\n+\tskel-\u003ebss-\u003epads_ran = 0;\n+\n+\terr = bpf_prog_test_run_opts(bpf_program__fd(prog), \u0026topts);\n+\tif (!ASSERT_OK(err, \"run\"))\n+\t\treturn;\n+\tASSERT_EQ(topts.retval, retval, \"retval\");\n+\tASSERT_EQ(skel-\u003ebss-\u003epads_ran, pads, \"pads_ran\");\n+}\n+\n+static void test_freplace(struct exceptions_cleanup_shapes *skel)\n+{\n+\tstruct exceptions_cleanup_freplace *fr;\n+\tstruct bpf_link *link;\n+\tint tgt_fd;\n+\n+\ttgt_fd = bpf_program__fd(skel-\u003eprogs.entry_freplace);\n+\n+\tfr = exceptions_cleanup_freplace__open();\n+\tif (!ASSERT_OK_PTR(fr, \"freplace open\"))\n+\t\treturn;\n+\n+\tif (!ASSERT_OK(bpf_program__set_attach_target(fr-\u003eprogs.new_fr_callee,\n+\t\t\t\t\t\t tgt_fd, \"fr_callee\"),\n+\t\t \"set_attach_target\"))\n+\t\tgoto out;\n+\tif (!ASSERT_OK(exceptions_cleanup_freplace__load(fr), \"freplace load\"))\n+\t\tgoto out;\n+\n+\tlink = bpf_program__attach_freplace(fr-\u003eprogs.new_fr_callee, tgt_fd,\n+\t\t\t\t\t \"fr_callee\");\n+\tif (!ASSERT_OK_PTR(link, \"attach_freplace\"))\n+\t\tgoto out;\n+\n+\trun_shape(skel, skel-\u003eprogs.entry_freplace, 101, THROW_COOKIE, 0);\n+\tbpf_link__destroy(link);\n+out:\n+\texceptions_cleanup_freplace__destroy(fr);\n+}\n+\n+static void test_pad_calls_freplace(struct exceptions_cleanup_shapes *skel)\n+{\n+\tstruct exceptions_cleanup_pad_freplace *fr;\n+\tstruct bpf_link *link;\n+\t__u64 ctx = 0;\n+\tint tgt_fd, err;\n+\n+\tLIBBPF_OPTS(bpf_test_run_opts, topts,\n+\t\t .ctx_in = \u0026ctx,\n+\t\t .ctx_size_in = sizeof(ctx),\n+\t);\n+\n+\ttgt_fd = bpf_program__fd(skel-\u003eprogs.entry_pad_calls);\n+\n+\tfr = exceptions_cleanup_pad_freplace__open();\n+\tif (!ASSERT_OK_PTR(fr, \"pad freplace open\"))\n+\t\treturn;\n+\n+\tif (!ASSERT_OK(bpf_program__set_attach_target(fr-\u003eprogs.new_pad_callee,\n+\t\t\t\t\t\t tgt_fd, \"pad_callee\"),\n+\t\t \"set_attach_target\"))\n+\t\tgoto out;\n+\tif (!ASSERT_OK(exceptions_cleanup_pad_freplace__load(fr), \"pad freplace load\"))\n+\t\tgoto out;\n+\n+\tlink = bpf_program__attach_freplace(fr-\u003eprogs.new_pad_callee, tgt_fd,\n+\t\t\t\t\t \"pad_callee\");\n+\tif (!ASSERT_OK_PTR(link, \"attach_freplace\"))\n+\t\tgoto out;\n+\n+\tskel-\u003ebss-\u003einput = 101;\n+\tskel-\u003ebss-\u003epads_ran = 0;\n+\tskel-\u003ebss-\u003epad_runs = 0;\n+\n+\terr = bpf_prog_test_run_opts(tgt_fd, \u0026topts);\n+\tif (!ASSERT_OK(err, \"run\"))\n+\t\tgoto out_link;\n+\n+\tASSERT_EQ(skel-\u003ebss-\u003epad_runs, 1, \"pad_runs\");\n+\tASSERT_EQ(skel-\u003ebss-\u003epads_ran, RAN_PAD_CALLS, \"pads_ran\");\n+\tASSERT_EQ(topts.retval, THROW_COOKIE, \"retval\");\n+out_link:\n+\tbpf_link__destroy(link);\n+out:\n+\texceptions_cleanup_pad_freplace__destroy(fr);\n+}\n+\n+static void test_ext_table(struct exceptions_cleanup_shapes *skel)\n+{\n+\tstruct exceptions_cleanup_ext_table *fr;\n+\tstruct bpf_link *link;\n+\tint tgt_fd;\n+\n+\ttgt_fd = bpf_program__fd(skel-\u003eprogs.entry_freplace);\n+\n+\tfr = exceptions_cleanup_ext_table__open();\n+\tif (!ASSERT_OK_PTR(fr, \"ext table open\"))\n+\t\treturn;\n+\n+\tif (!ASSERT_OK(bpf_program__set_attach_target(fr-\u003eprogs.new_fr_callee,\n+\t\t\t\t\t\t tgt_fd, \"fr_callee\"),\n+\t\t \"set_attach_target\"))\n+\t\tgoto out;\n+\tif (!ASSERT_OK(exceptions_cleanup_ext_table__load(fr), \"ext table load\"))\n+\t\tgoto out;\n+\n+\tlink = bpf_program__attach_freplace(fr-\u003eprogs.new_fr_callee, tgt_fd,\n+\t\t\t\t\t \"fr_callee\");\n+\tif (!ASSERT_OK_PTR(link, \"attach_freplace\"))\n+\t\tgoto out;\n+\n+\tfr-\u003ebss-\u003eext_pad_ran = 0;\n+\trun_shape(skel, skel-\u003eprogs.entry_freplace, 101, THROW_COOKIE, 0);\n+\tASSERT_EQ(fr-\u003ebss-\u003eext_pad_ran, 1, \"ext_pad_ran\");\n+\n+\tbpf_link__destroy(link);\n+out:\n+\texceptions_cleanup_ext_table__destroy(fr);\n+}\n+\n+static void test_light_skeleton(void)\n+{\n+\tstruct exceptions_cleanup_light_lskel *skel;\n+\t__u64 ctx = 0;\n+\tint err;\n+\n+\tLIBBPF_OPTS(bpf_test_run_opts, topts,\n+\t\t .ctx_in = \u0026ctx,\n+\t\t .ctx_size_in = sizeof(ctx),\n+\t);\n+\n+\tskel = exceptions_cleanup_light_lskel__open_and_load();\n+\tif (!ASSERT_OK_PTR(skel, \"light open_and_load\"))\n+\t\treturn;\n+\n+\terr = bpf_prog_test_run_opts(skel-\u003eprogs.entry_light.prog_fd, \u0026topts);\n+\tif (!ASSERT_OK(err, \"run\"))\n+\t\tgoto out;\n+\tASSERT_EQ(topts.retval, THROW_COOKIE, \"retval\");\n+\tASSERT_EQ(skel-\u003ebss-\u003epads_ran, RAN_LIGHT, \"pads_ran\");\n+out:\n+\texceptions_cleanup_light_lskel__destroy(skel);\n+}\n+\n+static void test_shapes(void)\n+{\n+\tstruct exceptions_cleanup_shapes *skel;\n+\n+\tskel = exceptions_cleanup_shapes__open_and_load();\n+\tif (!ASSERT_OK_PTR(skel, \"shapes open_and_load\"))\n+\t\treturn;\n+\n+\t/* The frame loads at all only if everything unreachable in it went. */\n+\tif (test__start_subtest(\"sweep_no_throw\"))\n+\t\trun_shape(skel, skel-\u003eprogs.entry_sweep, 1, 0, 0);\n+\tif (test__start_subtest(\"sweep_throw\"))\n+\t\trun_shape(skel, skel-\u003eprogs.entry_sweep, 101, THROW_COOKIE, RAN_SWEEP);\n+\n+\t/* The covered call unwinds to the pad; the uncovered one walks past. */\n+\tif (test__start_subtest(\"shared_callee_no_throw\")) {\n+\t\tskel-\u003ebss-\u003eouter_input = 0;\n+\t\trun_shape(skel, skel-\u003eprogs.entry_shared, 1, 2, 0);\n+\t}\n+\tif (test__start_subtest(\"shared_callee_throw\")) {\n+\t\tskel-\u003ebss-\u003eouter_input = 0;\n+\t\trun_shape(skel, skel-\u003eprogs.entry_shared, 101, THROW_COOKIE, RAN_SHARED);\n+\t}\n+\tif (test__start_subtest(\"shared_callee_uncovered_throw\")) {\n+\t\tskel-\u003ebss-\u003eouter_input = 101;\n+\t\trun_shape(skel, skel-\u003eprogs.entry_shared, 1, THROW_COOKIE, 0);\n+\t\tskel-\u003ebss-\u003eouter_input = 0;\n+\t}\n+\n+\t/* The pad only sets its bit if it got the frame's own r6-r9 back. */\n+\tif (test__start_subtest(\"pad_sees_callee_saved\"))\n+\t\trun_shape(skel, skel-\u003eprogs.entry_regs, 101, THROW_COOKIE, RAN_REGS);\n+\n+\t/* Same check, with a tail-call-reachable callee: its spill moves. */\n+\tif (test__start_subtest(\"tail_call_no_throw\"))\n+\t\trun_shape(skel, skel-\u003eprogs.entry_tail_call, 1, 0, 0);\n+\tif (test__start_subtest(\"tail_call_throw\"))\n+\t\trun_shape(skel, skel-\u003eprogs.entry_tail_call, 101, THROW_COOKIE,\n+\t\t\t RAN_TAIL_CALL);\n+\n+\t/* A region around a nounwind call: no pad dispatched, still loads. */\n+\tif (test__start_subtest(\"nounwind_region\"))\n+\t\trun_shape(skel, skel-\u003eprogs.entry_nounwind_rec, 1, 0, 0);\n+\n+\t/* A pad in the main program's own frame, not in a subprogram. */\n+\tif (test__start_subtest(\"main_program_pad\"))\n+\t\trun_shape(skel, skel-\u003eprogs.entry_main_pad, 101, THROW_COOKIE,\n+\t\t\t RAN_MAIN_PAD);\n+\n+\t/* The same call site either way: the subprogram's throw unwinds into\n+\t * this frame and runs its pad, an extension's stops at its own boundary.\n+\t */\n+\tif (test__start_subtest(\"freplace_subprog_throws\"))\n+\t\trun_shape(skel, skel-\u003eprogs.entry_freplace, 7, THROW_COOKIE,\n+\t\t\t RAN_FREPLACE);\n+\tif (test__start_subtest(\"freplace_extension_throws\"))\n+\t\ttest_freplace(skel);\n+\n+\t/* A tail call that is taken: the walk ends at the target, so the cookie\n+\t * comes back from there and this frame's pad does not run -- though it\n+\t * is reachable, so a walk past the boundary would find it.\n+\t */\n+\tif (test__start_subtest(\"tail_call_taken\")) {\n+\t\tint key = 0, prog_fd = bpf_program__fd(skel-\u003eprogs.tc_target);\n+\n+\t\tif (ASSERT_OK(bpf_map_update_elem(bpf_map__fd(skel-\u003emaps.taken_table),\n+\t\t\t\t\t\t \u0026key, \u0026prog_fd, BPF_ANY),\n+\t\t\t \"populate taken_table\"))\n+\t\t\trun_shape(skel, skel-\u003eprogs.entry_tail_taken, 101,\n+\t\t\t\t THROW_COOKIE, 0);\n+\t}\n+\n+\t/* A throwing subprog named by a BPF_PSEUDO_FUNC and handed to a\n+\t * bpf_loop() the verifier never reaches: the callback check has to fire\n+\t * on the helper call, not on the ld_imm64.\n+\t */\n+\tif (test__start_subtest(\"addr_taken_no_throw\"))\n+\t\trun_shape(skel, skel-\u003eprogs.entry_addr_taken, 1, 2, 0);\n+\tif (test__start_subtest(\"addr_taken_throw\"))\n+\t\trun_shape(skel, skel-\u003eprogs.entry_addr_taken, 101, THROW_COOKIE,\n+\t\t\t RAN_ADDR_TAKEN);\n+\n+\t/* A record covering bpf_throw() itself rather than a call to a frame\n+\t * that throws: raised, caught up with and delivered in one frame.\n+\t */\n+\tif (test__start_subtest(\"no_subprog_no_throw\"))\n+\t\trun_shape(skel, skel-\u003eprogs.entry_no_subprog, 1, 0, 0);\n+\tif (test__start_subtest(\"no_subprog_throw\"))\n+\t\trun_shape(skel, skel-\u003eprogs.entry_no_subprog, 101, THROW_COOKIE,\n+\t\t\t RAN_NO_SUBPROG);\n+\n+\t/* A pad that calls a subprogram; with a throwing extension in its place,\n+\t * the nested exception has to stop there, not restart this pad.\n+\t */\n+\tif (test__start_subtest(\"pad_calls_subprog\")) {\n+\t\tskel-\u003ebss-\u003epad_runs = 0;\n+\t\trun_shape(skel, skel-\u003eprogs.entry_pad_calls, 101, THROW_COOKIE,\n+\t\t\t RAN_PAD_CALLS);\n+\t\tASSERT_EQ(skel-\u003ebss-\u003epad_runs, 1, \"pad_runs\");\n+\t}\n+\tif (test__start_subtest(\"pad_calls_throwing_extension\"))\n+\t\ttest_pad_calls_freplace(skel);\n+\n+\t/* A covered throw the sweep leaves last, where the default exception\n+\t * callback is patched in; the pad's bit needs r6-r9 still spilled.\n+\t */\n+\tif (test__start_subtest(\"pad_before_throw\"))\n+\t\trun_shape(skel, skel-\u003eprogs.entry_pad_first, 101, THROW_COOKIE,\n+\t\t\t RAN_PAD_FIRST);\n+\n+\t/* A region whose last instruction is a 16-byte one, so that end - 1\n+\t * names the half of it that is not an instruction.\n+\t */\n+\tif (test__start_subtest(\"region_ends_on_ldimm64\"))\n+\t\trun_shape(skel, skel-\u003eprogs.entry_wide_rec, 101, THROW_COOKIE,\n+\t\t\t RAN_WIDE_REC);\n+\n+\t/* A pad that reloads from and writes to its own frame's stack, which a\n+\t * JIT addressing the frame through the stack pointer gets wrong.\n+\t */\n+\tif (test__start_subtest(\"pad_uses_own_frame\"))\n+\t\trun_shape(skel, skel-\u003eprogs.entry_pad_stack, 101, THROW_COOKIE,\n+\t\t\t RAN_PAD_STACK);\n+\n+\t/* The same, with an uncovered frame between the throw and the pad. */\n+\tif (test__start_subtest(\"pad_two_frames_up\"))\n+\t\trun_shape(skel, skel-\u003eprogs.entry_deep_pad, 101, THROW_COOKIE,\n+\t\t\t RAN_DEEP_PAD);\n+\n+\t/* An extension program with a cleanup table of its own. */\n+\tif (test__start_subtest(\"extension_carries_table\"))\n+\t\ttest_ext_table(skel);\n+\n+\t/* A pad terminated by _Unwind_Resume, which libbpf maps onto the kfunc;\n+\t * every other program here calls bpf_unwind_resume directly.\n+\t */\n+\tif (test__start_subtest(\"resume_alias\"))\n+\t\trun_shape(skel, skel-\u003eprogs.entry_resume_alias, 101,\n+\t\t\t THROW_COOKIE, RAN_RESUME_ALIAS);\n+\n+\t/* A pad that calls a subprogram which tail calls, array empty and then\n+\t * populated: the tail call releases only the callee's own prologue.\n+\t */\n+\tif (test__start_subtest(\"pad_callee_tail_call\")) {\n+\t\tint key = 0, prog_fd = bpf_program__fd(skel-\u003eprogs.pad_tc_target);\n+\n+\t\tskel-\u003ebss-\u003epad_tc_target_ran = 0;\n+\t\tskel-\u003ebss-\u003epad_runs = 0;\n+\t\trun_shape(skel, skel-\u003eprogs.entry_pad_tail_call, 101,\n+\t\t\t THROW_COOKIE, RAN_PAD_TAIL_CALL);\n+\t\tASSERT_EQ(skel-\u003ebss-\u003epad_tc_target_ran, 0, \"target not run\");\n+\t\tASSERT_EQ(skel-\u003ebss-\u003epad_runs, 1, \"pad_runs\");\n+\n+\t\tif (ASSERT_OK(bpf_map_update_elem(bpf_map__fd(skel-\u003emaps.pad_tc_table),\n+\t\t\t\t\t\t \u0026key, \u0026prog_fd, BPF_ANY),\n+\t\t\t \"populate pad_tc_table\")) {\n+\t\t\tskel-\u003ebss-\u003epad_runs = 0;\n+\t\t\trun_shape(skel, skel-\u003eprogs.entry_pad_tail_call, 101,\n+\t\t\t\t THROW_COOKIE, RAN_PAD_TAIL_CALL);\n+\t\t\tASSERT_EQ(skel-\u003ebss-\u003epad_tc_target_ran, 1, \"target ran\");\n+\t\t\tASSERT_EQ(skel-\u003ebss-\u003epad_runs, 1, \"pad_runs\");\n+\t\t}\n+\t}\n+\n+\t/* The same, into a target that carries a table and throws: that target\n+\t * is a boundary, so the outer pad runs once, not twice.\n+\t */\n+\tif (test__start_subtest(\"pad_callee_tail_call_throws\")) {\n+\t\tint key = 0, prog_fd = bpf_program__fd(skel-\u003eprogs.pad_tc_throw_target);\n+\n+\t\tif (ASSERT_OK(bpf_map_update_elem(bpf_map__fd(skel-\u003emaps.pad_tc_table),\n+\t\t\t\t\t\t \u0026key, \u0026prog_fd, BPF_ANY),\n+\t\t\t \"populate pad_tc_table\")) {\n+\t\t\tskel-\u003ebss-\u003epad_runs = 0;\n+\t\t\tskel-\u003ebss-\u003etc_target_pad_runs = 0;\n+\t\t\trun_shape(skel, skel-\u003eprogs.entry_pad_tail_call, 101,\n+\t\t\t\t THROW_COOKIE, RAN_PAD_TAIL_CALL);\n+\t\t\t/* The target cleaned up after itself, once. */\n+\t\t\tASSERT_EQ(skel-\u003ebss-\u003etc_target_pad_runs, 1,\n+\t\t\t\t \"tc_target_pad_runs\");\n+\t\t\t/* And the outer pad was not started over. */\n+\t\t\tASSERT_EQ(skel-\u003ebss-\u003epad_runs, 1, \"pad_runs\");\n+\t\t}\n+\t}\n+\n+\t/* A pad whose first instruction opt_remove_nops() deletes: the record\n+\t * has to follow the pad rather than be dropped with the nop.\n+\t */\n+\tif (test__start_subtest(\"nop_at_pad_head\"))\n+\t\trun_shape(skel, skel-\u003eprogs.entry_nop_pad, 101, THROW_COOKIE,\n+\t\t\t RAN_NOP_PAD);\n+\n+\t/* A pad that indexes its frame's stack by a register the frame set\n+\t * before the throwing call, marked precise back across the unwind edge.\n+\t */\n+\tif (test__start_subtest(\"pad_var_stack_offset\"))\n+\t\trun_shape(skel, skel-\u003eprogs.entry_var_stack, 101, THROW_COOKIE,\n+\t\t\t RAN_VAR_STACK);\n+\n+\t/* The same, where the throw is in a global subprogram, which the\n+\t * verifier walks without a frame of its own.\n+\t */\n+\tif (test__start_subtest(\"pad_over_global_subprog\"))\n+\t\trun_shape(skel, skel-\u003eprogs.entry_global_pad, 101, THROW_COOKIE,\n+\t\t\t RAN_GLOBAL_PAD);\n+\n+\texceptions_cleanup_shapes__destroy(skel);\n+}\n+\n+void test_exceptions_cleanup(void)\n+{\n+\tchar log[8192] = {};\n+\n+\tLIBBPF_OPTS(bpf_object_open_opts, opts,\n+\t\t .kernel_log_buf = log,\n+\t\t .kernel_log_size = sizeof(log));\n+\tstruct exceptions_cleanup *skel;\n+\tint err;\n+\n+\tskel = exceptions_cleanup__open_opts(\u0026opts);\n+\tif (!ASSERT_OK_PTR(skel, \"open\"))\n+\t\treturn;\n+\n+\terr = exceptions_cleanup__load(skel);\n+\tif (err) {\n+\t\tif (err == -EOPNOTSUPP \u0026\u0026\n+\t\t strstr(log, \"exception cleanup needs a JIT that can dispatch landing pads\"))\n+\t\t\ttest__skip();\n+\t\telse if (!ASSERT_OK(err, \"load\"))\n+\t\t\tfprintf(stderr, \"%s\", log);\n+\t\texceptions_cleanup__destroy(skel);\n+\t\treturn;\n+\t}\n+\n+\t/* No throw: foo3 returns 1 ^ 1 == 0, foo2 adds one, no pad runs. */\n+\tif (test__start_subtest(\"no_throw\"))\n+\t\trun(skel, 1, 1, 0);\n+\n+\t/* foo3 throws; every pad runs and the cookie is delivered at entry. */\n+\tif (test__start_subtest(\"throw_from_foo3\"))\n+\t\trun(skel, 101, THROW_COOKIE, PADS_FOO3_THREW);\n+\n+\t/* foo3 returns 2 ^ 1 == 3, so foo2 throws from its own second region;\n+\t * foo3's frame is long gone, so its pad must not run.\n+\t */\n+\tif (test__start_subtest(\"throw_from_foo2\"))\n+\t\trun(skel, 2, THROW_COOKIE, PADS_FOO2_THREW);\n+\n+\texceptions_cleanup__destroy(skel);\n+\n+\tif (test__start_subtest(\"light_skeleton\"))\n+\t\ttest_light_skeleton();\n+\n+\ttest_shapes();\n+\n+\tRUN_TESTS(exceptions_cleanup_fail);\n+}\ndiff --git a/tools/testing/selftests/bpf/progs/exceptions_cleanup.c b/tools/testing/selftests/bpf/progs/exceptions_cleanup.c\nnew file mode 100644\nindex 0000000000000..95199a2828faa\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/progs/exceptions_cleanup.c\n@@ -0,0 +1,152 @@\n+// SPDX-License-Identifier: GPL-2.0\n+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */\n+#include \u003cvmlinux.h\u003e\n+#include \u003cbpf/bpf_helpers.h\u003e\n+#include \"bpf_misc.h\"\n+#include \"exceptions_cleanup.h\"\n+\n+static __used __noinline void __kfunc_btf_anchor(void)\n+{\n+\tbpf_throw(0);\n+\tbpf_rcu_read_lock();\n+\tbpf_rcu_read_unlock();\n+\tbpf_preempt_disable();\n+\tbpf_preempt_enable();\n+\tbpf_unwind_resume();\n+}\n+\n+__u64 input = 0;\n+__u64 pads_ran = 0;\n+__u64 result = 0;\n+\n+static __used __noinline __u64 foo3(__u64 x)\n+{\n+\tbpf_preempt_disable();\n+\tif (x \u003e 100)\n+\t\tasm volatile (\n+\t\t\"r1 = %[cookie];\"\n+\t\"1:\"\t\"call bpf_throw;\"\t\t/* cleanup region */\n+\t\"2:\"\n+\t\t\"goto 3f;\"\n+\t\"4:\"\t\t\t\t\t/* landing pad */\n+\t\t\"r7 = r0;\"\n+\t\t\"call bpf_preempt_enable;\"\n+\t\tPAD_RAN(\"%[ran]\")\n+\t\t\"r1 = r7;\"\n+\t\t\"call bpf_unwind_resume;\"\n+\t\"3:\"\n+\t\tCLEANUP_REC(\"1b\", \"2b\", \"4b\")\n+\t\t:\n+\t\t: [cookie]\"i\"(THROW_COOKIE), [ran]\"i\"(RAN_FOO3_PREEMPT),\n+\t\t __imm_addr(pads_ran)\n+\t\t: __clobber_all);\n+\tbpf_preempt_enable();\n+\treturn x ^ 1;\n+}\n+\n+__u64 never = 0;\n+\n+static __used __naked __noinline void drop_glue(void)\n+{\n+\tasm volatile (\n+\tPAD_RAN(\"%[ran]\")\n+\t\"exit;\"\n+\t:\n+\t: [ran]\"i\"(RAN_FOO2_DROP), __imm_addr(pads_ran)\n+\t: __clobber_all);\n+}\n+\n+static __used __naked __noinline __u64 foo2(void)\n+{\n+\tasm volatile (\n+\t\"r6 = r1;\"\n+\t\"call bpf_rcu_read_lock;\"\n+\t\"r1 = r6;\"\n+\"1:\"\t\"call foo3;\"\t\t\t/* cleanup region #1 */\n+\"2:\"\n+\t\"r6 = r0;\"\n+\t\"if r6 == 0 goto 5f;\"\n+\t\"r1 = %[cookie];\"\n+\"3:\"\t\"call bpf_throw;\"\t\t/* cleanup region #2 */\n+\"4:\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\"5:\"\n+\t\"call bpf_rcu_read_unlock;\"\n+\t\"r0 = r6;\"\n+\t\"r0 += 1;\"\n+\t\"exit;\"\n+\"6:\"\t\t\t\t\t/* landing pad, shared by both regions */\n+\t\"call drop_glue;\"\n+\t\"call bpf_rcu_read_unlock;\"\n+\tPAD_RAN(\"%[ran_rcu]\")\n+\t\"call bpf_unwind_resume;\"\n+\t\"exit;\"\n+\tCLEANUP_REC(\"1b\", \"2b\", \"6b\")\n+\tCLEANUP_REC(\"3b\", \"4b\", \"6b\")\n+\t:\n+\t: [cookie]\"i\"(THROW_COOKIE), [ran_rcu]\"i\"(RAN_FOO2_RCU),\n+\t __imm_addr(pads_ran)\n+\t: __clobber_all);\n+}\n+\n+static __used __naked __noinline void foo1v(void)\n+{\n+\tasm volatile (\n+\t\"call bpf_preempt_disable;\"\n+\t\"r1 = %[input] ll;\"\n+\t\"r1 = *(u64 *)(r1 + 0);\"\n+\"1:\"\t\"call foo2;\"\t\t\t/* cleanup region */\n+\"2:\"\n+\t\"r6 = r0;\"\n+\t\"call bpf_preempt_enable;\"\n+\t\"r1 = %[result] ll;\"\n+\t\"*(u64 *)(r1 + 0) = r6;\"\n+\t\"goto 7f;\"\n+\"8:\"\t\t\t\t\t/* landing pad */\n+\t\"call bpf_preempt_enable;\"\n+\tPAD_RAN(\"%[ran]\")\n+\t\"goto 9f;\"\n+\"7:\"\t\t\t\t\t/* the frame's own exit block */\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\"9:\"\t\t\t\t\t/* shared resume block */\n+\t\"call bpf_unwind_resume;\"\n+\t\"exit;\"\n+\tCLEANUP_REC(\"1b\", \"2b\", \"8b\")\n+\t:\n+\t: [ran]\"i\"(RAN_FOO1V_PREEMPT), __imm_addr(input),\n+\t __imm_addr(result), __imm_addr(pads_ran)\n+\t: __clobber_all);\n+}\n+\n+static __used __naked __noinline void bump(void)\n+{\n+\tasm volatile (\n+\tPAD_RAN(\"%[ran]\")\n+\t\"r1 = %[never] ll;\"\n+\t\"r1 = *(u64 *)(r1 + 0);\"\n+\t\"if r1 == 0 goto 1f;\"\n+\t\"r1 = 0;\"\n+\t\"call bpf_throw;\"\n+\"1:\"\n+\t\"exit;\"\t\t\t\t/* r0 deliberately left alone */\n+\t:\n+\t: [ran]\"i\"(RAN_BUMP), __imm_addr(never), __imm_addr(pads_ran)\n+\t: __clobber_all);\n+}\n+\n+__noinline __u64 foo1(void)\n+{\n+\tbump();\n+\tfoo1v();\n+\treturn result;\n+}\n+\n+SEC(\"syscall\")\n+int entry(void *ctx)\n+{\n+\treturn foo1();\n+}\n+\n+char _license[] SEC(\"license\") = \"GPL\";\ndiff --git a/tools/testing/selftests/bpf/progs/exceptions_cleanup_ext_table.c b/tools/testing/selftests/bpf/progs/exceptions_cleanup_ext_table.c\nnew file mode 100644\nindex 0000000000000..d14db48d6b29b\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/progs/exceptions_cleanup_ext_table.c\n@@ -0,0 +1,48 @@\n+// SPDX-License-Identifier: GPL-2.0\n+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */\n+#include \u003cvmlinux.h\u003e\n+#include \u003cbpf/bpf_helpers.h\u003e\n+#include \"bpf_misc.h\"\n+#include \"exceptions_cleanup.h\"\n+\n+__u64 ext_pad_ran = 0;\n+\n+/* Without a 32-bit int in BTF, libbpf's dummy_ksym var gets type id 0. */\n+int btf_int_anchor;\n+\n+static __used __noinline void __kfunc_btf_anchor(void)\n+{\n+\tbpf_throw(0);\n+\tbpf_preempt_disable();\n+\tbpf_preempt_enable();\n+\tbpf_unwind_resume();\n+}\n+\n+static __used __naked __noinline __u64 ext_frame(void)\n+{\n+\tasm volatile (\n+\t\"call bpf_preempt_disable;\"\n+\t\"r1 = %[cookie];\"\n+\"1:\"\t\"call bpf_throw;\"\t\t/* cleanup region */\n+\"2:\"\n+\t\"exit;\"\n+\"3:\"\t\t\t\t\t/* landing pad */\n+\t\"call bpf_preempt_enable;\"\n+\t\"r1 = %[ext_pad_ran] ll;\"\n+\t\"r2 = 1;\"\n+\t\"*(u64 *)(r1 + 0) = r2;\"\n+\t\"call bpf_unwind_resume;\"\n+\t\"exit;\"\n+\tCLEANUP_REC(\"1b\", \"2b\", \"3b\")\n+\t:\n+\t: [cookie]\"i\"(THROW_COOKIE), __imm_addr(ext_pad_ran)\n+\t: __clobber_all);\n+}\n+\n+SEC(\"freplace/fr_callee\")\n+__u64 new_fr_callee(__u64 x)\n+{\n+\treturn ext_frame();\n+}\n+\n+char _license[] SEC(\"license\") = \"GPL\";\ndiff --git a/tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c b/tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c\nnew file mode 100644\nindex 0000000000000..db6ac7d8bd6de\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c\n@@ -0,0 +1,662 @@\n+// SPDX-License-Identifier: GPL-2.0\n+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */\n+#include \u003cvmlinux.h\u003e\n+#include \u003cbpf/bpf_helpers.h\u003e\n+#include \"bpf_experimental.h\"\n+#include \"bpf_misc.h\"\n+#include \"../test_kmods/bpf_testmod_kfunc.h\"\n+#include \"exceptions_cleanup.h\"\n+\n+__u64 input = 0;\n+\n+static __used __noinline void __kfunc_btf_anchor(void)\n+{\n+\tbpf_throw(0);\n+\tbpf_preempt_disable();\n+\tbpf_preempt_enable();\n+\tbpf_unwind_resume();\n+}\n+\n+/* 1. A subprogram that may unwind, also used as a helper callback:\n+ * bpf_loop()'s own kernel frame would end the walk before it found a\n+ * boundary.\n+ */\n+static int throwing_cb(__u32 idx, void *ctx)\n+{\n+\tbpf_throw(0xbad);\n+\treturn 0;\n+}\n+\n+static __used __naked __noinline __u64 cb_frame(void)\n+{\n+\tasm volatile (\n+\t\"call bpf_preempt_disable;\"\n+\"1:\"\t\"call throwing_cb;\"\t\t/* cleanup region */\n+\"2:\"\n+\t\"call bpf_preempt_enable;\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\"3:\"\t\t\t\t\t/* landing pad */\n+\t\"call bpf_preempt_enable;\"\n+\t\"call bpf_unwind_resume;\"\n+\t\"exit;\"\n+\tCLEANUP_REC(\"1b\", \"2b\", \"3b\")\n+\t::: __clobber_all);\n+}\n+\n+SEC(\"?syscall\")\n+__failure __msg(\"may unwind and is used as a callback\")\n+int callback_may_unwind(void *ctx)\n+{\n+\tbpf_loop(1, throwing_cb, NULL, 0);\n+\treturn cb_frame();\n+}\n+\n+/* 2. A landing pad that reaches both an unwind resume and a plain exit, so\n+ * nothing says whether it is a cleanup pad or a catch pad.\n+ */\n+static __used __naked __noinline __u64 inner_throw(void)\n+{\n+\tasm volatile (\n+\t\"r1 = 1;\"\n+\t\"call bpf_throw;\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t::: __clobber_all);\n+}\n+\n+static __used __naked __noinline __u64 ambiguous_pad_frame(void)\n+{\n+\tasm volatile (\n+\t\"r6 = r1;\"\n+\t\"call bpf_preempt_disable;\"\n+\"1:\"\t\"call inner_throw;\"\t\t/* cleanup region */\n+\"2:\"\n+\t\"call bpf_preempt_enable;\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\"3:\"\t\t\t\t\t/* landing pad: two ways out */\n+\t\"call bpf_preempt_enable;\"\n+\t\"if r6 \u003e 10 goto 4f;\"\n+\t\"call bpf_unwind_resume;\"\n+\t\"exit;\"\n+\"4:\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\tCLEANUP_REC(\"1b\", \"2b\", \"3b\")\n+\t::: __clobber_all);\n+}\n+\n+SEC(\"?syscall\")\n+__failure __msg(\"reaches both bpf_unwind_resume() and a plain exit\")\n+int ambiguous_landing_pad(void *ctx)\n+{\n+\treturn ambiguous_pad_frame();\n+}\n+\n+/* 3. A throw from inside a landing pad: a second walk over the frames the\n+ * first one is in the middle of discarding.\n+ */\n+static __used __naked __noinline __u64 throw_in_pad_frame(void)\n+{\n+\tasm volatile (\n+\t\"call bpf_preempt_disable;\"\n+\"1:\"\t\"call inner_throw;\"\t\t/* cleanup region */\n+\"2:\"\n+\t\"call bpf_preempt_enable;\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\"3:\"\t\t\t\t\t/* landing pad that throws again */\n+\t\"call bpf_preempt_enable;\"\n+\t\"r1 = 2;\"\n+\t\"call bpf_throw;\"\n+\t\"call bpf_unwind_resume;\"\n+\t\"exit;\"\n+\tCLEANUP_REC(\"1b\", \"2b\", \"3b\")\n+\t::: __clobber_all);\n+}\n+\n+SEC(\"?syscall\")\n+__failure __msg(\"can throw while an exception is in flight\")\n+int throw_from_landing_pad(void *ctx)\n+{\n+\treturn throw_in_pad_frame();\n+}\n+\n+/* 4. A cleanup table in a program that also installs an exception callback,\n+ * two different answers to what runs on the way out.\n+ */\n+__noinline int unused_exc_cb(u64 cookie)\n+{\n+\treturn 0;\n+}\n+\n+static __used __naked __noinline __u64 cb_and_table_frame(void)\n+{\n+\tasm volatile (\n+\t\"call bpf_preempt_disable;\"\n+\t\"r1 = 9;\"\n+\"1:\"\t\"call bpf_throw;\"\t\t/* cleanup region */\n+\"2:\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\"3:\"\t\t\t\t\t/* landing pad */\n+\t\"call bpf_preempt_enable;\"\n+\t\"call bpf_unwind_resume;\"\n+\t\"exit;\"\n+\tCLEANUP_REC(\"1b\", \"2b\", \"3b\")\n+\t::: __clobber_all);\n+}\n+\n+SEC(\"?syscall\")\n+__exception_cb(unused_exc_cb)\n+__failure __msg(\"cannot be combined with an exception callback\")\n+int table_with_exception_cb(void *ctx)\n+{\n+\treturn cb_and_table_frame();\n+}\n+\n+__u64 never;\n+\n+/* 5. A landing pad that calls a subprogram which can throw. Not case 3: the\n+ * throw is in another subprogram, so what catches it is the walk of the pad's\n+ * body, off subprog_info.might_throw.\n+ */\n+static __used __noinline void pad_callee_that_throws(void)\n+{\n+\tif (never)\n+\t\tbpf_throw(0);\n+}\n+\n+static __used __naked __noinline __u64 pad_calls_thrower_frame(void)\n+{\n+\tasm volatile (\n+\t\"call bpf_preempt_disable;\"\n+\t\"r1 = 11;\"\n+\"1:\"\t\"call bpf_throw;\"\t\t/* cleanup region */\n+\"2:\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\"3:\"\t\t\t\t\t/* landing pad */\n+\t\"call pad_callee_that_throws;\"\t/* ...which can throw: refused */\n+\t\"call bpf_preempt_enable;\"\n+\t\"call bpf_unwind_resume;\"\n+\t\"exit;\"\n+\tCLEANUP_REC(\"1b\", \"2b\", \"3b\")\n+\t::: __clobber_all);\n+}\n+\n+SEC(\"?syscall\")\n+__failure __msg(\"which can throw while an exception is in flight\")\n+int pad_calls_thrower(void *ctx)\n+{\n+\treturn pad_calls_thrower_frame();\n+}\n+\n+/* 6. A catch pad: it ends in a plain exit rather than a resume, and a walker\n+ * that calls pads as subroutines cannot hand a frame back its own execution.\n+ */\n+static __used __naked __noinline __u64 catch_pad_frame(void)\n+{\n+\tasm volatile (\n+\t\"call bpf_preempt_disable;\"\n+\t\"r1 = 12;\"\n+\"1:\"\t\"call bpf_throw;\"\t\t/* cleanup region */\n+\"2:\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\"3:\"\t\t\t\t\t/* catch pad: no resume, it stops here */\n+\t\"call bpf_preempt_enable;\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\tCLEANUP_REC(\"1b\", \"2b\", \"3b\")\n+\t::: __clobber_all);\n+}\n+\n+SEC(\"?syscall\")\n+__failure __msg(\"is not supported yet, only cleanup pads that resume\")\n+int catch_landing_pad(void *ctx)\n+{\n+\treturn catch_pad_frame();\n+}\n+\n+/* 7. An exception reaching the boundary of a program type that constrains its\n+ * return value: delivery makes the cookie that return value, and fentry has\n+ * to return 0.\n+ */\n+static __used __naked __noinline __u64 boundary_throw_frame(void)\n+{\n+\tasm volatile (\n+\t\"call bpf_preempt_disable;\"\n+\t\"r1 = 7;\"\n+\"1:\"\t\"call bpf_throw;\"\t\t/* cleanup region */\n+\"2:\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\"3:\"\t\t\t\t\t/* landing pad */\n+\t\"call bpf_preempt_enable;\"\n+\t\"call bpf_unwind_resume;\"\n+\t\"exit;\"\n+\tCLEANUP_REC(\"1b\", \"2b\", \"3b\")\n+\t::: __clobber_all);\n+}\n+\n+SEC(\"?fentry/bpf_fentry_test1\")\n+__failure __msg(\"the register R1 has smin=7 smax=7 should have been in [0, 0]\")\n+int boundary_delivers(void *ctx)\n+{\n+\treturn boundary_throw_frame();\n+}\n+\n+/* 8. A bpf_unwind_resume() outside any landing pad. Both JITs lower it as the\n+ * way back out of a pad, which in ordinary code leaves a live frame standing\n+ * with its epilogue skipped.\n+ */\n+static __used __naked __noinline __u64 stray_resume_frame(void)\n+{\n+\tasm volatile (\n+\t\"call bpf_preempt_disable;\"\n+\t\"r1 = 13;\"\n+\"1:\"\t\"call bpf_throw;\"\t\t/* cleanup region */\n+\"2:\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\"3:\"\t\t\t\t\t/* landing pad */\n+\t\"call bpf_preempt_enable;\"\n+\t\"call bpf_unwind_resume;\"\n+\t\"exit;\"\n+\tCLEANUP_REC(\"1b\", \"2b\", \"3b\")\n+\t::: __clobber_all);\n+}\n+\n+SEC(\"?syscall\")\n+__failure __msg(\"is not in an exception cleanup landing pad\")\n+int resume_outside_pad(void *ctx)\n+{\n+\t/* Never taken, but reachable, which is all the verifier needs. */\n+\tif (never)\n+\t\tbpf_unwind_resume();\n+\treturn stray_resume_frame();\n+}\n+\n+/* 9. A bpf_unwind_resume() in a subprogram a landing pad calls. The verifier's\n+ * walk cannot tell it from a resume in the pad itself -- an exception is in\n+ * flight either way -- so the rule is static: a resume sits in a pad body.\n+ */\n+static __used __naked __noinline void resume_in_callee(void)\n+{\n+\tasm volatile (\n+\t\"call bpf_unwind_resume;\"\n+\t\"exit;\"\n+\t::: __clobber_all);\n+}\n+\n+static __used __naked __noinline __u64 pad_calls_resumer_frame(void)\n+{\n+\tasm volatile (\n+\t\"call bpf_preempt_disable;\"\n+\t\"r1 = 14;\"\n+\"1:\"\t\"call bpf_throw;\"\t\t/* cleanup region */\n+\"2:\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\"3:\"\t\t\t\t\t/* landing pad */\n+\t\"call bpf_preempt_enable;\"\n+\t\"call resume_in_callee;\"\t/* ...which resumes: refused */\n+\t\"call bpf_unwind_resume;\"\n+\t\"exit;\"\n+\tCLEANUP_REC(\"1b\", \"2b\", \"3b\")\n+\t::: __clobber_all);\n+}\n+\n+SEC(\"?syscall\")\n+__failure __msg(\"is not in an exception cleanup landing pad\")\n+int resume_in_pad_callee(void *ctx)\n+{\n+\treturn pad_calls_resumer_frame();\n+}\n+\n+/* 10. A bpf_unwind_resume() in a program carrying no cleanup table, where\n+ * that static rule does not run at all. do_check() refuses it on the state\n+ * not unwinding, and has to: the JITs lower every one of these the same way.\n+ */\n+static __used __naked __noinline __u64 no_table_resume_frame(void)\n+{\n+\tasm volatile (\n+\t\"call bpf_unwind_resume;\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t::: __clobber_all);\n+}\n+\n+SEC(\"?syscall\")\n+__failure __msg(\"reached without an exception in flight\")\n+int resume_without_table(void *ctx)\n+{\n+\treturn no_table_resume_frame();\n+}\n+\n+/* 11. A landing pad that is itself a covered call site, so an exception out\n+ * of it would have nowhere to go. Hand-written only: LLVM sinks a function's\n+ * pads past every range it emits.\n+ */\n+static __used __naked __noinline __u64 nested_pad_frame(void)\n+{\n+\tasm volatile (\n+\t\"call bpf_preempt_disable;\"\n+\"1:\"\t\"call inner_throw;\"\t\t/* first cleanup region */\n+\"2:\"\n+\t\"call bpf_preempt_enable;\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\"3:\"\t\t\t\t\t/* first pad, second region's call */\n+\t\"call bpf_preempt_enable;\"\n+\"4:\"\n+\t\"call bpf_unwind_resume;\"\n+\t\"exit;\"\n+\"5:\"\t\t\t\t\t/* second pad */\n+\t\"call bpf_preempt_enable;\"\n+\t\"call bpf_unwind_resume;\"\n+\t\"exit;\"\n+\tCLEANUP_REC(\"1b\", \"2b\", \"3b\")\n+\tCLEANUP_REC(\"3b\", \"4b\", \"5b\")\n+\t::: __clobber_all);\n+}\n+\n+SEC(\"?syscall\")\n+__failure __msg(\"is inside the call-site range of\")\n+int nested_landing_pad(void *ctx)\n+{\n+\treturn nested_pad_frame();\n+}\n+\n+/* 12. A tail call in a landing pad: it unwinds the prologue off the stack\n+ * pointer, which in a pad is the walker's.\n+ */\n+struct {\n+\t__uint(type, BPF_MAP_TYPE_PROG_ARRAY);\n+\t__uint(max_entries, 1);\n+\t__uint(key_size, sizeof(__u32));\n+\t__uint(value_size, sizeof(__u32));\n+} tc_map SEC(\".maps\");\n+\n+static __used __naked __noinline __u64 tail_call_pad_frame(void)\n+{\n+\tasm volatile (\n+\t\"r6 = r1;\"\n+\"1:\"\t\"call inner_throw;\"\t\t/* cleanup region */\n+\"2:\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\"3:\"\t\t\t\t\t/* landing pad */\n+\t\"r1 = r6;\"\n+\t\"r2 = %[tc_map] ll;\"\n+\t\"r3 = 0;\"\n+\t\"call %[bpf_tail_call];\"\n+\t\"call bpf_unwind_resume;\"\n+\t\"exit;\"\n+\tCLEANUP_REC(\"1b\", \"2b\", \"3b\")\n+\t:\n+\t: __imm(bpf_tail_call), __imm_addr(tc_map)\n+\t: __clobber_all);\n+}\n+\n+SEC(\"?syscall\")\n+__failure __msg(\"is in an exception cleanup landing pad\")\n+int tail_call_in_pad(void *ctx)\n+{\n+\treturn tail_call_pad_frame();\n+}\n+\n+#if defined(__BPF_FEATURE_STACK_ARGUMENT)\n+\n+/* 13. A call that passes an argument on the stack, in a landing pad: the\n+ * outgoing area the callee reads is not the one the caller wrote, the frame\n+ * being the unwinding one and the stack pointer the walker's.\n+ */\n+static __used __noinline __u64 six_args(__u64 a, __u64 b, __u64 c, __u64 d,\n+\t\t\t\t\t__u64 e, __u64 f)\n+{\n+\treturn a + b + c + d + e + f;\n+}\n+\n+static __used __naked __noinline __u64 stack_arg_pad_frame(void)\n+{\n+\tasm volatile (\n+\t\"call bpf_preempt_disable;\"\n+\"1:\"\t\"call inner_throw;\"\t\t/* cleanup region */\n+\"2:\"\n+\t\"call bpf_preempt_enable;\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\"3:\"\t\t\t\t\t/* landing pad */\n+\t\"call bpf_preempt_enable;\"\n+\t\"r1 = 1;\"\n+\t\"r2 = 2;\"\n+\t\"r3 = 3;\"\n+\t\"r4 = 4;\"\n+\t\"r5 = 5;\"\n+\t\"*(u64 *)(r11 - 8) = 6;\"\t/* the sixth argument */\n+\t\"call six_args;\"\n+\t\"call bpf_unwind_resume;\"\n+\t\"exit;\"\n+\tCLEANUP_REC(\"1b\", \"2b\", \"3b\")\n+\t::: __clobber_all);\n+}\n+\n+SEC(\"?syscall\")\n+__failure __msg(\"on-stack call argument in an exception cleanup landing pad\")\n+int stack_arg_in_pad(void *ctx)\n+{\n+\treturn stack_arg_pad_frame();\n+}\n+\n+/* 14. The same, reached the other way: a kfunc whose by-value argument runs\n+ * past the five argument registers, where the JIT fills the outgoing area and\n+ * the rule above has no store to catch. The C call gives the extern its BTF.\n+ */\n+static __used __noinline void __nofit_btf_anchor(void)\n+{\n+\tstruct prog_test_pair_arg s = {};\n+\n+\tbpf_kfunc_call_test_pair_arg_nofit(1, 2, 3, 4, s);\n+}\n+\n+static __used __naked __noinline __u64 kfunc_arg_pad_frame(void)\n+{\n+\tasm volatile (\n+\t\"call bpf_preempt_disable;\"\n+\"1:\"\t\"call inner_throw;\"\t\t/* cleanup region */\n+\"2:\"\n+\t\"call bpf_preempt_enable;\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\"3:\"\t\t\t\t\t/* landing pad */\n+\t\"call bpf_preempt_enable;\"\n+\t\"r1 = 1;\"\n+\t\"r2 = 2;\"\n+\t\"r3 = 3;\"\n+\t\"r4 = 4;\"\n+\t\"r5 = 5;\"\n+\t\"call bpf_kfunc_call_test_pair_arg_nofit;\"\n+\t\"call bpf_unwind_resume;\"\n+\t\"exit;\"\n+\tCLEANUP_REC(\"1b\", \"2b\", \"3b\")\n+\t::: __clobber_all);\n+}\n+\n+SEC(\"?syscall\")\n+__failure __msg(\"on-stack call argument in an exception cleanup landing pad\")\n+int kfunc_stack_arg_in_pad(void *ctx)\n+{\n+\treturn kfunc_arg_pad_frame();\n+}\n+\n+#endif /* __BPF_FEATURE_STACK_ARGUMENT */\n+\n+/* 15. A landing pad entered by ordinary control flow, arriving with none of\n+ * what the walker sets up. Nothing static sees it -- the resume really is in\n+ * a pad body -- so do_check() refuses it on the state not unwinding.\n+ */\n+static __used __naked __noinline __u64 jump_into_pad_frame(void)\n+{\n+\tasm volatile (\n+\t\"r1 = %[input] ll;\"\n+\t\"r6 = *(u64 *)(r1 + 0);\"\n+\t\"if r6 \u003e 7 goto 4f;\"\t\t/* an ordinary branch into the pad */\n+\"1:\"\t\"call inner_throw;\"\t\t/* cleanup region */\n+\"2:\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\"3:\"\t\t\t\t\t/* landing pad */\n+\t\"r7 = r0;\"\n+\"4:\"\t\t\t\t\t/* ... and its second instruction */\n+\t\"call bpf_unwind_resume;\"\n+\t\"exit;\"\n+\tCLEANUP_REC(\"1b\", \"2b\", \"3b\")\n+\t:\n+\t: __imm_addr(input)\n+\t: __clobber_all);\n+}\n+\n+SEC(\"?syscall\")\n+__failure __msg(\"reached without an exception in flight\")\n+int jump_into_pad(void *ctx)\n+{\n+\treturn jump_into_pad_frame();\n+}\n+\n+#if defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64)\n+\n+/* 16. A landing pad that reaches an indirect jump, which cannot be told from\n+ * a catch pad. SEC(\"socket\") because a jump table entry is an offset from the\n+ * program's section symbol, and \"?syscall\" is not a name assembly can use.\n+ */\n+static __used __naked __noinline void gotox_thrower(void)\n+{\n+\tasm volatile (\n+\t\"r1 = 15;\"\n+\t\"call bpf_throw;\"\n+\t\"exit;\"\n+\t::: __clobber_all);\n+}\n+\n+SEC(\"socket\")\n+__failure __msg(\"reaches an indirect jump\")\n+__naked void gotox_in_pad(void)\n+{\n+\tasm volatile (\n+\t\".pushsection .jumptables,\\\"\\\",@progbits;\"\n+\"jt0_%=:\"\n+\t\".quad l0_%= - socket;\"\n+\t\".quad l1_%= - socket;\"\n+\t\".size jt0_%=, 16;\"\n+\t\".global jt0_%=;\"\n+\t\".popsection;\"\n+\n+\"1:\"\t\"call gotox_thrower;\"\t\t/* cleanup region */\n+\"2:\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\"3:\"\t\t\t\t\t/* landing pad */\n+\t\"r1 = jt0_%= ll;\"\n+\t\"r1 += 8;\"\n+\t\"r2 = *(u64 *)(r1 + 0);\"\n+\t/* gotox r2, as a raw insn: the mnemonic only reached the LLVM\n+\t * assembler in llvm 22, and BPF_RAW_INSN() needs \u003clinux/bpf.h\u003e, which\n+\t * vmlinux.h rules out.\n+\t */\n+\t\".8byte 0x20d;\"\n+\"l0_%=:\"\n+\t\"call bpf_unwind_resume;\"\n+\t\"exit;\"\n+\"l1_%=:\"\n+\t\"call bpf_unwind_resume;\"\n+\t\"exit;\"\n+\tCLEANUP_REC(\"1b\", \"2b\", \"3b\")\n+\t::: __clobber_all);\n+}\n+\n+#endif /* x86 || arm64 */\n+\n+/* 17. A BPF_LD_[ABS|IND] in a landing pad. A failed load leaves the\n+ * subprogram through the hidden \"r0 = 0; exit\" gen_ld_abs() patches in, and\n+ * that exit is the epilogue, which unwinds a stack the pad does not own.\n+ */\n+static __used __naked __noinline __u64 ld_abs_pad_frame(void)\n+{\n+\tasm volatile (\n+\t\"r6 = r1;\"\t\t\t/* the skb BPF_LD_ABS reads */\n+\"1:\"\t\"call inner_throw;\"\t\t/* cleanup region */\n+\"2:\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\"3:\"\t\t\t\t\t/* landing pad */\n+\t/* r0 = *(u32 *)skb[0]. Spelled as a raw insn because BPF_LD_ABS()\n+\t * needs \u003clinux/filter.h\u003e, which this file cannot have -- vmlinux.h\n+\t * already defines the uapi enums.\n+\t */\n+\t\".8byte 0x20;\"\n+\t\"call bpf_unwind_resume;\"\n+\t\"exit;\"\n+\tCLEANUP_REC(\"1b\", \"2b\", \"3b\")\n+\t::: __clobber_all);\n+}\n+\n+SEC(\"?tc\")\n+__failure __msg(\"is in an exception cleanup landing pad\")\n+__naked void ld_abs_in_pad(void)\n+{\n+\tasm volatile (\n+\t\"call ld_abs_pad_frame;\"\n+\t\"exit;\"\n+\t::: __clobber_all);\n+}\n+\n+/* 18. A bpf_unwind_resume() in a subprogram a landing pad called, which has a\n+ * pad of its own and reached it by ordinary control flow. Not case 9: an\n+ * exception is in flight, but this is not the frame whose pad the walker ran.\n+ */\n+static __used __naked __noinline __u64 own_pad_callee(void)\n+{\n+\tasm volatile (\n+\t\"r1 = %[input] ll;\"\n+\t\"r1 = *(u64 *)(r1 + 0);\"\n+\t\"if r1 == 0 goto 3f;\"\t\t/* an ordinary branch into its pad */\n+\"1:\"\t\"call bpf_preempt_disable;\"\t/* cleanup region: nothing that throws */\n+\t\"call bpf_preempt_enable;\"\n+\"2:\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\"3:\"\t\t\t\t\t/* its landing pad */\n+\t\"call bpf_unwind_resume;\"\n+\t\"exit;\"\n+\tCLEANUP_REC(\"1b\", \"2b\", \"3b\")\n+\t:\n+\t: __imm_addr(input)\n+\t: __clobber_all);\n+}\n+\n+static __used __naked __noinline __u64 pad_calls_own_pad_frame(void)\n+{\n+\tasm volatile (\n+\"1:\"\t\"call inner_throw;\"\t\t/* cleanup region */\n+\"2:\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\"3:\"\t\t\t\t\t/* landing pad, which calls the above */\n+\t\"call own_pad_callee;\"\n+\t\"call bpf_unwind_resume;\"\n+\t\"exit;\"\n+\tCLEANUP_REC(\"1b\", \"2b\", \"3b\")\n+\t::: __clobber_all);\n+}\n+\n+SEC(\"?syscall\")\n+__failure __msg(\"is in frame 2, not frame 1 whose landing pad the exception entered\")\n+int resume_in_callee_own_pad(void *ctx)\n+{\n+\treturn pad_calls_own_pad_frame();\n+}\n+\n+char _license[] SEC(\"license\") = \"GPL\";\ndiff --git a/tools/testing/selftests/bpf/progs/exceptions_cleanup_freplace.c b/tools/testing/selftests/bpf/progs/exceptions_cleanup_freplace.c\nnew file mode 100644\nindex 0000000000000..afb358fd3d405\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/progs/exceptions_cleanup_freplace.c\n@@ -0,0 +1,17 @@\n+// SPDX-License-Identifier: GPL-2.0\n+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */\n+#include \u003cvmlinux.h\u003e\n+#include \u003cbpf/bpf_helpers.h\u003e\n+#include \"exceptions_cleanup.h\"\n+\n+/* Without a 32-bit int in BTF, libbpf's dummy_ksym var gets type id 0. */\n+int btf_int_anchor;\n+\n+SEC(\"freplace/fr_callee\")\n+__u64 new_fr_callee(__u64 x)\n+{\n+\tbpf_throw(THROW_COOKIE);\n+\treturn 0;\n+}\n+\n+char _license[] SEC(\"license\") = \"GPL\";\ndiff --git a/tools/testing/selftests/bpf/progs/exceptions_cleanup_light.c b/tools/testing/selftests/bpf/progs/exceptions_cleanup_light.c\nnew file mode 100644\nindex 0000000000000..de7ae478d67f3\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/progs/exceptions_cleanup_light.c\n@@ -0,0 +1,39 @@\n+// SPDX-License-Identifier: GPL-2.0\n+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */\n+#include \u003cvmlinux.h\u003e\n+#include \u003cbpf/bpf_helpers.h\u003e\n+#include \"bpf_misc.h\"\n+#include \"exceptions_cleanup.h\"\n+\n+static __used __noinline void __kfunc_btf_anchor(void)\n+{\n+\tbpf_throw(0);\n+\tbpf_preempt_disable();\n+\tbpf_preempt_enable();\n+\tbpf_unwind_resume();\n+}\n+\n+__u64 pads_ran = 0;\n+\n+SEC(\"syscall\")\n+__naked int entry_light(void)\n+{\n+\tasm volatile (\n+\t\"call bpf_preempt_disable;\"\n+\t\"r1 = %[cookie];\"\n+\"1:\"\t\"call bpf_throw;\"\t\t/* cleanup region */\n+\"2:\"\n+\t\"exit;\"\n+\"3:\"\t\t\t\t\t/* landing pad */\n+\t\"call bpf_preempt_enable;\"\n+\tPAD_RAN(\"%[ran]\")\n+\t\"call bpf_unwind_resume;\"\n+\t\"exit;\"\n+\tCLEANUP_REC(\"1b\", \"2b\", \"3b\")\n+\t:\n+\t: [cookie]\"i\"(THROW_COOKIE), [ran]\"i\"(RAN_LIGHT),\n+\t __imm_addr(pads_ran)\n+\t: __clobber_all);\n+}\n+\n+char _license[] SEC(\"license\") = \"GPL\";\ndiff --git a/tools/testing/selftests/bpf/progs/exceptions_cleanup_pad_freplace.c b/tools/testing/selftests/bpf/progs/exceptions_cleanup_pad_freplace.c\nnew file mode 100644\nindex 0000000000000..eabac6baabb70\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/progs/exceptions_cleanup_pad_freplace.c\n@@ -0,0 +1,17 @@\n+// SPDX-License-Identifier: GPL-2.0\n+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */\n+#include \u003cvmlinux.h\u003e\n+#include \u003cbpf/bpf_helpers.h\u003e\n+#include \"exceptions_cleanup.h\"\n+\n+/* Without a 32-bit int in BTF, libbpf's dummy_ksym var gets type id 0. */\n+int btf_int_anchor;\n+\n+SEC(\"freplace/pad_callee\")\n+__u64 new_pad_callee(__u64 x)\n+{\n+\tbpf_throw(INNER_COOKIE);\n+\treturn 0;\n+}\n+\n+char _license[] SEC(\"license\") = \"GPL\";\ndiff --git a/tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c b/tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c\nnew file mode 100644\nindex 0000000000000..493f8e0306528\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c\n@@ -0,0 +1,984 @@\n+// SPDX-License-Identifier: GPL-2.0\n+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */\n+#include \u003cvmlinux.h\u003e\n+#include \u003cbpf/bpf_helpers.h\u003e\n+#include \"bpf_misc.h\"\n+#include \"exceptions_cleanup.h\"\n+\n+#define PAD_COUNT\t\t\t\t\t\t\\\n+\t\"r1 = %[pad_runs] ll;\"\t\t\t\t\t\\\n+\t\"r2 = *(u64 *)(r1 + 0);\"\t\t\t\t\\\n+\t\"r2 += 1;\"\t\t\t\t\t\t\\\n+\t\"*(u64 *)(r1 + 0) = r2;\"\n+\n+static __used __noinline void __kfunc_btf_anchor(void)\n+{\n+\tbpf_throw(0);\n+\tbpf_rcu_read_lock();\n+\tbpf_rcu_read_unlock();\n+\tbpf_preempt_disable();\n+\tbpf_preempt_enable();\n+\tbpf_unwind_resume();\n+}\n+\n+__u64 input = 0;\n+__u64 outer_input = 0;\n+__u64 magic = 0x5eed;\n+__u64 pads_ran = 0;\n+__u64 pad_runs = 0;\n+\n+/* 1. Everything a cleanup table leaves dead: a throw's continuation, the tail\n+ * after a resume with an ld_imm64 and a branch in it, and the block only that\n+ * continuation reaches.\n+ */\n+static __used __naked __noinline __u64 sweep_frame(void)\n+{\n+\tasm volatile (\n+\t\"r1 = %[input] ll;\"\n+\t\"r6 = *(u64 *)(r1 + 0);\"\n+\t\"call bpf_preempt_disable;\"\n+\t\"if r6 \u003c 101 goto 6f;\"\n+\t\"r1 = %[cookie];\"\n+\"1:\"\t\"call bpf_throw;\"\t\t/* cleanup region */\n+\"2:\"\n+\t\"goto 3f;\"\n+\"4:\"\t\t\t\t\t/* landing pad */\n+\t\"r7 = r0;\"\n+\t\"call bpf_preempt_enable;\"\n+\tPAD_RAN(\"%[ran]\")\n+\t\"r1 = r7;\"\n+\t\"call bpf_unwind_resume;\"\n+\t\"r1 = %[pads_ran] ll;\"\n+\t\"r2 = *(u64 *)(r1 + 0);\"\n+\t\"if r2 == 0 goto 5f;\"\n+\t\"call bpf_preempt_enable;\"\n+\t\"r0 = 7;\"\n+\t\"exit;\"\n+\"5:\"\n+\t\"r0 = 8;\"\n+\t\"exit;\"\n+\"3:\"\t\t\t\t\t/* dead: only the dead goto reaches it */\n+\t\"r0 = 9;\"\n+\t\"exit;\"\n+\"6:\"\t\t\t\t\t/* live: the ordinary return */\n+\t\"call bpf_preempt_enable;\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\tCLEANUP_REC(\"1b\", \"2b\", \"4b\")\n+\t:\n+\t: [cookie]\"i\"(THROW_COOKIE), [ran]\"i\"(RAN_SWEEP),\n+\t __imm_addr(input), __imm_addr(pads_ran)\n+\t: __clobber_all);\n+}\n+\n+SEC(\"syscall\")\n+int entry_sweep(void *ctx)\n+{\n+\treturn sweep_frame();\n+}\n+\n+/* 2. A callee called from both a covered and an uncovered site: the pad\n+ * belongs to the call site, not the callee. Either site can throw, and the RCU\n+ * lock between them is what an unwind out of the uncovered one leaks.\n+ */\n+static __used __noinline __u64 shared_callee(__u64 x)\n+{\n+\tif (x \u003e 100)\n+\t\tbpf_throw(THROW_COOKIE);\n+\treturn x + 1;\n+}\n+\n+static __used __naked __noinline __u64 shared_frame(void)\n+{\n+\tasm volatile (\n+\t\"r1 = %[input] ll;\"\n+\t\"r6 = *(u64 *)(r1 + 0);\"\n+\t\"r1 = %[outer_input] ll;\"\n+\t\"r1 = *(u64 *)(r1 + 0);\"\n+\t\"call shared_callee;\"\t\t/* uncovered: no pad for its unwind */\n+\t\"call bpf_rcu_read_lock;\"\n+\t\"r1 = r6;\"\n+\"1:\"\t\"call shared_callee;\"\t\t/* cleanup region */\n+\"2:\"\n+\t\"r6 = r0;\"\n+\t\"call bpf_rcu_read_unlock;\"\n+\t\"r0 = r6;\"\n+\t\"exit;\"\n+\"3:\"\t\t\t\t\t/* landing pad */\n+\t\"r7 = r0;\"\n+\t\"call bpf_rcu_read_unlock;\"\n+\tPAD_RAN(\"%[ran]\")\n+\t\"r1 = r7;\"\n+\t\"call bpf_unwind_resume;\"\n+\t\"exit;\"\n+\tCLEANUP_REC(\"1b\", \"2b\", \"3b\")\n+\t:\n+\t: [ran]\"i\"(RAN_SHARED), __imm_addr(input), __imm_addr(outer_input),\n+\t __imm_addr(pads_ran)\n+\t: __clobber_all);\n+}\n+\n+SEC(\"syscall\")\n+int entry_shared(void *ctx)\n+{\n+\treturn shared_frame();\n+}\n+\n+/* 3. A landing pad that reads its frame's callee-saved registers: the callee\n+ * overwrites r6-r9 before it throws, so the check passes only if the walker\n+ * found the spill in the discarded callee's prologue.\n+ */\n+#define LOAD_MAGIC_REGS\t\t\t\t\t\t\\\n+\t\"r1 = %[magic] ll;\"\t\t\t\t\t\\\n+\t\"r6 = *(u64 *)(r1 + 0);\"\t\t\t\t\\\n+\t\"r7 = r6;\"\t\t\t\t\t\t\\\n+\t\"r7 += 1;\"\t\t\t\t\t\t\\\n+\t\"r8 = r6;\"\t\t\t\t\t\t\\\n+\t\"r8 += 2;\"\t\t\t\t\t\t\\\n+\t\"r9 = r6;\"\t\t\t\t\t\t\\\n+\t\"r9 += 3;\"\n+\n+/* Set @bit only if r6-r9 still hold what LOAD_MAGIC_REGS put there. */\n+#define CHECK_MAGIC_REGS(bit)\t\t\t\t\t\\\n+\t\"r1 = %[magic] ll;\"\t\t\t\t\t\\\n+\t\"r2 = *(u64 *)(r1 + 0);\"\t\t\t\t\\\n+\t\"if r6 != r2 goto 9f;\"\t\t\t\t\t\\\n+\t\"r2 += 1;\"\t\t\t\t\t\t\\\n+\t\"if r7 != r2 goto 9f;\"\t\t\t\t\t\\\n+\t\"r2 += 1;\"\t\t\t\t\t\t\\\n+\t\"if r8 != r2 goto 9f;\"\t\t\t\t\t\\\n+\t\"r2 += 1;\"\t\t\t\t\t\t\\\n+\t\"if r9 != r2 goto 9f;\"\t\t\t\t\t\\\n+\tPAD_RAN(bit)\t\t\t\t\t\t\\\n+\t\"9:\"\n+\n+static __used __naked __noinline __u64 regs_thrower(void)\n+{\n+\tasm volatile (\n+\t/* Not this frame's to keep, and that is the point. */\n+\t\"r6 = 0xdead;\"\n+\t\"r7 = 0xbeef;\"\n+\t\"r8 = 0xcafe;\"\n+\t\"r9 = 0xf00d;\"\n+\t\"r1 = %[cookie];\"\n+\t\"call bpf_throw;\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:\n+\t: [cookie]\"i\"(THROW_COOKIE)\n+\t: __clobber_all);\n+}\n+\n+static __used __naked __noinline __u64 regs_frame(void)\n+{\n+\tasm volatile (\n+\tLOAD_MAGIC_REGS\n+\t\"call bpf_preempt_disable;\"\n+\"1:\"\t\"call regs_thrower;\"\t\t/* cleanup region */\n+\"2:\"\n+\t\"call bpf_preempt_enable;\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\"3:\"\t\t\t\t\t/* landing pad */\n+\t\"call bpf_preempt_enable;\"\n+\tCHECK_MAGIC_REGS(\"%[ran]\")\n+\t\"call bpf_unwind_resume;\"\n+\t\"exit;\"\n+\tCLEANUP_REC(\"1b\", \"2b\", \"3b\")\n+\t:\n+\t: [cookie]\"i\"(THROW_COOKIE), [ran]\"i\"(RAN_REGS),\n+\t __imm_addr(magic), __imm_addr(pads_ran)\n+\t: __clobber_all);\n+}\n+\n+SEC(\"syscall\")\n+int entry_regs(void *ctx)\n+{\n+\treturn regs_frame();\n+}\n+\n+/* 4. The same, with a tail-call-reachable callee: its prologue pushes the tail\n+ * call counter, which moves the spill the walker reads. The array is left\n+ * empty; being reachable is the point.\n+ */\n+struct {\n+\t__uint(type, BPF_MAP_TYPE_PROG_ARRAY);\n+\t__uint(max_entries, 1);\n+\t__uint(key_size, sizeof(__u32));\n+\t__uint(value_size, sizeof(__u32));\n+} jmp_table SEC(\".maps\");\n+\n+static __used __noinline __u64 tc_thrower(void *ctx)\n+{\n+\t/* Never taken; its presence is what makes this frame, whose spill the\n+\t * walker reads, tail-call-reachable.\n+\t */\n+\tbpf_tail_call_static(ctx, \u0026jmp_table, 0);\n+\tasm volatile (\n+\t\"r6 = 0xdead;\"\n+\t\"r7 = 0xbeef;\"\n+\t\"r8 = 0xcafe;\"\n+\t\"r9 = 0xf00d;\"\n+\t\"r1 = %[cookie];\"\n+\t\"call bpf_throw;\"\n+\t:\n+\t: [cookie]\"i\"(THROW_COOKIE)\n+\t: __clobber_all);\n+\treturn 0;\n+}\n+\n+/* The frame with the pad is the program itself, and __naked: r1 at entry is\n+ * the only place to get a context for bpf_tail_call().\n+ */\n+SEC(\"syscall\")\n+__naked int entry_tail_call(void)\n+{\n+\tasm volatile (\n+\t\"*(u64 *)(r10 - 8) = r1;\"\t/* the context, straight from entry */\n+\tLOAD_MAGIC_REGS\n+\t\"r1 = %[input] ll;\"\n+\t\"r1 = *(u64 *)(r1 + 0);\"\n+\t\"if r1 \u003c 101 goto 8f;\"\n+\t\"r1 = *(u64 *)(r10 - 8);\"\n+\"1:\"\t\"call tc_thrower;\"\t\t/* cleanup region */\n+\"2:\"\n+\"8:\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\"3:\"\t\t\t\t\t/* landing pad */\n+\tCHECK_MAGIC_REGS(\"%[ran]\")\n+\t\"call bpf_unwind_resume;\"\n+\t\"exit;\"\n+\tCLEANUP_REC(\"1b\", \"2b\", \"3b\")\n+\t:\n+\t: [ran]\"i\"(RAN_TAIL_CALL), __imm_addr(input),\n+\t __imm_addr(magic), __imm_addr(pads_ran)\n+\t: __clobber_all);\n+}\n+\n+/* 5. A landing pad in the main program's own frame. jit_subprogs() compiles it\n+ * as func[0], but the walker finds the outer bpf_prog's ksym, so the table has\n+ * to be handed over or the pad is never dispatched.\n+ */\n+SEC(\"syscall\")\n+__naked int entry_main_pad(void)\n+{\n+\tasm volatile (\n+\tLOAD_MAGIC_REGS\n+\t\"r1 = %[input] ll;\"\n+\t\"r1 = *(u64 *)(r1 + 0);\"\n+\t\"if r1 \u003c 101 goto 8f;\"\n+\"1:\"\t\"call regs_thrower;\"\t\t/* cleanup region */\n+\"2:\"\n+\"8:\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\"3:\"\t\t\t\t\t/* landing pad */\n+\tCHECK_MAGIC_REGS(\"%[ran]\")\n+\t\"call bpf_unwind_resume;\"\n+\t\"exit;\"\n+\tCLEANUP_REC(\"1b\", \"2b\", \"3b\")\n+\t:\n+\t: [ran]\"i\"(RAN_MAIN_PAD), __imm_addr(input),\n+\t __imm_addr(magic), __imm_addr(pads_ran)\n+\t: __clobber_all);\n+}\n+\n+/* 6. A tail call that is really taken: the target is a program in its own\n+ * right, so the walk ends there and this frame's pad does not run. The callee\n+ * can also throw, which keeps the pad out of the sweep.\n+ */\n+struct {\n+\t__uint(type, BPF_MAP_TYPE_PROG_ARRAY);\n+\t__uint(max_entries, 1);\n+\t__uint(key_size, sizeof(__u32));\n+\t__uint(value_size, sizeof(__u32));\n+} taken_table SEC(\".maps\");\n+\n+SEC(\"syscall\")\n+int tc_target(void *ctx)\n+{\n+\tbpf_throw(THROW_COOKIE);\n+\treturn 0;\n+}\n+\n+static __used __noinline __u64 tc_taken_callee(void *ctx, __u64 x)\n+{\n+\t/* Never true at run time; the verifier cannot know that, and its\n+\t * unwind out of here is what keeps the caller's pad alive.\n+\t */\n+\tif (x == 7)\n+\t\tbpf_throw(THROW_COOKIE);\n+\tbpf_tail_call_static(ctx, \u0026taken_table, 0);\n+\treturn 0;\n+}\n+\n+SEC(\"syscall\")\n+__naked int entry_tail_taken(void)\n+{\n+\tasm volatile (\n+\t\"*(u64 *)(r10 - 8) = r1;\"\t/* the context, straight from entry */\n+\t\"r1 = %[input] ll;\"\n+\t/* Unnarrowed, the way entry_freplace hands fr_callee its argument: a\n+\t * guard here would prune the callee's throw and let the sweep take the\n+\t * pad, leaving no record for a walk past the boundary to match.\n+\t */\n+\t\"r2 = *(u64 *)(r1 + 0);\"\n+\t\"r1 = *(u64 *)(r10 - 8);\"\n+\"1:\"\t\"call tc_taken_callee;\"\t\t/* cleanup region */\n+\"2:\"\n+\t\"exit;\"\t\t\t\t/* the cookie, delivered at tc_target */\n+\"3:\"\t\t\t\t\t/* landing pad: must not run */\n+\tPAD_RAN(\"%[ran]\")\n+\t\"call bpf_unwind_resume;\"\n+\t\"exit;\"\n+\tCLEANUP_REC(\"1b\", \"2b\", \"3b\")\n+\t:\n+\t: [ran]\"i\"(RAN_TC_TAKEN), __imm_addr(input), __imm_addr(pads_ran)\n+\t: __clobber_all);\n+}\n+\n+/* 7. An extension program over the callee of a covered call. The walk ends in\n+ * the extension's frame, as it does for a tail call target, so the pad does\n+ * not run; fr_callee() can also throw by itself.\n+ */\n+__noinline __u64 fr_callee(__u64 x)\n+{\n+\tif (x == 7)\n+\t\tbpf_throw(THROW_COOKIE);\n+\treturn x + 1;\n+}\n+\n+SEC(\"syscall\")\n+__naked int entry_freplace(void)\n+{\n+\tasm volatile (\n+\t\"r1 = %[input] ll;\"\n+\t\"r1 = *(u64 *)(r1 + 0);\"\n+\"1:\"\t\"call fr_callee;\"\t\t/* cleanup region */\n+\"2:\"\n+\t\"exit;\"\n+\"3:\"\t\t\t\t\t/* landing pad */\n+\tPAD_RAN(\"%[ran]\")\n+\t\"call bpf_unwind_resume;\"\n+\t\"exit;\"\n+\tCLEANUP_REC(\"1b\", \"2b\", \"3b\")\n+\t:\n+\t: [ran]\"i\"(RAN_FREPLACE), __imm_addr(input), __imm_addr(pads_ran)\n+\t: __clobber_all);\n+}\n+\n+/* 8. A throwing subprogram named by a BPF_PSEUDO_FUNC on a path never taken.\n+ * Handing one to a helper is what is refused, not naming it.\n+ */\n+static __used __noinline int cb_thrower(__u32 idx, void *ctx)\n+{\n+\tbpf_throw(THROW_COOKIE);\n+\treturn 0;\n+}\n+\n+static __used __noinline __u64 addr_taken_callee(__u64 x)\n+{\n+\tif (x \u003c= 100)\n+\t\treturn x + 1;\n+\tbpf_throw(THROW_COOKIE);\n+\treturn bpf_loop(1, cb_thrower, NULL, 0);\n+}\n+\n+SEC(\"syscall\")\n+__naked int entry_addr_taken(void)\n+{\n+\tasm volatile (\n+\t\"r1 = %[input] ll;\"\n+\t\"r1 = *(u64 *)(r1 + 0);\"\n+\"1:\"\t\"call addr_taken_callee;\"\t/* cleanup region */\n+\"2:\"\n+\t\"exit;\"\n+\"3:\"\t\t\t\t\t/* landing pad */\n+\tPAD_RAN(\"%[ran]\")\n+\t\"call bpf_unwind_resume;\"\n+\t\"exit;\"\n+\tCLEANUP_REC(\"1b\", \"2b\", \"3b\")\n+\t:\n+\t: [ran]\"i\"(RAN_ADDR_TAKEN), __imm_addr(input), __imm_addr(pads_ran)\n+\t: __clobber_all);\n+}\n+\n+/* 9. A record that covers bpf_throw() itself: the throwing frame is both the\n+ * frame the record covers and the boundary, so the pad runs on the way to\n+ * delivering the cookie.\n+ */\n+SEC(\"syscall\")\n+__naked int entry_no_subprog(void)\n+{\n+\tasm volatile (\n+\t\"r1 = %[input] ll;\"\n+\t\"r1 = *(u64 *)(r1 + 0);\"\n+\t\"if r1 \u003c 101 goto 8f;\"\n+\t\"call bpf_preempt_disable;\"\n+\t\"r1 = %[cookie];\"\n+\"1:\"\t\"call bpf_throw;\"\t\t/* cleanup region */\n+\"2:\"\n+\t\"exit;\"\n+\"8:\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\"3:\"\t\t\t\t\t/* landing pad */\n+\t\"call bpf_preempt_enable;\"\n+\tPAD_RAN(\"%[ran]\")\n+\t\"call bpf_unwind_resume;\"\n+\t\"exit;\"\n+\tCLEANUP_REC(\"1b\", \"2b\", \"3b\")\n+\t:\n+\t: [cookie]\"i\"(THROW_COOKIE), [ran]\"i\"(RAN_NO_SUBPROG),\n+\t __imm_addr(input), __imm_addr(pads_ran)\n+\t: __clobber_all);\n+}\n+\n+/* 10. A landing pad that calls a subprogram an extension can replace. The\n+ * load-time rule cannot see it coming, so what stops a nested exception is the\n+ * walk ending in the extension's frame; pad_runs says the pad ran once.\n+ */\n+__noinline __u64 pad_callee(__u64 x)\n+{\n+\treturn x + 1;\n+}\n+\n+static __used __noinline __u64 pc_thrower(__u64 x)\n+{\n+\tif (x \u003e 100)\n+\t\tbpf_throw(THROW_COOKIE);\n+\treturn x + 1;\n+}\n+\n+static __used __naked __noinline __u64 pad_calls_frame(void)\n+{\n+\tasm volatile (\n+\t\"r1 = %[input] ll;\"\n+\t\"r1 = *(u64 *)(r1 + 0);\"\n+\"1:\"\t\"call pc_thrower;\"\t\t/* cleanup region */\n+\"2:\"\n+\t\"exit;\"\n+\"3:\"\t\t\t\t\t/* landing pad */\n+\t\"r6 = r0;\"\n+\t\"r1 = 1;\"\n+\t\"call pad_callee;\"\t\t/* an extension can stand in here */\n+\tPAD_COUNT\n+\tPAD_RAN(\"%[ran]\")\n+\t\"r1 = r6;\"\n+\t\"call bpf_unwind_resume;\"\n+\t\"exit;\"\n+\tCLEANUP_REC(\"1b\", \"2b\", \"3b\")\n+\t:\n+\t: [ran]\"i\"(RAN_PAD_CALLS), __imm_addr(input), __imm_addr(pads_ran),\n+\t __imm_addr(pad_runs)\n+\t: __clobber_all);\n+}\n+\n+SEC(\"syscall\")\n+int entry_pad_calls(void *ctx)\n+{\n+\treturn pad_calls_frame();\n+}\n+\n+/* 11. A covered bpf_throw() the sweep leaves last, where the exception\n+ * callback patchlet -- the one that does not keep the call it replaced in the\n+ * last slot -- has to carry the marks with it; r6-r9 reports a lost one.\n+ */\n+SEC(\"syscall\")\n+__naked int entry_pad_first(void)\n+{\n+\tasm volatile (\n+\tLOAD_MAGIC_REGS\n+\t\"r1 = %[input] ll;\"\n+\t\"r1 = *(u64 *)(r1 + 0);\"\n+\t\"if r1 \u003c 101 goto 7f;\"\n+\t\"goto 4f;\"\n+\"3:\"\t\t\t\t\t/* landing pad, ahead of the call */\n+\tCHECK_MAGIC_REGS(\"%[ran]\")\n+\t\"call bpf_unwind_resume;\"\n+\t\"exit;\"\n+\"7:\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\"4:\"\n+\t\"r1 = %[cookie];\"\n+\"1:\"\t\"call bpf_throw;\"\t\t/* cleanup region */\n+\"2:\"\n+\t\"exit;\"\t\t\t\t/* dead: swept, leaving the call last */\n+\tCLEANUP_REC(\"1b\", \"2b\", \"3b\")\n+\t:\n+\t: [cookie]\"i\"(THROW_COOKIE), [ran]\"i\"(RAN_PAD_FIRST),\n+\t __imm_addr(input), __imm_addr(magic), __imm_addr(pads_ran)\n+\t: __clobber_all);\n+}\n+\n+/* 12. A cleanup region whose last instruction is a 16-byte one, so end - 1\n+ * names the half that is not an instruction of its own. Well formed, and a\n+ * rule against it would turn it away.\n+ */\n+static __used __naked __noinline __u64 wide_rec_frame(void)\n+{\n+\tasm volatile (\n+\t\"r1 = %[input] ll;\"\n+\t\"r6 = *(u64 *)(r1 + 0);\"\n+\t\"call bpf_rcu_read_lock;\"\n+\t\"r1 = r6;\"\n+\"1:\"\t\"call shared_callee;\"\t\t/* cleanup region begins */\n+\t\"r1 = %[magic] ll;\"\t\t/* ... and ends on this pair */\n+\"2:\"\n+\t\"r6 = r0;\"\n+\t\"call bpf_rcu_read_unlock;\"\n+\t\"r0 = r6;\"\n+\t\"exit;\"\n+\"3:\"\t\t\t\t\t/* landing pad */\n+\t\"r7 = r0;\"\n+\t\"call bpf_rcu_read_unlock;\"\n+\tPAD_RAN(\"%[ran]\")\n+\t\"r1 = r7;\"\n+\t\"call bpf_unwind_resume;\"\n+\t\"exit;\"\n+\tCLEANUP_REC(\"1b\", \"2b\", \"3b\")\n+\t:\n+\t: [ran]\"i\"(RAN_WIDE_REC), __imm_addr(input), __imm_addr(magic),\n+\t __imm_addr(pads_ran)\n+\t: __clobber_all);\n+}\n+\n+SEC(\"syscall\")\n+int entry_wide_rec(void *ctx)\n+{\n+\treturn wide_rec_frame();\n+}\n+\n+/* 13. A pad that reloads from and stores to its own frame -- the shape every\n+ * compiler-generated pad has. A JIT that addresses the frame through the stack\n+ * pointer, arm64, has to find a pad's frame another way.\n+ */\n+static __used __naked __noinline __u64 pad_stack_frame(void)\n+{\n+\tasm volatile (\n+\t\"r1 = %[magic] ll;\"\n+\t\"r1 = *(u64 *)(r1 + 0);\"\n+\t\"*(u64 *)(r10 - 8) = r1;\"\t/* what the pad will want */\n+\t\"r1 = %[input] ll;\"\n+\t\"r1 = *(u64 *)(r1 + 0);\"\n+\"1:\"\t\"call pc_thrower;\"\t\t/* cleanup region */\n+\"2:\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\"3:\"\t\t\t\t\t/* landing pad */\n+\t\"r6 = r0;\"\n+\t\"r7 = *(u64 *)(r10 - 8);\"\t/* reload it out of the frame */\n+\t\"*(u64 *)(r10 - 16) = r7;\"\t/* and write the frame while here */\n+\t\"r1 = %[magic] ll;\"\n+\t\"r2 = *(u64 *)(r1 + 0);\"\n+\t\"if r7 != r2 goto 9f;\"\n+\t\"r3 = *(u64 *)(r10 - 16);\"\n+\t\"if r3 != r2 goto 9f;\"\n+\tPAD_RAN(\"%[ran]\")\n+\"9:\"\n+\t\"r1 = r6;\"\n+\t\"call bpf_unwind_resume;\"\n+\t\"exit;\"\n+\tCLEANUP_REC(\"1b\", \"2b\", \"3b\")\n+\t:\n+\t: [ran]\"i\"(RAN_PAD_STACK), __imm_addr(input), __imm_addr(magic),\n+\t __imm_addr(pads_ran)\n+\t: __clobber_all);\n+}\n+\n+SEC(\"syscall\")\n+int entry_pad_stack(void *ctx)\n+{\n+\treturn pad_stack_frame();\n+}\n+\n+/* 14. The same, with a frame in between that has no pad of its own, so the\n+ * liveness query for an outer frame has more than one frame to walk.\n+ */\n+static __used __noinline __u64 deep_mid(__u64 x)\n+{\n+\treturn pc_thrower(x) + 1;\n+}\n+\n+static __used __naked __noinline __u64 deep_frame(void)\n+{\n+\tasm volatile (\n+\t\"r1 = %[magic] ll;\"\n+\t\"r1 = *(u64 *)(r1 + 0);\"\n+\t\"*(u64 *)(r10 - 8) = r1;\"\t/* nothing but the pad reads this */\n+\t\"r1 = %[input] ll;\"\n+\t\"r1 = *(u64 *)(r1 + 0);\"\n+\"1:\"\t\"call deep_mid;\"\t\t/* cleanup region */\n+\"2:\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\"3:\"\t\t\t\t\t/* landing pad */\n+\t\"r6 = r0;\"\n+\t\"r7 = *(u64 *)(r10 - 8);\"\n+\t\"r1 = %[magic] ll;\"\n+\t\"r2 = *(u64 *)(r1 + 0);\"\n+\t\"if r7 != r2 goto 9f;\"\n+\tPAD_RAN(\"%[ran]\")\n+\"9:\"\n+\t\"r1 = r6;\"\n+\t\"call bpf_unwind_resume;\"\n+\t\"exit;\"\n+\tCLEANUP_REC(\"1b\", \"2b\", \"3b\")\n+\t:\n+\t: [ran]\"i\"(RAN_DEEP_PAD), __imm_addr(input), __imm_addr(magic),\n+\t __imm_addr(pads_ran)\n+\t: __clobber_all);\n+}\n+\n+SEC(\"syscall\")\n+int entry_deep_pad(void *ctx)\n+{\n+\treturn deep_frame();\n+}\n+\n+/* 15. A region around a call the kernel knows cannot unwind: no call site is\n+ * marked, nothing reaches the pad, and the sweep removes it. The program is\n+ * otherwise ordinary and has to load.\n+ */\n+static __used __naked __noinline __u64 nounwind_rec_frame(void)\n+{\n+\tasm volatile (\n+\t\"call bpf_preempt_disable;\"\n+\"1:\"\t\"call bpf_preempt_enable;\"\t/* cleanup region: nounwind */\n+\"2:\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\"3:\"\t\t\t\t\t/* landing pad, never dispatched */\n+\tPAD_RAN(\"%[ran]\")\n+\t\"call bpf_unwind_resume;\"\n+\t\"exit;\"\n+\tCLEANUP_REC(\"1b\", \"2b\", \"3b\")\n+\t:\n+\t: [ran]\"i\"(RAN_NOUNWIND_REC), __imm_addr(pads_ran)\n+\t: __clobber_all);\n+}\n+\n+SEC(\"syscall\")\n+int entry_nounwind_rec(void *ctx)\n+{\n+\treturn nounwind_rec_frame();\n+}\n+\n+/* 16. The name a frontend gives the resume: LLVM emits _Unwind_Resume() and\n+ * libbpf maps it onto bpf_unwind_resume(), which every other pad here calls.\n+ */\n+extern void _Unwind_Resume(void) __ksym;\n+\n+static __used __noinline void __resume_alias_btf_anchor(void)\n+{\n+\t_Unwind_Resume();\n+}\n+\n+static __used __naked __noinline __u64 resume_alias_frame(void)\n+{\n+\tasm volatile (\n+\"1:\"\t\"call regs_thrower;\"\t\t/* cleanup region */\n+\"2:\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\"3:\"\t\t\t\t\t/* landing pad */\n+\tPAD_RAN(\"%[ran]\")\n+\t\"call _Unwind_Resume;\"\t\t/* the frontend's name for it */\n+\t\"exit;\"\n+\tCLEANUP_REC(\"1b\", \"2b\", \"3b\")\n+\t:\n+\t: [ran]\"i\"(RAN_RESUME_ALIAS), __imm_addr(pads_ran)\n+\t: __clobber_all);\n+}\n+\n+SEC(\"syscall\")\n+int entry_resume_alias(void *ctx)\n+{\n+\treturn resume_alias_frame();\n+}\n+\n+/* 17. A landing pad that calls a subprogram which tail calls. A tail call in\n+ * the pad itself is refused, but the callee's prologue really did run, so its\n+ * tail call releases exactly that and the target returns into the pad.\n+ */\n+struct {\n+\t__uint(type, BPF_MAP_TYPE_PROG_ARRAY);\n+\t__uint(max_entries, 1);\n+\t__uint(key_size, sizeof(__u32));\n+\t__uint(value_size, sizeof(__u32));\n+} pad_tc_table SEC(\".maps\");\n+\n+__u64 pad_tc_target_ran = 0;\n+\n+SEC(\"syscall\")\n+int pad_tc_target(void *ctx)\n+{\n+\tpad_tc_target_ran += 1;\n+\treturn 0;\n+}\n+\n+static __used __noinline __u64 pad_tc_callee(void *ctx)\n+{\n+\t/* Taken only once the test has populated the array. */\n+\tbpf_tail_call_static(ctx, \u0026pad_tc_table, 0);\n+\treturn 0;\n+}\n+\n+/* The frame with the pad is the program itself, and __naked: r1 at entry is\n+ * the only place to get a context for bpf_tail_call(); the pad reloads it\n+ * from the frame's stack.\n+ */\n+SEC(\"syscall\")\n+__naked int entry_pad_tail_call(void)\n+{\n+\tasm volatile (\n+\t\"*(u64 *)(r10 - 8) = r1;\"\t/* the context, straight from entry */\n+\tLOAD_MAGIC_REGS\n+\t\"r1 = %[input] ll;\"\n+\t\"r1 = *(u64 *)(r1 + 0);\"\n+\t\"if r1 \u003c 101 goto 8f;\"\n+\"1:\"\t\"call regs_thrower;\"\t\t/* cleanup region */\n+\"2:\"\n+\"8:\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\"3:\"\t\t\t\t\t/* landing pad */\n+\tPAD_COUNT\n+\t\"r1 = *(u64 *)(r10 - 8);\"\n+\t\"call pad_tc_callee;\"\n+\t/* Only if the frame survived the callee's tail call. */\n+\tCHECK_MAGIC_REGS(\"%[ran]\")\n+\t\"call bpf_unwind_resume;\"\n+\t\"exit;\"\n+\tCLEANUP_REC(\"1b\", \"2b\", \"3b\")\n+\t:\n+\t: [ran]\"i\"(RAN_PAD_TAIL_CALL), __imm_addr(input),\n+\t __imm_addr(magic), __imm_addr(pads_ran), __imm_addr(pad_runs)\n+\t: __clobber_all);\n+}\n+\n+/* 18. The other target for that same tail call: a program with a table of its\n+ * own, throwing inside the outer exception. The tail call made it a boundary,\n+ * so the inner walk ends there and the outer pad, cookie and r6-r9 survive.\n+ */\n+__u64 tc_target_pad_runs = 0;\n+__u64 inner_magic = 0xd00d;\n+\n+static __used __naked __noinline __u64 inner_thrower(void)\n+{\n+\tasm volatile (\n+\t/* Not this frame's to keep, the same as regs_thrower. */\n+\t\"r6 = 0xf00d;\"\n+\t\"r7 = 0xcafe;\"\n+\t\"r8 = 0xbeef;\"\n+\t\"r9 = 0xdead;\"\n+\t\"r1 = %[cookie];\"\n+\t\"call bpf_throw;\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:\n+\t: [cookie]\"i\"(INNER_COOKIE)\n+\t: __clobber_all);\n+}\n+\n+SEC(\"syscall\")\n+__naked int pad_tc_throw_target(void)\n+{\n+\tasm volatile (\n+\t/* Distinct from the outer pad's, so neither can stand in for it. */\n+\t\"r1 = %[inner_magic] ll;\"\n+\t\"r6 = *(u64 *)(r1 + 0);\"\n+\t\"r7 = r6;\"\n+\t\"r7 += 1;\"\n+\t\"r8 = r6;\"\n+\t\"r8 += 2;\"\n+\t\"r9 = r6;\"\n+\t\"r9 += 3;\"\n+\t\"r1 = %[input] ll;\"\n+\t\"r1 = *(u64 *)(r1 + 0);\"\n+\t\"if r1 \u003c 101 goto 8f;\"\n+\"1:\"\t\"call inner_thrower;\"\t\t/* cleanup region */\n+\"2:\"\n+\"8:\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\"3:\"\t\t\t\t\t/* landing pad */\n+\t/* This frame's own r6-r9, not the outer pad's. */\n+\t\"r1 = %[inner_magic] ll;\"\n+\t\"r2 = *(u64 *)(r1 + 0);\"\n+\t\"if r6 != r2 goto 9f;\"\n+\t\"r2 += 1;\"\n+\t\"if r7 != r2 goto 9f;\"\n+\t\"r2 += 1;\"\n+\t\"if r8 != r2 goto 9f;\"\n+\t\"r2 += 1;\"\n+\t\"if r9 != r2 goto 9f;\"\n+\t\"r1 = %[tc_target_pad_runs] ll;\"\n+\t\"r2 = *(u64 *)(r1 + 0);\"\n+\t\"r2 += 1;\"\n+\t\"*(u64 *)(r1 + 0) = r2;\"\n+\"9:\"\n+\t\"call bpf_unwind_resume;\"\n+\t\"exit;\"\n+\tCLEANUP_REC(\"1b\", \"2b\", \"3b\")\n+\t:\n+\t: __imm_addr(input), __imm_addr(inner_magic),\n+\t __imm_addr(tc_target_pad_runs)\n+\t: __clobber_all);\n+}\n+\n+/* 19. A landing pad whose first instruction is a nop. opt_remove_nops() runs\n+ * long after the cleanup walk, so the record has to follow the pad to the\n+ * instruction that takes its place rather than be dropped with the nop.\n+ */\n+static __used __naked __noinline __u64 nop_pad_frame(void)\n+{\n+\tasm volatile (\n+\t\"r1 = %[input] ll;\"\n+\t\"r6 = *(u64 *)(r1 + 0);\"\n+\t\"if r6 \u003c 101 goto 6f;\"\n+\t\"r1 = %[cookie];\"\n+\"1:\"\t\"call bpf_throw;\"\t\t/* cleanup region */\n+\"2:\"\n+\"6:\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\"3:\"\t\t\t\t\t/* landing pad: a nop, then its body */\n+\t\"goto +0;\"\n+\tPAD_RAN(\"%[ran]\")\n+\t\"call bpf_unwind_resume;\"\n+\t\"exit;\"\n+\tCLEANUP_REC(\"1b\", \"2b\", \"3b\")\n+\t:\n+\t: [cookie]\"i\"(THROW_COOKIE), [ran]\"i\"(RAN_NOP_PAD),\n+\t __imm_addr(input), __imm_addr(pads_ran)\n+\t: __clobber_all);\n+}\n+\n+SEC(\"syscall\")\n+int entry_nop_pad(void *ctx)\n+{\n+\treturn nop_pad_frame();\n+}\n+\n+/* 20. A landing pad that indexes its own frame's stack by a register the frame\n+ * set before the throwing call. The offset is not a constant, so r6 has to be\n+ * marked precise from inside the pad, back across the unwind edge.\n+ */\n+/* A thrower that touches none of r6-r9, so nothing in it can answer for the\n+ * pad frame's r6 and the walk has to leave this frame to look.\n+ */\n+static __used __naked __noinline __u64 var_thrower(void)\n+{\n+\tasm volatile (\n+\t\"if r1 \u003c 101 goto 1f;\"\n+\t\"r1 = %[cookie];\"\n+\t\"call bpf_throw;\"\n+\"1:\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:\n+\t: [cookie]\"i\"(THROW_COOKIE)\n+\t: __clobber_all);\n+}\n+\n+static __used __naked __noinline __u64 var_stack_frame(void)\n+{\n+\tasm volatile (\n+\t\"r1 = %[magic] ll;\"\n+\t\"r1 = *(u64 *)(r1 + 0);\"\n+\t\"*(u64 *)(r10 - 8) = r1;\"\t/* the slot the pad will read... */\n+\t\"*(u64 *)(r10 - 16) = r1;\"\t/* ...whichever of the two it is */\n+\t\"r1 = %[input] ll;\"\n+\t\"r6 = *(u64 *)(r1 + 0);\"\n+\t\"r6 \u0026= 1;\"\t\t\t/* an unknown slot number... */\n+\t\"r6 \u003c\u003c= 3;\"\t\t\t/* ...as an aligned byte offset */\n+\t\"r1 = %[input] ll;\"\n+\t\"r1 = *(u64 *)(r1 + 0);\"\n+\"1:\"\t\"call var_thrower;\"\t\t/* cleanup region */\n+\"2:\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\"3:\"\t\t\t\t\t/* landing pad */\n+\t\"r7 = r0;\"\n+\t\"r1 = r10;\"\n+\t\"r1 += r6;\"\t\t\t/* variable offset into the frame */\n+\t\"r2 = *(u64 *)(r1 - 16);\"\n+\t\"r3 = %[magic] ll;\"\n+\t\"r3 = *(u64 *)(r3 + 0);\"\n+\t\"if r2 != r3 goto 9f;\"\n+\tPAD_RAN(\"%[ran]\")\n+\"9:\"\n+\t\"r1 = r7;\"\n+\t\"call bpf_unwind_resume;\"\n+\t\"exit;\"\n+\tCLEANUP_REC(\"1b\", \"2b\", \"3b\")\n+\t:\n+\t: [ran]\"i\"(RAN_VAR_STACK), __imm_addr(input), __imm_addr(magic),\n+\t __imm_addr(pads_ran)\n+\t: __clobber_all);\n+}\n+\n+SEC(\"syscall\")\n+int entry_var_stack(void *ctx)\n+{\n+\treturn var_stack_frame();\n+}\n+\n+/* 21. The same walk, over a call to a global subprogram: the verifier enters\n+ * no frame for one, so the exception arrives at the pad from the call site\n+ * rather than from a throw some frames deeper.\n+ */\n+__noinline __u64 global_thrower(__u64 x)\n+{\n+\tif (x \u003e 100)\n+\t\tbpf_throw(THROW_COOKIE);\n+\treturn x + 1;\n+}\n+\n+static __used __naked __noinline __u64 global_pad_frame(void)\n+{\n+\tasm volatile (\n+\t\"r1 = %[magic] ll;\"\n+\t\"r1 = *(u64 *)(r1 + 0);\"\n+\t\"*(u64 *)(r10 - 8) = r1;\"\n+\t\"*(u64 *)(r10 - 16) = r1;\"\n+\t\"r1 = %[input] ll;\"\n+\t\"r6 = *(u64 *)(r1 + 0);\"\n+\t\"r6 \u0026= 1;\"\n+\t\"r6 \u003c\u003c= 3;\"\n+\t\"r1 = %[input] ll;\"\n+\t\"r1 = *(u64 *)(r1 + 0);\"\n+\"1:\"\t\"call global_thrower;\"\t\t/* cleanup region */\n+\"2:\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\"3:\"\t\t\t\t\t/* landing pad */\n+\t\"r7 = r0;\"\n+\t\"r1 = r10;\"\n+\t\"r1 += r6;\"\n+\t\"r2 = *(u64 *)(r1 - 16);\"\n+\t\"r3 = %[magic] ll;\"\n+\t\"r3 = *(u64 *)(r3 + 0);\"\n+\t\"if r2 != r3 goto 9f;\"\n+\tPAD_RAN(\"%[ran]\")\n+\"9:\"\n+\t\"r1 = r7;\"\n+\t\"call bpf_unwind_resume;\"\n+\t\"exit;\"\n+\tCLEANUP_REC(\"1b\", \"2b\", \"3b\")\n+\t:\n+\t: [ran]\"i\"(RAN_GLOBAL_PAD), __imm_addr(input), __imm_addr(magic),\n+\t __imm_addr(pads_ran)\n+\t: __clobber_all);\n+}\n+\n+SEC(\"syscall\")\n+int entry_global_pad(void *ctx)\n+{\n+\treturn global_pad_frame();\n+}\n+\n+char _license[] SEC(\"license\") = \"GPL\";\n"
} |
| 2/1 | 2026/09/22 09:05 | agent | patch-evaluator |
2mModel:core Args: null Results: {
"EnableConfigs": [
"BPF_SYSCALL",
"BPF_JIT",
"UNWINDER_ORC",
"DEBUG_INFO_BTF"
],
"FocusSymbols": [
"bpf_prepare_cleanup_exceptions",
"bpf_check_cleanup_exceptions",
"bpf_cleanup_attach_main_prog",
"bpf_cleanup_fill_native_ranges",
"bpf_cleanup_pad_for_ip",
"bpf_unwind_resume",
"arch_bpf_run_cleanup_pad",
"bpf_throw"
],
"Reasoning": "The patch series implements BPF exception cleanup landing pad support across core BPF (verifier CFG, reachability, liveness, backtracking, dead-code elimination) and architecture-specific JIT/unwind dispatch on x86-64 and arm64. This adds new user-facing fields in union bpf_attr for BPF_PROG_LOAD, verifier checks, kfuncs (bpf_unwind_resume), runtime stack unwinding, and assembly dispatchers. The changes touch core subsystems (kernel/bpf and arch/x86/net) and are fully reachable in virtualized environments via the bpf() syscall. Therefore, this warrants fuzzing.",
"WorthFuzzing": true
}Instruction: You are an expert Linux kernel maintainer and security engineer.
Your job is to review a provided patch series and evaluate whether it warrants fuzzing with syzkaller.
IMPORTANT: The changes have ALREADY been applied and committed as the HEAD commit in
your workspace. Do NOT rely on internal assumptions. You must actively use your code access
tools to inspect the actual source code, callers, and surrounding context.
================================================================================
1. CORE TRIAGE PHILOSOPHY
================================================================================
The goal of patch fuzzing is to discover crashes, regressions, exposed latent bugs,
and newly triggered assertions introduced by the patch series.
- REACHABILITY IS THE PRIMARY GATE:
Fuzzing can only discover bugs in code that can actually execute in standard virtualized
environments (GCE or QEMU, utilizing software-emulated devices like USB gadgets, netdev, tun/tap).
If the modified code is structurally unreachable (see Section 2), it MUST NOT be fuzzed,
regardless of whether it adds assertions or complex logic.
- DO NOT BLINDLY TRUST "NO FUNCTIONAL CHANGE" (NFCI) OR "REFACTORING" CLAIMS:
Patch authors routinely label changes as "cleanups", "refactorings", or state
"No functional change intended". Do NOT take these claims at face value.
Code refactorings that rearrange logic, introduce helper functions, or alter state management
in core subsystems frequently introduce subtle semantic shifts or uncover latent kernel bugs.
If reachable executable code is modified or refactored, it MUST be fuzzed.
- NEW OR MODIFIED ASSERTIONS IN REACHABLE CODE MUST BE FUZZED:
When a patch introduces or modifies runtime checks or assertions (e.g., WARN_ON*, VM_WARN_ON*,
BUG_ON*, lockdep_assert*) in reachable code paths, it enforces new or stricter invariants.
Even if the author believes the invariant always holds, fuzzing is essential to verify whether
an unusual sequence of operations can violate it.
================================================================================
2. WHEN TO RETURN WorthFuzzing=false (NEGATIVE CRITERIA)
================================================================================
Return WorthFuzzing=false ONLY IF all modified code falls strictly into one or more of these categories:
- Non-kernel and non-executable changes:
* Modifications to Documentation/, comments, or spelling fixes.
* User-space directories, self-tests, samples, or scripts (e.g., tools/, samples/, scripts/, usr/)
that do not affect the compiled kernel image (vmlinux) or kernel modules.
* Purely decorative logging (e.g., message strings in pr_err, printk, dev_info) or tracepoints
that do not alter control flow or data structures.
* Build system or Kconfig changes that do not alter compiled C logic.
- Structurally unreachable hardware:
* Vendor-specific PCIe switches, SmartNICs, or GPU drivers (e.g., mlxsw, pds_core, qed,
ionic, amdgpu) requiring physical ASIC/PCIe cards not emulated in standard QEMU.
- Unreachable execution paths:
* Driver teardown callbacks (.remove, .shutdown, pci_unregister_driver) executed only during
physical PCI hot-unplug or manual sysfs driver unbinding.
* Code paths exclusive to architectures other than the target architecture.
================================================================================
3. WHEN TO RETURN WorthFuzzing=true (POSITIVE CRITERIA)
================================================================================
Return WorthFuzzing=true whenever the patch touches reachable executable code, including:
- Core Subsystems:
* Any logic modifications in memory management (mm/), synchronization/locking (kernel/locking/),
BPF, scheduler, core networking, VFS, or syscall handling.
- Refactorings and Code Cleanups:
* Any restructuring of reachable data structures, helper abstractions, or algorithm flows.
- Runtime Assertions and Defensive Checks:
* Any introduction or alteration of assertions (WARN_ON*, VM_WARN_ON*, BUG_ON*, etc.) in reachable paths.
- Reachable Drivers and Protocols:
* Drivers accessible via virtual buses (virtio, USB gadget, loopback, netlink, binder, sockets, etc.).
================================================================================
4. EXTRACTING FocusSymbols (PREVENTING DILUTION)
================================================================================
When WorthFuzzing=true, you must extract specific kernel functions into FocusSymbols to guide the fuzzer:
- AVOID UBIQUITOUS LIFECYCLE HOT-PATHS:
Do NOT list generic, ubiquitous functions called by almost every program in the corpus
(including, but not limited to: general memory allocators and deallocators, page fault
and trap handlers, or core synchronization primitives; this is not an exhaustive list).
Listing ubiquitous functions causes the fuzzer to classify thousands of unrelated tests as "focused",
which severely dilutes fuzzing effort away from the actual changes.
- TARGET SPECIFIC FEATURE LOGIC AND ENTRYPOINTS:
List functions that specifically implement the logic being added or altered, or direct API entrypoints
for the subsystem feature under review.
- HANDLING STATIC INLINE FUNCTIONS IN HEADERS (.h):
Compiler-inlined static functions (such as static inlines in mm/*.h or include/linux/*.h) lack
distinct symbol addresses in vmlinux and cannot be targeted directly by symbol coverage filters.
If the changes are primarily in static inline helpers, identify non-static, feature-specific caller
functions in .c files that exercise them (avoiding ubiquitous lifecycle wrappers).
================================================================================
5. IDENTIFYING EnableConfigs
================================================================================
Identify any specific CONFIG_ options required to properly compile and reach the modified code:
- Inspect Kconfig files and #ifdef guards; do not make assumptions.
- Check "depends on" lines in Kconfig to include any non-standard parent subsystem configs needed.
- Strip any 'CONFIG_' prefix (e.g., return "NET_IPV4" instead of "CONFIG_NET_IPV4").
Prefer calling several tools at the same time to save round-trips.
Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.
Prompt: Target architecture: amd64
For your convenience, here is the diff of the changes:
commit 68c10631eef7b88304d24e3ffb2d009f8db416fc
Author: syz-cluster <triage@syzkaller.com>
Date: Tue Sep 22 09:05:55 2026 +0000
syz-cluster: applied patch under review
diff --git a/arch/arm64/net/Makefile b/arch/arm64/net/Makefile
index 3ae382bfca879..ebec2a44a52b6 100644
--- a/arch/arm64/net/Makefile
+++ b/arch/arm64/net/Makefile
@@ -2,4 +2,4 @@
#
# ARM64 networking code
#
-obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o bpf_timed_may_goto.o
+obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o bpf_timed_may_goto.o bpf_cleanup_pad.o
diff --git a/arch/arm64/net/bpf_cleanup_pad.S b/arch/arm64/net/bpf_cleanup_pad.S
new file mode 100644
index 0000000000000..ef441241949ec
--- /dev/null
+++ b/arch/arm64/net/bpf_cleanup_pad.S
@@ -0,0 +1,95 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+#include <linux/bpf_cleanup_abi.h>
+#include <linux/linkage.h>
+
+/*
+ * A frame's prologue pushes the tail call counter pair, and then -- once
+ * bpf_cleanup_force_spill() says so -- x19/x20, x21/x22, x23/x24, x25/x26 and
+ * x27/x28. Each A64_PUSH pre-decrements, so the lowest address of the spill
+ * area holds x27 and the highest x20:
+ *
+ * spill_base + 0 x27 (private stack pointer)
+ * spill_base + 8 x28 (arena base)
+ * spill_base + 16 x25 (BPF r10, the frame pointer)
+ * spill_base + 24 x26 (tail call counter pointer)
+ * spill_base + 32 x23 -- the pad's own, see below
+ * spill_base + 40 x24 -- likewise
+ * spill_base + 48 x21 (BPF r8)
+ * spill_base + 56 x22 (BPF r9)
+ * spill_base + 64 x19 (BPF r6)
+ * spill_base + 72 x20 (BPF r7)
+ *
+ * Neither x23 nor x24 is restored from that spill: the pad has its own use for
+ * both. bpf2a64[] maps nothing to either -- they are pushed only to keep the
+ * frame shape the exception callback expects -- so nothing else in generated
+ * code touches them, and being callee-saved they survive every call the pad
+ * makes.
+ *
+ * x23 is the pad's return address. Unlike x86-64 a pad cannot simply return:
+ * every call it makes clobbers x30, so nothing is left to return through by
+ * the time it reaches its resume. The JIT emits "br x23" for the pad's
+ * bpf_unwind_resume() and this routine puts .Lcleanup_pad_done there.
+ *
+ * x24 is where the pad's frame is anchored. Generated code addresses the BPF
+ * frame through the stack pointer, which here is this routine's rather than
+ * the unwinding frame's, so the JIT has the pad recompute the equivalent from
+ * BPF r10 on entry and address its frame off x24 for as long as it runs.
+ *
+ * Both of those branches are indirect, so both targets carry a BTI landing
+ * marker: the JIT emits one at each pad, and .Lcleanup_pad_done below has one
+ * of its own.
+ */
+
+ .text
+
+/*
+ * void arch_bpf_run_cleanup_pad(u64 pad, u64 frame_fp, u64 spill_base)
+ *
+ * x0 = native address of the landing pad
+ * x1 = frame pointer of the frame the pad belongs to (unused here: BPF r10 is
+ * x25, which the spill area already holds)
+ * x2 = spill area holding that frame's BPF callee-saved registers
+ *
+ * Give the pad the register state of its own frame and call it. It runs on
+ * this stack, far below the frame it is cleaning up after, so nothing it
+ * calls can reach into that frame.
+ */
+SYM_FUNC_START(arch_bpf_run_cleanup_pad)
+ /* Save the kernel's callee-saved registers; the pad owns them next. */
+ stp x29, x30, [sp, #-96]!
+ mov x29, sp
+ stp x19, x20, [sp, #16]
+ stp x21, x22, [sp, #32]
+ stp x23, x24, [sp, #48]
+ stp x25, x26, [sp, #64]
+ stp x27, x28, [sp, #80]
+
+ /* x9 is BPF_REG_AX, so the pad's address does not stay in BPF r1. */
+ mov x9, x0
+
+ ldp x27, x28, [x2, #0]
+ ldp x25, x26, [x2, #16]
+ ldp x21, x22, [x2, #48]
+ ldp x19, x20, [x2, #64]
+
+ /* BPF r0 (x8) on the way into a pad, not whatever the kernel left. */
+ mov x8, #BPF_PAD_ENTRY_R0
+
+ /* Where the pad's resume branches back to. */
+ adr x23, .Lcleanup_pad_done
+
+ br x9
+
+.Lcleanup_pad_done:
+ /* Reached by the pad's "br x23", so it is an indirect branch target. */
+ bti j
+ ldp x19, x20, [sp, #16]
+ ldp x21, x22, [sp, #32]
+ ldp x23, x24, [sp, #48]
+ ldp x25, x26, [sp, #64]
+ ldp x27, x28, [sp, #80]
+ ldp x29, x30, [sp], #96
+ ret
+SYM_FUNC_END(arch_bpf_run_cleanup_pad)
diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
index 6c04fee468766..7d26e43e29c3a 100644
--- a/arch/arm64/net/bpf_jit_comp.c
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -11,6 +11,7 @@
#include <linux/bitfield.h>
#include <linux/bpf.h>
#include <linux/cfi.h>
+#include <linux/bpf_verifier.h>
#include <linux/filter.h>
#include <linux/memory.h>
#include <linux/printk.h>
@@ -75,6 +76,19 @@ static const int bpf2a64[] = {
[ARENA_VM_START] = A64_R(28),
};
+/*
+ * Throw-site spill: the five pairs push_callee_regs() forces on, same size and
+ * slot order, so arch_bpf_run_cleanup_pad() reads both alike.
+ */
+#define A64_CLEANUP_SPILL_SZ (5 * 16)
+
+/*
+ * Where a landing pad's frame is anchored, since the stack pointer generated
+ * code normally addresses it through is the walker's inside a pad. bpf2a64[]
+ * maps nothing to x24, so nothing else in generated code touches it.
+ */
+#define A64_CLEANUP_FP A64_R(24)
+
struct jit_ctx {
const struct bpf_prog *prog;
int idx;
@@ -87,6 +101,8 @@ struct jit_ctx {
__le32 *ro_image;
u32 stack_size;
u16 stack_arg_size;
+ /* Bytes reserved for the throw-site spill; see bpf_cleanup_force_spill(). */
+ u32 throw_spill;
u64 user_vm_start;
u64 arena_vm_start;
bool fp_used;
@@ -432,7 +448,7 @@ static void push_callee_regs(struct jit_ctx *ctx)
* Callee-saved registers as the exception callback needs to recover
* all ARM64 Callee-saved registers in its epilogue.
*/
- if (ctx->prog->aux->exception_boundary) {
+ if (ctx->prog->aux->exception_boundary || bpf_cleanup_force_spill(ctx->prog)) {
emit(A64_PUSH(A64_R(19), A64_R(20), A64_SP), ctx);
emit(A64_PUSH(A64_R(21), A64_R(22), A64_SP), ctx);
emit(A64_PUSH(A64_R(23), A64_R(24), A64_SP), ctx);
@@ -466,7 +482,8 @@ static void pop_callee_regs(struct jit_ctx *ctx)
* program's stack frame, so recover these extra registers in the above
* two cases.
*/
- if (aux->exception_boundary || aux->exception_cb) {
+ if (aux->exception_boundary || aux->exception_cb ||
+ bpf_cleanup_force_spill(ctx->prog)) {
emit(A64_POP(A64_R(27), A64_R(28), A64_SP), ctx);
emit(A64_POP(A64_R(25), A64_R(26), A64_SP), ctx);
emit(A64_POP(A64_R(23), A64_R(24), A64_SP), ctx);
@@ -602,6 +619,20 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)
emit(A64_SUB_I(1, A64_SP, A64_FP, 96), ctx);
}
+ /*
+ * Lowest address of each spill area, as an offset from A64_FP; see
+ * bpf_cleanup_pad.S for the layout. The 16 is the tail call counter
+ * pair pushed just below the frame record, and the throw-site area
+ * sits below the callee-saved one rather than in the program stack.
+ */
+ if (bpf_cleanup_force_spill(prog)) {
+ prog->aux->exc->spill_off = -(16 + A64_CLEANUP_SPILL_SZ);
+ ctx->throw_spill = A64_CLEANUP_SPILL_SZ;
+ emit(A64_SUB_I(1, A64_SP, A64_SP, ctx->throw_spill), ctx);
+ prog->aux->exc->throw_spill_off =
+ -(16 + A64_CLEANUP_SPILL_SZ) - ctx->throw_spill;
+ }
+
/* Stack must be multiples of 16B */
ctx->stack_size = round_up(prog->aux->stack_depth, 16);
@@ -691,6 +722,10 @@ static int emit_bpf_tail_call(struct jit_ctx *ctx)
if (ctx->stack_size && !ctx->priv_sp_used)
emit(A64_ADD_I(1, A64_SP, A64_SP, ctx->stack_size), ctx);
+ /* Release it for the same reason build_epilogue() does. */
+ if (ctx->throw_spill)
+ emit(A64_ADD_I(1, A64_SP, A64_SP, ctx->throw_spill), ctx);
+
pop_callee_regs(ctx);
/* goto *(prog->bpf_func + prologue_offset); */
@@ -1055,6 +1090,9 @@ static void build_epilogue(struct jit_ctx *ctx, bool was_classic)
if (ctx->stack_size && !ctx->priv_sp_used)
emit(A64_ADD_I(1, A64_SP, A64_SP, ctx->stack_size), ctx);
+ if (ctx->throw_spill)
+ emit(A64_ADD_I(1, A64_SP, A64_SP, ctx->throw_spill), ctx);
+
pop_callee_regs(ctx);
emit(A64_POP(A64_ZR, ptr, A64_SP), ctx);
@@ -1367,6 +1405,8 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
const s16 off = insn->off;
const s32 imm = insn->imm;
const int i = insn - ctx->prog->insnsi;
+ const bool in_pad = bpf_cleanup_insn_in_pad(ctx->prog, i);
+ const bool pad_head = bpf_cleanup_insn_is_pad(ctx->prog, i);
const bool is64 = BPF_CLASS(code) == BPF_ALU64 ||
BPF_CLASS(code) == BPF_JMP;
u8 jmp_cond;
@@ -1378,9 +1418,13 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
int ret;
bool sign_extend;
- if (bpf_insn_is_indirect_target(env, ctx->prog, i))
+ if (bpf_insn_is_indirect_target(env, ctx->prog, i) || pad_head)
emit_bti(A64_BTI_J, ctx);
+ if (pad_head)
+ emit(A64_SUB_I(1, A64_CLEANUP_FP, fp,
+ ctx->stack_size + ctx->stack_arg_size), ctx);
+
switch (code) {
/* dst = src */
case BPF_ALU | BPF_MOV | BPF_X:
@@ -1743,6 +1787,26 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
u64 func_addr;
u32 cpu_offset;
+ if (bpf_cleanup_insn_is_throw(ctx->prog, i)) {
+ /* Spill where the bpf_throw() walker looks. */
+ const s32 off = ctx->prog->aux->exc->throw_spill_off;
+
+ emit(A64_SUB_I(1, tmp, A64_FP, -off), ctx);
+ emit(A64_STR64I(bpf2a64[PRIVATE_SP], tmp, 0), ctx);
+ emit(A64_STR64I(bpf2a64[ARENA_VM_START], tmp, 8), ctx);
+ emit(A64_STR64I(bpf2a64[BPF_REG_FP], tmp, 16), ctx);
+ emit(A64_STR64I(bpf2a64[TCCNT_PTR], tmp, 24), ctx);
+ emit(A64_STR64I(bpf2a64[BPF_REG_8], tmp, 48), ctx);
+ emit(A64_STR64I(bpf2a64[BPF_REG_9], tmp, 56), ctx);
+ emit(A64_STR64I(bpf2a64[BPF_REG_6], tmp, 64), ctx);
+ emit(A64_STR64I(bpf2a64[BPF_REG_7], tmp, 72), ctx);
+ }
+
+ if (bpf_cleanup_insn_is_resume(ctx->prog, i)) {
+ emit(A64_BR(A64_R(23)), ctx);
+ break;
+ }
+
/* Implement helper call to bpf_get_smp_processor_id() inline */
if (insn->src_reg == 0 && insn->imm == BPF_FUNC_get_smp_processor_id) {
cpu_offset = offsetof(struct thread_info, cpu);
@@ -1854,7 +1918,8 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
src = tmp2;
}
if (src == fp) {
- src_adj = ctx->priv_sp_used ? priv_sp : A64_SP;
+ src_adj = ctx->priv_sp_used ? priv_sp :
+ in_pad ? A64_CLEANUP_FP : A64_SP;
off_adj = off + ctx->stack_size;
if (!ctx->priv_sp_used)
off_adj += ctx->stack_arg_size;
@@ -1952,7 +2017,8 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
dst = tmp3;
}
if (dst == fp) {
- dst_adj = ctx->priv_sp_used ? priv_sp : A64_SP;
+ dst_adj = ctx->priv_sp_used ? priv_sp :
+ in_pad ? A64_CLEANUP_FP : A64_SP;
off_adj = off + ctx->stack_size;
if (!ctx->priv_sp_used)
off_adj += ctx->stack_arg_size;
@@ -2021,7 +2087,8 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
dst = tmp2;
}
if (dst == fp) {
- dst_adj = ctx->priv_sp_used ? priv_sp : A64_SP;
+ dst_adj = ctx->priv_sp_used ? priv_sp :
+ in_pad ? A64_CLEANUP_FP : A64_SP;
off_adj = off + ctx->stack_size;
if (!ctx->priv_sp_used)
off_adj += ctx->stack_arg_size;
@@ -2410,6 +2477,13 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
* reasons, expects to point to the next instruction)
*/
bpf_prog_update_insn_ptrs(prog, ctx.offset, ctx.ro_image);
+
+ /*
+ * Same byte offsets, consumed by the bpf_throw() frame walker:
+ * turn the cleanup records into native address ranges now that
+ * the image is final.
+ */
+ bpf_cleanup_fill_native_ranges(prog, ctx.offset, ctx.ro_image);
out_off:
if (!ro_header && priv_stack_ptr) {
free_percpu(priv_stack_ptr);
@@ -3385,6 +3459,11 @@ bool bpf_jit_supports_exceptions(void)
return true;
}
+bool bpf_jit_supports_cleanup_pads(void)
+{
+ return true;
+}
+
bool bpf_jit_supports_arena(void)
{
return true;
diff --git a/arch/x86/net/Makefile b/arch/x86/net/Makefile
index dddbefc0f4398..8bb22644ba10b 100644
--- a/arch/x86/net/Makefile
+++ b/arch/x86/net/Makefile
@@ -7,4 +7,7 @@ ifeq ($(CONFIG_X86_32),y)
obj-$(CONFIG_BPF_JIT) += bpf_jit_comp32.o
else
obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o bpf_timed_may_goto.o
+ ifdef CONFIG_UNWINDER_ORC
+ obj-$(CONFIG_BPF_JIT) += bpf_cleanup_pad.o
+ endif
endif
diff --git a/arch/x86/net/bpf_cleanup_pad.S b/arch/x86/net/bpf_cleanup_pad.S
new file mode 100644
index 0000000000000..025a54e718a40
--- /dev/null
+++ b/arch/x86/net/bpf_cleanup_pad.S
@@ -0,0 +1,80 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+#include <linux/bpf_cleanup_abi.h>
+#include <linux/linkage.h>
+#include <asm/nospec-branch.h>
+
+/*
+ * The x86-64 BPF JIT prologue spills, once bpf_cleanup_force_spill() makes it
+ * unconditional, r12, rbx, r13, r14 and r15 in that order -- so within the
+ * spill area the lowest address holds r15 and the highest r12. The throw-site
+ * spill the JIT emits uses the same layout, so the routine below reads both
+ * the same way:
+ *
+ * spill_base + 0 BPF r9 (r15)
+ * spill_base + 8 BPF r8 (r14)
+ * spill_base + 16 BPF r7 (r13)
+ * spill_base + 24 BPF r6 (rbx)
+ * spill_base + 32 r12 (arena base, not a BPF register)
+ */
+
+ .code64
+ .section .text, "ax"
+
+/*
+ * void arch_bpf_run_cleanup_pad(u64 pad, u64 frame_fp, u64 spill_base)
+ *
+ * rdi = native address of the landing pad
+ * rsi = frame pointer of the frame the pad belongs to
+ * rdx = spill area holding that frame's BPF callee-saved registers
+ *
+ * Give the pad the register state of its own frame and call it. The pad ends
+ * in the bare return the JIT emits for its bpf_unwind_resume(), so it comes
+ * back here rather than returning to its frame's caller. It runs on this
+ * stack, far below the frame it is cleaning up after, so nothing it calls can
+ * reach into that frame.
+ *
+ * rbp addresses the pad's frame and rsp this one, so the two are not the
+ * neighbours a JITed frame expects them to be. That is why
+ * cleanup_check_pad_insn() refuses a call in a pad that passes an argument on
+ * the stack: the JIT stages those relative to rbp, and the callee reads them
+ * relative to rsp.
+ */
+SYM_FUNC_START(arch_bpf_run_cleanup_pad)
+ ANNOTATE_NOENDBR
+
+ pushq %rbp
+ pushq %rbx
+ pushq %r12
+ pushq %r13
+ pushq %r14
+ pushq %r15
+ /* Keep the pad's entry rsp congruent to a normal call's. */
+ subq $8, %rsp
+
+ movq 0(%rdx), %r15
+ movq 8(%rdx), %r14
+ movq 16(%rdx), %r13
+ movq 24(%rdx), %rbx
+ movq 32(%rdx), %r12
+ /* rbp is BPF r10, so this is the whole of the pad's frame setup. */
+ movq %rsi, %rbp
+
+ /* CALL_NOSPEC needs the target in a register; rcx is BPF r4, dead. */
+ movq %rdi, %rcx
+
+ /* BPF r0 on the way into a pad, not whatever the kernel left in rax. */
+ movl $BPF_PAD_ENTRY_R0, %eax
+
+ CALL_NOSPEC rcx
+
+ addq $8, %rsp
+ popq %r15
+ popq %r14
+ popq %r13
+ popq %r12
+ popq %rbx
+ popq %rbp
+ RET
+SYM_FUNC_END(arch_bpf_run_cleanup_pad)
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index d4a980140b48d..04db7d7fc397e 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -357,6 +357,12 @@ struct jit_context {
/* Number of bytes that will be skipped on tailcall */
#define X86_TAIL_CALL_OFFSET (12 + ENDBR_INSN_SIZE)
+/*
+ * Throw-site spill: r15, r14, r13, rbx, r12 low to high, the layout the
+ * prologue's pushes leave, so arch_bpf_run_cleanup_pad() reads both alike.
+ */
+#define X86_CLEANUP_SPILL_SZ (5 * 8)
+
static void push_r9(u8 **pprog)
{
u8 *prog = *pprog;
@@ -832,7 +838,7 @@ static void emit_bpf_tail_call_indirect(struct bpf_prog *bpf_prog,
/* Inc tail_call_cnt if the slot is populated. */
EMIT4(0x48, 0x83, 0x00, 0x01); /* add qword ptr [rax], 1 */
- if (bpf_prog->aux->exception_boundary) {
+ if (bpf_prog->aux->exception_boundary || bpf_cleanup_force_spill(bpf_prog)) {
pop_callee_regs(&prog, all_callee_regs_used);
pop_r12(&prog);
} else {
@@ -899,7 +905,7 @@ static void emit_bpf_tail_call_direct(struct bpf_prog *bpf_prog,
/* Inc tail_call_cnt if the slot is populated. */
EMIT4(0x48, 0x83, 0x00, 0x01); /* add qword ptr [rax], 1 */
- if (bpf_prog->aux->exception_boundary) {
+ if (bpf_prog->aux->exception_boundary || bpf_cleanup_force_spill(bpf_prog)) {
pop_callee_regs(&prog, all_callee_regs_used);
pop_r12(&prog);
} else {
@@ -1977,6 +1983,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
u8 *ip, *prog = temp;
u32 stack_depth;
int callee_saved_size;
+ u32 throw_spill, prologue_depth;
s32 outgoing_arg_base;
int err;
@@ -2015,7 +2022,10 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
detect_reg_usage(insn, insn_cnt, callee_regs_used);
- emit_prologue(&prog, image, stack_depth,
+ throw_spill = bpf_cleanup_force_spill(bpf_prog) ? X86_CLEANUP_SPILL_SZ : 0;
+ prologue_depth = stack_depth + throw_spill;
+
+ emit_prologue(&prog, image, prologue_depth,
bpf_prog_was_classic(bpf_prog), tail_call_reachable,
bpf_is_subprog(bpf_prog), bpf_prog->aux->exception_cb);
@@ -2024,7 +2034,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
/* Exception callback will clobber callee regs for its own use, and
* restore the original callee regs from main prog's stack frame.
*/
- if (bpf_prog->aux->exception_boundary) {
+ if (bpf_prog->aux->exception_boundary || bpf_cleanup_force_spill(bpf_prog)) {
/* We also need to save r12, which is not mapped to any BPF
* register, as we throw after entry into the kernel, which may
* overwrite r12.
@@ -2039,9 +2049,10 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
/* Compute callee-saved register area size. */
callee_saved_size = 0;
- if (bpf_prog->aux->exception_boundary || arena_vm_start)
+ if (bpf_prog->aux->exception_boundary || bpf_cleanup_force_spill(bpf_prog) ||
+ arena_vm_start)
callee_saved_size += 8; /* r12 */
- if (bpf_prog->aux->exception_boundary) {
+ if (bpf_prog->aux->exception_boundary || bpf_cleanup_force_spill(bpf_prog)) {
callee_saved_size += 4 * 8; /* rbx, r13, r14, r15 */
} else {
int j;
@@ -2063,7 +2074,19 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
* Note that tail_call_reachable is guaranteed to be false when
* stack args exist, so tcc pushes need not be accounted for.
*/
- outgoing_arg_base = -(round_up(stack_depth, 8) + callee_saved_size);
+ outgoing_arg_base = -(round_up(stack_depth, 8) + throw_spill + callee_saved_size);
+
+ /*
+ * Lowest address of each spill area, as an offset from rbp; see
+ * bpf_cleanup_pad.S for the layout. The 16 is the tail call counter
+ * pair emit_prologue_tail_call() pushes above the callee-saved one.
+ */
+ if (bpf_cleanup_force_spill(bpf_prog)) {
+ bpf_prog->aux->exc->spill_off = -(round_up(stack_depth, 8) + throw_spill +
+ (tail_call_reachable ? 16 : 0) +
+ callee_saved_size);
+ bpf_prog->aux->exc->throw_spill_off = -(round_up(stack_depth, 8) + throw_spill);
+ }
/*
* Allocate outgoing stack arg area for args 7+ only.
@@ -2110,7 +2133,8 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
dst_reg = X86_REG_R9;
}
- if (bpf_insn_is_indirect_target(env, bpf_prog, i - 1))
+ if (bpf_insn_is_indirect_target(env, bpf_prog, i - 1) ||
+ bpf_cleanup_insn_is_pad(bpf_prog, i - 1))
EMIT_ENDBR();
ip = image + addrs[i - 1] + (prog - temp);
@@ -2903,9 +2927,27 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
case BPF_JMP | BPF_CALL: {
const struct btf_func_model *fm = NULL;
+ if (bpf_cleanup_insn_is_throw(bpf_prog, i - 1)) {
+ /* Spill r6-r9 and r12 where the bpf_throw() walker looks. */
+ s32 off = bpf_prog->aux->exc->throw_spill_off;
+ u8 *spill = prog;
+
+ emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_9, off + 0);
+ emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_8, off + 8);
+ emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_7, off + 16);
+ emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_6, off + 24);
+ emit_stx(&prog, BPF_DW, BPF_REG_FP, X86_REG_R12, off + 32);
+ ip += prog - spill;
+ }
+
+ if (bpf_cleanup_insn_is_resume(bpf_prog, i - 1)) {
+ emit_return(&prog, image + addrs[i - 1] + (prog - temp));
+ break;
+ }
+
func = (u8 *) __bpf_call_base + imm32;
if (src_reg == BPF_PSEUDO_CALL && tail_call_reachable) {
- LOAD_TAIL_CALL_CNT_PTR(stack_depth);
+ LOAD_TAIL_CALL_CNT_PTR(prologue_depth);
ip += 7;
}
if (!imm32)
@@ -2948,13 +2990,13 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
&prog,
ip,
callee_regs_used,
- stack_depth,
+ prologue_depth,
ctx);
else
emit_bpf_tail_call_indirect(bpf_prog,
&prog,
callee_regs_used,
- stack_depth,
+ prologue_depth,
ip,
ctx);
break;
@@ -3215,7 +3257,8 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
}
/* Deallocate outgoing args 7+ area. */
emit_add_rsp(&prog, outgoing_rsp);
- if (bpf_prog->aux->exception_boundary) {
+ if (bpf_prog->aux->exception_boundary ||
+ bpf_cleanup_force_spill(bpf_prog)) {
pop_callee_regs(&prog, all_callee_regs_used);
pop_r12(&prog);
} else {
@@ -4385,6 +4428,13 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
*/
bpf_prog_update_insn_ptrs(prog, addrs, image);
+ /*
+ * Same mapping, consumed by the bpf_throw() frame walker:
+ * turn the cleanup records into native address ranges now
+ * that the image is final.
+ */
+ bpf_cleanup_fill_native_ranges(prog, addrs, image);
+
/*
* ctx.prog_offset is used when CFI preambles put code *before*
* the function. See emit_cfi(). For FineIBT specifically this code
@@ -4501,6 +4551,11 @@ bool bpf_jit_supports_exceptions(void)
return IS_ENABLED(CONFIG_UNWINDER_ORC);
}
+bool bpf_jit_supports_cleanup_pads(void)
+{
+ return IS_ENABLED(CONFIG_UNWINDER_ORC);
+}
+
bool bpf_jit_supports_private_stack(void)
{
return true;
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 5033b934ffd90..0502b9fc4a6de 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -1774,6 +1774,89 @@ enum bpf_sig_keyring {
BPF_SIG_KEYRING_BPF,
};
+/*
+ * One cleanup region of a JITed (sub)program: @pad is the landing pad to run
+ * for a return address in (begin, end], the native code of its call sites.
+ */
+struct bpf_cleanup_range {
+ u64 begin;
+ u64 end;
+ u64 pad;
+};
+
+struct bpf_exception_info {
+ struct bpf_cleanup_info *info;
+ struct bpf_cleanup_range *ranges;
+ /* Landing pad instruction indices, sorted and deduplicated. */
+ u32 *pad_at;
+ /* bpf_throw() call instruction indices, sorted. */
+ u32 *throw_at;
+ /* Likewise for bpf_unwind_resume(), the way back out of a pad. */
+ u32 *resume_at;
+ /* One bit per instruction that only runs while unwinding. */
+ unsigned long *pad_body;
+ u32 nr_info;
+ u32 nr_ranges;
+ u32 nr_pad_at;
+ u32 nr_throw_at;
+ u32 nr_resume_at;
+ u32 nr_pad_body;
+ /* Offset from a frame's FP to the caller's spilled r6-r9. */
+ s32 spill_off;
+ /* Likewise, to the registers a frame spills before calling bpf_throw(). */
+ s32 throw_spill_off;
+};
+
+#ifdef CONFIG_BPF_SYSCALL
+bool bpf_cleanup_force_spill(const struct bpf_prog *prog);
+bool bpf_cleanup_insn_is_pad(const struct bpf_prog *prog, u32 idx);
+bool bpf_cleanup_insn_in_pad(const struct bpf_prog *prog, u32 idx);
+bool bpf_cleanup_insn_is_throw(const struct bpf_prog *prog, u32 idx);
+bool bpf_cleanup_insn_is_resume(const struct bpf_prog *prog, u32 idx);
+int bpf_cleanup_attach_main_prog(struct bpf_verifier_env *env, struct bpf_prog *prog);
+void bpf_cleanup_fill_native_ranges(struct bpf_prog *prog, u32 *addrs, void *image);
+void bpf_cleanup_free_info(struct bpf_prog_aux *aux);
+#else
+static inline bool bpf_cleanup_force_spill(const struct bpf_prog *prog)
+{
+ return false;
+}
+
+static inline bool bpf_cleanup_insn_is_pad(const struct bpf_prog *prog, u32 idx)
+{
+ return false;
+}
+
+static inline bool bpf_cleanup_insn_in_pad(const struct bpf_prog *prog, u32 idx)
+{
+ return false;
+}
+
+static inline bool bpf_cleanup_insn_is_throw(const struct bpf_prog *prog, u32 idx)
+{
+ return false;
+}
+
+static inline bool bpf_cleanup_insn_is_resume(const struct bpf_prog *prog, u32 idx)
+{
+ return false;
+}
+
+static inline int bpf_cleanup_attach_main_prog(struct bpf_verifier_env *env,
+ struct bpf_prog *prog)
+{
+ return 0;
+}
+
+static inline void bpf_cleanup_fill_native_ranges(struct bpf_prog *prog, u32 *addrs, void *image)
+{
+}
+
+static inline void bpf_cleanup_free_info(struct bpf_prog_aux *aux)
+{
+}
+#endif
+
struct bpf_prog_aux {
atomic64_t refcnt;
u32 used_map_cnt;
@@ -1854,6 +1937,7 @@ struct bpf_prog_aux {
char name[BPF_OBJ_NAME_LEN];
u64 (*bpf_exception_cb)(u64 cookie, u64 sp, u64 bp, u64, u64);
u16 stack_arg_sp_adjust;
+ struct bpf_exception_info *exc;
#ifdef CONFIG_SECURITY
void *security;
#endif
diff --git a/include/linux/bpf_cleanup_abi.h b/include/linux/bpf_cleanup_abi.h
new file mode 100644
index 0000000000000..b6c1d589abda7
--- /dev/null
+++ b/include/linux/bpf_cleanup_abi.h
@@ -0,0 +1,16 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#ifndef _LINUX_BPF_CLEANUP_ABI_H
+#define _LINUX_BPF_CLEANUP_ABI_H
+
+/*
+ * Value arch_bpf_run_cleanup_pad() leaves in r0 on the way into a landing pad.
+ * It has to be a constant the verifier knows: LLVM names r0 as both the
+ * exception pointer and the exception selector register, so every pad reads it
+ * before anything else and is free to store what it read. Kept on its own
+ * because the verifier and the arch dispatchers, which are assembly, have to
+ * agree on it.
+ */
+#define BPF_PAD_ENTRY_R0 1
+
+#endif /* _LINUX_BPF_CLEANUP_ABI_H */
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index cf85141ea1674..0721461aaabbf 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -429,7 +429,8 @@ struct bpf_jmp_history_entry {
u32 prev_idx : 20;
/* special INSN_F_xxx flags */
u32 flags : 4;
- u32 : 8;
+ u32 unwind_frames : 4; /* frames the unwind popped to get here */
+ u32 : 4;
/*
* additional registers that need precision tracking when this
* jump is backtracked, vector of five 11-bit records
@@ -509,6 +510,8 @@ struct bpf_verifier_state {
bool speculative;
bool in_sleepable;
+ bool unwinding; /* an exception is in flight */
+ u8 unwind_frameno; /* the frame whose landing pad it entered */
/* first and last insn idx of this verifier state */
u32 first_insn_idx;
@@ -681,6 +684,14 @@ struct bpf_insn_aux_data {
bool needs_zext; /* alu op needs to clear upper bits */
bool non_sleepable; /* helper/kfunc may be called from non-sleepable context */
bool is_iter_next; /* bpf_iter_<type>_next() kfunc call */
+ bool cleanup_throw_site; /* call to bpf_throw() */
+ bool cleanup_resume_site; /* call to bpf_unwind_resume() */
+ bool in_cleanup_pad; /* only runs with an exception in flight */
+ /*
+ * 1 + the instruction index of the exception cleanup landing pad this
+ * call site unwinds to, or 0 for none.
+ */
+ u32 cleanup_pad;
bool call_with_percpu_alloc_ptr; /* {this,per}_cpu_ptr() with prog percpu alloc */
u8 alu_state; /* used in combination with alu_limit */
/* true if STX or LDX instruction is a part of a spill/fill
@@ -983,10 +994,13 @@ struct bpf_verifier_env {
} cfg;
struct backtrack_state bt;
struct bpf_jmp_history_entry *cur_hist_ent;
+ u8 unwind_frames; /* scratch: pops for the insn about to be recorded */
/* Per-callsite copy of parent's converged at_stack_in for cross-frame fills. */
struct arg_track **callsite_at_stack;
u32 pass_cnt; /* number of times do_check() was called */
u32 subprog_cnt;
+ struct bpf_cleanup_info *cleanup_info;
+ u32 cleanup_info_cnt;
/* number of instructions analyzed by the verifier */
u32 prev_insn_processed, insn_processed;
/* number of jmps, calls, exits analyzed so far */
@@ -1516,6 +1530,7 @@ u32 btf_func_arg_align(const struct btf *btf, const struct btf_type *t);
int bpf_find_subprog(struct bpf_verifier_env *env, int off);
bool bpf_is_throw_kfunc(struct bpf_insn *insn);
+bool bpf_is_unwind_resume_kfunc(const struct bpf_insn *insn);
int bpf_compute_const_regs(struct bpf_verifier_env *env);
int bpf_prune_dead_branches(struct bpf_verifier_env *env);
int bpf_check_cfg(struct bpf_verifier_env *env);
diff --git a/include/linux/filter.h b/include/linux/filter.h
index 422284b4fa96f..b0c495e94f740 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -1242,6 +1242,8 @@ bool bpf_jit_supports_stack_args(void);
bool bpf_jit_supports_arena_args(void);
bool bpf_jit_supports_far_kfunc_call(void);
bool bpf_jit_supports_exceptions(void);
+bool bpf_jit_supports_cleanup_pads(void);
+void arch_bpf_run_cleanup_pad(u64 pad, u64 frame_fp, u64 spill_base);
bool bpf_jit_supports_ptr_xchg(void);
bool bpf_jit_supports_arena(void);
bool bpf_jit_supports_insn(struct bpf_insn *insn, bool in_arena);
diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
index 6330b7d745c57..a69cc04275751 100644
--- a/include/uapi/linux/bpf.h
+++ b/include/uapi/linux/bpf.h
@@ -1669,6 +1669,9 @@ union bpf_attr {
* verification.
*/
__s32 keyring_id;
+ __aligned_u64 cleanup_info; /* exception cleanup table */
+ __u32 cleanup_info_rec_size; /* userspace bpf_cleanup_info size */
+ __u32 cleanup_info_cnt; /* number of bpf_cleanup_info records */
};
struct { /* anonymous struct used by BPF_OBJ_* commands */
@@ -7599,6 +7602,12 @@ struct bpf_line_info {
__u32 line_col;
};
+struct bpf_cleanup_info {
+ __u32 begin_off;
+ __u32 end_off;
+ __u32 landing_pad_off;
+};
+
struct bpf_spin_lock {
__u32 val;
};
diff --git a/kernel/bpf/Makefile b/kernel/bpf/Makefile
index 9a92c348bbda6..af9bc60428ad1 100644
--- a/kernel/bpf/Makefile
+++ b/kernel/bpf/Makefile
@@ -11,7 +11,7 @@ obj-$(CONFIG_BPF_SYSCALL) += bpf_iter.o map_iter.o task_iter.o prog_iter.o link_
obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o bloom_filter.o
obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o bpf_insn_array.o
obj-$(CONFIG_BPF_SYSCALL) += bpf_local_storage.o bpf_task_storage.o
-obj-$(CONFIG_BPF_SYSCALL) += fixups.o cfg.o states.o backtrack.o check_btf.o
+obj-$(CONFIG_BPF_SYSCALL) += fixups.o cfg.o states.o backtrack.o check_btf.o exception.o
obj-${CONFIG_BPF_LSM} += bpf_inode_storage.o
obj-$(CONFIG_BPF_SYSCALL) += disasm.o mprog.o
obj-$(CONFIG_BPF_JIT) += trampoline.o
diff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c
index 507a366dffa47..c19e4fe247ce5 100644
--- a/kernel/bpf/backtrack.c
+++ b/kernel/bpf/backtrack.c
@@ -4,6 +4,7 @@
#include <linux/bpf_verifier.h>
#include <linux/filter.h>
#include <linux/bitmap.h>
+#include "exception.h"
#define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
@@ -47,6 +48,7 @@ int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state
p->flags = insn_flags;
p->spi = spi;
p->frame = frame;
+ p->unwind_frames = 0;
p->linked_regs = linked_regs;
cur->jmp_history_cnt = cnt;
env->cur_hist_ent = p;
@@ -419,10 +421,12 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
* extra instructions from subprog; the next
* instruction after call to global subprog
* should be literally next instruction in
- * caller program
+ * caller program -- or, if the callee threw,
+ * the landing pad of this call site
*/
- verifier_bug_if(idx + 1 != subseq_idx, env,
- "extra insn from subprog");
+ verifier_bug_if(idx + 1 != subseq_idx &&
+ bpf_cleanup_pad_of_call(env, idx) != subseq_idx,
+ env, "extra insn from subprog");
/* global subprog always sets R0 */
bt_clear_reg(bt, BPF_REG_0);
/* and if it does not set R2, main pass would catch it */
@@ -888,11 +892,11 @@ int bpf_mark_chain_precision(struct bpf_verifier_env *env,
}
for (i = last_idx;;) {
+ hist = get_jmp_hist_entry(st, history, i);
if (skip_first) {
err = 0;
skip_first = false;
} else {
- hist = get_jmp_hist_entry(st, history, i);
err = backtrack_insn(env, i, subseq_idx, hist, bt);
}
if (err == -ENOTSUPP) {
@@ -909,6 +913,15 @@ int bpf_mark_chain_precision(struct bpf_verifier_env *env,
*/
return 0;
subseq_idx = i;
+ /* An exception unwind reached this insn, a landing
+ * pad, from a throw or a resume that many frames
+ * deeper. There is no insn in between to backtrack
+ * over, so enter those frames here, the way BPF_EXIT
+ * does one at a time.
+ */
+ for (fr = 0; hist && fr < hist->unwind_frames; fr++)
+ if (bt_subprog_enter(bt))
+ return -EFAULT;
i = get_prev_insn_idx(st, i, &history);
if (i == -ENOENT)
break;
diff --git a/kernel/bpf/cfg.c b/kernel/bpf/cfg.c
index 842c7d1eabccc..3fd98dcadce89 100644
--- a/kernel/bpf/cfg.c
+++ b/kernel/bpf/cfg.c
@@ -6,6 +6,7 @@
#include <linux/sort.h>
#include "diagnostics.h"
+#include "exception.h"
#define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
@@ -158,17 +159,64 @@ static int push_insn(int t, int w, int e, struct bpf_verifier_env *env)
return DONE_EXPLORING;
}
+static int visit_cleanup_pad_edge(int t, struct bpf_verifier_env *env)
+{
+ int *insn_stack = env->cfg.insn_stack;
+ int *insn_state = env->cfg.insn_state;
+ int w;
+
+ if (!env->cleanup_info_cnt)
+ return DONE_EXPLORING;
+ w = bpf_cleanup_pad_of_call(env, t);
+ if (w < 0)
+ return DONE_EXPLORING;
+
+ /*
+ * @t is a call that may branch here, and @w is the target of that
+ * branch, so both are prune points. @w especially: every covered call
+ * site in a region unwinds to the same pad, and without a prune point
+ * at its head the verifier walks the pad again for each of them.
+ */
+ mark_prune_point(env, t);
+ mark_prune_point(env, w);
+ mark_jmp_point(env, w);
+ mark_jump_target(env, w);
+
+ if (insn_state[w])
+ return DONE_EXPLORING;
+ if (env->cfg.cur_stack >= env->prog->len)
+ return -E2BIG;
+ insn_stack[env->cfg.cur_stack++] = w;
+ insn_state[w] |= DISCOVERED;
+ return KEEP_EXPLORING;
+}
+
+static int merge_visit_ret(int a, int b)
+{
+ if (a < 0)
+ return a;
+ if (b < 0)
+ return b;
+ if (a == KEEP_EXPLORING || b == KEEP_EXPLORING)
+ return KEEP_EXPLORING;
+ return DONE_EXPLORING;
+}
+
static int visit_func_call_insn(int t, struct bpf_insn *insns,
struct bpf_verifier_env *env,
bool visit_callee)
{
- int ret, insn_sz;
+ int ret, insn_sz, pad_ret;
int w;
+ pad_ret = visit_cleanup_pad_edge(t, env);
+ if (pad_ret < 0)
+ return pad_ret;
+
insn_sz = bpf_is_ldimm64(&insns[t]) ? 2 : 1;
ret = push_insn(t, t + insn_sz, FALLTHROUGH, env);
if (ret)
- return ret;
+ return merge_visit_ret(pad_ret, ret);
mark_prune_point(env, t + insn_sz);
/* when we exit from subprog, we need to record non-linear history */
@@ -180,7 +228,7 @@ static int visit_func_call_insn(int t, struct bpf_insn *insns,
merge_callee_effects(env, t, w);
ret = push_insn(t, w, BRANCH, env);
}
- return ret;
+ return merge_visit_ret(pad_ret, ret);
}
struct bpf_iarray *bpf_iarray_realloc(struct bpf_iarray *old, size_t n_elem)
@@ -592,6 +640,7 @@ int bpf_check_cfg(struct bpf_verifier_env *env)
int insn_cnt = env->prog->len;
int *insn_stack, *insn_state;
int ex_insn_beg, i, ret = 0;
+ u32 pad_idx = 0;
insn_state = env->cfg.insn_state = kvzalloc_objs(int, insn_cnt,
GFP_KERNEL_ACCOUNT);
@@ -647,6 +696,22 @@ int bpf_check_cfg(struct bpf_verifier_env *env)
goto walk_cfg;
}
+ /*
+ * A landing pad no call site was marked with -- a record whose range
+ * holds no call an exception can unwind out of -- is reached by
+ * nothing. Walk it from here, and let the dead code sweep remove it.
+ */
+ while (pad_idx < env->cleanup_info_cnt) {
+ u32 pad = env->cleanup_info[pad_idx++].landing_pad_off;
+
+ if (insn_state[pad] != EXPLORED) {
+ insn_state[pad] = DISCOVERED;
+ insn_stack[0] = pad;
+ env->cfg.cur_stack = 1;
+ goto walk_cfg;
+ }
+ }
+
for (i = 0; i < insn_cnt; i++) {
struct bpf_insn *insn = &env->prog->insnsi[i];
diff --git a/kernel/bpf/check_btf.c b/kernel/bpf/check_btf.c
index 0e8b3ccc7a5b9..d03dc791042a2 100644
--- a/kernel/bpf/check_btf.c
+++ b/kernel/bpf/check_btf.c
@@ -407,6 +407,149 @@ static int check_core_relo(struct bpf_verifier_env *env,
return err;
}
+static int cleanup_insn_subprog(struct bpf_verifier_env *env, u32 off)
+{
+ struct bpf_subprog_info *info;
+
+ if (off >= env->prog->len)
+ return -1;
+ info = bpf_find_containing_subprog(env, off);
+ return info ? info - env->subprog_info : -1;
+}
+
+#define MIN_BPF_CLEANUP_INFO_SIZE 12
+#define MAX_CLEANUP_INFO_REC_SIZE MAX_FUNCINFO_REC_SIZE
+
+static int check_cleanup_info(struct bpf_verifier_env *env,
+ const union bpf_attr *attr,
+ bpfptr_t uattr)
+{
+ u32 krec_size = sizeof(struct bpf_cleanup_info);
+ u32 i, nrec, urec_size, min_size, prev_end = 0;
+ struct bpf_cleanup_info *krecord;
+ bpfptr_t urecord;
+ int ret = -EINVAL;
+
+ nrec = attr->cleanup_info_cnt;
+ if (!nrec)
+ return 0;
+ if (nrec > INT_MAX / krec_size)
+ return -EINVAL;
+
+ urec_size = attr->cleanup_info_rec_size;
+ if (urec_size < MIN_BPF_CLEANUP_INFO_SIZE ||
+ urec_size > MAX_CLEANUP_INFO_REC_SIZE ||
+ urec_size % sizeof(u32)) {
+ verbose(env, "invalid cleanup info rec size %u\n", urec_size);
+ return -EINVAL;
+ }
+
+ krecord = kvcalloc(nrec, krec_size, GFP_KERNEL_ACCOUNT | __GFP_NOWARN);
+ if (!krecord)
+ return -ENOMEM;
+
+ min_size = min_t(u32, krec_size, urec_size);
+ urecord = make_bpfptr(attr->cleanup_info, uattr.is_kernel);
+ for (i = 0; i < nrec; i++) {
+ struct bpf_cleanup_info *rec = &krecord[i];
+ int sb, se, sl;
+
+ ret = bpf_check_uarg_tail_zero(urecord, krec_size, urec_size);
+ if (ret) {
+ if (ret == -E2BIG) {
+ verbose(env, "nonzero tailing record in cleanup info\n");
+ if (copy_to_bpfptr_offset(uattr,
+ offsetof(union bpf_attr,
+ cleanup_info_rec_size),
+ &min_size, sizeof(min_size)))
+ ret = -EFAULT;
+ }
+ goto err_free;
+ }
+
+ if (copy_from_bpfptr(rec, urecord, min_size)) {
+ ret = -EFAULT;
+ goto err_free;
+ }
+ bpfptr_add(&urecord, urec_size);
+
+ ret = -EINVAL;
+ if (rec->begin_off >= rec->end_off) {
+ verbose(env, "cleanup_info[%u]: begin %u >= end %u\n",
+ i, rec->begin_off, rec->end_off);
+ goto err_free;
+ }
+ if (i && rec->begin_off < prev_end) {
+ verbose(env,
+ "cleanup_info[%u]: range [%u,%u) is unsorted or overlaps the previous record\n",
+ i, rec->begin_off, rec->end_off);
+ goto err_free;
+ }
+ prev_end = rec->end_off;
+
+ sb = cleanup_insn_subprog(env, rec->begin_off);
+ se = cleanup_insn_subprog(env, rec->end_off - 1);
+ sl = cleanup_insn_subprog(env, rec->landing_pad_off);
+ if (sb < 0 || se < 0 || sl < 0) {
+ verbose(env, "cleanup_info[%u]: offset out of range\n", i);
+ goto err_free;
+ }
+ if (sb != se || sb != sl) {
+ verbose(env,
+ "cleanup_info[%u]: range/landing pad span multiple subprogs\n",
+ i);
+ goto err_free;
+ }
+ /*
+ * The second half of a 16-byte instruction carries a zero
+ * opcode and is not an instruction of its own, so no offset
+ * may name one. end_off is exclusive, so it may also be one
+ * past the last instruction of the program.
+ */
+ if (!env->prog->insnsi[rec->begin_off].code ||
+ !env->prog->insnsi[rec->landing_pad_off].code ||
+ (rec->end_off < env->prog->len &&
+ !env->prog->insnsi[rec->end_off].code)) {
+ verbose(env, "cleanup_info[%u]: points at invalid insn\n", i);
+ goto err_free;
+ }
+ }
+
+ /*
+ * Reject a landing pad that lies inside a call-site range, its own
+ * included: it would be both a pad and a call that unwinds to one, and
+ * an exception out of it would have nowhere to go.
+ */
+ ret = -EINVAL;
+ for (i = 0; i < nrec; i++) {
+ u32 pad = krecord[i].landing_pad_off;
+ u32 l = 0, r = nrec;
+
+ while (l < r) {
+ u32 m = l + (r - l) / 2;
+
+ if (pad < krecord[m].begin_off) {
+ r = m;
+ } else if (pad >= krecord[m].end_off) {
+ l = m + 1;
+ } else {
+ verbose(env,
+ "cleanup_info[%u]: landing pad %u is inside the call-site range of cleanup_info[%u]\n",
+ i, pad, m);
+ goto err_free;
+ }
+ }
+ }
+
+ env->cleanup_info = krecord;
+ env->cleanup_info_cnt = nrec;
+ return 0;
+
+err_free:
+ kvfree(krecord);
+ return ret;
+}
+
int bpf_prepare_btf_info(struct bpf_verifier_env *env,
const union bpf_attr *attr,
bpfptr_t uattr)
@@ -441,6 +584,10 @@ int bpf_check_btf_info(struct bpf_verifier_env *env,
{
int err;
+ err = check_cleanup_info(env, attr, uattr);
+ if (err)
+ return err;
+
if (!attr->func_info_cnt && !attr->line_info_cnt) {
if (check_abnormal_return(env))
return -EINVAL;
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 227211166dccf..f19712be12a7d 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -292,6 +292,7 @@ void __bpf_prog_free(struct bpf_prog *fp)
mutex_destroy(&fp->aux->dst_mutex);
mutex_destroy(&fp->aux->st_ops_assoc_mutex);
kfree(fp->aux->poke_tab);
+ bpf_cleanup_free_info(fp->aux);
kfree(fp->aux);
}
free_percpu(fp->stats);
@@ -2625,13 +2626,21 @@ static bool bpf_prog_select_interpreter(struct bpf_prog *fp)
return select_interpreter;
}
-static struct bpf_prog *bpf_prog_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog)
+static struct bpf_prog *bpf_prog_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog,
+ int *err)
{
#ifdef CONFIG_BPF_JIT
struct bpf_prog *orig_prog;
+ int ret;
- if (!bpf_prog_need_blind(prog))
+ if (!bpf_prog_need_blind(prog)) {
+ ret = bpf_cleanup_attach_main_prog(env, prog);
+ if (ret) {
+ *err = ret;
+ return prog;
+ }
return bpf_int_jit_compile(env, prog);
+ }
orig_prog = prog;
prog = bpf_jit_blind_constants(env, prog);
@@ -2642,6 +2651,13 @@ static struct bpf_prog *bpf_prog_jit_compile(struct bpf_verifier_env *env, struc
if (IS_ERR(prog))
goto out_restore;
+ ret = bpf_cleanup_attach_main_prog(env, prog);
+ if (ret) {
+ *err = ret;
+ bpf_jit_prog_release_other(orig_prog, prog);
+ goto out_restore;
+ }
+
prog = bpf_int_jit_compile(env, prog);
if (prog->jited) {
bpf_jit_prog_release_other(prog, orig_prog);
@@ -2681,8 +2697,10 @@ struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct
if (*err)
return fp;
- fp = bpf_prog_jit_compile(env, fp);
+ fp = bpf_prog_jit_compile(env, fp, err);
bpf_prog_jit_attempt_done(fp);
+ if (*err)
+ return fp;
if (!fp->jited && jit_needed) {
*err = -ENOTSUPP;
return fp;
@@ -3475,6 +3493,17 @@ void __weak arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp,
{
}
+bool __weak bpf_jit_supports_cleanup_pads(void)
+{
+ return false;
+}
+
+/* Call @pad with the frame pointer @frame_fp and r6-r9 spilled at @spill_base. */
+void __weak arch_bpf_run_cleanup_pad(u64 pad, u64 frame_fp, u64 spill_base)
+{
+ WARN_ON_ONCE(1);
+}
+
bool __weak bpf_jit_supports_timed_may_goto(void)
{
return false;
diff --git a/kernel/bpf/exception.c b/kernel/bpf/exception.c
new file mode 100644
index 0000000000000..d4dcbcf4fb257
--- /dev/null
+++ b/kernel/bpf/exception.c
@@ -0,0 +1,664 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <linux/bitmap.h>
+#include <linux/bpf.h>
+#include <linux/bpf_verifier.h>
+#include <linux/bsearch.h>
+#include <linux/btf.h>
+#include <linux/btf_ids.h>
+#include <linux/filter.h>
+#include <linux/slab.h>
+#include <linux/sort.h>
+#include "exception.h"
+
+#define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
+
+enum exc_kfunc {
+ EXC_KF_bpf_unwind_resume,
+};
+
+BTF_ID_LIST(exc_kfunc_list)
+BTF_ID(func, bpf_unwind_resume)
+
+static bool insn_is_exc_kfunc(const struct bpf_insn *insn, int kf)
+{
+ return bpf_pseudo_kfunc_call(insn) && insn->off == 0 &&
+ insn->imm == exc_kfunc_list[kf];
+}
+
+/* What an instruction does to intra-subprog control flow. */
+enum cleanup_insn_kind {
+ CLEANUP_INSN_PLAIN, /* the next insn runs */
+ CLEANUP_INSN_JUMP, /* unconditional jump */
+ CLEANUP_INSN_COND, /* the next insn runs, or the branch target */
+ CLEANUP_INSN_EXIT,
+ CLEANUP_INSN_THROW, /* call bpf_throw: nothing after it runs */
+ CLEANUP_INSN_RESUME, /* call bpf_unwind_resume: likewise */
+ CLEANUP_INSN_CALL, /* call to another subprog */
+ CLEANUP_INSN_GOTOX, /* indirect jump: successors not known here */
+};
+
+/* What each instruction can reach, computed once by cleanup_reachability(). */
+#define CLEANUP_REACH_RESUME BIT(0) /* a bpf_unwind_resume() call */
+#define CLEANUP_REACH_EXIT BIT(1) /* a plain BPF_EXIT */
+#define CLEANUP_REACH_UNKNOWN BIT(2) /* an indirect jump */
+#define CLEANUP_REACH_THROW BIT(3) /* a bpf_throw() call */
+
+/* Scratch shared by the analyses, sized once so no walker has to allocate. */
+struct cleanup_ctx {
+ struct bpf_verifier_env *env;
+ u8 *reach; /* per insn: CLEANUP_REACH_* mask */
+ u32 *stack; /* per insn: DFS stack */
+ void *scratch; /* the one allocation all of the above live in */
+};
+
+static bool in_pad(struct bpf_verifier_env *env, u32 i)
+{
+ return env->insn_aux_data[i].in_cleanup_pad;
+}
+
+/* One scratch array for cleanup_alloc() to hand out. */
+struct cleanup_alloc_req {
+ void **dst;
+ size_t n, sz;
+};
+
+static void *cleanup_alloc(const struct cleanup_alloc_req *tab, u32 cnt)
+{
+ size_t total = 0;
+ char *block, *p;
+ u32 i;
+
+ for (i = 0; i < cnt; i++)
+ total += round_up(tab[i].n * tab[i].sz, 8);
+
+ block = kvzalloc(total, GFP_KERNEL_ACCOUNT | __GFP_NOWARN);
+ if (!block)
+ return NULL;
+
+ for (i = 0, p = block; i < cnt; i++) {
+ *tab[i].dst = p;
+ p += round_up(tab[i].n * tab[i].sz, 8);
+ }
+ return block;
+}
+
+static int cleanup_subprog_of(struct bpf_verifier_env *env, u32 off)
+{
+ struct bpf_subprog_info *info = bpf_find_containing_subprog(env, off);
+
+ return info ? info - env->subprog_info : -1;
+}
+
+/* The subprogram a linear pass is currently in. */
+struct cleanup_cursor {
+ u32 start, end; /* [start, end) of the current subprogram */
+ int sub; /* its index */
+};
+
+#define CLEANUP_CURSOR_INIT { .sub = -1 }
+
+static void cleanup_cursor_to(struct bpf_verifier_env *env, struct cleanup_cursor *c, u32 i)
+{
+ while (i >= c->end) {
+ c->sub++;
+ c->start = env->subprog_info[c->sub].start;
+ c->end = env->subprog_info[c->sub + 1].start;
+ }
+}
+
+static enum cleanup_insn_kind cleanup_classify(struct bpf_verifier_env *env, u32 i,
+ int *next, int *target)
+{
+ struct bpf_insn *insn = &env->prog->insnsi[i];
+ u8 class = BPF_CLASS(insn->code);
+
+ *next = i + 1;
+ *target = -1;
+
+ if (insn->code == (BPF_LD | BPF_IMM | BPF_DW)) {
+ *next = i + 2;
+ return CLEANUP_INSN_PLAIN;
+ }
+ if (class != BPF_JMP && class != BPF_JMP32)
+ return CLEANUP_INSN_PLAIN;
+
+ switch (BPF_OP(insn->code)) {
+ case BPF_EXIT:
+ *next = -1;
+ return CLEANUP_INSN_EXIT;
+ case BPF_JA:
+ *next = -1;
+ if (BPF_SRC(insn->code) == BPF_X)
+ return CLEANUP_INSN_GOTOX;
+ *target = class == BPF_JMP32 ? i + insn->imm + 1 : i + insn->off + 1;
+ return CLEANUP_INSN_JUMP;
+ case BPF_CALL:
+ if (bpf_is_throw_kfunc(insn)) {
+ *next = -1;
+ return CLEANUP_INSN_THROW;
+ }
+ if (insn_is_exc_kfunc(insn, EXC_KF_bpf_unwind_resume)) {
+ *next = -1;
+ return CLEANUP_INSN_RESUME;
+ }
+ return bpf_pseudo_call(insn) ? CLEANUP_INSN_CALL : CLEANUP_INSN_PLAIN;
+ default:
+ /* Conditional jump, including BPF_JCOND. */
+ *target = i + insn->off + 1;
+ return CLEANUP_INSN_COND;
+ }
+}
+
+static void cleanup_mark_kfunc_sites(struct bpf_verifier_env *env)
+{
+ u32 i;
+
+ for (i = 0; i < env->prog->len; i++) {
+ struct bpf_insn *insn = &env->prog->insnsi[i];
+
+ if (bpf_is_throw_kfunc(insn))
+ env->insn_aux_data[i].cleanup_throw_site = true;
+ else if (insn_is_exc_kfunc(insn, EXC_KF_bpf_unwind_resume))
+ env->insn_aux_data[i].cleanup_resume_site = true;
+ }
+}
+
+int bpf_cleanup_check_callback(struct bpf_verifier_env *env, int subprog)
+{
+ if (!env->cleanup_info_cnt || !env->subprog_info[subprog].might_throw)
+ return 0;
+
+ verbose(env, "subprog %d may unwind and is used as a callback\n", subprog);
+ return -EINVAL;
+}
+
+/* Intra-subprog successors of @i, or -1 each when absent. */
+static enum cleanup_insn_kind cleanup_succ(struct bpf_verifier_env *env, u32 i,
+ u32 start, u32 end, int *next, int *target)
+{
+ enum cleanup_insn_kind kind = cleanup_classify(env, i, next, target);
+
+ if (*next < (int)start || *next >= (int)end)
+ *next = -1;
+ if (*target < (int)start || *target >= (int)end)
+ *target = -1;
+ return kind;
+}
+
+static void cleanup_add_pred(u32 *head, u32 *link, u32 to, u32 e)
+{
+ link[e] = head[to];
+ head[to] = e + 1;
+}
+
+/* What every instruction can reach along intra-subprog edges, for
+ * cleanup_pad_is_catch(). One backward walk over a predecessor index, rather
+ * than a forward walk from each landing pad, which would be quadratic.
+ */
+static int cleanup_reachability(struct cleanup_ctx *ctx)
+{
+ struct bpf_verifier_env *env = ctx->env;
+ u32 len = env->prog->len;
+ struct cleanup_cursor c = CLEANUP_CURSOR_INIT;
+ u32 *head = NULL, *link = NULL;
+ bool *queued = NULL;
+ u32 i, sp = 0;
+ void *scratch;
+ const struct cleanup_alloc_req tab[] = {
+ { (void **)&head, len, sizeof(*head) },
+ { (void **)&link, 2 * (size_t)len, sizeof(*link) },
+ { (void **)&queued, len, sizeof(*queued) },
+ };
+
+ scratch = cleanup_alloc(tab, ARRAY_SIZE(tab));
+ if (!scratch)
+ return -ENOMEM;
+
+ /* Index the predecessors, and seed the walk at the terminators. */
+ for (i = 0; i < len; i++) {
+ enum cleanup_insn_kind kind;
+ int next, target;
+
+ cleanup_cursor_to(env, &c, i);
+ kind = cleanup_succ(env, i, c.start, c.end, &next, &target);
+
+ if (kind == CLEANUP_INSN_RESUME)
+ ctx->reach[i] |= CLEANUP_REACH_RESUME;
+ else if (kind == CLEANUP_INSN_EXIT)
+ ctx->reach[i] |= CLEANUP_REACH_EXIT;
+ else if (kind == CLEANUP_INSN_GOTOX)
+ ctx->reach[i] |= CLEANUP_REACH_UNKNOWN;
+ else if (kind == CLEANUP_INSN_THROW)
+ ctx->reach[i] |= CLEANUP_REACH_THROW;
+
+ if (next >= 0)
+ cleanup_add_pred(head, link, next, 2 * i);
+ if (target >= 0)
+ cleanup_add_pred(head, link, target, 2 * i + 1);
+
+ if (ctx->reach[i]) {
+ queued[i] = true;
+ ctx->stack[sp++] = i;
+ }
+ }
+
+ /* Each instruction re-enters the worklist at most once per bit it
+ * gains, so this is linear in the number of edges.
+ */
+ while (sp) {
+ u32 j = ctx->stack[--sp];
+ u8 flags = ctx->reach[j];
+ u32 e;
+
+ queued[j] = false;
+ for (e = head[j]; e; e = link[e - 1]) {
+ u32 p = (e - 1) / 2;
+
+ if ((ctx->reach[p] | flags) == ctx->reach[p])
+ continue;
+ ctx->reach[p] |= flags;
+ if (!queued[p]) {
+ queued[p] = true;
+ ctx->stack[sp++] = p;
+ }
+ }
+ }
+ kvfree(scratch);
+ return 0;
+}
+
+static int cleanup_pad_is_catch(struct cleanup_ctx *ctx, u32 pad)
+{
+ u8 reach = ctx->reach[pad];
+
+ if (reach & CLEANUP_REACH_UNKNOWN) {
+ verbose(ctx->env, "cleanup landing pad %u reaches an indirect jump\n", pad);
+ return -EINVAL;
+ }
+ if (reach & CLEANUP_REACH_THROW) {
+ verbose(ctx->env,
+ "cleanup landing pad %u can throw while an exception is in flight\n",
+ pad);
+ return -EINVAL;
+ }
+ if (!(reach & CLEANUP_REACH_RESUME) == !(reach & CLEANUP_REACH_EXIT)) {
+ verbose(ctx->env, "cleanup landing pad %u %s\n", pad,
+ (reach & CLEANUP_REACH_RESUME) ?
+ "reaches both bpf_unwind_resume() and a plain exit" :
+ "reaches neither bpf_unwind_resume() nor an exit");
+ return -EINVAL;
+ }
+ return !!(reach & CLEANUP_REACH_EXIT);
+}
+
+static int cleanup_check_pad_insn(struct bpf_verifier_env *env, u32 i)
+{
+ struct bpf_insn *insn = &env->prog->insnsi[i];
+
+ if (bpf_helper_call(insn) && insn->imm == BPF_FUNC_tail_call) {
+ verbose(env,
+ "bpf_tail_call() at insn %u is in an exception cleanup landing pad\n",
+ i);
+ return -EINVAL;
+ }
+ /* A failed BPF_LD_[ABS|IND] leaves the frame through its epilogue. */
+ if (BPF_CLASS(insn->code) == BPF_LD &&
+ (BPF_MODE(insn->code) == BPF_ABS || BPF_MODE(insn->code) == BPF_IND)) {
+ verbose(env,
+ "BPF_LD_[ABS|IND] at insn %u is in an exception cleanup landing pad\n",
+ i);
+ return -EINVAL;
+ }
+ /* Stack arguments are not supported. */
+ if (is_stack_arg_st(insn) || is_stack_arg_stx(insn)) {
+ verbose(env,
+ "insn %u passes an on-stack call argument in an exception cleanup landing pad\n",
+ i);
+ return -EINVAL;
+ }
+ /* Likewise, stack arguments are not supported. */
+ if (bpf_pseudo_kfunc_call(insn)) {
+ struct bpf_call_summary cs;
+
+ if (bpf_get_call_summary(env, insn, &cs) &&
+ cs.arg_slot_cnt > MAX_BPF_FUNC_REG_ARGS) {
+ verbose(env,
+ "insn %u passes an on-stack call argument in an exception cleanup landing pad\n",
+ i);
+ return -EINVAL;
+ }
+ }
+ return 0;
+}
+
+static int cleanup_mark_pad_bodies(struct cleanup_ctx *ctx)
+{
+ struct bpf_verifier_env *env = ctx->env;
+ u32 i, sp = 0;
+ int ret;
+
+ for (i = 0; i < env->cleanup_info_cnt; i++) {
+ u32 pad = env->cleanup_info[i].landing_pad_off;
+
+ if (in_pad(env, pad))
+ continue;
+
+ ret = cleanup_pad_is_catch(ctx, pad);
+ if (ret < 0)
+ return ret;
+ if (ret) {
+ verbose(env,
+ "catch landing pad %u is not supported yet, only cleanup pads that resume\n",
+ pad);
+ return -EOPNOTSUPP;
+ }
+ env->insn_aux_data[pad].in_cleanup_pad = true;
+ ctx->stack[sp++] = pad;
+ }
+
+ while (sp) {
+ u32 j = ctx->stack[--sp];
+ enum cleanup_insn_kind kind;
+ int next, target, sub;
+ u32 start, end;
+
+ ret = cleanup_check_pad_insn(env, j);
+ if (ret)
+ return ret;
+
+ sub = cleanup_subprog_of(env, j);
+ start = env->subprog_info[sub].start;
+ end = env->subprog_info[sub + 1].start;
+ kind = cleanup_succ(env, j, start, end, &next, &target);
+
+ if (kind == CLEANUP_INSN_CALL) {
+ int callee = cleanup_subprog_of(env, j + env->prog->insnsi[j].imm + 1);
+
+ if (env->subprog_info[callee].might_throw) {
+ verbose(env,
+ "cleanup landing pad calls subprog %d at insn %u, which can throw while an exception is in flight\n",
+ callee, j);
+ return -EINVAL;
+ }
+ }
+
+ if (next >= 0 && !in_pad(env, next)) {
+ env->insn_aux_data[next].in_cleanup_pad = true;
+ ctx->stack[sp++] = next;
+ }
+ if (target >= 0 && !in_pad(env, target)) {
+ env->insn_aux_data[target].in_cleanup_pad = true;
+ ctx->stack[sp++] = target;
+ }
+ }
+ return 0;
+}
+
+static int cleanup_check_resumes(struct cleanup_ctx *ctx)
+{
+ struct bpf_verifier_env *env = ctx->env;
+ u32 i;
+
+ for (i = 0; i < env->prog->len; i++) {
+ if (!insn_is_exc_kfunc(&env->prog->insnsi[i], EXC_KF_bpf_unwind_resume))
+ continue;
+ if (in_pad(env, i))
+ continue;
+ verbose(env,
+ "bpf_unwind_resume() at insn %u is not in an exception cleanup landing pad\n",
+ i);
+ return -EINVAL;
+ }
+ return 0;
+}
+
+static void cleanup_mark_call_sites(struct bpf_verifier_env *env)
+{
+ u32 i, j;
+
+ for (i = 0; i < env->cleanup_info_cnt; i++) {
+ struct bpf_cleanup_info *rec = &env->cleanup_info[i];
+
+ for (j = rec->begin_off; j < rec->end_off; j++) {
+ struct bpf_insn *insn = &env->prog->insnsi[j];
+
+ if (!bpf_pseudo_call(insn) && !bpf_is_throw_kfunc(insn))
+ continue;
+ env->insn_aux_data[j].cleanup_pad = rec->landing_pad_off + 1;
+ }
+ }
+}
+
+int bpf_prepare_cleanup_exceptions(struct bpf_verifier_env *env)
+{
+ if (!env->cleanup_info_cnt)
+ return 0;
+
+ if (bpf_prog_is_offloaded(env->prog->aux)) {
+ verbose(env,
+ "exception cleanup is not supported for offloaded programs\n");
+ return -EINVAL;
+ }
+
+ if (!bpf_jit_supports_cleanup_pads() || !env->prog->jit_requested) {
+ verbose(env,
+ "exception cleanup needs a JIT that can dispatch landing pads\n");
+ return -EOPNOTSUPP;
+ }
+ env->prog->jit_required = 1;
+
+ if (env->exception_callback_subprog) {
+ verbose(env,
+ "exception cleanup table cannot be combined with an exception callback\n");
+ return -EINVAL;
+ }
+
+ cleanup_mark_kfunc_sites(env);
+ cleanup_mark_call_sites(env);
+ return 0;
+}
+
+int bpf_check_cleanup_exceptions(struct bpf_verifier_env *env)
+{
+ u32 len = env->prog->len;
+ struct cleanup_ctx ctx = { .env = env };
+ const struct cleanup_alloc_req tab[] = {
+ { (void **)&ctx.reach, len, sizeof(*ctx.reach) },
+ { (void **)&ctx.stack, len, sizeof(*ctx.stack) },
+ };
+ int ret;
+
+ if (!env->cleanup_info_cnt)
+ return 0;
+
+ ctx.scratch = cleanup_alloc(tab, ARRAY_SIZE(tab));
+ if (!ctx.scratch)
+ return -ENOMEM;
+
+ ret = cleanup_reachability(&ctx);
+ if (ret)
+ goto out;
+
+ ret = cleanup_mark_pad_bodies(&ctx);
+ if (ret)
+ goto out;
+
+ ret = cleanup_check_resumes(&ctx);
+out:
+ kvfree(ctx.scratch);
+ return ret;
+}
+
+bool bpf_is_unwind_resume_kfunc(const struct bpf_insn *insn)
+{
+ return insn_is_exc_kfunc(insn, EXC_KF_bpf_unwind_resume);
+}
+
+int bpf_cleanup_pad_of_call(struct bpf_verifier_env *env, u32 idx)
+{
+ u32 pad = env->insn_aux_data[idx].cleanup_pad;
+
+ return pad ? (int)pad - 1 : -1;
+}
+
+/*
+ * Every subprogram of a cleanup-carrying program spills the BPF callee-saved
+ * registers, even one that never throws: a frame's spill holds its caller's
+ * registers, and that is what the walker restores before running the caller's
+ * pad. The exception callback does not, because it reuses the boundary frame
+ * rather than building one of its own.
+ */
+bool bpf_cleanup_force_spill(const struct bpf_prog *prog)
+{
+ return prog->aux->exc && !prog->aux->exception_cb;
+}
+
+const struct bpf_cleanup_range *bpf_cleanup_pad_for_ip(const struct bpf_prog *prog, u64 ip)
+{
+ const struct bpf_exception_info *exc = prog->aux->exc;
+ u32 l = 0, r = exc ? exc->nr_ranges : 0;
+
+ while (l < r) {
+ u32 m = l + (r - l) / 2;
+ const struct bpf_cleanup_range *rec = &exc->ranges[m];
+
+ if (ip <= rec->begin)
+ r = m;
+ else if (ip > rec->end)
+ l = m + 1;
+ else
+ return rec;
+ }
+ return NULL;
+}
+
+static int cmp_u32(const void *a, const void *b)
+{
+ u32 x = *(const u32 *)a, y = *(const u32 *)b;
+
+ return x < y ? -1 : x > y;
+}
+
+int bpf_cleanup_alloc_info(struct bpf_prog_aux *aux)
+{
+ if (aux->exc)
+ return 0;
+ aux->exc = kzalloc_obj(struct bpf_exception_info, GFP_KERNEL_ACCOUNT | __GFP_NOWARN);
+ return aux->exc ? 0 : -ENOMEM;
+}
+
+int bpf_cleanup_attach_info(struct bpf_prog_aux *aux, struct bpf_cleanup_info *recs, u32 cnt)
+{
+ struct bpf_exception_info *exc = aux->exc;
+ struct bpf_cleanup_range *ranges;
+ u32 i, n_at, *at;
+
+ if (!cnt) {
+ kvfree(recs);
+ return 0;
+ }
+
+ ranges = kvcalloc(cnt, sizeof(*ranges), GFP_KERNEL_ACCOUNT | __GFP_NOWARN);
+ if (!ranges) {
+ kvfree(recs);
+ return -ENOMEM;
+ }
+
+ /* The pads on their own, sorted and deduplicated. */
+ at = kvmalloc_array(cnt, sizeof(*at), GFP_KERNEL_ACCOUNT | __GFP_NOWARN);
+ if (!at) {
+ kvfree(ranges);
+ kvfree(recs);
+ return -ENOMEM;
+ }
+ for (i = 0; i < cnt; i++)
+ at[i] = recs[i].landing_pad_off;
+ sort(at, cnt, sizeof(*at), cmp_u32, NULL);
+ for (i = 0, n_at = 0; i < cnt; i++)
+ if (!n_at || at[n_at - 1] != at[i])
+ at[n_at++] = at[i];
+
+ exc->pad_at = at;
+ exc->nr_pad_at = n_at;
+ exc->info = recs;
+ exc->nr_info = cnt;
+ exc->ranges = ranges;
+ /* Withheld until the JIT has filled the table in. */
+ exc->nr_ranges = 0;
+ return 0;
+}
+
+void bpf_cleanup_fill_native_ranges(struct bpf_prog *prog, u32 *addrs, void *image)
+{
+ struct bpf_exception_info *exc = prog->aux->exc;
+ u32 i, n;
+
+ if (!exc || !exc->nr_info || !exc->ranges)
+ return;
+
+ n = exc->nr_info;
+ for (i = 0; i < n; i++) {
+ const struct bpf_cleanup_info *rec = &exc->info[i];
+
+ if (WARN_ON_ONCE(rec->begin_off >= prog->len ||
+ rec->end_off > prog->len ||
+ rec->landing_pad_off >= prog->len))
+ return;
+ exc->ranges[i].begin = (u64)(long)image + addrs[rec->begin_off];
+ exc->ranges[i].end = (u64)(long)image + addrs[rec->end_off];
+ exc->ranges[i].pad = (u64)(long)image + addrs[rec->landing_pad_off];
+ }
+ exc->nr_ranges = n;
+}
+
+void bpf_cleanup_free_info(struct bpf_prog_aux *aux)
+{
+ struct bpf_exception_info *exc = aux->exc;
+
+ if (!exc)
+ return;
+ kvfree(exc->ranges);
+ kvfree(exc->info);
+ kvfree(exc->pad_at);
+ kvfree(exc->throw_at);
+ kvfree(exc->resume_at);
+ bitmap_free(exc->pad_body);
+ kfree(exc);
+ aux->exc = NULL;
+}
+
+/* Is @idx in the sorted array @at of @n instruction indices? */
+static bool insn_idx_in(const u32 *at, u32 n, u32 idx)
+{
+ return bsearch(&idx, at, n, sizeof(*at), cmp_u32);
+}
+
+bool bpf_cleanup_insn_is_pad(const struct bpf_prog *prog, u32 idx)
+{
+ const struct bpf_exception_info *exc = prog->aux->exc;
+
+ return exc && insn_idx_in(exc->pad_at, exc->nr_pad_at, idx);
+}
+
+bool bpf_cleanup_insn_is_throw(const struct bpf_prog *prog, u32 idx)
+{
+ const struct bpf_exception_info *exc = prog->aux->exc;
+
+ return exc && insn_idx_in(exc->throw_at, exc->nr_throw_at, idx);
+}
+
+bool bpf_cleanup_insn_is_resume(const struct bpf_prog *prog, u32 idx)
+{
+ const struct bpf_exception_info *exc = prog->aux->exc;
+
+ return exc && insn_idx_in(exc->resume_at, exc->nr_resume_at, idx);
+}
+
+bool bpf_cleanup_insn_in_pad(const struct bpf_prog *prog, u32 idx)
+{
+ const struct bpf_exception_info *exc = prog->aux->exc;
+
+ return exc && exc->pad_body && idx < exc->nr_pad_body &&
+ test_bit(idx, exc->pad_body);
+}
diff --git a/kernel/bpf/exception.h b/kernel/bpf/exception.h
new file mode 100644
index 0000000000000..c0e68ce227c83
--- /dev/null
+++ b/kernel/bpf/exception.h
@@ -0,0 +1,22 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#ifndef _LINUX_BPF_EXCEPTION_H
+#define _LINUX_BPF_EXCEPTION_H
+
+#include <linux/types.h>
+
+struct bpf_cleanup_info;
+struct bpf_cleanup_range;
+struct bpf_prog;
+struct bpf_prog_aux;
+struct bpf_verifier_env;
+
+int bpf_prepare_cleanup_exceptions(struct bpf_verifier_env *env);
+int bpf_check_cleanup_exceptions(struct bpf_verifier_env *env);
+int bpf_cleanup_check_callback(struct bpf_verifier_env *env, int subprog);
+int bpf_cleanup_pad_of_call(struct bpf_verifier_env *env, u32 idx);
+int bpf_cleanup_alloc_info(struct bpf_prog_aux *aux);
+int bpf_cleanup_attach_info(struct bpf_prog_aux *aux, struct bpf_cleanup_info *recs, u32 cnt);
+const struct bpf_cleanup_range *bpf_cleanup_pad_for_ip(const struct bpf_prog *prog, u64 ip);
+
+#endif /* _LINUX_BPF_EXCEPTION_H */
diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
index 2add8001c3ec3..169cbf9f44159 100644
--- a/kernel/bpf/fixups.c
+++ b/kernel/bpf/fixups.c
@@ -1,5 +1,6 @@
// SPDX-License-Identifier: GPL-2.0-only
/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <linux/bitmap.h>
#include <linux/bpf.h>
#include <linux/btf.h>
#include <linux/bpf_verifier.h>
@@ -10,6 +11,7 @@
#include <linux/perf_event.h>
#include <net/xdp.h>
#include "disasm.h"
+#include "exception.h"
#define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
@@ -252,15 +254,29 @@ static void adjust_insn_aux_data(struct bpf_verifier_env *env,
/* Expand insni[off]'s seen count to the patched range. */
data[i].seen = old_seen;
data[i].zext_dst = bpf_insn_def32(new_prog, insn + i) >= 0;
+ data[i].in_cleanup_pad = data[off + cnt - 1].in_cleanup_pad;
if (!memcmp(insn + i, original_insn, sizeof(struct bpf_insn))) {
data[i].non_stack_access =
data[off + cnt - 1].non_stack_access;
data[off + cnt - 1].non_stack_access = false;
+ data[i].cleanup_throw_site =
+ data[off + cnt - 1].cleanup_throw_site;
+ data[off + cnt - 1].cleanup_throw_site = false;
+ data[i].cleanup_resume_site =
+ data[off + cnt - 1].cleanup_resume_site;
+ data[off + cnt - 1].cleanup_resume_site = false;
+ data[i].cleanup_pad = data[off + cnt - 1].cleanup_pad;
+ data[off + cnt - 1].cleanup_pad = 0;
} else if (bpf_is_mem_insn(insn + i)) {
data[i].non_stack_access = true;
}
}
+ if (env->cleanup_info_cnt)
+ for (i = 0; i < prog_len; i++)
+ if (data[i].cleanup_pad > off + 1)
+ data[i].cleanup_pad += cnt - 1;
+
/*
* Last slot instruction could be a newly generated
* BPF_ST/BPF_LDX/BPF_STX, systematically mark it for non-stack access
@@ -544,11 +560,13 @@ void bpf_clear_insn_aux_data(struct bpf_verifier_env *env, int start, int len)
}
}
-static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt)
+static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt,
+ bool falls_through)
{
struct bpf_insn_aux_data *aux_data = env->insn_aux_data;
unsigned int orig_prog_len = env->prog->len;
int err;
+ u32 i;
if (bpf_prog_is_offloaded(env->prog->aux))
bpf_prog_offload_remove_insns(env, off, cnt);
@@ -573,6 +591,17 @@ static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt)
sizeof(*aux_data) * (orig_prog_len - off - cnt));
env->insn_aux_data_len -= cnt;
+ if (env->cleanup_info_cnt) {
+ for (i = 0; i < env->insn_aux_data_len; i++) {
+ u32 pad = aux_data[i].cleanup_pad;
+
+ if (pad > off + cnt)
+ aux_data[i].cleanup_pad = pad - cnt;
+ else if (pad > off)
+ aux_data[i].cleanup_pad = falls_through ? off + 1 : 0;
+ }
+ }
+
return 0;
}
@@ -634,7 +663,7 @@ int bpf_opt_remove_dead_code(struct bpf_verifier_env *env)
if (!j)
continue;
- err = verifier_remove_insns(env, i, j);
+ err = verifier_remove_insns(env, i, j, false);
if (err)
return err;
insn_cnt = env->prog->len;
@@ -657,7 +686,7 @@ int bpf_opt_remove_nops(struct bpf_verifier_env *env)
if (!is_may_goto_0 && !is_ja)
continue;
- err = verifier_remove_insns(env, i, 1);
+ err = verifier_remove_insns(env, i, 1, true);
if (err)
return err;
insn_cnt--;
@@ -1095,6 +1124,129 @@ static void bpf_restore_subprog_starts(struct bpf_verifier_env *env, u32 *orig_s
env->subprog_info[env->subprog_cnt].start = env->prog->len;
}
+static bool cleanup_kfunc_site(const struct bpf_insn_aux_data *aux, bool resume)
+{
+ return resume ? aux->cleanup_resume_site : aux->cleanup_throw_site;
+}
+
+static int cleanup_kfunc_sites_for_subprog(struct bpf_verifier_env *env, u32 start, u32 end,
+ bool resume, u32 **at_p, u32 *nr_p)
+{
+ u32 i, cnt = 0, *at;
+
+ for (i = start; i < end; i++)
+ if (cleanup_kfunc_site(&env->insn_aux_data[i], resume))
+ cnt++;
+ if (!cnt)
+ return 0;
+
+ at = kvmalloc_array(cnt, sizeof(*at), GFP_KERNEL_ACCOUNT | __GFP_NOWARN);
+ if (!at)
+ return -ENOMEM;
+
+ for (i = start, cnt = 0; i < end; i++) {
+ if (!cleanup_kfunc_site(&env->insn_aux_data[i], resume))
+ continue;
+ at[cnt++] = i - start;
+ }
+
+ *at_p = at;
+ *nr_p = cnt;
+ return 0;
+}
+
+static int cleanup_pad_body_for_subprog(struct bpf_verifier_env *env, struct bpf_prog *sub,
+ u32 start, u32 end)
+{
+ unsigned long *bits;
+ u32 i, cnt = 0;
+
+ for (i = start; i < end; i++)
+ if (env->insn_aux_data[i].in_cleanup_pad)
+ cnt++;
+ if (!cnt)
+ return 0;
+
+ bits = bitmap_zalloc(end - start, GFP_KERNEL_ACCOUNT | __GFP_NOWARN);
+ if (!bits)
+ return -ENOMEM;
+
+ for (i = start; i < end; i++)
+ if (env->insn_aux_data[i].in_cleanup_pad)
+ __set_bit(i - start, bits);
+
+ sub->aux->exc->pad_body = bits;
+ sub->aux->exc->nr_pad_body = end - start;
+ return 0;
+}
+
+static int cleanup_info_for_subprog(struct bpf_verifier_env *env, struct bpf_prog *sub,
+ u32 start, u32 end)
+{
+ struct bpf_cleanup_info *recs;
+ u32 i, cnt = 0;
+ int err;
+
+ if (!env->cleanup_info_cnt)
+ return 0;
+
+ err = bpf_cleanup_alloc_info(sub->aux);
+ if (err)
+ return err;
+
+ err = cleanup_kfunc_sites_for_subprog(env, start, end, false,
+ &sub->aux->exc->throw_at,
+ &sub->aux->exc->nr_throw_at);
+ if (err)
+ return err;
+
+ err = cleanup_kfunc_sites_for_subprog(env, start, end, true,
+ &sub->aux->exc->resume_at,
+ &sub->aux->exc->nr_resume_at);
+ if (err)
+ return err;
+
+ err = cleanup_pad_body_for_subprog(env, sub, start, end);
+ if (err)
+ return err;
+
+ for (i = start; i < end; i++)
+ if (env->insn_aux_data[i].cleanup_pad)
+ cnt++;
+ if (!cnt)
+ return 0;
+
+ recs = kvmalloc_array(cnt, sizeof(*recs), GFP_KERNEL_ACCOUNT | __GFP_NOWARN);
+ if (!recs)
+ return -ENOMEM;
+
+ for (i = start, cnt = 0; i < end; i++) {
+ u32 pad = env->insn_aux_data[i].cleanup_pad;
+
+ if (!pad)
+ continue;
+ pad--;
+ if (verifier_bug_if(pad < start || pad >= end, env,
+ "insn %u is covered by a landing pad at %u outside its subprog [%u, %u)",
+ i, pad, start, end)) {
+ kvfree(recs);
+ return -EFAULT;
+ }
+ recs[cnt].begin_off = i - start;
+ recs[cnt].end_off = i - start + 1;
+ recs[cnt].landing_pad_off = pad - start;
+ cnt++;
+ }
+ return bpf_cleanup_attach_info(sub->aux, recs, cnt);
+}
+
+int bpf_cleanup_attach_main_prog(struct bpf_verifier_env *env, struct bpf_prog *prog)
+{
+ if (!env || env->subprog_cnt > 1)
+ return 0;
+ return cleanup_info_for_subprog(env, prog, 0, prog->len);
+}
+
static int jit_subprogs(struct bpf_verifier_env *env)
{
struct bpf_prog *prog = env->prog, **func, *tmp;
@@ -1232,6 +1384,10 @@ static int jit_subprogs(struct bpf_verifier_env *env)
func[i]->aux->token = prog->aux->token;
if (!i)
func[i]->aux->exception_boundary = env->seen_exception;
+ err = cleanup_info_for_subprog(env, func[i], subprog_start,
+ subprog_end);
+ if (err)
+ goto out_free;
func[i] = bpf_int_jit_compile(env, func[i]);
if (!func[i]->jited) {
err = -ENOTSUPP;
@@ -1336,6 +1492,8 @@ static int jit_subprogs(struct bpf_verifier_env *env)
prog->aux->bpf_exception_cb = (void *)func[env->exception_callback_subprog]->bpf_func;
prog->aux->exception_boundary = func[0]->aux->exception_boundary;
prog->aux->stack_arg_sp_adjust = func[0]->aux->stack_arg_sp_adjust;
+ prog->aux->exc = func[0]->aux->exc;
+ func[0]->aux->exc = NULL;
bpf_prog_jit_attempt_done(prog);
return 0;
out_free:
@@ -1916,6 +2074,8 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)
goto next_insn;
if (insn->src_reg == BPF_PSEUDO_CALL)
goto next_insn;
+ if (env->insn_aux_data[i + delta].cleanup_resume_site)
+ goto next_insn;
if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL) {
ret = bpf_fixup_kfunc_call(env, insn, insn_buf, i + delta, &cnt);
if (ret)
diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c
index 82402d97ce67c..d639dfbc3cbbf 100644
--- a/kernel/bpf/helpers.c
+++ b/kernel/bpf/helpers.c
@@ -31,6 +31,7 @@
#include <linux/buildid.h>
#include "../../lib/kstrtox.h"
+#include "exception.h"
/* If kernel subsystem is allowing eBPF programs to call this function,
* inside its own verifier_ops->get_func_proto() callback it should return
@@ -3398,8 +3399,36 @@ struct bpf_throw_ctx {
u64 sp;
u64 bp;
int cnt;
+ const struct bpf_prog *callee;
+ u64 callee_fp;
};
+static void bpf_run_cleanup_pad(struct bpf_throw_ctx *ctx, const struct bpf_prog *prog,
+ u64 ip, u64 fp)
+{
+ const struct bpf_exception_info *exc = prog->aux->exc;
+ const struct bpf_cleanup_range *rec;
+ u64 spill_base;
+
+ if (!exc || !exc->nr_ranges)
+ return;
+ rec = bpf_cleanup_pad_for_ip(prog, ip);
+ if (!rec)
+ return;
+
+ /*
+ * The callee is always another subprogram of this program -- the walk
+ * ends at any frame that is not one -- so its prologue spilled these
+ * registers and its exc is there to say where.
+ */
+ if (ctx->callee)
+ spill_base = ctx->callee_fp + ctx->callee->aux->exc->spill_off;
+ else
+ spill_base = fp + exc->throw_spill_off;
+
+ arch_bpf_run_cleanup_pad(rec->pad, fp, spill_base);
+}
+
static bool bpf_stack_walker(void *cookie, u64 ip, u64 sp, u64 bp)
{
struct bpf_throw_ctx *ctx = cookie;
@@ -3416,6 +3445,11 @@ static bool bpf_stack_walker(void *cookie, u64 ip, u64 sp, u64 bp)
if (!prog)
return !ctx->cnt;
ctx->cnt++;
+
+ bpf_run_cleanup_pad(ctx, prog, ip, bp);
+ ctx->callee = prog;
+ ctx->callee_fp = bp;
+
if (bpf_is_subprog(prog))
return true;
ctx->aux = prog->aux;
@@ -3445,6 +3479,17 @@ __bpf_kfunc void bpf_throw(u64 cookie)
WARN(1, "A call to BPF exception callback should never return\n");
}
+/*
+ * Terminator of a compiler-emitted cleanup landing pad. The compiler names
+ * this _Unwind_Resume, the base unwind ABI's entry point for carrying an
+ * unwind on once a frame's cleanups have run. To match kernel kfunc
+ * convention, the kernel calls it bpf_unwind_resume and libbpf maps the
+ * compiler's name onto it.
+ */
+__bpf_kfunc void bpf_unwind_resume(void)
+{
+}
+
__bpf_kfunc int bpf_wq_init(struct bpf_wq *wq, void *p__const_map, unsigned int flags)
{
struct bpf_async_kern *async = (struct bpf_async_kern *)wq;
@@ -4981,6 +5026,7 @@ BTF_ID_FLAGS(func, bpf_task_get_cgroup1, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_task_from_pid, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_task_from_vpid, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_throw)
+BTF_ID_FLAGS(func, bpf_unwind_resume)
#ifdef CONFIG_BPF_EVENTS
BTF_ID_FLAGS(func, bpf_send_signal_task)
#endif
diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index 44ecdc5b4ec2d..9cfd05f970bc9 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -8,6 +8,8 @@
#include <linux/slab.h>
#include <linux/sort.h>
+#include "exception.h"
+
#define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
struct per_frame_masks {
@@ -256,6 +258,9 @@ bpf_insn_successors(struct bpf_verifier_env *env, u32 idx)
succ = env->succ;
succ->cnt = 0;
+ if (unlikely(bpf_is_unwind_resume_kfunc(insn)))
+ return succ;
+
opcode_info = &opcode_info_tbl[BPF_CLASS(insn->code) | BPF_OP(insn->code)];
insn_sz = bpf_is_ldimm64(insn) ? 2 : 1;
if (opcode_info->can_fallthrough)
@@ -264,6 +269,13 @@ bpf_insn_successors(struct bpf_verifier_env *env, u32 idx)
if (opcode_info->can_jump)
succ->items[succ->cnt++] = idx + bpf_jmp_offset(insn) + 1;
+ if (unlikely(env->cleanup_info_cnt)) {
+ int pad = bpf_cleanup_pad_of_call(env, idx);
+
+ if (pad >= 0)
+ succ->items[succ->cnt++] = pad;
+ }
+
return succ;
}
@@ -397,6 +409,14 @@ bool bpf_stack_slot_alive(struct bpf_verifier_env *env, u32 frameno, u32 half_sp
alive = bpf_calls_callback(env, callsite)
? is_live_before(instance, callsite, rel, half_spi)
: is_live_before(instance, callsite + 1, rel, half_spi);
+
+ /* Control may also go to the landing pad. */
+ if (!alive && unlikely(env->cleanup_info_cnt)) {
+ int pad = bpf_cleanup_pad_of_call(env, callsite);
+
+ if (pad >= 0)
+ alive = is_live_before(instance, pad, rel, half_spi);
+ }
if (alive)
return true;
}
diff --git a/kernel/bpf/states.c b/kernel/bpf/states.c
index 66fb11b6c6a76..b101baa43171c 100644
--- a/kernel/bpf/states.c
+++ b/kernel/bpf/states.c
@@ -996,6 +996,12 @@ static bool states_equal(struct bpf_verifier_env *env,
if (old->in_sleepable != cur->in_sleepable)
return false;
+ if (old->unwinding != cur->unwinding)
+ return false;
+
+ if (old->unwinding && old->unwind_frameno != cur->unwind_frameno)
+ return false;
+
if (!refsafe(old, cur, &env->idmap_scratch))
return false;
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 113486b15d29f..049717dc3b7d7 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -2915,7 +2915,7 @@ int __init __used bpf_multi_func(void) { return 0; }
BTF_ID_LIST_GLOBAL_SINGLE(bpf_multi_func_btf_id, func, bpf_multi_func)
/* last field in 'union bpf_attr' used by this command */
-#define BPF_PROG_LOAD_LAST_FIELD keyring_id
+#define BPF_PROG_LOAD_LAST_FIELD cleanup_info_cnt
static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, struct bpf_log_attr *attr_log)
{
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 18aad4886f9cf..b978e710d7b66 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -10,6 +10,7 @@
#include <linux/slab.h>
#include <linux/bpf.h>
#include <linux/btf.h>
+#include <linux/bpf_cleanup_abi.h>
#include <linux/bpf_verifier.h>
#include <linux/filter.h>
#include <net/netlink.h>
@@ -37,6 +38,7 @@
#include "diagnostics.h"
#include "disasm.h"
+#include "exception.h"
static const struct bpf_verifier_ops * const bpf_verifier_ops[] = {
#define BPF_PROG_TYPE(_id, _name, prog_ctx_type, kern_ctx_type) \
@@ -1714,6 +1716,8 @@ int bpf_copy_verifier_state(struct bpf_verifier_state *dst_state,
return err;
dst_state->speculative = src->speculative;
dst_state->in_sleepable = src->in_sleepable;
+ dst_state->unwinding = src->unwinding;
+ dst_state->unwind_frameno = src->unwind_frameno;
dst_state->curframe = src->curframe;
dst_state->branches = src->branches;
dst_state->parent = src->parent;
@@ -5588,6 +5592,14 @@ static int check_max_stack_depth(struct bpf_verifier_env *env)
}
}
+ /*
+ * A pad rebuilds its frame from a spill area, and on x86-64 a private
+ * stack's frame pointer lives in r9, which no spill area holds.
+ * Refused on every arch rather than just that one.
+ */
+ if (env->cleanup_info_cnt)
+ priv_stack_mode = NO_PRIV_STACK;
+
if (priv_stack_mode == PRIV_STACK_UNKNOWN)
priv_stack_mode = bpf_enable_priv_stack(env->prog);
@@ -10502,6 +10514,10 @@ static int push_callback_call(struct bpf_verifier_env *env, struct bpf_insn *ins
* callbacks
*/
env->subprog_info[subprog].is_cb = true;
+ err = bpf_cleanup_check_callback(env, subprog);
+ if (err)
+ return err;
+
if (bpf_pseudo_kfunc_call(insn) &&
!is_callback_calling_kfunc(insn->imm)) {
verifier_bug(env, "kfunc %s#%d not marked as callback-calling",
@@ -10553,8 +10569,8 @@ static int push_callback_call(struct bpf_verifier_env *env, struct bpf_insn *ins
return 0;
}
-static int process_bpf_exit_full(struct bpf_verifier_env *env,
- bool *do_print_state, bool exception_exit);
+static int process_bpf_exit_full(struct bpf_verifier_env *env, bool *do_print_state);
+static int unwind_step(struct bpf_verifier_env *env, u32 callsite, int *insn_idx);
static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
int *insn_idx)
@@ -10644,7 +10660,7 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
verbose(env, "failed to push state for global subprog exception path\n");
return PTR_ERR(branch);
}
- return process_bpf_exit_full(env, NULL, true);
+ return unwind_step(env, *insn_idx, insn_idx);
}
/* continue with next insn after call */
@@ -14545,7 +14561,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
env->prog->call_session_cookie = true;
if (bpf_is_throw_kfunc(insn))
- return process_bpf_exit_full(env, NULL, true);
+ return unwind_step(env, insn_idx, &env->insn_idx);
return 0;
}
@@ -18470,9 +18486,103 @@ enum {
INSN_IDX_UPDATED = 2,
};
-static int process_bpf_exit_full(struct bpf_verifier_env *env,
- bool *do_print_state,
- bool exception_exit)
+static u32 unwind_pop_frame(struct bpf_verifier_env *env)
+{
+ struct bpf_verifier_state *state = env->cur_state;
+ struct bpf_func_state *callee = state->frame[state->curframe];
+ u32 callsite = callee->callsite;
+ struct bpf_func_state *caller;
+
+ caller = state->frame[state->curframe - 1];
+ account_processed_insns(env, callee, caller);
+ free_func_state(callee);
+ state->frame[state->curframe--] = NULL;
+ invalidate_outgoing_stack_args(env, caller);
+ return callsite;
+}
+
+static void unwind_enter_pad(struct bpf_verifier_env *env)
+{
+ struct bpf_verifier_state *state = env->cur_state;
+ struct bpf_func_state *frame = cur_func(env);
+
+ state->unwind_frameno = state->curframe;
+ clear_caller_saved_regs(env, frame->regs);
+ mark_reg_unknown(env, frame->regs, BPF_REG_0);
+ __mark_reg_known(&frame->regs[BPF_REG_0], BPF_PAD_ENTRY_R0);
+}
+
+static int unwind_finish(struct bpf_verifier_env *env)
+{
+ int err = check_resource_leak(env, true, true, "bpf_throw");
+
+ if (err)
+ return err;
+ return PROCESS_BPF_EXIT;
+}
+
+static int unwind_step(struct bpf_verifier_env *env, u32 callsite, int *insn_idx)
+{
+ struct bpf_verifier_state *state = env->cur_state;
+ u8 popped = 0;
+
+ state->unwinding = true;
+ for (;;) {
+ int pad = bpf_cleanup_pad_of_call(env, callsite);
+
+ if (pad >= 0) {
+ unwind_enter_pad(env);
+ /*
+ * The edge from @callsite to the pad crosses @popped
+ * frames, and nothing in the instruction stream says
+ * so. Record it for mark_chain_precision(), which has
+ * to walk back through the same frames.
+ */
+ env->unwind_frames = popped;
+ *insn_idx = pad;
+ return INSN_IDX_UPDATED;
+ }
+ if (!state->curframe)
+ return unwind_finish(env);
+ callsite = unwind_pop_frame(env);
+ popped++;
+ }
+}
+
+static int process_cleanup_resume(struct bpf_verifier_env *env, int *insn_idx)
+{
+ struct bpf_verifier_state *state = env->cur_state;
+ int err;
+
+ /* A pad entered by ordinary control flow. */
+ if (!state->unwinding) {
+ verbose(env,
+ "bpf_unwind_resume() at insn %d reached without an exception in flight\n",
+ *insn_idx);
+ return -EINVAL;
+ }
+ /*
+ * A resume in a subprogram the pad called, which has a landing pad of
+ * its own and reached it by ordinary control flow. A JIT lowers a
+ * resume as the way back out of a pad, which reaches the walker only
+ * from the frame whose pad it called.
+ */
+ if (state->curframe != state->unwind_frameno) {
+ verbose(env,
+ "bpf_unwind_resume() at insn %d is in frame %d, not frame %d whose landing pad the exception entered\n",
+ *insn_idx, state->curframe, state->unwind_frameno);
+ return -EINVAL;
+ }
+ if (!state->curframe)
+ return unwind_finish(env);
+ err = unwind_step(env, unwind_pop_frame(env), insn_idx);
+ /* unwind_step() counted the frames it popped, not this one. */
+ if (err == INSN_IDX_UPDATED)
+ env->unwind_frames++;
+ return err;
+}
+
+static int process_bpf_exit_full(struct bpf_verifier_env *env, bool *do_print_state)
{
struct bpf_func_state *cur_frame = cur_func(env);
@@ -18482,25 +18592,11 @@ static int process_bpf_exit_full(struct bpf_verifier_env *env,
* for which reference_state must match caller reference
* state when it exits.
*/
- int err = check_resource_leak(env, exception_exit,
- exception_exit || !env->cur_state->curframe,
- exception_exit ? "bpf_throw" :
+ int err = check_resource_leak(env, false, !env->cur_state->curframe,
"BPF_EXIT instruction in main prog");
if (err)
return err;
- /* The side effect of the prepare_func_exit which is
- * being skipped is that it frees bpf_func_state.
- * Typically, process_bpf_exit will only be hit with
- * outermost exit. copy_verifier_state in pop_stack will
- * handle freeing of any extra bpf_func_state left over
- * from not processing all nested function exits. We
- * also skip return code checks as they are not needed
- * for exceptional exits.
- */
- if (exception_exit)
- return PROCESS_BPF_EXIT;
-
if (env->cur_state->curframe) {
/* exit from nested function */
err = prepare_func_exit(env, &env->insn_idx);
@@ -18674,6 +18770,8 @@ static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state)
env->jmps_processed++;
if (opcode == BPF_CALL) {
+ if (bpf_is_unwind_resume_kfunc(insn))
+ return process_cleanup_resume(env, &env->insn_idx);
if (env->cur_state->active_locks) {
if ((insn->src_reg == BPF_REG_0 &&
insn->imm != BPF_FUNC_spin_unlock &&
@@ -18707,7 +18805,7 @@ static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state)
env->insn_idx += insn->imm + 1;
return INSN_IDX_UPDATED;
} else if (opcode == BPF_EXIT) {
- return process_bpf_exit_full(env, do_print_state, false);
+ return process_bpf_exit_full(env, do_print_state);
}
return check_cond_jmp_op(env, insn, &env->insn_idx);
}
@@ -18744,10 +18842,14 @@ static int do_check(struct bpf_verifier_env *env)
for (;;) {
struct bpf_insn *insn;
struct bpf_insn_aux_data *insn_aux;
+ u8 unwind_frames;
int err;
/* reset current history entry on each new instruction */
env->cur_hist_ent = NULL;
+ /* frames the unwind popped to reach this insn, if it is a pad */
+ unwind_frames = env->unwind_frames;
+ env->unwind_frames = 0;
env->prev_insn_idx = prev_insn_idx;
if (env->insn_idx >= insn_cnt) {
@@ -18811,10 +18913,16 @@ static int do_check(struct bpf_verifier_env *env)
}
}
- if (bpf_is_jmp_point(env, env->insn_idx)) {
+ /*
+ * A landing pad is a jump point, but record the edge even if
+ * that ever stops being true: it is the only place the frames
+ * the unwind popped are written down.
+ */
+ if (bpf_is_jmp_point(env, env->insn_idx) || unwind_frames) {
err = bpf_push_jmp_history(env, state, 0, 0, 0, 0);
if (err)
return err;
+ env->cur_hist_ent->unwind_frames = unwind_frames;
}
if (signal_pending(current))
@@ -21677,6 +21785,11 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
if (ret < 0)
goto skip_full_check;
+ /* The CFG needs an edge from a call in a cleanup range to its pad. */
+ ret = bpf_prepare_cleanup_exceptions(env);
+ if (ret < 0)
+ goto skip_full_check;
+
/* Validate instructions and resolve the program's referenced resources. */
ret = check_and_resolve_insns(env);
if (ret < 0)
@@ -21697,6 +21810,10 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
if (ret < 0)
goto skip_full_check;
+ ret = bpf_check_cleanup_exceptions(env);
+ if (ret < 0)
+ goto skip_full_check;
+
ret = bpf_compute_postorder(env);
if (ret < 0)
goto skip_full_check;
@@ -21884,6 +22001,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
kvfree(env->succ);
kvfree(env->gotox_tmp_buf);
bpf_diag_free(env);
+ kvfree(env->cleanup_info);
kvfree(env);
return ret;
}
diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h
index 6330b7d745c57..a69cc04275751 100644
--- a/tools/include/uapi/linux/bpf.h
+++ b/tools/include/uapi/linux/bpf.h
@@ -1669,6 +1669,9 @@ union bpf_attr {
* verification.
*/
__s32 keyring_id;
+ __aligned_u64 cleanup_info; /* exception cleanup table */
+ __u32 cleanup_info_rec_size; /* userspace bpf_cleanup_info size */
+ __u32 cleanup_info_cnt; /* number of bpf_cleanup_info records */
};
struct { /* anonymous struct used by BPF_OBJ_* commands */
@@ -7599,6 +7602,12 @@ struct bpf_line_info {
__u32 line_col;
};
+struct bpf_cleanup_info {
+ __u32 begin_off;
+ __u32 end_off;
+ __u32 landing_pad_off;
+};
+
struct bpf_spin_lock {
__u32 val;
};
diff --git a/tools/lib/bpf/bpf.c b/tools/lib/bpf/bpf.c
index a9de7f107cf7b..f11d042a28504 100644
--- a/tools/lib/bpf/bpf.c
+++ b/tools/lib/bpf/bpf.c
@@ -295,7 +295,7 @@ int bpf_prog_load(enum bpf_prog_type prog_type,
const struct bpf_insn *insns, size_t insn_cnt,
struct bpf_prog_load_opts *opts)
{
- const size_t attr_sz = offsetofend(union bpf_attr, keyring_id);
+ const size_t attr_sz = offsetofend(union bpf_attr, cleanup_info_cnt);
void *finfo = NULL, *linfo = NULL;
const char *func_info, *line_info;
__u32 log_size, log_level, attach_prog_fd, attach_btf_obj_fd;
@@ -370,6 +370,10 @@ int bpf_prog_load(enum bpf_prog_type prog_type,
attr.fd_array = ptr_to_u64(OPTS_GET(opts, fd_array, NULL));
attr.fd_array_cnt = OPTS_GET(opts, fd_array_cnt, 0);
+ attr.cleanup_info = ptr_to_u64(OPTS_GET(opts, cleanup_info, NULL));
+ attr.cleanup_info_cnt = OPTS_GET(opts, cleanup_info_cnt, 0);
+ attr.cleanup_info_rec_size = OPTS_GET(opts, cleanup_info_rec_size, 0);
+
if (log_level) {
attr.log_buf = ptr_to_u64(log_buf);
attr.log_size = log_size;
diff --git a/tools/lib/bpf/bpf.h b/tools/lib/bpf/bpf.h
index 490e8cb4ba537..ec542fd68f631 100644
--- a/tools/lib/bpf/bpf.h
+++ b/tools/lib/bpf/bpf.h
@@ -128,9 +128,14 @@ struct bpf_prog_load_opts {
/* if set, provides the length of fd_array */
__u32 fd_array_cnt;
+
+ /* exception cleanup table, from the .bpf_cleanup section */
+ const void *cleanup_info;
+ __u32 cleanup_info_cnt;
+ __u32 cleanup_info_rec_size;
size_t :0;
};
-#define bpf_prog_load_opts__last_field fd_array_cnt
+#define bpf_prog_load_opts__last_field cleanup_info_rec_size
LIBBPF_API int bpf_prog_load(enum bpf_prog_type prog_type,
const char *prog_name, const char *license,
diff --git a/tools/lib/bpf/gen_loader.c b/tools/lib/bpf/gen_loader.c
index af3a04f161ac1..2345fbdd46f56 100644
--- a/tools/lib/bpf/gen_loader.c
+++ b/tools/lib/bpf/gen_loader.c
@@ -981,13 +981,15 @@ static void cleanup_relos(struct bpf_gen *gen, int insns)
cleanup_core_relo(gen);
}
-/* Convert func, line, and core relo info blobs to target endianness */
+/* Convert func, line, core relo and cleanup info blobs to target endianness */
static void info_blob_bswap(struct bpf_gen *gen, int func_info, int line_info,
- int core_relos, struct bpf_prog_load_opts *load_attr)
+ int core_relos, int cleanup_info,
+ struct bpf_prog_load_opts *load_attr)
{
struct bpf_func_info *fi = gen->data_start + func_info;
struct bpf_line_info *li = gen->data_start + line_info;
struct bpf_core_relo *cr = gen->data_start + core_relos;
+ struct bpf_cleanup_info *ci = gen->data_start + cleanup_info;
int i;
for (i = 0; i < load_attr->func_info_cnt; i++)
@@ -998,6 +1000,9 @@ static void info_blob_bswap(struct bpf_gen *gen, int func_info, int line_info,
for (i = 0; i < gen->core_relo_cnt; i++)
bpf_core_relo_bswap(cr++);
+
+ for (i = 0; i < load_attr->cleanup_info_cnt; i++)
+ bpf_cleanup_info_bswap(ci++);
}
void bpf_gen__prog_load(struct bpf_gen *gen,
@@ -1011,8 +1016,11 @@ void bpf_gen__prog_load(struct bpf_gen *gen,
load_attr->line_info_rec_size;
int core_relo_tot_sz = gen->core_relo_cnt *
sizeof(struct bpf_core_relo);
+ int cleanup_info_tot_sz = load_attr->cleanup_info_cnt *
+ load_attr->cleanup_info_rec_size;
int prog_load_attr, license_off, insns_off, func_info, line_info, core_relos;
- int attr_size = offsetofend(union bpf_attr, core_relo_rec_size);
+ int attr_size = offsetofend(union bpf_attr, cleanup_info_cnt);
+ int cleanup_info;
union bpf_attr attr;
memset(&attr, 0, attr_size);
@@ -1061,9 +1069,17 @@ void bpf_gen__prog_load(struct bpf_gen *gen,
core_relos, gen->core_relo_cnt,
sizeof(struct bpf_core_relo));
+ attr.cleanup_info_rec_size = tgt_endian(load_attr->cleanup_info_rec_size);
+ attr.cleanup_info_cnt = tgt_endian(load_attr->cleanup_info_cnt);
+ cleanup_info = add_data(gen, load_attr->cleanup_info, cleanup_info_tot_sz);
+ pr_debug("gen: prog_load: cleanup_info: off %d cnt %u rec size %u\n",
+ cleanup_info, load_attr->cleanup_info_cnt,
+ load_attr->cleanup_info_rec_size);
+
/* convert all info blobs to target endianness */
if (gen->swapped_endian && !gen->error)
- info_blob_bswap(gen, func_info, line_info, core_relos, load_attr);
+ info_blob_bswap(gen, func_info, line_info, core_relos, cleanup_info,
+ load_attr);
libbpf_strlcpy(attr.prog_name, prog_name, sizeof(attr.prog_name));
prog_load_attr = add_data(gen, &attr, attr_size);
@@ -1085,6 +1101,11 @@ void bpf_gen__prog_load(struct bpf_gen *gen,
/* populate union bpf_attr with a pointer to core_relos */
emit_rel_store(gen, attr_field(prog_load_attr, core_relos), core_relos);
+ /* populate union bpf_attr with a pointer to cleanup_info, if there is one */
+ if (load_attr->cleanup_info_cnt)
+ emit_rel_store(gen, attr_field(prog_load_attr, cleanup_info),
+ cleanup_info);
+
/* populate union bpf_attr fd_array with a pointer to data where map_fds are saved */
emit_rel_store(gen, attr_field(prog_load_attr, fd_array), gen->fd_array);
diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
index 2f53afc985cfb..a8da2b680685c 100644
--- a/tools/lib/bpf/libbpf.c
+++ b/tools/lib/bpf/libbpf.c
@@ -514,6 +514,11 @@ struct bpf_program {
void *line_info;
__u32 line_info_rec_size;
__u32 line_info_cnt;
+
+ struct bpf_cleanup_info *cleanup_info;
+ __u32 cleanup_info_rec_size;
+ __u32 cleanup_info_cnt;
+
__u32 prog_flags;
__u8 hash[SHA256_DIGEST_LENGTH];
@@ -549,6 +554,7 @@ struct bpf_struct_ops {
#define STRUCT_OPS_SEC ".struct_ops"
#define STRUCT_OPS_LINK_SEC ".struct_ops.link"
#define ARENA_SEC ".addr_space.1"
+#define CLEANUP_SEC ".bpf_cleanup"
enum libbpf_map_type {
LIBBPF_MAP_UNSPEC,
@@ -677,6 +683,25 @@ struct elf_sec_desc {
Elf_Data *data;
};
+#define CLEANUP_REC_FIELDS (sizeof(struct bpf_cleanup_info) / sizeof(__u32))
+
+/* Index of each field of struct bpf_cleanup_info, read as an array of __u32. */
+enum {
+ CLEANUP_REC_BEGIN,
+ CLEANUP_REC_END,
+ CLEANUP_REC_PAD,
+};
+
+/* One (begin, end, landing_pad) triple from .bpf_cleanup, with each field
+ * resolved from its relocation to an ELF section plus a section-relative
+ * instruction index. The mapping to final program instruction indices can only
+ * happen after subprogram placement, which differs per main program.
+ */
+struct cleanup_raw_rec {
+ int sec_idx[CLEANUP_REC_FIELDS];
+ size_t insn_idx[CLEANUP_REC_FIELDS];
+};
+
struct elf_state {
int fd;
const void *obj_buf;
@@ -696,6 +721,8 @@ struct elf_state {
bool has_st_ops;
int arena_data_shndx;
int jumptables_data_shndx;
+ Elf_Data *cleanup_data;
+ int cleanup_shndx;
};
struct usdt_manager;
@@ -771,6 +798,9 @@ struct bpf_object {
void *jumptables_data;
size_t jumptables_data_sz;
+ struct cleanup_raw_rec *cleanup_recs;
+ size_t cleanup_rec_cnt;
+
struct {
struct bpf_program *prog;
unsigned int sym_off;
@@ -817,7 +847,10 @@ static void bpf_program__exit(struct bpf_program *prog)
zfree(&prog->sec_name);
zfree(&prog->insns);
zfree(&prog->reloc_desc);
+ zfree(&prog->cleanup_info);
+ prog->cleanup_info_rec_size = 0;
+ prog->cleanup_info_cnt = 0;
prog->nr_reloc = 0;
prog->insns_cnt = 0;
prog->sec_idx = -1;
@@ -1559,6 +1592,7 @@ static struct bpf_object *bpf_object__new(const char *path,
obj->efile.obj_buf = obj_buf;
obj->efile.obj_buf_sz = obj_buf_sz;
obj->efile.btf_maps_shndx = -1;
+ obj->efile.cleanup_shndx = -1;
obj->kconfig_map_idx = -1;
obj->arena_map_idx = -1;
@@ -4045,6 +4079,9 @@ static int bpf_object__elf_collect(struct bpf_object *obj)
sec_desc->shdr = sh;
sec_desc->data = data;
obj->efile.has_st_ops = true;
+ } else if (strcmp(name, CLEANUP_SEC) == 0) {
+ obj->efile.cleanup_data = data;
+ obj->efile.cleanup_shndx = idx;
} else if (strcmp(name, ARENA_SEC) == 0) {
obj->efile.arena_data = data;
obj->efile.arena_data_shndx = idx;
@@ -4072,6 +4109,7 @@ static int bpf_object__elf_collect(struct bpf_object *obj)
strcmp(name, ".rel" STRUCT_OPS_LINK_SEC) &&
strcmp(name, ".rel?" STRUCT_OPS_SEC) &&
strcmp(name, ".rel?" STRUCT_OPS_LINK_SEC) &&
+ strcmp(name, ".rel" CLEANUP_SEC) &&
strcmp(name, ".rel" MAPS_ELF_SEC)) {
pr_info("elf: skipping relo section(%d) %s for section(%d) %s\n",
idx, name, targ_sec_idx,
@@ -4852,6 +4890,229 @@ static struct bpf_program *find_prog_by_sec_insn(const struct bpf_object *obj,
return NULL;
}
+static int bpf_object__init_cleanup_info(struct bpf_object *obj)
+{
+ Elf_Data *data = obj->efile.cleanup_data;
+ Elf_Data *relo = NULL;
+ size_t i, nrels, nslots, nrecs;
+ struct cleanup_raw_rec *recs;
+ int *slot_sec, ret = 0;
+ size_t *slot_val;
+ const __u32 *vals;
+ bool native;
+
+ if (!data || obj->efile.cleanup_shndx < 0)
+ return 0;
+
+ native = is_native_endianness(obj);
+
+ for (i = 0; i < obj->efile.sec_cnt; i++) {
+ struct elf_sec_desc *sd = &obj->efile.secs[i];
+
+ if (sd->sec_type == SEC_RELO && sd->shdr &&
+ sd->shdr->sh_info == (Elf64_Word)obj->efile.cleanup_shndx) {
+ relo = sd->data;
+ break;
+ }
+ }
+ if (!relo) {
+ pr_warn("%s present without relocations\n", CLEANUP_SEC);
+ return -LIBBPF_ERRNO__FORMAT;
+ }
+ if (data->d_size % sizeof(struct bpf_cleanup_info)) {
+ pr_warn("%s size %zu is not a multiple of the record size %zu\n",
+ CLEANUP_SEC, data->d_size, sizeof(struct bpf_cleanup_info));
+ return -LIBBPF_ERRNO__FORMAT;
+ }
+
+ vals = data->d_buf;
+ nslots = data->d_size / sizeof(__u32);
+ nrecs = data->d_size / sizeof(struct bpf_cleanup_info);
+
+ slot_sec = calloc(nslots, sizeof(*slot_sec));
+ slot_val = calloc(nslots, sizeof(*slot_val));
+ recs = calloc(nrecs ?: 1, sizeof(*recs));
+ if (!slot_sec || !slot_val || !recs) {
+ ret = -ENOMEM;
+ goto out;
+ }
+ for (i = 0; i < nslots; i++)
+ slot_sec[i] = -1;
+
+ /* One relocation per 4-byte field, naming the section it points into. */
+ nrels = relo->d_size / sizeof(Elf64_Rel);
+ for (i = 0; i < nrels; i++) {
+ Elf64_Rel *rel = elf_rel_by_idx(relo, i);
+ Elf64_Sym *sym = elf_sym_by_idx(obj, ELF64_R_SYM(rel->r_info));
+ size_t type = ELF64_R_TYPE(rel->r_info);
+ size_t slot = rel->r_offset / sizeof(__u32);
+
+ if (type != R_BPF_64_NODYLD32 && type != R_BPF_64_ABS32) {
+ pr_warn("%s: relocation %zu has unexpected type %zu\n",
+ CLEANUP_SEC, i, type);
+ ret = -LIBBPF_ERRNO__FORMAT;
+ goto out;
+ }
+ if (!sym || slot >= nslots || rel->r_offset % sizeof(__u32)) {
+ pr_warn("%s: bad relocation %zu\n", CLEANUP_SEC, i);
+ ret = -LIBBPF_ERRNO__FORMAT;
+ goto out;
+ }
+ slot_sec[slot] = sym->st_shndx;
+ /* The addend lives in the section data, which libelf leaves in
+ * the object's byte order; a non-section symbol additionally
+ * contributes its own value.
+ */
+ slot_val[slot] = (native ? vals[slot] : bswap_32(vals[slot])) +
+ sym->st_value;
+ }
+
+ for (i = 0; i < nslots; i++) {
+ struct cleanup_raw_rec *rec = &recs[i / CLEANUP_REC_FIELDS];
+ size_t field = i % CLEANUP_REC_FIELDS;
+
+ if (slot_sec[i] < 0) {
+ pr_warn("%s: field %zu has no relocation\n", CLEANUP_SEC, i);
+ ret = -LIBBPF_ERRNO__FORMAT;
+ goto out;
+ }
+ if (slot_val[i] % BPF_INSN_SZ) {
+ pr_warn("%s: field %zu offset %zu is not instruction aligned\n",
+ CLEANUP_SEC, i, slot_val[i]);
+ ret = -LIBBPF_ERRNO__FORMAT;
+ goto out;
+ }
+ rec->sec_idx[field] = slot_sec[i];
+ rec->insn_idx[field] = slot_val[i] / BPF_INSN_SZ;
+ }
+
+ obj->cleanup_recs = recs;
+ obj->cleanup_rec_cnt = nrecs;
+ recs = NULL;
+out:
+ free(recs);
+ free(slot_val);
+ free(slot_sec);
+ return ret;
+}
+
+static int cmp_cleanup_info(const void *a, const void *b)
+{
+ const struct bpf_cleanup_info *x = a, *y = b;
+
+ if (x->begin_off == y->begin_off)
+ return 0;
+ return x->begin_off < y->begin_off ? -1 : 1;
+}
+
+static int bpf_prog_collect_cleanup_info(struct bpf_object *obj,
+ struct bpf_program *prog)
+{
+ size_t i;
+ int j;
+
+ for (i = 0; i < obj->cleanup_rec_cnt; i++) {
+ struct cleanup_raw_rec *raw = &obj->cleanup_recs[i];
+ struct bpf_program *owner = NULL;
+ struct bpf_cleanup_info ci = {};
+ __u32 fields[CLEANUP_REC_FIELDS];
+ void *tmp;
+
+ if (raw->sec_idx[CLEANUP_REC_BEGIN] == raw->sec_idx[CLEANUP_REC_END] &&
+ raw->insn_idx[CLEANUP_REC_BEGIN] >= raw->insn_idx[CLEANUP_REC_END]) {
+ pr_warn("%s: record %zu is an empty range [%zu,%zu)\n",
+ CLEANUP_SEC, i, raw->insn_idx[CLEANUP_REC_BEGIN],
+ raw->insn_idx[CLEANUP_REC_END]);
+ return -LIBBPF_ERRNO__FORMAT;
+ }
+
+ for (j = 0; j < CLEANUP_REC_FIELDS; j++) {
+ size_t idx = raw->insn_idx[j], final;
+ struct bpf_program *p;
+
+ /* The end of a range is exclusive, so it may name the
+ * instruction just past the last one of a function,
+ * which belongs to the next function or to nothing at
+ * all. Ask about the last instruction the range covers,
+ * the way the kernel does.
+ */
+ if (j == CLEANUP_REC_END) {
+ if (!idx) {
+ pr_warn("%s: record %zu ends at instruction 0\n",
+ CLEANUP_SEC, i);
+ return -LIBBPF_ERRNO__FORMAT;
+ }
+ idx--;
+ }
+
+ p = find_prog_by_sec_insn(obj, raw->sec_idx[j], idx);
+ if (!p) {
+ pr_warn("%s: record %zu field %d is not inside a function\n",
+ CLEANUP_SEC, i, j);
+ return -LIBBPF_ERRNO__FORMAT;
+ }
+ if (!owner) {
+ owner = p;
+ } else if (owner != p) {
+ pr_warn("%s: record %zu spans functions '%s' and '%s'\n",
+ CLEANUP_SEC, i, owner->name, p->name);
+ return -LIBBPF_ERRNO__FORMAT;
+ }
+
+ if (owner == prog) {
+ final = raw->insn_idx[j] - prog->sec_insn_off;
+ } else if (prog_is_subprog(obj, owner) && owner->sub_insn_off) {
+ /* sub_insn_off is where this subprogram was
+ * appended to the main program being relocated;
+ * zero means it is not part of it.
+ */
+ final = owner->sub_insn_off +
+ raw->insn_idx[j] - owner->sec_insn_off;
+ } else {
+ owner = NULL;
+ break;
+ }
+ fields[j] = final;
+ }
+ if (!owner)
+ continue;
+
+ ci.begin_off = fields[CLEANUP_REC_BEGIN];
+ ci.end_off = fields[CLEANUP_REC_END];
+ ci.landing_pad_off = fields[CLEANUP_REC_PAD];
+
+ tmp = libbpf_reallocarray(prog->cleanup_info, prog->cleanup_info_cnt + 1,
+ sizeof(*prog->cleanup_info));
+ if (!tmp)
+ return -ENOMEM;
+ prog->cleanup_info = tmp;
+ prog->cleanup_info_rec_size = sizeof(struct bpf_cleanup_info);
+ prog->cleanup_info[prog->cleanup_info_cnt++] = ci;
+
+ pr_debug("prog '%s': cleanup region [%u,%u) -> landing pad %u\n",
+ prog->name, ci.begin_off, ci.end_off, ci.landing_pad_off);
+ }
+
+ if (!prog->cleanup_info_cnt)
+ return 0;
+
+ qsort(prog->cleanup_info, prog->cleanup_info_cnt,
+ sizeof(*prog->cleanup_info), cmp_cleanup_info);
+ for (i = 1; i < prog->cleanup_info_cnt; i++) {
+ struct bpf_cleanup_info *prev = &prog->cleanup_info[i - 1];
+ struct bpf_cleanup_info *cur = &prog->cleanup_info[i];
+
+ if (cur->begin_off < prev->end_off) {
+ pr_warn("prog '%s': overlapping cleanup regions [%u,%u) and [%u,%u)\n",
+ prog->name, prev->begin_off, prev->end_off,
+ cur->begin_off, cur->end_off);
+ return -LIBBPF_ERRNO__FORMAT;
+ }
+ }
+
+ return 0;
+}
+
static int
bpf_object__collect_prog_relos(struct bpf_object *obj, Elf64_Shdr *shdr, Elf_Data *data)
{
@@ -7561,6 +7822,13 @@ static int bpf_object__relocate(struct bpf_object *obj, const char *targ_btf_pat
return err;
}
}
+
+ err = bpf_prog_collect_cleanup_info(obj, prog);
+ if (err) {
+ pr_warn("prog '%s': failed to collect cleanup info: %s\n",
+ prog->name, errstr(err));
+ return err;
+ }
}
for (i = 0; i < obj->nr_programs; i++) {
prog = &obj->programs[i];
@@ -7751,6 +8019,9 @@ static int bpf_object__collect_relos(struct bpf_object *obj)
return -LIBBPF_ERRNO__INTERNAL;
}
+ if (idx == obj->efile.cleanup_shndx)
+ continue;
+
if (obj->efile.secs[idx].sec_type == SEC_ST_OPS)
err = bpf_object__collect_st_ops_relos(obj, shdr, data);
else if (idx == obj->efile.btf_maps_shndx)
@@ -8023,6 +8294,11 @@ static int bpf_object_load_prog(struct bpf_object *obj, struct bpf_program *prog
load_attr.line_info_rec_size = prog->line_info_rec_size;
load_attr.line_info_cnt = prog->line_info_cnt;
}
+ if (prog->cleanup_info_cnt) {
+ load_attr.cleanup_info = prog->cleanup_info;
+ load_attr.cleanup_info_cnt = prog->cleanup_info_cnt;
+ load_attr.cleanup_info_rec_size = prog->cleanup_info_rec_size;
+ }
load_attr.log_level = log_level;
load_attr.prog_flags = prog->prog_flags;
load_attr.fd_array = obj->fd_array;
@@ -8351,6 +8627,15 @@ static void fixup_verifier_log(struct bpf_program *prog, char *buf, size_t buf_s
}
}
+/* LLVM terminates a cleanup landing pad with a call to _Unwind_Resume, the
+ * base unwind ABI's entry point for carrying an unwind on once a frame's
+ * cleanups have run. The kernel knows it as bpf_unwind_resume.
+ */
+static const char *kern_extern_name(const char *name)
+{
+ return strcmp(name, "_Unwind_Resume") ? name : "bpf_unwind_resume";
+}
+
static int bpf_program_record_relos(struct bpf_program *prog)
{
struct bpf_object *obj = prog->obj;
@@ -8367,12 +8652,12 @@ static int bpf_program_record_relos(struct bpf_program *prog)
continue;
kind = btf_is_var(btf__type_by_id(obj->btf, ext->btf_id)) ?
BTF_KIND_VAR : BTF_KIND_FUNC;
- bpf_gen__record_extern(obj->gen_loader, ext->name,
+ bpf_gen__record_extern(obj->gen_loader, kern_extern_name(ext->name),
ext->is_weak, !ext->ksym.type_id,
true, kind, relo->insn_idx);
break;
case RELO_EXTERN_CALL:
- bpf_gen__record_extern(obj->gen_loader, ext->name,
+ bpf_gen__record_extern(obj->gen_loader, kern_extern_name(ext->name),
ext->is_weak, false, false, BTF_KIND_FUNC,
relo->insn_idx);
break;
@@ -8570,6 +8855,7 @@ static struct bpf_object *bpf_object_open(const char *path, const void *obj_buf,
err = err ? : bpf_object__init_maps(obj, opts);
err = err ? : bpf_object_init_progs(obj, opts);
err = err ? : bpf_object__collect_relos(obj);
+ err = err ? : bpf_object__init_cleanup_info(obj);
if (err)
goto out;
@@ -8812,17 +9098,19 @@ static int bpf_object__resolve_ksym_func_btf_id(struct bpf_object *obj,
struct module_btf *mod_btf = NULL;
const struct btf_type *kern_func;
struct btf *kern_btf = NULL;
+ const char *kern_name;
int ret;
local_func_proto_id = ext->ksym.type_id;
- kfunc_id = find_ksym_btf_id(obj, ext->essent_name ?: ext->name, BTF_KIND_FUNC, &kern_btf,
- &mod_btf);
+ kern_name = kern_extern_name(ext->essent_name ?: ext->name);
+
+ kfunc_id = find_ksym_btf_id(obj, kern_name, BTF_KIND_FUNC, &kern_btf, &mod_btf);
if (kfunc_id < 0) {
if (kfunc_id == -ESRCH && ext->is_weak)
return 0;
pr_warn("extern (func ksym) '%s': not found in kernel or module BTFs\n",
- ext->name);
+ strcmp(kern_name, "bpf_unwind_resume") ? ext->name : kern_name);
return kfunc_id;
}
@@ -9681,6 +9969,9 @@ void bpf_object__close(struct bpf_object *obj)
zfree(&obj->jumptables_data);
obj->jumptables_data_sz = 0;
+ zfree(&obj->cleanup_recs);
+ obj->cleanup_rec_cnt = 0;
+
for (i = 0; i < obj->jumptable_map_cnt; i++)
close(obj->jumptable_maps[i].fd);
zfree(&obj->jumptable_maps);
@@ -10073,6 +10364,25 @@ int bpf_program__clone(struct bpf_program *prog, const struct bpf_prog_load_opts
attr.line_info_rec_size = info ? info_rec_size : prog->line_info_rec_size;
}
+ /* exception cleanup table */
+ info = OPTS_GET(opts, cleanup_info, NULL);
+ info_cnt = OPTS_GET(opts, cleanup_info_cnt, 0);
+ info_rec_size = OPTS_GET(opts, cleanup_info_rec_size, 0);
+ if (!!info != !!info_cnt || !!info != !!info_rec_size) {
+ pr_warn("prog '%s': cleanup_info, cleanup_info_cnt, and cleanup_info_rec_size must all be specified or all omitted\n",
+ prog->name);
+ return libbpf_err(-EINVAL);
+ }
+ if (info) {
+ attr.cleanup_info = info;
+ attr.cleanup_info_cnt = info_cnt;
+ attr.cleanup_info_rec_size = info_rec_size;
+ } else if (prog->cleanup_info_cnt) {
+ attr.cleanup_info = prog->cleanup_info;
+ attr.cleanup_info_cnt = prog->cleanup_info_cnt;
+ attr.cleanup_info_rec_size = prog->cleanup_info_rec_size;
+ }
+
/* Logging is caller-controlled; no fallback to prog/obj log settings */
attr.log_buf = OPTS_GET(opts, log_buf, NULL);
attr.log_size = OPTS_GET(opts, log_size, 0);
diff --git a/tools/lib/bpf/libbpf_internal.h b/tools/lib/bpf/libbpf_internal.h
index cb4d962338441..78519f24fb404 100644
--- a/tools/lib/bpf/libbpf_internal.h
+++ b/tools/lib/bpf/libbpf_internal.h
@@ -56,6 +56,9 @@
#ifndef R_BPF_64_ABS32
#define R_BPF_64_ABS32 3
#endif
+#ifndef R_BPF_64_NODYLD32
+#define R_BPF_64_NODYLD32 4
+#endif
#ifndef R_BPF_64_32
#define R_BPF_64_32 10
#endif
@@ -569,6 +572,13 @@ static inline void bpf_core_relo_bswap(struct bpf_core_relo *i)
i->kind = bswap_32(i->kind);
}
+static inline void bpf_cleanup_info_bswap(struct bpf_cleanup_info *i)
+{
+ i->begin_off = bswap_32(i->begin_off);
+ i->end_off = bswap_32(i->end_off);
+ i->landing_pad_off = bswap_32(i->landing_pad_off);
+}
+
enum btf_field_iter_kind {
BTF_FIELD_ITER_IDS,
BTF_FIELD_ITER_STRS,
diff --git a/tools/lib/bpf/linker.c b/tools/lib/bpf/linker.c
index 78f92c39290af..fac7390bd109d 100644
--- a/tools/lib/bpf/linker.c
+++ b/tools/lib/bpf/linker.c
@@ -1036,7 +1036,8 @@ static int linker_sanity_check_elf_relos(struct src_obj *obj, struct src_sec *se
size_t sym_type = ELF64_R_TYPE(relo->r_info);
if (sym_type != R_BPF_64_64 && sym_type != R_BPF_64_32 &&
- sym_type != R_BPF_64_ABS64 && sym_type != R_BPF_64_ABS32) {
+ sym_type != R_BPF_64_ABS64 && sym_type != R_BPF_64_ABS32 &&
+ sym_type != R_BPF_64_NODYLD32) {
pr_warn("ELF relo #%d in section #%zu has unexpected type %zu in %s\n",
i, sec->sec_idx, sym_type, obj->filename);
return -EINVAL;
@@ -2274,6 +2275,39 @@ static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob
insn->imm += sec->dst_off / sizeof(struct bpf_insn);
else
insn->imm += sec->dst_off;
+ } else if (sym_type == R_BPF_64_NODYLD32 ||
+ sym_type == R_BPF_64_ABS32) {
+ __u32 *val;
+
+ /* Two spellings of the one thing: LLVM
+ * emits NODYLD32 for a .long against a
+ * section symbol, GNU as emits ABS32
+ * (bpf_reloc_type_lookup() maps
+ * BFD_RELOC_32 to it), and the value
+ * they describe is the same.
+ */
+
+ /* An SHT_NOBITS section has no
+ * raw_data, and r_offset is sanity
+ * checked only for sections holding
+ * instructions. Check both here
+ * before writing.
+ */
+ if (!dst_linked_sec->raw_data ||
+ dst_rel->r_offset % sizeof(*val) ||
+ dst_rel->r_offset + sizeof(*val) >
+ (size_t)dst_linked_sec->sec_sz) {
+ pr_warn("ELF relo #%d in section #%zu points outside the data of section '%s' in %s\n",
+ j, src_sec->sec_idx,
+ dst_linked_sec->sec_name,
+ obj->filename);
+ return -EINVAL;
+ }
+ val = dst_linked_sec->raw_data + dst_rel->r_offset;
+ if (linker->swapped_endian)
+ *val = bswap_32(bswap_32(*val) + sec->dst_off);
+ else
+ *val += sec->dst_off;
} else {
pr_warn("relocation against STT_SECTION in non-exec section is not supported!\n");
return -EINVAL;
diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile
index 7ea5ba1df29e3..981cc26430495 100644
--- a/tools/testing/selftests/bpf/Makefile
+++ b/tools/testing/selftests/bpf/Makefile
@@ -523,7 +523,7 @@ LINKED_SKELS := test_static_linked.skel.h linked_funcs.skel.h \
LSKELS := fexit_sleep.c trace_printk.c trace_vprintk.c map_ptr_kern.c \
core_kern.c core_kern_overflow.c test_ringbuf.c \
test_ringbuf_n.c test_ringbuf_map_key.c test_ringbuf_write.c \
- test_ringbuf_overwrite.c
+ test_ringbuf_overwrite.c exceptions_cleanup_light.c
LSKELS_SIGNED := fentry_test.c fexit_test.c atomics.c
diff --git a/tools/testing/selftests/bpf/exceptions_cleanup.h b/tools/testing/selftests/bpf/exceptions_cleanup.h
new file mode 100644
index 0000000000000..e14b651263b88
--- /dev/null
+++ b/tools/testing/selftests/bpf/exceptions_cleanup.h
@@ -0,0 +1,55 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#ifndef __EXCEPTIONS_CLEANUP_H__
+#define __EXCEPTIONS_CLEANUP_H__
+
+#define THROW_COOKIE 0x100
+#define INNER_COOKIE 0x200
+
+/* progs/exceptions_cleanup.c: one bit per frame that reports it ran. */
+#define RAN_FOO3_PREEMPT 0x1
+#define RAN_FOO2_RCU 0x2
+#define RAN_FOO1V_PREEMPT 0x4
+#define RAN_FOO2_DROP 0x8
+#define RAN_BUMP 0x10
+
+/* progs/exceptions_cleanup_shapes.c: one bit per shape, numbered its own way. */
+#define RAN_SWEEP 0x1
+#define RAN_SHARED 0x2
+#define RAN_REGS 0x4
+#define RAN_TAIL_CALL 0x8
+#define RAN_MAIN_PAD 0x10
+#define RAN_TC_TAKEN 0x20
+#define RAN_FREPLACE 0x40
+#define RAN_ADDR_TAKEN 0x80
+#define RAN_NO_SUBPROG 0x100
+#define RAN_PAD_CALLS 0x200
+#define RAN_PAD_FIRST 0x400
+#define RAN_WIDE_REC 0x800
+#define RAN_PAD_STACK 0x1000
+#define RAN_DEEP_PAD 0x2000
+#define RAN_NOUNWIND_REC 0x4000
+#define RAN_RESUME_ALIAS 0x8000
+#define RAN_PAD_TAIL_CALL 0x10000
+#define RAN_NOP_PAD 0x20000
+#define RAN_VAR_STACK 0x40000
+#define RAN_GLOBAL_PAD 0x80000
+
+/* progs/exceptions_cleanup_light.c: the one pad it has. */
+#define RAN_LIGHT 0x1
+
+#define CLEANUP_REC(begin, end, landing_pad) \
+ ".pushsection .bpf_cleanup,\"a\",@progbits;" \
+ ".long " begin ";" \
+ ".long " end ";" \
+ ".long " landing_pad ";" \
+ ".popsection;"
+
+/* Set a bit in @pads_ran. */
+#define PAD_RAN(bit) \
+ "r1 = %[pads_ran] ll;" \
+ "r2 = *(u64 *)(r1 + 0);" \
+ "r2 |= " bit ";" \
+ "*(u64 *)(r1 + 0) = r2;"
+
+#endif /* __EXCEPTIONS_CLEANUP_H__ */
diff --git a/tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c b/tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c
new file mode 100644
index 0000000000000..b69be77b2e26c
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c
@@ -0,0 +1,454 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <test_progs.h>
+#include "exceptions_cleanup.h"
+#include "exceptions_cleanup.skel.h"
+#include "exceptions_cleanup_fail.skel.h"
+#include "exceptions_cleanup_shapes.skel.h"
+#include "exceptions_cleanup_freplace.skel.h"
+#include "exceptions_cleanup_pad_freplace.skel.h"
+#include "exceptions_cleanup_ext_table.skel.h"
+#include "exceptions_cleanup_light.lskel.h"
+
+/* foo3 threw: every frame that has a pad ran it. */
+#define PADS_FOO3_THREW \
+ (RAN_FOO3_PREEMPT | RAN_FOO2_RCU | RAN_FOO1V_PREEMPT | RAN_FOO2_DROP)
+
+/* foo2 threw after foo3 returned normally: foo3's pad must not run. */
+#define PADS_FOO2_THREW \
+ (RAN_FOO2_RCU | RAN_FOO1V_PREEMPT | RAN_FOO2_DROP)
+
+static void run(struct exceptions_cleanup *skel, __u64 input, __u32 retval,
+ __u64 pads)
+{
+ __u64 ctx = 0;
+ int err;
+
+ LIBBPF_OPTS(bpf_test_run_opts, topts,
+ .ctx_in = &ctx,
+ .ctx_size_in = sizeof(ctx),
+ );
+
+ skel->bss->input = input;
+ skel->bss->pads_ran = 0;
+ skel->bss->result = 0;
+
+ err = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.entry), &topts);
+ if (!ASSERT_OK(err, "run"))
+ return;
+ ASSERT_EQ(topts.retval, retval, "retval");
+ ASSERT_EQ(skel->bss->pads_ran, pads | RAN_BUMP, "pads_ran");
+}
+
+static void run_shape(struct exceptions_cleanup_shapes *skel, struct bpf_program *prog,
+ __u64 input, __u32 retval, __u64 pads)
+{
+ __u64 ctx = 0;
+ int err;
+
+ LIBBPF_OPTS(bpf_test_run_opts, topts,
+ .ctx_in = &ctx,
+ .ctx_size_in = sizeof(ctx),
+ );
+
+ skel->bss->input = input;
+ skel->bss->pads_ran = 0;
+
+ err = bpf_prog_test_run_opts(bpf_program__fd(prog), &topts);
+ if (!ASSERT_OK(err, "run"))
+ return;
+ ASSERT_EQ(topts.retval, retval, "retval");
+ ASSERT_EQ(skel->bss->pads_ran, pads, "pads_ran");
+}
+
+static void test_freplace(struct exceptions_cleanup_shapes *skel)
+{
+ struct exceptions_cleanup_freplace *fr;
+ struct bpf_link *link;
+ int tgt_fd;
+
+ tgt_fd = bpf_program__fd(skel->progs.entry_freplace);
+
+ fr = exceptions_cleanup_freplace__open();
+ if (!ASSERT_OK_PTR(fr, "freplace open"))
+ return;
+
+ if (!ASSERT_OK(bpf_program__set_attach_target(fr->progs.new_fr_callee,
+ tgt_fd, "fr_callee"),
+ "set_attach_target"))
+ goto out;
+ if (!ASSERT_OK(exceptions_cleanup_freplace__load(fr), "freplace load"))
+ goto out;
+
+ link = bpf_program__attach_freplace(fr->progs.new_fr_callee, tgt_fd,
+ "fr_callee");
+ if (!ASSERT_OK_PTR(link, "attach_freplace"))
+ goto out;
+
+ run_shape(skel, skel->progs.entry_freplace, 101, THROW_COOKIE, 0);
+ bpf_link__destroy(link);
+out:
+ exceptions_cleanup_freplace__destroy(fr);
+}
+
+static void test_pad_calls_freplace(struct exceptions_cleanup_shapes *skel)
+{
+ struct exceptions_cleanup_pad_freplace *fr;
+ struct bpf_link *link;
+ __u64 ctx = 0;
+ int tgt_fd, err;
+
+ LIBBPF_OPTS(bpf_test_run_opts, topts,
+ .ctx_in = &ctx,
+ .ctx_size_in = sizeof(ctx),
+ );
+
+ tgt_fd = bpf_program__fd(skel->progs.entry_pad_calls);
+
+ fr = exceptions_cleanup_pad_freplace__open();
+ if (!ASSERT_OK_PTR(fr, "pad freplace open"))
+ return;
+
+ if (!ASSERT_OK(bpf_program__set_attach_target(fr->progs.new_pad_callee,
+ tgt_fd, "pad_callee"),
+ "set_attach_target"))
+ goto out;
+ if (!ASSERT_OK(exceptions_cleanup_pad_freplace__load(fr), "pad freplace load"))
+ goto out;
+
+ link = bpf_program__attach_freplace(fr->progs.new_pad_callee, tgt_fd,
+ "pad_callee");
+ if (!ASSERT_OK_PTR(link, "attach_freplace"))
+ goto out;
+
+ skel->bss->input = 101;
+ skel->bss->pads_ran = 0;
+ skel->bss->pad_runs = 0;
+
+ err = bpf_prog_test_run_opts(tgt_fd, &topts);
+ if (!ASSERT_OK(err, "run"))
+ goto out_link;
+
+ ASSERT_EQ(skel->bss->pad_runs, 1, "pad_runs");
+ ASSERT_EQ(skel->bss->pads_ran, RAN_PAD_CALLS, "pads_ran");
+ ASSERT_EQ(topts.retval, THROW_COOKIE, "retval");
+out_link:
+ bpf_link__destroy(link);
+out:
+ exceptions_cleanup_pad_freplace__destroy(fr);
+}
+
+static void test_ext_table(struct exceptions_cleanup_shapes *skel)
+{
+ struct exceptions_cleanup_ext_table *fr;
+ struct bpf_link *link;
+ int tgt_fd;
+
+ tgt_fd = bpf_program__fd(skel->progs.entry_freplace);
+
+ fr = exceptions_cleanup_ext_table__open();
+ if (!ASSERT_OK_PTR(fr, "ext table open"))
+ return;
+
+ if (!ASSERT_OK(bpf_program__set_attach_target(fr->progs.new_fr_callee,
+ tgt_fd, "fr_callee"),
+ "set_attach_target"))
+ goto out;
+ if (!ASSERT_OK(exceptions_cleanup_ext_table__load(fr), "ext table load"))
+ goto out;
+
+ link = bpf_program__attach_freplace(fr->progs.new_fr_callee, tgt_fd,
+ "fr_callee");
+ if (!ASSERT_OK_PTR(link, "attach_freplace"))
+ goto out;
+
+ fr->bss->ext_pad_ran = 0;
+ run_shape(skel, skel->progs.entry_freplace, 101, THROW_COOKIE, 0);
+ ASSERT_EQ(fr->bss->ext_pad_ran, 1, "ext_pad_ran");
+
+ bpf_link__destroy(link);
+out:
+ exceptions_cleanup_ext_table__destroy(fr);
+}
+
+static void test_light_skeleton(void)
+{
+ struct exceptions_cleanup_light_lskel *skel;
+ __u64 ctx = 0;
+ int err;
+
+ LIBBPF_OPTS(bpf_test_run_opts, topts,
+ .ctx_in = &ctx,
+ .ctx_size_in = sizeof(ctx),
+ );
+
+ skel = exceptions_cleanup_light_lskel__open_and_load();
+ if (!ASSERT_OK_PTR(skel, "light open_and_load"))
+ return;
+
+ err = bpf_prog_test_run_opts(skel->progs.entry_light.prog_fd, &topts);
+ if (!ASSERT_OK(err, "run"))
+ goto out;
+ ASSERT_EQ(topts.retval, THROW_COOKIE, "retval");
+ ASSERT_EQ(skel->bss->pads_ran, RAN_LIGHT, "pads_ran");
+out:
+ exceptions_cleanup_light_lskel__destroy(skel);
+}
+
+static void test_shapes(void)
+{
+ struct exceptions_cleanup_shapes *skel;
+
+ skel = exceptions_cleanup_shapes__open_and_load();
+ if (!ASSERT_OK_PTR(skel, "shapes open_and_load"))
+ return;
+
+ /* The frame loads at all only if everything unreachable in it went. */
+ if (test__start_subtest("sweep_no_throw"))
+ run_shape(skel, skel->progs.entry_sweep, 1, 0, 0);
+ if (test__start_subtest("sweep_throw"))
+ run_shape(skel, skel->progs.entry_sweep, 101, THROW_COOKIE, RAN_SWEEP);
+
+ /* The covered call unwinds to the pad; the uncovered one walks past. */
+ if (test__start_subtest("shared_callee_no_throw")) {
+ skel->bss->outer_input = 0;
+ run_shape(skel, skel->progs.entry_shared, 1, 2, 0);
+ }
+ if (test__start_subtest("shared_callee_throw")) {
+ skel->bss->outer_input = 0;
+ run_shape(skel, skel->progs.entry_shared, 101, THROW_COOKIE, RAN_SHARED);
+ }
+ if (test__start_subtest("shared_callee_uncovered_throw")) {
+ skel->bss->outer_input = 101;
+ run_shape(skel, skel->progs.entry_shared, 1, THROW_COOKIE, 0);
+ skel->bss->outer_input = 0;
+ }
+
+ /* The pad only sets its bit if it got the frame's own r6-r9 back. */
+ if (test__start_subtest("pad_sees_callee_saved"))
+ run_shape(skel, skel->progs.entry_regs, 101, THROW_COOKIE, RAN_REGS);
+
+ /* Same check, with a tail-call-reachable callee: its spill moves. */
+ if (test__start_subtest("tail_call_no_throw"))
+ run_shape(skel, skel->progs.entry_tail_call, 1, 0, 0);
+ if (test__start_subtest("tail_call_throw"))
+ run_shape(skel, skel->progs.entry_tail_call, 101, THROW_COOKIE,
+ RAN_TAIL_CALL);
+
+ /* A region around a nounwind call: no pad dispatched, still loads. */
+ if (test__start_subtest("nounwind_region"))
+ run_shape(skel, skel->progs.entry_nounwind_rec, 1, 0, 0);
+
+ /* A pad in the main program's own frame, not in a subprogram. */
+ if (test__start_subtest("main_program_pad"))
+ run_shape(skel, skel->progs.entry_main_pad, 101, THROW_COOKIE,
+ RAN_MAIN_PAD);
+
+ /* The same call site either way: the subprogram's throw unwinds into
+ * this frame and runs its pad, an extension's stops at its own boundary.
+ */
+ if (test__start_subtest("freplace_subprog_throws"))
+ run_shape(skel, skel->progs.entry_freplace, 7, THROW_COOKIE,
+ RAN_FREPLACE);
+ if (test__start_subtest("freplace_extension_throws"))
+ test_freplace(skel);
+
+ /* A tail call that is taken: the walk ends at the target, so the cookie
+ * comes back from there and this frame's pad does not run -- though it
+ * is reachable, so a walk past the boundary would find it.
+ */
+ if (test__start_subtest("tail_call_taken")) {
+ int key = 0, prog_fd = bpf_program__fd(skel->progs.tc_target);
+
+ if (ASSERT_OK(bpf_map_update_elem(bpf_map__fd(skel->maps.taken_table),
+ &key, &prog_fd, BPF_ANY),
+ "populate taken_table"))
+ run_shape(skel, skel->progs.entry_tail_taken, 101,
+ THROW_COOKIE, 0);
+ }
+
+ /* A throwing subprog named by a BPF_PSEUDO_FUNC and handed to a
+ * bpf_loop() the verifier never reaches: the callback check has to fire
+ * on the helper call, not on the ld_imm64.
+ */
+ if (test__start_subtest("addr_taken_no_throw"))
+ run_shape(skel, skel->progs.entry_addr_taken, 1, 2, 0);
+ if (test__start_subtest("addr_taken_throw"))
+ run_shape(skel, skel->progs.entry_addr_taken, 101, THROW_COOKIE,
+ RAN_ADDR_TAKEN);
+
+ /* A record covering bpf_throw() itself rather than a call to a frame
+ * that throws: raised, caught up with and delivered in one frame.
+ */
+ if (test__start_subtest("no_subprog_no_throw"))
+ run_shape(skel, skel->progs.entry_no_subprog, 1, 0, 0);
+ if (test__start_subtest("no_subprog_throw"))
+ run_shape(skel, skel->progs.entry_no_subprog, 101, THROW_COOKIE,
+ RAN_NO_SUBPROG);
+
+ /* A pad that calls a subprogram; with a throwing extension in its place,
+ * the nested exception has to stop there, not restart this pad.
+ */
+ if (test__start_subtest("pad_calls_subprog")) {
+ skel->bss->pad_runs = 0;
+ run_shape(skel, skel->progs.entry_pad_calls, 101, THROW_COOKIE,
+ RAN_PAD_CALLS);
+ ASSERT_EQ(skel->bss->pad_runs, 1, "pad_runs");
+ }
+ if (test__start_subtest("pad_calls_throwing_extension"))
+ test_pad_calls_freplace(skel);
+
+ /* A covered throw the sweep leaves last, where the default exception
+ * callback is patched in; the pad's bit needs r6-r9 still spilled.
+ */
+ if (test__start_subtest("pad_before_throw"))
+ run_shape(skel, skel->progs.entry_pad_first, 101, THROW_COOKIE,
+ RAN_PAD_FIRST);
+
+ /* A region whose last instruction is a 16-byte one, so that end - 1
+ * names the half of it that is not an instruction.
+ */
+ if (test__start_subtest("region_ends_on_ldimm64"))
+ run_shape(skel, skel->progs.entry_wide_rec, 101, THROW_COOKIE,
+ RAN_WIDE_REC);
+
+ /* A pad that reloads from and writes to its own frame's stack, which a
+ * JIT addressing the frame through the stack pointer gets wrong.
+ */
+ if (test__start_subtest("pad_uses_own_frame"))
+ run_shape(skel, skel->progs.entry_pad_stack, 101, THROW_COOKIE,
+ RAN_PAD_STACK);
+
+ /* The same, with an uncovered frame between the throw and the pad. */
+ if (test__start_subtest("pad_two_frames_up"))
+ run_shape(skel, skel->progs.entry_deep_pad, 101, THROW_COOKIE,
+ RAN_DEEP_PAD);
+
+ /* An extension program with a cleanup table of its own. */
+ if (test__start_subtest("extension_carries_table"))
+ test_ext_table(skel);
+
+ /* A pad terminated by _Unwind_Resume, which libbpf maps onto the kfunc;
+ * every other program here calls bpf_unwind_resume directly.
+ */
+ if (test__start_subtest("resume_alias"))
+ run_shape(skel, skel->progs.entry_resume_alias, 101,
+ THROW_COOKIE, RAN_RESUME_ALIAS);
+
+ /* A pad that calls a subprogram which tail calls, array empty and then
+ * populated: the tail call releases only the callee's own prologue.
+ */
+ if (test__start_subtest("pad_callee_tail_call")) {
+ int key = 0, prog_fd = bpf_program__fd(skel->progs.pad_tc_target);
+
+ skel->bss->pad_tc_target_ran = 0;
+ skel->bss->pad_runs = 0;
+ run_shape(skel, skel->progs.entry_pad_tail_call, 101,
+ THROW_COOKIE, RAN_PAD_TAIL_CALL);
+ ASSERT_EQ(skel->bss->pad_tc_target_ran, 0, "target not run");
+ ASSERT_EQ(skel->bss->pad_runs, 1, "pad_runs");
+
+ if (ASSERT_OK(bpf_map_update_elem(bpf_map__fd(skel->maps.pad_tc_table),
+ &key, &prog_fd, BPF_ANY),
+ "populate pad_tc_table")) {
+ skel->bss->pad_runs = 0;
+ run_shape(skel, skel->progs.entry_pad_tail_call, 101,
+ THROW_COOKIE, RAN_PAD_TAIL_CALL);
+ ASSERT_EQ(skel->bss->pad_tc_target_ran, 1, "target ran");
+ ASSERT_EQ(skel->bss->pad_runs, 1, "pad_runs");
+ }
+ }
+
+ /* The same, into a target that carries a table and throws: that target
+ * is a boundary, so the outer pad runs once, not twice.
+ */
+ if (test__start_subtest("pad_callee_tail_call_throws")) {
+ int key = 0, prog_fd = bpf_program__fd(skel->progs.pad_tc_throw_target);
+
+ if (ASSERT_OK(bpf_map_update_elem(bpf_map__fd(skel->maps.pad_tc_table),
+ &key, &prog_fd, BPF_ANY),
+ "populate pad_tc_table")) {
+ skel->bss->pad_runs = 0;
+ skel->bss->tc_target_pad_runs = 0;
+ run_shape(skel, skel->progs.entry_pad_tail_call, 101,
+ THROW_COOKIE, RAN_PAD_TAIL_CALL);
+ /* The target cleaned up after itself, once. */
+ ASSERT_EQ(skel->bss->tc_target_pad_runs, 1,
+ "tc_target_pad_runs");
+ /* And the outer pad was not started over. */
+ ASSERT_EQ(skel->bss->pad_runs, 1, "pad_runs");
+ }
+ }
+
+ /* A pad whose first instruction opt_remove_nops() deletes: the record
+ * has to follow the pad rather than be dropped with the nop.
+ */
+ if (test__start_subtest("nop_at_pad_head"))
+ run_shape(skel, skel->progs.entry_nop_pad, 101, THROW_COOKIE,
+ RAN_NOP_PAD);
+
+ /* A pad that indexes its frame's stack by a register the frame set
+ * before the throwing call, marked precise back across the unwind edge.
+ */
+ if (test__start_subtest("pad_var_stack_offset"))
+ run_shape(skel, skel->progs.entry_var_stack, 101, THROW_COOKIE,
+ RAN_VAR_STACK);
+
+ /* The same, where the throw is in a global subprogram, which the
+ * verifier walks without a frame of its own.
+ */
+ if (test__start_subtest("pad_over_global_subprog"))
+ run_shape(skel, skel->progs.entry_global_pad, 101, THROW_COOKIE,
+ RAN_GLOBAL_PAD);
+
+ exceptions_cleanup_shapes__destroy(skel);
+}
+
+void test_exceptions_cleanup(void)
+{
+ char log[8192] = {};
+
+ LIBBPF_OPTS(bpf_object_open_opts, opts,
+ .kernel_log_buf = log,
+ .kernel_log_size = sizeof(log));
+ struct exceptions_cleanup *skel;
+ int err;
+
+ skel = exceptions_cleanup__open_opts(&opts);
+ if (!ASSERT_OK_PTR(skel, "open"))
+ return;
+
+ err = exceptions_cleanup__load(skel);
+ if (err) {
+ if (err == -EOPNOTSUPP &&
+ strstr(log, "exception cleanup needs a JIT that can dispatch landing pads"))
+ test__skip();
+ else if (!ASSERT_OK(err, "load"))
+ fprintf(stderr, "%s", log);
+ exceptions_cleanup__destroy(skel);
+ return;
+ }
+
+ /* No throw: foo3 returns 1 ^ 1 == 0, foo2 adds one, no pad runs. */
+ if (test__start_subtest("no_throw"))
+ run(skel, 1, 1, 0);
+
+ /* foo3 throws; every pad runs and the cookie is delivered at entry. */
+ if (test__start_subtest("throw_from_foo3"))
+ run(skel, 101, THROW_COOKIE, PADS_FOO3_THREW);
+
+ /* foo3 returns 2 ^ 1 == 3, so foo2 throws from its own second region;
+ * foo3's frame is long gone, so its pad must not run.
+ */
+ if (test__start_subtest("throw_from_foo2"))
+ run(skel, 2, THROW_COOKIE, PADS_FOO2_THREW);
+
+ exceptions_cleanup__destroy(skel);
+
+ if (test__start_subtest("light_skeleton"))
+ test_light_skeleton();
+
+ test_shapes();
+
+ RUN_TESTS(exceptions_cleanup_fail);
+}
diff --git a/tools/testing/selftests/bpf/progs/exceptions_cleanup.c b/tools/testing/selftests/bpf/progs/exceptions_cleanup.c
new file mode 100644
index 0000000000000..95199a2828faa
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/exceptions_cleanup.c
@@ -0,0 +1,152 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+#include "exceptions_cleanup.h"
+
+static __used __noinline void __kfunc_btf_anchor(void)
+{
+ bpf_throw(0);
+ bpf_rcu_read_lock();
+ bpf_rcu_read_unlock();
+ bpf_preempt_disable();
+ bpf_preempt_enable();
+ bpf_unwind_resume();
+}
+
+__u64 input = 0;
+__u64 pads_ran = 0;
+__u64 result = 0;
+
+static __used __noinline __u64 foo3(__u64 x)
+{
+ bpf_preempt_disable();
+ if (x > 100)
+ asm volatile (
+ "r1 = %[cookie];"
+ "1:" "call bpf_throw;" /* cleanup region */
+ "2:"
+ "goto 3f;"
+ "4:" /* landing pad */
+ "r7 = r0;"
+ "call bpf_preempt_enable;"
+ PAD_RAN("%[ran]")
+ "r1 = r7;"
+ "call bpf_unwind_resume;"
+ "3:"
+ CLEANUP_REC("1b", "2b", "4b")
+ :
+ : [cookie]"i"(THROW_COOKIE), [ran]"i"(RAN_FOO3_PREEMPT),
+ __imm_addr(pads_ran)
+ : __clobber_all);
+ bpf_preempt_enable();
+ return x ^ 1;
+}
+
+__u64 never = 0;
+
+static __used __naked __noinline void drop_glue(void)
+{
+ asm volatile (
+ PAD_RAN("%[ran]")
+ "exit;"
+ :
+ : [ran]"i"(RAN_FOO2_DROP), __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+static __used __naked __noinline __u64 foo2(void)
+{
+ asm volatile (
+ "r6 = r1;"
+ "call bpf_rcu_read_lock;"
+ "r1 = r6;"
+"1:" "call foo3;" /* cleanup region #1 */
+"2:"
+ "r6 = r0;"
+ "if r6 == 0 goto 5f;"
+ "r1 = %[cookie];"
+"3:" "call bpf_throw;" /* cleanup region #2 */
+"4:"
+ "r0 = 0;"
+ "exit;"
+"5:"
+ "call bpf_rcu_read_unlock;"
+ "r0 = r6;"
+ "r0 += 1;"
+ "exit;"
+"6:" /* landing pad, shared by both regions */
+ "call drop_glue;"
+ "call bpf_rcu_read_unlock;"
+ PAD_RAN("%[ran_rcu]")
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "6b")
+ CLEANUP_REC("3b", "4b", "6b")
+ :
+ : [cookie]"i"(THROW_COOKIE), [ran_rcu]"i"(RAN_FOO2_RCU),
+ __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+static __used __naked __noinline void foo1v(void)
+{
+ asm volatile (
+ "call bpf_preempt_disable;"
+ "r1 = %[input] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+"1:" "call foo2;" /* cleanup region */
+"2:"
+ "r6 = r0;"
+ "call bpf_preempt_enable;"
+ "r1 = %[result] ll;"
+ "*(u64 *)(r1 + 0) = r6;"
+ "goto 7f;"
+"8:" /* landing pad */
+ "call bpf_preempt_enable;"
+ PAD_RAN("%[ran]")
+ "goto 9f;"
+"7:" /* the frame's own exit block */
+ "r0 = 0;"
+ "exit;"
+"9:" /* shared resume block */
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "8b")
+ :
+ : [ran]"i"(RAN_FOO1V_PREEMPT), __imm_addr(input),
+ __imm_addr(result), __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+static __used __naked __noinline void bump(void)
+{
+ asm volatile (
+ PAD_RAN("%[ran]")
+ "r1 = %[never] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+ "if r1 == 0 goto 1f;"
+ "r1 = 0;"
+ "call bpf_throw;"
+"1:"
+ "exit;" /* r0 deliberately left alone */
+ :
+ : [ran]"i"(RAN_BUMP), __imm_addr(never), __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+__noinline __u64 foo1(void)
+{
+ bump();
+ foo1v();
+ return result;
+}
+
+SEC("syscall")
+int entry(void *ctx)
+{
+ return foo1();
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/exceptions_cleanup_ext_table.c b/tools/testing/selftests/bpf/progs/exceptions_cleanup_ext_table.c
new file mode 100644
index 0000000000000..d14db48d6b29b
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/exceptions_cleanup_ext_table.c
@@ -0,0 +1,48 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+#include "exceptions_cleanup.h"
+
+__u64 ext_pad_ran = 0;
+
+/* Without a 32-bit int in BTF, libbpf's dummy_ksym var gets type id 0. */
+int btf_int_anchor;
+
+static __used __noinline void __kfunc_btf_anchor(void)
+{
+ bpf_throw(0);
+ bpf_preempt_disable();
+ bpf_preempt_enable();
+ bpf_unwind_resume();
+}
+
+static __used __naked __noinline __u64 ext_frame(void)
+{
+ asm volatile (
+ "call bpf_preempt_disable;"
+ "r1 = %[cookie];"
+"1:" "call bpf_throw;" /* cleanup region */
+"2:"
+ "exit;"
+"3:" /* landing pad */
+ "call bpf_preempt_enable;"
+ "r1 = %[ext_pad_ran] ll;"
+ "r2 = 1;"
+ "*(u64 *)(r1 + 0) = r2;"
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : [cookie]"i"(THROW_COOKIE), __imm_addr(ext_pad_ran)
+ : __clobber_all);
+}
+
+SEC("freplace/fr_callee")
+__u64 new_fr_callee(__u64 x)
+{
+ return ext_frame();
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c b/tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c
new file mode 100644
index 0000000000000..db6ac7d8bd6de
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c
@@ -0,0 +1,662 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_experimental.h"
+#include "bpf_misc.h"
+#include "../test_kmods/bpf_testmod_kfunc.h"
+#include "exceptions_cleanup.h"
+
+__u64 input = 0;
+
+static __used __noinline void __kfunc_btf_anchor(void)
+{
+ bpf_throw(0);
+ bpf_preempt_disable();
+ bpf_preempt_enable();
+ bpf_unwind_resume();
+}
+
+/* 1. A subprogram that may unwind, also used as a helper callback:
+ * bpf_loop()'s own kernel frame would end the walk before it found a
+ * boundary.
+ */
+static int throwing_cb(__u32 idx, void *ctx)
+{
+ bpf_throw(0xbad);
+ return 0;
+}
+
+static __used __naked __noinline __u64 cb_frame(void)
+{
+ asm volatile (
+ "call bpf_preempt_disable;"
+"1:" "call throwing_cb;" /* cleanup region */
+"2:"
+ "call bpf_preempt_enable;"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ "call bpf_preempt_enable;"
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ ::: __clobber_all);
+}
+
+SEC("?syscall")
+__failure __msg("may unwind and is used as a callback")
+int callback_may_unwind(void *ctx)
+{
+ bpf_loop(1, throwing_cb, NULL, 0);
+ return cb_frame();
+}
+
+/* 2. A landing pad that reaches both an unwind resume and a plain exit, so
+ * nothing says whether it is a cleanup pad or a catch pad.
+ */
+static __used __naked __noinline __u64 inner_throw(void)
+{
+ asm volatile (
+ "r1 = 1;"
+ "call bpf_throw;"
+ "r0 = 0;"
+ "exit;"
+ ::: __clobber_all);
+}
+
+static __used __naked __noinline __u64 ambiguous_pad_frame(void)
+{
+ asm volatile (
+ "r6 = r1;"
+ "call bpf_preempt_disable;"
+"1:" "call inner_throw;" /* cleanup region */
+"2:"
+ "call bpf_preempt_enable;"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad: two ways out */
+ "call bpf_preempt_enable;"
+ "if r6 > 10 goto 4f;"
+ "call bpf_unwind_resume;"
+ "exit;"
+"4:"
+ "r0 = 0;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ ::: __clobber_all);
+}
+
+SEC("?syscall")
+__failure __msg("reaches both bpf_unwind_resume() and a plain exit")
+int ambiguous_landing_pad(void *ctx)
+{
+ return ambiguous_pad_frame();
+}
+
+/* 3. A throw from inside a landing pad: a second walk over the frames the
+ * first one is in the middle of discarding.
+ */
+static __used __naked __noinline __u64 throw_in_pad_frame(void)
+{
+ asm volatile (
+ "call bpf_preempt_disable;"
+"1:" "call inner_throw;" /* cleanup region */
+"2:"
+ "call bpf_preempt_enable;"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad that throws again */
+ "call bpf_preempt_enable;"
+ "r1 = 2;"
+ "call bpf_throw;"
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ ::: __clobber_all);
+}
+
+SEC("?syscall")
+__failure __msg("can throw while an exception is in flight")
+int throw_from_landing_pad(void *ctx)
+{
+ return throw_in_pad_frame();
+}
+
+/* 4. A cleanup table in a program that also installs an exception callback,
+ * two different answers to what runs on the way out.
+ */
+__noinline int unused_exc_cb(u64 cookie)
+{
+ return 0;
+}
+
+static __used __naked __noinline __u64 cb_and_table_frame(void)
+{
+ asm volatile (
+ "call bpf_preempt_disable;"
+ "r1 = 9;"
+"1:" "call bpf_throw;" /* cleanup region */
+"2:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ "call bpf_preempt_enable;"
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ ::: __clobber_all);
+}
+
+SEC("?syscall")
+__exception_cb(unused_exc_cb)
+__failure __msg("cannot be combined with an exception callback")
+int table_with_exception_cb(void *ctx)
+{
+ return cb_and_table_frame();
+}
+
+__u64 never;
+
+/* 5. A landing pad that calls a subprogram which can throw. Not case 3: the
+ * throw is in another subprogram, so what catches it is the walk of the pad's
+ * body, off subprog_info.might_throw.
+ */
+static __used __noinline void pad_callee_that_throws(void)
+{
+ if (never)
+ bpf_throw(0);
+}
+
+static __used __naked __noinline __u64 pad_calls_thrower_frame(void)
+{
+ asm volatile (
+ "call bpf_preempt_disable;"
+ "r1 = 11;"
+"1:" "call bpf_throw;" /* cleanup region */
+"2:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ "call pad_callee_that_throws;" /* ...which can throw: refused */
+ "call bpf_preempt_enable;"
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ ::: __clobber_all);
+}
+
+SEC("?syscall")
+__failure __msg("which can throw while an exception is in flight")
+int pad_calls_thrower(void *ctx)
+{
+ return pad_calls_thrower_frame();
+}
+
+/* 6. A catch pad: it ends in a plain exit rather than a resume, and a walker
+ * that calls pads as subroutines cannot hand a frame back its own execution.
+ */
+static __used __naked __noinline __u64 catch_pad_frame(void)
+{
+ asm volatile (
+ "call bpf_preempt_disable;"
+ "r1 = 12;"
+"1:" "call bpf_throw;" /* cleanup region */
+"2:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* catch pad: no resume, it stops here */
+ "call bpf_preempt_enable;"
+ "r0 = 0;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ ::: __clobber_all);
+}
+
+SEC("?syscall")
+__failure __msg("is not supported yet, only cleanup pads that resume")
+int catch_landing_pad(void *ctx)
+{
+ return catch_pad_frame();
+}
+
+/* 7. An exception reaching the boundary of a program type that constrains its
+ * return value: delivery makes the cookie that return value, and fentry has
+ * to return 0.
+ */
+static __used __naked __noinline __u64 boundary_throw_frame(void)
+{
+ asm volatile (
+ "call bpf_preempt_disable;"
+ "r1 = 7;"
+"1:" "call bpf_throw;" /* cleanup region */
+"2:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ "call bpf_preempt_enable;"
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ ::: __clobber_all);
+}
+
+SEC("?fentry/bpf_fentry_test1")
+__failure __msg("the register R1 has smin=7 smax=7 should have been in [0, 0]")
+int boundary_delivers(void *ctx)
+{
+ return boundary_throw_frame();
+}
+
+/* 8. A bpf_unwind_resume() outside any landing pad. Both JITs lower it as the
+ * way back out of a pad, which in ordinary code leaves a live frame standing
+ * with its epilogue skipped.
+ */
+static __used __naked __noinline __u64 stray_resume_frame(void)
+{
+ asm volatile (
+ "call bpf_preempt_disable;"
+ "r1 = 13;"
+"1:" "call bpf_throw;" /* cleanup region */
+"2:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ "call bpf_preempt_enable;"
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ ::: __clobber_all);
+}
+
+SEC("?syscall")
+__failure __msg("is not in an exception cleanup landing pad")
+int resume_outside_pad(void *ctx)
+{
+ /* Never taken, but reachable, which is all the verifier needs. */
+ if (never)
+ bpf_unwind_resume();
+ return stray_resume_frame();
+}
+
+/* 9. A bpf_unwind_resume() in a subprogram a landing pad calls. The verifier's
+ * walk cannot tell it from a resume in the pad itself -- an exception is in
+ * flight either way -- so the rule is static: a resume sits in a pad body.
+ */
+static __used __naked __noinline void resume_in_callee(void)
+{
+ asm volatile (
+ "call bpf_unwind_resume;"
+ "exit;"
+ ::: __clobber_all);
+}
+
+static __used __naked __noinline __u64 pad_calls_resumer_frame(void)
+{
+ asm volatile (
+ "call bpf_preempt_disable;"
+ "r1 = 14;"
+"1:" "call bpf_throw;" /* cleanup region */
+"2:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ "call bpf_preempt_enable;"
+ "call resume_in_callee;" /* ...which resumes: refused */
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ ::: __clobber_all);
+}
+
+SEC("?syscall")
+__failure __msg("is not in an exception cleanup landing pad")
+int resume_in_pad_callee(void *ctx)
+{
+ return pad_calls_resumer_frame();
+}
+
+/* 10. A bpf_unwind_resume() in a program carrying no cleanup table, where
+ * that static rule does not run at all. do_check() refuses it on the state
+ * not unwinding, and has to: the JITs lower every one of these the same way.
+ */
+static __used __naked __noinline __u64 no_table_resume_frame(void)
+{
+ asm volatile (
+ "call bpf_unwind_resume;"
+ "r0 = 0;"
+ "exit;"
+ ::: __clobber_all);
+}
+
+SEC("?syscall")
+__failure __msg("reached without an exception in flight")
+int resume_without_table(void *ctx)
+{
+ return no_table_resume_frame();
+}
+
+/* 11. A landing pad that is itself a covered call site, so an exception out
+ * of it would have nowhere to go. Hand-written only: LLVM sinks a function's
+ * pads past every range it emits.
+ */
+static __used __naked __noinline __u64 nested_pad_frame(void)
+{
+ asm volatile (
+ "call bpf_preempt_disable;"
+"1:" "call inner_throw;" /* first cleanup region */
+"2:"
+ "call bpf_preempt_enable;"
+ "r0 = 0;"
+ "exit;"
+"3:" /* first pad, second region's call */
+ "call bpf_preempt_enable;"
+"4:"
+ "call bpf_unwind_resume;"
+ "exit;"
+"5:" /* second pad */
+ "call bpf_preempt_enable;"
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ CLEANUP_REC("3b", "4b", "5b")
+ ::: __clobber_all);
+}
+
+SEC("?syscall")
+__failure __msg("is inside the call-site range of")
+int nested_landing_pad(void *ctx)
+{
+ return nested_pad_frame();
+}
+
+/* 12. A tail call in a landing pad: it unwinds the prologue off the stack
+ * pointer, which in a pad is the walker's.
+ */
+struct {
+ __uint(type, BPF_MAP_TYPE_PROG_ARRAY);
+ __uint(max_entries, 1);
+ __uint(key_size, sizeof(__u32));
+ __uint(value_size, sizeof(__u32));
+} tc_map SEC(".maps");
+
+static __used __naked __noinline __u64 tail_call_pad_frame(void)
+{
+ asm volatile (
+ "r6 = r1;"
+"1:" "call inner_throw;" /* cleanup region */
+"2:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ "r1 = r6;"
+ "r2 = %[tc_map] ll;"
+ "r3 = 0;"
+ "call %[bpf_tail_call];"
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : __imm(bpf_tail_call), __imm_addr(tc_map)
+ : __clobber_all);
+}
+
+SEC("?syscall")
+__failure __msg("is in an exception cleanup landing pad")
+int tail_call_in_pad(void *ctx)
+{
+ return tail_call_pad_frame();
+}
+
+#if defined(__BPF_FEATURE_STACK_ARGUMENT)
+
+/* 13. A call that passes an argument on the stack, in a landing pad: the
+ * outgoing area the callee reads is not the one the caller wrote, the frame
+ * being the unwinding one and the stack pointer the walker's.
+ */
+static __used __noinline __u64 six_args(__u64 a, __u64 b, __u64 c, __u64 d,
+ __u64 e, __u64 f)
+{
+ return a + b + c + d + e + f;
+}
+
+static __used __naked __noinline __u64 stack_arg_pad_frame(void)
+{
+ asm volatile (
+ "call bpf_preempt_disable;"
+"1:" "call inner_throw;" /* cleanup region */
+"2:"
+ "call bpf_preempt_enable;"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ "call bpf_preempt_enable;"
+ "r1 = 1;"
+ "r2 = 2;"
+ "r3 = 3;"
+ "r4 = 4;"
+ "r5 = 5;"
+ "*(u64 *)(r11 - 8) = 6;" /* the sixth argument */
+ "call six_args;"
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ ::: __clobber_all);
+}
+
+SEC("?syscall")
+__failure __msg("on-stack call argument in an exception cleanup landing pad")
+int stack_arg_in_pad(void *ctx)
+{
+ return stack_arg_pad_frame();
+}
+
+/* 14. The same, reached the other way: a kfunc whose by-value argument runs
+ * past the five argument registers, where the JIT fills the outgoing area and
+ * the rule above has no store to catch. The C call gives the extern its BTF.
+ */
+static __used __noinline void __nofit_btf_anchor(void)
+{
+ struct prog_test_pair_arg s = {};
+
+ bpf_kfunc_call_test_pair_arg_nofit(1, 2, 3, 4, s);
+}
+
+static __used __naked __noinline __u64 kfunc_arg_pad_frame(void)
+{
+ asm volatile (
+ "call bpf_preempt_disable;"
+"1:" "call inner_throw;" /* cleanup region */
+"2:"
+ "call bpf_preempt_enable;"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ "call bpf_preempt_enable;"
+ "r1 = 1;"
+ "r2 = 2;"
+ "r3 = 3;"
+ "r4 = 4;"
+ "r5 = 5;"
+ "call bpf_kfunc_call_test_pair_arg_nofit;"
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ ::: __clobber_all);
+}
+
+SEC("?syscall")
+__failure __msg("on-stack call argument in an exception cleanup landing pad")
+int kfunc_stack_arg_in_pad(void *ctx)
+{
+ return kfunc_arg_pad_frame();
+}
+
+#endif /* __BPF_FEATURE_STACK_ARGUMENT */
+
+/* 15. A landing pad entered by ordinary control flow, arriving with none of
+ * what the walker sets up. Nothing static sees it -- the resume really is in
+ * a pad body -- so do_check() refuses it on the state not unwinding.
+ */
+static __used __naked __noinline __u64 jump_into_pad_frame(void)
+{
+ asm volatile (
+ "r1 = %[input] ll;"
+ "r6 = *(u64 *)(r1 + 0);"
+ "if r6 > 7 goto 4f;" /* an ordinary branch into the pad */
+"1:" "call inner_throw;" /* cleanup region */
+"2:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ "r7 = r0;"
+"4:" /* ... and its second instruction */
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : __imm_addr(input)
+ : __clobber_all);
+}
+
+SEC("?syscall")
+__failure __msg("reached without an exception in flight")
+int jump_into_pad(void *ctx)
+{
+ return jump_into_pad_frame();
+}
+
+#if defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64)
+
+/* 16. A landing pad that reaches an indirect jump, which cannot be told from
+ * a catch pad. SEC("socket") because a jump table entry is an offset from the
+ * program's section symbol, and "?syscall" is not a name assembly can use.
+ */
+static __used __naked __noinline void gotox_thrower(void)
+{
+ asm volatile (
+ "r1 = 15;"
+ "call bpf_throw;"
+ "exit;"
+ ::: __clobber_all);
+}
+
+SEC("socket")
+__failure __msg("reaches an indirect jump")
+__naked void gotox_in_pad(void)
+{
+ asm volatile (
+ ".pushsection .jumptables,\"\",@progbits;"
+"jt0_%=:"
+ ".quad l0_%= - socket;"
+ ".quad l1_%= - socket;"
+ ".size jt0_%=, 16;"
+ ".global jt0_%=;"
+ ".popsection;"
+
+"1:" "call gotox_thrower;" /* cleanup region */
+"2:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ "r1 = jt0_%= ll;"
+ "r1 += 8;"
+ "r2 = *(u64 *)(r1 + 0);"
+ /* gotox r2, as a raw insn: the mnemonic only reached the LLVM
+ * assembler in llvm 22, and BPF_RAW_INSN() needs <linux/bpf.h>, which
+ * vmlinux.h rules out.
+ */
+ ".8byte 0x20d;"
+"l0_%=:"
+ "call bpf_unwind_resume;"
+ "exit;"
+"l1_%=:"
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ ::: __clobber_all);
+}
+
+#endif /* x86 || arm64 */
+
+/* 17. A BPF_LD_[ABS|IND] in a landing pad. A failed load leaves the
+ * subprogram through the hidden "r0 = 0; exit" gen_ld_abs() patches in, and
+ * that exit is the epilogue, which unwinds a stack the pad does not own.
+ */
+static __used __naked __noinline __u64 ld_abs_pad_frame(void)
+{
+ asm volatile (
+ "r6 = r1;" /* the skb BPF_LD_ABS reads */
+"1:" "call inner_throw;" /* cleanup region */
+"2:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ /* r0 = *(u32 *)skb[0]. Spelled as a raw insn because BPF_LD_ABS()
+ * needs <linux/filter.h>, which this file cannot have -- vmlinux.h
+ * already defines the uapi enums.
+ */
+ ".8byte 0x20;"
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ ::: __clobber_all);
+}
+
+SEC("?tc")
+__failure __msg("is in an exception cleanup landing pad")
+__naked void ld_abs_in_pad(void)
+{
+ asm volatile (
+ "call ld_abs_pad_frame;"
+ "exit;"
+ ::: __clobber_all);
+}
+
+/* 18. A bpf_unwind_resume() in a subprogram a landing pad called, which has a
+ * pad of its own and reached it by ordinary control flow. Not case 9: an
+ * exception is in flight, but this is not the frame whose pad the walker ran.
+ */
+static __used __naked __noinline __u64 own_pad_callee(void)
+{
+ asm volatile (
+ "r1 = %[input] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+ "if r1 == 0 goto 3f;" /* an ordinary branch into its pad */
+"1:" "call bpf_preempt_disable;" /* cleanup region: nothing that throws */
+ "call bpf_preempt_enable;"
+"2:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* its landing pad */
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : __imm_addr(input)
+ : __clobber_all);
+}
+
+static __used __naked __noinline __u64 pad_calls_own_pad_frame(void)
+{
+ asm volatile (
+"1:" "call inner_throw;" /* cleanup region */
+"2:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad, which calls the above */
+ "call own_pad_callee;"
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ ::: __clobber_all);
+}
+
+SEC("?syscall")
+__failure __msg("is in frame 2, not frame 1 whose landing pad the exception entered")
+int resume_in_callee_own_pad(void *ctx)
+{
+ return pad_calls_own_pad_frame();
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/exceptions_cleanup_freplace.c b/tools/testing/selftests/bpf/progs/exceptions_cleanup_freplace.c
new file mode 100644
index 0000000000000..afb358fd3d405
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/exceptions_cleanup_freplace.c
@@ -0,0 +1,17 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "exceptions_cleanup.h"
+
+/* Without a 32-bit int in BTF, libbpf's dummy_ksym var gets type id 0. */
+int btf_int_anchor;
+
+SEC("freplace/fr_callee")
+__u64 new_fr_callee(__u64 x)
+{
+ bpf_throw(THROW_COOKIE);
+ return 0;
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/exceptions_cleanup_light.c b/tools/testing/selftests/bpf/progs/exceptions_cleanup_light.c
new file mode 100644
index 0000000000000..de7ae478d67f3
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/exceptions_cleanup_light.c
@@ -0,0 +1,39 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+#include "exceptions_cleanup.h"
+
+static __used __noinline void __kfunc_btf_anchor(void)
+{
+ bpf_throw(0);
+ bpf_preempt_disable();
+ bpf_preempt_enable();
+ bpf_unwind_resume();
+}
+
+__u64 pads_ran = 0;
+
+SEC("syscall")
+__naked int entry_light(void)
+{
+ asm volatile (
+ "call bpf_preempt_disable;"
+ "r1 = %[cookie];"
+"1:" "call bpf_throw;" /* cleanup region */
+"2:"
+ "exit;"
+"3:" /* landing pad */
+ "call bpf_preempt_enable;"
+ PAD_RAN("%[ran]")
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : [cookie]"i"(THROW_COOKIE), [ran]"i"(RAN_LIGHT),
+ __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/exceptions_cleanup_pad_freplace.c b/tools/testing/selftests/bpf/progs/exceptions_cleanup_pad_freplace.c
new file mode 100644
index 0000000000000..eabac6baabb70
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/exceptions_cleanup_pad_freplace.c
@@ -0,0 +1,17 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "exceptions_cleanup.h"
+
+/* Without a 32-bit int in BTF, libbpf's dummy_ksym var gets type id 0. */
+int btf_int_anchor;
+
+SEC("freplace/pad_callee")
+__u64 new_pad_callee(__u64 x)
+{
+ bpf_throw(INNER_COOKIE);
+ return 0;
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c b/tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c
new file mode 100644
index 0000000000000..493f8e0306528
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c
@@ -0,0 +1,984 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+#include "exceptions_cleanup.h"
+
+#define PAD_COUNT \
+ "r1 = %[pad_runs] ll;" \
+ "r2 = *(u64 *)(r1 + 0);" \
+ "r2 += 1;" \
+ "*(u64 *)(r1 + 0) = r2;"
+
+static __used __noinline void __kfunc_btf_anchor(void)
+{
+ bpf_throw(0);
+ bpf_rcu_read_lock();
+ bpf_rcu_read_unlock();
+ bpf_preempt_disable();
+ bpf_preempt_enable();
+ bpf_unwind_resume();
+}
+
+__u64 input = 0;
+__u64 outer_input = 0;
+__u64 magic = 0x5eed;
+__u64 pads_ran = 0;
+__u64 pad_runs = 0;
+
+/* 1. Everything a cleanup table leaves dead: a throw's continuation, the tail
+ * after a resume with an ld_imm64 and a branch in it, and the block only that
+ * continuation reaches.
+ */
+static __used __naked __noinline __u64 sweep_frame(void)
+{
+ asm volatile (
+ "r1 = %[input] ll;"
+ "r6 = *(u64 *)(r1 + 0);"
+ "call bpf_preempt_disable;"
+ "if r6 < 101 goto 6f;"
+ "r1 = %[cookie];"
+"1:" "call bpf_throw;" /* cleanup region */
+"2:"
+ "goto 3f;"
+"4:" /* landing pad */
+ "r7 = r0;"
+ "call bpf_preempt_enable;"
+ PAD_RAN("%[ran]")
+ "r1 = r7;"
+ "call bpf_unwind_resume;"
+ "r1 = %[pads_ran] ll;"
+ "r2 = *(u64 *)(r1 + 0);"
+ "if r2 == 0 goto 5f;"
+ "call bpf_preempt_enable;"
+ "r0 = 7;"
+ "exit;"
+"5:"
+ "r0 = 8;"
+ "exit;"
+"3:" /* dead: only the dead goto reaches it */
+ "r0 = 9;"
+ "exit;"
+"6:" /* live: the ordinary return */
+ "call bpf_preempt_enable;"
+ "r0 = 0;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "4b")
+ :
+ : [cookie]"i"(THROW_COOKIE), [ran]"i"(RAN_SWEEP),
+ __imm_addr(input), __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+SEC("syscall")
+int entry_sweep(void *ctx)
+{
+ return sweep_frame();
+}
+
+/* 2. A callee called from both a covered and an uncovered site: the pad
+ * belongs to the call site, not the callee. Either site can throw, and the RCU
+ * lock between them is what an unwind out of the uncovered one leaks.
+ */
+static __used __noinline __u64 shared_callee(__u64 x)
+{
+ if (x > 100)
+ bpf_throw(THROW_COOKIE);
+ return x + 1;
+}
+
+static __used __naked __noinline __u64 shared_frame(void)
+{
+ asm volatile (
+ "r1 = %[input] ll;"
+ "r6 = *(u64 *)(r1 + 0);"
+ "r1 = %[outer_input] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+ "call shared_callee;" /* uncovered: no pad for its unwind */
+ "call bpf_rcu_read_lock;"
+ "r1 = r6;"
+"1:" "call shared_callee;" /* cleanup region */
+"2:"
+ "r6 = r0;"
+ "call bpf_rcu_read_unlock;"
+ "r0 = r6;"
+ "exit;"
+"3:" /* landing pad */
+ "r7 = r0;"
+ "call bpf_rcu_read_unlock;"
+ PAD_RAN("%[ran]")
+ "r1 = r7;"
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : [ran]"i"(RAN_SHARED), __imm_addr(input), __imm_addr(outer_input),
+ __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+SEC("syscall")
+int entry_shared(void *ctx)
+{
+ return shared_frame();
+}
+
+/* 3. A landing pad that reads its frame's callee-saved registers: the callee
+ * overwrites r6-r9 before it throws, so the check passes only if the walker
+ * found the spill in the discarded callee's prologue.
+ */
+#define LOAD_MAGIC_REGS \
+ "r1 = %[magic] ll;" \
+ "r6 = *(u64 *)(r1 + 0);" \
+ "r7 = r6;" \
+ "r7 += 1;" \
+ "r8 = r6;" \
+ "r8 += 2;" \
+ "r9 = r6;" \
+ "r9 += 3;"
+
+/* Set @bit only if r6-r9 still hold what LOAD_MAGIC_REGS put there. */
+#define CHECK_MAGIC_REGS(bit) \
+ "r1 = %[magic] ll;" \
+ "r2 = *(u64 *)(r1 + 0);" \
+ "if r6 != r2 goto 9f;" \
+ "r2 += 1;" \
+ "if r7 != r2 goto 9f;" \
+ "r2 += 1;" \
+ "if r8 != r2 goto 9f;" \
+ "r2 += 1;" \
+ "if r9 != r2 goto 9f;" \
+ PAD_RAN(bit) \
+ "9:"
+
+static __used __naked __noinline __u64 regs_thrower(void)
+{
+ asm volatile (
+ /* Not this frame's to keep, and that is the point. */
+ "r6 = 0xdead;"
+ "r7 = 0xbeef;"
+ "r8 = 0xcafe;"
+ "r9 = 0xf00d;"
+ "r1 = %[cookie];"
+ "call bpf_throw;"
+ "r0 = 0;"
+ "exit;"
+ :
+ : [cookie]"i"(THROW_COOKIE)
+ : __clobber_all);
+}
+
+static __used __naked __noinline __u64 regs_frame(void)
+{
+ asm volatile (
+ LOAD_MAGIC_REGS
+ "call bpf_preempt_disable;"
+"1:" "call regs_thrower;" /* cleanup region */
+"2:"
+ "call bpf_preempt_enable;"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ "call bpf_preempt_enable;"
+ CHECK_MAGIC_REGS("%[ran]")
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : [cookie]"i"(THROW_COOKIE), [ran]"i"(RAN_REGS),
+ __imm_addr(magic), __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+SEC("syscall")
+int entry_regs(void *ctx)
+{
+ return regs_frame();
+}
+
+/* 4. The same, with a tail-call-reachable callee: its prologue pushes the tail
+ * call counter, which moves the spill the walker reads. The array is left
+ * empty; being reachable is the point.
+ */
+struct {
+ __uint(type, BPF_MAP_TYPE_PROG_ARRAY);
+ __uint(max_entries, 1);
+ __uint(key_size, sizeof(__u32));
+ __uint(value_size, sizeof(__u32));
+} jmp_table SEC(".maps");
+
+static __used __noinline __u64 tc_thrower(void *ctx)
+{
+ /* Never taken; its presence is what makes this frame, whose spill the
+ * walker reads, tail-call-reachable.
+ */
+ bpf_tail_call_static(ctx, &jmp_table, 0);
+ asm volatile (
+ "r6 = 0xdead;"
+ "r7 = 0xbeef;"
+ "r8 = 0xcafe;"
+ "r9 = 0xf00d;"
+ "r1 = %[cookie];"
+ "call bpf_throw;"
+ :
+ : [cookie]"i"(THROW_COOKIE)
+ : __clobber_all);
+ return 0;
+}
+
+/* The frame with the pad is the program itself, and __naked: r1 at entry is
+ * the only place to get a context for bpf_tail_call().
+ */
+SEC("syscall")
+__naked int entry_tail_call(void)
+{
+ asm volatile (
+ "*(u64 *)(r10 - 8) = r1;" /* the context, straight from entry */
+ LOAD_MAGIC_REGS
+ "r1 = %[input] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+ "if r1 < 101 goto 8f;"
+ "r1 = *(u64 *)(r10 - 8);"
+"1:" "call tc_thrower;" /* cleanup region */
+"2:"
+"8:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ CHECK_MAGIC_REGS("%[ran]")
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : [ran]"i"(RAN_TAIL_CALL), __imm_addr(input),
+ __imm_addr(magic), __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+/* 5. A landing pad in the main program's own frame. jit_subprogs() compiles it
+ * as func[0], but the walker finds the outer bpf_prog's ksym, so the table has
+ * to be handed over or the pad is never dispatched.
+ */
+SEC("syscall")
+__naked int entry_main_pad(void)
+{
+ asm volatile (
+ LOAD_MAGIC_REGS
+ "r1 = %[input] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+ "if r1 < 101 goto 8f;"
+"1:" "call regs_thrower;" /* cleanup region */
+"2:"
+"8:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ CHECK_MAGIC_REGS("%[ran]")
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : [ran]"i"(RAN_MAIN_PAD), __imm_addr(input),
+ __imm_addr(magic), __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+/* 6. A tail call that is really taken: the target is a program in its own
+ * right, so the walk ends there and this frame's pad does not run. The callee
+ * can also throw, which keeps the pad out of the sweep.
+ */
+struct {
+ __uint(type, BPF_MAP_TYPE_PROG_ARRAY);
+ __uint(max_entries, 1);
+ __uint(key_size, sizeof(__u32));
+ __uint(value_size, sizeof(__u32));
+} taken_table SEC(".maps");
+
+SEC("syscall")
+int tc_target(void *ctx)
+{
+ bpf_throw(THROW_COOKIE);
+ return 0;
+}
+
+static __used __noinline __u64 tc_taken_callee(void *ctx, __u64 x)
+{
+ /* Never true at run time; the verifier cannot know that, and its
+ * unwind out of here is what keeps the caller's pad alive.
+ */
+ if (x == 7)
+ bpf_throw(THROW_COOKIE);
+ bpf_tail_call_static(ctx, &taken_table, 0);
+ return 0;
+}
+
+SEC("syscall")
+__naked int entry_tail_taken(void)
+{
+ asm volatile (
+ "*(u64 *)(r10 - 8) = r1;" /* the context, straight from entry */
+ "r1 = %[input] ll;"
+ /* Unnarrowed, the way entry_freplace hands fr_callee its argument: a
+ * guard here would prune the callee's throw and let the sweep take the
+ * pad, leaving no record for a walk past the boundary to match.
+ */
+ "r2 = *(u64 *)(r1 + 0);"
+ "r1 = *(u64 *)(r10 - 8);"
+"1:" "call tc_taken_callee;" /* cleanup region */
+"2:"
+ "exit;" /* the cookie, delivered at tc_target */
+"3:" /* landing pad: must not run */
+ PAD_RAN("%[ran]")
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : [ran]"i"(RAN_TC_TAKEN), __imm_addr(input), __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+/* 7. An extension program over the callee of a covered call. The walk ends in
+ * the extension's frame, as it does for a tail call target, so the pad does
+ * not run; fr_callee() can also throw by itself.
+ */
+__noinline __u64 fr_callee(__u64 x)
+{
+ if (x == 7)
+ bpf_throw(THROW_COOKIE);
+ return x + 1;
+}
+
+SEC("syscall")
+__naked int entry_freplace(void)
+{
+ asm volatile (
+ "r1 = %[input] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+"1:" "call fr_callee;" /* cleanup region */
+"2:"
+ "exit;"
+"3:" /* landing pad */
+ PAD_RAN("%[ran]")
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : [ran]"i"(RAN_FREPLACE), __imm_addr(input), __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+/* 8. A throwing subprogram named by a BPF_PSEUDO_FUNC on a path never taken.
+ * Handing one to a helper is what is refused, not naming it.
+ */
+static __used __noinline int cb_thrower(__u32 idx, void *ctx)
+{
+ bpf_throw(THROW_COOKIE);
+ return 0;
+}
+
+static __used __noinline __u64 addr_taken_callee(__u64 x)
+{
+ if (x <= 100)
+ return x + 1;
+ bpf_throw(THROW_COOKIE);
+ return bpf_loop(1, cb_thrower, NULL, 0);
+}
+
+SEC("syscall")
+__naked int entry_addr_taken(void)
+{
+ asm volatile (
+ "r1 = %[input] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+"1:" "call addr_taken_callee;" /* cleanup region */
+"2:"
+ "exit;"
+"3:" /* landing pad */
+ PAD_RAN("%[ran]")
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : [ran]"i"(RAN_ADDR_TAKEN), __imm_addr(input), __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+/* 9. A record that covers bpf_throw() itself: the throwing frame is both the
+ * frame the record covers and the boundary, so the pad runs on the way to
+ * delivering the cookie.
+ */
+SEC("syscall")
+__naked int entry_no_subprog(void)
+{
+ asm volatile (
+ "r1 = %[input] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+ "if r1 < 101 goto 8f;"
+ "call bpf_preempt_disable;"
+ "r1 = %[cookie];"
+"1:" "call bpf_throw;" /* cleanup region */
+"2:"
+ "exit;"
+"8:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ "call bpf_preempt_enable;"
+ PAD_RAN("%[ran]")
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : [cookie]"i"(THROW_COOKIE), [ran]"i"(RAN_NO_SUBPROG),
+ __imm_addr(input), __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+/* 10. A landing pad that calls a subprogram an extension can replace. The
+ * load-time rule cannot see it coming, so what stops a nested exception is the
+ * walk ending in the extension's frame; pad_runs says the pad ran once.
+ */
+__noinline __u64 pad_callee(__u64 x)
+{
+ return x + 1;
+}
+
+static __used __noinline __u64 pc_thrower(__u64 x)
+{
+ if (x > 100)
+ bpf_throw(THROW_COOKIE);
+ return x + 1;
+}
+
+static __used __naked __noinline __u64 pad_calls_frame(void)
+{
+ asm volatile (
+ "r1 = %[input] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+"1:" "call pc_thrower;" /* cleanup region */
+"2:"
+ "exit;"
+"3:" /* landing pad */
+ "r6 = r0;"
+ "r1 = 1;"
+ "call pad_callee;" /* an extension can stand in here */
+ PAD_COUNT
+ PAD_RAN("%[ran]")
+ "r1 = r6;"
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : [ran]"i"(RAN_PAD_CALLS), __imm_addr(input), __imm_addr(pads_ran),
+ __imm_addr(pad_runs)
+ : __clobber_all);
+}
+
+SEC("syscall")
+int entry_pad_calls(void *ctx)
+{
+ return pad_calls_frame();
+}
+
+/* 11. A covered bpf_throw() the sweep leaves last, where the exception
+ * callback patchlet -- the one that does not keep the call it replaced in the
+ * last slot -- has to carry the marks with it; r6-r9 reports a lost one.
+ */
+SEC("syscall")
+__naked int entry_pad_first(void)
+{
+ asm volatile (
+ LOAD_MAGIC_REGS
+ "r1 = %[input] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+ "if r1 < 101 goto 7f;"
+ "goto 4f;"
+"3:" /* landing pad, ahead of the call */
+ CHECK_MAGIC_REGS("%[ran]")
+ "call bpf_unwind_resume;"
+ "exit;"
+"7:"
+ "r0 = 0;"
+ "exit;"
+"4:"
+ "r1 = %[cookie];"
+"1:" "call bpf_throw;" /* cleanup region */
+"2:"
+ "exit;" /* dead: swept, leaving the call last */
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : [cookie]"i"(THROW_COOKIE), [ran]"i"(RAN_PAD_FIRST),
+ __imm_addr(input), __imm_addr(magic), __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+/* 12. A cleanup region whose last instruction is a 16-byte one, so end - 1
+ * names the half that is not an instruction of its own. Well formed, and a
+ * rule against it would turn it away.
+ */
+static __used __naked __noinline __u64 wide_rec_frame(void)
+{
+ asm volatile (
+ "r1 = %[input] ll;"
+ "r6 = *(u64 *)(r1 + 0);"
+ "call bpf_rcu_read_lock;"
+ "r1 = r6;"
+"1:" "call shared_callee;" /* cleanup region begins */
+ "r1 = %[magic] ll;" /* ... and ends on this pair */
+"2:"
+ "r6 = r0;"
+ "call bpf_rcu_read_unlock;"
+ "r0 = r6;"
+ "exit;"
+"3:" /* landing pad */
+ "r7 = r0;"
+ "call bpf_rcu_read_unlock;"
+ PAD_RAN("%[ran]")
+ "r1 = r7;"
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : [ran]"i"(RAN_WIDE_REC), __imm_addr(input), __imm_addr(magic),
+ __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+SEC("syscall")
+int entry_wide_rec(void *ctx)
+{
+ return wide_rec_frame();
+}
+
+/* 13. A pad that reloads from and stores to its own frame -- the shape every
+ * compiler-generated pad has. A JIT that addresses the frame through the stack
+ * pointer, arm64, has to find a pad's frame another way.
+ */
+static __used __naked __noinline __u64 pad_stack_frame(void)
+{
+ asm volatile (
+ "r1 = %[magic] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+ "*(u64 *)(r10 - 8) = r1;" /* what the pad will want */
+ "r1 = %[input] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+"1:" "call pc_thrower;" /* cleanup region */
+"2:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ "r6 = r0;"
+ "r7 = *(u64 *)(r10 - 8);" /* reload it out of the frame */
+ "*(u64 *)(r10 - 16) = r7;" /* and write the frame while here */
+ "r1 = %[magic] ll;"
+ "r2 = *(u64 *)(r1 + 0);"
+ "if r7 != r2 goto 9f;"
+ "r3 = *(u64 *)(r10 - 16);"
+ "if r3 != r2 goto 9f;"
+ PAD_RAN("%[ran]")
+"9:"
+ "r1 = r6;"
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : [ran]"i"(RAN_PAD_STACK), __imm_addr(input), __imm_addr(magic),
+ __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+SEC("syscall")
+int entry_pad_stack(void *ctx)
+{
+ return pad_stack_frame();
+}
+
+/* 14. The same, with a frame in between that has no pad of its own, so the
+ * liveness query for an outer frame has more than one frame to walk.
+ */
+static __used __noinline __u64 deep_mid(__u64 x)
+{
+ return pc_thrower(x) + 1;
+}
+
+static __used __naked __noinline __u64 deep_frame(void)
+{
+ asm volatile (
+ "r1 = %[magic] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+ "*(u64 *)(r10 - 8) = r1;" /* nothing but the pad reads this */
+ "r1 = %[input] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+"1:" "call deep_mid;" /* cleanup region */
+"2:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ "r6 = r0;"
+ "r7 = *(u64 *)(r10 - 8);"
+ "r1 = %[magic] ll;"
+ "r2 = *(u64 *)(r1 + 0);"
+ "if r7 != r2 goto 9f;"
+ PAD_RAN("%[ran]")
+"9:"
+ "r1 = r6;"
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : [ran]"i"(RAN_DEEP_PAD), __imm_addr(input), __imm_addr(magic),
+ __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+SEC("syscall")
+int entry_deep_pad(void *ctx)
+{
+ return deep_frame();
+}
+
+/* 15. A region around a call the kernel knows cannot unwind: no call site is
+ * marked, nothing reaches the pad, and the sweep removes it. The program is
+ * otherwise ordinary and has to load.
+ */
+static __used __naked __noinline __u64 nounwind_rec_frame(void)
+{
+ asm volatile (
+ "call bpf_preempt_disable;"
+"1:" "call bpf_preempt_enable;" /* cleanup region: nounwind */
+"2:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad, never dispatched */
+ PAD_RAN("%[ran]")
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : [ran]"i"(RAN_NOUNWIND_REC), __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+SEC("syscall")
+int entry_nounwind_rec(void *ctx)
+{
+ return nounwind_rec_frame();
+}
+
+/* 16. The name a frontend gives the resume: LLVM emits _Unwind_Resume() and
+ * libbpf maps it onto bpf_unwind_resume(), which every other pad here calls.
+ */
+extern void _Unwind_Resume(void) __ksym;
+
+static __used __noinline void __resume_alias_btf_anchor(void)
+{
+ _Unwind_Resume();
+}
+
+static __used __naked __noinline __u64 resume_alias_frame(void)
+{
+ asm volatile (
+"1:" "call regs_thrower;" /* cleanup region */
+"2:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ PAD_RAN("%[ran]")
+ "call _Unwind_Resume;" /* the frontend's name for it */
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : [ran]"i"(RAN_RESUME_ALIAS), __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+SEC("syscall")
+int entry_resume_alias(void *ctx)
+{
+ return resume_alias_frame();
+}
+
+/* 17. A landing pad that calls a subprogram which tail calls. A tail call in
+ * the pad itself is refused, but the callee's prologue really did run, so its
+ * tail call releases exactly that and the target returns into the pad.
+ */
+struct {
+ __uint(type, BPF_MAP_TYPE_PROG_ARRAY);
+ __uint(max_entries, 1);
+ __uint(key_size, sizeof(__u32));
+ __uint(value_size, sizeof(__u32));
+} pad_tc_table SEC(".maps");
+
+__u64 pad_tc_target_ran = 0;
+
+SEC("syscall")
+int pad_tc_target(void *ctx)
+{
+ pad_tc_target_ran += 1;
+ return 0;
+}
+
+static __used __noinline __u64 pad_tc_callee(void *ctx)
+{
+ /* Taken only once the test has populated the array. */
+ bpf_tail_call_static(ctx, &pad_tc_table, 0);
+ return 0;
+}
+
+/* The frame with the pad is the program itself, and __naked: r1 at entry is
+ * the only place to get a context for bpf_tail_call(); the pad reloads it
+ * from the frame's stack.
+ */
+SEC("syscall")
+__naked int entry_pad_tail_call(void)
+{
+ asm volatile (
+ "*(u64 *)(r10 - 8) = r1;" /* the context, straight from entry */
+ LOAD_MAGIC_REGS
+ "r1 = %[input] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+ "if r1 < 101 goto 8f;"
+"1:" "call regs_thrower;" /* cleanup region */
+"2:"
+"8:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ PAD_COUNT
+ "r1 = *(u64 *)(r10 - 8);"
+ "call pad_tc_callee;"
+ /* Only if the frame survived the callee's tail call. */
+ CHECK_MAGIC_REGS("%[ran]")
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : [ran]"i"(RAN_PAD_TAIL_CALL), __imm_addr(input),
+ __imm_addr(magic), __imm_addr(pads_ran), __imm_addr(pad_runs)
+ : __clobber_all);
+}
+
+/* 18. The other target for that same tail call: a program with a table of its
+ * own, throwing inside the outer exception. The tail call made it a boundary,
+ * so the inner walk ends there and the outer pad, cookie and r6-r9 survive.
+ */
+__u64 tc_target_pad_runs = 0;
+__u64 inner_magic = 0xd00d;
+
+static __used __naked __noinline __u64 inner_thrower(void)
+{
+ asm volatile (
+ /* Not this frame's to keep, the same as regs_thrower. */
+ "r6 = 0xf00d;"
+ "r7 = 0xcafe;"
+ "r8 = 0xbeef;"
+ "r9 = 0xdead;"
+ "r1 = %[cookie];"
+ "call bpf_throw;"
+ "r0 = 0;"
+ "exit;"
+ :
+ : [cookie]"i"(INNER_COOKIE)
+ : __clobber_all);
+}
+
+SEC("syscall")
+__naked int pad_tc_throw_target(void)
+{
+ asm volatile (
+ /* Distinct from the outer pad's, so neither can stand in for it. */
+ "r1 = %[inner_magic] ll;"
+ "r6 = *(u64 *)(r1 + 0);"
+ "r7 = r6;"
+ "r7 += 1;"
+ "r8 = r6;"
+ "r8 += 2;"
+ "r9 = r6;"
+ "r9 += 3;"
+ "r1 = %[input] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+ "if r1 < 101 goto 8f;"
+"1:" "call inner_thrower;" /* cleanup region */
+"2:"
+"8:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ /* This frame's own r6-r9, not the outer pad's. */
+ "r1 = %[inner_magic] ll;"
+ "r2 = *(u64 *)(r1 + 0);"
+ "if r6 != r2 goto 9f;"
+ "r2 += 1;"
+ "if r7 != r2 goto 9f;"
+ "r2 += 1;"
+ "if r8 != r2 goto 9f;"
+ "r2 += 1;"
+ "if r9 != r2 goto 9f;"
+ "r1 = %[tc_target_pad_runs] ll;"
+ "r2 = *(u64 *)(r1 + 0);"
+ "r2 += 1;"
+ "*(u64 *)(r1 + 0) = r2;"
+"9:"
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : __imm_addr(input), __imm_addr(inner_magic),
+ __imm_addr(tc_target_pad_runs)
+ : __clobber_all);
+}
+
+/* 19. A landing pad whose first instruction is a nop. opt_remove_nops() runs
+ * long after the cleanup walk, so the record has to follow the pad to the
+ * instruction that takes its place rather than be dropped with the nop.
+ */
+static __used __naked __noinline __u64 nop_pad_frame(void)
+{
+ asm volatile (
+ "r1 = %[input] ll;"
+ "r6 = *(u64 *)(r1 + 0);"
+ "if r6 < 101 goto 6f;"
+ "r1 = %[cookie];"
+"1:" "call bpf_throw;" /* cleanup region */
+"2:"
+"6:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad: a nop, then its body */
+ "goto +0;"
+ PAD_RAN("%[ran]")
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : [cookie]"i"(THROW_COOKIE), [ran]"i"(RAN_NOP_PAD),
+ __imm_addr(input), __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+SEC("syscall")
+int entry_nop_pad(void *ctx)
+{
+ return nop_pad_frame();
+}
+
+/* 20. A landing pad that indexes its own frame's stack by a register the frame
+ * set before the throwing call. The offset is not a constant, so r6 has to be
+ * marked precise from inside the pad, back across the unwind edge.
+ */
+/* A thrower that touches none of r6-r9, so nothing in it can answer for the
+ * pad frame's r6 and the walk has to leave this frame to look.
+ */
+static __used __naked __noinline __u64 var_thrower(void)
+{
+ asm volatile (
+ "if r1 < 101 goto 1f;"
+ "r1 = %[cookie];"
+ "call bpf_throw;"
+"1:"
+ "r0 = 0;"
+ "exit;"
+ :
+ : [cookie]"i"(THROW_COOKIE)
+ : __clobber_all);
+}
+
+static __used __naked __noinline __u64 var_stack_frame(void)
+{
+ asm volatile (
+ "r1 = %[magic] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+ "*(u64 *)(r10 - 8) = r1;" /* the slot the pad will read... */
+ "*(u64 *)(r10 - 16) = r1;" /* ...whichever of the two it is */
+ "r1 = %[input] ll;"
+ "r6 = *(u64 *)(r1 + 0);"
+ "r6 &= 1;" /* an unknown slot number... */
+ "r6 <<= 3;" /* ...as an aligned byte offset */
+ "r1 = %[input] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+"1:" "call var_thrower;" /* cleanup region */
+"2:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ "r7 = r0;"
+ "r1 = r10;"
+ "r1 += r6;" /* variable offset into the frame */
+ "r2 = *(u64 *)(r1 - 16);"
+ "r3 = %[magic] ll;"
+ "r3 = *(u64 *)(r3 + 0);"
+ "if r2 != r3 goto 9f;"
+ PAD_RAN("%[ran]")
+"9:"
+ "r1 = r7;"
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : [ran]"i"(RAN_VAR_STACK), __imm_addr(input), __imm_addr(magic),
+ __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+SEC("syscall")
+int entry_var_stack(void *ctx)
+{
+ return var_stack_frame();
+}
+
+/* 21. The same walk, over a call to a global subprogram: the verifier enters
+ * no frame for one, so the exception arrives at the pad from the call site
+ * rather than from a throw some frames deeper.
+ */
+__noinline __u64 global_thrower(__u64 x)
+{
+ if (x > 100)
+ bpf_throw(THROW_COOKIE);
+ return x + 1;
+}
+
+static __used __naked __noinline __u64 global_pad_frame(void)
+{
+ asm volatile (
+ "r1 = %[magic] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+ "*(u64 *)(r10 - 8) = r1;"
+ "*(u64 *)(r10 - 16) = r1;"
+ "r1 = %[input] ll;"
+ "r6 = *(u64 *)(r1 + 0);"
+ "r6 &= 1;"
+ "r6 <<= 3;"
+ "r1 = %[input] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+"1:" "call global_thrower;" /* cleanup region */
+"2:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ "r7 = r0;"
+ "r1 = r10;"
+ "r1 += r6;"
+ "r2 = *(u64 *)(r1 - 16);"
+ "r3 = %[magic] ll;"
+ "r3 = *(u64 *)(r3 + 0);"
+ "if r2 != r3 goto 9f;"
+ PAD_RAN("%[ran]")
+"9:"
+ "r1 = r7;"
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : [ran]"i"(RAN_GLOBAL_PAD), __imm_addr(input), __imm_addr(magic),
+ __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+SEC("syscall")
+int entry_global_pad(void *ctx)
+{
+ return global_pad_frame();
+}
+
+char _license[] SEC("license") = "GPL";
|
| 3/2 | 2026/09/22 09:05 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 79192 output: 32 thoughts: 656 Thoughts: **BPF Exception Cleanup Landing Pad Support: Deep Dive** Okay, so this patch is introducing BPF exception cleanup landing pad support. Let's dig in. It's targeting amd64, which is good; the x86-64 code directly applies, and the core kernel/bpf code is relevant as well. The changes are all over the place: arm64 and x86 architectures in `arch/*`, core BPF logic in `kernel/bpf/*`, and a bunch of includes and tool updates. This is a pretty significant spread. Given that BPF is explicitly listed as a core subsystem, any changes here warrant careful attention. And absolutely, this code is reachable and fuzzable! We've got the BPF verifier and JIT changes which are key targets for syzkaller. Now, how does this all work? Well, it looks like it's hooked into `sys_bpf` and `BPF_PROG_LOAD`, specifically with the new fields in `union bpf_attr`: `attr.cleanup_info`, `attr.cleanup_info_rec_size`, and `attr.cleanup_info_cnt`. This means the verifier is going to get new checks, and we're seeing x86 JIT support for these cleanup pads, like `bpf_cleanup_force_spill`, and handling throw/unwind resume. Time to figure out what config options we need to enable this. Let's see what the Kconfig files or the code itself can tell us. |
| 4/2 | 2026/09/22 09:06 | tool | grepper |
0mArgs:{
"Expression": "UNWINDER_ORC",
"PathPrefix": "arch/x86/"
}Results: {
"Output": "arch/x86/Kconfig=58=config X86\n--\narch/x86/Kconfig-292-\tselect HAVE_REGS_AND_STACK_ACCESS_API\narch/x86/Kconfig:293:\tselect HAVE_RELIABLE_STACKTRACE\t\tif UNWINDER_ORC || STACK_VALIDATION\narch/x86/Kconfig-294-\tselect HAVE_FUNCTION_ARG_ACCESS_API\n--\narch/x86/Kconfig.debug=224=choice\narch/x86/Kconfig.debug-225-\tprompt \"Choose kernel unwinder\"\narch/x86/Kconfig.debug:226:\tdefault UNWINDER_ORC if X86_64\narch/x86/Kconfig.debug-227-\tdefault UNWINDER_FRAME_POINTER if X86_32\n--\narch/x86/Kconfig.debug-232-\narch/x86/Kconfig.debug:233:config UNWINDER_ORC\narch/x86/Kconfig.debug-234-\tbool \"ORC unwinder\"\n--\narch/x86/Makefile=327=endif\narch/x86/Makefile-328-\narch/x86/Makefile:329:ifdef CONFIG_UNWINDER_ORC\narch/x86/Makefile-330-orc_hash_h := arch/$(SRCARCH)/include/generated/asm/orc_hash.h\n--\narch/x86/include/asm/module.h=15=struct mod_arch_specific {\narch/x86/include/asm/module.h:16:#ifdef CONFIG_UNWINDER_ORC\narch/x86/include/asm/module.h-17-\tunsigned int num_orcs;\n--\narch/x86/include/asm/unwind.h=14=struct unwind_state {\n--\narch/x86/include/asm/unwind.h-22-\tbool error;\narch/x86/include/asm/unwind.h:23:#if defined(CONFIG_UNWINDER_ORC)\narch/x86/include/asm/unwind.h-24-\tbool signal, full_regs;\n--\narch/x86/include/asm/unwind.h=59=void unwind_start(struct unwind_state *state, struct task_struct *task,\n--\narch/x86/include/asm/unwind.h-66-\narch/x86/include/asm/unwind.h:67:#if defined(CONFIG_UNWINDER_ORC) || defined(CONFIG_UNWINDER_FRAME_POINTER)\narch/x86/include/asm/unwind.h-68-/*\n--\narch/x86/include/asm/unwind.h=71=static inline struct pt_regs *unwind_get_entry_regs(struct unwind_state *state,\n--\narch/x86/include/asm/unwind.h-77-\tif (partial) {\narch/x86/include/asm/unwind.h:78:#ifdef CONFIG_UNWINDER_ORC\narch/x86/include/asm/unwind.h-79-\t\t*partial = !state-\u003efull_regs;\n--\narch/x86/include/asm/unwind.h=88=static inline struct pt_regs *unwind_get_entry_regs(struct unwind_state *state,\n--\narch/x86/include/asm/unwind.h-94-\narch/x86/include/asm/unwind.h:95:#ifdef CONFIG_UNWINDER_ORC\narch/x86/include/asm/unwind.h-96-void unwind_init(void);\n--\narch/x86/kernel/Makefile=159=obj-$(CONFIG_X86_UMIP)\t\t\t+= umip.o\narch/x86/kernel/Makefile-160-\narch/x86/kernel/Makefile:161:obj-$(CONFIG_UNWINDER_ORC)\t\t+= unwind_orc.o\narch/x86/kernel/Makefile-162-obj-$(CONFIG_UNWINDER_FRAME_POINTER)\t+= unwind_frame.o\n--\narch/x86/net/Makefile=8=else\narch/x86/net/Makefile-9- obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o bpf_timed_may_goto.o\narch/x86/net/Makefile:10: ifdef CONFIG_UNWINDER_ORC\narch/x86/net/Makefile-11- obj-$(CONFIG_BPF_JIT) += bpf_cleanup_pad.o\n--\narch/x86/net/bpf_jit_comp.c=4545=bool bpf_jit_supports_exceptions(void)\n--\narch/x86/net/bpf_jit_comp.c-4550-\t */\narch/x86/net/bpf_jit_comp.c:4551:\treturn IS_ENABLED(CONFIG_UNWINDER_ORC);\narch/x86/net/bpf_jit_comp.c-4552-}\n--\narch/x86/net/bpf_jit_comp.c=4554=bool bpf_jit_supports_cleanup_pads(void)\narch/x86/net/bpf_jit_comp.c-4555-{\narch/x86/net/bpf_jit_comp.c:4556:\treturn IS_ENABLED(CONFIG_UNWINDER_ORC);\narch/x86/net/bpf_jit_comp.c-4557-}\n--\narch/x86/net/bpf_jit_comp.c=4564=void arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, u64 bp), void *cookie)\narch/x86/net/bpf_jit_comp.c-4565-{\narch/x86/net/bpf_jit_comp.c:4566:#if defined(CONFIG_UNWINDER_ORC)\narch/x86/net/bpf_jit_comp.c-4567-\tstruct unwind_state state;\n"
} |
| 5/2 | 2026/09/22 09:06 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 81792 output: 28 thoughts: 83 Thoughts: **My Kernel BPF Configuration Considerations** Okay, so I'm diving into this BPF JIT cleanup pad support on x86, and my initial thought is, "What are the dependencies?" It looks like `bpf_jit_supports_cleanup_pads()` needs `CONFIG_UNWINDER_ORC` and, obviously, `CONFIG_BPF_JIT`. That makes sense. I also remember that `CONFIG_BPF_SYSCALL` is a core requirement for pretty much everything BPF related – the syscall interface, the verifier, and all that good stuff. Time to double-check my assumptions and see if there are any gotchas in the build configuration. I'll need to go through the `kernel/bpf/Makefile` and the Kconfig files. This should give me a solid picture of the build requirements and dependencies for this feature. |
| 6/2 | 2026/09/22 09:06 | tool | grepper |
0mArgs:{
"Expression": "exception.o",
"PathPrefix": "kernel/bpf/"
}Results: {
"Output": "kernel/bpf/Makefile=13=obj-$(CONFIG_BPF_SYSCALL) += bpf_local_storage.o bpf_task_storage.o\nkernel/bpf/Makefile:14:obj-$(CONFIG_BPF_SYSCALL) += fixups.o cfg.o states.o backtrack.o check_btf.o exception.o\nkernel/bpf/Makefile-15-obj-${CONFIG_BPF_LSM}\t += bpf_inode_storage.o\n--\nkernel/bpf/btf.c=7148=static int btf_struct_walk(struct bpf_verifier_log *log, const struct btf *btf,\n--\nkernel/bpf/btf.c-7171-\t\t * walking unions yields untrusted pointers\nkernel/bpf/btf.c:7172:\t\t * with exception of __bpf_md_ptr and other\nkernel/bpf/btf.c-7173-\t\t * unions with a single member\n--\nkernel/bpf/check_btf.c=423=static int check_cleanup_info(struct bpf_verifier_env *env,\n--\nkernel/bpf/check_btf.c-520-\t * included: it would be both a pad and a call that unwinds to one, and\nkernel/bpf/check_btf.c:521:\t * an exception out of it would have nowhere to go.\nkernel/bpf/check_btf.c-522-\t */\n"
} |
| 7/2 | 2026/09/22 09:06 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 82467 output: 31 thoughts: 220 Thoughts: **Examining the BPF JIT Compilation for x86 Architecture** Okay, let's dive into this `arch/x86/net/Makefile` to understand how the BPF JIT compilation is handled on x86. I'm specifically interested in how it relates to our current work, particularly around the interaction with kfuncs and exception cleanup. The makefile conditional, based on `CONFIG_X86_32`, immediately tells me we're dealing with different object files depending on the architecture. If it's a 32-bit x86 (`CONFIG_X86_32=y`), we compile `bpf_jit_comp32.o`. But for 64-bit (x86_64, which is the more common case nowadays), the inclusion is more involved. Here, we're pulling in `bpf_jit_comp.o` and `bpf_timed_may_goto.o`. And then, conditionally, if `CONFIG_UNWINDER_ORC` is defined, `bpf_cleanup_pad.o` is also included. So, on AMD64, it's pretty clear that `CONFIG_BPF_SYSCALL`, `CONFIG_BPF_JIT` and `CONFIG_UNWINDER_ORC` are active. Now, I should make sure that there aren't any other related configurations such as `CONFIG_DEBUG_INFO_BTF` that affect things. Next step: I need to investigate if BTF (BTF debug information) is strictly *required* for either kfuncs or the exception cleanup. It could be that BTF is involved, but not essential for certain configurations, or perhaps it's a hard dependency for these features, based on the `UNWINDER_ORC`. I need to figure that out before proceeding further. |
| 8/2 | 2026/09/22 09:06 | tool | grepper |
0mArgs:{
"Expression": "exc_kfunc_list",
"PathPrefix": "kernel/bpf/"
}Results: {
"Output": "kernel/bpf/exception.c=16=enum exc_kfunc {\n--\nkernel/bpf/exception.c-19-\nkernel/bpf/exception.c:20:BTF_ID_LIST(exc_kfunc_list)\nkernel/bpf/exception.c-21-BTF_ID(func, bpf_unwind_resume)\n--\nkernel/bpf/exception.c=23=static bool insn_is_exc_kfunc(const struct bpf_insn *insn, int kf)\n--\nkernel/bpf/exception.c-25-\treturn bpf_pseudo_kfunc_call(insn) \u0026\u0026 insn-\u003eoff == 0 \u0026\u0026\nkernel/bpf/exception.c:26:\t insn-\u003eimm == exc_kfunc_list[kf];\nkernel/bpf/exception.c-27-}\n"
} |
| 9/2 | 2026/09/22 09:06 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 83341 output: 32 thoughts: 119 Thoughts: **My Initial Thoughts on `bpf_unwind_resume` and BTF Integration** Okay, so I've just spotted something interesting: `bpf_unwind_resume` is a kfunc that's been registered with `BTF_ID_FLAGS`. I remember seeing that construct, `BTF_ID_FLAGS(func, bpf_unwind_resume)`, in `kernel/bpf/helpers.c`. And that reminds me, there's also `BTF_ID_LIST(exc_kfunc_list)` over in `kernel/bpf/exception.c`. The key question now is whether the usage of `BTF_ID_FLAGS`, or indeed any kfuncs, has a dependency on `DEBUG_INFO_BTF` being enabled. It's a valid question, as the way BTF information is handled could be impacted. Time to dive into `kernel/bpf/helpers.c` and confirm my assumptions. I need to understand the connection more fully, and if this impacts the overall BTF requirements within the kernel. |
| 10/2 | 2026/09/22 09:06 | tool | grepper |
0mArgs:{
"Expression": "bpf_unwind_resume",
"PathPrefix": "kernel/bpf/"
}Results: {
"Output": "kernel/bpf/exception.c=16=enum exc_kfunc {\nkernel/bpf/exception.c:17:\tEXC_KF_bpf_unwind_resume,\nkernel/bpf/exception.c-18-};\n--\nkernel/bpf/exception.c=20=BTF_ID_LIST(exc_kfunc_list)\nkernel/bpf/exception.c:21:BTF_ID(func, bpf_unwind_resume)\nkernel/bpf/exception.c-22-\n--\nkernel/bpf/exception.c=30=enum cleanup_insn_kind {\n--\nkernel/bpf/exception.c-35-\tCLEANUP_INSN_THROW,\t/* call bpf_throw: nothing after it runs */\nkernel/bpf/exception.c:36:\tCLEANUP_INSN_RESUME,\t/* call bpf_unwind_resume: likewise */\nkernel/bpf/exception.c-37-\tCLEANUP_INSN_CALL,\t/* call to another subprog */\n--\nkernel/bpf/exception.c-41-/* What each instruction can reach, computed once by cleanup_reachability(). */\nkernel/bpf/exception.c:42:#define CLEANUP_REACH_RESUME\tBIT(0)\t/* a bpf_unwind_resume() call */\nkernel/bpf/exception.c-43-#define CLEANUP_REACH_EXIT\tBIT(1)\t/* a plain BPF_EXIT */\n--\nkernel/bpf/exception.c=110=static enum cleanup_insn_kind cleanup_classify(struct bpf_verifier_env *env, u32 i,\n--\nkernel/bpf/exception.c-140-\t\t}\nkernel/bpf/exception.c:141:\t\tif (insn_is_exc_kfunc(insn, EXC_KF_bpf_unwind_resume)) {\nkernel/bpf/exception.c-142-\t\t\t*next = -1;\n--\nkernel/bpf/exception.c=153=static void cleanup_mark_kfunc_sites(struct bpf_verifier_env *env)\n--\nkernel/bpf/exception.c-161-\t\t\tenv-\u003einsn_aux_data[i].cleanup_throw_site = true;\nkernel/bpf/exception.c:162:\t\telse if (insn_is_exc_kfunc(insn, EXC_KF_bpf_unwind_resume))\nkernel/bpf/exception.c-163-\t\t\tenv-\u003einsn_aux_data[i].cleanup_resume_site = true;\n--\nkernel/bpf/exception.c=271=static int cleanup_pad_is_catch(struct cleanup_ctx *ctx, u32 pad)\n--\nkernel/bpf/exception.c-287-\t\t\t(reach \u0026 CLEANUP_REACH_RESUME) ?\nkernel/bpf/exception.c:288:\t\t\t\"reaches both bpf_unwind_resume() and a plain exit\" :\nkernel/bpf/exception.c:289:\t\t\t\"reaches neither bpf_unwind_resume() nor an exit\");\nkernel/bpf/exception.c-290-\t\treturn -EINVAL;\n--\nkernel/bpf/exception.c=398=static int cleanup_check_resumes(struct cleanup_ctx *ctx)\n--\nkernel/bpf/exception.c-403-\tfor (i = 0; i \u003c env-\u003eprog-\u003elen; i++) {\nkernel/bpf/exception.c:404:\t\tif (!insn_is_exc_kfunc(\u0026env-\u003eprog-\u003einsnsi[i], EXC_KF_bpf_unwind_resume))\nkernel/bpf/exception.c-405-\t\t\tcontinue;\n--\nkernel/bpf/exception.c-408-\t\tverbose(env,\nkernel/bpf/exception.c:409:\t\t\t\"bpf_unwind_resume() at insn %u is not in an exception cleanup landing pad\\n\",\nkernel/bpf/exception.c-410-\t\t\ti);\n--\nkernel/bpf/exception.c=493=bool bpf_is_unwind_resume_kfunc(const struct bpf_insn *insn)\nkernel/bpf/exception.c-494-{\nkernel/bpf/exception.c:495:\treturn insn_is_exc_kfunc(insn, EXC_KF_bpf_unwind_resume);\nkernel/bpf/exception.c-496-}\n--\nkernel/bpf/helpers.c=3461=__bpf_kfunc void bpf_throw(u64 cookie)\n--\nkernel/bpf/helpers.c-3485- * unwind on once a frame's cleanups have run. To match kernel kfunc\nkernel/bpf/helpers.c:3486: * convention, the kernel calls it bpf_unwind_resume and libbpf maps the\nkernel/bpf/helpers.c-3487- * compiler's name onto it.\nkernel/bpf/helpers.c-3488- */\nkernel/bpf/helpers.c:3489:__bpf_kfunc void bpf_unwind_resume(void)\nkernel/bpf/helpers.c-3490-{\n--\nkernel/bpf/helpers.c=5028=BTF_ID_FLAGS(func, bpf_throw)\nkernel/bpf/helpers.c:5029:BTF_ID_FLAGS(func, bpf_unwind_resume)\nkernel/bpf/helpers.c-5030-#ifdef CONFIG_BPF_EVENTS\n--\nkernel/bpf/verifier.c=18552=static int process_cleanup_resume(struct bpf_verifier_env *env, int *insn_idx)\n--\nkernel/bpf/verifier.c-18559-\t\tverbose(env,\nkernel/bpf/verifier.c:18560:\t\t\t\"bpf_unwind_resume() at insn %d reached without an exception in flight\\n\",\nkernel/bpf/verifier.c-18561-\t\t\t*insn_idx);\n--\nkernel/bpf/verifier.c-18571-\t\tverbose(env,\nkernel/bpf/verifier.c:18572:\t\t\t\"bpf_unwind_resume() at insn %d is in frame %d, not frame %d whose landing pad the exception entered\\n\",\nkernel/bpf/verifier.c-18573-\t\t\t*insn_idx, state-\u003ecurframe, state-\u003eunwind_frameno);\n"
} |
| 11/2 | 2026/09/22 09:06 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 85360 output: 31 thoughts: 23 Thoughts: Here's my attempt at summarizing the thought process, tailored for an expert audience: **Investigating BTF_ID_LIST & DEBUG_INFO_BTF Dependencies** Okay, so I need to quickly refresh my understanding of how `BTF_ID_LIST` functions and if the `DEBUG_INFO_BTF` section is actually a hard requirement. I need to figure out if there is some nuance to consider. I'm probably looking at a specific use case where this interplay matters, perhaps related to symbol resolution in eBPF contexts. I need to clarify if `BTF_ID_LIST` can operate independently or if it inherently relies on the presence of the debug information that `DEBUG_INFO_BTF` provides, or if its use is conditional. This likely impacts things like relocation handling, type checking, or even program verification strategies. I should probably dive into the kernel source and the spec to make sure I haven't missed something fundamental in my prior analyses. Ultimately, the question is how this affects the integrity of my program in a given environment. |
| 12/2 | 2026/09/22 09:06 | tool | grepper |
0mArgs:{
"Expression": "DEBUG_INFO_BTF",
"PathPrefix": "kernel/bpf/"
}Results: {
"Output": "kernel/bpf/Makefile=43=ifeq ($(CONFIG_SYSFS),y)\nkernel/bpf/Makefile:44:obj-$(CONFIG_DEBUG_INFO_BTF) += sysfs_btf.o\nkernel/bpf/Makefile-45-endif\n--\nkernel/bpf/btf.c=6417=static struct btf *btf_parse_base(struct btf_verifier_env *env, const char *name,\n--\nkernel/bpf/btf.c-6422-\nkernel/bpf/btf.c:6423:\tif (!IS_ENABLED(CONFIG_DEBUG_INFO_BTF))\nkernel/bpf/btf.c-6424-\t\treturn ERR_PTR(-ENOENT);\n--\nkernel/bpf/btf.c=6503=__u32 btf_relocate_id(const struct btf *btf, __u32 id)\n--\nkernel/bpf/btf.c-6509-\nkernel/bpf/btf.c:6510:#ifdef CONFIG_DEBUG_INFO_BTF_MODULES\nkernel/bpf/btf.c-6511-\nkernel/bpf/btf.c=6512=static struct btf *btf_parse_module(const char *module_name, const void *data,\n--\nkernel/bpf/btf.c-6607-\nkernel/bpf/btf.c:6608:#endif /* CONFIG_DEBUG_INFO_BTF_MODULES */\nkernel/bpf/btf.c-6609-\n--\nkernel/bpf/btf.c=8588=enum {\n--\nkernel/bpf/btf.c-8591-\nkernel/bpf/btf.c:8592:#ifdef CONFIG_DEBUG_INFO_BTF_MODULES\nkernel/bpf/btf.c-8593-struct btf_module {\n--\nkernel/bpf/btf.c=8730=fs_initcall(btf_module_init);\nkernel/bpf/btf.c:8731:#endif /* CONFIG_DEBUG_INFO_BTF_MODULES */\nkernel/bpf/btf.c-8732-\nkernel/bpf/btf.c=8733=struct module *btf_try_get_module(const struct btf *btf)\n--\nkernel/bpf/btf.c-8735-\tstruct module *res = NULL;\nkernel/bpf/btf.c:8736:#ifdef CONFIG_DEBUG_INFO_BTF_MODULES\nkernel/bpf/btf.c-8737-\tstruct btf_module *btf_mod, *tmp;\n--\nkernel/bpf/btf.c=8763=struct btf *btf_get_module_btf(const struct module *module)\nkernel/bpf/btf.c-8764-{\nkernel/bpf/btf.c:8765:#ifdef CONFIG_DEBUG_INFO_BTF_MODULES\nkernel/bpf/btf.c-8766-\tstruct btf_module *btf_mod, *tmp;\n--\nkernel/bpf/btf.c-8776-\nkernel/bpf/btf.c:8777:#ifdef CONFIG_DEBUG_INFO_BTF_MODULES\nkernel/bpf/btf.c-8778-\tmutex_lock(\u0026btf_module_mutex);\n--\nkernel/bpf/btf.c=8793=static int check_btf_kconfigs(const struct module *module, const char *feature)\nkernel/bpf/btf.c-8794-{\nkernel/bpf/btf.c:8795:\tif (!module \u0026\u0026 IS_ENABLED(CONFIG_DEBUG_INFO_BTF)) {\nkernel/bpf/btf.c-8796-\t\tpr_err(\"missing vmlinux BTF, cannot register %s\\n\", feature);\n--\nkernel/bpf/btf.c-8798-\t}\nkernel/bpf/btf.c:8799:\tif (module \u0026\u0026 IS_ENABLED(CONFIG_DEBUG_INFO_BTF_MODULES))\nkernel/bpf/btf.c-8800-\t\tpr_warn(\"missing module BTF, cannot register %s\\n\", feature);\n--\nkernel/bpf/btf.c=8941=static int btf_check_kfunc_name(struct btf *btf, const char *func_name, u32 kind)\nkernel/bpf/btf.c-8942-{\nkernel/bpf/btf.c:8943:#ifdef CONFIG_DEBUG_INFO_BTF_MODULES\nkernel/bpf/btf.c-8944-\tstruct btf_module *btf_mod, *tmp;\n--\nkernel/bpf/btf.c-8957-\nkernel/bpf/btf.c:8958:#ifdef CONFIG_DEBUG_INFO_BTF_MODULES\nkernel/bpf/btf.c-8959-\tguard(mutex)(\u0026btf_module_mutex);\n--\nkernel/bpf/btf.c=9606=static struct bpf_cand_cache *populate_cand_cache(struct bpf_cand_cache *cands,\n--\nkernel/bpf/btf.c-9633-\nkernel/bpf/btf.c:9634:#ifdef CONFIG_DEBUG_INFO_BTF_MODULES\nkernel/bpf/btf.c-9635-static void __purge_cand_cache(struct btf *btf, struct bpf_cand_cache **cache,\n--\nkernel/bpf/verifier.c=2853=int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset)\n--\nkernel/bpf/verifier.c-2869-\t\tif (!btf_vmlinux) {\nkernel/bpf/verifier.c:2870:\t\t\tverbose(env, \"calling kernel function is not supported without CONFIG_DEBUG_INFO_BTF\\n\");\nkernel/bpf/verifier.c-2871-\t\t\treturn -ENOTSUPP;\n--\nkernel/bpf/verifier.c=6247=static int check_ptr_to_map_access(struct bpf_verifier_env *env,\n--\nkernel/bpf/verifier.c-6261-\tif (!btf_vmlinux) {\nkernel/bpf/verifier.c:6262:\t\tverbose(env, \"map_ptr access not supported without CONFIG_DEBUG_INFO_BTF\\n\");\nkernel/bpf/verifier.c-6263-\t\treturn -ENOTSUPP;\n--\nkernel/bpf/verifier.c=19215=static int check_pseudo_btf_id(struct bpf_verifier_env *env,\n--\nkernel/bpf/verifier.c-19231-\t\tif (!btf_vmlinux) {\nkernel/bpf/verifier.c:19232:\t\t\tverbose(env, \"kernel is missing BTF, make sure CONFIG_DEBUG_INFO_BTF=y is specified in Kconfig.\\n\");\nkernel/bpf/verifier.c-19233-\t\t\treturn -EINVAL;\n--\nkernel/bpf/verifier.c=21126=struct btf *bpf_get_btf_vmlinux(void)\n--\nkernel/bpf/verifier.c-21130-\nkernel/bpf/verifier.c:21131:\tif (!btf \u0026\u0026 IS_ENABLED(CONFIG_DEBUG_INFO_BTF)) {\nkernel/bpf/verifier.c-21132-\t\tmutex_lock(\u0026btf_vmlinux_lock);\n"
} |
| 13/2 | 2026/09/22 09:06 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 87591 output: 28 thoughts: 2810 Thoughts: **BPF Exception and Cleanup Pad Fuzzing Strategy** Okay, so I'm looking at this patch and immediately, I see a prime opportunity for fuzzing within the BPF subsystem of the Linux kernel. The presence of kfuncs like `bpf_throw` and `bpf_unwind_resume` screams `DEBUG_INFO_BTF`, and the code references line 2870 which explicitly states that these features depend on it. That's a key configuration item right there. I also see that `BPF_SYSCALL`, `BPF_JIT`, and `UNWINDER_ORC` are probably important too. First, I need to confirm whether this code is even reachable via fuzzing. Yes, the changes are deeply integrated within the BPF verifier, runtime, and JIT. Specifically, I'm seeing changes in: * BPF verifier (cleanup info handling, CFG analysis, and error handling). * BPF runtime and JIT (especially `bpf_throw`, `bpf_unwind_resume`, and cleanup pad execution). The fact that userspace can directly interact with this logic via `sys_bpf` and `BPF_PROG_LOAD`, supplying `cleanup_info`, `cleanup_info_cnt`, and `cleanup_info_rec_size`, is fantastic. This is textbook BPF fuzzing material. Syzkaller can be wielded to generate programs that specifically target exceptions, throws, cleanup pads, and unwinding. Also, the presence of assertions (`WARN_ON_ONCE`, `verifier_bug_if`) and complex algorithms in the verifier are very enticing. This is textbook high-value BPF fuzzing material. Verifier and JIT bugs are prime targets for syzkaller. Therefore, worthFuzzing = true. Next, I need to establish the necessary configurations. It's pretty straightforward, but I have to be precise: 1. `BPF_SYSCALL`: Obviously needs to be enabled for using the BPF syscall itself. 2. `BPF_JIT`: This is non-negotiable, required for the JIT and specifically for exception landing pads (`bpf_jit_supports_cleanup_pads()`). 3. `DEBUG_INFO_BTF`: Necessary to support kfuncs (`bpf_throw`, `bpf_unwind_resume`). 4. `UNWINDER_ORC`: Required on x86 for cleanup pad support. This is also needed for `bpf_jit_supports_cleanup_pads()` and `bpf_jit_supports_exceptions()` to be true. It’s also important since it compiles `bpf_cleanup_pad.o`. I'm double-checking by inspecting `arch/x86/net/Makefile` and `bpf_jit_comp.c` to confirm these dependencies are correctly identified. Without `UNWINDER_ORC`, the code path that uses landing pads will fail with -EOPNOTSUPP, because the JIT will not support cleanup pads and exceptions. JIT has to be enabled (jit_requested), which requires `CONFIG_BPF_JIT`. Now, I need to focus on the key symbols, guided by the principle of avoiding generic code and instead targeting the specific feature implementation and API entry points. I reviewed the guidelines and then compiled a list of potential symbol focuses from the patched files: From `kernel/bpf/exception.c`: * `bpf_prepare_cleanup_exceptions` * `bpf_check_cleanup_exceptions` * `bpf_cleanup_pad_of_call` * `bpf_cleanup_fill_native_ranges` * `bpf_cleanup_pad_for_ip` From `kernel/bpf/check_btf.c`: * `check_cleanup_info` (or its caller `bpf_check_btf_info`) From `kernel/bpf/fixups.c`: * `bpf_cleanup_attach_main_prog` From `kernel/bpf/helpers.c`: * `bpf_throw` * `bpf_unwind_resume` From `arch/x86/net/`: * `arch_bpf_run_cleanup_pad` I verified that these functions are actually defined as symbols in vmlinux, specifically by checking via grepping. They *are* all specific entry points, so they are suitable targets for coverage filters. These are only hit by BPF programs using exceptions or throws, never by unrelated tests. The syscall is `bpf` (command `BPF_PROG_LOAD |
| 14/2 | 2026/09/22 09:06 | tool | grepper |
0mError:no matches Args: {
"Expression": "config BPF_JIT",
"PathPrefix": "init/"
}Results: {
"error": "no matches"
} |
| 15/2 | 2026/09/22 09:06 | llm | patch-evaluator |
1mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 91468 output: 22 thoughts: 20 Thoughts: Here's my attempt at summarizing the thought process, tailored for an expert audience: **Investigating BPF JIT Configuration** Okay, so I need to get a handle on the configuration options for the BPF Just-In-Time compiler. My immediate thought is to track down the `config BPF_JIT` symbol within the kernel's configuration files. This is the place to start, as I know Kconfig files are the central repository for kernel configuration variables. I need to figure out where this is defined and potentially see how it relates to other BPF-related options. This will help me understand the implications of different JIT settings and how they might interact with other subsystems. I'll need to do a full search through the Kconfig files across the source tree to pinpoint this, and then begin to follow the dependencies. |
| 16/2 | 2026/09/22 09:07 | tool | grepper |
0mArgs:{
"Expression": "config BPF_JIT\\b"
}Results: {
"Output": "kernel/bpf/Kconfig=31=config BPF_SYSCALL\n--\nkernel/bpf/Kconfig-45-\nkernel/bpf/Kconfig:46:config BPF_JIT\nkernel/bpf/Kconfig-47-\tbool \"Enable BPF Just In Time compiler\"\n"
} |
| 17/2 | 2026/09/22 09:07 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 91770 output: 34 thoughts: 19 Thoughts: Here's my summary, as though I'm reading and thinking about the Kconfig file: **Navigating the BPF Kernel Configuration** Okay, time to dive into the `kernel/bpf/Kconfig` file. I'm going to jump straight to around line 46, because my experience tells me that's a likely spot for the core options related to BPF functionality within the kernel. I'm expecting to see the main switches: enabling BPF itself, maybe things about JIT compilation, various features, or security settings. I need to get a clear picture of what the fundamental configuration choices are. This is where the real architecture begins. Time to understand the building blocks for the rest of the BPF code. |
| 18/2 | 2026/09/22 09:07 | tool | read-file |
0mArgs:{
"File": "kernel/bpf/Kconfig",
"FirstLine": 30,
"LineCount": 40
}Results: {
"Contents": " 30:\t\n 31:\tconfig BPF_SYSCALL\n 32:\t\tbool \"Enable bpf() system call\"\n 33:\t\tselect BPF\n 34:\t\tselect IRQ_WORK\n 35:\t\tselect NEED_TASKS_RCU\n 36:\t\tselect TASKS_TRACE_RCU\n 37:\t\tselect BINARY_PRINTF\n 38:\t\tselect NET_SOCK_MSG if NET\n 39:\t\tselect NET_XGRESS if NET\n 40:\t\tselect PAGE_POOL if NET\n 41:\t\tdefault n\n 42:\t\thelp\n 43:\t\t Enable the bpf() system call that allows to manipulate BPF programs\n 44:\t\t and maps via file descriptors.\n 45:\t\n 46:\tconfig BPF_JIT\n 47:\t\tbool \"Enable BPF Just In Time compiler\"\n 48:\t\tdepends on BPF\n 49:\t\tdepends on HAVE_CBPF_JIT || HAVE_EBPF_JIT\n 50:\t\tselect EXECMEM\n 51:\t\thelp\n 52:\t\t BPF programs are normally handled by a BPF interpreter. This option\n 53:\t\t allows the kernel to generate native code when a program is loaded\n 54:\t\t into the kernel. This will significantly speed-up processing of BPF\n 55:\t\t programs.\n 56:\t\n 57:\t\t Note, an admin should enable this feature changing:\n 58:\t\t /proc/sys/net/core/bpf_jit_enable\n 59:\t\t /proc/sys/net/core/bpf_jit_harden (optional)\n 60:\t\t /proc/sys/net/core/bpf_jit_kallsyms (optional)\n 61:\t\n 62:\tconfig BPF_JIT_ALWAYS_ON\n 63:\t\tbool \"Permanently enable BPF JIT and remove BPF interpreter\"\n 64:\t\tdepends on BPF_SYSCALL \u0026\u0026 HAVE_EBPF_JIT \u0026\u0026 BPF_JIT\n 65:\t\thelp\n 66:\t\t Enables BPF JIT and removes BPF interpreter to avoid speculative\n 67:\t\t execution of BPF instructions by the interpreter.\n 68:\t\n 69:\t\t When CONFIG_BPF_JIT_ALWAYS_ON is enabled, /proc/sys/net/core/bpf_jit_enable\n"
} |
| 19/2 | 2026/09/22 09:07 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 92587 output: 298 thoughts: 479 Thoughts: **Triage Review: BPF Exception Handling** Okay, let's see. I'm looking at BPF exception handling, and my initial thought is, what are the relevant configurations? I've got `BPF_JIT`, `BPF_SYSCALL`, `DEBUG_INFO_BTF`, and `UNWINDER_ORC` flagged. Let me quickly double-check that `CONFIG_UNWINDER_ORC` in `arch/x86/Kconfig.debug`; Yep, it depends on the standard `X86_64`. `DEBUG_INFO_BTF` depends on `DEBUG_INFO`. Confirmed. So, the key configs are: * `BPF_SYSCALL` * `BPF_JIT` * `DEBUG_INFO_BTF` * `UNWINDER_ORC` Now, let's revisit those `FocusSymbols`. I want to make sure these are distinct, focused on these changes, and not overly general. I've got a list here: * `bpf_prepare_cleanup_exceptions` * `bpf_check_cleanup_exceptions` * `bpf_cleanup_attach_main_prog` * `bpf_cleanup_fill_native_ranges` * `bpf_cleanup_pad_for_ip` * `bpf_unwind_resume` * `arch_bpf_run_cleanup_pad` * `bpf_throw` Okay, let's break these down: * `bpf_prepare_cleanup_exceptions`: This is the preparation step for exception metadata, part of verification. * `bpf_check_cleanup_exceptions`: Checks reachability. This verifies landing pads. * `bpf_cleanup_attach_main_prog`: Attaches the cleanup information to the main/sub BPF program. * `bpf_cleanup_fill_native_ranges`: Fills the native addresses after JIT. * `bpf_cleanup_pad_for_ip`: Looks up cleanup pad for a given IP during unwinding. * `bpf_unwind_resume`: A kfunc that's called by cleanup landing pads. * `arch_bpf_run_cleanup_pad`: The architecture-specific entry for the landing pad. * `bpf_throw`: A kfunc triggering exception unwinding, thus exercising the pads. That all seems perfectly clear. The symbols are well-defined and tied directly to the exception handling logic. They are all distinct, focused, and specific to the changes. This all aligns with the triage guidelines. Alright, with that, I'm confident in the review. Let's call `set-results`. |
| 20/2 | 2026/09/22 09:08 | tool | set-results |
0mArgs:{
"EnableConfigs": [
"BPF_SYSCALL",
"BPF_JIT",
"UNWINDER_ORC",
"DEBUG_INFO_BTF"
],
"FocusSymbols": [
"bpf_prepare_cleanup_exceptions",
"bpf_check_cleanup_exceptions",
"bpf_cleanup_attach_main_prog",
"bpf_cleanup_fill_native_ranges",
"bpf_cleanup_pad_for_ip",
"bpf_unwind_resume",
"arch_bpf_run_cleanup_pad",
"bpf_throw"
],
"Reasoning": "The patch series implements BPF exception cleanup landing pad support across core BPF (verifier CFG, reachability, liveness, backtracking, dead-code elimination) and architecture-specific JIT/unwind dispatch on x86-64 and arm64. This adds new user-facing fields in union bpf_attr for BPF_PROG_LOAD, verifier checks, kfuncs (bpf_unwind_resume), runtime stack unwinding, and assembly dispatchers. The changes touch core subsystems (kernel/bpf and arch/x86/net) and are fully reachable in virtualized environments via the bpf() syscall. Therefore, this warrants fuzzing.",
"WorthFuzzing": true
}Results: {
"EnableConfigs": [
"BPF_SYSCALL",
"BPF_JIT",
"UNWINDER_ORC",
"DEBUG_INFO_BTF"
],
"FocusSymbols": [
"bpf_prepare_cleanup_exceptions",
"bpf_check_cleanup_exceptions",
"bpf_cleanup_attach_main_prog",
"bpf_cleanup_fill_native_ranges",
"bpf_cleanup_pad_for_ip",
"bpf_unwind_resume",
"arch_bpf_run_cleanup_pad",
"bpf_throw"
],
"Reasoning": "The patch series implements BPF exception cleanup landing pad support across core BPF (verifier CFG, reachability, liveness, backtracking, dead-code elimination) and architecture-specific JIT/unwind dispatch on x86-64 and arm64. This adds new user-facing fields in union bpf_attr for BPF_PROG_LOAD, verifier checks, kfuncs (bpf_unwind_resume), runtime stack unwinding, and assembly dispatchers. The changes touch core subsystems (kernel/bpf and arch/x86/net) and are fully reachable in virtualized environments via the bpf() syscall. Therefore, this warrants fuzzing.",
"WorthFuzzing": true
} |
| 21/1 | 2026/09/22 09:08 | action | If |
5mArgs:{
"WorthFuzzing": true
}Results: null |
| 22/2 | 2026/09/22 09:08 | agent | kmsan-evaluator |
5mModel:core Args: null Results: {
"KMSANReasoning": "The patch series implements support for BPF cleanup landing pads and exception unwinding across x86-64 and arm64 architectures, adding verifier checks, CFG analysis, JIT spills/epilogues, and the bpf_unwind_resume kfunc.\n\nA dedicated KMSAN session is not required based on the following analysis:\n1. Heap and Stack Memory Initialization:\n - All newly allocated kernel data structures (such as `struct bpf_exception_info`, `struct bpf_cleanup_range`, `struct bpf_cleanup_info`, and DFS scratch arrays in `cleanup_ctx`) are allocated using zeroing allocators (`kzalloc_obj`, `kvcalloc`, `kvzalloc`, `bitmap_zalloc`).\n - Verifier state tracking structures (`struct bpf_verifier_state`, `struct bpf_insn_aux_data`) are zero-initialized via `kzalloc_obj` and `__vmalloc(..., __GFP_ZERO)`. Added fields (`unwinding`, `unwind_frameno`, `unwind_frames`) are explicitly initialized and reset.\n - The stack context used during unwinding (`struct bpf_throw_ctx ctx = {}`) is zero-initialized on the stack.\n\n2. Kernel-to-User Space Copies / Info Leaks:\n - The only user copy introduced is writing `min_size` (a 4-byte scalar `u32`) to user space via `copy_to_bpfptr_offset` on `-E2BIG` in `check_cleanup_info()`. It is fully initialized with `min_t(u32, krec_size, urec_size)`.\n - No other kernel structures, struct padding, or uninitialized buffers are copied to user space.\n\n3. JIT and Runtime Register Spills:\n - On x86-64, JIT prologue and throw-site register spills write all 5 callee-saved / context registers before `arch_bpf_run_cleanup_pad()` reads them.\n - The BPF verifier enforces that landing pad entry clears caller-saved registers (`clear_caller_saved_regs`), initializes `BPF_REG_0` as known, and prevents reading uninitialized registers or stack slots within landing pads.\n\nPotential failure modes in this patch involve CFG exploration, jump history backtracking, verifier prune points, or stack offset calculations—bugs that fall under control-flow logic, out-of-bounds access, or memory corruption detectable by KASAN and standard kernel assertions rather than KMSAN uninitialized memory detectors.",
"NeedsKMSAN": false
}Instruction: You are an expert Linux kernel security engineer specializing in kernel memory
error detectors (KASAN and KMSAN). Your job is to review the provided patch series and
determine if the code changes justify spawning a dedicated KMSAN (KernelMemorySanitizer)
fuzzing session in addition to standard KASAN fuzzing.
CRITICAL DISTINCTION BETWEEN KASAN AND KMSAN:
- Standard KASAN kernel builds (upstream-apparmor-kasan.config) already enable
a comprehensive suite of debugging tools and sanitizers, including KASAN
(out-of-bounds accesses, use-after-free, double free, invalid free), LOCKDEP
(locking bugs and deadlocks), UB-sanitizers, and memory corruption checks.
- KMSAN (KernelMemorySanitizer) detects reads of UNINITIALIZED memory (stack, heap,
or page allocations) and kernel-to-user memory info-leaks.
Rule: THERE IS NO SENSE IN RUNNING A KMSAN SESSION IF A BUG CAN BE CAUGHT BY KASAN,
LOCKDEP, OR OTHER STANDARD BUG DETECTORS.
A dedicated KMSAN fuzzing session incurs significant resource costs. You must ONLY
set NeedsKMSAN=true if the code changes introduce or expose UNINITIALIZED MEMORY risks
that are detected ONLY by KMSAN.
Look holistically at the patch series and surrounding code. Even if no direct
uninitialized field accesses or new buffer allocations are added in the diff itself,
a patch may alter control flow, bounds checking, or data length calculations in ways
that change how the rest of the code operates on existing buffers (e.g. allowing
uninitialized stack/heap memory to be read, copied to user space, or used in control
flow). Do not hesitate to use your code access tools to inspect the surrounding code,
called functions, and callers.
Set NeedsKMSAN=true ONLY IF the patch introduces or modifies:
1. Kernel structures sent to user space (via copy_to_user, put_user, netlink skb
attributes, ioctl output arguments, socket options, or BPF buffers) where fields
or structure padding might not be fully initialized/zeroed.
2. Conditional logic or branching that depends on potentially uninitialized variables
or struct fields.
3. Allocation or initialization of complex data structures where uninitialized fields
could be read later in reachable code paths.
4. Bounds checks, lengths, or logic in a way that allows surrounding code to access
uninitialized bytes of existing buffers.
Set NeedsKMSAN=false IF:
- The code changes primarily risk out-of-bounds access, array overflows, NULL pointer
dereferences, locking deadlocks, or use-after-free bugs (these are already caught
by KASAN, LOCKDEP, or standard bug detectors).
- All stack/heap structures touched or introduced by the patch are fully zeroed
or initialized (e.g. using = {0}, memset, kzalloc) before being read or copied.
- The patch does not introduce any risk of uninitialized memory usage or info-leaks.
Use your code access tools to inspect the surrounding code if necessary, then provide
detailed KMSANReasoning contrasting KASAN vs KMSAN applicability for this patch.
Prefer calling several tools at the same time to save round-trips.
Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.
Prompt: Target architecture: amd64
For your convenience, here is the diff of the changes:
commit 68c10631eef7b88304d24e3ffb2d009f8db416fc
Author: syz-cluster <triage@syzkaller.com>
Date: Tue Sep 22 09:05:55 2026 +0000
syz-cluster: applied patch under review
diff --git a/arch/arm64/net/Makefile b/arch/arm64/net/Makefile
index 3ae382bfca879..ebec2a44a52b6 100644
--- a/arch/arm64/net/Makefile
+++ b/arch/arm64/net/Makefile
@@ -2,4 +2,4 @@
#
# ARM64 networking code
#
-obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o bpf_timed_may_goto.o
+obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o bpf_timed_may_goto.o bpf_cleanup_pad.o
diff --git a/arch/arm64/net/bpf_cleanup_pad.S b/arch/arm64/net/bpf_cleanup_pad.S
new file mode 100644
index 0000000000000..ef441241949ec
--- /dev/null
+++ b/arch/arm64/net/bpf_cleanup_pad.S
@@ -0,0 +1,95 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+#include <linux/bpf_cleanup_abi.h>
+#include <linux/linkage.h>
+
+/*
+ * A frame's prologue pushes the tail call counter pair, and then -- once
+ * bpf_cleanup_force_spill() says so -- x19/x20, x21/x22, x23/x24, x25/x26 and
+ * x27/x28. Each A64_PUSH pre-decrements, so the lowest address of the spill
+ * area holds x27 and the highest x20:
+ *
+ * spill_base + 0 x27 (private stack pointer)
+ * spill_base + 8 x28 (arena base)
+ * spill_base + 16 x25 (BPF r10, the frame pointer)
+ * spill_base + 24 x26 (tail call counter pointer)
+ * spill_base + 32 x23 -- the pad's own, see below
+ * spill_base + 40 x24 -- likewise
+ * spill_base + 48 x21 (BPF r8)
+ * spill_base + 56 x22 (BPF r9)
+ * spill_base + 64 x19 (BPF r6)
+ * spill_base + 72 x20 (BPF r7)
+ *
+ * Neither x23 nor x24 is restored from that spill: the pad has its own use for
+ * both. bpf2a64[] maps nothing to either -- they are pushed only to keep the
+ * frame shape the exception callback expects -- so nothing else in generated
+ * code touches them, and being callee-saved they survive every call the pad
+ * makes.
+ *
+ * x23 is the pad's return address. Unlike x86-64 a pad cannot simply return:
+ * every call it makes clobbers x30, so nothing is left to return through by
+ * the time it reaches its resume. The JIT emits "br x23" for the pad's
+ * bpf_unwind_resume() and this routine puts .Lcleanup_pad_done there.
+ *
+ * x24 is where the pad's frame is anchored. Generated code addresses the BPF
+ * frame through the stack pointer, which here is this routine's rather than
+ * the unwinding frame's, so the JIT has the pad recompute the equivalent from
+ * BPF r10 on entry and address its frame off x24 for as long as it runs.
+ *
+ * Both of those branches are indirect, so both targets carry a BTI landing
+ * marker: the JIT emits one at each pad, and .Lcleanup_pad_done below has one
+ * of its own.
+ */
+
+ .text
+
+/*
+ * void arch_bpf_run_cleanup_pad(u64 pad, u64 frame_fp, u64 spill_base)
+ *
+ * x0 = native address of the landing pad
+ * x1 = frame pointer of the frame the pad belongs to (unused here: BPF r10 is
+ * x25, which the spill area already holds)
+ * x2 = spill area holding that frame's BPF callee-saved registers
+ *
+ * Give the pad the register state of its own frame and call it. It runs on
+ * this stack, far below the frame it is cleaning up after, so nothing it
+ * calls can reach into that frame.
+ */
+SYM_FUNC_START(arch_bpf_run_cleanup_pad)
+ /* Save the kernel's callee-saved registers; the pad owns them next. */
+ stp x29, x30, [sp, #-96]!
+ mov x29, sp
+ stp x19, x20, [sp, #16]
+ stp x21, x22, [sp, #32]
+ stp x23, x24, [sp, #48]
+ stp x25, x26, [sp, #64]
+ stp x27, x28, [sp, #80]
+
+ /* x9 is BPF_REG_AX, so the pad's address does not stay in BPF r1. */
+ mov x9, x0
+
+ ldp x27, x28, [x2, #0]
+ ldp x25, x26, [x2, #16]
+ ldp x21, x22, [x2, #48]
+ ldp x19, x20, [x2, #64]
+
+ /* BPF r0 (x8) on the way into a pad, not whatever the kernel left. */
+ mov x8, #BPF_PAD_ENTRY_R0
+
+ /* Where the pad's resume branches back to. */
+ adr x23, .Lcleanup_pad_done
+
+ br x9
+
+.Lcleanup_pad_done:
+ /* Reached by the pad's "br x23", so it is an indirect branch target. */
+ bti j
+ ldp x19, x20, [sp, #16]
+ ldp x21, x22, [sp, #32]
+ ldp x23, x24, [sp, #48]
+ ldp x25, x26, [sp, #64]
+ ldp x27, x28, [sp, #80]
+ ldp x29, x30, [sp], #96
+ ret
+SYM_FUNC_END(arch_bpf_run_cleanup_pad)
diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
index 6c04fee468766..7d26e43e29c3a 100644
--- a/arch/arm64/net/bpf_jit_comp.c
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -11,6 +11,7 @@
#include <linux/bitfield.h>
#include <linux/bpf.h>
#include <linux/cfi.h>
+#include <linux/bpf_verifier.h>
#include <linux/filter.h>
#include <linux/memory.h>
#include <linux/printk.h>
@@ -75,6 +76,19 @@ static const int bpf2a64[] = {
[ARENA_VM_START] = A64_R(28),
};
+/*
+ * Throw-site spill: the five pairs push_callee_regs() forces on, same size and
+ * slot order, so arch_bpf_run_cleanup_pad() reads both alike.
+ */
+#define A64_CLEANUP_SPILL_SZ (5 * 16)
+
+/*
+ * Where a landing pad's frame is anchored, since the stack pointer generated
+ * code normally addresses it through is the walker's inside a pad. bpf2a64[]
+ * maps nothing to x24, so nothing else in generated code touches it.
+ */
+#define A64_CLEANUP_FP A64_R(24)
+
struct jit_ctx {
const struct bpf_prog *prog;
int idx;
@@ -87,6 +101,8 @@ struct jit_ctx {
__le32 *ro_image;
u32 stack_size;
u16 stack_arg_size;
+ /* Bytes reserved for the throw-site spill; see bpf_cleanup_force_spill(). */
+ u32 throw_spill;
u64 user_vm_start;
u64 arena_vm_start;
bool fp_used;
@@ -432,7 +448,7 @@ static void push_callee_regs(struct jit_ctx *ctx)
* Callee-saved registers as the exception callback needs to recover
* all ARM64 Callee-saved registers in its epilogue.
*/
- if (ctx->prog->aux->exception_boundary) {
+ if (ctx->prog->aux->exception_boundary || bpf_cleanup_force_spill(ctx->prog)) {
emit(A64_PUSH(A64_R(19), A64_R(20), A64_SP), ctx);
emit(A64_PUSH(A64_R(21), A64_R(22), A64_SP), ctx);
emit(A64_PUSH(A64_R(23), A64_R(24), A64_SP), ctx);
@@ -466,7 +482,8 @@ static void pop_callee_regs(struct jit_ctx *ctx)
* program's stack frame, so recover these extra registers in the above
* two cases.
*/
- if (aux->exception_boundary || aux->exception_cb) {
+ if (aux->exception_boundary || aux->exception_cb ||
+ bpf_cleanup_force_spill(ctx->prog)) {
emit(A64_POP(A64_R(27), A64_R(28), A64_SP), ctx);
emit(A64_POP(A64_R(25), A64_R(26), A64_SP), ctx);
emit(A64_POP(A64_R(23), A64_R(24), A64_SP), ctx);
@@ -602,6 +619,20 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)
emit(A64_SUB_I(1, A64_SP, A64_FP, 96), ctx);
}
+ /*
+ * Lowest address of each spill area, as an offset from A64_FP; see
+ * bpf_cleanup_pad.S for the layout. The 16 is the tail call counter
+ * pair pushed just below the frame record, and the throw-site area
+ * sits below the callee-saved one rather than in the program stack.
+ */
+ if (bpf_cleanup_force_spill(prog)) {
+ prog->aux->exc->spill_off = -(16 + A64_CLEANUP_SPILL_SZ);
+ ctx->throw_spill = A64_CLEANUP_SPILL_SZ;
+ emit(A64_SUB_I(1, A64_SP, A64_SP, ctx->throw_spill), ctx);
+ prog->aux->exc->throw_spill_off =
+ -(16 + A64_CLEANUP_SPILL_SZ) - ctx->throw_spill;
+ }
+
/* Stack must be multiples of 16B */
ctx->stack_size = round_up(prog->aux->stack_depth, 16);
@@ -691,6 +722,10 @@ static int emit_bpf_tail_call(struct jit_ctx *ctx)
if (ctx->stack_size && !ctx->priv_sp_used)
emit(A64_ADD_I(1, A64_SP, A64_SP, ctx->stack_size), ctx);
+ /* Release it for the same reason build_epilogue() does. */
+ if (ctx->throw_spill)
+ emit(A64_ADD_I(1, A64_SP, A64_SP, ctx->throw_spill), ctx);
+
pop_callee_regs(ctx);
/* goto *(prog->bpf_func + prologue_offset); */
@@ -1055,6 +1090,9 @@ static void build_epilogue(struct jit_ctx *ctx, bool was_classic)
if (ctx->stack_size && !ctx->priv_sp_used)
emit(A64_ADD_I(1, A64_SP, A64_SP, ctx->stack_size), ctx);
+ if (ctx->throw_spill)
+ emit(A64_ADD_I(1, A64_SP, A64_SP, ctx->throw_spill), ctx);
+
pop_callee_regs(ctx);
emit(A64_POP(A64_ZR, ptr, A64_SP), ctx);
@@ -1367,6 +1405,8 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
const s16 off = insn->off;
const s32 imm = insn->imm;
const int i = insn - ctx->prog->insnsi;
+ const bool in_pad = bpf_cleanup_insn_in_pad(ctx->prog, i);
+ const bool pad_head = bpf_cleanup_insn_is_pad(ctx->prog, i);
const bool is64 = BPF_CLASS(code) == BPF_ALU64 ||
BPF_CLASS(code) == BPF_JMP;
u8 jmp_cond;
@@ -1378,9 +1418,13 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
int ret;
bool sign_extend;
- if (bpf_insn_is_indirect_target(env, ctx->prog, i))
+ if (bpf_insn_is_indirect_target(env, ctx->prog, i) || pad_head)
emit_bti(A64_BTI_J, ctx);
+ if (pad_head)
+ emit(A64_SUB_I(1, A64_CLEANUP_FP, fp,
+ ctx->stack_size + ctx->stack_arg_size), ctx);
+
switch (code) {
/* dst = src */
case BPF_ALU | BPF_MOV | BPF_X:
@@ -1743,6 +1787,26 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
u64 func_addr;
u32 cpu_offset;
+ if (bpf_cleanup_insn_is_throw(ctx->prog, i)) {
+ /* Spill where the bpf_throw() walker looks. */
+ const s32 off = ctx->prog->aux->exc->throw_spill_off;
+
+ emit(A64_SUB_I(1, tmp, A64_FP, -off), ctx);
+ emit(A64_STR64I(bpf2a64[PRIVATE_SP], tmp, 0), ctx);
+ emit(A64_STR64I(bpf2a64[ARENA_VM_START], tmp, 8), ctx);
+ emit(A64_STR64I(bpf2a64[BPF_REG_FP], tmp, 16), ctx);
+ emit(A64_STR64I(bpf2a64[TCCNT_PTR], tmp, 24), ctx);
+ emit(A64_STR64I(bpf2a64[BPF_REG_8], tmp, 48), ctx);
+ emit(A64_STR64I(bpf2a64[BPF_REG_9], tmp, 56), ctx);
+ emit(A64_STR64I(bpf2a64[BPF_REG_6], tmp, 64), ctx);
+ emit(A64_STR64I(bpf2a64[BPF_REG_7], tmp, 72), ctx);
+ }
+
+ if (bpf_cleanup_insn_is_resume(ctx->prog, i)) {
+ emit(A64_BR(A64_R(23)), ctx);
+ break;
+ }
+
/* Implement helper call to bpf_get_smp_processor_id() inline */
if (insn->src_reg == 0 && insn->imm == BPF_FUNC_get_smp_processor_id) {
cpu_offset = offsetof(struct thread_info, cpu);
@@ -1854,7 +1918,8 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
src = tmp2;
}
if (src == fp) {
- src_adj = ctx->priv_sp_used ? priv_sp : A64_SP;
+ src_adj = ctx->priv_sp_used ? priv_sp :
+ in_pad ? A64_CLEANUP_FP : A64_SP;
off_adj = off + ctx->stack_size;
if (!ctx->priv_sp_used)
off_adj += ctx->stack_arg_size;
@@ -1952,7 +2017,8 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
dst = tmp3;
}
if (dst == fp) {
- dst_adj = ctx->priv_sp_used ? priv_sp : A64_SP;
+ dst_adj = ctx->priv_sp_used ? priv_sp :
+ in_pad ? A64_CLEANUP_FP : A64_SP;
off_adj = off + ctx->stack_size;
if (!ctx->priv_sp_used)
off_adj += ctx->stack_arg_size;
@@ -2021,7 +2087,8 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
dst = tmp2;
}
if (dst == fp) {
- dst_adj = ctx->priv_sp_used ? priv_sp : A64_SP;
+ dst_adj = ctx->priv_sp_used ? priv_sp :
+ in_pad ? A64_CLEANUP_FP : A64_SP;
off_adj = off + ctx->stack_size;
if (!ctx->priv_sp_used)
off_adj += ctx->stack_arg_size;
@@ -2410,6 +2477,13 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
* reasons, expects to point to the next instruction)
*/
bpf_prog_update_insn_ptrs(prog, ctx.offset, ctx.ro_image);
+
+ /*
+ * Same byte offsets, consumed by the bpf_throw() frame walker:
+ * turn the cleanup records into native address ranges now that
+ * the image is final.
+ */
+ bpf_cleanup_fill_native_ranges(prog, ctx.offset, ctx.ro_image);
out_off:
if (!ro_header && priv_stack_ptr) {
free_percpu(priv_stack_ptr);
@@ -3385,6 +3459,11 @@ bool bpf_jit_supports_exceptions(void)
return true;
}
+bool bpf_jit_supports_cleanup_pads(void)
+{
+ return true;
+}
+
bool bpf_jit_supports_arena(void)
{
return true;
diff --git a/arch/x86/net/Makefile b/arch/x86/net/Makefile
index dddbefc0f4398..8bb22644ba10b 100644
--- a/arch/x86/net/Makefile
+++ b/arch/x86/net/Makefile
@@ -7,4 +7,7 @@ ifeq ($(CONFIG_X86_32),y)
obj-$(CONFIG_BPF_JIT) += bpf_jit_comp32.o
else
obj-$(CONFIG_BPF_JIT) += bpf_jit_comp.o bpf_timed_may_goto.o
+ ifdef CONFIG_UNWINDER_ORC
+ obj-$(CONFIG_BPF_JIT) += bpf_cleanup_pad.o
+ endif
endif
diff --git a/arch/x86/net/bpf_cleanup_pad.S b/arch/x86/net/bpf_cleanup_pad.S
new file mode 100644
index 0000000000000..025a54e718a40
--- /dev/null
+++ b/arch/x86/net/bpf_cleanup_pad.S
@@ -0,0 +1,80 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+#include <linux/bpf_cleanup_abi.h>
+#include <linux/linkage.h>
+#include <asm/nospec-branch.h>
+
+/*
+ * The x86-64 BPF JIT prologue spills, once bpf_cleanup_force_spill() makes it
+ * unconditional, r12, rbx, r13, r14 and r15 in that order -- so within the
+ * spill area the lowest address holds r15 and the highest r12. The throw-site
+ * spill the JIT emits uses the same layout, so the routine below reads both
+ * the same way:
+ *
+ * spill_base + 0 BPF r9 (r15)
+ * spill_base + 8 BPF r8 (r14)
+ * spill_base + 16 BPF r7 (r13)
+ * spill_base + 24 BPF r6 (rbx)
+ * spill_base + 32 r12 (arena base, not a BPF register)
+ */
+
+ .code64
+ .section .text, "ax"
+
+/*
+ * void arch_bpf_run_cleanup_pad(u64 pad, u64 frame_fp, u64 spill_base)
+ *
+ * rdi = native address of the landing pad
+ * rsi = frame pointer of the frame the pad belongs to
+ * rdx = spill area holding that frame's BPF callee-saved registers
+ *
+ * Give the pad the register state of its own frame and call it. The pad ends
+ * in the bare return the JIT emits for its bpf_unwind_resume(), so it comes
+ * back here rather than returning to its frame's caller. It runs on this
+ * stack, far below the frame it is cleaning up after, so nothing it calls can
+ * reach into that frame.
+ *
+ * rbp addresses the pad's frame and rsp this one, so the two are not the
+ * neighbours a JITed frame expects them to be. That is why
+ * cleanup_check_pad_insn() refuses a call in a pad that passes an argument on
+ * the stack: the JIT stages those relative to rbp, and the callee reads them
+ * relative to rsp.
+ */
+SYM_FUNC_START(arch_bpf_run_cleanup_pad)
+ ANNOTATE_NOENDBR
+
+ pushq %rbp
+ pushq %rbx
+ pushq %r12
+ pushq %r13
+ pushq %r14
+ pushq %r15
+ /* Keep the pad's entry rsp congruent to a normal call's. */
+ subq $8, %rsp
+
+ movq 0(%rdx), %r15
+ movq 8(%rdx), %r14
+ movq 16(%rdx), %r13
+ movq 24(%rdx), %rbx
+ movq 32(%rdx), %r12
+ /* rbp is BPF r10, so this is the whole of the pad's frame setup. */
+ movq %rsi, %rbp
+
+ /* CALL_NOSPEC needs the target in a register; rcx is BPF r4, dead. */
+ movq %rdi, %rcx
+
+ /* BPF r0 on the way into a pad, not whatever the kernel left in rax. */
+ movl $BPF_PAD_ENTRY_R0, %eax
+
+ CALL_NOSPEC rcx
+
+ addq $8, %rsp
+ popq %r15
+ popq %r14
+ popq %r13
+ popq %r12
+ popq %rbx
+ popq %rbp
+ RET
+SYM_FUNC_END(arch_bpf_run_cleanup_pad)
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index d4a980140b48d..04db7d7fc397e 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -357,6 +357,12 @@ struct jit_context {
/* Number of bytes that will be skipped on tailcall */
#define X86_TAIL_CALL_OFFSET (12 + ENDBR_INSN_SIZE)
+/*
+ * Throw-site spill: r15, r14, r13, rbx, r12 low to high, the layout the
+ * prologue's pushes leave, so arch_bpf_run_cleanup_pad() reads both alike.
+ */
+#define X86_CLEANUP_SPILL_SZ (5 * 8)
+
static void push_r9(u8 **pprog)
{
u8 *prog = *pprog;
@@ -832,7 +838,7 @@ static void emit_bpf_tail_call_indirect(struct bpf_prog *bpf_prog,
/* Inc tail_call_cnt if the slot is populated. */
EMIT4(0x48, 0x83, 0x00, 0x01); /* add qword ptr [rax], 1 */
- if (bpf_prog->aux->exception_boundary) {
+ if (bpf_prog->aux->exception_boundary || bpf_cleanup_force_spill(bpf_prog)) {
pop_callee_regs(&prog, all_callee_regs_used);
pop_r12(&prog);
} else {
@@ -899,7 +905,7 @@ static void emit_bpf_tail_call_direct(struct bpf_prog *bpf_prog,
/* Inc tail_call_cnt if the slot is populated. */
EMIT4(0x48, 0x83, 0x00, 0x01); /* add qword ptr [rax], 1 */
- if (bpf_prog->aux->exception_boundary) {
+ if (bpf_prog->aux->exception_boundary || bpf_cleanup_force_spill(bpf_prog)) {
pop_callee_regs(&prog, all_callee_regs_used);
pop_r12(&prog);
} else {
@@ -1977,6 +1983,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
u8 *ip, *prog = temp;
u32 stack_depth;
int callee_saved_size;
+ u32 throw_spill, prologue_depth;
s32 outgoing_arg_base;
int err;
@@ -2015,7 +2022,10 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
detect_reg_usage(insn, insn_cnt, callee_regs_used);
- emit_prologue(&prog, image, stack_depth,
+ throw_spill = bpf_cleanup_force_spill(bpf_prog) ? X86_CLEANUP_SPILL_SZ : 0;
+ prologue_depth = stack_depth + throw_spill;
+
+ emit_prologue(&prog, image, prologue_depth,
bpf_prog_was_classic(bpf_prog), tail_call_reachable,
bpf_is_subprog(bpf_prog), bpf_prog->aux->exception_cb);
@@ -2024,7 +2034,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
/* Exception callback will clobber callee regs for its own use, and
* restore the original callee regs from main prog's stack frame.
*/
- if (bpf_prog->aux->exception_boundary) {
+ if (bpf_prog->aux->exception_boundary || bpf_cleanup_force_spill(bpf_prog)) {
/* We also need to save r12, which is not mapped to any BPF
* register, as we throw after entry into the kernel, which may
* overwrite r12.
@@ -2039,9 +2049,10 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
/* Compute callee-saved register area size. */
callee_saved_size = 0;
- if (bpf_prog->aux->exception_boundary || arena_vm_start)
+ if (bpf_prog->aux->exception_boundary || bpf_cleanup_force_spill(bpf_prog) ||
+ arena_vm_start)
callee_saved_size += 8; /* r12 */
- if (bpf_prog->aux->exception_boundary) {
+ if (bpf_prog->aux->exception_boundary || bpf_cleanup_force_spill(bpf_prog)) {
callee_saved_size += 4 * 8; /* rbx, r13, r14, r15 */
} else {
int j;
@@ -2063,7 +2074,19 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
* Note that tail_call_reachable is guaranteed to be false when
* stack args exist, so tcc pushes need not be accounted for.
*/
- outgoing_arg_base = -(round_up(stack_depth, 8) + callee_saved_size);
+ outgoing_arg_base = -(round_up(stack_depth, 8) + throw_spill + callee_saved_size);
+
+ /*
+ * Lowest address of each spill area, as an offset from rbp; see
+ * bpf_cleanup_pad.S for the layout. The 16 is the tail call counter
+ * pair emit_prologue_tail_call() pushes above the callee-saved one.
+ */
+ if (bpf_cleanup_force_spill(bpf_prog)) {
+ bpf_prog->aux->exc->spill_off = -(round_up(stack_depth, 8) + throw_spill +
+ (tail_call_reachable ? 16 : 0) +
+ callee_saved_size);
+ bpf_prog->aux->exc->throw_spill_off = -(round_up(stack_depth, 8) + throw_spill);
+ }
/*
* Allocate outgoing stack arg area for args 7+ only.
@@ -2110,7 +2133,8 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
dst_reg = X86_REG_R9;
}
- if (bpf_insn_is_indirect_target(env, bpf_prog, i - 1))
+ if (bpf_insn_is_indirect_target(env, bpf_prog, i - 1) ||
+ bpf_cleanup_insn_is_pad(bpf_prog, i - 1))
EMIT_ENDBR();
ip = image + addrs[i - 1] + (prog - temp);
@@ -2903,9 +2927,27 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
case BPF_JMP | BPF_CALL: {
const struct btf_func_model *fm = NULL;
+ if (bpf_cleanup_insn_is_throw(bpf_prog, i - 1)) {
+ /* Spill r6-r9 and r12 where the bpf_throw() walker looks. */
+ s32 off = bpf_prog->aux->exc->throw_spill_off;
+ u8 *spill = prog;
+
+ emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_9, off + 0);
+ emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_8, off + 8);
+ emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_7, off + 16);
+ emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_6, off + 24);
+ emit_stx(&prog, BPF_DW, BPF_REG_FP, X86_REG_R12, off + 32);
+ ip += prog - spill;
+ }
+
+ if (bpf_cleanup_insn_is_resume(bpf_prog, i - 1)) {
+ emit_return(&prog, image + addrs[i - 1] + (prog - temp));
+ break;
+ }
+
func = (u8 *) __bpf_call_base + imm32;
if (src_reg == BPF_PSEUDO_CALL && tail_call_reachable) {
- LOAD_TAIL_CALL_CNT_PTR(stack_depth);
+ LOAD_TAIL_CALL_CNT_PTR(prologue_depth);
ip += 7;
}
if (!imm32)
@@ -2948,13 +2990,13 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
&prog,
ip,
callee_regs_used,
- stack_depth,
+ prologue_depth,
ctx);
else
emit_bpf_tail_call_indirect(bpf_prog,
&prog,
callee_regs_used,
- stack_depth,
+ prologue_depth,
ip,
ctx);
break;
@@ -3215,7 +3257,8 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
}
/* Deallocate outgoing args 7+ area. */
emit_add_rsp(&prog, outgoing_rsp);
- if (bpf_prog->aux->exception_boundary) {
+ if (bpf_prog->aux->exception_boundary ||
+ bpf_cleanup_force_spill(bpf_prog)) {
pop_callee_regs(&prog, all_callee_regs_used);
pop_r12(&prog);
} else {
@@ -4385,6 +4428,13 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
*/
bpf_prog_update_insn_ptrs(prog, addrs, image);
+ /*
+ * Same mapping, consumed by the bpf_throw() frame walker:
+ * turn the cleanup records into native address ranges now
+ * that the image is final.
+ */
+ bpf_cleanup_fill_native_ranges(prog, addrs, image);
+
/*
* ctx.prog_offset is used when CFI preambles put code *before*
* the function. See emit_cfi(). For FineIBT specifically this code
@@ -4501,6 +4551,11 @@ bool bpf_jit_supports_exceptions(void)
return IS_ENABLED(CONFIG_UNWINDER_ORC);
}
+bool bpf_jit_supports_cleanup_pads(void)
+{
+ return IS_ENABLED(CONFIG_UNWINDER_ORC);
+}
+
bool bpf_jit_supports_private_stack(void)
{
return true;
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 5033b934ffd90..0502b9fc4a6de 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -1774,6 +1774,89 @@ enum bpf_sig_keyring {
BPF_SIG_KEYRING_BPF,
};
+/*
+ * One cleanup region of a JITed (sub)program: @pad is the landing pad to run
+ * for a return address in (begin, end], the native code of its call sites.
+ */
+struct bpf_cleanup_range {
+ u64 begin;
+ u64 end;
+ u64 pad;
+};
+
+struct bpf_exception_info {
+ struct bpf_cleanup_info *info;
+ struct bpf_cleanup_range *ranges;
+ /* Landing pad instruction indices, sorted and deduplicated. */
+ u32 *pad_at;
+ /* bpf_throw() call instruction indices, sorted. */
+ u32 *throw_at;
+ /* Likewise for bpf_unwind_resume(), the way back out of a pad. */
+ u32 *resume_at;
+ /* One bit per instruction that only runs while unwinding. */
+ unsigned long *pad_body;
+ u32 nr_info;
+ u32 nr_ranges;
+ u32 nr_pad_at;
+ u32 nr_throw_at;
+ u32 nr_resume_at;
+ u32 nr_pad_body;
+ /* Offset from a frame's FP to the caller's spilled r6-r9. */
+ s32 spill_off;
+ /* Likewise, to the registers a frame spills before calling bpf_throw(). */
+ s32 throw_spill_off;
+};
+
+#ifdef CONFIG_BPF_SYSCALL
+bool bpf_cleanup_force_spill(const struct bpf_prog *prog);
+bool bpf_cleanup_insn_is_pad(const struct bpf_prog *prog, u32 idx);
+bool bpf_cleanup_insn_in_pad(const struct bpf_prog *prog, u32 idx);
+bool bpf_cleanup_insn_is_throw(const struct bpf_prog *prog, u32 idx);
+bool bpf_cleanup_insn_is_resume(const struct bpf_prog *prog, u32 idx);
+int bpf_cleanup_attach_main_prog(struct bpf_verifier_env *env, struct bpf_prog *prog);
+void bpf_cleanup_fill_native_ranges(struct bpf_prog *prog, u32 *addrs, void *image);
+void bpf_cleanup_free_info(struct bpf_prog_aux *aux);
+#else
+static inline bool bpf_cleanup_force_spill(const struct bpf_prog *prog)
+{
+ return false;
+}
+
+static inline bool bpf_cleanup_insn_is_pad(const struct bpf_prog *prog, u32 idx)
+{
+ return false;
+}
+
+static inline bool bpf_cleanup_insn_in_pad(const struct bpf_prog *prog, u32 idx)
+{
+ return false;
+}
+
+static inline bool bpf_cleanup_insn_is_throw(const struct bpf_prog *prog, u32 idx)
+{
+ return false;
+}
+
+static inline bool bpf_cleanup_insn_is_resume(const struct bpf_prog *prog, u32 idx)
+{
+ return false;
+}
+
+static inline int bpf_cleanup_attach_main_prog(struct bpf_verifier_env *env,
+ struct bpf_prog *prog)
+{
+ return 0;
+}
+
+static inline void bpf_cleanup_fill_native_ranges(struct bpf_prog *prog, u32 *addrs, void *image)
+{
+}
+
+static inline void bpf_cleanup_free_info(struct bpf_prog_aux *aux)
+{
+}
+#endif
+
struct bpf_prog_aux {
atomic64_t refcnt;
u32 used_map_cnt;
@@ -1854,6 +1937,7 @@ struct bpf_prog_aux {
char name[BPF_OBJ_NAME_LEN];
u64 (*bpf_exception_cb)(u64 cookie, u64 sp, u64 bp, u64, u64);
u16 stack_arg_sp_adjust;
+ struct bpf_exception_info *exc;
#ifdef CONFIG_SECURITY
void *security;
#endif
diff --git a/include/linux/bpf_cleanup_abi.h b/include/linux/bpf_cleanup_abi.h
new file mode 100644
index 0000000000000..b6c1d589abda7
--- /dev/null
+++ b/include/linux/bpf_cleanup_abi.h
@@ -0,0 +1,16 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#ifndef _LINUX_BPF_CLEANUP_ABI_H
+#define _LINUX_BPF_CLEANUP_ABI_H
+
+/*
+ * Value arch_bpf_run_cleanup_pad() leaves in r0 on the way into a landing pad.
+ * It has to be a constant the verifier knows: LLVM names r0 as both the
+ * exception pointer and the exception selector register, so every pad reads it
+ * before anything else and is free to store what it read. Kept on its own
+ * because the verifier and the arch dispatchers, which are assembly, have to
+ * agree on it.
+ */
+#define BPF_PAD_ENTRY_R0 1
+
+#endif /* _LINUX_BPF_CLEANUP_ABI_H */
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index cf85141ea1674..0721461aaabbf 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -429,7 +429,8 @@ struct bpf_jmp_history_entry {
u32 prev_idx : 20;
/* special INSN_F_xxx flags */
u32 flags : 4;
- u32 : 8;
+ u32 unwind_frames : 4; /* frames the unwind popped to get here */
+ u32 : 4;
/*
* additional registers that need precision tracking when this
* jump is backtracked, vector of five 11-bit records
@@ -509,6 +510,8 @@ struct bpf_verifier_state {
bool speculative;
bool in_sleepable;
+ bool unwinding; /* an exception is in flight */
+ u8 unwind_frameno; /* the frame whose landing pad it entered */
/* first and last insn idx of this verifier state */
u32 first_insn_idx;
@@ -681,6 +684,14 @@ struct bpf_insn_aux_data {
bool needs_zext; /* alu op needs to clear upper bits */
bool non_sleepable; /* helper/kfunc may be called from non-sleepable context */
bool is_iter_next; /* bpf_iter_<type>_next() kfunc call */
+ bool cleanup_throw_site; /* call to bpf_throw() */
+ bool cleanup_resume_site; /* call to bpf_unwind_resume() */
+ bool in_cleanup_pad; /* only runs with an exception in flight */
+ /*
+ * 1 + the instruction index of the exception cleanup landing pad this
+ * call site unwinds to, or 0 for none.
+ */
+ u32 cleanup_pad;
bool call_with_percpu_alloc_ptr; /* {this,per}_cpu_ptr() with prog percpu alloc */
u8 alu_state; /* used in combination with alu_limit */
/* true if STX or LDX instruction is a part of a spill/fill
@@ -983,10 +994,13 @@ struct bpf_verifier_env {
} cfg;
struct backtrack_state bt;
struct bpf_jmp_history_entry *cur_hist_ent;
+ u8 unwind_frames; /* scratch: pops for the insn about to be recorded */
/* Per-callsite copy of parent's converged at_stack_in for cross-frame fills. */
struct arg_track **callsite_at_stack;
u32 pass_cnt; /* number of times do_check() was called */
u32 subprog_cnt;
+ struct bpf_cleanup_info *cleanup_info;
+ u32 cleanup_info_cnt;
/* number of instructions analyzed by the verifier */
u32 prev_insn_processed, insn_processed;
/* number of jmps, calls, exits analyzed so far */
@@ -1516,6 +1530,7 @@ u32 btf_func_arg_align(const struct btf *btf, const struct btf_type *t);
int bpf_find_subprog(struct bpf_verifier_env *env, int off);
bool bpf_is_throw_kfunc(struct bpf_insn *insn);
+bool bpf_is_unwind_resume_kfunc(const struct bpf_insn *insn);
int bpf_compute_const_regs(struct bpf_verifier_env *env);
int bpf_prune_dead_branches(struct bpf_verifier_env *env);
int bpf_check_cfg(struct bpf_verifier_env *env);
diff --git a/include/linux/filter.h b/include/linux/filter.h
index 422284b4fa96f..b0c495e94f740 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -1242,6 +1242,8 @@ bool bpf_jit_supports_stack_args(void);
bool bpf_jit_supports_arena_args(void);
bool bpf_jit_supports_far_kfunc_call(void);
bool bpf_jit_supports_exceptions(void);
+bool bpf_jit_supports_cleanup_pads(void);
+void arch_bpf_run_cleanup_pad(u64 pad, u64 frame_fp, u64 spill_base);
bool bpf_jit_supports_ptr_xchg(void);
bool bpf_jit_supports_arena(void);
bool bpf_jit_supports_insn(struct bpf_insn *insn, bool in_arena);
diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
index 6330b7d745c57..a69cc04275751 100644
--- a/include/uapi/linux/bpf.h
+++ b/include/uapi/linux/bpf.h
@@ -1669,6 +1669,9 @@ union bpf_attr {
* verification.
*/
__s32 keyring_id;
+ __aligned_u64 cleanup_info; /* exception cleanup table */
+ __u32 cleanup_info_rec_size; /* userspace bpf_cleanup_info size */
+ __u32 cleanup_info_cnt; /* number of bpf_cleanup_info records */
};
struct { /* anonymous struct used by BPF_OBJ_* commands */
@@ -7599,6 +7602,12 @@ struct bpf_line_info {
__u32 line_col;
};
+struct bpf_cleanup_info {
+ __u32 begin_off;
+ __u32 end_off;
+ __u32 landing_pad_off;
+};
+
struct bpf_spin_lock {
__u32 val;
};
diff --git a/kernel/bpf/Makefile b/kernel/bpf/Makefile
index 9a92c348bbda6..af9bc60428ad1 100644
--- a/kernel/bpf/Makefile
+++ b/kernel/bpf/Makefile
@@ -11,7 +11,7 @@ obj-$(CONFIG_BPF_SYSCALL) += bpf_iter.o map_iter.o task_iter.o prog_iter.o link_
obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o bloom_filter.o
obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o bpf_insn_array.o
obj-$(CONFIG_BPF_SYSCALL) += bpf_local_storage.o bpf_task_storage.o
-obj-$(CONFIG_BPF_SYSCALL) += fixups.o cfg.o states.o backtrack.o check_btf.o
+obj-$(CONFIG_BPF_SYSCALL) += fixups.o cfg.o states.o backtrack.o check_btf.o exception.o
obj-${CONFIG_BPF_LSM} += bpf_inode_storage.o
obj-$(CONFIG_BPF_SYSCALL) += disasm.o mprog.o
obj-$(CONFIG_BPF_JIT) += trampoline.o
diff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c
index 507a366dffa47..c19e4fe247ce5 100644
--- a/kernel/bpf/backtrack.c
+++ b/kernel/bpf/backtrack.c
@@ -4,6 +4,7 @@
#include <linux/bpf_verifier.h>
#include <linux/filter.h>
#include <linux/bitmap.h>
+#include "exception.h"
#define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
@@ -47,6 +48,7 @@ int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state
p->flags = insn_flags;
p->spi = spi;
p->frame = frame;
+ p->unwind_frames = 0;
p->linked_regs = linked_regs;
cur->jmp_history_cnt = cnt;
env->cur_hist_ent = p;
@@ -419,10 +421,12 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
* extra instructions from subprog; the next
* instruction after call to global subprog
* should be literally next instruction in
- * caller program
+ * caller program -- or, if the callee threw,
+ * the landing pad of this call site
*/
- verifier_bug_if(idx + 1 != subseq_idx, env,
- "extra insn from subprog");
+ verifier_bug_if(idx + 1 != subseq_idx &&
+ bpf_cleanup_pad_of_call(env, idx) != subseq_idx,
+ env, "extra insn from subprog");
/* global subprog always sets R0 */
bt_clear_reg(bt, BPF_REG_0);
/* and if it does not set R2, main pass would catch it */
@@ -888,11 +892,11 @@ int bpf_mark_chain_precision(struct bpf_verifier_env *env,
}
for (i = last_idx;;) {
+ hist = get_jmp_hist_entry(st, history, i);
if (skip_first) {
err = 0;
skip_first = false;
} else {
- hist = get_jmp_hist_entry(st, history, i);
err = backtrack_insn(env, i, subseq_idx, hist, bt);
}
if (err == -ENOTSUPP) {
@@ -909,6 +913,15 @@ int bpf_mark_chain_precision(struct bpf_verifier_env *env,
*/
return 0;
subseq_idx = i;
+ /* An exception unwind reached this insn, a landing
+ * pad, from a throw or a resume that many frames
+ * deeper. There is no insn in between to backtrack
+ * over, so enter those frames here, the way BPF_EXIT
+ * does one at a time.
+ */
+ for (fr = 0; hist && fr < hist->unwind_frames; fr++)
+ if (bt_subprog_enter(bt))
+ return -EFAULT;
i = get_prev_insn_idx(st, i, &history);
if (i == -ENOENT)
break;
diff --git a/kernel/bpf/cfg.c b/kernel/bpf/cfg.c
index 842c7d1eabccc..3fd98dcadce89 100644
--- a/kernel/bpf/cfg.c
+++ b/kernel/bpf/cfg.c
@@ -6,6 +6,7 @@
#include <linux/sort.h>
#include "diagnostics.h"
+#include "exception.h"
#define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
@@ -158,17 +159,64 @@ static int push_insn(int t, int w, int e, struct bpf_verifier_env *env)
return DONE_EXPLORING;
}
+static int visit_cleanup_pad_edge(int t, struct bpf_verifier_env *env)
+{
+ int *insn_stack = env->cfg.insn_stack;
+ int *insn_state = env->cfg.insn_state;
+ int w;
+
+ if (!env->cleanup_info_cnt)
+ return DONE_EXPLORING;
+ w = bpf_cleanup_pad_of_call(env, t);
+ if (w < 0)
+ return DONE_EXPLORING;
+
+ /*
+ * @t is a call that may branch here, and @w is the target of that
+ * branch, so both are prune points. @w especially: every covered call
+ * site in a region unwinds to the same pad, and without a prune point
+ * at its head the verifier walks the pad again for each of them.
+ */
+ mark_prune_point(env, t);
+ mark_prune_point(env, w);
+ mark_jmp_point(env, w);
+ mark_jump_target(env, w);
+
+ if (insn_state[w])
+ return DONE_EXPLORING;
+ if (env->cfg.cur_stack >= env->prog->len)
+ return -E2BIG;
+ insn_stack[env->cfg.cur_stack++] = w;
+ insn_state[w] |= DISCOVERED;
+ return KEEP_EXPLORING;
+}
+
+static int merge_visit_ret(int a, int b)
+{
+ if (a < 0)
+ return a;
+ if (b < 0)
+ return b;
+ if (a == KEEP_EXPLORING || b == KEEP_EXPLORING)
+ return KEEP_EXPLORING;
+ return DONE_EXPLORING;
+}
+
static int visit_func_call_insn(int t, struct bpf_insn *insns,
struct bpf_verifier_env *env,
bool visit_callee)
{
- int ret, insn_sz;
+ int ret, insn_sz, pad_ret;
int w;
+ pad_ret = visit_cleanup_pad_edge(t, env);
+ if (pad_ret < 0)
+ return pad_ret;
+
insn_sz = bpf_is_ldimm64(&insns[t]) ? 2 : 1;
ret = push_insn(t, t + insn_sz, FALLTHROUGH, env);
if (ret)
- return ret;
+ return merge_visit_ret(pad_ret, ret);
mark_prune_point(env, t + insn_sz);
/* when we exit from subprog, we need to record non-linear history */
@@ -180,7 +228,7 @@ static int visit_func_call_insn(int t, struct bpf_insn *insns,
merge_callee_effects(env, t, w);
ret = push_insn(t, w, BRANCH, env);
}
- return ret;
+ return merge_visit_ret(pad_ret, ret);
}
struct bpf_iarray *bpf_iarray_realloc(struct bpf_iarray *old, size_t n_elem)
@@ -592,6 +640,7 @@ int bpf_check_cfg(struct bpf_verifier_env *env)
int insn_cnt = env->prog->len;
int *insn_stack, *insn_state;
int ex_insn_beg, i, ret = 0;
+ u32 pad_idx = 0;
insn_state = env->cfg.insn_state = kvzalloc_objs(int, insn_cnt,
GFP_KERNEL_ACCOUNT);
@@ -647,6 +696,22 @@ int bpf_check_cfg(struct bpf_verifier_env *env)
goto walk_cfg;
}
+ /*
+ * A landing pad no call site was marked with -- a record whose range
+ * holds no call an exception can unwind out of -- is reached by
+ * nothing. Walk it from here, and let the dead code sweep remove it.
+ */
+ while (pad_idx < env->cleanup_info_cnt) {
+ u32 pad = env->cleanup_info[pad_idx++].landing_pad_off;
+
+ if (insn_state[pad] != EXPLORED) {
+ insn_state[pad] = DISCOVERED;
+ insn_stack[0] = pad;
+ env->cfg.cur_stack = 1;
+ goto walk_cfg;
+ }
+ }
+
for (i = 0; i < insn_cnt; i++) {
struct bpf_insn *insn = &env->prog->insnsi[i];
diff --git a/kernel/bpf/check_btf.c b/kernel/bpf/check_btf.c
index 0e8b3ccc7a5b9..d03dc791042a2 100644
--- a/kernel/bpf/check_btf.c
+++ b/kernel/bpf/check_btf.c
@@ -407,6 +407,149 @@ static int check_core_relo(struct bpf_verifier_env *env,
return err;
}
+static int cleanup_insn_subprog(struct bpf_verifier_env *env, u32 off)
+{
+ struct bpf_subprog_info *info;
+
+ if (off >= env->prog->len)
+ return -1;
+ info = bpf_find_containing_subprog(env, off);
+ return info ? info - env->subprog_info : -1;
+}
+
+#define MIN_BPF_CLEANUP_INFO_SIZE 12
+#define MAX_CLEANUP_INFO_REC_SIZE MAX_FUNCINFO_REC_SIZE
+
+static int check_cleanup_info(struct bpf_verifier_env *env,
+ const union bpf_attr *attr,
+ bpfptr_t uattr)
+{
+ u32 krec_size = sizeof(struct bpf_cleanup_info);
+ u32 i, nrec, urec_size, min_size, prev_end = 0;
+ struct bpf_cleanup_info *krecord;
+ bpfptr_t urecord;
+ int ret = -EINVAL;
+
+ nrec = attr->cleanup_info_cnt;
+ if (!nrec)
+ return 0;
+ if (nrec > INT_MAX / krec_size)
+ return -EINVAL;
+
+ urec_size = attr->cleanup_info_rec_size;
+ if (urec_size < MIN_BPF_CLEANUP_INFO_SIZE ||
+ urec_size > MAX_CLEANUP_INFO_REC_SIZE ||
+ urec_size % sizeof(u32)) {
+ verbose(env, "invalid cleanup info rec size %u\n", urec_size);
+ return -EINVAL;
+ }
+
+ krecord = kvcalloc(nrec, krec_size, GFP_KERNEL_ACCOUNT | __GFP_NOWARN);
+ if (!krecord)
+ return -ENOMEM;
+
+ min_size = min_t(u32, krec_size, urec_size);
+ urecord = make_bpfptr(attr->cleanup_info, uattr.is_kernel);
+ for (i = 0; i < nrec; i++) {
+ struct bpf_cleanup_info *rec = &krecord[i];
+ int sb, se, sl;
+
+ ret = bpf_check_uarg_tail_zero(urecord, krec_size, urec_size);
+ if (ret) {
+ if (ret == -E2BIG) {
+ verbose(env, "nonzero tailing record in cleanup info\n");
+ if (copy_to_bpfptr_offset(uattr,
+ offsetof(union bpf_attr,
+ cleanup_info_rec_size),
+ &min_size, sizeof(min_size)))
+ ret = -EFAULT;
+ }
+ goto err_free;
+ }
+
+ if (copy_from_bpfptr(rec, urecord, min_size)) {
+ ret = -EFAULT;
+ goto err_free;
+ }
+ bpfptr_add(&urecord, urec_size);
+
+ ret = -EINVAL;
+ if (rec->begin_off >= rec->end_off) {
+ verbose(env, "cleanup_info[%u]: begin %u >= end %u\n",
+ i, rec->begin_off, rec->end_off);
+ goto err_free;
+ }
+ if (i && rec->begin_off < prev_end) {
+ verbose(env,
+ "cleanup_info[%u]: range [%u,%u) is unsorted or overlaps the previous record\n",
+ i, rec->begin_off, rec->end_off);
+ goto err_free;
+ }
+ prev_end = rec->end_off;
+
+ sb = cleanup_insn_subprog(env, rec->begin_off);
+ se = cleanup_insn_subprog(env, rec->end_off - 1);
+ sl = cleanup_insn_subprog(env, rec->landing_pad_off);
+ if (sb < 0 || se < 0 || sl < 0) {
+ verbose(env, "cleanup_info[%u]: offset out of range\n", i);
+ goto err_free;
+ }
+ if (sb != se || sb != sl) {
+ verbose(env,
+ "cleanup_info[%u]: range/landing pad span multiple subprogs\n",
+ i);
+ goto err_free;
+ }
+ /*
+ * The second half of a 16-byte instruction carries a zero
+ * opcode and is not an instruction of its own, so no offset
+ * may name one. end_off is exclusive, so it may also be one
+ * past the last instruction of the program.
+ */
+ if (!env->prog->insnsi[rec->begin_off].code ||
+ !env->prog->insnsi[rec->landing_pad_off].code ||
+ (rec->end_off < env->prog->len &&
+ !env->prog->insnsi[rec->end_off].code)) {
+ verbose(env, "cleanup_info[%u]: points at invalid insn\n", i);
+ goto err_free;
+ }
+ }
+
+ /*
+ * Reject a landing pad that lies inside a call-site range, its own
+ * included: it would be both a pad and a call that unwinds to one, and
+ * an exception out of it would have nowhere to go.
+ */
+ ret = -EINVAL;
+ for (i = 0; i < nrec; i++) {
+ u32 pad = krecord[i].landing_pad_off;
+ u32 l = 0, r = nrec;
+
+ while (l < r) {
+ u32 m = l + (r - l) / 2;
+
+ if (pad < krecord[m].begin_off) {
+ r = m;
+ } else if (pad >= krecord[m].end_off) {
+ l = m + 1;
+ } else {
+ verbose(env,
+ "cleanup_info[%u]: landing pad %u is inside the call-site range of cleanup_info[%u]\n",
+ i, pad, m);
+ goto err_free;
+ }
+ }
+ }
+
+ env->cleanup_info = krecord;
+ env->cleanup_info_cnt = nrec;
+ return 0;
+
+err_free:
+ kvfree(krecord);
+ return ret;
+}
+
int bpf_prepare_btf_info(struct bpf_verifier_env *env,
const union bpf_attr *attr,
bpfptr_t uattr)
@@ -441,6 +584,10 @@ int bpf_check_btf_info(struct bpf_verifier_env *env,
{
int err;
+ err = check_cleanup_info(env, attr, uattr);
+ if (err)
+ return err;
+
if (!attr->func_info_cnt && !attr->line_info_cnt) {
if (check_abnormal_return(env))
return -EINVAL;
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 227211166dccf..f19712be12a7d 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -292,6 +292,7 @@ void __bpf_prog_free(struct bpf_prog *fp)
mutex_destroy(&fp->aux->dst_mutex);
mutex_destroy(&fp->aux->st_ops_assoc_mutex);
kfree(fp->aux->poke_tab);
+ bpf_cleanup_free_info(fp->aux);
kfree(fp->aux);
}
free_percpu(fp->stats);
@@ -2625,13 +2626,21 @@ static bool bpf_prog_select_interpreter(struct bpf_prog *fp)
return select_interpreter;
}
-static struct bpf_prog *bpf_prog_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog)
+static struct bpf_prog *bpf_prog_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog,
+ int *err)
{
#ifdef CONFIG_BPF_JIT
struct bpf_prog *orig_prog;
+ int ret;
- if (!bpf_prog_need_blind(prog))
+ if (!bpf_prog_need_blind(prog)) {
+ ret = bpf_cleanup_attach_main_prog(env, prog);
+ if (ret) {
+ *err = ret;
+ return prog;
+ }
return bpf_int_jit_compile(env, prog);
+ }
orig_prog = prog;
prog = bpf_jit_blind_constants(env, prog);
@@ -2642,6 +2651,13 @@ static struct bpf_prog *bpf_prog_jit_compile(struct bpf_verifier_env *env, struc
if (IS_ERR(prog))
goto out_restore;
+ ret = bpf_cleanup_attach_main_prog(env, prog);
+ if (ret) {
+ *err = ret;
+ bpf_jit_prog_release_other(orig_prog, prog);
+ goto out_restore;
+ }
+
prog = bpf_int_jit_compile(env, prog);
if (prog->jited) {
bpf_jit_prog_release_other(prog, orig_prog);
@@ -2681,8 +2697,10 @@ struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct
if (*err)
return fp;
- fp = bpf_prog_jit_compile(env, fp);
+ fp = bpf_prog_jit_compile(env, fp, err);
bpf_prog_jit_attempt_done(fp);
+ if (*err)
+ return fp;
if (!fp->jited && jit_needed) {
*err = -ENOTSUPP;
return fp;
@@ -3475,6 +3493,17 @@ void __weak arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp,
{
}
+bool __weak bpf_jit_supports_cleanup_pads(void)
+{
+ return false;
+}
+
+/* Call @pad with the frame pointer @frame_fp and r6-r9 spilled at @spill_base. */
+void __weak arch_bpf_run_cleanup_pad(u64 pad, u64 frame_fp, u64 spill_base)
+{
+ WARN_ON_ONCE(1);
+}
+
bool __weak bpf_jit_supports_timed_may_goto(void)
{
return false;
diff --git a/kernel/bpf/exception.c b/kernel/bpf/exception.c
new file mode 100644
index 0000000000000..d4dcbcf4fb257
--- /dev/null
+++ b/kernel/bpf/exception.c
@@ -0,0 +1,664 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <linux/bitmap.h>
+#include <linux/bpf.h>
+#include <linux/bpf_verifier.h>
+#include <linux/bsearch.h>
+#include <linux/btf.h>
+#include <linux/btf_ids.h>
+#include <linux/filter.h>
+#include <linux/slab.h>
+#include <linux/sort.h>
+#include "exception.h"
+
+#define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
+
+enum exc_kfunc {
+ EXC_KF_bpf_unwind_resume,
+};
+
+BTF_ID_LIST(exc_kfunc_list)
+BTF_ID(func, bpf_unwind_resume)
+
+static bool insn_is_exc_kfunc(const struct bpf_insn *insn, int kf)
+{
+ return bpf_pseudo_kfunc_call(insn) && insn->off == 0 &&
+ insn->imm == exc_kfunc_list[kf];
+}
+
+/* What an instruction does to intra-subprog control flow. */
+enum cleanup_insn_kind {
+ CLEANUP_INSN_PLAIN, /* the next insn runs */
+ CLEANUP_INSN_JUMP, /* unconditional jump */
+ CLEANUP_INSN_COND, /* the next insn runs, or the branch target */
+ CLEANUP_INSN_EXIT,
+ CLEANUP_INSN_THROW, /* call bpf_throw: nothing after it runs */
+ CLEANUP_INSN_RESUME, /* call bpf_unwind_resume: likewise */
+ CLEANUP_INSN_CALL, /* call to another subprog */
+ CLEANUP_INSN_GOTOX, /* indirect jump: successors not known here */
+};
+
+/* What each instruction can reach, computed once by cleanup_reachability(). */
+#define CLEANUP_REACH_RESUME BIT(0) /* a bpf_unwind_resume() call */
+#define CLEANUP_REACH_EXIT BIT(1) /* a plain BPF_EXIT */
+#define CLEANUP_REACH_UNKNOWN BIT(2) /* an indirect jump */
+#define CLEANUP_REACH_THROW BIT(3) /* a bpf_throw() call */
+
+/* Scratch shared by the analyses, sized once so no walker has to allocate. */
+struct cleanup_ctx {
+ struct bpf_verifier_env *env;
+ u8 *reach; /* per insn: CLEANUP_REACH_* mask */
+ u32 *stack; /* per insn: DFS stack */
+ void *scratch; /* the one allocation all of the above live in */
+};
+
+static bool in_pad(struct bpf_verifier_env *env, u32 i)
+{
+ return env->insn_aux_data[i].in_cleanup_pad;
+}
+
+/* One scratch array for cleanup_alloc() to hand out. */
+struct cleanup_alloc_req {
+ void **dst;
+ size_t n, sz;
+};
+
+static void *cleanup_alloc(const struct cleanup_alloc_req *tab, u32 cnt)
+{
+ size_t total = 0;
+ char *block, *p;
+ u32 i;
+
+ for (i = 0; i < cnt; i++)
+ total += round_up(tab[i].n * tab[i].sz, 8);
+
+ block = kvzalloc(total, GFP_KERNEL_ACCOUNT | __GFP_NOWARN);
+ if (!block)
+ return NULL;
+
+ for (i = 0, p = block; i < cnt; i++) {
+ *tab[i].dst = p;
+ p += round_up(tab[i].n * tab[i].sz, 8);
+ }
+ return block;
+}
+
+static int cleanup_subprog_of(struct bpf_verifier_env *env, u32 off)
+{
+ struct bpf_subprog_info *info = bpf_find_containing_subprog(env, off);
+
+ return info ? info - env->subprog_info : -1;
+}
+
+/* The subprogram a linear pass is currently in. */
+struct cleanup_cursor {
+ u32 start, end; /* [start, end) of the current subprogram */
+ int sub; /* its index */
+};
+
+#define CLEANUP_CURSOR_INIT { .sub = -1 }
+
+static void cleanup_cursor_to(struct bpf_verifier_env *env, struct cleanup_cursor *c, u32 i)
+{
+ while (i >= c->end) {
+ c->sub++;
+ c->start = env->subprog_info[c->sub].start;
+ c->end = env->subprog_info[c->sub + 1].start;
+ }
+}
+
+static enum cleanup_insn_kind cleanup_classify(struct bpf_verifier_env *env, u32 i,
+ int *next, int *target)
+{
+ struct bpf_insn *insn = &env->prog->insnsi[i];
+ u8 class = BPF_CLASS(insn->code);
+
+ *next = i + 1;
+ *target = -1;
+
+ if (insn->code == (BPF_LD | BPF_IMM | BPF_DW)) {
+ *next = i + 2;
+ return CLEANUP_INSN_PLAIN;
+ }
+ if (class != BPF_JMP && class != BPF_JMP32)
+ return CLEANUP_INSN_PLAIN;
+
+ switch (BPF_OP(insn->code)) {
+ case BPF_EXIT:
+ *next = -1;
+ return CLEANUP_INSN_EXIT;
+ case BPF_JA:
+ *next = -1;
+ if (BPF_SRC(insn->code) == BPF_X)
+ return CLEANUP_INSN_GOTOX;
+ *target = class == BPF_JMP32 ? i + insn->imm + 1 : i + insn->off + 1;
+ return CLEANUP_INSN_JUMP;
+ case BPF_CALL:
+ if (bpf_is_throw_kfunc(insn)) {
+ *next = -1;
+ return CLEANUP_INSN_THROW;
+ }
+ if (insn_is_exc_kfunc(insn, EXC_KF_bpf_unwind_resume)) {
+ *next = -1;
+ return CLEANUP_INSN_RESUME;
+ }
+ return bpf_pseudo_call(insn) ? CLEANUP_INSN_CALL : CLEANUP_INSN_PLAIN;
+ default:
+ /* Conditional jump, including BPF_JCOND. */
+ *target = i + insn->off + 1;
+ return CLEANUP_INSN_COND;
+ }
+}
+
+static void cleanup_mark_kfunc_sites(struct bpf_verifier_env *env)
+{
+ u32 i;
+
+ for (i = 0; i < env->prog->len; i++) {
+ struct bpf_insn *insn = &env->prog->insnsi[i];
+
+ if (bpf_is_throw_kfunc(insn))
+ env->insn_aux_data[i].cleanup_throw_site = true;
+ else if (insn_is_exc_kfunc(insn, EXC_KF_bpf_unwind_resume))
+ env->insn_aux_data[i].cleanup_resume_site = true;
+ }
+}
+
+int bpf_cleanup_check_callback(struct bpf_verifier_env *env, int subprog)
+{
+ if (!env->cleanup_info_cnt || !env->subprog_info[subprog].might_throw)
+ return 0;
+
+ verbose(env, "subprog %d may unwind and is used as a callback\n", subprog);
+ return -EINVAL;
+}
+
+/* Intra-subprog successors of @i, or -1 each when absent. */
+static enum cleanup_insn_kind cleanup_succ(struct bpf_verifier_env *env, u32 i,
+ u32 start, u32 end, int *next, int *target)
+{
+ enum cleanup_insn_kind kind = cleanup_classify(env, i, next, target);
+
+ if (*next < (int)start || *next >= (int)end)
+ *next = -1;
+ if (*target < (int)start || *target >= (int)end)
+ *target = -1;
+ return kind;
+}
+
+static void cleanup_add_pred(u32 *head, u32 *link, u32 to, u32 e)
+{
+ link[e] = head[to];
+ head[to] = e + 1;
+}
+
+/* What every instruction can reach along intra-subprog edges, for
+ * cleanup_pad_is_catch(). One backward walk over a predecessor index, rather
+ * than a forward walk from each landing pad, which would be quadratic.
+ */
+static int cleanup_reachability(struct cleanup_ctx *ctx)
+{
+ struct bpf_verifier_env *env = ctx->env;
+ u32 len = env->prog->len;
+ struct cleanup_cursor c = CLEANUP_CURSOR_INIT;
+ u32 *head = NULL, *link = NULL;
+ bool *queued = NULL;
+ u32 i, sp = 0;
+ void *scratch;
+ const struct cleanup_alloc_req tab[] = {
+ { (void **)&head, len, sizeof(*head) },
+ { (void **)&link, 2 * (size_t)len, sizeof(*link) },
+ { (void **)&queued, len, sizeof(*queued) },
+ };
+
+ scratch = cleanup_alloc(tab, ARRAY_SIZE(tab));
+ if (!scratch)
+ return -ENOMEM;
+
+ /* Index the predecessors, and seed the walk at the terminators. */
+ for (i = 0; i < len; i++) {
+ enum cleanup_insn_kind kind;
+ int next, target;
+
+ cleanup_cursor_to(env, &c, i);
+ kind = cleanup_succ(env, i, c.start, c.end, &next, &target);
+
+ if (kind == CLEANUP_INSN_RESUME)
+ ctx->reach[i] |= CLEANUP_REACH_RESUME;
+ else if (kind == CLEANUP_INSN_EXIT)
+ ctx->reach[i] |= CLEANUP_REACH_EXIT;
+ else if (kind == CLEANUP_INSN_GOTOX)
+ ctx->reach[i] |= CLEANUP_REACH_UNKNOWN;
+ else if (kind == CLEANUP_INSN_THROW)
+ ctx->reach[i] |= CLEANUP_REACH_THROW;
+
+ if (next >= 0)
+ cleanup_add_pred(head, link, next, 2 * i);
+ if (target >= 0)
+ cleanup_add_pred(head, link, target, 2 * i + 1);
+
+ if (ctx->reach[i]) {
+ queued[i] = true;
+ ctx->stack[sp++] = i;
+ }
+ }
+
+ /* Each instruction re-enters the worklist at most once per bit it
+ * gains, so this is linear in the number of edges.
+ */
+ while (sp) {
+ u32 j = ctx->stack[--sp];
+ u8 flags = ctx->reach[j];
+ u32 e;
+
+ queued[j] = false;
+ for (e = head[j]; e; e = link[e - 1]) {
+ u32 p = (e - 1) / 2;
+
+ if ((ctx->reach[p] | flags) == ctx->reach[p])
+ continue;
+ ctx->reach[p] |= flags;
+ if (!queued[p]) {
+ queued[p] = true;
+ ctx->stack[sp++] = p;
+ }
+ }
+ }
+ kvfree(scratch);
+ return 0;
+}
+
+static int cleanup_pad_is_catch(struct cleanup_ctx *ctx, u32 pad)
+{
+ u8 reach = ctx->reach[pad];
+
+ if (reach & CLEANUP_REACH_UNKNOWN) {
+ verbose(ctx->env, "cleanup landing pad %u reaches an indirect jump\n", pad);
+ return -EINVAL;
+ }
+ if (reach & CLEANUP_REACH_THROW) {
+ verbose(ctx->env,
+ "cleanup landing pad %u can throw while an exception is in flight\n",
+ pad);
+ return -EINVAL;
+ }
+ if (!(reach & CLEANUP_REACH_RESUME) == !(reach & CLEANUP_REACH_EXIT)) {
+ verbose(ctx->env, "cleanup landing pad %u %s\n", pad,
+ (reach & CLEANUP_REACH_RESUME) ?
+ "reaches both bpf_unwind_resume() and a plain exit" :
+ "reaches neither bpf_unwind_resume() nor an exit");
+ return -EINVAL;
+ }
+ return !!(reach & CLEANUP_REACH_EXIT);
+}
+
+static int cleanup_check_pad_insn(struct bpf_verifier_env *env, u32 i)
+{
+ struct bpf_insn *insn = &env->prog->insnsi[i];
+
+ if (bpf_helper_call(insn) && insn->imm == BPF_FUNC_tail_call) {
+ verbose(env,
+ "bpf_tail_call() at insn %u is in an exception cleanup landing pad\n",
+ i);
+ return -EINVAL;
+ }
+ /* A failed BPF_LD_[ABS|IND] leaves the frame through its epilogue. */
+ if (BPF_CLASS(insn->code) == BPF_LD &&
+ (BPF_MODE(insn->code) == BPF_ABS || BPF_MODE(insn->code) == BPF_IND)) {
+ verbose(env,
+ "BPF_LD_[ABS|IND] at insn %u is in an exception cleanup landing pad\n",
+ i);
+ return -EINVAL;
+ }
+ /* Stack arguments are not supported. */
+ if (is_stack_arg_st(insn) || is_stack_arg_stx(insn)) {
+ verbose(env,
+ "insn %u passes an on-stack call argument in an exception cleanup landing pad\n",
+ i);
+ return -EINVAL;
+ }
+ /* Likewise, stack arguments are not supported. */
+ if (bpf_pseudo_kfunc_call(insn)) {
+ struct bpf_call_summary cs;
+
+ if (bpf_get_call_summary(env, insn, &cs) &&
+ cs.arg_slot_cnt > MAX_BPF_FUNC_REG_ARGS) {
+ verbose(env,
+ "insn %u passes an on-stack call argument in an exception cleanup landing pad\n",
+ i);
+ return -EINVAL;
+ }
+ }
+ return 0;
+}
+
+static int cleanup_mark_pad_bodies(struct cleanup_ctx *ctx)
+{
+ struct bpf_verifier_env *env = ctx->env;
+ u32 i, sp = 0;
+ int ret;
+
+ for (i = 0; i < env->cleanup_info_cnt; i++) {
+ u32 pad = env->cleanup_info[i].landing_pad_off;
+
+ if (in_pad(env, pad))
+ continue;
+
+ ret = cleanup_pad_is_catch(ctx, pad);
+ if (ret < 0)
+ return ret;
+ if (ret) {
+ verbose(env,
+ "catch landing pad %u is not supported yet, only cleanup pads that resume\n",
+ pad);
+ return -EOPNOTSUPP;
+ }
+ env->insn_aux_data[pad].in_cleanup_pad = true;
+ ctx->stack[sp++] = pad;
+ }
+
+ while (sp) {
+ u32 j = ctx->stack[--sp];
+ enum cleanup_insn_kind kind;
+ int next, target, sub;
+ u32 start, end;
+
+ ret = cleanup_check_pad_insn(env, j);
+ if (ret)
+ return ret;
+
+ sub = cleanup_subprog_of(env, j);
+ start = env->subprog_info[sub].start;
+ end = env->subprog_info[sub + 1].start;
+ kind = cleanup_succ(env, j, start, end, &next, &target);
+
+ if (kind == CLEANUP_INSN_CALL) {
+ int callee = cleanup_subprog_of(env, j + env->prog->insnsi[j].imm + 1);
+
+ if (env->subprog_info[callee].might_throw) {
+ verbose(env,
+ "cleanup landing pad calls subprog %d at insn %u, which can throw while an exception is in flight\n",
+ callee, j);
+ return -EINVAL;
+ }
+ }
+
+ if (next >= 0 && !in_pad(env, next)) {
+ env->insn_aux_data[next].in_cleanup_pad = true;
+ ctx->stack[sp++] = next;
+ }
+ if (target >= 0 && !in_pad(env, target)) {
+ env->insn_aux_data[target].in_cleanup_pad = true;
+ ctx->stack[sp++] = target;
+ }
+ }
+ return 0;
+}
+
+static int cleanup_check_resumes(struct cleanup_ctx *ctx)
+{
+ struct bpf_verifier_env *env = ctx->env;
+ u32 i;
+
+ for (i = 0; i < env->prog->len; i++) {
+ if (!insn_is_exc_kfunc(&env->prog->insnsi[i], EXC_KF_bpf_unwind_resume))
+ continue;
+ if (in_pad(env, i))
+ continue;
+ verbose(env,
+ "bpf_unwind_resume() at insn %u is not in an exception cleanup landing pad\n",
+ i);
+ return -EINVAL;
+ }
+ return 0;
+}
+
+static void cleanup_mark_call_sites(struct bpf_verifier_env *env)
+{
+ u32 i, j;
+
+ for (i = 0; i < env->cleanup_info_cnt; i++) {
+ struct bpf_cleanup_info *rec = &env->cleanup_info[i];
+
+ for (j = rec->begin_off; j < rec->end_off; j++) {
+ struct bpf_insn *insn = &env->prog->insnsi[j];
+
+ if (!bpf_pseudo_call(insn) && !bpf_is_throw_kfunc(insn))
+ continue;
+ env->insn_aux_data[j].cleanup_pad = rec->landing_pad_off + 1;
+ }
+ }
+}
+
+int bpf_prepare_cleanup_exceptions(struct bpf_verifier_env *env)
+{
+ if (!env->cleanup_info_cnt)
+ return 0;
+
+ if (bpf_prog_is_offloaded(env->prog->aux)) {
+ verbose(env,
+ "exception cleanup is not supported for offloaded programs\n");
+ return -EINVAL;
+ }
+
+ if (!bpf_jit_supports_cleanup_pads() || !env->prog->jit_requested) {
+ verbose(env,
+ "exception cleanup needs a JIT that can dispatch landing pads\n");
+ return -EOPNOTSUPP;
+ }
+ env->prog->jit_required = 1;
+
+ if (env->exception_callback_subprog) {
+ verbose(env,
+ "exception cleanup table cannot be combined with an exception callback\n");
+ return -EINVAL;
+ }
+
+ cleanup_mark_kfunc_sites(env);
+ cleanup_mark_call_sites(env);
+ return 0;
+}
+
+int bpf_check_cleanup_exceptions(struct bpf_verifier_env *env)
+{
+ u32 len = env->prog->len;
+ struct cleanup_ctx ctx = { .env = env };
+ const struct cleanup_alloc_req tab[] = {
+ { (void **)&ctx.reach, len, sizeof(*ctx.reach) },
+ { (void **)&ctx.stack, len, sizeof(*ctx.stack) },
+ };
+ int ret;
+
+ if (!env->cleanup_info_cnt)
+ return 0;
+
+ ctx.scratch = cleanup_alloc(tab, ARRAY_SIZE(tab));
+ if (!ctx.scratch)
+ return -ENOMEM;
+
+ ret = cleanup_reachability(&ctx);
+ if (ret)
+ goto out;
+
+ ret = cleanup_mark_pad_bodies(&ctx);
+ if (ret)
+ goto out;
+
+ ret = cleanup_check_resumes(&ctx);
+out:
+ kvfree(ctx.scratch);
+ return ret;
+}
+
+bool bpf_is_unwind_resume_kfunc(const struct bpf_insn *insn)
+{
+ return insn_is_exc_kfunc(insn, EXC_KF_bpf_unwind_resume);
+}
+
+int bpf_cleanup_pad_of_call(struct bpf_verifier_env *env, u32 idx)
+{
+ u32 pad = env->insn_aux_data[idx].cleanup_pad;
+
+ return pad ? (int)pad - 1 : -1;
+}
+
+/*
+ * Every subprogram of a cleanup-carrying program spills the BPF callee-saved
+ * registers, even one that never throws: a frame's spill holds its caller's
+ * registers, and that is what the walker restores before running the caller's
+ * pad. The exception callback does not, because it reuses the boundary frame
+ * rather than building one of its own.
+ */
+bool bpf_cleanup_force_spill(const struct bpf_prog *prog)
+{
+ return prog->aux->exc && !prog->aux->exception_cb;
+}
+
+const struct bpf_cleanup_range *bpf_cleanup_pad_for_ip(const struct bpf_prog *prog, u64 ip)
+{
+ const struct bpf_exception_info *exc = prog->aux->exc;
+ u32 l = 0, r = exc ? exc->nr_ranges : 0;
+
+ while (l < r) {
+ u32 m = l + (r - l) / 2;
+ const struct bpf_cleanup_range *rec = &exc->ranges[m];
+
+ if (ip <= rec->begin)
+ r = m;
+ else if (ip > rec->end)
+ l = m + 1;
+ else
+ return rec;
+ }
+ return NULL;
+}
+
+static int cmp_u32(const void *a, const void *b)
+{
+ u32 x = *(const u32 *)a, y = *(const u32 *)b;
+
+ return x < y ? -1 : x > y;
+}
+
+int bpf_cleanup_alloc_info(struct bpf_prog_aux *aux)
+{
+ if (aux->exc)
+ return 0;
+ aux->exc = kzalloc_obj(struct bpf_exception_info, GFP_KERNEL_ACCOUNT | __GFP_NOWARN);
+ return aux->exc ? 0 : -ENOMEM;
+}
+
+int bpf_cleanup_attach_info(struct bpf_prog_aux *aux, struct bpf_cleanup_info *recs, u32 cnt)
+{
+ struct bpf_exception_info *exc = aux->exc;
+ struct bpf_cleanup_range *ranges;
+ u32 i, n_at, *at;
+
+ if (!cnt) {
+ kvfree(recs);
+ return 0;
+ }
+
+ ranges = kvcalloc(cnt, sizeof(*ranges), GFP_KERNEL_ACCOUNT | __GFP_NOWARN);
+ if (!ranges) {
+ kvfree(recs);
+ return -ENOMEM;
+ }
+
+ /* The pads on their own, sorted and deduplicated. */
+ at = kvmalloc_array(cnt, sizeof(*at), GFP_KERNEL_ACCOUNT | __GFP_NOWARN);
+ if (!at) {
+ kvfree(ranges);
+ kvfree(recs);
+ return -ENOMEM;
+ }
+ for (i = 0; i < cnt; i++)
+ at[i] = recs[i].landing_pad_off;
+ sort(at, cnt, sizeof(*at), cmp_u32, NULL);
+ for (i = 0, n_at = 0; i < cnt; i++)
+ if (!n_at || at[n_at - 1] != at[i])
+ at[n_at++] = at[i];
+
+ exc->pad_at = at;
+ exc->nr_pad_at = n_at;
+ exc->info = recs;
+ exc->nr_info = cnt;
+ exc->ranges = ranges;
+ /* Withheld until the JIT has filled the table in. */
+ exc->nr_ranges = 0;
+ return 0;
+}
+
+void bpf_cleanup_fill_native_ranges(struct bpf_prog *prog, u32 *addrs, void *image)
+{
+ struct bpf_exception_info *exc = prog->aux->exc;
+ u32 i, n;
+
+ if (!exc || !exc->nr_info || !exc->ranges)
+ return;
+
+ n = exc->nr_info;
+ for (i = 0; i < n; i++) {
+ const struct bpf_cleanup_info *rec = &exc->info[i];
+
+ if (WARN_ON_ONCE(rec->begin_off >= prog->len ||
+ rec->end_off > prog->len ||
+ rec->landing_pad_off >= prog->len))
+ return;
+ exc->ranges[i].begin = (u64)(long)image + addrs[rec->begin_off];
+ exc->ranges[i].end = (u64)(long)image + addrs[rec->end_off];
+ exc->ranges[i].pad = (u64)(long)image + addrs[rec->landing_pad_off];
+ }
+ exc->nr_ranges = n;
+}
+
+void bpf_cleanup_free_info(struct bpf_prog_aux *aux)
+{
+ struct bpf_exception_info *exc = aux->exc;
+
+ if (!exc)
+ return;
+ kvfree(exc->ranges);
+ kvfree(exc->info);
+ kvfree(exc->pad_at);
+ kvfree(exc->throw_at);
+ kvfree(exc->resume_at);
+ bitmap_free(exc->pad_body);
+ kfree(exc);
+ aux->exc = NULL;
+}
+
+/* Is @idx in the sorted array @at of @n instruction indices? */
+static bool insn_idx_in(const u32 *at, u32 n, u32 idx)
+{
+ return bsearch(&idx, at, n, sizeof(*at), cmp_u32);
+}
+
+bool bpf_cleanup_insn_is_pad(const struct bpf_prog *prog, u32 idx)
+{
+ const struct bpf_exception_info *exc = prog->aux->exc;
+
+ return exc && insn_idx_in(exc->pad_at, exc->nr_pad_at, idx);
+}
+
+bool bpf_cleanup_insn_is_throw(const struct bpf_prog *prog, u32 idx)
+{
+ const struct bpf_exception_info *exc = prog->aux->exc;
+
+ return exc && insn_idx_in(exc->throw_at, exc->nr_throw_at, idx);
+}
+
+bool bpf_cleanup_insn_is_resume(const struct bpf_prog *prog, u32 idx)
+{
+ const struct bpf_exception_info *exc = prog->aux->exc;
+
+ return exc && insn_idx_in(exc->resume_at, exc->nr_resume_at, idx);
+}
+
+bool bpf_cleanup_insn_in_pad(const struct bpf_prog *prog, u32 idx)
+{
+ const struct bpf_exception_info *exc = prog->aux->exc;
+
+ return exc && exc->pad_body && idx < exc->nr_pad_body &&
+ test_bit(idx, exc->pad_body);
+}
diff --git a/kernel/bpf/exception.h b/kernel/bpf/exception.h
new file mode 100644
index 0000000000000..c0e68ce227c83
--- /dev/null
+++ b/kernel/bpf/exception.h
@@ -0,0 +1,22 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#ifndef _LINUX_BPF_EXCEPTION_H
+#define _LINUX_BPF_EXCEPTION_H
+
+#include <linux/types.h>
+
+struct bpf_cleanup_info;
+struct bpf_cleanup_range;
+struct bpf_prog;
+struct bpf_prog_aux;
+struct bpf_verifier_env;
+
+int bpf_prepare_cleanup_exceptions(struct bpf_verifier_env *env);
+int bpf_check_cleanup_exceptions(struct bpf_verifier_env *env);
+int bpf_cleanup_check_callback(struct bpf_verifier_env *env, int subprog);
+int bpf_cleanup_pad_of_call(struct bpf_verifier_env *env, u32 idx);
+int bpf_cleanup_alloc_info(struct bpf_prog_aux *aux);
+int bpf_cleanup_attach_info(struct bpf_prog_aux *aux, struct bpf_cleanup_info *recs, u32 cnt);
+const struct bpf_cleanup_range *bpf_cleanup_pad_for_ip(const struct bpf_prog *prog, u64 ip);
+
+#endif /* _LINUX_BPF_EXCEPTION_H */
diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
index 2add8001c3ec3..169cbf9f44159 100644
--- a/kernel/bpf/fixups.c
+++ b/kernel/bpf/fixups.c
@@ -1,5 +1,6 @@
// SPDX-License-Identifier: GPL-2.0-only
/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <linux/bitmap.h>
#include <linux/bpf.h>
#include <linux/btf.h>
#include <linux/bpf_verifier.h>
@@ -10,6 +11,7 @@
#include <linux/perf_event.h>
#include <net/xdp.h>
#include "disasm.h"
+#include "exception.h"
#define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
@@ -252,15 +254,29 @@ static void adjust_insn_aux_data(struct bpf_verifier_env *env,
/* Expand insni[off]'s seen count to the patched range. */
data[i].seen = old_seen;
data[i].zext_dst = bpf_insn_def32(new_prog, insn + i) >= 0;
+ data[i].in_cleanup_pad = data[off + cnt - 1].in_cleanup_pad;
if (!memcmp(insn + i, original_insn, sizeof(struct bpf_insn))) {
data[i].non_stack_access =
data[off + cnt - 1].non_stack_access;
data[off + cnt - 1].non_stack_access = false;
+ data[i].cleanup_throw_site =
+ data[off + cnt - 1].cleanup_throw_site;
+ data[off + cnt - 1].cleanup_throw_site = false;
+ data[i].cleanup_resume_site =
+ data[off + cnt - 1].cleanup_resume_site;
+ data[off + cnt - 1].cleanup_resume_site = false;
+ data[i].cleanup_pad = data[off + cnt - 1].cleanup_pad;
+ data[off + cnt - 1].cleanup_pad = 0;
} else if (bpf_is_mem_insn(insn + i)) {
data[i].non_stack_access = true;
}
}
+ if (env->cleanup_info_cnt)
+ for (i = 0; i < prog_len; i++)
+ if (data[i].cleanup_pad > off + 1)
+ data[i].cleanup_pad += cnt - 1;
+
/*
* Last slot instruction could be a newly generated
* BPF_ST/BPF_LDX/BPF_STX, systematically mark it for non-stack access
@@ -544,11 +560,13 @@ void bpf_clear_insn_aux_data(struct bpf_verifier_env *env, int start, int len)
}
}
-static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt)
+static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt,
+ bool falls_through)
{
struct bpf_insn_aux_data *aux_data = env->insn_aux_data;
unsigned int orig_prog_len = env->prog->len;
int err;
+ u32 i;
if (bpf_prog_is_offloaded(env->prog->aux))
bpf_prog_offload_remove_insns(env, off, cnt);
@@ -573,6 +591,17 @@ static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt)
sizeof(*aux_data) * (orig_prog_len - off - cnt));
env->insn_aux_data_len -= cnt;
+ if (env->cleanup_info_cnt) {
+ for (i = 0; i < env->insn_aux_data_len; i++) {
+ u32 pad = aux_data[i].cleanup_pad;
+
+ if (pad > off + cnt)
+ aux_data[i].cleanup_pad = pad - cnt;
+ else if (pad > off)
+ aux_data[i].cleanup_pad = falls_through ? off + 1 : 0;
+ }
+ }
+
return 0;
}
@@ -634,7 +663,7 @@ int bpf_opt_remove_dead_code(struct bpf_verifier_env *env)
if (!j)
continue;
- err = verifier_remove_insns(env, i, j);
+ err = verifier_remove_insns(env, i, j, false);
if (err)
return err;
insn_cnt = env->prog->len;
@@ -657,7 +686,7 @@ int bpf_opt_remove_nops(struct bpf_verifier_env *env)
if (!is_may_goto_0 && !is_ja)
continue;
- err = verifier_remove_insns(env, i, 1);
+ err = verifier_remove_insns(env, i, 1, true);
if (err)
return err;
insn_cnt--;
@@ -1095,6 +1124,129 @@ static void bpf_restore_subprog_starts(struct bpf_verifier_env *env, u32 *orig_s
env->subprog_info[env->subprog_cnt].start = env->prog->len;
}
+static bool cleanup_kfunc_site(const struct bpf_insn_aux_data *aux, bool resume)
+{
+ return resume ? aux->cleanup_resume_site : aux->cleanup_throw_site;
+}
+
+static int cleanup_kfunc_sites_for_subprog(struct bpf_verifier_env *env, u32 start, u32 end,
+ bool resume, u32 **at_p, u32 *nr_p)
+{
+ u32 i, cnt = 0, *at;
+
+ for (i = start; i < end; i++)
+ if (cleanup_kfunc_site(&env->insn_aux_data[i], resume))
+ cnt++;
+ if (!cnt)
+ return 0;
+
+ at = kvmalloc_array(cnt, sizeof(*at), GFP_KERNEL_ACCOUNT | __GFP_NOWARN);
+ if (!at)
+ return -ENOMEM;
+
+ for (i = start, cnt = 0; i < end; i++) {
+ if (!cleanup_kfunc_site(&env->insn_aux_data[i], resume))
+ continue;
+ at[cnt++] = i - start;
+ }
+
+ *at_p = at;
+ *nr_p = cnt;
+ return 0;
+}
+
+static int cleanup_pad_body_for_subprog(struct bpf_verifier_env *env, struct bpf_prog *sub,
+ u32 start, u32 end)
+{
+ unsigned long *bits;
+ u32 i, cnt = 0;
+
+ for (i = start; i < end; i++)
+ if (env->insn_aux_data[i].in_cleanup_pad)
+ cnt++;
+ if (!cnt)
+ return 0;
+
+ bits = bitmap_zalloc(end - start, GFP_KERNEL_ACCOUNT | __GFP_NOWARN);
+ if (!bits)
+ return -ENOMEM;
+
+ for (i = start; i < end; i++)
+ if (env->insn_aux_data[i].in_cleanup_pad)
+ __set_bit(i - start, bits);
+
+ sub->aux->exc->pad_body = bits;
+ sub->aux->exc->nr_pad_body = end - start;
+ return 0;
+}
+
+static int cleanup_info_for_subprog(struct bpf_verifier_env *env, struct bpf_prog *sub,
+ u32 start, u32 end)
+{
+ struct bpf_cleanup_info *recs;
+ u32 i, cnt = 0;
+ int err;
+
+ if (!env->cleanup_info_cnt)
+ return 0;
+
+ err = bpf_cleanup_alloc_info(sub->aux);
+ if (err)
+ return err;
+
+ err = cleanup_kfunc_sites_for_subprog(env, start, end, false,
+ &sub->aux->exc->throw_at,
+ &sub->aux->exc->nr_throw_at);
+ if (err)
+ return err;
+
+ err = cleanup_kfunc_sites_for_subprog(env, start, end, true,
+ &sub->aux->exc->resume_at,
+ &sub->aux->exc->nr_resume_at);
+ if (err)
+ return err;
+
+ err = cleanup_pad_body_for_subprog(env, sub, start, end);
+ if (err)
+ return err;
+
+ for (i = start; i < end; i++)
+ if (env->insn_aux_data[i].cleanup_pad)
+ cnt++;
+ if (!cnt)
+ return 0;
+
+ recs = kvmalloc_array(cnt, sizeof(*recs), GFP_KERNEL_ACCOUNT | __GFP_NOWARN);
+ if (!recs)
+ return -ENOMEM;
+
+ for (i = start, cnt = 0; i < end; i++) {
+ u32 pad = env->insn_aux_data[i].cleanup_pad;
+
+ if (!pad)
+ continue;
+ pad--;
+ if (verifier_bug_if(pad < start || pad >= end, env,
+ "insn %u is covered by a landing pad at %u outside its subprog [%u, %u)",
+ i, pad, start, end)) {
+ kvfree(recs);
+ return -EFAULT;
+ }
+ recs[cnt].begin_off = i - start;
+ recs[cnt].end_off = i - start + 1;
+ recs[cnt].landing_pad_off = pad - start;
+ cnt++;
+ }
+ return bpf_cleanup_attach_info(sub->aux, recs, cnt);
+}
+
+int bpf_cleanup_attach_main_prog(struct bpf_verifier_env *env, struct bpf_prog *prog)
+{
+ if (!env || env->subprog_cnt > 1)
+ return 0;
+ return cleanup_info_for_subprog(env, prog, 0, prog->len);
+}
+
static int jit_subprogs(struct bpf_verifier_env *env)
{
struct bpf_prog *prog = env->prog, **func, *tmp;
@@ -1232,6 +1384,10 @@ static int jit_subprogs(struct bpf_verifier_env *env)
func[i]->aux->token = prog->aux->token;
if (!i)
func[i]->aux->exception_boundary = env->seen_exception;
+ err = cleanup_info_for_subprog(env, func[i], subprog_start,
+ subprog_end);
+ if (err)
+ goto out_free;
func[i] = bpf_int_jit_compile(env, func[i]);
if (!func[i]->jited) {
err = -ENOTSUPP;
@@ -1336,6 +1492,8 @@ static int jit_subprogs(struct bpf_verifier_env *env)
prog->aux->bpf_exception_cb = (void *)func[env->exception_callback_subprog]->bpf_func;
prog->aux->exception_boundary = func[0]->aux->exception_boundary;
prog->aux->stack_arg_sp_adjust = func[0]->aux->stack_arg_sp_adjust;
+ prog->aux->exc = func[0]->aux->exc;
+ func[0]->aux->exc = NULL;
bpf_prog_jit_attempt_done(prog);
return 0;
out_free:
@@ -1916,6 +2074,8 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)
goto next_insn;
if (insn->src_reg == BPF_PSEUDO_CALL)
goto next_insn;
+ if (env->insn_aux_data[i + delta].cleanup_resume_site)
+ goto next_insn;
if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL) {
ret = bpf_fixup_kfunc_call(env, insn, insn_buf, i + delta, &cnt);
if (ret)
diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c
index 82402d97ce67c..d639dfbc3cbbf 100644
--- a/kernel/bpf/helpers.c
+++ b/kernel/bpf/helpers.c
@@ -31,6 +31,7 @@
#include <linux/buildid.h>
#include "../../lib/kstrtox.h"
+#include "exception.h"
/* If kernel subsystem is allowing eBPF programs to call this function,
* inside its own verifier_ops->get_func_proto() callback it should return
@@ -3398,8 +3399,36 @@ struct bpf_throw_ctx {
u64 sp;
u64 bp;
int cnt;
+ const struct bpf_prog *callee;
+ u64 callee_fp;
};
+static void bpf_run_cleanup_pad(struct bpf_throw_ctx *ctx, const struct bpf_prog *prog,
+ u64 ip, u64 fp)
+{
+ const struct bpf_exception_info *exc = prog->aux->exc;
+ const struct bpf_cleanup_range *rec;
+ u64 spill_base;
+
+ if (!exc || !exc->nr_ranges)
+ return;
+ rec = bpf_cleanup_pad_for_ip(prog, ip);
+ if (!rec)
+ return;
+
+ /*
+ * The callee is always another subprogram of this program -- the walk
+ * ends at any frame that is not one -- so its prologue spilled these
+ * registers and its exc is there to say where.
+ */
+ if (ctx->callee)
+ spill_base = ctx->callee_fp + ctx->callee->aux->exc->spill_off;
+ else
+ spill_base = fp + exc->throw_spill_off;
+
+ arch_bpf_run_cleanup_pad(rec->pad, fp, spill_base);
+}
+
static bool bpf_stack_walker(void *cookie, u64 ip, u64 sp, u64 bp)
{
struct bpf_throw_ctx *ctx = cookie;
@@ -3416,6 +3445,11 @@ static bool bpf_stack_walker(void *cookie, u64 ip, u64 sp, u64 bp)
if (!prog)
return !ctx->cnt;
ctx->cnt++;
+
+ bpf_run_cleanup_pad(ctx, prog, ip, bp);
+ ctx->callee = prog;
+ ctx->callee_fp = bp;
+
if (bpf_is_subprog(prog))
return true;
ctx->aux = prog->aux;
@@ -3445,6 +3479,17 @@ __bpf_kfunc void bpf_throw(u64 cookie)
WARN(1, "A call to BPF exception callback should never return\n");
}
+/*
+ * Terminator of a compiler-emitted cleanup landing pad. The compiler names
+ * this _Unwind_Resume, the base unwind ABI's entry point for carrying an
+ * unwind on once a frame's cleanups have run. To match kernel kfunc
+ * convention, the kernel calls it bpf_unwind_resume and libbpf maps the
+ * compiler's name onto it.
+ */
+__bpf_kfunc void bpf_unwind_resume(void)
+{
+}
+
__bpf_kfunc int bpf_wq_init(struct bpf_wq *wq, void *p__const_map, unsigned int flags)
{
struct bpf_async_kern *async = (struct bpf_async_kern *)wq;
@@ -4981,6 +5026,7 @@ BTF_ID_FLAGS(func, bpf_task_get_cgroup1, KF_ACQUIRE | KF_RCU | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_task_from_pid, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_task_from_vpid, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_throw)
+BTF_ID_FLAGS(func, bpf_unwind_resume)
#ifdef CONFIG_BPF_EVENTS
BTF_ID_FLAGS(func, bpf_send_signal_task)
#endif
diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index 44ecdc5b4ec2d..9cfd05f970bc9 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -8,6 +8,8 @@
#include <linux/slab.h>
#include <linux/sort.h>
+#include "exception.h"
+
#define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
struct per_frame_masks {
@@ -256,6 +258,9 @@ bpf_insn_successors(struct bpf_verifier_env *env, u32 idx)
succ = env->succ;
succ->cnt = 0;
+ if (unlikely(bpf_is_unwind_resume_kfunc(insn)))
+ return succ;
+
opcode_info = &opcode_info_tbl[BPF_CLASS(insn->code) | BPF_OP(insn->code)];
insn_sz = bpf_is_ldimm64(insn) ? 2 : 1;
if (opcode_info->can_fallthrough)
@@ -264,6 +269,13 @@ bpf_insn_successors(struct bpf_verifier_env *env, u32 idx)
if (opcode_info->can_jump)
succ->items[succ->cnt++] = idx + bpf_jmp_offset(insn) + 1;
+ if (unlikely(env->cleanup_info_cnt)) {
+ int pad = bpf_cleanup_pad_of_call(env, idx);
+
+ if (pad >= 0)
+ succ->items[succ->cnt++] = pad;
+ }
+
return succ;
}
@@ -397,6 +409,14 @@ bool bpf_stack_slot_alive(struct bpf_verifier_env *env, u32 frameno, u32 half_sp
alive = bpf_calls_callback(env, callsite)
? is_live_before(instance, callsite, rel, half_spi)
: is_live_before(instance, callsite + 1, rel, half_spi);
+
+ /* Control may also go to the landing pad. */
+ if (!alive && unlikely(env->cleanup_info_cnt)) {
+ int pad = bpf_cleanup_pad_of_call(env, callsite);
+
+ if (pad >= 0)
+ alive = is_live_before(instance, pad, rel, half_spi);
+ }
if (alive)
return true;
}
diff --git a/kernel/bpf/states.c b/kernel/bpf/states.c
index 66fb11b6c6a76..b101baa43171c 100644
--- a/kernel/bpf/states.c
+++ b/kernel/bpf/states.c
@@ -996,6 +996,12 @@ static bool states_equal(struct bpf_verifier_env *env,
if (old->in_sleepable != cur->in_sleepable)
return false;
+ if (old->unwinding != cur->unwinding)
+ return false;
+
+ if (old->unwinding && old->unwind_frameno != cur->unwind_frameno)
+ return false;
+
if (!refsafe(old, cur, &env->idmap_scratch))
return false;
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 113486b15d29f..049717dc3b7d7 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -2915,7 +2915,7 @@ int __init __used bpf_multi_func(void) { return 0; }
BTF_ID_LIST_GLOBAL_SINGLE(bpf_multi_func_btf_id, func, bpf_multi_func)
/* last field in 'union bpf_attr' used by this command */
-#define BPF_PROG_LOAD_LAST_FIELD keyring_id
+#define BPF_PROG_LOAD_LAST_FIELD cleanup_info_cnt
static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, struct bpf_log_attr *attr_log)
{
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 18aad4886f9cf..b978e710d7b66 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -10,6 +10,7 @@
#include <linux/slab.h>
#include <linux/bpf.h>
#include <linux/btf.h>
+#include <linux/bpf_cleanup_abi.h>
#include <linux/bpf_verifier.h>
#include <linux/filter.h>
#include <net/netlink.h>
@@ -37,6 +38,7 @@
#include "diagnostics.h"
#include "disasm.h"
+#include "exception.h"
static const struct bpf_verifier_ops * const bpf_verifier_ops[] = {
#define BPF_PROG_TYPE(_id, _name, prog_ctx_type, kern_ctx_type) \
@@ -1714,6 +1716,8 @@ int bpf_copy_verifier_state(struct bpf_verifier_state *dst_state,
return err;
dst_state->speculative = src->speculative;
dst_state->in_sleepable = src->in_sleepable;
+ dst_state->unwinding = src->unwinding;
+ dst_state->unwind_frameno = src->unwind_frameno;
dst_state->curframe = src->curframe;
dst_state->branches = src->branches;
dst_state->parent = src->parent;
@@ -5588,6 +5592,14 @@ static int check_max_stack_depth(struct bpf_verifier_env *env)
}
}
+ /*
+ * A pad rebuilds its frame from a spill area, and on x86-64 a private
+ * stack's frame pointer lives in r9, which no spill area holds.
+ * Refused on every arch rather than just that one.
+ */
+ if (env->cleanup_info_cnt)
+ priv_stack_mode = NO_PRIV_STACK;
+
if (priv_stack_mode == PRIV_STACK_UNKNOWN)
priv_stack_mode = bpf_enable_priv_stack(env->prog);
@@ -10502,6 +10514,10 @@ static int push_callback_call(struct bpf_verifier_env *env, struct bpf_insn *ins
* callbacks
*/
env->subprog_info[subprog].is_cb = true;
+ err = bpf_cleanup_check_callback(env, subprog);
+ if (err)
+ return err;
+
if (bpf_pseudo_kfunc_call(insn) &&
!is_callback_calling_kfunc(insn->imm)) {
verifier_bug(env, "kfunc %s#%d not marked as callback-calling",
@@ -10553,8 +10569,8 @@ static int push_callback_call(struct bpf_verifier_env *env, struct bpf_insn *ins
return 0;
}
-static int process_bpf_exit_full(struct bpf_verifier_env *env,
- bool *do_print_state, bool exception_exit);
+static int process_bpf_exit_full(struct bpf_verifier_env *env, bool *do_print_state);
+static int unwind_step(struct bpf_verifier_env *env, u32 callsite, int *insn_idx);
static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
int *insn_idx)
@@ -10644,7 +10660,7 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
verbose(env, "failed to push state for global subprog exception path\n");
return PTR_ERR(branch);
}
- return process_bpf_exit_full(env, NULL, true);
+ return unwind_step(env, *insn_idx, insn_idx);
}
/* continue with next insn after call */
@@ -14545,7 +14561,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
env->prog->call_session_cookie = true;
if (bpf_is_throw_kfunc(insn))
- return process_bpf_exit_full(env, NULL, true);
+ return unwind_step(env, insn_idx, &env->insn_idx);
return 0;
}
@@ -18470,9 +18486,103 @@ enum {
INSN_IDX_UPDATED = 2,
};
-static int process_bpf_exit_full(struct bpf_verifier_env *env,
- bool *do_print_state,
- bool exception_exit)
+static u32 unwind_pop_frame(struct bpf_verifier_env *env)
+{
+ struct bpf_verifier_state *state = env->cur_state;
+ struct bpf_func_state *callee = state->frame[state->curframe];
+ u32 callsite = callee->callsite;
+ struct bpf_func_state *caller;
+
+ caller = state->frame[state->curframe - 1];
+ account_processed_insns(env, callee, caller);
+ free_func_state(callee);
+ state->frame[state->curframe--] = NULL;
+ invalidate_outgoing_stack_args(env, caller);
+ return callsite;
+}
+
+static void unwind_enter_pad(struct bpf_verifier_env *env)
+{
+ struct bpf_verifier_state *state = env->cur_state;
+ struct bpf_func_state *frame = cur_func(env);
+
+ state->unwind_frameno = state->curframe;
+ clear_caller_saved_regs(env, frame->regs);
+ mark_reg_unknown(env, frame->regs, BPF_REG_0);
+ __mark_reg_known(&frame->regs[BPF_REG_0], BPF_PAD_ENTRY_R0);
+}
+
+static int unwind_finish(struct bpf_verifier_env *env)
+{
+ int err = check_resource_leak(env, true, true, "bpf_throw");
+
+ if (err)
+ return err;
+ return PROCESS_BPF_EXIT;
+}
+
+static int unwind_step(struct bpf_verifier_env *env, u32 callsite, int *insn_idx)
+{
+ struct bpf_verifier_state *state = env->cur_state;
+ u8 popped = 0;
+
+ state->unwinding = true;
+ for (;;) {
+ int pad = bpf_cleanup_pad_of_call(env, callsite);
+
+ if (pad >= 0) {
+ unwind_enter_pad(env);
+ /*
+ * The edge from @callsite to the pad crosses @popped
+ * frames, and nothing in the instruction stream says
+ * so. Record it for mark_chain_precision(), which has
+ * to walk back through the same frames.
+ */
+ env->unwind_frames = popped;
+ *insn_idx = pad;
+ return INSN_IDX_UPDATED;
+ }
+ if (!state->curframe)
+ return unwind_finish(env);
+ callsite = unwind_pop_frame(env);
+ popped++;
+ }
+}
+
+static int process_cleanup_resume(struct bpf_verifier_env *env, int *insn_idx)
+{
+ struct bpf_verifier_state *state = env->cur_state;
+ int err;
+
+ /* A pad entered by ordinary control flow. */
+ if (!state->unwinding) {
+ verbose(env,
+ "bpf_unwind_resume() at insn %d reached without an exception in flight\n",
+ *insn_idx);
+ return -EINVAL;
+ }
+ /*
+ * A resume in a subprogram the pad called, which has a landing pad of
+ * its own and reached it by ordinary control flow. A JIT lowers a
+ * resume as the way back out of a pad, which reaches the walker only
+ * from the frame whose pad it called.
+ */
+ if (state->curframe != state->unwind_frameno) {
+ verbose(env,
+ "bpf_unwind_resume() at insn %d is in frame %d, not frame %d whose landing pad the exception entered\n",
+ *insn_idx, state->curframe, state->unwind_frameno);
+ return -EINVAL;
+ }
+ if (!state->curframe)
+ return unwind_finish(env);
+ err = unwind_step(env, unwind_pop_frame(env), insn_idx);
+ /* unwind_step() counted the frames it popped, not this one. */
+ if (err == INSN_IDX_UPDATED)
+ env->unwind_frames++;
+ return err;
+}
+
+static int process_bpf_exit_full(struct bpf_verifier_env *env, bool *do_print_state)
{
struct bpf_func_state *cur_frame = cur_func(env);
@@ -18482,25 +18592,11 @@ static int process_bpf_exit_full(struct bpf_verifier_env *env,
* for which reference_state must match caller reference
* state when it exits.
*/
- int err = check_resource_leak(env, exception_exit,
- exception_exit || !env->cur_state->curframe,
- exception_exit ? "bpf_throw" :
+ int err = check_resource_leak(env, false, !env->cur_state->curframe,
"BPF_EXIT instruction in main prog");
if (err)
return err;
- /* The side effect of the prepare_func_exit which is
- * being skipped is that it frees bpf_func_state.
- * Typically, process_bpf_exit will only be hit with
- * outermost exit. copy_verifier_state in pop_stack will
- * handle freeing of any extra bpf_func_state left over
- * from not processing all nested function exits. We
- * also skip return code checks as they are not needed
- * for exceptional exits.
- */
- if (exception_exit)
- return PROCESS_BPF_EXIT;
-
if (env->cur_state->curframe) {
/* exit from nested function */
err = prepare_func_exit(env, &env->insn_idx);
@@ -18674,6 +18770,8 @@ static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state)
env->jmps_processed++;
if (opcode == BPF_CALL) {
+ if (bpf_is_unwind_resume_kfunc(insn))
+ return process_cleanup_resume(env, &env->insn_idx);
if (env->cur_state->active_locks) {
if ((insn->src_reg == BPF_REG_0 &&
insn->imm != BPF_FUNC_spin_unlock &&
@@ -18707,7 +18805,7 @@ static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state)
env->insn_idx += insn->imm + 1;
return INSN_IDX_UPDATED;
} else if (opcode == BPF_EXIT) {
- return process_bpf_exit_full(env, do_print_state, false);
+ return process_bpf_exit_full(env, do_print_state);
}
return check_cond_jmp_op(env, insn, &env->insn_idx);
}
@@ -18744,10 +18842,14 @@ static int do_check(struct bpf_verifier_env *env)
for (;;) {
struct bpf_insn *insn;
struct bpf_insn_aux_data *insn_aux;
+ u8 unwind_frames;
int err;
/* reset current history entry on each new instruction */
env->cur_hist_ent = NULL;
+ /* frames the unwind popped to reach this insn, if it is a pad */
+ unwind_frames = env->unwind_frames;
+ env->unwind_frames = 0;
env->prev_insn_idx = prev_insn_idx;
if (env->insn_idx >= insn_cnt) {
@@ -18811,10 +18913,16 @@ static int do_check(struct bpf_verifier_env *env)
}
}
- if (bpf_is_jmp_point(env, env->insn_idx)) {
+ /*
+ * A landing pad is a jump point, but record the edge even if
+ * that ever stops being true: it is the only place the frames
+ * the unwind popped are written down.
+ */
+ if (bpf_is_jmp_point(env, env->insn_idx) || unwind_frames) {
err = bpf_push_jmp_history(env, state, 0, 0, 0, 0);
if (err)
return err;
+ env->cur_hist_ent->unwind_frames = unwind_frames;
}
if (signal_pending(current))
@@ -21677,6 +21785,11 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
if (ret < 0)
goto skip_full_check;
+ /* The CFG needs an edge from a call in a cleanup range to its pad. */
+ ret = bpf_prepare_cleanup_exceptions(env);
+ if (ret < 0)
+ goto skip_full_check;
+
/* Validate instructions and resolve the program's referenced resources. */
ret = check_and_resolve_insns(env);
if (ret < 0)
@@ -21697,6 +21810,10 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
if (ret < 0)
goto skip_full_check;
+ ret = bpf_check_cleanup_exceptions(env);
+ if (ret < 0)
+ goto skip_full_check;
+
ret = bpf_compute_postorder(env);
if (ret < 0)
goto skip_full_check;
@@ -21884,6 +22001,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
kvfree(env->succ);
kvfree(env->gotox_tmp_buf);
bpf_diag_free(env);
+ kvfree(env->cleanup_info);
kvfree(env);
return ret;
}
diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h
index 6330b7d745c57..a69cc04275751 100644
--- a/tools/include/uapi/linux/bpf.h
+++ b/tools/include/uapi/linux/bpf.h
@@ -1669,6 +1669,9 @@ union bpf_attr {
* verification.
*/
__s32 keyring_id;
+ __aligned_u64 cleanup_info; /* exception cleanup table */
+ __u32 cleanup_info_rec_size; /* userspace bpf_cleanup_info size */
+ __u32 cleanup_info_cnt; /* number of bpf_cleanup_info records */
};
struct { /* anonymous struct used by BPF_OBJ_* commands */
@@ -7599,6 +7602,12 @@ struct bpf_line_info {
__u32 line_col;
};
+struct bpf_cleanup_info {
+ __u32 begin_off;
+ __u32 end_off;
+ __u32 landing_pad_off;
+};
+
struct bpf_spin_lock {
__u32 val;
};
diff --git a/tools/lib/bpf/bpf.c b/tools/lib/bpf/bpf.c
index a9de7f107cf7b..f11d042a28504 100644
--- a/tools/lib/bpf/bpf.c
+++ b/tools/lib/bpf/bpf.c
@@ -295,7 +295,7 @@ int bpf_prog_load(enum bpf_prog_type prog_type,
const struct bpf_insn *insns, size_t insn_cnt,
struct bpf_prog_load_opts *opts)
{
- const size_t attr_sz = offsetofend(union bpf_attr, keyring_id);
+ const size_t attr_sz = offsetofend(union bpf_attr, cleanup_info_cnt);
void *finfo = NULL, *linfo = NULL;
const char *func_info, *line_info;
__u32 log_size, log_level, attach_prog_fd, attach_btf_obj_fd;
@@ -370,6 +370,10 @@ int bpf_prog_load(enum bpf_prog_type prog_type,
attr.fd_array = ptr_to_u64(OPTS_GET(opts, fd_array, NULL));
attr.fd_array_cnt = OPTS_GET(opts, fd_array_cnt, 0);
+ attr.cleanup_info = ptr_to_u64(OPTS_GET(opts, cleanup_info, NULL));
+ attr.cleanup_info_cnt = OPTS_GET(opts, cleanup_info_cnt, 0);
+ attr.cleanup_info_rec_size = OPTS_GET(opts, cleanup_info_rec_size, 0);
+
if (log_level) {
attr.log_buf = ptr_to_u64(log_buf);
attr.log_size = log_size;
diff --git a/tools/lib/bpf/bpf.h b/tools/lib/bpf/bpf.h
index 490e8cb4ba537..ec542fd68f631 100644
--- a/tools/lib/bpf/bpf.h
+++ b/tools/lib/bpf/bpf.h
@@ -128,9 +128,14 @@ struct bpf_prog_load_opts {
/* if set, provides the length of fd_array */
__u32 fd_array_cnt;
+
+ /* exception cleanup table, from the .bpf_cleanup section */
+ const void *cleanup_info;
+ __u32 cleanup_info_cnt;
+ __u32 cleanup_info_rec_size;
size_t :0;
};
-#define bpf_prog_load_opts__last_field fd_array_cnt
+#define bpf_prog_load_opts__last_field cleanup_info_rec_size
LIBBPF_API int bpf_prog_load(enum bpf_prog_type prog_type,
const char *prog_name, const char *license,
diff --git a/tools/lib/bpf/gen_loader.c b/tools/lib/bpf/gen_loader.c
index af3a04f161ac1..2345fbdd46f56 100644
--- a/tools/lib/bpf/gen_loader.c
+++ b/tools/lib/bpf/gen_loader.c
@@ -981,13 +981,15 @@ static void cleanup_relos(struct bpf_gen *gen, int insns)
cleanup_core_relo(gen);
}
-/* Convert func, line, and core relo info blobs to target endianness */
+/* Convert func, line, core relo and cleanup info blobs to target endianness */
static void info_blob_bswap(struct bpf_gen *gen, int func_info, int line_info,
- int core_relos, struct bpf_prog_load_opts *load_attr)
+ int core_relos, int cleanup_info,
+ struct bpf_prog_load_opts *load_attr)
{
struct bpf_func_info *fi = gen->data_start + func_info;
struct bpf_line_info *li = gen->data_start + line_info;
struct bpf_core_relo *cr = gen->data_start + core_relos;
+ struct bpf_cleanup_info *ci = gen->data_start + cleanup_info;
int i;
for (i = 0; i < load_attr->func_info_cnt; i++)
@@ -998,6 +1000,9 @@ static void info_blob_bswap(struct bpf_gen *gen, int func_info, int line_info,
for (i = 0; i < gen->core_relo_cnt; i++)
bpf_core_relo_bswap(cr++);
+
+ for (i = 0; i < load_attr->cleanup_info_cnt; i++)
+ bpf_cleanup_info_bswap(ci++);
}
void bpf_gen__prog_load(struct bpf_gen *gen,
@@ -1011,8 +1016,11 @@ void bpf_gen__prog_load(struct bpf_gen *gen,
load_attr->line_info_rec_size;
int core_relo_tot_sz = gen->core_relo_cnt *
sizeof(struct bpf_core_relo);
+ int cleanup_info_tot_sz = load_attr->cleanup_info_cnt *
+ load_attr->cleanup_info_rec_size;
int prog_load_attr, license_off, insns_off, func_info, line_info, core_relos;
- int attr_size = offsetofend(union bpf_attr, core_relo_rec_size);
+ int attr_size = offsetofend(union bpf_attr, cleanup_info_cnt);
+ int cleanup_info;
union bpf_attr attr;
memset(&attr, 0, attr_size);
@@ -1061,9 +1069,17 @@ void bpf_gen__prog_load(struct bpf_gen *gen,
core_relos, gen->core_relo_cnt,
sizeof(struct bpf_core_relo));
+ attr.cleanup_info_rec_size = tgt_endian(load_attr->cleanup_info_rec_size);
+ attr.cleanup_info_cnt = tgt_endian(load_attr->cleanup_info_cnt);
+ cleanup_info = add_data(gen, load_attr->cleanup_info, cleanup_info_tot_sz);
+ pr_debug("gen: prog_load: cleanup_info: off %d cnt %u rec size %u\n",
+ cleanup_info, load_attr->cleanup_info_cnt,
+ load_attr->cleanup_info_rec_size);
+
/* convert all info blobs to target endianness */
if (gen->swapped_endian && !gen->error)
- info_blob_bswap(gen, func_info, line_info, core_relos, load_attr);
+ info_blob_bswap(gen, func_info, line_info, core_relos, cleanup_info,
+ load_attr);
libbpf_strlcpy(attr.prog_name, prog_name, sizeof(attr.prog_name));
prog_load_attr = add_data(gen, &attr, attr_size);
@@ -1085,6 +1101,11 @@ void bpf_gen__prog_load(struct bpf_gen *gen,
/* populate union bpf_attr with a pointer to core_relos */
emit_rel_store(gen, attr_field(prog_load_attr, core_relos), core_relos);
+ /* populate union bpf_attr with a pointer to cleanup_info, if there is one */
+ if (load_attr->cleanup_info_cnt)
+ emit_rel_store(gen, attr_field(prog_load_attr, cleanup_info),
+ cleanup_info);
+
/* populate union bpf_attr fd_array with a pointer to data where map_fds are saved */
emit_rel_store(gen, attr_field(prog_load_attr, fd_array), gen->fd_array);
diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
index 2f53afc985cfb..a8da2b680685c 100644
--- a/tools/lib/bpf/libbpf.c
+++ b/tools/lib/bpf/libbpf.c
@@ -514,6 +514,11 @@ struct bpf_program {
void *line_info;
__u32 line_info_rec_size;
__u32 line_info_cnt;
+
+ struct bpf_cleanup_info *cleanup_info;
+ __u32 cleanup_info_rec_size;
+ __u32 cleanup_info_cnt;
+
__u32 prog_flags;
__u8 hash[SHA256_DIGEST_LENGTH];
@@ -549,6 +554,7 @@ struct bpf_struct_ops {
#define STRUCT_OPS_SEC ".struct_ops"
#define STRUCT_OPS_LINK_SEC ".struct_ops.link"
#define ARENA_SEC ".addr_space.1"
+#define CLEANUP_SEC ".bpf_cleanup"
enum libbpf_map_type {
LIBBPF_MAP_UNSPEC,
@@ -677,6 +683,25 @@ struct elf_sec_desc {
Elf_Data *data;
};
+#define CLEANUP_REC_FIELDS (sizeof(struct bpf_cleanup_info) / sizeof(__u32))
+
+/* Index of each field of struct bpf_cleanup_info, read as an array of __u32. */
+enum {
+ CLEANUP_REC_BEGIN,
+ CLEANUP_REC_END,
+ CLEANUP_REC_PAD,
+};
+
+/* One (begin, end, landing_pad) triple from .bpf_cleanup, with each field
+ * resolved from its relocation to an ELF section plus a section-relative
+ * instruction index. The mapping to final program instruction indices can only
+ * happen after subprogram placement, which differs per main program.
+ */
+struct cleanup_raw_rec {
+ int sec_idx[CLEANUP_REC_FIELDS];
+ size_t insn_idx[CLEANUP_REC_FIELDS];
+};
+
struct elf_state {
int fd;
const void *obj_buf;
@@ -696,6 +721,8 @@ struct elf_state {
bool has_st_ops;
int arena_data_shndx;
int jumptables_data_shndx;
+ Elf_Data *cleanup_data;
+ int cleanup_shndx;
};
struct usdt_manager;
@@ -771,6 +798,9 @@ struct bpf_object {
void *jumptables_data;
size_t jumptables_data_sz;
+ struct cleanup_raw_rec *cleanup_recs;
+ size_t cleanup_rec_cnt;
+
struct {
struct bpf_program *prog;
unsigned int sym_off;
@@ -817,7 +847,10 @@ static void bpf_program__exit(struct bpf_program *prog)
zfree(&prog->sec_name);
zfree(&prog->insns);
zfree(&prog->reloc_desc);
+ zfree(&prog->cleanup_info);
+ prog->cleanup_info_rec_size = 0;
+ prog->cleanup_info_cnt = 0;
prog->nr_reloc = 0;
prog->insns_cnt = 0;
prog->sec_idx = -1;
@@ -1559,6 +1592,7 @@ static struct bpf_object *bpf_object__new(const char *path,
obj->efile.obj_buf = obj_buf;
obj->efile.obj_buf_sz = obj_buf_sz;
obj->efile.btf_maps_shndx = -1;
+ obj->efile.cleanup_shndx = -1;
obj->kconfig_map_idx = -1;
obj->arena_map_idx = -1;
@@ -4045,6 +4079,9 @@ static int bpf_object__elf_collect(struct bpf_object *obj)
sec_desc->shdr = sh;
sec_desc->data = data;
obj->efile.has_st_ops = true;
+ } else if (strcmp(name, CLEANUP_SEC) == 0) {
+ obj->efile.cleanup_data = data;
+ obj->efile.cleanup_shndx = idx;
} else if (strcmp(name, ARENA_SEC) == 0) {
obj->efile.arena_data = data;
obj->efile.arena_data_shndx = idx;
@@ -4072,6 +4109,7 @@ static int bpf_object__elf_collect(struct bpf_object *obj)
strcmp(name, ".rel" STRUCT_OPS_LINK_SEC) &&
strcmp(name, ".rel?" STRUCT_OPS_SEC) &&
strcmp(name, ".rel?" STRUCT_OPS_LINK_SEC) &&
+ strcmp(name, ".rel" CLEANUP_SEC) &&
strcmp(name, ".rel" MAPS_ELF_SEC)) {
pr_info("elf: skipping relo section(%d) %s for section(%d) %s\n",
idx, name, targ_sec_idx,
@@ -4852,6 +4890,229 @@ static struct bpf_program *find_prog_by_sec_insn(const struct bpf_object *obj,
return NULL;
}
+static int bpf_object__init_cleanup_info(struct bpf_object *obj)
+{
+ Elf_Data *data = obj->efile.cleanup_data;
+ Elf_Data *relo = NULL;
+ size_t i, nrels, nslots, nrecs;
+ struct cleanup_raw_rec *recs;
+ int *slot_sec, ret = 0;
+ size_t *slot_val;
+ const __u32 *vals;
+ bool native;
+
+ if (!data || obj->efile.cleanup_shndx < 0)
+ return 0;
+
+ native = is_native_endianness(obj);
+
+ for (i = 0; i < obj->efile.sec_cnt; i++) {
+ struct elf_sec_desc *sd = &obj->efile.secs[i];
+
+ if (sd->sec_type == SEC_RELO && sd->shdr &&
+ sd->shdr->sh_info == (Elf64_Word)obj->efile.cleanup_shndx) {
+ relo = sd->data;
+ break;
+ }
+ }
+ if (!relo) {
+ pr_warn("%s present without relocations\n", CLEANUP_SEC);
+ return -LIBBPF_ERRNO__FORMAT;
+ }
+ if (data->d_size % sizeof(struct bpf_cleanup_info)) {
+ pr_warn("%s size %zu is not a multiple of the record size %zu\n",
+ CLEANUP_SEC, data->d_size, sizeof(struct bpf_cleanup_info));
+ return -LIBBPF_ERRNO__FORMAT;
+ }
+
+ vals = data->d_buf;
+ nslots = data->d_size / sizeof(__u32);
+ nrecs = data->d_size / sizeof(struct bpf_cleanup_info);
+
+ slot_sec = calloc(nslots, sizeof(*slot_sec));
+ slot_val = calloc(nslots, sizeof(*slot_val));
+ recs = calloc(nrecs ?: 1, sizeof(*recs));
+ if (!slot_sec || !slot_val || !recs) {
+ ret = -ENOMEM;
+ goto out;
+ }
+ for (i = 0; i < nslots; i++)
+ slot_sec[i] = -1;
+
+ /* One relocation per 4-byte field, naming the section it points into. */
+ nrels = relo->d_size / sizeof(Elf64_Rel);
+ for (i = 0; i < nrels; i++) {
+ Elf64_Rel *rel = elf_rel_by_idx(relo, i);
+ Elf64_Sym *sym = elf_sym_by_idx(obj, ELF64_R_SYM(rel->r_info));
+ size_t type = ELF64_R_TYPE(rel->r_info);
+ size_t slot = rel->r_offset / sizeof(__u32);
+
+ if (type != R_BPF_64_NODYLD32 && type != R_BPF_64_ABS32) {
+ pr_warn("%s: relocation %zu has unexpected type %zu\n",
+ CLEANUP_SEC, i, type);
+ ret = -LIBBPF_ERRNO__FORMAT;
+ goto out;
+ }
+ if (!sym || slot >= nslots || rel->r_offset % sizeof(__u32)) {
+ pr_warn("%s: bad relocation %zu\n", CLEANUP_SEC, i);
+ ret = -LIBBPF_ERRNO__FORMAT;
+ goto out;
+ }
+ slot_sec[slot] = sym->st_shndx;
+ /* The addend lives in the section data, which libelf leaves in
+ * the object's byte order; a non-section symbol additionally
+ * contributes its own value.
+ */
+ slot_val[slot] = (native ? vals[slot] : bswap_32(vals[slot])) +
+ sym->st_value;
+ }
+
+ for (i = 0; i < nslots; i++) {
+ struct cleanup_raw_rec *rec = &recs[i / CLEANUP_REC_FIELDS];
+ size_t field = i % CLEANUP_REC_FIELDS;
+
+ if (slot_sec[i] < 0) {
+ pr_warn("%s: field %zu has no relocation\n", CLEANUP_SEC, i);
+ ret = -LIBBPF_ERRNO__FORMAT;
+ goto out;
+ }
+ if (slot_val[i] % BPF_INSN_SZ) {
+ pr_warn("%s: field %zu offset %zu is not instruction aligned\n",
+ CLEANUP_SEC, i, slot_val[i]);
+ ret = -LIBBPF_ERRNO__FORMAT;
+ goto out;
+ }
+ rec->sec_idx[field] = slot_sec[i];
+ rec->insn_idx[field] = slot_val[i] / BPF_INSN_SZ;
+ }
+
+ obj->cleanup_recs = recs;
+ obj->cleanup_rec_cnt = nrecs;
+ recs = NULL;
+out:
+ free(recs);
+ free(slot_val);
+ free(slot_sec);
+ return ret;
+}
+
+static int cmp_cleanup_info(const void *a, const void *b)
+{
+ const struct bpf_cleanup_info *x = a, *y = b;
+
+ if (x->begin_off == y->begin_off)
+ return 0;
+ return x->begin_off < y->begin_off ? -1 : 1;
+}
+
+static int bpf_prog_collect_cleanup_info(struct bpf_object *obj,
+ struct bpf_program *prog)
+{
+ size_t i;
+ int j;
+
+ for (i = 0; i < obj->cleanup_rec_cnt; i++) {
+ struct cleanup_raw_rec *raw = &obj->cleanup_recs[i];
+ struct bpf_program *owner = NULL;
+ struct bpf_cleanup_info ci = {};
+ __u32 fields[CLEANUP_REC_FIELDS];
+ void *tmp;
+
+ if (raw->sec_idx[CLEANUP_REC_BEGIN] == raw->sec_idx[CLEANUP_REC_END] &&
+ raw->insn_idx[CLEANUP_REC_BEGIN] >= raw->insn_idx[CLEANUP_REC_END]) {
+ pr_warn("%s: record %zu is an empty range [%zu,%zu)\n",
+ CLEANUP_SEC, i, raw->insn_idx[CLEANUP_REC_BEGIN],
+ raw->insn_idx[CLEANUP_REC_END]);
+ return -LIBBPF_ERRNO__FORMAT;
+ }
+
+ for (j = 0; j < CLEANUP_REC_FIELDS; j++) {
+ size_t idx = raw->insn_idx[j], final;
+ struct bpf_program *p;
+
+ /* The end of a range is exclusive, so it may name the
+ * instruction just past the last one of a function,
+ * which belongs to the next function or to nothing at
+ * all. Ask about the last instruction the range covers,
+ * the way the kernel does.
+ */
+ if (j == CLEANUP_REC_END) {
+ if (!idx) {
+ pr_warn("%s: record %zu ends at instruction 0\n",
+ CLEANUP_SEC, i);
+ return -LIBBPF_ERRNO__FORMAT;
+ }
+ idx--;
+ }
+
+ p = find_prog_by_sec_insn(obj, raw->sec_idx[j], idx);
+ if (!p) {
+ pr_warn("%s: record %zu field %d is not inside a function\n",
+ CLEANUP_SEC, i, j);
+ return -LIBBPF_ERRNO__FORMAT;
+ }
+ if (!owner) {
+ owner = p;
+ } else if (owner != p) {
+ pr_warn("%s: record %zu spans functions '%s' and '%s'\n",
+ CLEANUP_SEC, i, owner->name, p->name);
+ return -LIBBPF_ERRNO__FORMAT;
+ }
+
+ if (owner == prog) {
+ final = raw->insn_idx[j] - prog->sec_insn_off;
+ } else if (prog_is_subprog(obj, owner) && owner->sub_insn_off) {
+ /* sub_insn_off is where this subprogram was
+ * appended to the main program being relocated;
+ * zero means it is not part of it.
+ */
+ final = owner->sub_insn_off +
+ raw->insn_idx[j] - owner->sec_insn_off;
+ } else {
+ owner = NULL;
+ break;
+ }
+ fields[j] = final;
+ }
+ if (!owner)
+ continue;
+
+ ci.begin_off = fields[CLEANUP_REC_BEGIN];
+ ci.end_off = fields[CLEANUP_REC_END];
+ ci.landing_pad_off = fields[CLEANUP_REC_PAD];
+
+ tmp = libbpf_reallocarray(prog->cleanup_info, prog->cleanup_info_cnt + 1,
+ sizeof(*prog->cleanup_info));
+ if (!tmp)
+ return -ENOMEM;
+ prog->cleanup_info = tmp;
+ prog->cleanup_info_rec_size = sizeof(struct bpf_cleanup_info);
+ prog->cleanup_info[prog->cleanup_info_cnt++] = ci;
+
+ pr_debug("prog '%s': cleanup region [%u,%u) -> landing pad %u\n",
+ prog->name, ci.begin_off, ci.end_off, ci.landing_pad_off);
+ }
+
+ if (!prog->cleanup_info_cnt)
+ return 0;
+
+ qsort(prog->cleanup_info, prog->cleanup_info_cnt,
+ sizeof(*prog->cleanup_info), cmp_cleanup_info);
+ for (i = 1; i < prog->cleanup_info_cnt; i++) {
+ struct bpf_cleanup_info *prev = &prog->cleanup_info[i - 1];
+ struct bpf_cleanup_info *cur = &prog->cleanup_info[i];
+
+ if (cur->begin_off < prev->end_off) {
+ pr_warn("prog '%s': overlapping cleanup regions [%u,%u) and [%u,%u)\n",
+ prog->name, prev->begin_off, prev->end_off,
+ cur->begin_off, cur->end_off);
+ return -LIBBPF_ERRNO__FORMAT;
+ }
+ }
+
+ return 0;
+}
+
static int
bpf_object__collect_prog_relos(struct bpf_object *obj, Elf64_Shdr *shdr, Elf_Data *data)
{
@@ -7561,6 +7822,13 @@ static int bpf_object__relocate(struct bpf_object *obj, const char *targ_btf_pat
return err;
}
}
+
+ err = bpf_prog_collect_cleanup_info(obj, prog);
+ if (err) {
+ pr_warn("prog '%s': failed to collect cleanup info: %s\n",
+ prog->name, errstr(err));
+ return err;
+ }
}
for (i = 0; i < obj->nr_programs; i++) {
prog = &obj->programs[i];
@@ -7751,6 +8019,9 @@ static int bpf_object__collect_relos(struct bpf_object *obj)
return -LIBBPF_ERRNO__INTERNAL;
}
+ if (idx == obj->efile.cleanup_shndx)
+ continue;
+
if (obj->efile.secs[idx].sec_type == SEC_ST_OPS)
err = bpf_object__collect_st_ops_relos(obj, shdr, data);
else if (idx == obj->efile.btf_maps_shndx)
@@ -8023,6 +8294,11 @@ static int bpf_object_load_prog(struct bpf_object *obj, struct bpf_program *prog
load_attr.line_info_rec_size = prog->line_info_rec_size;
load_attr.line_info_cnt = prog->line_info_cnt;
}
+ if (prog->cleanup_info_cnt) {
+ load_attr.cleanup_info = prog->cleanup_info;
+ load_attr.cleanup_info_cnt = prog->cleanup_info_cnt;
+ load_attr.cleanup_info_rec_size = prog->cleanup_info_rec_size;
+ }
load_attr.log_level = log_level;
load_attr.prog_flags = prog->prog_flags;
load_attr.fd_array = obj->fd_array;
@@ -8351,6 +8627,15 @@ static void fixup_verifier_log(struct bpf_program *prog, char *buf, size_t buf_s
}
}
+/* LLVM terminates a cleanup landing pad with a call to _Unwind_Resume, the
+ * base unwind ABI's entry point for carrying an unwind on once a frame's
+ * cleanups have run. The kernel knows it as bpf_unwind_resume.
+ */
+static const char *kern_extern_name(const char *name)
+{
+ return strcmp(name, "_Unwind_Resume") ? name : "bpf_unwind_resume";
+}
+
static int bpf_program_record_relos(struct bpf_program *prog)
{
struct bpf_object *obj = prog->obj;
@@ -8367,12 +8652,12 @@ static int bpf_program_record_relos(struct bpf_program *prog)
continue;
kind = btf_is_var(btf__type_by_id(obj->btf, ext->btf_id)) ?
BTF_KIND_VAR : BTF_KIND_FUNC;
- bpf_gen__record_extern(obj->gen_loader, ext->name,
+ bpf_gen__record_extern(obj->gen_loader, kern_extern_name(ext->name),
ext->is_weak, !ext->ksym.type_id,
true, kind, relo->insn_idx);
break;
case RELO_EXTERN_CALL:
- bpf_gen__record_extern(obj->gen_loader, ext->name,
+ bpf_gen__record_extern(obj->gen_loader, kern_extern_name(ext->name),
ext->is_weak, false, false, BTF_KIND_FUNC,
relo->insn_idx);
break;
@@ -8570,6 +8855,7 @@ static struct bpf_object *bpf_object_open(const char *path, const void *obj_buf,
err = err ? : bpf_object__init_maps(obj, opts);
err = err ? : bpf_object_init_progs(obj, opts);
err = err ? : bpf_object__collect_relos(obj);
+ err = err ? : bpf_object__init_cleanup_info(obj);
if (err)
goto out;
@@ -8812,17 +9098,19 @@ static int bpf_object__resolve_ksym_func_btf_id(struct bpf_object *obj,
struct module_btf *mod_btf = NULL;
const struct btf_type *kern_func;
struct btf *kern_btf = NULL;
+ const char *kern_name;
int ret;
local_func_proto_id = ext->ksym.type_id;
- kfunc_id = find_ksym_btf_id(obj, ext->essent_name ?: ext->name, BTF_KIND_FUNC, &kern_btf,
- &mod_btf);
+ kern_name = kern_extern_name(ext->essent_name ?: ext->name);
+
+ kfunc_id = find_ksym_btf_id(obj, kern_name, BTF_KIND_FUNC, &kern_btf, &mod_btf);
if (kfunc_id < 0) {
if (kfunc_id == -ESRCH && ext->is_weak)
return 0;
pr_warn("extern (func ksym) '%s': not found in kernel or module BTFs\n",
- ext->name);
+ strcmp(kern_name, "bpf_unwind_resume") ? ext->name : kern_name);
return kfunc_id;
}
@@ -9681,6 +9969,9 @@ void bpf_object__close(struct bpf_object *obj)
zfree(&obj->jumptables_data);
obj->jumptables_data_sz = 0;
+ zfree(&obj->cleanup_recs);
+ obj->cleanup_rec_cnt = 0;
+
for (i = 0; i < obj->jumptable_map_cnt; i++)
close(obj->jumptable_maps[i].fd);
zfree(&obj->jumptable_maps);
@@ -10073,6 +10364,25 @@ int bpf_program__clone(struct bpf_program *prog, const struct bpf_prog_load_opts
attr.line_info_rec_size = info ? info_rec_size : prog->line_info_rec_size;
}
+ /* exception cleanup table */
+ info = OPTS_GET(opts, cleanup_info, NULL);
+ info_cnt = OPTS_GET(opts, cleanup_info_cnt, 0);
+ info_rec_size = OPTS_GET(opts, cleanup_info_rec_size, 0);
+ if (!!info != !!info_cnt || !!info != !!info_rec_size) {
+ pr_warn("prog '%s': cleanup_info, cleanup_info_cnt, and cleanup_info_rec_size must all be specified or all omitted\n",
+ prog->name);
+ return libbpf_err(-EINVAL);
+ }
+ if (info) {
+ attr.cleanup_info = info;
+ attr.cleanup_info_cnt = info_cnt;
+ attr.cleanup_info_rec_size = info_rec_size;
+ } else if (prog->cleanup_info_cnt) {
+ attr.cleanup_info = prog->cleanup_info;
+ attr.cleanup_info_cnt = prog->cleanup_info_cnt;
+ attr.cleanup_info_rec_size = prog->cleanup_info_rec_size;
+ }
+
/* Logging is caller-controlled; no fallback to prog/obj log settings */
attr.log_buf = OPTS_GET(opts, log_buf, NULL);
attr.log_size = OPTS_GET(opts, log_size, 0);
diff --git a/tools/lib/bpf/libbpf_internal.h b/tools/lib/bpf/libbpf_internal.h
index cb4d962338441..78519f24fb404 100644
--- a/tools/lib/bpf/libbpf_internal.h
+++ b/tools/lib/bpf/libbpf_internal.h
@@ -56,6 +56,9 @@
#ifndef R_BPF_64_ABS32
#define R_BPF_64_ABS32 3
#endif
+#ifndef R_BPF_64_NODYLD32
+#define R_BPF_64_NODYLD32 4
+#endif
#ifndef R_BPF_64_32
#define R_BPF_64_32 10
#endif
@@ -569,6 +572,13 @@ static inline void bpf_core_relo_bswap(struct bpf_core_relo *i)
i->kind = bswap_32(i->kind);
}
+static inline void bpf_cleanup_info_bswap(struct bpf_cleanup_info *i)
+{
+ i->begin_off = bswap_32(i->begin_off);
+ i->end_off = bswap_32(i->end_off);
+ i->landing_pad_off = bswap_32(i->landing_pad_off);
+}
+
enum btf_field_iter_kind {
BTF_FIELD_ITER_IDS,
BTF_FIELD_ITER_STRS,
diff --git a/tools/lib/bpf/linker.c b/tools/lib/bpf/linker.c
index 78f92c39290af..fac7390bd109d 100644
--- a/tools/lib/bpf/linker.c
+++ b/tools/lib/bpf/linker.c
@@ -1036,7 +1036,8 @@ static int linker_sanity_check_elf_relos(struct src_obj *obj, struct src_sec *se
size_t sym_type = ELF64_R_TYPE(relo->r_info);
if (sym_type != R_BPF_64_64 && sym_type != R_BPF_64_32 &&
- sym_type != R_BPF_64_ABS64 && sym_type != R_BPF_64_ABS32) {
+ sym_type != R_BPF_64_ABS64 && sym_type != R_BPF_64_ABS32 &&
+ sym_type != R_BPF_64_NODYLD32) {
pr_warn("ELF relo #%d in section #%zu has unexpected type %zu in %s\n",
i, sec->sec_idx, sym_type, obj->filename);
return -EINVAL;
@@ -2274,6 +2275,39 @@ static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob
insn->imm += sec->dst_off / sizeof(struct bpf_insn);
else
insn->imm += sec->dst_off;
+ } else if (sym_type == R_BPF_64_NODYLD32 ||
+ sym_type == R_BPF_64_ABS32) {
+ __u32 *val;
+
+ /* Two spellings of the one thing: LLVM
+ * emits NODYLD32 for a .long against a
+ * section symbol, GNU as emits ABS32
+ * (bpf_reloc_type_lookup() maps
+ * BFD_RELOC_32 to it), and the value
+ * they describe is the same.
+ */
+
+ /* An SHT_NOBITS section has no
+ * raw_data, and r_offset is sanity
+ * checked only for sections holding
+ * instructions. Check both here
+ * before writing.
+ */
+ if (!dst_linked_sec->raw_data ||
+ dst_rel->r_offset % sizeof(*val) ||
+ dst_rel->r_offset + sizeof(*val) >
+ (size_t)dst_linked_sec->sec_sz) {
+ pr_warn("ELF relo #%d in section #%zu points outside the data of section '%s' in %s\n",
+ j, src_sec->sec_idx,
+ dst_linked_sec->sec_name,
+ obj->filename);
+ return -EINVAL;
+ }
+ val = dst_linked_sec->raw_data + dst_rel->r_offset;
+ if (linker->swapped_endian)
+ *val = bswap_32(bswap_32(*val) + sec->dst_off);
+ else
+ *val += sec->dst_off;
} else {
pr_warn("relocation against STT_SECTION in non-exec section is not supported!\n");
return -EINVAL;
diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile
index 7ea5ba1df29e3..981cc26430495 100644
--- a/tools/testing/selftests/bpf/Makefile
+++ b/tools/testing/selftests/bpf/Makefile
@@ -523,7 +523,7 @@ LINKED_SKELS := test_static_linked.skel.h linked_funcs.skel.h \
LSKELS := fexit_sleep.c trace_printk.c trace_vprintk.c map_ptr_kern.c \
core_kern.c core_kern_overflow.c test_ringbuf.c \
test_ringbuf_n.c test_ringbuf_map_key.c test_ringbuf_write.c \
- test_ringbuf_overwrite.c
+ test_ringbuf_overwrite.c exceptions_cleanup_light.c
LSKELS_SIGNED := fentry_test.c fexit_test.c atomics.c
diff --git a/tools/testing/selftests/bpf/exceptions_cleanup.h b/tools/testing/selftests/bpf/exceptions_cleanup.h
new file mode 100644
index 0000000000000..e14b651263b88
--- /dev/null
+++ b/tools/testing/selftests/bpf/exceptions_cleanup.h
@@ -0,0 +1,55 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#ifndef __EXCEPTIONS_CLEANUP_H__
+#define __EXCEPTIONS_CLEANUP_H__
+
+#define THROW_COOKIE 0x100
+#define INNER_COOKIE 0x200
+
+/* progs/exceptions_cleanup.c: one bit per frame that reports it ran. */
+#define RAN_FOO3_PREEMPT 0x1
+#define RAN_FOO2_RCU 0x2
+#define RAN_FOO1V_PREEMPT 0x4
+#define RAN_FOO2_DROP 0x8
+#define RAN_BUMP 0x10
+
+/* progs/exceptions_cleanup_shapes.c: one bit per shape, numbered its own way. */
+#define RAN_SWEEP 0x1
+#define RAN_SHARED 0x2
+#define RAN_REGS 0x4
+#define RAN_TAIL_CALL 0x8
+#define RAN_MAIN_PAD 0x10
+#define RAN_TC_TAKEN 0x20
+#define RAN_FREPLACE 0x40
+#define RAN_ADDR_TAKEN 0x80
+#define RAN_NO_SUBPROG 0x100
+#define RAN_PAD_CALLS 0x200
+#define RAN_PAD_FIRST 0x400
+#define RAN_WIDE_REC 0x800
+#define RAN_PAD_STACK 0x1000
+#define RAN_DEEP_PAD 0x2000
+#define RAN_NOUNWIND_REC 0x4000
+#define RAN_RESUME_ALIAS 0x8000
+#define RAN_PAD_TAIL_CALL 0x10000
+#define RAN_NOP_PAD 0x20000
+#define RAN_VAR_STACK 0x40000
+#define RAN_GLOBAL_PAD 0x80000
+
+/* progs/exceptions_cleanup_light.c: the one pad it has. */
+#define RAN_LIGHT 0x1
+
+#define CLEANUP_REC(begin, end, landing_pad) \
+ ".pushsection .bpf_cleanup,\"a\",@progbits;" \
+ ".long " begin ";" \
+ ".long " end ";" \
+ ".long " landing_pad ";" \
+ ".popsection;"
+
+/* Set a bit in @pads_ran. */
+#define PAD_RAN(bit) \
+ "r1 = %[pads_ran] ll;" \
+ "r2 = *(u64 *)(r1 + 0);" \
+ "r2 |= " bit ";" \
+ "*(u64 *)(r1 + 0) = r2;"
+
+#endif /* __EXCEPTIONS_CLEANUP_H__ */
diff --git a/tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c b/tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c
new file mode 100644
index 0000000000000..b69be77b2e26c
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c
@@ -0,0 +1,454 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <test_progs.h>
+#include "exceptions_cleanup.h"
+#include "exceptions_cleanup.skel.h"
+#include "exceptions_cleanup_fail.skel.h"
+#include "exceptions_cleanup_shapes.skel.h"
+#include "exceptions_cleanup_freplace.skel.h"
+#include "exceptions_cleanup_pad_freplace.skel.h"
+#include "exceptions_cleanup_ext_table.skel.h"
+#include "exceptions_cleanup_light.lskel.h"
+
+/* foo3 threw: every frame that has a pad ran it. */
+#define PADS_FOO3_THREW \
+ (RAN_FOO3_PREEMPT | RAN_FOO2_RCU | RAN_FOO1V_PREEMPT | RAN_FOO2_DROP)
+
+/* foo2 threw after foo3 returned normally: foo3's pad must not run. */
+#define PADS_FOO2_THREW \
+ (RAN_FOO2_RCU | RAN_FOO1V_PREEMPT | RAN_FOO2_DROP)
+
+static void run(struct exceptions_cleanup *skel, __u64 input, __u32 retval,
+ __u64 pads)
+{
+ __u64 ctx = 0;
+ int err;
+
+ LIBBPF_OPTS(bpf_test_run_opts, topts,
+ .ctx_in = &ctx,
+ .ctx_size_in = sizeof(ctx),
+ );
+
+ skel->bss->input = input;
+ skel->bss->pads_ran = 0;
+ skel->bss->result = 0;
+
+ err = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.entry), &topts);
+ if (!ASSERT_OK(err, "run"))
+ return;
+ ASSERT_EQ(topts.retval, retval, "retval");
+ ASSERT_EQ(skel->bss->pads_ran, pads | RAN_BUMP, "pads_ran");
+}
+
+static void run_shape(struct exceptions_cleanup_shapes *skel, struct bpf_program *prog,
+ __u64 input, __u32 retval, __u64 pads)
+{
+ __u64 ctx = 0;
+ int err;
+
+ LIBBPF_OPTS(bpf_test_run_opts, topts,
+ .ctx_in = &ctx,
+ .ctx_size_in = sizeof(ctx),
+ );
+
+ skel->bss->input = input;
+ skel->bss->pads_ran = 0;
+
+ err = bpf_prog_test_run_opts(bpf_program__fd(prog), &topts);
+ if (!ASSERT_OK(err, "run"))
+ return;
+ ASSERT_EQ(topts.retval, retval, "retval");
+ ASSERT_EQ(skel->bss->pads_ran, pads, "pads_ran");
+}
+
+static void test_freplace(struct exceptions_cleanup_shapes *skel)
+{
+ struct exceptions_cleanup_freplace *fr;
+ struct bpf_link *link;
+ int tgt_fd;
+
+ tgt_fd = bpf_program__fd(skel->progs.entry_freplace);
+
+ fr = exceptions_cleanup_freplace__open();
+ if (!ASSERT_OK_PTR(fr, "freplace open"))
+ return;
+
+ if (!ASSERT_OK(bpf_program__set_attach_target(fr->progs.new_fr_callee,
+ tgt_fd, "fr_callee"),
+ "set_attach_target"))
+ goto out;
+ if (!ASSERT_OK(exceptions_cleanup_freplace__load(fr), "freplace load"))
+ goto out;
+
+ link = bpf_program__attach_freplace(fr->progs.new_fr_callee, tgt_fd,
+ "fr_callee");
+ if (!ASSERT_OK_PTR(link, "attach_freplace"))
+ goto out;
+
+ run_shape(skel, skel->progs.entry_freplace, 101, THROW_COOKIE, 0);
+ bpf_link__destroy(link);
+out:
+ exceptions_cleanup_freplace__destroy(fr);
+}
+
+static void test_pad_calls_freplace(struct exceptions_cleanup_shapes *skel)
+{
+ struct exceptions_cleanup_pad_freplace *fr;
+ struct bpf_link *link;
+ __u64 ctx = 0;
+ int tgt_fd, err;
+
+ LIBBPF_OPTS(bpf_test_run_opts, topts,
+ .ctx_in = &ctx,
+ .ctx_size_in = sizeof(ctx),
+ );
+
+ tgt_fd = bpf_program__fd(skel->progs.entry_pad_calls);
+
+ fr = exceptions_cleanup_pad_freplace__open();
+ if (!ASSERT_OK_PTR(fr, "pad freplace open"))
+ return;
+
+ if (!ASSERT_OK(bpf_program__set_attach_target(fr->progs.new_pad_callee,
+ tgt_fd, "pad_callee"),
+ "set_attach_target"))
+ goto out;
+ if (!ASSERT_OK(exceptions_cleanup_pad_freplace__load(fr), "pad freplace load"))
+ goto out;
+
+ link = bpf_program__attach_freplace(fr->progs.new_pad_callee, tgt_fd,
+ "pad_callee");
+ if (!ASSERT_OK_PTR(link, "attach_freplace"))
+ goto out;
+
+ skel->bss->input = 101;
+ skel->bss->pads_ran = 0;
+ skel->bss->pad_runs = 0;
+
+ err = bpf_prog_test_run_opts(tgt_fd, &topts);
+ if (!ASSERT_OK(err, "run"))
+ goto out_link;
+
+ ASSERT_EQ(skel->bss->pad_runs, 1, "pad_runs");
+ ASSERT_EQ(skel->bss->pads_ran, RAN_PAD_CALLS, "pads_ran");
+ ASSERT_EQ(topts.retval, THROW_COOKIE, "retval");
+out_link:
+ bpf_link__destroy(link);
+out:
+ exceptions_cleanup_pad_freplace__destroy(fr);
+}
+
+static void test_ext_table(struct exceptions_cleanup_shapes *skel)
+{
+ struct exceptions_cleanup_ext_table *fr;
+ struct bpf_link *link;
+ int tgt_fd;
+
+ tgt_fd = bpf_program__fd(skel->progs.entry_freplace);
+
+ fr = exceptions_cleanup_ext_table__open();
+ if (!ASSERT_OK_PTR(fr, "ext table open"))
+ return;
+
+ if (!ASSERT_OK(bpf_program__set_attach_target(fr->progs.new_fr_callee,
+ tgt_fd, "fr_callee"),
+ "set_attach_target"))
+ goto out;
+ if (!ASSERT_OK(exceptions_cleanup_ext_table__load(fr), "ext table load"))
+ goto out;
+
+ link = bpf_program__attach_freplace(fr->progs.new_fr_callee, tgt_fd,
+ "fr_callee");
+ if (!ASSERT_OK_PTR(link, "attach_freplace"))
+ goto out;
+
+ fr->bss->ext_pad_ran = 0;
+ run_shape(skel, skel->progs.entry_freplace, 101, THROW_COOKIE, 0);
+ ASSERT_EQ(fr->bss->ext_pad_ran, 1, "ext_pad_ran");
+
+ bpf_link__destroy(link);
+out:
+ exceptions_cleanup_ext_table__destroy(fr);
+}
+
+static void test_light_skeleton(void)
+{
+ struct exceptions_cleanup_light_lskel *skel;
+ __u64 ctx = 0;
+ int err;
+
+ LIBBPF_OPTS(bpf_test_run_opts, topts,
+ .ctx_in = &ctx,
+ .ctx_size_in = sizeof(ctx),
+ );
+
+ skel = exceptions_cleanup_light_lskel__open_and_load();
+ if (!ASSERT_OK_PTR(skel, "light open_and_load"))
+ return;
+
+ err = bpf_prog_test_run_opts(skel->progs.entry_light.prog_fd, &topts);
+ if (!ASSERT_OK(err, "run"))
+ goto out;
+ ASSERT_EQ(topts.retval, THROW_COOKIE, "retval");
+ ASSERT_EQ(skel->bss->pads_ran, RAN_LIGHT, "pads_ran");
+out:
+ exceptions_cleanup_light_lskel__destroy(skel);
+}
+
+static void test_shapes(void)
+{
+ struct exceptions_cleanup_shapes *skel;
+
+ skel = exceptions_cleanup_shapes__open_and_load();
+ if (!ASSERT_OK_PTR(skel, "shapes open_and_load"))
+ return;
+
+ /* The frame loads at all only if everything unreachable in it went. */
+ if (test__start_subtest("sweep_no_throw"))
+ run_shape(skel, skel->progs.entry_sweep, 1, 0, 0);
+ if (test__start_subtest("sweep_throw"))
+ run_shape(skel, skel->progs.entry_sweep, 101, THROW_COOKIE, RAN_SWEEP);
+
+ /* The covered call unwinds to the pad; the uncovered one walks past. */
+ if (test__start_subtest("shared_callee_no_throw")) {
+ skel->bss->outer_input = 0;
+ run_shape(skel, skel->progs.entry_shared, 1, 2, 0);
+ }
+ if (test__start_subtest("shared_callee_throw")) {
+ skel->bss->outer_input = 0;
+ run_shape(skel, skel->progs.entry_shared, 101, THROW_COOKIE, RAN_SHARED);
+ }
+ if (test__start_subtest("shared_callee_uncovered_throw")) {
+ skel->bss->outer_input = 101;
+ run_shape(skel, skel->progs.entry_shared, 1, THROW_COOKIE, 0);
+ skel->bss->outer_input = 0;
+ }
+
+ /* The pad only sets its bit if it got the frame's own r6-r9 back. */
+ if (test__start_subtest("pad_sees_callee_saved"))
+ run_shape(skel, skel->progs.entry_regs, 101, THROW_COOKIE, RAN_REGS);
+
+ /* Same check, with a tail-call-reachable callee: its spill moves. */
+ if (test__start_subtest("tail_call_no_throw"))
+ run_shape(skel, skel->progs.entry_tail_call, 1, 0, 0);
+ if (test__start_subtest("tail_call_throw"))
+ run_shape(skel, skel->progs.entry_tail_call, 101, THROW_COOKIE,
+ RAN_TAIL_CALL);
+
+ /* A region around a nounwind call: no pad dispatched, still loads. */
+ if (test__start_subtest("nounwind_region"))
+ run_shape(skel, skel->progs.entry_nounwind_rec, 1, 0, 0);
+
+ /* A pad in the main program's own frame, not in a subprogram. */
+ if (test__start_subtest("main_program_pad"))
+ run_shape(skel, skel->progs.entry_main_pad, 101, THROW_COOKIE,
+ RAN_MAIN_PAD);
+
+ /* The same call site either way: the subprogram's throw unwinds into
+ * this frame and runs its pad, an extension's stops at its own boundary.
+ */
+ if (test__start_subtest("freplace_subprog_throws"))
+ run_shape(skel, skel->progs.entry_freplace, 7, THROW_COOKIE,
+ RAN_FREPLACE);
+ if (test__start_subtest("freplace_extension_throws"))
+ test_freplace(skel);
+
+ /* A tail call that is taken: the walk ends at the target, so the cookie
+ * comes back from there and this frame's pad does not run -- though it
+ * is reachable, so a walk past the boundary would find it.
+ */
+ if (test__start_subtest("tail_call_taken")) {
+ int key = 0, prog_fd = bpf_program__fd(skel->progs.tc_target);
+
+ if (ASSERT_OK(bpf_map_update_elem(bpf_map__fd(skel->maps.taken_table),
+ &key, &prog_fd, BPF_ANY),
+ "populate taken_table"))
+ run_shape(skel, skel->progs.entry_tail_taken, 101,
+ THROW_COOKIE, 0);
+ }
+
+ /* A throwing subprog named by a BPF_PSEUDO_FUNC and handed to a
+ * bpf_loop() the verifier never reaches: the callback check has to fire
+ * on the helper call, not on the ld_imm64.
+ */
+ if (test__start_subtest("addr_taken_no_throw"))
+ run_shape(skel, skel->progs.entry_addr_taken, 1, 2, 0);
+ if (test__start_subtest("addr_taken_throw"))
+ run_shape(skel, skel->progs.entry_addr_taken, 101, THROW_COOKIE,
+ RAN_ADDR_TAKEN);
+
+ /* A record covering bpf_throw() itself rather than a call to a frame
+ * that throws: raised, caught up with and delivered in one frame.
+ */
+ if (test__start_subtest("no_subprog_no_throw"))
+ run_shape(skel, skel->progs.entry_no_subprog, 1, 0, 0);
+ if (test__start_subtest("no_subprog_throw"))
+ run_shape(skel, skel->progs.entry_no_subprog, 101, THROW_COOKIE,
+ RAN_NO_SUBPROG);
+
+ /* A pad that calls a subprogram; with a throwing extension in its place,
+ * the nested exception has to stop there, not restart this pad.
+ */
+ if (test__start_subtest("pad_calls_subprog")) {
+ skel->bss->pad_runs = 0;
+ run_shape(skel, skel->progs.entry_pad_calls, 101, THROW_COOKIE,
+ RAN_PAD_CALLS);
+ ASSERT_EQ(skel->bss->pad_runs, 1, "pad_runs");
+ }
+ if (test__start_subtest("pad_calls_throwing_extension"))
+ test_pad_calls_freplace(skel);
+
+ /* A covered throw the sweep leaves last, where the default exception
+ * callback is patched in; the pad's bit needs r6-r9 still spilled.
+ */
+ if (test__start_subtest("pad_before_throw"))
+ run_shape(skel, skel->progs.entry_pad_first, 101, THROW_COOKIE,
+ RAN_PAD_FIRST);
+
+ /* A region whose last instruction is a 16-byte one, so that end - 1
+ * names the half of it that is not an instruction.
+ */
+ if (test__start_subtest("region_ends_on_ldimm64"))
+ run_shape(skel, skel->progs.entry_wide_rec, 101, THROW_COOKIE,
+ RAN_WIDE_REC);
+
+ /* A pad that reloads from and writes to its own frame's stack, which a
+ * JIT addressing the frame through the stack pointer gets wrong.
+ */
+ if (test__start_subtest("pad_uses_own_frame"))
+ run_shape(skel, skel->progs.entry_pad_stack, 101, THROW_COOKIE,
+ RAN_PAD_STACK);
+
+ /* The same, with an uncovered frame between the throw and the pad. */
+ if (test__start_subtest("pad_two_frames_up"))
+ run_shape(skel, skel->progs.entry_deep_pad, 101, THROW_COOKIE,
+ RAN_DEEP_PAD);
+
+ /* An extension program with a cleanup table of its own. */
+ if (test__start_subtest("extension_carries_table"))
+ test_ext_table(skel);
+
+ /* A pad terminated by _Unwind_Resume, which libbpf maps onto the kfunc;
+ * every other program here calls bpf_unwind_resume directly.
+ */
+ if (test__start_subtest("resume_alias"))
+ run_shape(skel, skel->progs.entry_resume_alias, 101,
+ THROW_COOKIE, RAN_RESUME_ALIAS);
+
+ /* A pad that calls a subprogram which tail calls, array empty and then
+ * populated: the tail call releases only the callee's own prologue.
+ */
+ if (test__start_subtest("pad_callee_tail_call")) {
+ int key = 0, prog_fd = bpf_program__fd(skel->progs.pad_tc_target);
+
+ skel->bss->pad_tc_target_ran = 0;
+ skel->bss->pad_runs = 0;
+ run_shape(skel, skel->progs.entry_pad_tail_call, 101,
+ THROW_COOKIE, RAN_PAD_TAIL_CALL);
+ ASSERT_EQ(skel->bss->pad_tc_target_ran, 0, "target not run");
+ ASSERT_EQ(skel->bss->pad_runs, 1, "pad_runs");
+
+ if (ASSERT_OK(bpf_map_update_elem(bpf_map__fd(skel->maps.pad_tc_table),
+ &key, &prog_fd, BPF_ANY),
+ "populate pad_tc_table")) {
+ skel->bss->pad_runs = 0;
+ run_shape(skel, skel->progs.entry_pad_tail_call, 101,
+ THROW_COOKIE, RAN_PAD_TAIL_CALL);
+ ASSERT_EQ(skel->bss->pad_tc_target_ran, 1, "target ran");
+ ASSERT_EQ(skel->bss->pad_runs, 1, "pad_runs");
+ }
+ }
+
+ /* The same, into a target that carries a table and throws: that target
+ * is a boundary, so the outer pad runs once, not twice.
+ */
+ if (test__start_subtest("pad_callee_tail_call_throws")) {
+ int key = 0, prog_fd = bpf_program__fd(skel->progs.pad_tc_throw_target);
+
+ if (ASSERT_OK(bpf_map_update_elem(bpf_map__fd(skel->maps.pad_tc_table),
+ &key, &prog_fd, BPF_ANY),
+ "populate pad_tc_table")) {
+ skel->bss->pad_runs = 0;
+ skel->bss->tc_target_pad_runs = 0;
+ run_shape(skel, skel->progs.entry_pad_tail_call, 101,
+ THROW_COOKIE, RAN_PAD_TAIL_CALL);
+ /* The target cleaned up after itself, once. */
+ ASSERT_EQ(skel->bss->tc_target_pad_runs, 1,
+ "tc_target_pad_runs");
+ /* And the outer pad was not started over. */
+ ASSERT_EQ(skel->bss->pad_runs, 1, "pad_runs");
+ }
+ }
+
+ /* A pad whose first instruction opt_remove_nops() deletes: the record
+ * has to follow the pad rather than be dropped with the nop.
+ */
+ if (test__start_subtest("nop_at_pad_head"))
+ run_shape(skel, skel->progs.entry_nop_pad, 101, THROW_COOKIE,
+ RAN_NOP_PAD);
+
+ /* A pad that indexes its frame's stack by a register the frame set
+ * before the throwing call, marked precise back across the unwind edge.
+ */
+ if (test__start_subtest("pad_var_stack_offset"))
+ run_shape(skel, skel->progs.entry_var_stack, 101, THROW_COOKIE,
+ RAN_VAR_STACK);
+
+ /* The same, where the throw is in a global subprogram, which the
+ * verifier walks without a frame of its own.
+ */
+ if (test__start_subtest("pad_over_global_subprog"))
+ run_shape(skel, skel->progs.entry_global_pad, 101, THROW_COOKIE,
+ RAN_GLOBAL_PAD);
+
+ exceptions_cleanup_shapes__destroy(skel);
+}
+
+void test_exceptions_cleanup(void)
+{
+ char log[8192] = {};
+
+ LIBBPF_OPTS(bpf_object_open_opts, opts,
+ .kernel_log_buf = log,
+ .kernel_log_size = sizeof(log));
+ struct exceptions_cleanup *skel;
+ int err;
+
+ skel = exceptions_cleanup__open_opts(&opts);
+ if (!ASSERT_OK_PTR(skel, "open"))
+ return;
+
+ err = exceptions_cleanup__load(skel);
+ if (err) {
+ if (err == -EOPNOTSUPP &&
+ strstr(log, "exception cleanup needs a JIT that can dispatch landing pads"))
+ test__skip();
+ else if (!ASSERT_OK(err, "load"))
+ fprintf(stderr, "%s", log);
+ exceptions_cleanup__destroy(skel);
+ return;
+ }
+
+ /* No throw: foo3 returns 1 ^ 1 == 0, foo2 adds one, no pad runs. */
+ if (test__start_subtest("no_throw"))
+ run(skel, 1, 1, 0);
+
+ /* foo3 throws; every pad runs and the cookie is delivered at entry. */
+ if (test__start_subtest("throw_from_foo3"))
+ run(skel, 101, THROW_COOKIE, PADS_FOO3_THREW);
+
+ /* foo3 returns 2 ^ 1 == 3, so foo2 throws from its own second region;
+ * foo3's frame is long gone, so its pad must not run.
+ */
+ if (test__start_subtest("throw_from_foo2"))
+ run(skel, 2, THROW_COOKIE, PADS_FOO2_THREW);
+
+ exceptions_cleanup__destroy(skel);
+
+ if (test__start_subtest("light_skeleton"))
+ test_light_skeleton();
+
+ test_shapes();
+
+ RUN_TESTS(exceptions_cleanup_fail);
+}
diff --git a/tools/testing/selftests/bpf/progs/exceptions_cleanup.c b/tools/testing/selftests/bpf/progs/exceptions_cleanup.c
new file mode 100644
index 0000000000000..95199a2828faa
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/exceptions_cleanup.c
@@ -0,0 +1,152 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+#include "exceptions_cleanup.h"
+
+static __used __noinline void __kfunc_btf_anchor(void)
+{
+ bpf_throw(0);
+ bpf_rcu_read_lock();
+ bpf_rcu_read_unlock();
+ bpf_preempt_disable();
+ bpf_preempt_enable();
+ bpf_unwind_resume();
+}
+
+__u64 input = 0;
+__u64 pads_ran = 0;
+__u64 result = 0;
+
+static __used __noinline __u64 foo3(__u64 x)
+{
+ bpf_preempt_disable();
+ if (x > 100)
+ asm volatile (
+ "r1 = %[cookie];"
+ "1:" "call bpf_throw;" /* cleanup region */
+ "2:"
+ "goto 3f;"
+ "4:" /* landing pad */
+ "r7 = r0;"
+ "call bpf_preempt_enable;"
+ PAD_RAN("%[ran]")
+ "r1 = r7;"
+ "call bpf_unwind_resume;"
+ "3:"
+ CLEANUP_REC("1b", "2b", "4b")
+ :
+ : [cookie]"i"(THROW_COOKIE), [ran]"i"(RAN_FOO3_PREEMPT),
+ __imm_addr(pads_ran)
+ : __clobber_all);
+ bpf_preempt_enable();
+ return x ^ 1;
+}
+
+__u64 never = 0;
+
+static __used __naked __noinline void drop_glue(void)
+{
+ asm volatile (
+ PAD_RAN("%[ran]")
+ "exit;"
+ :
+ : [ran]"i"(RAN_FOO2_DROP), __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+static __used __naked __noinline __u64 foo2(void)
+{
+ asm volatile (
+ "r6 = r1;"
+ "call bpf_rcu_read_lock;"
+ "r1 = r6;"
+"1:" "call foo3;" /* cleanup region #1 */
+"2:"
+ "r6 = r0;"
+ "if r6 == 0 goto 5f;"
+ "r1 = %[cookie];"
+"3:" "call bpf_throw;" /* cleanup region #2 */
+"4:"
+ "r0 = 0;"
+ "exit;"
+"5:"
+ "call bpf_rcu_read_unlock;"
+ "r0 = r6;"
+ "r0 += 1;"
+ "exit;"
+"6:" /* landing pad, shared by both regions */
+ "call drop_glue;"
+ "call bpf_rcu_read_unlock;"
+ PAD_RAN("%[ran_rcu]")
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "6b")
+ CLEANUP_REC("3b", "4b", "6b")
+ :
+ : [cookie]"i"(THROW_COOKIE), [ran_rcu]"i"(RAN_FOO2_RCU),
+ __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+static __used __naked __noinline void foo1v(void)
+{
+ asm volatile (
+ "call bpf_preempt_disable;"
+ "r1 = %[input] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+"1:" "call foo2;" /* cleanup region */
+"2:"
+ "r6 = r0;"
+ "call bpf_preempt_enable;"
+ "r1 = %[result] ll;"
+ "*(u64 *)(r1 + 0) = r6;"
+ "goto 7f;"
+"8:" /* landing pad */
+ "call bpf_preempt_enable;"
+ PAD_RAN("%[ran]")
+ "goto 9f;"
+"7:" /* the frame's own exit block */
+ "r0 = 0;"
+ "exit;"
+"9:" /* shared resume block */
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "8b")
+ :
+ : [ran]"i"(RAN_FOO1V_PREEMPT), __imm_addr(input),
+ __imm_addr(result), __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+static __used __naked __noinline void bump(void)
+{
+ asm volatile (
+ PAD_RAN("%[ran]")
+ "r1 = %[never] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+ "if r1 == 0 goto 1f;"
+ "r1 = 0;"
+ "call bpf_throw;"
+"1:"
+ "exit;" /* r0 deliberately left alone */
+ :
+ : [ran]"i"(RAN_BUMP), __imm_addr(never), __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+__noinline __u64 foo1(void)
+{
+ bump();
+ foo1v();
+ return result;
+}
+
+SEC("syscall")
+int entry(void *ctx)
+{
+ return foo1();
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/exceptions_cleanup_ext_table.c b/tools/testing/selftests/bpf/progs/exceptions_cleanup_ext_table.c
new file mode 100644
index 0000000000000..d14db48d6b29b
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/exceptions_cleanup_ext_table.c
@@ -0,0 +1,48 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+#include "exceptions_cleanup.h"
+
+__u64 ext_pad_ran = 0;
+
+/* Without a 32-bit int in BTF, libbpf's dummy_ksym var gets type id 0. */
+int btf_int_anchor;
+
+static __used __noinline void __kfunc_btf_anchor(void)
+{
+ bpf_throw(0);
+ bpf_preempt_disable();
+ bpf_preempt_enable();
+ bpf_unwind_resume();
+}
+
+static __used __naked __noinline __u64 ext_frame(void)
+{
+ asm volatile (
+ "call bpf_preempt_disable;"
+ "r1 = %[cookie];"
+"1:" "call bpf_throw;" /* cleanup region */
+"2:"
+ "exit;"
+"3:" /* landing pad */
+ "call bpf_preempt_enable;"
+ "r1 = %[ext_pad_ran] ll;"
+ "r2 = 1;"
+ "*(u64 *)(r1 + 0) = r2;"
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : [cookie]"i"(THROW_COOKIE), __imm_addr(ext_pad_ran)
+ : __clobber_all);
+}
+
+SEC("freplace/fr_callee")
+__u64 new_fr_callee(__u64 x)
+{
+ return ext_frame();
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c b/tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c
new file mode 100644
index 0000000000000..db6ac7d8bd6de
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c
@@ -0,0 +1,662 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_experimental.h"
+#include "bpf_misc.h"
+#include "../test_kmods/bpf_testmod_kfunc.h"
+#include "exceptions_cleanup.h"
+
+__u64 input = 0;
+
+static __used __noinline void __kfunc_btf_anchor(void)
+{
+ bpf_throw(0);
+ bpf_preempt_disable();
+ bpf_preempt_enable();
+ bpf_unwind_resume();
+}
+
+/* 1. A subprogram that may unwind, also used as a helper callback:
+ * bpf_loop()'s own kernel frame would end the walk before it found a
+ * boundary.
+ */
+static int throwing_cb(__u32 idx, void *ctx)
+{
+ bpf_throw(0xbad);
+ return 0;
+}
+
+static __used __naked __noinline __u64 cb_frame(void)
+{
+ asm volatile (
+ "call bpf_preempt_disable;"
+"1:" "call throwing_cb;" /* cleanup region */
+"2:"
+ "call bpf_preempt_enable;"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ "call bpf_preempt_enable;"
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ ::: __clobber_all);
+}
+
+SEC("?syscall")
+__failure __msg("may unwind and is used as a callback")
+int callback_may_unwind(void *ctx)
+{
+ bpf_loop(1, throwing_cb, NULL, 0);
+ return cb_frame();
+}
+
+/* 2. A landing pad that reaches both an unwind resume and a plain exit, so
+ * nothing says whether it is a cleanup pad or a catch pad.
+ */
+static __used __naked __noinline __u64 inner_throw(void)
+{
+ asm volatile (
+ "r1 = 1;"
+ "call bpf_throw;"
+ "r0 = 0;"
+ "exit;"
+ ::: __clobber_all);
+}
+
+static __used __naked __noinline __u64 ambiguous_pad_frame(void)
+{
+ asm volatile (
+ "r6 = r1;"
+ "call bpf_preempt_disable;"
+"1:" "call inner_throw;" /* cleanup region */
+"2:"
+ "call bpf_preempt_enable;"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad: two ways out */
+ "call bpf_preempt_enable;"
+ "if r6 > 10 goto 4f;"
+ "call bpf_unwind_resume;"
+ "exit;"
+"4:"
+ "r0 = 0;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ ::: __clobber_all);
+}
+
+SEC("?syscall")
+__failure __msg("reaches both bpf_unwind_resume() and a plain exit")
+int ambiguous_landing_pad(void *ctx)
+{
+ return ambiguous_pad_frame();
+}
+
+/* 3. A throw from inside a landing pad: a second walk over the frames the
+ * first one is in the middle of discarding.
+ */
+static __used __naked __noinline __u64 throw_in_pad_frame(void)
+{
+ asm volatile (
+ "call bpf_preempt_disable;"
+"1:" "call inner_throw;" /* cleanup region */
+"2:"
+ "call bpf_preempt_enable;"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad that throws again */
+ "call bpf_preempt_enable;"
+ "r1 = 2;"
+ "call bpf_throw;"
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ ::: __clobber_all);
+}
+
+SEC("?syscall")
+__failure __msg("can throw while an exception is in flight")
+int throw_from_landing_pad(void *ctx)
+{
+ return throw_in_pad_frame();
+}
+
+/* 4. A cleanup table in a program that also installs an exception callback,
+ * two different answers to what runs on the way out.
+ */
+__noinline int unused_exc_cb(u64 cookie)
+{
+ return 0;
+}
+
+static __used __naked __noinline __u64 cb_and_table_frame(void)
+{
+ asm volatile (
+ "call bpf_preempt_disable;"
+ "r1 = 9;"
+"1:" "call bpf_throw;" /* cleanup region */
+"2:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ "call bpf_preempt_enable;"
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ ::: __clobber_all);
+}
+
+SEC("?syscall")
+__exception_cb(unused_exc_cb)
+__failure __msg("cannot be combined with an exception callback")
+int table_with_exception_cb(void *ctx)
+{
+ return cb_and_table_frame();
+}
+
+__u64 never;
+
+/* 5. A landing pad that calls a subprogram which can throw. Not case 3: the
+ * throw is in another subprogram, so what catches it is the walk of the pad's
+ * body, off subprog_info.might_throw.
+ */
+static __used __noinline void pad_callee_that_throws(void)
+{
+ if (never)
+ bpf_throw(0);
+}
+
+static __used __naked __noinline __u64 pad_calls_thrower_frame(void)
+{
+ asm volatile (
+ "call bpf_preempt_disable;"
+ "r1 = 11;"
+"1:" "call bpf_throw;" /* cleanup region */
+"2:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ "call pad_callee_that_throws;" /* ...which can throw: refused */
+ "call bpf_preempt_enable;"
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ ::: __clobber_all);
+}
+
+SEC("?syscall")
+__failure __msg("which can throw while an exception is in flight")
+int pad_calls_thrower(void *ctx)
+{
+ return pad_calls_thrower_frame();
+}
+
+/* 6. A catch pad: it ends in a plain exit rather than a resume, and a walker
+ * that calls pads as subroutines cannot hand a frame back its own execution.
+ */
+static __used __naked __noinline __u64 catch_pad_frame(void)
+{
+ asm volatile (
+ "call bpf_preempt_disable;"
+ "r1 = 12;"
+"1:" "call bpf_throw;" /* cleanup region */
+"2:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* catch pad: no resume, it stops here */
+ "call bpf_preempt_enable;"
+ "r0 = 0;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ ::: __clobber_all);
+}
+
+SEC("?syscall")
+__failure __msg("is not supported yet, only cleanup pads that resume")
+int catch_landing_pad(void *ctx)
+{
+ return catch_pad_frame();
+}
+
+/* 7. An exception reaching the boundary of a program type that constrains its
+ * return value: delivery makes the cookie that return value, and fentry has
+ * to return 0.
+ */
+static __used __naked __noinline __u64 boundary_throw_frame(void)
+{
+ asm volatile (
+ "call bpf_preempt_disable;"
+ "r1 = 7;"
+"1:" "call bpf_throw;" /* cleanup region */
+"2:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ "call bpf_preempt_enable;"
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ ::: __clobber_all);
+}
+
+SEC("?fentry/bpf_fentry_test1")
+__failure __msg("the register R1 has smin=7 smax=7 should have been in [0, 0]")
+int boundary_delivers(void *ctx)
+{
+ return boundary_throw_frame();
+}
+
+/* 8. A bpf_unwind_resume() outside any landing pad. Both JITs lower it as the
+ * way back out of a pad, which in ordinary code leaves a live frame standing
+ * with its epilogue skipped.
+ */
+static __used __naked __noinline __u64 stray_resume_frame(void)
+{
+ asm volatile (
+ "call bpf_preempt_disable;"
+ "r1 = 13;"
+"1:" "call bpf_throw;" /* cleanup region */
+"2:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ "call bpf_preempt_enable;"
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ ::: __clobber_all);
+}
+
+SEC("?syscall")
+__failure __msg("is not in an exception cleanup landing pad")
+int resume_outside_pad(void *ctx)
+{
+ /* Never taken, but reachable, which is all the verifier needs. */
+ if (never)
+ bpf_unwind_resume();
+ return stray_resume_frame();
+}
+
+/* 9. A bpf_unwind_resume() in a subprogram a landing pad calls. The verifier's
+ * walk cannot tell it from a resume in the pad itself -- an exception is in
+ * flight either way -- so the rule is static: a resume sits in a pad body.
+ */
+static __used __naked __noinline void resume_in_callee(void)
+{
+ asm volatile (
+ "call bpf_unwind_resume;"
+ "exit;"
+ ::: __clobber_all);
+}
+
+static __used __naked __noinline __u64 pad_calls_resumer_frame(void)
+{
+ asm volatile (
+ "call bpf_preempt_disable;"
+ "r1 = 14;"
+"1:" "call bpf_throw;" /* cleanup region */
+"2:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ "call bpf_preempt_enable;"
+ "call resume_in_callee;" /* ...which resumes: refused */
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ ::: __clobber_all);
+}
+
+SEC("?syscall")
+__failure __msg("is not in an exception cleanup landing pad")
+int resume_in_pad_callee(void *ctx)
+{
+ return pad_calls_resumer_frame();
+}
+
+/* 10. A bpf_unwind_resume() in a program carrying no cleanup table, where
+ * that static rule does not run at all. do_check() refuses it on the state
+ * not unwinding, and has to: the JITs lower every one of these the same way.
+ */
+static __used __naked __noinline __u64 no_table_resume_frame(void)
+{
+ asm volatile (
+ "call bpf_unwind_resume;"
+ "r0 = 0;"
+ "exit;"
+ ::: __clobber_all);
+}
+
+SEC("?syscall")
+__failure __msg("reached without an exception in flight")
+int resume_without_table(void *ctx)
+{
+ return no_table_resume_frame();
+}
+
+/* 11. A landing pad that is itself a covered call site, so an exception out
+ * of it would have nowhere to go. Hand-written only: LLVM sinks a function's
+ * pads past every range it emits.
+ */
+static __used __naked __noinline __u64 nested_pad_frame(void)
+{
+ asm volatile (
+ "call bpf_preempt_disable;"
+"1:" "call inner_throw;" /* first cleanup region */
+"2:"
+ "call bpf_preempt_enable;"
+ "r0 = 0;"
+ "exit;"
+"3:" /* first pad, second region's call */
+ "call bpf_preempt_enable;"
+"4:"
+ "call bpf_unwind_resume;"
+ "exit;"
+"5:" /* second pad */
+ "call bpf_preempt_enable;"
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ CLEANUP_REC("3b", "4b", "5b")
+ ::: __clobber_all);
+}
+
+SEC("?syscall")
+__failure __msg("is inside the call-site range of")
+int nested_landing_pad(void *ctx)
+{
+ return nested_pad_frame();
+}
+
+/* 12. A tail call in a landing pad: it unwinds the prologue off the stack
+ * pointer, which in a pad is the walker's.
+ */
+struct {
+ __uint(type, BPF_MAP_TYPE_PROG_ARRAY);
+ __uint(max_entries, 1);
+ __uint(key_size, sizeof(__u32));
+ __uint(value_size, sizeof(__u32));
+} tc_map SEC(".maps");
+
+static __used __naked __noinline __u64 tail_call_pad_frame(void)
+{
+ asm volatile (
+ "r6 = r1;"
+"1:" "call inner_throw;" /* cleanup region */
+"2:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ "r1 = r6;"
+ "r2 = %[tc_map] ll;"
+ "r3 = 0;"
+ "call %[bpf_tail_call];"
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : __imm(bpf_tail_call), __imm_addr(tc_map)
+ : __clobber_all);
+}
+
+SEC("?syscall")
+__failure __msg("is in an exception cleanup landing pad")
+int tail_call_in_pad(void *ctx)
+{
+ return tail_call_pad_frame();
+}
+
+#if defined(__BPF_FEATURE_STACK_ARGUMENT)
+
+/* 13. A call that passes an argument on the stack, in a landing pad: the
+ * outgoing area the callee reads is not the one the caller wrote, the frame
+ * being the unwinding one and the stack pointer the walker's.
+ */
+static __used __noinline __u64 six_args(__u64 a, __u64 b, __u64 c, __u64 d,
+ __u64 e, __u64 f)
+{
+ return a + b + c + d + e + f;
+}
+
+static __used __naked __noinline __u64 stack_arg_pad_frame(void)
+{
+ asm volatile (
+ "call bpf_preempt_disable;"
+"1:" "call inner_throw;" /* cleanup region */
+"2:"
+ "call bpf_preempt_enable;"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ "call bpf_preempt_enable;"
+ "r1 = 1;"
+ "r2 = 2;"
+ "r3 = 3;"
+ "r4 = 4;"
+ "r5 = 5;"
+ "*(u64 *)(r11 - 8) = 6;" /* the sixth argument */
+ "call six_args;"
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ ::: __clobber_all);
+}
+
+SEC("?syscall")
+__failure __msg("on-stack call argument in an exception cleanup landing pad")
+int stack_arg_in_pad(void *ctx)
+{
+ return stack_arg_pad_frame();
+}
+
+/* 14. The same, reached the other way: a kfunc whose by-value argument runs
+ * past the five argument registers, where the JIT fills the outgoing area and
+ * the rule above has no store to catch. The C call gives the extern its BTF.
+ */
+static __used __noinline void __nofit_btf_anchor(void)
+{
+ struct prog_test_pair_arg s = {};
+
+ bpf_kfunc_call_test_pair_arg_nofit(1, 2, 3, 4, s);
+}
+
+static __used __naked __noinline __u64 kfunc_arg_pad_frame(void)
+{
+ asm volatile (
+ "call bpf_preempt_disable;"
+"1:" "call inner_throw;" /* cleanup region */
+"2:"
+ "call bpf_preempt_enable;"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ "call bpf_preempt_enable;"
+ "r1 = 1;"
+ "r2 = 2;"
+ "r3 = 3;"
+ "r4 = 4;"
+ "r5 = 5;"
+ "call bpf_kfunc_call_test_pair_arg_nofit;"
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ ::: __clobber_all);
+}
+
+SEC("?syscall")
+__failure __msg("on-stack call argument in an exception cleanup landing pad")
+int kfunc_stack_arg_in_pad(void *ctx)
+{
+ return kfunc_arg_pad_frame();
+}
+
+#endif /* __BPF_FEATURE_STACK_ARGUMENT */
+
+/* 15. A landing pad entered by ordinary control flow, arriving with none of
+ * what the walker sets up. Nothing static sees it -- the resume really is in
+ * a pad body -- so do_check() refuses it on the state not unwinding.
+ */
+static __used __naked __noinline __u64 jump_into_pad_frame(void)
+{
+ asm volatile (
+ "r1 = %[input] ll;"
+ "r6 = *(u64 *)(r1 + 0);"
+ "if r6 > 7 goto 4f;" /* an ordinary branch into the pad */
+"1:" "call inner_throw;" /* cleanup region */
+"2:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ "r7 = r0;"
+"4:" /* ... and its second instruction */
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : __imm_addr(input)
+ : __clobber_all);
+}
+
+SEC("?syscall")
+__failure __msg("reached without an exception in flight")
+int jump_into_pad(void *ctx)
+{
+ return jump_into_pad_frame();
+}
+
+#if defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64)
+
+/* 16. A landing pad that reaches an indirect jump, which cannot be told from
+ * a catch pad. SEC("socket") because a jump table entry is an offset from the
+ * program's section symbol, and "?syscall" is not a name assembly can use.
+ */
+static __used __naked __noinline void gotox_thrower(void)
+{
+ asm volatile (
+ "r1 = 15;"
+ "call bpf_throw;"
+ "exit;"
+ ::: __clobber_all);
+}
+
+SEC("socket")
+__failure __msg("reaches an indirect jump")
+__naked void gotox_in_pad(void)
+{
+ asm volatile (
+ ".pushsection .jumptables,\"\",@progbits;"
+"jt0_%=:"
+ ".quad l0_%= - socket;"
+ ".quad l1_%= - socket;"
+ ".size jt0_%=, 16;"
+ ".global jt0_%=;"
+ ".popsection;"
+
+"1:" "call gotox_thrower;" /* cleanup region */
+"2:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ "r1 = jt0_%= ll;"
+ "r1 += 8;"
+ "r2 = *(u64 *)(r1 + 0);"
+ /* gotox r2, as a raw insn: the mnemonic only reached the LLVM
+ * assembler in llvm 22, and BPF_RAW_INSN() needs <linux/bpf.h>, which
+ * vmlinux.h rules out.
+ */
+ ".8byte 0x20d;"
+"l0_%=:"
+ "call bpf_unwind_resume;"
+ "exit;"
+"l1_%=:"
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ ::: __clobber_all);
+}
+
+#endif /* x86 || arm64 */
+
+/* 17. A BPF_LD_[ABS|IND] in a landing pad. A failed load leaves the
+ * subprogram through the hidden "r0 = 0; exit" gen_ld_abs() patches in, and
+ * that exit is the epilogue, which unwinds a stack the pad does not own.
+ */
+static __used __naked __noinline __u64 ld_abs_pad_frame(void)
+{
+ asm volatile (
+ "r6 = r1;" /* the skb BPF_LD_ABS reads */
+"1:" "call inner_throw;" /* cleanup region */
+"2:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ /* r0 = *(u32 *)skb[0]. Spelled as a raw insn because BPF_LD_ABS()
+ * needs <linux/filter.h>, which this file cannot have -- vmlinux.h
+ * already defines the uapi enums.
+ */
+ ".8byte 0x20;"
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ ::: __clobber_all);
+}
+
+SEC("?tc")
+__failure __msg("is in an exception cleanup landing pad")
+__naked void ld_abs_in_pad(void)
+{
+ asm volatile (
+ "call ld_abs_pad_frame;"
+ "exit;"
+ ::: __clobber_all);
+}
+
+/* 18. A bpf_unwind_resume() in a subprogram a landing pad called, which has a
+ * pad of its own and reached it by ordinary control flow. Not case 9: an
+ * exception is in flight, but this is not the frame whose pad the walker ran.
+ */
+static __used __naked __noinline __u64 own_pad_callee(void)
+{
+ asm volatile (
+ "r1 = %[input] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+ "if r1 == 0 goto 3f;" /* an ordinary branch into its pad */
+"1:" "call bpf_preempt_disable;" /* cleanup region: nothing that throws */
+ "call bpf_preempt_enable;"
+"2:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* its landing pad */
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : __imm_addr(input)
+ : __clobber_all);
+}
+
+static __used __naked __noinline __u64 pad_calls_own_pad_frame(void)
+{
+ asm volatile (
+"1:" "call inner_throw;" /* cleanup region */
+"2:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad, which calls the above */
+ "call own_pad_callee;"
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ ::: __clobber_all);
+}
+
+SEC("?syscall")
+__failure __msg("is in frame 2, not frame 1 whose landing pad the exception entered")
+int resume_in_callee_own_pad(void *ctx)
+{
+ return pad_calls_own_pad_frame();
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/exceptions_cleanup_freplace.c b/tools/testing/selftests/bpf/progs/exceptions_cleanup_freplace.c
new file mode 100644
index 0000000000000..afb358fd3d405
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/exceptions_cleanup_freplace.c
@@ -0,0 +1,17 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "exceptions_cleanup.h"
+
+/* Without a 32-bit int in BTF, libbpf's dummy_ksym var gets type id 0. */
+int btf_int_anchor;
+
+SEC("freplace/fr_callee")
+__u64 new_fr_callee(__u64 x)
+{
+ bpf_throw(THROW_COOKIE);
+ return 0;
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/exceptions_cleanup_light.c b/tools/testing/selftests/bpf/progs/exceptions_cleanup_light.c
new file mode 100644
index 0000000000000..de7ae478d67f3
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/exceptions_cleanup_light.c
@@ -0,0 +1,39 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+#include "exceptions_cleanup.h"
+
+static __used __noinline void __kfunc_btf_anchor(void)
+{
+ bpf_throw(0);
+ bpf_preempt_disable();
+ bpf_preempt_enable();
+ bpf_unwind_resume();
+}
+
+__u64 pads_ran = 0;
+
+SEC("syscall")
+__naked int entry_light(void)
+{
+ asm volatile (
+ "call bpf_preempt_disable;"
+ "r1 = %[cookie];"
+"1:" "call bpf_throw;" /* cleanup region */
+"2:"
+ "exit;"
+"3:" /* landing pad */
+ "call bpf_preempt_enable;"
+ PAD_RAN("%[ran]")
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : [cookie]"i"(THROW_COOKIE), [ran]"i"(RAN_LIGHT),
+ __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/exceptions_cleanup_pad_freplace.c b/tools/testing/selftests/bpf/progs/exceptions_cleanup_pad_freplace.c
new file mode 100644
index 0000000000000..eabac6baabb70
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/exceptions_cleanup_pad_freplace.c
@@ -0,0 +1,17 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "exceptions_cleanup.h"
+
+/* Without a 32-bit int in BTF, libbpf's dummy_ksym var gets type id 0. */
+int btf_int_anchor;
+
+SEC("freplace/pad_callee")
+__u64 new_pad_callee(__u64 x)
+{
+ bpf_throw(INNER_COOKIE);
+ return 0;
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c b/tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c
new file mode 100644
index 0000000000000..493f8e0306528
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/exceptions_cleanup_shapes.c
@@ -0,0 +1,984 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+#include "exceptions_cleanup.h"
+
+#define PAD_COUNT \
+ "r1 = %[pad_runs] ll;" \
+ "r2 = *(u64 *)(r1 + 0);" \
+ "r2 += 1;" \
+ "*(u64 *)(r1 + 0) = r2;"
+
+static __used __noinline void __kfunc_btf_anchor(void)
+{
+ bpf_throw(0);
+ bpf_rcu_read_lock();
+ bpf_rcu_read_unlock();
+ bpf_preempt_disable();
+ bpf_preempt_enable();
+ bpf_unwind_resume();
+}
+
+__u64 input = 0;
+__u64 outer_input = 0;
+__u64 magic = 0x5eed;
+__u64 pads_ran = 0;
+__u64 pad_runs = 0;
+
+/* 1. Everything a cleanup table leaves dead: a throw's continuation, the tail
+ * after a resume with an ld_imm64 and a branch in it, and the block only that
+ * continuation reaches.
+ */
+static __used __naked __noinline __u64 sweep_frame(void)
+{
+ asm volatile (
+ "r1 = %[input] ll;"
+ "r6 = *(u64 *)(r1 + 0);"
+ "call bpf_preempt_disable;"
+ "if r6 < 101 goto 6f;"
+ "r1 = %[cookie];"
+"1:" "call bpf_throw;" /* cleanup region */
+"2:"
+ "goto 3f;"
+"4:" /* landing pad */
+ "r7 = r0;"
+ "call bpf_preempt_enable;"
+ PAD_RAN("%[ran]")
+ "r1 = r7;"
+ "call bpf_unwind_resume;"
+ "r1 = %[pads_ran] ll;"
+ "r2 = *(u64 *)(r1 + 0);"
+ "if r2 == 0 goto 5f;"
+ "call bpf_preempt_enable;"
+ "r0 = 7;"
+ "exit;"
+"5:"
+ "r0 = 8;"
+ "exit;"
+"3:" /* dead: only the dead goto reaches it */
+ "r0 = 9;"
+ "exit;"
+"6:" /* live: the ordinary return */
+ "call bpf_preempt_enable;"
+ "r0 = 0;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "4b")
+ :
+ : [cookie]"i"(THROW_COOKIE), [ran]"i"(RAN_SWEEP),
+ __imm_addr(input), __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+SEC("syscall")
+int entry_sweep(void *ctx)
+{
+ return sweep_frame();
+}
+
+/* 2. A callee called from both a covered and an uncovered site: the pad
+ * belongs to the call site, not the callee. Either site can throw, and the RCU
+ * lock between them is what an unwind out of the uncovered one leaks.
+ */
+static __used __noinline __u64 shared_callee(__u64 x)
+{
+ if (x > 100)
+ bpf_throw(THROW_COOKIE);
+ return x + 1;
+}
+
+static __used __naked __noinline __u64 shared_frame(void)
+{
+ asm volatile (
+ "r1 = %[input] ll;"
+ "r6 = *(u64 *)(r1 + 0);"
+ "r1 = %[outer_input] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+ "call shared_callee;" /* uncovered: no pad for its unwind */
+ "call bpf_rcu_read_lock;"
+ "r1 = r6;"
+"1:" "call shared_callee;" /* cleanup region */
+"2:"
+ "r6 = r0;"
+ "call bpf_rcu_read_unlock;"
+ "r0 = r6;"
+ "exit;"
+"3:" /* landing pad */
+ "r7 = r0;"
+ "call bpf_rcu_read_unlock;"
+ PAD_RAN("%[ran]")
+ "r1 = r7;"
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : [ran]"i"(RAN_SHARED), __imm_addr(input), __imm_addr(outer_input),
+ __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+SEC("syscall")
+int entry_shared(void *ctx)
+{
+ return shared_frame();
+}
+
+/* 3. A landing pad that reads its frame's callee-saved registers: the callee
+ * overwrites r6-r9 before it throws, so the check passes only if the walker
+ * found the spill in the discarded callee's prologue.
+ */
+#define LOAD_MAGIC_REGS \
+ "r1 = %[magic] ll;" \
+ "r6 = *(u64 *)(r1 + 0);" \
+ "r7 = r6;" \
+ "r7 += 1;" \
+ "r8 = r6;" \
+ "r8 += 2;" \
+ "r9 = r6;" \
+ "r9 += 3;"
+
+/* Set @bit only if r6-r9 still hold what LOAD_MAGIC_REGS put there. */
+#define CHECK_MAGIC_REGS(bit) \
+ "r1 = %[magic] ll;" \
+ "r2 = *(u64 *)(r1 + 0);" \
+ "if r6 != r2 goto 9f;" \
+ "r2 += 1;" \
+ "if r7 != r2 goto 9f;" \
+ "r2 += 1;" \
+ "if r8 != r2 goto 9f;" \
+ "r2 += 1;" \
+ "if r9 != r2 goto 9f;" \
+ PAD_RAN(bit) \
+ "9:"
+
+static __used __naked __noinline __u64 regs_thrower(void)
+{
+ asm volatile (
+ /* Not this frame's to keep, and that is the point. */
+ "r6 = 0xdead;"
+ "r7 = 0xbeef;"
+ "r8 = 0xcafe;"
+ "r9 = 0xf00d;"
+ "r1 = %[cookie];"
+ "call bpf_throw;"
+ "r0 = 0;"
+ "exit;"
+ :
+ : [cookie]"i"(THROW_COOKIE)
+ : __clobber_all);
+}
+
+static __used __naked __noinline __u64 regs_frame(void)
+{
+ asm volatile (
+ LOAD_MAGIC_REGS
+ "call bpf_preempt_disable;"
+"1:" "call regs_thrower;" /* cleanup region */
+"2:"
+ "call bpf_preempt_enable;"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ "call bpf_preempt_enable;"
+ CHECK_MAGIC_REGS("%[ran]")
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : [cookie]"i"(THROW_COOKIE), [ran]"i"(RAN_REGS),
+ __imm_addr(magic), __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+SEC("syscall")
+int entry_regs(void *ctx)
+{
+ return regs_frame();
+}
+
+/* 4. The same, with a tail-call-reachable callee: its prologue pushes the tail
+ * call counter, which moves the spill the walker reads. The array is left
+ * empty; being reachable is the point.
+ */
+struct {
+ __uint(type, BPF_MAP_TYPE_PROG_ARRAY);
+ __uint(max_entries, 1);
+ __uint(key_size, sizeof(__u32));
+ __uint(value_size, sizeof(__u32));
+} jmp_table SEC(".maps");
+
+static __used __noinline __u64 tc_thrower(void *ctx)
+{
+ /* Never taken; its presence is what makes this frame, whose spill the
+ * walker reads, tail-call-reachable.
+ */
+ bpf_tail_call_static(ctx, &jmp_table, 0);
+ asm volatile (
+ "r6 = 0xdead;"
+ "r7 = 0xbeef;"
+ "r8 = 0xcafe;"
+ "r9 = 0xf00d;"
+ "r1 = %[cookie];"
+ "call bpf_throw;"
+ :
+ : [cookie]"i"(THROW_COOKIE)
+ : __clobber_all);
+ return 0;
+}
+
+/* The frame with the pad is the program itself, and __naked: r1 at entry is
+ * the only place to get a context for bpf_tail_call().
+ */
+SEC("syscall")
+__naked int entry_tail_call(void)
+{
+ asm volatile (
+ "*(u64 *)(r10 - 8) = r1;" /* the context, straight from entry */
+ LOAD_MAGIC_REGS
+ "r1 = %[input] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+ "if r1 < 101 goto 8f;"
+ "r1 = *(u64 *)(r10 - 8);"
+"1:" "call tc_thrower;" /* cleanup region */
+"2:"
+"8:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ CHECK_MAGIC_REGS("%[ran]")
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : [ran]"i"(RAN_TAIL_CALL), __imm_addr(input),
+ __imm_addr(magic), __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+/* 5. A landing pad in the main program's own frame. jit_subprogs() compiles it
+ * as func[0], but the walker finds the outer bpf_prog's ksym, so the table has
+ * to be handed over or the pad is never dispatched.
+ */
+SEC("syscall")
+__naked int entry_main_pad(void)
+{
+ asm volatile (
+ LOAD_MAGIC_REGS
+ "r1 = %[input] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+ "if r1 < 101 goto 8f;"
+"1:" "call regs_thrower;" /* cleanup region */
+"2:"
+"8:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ CHECK_MAGIC_REGS("%[ran]")
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : [ran]"i"(RAN_MAIN_PAD), __imm_addr(input),
+ __imm_addr(magic), __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+/* 6. A tail call that is really taken: the target is a program in its own
+ * right, so the walk ends there and this frame's pad does not run. The callee
+ * can also throw, which keeps the pad out of the sweep.
+ */
+struct {
+ __uint(type, BPF_MAP_TYPE_PROG_ARRAY);
+ __uint(max_entries, 1);
+ __uint(key_size, sizeof(__u32));
+ __uint(value_size, sizeof(__u32));
+} taken_table SEC(".maps");
+
+SEC("syscall")
+int tc_target(void *ctx)
+{
+ bpf_throw(THROW_COOKIE);
+ return 0;
+}
+
+static __used __noinline __u64 tc_taken_callee(void *ctx, __u64 x)
+{
+ /* Never true at run time; the verifier cannot know that, and its
+ * unwind out of here is what keeps the caller's pad alive.
+ */
+ if (x == 7)
+ bpf_throw(THROW_COOKIE);
+ bpf_tail_call_static(ctx, &taken_table, 0);
+ return 0;
+}
+
+SEC("syscall")
+__naked int entry_tail_taken(void)
+{
+ asm volatile (
+ "*(u64 *)(r10 - 8) = r1;" /* the context, straight from entry */
+ "r1 = %[input] ll;"
+ /* Unnarrowed, the way entry_freplace hands fr_callee its argument: a
+ * guard here would prune the callee's throw and let the sweep take the
+ * pad, leaving no record for a walk past the boundary to match.
+ */
+ "r2 = *(u64 *)(r1 + 0);"
+ "r1 = *(u64 *)(r10 - 8);"
+"1:" "call tc_taken_callee;" /* cleanup region */
+"2:"
+ "exit;" /* the cookie, delivered at tc_target */
+"3:" /* landing pad: must not run */
+ PAD_RAN("%[ran]")
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : [ran]"i"(RAN_TC_TAKEN), __imm_addr(input), __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+/* 7. An extension program over the callee of a covered call. The walk ends in
+ * the extension's frame, as it does for a tail call target, so the pad does
+ * not run; fr_callee() can also throw by itself.
+ */
+__noinline __u64 fr_callee(__u64 x)
+{
+ if (x == 7)
+ bpf_throw(THROW_COOKIE);
+ return x + 1;
+}
+
+SEC("syscall")
+__naked int entry_freplace(void)
+{
+ asm volatile (
+ "r1 = %[input] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+"1:" "call fr_callee;" /* cleanup region */
+"2:"
+ "exit;"
+"3:" /* landing pad */
+ PAD_RAN("%[ran]")
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : [ran]"i"(RAN_FREPLACE), __imm_addr(input), __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+/* 8. A throwing subprogram named by a BPF_PSEUDO_FUNC on a path never taken.
+ * Handing one to a helper is what is refused, not naming it.
+ */
+static __used __noinline int cb_thrower(__u32 idx, void *ctx)
+{
+ bpf_throw(THROW_COOKIE);
+ return 0;
+}
+
+static __used __noinline __u64 addr_taken_callee(__u64 x)
+{
+ if (x <= 100)
+ return x + 1;
+ bpf_throw(THROW_COOKIE);
+ return bpf_loop(1, cb_thrower, NULL, 0);
+}
+
+SEC("syscall")
+__naked int entry_addr_taken(void)
+{
+ asm volatile (
+ "r1 = %[input] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+"1:" "call addr_taken_callee;" /* cleanup region */
+"2:"
+ "exit;"
+"3:" /* landing pad */
+ PAD_RAN("%[ran]")
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : [ran]"i"(RAN_ADDR_TAKEN), __imm_addr(input), __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+/* 9. A record that covers bpf_throw() itself: the throwing frame is both the
+ * frame the record covers and the boundary, so the pad runs on the way to
+ * delivering the cookie.
+ */
+SEC("syscall")
+__naked int entry_no_subprog(void)
+{
+ asm volatile (
+ "r1 = %[input] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+ "if r1 < 101 goto 8f;"
+ "call bpf_preempt_disable;"
+ "r1 = %[cookie];"
+"1:" "call bpf_throw;" /* cleanup region */
+"2:"
+ "exit;"
+"8:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ "call bpf_preempt_enable;"
+ PAD_RAN("%[ran]")
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : [cookie]"i"(THROW_COOKIE), [ran]"i"(RAN_NO_SUBPROG),
+ __imm_addr(input), __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+/* 10. A landing pad that calls a subprogram an extension can replace. The
+ * load-time rule cannot see it coming, so what stops a nested exception is the
+ * walk ending in the extension's frame; pad_runs says the pad ran once.
+ */
+__noinline __u64 pad_callee(__u64 x)
+{
+ return x + 1;
+}
+
+static __used __noinline __u64 pc_thrower(__u64 x)
+{
+ if (x > 100)
+ bpf_throw(THROW_COOKIE);
+ return x + 1;
+}
+
+static __used __naked __noinline __u64 pad_calls_frame(void)
+{
+ asm volatile (
+ "r1 = %[input] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+"1:" "call pc_thrower;" /* cleanup region */
+"2:"
+ "exit;"
+"3:" /* landing pad */
+ "r6 = r0;"
+ "r1 = 1;"
+ "call pad_callee;" /* an extension can stand in here */
+ PAD_COUNT
+ PAD_RAN("%[ran]")
+ "r1 = r6;"
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : [ran]"i"(RAN_PAD_CALLS), __imm_addr(input), __imm_addr(pads_ran),
+ __imm_addr(pad_runs)
+ : __clobber_all);
+}
+
+SEC("syscall")
+int entry_pad_calls(void *ctx)
+{
+ return pad_calls_frame();
+}
+
+/* 11. A covered bpf_throw() the sweep leaves last, where the exception
+ * callback patchlet -- the one that does not keep the call it replaced in the
+ * last slot -- has to carry the marks with it; r6-r9 reports a lost one.
+ */
+SEC("syscall")
+__naked int entry_pad_first(void)
+{
+ asm volatile (
+ LOAD_MAGIC_REGS
+ "r1 = %[input] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+ "if r1 < 101 goto 7f;"
+ "goto 4f;"
+"3:" /* landing pad, ahead of the call */
+ CHECK_MAGIC_REGS("%[ran]")
+ "call bpf_unwind_resume;"
+ "exit;"
+"7:"
+ "r0 = 0;"
+ "exit;"
+"4:"
+ "r1 = %[cookie];"
+"1:" "call bpf_throw;" /* cleanup region */
+"2:"
+ "exit;" /* dead: swept, leaving the call last */
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : [cookie]"i"(THROW_COOKIE), [ran]"i"(RAN_PAD_FIRST),
+ __imm_addr(input), __imm_addr(magic), __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+/* 12. A cleanup region whose last instruction is a 16-byte one, so end - 1
+ * names the half that is not an instruction of its own. Well formed, and a
+ * rule against it would turn it away.
+ */
+static __used __naked __noinline __u64 wide_rec_frame(void)
+{
+ asm volatile (
+ "r1 = %[input] ll;"
+ "r6 = *(u64 *)(r1 + 0);"
+ "call bpf_rcu_read_lock;"
+ "r1 = r6;"
+"1:" "call shared_callee;" /* cleanup region begins */
+ "r1 = %[magic] ll;" /* ... and ends on this pair */
+"2:"
+ "r6 = r0;"
+ "call bpf_rcu_read_unlock;"
+ "r0 = r6;"
+ "exit;"
+"3:" /* landing pad */
+ "r7 = r0;"
+ "call bpf_rcu_read_unlock;"
+ PAD_RAN("%[ran]")
+ "r1 = r7;"
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : [ran]"i"(RAN_WIDE_REC), __imm_addr(input), __imm_addr(magic),
+ __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+SEC("syscall")
+int entry_wide_rec(void *ctx)
+{
+ return wide_rec_frame();
+}
+
+/* 13. A pad that reloads from and stores to its own frame -- the shape every
+ * compiler-generated pad has. A JIT that addresses the frame through the stack
+ * pointer, arm64, has to find a pad's frame another way.
+ */
+static __used __naked __noinline __u64 pad_stack_frame(void)
+{
+ asm volatile (
+ "r1 = %[magic] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+ "*(u64 *)(r10 - 8) = r1;" /* what the pad will want */
+ "r1 = %[input] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+"1:" "call pc_thrower;" /* cleanup region */
+"2:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ "r6 = r0;"
+ "r7 = *(u64 *)(r10 - 8);" /* reload it out of the frame */
+ "*(u64 *)(r10 - 16) = r7;" /* and write the frame while here */
+ "r1 = %[magic] ll;"
+ "r2 = *(u64 *)(r1 + 0);"
+ "if r7 != r2 goto 9f;"
+ "r3 = *(u64 *)(r10 - 16);"
+ "if r3 != r2 goto 9f;"
+ PAD_RAN("%[ran]")
+"9:"
+ "r1 = r6;"
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : [ran]"i"(RAN_PAD_STACK), __imm_addr(input), __imm_addr(magic),
+ __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+SEC("syscall")
+int entry_pad_stack(void *ctx)
+{
+ return pad_stack_frame();
+}
+
+/* 14. The same, with a frame in between that has no pad of its own, so the
+ * liveness query for an outer frame has more than one frame to walk.
+ */
+static __used __noinline __u64 deep_mid(__u64 x)
+{
+ return pc_thrower(x) + 1;
+}
+
+static __used __naked __noinline __u64 deep_frame(void)
+{
+ asm volatile (
+ "r1 = %[magic] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+ "*(u64 *)(r10 - 8) = r1;" /* nothing but the pad reads this */
+ "r1 = %[input] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+"1:" "call deep_mid;" /* cleanup region */
+"2:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ "r6 = r0;"
+ "r7 = *(u64 *)(r10 - 8);"
+ "r1 = %[magic] ll;"
+ "r2 = *(u64 *)(r1 + 0);"
+ "if r7 != r2 goto 9f;"
+ PAD_RAN("%[ran]")
+"9:"
+ "r1 = r6;"
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : [ran]"i"(RAN_DEEP_PAD), __imm_addr(input), __imm_addr(magic),
+ __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+SEC("syscall")
+int entry_deep_pad(void *ctx)
+{
+ return deep_frame();
+}
+
+/* 15. A region around a call the kernel knows cannot unwind: no call site is
+ * marked, nothing reaches the pad, and the sweep removes it. The program is
+ * otherwise ordinary and has to load.
+ */
+static __used __naked __noinline __u64 nounwind_rec_frame(void)
+{
+ asm volatile (
+ "call bpf_preempt_disable;"
+"1:" "call bpf_preempt_enable;" /* cleanup region: nounwind */
+"2:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad, never dispatched */
+ PAD_RAN("%[ran]")
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : [ran]"i"(RAN_NOUNWIND_REC), __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+SEC("syscall")
+int entry_nounwind_rec(void *ctx)
+{
+ return nounwind_rec_frame();
+}
+
+/* 16. The name a frontend gives the resume: LLVM emits _Unwind_Resume() and
+ * libbpf maps it onto bpf_unwind_resume(), which every other pad here calls.
+ */
+extern void _Unwind_Resume(void) __ksym;
+
+static __used __noinline void __resume_alias_btf_anchor(void)
+{
+ _Unwind_Resume();
+}
+
+static __used __naked __noinline __u64 resume_alias_frame(void)
+{
+ asm volatile (
+"1:" "call regs_thrower;" /* cleanup region */
+"2:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ PAD_RAN("%[ran]")
+ "call _Unwind_Resume;" /* the frontend's name for it */
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : [ran]"i"(RAN_RESUME_ALIAS), __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+SEC("syscall")
+int entry_resume_alias(void *ctx)
+{
+ return resume_alias_frame();
+}
+
+/* 17. A landing pad that calls a subprogram which tail calls. A tail call in
+ * the pad itself is refused, but the callee's prologue really did run, so its
+ * tail call releases exactly that and the target returns into the pad.
+ */
+struct {
+ __uint(type, BPF_MAP_TYPE_PROG_ARRAY);
+ __uint(max_entries, 1);
+ __uint(key_size, sizeof(__u32));
+ __uint(value_size, sizeof(__u32));
+} pad_tc_table SEC(".maps");
+
+__u64 pad_tc_target_ran = 0;
+
+SEC("syscall")
+int pad_tc_target(void *ctx)
+{
+ pad_tc_target_ran += 1;
+ return 0;
+}
+
+static __used __noinline __u64 pad_tc_callee(void *ctx)
+{
+ /* Taken only once the test has populated the array. */
+ bpf_tail_call_static(ctx, &pad_tc_table, 0);
+ return 0;
+}
+
+/* The frame with the pad is the program itself, and __naked: r1 at entry is
+ * the only place to get a context for bpf_tail_call(); the pad reloads it
+ * from the frame's stack.
+ */
+SEC("syscall")
+__naked int entry_pad_tail_call(void)
+{
+ asm volatile (
+ "*(u64 *)(r10 - 8) = r1;" /* the context, straight from entry */
+ LOAD_MAGIC_REGS
+ "r1 = %[input] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+ "if r1 < 101 goto 8f;"
+"1:" "call regs_thrower;" /* cleanup region */
+"2:"
+"8:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ PAD_COUNT
+ "r1 = *(u64 *)(r10 - 8);"
+ "call pad_tc_callee;"
+ /* Only if the frame survived the callee's tail call. */
+ CHECK_MAGIC_REGS("%[ran]")
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : [ran]"i"(RAN_PAD_TAIL_CALL), __imm_addr(input),
+ __imm_addr(magic), __imm_addr(pads_ran), __imm_addr(pad_runs)
+ : __clobber_all);
+}
+
+/* 18. The other target for that same tail call: a program with a table of its
+ * own, throwing inside the outer exception. The tail call made it a boundary,
+ * so the inner walk ends there and the outer pad, cookie and r6-r9 survive.
+ */
+__u64 tc_target_pad_runs = 0;
+__u64 inner_magic = 0xd00d;
+
+static __used __naked __noinline __u64 inner_thrower(void)
+{
+ asm volatile (
+ /* Not this frame's to keep, the same as regs_thrower. */
+ "r6 = 0xf00d;"
+ "r7 = 0xcafe;"
+ "r8 = 0xbeef;"
+ "r9 = 0xdead;"
+ "r1 = %[cookie];"
+ "call bpf_throw;"
+ "r0 = 0;"
+ "exit;"
+ :
+ : [cookie]"i"(INNER_COOKIE)
+ : __clobber_all);
+}
+
+SEC("syscall")
+__naked int pad_tc_throw_target(void)
+{
+ asm volatile (
+ /* Distinct from the outer pad's, so neither can stand in for it. */
+ "r1 = %[inner_magic] ll;"
+ "r6 = *(u64 *)(r1 + 0);"
+ "r7 = r6;"
+ "r7 += 1;"
+ "r8 = r6;"
+ "r8 += 2;"
+ "r9 = r6;"
+ "r9 += 3;"
+ "r1 = %[input] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+ "if r1 < 101 goto 8f;"
+"1:" "call inner_thrower;" /* cleanup region */
+"2:"
+"8:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ /* This frame's own r6-r9, not the outer pad's. */
+ "r1 = %[inner_magic] ll;"
+ "r2 = *(u64 *)(r1 + 0);"
+ "if r6 != r2 goto 9f;"
+ "r2 += 1;"
+ "if r7 != r2 goto 9f;"
+ "r2 += 1;"
+ "if r8 != r2 goto 9f;"
+ "r2 += 1;"
+ "if r9 != r2 goto 9f;"
+ "r1 = %[tc_target_pad_runs] ll;"
+ "r2 = *(u64 *)(r1 + 0);"
+ "r2 += 1;"
+ "*(u64 *)(r1 + 0) = r2;"
+"9:"
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : __imm_addr(input), __imm_addr(inner_magic),
+ __imm_addr(tc_target_pad_runs)
+ : __clobber_all);
+}
+
+/* 19. A landing pad whose first instruction is a nop. opt_remove_nops() runs
+ * long after the cleanup walk, so the record has to follow the pad to the
+ * instruction that takes its place rather than be dropped with the nop.
+ */
+static __used __naked __noinline __u64 nop_pad_frame(void)
+{
+ asm volatile (
+ "r1 = %[input] ll;"
+ "r6 = *(u64 *)(r1 + 0);"
+ "if r6 < 101 goto 6f;"
+ "r1 = %[cookie];"
+"1:" "call bpf_throw;" /* cleanup region */
+"2:"
+"6:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad: a nop, then its body */
+ "goto +0;"
+ PAD_RAN("%[ran]")
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : [cookie]"i"(THROW_COOKIE), [ran]"i"(RAN_NOP_PAD),
+ __imm_addr(input), __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+SEC("syscall")
+int entry_nop_pad(void *ctx)
+{
+ return nop_pad_frame();
+}
+
+/* 20. A landing pad that indexes its own frame's stack by a register the frame
+ * set before the throwing call. The offset is not a constant, so r6 has to be
+ * marked precise from inside the pad, back across the unwind edge.
+ */
+/* A thrower that touches none of r6-r9, so nothing in it can answer for the
+ * pad frame's r6 and the walk has to leave this frame to look.
+ */
+static __used __naked __noinline __u64 var_thrower(void)
+{
+ asm volatile (
+ "if r1 < 101 goto 1f;"
+ "r1 = %[cookie];"
+ "call bpf_throw;"
+"1:"
+ "r0 = 0;"
+ "exit;"
+ :
+ : [cookie]"i"(THROW_COOKIE)
+ : __clobber_all);
+}
+
+static __used __naked __noinline __u64 var_stack_frame(void)
+{
+ asm volatile (
+ "r1 = %[magic] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+ "*(u64 *)(r10 - 8) = r1;" /* the slot the pad will read... */
+ "*(u64 *)(r10 - 16) = r1;" /* ...whichever of the two it is */
+ "r1 = %[input] ll;"
+ "r6 = *(u64 *)(r1 + 0);"
+ "r6 &= 1;" /* an unknown slot number... */
+ "r6 <<= 3;" /* ...as an aligned byte offset */
+ "r1 = %[input] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+"1:" "call var_thrower;" /* cleanup region */
+"2:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ "r7 = r0;"
+ "r1 = r10;"
+ "r1 += r6;" /* variable offset into the frame */
+ "r2 = *(u64 *)(r1 - 16);"
+ "r3 = %[magic] ll;"
+ "r3 = *(u64 *)(r3 + 0);"
+ "if r2 != r3 goto 9f;"
+ PAD_RAN("%[ran]")
+"9:"
+ "r1 = r7;"
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : [ran]"i"(RAN_VAR_STACK), __imm_addr(input), __imm_addr(magic),
+ __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+SEC("syscall")
+int entry_var_stack(void *ctx)
+{
+ return var_stack_frame();
+}
+
+/* 21. The same walk, over a call to a global subprogram: the verifier enters
+ * no frame for one, so the exception arrives at the pad from the call site
+ * rather than from a throw some frames deeper.
+ */
+__noinline __u64 global_thrower(__u64 x)
+{
+ if (x > 100)
+ bpf_throw(THROW_COOKIE);
+ return x + 1;
+}
+
+static __used __naked __noinline __u64 global_pad_frame(void)
+{
+ asm volatile (
+ "r1 = %[magic] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+ "*(u64 *)(r10 - 8) = r1;"
+ "*(u64 *)(r10 - 16) = r1;"
+ "r1 = %[input] ll;"
+ "r6 = *(u64 *)(r1 + 0);"
+ "r6 &= 1;"
+ "r6 <<= 3;"
+ "r1 = %[input] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+"1:" "call global_thrower;" /* cleanup region */
+"2:"
+ "r0 = 0;"
+ "exit;"
+"3:" /* landing pad */
+ "r7 = r0;"
+ "r1 = r10;"
+ "r1 += r6;"
+ "r2 = *(u64 *)(r1 - 16);"
+ "r3 = %[magic] ll;"
+ "r3 = *(u64 *)(r3 + 0);"
+ "if r2 != r3 goto 9f;"
+ PAD_RAN("%[ran]")
+"9:"
+ "r1 = r7;"
+ "call bpf_unwind_resume;"
+ "exit;"
+ CLEANUP_REC("1b", "2b", "3b")
+ :
+ : [ran]"i"(RAN_GLOBAL_PAD), __imm_addr(input), __imm_addr(magic),
+ __imm_addr(pads_ran)
+ : __clobber_all);
+}
+
+SEC("syscall")
+int entry_global_pad(void *ctx)
+{
+ return global_pad_frame();
+}
+
+char _license[] SEC("license") = "GPL";
|
| 23/3 | 2026/09/22 09:08 | llm | kmsan-evaluator |
2mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 78404 output: 17 thoughts: 2031 Thoughts: **KMSAN Assessment: BPF Cleanup Pad Implementation** Okay, let's break this down. My goal is to determine if this patch series, implementing BPF cleanup pads and exception handling, necessitates a KMSAN fuzzing session. The target is amd64, but the code changes apply to arm64 as well. I need to figure out if uninitialized memory reads or information leaks are introduced by this. First, I need to remind myself of the rules. If standard checks like KASAN (OOB, UAF, etc.), LOCKDEP, or UBSAN would catch the bug, there's no need for KMSAN. KMSAN is expensive, so I only use it if it exposes uninitialized memory usage *only*. Here are the conditions that would necessitate setting `NeedsKMSAN=true`: 1. Kernel structures sent to userspace (e.g., via `copy_to_user`, netlink, ioctl) with potentially uninitialized fields or padding. 2. Conditional logic based on uninitialized variables. 3. Allocation of complex data structures where uninitialized fields are read later. 4. Bounds checks that allow access to uninitialized bytes of existing buffers. If the changes primarily risk out-of-bounds access, etc., or all structures are initialized, I set `NeedsKMSAN=false`. Now, looking at the patch, it seems to add BPF cleanup pad functionality, dealing with exception handling within BPF programs. The files modified are extensive, spanning architecture-specific assembly, core BPF code, headers, and tools. My first thought is, are there any potential information leaks to userspace? Specifically, let's look at the kernel changes. I'm seeing additions to `uapi/linux/bpf.h`. There's a new `union bpf_attr` member, `cleanup_info`, which contains a pointer to a struct. The `bpf_cleanup_info` struct is also added. It's used in `kernel/bpf/check_btf.c`. Specifically, in `check_cleanup_info`, the code allocates memory for `krecord` with `kvcalloc`, which *should* zero the memory, meaning that there shouldn't be any uninitialized memory read issues at first glance. It then copies user-provided data into this `krecord` with `copy_from_bpfptr`. However, this copy is done in `min_size` chunks, and if the user provides data smaller than the kernel structure size, the remaining fields will be initialized to zero thanks to `kvcalloc`. Moreover, after the copy, the code sets `min_size` to the minimum of the user and kernel structure size. It then copies the minimum size back to the user via `copy_to_bpfptr_offset`. This operation looks fine. Is *any* of the cleanup info returned to the user? I don't see any other places where `cleanup_info` or `bpf_cleanup_info` are copied back to user space. To be sure, I'll quickly check if `bpf_prog_get_info_by_fd` has been touched; it's a common place to expose information. Since the diff doesn't touch it, I also want to make sure it's not exposed in `BPF_OBJ_GET_INFO_BY_FD` through the current patch. Let's do a quick grep. In short, based on my analysis, it doesn't look like any uninitialized data is being leaked to userspace. All kernel structures related to `bpf_cleanup_info` are properly initialized (or partially initialized with zeroing) before being used, and no data is being copied back to user space that would expose uninitialized memory. Therefore, I believe there's no need for a dedicated KMSAN fuzzing session for this patch series. |
| 24/3 | 2026/09/22 09:10 | tool | grepper |
0mArgs:{
"Expression": "cleanup_info"
}Results: {
"Output": "drivers/net/ethernet/broadcom/cnic_defs.h=2411=struct fcoe_cached_sge_ctx {\n--\ndrivers/net/ethernet/broadcom/cnic_defs.h-2421- */\ndrivers/net/ethernet/broadcom/cnic_defs.h:2422:struct fcoe_cleanup_info {\ndrivers/net/ethernet/broadcom/cnic_defs.h-2423-\t__le16 cleaned_task_id;\n--\ndrivers/net/ethernet/broadcom/cnic_defs.h=2511=struct fcoe_ext_abts_info {\n--\ndrivers/net/ethernet/broadcom/cnic_defs.h-2519- */\ndrivers/net/ethernet/broadcom/cnic_defs.h:2520:struct fcoe_ext_cleanup_info {\ndrivers/net/ethernet/broadcom/cnic_defs.h-2521-\t__le32 rsrv0[6];\ndrivers/net/ethernet/broadcom/cnic_defs.h:2522:\tstruct fcoe_cleanup_info ctx;\ndrivers/net/ethernet/broadcom/cnic_defs.h-2523-};\n--\ndrivers/net/ethernet/broadcom/cnic_defs.h=2962=union fcoe_tx_wr_rx_rd_union_ctx {\n--\ndrivers/net/ethernet/broadcom/cnic_defs.h-2964-\tstruct fcoe_fcp_cmd_payload fcp_cmd;\ndrivers/net/ethernet/broadcom/cnic_defs.h:2965:\tstruct fcoe_ext_cleanup_info cleanup;\ndrivers/net/ethernet/broadcom/cnic_defs.h-2966-\tstruct fcoe_ext_abts_info abts;\n--\ndrivers/net/ethernet/broadcom/cnic_defs.h=3287=union fcoe_u_tce_tx_wr_rx_rd_union {\ndrivers/net/ethernet/broadcom/cnic_defs.h-3288-\tstruct fcoe_abts_info abts;\ndrivers/net/ethernet/broadcom/cnic_defs.h:3289:\tstruct fcoe_cleanup_info cleanup;\ndrivers/net/ethernet/broadcom/cnic_defs.h-3290-\tstruct fcoe_fw_tx_seq_ctx tx_seq_ctx;\n--\ndrivers/scsi/bnx2fc/57xx_hsi_bnx2fc.h=109=struct fcoe_cached_sge_ctx {\n--\ndrivers/scsi/bnx2fc/57xx_hsi_bnx2fc.h-119- */\ndrivers/scsi/bnx2fc/57xx_hsi_bnx2fc.h:120:struct fcoe_cleanup_info {\ndrivers/scsi/bnx2fc/57xx_hsi_bnx2fc.h-121-\t__le16 cleaned_task_id;\n--\ndrivers/scsi/bnx2fc/57xx_hsi_bnx2fc.h=209=struct fcoe_ext_abts_info {\n--\ndrivers/scsi/bnx2fc/57xx_hsi_bnx2fc.h-217- */\ndrivers/scsi/bnx2fc/57xx_hsi_bnx2fc.h:218:struct fcoe_ext_cleanup_info {\ndrivers/scsi/bnx2fc/57xx_hsi_bnx2fc.h-219-\t__le32 rsrv0[6];\ndrivers/scsi/bnx2fc/57xx_hsi_bnx2fc.h:220:\tstruct fcoe_cleanup_info ctx;\ndrivers/scsi/bnx2fc/57xx_hsi_bnx2fc.h-221-};\n--\ndrivers/scsi/bnx2fc/57xx_hsi_bnx2fc.h=660=union fcoe_tx_wr_rx_rd_union_ctx {\n--\ndrivers/scsi/bnx2fc/57xx_hsi_bnx2fc.h-662-\tstruct fcoe_fcp_cmd_payload fcp_cmd;\ndrivers/scsi/bnx2fc/57xx_hsi_bnx2fc.h:663:\tstruct fcoe_ext_cleanup_info cleanup;\ndrivers/scsi/bnx2fc/57xx_hsi_bnx2fc.h-664-\tstruct fcoe_ext_abts_info abts;\n--\ndrivers/usb/host/xhci-mem.c=609=struct xhci_stream_info *xhci_alloc_stream_info(struct xhci_hcd *xhci,\n--\ndrivers/usb/host/xhci-mem.c-641-\tif (!stream_info-\u003estream_rings)\ndrivers/usb/host/xhci-mem.c:642:\t\tgoto cleanup_info;\ndrivers/usb/host/xhci-mem.c-643-\n--\ndrivers/usb/host/xhci-mem.c-715-\tkfree(stream_info-\u003estream_rings);\ndrivers/usb/host/xhci-mem.c:716:cleanup_info:\ndrivers/usb/host/xhci-mem.c-717-\tkfree(stream_info);\n--\ninclude/linux/bpf.h=1787=struct bpf_exception_info {\ninclude/linux/bpf.h:1788:\tstruct bpf_cleanup_info *info;\ninclude/linux/bpf.h-1789-\tstruct bpf_cleanup_range *ranges;\n--\ninclude/linux/bpf_verifier.h=932=struct bpf_verifier_env {\n--\ninclude/linux/bpf_verifier.h-1001-\tu32 subprog_cnt;\ninclude/linux/bpf_verifier.h:1002:\tstruct bpf_cleanup_info *cleanup_info;\ninclude/linux/bpf_verifier.h:1003:\tu32 cleanup_info_cnt;\ninclude/linux/bpf_verifier.h-1004-\t/* number of instructions analyzed by the verifier */\n--\ninclude/uapi/linux/bpf.h=1527=union bpf_attr {\n--\ninclude/uapi/linux/bpf.h-1671-\t\t__s32\t\tkeyring_id;\ninclude/uapi/linux/bpf.h:1672:\t\t__aligned_u64\tcleanup_info;\t/* exception cleanup table */\ninclude/uapi/linux/bpf.h:1673:\t\t__u32\t\tcleanup_info_rec_size; /* userspace bpf_cleanup_info size */\ninclude/uapi/linux/bpf.h:1674:\t\t__u32\t\tcleanup_info_cnt; /* number of bpf_cleanup_info records */\ninclude/uapi/linux/bpf.h-1675-\t};\n--\ninclude/uapi/linux/bpf.h=7598=struct bpf_line_info {\n--\ninclude/uapi/linux/bpf.h-7604-\ninclude/uapi/linux/bpf.h:7605:struct bpf_cleanup_info {\ninclude/uapi/linux/bpf.h-7606-\t__u32\tbegin_off;\n--\nkernel/bpf/cfg.c=162=static int visit_cleanup_pad_edge(int t, struct bpf_verifier_env *env)\n--\nkernel/bpf/cfg.c-167-\nkernel/bpf/cfg.c:168:\tif (!env-\u003ecleanup_info_cnt)\nkernel/bpf/cfg.c-169-\t\treturn DONE_EXPLORING;\n--\nkernel/bpf/cfg.c=638=int bpf_check_cfg(struct bpf_verifier_env *env)\n--\nkernel/bpf/cfg.c-703-\t */\nkernel/bpf/cfg.c:704:\twhile (pad_idx \u003c env-\u003ecleanup_info_cnt) {\nkernel/bpf/cfg.c:705:\t\tu32 pad = env-\u003ecleanup_info[pad_idx++].landing_pad_off;\nkernel/bpf/cfg.c-706-\n--\nkernel/bpf/check_btf.c=410=static int cleanup_insn_subprog(struct bpf_verifier_env *env, u32 off)\n--\nkernel/bpf/check_btf.c-422-\nkernel/bpf/check_btf.c:423:static int check_cleanup_info(struct bpf_verifier_env *env,\nkernel/bpf/check_btf.c-424-\t\t\t const union bpf_attr *attr,\n--\nkernel/bpf/check_btf.c-426-{\nkernel/bpf/check_btf.c:427:\tu32 krec_size = sizeof(struct bpf_cleanup_info);\nkernel/bpf/check_btf.c-428-\tu32 i, nrec, urec_size, min_size, prev_end = 0;\nkernel/bpf/check_btf.c:429:\tstruct bpf_cleanup_info *krecord;\nkernel/bpf/check_btf.c-430-\tbpfptr_t urecord;\n--\nkernel/bpf/check_btf.c-432-\nkernel/bpf/check_btf.c:433:\tnrec = attr-\u003ecleanup_info_cnt;\nkernel/bpf/check_btf.c-434-\tif (!nrec)\n--\nkernel/bpf/check_btf.c-438-\nkernel/bpf/check_btf.c:439:\turec_size = attr-\u003ecleanup_info_rec_size;\nkernel/bpf/check_btf.c-440-\tif (urec_size \u003c MIN_BPF_CLEANUP_INFO_SIZE ||\n--\nkernel/bpf/check_btf.c-451-\tmin_size = min_t(u32, krec_size, urec_size);\nkernel/bpf/check_btf.c:452:\turecord = make_bpfptr(attr-\u003ecleanup_info, uattr.is_kernel);\nkernel/bpf/check_btf.c-453-\tfor (i = 0; i \u003c nrec; i++) {\nkernel/bpf/check_btf.c:454:\t\tstruct bpf_cleanup_info *rec = \u0026krecord[i];\nkernel/bpf/check_btf.c-455-\t\tint sb, se, sl;\n--\nkernel/bpf/check_btf.c-462-\t\t\t\t\t\t\t offsetof(union bpf_attr,\nkernel/bpf/check_btf.c:463:\t\t\t\t\t\t\t\t cleanup_info_rec_size),\nkernel/bpf/check_btf.c-464-\t\t\t\t\t\t\t \u0026min_size, sizeof(min_size)))\n--\nkernel/bpf/check_btf.c-477-\t\tif (rec-\u003ebegin_off \u003e= rec-\u003eend_off) {\nkernel/bpf/check_btf.c:478:\t\t\tverbose(env, \"cleanup_info[%u]: begin %u \u003e= end %u\\n\",\nkernel/bpf/check_btf.c-479-\t\t\t\ti, rec-\u003ebegin_off, rec-\u003eend_off);\n--\nkernel/bpf/check_btf.c-483-\t\t\tverbose(env,\nkernel/bpf/check_btf.c:484:\t\t\t\t\"cleanup_info[%u]: range [%u,%u) is unsorted or overlaps the previous record\\n\",\nkernel/bpf/check_btf.c-485-\t\t\t\ti, rec-\u003ebegin_off, rec-\u003eend_off);\n--\nkernel/bpf/check_btf.c-493-\t\tif (sb \u003c 0 || se \u003c 0 || sl \u003c 0) {\nkernel/bpf/check_btf.c:494:\t\t\tverbose(env, \"cleanup_info[%u]: offset out of range\\n\", i);\nkernel/bpf/check_btf.c-495-\t\t\tgoto err_free;\n--\nkernel/bpf/check_btf.c-498-\t\t\tverbose(env,\nkernel/bpf/check_btf.c:499:\t\t\t\t\"cleanup_info[%u]: range/landing pad span multiple subprogs\\n\",\nkernel/bpf/check_btf.c-500-\t\t\t\ti);\n--\nkernel/bpf/check_btf.c-512-\t\t !env-\u003eprog-\u003einsnsi[rec-\u003eend_off].code)) {\nkernel/bpf/check_btf.c:513:\t\t\tverbose(env, \"cleanup_info[%u]: points at invalid insn\\n\", i);\nkernel/bpf/check_btf.c-514-\t\t\tgoto err_free;\n--\nkernel/bpf/check_btf.c-536-\t\t\t\tverbose(env,\nkernel/bpf/check_btf.c:537:\t\t\t\t\t\"cleanup_info[%u]: landing pad %u is inside the call-site range of cleanup_info[%u]\\n\",\nkernel/bpf/check_btf.c-538-\t\t\t\t\ti, pad, m);\n--\nkernel/bpf/check_btf.c-543-\nkernel/bpf/check_btf.c:544:\tenv-\u003ecleanup_info = krecord;\nkernel/bpf/check_btf.c:545:\tenv-\u003ecleanup_info_cnt = nrec;\nkernel/bpf/check_btf.c-546-\treturn 0;\n--\nkernel/bpf/check_btf.c=581=int bpf_check_btf_info(struct bpf_verifier_env *env,\n--\nkernel/bpf/check_btf.c-586-\nkernel/bpf/check_btf.c:587:\terr = check_cleanup_info(env, attr, uattr);\nkernel/bpf/check_btf.c-588-\tif (err)\n--\nkernel/bpf/exception.c=167=int bpf_cleanup_check_callback(struct bpf_verifier_env *env, int subprog)\nkernel/bpf/exception.c-168-{\nkernel/bpf/exception.c:169:\tif (!env-\u003ecleanup_info_cnt || !env-\u003esubprog_info[subprog].might_throw)\nkernel/bpf/exception.c-170-\t\treturn 0;\n--\nkernel/bpf/exception.c=335=static int cleanup_mark_pad_bodies(struct cleanup_ctx *ctx)\n--\nkernel/bpf/exception.c-340-\nkernel/bpf/exception.c:341:\tfor (i = 0; i \u003c env-\u003ecleanup_info_cnt; i++) {\nkernel/bpf/exception.c:342:\t\tu32 pad = env-\u003ecleanup_info[i].landing_pad_off;\nkernel/bpf/exception.c-343-\n--\nkernel/bpf/exception.c=416=static void cleanup_mark_call_sites(struct bpf_verifier_env *env)\n--\nkernel/bpf/exception.c-419-\nkernel/bpf/exception.c:420:\tfor (i = 0; i \u003c env-\u003ecleanup_info_cnt; i++) {\nkernel/bpf/exception.c:421:\t\tstruct bpf_cleanup_info *rec = \u0026env-\u003ecleanup_info[i];\nkernel/bpf/exception.c-422-\n--\nkernel/bpf/exception.c=433=int bpf_prepare_cleanup_exceptions(struct bpf_verifier_env *env)\nkernel/bpf/exception.c-434-{\nkernel/bpf/exception.c:435:\tif (!env-\u003ecleanup_info_cnt)\nkernel/bpf/exception.c-436-\t\treturn 0;\n--\nkernel/bpf/exception.c=462=int bpf_check_cleanup_exceptions(struct bpf_verifier_env *env)\n--\nkernel/bpf/exception.c-471-\nkernel/bpf/exception.c:472:\tif (!env-\u003ecleanup_info_cnt)\nkernel/bpf/exception.c-473-\t\treturn 0;\n--\nkernel/bpf/exception.c=543=int bpf_cleanup_alloc_info(struct bpf_prog_aux *aux)\n--\nkernel/bpf/exception.c-550-\nkernel/bpf/exception.c:551:int bpf_cleanup_attach_info(struct bpf_prog_aux *aux, struct bpf_cleanup_info *recs, u32 cnt)\nkernel/bpf/exception.c-552-{\n--\nkernel/bpf/exception.c=592=void bpf_cleanup_fill_native_ranges(struct bpf_prog *prog, u32 *addrs, void *image)\n--\nkernel/bpf/exception.c-601-\tfor (i = 0; i \u003c n; i++) {\nkernel/bpf/exception.c:602:\t\tconst struct bpf_cleanup_info *rec = \u0026exc-\u003einfo[i];\nkernel/bpf/exception.c-603-\n--\nkernel/bpf/exception.h-7-\nkernel/bpf/exception.h:8:struct bpf_cleanup_info;\nkernel/bpf/exception.h-9-struct bpf_cleanup_range;\n--\nkernel/bpf/exception.h=18=int bpf_cleanup_alloc_info(struct bpf_prog_aux *aux);\nkernel/bpf/exception.h:19:int bpf_cleanup_attach_info(struct bpf_prog_aux *aux, struct bpf_cleanup_info *recs, u32 cnt);\nkernel/bpf/exception.h-20-const struct bpf_cleanup_range *bpf_cleanup_pad_for_ip(const struct bpf_prog *prog, u64 ip);\n--\nkernel/bpf/fixups.c=222=static void adjust_insn_aux_data(struct bpf_verifier_env *env,\n--\nkernel/bpf/fixups.c-274-\nkernel/bpf/fixups.c:275:\tif (env-\u003ecleanup_info_cnt)\nkernel/bpf/fixups.c-276-\t\tfor (i = 0; i \u003c prog_len; i++)\n--\nkernel/bpf/fixups.c=563=static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt,\n--\nkernel/bpf/fixups.c-593-\nkernel/bpf/fixups.c:594:\tif (env-\u003ecleanup_info_cnt) {\nkernel/bpf/fixups.c-595-\t\tfor (i = 0; i \u003c env-\u003einsn_aux_data_len; i++) {\n--\nkernel/bpf/fixups.c=1158=static int cleanup_pad_body_for_subprog(struct bpf_verifier_env *env, struct bpf_prog *sub,\n--\nkernel/bpf/fixups.c-1182-\nkernel/bpf/fixups.c:1183:static int cleanup_info_for_subprog(struct bpf_verifier_env *env, struct bpf_prog *sub,\nkernel/bpf/fixups.c-1184-\t\t\t\t u32 start, u32 end)\nkernel/bpf/fixups.c-1185-{\nkernel/bpf/fixups.c:1186:\tstruct bpf_cleanup_info *recs;\nkernel/bpf/fixups.c-1187-\tu32 i, cnt = 0;\n--\nkernel/bpf/fixups.c-1189-\nkernel/bpf/fixups.c:1190:\tif (!env-\u003ecleanup_info_cnt)\nkernel/bpf/fixups.c-1191-\t\treturn 0;\n--\nkernel/bpf/fixups.c=1243=int bpf_cleanup_attach_main_prog(struct bpf_verifier_env *env, struct bpf_prog *prog)\n--\nkernel/bpf/fixups.c-1246-\t\treturn 0;\nkernel/bpf/fixups.c:1247:\treturn cleanup_info_for_subprog(env, prog, 0, prog-\u003elen);\nkernel/bpf/fixups.c-1248-}\n--\nkernel/bpf/fixups.c=1250=static int jit_subprogs(struct bpf_verifier_env *env)\n--\nkernel/bpf/fixups.c-1386-\t\t\tfunc[i]-\u003eaux-\u003eexception_boundary = env-\u003eseen_exception;\nkernel/bpf/fixups.c:1387:\t\terr = cleanup_info_for_subprog(env, func[i], subprog_start,\nkernel/bpf/fixups.c-1388-\t\t\t\t\t subprog_end);\n--\nkernel/bpf/liveness.c=220=bpf_insn_successors(struct bpf_verifier_env *env, u32 idx)\n--\nkernel/bpf/liveness.c-271-\nkernel/bpf/liveness.c:272:\tif (unlikely(env-\u003ecleanup_info_cnt)) {\nkernel/bpf/liveness.c-273-\t\tint pad = bpf_cleanup_pad_of_call(env, idx);\n--\nkernel/bpf/liveness.c=371=bool bpf_stack_slot_alive(struct bpf_verifier_env *env, u32 frameno, u32 half_spi)\n--\nkernel/bpf/liveness.c-413-\t\t/* Control may also go to the landing pad. */\nkernel/bpf/liveness.c:414:\t\tif (!alive \u0026\u0026 unlikely(env-\u003ecleanup_info_cnt)) {\nkernel/bpf/liveness.c-415-\t\t\tint pad = bpf_cleanup_pad_of_call(env, callsite);\n--\nkernel/bpf/syscall.c=2915=BTF_ID_LIST_GLOBAL_SINGLE(bpf_multi_func_btf_id, func, bpf_multi_func)\n--\nkernel/bpf/syscall.c-2917-/* last field in 'union bpf_attr' used by this command */\nkernel/bpf/syscall.c:2918:#define BPF_PROG_LOAD_LAST_FIELD cleanup_info_cnt\nkernel/bpf/syscall.c-2919-\n--\nkernel/bpf/verifier.c=5576=static int check_max_stack_depth(struct bpf_verifier_env *env)\n--\nkernel/bpf/verifier.c-5599-\t */\nkernel/bpf/verifier.c:5600:\tif (env-\u003ecleanup_info_cnt)\nkernel/bpf/verifier.c-5601-\t\tpriv_stack_mode = NO_PRIV_STACK;\n--\nkernel/bpf/verifier.c=21658=int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,\n--\nkernel/bpf/verifier.c-22003-\tbpf_diag_free(env);\nkernel/bpf/verifier.c:22004:\tkvfree(env-\u003ecleanup_info);\nkernel/bpf/verifier.c-22005-\tkvfree(env);\n--\ntools/include/uapi/linux/bpf.h=1527=union bpf_attr {\n--\ntools/include/uapi/linux/bpf.h-1671-\t\t__s32\t\tkeyring_id;\ntools/include/uapi/linux/bpf.h:1672:\t\t__aligned_u64\tcleanup_info;\t/* exception cleanup table */\ntools/include/uapi/linux/bpf.h:1673:\t\t__u32\t\tcleanup_info_rec_size; /* userspace bpf_cleanup_info size */\ntools/include/uapi/linux/bpf.h:1674:\t\t__u32\t\tcleanup_info_cnt; /* number of bpf_cleanup_info records */\ntools/include/uapi/linux/bpf.h-1675-\t};\n--\ntools/include/uapi/linux/bpf.h=7598=struct bpf_line_info {\n--\ntools/include/uapi/linux/bpf.h-7604-\ntools/include/uapi/linux/bpf.h:7605:struct bpf_cleanup_info {\ntools/include/uapi/linux/bpf.h-7606-\t__u32\tbegin_off;\n--\ntools/lib/bpf/bpf.c=293=int bpf_prog_load(enum bpf_prog_type prog_type,\n--\ntools/lib/bpf/bpf.c-297-{\ntools/lib/bpf/bpf.c:298:\tconst size_t attr_sz = offsetofend(union bpf_attr, cleanup_info_cnt);\ntools/lib/bpf/bpf.c-299-\tvoid *finfo = NULL, *linfo = NULL;\n--\ntools/lib/bpf/bpf.c-372-\ntools/lib/bpf/bpf.c:373:\tattr.cleanup_info = ptr_to_u64(OPTS_GET(opts, cleanup_info, NULL));\ntools/lib/bpf/bpf.c:374:\tattr.cleanup_info_cnt = OPTS_GET(opts, cleanup_info_cnt, 0);\ntools/lib/bpf/bpf.c:375:\tattr.cleanup_info_rec_size = OPTS_GET(opts, cleanup_info_rec_size, 0);\ntools/lib/bpf/bpf.c-376-\n--\ntools/lib/bpf/bpf.h=86=struct bpf_prog_load_opts {\n--\ntools/lib/bpf/bpf.h-132-\t/* exception cleanup table, from the .bpf_cleanup section */\ntools/lib/bpf/bpf.h:133:\tconst void *cleanup_info;\ntools/lib/bpf/bpf.h:134:\t__u32 cleanup_info_cnt;\ntools/lib/bpf/bpf.h:135:\t__u32 cleanup_info_rec_size;\ntools/lib/bpf/bpf.h-136-\tsize_t :0;\ntools/lib/bpf/bpf.h-137-};\ntools/lib/bpf/bpf.h:138:#define bpf_prog_load_opts__last_field cleanup_info_rec_size\ntools/lib/bpf/bpf.h-139-\n--\ntools/lib/bpf/gen_loader.c=985=static void info_blob_bswap(struct bpf_gen *gen, int func_info, int line_info,\ntools/lib/bpf/gen_loader.c:986:\t\t\t int core_relos, int cleanup_info,\ntools/lib/bpf/gen_loader.c-987-\t\t\t struct bpf_prog_load_opts *load_attr)\n--\ntools/lib/bpf/gen_loader.c-991-\tstruct bpf_core_relo *cr = gen-\u003edata_start + core_relos;\ntools/lib/bpf/gen_loader.c:992:\tstruct bpf_cleanup_info *ci = gen-\u003edata_start + cleanup_info;\ntools/lib/bpf/gen_loader.c-993-\tint i;\n--\ntools/lib/bpf/gen_loader.c-1003-\ntools/lib/bpf/gen_loader.c:1004:\tfor (i = 0; i \u003c load_attr-\u003ecleanup_info_cnt; i++)\ntools/lib/bpf/gen_loader.c:1005:\t\tbpf_cleanup_info_bswap(ci++);\ntools/lib/bpf/gen_loader.c-1006-}\n--\ntools/lib/bpf/gen_loader.c=1008=void bpf_gen__prog_load(struct bpf_gen *gen,\n--\ntools/lib/bpf/gen_loader.c-1018-\t\t\t sizeof(struct bpf_core_relo);\ntools/lib/bpf/gen_loader.c:1019:\tint cleanup_info_tot_sz = load_attr-\u003ecleanup_info_cnt *\ntools/lib/bpf/gen_loader.c:1020:\t\t\t\t load_attr-\u003ecleanup_info_rec_size;\ntools/lib/bpf/gen_loader.c-1021-\tint prog_load_attr, license_off, insns_off, func_info, line_info, core_relos;\ntools/lib/bpf/gen_loader.c:1022:\tint attr_size = offsetofend(union bpf_attr, cleanup_info_cnt);\ntools/lib/bpf/gen_loader.c:1023:\tint cleanup_info;\ntools/lib/bpf/gen_loader.c-1024-\tunion bpf_attr attr;\n--\ntools/lib/bpf/gen_loader.c-1071-\ntools/lib/bpf/gen_loader.c:1072:\tattr.cleanup_info_rec_size = tgt_endian(load_attr-\u003ecleanup_info_rec_size);\ntools/lib/bpf/gen_loader.c:1073:\tattr.cleanup_info_cnt = tgt_endian(load_attr-\u003ecleanup_info_cnt);\ntools/lib/bpf/gen_loader.c:1074:\tcleanup_info = add_data(gen, load_attr-\u003ecleanup_info, cleanup_info_tot_sz);\ntools/lib/bpf/gen_loader.c:1075:\tpr_debug(\"gen: prog_load: cleanup_info: off %d cnt %u rec size %u\\n\",\ntools/lib/bpf/gen_loader.c:1076:\t\t cleanup_info, load_attr-\u003ecleanup_info_cnt,\ntools/lib/bpf/gen_loader.c:1077:\t\t load_attr-\u003ecleanup_info_rec_size);\ntools/lib/bpf/gen_loader.c-1078-\n--\ntools/lib/bpf/gen_loader.c-1080-\tif (gen-\u003eswapped_endian \u0026\u0026 !gen-\u003eerror)\ntools/lib/bpf/gen_loader.c:1081:\t\tinfo_blob_bswap(gen, func_info, line_info, core_relos, cleanup_info,\ntools/lib/bpf/gen_loader.c-1082-\t\t\t\tload_attr);\n--\ntools/lib/bpf/gen_loader.c-1103-\ntools/lib/bpf/gen_loader.c:1104:\t/* populate union bpf_attr with a pointer to cleanup_info, if there is one */\ntools/lib/bpf/gen_loader.c:1105:\tif (load_attr-\u003ecleanup_info_cnt)\ntools/lib/bpf/gen_loader.c:1106:\t\temit_rel_store(gen, attr_field(prog_load_attr, cleanup_info),\ntools/lib/bpf/gen_loader.c:1107:\t\t\t cleanup_info);\ntools/lib/bpf/gen_loader.c-1108-\n--\ntools/lib/bpf/libbpf.c=450=struct bpf_program {\n--\ntools/lib/bpf/libbpf.c-517-\ntools/lib/bpf/libbpf.c:518:\tstruct bpf_cleanup_info *cleanup_info;\ntools/lib/bpf/libbpf.c:519:\t__u32 cleanup_info_rec_size;\ntools/lib/bpf/libbpf.c:520:\t__u32 cleanup_info_cnt;\ntools/lib/bpf/libbpf.c-521-\n--\ntools/lib/bpf/libbpf.c=680=struct elf_sec_desc {\n--\ntools/lib/bpf/libbpf.c-685-\ntools/lib/bpf/libbpf.c:686:#define CLEANUP_REC_FIELDS\t(sizeof(struct bpf_cleanup_info) / sizeof(__u32))\ntools/lib/bpf/libbpf.c-687-\ntools/lib/bpf/libbpf.c:688:/* Index of each field of struct bpf_cleanup_info, read as an array of __u32. */\ntools/lib/bpf/libbpf.c-689-enum {\n--\ntools/lib/bpf/libbpf.c=840=static void bpf_program__exit(struct bpf_program *prog)\n--\ntools/lib/bpf/libbpf.c-849-\tzfree(\u0026prog-\u003ereloc_desc);\ntools/lib/bpf/libbpf.c:850:\tzfree(\u0026prog-\u003ecleanup_info);\ntools/lib/bpf/libbpf.c-851-\ntools/lib/bpf/libbpf.c:852:\tprog-\u003ecleanup_info_rec_size = 0;\ntools/lib/bpf/libbpf.c:853:\tprog-\u003ecleanup_info_cnt = 0;\ntools/lib/bpf/libbpf.c-854-\tprog-\u003enr_reloc = 0;\n--\ntools/lib/bpf/libbpf.c=4865=static struct bpf_program *find_prog_by_sec_insn(const struct bpf_object *obj,\n--\ntools/lib/bpf/libbpf.c-4892-\ntools/lib/bpf/libbpf.c:4893:static int bpf_object__init_cleanup_info(struct bpf_object *obj)\ntools/lib/bpf/libbpf.c-4894-{\n--\ntools/lib/bpf/libbpf.c-4921-\t}\ntools/lib/bpf/libbpf.c:4922:\tif (data-\u003ed_size % sizeof(struct bpf_cleanup_info)) {\ntools/lib/bpf/libbpf.c-4923-\t\tpr_warn(\"%s size %zu is not a multiple of the record size %zu\\n\",\ntools/lib/bpf/libbpf.c:4924:\t\t\tCLEANUP_SEC, data-\u003ed_size, sizeof(struct bpf_cleanup_info));\ntools/lib/bpf/libbpf.c-4925-\t\treturn -LIBBPF_ERRNO__FORMAT;\n--\ntools/lib/bpf/libbpf.c-4929-\tnslots = data-\u003ed_size / sizeof(__u32);\ntools/lib/bpf/libbpf.c:4930:\tnrecs = data-\u003ed_size / sizeof(struct bpf_cleanup_info);\ntools/lib/bpf/libbpf.c-4931-\n--\ntools/lib/bpf/libbpf.c-4998-\ntools/lib/bpf/libbpf.c:4999:static int cmp_cleanup_info(const void *a, const void *b)\ntools/lib/bpf/libbpf.c-5000-{\ntools/lib/bpf/libbpf.c:5001:\tconst struct bpf_cleanup_info *x = a, *y = b;\ntools/lib/bpf/libbpf.c-5002-\n--\ntools/lib/bpf/libbpf.c-5007-\ntools/lib/bpf/libbpf.c:5008:static int bpf_prog_collect_cleanup_info(struct bpf_object *obj,\ntools/lib/bpf/libbpf.c-5009-\t\t\t\t\t struct bpf_program *prog)\n--\ntools/lib/bpf/libbpf.c-5016-\t\tstruct bpf_program *owner = NULL;\ntools/lib/bpf/libbpf.c:5017:\t\tstruct bpf_cleanup_info ci = {};\ntools/lib/bpf/libbpf.c-5018-\t\t__u32 fields[CLEANUP_REC_FIELDS];\n--\ntools/lib/bpf/libbpf.c-5083-\ntools/lib/bpf/libbpf.c:5084:\t\ttmp = libbpf_reallocarray(prog-\u003ecleanup_info, prog-\u003ecleanup_info_cnt + 1,\ntools/lib/bpf/libbpf.c:5085:\t\t\t\t\t sizeof(*prog-\u003ecleanup_info));\ntools/lib/bpf/libbpf.c-5086-\t\tif (!tmp)\ntools/lib/bpf/libbpf.c-5087-\t\t\treturn -ENOMEM;\ntools/lib/bpf/libbpf.c:5088:\t\tprog-\u003ecleanup_info = tmp;\ntools/lib/bpf/libbpf.c:5089:\t\tprog-\u003ecleanup_info_rec_size = sizeof(struct bpf_cleanup_info);\ntools/lib/bpf/libbpf.c:5090:\t\tprog-\u003ecleanup_info[prog-\u003ecleanup_info_cnt++] = ci;\ntools/lib/bpf/libbpf.c-5091-\n--\ntools/lib/bpf/libbpf.c-5095-\ntools/lib/bpf/libbpf.c:5096:\tif (!prog-\u003ecleanup_info_cnt)\ntools/lib/bpf/libbpf.c-5097-\t\treturn 0;\ntools/lib/bpf/libbpf.c-5098-\ntools/lib/bpf/libbpf.c:5099:\tqsort(prog-\u003ecleanup_info, prog-\u003ecleanup_info_cnt,\ntools/lib/bpf/libbpf.c:5100:\t sizeof(*prog-\u003ecleanup_info), cmp_cleanup_info);\ntools/lib/bpf/libbpf.c:5101:\tfor (i = 1; i \u003c prog-\u003ecleanup_info_cnt; i++) {\ntools/lib/bpf/libbpf.c:5102:\t\tstruct bpf_cleanup_info *prev = \u0026prog-\u003ecleanup_info[i - 1];\ntools/lib/bpf/libbpf.c:5103:\t\tstruct bpf_cleanup_info *cur = \u0026prog-\u003ecleanup_info[i];\ntools/lib/bpf/libbpf.c-5104-\n--\ntools/lib/bpf/libbpf.c=7729=static int bpf_object__relocate(struct bpf_object *obj, const char *targ_btf_path)\n--\ntools/lib/bpf/libbpf.c-7825-\ntools/lib/bpf/libbpf.c:7826:\t\terr = bpf_prog_collect_cleanup_info(obj, prog);\ntools/lib/bpf/libbpf.c-7827-\t\tif (err) {\n--\ntools/lib/bpf/libbpf.c=8240=static int bpf_object_load_prog(struct bpf_object *obj, struct bpf_program *prog,\n--\ntools/lib/bpf/libbpf.c-8296-\t}\ntools/lib/bpf/libbpf.c:8297:\tif (prog-\u003ecleanup_info_cnt) {\ntools/lib/bpf/libbpf.c:8298:\t\tload_attr.cleanup_info = prog-\u003ecleanup_info;\ntools/lib/bpf/libbpf.c:8299:\t\tload_attr.cleanup_info_cnt = prog-\u003ecleanup_info_cnt;\ntools/lib/bpf/libbpf.c:8300:\t\tload_attr.cleanup_info_rec_size = prog-\u003ecleanup_info_rec_size;\ntools/lib/bpf/libbpf.c-8301-\t}\n--\ntools/lib/bpf/libbpf.c=8764=static struct bpf_object *bpf_object_open(const char *path, const void *obj_buf, size_t obj_buf_sz,\n--\ntools/lib/bpf/libbpf.c-8857-\terr = err ? : bpf_object__collect_relos(obj);\ntools/lib/bpf/libbpf.c:8858:\terr = err ? : bpf_object__init_cleanup_info(obj);\ntools/lib/bpf/libbpf.c-8859-\tif (err)\n--\ntools/lib/bpf/libbpf.c=10301=int bpf_program__clone(struct bpf_program *prog, const struct bpf_prog_load_opts *opts)\n--\ntools/lib/bpf/libbpf.c-10367-\t/* exception cleanup table */\ntools/lib/bpf/libbpf.c:10368:\tinfo = OPTS_GET(opts, cleanup_info, NULL);\ntools/lib/bpf/libbpf.c:10369:\tinfo_cnt = OPTS_GET(opts, cleanup_info_cnt, 0);\ntools/lib/bpf/libbpf.c:10370:\tinfo_rec_size = OPTS_GET(opts, cleanup_info_rec_size, 0);\ntools/lib/bpf/libbpf.c-10371-\tif (!!info != !!info_cnt || !!info != !!info_rec_size) {\ntools/lib/bpf/libbpf.c:10372:\t\tpr_warn(\"prog '%s': cleanup_info, cleanup_info_cnt, and cleanup_info_rec_size must all be specified or all omitted\\n\",\ntools/lib/bpf/libbpf.c-10373-\t\t\tprog-\u003ename);\n--\ntools/lib/bpf/libbpf.c-10376-\tif (info) {\ntools/lib/bpf/libbpf.c:10377:\t\tattr.cleanup_info = info;\ntools/lib/bpf/libbpf.c:10378:\t\tattr.cleanup_info_cnt = info_cnt;\ntools/lib/bpf/libbpf.c:10379:\t\tattr.cleanup_info_rec_size = info_rec_size;\ntools/lib/bpf/libbpf.c:10380:\t} else if (prog-\u003ecleanup_info_cnt) {\ntools/lib/bpf/libbpf.c:10381:\t\tattr.cleanup_info = prog-\u003ecleanup_info;\ntools/lib/bpf/libbpf.c:10382:\t\tattr.cleanup_info_cnt = prog-\u003ecleanup_info_cnt;\ntools/lib/bpf/libbpf.c:10383:\t\tattr.cleanup_info_rec_size = prog-\u003ecleanup_info_rec_size;\ntools/lib/bpf/libbpf.c-10384-\t}\n--\ntools/lib/bpf/libbpf_internal.h=567=static inline void bpf_core_relo_bswap(struct bpf_core_relo *i)\n--\ntools/lib/bpf/libbpf_internal.h-574-\ntools/lib/bpf/libbpf_internal.h:575:static inline void bpf_cleanup_info_bswap(struct bpf_cleanup_info *i)\ntools/lib/bpf/libbpf_internal.h-576-{\n"
} |
| 25/3 | 2026/09/22 09:10 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 92457 output: 35 thoughts: 1045 Thoughts: **BPF Patch Analysis: Memory, Initialization, and Security** Okay, let's dive into this patch and scrutinize it for any potential memory-related issues, paying close attention to memory allocations, data copying to user space, and the use of uninitialized variables. We need to be meticulous. First up, `arch/x86/net/bpf_cleanup_pad.S`. This assembly routine, `arch_bpf_run_cleanup_pad`, looks like it could be a spot for uninitialized memory to rear its head, especially with the register spilling and restoration. The prologue and throw-site are the key areas. But, wait, the `bpf_cleanup_force_spill()` function looks promising. This decides whether we need to spill callee-saved registers. We're looking at `arch/x86/net/bpf_jit_comp.c` to see how these registers are saved. The code spills r6-r9 and r12. It looks like the prologue is saving the registers. We're good. But, a quick glance at the arm64 architecture reveals a similar mechanism is in place, as it should be. Now, let's shift to `kernel/bpf/backtrack.c`. We're going over the initialization of `p->unwind_frames` within `bpf_push_jmp_history` - great, it's explicitly initialized to zero, which is what we want. Next, it's `kernel/bpf/cfg.c`. The `pad_idx` variable in `visit_cleanup_pad_edge` gets a similar treatment, being set to zero as well. This file seems to be handling control flow analysis, so this makes sense. `kernel/bpf/check_btf.c` calls for careful examination: `check_cleanup_info` uses `kvcalloc` for memory allocation, which is good. We also see `copy_from_bpfptr` to copy user space data. This needs careful validation, and the code includes bounds checks and validation of records. We are looking for vulnerabilities here, and it looks like `copy_to_bpfptr_offset` has checks for `-E2BIG` which is correct. Moving on to `kernel/bpf/core.c`, the `__bpf_prog_free` function looks safe and is calling `bpf_cleanup_free_info`. Furthermore, `bpf_cleanup_attach_main_prog` is being called with error handling, which looks sound. Finally, we're going to give `kernel/bpf/exception.c` a thorough once-over. We need to look for any hidden issues there. |
| 26/3 | 2026/09/22 09:10 | tool | read-file |
0mArgs:{
"File": "kernel/bpf/exception.c",
"FirstLine": 1,
"LineCount": 100
}Results: {
"Contents": " 1:\t// SPDX-License-Identifier: GPL-2.0-only\n 2:\t/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */\n 3:\t#include \u003clinux/bitmap.h\u003e\n 4:\t#include \u003clinux/bpf.h\u003e\n 5:\t#include \u003clinux/bpf_verifier.h\u003e\n 6:\t#include \u003clinux/bsearch.h\u003e\n 7:\t#include \u003clinux/btf.h\u003e\n 8:\t#include \u003clinux/btf_ids.h\u003e\n 9:\t#include \u003clinux/filter.h\u003e\n 10:\t#include \u003clinux/slab.h\u003e\n 11:\t#include \u003clinux/sort.h\u003e\n 12:\t#include \"exception.h\"\n 13:\t\n 14:\t#define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)\n 15:\t\n 16:\tenum exc_kfunc {\n 17:\t\tEXC_KF_bpf_unwind_resume,\n 18:\t};\n 19:\t\n 20:\tBTF_ID_LIST(exc_kfunc_list)\n 21:\tBTF_ID(func, bpf_unwind_resume)\n 22:\t\n 23:\tstatic bool insn_is_exc_kfunc(const struct bpf_insn *insn, int kf)\n 24:\t{\n 25:\t\treturn bpf_pseudo_kfunc_call(insn) \u0026\u0026 insn-\u003eoff == 0 \u0026\u0026\n 26:\t\t insn-\u003eimm == exc_kfunc_list[kf];\n 27:\t}\n 28:\t\n 29:\t/* What an instruction does to intra-subprog control flow. */\n 30:\tenum cleanup_insn_kind {\n 31:\t\tCLEANUP_INSN_PLAIN,\t/* the next insn runs */\n 32:\t\tCLEANUP_INSN_JUMP,\t/* unconditional jump */\n 33:\t\tCLEANUP_INSN_COND,\t/* the next insn runs, or the branch target */\n 34:\t\tCLEANUP_INSN_EXIT,\n 35:\t\tCLEANUP_INSN_THROW,\t/* call bpf_throw: nothing after it runs */\n 36:\t\tCLEANUP_INSN_RESUME,\t/* call bpf_unwind_resume: likewise */\n 37:\t\tCLEANUP_INSN_CALL,\t/* call to another subprog */\n 38:\t\tCLEANUP_INSN_GOTOX,\t/* indirect jump: successors not known here */\n 39:\t};\n 40:\t\n 41:\t/* What each instruction can reach, computed once by cleanup_reachability(). */\n 42:\t#define CLEANUP_REACH_RESUME\tBIT(0)\t/* a bpf_unwind_resume() call */\n 43:\t#define CLEANUP_REACH_EXIT\tBIT(1)\t/* a plain BPF_EXIT */\n 44:\t#define CLEANUP_REACH_UNKNOWN\tBIT(2)\t/* an indirect jump */\n 45:\t#define CLEANUP_REACH_THROW\tBIT(3)\t/* a bpf_throw() call */\n 46:\t\n 47:\t/* Scratch shared by the analyses, sized once so no walker has to allocate. */\n 48:\tstruct cleanup_ctx {\n 49:\t\tstruct bpf_verifier_env *env;\n 50:\t\tu8 *reach;\t\t/* per insn: CLEANUP_REACH_* mask */\n 51:\t\tu32 *stack;\t\t/* per insn: DFS stack */\n 52:\t\tvoid *scratch;\t\t/* the one allocation all of the above live in */\n 53:\t};\n 54:\t\n 55:\tstatic bool in_pad(struct bpf_verifier_env *env, u32 i)\n 56:\t{\n 57:\t\treturn env-\u003einsn_aux_data[i].in_cleanup_pad;\n 58:\t}\n 59:\t\n 60:\t/* One scratch array for cleanup_alloc() to hand out. */\n 61:\tstruct cleanup_alloc_req {\n 62:\t\tvoid **dst;\n 63:\t\tsize_t n, sz;\n 64:\t};\n 65:\t\n 66:\tstatic void *cleanup_alloc(const struct cleanup_alloc_req *tab, u32 cnt)\n 67:\t{\n 68:\t\tsize_t total = 0;\n 69:\t\tchar *block, *p;\n 70:\t\tu32 i;\n 71:\t\n 72:\t\tfor (i = 0; i \u003c cnt; i++)\n 73:\t\t\ttotal += round_up(tab[i].n * tab[i].sz, 8);\n 74:\t\n 75:\t\tblock = kvzalloc(total, GFP_KERNEL_ACCOUNT | __GFP_NOWARN);\n 76:\t\tif (!block)\n 77:\t\t\treturn NULL;\n 78:\t\n 79:\t\tfor (i = 0, p = block; i \u003c cnt; i++) {\n 80:\t\t\t*tab[i].dst = p;\n 81:\t\t\tp += round_up(tab[i].n * tab[i].sz, 8);\n 82:\t\t}\n 83:\t\treturn block;\n 84:\t}\n 85:\t\n 86:\tstatic int cleanup_subprog_of(struct bpf_verifier_env *env, u32 off)\n 87:\t{\n 88:\t\tstruct bpf_subprog_info *info = bpf_find_containing_subprog(env, off);\n 89:\t\n 90:\t\treturn info ? info - env-\u003esubprog_info : -1;\n 91:\t}\n 92:\t\n 93:\t/* The subprogram a linear pass is currently in. */\n 94:\tstruct cleanup_cursor {\n 95:\t\tu32 start, end;\t\t/* [start, end) of the current subprogram */\n 96:\t\tint sub;\t\t/* its index */\n 97:\t};\n 98:\t\n 99:\t#define CLEANUP_CURSOR_INIT { .sub = -1 }\n 100:\t\n"
} |
| 27/3 | 2026/09/22 09:10 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 95720 output: 37 thoughts: 17 Thoughts: Okay, let's dive into the core of this `exception.c` file within the BPF kernel module. **My Analysis of `kernel/bpf/exception.c`** Alright, I'm thinking, let's get into the details of handling exceptions within the BPF context. This is where things can get interesting, particularly in the realm of security and robustness. I need to understand how the kernel gracefully manages potential issues that might arise during BPF program execution. Specifically, I want to see how they're handling things like out-of-bounds access, division by zero, and invalid instructions. I'm also curious about the mechanisms in place to prevent a malicious BPF program from crashing the system. I'll need to pay close attention to the error handling and recovery strategies. Are they using traps, signals, or something more bespoke? Furthermore, I'll be looking to see if the exception handling differentiates between user-space and kernel-space BPF programs. I expect to find mechanisms to ensure that an exception within a BPF program doesn't compromise the kernel's stability. Any advanced techniques such as sandboxing or seccomp are a plus. I also want to see how these exceptions interact with the larger BPF verifier and interpreter flow. How does the system signal and handle these issues without interfering with the program execution? |
| 28/3 | 2026/09/22 09:10 | tool | read-file |
0mArgs:{
"File": "kernel/bpf/exception.c",
"FirstLine": 101,
"LineCount": 100
}Results: {
"Contents": " 101:\tstatic void cleanup_cursor_to(struct bpf_verifier_env *env, struct cleanup_cursor *c, u32 i)\n 102:\t{\n 103:\t\twhile (i \u003e= c-\u003eend) {\n 104:\t\t\tc-\u003esub++;\n 105:\t\t\tc-\u003estart = env-\u003esubprog_info[c-\u003esub].start;\n 106:\t\t\tc-\u003eend = env-\u003esubprog_info[c-\u003esub + 1].start;\n 107:\t\t}\n 108:\t}\n 109:\t\n 110:\tstatic enum cleanup_insn_kind cleanup_classify(struct bpf_verifier_env *env, u32 i,\n 111:\t\t\t\t\t\t int *next, int *target)\n 112:\t{\n 113:\t\tstruct bpf_insn *insn = \u0026env-\u003eprog-\u003einsnsi[i];\n 114:\t\tu8 class = BPF_CLASS(insn-\u003ecode);\n 115:\t\n 116:\t\t*next = i + 1;\n 117:\t\t*target = -1;\n 118:\t\n 119:\t\tif (insn-\u003ecode == (BPF_LD | BPF_IMM | BPF_DW)) {\n 120:\t\t\t*next = i + 2;\n 121:\t\t\treturn CLEANUP_INSN_PLAIN;\n 122:\t\t}\n 123:\t\tif (class != BPF_JMP \u0026\u0026 class != BPF_JMP32)\n 124:\t\t\treturn CLEANUP_INSN_PLAIN;\n 125:\t\n 126:\t\tswitch (BPF_OP(insn-\u003ecode)) {\n 127:\t\tcase BPF_EXIT:\n 128:\t\t\t*next = -1;\n 129:\t\t\treturn CLEANUP_INSN_EXIT;\n 130:\t\tcase BPF_JA:\n 131:\t\t\t*next = -1;\n 132:\t\t\tif (BPF_SRC(insn-\u003ecode) == BPF_X)\n 133:\t\t\t\treturn CLEANUP_INSN_GOTOX;\n 134:\t\t\t*target = class == BPF_JMP32 ? i + insn-\u003eimm + 1 : i + insn-\u003eoff + 1;\n 135:\t\t\treturn CLEANUP_INSN_JUMP;\n 136:\t\tcase BPF_CALL:\n 137:\t\t\tif (bpf_is_throw_kfunc(insn)) {\n 138:\t\t\t\t*next = -1;\n 139:\t\t\t\treturn CLEANUP_INSN_THROW;\n 140:\t\t\t}\n 141:\t\t\tif (insn_is_exc_kfunc(insn, EXC_KF_bpf_unwind_resume)) {\n 142:\t\t\t\t*next = -1;\n 143:\t\t\t\treturn CLEANUP_INSN_RESUME;\n 144:\t\t\t}\n 145:\t\t\treturn bpf_pseudo_call(insn) ? CLEANUP_INSN_CALL : CLEANUP_INSN_PLAIN;\n 146:\t\tdefault:\n 147:\t\t\t/* Conditional jump, including BPF_JCOND. */\n 148:\t\t\t*target = i + insn-\u003eoff + 1;\n 149:\t\t\treturn CLEANUP_INSN_COND;\n 150:\t\t}\n 151:\t}\n 152:\t\n 153:\tstatic void cleanup_mark_kfunc_sites(struct bpf_verifier_env *env)\n 154:\t{\n 155:\t\tu32 i;\n 156:\t\n 157:\t\tfor (i = 0; i \u003c env-\u003eprog-\u003elen; i++) {\n 158:\t\t\tstruct bpf_insn *insn = \u0026env-\u003eprog-\u003einsnsi[i];\n 159:\t\n 160:\t\t\tif (bpf_is_throw_kfunc(insn))\n 161:\t\t\t\tenv-\u003einsn_aux_data[i].cleanup_throw_site = true;\n 162:\t\t\telse if (insn_is_exc_kfunc(insn, EXC_KF_bpf_unwind_resume))\n 163:\t\t\t\tenv-\u003einsn_aux_data[i].cleanup_resume_site = true;\n 164:\t\t}\n 165:\t}\n 166:\t\n 167:\tint bpf_cleanup_check_callback(struct bpf_verifier_env *env, int subprog)\n 168:\t{\n 169:\t\tif (!env-\u003ecleanup_info_cnt || !env-\u003esubprog_info[subprog].might_throw)\n 170:\t\t\treturn 0;\n 171:\t\n 172:\t\tverbose(env, \"subprog %d may unwind and is used as a callback\\n\", subprog);\n 173:\t\treturn -EINVAL;\n 174:\t}\n 175:\t\n 176:\t/* Intra-subprog successors of @i, or -1 each when absent. */\n 177:\tstatic enum cleanup_insn_kind cleanup_succ(struct bpf_verifier_env *env, u32 i,\n 178:\t\t\t\t\t\t u32 start, u32 end, int *next, int *target)\n 179:\t{\n 180:\t\tenum cleanup_insn_kind kind = cleanup_classify(env, i, next, target);\n 181:\t\n 182:\t\tif (*next \u003c (int)start || *next \u003e= (int)end)\n 183:\t\t\t*next = -1;\n 184:\t\tif (*target \u003c (int)start || *target \u003e= (int)end)\n 185:\t\t\t*target = -1;\n 186:\t\treturn kind;\n 187:\t}\n 188:\t\n 189:\tstatic void cleanup_add_pred(u32 *head, u32 *link, u32 to, u32 e)\n 190:\t{\n 191:\t\tlink[e] = head[to];\n 192:\t\thead[to] = e + 1;\n 193:\t}\n 194:\t\n 195:\t/* What every instruction can reach along intra-subprog edges, for\n 196:\t * cleanup_pad_is_catch(). One backward walk over a predecessor index, rather\n 197:\t * than a forward walk from each landing pad, which would be quadratic.\n 198:\t */\n 199:\tstatic int cleanup_reachability(struct cleanup_ctx *ctx)\n 200:\t{\n"
} |
| 29/3 | 2026/09/22 09:10 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 97758 output: 37 thoughts: 22 Thoughts: Okay, here's my take on those instructions, interpreted as my own thought process, as requested: **Diving into the BPF Exception Handling** Alright, time to get my hands dirty. I'm going to take a deep dive into the `kernel/bpf/exception.c` file. Specifically, I'm focusing on the next hundred lines of code. Given my expertise, I know this file likely deals with how the kernel handles exceptions related to Berkeley Packet Filter (BPF) programs. I anticipate seeing the mechanics of things like invalid memory access within the BPF context, potential division by zero errors, or perhaps even safeguards against malicious code trying to trigger a kernel panic through BPF. I'm hoping to understand how these exceptions are trapped, processed, and handled in a safe and controlled manner, ideally without impacting the system's overall stability. I expect the code will reveal details about how the BPF verifier interacts with the exception handling, if it does, and I'll be looking for any hints about the trade-offs made between performance and robust exception handling. I'm also curious to see if there are any specific BPF-related error codes or signals used, and how these are communicated back to the user space BPF program. This is probably going to be interesting. |
| 30/3 | 2026/09/22 09:11 | tool | read-file |
0mArgs:{
"File": "kernel/bpf/exception.c",
"FirstLine": 201,
"LineCount": 100
}Results: {
"Contents": " 201:\t\tstruct bpf_verifier_env *env = ctx-\u003eenv;\n 202:\t\tu32 len = env-\u003eprog-\u003elen;\n 203:\t\tstruct cleanup_cursor c = CLEANUP_CURSOR_INIT;\n 204:\t\tu32 *head = NULL, *link = NULL;\n 205:\t\tbool *queued = NULL;\n 206:\t\tu32 i, sp = 0;\n 207:\t\tvoid *scratch;\n 208:\t\tconst struct cleanup_alloc_req tab[] = {\n 209:\t\t\t{ (void **)\u0026head, len, sizeof(*head) },\n 210:\t\t\t{ (void **)\u0026link, 2 * (size_t)len, sizeof(*link) },\n 211:\t\t\t{ (void **)\u0026queued, len, sizeof(*queued) },\n 212:\t\t};\n 213:\t\n 214:\t\tscratch = cleanup_alloc(tab, ARRAY_SIZE(tab));\n 215:\t\tif (!scratch)\n 216:\t\t\treturn -ENOMEM;\n 217:\t\n 218:\t\t/* Index the predecessors, and seed the walk at the terminators. */\n 219:\t\tfor (i = 0; i \u003c len; i++) {\n 220:\t\t\tenum cleanup_insn_kind kind;\n 221:\t\t\tint next, target;\n 222:\t\n 223:\t\t\tcleanup_cursor_to(env, \u0026c, i);\n 224:\t\t\tkind = cleanup_succ(env, i, c.start, c.end, \u0026next, \u0026target);\n 225:\t\n 226:\t\t\tif (kind == CLEANUP_INSN_RESUME)\n 227:\t\t\t\tctx-\u003ereach[i] |= CLEANUP_REACH_RESUME;\n 228:\t\t\telse if (kind == CLEANUP_INSN_EXIT)\n 229:\t\t\t\tctx-\u003ereach[i] |= CLEANUP_REACH_EXIT;\n 230:\t\t\telse if (kind == CLEANUP_INSN_GOTOX)\n 231:\t\t\t\tctx-\u003ereach[i] |= CLEANUP_REACH_UNKNOWN;\n 232:\t\t\telse if (kind == CLEANUP_INSN_THROW)\n 233:\t\t\t\tctx-\u003ereach[i] |= CLEANUP_REACH_THROW;\n 234:\t\n 235:\t\t\tif (next \u003e= 0)\n 236:\t\t\t\tcleanup_add_pred(head, link, next, 2 * i);\n 237:\t\t\tif (target \u003e= 0)\n 238:\t\t\t\tcleanup_add_pred(head, link, target, 2 * i + 1);\n 239:\t\n 240:\t\t\tif (ctx-\u003ereach[i]) {\n 241:\t\t\t\tqueued[i] = true;\n 242:\t\t\t\tctx-\u003estack[sp++] = i;\n 243:\t\t\t}\n 244:\t\t}\n 245:\t\n 246:\t\t/* Each instruction re-enters the worklist at most once per bit it\n 247:\t\t * gains, so this is linear in the number of edges.\n 248:\t\t */\n 249:\t\twhile (sp) {\n 250:\t\t\tu32 j = ctx-\u003estack[--sp];\n 251:\t\t\tu8 flags = ctx-\u003ereach[j];\n 252:\t\t\tu32 e;\n 253:\t\n 254:\t\t\tqueued[j] = false;\n 255:\t\t\tfor (e = head[j]; e; e = link[e - 1]) {\n 256:\t\t\t\tu32 p = (e - 1) / 2;\n 257:\t\n 258:\t\t\t\tif ((ctx-\u003ereach[p] | flags) == ctx-\u003ereach[p])\n 259:\t\t\t\t\tcontinue;\n 260:\t\t\t\tctx-\u003ereach[p] |= flags;\n 261:\t\t\t\tif (!queued[p]) {\n 262:\t\t\t\t\tqueued[p] = true;\n 263:\t\t\t\t\tctx-\u003estack[sp++] = p;\n 264:\t\t\t\t}\n 265:\t\t\t}\n 266:\t\t}\n 267:\t\tkvfree(scratch);\n 268:\t\treturn 0;\n 269:\t}\n 270:\t\n 271:\tstatic int cleanup_pad_is_catch(struct cleanup_ctx *ctx, u32 pad)\n 272:\t{\n 273:\t\tu8 reach = ctx-\u003ereach[pad];\n 274:\t\n 275:\t\tif (reach \u0026 CLEANUP_REACH_UNKNOWN) {\n 276:\t\t\tverbose(ctx-\u003eenv, \"cleanup landing pad %u reaches an indirect jump\\n\", pad);\n 277:\t\t\treturn -EINVAL;\n 278:\t\t}\n 279:\t\tif (reach \u0026 CLEANUP_REACH_THROW) {\n 280:\t\t\tverbose(ctx-\u003eenv,\n 281:\t\t\t\t\"cleanup landing pad %u can throw while an exception is in flight\\n\",\n 282:\t\t\t\tpad);\n 283:\t\t\treturn -EINVAL;\n 284:\t\t}\n 285:\t\tif (!(reach \u0026 CLEANUP_REACH_RESUME) == !(reach \u0026 CLEANUP_REACH_EXIT)) {\n 286:\t\t\tverbose(ctx-\u003eenv, \"cleanup landing pad %u %s\\n\", pad,\n 287:\t\t\t\t(reach \u0026 CLEANUP_REACH_RESUME) ?\n 288:\t\t\t\t\"reaches both bpf_unwind_resume() and a plain exit\" :\n 289:\t\t\t\t\"reaches neither bpf_unwind_resume() nor an exit\");\n 290:\t\t\treturn -EINVAL;\n 291:\t\t}\n 292:\t\treturn !!(reach \u0026 CLEANUP_REACH_EXIT);\n 293:\t}\n 294:\t\n 295:\tstatic int cleanup_check_pad_insn(struct bpf_verifier_env *env, u32 i)\n 296:\t{\n 297:\t\tstruct bpf_insn *insn = \u0026env-\u003eprog-\u003einsnsi[i];\n 298:\t\n 299:\t\tif (bpf_helper_call(insn) \u0026\u0026 insn-\u003eimm == BPF_FUNC_tail_call) {\n 300:\t\t\tverbose(env,\n"
} |
| 31/3 | 2026/09/22 09:11 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 99710 output: 37 thoughts: 22 Thoughts: Okay, here's my take on those instructions, interpreted as my own thought process, as requested: **Diving into the BPF Exception Handling** Alright, time to get my hands dirty. I'm going to take a deep dive into the `kernel/bpf/exception.c` file. Specifically, I'm focusing on the next hundred lines of code. Given my expertise, I know this file likely deals with how the kernel handles exceptions related to Berkeley Packet Filter (BPF) programs. I anticipate seeing the mechanics of things like invalid memory access within the BPF context, potential division by zero errors, or perhaps even safeguards against malicious code trying to trigger a kernel panic through BPF. I'm hoping to understand how these exceptions are trapped, processed, and handled in a safe and controlled manner, ideally without impacting the system's overall stability. I expect the code will reveal details about how the BPF verifier interacts with the exception handling, if it does, and I'll be looking for any hints about the trade-offs made between performance and robust exception handling. I'm also curious to see if there are any specific BPF-related error codes or signals used, and how these are communicated back to the user space BPF program. This is probably going to be interesting. |
| 32/3 | 2026/09/22 09:11 | tool | read-file |
0mArgs:{
"File": "kernel/bpf/exception.c",
"FirstLine": 301,
"LineCount": 100
}Results: {
"Contents": " 301:\t\t\t\t\"bpf_tail_call() at insn %u is in an exception cleanup landing pad\\n\",\n 302:\t\t\t\ti);\n 303:\t\t\treturn -EINVAL;\n 304:\t\t}\n 305:\t\t/* A failed BPF_LD_[ABS|IND] leaves the frame through its epilogue. */\n 306:\t\tif (BPF_CLASS(insn-\u003ecode) == BPF_LD \u0026\u0026\n 307:\t\t (BPF_MODE(insn-\u003ecode) == BPF_ABS || BPF_MODE(insn-\u003ecode) == BPF_IND)) {\n 308:\t\t\tverbose(env,\n 309:\t\t\t\t\"BPF_LD_[ABS|IND] at insn %u is in an exception cleanup landing pad\\n\",\n 310:\t\t\t\ti);\n 311:\t\t\treturn -EINVAL;\n 312:\t\t}\n 313:\t\t/* Stack arguments are not supported. */\n 314:\t\tif (is_stack_arg_st(insn) || is_stack_arg_stx(insn)) {\n 315:\t\t\tverbose(env,\n 316:\t\t\t\t\"insn %u passes an on-stack call argument in an exception cleanup landing pad\\n\",\n 317:\t\t\t\ti);\n 318:\t\t\treturn -EINVAL;\n 319:\t\t}\n 320:\t\t/* Likewise, stack arguments are not supported. */\n 321:\t\tif (bpf_pseudo_kfunc_call(insn)) {\n 322:\t\t\tstruct bpf_call_summary cs;\n 323:\t\n 324:\t\t\tif (bpf_get_call_summary(env, insn, \u0026cs) \u0026\u0026\n 325:\t\t\t cs.arg_slot_cnt \u003e MAX_BPF_FUNC_REG_ARGS) {\n 326:\t\t\t\tverbose(env,\n 327:\t\t\t\t\t\"insn %u passes an on-stack call argument in an exception cleanup landing pad\\n\",\n 328:\t\t\t\t\ti);\n 329:\t\t\t\treturn -EINVAL;\n 330:\t\t\t}\n 331:\t\t}\n 332:\t\treturn 0;\n 333:\t}\n 334:\t\n 335:\tstatic int cleanup_mark_pad_bodies(struct cleanup_ctx *ctx)\n 336:\t{\n 337:\t\tstruct bpf_verifier_env *env = ctx-\u003eenv;\n 338:\t\tu32 i, sp = 0;\n 339:\t\tint ret;\n 340:\t\n 341:\t\tfor (i = 0; i \u003c env-\u003ecleanup_info_cnt; i++) {\n 342:\t\t\tu32 pad = env-\u003ecleanup_info[i].landing_pad_off;\n 343:\t\n 344:\t\t\tif (in_pad(env, pad))\n 345:\t\t\t\tcontinue;\n 346:\t\n 347:\t\t\tret = cleanup_pad_is_catch(ctx, pad);\n 348:\t\t\tif (ret \u003c 0)\n 349:\t\t\t\treturn ret;\n 350:\t\t\tif (ret) {\n 351:\t\t\t\tverbose(env,\n 352:\t\t\t\t\t\"catch landing pad %u is not supported yet, only cleanup pads that resume\\n\",\n 353:\t\t\t\t\tpad);\n 354:\t\t\t\treturn -EOPNOTSUPP;\n 355:\t\t\t}\n 356:\t\t\tenv-\u003einsn_aux_data[pad].in_cleanup_pad = true;\n 357:\t\t\tctx-\u003estack[sp++] = pad;\n 358:\t\t}\n 359:\t\n 360:\t\twhile (sp) {\n 361:\t\t\tu32 j = ctx-\u003estack[--sp];\n 362:\t\t\tenum cleanup_insn_kind kind;\n 363:\t\t\tint next, target, sub;\n 364:\t\t\tu32 start, end;\n 365:\t\n 366:\t\t\tret = cleanup_check_pad_insn(env, j);\n 367:\t\t\tif (ret)\n 368:\t\t\t\treturn ret;\n 369:\t\n 370:\t\t\tsub = cleanup_subprog_of(env, j);\n 371:\t\t\tstart = env-\u003esubprog_info[sub].start;\n 372:\t\t\tend = env-\u003esubprog_info[sub + 1].start;\n 373:\t\t\tkind = cleanup_succ(env, j, start, end, \u0026next, \u0026target);\n 374:\t\n 375:\t\t\tif (kind == CLEANUP_INSN_CALL) {\n 376:\t\t\t\tint callee = cleanup_subprog_of(env, j + env-\u003eprog-\u003einsnsi[j].imm + 1);\n 377:\t\n 378:\t\t\t\tif (env-\u003esubprog_info[callee].might_throw) {\n 379:\t\t\t\t\tverbose(env,\n 380:\t\t\t\t\t\t\"cleanup landing pad calls subprog %d at insn %u, which can throw while an exception is in flight\\n\",\n 381:\t\t\t\t\t\tcallee, j);\n 382:\t\t\t\t\treturn -EINVAL;\n 383:\t\t\t\t}\n 384:\t\t\t}\n 385:\t\n 386:\t\t\tif (next \u003e= 0 \u0026\u0026 !in_pad(env, next)) {\n 387:\t\t\t\tenv-\u003einsn_aux_data[next].in_cleanup_pad = true;\n 388:\t\t\t\tctx-\u003estack[sp++] = next;\n 389:\t\t\t}\n 390:\t\t\tif (target \u003e= 0 \u0026\u0026 !in_pad(env, target)) {\n 391:\t\t\t\tenv-\u003einsn_aux_data[target].in_cleanup_pad = true;\n 392:\t\t\t\tctx-\u003estack[sp++] = target;\n 393:\t\t\t}\n 394:\t\t}\n 395:\t\treturn 0;\n 396:\t}\n 397:\t\n 398:\tstatic int cleanup_check_resumes(struct cleanup_ctx *ctx)\n 399:\t{\n 400:\t\tstruct bpf_verifier_env *env = ctx-\u003eenv;\n"
} |
| 33/3 | 2026/09/22 09:11 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 101621 output: 37 thoughts: 17 Thoughts: Okay, let's dive into the core of this `exception.c` file within the BPF kernel module. **My Analysis of `kernel/bpf/exception.c`** Alright, I'm thinking, let's get into the details of handling exceptions within the BPF context. This is where things can get interesting, particularly in the realm of security and robustness. I need to understand how the kernel gracefully manages potential issues that might arise during BPF program execution. Specifically, I want to see how they're handling things like out-of-bounds access, division by zero, and invalid instructions. I'm also curious about the mechanisms in place to prevent a malicious BPF program from crashing the system. I'll need to pay close attention to the error handling and recovery strategies. Are they using traps, signals, or something more bespoke? Furthermore, I'll be looking to see if the exception handling differentiates between user-space and kernel-space BPF programs. I expect to find mechanisms to ensure that an exception within a BPF program doesn't compromise the kernel's stability. Any advanced techniques such as sandboxing or seccomp are a plus. I also want to see how these exceptions interact with the larger BPF verifier and interpreter flow. How does the system signal and handle these issues without interfering with the program execution? |
| 34/3 | 2026/09/22 09:11 | tool | read-file |
0mArgs:{
"File": "kernel/bpf/exception.c",
"FirstLine": 401,
"LineCount": 100
}Results: {
"Contents": " 401:\t\tu32 i;\n 402:\t\n 403:\t\tfor (i = 0; i \u003c env-\u003eprog-\u003elen; i++) {\n 404:\t\t\tif (!insn_is_exc_kfunc(\u0026env-\u003eprog-\u003einsnsi[i], EXC_KF_bpf_unwind_resume))\n 405:\t\t\t\tcontinue;\n 406:\t\t\tif (in_pad(env, i))\n 407:\t\t\t\tcontinue;\n 408:\t\t\tverbose(env,\n 409:\t\t\t\t\"bpf_unwind_resume() at insn %u is not in an exception cleanup landing pad\\n\",\n 410:\t\t\t\ti);\n 411:\t\t\treturn -EINVAL;\n 412:\t\t}\n 413:\t\treturn 0;\n 414:\t}\n 415:\t\n 416:\tstatic void cleanup_mark_call_sites(struct bpf_verifier_env *env)\n 417:\t{\n 418:\t\tu32 i, j;\n 419:\t\n 420:\t\tfor (i = 0; i \u003c env-\u003ecleanup_info_cnt; i++) {\n 421:\t\t\tstruct bpf_cleanup_info *rec = \u0026env-\u003ecleanup_info[i];\n 422:\t\n 423:\t\t\tfor (j = rec-\u003ebegin_off; j \u003c rec-\u003eend_off; j++) {\n 424:\t\t\t\tstruct bpf_insn *insn = \u0026env-\u003eprog-\u003einsnsi[j];\n 425:\t\n 426:\t\t\t\tif (!bpf_pseudo_call(insn) \u0026\u0026 !bpf_is_throw_kfunc(insn))\n 427:\t\t\t\t\tcontinue;\n 428:\t\t\t\tenv-\u003einsn_aux_data[j].cleanup_pad = rec-\u003elanding_pad_off + 1;\n 429:\t\t\t}\n 430:\t\t}\n 431:\t}\n 432:\t\n 433:\tint bpf_prepare_cleanup_exceptions(struct bpf_verifier_env *env)\n 434:\t{\n 435:\t\tif (!env-\u003ecleanup_info_cnt)\n 436:\t\t\treturn 0;\n 437:\t\n 438:\t\tif (bpf_prog_is_offloaded(env-\u003eprog-\u003eaux)) {\n 439:\t\t\tverbose(env,\n 440:\t\t\t\t\"exception cleanup is not supported for offloaded programs\\n\");\n 441:\t\t\treturn -EINVAL;\n 442:\t\t}\n 443:\t\n 444:\t\tif (!bpf_jit_supports_cleanup_pads() || !env-\u003eprog-\u003ejit_requested) {\n 445:\t\t\tverbose(env,\n 446:\t\t\t\t\"exception cleanup needs a JIT that can dispatch landing pads\\n\");\n 447:\t\t\treturn -EOPNOTSUPP;\n 448:\t\t}\n 449:\t\tenv-\u003eprog-\u003ejit_required = 1;\n 450:\t\n 451:\t\tif (env-\u003eexception_callback_subprog) {\n 452:\t\t\tverbose(env,\n 453:\t\t\t\t\"exception cleanup table cannot be combined with an exception callback\\n\");\n 454:\t\t\treturn -EINVAL;\n 455:\t\t}\n 456:\t\n 457:\t\tcleanup_mark_kfunc_sites(env);\n 458:\t\tcleanup_mark_call_sites(env);\n 459:\t\treturn 0;\n 460:\t}\n 461:\t\n 462:\tint bpf_check_cleanup_exceptions(struct bpf_verifier_env *env)\n 463:\t{\n 464:\t\tu32 len = env-\u003eprog-\u003elen;\n 465:\t\tstruct cleanup_ctx ctx = { .env = env };\n 466:\t\tconst struct cleanup_alloc_req tab[] = {\n 467:\t\t\t{ (void **)\u0026ctx.reach, len, sizeof(*ctx.reach) },\n 468:\t\t\t{ (void **)\u0026ctx.stack, len, sizeof(*ctx.stack) },\n 469:\t\t};\n 470:\t\tint ret;\n 471:\t\n 472:\t\tif (!env-\u003ecleanup_info_cnt)\n 473:\t\t\treturn 0;\n 474:\t\n 475:\t\tctx.scratch = cleanup_alloc(tab, ARRAY_SIZE(tab));\n 476:\t\tif (!ctx.scratch)\n 477:\t\t\treturn -ENOMEM;\n 478:\t\n 479:\t\tret = cleanup_reachability(\u0026ctx);\n 480:\t\tif (ret)\n 481:\t\t\tgoto out;\n 482:\t\n 483:\t\tret = cleanup_mark_pad_bodies(\u0026ctx);\n 484:\t\tif (ret)\n 485:\t\t\tgoto out;\n 486:\t\n 487:\t\tret = cleanup_check_resumes(\u0026ctx);\n 488:\tout:\n 489:\t\tkvfree(ctx.scratch);\n 490:\t\treturn ret;\n 491:\t}\n 492:\t\n 493:\tbool bpf_is_unwind_resume_kfunc(const struct bpf_insn *insn)\n 494:\t{\n 495:\t\treturn insn_is_exc_kfunc(insn, EXC_KF_bpf_unwind_resume);\n 496:\t}\n 497:\t\n 498:\tint bpf_cleanup_pad_of_call(struct bpf_verifier_env *env, u32 idx)\n 499:\t{\n 500:\t\tu32 pad = env-\u003einsn_aux_data[idx].cleanup_pad;\n"
} |
| 35/3 | 2026/09/22 09:11 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 103449 output: 37 thoughts: 22 Thoughts: Here's my take on the kernel BPF exception handling, focusing on the tail end of `kernel/bpf/exception.c`: **Diving into BPF Exception Handling: The Endgame** Okay, time to zero in on the nitty-gritty. I'm focusing on the code from line 501 to the end of `kernel/bpf/exception.c`. That chunk of code is where the kernel's handling of BPF exceptions is getting finalized. I expect to see the meat of how the kernel actually *deals* with those nasty little runtime errors that can pop up in a BPF program. Specifically, this should encompass the final stages. That means things like: * **Cleanup and Error Reporting:** What exactly gets cleaned up when an exception hits? Do we free resources, log the error, and notify the user space? Where does the error information actually go? I'll be looking for the mechanisms that funnel that crucial exception information back up to user space so we can debug the broken BPF program. * **Safety Net:** What does the kernel do to ensure that even a badly behaved BPF program doesn't bring the entire system down? We're dealing with potentially untrusted code, and the kernel must have extremely robust checks and fail-safes. I'm anticipating the presence of defensive measures here to protect kernel integrity. * **Context Preservation (or Destruction):** Does the kernel attempt to preserve any context when an exception occurs, or does it basically say "game over" and nuke the affected context? If there's context preservation, how is that achieved? If it's destruction, how does it ensure the kernel is left in a safe state, and how is it managed? * **Error Codes and Signals:** What specific error codes are used, and how are these mapped back to specific issues within the BPF code? Are signals used to communicate exceptions to the application? I'll be looking for the mapping between internal error codes and the information user-space ultimately receives. Essentially, I'm expecting to see the final stages of the exception handling process that determines how a buggy BPF program impacts the rest of the system. I'm hoping to understand how robust and well-designed this handling is, given the critical security implications of running potentially untrusted code in the kernel. Let's dig in and see what they've come up with. |
| 36/3 | 2026/09/22 09:11 | tool | read-file |
0mArgs:{
"File": "kernel/bpf/exception.c",
"FirstLine": 501,
"LineCount": 100
}Results: {
"Contents": " 501:\t\n 502:\t\treturn pad ? (int)pad - 1 : -1;\n 503:\t}\n 504:\t\n 505:\t/*\n 506:\t * Every subprogram of a cleanup-carrying program spills the BPF callee-saved\n 507:\t * registers, even one that never throws: a frame's spill holds its caller's\n 508:\t * registers, and that is what the walker restores before running the caller's\n 509:\t * pad. The exception callback does not, because it reuses the boundary frame\n 510:\t * rather than building one of its own.\n 511:\t */\n 512:\tbool bpf_cleanup_force_spill(const struct bpf_prog *prog)\n 513:\t{\n 514:\t\treturn prog-\u003eaux-\u003eexc \u0026\u0026 !prog-\u003eaux-\u003eexception_cb;\n 515:\t}\n 516:\t\n 517:\tconst struct bpf_cleanup_range *bpf_cleanup_pad_for_ip(const struct bpf_prog *prog, u64 ip)\n 518:\t{\n 519:\t\tconst struct bpf_exception_info *exc = prog-\u003eaux-\u003eexc;\n 520:\t\tu32 l = 0, r = exc ? exc-\u003enr_ranges : 0;\n 521:\t\n 522:\t\twhile (l \u003c r) {\n 523:\t\t\tu32 m = l + (r - l) / 2;\n 524:\t\t\tconst struct bpf_cleanup_range *rec = \u0026exc-\u003eranges[m];\n 525:\t\n 526:\t\t\tif (ip \u003c= rec-\u003ebegin)\n 527:\t\t\t\tr = m;\n 528:\t\t\telse if (ip \u003e rec-\u003eend)\n 529:\t\t\t\tl = m + 1;\n 530:\t\t\telse\n 531:\t\t\t\treturn rec;\n 532:\t\t}\n 533:\t\treturn NULL;\n 534:\t}\n 535:\t\n 536:\tstatic int cmp_u32(const void *a, const void *b)\n 537:\t{\n 538:\t\tu32 x = *(const u32 *)a, y = *(const u32 *)b;\n 539:\t\n 540:\t\treturn x \u003c y ? -1 : x \u003e y;\n 541:\t}\n 542:\t\n 543:\tint bpf_cleanup_alloc_info(struct bpf_prog_aux *aux)\n 544:\t{\n 545:\t\tif (aux-\u003eexc)\n 546:\t\t\treturn 0;\n 547:\t\taux-\u003eexc = kzalloc_obj(struct bpf_exception_info, GFP_KERNEL_ACCOUNT | __GFP_NOWARN);\n 548:\t\treturn aux-\u003eexc ? 0 : -ENOMEM;\n 549:\t}\n 550:\t\n 551:\tint bpf_cleanup_attach_info(struct bpf_prog_aux *aux, struct bpf_cleanup_info *recs, u32 cnt)\n 552:\t{\n 553:\t\tstruct bpf_exception_info *exc = aux-\u003eexc;\n 554:\t\tstruct bpf_cleanup_range *ranges;\n 555:\t\tu32 i, n_at, *at;\n 556:\t\n 557:\t\tif (!cnt) {\n 558:\t\t\tkvfree(recs);\n 559:\t\t\treturn 0;\n 560:\t\t}\n 561:\t\n 562:\t\tranges = kvcalloc(cnt, sizeof(*ranges), GFP_KERNEL_ACCOUNT | __GFP_NOWARN);\n 563:\t\tif (!ranges) {\n 564:\t\t\tkvfree(recs);\n 565:\t\t\treturn -ENOMEM;\n 566:\t\t}\n 567:\t\n 568:\t\t/* The pads on their own, sorted and deduplicated. */\n 569:\t\tat = kvmalloc_array(cnt, sizeof(*at), GFP_KERNEL_ACCOUNT | __GFP_NOWARN);\n 570:\t\tif (!at) {\n 571:\t\t\tkvfree(ranges);\n 572:\t\t\tkvfree(recs);\n 573:\t\t\treturn -ENOMEM;\n 574:\t\t}\n 575:\t\tfor (i = 0; i \u003c cnt; i++)\n 576:\t\t\tat[i] = recs[i].landing_pad_off;\n 577:\t\tsort(at, cnt, sizeof(*at), cmp_u32, NULL);\n 578:\t\tfor (i = 0, n_at = 0; i \u003c cnt; i++)\n 579:\t\t\tif (!n_at || at[n_at - 1] != at[i])\n 580:\t\t\t\tat[n_at++] = at[i];\n 581:\t\n 582:\t\texc-\u003epad_at = at;\n 583:\t\texc-\u003enr_pad_at = n_at;\n 584:\t\texc-\u003einfo = recs;\n 585:\t\texc-\u003enr_info = cnt;\n 586:\t\texc-\u003eranges = ranges;\n 587:\t\t/* Withheld until the JIT has filled the table in. */\n 588:\t\texc-\u003enr_ranges = 0;\n 589:\t\treturn 0;\n 590:\t}\n 591:\t\n 592:\tvoid bpf_cleanup_fill_native_ranges(struct bpf_prog *prog, u32 *addrs, void *image)\n 593:\t{\n 594:\t\tstruct bpf_exception_info *exc = prog-\u003eaux-\u003eexc;\n 595:\t\tu32 i, n;\n 596:\t\n 597:\t\tif (!exc || !exc-\u003enr_info || !exc-\u003eranges)\n 598:\t\t\treturn;\n 599:\t\n 600:\t\tn = exc-\u003enr_info;\n"
} |
| 37/3 | 2026/09/22 09:11 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 105580 output: 36 thoughts: 27 Thoughts: Okay, let's dive into the remaining lines of `kernel/bpf/exception.c`, specifically from line 601 to 664. I'm keen to see what these lines are addressing, as they're likely handling specific edge cases or optimizations related to BPF exceptions. Given the context of the file, I'm expecting to find either exception handling routines, specific error code interpretations, or potentially, code related to the interaction between the BPF interpreter and the kernel's exception handling mechanisms. It could also involve debugging information or logging related to exceptions. I'll be looking for patterns, function calls, and data structures that shed light on how the BPF subsystem responds to and recovers from unexpected events during bytecode execution. Essentially, I'm trying to gain a deeper understanding of the robustness and fault-tolerance of this critical component. |
| 38/3 | 2026/09/22 09:11 | tool | read-file |
0mArgs:{
"File": "kernel/bpf/exception.c",
"FirstLine": 601,
"LineCount": 70
}Results: {
"Contents": " 601:\t\tfor (i = 0; i \u003c n; i++) {\n 602:\t\t\tconst struct bpf_cleanup_info *rec = \u0026exc-\u003einfo[i];\n 603:\t\n 604:\t\t\tif (WARN_ON_ONCE(rec-\u003ebegin_off \u003e= prog-\u003elen ||\n 605:\t\t\t\t\t rec-\u003eend_off \u003e prog-\u003elen ||\n 606:\t\t\t\t\t rec-\u003elanding_pad_off \u003e= prog-\u003elen))\n 607:\t\t\t\treturn;\n 608:\t\t\texc-\u003eranges[i].begin = (u64)(long)image + addrs[rec-\u003ebegin_off];\n 609:\t\t\texc-\u003eranges[i].end = (u64)(long)image + addrs[rec-\u003eend_off];\n 610:\t\t\texc-\u003eranges[i].pad = (u64)(long)image + addrs[rec-\u003elanding_pad_off];\n 611:\t\t}\n 612:\t\texc-\u003enr_ranges = n;\n 613:\t}\n 614:\t\n 615:\tvoid bpf_cleanup_free_info(struct bpf_prog_aux *aux)\n 616:\t{\n 617:\t\tstruct bpf_exception_info *exc = aux-\u003eexc;\n 618:\t\n 619:\t\tif (!exc)\n 620:\t\t\treturn;\n 621:\t\tkvfree(exc-\u003eranges);\n 622:\t\tkvfree(exc-\u003einfo);\n 623:\t\tkvfree(exc-\u003epad_at);\n 624:\t\tkvfree(exc-\u003ethrow_at);\n 625:\t\tkvfree(exc-\u003eresume_at);\n 626:\t\tbitmap_free(exc-\u003epad_body);\n 627:\t\tkfree(exc);\n 628:\t\taux-\u003eexc = NULL;\n 629:\t}\n 630:\t\n 631:\t/* Is @idx in the sorted array @at of @n instruction indices? */\n 632:\tstatic bool insn_idx_in(const u32 *at, u32 n, u32 idx)\n 633:\t{\n 634:\t\treturn bsearch(\u0026idx, at, n, sizeof(*at), cmp_u32);\n 635:\t}\n 636:\t\n 637:\tbool bpf_cleanup_insn_is_pad(const struct bpf_prog *prog, u32 idx)\n 638:\t{\n 639:\t\tconst struct bpf_exception_info *exc = prog-\u003eaux-\u003eexc;\n 640:\t\n 641:\t\treturn exc \u0026\u0026 insn_idx_in(exc-\u003epad_at, exc-\u003enr_pad_at, idx);\n 642:\t}\n 643:\t\n 644:\tbool bpf_cleanup_insn_is_throw(const struct bpf_prog *prog, u32 idx)\n 645:\t{\n 646:\t\tconst struct bpf_exception_info *exc = prog-\u003eaux-\u003eexc;\n 647:\t\n 648:\t\treturn exc \u0026\u0026 insn_idx_in(exc-\u003ethrow_at, exc-\u003enr_throw_at, idx);\n 649:\t}\n 650:\t\n 651:\tbool bpf_cleanup_insn_is_resume(const struct bpf_prog *prog, u32 idx)\n 652:\t{\n 653:\t\tconst struct bpf_exception_info *exc = prog-\u003eaux-\u003eexc;\n 654:\t\n 655:\t\treturn exc \u0026\u0026 insn_idx_in(exc-\u003eresume_at, exc-\u003enr_resume_at, idx);\n 656:\t}\n 657:\t\n 658:\tbool bpf_cleanup_insn_in_pad(const struct bpf_prog *prog, u32 idx)\n 659:\t{\n 660:\t\tconst struct bpf_exception_info *exc = prog-\u003eaux-\u003eexc;\n 661:\t\n 662:\t\treturn exc \u0026\u0026 exc-\u003epad_body \u0026\u0026 idx \u003c exc-\u003enr_pad_body \u0026\u0026\n 663:\t\t test_bit(idx, exc-\u003epad_body);\n 664:\t}\n"
} |
| 39/3 | 2026/09/22 09:11 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 106910 output: 37 thoughts: 1275 Thoughts: **Analyzing BPF Cleanup Mechanisms** Okay, let's dive into this. My initial focus is on the changes within `kernel/bpf/fixups.c`. I need to understand how the new cleanup mechanisms are integrated within the BPF verifier. First, I examine the `adjust_insn_aux_data` function. It seems the core addition here is related to handling `cleanup_pad` information during instruction expansion. I note that data is being propagated and potentially modified based on whether the instructions are identical to original instructions or if it's a memory access instruction. Specifically, the code appears to be tracking instruction offsets and sizes related to cleanup regions, ensuring that the `cleanup_pad` offsets are adjusted correctly when instructions are inserted or removed. The loop over `env->cleanup_info_cnt` indicates this is a conditional feature and not core behavior. Next, I look at the modifications in `verifier_remove_insns`. Here, the code adjusts `cleanup_pad` offsets when instructions are removed. The handling of `falls_through` is intriguing; it implies there is a need to identify cases where execution falls through to the next instruction after the removal. Now, I shift my attention to `cleanup_kfunc_sites_for_subprog`. I observe that it allocates an array (`at`) which seems to be a list of instruction indices. And this populates that array. Then I look at `cleanup_pad_body_for_subprog`. This uses a bitmap (`bits`) to represent the cleanup pad. It is setting bits using `__set_bit`. Finally, the `cleanup_info_for_subprog` function allocates an array of `recs`. I can see it's populating `begin_off`, `end_off`, and `landing_pad_off`, and then calls `bpf_cleanup_attach_info`. This strongly suggests the creation of a cleanup information structure. Each struct is fully populated, so nothing is uninitialized. The call to `bpf_cleanup_attach_info` implies that it's attaching this information to the BPF program's auxiliary data. Now, I shift my focus to `kernel/bpf/helpers.c`. Here, `bpf_run_cleanup_pad` is defined. This looks like the runtime execution function for the cleanup pad. It retrieves `struct bpf_exception_info` from the program's auxiliary data, checks for an exception range, and, if found, calls `arch_bpf_run_cleanup_pad`. This function seems to be taking `fp` and `ip` in consideration, that is: frame pointer and instruction pointer, and I need to consider `callee` too. It's using `bpf_cleanup_pad_for_ip` to identify the correct cleanup range. The spill base calculation is interesting; it determines where spilled registers are located based on the context, especially the callee and exception information. The `bpf_stack_walker` function also calls `bpf_run_cleanup_pad`, but what is the meaning of `ctx`? I'll need to look at the surrounding code to determine how `ctx` is initialized and used. I see that it saves `prog` and `bp` to `ctx` itself. It appears to be carrying information about the current call stack context during a BPF stack walk, so I need to check how it's being used. |
| 40/3 | 2026/09/22 09:12 | tool | read-file |
0mArgs:{
"File": "kernel/bpf/helpers.c",
"FirstLine": 3415,
"LineCount": 50
}Results: {
"Contents": "3415:\t\trec = bpf_cleanup_pad_for_ip(prog, ip);\n3416:\t\tif (!rec)\n3417:\t\t\treturn;\n3418:\t\n3419:\t\t/*\n3420:\t\t * The callee is always another subprogram of this program -- the walk\n3421:\t\t * ends at any frame that is not one -- so its prologue spilled these\n3422:\t\t * registers and its exc is there to say where.\n3423:\t\t */\n3424:\t\tif (ctx-\u003ecallee)\n3425:\t\t\tspill_base = ctx-\u003ecallee_fp + ctx-\u003ecallee-\u003eaux-\u003eexc-\u003espill_off;\n3426:\t\telse\n3427:\t\t\tspill_base = fp + exc-\u003ethrow_spill_off;\n3428:\t\n3429:\t\tarch_bpf_run_cleanup_pad(rec-\u003epad, fp, spill_base);\n3430:\t}\n3431:\t\n3432:\tstatic bool bpf_stack_walker(void *cookie, u64 ip, u64 sp, u64 bp)\n3433:\t{\n3434:\t\tstruct bpf_throw_ctx *ctx = cookie;\n3435:\t\tstruct bpf_prog *prog;\n3436:\t\n3437:\t\t/*\n3438:\t\t * The RCU read lock is held to safely traverse the latch tree, but we\n3439:\t\t * don't need its protection when accessing the prog, since it has an\n3440:\t\t * active stack frame on the current stack trace, and won't disappear.\n3441:\t\t */\n3442:\t\trcu_read_lock();\n3443:\t\tprog = bpf_prog_ksym_find(ip);\n3444:\t\trcu_read_unlock();\n3445:\t\tif (!prog)\n3446:\t\t\treturn !ctx-\u003ecnt;\n3447:\t\tctx-\u003ecnt++;\n3448:\t\n3449:\t\tbpf_run_cleanup_pad(ctx, prog, ip, bp);\n3450:\t\tctx-\u003ecallee = prog;\n3451:\t\tctx-\u003ecallee_fp = bp;\n3452:\t\n3453:\t\tif (bpf_is_subprog(prog))\n3454:\t\t\treturn true;\n3455:\t\tctx-\u003eaux = prog-\u003eaux;\n3456:\t\tctx-\u003esp = sp;\n3457:\t\tctx-\u003ebp = bp;\n3458:\t\treturn false;\n3459:\t}\n3460:\t\n3461:\t__bpf_kfunc void bpf_throw(u64 cookie)\n3462:\t{\n3463:\t\tstruct bpf_throw_ctx ctx = {};\n3464:\t\n"
} |
| 41/3 | 2026/09/22 09:12 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 109696 output: 39 thoughts: 731 Thoughts: **BPF Verifier Deep Dive**
Okay, so I'm diving into this BPF verifier code and immediately, I'm noting the `struct bpf_throw_ctx ctx = {};` initialization. It's zeroed out on the stack, which means `ctx.callee` and `ctx.callee_fp` start as NULL/0. That's crucial. I'm thinking through the implications: When `ctx.callee` is NULL, `spill_base` is calculated relative to the frame pointer (`fp`), using the `exc->throw_spill_off` offset. But when `ctx.callee` *isn't* NULL, then `spill_base` depends on `ctx->callee_fp` and the callee's exception spill offset. And remembering that this struct starts out zeroed is key.
Now, shifting gears, I'm checking the `kernel/bpf/liveness.c` code. Specifically, the `bpf_insn_successors` function, it's adding a pad to successors when `cleanup_pad_of_call` is >= 0, and the `bpf_stack_slot_alive` function is checking if stack slots are alive *before* any padding. That's important for understanding stack behavior around cleanup.
Then, moving to `kernel/bpf/states.c`, the `states_equal` function has caught my eye. It's comparing states and making sure that if `unwinding` flags are different, then they are not equal, but if unwinding, it also checks the frame number. That has me thinking about how state changes are detected.
Okay, back to the verifier itself in `kernel/bpf/verifier.c`. I'm seeing a handful of interesting changes:
1. `bpf_copy_verifier_state` is now copying the `unwinding` and `unwind_frameno` fields, along with speculative and in_sleepable state. This suggests they are becoming critical parts of the state.
2. `check_max_stack_depth` got an addition where if `env->cleanup_info_cnt` isn't zero, it sets `priv_stack_mode` to `NO_PRIV_STACK`. I wonder what that means about stack depth with cleanup.
3. `push_callback_call` has a call to `bpf_cleanup_check_callback`, which is new and might be key to dealing with cleanup handlers and callbacks.
4. I see that the handling of `bpf_is_throw_kfunc` has changed in `check_func_call` and `check_kfunc_call`. It now calls `unwind_step` instead of the old `process_bpf_exit_full`.
5. I've got to carefully read `unwind_pop_frame`, `unwind_enter_pad`, `unwind_finish`, `unwind_step`, and `process_cleanup_resume`. The new code around unwinding is complex, and those are going to be key. I'll read from lines 18485 to 18600 of `kernel/bpf/verifier.c` in order to see how this all connects.
|
| 42/3 | 2026/09/22 09:12 | tool | read-file |
0mArgs:{
"File": "kernel/bpf/verifier.c",
"FirstLine": 18485,
"LineCount": 100
}Results: {
"Contents": "18485:\t\tPROCESS_BPF_EXIT = 1,\n18486:\t\tINSN_IDX_UPDATED = 2,\n18487:\t};\n18488:\t\n18489:\tstatic u32 unwind_pop_frame(struct bpf_verifier_env *env)\n18490:\t{\n18491:\t\tstruct bpf_verifier_state *state = env-\u003ecur_state;\n18492:\t\tstruct bpf_func_state *callee = state-\u003eframe[state-\u003ecurframe];\n18493:\t\tu32 callsite = callee-\u003ecallsite;\n18494:\t\tstruct bpf_func_state *caller;\n18495:\t\n18496:\t\tcaller = state-\u003eframe[state-\u003ecurframe - 1];\n18497:\t\taccount_processed_insns(env, callee, caller);\n18498:\t\tfree_func_state(callee);\n18499:\t\tstate-\u003eframe[state-\u003ecurframe--] = NULL;\n18500:\t\tinvalidate_outgoing_stack_args(env, caller);\n18501:\t\treturn callsite;\n18502:\t}\n18503:\t\n18504:\tstatic void unwind_enter_pad(struct bpf_verifier_env *env)\n18505:\t{\n18506:\t\tstruct bpf_verifier_state *state = env-\u003ecur_state;\n18507:\t\tstruct bpf_func_state *frame = cur_func(env);\n18508:\t\n18509:\t\tstate-\u003eunwind_frameno = state-\u003ecurframe;\n18510:\t\tclear_caller_saved_regs(env, frame-\u003eregs);\n18511:\t\tmark_reg_unknown(env, frame-\u003eregs, BPF_REG_0);\n18512:\t\t__mark_reg_known(\u0026frame-\u003eregs[BPF_REG_0], BPF_PAD_ENTRY_R0);\n18513:\t}\n18514:\t\n18515:\tstatic int unwind_finish(struct bpf_verifier_env *env)\n18516:\t{\n18517:\t\tint err = check_resource_leak(env, true, true, \"bpf_throw\");\n18518:\t\n18519:\t\tif (err)\n18520:\t\t\treturn err;\n18521:\t\treturn PROCESS_BPF_EXIT;\n18522:\t}\n18523:\t\n18524:\tstatic int unwind_step(struct bpf_verifier_env *env, u32 callsite, int *insn_idx)\n18525:\t{\n18526:\t\tstruct bpf_verifier_state *state = env-\u003ecur_state;\n18527:\t\tu8 popped = 0;\n18528:\t\n18529:\t\tstate-\u003eunwinding = true;\n18530:\t\tfor (;;) {\n18531:\t\t\tint pad = bpf_cleanup_pad_of_call(env, callsite);\n18532:\t\n18533:\t\t\tif (pad \u003e= 0) {\n18534:\t\t\t\tunwind_enter_pad(env);\n18535:\t\t\t\t/*\n18536:\t\t\t\t * The edge from @callsite to the pad crosses @popped\n18537:\t\t\t\t * frames, and nothing in the instruction stream says\n18538:\t\t\t\t * so. Record it for mark_chain_precision(), which has\n18539:\t\t\t\t * to walk back through the same frames.\n18540:\t\t\t\t */\n18541:\t\t\t\tenv-\u003eunwind_frames = popped;\n18542:\t\t\t\t*insn_idx = pad;\n18543:\t\t\t\treturn INSN_IDX_UPDATED;\n18544:\t\t\t}\n18545:\t\t\tif (!state-\u003ecurframe)\n18546:\t\t\t\treturn unwind_finish(env);\n18547:\t\t\tcallsite = unwind_pop_frame(env);\n18548:\t\t\tpopped++;\n18549:\t\t}\n18550:\t}\n18551:\t\n18552:\tstatic int process_cleanup_resume(struct bpf_verifier_env *env, int *insn_idx)\n18553:\t{\n18554:\t\tstruct bpf_verifier_state *state = env-\u003ecur_state;\n18555:\t\tint err;\n18556:\t\n18557:\t\t/* A pad entered by ordinary control flow. */\n18558:\t\tif (!state-\u003eunwinding) {\n18559:\t\t\tverbose(env,\n18560:\t\t\t\t\"bpf_unwind_resume() at insn %d reached without an exception in flight\\n\",\n18561:\t\t\t\t*insn_idx);\n18562:\t\t\treturn -EINVAL;\n18563:\t\t}\n18564:\t\t/*\n18565:\t\t * A resume in a subprogram the pad called, which has a landing pad of\n18566:\t\t * its own and reached it by ordinary control flow. A JIT lowers a\n18567:\t\t * resume as the way back out of a pad, which reaches the walker only\n18568:\t\t * from the frame whose pad it called.\n18569:\t\t */\n18570:\t\tif (state-\u003ecurframe != state-\u003eunwind_frameno) {\n18571:\t\t\tverbose(env,\n18572:\t\t\t\t\"bpf_unwind_resume() at insn %d is in frame %d, not frame %d whose landing pad the exception entered\\n\",\n18573:\t\t\t\t*insn_idx, state-\u003ecurframe, state-\u003eunwind_frameno);\n18574:\t\t\treturn -EINVAL;\n18575:\t\t}\n18576:\t\tif (!state-\u003ecurframe)\n18577:\t\t\treturn unwind_finish(env);\n18578:\t\terr = unwind_step(env, unwind_pop_frame(env), insn_idx);\n18579:\t\t/* unwind_step() counted the frames it popped, not this one. */\n18580:\t\tif (err == INSN_IDX_UPDATED)\n18581:\t\t\tenv-\u003eunwind_frames++;\n18582:\t\treturn err;\n18583:\t}\n18584:\t\n"
} |
| 43/3 | 2026/09/22 09:12 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 112853 output: 23 thoughts: 1124 Thoughts: **Unraveling the Unwind Entry Pad and KMSAN Concerns**
Okay, let's dissect this `unwind_enter_pad` code. I see it's setting the `unwind_frameno`, clearing caller-saved registers, marking `BPF_REG_0` as known with `BPF_PAD_ENTRY_R0`, and crucially, retaining the state of callee-saved registers (r6-r9) and the frame pointer (r10) from the original frame. The verifier's in place to prevent any nasty reads from uninitialized registers or leaking resources within the pad. Good.
But wait, this calls for a deeper look. I need to rigorously examine if this introduction has any potential to expose uninitialized memory. Let's apply the KMSAN criteria.
My mind immediately runs through the four `NeedsKMSAN=true` criteria:
1. **Kernel structures sent to user space:** I need to find all instances where data is copied from kernel space to user space, paying special attention to potential uninitialized padding. The code in question includes `check_cleanup_info` which contains the following `copy_to_bpfptr_offset`: I see the code is copying `min_size` which is a 4-byte `u32`. It is initialized. No issues there. I don't see any other `copy_to_user` calls within this patch. That's a relief.
2. **Conditional logic based on uninitialized variables:** I must go through every newly introduced data structure and its initialization process. Let's see...
* `struct bpf_exception_info`: Initialized with `kzalloc_obj`, so that's good - zeroed allocation.
* `struct bpf_cleanup_range *ranges`: Allocated with `kvcalloc`, also a zeroed allocation.
* `exc->pad_at`: Allocated with `kvmalloc_array`, each element initialized with data.
* `exc->throw_at` and `exc->resume_at`: Allocated and fully populated.
* `exc->pad_body`: Allocated with `bitmap_zalloc`, which is zeroed.
* `krecord` in `check_cleanup_info`: Allocated with `kvcalloc`, so zeroed.
* `struct bpf_jmp_history_entry`: `unwind_frames` is explicitly initialized or assigned later.
* `struct bpf_verifier_state`: `unwinding` and `unwind_frameno` are copied. My attention is drawn to the state initialization, I will grep the code to be sure the verifier state is either allocated with `kzalloc` or initialized.
So, it seems that all the added structs and fields are either zeroed during allocation or have their values explicitly set. That's a good sign.
3. **Allocation of complex data structures with uninitialized fields:** I have just checked, so that looks like it's covered.
4. **Bounds checks exposing uninitialized bytes:** I've not come across any worrisome boundary conditions that could lead to reading uninitialized memory.
Overall, it looks good from a KMSAN perspective. But I need to be 100% sure the verifier's state initialization is solid and no gotchas are hidden in the code. I must grep for `kzalloc` or similar methods used for allocating `bpf_verifier_state` to be sure. It looks like this unwind code is relatively safe.
|
| 44/3 | 2026/09/22 09:12 | tool | grepper |
0mArgs:{
"Expression": "kzalloc.*bpf_verifier_state"
}Results: {
"Output": "kernel/bpf/states.c=1242=int bpf_is_state_visited(struct bpf_verifier_env *env, int insn_idx)\n--\nkernel/bpf/states.c-1562-\t */\nkernel/bpf/states.c:1563:\tnew_sl = kzalloc_obj(struct bpf_verifier_state_list, GFP_KERNEL_ACCOUNT);\nkernel/bpf/states.c-1564-\tif (!new_sl)\n--\nkernel/bpf/verifier.c=19930=static int do_check_common(struct bpf_verifier_env *env, int subprog)\n--\nkernel/bpf/verifier.c-19942-\nkernel/bpf/verifier.c:19943:\tstate = kzalloc_obj(struct bpf_verifier_state, GFP_KERNEL_ACCOUNT);\nkernel/bpf/verifier.c-19944-\tif (!state)\n"
} |
| 45/3 | 2026/09/22 09:12 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 115015 output: 22 thoughts: 126 Thoughts: **My Initial Thoughts on BPF Verifier State Initialization** Okay, so `kzalloc_obj` is called on the `bpf_verifier_state`, right? That means the entire state is zeroed out. Good, that makes sense. `unwinding` and `unwind_frameno` are set to zero from the get-go. No surprises there. But wait a second... what about `insn_aux_data`? I know that's part of the state, and I've seen some interesting behavior around that field. I need to be sure I understand its initialization. I've got to consider what happens with these things: `cleanup_throw_site`, `cleanup_resume_site`, `in_cleanup_pad`, and `cleanup_pad`. Right, `insn_aux_data` isn't a simple `kzalloc`. I recall it being allocated separately using `vzalloc` or `kvcalloc` *within* the verifier itself. So, I need to trace back through that allocation path within the verifier. I'll need to figure out exactly how the `env->insn_aux_data` pointer is set up and when. I have to go deeper than just the state zeroing, and focus on the allocation strategy for `insn_aux_data`. |
| 46/3 | 2026/09/22 09:12 | tool | grepper |
0mArgs:{
"Expression": "insn_aux_data = "
}Results: {
"Output": "kernel/bpf/fixups.c=350=struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 off,\n--\nkernel/bpf/fixups.c-364-\nkernel/bpf/fixups.c:365:\t\tenv-\u003einsn_aux_data = new_data;\nkernel/bpf/fixups.c-366-\t}\n"
} |
| 47/3 | 2026/09/22 09:12 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 115565 output: 34 thoughts: 17 Thoughts: Here's my summary, as though I'm thinking through this process: **Tracing the Allocation of `insn_aux_data`** Okay, so I need to figure out where `insn_aux_data` is being allocated in the first place. This is crucial for understanding its lifecycle and potential memory issues. I need to trace this back to the source. Let's start by looking for the initial allocation, the point of instantiation. Is it dynamically allocated? Statically? This is fundamental, and will inform every subsequent investigation into this data structure's use. Hopefully there are clear and concise references to follow. Let's get to work. |
| 48/3 | 2026/09/22 09:12 | tool | grepper |
0mArgs:{
"Expression": "insn_aux_data",
"PathPrefix": "kernel/bpf/verifier.c"
}Results: {
"Output": "kernel/bpf/verifier.c=224=static void scalar_min_max_add(struct bpf_reg_state *dst_reg, struct bpf_reg_state *src_reg);\nkernel/bpf/verifier.c-225-\nkernel/bpf/verifier.c:226:static void bpf_map_ptr_store(struct bpf_insn_aux_data *aux,\nkernel/bpf/verifier.c-227-\t\t\t struct bpf_map *map,\n--\nkernel/bpf/verifier.c-235-\nkernel/bpf/verifier.c:236:static void bpf_map_key_store(struct bpf_insn_aux_data *aux, u64 state)\nkernel/bpf/verifier.c-237-{\n--\nkernel/bpf/verifier.c=3288=static void mark_indirect_target(struct bpf_verifier_env *env, int idx)\nkernel/bpf/verifier.c-3289-{\nkernel/bpf/verifier.c:3290:\tenv-\u003einsn_aux_data[idx].indirect_target = true;\nkernel/bpf/verifier.c-3291-}\n--\nkernel/bpf/verifier.c=3293=static void mark_non_stack_access(struct bpf_verifier_env *env, int idx)\nkernel/bpf/verifier.c-3294-{\nkernel/bpf/verifier.c:3295:\tenv-\u003einsn_aux_data[idx].non_stack_access = true;\nkernel/bpf/verifier.c-3296-}\n--\nkernel/bpf/verifier.c=3545=static void check_fastcall_stack_contract(struct bpf_verifier_env *env,\n--\nkernel/bpf/verifier.c-3548-\tstruct bpf_subprog_info *subprog = \u0026env-\u003esubprog_info[state-\u003esubprogno];\nkernel/bpf/verifier.c:3549:\tstruct bpf_insn_aux_data *aux = env-\u003einsn_aux_data;\nkernel/bpf/verifier.c-3550-\tint i;\n--\nkernel/bpf/verifier.c=3584=static int check_stack_write_fixed_off(struct bpf_verifier_env *env,\n--\nkernel/bpf/verifier.c-3633-\t\tif (sanitize)\nkernel/bpf/verifier.c:3634:\t\t\tenv-\u003einsn_aux_data[insn_idx].nospec_result = true;\nkernel/bpf/verifier.c-3635-\t}\n--\nkernel/bpf/verifier.c=4963=static int __check_ctx_access(struct bpf_verifier_env *env, int insn_idx, int off, int size,\n--\nkernel/bpf/verifier.c-4982-\t\t} else {\nkernel/bpf/verifier.c:4983:\t\t\tenv-\u003einsn_aux_data[insn_idx].ctx_field_size = info-\u003ectx_field_size;\nkernel/bpf/verifier.c-4984-\t\t}\n--\nkernel/bpf/verifier.c=5045=static int check_sock_access(struct bpf_verifier_env *env, int insn_idx,\n--\nkernel/bpf/verifier.c-5075-\tif (valid) {\nkernel/bpf/verifier.c:5076:\t\tenv-\u003einsn_aux_data[insn_idx].ctx_field_size =\nkernel/bpf/verifier.c-5077-\t\t\tinfo.ctx_field_size;\n--\nkernel/bpf/verifier.c=8278=static int get_bpf_res_spin_lock_kfunc_flags(const struct bpf_call_arg_meta *meta);\nkernel/bpf/verifier.c:8279:static struct bpf_insn_aux_data *cur_aux(const struct bpf_verifier_env *env);\nkernel/bpf/verifier.c-8280-static int process_irq_flag(struct bpf_verifier_env *env,\n--\nkernel/bpf/verifier.c=10746=static int set_map_elem_callback_state(struct bpf_verifier_env *env,\n--\nkernel/bpf/verifier.c-10750-{\nkernel/bpf/verifier.c:10751:\tstruct bpf_insn_aux_data *insn_aux = \u0026env-\u003einsn_aux_data[insn_idx];\nkernel/bpf/verifier.c-10752-\tstruct bpf_map *map;\n--\nkernel/bpf/verifier.c=11180=record_func_map(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta,\n--\nkernel/bpf/verifier.c-11182-{\nkernel/bpf/verifier.c:11183:\tstruct bpf_insn_aux_data *aux = \u0026env-\u003einsn_aux_data[insn_idx];\nkernel/bpf/verifier.c-11184-\tstruct bpf_map *map = meta-\u003emap.ptr;\n--\nkernel/bpf/verifier.c=11226=record_func_key(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta,\n--\nkernel/bpf/verifier.c-11228-{\nkernel/bpf/verifier.c:11229:\tstruct bpf_insn_aux_data *aux = \u0026env-\u003einsn_aux_data[insn_idx];\nkernel/bpf/verifier.c-11230-\tstruct bpf_reg_state *reg;\n--\nkernel/bpf/verifier.c=11370=static int check_get_func_ip(struct bpf_verifier_env *env)\n--\nkernel/bpf/verifier.c-11390-\nkernel/bpf/verifier.c:11391:static struct bpf_insn_aux_data *cur_aux(const struct bpf_verifier_env *env)\nkernel/bpf/verifier.c-11392-{\nkernel/bpf/verifier.c:11393:\treturn \u0026env-\u003einsn_aux_data[env-\u003einsn_idx];\nkernel/bpf/verifier.c-11394-}\n--\nkernel/bpf/verifier.c=11509=static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn,\n--\nkernel/bpf/verifier.c-11589-\tif (!in_sleepable_context(env))\nkernel/bpf/verifier.c:11590:\t\tenv-\u003einsn_aux_data[insn_idx].non_sleepable = true;\nkernel/bpf/verifier.c-11591-\n--\nkernel/bpf/verifier.c-11760-\t\t\treturns_cpu_specific_alloc_ptr = true;\nkernel/bpf/verifier.c:11761:\t\t\tenv-\u003einsn_aux_data[insn_idx].call_with_percpu_alloc_ptr = true;\nkernel/bpf/verifier.c-11762-\t\t}\n--\nkernel/bpf/verifier.c=13734=s64 bpf_helper_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn *insn,\n--\nkernel/bpf/verifier.c-13736-{\nkernel/bpf/verifier.c:13737:\tstruct bpf_insn_aux_data *aux = \u0026env-\u003einsn_aux_data[insn_idx];\nkernel/bpf/verifier.c-13738-\tconst struct bpf_func_proto *fn;\n--\nkernel/bpf/verifier.c=13852=s64 bpf_kfunc_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn *insn,\n--\nkernel/bpf/verifier.c-13854-{\nkernel/bpf/verifier.c:13855:\tstruct bpf_insn_aux_data *aux = \u0026env-\u003einsn_aux_data[insn_idx];\nkernel/bpf/verifier.c-13856-\tstruct bpf_call_arg_meta meta;\n--\nkernel/bpf/verifier.c=13927=static int check_special_kfunc(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta,\nkernel/bpf/verifier.c:13928:\t\t\t struct bpf_reg_state *regs, struct bpf_insn_aux_data *insn_aux,\nkernel/bpf/verifier.c-13929-\t\t\t const struct btf_type *ptr_type, struct btf *desc_btf)\n--\nkernel/bpf/verifier.c=14098=static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,\n--\nkernel/bpf/verifier.c-14106-\tstruct bpf_call_arg_meta meta;\nkernel/bpf/verifier.c:14107:\tstruct bpf_insn_aux_data *insn_aux;\nkernel/bpf/verifier.c-14108-\tconst char *operation;\n--\nkernel/bpf/verifier.c-14130-\tfunc_name = meta.func_name;\nkernel/bpf/verifier.c:14131:\tinsn_aux = \u0026env-\u003einsn_aux_data[insn_idx];\nkernel/bpf/verifier.c-14132-\n--\nkernel/bpf/verifier.c=14658=static bool can_skip_alu_sanitation(const struct bpf_verifier_env *env,\n--\nkernel/bpf/verifier.c-14665-\nkernel/bpf/verifier.c:14666:static int update_alu_sanitation_state(struct bpf_insn_aux_data *aux,\nkernel/bpf/verifier.c-14667-\t\t\t\t u32 alu_state, u32 alu_limit)\n--\nkernel/bpf/verifier.c=14683=static int sanitize_val_alu(struct bpf_verifier_env *env,\n--\nkernel/bpf/verifier.c-14685-{\nkernel/bpf/verifier.c:14686:\tstruct bpf_insn_aux_data *aux = cur_aux(env);\nkernel/bpf/verifier.c-14687-\n--\nkernel/bpf/verifier.c=14699=struct bpf_sanitize_info {\nkernel/bpf/verifier.c:14700:\tstruct bpf_insn_aux_data aux;\nkernel/bpf/verifier.c-14701-\tbool mask_to_left;\n--\nkernel/bpf/verifier.c=14724=static int sanitize_ptr_alu(struct bpf_verifier_env *env,\n--\nkernel/bpf/verifier.c-14731-{\nkernel/bpf/verifier.c:14732:\tstruct bpf_insn_aux_data *aux = commit_window ? cur_aux(env) : \u0026info-\u003eaux;\nkernel/bpf/verifier.c-14733-\tstruct bpf_verifier_state *vstate = env-\u003ecur_state;\n--\nkernel/bpf/verifier.c=14820=static void sanitize_mark_insn_seen(struct bpf_verifier_env *env)\n--\nkernel/bpf/verifier.c-14829-\tif (!vstate-\u003especulative)\nkernel/bpf/verifier.c:14830:\t\tenv-\u003einsn_aux_data[env-\u003einsn_idx].seen = env-\u003epass_cnt;\nkernel/bpf/verifier.c-14831-}\n--\nkernel/bpf/verifier.c=16096=static int adjust_reg_min_max_vals(struct bpf_verifier_env *env,\n--\nkernel/bpf/verifier.c-16114-\tif (dst_reg-\u003etype == PTR_TO_ARENA || (src_reg \u0026\u0026 src_reg-\u003etype == PTR_TO_ARENA)) {\nkernel/bpf/verifier.c:16115:\t\tstruct bpf_insn_aux_data *aux = cur_aux(env);\nkernel/bpf/verifier.c-16116-\n--\nkernel/bpf/verifier.c=17205=static void collect_linked_regs(struct bpf_verifier_env *env,\n--\nkernel/bpf/verifier.c-17209-{\nkernel/bpf/verifier.c:17210:\tstruct bpf_insn_aux_data *aux = env-\u003einsn_aux_data;\nkernel/bpf/verifier.c-17211-\tstruct bpf_func_state *func;\n--\nkernel/bpf/verifier.c=17548=static int check_ld_imm(struct bpf_verifier_env *env, struct bpf_insn *insn)\nkernel/bpf/verifier.c-17549-{\nkernel/bpf/verifier.c:17550:\tstruct bpf_insn_aux_data *aux = cur_aux(env);\nkernel/bpf/verifier.c-17551-\tstruct bpf_reg_state *regs = cur_regs(env);\n--\nkernel/bpf/verifier.c=18076=bool bpf_get_call_summary(struct bpf_verifier_env *env, struct bpf_insn *call,\n--\nkernel/bpf/verifier.c-18163- * - look for such patterns;\nkernel/bpf/verifier.c:18164: * - mark spill and fill instructions in env-\u003einsn_aux_data[*].fastcall_pattern;\nkernel/bpf/verifier.c:18165: * - mark set env-\u003einsn_aux_data[*].fastcall_spills_num for call instruction;\nkernel/bpf/verifier.c-18166- * - update env-\u003esubprog_info[*]-\u003efastcall_stack_off to find an offset\n--\nkernel/bpf/verifier.c=18194=static void mark_fastcall_pattern_for_call(struct bpf_verifier_env *env,\n--\nkernel/bpf/verifier.c-18255-\t\texpected_regs_mask \u0026= ~BIT(stx-\u003esrc_reg);\nkernel/bpf/verifier.c:18256:\t\tenv-\u003einsn_aux_data[insn_idx - i].fastcall_pattern = 1;\nkernel/bpf/verifier.c:18257:\t\tenv-\u003einsn_aux_data[insn_idx + i].fastcall_pattern = 1;\nkernel/bpf/verifier.c-18258-\t}\n--\nkernel/bpf/verifier.c-18278-\tif (cs.fastcall)\nkernel/bpf/verifier.c:18279:\t\tenv-\u003einsn_aux_data[insn_idx].fastcall_spills_num = i - 1;\nkernel/bpf/verifier.c-18280-\telse\n--\nkernel/bpf/verifier.c=18445=static int save_aux_ptr_type(struct bpf_verifier_env *env, enum bpf_reg_type type,\n--\nkernel/bpf/verifier.c-18447-{\nkernel/bpf/verifier.c:18448:\tenum bpf_reg_type *prev_type = \u0026env-\u003einsn_aux_data[env-\u003einsn_idx].ptr_type;\nkernel/bpf/verifier.c-18449-\n--\nkernel/bpf/verifier.c=18833=static int do_check(struct bpf_verifier_env *env)\n--\nkernel/bpf/verifier.c-18843-\t\tstruct bpf_insn *insn;\nkernel/bpf/verifier.c:18844:\t\tstruct bpf_insn_aux_data *insn_aux;\nkernel/bpf/verifier.c-18845-\t\tu8 unwind_frames;\n--\nkernel/bpf/verifier.c-18861-\t\tinsn = \u0026insns[env-\u003einsn_idx];\nkernel/bpf/verifier.c:18862:\t\tinsn_aux = \u0026env-\u003einsn_aux_data[env-\u003einsn_idx];\nkernel/bpf/verifier.c-18863-\n--\nkernel/bpf/verifier.c=19132=static int __check_pseudo_btf_id(struct bpf_verifier_env *env,\nkernel/bpf/verifier.c-19133-\t\t\t\t struct bpf_insn *insn,\nkernel/bpf/verifier.c:19134:\t\t\t\t struct bpf_insn_aux_data *aux,\nkernel/bpf/verifier.c-19135-\t\t\t\t struct btf *btf)\n--\nkernel/bpf/verifier.c=19215=static int check_pseudo_btf_id(struct bpf_verifier_env *env,\nkernel/bpf/verifier.c-19216-\t\t\t struct bpf_insn *insn,\nkernel/bpf/verifier.c:19217:\t\t\t struct bpf_insn_aux_data *aux)\nkernel/bpf/verifier.c-19218-{\n--\nkernel/bpf/verifier.c=19680=static int check_and_resolve_insns(struct bpf_verifier_env *env)\n--\nkernel/bpf/verifier.c-19700-\t\tif (insn[0].code == (BPF_LD | BPF_IMM | BPF_DW)) {\nkernel/bpf/verifier.c:19701:\t\t\tstruct bpf_insn_aux_data *aux;\nkernel/bpf/verifier.c-19702-\t\t\tstruct bpf_map *map;\n--\nkernel/bpf/verifier.c-19722-\t\t\tif (insn[0].src_reg == BPF_PSEUDO_BTF_ID) {\nkernel/bpf/verifier.c:19723:\t\t\t\taux = \u0026env-\u003einsn_aux_data[i];\nkernel/bpf/verifier.c-19724-\t\t\t\terr = check_pseudo_btf_id(env, insn, aux);\n--\nkernel/bpf/verifier.c-19730-\t\t\tif (insn[0].src_reg == BPF_PSEUDO_FUNC) {\nkernel/bpf/verifier.c:19731:\t\t\t\taux = \u0026env-\u003einsn_aux_data[i];\nkernel/bpf/verifier.c-19732-\t\t\t\taux-\u003eptr_type = PTR_TO_FUNC;\n--\nkernel/bpf/verifier.c-19770-\nkernel/bpf/verifier.c:19771:\t\t\taux = \u0026env-\u003einsn_aux_data[i];\nkernel/bpf/verifier.c-19772-\t\t\taux-\u003emap_index = map_idx;\n--\nkernel/bpf/verifier.c=19871=static void sanitize_dead_code(struct bpf_verifier_env *env)\nkernel/bpf/verifier.c-19872-{\nkernel/bpf/verifier.c:19873:\tstruct bpf_insn_aux_data *aux_data = env-\u003einsn_aux_data;\nkernel/bpf/verifier.c-19874-\tstruct bpf_insn trap = BPF_JMP_IMM(BPF_JA, 0, 0, -1);\n--\nkernel/bpf/verifier.c=21246=static int specialize_kfunc(struct bpf_verifier_env *env, struct bpf_kfunc_desc *desc, int insn_idx)\n--\nkernel/bpf/verifier.c-21281-\t} else if (func_id == special_kfunc_list[KF_bpf_dynptr_from_file]) {\nkernel/bpf/verifier.c:21282:\t\tif (!env-\u003einsn_aux_data[insn_idx].non_sleepable)\nkernel/bpf/verifier.c-21283-\t\t\taddr = (unsigned long)bpf_dynptr_from_file_sleepable;\nkernel/bpf/verifier.c-21284-\t} else if (func_id == special_kfunc_list[KF_bpf_arena_alloc_pages]) {\nkernel/bpf/verifier.c:21285:\t\tif (env-\u003einsn_aux_data[insn_idx].non_sleepable)\nkernel/bpf/verifier.c-21286-\t\t\taddr = (unsigned long)bpf_arena_alloc_pages_non_sleepable;\nkernel/bpf/verifier.c-21287-\t} else if (func_id == special_kfunc_list[KF_bpf_arena_free_pages]) {\nkernel/bpf/verifier.c:21288:\t\tif (env-\u003einsn_aux_data[insn_idx].non_sleepable)\nkernel/bpf/verifier.c-21289-\t\t\taddr = (unsigned long)bpf_arena_free_pages_non_sleepable;\n--\nkernel/bpf/verifier.c-21294-\nkernel/bpf/verifier.c:21295:static void __fixup_collection_insert_kfunc(struct bpf_insn_aux_data *insn_aux,\nkernel/bpf/verifier.c-21296-\t\t\t\t\t u16 struct_meta_reg,\n--\nkernel/bpf/verifier.c=21312=int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,\n--\nkernel/bpf/verifier.c-21343-\tif (is_bpf_obj_new_kfunc(desc-\u003efunc_id) || is_bpf_percpu_obj_new_kfunc(desc-\u003efunc_id)) {\nkernel/bpf/verifier.c:21344:\t\tstruct btf_struct_meta *kptr_struct_meta = env-\u003einsn_aux_data[insn_idx].kptr_struct_meta;\nkernel/bpf/verifier.c-21345-\t\tstruct bpf_insn addr[2] = { BPF_LD_IMM64(BPF_REG_2, (long)kptr_struct_meta) };\nkernel/bpf/verifier.c:21346:\t\tu64 obj_new_size = env-\u003einsn_aux_data[insn_idx].obj_new_size;\nkernel/bpf/verifier.c-21347-\n--\nkernel/bpf/verifier.c-21361-\t\t is_bpf_refcount_acquire_kfunc(desc-\u003efunc_id)) {\nkernel/bpf/verifier.c:21362:\t\tstruct btf_struct_meta *kptr_struct_meta = env-\u003einsn_aux_data[insn_idx].kptr_struct_meta;\nkernel/bpf/verifier.c-21363-\t\tstruct bpf_insn addr[2] = { BPF_LD_IMM64(BPF_REG_2, (long)kptr_struct_meta) };\n--\nkernel/bpf/verifier.c-21382-\t\t is_bpf_rbtree_add_kfunc(desc-\u003efunc_id)) {\nkernel/bpf/verifier.c:21383:\t\tstruct btf_struct_meta *kptr_struct_meta = env-\u003einsn_aux_data[insn_idx].kptr_struct_meta;\nkernel/bpf/verifier.c-21384-\t\tint struct_meta_reg = BPF_REG_3;\n--\nkernel/bpf/verifier.c-21401-\nkernel/bpf/verifier.c:21402:\t\t__fixup_collection_insert_kfunc(\u0026env-\u003einsn_aux_data[insn_idx], struct_meta_reg,\nkernel/bpf/verifier.c-21403-\t\t\t\t\t\tnode_offset_reg, insn, insn_buf, cnt);\n--\nkernel/bpf/verifier.c-21487-\nkernel/bpf/verifier.c:21488:\tif (env-\u003einsn_aux_data[insn_idx].arg_prog) {\nkernel/bpf/verifier.c:21489:\t\tu32 regno = env-\u003einsn_aux_data[insn_idx].arg_prog;\nkernel/bpf/verifier.c-21490-\t\tstruct bpf_insn ld_addrs[2] = { BPF_LD_IMM64(regno, (long)env-\u003eprog-\u003eaux) };\n--\nkernel/bpf/verifier.c=21658=int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,\n--\nkernel/bpf/verifier.c-21725-\nkernel/bpf/verifier.c:21726:\tlen = env-\u003einsn_aux_data_len = env-\u003eprog-\u003elen;\nkernel/bpf/verifier.c:21727:\tenv-\u003einsn_aux_data =\nkernel/bpf/verifier.c:21728:\t\t__vmalloc(array_size(sizeof(struct bpf_insn_aux_data), len),\nkernel/bpf/verifier.c-21729-\t\t\t GFP_KERNEL_ACCOUNT | __GFP_ZERO);\nkernel/bpf/verifier.c-21730-\tret = -ENOMEM;\nkernel/bpf/verifier.c:21731:\tif (!env-\u003einsn_aux_data)\nkernel/bpf/verifier.c-21732-\t\tgoto skip_full_check;\nkernel/bpf/verifier.c-21733-\tfor (i = 0; i \u003c len; i++)\nkernel/bpf/verifier.c:21734:\t\tenv-\u003einsn_aux_data[i].orig_idx = i;\nkernel/bpf/verifier.c-21735-\tenv-\u003esucc = bpf_iarray_realloc(NULL, 2);\n--\nkernel/bpf/verifier.c-21993-err_free_env:\nkernel/bpf/verifier.c:21994:\tif (env-\u003einsn_aux_data)\nkernel/bpf/verifier.c:21995:\t\tbpf_clear_insn_aux_data(env, 0, env-\u003einsn_aux_data_len);\nkernel/bpf/verifier.c:21996:\tvfree(env-\u003einsn_aux_data);\nkernel/bpf/verifier.c-21997-\tkvfree(env-\u003efd_array);\n"
} |
| 49/3 | 2026/09/22 09:12 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 122620 output: 34 thoughts: 74 Thoughts: Here's my take on those code snippets, thinking through the implications: **Zero-Allocation and Data Patching in BPF** Okay, so I'm seeing `env->insn_aux_data` is being allocated with `__vmalloc` and initialized to zero immediately using `__GFP_ZERO`. This is standard, but the key is that it's *zero-initialized*. This tells me the kernel is expecting a clean slate, a starting point of known state for this auxiliary data associated with the BPF instruction. This suggests the auxiliary data likely holds important contextual information that needs to be pristine at initialization. Now, moving onto `bpf_patch_insn_data`. My mental gears are already turning, wondering how this function handles its own data allocation. I need to dig in and see how the `new_data` is allocated here. Specifically, I want to understand: * Is it using `kmalloc` or `vmalloc`? * Is there an initial zeroing operation done? * Are there any memory barriers or synchronization concerns involved in the data transfer? It's crucial to understand this data patch process for several reasons: * **Security:** How we allocate and handle this data is a potential vector for vulnerability. Any improper handling could lead to data leaks or injection vulnerabilities. * **Performance:** Unnecessary copying or allocation overhead here could become a bottleneck for this patch operation, so I need to analyze its efficiency. * **Correctness:** If the allocation isn't handled correctly, the auxiliary data may be corrupted, and could lead to incorrect program behavior. |
| 50/3 | 2026/09/22 09:13 | tool | grepper |
0mArgs:{
"Expression": "new_data = ",
"PathPrefix": "kernel/bpf/fixups.c"
}Results: {
"Output": "kernel/bpf/fixups.c=350=struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 off,\n--\nkernel/bpf/fixups.c-353-\tstruct bpf_prog *new_prog;\nkernel/bpf/fixups.c:354:\tstruct bpf_insn_aux_data *new_data = NULL;\nkernel/bpf/fixups.c-355-\tstruct bpf_insn original_insn;\n--\nkernel/bpf/fixups.c-357-\tif (len \u003e 1) {\nkernel/bpf/fixups.c:358:\t\tnew_data = vrealloc(env-\u003einsn_aux_data,\nkernel/bpf/fixups.c-359-\t\t\t\t array_size(env-\u003eprog-\u003elen + len - 1,\n"
} |
| 51/3 | 2026/09/22 09:13 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 123306 output: 37 thoughts: 37 Thoughts: Here's my summary, as if I were thinking these thoughts: **Examining BPF Instruction Data Patching** Alright, let's dive in. I need to refresh my memory on how `bpf_patch_insn_data` actually works within the kernel. My initial thought is to zero in on `kernel/bpf/fixups.c`, specifically around line 355. That should be the sweet spot, give or take a few lines, where the magic happens. I figure by examining the code there, I'll quickly grasp the specifics of how instruction data is patched within the BPF framework. Understanding this is key to debugging [insert relevant debugging scenario/problem] and to ensuring [insert relevant positive outcome/goal]. Let's see what the source says! |
| 52/3 | 2026/09/22 09:13 | tool | read-file |
0mArgs:{
"File": "kernel/bpf/fixups.c",
"FirstLine": 355,
"LineCount": 40
}Results: {
"Contents": " 355:\t\tstruct bpf_insn original_insn;\n 356:\t\n 357:\t\tif (len \u003e 1) {\n 358:\t\t\tnew_data = vrealloc(env-\u003einsn_aux_data,\n 359:\t\t\t\t\t array_size(env-\u003eprog-\u003elen + len - 1,\n 360:\t\t\t\t\t\t sizeof(struct bpf_insn_aux_data)),\n 361:\t\t\t\t\t GFP_KERNEL_ACCOUNT | __GFP_ZERO);\n 362:\t\t\tif (!new_data)\n 363:\t\t\t\treturn NULL;\n 364:\t\n 365:\t\t\tenv-\u003einsn_aux_data = new_data;\n 366:\t\t}\n 367:\t\n 368:\t\tmemcpy(\u0026original_insn, env-\u003eprog-\u003einsnsi + off, sizeof(struct bpf_insn));\n 369:\t\tnew_prog = bpf_patch_insn_single(env-\u003eprog, off, patch, len);\n 370:\t\tif (IS_ERR(new_prog)) {\n 371:\t\t\tif (PTR_ERR(new_prog) == -ERANGE)\n 372:\t\t\t\tverbose(env,\n 373:\t\t\t\t\t\"insn %d cannot be patched due to 16-bit range\\n\",\n 374:\t\t\t\t\tenv-\u003einsn_aux_data[off].orig_idx);\n 375:\t\t\treturn NULL;\n 376:\t\t}\n 377:\t\tadjust_insn_aux_data(env, new_prog, off, len, \u0026original_insn);\n 378:\t\tadjust_subprog_starts(env, off, len);\n 379:\t\tadjust_insn_arrays(env, off, len);\n 380:\t\tadjust_poke_descs(new_prog, off, len);\n 381:\t\treturn new_prog;\n 382:\t}\n 383:\t\n 384:\t/*\n 385:\t * For all jmp insns in a given 'prog' that point to 'tgt_idx' insn adjust the\n 386:\t * jump offset by 'delta'.\n 387:\t */\n 388:\tstatic int adjust_jmp_off(struct bpf_prog *prog, u32 tgt_idx, u32 delta)\n 389:\t{\n 390:\t\tstruct bpf_insn *insn = prog-\u003einsnsi;\n 391:\t\tu32 insn_cnt = prog-\u003elen, i;\n 392:\t\ts32 imm;\n 393:\t\ts16 off;\n 394:\t\n"
} |
| 53/3 | 2026/09/22 09:13 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 124233 output: 29 thoughts: 2531 Thoughts: **BPF JIT Compilation and Memory Safety: An X86 Perspective** Alright, let's dive into how the X86 BPF JIT compiler handles memory management, especially in the context of cleanup and exception handling. My immediate thought is, hey, `vrealloc` and friends are indeed zero-allocated, which is a good baseline. Now, I'm digging into `arch/x86/net/bpf_jit_comp.c`. The first thing that jumps out is how the compiler accounts for spill space. There's a check for `bpf_cleanup_force_spill`, and that, along with the stack depth, determines the prologue depth. The prologue then gets emitted to set up the stack frame. Callee-saved registers are the next point of interest. The code carefully considers scenarios involving exception boundaries, forced spills, and potential arena starts. These conditions lead to adding extra space on the stack for those callee-saved registers (like `r12`, `rbx`, `r13`, `r14`, and `r15`). Next up is dealing with outgoing arguments. The compiler calculates `outgoing_arg_base` by considering stack depth, any spill space, and those callee-saved registers. We also see how `spill_off` and `throw_spill_off` are computed to offset for spilling, especially before a `throw`. The code then explicitly spills registers to pre-defined offsets on the stack *before* calling `bpf_throw()`. This is followed by emitting a `return` when a resume instruction is found. There's also an epilogue that pops the callee regs to clean up the stack. Then there's `arch_bpf_run_cleanup_pad`, the assembly code for the cleanup pad itself. We see that the stack frame is setup properly here, and register values are moved around, but it is clear from the code that these values are taken from rdx, not from anywhere else. Finally, a `CALL_NOSPEC` is used to jump to the cleanup pad function. My question is, does the BPF verifier or runtime ever read uninitialized stack or heap memory? Good! The verifier itself enforces that BPF programs cannot access uninitialized stack slots. Before a pad is entered, caller-saved registers are cleared, the return value is initialized, and registers that persist across calls are retained. The verifier validates instructions in landing pads statically. Now, concerning the kernel C code that handles memory allocation for exceptions and cleanup: The `cleanup_alloc` function uses `kvzalloc` to zero allocated memory, which takes care of a good portion of memory safety concerns. Structures like `cleanup_reachability`, `cleanup_mark_pad_bodies`, and `cleanup_check_resumes` all rely on zeroed memory. Specifically, the cleanup info for each subprogram, exception information, ranges, and related data are allocated using zeroed memory. Importantly, let's address any potential information leaks to userspace. Does the code ever expose uninitialized data? We need to verify the BPF syscall commands to make sure that `exc` or `cleanup_info` or any other related structures are not copied to userspace without proper initialization. |
| 54/3 | 2026/09/22 09:13 | tool | grepper |
0mArgs:{
"Expression": "exc",
"PathPrefix": "kernel/bpf/syscall.c"
}Results: {
"Output": "kernel/bpf/syscall.c=885=static void bpf_map_free(struct bpf_map *map)\n--\nkernel/bpf/syscall.c-893-\t */\nkernel/bpf/syscall.c:894:\tkfree(map-\u003eexcl_prog_sha);\nkernel/bpf/syscall.c-895-\tmigrate_disable();\n--\nkernel/bpf/syscall.c=1055=static ssize_t bpf_dummy_write(struct file *filp, const char __user *buf,\n--\nkernel/bpf/syscall.c-1063-\nkernel/bpf/syscall.c:1064:/* called for any extra memory-mapped regions (except initial) */\nkernel/bpf/syscall.c-1065-static void bpf_map_mmap_open(struct vm_area_struct *vma)\n--\nkernel/bpf/syscall.c=1258=static int map_check_btf(struct bpf_map *map, struct bpf_token *token,\n--\nkernel/bpf/syscall.c-1382-\nkernel/bpf/syscall.c:1383:#define BPF_MAP_CREATE_LAST_FIELD excl_prog_hash_size\nkernel/bpf/syscall.c-1384-/* called via syscall */\nkernel/bpf/syscall.c=1385=static int map_create_alloc(union bpf_attr *attr, bpfptr_t uattr, struct bpf_verifier_log *log,\n--\nkernel/bpf/syscall.c-1413-\t\tif (attr-\u003ebtf_key_type_id || attr-\u003ebtf_value_type_id) {\nkernel/bpf/syscall.c:1414:\t\t\tbpf_log(log, \"btf_vmlinux_value_type_id is mutually exclusive with btf_key_type_id and btf_value_type_id.\\n\");\nkernel/bpf/syscall.c-1415-\t\t\treturn -EINVAL;\n--\nkernel/bpf/syscall.c-1596-\nkernel/bpf/syscall.c:1597:\tif (attr-\u003eexcl_prog_hash) {\nkernel/bpf/syscall.c:1598:\t\tbpfptr_t uprog_hash = make_bpfptr(attr-\u003eexcl_prog_hash, uattr.is_kernel);\nkernel/bpf/syscall.c-1599-\nkernel/bpf/syscall.c:1600:\t\tif (attr-\u003eexcl_prog_hash_size != SHA256_DIGEST_SIZE) {\nkernel/bpf/syscall.c:1601:\t\t\tbpf_log(log, \"Invalid excl_prog_hash_size.\\n\");\nkernel/bpf/syscall.c-1602-\t\t\terr = -EINVAL;\n--\nkernel/bpf/syscall.c-1605-\nkernel/bpf/syscall.c:1606:\t\tmap-\u003eexcl_prog_sha = kzalloc(SHA256_DIGEST_SIZE, GFP_KERNEL);\nkernel/bpf/syscall.c:1607:\t\tif (!map-\u003eexcl_prog_sha) {\nkernel/bpf/syscall.c-1608-\t\t\terr = -ENOMEM;\n--\nkernel/bpf/syscall.c-1611-\nkernel/bpf/syscall.c:1612:\t\tif (copy_from_bpfptr(map-\u003eexcl_prog_sha, uprog_hash, SHA256_DIGEST_SIZE)) {\nkernel/bpf/syscall.c-1613-\t\t\terr = -EFAULT;\n--\nkernel/bpf/syscall.c-1615-\t\t}\nkernel/bpf/syscall.c:1616:\t} else if (attr-\u003eexcl_prog_hash_size) {\nkernel/bpf/syscall.c:1617:\t\tbpf_log(log, \"Invalid excl_prog_hash_size.\\n\");\nkernel/bpf/syscall.c-1618-\t\terr = -EINVAL;\n--\nkernel/bpf/syscall.c=3327=static void bpf_link_free(struct bpf_link *link)\n--\nkernel/bpf/syscall.c-3347-\t\t * don't need to chain SRCU grace period waits, however, for the\nkernel/bpf/syscall.c:3348:\t\t * faultable case, since it exclusively uses RCU Tasks Trace.\nkernel/bpf/syscall.c-3349-\t\t */\n"
} |
| 55/3 | 2026/09/22 09:13 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 128628 output: 54 thoughts: 1066 Thoughts: **Analyzing BPF Exception Handling and Register Spilling** Okay, so `cleanup_info` is just an input table, purely user-space, passed in at BPF program load, not something returned. Got it. But hold on, what about stack unwinding and those pesky spill areas when an exception is thrown? I need to understand this exception unwinder better. Let's dig into the code, specifically `kernel/bpf/helpers.c`. Aha, `bpf_run_cleanup_pad` is the key function here. It uses `bpf_cleanup_pad_for_ip` to find the relevant cleanup range. If an exception is encountered, we need to know where the registers are that need cleaning up. The interesting part is the `spill_base`. When a `bpf_throw` is called, it might be in the current frame, or a nested function frame. When `ctx->callee` is set, it means it's within a nested function call, and that `spill_base` will point to the `callee_fp` and an offset stored in the nested function's exception metadata. Otherwise it uses the `fp` in the current frame. Either way, it points to the stack location where the callee saved registers are spilled before the throw. Now, what is `arch_bpf_run_cleanup_pad` doing? That's the real question. Let's look at the assembly in `arch/x86/net/bpf_cleanup_pad.S`. Okay, it's essentially a series of `movq` instructions. It's reading values from the `spill_base` address – specifically, `0(%rdx)`, `8(%rdx)`, `16(%rdx)`, `24(%rdx)`, and `32(%rdx)`. Those are register spills! So, what registers are being spilled, and by whom? If `!ctx->callee`, this must be the frame that *called* `bpf_throw()`. Let's check the code that precedes the call to `bpf_throw()` in `arch/x86/net/bpf_jit_comp.c`. Yes, it looks like before `bpf_throw()` is called, there's a spill operation! It stores registers into locations on the stack based on `throw_spill_off`. It's spilling `r9`, `r8`, `r7`, `r6`, and `r12` using `emit_stx`, at offsets `0`, `8`, `16`, `24`, and `32` from `throw_spill_off` relative to the frame pointer. Those registers are being restored by `arch_bpf_run_cleanup_pad`. That's five qwords. And since the `rdx` register in the assembly is the `spill_base = fp + exc->throw_spill_off`, the offsets in `arch/x86/net/bpf_cleanup_pad.S` are working as expected. And in the case when `ctx->callee` is set? That means the callee saved registers are in another frame, so `spill_base = ctx->callee_fp + ctx->callee->aux->exc->spill_off`. So, where does the prologue spill these registers when a callee has been called? I need to examine how the prologue is generated in the JIT compiler, looking specifically for register pushes, let's search for `callee_saved_size` or pushes in `arch/x86/net/bpf_jit_comp.c`. Now let's trace how the callee registers are pushed in x86 JIT in `emit_prologue`. |
| 56/3 | 2026/09/22 09:13 | tool | grepper |
0mArgs:{
"Expression": "push_r12|pop_r12|push_callee_regs",
"PathPrefix": "arch/x86/net/bpf_jit_comp.c"
}Results: {
"Output": "arch/x86/net/bpf_jit_comp.c=374=static void pop_r9(u8 **pprog)\n--\narch/x86/net/bpf_jit_comp.c-381-\narch/x86/net/bpf_jit_comp.c:382:static void push_r12(u8 **pprog)\narch/x86/net/bpf_jit_comp.c-383-{\n--\narch/x86/net/bpf_jit_comp.c-389-\narch/x86/net/bpf_jit_comp.c:390:static void push_callee_regs(u8 **pprog, bool *callee_regs_used)\narch/x86/net/bpf_jit_comp.c-391-{\n--\narch/x86/net/bpf_jit_comp.c-404-\narch/x86/net/bpf_jit_comp.c:405:static void pop_r12(u8 **pprog)\narch/x86/net/bpf_jit_comp.c-406-{\n--\narch/x86/net/bpf_jit_comp.c=569=static void emit_prologue(u8 **pprog, u8 *ip, u32 stack_depth, bool ebpf_from_cbpf,\n--\narch/x86/net/bpf_jit_comp.c-602-\t\tpop_callee_regs(\u0026prog, all_callee_regs_used);\narch/x86/net/bpf_jit_comp.c:603:\t\tpop_r12(\u0026prog);\narch/x86/net/bpf_jit_comp.c-604-\t\t/* Reset the stack frame. */\n--\narch/x86/net/bpf_jit_comp.c=789=static void emit_bpf_tail_call_indirect(struct bpf_prog *bpf_prog,\n--\narch/x86/net/bpf_jit_comp.c-842-\t\tpop_callee_regs(\u0026prog, all_callee_regs_used);\narch/x86/net/bpf_jit_comp.c:843:\t\tpop_r12(\u0026prog);\narch/x86/net/bpf_jit_comp.c-844-\t} else {\n--\narch/x86/net/bpf_jit_comp.c-846-\t\tif (bpf_arena_get_kern_vm_start(bpf_prog-\u003eaux-\u003earena))\narch/x86/net/bpf_jit_comp.c:847:\t\t\tpop_r12(\u0026prog);\narch/x86/net/bpf_jit_comp.c-848-\t}\n--\narch/x86/net/bpf_jit_comp.c=877=static void emit_bpf_tail_call_direct(struct bpf_prog *bpf_prog,\n--\narch/x86/net/bpf_jit_comp.c-909-\t\tpop_callee_regs(\u0026prog, all_callee_regs_used);\narch/x86/net/bpf_jit_comp.c:910:\t\tpop_r12(\u0026prog);\narch/x86/net/bpf_jit_comp.c-911-\t} else {\n--\narch/x86/net/bpf_jit_comp.c-913-\t\tif (bpf_arena_get_kern_vm_start(bpf_prog-\u003eaux-\u003earena))\narch/x86/net/bpf_jit_comp.c:914:\t\t\tpop_r12(\u0026prog);\narch/x86/net/bpf_jit_comp.c-915-\t}\n--\narch/x86/net/bpf_jit_comp.c=1968=static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *addrs, u8 *image,\n--\narch/x86/net/bpf_jit_comp.c-2041-\t\t */\narch/x86/net/bpf_jit_comp.c:2042:\t\tpush_r12(\u0026prog);\narch/x86/net/bpf_jit_comp.c:2043:\t\tpush_callee_regs(\u0026prog, all_callee_regs_used);\narch/x86/net/bpf_jit_comp.c-2044-\t} else {\narch/x86/net/bpf_jit_comp.c-2045-\t\tif (arena_vm_start)\narch/x86/net/bpf_jit_comp.c:2046:\t\t\tpush_r12(\u0026prog);\narch/x86/net/bpf_jit_comp.c:2047:\t\tpush_callee_regs(\u0026prog, callee_regs_used);\narch/x86/net/bpf_jit_comp.c-2048-\t}\n--\narch/x86/net/bpf_jit_comp.c-3262-\t\t\t\tpop_callee_regs(\u0026prog, all_callee_regs_used);\narch/x86/net/bpf_jit_comp.c:3263:\t\t\t\tpop_r12(\u0026prog);\narch/x86/net/bpf_jit_comp.c-3264-\t\t\t} else {\n--\narch/x86/net/bpf_jit_comp.c-3266-\t\t\t\tif (arena_vm_start)\narch/x86/net/bpf_jit_comp.c:3267:\t\t\t\t\tpop_r12(\u0026prog);\narch/x86/net/bpf_jit_comp.c-3268-\t\t\t}\n"
} |
| 57/3 | 2026/09/22 09:13 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 132055 output: 44 thoughts: 116 Thoughts: **Analyzing x86 BPF JIT Compilation Register Usage** Okay, let's dive into this x86 BPF JIT compilation code. I need to understand exactly what's being pushed onto the stack at the start of a BPF program. Specifically, I'm interested in the register preservation. I see `push_r12` called followed by `push_callee_regs`. Right, so `push_r12` is straightforward enough, it's pushing register R12. But I need to fully grok what `push_callee_regs` does. Let's trace it and see which registers that function is preserving. I'll need to read the code in `arch/x86/net/bpf_jit_comp.c`, and I'll jump to the vicinity of line 390. This should clarify which callee-saved registers are being pushed, and what the overall stack frame setup looks like. |
| 58/3 | 2026/09/22 09:13 | tool | read-file |
0mArgs:{
"File": "arch/x86/net/bpf_jit_comp.c",
"FirstLine": 380,
"LineCount": 30
}Results: {
"Contents": " 380:\t}\n 381:\t\n 382:\tstatic void push_r12(u8 **pprog)\n 383:\t{\n 384:\t\tu8 *prog = *pprog;\n 385:\t\n 386:\t\tEMIT2(0x41, 0x54); /* push r12 */\n 387:\t\t*pprog = prog;\n 388:\t}\n 389:\t\n 390:\tstatic void push_callee_regs(u8 **pprog, bool *callee_regs_used)\n 391:\t{\n 392:\t\tu8 *prog = *pprog;\n 393:\t\n 394:\t\tif (callee_regs_used[0])\n 395:\t\t\tEMIT1(0x53); /* push rbx */\n 396:\t\tif (callee_regs_used[1])\n 397:\t\t\tEMIT2(0x41, 0x55); /* push r13 */\n 398:\t\tif (callee_regs_used[2])\n 399:\t\t\tEMIT2(0x41, 0x56); /* push r14 */\n 400:\t\tif (callee_regs_used[3])\n 401:\t\t\tEMIT2(0x41, 0x57); /* push r15 */\n 402:\t\t*pprog = prog;\n 403:\t}\n 404:\t\n 405:\tstatic void pop_r12(u8 **pprog)\n 406:\t{\n 407:\t\tu8 *prog = *pprog;\n 408:\t\n 409:\t\tEMIT2(0x41, 0x5C); /* pop r12 */\n"
} |
| 59/3 | 2026/09/22 09:13 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 132910 output: 584 thoughts: 1525 Thoughts: **KMSAN Assessment: No Need for Dedicated Fuzzing** Okay, let's break this down. First, I need to understand the implications of the `push_r12` and `push_callee_regs` sequence, which I've just mapped out in terms of stack layout. It confirms that the callee-saved registers (rbx, r13, r14, r15, r12) are *always* pushed when `all_callee_regs_used` is passed, due to either an exception boundary or a forced spill from `bpf_cleanup_force_spill()`. And the code initialization matches my expectations. Now, let's carefully evaluate this patch against the KMSAN requirements. I need to be absolutely certain if this warrants a `NeedsKMSAN=true` flag. I'm going to follow each point systematically. 1. **Kernel structures to userspace:** The only data passed to userspace is the `min_size` (a `u32`), which is fully initialized. No uninitialized data, no structure leaks. Definitely not an issue. 2. **Conditional logic and uninitialized variables:** No, there are no uninitialized variables that dictate conditional branching. All allocated structures, `bpf_exception_info`, `cleanup_alloc`, `krecord`, `ranges`, `bits`, are initialized with zeroing allocators or explicit initialization. The verifier state allocations are zeroed. This is safe. 3. **Complex data structures, uninitialized fields:** Again, no. All dynamically allocated buffers and structs, like `bpf_exception_info`, `bpf_cleanup_info`, `bpf_cleanup_range`, `insn_aux_data`, and DFS scratch arrays are *fully* initialized, either zeroed or written to before being used. 4. **Bounds checks and uninitialized bytes:** No, there's nothing that allows for accessing uninitialized parts of existing buffers due to these modifications. 5. Regarding generated JIT code, specifically, spill areas for callee-saved registers are fully written before being loaded back. The verifier enforces proper instruction sequences to prevent reading uninitialized values. This is a critical point that helps demonstrate safety. 6. Considering possible issues from this patch, they are most likely related to verifier bugs, control flow graph construction errors, CFI/BTI branch landing problems, or stack frame issues. These types of problems could lead to out-of-bounds stack access or corrupt control flow. I do not see any specific risks of uninitialized memory reads. Therefore, after a careful review, the risk profile points clearly away from needing KMSAN. These types of bugs are already covered by KASAN/KASAN-extra/stack guard pages, not by KMSAN. It's a logic or memory safety issue, and I'm confident KASAN will catch them. In short, there's no way this patch introduces or exposes uninitialized memory risks detectable *only* by KMSAN. All relevant structures are properly initialized, and the potential issues identified are in areas better covered by existing bug detectors. A KMSAN session is unnecessary and would be a waste of resources. Okay, let's call the `set-results` tool now and report this conclusion. |
| 60/3 | 2026/09/22 09:13 | tool | set-results |
0mArgs:{
"KMSANReasoning": "The patch series implements support for BPF cleanup landing pads and exception unwinding across x86-64 and arm64 architectures, adding verifier checks, CFG analysis, JIT spills/epilogues, and the bpf_unwind_resume kfunc.\n\nA dedicated KMSAN session is not required based on the following analysis:\n1. Heap and Stack Memory Initialization:\n - All newly allocated kernel data structures (such as `struct bpf_exception_info`, `struct bpf_cleanup_range`, `struct bpf_cleanup_info`, and DFS scratch arrays in `cleanup_ctx`) are allocated using zeroing allocators (`kzalloc_obj`, `kvcalloc`, `kvzalloc`, `bitmap_zalloc`).\n - Verifier state tracking structures (`struct bpf_verifier_state`, `struct bpf_insn_aux_data`) are zero-initialized via `kzalloc_obj` and `__vmalloc(..., __GFP_ZERO)`. Added fields (`unwinding`, `unwind_frameno`, `unwind_frames`) are explicitly initialized and reset.\n - The stack context used during unwinding (`struct bpf_throw_ctx ctx = {}`) is zero-initialized on the stack.\n\n2. Kernel-to-User Space Copies / Info Leaks:\n - The only user copy introduced is writing `min_size` (a 4-byte scalar `u32`) to user space via `copy_to_bpfptr_offset` on `-E2BIG` in `check_cleanup_info()`. It is fully initialized with `min_t(u32, krec_size, urec_size)`.\n - No other kernel structures, struct padding, or uninitialized buffers are copied to user space.\n\n3. JIT and Runtime Register Spills:\n - On x86-64, JIT prologue and throw-site register spills write all 5 callee-saved / context registers before `arch_bpf_run_cleanup_pad()` reads them.\n - The BPF verifier enforces that landing pad entry clears caller-saved registers (`clear_caller_saved_regs`), initializes `BPF_REG_0` as known, and prevents reading uninitialized registers or stack slots within landing pads.\n\nPotential failure modes in this patch involve CFG exploration, jump history backtracking, verifier prune points, or stack offset calculations—bugs that fall under control-flow logic, out-of-bounds access, or memory corruption detectable by KASAN and standard kernel assertions rather than KMSAN uninitialized memory detectors.",
"NeedsKMSAN": false
}Results: {
"KMSANReasoning": "The patch series implements support for BPF cleanup landing pads and exception unwinding across x86-64 and arm64 architectures, adding verifier checks, CFG analysis, JIT spills/epilogues, and the bpf_unwind_resume kfunc.\n\nA dedicated KMSAN session is not required based on the following analysis:\n1. Heap and Stack Memory Initialization:\n - All newly allocated kernel data structures (such as `struct bpf_exception_info`, `struct bpf_cleanup_range`, `struct bpf_cleanup_info`, and DFS scratch arrays in `cleanup_ctx`) are allocated using zeroing allocators (`kzalloc_obj`, `kvcalloc`, `kvzalloc`, `bitmap_zalloc`).\n - Verifier state tracking structures (`struct bpf_verifier_state`, `struct bpf_insn_aux_data`) are zero-initialized via `kzalloc_obj` and `__vmalloc(..., __GFP_ZERO)`. Added fields (`unwinding`, `unwind_frameno`, `unwind_frames`) are explicitly initialized and reset.\n - The stack context used during unwinding (`struct bpf_throw_ctx ctx = {}`) is zero-initialized on the stack.\n\n2. Kernel-to-User Space Copies / Info Leaks:\n - The only user copy introduced is writing `min_size` (a 4-byte scalar `u32`) to user space via `copy_to_bpfptr_offset` on `-E2BIG` in `check_cleanup_info()`. It is fully initialized with `min_t(u32, krec_size, urec_size)`.\n - No other kernel structures, struct padding, or uninitialized buffers are copied to user space.\n\n3. JIT and Runtime Register Spills:\n - On x86-64, JIT prologue and throw-site register spills write all 5 callee-saved / context registers before `arch_bpf_run_cleanup_pad()` reads them.\n - The BPF verifier enforces that landing pad entry clears caller-saved registers (`clear_caller_saved_regs`), initializes `BPF_REG_0` as known, and prevents reading uninitialized registers or stack slots within landing pads.\n\nPotential failure modes in this patch involve CFG exploration, jump history backtracking, verifier prune points, or stack offset calculations—bugs that fall under control-flow logic, out-of-bounds access, or memory corruption detectable by KASAN and standard kernel assertions rather than KMSAN uninitialized memory detectors.",
"NeedsKMSAN": false
} |
| Total Calls | Total Tokens | Avg Tokens | Total Duration (Seconds) | Avg Duration (Seconds) |
|---|
| Total Calls | Total Duration (Seconds) | Avg Duration (Seconds) |
|---|