| Seq | Timestamp | Type | Name | Duration |
|---|---|---|---|---|
| 0/0 | 2026/09/26 01:41 | flow | patch-triage |
2mArgs:null Results: {
"EnableConfigs": [
"BPF_SYSCALL",
"BPF_JIT"
],
"FocusSymbols": [
"bpf_jit_supports_large_stack",
"bpf_id_scratch_reserve",
"bpf_push_jmp_history",
"bpf_bt_sync_linked_regs",
"bpf_mark_chain_precision",
"bpf_compute_subprog_arg_access",
"stacksafe",
"check_max_stack_depth_subprog",
"check_stack_slot_within_bounds",
"check_stack_write_fixed_off",
"check_stack_read_fixed_off",
"widen_frame_masks"
],
"KMSANReasoning": "The patch increases the maximum stack depth budget for BPF programs from 512 bytes (MAX_BPF_STACK) to 2048 bytes (MAX_BPF_STACK_JIT) on architectures whose JIT supports large stacks (x86-64 and arm64). It updates the BPF verifier's stack tracking, liveness analysis, backtracking, and scratch data structures to handle up to 2 KiB frames.\n\nAnalysis of memory safety and detector applicability:\n1. No kernel structures or padding are copied or leaked to user space via copy_to_user or BPF buffers.\n2. All modified and newly allocated data structures are properly initialized:\n - Dynamic scratch buffers (bpf_idmap, bpf_idset) are managed with bpf_id_scratch_reserve() where written entries are initialized before use and freed on exit.\n - Dynamic liveness frame masks (struct frame_masks) are allocated using kvzalloc_flex() and widened with zero-padding intact.\n - Backtracking state (bt) and its stack masks are zero-initialized via bt_reset() (memset) and bitmap_zero().\n - The jump history entries (bpf_jmp_history_entry) properly track linked_regs_cnt and initialize all fields prior to read/unpacking.\n3. BPF verifier still enforces stack slot initialization semantics (tracking STACK_INVALID, STACK_MISC, STACK_ZERO, etc.) over the expanded stack range.\n4. Any potential issues stemming from this change would be stack overflows, verifier state tracking inconsistencies, out-of-bounds accesses during liveness tracking or backtracking, or memory corruption, which are covered by KASAN, stack guards, and other standard debug sanitizers.\n\nBecause no uninitialized memory risks or kernel-to-user info-leaks are introduced, a dedicated KMSAN fuzzing session is not warranted.",
"NeedsKMSAN": false,
"Reasoning": "The patch series expands BPF stack capacity from 512 bytes to 2048 bytes (MAX_BPF_STACK_JIT) on architectures with JIT support (x86_64, arm64). To support this, it modifies core BPF verifier subsystems: jump history linked register bit packing, stack backtracking with enlarged bitmaps, dynamically grown ID scratch maps/sets, dynamically sized per-frame liveness masks, and stack bounds verification. All of this logic is directly reachable via the bpf(BPF_PROG_LOAD) syscall and warrants fuzzing for potential verifier correctness issues, precision tracking regressions, and memory safety flaws.",
"WorthFuzzing": true
} |
| 1/1 | 2026/09/26 01:41 | action | read-patch-diff |
0mArgs:null Results: {
"PatchDiff": "commit 5a159253784d7a46f42cd32160c732f134182d57\nAuthor: syz-cluster \u003ctriage@syzkaller.com\u003e\nDate: Sat Sep 26 01:41:58 2026 +0000\n\n syz-cluster: applied patch under review\n\ndiff --git a/Documentation/bpf/bpf_design_QA.rst b/Documentation/bpf/bpf_design_QA.rst\nindex eb19c945f4d57..f4e4a7f4f9fdb 100644\n--- a/Documentation/bpf/bpf_design_QA.rst\n+++ b/Documentation/bpf/bpf_design_QA.rst\n@@ -221,9 +221,13 @@ newer kernels. BPF programs need to change accordingly when this happens.\n \n Q: How much stack space a BPF program uses?\n -------------------------------------------\n-A: Currently all program types are limited to 512 bytes of stack\n-space, but the verifier computes the actual amount of stack used\n-and both interpreter and most JITed code consume necessary amount.\n+A: A program may use up to 2 KiB of stack, combined over its call\n+chain, when the JIT of the architecture reports support for large\n+stacks (currently x86-64 and arm64); a single function may use all of\n+it, and every frame of a program running on a private stack gets the\n+whole amount. Elsewhere, and whenever the interpreter is used, the\n+limit is 512 bytes. The verifier computes the actual amount of stack\n+used and both interpreter and most JITed code consume necessary amount.\n \n Q: Can BPF be offloaded to HW?\n ------------------------------\ndiff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c\nindex 6c04fee468766..d92d7754578ab 100644\n--- a/arch/arm64/net/bpf_jit_comp.c\n+++ b/arch/arm64/net/bpf_jit_comp.c\n@@ -2485,6 +2485,11 @@ bool bpf_jit_supports_subprog_tailcalls(void)\n \treturn true;\n }\n \n+bool bpf_jit_supports_large_stack(void)\n+{\n+\treturn true;\n+}\n+\n static void invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_node *node,\n \t\t\t int bargs_off, int retval_off, int run_ctx_off,\n \t\t\t bool save_ret)\ndiff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c\nindex d4a980140b48d..d6998c909754f 100644\n--- a/arch/x86/net/bpf_jit_comp.c\n+++ b/arch/x86/net/bpf_jit_comp.c\n@@ -4452,6 +4452,18 @@ bool bpf_jit_supports_subprog_tailcalls(void)\n \treturn true;\n }\n \n+/*\n+ * Frame sizes are 32-bit immediates in the prologue, epilogue and tail call\n+ * sequences, a tail call pops the caller's frame and lands in the target's\n+ * prologue before the target allocates its own, and private stacks are\n+ * allocated from the program's own depth, so MAX_BPF_STACK_JIT frames need\n+ * nothing special.\n+ */\n+bool bpf_jit_supports_large_stack(void)\n+{\n+\treturn true;\n+}\n+\n bool bpf_jit_supports_percpu_insn(void)\n {\n \treturn true;\ndiff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h\nindex 92f528c456052..a5d493b3876fc 100644\n--- a/include/linux/bpf_verifier.h\n+++ b/include/linux/bpf_verifier.h\n@@ -19,11 +19,12 @@\n * that converting umax_value to int cannot overflow.\n */\n #define BPF_MAX_VAR_SIZ\t(1 \u003c\u003c 29)\n-/* size of tmp_str_buf in bpf_verifier.\n- * we need at least 306 bytes to fit full stack mask representation\n- * (in the \"-8,-16,...,-512\" form)\n+/*\n+ * size of tmp_str_buf in bpf_verifier.\n+ * we need at least 1399 bytes to fit full stack mask representation\n+ * (in the \"-8,-16,...,-2048\" form)\n */\n-#define TMP_STR_BUF_LEN 320\n+#define TMP_STR_BUF_LEN 1408\n /* Patch buffer size */\n #define INSN_BUF_SIZE 32\n \n@@ -242,55 +243,18 @@ enum bpf_stack_slot_type {\n \n #define BPF_REG_SIZE 8\t/* size of eBPF register in bytes */\n \n+/*\n+ * Largest number of BPF_REG_SIZE stack slots a single frame can have, sized\n+ * for the largest stack budget any JIT supports. A frame may use any part of\n+ * its program's budget; check_max_stack_depth() enforces the budget on the\n+ * combined depth of frames sharing the kernel stack and on each frame using\n+ * a private stack.\n+ */\n+#define MAX_BPF_STACK_SLOTS\t(MAX_BPF_STACK_JIT / BPF_REG_SIZE)\n+\n /* 4-byte stack slot granularity for liveness analysis */\n #define BPF_HALF_REG_SIZE\t4\n #define STACK_SLOT_SZ\t\t4\n-#define STACK_SLOTS\t\t(MAX_BPF_STACK / BPF_HALF_REG_SIZE)\t/* 128 */\n-\n-typedef struct {\n-\tu64 v[2];\n-} spis_t;\n-\n-#define SPIS_ZERO\t((spis_t){})\n-#define SPIS_ALL\t((spis_t){{ U64_MAX, U64_MAX }})\n-\n-static inline bool spis_is_zero(spis_t s)\n-{\n-\treturn s.v[0] == 0 \u0026\u0026 s.v[1] == 0;\n-}\n-\n-static inline bool spis_equal(spis_t a, spis_t b)\n-{\n-\treturn a.v[0] == b.v[0] \u0026\u0026 a.v[1] == b.v[1];\n-}\n-\n-static inline spis_t spis_or(spis_t a, spis_t b)\n-{\n-\treturn (spis_t){{ a.v[0] | b.v[0], a.v[1] | b.v[1] }};\n-}\n-\n-static inline spis_t spis_and(spis_t a, spis_t b)\n-{\n-\treturn (spis_t){{ a.v[0] \u0026 b.v[0], a.v[1] \u0026 b.v[1] }};\n-}\n-\n-static inline spis_t spis_not(spis_t s)\n-{\n-\treturn (spis_t){{ ~s.v[0], ~s.v[1] }};\n-}\n-\n-static inline bool spis_test_bit(spis_t s, u32 slot)\n-{\n-\treturn s.v[slot / 64] \u0026 BIT_ULL(slot % 64);\n-}\n-\n-static inline void spis_or_range(spis_t *mask, u32 lo, u32 hi)\n-{\n-\tu32 w;\n-\n-\tfor (w = lo; w \u003c= hi \u0026\u0026 w \u003c STACK_SLOTS; w++)\n-\t\tmask-\u003ev[w / 64] |= BIT_ULL(w % 64);\n-}\n \n #define BPF_REGMASK_ARGS ((1 \u003c\u003c BPF_REG_1) | (1 \u003c\u003c BPF_REG_2) | \\\n \t\t\t (1 \u003c\u003c BPF_REG_3) | (1 \u003c\u003c BPF_REG_4) | \\\n@@ -420,30 +384,33 @@ enum {\n \tINSN_F_STACK_ARG_ACCESS = BIT(3),\n };\n \n+/* Registers linked to one jump condition that a history entry can record */\n+#define BPF_LINKED_REGS_MAX\t5\n+\n struct bpf_jmp_history_entry {\n \t/* insn idx can't be bigger than 1 million */\n \tu32 idx : 20;\n \tu32 frame : 4;\t/* stack access frame number */\n-\tu32 spi : 6;\t/* stack slot index (0..63) */\n-\tu32 : 2;\n-\tu32 prev_idx : 20;\n \t/* special INSN_F_xxx flags */\n \tu32 flags : 4;\n-\tu32 : 8;\n+\tu32 : 4;\n+\tu32 prev_idx : 20;\n+\tu32 spi : 12;\t/* stack slot index */\n \t/*\n-\t * additional registers that need precision tracking when this\n-\t * jump is backtracked, vector of five 11-bit records\n+\t * Scalar registers and spilled scalars linked to the condition of\n+\t * this jump, which need precision tracking together when the jump is\n+\t * backtracked. Each is packed as 4 bits of frame number, one bit\n+\t * telling a register from a stack slot and 11 bits of register or\n+\t * slot index, see linked_regs_pack().\n \t */\n-\tu64 linked_regs;\n+\tu16 linked_regs[BPF_LINKED_REGS_MAX];\n+\tu8 linked_regs_cnt;\n };\n \n static_assert(MAX_CALL_FRAMES \u003c= (1 \u003c\u003c 4));\n-static_assert(MAX_BPF_STACK / 8 \u003c= (1 \u003c\u003c 6));\n+static_assert(MAX_BPF_STACK_SLOTS \u003c= (1 \u003c\u003c 12));\n \n-/* Maximum number of bpf_reg_state objects that can exist at once */\n #define MAX_STACK_ARG_SLOTS (MAX_BPF_FUNC_ARGS - MAX_BPF_FUNC_REG_ARGS)\n-#define BPF_ID_MAP_SIZE ((MAX_BPF_REG + MAX_BPF_STACK / BPF_REG_SIZE + \\\n-\t\t\t MAX_STACK_ARG_SLOTS) * MAX_CALL_FRAMES)\n struct bpf_verifier_state {\n \t/* call stack tracking */\n \tstruct bpf_func_state *frame[MAX_CALL_FRAMES];\n@@ -529,12 +496,27 @@ struct bpf_verifier_state {\n \tu32 may_goto_depth;\n };\n \n+/* Number of BPF_REG_SIZE stack slots tracked for the frame so far. */\n+static inline u32 bpf_stack_nr_slots(const struct bpf_func_state *frame)\n+{\n+\treturn frame-\u003eallocated_stack / BPF_REG_SIZE;\n+}\n+\n+/*\n+ * Stack slot @spi of @frame, covering bytes [fp - (spi + 1) * 8, fp - spi * 8).\n+ * The caller must ensure spi \u003c bpf_stack_nr_slots(frame), see grow_stack_state().\n+ */\n+static inline struct bpf_stack_state *bpf_stack_slot(const struct bpf_func_state *frame, u32 spi)\n+{\n+\treturn \u0026frame-\u003estack[spi];\n+}\n+\n static inline struct bpf_reg_state *\n bpf_get_spilled_reg(int slot, struct bpf_func_state *frame, u32 mask)\n {\n-\tif (slot \u003c frame-\u003eallocated_stack / BPF_REG_SIZE \u0026\u0026\n-\t (1 \u003c\u003c frame-\u003estack[slot].slot_type[BPF_REG_SIZE - 1]) \u0026 mask)\n-\t\treturn \u0026frame-\u003estack[slot].spilled_ptr;\n+\tif (slot \u003c bpf_stack_nr_slots(frame) \u0026\u0026\n+\t (1 \u003c\u003c bpf_stack_slot(frame, slot)-\u003eslot_type[BPF_REG_SIZE - 1]) \u0026 mask)\n+\t\treturn \u0026bpf_stack_slot(frame, slot)-\u003espilled_ptr;\n \treturn NULL;\n }\n \n@@ -550,7 +532,7 @@ bpf_get_spilled_stack_arg(int slot, struct bpf_func_state *frame)\n /* Iterate over 'frame', setting 'reg' to either NULL or a spilled register. */\n #define bpf_for_each_spilled_reg(iter, frame, reg, mask)\t\t\t\\\n \tfor (iter = 0, reg = bpf_get_spilled_reg(iter, frame, mask);\t\t\\\n-\t iter \u003c frame-\u003eallocated_stack / BPF_REG_SIZE;\t\t\\\n+\t iter \u003c bpf_stack_nr_slots(frame);\t\t\t\t\\\n \t iter++, reg = bpf_get_spilled_reg(iter, frame, mask))\n \n /* Iterate over 'frame', setting 'reg' to either NULL or a spilled stack arg. */\n@@ -575,7 +557,7 @@ bpf_get_spilled_stack_arg(int slot, struct bpf_func_state *frame)\n \t\t\tbpf_for_each_spilled_reg(___j, __state, __reg, __mask) { \\\n \t\t\t\tif (!__reg) \\\n \t\t\t\t\tcontinue; \\\n-\t\t\t\t__stack = \u0026__state-\u003estack[___j]; \\\n+\t\t\t\t__stack = bpf_stack_slot(__state, ___j); \\\n \t\t\t\t(void)(__expr); \\\n \t\t\t} \\\n \t\t\t__stack = NULL; \\\n@@ -850,7 +832,7 @@ struct backtrack_state {\n \tstruct bpf_verifier_env *env;\n \tu32 frame;\n \tu32 reg_masks[MAX_CALL_FRAMES];\n-\tu64 stack_masks[MAX_CALL_FRAMES];\n+\tunsigned long stack_masks[MAX_CALL_FRAMES][BITS_TO_LONGS(MAX_BPF_STACK_SLOTS)];\n \tu8 stack_arg_masks[MAX_CALL_FRAMES];\n };\n \n@@ -859,18 +841,27 @@ struct bpf_id_pair {\n \tu32 cur;\n };\n \n+/*\n+ * Scratch map from the ids of one verifier state to those of another, also\n+ * used as a stack of ids. Grown on demand by bpf_id_scratch_reserve().\n+ */\n struct bpf_idmap {\n \tu32 tmp_id_gen;\n \tu32 cnt;\n-\tstruct bpf_id_pair map[BPF_ID_MAP_SIZE];\n+\tu32 cap;\n+\tstruct bpf_id_pair *map;\n+};\n+\n+struct bpf_idset_entry {\n+\tu32 id;\n+\tu32 cnt;\n };\n \n+/* Scratch set of ids with a use count each, grown on demand */\n struct bpf_idset {\n \tu32 num_ids;\n-\tstruct {\n-\t\tu32 id;\n-\t\tu32 cnt;\n-\t} entries[BPF_ID_MAP_SIZE];\n+\tu32 cap;\n+\tstruct bpf_idset_entry *entries;\n };\n \n /* see verifier.c:compute_scc_callchain() */\n@@ -965,10 +956,8 @@ struct bpf_verifier_env {\n \tstruct bpf_subprog_info subprog_info[BPF_MAX_SUBPROGS + 2]; /* max + 2 for the fake and exception subprogs */\n \t/* subprog indices sorted in topological order: leaves first, callers last */\n \tint subprog_topo_order[BPF_MAX_SUBPROGS + 2];\n-\tunion {\n-\t\tstruct bpf_idmap idmap_scratch;\n-\t\tstruct bpf_idset idset_scratch;\n-\t};\n+\tstruct bpf_idmap idmap_scratch;\n+\tstruct bpf_idset idset_scratch;\n \tstruct {\n \t\tint *insn_state;\n \t\tint *insn_stack;\n@@ -993,6 +982,8 @@ struct bpf_verifier_env {\n \tu32 prev_jmps_processed, jmps_processed;\n \t/* maximum combined stack depth */\n \tu32 max_stack_depth;\n+\t/* stack budget of the program, see bpf_prog_stack_limit() */\n+\tu32 stack_limit;\n \t/* total verification time */\n \tu64 verification_time;\n \t/* maximum number of verifier states kept in 'branching' instructions */\n@@ -1028,7 +1019,7 @@ struct bpf_verifier_env {\n \t */\n \tu32 scratched_regs;\n \t/* Same as scratched_regs but for stack slots */\n-\tu64 scratched_stack_slots;\n+\tDECLARE_BITMAP(scratched_stack_slots, MAX_BPF_STACK_SLOTS);\n \tu64 prev_log_pos, prev_insn_print_pos;\n \t/* buffer used to temporary hold constants as scalar registers */\n \tstruct bpf_reg_state fake_reg[1];\n@@ -1226,8 +1217,10 @@ int bpf_copy_verifier_state(struct bpf_verifier_state *dst_state,\n struct list_head *bpf_explored_state(struct bpf_verifier_env *env, int idx);\n void bpf_free_verifier_state(struct bpf_verifier_state *state, bool free_self);\n void bpf_free_backedges(struct bpf_scc_visit *visit);\n+bool bpf_id_scratch_reserve(void **arr, u32 *cap, u32 cnt, size_t elem_size);\n int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state *cur,\n-\t\t\t int insn_flags, int spi, int frame, u64 linked_regs);\n+\t\t\t int insn_flags, int spi, int frame, const u16 *linked_regs,\n+\t\t\t u8 linked_regs_cnt);\n void bpf_bt_sync_linked_regs(struct backtrack_state *bt, struct bpf_jmp_history_entry *hist);\n void bpf_mark_reg_not_init(const struct bpf_verifier_env *env,\n \t\t\t struct bpf_reg_state *reg);\n@@ -1244,6 +1237,24 @@ static inline int bpf_get_spi(s32 off)\n \treturn (-off - 1) / BPF_REG_SIZE;\n }\n \n+/*\n+ * Stack a program may use in total: combined over the frames of a call\n+ * chain on the kernel stack, or per frame on a private stack. Any single\n+ * frame may reach that deep. Only a JIT that lays out such frames may go\n+ * beyond MAX_BPF_STACK, the interpreter's frame size, and only one whose\n+ * tail calls let the target set up its own frame: without subprogram\n+ * tail calls, do_misc_fixups() gives every program with tail calls a\n+ * MAX_BPF_STACK frame, which a deeper frame would overrun.\n+ */\n+static inline u32 bpf_prog_stack_limit(const struct bpf_prog *prog)\n+{\n+\t/* an offloaded program never runs on the host JIT, whatever it supports */\n+\tif (prog-\u003ejit_requested \u0026\u0026 !bpf_prog_is_offloaded(prog-\u003eaux) \u0026\u0026\n+\t bpf_jit_supports_large_stack() \u0026\u0026 bpf_jit_supports_subprog_tailcalls())\n+\t\treturn MAX_BPF_STACK_JIT;\n+\treturn MAX_BPF_STACK;\n+}\n+\n static inline struct bpf_func_state *bpf_func(struct bpf_verifier_env *env,\n \t\t\t\t\t const struct bpf_reg_state *reg)\n {\n@@ -1287,12 +1298,7 @@ static inline void bpf_bt_set_frame_reg(struct backtrack_state *bt, u32 frame, u\n \n static inline void bpf_bt_set_frame_slot(struct backtrack_state *bt, u32 frame, u32 slot)\n {\n-\tbt-\u003estack_masks[frame] |= 1ull \u003c\u003c slot;\n-}\n-\n-static inline void bpf_bt_set_frame_slot_mask(struct backtrack_state *bt, u32 frame, u64 mask)\n-{\n-\tbt-\u003estack_masks[frame] |= mask;\n+\t__set_bit(slot, bt-\u003estack_masks[frame]);\n }\n \n static inline void bt_set_frame_stack_arg_slot(struct backtrack_state *bt, u32 frame, u32 slot)\n@@ -1307,7 +1313,7 @@ static inline bool bt_is_frame_reg_set(struct backtrack_state *bt, u32 frame, u3\n \n static inline bool bt_is_frame_slot_set(struct backtrack_state *bt, u32 frame, u32 slot)\n {\n-\treturn bt-\u003estack_masks[frame] \u0026 (1ull \u003c\u003c slot);\n+\treturn test_bit(slot, bt-\u003estack_masks[frame]);\n }\n \n bool bpf_map_is_rdonly(const struct bpf_map *map);\n@@ -1438,7 +1444,7 @@ static inline void mark_reg_scratched(struct bpf_verifier_env *env, u32 regno)\n \n static inline void mark_stack_slot_scratched(struct bpf_verifier_env *env, u32 spi)\n {\n-\tenv-\u003escratched_stack_slots |= 1ULL \u003c\u003c spi;\n+\t__set_bit(spi, env-\u003escratched_stack_slots);\n }\n \n static inline bool reg_scratched(const struct bpf_verifier_env *env, u32 regno)\n@@ -1446,27 +1452,28 @@ static inline bool reg_scratched(const struct bpf_verifier_env *env, u32 regno)\n \treturn (env-\u003escratched_regs \u003e\u003e regno) \u0026 1;\n }\n \n-static inline bool stack_slot_scratched(const struct bpf_verifier_env *env, u64 regno)\n+static inline bool stack_slot_scratched(const struct bpf_verifier_env *env, u32 spi)\n {\n-\treturn (env-\u003escratched_stack_slots \u003e\u003e regno) \u0026 1;\n+\treturn test_bit(spi, env-\u003escratched_stack_slots);\n }\n \n static inline bool verifier_state_scratched(const struct bpf_verifier_env *env)\n {\n-\treturn env-\u003escratched_regs || env-\u003escratched_stack_slots;\n+\treturn env-\u003escratched_regs ||\n+\t !bitmap_empty(env-\u003escratched_stack_slots, MAX_BPF_STACK_SLOTS);\n }\n \n static inline void mark_verifier_state_clean(struct bpf_verifier_env *env)\n {\n \tenv-\u003escratched_regs = 0U;\n-\tenv-\u003escratched_stack_slots = 0ULL;\n+\tbitmap_zero(env-\u003escratched_stack_slots, MAX_BPF_STACK_SLOTS);\n }\n \n /* Used for printing the entire verifier state. */\n static inline void mark_verifier_state_scratched(struct bpf_verifier_env *env)\n {\n \tenv-\u003escratched_regs = ~0U;\n-\tenv-\u003escratched_stack_slots = ~0ULL;\n+\tbitmap_fill(env-\u003escratched_stack_slots, MAX_BPF_STACK_SLOTS);\n }\n \n static inline bool bpf_stack_narrow_access_ok(int off, int fill_size, int spill_size)\n@@ -1501,7 +1508,7 @@ struct bpf_subprog_info *bpf_find_containing_subprog(struct bpf_verifier_env *en\n const char *bpf_subprog_name(const struct bpf_verifier_env *env, int subprog);\n int bpf_jmp_offset(struct bpf_insn *insn);\n struct bpf_iarray *bpf_insn_successors(struct bpf_verifier_env *env, u32 idx);\n-void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask);\n+void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, const unsigned long *stack_mask);\n bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog);\n \n /* Kinds of member a by-value struct or union may be composed of. */\ndiff --git a/include/linux/filter.h b/include/linux/filter.h\nindex 422284b4fa96f..cdd16bdd4dfb6 100644\n--- a/include/linux/filter.h\n+++ b/include/linux/filter.h\n@@ -98,6 +98,11 @@ struct ctl_table_header;\n \n /* BPF program can access up to 512 bytes of stack space. */\n #define MAX_BPF_STACK\t512\n+/*\n+ * Stack budget of a program on a JIT that lays out frames of that size.\n+ * The interpreter and JITs without such support keep MAX_BPF_STACK.\n+ */\n+#define MAX_BPF_STACK_JIT\t2048\n \n /* Helper macros for filter block array initializers. */\n \n@@ -1246,6 +1251,7 @@ bool bpf_jit_supports_ptr_xchg(void);\n bool bpf_jit_supports_arena(void);\n bool bpf_jit_supports_insn(struct bpf_insn *insn, bool in_arena);\n bool bpf_jit_supports_private_stack(void);\n+bool bpf_jit_supports_large_stack(void);\n bool bpf_jit_supports_timed_may_goto(void);\n bool bpf_jit_supports_fsession(void);\n \ndiff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c\nindex 507a366dffa47..db1be14d0a680 100644\n--- a/kernel/bpf/backtrack.c\n+++ b/kernel/bpf/backtrack.c\n@@ -9,7 +9,8 @@\n \n /* for any branch, call, exit record the history of jmps in the given state */\n int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state *cur,\n-\t\t\t int insn_flags, int spi, int frame, u64 linked_regs)\n+\t\t\t int insn_flags, int spi, int frame, const u16 *linked_regs,\n+\t\t\t u8 linked_regs_cnt)\n {\n \tu32 cnt = cur-\u003ejmp_history_cnt;\n \tstruct bpf_jmp_history_entry *p;\n@@ -27,10 +28,13 @@ int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state\n \t\tenv-\u003ecur_hist_ent-\u003eflags |= insn_flags;\n \t\tenv-\u003ecur_hist_ent-\u003espi = spi;\n \t\tenv-\u003ecur_hist_ent-\u003eframe = frame;\n-\t\tverifier_bug_if(env-\u003ecur_hist_ent-\u003elinked_regs != 0, env,\n-\t\t\t\t\"insn history: insn_idx %d linked_regs: %#llx\",\n-\t\t\t\tenv-\u003einsn_idx, env-\u003ecur_hist_ent-\u003elinked_regs);\n-\t\tenv-\u003ecur_hist_ent-\u003elinked_regs = linked_regs;\n+\t\tverifier_bug_if(env-\u003ecur_hist_ent-\u003elinked_regs_cnt != 0, env,\n+\t\t\t\t\"insn history: insn_idx %d has %u linked regs\",\n+\t\t\t\tenv-\u003einsn_idx, env-\u003ecur_hist_ent-\u003elinked_regs_cnt);\n+\t\tif (linked_regs_cnt)\n+\t\t\tmemcpy(env-\u003ecur_hist_ent-\u003elinked_regs, linked_regs,\n+\t\t\t linked_regs_cnt * sizeof(*linked_regs));\n+\t\tenv-\u003ecur_hist_ent-\u003elinked_regs_cnt = linked_regs_cnt;\n \t\treturn 0;\n \t}\n \n@@ -47,7 +51,9 @@ int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state\n \tp-\u003eflags = insn_flags;\n \tp-\u003espi = spi;\n \tp-\u003eframe = frame;\n-\tp-\u003elinked_regs = linked_regs;\n+\tif (linked_regs_cnt)\n+\t\tmemcpy(p-\u003elinked_regs, linked_regs, linked_regs_cnt * sizeof(*linked_regs));\n+\tp-\u003elinked_regs_cnt = linked_regs_cnt;\n \tcur-\u003ejmp_history_cnt = cnt;\n \tenv-\u003ecur_hist_ent = p;\n \n@@ -123,13 +129,26 @@ static inline void bt_reset(struct backtrack_state *bt)\n \tbt-\u003eenv = env;\n }\n \n-static inline u32 bt_empty(struct backtrack_state *bt)\n+static inline bool bt_frame_stack_empty(struct backtrack_state *bt, u32 frame)\n {\n-\tu64 mask = 0;\n+\treturn bitmap_empty(bt-\u003estack_masks[frame], MAX_BPF_STACK_SLOTS);\n+}\n+\n+static inline bool bt_stack_empty(struct backtrack_state *bt)\n+{\n+\treturn bt_frame_stack_empty(bt, bt-\u003eframe);\n+}\n+\n+static inline bool bt_empty(struct backtrack_state *bt)\n+{\n+\tu32 mask = 0;\n \tint i;\n \n-\tfor (i = 0; i \u003c= bt-\u003eframe; i++)\n-\t\tmask |= bt-\u003ereg_masks[i] | bt-\u003estack_masks[i] | bt-\u003estack_arg_masks[i];\n+\tfor (i = 0; i \u003c= bt-\u003eframe; i++) {\n+\t\tmask |= bt-\u003ereg_masks[i] | bt-\u003estack_arg_masks[i];\n+\t\tif (!bt_frame_stack_empty(bt, i))\n+\t\t\treturn false;\n+\t}\n \n \treturn mask == 0;\n }\n@@ -181,7 +200,7 @@ static inline void bt_clear_reg(struct backtrack_state *bt, u32 reg)\n \n static inline void bt_clear_frame_slot(struct backtrack_state *bt, u32 frame, u32 slot)\n {\n-\tbt-\u003estack_masks[frame] \u0026= ~(1ull \u003c\u003c slot);\n+\t__clear_bit(slot, bt-\u003estack_masks[frame]);\n }\n \n static inline u32 bt_frame_reg_mask(struct backtrack_state *bt, u32 frame)\n@@ -194,14 +213,14 @@ static inline u32 bt_reg_mask(struct backtrack_state *bt)\n \treturn bt-\u003ereg_masks[bt-\u003eframe];\n }\n \n-static inline u64 bt_frame_stack_mask(struct backtrack_state *bt, u32 frame)\n+static inline unsigned long *bt_frame_stack_mask(struct backtrack_state *bt, u32 frame)\n {\n \treturn bt-\u003estack_masks[frame];\n }\n \n-static inline u64 bt_stack_mask(struct backtrack_state *bt)\n+static inline unsigned long *bt_stack_mask(struct backtrack_state *bt)\n {\n-\treturn bt-\u003estack_masks[bt-\u003eframe];\n+\treturn bt_frame_stack_mask(bt, bt-\u003eframe);\n }\n \n static inline u8 bt_stack_arg_mask(struct backtrack_state *bt)\n@@ -233,17 +252,16 @@ static void fmt_reg_mask(char *buf, ssize_t buf_sz, u32 reg_mask)\n \t\t\tbreak;\n \t}\n }\n-/* format stack slots bitmask, e.g., \"-8,-24,-40\" for 0x15 mask */\n-void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask)\n+\n+/* format stack slots bitmask, e.g., \"-8,-24,-40\" for slots 0, 2 and 4 */\n+void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, const unsigned long *stack_mask)\n {\n-\tDECLARE_BITMAP(mask, 64);\n \tbool first = true;\n \tint i, n;\n \n \tbuf[0] = '\\0';\n \n-\tbitmap_from_u64(mask, stack_mask);\n-\tfor_each_set_bit(i, mask, 64) {\n+\tfor_each_set_bit(i, stack_mask, MAX_BPF_STACK_SLOTS) {\n \t\tn = snprintf(buf, buf_sz, \"%s%d\", first ? \"\" : \",\", -(i + 1) * 8);\n \t\tfirst = false;\n \t\tbuf += n;\n@@ -452,10 +470,11 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,\n \t\t\t\t/* we are now tracking register spills correctly,\n \t\t\t\t * so any instance of leftover slots is a bug\n \t\t\t\t */\n-\t\t\t\tif (bt_stack_mask(bt) != 0) {\n-\t\t\t\t\tverifier_bug(env,\n-\t\t\t\t\t\t \"static subprog leftover stack slots %llx\",\n-\t\t\t\t\t\t bt_stack_mask(bt));\n+\t\t\t\tif (!bt_stack_empty(bt)) {\n+\t\t\t\t\tbpf_fmt_stack_mask(env-\u003etmp_str_buf, TMP_STR_BUF_LEN,\n+\t\t\t\t\t\t\t bt_stack_mask(bt));\n+\t\t\t\t\tverifier_bug(env, \"static subprog leftover stack slots %s\",\n+\t\t\t\t\t\t env-\u003etmp_str_buf);\n \t\t\t\t\treturn -EFAULT;\n \t\t\t\t}\n \t\t\t\t/* propagate r1-r5 to the caller */\n@@ -488,9 +507,11 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,\n \t\t\t\t\t bt_reg_mask(bt));\n \t\t\t\treturn -EFAULT;\n \t\t\t}\n-\t\t\tif (bt_stack_mask(bt) != 0) {\n-\t\t\t\tverifier_bug(env, \"callback leftover stack slots %llx\",\n-\t\t\t\t\t bt_stack_mask(bt));\n+\t\t\tif (!bt_stack_empty(bt)) {\n+\t\t\t\tbpf_fmt_stack_mask(env-\u003etmp_str_buf, TMP_STR_BUF_LEN,\n+\t\t\t\t\t\t bt_stack_mask(bt));\n+\t\t\t\tverifier_bug(env, \"callback leftover stack slots %s\",\n+\t\t\t\t\t env-\u003etmp_str_buf);\n \t\t\t\treturn -EFAULT;\n \t\t\t}\n \t\t\t/* clear r1-r5 in callback subprog's mask */\n@@ -707,10 +728,10 @@ void bpf_mark_all_scalars_precise(struct bpf_verifier_env *env,\n \t\t\t\t\t\ti, j);\n \t\t\t\t}\n \t\t\t}\n-\t\t\tfor (j = 0; j \u003c func-\u003eallocated_stack / BPF_REG_SIZE; j++) {\n-\t\t\t\tif (!bpf_is_spilled_reg(\u0026func-\u003estack[j]))\n+\t\t\tfor (j = 0; j \u003c bpf_stack_nr_slots(func); j++) {\n+\t\t\t\tif (!bpf_is_spilled_reg(bpf_stack_slot(func, j)))\n \t\t\t\t\tcontinue;\n-\t\t\t\treg = \u0026func-\u003estack[j].spilled_ptr;\n+\t\t\t\treg = \u0026bpf_stack_slot(func, j)-\u003espilled_ptr;\n \t\t\t\tif (reg-\u003etype != SCALAR_VALUE || reg-\u003eprecise)\n \t\t\t\t\tcontinue;\n \t\t\t\treg-\u003eprecise = true;\n@@ -868,7 +889,7 @@ int bpf_mark_chain_precision(struct bpf_verifier_env *env,\n \t\t\tif (st-\u003ecurframe == 0 \u0026\u0026\n \t\t\t st-\u003eframe[0]-\u003esubprogno \u003e 0 \u0026\u0026\n \t\t\t st-\u003eframe[0]-\u003ecallsite == BPF_MAIN_FUNC \u0026\u0026\n-\t\t\t bt_stack_mask(bt) == 0 \u0026\u0026\n+\t\t\t bt_stack_empty(bt) \u0026\u0026\n \t\t\t (bt_reg_mask(bt) \u0026 ~BPF_REGMASK_ARGS) == 0) {\n \t\t\t\tbitmap_from_u64(mask, bt_reg_mask(bt));\n \t\t\t\tfor_each_set_bit(i, mask, 32) {\n@@ -882,8 +903,9 @@ int bpf_mark_chain_precision(struct bpf_verifier_env *env,\n \t\t\t\treturn 0;\n \t\t\t}\n \n-\t\t\tverifier_bug(env, \"backtracking func entry subprog %d reg_mask %x stack_mask %llx\",\n-\t\t\t\t st-\u003eframe[0]-\u003esubprogno, bt_reg_mask(bt), bt_stack_mask(bt));\n+\t\t\tbpf_fmt_stack_mask(env-\u003etmp_str_buf, TMP_STR_BUF_LEN, bt_stack_mask(bt));\n+\t\t\tverifier_bug(env, \"backtracking func entry subprog %d reg_mask %x stack_mask %s\",\n+\t\t\t\t st-\u003eframe[0]-\u003esubprogno, bt_reg_mask(bt), env-\u003etmp_str_buf);\n \t\t\treturn -EFAULT;\n \t\t}\n \n@@ -944,18 +966,17 @@ int bpf_mark_chain_precision(struct bpf_verifier_env *env,\n \t\t\t\t}\n \t\t\t}\n \n-\t\t\tbitmap_from_u64(mask, bt_frame_stack_mask(bt, fr));\n-\t\t\tfor_each_set_bit(i, mask, 64) {\n-\t\t\t\tif (verifier_bug_if(i \u003e= func-\u003eallocated_stack / BPF_REG_SIZE,\n+\t\t\tfor_each_set_bit(i, bt_frame_stack_mask(bt, fr), MAX_BPF_STACK_SLOTS) {\n+\t\t\t\tif (verifier_bug_if(i \u003e= bpf_stack_nr_slots(func),\n \t\t\t\t\t\t env, \"stack slot %d, total slots %d\",\n-\t\t\t\t\t\t i, func-\u003eallocated_stack / BPF_REG_SIZE))\n+\t\t\t\t\t\t i, bpf_stack_nr_slots(func)))\n \t\t\t\t\treturn -EFAULT;\n \n-\t\t\t\tif (!bpf_is_spilled_scalar_reg(\u0026func-\u003estack[i])) {\n+\t\t\t\tif (!bpf_is_spilled_scalar_reg(bpf_stack_slot(func, i))) {\n \t\t\t\t\tbt_clear_frame_slot(bt, fr, i);\n \t\t\t\t\tcontinue;\n \t\t\t\t}\n-\t\t\t\treg = \u0026func-\u003estack[i].spilled_ptr;\n+\t\t\t\treg = \u0026bpf_stack_slot(func, i)-\u003espilled_ptr;\n \t\t\t\tif (reg-\u003eprecise) {\n \t\t\t\t\tbt_clear_frame_slot(bt, fr, i);\n \t\t\t\t} else {\ndiff --git a/kernel/bpf/core.c b/kernel/bpf/core.c\nindex 227211166dccf..fbb2d8a840ef3 100644\n--- a/kernel/bpf/core.c\n+++ b/kernel/bpf/core.c\n@@ -3471,6 +3471,19 @@ bool __weak bpf_jit_supports_private_stack(void)\n \treturn false;\n }\n \n+/*\n+ * Return TRUE if the JIT lays out frames of up to MAX_BPF_STACK_JIT bytes.\n+ * Its prologue, epilogue and tail call sequences must encode such frame\n+ * sizes and a private stack must be sized from the program's depth. The\n+ * budget is only granted alongside bpf_jit_supports_subprog_tailcalls(),\n+ * whose tail calls land before the target sets up its own frame; see\n+ * bpf_prog_stack_limit().\n+ */\n+bool __weak bpf_jit_supports_large_stack(void)\n+{\n+\treturn false;\n+}\n+\n void __weak arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, u64 bp), void *cookie)\n {\n }\ndiff --git a/kernel/bpf/diagnostics.c b/kernel/bpf/diagnostics.c\nindex 5ecfa86ed49f4..a8ed6130c1373 100644\n--- a/kernel/bpf/diagnostics.c\n+++ b/kernel/bpf/diagnostics.c\n@@ -1600,9 +1600,9 @@ static struct bpf_reg_state *target_to_reg(struct bpf_verifier_env *env,\n \t\t\treturn NULL;\n \t\treturn \u0026state-\u003estack_arg_regs[target-\u003estack_arg];\n \tcase BPF_DIAG_MOD_TARGET_STACK_SLOT:\n-\t\tif (target-\u003espi \u003e= state-\u003eallocated_stack / BPF_REG_SIZE)\n+\t\tif (target-\u003espi \u003e= bpf_stack_nr_slots(state))\n \t\t\treturn NULL;\n-\t\treturn \u0026state-\u003estack[target-\u003espi].spilled_ptr;\n+\t\treturn \u0026bpf_stack_slot(state, target-\u003espi)-\u003espilled_ptr;\n \tdefault:\n \t\treturn NULL;\n \t}\n@@ -1618,7 +1618,7 @@ static bool reg_to_target(struct bpf_verifier_env *env, const struct bpf_reg_sta\n \tfor (frame = 0; frame \u003c= vstate-\u003ecurframe; frame++) {\n \t\tstruct bpf_func_state *state = vstate-\u003eframe[frame];\n \t\tunsigned long start, end;\n-\t\tu32 nslots = state-\u003eallocated_stack / BPF_REG_SIZE;\n+\t\tu32 nslots = bpf_stack_nr_slots(state);\n \t\tint spi;\n \n \t\tstart = (unsigned long)state-\u003eregs;\ndiff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c\nindex 44ecdc5b4ec2d..f9beef6695c44 100644\n--- a/kernel/bpf/liveness.c\n+++ b/kernel/bpf/liveness.c\n@@ -10,10 +10,39 @@\n \n #define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)\n \n-struct per_frame_masks {\n-\tspis_t may_read;\t/* stack slots that may be read by this instruction */\n-\tspis_t must_write;\t/* stack slots written by this instruction */\n-\tspis_t live_before;\t/* stack slots that may be read by this insn and its successors */\n+/*\n+ * Stack liveness is tracked with a 4-byte (half register) granularity.\n+ * Half-slot 0 covers [fp-4, fp), half-slot 1 covers [fp-8, fp-4), and so on,\n+ * hence FRAME_HALF_SPIS - 1 is the deepest half-slot a frame can have.\n+ */\n+#define FRAME_HALF_SPIS\t\t(MAX_BPF_STACK_JIT / BPF_HALF_REG_SIZE)\n+#define FRAME_MAX_WORDS\t\tBITS_TO_LONGS(FRAME_HALF_SPIS)\n+\n+/* Masks tracked for each instruction of a frame */\n+enum {\n+\tFM_MAY_READ,\t/* stack slots that may be read by this instruction */\n+\tFM_MUST_WRITE,\t/* stack slots written by this instruction */\n+\tFM_LIVE_BEFORE,\t/* stack slots that may be read by this insn and its successors */\n+\tFM_MASK_CNT,\n+};\n+\n+/*\n+ * Per instruction stack masks for one frame of a function instance.\n+ *\n+ * Most frames use only a fraction of the stack budget, so instead of masks\n+ * wide enough for every half-slot of the largest possible frame, all masks\n+ * of one array share the width @words, and the marking functions widen the\n+ * array as deeper half-slots are recorded. Mask @kind of the instruction at\n+ * relative index @i is at \u0026bits[(i * FM_MASK_CNT + kind) * words]. A\n+ * half-slot at or past @words * BITS_PER_LONG is never read by this frame,\n+ * hence never live. An instruction that may read the whole frame, such as a\n+ * call passing a frame pointer to another subprog, widens the array to the\n+ * program's stack budget, the deepest an accepted program can reach, so\n+ * that the read cannot lose half-slots to a later widening.\n+ */\n+struct frame_masks {\n+\tu32 words;\n+\tunsigned long bits[];\n };\n \n /*\n@@ -29,7 +58,7 @@ struct func_instance {\n \tu32 subprog_start;\t/* cached env-\u003esubprog_info[subprog].start */\n \tu32 insn_cnt;\t\t/* cached number of insns in the function */\n \t/* Per frame, per instruction masks, frames allocated lazily. */\n-\tstruct per_frame_masks *frames[MAX_CALL_FRAMES];\n+\tstruct frame_masks *frames[MAX_CALL_FRAMES];\n \tbool must_write_initialized;\n };\n \n@@ -151,50 +180,122 @@ static int relative_idx(struct func_instance *instance, u32 insn_idx)\n \treturn insn_idx - instance-\u003esubprog_start;\n }\n \n-static struct per_frame_masks *get_frame_masks(struct func_instance *instance,\n-\t\t\t\t\t u32 frame, u32 insn_idx)\n+static u32 frame_mask_bits(struct frame_masks *fm)\n {\n-\tif (!instance-\u003eframes[frame])\n-\t\treturn NULL;\n+\treturn fm-\u003ewords * BITS_PER_LONG;\n+}\n \n-\treturn \u0026instance-\u003eframes[frame][relative_idx(instance, insn_idx)];\n+static size_t frame_mask_words(struct func_instance *instance, u32 words)\n+{\n+\treturn (size_t)instance-\u003einsn_cnt * FM_MASK_CNT * words;\n }\n \n-static struct per_frame_masks *alloc_frame_masks(struct func_instance *instance,\n-\t\t\t\t\t\t u32 frame, u32 insn_idx)\n+/* Mask @kind of the instruction at relative index @rel */\n+static unsigned long *rel_mask(struct frame_masks *fm, u32 rel, u32 kind)\n {\n-\tstruct per_frame_masks *arr;\n+\treturn fm-\u003ebits + ((size_t)rel * FM_MASK_CNT + kind) * fm-\u003ewords;\n+}\n \n-\tif (!instance-\u003eframes[frame]) {\n-\t\tarr = kvzalloc_objs(*arr, instance-\u003einsn_cnt,\n-\t\t\t\t GFP_KERNEL_ACCOUNT);\n-\t\tinstance-\u003eframes[frame] = arr;\n-\t\tif (!arr)\n-\t\t\treturn ERR_PTR(-ENOMEM);\n+/*\n+ * Make sure @frame has a mask array at least @words wide, allocating it or\n+ * copying the existing masks into the wider stride as needed.\n+ * @words must be in range [1, FRAME_MAX_WORDS].\n+ */\n+static struct frame_masks *widen_frame_masks(struct func_instance *instance,\n+\t\t\t\t\t u32 frame, u32 words)\n+{\n+\tstruct frame_masks *old = instance-\u003eframes[frame], *new;\n+\tu32 i, kind;\n+\n+\tif (old \u0026\u0026 old-\u003ewords \u003e= words)\n+\t\treturn old;\n+\tnew = kvzalloc_flex(*new, bits, frame_mask_words(instance, words), GFP_KERNEL_ACCOUNT);\n+\tif (!new)\n+\t\treturn NULL;\n+\tnew-\u003ewords = words;\n+\tif (old) {\n+\t\tfor (i = 0; i \u003c instance-\u003einsn_cnt; i++)\n+\t\t\tfor (kind = 0; kind \u003c FM_MASK_CNT; kind++)\n+\t\t\t\tmemcpy(rel_mask(new, i, kind), rel_mask(old, i, kind),\n+\t\t\t\t old-\u003ewords * sizeof(*old-\u003ebits));\n+\t\tkvfree(old);\n \t}\n-\treturn get_frame_masks(instance, frame, insn_idx);\n+\tinstance-\u003eframes[frame] = new;\n+\treturn new;\n }\n \n-/* Accumulate may_read masks for @frame at @insn_idx */\n-static int mark_stack_read(struct func_instance *instance, u32 frame, u32 insn_idx, spis_t mask)\n+/*\n+ * Set the inclusive half-slot range [lo, hi] in mask @kind of @frame at @insn_idx.\n+ * An empty range, including one with a negative @hi as computed for a write\n+ * that does not fully cover any half-slot, marks nothing.\n+ */\n+static int mark_stack_range(struct func_instance *instance, u32 frame, u32 insn_idx,\n+\t\t\t u32 kind, s32 lo, s32 hi)\n {\n-\tstruct per_frame_masks *masks;\n+\tstruct frame_masks *fm;\n \n-\tmasks = alloc_frame_masks(instance, frame, insn_idx);\n-\tif (IS_ERR(masks))\n-\t\treturn PTR_ERR(masks);\n-\tmasks-\u003emay_read = spis_or(masks-\u003emay_read, mask);\n+\t/*\n+\t * An access past the frame bottom is rejected by the main verifier\n+\t * pass later, liveness only has to avoid running off the masks.\n+\t */\n+\thi = min_t(s32, hi, FRAME_HALF_SPIS - 1);\n+\tif (lo \u003e hi)\n+\t\treturn 0;\n+\tfm = widen_frame_masks(instance, frame, BITS_TO_LONGS(hi + 1));\n+\tif (!fm)\n+\t\treturn -ENOMEM;\n+\tbitmap_set(rel_mask(fm, relative_idx(instance, insn_idx), kind), lo, hi - lo + 1);\n \treturn 0;\n }\n \n-static int mark_stack_write(struct func_instance *instance, u32 frame, u32 insn_idx, spis_t mask)\n+/* Accumulate may_read for half-slots [lo, hi] of @frame at @insn_idx */\n+static int mark_stack_read(struct func_instance *instance, u32 frame, u32 insn_idx,\n+\t\t\t s32 lo, s32 hi)\n+{\n+\treturn mark_stack_range(instance, frame, insn_idx, FM_MAY_READ, lo, hi);\n+}\n+\n+/* Accumulate must_write for half-slots [lo, hi] of @frame at @insn_idx */\n+static int mark_stack_write(struct func_instance *instance, u32 frame, u32 insn_idx,\n+\t\t\t s32 lo, s32 hi)\n {\n-\tstruct per_frame_masks *masks;\n+\treturn mark_stack_range(instance, frame, insn_idx, FM_MUST_WRITE, lo, hi);\n+}\n \n-\tmasks = alloc_frame_masks(instance, frame, insn_idx);\n-\tif (IS_ERR(masks))\n-\t\treturn PTR_ERR(masks);\n-\tmasks-\u003emust_write = spis_or(masks-\u003emust_write, mask);\n+/*\n+ * Mark every half-slot of @frame as possibly read by @insn_idx. This widens\n+ * the masks to the program's stack budget: a full read recorded at a narrower\n+ * width would leave the bits added by a later widening clear and lose part of\n+ * it, and an access past the budget is rejected by the main pass later, so no\n+ * widening of an accepted program goes further.\n+ */\n+static int mark_stack_read_all(struct bpf_verifier_env *env, struct func_instance *instance,\n+\t\t\t u32 frame, u32 insn_idx)\n+{\n+\treturn mark_stack_read(instance, frame, insn_idx, 0,\n+\t\t\t env-\u003estack_limit / BPF_HALF_REG_SIZE - 1);\n+}\n+\n+/* Accumulate @src, a mask @src_words wide, into may_read of @frame at @insn_idx */\n+static int mark_stack_read_mask(struct func_instance *instance, u32 frame, u32 insn_idx,\n+\t\t\t\tconst unsigned long *src, u32 src_words)\n+{\n+\tu32 nbits = src_words * BITS_PER_LONG;\n+\tstruct frame_masks *fm;\n+\tunsigned long *dst;\n+\tu32 last, w;\n+\n+\tlast = find_last_bit(src, nbits);\n+\tif (last == nbits)\n+\t\treturn 0;\n+\tfm = widen_frame_masks(instance, frame, BITS_TO_LONGS(last + 1));\n+\tif (!fm)\n+\t\treturn -ENOMEM;\n+\tdst = rel_mask(fm, relative_idx(instance, insn_idx), FM_MAY_READ);\n+\t/* @src has no bits set past @last, hence none past @fm-\u003ewords either */\n+\tsrc_words = min(src_words, fm-\u003ewords);\n+\tfor (w = 0; w \u003c src_words; w++)\n+\t\tdst[w] |= src[w];\n \treturn 0;\n }\n \n@@ -272,33 +373,42 @@ __diag_pop();\n static inline bool update_insn(struct bpf_verifier_env *env,\n \t\t\t struct func_instance *instance, u32 frame, u32 insn_idx)\n {\n-\tspis_t new_before, new_after;\n-\tstruct per_frame_masks *insn, *succ_insn;\n+\tunsigned long new_after[FRAME_MAX_WORDS] = {};\n+\tunsigned long *may_read, *must_write, *live_before;\n+\tstruct frame_masks *fm = instance-\u003eframes[frame];\n+\tu32 rel = relative_idx(instance, insn_idx);\n \tstruct bpf_iarray *succ;\n-\tu32 s;\n-\tbool changed;\n+\tbool changed = false;\n+\tu32 s, w;\n \n \tsucc = bpf_insn_successors(env, insn_idx);\n \tif (succ-\u003ecnt == 0)\n \t\treturn false;\n \n-\tchanged = false;\n-\tinsn = get_frame_masks(instance, frame, insn_idx);\n-\tnew_before = SPIS_ZERO;\n-\tnew_after = SPIS_ZERO;\n+\t/* All instructions of one frame array share the same mask width */\n \tfor (s = 0; s \u003c succ-\u003ecnt; ++s) {\n-\t\tsucc_insn = get_frame_masks(instance, frame, succ-\u003eitems[s]);\n-\t\tnew_after = spis_or(new_after, succ_insn-\u003elive_before);\n+\t\tunsigned long *succ_live;\n+\n+\t\tsucc_live = rel_mask(fm, relative_idx(instance, succ-\u003eitems[s]), FM_LIVE_BEFORE);\n+\t\tfor (w = 0; w \u003c fm-\u003ewords; w++)\n+\t\t\tnew_after[w] |= succ_live[w];\n \t}\n+\tmay_read = rel_mask(fm, rel, FM_MAY_READ);\n+\tmust_write = rel_mask(fm, rel, FM_MUST_WRITE);\n+\tlive_before = rel_mask(fm, rel, FM_LIVE_BEFORE);\n \t/*\n \t * New \"live_before\" is a union of all \"live_before\" of successors\n \t * minus slots written by instruction plus slots read by instruction.\n \t * new_before = (new_after \u0026 ~insn-\u003emust_write) | insn-\u003emay_read\n \t */\n-\tnew_before = spis_or(spis_and(new_after, spis_not(insn-\u003emust_write)),\n-\t\t\t insn-\u003emay_read);\n-\tchanged |= !spis_equal(new_before, insn-\u003elive_before);\n-\tinsn-\u003elive_before = new_before;\n+\tfor (w = 0; w \u003c fm-\u003ewords; w++) {\n+\t\tunsigned long new_before = (new_after[w] \u0026 ~must_write[w]) | may_read[w];\n+\n+\t\tif (new_before != live_before[w]) {\n+\t\t\tlive_before[w] = new_before;\n+\t\t\tchanged = true;\n+\t\t}\n+\t}\n \treturn changed;\n }\n \n@@ -329,10 +439,12 @@ static void update_instance(struct bpf_verifier_env *env, struct func_instance *\n \n static bool is_live_before(struct func_instance *instance, u32 insn_idx, u32 frameno, u32 half_spi)\n {\n-\tstruct per_frame_masks *masks;\n+\tstruct frame_masks *fm = instance-\u003eframes[frameno];\n \n-\tmasks = get_frame_masks(instance, frameno, insn_idx);\n-\treturn masks \u0026\u0026 spis_test_bit(masks-\u003elive_before, half_spi);\n+\t/* No recorded access reaches past the masks, so nothing there is live */\n+\tif (!fm || half_spi \u003e= frame_mask_bits(fm))\n+\t\treturn false;\n+\treturn test_bit(half_spi, rel_mask(fm, relative_idx(instance, insn_idx), FM_LIVE_BEFORE));\n }\n \n int bpf_live_stack_query_init(struct bpf_verifier_env *env, struct bpf_verifier_state *st)\n@@ -430,17 +542,19 @@ static int spi_off(int spi)\n * When only one half is set, print as \"-4h\",\"-8h\",...\n * Runs of 3+ consecutive fully-set SPIs are collapsed: \"fp0-8..-24\"\n */\n-static char *fmt_spis_mask(struct bpf_verifier_env *env, int frame, bool first, spis_t spis)\n+static char *fmt_spis_mask(struct bpf_verifier_env *env, int frame, bool first,\n+\t\t\t const unsigned long *spis, u32 words)\n {\n \tint buf_sz = sizeof(env-\u003etmp_str_buf);\n+\tint spi_cnt = words * BITS_PER_LONG / 2;\n \tchar *buf = env-\u003etmp_str_buf;\n \tint spi, n, run_start;\n \n \tbuf[0] = '\\0';\n \n-\tfor (spi = 0; spi \u003c STACK_SLOTS / 2 \u0026\u0026 buf_sz \u003e 0; spi++) {\n-\t\tbool lo = spis_test_bit(spis, spi * 2);\n-\t\tbool hi = spis_test_bit(spis, spi * 2 + 1);\n+\tfor (spi = 0; spi \u003c spi_cnt \u0026\u0026 buf_sz \u003e 0; spi++) {\n+\t\tbool lo = test_bit(spi * 2, spis);\n+\t\tbool hi = test_bit(spi * 2 + 1, spis);\n \t\tconst char *space = first ? \"\" : \" \";\n \n \t\tif (!lo \u0026\u0026 !hi)\n@@ -450,16 +564,16 @@ static char *fmt_spis_mask(struct bpf_verifier_env *env, int frame, bool first,\n \t\t\t/* half-spi */\n \t\t\tn = scnprintf(buf, buf_sz, \"%sfp%d%d%s\",\n \t\t\t\t space, frame, spi_off(spi) + (lo ? STACK_SLOT_SZ : 0), \"h\");\n-\t\t} else if (spi + 2 \u003c STACK_SLOTS / 2 \u0026\u0026\n-\t\t\t spis_test_bit(spis, spi * 2 + 2) \u0026\u0026\n-\t\t\t spis_test_bit(spis, spi * 2 + 3) \u0026\u0026\n-\t\t\t spis_test_bit(spis, spi * 2 + 4) \u0026\u0026\n-\t\t\t spis_test_bit(spis, spi * 2 + 5)) {\n+\t\t} else if (spi + 2 \u003c spi_cnt \u0026\u0026\n+\t\t\t test_bit(spi * 2 + 2, spis) \u0026\u0026\n+\t\t\t test_bit(spi * 2 + 3, spis) \u0026\u0026\n+\t\t\t test_bit(spi * 2 + 4, spis) \u0026\u0026\n+\t\t\t test_bit(spi * 2 + 5, spis)) {\n \t\t\t/* 3+ consecutive full spis */\n \t\t\trun_start = spi;\n-\t\t\twhile (spi + 1 \u003c STACK_SLOTS / 2 \u0026\u0026\n-\t\t\t spis_test_bit(spis, (spi + 1) * 2) \u0026\u0026\n-\t\t\t spis_test_bit(spis, (spi + 1) * 2 + 1))\n+\t\t\twhile (spi + 1 \u003c spi_cnt \u0026\u0026\n+\t\t\t test_bit((spi + 1) * 2, spis) \u0026\u0026\n+\t\t\t test_bit((spi + 1) * 2 + 1, spis))\n \t\t\t\tspi++;\n \t\t\tn = scnprintf(buf, buf_sz, \"%sfp%d%d..%d\",\n \t\t\t\t space, frame, spi_off(run_start), spi_off(spi));\n@@ -478,7 +592,8 @@ static void print_instance(struct bpf_verifier_env *env, struct func_instance *i\n {\n \tint start = env-\u003esubprog_info[instance-\u003esubprog].start;\n \tstruct bpf_insn *insns = env-\u003eprog-\u003einsnsi;\n-\tstruct per_frame_masks *masks;\n+\tstruct frame_masks *fm;\n+\tunsigned long *mask;\n \tint len = instance-\u003einsn_cnt;\n \tint insn_idx, frame, i;\n \tbool has_use, has_def;\n@@ -501,10 +616,13 @@ static void print_instance(struct bpf_verifier_env *env, struct func_instance *i\n \t\tpos = env-\u003elog.end_pos;\n \t\tverbose(env, \" use: \");\n \t\tfor (frame = instance-\u003edepth; frame \u003e= 0; --frame) {\n-\t\t\tmasks = get_frame_masks(instance, frame, insn_idx);\n-\t\t\tif (!masks || spis_is_zero(masks-\u003emay_read))\n+\t\t\tfm = instance-\u003eframes[frame];\n+\t\t\tif (!fm)\n+\t\t\t\tcontinue;\n+\t\t\tmask = rel_mask(fm, i, FM_MAY_READ);\n+\t\t\tif (bitmap_empty(mask, frame_mask_bits(fm)))\n \t\t\t\tcontinue;\n-\t\t\tverbose(env, \"%s\", fmt_spis_mask(env, frame, !has_use, masks-\u003emay_read));\n+\t\t\tverbose(env, \"%s\", fmt_spis_mask(env, frame, !has_use, mask, fm-\u003ewords));\n \t\t\thas_use = true;\n \t\t}\n \t\tif (!has_use)\n@@ -512,10 +630,13 @@ static void print_instance(struct bpf_verifier_env *env, struct func_instance *i\n \t\tpos = env-\u003elog.end_pos;\n \t\tverbose(env, \" def: \");\n \t\tfor (frame = instance-\u003edepth; frame \u003e= 0; --frame) {\n-\t\t\tmasks = get_frame_masks(instance, frame, insn_idx);\n-\t\t\tif (!masks || spis_is_zero(masks-\u003emust_write))\n+\t\t\tfm = instance-\u003eframes[frame];\n+\t\t\tif (!fm)\n+\t\t\t\tcontinue;\n+\t\t\tmask = rel_mask(fm, i, FM_MUST_WRITE);\n+\t\t\tif (bitmap_empty(mask, frame_mask_bits(fm)))\n \t\t\t\tcontinue;\n-\t\t\tverbose(env, \"%s\", fmt_spis_mask(env, frame, !has_def, masks-\u003emust_write));\n+\t\t\tverbose(env, \"%s\", fmt_spis_mask(env, frame, !has_def, mask, fm-\u003ewords));\n \t\t\thas_def = true;\n \t\t}\n \t\tif (!has_def)\n@@ -584,9 +705,9 @@ static int print_instances(struct bpf_verifier_env *env)\n * - same frame + different offset -\u003e offset-imprecise\n * - different frames -\u003e fully-imprecise (bitmask OR)\n *\n- * At memory access sites (LDX/STX/ST), offset-imprecise marks only\n- * the known frame's access mask as SPIS_ALL, while fully-imprecise\n- * iterates bits in the bitmask and routes each frame to its target.\n+ * At memory access sites (LDX/STX/ST), offset-imprecise marks the known\n+ * frame as fully read, while fully-imprecise iterates bits in the bitmask\n+ * and routes each frame to its target.\n */\n #define MAX_ARG_OFFSETS 4\n \n@@ -1235,7 +1356,6 @@ static int record_stack_access_off(struct func_instance *instance, s64 fp_off,\n \t\t\t\t s64 access_bytes, u32 frame, u32 insn_idx)\n {\n \ts32 slot_hi, slot_lo;\n-\tspis_t mask;\n \n \tif (fp_off \u003e= 0)\n \t\t/*\n@@ -1247,27 +1367,19 @@ static int record_stack_access_off(struct func_instance *instance, s64 fp_off,\n \tif (access_bytes == S64_MIN) {\n \t\t/* helper/kfunc read unknown amount of bytes from fp_off until fp+0 */\n \t\tslot_hi = (-fp_off - 1) / STACK_SLOT_SZ;\n-\t\tmask = SPIS_ZERO;\n-\t\tspis_or_range(\u0026mask, 0, slot_hi);\n-\t\treturn mark_stack_read(instance, frame, insn_idx, mask);\n+\t\treturn mark_stack_read(instance, frame, insn_idx, 0, slot_hi);\n \t}\n \tif (access_bytes \u003e 0) {\n \t\t/* Mark any touched slot as use */\n \t\tslot_hi = (-fp_off - 1) / STACK_SLOT_SZ;\n \t\tslot_lo = max_t(s32, (-fp_off - access_bytes) / STACK_SLOT_SZ, 0);\n-\t\tmask = SPIS_ZERO;\n-\t\tspis_or_range(\u0026mask, slot_lo, slot_hi);\n-\t\treturn mark_stack_read(instance, frame, insn_idx, mask);\n+\t\treturn mark_stack_read(instance, frame, insn_idx, slot_lo, slot_hi);\n \t} else if (access_bytes \u003c 0) {\n \t\t/* Mark only fully covered slots as def */\n \t\taccess_bytes = -access_bytes;\n \t\tslot_hi = (-fp_off) / STACK_SLOT_SZ - 1;\n \t\tslot_lo = max_t(s32, (-fp_off - access_bytes + STACK_SLOT_SZ - 1) / STACK_SLOT_SZ, 0);\n-\t\tif (slot_lo \u003c= slot_hi) {\n-\t\t\tmask = SPIS_ZERO;\n-\t\t\tspis_or_range(\u0026mask, slot_lo, slot_hi);\n-\t\t\treturn mark_stack_write(instance, frame, insn_idx, mask);\n-\t\t}\n+\t\treturn mark_stack_write(instance, frame, insn_idx, slot_lo, slot_hi);\n \t}\n \treturn 0;\n }\n@@ -1276,7 +1388,7 @@ static int record_stack_access_off(struct func_instance *instance, s64 fp_off,\n * 'arg' is FP-derived argument to helper/kfunc or load/store that\n * reads (positive) or writes (negative) 'access_bytes' into 'use' or 'def'.\n */\n-static int record_stack_access(struct func_instance *instance,\n+static int record_stack_access(struct bpf_verifier_env *env, struct func_instance *instance,\n \t\t\t const struct arg_track *arg,\n \t\t\t s64 access_bytes, u32 frame, u32 insn_idx)\n {\n@@ -1286,7 +1398,7 @@ static int record_stack_access(struct func_instance *instance,\n \t\treturn 0;\n \tif (arg-\u003eoff_cnt == 0) {\n \t\tif (access_bytes \u003e 0 || access_bytes == S64_MIN)\n-\t\t\treturn mark_stack_read(instance, frame, insn_idx, SPIS_ALL);\n+\t\t\treturn mark_stack_read_all(env, instance, frame, insn_idx);\n \t\treturn 0;\n \t}\n \tif (access_bytes != S64_MIN \u0026\u0026 access_bytes \u003c 0 \u0026\u0026 arg-\u003eoff_cnt != 1)\n@@ -1305,7 +1417,8 @@ static int record_stack_access(struct func_instance *instance,\n * When a pointer is ARG_IMPRECISE, conservatively mark every frame in\n * the bitmask as fully used.\n */\n-static int record_imprecise(struct func_instance *instance, u32 mask, u32 insn_idx)\n+static int record_imprecise(struct bpf_verifier_env *env, struct func_instance *instance,\n+\t\t\t u32 mask, u32 insn_idx)\n {\n \tint depth = instance-\u003edepth;\n \tint f, err;\n@@ -1314,7 +1427,7 @@ static int record_imprecise(struct func_instance *instance, u32 mask, u32 insn_i\n \t\tif (!(mask \u0026 1))\n \t\t\tcontinue;\n \t\tif (f \u003c= depth) {\n-\t\t\terr = mark_stack_read(instance, f, insn_idx, SPIS_ALL);\n+\t\t\terr = mark_stack_read_all(env, instance, f, insn_idx);\n \t\t\tif (err)\n \t\t\t\treturn err;\n \t\t}\n@@ -1383,9 +1496,9 @@ static int record_load_store_access(struct bpf_verifier_env *env,\n \t}\n \n \tif (ptr-\u003eframe \u003e= 0 \u0026\u0026 ptr-\u003eframe \u003c= depth)\n-\t\treturn record_stack_access(instance, ptr, sz, ptr-\u003eframe, insn_idx);\n+\t\treturn record_stack_access(env, instance, ptr, sz, ptr-\u003eframe, insn_idx);\n \tif (ptr-\u003eframe == ARG_IMPRECISE)\n-\t\treturn record_imprecise(instance, ptr-\u003emask, insn_idx);\n+\t\treturn record_imprecise(env, instance, ptr-\u003emask, insn_idx);\n \t/* ARG_NONE: not derived from any frame pointer, skip */\n \treturn 0;\n }\n@@ -1410,7 +1523,7 @@ static int record_arg_access(struct bpf_verifier_env *env,\n \t\tbytes = bpf_kfunc_stack_access_bytes(env, insn, arg_idx, insn_idx);\n \t} else {\n \t\tfor (int f = 0; f \u003c= depth; f++) {\n-\t\t\terr = mark_stack_read(instance, f, insn_idx, SPIS_ALL);\n+\t\t\terr = mark_stack_read_all(env, instance, f, insn_idx);\n \t\t\tif (err)\n \t\t\t\treturn err;\n \t\t}\n@@ -1420,9 +1533,9 @@ static int record_arg_access(struct bpf_verifier_env *env,\n \t\treturn 0;\n \n \tif (frame \u003e= 0 \u0026\u0026 frame \u003c= depth)\n-\t\terr = record_stack_access(instance, at, bytes, frame, insn_idx);\n+\t\terr = record_stack_access(env, instance, at, bytes, frame, insn_idx);\n \telse if (frame == ARG_IMPRECISE)\n-\t\terr = record_imprecise(instance, at-\u003emask, insn_idx);\n+\t\terr = record_imprecise(env, instance, at-\u003emask, insn_idx);\n \treturn err;\n }\n \n@@ -1772,36 +1885,52 @@ static bool has_fp_args(struct arg_track *args)\n * may_read: union (any pass might read the slot).\n * must_write: intersection (only slots written on ALL passes are guaranteed).\n * live_before is recomputed by a subsequent update_instance() on @dst.\n+ *\n+ * The two instances may have settled on different mask widths for the same\n+ * frame, so @dst is widened to cover @src first. A word only @dst has counts\n+ * as zero on the @src side: it unions into may_read as a no-op and intersects\n+ * must_write to empty.\n */\n-static void merge_instances(struct func_instance *dst, struct func_instance *src)\n+static int merge_instances(struct func_instance *dst, struct func_instance *src)\n {\n-\tint f, i;\n+\tstruct frame_masks *d, *s;\n+\tu32 f, i, w;\n \n \tfor (f = 0; f \u003c= dst-\u003edepth; f++) {\n-\t\tif (!src-\u003eframes[f]) {\n+\t\ts = src-\u003eframes[f];\n+\t\td = dst-\u003eframes[f];\n+\t\tif (!s) {\n \t\t\t/* This pass didn't touch frame f — must_write intersects with empty. */\n-\t\t\tif (dst-\u003eframes[f])\n+\t\t\tif (d)\n \t\t\t\tfor (i = 0; i \u003c dst-\u003einsn_cnt; i++)\n-\t\t\t\t\tdst-\u003eframes[f][i].must_write = SPIS_ZERO;\n+\t\t\t\t\tbitmap_zero(rel_mask(d, i, FM_MUST_WRITE),\n+\t\t\t\t\t\t frame_mask_bits(d));\n \t\t\tcontinue;\n \t\t}\n-\t\tif (!dst-\u003eframes[f]) {\n+\t\tif (!d) {\n \t\t\t/* Previous pass didn't touch frame f — take src, zero must_write. */\n-\t\t\tdst-\u003eframes[f] = src-\u003eframes[f];\n+\t\t\tdst-\u003eframes[f] = s;\n \t\t\tsrc-\u003eframes[f] = NULL;\n \t\t\tfor (i = 0; i \u003c dst-\u003einsn_cnt; i++)\n-\t\t\t\tdst-\u003eframes[f][i].must_write = SPIS_ZERO;\n+\t\t\t\tbitmap_zero(rel_mask(s, i, FM_MUST_WRITE), frame_mask_bits(s));\n \t\t\tcontinue;\n \t\t}\n+\t\td = widen_frame_masks(dst, f, s-\u003ewords);\n+\t\tif (!d)\n+\t\t\treturn -ENOMEM;\n \t\tfor (i = 0; i \u003c dst-\u003einsn_cnt; i++) {\n-\t\t\tdst-\u003eframes[f][i].may_read =\n-\t\t\t\tspis_or(dst-\u003eframes[f][i].may_read,\n-\t\t\t\t\tsrc-\u003eframes[f][i].may_read);\n-\t\t\tdst-\u003eframes[f][i].must_write =\n-\t\t\t\tspis_and(dst-\u003eframes[f][i].must_write,\n-\t\t\t\t\t src-\u003eframes[f][i].must_write);\n+\t\t\tunsigned long *dst_read = rel_mask(d, i, FM_MAY_READ);\n+\t\t\tunsigned long *dst_write = rel_mask(d, i, FM_MUST_WRITE);\n+\t\t\tunsigned long *src_read = rel_mask(s, i, FM_MAY_READ);\n+\t\t\tunsigned long *src_write = rel_mask(s, i, FM_MUST_WRITE);\n+\n+\t\t\tfor (w = 0; w \u003c d-\u003ewords; w++) {\n+\t\t\t\tdst_read[w] |= w \u003c s-\u003ewords ? src_read[w] : 0;\n+\t\t\t\tdst_write[w] \u0026= w \u003c s-\u003ewords ? src_write[w] : 0;\n+\t\t\t}\n \t\t}\n \t}\n+\treturn 0;\n }\n \n static struct func_instance *fresh_instance(struct func_instance *src)\n@@ -1916,7 +2045,7 @@ static int analyze_subprog(struct bpf_verifier_env *env,\n \t\t\t\tif (info[subprog].at_in[j][caller_reg].frame == ARG_NONE)\n \t\t\t\t\tcontinue;\n \t\t\t\tfor (int f = 0; f \u003c= depth; f++) {\n-\t\t\t\t\terr = mark_stack_read(instance, f, idx, SPIS_ALL);\n+\t\t\t\t\terr = mark_stack_read_all(env, instance, f, idx);\n \t\t\t\t\tif (err)\n \t\t\t\t\t\tgoto out_free;\n \t\t\t\t}\n@@ -1955,13 +2084,18 @@ static int analyze_subprog(struct bpf_verifier_env *env,\n \t\t/* Pull callee's entry liveness back to caller's callsite */\n \t\t{\n \t\t\tu32 callee_start = callee_instance-\u003esubprog_start;\n-\t\t\tstruct per_frame_masks *entry;\n+\t\t\tstruct frame_masks *callee_fm;\n \n \t\t\tfor (int f = 0; f \u003c callee_instance-\u003edepth; f++) {\n-\t\t\t\tentry = get_frame_masks(callee_instance, f, callee_start);\n-\t\t\t\tif (!entry)\n+\t\t\t\tcallee_fm = callee_instance-\u003eframes[f];\n+\t\t\t\tif (!callee_fm)\n \t\t\t\t\tcontinue;\n-\t\t\t\terr = mark_stack_read(instance, f, idx, entry-\u003elive_before);\n+\t\t\t\terr = mark_stack_read_mask(instance, f, idx,\n+\t\t\t\t\t\t\t rel_mask(callee_fm,\n+\t\t\t\t\t\t\t\t relative_idx(callee_instance,\n+\t\t\t\t\t\t\t\t\t\t callee_start),\n+\t\t\t\t\t\t\t\t FM_LIVE_BEFORE),\n+\t\t\t\t\t\t\t callee_fm-\u003ewords);\n \t\t\t\tif (err)\n \t\t\t\t\tgoto out_free;\n \t\t\t}\n@@ -1969,9 +2103,11 @@ static int analyze_subprog(struct bpf_verifier_env *env,\n \t}\n \n \tif (prev_instance) {\n-\t\tmerge_instances(prev_instance, instance);\n+\t\terr = merge_instances(prev_instance, instance);\n \t\tfree_instance(instance);\n \t\tinstance = prev_instance;\n+\t\tif (err)\n+\t\t\treturn err;\n \t}\n \tupdate_instance(env, instance);\n \treturn 0;\ndiff --git a/kernel/bpf/log.c b/kernel/bpf/log.c\nindex fb032dfdc0dee..d850a7863d2ed 100644\n--- a/kernel/bpf/log.c\n+++ b/kernel/bpf/log.c\n@@ -716,7 +716,8 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifie\n \t\tverbose(env, \"=\");\n \t\tprint_reg_state(env, state, reg);\n \t}\n-\tfor (i = 0; i \u003c state-\u003eallocated_stack / BPF_REG_SIZE; i++) {\n+\tfor (i = 0; i \u003c bpf_stack_nr_slots(state); i++) {\n+\t\tstruct bpf_stack_state *slot = bpf_stack_slot(state, i);\n \t\tchar types_buf[BPF_REG_SIZE + 1];\n \t\tconst char *sep = \"\";\n \t\tbool valid = false;\n@@ -727,7 +728,7 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifie\n \t\t\tcontinue;\n \n \t\tfor (j = 0; j \u003c BPF_REG_SIZE; j++) {\n-\t\t\tslot_type = state-\u003estack[i].slot_type[j];\n+\t\t\tslot_type = slot-\u003eslot_type[j];\n \t\t\tif (slot_type != STACK_INVALID \u0026\u0026 slot_type != STACK_POISON)\n \t\t\t\tvalid = true;\n \t\t\ttypes_buf[j] = slot_type_char[slot_type];\n@@ -736,12 +737,12 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifie\n \t\tif (!valid)\n \t\t\tcontinue;\n \n-\t\treg = \u0026state-\u003estack[i].spilled_ptr;\n-\t\tswitch (state-\u003estack[i].slot_type[BPF_REG_SIZE - 1]) {\n+\t\treg = \u0026slot-\u003espilled_ptr;\n+\t\tswitch (slot-\u003eslot_type[BPF_REG_SIZE - 1]) {\n \t\tcase STACK_SPILL:\n \t\t\t/* print MISC/ZERO/INVALID slots above subreg spill */\n \t\t\tfor (j = 0; j \u003c BPF_REG_SIZE; j++)\n-\t\t\t\tif (state-\u003estack[i].slot_type[j] == STACK_SPILL)\n+\t\t\t\tif (slot-\u003eslot_type[j] == STACK_SPILL)\n \t\t\t\t\tbreak;\n \t\t\ttypes_buf[j] = '\\0';\n \n@@ -751,7 +752,7 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifie\n \t\tcase STACK_DYNPTR:\n \t\t\t/* skip to main dynptr slot */\n \t\t\ti += BPF_DYNPTR_NR_SLOTS - 1;\n-\t\t\treg = \u0026state-\u003estack[i].spilled_ptr;\n+\t\t\treg = \u0026bpf_stack_slot(state, i)-\u003espilled_ptr;\n \n \t\t\tverbose(env, \" fp%d\", (-i - 1) * BPF_REG_SIZE);\n \t\t\tverbose(env, \"=dynptr_%s(\", dynptr_type_str(reg-\u003edynptr.type));\ndiff --git a/kernel/bpf/states.c b/kernel/bpf/states.c\nindex 66fb11b6c6a76..5078e4832c6e1 100644\n--- a/kernel/bpf/states.c\n+++ b/kernel/bpf/states.c\n@@ -335,21 +335,18 @@ static bool check_ids(u32 old_id, u32 cur_id, struct bpf_idmap *idmap)\n \t\t\treturn false;\n \t}\n \n-\t/* Reached the end of known mappings; haven't seen this id before */\n-\tif (idmap-\u003ecnt \u003c BPF_ID_MAP_SIZE) {\n-\t\tmap[idmap-\u003ecnt].old = old_id;\n-\t\tmap[idmap-\u003ecnt].cur = cur_id;\n-\t\tidmap-\u003ecnt++;\n-\t\treturn true;\n-\t}\n-\n \t/*\n-\t * idmap slots are bounded by the number of registers and stack slots.\n-\t * Since referenced dynptrs acquire intermediate references that do\n-\t * not live in either, so the map can be exhausted. Since it is unlikely,\n-\t * fail the verification by treating the states as not equivalent.\n+\t * Reached the end of known mappings; haven't seen this id before. If\n+\t * the map cannot grow, treat the states as not equivalent, which only\n+\t * costs pruning.\n \t */\n-\treturn false;\n+\tif (!bpf_id_scratch_reserve((void **)\u0026idmap-\u003emap, \u0026idmap-\u003ecap, idmap-\u003ecnt, sizeof(*map)))\n+\t\treturn false;\n+\tmap = idmap-\u003emap;\n+\tmap[idmap-\u003ecnt].old = old_id;\n+\tmap[idmap-\u003ecnt].cur = cur_id;\n+\tidmap-\u003ecnt++;\n+\treturn true;\n }\n \n /*\n@@ -415,14 +412,14 @@ static void __clean_func_state(struct bpf_verifier_env *env,\n \t * half_spi 2*i → lower half: slot_type[0..3] (closer to FP)\n \t * half_spi 2*i+1 → upper half: slot_type[4..7] (farther from FP)\n \t */\n-\tfor (i = 0; i \u003c st-\u003eallocated_stack / BPF_REG_SIZE; i++) {\n+\tfor (i = 0; i \u003c bpf_stack_nr_slots(st); i++) {\n \t\tbool lo_live = bpf_stack_slot_alive(env, frame, i * 2);\n \t\tbool hi_live = bpf_stack_slot_alive(env, frame, i * 2 + 1);\n \n \t\tif (!hi_live || !lo_live) {\n \t\t\tint start = !lo_live ? 0 : BPF_REG_SIZE / 2;\n \t\t\tint end = !hi_live ? BPF_REG_SIZE : BPF_REG_SIZE / 2;\n-\t\t\tu8 stype = st-\u003estack[i].slot_type[7];\n+\t\t\tu8 stype = bpf_stack_slot(st, i)-\u003eslot_type[7];\n \n \t\t\t/*\n \t\t\t * Don't clear special slots.\n@@ -442,7 +439,7 @@ static void __clean_func_state(struct bpf_verifier_env *env,\n \t\t\t * rejecting as non-scalar register fills.\n \t\t\t */\n \t\t\tif (!hi_live) {\n-\t\t\t\tstruct bpf_reg_state *spill = \u0026st-\u003estack[i].spilled_ptr;\n+\t\t\t\tstruct bpf_reg_state *spill = \u0026bpf_stack_slot(st, i)-\u003espilled_ptr;\n \n \t\t\t\tif (lo_live \u0026\u0026 stype == STACK_SPILL) {\n \t\t\t\t\tif (spill-\u003etype != SCALAR_VALUE)\n@@ -454,7 +451,7 @@ static void __clean_func_state(struct bpf_verifier_env *env,\n \t\t\t\t\tif (bpf_register_is_null(spill))\n \t\t\t\t\t\tcontinue;\n \t\t\t\t\tfor (j = 0; j \u003c 4; j++) {\n-\t\t\t\t\t\tu8 *t = \u0026st-\u003estack[i].slot_type[j];\n+\t\t\t\t\t\tu8 *t = \u0026bpf_stack_slot(st, i)-\u003eslot_type[j];\n \n \t\t\t\t\t\tif (*t == STACK_SPILL)\n \t\t\t\t\t\t\t*t = STACK_MISC;\n@@ -463,7 +460,7 @@ static void __clean_func_state(struct bpf_verifier_env *env,\n \t\t\t\tbpf_mark_reg_not_init(env, spill);\n \t\t\t}\n \t\t\tfor (j = start; j \u003c end; j++)\n-\t\t\t\tst-\u003estack[i].slot_type[j] = STACK_POISON;\n+\t\t\t\tbpf_stack_slot(st, i)-\u003eslot_type[j] = STACK_POISON;\n \t\t}\n \t}\n }\n@@ -707,37 +704,38 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,\n \t * didn't use them\n \t */\n \tfor (i = 0; i \u003c old-\u003eallocated_stack; i++) {\n+\t\tstruct bpf_stack_state *old_slot, *cur_slot;\n \t\tstruct bpf_reg_state *old_reg, *cur_reg;\n \t\tint im = i % BPF_REG_SIZE;\n+\t\tu8 old_type;\n \n \t\tspi = i / BPF_REG_SIZE;\n+\t\told_slot = bpf_stack_slot(old, spi);\n+\t\told_type = old_slot-\u003eslot_type[im];\n+\t\tcur_slot = i \u003c cur-\u003eallocated_stack ? bpf_stack_slot(cur, spi) : NULL;\n \n \t\tif (exact == EXACT) {\n-\t\t\tu8 old_type = old-\u003estack[spi].slot_type[i % BPF_REG_SIZE];\n-\t\t\tu8 cur_type = i \u003c cur-\u003eallocated_stack ?\n-\t\t\t\t cur-\u003estack[spi].slot_type[i % BPF_REG_SIZE] : STACK_INVALID;\n+\t\t\tu8 cur_type = cur_slot ? cur_slot-\u003eslot_type[im] : STACK_INVALID;\n \n \t\t\t/* STACK_INVALID and STACK_POISON are equivalent for pruning */\n \t\t\tif (old_type == STACK_POISON)\n \t\t\t\told_type = STACK_INVALID;\n \t\t\tif (cur_type == STACK_POISON)\n \t\t\t\tcur_type = STACK_INVALID;\n-\t\t\tif (i \u003e= cur-\u003eallocated_stack || old_type != cur_type)\n+\t\t\tif (!cur_slot || old_type != cur_type)\n \t\t\t\treturn false;\n \t\t}\n \n-\t\tif (old-\u003estack[spi].slot_type[i % BPF_REG_SIZE] == STACK_INVALID ||\n-\t\t old-\u003estack[spi].slot_type[i % BPF_REG_SIZE] == STACK_POISON)\n+\t\tif (old_type == STACK_INVALID || old_type == STACK_POISON)\n \t\t\tcontinue;\n \n-\t\tif (env-\u003eallow_uninit_stack \u0026\u0026\n-\t\t old-\u003estack[spi].slot_type[i % BPF_REG_SIZE] == STACK_MISC)\n+\t\tif (env-\u003eallow_uninit_stack \u0026\u0026 old_type == STACK_MISC)\n \t\t\tcontinue;\n \n \t\t/* explored stack has more populated slots than current stack\n \t\t * and these slots were used\n \t\t */\n-\t\tif (i \u003e= cur-\u003eallocated_stack)\n+\t\tif (!cur_slot)\n \t\t\treturn false;\n \n \t\t/*\n@@ -747,8 +745,8 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,\n \t\t * regsafe() to ensure scalar ids are compared.\n \t\t */\n \t\tif (im == 0 || im == 4) {\n-\t\t\told_reg = scalar_reg_for_stack(env, \u0026old-\u003estack[spi], im);\n-\t\t\tcur_reg = scalar_reg_for_stack(env, \u0026cur-\u003estack[spi], im);\n+\t\t\told_reg = scalar_reg_for_stack(env, old_slot, im);\n+\t\t\tcur_reg = scalar_reg_for_stack(env, cur_slot, im);\n \t\t\tif (old_reg \u0026\u0026 cur_reg) {\n \t\t\t\tif (!regsafe(env, old_reg, cur_reg, idmap, exact))\n \t\t\t\t\treturn false;\n@@ -761,21 +759,19 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,\n \t\t * it will be safe with zero-initialized stack.\n \t\t * The opposite is not true\n \t\t */\n-\t\tif (old-\u003estack[spi].slot_type[i % BPF_REG_SIZE] == STACK_MISC \u0026\u0026\n-\t\t cur-\u003estack[spi].slot_type[i % BPF_REG_SIZE] == STACK_ZERO)\n+\t\tif (old_type == STACK_MISC \u0026\u0026 cur_slot-\u003eslot_type[im] == STACK_ZERO)\n \t\t\tcontinue;\n-\t\tif (old-\u003estack[spi].slot_type[i % BPF_REG_SIZE] !=\n-\t\t cur-\u003estack[spi].slot_type[i % BPF_REG_SIZE])\n+\t\tif (old_type != cur_slot-\u003eslot_type[im])\n \t\t\t/* Ex: old explored (safe) state has STACK_SPILL in\n \t\t\t * this stack slot, but current has STACK_MISC -\u003e\n \t\t\t * this verifier states are not equivalent,\n \t\t\t * return false to continue verification of this path\n \t\t\t */\n \t\t\treturn false;\n-\t\tif (i % BPF_REG_SIZE != BPF_REG_SIZE - 1)\n+\t\tif (im != BPF_REG_SIZE - 1)\n \t\t\tcontinue;\n \t\t/* Both old and cur are having same slot_type */\n-\t\tswitch (old-\u003estack[spi].slot_type[BPF_REG_SIZE - 1]) {\n+\t\tswitch (old_type) {\n \t\tcase STACK_SPILL:\n \t\t\t/* when explored and current stack slot are both storing\n \t\t\t * spilled registers, check that stored pointers types\n@@ -787,13 +783,13 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,\n \t\t\t * such verifier states are not equivalent.\n \t\t\t * return false to continue verification of this path\n \t\t\t */\n-\t\t\tif (!regsafe(env, \u0026old-\u003estack[spi].spilled_ptr,\n-\t\t\t\t \u0026cur-\u003estack[spi].spilled_ptr, idmap, exact))\n+\t\t\tif (!regsafe(env, \u0026old_slot-\u003espilled_ptr, \u0026cur_slot-\u003espilled_ptr,\n+\t\t\t\t idmap, exact))\n \t\t\t\treturn false;\n \t\t\tbreak;\n \t\tcase STACK_DYNPTR:\n-\t\t\told_reg = \u0026old-\u003estack[spi].spilled_ptr;\n-\t\t\tcur_reg = \u0026cur-\u003estack[spi].spilled_ptr;\n+\t\t\told_reg = \u0026old_slot-\u003espilled_ptr;\n+\t\t\tcur_reg = \u0026cur_slot-\u003espilled_ptr;\n \t\t\tif (old_reg-\u003edynptr.type != cur_reg-\u003edynptr.type ||\n \t\t\t old_reg-\u003edynptr.first_slot != cur_reg-\u003edynptr.first_slot ||\n \t\t\t !check_ids(old_reg-\u003eid, cur_reg-\u003eid, idmap) ||\n@@ -801,8 +797,8 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,\n \t\t\t\treturn false;\n \t\t\tbreak;\n \t\tcase STACK_ITER:\n-\t\t\told_reg = \u0026old-\u003estack[spi].spilled_ptr;\n-\t\t\tcur_reg = \u0026cur-\u003estack[spi].spilled_ptr;\n+\t\t\told_reg = \u0026old_slot-\u003espilled_ptr;\n+\t\t\tcur_reg = \u0026cur_slot-\u003espilled_ptr;\n \t\t\t/* iter.depth is not compared between states as it\n \t\t\t * doesn't matter for correctness and would otherwise\n \t\t\t * prevent convergence; we maintain it only to prevent\n@@ -818,8 +814,8 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,\n \t\t\t\treturn false;\n \t\t\tbreak;\n \t\tcase STACK_IRQ_FLAG:\n-\t\t\told_reg = \u0026old-\u003estack[spi].spilled_ptr;\n-\t\t\tcur_reg = \u0026cur-\u003estack[spi].spilled_ptr;\n+\t\t\told_reg = \u0026old_slot-\u003espilled_ptr;\n+\t\t\tcur_reg = \u0026cur_slot-\u003espilled_ptr;\n \t\t\tif (!check_ids(old_reg-\u003eid, cur_reg-\u003eid, idmap) ||\n \t\t\t old_reg-\u003eirq.kfunc_class != cur_reg-\u003eirq.kfunc_class)\n \t\t\t\treturn false;\n@@ -966,6 +962,27 @@ static bool func_states_equal(struct bpf_verifier_env *env, struct bpf_func_stat\n \treturn true;\n }\n \n+/*\n+ * Make room for one more entry in an id scratch array, doubling it as needed.\n+ * Returns false if it could not grow; callers then treat the id as unknown\n+ * or the states as different, which is always safe.\n+ */\n+bool bpf_id_scratch_reserve(void **arr, u32 *cap, u32 cnt, size_t elem_size)\n+{\n+\tu32 new_cap;\n+\tvoid *p;\n+\n+\tif (cnt \u003c *cap)\n+\t\treturn true;\n+\tnew_cap = *cap ? *cap * 2 : 64;\n+\tp = krealloc_array(*arr, new_cap, elem_size, GFP_KERNEL_ACCOUNT | __GFP_NOWARN);\n+\tif (!p)\n+\t\treturn false;\n+\t*arr = p;\n+\t*cap = new_cap;\n+\treturn true;\n+}\n+\n static void reset_idmap_scratch(struct bpf_verifier_env *env)\n {\n \tstruct bpf_idmap *idmap = \u0026env-\u003eidmap_scratch;\n@@ -1043,10 +1060,10 @@ static int propagate_precision(struct bpf_verifier_env *env,\n \t\t\tfirst = false;\n \t\t}\n \n-\t\tfor (i = 0; i \u003c state-\u003eallocated_stack / BPF_REG_SIZE; i++) {\n-\t\t\tif (!bpf_is_spilled_reg(\u0026state-\u003estack[i]))\n+\t\tfor (i = 0; i \u003c bpf_stack_nr_slots(state); i++) {\n+\t\t\tif (!bpf_is_spilled_reg(bpf_stack_slot(state, i)))\n \t\t\t\tcontinue;\n-\t\t\tstate_reg = \u0026state-\u003estack[i].spilled_ptr;\n+\t\t\tstate_reg = \u0026bpf_stack_slot(state, i)-\u003espilled_ptr;\n \t\t\tif (state_reg-\u003etype != SCALAR_VALUE ||\n \t\t\t !state_reg-\u003eprecise)\n \t\t\t\tcontinue;\n@@ -1192,15 +1209,15 @@ static bool iter_active_depths_differ(struct bpf_verifier_state *old, struct bpf\n \n \tfor (fr = old-\u003ecurframe; fr \u003e= 0; fr--) {\n \t\tstate = old-\u003eframe[fr];\n-\t\tfor (i = 0; i \u003c state-\u003eallocated_stack / BPF_REG_SIZE; i++) {\n-\t\t\tif (state-\u003estack[i].slot_type[0] != STACK_ITER)\n+\t\tfor (i = 0; i \u003c bpf_stack_nr_slots(state); i++) {\n+\t\t\tif (bpf_stack_slot(state, i)-\u003eslot_type[0] != STACK_ITER)\n \t\t\t\tcontinue;\n \n-\t\t\tslot = \u0026state-\u003estack[i].spilled_ptr;\n+\t\t\tslot = \u0026bpf_stack_slot(state, i)-\u003espilled_ptr;\n \t\t\tif (slot-\u003eiter.state != BPF_ITER_STATE_ACTIVE)\n \t\t\t\tcontinue;\n \n-\t\t\tcur_slot = \u0026cur-\u003eframe[fr]-\u003estack[i].spilled_ptr;\n+\t\t\tcur_slot = \u0026bpf_stack_slot(cur-\u003eframe[fr], i)-\u003espilled_ptr;\n \t\t\tif (cur_slot-\u003eiter.depth != slot-\u003eiter.depth)\n \t\t\t\treturn true;\n \t\t}\n@@ -1222,10 +1239,10 @@ static void mark_all_scalars_imprecise(struct bpf_verifier_env *env, struct bpf_\n \t\t\t\tcontinue;\n \t\t\treg-\u003eprecise = false;\n \t\t}\n-\t\tfor (j = 0; j \u003c func-\u003eallocated_stack / BPF_REG_SIZE; j++) {\n-\t\t\tif (!bpf_is_spilled_reg(\u0026func-\u003estack[j]))\n+\t\tfor (j = 0; j \u003c bpf_stack_nr_slots(func); j++) {\n+\t\t\tif (!bpf_is_spilled_reg(bpf_stack_slot(func, j)))\n \t\t\t\tcontinue;\n-\t\t\treg = \u0026func-\u003estack[j].spilled_ptr;\n+\t\t\treg = \u0026bpf_stack_slot(func, j)-\u003espilled_ptr;\n \t\t\tif (reg-\u003etype != SCALAR_VALUE)\n \t\t\t\tcontinue;\n \t\t\treg-\u003eprecise = false;\n@@ -1328,7 +1345,7 @@ int bpf_is_state_visited(struct bpf_verifier_env *env, int insn_idx)\n \t\t\t */\n \t\t\tif (is_iter_next_insn(env, insn_idx)) {\n \t\t\t\tif (states_equal(env, \u0026sl-\u003estate, cur, RANGE_WITHIN)) {\n-\t\t\t\t\tstruct bpf_func_state *cur_frame;\n+\t\t\t\t\tstruct bpf_func_state *cur_frame, *iter_frame;\n \t\t\t\t\tstruct bpf_reg_state *iter_state, *iter_reg;\n \t\t\t\t\tint spi;\n \n@@ -1342,7 +1359,8 @@ int bpf_is_state_visited(struct bpf_verifier_env *env, int insn_idx)\n \t\t\t\t\t * no need for extra (re-)validations\n \t\t\t\t\t */\n \t\t\t\t\tspi = bpf_get_spi(iter_reg-\u003evar_off.value);\n-\t\t\t\t\titer_state = \u0026bpf_func(env, iter_reg)-\u003estack[spi].spilled_ptr;\n+\t\t\t\t\titer_frame = bpf_func(env, iter_reg);\n+\t\t\t\t\titer_state = \u0026bpf_stack_slot(iter_frame, spi)-\u003espilled_ptr;\n \t\t\t\t\tif (iter_state-\u003eiter.state == BPF_ITER_STATE_ACTIVE) {\n \t\t\t\t\t\tloop = true;\n \t\t\t\t\t\tgoto hit;\n@@ -1410,7 +1428,7 @@ int bpf_is_state_visited(struct bpf_verifier_env *env, int insn_idx)\n \t\t\t */\n \t\t\terr = 0;\n \t\t\tif (bpf_is_jmp_point(env, env-\u003einsn_idx))\n-\t\t\t\terr = bpf_push_jmp_history(env, cur, 0, 0, 0, 0);\n+\t\t\t\terr = bpf_push_jmp_history(env, cur, 0, 0, 0, NULL, 0);\n \t\t\terr = err ? : propagate_precision(env, \u0026sl-\u003estate, cur, NULL);\n \t\t\tif (err)\n \t\t\t\treturn err;\ndiff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c\nindex a7c9e2d8965d5..9bd7f9b3a67cb 100644\n--- a/kernel/bpf/verifier.c\n+++ b/kernel/bpf/verifier.c\n@@ -598,16 +598,17 @@ bool bpf_is_may_goto_insn(struct bpf_insn *insn)\n \n static bool is_spi_bounds_valid(struct bpf_func_state *state, int spi, int nr_slots)\n {\n- int allocated_slots = state-\u003eallocated_stack / BPF_REG_SIZE;\n+\tint allocated_slots = bpf_stack_nr_slots(state);\n \n- /* We need to check that slots between [spi - nr_slots + 1, spi] are\n-\t* within [0, allocated_stack).\n-\t*\n-\t* Please note that the spi grows downwards. For example, a dynptr\n-\t* takes the size of two stack slots; the first slot will be at\n-\t* spi and the second slot will be at spi - 1.\n-\t*/\n- return spi - nr_slots + 1 \u003e= 0 \u0026\u0026 spi \u003c allocated_slots;\n+\t/*\n+\t * We need to check that slots between [spi - nr_slots + 1, spi] are\n+\t * within [0, allocated_stack).\n+\t *\n+\t * Please note that the spi grows downwards. For example, a dynptr\n+\t * takes the size of two stack slots; the first slot will be at\n+\t * spi and the second slot will be at spi - 1.\n+\t */\n+\treturn spi - nr_slots + 1 \u003e= 0 \u0026\u0026 spi \u003c allocated_slots;\n }\n \n static int stack_slot_obj_get_spi(struct bpf_verifier_env *env, struct bpf_reg_state *reg,\n@@ -751,8 +752,8 @@ static int mark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_\n \t\treturn err;\n \n \tfor (i = 0; i \u003c BPF_REG_SIZE; i++) {\n-\t\tstate-\u003estack[spi].slot_type[i] = STACK_DYNPTR;\n-\t\tstate-\u003estack[spi - 1].slot_type[i] = STACK_DYNPTR;\n+\t\tbpf_stack_slot(state, spi)-\u003eslot_type[i] = STACK_DYNPTR;\n+\t\tbpf_stack_slot(state, spi - 1)-\u003eslot_type[i] = STACK_DYNPTR;\n \t}\n \n \ttype = arg_to_dynptr_type(arg_type);\n@@ -785,8 +786,8 @@ static int mark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_\n \t\tparent_id = dynptr-\u003eparent_id;\n \t}\n \n-\tmark_dynptr_stack_regs(env, \u0026state-\u003estack[spi].spilled_ptr,\n-\t\t\t \u0026state-\u003estack[spi - 1].spilled_ptr, type, parent_id);\n+\tmark_dynptr_stack_regs(env, \u0026bpf_stack_slot(state, spi)-\u003espilled_ptr,\n+\t\t\t \u0026bpf_stack_slot(state, spi - 1)-\u003espilled_ptr, type, parent_id);\n \n \treturn 0;\n }\n@@ -818,7 +819,7 @@ static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_re\n \t * all clones and derived slices. For non-referenced dynptr, only\n \t * the dynptr and slices derived from it will be invalidated.\n \t */\n-\treg = \u0026state-\u003estack[spi].spilled_ptr;\n+\treg = \u0026bpf_stack_slot(state, spi)-\u003espilled_ptr;\n \treturn release_reference(env, dynptr_type_referenced(reg-\u003edynptr.type)\n \t\t\t\t ? reg-\u003eparent_id\n \t\t\t\t : reg-\u003eid);\n@@ -857,6 +858,7 @@ static int dynptr_ref_cnt(struct bpf_verifier_env *env, int v_parent_id)\n static int destroy_if_dynptr_stack_slot(struct bpf_verifier_env *env,\n \t\t\t\t struct bpf_func_state *state, int spi)\n {\n+\tstruct bpf_stack_state *slot = bpf_stack_slot(state, spi);\n \tint err = 0;\n \n \t/* We always ensure that STACK_DYNPTR is never set partially,\n@@ -864,20 +866,22 @@ static int destroy_if_dynptr_stack_slot(struct bpf_verifier_env *env,\n \t * different for STACK_SPILL, where it may be only set for\n \t * 1 byte, so code has to use is_spilled_reg.\n \t */\n-\tif (state-\u003estack[spi].slot_type[0] != STACK_DYNPTR)\n+\tif (slot-\u003eslot_type[0] != STACK_DYNPTR)\n \t\treturn 0;\n \n \t/* Reposition spi to first slot */\n-\tif (!state-\u003estack[spi].spilled_ptr.dynptr.first_slot)\n+\tif (!slot-\u003espilled_ptr.dynptr.first_slot) {\n \t\tspi = spi + 1;\n+\t\tslot = bpf_stack_slot(state, spi);\n+\t}\n \n \t/*\n \t * A referenced dynptr can be overwritten only if there is at\n \t * least one other dynptr sharing the same virtual ref parent,\n \t * ensuring the reference can still be properly released.\n \t */\n-\tif (dynptr_type_referenced(state-\u003estack[spi].spilled_ptr.dynptr.type) \u0026\u0026\n-\t dynptr_ref_cnt(env, state-\u003estack[spi].spilled_ptr.parent_id) \u003c= 1) {\n+\tif (dynptr_type_referenced(slot-\u003espilled_ptr.dynptr.type) \u0026\u0026\n+\t dynptr_ref_cnt(env, slot-\u003espilled_ptr.parent_id) \u003c= 1) {\n \t\tverbose(env, \"cannot overwrite referenced dynptr\\n\");\n \t\tbpf_diag_res(\n \t\t\tenv, env-\u003einsn_idx, \"referenced dynptr overwrite\",\n@@ -887,7 +891,7 @@ static int destroy_if_dynptr_stack_slot(struct bpf_verifier_env *env,\n \t}\n \n \t/* Invalidate the dynptr and any derived slices */\n-\terr = release_reference(env, state-\u003estack[spi].spilled_ptr.id);\n+\terr = release_reference(env, slot-\u003espilled_ptr.id);\n \tif (!err) {\n \t\tmark_stack_slot_scratched(env, spi);\n \t\tmark_stack_slot_scratched(env, spi - 1);\n@@ -927,6 +931,7 @@ static bool is_dynptr_reg_valid_uninit(struct bpf_verifier_env *env, struct bpf_\n static bool is_dynptr_reg_valid_init(struct bpf_verifier_env *env, struct bpf_reg_state *reg)\n {\n \tstruct bpf_func_state *state = bpf_func(env, reg);\n+\tstruct bpf_stack_state *slot;\n \tint i, spi;\n \n \t/* This already represents first slot of initialized bpf_dynptr.\n@@ -941,12 +946,13 @@ static bool is_dynptr_reg_valid_init(struct bpf_verifier_env *env, struct bpf_re\n \tspi = dynptr_get_spi(env, reg);\n \tif (spi \u003c 0)\n \t\treturn false;\n-\tif (!state-\u003estack[spi].spilled_ptr.dynptr.first_slot)\n+\tslot = bpf_stack_slot(state, spi);\n+\tif (!slot-\u003espilled_ptr.dynptr.first_slot)\n \t\treturn false;\n \n \tfor (i = 0; i \u003c BPF_REG_SIZE; i++) {\n-\t\tif (state-\u003estack[spi].slot_type[i] != STACK_DYNPTR ||\n-\t\t state-\u003estack[spi - 1].slot_type[i] != STACK_DYNPTR)\n+\t\tif (slot-\u003eslot_type[i] != STACK_DYNPTR ||\n+\t\t bpf_stack_slot(state, spi - 1)-\u003eslot_type[i] != STACK_DYNPTR)\n \t\t\treturn false;\n \t}\n \n@@ -965,7 +971,7 @@ static enum bpf_dynptr_type dynptr_reg_type(struct bpf_verifier_env *env, struct\n \tif (spi \u003c 0)\n \t\treturn BPF_DYNPTR_TYPE_INVALID;\n \tstate = bpf_func(env, reg);\n-\treturn state-\u003estack[spi].spilled_ptr.dynptr.type;\n+\treturn bpf_stack_slot(state, spi)-\u003espilled_ptr.dynptr.type;\n }\n \n static bool is_dynptr_type_expected(struct bpf_verifier_env *env, struct bpf_reg_state *reg,\n@@ -1005,7 +1011,7 @@ static int mark_stack_slots_iter(struct bpf_verifier_env *env,\n \t\treturn id;\n \n \tfor (i = 0; i \u003c nr_slots; i++) {\n-\t\tstruct bpf_stack_state *slot = \u0026state-\u003estack[spi - i];\n+\t\tstruct bpf_stack_state *slot = bpf_stack_slot(state, spi - i);\n \t\tstruct bpf_reg_state *st = \u0026slot-\u003espilled_ptr;\n \n \t\t__mark_reg_known_zero(st);\n@@ -1042,7 +1048,7 @@ static int unmark_stack_slots_iter(struct bpf_verifier_env *env,\n \t\treturn spi;\n \n \tfor (i = 0; i \u003c nr_slots; i++) {\n-\t\tstruct bpf_stack_state *slot = \u0026state-\u003estack[spi - i];\n+\t\tstruct bpf_stack_state *slot = bpf_stack_slot(state, spi - i);\n \t\tstruct bpf_reg_state *st = \u0026slot-\u003espilled_ptr;\n \n \t\tif (i == 0)\n@@ -1076,7 +1082,7 @@ static bool is_iter_reg_valid_uninit(struct bpf_verifier_env *env,\n \t\treturn false;\n \n \tfor (i = 0; i \u003c nr_slots; i++) {\n-\t\tstruct bpf_stack_state *slot = \u0026state-\u003estack[spi - i];\n+\t\tstruct bpf_stack_state *slot = bpf_stack_slot(state, spi - i);\n \n \t\tfor (j = 0; j \u003c BPF_REG_SIZE; j++)\n \t\t\tif (slot-\u003eslot_type[j] == STACK_ITER)\n@@ -1097,7 +1103,7 @@ static int is_iter_reg_valid_init(struct bpf_verifier_env *env, struct bpf_reg_s\n \t\treturn -EINVAL;\n \n \tfor (i = 0; i \u003c nr_slots; i++) {\n-\t\tstruct bpf_stack_state *slot = \u0026state-\u003estack[spi - i];\n+\t\tstruct bpf_stack_state *slot = bpf_stack_slot(state, spi - i);\n \t\tstruct bpf_reg_state *st = \u0026slot-\u003espilled_ptr;\n \n \t\tif (st-\u003etype \u0026 PTR_UNTRUSTED)\n@@ -1139,7 +1145,7 @@ static int mark_stack_slot_irq_flag(struct bpf_verifier_env *env,\n \tif (id \u003c 0)\n \t\treturn id;\n \n-\tslot = \u0026state-\u003estack[spi];\n+\tslot = bpf_stack_slot(state, spi);\n \tst = \u0026slot-\u003espilled_ptr;\n \n \t__mark_reg_known_zero(st);\n@@ -1166,7 +1172,7 @@ static int unmark_stack_slot_irq_flag(struct bpf_verifier_env *env, struct bpf_r\n \tif (spi \u003c 0)\n \t\treturn spi;\n \n-\tslot = \u0026state-\u003estack[spi];\n+\tslot = bpf_stack_slot(state, spi);\n \tst = \u0026slot-\u003espilled_ptr;\n \n \tif (st-\u003eirq.kfunc_class != kfunc_class) {\n@@ -1235,7 +1241,7 @@ static bool is_irq_flag_reg_valid_uninit(struct bpf_verifier_env *env, struct bp\n \tif (spi \u003c 0)\n \t\treturn false;\n \n-\tslot = \u0026state-\u003estack[spi];\n+\tslot = bpf_stack_slot(state, spi);\n \n \tfor (i = 0; i \u003c BPF_REG_SIZE; i++)\n \t\tif (slot-\u003eslot_type[i] == STACK_IRQ_FLAG)\n@@ -1254,7 +1260,7 @@ static int is_irq_flag_reg_valid_init(struct bpf_verifier_env *env, struct bpf_r\n \tif (spi \u003c 0)\n \t\treturn -EINVAL;\n \n-\tslot = \u0026state-\u003estack[spi];\n+\tslot = bpf_stack_slot(state, spi);\n \tst = \u0026slot-\u003espilled_ptr;\n \n \tif (!st-\u003eid)\n@@ -1401,7 +1407,7 @@ static int copy_reference_state(struct bpf_verifier_state *dst, const struct bpf\n \n static int copy_stack_state(struct bpf_func_state *dst, const struct bpf_func_state *src)\n {\n-\tsize_t n = src-\u003eallocated_stack / BPF_REG_SIZE;\n+\tsize_t n = bpf_stack_nr_slots(src);\n \n \tdst-\u003estack = copy_array(dst-\u003estack, src-\u003estack, n, sizeof(struct bpf_stack_state),\n \t\t\t\tGFP_KERNEL_ACCOUNT);\n@@ -1440,7 +1446,7 @@ static int resize_reference_state(struct bpf_verifier_state *state, size_t n)\n */\n static int grow_stack_state(struct bpf_verifier_env *env, struct bpf_func_state *state, int size)\n {\n-\tsize_t old_n = state-\u003eallocated_stack / BPF_REG_SIZE, n;\n+\tsize_t old_n = bpf_stack_nr_slots(state), n;\n \n \t/* The stack size is always a multiple of BPF_REG_SIZE. */\n \tsize = round_up(size, BPF_REG_SIZE);\n@@ -3302,26 +3308,25 @@ static void mark_non_stack_access(struct bpf_verifier_env *env, int idx)\n \tenv-\u003einsn_aux_data[idx].non_stack_access = true;\n }\n \n+/* Layout of one packed linked register in the jump history, see linked_regs_pack() */\n #define LR_FRAMENO_BITS\t4\n-#define LR_SPI_BITS\t6\n-#define LR_ENTRY_BITS\t(LR_SPI_BITS + LR_FRAMENO_BITS + 1)\n-#define LR_SIZE_BITS\t4\n-#define LR_FRAMENO_MASK\t((1ull \u003c\u003c LR_FRAMENO_BITS) - 1)\n-#define LR_SPI_MASK\t((1ull \u003c\u003c LR_SPI_BITS) - 1)\n-#define LR_SIZE_MASK\t((1ull \u003c\u003c LR_SIZE_BITS) - 1)\n-#define LR_SPI_OFF\tLR_FRAMENO_BITS\n-#define LR_IS_REG_OFF\t(LR_SPI_BITS + LR_FRAMENO_BITS)\n-#define LINKED_REGS_MAX\t5\n+#define LR_INDEX_BITS\t11\n+#define LR_FRAMENO_MASK\t((1u \u003c\u003c LR_FRAMENO_BITS) - 1)\n+#define LR_IS_REG\tBIT(LR_FRAMENO_BITS)\n+#define LR_INDEX_OFF\t(LR_FRAMENO_BITS + 1)\n+#define LR_INDEX_MASK\t((1u \u003c\u003c LR_INDEX_BITS) - 1)\n+#define LINKED_REGS_MAX\tBPF_LINKED_REGS_MAX\n \n static_assert(MAX_CALL_FRAMES \u003c= (1 \u003c\u003c LR_FRAMENO_BITS));\n-static_assert(LINKED_REGS_MAX \u003c (1 \u003c\u003c LR_SIZE_BITS));\n-static_assert(LINKED_REGS_MAX * LR_ENTRY_BITS + LR_SIZE_BITS \u003c= 64);\n+static_assert(MAX_BPF_REG \u003c= (1 \u003c\u003c LR_INDEX_BITS));\n+static_assert(MAX_BPF_STACK_SLOTS \u003c= (1 \u003c\u003c LR_INDEX_BITS));\n+static_assert(LR_INDEX_OFF + LR_INDEX_BITS \u003c= 16);\n \n struct linked_reg {\n \tu8 frameno;\n \tunion {\n-\t\tu8 spi;\n-\t\tu8 regno;\n+\t\tu16 spi;\n+\t\tu16 regno;\n \t};\n \tbool is_reg;\n };\n@@ -3340,48 +3345,34 @@ static struct linked_reg *linked_regs_push(struct linked_regs *s)\n }\n \n /*\n- * Use u64 as a vector of 5 11-bit values, use first 4-bits to track\n- * number of elements currently in stack.\n- * Pack one history entry for linked registers as 11 bits in the following format:\n- * - 4-bits frameno\n- * - 6-bits spi_or_reg\n- * - 1-bit is_reg\n+ * Pack linked registers for a jump history entry, one u16 each:\n+ * - 4 bits frameno\n+ * - 1 bit is_reg\n+ * - 11 bits register or stack slot index\n */\n-static u64 linked_regs_pack(struct linked_regs *s)\n+static void linked_regs_pack(const struct linked_regs *s, u16 *packed)\n {\n-\tu64 val = 0;\n \tint i;\n \n \tfor (i = 0; i \u003c s-\u003ecnt; ++i) {\n-\t\tstruct linked_reg *e = \u0026s-\u003eentries[i];\n-\t\tu64 tmp = 0;\n-\n-\t\ttmp |= e-\u003eframeno;\n-\t\ttmp |= e-\u003espi \u003c\u003c LR_SPI_OFF;\n-\t\ttmp |= (e-\u003eis_reg ? 1 : 0) \u003c\u003c LR_IS_REG_OFF;\n+\t\tconst struct linked_reg *e = \u0026s-\u003eentries[i];\n \n-\t\tval \u003c\u003c= LR_ENTRY_BITS;\n-\t\tval |= tmp;\n+\t\tpacked[i] = e-\u003eframeno | (e-\u003eis_reg ? LR_IS_REG : 0) | (e-\u003espi \u003c\u003c LR_INDEX_OFF);\n \t}\n-\tval \u003c\u003c= LR_SIZE_BITS;\n-\tval |= s-\u003ecnt;\n-\treturn val;\n }\n \n-static void linked_regs_unpack(u64 val, struct linked_regs *s)\n+static void linked_regs_unpack(const struct bpf_jmp_history_entry *hist, struct linked_regs *s)\n {\n \tint i;\n \n-\ts-\u003ecnt = val \u0026 LR_SIZE_MASK;\n-\tval \u003e\u003e= LR_SIZE_BITS;\n-\n+\ts-\u003ecnt = hist-\u003elinked_regs_cnt;\n \tfor (i = 0; i \u003c s-\u003ecnt; ++i) {\n \t\tstruct linked_reg *e = \u0026s-\u003eentries[i];\n+\t\tu16 packed = hist-\u003elinked_regs[i];\n \n-\t\te-\u003eframeno = val \u0026 LR_FRAMENO_MASK;\n-\t\te-\u003espi = (val \u003e\u003e LR_SPI_OFF) \u0026 LR_SPI_MASK;\n-\t\te-\u003eis_reg = (val \u003e\u003e LR_IS_REG_OFF) \u0026 0x1;\n-\t\tval \u003e\u003e= LR_ENTRY_BITS;\n+\t\te-\u003eframeno = packed \u0026 LR_FRAMENO_MASK;\n+\t\te-\u003eis_reg = packed \u0026 LR_IS_REG;\n+\t\te-\u003espi = (packed \u003e\u003e LR_INDEX_OFF) \u0026 LR_INDEX_MASK;\n \t}\n }\n \n@@ -3423,10 +3414,10 @@ void bpf_bt_sync_linked_regs(struct backtrack_state *bt, struct bpf_jmp_history_\n \tbool some_precise = false;\n \tint i;\n \n-\tif (!hist || hist-\u003elinked_regs == 0)\n+\tif (!hist || !hist-\u003elinked_regs_cnt)\n \t\treturn;\n \n-\tlinked_regs_unpack(hist-\u003elinked_regs, \u0026linked_regs);\n+\tlinked_regs_unpack(hist, \u0026linked_regs);\n \tfor (i = 0; i \u003c linked_regs.cnt; ++i) {\n \t\tstruct linked_reg *e = \u0026linked_regs.entries[i];\n \n@@ -3523,17 +3514,18 @@ static void save_register_state(struct bpf_verifier_env *env,\n \t\t\t\tint spi, struct bpf_reg_state *reg,\n \t\t\t\tint size)\n {\n+\tstruct bpf_stack_state *slot = bpf_stack_slot(state, spi);\n \tint i;\n \n-\tbpf_diag_mod_begin(env, \u0026state-\u003estack[spi].spilled_ptr, reg, BPF_DIAG_MOD_SPILL);\n-\tstate-\u003estack[spi].spilled_ptr = *reg;\n+\tbpf_diag_mod_begin(env, \u0026slot-\u003espilled_ptr, reg, BPF_DIAG_MOD_SPILL);\n+\tslot-\u003espilled_ptr = *reg;\n \n \tfor (i = BPF_REG_SIZE; i \u003e BPF_REG_SIZE - size; i--)\n-\t\tstate-\u003estack[spi].slot_type[i - 1] = STACK_SPILL;\n+\t\tslot-\u003eslot_type[i - 1] = STACK_SPILL;\n \n \t/* size \u003c 8 bytes spill */\n \tfor (; i; i--)\n-\t\tmark_stack_slot_misc(env, \u0026state-\u003estack[spi].slot_type[i - 1]);\n+\t\tmark_stack_slot_misc(env, \u0026slot-\u003eslot_type[i - 1]);\n \n \tbpf_diag_mod_end(env);\n }\n@@ -3575,14 +3567,15 @@ static void check_fastcall_stack_contract(struct bpf_verifier_env *env,\n \n static void scrub_special_slot(struct bpf_func_state *state, int spi)\n {\n+\tstruct bpf_stack_state *slot = bpf_stack_slot(state, spi);\n \tint i;\n \n \t/* regular write of data into stack destroys any spilled ptr */\n-\tstate-\u003estack[spi].spilled_ptr.type = NOT_INIT;\n+\tslot-\u003espilled_ptr.type = NOT_INIT;\n \t/* Mark slots as STACK_MISC if they belonged to spilled ptr/dynptr/iter. */\n-\tif (is_stack_slot_special(\u0026state-\u003estack[spi]))\n+\tif (is_stack_slot_special(slot))\n \t\tfor (i = 0; i \u003c BPF_REG_SIZE; i++)\n-\t\t\tscrub_spilled_slot(\u0026state-\u003estack[spi].slot_type[i]);\n+\t\t\tscrub_spilled_slot(\u0026slot-\u003eslot_type[i]);\n }\n \n /* check_stack_{read,write}_fixed_off functions track spill/fill of registers,\n@@ -3600,13 +3593,15 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env,\n \tstruct bpf_reg_state *reg = NULL;\n \tint insn_flags = INSN_F_STACK_ACCESS;\n \tint hist_spi = spi, hist_frame = state-\u003eframeno;\n+\tstruct bpf_stack_state *ss = bpf_stack_slot(state, spi);\n \n-\t/* caller checked that off % size == 0 and -MAX_BPF_STACK \u003c= off \u003c 0,\n+\t/*\n+\t * caller checked that off % size == 0 and -env-\u003estack_limit \u003c= off \u003c 0,\n \t * so it's aligned access and [off, off + size) are within stack limits\n \t */\n \tif (!env-\u003eallow_ptr_leaks \u0026\u0026\n-\t bpf_is_spilled_reg(\u0026state-\u003estack[spi]) \u0026\u0026\n-\t !bpf_is_spilled_scalar_reg(\u0026state-\u003estack[spi]) \u0026\u0026\n+\t bpf_is_spilled_reg(ss) \u0026\u0026\n+\t !bpf_is_spilled_scalar_reg(ss) \u0026\u0026\n \t size != BPF_REG_SIZE) {\n \t\tconst char *reason;\n \n@@ -3628,7 +3623,7 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env,\n \t\tbool sanitize = reg \u0026\u0026 is_pointer_regtype(reg-\u003etype);\n \n \t\tfor (i = 0; i \u003c size; i++) {\n-\t\t\tu8 type = state-\u003estack[spi].slot_type[(slot - i) %\n+\t\t\tu8 type = ss-\u003eslot_type[(slot - i) %\n \t\t\t\t\t\t\t BPF_REG_SIZE];\n \n \t\t\tif (type != STACK_MISC \u0026\u0026 type != STACK_ZERO) {\n@@ -3657,7 +3652,7 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env,\n \t\tsave_register_state(env, state, spi, reg, size);\n \t\t/* Break the relation on a narrowing spill. */\n \t\tif (!reg_value_fits)\n-\t\t\tstate-\u003estack[spi].spilled_ptr.id = 0;\n+\t\t\tss-\u003espilled_ptr.id = 0;\n \t} else if (!reg \u0026\u0026 !(off % BPF_REG_SIZE) \u0026\u0026 is_bpf_st_mem(insn) \u0026\u0026\n \t\t env-\u003ebpf_capable) {\n \t\tstruct bpf_reg_state *tmp_reg = \u0026env-\u003efake_reg[0];\n@@ -3681,8 +3676,8 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env,\n \t} else {\n \t\tu8 type = STACK_MISC;\n \n-\t\tif (bpf_is_spilled_reg(\u0026state-\u003estack[spi]))\n-\t\t\tbpf_diag_record_scrub(env, \u0026state-\u003estack[spi].spilled_ptr,\n+\t\tif (bpf_is_spilled_reg(ss))\n+\t\t\tbpf_diag_record_scrub(env, \u0026ss-\u003espilled_ptr,\n \t\t\t\t\t BPF_DIAG_MOD_WRITE);\n \t\tscrub_special_slot(state, spi);\n \n@@ -3703,13 +3698,13 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env,\n \n \t\t/* Mark slots affected by this stack write. */\n \t\tfor (i = 0; i \u003c size; i++)\n-\t\t\tstate-\u003estack[spi].slot_type[(slot - i) % BPF_REG_SIZE] = type;\n+\t\t\tss-\u003eslot_type[(slot - i) % BPF_REG_SIZE] = type;\n \t\tinsn_flags = 0; /* not a register spill */\n \t}\n \n \tif (insn_flags)\n \t\treturn bpf_push_jmp_history(env, env-\u003ecur_state, insn_flags,\n-\t\t\t\t\t hist_spi, hist_frame, 0);\n+\t\t\t\t\t hist_spi, hist_frame, NULL, 0);\n \treturn 0;\n }\n \n@@ -3774,7 +3769,7 @@ static int check_stack_write_var_off(struct bpf_verifier_env *env,\n \n \t\tslot = -i - 1;\n \t\tspi = slot / BPF_REG_SIZE;\n-\t\tstype = \u0026state-\u003estack[spi].slot_type[slot % BPF_REG_SIZE];\n+\t\tstype = \u0026bpf_stack_slot(state, spi)-\u003eslot_type[slot % BPF_REG_SIZE];\n \t\tmark_stack_slot_scratched(env, spi);\n \n \t\tif (!env-\u003eallow_ptr_leaks \u0026\u0026 *stype != STACK_MISC \u0026\u0026 *stype != STACK_ZERO) {\n@@ -3798,8 +3793,8 @@ static int check_stack_write_var_off(struct bpf_verifier_env *env,\n \t\t * maintain the spill type.\n \t\t */\n \t\tif (writing_zero \u0026\u0026 *stype == STACK_SPILL \u0026\u0026\n-\t\t bpf_is_spilled_scalar_reg(\u0026state-\u003estack[spi])) {\n-\t\t\tstruct bpf_reg_state *spill_reg = \u0026state-\u003estack[spi].spilled_ptr;\n+\t\t bpf_is_spilled_scalar_reg(bpf_stack_slot(state, spi))) {\n+\t\t\tstruct bpf_reg_state *spill_reg = \u0026bpf_stack_slot(state, spi)-\u003espilled_ptr;\n \n \t\t\tif (tnum_is_const(spill_reg-\u003evar_off) \u0026\u0026 spill_reg-\u003evar_off.value == 0) {\n \t\t\t\tzero_used = true;\n@@ -3870,23 +3865,22 @@ static int mark_reg_stack_read(struct bpf_verifier_env *env,\n {\n \tstruct bpf_verifier_state *vstate = env-\u003ecur_state;\n \tstruct bpf_func_state *state = vstate-\u003eframe[vstate-\u003ecurframe];\n-\tu64 zero_spill_mask = 0;\n \tint i, slot, spi;\n \tu8 *stype;\n-\tint zeros = 0;\n+\tint zeros = 0, zero_spills = 0;\n \n \tfor (i = min_off; i \u003c max_off; i++) {\n \t\tslot = -i - 1;\n \t\tspi = slot / BPF_REG_SIZE;\n \t\tmark_stack_slot_scratched(env, spi);\n-\t\tstype = ptr_state-\u003estack[spi].slot_type;\n+\t\tstype = bpf_stack_slot(ptr_state, spi)-\u003eslot_type;\n \t\tif (stype[slot % BPF_REG_SIZE] == STACK_ZERO) {\n \t\t\tzeros++;\n \t\t\tcontinue;\n \t\t}\n \t\tif (stype[slot % BPF_REG_SIZE] == STACK_SPILL \u0026\u0026\n-\t\t bpf_register_is_null(\u0026ptr_state-\u003estack[spi].spilled_ptr)) {\n-\t\t\tzero_spill_mask |= 1ull \u003c\u003c spi;\n+\t\t bpf_register_is_null(\u0026bpf_stack_slot(ptr_state, spi)-\u003espilled_ptr)) {\n+\t\t\tzero_spills++;\n \t\t\tzeros++;\n \t\t\tcontinue;\n \t\t}\n@@ -3897,8 +3891,14 @@ static int mark_reg_stack_read(struct bpf_verifier_env *env,\n \t\t * so the whole register == const_zero.\n \t\t */\n \t\t__mark_reg_const_zero(env, \u0026state-\u003eregs[dst_regno]);\n-\t\tif (zero_spill_mask) {\n-\t\t\tbpf_bt_set_frame_slot_mask(\u0026env-\u003ebt, ptr_state-\u003eframeno, zero_spill_mask);\n+\t\tif (zero_spills) {\n+\t\t\tfor (i = min_off; i \u003c max_off; i++) {\n+\t\t\t\tslot = -i - 1;\n+\t\t\t\tspi = slot / BPF_REG_SIZE;\n+\t\t\t\tstype = bpf_stack_slot(ptr_state, spi)-\u003eslot_type;\n+\t\t\t\tif (stype[slot % BPF_REG_SIZE] == STACK_SPILL)\n+\t\t\t\t\tbpf_bt_set_frame_slot(\u0026env-\u003ebt, ptr_state-\u003eframeno, spi);\n+\t\t\t}\n \t\t\treturn mark_chain_precision_batch(env, env-\u003ecur_state);\n \t\t}\n \t} else {\n@@ -3946,9 +3946,10 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env,\n \tint err;\n \tint insn_flags = INSN_F_STACK_ACCESS;\n \tint hist_spi = spi, hist_frame = reg_state-\u003eframeno;\n+\tstruct bpf_stack_state *ss = bpf_stack_slot(reg_state, spi);\n \n-\tstype = reg_state-\u003estack[spi].slot_type;\n-\treg = \u0026reg_state-\u003estack[spi].spilled_ptr;\n+\tstype = ss-\u003eslot_type;\n+\treg = \u0026ss-\u003espilled_ptr;\n \n \tmark_stack_slot_scratched(env, spi);\n \tcheck_fastcall_stack_contract(env, state, env-\u003einsn_idx, off);\n@@ -3959,7 +3960,7 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env,\n \tif (dst_regno \u003e= 0)\n \t\tbpf_diag_mod_begin(env, \u0026state-\u003eregs[dst_regno], reg, BPF_DIAG_MOD_WRITE);\n \n-\tif (bpf_is_spilled_reg(\u0026reg_state-\u003estack[spi])) {\n+\tif (bpf_is_spilled_reg(ss)) {\n \t\tu8 spill_size = 1;\n \n \t\tfor (i = BPF_REG_SIZE - 1; i \u003e 0 \u0026\u0026 stype[i - 1] == STACK_SPILL; i--)\n@@ -4085,7 +4086,7 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env,\n \t}\n \tif (insn_flags)\n \t\treturn bpf_push_jmp_history(env, env-\u003ecur_state, insn_flags,\n-\t\t\t\t\t hist_spi, hist_frame, 0);\n+\t\t\t\t\t hist_spi, hist_frame, NULL, 0);\n \treturn 0;\n }\n \n@@ -4275,7 +4276,7 @@ static int check_stack_arg_write(struct bpf_verifier_env *env, struct bpf_func_s\n \tbpf_diag_mod_end(env);\n \tstate-\u003eno_stack_arg_load = true;\n \treturn bpf_push_jmp_history(env, env-\u003ecur_state,\n-\t\t\t\t INSN_F_STACK_ARG_ACCESS, spi, 0, 0);\n+\t\t\t\t INSN_F_STACK_ARG_ACCESS, spi, 0, NULL, 0);\n }\n \n /*\n@@ -4309,7 +4310,7 @@ static int check_stack_arg_read(struct bpf_verifier_env *env, struct bpf_func_st\n \tcur-\u003eregs[dst_regno] = *arg;\n \tbpf_diag_mod_end(env);\n \treturn bpf_push_jmp_history(env, env-\u003ecur_state,\n-\t\t\t\t INSN_F_STACK_ARG_ACCESS, spi, 0, 0);\n+\t\t\t\t INSN_F_STACK_ARG_ACCESS, spi, 0, NULL, 0);\n }\n \n static int mark_stack_arg_precision(struct bpf_verifier_env *env, int arg_idx)\n@@ -5448,7 +5449,7 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx,\n \tif (subprog[idx].priv_stack_mode == PRIV_STACK_ADAPTIVE) {\n \t\tif (subprog_depth \u003e env-\u003emax_stack_depth)\n \t\t\tenv-\u003emax_stack_depth = subprog_depth;\n-\t\tif (subprog_depth \u003e MAX_BPF_STACK) {\n+\t\tif (subprog_depth \u003e env-\u003estack_limit) {\n \t\t\tverbose(env, \"stack size of subprog %d is %d. Too large\\n\",\n \t\t\t\tidx, subprog_depth);\n \t\t\treturn -EACCES;\n@@ -5457,7 +5458,7 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx,\n \t\tdepth += subprog_depth;\n \t\tif (depth \u003e env-\u003emax_stack_depth)\n \t\t\tenv-\u003emax_stack_depth = depth;\n-\t\tif (depth \u003e MAX_BPF_STACK) {\n+\t\tif (depth \u003e env-\u003estack_limit) {\n \t\t\ttotal = 0;\n \t\t\tfor (tmp = idx; tmp \u003e= 0; tmp = dinfo[tmp].caller)\n \t\t\t\ttotal++;\n@@ -6308,10 +6309,11 @@ static int check_ptr_to_map_access(struct bpf_verifier_env *env,\n \treturn 0;\n }\n \n-/* Check that the stack access at the given offset is within bounds. The\n+/*\n+ * Check that the stack access at the given offset is within bounds. The\n * maximum valid offset is -1.\n *\n- * The minimum valid offset is -MAX_BPF_STACK for writes, and\n+ * The minimum valid offset is -env-\u003estack_limit for writes, and\n * -state-\u003eallocated_stack for reads.\n */\n static int check_stack_slot_within_bounds(struct bpf_verifier_env *env,\n@@ -6322,7 +6324,7 @@ static int check_stack_slot_within_bounds(struct bpf_verifier_env *env,\n \tint min_valid_off;\n \n \tif (t == BPF_WRITE || env-\u003eallow_uninit_stack)\n-\t\tmin_valid_off = -MAX_BPF_STACK;\n+\t\tmin_valid_off = -(int)env-\u003estack_limit;\n \telse\n \t\tmin_valid_off = -state-\u003eallocated_stack;\n \n@@ -7048,6 +7050,7 @@ static int check_stack_range_initialized(\n \t}\n \n \tfor (i = min_off; i \u003c max_off + access_size; i++) {\n+\t\tstruct bpf_stack_state *ss;\n \t\tu8 *stype;\n \n \t\tslot = -i - 1;\n@@ -7057,7 +7060,8 @@ static int check_stack_range_initialized(\n \t\t\treturn -EFAULT;\n \t\t}\n \n-\t\tstype = \u0026state-\u003estack[spi].slot_type[slot % BPF_REG_SIZE];\n+\t\tss = bpf_stack_slot(state, spi);\n+\t\tstype = \u0026ss-\u003eslot_type[slot % BPF_REG_SIZE];\n \t\tif (*stype == STACK_MISC)\n \t\t\tgoto mark;\n \t\tif ((*stype == STACK_ZERO) ||\n@@ -7069,13 +7073,13 @@ static int check_stack_range_initialized(\n \t\t\tgoto mark;\n \t\t}\n \n-\t\tif (bpf_is_spilled_reg(\u0026state-\u003estack[spi]) \u0026\u0026\n-\t\t (state-\u003estack[spi].spilled_ptr.type == SCALAR_VALUE ||\n+\t\tif (bpf_is_spilled_reg(ss) \u0026\u0026\n+\t\t (ss-\u003espilled_ptr.type == SCALAR_VALUE ||\n \t\t env-\u003eallow_ptr_leaks)) {\n \t\t\tif (clobber) {\n-\t\t\t\t__mark_reg_unknown(env, \u0026state-\u003estack[spi].spilled_ptr);\n+\t\t\t\t__mark_reg_unknown(env, \u0026ss-\u003espilled_ptr);\n \t\t\t\tfor (j = 0; j \u003c BPF_REG_SIZE; j++)\n-\t\t\t\t\tscrub_spilled_slot(\u0026state-\u003estack[spi].slot_type[j]);\n+\t\t\t\t\tscrub_spilled_slot(\u0026ss-\u003eslot_type[j]);\n \t\t\t}\n \t\t\tgoto mark;\n \t\t}\n@@ -7805,7 +7809,7 @@ static int process_dynptr_func(struct bpf_verifier_env *env, struct bpf_reg_stat\n \n \t\t\tmark_stack_slots_scratched(env, spi, BPF_DYNPTR_NR_SLOTS);\n \n-\t\t\treg = \u0026state-\u003estack[spi].spilled_ptr;\n+\t\t\treg = \u0026bpf_stack_slot(state, spi)-\u003espilled_ptr;\n \t\t}\n \n \t\tmeta-\u003edynptr.type = reg-\u003edynptr.type;\n@@ -7938,7 +7942,7 @@ static int process_iter_arg(struct bpf_verifier_env *env, struct bpf_reg_state *\n \t\t/* remember meta-\u003eiter info for process_iter_next_call() */\n \t\tmeta-\u003eiter.spi = spi;\n \t\tmeta-\u003eiter.frameno = reg-\u003eframeno;\n-\t\tupdate_ref_obj(\u0026meta-\u003eref_obj, \u0026state-\u003estack[spi].spilled_ptr);\n+\t\tupdate_ref_obj(\u0026meta-\u003eref_obj, \u0026bpf_stack_slot(state, spi)-\u003espilled_ptr);\n \n \t\tif (is_iter_destroy_kfunc(meta)) {\n \t\t\terr = unmark_stack_slots_iter(env, reg, nr_slots);\n@@ -8015,16 +8019,15 @@ static int widen_imprecise_scalars(struct bpf_verifier_env *env,\n \t\t\t\t\t\u0026fold-\u003eregs[i],\n \t\t\t\t\t\u0026fcur-\u003eregs[i]);\n \n-\t\tnum_slots = min(fold-\u003eallocated_stack / BPF_REG_SIZE,\n-\t\t\t\tfcur-\u003eallocated_stack / BPF_REG_SIZE);\n+\t\tnum_slots = min(bpf_stack_nr_slots(fold), bpf_stack_nr_slots(fcur));\n \t\tfor (i = 0; i \u003c num_slots; i++) {\n-\t\t\tif (!bpf_is_spilled_reg(\u0026fold-\u003estack[i]) ||\n-\t\t\t !bpf_is_spilled_reg(\u0026fcur-\u003estack[i]))\n+\t\t\tif (!bpf_is_spilled_reg(bpf_stack_slot(fold, i)) ||\n+\t\t\t !bpf_is_spilled_reg(bpf_stack_slot(fcur, i)))\n \t\t\t\tcontinue;\n \n \t\t\tmaybe_widen_reg(env,\n-\t\t\t\t\t\u0026fold-\u003estack[i].spilled_ptr,\n-\t\t\t\t\t\u0026fcur-\u003estack[i].spilled_ptr);\n+\t\t\t\t\t\u0026bpf_stack_slot(fold, i)-\u003espilled_ptr,\n+\t\t\t\t\t\u0026bpf_stack_slot(fcur, i)-\u003espilled_ptr);\n \t\t}\n \t}\n \treturn 0;\n@@ -8036,7 +8039,7 @@ static struct bpf_reg_state *get_iter_from_state(struct bpf_verifier_state *cur_\n \tint iter_frameno = meta-\u003eiter.frameno;\n \tint iter_spi = meta-\u003eiter.spi;\n \n-\treturn \u0026cur_st-\u003eframe[iter_frameno]-\u003estack[iter_spi].spilled_ptr;\n+\treturn \u0026bpf_stack_slot(cur_st-\u003eframe[iter_frameno], iter_spi)-\u003espilled_ptr;\n }\n \n /* process_iter_next_call() is called when verifier gets to iterator's next\n@@ -8841,7 +8844,7 @@ static int get_constant_map_key(struct bpf_verifier_env *env,\n \tslot = -stack_off - 1;\n \tspi = slot / BPF_REG_SIZE;\n \toff = slot % BPF_REG_SIZE;\n-\tstype = state-\u003estack[spi].slot_type;\n+\tstype = bpf_stack_slot(state, spi)-\u003eslot_type;\n \n \t/* First handle precisely tracked STACK_ZERO */\n \tfor (i = off; i \u003e= 0 \u0026\u0026 stype[i] == STACK_ZERO; i--)\n@@ -8852,14 +8855,14 @@ static int get_constant_map_key(struct bpf_verifier_env *env,\n \t}\n \n \t/* Check that stack contains a scalar spill of expected size */\n-\tif (!bpf_is_spilled_scalar_reg(\u0026state-\u003estack[spi]))\n+\tif (!bpf_is_spilled_scalar_reg(bpf_stack_slot(state, spi)))\n \t\treturn -EOPNOTSUPP;\n \tfor (i = off; i \u003e= 0 \u0026\u0026 stype[i] == STACK_SPILL; i--)\n \t\tspill_size++;\n \tif (spill_size != key_size)\n \t\treturn -EOPNOTSUPP;\n \n-\treg = \u0026state-\u003estack[spi].spilled_ptr;\n+\treg = \u0026bpf_stack_slot(state, spi)-\u003espilled_ptr;\n \tif (!tnum_is_const(reg-\u003evar_off))\n \t\t/* Stack value not statically known */\n \t\treturn -EOPNOTSUPP;\n@@ -10135,8 +10138,9 @@ static int idstack_push(struct bpf_idmap *idmap, u32 id)\n \t\tif (idmap-\u003emap[i].old == id)\n \t\t\treturn 0;\n \n-\tif (WARN_ON_ONCE(idmap-\u003ecnt \u003e= BPF_ID_MAP_SIZE))\n-\t\treturn -EFAULT;\n+\tif (!bpf_id_scratch_reserve((void **)\u0026idmap-\u003emap, \u0026idmap-\u003ecap, idmap-\u003ecnt,\n+\t\t\t\t sizeof(*idmap-\u003emap)))\n+\t\treturn -ENOMEM;\n \n \tidmap-\u003emap[idmap-\u003ecnt++].old = id;\n \treturn 0;\n@@ -13893,11 +13897,11 @@ s64 bpf_helper_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn\n \t\t\t}\n \t\t\t/*\n \t\t\t * Size arg is const on each path but differs across merged\n-\t\t\t * paths. MAX_BPF_STACK is a safe upper bound for reads.\n+\t\t\t * paths. Reads may extend anywhere up to the frame top.\n \t\t\t */\n \t\t\tif (full_write)\n \t\t\t\treturn 0;\n-\t\t\treturn MAX_BPF_STACK;\n+\t\t\treturn S64_MIN;\n \t\t}\n \t\treturn S64_MIN;\n \tcase ARG_PTR_TO_DYNPTR:\n@@ -13983,7 +13987,8 @@ s64 bpf_kfunc_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn *\n \t\t\tsize = (s64)aux-\u003econst_reg_vals[size_reg];\n \t\t\tgoto out;\n \t\t}\n-\t\treturn MAX_BPF_STACK;\n+\t\t/* Unknown size: the read may extend anywhere up to the frame top. */\n+\t\treturn S64_MIN;\n \t}\n \n \t/* fixed-size pointed-to type: resolve via BTF */\n@@ -14725,7 +14730,8 @@ enum {\n \tREASON_STACK\t= -5,\n };\n \n-static int retrieve_ptr_limit(const struct bpf_reg_state *ptr_reg,\n+static int retrieve_ptr_limit(const struct bpf_verifier_env *env,\n+\t\t\t const struct bpf_reg_state *ptr_reg,\n \t\t\t u32 *alu_limit, bool mask_to_left)\n {\n \tu32 max = 0, ptr_limit = 0;\n@@ -14737,7 +14743,7 @@ static int retrieve_ptr_limit(const struct bpf_reg_state *ptr_reg,\n \t\t * offset where we would need to deal with min/max bounds is\n \t\t * currently prohibited for unprivileged.\n \t\t */\n-\t\tmax = MAX_BPF_STACK + mask_to_left;\n+\t\tmax = env-\u003estack_limit + mask_to_left;\n \t\tptr_limit = -ptr_reg-\u003evar_off.value;\n \t\tbreak;\n \tcase PTR_TO_MAP_VALUE:\n@@ -14857,7 +14863,7 @@ static int sanitize_ptr_alu(struct bpf_verifier_env *env,\n \t\t\t\t (opcode == BPF_SUB \u0026\u0026 !off_is_neg);\n \t}\n \n-\terr = retrieve_ptr_limit(ptr_reg, \u0026alu_limit, info-\u003emask_to_left);\n+\terr = retrieve_ptr_limit(env, ptr_reg, \u0026alu_limit, info-\u003emask_to_left);\n \tif (err \u003c 0)\n \t\treturn err;\n \n@@ -14987,7 +14993,7 @@ static int check_stack_access_for_ptr_arithmetic(\n \t\treturn -EACCES;\n \t}\n \n-\tif (off \u003e= 0 || off \u003c -MAX_BPF_STACK) {\n+\tif (off \u003e= 0 || off \u003c -(int)env-\u003estack_limit) {\n \t\tverbose(env, \"R%d stack pointer arithmetic goes out of range, \"\n \t\t\t\"prohibited for !root; off=%d\\n\", regno, off);\n \t\treturn -EACCES;\n@@ -17322,10 +17328,10 @@ static void collect_linked_regs(struct bpf_verifier_env *env,\n \t\t\treg = \u0026func-\u003eregs[j];\n \t\t\t__collect_linked_regs(linked_regs, reg, id, i, j, true);\n \t\t}\n-\t\tfor (j = 0; j \u003c func-\u003eallocated_stack / BPF_REG_SIZE; j++) {\n-\t\t\tif (!bpf_is_spilled_reg(\u0026func-\u003estack[j]))\n+\t\tfor (j = 0; j \u003c bpf_stack_nr_slots(func); j++) {\n+\t\t\tif (!bpf_is_spilled_reg(bpf_stack_slot(func, j)))\n \t\t\t\tcontinue;\n-\t\t\treg = \u0026func-\u003estack[j].spilled_ptr;\n+\t\t\treg = \u0026bpf_stack_slot(func, j)-\u003espilled_ptr;\n \t\t\t__collect_linked_regs(linked_regs, reg, id, i, j, false);\n \t\t}\n \t}\n@@ -17345,7 +17351,7 @@ static void sync_linked_regs(struct bpf_verifier_env *env, struct bpf_verifier_s\n \tfor (i = 0; i \u003c linked_regs-\u003ecnt; ++i) {\n \t\te = \u0026linked_regs-\u003eentries[i];\n \t\treg = e-\u003eis_reg ? \u0026vstate-\u003eframe[e-\u003eframeno]-\u003eregs[e-\u003eregno]\n-\t\t\t\t: \u0026vstate-\u003eframe[e-\u003eframeno]-\u003estack[e-\u003espi].spilled_ptr;\n+\t\t\t\t: \u0026bpf_stack_slot(vstate-\u003eframe[e-\u003eframeno], e-\u003espi)-\u003espilled_ptr;\n \t\tif (reg-\u003etype != SCALAR_VALUE || reg == known_reg)\n \t\t\tcontinue;\n \t\tif ((reg-\u003eid \u0026 ~BPF_ADD_CONST) != (known_reg-\u003eid \u0026 ~BPF_ADD_CONST))\n@@ -17463,7 +17469,7 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env,\n \t}\n \n \tif (insn_flags) {\n-\t\terr = bpf_push_jmp_history(env, this_branch, insn_flags, 0, 0, 0);\n+\t\terr = bpf_push_jmp_history(env, this_branch, insn_flags, 0, 0, NULL, 0);\n \t\tif (err)\n \t\t\treturn err;\n \t}\n@@ -17533,7 +17539,10 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env,\n \t * if parent state is created.\n \t */\n \tif (linked_regs.cnt \u003e 1) {\n-\t\terr = bpf_push_jmp_history(env, this_branch, 0, 0, 0, linked_regs_pack(\u0026linked_regs));\n+\t\tu16 packed[LINKED_REGS_MAX];\n+\n+\t\tlinked_regs_pack(\u0026linked_regs, packed);\n+\t\terr = bpf_push_jmp_history(env, this_branch, 0, 0, 0, packed, linked_regs.cnt);\n \t\tif (err)\n \t\t\treturn err;\n \t}\n@@ -18433,12 +18442,13 @@ static void idset_cnt_inc(struct bpf_idset *idset, u32 id)\n \t\t\treturn;\n \t\t}\n \t}\n-\t/* New id */\n-\tif (idset-\u003enum_ids \u003c BPF_ID_MAP_SIZE) {\n-\t\tidset-\u003eentries[idset-\u003enum_ids].id = id;\n-\t\tidset-\u003eentries[idset-\u003enum_ids].cnt = 1;\n-\t\tidset-\u003enum_ids++;\n-\t}\n+\t/* New id; one that cannot be recorded counts as shared and is kept */\n+\tif (!bpf_id_scratch_reserve((void **)\u0026idset-\u003eentries, \u0026idset-\u003ecap, idset-\u003enum_ids,\n+\t\t\t\t sizeof(*idset-\u003eentries)))\n+\t\treturn;\n+\tidset-\u003eentries[idset-\u003enum_ids].id = id;\n+\tidset-\u003eentries[idset-\u003enum_ids].cnt = 1;\n+\tidset-\u003enum_ids++;\n }\n \n /* Find id in idset and return its count, or 0 if not found */\n@@ -18927,7 +18937,7 @@ static int do_check(struct bpf_verifier_env *env)\n \t\t}\n \n \t\tif (bpf_is_jmp_point(env, env-\u003einsn_idx)) {\n-\t\t\terr = bpf_push_jmp_history(env, state, 0, 0, 0, 0);\n+\t\t\terr = bpf_push_jmp_history(env, state, 0, 0, 0, NULL, 0);\n \t\t\tif (err)\n \t\t\t\treturn err;\n \t\t}\n@@ -21686,6 +21696,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,\n \tenv-\u003ebt.env = env;\n \tenv-\u003eprog = *prog;\n \tenv-\u003eops = bpf_verifier_ops[env-\u003eprog-\u003etype];\n+\tenv-\u003estack_limit = bpf_prog_stack_limit(env-\u003eprog);\n \n \tenv-\u003eallow_ptr_leaks = bpf_allow_ptr_leaks(env-\u003eprog-\u003eaux-\u003etoken);\n \tenv-\u003eallow_uninit_stack = bpf_allow_uninit_stack(env-\u003eprog-\u003eaux-\u003etoken);\n@@ -21998,6 +22009,8 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,\n \tkvfree(env-\u003escc_info);\n \tkvfree(env-\u003esucc);\n \tkvfree(env-\u003egotox_tmp_buf);\n+\tkfree(env-\u003eidmap_scratch.map);\n+\tkfree(env-\u003eidset_scratch.entries);\n \tbpf_diag_free(env);\n \tkvfree(env);\n \treturn ret;\ndiff --git a/tools/testing/selftests/bpf/prog_tests/struct_ops_private_stack.c b/tools/testing/selftests/bpf/prog_tests/struct_ops_private_stack.c\nindex 98db9bafa44b2..2b3ec2b790918 100644\n--- a/tools/testing/selftests/bpf/prog_tests/struct_ops_private_stack.c\n+++ b/tools/testing/selftests/bpf/prog_tests/struct_ops_private_stack.c\n@@ -4,6 +4,7 @@\n #include \"struct_ops_private_stack.skel.h\"\n #include \"struct_ops_private_stack_fail.skel.h\"\n #include \"struct_ops_private_stack_recur.skel.h\"\n+#include \"struct_ops_private_stack_large.skel.h\"\n \n #if defined(__x86_64__) || defined(__aarch64__) || defined(__powerpc64__)\n static void test_private_stack(void)\n@@ -78,6 +79,34 @@ static void test_private_stack_recur(void)\n \tstruct_ops_private_stack_recur__destroy(skel);\n }\n \n+/* Two frames of 2 KiB each on the private stack */\n+static void test_private_stack_large(void)\n+{\n+\tstruct struct_ops_private_stack_large *skel;\n+\tstruct bpf_link *link;\n+\n+\tif (!is_large_stack_supported()) {\n+\t\ttest__skip();\n+\t\treturn;\n+\t}\n+\n+\tskel = struct_ops_private_stack_large__open_and_load();\n+\tif (!ASSERT_OK_PTR(skel, \"struct_ops_private_stack_large__open_and_load\"))\n+\t\treturn;\n+\n+\tlink = bpf_map__attach_struct_ops(skel-\u003emaps.testmod_1);\n+\tif (!ASSERT_OK_PTR(link, \"attach_struct_ops\"))\n+\t\tgoto cleanup;\n+\n+\tASSERT_OK(trigger_module_test_read(256), \"trigger_read\");\n+\n+\tASSERT_EQ(skel-\u003ebss-\u003eval, 100 + 30 + 12, \"val\");\n+\n+\tbpf_link__destroy(link);\n+cleanup:\n+\tstruct_ops_private_stack_large__destroy(skel);\n+}\n+\n static void __test_struct_ops_private_stack(void)\n {\n \tif (test__start_subtest(\"private_stack\"))\n@@ -86,6 +115,8 @@ static void __test_struct_ops_private_stack(void)\n \t\ttest_private_stack_fail();\n \tif (test__start_subtest(\"private_stack_recur\"))\n \t\ttest_private_stack_recur();\n+\tif (test__start_subtest(\"private_stack_large\"))\n+\t\ttest_private_stack_large();\n }\n #else\n static void __test_struct_ops_private_stack(void)\ndiff --git a/tools/testing/selftests/bpf/prog_tests/tailcalls.c b/tools/testing/selftests/bpf/prog_tests/tailcalls.c\nindex c5c9d6c359bb0..c5e3f198ca7f6 100644\n--- a/tools/testing/selftests/bpf/prog_tests/tailcalls.c\n+++ b/tools/testing/selftests/bpf/prog_tests/tailcalls.c\n@@ -9,6 +9,7 @@\n #include \"tc_bpf2bpf.skel.h\"\n #include \"tailcall_fail.skel.h\"\n #include \"tailcall_cgrp_storage_owner.skel.h\"\n+#include \"tailcall_large_stack.skel.h\"\n #include \"tailcall_cgrp_storage_no_storage.skel.h\"\n #include \"tailcall_cgrp_storage.skel.h\"\n #include \"tailcall_sleepable.skel.h\"\n@@ -1953,6 +1954,45 @@ static void test_tailcall_bpf2bpf_fexit_links(void)\n \ttailcall_bpf2bpf2__destroy(skel_tc);\n }\n \n+/*\n+ * test_tailcall_large_stack runs a tail call made from a subprog with a 1536\n+ * byte frame, under a 240-byte caller, into a program with a 2 KiB frame:\n+ *\n+ * entry (240) --call-\u003e subprog_tail (1536) --tailcall-\u003e classifier_0 (2048)\n+ */\n+static void test_tailcall_large_stack(void)\n+{\n+\tstruct tailcall_large_stack *skel;\n+\tint err, prog_fd, map_fd, key = 0;\n+\tchar buff[128] = {};\n+\tLIBBPF_OPTS(bpf_test_run_opts, topts,\n+\t\t .data_in = buff,\n+\t\t .data_size_in = sizeof(buff),\n+\t\t .repeat = 1,\n+\t);\n+\n+\tif (!is_large_stack_supported()) {\n+\t\ttest__skip();\n+\t\treturn;\n+\t}\n+\n+\tskel = tailcall_large_stack__open_and_load();\n+\tif (!ASSERT_OK_PTR(skel, \"tailcall_large_stack__open_and_load\"))\n+\t\treturn;\n+\n+\tprog_fd = bpf_program__fd(skel-\u003eprogs.classifier_0);\n+\tmap_fd = bpf_map__fd(skel-\u003emaps.jmp_table);\n+\terr = bpf_map_update_elem(map_fd, \u0026key, \u0026prog_fd, BPF_ANY);\n+\tif (!ASSERT_OK(err, \"update jmp_table\"))\n+\t\tgoto out;\n+\n+\terr = bpf_prog_test_run_opts(bpf_program__fd(skel-\u003eprogs.entry), \u0026topts);\n+\tASSERT_OK(err, \"test_run\");\n+\tASSERT_EQ(topts.retval, 42 + 7, \"retval\");\n+out:\n+\ttailcall_large_stack__destroy(skel);\n+}\n+\n void test_tailcalls(void)\n {\n \tif (test__start_subtest(\"tailcall_1\"))\n@@ -2022,4 +2062,6 @@ void test_tailcalls(void)\n \ttest_tailcall_callback();\n \tif (test__start_subtest(\"tailcall_bpf2bpf_fexit_links\"))\n \t\ttest_tailcall_bpf2bpf_fexit_links();\n+\tif (test__start_subtest(\"tailcall_large_stack\"))\n+\t\ttest_tailcall_large_stack();\n }\ndiff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c\nindex 4f1e1c1cd5ab3..ced8a2f1c89c3 100644\n--- a/tools/testing/selftests/bpf/prog_tests/verifier.c\n+++ b/tools/testing/selftests/bpf/prog_tests/verifier.c\n@@ -59,6 +59,7 @@\n #include \"verifier_kfunc_uninit.skel.h\"\n #include \"verifier_kfunc_uninit_multi.skel.h\"\n #include \"verifier_ld_ind.skel.h\"\n+#include \"verifier_large_stack.skel.h\"\n #include \"verifier_ldsx.skel.h\"\n #include \"verifier_leak_ptr.skel.h\"\n #include \"verifier_linked_scalars.skel.h\"\n@@ -229,6 +230,7 @@ void test_verifier_kfunc_uninit(void) { RUN_TESTS(verifier_kfunc_uninit)\n void test_verifier_kfunc_uninit_multi(void) { RUN_TESTS(verifier_kfunc_uninit_multi); }\n void test_verifier_load_acquire(void) { RUN(verifier_load_acquire); }\n void test_verifier_ld_ind(void) { RUN(verifier_ld_ind); }\n+void test_verifier_large_stack(void) { RUN(verifier_large_stack); }\n void test_verifier_ldsx(void) { RUN(verifier_ldsx); }\n void test_verifier_leak_ptr(void) { RUN(verifier_leak_ptr); }\n void test_verifier_linked_scalars(void) { RUN(verifier_linked_scalars); }\ndiff --git a/tools/testing/selftests/bpf/progs/async_stack_depth.c b/tools/testing/selftests/bpf/progs/async_stack_depth.c\nindex 36734683acbdb..9cd874a90b39a 100644\n--- a/tools/testing/selftests/bpf/progs/async_stack_depth.c\n+++ b/tools/testing/selftests/bpf/progs/async_stack_depth.c\n@@ -29,7 +29,49 @@ static int bad_timer_cb(void *map, int *key, struct bpf_timer *timer)\n \treturn buf[255] + timer_cb(NULL, NULL, NULL);\n }\n \n+/*\n+ * The same shapes scaled to the 2 KiB budget of JITs with large stacks. The\n+ * compiler caps a single function at 512 bytes, so the depth comes from a\n+ * chain of 480-byte frames.\n+ */\n+__attribute__((noinline))\n+static int timer_cb_large_0(void *map, int *key, struct bpf_timer *timer)\n+{\n+\tvolatile char buf[480] = {};\n+\treturn buf[69];\n+}\n+\n+__attribute__((noinline))\n+static int timer_cb_large_1(void *map, int *key, struct bpf_timer *timer)\n+{\n+\tvolatile char buf[480] = {};\n+\treturn buf[69] + timer_cb_large_0(map, key, timer);\n+}\n+\n+__attribute__((noinline))\n+static int timer_cb_large_2(void *map, int *key, struct bpf_timer *timer)\n+{\n+\tvolatile char buf[480] = {};\n+\treturn buf[69] + timer_cb_large_1(map, key, timer);\n+}\n+\n+__attribute__((noinline))\n+static int timer_cb_large_3(void *map, int *key, struct bpf_timer *timer)\n+{\n+\tvolatile char buf[480] = {};\n+\treturn buf[69] + timer_cb_large_2(map, key, timer);\n+}\n+\n+/* 5 * 480 = 2400 bytes on its own */\n+__attribute__((noinline))\n+static int bad_timer_cb_large(void *map, int *key, struct bpf_timer *timer)\n+{\n+\tvolatile char buf[480] = {};\n+\treturn buf[255] + timer_cb_large_3(map, key, timer);\n+}\n+\n SEC(\"tc\")\n+__load_if_no_large_stack()\n __failure __msg(\"combined stack size of 2 calls is\")\n int pseudo_call_check(struct __sk_buff *ctx)\n {\n@@ -44,7 +86,25 @@ int pseudo_call_check(struct __sk_buff *ctx)\n \treturn bpf_timer_set_callback(\u0026elem-\u003etimer, timer_cb) + buf[0];\n }\n \n+/* main plus the four frames under timer_cb_large_3: 2400 bytes */\n SEC(\"tc\")\n+__load_if_large_stack()\n+__failure __msg(\"combined stack size of 5 calls is\")\n+int pseudo_call_check_large(struct __sk_buff *ctx)\n+{\n+\tstruct hmap_elem *elem;\n+\tvolatile char buf[480] = {};\n+\n+\telem = bpf_map_lookup_elem(\u0026hmap, \u0026(int){0});\n+\tif (!elem)\n+\t\treturn 0;\n+\n+\ttimer_cb_large_3(NULL, NULL, NULL);\n+\treturn bpf_timer_set_callback(\u0026elem-\u003etimer, timer_cb_large_3) + buf[0];\n+}\n+\n+SEC(\"tc\")\n+__load_if_no_large_stack()\n __failure __msg(\"combined stack size of 2 calls is\")\n int async_call_root_check(struct __sk_buff *ctx)\n {\n@@ -58,4 +118,19 @@ int async_call_root_check(struct __sk_buff *ctx)\n \treturn bpf_timer_set_callback(\u0026elem-\u003etimer, bad_timer_cb) + buf[0];\n }\n \n+SEC(\"tc\")\n+__load_if_large_stack()\n+__failure __msg(\"combined stack size of 5 calls is\")\n+int async_call_root_check_large(struct __sk_buff *ctx)\n+{\n+\tstruct hmap_elem *elem;\n+\tvolatile char buf[480] = {};\n+\n+\telem = bpf_map_lookup_elem(\u0026hmap, \u0026(int){0});\n+\tif (!elem)\n+\t\treturn 0;\n+\n+\treturn bpf_timer_set_callback(\u0026elem-\u003etimer, bad_timer_cb_large) + buf[0];\n+}\n+\n char _license[] SEC(\"license\") = \"GPL\";\ndiff --git a/tools/testing/selftests/bpf/progs/bpf_misc.h b/tools/testing/selftests/bpf/progs/bpf_misc.h\nindex 2ced1d751acea..f3dbc3b59bff2 100644\n--- a/tools/testing/selftests/bpf/progs/bpf_misc.h\n+++ b/tools/testing/selftests/bpf/progs/bpf_misc.h\n@@ -175,6 +175,9 @@\n #define __prepare_priv\t\t__test_tag(\"test_prepare_priv\")\n #define __load_if_JITed()\t__test_tag(\"load_mode=jited\")\n #define __load_if_no_JITed()\t__test_tag(\"load_mode=no_jited\")\n+/* Whether programs may use more than 512 bytes of stack on this kernel and JIT */\n+#define __load_if_large_stack()\t\t__test_tag(\"stack_mode=large\")\n+#define __load_if_no_large_stack()\t__test_tag(\"stack_mode=small\")\n #define __stderr(msg)\t\t__test_tag(\"test_expect_stderr=\" msg)\n #define __stderr_unpriv(msg)\t__test_tag(\"test_expect_stderr_unpriv=\" msg)\n #define __stdout(msg)\t\t__test_tag(\"test_expect_stdout=\" msg)\ndiff --git a/tools/testing/selftests/bpf/progs/struct_ops_private_stack_fail.c b/tools/testing/selftests/bpf/progs/struct_ops_private_stack_fail.c\nindex 1442728f56046..c8cb35b37867b 100644\n--- a/tools/testing/selftests/bpf/progs/struct_ops_private_stack_fail.c\n+++ b/tools/testing/selftests/bpf/progs/struct_ops_private_stack_fail.c\n@@ -4,6 +4,7 @@\n #include \u003cbpf/bpf_helpers.h\u003e\n #include \u003cbpf/bpf_tracing.h\u003e\n #include \"../test_kmods/bpf_testmod.h\"\n+#include \"bpf_misc.h\"\n \n char _license[] SEC(\"license\") = \"GPL\";\n \n@@ -25,6 +26,44 @@ __noinline static int subprog1(int *a)\n \treturn subprog2(a, b);\n }\n \n+/*\n+ * A chain of 480-byte frames under test_2, so that its call chain exceeds\n+ * the 2 KiB budget of JITs with large stacks as well as the 512 bytes\n+ * allowed elsewhere. The compiler caps a single function at 512 bytes, and\n+ * the buffers are volatile so that it cannot shrink them.\n+ */\n+__noinline static int subprog_deep4(int *a)\n+{\n+\tvolatile char b[480] = {};\n+\n+\t__sink(b[479]);\n+\treturn a[10] + b[20];\n+}\n+\n+__noinline static int subprog_deep3(int *a)\n+{\n+\tvolatile char b[480] = {};\n+\n+\t__sink(b[479]);\n+\treturn subprog_deep4(a) + b[20];\n+}\n+\n+__noinline static int subprog_deep2(int *a)\n+{\n+\tvolatile char b[480] = {};\n+\n+\t__sink(b[479]);\n+\treturn subprog_deep3(a) + b[20];\n+}\n+\n+__noinline static int subprog_deep1(int *a)\n+{\n+\tvolatile char b[480] = {};\n+\n+\t__sink(b[479]);\n+\treturn subprog_deep2(a) + b[20];\n+}\n+\n \n SEC(\"struct_ops\")\n int BPF_PROG(test_1)\n@@ -41,11 +80,13 @@ int BPF_PROG(test_1)\n SEC(\"struct_ops\")\n int BPF_PROG(test_2)\n {\n-\t/* stack size 400 bytes */\n-\tint a[100] = {};\n+\t/* stack size 476 bytes, over 2 KiB with the four 480-byte deep subprogs */\n+\tvolatile char buf[376] = {};\n+\tint a[25] = {};\n \n+\t__sink(buf[375]);\n \ta[10] = 3;\n-\tval_j = subprog1(a);\n+\tval_j = subprog1(a) + subprog_deep1(a);\n \treturn 0;\n }\n \ndiff --git a/tools/testing/selftests/bpf/progs/struct_ops_private_stack_large.c b/tools/testing/selftests/bpf/progs/struct_ops_private_stack_large.c\nnew file mode 100644\nindex 0000000000000..94a25a2cff6ec\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/progs/struct_ops_private_stack_large.c\n@@ -0,0 +1,51 @@\n+// SPDX-License-Identifier: GPL-2.0\n+\n+#include \u003cvmlinux.h\u003e\n+#include \u003cbpf/bpf_helpers.h\u003e\n+#include \u003cbpf/bpf_tracing.h\u003e\n+#include \"../test_kmods/bpf_testmod.h\"\n+#include \"bpf_misc.h\"\n+\n+char _license[] SEC(\"license\") = \"GPL\";\n+\n+long val;\n+\n+/* On a private stack every frame gets the whole 2 KiB budget. */\n+__used __naked\n+static long frame_2048_leaf(void)\n+{\n+\tasm volatile (\"\t\t\t\t\t\\\n+\tr1 = 30;\t\t\t\t\t\\\n+\t*(u64 *)(r10 - 2048) = r1;\t\t\t\\\n+\tr1 = 12;\t\t\t\t\t\\\n+\t*(u64 *)(r10 - 8) = r1;\t\t\t\t\\\n+\tr0 = *(u64 *)(r10 - 2048);\t\t\t\\\n+\tr1 = *(u64 *)(r10 - 8);\t\t\t\t\\\n+\tr0 += r1;\t\t\t\t\t\\\n+\texit;\t\t\t\t\t\t\\\n+\"\t::: __clobber_all);\n+}\n+\n+/* test_1 is the member bpf_testmod requests a private stack for */\n+SEC(\"struct_ops\")\n+__naked int test_1(void)\n+{\n+\tasm volatile (\"\t\t\t\t\t\\\n+\tr1 = 100;\t\t\t\t\t\\\n+\t*(u64 *)(r10 - 2048) = r1;\t\t\t\\\n+\tcall frame_2048_leaf;\t\t\t\t\\\n+\tr1 = *(u64 *)(r10 - 2048);\t\t\t\\\n+\tr0 += r1;\t\t\t\t\t\\\n+\tr1 = %[val] ll;\t\t\t\t\t\\\n+\t*(u64 *)(r1 + 0) = r0;\t\t\t\t\\\n+\tr0 = 0;\t\t\t\t\t\t\\\n+\texit;\t\t\t\t\t\t\\\n+\"\t:\n+\t: __imm_addr(val)\n+\t: __clobber_all);\n+}\n+\n+SEC(\".struct_ops\")\n+struct bpf_testmod_ops3 testmod_1 = {\n+\t.test_1 = (void *)test_1,\n+};\ndiff --git a/tools/testing/selftests/bpf/progs/tailcall_large_stack.c b/tools/testing/selftests/bpf/progs/tailcall_large_stack.c\nnew file mode 100644\nindex 0000000000000..977197dac5d37\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/progs/tailcall_large_stack.c\n@@ -0,0 +1,62 @@\n+// SPDX-License-Identifier: GPL-2.0\n+#include \u003clinux/bpf.h\u003e\n+#include \u003cbpf/bpf_helpers.h\u003e\n+#include \"bpf_misc.h\"\n+\n+struct {\n+\t__uint(type, BPF_MAP_TYPE_PROG_ARRAY);\n+\t__uint(max_entries, 1);\n+\t__uint(key_size, sizeof(__u32));\n+\t__uint(value_size, sizeof(__u32));\n+} jmp_table SEC(\".maps\");\n+\n+/* The tail call target sets up a 2 KiB frame of its own and uses both of its ends. */\n+SEC(\"tc\")\n+__naked int classifier_0(void)\n+{\n+\tasm volatile (\"\t\t\t\t\t\\\n+\tr1 = 42;\t\t\t\t\t\\\n+\t*(u64 *)(r10 - 2048) = r1;\t\t\t\\\n+\tr1 = 7;\t\t\t\t\t\t\\\n+\t*(u64 *)(r10 - 8) = r1;\t\t\t\t\\\n+\tr0 = *(u64 *)(r10 - 2048);\t\t\t\\\n+\tr1 = *(u64 *)(r10 - 8);\t\t\t\t\\\n+\tr0 += r1;\t\t\t\t\t\\\n+\texit;\t\t\t\t\t\t\\\n+\"\t::: __clobber_all);\n+}\n+\n+/*\n+ * The frame of the subprog doing the tail call is unwound by it, so it may be\n+ * large; only the frames of its callers stay behind and are limited to 256\n+ * bytes in total. Returns 1 when the tail call falls through.\n+ */\n+__used __naked\n+static int subprog_tail(void)\n+{\n+\tasm volatile (\"\t\t\t\t\t\\\n+\tr2 = 1;\t\t\t\t\t\t\\\n+\t*(u64 *)(r10 - 1536) = r2;\t\t\t\\\n+\tr2 = %[jmp_table] ll;\t\t\t\t\\\n+\tr3 = 0;\t\t\t\t\t\t\\\n+\tcall %[bpf_tail_call];\t\t\t\t\\\n+\tr0 = 1;\t\t\t\t\t\t\\\n+\texit;\t\t\t\t\t\t\\\n+\"\t:\n+\t: __imm(bpf_tail_call),\n+\t __imm_addr(jmp_table)\n+\t: __clobber_all);\n+}\n+\n+SEC(\"tc\")\n+__naked int entry(void)\n+{\n+\tasm volatile (\"\t\t\t\t\t\\\n+\tr2 = 2;\t\t\t\t\t\t\\\n+\t*(u64 *)(r10 - 240) = r2;\t\t\t\\\n+\tcall subprog_tail;\t\t\t\t\\\n+\texit;\t\t\t\t\t\t\\\n+\"\t::: __clobber_all);\n+}\n+\n+char _license[] SEC(\"license\") = \"GPL\";\ndiff --git a/tools/testing/selftests/bpf/progs/test_global_func1.c b/tools/testing/selftests/bpf/progs/test_global_func1.c\nindex fc69ff18880d5..f0eca282e0d43 100644\n--- a/tools/testing/selftests/bpf/progs/test_global_func1.c\n+++ b/tools/testing/selftests/bpf/progs/test_global_func1.c\n@@ -48,8 +48,73 @@ int f3(int val, struct __sk_buff *skb, int var)\n }\n \n SEC(\"tc\")\n+__load_if_no_large_stack()\n __failure __msg(\"combined stack size of 3 calls is\")\n int global_func1(struct __sk_buff *skb)\n {\n \treturn f0(1, skb) + f1(skb) + f2(2, skb) + f3(3, skb, 4);\n }\n+\n+/*\n+ * A chain of five frames that stay under 512 bytes each but add up to more\n+ * than the 2 KiB budget of JITs with large stacks; the chain also exceeds\n+ * 512 bytes after two frames, so it is rejected everywhere.\n+ */\n+#define MAX_STACK_LARGE 480\n+\n+__attribute__ ((noinline))\n+int g0(struct __sk_buff *skb)\n+{\n+\tvolatile char buf[MAX_STACK_LARGE] = {};\n+\n+\t__sink(buf[MAX_STACK_LARGE - 1]);\n+\n+\treturn skb-\u003elen;\n+}\n+\n+__attribute__ ((noinline))\n+int g1(struct __sk_buff *skb)\n+{\n+\tvolatile char buf[MAX_STACK_LARGE] = {};\n+\n+\t__sink(buf[MAX_STACK_LARGE - 1]);\n+\n+\treturn g0(skb) + skb-\u003elen;\n+}\n+\n+__attribute__ ((noinline))\n+int g2(struct __sk_buff *skb)\n+{\n+\tvolatile char buf[MAX_STACK_LARGE] = {};\n+\n+\t__sink(buf[MAX_STACK_LARGE - 1]);\n+\n+\treturn g1(skb) + skb-\u003elen;\n+}\n+\n+__attribute__ ((noinline))\n+int g3(struct __sk_buff *skb)\n+{\n+\tvolatile char buf[MAX_STACK_LARGE] = {};\n+\n+\t__sink(buf[MAX_STACK_LARGE - 1]);\n+\n+\treturn g2(skb) + skb-\u003elen;\n+}\n+\n+__attribute__ ((noinline))\n+int g4(struct __sk_buff *skb)\n+{\n+\tvolatile char buf[MAX_STACK_LARGE] = {};\n+\n+\t__sink(buf[MAX_STACK_LARGE - 1]);\n+\n+\treturn g3(skb) + skb-\u003elen;\n+}\n+\n+SEC(\"tc\")\n+__failure __msg(\"combined stack size of {{[0-9]+}} calls is\")\n+int global_func1_deep(struct __sk_buff *skb)\n+{\n+\treturn g4(skb);\n+}\ndiff --git a/tools/testing/selftests/bpf/progs/test_global_func_deep_stack.c b/tools/testing/selftests/bpf/progs/test_global_func_deep_stack.c\nindex 1b634b543b629..edb8a223a3cb2 100644\n--- a/tools/testing/selftests/bpf/progs/test_global_func_deep_stack.c\n+++ b/tools/testing/selftests/bpf/progs/test_global_func_deep_stack.c\n@@ -67,12 +67,30 @@ int XCAT(f, n)(unsigned long a) \\\n #define F_31 F_30 FN(31, 30)\n #define F_32 F_31 FN(32, 31)\n \n+/* Same, with a 480-byte frame, to exceed the 2 KiB budget of large stacks. */\n+#define FNB(n, prev) \\\n+__attribute__((noinline)) \\\n+int XCAT(f, n)(unsigned long a) \\\n+{ \\\n+\tvolatile char buf[480] = {}; \\\n+\tvolatile long b = XCAT(f, prev)(a - 1); \\\n+\tif (!b) \\\n+\t\treturn 0; \\\n+\treturn b + buf[479] + 1; \\\n+}\n+\n+#define F_33 F_32 FNB(33, 32)\n+#define F_34 F_33 FNB(34, 33)\n+#define F_35 F_34 FNB(35, 34)\n+#define F_36 F_35 FNB(36, 35)\n+#define F_37 F_36 FNB(37, 36)\n+\n #define CAT2(a, b) a ## b\n #define XCAT2(a, b) CAT2(a, b)\n \n #define F(n) XCAT2(F_, n)\n \n-F(32)\n+F(37)\n \n /* Ensure that even 32 levels deep, the function verifies. */\n SEC(\"syscall\")\n@@ -88,8 +106,21 @@ int global_func_deep_stack_success(struct __sk_buff *skb)\n * the size.\n */\n SEC(\"syscall\")\n+__load_if_no_large_stack()\n __failure __msg(\"combined stack size of 34 calls\")\n int global_func_deep_stack_fail(struct __sk_buff *skb)\n {\n \treturn f32(123);\n }\n+\n+/*\n+ * Five 480-byte frames on top of the chain: 5 * 480 + 33 * 16 = 2928 bytes,\n+ * more than the 2 KiB budget of JITs with large stacks, and more than 512\n+ * bytes after the second frame everywhere else.\n+ */\n+SEC(\"syscall\")\n+__failure __msg(\"combined stack size of {{[0-9]+}} calls\")\n+int global_func_deep_stack_fail_large(struct __sk_buff *skb)\n+{\n+\treturn f37(123);\n+}\ndiff --git a/tools/testing/selftests/bpf/progs/verifier_large_stack.c b/tools/testing/selftests/bpf/progs/verifier_large_stack.c\nnew file mode 100644\nindex 0000000000000..2d4c81a3f0cb9\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/progs/verifier_large_stack.c\n@@ -0,0 +1,377 @@\n+// SPDX-License-Identifier: GPL-2.0\n+\n+#include \u003clinux/bpf.h\u003e\n+#include \u003cbpf/bpf_helpers.h\u003e\n+#include \"bpf_misc.h\"\n+\n+/*\n+ * Programs may use MAX_BPF_STACK_JIT (2 KiB) of stack on JITs that support\n+ * large stacks, combined over a call chain, with no separate limit on a\n+ * single frame. Interpreted programs and other JITs keep 512 bytes.\n+ */\n+\n+SEC(\"socket\")\n+__description(\"single frame of 2048 bytes\")\n+__load_if_large_stack()\n+__success __success_unpriv __retval(42)\n+__naked void single_frame_2048(void)\n+{\n+\tasm volatile (\"\t\t\t\t\t\\\n+\tr1 = r10;\t\t\t\t\t\\\n+\tr1 += -2048;\t\t\t\t\t\\\n+\tr0 = 42;\t\t\t\t\t\\\n+\t*(u64*)(r1 + 0) = r0;\t\t\t\t\\\n+\tr0 = *(u64*)(r1 + 0);\t\t\t\t\\\n+\texit;\t\t\t\t\t\t\\\n+\"\t::: __clobber_all);\n+}\n+\n+SEC(\"socket\")\n+__description(\"single frame of 2048 bytes without large stack support\")\n+__load_if_no_large_stack()\n+__failure __msg(\"invalid write to stack R1 off=-2048 size=8\")\n+__naked void single_frame_2048_no_large_stack(void)\n+{\n+\tasm volatile (\"\t\t\t\t\t\\\n+\tr1 = r10;\t\t\t\t\t\\\n+\tr1 += -2048;\t\t\t\t\t\\\n+\tr0 = 42;\t\t\t\t\t\\\n+\t*(u64*)(r1 + 0) = r0;\t\t\t\t\\\n+\texit;\t\t\t\t\t\t\\\n+\"\t::: __clobber_all);\n+}\n+\n+__used __naked\n+static void frame_512_leaf(void)\n+{\n+\tasm volatile (\"\t\t\t\t\t\\\n+\tr1 = 1;\t\t\t\t\t\t\\\n+\t*(u64 *)(r10 - 512) = r1;\t\t\t\\\n+\texit;\t\t\t\t\t\t\\\n+\"\t::: __clobber_all);\n+}\n+\n+__used __naked\n+static void frame_512_depth_2(void)\n+{\n+\tasm volatile (\"\t\t\t\t\t\\\n+\tr1 = 2;\t\t\t\t\t\t\\\n+\t*(u64 *)(r10 - 512) = r1;\t\t\t\\\n+\tcall frame_512_leaf;\t\t\t\t\\\n+\texit;\t\t\t\t\t\t\\\n+\"\t::: __clobber_all);\n+}\n+\n+__used __naked\n+static void frame_512_depth_3(void)\n+{\n+\tasm volatile (\"\t\t\t\t\t\\\n+\tr1 = 3;\t\t\t\t\t\t\\\n+\t*(u64 *)(r10 - 512) = r1;\t\t\t\\\n+\tcall frame_512_depth_2;\t\t\t\t\\\n+\texit;\t\t\t\t\t\t\\\n+\"\t::: __clobber_all);\n+}\n+\n+__used __naked\n+static void frame_512_depth_4(void)\n+{\n+\tasm volatile (\"\t\t\t\t\t\\\n+\tr1 = 4;\t\t\t\t\t\t\\\n+\t*(u64 *)(r10 - 512) = r1;\t\t\t\\\n+\tcall frame_512_depth_3;\t\t\t\t\\\n+\texit;\t\t\t\t\t\t\\\n+\"\t::: __clobber_all);\n+}\n+\n+SEC(\"socket\")\n+__description(\"four frames of 512 bytes fit the 2 KiB budget\")\n+__load_if_large_stack()\n+__success __log_level(4) __msg(\"stack depth max 2048\")\n+__naked void four_frames_of_512(void)\n+{\n+\tasm volatile (\"\t\t\t\t\t\\\n+\tcall frame_512_depth_4;\t\t\t\t\\\n+\tr0 = 0;\t\t\t\t\t\t\\\n+\texit;\t\t\t\t\t\t\\\n+\"\t::: __clobber_all);\n+}\n+\n+SEC(\"socket\")\n+__description(\"five frames of 512 bytes exceed the 2 KiB budget\")\n+__load_if_large_stack()\n+__failure __msg(\"combined stack size of 5 calls is 2560. Too large\")\n+__naked void five_frames_of_512(void)\n+{\n+\tasm volatile (\"\t\t\t\t\t\\\n+\tr1 = 5;\t\t\t\t\t\t\\\n+\t*(u64 *)(r10 - 512) = r1;\t\t\t\\\n+\tcall frame_512_depth_4;\t\t\t\t\\\n+\tr0 = 0;\t\t\t\t\t\t\\\n+\texit;\t\t\t\t\t\t\\\n+\"\t::: __clobber_all);\n+}\n+\n+__used __naked\n+static void frame_1536_leaf(void)\n+{\n+\tasm volatile (\"\t\t\t\t\t\\\n+\tr1 = 1;\t\t\t\t\t\t\\\n+\t*(u64 *)(r10 - 1536) = r1;\t\t\t\\\n+\texit;\t\t\t\t\t\t\\\n+\"\t::: __clobber_all);\n+}\n+\n+SEC(\"socket\")\n+__description(\"512-byte frame calling a 1536-byte frame\")\n+__load_if_large_stack()\n+__success __log_level(4) __msg(\"stack depth max 2048\")\n+__naked void uneven_frames_fit(void)\n+{\n+\tasm volatile (\"\t\t\t\t\t\\\n+\tr1 = 2;\t\t\t\t\t\t\\\n+\t*(u64 *)(r10 - 512) = r1;\t\t\t\\\n+\tcall frame_1536_leaf;\t\t\t\t\\\n+\tr0 = 0;\t\t\t\t\t\t\\\n+\texit;\t\t\t\t\t\t\\\n+\"\t::: __clobber_all);\n+}\n+\n+SEC(\"socket\")\n+__description(\"520-byte frame calling a 1536-byte frame\")\n+__load_if_large_stack()\n+__failure __msg(\"combined stack size of 2 calls is 2064. Too large\")\n+__naked void uneven_frames_exceed(void)\n+{\n+\tasm volatile (\"\t\t\t\t\t\\\n+\tr1 = 2;\t\t\t\t\t\t\\\n+\t*(u64 *)(r10 - 520) = r1;\t\t\t\\\n+\tcall frame_1536_leaf;\t\t\t\t\\\n+\tr0 = 0;\t\t\t\t\t\t\\\n+\texit;\t\t\t\t\t\t\\\n+\"\t::: __clobber_all);\n+}\n+\n+#ifdef __BPF_FEATURE_MAY_GOTO\n+/* may_goto adds its counter below the frame; a JIT does not hold that against the budget */\n+SEC(\"socket\")\n+__description(\"frame of 2048 bytes with may_goto\")\n+__load_if_large_stack()\n+__success __retval(42)\n+__naked void frame_2048_with_may_goto(void)\n+{\n+\tasm volatile (\"\t\t\t\t\t\\\n+\tr1 = r10;\t\t\t\t\t\\\n+\tr1 += -2048;\t\t\t\t\t\\\n+\tr0 = 42;\t\t\t\t\t\\\n+\t*(u32*)(r1 + 0) = r0;\t\t\t\t\\\n+\tmay_goto l0_%=;\t\t\t\t\t\\\n+\tr2 = 100;\t\t\t\t\t\\\n+\tl0_%=:\t\t\t\t\t\t\\\n+\texit;\t\t\t\t\t\t\\\n+\"\t::: __clobber_all);\n+}\n+#endif\n+\n+SEC(\"socket\")\n+__description(\"variable offset write reaching 2048 bytes deep\")\n+__load_if_large_stack()\n+__success\n+__naked void var_off_write_to_2048(void)\n+{\n+\tasm volatile (\"\t\t\t\t\t\\\n+\tcall %[bpf_get_prandom_u32];\t\t\t\\\n+\tr0 \u0026= 8;\t\t\t\t\t\\\n+\tr2 = r10;\t\t\t\t\t\\\n+\tr2 += -2048;\t\t\t\t\t\\\n+\tr2 += r0;\t\t\t\t\t\\\n+\tr1 = 0;\t\t\t\t\t\t\\\n+\t*(u64*)(r2 + 0) = r1;\t\t\t\t\\\n+\tr0 = 0;\t\t\t\t\t\t\\\n+\texit;\t\t\t\t\t\t\\\n+\"\t:\n+\t: __imm(bpf_get_prandom_u32)\n+\t: __clobber_all);\n+}\n+\n+SEC(\"socket\")\n+__description(\"variable offset write reaching 2056 bytes deep\")\n+__load_if_large_stack()\n+__failure __msg(\"invalid variable-offset write to stack R2\")\n+__naked void var_off_write_to_2056(void)\n+{\n+\tasm volatile (\"\t\t\t\t\t\\\n+\tcall %[bpf_get_prandom_u32];\t\t\t\\\n+\tr0 \u0026= 8;\t\t\t\t\t\\\n+\tr2 = r10;\t\t\t\t\t\\\n+\tr2 += -2056;\t\t\t\t\t\\\n+\tr2 += r0;\t\t\t\t\t\\\n+\tr1 = 0;\t\t\t\t\t\t\\\n+\t*(u64*)(r2 + 0) = r1;\t\t\t\t\\\n+\tr0 = 0;\t\t\t\t\t\t\\\n+\texit;\t\t\t\t\t\t\\\n+\"\t:\n+\t: __imm(bpf_get_prandom_u32)\n+\t: __clobber_all);\n+}\n+\n+/* Each frame of a private stack gets the whole budget. */\n+__used __naked\n+static void priv_stack_frame_2048(void)\n+{\n+\tasm volatile (\"\t\t\t\t\t\\\n+\tr1 = 1;\t\t\t\t\t\t\\\n+\t*(u64 *)(r10 - 2048) = r1;\t\t\t\\\n+\texit;\t\t\t\t\t\t\\\n+\"\t::: __clobber_all);\n+}\n+\n+SEC(\"kprobe\")\n+__description(\"private stack: two frames of 2048 bytes\")\n+__load_if_large_stack()\n+__arch_x86_64\n+__arch_arm64\n+__success __log_level(4)\n+__msg(\"stack depth max 2048\")\n+__msg(\"subprog 0 (private_stack_two_frames) main {{.*}} stack 2048\")\n+__msg(\"subprog 1 (priv_stack_frame_2048) static {{.*}} stack 2048\")\n+__naked void private_stack_two_frames(void)\n+{\n+\tasm volatile (\"\t\t\t\t\t\\\n+\tr1 = 2;\t\t\t\t\t\t\\\n+\t*(u64 *)(r10 - 2048) = r1;\t\t\t\\\n+\tcall priv_stack_frame_2048;\t\t\t\\\n+\tr0 = 0;\t\t\t\t\t\t\\\n+\texit;\t\t\t\t\t\t\\\n+\"\t::: __clobber_all);\n+}\n+\n+struct {\n+\t__uint(type, BPF_MAP_TYPE_PROG_ARRAY);\n+\t__uint(max_entries, 1);\n+\t__uint(key_size, sizeof(__u32));\n+\t__uint(value_size, sizeof(__u32));\n+} jmp_table SEC(\".maps\");\n+\n+/*\n+ * A tail call unwinds the frame of the program doing it, so a large main\n+ * frame is fine; the 256-byte rule only concerns the frames of callers of a\n+ * subprog that tail calls.\n+ */\n+SEC(\"tc\")\n+__description(\"tail call from a 1 KiB frame\")\n+__load_if_large_stack()\n+__success\n+__naked void tail_call_from_large_frame(void)\n+{\n+\tasm volatile (\"\t\t\t\t\t\\\n+\tr2 = 42;\t\t\t\t\t\\\n+\t*(u64 *)(r10 - 1024) = r2;\t\t\t\\\n+\tr2 = %[jmp_table] ll;\t\t\t\t\\\n+\tr3 = 0;\t\t\t\t\t\t\\\n+\tcall %[bpf_tail_call];\t\t\t\t\\\n+\tr0 = 0;\t\t\t\t\t\t\\\n+\texit;\t\t\t\t\t\t\\\n+\"\t:\n+\t: __imm(bpf_tail_call),\n+\t __imm_addr(jmp_table)\n+\t: __clobber_all);\n+}\n+\n+/* Global subprogs are verified on their own but share the call chain budget. */\n+__used __naked int global_frame_1536(void)\n+{\n+\tasm volatile (\"\t\t\t\t\t\\\n+\tr1 = 1;\t\t\t\t\t\t\\\n+\t*(u64 *)(r10 - 1536) = r1;\t\t\t\\\n+\tr0 = 0;\t\t\t\t\t\t\\\n+\texit;\t\t\t\t\t\t\\\n+\"\t::: __clobber_all);\n+}\n+\n+SEC(\"socket\")\n+__description(\"512-byte frame calling a 1536-byte global subprog\")\n+__load_if_large_stack()\n+__success __log_level(4) __msg(\"stack depth max 2048\")\n+__naked void global_subprog_fits(void)\n+{\n+\tasm volatile (\"\t\t\t\t\t\\\n+\tr1 = 2;\t\t\t\t\t\t\\\n+\t*(u64 *)(r10 - 512) = r1;\t\t\t\\\n+\tcall global_frame_1536;\t\t\t\t\\\n+\tr0 = 0;\t\t\t\t\t\t\\\n+\texit;\t\t\t\t\t\t\\\n+\"\t::: __clobber_all);\n+}\n+\n+SEC(\"socket\")\n+__description(\"520-byte frame calling a 1536-byte global subprog\")\n+__load_if_large_stack()\n+__failure __msg(\"combined stack size of 2 calls is 2064. Too large\")\n+__naked void global_subprog_exceeds(void)\n+{\n+\tasm volatile (\"\t\t\t\t\t\\\n+\tr1 = 2;\t\t\t\t\t\t\\\n+\t*(u64 *)(r10 - 520) = r1;\t\t\t\\\n+\tcall global_frame_1536;\t\t\t\t\\\n+\tr0 = 0;\t\t\t\t\t\t\\\n+\texit;\t\t\t\t\t\t\\\n+\"\t::: __clobber_all);\n+}\n+\n+/* Callback frames are part of the chain of the helper that calls them. */\n+static __naked int loop_cb_1536(void)\n+{\n+\tasm volatile (\"\t\t\t\t\t\\\n+\tr1 = 1;\t\t\t\t\t\t\\\n+\t*(u64 *)(r10 - 1536) = r1;\t\t\t\\\n+\tr0 = 0;\t\t\t\t\t\t\\\n+\texit;\t\t\t\t\t\t\\\n+\"\t::: __clobber_all);\n+}\n+\n+SEC(\"socket\")\n+__description(\"512-byte frame with a 1536-byte bpf_loop callback\")\n+__load_if_large_stack()\n+__success __log_level(4) __msg(\"stack depth max 2048\")\n+__naked void loop_callback_fits(void)\n+{\n+\tasm volatile (\"\t\t\t\t\t\\\n+\tr1 = 2;\t\t\t\t\t\t\\\n+\t*(u64 *)(r10 - 512) = r1;\t\t\t\\\n+\tr1 = 1;\t\t\t\t\t\t\\\n+\tr2 = %[loop_cb_1536];\t\t\t\t\\\n+\tr3 = 0;\t\t\t\t\t\t\\\n+\tr4 = 0;\t\t\t\t\t\t\\\n+\tcall %[bpf_loop];\t\t\t\t\\\n+\tr0 = 0;\t\t\t\t\t\t\\\n+\texit;\t\t\t\t\t\t\\\n+\"\t:\n+\t: __imm_ptr(loop_cb_1536),\n+\t __imm(bpf_loop)\n+\t: __clobber_common);\n+}\n+\n+SEC(\"socket\")\n+__description(\"520-byte frame with a 1536-byte bpf_loop callback\")\n+__load_if_large_stack()\n+__failure __msg(\"combined stack size of 2 calls is 2064. Too large\")\n+__naked void loop_callback_exceeds(void)\n+{\n+\tasm volatile (\"\t\t\t\t\t\\\n+\tr1 = 2;\t\t\t\t\t\t\\\n+\t*(u64 *)(r10 - 520) = r1;\t\t\t\\\n+\tr1 = 1;\t\t\t\t\t\t\\\n+\tr2 = %[loop_cb_1536];\t\t\t\t\\\n+\tr3 = 0;\t\t\t\t\t\t\\\n+\tr4 = 0;\t\t\t\t\t\t\\\n+\tcall %[bpf_loop];\t\t\t\t\\\n+\tr0 = 0;\t\t\t\t\t\t\\\n+\texit;\t\t\t\t\t\t\\\n+\"\t:\n+\t: __imm_ptr(loop_cb_1536),\n+\t __imm(bpf_loop)\n+\t: __clobber_common);\n+}\n+\n+char _license[] SEC(\"license\") = \"GPL\";\ndiff --git a/tools/testing/selftests/bpf/progs/verifier_live_stack.c b/tools/testing/selftests/bpf/progs/verifier_live_stack.c\nindex 7a1a0670f851d..7be813f06a5ae 100644\n--- a/tools/testing/selftests/bpf/progs/verifier_live_stack.c\n+++ b/tools/testing/selftests/bpf/progs/verifier_live_stack.c\n@@ -318,7 +318,7 @@ struct {\n } map_array SEC(\".maps\");\n \n SEC(\"socket\")\n-__failure __msg(\"invalid read from stack R2 off=-1024 size=8\")\n+__failure __msg(\"invalid read from stack R2 off=-4096 size=8\")\n __flag(BPF_F_TEST_STATE_FREQ)\n __naked unsigned long caller_stack_write_tail_call(void)\n {\n@@ -329,7 +329,7 @@ __naked unsigned long caller_stack_write_tail_call(void)\n \"if r0 != 42 goto 1f;\"\n \"goto 2f;\"\n \"1:\"\n- \"*(u64 *)(r10 - 8) = -1024;\"\n+ \"*(u64 *)(r10 - 8) = -4096;\"\n \"2:\"\n \"r1 = r6;\"\n \"r2 = r10;\"\n@@ -1953,7 +1953,7 @@ static __used __naked void fwd_parent_key_to_helper(void)\n SEC(\"socket\")\n __log_level(2)\n __success\n-__msg(\"call bpf_map_update_elem{{.*}}; use: fp1-8..-512 fp0-8\")\n+__msg(\"call bpf_map_update_elem{{.*}}; use: fp1-8..-{{(512|2048)}} fp0-8\")\n __naked void helper_arg_fallback_keeps_scanning(void)\n {\n \tasm volatile (\n@@ -2267,7 +2267,7 @@ static __used __naked void merge_leaf_read(void)\n SEC(\"socket\")\n __log_level(2)\n __success\n-__msg(\"call bpf_loop#181 ; use: fp2-8..-512 fp1-8..-512 fp0-8..-512\")\n+__msg(\"call bpf_loop#181 ; use: fp2-8..-{{(512|2048)}} fp1-8..-{{(512|2048)}} fp0-8..-{{(512|2048)}}\")\n __naked void bpf_loop_two_callbacks(void)\n {\n \tasm volatile (\n@@ -2839,3 +2839,74 @@ static __used __naked void imprecise_dst_spill_join_sub(void)\n \t:: __imm(bpf_get_prandom_u32)\n \t: __clobber_all);\n }\n+\n+/*\n+ * A store that does not fully cover a 4-byte half-slot defines nothing, so a\n+ * narrow store at the top of the frame must not turn any slot into a \"def\",\n+ * least of all every slot of the frame: the earlier data at fp-8 stays live.\n+ */\n+SEC(\"socket\")\n+__log_level(2)\n+__msg(\"0: (79) r0 = *(u64 *)(r10 -8) ; use: fp0-8\")\n+__msg(\"1: (73) *(u8 *)(r10 -1) = r0{{$}}\")\n+__msg(\"2: (6b) *(u16 *)(r10 -4) = r0{{$}}\")\n+__msg(\"3: (79) r0 = *(u64 *)(r10 -8) ; use: fp0-8\")\n+__naked void narrow_store_defines_nothing(void)\n+{\n+\tasm volatile (\n+\t\"r0 = *(u64 *)(r10 - 8);\"\n+\t\"*(u8 *)(r10 - 1) = r0;\"\n+\t\"*(u16 *)(r10 - 4) = r0;\"\n+\t\"r0 = *(u64 *)(r10 - 8);\"\n+\t\"exit;\"\n+\t::: __clobber_all);\n+}\n+\n+/*\n+ * The same callee instance is analyzed twice: the call sites are visited in\n+ * postorder, so the second one goes first with a precise pointer 264 bytes\n+ * into the main frame, and the first one then passes a pointer of unknown\n+ * offset, which reads the whole frame. The masks of the two passes differ in\n+ * width; merging the second into the first must keep the whole-frame read.\n+ */\n+SEC(\"socket\")\n+__log_level(2)\n+__msg(\"stack use/def subprog#{{[0-9]+}} merge_read_all_callee (d2,cs{{[0-9]+}}):\")\n+__msg(\"(79) r0 = *(u64 *)(r1 +0){{.*}}; use: fp0-8..-{{(512|2048)}}\")\n+__naked void merge_keeps_whole_frame_read(void)\n+{\n+\tasm volatile (\n+\t\"r1 = 0;\"\n+\t\"*(u64 *)(r10 - 8) = r1;\"\n+\t\"*(u64 *)(r10 - 16) = r1;\"\n+\t\"*(u64 *)(r10 - 264) = r1;\"\n+\t\"call %[bpf_get_prandom_u32];\"\n+\t\"r0 \u0026= 8;\"\n+\t\"r1 = r10;\"\n+\t\"r1 += -16;\"\n+\t\"r1 += r0;\"\n+\t\"call merge_read_all_mid;\"\n+\t\"r1 = r10;\"\n+\t\"r1 += -264;\"\n+\t\"call merge_read_all_mid;\"\n+\t\"r0 = 0;\"\n+\t\"exit;\"\n+\t:: __imm(bpf_get_prandom_u32)\n+\t: __clobber_all);\n+}\n+\n+static __used __naked void merge_read_all_mid(void)\n+{\n+\tasm volatile (\n+\t\"call merge_read_all_callee;\"\n+\t\"exit;\"\n+\t::: __clobber_all);\n+}\n+\n+static __used __naked void merge_read_all_callee(void)\n+{\n+\tasm volatile (\n+\t\"r0 = *(u64 *)(r1 + 0);\"\n+\t\"exit;\"\n+\t::: __clobber_all);\n+}\ndiff --git a/tools/testing/selftests/bpf/progs/verifier_raw_stack.c b/tools/testing/selftests/bpf/progs/verifier_raw_stack.c\nindex 9f0f48ecb4216..0fe631411b9cc 100644\n--- a/tools/testing/selftests/bpf/progs/verifier_raw_stack.c\n+++ b/tools/testing/selftests/bpf/progs/verifier_raw_stack.c\n@@ -240,6 +240,7 @@ __naked void load_bytes_spilled_regs_data(void)\n \n SEC(\"tc\")\n __description(\"raw_stack: skb_load_bytes, invalid access 1\")\n+__load_if_no_large_stack()\n __failure __msg(\"invalid write to stack R3 off=-513 size=8\")\n __naked void load_bytes_invalid_access_1(void)\n {\n@@ -257,6 +258,26 @@ __naked void load_bytes_invalid_access_1(void)\n \t: __clobber_all);\n }\n \n+SEC(\"tc\")\n+__description(\"raw_stack: skb_load_bytes, invalid access 1, large stack\")\n+__load_if_large_stack()\n+__failure __msg(\"invalid write to stack R3 off=-2049 size=8\")\n+__naked void load_bytes_invalid_access_1_large(void)\n+{\n+\tasm volatile (\"\t\t\t\t\t\\\n+\tr2 = 4;\t\t\t\t\t\t\\\n+\tr6 = r10;\t\t\t\t\t\\\n+\tr6 += -2049;\t\t\t\t\t\\\n+\tr3 = r6;\t\t\t\t\t\\\n+\tr4 = 8;\t\t\t\t\t\t\\\n+\tcall %[bpf_skb_load_bytes];\t\t\t\\\n+\tr0 = *(u64*)(r6 + 0);\t\t\t\t\\\n+\texit;\t\t\t\t\t\t\\\n+\"\t:\n+\t: __imm(bpf_skb_load_bytes)\n+\t: __clobber_all);\n+}\n+\n SEC(\"tc\")\n __description(\"raw_stack: skb_load_bytes, invalid access 2\")\n __failure __msg(\"invalid write to stack R3 off=-1 size=8\")\ndiff --git a/tools/testing/selftests/bpf/progs/verifier_stack_ptr.c b/tools/testing/selftests/bpf/progs/verifier_stack_ptr.c\nindex 8e8cf8232255f..3e0bea9819cab 100644\n--- a/tools/testing/selftests/bpf/progs/verifier_stack_ptr.c\n+++ b/tools/testing/selftests/bpf/progs/verifier_stack_ptr.c\n@@ -235,6 +235,7 @@ __naked void to_stack_check_low_1(void)\n \n SEC(\"socket\")\n __description(\"PTR_TO_STACK check low 2\")\n+__load_if_no_large_stack()\n __success __failure_unpriv\n __msg_unpriv(\"R1 stack pointer arithmetic goes out of range\")\n __retval(42)\n@@ -250,8 +251,27 @@ __naked void to_stack_check_low_2(void)\n \"\t::: __clobber_all);\n }\n \n+SEC(\"socket\")\n+__description(\"PTR_TO_STACK check low 2, large stack\")\n+__load_if_large_stack()\n+__success __failure_unpriv\n+__msg_unpriv(\"R1 stack pointer arithmetic goes out of range\")\n+__retval(42)\n+__naked void to_stack_check_low_2_large(void)\n+{\n+\tasm volatile (\"\t\t\t\t\t\\\n+\tr1 = r10;\t\t\t\t\t\\\n+\tr1 += -2049;\t\t\t\t\t\\\n+\tr0 = 42;\t\t\t\t\t\\\n+\t*(u8*)(r1 + 1) = r0;\t\t\t\t\\\n+\tr0 = *(u8*)(r1 + 1);\t\t\t\t\\\n+\texit;\t\t\t\t\t\t\\\n+\"\t::: __clobber_all);\n+}\n+\n SEC(\"socket\")\n __description(\"PTR_TO_STACK check low 3\")\n+__load_if_no_large_stack()\n __failure __msg(\"invalid write to stack R1 off=-513 size=1\")\n __msg_unpriv(\"R1 stack pointer arithmetic goes out of range\")\n __naked void to_stack_check_low_3(void)\n@@ -266,6 +286,23 @@ __naked void to_stack_check_low_3(void)\n \"\t::: __clobber_all);\n }\n \n+SEC(\"socket\")\n+__description(\"PTR_TO_STACK check low 3, large stack\")\n+__load_if_large_stack()\n+__failure __msg(\"invalid write to stack R1 off=-2049 size=1\")\n+__msg_unpriv(\"R1 stack pointer arithmetic goes out of range\")\n+__naked void to_stack_check_low_3_large(void)\n+{\n+\tasm volatile (\"\t\t\t\t\t\\\n+\tr1 = r10;\t\t\t\t\t\\\n+\tr1 += -2049;\t\t\t\t\t\\\n+\tr0 = 42;\t\t\t\t\t\\\n+\t*(u8*)(r1 + 0) = r0;\t\t\t\t\\\n+\tr0 = *(u8*)(r1 + 0);\t\t\t\t\\\n+\texit;\t\t\t\t\t\t\\\n+\"\t::: __clobber_all);\n+}\n+\n SEC(\"socket\")\n __description(\"PTR_TO_STACK check low 4\")\n __failure __msg(\"math between fp pointer\")\n@@ -483,6 +520,7 @@ l1_%=:\tr0 = 42;\t\t\t\t\t\\\n \n SEC(\"socket\")\n __description(\"PTR_TO_STACK stack size \u003e 512\")\n+__load_if_no_large_stack()\n __failure __msg(\"invalid write to stack R1 off=-520 size=8\")\n __naked void stack_check_size_gt_512(void)\n {\n@@ -495,6 +533,21 @@ __naked void stack_check_size_gt_512(void)\n \"\t::: __clobber_all);\n }\n \n+SEC(\"socket\")\n+__description(\"PTR_TO_STACK stack size \u003e 2048\")\n+__load_if_large_stack()\n+__failure __msg(\"invalid write to stack R1 off=-2056 size=8\")\n+__naked void stack_check_size_gt_2048(void)\n+{\n+\tasm volatile (\"\t\t\t\t\t\\\n+\tr1 = r10;\t\t\t\t\t\\\n+\tr1 += -2056;\t\t\t\t\t\\\n+\tr0 = 42;\t\t\t\t\t\\\n+\t*(u64*)(r1 + 0) = r0;\t\t\t\t\\\n+\texit;\t\t\t\t\t\t\\\n+\"\t::: __clobber_all);\n+}\n+\n #ifdef __BPF_FEATURE_MAY_GOTO\n SEC(\"socket\")\n __description(\"PTR_TO_STACK stack size 512 with may_goto with jit\")\ndiff --git a/tools/testing/selftests/bpf/progs/verifier_tailcall.c b/tools/testing/selftests/bpf/progs/verifier_tailcall.c\nindex b4acce60fb9b9..51687da972257 100644\n--- a/tools/testing/selftests/bpf/progs/verifier_tailcall.c\n+++ b/tools/testing/selftests/bpf/progs/verifier_tailcall.c\n@@ -28,4 +28,61 @@ __naked void invalid_map_for_tail_call(void)\n \t: __clobber_all);\n }\n \n+struct {\n+\t__uint(type, BPF_MAP_TYPE_PROG_ARRAY);\n+\t__uint(max_entries, 1);\n+\t__uint(key_size, sizeof(__u32));\n+\t__uint(value_size, sizeof(__u32));\n+} jmp_table SEC(\".maps\");\n+\n+__used __naked\n+static int subprog_tail_call(void)\n+{\n+\tasm volatile (\"\t\t\t\\\n+\tr2 = %[jmp_table] ll;\t\t\\\n+\tr3 = 0;\t\t\t\t\\\n+\tcall %[bpf_tail_call];\t\t\\\n+\tr0 = 0;\t\t\t\t\\\n+\texit;\t\t\t\t\\\n+\"\t:\n+\t: __imm(bpf_tail_call),\n+\t __imm_addr(jmp_table)\n+\t: __clobber_all);\n+}\n+\n+/*\n+ * A tail call unwinds only the frame of the subprog doing it, so the\n+ * frames of its callers stay on the stack. With up to 33 tail calls in\n+ * a chain the verifier caps the stack those frames may add up to at\n+ * 256 bytes.\n+ */\n+SEC(\"tc\")\n+__description(\"tail call from subprog with 240 bytes of caller stack\")\n+__success\n+__naked void tail_call_caller_stack_ok(void)\n+{\n+\tasm volatile (\"\t\t\t\\\n+\tr2 = 42;\t\t\t\\\n+\t*(u64 *)(r10 - 240) = r2;\t\\\n+\tcall subprog_tail_call;\t\t\\\n+\tr0 = 0;\t\t\t\t\\\n+\texit;\t\t\t\t\\\n+\"\t::: __clobber_all);\n+}\n+\n+SEC(\"tc\")\n+__description(\"tail call from subprog with 256 bytes of caller stack\")\n+__failure\n+__msg(\"tail_calls are not allowed when call stack of previous frames is 256 bytes. Too large\")\n+__naked void tail_call_caller_stack_too_large(void)\n+{\n+\tasm volatile (\"\t\t\t\\\n+\tr2 = 42;\t\t\t\\\n+\t*(u64 *)(r10 - 256) = r2;\t\\\n+\tcall subprog_tail_call;\t\t\\\n+\tr0 = 0;\t\t\t\t\\\n+\texit;\t\t\t\t\\\n+\"\t::: __clobber_all);\n+}\n+\n char _license[] SEC(\"license\") = \"GPL\";\ndiff --git a/tools/testing/selftests/bpf/progs/verifier_var_off.c b/tools/testing/selftests/bpf/progs/verifier_var_off.c\nindex a63e336750918..399884911ea53 100644\n--- a/tools/testing/selftests/bpf/progs/verifier_var_off.c\n+++ b/tools/testing/selftests/bpf/progs/verifier_var_off.c\n@@ -406,6 +406,7 @@ __naked void zero_sized_access_max_out_of_bound(void)\n \n SEC(\"lwt_in\")\n __description(\"indirect variable-offset stack access, min out of bound\")\n+__load_if_no_large_stack()\n __failure __msg(\"invalid variable-offset read from stack R2\")\n __naked void access_min_out_of_bound(void)\n {\n@@ -433,6 +434,37 @@ __naked void access_min_out_of_bound(void)\n \t: __clobber_all);\n }\n \n+SEC(\"lwt_in\")\n+__description(\"indirect variable-offset stack access, min out of bound, large stack\")\n+__load_if_large_stack()\n+__failure __msg(\"invalid variable-offset read from stack R2\")\n+__naked void access_min_out_of_bound_large(void)\n+{\n+\tasm volatile (\"\t\t\t\t\t\\\n+\t/* Fill the top 8 bytes of the stack */\t\t\\\n+\tr2 = 0;\t\t\t\t\t\t\\\n+\t*(u64*)(r10 - 8) = r2;\t\t\t\t\\\n+\t/* Get an unknown value */\t\t\t\\\n+\tr2 = *(u32*)(r1 + 0);\t\t\t\t\\\n+\t/* Make it small and 4-byte aligned */\t\t\\\n+\tr2 \u0026= 4;\t\t\t\t\t\\\n+\tr2 -= 2052;\t\t\t\t\t\\\n+\t/*\t\t\t\t\t\t\\\n+\t * add it to fp. We now have either fp-2052 or fp-2048, but\\\n+\t * we don't know which\t\t\t\t\\\n+\t */\t\t\t\t\t\t\\\n+\tr2 += r10;\t\t\t\t\t\\\n+\t/* dereference it indirectly */\t\t\t\\\n+\tr1 = %[map_hash_8b] ll;\t\t\t\t\\\n+\tcall %[bpf_map_lookup_elem];\t\t\t\\\n+\tr0 = 0;\t\t\t\t\t\t\\\n+\texit;\t\t\t\t\t\t\\\n+\"\t:\n+\t: __imm(bpf_map_lookup_elem),\n+\t __imm_addr(map_hash_8b)\n+\t: __clobber_all);\n+}\n+\n SEC(\"cgroup/skb\")\n __description(\"indirect variable-offset stack access, min_off \u003c min_initialized\")\n __success\ndiff --git a/tools/testing/selftests/bpf/test_loader.c b/tools/testing/selftests/bpf/test_loader.c\nindex a6e3fcc1079c6..25eeb1c1248b3 100644\n--- a/tools/testing/selftests/bpf/test_loader.c\n+++ b/tools/testing/selftests/bpf/test_loader.c\n@@ -45,6 +45,11 @@ enum load_mode {\n \tNO_JITED\t= 1 \u003c\u003c 1,\n };\n \n+enum stack_mode {\n+\tLARGE_STACK\t= 1 \u003c\u003c 0,\n+\tSMALL_STACK\t= 1 \u003c\u003c 1,\n+};\n+\n struct test_subspec {\n \tchar *name;\n \tchar *description;\n@@ -70,6 +75,7 @@ struct test_spec {\n \tint mode_mask;\n \tint arch_mask;\n \tint load_mask;\n+\tint stack_mask;\n \tint linear_sz;\n \tconst char *skip_reason;\n \tbool prepare_priv;\n@@ -425,6 +431,7 @@ static int parse_test_spec(struct test_loader *tester,\n \tint err = 0;\n \tu32 arch_mask = 0;\n \tu32 load_mask = 0;\n+\tu32 stack_mask = 0;\n \tstruct btf *btf;\n \tenum arch arch;\n \n@@ -620,6 +627,16 @@ static int parse_test_spec(struct test_loader *tester,\n \t\t\t\terr = -EINVAL;\n \t\t\t\tgoto cleanup;\n \t\t\t}\n+\t\t} else if ((val = str_has_pfx(s, \"stack_mode=\"))) {\n+\t\t\tif (strcmp(val, \"large\") == 0) {\n+\t\t\t\tstack_mask = LARGE_STACK;\n+\t\t\t} else if (strcmp(val, \"small\") == 0) {\n+\t\t\t\tstack_mask = SMALL_STACK;\n+\t\t\t} else {\n+\t\t\t\tPRINT_FAIL(\"bad stack spec: '%s'\", val);\n+\t\t\t\terr = -EINVAL;\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n \t\t} else if ((msg = str_has_pfx(s, \"test_expect_stderr=\"))) {\n \t\t\terr = push_disasm_msg(msg, \u0026stderr_on_next_line,\n \t\t\t\t\t \u0026spec-\u003epriv.stderr);\n@@ -659,6 +676,7 @@ static int parse_test_spec(struct test_loader *tester,\n \n \tspec-\u003earch_mask = arch_mask ?: -1;\n \tspec-\u003eload_mask = load_mask ?: (JITED | NO_JITED);\n+\tspec-\u003estack_mask = stack_mask ?: (LARGE_STACK | SMALL_STACK);\n \n \tif (spec-\u003emode_mask == 0)\n \t\tspec-\u003emode_mask = PRIV;\n@@ -1331,6 +1349,7 @@ void run_subtest(struct test_loader *tester,\n {\n \tstruct test_subspec *subspec = unpriv ? \u0026spec-\u003eunpriv : \u0026spec-\u003epriv;\n \tint current_runtime = is_jit_enabled() ? JITED : NO_JITED;\n+\tint current_stack = is_large_stack_supported() ? LARGE_STACK : SMALL_STACK;\n \tstruct bpf_program *tprog = NULL, *tprog_iter;\n \tstruct bpf_link *link, *links[32] = {};\n \tstruct test_spec *spec_iter;\n@@ -1360,6 +1379,11 @@ void run_subtest(struct test_loader *tester,\n \t\treturn;\n \t}\n \n+\tif ((current_stack \u0026 spec-\u003estack_mask) == 0) {\n+\t\ttest__skip();\n+\t\treturn;\n+\t}\n+\n \tif (unpriv) {\n \t\tif (!can_execute_unpriv(tester, spec)) {\n \t\t\ttest__skip();\ndiff --git a/tools/testing/selftests/bpf/testing_helpers.c b/tools/testing/selftests/bpf/testing_helpers.c\nindex d1d60451c5bcc..47fe61a1ebff1 100644\n--- a/tools/testing/selftests/bpf/testing_helpers.c\n+++ b/tools/testing/selftests/bpf/testing_helpers.c\n@@ -517,6 +517,47 @@ bool is_jit_enabled(void)\n \treturn enabled;\n }\n \n+/*\n+ * Whether the kernel accepts a program using more than 512 bytes of stack,\n+ * which depends on the JIT in use. Probed once with a program that stores\n+ * at the 2 KiB depth. Only the verifier's verdict on that store is cached:\n+ * a load that fails for another reason, such as a missing capability, is\n+ * reported and probed again on the next call.\n+ */\n+bool is_large_stack_supported(void)\n+{\n+\tstatic int supported = -1;\n+\tstruct bpf_insn insns[] = {\n+\t\tBPF_ST_MEM(BPF_DW, BPF_REG_10, -2048, 0),\n+\t\tBPF_MOV64_IMM(BPF_REG_0, 0),\n+\t\tBPF_EXIT_INSN(),\n+\t};\n+\tchar log[1024] = {};\n+\tLIBBPF_OPTS(bpf_prog_load_opts, opts,\n+\t\t.log_buf = log,\n+\t\t.log_size = sizeof(log),\n+\t\t.log_level = 1,\n+\t);\n+\tint fd;\n+\n+\tif (supported \u003e= 0)\n+\t\treturn supported;\n+\n+\tfd = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, NULL, \"GPL\", insns, ARRAY_SIZE(insns),\n+\t\t\t \u0026opts);\n+\tif (fd \u003e= 0) {\n+\t\tclose(fd);\n+\t\tsupported = 1;\n+\t} else if (strstr(log, \"invalid write to stack\")) {\n+\t\tsupported = 0;\n+\t} else {\n+\t\tfprintf(stderr, \"%s: probe failed with errno %d, assuming 512 bytes:\\n%s\",\n+\t\t\t__func__, errno, log);\n+\t\treturn false;\n+\t}\n+\treturn supported;\n+}\n+\n int stack_mprotect(void)\n {\n \tvoid *buf;\ndiff --git a/tools/testing/selftests/bpf/testing_helpers.h b/tools/testing/selftests/bpf/testing_helpers.h\nindex 1c58a2f08b645..f1505108e26a4 100644\n--- a/tools/testing/selftests/bpf/testing_helpers.h\n+++ b/tools/testing/selftests/bpf/testing_helpers.h\n@@ -59,6 +59,7 @@ struct bpf_insn;\n int get_xlated_program(int fd_prog, struct bpf_insn **buf, __u32 *cnt);\n int testing_prog_flags(void);\n bool is_jit_enabled(void);\n+bool is_large_stack_supported(void);\n int stack_mprotect(void);\n \n /* Runs diff(1) on mismatch */\ndiff --git a/tools/testing/selftests/bpf/verifier/calls.c b/tools/testing/selftests/bpf/verifier/calls.c\nindex 8b94b87135bcf..0af237c02ddf9 100644\n--- a/tools/testing/selftests/bpf/verifier/calls.c\n+++ b/tools/testing/selftests/bpf/verifier/calls.c\n@@ -1037,15 +1037,34 @@\n \t.result = ACCEPT,\n },\n {\n-\t\"calls: stack overflow using two frames (pre-call access)\",\n+\t/*\n+\t * Five 480-byte frames exceed the 2 KiB budget of JITs with large\n+\t * stacks, and two of them the 512 bytes allowed elsewhere.\n+\t */\n+\t\"calls: stack overflow using five frames (pre-call access)\",\n \t.insns = {\n \t/* prog 1 */\n-\tBPF_ST_MEM(BPF_B, BPF_REG_10, -300, 0),\n+\tBPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),\n \tBPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1),\n \tBPF_EXIT_INSN(),\n \n \t/* prog 2 */\n-\tBPF_ST_MEM(BPF_B, BPF_REG_10, -300, 0),\n+\tBPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),\n+\tBPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1),\n+\tBPF_EXIT_INSN(),\n+\n+\t/* prog 3 */\n+\tBPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),\n+\tBPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1),\n+\tBPF_EXIT_INSN(),\n+\n+\t/* prog 4 */\n+\tBPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),\n+\tBPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1),\n+\tBPF_EXIT_INSN(),\n+\n+\t/* prog 5 */\n+\tBPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),\n \tBPF_MOV64_IMM(BPF_REG_0, 0),\n \tBPF_EXIT_INSN(),\n \t},\n@@ -1054,15 +1073,30 @@\n \t.result = REJECT,\n },\n {\n-\t\"calls: stack overflow using two frames (post-call access)\",\n+\t\"calls: stack overflow using five frames (post-call access)\",\n \t.insns = {\n \t/* prog 1 */\n \tBPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 2),\n-\tBPF_ST_MEM(BPF_B, BPF_REG_10, -300, 0),\n+\tBPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),\n \tBPF_EXIT_INSN(),\n \n \t/* prog 2 */\n-\tBPF_ST_MEM(BPF_B, BPF_REG_10, -300, 0),\n+\tBPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 2),\n+\tBPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),\n+\tBPF_EXIT_INSN(),\n+\n+\t/* prog 3 */\n+\tBPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 2),\n+\tBPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),\n+\tBPF_EXIT_INSN(),\n+\n+\t/* prog 4 */\n+\tBPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 2),\n+\tBPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),\n+\tBPF_EXIT_INSN(),\n+\n+\t/* prog 5 */\n+\tBPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),\n \tBPF_MOV64_IMM(BPF_REG_0, 0),\n \tBPF_EXIT_INSN(),\n \t},\n@@ -1127,7 +1161,7 @@\n \t.result = ACCEPT,\n },\n {\n-\t\"calls: stack depth check using three frames. test3\",\n+\t\"calls: stack depth check using five frames. test3\",\n \t.insns = {\n \t/* main */\n \tBPF_MOV64_REG(BPF_REG_6, BPF_REG_1),\n@@ -1135,66 +1169,104 @@\n \tBPF_MOV64_REG(BPF_REG_1, BPF_REG_6),\n \tBPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 8), /* call B */\n \tBPF_JMP_IMM(BPF_JGE, BPF_REG_6, 0, 1),\n-\tBPF_ST_MEM(BPF_B, BPF_REG_10, -64, 0),\n+\tBPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),\n \tBPF_MOV64_IMM(BPF_REG_0, 0),\n \tBPF_EXIT_INSN(),\n \t/* A */\n \tBPF_JMP_IMM(BPF_JLT, BPF_REG_1, 10, 1),\n \tBPF_EXIT_INSN(),\n-\tBPF_ST_MEM(BPF_B, BPF_REG_10, -224, 0),\n+\tBPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),\n \tBPF_JMP_IMM(BPF_JA, 0, 0, -3),\n \t/* B */\n \tBPF_JMP_IMM(BPF_JGT, BPF_REG_1, 2, 1),\n-\tBPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, -6), /* call A */\n-\tBPF_ST_MEM(BPF_B, BPF_REG_10, -256, 0),\n+\tBPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 2), /* call C */\n+\tBPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),\n+\tBPF_EXIT_INSN(),\n+\t/* C */\n+\tBPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 2), /* call D */\n+\tBPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),\n+\tBPF_EXIT_INSN(),\n+\t/* D */\n+\tBPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, -12), /* call A */\n+\tBPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),\n \tBPF_EXIT_INSN(),\n \t},\n \t.prog_type = BPF_PROG_TYPE_XDP,\n-\t/* stack_main=64, stack_A=224, stack_B=256\n-\t * and max(main+A, main+A+B) \u003e 512\n+\t/*\n+\t * every frame is 480 bytes, main+A = 960 \u003e 512 and\n+\t * max(main+A, main+B+C+D+A) = 2400 \u003e 2048\n \t */\n \t.errstr = \"combined stack\",\n \t.result = REJECT,\n },\n {\n-\t\"calls: stack depth check using three frames. test4\",\n-\t/* void main(void) {\n+\t\"calls: stack depth check using five frames. test4\",\n+\t/*\n+\t * void main(void) {\n \t * func1(0);\n \t * func1(1);\n \t * func2(1);\n \t * }\n-\t * void func1(int alloc_or_recurse) {\n+\t * void funcN(int alloc_or_recurse) { N = 1..4\n \t * if (alloc_or_recurse) {\n-\t * frame_pointer[-300] = 1;\n+\t * frame_pointer[-480] = 1;\n \t * } else {\n-\t * func2(alloc_or_recurse);\n+\t * funcN+1(alloc_or_recurse);\n \t * }\n \t * }\n-\t * void func2(int alloc_or_recurse) {\n+\t * void func5(int alloc_or_recurse) {\n \t * if (alloc_or_recurse) {\n-\t * frame_pointer[-300] = 1;\n+\t * frame_pointer[-480] = 1;\n \t * }\n \t * }\n+\t * main also calls func2 to func5 with 1 so that every function has a\n+\t * path allocating its 480 bytes, and the chain adds up to 2400 bytes,\n+\t * more than the 2 KiB budget of JITs with large stacks, and to 960\n+\t * bytes after two frames, more than the 512 bytes allowed elsewhere.\n \t */\n \t.insns = {\n \t/* main */\n \tBPF_MOV64_IMM(BPF_REG_1, 0),\n-\tBPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 6), /* call A */\n+\tBPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 12), /* call A */\n \tBPF_MOV64_IMM(BPF_REG_1, 1),\n-\tBPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 4), /* call A */\n+\tBPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 10), /* call A */\n+\tBPF_MOV64_IMM(BPF_REG_1, 1),\n+\tBPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 13), /* call B */\n+\tBPF_MOV64_IMM(BPF_REG_1, 1),\n+\tBPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 16), /* call C */\n \tBPF_MOV64_IMM(BPF_REG_1, 1),\n-\tBPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 7), /* call B */\n+\tBPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 19), /* call D */\n+\tBPF_MOV64_IMM(BPF_REG_1, 1),\n+\tBPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 22), /* call E */\n \tBPF_MOV64_IMM(BPF_REG_0, 0),\n \tBPF_EXIT_INSN(),\n \t/* A */\n \tBPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 2),\n-\tBPF_ST_MEM(BPF_B, BPF_REG_10, -300, 0),\n+\tBPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),\n \tBPF_EXIT_INSN(),\n \tBPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call B */\n \tBPF_EXIT_INSN(),\n \t/* B */\n+\tBPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 2),\n+\tBPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),\n+\tBPF_EXIT_INSN(),\n+\tBPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call C */\n+\tBPF_EXIT_INSN(),\n+\t/* C */\n+\tBPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 2),\n+\tBPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),\n+\tBPF_EXIT_INSN(),\n+\tBPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call D */\n+\tBPF_EXIT_INSN(),\n+\t/* D */\n+\tBPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 2),\n+\tBPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),\n+\tBPF_EXIT_INSN(),\n+\tBPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call E */\n+\tBPF_EXIT_INSN(),\n+\t/* E */\n \tBPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 1),\n-\tBPF_ST_MEM(BPF_B, BPF_REG_10, -300, 0),\n+\tBPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),\n \tBPF_EXIT_INSN(),\n \t},\n \t.prog_type = BPF_PROG_TYPE_XDP,\n"
} |
| 2/1 | 2026/09/26 01:41 | agent | patch-evaluator |
1mModel:core Args: null Results: {
"EnableConfigs": [
"BPF_SYSCALL",
"BPF_JIT"
],
"FocusSymbols": [
"bpf_jit_supports_large_stack",
"bpf_id_scratch_reserve",
"bpf_push_jmp_history",
"bpf_bt_sync_linked_regs",
"bpf_mark_chain_precision",
"bpf_compute_subprog_arg_access",
"stacksafe",
"check_max_stack_depth_subprog",
"check_stack_slot_within_bounds",
"check_stack_write_fixed_off",
"check_stack_read_fixed_off",
"widen_frame_masks"
],
"Reasoning": "The patch series expands BPF stack capacity from 512 bytes to 2048 bytes (MAX_BPF_STACK_JIT) on architectures with JIT support (x86_64, arm64). To support this, it modifies core BPF verifier subsystems: jump history linked register bit packing, stack backtracking with enlarged bitmaps, dynamically grown ID scratch maps/sets, dynamically sized per-frame liveness masks, and stack bounds verification. All of this logic is directly reachable via the bpf(BPF_PROG_LOAD) syscall and warrants fuzzing for potential verifier correctness issues, precision tracking regressions, and memory safety flaws.",
"WorthFuzzing": true
}Instruction: You are an expert Linux kernel maintainer and security engineer.
Your job is to review a provided patch series and evaluate whether it warrants fuzzing with syzkaller.
IMPORTANT: The changes have ALREADY been applied and committed as the HEAD commit in
your workspace. Do NOT rely on internal assumptions. You must actively use your code access
tools to inspect the actual source code, callers, and surrounding context.
================================================================================
1. CORE TRIAGE PHILOSOPHY
================================================================================
The goal of patch fuzzing is to discover crashes, regressions, exposed latent bugs,
and newly triggered assertions introduced by the patch series.
- REACHABILITY IS THE PRIMARY GATE:
Fuzzing can only discover bugs in code that can actually execute in standard virtualized
environments (GCE or QEMU, utilizing software-emulated devices like USB gadgets, netdev, tun/tap).
If the modified code is structurally unreachable (see Section 2), it MUST NOT be fuzzed,
regardless of whether it adds assertions or complex logic.
- DO NOT BLINDLY TRUST "NO FUNCTIONAL CHANGE" (NFCI) OR "REFACTORING" CLAIMS:
Patch authors routinely label changes as "cleanups", "refactorings", or state
"No functional change intended". Do NOT take these claims at face value.
Code refactorings that rearrange logic, introduce helper functions, or alter state management
in core subsystems frequently introduce subtle semantic shifts or uncover latent kernel bugs.
If reachable executable code is modified or refactored, it MUST be fuzzed.
- NEW OR MODIFIED ASSERTIONS IN REACHABLE CODE MUST BE FUZZED:
When a patch introduces or modifies runtime checks or assertions (e.g., WARN_ON*, VM_WARN_ON*,
BUG_ON*, lockdep_assert*) in reachable code paths, it enforces new or stricter invariants.
Even if the author believes the invariant always holds, fuzzing is essential to verify whether
an unusual sequence of operations can violate it.
================================================================================
2. WHEN TO RETURN WorthFuzzing=false (NEGATIVE CRITERIA)
================================================================================
Return WorthFuzzing=false ONLY IF all modified code falls strictly into one or more of these categories:
- Non-kernel and non-executable changes:
* Modifications to Documentation/, comments, or spelling fixes.
* User-space directories, self-tests, samples, or scripts (e.g., tools/, samples/, scripts/, usr/)
that do not affect the compiled kernel image (vmlinux) or kernel modules.
* Purely decorative logging (e.g., message strings in pr_err, printk, dev_info) or tracepoints
that do not alter control flow or data structures.
* Build system or Kconfig changes that do not alter compiled C logic.
- Structurally unreachable hardware:
* Vendor-specific PCIe switches, SmartNICs, or GPU drivers (e.g., mlxsw, pds_core, qed,
ionic, amdgpu) requiring physical ASIC/PCIe cards not emulated in standard QEMU.
- Unreachable execution paths:
* Driver teardown callbacks (.remove, .shutdown, pci_unregister_driver) executed only during
physical PCI hot-unplug or manual sysfs driver unbinding.
* Code paths exclusive to architectures other than the target architecture.
================================================================================
3. WHEN TO RETURN WorthFuzzing=true (POSITIVE CRITERIA)
================================================================================
Return WorthFuzzing=true whenever the patch touches reachable executable code, including:
- Core Subsystems:
* Any logic modifications in memory management (mm/), synchronization/locking (kernel/locking/),
BPF, scheduler, core networking, VFS, or syscall handling.
- Refactorings and Code Cleanups:
* Any restructuring of reachable data structures, helper abstractions, or algorithm flows.
- Runtime Assertions and Defensive Checks:
* Any introduction or alteration of assertions (WARN_ON*, VM_WARN_ON*, BUG_ON*, etc.) in reachable paths.
- Reachable Drivers and Protocols:
* Drivers accessible via virtual buses (virtio, USB gadget, loopback, netlink, binder, sockets, etc.).
================================================================================
4. EXTRACTING FocusSymbols (PREVENTING DILUTION)
================================================================================
When WorthFuzzing=true, you must extract specific kernel functions into FocusSymbols to guide the fuzzer:
- AVOID UBIQUITOUS LIFECYCLE HOT-PATHS:
Do NOT list generic, ubiquitous functions called by almost every program in the corpus
(including, but not limited to: general memory allocators and deallocators, page fault
and trap handlers, or core synchronization primitives; this is not an exhaustive list).
Listing ubiquitous functions causes the fuzzer to classify thousands of unrelated tests as "focused",
which severely dilutes fuzzing effort away from the actual changes.
- TARGET SPECIFIC FEATURE LOGIC AND ENTRYPOINTS:
List functions that specifically implement the logic being added or altered, or direct API entrypoints
for the subsystem feature under review.
- HANDLING STATIC INLINE FUNCTIONS IN HEADERS (.h):
Compiler-inlined static functions (such as static inlines in mm/*.h or include/linux/*.h) lack
distinct symbol addresses in vmlinux and cannot be targeted directly by symbol coverage filters.
If the changes are primarily in static inline helpers, identify non-static, feature-specific caller
functions in .c files that exercise them (avoiding ubiquitous lifecycle wrappers).
================================================================================
5. IDENTIFYING EnableConfigs
================================================================================
Identify any specific CONFIG_ options required to properly compile and reach the modified code:
- Inspect Kconfig files and #ifdef guards; do not make assumptions.
- Check "depends on" lines in Kconfig to include any non-standard parent subsystem configs needed.
- Strip any 'CONFIG_' prefix (e.g., return "NET_IPV4" instead of "CONFIG_NET_IPV4").
Prefer calling several tools at the same time to save round-trips.
Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.
Prompt: Target architecture: amd64
For your convenience, here is the diff of the changes:
commit 5a159253784d7a46f42cd32160c732f134182d57
Author: syz-cluster <triage@syzkaller.com>
Date: Sat Sep 26 01:41:58 2026 +0000
syz-cluster: applied patch under review
diff --git a/Documentation/bpf/bpf_design_QA.rst b/Documentation/bpf/bpf_design_QA.rst
index eb19c945f4d57..f4e4a7f4f9fdb 100644
--- a/Documentation/bpf/bpf_design_QA.rst
+++ b/Documentation/bpf/bpf_design_QA.rst
@@ -221,9 +221,13 @@ newer kernels. BPF programs need to change accordingly when this happens.
Q: How much stack space a BPF program uses?
-------------------------------------------
-A: Currently all program types are limited to 512 bytes of stack
-space, but the verifier computes the actual amount of stack used
-and both interpreter and most JITed code consume necessary amount.
+A: A program may use up to 2 KiB of stack, combined over its call
+chain, when the JIT of the architecture reports support for large
+stacks (currently x86-64 and arm64); a single function may use all of
+it, and every frame of a program running on a private stack gets the
+whole amount. Elsewhere, and whenever the interpreter is used, the
+limit is 512 bytes. The verifier computes the actual amount of stack
+used and both interpreter and most JITed code consume necessary amount.
Q: Can BPF be offloaded to HW?
------------------------------
diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
index 6c04fee468766..d92d7754578ab 100644
--- a/arch/arm64/net/bpf_jit_comp.c
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -2485,6 +2485,11 @@ bool bpf_jit_supports_subprog_tailcalls(void)
return true;
}
+bool bpf_jit_supports_large_stack(void)
+{
+ return true;
+}
+
static void invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_node *node,
int bargs_off, int retval_off, int run_ctx_off,
bool save_ret)
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index d4a980140b48d..d6998c909754f 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -4452,6 +4452,18 @@ bool bpf_jit_supports_subprog_tailcalls(void)
return true;
}
+/*
+ * Frame sizes are 32-bit immediates in the prologue, epilogue and tail call
+ * sequences, a tail call pops the caller's frame and lands in the target's
+ * prologue before the target allocates its own, and private stacks are
+ * allocated from the program's own depth, so MAX_BPF_STACK_JIT frames need
+ * nothing special.
+ */
+bool bpf_jit_supports_large_stack(void)
+{
+ return true;
+}
+
bool bpf_jit_supports_percpu_insn(void)
{
return true;
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 92f528c456052..a5d493b3876fc 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -19,11 +19,12 @@
* that converting umax_value to int cannot overflow.
*/
#define BPF_MAX_VAR_SIZ (1 << 29)
-/* size of tmp_str_buf in bpf_verifier.
- * we need at least 306 bytes to fit full stack mask representation
- * (in the "-8,-16,...,-512" form)
+/*
+ * size of tmp_str_buf in bpf_verifier.
+ * we need at least 1399 bytes to fit full stack mask representation
+ * (in the "-8,-16,...,-2048" form)
*/
-#define TMP_STR_BUF_LEN 320
+#define TMP_STR_BUF_LEN 1408
/* Patch buffer size */
#define INSN_BUF_SIZE 32
@@ -242,55 +243,18 @@ enum bpf_stack_slot_type {
#define BPF_REG_SIZE 8 /* size of eBPF register in bytes */
+/*
+ * Largest number of BPF_REG_SIZE stack slots a single frame can have, sized
+ * for the largest stack budget any JIT supports. A frame may use any part of
+ * its program's budget; check_max_stack_depth() enforces the budget on the
+ * combined depth of frames sharing the kernel stack and on each frame using
+ * a private stack.
+ */
+#define MAX_BPF_STACK_SLOTS (MAX_BPF_STACK_JIT / BPF_REG_SIZE)
+
/* 4-byte stack slot granularity for liveness analysis */
#define BPF_HALF_REG_SIZE 4
#define STACK_SLOT_SZ 4
-#define STACK_SLOTS (MAX_BPF_STACK / BPF_HALF_REG_SIZE) /* 128 */
-
-typedef struct {
- u64 v[2];
-} spis_t;
-
-#define SPIS_ZERO ((spis_t){})
-#define SPIS_ALL ((spis_t){{ U64_MAX, U64_MAX }})
-
-static inline bool spis_is_zero(spis_t s)
-{
- return s.v[0] == 0 && s.v[1] == 0;
-}
-
-static inline bool spis_equal(spis_t a, spis_t b)
-{
- return a.v[0] == b.v[0] && a.v[1] == b.v[1];
-}
-
-static inline spis_t spis_or(spis_t a, spis_t b)
-{
- return (spis_t){{ a.v[0] | b.v[0], a.v[1] | b.v[1] }};
-}
-
-static inline spis_t spis_and(spis_t a, spis_t b)
-{
- return (spis_t){{ a.v[0] & b.v[0], a.v[1] & b.v[1] }};
-}
-
-static inline spis_t spis_not(spis_t s)
-{
- return (spis_t){{ ~s.v[0], ~s.v[1] }};
-}
-
-static inline bool spis_test_bit(spis_t s, u32 slot)
-{
- return s.v[slot / 64] & BIT_ULL(slot % 64);
-}
-
-static inline void spis_or_range(spis_t *mask, u32 lo, u32 hi)
-{
- u32 w;
-
- for (w = lo; w <= hi && w < STACK_SLOTS; w++)
- mask->v[w / 64] |= BIT_ULL(w % 64);
-}
#define BPF_REGMASK_ARGS ((1 << BPF_REG_1) | (1 << BPF_REG_2) | \
(1 << BPF_REG_3) | (1 << BPF_REG_4) | \
@@ -420,30 +384,33 @@ enum {
INSN_F_STACK_ARG_ACCESS = BIT(3),
};
+/* Registers linked to one jump condition that a history entry can record */
+#define BPF_LINKED_REGS_MAX 5
+
struct bpf_jmp_history_entry {
/* insn idx can't be bigger than 1 million */
u32 idx : 20;
u32 frame : 4; /* stack access frame number */
- u32 spi : 6; /* stack slot index (0..63) */
- u32 : 2;
- u32 prev_idx : 20;
/* special INSN_F_xxx flags */
u32 flags : 4;
- u32 : 8;
+ u32 : 4;
+ u32 prev_idx : 20;
+ u32 spi : 12; /* stack slot index */
/*
- * additional registers that need precision tracking when this
- * jump is backtracked, vector of five 11-bit records
+ * Scalar registers and spilled scalars linked to the condition of
+ * this jump, which need precision tracking together when the jump is
+ * backtracked. Each is packed as 4 bits of frame number, one bit
+ * telling a register from a stack slot and 11 bits of register or
+ * slot index, see linked_regs_pack().
*/
- u64 linked_regs;
+ u16 linked_regs[BPF_LINKED_REGS_MAX];
+ u8 linked_regs_cnt;
};
static_assert(MAX_CALL_FRAMES <= (1 << 4));
-static_assert(MAX_BPF_STACK / 8 <= (1 << 6));
+static_assert(MAX_BPF_STACK_SLOTS <= (1 << 12));
-/* Maximum number of bpf_reg_state objects that can exist at once */
#define MAX_STACK_ARG_SLOTS (MAX_BPF_FUNC_ARGS - MAX_BPF_FUNC_REG_ARGS)
-#define BPF_ID_MAP_SIZE ((MAX_BPF_REG + MAX_BPF_STACK / BPF_REG_SIZE + \
- MAX_STACK_ARG_SLOTS) * MAX_CALL_FRAMES)
struct bpf_verifier_state {
/* call stack tracking */
struct bpf_func_state *frame[MAX_CALL_FRAMES];
@@ -529,12 +496,27 @@ struct bpf_verifier_state {
u32 may_goto_depth;
};
+/* Number of BPF_REG_SIZE stack slots tracked for the frame so far. */
+static inline u32 bpf_stack_nr_slots(const struct bpf_func_state *frame)
+{
+ return frame->allocated_stack / BPF_REG_SIZE;
+}
+
+/*
+ * Stack slot @spi of @frame, covering bytes [fp - (spi + 1) * 8, fp - spi * 8).
+ * The caller must ensure spi < bpf_stack_nr_slots(frame), see grow_stack_state().
+ */
+static inline struct bpf_stack_state *bpf_stack_slot(const struct bpf_func_state *frame, u32 spi)
+{
+ return &frame->stack[spi];
+}
+
static inline struct bpf_reg_state *
bpf_get_spilled_reg(int slot, struct bpf_func_state *frame, u32 mask)
{
- if (slot < frame->allocated_stack / BPF_REG_SIZE &&
- (1 << frame->stack[slot].slot_type[BPF_REG_SIZE - 1]) & mask)
- return &frame->stack[slot].spilled_ptr;
+ if (slot < bpf_stack_nr_slots(frame) &&
+ (1 << bpf_stack_slot(frame, slot)->slot_type[BPF_REG_SIZE - 1]) & mask)
+ return &bpf_stack_slot(frame, slot)->spilled_ptr;
return NULL;
}
@@ -550,7 +532,7 @@ bpf_get_spilled_stack_arg(int slot, struct bpf_func_state *frame)
/* Iterate over 'frame', setting 'reg' to either NULL or a spilled register. */
#define bpf_for_each_spilled_reg(iter, frame, reg, mask) \
for (iter = 0, reg = bpf_get_spilled_reg(iter, frame, mask); \
- iter < frame->allocated_stack / BPF_REG_SIZE; \
+ iter < bpf_stack_nr_slots(frame); \
iter++, reg = bpf_get_spilled_reg(iter, frame, mask))
/* Iterate over 'frame', setting 'reg' to either NULL or a spilled stack arg. */
@@ -575,7 +557,7 @@ bpf_get_spilled_stack_arg(int slot, struct bpf_func_state *frame)
bpf_for_each_spilled_reg(___j, __state, __reg, __mask) { \
if (!__reg) \
continue; \
- __stack = &__state->stack[___j]; \
+ __stack = bpf_stack_slot(__state, ___j); \
(void)(__expr); \
} \
__stack = NULL; \
@@ -850,7 +832,7 @@ struct backtrack_state {
struct bpf_verifier_env *env;
u32 frame;
u32 reg_masks[MAX_CALL_FRAMES];
- u64 stack_masks[MAX_CALL_FRAMES];
+ unsigned long stack_masks[MAX_CALL_FRAMES][BITS_TO_LONGS(MAX_BPF_STACK_SLOTS)];
u8 stack_arg_masks[MAX_CALL_FRAMES];
};
@@ -859,18 +841,27 @@ struct bpf_id_pair {
u32 cur;
};
+/*
+ * Scratch map from the ids of one verifier state to those of another, also
+ * used as a stack of ids. Grown on demand by bpf_id_scratch_reserve().
+ */
struct bpf_idmap {
u32 tmp_id_gen;
u32 cnt;
- struct bpf_id_pair map[BPF_ID_MAP_SIZE];
+ u32 cap;
+ struct bpf_id_pair *map;
+};
+
+struct bpf_idset_entry {
+ u32 id;
+ u32 cnt;
};
+/* Scratch set of ids with a use count each, grown on demand */
struct bpf_idset {
u32 num_ids;
- struct {
- u32 id;
- u32 cnt;
- } entries[BPF_ID_MAP_SIZE];
+ u32 cap;
+ struct bpf_idset_entry *entries;
};
/* see verifier.c:compute_scc_callchain() */
@@ -965,10 +956,8 @@ struct bpf_verifier_env {
struct bpf_subprog_info subprog_info[BPF_MAX_SUBPROGS + 2]; /* max + 2 for the fake and exception subprogs */
/* subprog indices sorted in topological order: leaves first, callers last */
int subprog_topo_order[BPF_MAX_SUBPROGS + 2];
- union {
- struct bpf_idmap idmap_scratch;
- struct bpf_idset idset_scratch;
- };
+ struct bpf_idmap idmap_scratch;
+ struct bpf_idset idset_scratch;
struct {
int *insn_state;
int *insn_stack;
@@ -993,6 +982,8 @@ struct bpf_verifier_env {
u32 prev_jmps_processed, jmps_processed;
/* maximum combined stack depth */
u32 max_stack_depth;
+ /* stack budget of the program, see bpf_prog_stack_limit() */
+ u32 stack_limit;
/* total verification time */
u64 verification_time;
/* maximum number of verifier states kept in 'branching' instructions */
@@ -1028,7 +1019,7 @@ struct bpf_verifier_env {
*/
u32 scratched_regs;
/* Same as scratched_regs but for stack slots */
- u64 scratched_stack_slots;
+ DECLARE_BITMAP(scratched_stack_slots, MAX_BPF_STACK_SLOTS);
u64 prev_log_pos, prev_insn_print_pos;
/* buffer used to temporary hold constants as scalar registers */
struct bpf_reg_state fake_reg[1];
@@ -1226,8 +1217,10 @@ int bpf_copy_verifier_state(struct bpf_verifier_state *dst_state,
struct list_head *bpf_explored_state(struct bpf_verifier_env *env, int idx);
void bpf_free_verifier_state(struct bpf_verifier_state *state, bool free_self);
void bpf_free_backedges(struct bpf_scc_visit *visit);
+bool bpf_id_scratch_reserve(void **arr, u32 *cap, u32 cnt, size_t elem_size);
int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state *cur,
- int insn_flags, int spi, int frame, u64 linked_regs);
+ int insn_flags, int spi, int frame, const u16 *linked_regs,
+ u8 linked_regs_cnt);
void bpf_bt_sync_linked_regs(struct backtrack_state *bt, struct bpf_jmp_history_entry *hist);
void bpf_mark_reg_not_init(const struct bpf_verifier_env *env,
struct bpf_reg_state *reg);
@@ -1244,6 +1237,24 @@ static inline int bpf_get_spi(s32 off)
return (-off - 1) / BPF_REG_SIZE;
}
+/*
+ * Stack a program may use in total: combined over the frames of a call
+ * chain on the kernel stack, or per frame on a private stack. Any single
+ * frame may reach that deep. Only a JIT that lays out such frames may go
+ * beyond MAX_BPF_STACK, the interpreter's frame size, and only one whose
+ * tail calls let the target set up its own frame: without subprogram
+ * tail calls, do_misc_fixups() gives every program with tail calls a
+ * MAX_BPF_STACK frame, which a deeper frame would overrun.
+ */
+static inline u32 bpf_prog_stack_limit(const struct bpf_prog *prog)
+{
+ /* an offloaded program never runs on the host JIT, whatever it supports */
+ if (prog->jit_requested && !bpf_prog_is_offloaded(prog->aux) &&
+ bpf_jit_supports_large_stack() && bpf_jit_supports_subprog_tailcalls())
+ return MAX_BPF_STACK_JIT;
+ return MAX_BPF_STACK;
+}
+
static inline struct bpf_func_state *bpf_func(struct bpf_verifier_env *env,
const struct bpf_reg_state *reg)
{
@@ -1287,12 +1298,7 @@ static inline void bpf_bt_set_frame_reg(struct backtrack_state *bt, u32 frame, u
static inline void bpf_bt_set_frame_slot(struct backtrack_state *bt, u32 frame, u32 slot)
{
- bt->stack_masks[frame] |= 1ull << slot;
-}
-
-static inline void bpf_bt_set_frame_slot_mask(struct backtrack_state *bt, u32 frame, u64 mask)
-{
- bt->stack_masks[frame] |= mask;
+ __set_bit(slot, bt->stack_masks[frame]);
}
static inline void bt_set_frame_stack_arg_slot(struct backtrack_state *bt, u32 frame, u32 slot)
@@ -1307,7 +1313,7 @@ static inline bool bt_is_frame_reg_set(struct backtrack_state *bt, u32 frame, u3
static inline bool bt_is_frame_slot_set(struct backtrack_state *bt, u32 frame, u32 slot)
{
- return bt->stack_masks[frame] & (1ull << slot);
+ return test_bit(slot, bt->stack_masks[frame]);
}
bool bpf_map_is_rdonly(const struct bpf_map *map);
@@ -1438,7 +1444,7 @@ static inline void mark_reg_scratched(struct bpf_verifier_env *env, u32 regno)
static inline void mark_stack_slot_scratched(struct bpf_verifier_env *env, u32 spi)
{
- env->scratched_stack_slots |= 1ULL << spi;
+ __set_bit(spi, env->scratched_stack_slots);
}
static inline bool reg_scratched(const struct bpf_verifier_env *env, u32 regno)
@@ -1446,27 +1452,28 @@ static inline bool reg_scratched(const struct bpf_verifier_env *env, u32 regno)
return (env->scratched_regs >> regno) & 1;
}
-static inline bool stack_slot_scratched(const struct bpf_verifier_env *env, u64 regno)
+static inline bool stack_slot_scratched(const struct bpf_verifier_env *env, u32 spi)
{
- return (env->scratched_stack_slots >> regno) & 1;
+ return test_bit(spi, env->scratched_stack_slots);
}
static inline bool verifier_state_scratched(const struct bpf_verifier_env *env)
{
- return env->scratched_regs || env->scratched_stack_slots;
+ return env->scratched_regs ||
+ !bitmap_empty(env->scratched_stack_slots, MAX_BPF_STACK_SLOTS);
}
static inline void mark_verifier_state_clean(struct bpf_verifier_env *env)
{
env->scratched_regs = 0U;
- env->scratched_stack_slots = 0ULL;
+ bitmap_zero(env->scratched_stack_slots, MAX_BPF_STACK_SLOTS);
}
/* Used for printing the entire verifier state. */
static inline void mark_verifier_state_scratched(struct bpf_verifier_env *env)
{
env->scratched_regs = ~0U;
- env->scratched_stack_slots = ~0ULL;
+ bitmap_fill(env->scratched_stack_slots, MAX_BPF_STACK_SLOTS);
}
static inline bool bpf_stack_narrow_access_ok(int off, int fill_size, int spill_size)
@@ -1501,7 +1508,7 @@ struct bpf_subprog_info *bpf_find_containing_subprog(struct bpf_verifier_env *en
const char *bpf_subprog_name(const struct bpf_verifier_env *env, int subprog);
int bpf_jmp_offset(struct bpf_insn *insn);
struct bpf_iarray *bpf_insn_successors(struct bpf_verifier_env *env, u32 idx);
-void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask);
+void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, const unsigned long *stack_mask);
bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog);
/* Kinds of member a by-value struct or union may be composed of. */
diff --git a/include/linux/filter.h b/include/linux/filter.h
index 422284b4fa96f..cdd16bdd4dfb6 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -98,6 +98,11 @@ struct ctl_table_header;
/* BPF program can access up to 512 bytes of stack space. */
#define MAX_BPF_STACK 512
+/*
+ * Stack budget of a program on a JIT that lays out frames of that size.
+ * The interpreter and JITs without such support keep MAX_BPF_STACK.
+ */
+#define MAX_BPF_STACK_JIT 2048
/* Helper macros for filter block array initializers. */
@@ -1246,6 +1251,7 @@ bool bpf_jit_supports_ptr_xchg(void);
bool bpf_jit_supports_arena(void);
bool bpf_jit_supports_insn(struct bpf_insn *insn, bool in_arena);
bool bpf_jit_supports_private_stack(void);
+bool bpf_jit_supports_large_stack(void);
bool bpf_jit_supports_timed_may_goto(void);
bool bpf_jit_supports_fsession(void);
diff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c
index 507a366dffa47..db1be14d0a680 100644
--- a/kernel/bpf/backtrack.c
+++ b/kernel/bpf/backtrack.c
@@ -9,7 +9,8 @@
/* for any branch, call, exit record the history of jmps in the given state */
int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state *cur,
- int insn_flags, int spi, int frame, u64 linked_regs)
+ int insn_flags, int spi, int frame, const u16 *linked_regs,
+ u8 linked_regs_cnt)
{
u32 cnt = cur->jmp_history_cnt;
struct bpf_jmp_history_entry *p;
@@ -27,10 +28,13 @@ int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state
env->cur_hist_ent->flags |= insn_flags;
env->cur_hist_ent->spi = spi;
env->cur_hist_ent->frame = frame;
- verifier_bug_if(env->cur_hist_ent->linked_regs != 0, env,
- "insn history: insn_idx %d linked_regs: %#llx",
- env->insn_idx, env->cur_hist_ent->linked_regs);
- env->cur_hist_ent->linked_regs = linked_regs;
+ verifier_bug_if(env->cur_hist_ent->linked_regs_cnt != 0, env,
+ "insn history: insn_idx %d has %u linked regs",
+ env->insn_idx, env->cur_hist_ent->linked_regs_cnt);
+ if (linked_regs_cnt)
+ memcpy(env->cur_hist_ent->linked_regs, linked_regs,
+ linked_regs_cnt * sizeof(*linked_regs));
+ env->cur_hist_ent->linked_regs_cnt = linked_regs_cnt;
return 0;
}
@@ -47,7 +51,9 @@ int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state
p->flags = insn_flags;
p->spi = spi;
p->frame = frame;
- p->linked_regs = linked_regs;
+ if (linked_regs_cnt)
+ memcpy(p->linked_regs, linked_regs, linked_regs_cnt * sizeof(*linked_regs));
+ p->linked_regs_cnt = linked_regs_cnt;
cur->jmp_history_cnt = cnt;
env->cur_hist_ent = p;
@@ -123,13 +129,26 @@ static inline void bt_reset(struct backtrack_state *bt)
bt->env = env;
}
-static inline u32 bt_empty(struct backtrack_state *bt)
+static inline bool bt_frame_stack_empty(struct backtrack_state *bt, u32 frame)
{
- u64 mask = 0;
+ return bitmap_empty(bt->stack_masks[frame], MAX_BPF_STACK_SLOTS);
+}
+
+static inline bool bt_stack_empty(struct backtrack_state *bt)
+{
+ return bt_frame_stack_empty(bt, bt->frame);
+}
+
+static inline bool bt_empty(struct backtrack_state *bt)
+{
+ u32 mask = 0;
int i;
- for (i = 0; i <= bt->frame; i++)
- mask |= bt->reg_masks[i] | bt->stack_masks[i] | bt->stack_arg_masks[i];
+ for (i = 0; i <= bt->frame; i++) {
+ mask |= bt->reg_masks[i] | bt->stack_arg_masks[i];
+ if (!bt_frame_stack_empty(bt, i))
+ return false;
+ }
return mask == 0;
}
@@ -181,7 +200,7 @@ static inline void bt_clear_reg(struct backtrack_state *bt, u32 reg)
static inline void bt_clear_frame_slot(struct backtrack_state *bt, u32 frame, u32 slot)
{
- bt->stack_masks[frame] &= ~(1ull << slot);
+ __clear_bit(slot, bt->stack_masks[frame]);
}
static inline u32 bt_frame_reg_mask(struct backtrack_state *bt, u32 frame)
@@ -194,14 +213,14 @@ static inline u32 bt_reg_mask(struct backtrack_state *bt)
return bt->reg_masks[bt->frame];
}
-static inline u64 bt_frame_stack_mask(struct backtrack_state *bt, u32 frame)
+static inline unsigned long *bt_frame_stack_mask(struct backtrack_state *bt, u32 frame)
{
return bt->stack_masks[frame];
}
-static inline u64 bt_stack_mask(struct backtrack_state *bt)
+static inline unsigned long *bt_stack_mask(struct backtrack_state *bt)
{
- return bt->stack_masks[bt->frame];
+ return bt_frame_stack_mask(bt, bt->frame);
}
static inline u8 bt_stack_arg_mask(struct backtrack_state *bt)
@@ -233,17 +252,16 @@ static void fmt_reg_mask(char *buf, ssize_t buf_sz, u32 reg_mask)
break;
}
}
-/* format stack slots bitmask, e.g., "-8,-24,-40" for 0x15 mask */
-void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask)
+
+/* format stack slots bitmask, e.g., "-8,-24,-40" for slots 0, 2 and 4 */
+void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, const unsigned long *stack_mask)
{
- DECLARE_BITMAP(mask, 64);
bool first = true;
int i, n;
buf[0] = '\0';
- bitmap_from_u64(mask, stack_mask);
- for_each_set_bit(i, mask, 64) {
+ for_each_set_bit(i, stack_mask, MAX_BPF_STACK_SLOTS) {
n = snprintf(buf, buf_sz, "%s%d", first ? "" : ",", -(i + 1) * 8);
first = false;
buf += n;
@@ -452,10 +470,11 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
/* we are now tracking register spills correctly,
* so any instance of leftover slots is a bug
*/
- if (bt_stack_mask(bt) != 0) {
- verifier_bug(env,
- "static subprog leftover stack slots %llx",
- bt_stack_mask(bt));
+ if (!bt_stack_empty(bt)) {
+ bpf_fmt_stack_mask(env->tmp_str_buf, TMP_STR_BUF_LEN,
+ bt_stack_mask(bt));
+ verifier_bug(env, "static subprog leftover stack slots %s",
+ env->tmp_str_buf);
return -EFAULT;
}
/* propagate r1-r5 to the caller */
@@ -488,9 +507,11 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
bt_reg_mask(bt));
return -EFAULT;
}
- if (bt_stack_mask(bt) != 0) {
- verifier_bug(env, "callback leftover stack slots %llx",
- bt_stack_mask(bt));
+ if (!bt_stack_empty(bt)) {
+ bpf_fmt_stack_mask(env->tmp_str_buf, TMP_STR_BUF_LEN,
+ bt_stack_mask(bt));
+ verifier_bug(env, "callback leftover stack slots %s",
+ env->tmp_str_buf);
return -EFAULT;
}
/* clear r1-r5 in callback subprog's mask */
@@ -707,10 +728,10 @@ void bpf_mark_all_scalars_precise(struct bpf_verifier_env *env,
i, j);
}
}
- for (j = 0; j < func->allocated_stack / BPF_REG_SIZE; j++) {
- if (!bpf_is_spilled_reg(&func->stack[j]))
+ for (j = 0; j < bpf_stack_nr_slots(func); j++) {
+ if (!bpf_is_spilled_reg(bpf_stack_slot(func, j)))
continue;
- reg = &func->stack[j].spilled_ptr;
+ reg = &bpf_stack_slot(func, j)->spilled_ptr;
if (reg->type != SCALAR_VALUE || reg->precise)
continue;
reg->precise = true;
@@ -868,7 +889,7 @@ int bpf_mark_chain_precision(struct bpf_verifier_env *env,
if (st->curframe == 0 &&
st->frame[0]->subprogno > 0 &&
st->frame[0]->callsite == BPF_MAIN_FUNC &&
- bt_stack_mask(bt) == 0 &&
+ bt_stack_empty(bt) &&
(bt_reg_mask(bt) & ~BPF_REGMASK_ARGS) == 0) {
bitmap_from_u64(mask, bt_reg_mask(bt));
for_each_set_bit(i, mask, 32) {
@@ -882,8 +903,9 @@ int bpf_mark_chain_precision(struct bpf_verifier_env *env,
return 0;
}
- verifier_bug(env, "backtracking func entry subprog %d reg_mask %x stack_mask %llx",
- st->frame[0]->subprogno, bt_reg_mask(bt), bt_stack_mask(bt));
+ bpf_fmt_stack_mask(env->tmp_str_buf, TMP_STR_BUF_LEN, bt_stack_mask(bt));
+ verifier_bug(env, "backtracking func entry subprog %d reg_mask %x stack_mask %s",
+ st->frame[0]->subprogno, bt_reg_mask(bt), env->tmp_str_buf);
return -EFAULT;
}
@@ -944,18 +966,17 @@ int bpf_mark_chain_precision(struct bpf_verifier_env *env,
}
}
- bitmap_from_u64(mask, bt_frame_stack_mask(bt, fr));
- for_each_set_bit(i, mask, 64) {
- if (verifier_bug_if(i >= func->allocated_stack / BPF_REG_SIZE,
+ for_each_set_bit(i, bt_frame_stack_mask(bt, fr), MAX_BPF_STACK_SLOTS) {
+ if (verifier_bug_if(i >= bpf_stack_nr_slots(func),
env, "stack slot %d, total slots %d",
- i, func->allocated_stack / BPF_REG_SIZE))
+ i, bpf_stack_nr_slots(func)))
return -EFAULT;
- if (!bpf_is_spilled_scalar_reg(&func->stack[i])) {
+ if (!bpf_is_spilled_scalar_reg(bpf_stack_slot(func, i))) {
bt_clear_frame_slot(bt, fr, i);
continue;
}
- reg = &func->stack[i].spilled_ptr;
+ reg = &bpf_stack_slot(func, i)->spilled_ptr;
if (reg->precise) {
bt_clear_frame_slot(bt, fr, i);
} else {
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 227211166dccf..fbb2d8a840ef3 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -3471,6 +3471,19 @@ bool __weak bpf_jit_supports_private_stack(void)
return false;
}
+/*
+ * Return TRUE if the JIT lays out frames of up to MAX_BPF_STACK_JIT bytes.
+ * Its prologue, epilogue and tail call sequences must encode such frame
+ * sizes and a private stack must be sized from the program's depth. The
+ * budget is only granted alongside bpf_jit_supports_subprog_tailcalls(),
+ * whose tail calls land before the target sets up its own frame; see
+ * bpf_prog_stack_limit().
+ */
+bool __weak bpf_jit_supports_large_stack(void)
+{
+ return false;
+}
+
void __weak arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, u64 bp), void *cookie)
{
}
diff --git a/kernel/bpf/diagnostics.c b/kernel/bpf/diagnostics.c
index 5ecfa86ed49f4..a8ed6130c1373 100644
--- a/kernel/bpf/diagnostics.c
+++ b/kernel/bpf/diagnostics.c
@@ -1600,9 +1600,9 @@ static struct bpf_reg_state *target_to_reg(struct bpf_verifier_env *env,
return NULL;
return &state->stack_arg_regs[target->stack_arg];
case BPF_DIAG_MOD_TARGET_STACK_SLOT:
- if (target->spi >= state->allocated_stack / BPF_REG_SIZE)
+ if (target->spi >= bpf_stack_nr_slots(state))
return NULL;
- return &state->stack[target->spi].spilled_ptr;
+ return &bpf_stack_slot(state, target->spi)->spilled_ptr;
default:
return NULL;
}
@@ -1618,7 +1618,7 @@ static bool reg_to_target(struct bpf_verifier_env *env, const struct bpf_reg_sta
for (frame = 0; frame <= vstate->curframe; frame++) {
struct bpf_func_state *state = vstate->frame[frame];
unsigned long start, end;
- u32 nslots = state->allocated_stack / BPF_REG_SIZE;
+ u32 nslots = bpf_stack_nr_slots(state);
int spi;
start = (unsigned long)state->regs;
diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index 44ecdc5b4ec2d..f9beef6695c44 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -10,10 +10,39 @@
#define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
-struct per_frame_masks {
- spis_t may_read; /* stack slots that may be read by this instruction */
- spis_t must_write; /* stack slots written by this instruction */
- spis_t live_before; /* stack slots that may be read by this insn and its successors */
+/*
+ * Stack liveness is tracked with a 4-byte (half register) granularity.
+ * Half-slot 0 covers [fp-4, fp), half-slot 1 covers [fp-8, fp-4), and so on,
+ * hence FRAME_HALF_SPIS - 1 is the deepest half-slot a frame can have.
+ */
+#define FRAME_HALF_SPIS (MAX_BPF_STACK_JIT / BPF_HALF_REG_SIZE)
+#define FRAME_MAX_WORDS BITS_TO_LONGS(FRAME_HALF_SPIS)
+
+/* Masks tracked for each instruction of a frame */
+enum {
+ FM_MAY_READ, /* stack slots that may be read by this instruction */
+ FM_MUST_WRITE, /* stack slots written by this instruction */
+ FM_LIVE_BEFORE, /* stack slots that may be read by this insn and its successors */
+ FM_MASK_CNT,
+};
+
+/*
+ * Per instruction stack masks for one frame of a function instance.
+ *
+ * Most frames use only a fraction of the stack budget, so instead of masks
+ * wide enough for every half-slot of the largest possible frame, all masks
+ * of one array share the width @words, and the marking functions widen the
+ * array as deeper half-slots are recorded. Mask @kind of the instruction at
+ * relative index @i is at &bits[(i * FM_MASK_CNT + kind) * words]. A
+ * half-slot at or past @words * BITS_PER_LONG is never read by this frame,
+ * hence never live. An instruction that may read the whole frame, such as a
+ * call passing a frame pointer to another subprog, widens the array to the
+ * program's stack budget, the deepest an accepted program can reach, so
+ * that the read cannot lose half-slots to a later widening.
+ */
+struct frame_masks {
+ u32 words;
+ unsigned long bits[];
};
/*
@@ -29,7 +58,7 @@ struct func_instance {
u32 subprog_start; /* cached env->subprog_info[subprog].start */
u32 insn_cnt; /* cached number of insns in the function */
/* Per frame, per instruction masks, frames allocated lazily. */
- struct per_frame_masks *frames[MAX_CALL_FRAMES];
+ struct frame_masks *frames[MAX_CALL_FRAMES];
bool must_write_initialized;
};
@@ -151,50 +180,122 @@ static int relative_idx(struct func_instance *instance, u32 insn_idx)
return insn_idx - instance->subprog_start;
}
-static struct per_frame_masks *get_frame_masks(struct func_instance *instance,
- u32 frame, u32 insn_idx)
+static u32 frame_mask_bits(struct frame_masks *fm)
{
- if (!instance->frames[frame])
- return NULL;
+ return fm->words * BITS_PER_LONG;
+}
- return &instance->frames[frame][relative_idx(instance, insn_idx)];
+static size_t frame_mask_words(struct func_instance *instance, u32 words)
+{
+ return (size_t)instance->insn_cnt * FM_MASK_CNT * words;
}
-static struct per_frame_masks *alloc_frame_masks(struct func_instance *instance,
- u32 frame, u32 insn_idx)
+/* Mask @kind of the instruction at relative index @rel */
+static unsigned long *rel_mask(struct frame_masks *fm, u32 rel, u32 kind)
{
- struct per_frame_masks *arr;
+ return fm->bits + ((size_t)rel * FM_MASK_CNT + kind) * fm->words;
+}
- if (!instance->frames[frame]) {
- arr = kvzalloc_objs(*arr, instance->insn_cnt,
- GFP_KERNEL_ACCOUNT);
- instance->frames[frame] = arr;
- if (!arr)
- return ERR_PTR(-ENOMEM);
+/*
+ * Make sure @frame has a mask array at least @words wide, allocating it or
+ * copying the existing masks into the wider stride as needed.
+ * @words must be in range [1, FRAME_MAX_WORDS].
+ */
+static struct frame_masks *widen_frame_masks(struct func_instance *instance,
+ u32 frame, u32 words)
+{
+ struct frame_masks *old = instance->frames[frame], *new;
+ u32 i, kind;
+
+ if (old && old->words >= words)
+ return old;
+ new = kvzalloc_flex(*new, bits, frame_mask_words(instance, words), GFP_KERNEL_ACCOUNT);
+ if (!new)
+ return NULL;
+ new->words = words;
+ if (old) {
+ for (i = 0; i < instance->insn_cnt; i++)
+ for (kind = 0; kind < FM_MASK_CNT; kind++)
+ memcpy(rel_mask(new, i, kind), rel_mask(old, i, kind),
+ old->words * sizeof(*old->bits));
+ kvfree(old);
}
- return get_frame_masks(instance, frame, insn_idx);
+ instance->frames[frame] = new;
+ return new;
}
-/* Accumulate may_read masks for @frame at @insn_idx */
-static int mark_stack_read(struct func_instance *instance, u32 frame, u32 insn_idx, spis_t mask)
+/*
+ * Set the inclusive half-slot range [lo, hi] in mask @kind of @frame at @insn_idx.
+ * An empty range, including one with a negative @hi as computed for a write
+ * that does not fully cover any half-slot, marks nothing.
+ */
+static int mark_stack_range(struct func_instance *instance, u32 frame, u32 insn_idx,
+ u32 kind, s32 lo, s32 hi)
{
- struct per_frame_masks *masks;
+ struct frame_masks *fm;
- masks = alloc_frame_masks(instance, frame, insn_idx);
- if (IS_ERR(masks))
- return PTR_ERR(masks);
- masks->may_read = spis_or(masks->may_read, mask);
+ /*
+ * An access past the frame bottom is rejected by the main verifier
+ * pass later, liveness only has to avoid running off the masks.
+ */
+ hi = min_t(s32, hi, FRAME_HALF_SPIS - 1);
+ if (lo > hi)
+ return 0;
+ fm = widen_frame_masks(instance, frame, BITS_TO_LONGS(hi + 1));
+ if (!fm)
+ return -ENOMEM;
+ bitmap_set(rel_mask(fm, relative_idx(instance, insn_idx), kind), lo, hi - lo + 1);
return 0;
}
-static int mark_stack_write(struct func_instance *instance, u32 frame, u32 insn_idx, spis_t mask)
+/* Accumulate may_read for half-slots [lo, hi] of @frame at @insn_idx */
+static int mark_stack_read(struct func_instance *instance, u32 frame, u32 insn_idx,
+ s32 lo, s32 hi)
+{
+ return mark_stack_range(instance, frame, insn_idx, FM_MAY_READ, lo, hi);
+}
+
+/* Accumulate must_write for half-slots [lo, hi] of @frame at @insn_idx */
+static int mark_stack_write(struct func_instance *instance, u32 frame, u32 insn_idx,
+ s32 lo, s32 hi)
{
- struct per_frame_masks *masks;
+ return mark_stack_range(instance, frame, insn_idx, FM_MUST_WRITE, lo, hi);
+}
- masks = alloc_frame_masks(instance, frame, insn_idx);
- if (IS_ERR(masks))
- return PTR_ERR(masks);
- masks->must_write = spis_or(masks->must_write, mask);
+/*
+ * Mark every half-slot of @frame as possibly read by @insn_idx. This widens
+ * the masks to the program's stack budget: a full read recorded at a narrower
+ * width would leave the bits added by a later widening clear and lose part of
+ * it, and an access past the budget is rejected by the main pass later, so no
+ * widening of an accepted program goes further.
+ */
+static int mark_stack_read_all(struct bpf_verifier_env *env, struct func_instance *instance,
+ u32 frame, u32 insn_idx)
+{
+ return mark_stack_read(instance, frame, insn_idx, 0,
+ env->stack_limit / BPF_HALF_REG_SIZE - 1);
+}
+
+/* Accumulate @src, a mask @src_words wide, into may_read of @frame at @insn_idx */
+static int mark_stack_read_mask(struct func_instance *instance, u32 frame, u32 insn_idx,
+ const unsigned long *src, u32 src_words)
+{
+ u32 nbits = src_words * BITS_PER_LONG;
+ struct frame_masks *fm;
+ unsigned long *dst;
+ u32 last, w;
+
+ last = find_last_bit(src, nbits);
+ if (last == nbits)
+ return 0;
+ fm = widen_frame_masks(instance, frame, BITS_TO_LONGS(last + 1));
+ if (!fm)
+ return -ENOMEM;
+ dst = rel_mask(fm, relative_idx(instance, insn_idx), FM_MAY_READ);
+ /* @src has no bits set past @last, hence none past @fm->words either */
+ src_words = min(src_words, fm->words);
+ for (w = 0; w < src_words; w++)
+ dst[w] |= src[w];
return 0;
}
@@ -272,33 +373,42 @@ __diag_pop();
static inline bool update_insn(struct bpf_verifier_env *env,
struct func_instance *instance, u32 frame, u32 insn_idx)
{
- spis_t new_before, new_after;
- struct per_frame_masks *insn, *succ_insn;
+ unsigned long new_after[FRAME_MAX_WORDS] = {};
+ unsigned long *may_read, *must_write, *live_before;
+ struct frame_masks *fm = instance->frames[frame];
+ u32 rel = relative_idx(instance, insn_idx);
struct bpf_iarray *succ;
- u32 s;
- bool changed;
+ bool changed = false;
+ u32 s, w;
succ = bpf_insn_successors(env, insn_idx);
if (succ->cnt == 0)
return false;
- changed = false;
- insn = get_frame_masks(instance, frame, insn_idx);
- new_before = SPIS_ZERO;
- new_after = SPIS_ZERO;
+ /* All instructions of one frame array share the same mask width */
for (s = 0; s < succ->cnt; ++s) {
- succ_insn = get_frame_masks(instance, frame, succ->items[s]);
- new_after = spis_or(new_after, succ_insn->live_before);
+ unsigned long *succ_live;
+
+ succ_live = rel_mask(fm, relative_idx(instance, succ->items[s]), FM_LIVE_BEFORE);
+ for (w = 0; w < fm->words; w++)
+ new_after[w] |= succ_live[w];
}
+ may_read = rel_mask(fm, rel, FM_MAY_READ);
+ must_write = rel_mask(fm, rel, FM_MUST_WRITE);
+ live_before = rel_mask(fm, rel, FM_LIVE_BEFORE);
/*
* New "live_before" is a union of all "live_before" of successors
* minus slots written by instruction plus slots read by instruction.
* new_before = (new_after & ~insn->must_write) | insn->may_read
*/
- new_before = spis_or(spis_and(new_after, spis_not(insn->must_write)),
- insn->may_read);
- changed |= !spis_equal(new_before, insn->live_before);
- insn->live_before = new_before;
+ for (w = 0; w < fm->words; w++) {
+ unsigned long new_before = (new_after[w] & ~must_write[w]) | may_read[w];
+
+ if (new_before != live_before[w]) {
+ live_before[w] = new_before;
+ changed = true;
+ }
+ }
return changed;
}
@@ -329,10 +439,12 @@ static void update_instance(struct bpf_verifier_env *env, struct func_instance *
static bool is_live_before(struct func_instance *instance, u32 insn_idx, u32 frameno, u32 half_spi)
{
- struct per_frame_masks *masks;
+ struct frame_masks *fm = instance->frames[frameno];
- masks = get_frame_masks(instance, frameno, insn_idx);
- return masks && spis_test_bit(masks->live_before, half_spi);
+ /* No recorded access reaches past the masks, so nothing there is live */
+ if (!fm || half_spi >= frame_mask_bits(fm))
+ return false;
+ return test_bit(half_spi, rel_mask(fm, relative_idx(instance, insn_idx), FM_LIVE_BEFORE));
}
int bpf_live_stack_query_init(struct bpf_verifier_env *env, struct bpf_verifier_state *st)
@@ -430,17 +542,19 @@ static int spi_off(int spi)
* When only one half is set, print as "-4h","-8h",...
* Runs of 3+ consecutive fully-set SPIs are collapsed: "fp0-8..-24"
*/
-static char *fmt_spis_mask(struct bpf_verifier_env *env, int frame, bool first, spis_t spis)
+static char *fmt_spis_mask(struct bpf_verifier_env *env, int frame, bool first,
+ const unsigned long *spis, u32 words)
{
int buf_sz = sizeof(env->tmp_str_buf);
+ int spi_cnt = words * BITS_PER_LONG / 2;
char *buf = env->tmp_str_buf;
int spi, n, run_start;
buf[0] = '\0';
- for (spi = 0; spi < STACK_SLOTS / 2 && buf_sz > 0; spi++) {
- bool lo = spis_test_bit(spis, spi * 2);
- bool hi = spis_test_bit(spis, spi * 2 + 1);
+ for (spi = 0; spi < spi_cnt && buf_sz > 0; spi++) {
+ bool lo = test_bit(spi * 2, spis);
+ bool hi = test_bit(spi * 2 + 1, spis);
const char *space = first ? "" : " ";
if (!lo && !hi)
@@ -450,16 +564,16 @@ static char *fmt_spis_mask(struct bpf_verifier_env *env, int frame, bool first,
/* half-spi */
n = scnprintf(buf, buf_sz, "%sfp%d%d%s",
space, frame, spi_off(spi) + (lo ? STACK_SLOT_SZ : 0), "h");
- } else if (spi + 2 < STACK_SLOTS / 2 &&
- spis_test_bit(spis, spi * 2 + 2) &&
- spis_test_bit(spis, spi * 2 + 3) &&
- spis_test_bit(spis, spi * 2 + 4) &&
- spis_test_bit(spis, spi * 2 + 5)) {
+ } else if (spi + 2 < spi_cnt &&
+ test_bit(spi * 2 + 2, spis) &&
+ test_bit(spi * 2 + 3, spis) &&
+ test_bit(spi * 2 + 4, spis) &&
+ test_bit(spi * 2 + 5, spis)) {
/* 3+ consecutive full spis */
run_start = spi;
- while (spi + 1 < STACK_SLOTS / 2 &&
- spis_test_bit(spis, (spi + 1) * 2) &&
- spis_test_bit(spis, (spi + 1) * 2 + 1))
+ while (spi + 1 < spi_cnt &&
+ test_bit((spi + 1) * 2, spis) &&
+ test_bit((spi + 1) * 2 + 1, spis))
spi++;
n = scnprintf(buf, buf_sz, "%sfp%d%d..%d",
space, frame, spi_off(run_start), spi_off(spi));
@@ -478,7 +592,8 @@ static void print_instance(struct bpf_verifier_env *env, struct func_instance *i
{
int start = env->subprog_info[instance->subprog].start;
struct bpf_insn *insns = env->prog->insnsi;
- struct per_frame_masks *masks;
+ struct frame_masks *fm;
+ unsigned long *mask;
int len = instance->insn_cnt;
int insn_idx, frame, i;
bool has_use, has_def;
@@ -501,10 +616,13 @@ static void print_instance(struct bpf_verifier_env *env, struct func_instance *i
pos = env->log.end_pos;
verbose(env, " use: ");
for (frame = instance->depth; frame >= 0; --frame) {
- masks = get_frame_masks(instance, frame, insn_idx);
- if (!masks || spis_is_zero(masks->may_read))
+ fm = instance->frames[frame];
+ if (!fm)
+ continue;
+ mask = rel_mask(fm, i, FM_MAY_READ);
+ if (bitmap_empty(mask, frame_mask_bits(fm)))
continue;
- verbose(env, "%s", fmt_spis_mask(env, frame, !has_use, masks->may_read));
+ verbose(env, "%s", fmt_spis_mask(env, frame, !has_use, mask, fm->words));
has_use = true;
}
if (!has_use)
@@ -512,10 +630,13 @@ static void print_instance(struct bpf_verifier_env *env, struct func_instance *i
pos = env->log.end_pos;
verbose(env, " def: ");
for (frame = instance->depth; frame >= 0; --frame) {
- masks = get_frame_masks(instance, frame, insn_idx);
- if (!masks || spis_is_zero(masks->must_write))
+ fm = instance->frames[frame];
+ if (!fm)
+ continue;
+ mask = rel_mask(fm, i, FM_MUST_WRITE);
+ if (bitmap_empty(mask, frame_mask_bits(fm)))
continue;
- verbose(env, "%s", fmt_spis_mask(env, frame, !has_def, masks->must_write));
+ verbose(env, "%s", fmt_spis_mask(env, frame, !has_def, mask, fm->words));
has_def = true;
}
if (!has_def)
@@ -584,9 +705,9 @@ static int print_instances(struct bpf_verifier_env *env)
* - same frame + different offset -> offset-imprecise
* - different frames -> fully-imprecise (bitmask OR)
*
- * At memory access sites (LDX/STX/ST), offset-imprecise marks only
- * the known frame's access mask as SPIS_ALL, while fully-imprecise
- * iterates bits in the bitmask and routes each frame to its target.
+ * At memory access sites (LDX/STX/ST), offset-imprecise marks the known
+ * frame as fully read, while fully-imprecise iterates bits in the bitmask
+ * and routes each frame to its target.
*/
#define MAX_ARG_OFFSETS 4
@@ -1235,7 +1356,6 @@ static int record_stack_access_off(struct func_instance *instance, s64 fp_off,
s64 access_bytes, u32 frame, u32 insn_idx)
{
s32 slot_hi, slot_lo;
- spis_t mask;
if (fp_off >= 0)
/*
@@ -1247,27 +1367,19 @@ static int record_stack_access_off(struct func_instance *instance, s64 fp_off,
if (access_bytes == S64_MIN) {
/* helper/kfunc read unknown amount of bytes from fp_off until fp+0 */
slot_hi = (-fp_off - 1) / STACK_SLOT_SZ;
- mask = SPIS_ZERO;
- spis_or_range(&mask, 0, slot_hi);
- return mark_stack_read(instance, frame, insn_idx, mask);
+ return mark_stack_read(instance, frame, insn_idx, 0, slot_hi);
}
if (access_bytes > 0) {
/* Mark any touched slot as use */
slot_hi = (-fp_off - 1) / STACK_SLOT_SZ;
slot_lo = max_t(s32, (-fp_off - access_bytes) / STACK_SLOT_SZ, 0);
- mask = SPIS_ZERO;
- spis_or_range(&mask, slot_lo, slot_hi);
- return mark_stack_read(instance, frame, insn_idx, mask);
+ return mark_stack_read(instance, frame, insn_idx, slot_lo, slot_hi);
} else if (access_bytes < 0) {
/* Mark only fully covered slots as def */
access_bytes = -access_bytes;
slot_hi = (-fp_off) / STACK_SLOT_SZ - 1;
slot_lo = max_t(s32, (-fp_off - access_bytes + STACK_SLOT_SZ - 1) / STACK_SLOT_SZ, 0);
- if (slot_lo <= slot_hi) {
- mask = SPIS_ZERO;
- spis_or_range(&mask, slot_lo, slot_hi);
- return mark_stack_write(instance, frame, insn_idx, mask);
- }
+ return mark_stack_write(instance, frame, insn_idx, slot_lo, slot_hi);
}
return 0;
}
@@ -1276,7 +1388,7 @@ static int record_stack_access_off(struct func_instance *instance, s64 fp_off,
* 'arg' is FP-derived argument to helper/kfunc or load/store that
* reads (positive) or writes (negative) 'access_bytes' into 'use' or 'def'.
*/
-static int record_stack_access(struct func_instance *instance,
+static int record_stack_access(struct bpf_verifier_env *env, struct func_instance *instance,
const struct arg_track *arg,
s64 access_bytes, u32 frame, u32 insn_idx)
{
@@ -1286,7 +1398,7 @@ static int record_stack_access(struct func_instance *instance,
return 0;
if (arg->off_cnt == 0) {
if (access_bytes > 0 || access_bytes == S64_MIN)
- return mark_stack_read(instance, frame, insn_idx, SPIS_ALL);
+ return mark_stack_read_all(env, instance, frame, insn_idx);
return 0;
}
if (access_bytes != S64_MIN && access_bytes < 0 && arg->off_cnt != 1)
@@ -1305,7 +1417,8 @@ static int record_stack_access(struct func_instance *instance,
* When a pointer is ARG_IMPRECISE, conservatively mark every frame in
* the bitmask as fully used.
*/
-static int record_imprecise(struct func_instance *instance, u32 mask, u32 insn_idx)
+static int record_imprecise(struct bpf_verifier_env *env, struct func_instance *instance,
+ u32 mask, u32 insn_idx)
{
int depth = instance->depth;
int f, err;
@@ -1314,7 +1427,7 @@ static int record_imprecise(struct func_instance *instance, u32 mask, u32 insn_i
if (!(mask & 1))
continue;
if (f <= depth) {
- err = mark_stack_read(instance, f, insn_idx, SPIS_ALL);
+ err = mark_stack_read_all(env, instance, f, insn_idx);
if (err)
return err;
}
@@ -1383,9 +1496,9 @@ static int record_load_store_access(struct bpf_verifier_env *env,
}
if (ptr->frame >= 0 && ptr->frame <= depth)
- return record_stack_access(instance, ptr, sz, ptr->frame, insn_idx);
+ return record_stack_access(env, instance, ptr, sz, ptr->frame, insn_idx);
if (ptr->frame == ARG_IMPRECISE)
- return record_imprecise(instance, ptr->mask, insn_idx);
+ return record_imprecise(env, instance, ptr->mask, insn_idx);
/* ARG_NONE: not derived from any frame pointer, skip */
return 0;
}
@@ -1410,7 +1523,7 @@ static int record_arg_access(struct bpf_verifier_env *env,
bytes = bpf_kfunc_stack_access_bytes(env, insn, arg_idx, insn_idx);
} else {
for (int f = 0; f <= depth; f++) {
- err = mark_stack_read(instance, f, insn_idx, SPIS_ALL);
+ err = mark_stack_read_all(env, instance, f, insn_idx);
if (err)
return err;
}
@@ -1420,9 +1533,9 @@ static int record_arg_access(struct bpf_verifier_env *env,
return 0;
if (frame >= 0 && frame <= depth)
- err = record_stack_access(instance, at, bytes, frame, insn_idx);
+ err = record_stack_access(env, instance, at, bytes, frame, insn_idx);
else if (frame == ARG_IMPRECISE)
- err = record_imprecise(instance, at->mask, insn_idx);
+ err = record_imprecise(env, instance, at->mask, insn_idx);
return err;
}
@@ -1772,36 +1885,52 @@ static bool has_fp_args(struct arg_track *args)
* may_read: union (any pass might read the slot).
* must_write: intersection (only slots written on ALL passes are guaranteed).
* live_before is recomputed by a subsequent update_instance() on @dst.
+ *
+ * The two instances may have settled on different mask widths for the same
+ * frame, so @dst is widened to cover @src first. A word only @dst has counts
+ * as zero on the @src side: it unions into may_read as a no-op and intersects
+ * must_write to empty.
*/
-static void merge_instances(struct func_instance *dst, struct func_instance *src)
+static int merge_instances(struct func_instance *dst, struct func_instance *src)
{
- int f, i;
+ struct frame_masks *d, *s;
+ u32 f, i, w;
for (f = 0; f <= dst->depth; f++) {
- if (!src->frames[f]) {
+ s = src->frames[f];
+ d = dst->frames[f];
+ if (!s) {
/* This pass didn't touch frame f — must_write intersects with empty. */
- if (dst->frames[f])
+ if (d)
for (i = 0; i < dst->insn_cnt; i++)
- dst->frames[f][i].must_write = SPIS_ZERO;
+ bitmap_zero(rel_mask(d, i, FM_MUST_WRITE),
+ frame_mask_bits(d));
continue;
}
- if (!dst->frames[f]) {
+ if (!d) {
/* Previous pass didn't touch frame f — take src, zero must_write. */
- dst->frames[f] = src->frames[f];
+ dst->frames[f] = s;
src->frames[f] = NULL;
for (i = 0; i < dst->insn_cnt; i++)
- dst->frames[f][i].must_write = SPIS_ZERO;
+ bitmap_zero(rel_mask(s, i, FM_MUST_WRITE), frame_mask_bits(s));
continue;
}
+ d = widen_frame_masks(dst, f, s->words);
+ if (!d)
+ return -ENOMEM;
for (i = 0; i < dst->insn_cnt; i++) {
- dst->frames[f][i].may_read =
- spis_or(dst->frames[f][i].may_read,
- src->frames[f][i].may_read);
- dst->frames[f][i].must_write =
- spis_and(dst->frames[f][i].must_write,
- src->frames[f][i].must_write);
+ unsigned long *dst_read = rel_mask(d, i, FM_MAY_READ);
+ unsigned long *dst_write = rel_mask(d, i, FM_MUST_WRITE);
+ unsigned long *src_read = rel_mask(s, i, FM_MAY_READ);
+ unsigned long *src_write = rel_mask(s, i, FM_MUST_WRITE);
+
+ for (w = 0; w < d->words; w++) {
+ dst_read[w] |= w < s->words ? src_read[w] : 0;
+ dst_write[w] &= w < s->words ? src_write[w] : 0;
+ }
}
}
+ return 0;
}
static struct func_instance *fresh_instance(struct func_instance *src)
@@ -1916,7 +2045,7 @@ static int analyze_subprog(struct bpf_verifier_env *env,
if (info[subprog].at_in[j][caller_reg].frame == ARG_NONE)
continue;
for (int f = 0; f <= depth; f++) {
- err = mark_stack_read(instance, f, idx, SPIS_ALL);
+ err = mark_stack_read_all(env, instance, f, idx);
if (err)
goto out_free;
}
@@ -1955,13 +2084,18 @@ static int analyze_subprog(struct bpf_verifier_env *env,
/* Pull callee's entry liveness back to caller's callsite */
{
u32 callee_start = callee_instance->subprog_start;
- struct per_frame_masks *entry;
+ struct frame_masks *callee_fm;
for (int f = 0; f < callee_instance->depth; f++) {
- entry = get_frame_masks(callee_instance, f, callee_start);
- if (!entry)
+ callee_fm = callee_instance->frames[f];
+ if (!callee_fm)
continue;
- err = mark_stack_read(instance, f, idx, entry->live_before);
+ err = mark_stack_read_mask(instance, f, idx,
+ rel_mask(callee_fm,
+ relative_idx(callee_instance,
+ callee_start),
+ FM_LIVE_BEFORE),
+ callee_fm->words);
if (err)
goto out_free;
}
@@ -1969,9 +2103,11 @@ static int analyze_subprog(struct bpf_verifier_env *env,
}
if (prev_instance) {
- merge_instances(prev_instance, instance);
+ err = merge_instances(prev_instance, instance);
free_instance(instance);
instance = prev_instance;
+ if (err)
+ return err;
}
update_instance(env, instance);
return 0;
diff --git a/kernel/bpf/log.c b/kernel/bpf/log.c
index fb032dfdc0dee..d850a7863d2ed 100644
--- a/kernel/bpf/log.c
+++ b/kernel/bpf/log.c
@@ -716,7 +716,8 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifie
verbose(env, "=");
print_reg_state(env, state, reg);
}
- for (i = 0; i < state->allocated_stack / BPF_REG_SIZE; i++) {
+ for (i = 0; i < bpf_stack_nr_slots(state); i++) {
+ struct bpf_stack_state *slot = bpf_stack_slot(state, i);
char types_buf[BPF_REG_SIZE + 1];
const char *sep = "";
bool valid = false;
@@ -727,7 +728,7 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifie
continue;
for (j = 0; j < BPF_REG_SIZE; j++) {
- slot_type = state->stack[i].slot_type[j];
+ slot_type = slot->slot_type[j];
if (slot_type != STACK_INVALID && slot_type != STACK_POISON)
valid = true;
types_buf[j] = slot_type_char[slot_type];
@@ -736,12 +737,12 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifie
if (!valid)
continue;
- reg = &state->stack[i].spilled_ptr;
- switch (state->stack[i].slot_type[BPF_REG_SIZE - 1]) {
+ reg = &slot->spilled_ptr;
+ switch (slot->slot_type[BPF_REG_SIZE - 1]) {
case STACK_SPILL:
/* print MISC/ZERO/INVALID slots above subreg spill */
for (j = 0; j < BPF_REG_SIZE; j++)
- if (state->stack[i].slot_type[j] == STACK_SPILL)
+ if (slot->slot_type[j] == STACK_SPILL)
break;
types_buf[j] = '\0';
@@ -751,7 +752,7 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifie
case STACK_DYNPTR:
/* skip to main dynptr slot */
i += BPF_DYNPTR_NR_SLOTS - 1;
- reg = &state->stack[i].spilled_ptr;
+ reg = &bpf_stack_slot(state, i)->spilled_ptr;
verbose(env, " fp%d", (-i - 1) * BPF_REG_SIZE);
verbose(env, "=dynptr_%s(", dynptr_type_str(reg->dynptr.type));
diff --git a/kernel/bpf/states.c b/kernel/bpf/states.c
index 66fb11b6c6a76..5078e4832c6e1 100644
--- a/kernel/bpf/states.c
+++ b/kernel/bpf/states.c
@@ -335,21 +335,18 @@ static bool check_ids(u32 old_id, u32 cur_id, struct bpf_idmap *idmap)
return false;
}
- /* Reached the end of known mappings; haven't seen this id before */
- if (idmap->cnt < BPF_ID_MAP_SIZE) {
- map[idmap->cnt].old = old_id;
- map[idmap->cnt].cur = cur_id;
- idmap->cnt++;
- return true;
- }
-
/*
- * idmap slots are bounded by the number of registers and stack slots.
- * Since referenced dynptrs acquire intermediate references that do
- * not live in either, so the map can be exhausted. Since it is unlikely,
- * fail the verification by treating the states as not equivalent.
+ * Reached the end of known mappings; haven't seen this id before. If
+ * the map cannot grow, treat the states as not equivalent, which only
+ * costs pruning.
*/
- return false;
+ if (!bpf_id_scratch_reserve((void **)&idmap->map, &idmap->cap, idmap->cnt, sizeof(*map)))
+ return false;
+ map = idmap->map;
+ map[idmap->cnt].old = old_id;
+ map[idmap->cnt].cur = cur_id;
+ idmap->cnt++;
+ return true;
}
/*
@@ -415,14 +412,14 @@ static void __clean_func_state(struct bpf_verifier_env *env,
* half_spi 2*i → lower half: slot_type[0..3] (closer to FP)
* half_spi 2*i+1 → upper half: slot_type[4..7] (farther from FP)
*/
- for (i = 0; i < st->allocated_stack / BPF_REG_SIZE; i++) {
+ for (i = 0; i < bpf_stack_nr_slots(st); i++) {
bool lo_live = bpf_stack_slot_alive(env, frame, i * 2);
bool hi_live = bpf_stack_slot_alive(env, frame, i * 2 + 1);
if (!hi_live || !lo_live) {
int start = !lo_live ? 0 : BPF_REG_SIZE / 2;
int end = !hi_live ? BPF_REG_SIZE : BPF_REG_SIZE / 2;
- u8 stype = st->stack[i].slot_type[7];
+ u8 stype = bpf_stack_slot(st, i)->slot_type[7];
/*
* Don't clear special slots.
@@ -442,7 +439,7 @@ static void __clean_func_state(struct bpf_verifier_env *env,
* rejecting as non-scalar register fills.
*/
if (!hi_live) {
- struct bpf_reg_state *spill = &st->stack[i].spilled_ptr;
+ struct bpf_reg_state *spill = &bpf_stack_slot(st, i)->spilled_ptr;
if (lo_live && stype == STACK_SPILL) {
if (spill->type != SCALAR_VALUE)
@@ -454,7 +451,7 @@ static void __clean_func_state(struct bpf_verifier_env *env,
if (bpf_register_is_null(spill))
continue;
for (j = 0; j < 4; j++) {
- u8 *t = &st->stack[i].slot_type[j];
+ u8 *t = &bpf_stack_slot(st, i)->slot_type[j];
if (*t == STACK_SPILL)
*t = STACK_MISC;
@@ -463,7 +460,7 @@ static void __clean_func_state(struct bpf_verifier_env *env,
bpf_mark_reg_not_init(env, spill);
}
for (j = start; j < end; j++)
- st->stack[i].slot_type[j] = STACK_POISON;
+ bpf_stack_slot(st, i)->slot_type[j] = STACK_POISON;
}
}
}
@@ -707,37 +704,38 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
* didn't use them
*/
for (i = 0; i < old->allocated_stack; i++) {
+ struct bpf_stack_state *old_slot, *cur_slot;
struct bpf_reg_state *old_reg, *cur_reg;
int im = i % BPF_REG_SIZE;
+ u8 old_type;
spi = i / BPF_REG_SIZE;
+ old_slot = bpf_stack_slot(old, spi);
+ old_type = old_slot->slot_type[im];
+ cur_slot = i < cur->allocated_stack ? bpf_stack_slot(cur, spi) : NULL;
if (exact == EXACT) {
- u8 old_type = old->stack[spi].slot_type[i % BPF_REG_SIZE];
- u8 cur_type = i < cur->allocated_stack ?
- cur->stack[spi].slot_type[i % BPF_REG_SIZE] : STACK_INVALID;
+ u8 cur_type = cur_slot ? cur_slot->slot_type[im] : STACK_INVALID;
/* STACK_INVALID and STACK_POISON are equivalent for pruning */
if (old_type == STACK_POISON)
old_type = STACK_INVALID;
if (cur_type == STACK_POISON)
cur_type = STACK_INVALID;
- if (i >= cur->allocated_stack || old_type != cur_type)
+ if (!cur_slot || old_type != cur_type)
return false;
}
- if (old->stack[spi].slot_type[i % BPF_REG_SIZE] == STACK_INVALID ||
- old->stack[spi].slot_type[i % BPF_REG_SIZE] == STACK_POISON)
+ if (old_type == STACK_INVALID || old_type == STACK_POISON)
continue;
- if (env->allow_uninit_stack &&
- old->stack[spi].slot_type[i % BPF_REG_SIZE] == STACK_MISC)
+ if (env->allow_uninit_stack && old_type == STACK_MISC)
continue;
/* explored stack has more populated slots than current stack
* and these slots were used
*/
- if (i >= cur->allocated_stack)
+ if (!cur_slot)
return false;
/*
@@ -747,8 +745,8 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
* regsafe() to ensure scalar ids are compared.
*/
if (im == 0 || im == 4) {
- old_reg = scalar_reg_for_stack(env, &old->stack[spi], im);
- cur_reg = scalar_reg_for_stack(env, &cur->stack[spi], im);
+ old_reg = scalar_reg_for_stack(env, old_slot, im);
+ cur_reg = scalar_reg_for_stack(env, cur_slot, im);
if (old_reg && cur_reg) {
if (!regsafe(env, old_reg, cur_reg, idmap, exact))
return false;
@@ -761,21 +759,19 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
* it will be safe with zero-initialized stack.
* The opposite is not true
*/
- if (old->stack[spi].slot_type[i % BPF_REG_SIZE] == STACK_MISC &&
- cur->stack[spi].slot_type[i % BPF_REG_SIZE] == STACK_ZERO)
+ if (old_type == STACK_MISC && cur_slot->slot_type[im] == STACK_ZERO)
continue;
- if (old->stack[spi].slot_type[i % BPF_REG_SIZE] !=
- cur->stack[spi].slot_type[i % BPF_REG_SIZE])
+ if (old_type != cur_slot->slot_type[im])
/* Ex: old explored (safe) state has STACK_SPILL in
* this stack slot, but current has STACK_MISC ->
* this verifier states are not equivalent,
* return false to continue verification of this path
*/
return false;
- if (i % BPF_REG_SIZE != BPF_REG_SIZE - 1)
+ if (im != BPF_REG_SIZE - 1)
continue;
/* Both old and cur are having same slot_type */
- switch (old->stack[spi].slot_type[BPF_REG_SIZE - 1]) {
+ switch (old_type) {
case STACK_SPILL:
/* when explored and current stack slot are both storing
* spilled registers, check that stored pointers types
@@ -787,13 +783,13 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
* such verifier states are not equivalent.
* return false to continue verification of this path
*/
- if (!regsafe(env, &old->stack[spi].spilled_ptr,
- &cur->stack[spi].spilled_ptr, idmap, exact))
+ if (!regsafe(env, &old_slot->spilled_ptr, &cur_slot->spilled_ptr,
+ idmap, exact))
return false;
break;
case STACK_DYNPTR:
- old_reg = &old->stack[spi].spilled_ptr;
- cur_reg = &cur->stack[spi].spilled_ptr;
+ old_reg = &old_slot->spilled_ptr;
+ cur_reg = &cur_slot->spilled_ptr;
if (old_reg->dynptr.type != cur_reg->dynptr.type ||
old_reg->dynptr.first_slot != cur_reg->dynptr.first_slot ||
!check_ids(old_reg->id, cur_reg->id, idmap) ||
@@ -801,8 +797,8 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
return false;
break;
case STACK_ITER:
- old_reg = &old->stack[spi].spilled_ptr;
- cur_reg = &cur->stack[spi].spilled_ptr;
+ old_reg = &old_slot->spilled_ptr;
+ cur_reg = &cur_slot->spilled_ptr;
/* iter.depth is not compared between states as it
* doesn't matter for correctness and would otherwise
* prevent convergence; we maintain it only to prevent
@@ -818,8 +814,8 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
return false;
break;
case STACK_IRQ_FLAG:
- old_reg = &old->stack[spi].spilled_ptr;
- cur_reg = &cur->stack[spi].spilled_ptr;
+ old_reg = &old_slot->spilled_ptr;
+ cur_reg = &cur_slot->spilled_ptr;
if (!check_ids(old_reg->id, cur_reg->id, idmap) ||
old_reg->irq.kfunc_class != cur_reg->irq.kfunc_class)
return false;
@@ -966,6 +962,27 @@ static bool func_states_equal(struct bpf_verifier_env *env, struct bpf_func_stat
return true;
}
+/*
+ * Make room for one more entry in an id scratch array, doubling it as needed.
+ * Returns false if it could not grow; callers then treat the id as unknown
+ * or the states as different, which is always safe.
+ */
+bool bpf_id_scratch_reserve(void **arr, u32 *cap, u32 cnt, size_t elem_size)
+{
+ u32 new_cap;
+ void *p;
+
+ if (cnt < *cap)
+ return true;
+ new_cap = *cap ? *cap * 2 : 64;
+ p = krealloc_array(*arr, new_cap, elem_size, GFP_KERNEL_ACCOUNT | __GFP_NOWARN);
+ if (!p)
+ return false;
+ *arr = p;
+ *cap = new_cap;
+ return true;
+}
+
static void reset_idmap_scratch(struct bpf_verifier_env *env)
{
struct bpf_idmap *idmap = &env->idmap_scratch;
@@ -1043,10 +1060,10 @@ static int propagate_precision(struct bpf_verifier_env *env,
first = false;
}
- for (i = 0; i < state->allocated_stack / BPF_REG_SIZE; i++) {
- if (!bpf_is_spilled_reg(&state->stack[i]))
+ for (i = 0; i < bpf_stack_nr_slots(state); i++) {
+ if (!bpf_is_spilled_reg(bpf_stack_slot(state, i)))
continue;
- state_reg = &state->stack[i].spilled_ptr;
+ state_reg = &bpf_stack_slot(state, i)->spilled_ptr;
if (state_reg->type != SCALAR_VALUE ||
!state_reg->precise)
continue;
@@ -1192,15 +1209,15 @@ static bool iter_active_depths_differ(struct bpf_verifier_state *old, struct bpf
for (fr = old->curframe; fr >= 0; fr--) {
state = old->frame[fr];
- for (i = 0; i < state->allocated_stack / BPF_REG_SIZE; i++) {
- if (state->stack[i].slot_type[0] != STACK_ITER)
+ for (i = 0; i < bpf_stack_nr_slots(state); i++) {
+ if (bpf_stack_slot(state, i)->slot_type[0] != STACK_ITER)
continue;
- slot = &state->stack[i].spilled_ptr;
+ slot = &bpf_stack_slot(state, i)->spilled_ptr;
if (slot->iter.state != BPF_ITER_STATE_ACTIVE)
continue;
- cur_slot = &cur->frame[fr]->stack[i].spilled_ptr;
+ cur_slot = &bpf_stack_slot(cur->frame[fr], i)->spilled_ptr;
if (cur_slot->iter.depth != slot->iter.depth)
return true;
}
@@ -1222,10 +1239,10 @@ static void mark_all_scalars_imprecise(struct bpf_verifier_env *env, struct bpf_
continue;
reg->precise = false;
}
- for (j = 0; j < func->allocated_stack / BPF_REG_SIZE; j++) {
- if (!bpf_is_spilled_reg(&func->stack[j]))
+ for (j = 0; j < bpf_stack_nr_slots(func); j++) {
+ if (!bpf_is_spilled_reg(bpf_stack_slot(func, j)))
continue;
- reg = &func->stack[j].spilled_ptr;
+ reg = &bpf_stack_slot(func, j)->spilled_ptr;
if (reg->type != SCALAR_VALUE)
continue;
reg->precise = false;
@@ -1328,7 +1345,7 @@ int bpf_is_state_visited(struct bpf_verifier_env *env, int insn_idx)
*/
if (is_iter_next_insn(env, insn_idx)) {
if (states_equal(env, &sl->state, cur, RANGE_WITHIN)) {
- struct bpf_func_state *cur_frame;
+ struct bpf_func_state *cur_frame, *iter_frame;
struct bpf_reg_state *iter_state, *iter_reg;
int spi;
@@ -1342,7 +1359,8 @@ int bpf_is_state_visited(struct bpf_verifier_env *env, int insn_idx)
* no need for extra (re-)validations
*/
spi = bpf_get_spi(iter_reg->var_off.value);
- iter_state = &bpf_func(env, iter_reg)->stack[spi].spilled_ptr;
+ iter_frame = bpf_func(env, iter_reg);
+ iter_state = &bpf_stack_slot(iter_frame, spi)->spilled_ptr;
if (iter_state->iter.state == BPF_ITER_STATE_ACTIVE) {
loop = true;
goto hit;
@@ -1410,7 +1428,7 @@ int bpf_is_state_visited(struct bpf_verifier_env *env, int insn_idx)
*/
err = 0;
if (bpf_is_jmp_point(env, env->insn_idx))
- err = bpf_push_jmp_history(env, cur, 0, 0, 0, 0);
+ err = bpf_push_jmp_history(env, cur, 0, 0, 0, NULL, 0);
err = err ? : propagate_precision(env, &sl->state, cur, NULL);
if (err)
return err;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index a7c9e2d8965d5..9bd7f9b3a67cb 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -598,16 +598,17 @@ bool bpf_is_may_goto_insn(struct bpf_insn *insn)
static bool is_spi_bounds_valid(struct bpf_func_state *state, int spi, int nr_slots)
{
- int allocated_slots = state->allocated_stack / BPF_REG_SIZE;
+ int allocated_slots = bpf_stack_nr_slots(state);
- /* We need to check that slots between [spi - nr_slots + 1, spi] are
- * within [0, allocated_stack).
- *
- * Please note that the spi grows downwards. For example, a dynptr
- * takes the size of two stack slots; the first slot will be at
- * spi and the second slot will be at spi - 1.
- */
- return spi - nr_slots + 1 >= 0 && spi < allocated_slots;
+ /*
+ * We need to check that slots between [spi - nr_slots + 1, spi] are
+ * within [0, allocated_stack).
+ *
+ * Please note that the spi grows downwards. For example, a dynptr
+ * takes the size of two stack slots; the first slot will be at
+ * spi and the second slot will be at spi - 1.
+ */
+ return spi - nr_slots + 1 >= 0 && spi < allocated_slots;
}
static int stack_slot_obj_get_spi(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
@@ -751,8 +752,8 @@ static int mark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_
return err;
for (i = 0; i < BPF_REG_SIZE; i++) {
- state->stack[spi].slot_type[i] = STACK_DYNPTR;
- state->stack[spi - 1].slot_type[i] = STACK_DYNPTR;
+ bpf_stack_slot(state, spi)->slot_type[i] = STACK_DYNPTR;
+ bpf_stack_slot(state, spi - 1)->slot_type[i] = STACK_DYNPTR;
}
type = arg_to_dynptr_type(arg_type);
@@ -785,8 +786,8 @@ static int mark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_
parent_id = dynptr->parent_id;
}
- mark_dynptr_stack_regs(env, &state->stack[spi].spilled_ptr,
- &state->stack[spi - 1].spilled_ptr, type, parent_id);
+ mark_dynptr_stack_regs(env, &bpf_stack_slot(state, spi)->spilled_ptr,
+ &bpf_stack_slot(state, spi - 1)->spilled_ptr, type, parent_id);
return 0;
}
@@ -818,7 +819,7 @@ static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_re
* all clones and derived slices. For non-referenced dynptr, only
* the dynptr and slices derived from it will be invalidated.
*/
- reg = &state->stack[spi].spilled_ptr;
+ reg = &bpf_stack_slot(state, spi)->spilled_ptr;
return release_reference(env, dynptr_type_referenced(reg->dynptr.type)
? reg->parent_id
: reg->id);
@@ -857,6 +858,7 @@ static int dynptr_ref_cnt(struct bpf_verifier_env *env, int v_parent_id)
static int destroy_if_dynptr_stack_slot(struct bpf_verifier_env *env,
struct bpf_func_state *state, int spi)
{
+ struct bpf_stack_state *slot = bpf_stack_slot(state, spi);
int err = 0;
/* We always ensure that STACK_DYNPTR is never set partially,
@@ -864,20 +866,22 @@ static int destroy_if_dynptr_stack_slot(struct bpf_verifier_env *env,
* different for STACK_SPILL, where it may be only set for
* 1 byte, so code has to use is_spilled_reg.
*/
- if (state->stack[spi].slot_type[0] != STACK_DYNPTR)
+ if (slot->slot_type[0] != STACK_DYNPTR)
return 0;
/* Reposition spi to first slot */
- if (!state->stack[spi].spilled_ptr.dynptr.first_slot)
+ if (!slot->spilled_ptr.dynptr.first_slot) {
spi = spi + 1;
+ slot = bpf_stack_slot(state, spi);
+ }
/*
* A referenced dynptr can be overwritten only if there is at
* least one other dynptr sharing the same virtual ref parent,
* ensuring the reference can still be properly released.
*/
- if (dynptr_type_referenced(state->stack[spi].spilled_ptr.dynptr.type) &&
- dynptr_ref_cnt(env, state->stack[spi].spilled_ptr.parent_id) <= 1) {
+ if (dynptr_type_referenced(slot->spilled_ptr.dynptr.type) &&
+ dynptr_ref_cnt(env, slot->spilled_ptr.parent_id) <= 1) {
verbose(env, "cannot overwrite referenced dynptr\n");
bpf_diag_res(
env, env->insn_idx, "referenced dynptr overwrite",
@@ -887,7 +891,7 @@ static int destroy_if_dynptr_stack_slot(struct bpf_verifier_env *env,
}
/* Invalidate the dynptr and any derived slices */
- err = release_reference(env, state->stack[spi].spilled_ptr.id);
+ err = release_reference(env, slot->spilled_ptr.id);
if (!err) {
mark_stack_slot_scratched(env, spi);
mark_stack_slot_scratched(env, spi - 1);
@@ -927,6 +931,7 @@ static bool is_dynptr_reg_valid_uninit(struct bpf_verifier_env *env, struct bpf_
static bool is_dynptr_reg_valid_init(struct bpf_verifier_env *env, struct bpf_reg_state *reg)
{
struct bpf_func_state *state = bpf_func(env, reg);
+ struct bpf_stack_state *slot;
int i, spi;
/* This already represents first slot of initialized bpf_dynptr.
@@ -941,12 +946,13 @@ static bool is_dynptr_reg_valid_init(struct bpf_verifier_env *env, struct bpf_re
spi = dynptr_get_spi(env, reg);
if (spi < 0)
return false;
- if (!state->stack[spi].spilled_ptr.dynptr.first_slot)
+ slot = bpf_stack_slot(state, spi);
+ if (!slot->spilled_ptr.dynptr.first_slot)
return false;
for (i = 0; i < BPF_REG_SIZE; i++) {
- if (state->stack[spi].slot_type[i] != STACK_DYNPTR ||
- state->stack[spi - 1].slot_type[i] != STACK_DYNPTR)
+ if (slot->slot_type[i] != STACK_DYNPTR ||
+ bpf_stack_slot(state, spi - 1)->slot_type[i] != STACK_DYNPTR)
return false;
}
@@ -965,7 +971,7 @@ static enum bpf_dynptr_type dynptr_reg_type(struct bpf_verifier_env *env, struct
if (spi < 0)
return BPF_DYNPTR_TYPE_INVALID;
state = bpf_func(env, reg);
- return state->stack[spi].spilled_ptr.dynptr.type;
+ return bpf_stack_slot(state, spi)->spilled_ptr.dynptr.type;
}
static bool is_dynptr_type_expected(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
@@ -1005,7 +1011,7 @@ static int mark_stack_slots_iter(struct bpf_verifier_env *env,
return id;
for (i = 0; i < nr_slots; i++) {
- struct bpf_stack_state *slot = &state->stack[spi - i];
+ struct bpf_stack_state *slot = bpf_stack_slot(state, spi - i);
struct bpf_reg_state *st = &slot->spilled_ptr;
__mark_reg_known_zero(st);
@@ -1042,7 +1048,7 @@ static int unmark_stack_slots_iter(struct bpf_verifier_env *env,
return spi;
for (i = 0; i < nr_slots; i++) {
- struct bpf_stack_state *slot = &state->stack[spi - i];
+ struct bpf_stack_state *slot = bpf_stack_slot(state, spi - i);
struct bpf_reg_state *st = &slot->spilled_ptr;
if (i == 0)
@@ -1076,7 +1082,7 @@ static bool is_iter_reg_valid_uninit(struct bpf_verifier_env *env,
return false;
for (i = 0; i < nr_slots; i++) {
- struct bpf_stack_state *slot = &state->stack[spi - i];
+ struct bpf_stack_state *slot = bpf_stack_slot(state, spi - i);
for (j = 0; j < BPF_REG_SIZE; j++)
if (slot->slot_type[j] == STACK_ITER)
@@ -1097,7 +1103,7 @@ static int is_iter_reg_valid_init(struct bpf_verifier_env *env, struct bpf_reg_s
return -EINVAL;
for (i = 0; i < nr_slots; i++) {
- struct bpf_stack_state *slot = &state->stack[spi - i];
+ struct bpf_stack_state *slot = bpf_stack_slot(state, spi - i);
struct bpf_reg_state *st = &slot->spilled_ptr;
if (st->type & PTR_UNTRUSTED)
@@ -1139,7 +1145,7 @@ static int mark_stack_slot_irq_flag(struct bpf_verifier_env *env,
if (id < 0)
return id;
- slot = &state->stack[spi];
+ slot = bpf_stack_slot(state, spi);
st = &slot->spilled_ptr;
__mark_reg_known_zero(st);
@@ -1166,7 +1172,7 @@ static int unmark_stack_slot_irq_flag(struct bpf_verifier_env *env, struct bpf_r
if (spi < 0)
return spi;
- slot = &state->stack[spi];
+ slot = bpf_stack_slot(state, spi);
st = &slot->spilled_ptr;
if (st->irq.kfunc_class != kfunc_class) {
@@ -1235,7 +1241,7 @@ static bool is_irq_flag_reg_valid_uninit(struct bpf_verifier_env *env, struct bp
if (spi < 0)
return false;
- slot = &state->stack[spi];
+ slot = bpf_stack_slot(state, spi);
for (i = 0; i < BPF_REG_SIZE; i++)
if (slot->slot_type[i] == STACK_IRQ_FLAG)
@@ -1254,7 +1260,7 @@ static int is_irq_flag_reg_valid_init(struct bpf_verifier_env *env, struct bpf_r
if (spi < 0)
return -EINVAL;
- slot = &state->stack[spi];
+ slot = bpf_stack_slot(state, spi);
st = &slot->spilled_ptr;
if (!st->id)
@@ -1401,7 +1407,7 @@ static int copy_reference_state(struct bpf_verifier_state *dst, const struct bpf
static int copy_stack_state(struct bpf_func_state *dst, const struct bpf_func_state *src)
{
- size_t n = src->allocated_stack / BPF_REG_SIZE;
+ size_t n = bpf_stack_nr_slots(src);
dst->stack = copy_array(dst->stack, src->stack, n, sizeof(struct bpf_stack_state),
GFP_KERNEL_ACCOUNT);
@@ -1440,7 +1446,7 @@ static int resize_reference_state(struct bpf_verifier_state *state, size_t n)
*/
static int grow_stack_state(struct bpf_verifier_env *env, struct bpf_func_state *state, int size)
{
- size_t old_n = state->allocated_stack / BPF_REG_SIZE, n;
+ size_t old_n = bpf_stack_nr_slots(state), n;
/* The stack size is always a multiple of BPF_REG_SIZE. */
size = round_up(size, BPF_REG_SIZE);
@@ -3302,26 +3308,25 @@ static void mark_non_stack_access(struct bpf_verifier_env *env, int idx)
env->insn_aux_data[idx].non_stack_access = true;
}
+/* Layout of one packed linked register in the jump history, see linked_regs_pack() */
#define LR_FRAMENO_BITS 4
-#define LR_SPI_BITS 6
-#define LR_ENTRY_BITS (LR_SPI_BITS + LR_FRAMENO_BITS + 1)
-#define LR_SIZE_BITS 4
-#define LR_FRAMENO_MASK ((1ull << LR_FRAMENO_BITS) - 1)
-#define LR_SPI_MASK ((1ull << LR_SPI_BITS) - 1)
-#define LR_SIZE_MASK ((1ull << LR_SIZE_BITS) - 1)
-#define LR_SPI_OFF LR_FRAMENO_BITS
-#define LR_IS_REG_OFF (LR_SPI_BITS + LR_FRAMENO_BITS)
-#define LINKED_REGS_MAX 5
+#define LR_INDEX_BITS 11
+#define LR_FRAMENO_MASK ((1u << LR_FRAMENO_BITS) - 1)
+#define LR_IS_REG BIT(LR_FRAMENO_BITS)
+#define LR_INDEX_OFF (LR_FRAMENO_BITS + 1)
+#define LR_INDEX_MASK ((1u << LR_INDEX_BITS) - 1)
+#define LINKED_REGS_MAX BPF_LINKED_REGS_MAX
static_assert(MAX_CALL_FRAMES <= (1 << LR_FRAMENO_BITS));
-static_assert(LINKED_REGS_MAX < (1 << LR_SIZE_BITS));
-static_assert(LINKED_REGS_MAX * LR_ENTRY_BITS + LR_SIZE_BITS <= 64);
+static_assert(MAX_BPF_REG <= (1 << LR_INDEX_BITS));
+static_assert(MAX_BPF_STACK_SLOTS <= (1 << LR_INDEX_BITS));
+static_assert(LR_INDEX_OFF + LR_INDEX_BITS <= 16);
struct linked_reg {
u8 frameno;
union {
- u8 spi;
- u8 regno;
+ u16 spi;
+ u16 regno;
};
bool is_reg;
};
@@ -3340,48 +3345,34 @@ static struct linked_reg *linked_regs_push(struct linked_regs *s)
}
/*
- * Use u64 as a vector of 5 11-bit values, use first 4-bits to track
- * number of elements currently in stack.
- * Pack one history entry for linked registers as 11 bits in the following format:
- * - 4-bits frameno
- * - 6-bits spi_or_reg
- * - 1-bit is_reg
+ * Pack linked registers for a jump history entry, one u16 each:
+ * - 4 bits frameno
+ * - 1 bit is_reg
+ * - 11 bits register or stack slot index
*/
-static u64 linked_regs_pack(struct linked_regs *s)
+static void linked_regs_pack(const struct linked_regs *s, u16 *packed)
{
- u64 val = 0;
int i;
for (i = 0; i < s->cnt; ++i) {
- struct linked_reg *e = &s->entries[i];
- u64 tmp = 0;
-
- tmp |= e->frameno;
- tmp |= e->spi << LR_SPI_OFF;
- tmp |= (e->is_reg ? 1 : 0) << LR_IS_REG_OFF;
+ const struct linked_reg *e = &s->entries[i];
- val <<= LR_ENTRY_BITS;
- val |= tmp;
+ packed[i] = e->frameno | (e->is_reg ? LR_IS_REG : 0) | (e->spi << LR_INDEX_OFF);
}
- val <<= LR_SIZE_BITS;
- val |= s->cnt;
- return val;
}
-static void linked_regs_unpack(u64 val, struct linked_regs *s)
+static void linked_regs_unpack(const struct bpf_jmp_history_entry *hist, struct linked_regs *s)
{
int i;
- s->cnt = val & LR_SIZE_MASK;
- val >>= LR_SIZE_BITS;
-
+ s->cnt = hist->linked_regs_cnt;
for (i = 0; i < s->cnt; ++i) {
struct linked_reg *e = &s->entries[i];
+ u16 packed = hist->linked_regs[i];
- e->frameno = val & LR_FRAMENO_MASK;
- e->spi = (val >> LR_SPI_OFF) & LR_SPI_MASK;
- e->is_reg = (val >> LR_IS_REG_OFF) & 0x1;
- val >>= LR_ENTRY_BITS;
+ e->frameno = packed & LR_FRAMENO_MASK;
+ e->is_reg = packed & LR_IS_REG;
+ e->spi = (packed >> LR_INDEX_OFF) & LR_INDEX_MASK;
}
}
@@ -3423,10 +3414,10 @@ void bpf_bt_sync_linked_regs(struct backtrack_state *bt, struct bpf_jmp_history_
bool some_precise = false;
int i;
- if (!hist || hist->linked_regs == 0)
+ if (!hist || !hist->linked_regs_cnt)
return;
- linked_regs_unpack(hist->linked_regs, &linked_regs);
+ linked_regs_unpack(hist, &linked_regs);
for (i = 0; i < linked_regs.cnt; ++i) {
struct linked_reg *e = &linked_regs.entries[i];
@@ -3523,17 +3514,18 @@ static void save_register_state(struct bpf_verifier_env *env,
int spi, struct bpf_reg_state *reg,
int size)
{
+ struct bpf_stack_state *slot = bpf_stack_slot(state, spi);
int i;
- bpf_diag_mod_begin(env, &state->stack[spi].spilled_ptr, reg, BPF_DIAG_MOD_SPILL);
- state->stack[spi].spilled_ptr = *reg;
+ bpf_diag_mod_begin(env, &slot->spilled_ptr, reg, BPF_DIAG_MOD_SPILL);
+ slot->spilled_ptr = *reg;
for (i = BPF_REG_SIZE; i > BPF_REG_SIZE - size; i--)
- state->stack[spi].slot_type[i - 1] = STACK_SPILL;
+ slot->slot_type[i - 1] = STACK_SPILL;
/* size < 8 bytes spill */
for (; i; i--)
- mark_stack_slot_misc(env, &state->stack[spi].slot_type[i - 1]);
+ mark_stack_slot_misc(env, &slot->slot_type[i - 1]);
bpf_diag_mod_end(env);
}
@@ -3575,14 +3567,15 @@ static void check_fastcall_stack_contract(struct bpf_verifier_env *env,
static void scrub_special_slot(struct bpf_func_state *state, int spi)
{
+ struct bpf_stack_state *slot = bpf_stack_slot(state, spi);
int i;
/* regular write of data into stack destroys any spilled ptr */
- state->stack[spi].spilled_ptr.type = NOT_INIT;
+ slot->spilled_ptr.type = NOT_INIT;
/* Mark slots as STACK_MISC if they belonged to spilled ptr/dynptr/iter. */
- if (is_stack_slot_special(&state->stack[spi]))
+ if (is_stack_slot_special(slot))
for (i = 0; i < BPF_REG_SIZE; i++)
- scrub_spilled_slot(&state->stack[spi].slot_type[i]);
+ scrub_spilled_slot(&slot->slot_type[i]);
}
/* check_stack_{read,write}_fixed_off functions track spill/fill of registers,
@@ -3600,13 +3593,15 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env,
struct bpf_reg_state *reg = NULL;
int insn_flags = INSN_F_STACK_ACCESS;
int hist_spi = spi, hist_frame = state->frameno;
+ struct bpf_stack_state *ss = bpf_stack_slot(state, spi);
- /* caller checked that off % size == 0 and -MAX_BPF_STACK <= off < 0,
+ /*
+ * caller checked that off % size == 0 and -env->stack_limit <= off < 0,
* so it's aligned access and [off, off + size) are within stack limits
*/
if (!env->allow_ptr_leaks &&
- bpf_is_spilled_reg(&state->stack[spi]) &&
- !bpf_is_spilled_scalar_reg(&state->stack[spi]) &&
+ bpf_is_spilled_reg(ss) &&
+ !bpf_is_spilled_scalar_reg(ss) &&
size != BPF_REG_SIZE) {
const char *reason;
@@ -3628,7 +3623,7 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env,
bool sanitize = reg && is_pointer_regtype(reg->type);
for (i = 0; i < size; i++) {
- u8 type = state->stack[spi].slot_type[(slot - i) %
+ u8 type = ss->slot_type[(slot - i) %
BPF_REG_SIZE];
if (type != STACK_MISC && type != STACK_ZERO) {
@@ -3657,7 +3652,7 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env,
save_register_state(env, state, spi, reg, size);
/* Break the relation on a narrowing spill. */
if (!reg_value_fits)
- state->stack[spi].spilled_ptr.id = 0;
+ ss->spilled_ptr.id = 0;
} else if (!reg && !(off % BPF_REG_SIZE) && is_bpf_st_mem(insn) &&
env->bpf_capable) {
struct bpf_reg_state *tmp_reg = &env->fake_reg[0];
@@ -3681,8 +3676,8 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env,
} else {
u8 type = STACK_MISC;
- if (bpf_is_spilled_reg(&state->stack[spi]))
- bpf_diag_record_scrub(env, &state->stack[spi].spilled_ptr,
+ if (bpf_is_spilled_reg(ss))
+ bpf_diag_record_scrub(env, &ss->spilled_ptr,
BPF_DIAG_MOD_WRITE);
scrub_special_slot(state, spi);
@@ -3703,13 +3698,13 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env,
/* Mark slots affected by this stack write. */
for (i = 0; i < size; i++)
- state->stack[spi].slot_type[(slot - i) % BPF_REG_SIZE] = type;
+ ss->slot_type[(slot - i) % BPF_REG_SIZE] = type;
insn_flags = 0; /* not a register spill */
}
if (insn_flags)
return bpf_push_jmp_history(env, env->cur_state, insn_flags,
- hist_spi, hist_frame, 0);
+ hist_spi, hist_frame, NULL, 0);
return 0;
}
@@ -3774,7 +3769,7 @@ static int check_stack_write_var_off(struct bpf_verifier_env *env,
slot = -i - 1;
spi = slot / BPF_REG_SIZE;
- stype = &state->stack[spi].slot_type[slot % BPF_REG_SIZE];
+ stype = &bpf_stack_slot(state, spi)->slot_type[slot % BPF_REG_SIZE];
mark_stack_slot_scratched(env, spi);
if (!env->allow_ptr_leaks && *stype != STACK_MISC && *stype != STACK_ZERO) {
@@ -3798,8 +3793,8 @@ static int check_stack_write_var_off(struct bpf_verifier_env *env,
* maintain the spill type.
*/
if (writing_zero && *stype == STACK_SPILL &&
- bpf_is_spilled_scalar_reg(&state->stack[spi])) {
- struct bpf_reg_state *spill_reg = &state->stack[spi].spilled_ptr;
+ bpf_is_spilled_scalar_reg(bpf_stack_slot(state, spi))) {
+ struct bpf_reg_state *spill_reg = &bpf_stack_slot(state, spi)->spilled_ptr;
if (tnum_is_const(spill_reg->var_off) && spill_reg->var_off.value == 0) {
zero_used = true;
@@ -3870,23 +3865,22 @@ static int mark_reg_stack_read(struct bpf_verifier_env *env,
{
struct bpf_verifier_state *vstate = env->cur_state;
struct bpf_func_state *state = vstate->frame[vstate->curframe];
- u64 zero_spill_mask = 0;
int i, slot, spi;
u8 *stype;
- int zeros = 0;
+ int zeros = 0, zero_spills = 0;
for (i = min_off; i < max_off; i++) {
slot = -i - 1;
spi = slot / BPF_REG_SIZE;
mark_stack_slot_scratched(env, spi);
- stype = ptr_state->stack[spi].slot_type;
+ stype = bpf_stack_slot(ptr_state, spi)->slot_type;
if (stype[slot % BPF_REG_SIZE] == STACK_ZERO) {
zeros++;
continue;
}
if (stype[slot % BPF_REG_SIZE] == STACK_SPILL &&
- bpf_register_is_null(&ptr_state->stack[spi].spilled_ptr)) {
- zero_spill_mask |= 1ull << spi;
+ bpf_register_is_null(&bpf_stack_slot(ptr_state, spi)->spilled_ptr)) {
+ zero_spills++;
zeros++;
continue;
}
@@ -3897,8 +3891,14 @@ static int mark_reg_stack_read(struct bpf_verifier_env *env,
* so the whole register == const_zero.
*/
__mark_reg_const_zero(env, &state->regs[dst_regno]);
- if (zero_spill_mask) {
- bpf_bt_set_frame_slot_mask(&env->bt, ptr_state->frameno, zero_spill_mask);
+ if (zero_spills) {
+ for (i = min_off; i < max_off; i++) {
+ slot = -i - 1;
+ spi = slot / BPF_REG_SIZE;
+ stype = bpf_stack_slot(ptr_state, spi)->slot_type;
+ if (stype[slot % BPF_REG_SIZE] == STACK_SPILL)
+ bpf_bt_set_frame_slot(&env->bt, ptr_state->frameno, spi);
+ }
return mark_chain_precision_batch(env, env->cur_state);
}
} else {
@@ -3946,9 +3946,10 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env,
int err;
int insn_flags = INSN_F_STACK_ACCESS;
int hist_spi = spi, hist_frame = reg_state->frameno;
+ struct bpf_stack_state *ss = bpf_stack_slot(reg_state, spi);
- stype = reg_state->stack[spi].slot_type;
- reg = ®_state->stack[spi].spilled_ptr;
+ stype = ss->slot_type;
+ reg = &ss->spilled_ptr;
mark_stack_slot_scratched(env, spi);
check_fastcall_stack_contract(env, state, env->insn_idx, off);
@@ -3959,7 +3960,7 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env,
if (dst_regno >= 0)
bpf_diag_mod_begin(env, &state->regs[dst_regno], reg, BPF_DIAG_MOD_WRITE);
- if (bpf_is_spilled_reg(®_state->stack[spi])) {
+ if (bpf_is_spilled_reg(ss)) {
u8 spill_size = 1;
for (i = BPF_REG_SIZE - 1; i > 0 && stype[i - 1] == STACK_SPILL; i--)
@@ -4085,7 +4086,7 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env,
}
if (insn_flags)
return bpf_push_jmp_history(env, env->cur_state, insn_flags,
- hist_spi, hist_frame, 0);
+ hist_spi, hist_frame, NULL, 0);
return 0;
}
@@ -4275,7 +4276,7 @@ static int check_stack_arg_write(struct bpf_verifier_env *env, struct bpf_func_s
bpf_diag_mod_end(env);
state->no_stack_arg_load = true;
return bpf_push_jmp_history(env, env->cur_state,
- INSN_F_STACK_ARG_ACCESS, spi, 0, 0);
+ INSN_F_STACK_ARG_ACCESS, spi, 0, NULL, 0);
}
/*
@@ -4309,7 +4310,7 @@ static int check_stack_arg_read(struct bpf_verifier_env *env, struct bpf_func_st
cur->regs[dst_regno] = *arg;
bpf_diag_mod_end(env);
return bpf_push_jmp_history(env, env->cur_state,
- INSN_F_STACK_ARG_ACCESS, spi, 0, 0);
+ INSN_F_STACK_ARG_ACCESS, spi, 0, NULL, 0);
}
static int mark_stack_arg_precision(struct bpf_verifier_env *env, int arg_idx)
@@ -5448,7 +5449,7 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx,
if (subprog[idx].priv_stack_mode == PRIV_STACK_ADAPTIVE) {
if (subprog_depth > env->max_stack_depth)
env->max_stack_depth = subprog_depth;
- if (subprog_depth > MAX_BPF_STACK) {
+ if (subprog_depth > env->stack_limit) {
verbose(env, "stack size of subprog %d is %d. Too large\n",
idx, subprog_depth);
return -EACCES;
@@ -5457,7 +5458,7 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx,
depth += subprog_depth;
if (depth > env->max_stack_depth)
env->max_stack_depth = depth;
- if (depth > MAX_BPF_STACK) {
+ if (depth > env->stack_limit) {
total = 0;
for (tmp = idx; tmp >= 0; tmp = dinfo[tmp].caller)
total++;
@@ -6308,10 +6309,11 @@ static int check_ptr_to_map_access(struct bpf_verifier_env *env,
return 0;
}
-/* Check that the stack access at the given offset is within bounds. The
+/*
+ * Check that the stack access at the given offset is within bounds. The
* maximum valid offset is -1.
*
- * The minimum valid offset is -MAX_BPF_STACK for writes, and
+ * The minimum valid offset is -env->stack_limit for writes, and
* -state->allocated_stack for reads.
*/
static int check_stack_slot_within_bounds(struct bpf_verifier_env *env,
@@ -6322,7 +6324,7 @@ static int check_stack_slot_within_bounds(struct bpf_verifier_env *env,
int min_valid_off;
if (t == BPF_WRITE || env->allow_uninit_stack)
- min_valid_off = -MAX_BPF_STACK;
+ min_valid_off = -(int)env->stack_limit;
else
min_valid_off = -state->allocated_stack;
@@ -7048,6 +7050,7 @@ static int check_stack_range_initialized(
}
for (i = min_off; i < max_off + access_size; i++) {
+ struct bpf_stack_state *ss;
u8 *stype;
slot = -i - 1;
@@ -7057,7 +7060,8 @@ static int check_stack_range_initialized(
return -EFAULT;
}
- stype = &state->stack[spi].slot_type[slot % BPF_REG_SIZE];
+ ss = bpf_stack_slot(state, spi);
+ stype = &ss->slot_type[slot % BPF_REG_SIZE];
if (*stype == STACK_MISC)
goto mark;
if ((*stype == STACK_ZERO) ||
@@ -7069,13 +7073,13 @@ static int check_stack_range_initialized(
goto mark;
}
- if (bpf_is_spilled_reg(&state->stack[spi]) &&
- (state->stack[spi].spilled_ptr.type == SCALAR_VALUE ||
+ if (bpf_is_spilled_reg(ss) &&
+ (ss->spilled_ptr.type == SCALAR_VALUE ||
env->allow_ptr_leaks)) {
if (clobber) {
- __mark_reg_unknown(env, &state->stack[spi].spilled_ptr);
+ __mark_reg_unknown(env, &ss->spilled_ptr);
for (j = 0; j < BPF_REG_SIZE; j++)
- scrub_spilled_slot(&state->stack[spi].slot_type[j]);
+ scrub_spilled_slot(&ss->slot_type[j]);
}
goto mark;
}
@@ -7805,7 +7809,7 @@ static int process_dynptr_func(struct bpf_verifier_env *env, struct bpf_reg_stat
mark_stack_slots_scratched(env, spi, BPF_DYNPTR_NR_SLOTS);
- reg = &state->stack[spi].spilled_ptr;
+ reg = &bpf_stack_slot(state, spi)->spilled_ptr;
}
meta->dynptr.type = reg->dynptr.type;
@@ -7938,7 +7942,7 @@ static int process_iter_arg(struct bpf_verifier_env *env, struct bpf_reg_state *
/* remember meta->iter info for process_iter_next_call() */
meta->iter.spi = spi;
meta->iter.frameno = reg->frameno;
- update_ref_obj(&meta->ref_obj, &state->stack[spi].spilled_ptr);
+ update_ref_obj(&meta->ref_obj, &bpf_stack_slot(state, spi)->spilled_ptr);
if (is_iter_destroy_kfunc(meta)) {
err = unmark_stack_slots_iter(env, reg, nr_slots);
@@ -8015,16 +8019,15 @@ static int widen_imprecise_scalars(struct bpf_verifier_env *env,
&fold->regs[i],
&fcur->regs[i]);
- num_slots = min(fold->allocated_stack / BPF_REG_SIZE,
- fcur->allocated_stack / BPF_REG_SIZE);
+ num_slots = min(bpf_stack_nr_slots(fold), bpf_stack_nr_slots(fcur));
for (i = 0; i < num_slots; i++) {
- if (!bpf_is_spilled_reg(&fold->stack[i]) ||
- !bpf_is_spilled_reg(&fcur->stack[i]))
+ if (!bpf_is_spilled_reg(bpf_stack_slot(fold, i)) ||
+ !bpf_is_spilled_reg(bpf_stack_slot(fcur, i)))
continue;
maybe_widen_reg(env,
- &fold->stack[i].spilled_ptr,
- &fcur->stack[i].spilled_ptr);
+ &bpf_stack_slot(fold, i)->spilled_ptr,
+ &bpf_stack_slot(fcur, i)->spilled_ptr);
}
}
return 0;
@@ -8036,7 +8039,7 @@ static struct bpf_reg_state *get_iter_from_state(struct bpf_verifier_state *cur_
int iter_frameno = meta->iter.frameno;
int iter_spi = meta->iter.spi;
- return &cur_st->frame[iter_frameno]->stack[iter_spi].spilled_ptr;
+ return &bpf_stack_slot(cur_st->frame[iter_frameno], iter_spi)->spilled_ptr;
}
/* process_iter_next_call() is called when verifier gets to iterator's next
@@ -8841,7 +8844,7 @@ static int get_constant_map_key(struct bpf_verifier_env *env,
slot = -stack_off - 1;
spi = slot / BPF_REG_SIZE;
off = slot % BPF_REG_SIZE;
- stype = state->stack[spi].slot_type;
+ stype = bpf_stack_slot(state, spi)->slot_type;
/* First handle precisely tracked STACK_ZERO */
for (i = off; i >= 0 && stype[i] == STACK_ZERO; i--)
@@ -8852,14 +8855,14 @@ static int get_constant_map_key(struct bpf_verifier_env *env,
}
/* Check that stack contains a scalar spill of expected size */
- if (!bpf_is_spilled_scalar_reg(&state->stack[spi]))
+ if (!bpf_is_spilled_scalar_reg(bpf_stack_slot(state, spi)))
return -EOPNOTSUPP;
for (i = off; i >= 0 && stype[i] == STACK_SPILL; i--)
spill_size++;
if (spill_size != key_size)
return -EOPNOTSUPP;
- reg = &state->stack[spi].spilled_ptr;
+ reg = &bpf_stack_slot(state, spi)->spilled_ptr;
if (!tnum_is_const(reg->var_off))
/* Stack value not statically known */
return -EOPNOTSUPP;
@@ -10135,8 +10138,9 @@ static int idstack_push(struct bpf_idmap *idmap, u32 id)
if (idmap->map[i].old == id)
return 0;
- if (WARN_ON_ONCE(idmap->cnt >= BPF_ID_MAP_SIZE))
- return -EFAULT;
+ if (!bpf_id_scratch_reserve((void **)&idmap->map, &idmap->cap, idmap->cnt,
+ sizeof(*idmap->map)))
+ return -ENOMEM;
idmap->map[idmap->cnt++].old = id;
return 0;
@@ -13893,11 +13897,11 @@ s64 bpf_helper_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn
}
/*
* Size arg is const on each path but differs across merged
- * paths. MAX_BPF_STACK is a safe upper bound for reads.
+ * paths. Reads may extend anywhere up to the frame top.
*/
if (full_write)
return 0;
- return MAX_BPF_STACK;
+ return S64_MIN;
}
return S64_MIN;
case ARG_PTR_TO_DYNPTR:
@@ -13983,7 +13987,8 @@ s64 bpf_kfunc_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn *
size = (s64)aux->const_reg_vals[size_reg];
goto out;
}
- return MAX_BPF_STACK;
+ /* Unknown size: the read may extend anywhere up to the frame top. */
+ return S64_MIN;
}
/* fixed-size pointed-to type: resolve via BTF */
@@ -14725,7 +14730,8 @@ enum {
REASON_STACK = -5,
};
-static int retrieve_ptr_limit(const struct bpf_reg_state *ptr_reg,
+static int retrieve_ptr_limit(const struct bpf_verifier_env *env,
+ const struct bpf_reg_state *ptr_reg,
u32 *alu_limit, bool mask_to_left)
{
u32 max = 0, ptr_limit = 0;
@@ -14737,7 +14743,7 @@ static int retrieve_ptr_limit(const struct bpf_reg_state *ptr_reg,
* offset where we would need to deal with min/max bounds is
* currently prohibited for unprivileged.
*/
- max = MAX_BPF_STACK + mask_to_left;
+ max = env->stack_limit + mask_to_left;
ptr_limit = -ptr_reg->var_off.value;
break;
case PTR_TO_MAP_VALUE:
@@ -14857,7 +14863,7 @@ static int sanitize_ptr_alu(struct bpf_verifier_env *env,
(opcode == BPF_SUB && !off_is_neg);
}
- err = retrieve_ptr_limit(ptr_reg, &alu_limit, info->mask_to_left);
+ err = retrieve_ptr_limit(env, ptr_reg, &alu_limit, info->mask_to_left);
if (err < 0)
return err;
@@ -14987,7 +14993,7 @@ static int check_stack_access_for_ptr_arithmetic(
return -EACCES;
}
- if (off >= 0 || off < -MAX_BPF_STACK) {
+ if (off >= 0 || off < -(int)env->stack_limit) {
verbose(env, "R%d stack pointer arithmetic goes out of range, "
"prohibited for !root; off=%d\n", regno, off);
return -EACCES;
@@ -17322,10 +17328,10 @@ static void collect_linked_regs(struct bpf_verifier_env *env,
reg = &func->regs[j];
__collect_linked_regs(linked_regs, reg, id, i, j, true);
}
- for (j = 0; j < func->allocated_stack / BPF_REG_SIZE; j++) {
- if (!bpf_is_spilled_reg(&func->stack[j]))
+ for (j = 0; j < bpf_stack_nr_slots(func); j++) {
+ if (!bpf_is_spilled_reg(bpf_stack_slot(func, j)))
continue;
- reg = &func->stack[j].spilled_ptr;
+ reg = &bpf_stack_slot(func, j)->spilled_ptr;
__collect_linked_regs(linked_regs, reg, id, i, j, false);
}
}
@@ -17345,7 +17351,7 @@ static void sync_linked_regs(struct bpf_verifier_env *env, struct bpf_verifier_s
for (i = 0; i < linked_regs->cnt; ++i) {
e = &linked_regs->entries[i];
reg = e->is_reg ? &vstate->frame[e->frameno]->regs[e->regno]
- : &vstate->frame[e->frameno]->stack[e->spi].spilled_ptr;
+ : &bpf_stack_slot(vstate->frame[e->frameno], e->spi)->spilled_ptr;
if (reg->type != SCALAR_VALUE || reg == known_reg)
continue;
if ((reg->id & ~BPF_ADD_CONST) != (known_reg->id & ~BPF_ADD_CONST))
@@ -17463,7 +17469,7 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env,
}
if (insn_flags) {
- err = bpf_push_jmp_history(env, this_branch, insn_flags, 0, 0, 0);
+ err = bpf_push_jmp_history(env, this_branch, insn_flags, 0, 0, NULL, 0);
if (err)
return err;
}
@@ -17533,7 +17539,10 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env,
* if parent state is created.
*/
if (linked_regs.cnt > 1) {
- err = bpf_push_jmp_history(env, this_branch, 0, 0, 0, linked_regs_pack(&linked_regs));
+ u16 packed[LINKED_REGS_MAX];
+
+ linked_regs_pack(&linked_regs, packed);
+ err = bpf_push_jmp_history(env, this_branch, 0, 0, 0, packed, linked_regs.cnt);
if (err)
return err;
}
@@ -18433,12 +18442,13 @@ static void idset_cnt_inc(struct bpf_idset *idset, u32 id)
return;
}
}
- /* New id */
- if (idset->num_ids < BPF_ID_MAP_SIZE) {
- idset->entries[idset->num_ids].id = id;
- idset->entries[idset->num_ids].cnt = 1;
- idset->num_ids++;
- }
+ /* New id; one that cannot be recorded counts as shared and is kept */
+ if (!bpf_id_scratch_reserve((void **)&idset->entries, &idset->cap, idset->num_ids,
+ sizeof(*idset->entries)))
+ return;
+ idset->entries[idset->num_ids].id = id;
+ idset->entries[idset->num_ids].cnt = 1;
+ idset->num_ids++;
}
/* Find id in idset and return its count, or 0 if not found */
@@ -18927,7 +18937,7 @@ static int do_check(struct bpf_verifier_env *env)
}
if (bpf_is_jmp_point(env, env->insn_idx)) {
- err = bpf_push_jmp_history(env, state, 0, 0, 0, 0);
+ err = bpf_push_jmp_history(env, state, 0, 0, 0, NULL, 0);
if (err)
return err;
}
@@ -21686,6 +21696,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
env->bt.env = env;
env->prog = *prog;
env->ops = bpf_verifier_ops[env->prog->type];
+ env->stack_limit = bpf_prog_stack_limit(env->prog);
env->allow_ptr_leaks = bpf_allow_ptr_leaks(env->prog->aux->token);
env->allow_uninit_stack = bpf_allow_uninit_stack(env->prog->aux->token);
@@ -21998,6 +22009,8 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
kvfree(env->scc_info);
kvfree(env->succ);
kvfree(env->gotox_tmp_buf);
+ kfree(env->idmap_scratch.map);
+ kfree(env->idset_scratch.entries);
bpf_diag_free(env);
kvfree(env);
return ret;
diff --git a/tools/testing/selftests/bpf/prog_tests/struct_ops_private_stack.c b/tools/testing/selftests/bpf/prog_tests/struct_ops_private_stack.c
index 98db9bafa44b2..2b3ec2b790918 100644
--- a/tools/testing/selftests/bpf/prog_tests/struct_ops_private_stack.c
+++ b/tools/testing/selftests/bpf/prog_tests/struct_ops_private_stack.c
@@ -4,6 +4,7 @@
#include "struct_ops_private_stack.skel.h"
#include "struct_ops_private_stack_fail.skel.h"
#include "struct_ops_private_stack_recur.skel.h"
+#include "struct_ops_private_stack_large.skel.h"
#if defined(__x86_64__) || defined(__aarch64__) || defined(__powerpc64__)
static void test_private_stack(void)
@@ -78,6 +79,34 @@ static void test_private_stack_recur(void)
struct_ops_private_stack_recur__destroy(skel);
}
+/* Two frames of 2 KiB each on the private stack */
+static void test_private_stack_large(void)
+{
+ struct struct_ops_private_stack_large *skel;
+ struct bpf_link *link;
+
+ if (!is_large_stack_supported()) {
+ test__skip();
+ return;
+ }
+
+ skel = struct_ops_private_stack_large__open_and_load();
+ if (!ASSERT_OK_PTR(skel, "struct_ops_private_stack_large__open_and_load"))
+ return;
+
+ link = bpf_map__attach_struct_ops(skel->maps.testmod_1);
+ if (!ASSERT_OK_PTR(link, "attach_struct_ops"))
+ goto cleanup;
+
+ ASSERT_OK(trigger_module_test_read(256), "trigger_read");
+
+ ASSERT_EQ(skel->bss->val, 100 + 30 + 12, "val");
+
+ bpf_link__destroy(link);
+cleanup:
+ struct_ops_private_stack_large__destroy(skel);
+}
+
static void __test_struct_ops_private_stack(void)
{
if (test__start_subtest("private_stack"))
@@ -86,6 +115,8 @@ static void __test_struct_ops_private_stack(void)
test_private_stack_fail();
if (test__start_subtest("private_stack_recur"))
test_private_stack_recur();
+ if (test__start_subtest("private_stack_large"))
+ test_private_stack_large();
}
#else
static void __test_struct_ops_private_stack(void)
diff --git a/tools/testing/selftests/bpf/prog_tests/tailcalls.c b/tools/testing/selftests/bpf/prog_tests/tailcalls.c
index c5c9d6c359bb0..c5e3f198ca7f6 100644
--- a/tools/testing/selftests/bpf/prog_tests/tailcalls.c
+++ b/tools/testing/selftests/bpf/prog_tests/tailcalls.c
@@ -9,6 +9,7 @@
#include "tc_bpf2bpf.skel.h"
#include "tailcall_fail.skel.h"
#include "tailcall_cgrp_storage_owner.skel.h"
+#include "tailcall_large_stack.skel.h"
#include "tailcall_cgrp_storage_no_storage.skel.h"
#include "tailcall_cgrp_storage.skel.h"
#include "tailcall_sleepable.skel.h"
@@ -1953,6 +1954,45 @@ static void test_tailcall_bpf2bpf_fexit_links(void)
tailcall_bpf2bpf2__destroy(skel_tc);
}
+/*
+ * test_tailcall_large_stack runs a tail call made from a subprog with a 1536
+ * byte frame, under a 240-byte caller, into a program with a 2 KiB frame:
+ *
+ * entry (240) --call-> subprog_tail (1536) --tailcall-> classifier_0 (2048)
+ */
+static void test_tailcall_large_stack(void)
+{
+ struct tailcall_large_stack *skel;
+ int err, prog_fd, map_fd, key = 0;
+ char buff[128] = {};
+ LIBBPF_OPTS(bpf_test_run_opts, topts,
+ .data_in = buff,
+ .data_size_in = sizeof(buff),
+ .repeat = 1,
+ );
+
+ if (!is_large_stack_supported()) {
+ test__skip();
+ return;
+ }
+
+ skel = tailcall_large_stack__open_and_load();
+ if (!ASSERT_OK_PTR(skel, "tailcall_large_stack__open_and_load"))
+ return;
+
+ prog_fd = bpf_program__fd(skel->progs.classifier_0);
+ map_fd = bpf_map__fd(skel->maps.jmp_table);
+ err = bpf_map_update_elem(map_fd, &key, &prog_fd, BPF_ANY);
+ if (!ASSERT_OK(err, "update jmp_table"))
+ goto out;
+
+ err = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.entry), &topts);
+ ASSERT_OK(err, "test_run");
+ ASSERT_EQ(topts.retval, 42 + 7, "retval");
+out:
+ tailcall_large_stack__destroy(skel);
+}
+
void test_tailcalls(void)
{
if (test__start_subtest("tailcall_1"))
@@ -2022,4 +2062,6 @@ void test_tailcalls(void)
test_tailcall_callback();
if (test__start_subtest("tailcall_bpf2bpf_fexit_links"))
test_tailcall_bpf2bpf_fexit_links();
+ if (test__start_subtest("tailcall_large_stack"))
+ test_tailcall_large_stack();
}
diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c
index 4f1e1c1cd5ab3..ced8a2f1c89c3 100644
--- a/tools/testing/selftests/bpf/prog_tests/verifier.c
+++ b/tools/testing/selftests/bpf/prog_tests/verifier.c
@@ -59,6 +59,7 @@
#include "verifier_kfunc_uninit.skel.h"
#include "verifier_kfunc_uninit_multi.skel.h"
#include "verifier_ld_ind.skel.h"
+#include "verifier_large_stack.skel.h"
#include "verifier_ldsx.skel.h"
#include "verifier_leak_ptr.skel.h"
#include "verifier_linked_scalars.skel.h"
@@ -229,6 +230,7 @@ void test_verifier_kfunc_uninit(void) { RUN_TESTS(verifier_kfunc_uninit)
void test_verifier_kfunc_uninit_multi(void) { RUN_TESTS(verifier_kfunc_uninit_multi); }
void test_verifier_load_acquire(void) { RUN(verifier_load_acquire); }
void test_verifier_ld_ind(void) { RUN(verifier_ld_ind); }
+void test_verifier_large_stack(void) { RUN(verifier_large_stack); }
void test_verifier_ldsx(void) { RUN(verifier_ldsx); }
void test_verifier_leak_ptr(void) { RUN(verifier_leak_ptr); }
void test_verifier_linked_scalars(void) { RUN(verifier_linked_scalars); }
diff --git a/tools/testing/selftests/bpf/progs/async_stack_depth.c b/tools/testing/selftests/bpf/progs/async_stack_depth.c
index 36734683acbdb..9cd874a90b39a 100644
--- a/tools/testing/selftests/bpf/progs/async_stack_depth.c
+++ b/tools/testing/selftests/bpf/progs/async_stack_depth.c
@@ -29,7 +29,49 @@ static int bad_timer_cb(void *map, int *key, struct bpf_timer *timer)
return buf[255] + timer_cb(NULL, NULL, NULL);
}
+/*
+ * The same shapes scaled to the 2 KiB budget of JITs with large stacks. The
+ * compiler caps a single function at 512 bytes, so the depth comes from a
+ * chain of 480-byte frames.
+ */
+__attribute__((noinline))
+static int timer_cb_large_0(void *map, int *key, struct bpf_timer *timer)
+{
+ volatile char buf[480] = {};
+ return buf[69];
+}
+
+__attribute__((noinline))
+static int timer_cb_large_1(void *map, int *key, struct bpf_timer *timer)
+{
+ volatile char buf[480] = {};
+ return buf[69] + timer_cb_large_0(map, key, timer);
+}
+
+__attribute__((noinline))
+static int timer_cb_large_2(void *map, int *key, struct bpf_timer *timer)
+{
+ volatile char buf[480] = {};
+ return buf[69] + timer_cb_large_1(map, key, timer);
+}
+
+__attribute__((noinline))
+static int timer_cb_large_3(void *map, int *key, struct bpf_timer *timer)
+{
+ volatile char buf[480] = {};
+ return buf[69] + timer_cb_large_2(map, key, timer);
+}
+
+/* 5 * 480 = 2400 bytes on its own */
+__attribute__((noinline))
+static int bad_timer_cb_large(void *map, int *key, struct bpf_timer *timer)
+{
+ volatile char buf[480] = {};
+ return buf[255] + timer_cb_large_3(map, key, timer);
+}
+
SEC("tc")
+__load_if_no_large_stack()
__failure __msg("combined stack size of 2 calls is")
int pseudo_call_check(struct __sk_buff *ctx)
{
@@ -44,7 +86,25 @@ int pseudo_call_check(struct __sk_buff *ctx)
return bpf_timer_set_callback(&elem->timer, timer_cb) + buf[0];
}
+/* main plus the four frames under timer_cb_large_3: 2400 bytes */
SEC("tc")
+__load_if_large_stack()
+__failure __msg("combined stack size of 5 calls is")
+int pseudo_call_check_large(struct __sk_buff *ctx)
+{
+ struct hmap_elem *elem;
+ volatile char buf[480] = {};
+
+ elem = bpf_map_lookup_elem(&hmap, &(int){0});
+ if (!elem)
+ return 0;
+
+ timer_cb_large_3(NULL, NULL, NULL);
+ return bpf_timer_set_callback(&elem->timer, timer_cb_large_3) + buf[0];
+}
+
+SEC("tc")
+__load_if_no_large_stack()
__failure __msg("combined stack size of 2 calls is")
int async_call_root_check(struct __sk_buff *ctx)
{
@@ -58,4 +118,19 @@ int async_call_root_check(struct __sk_buff *ctx)
return bpf_timer_set_callback(&elem->timer, bad_timer_cb) + buf[0];
}
+SEC("tc")
+__load_if_large_stack()
+__failure __msg("combined stack size of 5 calls is")
+int async_call_root_check_large(struct __sk_buff *ctx)
+{
+ struct hmap_elem *elem;
+ volatile char buf[480] = {};
+
+ elem = bpf_map_lookup_elem(&hmap, &(int){0});
+ if (!elem)
+ return 0;
+
+ return bpf_timer_set_callback(&elem->timer, bad_timer_cb_large) + buf[0];
+}
+
char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/bpf_misc.h b/tools/testing/selftests/bpf/progs/bpf_misc.h
index 2ced1d751acea..f3dbc3b59bff2 100644
--- a/tools/testing/selftests/bpf/progs/bpf_misc.h
+++ b/tools/testing/selftests/bpf/progs/bpf_misc.h
@@ -175,6 +175,9 @@
#define __prepare_priv __test_tag("test_prepare_priv")
#define __load_if_JITed() __test_tag("load_mode=jited")
#define __load_if_no_JITed() __test_tag("load_mode=no_jited")
+/* Whether programs may use more than 512 bytes of stack on this kernel and JIT */
+#define __load_if_large_stack() __test_tag("stack_mode=large")
+#define __load_if_no_large_stack() __test_tag("stack_mode=small")
#define __stderr(msg) __test_tag("test_expect_stderr=" msg)
#define __stderr_unpriv(msg) __test_tag("test_expect_stderr_unpriv=" msg)
#define __stdout(msg) __test_tag("test_expect_stdout=" msg)
diff --git a/tools/testing/selftests/bpf/progs/struct_ops_private_stack_fail.c b/tools/testing/selftests/bpf/progs/struct_ops_private_stack_fail.c
index 1442728f56046..c8cb35b37867b 100644
--- a/tools/testing/selftests/bpf/progs/struct_ops_private_stack_fail.c
+++ b/tools/testing/selftests/bpf/progs/struct_ops_private_stack_fail.c
@@ -4,6 +4,7 @@
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
#include "../test_kmods/bpf_testmod.h"
+#include "bpf_misc.h"
char _license[] SEC("license") = "GPL";
@@ -25,6 +26,44 @@ __noinline static int subprog1(int *a)
return subprog2(a, b);
}
+/*
+ * A chain of 480-byte frames under test_2, so that its call chain exceeds
+ * the 2 KiB budget of JITs with large stacks as well as the 512 bytes
+ * allowed elsewhere. The compiler caps a single function at 512 bytes, and
+ * the buffers are volatile so that it cannot shrink them.
+ */
+__noinline static int subprog_deep4(int *a)
+{
+ volatile char b[480] = {};
+
+ __sink(b[479]);
+ return a[10] + b[20];
+}
+
+__noinline static int subprog_deep3(int *a)
+{
+ volatile char b[480] = {};
+
+ __sink(b[479]);
+ return subprog_deep4(a) + b[20];
+}
+
+__noinline static int subprog_deep2(int *a)
+{
+ volatile char b[480] = {};
+
+ __sink(b[479]);
+ return subprog_deep3(a) + b[20];
+}
+
+__noinline static int subprog_deep1(int *a)
+{
+ volatile char b[480] = {};
+
+ __sink(b[479]);
+ return subprog_deep2(a) + b[20];
+}
+
SEC("struct_ops")
int BPF_PROG(test_1)
@@ -41,11 +80,13 @@ int BPF_PROG(test_1)
SEC("struct_ops")
int BPF_PROG(test_2)
{
- /* stack size 400 bytes */
- int a[100] = {};
+ /* stack size 476 bytes, over 2 KiB with the four 480-byte deep subprogs */
+ volatile char buf[376] = {};
+ int a[25] = {};
+ __sink(buf[375]);
a[10] = 3;
- val_j = subprog1(a);
+ val_j = subprog1(a) + subprog_deep1(a);
return 0;
}
diff --git a/tools/testing/selftests/bpf/progs/struct_ops_private_stack_large.c b/tools/testing/selftests/bpf/progs/struct_ops_private_stack_large.c
new file mode 100644
index 0000000000000..94a25a2cff6ec
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/struct_ops_private_stack_large.c
@@ -0,0 +1,51 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+#include "../test_kmods/bpf_testmod.h"
+#include "bpf_misc.h"
+
+char _license[] SEC("license") = "GPL";
+
+long val;
+
+/* On a private stack every frame gets the whole 2 KiB budget. */
+__used __naked
+static long frame_2048_leaf(void)
+{
+ asm volatile (" \
+ r1 = 30; \
+ *(u64 *)(r10 - 2048) = r1; \
+ r1 = 12; \
+ *(u64 *)(r10 - 8) = r1; \
+ r0 = *(u64 *)(r10 - 2048); \
+ r1 = *(u64 *)(r10 - 8); \
+ r0 += r1; \
+ exit; \
+" ::: __clobber_all);
+}
+
+/* test_1 is the member bpf_testmod requests a private stack for */
+SEC("struct_ops")
+__naked int test_1(void)
+{
+ asm volatile (" \
+ r1 = 100; \
+ *(u64 *)(r10 - 2048) = r1; \
+ call frame_2048_leaf; \
+ r1 = *(u64 *)(r10 - 2048); \
+ r0 += r1; \
+ r1 = %[val] ll; \
+ *(u64 *)(r1 + 0) = r0; \
+ r0 = 0; \
+ exit; \
+" :
+ : __imm_addr(val)
+ : __clobber_all);
+}
+
+SEC(".struct_ops")
+struct bpf_testmod_ops3 testmod_1 = {
+ .test_1 = (void *)test_1,
+};
diff --git a/tools/testing/selftests/bpf/progs/tailcall_large_stack.c b/tools/testing/selftests/bpf/progs/tailcall_large_stack.c
new file mode 100644
index 0000000000000..977197dac5d37
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/tailcall_large_stack.c
@@ -0,0 +1,62 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+struct {
+ __uint(type, BPF_MAP_TYPE_PROG_ARRAY);
+ __uint(max_entries, 1);
+ __uint(key_size, sizeof(__u32));
+ __uint(value_size, sizeof(__u32));
+} jmp_table SEC(".maps");
+
+/* The tail call target sets up a 2 KiB frame of its own and uses both of its ends. */
+SEC("tc")
+__naked int classifier_0(void)
+{
+ asm volatile (" \
+ r1 = 42; \
+ *(u64 *)(r10 - 2048) = r1; \
+ r1 = 7; \
+ *(u64 *)(r10 - 8) = r1; \
+ r0 = *(u64 *)(r10 - 2048); \
+ r1 = *(u64 *)(r10 - 8); \
+ r0 += r1; \
+ exit; \
+" ::: __clobber_all);
+}
+
+/*
+ * The frame of the subprog doing the tail call is unwound by it, so it may be
+ * large; only the frames of its callers stay behind and are limited to 256
+ * bytes in total. Returns 1 when the tail call falls through.
+ */
+__used __naked
+static int subprog_tail(void)
+{
+ asm volatile (" \
+ r2 = 1; \
+ *(u64 *)(r10 - 1536) = r2; \
+ r2 = %[jmp_table] ll; \
+ r3 = 0; \
+ call %[bpf_tail_call]; \
+ r0 = 1; \
+ exit; \
+" :
+ : __imm(bpf_tail_call),
+ __imm_addr(jmp_table)
+ : __clobber_all);
+}
+
+SEC("tc")
+__naked int entry(void)
+{
+ asm volatile (" \
+ r2 = 2; \
+ *(u64 *)(r10 - 240) = r2; \
+ call subprog_tail; \
+ exit; \
+" ::: __clobber_all);
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/test_global_func1.c b/tools/testing/selftests/bpf/progs/test_global_func1.c
index fc69ff18880d5..f0eca282e0d43 100644
--- a/tools/testing/selftests/bpf/progs/test_global_func1.c
+++ b/tools/testing/selftests/bpf/progs/test_global_func1.c
@@ -48,8 +48,73 @@ int f3(int val, struct __sk_buff *skb, int var)
}
SEC("tc")
+__load_if_no_large_stack()
__failure __msg("combined stack size of 3 calls is")
int global_func1(struct __sk_buff *skb)
{
return f0(1, skb) + f1(skb) + f2(2, skb) + f3(3, skb, 4);
}
+
+/*
+ * A chain of five frames that stay under 512 bytes each but add up to more
+ * than the 2 KiB budget of JITs with large stacks; the chain also exceeds
+ * 512 bytes after two frames, so it is rejected everywhere.
+ */
+#define MAX_STACK_LARGE 480
+
+__attribute__ ((noinline))
+int g0(struct __sk_buff *skb)
+{
+ volatile char buf[MAX_STACK_LARGE] = {};
+
+ __sink(buf[MAX_STACK_LARGE - 1]);
+
+ return skb->len;
+}
+
+__attribute__ ((noinline))
+int g1(struct __sk_buff *skb)
+{
+ volatile char buf[MAX_STACK_LARGE] = {};
+
+ __sink(buf[MAX_STACK_LARGE - 1]);
+
+ return g0(skb) + skb->len;
+}
+
+__attribute__ ((noinline))
+int g2(struct __sk_buff *skb)
+{
+ volatile char buf[MAX_STACK_LARGE] = {};
+
+ __sink(buf[MAX_STACK_LARGE - 1]);
+
+ return g1(skb) + skb->len;
+}
+
+__attribute__ ((noinline))
+int g3(struct __sk_buff *skb)
+{
+ volatile char buf[MAX_STACK_LARGE] = {};
+
+ __sink(buf[MAX_STACK_LARGE - 1]);
+
+ return g2(skb) + skb->len;
+}
+
+__attribute__ ((noinline))
+int g4(struct __sk_buff *skb)
+{
+ volatile char buf[MAX_STACK_LARGE] = {};
+
+ __sink(buf[MAX_STACK_LARGE - 1]);
+
+ return g3(skb) + skb->len;
+}
+
+SEC("tc")
+__failure __msg("combined stack size of {{[0-9]+}} calls is")
+int global_func1_deep(struct __sk_buff *skb)
+{
+ return g4(skb);
+}
diff --git a/tools/testing/selftests/bpf/progs/test_global_func_deep_stack.c b/tools/testing/selftests/bpf/progs/test_global_func_deep_stack.c
index 1b634b543b629..edb8a223a3cb2 100644
--- a/tools/testing/selftests/bpf/progs/test_global_func_deep_stack.c
+++ b/tools/testing/selftests/bpf/progs/test_global_func_deep_stack.c
@@ -67,12 +67,30 @@ int XCAT(f, n)(unsigned long a) \
#define F_31 F_30 FN(31, 30)
#define F_32 F_31 FN(32, 31)
+/* Same, with a 480-byte frame, to exceed the 2 KiB budget of large stacks. */
+#define FNB(n, prev) \
+__attribute__((noinline)) \
+int XCAT(f, n)(unsigned long a) \
+{ \
+ volatile char buf[480] = {}; \
+ volatile long b = XCAT(f, prev)(a - 1); \
+ if (!b) \
+ return 0; \
+ return b + buf[479] + 1; \
+}
+
+#define F_33 F_32 FNB(33, 32)
+#define F_34 F_33 FNB(34, 33)
+#define F_35 F_34 FNB(35, 34)
+#define F_36 F_35 FNB(36, 35)
+#define F_37 F_36 FNB(37, 36)
+
#define CAT2(a, b) a ## b
#define XCAT2(a, b) CAT2(a, b)
#define F(n) XCAT2(F_, n)
-F(32)
+F(37)
/* Ensure that even 32 levels deep, the function verifies. */
SEC("syscall")
@@ -88,8 +106,21 @@ int global_func_deep_stack_success(struct __sk_buff *skb)
* the size.
*/
SEC("syscall")
+__load_if_no_large_stack()
__failure __msg("combined stack size of 34 calls")
int global_func_deep_stack_fail(struct __sk_buff *skb)
{
return f32(123);
}
+
+/*
+ * Five 480-byte frames on top of the chain: 5 * 480 + 33 * 16 = 2928 bytes,
+ * more than the 2 KiB budget of JITs with large stacks, and more than 512
+ * bytes after the second frame everywhere else.
+ */
+SEC("syscall")
+__failure __msg("combined stack size of {{[0-9]+}} calls")
+int global_func_deep_stack_fail_large(struct __sk_buff *skb)
+{
+ return f37(123);
+}
diff --git a/tools/testing/selftests/bpf/progs/verifier_large_stack.c b/tools/testing/selftests/bpf/progs/verifier_large_stack.c
new file mode 100644
index 0000000000000..2d4c81a3f0cb9
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_large_stack.c
@@ -0,0 +1,377 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+/*
+ * Programs may use MAX_BPF_STACK_JIT (2 KiB) of stack on JITs that support
+ * large stacks, combined over a call chain, with no separate limit on a
+ * single frame. Interpreted programs and other JITs keep 512 bytes.
+ */
+
+SEC("socket")
+__description("single frame of 2048 bytes")
+__load_if_large_stack()
+__success __success_unpriv __retval(42)
+__naked void single_frame_2048(void)
+{
+ asm volatile (" \
+ r1 = r10; \
+ r1 += -2048; \
+ r0 = 42; \
+ *(u64*)(r1 + 0) = r0; \
+ r0 = *(u64*)(r1 + 0); \
+ exit; \
+" ::: __clobber_all);
+}
+
+SEC("socket")
+__description("single frame of 2048 bytes without large stack support")
+__load_if_no_large_stack()
+__failure __msg("invalid write to stack R1 off=-2048 size=8")
+__naked void single_frame_2048_no_large_stack(void)
+{
+ asm volatile (" \
+ r1 = r10; \
+ r1 += -2048; \
+ r0 = 42; \
+ *(u64*)(r1 + 0) = r0; \
+ exit; \
+" ::: __clobber_all);
+}
+
+__used __naked
+static void frame_512_leaf(void)
+{
+ asm volatile (" \
+ r1 = 1; \
+ *(u64 *)(r10 - 512) = r1; \
+ exit; \
+" ::: __clobber_all);
+}
+
+__used __naked
+static void frame_512_depth_2(void)
+{
+ asm volatile (" \
+ r1 = 2; \
+ *(u64 *)(r10 - 512) = r1; \
+ call frame_512_leaf; \
+ exit; \
+" ::: __clobber_all);
+}
+
+__used __naked
+static void frame_512_depth_3(void)
+{
+ asm volatile (" \
+ r1 = 3; \
+ *(u64 *)(r10 - 512) = r1; \
+ call frame_512_depth_2; \
+ exit; \
+" ::: __clobber_all);
+}
+
+__used __naked
+static void frame_512_depth_4(void)
+{
+ asm volatile (" \
+ r1 = 4; \
+ *(u64 *)(r10 - 512) = r1; \
+ call frame_512_depth_3; \
+ exit; \
+" ::: __clobber_all);
+}
+
+SEC("socket")
+__description("four frames of 512 bytes fit the 2 KiB budget")
+__load_if_large_stack()
+__success __log_level(4) __msg("stack depth max 2048")
+__naked void four_frames_of_512(void)
+{
+ asm volatile (" \
+ call frame_512_depth_4; \
+ r0 = 0; \
+ exit; \
+" ::: __clobber_all);
+}
+
+SEC("socket")
+__description("five frames of 512 bytes exceed the 2 KiB budget")
+__load_if_large_stack()
+__failure __msg("combined stack size of 5 calls is 2560. Too large")
+__naked void five_frames_of_512(void)
+{
+ asm volatile (" \
+ r1 = 5; \
+ *(u64 *)(r10 - 512) = r1; \
+ call frame_512_depth_4; \
+ r0 = 0; \
+ exit; \
+" ::: __clobber_all);
+}
+
+__used __naked
+static void frame_1536_leaf(void)
+{
+ asm volatile (" \
+ r1 = 1; \
+ *(u64 *)(r10 - 1536) = r1; \
+ exit; \
+" ::: __clobber_all);
+}
+
+SEC("socket")
+__description("512-byte frame calling a 1536-byte frame")
+__load_if_large_stack()
+__success __log_level(4) __msg("stack depth max 2048")
+__naked void uneven_frames_fit(void)
+{
+ asm volatile (" \
+ r1 = 2; \
+ *(u64 *)(r10 - 512) = r1; \
+ call frame_1536_leaf; \
+ r0 = 0; \
+ exit; \
+" ::: __clobber_all);
+}
+
+SEC("socket")
+__description("520-byte frame calling a 1536-byte frame")
+__load_if_large_stack()
+__failure __msg("combined stack size of 2 calls is 2064. Too large")
+__naked void uneven_frames_exceed(void)
+{
+ asm volatile (" \
+ r1 = 2; \
+ *(u64 *)(r10 - 520) = r1; \
+ call frame_1536_leaf; \
+ r0 = 0; \
+ exit; \
+" ::: __clobber_all);
+}
+
+#ifdef __BPF_FEATURE_MAY_GOTO
+/* may_goto adds its counter below the frame; a JIT does not hold that against the budget */
+SEC("socket")
+__description("frame of 2048 bytes with may_goto")
+__load_if_large_stack()
+__success __retval(42)
+__naked void frame_2048_with_may_goto(void)
+{
+ asm volatile (" \
+ r1 = r10; \
+ r1 += -2048; \
+ r0 = 42; \
+ *(u32*)(r1 + 0) = r0; \
+ may_goto l0_%=; \
+ r2 = 100; \
+ l0_%=: \
+ exit; \
+" ::: __clobber_all);
+}
+#endif
+
+SEC("socket")
+__description("variable offset write reaching 2048 bytes deep")
+__load_if_large_stack()
+__success
+__naked void var_off_write_to_2048(void)
+{
+ asm volatile (" \
+ call %[bpf_get_prandom_u32]; \
+ r0 &= 8; \
+ r2 = r10; \
+ r2 += -2048; \
+ r2 += r0; \
+ r1 = 0; \
+ *(u64*)(r2 + 0) = r1; \
+ r0 = 0; \
+ exit; \
+" :
+ : __imm(bpf_get_prandom_u32)
+ : __clobber_all);
+}
+
+SEC("socket")
+__description("variable offset write reaching 2056 bytes deep")
+__load_if_large_stack()
+__failure __msg("invalid variable-offset write to stack R2")
+__naked void var_off_write_to_2056(void)
+{
+ asm volatile (" \
+ call %[bpf_get_prandom_u32]; \
+ r0 &= 8; \
+ r2 = r10; \
+ r2 += -2056; \
+ r2 += r0; \
+ r1 = 0; \
+ *(u64*)(r2 + 0) = r1; \
+ r0 = 0; \
+ exit; \
+" :
+ : __imm(bpf_get_prandom_u32)
+ : __clobber_all);
+}
+
+/* Each frame of a private stack gets the whole budget. */
+__used __naked
+static void priv_stack_frame_2048(void)
+{
+ asm volatile (" \
+ r1 = 1; \
+ *(u64 *)(r10 - 2048) = r1; \
+ exit; \
+" ::: __clobber_all);
+}
+
+SEC("kprobe")
+__description("private stack: two frames of 2048 bytes")
+__load_if_large_stack()
+__arch_x86_64
+__arch_arm64
+__success __log_level(4)
+__msg("stack depth max 2048")
+__msg("subprog 0 (private_stack_two_frames) main {{.*}} stack 2048")
+__msg("subprog 1 (priv_stack_frame_2048) static {{.*}} stack 2048")
+__naked void private_stack_two_frames(void)
+{
+ asm volatile (" \
+ r1 = 2; \
+ *(u64 *)(r10 - 2048) = r1; \
+ call priv_stack_frame_2048; \
+ r0 = 0; \
+ exit; \
+" ::: __clobber_all);
+}
+
+struct {
+ __uint(type, BPF_MAP_TYPE_PROG_ARRAY);
+ __uint(max_entries, 1);
+ __uint(key_size, sizeof(__u32));
+ __uint(value_size, sizeof(__u32));
+} jmp_table SEC(".maps");
+
+/*
+ * A tail call unwinds the frame of the program doing it, so a large main
+ * frame is fine; the 256-byte rule only concerns the frames of callers of a
+ * subprog that tail calls.
+ */
+SEC("tc")
+__description("tail call from a 1 KiB frame")
+__load_if_large_stack()
+__success
+__naked void tail_call_from_large_frame(void)
+{
+ asm volatile (" \
+ r2 = 42; \
+ *(u64 *)(r10 - 1024) = r2; \
+ r2 = %[jmp_table] ll; \
+ r3 = 0; \
+ call %[bpf_tail_call]; \
+ r0 = 0; \
+ exit; \
+" :
+ : __imm(bpf_tail_call),
+ __imm_addr(jmp_table)
+ : __clobber_all);
+}
+
+/* Global subprogs are verified on their own but share the call chain budget. */
+__used __naked int global_frame_1536(void)
+{
+ asm volatile (" \
+ r1 = 1; \
+ *(u64 *)(r10 - 1536) = r1; \
+ r0 = 0; \
+ exit; \
+" ::: __clobber_all);
+}
+
+SEC("socket")
+__description("512-byte frame calling a 1536-byte global subprog")
+__load_if_large_stack()
+__success __log_level(4) __msg("stack depth max 2048")
+__naked void global_subprog_fits(void)
+{
+ asm volatile (" \
+ r1 = 2; \
+ *(u64 *)(r10 - 512) = r1; \
+ call global_frame_1536; \
+ r0 = 0; \
+ exit; \
+" ::: __clobber_all);
+}
+
+SEC("socket")
+__description("520-byte frame calling a 1536-byte global subprog")
+__load_if_large_stack()
+__failure __msg("combined stack size of 2 calls is 2064. Too large")
+__naked void global_subprog_exceeds(void)
+{
+ asm volatile (" \
+ r1 = 2; \
+ *(u64 *)(r10 - 520) = r1; \
+ call global_frame_1536; \
+ r0 = 0; \
+ exit; \
+" ::: __clobber_all);
+}
+
+/* Callback frames are part of the chain of the helper that calls them. */
+static __naked int loop_cb_1536(void)
+{
+ asm volatile (" \
+ r1 = 1; \
+ *(u64 *)(r10 - 1536) = r1; \
+ r0 = 0; \
+ exit; \
+" ::: __clobber_all);
+}
+
+SEC("socket")
+__description("512-byte frame with a 1536-byte bpf_loop callback")
+__load_if_large_stack()
+__success __log_level(4) __msg("stack depth max 2048")
+__naked void loop_callback_fits(void)
+{
+ asm volatile (" \
+ r1 = 2; \
+ *(u64 *)(r10 - 512) = r1; \
+ r1 = 1; \
+ r2 = %[loop_cb_1536]; \
+ r3 = 0; \
+ r4 = 0; \
+ call %[bpf_loop]; \
+ r0 = 0; \
+ exit; \
+" :
+ : __imm_ptr(loop_cb_1536),
+ __imm(bpf_loop)
+ : __clobber_common);
+}
+
+SEC("socket")
+__description("520-byte frame with a 1536-byte bpf_loop callback")
+__load_if_large_stack()
+__failure __msg("combined stack size of 2 calls is 2064. Too large")
+__naked void loop_callback_exceeds(void)
+{
+ asm volatile (" \
+ r1 = 2; \
+ *(u64 *)(r10 - 520) = r1; \
+ r1 = 1; \
+ r2 = %[loop_cb_1536]; \
+ r3 = 0; \
+ r4 = 0; \
+ call %[bpf_loop]; \
+ r0 = 0; \
+ exit; \
+" :
+ : __imm_ptr(loop_cb_1536),
+ __imm(bpf_loop)
+ : __clobber_common);
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/verifier_live_stack.c b/tools/testing/selftests/bpf/progs/verifier_live_stack.c
index 7a1a0670f851d..7be813f06a5ae 100644
--- a/tools/testing/selftests/bpf/progs/verifier_live_stack.c
+++ b/tools/testing/selftests/bpf/progs/verifier_live_stack.c
@@ -318,7 +318,7 @@ struct {
} map_array SEC(".maps");
SEC("socket")
-__failure __msg("invalid read from stack R2 off=-1024 size=8")
+__failure __msg("invalid read from stack R2 off=-4096 size=8")
__flag(BPF_F_TEST_STATE_FREQ)
__naked unsigned long caller_stack_write_tail_call(void)
{
@@ -329,7 +329,7 @@ __naked unsigned long caller_stack_write_tail_call(void)
"if r0 != 42 goto 1f;"
"goto 2f;"
"1:"
- "*(u64 *)(r10 - 8) = -1024;"
+ "*(u64 *)(r10 - 8) = -4096;"
"2:"
"r1 = r6;"
"r2 = r10;"
@@ -1953,7 +1953,7 @@ static __used __naked void fwd_parent_key_to_helper(void)
SEC("socket")
__log_level(2)
__success
-__msg("call bpf_map_update_elem{{.*}}; use: fp1-8..-512 fp0-8")
+__msg("call bpf_map_update_elem{{.*}}; use: fp1-8..-{{(512|2048)}} fp0-8")
__naked void helper_arg_fallback_keeps_scanning(void)
{
asm volatile (
@@ -2267,7 +2267,7 @@ static __used __naked void merge_leaf_read(void)
SEC("socket")
__log_level(2)
__success
-__msg("call bpf_loop#181 ; use: fp2-8..-512 fp1-8..-512 fp0-8..-512")
+__msg("call bpf_loop#181 ; use: fp2-8..-{{(512|2048)}} fp1-8..-{{(512|2048)}} fp0-8..-{{(512|2048)}}")
__naked void bpf_loop_two_callbacks(void)
{
asm volatile (
@@ -2839,3 +2839,74 @@ static __used __naked void imprecise_dst_spill_join_sub(void)
:: __imm(bpf_get_prandom_u32)
: __clobber_all);
}
+
+/*
+ * A store that does not fully cover a 4-byte half-slot defines nothing, so a
+ * narrow store at the top of the frame must not turn any slot into a "def",
+ * least of all every slot of the frame: the earlier data at fp-8 stays live.
+ */
+SEC("socket")
+__log_level(2)
+__msg("0: (79) r0 = *(u64 *)(r10 -8) ; use: fp0-8")
+__msg("1: (73) *(u8 *)(r10 -1) = r0{{$}}")
+__msg("2: (6b) *(u16 *)(r10 -4) = r0{{$}}")
+__msg("3: (79) r0 = *(u64 *)(r10 -8) ; use: fp0-8")
+__naked void narrow_store_defines_nothing(void)
+{
+ asm volatile (
+ "r0 = *(u64 *)(r10 - 8);"
+ "*(u8 *)(r10 - 1) = r0;"
+ "*(u16 *)(r10 - 4) = r0;"
+ "r0 = *(u64 *)(r10 - 8);"
+ "exit;"
+ ::: __clobber_all);
+}
+
+/*
+ * The same callee instance is analyzed twice: the call sites are visited in
+ * postorder, so the second one goes first with a precise pointer 264 bytes
+ * into the main frame, and the first one then passes a pointer of unknown
+ * offset, which reads the whole frame. The masks of the two passes differ in
+ * width; merging the second into the first must keep the whole-frame read.
+ */
+SEC("socket")
+__log_level(2)
+__msg("stack use/def subprog#{{[0-9]+}} merge_read_all_callee (d2,cs{{[0-9]+}}):")
+__msg("(79) r0 = *(u64 *)(r1 +0){{.*}}; use: fp0-8..-{{(512|2048)}}")
+__naked void merge_keeps_whole_frame_read(void)
+{
+ asm volatile (
+ "r1 = 0;"
+ "*(u64 *)(r10 - 8) = r1;"
+ "*(u64 *)(r10 - 16) = r1;"
+ "*(u64 *)(r10 - 264) = r1;"
+ "call %[bpf_get_prandom_u32];"
+ "r0 &= 8;"
+ "r1 = r10;"
+ "r1 += -16;"
+ "r1 += r0;"
+ "call merge_read_all_mid;"
+ "r1 = r10;"
+ "r1 += -264;"
+ "call merge_read_all_mid;"
+ "r0 = 0;"
+ "exit;"
+ :: __imm(bpf_get_prandom_u32)
+ : __clobber_all);
+}
+
+static __used __naked void merge_read_all_mid(void)
+{
+ asm volatile (
+ "call merge_read_all_callee;"
+ "exit;"
+ ::: __clobber_all);
+}
+
+static __used __naked void merge_read_all_callee(void)
+{
+ asm volatile (
+ "r0 = *(u64 *)(r1 + 0);"
+ "exit;"
+ ::: __clobber_all);
+}
diff --git a/tools/testing/selftests/bpf/progs/verifier_raw_stack.c b/tools/testing/selftests/bpf/progs/verifier_raw_stack.c
index 9f0f48ecb4216..0fe631411b9cc 100644
--- a/tools/testing/selftests/bpf/progs/verifier_raw_stack.c
+++ b/tools/testing/selftests/bpf/progs/verifier_raw_stack.c
@@ -240,6 +240,7 @@ __naked void load_bytes_spilled_regs_data(void)
SEC("tc")
__description("raw_stack: skb_load_bytes, invalid access 1")
+__load_if_no_large_stack()
__failure __msg("invalid write to stack R3 off=-513 size=8")
__naked void load_bytes_invalid_access_1(void)
{
@@ -257,6 +258,26 @@ __naked void load_bytes_invalid_access_1(void)
: __clobber_all);
}
+SEC("tc")
+__description("raw_stack: skb_load_bytes, invalid access 1, large stack")
+__load_if_large_stack()
+__failure __msg("invalid write to stack R3 off=-2049 size=8")
+__naked void load_bytes_invalid_access_1_large(void)
+{
+ asm volatile (" \
+ r2 = 4; \
+ r6 = r10; \
+ r6 += -2049; \
+ r3 = r6; \
+ r4 = 8; \
+ call %[bpf_skb_load_bytes]; \
+ r0 = *(u64*)(r6 + 0); \
+ exit; \
+" :
+ : __imm(bpf_skb_load_bytes)
+ : __clobber_all);
+}
+
SEC("tc")
__description("raw_stack: skb_load_bytes, invalid access 2")
__failure __msg("invalid write to stack R3 off=-1 size=8")
diff --git a/tools/testing/selftests/bpf/progs/verifier_stack_ptr.c b/tools/testing/selftests/bpf/progs/verifier_stack_ptr.c
index 8e8cf8232255f..3e0bea9819cab 100644
--- a/tools/testing/selftests/bpf/progs/verifier_stack_ptr.c
+++ b/tools/testing/selftests/bpf/progs/verifier_stack_ptr.c
@@ -235,6 +235,7 @@ __naked void to_stack_check_low_1(void)
SEC("socket")
__description("PTR_TO_STACK check low 2")
+__load_if_no_large_stack()
__success __failure_unpriv
__msg_unpriv("R1 stack pointer arithmetic goes out of range")
__retval(42)
@@ -250,8 +251,27 @@ __naked void to_stack_check_low_2(void)
" ::: __clobber_all);
}
+SEC("socket")
+__description("PTR_TO_STACK check low 2, large stack")
+__load_if_large_stack()
+__success __failure_unpriv
+__msg_unpriv("R1 stack pointer arithmetic goes out of range")
+__retval(42)
+__naked void to_stack_check_low_2_large(void)
+{
+ asm volatile (" \
+ r1 = r10; \
+ r1 += -2049; \
+ r0 = 42; \
+ *(u8*)(r1 + 1) = r0; \
+ r0 = *(u8*)(r1 + 1); \
+ exit; \
+" ::: __clobber_all);
+}
+
SEC("socket")
__description("PTR_TO_STACK check low 3")
+__load_if_no_large_stack()
__failure __msg("invalid write to stack R1 off=-513 size=1")
__msg_unpriv("R1 stack pointer arithmetic goes out of range")
__naked void to_stack_check_low_3(void)
@@ -266,6 +286,23 @@ __naked void to_stack_check_low_3(void)
" ::: __clobber_all);
}
+SEC("socket")
+__description("PTR_TO_STACK check low 3, large stack")
+__load_if_large_stack()
+__failure __msg("invalid write to stack R1 off=-2049 size=1")
+__msg_unpriv("R1 stack pointer arithmetic goes out of range")
+__naked void to_stack_check_low_3_large(void)
+{
+ asm volatile (" \
+ r1 = r10; \
+ r1 += -2049; \
+ r0 = 42; \
+ *(u8*)(r1 + 0) = r0; \
+ r0 = *(u8*)(r1 + 0); \
+ exit; \
+" ::: __clobber_all);
+}
+
SEC("socket")
__description("PTR_TO_STACK check low 4")
__failure __msg("math between fp pointer")
@@ -483,6 +520,7 @@ l1_%=: r0 = 42; \
SEC("socket")
__description("PTR_TO_STACK stack size > 512")
+__load_if_no_large_stack()
__failure __msg("invalid write to stack R1 off=-520 size=8")
__naked void stack_check_size_gt_512(void)
{
@@ -495,6 +533,21 @@ __naked void stack_check_size_gt_512(void)
" ::: __clobber_all);
}
+SEC("socket")
+__description("PTR_TO_STACK stack size > 2048")
+__load_if_large_stack()
+__failure __msg("invalid write to stack R1 off=-2056 size=8")
+__naked void stack_check_size_gt_2048(void)
+{
+ asm volatile (" \
+ r1 = r10; \
+ r1 += -2056; \
+ r0 = 42; \
+ *(u64*)(r1 + 0) = r0; \
+ exit; \
+" ::: __clobber_all);
+}
+
#ifdef __BPF_FEATURE_MAY_GOTO
SEC("socket")
__description("PTR_TO_STACK stack size 512 with may_goto with jit")
diff --git a/tools/testing/selftests/bpf/progs/verifier_tailcall.c b/tools/testing/selftests/bpf/progs/verifier_tailcall.c
index b4acce60fb9b9..51687da972257 100644
--- a/tools/testing/selftests/bpf/progs/verifier_tailcall.c
+++ b/tools/testing/selftests/bpf/progs/verifier_tailcall.c
@@ -28,4 +28,61 @@ __naked void invalid_map_for_tail_call(void)
: __clobber_all);
}
+struct {
+ __uint(type, BPF_MAP_TYPE_PROG_ARRAY);
+ __uint(max_entries, 1);
+ __uint(key_size, sizeof(__u32));
+ __uint(value_size, sizeof(__u32));
+} jmp_table SEC(".maps");
+
+__used __naked
+static int subprog_tail_call(void)
+{
+ asm volatile (" \
+ r2 = %[jmp_table] ll; \
+ r3 = 0; \
+ call %[bpf_tail_call]; \
+ r0 = 0; \
+ exit; \
+" :
+ : __imm(bpf_tail_call),
+ __imm_addr(jmp_table)
+ : __clobber_all);
+}
+
+/*
+ * A tail call unwinds only the frame of the subprog doing it, so the
+ * frames of its callers stay on the stack. With up to 33 tail calls in
+ * a chain the verifier caps the stack those frames may add up to at
+ * 256 bytes.
+ */
+SEC("tc")
+__description("tail call from subprog with 240 bytes of caller stack")
+__success
+__naked void tail_call_caller_stack_ok(void)
+{
+ asm volatile (" \
+ r2 = 42; \
+ *(u64 *)(r10 - 240) = r2; \
+ call subprog_tail_call; \
+ r0 = 0; \
+ exit; \
+" ::: __clobber_all);
+}
+
+SEC("tc")
+__description("tail call from subprog with 256 bytes of caller stack")
+__failure
+__msg("tail_calls are not allowed when call stack of previous frames is 256 bytes. Too large")
+__naked void tail_call_caller_stack_too_large(void)
+{
+ asm volatile (" \
+ r2 = 42; \
+ *(u64 *)(r10 - 256) = r2; \
+ call subprog_tail_call; \
+ r0 = 0; \
+ exit; \
+" ::: __clobber_all);
+}
+
char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/verifier_var_off.c b/tools/testing/selftests/bpf/progs/verifier_var_off.c
index a63e336750918..399884911ea53 100644
--- a/tools/testing/selftests/bpf/progs/verifier_var_off.c
+++ b/tools/testing/selftests/bpf/progs/verifier_var_off.c
@@ -406,6 +406,7 @@ __naked void zero_sized_access_max_out_of_bound(void)
SEC("lwt_in")
__description("indirect variable-offset stack access, min out of bound")
+__load_if_no_large_stack()
__failure __msg("invalid variable-offset read from stack R2")
__naked void access_min_out_of_bound(void)
{
@@ -433,6 +434,37 @@ __naked void access_min_out_of_bound(void)
: __clobber_all);
}
+SEC("lwt_in")
+__description("indirect variable-offset stack access, min out of bound, large stack")
+__load_if_large_stack()
+__failure __msg("invalid variable-offset read from stack R2")
+__naked void access_min_out_of_bound_large(void)
+{
+ asm volatile (" \
+ /* Fill the top 8 bytes of the stack */ \
+ r2 = 0; \
+ *(u64*)(r10 - 8) = r2; \
+ /* Get an unknown value */ \
+ r2 = *(u32*)(r1 + 0); \
+ /* Make it small and 4-byte aligned */ \
+ r2 &= 4; \
+ r2 -= 2052; \
+ /* \
+ * add it to fp. We now have either fp-2052 or fp-2048, but\
+ * we don't know which \
+ */ \
+ r2 += r10; \
+ /* dereference it indirectly */ \
+ r1 = %[map_hash_8b] ll; \
+ call %[bpf_map_lookup_elem]; \
+ r0 = 0; \
+ exit; \
+" :
+ : __imm(bpf_map_lookup_elem),
+ __imm_addr(map_hash_8b)
+ : __clobber_all);
+}
+
SEC("cgroup/skb")
__description("indirect variable-offset stack access, min_off < min_initialized")
__success
diff --git a/tools/testing/selftests/bpf/test_loader.c b/tools/testing/selftests/bpf/test_loader.c
index a6e3fcc1079c6..25eeb1c1248b3 100644
--- a/tools/testing/selftests/bpf/test_loader.c
+++ b/tools/testing/selftests/bpf/test_loader.c
@@ -45,6 +45,11 @@ enum load_mode {
NO_JITED = 1 << 1,
};
+enum stack_mode {
+ LARGE_STACK = 1 << 0,
+ SMALL_STACK = 1 << 1,
+};
+
struct test_subspec {
char *name;
char *description;
@@ -70,6 +75,7 @@ struct test_spec {
int mode_mask;
int arch_mask;
int load_mask;
+ int stack_mask;
int linear_sz;
const char *skip_reason;
bool prepare_priv;
@@ -425,6 +431,7 @@ static int parse_test_spec(struct test_loader *tester,
int err = 0;
u32 arch_mask = 0;
u32 load_mask = 0;
+ u32 stack_mask = 0;
struct btf *btf;
enum arch arch;
@@ -620,6 +627,16 @@ static int parse_test_spec(struct test_loader *tester,
err = -EINVAL;
goto cleanup;
}
+ } else if ((val = str_has_pfx(s, "stack_mode="))) {
+ if (strcmp(val, "large") == 0) {
+ stack_mask = LARGE_STACK;
+ } else if (strcmp(val, "small") == 0) {
+ stack_mask = SMALL_STACK;
+ } else {
+ PRINT_FAIL("bad stack spec: '%s'", val);
+ err = -EINVAL;
+ goto cleanup;
+ }
} else if ((msg = str_has_pfx(s, "test_expect_stderr="))) {
err = push_disasm_msg(msg, &stderr_on_next_line,
&spec->priv.stderr);
@@ -659,6 +676,7 @@ static int parse_test_spec(struct test_loader *tester,
spec->arch_mask = arch_mask ?: -1;
spec->load_mask = load_mask ?: (JITED | NO_JITED);
+ spec->stack_mask = stack_mask ?: (LARGE_STACK | SMALL_STACK);
if (spec->mode_mask == 0)
spec->mode_mask = PRIV;
@@ -1331,6 +1349,7 @@ void run_subtest(struct test_loader *tester,
{
struct test_subspec *subspec = unpriv ? &spec->unpriv : &spec->priv;
int current_runtime = is_jit_enabled() ? JITED : NO_JITED;
+ int current_stack = is_large_stack_supported() ? LARGE_STACK : SMALL_STACK;
struct bpf_program *tprog = NULL, *tprog_iter;
struct bpf_link *link, *links[32] = {};
struct test_spec *spec_iter;
@@ -1360,6 +1379,11 @@ void run_subtest(struct test_loader *tester,
return;
}
+ if ((current_stack & spec->stack_mask) == 0) {
+ test__skip();
+ return;
+ }
+
if (unpriv) {
if (!can_execute_unpriv(tester, spec)) {
test__skip();
diff --git a/tools/testing/selftests/bpf/testing_helpers.c b/tools/testing/selftests/bpf/testing_helpers.c
index d1d60451c5bcc..47fe61a1ebff1 100644
--- a/tools/testing/selftests/bpf/testing_helpers.c
+++ b/tools/testing/selftests/bpf/testing_helpers.c
@@ -517,6 +517,47 @@ bool is_jit_enabled(void)
return enabled;
}
+/*
+ * Whether the kernel accepts a program using more than 512 bytes of stack,
+ * which depends on the JIT in use. Probed once with a program that stores
+ * at the 2 KiB depth. Only the verifier's verdict on that store is cached:
+ * a load that fails for another reason, such as a missing capability, is
+ * reported and probed again on the next call.
+ */
+bool is_large_stack_supported(void)
+{
+ static int supported = -1;
+ struct bpf_insn insns[] = {
+ BPF_ST_MEM(BPF_DW, BPF_REG_10, -2048, 0),
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_EXIT_INSN(),
+ };
+ char log[1024] = {};
+ LIBBPF_OPTS(bpf_prog_load_opts, opts,
+ .log_buf = log,
+ .log_size = sizeof(log),
+ .log_level = 1,
+ );
+ int fd;
+
+ if (supported >= 0)
+ return supported;
+
+ fd = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, NULL, "GPL", insns, ARRAY_SIZE(insns),
+ &opts);
+ if (fd >= 0) {
+ close(fd);
+ supported = 1;
+ } else if (strstr(log, "invalid write to stack")) {
+ supported = 0;
+ } else {
+ fprintf(stderr, "%s: probe failed with errno %d, assuming 512 bytes:\n%s",
+ __func__, errno, log);
+ return false;
+ }
+ return supported;
+}
+
int stack_mprotect(void)
{
void *buf;
diff --git a/tools/testing/selftests/bpf/testing_helpers.h b/tools/testing/selftests/bpf/testing_helpers.h
index 1c58a2f08b645..f1505108e26a4 100644
--- a/tools/testing/selftests/bpf/testing_helpers.h
+++ b/tools/testing/selftests/bpf/testing_helpers.h
@@ -59,6 +59,7 @@ struct bpf_insn;
int get_xlated_program(int fd_prog, struct bpf_insn **buf, __u32 *cnt);
int testing_prog_flags(void);
bool is_jit_enabled(void);
+bool is_large_stack_supported(void);
int stack_mprotect(void);
/* Runs diff(1) on mismatch */
diff --git a/tools/testing/selftests/bpf/verifier/calls.c b/tools/testing/selftests/bpf/verifier/calls.c
index 8b94b87135bcf..0af237c02ddf9 100644
--- a/tools/testing/selftests/bpf/verifier/calls.c
+++ b/tools/testing/selftests/bpf/verifier/calls.c
@@ -1037,15 +1037,34 @@
.result = ACCEPT,
},
{
- "calls: stack overflow using two frames (pre-call access)",
+ /*
+ * Five 480-byte frames exceed the 2 KiB budget of JITs with large
+ * stacks, and two of them the 512 bytes allowed elsewhere.
+ */
+ "calls: stack overflow using five frames (pre-call access)",
.insns = {
/* prog 1 */
- BPF_ST_MEM(BPF_B, BPF_REG_10, -300, 0),
+ BPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),
BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1),
BPF_EXIT_INSN(),
/* prog 2 */
- BPF_ST_MEM(BPF_B, BPF_REG_10, -300, 0),
+ BPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1),
+ BPF_EXIT_INSN(),
+
+ /* prog 3 */
+ BPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1),
+ BPF_EXIT_INSN(),
+
+ /* prog 4 */
+ BPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1),
+ BPF_EXIT_INSN(),
+
+ /* prog 5 */
+ BPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),
BPF_MOV64_IMM(BPF_REG_0, 0),
BPF_EXIT_INSN(),
},
@@ -1054,15 +1073,30 @@
.result = REJECT,
},
{
- "calls: stack overflow using two frames (post-call access)",
+ "calls: stack overflow using five frames (post-call access)",
.insns = {
/* prog 1 */
BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 2),
- BPF_ST_MEM(BPF_B, BPF_REG_10, -300, 0),
+ BPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),
BPF_EXIT_INSN(),
/* prog 2 */
- BPF_ST_MEM(BPF_B, BPF_REG_10, -300, 0),
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 2),
+ BPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),
+ BPF_EXIT_INSN(),
+
+ /* prog 3 */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 2),
+ BPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),
+ BPF_EXIT_INSN(),
+
+ /* prog 4 */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 2),
+ BPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),
+ BPF_EXIT_INSN(),
+
+ /* prog 5 */
+ BPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),
BPF_MOV64_IMM(BPF_REG_0, 0),
BPF_EXIT_INSN(),
},
@@ -1127,7 +1161,7 @@
.result = ACCEPT,
},
{
- "calls: stack depth check using three frames. test3",
+ "calls: stack depth check using five frames. test3",
.insns = {
/* main */
BPF_MOV64_REG(BPF_REG_6, BPF_REG_1),
@@ -1135,66 +1169,104 @@
BPF_MOV64_REG(BPF_REG_1, BPF_REG_6),
BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 8), /* call B */
BPF_JMP_IMM(BPF_JGE, BPF_REG_6, 0, 1),
- BPF_ST_MEM(BPF_B, BPF_REG_10, -64, 0),
+ BPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),
BPF_MOV64_IMM(BPF_REG_0, 0),
BPF_EXIT_INSN(),
/* A */
BPF_JMP_IMM(BPF_JLT, BPF_REG_1, 10, 1),
BPF_EXIT_INSN(),
- BPF_ST_MEM(BPF_B, BPF_REG_10, -224, 0),
+ BPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),
BPF_JMP_IMM(BPF_JA, 0, 0, -3),
/* B */
BPF_JMP_IMM(BPF_JGT, BPF_REG_1, 2, 1),
- BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, -6), /* call A */
- BPF_ST_MEM(BPF_B, BPF_REG_10, -256, 0),
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 2), /* call C */
+ BPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),
+ BPF_EXIT_INSN(),
+ /* C */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 2), /* call D */
+ BPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),
+ BPF_EXIT_INSN(),
+ /* D */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, -12), /* call A */
+ BPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),
BPF_EXIT_INSN(),
},
.prog_type = BPF_PROG_TYPE_XDP,
- /* stack_main=64, stack_A=224, stack_B=256
- * and max(main+A, main+A+B) > 512
+ /*
+ * every frame is 480 bytes, main+A = 960 > 512 and
+ * max(main+A, main+B+C+D+A) = 2400 > 2048
*/
.errstr = "combined stack",
.result = REJECT,
},
{
- "calls: stack depth check using three frames. test4",
- /* void main(void) {
+ "calls: stack depth check using five frames. test4",
+ /*
+ * void main(void) {
* func1(0);
* func1(1);
* func2(1);
* }
- * void func1(int alloc_or_recurse) {
+ * void funcN(int alloc_or_recurse) { N = 1..4
* if (alloc_or_recurse) {
- * frame_pointer[-300] = 1;
+ * frame_pointer[-480] = 1;
* } else {
- * func2(alloc_or_recurse);
+ * funcN+1(alloc_or_recurse);
* }
* }
- * void func2(int alloc_or_recurse) {
+ * void func5(int alloc_or_recurse) {
* if (alloc_or_recurse) {
- * frame_pointer[-300] = 1;
+ * frame_pointer[-480] = 1;
* }
* }
+ * main also calls func2 to func5 with 1 so that every function has a
+ * path allocating its 480 bytes, and the chain adds up to 2400 bytes,
+ * more than the 2 KiB budget of JITs with large stacks, and to 960
+ * bytes after two frames, more than the 512 bytes allowed elsewhere.
*/
.insns = {
/* main */
BPF_MOV64_IMM(BPF_REG_1, 0),
- BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 6), /* call A */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 12), /* call A */
BPF_MOV64_IMM(BPF_REG_1, 1),
- BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 4), /* call A */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 10), /* call A */
+ BPF_MOV64_IMM(BPF_REG_1, 1),
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 13), /* call B */
+ BPF_MOV64_IMM(BPF_REG_1, 1),
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 16), /* call C */
BPF_MOV64_IMM(BPF_REG_1, 1),
- BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 7), /* call B */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 19), /* call D */
+ BPF_MOV64_IMM(BPF_REG_1, 1),
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 22), /* call E */
BPF_MOV64_IMM(BPF_REG_0, 0),
BPF_EXIT_INSN(),
/* A */
BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 2),
- BPF_ST_MEM(BPF_B, BPF_REG_10, -300, 0),
+ BPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),
BPF_EXIT_INSN(),
BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call B */
BPF_EXIT_INSN(),
/* B */
+ BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 2),
+ BPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),
+ BPF_EXIT_INSN(),
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call C */
+ BPF_EXIT_INSN(),
+ /* C */
+ BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 2),
+ BPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),
+ BPF_EXIT_INSN(),
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call D */
+ BPF_EXIT_INSN(),
+ /* D */
+ BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 2),
+ BPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),
+ BPF_EXIT_INSN(),
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call E */
+ BPF_EXIT_INSN(),
+ /* E */
BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 1),
- BPF_ST_MEM(BPF_B, BPF_REG_10, -300, 0),
+ BPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),
BPF_EXIT_INSN(),
},
.prog_type = BPF_PROG_TYPE_XDP,
|
| 3/2 | 2026/09/26 01:41 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 64784 output: 30 thoughts: 642 Thoughts: **Analyzing the BPF Patch for Potential Impact** Okay, let's dive into this patch and review request. My initial thought is to understand the scope and potential implications. First, I need to get a handle on what this commit is actually doing. Based on the commit message and the diff, it looks like changes are scattered across quite a few files: documentation, arch-specific JIT comp (both arm64 and x86), some include headers, and most significantly, a large chunk of the core BPF subsystem files: the verifier, backtracking, diagnostics, liveness analysis, state management, and logging. Plus, there are changes in selftests, which is a good sign that testing is involved. The target architecture mentioned is amd64, so I'll keep that in mind. The subsystem clearly points to BPF. The crux of the changes seems to be enabling support for larger stack sizes in BPF programs – up to 2048 bytes on both x86-64 and arm64. This is a pretty significant increase. This stack size change necessitates modifications to the verifier data structures. I see that `MAX_BPF_STACK_JIT` is now 2048, and consequently, `MAX_BPF_STACK_SLOTS` is 256. More importantly, the stack masks have evolved from simple u64 or fixed bitmaps to dynamic or larger bitmaps using `DECLARE_BITMAP` and dynamic allocation (like `stack_masks` and `frame_masks`). We're also seeing some dynamic growth in idmap and idsetscratch with `bpf_id_scratch_reserve`. The encoding for linked registers has also been touched. This suggests a pretty fundamental restructuring of how stack management is handled. The bulk of the changes appear to be in `kernel/bpf/liveness.c`. This is where the variable width frame masks are added which requires attention. I'm seeing functions like `widen_frame_masks`, `mark_stack_range`, `merge_instances`, and `update_insn`. Also, changes are noticeable in `kernel/bpf/verifier.c`, `kernel/bpf/states.c`, and `kernel/bpf/backtrack.c`, which tells me the core BPF logic is being heavily modified. Considering these modifications, I need to determine if executable code is being touched. Definitely! The changes to the core BPF verifier logic, backtracking, liveness analysis, and stack-safety are key. The BPF subsystem is reachable from userspace via the `bpf()` syscall, specifically `BPF_PROG_LOAD`. This raises a red flag in terms of potential vulnerabilities, making this patch definitely `WorthFuzzing=true`. Next, I need to understand the configuration implications. The BPF verifier is enabled when `CONFIG_BPF_SYSCALL` is set. The JIT on x86 is enabled with `CONFIG_BPF_JIT`. Now, I need to identify if there are any other configuration dependencies. I'll use `grepper` to search through the Kconfig files for bpf jit, or check the kernel configuration directly to identify any other requirements. |
| 4/2 | 2026/09/26 01:42 | tool | grepper |
0mError:no matches Args: {
"Expression": "config BPF_JIT",
"PathPrefix": "init/Kconfig"
}Results: {
"error": "no matches"
} |
| 5/2 | 2026/09/26 01:42 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 66150 output: 21 thoughts: 20 Thoughts: Here's my thought process, as if I were analyzing that: **Investigating BPF_JIT and BPF_SYSCALL Definitions** Okay, so I need to track down the exact definitions of `BPF_JIT` and `BPF_SYSCALL`. I'm going to start by looking for their declaration, probably in the kernel source code. Knowing where these are defined will be crucial for understanding their role in BPF operation, especially related to just-in-time compilation and interaction with system calls. The headers will likely provide insight into their values, and ultimately, how they influence how BPF programs are handled at a low level. I'll probably start with the standard BPF headers, maybe looking in the `linux/bpf.h` file, as that's usually the go-to place for BPF-related constants and structures. I'll need to check the kernel configuration files and build flags to understand if these symbols are conditional, or version dependent. Then, I can start tracking down where these are used within the kernel. |
| 6/2 | 2026/09/26 01:42 | tool | grepper |
0mArgs:{
"Expression": "config BPF_JIT$"
}Results: {
"Output": "kernel/bpf/Kconfig=31=config BPF_SYSCALL\n--\nkernel/bpf/Kconfig-45-\nkernel/bpf/Kconfig:46:config BPF_JIT\nkernel/bpf/Kconfig-47-\tbool \"Enable BPF Just In Time compiler\"\n"
} |
| 7/2 | 2026/09/26 01:42 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 66502 output: 34 thoughts: 22 Thoughts: Here's a first-person summary, tailored for an expert audience, as requested: **Diving into the BPF Kernel Configuration** Okay, time to dig into this. I'm going to navigate directly to the `kernel/bpf/Kconfig` file. Specifically, I'm thinking the meat of the configuration options related to BPF are likely going to be clustered between lines 30 and 60, give or take. That's usually where the more core and fundamental settings tend to reside, assuming a logical organization. I'll focus on those lines to get a quick overview of what's available and if there's any immediate indication of what's going on with the particular issue I'm looking into. Hopefully, I'll find something illuminating within that range. Let's see what's there. |
| 8/2 | 2026/09/26 01:42 | tool | read-file |
0mArgs:{
"File": "kernel/bpf/Kconfig",
"FirstLine": 30,
"LineCount": 40
}Results: {
"Contents": " 30:\t\n 31:\tconfig BPF_SYSCALL\n 32:\t\tbool \"Enable bpf() system call\"\n 33:\t\tselect BPF\n 34:\t\tselect IRQ_WORK\n 35:\t\tselect NEED_TASKS_RCU\n 36:\t\tselect TASKS_TRACE_RCU\n 37:\t\tselect BINARY_PRINTF\n 38:\t\tselect NET_SOCK_MSG if NET\n 39:\t\tselect NET_XGRESS if NET\n 40:\t\tselect PAGE_POOL if NET\n 41:\t\tdefault n\n 42:\t\thelp\n 43:\t\t Enable the bpf() system call that allows to manipulate BPF programs\n 44:\t\t and maps via file descriptors.\n 45:\t\n 46:\tconfig BPF_JIT\n 47:\t\tbool \"Enable BPF Just In Time compiler\"\n 48:\t\tdepends on BPF\n 49:\t\tdepends on HAVE_CBPF_JIT || HAVE_EBPF_JIT\n 50:\t\tselect EXECMEM\n 51:\t\thelp\n 52:\t\t BPF programs are normally handled by a BPF interpreter. This option\n 53:\t\t allows the kernel to generate native code when a program is loaded\n 54:\t\t into the kernel. This will significantly speed-up processing of BPF\n 55:\t\t programs.\n 56:\t\n 57:\t\t Note, an admin should enable this feature changing:\n 58:\t\t /proc/sys/net/core/bpf_jit_enable\n 59:\t\t /proc/sys/net/core/bpf_jit_harden (optional)\n 60:\t\t /proc/sys/net/core/bpf_jit_kallsyms (optional)\n 61:\t\n 62:\tconfig BPF_JIT_ALWAYS_ON\n 63:\t\tbool \"Permanently enable BPF JIT and remove BPF interpreter\"\n 64:\t\tdepends on BPF_SYSCALL \u0026\u0026 HAVE_EBPF_JIT \u0026\u0026 BPF_JIT\n 65:\t\thelp\n 66:\t\t Enables BPF JIT and removes BPF interpreter to avoid speculative\n 67:\t\t execution of BPF instructions by the interpreter.\n 68:\t\n 69:\t\t When CONFIG_BPF_JIT_ALWAYS_ON is enabled, /proc/sys/net/core/bpf_jit_enable\n"
} |
| 9/2 | 2026/09/26 01:42 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 67344 output: 28 thoughts: 181 Thoughts: Here's my analysis of the provided code snippet: **Analyzing `bpf_prog_stack_limit()`** Alright, let's break this down. The core of this function, `bpf_prog_stack_limit()`, seems straightforward: it's determining the maximum stack size allowed for a BPF program. I see two potential limits: `MAX_BPF_STACK` and `MAX_BPF_STACK_JIT`. The crucial factor in choosing between them is whether the program is intended to be JIT compiled. The logic is fairly clear. If the program is marked as `prog->jit_requested`, *and* it's not offloaded (using `bpf_prog_is_offloaded()`, checking `prog->aux`), *and* the JIT environment supports a large stack *and* subprogram tailcalls, we use `MAX_BPF_STACK_JIT`. Otherwise, we fall back to `MAX_BPF_STACK`. This makes sense; if the JIT can handle a larger stack safely, we'll give it more room. That is dependent on the target platform supporting a large stack, and supporting tailcalls. But, and here's the key question, I need to understand when and how `prog->jit_requested` is actually set. I'd better trace back how this member is initialized, or at the very least, where it's checked. I need to be sure I understand the semantics of `jit_requested`. Knowing the conditions that set this flag is essential to fully understand the behavior of this stack limit calculation. Knowing this will allow me to reason about the implications of the stack limit more accurately. |
| 10/2 | 2026/09/26 01:42 | tool | grepper |
0mArgs:{
"Expression": "jit_requested",
"PathPrefix": "kernel/bpf/"
}Results: {
"Output": "kernel/bpf/core.c=100=struct bpf_prog *bpf_prog_alloc_no_stats(unsigned int size, gfp_t gfp_extra_flags)\n--\nkernel/bpf/core.c-127-\tfp-\u003eaux-\u003eprog = fp;\nkernel/bpf/core.c:128:\tfp-\u003ejit_requested = ebpf_jit_enabled();\nkernel/bpf/core.c-129-\tfp-\u003ejit_required = IS_ENABLED(CONFIG_BPF_JIT_ALWAYS_ON);\n--\nkernel/bpf/core.c=179=int bpf_prog_alloc_jited_linfo(struct bpf_prog *prog)\nkernel/bpf/core.c-180-{\nkernel/bpf/core.c:181:\tif (!prog-\u003eaux-\u003enr_linfo || !prog-\u003ejit_requested)\nkernel/bpf/core.c-182-\t\treturn 0;\n--\nkernel/bpf/fixups.c=1098=static int jit_subprogs(struct bpf_verifier_env *env)\n--\nkernel/bpf/fixups.c-1198-\nkernel/bpf/fixups.c:1199:\t\tfunc[i]-\u003ejit_requested = 1;\nkernel/bpf/fixups.c-1200-\t\tfunc[i]-\u003eblinding_requested = prog-\u003eblinding_requested;\n--\nkernel/bpf/fixups.c=1366=int bpf_jit_subprogs(struct bpf_verifier_env *env)\n--\nkernel/bpf/fixups.c-1431-\t/* cleanup main prog to be interpreted */\nkernel/bpf/fixups.c:1432:\tprog-\u003ejit_requested = 0;\nkernel/bpf/fixups.c-1433-\tprog-\u003eblinding_requested = 0;\n--\nkernel/bpf/fixups.c=1437=int bpf_fixup_call_args(struct bpf_verifier_env *env)\n--\nkernel/bpf/fixups.c-1457-\nkernel/bpf/fixups.c:1458:\tif (env-\u003eprog-\u003ejit_requested \u0026\u0026\nkernel/bpf/fixups.c-1459-\t !bpf_prog_is_offloaded(env-\u003eprog-\u003eaux)) {\n--\nkernel/bpf/fixups.c=1540=int bpf_do_misc_fixups(struct bpf_verifier_env *env)\n--\nkernel/bpf/fixups.c-1968-\t\t\tif (env-\u003ebpf_capable \u0026\u0026 !prog-\u003eblinding_requested \u0026\u0026\nkernel/bpf/fixups.c:1969:\t\t\t prog-\u003ejit_requested \u0026\u0026\nkernel/bpf/fixups.c-1970-\t\t\t !bpf_map_key_poisoned(aux) \u0026\u0026\n--\nkernel/bpf/fixups.c-2079-\t\t */\nkernel/bpf/fixups.c:2080:\t\tif (prog-\u003ejit_requested \u0026\u0026 BITS_PER_LONG == 64 \u0026\u0026\nkernel/bpf/fixups.c-2081-\t\t (insn-\u003eimm == BPF_FUNC_map_lookup_elem ||\n--\nkernel/bpf/fixups.c-2178-\t\t/* Implement bpf_jiffies64 inline. */\nkernel/bpf/fixups.c:2179:\t\tif (prog-\u003ejit_requested \u0026\u0026 BITS_PER_LONG == 64 \u0026\u0026\nkernel/bpf/fixups.c-2180-\t\t insn-\u003eimm == BPF_FUNC_jiffies64) {\n--\nkernel/bpf/fixups.c-2359-\t\tif (IS_ENABLED(CONFIG_PERF_EVENTS) \u0026\u0026\nkernel/bpf/fixups.c:2360:\t\t prog-\u003ejit_requested \u0026\u0026 BITS_PER_LONG == 64 \u0026\u0026\nkernel/bpf/fixups.c-2361-\t\t insn-\u003eimm == BPF_FUNC_get_branch_snapshot) {\n--\nkernel/bpf/fixups.c-2414-\t\t/* Implement bpf_kptr_xchg inline */\nkernel/bpf/fixups.c:2415:\t\tif (prog-\u003ejit_requested \u0026\u0026 BITS_PER_LONG == 64 \u0026\u0026\nkernel/bpf/fixups.c-2416-\t\t insn-\u003eimm == BPF_FUNC_kptr_xchg \u0026\u0026\n--\nkernel/bpf/fixups.c-2448-\t\t\tstack_depth = subprogs[cur_subprog].stack_depth;\nkernel/bpf/fixups.c:2449:\t\t\tif (stack_depth \u003e MAX_BPF_STACK \u0026\u0026 !prog-\u003ejit_requested) {\nkernel/bpf/fixups.c-2450-\t\t\t\tverbose(env, \"stack size %d(extra %d) is too large\\n\",\n--\nkernel/bpf/verifier.c=437=static int bpf_compute_subprog_ret_regs(struct bpf_verifier_env *env)\n--\nkernel/bpf/verifier.c-443-\nkernel/bpf/verifier.c:444:\tif (!env-\u003eprog-\u003ejit_requested || bpf_prog_is_offloaded(env-\u003eprog-\u003eaux))\nkernel/bpf/verifier.c-445-\t\treturn 0;\n--\nkernel/bpf/verifier.c=2866=int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset)\n--\nkernel/bpf/verifier.c-2886-\nkernel/bpf/verifier.c:2887:\t\tif (!env-\u003eprog-\u003ejit_requested) {\nkernel/bpf/verifier.c-2888-\t\t\tverbose(env, \"JIT is required for calling kernel function\\n\");\n--\nkernel/bpf/verifier.c=5360=static int round_up_stack_depth(struct bpf_verifier_env *env, int stack_depth)\nkernel/bpf/verifier.c-5361-{\nkernel/bpf/verifier.c:5362:\tif (env-\u003eprog-\u003ejit_requested)\nkernel/bpf/verifier.c-5363-\t\treturn round_up(stack_depth, 16);\n--\nkernel/bpf/verifier.c=9669=bool bpf_allow_tail_call_in_subprogs(struct bpf_verifier_env *env)\nkernel/bpf/verifier.c-9670-{\nkernel/bpf/verifier.c:9671:\treturn env-\u003eprog-\u003ejit_requested \u0026\u0026\nkernel/bpf/verifier.c-9672-\t bpf_jit_supports_subprog_tailcalls();\n--\nkernel/bpf/verifier.c=18165=bool bpf_verifier_inlines_helper_call(struct bpf_verifier_env *env, s32 imm)\n--\nkernel/bpf/verifier.c-18173-#endif\nkernel/bpf/verifier.c:18174:\t\treturn env-\u003eprog-\u003ejit_requested \u0026\u0026 bpf_jit_supports_percpu_insn();\nkernel/bpf/verifier.c-18175-#endif\n--\nkernel/bpf/verifier.c=19285=static int check_map_prog_compatibility(struct bpf_verifier_env *env,\n--\nkernel/bpf/verifier.c-19375-\t\t}\nkernel/bpf/verifier.c:19376:\t\tif (!env-\u003eprog-\u003ejit_requested) {\nkernel/bpf/verifier.c-19377-\t\t\tverbose(env, \"JIT is required to use arena\\n\");\n--\nkernel/bpf/verifier.c=20563=int bpf_check_attach_target(struct bpf_verifier_log *log,\n--\nkernel/bpf/verifier.c-20644-\t\t\t}\nkernel/bpf/verifier.c:20645:\t\t\tif (!prog-\u003ejit_requested) {\nkernel/bpf/verifier.c-20646-\t\t\t\tbpf_log(log,\n"
} |
| 11/2 | 2026/09/26 01:42 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 69970 output: 28 thoughts: 31 Thoughts: Here's my take on the thought process, as requested: **Checking if JIT Compilation is Desired** Okay, so I'm looking at this line: `fp->jit_requested = ebpf_jit_enabled();`. My immediate thought is, "Right, we're setting a flag, `jit_requested`, within the function's internal state (`fp`)." It looks like we're determining if JIT (Just-In-Time) compilation should be enabled. That makes sense, JIT can give us a significant performance boost in many cases. Now, what does `ebpf_jit_enabled()` actually *do*? I need to understand the underlying logic. Is it checking for command-line flags, environment variables, system capabilities, or perhaps configuration files? I need to dig into that function to understand the criteria used to decide whether JIT is requested. This will impact the overall performance of the eBPF program, so understanding the logic here is crucial to ensure optimal execution. |
| 12/2 | 2026/09/26 01:42 | tool | grepper |
0mArgs:{
"Expression": "ebpf_jit_enabled",
"PathPrefix": "include/"
}Results: {
"Output": "include/linux/filter.h=1472=static inline bool bpf_jit_is_ebpf(void)\n--\ninclude/linux/filter.h-1480-\ninclude/linux/filter.h:1481:static inline bool ebpf_jit_enabled(void)\ninclude/linux/filter.h-1482-{\n--\ninclude/linux/filter.h=1532=void bpf_prog_kallsyms_del(struct bpf_prog *fp);\n--\ninclude/linux/filter.h-1535-\ninclude/linux/filter.h:1536:static inline bool ebpf_jit_enabled(void)\ninclude/linux/filter.h-1537-{\n"
} |
| 13/2 | 2026/09/26 01:42 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 70431 output: 36 thoughts: 20 Thoughts: Okay, here's my interpretation of what's going on, formatted as you requested: **Diving into eBPF Filter Implementation** Alright, let's get down to brass tacks and dig into the implementation details. I'm going to navigate directly to `include/linux/filter.h`, specifically around line 1481. This is where the real meat of the eBPF filter structure likely lives, and it's where I need to refresh my memory on the core data structures involved in instruction processing and filter evaluation. I need a clear understanding of what that section specifically holds to understand what the next steps are, but I am expecting to find details around the bytecode representation of the eBPF programs. I need a refresher on the internal representation to be able to follow through with the next phase of this. |
| 14/2 | 2026/09/26 01:42 | tool | read-file |
0mArgs:{
"File": "include/linux/filter.h",
"FirstLine": 1475,
"LineCount": 25
}Results: {
"Contents": "1475:\t\treturn true;\n1476:\t# else\n1477:\t\treturn false;\n1478:\t# endif\n1479:\t}\n1480:\t\n1481:\tstatic inline bool ebpf_jit_enabled(void)\n1482:\t{\n1483:\t\treturn bpf_jit_enable \u0026\u0026 bpf_jit_is_ebpf();\n1484:\t}\n1485:\t\n1486:\tstatic inline bool bpf_prog_ebpf_jited(const struct bpf_prog *fp)\n1487:\t{\n1488:\t\treturn fp-\u003ejited \u0026\u0026 bpf_jit_is_ebpf();\n1489:\t}\n1490:\t\n1491:\tstatic inline bool bpf_jit_blinding_enabled(struct bpf_prog *prog)\n1492:\t{\n1493:\t\t/* These are the prerequisites, should someone ever have the\n1494:\t\t * idea to call blinding outside of them, we make sure to\n1495:\t\t * bail out.\n1496:\t\t */\n1497:\t\tif (!bpf_jit_is_ebpf())\n1498:\t\t\treturn false;\n1499:\t\tif (!prog-\u003ejit_requested)\n"
} |
| 15/2 | 2026/09/26 01:42 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 71007 output: 29 thoughts: 510 Thoughts: **Analyzing BPF JIT and Liveness Changes** Okay, so I'm looking at this commit related to BPF, and the key configuration options here seem to be `BPF_SYSCALL`, `BPF_JIT`, and possibly `BPF_JIT_ALWAYS_ON`. My goal is to understand what's changed and figure out the best way to define the `FocusSymbols` based on the code changes. I need to get this right, so I'm going to review the diff carefully. First, I'm noting the files touched and the functions modified or added. `arch/x86/net/bpf_jit_comp.c` and `kernel/bpf/core.c` both contain a function `bpf_jit_supports_large_stack`. Then, I have a more interesting set of changes in `kernel/bpf/backtrack.c`, including several new functions: `bpf_push_jmp_history`, `bpf_fmt_stack_mask`, `backtrack_insn`, `bpf_mark_all_scalars_precise`, and `bpf_mark_chain_precision`. However, the real meat of the changes seems to be in `kernel/bpf/liveness.c`. This file has a *ton* of new functions, most of which are static. I'm seeing a lot of stack manipulation and analysis functions, which makes sense. I see static variables like `frame_mask_bits`, `frame_mask_words`, and `rel_mask`, and static functions like `widen_frame_masks`, `mark_stack_range`, `mark_stack_read`, `mark_stack_write`, `mark_stack_read_all`, `mark_stack_read_mask`, `update_insn`, `is_live_before`, `fmt_spis_mask`, `print_instance`, `record_stack_access_off`, `record_stack_access`, `record_imprecise`, `record_load_store_access`, `record_arg_access`, and `merge_instances`. `update_insn` is also inline. I also see a non-static `update_instance` and a non-static `analyze_subprog`. Now, I need to focus on `liveness.c` and understand the implications, specifically if and how these static functions are used within `liveness.c`, and if any functions within `liveness.c` are entry points that are called from elsewhere. I'm going to determine which functions in `liveness.c` are exported or non-static and *how* they utilize all those static stack-related functions. I need to see what's *calling* `analyze_subprog`. That's likely going to be the key to understanding the control flow and impact of these changes. |
| 16/2 | 2026/09/26 01:42 | tool | grepper |
0mArgs:{
"Expression": "analyze_subprog",
"PathPrefix": "kernel/bpf/"
}Results: {
"Output": "kernel/bpf/liveness.c=72=struct bpf_liveness {\n--\nkernel/bpf/liveness.c-74-\tstruct live_stack_query live_stack_query;\t/* cache to avoid repetitive ht lookups */\nkernel/bpf/liveness.c:75:\tu32 subprog_calls;\t\t\t\t/* analyze_subprog() invocations */\nkernel/bpf/liveness.c-76-};\n--\nkernel/bpf/liveness.c=1951=static void free_instance(struct func_instance *instance)\n--\nkernel/bpf/liveness.c-1968- */\nkernel/bpf/liveness.c:1969:static int analyze_subprog(struct bpf_verifier_env *env,\nkernel/bpf/liveness.c-1970-\t\t\t struct arg_track *entry_args,\n--\nkernel/bpf/liveness.c-2079-\t\tcallsites[depth] = idx;\nkernel/bpf/liveness.c:2080:\t\terr = analyze_subprog(env, callee_args, info, callee_instance, callsites);\nkernel/bpf/liveness.c-2081-\t\tif (err)\n--\nkernel/bpf/liveness.c=2121=int bpf_compute_subprog_arg_access(struct bpf_verifier_env *env)\n--\nkernel/bpf/liveness.c-2142-\t * before callees) so that each subprog is analyzed before its\nkernel/bpf/liveness.c:2143:\t * callees, allowing the recursive walk inside analyze_subprog()\nkernel/bpf/liveness.c-2144-\t * to naturally reach callees that receive FP-derived args.\n--\nkernel/bpf/liveness.c-2159-\t\t}\nkernel/bpf/liveness.c:2160:\t\terr = analyze_subprog(env, NULL, info, instance, callsites);\nkernel/bpf/liveness.c-2161-\t\tif (err)\n"
} |
| 17/2 | 2026/09/26 01:42 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 72689 output: 24 thoughts: 850 Thoughts: **Analyzing BPF Function Visibility and Suitability for Targeted Coverage** Okay, so I'm looking at BPF code and trying to figure out which functions are good candidates for targeted coverage analysis. My initial thought is to examine the `bpf_compute_subprog_arg_access` function in `kernel/bpf/liveness.c`. It's non-static, which is a good starting point. Now, let's explore functions in other relevant files. I'll check `kernel/bpf/states.c`: I see `bpf_id_scratch_reserve` (non-static), which is also promising, along with some static functions like `stacksafe`, `func_states_equal`, `propagate_precision`, and `__clean_func_state`. There's also `bpf_is_state_visited`, which is non-static and another one to keep on the radar. Next, I'll switch to `kernel/bpf/verifier.c`. Here, there are several static functions like `check_stack_write_fixed_off`, `check_stack_write_var_off`, `check_stack_read_fixed_off`, `check_max_stack_depth_subprog`, `check_stack_slot_within_bounds`, `check_stack_range_initialized`, `check_cond_jmp_op`, and `do_check`. I also notice `bpf_push_jmp_history` in `backtrack.c` and `bpf_bt_sync_linked_regs` (defined in verifier.c/backtrack.c) which are both non-static. Finally, `bpf_check` is non-static. Now, I need to consider whether static functions can be FocusSymbols in the context of tools like syzkaller and KCOV/ftrace. I remember that FocusSymbols are typically based on the symbol table in vmlinux or kallsyms. Static functions, unless inlined by the compiler, are indeed present in the symbol table, in .c files. But the documentation gives specific guidance: I must avoid those ubiquitous, generic functions, and instead target feature-specific logic. The key is to find non-static or key functions. Also, if there are changes primarily in static inline helpers, I should identify non-static, feature-specific caller functions in the .c files that exercise them. So, I'll compile a list of non-static functions from my checks: `bpf_jit_supports_large_stack`, `bpf_push_jmp_history`, `bpf_mark_chain_precision`, `bpf_compute_subprog_arg_access`, `bpf_id_scratch_reserve`. Also `check_stack_write_fixed_off`, `check_stack_read_fixed_off`, `check_max_stack_depth_subprog`, `stacksafe`. I'll revisit that list and confirm whether they exist in the kernel and review their definitions and usage, and see if I'm targeting code that fulfills the requirements of targeted coverage. |
| 18/2 | 2026/09/26 01:42 | tool | grepper |
0mArgs:{
"Expression": "bpf_jit_supports_large_stack"
}Results: {
"Output": "arch/arm64/net/bpf_jit_comp.c=2483=bool bpf_jit_supports_subprog_tailcalls(void)\n--\narch/arm64/net/bpf_jit_comp.c-2487-\narch/arm64/net/bpf_jit_comp.c:2488:bool bpf_jit_supports_large_stack(void)\narch/arm64/net/bpf_jit_comp.c-2489-{\n--\narch/x86/net/bpf_jit_comp.c=4450=bool bpf_jit_supports_subprog_tailcalls(void)\n--\narch/x86/net/bpf_jit_comp.c-4461- */\narch/x86/net/bpf_jit_comp.c:4462:bool bpf_jit_supports_large_stack(void)\narch/x86/net/bpf_jit_comp.c-4463-{\n--\ninclude/linux/bpf_verifier.h=1249=static inline u32 bpf_prog_stack_limit(const struct bpf_prog *prog)\n--\ninclude/linux/bpf_verifier.h-1252-\tif (prog-\u003ejit_requested \u0026\u0026 !bpf_prog_is_offloaded(prog-\u003eaux) \u0026\u0026\ninclude/linux/bpf_verifier.h:1253:\t bpf_jit_supports_large_stack() \u0026\u0026 bpf_jit_supports_subprog_tailcalls())\ninclude/linux/bpf_verifier.h-1254-\t\treturn MAX_BPF_STACK_JIT;\n--\ninclude/linux/filter.h=1253=bool bpf_jit_supports_private_stack(void);\ninclude/linux/filter.h:1254:bool bpf_jit_supports_large_stack(void);\ninclude/linux/filter.h-1255-bool bpf_jit_supports_timed_may_goto(void);\n--\nkernel/bpf/core.c=3469=bool __weak bpf_jit_supports_private_stack(void)\n--\nkernel/bpf/core.c-3481- */\nkernel/bpf/core.c:3482:bool __weak bpf_jit_supports_large_stack(void)\nkernel/bpf/core.c-3483-{\n"
} |
| 19/2 | 2026/09/26 01:42 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 74833 output: 22 thoughts: 14 Thoughts: Okay, here's my interpretation of that thought process, tailored for an expert audience, and written in the first person: **Scratching the Surface of `bpf_id_scratch_reserve`** Right, let's dive into this `bpf_id_scratch_reserve` function. The name itself suggests a function related to reserving scratch space within the BPF context, likely for intermediate data or temporary calculations. Given the "ID" component, it could be tied to specific BPF program instances or perhaps per-CPU scratch areas identified by an ID. My initial questions would revolve around the details of the reservation: Is it allocation-based, or is it more like a claim on pre-allocated memory? Does this function manage the lifecycle of the reserved memory - allocation, deallocation, bounds checking etc? What are the potential constraints on the size of this scratch area? Is this reservation per-program, per-CPU, or something else entirely? The error codes and return values are going to be critical to understand how the function is used correctly. I'll need to look at how it interacts with other related BPF infrastructure, such as the BPF map interfaces or perhaps the BPF verifier. The intent is to avoid common mistakes like memory corruption and out-of-bounds access. I’m particularly curious about how this ties into the overall BPF security model. Are there any built-in safeguards to prevent malicious programs from exhausting this scratch space and potentially leading to denial-of-service? Security implications are always front of mind, particularly with anything involving dynamic memory manipulation within the kernel. I anticipate this reservation strategy is designed to be lightweight and efficient, so the implications are going to be key to evaluating performance and resource utilization. |
| 20/2 | 2026/09/26 01:42 | tool | grepper |
0mArgs:{
"Expression": "bpf_id_scratch_reserve"
}Results: {
"Output": "include/linux/bpf_verifier.h=839=struct bpf_id_pair {\n--\ninclude/linux/bpf_verifier.h-845- * Scratch map from the ids of one verifier state to those of another, also\ninclude/linux/bpf_verifier.h:846: * used as a stack of ids. Grown on demand by bpf_id_scratch_reserve().\ninclude/linux/bpf_verifier.h-847- */\n--\ninclude/linux/bpf_verifier.h=1219=void bpf_free_backedges(struct bpf_scc_visit *visit);\ninclude/linux/bpf_verifier.h:1220:bool bpf_id_scratch_reserve(void **arr, u32 *cap, u32 cnt, size_t elem_size);\ninclude/linux/bpf_verifier.h-1221-int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state *cur,\n--\nkernel/bpf/states.c=319=static bool check_ids(u32 old_id, u32 cur_id, struct bpf_idmap *idmap)\n--\nkernel/bpf/states.c-342-\t */\nkernel/bpf/states.c:343:\tif (!bpf_id_scratch_reserve((void **)\u0026idmap-\u003emap, \u0026idmap-\u003ecap, idmap-\u003ecnt, sizeof(*map)))\nkernel/bpf/states.c-344-\t\treturn false;\n--\nkernel/bpf/states.c=938=static bool func_states_equal(struct bpf_verifier_env *env, struct bpf_func_state *old,\n--\nkernel/bpf/states.c-969- */\nkernel/bpf/states.c:970:bool bpf_id_scratch_reserve(void **arr, u32 *cap, u32 cnt, size_t elem_size)\nkernel/bpf/states.c-971-{\n--\nkernel/bpf/verifier.c=10130=static int idstack_push(struct bpf_idmap *idmap, u32 id)\n--\nkernel/bpf/verifier.c-10140-\nkernel/bpf/verifier.c:10141:\tif (!bpf_id_scratch_reserve((void **)\u0026idmap-\u003emap, \u0026idmap-\u003ecap, idmap-\u003ecnt,\nkernel/bpf/verifier.c-10142-\t\t\t\t sizeof(*idmap-\u003emap)))\n--\nkernel/bpf/verifier.c=18435=static void idset_cnt_inc(struct bpf_idset *idset, u32 id)\n--\nkernel/bpf/verifier.c-18445-\t/* New id; one that cannot be recorded counts as shared and is kept */\nkernel/bpf/verifier.c:18446:\tif (!bpf_id_scratch_reserve((void **)\u0026idset-\u003eentries, \u0026idset-\u003ecap, idset-\u003enum_ids,\nkernel/bpf/verifier.c-18447-\t\t\t\t sizeof(*idset-\u003eentries)))\n"
} |
| 21/2 | 2026/09/26 01:42 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 76023 output: 32 thoughts: 1947 Thoughts: **Analysis of BPF Stack Limit and Verifier Modifications** Okay, let's break this down for the patch review. The core focus seems to be around stack limits in BPF programs, which is critical for security. I'm starting by tracing how `bpf_prog_stack_limit` is used. It's a static inline function, but its returned value impacts several key areas of the verifier, namely `mark_stack_read_all`, `check_max_stack_depth_subprog`, and several other stack access checks. So `bpf_prog_stack_limit` is foundational. Now I'm evaluating which functions are *directly* relevant to the logic being modified. `bpf_check` is the verifier entry point, but it's too ubiquitous. The guidance tells us to avoid listing general functions, and instead focus on what is specific. `check_stack_write_fixed_off`, `check_stack_read_fixed_off`, and `check_max_stack_depth_subprog`, etc., are much more specific to the stack checking. These are the functions I need to focus on. I'm now analyzing some of the specifics. `widen_frame_masks` in `liveness.c` looks like it's a good candidate, but it's 15 lines. Depending on inlining, it might or might not be a key thing to add. Now I'm looking at non-static functions and those that directly implement the new logic: `bpf_compute_subprog_arg_access` (liveness.c), `bpf_id_scratch_reserve` (states.c), `bpf_push_jmp_history` (backtrack.c), `bpf_bt_sync_linked_regs`, `bpf_mark_chain_precision`, `stacksafe` (states.c) are all non-static and *directly* implement some new logic. `bpf_jit_supports_large_stack` is a global function. The `stacksafe` function seems to have been heavily modified. I should verify the inlining with the compiler. I should note the changes to the signatures, bodies, and use of stack slot information in those modified functions. I need to make sure I understand the modifications in `bpf_mark_chain_precision`, `check_max_stack_depth_subprog`, `check_stack_write_fixed_off`, `check_stack_read_fixed_off`, and `check_stack_slot_within_bounds` since those are the key stack access checks. Finally, I'll need to confirm the required configurations. It looks like `BPF_SYSCALL` and `BPF_JIT` are required, and the stack limit itself depends on JIT support, specifically `bpf_jit_supports_large_stack()` and `bpf_jit_supports_subprog_tailcalls()`. I need to ensure that the code correctly handles cases where JIT is enabled (e.g., via `CONFIG_BPF_JIT_ALWAYS_ON`) and when it's not and the related `bpf_jit_enable`. These configurations and their impact on stack limits are crucial. |
| 22/2 | 2026/09/26 01:42 | tool | grepper |
0mArgs:{
"Expression": "int bpf_jit_enable",
"PathPrefix": "kernel/bpf/"
}Results: {
"Output": "kernel/bpf/core.c=541=void bpf_prog_kallsyms_del_all(struct bpf_prog *fp)\n--\nkernel/bpf/core.c-548-/* All BPF JIT sysctl knobs here. */\nkernel/bpf/core.c:549:int bpf_jit_enable __read_mostly = IS_BUILTIN(CONFIG_BPF_JIT_DEFAULT_ON);\nkernel/bpf/core.c-550-int bpf_jit_kallsyms __read_mostly = IS_BUILTIN(CONFIG_BPF_JIT_DEFAULT_ON);\n"
} |
| 23/2 | 2026/09/26 01:42 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 78865 output: 23 thoughts: 61 Thoughts: Okay, here's my take on those thoughts, framed as my own reasoning: **Checking BPF JIT Configuration** Right, so I see that either `CONFIG_BPF_JIT_ALWAYS_ON` or `CONFIG_BPF_JIT_DEFAULT_ON` are going to set `bpf_jit_enable = 1`. That's straightforward enough. Now, I need to understand the nuances of this configuration. Let's delve into the `Kconfig` for `CONFIG_BPF_JIT_DEFAULT_ON`. I'm going to look at what conditions trigger it, and what its implications are. Is it enabled by default on specific architectures or configurations? Are there any dependencies I need to be aware of? This is key to properly understanding the default behavior and any potential runtime consequences. |
| 24/2 | 2026/09/26 01:42 | tool | grepper |
0mArgs:{
"Expression": "BPF_JIT_DEFAULT_ON"
}Results: {
"Output": "Documentation/scheduler/sched-ext.rst=35=options should be enabled to use sched_ext:\n--\nDocumentation/scheduler/sched-ext.rst-44- CONFIG_BPF_JIT_ALWAYS_ON=y\nDocumentation/scheduler/sched-ext.rst:45: CONFIG_BPF_JIT_DEFAULT_ON=y\nDocumentation/scheduler/sched-ext.rst-46-\n--\nkernel/bpf/Kconfig=62=config BPF_JIT_ALWAYS_ON\n--\nkernel/bpf/Kconfig-72-\nkernel/bpf/Kconfig:73:config BPF_JIT_DEFAULT_ON\nkernel/bpf/Kconfig-74-\tdef_bool ARCH_WANT_DEFAULT_BPF_JIT || BPF_JIT_ALWAYS_ON\n--\nkernel/bpf/core.c=541=void bpf_prog_kallsyms_del_all(struct bpf_prog *fp)\n--\nkernel/bpf/core.c-548-/* All BPF JIT sysctl knobs here. */\nkernel/bpf/core.c:549:int bpf_jit_enable __read_mostly = IS_BUILTIN(CONFIG_BPF_JIT_DEFAULT_ON);\nkernel/bpf/core.c:550:int bpf_jit_kallsyms __read_mostly = IS_BUILTIN(CONFIG_BPF_JIT_DEFAULT_ON);\nkernel/bpf/core.c-551-int bpf_jit_harden __read_mostly;\n--\ntools/sched_ext/Kconfig=7=CONFIG_BPF_JIT_ALWAYS_ON=y\ntools/sched_ext/Kconfig:8:CONFIG_BPF_JIT_DEFAULT_ON=y\ntools/sched_ext/Kconfig-9-CONFIG_SCHED_CLASS_EXT=y\n--\ntools/sched_ext/README.md=61=CONFIG_BPF_JIT_ALWAYS_ON=y\ntools/sched_ext/README.md:62:CONFIG_BPF_JIT_DEFAULT_ON=y\ntools/sched_ext/README.md-63-```\n--\ntools/testing/selftests/bpf/config.aarch64=14=CONFIG_BPF_JIT_ALWAYS_ON=y\ntools/testing/selftests/bpf/config.aarch64:15:CONFIG_BPF_JIT_DEFAULT_ON=y\ntools/testing/selftests/bpf/config.aarch64-16-CONFIG_BPF_PRELOAD_UMD=y\n--\ntools/testing/selftests/bpf/config.s390x=8=CONFIG_BPF_JIT_ALWAYS_ON=y\ntools/testing/selftests/bpf/config.s390x:9:CONFIG_BPF_JIT_DEFAULT_ON=y\ntools/testing/selftests/bpf/config.s390x-10-CONFIG_BPF_PRELOAD=y\n"
} |
| 25/2 | 2026/09/26 01:42 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 79871 output: 26 thoughts: 23 Thoughts: Here's my take on those thoughts: **X86 and the JIT: Checking the Status** Alright, let's dive into this. My primary concern right now is the `ARCH_WANT_DEFAULT_BPF_JIT` configuration option, specifically how it's handled on x86 architectures. I need to quickly ascertain the current state and implications there. I'm likely looking for any recent commits, discussions, or documentation that touch on how the BPF JIT is behaving, whether it's enabled by default, and if any changes or regressions have cropped up recently. Basically, I need to understand if the x86 implementation is meeting the current performance and security requirements in relation to the default JIT behavior, or if there is something specific to investigate. I'm running through the mental checklist: possible optimization opportunities, potential security vulnerabilities, any potential regressions since the last kernel version. Need to have a clear picture. |
| 26/2 | 2026/09/26 01:42 | tool | grepper |
0mArgs:{
"Expression": "ARCH_WANT_DEFAULT_BPF_JIT"
}Results: {
"Output": "arch/arm64/Kconfig=2=config ARM64\n--\narch/arm64/Kconfig-91-\tselect ARCH_WANT_COMPAT_IPC_PARSE_VERSION if COMPAT\narch/arm64/Kconfig:92:\tselect ARCH_WANT_DEFAULT_BPF_JIT\narch/arm64/Kconfig-93-\tselect ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT\n--\narch/loongarch/Kconfig=2=config LOONGARCH\n--\narch/loongarch/Kconfig-80-\tselect ARCH_USE_QUEUED_SPINLOCKS\narch/loongarch/Kconfig:81:\tselect ARCH_WANT_DEFAULT_BPF_JIT if HAVE_EBPF_JIT\narch/loongarch/Kconfig-82-\tselect ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT\n--\narch/parisc/Kconfig=2=config PARISC\n--\narch/parisc/Kconfig-73-\tselect HAVE_EBPF_JIT\narch/parisc/Kconfig:74:\tselect ARCH_WANT_DEFAULT_BPF_JIT\narch/parisc/Kconfig-75-\tselect HAVE_REGS_AND_STACK_ACCESS_API\n--\narch/powerpc/Kconfig=118=config PPC\n--\narch/powerpc/Kconfig-182-\tselect ARCH_USE_QUEUED_RWLOCKS\t\tif PPC_QUEUED_SPINLOCKS\narch/powerpc/Kconfig:183:\tselect ARCH_WANT_DEFAULT_BPF_JIT\narch/powerpc/Kconfig-184-\tselect ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT\n--\narch/s390/Kconfig=78=config S390\n--\narch/s390/Kconfig-164-\tselect ARCH_WANTS_NO_INSTR\narch/s390/Kconfig:165:\tselect ARCH_WANT_DEFAULT_BPF_JIT\narch/s390/Kconfig-166-\tselect ARCH_WANT_IPC_PARSE_VERSION\n--\narch/x86/Kconfig=58=config X86\n--\narch/x86/Kconfig-143-\tselect ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH\narch/x86/Kconfig:144:\tselect ARCH_WANT_DEFAULT_BPF_JIT\tif X86_64\narch/x86/Kconfig-145-\tselect ARCH_WANTS_CLOCKSOURCE_READ_INLINE\tif X86_64\n--\nkernel/bpf/Kconfig=21=config HAVE_EBPF_JIT_KASAN\n--\nkernel/bpf/Kconfig-25-# default for kernels that were compiled with BPF JIT support.\nkernel/bpf/Kconfig:26:config ARCH_WANT_DEFAULT_BPF_JIT\nkernel/bpf/Kconfig-27-\tbool\n--\nkernel/bpf/Kconfig=73=config BPF_JIT_DEFAULT_ON\nkernel/bpf/Kconfig:74:\tdef_bool ARCH_WANT_DEFAULT_BPF_JIT || BPF_JIT_ALWAYS_ON\nkernel/bpf/Kconfig-75-\tdepends on HAVE_EBPF_JIT \u0026\u0026 BPF_JIT\n--\ntools/testing/selftests/bpf/config.aarch64=1=CONFIG_ARCH_VEXPRESS=y\ntools/testing/selftests/bpf/config.aarch64:2:CONFIG_ARCH_WANT_DEFAULT_BPF_JIT=y\ntools/testing/selftests/bpf/config.aarch64-3-CONFIG_ARM_SMMU_V3=y\n--\ntools/testing/selftests/bpf/config.s390x:1:CONFIG_ARCH_WANT_DEFAULT_BPF_JIT=y\ntools/testing/selftests/bpf/config.s390x-2-CONFIG_AUDIT=y\n"
} |
| 27/2 | 2026/09/26 01:42 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 81132 output: 321 thoughts: 923 Thoughts: **BPF Stack Expansion and Verification Analysis** Okay, so looking at this, `ARCH_WANT_DEFAULT_BPF_JIT` on x86_64 means `BPF_JIT_DEFAULT_ON` is automatically enabled when `BPF_JIT` is enabled. That simplifies things a bit, so `BPF_SYSCALL` and `BPF_JIT` are the key configs needed. Let's make sure that's clear. Now, let's drill down into these code changes. I need to understand the scope. I see modifications across `kernel/bpf/backtrack.c`, `kernel/bpf/states.c`, `kernel/bpf/liveness.c`, `kernel/bpf/verifier.c`, and even `arch/x86/net/bpf_jit_comp.c`. The symbols they're introducing or changing are: `bpf_push_jmp_history`, `bpf_mark_chain_precision`, `bpf_mark_all_scalars_precise`, `backtrack_insn`, `bpf_fmt_stack_mask`, `bpf_id_scratch_reserve`, `stacksafe`, `propagate_precision`, `bpf_is_state_visited`, `bpf_compute_subprog_arg_access`, `widen_frame_masks`, `mark_stack_range`, `merge_instances`, `bpf_bt_sync_linked_regs`, `check_stack_write_fixed_off`, `check_stack_read_fixed_off`, `check_max_stack_depth_subprog`, `check_stack_slot_within_bounds`, and finally `bpf_jit_supports_large_stack`. Yep, these are definitely focused and not ubiquitous. They all relate directly to BPF stack verification, jump history, stack backtracking, dynamic ID map/set allocation, and liveness tracking. This all seems tightly coupled with the BPF stack changes. This is a pretty significant change. The big picture is that this patch series is increasing the BPF stack from 512 bytes to 2048 bytes (MAX_BPF_STACK_JIT) on architectures like x86_64. That necessitates *substantial* changes to core BPF verifier logic: adapting data structures, jump history handling, and backtracking to support the increased stack size. It looks like it's growing from 64 to 256 stack slots. Additionally, they are using dynamically sized allocations for scratch buffers (replacing fixed-size buffers) and refactoring stack liveness tracking to use variable width masks. Bounds checks and stack depth enforcement have also been modified. This code will be executed within the kernel's BPF verifier when BPF programs are loaded via `sys_bpf(BPF_PROG_LOAD, ...)`. Therefore, fuzzing is absolutely crucial. I need to look for any memory corruption, verifier bugs, any unhandled bounds or overflows, or logic flaws in precision tracking and liveness analysis. This is definitely worth fuzzing. I will flag it as `WorthFuzzing: true`. The required configs are `BPF_SYSCALL` and `BPF_JIT`. The focus symbols are all listed. Alright, I think I have a solid understanding of the implications here. Let's call `set-results`. |
| 28/2 | 2026/09/26 01:43 | tool | set-results |
0mArgs:{
"EnableConfigs": [
"BPF_SYSCALL",
"BPF_JIT"
],
"FocusSymbols": [
"bpf_jit_supports_large_stack",
"bpf_id_scratch_reserve",
"bpf_push_jmp_history",
"bpf_bt_sync_linked_regs",
"bpf_mark_chain_precision",
"bpf_compute_subprog_arg_access",
"stacksafe",
"check_max_stack_depth_subprog",
"check_stack_slot_within_bounds",
"check_stack_write_fixed_off",
"check_stack_read_fixed_off",
"widen_frame_masks"
],
"Reasoning": "The patch series expands BPF stack capacity from 512 bytes to 2048 bytes (MAX_BPF_STACK_JIT) on architectures with JIT support (x86_64, arm64). To support this, it modifies core BPF verifier subsystems: jump history linked register bit packing, stack backtracking with enlarged bitmaps, dynamically grown ID scratch maps/sets, dynamically sized per-frame liveness masks, and stack bounds verification. All of this logic is directly reachable via the bpf(BPF_PROG_LOAD) syscall and warrants fuzzing for potential verifier correctness issues, precision tracking regressions, and memory safety flaws.",
"WorthFuzzing": true
}Results: {
"EnableConfigs": [
"BPF_SYSCALL",
"BPF_JIT"
],
"FocusSymbols": [
"bpf_jit_supports_large_stack",
"bpf_id_scratch_reserve",
"bpf_push_jmp_history",
"bpf_bt_sync_linked_regs",
"bpf_mark_chain_precision",
"bpf_compute_subprog_arg_access",
"stacksafe",
"check_max_stack_depth_subprog",
"check_stack_slot_within_bounds",
"check_stack_write_fixed_off",
"check_stack_read_fixed_off",
"widen_frame_masks"
],
"Reasoning": "The patch series expands BPF stack capacity from 512 bytes to 2048 bytes (MAX_BPF_STACK_JIT) on architectures with JIT support (x86_64, arm64). To support this, it modifies core BPF verifier subsystems: jump history linked register bit packing, stack backtracking with enlarged bitmaps, dynamically grown ID scratch maps/sets, dynamically sized per-frame liveness masks, and stack bounds verification. All of this logic is directly reachable via the bpf(BPF_PROG_LOAD) syscall and warrants fuzzing for potential verifier correctness issues, precision tracking regressions, and memory safety flaws.",
"WorthFuzzing": true
} |
| 29/1 | 2026/09/26 01:43 | action | If |
1mArgs:{
"WorthFuzzing": true
}Results: null |
| 30/2 | 2026/09/26 01:43 | agent | kmsan-evaluator |
1mModel:core Args: null Results: {
"KMSANReasoning": "The patch increases the maximum stack depth budget for BPF programs from 512 bytes (MAX_BPF_STACK) to 2048 bytes (MAX_BPF_STACK_JIT) on architectures whose JIT supports large stacks (x86-64 and arm64). It updates the BPF verifier's stack tracking, liveness analysis, backtracking, and scratch data structures to handle up to 2 KiB frames.\n\nAnalysis of memory safety and detector applicability:\n1. No kernel structures or padding are copied or leaked to user space via copy_to_user or BPF buffers.\n2. All modified and newly allocated data structures are properly initialized:\n - Dynamic scratch buffers (bpf_idmap, bpf_idset) are managed with bpf_id_scratch_reserve() where written entries are initialized before use and freed on exit.\n - Dynamic liveness frame masks (struct frame_masks) are allocated using kvzalloc_flex() and widened with zero-padding intact.\n - Backtracking state (bt) and its stack masks are zero-initialized via bt_reset() (memset) and bitmap_zero().\n - The jump history entries (bpf_jmp_history_entry) properly track linked_regs_cnt and initialize all fields prior to read/unpacking.\n3. BPF verifier still enforces stack slot initialization semantics (tracking STACK_INVALID, STACK_MISC, STACK_ZERO, etc.) over the expanded stack range.\n4. Any potential issues stemming from this change would be stack overflows, verifier state tracking inconsistencies, out-of-bounds accesses during liveness tracking or backtracking, or memory corruption, which are covered by KASAN, stack guards, and other standard debug sanitizers.\n\nBecause no uninitialized memory risks or kernel-to-user info-leaks are introduced, a dedicated KMSAN fuzzing session is not warranted.",
"NeedsKMSAN": false
}Instruction: You are an expert Linux kernel security engineer specializing in kernel memory
error detectors (KASAN and KMSAN). Your job is to review the provided patch series and
determine if the code changes justify spawning a dedicated KMSAN (KernelMemorySanitizer)
fuzzing session in addition to standard KASAN fuzzing.
CRITICAL DISTINCTION BETWEEN KASAN AND KMSAN:
- Standard KASAN kernel builds (upstream-apparmor-kasan.config) already enable
a comprehensive suite of debugging tools and sanitizers, including KASAN
(out-of-bounds accesses, use-after-free, double free, invalid free), LOCKDEP
(locking bugs and deadlocks), UB-sanitizers, and memory corruption checks.
- KMSAN (KernelMemorySanitizer) detects reads of UNINITIALIZED memory (stack, heap,
or page allocations) and kernel-to-user memory info-leaks.
Rule: THERE IS NO SENSE IN RUNNING A KMSAN SESSION IF A BUG CAN BE CAUGHT BY KASAN,
LOCKDEP, OR OTHER STANDARD BUG DETECTORS.
A dedicated KMSAN fuzzing session incurs significant resource costs. You must ONLY
set NeedsKMSAN=true if the code changes introduce or expose UNINITIALIZED MEMORY risks
that are detected ONLY by KMSAN.
Look holistically at the patch series and surrounding code. Even if no direct
uninitialized field accesses or new buffer allocations are added in the diff itself,
a patch may alter control flow, bounds checking, or data length calculations in ways
that change how the rest of the code operates on existing buffers (e.g. allowing
uninitialized stack/heap memory to be read, copied to user space, or used in control
flow). Do not hesitate to use your code access tools to inspect the surrounding code,
called functions, and callers.
Set NeedsKMSAN=true ONLY IF the patch introduces or modifies:
1. Kernel structures sent to user space (via copy_to_user, put_user, netlink skb
attributes, ioctl output arguments, socket options, or BPF buffers) where fields
or structure padding might not be fully initialized/zeroed.
2. Conditional logic or branching that depends on potentially uninitialized variables
or struct fields.
3. Allocation or initialization of complex data structures where uninitialized fields
could be read later in reachable code paths.
4. Bounds checks, lengths, or logic in a way that allows surrounding code to access
uninitialized bytes of existing buffers.
Set NeedsKMSAN=false IF:
- The code changes primarily risk out-of-bounds access, array overflows, NULL pointer
dereferences, locking deadlocks, or use-after-free bugs (these are already caught
by KASAN, LOCKDEP, or standard bug detectors).
- All stack/heap structures touched or introduced by the patch are fully zeroed
or initialized (e.g. using = {0}, memset, kzalloc) before being read or copied.
- The patch does not introduce any risk of uninitialized memory usage or info-leaks.
Use your code access tools to inspect the surrounding code if necessary, then provide
detailed KMSANReasoning contrasting KASAN vs KMSAN applicability for this patch.
Prefer calling several tools at the same time to save round-trips.
Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.
Prompt: Target architecture: amd64
For your convenience, here is the diff of the changes:
commit 5a159253784d7a46f42cd32160c732f134182d57
Author: syz-cluster <triage@syzkaller.com>
Date: Sat Sep 26 01:41:58 2026 +0000
syz-cluster: applied patch under review
diff --git a/Documentation/bpf/bpf_design_QA.rst b/Documentation/bpf/bpf_design_QA.rst
index eb19c945f4d57..f4e4a7f4f9fdb 100644
--- a/Documentation/bpf/bpf_design_QA.rst
+++ b/Documentation/bpf/bpf_design_QA.rst
@@ -221,9 +221,13 @@ newer kernels. BPF programs need to change accordingly when this happens.
Q: How much stack space a BPF program uses?
-------------------------------------------
-A: Currently all program types are limited to 512 bytes of stack
-space, but the verifier computes the actual amount of stack used
-and both interpreter and most JITed code consume necessary amount.
+A: A program may use up to 2 KiB of stack, combined over its call
+chain, when the JIT of the architecture reports support for large
+stacks (currently x86-64 and arm64); a single function may use all of
+it, and every frame of a program running on a private stack gets the
+whole amount. Elsewhere, and whenever the interpreter is used, the
+limit is 512 bytes. The verifier computes the actual amount of stack
+used and both interpreter and most JITed code consume necessary amount.
Q: Can BPF be offloaded to HW?
------------------------------
diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
index 6c04fee468766..d92d7754578ab 100644
--- a/arch/arm64/net/bpf_jit_comp.c
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -2485,6 +2485,11 @@ bool bpf_jit_supports_subprog_tailcalls(void)
return true;
}
+bool bpf_jit_supports_large_stack(void)
+{
+ return true;
+}
+
static void invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_node *node,
int bargs_off, int retval_off, int run_ctx_off,
bool save_ret)
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index d4a980140b48d..d6998c909754f 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -4452,6 +4452,18 @@ bool bpf_jit_supports_subprog_tailcalls(void)
return true;
}
+/*
+ * Frame sizes are 32-bit immediates in the prologue, epilogue and tail call
+ * sequences, a tail call pops the caller's frame and lands in the target's
+ * prologue before the target allocates its own, and private stacks are
+ * allocated from the program's own depth, so MAX_BPF_STACK_JIT frames need
+ * nothing special.
+ */
+bool bpf_jit_supports_large_stack(void)
+{
+ return true;
+}
+
bool bpf_jit_supports_percpu_insn(void)
{
return true;
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 92f528c456052..a5d493b3876fc 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -19,11 +19,12 @@
* that converting umax_value to int cannot overflow.
*/
#define BPF_MAX_VAR_SIZ (1 << 29)
-/* size of tmp_str_buf in bpf_verifier.
- * we need at least 306 bytes to fit full stack mask representation
- * (in the "-8,-16,...,-512" form)
+/*
+ * size of tmp_str_buf in bpf_verifier.
+ * we need at least 1399 bytes to fit full stack mask representation
+ * (in the "-8,-16,...,-2048" form)
*/
-#define TMP_STR_BUF_LEN 320
+#define TMP_STR_BUF_LEN 1408
/* Patch buffer size */
#define INSN_BUF_SIZE 32
@@ -242,55 +243,18 @@ enum bpf_stack_slot_type {
#define BPF_REG_SIZE 8 /* size of eBPF register in bytes */
+/*
+ * Largest number of BPF_REG_SIZE stack slots a single frame can have, sized
+ * for the largest stack budget any JIT supports. A frame may use any part of
+ * its program's budget; check_max_stack_depth() enforces the budget on the
+ * combined depth of frames sharing the kernel stack and on each frame using
+ * a private stack.
+ */
+#define MAX_BPF_STACK_SLOTS (MAX_BPF_STACK_JIT / BPF_REG_SIZE)
+
/* 4-byte stack slot granularity for liveness analysis */
#define BPF_HALF_REG_SIZE 4
#define STACK_SLOT_SZ 4
-#define STACK_SLOTS (MAX_BPF_STACK / BPF_HALF_REG_SIZE) /* 128 */
-
-typedef struct {
- u64 v[2];
-} spis_t;
-
-#define SPIS_ZERO ((spis_t){})
-#define SPIS_ALL ((spis_t){{ U64_MAX, U64_MAX }})
-
-static inline bool spis_is_zero(spis_t s)
-{
- return s.v[0] == 0 && s.v[1] == 0;
-}
-
-static inline bool spis_equal(spis_t a, spis_t b)
-{
- return a.v[0] == b.v[0] && a.v[1] == b.v[1];
-}
-
-static inline spis_t spis_or(spis_t a, spis_t b)
-{
- return (spis_t){{ a.v[0] | b.v[0], a.v[1] | b.v[1] }};
-}
-
-static inline spis_t spis_and(spis_t a, spis_t b)
-{
- return (spis_t){{ a.v[0] & b.v[0], a.v[1] & b.v[1] }};
-}
-
-static inline spis_t spis_not(spis_t s)
-{
- return (spis_t){{ ~s.v[0], ~s.v[1] }};
-}
-
-static inline bool spis_test_bit(spis_t s, u32 slot)
-{
- return s.v[slot / 64] & BIT_ULL(slot % 64);
-}
-
-static inline void spis_or_range(spis_t *mask, u32 lo, u32 hi)
-{
- u32 w;
-
- for (w = lo; w <= hi && w < STACK_SLOTS; w++)
- mask->v[w / 64] |= BIT_ULL(w % 64);
-}
#define BPF_REGMASK_ARGS ((1 << BPF_REG_1) | (1 << BPF_REG_2) | \
(1 << BPF_REG_3) | (1 << BPF_REG_4) | \
@@ -420,30 +384,33 @@ enum {
INSN_F_STACK_ARG_ACCESS = BIT(3),
};
+/* Registers linked to one jump condition that a history entry can record */
+#define BPF_LINKED_REGS_MAX 5
+
struct bpf_jmp_history_entry {
/* insn idx can't be bigger than 1 million */
u32 idx : 20;
u32 frame : 4; /* stack access frame number */
- u32 spi : 6; /* stack slot index (0..63) */
- u32 : 2;
- u32 prev_idx : 20;
/* special INSN_F_xxx flags */
u32 flags : 4;
- u32 : 8;
+ u32 : 4;
+ u32 prev_idx : 20;
+ u32 spi : 12; /* stack slot index */
/*
- * additional registers that need precision tracking when this
- * jump is backtracked, vector of five 11-bit records
+ * Scalar registers and spilled scalars linked to the condition of
+ * this jump, which need precision tracking together when the jump is
+ * backtracked. Each is packed as 4 bits of frame number, one bit
+ * telling a register from a stack slot and 11 bits of register or
+ * slot index, see linked_regs_pack().
*/
- u64 linked_regs;
+ u16 linked_regs[BPF_LINKED_REGS_MAX];
+ u8 linked_regs_cnt;
};
static_assert(MAX_CALL_FRAMES <= (1 << 4));
-static_assert(MAX_BPF_STACK / 8 <= (1 << 6));
+static_assert(MAX_BPF_STACK_SLOTS <= (1 << 12));
-/* Maximum number of bpf_reg_state objects that can exist at once */
#define MAX_STACK_ARG_SLOTS (MAX_BPF_FUNC_ARGS - MAX_BPF_FUNC_REG_ARGS)
-#define BPF_ID_MAP_SIZE ((MAX_BPF_REG + MAX_BPF_STACK / BPF_REG_SIZE + \
- MAX_STACK_ARG_SLOTS) * MAX_CALL_FRAMES)
struct bpf_verifier_state {
/* call stack tracking */
struct bpf_func_state *frame[MAX_CALL_FRAMES];
@@ -529,12 +496,27 @@ struct bpf_verifier_state {
u32 may_goto_depth;
};
+/* Number of BPF_REG_SIZE stack slots tracked for the frame so far. */
+static inline u32 bpf_stack_nr_slots(const struct bpf_func_state *frame)
+{
+ return frame->allocated_stack / BPF_REG_SIZE;
+}
+
+/*
+ * Stack slot @spi of @frame, covering bytes [fp - (spi + 1) * 8, fp - spi * 8).
+ * The caller must ensure spi < bpf_stack_nr_slots(frame), see grow_stack_state().
+ */
+static inline struct bpf_stack_state *bpf_stack_slot(const struct bpf_func_state *frame, u32 spi)
+{
+ return &frame->stack[spi];
+}
+
static inline struct bpf_reg_state *
bpf_get_spilled_reg(int slot, struct bpf_func_state *frame, u32 mask)
{
- if (slot < frame->allocated_stack / BPF_REG_SIZE &&
- (1 << frame->stack[slot].slot_type[BPF_REG_SIZE - 1]) & mask)
- return &frame->stack[slot].spilled_ptr;
+ if (slot < bpf_stack_nr_slots(frame) &&
+ (1 << bpf_stack_slot(frame, slot)->slot_type[BPF_REG_SIZE - 1]) & mask)
+ return &bpf_stack_slot(frame, slot)->spilled_ptr;
return NULL;
}
@@ -550,7 +532,7 @@ bpf_get_spilled_stack_arg(int slot, struct bpf_func_state *frame)
/* Iterate over 'frame', setting 'reg' to either NULL or a spilled register. */
#define bpf_for_each_spilled_reg(iter, frame, reg, mask) \
for (iter = 0, reg = bpf_get_spilled_reg(iter, frame, mask); \
- iter < frame->allocated_stack / BPF_REG_SIZE; \
+ iter < bpf_stack_nr_slots(frame); \
iter++, reg = bpf_get_spilled_reg(iter, frame, mask))
/* Iterate over 'frame', setting 'reg' to either NULL or a spilled stack arg. */
@@ -575,7 +557,7 @@ bpf_get_spilled_stack_arg(int slot, struct bpf_func_state *frame)
bpf_for_each_spilled_reg(___j, __state, __reg, __mask) { \
if (!__reg) \
continue; \
- __stack = &__state->stack[___j]; \
+ __stack = bpf_stack_slot(__state, ___j); \
(void)(__expr); \
} \
__stack = NULL; \
@@ -850,7 +832,7 @@ struct backtrack_state {
struct bpf_verifier_env *env;
u32 frame;
u32 reg_masks[MAX_CALL_FRAMES];
- u64 stack_masks[MAX_CALL_FRAMES];
+ unsigned long stack_masks[MAX_CALL_FRAMES][BITS_TO_LONGS(MAX_BPF_STACK_SLOTS)];
u8 stack_arg_masks[MAX_CALL_FRAMES];
};
@@ -859,18 +841,27 @@ struct bpf_id_pair {
u32 cur;
};
+/*
+ * Scratch map from the ids of one verifier state to those of another, also
+ * used as a stack of ids. Grown on demand by bpf_id_scratch_reserve().
+ */
struct bpf_idmap {
u32 tmp_id_gen;
u32 cnt;
- struct bpf_id_pair map[BPF_ID_MAP_SIZE];
+ u32 cap;
+ struct bpf_id_pair *map;
+};
+
+struct bpf_idset_entry {
+ u32 id;
+ u32 cnt;
};
+/* Scratch set of ids with a use count each, grown on demand */
struct bpf_idset {
u32 num_ids;
- struct {
- u32 id;
- u32 cnt;
- } entries[BPF_ID_MAP_SIZE];
+ u32 cap;
+ struct bpf_idset_entry *entries;
};
/* see verifier.c:compute_scc_callchain() */
@@ -965,10 +956,8 @@ struct bpf_verifier_env {
struct bpf_subprog_info subprog_info[BPF_MAX_SUBPROGS + 2]; /* max + 2 for the fake and exception subprogs */
/* subprog indices sorted in topological order: leaves first, callers last */
int subprog_topo_order[BPF_MAX_SUBPROGS + 2];
- union {
- struct bpf_idmap idmap_scratch;
- struct bpf_idset idset_scratch;
- };
+ struct bpf_idmap idmap_scratch;
+ struct bpf_idset idset_scratch;
struct {
int *insn_state;
int *insn_stack;
@@ -993,6 +982,8 @@ struct bpf_verifier_env {
u32 prev_jmps_processed, jmps_processed;
/* maximum combined stack depth */
u32 max_stack_depth;
+ /* stack budget of the program, see bpf_prog_stack_limit() */
+ u32 stack_limit;
/* total verification time */
u64 verification_time;
/* maximum number of verifier states kept in 'branching' instructions */
@@ -1028,7 +1019,7 @@ struct bpf_verifier_env {
*/
u32 scratched_regs;
/* Same as scratched_regs but for stack slots */
- u64 scratched_stack_slots;
+ DECLARE_BITMAP(scratched_stack_slots, MAX_BPF_STACK_SLOTS);
u64 prev_log_pos, prev_insn_print_pos;
/* buffer used to temporary hold constants as scalar registers */
struct bpf_reg_state fake_reg[1];
@@ -1226,8 +1217,10 @@ int bpf_copy_verifier_state(struct bpf_verifier_state *dst_state,
struct list_head *bpf_explored_state(struct bpf_verifier_env *env, int idx);
void bpf_free_verifier_state(struct bpf_verifier_state *state, bool free_self);
void bpf_free_backedges(struct bpf_scc_visit *visit);
+bool bpf_id_scratch_reserve(void **arr, u32 *cap, u32 cnt, size_t elem_size);
int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state *cur,
- int insn_flags, int spi, int frame, u64 linked_regs);
+ int insn_flags, int spi, int frame, const u16 *linked_regs,
+ u8 linked_regs_cnt);
void bpf_bt_sync_linked_regs(struct backtrack_state *bt, struct bpf_jmp_history_entry *hist);
void bpf_mark_reg_not_init(const struct bpf_verifier_env *env,
struct bpf_reg_state *reg);
@@ -1244,6 +1237,24 @@ static inline int bpf_get_spi(s32 off)
return (-off - 1) / BPF_REG_SIZE;
}
+/*
+ * Stack a program may use in total: combined over the frames of a call
+ * chain on the kernel stack, or per frame on a private stack. Any single
+ * frame may reach that deep. Only a JIT that lays out such frames may go
+ * beyond MAX_BPF_STACK, the interpreter's frame size, and only one whose
+ * tail calls let the target set up its own frame: without subprogram
+ * tail calls, do_misc_fixups() gives every program with tail calls a
+ * MAX_BPF_STACK frame, which a deeper frame would overrun.
+ */
+static inline u32 bpf_prog_stack_limit(const struct bpf_prog *prog)
+{
+ /* an offloaded program never runs on the host JIT, whatever it supports */
+ if (prog->jit_requested && !bpf_prog_is_offloaded(prog->aux) &&
+ bpf_jit_supports_large_stack() && bpf_jit_supports_subprog_tailcalls())
+ return MAX_BPF_STACK_JIT;
+ return MAX_BPF_STACK;
+}
+
static inline struct bpf_func_state *bpf_func(struct bpf_verifier_env *env,
const struct bpf_reg_state *reg)
{
@@ -1287,12 +1298,7 @@ static inline void bpf_bt_set_frame_reg(struct backtrack_state *bt, u32 frame, u
static inline void bpf_bt_set_frame_slot(struct backtrack_state *bt, u32 frame, u32 slot)
{
- bt->stack_masks[frame] |= 1ull << slot;
-}
-
-static inline void bpf_bt_set_frame_slot_mask(struct backtrack_state *bt, u32 frame, u64 mask)
-{
- bt->stack_masks[frame] |= mask;
+ __set_bit(slot, bt->stack_masks[frame]);
}
static inline void bt_set_frame_stack_arg_slot(struct backtrack_state *bt, u32 frame, u32 slot)
@@ -1307,7 +1313,7 @@ static inline bool bt_is_frame_reg_set(struct backtrack_state *bt, u32 frame, u3
static inline bool bt_is_frame_slot_set(struct backtrack_state *bt, u32 frame, u32 slot)
{
- return bt->stack_masks[frame] & (1ull << slot);
+ return test_bit(slot, bt->stack_masks[frame]);
}
bool bpf_map_is_rdonly(const struct bpf_map *map);
@@ -1438,7 +1444,7 @@ static inline void mark_reg_scratched(struct bpf_verifier_env *env, u32 regno)
static inline void mark_stack_slot_scratched(struct bpf_verifier_env *env, u32 spi)
{
- env->scratched_stack_slots |= 1ULL << spi;
+ __set_bit(spi, env->scratched_stack_slots);
}
static inline bool reg_scratched(const struct bpf_verifier_env *env, u32 regno)
@@ -1446,27 +1452,28 @@ static inline bool reg_scratched(const struct bpf_verifier_env *env, u32 regno)
return (env->scratched_regs >> regno) & 1;
}
-static inline bool stack_slot_scratched(const struct bpf_verifier_env *env, u64 regno)
+static inline bool stack_slot_scratched(const struct bpf_verifier_env *env, u32 spi)
{
- return (env->scratched_stack_slots >> regno) & 1;
+ return test_bit(spi, env->scratched_stack_slots);
}
static inline bool verifier_state_scratched(const struct bpf_verifier_env *env)
{
- return env->scratched_regs || env->scratched_stack_slots;
+ return env->scratched_regs ||
+ !bitmap_empty(env->scratched_stack_slots, MAX_BPF_STACK_SLOTS);
}
static inline void mark_verifier_state_clean(struct bpf_verifier_env *env)
{
env->scratched_regs = 0U;
- env->scratched_stack_slots = 0ULL;
+ bitmap_zero(env->scratched_stack_slots, MAX_BPF_STACK_SLOTS);
}
/* Used for printing the entire verifier state. */
static inline void mark_verifier_state_scratched(struct bpf_verifier_env *env)
{
env->scratched_regs = ~0U;
- env->scratched_stack_slots = ~0ULL;
+ bitmap_fill(env->scratched_stack_slots, MAX_BPF_STACK_SLOTS);
}
static inline bool bpf_stack_narrow_access_ok(int off, int fill_size, int spill_size)
@@ -1501,7 +1508,7 @@ struct bpf_subprog_info *bpf_find_containing_subprog(struct bpf_verifier_env *en
const char *bpf_subprog_name(const struct bpf_verifier_env *env, int subprog);
int bpf_jmp_offset(struct bpf_insn *insn);
struct bpf_iarray *bpf_insn_successors(struct bpf_verifier_env *env, u32 idx);
-void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask);
+void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, const unsigned long *stack_mask);
bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog);
/* Kinds of member a by-value struct or union may be composed of. */
diff --git a/include/linux/filter.h b/include/linux/filter.h
index 422284b4fa96f..cdd16bdd4dfb6 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -98,6 +98,11 @@ struct ctl_table_header;
/* BPF program can access up to 512 bytes of stack space. */
#define MAX_BPF_STACK 512
+/*
+ * Stack budget of a program on a JIT that lays out frames of that size.
+ * The interpreter and JITs without such support keep MAX_BPF_STACK.
+ */
+#define MAX_BPF_STACK_JIT 2048
/* Helper macros for filter block array initializers. */
@@ -1246,6 +1251,7 @@ bool bpf_jit_supports_ptr_xchg(void);
bool bpf_jit_supports_arena(void);
bool bpf_jit_supports_insn(struct bpf_insn *insn, bool in_arena);
bool bpf_jit_supports_private_stack(void);
+bool bpf_jit_supports_large_stack(void);
bool bpf_jit_supports_timed_may_goto(void);
bool bpf_jit_supports_fsession(void);
diff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c
index 507a366dffa47..db1be14d0a680 100644
--- a/kernel/bpf/backtrack.c
+++ b/kernel/bpf/backtrack.c
@@ -9,7 +9,8 @@
/* for any branch, call, exit record the history of jmps in the given state */
int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state *cur,
- int insn_flags, int spi, int frame, u64 linked_regs)
+ int insn_flags, int spi, int frame, const u16 *linked_regs,
+ u8 linked_regs_cnt)
{
u32 cnt = cur->jmp_history_cnt;
struct bpf_jmp_history_entry *p;
@@ -27,10 +28,13 @@ int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state
env->cur_hist_ent->flags |= insn_flags;
env->cur_hist_ent->spi = spi;
env->cur_hist_ent->frame = frame;
- verifier_bug_if(env->cur_hist_ent->linked_regs != 0, env,
- "insn history: insn_idx %d linked_regs: %#llx",
- env->insn_idx, env->cur_hist_ent->linked_regs);
- env->cur_hist_ent->linked_regs = linked_regs;
+ verifier_bug_if(env->cur_hist_ent->linked_regs_cnt != 0, env,
+ "insn history: insn_idx %d has %u linked regs",
+ env->insn_idx, env->cur_hist_ent->linked_regs_cnt);
+ if (linked_regs_cnt)
+ memcpy(env->cur_hist_ent->linked_regs, linked_regs,
+ linked_regs_cnt * sizeof(*linked_regs));
+ env->cur_hist_ent->linked_regs_cnt = linked_regs_cnt;
return 0;
}
@@ -47,7 +51,9 @@ int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state
p->flags = insn_flags;
p->spi = spi;
p->frame = frame;
- p->linked_regs = linked_regs;
+ if (linked_regs_cnt)
+ memcpy(p->linked_regs, linked_regs, linked_regs_cnt * sizeof(*linked_regs));
+ p->linked_regs_cnt = linked_regs_cnt;
cur->jmp_history_cnt = cnt;
env->cur_hist_ent = p;
@@ -123,13 +129,26 @@ static inline void bt_reset(struct backtrack_state *bt)
bt->env = env;
}
-static inline u32 bt_empty(struct backtrack_state *bt)
+static inline bool bt_frame_stack_empty(struct backtrack_state *bt, u32 frame)
{
- u64 mask = 0;
+ return bitmap_empty(bt->stack_masks[frame], MAX_BPF_STACK_SLOTS);
+}
+
+static inline bool bt_stack_empty(struct backtrack_state *bt)
+{
+ return bt_frame_stack_empty(bt, bt->frame);
+}
+
+static inline bool bt_empty(struct backtrack_state *bt)
+{
+ u32 mask = 0;
int i;
- for (i = 0; i <= bt->frame; i++)
- mask |= bt->reg_masks[i] | bt->stack_masks[i] | bt->stack_arg_masks[i];
+ for (i = 0; i <= bt->frame; i++) {
+ mask |= bt->reg_masks[i] | bt->stack_arg_masks[i];
+ if (!bt_frame_stack_empty(bt, i))
+ return false;
+ }
return mask == 0;
}
@@ -181,7 +200,7 @@ static inline void bt_clear_reg(struct backtrack_state *bt, u32 reg)
static inline void bt_clear_frame_slot(struct backtrack_state *bt, u32 frame, u32 slot)
{
- bt->stack_masks[frame] &= ~(1ull << slot);
+ __clear_bit(slot, bt->stack_masks[frame]);
}
static inline u32 bt_frame_reg_mask(struct backtrack_state *bt, u32 frame)
@@ -194,14 +213,14 @@ static inline u32 bt_reg_mask(struct backtrack_state *bt)
return bt->reg_masks[bt->frame];
}
-static inline u64 bt_frame_stack_mask(struct backtrack_state *bt, u32 frame)
+static inline unsigned long *bt_frame_stack_mask(struct backtrack_state *bt, u32 frame)
{
return bt->stack_masks[frame];
}
-static inline u64 bt_stack_mask(struct backtrack_state *bt)
+static inline unsigned long *bt_stack_mask(struct backtrack_state *bt)
{
- return bt->stack_masks[bt->frame];
+ return bt_frame_stack_mask(bt, bt->frame);
}
static inline u8 bt_stack_arg_mask(struct backtrack_state *bt)
@@ -233,17 +252,16 @@ static void fmt_reg_mask(char *buf, ssize_t buf_sz, u32 reg_mask)
break;
}
}
-/* format stack slots bitmask, e.g., "-8,-24,-40" for 0x15 mask */
-void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask)
+
+/* format stack slots bitmask, e.g., "-8,-24,-40" for slots 0, 2 and 4 */
+void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, const unsigned long *stack_mask)
{
- DECLARE_BITMAP(mask, 64);
bool first = true;
int i, n;
buf[0] = '\0';
- bitmap_from_u64(mask, stack_mask);
- for_each_set_bit(i, mask, 64) {
+ for_each_set_bit(i, stack_mask, MAX_BPF_STACK_SLOTS) {
n = snprintf(buf, buf_sz, "%s%d", first ? "" : ",", -(i + 1) * 8);
first = false;
buf += n;
@@ -452,10 +470,11 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
/* we are now tracking register spills correctly,
* so any instance of leftover slots is a bug
*/
- if (bt_stack_mask(bt) != 0) {
- verifier_bug(env,
- "static subprog leftover stack slots %llx",
- bt_stack_mask(bt));
+ if (!bt_stack_empty(bt)) {
+ bpf_fmt_stack_mask(env->tmp_str_buf, TMP_STR_BUF_LEN,
+ bt_stack_mask(bt));
+ verifier_bug(env, "static subprog leftover stack slots %s",
+ env->tmp_str_buf);
return -EFAULT;
}
/* propagate r1-r5 to the caller */
@@ -488,9 +507,11 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
bt_reg_mask(bt));
return -EFAULT;
}
- if (bt_stack_mask(bt) != 0) {
- verifier_bug(env, "callback leftover stack slots %llx",
- bt_stack_mask(bt));
+ if (!bt_stack_empty(bt)) {
+ bpf_fmt_stack_mask(env->tmp_str_buf, TMP_STR_BUF_LEN,
+ bt_stack_mask(bt));
+ verifier_bug(env, "callback leftover stack slots %s",
+ env->tmp_str_buf);
return -EFAULT;
}
/* clear r1-r5 in callback subprog's mask */
@@ -707,10 +728,10 @@ void bpf_mark_all_scalars_precise(struct bpf_verifier_env *env,
i, j);
}
}
- for (j = 0; j < func->allocated_stack / BPF_REG_SIZE; j++) {
- if (!bpf_is_spilled_reg(&func->stack[j]))
+ for (j = 0; j < bpf_stack_nr_slots(func); j++) {
+ if (!bpf_is_spilled_reg(bpf_stack_slot(func, j)))
continue;
- reg = &func->stack[j].spilled_ptr;
+ reg = &bpf_stack_slot(func, j)->spilled_ptr;
if (reg->type != SCALAR_VALUE || reg->precise)
continue;
reg->precise = true;
@@ -868,7 +889,7 @@ int bpf_mark_chain_precision(struct bpf_verifier_env *env,
if (st->curframe == 0 &&
st->frame[0]->subprogno > 0 &&
st->frame[0]->callsite == BPF_MAIN_FUNC &&
- bt_stack_mask(bt) == 0 &&
+ bt_stack_empty(bt) &&
(bt_reg_mask(bt) & ~BPF_REGMASK_ARGS) == 0) {
bitmap_from_u64(mask, bt_reg_mask(bt));
for_each_set_bit(i, mask, 32) {
@@ -882,8 +903,9 @@ int bpf_mark_chain_precision(struct bpf_verifier_env *env,
return 0;
}
- verifier_bug(env, "backtracking func entry subprog %d reg_mask %x stack_mask %llx",
- st->frame[0]->subprogno, bt_reg_mask(bt), bt_stack_mask(bt));
+ bpf_fmt_stack_mask(env->tmp_str_buf, TMP_STR_BUF_LEN, bt_stack_mask(bt));
+ verifier_bug(env, "backtracking func entry subprog %d reg_mask %x stack_mask %s",
+ st->frame[0]->subprogno, bt_reg_mask(bt), env->tmp_str_buf);
return -EFAULT;
}
@@ -944,18 +966,17 @@ int bpf_mark_chain_precision(struct bpf_verifier_env *env,
}
}
- bitmap_from_u64(mask, bt_frame_stack_mask(bt, fr));
- for_each_set_bit(i, mask, 64) {
- if (verifier_bug_if(i >= func->allocated_stack / BPF_REG_SIZE,
+ for_each_set_bit(i, bt_frame_stack_mask(bt, fr), MAX_BPF_STACK_SLOTS) {
+ if (verifier_bug_if(i >= bpf_stack_nr_slots(func),
env, "stack slot %d, total slots %d",
- i, func->allocated_stack / BPF_REG_SIZE))
+ i, bpf_stack_nr_slots(func)))
return -EFAULT;
- if (!bpf_is_spilled_scalar_reg(&func->stack[i])) {
+ if (!bpf_is_spilled_scalar_reg(bpf_stack_slot(func, i))) {
bt_clear_frame_slot(bt, fr, i);
continue;
}
- reg = &func->stack[i].spilled_ptr;
+ reg = &bpf_stack_slot(func, i)->spilled_ptr;
if (reg->precise) {
bt_clear_frame_slot(bt, fr, i);
} else {
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 227211166dccf..fbb2d8a840ef3 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -3471,6 +3471,19 @@ bool __weak bpf_jit_supports_private_stack(void)
return false;
}
+/*
+ * Return TRUE if the JIT lays out frames of up to MAX_BPF_STACK_JIT bytes.
+ * Its prologue, epilogue and tail call sequences must encode such frame
+ * sizes and a private stack must be sized from the program's depth. The
+ * budget is only granted alongside bpf_jit_supports_subprog_tailcalls(),
+ * whose tail calls land before the target sets up its own frame; see
+ * bpf_prog_stack_limit().
+ */
+bool __weak bpf_jit_supports_large_stack(void)
+{
+ return false;
+}
+
void __weak arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, u64 bp), void *cookie)
{
}
diff --git a/kernel/bpf/diagnostics.c b/kernel/bpf/diagnostics.c
index 5ecfa86ed49f4..a8ed6130c1373 100644
--- a/kernel/bpf/diagnostics.c
+++ b/kernel/bpf/diagnostics.c
@@ -1600,9 +1600,9 @@ static struct bpf_reg_state *target_to_reg(struct bpf_verifier_env *env,
return NULL;
return &state->stack_arg_regs[target->stack_arg];
case BPF_DIAG_MOD_TARGET_STACK_SLOT:
- if (target->spi >= state->allocated_stack / BPF_REG_SIZE)
+ if (target->spi >= bpf_stack_nr_slots(state))
return NULL;
- return &state->stack[target->spi].spilled_ptr;
+ return &bpf_stack_slot(state, target->spi)->spilled_ptr;
default:
return NULL;
}
@@ -1618,7 +1618,7 @@ static bool reg_to_target(struct bpf_verifier_env *env, const struct bpf_reg_sta
for (frame = 0; frame <= vstate->curframe; frame++) {
struct bpf_func_state *state = vstate->frame[frame];
unsigned long start, end;
- u32 nslots = state->allocated_stack / BPF_REG_SIZE;
+ u32 nslots = bpf_stack_nr_slots(state);
int spi;
start = (unsigned long)state->regs;
diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index 44ecdc5b4ec2d..f9beef6695c44 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -10,10 +10,39 @@
#define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
-struct per_frame_masks {
- spis_t may_read; /* stack slots that may be read by this instruction */
- spis_t must_write; /* stack slots written by this instruction */
- spis_t live_before; /* stack slots that may be read by this insn and its successors */
+/*
+ * Stack liveness is tracked with a 4-byte (half register) granularity.
+ * Half-slot 0 covers [fp-4, fp), half-slot 1 covers [fp-8, fp-4), and so on,
+ * hence FRAME_HALF_SPIS - 1 is the deepest half-slot a frame can have.
+ */
+#define FRAME_HALF_SPIS (MAX_BPF_STACK_JIT / BPF_HALF_REG_SIZE)
+#define FRAME_MAX_WORDS BITS_TO_LONGS(FRAME_HALF_SPIS)
+
+/* Masks tracked for each instruction of a frame */
+enum {
+ FM_MAY_READ, /* stack slots that may be read by this instruction */
+ FM_MUST_WRITE, /* stack slots written by this instruction */
+ FM_LIVE_BEFORE, /* stack slots that may be read by this insn and its successors */
+ FM_MASK_CNT,
+};
+
+/*
+ * Per instruction stack masks for one frame of a function instance.
+ *
+ * Most frames use only a fraction of the stack budget, so instead of masks
+ * wide enough for every half-slot of the largest possible frame, all masks
+ * of one array share the width @words, and the marking functions widen the
+ * array as deeper half-slots are recorded. Mask @kind of the instruction at
+ * relative index @i is at &bits[(i * FM_MASK_CNT + kind) * words]. A
+ * half-slot at or past @words * BITS_PER_LONG is never read by this frame,
+ * hence never live. An instruction that may read the whole frame, such as a
+ * call passing a frame pointer to another subprog, widens the array to the
+ * program's stack budget, the deepest an accepted program can reach, so
+ * that the read cannot lose half-slots to a later widening.
+ */
+struct frame_masks {
+ u32 words;
+ unsigned long bits[];
};
/*
@@ -29,7 +58,7 @@ struct func_instance {
u32 subprog_start; /* cached env->subprog_info[subprog].start */
u32 insn_cnt; /* cached number of insns in the function */
/* Per frame, per instruction masks, frames allocated lazily. */
- struct per_frame_masks *frames[MAX_CALL_FRAMES];
+ struct frame_masks *frames[MAX_CALL_FRAMES];
bool must_write_initialized;
};
@@ -151,50 +180,122 @@ static int relative_idx(struct func_instance *instance, u32 insn_idx)
return insn_idx - instance->subprog_start;
}
-static struct per_frame_masks *get_frame_masks(struct func_instance *instance,
- u32 frame, u32 insn_idx)
+static u32 frame_mask_bits(struct frame_masks *fm)
{
- if (!instance->frames[frame])
- return NULL;
+ return fm->words * BITS_PER_LONG;
+}
- return &instance->frames[frame][relative_idx(instance, insn_idx)];
+static size_t frame_mask_words(struct func_instance *instance, u32 words)
+{
+ return (size_t)instance->insn_cnt * FM_MASK_CNT * words;
}
-static struct per_frame_masks *alloc_frame_masks(struct func_instance *instance,
- u32 frame, u32 insn_idx)
+/* Mask @kind of the instruction at relative index @rel */
+static unsigned long *rel_mask(struct frame_masks *fm, u32 rel, u32 kind)
{
- struct per_frame_masks *arr;
+ return fm->bits + ((size_t)rel * FM_MASK_CNT + kind) * fm->words;
+}
- if (!instance->frames[frame]) {
- arr = kvzalloc_objs(*arr, instance->insn_cnt,
- GFP_KERNEL_ACCOUNT);
- instance->frames[frame] = arr;
- if (!arr)
- return ERR_PTR(-ENOMEM);
+/*
+ * Make sure @frame has a mask array at least @words wide, allocating it or
+ * copying the existing masks into the wider stride as needed.
+ * @words must be in range [1, FRAME_MAX_WORDS].
+ */
+static struct frame_masks *widen_frame_masks(struct func_instance *instance,
+ u32 frame, u32 words)
+{
+ struct frame_masks *old = instance->frames[frame], *new;
+ u32 i, kind;
+
+ if (old && old->words >= words)
+ return old;
+ new = kvzalloc_flex(*new, bits, frame_mask_words(instance, words), GFP_KERNEL_ACCOUNT);
+ if (!new)
+ return NULL;
+ new->words = words;
+ if (old) {
+ for (i = 0; i < instance->insn_cnt; i++)
+ for (kind = 0; kind < FM_MASK_CNT; kind++)
+ memcpy(rel_mask(new, i, kind), rel_mask(old, i, kind),
+ old->words * sizeof(*old->bits));
+ kvfree(old);
}
- return get_frame_masks(instance, frame, insn_idx);
+ instance->frames[frame] = new;
+ return new;
}
-/* Accumulate may_read masks for @frame at @insn_idx */
-static int mark_stack_read(struct func_instance *instance, u32 frame, u32 insn_idx, spis_t mask)
+/*
+ * Set the inclusive half-slot range [lo, hi] in mask @kind of @frame at @insn_idx.
+ * An empty range, including one with a negative @hi as computed for a write
+ * that does not fully cover any half-slot, marks nothing.
+ */
+static int mark_stack_range(struct func_instance *instance, u32 frame, u32 insn_idx,
+ u32 kind, s32 lo, s32 hi)
{
- struct per_frame_masks *masks;
+ struct frame_masks *fm;
- masks = alloc_frame_masks(instance, frame, insn_idx);
- if (IS_ERR(masks))
- return PTR_ERR(masks);
- masks->may_read = spis_or(masks->may_read, mask);
+ /*
+ * An access past the frame bottom is rejected by the main verifier
+ * pass later, liveness only has to avoid running off the masks.
+ */
+ hi = min_t(s32, hi, FRAME_HALF_SPIS - 1);
+ if (lo > hi)
+ return 0;
+ fm = widen_frame_masks(instance, frame, BITS_TO_LONGS(hi + 1));
+ if (!fm)
+ return -ENOMEM;
+ bitmap_set(rel_mask(fm, relative_idx(instance, insn_idx), kind), lo, hi - lo + 1);
return 0;
}
-static int mark_stack_write(struct func_instance *instance, u32 frame, u32 insn_idx, spis_t mask)
+/* Accumulate may_read for half-slots [lo, hi] of @frame at @insn_idx */
+static int mark_stack_read(struct func_instance *instance, u32 frame, u32 insn_idx,
+ s32 lo, s32 hi)
+{
+ return mark_stack_range(instance, frame, insn_idx, FM_MAY_READ, lo, hi);
+}
+
+/* Accumulate must_write for half-slots [lo, hi] of @frame at @insn_idx */
+static int mark_stack_write(struct func_instance *instance, u32 frame, u32 insn_idx,
+ s32 lo, s32 hi)
{
- struct per_frame_masks *masks;
+ return mark_stack_range(instance, frame, insn_idx, FM_MUST_WRITE, lo, hi);
+}
- masks = alloc_frame_masks(instance, frame, insn_idx);
- if (IS_ERR(masks))
- return PTR_ERR(masks);
- masks->must_write = spis_or(masks->must_write, mask);
+/*
+ * Mark every half-slot of @frame as possibly read by @insn_idx. This widens
+ * the masks to the program's stack budget: a full read recorded at a narrower
+ * width would leave the bits added by a later widening clear and lose part of
+ * it, and an access past the budget is rejected by the main pass later, so no
+ * widening of an accepted program goes further.
+ */
+static int mark_stack_read_all(struct bpf_verifier_env *env, struct func_instance *instance,
+ u32 frame, u32 insn_idx)
+{
+ return mark_stack_read(instance, frame, insn_idx, 0,
+ env->stack_limit / BPF_HALF_REG_SIZE - 1);
+}
+
+/* Accumulate @src, a mask @src_words wide, into may_read of @frame at @insn_idx */
+static int mark_stack_read_mask(struct func_instance *instance, u32 frame, u32 insn_idx,
+ const unsigned long *src, u32 src_words)
+{
+ u32 nbits = src_words * BITS_PER_LONG;
+ struct frame_masks *fm;
+ unsigned long *dst;
+ u32 last, w;
+
+ last = find_last_bit(src, nbits);
+ if (last == nbits)
+ return 0;
+ fm = widen_frame_masks(instance, frame, BITS_TO_LONGS(last + 1));
+ if (!fm)
+ return -ENOMEM;
+ dst = rel_mask(fm, relative_idx(instance, insn_idx), FM_MAY_READ);
+ /* @src has no bits set past @last, hence none past @fm->words either */
+ src_words = min(src_words, fm->words);
+ for (w = 0; w < src_words; w++)
+ dst[w] |= src[w];
return 0;
}
@@ -272,33 +373,42 @@ __diag_pop();
static inline bool update_insn(struct bpf_verifier_env *env,
struct func_instance *instance, u32 frame, u32 insn_idx)
{
- spis_t new_before, new_after;
- struct per_frame_masks *insn, *succ_insn;
+ unsigned long new_after[FRAME_MAX_WORDS] = {};
+ unsigned long *may_read, *must_write, *live_before;
+ struct frame_masks *fm = instance->frames[frame];
+ u32 rel = relative_idx(instance, insn_idx);
struct bpf_iarray *succ;
- u32 s;
- bool changed;
+ bool changed = false;
+ u32 s, w;
succ = bpf_insn_successors(env, insn_idx);
if (succ->cnt == 0)
return false;
- changed = false;
- insn = get_frame_masks(instance, frame, insn_idx);
- new_before = SPIS_ZERO;
- new_after = SPIS_ZERO;
+ /* All instructions of one frame array share the same mask width */
for (s = 0; s < succ->cnt; ++s) {
- succ_insn = get_frame_masks(instance, frame, succ->items[s]);
- new_after = spis_or(new_after, succ_insn->live_before);
+ unsigned long *succ_live;
+
+ succ_live = rel_mask(fm, relative_idx(instance, succ->items[s]), FM_LIVE_BEFORE);
+ for (w = 0; w < fm->words; w++)
+ new_after[w] |= succ_live[w];
}
+ may_read = rel_mask(fm, rel, FM_MAY_READ);
+ must_write = rel_mask(fm, rel, FM_MUST_WRITE);
+ live_before = rel_mask(fm, rel, FM_LIVE_BEFORE);
/*
* New "live_before" is a union of all "live_before" of successors
* minus slots written by instruction plus slots read by instruction.
* new_before = (new_after & ~insn->must_write) | insn->may_read
*/
- new_before = spis_or(spis_and(new_after, spis_not(insn->must_write)),
- insn->may_read);
- changed |= !spis_equal(new_before, insn->live_before);
- insn->live_before = new_before;
+ for (w = 0; w < fm->words; w++) {
+ unsigned long new_before = (new_after[w] & ~must_write[w]) | may_read[w];
+
+ if (new_before != live_before[w]) {
+ live_before[w] = new_before;
+ changed = true;
+ }
+ }
return changed;
}
@@ -329,10 +439,12 @@ static void update_instance(struct bpf_verifier_env *env, struct func_instance *
static bool is_live_before(struct func_instance *instance, u32 insn_idx, u32 frameno, u32 half_spi)
{
- struct per_frame_masks *masks;
+ struct frame_masks *fm = instance->frames[frameno];
- masks = get_frame_masks(instance, frameno, insn_idx);
- return masks && spis_test_bit(masks->live_before, half_spi);
+ /* No recorded access reaches past the masks, so nothing there is live */
+ if (!fm || half_spi >= frame_mask_bits(fm))
+ return false;
+ return test_bit(half_spi, rel_mask(fm, relative_idx(instance, insn_idx), FM_LIVE_BEFORE));
}
int bpf_live_stack_query_init(struct bpf_verifier_env *env, struct bpf_verifier_state *st)
@@ -430,17 +542,19 @@ static int spi_off(int spi)
* When only one half is set, print as "-4h","-8h",...
* Runs of 3+ consecutive fully-set SPIs are collapsed: "fp0-8..-24"
*/
-static char *fmt_spis_mask(struct bpf_verifier_env *env, int frame, bool first, spis_t spis)
+static char *fmt_spis_mask(struct bpf_verifier_env *env, int frame, bool first,
+ const unsigned long *spis, u32 words)
{
int buf_sz = sizeof(env->tmp_str_buf);
+ int spi_cnt = words * BITS_PER_LONG / 2;
char *buf = env->tmp_str_buf;
int spi, n, run_start;
buf[0] = '\0';
- for (spi = 0; spi < STACK_SLOTS / 2 && buf_sz > 0; spi++) {
- bool lo = spis_test_bit(spis, spi * 2);
- bool hi = spis_test_bit(spis, spi * 2 + 1);
+ for (spi = 0; spi < spi_cnt && buf_sz > 0; spi++) {
+ bool lo = test_bit(spi * 2, spis);
+ bool hi = test_bit(spi * 2 + 1, spis);
const char *space = first ? "" : " ";
if (!lo && !hi)
@@ -450,16 +564,16 @@ static char *fmt_spis_mask(struct bpf_verifier_env *env, int frame, bool first,
/* half-spi */
n = scnprintf(buf, buf_sz, "%sfp%d%d%s",
space, frame, spi_off(spi) + (lo ? STACK_SLOT_SZ : 0), "h");
- } else if (spi + 2 < STACK_SLOTS / 2 &&
- spis_test_bit(spis, spi * 2 + 2) &&
- spis_test_bit(spis, spi * 2 + 3) &&
- spis_test_bit(spis, spi * 2 + 4) &&
- spis_test_bit(spis, spi * 2 + 5)) {
+ } else if (spi + 2 < spi_cnt &&
+ test_bit(spi * 2 + 2, spis) &&
+ test_bit(spi * 2 + 3, spis) &&
+ test_bit(spi * 2 + 4, spis) &&
+ test_bit(spi * 2 + 5, spis)) {
/* 3+ consecutive full spis */
run_start = spi;
- while (spi + 1 < STACK_SLOTS / 2 &&
- spis_test_bit(spis, (spi + 1) * 2) &&
- spis_test_bit(spis, (spi + 1) * 2 + 1))
+ while (spi + 1 < spi_cnt &&
+ test_bit((spi + 1) * 2, spis) &&
+ test_bit((spi + 1) * 2 + 1, spis))
spi++;
n = scnprintf(buf, buf_sz, "%sfp%d%d..%d",
space, frame, spi_off(run_start), spi_off(spi));
@@ -478,7 +592,8 @@ static void print_instance(struct bpf_verifier_env *env, struct func_instance *i
{
int start = env->subprog_info[instance->subprog].start;
struct bpf_insn *insns = env->prog->insnsi;
- struct per_frame_masks *masks;
+ struct frame_masks *fm;
+ unsigned long *mask;
int len = instance->insn_cnt;
int insn_idx, frame, i;
bool has_use, has_def;
@@ -501,10 +616,13 @@ static void print_instance(struct bpf_verifier_env *env, struct func_instance *i
pos = env->log.end_pos;
verbose(env, " use: ");
for (frame = instance->depth; frame >= 0; --frame) {
- masks = get_frame_masks(instance, frame, insn_idx);
- if (!masks || spis_is_zero(masks->may_read))
+ fm = instance->frames[frame];
+ if (!fm)
+ continue;
+ mask = rel_mask(fm, i, FM_MAY_READ);
+ if (bitmap_empty(mask, frame_mask_bits(fm)))
continue;
- verbose(env, "%s", fmt_spis_mask(env, frame, !has_use, masks->may_read));
+ verbose(env, "%s", fmt_spis_mask(env, frame, !has_use, mask, fm->words));
has_use = true;
}
if (!has_use)
@@ -512,10 +630,13 @@ static void print_instance(struct bpf_verifier_env *env, struct func_instance *i
pos = env->log.end_pos;
verbose(env, " def: ");
for (frame = instance->depth; frame >= 0; --frame) {
- masks = get_frame_masks(instance, frame, insn_idx);
- if (!masks || spis_is_zero(masks->must_write))
+ fm = instance->frames[frame];
+ if (!fm)
+ continue;
+ mask = rel_mask(fm, i, FM_MUST_WRITE);
+ if (bitmap_empty(mask, frame_mask_bits(fm)))
continue;
- verbose(env, "%s", fmt_spis_mask(env, frame, !has_def, masks->must_write));
+ verbose(env, "%s", fmt_spis_mask(env, frame, !has_def, mask, fm->words));
has_def = true;
}
if (!has_def)
@@ -584,9 +705,9 @@ static int print_instances(struct bpf_verifier_env *env)
* - same frame + different offset -> offset-imprecise
* - different frames -> fully-imprecise (bitmask OR)
*
- * At memory access sites (LDX/STX/ST), offset-imprecise marks only
- * the known frame's access mask as SPIS_ALL, while fully-imprecise
- * iterates bits in the bitmask and routes each frame to its target.
+ * At memory access sites (LDX/STX/ST), offset-imprecise marks the known
+ * frame as fully read, while fully-imprecise iterates bits in the bitmask
+ * and routes each frame to its target.
*/
#define MAX_ARG_OFFSETS 4
@@ -1235,7 +1356,6 @@ static int record_stack_access_off(struct func_instance *instance, s64 fp_off,
s64 access_bytes, u32 frame, u32 insn_idx)
{
s32 slot_hi, slot_lo;
- spis_t mask;
if (fp_off >= 0)
/*
@@ -1247,27 +1367,19 @@ static int record_stack_access_off(struct func_instance *instance, s64 fp_off,
if (access_bytes == S64_MIN) {
/* helper/kfunc read unknown amount of bytes from fp_off until fp+0 */
slot_hi = (-fp_off - 1) / STACK_SLOT_SZ;
- mask = SPIS_ZERO;
- spis_or_range(&mask, 0, slot_hi);
- return mark_stack_read(instance, frame, insn_idx, mask);
+ return mark_stack_read(instance, frame, insn_idx, 0, slot_hi);
}
if (access_bytes > 0) {
/* Mark any touched slot as use */
slot_hi = (-fp_off - 1) / STACK_SLOT_SZ;
slot_lo = max_t(s32, (-fp_off - access_bytes) / STACK_SLOT_SZ, 0);
- mask = SPIS_ZERO;
- spis_or_range(&mask, slot_lo, slot_hi);
- return mark_stack_read(instance, frame, insn_idx, mask);
+ return mark_stack_read(instance, frame, insn_idx, slot_lo, slot_hi);
} else if (access_bytes < 0) {
/* Mark only fully covered slots as def */
access_bytes = -access_bytes;
slot_hi = (-fp_off) / STACK_SLOT_SZ - 1;
slot_lo = max_t(s32, (-fp_off - access_bytes + STACK_SLOT_SZ - 1) / STACK_SLOT_SZ, 0);
- if (slot_lo <= slot_hi) {
- mask = SPIS_ZERO;
- spis_or_range(&mask, slot_lo, slot_hi);
- return mark_stack_write(instance, frame, insn_idx, mask);
- }
+ return mark_stack_write(instance, frame, insn_idx, slot_lo, slot_hi);
}
return 0;
}
@@ -1276,7 +1388,7 @@ static int record_stack_access_off(struct func_instance *instance, s64 fp_off,
* 'arg' is FP-derived argument to helper/kfunc or load/store that
* reads (positive) or writes (negative) 'access_bytes' into 'use' or 'def'.
*/
-static int record_stack_access(struct func_instance *instance,
+static int record_stack_access(struct bpf_verifier_env *env, struct func_instance *instance,
const struct arg_track *arg,
s64 access_bytes, u32 frame, u32 insn_idx)
{
@@ -1286,7 +1398,7 @@ static int record_stack_access(struct func_instance *instance,
return 0;
if (arg->off_cnt == 0) {
if (access_bytes > 0 || access_bytes == S64_MIN)
- return mark_stack_read(instance, frame, insn_idx, SPIS_ALL);
+ return mark_stack_read_all(env, instance, frame, insn_idx);
return 0;
}
if (access_bytes != S64_MIN && access_bytes < 0 && arg->off_cnt != 1)
@@ -1305,7 +1417,8 @@ static int record_stack_access(struct func_instance *instance,
* When a pointer is ARG_IMPRECISE, conservatively mark every frame in
* the bitmask as fully used.
*/
-static int record_imprecise(struct func_instance *instance, u32 mask, u32 insn_idx)
+static int record_imprecise(struct bpf_verifier_env *env, struct func_instance *instance,
+ u32 mask, u32 insn_idx)
{
int depth = instance->depth;
int f, err;
@@ -1314,7 +1427,7 @@ static int record_imprecise(struct func_instance *instance, u32 mask, u32 insn_i
if (!(mask & 1))
continue;
if (f <= depth) {
- err = mark_stack_read(instance, f, insn_idx, SPIS_ALL);
+ err = mark_stack_read_all(env, instance, f, insn_idx);
if (err)
return err;
}
@@ -1383,9 +1496,9 @@ static int record_load_store_access(struct bpf_verifier_env *env,
}
if (ptr->frame >= 0 && ptr->frame <= depth)
- return record_stack_access(instance, ptr, sz, ptr->frame, insn_idx);
+ return record_stack_access(env, instance, ptr, sz, ptr->frame, insn_idx);
if (ptr->frame == ARG_IMPRECISE)
- return record_imprecise(instance, ptr->mask, insn_idx);
+ return record_imprecise(env, instance, ptr->mask, insn_idx);
/* ARG_NONE: not derived from any frame pointer, skip */
return 0;
}
@@ -1410,7 +1523,7 @@ static int record_arg_access(struct bpf_verifier_env *env,
bytes = bpf_kfunc_stack_access_bytes(env, insn, arg_idx, insn_idx);
} else {
for (int f = 0; f <= depth; f++) {
- err = mark_stack_read(instance, f, insn_idx, SPIS_ALL);
+ err = mark_stack_read_all(env, instance, f, insn_idx);
if (err)
return err;
}
@@ -1420,9 +1533,9 @@ static int record_arg_access(struct bpf_verifier_env *env,
return 0;
if (frame >= 0 && frame <= depth)
- err = record_stack_access(instance, at, bytes, frame, insn_idx);
+ err = record_stack_access(env, instance, at, bytes, frame, insn_idx);
else if (frame == ARG_IMPRECISE)
- err = record_imprecise(instance, at->mask, insn_idx);
+ err = record_imprecise(env, instance, at->mask, insn_idx);
return err;
}
@@ -1772,36 +1885,52 @@ static bool has_fp_args(struct arg_track *args)
* may_read: union (any pass might read the slot).
* must_write: intersection (only slots written on ALL passes are guaranteed).
* live_before is recomputed by a subsequent update_instance() on @dst.
+ *
+ * The two instances may have settled on different mask widths for the same
+ * frame, so @dst is widened to cover @src first. A word only @dst has counts
+ * as zero on the @src side: it unions into may_read as a no-op and intersects
+ * must_write to empty.
*/
-static void merge_instances(struct func_instance *dst, struct func_instance *src)
+static int merge_instances(struct func_instance *dst, struct func_instance *src)
{
- int f, i;
+ struct frame_masks *d, *s;
+ u32 f, i, w;
for (f = 0; f <= dst->depth; f++) {
- if (!src->frames[f]) {
+ s = src->frames[f];
+ d = dst->frames[f];
+ if (!s) {
/* This pass didn't touch frame f — must_write intersects with empty. */
- if (dst->frames[f])
+ if (d)
for (i = 0; i < dst->insn_cnt; i++)
- dst->frames[f][i].must_write = SPIS_ZERO;
+ bitmap_zero(rel_mask(d, i, FM_MUST_WRITE),
+ frame_mask_bits(d));
continue;
}
- if (!dst->frames[f]) {
+ if (!d) {
/* Previous pass didn't touch frame f — take src, zero must_write. */
- dst->frames[f] = src->frames[f];
+ dst->frames[f] = s;
src->frames[f] = NULL;
for (i = 0; i < dst->insn_cnt; i++)
- dst->frames[f][i].must_write = SPIS_ZERO;
+ bitmap_zero(rel_mask(s, i, FM_MUST_WRITE), frame_mask_bits(s));
continue;
}
+ d = widen_frame_masks(dst, f, s->words);
+ if (!d)
+ return -ENOMEM;
for (i = 0; i < dst->insn_cnt; i++) {
- dst->frames[f][i].may_read =
- spis_or(dst->frames[f][i].may_read,
- src->frames[f][i].may_read);
- dst->frames[f][i].must_write =
- spis_and(dst->frames[f][i].must_write,
- src->frames[f][i].must_write);
+ unsigned long *dst_read = rel_mask(d, i, FM_MAY_READ);
+ unsigned long *dst_write = rel_mask(d, i, FM_MUST_WRITE);
+ unsigned long *src_read = rel_mask(s, i, FM_MAY_READ);
+ unsigned long *src_write = rel_mask(s, i, FM_MUST_WRITE);
+
+ for (w = 0; w < d->words; w++) {
+ dst_read[w] |= w < s->words ? src_read[w] : 0;
+ dst_write[w] &= w < s->words ? src_write[w] : 0;
+ }
}
}
+ return 0;
}
static struct func_instance *fresh_instance(struct func_instance *src)
@@ -1916,7 +2045,7 @@ static int analyze_subprog(struct bpf_verifier_env *env,
if (info[subprog].at_in[j][caller_reg].frame == ARG_NONE)
continue;
for (int f = 0; f <= depth; f++) {
- err = mark_stack_read(instance, f, idx, SPIS_ALL);
+ err = mark_stack_read_all(env, instance, f, idx);
if (err)
goto out_free;
}
@@ -1955,13 +2084,18 @@ static int analyze_subprog(struct bpf_verifier_env *env,
/* Pull callee's entry liveness back to caller's callsite */
{
u32 callee_start = callee_instance->subprog_start;
- struct per_frame_masks *entry;
+ struct frame_masks *callee_fm;
for (int f = 0; f < callee_instance->depth; f++) {
- entry = get_frame_masks(callee_instance, f, callee_start);
- if (!entry)
+ callee_fm = callee_instance->frames[f];
+ if (!callee_fm)
continue;
- err = mark_stack_read(instance, f, idx, entry->live_before);
+ err = mark_stack_read_mask(instance, f, idx,
+ rel_mask(callee_fm,
+ relative_idx(callee_instance,
+ callee_start),
+ FM_LIVE_BEFORE),
+ callee_fm->words);
if (err)
goto out_free;
}
@@ -1969,9 +2103,11 @@ static int analyze_subprog(struct bpf_verifier_env *env,
}
if (prev_instance) {
- merge_instances(prev_instance, instance);
+ err = merge_instances(prev_instance, instance);
free_instance(instance);
instance = prev_instance;
+ if (err)
+ return err;
}
update_instance(env, instance);
return 0;
diff --git a/kernel/bpf/log.c b/kernel/bpf/log.c
index fb032dfdc0dee..d850a7863d2ed 100644
--- a/kernel/bpf/log.c
+++ b/kernel/bpf/log.c
@@ -716,7 +716,8 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifie
verbose(env, "=");
print_reg_state(env, state, reg);
}
- for (i = 0; i < state->allocated_stack / BPF_REG_SIZE; i++) {
+ for (i = 0; i < bpf_stack_nr_slots(state); i++) {
+ struct bpf_stack_state *slot = bpf_stack_slot(state, i);
char types_buf[BPF_REG_SIZE + 1];
const char *sep = "";
bool valid = false;
@@ -727,7 +728,7 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifie
continue;
for (j = 0; j < BPF_REG_SIZE; j++) {
- slot_type = state->stack[i].slot_type[j];
+ slot_type = slot->slot_type[j];
if (slot_type != STACK_INVALID && slot_type != STACK_POISON)
valid = true;
types_buf[j] = slot_type_char[slot_type];
@@ -736,12 +737,12 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifie
if (!valid)
continue;
- reg = &state->stack[i].spilled_ptr;
- switch (state->stack[i].slot_type[BPF_REG_SIZE - 1]) {
+ reg = &slot->spilled_ptr;
+ switch (slot->slot_type[BPF_REG_SIZE - 1]) {
case STACK_SPILL:
/* print MISC/ZERO/INVALID slots above subreg spill */
for (j = 0; j < BPF_REG_SIZE; j++)
- if (state->stack[i].slot_type[j] == STACK_SPILL)
+ if (slot->slot_type[j] == STACK_SPILL)
break;
types_buf[j] = '\0';
@@ -751,7 +752,7 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifie
case STACK_DYNPTR:
/* skip to main dynptr slot */
i += BPF_DYNPTR_NR_SLOTS - 1;
- reg = &state->stack[i].spilled_ptr;
+ reg = &bpf_stack_slot(state, i)->spilled_ptr;
verbose(env, " fp%d", (-i - 1) * BPF_REG_SIZE);
verbose(env, "=dynptr_%s(", dynptr_type_str(reg->dynptr.type));
diff --git a/kernel/bpf/states.c b/kernel/bpf/states.c
index 66fb11b6c6a76..5078e4832c6e1 100644
--- a/kernel/bpf/states.c
+++ b/kernel/bpf/states.c
@@ -335,21 +335,18 @@ static bool check_ids(u32 old_id, u32 cur_id, struct bpf_idmap *idmap)
return false;
}
- /* Reached the end of known mappings; haven't seen this id before */
- if (idmap->cnt < BPF_ID_MAP_SIZE) {
- map[idmap->cnt].old = old_id;
- map[idmap->cnt].cur = cur_id;
- idmap->cnt++;
- return true;
- }
-
/*
- * idmap slots are bounded by the number of registers and stack slots.
- * Since referenced dynptrs acquire intermediate references that do
- * not live in either, so the map can be exhausted. Since it is unlikely,
- * fail the verification by treating the states as not equivalent.
+ * Reached the end of known mappings; haven't seen this id before. If
+ * the map cannot grow, treat the states as not equivalent, which only
+ * costs pruning.
*/
- return false;
+ if (!bpf_id_scratch_reserve((void **)&idmap->map, &idmap->cap, idmap->cnt, sizeof(*map)))
+ return false;
+ map = idmap->map;
+ map[idmap->cnt].old = old_id;
+ map[idmap->cnt].cur = cur_id;
+ idmap->cnt++;
+ return true;
}
/*
@@ -415,14 +412,14 @@ static void __clean_func_state(struct bpf_verifier_env *env,
* half_spi 2*i → lower half: slot_type[0..3] (closer to FP)
* half_spi 2*i+1 → upper half: slot_type[4..7] (farther from FP)
*/
- for (i = 0; i < st->allocated_stack / BPF_REG_SIZE; i++) {
+ for (i = 0; i < bpf_stack_nr_slots(st); i++) {
bool lo_live = bpf_stack_slot_alive(env, frame, i * 2);
bool hi_live = bpf_stack_slot_alive(env, frame, i * 2 + 1);
if (!hi_live || !lo_live) {
int start = !lo_live ? 0 : BPF_REG_SIZE / 2;
int end = !hi_live ? BPF_REG_SIZE : BPF_REG_SIZE / 2;
- u8 stype = st->stack[i].slot_type[7];
+ u8 stype = bpf_stack_slot(st, i)->slot_type[7];
/*
* Don't clear special slots.
@@ -442,7 +439,7 @@ static void __clean_func_state(struct bpf_verifier_env *env,
* rejecting as non-scalar register fills.
*/
if (!hi_live) {
- struct bpf_reg_state *spill = &st->stack[i].spilled_ptr;
+ struct bpf_reg_state *spill = &bpf_stack_slot(st, i)->spilled_ptr;
if (lo_live && stype == STACK_SPILL) {
if (spill->type != SCALAR_VALUE)
@@ -454,7 +451,7 @@ static void __clean_func_state(struct bpf_verifier_env *env,
if (bpf_register_is_null(spill))
continue;
for (j = 0; j < 4; j++) {
- u8 *t = &st->stack[i].slot_type[j];
+ u8 *t = &bpf_stack_slot(st, i)->slot_type[j];
if (*t == STACK_SPILL)
*t = STACK_MISC;
@@ -463,7 +460,7 @@ static void __clean_func_state(struct bpf_verifier_env *env,
bpf_mark_reg_not_init(env, spill);
}
for (j = start; j < end; j++)
- st->stack[i].slot_type[j] = STACK_POISON;
+ bpf_stack_slot(st, i)->slot_type[j] = STACK_POISON;
}
}
}
@@ -707,37 +704,38 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
* didn't use them
*/
for (i = 0; i < old->allocated_stack; i++) {
+ struct bpf_stack_state *old_slot, *cur_slot;
struct bpf_reg_state *old_reg, *cur_reg;
int im = i % BPF_REG_SIZE;
+ u8 old_type;
spi = i / BPF_REG_SIZE;
+ old_slot = bpf_stack_slot(old, spi);
+ old_type = old_slot->slot_type[im];
+ cur_slot = i < cur->allocated_stack ? bpf_stack_slot(cur, spi) : NULL;
if (exact == EXACT) {
- u8 old_type = old->stack[spi].slot_type[i % BPF_REG_SIZE];
- u8 cur_type = i < cur->allocated_stack ?
- cur->stack[spi].slot_type[i % BPF_REG_SIZE] : STACK_INVALID;
+ u8 cur_type = cur_slot ? cur_slot->slot_type[im] : STACK_INVALID;
/* STACK_INVALID and STACK_POISON are equivalent for pruning */
if (old_type == STACK_POISON)
old_type = STACK_INVALID;
if (cur_type == STACK_POISON)
cur_type = STACK_INVALID;
- if (i >= cur->allocated_stack || old_type != cur_type)
+ if (!cur_slot || old_type != cur_type)
return false;
}
- if (old->stack[spi].slot_type[i % BPF_REG_SIZE] == STACK_INVALID ||
- old->stack[spi].slot_type[i % BPF_REG_SIZE] == STACK_POISON)
+ if (old_type == STACK_INVALID || old_type == STACK_POISON)
continue;
- if (env->allow_uninit_stack &&
- old->stack[spi].slot_type[i % BPF_REG_SIZE] == STACK_MISC)
+ if (env->allow_uninit_stack && old_type == STACK_MISC)
continue;
/* explored stack has more populated slots than current stack
* and these slots were used
*/
- if (i >= cur->allocated_stack)
+ if (!cur_slot)
return false;
/*
@@ -747,8 +745,8 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
* regsafe() to ensure scalar ids are compared.
*/
if (im == 0 || im == 4) {
- old_reg = scalar_reg_for_stack(env, &old->stack[spi], im);
- cur_reg = scalar_reg_for_stack(env, &cur->stack[spi], im);
+ old_reg = scalar_reg_for_stack(env, old_slot, im);
+ cur_reg = scalar_reg_for_stack(env, cur_slot, im);
if (old_reg && cur_reg) {
if (!regsafe(env, old_reg, cur_reg, idmap, exact))
return false;
@@ -761,21 +759,19 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
* it will be safe with zero-initialized stack.
* The opposite is not true
*/
- if (old->stack[spi].slot_type[i % BPF_REG_SIZE] == STACK_MISC &&
- cur->stack[spi].slot_type[i % BPF_REG_SIZE] == STACK_ZERO)
+ if (old_type == STACK_MISC && cur_slot->slot_type[im] == STACK_ZERO)
continue;
- if (old->stack[spi].slot_type[i % BPF_REG_SIZE] !=
- cur->stack[spi].slot_type[i % BPF_REG_SIZE])
+ if (old_type != cur_slot->slot_type[im])
/* Ex: old explored (safe) state has STACK_SPILL in
* this stack slot, but current has STACK_MISC ->
* this verifier states are not equivalent,
* return false to continue verification of this path
*/
return false;
- if (i % BPF_REG_SIZE != BPF_REG_SIZE - 1)
+ if (im != BPF_REG_SIZE - 1)
continue;
/* Both old and cur are having same slot_type */
- switch (old->stack[spi].slot_type[BPF_REG_SIZE - 1]) {
+ switch (old_type) {
case STACK_SPILL:
/* when explored and current stack slot are both storing
* spilled registers, check that stored pointers types
@@ -787,13 +783,13 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
* such verifier states are not equivalent.
* return false to continue verification of this path
*/
- if (!regsafe(env, &old->stack[spi].spilled_ptr,
- &cur->stack[spi].spilled_ptr, idmap, exact))
+ if (!regsafe(env, &old_slot->spilled_ptr, &cur_slot->spilled_ptr,
+ idmap, exact))
return false;
break;
case STACK_DYNPTR:
- old_reg = &old->stack[spi].spilled_ptr;
- cur_reg = &cur->stack[spi].spilled_ptr;
+ old_reg = &old_slot->spilled_ptr;
+ cur_reg = &cur_slot->spilled_ptr;
if (old_reg->dynptr.type != cur_reg->dynptr.type ||
old_reg->dynptr.first_slot != cur_reg->dynptr.first_slot ||
!check_ids(old_reg->id, cur_reg->id, idmap) ||
@@ -801,8 +797,8 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
return false;
break;
case STACK_ITER:
- old_reg = &old->stack[spi].spilled_ptr;
- cur_reg = &cur->stack[spi].spilled_ptr;
+ old_reg = &old_slot->spilled_ptr;
+ cur_reg = &cur_slot->spilled_ptr;
/* iter.depth is not compared between states as it
* doesn't matter for correctness and would otherwise
* prevent convergence; we maintain it only to prevent
@@ -818,8 +814,8 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
return false;
break;
case STACK_IRQ_FLAG:
- old_reg = &old->stack[spi].spilled_ptr;
- cur_reg = &cur->stack[spi].spilled_ptr;
+ old_reg = &old_slot->spilled_ptr;
+ cur_reg = &cur_slot->spilled_ptr;
if (!check_ids(old_reg->id, cur_reg->id, idmap) ||
old_reg->irq.kfunc_class != cur_reg->irq.kfunc_class)
return false;
@@ -966,6 +962,27 @@ static bool func_states_equal(struct bpf_verifier_env *env, struct bpf_func_stat
return true;
}
+/*
+ * Make room for one more entry in an id scratch array, doubling it as needed.
+ * Returns false if it could not grow; callers then treat the id as unknown
+ * or the states as different, which is always safe.
+ */
+bool bpf_id_scratch_reserve(void **arr, u32 *cap, u32 cnt, size_t elem_size)
+{
+ u32 new_cap;
+ void *p;
+
+ if (cnt < *cap)
+ return true;
+ new_cap = *cap ? *cap * 2 : 64;
+ p = krealloc_array(*arr, new_cap, elem_size, GFP_KERNEL_ACCOUNT | __GFP_NOWARN);
+ if (!p)
+ return false;
+ *arr = p;
+ *cap = new_cap;
+ return true;
+}
+
static void reset_idmap_scratch(struct bpf_verifier_env *env)
{
struct bpf_idmap *idmap = &env->idmap_scratch;
@@ -1043,10 +1060,10 @@ static int propagate_precision(struct bpf_verifier_env *env,
first = false;
}
- for (i = 0; i < state->allocated_stack / BPF_REG_SIZE; i++) {
- if (!bpf_is_spilled_reg(&state->stack[i]))
+ for (i = 0; i < bpf_stack_nr_slots(state); i++) {
+ if (!bpf_is_spilled_reg(bpf_stack_slot(state, i)))
continue;
- state_reg = &state->stack[i].spilled_ptr;
+ state_reg = &bpf_stack_slot(state, i)->spilled_ptr;
if (state_reg->type != SCALAR_VALUE ||
!state_reg->precise)
continue;
@@ -1192,15 +1209,15 @@ static bool iter_active_depths_differ(struct bpf_verifier_state *old, struct bpf
for (fr = old->curframe; fr >= 0; fr--) {
state = old->frame[fr];
- for (i = 0; i < state->allocated_stack / BPF_REG_SIZE; i++) {
- if (state->stack[i].slot_type[0] != STACK_ITER)
+ for (i = 0; i < bpf_stack_nr_slots(state); i++) {
+ if (bpf_stack_slot(state, i)->slot_type[0] != STACK_ITER)
continue;
- slot = &state->stack[i].spilled_ptr;
+ slot = &bpf_stack_slot(state, i)->spilled_ptr;
if (slot->iter.state != BPF_ITER_STATE_ACTIVE)
continue;
- cur_slot = &cur->frame[fr]->stack[i].spilled_ptr;
+ cur_slot = &bpf_stack_slot(cur->frame[fr], i)->spilled_ptr;
if (cur_slot->iter.depth != slot->iter.depth)
return true;
}
@@ -1222,10 +1239,10 @@ static void mark_all_scalars_imprecise(struct bpf_verifier_env *env, struct bpf_
continue;
reg->precise = false;
}
- for (j = 0; j < func->allocated_stack / BPF_REG_SIZE; j++) {
- if (!bpf_is_spilled_reg(&func->stack[j]))
+ for (j = 0; j < bpf_stack_nr_slots(func); j++) {
+ if (!bpf_is_spilled_reg(bpf_stack_slot(func, j)))
continue;
- reg = &func->stack[j].spilled_ptr;
+ reg = &bpf_stack_slot(func, j)->spilled_ptr;
if (reg->type != SCALAR_VALUE)
continue;
reg->precise = false;
@@ -1328,7 +1345,7 @@ int bpf_is_state_visited(struct bpf_verifier_env *env, int insn_idx)
*/
if (is_iter_next_insn(env, insn_idx)) {
if (states_equal(env, &sl->state, cur, RANGE_WITHIN)) {
- struct bpf_func_state *cur_frame;
+ struct bpf_func_state *cur_frame, *iter_frame;
struct bpf_reg_state *iter_state, *iter_reg;
int spi;
@@ -1342,7 +1359,8 @@ int bpf_is_state_visited(struct bpf_verifier_env *env, int insn_idx)
* no need for extra (re-)validations
*/
spi = bpf_get_spi(iter_reg->var_off.value);
- iter_state = &bpf_func(env, iter_reg)->stack[spi].spilled_ptr;
+ iter_frame = bpf_func(env, iter_reg);
+ iter_state = &bpf_stack_slot(iter_frame, spi)->spilled_ptr;
if (iter_state->iter.state == BPF_ITER_STATE_ACTIVE) {
loop = true;
goto hit;
@@ -1410,7 +1428,7 @@ int bpf_is_state_visited(struct bpf_verifier_env *env, int insn_idx)
*/
err = 0;
if (bpf_is_jmp_point(env, env->insn_idx))
- err = bpf_push_jmp_history(env, cur, 0, 0, 0, 0);
+ err = bpf_push_jmp_history(env, cur, 0, 0, 0, NULL, 0);
err = err ? : propagate_precision(env, &sl->state, cur, NULL);
if (err)
return err;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index a7c9e2d8965d5..9bd7f9b3a67cb 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -598,16 +598,17 @@ bool bpf_is_may_goto_insn(struct bpf_insn *insn)
static bool is_spi_bounds_valid(struct bpf_func_state *state, int spi, int nr_slots)
{
- int allocated_slots = state->allocated_stack / BPF_REG_SIZE;
+ int allocated_slots = bpf_stack_nr_slots(state);
- /* We need to check that slots between [spi - nr_slots + 1, spi] are
- * within [0, allocated_stack).
- *
- * Please note that the spi grows downwards. For example, a dynptr
- * takes the size of two stack slots; the first slot will be at
- * spi and the second slot will be at spi - 1.
- */
- return spi - nr_slots + 1 >= 0 && spi < allocated_slots;
+ /*
+ * We need to check that slots between [spi - nr_slots + 1, spi] are
+ * within [0, allocated_stack).
+ *
+ * Please note that the spi grows downwards. For example, a dynptr
+ * takes the size of two stack slots; the first slot will be at
+ * spi and the second slot will be at spi - 1.
+ */
+ return spi - nr_slots + 1 >= 0 && spi < allocated_slots;
}
static int stack_slot_obj_get_spi(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
@@ -751,8 +752,8 @@ static int mark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_
return err;
for (i = 0; i < BPF_REG_SIZE; i++) {
- state->stack[spi].slot_type[i] = STACK_DYNPTR;
- state->stack[spi - 1].slot_type[i] = STACK_DYNPTR;
+ bpf_stack_slot(state, spi)->slot_type[i] = STACK_DYNPTR;
+ bpf_stack_slot(state, spi - 1)->slot_type[i] = STACK_DYNPTR;
}
type = arg_to_dynptr_type(arg_type);
@@ -785,8 +786,8 @@ static int mark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_
parent_id = dynptr->parent_id;
}
- mark_dynptr_stack_regs(env, &state->stack[spi].spilled_ptr,
- &state->stack[spi - 1].spilled_ptr, type, parent_id);
+ mark_dynptr_stack_regs(env, &bpf_stack_slot(state, spi)->spilled_ptr,
+ &bpf_stack_slot(state, spi - 1)->spilled_ptr, type, parent_id);
return 0;
}
@@ -818,7 +819,7 @@ static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_re
* all clones and derived slices. For non-referenced dynptr, only
* the dynptr and slices derived from it will be invalidated.
*/
- reg = &state->stack[spi].spilled_ptr;
+ reg = &bpf_stack_slot(state, spi)->spilled_ptr;
return release_reference(env, dynptr_type_referenced(reg->dynptr.type)
? reg->parent_id
: reg->id);
@@ -857,6 +858,7 @@ static int dynptr_ref_cnt(struct bpf_verifier_env *env, int v_parent_id)
static int destroy_if_dynptr_stack_slot(struct bpf_verifier_env *env,
struct bpf_func_state *state, int spi)
{
+ struct bpf_stack_state *slot = bpf_stack_slot(state, spi);
int err = 0;
/* We always ensure that STACK_DYNPTR is never set partially,
@@ -864,20 +866,22 @@ static int destroy_if_dynptr_stack_slot(struct bpf_verifier_env *env,
* different for STACK_SPILL, where it may be only set for
* 1 byte, so code has to use is_spilled_reg.
*/
- if (state->stack[spi].slot_type[0] != STACK_DYNPTR)
+ if (slot->slot_type[0] != STACK_DYNPTR)
return 0;
/* Reposition spi to first slot */
- if (!state->stack[spi].spilled_ptr.dynptr.first_slot)
+ if (!slot->spilled_ptr.dynptr.first_slot) {
spi = spi + 1;
+ slot = bpf_stack_slot(state, spi);
+ }
/*
* A referenced dynptr can be overwritten only if there is at
* least one other dynptr sharing the same virtual ref parent,
* ensuring the reference can still be properly released.
*/
- if (dynptr_type_referenced(state->stack[spi].spilled_ptr.dynptr.type) &&
- dynptr_ref_cnt(env, state->stack[spi].spilled_ptr.parent_id) <= 1) {
+ if (dynptr_type_referenced(slot->spilled_ptr.dynptr.type) &&
+ dynptr_ref_cnt(env, slot->spilled_ptr.parent_id) <= 1) {
verbose(env, "cannot overwrite referenced dynptr\n");
bpf_diag_res(
env, env->insn_idx, "referenced dynptr overwrite",
@@ -887,7 +891,7 @@ static int destroy_if_dynptr_stack_slot(struct bpf_verifier_env *env,
}
/* Invalidate the dynptr and any derived slices */
- err = release_reference(env, state->stack[spi].spilled_ptr.id);
+ err = release_reference(env, slot->spilled_ptr.id);
if (!err) {
mark_stack_slot_scratched(env, spi);
mark_stack_slot_scratched(env, spi - 1);
@@ -927,6 +931,7 @@ static bool is_dynptr_reg_valid_uninit(struct bpf_verifier_env *env, struct bpf_
static bool is_dynptr_reg_valid_init(struct bpf_verifier_env *env, struct bpf_reg_state *reg)
{
struct bpf_func_state *state = bpf_func(env, reg);
+ struct bpf_stack_state *slot;
int i, spi;
/* This already represents first slot of initialized bpf_dynptr.
@@ -941,12 +946,13 @@ static bool is_dynptr_reg_valid_init(struct bpf_verifier_env *env, struct bpf_re
spi = dynptr_get_spi(env, reg);
if (spi < 0)
return false;
- if (!state->stack[spi].spilled_ptr.dynptr.first_slot)
+ slot = bpf_stack_slot(state, spi);
+ if (!slot->spilled_ptr.dynptr.first_slot)
return false;
for (i = 0; i < BPF_REG_SIZE; i++) {
- if (state->stack[spi].slot_type[i] != STACK_DYNPTR ||
- state->stack[spi - 1].slot_type[i] != STACK_DYNPTR)
+ if (slot->slot_type[i] != STACK_DYNPTR ||
+ bpf_stack_slot(state, spi - 1)->slot_type[i] != STACK_DYNPTR)
return false;
}
@@ -965,7 +971,7 @@ static enum bpf_dynptr_type dynptr_reg_type(struct bpf_verifier_env *env, struct
if (spi < 0)
return BPF_DYNPTR_TYPE_INVALID;
state = bpf_func(env, reg);
- return state->stack[spi].spilled_ptr.dynptr.type;
+ return bpf_stack_slot(state, spi)->spilled_ptr.dynptr.type;
}
static bool is_dynptr_type_expected(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
@@ -1005,7 +1011,7 @@ static int mark_stack_slots_iter(struct bpf_verifier_env *env,
return id;
for (i = 0; i < nr_slots; i++) {
- struct bpf_stack_state *slot = &state->stack[spi - i];
+ struct bpf_stack_state *slot = bpf_stack_slot(state, spi - i);
struct bpf_reg_state *st = &slot->spilled_ptr;
__mark_reg_known_zero(st);
@@ -1042,7 +1048,7 @@ static int unmark_stack_slots_iter(struct bpf_verifier_env *env,
return spi;
for (i = 0; i < nr_slots; i++) {
- struct bpf_stack_state *slot = &state->stack[spi - i];
+ struct bpf_stack_state *slot = bpf_stack_slot(state, spi - i);
struct bpf_reg_state *st = &slot->spilled_ptr;
if (i == 0)
@@ -1076,7 +1082,7 @@ static bool is_iter_reg_valid_uninit(struct bpf_verifier_env *env,
return false;
for (i = 0; i < nr_slots; i++) {
- struct bpf_stack_state *slot = &state->stack[spi - i];
+ struct bpf_stack_state *slot = bpf_stack_slot(state, spi - i);
for (j = 0; j < BPF_REG_SIZE; j++)
if (slot->slot_type[j] == STACK_ITER)
@@ -1097,7 +1103,7 @@ static int is_iter_reg_valid_init(struct bpf_verifier_env *env, struct bpf_reg_s
return -EINVAL;
for (i = 0; i < nr_slots; i++) {
- struct bpf_stack_state *slot = &state->stack[spi - i];
+ struct bpf_stack_state *slot = bpf_stack_slot(state, spi - i);
struct bpf_reg_state *st = &slot->spilled_ptr;
if (st->type & PTR_UNTRUSTED)
@@ -1139,7 +1145,7 @@ static int mark_stack_slot_irq_flag(struct bpf_verifier_env *env,
if (id < 0)
return id;
- slot = &state->stack[spi];
+ slot = bpf_stack_slot(state, spi);
st = &slot->spilled_ptr;
__mark_reg_known_zero(st);
@@ -1166,7 +1172,7 @@ static int unmark_stack_slot_irq_flag(struct bpf_verifier_env *env, struct bpf_r
if (spi < 0)
return spi;
- slot = &state->stack[spi];
+ slot = bpf_stack_slot(state, spi);
st = &slot->spilled_ptr;
if (st->irq.kfunc_class != kfunc_class) {
@@ -1235,7 +1241,7 @@ static bool is_irq_flag_reg_valid_uninit(struct bpf_verifier_env *env, struct bp
if (spi < 0)
return false;
- slot = &state->stack[spi];
+ slot = bpf_stack_slot(state, spi);
for (i = 0; i < BPF_REG_SIZE; i++)
if (slot->slot_type[i] == STACK_IRQ_FLAG)
@@ -1254,7 +1260,7 @@ static int is_irq_flag_reg_valid_init(struct bpf_verifier_env *env, struct bpf_r
if (spi < 0)
return -EINVAL;
- slot = &state->stack[spi];
+ slot = bpf_stack_slot(state, spi);
st = &slot->spilled_ptr;
if (!st->id)
@@ -1401,7 +1407,7 @@ static int copy_reference_state(struct bpf_verifier_state *dst, const struct bpf
static int copy_stack_state(struct bpf_func_state *dst, const struct bpf_func_state *src)
{
- size_t n = src->allocated_stack / BPF_REG_SIZE;
+ size_t n = bpf_stack_nr_slots(src);
dst->stack = copy_array(dst->stack, src->stack, n, sizeof(struct bpf_stack_state),
GFP_KERNEL_ACCOUNT);
@@ -1440,7 +1446,7 @@ static int resize_reference_state(struct bpf_verifier_state *state, size_t n)
*/
static int grow_stack_state(struct bpf_verifier_env *env, struct bpf_func_state *state, int size)
{
- size_t old_n = state->allocated_stack / BPF_REG_SIZE, n;
+ size_t old_n = bpf_stack_nr_slots(state), n;
/* The stack size is always a multiple of BPF_REG_SIZE. */
size = round_up(size, BPF_REG_SIZE);
@@ -3302,26 +3308,25 @@ static void mark_non_stack_access(struct bpf_verifier_env *env, int idx)
env->insn_aux_data[idx].non_stack_access = true;
}
+/* Layout of one packed linked register in the jump history, see linked_regs_pack() */
#define LR_FRAMENO_BITS 4
-#define LR_SPI_BITS 6
-#define LR_ENTRY_BITS (LR_SPI_BITS + LR_FRAMENO_BITS + 1)
-#define LR_SIZE_BITS 4
-#define LR_FRAMENO_MASK ((1ull << LR_FRAMENO_BITS) - 1)
-#define LR_SPI_MASK ((1ull << LR_SPI_BITS) - 1)
-#define LR_SIZE_MASK ((1ull << LR_SIZE_BITS) - 1)
-#define LR_SPI_OFF LR_FRAMENO_BITS
-#define LR_IS_REG_OFF (LR_SPI_BITS + LR_FRAMENO_BITS)
-#define LINKED_REGS_MAX 5
+#define LR_INDEX_BITS 11
+#define LR_FRAMENO_MASK ((1u << LR_FRAMENO_BITS) - 1)
+#define LR_IS_REG BIT(LR_FRAMENO_BITS)
+#define LR_INDEX_OFF (LR_FRAMENO_BITS + 1)
+#define LR_INDEX_MASK ((1u << LR_INDEX_BITS) - 1)
+#define LINKED_REGS_MAX BPF_LINKED_REGS_MAX
static_assert(MAX_CALL_FRAMES <= (1 << LR_FRAMENO_BITS));
-static_assert(LINKED_REGS_MAX < (1 << LR_SIZE_BITS));
-static_assert(LINKED_REGS_MAX * LR_ENTRY_BITS + LR_SIZE_BITS <= 64);
+static_assert(MAX_BPF_REG <= (1 << LR_INDEX_BITS));
+static_assert(MAX_BPF_STACK_SLOTS <= (1 << LR_INDEX_BITS));
+static_assert(LR_INDEX_OFF + LR_INDEX_BITS <= 16);
struct linked_reg {
u8 frameno;
union {
- u8 spi;
- u8 regno;
+ u16 spi;
+ u16 regno;
};
bool is_reg;
};
@@ -3340,48 +3345,34 @@ static struct linked_reg *linked_regs_push(struct linked_regs *s)
}
/*
- * Use u64 as a vector of 5 11-bit values, use first 4-bits to track
- * number of elements currently in stack.
- * Pack one history entry for linked registers as 11 bits in the following format:
- * - 4-bits frameno
- * - 6-bits spi_or_reg
- * - 1-bit is_reg
+ * Pack linked registers for a jump history entry, one u16 each:
+ * - 4 bits frameno
+ * - 1 bit is_reg
+ * - 11 bits register or stack slot index
*/
-static u64 linked_regs_pack(struct linked_regs *s)
+static void linked_regs_pack(const struct linked_regs *s, u16 *packed)
{
- u64 val = 0;
int i;
for (i = 0; i < s->cnt; ++i) {
- struct linked_reg *e = &s->entries[i];
- u64 tmp = 0;
-
- tmp |= e->frameno;
- tmp |= e->spi << LR_SPI_OFF;
- tmp |= (e->is_reg ? 1 : 0) << LR_IS_REG_OFF;
+ const struct linked_reg *e = &s->entries[i];
- val <<= LR_ENTRY_BITS;
- val |= tmp;
+ packed[i] = e->frameno | (e->is_reg ? LR_IS_REG : 0) | (e->spi << LR_INDEX_OFF);
}
- val <<= LR_SIZE_BITS;
- val |= s->cnt;
- return val;
}
-static void linked_regs_unpack(u64 val, struct linked_regs *s)
+static void linked_regs_unpack(const struct bpf_jmp_history_entry *hist, struct linked_regs *s)
{
int i;
- s->cnt = val & LR_SIZE_MASK;
- val >>= LR_SIZE_BITS;
-
+ s->cnt = hist->linked_regs_cnt;
for (i = 0; i < s->cnt; ++i) {
struct linked_reg *e = &s->entries[i];
+ u16 packed = hist->linked_regs[i];
- e->frameno = val & LR_FRAMENO_MASK;
- e->spi = (val >> LR_SPI_OFF) & LR_SPI_MASK;
- e->is_reg = (val >> LR_IS_REG_OFF) & 0x1;
- val >>= LR_ENTRY_BITS;
+ e->frameno = packed & LR_FRAMENO_MASK;
+ e->is_reg = packed & LR_IS_REG;
+ e->spi = (packed >> LR_INDEX_OFF) & LR_INDEX_MASK;
}
}
@@ -3423,10 +3414,10 @@ void bpf_bt_sync_linked_regs(struct backtrack_state *bt, struct bpf_jmp_history_
bool some_precise = false;
int i;
- if (!hist || hist->linked_regs == 0)
+ if (!hist || !hist->linked_regs_cnt)
return;
- linked_regs_unpack(hist->linked_regs, &linked_regs);
+ linked_regs_unpack(hist, &linked_regs);
for (i = 0; i < linked_regs.cnt; ++i) {
struct linked_reg *e = &linked_regs.entries[i];
@@ -3523,17 +3514,18 @@ static void save_register_state(struct bpf_verifier_env *env,
int spi, struct bpf_reg_state *reg,
int size)
{
+ struct bpf_stack_state *slot = bpf_stack_slot(state, spi);
int i;
- bpf_diag_mod_begin(env, &state->stack[spi].spilled_ptr, reg, BPF_DIAG_MOD_SPILL);
- state->stack[spi].spilled_ptr = *reg;
+ bpf_diag_mod_begin(env, &slot->spilled_ptr, reg, BPF_DIAG_MOD_SPILL);
+ slot->spilled_ptr = *reg;
for (i = BPF_REG_SIZE; i > BPF_REG_SIZE - size; i--)
- state->stack[spi].slot_type[i - 1] = STACK_SPILL;
+ slot->slot_type[i - 1] = STACK_SPILL;
/* size < 8 bytes spill */
for (; i; i--)
- mark_stack_slot_misc(env, &state->stack[spi].slot_type[i - 1]);
+ mark_stack_slot_misc(env, &slot->slot_type[i - 1]);
bpf_diag_mod_end(env);
}
@@ -3575,14 +3567,15 @@ static void check_fastcall_stack_contract(struct bpf_verifier_env *env,
static void scrub_special_slot(struct bpf_func_state *state, int spi)
{
+ struct bpf_stack_state *slot = bpf_stack_slot(state, spi);
int i;
/* regular write of data into stack destroys any spilled ptr */
- state->stack[spi].spilled_ptr.type = NOT_INIT;
+ slot->spilled_ptr.type = NOT_INIT;
/* Mark slots as STACK_MISC if they belonged to spilled ptr/dynptr/iter. */
- if (is_stack_slot_special(&state->stack[spi]))
+ if (is_stack_slot_special(slot))
for (i = 0; i < BPF_REG_SIZE; i++)
- scrub_spilled_slot(&state->stack[spi].slot_type[i]);
+ scrub_spilled_slot(&slot->slot_type[i]);
}
/* check_stack_{read,write}_fixed_off functions track spill/fill of registers,
@@ -3600,13 +3593,15 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env,
struct bpf_reg_state *reg = NULL;
int insn_flags = INSN_F_STACK_ACCESS;
int hist_spi = spi, hist_frame = state->frameno;
+ struct bpf_stack_state *ss = bpf_stack_slot(state, spi);
- /* caller checked that off % size == 0 and -MAX_BPF_STACK <= off < 0,
+ /*
+ * caller checked that off % size == 0 and -env->stack_limit <= off < 0,
* so it's aligned access and [off, off + size) are within stack limits
*/
if (!env->allow_ptr_leaks &&
- bpf_is_spilled_reg(&state->stack[spi]) &&
- !bpf_is_spilled_scalar_reg(&state->stack[spi]) &&
+ bpf_is_spilled_reg(ss) &&
+ !bpf_is_spilled_scalar_reg(ss) &&
size != BPF_REG_SIZE) {
const char *reason;
@@ -3628,7 +3623,7 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env,
bool sanitize = reg && is_pointer_regtype(reg->type);
for (i = 0; i < size; i++) {
- u8 type = state->stack[spi].slot_type[(slot - i) %
+ u8 type = ss->slot_type[(slot - i) %
BPF_REG_SIZE];
if (type != STACK_MISC && type != STACK_ZERO) {
@@ -3657,7 +3652,7 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env,
save_register_state(env, state, spi, reg, size);
/* Break the relation on a narrowing spill. */
if (!reg_value_fits)
- state->stack[spi].spilled_ptr.id = 0;
+ ss->spilled_ptr.id = 0;
} else if (!reg && !(off % BPF_REG_SIZE) && is_bpf_st_mem(insn) &&
env->bpf_capable) {
struct bpf_reg_state *tmp_reg = &env->fake_reg[0];
@@ -3681,8 +3676,8 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env,
} else {
u8 type = STACK_MISC;
- if (bpf_is_spilled_reg(&state->stack[spi]))
- bpf_diag_record_scrub(env, &state->stack[spi].spilled_ptr,
+ if (bpf_is_spilled_reg(ss))
+ bpf_diag_record_scrub(env, &ss->spilled_ptr,
BPF_DIAG_MOD_WRITE);
scrub_special_slot(state, spi);
@@ -3703,13 +3698,13 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env,
/* Mark slots affected by this stack write. */
for (i = 0; i < size; i++)
- state->stack[spi].slot_type[(slot - i) % BPF_REG_SIZE] = type;
+ ss->slot_type[(slot - i) % BPF_REG_SIZE] = type;
insn_flags = 0; /* not a register spill */
}
if (insn_flags)
return bpf_push_jmp_history(env, env->cur_state, insn_flags,
- hist_spi, hist_frame, 0);
+ hist_spi, hist_frame, NULL, 0);
return 0;
}
@@ -3774,7 +3769,7 @@ static int check_stack_write_var_off(struct bpf_verifier_env *env,
slot = -i - 1;
spi = slot / BPF_REG_SIZE;
- stype = &state->stack[spi].slot_type[slot % BPF_REG_SIZE];
+ stype = &bpf_stack_slot(state, spi)->slot_type[slot % BPF_REG_SIZE];
mark_stack_slot_scratched(env, spi);
if (!env->allow_ptr_leaks && *stype != STACK_MISC && *stype != STACK_ZERO) {
@@ -3798,8 +3793,8 @@ static int check_stack_write_var_off(struct bpf_verifier_env *env,
* maintain the spill type.
*/
if (writing_zero && *stype == STACK_SPILL &&
- bpf_is_spilled_scalar_reg(&state->stack[spi])) {
- struct bpf_reg_state *spill_reg = &state->stack[spi].spilled_ptr;
+ bpf_is_spilled_scalar_reg(bpf_stack_slot(state, spi))) {
+ struct bpf_reg_state *spill_reg = &bpf_stack_slot(state, spi)->spilled_ptr;
if (tnum_is_const(spill_reg->var_off) && spill_reg->var_off.value == 0) {
zero_used = true;
@@ -3870,23 +3865,22 @@ static int mark_reg_stack_read(struct bpf_verifier_env *env,
{
struct bpf_verifier_state *vstate = env->cur_state;
struct bpf_func_state *state = vstate->frame[vstate->curframe];
- u64 zero_spill_mask = 0;
int i, slot, spi;
u8 *stype;
- int zeros = 0;
+ int zeros = 0, zero_spills = 0;
for (i = min_off; i < max_off; i++) {
slot = -i - 1;
spi = slot / BPF_REG_SIZE;
mark_stack_slot_scratched(env, spi);
- stype = ptr_state->stack[spi].slot_type;
+ stype = bpf_stack_slot(ptr_state, spi)->slot_type;
if (stype[slot % BPF_REG_SIZE] == STACK_ZERO) {
zeros++;
continue;
}
if (stype[slot % BPF_REG_SIZE] == STACK_SPILL &&
- bpf_register_is_null(&ptr_state->stack[spi].spilled_ptr)) {
- zero_spill_mask |= 1ull << spi;
+ bpf_register_is_null(&bpf_stack_slot(ptr_state, spi)->spilled_ptr)) {
+ zero_spills++;
zeros++;
continue;
}
@@ -3897,8 +3891,14 @@ static int mark_reg_stack_read(struct bpf_verifier_env *env,
* so the whole register == const_zero.
*/
__mark_reg_const_zero(env, &state->regs[dst_regno]);
- if (zero_spill_mask) {
- bpf_bt_set_frame_slot_mask(&env->bt, ptr_state->frameno, zero_spill_mask);
+ if (zero_spills) {
+ for (i = min_off; i < max_off; i++) {
+ slot = -i - 1;
+ spi = slot / BPF_REG_SIZE;
+ stype = bpf_stack_slot(ptr_state, spi)->slot_type;
+ if (stype[slot % BPF_REG_SIZE] == STACK_SPILL)
+ bpf_bt_set_frame_slot(&env->bt, ptr_state->frameno, spi);
+ }
return mark_chain_precision_batch(env, env->cur_state);
}
} else {
@@ -3946,9 +3946,10 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env,
int err;
int insn_flags = INSN_F_STACK_ACCESS;
int hist_spi = spi, hist_frame = reg_state->frameno;
+ struct bpf_stack_state *ss = bpf_stack_slot(reg_state, spi);
- stype = reg_state->stack[spi].slot_type;
- reg = ®_state->stack[spi].spilled_ptr;
+ stype = ss->slot_type;
+ reg = &ss->spilled_ptr;
mark_stack_slot_scratched(env, spi);
check_fastcall_stack_contract(env, state, env->insn_idx, off);
@@ -3959,7 +3960,7 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env,
if (dst_regno >= 0)
bpf_diag_mod_begin(env, &state->regs[dst_regno], reg, BPF_DIAG_MOD_WRITE);
- if (bpf_is_spilled_reg(®_state->stack[spi])) {
+ if (bpf_is_spilled_reg(ss)) {
u8 spill_size = 1;
for (i = BPF_REG_SIZE - 1; i > 0 && stype[i - 1] == STACK_SPILL; i--)
@@ -4085,7 +4086,7 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env,
}
if (insn_flags)
return bpf_push_jmp_history(env, env->cur_state, insn_flags,
- hist_spi, hist_frame, 0);
+ hist_spi, hist_frame, NULL, 0);
return 0;
}
@@ -4275,7 +4276,7 @@ static int check_stack_arg_write(struct bpf_verifier_env *env, struct bpf_func_s
bpf_diag_mod_end(env);
state->no_stack_arg_load = true;
return bpf_push_jmp_history(env, env->cur_state,
- INSN_F_STACK_ARG_ACCESS, spi, 0, 0);
+ INSN_F_STACK_ARG_ACCESS, spi, 0, NULL, 0);
}
/*
@@ -4309,7 +4310,7 @@ static int check_stack_arg_read(struct bpf_verifier_env *env, struct bpf_func_st
cur->regs[dst_regno] = *arg;
bpf_diag_mod_end(env);
return bpf_push_jmp_history(env, env->cur_state,
- INSN_F_STACK_ARG_ACCESS, spi, 0, 0);
+ INSN_F_STACK_ARG_ACCESS, spi, 0, NULL, 0);
}
static int mark_stack_arg_precision(struct bpf_verifier_env *env, int arg_idx)
@@ -5448,7 +5449,7 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx,
if (subprog[idx].priv_stack_mode == PRIV_STACK_ADAPTIVE) {
if (subprog_depth > env->max_stack_depth)
env->max_stack_depth = subprog_depth;
- if (subprog_depth > MAX_BPF_STACK) {
+ if (subprog_depth > env->stack_limit) {
verbose(env, "stack size of subprog %d is %d. Too large\n",
idx, subprog_depth);
return -EACCES;
@@ -5457,7 +5458,7 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx,
depth += subprog_depth;
if (depth > env->max_stack_depth)
env->max_stack_depth = depth;
- if (depth > MAX_BPF_STACK) {
+ if (depth > env->stack_limit) {
total = 0;
for (tmp = idx; tmp >= 0; tmp = dinfo[tmp].caller)
total++;
@@ -6308,10 +6309,11 @@ static int check_ptr_to_map_access(struct bpf_verifier_env *env,
return 0;
}
-/* Check that the stack access at the given offset is within bounds. The
+/*
+ * Check that the stack access at the given offset is within bounds. The
* maximum valid offset is -1.
*
- * The minimum valid offset is -MAX_BPF_STACK for writes, and
+ * The minimum valid offset is -env->stack_limit for writes, and
* -state->allocated_stack for reads.
*/
static int check_stack_slot_within_bounds(struct bpf_verifier_env *env,
@@ -6322,7 +6324,7 @@ static int check_stack_slot_within_bounds(struct bpf_verifier_env *env,
int min_valid_off;
if (t == BPF_WRITE || env->allow_uninit_stack)
- min_valid_off = -MAX_BPF_STACK;
+ min_valid_off = -(int)env->stack_limit;
else
min_valid_off = -state->allocated_stack;
@@ -7048,6 +7050,7 @@ static int check_stack_range_initialized(
}
for (i = min_off; i < max_off + access_size; i++) {
+ struct bpf_stack_state *ss;
u8 *stype;
slot = -i - 1;
@@ -7057,7 +7060,8 @@ static int check_stack_range_initialized(
return -EFAULT;
}
- stype = &state->stack[spi].slot_type[slot % BPF_REG_SIZE];
+ ss = bpf_stack_slot(state, spi);
+ stype = &ss->slot_type[slot % BPF_REG_SIZE];
if (*stype == STACK_MISC)
goto mark;
if ((*stype == STACK_ZERO) ||
@@ -7069,13 +7073,13 @@ static int check_stack_range_initialized(
goto mark;
}
- if (bpf_is_spilled_reg(&state->stack[spi]) &&
- (state->stack[spi].spilled_ptr.type == SCALAR_VALUE ||
+ if (bpf_is_spilled_reg(ss) &&
+ (ss->spilled_ptr.type == SCALAR_VALUE ||
env->allow_ptr_leaks)) {
if (clobber) {
- __mark_reg_unknown(env, &state->stack[spi].spilled_ptr);
+ __mark_reg_unknown(env, &ss->spilled_ptr);
for (j = 0; j < BPF_REG_SIZE; j++)
- scrub_spilled_slot(&state->stack[spi].slot_type[j]);
+ scrub_spilled_slot(&ss->slot_type[j]);
}
goto mark;
}
@@ -7805,7 +7809,7 @@ static int process_dynptr_func(struct bpf_verifier_env *env, struct bpf_reg_stat
mark_stack_slots_scratched(env, spi, BPF_DYNPTR_NR_SLOTS);
- reg = &state->stack[spi].spilled_ptr;
+ reg = &bpf_stack_slot(state, spi)->spilled_ptr;
}
meta->dynptr.type = reg->dynptr.type;
@@ -7938,7 +7942,7 @@ static int process_iter_arg(struct bpf_verifier_env *env, struct bpf_reg_state *
/* remember meta->iter info for process_iter_next_call() */
meta->iter.spi = spi;
meta->iter.frameno = reg->frameno;
- update_ref_obj(&meta->ref_obj, &state->stack[spi].spilled_ptr);
+ update_ref_obj(&meta->ref_obj, &bpf_stack_slot(state, spi)->spilled_ptr);
if (is_iter_destroy_kfunc(meta)) {
err = unmark_stack_slots_iter(env, reg, nr_slots);
@@ -8015,16 +8019,15 @@ static int widen_imprecise_scalars(struct bpf_verifier_env *env,
&fold->regs[i],
&fcur->regs[i]);
- num_slots = min(fold->allocated_stack / BPF_REG_SIZE,
- fcur->allocated_stack / BPF_REG_SIZE);
+ num_slots = min(bpf_stack_nr_slots(fold), bpf_stack_nr_slots(fcur));
for (i = 0; i < num_slots; i++) {
- if (!bpf_is_spilled_reg(&fold->stack[i]) ||
- !bpf_is_spilled_reg(&fcur->stack[i]))
+ if (!bpf_is_spilled_reg(bpf_stack_slot(fold, i)) ||
+ !bpf_is_spilled_reg(bpf_stack_slot(fcur, i)))
continue;
maybe_widen_reg(env,
- &fold->stack[i].spilled_ptr,
- &fcur->stack[i].spilled_ptr);
+ &bpf_stack_slot(fold, i)->spilled_ptr,
+ &bpf_stack_slot(fcur, i)->spilled_ptr);
}
}
return 0;
@@ -8036,7 +8039,7 @@ static struct bpf_reg_state *get_iter_from_state(struct bpf_verifier_state *cur_
int iter_frameno = meta->iter.frameno;
int iter_spi = meta->iter.spi;
- return &cur_st->frame[iter_frameno]->stack[iter_spi].spilled_ptr;
+ return &bpf_stack_slot(cur_st->frame[iter_frameno], iter_spi)->spilled_ptr;
}
/* process_iter_next_call() is called when verifier gets to iterator's next
@@ -8841,7 +8844,7 @@ static int get_constant_map_key(struct bpf_verifier_env *env,
slot = -stack_off - 1;
spi = slot / BPF_REG_SIZE;
off = slot % BPF_REG_SIZE;
- stype = state->stack[spi].slot_type;
+ stype = bpf_stack_slot(state, spi)->slot_type;
/* First handle precisely tracked STACK_ZERO */
for (i = off; i >= 0 && stype[i] == STACK_ZERO; i--)
@@ -8852,14 +8855,14 @@ static int get_constant_map_key(struct bpf_verifier_env *env,
}
/* Check that stack contains a scalar spill of expected size */
- if (!bpf_is_spilled_scalar_reg(&state->stack[spi]))
+ if (!bpf_is_spilled_scalar_reg(bpf_stack_slot(state, spi)))
return -EOPNOTSUPP;
for (i = off; i >= 0 && stype[i] == STACK_SPILL; i--)
spill_size++;
if (spill_size != key_size)
return -EOPNOTSUPP;
- reg = &state->stack[spi].spilled_ptr;
+ reg = &bpf_stack_slot(state, spi)->spilled_ptr;
if (!tnum_is_const(reg->var_off))
/* Stack value not statically known */
return -EOPNOTSUPP;
@@ -10135,8 +10138,9 @@ static int idstack_push(struct bpf_idmap *idmap, u32 id)
if (idmap->map[i].old == id)
return 0;
- if (WARN_ON_ONCE(idmap->cnt >= BPF_ID_MAP_SIZE))
- return -EFAULT;
+ if (!bpf_id_scratch_reserve((void **)&idmap->map, &idmap->cap, idmap->cnt,
+ sizeof(*idmap->map)))
+ return -ENOMEM;
idmap->map[idmap->cnt++].old = id;
return 0;
@@ -13893,11 +13897,11 @@ s64 bpf_helper_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn
}
/*
* Size arg is const on each path but differs across merged
- * paths. MAX_BPF_STACK is a safe upper bound for reads.
+ * paths. Reads may extend anywhere up to the frame top.
*/
if (full_write)
return 0;
- return MAX_BPF_STACK;
+ return S64_MIN;
}
return S64_MIN;
case ARG_PTR_TO_DYNPTR:
@@ -13983,7 +13987,8 @@ s64 bpf_kfunc_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn *
size = (s64)aux->const_reg_vals[size_reg];
goto out;
}
- return MAX_BPF_STACK;
+ /* Unknown size: the read may extend anywhere up to the frame top. */
+ return S64_MIN;
}
/* fixed-size pointed-to type: resolve via BTF */
@@ -14725,7 +14730,8 @@ enum {
REASON_STACK = -5,
};
-static int retrieve_ptr_limit(const struct bpf_reg_state *ptr_reg,
+static int retrieve_ptr_limit(const struct bpf_verifier_env *env,
+ const struct bpf_reg_state *ptr_reg,
u32 *alu_limit, bool mask_to_left)
{
u32 max = 0, ptr_limit = 0;
@@ -14737,7 +14743,7 @@ static int retrieve_ptr_limit(const struct bpf_reg_state *ptr_reg,
* offset where we would need to deal with min/max bounds is
* currently prohibited for unprivileged.
*/
- max = MAX_BPF_STACK + mask_to_left;
+ max = env->stack_limit + mask_to_left;
ptr_limit = -ptr_reg->var_off.value;
break;
case PTR_TO_MAP_VALUE:
@@ -14857,7 +14863,7 @@ static int sanitize_ptr_alu(struct bpf_verifier_env *env,
(opcode == BPF_SUB && !off_is_neg);
}
- err = retrieve_ptr_limit(ptr_reg, &alu_limit, info->mask_to_left);
+ err = retrieve_ptr_limit(env, ptr_reg, &alu_limit, info->mask_to_left);
if (err < 0)
return err;
@@ -14987,7 +14993,7 @@ static int check_stack_access_for_ptr_arithmetic(
return -EACCES;
}
- if (off >= 0 || off < -MAX_BPF_STACK) {
+ if (off >= 0 || off < -(int)env->stack_limit) {
verbose(env, "R%d stack pointer arithmetic goes out of range, "
"prohibited for !root; off=%d\n", regno, off);
return -EACCES;
@@ -17322,10 +17328,10 @@ static void collect_linked_regs(struct bpf_verifier_env *env,
reg = &func->regs[j];
__collect_linked_regs(linked_regs, reg, id, i, j, true);
}
- for (j = 0; j < func->allocated_stack / BPF_REG_SIZE; j++) {
- if (!bpf_is_spilled_reg(&func->stack[j]))
+ for (j = 0; j < bpf_stack_nr_slots(func); j++) {
+ if (!bpf_is_spilled_reg(bpf_stack_slot(func, j)))
continue;
- reg = &func->stack[j].spilled_ptr;
+ reg = &bpf_stack_slot(func, j)->spilled_ptr;
__collect_linked_regs(linked_regs, reg, id, i, j, false);
}
}
@@ -17345,7 +17351,7 @@ static void sync_linked_regs(struct bpf_verifier_env *env, struct bpf_verifier_s
for (i = 0; i < linked_regs->cnt; ++i) {
e = &linked_regs->entries[i];
reg = e->is_reg ? &vstate->frame[e->frameno]->regs[e->regno]
- : &vstate->frame[e->frameno]->stack[e->spi].spilled_ptr;
+ : &bpf_stack_slot(vstate->frame[e->frameno], e->spi)->spilled_ptr;
if (reg->type != SCALAR_VALUE || reg == known_reg)
continue;
if ((reg->id & ~BPF_ADD_CONST) != (known_reg->id & ~BPF_ADD_CONST))
@@ -17463,7 +17469,7 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env,
}
if (insn_flags) {
- err = bpf_push_jmp_history(env, this_branch, insn_flags, 0, 0, 0);
+ err = bpf_push_jmp_history(env, this_branch, insn_flags, 0, 0, NULL, 0);
if (err)
return err;
}
@@ -17533,7 +17539,10 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env,
* if parent state is created.
*/
if (linked_regs.cnt > 1) {
- err = bpf_push_jmp_history(env, this_branch, 0, 0, 0, linked_regs_pack(&linked_regs));
+ u16 packed[LINKED_REGS_MAX];
+
+ linked_regs_pack(&linked_regs, packed);
+ err = bpf_push_jmp_history(env, this_branch, 0, 0, 0, packed, linked_regs.cnt);
if (err)
return err;
}
@@ -18433,12 +18442,13 @@ static void idset_cnt_inc(struct bpf_idset *idset, u32 id)
return;
}
}
- /* New id */
- if (idset->num_ids < BPF_ID_MAP_SIZE) {
- idset->entries[idset->num_ids].id = id;
- idset->entries[idset->num_ids].cnt = 1;
- idset->num_ids++;
- }
+ /* New id; one that cannot be recorded counts as shared and is kept */
+ if (!bpf_id_scratch_reserve((void **)&idset->entries, &idset->cap, idset->num_ids,
+ sizeof(*idset->entries)))
+ return;
+ idset->entries[idset->num_ids].id = id;
+ idset->entries[idset->num_ids].cnt = 1;
+ idset->num_ids++;
}
/* Find id in idset and return its count, or 0 if not found */
@@ -18927,7 +18937,7 @@ static int do_check(struct bpf_verifier_env *env)
}
if (bpf_is_jmp_point(env, env->insn_idx)) {
- err = bpf_push_jmp_history(env, state, 0, 0, 0, 0);
+ err = bpf_push_jmp_history(env, state, 0, 0, 0, NULL, 0);
if (err)
return err;
}
@@ -21686,6 +21696,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
env->bt.env = env;
env->prog = *prog;
env->ops = bpf_verifier_ops[env->prog->type];
+ env->stack_limit = bpf_prog_stack_limit(env->prog);
env->allow_ptr_leaks = bpf_allow_ptr_leaks(env->prog->aux->token);
env->allow_uninit_stack = bpf_allow_uninit_stack(env->prog->aux->token);
@@ -21998,6 +22009,8 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
kvfree(env->scc_info);
kvfree(env->succ);
kvfree(env->gotox_tmp_buf);
+ kfree(env->idmap_scratch.map);
+ kfree(env->idset_scratch.entries);
bpf_diag_free(env);
kvfree(env);
return ret;
diff --git a/tools/testing/selftests/bpf/prog_tests/struct_ops_private_stack.c b/tools/testing/selftests/bpf/prog_tests/struct_ops_private_stack.c
index 98db9bafa44b2..2b3ec2b790918 100644
--- a/tools/testing/selftests/bpf/prog_tests/struct_ops_private_stack.c
+++ b/tools/testing/selftests/bpf/prog_tests/struct_ops_private_stack.c
@@ -4,6 +4,7 @@
#include "struct_ops_private_stack.skel.h"
#include "struct_ops_private_stack_fail.skel.h"
#include "struct_ops_private_stack_recur.skel.h"
+#include "struct_ops_private_stack_large.skel.h"
#if defined(__x86_64__) || defined(__aarch64__) || defined(__powerpc64__)
static void test_private_stack(void)
@@ -78,6 +79,34 @@ static void test_private_stack_recur(void)
struct_ops_private_stack_recur__destroy(skel);
}
+/* Two frames of 2 KiB each on the private stack */
+static void test_private_stack_large(void)
+{
+ struct struct_ops_private_stack_large *skel;
+ struct bpf_link *link;
+
+ if (!is_large_stack_supported()) {
+ test__skip();
+ return;
+ }
+
+ skel = struct_ops_private_stack_large__open_and_load();
+ if (!ASSERT_OK_PTR(skel, "struct_ops_private_stack_large__open_and_load"))
+ return;
+
+ link = bpf_map__attach_struct_ops(skel->maps.testmod_1);
+ if (!ASSERT_OK_PTR(link, "attach_struct_ops"))
+ goto cleanup;
+
+ ASSERT_OK(trigger_module_test_read(256), "trigger_read");
+
+ ASSERT_EQ(skel->bss->val, 100 + 30 + 12, "val");
+
+ bpf_link__destroy(link);
+cleanup:
+ struct_ops_private_stack_large__destroy(skel);
+}
+
static void __test_struct_ops_private_stack(void)
{
if (test__start_subtest("private_stack"))
@@ -86,6 +115,8 @@ static void __test_struct_ops_private_stack(void)
test_private_stack_fail();
if (test__start_subtest("private_stack_recur"))
test_private_stack_recur();
+ if (test__start_subtest("private_stack_large"))
+ test_private_stack_large();
}
#else
static void __test_struct_ops_private_stack(void)
diff --git a/tools/testing/selftests/bpf/prog_tests/tailcalls.c b/tools/testing/selftests/bpf/prog_tests/tailcalls.c
index c5c9d6c359bb0..c5e3f198ca7f6 100644
--- a/tools/testing/selftests/bpf/prog_tests/tailcalls.c
+++ b/tools/testing/selftests/bpf/prog_tests/tailcalls.c
@@ -9,6 +9,7 @@
#include "tc_bpf2bpf.skel.h"
#include "tailcall_fail.skel.h"
#include "tailcall_cgrp_storage_owner.skel.h"
+#include "tailcall_large_stack.skel.h"
#include "tailcall_cgrp_storage_no_storage.skel.h"
#include "tailcall_cgrp_storage.skel.h"
#include "tailcall_sleepable.skel.h"
@@ -1953,6 +1954,45 @@ static void test_tailcall_bpf2bpf_fexit_links(void)
tailcall_bpf2bpf2__destroy(skel_tc);
}
+/*
+ * test_tailcall_large_stack runs a tail call made from a subprog with a 1536
+ * byte frame, under a 240-byte caller, into a program with a 2 KiB frame:
+ *
+ * entry (240) --call-> subprog_tail (1536) --tailcall-> classifier_0 (2048)
+ */
+static void test_tailcall_large_stack(void)
+{
+ struct tailcall_large_stack *skel;
+ int err, prog_fd, map_fd, key = 0;
+ char buff[128] = {};
+ LIBBPF_OPTS(bpf_test_run_opts, topts,
+ .data_in = buff,
+ .data_size_in = sizeof(buff),
+ .repeat = 1,
+ );
+
+ if (!is_large_stack_supported()) {
+ test__skip();
+ return;
+ }
+
+ skel = tailcall_large_stack__open_and_load();
+ if (!ASSERT_OK_PTR(skel, "tailcall_large_stack__open_and_load"))
+ return;
+
+ prog_fd = bpf_program__fd(skel->progs.classifier_0);
+ map_fd = bpf_map__fd(skel->maps.jmp_table);
+ err = bpf_map_update_elem(map_fd, &key, &prog_fd, BPF_ANY);
+ if (!ASSERT_OK(err, "update jmp_table"))
+ goto out;
+
+ err = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.entry), &topts);
+ ASSERT_OK(err, "test_run");
+ ASSERT_EQ(topts.retval, 42 + 7, "retval");
+out:
+ tailcall_large_stack__destroy(skel);
+}
+
void test_tailcalls(void)
{
if (test__start_subtest("tailcall_1"))
@@ -2022,4 +2062,6 @@ void test_tailcalls(void)
test_tailcall_callback();
if (test__start_subtest("tailcall_bpf2bpf_fexit_links"))
test_tailcall_bpf2bpf_fexit_links();
+ if (test__start_subtest("tailcall_large_stack"))
+ test_tailcall_large_stack();
}
diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c
index 4f1e1c1cd5ab3..ced8a2f1c89c3 100644
--- a/tools/testing/selftests/bpf/prog_tests/verifier.c
+++ b/tools/testing/selftests/bpf/prog_tests/verifier.c
@@ -59,6 +59,7 @@
#include "verifier_kfunc_uninit.skel.h"
#include "verifier_kfunc_uninit_multi.skel.h"
#include "verifier_ld_ind.skel.h"
+#include "verifier_large_stack.skel.h"
#include "verifier_ldsx.skel.h"
#include "verifier_leak_ptr.skel.h"
#include "verifier_linked_scalars.skel.h"
@@ -229,6 +230,7 @@ void test_verifier_kfunc_uninit(void) { RUN_TESTS(verifier_kfunc_uninit)
void test_verifier_kfunc_uninit_multi(void) { RUN_TESTS(verifier_kfunc_uninit_multi); }
void test_verifier_load_acquire(void) { RUN(verifier_load_acquire); }
void test_verifier_ld_ind(void) { RUN(verifier_ld_ind); }
+void test_verifier_large_stack(void) { RUN(verifier_large_stack); }
void test_verifier_ldsx(void) { RUN(verifier_ldsx); }
void test_verifier_leak_ptr(void) { RUN(verifier_leak_ptr); }
void test_verifier_linked_scalars(void) { RUN(verifier_linked_scalars); }
diff --git a/tools/testing/selftests/bpf/progs/async_stack_depth.c b/tools/testing/selftests/bpf/progs/async_stack_depth.c
index 36734683acbdb..9cd874a90b39a 100644
--- a/tools/testing/selftests/bpf/progs/async_stack_depth.c
+++ b/tools/testing/selftests/bpf/progs/async_stack_depth.c
@@ -29,7 +29,49 @@ static int bad_timer_cb(void *map, int *key, struct bpf_timer *timer)
return buf[255] + timer_cb(NULL, NULL, NULL);
}
+/*
+ * The same shapes scaled to the 2 KiB budget of JITs with large stacks. The
+ * compiler caps a single function at 512 bytes, so the depth comes from a
+ * chain of 480-byte frames.
+ */
+__attribute__((noinline))
+static int timer_cb_large_0(void *map, int *key, struct bpf_timer *timer)
+{
+ volatile char buf[480] = {};
+ return buf[69];
+}
+
+__attribute__((noinline))
+static int timer_cb_large_1(void *map, int *key, struct bpf_timer *timer)
+{
+ volatile char buf[480] = {};
+ return buf[69] + timer_cb_large_0(map, key, timer);
+}
+
+__attribute__((noinline))
+static int timer_cb_large_2(void *map, int *key, struct bpf_timer *timer)
+{
+ volatile char buf[480] = {};
+ return buf[69] + timer_cb_large_1(map, key, timer);
+}
+
+__attribute__((noinline))
+static int timer_cb_large_3(void *map, int *key, struct bpf_timer *timer)
+{
+ volatile char buf[480] = {};
+ return buf[69] + timer_cb_large_2(map, key, timer);
+}
+
+/* 5 * 480 = 2400 bytes on its own */
+__attribute__((noinline))
+static int bad_timer_cb_large(void *map, int *key, struct bpf_timer *timer)
+{
+ volatile char buf[480] = {};
+ return buf[255] + timer_cb_large_3(map, key, timer);
+}
+
SEC("tc")
+__load_if_no_large_stack()
__failure __msg("combined stack size of 2 calls is")
int pseudo_call_check(struct __sk_buff *ctx)
{
@@ -44,7 +86,25 @@ int pseudo_call_check(struct __sk_buff *ctx)
return bpf_timer_set_callback(&elem->timer, timer_cb) + buf[0];
}
+/* main plus the four frames under timer_cb_large_3: 2400 bytes */
SEC("tc")
+__load_if_large_stack()
+__failure __msg("combined stack size of 5 calls is")
+int pseudo_call_check_large(struct __sk_buff *ctx)
+{
+ struct hmap_elem *elem;
+ volatile char buf[480] = {};
+
+ elem = bpf_map_lookup_elem(&hmap, &(int){0});
+ if (!elem)
+ return 0;
+
+ timer_cb_large_3(NULL, NULL, NULL);
+ return bpf_timer_set_callback(&elem->timer, timer_cb_large_3) + buf[0];
+}
+
+SEC("tc")
+__load_if_no_large_stack()
__failure __msg("combined stack size of 2 calls is")
int async_call_root_check(struct __sk_buff *ctx)
{
@@ -58,4 +118,19 @@ int async_call_root_check(struct __sk_buff *ctx)
return bpf_timer_set_callback(&elem->timer, bad_timer_cb) + buf[0];
}
+SEC("tc")
+__load_if_large_stack()
+__failure __msg("combined stack size of 5 calls is")
+int async_call_root_check_large(struct __sk_buff *ctx)
+{
+ struct hmap_elem *elem;
+ volatile char buf[480] = {};
+
+ elem = bpf_map_lookup_elem(&hmap, &(int){0});
+ if (!elem)
+ return 0;
+
+ return bpf_timer_set_callback(&elem->timer, bad_timer_cb_large) + buf[0];
+}
+
char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/bpf_misc.h b/tools/testing/selftests/bpf/progs/bpf_misc.h
index 2ced1d751acea..f3dbc3b59bff2 100644
--- a/tools/testing/selftests/bpf/progs/bpf_misc.h
+++ b/tools/testing/selftests/bpf/progs/bpf_misc.h
@@ -175,6 +175,9 @@
#define __prepare_priv __test_tag("test_prepare_priv")
#define __load_if_JITed() __test_tag("load_mode=jited")
#define __load_if_no_JITed() __test_tag("load_mode=no_jited")
+/* Whether programs may use more than 512 bytes of stack on this kernel and JIT */
+#define __load_if_large_stack() __test_tag("stack_mode=large")
+#define __load_if_no_large_stack() __test_tag("stack_mode=small")
#define __stderr(msg) __test_tag("test_expect_stderr=" msg)
#define __stderr_unpriv(msg) __test_tag("test_expect_stderr_unpriv=" msg)
#define __stdout(msg) __test_tag("test_expect_stdout=" msg)
diff --git a/tools/testing/selftests/bpf/progs/struct_ops_private_stack_fail.c b/tools/testing/selftests/bpf/progs/struct_ops_private_stack_fail.c
index 1442728f56046..c8cb35b37867b 100644
--- a/tools/testing/selftests/bpf/progs/struct_ops_private_stack_fail.c
+++ b/tools/testing/selftests/bpf/progs/struct_ops_private_stack_fail.c
@@ -4,6 +4,7 @@
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
#include "../test_kmods/bpf_testmod.h"
+#include "bpf_misc.h"
char _license[] SEC("license") = "GPL";
@@ -25,6 +26,44 @@ __noinline static int subprog1(int *a)
return subprog2(a, b);
}
+/*
+ * A chain of 480-byte frames under test_2, so that its call chain exceeds
+ * the 2 KiB budget of JITs with large stacks as well as the 512 bytes
+ * allowed elsewhere. The compiler caps a single function at 512 bytes, and
+ * the buffers are volatile so that it cannot shrink them.
+ */
+__noinline static int subprog_deep4(int *a)
+{
+ volatile char b[480] = {};
+
+ __sink(b[479]);
+ return a[10] + b[20];
+}
+
+__noinline static int subprog_deep3(int *a)
+{
+ volatile char b[480] = {};
+
+ __sink(b[479]);
+ return subprog_deep4(a) + b[20];
+}
+
+__noinline static int subprog_deep2(int *a)
+{
+ volatile char b[480] = {};
+
+ __sink(b[479]);
+ return subprog_deep3(a) + b[20];
+}
+
+__noinline static int subprog_deep1(int *a)
+{
+ volatile char b[480] = {};
+
+ __sink(b[479]);
+ return subprog_deep2(a) + b[20];
+}
+
SEC("struct_ops")
int BPF_PROG(test_1)
@@ -41,11 +80,13 @@ int BPF_PROG(test_1)
SEC("struct_ops")
int BPF_PROG(test_2)
{
- /* stack size 400 bytes */
- int a[100] = {};
+ /* stack size 476 bytes, over 2 KiB with the four 480-byte deep subprogs */
+ volatile char buf[376] = {};
+ int a[25] = {};
+ __sink(buf[375]);
a[10] = 3;
- val_j = subprog1(a);
+ val_j = subprog1(a) + subprog_deep1(a);
return 0;
}
diff --git a/tools/testing/selftests/bpf/progs/struct_ops_private_stack_large.c b/tools/testing/selftests/bpf/progs/struct_ops_private_stack_large.c
new file mode 100644
index 0000000000000..94a25a2cff6ec
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/struct_ops_private_stack_large.c
@@ -0,0 +1,51 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+#include "../test_kmods/bpf_testmod.h"
+#include "bpf_misc.h"
+
+char _license[] SEC("license") = "GPL";
+
+long val;
+
+/* On a private stack every frame gets the whole 2 KiB budget. */
+__used __naked
+static long frame_2048_leaf(void)
+{
+ asm volatile (" \
+ r1 = 30; \
+ *(u64 *)(r10 - 2048) = r1; \
+ r1 = 12; \
+ *(u64 *)(r10 - 8) = r1; \
+ r0 = *(u64 *)(r10 - 2048); \
+ r1 = *(u64 *)(r10 - 8); \
+ r0 += r1; \
+ exit; \
+" ::: __clobber_all);
+}
+
+/* test_1 is the member bpf_testmod requests a private stack for */
+SEC("struct_ops")
+__naked int test_1(void)
+{
+ asm volatile (" \
+ r1 = 100; \
+ *(u64 *)(r10 - 2048) = r1; \
+ call frame_2048_leaf; \
+ r1 = *(u64 *)(r10 - 2048); \
+ r0 += r1; \
+ r1 = %[val] ll; \
+ *(u64 *)(r1 + 0) = r0; \
+ r0 = 0; \
+ exit; \
+" :
+ : __imm_addr(val)
+ : __clobber_all);
+}
+
+SEC(".struct_ops")
+struct bpf_testmod_ops3 testmod_1 = {
+ .test_1 = (void *)test_1,
+};
diff --git a/tools/testing/selftests/bpf/progs/tailcall_large_stack.c b/tools/testing/selftests/bpf/progs/tailcall_large_stack.c
new file mode 100644
index 0000000000000..977197dac5d37
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/tailcall_large_stack.c
@@ -0,0 +1,62 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+struct {
+ __uint(type, BPF_MAP_TYPE_PROG_ARRAY);
+ __uint(max_entries, 1);
+ __uint(key_size, sizeof(__u32));
+ __uint(value_size, sizeof(__u32));
+} jmp_table SEC(".maps");
+
+/* The tail call target sets up a 2 KiB frame of its own and uses both of its ends. */
+SEC("tc")
+__naked int classifier_0(void)
+{
+ asm volatile (" \
+ r1 = 42; \
+ *(u64 *)(r10 - 2048) = r1; \
+ r1 = 7; \
+ *(u64 *)(r10 - 8) = r1; \
+ r0 = *(u64 *)(r10 - 2048); \
+ r1 = *(u64 *)(r10 - 8); \
+ r0 += r1; \
+ exit; \
+" ::: __clobber_all);
+}
+
+/*
+ * The frame of the subprog doing the tail call is unwound by it, so it may be
+ * large; only the frames of its callers stay behind and are limited to 256
+ * bytes in total. Returns 1 when the tail call falls through.
+ */
+__used __naked
+static int subprog_tail(void)
+{
+ asm volatile (" \
+ r2 = 1; \
+ *(u64 *)(r10 - 1536) = r2; \
+ r2 = %[jmp_table] ll; \
+ r3 = 0; \
+ call %[bpf_tail_call]; \
+ r0 = 1; \
+ exit; \
+" :
+ : __imm(bpf_tail_call),
+ __imm_addr(jmp_table)
+ : __clobber_all);
+}
+
+SEC("tc")
+__naked int entry(void)
+{
+ asm volatile (" \
+ r2 = 2; \
+ *(u64 *)(r10 - 240) = r2; \
+ call subprog_tail; \
+ exit; \
+" ::: __clobber_all);
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/test_global_func1.c b/tools/testing/selftests/bpf/progs/test_global_func1.c
index fc69ff18880d5..f0eca282e0d43 100644
--- a/tools/testing/selftests/bpf/progs/test_global_func1.c
+++ b/tools/testing/selftests/bpf/progs/test_global_func1.c
@@ -48,8 +48,73 @@ int f3(int val, struct __sk_buff *skb, int var)
}
SEC("tc")
+__load_if_no_large_stack()
__failure __msg("combined stack size of 3 calls is")
int global_func1(struct __sk_buff *skb)
{
return f0(1, skb) + f1(skb) + f2(2, skb) + f3(3, skb, 4);
}
+
+/*
+ * A chain of five frames that stay under 512 bytes each but add up to more
+ * than the 2 KiB budget of JITs with large stacks; the chain also exceeds
+ * 512 bytes after two frames, so it is rejected everywhere.
+ */
+#define MAX_STACK_LARGE 480
+
+__attribute__ ((noinline))
+int g0(struct __sk_buff *skb)
+{
+ volatile char buf[MAX_STACK_LARGE] = {};
+
+ __sink(buf[MAX_STACK_LARGE - 1]);
+
+ return skb->len;
+}
+
+__attribute__ ((noinline))
+int g1(struct __sk_buff *skb)
+{
+ volatile char buf[MAX_STACK_LARGE] = {};
+
+ __sink(buf[MAX_STACK_LARGE - 1]);
+
+ return g0(skb) + skb->len;
+}
+
+__attribute__ ((noinline))
+int g2(struct __sk_buff *skb)
+{
+ volatile char buf[MAX_STACK_LARGE] = {};
+
+ __sink(buf[MAX_STACK_LARGE - 1]);
+
+ return g1(skb) + skb->len;
+}
+
+__attribute__ ((noinline))
+int g3(struct __sk_buff *skb)
+{
+ volatile char buf[MAX_STACK_LARGE] = {};
+
+ __sink(buf[MAX_STACK_LARGE - 1]);
+
+ return g2(skb) + skb->len;
+}
+
+__attribute__ ((noinline))
+int g4(struct __sk_buff *skb)
+{
+ volatile char buf[MAX_STACK_LARGE] = {};
+
+ __sink(buf[MAX_STACK_LARGE - 1]);
+
+ return g3(skb) + skb->len;
+}
+
+SEC("tc")
+__failure __msg("combined stack size of {{[0-9]+}} calls is")
+int global_func1_deep(struct __sk_buff *skb)
+{
+ return g4(skb);
+}
diff --git a/tools/testing/selftests/bpf/progs/test_global_func_deep_stack.c b/tools/testing/selftests/bpf/progs/test_global_func_deep_stack.c
index 1b634b543b629..edb8a223a3cb2 100644
--- a/tools/testing/selftests/bpf/progs/test_global_func_deep_stack.c
+++ b/tools/testing/selftests/bpf/progs/test_global_func_deep_stack.c
@@ -67,12 +67,30 @@ int XCAT(f, n)(unsigned long a) \
#define F_31 F_30 FN(31, 30)
#define F_32 F_31 FN(32, 31)
+/* Same, with a 480-byte frame, to exceed the 2 KiB budget of large stacks. */
+#define FNB(n, prev) \
+__attribute__((noinline)) \
+int XCAT(f, n)(unsigned long a) \
+{ \
+ volatile char buf[480] = {}; \
+ volatile long b = XCAT(f, prev)(a - 1); \
+ if (!b) \
+ return 0; \
+ return b + buf[479] + 1; \
+}
+
+#define F_33 F_32 FNB(33, 32)
+#define F_34 F_33 FNB(34, 33)
+#define F_35 F_34 FNB(35, 34)
+#define F_36 F_35 FNB(36, 35)
+#define F_37 F_36 FNB(37, 36)
+
#define CAT2(a, b) a ## b
#define XCAT2(a, b) CAT2(a, b)
#define F(n) XCAT2(F_, n)
-F(32)
+F(37)
/* Ensure that even 32 levels deep, the function verifies. */
SEC("syscall")
@@ -88,8 +106,21 @@ int global_func_deep_stack_success(struct __sk_buff *skb)
* the size.
*/
SEC("syscall")
+__load_if_no_large_stack()
__failure __msg("combined stack size of 34 calls")
int global_func_deep_stack_fail(struct __sk_buff *skb)
{
return f32(123);
}
+
+/*
+ * Five 480-byte frames on top of the chain: 5 * 480 + 33 * 16 = 2928 bytes,
+ * more than the 2 KiB budget of JITs with large stacks, and more than 512
+ * bytes after the second frame everywhere else.
+ */
+SEC("syscall")
+__failure __msg("combined stack size of {{[0-9]+}} calls")
+int global_func_deep_stack_fail_large(struct __sk_buff *skb)
+{
+ return f37(123);
+}
diff --git a/tools/testing/selftests/bpf/progs/verifier_large_stack.c b/tools/testing/selftests/bpf/progs/verifier_large_stack.c
new file mode 100644
index 0000000000000..2d4c81a3f0cb9
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_large_stack.c
@@ -0,0 +1,377 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+/*
+ * Programs may use MAX_BPF_STACK_JIT (2 KiB) of stack on JITs that support
+ * large stacks, combined over a call chain, with no separate limit on a
+ * single frame. Interpreted programs and other JITs keep 512 bytes.
+ */
+
+SEC("socket")
+__description("single frame of 2048 bytes")
+__load_if_large_stack()
+__success __success_unpriv __retval(42)
+__naked void single_frame_2048(void)
+{
+ asm volatile (" \
+ r1 = r10; \
+ r1 += -2048; \
+ r0 = 42; \
+ *(u64*)(r1 + 0) = r0; \
+ r0 = *(u64*)(r1 + 0); \
+ exit; \
+" ::: __clobber_all);
+}
+
+SEC("socket")
+__description("single frame of 2048 bytes without large stack support")
+__load_if_no_large_stack()
+__failure __msg("invalid write to stack R1 off=-2048 size=8")
+__naked void single_frame_2048_no_large_stack(void)
+{
+ asm volatile (" \
+ r1 = r10; \
+ r1 += -2048; \
+ r0 = 42; \
+ *(u64*)(r1 + 0) = r0; \
+ exit; \
+" ::: __clobber_all);
+}
+
+__used __naked
+static void frame_512_leaf(void)
+{
+ asm volatile (" \
+ r1 = 1; \
+ *(u64 *)(r10 - 512) = r1; \
+ exit; \
+" ::: __clobber_all);
+}
+
+__used __naked
+static void frame_512_depth_2(void)
+{
+ asm volatile (" \
+ r1 = 2; \
+ *(u64 *)(r10 - 512) = r1; \
+ call frame_512_leaf; \
+ exit; \
+" ::: __clobber_all);
+}
+
+__used __naked
+static void frame_512_depth_3(void)
+{
+ asm volatile (" \
+ r1 = 3; \
+ *(u64 *)(r10 - 512) = r1; \
+ call frame_512_depth_2; \
+ exit; \
+" ::: __clobber_all);
+}
+
+__used __naked
+static void frame_512_depth_4(void)
+{
+ asm volatile (" \
+ r1 = 4; \
+ *(u64 *)(r10 - 512) = r1; \
+ call frame_512_depth_3; \
+ exit; \
+" ::: __clobber_all);
+}
+
+SEC("socket")
+__description("four frames of 512 bytes fit the 2 KiB budget")
+__load_if_large_stack()
+__success __log_level(4) __msg("stack depth max 2048")
+__naked void four_frames_of_512(void)
+{
+ asm volatile (" \
+ call frame_512_depth_4; \
+ r0 = 0; \
+ exit; \
+" ::: __clobber_all);
+}
+
+SEC("socket")
+__description("five frames of 512 bytes exceed the 2 KiB budget")
+__load_if_large_stack()
+__failure __msg("combined stack size of 5 calls is 2560. Too large")
+__naked void five_frames_of_512(void)
+{
+ asm volatile (" \
+ r1 = 5; \
+ *(u64 *)(r10 - 512) = r1; \
+ call frame_512_depth_4; \
+ r0 = 0; \
+ exit; \
+" ::: __clobber_all);
+}
+
+__used __naked
+static void frame_1536_leaf(void)
+{
+ asm volatile (" \
+ r1 = 1; \
+ *(u64 *)(r10 - 1536) = r1; \
+ exit; \
+" ::: __clobber_all);
+}
+
+SEC("socket")
+__description("512-byte frame calling a 1536-byte frame")
+__load_if_large_stack()
+__success __log_level(4) __msg("stack depth max 2048")
+__naked void uneven_frames_fit(void)
+{
+ asm volatile (" \
+ r1 = 2; \
+ *(u64 *)(r10 - 512) = r1; \
+ call frame_1536_leaf; \
+ r0 = 0; \
+ exit; \
+" ::: __clobber_all);
+}
+
+SEC("socket")
+__description("520-byte frame calling a 1536-byte frame")
+__load_if_large_stack()
+__failure __msg("combined stack size of 2 calls is 2064. Too large")
+__naked void uneven_frames_exceed(void)
+{
+ asm volatile (" \
+ r1 = 2; \
+ *(u64 *)(r10 - 520) = r1; \
+ call frame_1536_leaf; \
+ r0 = 0; \
+ exit; \
+" ::: __clobber_all);
+}
+
+#ifdef __BPF_FEATURE_MAY_GOTO
+/* may_goto adds its counter below the frame; a JIT does not hold that against the budget */
+SEC("socket")
+__description("frame of 2048 bytes with may_goto")
+__load_if_large_stack()
+__success __retval(42)
+__naked void frame_2048_with_may_goto(void)
+{
+ asm volatile (" \
+ r1 = r10; \
+ r1 += -2048; \
+ r0 = 42; \
+ *(u32*)(r1 + 0) = r0; \
+ may_goto l0_%=; \
+ r2 = 100; \
+ l0_%=: \
+ exit; \
+" ::: __clobber_all);
+}
+#endif
+
+SEC("socket")
+__description("variable offset write reaching 2048 bytes deep")
+__load_if_large_stack()
+__success
+__naked void var_off_write_to_2048(void)
+{
+ asm volatile (" \
+ call %[bpf_get_prandom_u32]; \
+ r0 &= 8; \
+ r2 = r10; \
+ r2 += -2048; \
+ r2 += r0; \
+ r1 = 0; \
+ *(u64*)(r2 + 0) = r1; \
+ r0 = 0; \
+ exit; \
+" :
+ : __imm(bpf_get_prandom_u32)
+ : __clobber_all);
+}
+
+SEC("socket")
+__description("variable offset write reaching 2056 bytes deep")
+__load_if_large_stack()
+__failure __msg("invalid variable-offset write to stack R2")
+__naked void var_off_write_to_2056(void)
+{
+ asm volatile (" \
+ call %[bpf_get_prandom_u32]; \
+ r0 &= 8; \
+ r2 = r10; \
+ r2 += -2056; \
+ r2 += r0; \
+ r1 = 0; \
+ *(u64*)(r2 + 0) = r1; \
+ r0 = 0; \
+ exit; \
+" :
+ : __imm(bpf_get_prandom_u32)
+ : __clobber_all);
+}
+
+/* Each frame of a private stack gets the whole budget. */
+__used __naked
+static void priv_stack_frame_2048(void)
+{
+ asm volatile (" \
+ r1 = 1; \
+ *(u64 *)(r10 - 2048) = r1; \
+ exit; \
+" ::: __clobber_all);
+}
+
+SEC("kprobe")
+__description("private stack: two frames of 2048 bytes")
+__load_if_large_stack()
+__arch_x86_64
+__arch_arm64
+__success __log_level(4)
+__msg("stack depth max 2048")
+__msg("subprog 0 (private_stack_two_frames) main {{.*}} stack 2048")
+__msg("subprog 1 (priv_stack_frame_2048) static {{.*}} stack 2048")
+__naked void private_stack_two_frames(void)
+{
+ asm volatile (" \
+ r1 = 2; \
+ *(u64 *)(r10 - 2048) = r1; \
+ call priv_stack_frame_2048; \
+ r0 = 0; \
+ exit; \
+" ::: __clobber_all);
+}
+
+struct {
+ __uint(type, BPF_MAP_TYPE_PROG_ARRAY);
+ __uint(max_entries, 1);
+ __uint(key_size, sizeof(__u32));
+ __uint(value_size, sizeof(__u32));
+} jmp_table SEC(".maps");
+
+/*
+ * A tail call unwinds the frame of the program doing it, so a large main
+ * frame is fine; the 256-byte rule only concerns the frames of callers of a
+ * subprog that tail calls.
+ */
+SEC("tc")
+__description("tail call from a 1 KiB frame")
+__load_if_large_stack()
+__success
+__naked void tail_call_from_large_frame(void)
+{
+ asm volatile (" \
+ r2 = 42; \
+ *(u64 *)(r10 - 1024) = r2; \
+ r2 = %[jmp_table] ll; \
+ r3 = 0; \
+ call %[bpf_tail_call]; \
+ r0 = 0; \
+ exit; \
+" :
+ : __imm(bpf_tail_call),
+ __imm_addr(jmp_table)
+ : __clobber_all);
+}
+
+/* Global subprogs are verified on their own but share the call chain budget. */
+__used __naked int global_frame_1536(void)
+{
+ asm volatile (" \
+ r1 = 1; \
+ *(u64 *)(r10 - 1536) = r1; \
+ r0 = 0; \
+ exit; \
+" ::: __clobber_all);
+}
+
+SEC("socket")
+__description("512-byte frame calling a 1536-byte global subprog")
+__load_if_large_stack()
+__success __log_level(4) __msg("stack depth max 2048")
+__naked void global_subprog_fits(void)
+{
+ asm volatile (" \
+ r1 = 2; \
+ *(u64 *)(r10 - 512) = r1; \
+ call global_frame_1536; \
+ r0 = 0; \
+ exit; \
+" ::: __clobber_all);
+}
+
+SEC("socket")
+__description("520-byte frame calling a 1536-byte global subprog")
+__load_if_large_stack()
+__failure __msg("combined stack size of 2 calls is 2064. Too large")
+__naked void global_subprog_exceeds(void)
+{
+ asm volatile (" \
+ r1 = 2; \
+ *(u64 *)(r10 - 520) = r1; \
+ call global_frame_1536; \
+ r0 = 0; \
+ exit; \
+" ::: __clobber_all);
+}
+
+/* Callback frames are part of the chain of the helper that calls them. */
+static __naked int loop_cb_1536(void)
+{
+ asm volatile (" \
+ r1 = 1; \
+ *(u64 *)(r10 - 1536) = r1; \
+ r0 = 0; \
+ exit; \
+" ::: __clobber_all);
+}
+
+SEC("socket")
+__description("512-byte frame with a 1536-byte bpf_loop callback")
+__load_if_large_stack()
+__success __log_level(4) __msg("stack depth max 2048")
+__naked void loop_callback_fits(void)
+{
+ asm volatile (" \
+ r1 = 2; \
+ *(u64 *)(r10 - 512) = r1; \
+ r1 = 1; \
+ r2 = %[loop_cb_1536]; \
+ r3 = 0; \
+ r4 = 0; \
+ call %[bpf_loop]; \
+ r0 = 0; \
+ exit; \
+" :
+ : __imm_ptr(loop_cb_1536),
+ __imm(bpf_loop)
+ : __clobber_common);
+}
+
+SEC("socket")
+__description("520-byte frame with a 1536-byte bpf_loop callback")
+__load_if_large_stack()
+__failure __msg("combined stack size of 2 calls is 2064. Too large")
+__naked void loop_callback_exceeds(void)
+{
+ asm volatile (" \
+ r1 = 2; \
+ *(u64 *)(r10 - 520) = r1; \
+ r1 = 1; \
+ r2 = %[loop_cb_1536]; \
+ r3 = 0; \
+ r4 = 0; \
+ call %[bpf_loop]; \
+ r0 = 0; \
+ exit; \
+" :
+ : __imm_ptr(loop_cb_1536),
+ __imm(bpf_loop)
+ : __clobber_common);
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/verifier_live_stack.c b/tools/testing/selftests/bpf/progs/verifier_live_stack.c
index 7a1a0670f851d..7be813f06a5ae 100644
--- a/tools/testing/selftests/bpf/progs/verifier_live_stack.c
+++ b/tools/testing/selftests/bpf/progs/verifier_live_stack.c
@@ -318,7 +318,7 @@ struct {
} map_array SEC(".maps");
SEC("socket")
-__failure __msg("invalid read from stack R2 off=-1024 size=8")
+__failure __msg("invalid read from stack R2 off=-4096 size=8")
__flag(BPF_F_TEST_STATE_FREQ)
__naked unsigned long caller_stack_write_tail_call(void)
{
@@ -329,7 +329,7 @@ __naked unsigned long caller_stack_write_tail_call(void)
"if r0 != 42 goto 1f;"
"goto 2f;"
"1:"
- "*(u64 *)(r10 - 8) = -1024;"
+ "*(u64 *)(r10 - 8) = -4096;"
"2:"
"r1 = r6;"
"r2 = r10;"
@@ -1953,7 +1953,7 @@ static __used __naked void fwd_parent_key_to_helper(void)
SEC("socket")
__log_level(2)
__success
-__msg("call bpf_map_update_elem{{.*}}; use: fp1-8..-512 fp0-8")
+__msg("call bpf_map_update_elem{{.*}}; use: fp1-8..-{{(512|2048)}} fp0-8")
__naked void helper_arg_fallback_keeps_scanning(void)
{
asm volatile (
@@ -2267,7 +2267,7 @@ static __used __naked void merge_leaf_read(void)
SEC("socket")
__log_level(2)
__success
-__msg("call bpf_loop#181 ; use: fp2-8..-512 fp1-8..-512 fp0-8..-512")
+__msg("call bpf_loop#181 ; use: fp2-8..-{{(512|2048)}} fp1-8..-{{(512|2048)}} fp0-8..-{{(512|2048)}}")
__naked void bpf_loop_two_callbacks(void)
{
asm volatile (
@@ -2839,3 +2839,74 @@ static __used __naked void imprecise_dst_spill_join_sub(void)
:: __imm(bpf_get_prandom_u32)
: __clobber_all);
}
+
+/*
+ * A store that does not fully cover a 4-byte half-slot defines nothing, so a
+ * narrow store at the top of the frame must not turn any slot into a "def",
+ * least of all every slot of the frame: the earlier data at fp-8 stays live.
+ */
+SEC("socket")
+__log_level(2)
+__msg("0: (79) r0 = *(u64 *)(r10 -8) ; use: fp0-8")
+__msg("1: (73) *(u8 *)(r10 -1) = r0{{$}}")
+__msg("2: (6b) *(u16 *)(r10 -4) = r0{{$}}")
+__msg("3: (79) r0 = *(u64 *)(r10 -8) ; use: fp0-8")
+__naked void narrow_store_defines_nothing(void)
+{
+ asm volatile (
+ "r0 = *(u64 *)(r10 - 8);"
+ "*(u8 *)(r10 - 1) = r0;"
+ "*(u16 *)(r10 - 4) = r0;"
+ "r0 = *(u64 *)(r10 - 8);"
+ "exit;"
+ ::: __clobber_all);
+}
+
+/*
+ * The same callee instance is analyzed twice: the call sites are visited in
+ * postorder, so the second one goes first with a precise pointer 264 bytes
+ * into the main frame, and the first one then passes a pointer of unknown
+ * offset, which reads the whole frame. The masks of the two passes differ in
+ * width; merging the second into the first must keep the whole-frame read.
+ */
+SEC("socket")
+__log_level(2)
+__msg("stack use/def subprog#{{[0-9]+}} merge_read_all_callee (d2,cs{{[0-9]+}}):")
+__msg("(79) r0 = *(u64 *)(r1 +0){{.*}}; use: fp0-8..-{{(512|2048)}}")
+__naked void merge_keeps_whole_frame_read(void)
+{
+ asm volatile (
+ "r1 = 0;"
+ "*(u64 *)(r10 - 8) = r1;"
+ "*(u64 *)(r10 - 16) = r1;"
+ "*(u64 *)(r10 - 264) = r1;"
+ "call %[bpf_get_prandom_u32];"
+ "r0 &= 8;"
+ "r1 = r10;"
+ "r1 += -16;"
+ "r1 += r0;"
+ "call merge_read_all_mid;"
+ "r1 = r10;"
+ "r1 += -264;"
+ "call merge_read_all_mid;"
+ "r0 = 0;"
+ "exit;"
+ :: __imm(bpf_get_prandom_u32)
+ : __clobber_all);
+}
+
+static __used __naked void merge_read_all_mid(void)
+{
+ asm volatile (
+ "call merge_read_all_callee;"
+ "exit;"
+ ::: __clobber_all);
+}
+
+static __used __naked void merge_read_all_callee(void)
+{
+ asm volatile (
+ "r0 = *(u64 *)(r1 + 0);"
+ "exit;"
+ ::: __clobber_all);
+}
diff --git a/tools/testing/selftests/bpf/progs/verifier_raw_stack.c b/tools/testing/selftests/bpf/progs/verifier_raw_stack.c
index 9f0f48ecb4216..0fe631411b9cc 100644
--- a/tools/testing/selftests/bpf/progs/verifier_raw_stack.c
+++ b/tools/testing/selftests/bpf/progs/verifier_raw_stack.c
@@ -240,6 +240,7 @@ __naked void load_bytes_spilled_regs_data(void)
SEC("tc")
__description("raw_stack: skb_load_bytes, invalid access 1")
+__load_if_no_large_stack()
__failure __msg("invalid write to stack R3 off=-513 size=8")
__naked void load_bytes_invalid_access_1(void)
{
@@ -257,6 +258,26 @@ __naked void load_bytes_invalid_access_1(void)
: __clobber_all);
}
+SEC("tc")
+__description("raw_stack: skb_load_bytes, invalid access 1, large stack")
+__load_if_large_stack()
+__failure __msg("invalid write to stack R3 off=-2049 size=8")
+__naked void load_bytes_invalid_access_1_large(void)
+{
+ asm volatile (" \
+ r2 = 4; \
+ r6 = r10; \
+ r6 += -2049; \
+ r3 = r6; \
+ r4 = 8; \
+ call %[bpf_skb_load_bytes]; \
+ r0 = *(u64*)(r6 + 0); \
+ exit; \
+" :
+ : __imm(bpf_skb_load_bytes)
+ : __clobber_all);
+}
+
SEC("tc")
__description("raw_stack: skb_load_bytes, invalid access 2")
__failure __msg("invalid write to stack R3 off=-1 size=8")
diff --git a/tools/testing/selftests/bpf/progs/verifier_stack_ptr.c b/tools/testing/selftests/bpf/progs/verifier_stack_ptr.c
index 8e8cf8232255f..3e0bea9819cab 100644
--- a/tools/testing/selftests/bpf/progs/verifier_stack_ptr.c
+++ b/tools/testing/selftests/bpf/progs/verifier_stack_ptr.c
@@ -235,6 +235,7 @@ __naked void to_stack_check_low_1(void)
SEC("socket")
__description("PTR_TO_STACK check low 2")
+__load_if_no_large_stack()
__success __failure_unpriv
__msg_unpriv("R1 stack pointer arithmetic goes out of range")
__retval(42)
@@ -250,8 +251,27 @@ __naked void to_stack_check_low_2(void)
" ::: __clobber_all);
}
+SEC("socket")
+__description("PTR_TO_STACK check low 2, large stack")
+__load_if_large_stack()
+__success __failure_unpriv
+__msg_unpriv("R1 stack pointer arithmetic goes out of range")
+__retval(42)
+__naked void to_stack_check_low_2_large(void)
+{
+ asm volatile (" \
+ r1 = r10; \
+ r1 += -2049; \
+ r0 = 42; \
+ *(u8*)(r1 + 1) = r0; \
+ r0 = *(u8*)(r1 + 1); \
+ exit; \
+" ::: __clobber_all);
+}
+
SEC("socket")
__description("PTR_TO_STACK check low 3")
+__load_if_no_large_stack()
__failure __msg("invalid write to stack R1 off=-513 size=1")
__msg_unpriv("R1 stack pointer arithmetic goes out of range")
__naked void to_stack_check_low_3(void)
@@ -266,6 +286,23 @@ __naked void to_stack_check_low_3(void)
" ::: __clobber_all);
}
+SEC("socket")
+__description("PTR_TO_STACK check low 3, large stack")
+__load_if_large_stack()
+__failure __msg("invalid write to stack R1 off=-2049 size=1")
+__msg_unpriv("R1 stack pointer arithmetic goes out of range")
+__naked void to_stack_check_low_3_large(void)
+{
+ asm volatile (" \
+ r1 = r10; \
+ r1 += -2049; \
+ r0 = 42; \
+ *(u8*)(r1 + 0) = r0; \
+ r0 = *(u8*)(r1 + 0); \
+ exit; \
+" ::: __clobber_all);
+}
+
SEC("socket")
__description("PTR_TO_STACK check low 4")
__failure __msg("math between fp pointer")
@@ -483,6 +520,7 @@ l1_%=: r0 = 42; \
SEC("socket")
__description("PTR_TO_STACK stack size > 512")
+__load_if_no_large_stack()
__failure __msg("invalid write to stack R1 off=-520 size=8")
__naked void stack_check_size_gt_512(void)
{
@@ -495,6 +533,21 @@ __naked void stack_check_size_gt_512(void)
" ::: __clobber_all);
}
+SEC("socket")
+__description("PTR_TO_STACK stack size > 2048")
+__load_if_large_stack()
+__failure __msg("invalid write to stack R1 off=-2056 size=8")
+__naked void stack_check_size_gt_2048(void)
+{
+ asm volatile (" \
+ r1 = r10; \
+ r1 += -2056; \
+ r0 = 42; \
+ *(u64*)(r1 + 0) = r0; \
+ exit; \
+" ::: __clobber_all);
+}
+
#ifdef __BPF_FEATURE_MAY_GOTO
SEC("socket")
__description("PTR_TO_STACK stack size 512 with may_goto with jit")
diff --git a/tools/testing/selftests/bpf/progs/verifier_tailcall.c b/tools/testing/selftests/bpf/progs/verifier_tailcall.c
index b4acce60fb9b9..51687da972257 100644
--- a/tools/testing/selftests/bpf/progs/verifier_tailcall.c
+++ b/tools/testing/selftests/bpf/progs/verifier_tailcall.c
@@ -28,4 +28,61 @@ __naked void invalid_map_for_tail_call(void)
: __clobber_all);
}
+struct {
+ __uint(type, BPF_MAP_TYPE_PROG_ARRAY);
+ __uint(max_entries, 1);
+ __uint(key_size, sizeof(__u32));
+ __uint(value_size, sizeof(__u32));
+} jmp_table SEC(".maps");
+
+__used __naked
+static int subprog_tail_call(void)
+{
+ asm volatile (" \
+ r2 = %[jmp_table] ll; \
+ r3 = 0; \
+ call %[bpf_tail_call]; \
+ r0 = 0; \
+ exit; \
+" :
+ : __imm(bpf_tail_call),
+ __imm_addr(jmp_table)
+ : __clobber_all);
+}
+
+/*
+ * A tail call unwinds only the frame of the subprog doing it, so the
+ * frames of its callers stay on the stack. With up to 33 tail calls in
+ * a chain the verifier caps the stack those frames may add up to at
+ * 256 bytes.
+ */
+SEC("tc")
+__description("tail call from subprog with 240 bytes of caller stack")
+__success
+__naked void tail_call_caller_stack_ok(void)
+{
+ asm volatile (" \
+ r2 = 42; \
+ *(u64 *)(r10 - 240) = r2; \
+ call subprog_tail_call; \
+ r0 = 0; \
+ exit; \
+" ::: __clobber_all);
+}
+
+SEC("tc")
+__description("tail call from subprog with 256 bytes of caller stack")
+__failure
+__msg("tail_calls are not allowed when call stack of previous frames is 256 bytes. Too large")
+__naked void tail_call_caller_stack_too_large(void)
+{
+ asm volatile (" \
+ r2 = 42; \
+ *(u64 *)(r10 - 256) = r2; \
+ call subprog_tail_call; \
+ r0 = 0; \
+ exit; \
+" ::: __clobber_all);
+}
+
char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/verifier_var_off.c b/tools/testing/selftests/bpf/progs/verifier_var_off.c
index a63e336750918..399884911ea53 100644
--- a/tools/testing/selftests/bpf/progs/verifier_var_off.c
+++ b/tools/testing/selftests/bpf/progs/verifier_var_off.c
@@ -406,6 +406,7 @@ __naked void zero_sized_access_max_out_of_bound(void)
SEC("lwt_in")
__description("indirect variable-offset stack access, min out of bound")
+__load_if_no_large_stack()
__failure __msg("invalid variable-offset read from stack R2")
__naked void access_min_out_of_bound(void)
{
@@ -433,6 +434,37 @@ __naked void access_min_out_of_bound(void)
: __clobber_all);
}
+SEC("lwt_in")
+__description("indirect variable-offset stack access, min out of bound, large stack")
+__load_if_large_stack()
+__failure __msg("invalid variable-offset read from stack R2")
+__naked void access_min_out_of_bound_large(void)
+{
+ asm volatile (" \
+ /* Fill the top 8 bytes of the stack */ \
+ r2 = 0; \
+ *(u64*)(r10 - 8) = r2; \
+ /* Get an unknown value */ \
+ r2 = *(u32*)(r1 + 0); \
+ /* Make it small and 4-byte aligned */ \
+ r2 &= 4; \
+ r2 -= 2052; \
+ /* \
+ * add it to fp. We now have either fp-2052 or fp-2048, but\
+ * we don't know which \
+ */ \
+ r2 += r10; \
+ /* dereference it indirectly */ \
+ r1 = %[map_hash_8b] ll; \
+ call %[bpf_map_lookup_elem]; \
+ r0 = 0; \
+ exit; \
+" :
+ : __imm(bpf_map_lookup_elem),
+ __imm_addr(map_hash_8b)
+ : __clobber_all);
+}
+
SEC("cgroup/skb")
__description("indirect variable-offset stack access, min_off < min_initialized")
__success
diff --git a/tools/testing/selftests/bpf/test_loader.c b/tools/testing/selftests/bpf/test_loader.c
index a6e3fcc1079c6..25eeb1c1248b3 100644
--- a/tools/testing/selftests/bpf/test_loader.c
+++ b/tools/testing/selftests/bpf/test_loader.c
@@ -45,6 +45,11 @@ enum load_mode {
NO_JITED = 1 << 1,
};
+enum stack_mode {
+ LARGE_STACK = 1 << 0,
+ SMALL_STACK = 1 << 1,
+};
+
struct test_subspec {
char *name;
char *description;
@@ -70,6 +75,7 @@ struct test_spec {
int mode_mask;
int arch_mask;
int load_mask;
+ int stack_mask;
int linear_sz;
const char *skip_reason;
bool prepare_priv;
@@ -425,6 +431,7 @@ static int parse_test_spec(struct test_loader *tester,
int err = 0;
u32 arch_mask = 0;
u32 load_mask = 0;
+ u32 stack_mask = 0;
struct btf *btf;
enum arch arch;
@@ -620,6 +627,16 @@ static int parse_test_spec(struct test_loader *tester,
err = -EINVAL;
goto cleanup;
}
+ } else if ((val = str_has_pfx(s, "stack_mode="))) {
+ if (strcmp(val, "large") == 0) {
+ stack_mask = LARGE_STACK;
+ } else if (strcmp(val, "small") == 0) {
+ stack_mask = SMALL_STACK;
+ } else {
+ PRINT_FAIL("bad stack spec: '%s'", val);
+ err = -EINVAL;
+ goto cleanup;
+ }
} else if ((msg = str_has_pfx(s, "test_expect_stderr="))) {
err = push_disasm_msg(msg, &stderr_on_next_line,
&spec->priv.stderr);
@@ -659,6 +676,7 @@ static int parse_test_spec(struct test_loader *tester,
spec->arch_mask = arch_mask ?: -1;
spec->load_mask = load_mask ?: (JITED | NO_JITED);
+ spec->stack_mask = stack_mask ?: (LARGE_STACK | SMALL_STACK);
if (spec->mode_mask == 0)
spec->mode_mask = PRIV;
@@ -1331,6 +1349,7 @@ void run_subtest(struct test_loader *tester,
{
struct test_subspec *subspec = unpriv ? &spec->unpriv : &spec->priv;
int current_runtime = is_jit_enabled() ? JITED : NO_JITED;
+ int current_stack = is_large_stack_supported() ? LARGE_STACK : SMALL_STACK;
struct bpf_program *tprog = NULL, *tprog_iter;
struct bpf_link *link, *links[32] = {};
struct test_spec *spec_iter;
@@ -1360,6 +1379,11 @@ void run_subtest(struct test_loader *tester,
return;
}
+ if ((current_stack & spec->stack_mask) == 0) {
+ test__skip();
+ return;
+ }
+
if (unpriv) {
if (!can_execute_unpriv(tester, spec)) {
test__skip();
diff --git a/tools/testing/selftests/bpf/testing_helpers.c b/tools/testing/selftests/bpf/testing_helpers.c
index d1d60451c5bcc..47fe61a1ebff1 100644
--- a/tools/testing/selftests/bpf/testing_helpers.c
+++ b/tools/testing/selftests/bpf/testing_helpers.c
@@ -517,6 +517,47 @@ bool is_jit_enabled(void)
return enabled;
}
+/*
+ * Whether the kernel accepts a program using more than 512 bytes of stack,
+ * which depends on the JIT in use. Probed once with a program that stores
+ * at the 2 KiB depth. Only the verifier's verdict on that store is cached:
+ * a load that fails for another reason, such as a missing capability, is
+ * reported and probed again on the next call.
+ */
+bool is_large_stack_supported(void)
+{
+ static int supported = -1;
+ struct bpf_insn insns[] = {
+ BPF_ST_MEM(BPF_DW, BPF_REG_10, -2048, 0),
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_EXIT_INSN(),
+ };
+ char log[1024] = {};
+ LIBBPF_OPTS(bpf_prog_load_opts, opts,
+ .log_buf = log,
+ .log_size = sizeof(log),
+ .log_level = 1,
+ );
+ int fd;
+
+ if (supported >= 0)
+ return supported;
+
+ fd = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, NULL, "GPL", insns, ARRAY_SIZE(insns),
+ &opts);
+ if (fd >= 0) {
+ close(fd);
+ supported = 1;
+ } else if (strstr(log, "invalid write to stack")) {
+ supported = 0;
+ } else {
+ fprintf(stderr, "%s: probe failed with errno %d, assuming 512 bytes:\n%s",
+ __func__, errno, log);
+ return false;
+ }
+ return supported;
+}
+
int stack_mprotect(void)
{
void *buf;
diff --git a/tools/testing/selftests/bpf/testing_helpers.h b/tools/testing/selftests/bpf/testing_helpers.h
index 1c58a2f08b645..f1505108e26a4 100644
--- a/tools/testing/selftests/bpf/testing_helpers.h
+++ b/tools/testing/selftests/bpf/testing_helpers.h
@@ -59,6 +59,7 @@ struct bpf_insn;
int get_xlated_program(int fd_prog, struct bpf_insn **buf, __u32 *cnt);
int testing_prog_flags(void);
bool is_jit_enabled(void);
+bool is_large_stack_supported(void);
int stack_mprotect(void);
/* Runs diff(1) on mismatch */
diff --git a/tools/testing/selftests/bpf/verifier/calls.c b/tools/testing/selftests/bpf/verifier/calls.c
index 8b94b87135bcf..0af237c02ddf9 100644
--- a/tools/testing/selftests/bpf/verifier/calls.c
+++ b/tools/testing/selftests/bpf/verifier/calls.c
@@ -1037,15 +1037,34 @@
.result = ACCEPT,
},
{
- "calls: stack overflow using two frames (pre-call access)",
+ /*
+ * Five 480-byte frames exceed the 2 KiB budget of JITs with large
+ * stacks, and two of them the 512 bytes allowed elsewhere.
+ */
+ "calls: stack overflow using five frames (pre-call access)",
.insns = {
/* prog 1 */
- BPF_ST_MEM(BPF_B, BPF_REG_10, -300, 0),
+ BPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),
BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1),
BPF_EXIT_INSN(),
/* prog 2 */
- BPF_ST_MEM(BPF_B, BPF_REG_10, -300, 0),
+ BPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1),
+ BPF_EXIT_INSN(),
+
+ /* prog 3 */
+ BPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1),
+ BPF_EXIT_INSN(),
+
+ /* prog 4 */
+ BPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1),
+ BPF_EXIT_INSN(),
+
+ /* prog 5 */
+ BPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),
BPF_MOV64_IMM(BPF_REG_0, 0),
BPF_EXIT_INSN(),
},
@@ -1054,15 +1073,30 @@
.result = REJECT,
},
{
- "calls: stack overflow using two frames (post-call access)",
+ "calls: stack overflow using five frames (post-call access)",
.insns = {
/* prog 1 */
BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 2),
- BPF_ST_MEM(BPF_B, BPF_REG_10, -300, 0),
+ BPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),
BPF_EXIT_INSN(),
/* prog 2 */
- BPF_ST_MEM(BPF_B, BPF_REG_10, -300, 0),
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 2),
+ BPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),
+ BPF_EXIT_INSN(),
+
+ /* prog 3 */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 2),
+ BPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),
+ BPF_EXIT_INSN(),
+
+ /* prog 4 */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 2),
+ BPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),
+ BPF_EXIT_INSN(),
+
+ /* prog 5 */
+ BPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),
BPF_MOV64_IMM(BPF_REG_0, 0),
BPF_EXIT_INSN(),
},
@@ -1127,7 +1161,7 @@
.result = ACCEPT,
},
{
- "calls: stack depth check using three frames. test3",
+ "calls: stack depth check using five frames. test3",
.insns = {
/* main */
BPF_MOV64_REG(BPF_REG_6, BPF_REG_1),
@@ -1135,66 +1169,104 @@
BPF_MOV64_REG(BPF_REG_1, BPF_REG_6),
BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 8), /* call B */
BPF_JMP_IMM(BPF_JGE, BPF_REG_6, 0, 1),
- BPF_ST_MEM(BPF_B, BPF_REG_10, -64, 0),
+ BPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),
BPF_MOV64_IMM(BPF_REG_0, 0),
BPF_EXIT_INSN(),
/* A */
BPF_JMP_IMM(BPF_JLT, BPF_REG_1, 10, 1),
BPF_EXIT_INSN(),
- BPF_ST_MEM(BPF_B, BPF_REG_10, -224, 0),
+ BPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),
BPF_JMP_IMM(BPF_JA, 0, 0, -3),
/* B */
BPF_JMP_IMM(BPF_JGT, BPF_REG_1, 2, 1),
- BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, -6), /* call A */
- BPF_ST_MEM(BPF_B, BPF_REG_10, -256, 0),
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 2), /* call C */
+ BPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),
+ BPF_EXIT_INSN(),
+ /* C */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 2), /* call D */
+ BPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),
+ BPF_EXIT_INSN(),
+ /* D */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, -12), /* call A */
+ BPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),
BPF_EXIT_INSN(),
},
.prog_type = BPF_PROG_TYPE_XDP,
- /* stack_main=64, stack_A=224, stack_B=256
- * and max(main+A, main+A+B) > 512
+ /*
+ * every frame is 480 bytes, main+A = 960 > 512 and
+ * max(main+A, main+B+C+D+A) = 2400 > 2048
*/
.errstr = "combined stack",
.result = REJECT,
},
{
- "calls: stack depth check using three frames. test4",
- /* void main(void) {
+ "calls: stack depth check using five frames. test4",
+ /*
+ * void main(void) {
* func1(0);
* func1(1);
* func2(1);
* }
- * void func1(int alloc_or_recurse) {
+ * void funcN(int alloc_or_recurse) { N = 1..4
* if (alloc_or_recurse) {
- * frame_pointer[-300] = 1;
+ * frame_pointer[-480] = 1;
* } else {
- * func2(alloc_or_recurse);
+ * funcN+1(alloc_or_recurse);
* }
* }
- * void func2(int alloc_or_recurse) {
+ * void func5(int alloc_or_recurse) {
* if (alloc_or_recurse) {
- * frame_pointer[-300] = 1;
+ * frame_pointer[-480] = 1;
* }
* }
+ * main also calls func2 to func5 with 1 so that every function has a
+ * path allocating its 480 bytes, and the chain adds up to 2400 bytes,
+ * more than the 2 KiB budget of JITs with large stacks, and to 960
+ * bytes after two frames, more than the 512 bytes allowed elsewhere.
*/
.insns = {
/* main */
BPF_MOV64_IMM(BPF_REG_1, 0),
- BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 6), /* call A */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 12), /* call A */
BPF_MOV64_IMM(BPF_REG_1, 1),
- BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 4), /* call A */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 10), /* call A */
+ BPF_MOV64_IMM(BPF_REG_1, 1),
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 13), /* call B */
+ BPF_MOV64_IMM(BPF_REG_1, 1),
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 16), /* call C */
BPF_MOV64_IMM(BPF_REG_1, 1),
- BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 7), /* call B */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 19), /* call D */
+ BPF_MOV64_IMM(BPF_REG_1, 1),
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 22), /* call E */
BPF_MOV64_IMM(BPF_REG_0, 0),
BPF_EXIT_INSN(),
/* A */
BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 2),
- BPF_ST_MEM(BPF_B, BPF_REG_10, -300, 0),
+ BPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),
BPF_EXIT_INSN(),
BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call B */
BPF_EXIT_INSN(),
/* B */
+ BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 2),
+ BPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),
+ BPF_EXIT_INSN(),
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call C */
+ BPF_EXIT_INSN(),
+ /* C */
+ BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 2),
+ BPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),
+ BPF_EXIT_INSN(),
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call D */
+ BPF_EXIT_INSN(),
+ /* D */
+ BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 2),
+ BPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),
+ BPF_EXIT_INSN(),
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call E */
+ BPF_EXIT_INSN(),
+ /* E */
BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 1),
- BPF_ST_MEM(BPF_B, BPF_REG_10, -300, 0),
+ BPF_ST_MEM(BPF_B, BPF_REG_10, -480, 0),
BPF_EXIT_INSN(),
},
.prog_type = BPF_PROG_TYPE_XDP,
|
| 31/3 | 2026/09/26 01:43 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 63996 output: 28 thoughts: 1602 Thoughts: **KMSAN Evaluation of BPF Stack Increase Patch** Okay, let's break this down. My goal is to determine if this BPF stack increase warrants a dedicated KMSAN fuzzing session. The primary concern with KMSAN is uninitialized memory reads, which KASAN, LOCKDEP, and other standard checks may miss. I need to figure out if this patch introduces new opportunities for uninitialized memory reads or leaks. First, I need to understand what this patch actually *does*. It's bumping the BPF stack size on JIT-capable architectures (x86_64, arm64) from 512 bytes to 2KB. This means the BPF verifier has to be updated to handle this larger stack. Okay, so far, it seems like a straightforward increase, but I need to dive into the implications. Now, I'm thinking about how the BPF stack interacts with *kernel* memory. The BPF stack is *within* kernel memory when a BPF program runs. That's a key distinction. Could this change, by increasing the stack size, inadvertently allow for reading uninitialized memory in the kernel itself, or potentially leaking kernel memory to userspace through the stack? Does this patch introduce anything that could cause uninitialized stack slots to be read from within the kernel's execution context? I know that KMSAN's purpose here is only to catch reads of *uninitialized* kernel memory. KASAN is already enabled, so I'm not worried about the typical OOB, UAF, double-free issues unless this introduces an entirely *new* type of memory use. KMSAN detects reads of *uninitialized* memory on the stack, heap, or page allocations, and kernel-to-user memory leaks. My internal ruleset: KMSAN is only necessary if the patch touches any of these: kernel structures sent to userspace, conditional logic on potentially uninitialized variables or struct fields, complex data structure allocations that might leave fields uninitialized, or bounds checks that could lead to reading outside of initialized buffers. A key question emerges: does this patch *change* how BPF stack initialization is tracked? Internally, the BPF verifier already does its own stack tracking (`STACK_INVALID`, `STACK_ZERO`, etc.), but does *this* patch alter that in a way that creates a KMSAN concern? Now, how does this stack memory get *allocated*? I remember JIT uses `sub rsp, round_up()`. This means that BPF stack memory is within the kernel stack (or private stack). Crucially, can a verifier bug now potentially lead to uninitialized memory reads? Does KMSAN even instrument BPF JIT code? No, of course not. KMSAN instruments C code, not generated machine code. This is a very important point! So, I need to focus on what the *C code* does. I should focus on the interpreter and verifier. Alright, time to focus on the details. I need to carefully inspect the diff, line by line. I'm looking for new or modified data structures. I want to search for uses of uninitialized variables, new allocations, information leaks, and so on. Let's begin with the data structure that has changed, `bpf_jmp_history_entry`. I see that `linked_regs` has become an array, `u16 linked_regs[BPF_LINKED_REGS_MAX]; u8 linked_regs_cnt;`. How is this allocated and used? Is this correctly initialized? I need to know! Let's see how `bpf_jmp_history_entry` is handled in `kernel/bpf/backtrack.c` using `bpf_push_jmp_history()`. Let's also see where `cur->jmp_history` is allocated using `grepper`. |
| 32/3 | 2026/09/26 01:43 | tool | grepper |
0mArgs:{
"Expression": "jmp_history",
"PathPrefix": "kernel/bpf/"
}Results: {
"Output": "kernel/bpf/backtrack.c-10-/* for any branch, call, exit record the history of jmps in the given state */\nkernel/bpf/backtrack.c:11:int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state *cur,\nkernel/bpf/backtrack.c-12-\t\t\t int insn_flags, int spi, int frame, const u16 *linked_regs,\n--\nkernel/bpf/backtrack.c-14-{\nkernel/bpf/backtrack.c:15:\tu32 cnt = cur-\u003ejmp_history_cnt;\nkernel/bpf/backtrack.c:16:\tstruct bpf_jmp_history_entry *p;\nkernel/bpf/backtrack.c-17-\tsize_t alloc_size;\n--\nkernel/bpf/backtrack.c-42-\talloc_size = kmalloc_size_roundup(size_mul(cnt, sizeof(*p)));\nkernel/bpf/backtrack.c:43:\tp = krealloc(cur-\u003ejmp_history, alloc_size, GFP_KERNEL_ACCOUNT);\nkernel/bpf/backtrack.c-44-\tif (!p)\nkernel/bpf/backtrack.c-45-\t\treturn -ENOMEM;\nkernel/bpf/backtrack.c:46:\tcur-\u003ejmp_history = p;\nkernel/bpf/backtrack.c-47-\nkernel/bpf/backtrack.c:48:\tp = \u0026cur-\u003ejmp_history[cnt - 1];\nkernel/bpf/backtrack.c-49-\tp-\u003eidx = env-\u003einsn_idx;\n--\nkernel/bpf/backtrack.c-56-\tp-\u003elinked_regs_cnt = linked_regs_cnt;\nkernel/bpf/backtrack.c:57:\tcur-\u003ejmp_history_cnt = cnt;\nkernel/bpf/backtrack.c-58-\tenv-\u003ecur_hist_ent = p;\n--\nkernel/bpf/backtrack.c=90=static int get_prev_insn_idx(struct bpf_verifier_state *st, int i,\n--\nkernel/bpf/backtrack.c-97-\t\t\treturn -ENOENT;\nkernel/bpf/backtrack.c:98:\t\tif (cnt == 1 \u0026\u0026 st-\u003ejmp_history[0].idx == i)\nkernel/bpf/backtrack.c-99-\t\t\treturn -ENOENT;\n--\nkernel/bpf/backtrack.c-101-\nkernel/bpf/backtrack.c:102:\tif (cnt \u0026\u0026 st-\u003ejmp_history[cnt - 1].idx == i) {\nkernel/bpf/backtrack.c:103:\t\ti = st-\u003ejmp_history[cnt - 1].prev_idx;\nkernel/bpf/backtrack.c-104-\t\t(*history)--;\n--\nkernel/bpf/backtrack.c-110-\nkernel/bpf/backtrack.c:111:static struct bpf_jmp_history_entry *get_jmp_hist_entry(struct bpf_verifier_state *st,\nkernel/bpf/backtrack.c-112-\t\t\t\t\t\t u32 hist_end, int insn_idx)\nkernel/bpf/backtrack.c-113-{\nkernel/bpf/backtrack.c:114:\tif (hist_end \u003e 0 \u0026\u0026 st-\u003ejmp_history[hist_end - 1].idx == insn_idx)\nkernel/bpf/backtrack.c:115:\t\treturn \u0026st-\u003ejmp_history[hist_end - 1];\nkernel/bpf/backtrack.c-116-\treturn NULL;\n--\nkernel/bpf/backtrack.c=283=static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,\nkernel/bpf/backtrack.c:284:\t\t\t struct bpf_jmp_history_entry *hist, struct backtrack_state *bt)\nkernel/bpf/backtrack.c-285-{\n--\nkernel/bpf/backtrack.c=700=void bpf_mark_all_scalars_precise(struct bpf_verifier_env *env,\n--\nkernel/bpf/backtrack.c-753- * assembly instructions (see backtrack_insns() for exact instruction handling\nkernel/bpf/backtrack.c:754: * logic). This backtracking relies on recorded jmp_history and is able to\nkernel/bpf/backtrack.c-755- * traverse entire chain of parent states. This process ends only when all the\n--\nkernel/bpf/backtrack.c=834=int bpf_mark_chain_precision(struct bpf_verifier_env *env,\n--\nkernel/bpf/backtrack.c-874-\t\tDECLARE_BITMAP(mask, 64);\nkernel/bpf/backtrack.c:875:\t\tu32 history = st-\u003ejmp_history_cnt;\nkernel/bpf/backtrack.c:876:\t\tstruct bpf_jmp_history_entry *hist;\nkernel/bpf/backtrack.c-877-\n--\nkernel/bpf/states.c=1253=int bpf_is_state_visited(struct bpf_verifier_env *env, int insn_idx)\n--\nkernel/bpf/states.c-1263-\t\t\t /* Avoid accumulating infinitely long jmp history */\nkernel/bpf/states.c:1264:\t\t\t cur-\u003ejmp_history_cnt \u003e 40;\nkernel/bpf/states.c-1265-\n--\nkernel/bpf/states.c-1430-\t\t\tif (bpf_is_jmp_point(env, env-\u003einsn_idx))\nkernel/bpf/states.c:1431:\t\t\t\terr = bpf_push_jmp_history(env, cur, 0, 0, 0, NULL, 0);\nkernel/bpf/states.c-1432-\t\t\terr = err ? : propagate_precision(env, \u0026sl-\u003estate, cur, NULL);\n--\nkernel/bpf/states.c-1611-\tcur-\u003edfs_depth = new-\u003edfs_depth + 1;\nkernel/bpf/states.c:1612:\tbpf_clear_jmp_history(cur);\nkernel/bpf/states.c-1613-\tlist_add(\u0026new_sl-\u003enode, head);\n--\nkernel/bpf/verifier.c=1666=static void free_func_state(struct bpf_func_state *state)\n--\nkernel/bpf/verifier.c-1674-\nkernel/bpf/verifier.c:1675:void bpf_clear_jmp_history(struct bpf_verifier_state *state)\nkernel/bpf/verifier.c-1676-{\nkernel/bpf/verifier.c:1677:\tkfree(state-\u003ejmp_history);\nkernel/bpf/verifier.c:1678:\tstate-\u003ejmp_history = NULL;\nkernel/bpf/verifier.c:1679:\tstate-\u003ejmp_history_cnt = 0;\nkernel/bpf/verifier.c-1680-}\n--\nkernel/bpf/verifier.c=1682=void bpf_free_verifier_state(struct bpf_verifier_state *state,\n--\nkernel/bpf/verifier.c-1691-\tkfree(state-\u003erefs);\nkernel/bpf/verifier.c:1692:\tbpf_clear_jmp_history(state);\nkernel/bpf/verifier.c-1693-\tif (free_self)\n--\nkernel/bpf/verifier.c=1709=int bpf_copy_verifier_state(struct bpf_verifier_state *dst_state,\n--\nkernel/bpf/verifier.c-1714-\nkernel/bpf/verifier.c:1715:\tdst_state-\u003ejmp_history = copy_array(dst_state-\u003ejmp_history, src-\u003ejmp_history,\nkernel/bpf/verifier.c:1716:\t\t\t\t\t src-\u003ejmp_history_cnt, sizeof(*dst_state-\u003ejmp_history),\nkernel/bpf/verifier.c-1717-\t\t\t\t\t GFP_KERNEL_ACCOUNT);\nkernel/bpf/verifier.c:1718:\tif (!dst_state-\u003ejmp_history)\nkernel/bpf/verifier.c-1719-\t\treturn -ENOMEM;\nkernel/bpf/verifier.c:1720:\tdst_state-\u003ejmp_history_cnt = src-\u003ejmp_history_cnt;\nkernel/bpf/verifier.c-1721-\n--\nkernel/bpf/verifier.c=3353=static void linked_regs_pack(const struct linked_regs *s, u16 *packed)\n--\nkernel/bpf/verifier.c-3363-\nkernel/bpf/verifier.c:3364:static void linked_regs_unpack(const struct bpf_jmp_history_entry *hist, struct linked_regs *s)\nkernel/bpf/verifier.c-3365-{\n--\nkernel/bpf/verifier.c=3397=void bpf_verbose_insn(struct bpf_verifier_env *env, struct bpf_insn *insn)\n--\nkernel/bpf/verifier.c-3410- */\nkernel/bpf/verifier.c:3411:void bpf_bt_sync_linked_regs(struct backtrack_state *bt, struct bpf_jmp_history_entry *hist)\nkernel/bpf/verifier.c-3412-{\n--\nkernel/bpf/verifier.c=3584=static int check_stack_write_fixed_off(struct bpf_verifier_env *env,\n--\nkernel/bpf/verifier.c-3705-\tif (insn_flags)\nkernel/bpf/verifier.c:3706:\t\treturn bpf_push_jmp_history(env, env-\u003ecur_state, insn_flags,\nkernel/bpf/verifier.c-3707-\t\t\t\t\t hist_spi, hist_frame, NULL, 0);\n--\nkernel/bpf/verifier.c=3936=static int check_stack_read_fixed_off(struct bpf_verifier_env *env,\n--\nkernel/bpf/verifier.c-4087-\tif (insn_flags)\nkernel/bpf/verifier.c:4088:\t\treturn bpf_push_jmp_history(env, env-\u003ecur_state, insn_flags,\nkernel/bpf/verifier.c-4089-\t\t\t\t\t hist_spi, hist_frame, NULL, 0);\n--\nkernel/bpf/verifier.c=4243=static int check_stack_arg_write(struct bpf_verifier_env *env, struct bpf_func_state *state,\n--\nkernel/bpf/verifier.c-4277-\tstate-\u003eno_stack_arg_load = true;\nkernel/bpf/verifier.c:4278:\treturn bpf_push_jmp_history(env, env-\u003ecur_state,\nkernel/bpf/verifier.c-4279-\t\t\t\t INSN_F_STACK_ARG_ACCESS, spi, 0, NULL, 0);\n--\nkernel/bpf/verifier.c=4286=static int check_stack_arg_read(struct bpf_verifier_env *env, struct bpf_func_state *state,\n--\nkernel/bpf/verifier.c-4311-\tbpf_diag_mod_end(env);\nkernel/bpf/verifier.c:4312:\treturn bpf_push_jmp_history(env, env-\u003ecur_state,\nkernel/bpf/verifier.c-4313-\t\t\t\t INSN_F_STACK_ARG_ACCESS, spi, 0, NULL, 0);\n--\nkernel/bpf/verifier.c=17398=static int check_cond_jmp_op(struct bpf_verifier_env *env,\n--\nkernel/bpf/verifier.c-17471-\tif (insn_flags) {\nkernel/bpf/verifier.c:17472:\t\terr = bpf_push_jmp_history(env, this_branch, insn_flags, 0, 0, NULL, 0);\nkernel/bpf/verifier.c-17473-\t\tif (err)\n--\nkernel/bpf/verifier.c-17544-\t\tlinked_regs_pack(\u0026linked_regs, packed);\nkernel/bpf/verifier.c:17545:\t\terr = bpf_push_jmp_history(env, this_branch, 0, 0, 0, packed, linked_regs.cnt);\nkernel/bpf/verifier.c-17546-\t\tif (err)\n--\nkernel/bpf/verifier.c=18860=static int do_check(struct bpf_verifier_env *env)\n--\nkernel/bpf/verifier.c-18939-\t\tif (bpf_is_jmp_point(env, env-\u003einsn_idx)) {\nkernel/bpf/verifier.c:18940:\t\t\terr = bpf_push_jmp_history(env, state, 0, 0, 0, NULL, 0);\nkernel/bpf/verifier.c-18941-\t\t\tif (err)\n"
} |
| 33/3 | 2026/09/26 01:43 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 69793 output: 36 thoughts: 50 Thoughts: Here's my analysis of the provided snippet, presented as a first-person expert perspective: **Diving into `bpf_push_jmp_history` Initialization** Okay, I need to get a clear picture of how `bpf_push_jmp_history` sets up its internal pointer `p`. I know the core logic resides in `kernel/bpf/backtrack.c`, so I'll pull that file up and focus on the relevant section – lines 15 through 65. Let's dig in and see precisely how this is initialized. No need to go too deep. Let's assume that it's a pointer to some memory and that is gets filled in the code, but let's go over the code to make sure. I'll need to double check the allocation and any initialization that happens in the function. |
| 34/3 | 2026/09/26 01:43 | tool | read-file |
0mArgs:{
"File": "kernel/bpf/backtrack.c",
"FirstLine": 15,
"LineCount": 60
}Results: {
"Contents": " 15:\t\tu32 cnt = cur-\u003ejmp_history_cnt;\n 16:\t\tstruct bpf_jmp_history_entry *p;\n 17:\t\tsize_t alloc_size;\n 18:\t\n 19:\t\t/* combine instruction flags if we already recorded this instruction */\n 20:\t\tif (env-\u003ecur_hist_ent) {\n 21:\t\t\t/* atomic instructions push insn_flags twice, for READ and\n 22:\t\t\t * WRITE sides, but they should agree on stack slot\n 23:\t\t\t */\n 24:\t\t\tverifier_bug_if((env-\u003ecur_hist_ent-\u003eflags \u0026 insn_flags) \u0026\u0026\n 25:\t\t\t\t\t(env-\u003ecur_hist_ent-\u003eflags \u0026 insn_flags) != insn_flags,\n 26:\t\t\t\t\tenv, \"insn history: insn_idx %d cur flags %x new flags %x\",\n 27:\t\t\t\t\tenv-\u003einsn_idx, env-\u003ecur_hist_ent-\u003eflags, insn_flags);\n 28:\t\t\tenv-\u003ecur_hist_ent-\u003eflags |= insn_flags;\n 29:\t\t\tenv-\u003ecur_hist_ent-\u003espi = spi;\n 30:\t\t\tenv-\u003ecur_hist_ent-\u003eframe = frame;\n 31:\t\t\tverifier_bug_if(env-\u003ecur_hist_ent-\u003elinked_regs_cnt != 0, env,\n 32:\t\t\t\t\t\"insn history: insn_idx %d has %u linked regs\",\n 33:\t\t\t\t\tenv-\u003einsn_idx, env-\u003ecur_hist_ent-\u003elinked_regs_cnt);\n 34:\t\t\tif (linked_regs_cnt)\n 35:\t\t\t\tmemcpy(env-\u003ecur_hist_ent-\u003elinked_regs, linked_regs,\n 36:\t\t\t\t linked_regs_cnt * sizeof(*linked_regs));\n 37:\t\t\tenv-\u003ecur_hist_ent-\u003elinked_regs_cnt = linked_regs_cnt;\n 38:\t\t\treturn 0;\n 39:\t\t}\n 40:\t\n 41:\t\tcnt++;\n 42:\t\talloc_size = kmalloc_size_roundup(size_mul(cnt, sizeof(*p)));\n 43:\t\tp = krealloc(cur-\u003ejmp_history, alloc_size, GFP_KERNEL_ACCOUNT);\n 44:\t\tif (!p)\n 45:\t\t\treturn -ENOMEM;\n 46:\t\tcur-\u003ejmp_history = p;\n 47:\t\n 48:\t\tp = \u0026cur-\u003ejmp_history[cnt - 1];\n 49:\t\tp-\u003eidx = env-\u003einsn_idx;\n 50:\t\tp-\u003eprev_idx = env-\u003eprev_insn_idx;\n 51:\t\tp-\u003eflags = insn_flags;\n 52:\t\tp-\u003espi = spi;\n 53:\t\tp-\u003eframe = frame;\n 54:\t\tif (linked_regs_cnt)\n 55:\t\t\tmemcpy(p-\u003elinked_regs, linked_regs, linked_regs_cnt * sizeof(*linked_regs));\n 56:\t\tp-\u003elinked_regs_cnt = linked_regs_cnt;\n 57:\t\tcur-\u003ejmp_history_cnt = cnt;\n 58:\t\tenv-\u003ecur_hist_ent = p;\n 59:\t\n 60:\t\treturn 0;\n 61:\t}\n 62:\t\n 63:\tstatic bool is_atomic_load_insn(const struct bpf_insn *insn)\n 64:\t{\n 65:\t\treturn BPF_CLASS(insn-\u003ecode) == BPF_STX \u0026\u0026\n 66:\t\t BPF_MODE(insn-\u003ecode) == BPF_ATOMIC \u0026\u0026\n 67:\t\t insn-\u003eimm == BPF_LOAD_ACQ;\n 68:\t}\n 69:\t\n 70:\tstatic bool is_atomic_fetch_insn(const struct bpf_insn *insn)\n 71:\t{\n 72:\t\treturn BPF_CLASS(insn-\u003ecode) == BPF_STX \u0026\u0026\n 73:\t\t BPF_MODE(insn-\u003ecode) == BPF_ATOMIC \u0026\u0026\n 74:\t\t (insn-\u003eimm \u0026 BPF_FETCH);\n"
} |
| 35/3 | 2026/09/26 01:43 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 71106 output: 28 thoughts: 1428 Thoughts: **Analyzing Potential Uninitialized Memory Reads in BPF Verification** Okay, let's break this down. My initial focus is on the potential for uninitialized memory reads within the BPF verifier. The code snippets, specifically lines 48-56, have sparked a concern. `krealloc` doesn't zero memory, and I see `p = &cur->jmp_history[cnt - 1];` being used. That looks risky! My primary concern is this: if `krealloc` is used to grow the `jmp_history` and `p->linked_regs` isn't fully initialized, could there be a read beyond the bounds? My concern is if anything ever reads `p->linked_regs` beyond the `linked_regs_cnt`. So first, I need to check how that `linked_regs` is *used*. The immediate follow-up is to investigate `linked_regs_unpack`. Good, this confirms my initial suspicion; it iterates using `s->cnt`, which is set by `hist->linked_regs_cnt`. So, it *should* be safe. It looks like it only iterates from 0 to `hist->linked_regs_cnt`, so if there are any uninitialized elements, they are not accessed. Next, I need to confirm whether `jmp_history` itself is ever copied to userspace. If so, uninitialized memory could be leaked. Checking `bpf_copy_verifier_state` which copies `jmp_history`, it only does a *copy* of `sizeof(*dst_state->jmp_history)`. So, it doesn't seem to be a problem. Let's check where `jmp_history` is used across the kernel. A quick scan confirms my assumption: `jmp_history` is only utilized within the verifier's internal state. It is never copied to userspace. I think this is safe. Now, let's switch gears and examine `bpf_id_scratch_reserve`. This looks like a potentially similar issue. We're using `krealloc` to expand the buffer and then copying data. Specifically, I'm now looking at the use of `idmap` and `idset`. The code snippets for `check_ids`, `idset_cnt_inc`, and `idstack_push` are relevant. In `check_ids` and `idset_cnt_inc`, the code iterates up to `idmap->cnt - 1` and `num_ids - 1` respectively. Expansion initializes the new element *before* incrementing the count, so the new elements won't be read. That should be safe. However, in `idstack_push`, it seems to be accessing the `idmap` differently, and in this case, I'm trying to see if anything reads `idmap->map[i].cur`, so I need to check where `idstack_push` is called to confirm its usage and look for potential issues there. |
| 36/3 | 2026/09/26 01:43 | tool | grepper |
0mArgs:{
"Expression": "idstack_",
"PathPrefix": "kernel/bpf/"
}Results: {
"Output": "kernel/bpf/verifier.c=10120=static int release_reference_nomark(struct bpf_verifier_env *env, int id)\n--\nkernel/bpf/verifier.c-10129-\nkernel/bpf/verifier.c:10130:static int idstack_push(struct bpf_idmap *idmap, u32 id)\nkernel/bpf/verifier.c-10131-{\n--\nkernel/bpf/verifier.c-10148-\nkernel/bpf/verifier.c:10149:static int idstack_pop(struct bpf_idmap *idmap)\nkernel/bpf/verifier.c-10150-{\n--\nkernel/bpf/verifier.c=10158=static int release_reference(struct bpf_verifier_env *env, int id)\n--\nkernel/bpf/verifier.c-10168-\tidstack-\u003ecnt = 0;\nkernel/bpf/verifier.c:10169:\terr = idstack_push(idstack, id);\nkernel/bpf/verifier.c-10170-\tif (err)\n--\nkernel/bpf/verifier.c-10177-\nkernel/bpf/verifier.c:10178:\twhile ((id = idstack_pop(idstack))) {\nkernel/bpf/verifier.c-10179-\t\t/*\n--\nkernel/bpf/verifier.c-10198-\t\t\tif (reg-\u003eparent_id == id) {\nkernel/bpf/verifier.c:10199:\t\t\t\terr = idstack_push(idstack, reg-\u003eid);\nkernel/bpf/verifier.c-10200-\t\t\t\tif (err)\n"
} |
| 37/3 | 2026/09/26 01:43 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 73649 output: 38 thoughts: 33 Thoughts: Here's my take on the thought process: **Diving into the BPF Verifier** Okay, time to troubleshoot this BPF issue. First things first, I need to get a handle on what's going on with the `idstack_pop` function. Let's see if there's an obvious problem here. I'm going to navigate directly to the heart of the verifier code, specifically `kernel/bpf/verifier.c` to see how it's being handled. Around line 10130 should be a good starting point to zero in on how it's implemented. Hopefully, that should give me some solid clues. |
| 38/3 | 2026/09/26 01:43 | tool | read-file |
0mArgs:{
"File": "kernel/bpf/verifier.c",
"FirstLine": 10130,
"LineCount": 35
}Results: {
"Contents": "10130:\tstatic int idstack_push(struct bpf_idmap *idmap, u32 id)\n10131:\t{\n10132:\t\tint i;\n10133:\t\n10134:\t\tif (!id)\n10135:\t\t\treturn 0;\n10136:\t\n10137:\t\tfor (i = 0; i \u003c idmap-\u003ecnt; i++)\n10138:\t\t\tif (idmap-\u003emap[i].old == id)\n10139:\t\t\t\treturn 0;\n10140:\t\n10141:\t\tif (!bpf_id_scratch_reserve((void **)\u0026idmap-\u003emap, \u0026idmap-\u003ecap, idmap-\u003ecnt,\n10142:\t\t\t\t\t sizeof(*idmap-\u003emap)))\n10143:\t\t\treturn -ENOMEM;\n10144:\t\n10145:\t\tidmap-\u003emap[idmap-\u003ecnt++].old = id;\n10146:\t\treturn 0;\n10147:\t}\n10148:\t\n10149:\tstatic int idstack_pop(struct bpf_idmap *idmap)\n10150:\t{\n10151:\t\tif (!idmap-\u003ecnt)\n10152:\t\t\treturn 0;\n10153:\t\n10154:\t\treturn idmap-\u003emap[--idmap-\u003ecnt].old;\n10155:\t}\n10156:\t\n10157:\t/* Release id and objects derived from it iteratively in a DFS manner */\n10158:\tstatic int release_reference(struct bpf_verifier_env *env, int id)\n10159:\t{\n10160:\t\tu32 mask = (1 \u003c\u003c STACK_SPILL) | (1 \u003c\u003c STACK_DYNPTR);\n10161:\t\tstruct bpf_verifier_state *vstate = env-\u003ecur_state;\n10162:\t\tstruct bpf_idmap *idstack = \u0026env-\u003eidmap_scratch;\n10163:\t\tstruct bpf_stack_state *stack;\n10164:\t\tstruct bpf_func_state *state;\n"
} |
| 39/3 | 2026/09/26 01:43 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 74444 output: 28 thoughts: 641 Thoughts: **Analyzing Memory Initialization in BPF Code**
Okay, so I'm looking at this BPF code and trying to understand how memory is initialized. First, I see that `.old` is fully initialized, which is good.
Now, let's dive into `liveness.c`. The `widen_frame_masks` function is interesting. It uses `kvzalloc_flex`. *That's key*—`kvzalloc_flex` allocates *zeroed* memory. Any bits in the new allocation that weren't in the `old` allocation are guaranteed to be zero. Furthermore, `update_insn` also explicitly initializes `new_after` with zeros using `= {}`. Seems like a consistent zero-initialization strategy is at play here.
Then, there's `bpf_fmt_stack_mask`. It formats stack masks for printing. The crucial part here is the buffer used: callers pass `env->tmp_str_buf`. I recall that this buffer was enlarged to `TMP_STR_BUF_LEN 1408`. This enlargement provides a larger buffer for the mask formatting, helping with possible buffer overflows.
Finally, a quick mental note about `stack_masks`: I need to remember where `stack_masks` within `struct backtrack_state` are initialized! I need to trace the allocation and initialization of `struct backtrack_state` itself (or `env->bt`). Need to search for those.
|
| 40/3 | 2026/09/26 01:43 | tool | grepper |
0mArgs:{
"Expression": "bt_reset",
"PathPrefix": "kernel/bpf/"
}Results: {
"Output": "kernel/bpf/backtrack.c=119=static inline void bt_init(struct backtrack_state *bt, u32 frame)\n--\nkernel/bpf/backtrack.c-123-\nkernel/bpf/backtrack.c:124:static inline void bt_reset(struct backtrack_state *bt)\nkernel/bpf/backtrack.c-125-{\n--\nkernel/bpf/backtrack.c=834=int bpf_mark_chain_precision(struct bpf_verifier_env *env,\n--\nkernel/bpf/backtrack.c-921-\t\t\t\tbpf_mark_all_scalars_precise(env, starting_state);\nkernel/bpf/backtrack.c:922:\t\t\t\tbt_reset(bt);\nkernel/bpf/backtrack.c-923-\t\t\t\treturn 0;\n--\nkernel/bpf/backtrack.c-1023-\t\tbpf_mark_all_scalars_precise(env, starting_state);\nkernel/bpf/backtrack.c:1024:\t\tbt_reset(bt);\nkernel/bpf/backtrack.c-1025-\t}\n"
} |
| 41/3 | 2026/09/26 01:43 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 75718 output: 37 thoughts: 27 Thoughts: Okay, let's dive into this. Here's my summary, taking on the persona you requested: **Diving into `bt_init` and `bt_reset`** Right, time to examine the core of the backtracking mechanism in the BPF kernel code. I need to get a solid understanding of how these functions, specifically `bt_init` and `bt_reset`, are implemented within `kernel/bpf/backtrack.c`. My immediate focus is on how they contribute to the overarching backtracking functionality. I need to see how the state is initialized, maintained, and cleaned up as the BPF verifier navigates potentially complex program paths. Are there any subtle race conditions lurking here that I need to be aware of? Are there potential performance bottlenecks related to these initialisation/reset operations? I need to think this through in the context of the larger BPF ecosystem. Let's see how they handle memory allocation, and what data structures are in play. Any clever optimizations? Let's get to the code. |
| 42/3 | 2026/09/26 01:43 | tool | read-file |
0mArgs:{
"File": "kernel/bpf/backtrack.c",
"FirstLine": 115,
"LineCount": 25
}Results: {
"Contents": " 115:\t\t\treturn \u0026st-\u003ejmp_history[hist_end - 1];\n 116:\t\treturn NULL;\n 117:\t}\n 118:\t\n 119:\tstatic inline void bt_init(struct backtrack_state *bt, u32 frame)\n 120:\t{\n 121:\t\tbt-\u003eframe = frame;\n 122:\t}\n 123:\t\n 124:\tstatic inline void bt_reset(struct backtrack_state *bt)\n 125:\t{\n 126:\t\tstruct bpf_verifier_env *env = bt-\u003eenv;\n 127:\t\n 128:\t\tmemset(bt, 0, sizeof(*bt));\n 129:\t\tbt-\u003eenv = env;\n 130:\t}\n 131:\t\n 132:\tstatic inline bool bt_frame_stack_empty(struct backtrack_state *bt, u32 frame)\n 133:\t{\n 134:\t\treturn bitmap_empty(bt-\u003estack_masks[frame], MAX_BPF_STACK_SLOTS);\n 135:\t}\n 136:\t\n 137:\tstatic inline bool bt_stack_empty(struct backtrack_state *bt)\n 138:\t{\n 139:\t\treturn bt_frame_stack_empty(bt, bt-\u003eframe);\n"
} |
| 43/3 | 2026/09/26 01:43 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 76358 output: 29 thoughts: 55 Thoughts: Okay, here's my interpretation of those thoughts, crafted with the understanding that I'm discussing this with a fellow expert: **Investigating Memory Initialization in the BPF Context** Alright, so I'm thinking about the `bt_reset` function and how it's using `memset(bt, 0, sizeof(*bt))` to clear a structure. My immediate thought is, "Okay, that's standard for initialization, but let's double-check how the environment variable `env` is handled." I know `env` is allocated with `kzalloc` during its creation within `bpf_check`. So, the next thing I'm doing is mentally tracing that allocation path and need to make certain to verify the allocation of `env` within `bpf_check`. This is critical. Because `kzalloc` does zero-initialization, which aligns with my understanding of `memset` in `bt_reset`, but it's always good to be thorough and confirm the allocation strategy. This helps me confirm and contextualize the overall memory management strategy, and its potential implications. |
| 44/3 | 2026/09/26 01:43 | tool | grepper |
0mArgs:{
"Expression": "kzalloc.*env",
"PathPrefix": "kernel/bpf/"
}Results: {
"Output": "kernel/bpf/btf.c=5988=static struct btf *btf_parse(const union bpf_attr *attr, bpfptr_t uattr,\n--\nkernel/bpf/btf.c-6000-\nkernel/bpf/btf.c:6001:\tenv = kzalloc_obj(*env, GFP_KERNEL | __GFP_NOWARN);\nkernel/bpf/btf.c-6002-\tif (!env)\n--\nkernel/bpf/btf.c=6477=struct btf *btf_parse_vmlinux(void)\n--\nkernel/bpf/btf.c-6483-\nkernel/bpf/btf.c:6484:\tenv = kzalloc_obj(*env, GFP_KERNEL | __GFP_NOWARN);\nkernel/bpf/btf.c-6485-\tif (!env)\n--\nkernel/bpf/btf.c=6519=static struct btf *btf_parse_module(const char *module_name, const void *data,\n--\nkernel/bpf/btf.c-6533-\nkernel/bpf/btf.c:6534:\tenv = kzalloc_obj(*env, GFP_KERNEL | __GFP_NOWARN);\nkernel/bpf/btf.c-6535-\tif (!env)\n--\nkernel/bpf/fixups.c=1098=static int jit_subprogs(struct bpf_verifier_env *env)\n--\nkernel/bpf/fixups.c-1148-\terr = -ENOMEM;\nkernel/bpf/fixups.c:1149:\tfunc = kzalloc_objs(prog, env-\u003esubprog_cnt);\nkernel/bpf/fixups.c-1150-\tif (!func)\n"
} |
| 45/3 | 2026/09/26 01:43 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 77144 output: 31 thoughts: 10 Thoughts: Okay, here's my interpretation of that thought process, tailored for an expert audience, and written in the first person: **Analyzing `bpf_check` - A Quick Audit** Alright, let's dive into this `bpf_check` function. Given my background and experience, I'm immediately thinking of a few key questions. What's its purpose? What are the potential security implications if it fails, or is bypassed? I'm not going to approach this superficially; this needs a rigorous assessment. My immediate mental checklist starts: error handling - how robust is it? What edge cases does it account for? What's the context in which `bpf_check` is being called? Is it privileged code, user-space, kernel-space, or something else? I'm assuming this function validates BPF bytecode. So, I need to know what security mitigations it's implementing. Is it protecting against memory corruption vulnerabilities by preventing out-of-bounds access, instruction sequencing issues, or arithmetic overflows? Does it account for register usage correctly? I also want to consider possible issues with the verifier itself: potential side-channels, and bypasses, as well as the types of BPF programs it handles. I know these verifiers can be complex, and finding vulnerabilities is often about exploring the interaction between the program being checked and the verifier itself. Therefore, I need to look past the functionality and into the implementation as well. Given that I'm focusing on the underlying security context, the specific BPF program, and the system, I need to understand what this check *prevents*, and then what would be possible *if* the check were circumvented. My mind instantly jumps to potential attack vectors that could exploit flaws in the BPF runtime. I want to see if any of these are accounted for, and if not, how easily they could be exploited. |
| 46/3 | 2026/09/26 01:43 | tool | grepper |
0mArgs:{
"Expression": "env = ",
"PathPrefix": "kernel/bpf/verifier.c"
}Results: {
"Output": "kernel/bpf/verifier.c=320=__printf(2, 3) static void verbose(void *private_data, const char *fmt, ...)\nkernel/bpf/verifier.c-321-{\nkernel/bpf/verifier.c:322:\tstruct bpf_verifier_env *env = private_data;\nkernel/bpf/verifier.c-323-\tva_list args;\n--\nkernel/bpf/verifier.c=21675=int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,\n--\nkernel/bpf/verifier.c-21691-\t */\nkernel/bpf/verifier.c:21692:\tenv = kvzalloc_obj(struct bpf_verifier_env, GFP_KERNEL_ACCOUNT);\nkernel/bpf/verifier.c-21693-\tif (!env)\n--\nkernel/bpf/verifier.c-21695-\nkernel/bpf/verifier.c:21696:\tenv-\u003ebt.env = env;\nkernel/bpf/verifier.c-21697-\tenv-\u003eprog = *prog;\n"
} |
| 47/3 | 2026/09/26 01:43 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 77887 output: 37 thoughts: 149 Thoughts: Okay, let me break this down. **My Analysis of Stack Slot Management in BPF** Right, so the `env` struct, that's allocated using `kvzalloc_obj`. No surprises there; that's standard for kernel object allocation. And then there's `scratched_stack_slots`. I see that's declared as a bitmap, using `DECLARE_BITMAP`. `MAX_BPF_STACK_SLOTS` gives me a clue about the size, so it's a bitmask for tracking which stack slots are in use. Excellent. And it's initialized to zero, like I'd expect, likely via the `kvzalloc_obj` call. We're using the standard bitmap manipulation functions: `bitmap_zero`, `bitmap_fill`, `__set_bit`, `test_bit`, `bitmap_empty`. All perfectly reasonable and efficient for this purpose. Now, a key question pops up: what's going on with `grow_stack_state`? That's the function to pay attention to for stack memory allocation and growth, isn't it? I need to understand how stack slots are allocated and extended when a BPF program accesses stack memory. That's the real heart of the matter. So, I need to inspect `grow_stack_state` in `kernel/bpf/verifier.c`. That's where the control logic resides. Time to dig in and see how the state is actually managed. |
| 48/3 | 2026/09/26 01:43 | tool | read-file |
0mArgs:{
"File": "kernel/bpf/verifier.c",
"FirstLine": 1440,
"LineCount": 40
}Results: {
"Contents": "1440:\t\tstate-\u003eacquired_refs = n;\n1441:\t\treturn 0;\n1442:\t}\n1443:\t\n1444:\t/* Possibly update state-\u003eallocated_stack to be at least size bytes. Also\n1445:\t * possibly update the function's high-water mark in its bpf_subprog_info.\n1446:\t */\n1447:\tstatic int grow_stack_state(struct bpf_verifier_env *env, struct bpf_func_state *state, int size)\n1448:\t{\n1449:\t\tsize_t old_n = bpf_stack_nr_slots(state), n;\n1450:\t\n1451:\t\t/* The stack size is always a multiple of BPF_REG_SIZE. */\n1452:\t\tsize = round_up(size, BPF_REG_SIZE);\n1453:\t\tn = size / BPF_REG_SIZE;\n1454:\t\n1455:\t\tif (old_n \u003e= n)\n1456:\t\t\treturn 0;\n1457:\t\n1458:\t\tstate-\u003estack = realloc_array(state-\u003estack, old_n, n, sizeof(struct bpf_stack_state));\n1459:\t\tif (!state-\u003estack)\n1460:\t\t\treturn -ENOMEM;\n1461:\t\n1462:\t\tstate-\u003eallocated_stack = size;\n1463:\t\n1464:\t\t/* update known max for given subprogram */\n1465:\t\tif (env-\u003esubprog_info[state-\u003esubprogno].stack_depth \u003c size)\n1466:\t\t\tenv-\u003esubprog_info[state-\u003esubprogno].stack_depth = size;\n1467:\t\n1468:\t\treturn 0;\n1469:\t}\n1470:\t\n1471:\tstatic int grow_stack_arg_slots(struct bpf_verifier_env *env,\n1472:\t\t\t\t\tstruct bpf_func_state *state, int cnt)\n1473:\t{\n1474:\t\tsize_t old_n = state-\u003eout_stack_arg_cnt;\n1475:\t\n1476:\t\tif (old_n \u003e= cnt)\n1477:\t\t\treturn 0;\n1478:\t\n1479:\t\tstate-\u003estack_arg_regs = realloc_array(state-\u003estack_arg_regs, old_n, cnt,\n"
} |
| 49/3 | 2026/09/26 01:43 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 79031 output: 29 thoughts: 21 Thoughts: Okay, here's my take on those thoughts, tailored for an expert audience: **Investigating `realloc_array` and its Initialization Behavior** Alright, so `realloc_array`... I need to quickly refresh my memory on the specifics of its behavior, particularly concerning the initialization of newly allocated memory. It's crucial for me to have this detail concrete in my mind, as I am trying to determine the most performant method of growing my array. I believe that it zeros the newly allocated memory, but I had better confirm that. Let's delve back into the documentation and source, just to make sure I am not mistaken. I need to know definitively if I can rely on a zero-filled extension, or if I'll need to explicitly initialize it myself, depending on the particular implementation and environment. That'll influence how I design my overall memory management strategy. |
| 50/3 | 2026/09/26 01:43 | tool | grepper |
0mArgs:{
"Expression": "realloc_array",
"PathPrefix": "kernel/bpf/"
}Results: {
"Output": "kernel/bpf/core.c=830=int bpf_jit_add_poke_descriptor(struct bpf_prog *prog,\n--\nkernel/bpf/core.c-852-\nkernel/bpf/core.c:853:\ttab = krealloc_array(tab, size, sizeof(*poke), GFP_KERNEL);\nkernel/bpf/core.c-854-\tif (!tab)\n--\nkernel/bpf/states.c=970=bool bpf_id_scratch_reserve(void **arr, u32 *cap, u32 cnt, size_t elem_size)\n--\nkernel/bpf/states.c-977-\tnew_cap = *cap ? *cap * 2 : 64;\nkernel/bpf/states.c:978:\tp = krealloc_array(*arr, new_cap, elem_size, GFP_KERNEL_ACCOUNT | __GFP_NOWARN);\nkernel/bpf/states.c-979-\tif (!p)\n--\nkernel/bpf/verifier.c=1339=static void *copy_array(void *dst, const void *src, size_t n, size_t size, gfp_t flags)\n--\nkernel/bpf/verifier.c-1365- *\nkernel/bpf/verifier.c:1366: * Contrary to krealloc_array, does not free arr if new_n is zero.\nkernel/bpf/verifier.c-1367- */\nkernel/bpf/verifier.c:1368:static void *realloc_array(void *arr, size_t old_n, size_t new_n, size_t size)\nkernel/bpf/verifier.c-1369-{\n--\nkernel/bpf/verifier.c=1433=static int resize_reference_state(struct bpf_verifier_state *state, size_t n)\nkernel/bpf/verifier.c-1434-{\nkernel/bpf/verifier.c:1435:\tstate-\u003erefs = realloc_array(state-\u003erefs, state-\u003eacquired_refs, n,\nkernel/bpf/verifier.c-1436-\t\t\t\t sizeof(struct bpf_reference_state));\n--\nkernel/bpf/verifier.c=1447=static int grow_stack_state(struct bpf_verifier_env *env, struct bpf_func_state *state, int size)\n--\nkernel/bpf/verifier.c-1457-\nkernel/bpf/verifier.c:1458:\tstate-\u003estack = realloc_array(state-\u003estack, old_n, n, sizeof(struct bpf_stack_state));\nkernel/bpf/verifier.c-1459-\tif (!state-\u003estack)\n--\nkernel/bpf/verifier.c=1471=static int grow_stack_arg_slots(struct bpf_verifier_env *env,\n--\nkernel/bpf/verifier.c-1478-\nkernel/bpf/verifier.c:1479:\tstate-\u003estack_arg_regs = realloc_array(state-\u003estack_arg_regs, old_n, cnt,\nkernel/bpf/verifier.c-1480-\t\t\t\t\t sizeof(struct bpf_reg_state));\n"
} |
| 51/3 | 2026/09/26 01:43 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 80078 output: 37 thoughts: 28 Thoughts: Here's a summary of my thought process, as requested: **Diving into the BPF Verifier's Memory Allocation** Okay, time to get to the heart of this. I'm going to head over to the `kernel/bpf/verifier.c` file. Specifically, I need to zero in on the `realloc_array` implementation. I believe it's around line 1368, give or take. This is where the verifier's memory allocation logic lives, which is a critical piece when we're dealing with potentially untrusted BPF programs. I need to understand how this function handles reallocation to ensure it's doing so safely and efficiently within the BPF context. I'll need to examine the multiplication of size and number, how it handles potential overflows, and what protections it uses to avoid any vulnerabilities there. It's time to dig into the guts of the memory management within the verifier. |
| 52/3 | 2026/09/26 01:43 | tool | read-file |
0mArgs:{
"File": "kernel/bpf/verifier.c",
"FirstLine": 1365,
"LineCount": 30
}Results: {
"Contents": "1365:\t *\n1366:\t * Contrary to krealloc_array, does not free arr if new_n is zero.\n1367:\t */\n1368:\tstatic void *realloc_array(void *arr, size_t old_n, size_t new_n, size_t size)\n1369:\t{\n1370:\t\tsize_t alloc_size;\n1371:\t\tvoid *new_arr;\n1372:\t\n1373:\t\tif (!new_n || old_n == new_n)\n1374:\t\t\tgoto out;\n1375:\t\n1376:\t\talloc_size = kmalloc_size_roundup(size_mul(new_n, size));\n1377:\t\tnew_arr = krealloc(arr, alloc_size, GFP_KERNEL_ACCOUNT);\n1378:\t\tif (!new_arr) {\n1379:\t\t\tkfree(arr);\n1380:\t\t\treturn NULL;\n1381:\t\t}\n1382:\t\tarr = new_arr;\n1383:\t\n1384:\t\tif (new_n \u003e old_n)\n1385:\t\t\tmemset(arr + old_n * size, 0, (new_n - old_n) * size);\n1386:\t\n1387:\tout:\n1388:\t\treturn arr ? arr : ZERO_SIZE_PTR;\n1389:\t}\n1390:\t\n1391:\tstatic int copy_reference_state(struct bpf_verifier_state *dst, const struct bpf_verifier_state *src)\n1392:\t{\n1393:\t\tdst-\u003erefs = copy_array(dst-\u003erefs, src-\u003erefs, src-\u003eacquired_refs,\n1394:\t\t\t\t sizeof(struct bpf_reference_state), GFP_KERNEL_ACCOUNT);\n"
} |
| 53/3 | 2026/09/26 01:43 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 80830 output: 31 thoughts: 185 Thoughts: Here's my thought process, as requested: **Examining `realloc_array` and BPF Stack Limits** Okay, so I'm digging into this `realloc_array` function, and I'm reminded that it calls `memset` to zero out the newly allocated space, specifically `memset(arr + old_n * size, 0, (new_n - old_n) * size)`. That's standard practice, of course, but it's important to keep in mind. Now, the real question that's piquing my interest is what happens when we increase the BPF stack limit to, say, 2048? The immediate concern is, could this potentially allow BPF programs to somehow access uninitialized memory on the kernel stack? That would be a serious security vulnerability. Hold on a second, though! The BPF verifier is always on guard, verifying every single read and write. It's designed to prevent precisely this type of thing. The verifier tracks initialized stack slots through `slot_type`: `STACK_INVALID`, `STACK_SPILL`, `STACK_MISC`, `STACK_ZERO`, and others. So, uninitialized stack slots *should* be inaccessible. But then I stop and ask myself: under what conditions are uninitialized stack slots *allowed* to be read? That's where I need to look at `env->allow_uninit_stack`. What does this flag do? Time to crack open the kernel source and understand its implications. I need to figure out exactly when and why uninitialized stack reads are permitted. |
| 54/3 | 2026/09/26 01:43 | tool | grepper |
0mArgs:{
"Expression": "allow_uninit_stack",
"PathPrefix": "kernel/bpf/"
}Results: {
"Output": "kernel/bpf/states.c=668=static bool is_stack_misc_after(struct bpf_verifier_env *env,\n--\nkernel/bpf/states.c-675-\t\t ((stack-\u003eslot_type[i] == STACK_INVALID || stack-\u003eslot_type[i] == STACK_POISON) \u0026\u0026\nkernel/bpf/states.c:676:\t\t env-\u003eallow_uninit_stack))\nkernel/bpf/states.c-677-\t\t\tcontinue;\n--\nkernel/bpf/states.c=696=static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,\n--\nkernel/bpf/states.c-731-\nkernel/bpf/states.c:732:\t\tif (env-\u003eallow_uninit_stack \u0026\u0026 old_type == STACK_MISC)\nkernel/bpf/states.c-733-\t\t\tcontinue;\n--\nkernel/bpf/verifier.c=3729=static int check_stack_write_var_off(struct bpf_verifier_env *env,\n--\nkernel/bpf/verifier.c-3828-\t\tif ((*stype == STACK_INVALID || *stype == STACK_POISON) \u0026\u0026\nkernel/bpf/verifier.c:3829:\t\t !env-\u003eallow_uninit_stack) {\nkernel/bpf/verifier.c-3830-\t\t\tverbose(env, \"uninit stack in range of var-offset write prohibited for !root; insn %d, off: %d\",\n--\nkernel/bpf/verifier.c=3936=static int check_stack_read_fixed_off(struct bpf_verifier_env *env,\n--\nkernel/bpf/verifier.c-4008-\t\t\t\t\t}\nkernel/bpf/verifier.c:4009:\t\t\t\t\tif (type == STACK_INVALID \u0026\u0026 env-\u003eallow_uninit_stack)\nkernel/bpf/verifier.c-4010-\t\t\t\t\t\tcontinue;\n--\nkernel/bpf/verifier.c-4067-\t\t\t\tcontinue;\nkernel/bpf/verifier.c:4068:\t\t\tif (type == STACK_INVALID \u0026\u0026 env-\u003eallow_uninit_stack)\nkernel/bpf/verifier.c-4069-\t\t\t\tcontinue;\n--\nkernel/bpf/verifier.c=6319=static int check_stack_slot_within_bounds(struct bpf_verifier_env *env,\n--\nkernel/bpf/verifier.c-6325-\nkernel/bpf/verifier.c:6326:\tif (t == BPF_WRITE || env-\u003eallow_uninit_stack)\nkernel/bpf/verifier.c-6327-\t\tmin_valid_off = -(int)env-\u003estack_limit;\n--\nkernel/bpf/verifier.c=6985=static int check_stack_range_initialized(\n--\nkernel/bpf/verifier.c-7005-\t\t (meta-\u003earg_raw_mem.mask \u0026 BIT(arg_slot));\nkernel/bpf/verifier.c:7006:\tbool raw_mode = uninit \u0026\u0026 env-\u003eallow_uninit_stack \u0026\u0026\nkernel/bpf/verifier.c-7007-\t\t\t!(meta-\u003earg_raw_mem.var_size_mask \u0026 BIT(arg_slot));\n--\nkernel/bpf/verifier.c-7067-\t\tif ((*stype == STACK_ZERO) ||\nkernel/bpf/verifier.c:7068:\t\t (*stype == STACK_INVALID \u0026\u0026 (uninit || env-\u003eallow_uninit_stack))) {\nkernel/bpf/verifier.c:7069:\t\t\tif (clobber \u0026\u0026 (*stype != STACK_INVALID || env-\u003eallow_uninit_stack)) {\nkernel/bpf/verifier.c-7070-\t\t\t\t/* helper can write anything into the stack */\n--\nkernel/bpf/verifier.c-7088-\t\t\tif (allow_poison) {\nkernel/bpf/verifier.c:7089:\t\t\t\tif (uninit \u0026\u0026 env-\u003eallow_uninit_stack)\nkernel/bpf/verifier.c-7090-\t\t\t\t\t*stype = STACK_MISC;\n--\nkernel/bpf/verifier.c=13814=s64 bpf_helper_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn *insn,\n--\nkernel/bpf/verifier.c-13832-\tfull_write = (at \u0026 MEM_UNINIT) \u0026\u0026\nkernel/bpf/verifier.c:13833:\t\t (!arg_type_is_raw_mem(at) || env-\u003eallow_uninit_stack);\nkernel/bpf/verifier.c-13834-\n--\nkernel/bpf/verifier.c=13940=s64 bpf_kfunc_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn *insn,\n--\nkernel/bpf/verifier.c-14006-\tif (is_kfunc_arg_uninit(btf, \u0026args[i]) \u0026\u0026\nkernel/bpf/verifier.c:14007:\t (is_kfunc_arg_dynptr(btf, \u0026args[i]) || env-\u003eallow_uninit_stack))\nkernel/bpf/verifier.c-14008-\t\treturn -size;\n--\nkernel/bpf/verifier.c=21675=int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,\n--\nkernel/bpf/verifier.c-21701-\tenv-\u003eallow_ptr_leaks = bpf_allow_ptr_leaks(env-\u003eprog-\u003eaux-\u003etoken);\nkernel/bpf/verifier.c:21702:\tenv-\u003eallow_uninit_stack = bpf_allow_uninit_stack(env-\u003eprog-\u003eaux-\u003etoken);\nkernel/bpf/verifier.c-21703-\tenv-\u003ebypass_spec_v1 = bpf_bypass_spec_v1(env-\u003eprog-\u003eaux-\u003etoken);\n"
} |
| 55/3 | 2026/09/26 01:43 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 82921 output: 39 thoughts: 2966 Thoughts: **BPF Stack Size Increase and KMSAN Considerations** Okay, so the task is to determine if this patch, which increases the BPF stack size limit, necessitates a KMSAN check. Let's break this down systematically. First, the question of KMSAN and BPF: KMSAN detects uninitialized memory reads, but does KMSAN even instrument BPF? No, it doesn't. BPF programs are JIT-compiled. So, can a BPF program potentially leak kernel memory to userspace *or* trigger KMSAN issues in kernel functions if it reads uninitialized memory on its stack? It could, if the BPF program reads uninitialized stack and subsequently interacts with kernel helpers such as `bpf_probe_read_kernel`, passes a buffer, or writes to a map read by userspace. However, this has always been possible for privileged users with `allow_uninit_stack`. So, does this patch *change* `allow_uninit_stack` behavior? No, it *only* increases the stack size. Now, let's apply the KMSAN rules. I need to only trigger KMSAN if the change introduces uninitialized memory risks that are *only* detected by KMSAN. If the change risks bugs that are caught by KASAN, LOCKDEP, or standard bug detectors, then KMSAN is not needed. So, let's step through the 4 KMSAN conditions: 1. **Kernel structures to userspace:** Are kernel structures sent to userspace that might have uninitialized fields? In `bpf_check` - no. But, what about `verifier_log`? The verifier uses `env->tmp_str_buf` for formatting log messages. The patch has increased `TMP_STR_BUF_LEN` from 320 to 1408 bytes. Is there a structure copied to user space? No. And `bpf_fmt_stack_mask` is safe because it is null-terminated. So this case seems ok. 2. **Conditional logic or branching on uninitialized variables:** What about the `bpf_jmp_history_entry` structure and the `linked_regs` array? It is important to inspect the usage of fields in that struct. Here I have identified that only indices `0` to `hist->linked_regs_cnt - 1` are read, and that data at these indices are initialized within the same function and passed. So, nothing uninitialized is read there. What about `bpf_idmap` and `bpf_idset`? They are now separate structures within `bpf_verifier_env`, and their pointers are allocated and freed correctly. 3. **Complex data structure allocation/initialization:** Checking the `frame_masks` in `liveness.c` and its allocation with `kvzalloc_flex` and initialized. The same with `backtrack_state`, and `scratched_stack_slots`, it's allocated with `kvzalloc_obj` or `kvzalloc_flex` and zeroed. What about the changes in the `stacksafe()` function? It seems fine. Everything accessed in `stacksafe()` is within the allocated stack. 4. **Bounds checks affecting uninitialized bytes:** It appears the patch introduces no new bounds check risks. Finally, does the increased stack size in the JIT affect uninitialized memory use? The only changes are a boolean, `bpf_jit_supports_large_stack`, which seems inconsequential. Checking the x86 JIT code, does the x86 JIT allocate stack space? There is no allocation code directly. **Conclusion:** Based on this review, I don't see any evidence that the stack size increase introduces risks *only* detectable by KMSAN. All identified potential uninitialized memory areas were initialized correctly. Therefore, the changes don't need `NeedsKMSAN=true`. |
| 56/3 | 2026/09/26 01:44 | tool | grepper |
0mArgs:{
"Expression": "stack_depth",
"PathPrefix": "arch/x86/net/bpf_jit_comp.c"
}Results: {
"Output": "arch/x86/net/bpf_jit_comp.c=527=static void emit_prologue_tail_call(u8 **pprog, bool is_subprog)\n--\narch/x86/net/bpf_jit_comp.c-562- */\narch/x86/net/bpf_jit_comp.c:563:static void emit_prologue(u8 **pprog, u8 *ip, u32 stack_depth, bool ebpf_from_cbpf,\narch/x86/net/bpf_jit_comp.c-564-\t\t\t bool tail_call_reachable, bool is_subprog,\n--\narch/x86/net/bpf_jit_comp.c-607-\narch/x86/net/bpf_jit_comp.c:608:\t/* sub rsp, rounded_stack_depth */\narch/x86/net/bpf_jit_comp.c:609:\tif (stack_depth)\narch/x86/net/bpf_jit_comp.c:610:\t\tEMIT3_off32(0x48, 0x81, 0xEC, round_up(stack_depth, 8));\narch/x86/net/bpf_jit_comp.c-611-\tif (tail_call_reachable)\n--\narch/x86/net/bpf_jit_comp.c=783=static void emit_bpf_tail_call_indirect(struct bpf_prog *bpf_prog,\narch/x86/net/bpf_jit_comp.c-784-\t\t\t\t\tu8 **pprog, bool *callee_regs_used,\narch/x86/net/bpf_jit_comp.c:785:\t\t\t\t\tu32 stack_depth, u8 *ip,\narch/x86/net/bpf_jit_comp.c-786-\t\t\t\t\tstruct jit_context *ctx)\narch/x86/net/bpf_jit_comp.c-787-{\narch/x86/net/bpf_jit_comp.c:788:\tint tcc_ptr_off = BPF_TAIL_CALL_CNT_PTR_STACK_OFF(stack_depth);\narch/x86/net/bpf_jit_comp.c-789-\tu8 *prog = *pprog, *start = *pprog;\n--\narch/x86/net/bpf_jit_comp.c-849-\tEMIT1(0x58); /* pop rax */\narch/x86/net/bpf_jit_comp.c:850:\tif (stack_depth)\narch/x86/net/bpf_jit_comp.c-851-\t\tEMIT3_off32(0x48, 0x81, 0xC4, /* add rsp, sd */\narch/x86/net/bpf_jit_comp.c:852:\t\t\t round_up(stack_depth, 8));\narch/x86/net/bpf_jit_comp.c-853-\n--\narch/x86/net/bpf_jit_comp.c=871=static void emit_bpf_tail_call_direct(struct bpf_prog *bpf_prog,\n--\narch/x86/net/bpf_jit_comp.c-873-\t\t\t\t u8 **pprog, u8 *ip,\narch/x86/net/bpf_jit_comp.c:874:\t\t\t\t bool *callee_regs_used, u32 stack_depth,\narch/x86/net/bpf_jit_comp.c-875-\t\t\t\t struct jit_context *ctx)\narch/x86/net/bpf_jit_comp.c-876-{\narch/x86/net/bpf_jit_comp.c:877:\tint tcc_ptr_off = BPF_TAIL_CALL_CNT_PTR_STACK_OFF(stack_depth);\narch/x86/net/bpf_jit_comp.c-878-\tu8 *prog = *pprog, *start = *pprog;\n--\narch/x86/net/bpf_jit_comp.c-916-\tEMIT1(0x58); /* pop rax */\narch/x86/net/bpf_jit_comp.c:917:\tif (stack_depth)\narch/x86/net/bpf_jit_comp.c:918:\t\tEMIT3_off32(0x48, 0x81, 0xC4, round_up(stack_depth, 8));\narch/x86/net/bpf_jit_comp.c-919-\n--\narch/x86/net/bpf_jit_comp.c=1786=static void emit_priv_frame_ptr(u8 **pprog, void __percpu *priv_frame_ptr)\n--\narch/x86/net/bpf_jit_comp.c-1807-\tEMIT3_off32(0x48, 0x8B, 0x85, off)\narch/x86/net/bpf_jit_comp.c:1808:/* mov rax, qword ptr [rbp - rounded_stack_depth - 16] */\narch/x86/net/bpf_jit_comp.c-1809-#define LOAD_TAIL_CALL_CNT_PTR(stack)\t\t\t\t\\\n--\narch/x86/net/bpf_jit_comp.c=1962=static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *addrs, u8 *image,\n--\narch/x86/net/bpf_jit_comp.c-1977-\tu8 *ip, *prog = temp;\narch/x86/net/bpf_jit_comp.c:1978:\tu32 stack_depth;\narch/x86/net/bpf_jit_comp.c-1979-\tint callee_saved_size;\n--\narch/x86/net/bpf_jit_comp.c-1982-\narch/x86/net/bpf_jit_comp.c:1983:\tstack_depth = bpf_prog-\u003eaux-\u003estack_depth;\narch/x86/net/bpf_jit_comp.c-1984-\tout_stack_arg_cnt = bpf_out_stack_arg_cnt(env, bpf_prog);\n--\narch/x86/net/bpf_jit_comp.c-1986-\tif (priv_stack_ptr) {\narch/x86/net/bpf_jit_comp.c:1987:\t\tpriv_frame_ptr = priv_stack_ptr + PRIV_STACK_GUARD_SZ + round_up(stack_depth, 8);\narch/x86/net/bpf_jit_comp.c:1988:\t\tstack_depth = 0;\narch/x86/net/bpf_jit_comp.c-1989-\t}\n--\narch/x86/net/bpf_jit_comp.c-2017-\narch/x86/net/bpf_jit_comp.c:2018:\temit_prologue(\u0026prog, image, stack_depth,\narch/x86/net/bpf_jit_comp.c-2019-\t\t bpf_prog_was_classic(bpf_prog), tail_call_reachable,\n--\narch/x86/net/bpf_jit_comp.c-2065-\t */\narch/x86/net/bpf_jit_comp.c:2066:\toutgoing_arg_base = -(round_up(stack_depth, 8) + callee_saved_size);\narch/x86/net/bpf_jit_comp.c-2067-\n--\narch/x86/net/bpf_jit_comp.c-2907-\t\t\tif (src_reg == BPF_PSEUDO_CALL \u0026\u0026 tail_call_reachable) {\narch/x86/net/bpf_jit_comp.c:2908:\t\t\t\tLOAD_TAIL_CALL_CNT_PTR(stack_depth);\narch/x86/net/bpf_jit_comp.c-2909-\t\t\t\tip += 7;\n--\narch/x86/net/bpf_jit_comp.c-2950-\t\t\t\t\t\t\t callee_regs_used,\narch/x86/net/bpf_jit_comp.c:2951:\t\t\t\t\t\t\t stack_depth,\narch/x86/net/bpf_jit_comp.c-2952-\t\t\t\t\t\t\t ctx);\n--\narch/x86/net/bpf_jit_comp.c-2956-\t\t\t\t\t\t\t callee_regs_used,\narch/x86/net/bpf_jit_comp.c:2957:\t\t\t\t\t\t\t stack_depth,\narch/x86/net/bpf_jit_comp.c-2958-\t\t\t\t\t\t\t ip,\n--\narch/x86/net/bpf_jit_comp.c=3628=static int invoke_bpf_mod_ret(const struct btf_func_model *m, u8 **pprog,\n--\narch/x86/net/bpf_jit_comp.c-3665-\narch/x86/net/bpf_jit_comp.c:3666:/* mov rax, qword ptr [rbp - rounded_stack_depth - 8] */\narch/x86/net/bpf_jit_comp.c-3667-#define LOAD_TRAMP_TAIL_CALL_CNT_PTR(stack)\t\\\n--\narch/x86/net/bpf_jit_comp.c=4225=struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog)\n--\narch/x86/net/bpf_jit_comp.c-4257-\t\t */\narch/x86/net/bpf_jit_comp.c:4258:\t\tpriv_stack_alloc_sz = round_up(prog-\u003eaux-\u003estack_depth, 8) +\narch/x86/net/bpf_jit_comp.c-4259-\t\t\t\t 2 * PRIV_STACK_GUARD_SZ;\n--\narch/x86/net/bpf_jit_comp.c=4472=void bpf_jit_free(struct bpf_prog *prog)\n--\narch/x86/net/bpf_jit_comp.c-4495-\t\tif (priv_stack_ptr) {\narch/x86/net/bpf_jit_comp.c:4496:\t\t\tpriv_stack_alloc_sz = round_up(prog-\u003eaux-\u003estack_depth, 8) +\narch/x86/net/bpf_jit_comp.c-4497-\t\t\t\t\t 2 * PRIV_STACK_GUARD_SZ;\n"
} |
| 57/3 | 2026/09/26 01:44 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 89575 output: 436 thoughts: 1261 Thoughts: **Analysis of KMSAN Need for BPF Stack Increase** Alright, let's break this down. We're increasing the BPF stack limit, so the immediate question is whether this introduces or exposes uninitialized memory reads that KMSAN would uniquely catch. First, on the x86 side, it looks like the existing prologue/epilogue/tail call code uses 32-bit offsets, so a 2048-byte stack is no problem. This accommodates the changes as well as existing private stacks. The interpreter, however, remains at the old `MAX_BPF_STACK` limit. Now for the core question: memory safety. I can see potential risks: if the verifier messes up stack depth calculations, or if a JITed program overflows the kernel stack. That's a classic stack overflow, which triggers guard page faults, panics, KASAN OOB access. Also, if the verifier access out of bound into `state->stack`, or dynamic arrays like `idmap->map` or `idset->entries` overflow or have memory leaks, KMSAN detects those issues. Dynamic bitmasks in liveness analysis? KASAN will detect any out-of-bounds access. So, are there any uninitialized memory concerns? Let's check the criteria. 1. No kernel structures are sent to userspace. Good. 2. No conditional logic depends on potentially uninitialized variables. Everything in `bpf_jmp_history_entry`, `frame_masks`, `backtrack_state`, `bpf_idmap`, `bpf_idset` is initialized before use. 3. Newly allocated buffers are either zero-initialized or explicitly written before read. 4. The verifier diligently tracks stack byte initialization before reads, no uninitialized memory leaks. My conclusion is clear: this patch doesn't introduce or expose UNINITIALIZED MEMORY risks. Any issues are likely to be related to verification logic, stack depth computation (kernel stack overflows), bounds checking in analysis, or memory allocation/reallocation – all of which KASAN, stack guards, and standard bug detectors should catch. The rule is clear: *no KMSAN if other detectors are sufficient*. This change doesn't introduce uninitialized memory risks. Thus: `NeedsKMSAN = false`. I'll document my findings and set the flag accordingly. |
| 58/3 | 2026/09/26 01:44 | tool | set-results |
0mArgs:{
"KMSANReasoning": "The patch increases the maximum stack depth budget for BPF programs from 512 bytes (MAX_BPF_STACK) to 2048 bytes (MAX_BPF_STACK_JIT) on architectures whose JIT supports large stacks (x86-64 and arm64). It updates the BPF verifier's stack tracking, liveness analysis, backtracking, and scratch data structures to handle up to 2 KiB frames.\n\nAnalysis of memory safety and detector applicability:\n1. No kernel structures or padding are copied or leaked to user space via copy_to_user or BPF buffers.\n2. All modified and newly allocated data structures are properly initialized:\n - Dynamic scratch buffers (bpf_idmap, bpf_idset) are managed with bpf_id_scratch_reserve() where written entries are initialized before use and freed on exit.\n - Dynamic liveness frame masks (struct frame_masks) are allocated using kvzalloc_flex() and widened with zero-padding intact.\n - Backtracking state (bt) and its stack masks are zero-initialized via bt_reset() (memset) and bitmap_zero().\n - The jump history entries (bpf_jmp_history_entry) properly track linked_regs_cnt and initialize all fields prior to read/unpacking.\n3. BPF verifier still enforces stack slot initialization semantics (tracking STACK_INVALID, STACK_MISC, STACK_ZERO, etc.) over the expanded stack range.\n4. Any potential issues stemming from this change would be stack overflows, verifier state tracking inconsistencies, out-of-bounds accesses during liveness tracking or backtracking, or memory corruption, which are covered by KASAN, stack guards, and other standard debug sanitizers.\n\nBecause no uninitialized memory risks or kernel-to-user info-leaks are introduced, a dedicated KMSAN fuzzing session is not warranted.",
"NeedsKMSAN": false
}Results: {
"KMSANReasoning": "The patch increases the maximum stack depth budget for BPF programs from 512 bytes (MAX_BPF_STACK) to 2048 bytes (MAX_BPF_STACK_JIT) on architectures whose JIT supports large stacks (x86-64 and arm64). It updates the BPF verifier's stack tracking, liveness analysis, backtracking, and scratch data structures to handle up to 2 KiB frames.\n\nAnalysis of memory safety and detector applicability:\n1. No kernel structures or padding are copied or leaked to user space via copy_to_user or BPF buffers.\n2. All modified and newly allocated data structures are properly initialized:\n - Dynamic scratch buffers (bpf_idmap, bpf_idset) are managed with bpf_id_scratch_reserve() where written entries are initialized before use and freed on exit.\n - Dynamic liveness frame masks (struct frame_masks) are allocated using kvzalloc_flex() and widened with zero-padding intact.\n - Backtracking state (bt) and its stack masks are zero-initialized via bt_reset() (memset) and bitmap_zero().\n - The jump history entries (bpf_jmp_history_entry) properly track linked_regs_cnt and initialize all fields prior to read/unpacking.\n3. BPF verifier still enforces stack slot initialization semantics (tracking STACK_INVALID, STACK_MISC, STACK_ZERO, etc.) over the expanded stack range.\n4. Any potential issues stemming from this change would be stack overflows, verifier state tracking inconsistencies, out-of-bounds accesses during liveness tracking or backtracking, or memory corruption, which are covered by KASAN, stack guards, and other standard debug sanitizers.\n\nBecause no uninitialized memory risks or kernel-to-user info-leaks are introduced, a dedicated KMSAN fuzzing session is not warranted.",
"NeedsKMSAN": false
} |
| Total Calls | Total Tokens | Avg Tokens | Total Duration (Seconds) | Avg Duration (Seconds) |
|---|
| Total Calls | Total Duration (Seconds) | Avg Duration (Seconds) |
|---|