AI Trajectory

Trajectory: (Expand/Collapse All)
Seq	Timestamp	Type	Name	Duration
0/0	2026/08/13 16:51	flow	patch-triage	1m Args: null Results: { "EnableConfigs": [ "BPF_SYSCALL" ], "FocusSymbols": [ "bpf_do_misc_fixups", "bpf_map_direct_read", "percpu_array_map_direct_value_addr", "percpu_array_map_direct_value_meta" ], "KMSANReasoning": "The patch series adds support for global percpu data in BPF by allowing `.percpu` sections to be mapped to `BPF_MAP_TYPE_PERCPU_ARRAY` maps. The changes involve libbpf parsing the new section, bpftool generating skeleton code, and the BPF verifier handling direct value access and JIT patching for percpu arrays.\n\nThere are no new kernel structures sent to user space, no new uninitialized buffers allocated, and no complex data structures where padding might be leaked. The percpu array memory is allocated using existing map allocation routines (which handle initialization) and populated by libbpf from the ELF section.\n\nAny potential bugs introduced by these changes would likely be out-of-bounds accesses, invalid JIT patching, or verifier logic errors, which are well-covered by KASAN and standard bug detectors. There is no specific risk of uninitialized memory usage or info-leaks that would require a dedicated KMSAN fuzzing session.", "NeedsKMSAN": false, "Reasoning": "The patch introduces support for global percpu data in BPF programs by allowing `.percpu` sections to be backed by `BPF_MAP_TYPE_PERCPU_ARRAY`. It modifies the BPF verifier to handle direct memory reads from percpu arrays and updates JIT fixups to support direct loading of percpu variables. These changes affect core BPF verifier logic and map operations, which are reachable from user space via the `bpf()` syscall when loading and executing BPF programs.", "WorthFuzzing": true }
1/1	2026/08/13 16:51	action	read-patch-diff	0m Args: null Results: { "PatchDiff": "commit 3b76f69b75942f2595d46bc813574f1caa6e529f\nAuthor: syz-cluster \u003ctriage@syzkaller.com\u003e\nDate: Thu Aug 13 16:51:17 2026 +0000\n\n syz-cluster: applied patch under review\n\ndiff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c\nindex 248b4818178cd..34865701f7f75 100644\n--- a/kernel/bpf/arraymap.c\n+++ b/kernel/bpf/arraymap.c\n@@ -259,6 +259,37 @@ static void percpu_array_map_lookup_elem(struct bpf_map map, void key)\n \treturn this_cpu_ptr(array-\u003epptrs[index \u0026 array-\u003eindex_mask]);\n }\n \n+static int percpu_array_map_direct_value_addr(const struct bpf_map map, u64 imm, u32 off)\n+{\n+\tstruct bpf_array array = container_of(map, struct bpf_array, map);\n+\n+\tif (!bpf_jit_supports_percpu_insn())\n+\t\treturn -EOPNOTSUPP;\n+\tif (map-\u003emax_entries != 1)\n+\t\treturn -EOPNOTSUPP;\n+\tif (off \u003e= map-\u003evalue_size)\n+\t\treturn -EINVAL;\n+\n+\timm = (u64)(__force unsigned long) array-\u003epptrs[0];\n+\treturn 0;\n+}\n+\n+static int percpu_array_map_direct_value_meta(const struct bpf_map map, u64 imm, u32 off)\n+{\n+\tstruct bpf_array array = container_of(map, struct bpf_array, map);\n+\tu64 base = (u64)(__force unsigned long) array-\u003epptrs[0];\n+\n+\tif (!bpf_jit_supports_percpu_insn())\n+\t\treturn -EOPNOTSUPP;\n+\tif (map-\u003emax_entries != 1)\n+\t\treturn -EOPNOTSUPP;\n+\tif (imm \u003c base \|\| imm \u003e= base + array-\u003eelem_size)\n+\t\treturn -ENOENT;\n+\n+\toff = imm - base;\n+\treturn 0;\n+}\n+\n / emit BPF instructions equivalent to C code of percpu_array_map_lookup_elem() /\n static int percpu_array_map_gen_lookup(struct bpf_map map, struct bpf_insn insn_buf)\n {\n@@ -551,9 +582,10 @@ static int array_map_check_btf(struct bpf_map map,\n \t\t\t const struct btf_type key_type,\n \t\t\t const struct btf_type value_type)\n {\n-\t/* One exception for keyless BTF: .bss/.data/.rodata map /\n+\t/ One exception for keyless BTF: .bss/.data/.rodata/.percpu map /\n \tif (btf_type_is_void(key_type)) {\n-\t\tif (map-\u003emap_type != BPF_MAP_TYPE_ARRAY \|\|\n+\t\tif ((map-\u003emap_type != BPF_MAP_TYPE_ARRAY \u0026\u0026\n+\t\t map-\u003emap_type != BPF_MAP_TYPE_PERCPU_ARRAY) \|\|\n \t\t map-\u003emax_entries != 1)\n \t\t\treturn -EINVAL;\n \n@@ -832,6 +864,8 @@ const struct bpf_map_ops percpu_array_map_ops = {\n \t.map_get_next_key = bpf_array_get_next_key,\n \t.map_lookup_elem = percpu_array_map_lookup_elem,\n \t.map_gen_lookup = percpu_array_map_gen_lookup,\n+\t.map_direct_value_addr = percpu_array_map_direct_value_addr,\n+\t.map_direct_value_meta = percpu_array_map_direct_value_meta,\n \t.map_update_elem = array_map_update_elem,\n \t.map_delete_elem = array_map_delete_elem,\n \t.map_lookup_percpu_elem = percpu_array_map_lookup_percpu_elem,\ndiff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c\nindex 40bd04421a991..a2b18a9f1694c 100644\n--- a/kernel/bpf/backtrack.c\n+++ b/kernel/bpf/backtrack.c\n@@ -214,7 +214,6 @@ static inline bool bt_is_reg_set(struct backtrack_state bt, u32 reg)\n \treturn bt-\u003ereg_masks[bt-\u003eframe] \u0026 (1 \u003c\u003c reg);\n }\n \n-\n /* format registers bitmask, e.g., \"r0,r2,r4\" for 0x15 mask /\n static void fmt_reg_mask(char buf, ssize_t buf_sz, u32 reg_mask)\n {\n@@ -254,7 +253,6 @@ void bpf_fmt_stack_mask(char buf, ssize_t buf_sz, u64 stack_mask)\n \t}\n }\n \n-\n / For given verifier state backtrack_insn() is called from the last insn to\n * the first insn. Its purpose is to compute a bitmask of registers and\n * stack slots that needs precision in the parent verifier state.\ndiff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c\nindex 6606187ed4f43..87ffde865a503 100644\n--- a/kernel/bpf/btf.c\n+++ b/kernel/bpf/btf.c\n@@ -2534,7 +2534,6 @@ static void btf_bitfield_show(void data, u8 bits_offset,\n \tbtf_int128_print(show, print_num);\n }\n \n-\n static void btf_int_bits_show(const struct btf btf,\n \t\t\t const struct btf_type t,\n \t\t\t void data, u8 bits_offset,\ndiff --git a/kernel/bpf/cfg.c b/kernel/bpf/cfg.c\nindex db3416a7c9047..818f7afac83a5 100644\n--- a/kernel/bpf/cfg.c\n+++ b/kernel/bpf/cfg.c\n@@ -47,7 +47,6 @@ enum {\n \tBRANCH = 2,\n };\n \n-\n static void mark_subprog_changes_pkt_data(struct bpf_verifier_env env, int off)\n {\n \tstruct bpf_subprog_info subprog;\ndiff --git a/kernel/bpf/const_fold.c b/kernel/bpf/const_fold.c\nindex 4cf120c7b2cb4..7f1b30059cc87 100644\n--- a/kernel/bpf/const_fold.c\n+++ b/kernel/bpf/const_fold.c\n@@ -182,7 +182,6 @@ static void const_reg_xfer(struct bpf_verifier_env env, struct const_arg_info \n \t\tu64 val = 0;\n \n \t\tif (!bpf_map_is_rdonly(map) \|\| !map-\u003eops-\u003emap_direct_value_addr \|\|\n-\t\t map-\u003emap_type == BPF_MAP_TYPE_INSN_ARRAY \|\|\n \t\t off \u003c 0 \|\| off + size \u003e map-\u003evalue_size \|\|\n \t\t bpf_map_direct_read(map, off, size, \u0026val, is_ldsx)) {\n \t\t\tdst = unknown;\ndiff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c\nindex 2417a3461652d..177a3fcbb63ac 100644\n--- a/kernel/bpf/fixups.c\n+++ b/kernel/bpf/fixups.c\n@@ -1466,7 +1466,6 @@ int bpf_fixup_call_args(struct bpf_verifier_env env)\n \treturn err;\n }\n \n-\n /* The function requires that first instruction in 'patch' is insnsi[prog-\u003elen - 1] /\n static int add_hidden_subprog(struct bpf_verifier_env env, struct bpf_insn patch, int len)\n {\n@@ -1835,6 +1834,43 @@ int bpf_do_misc_fixups(struct bpf_verifier_env env)\n \t\t\tgoto next_insn;\n \t\t}\n \n+\t\tif (bpf_jit_supports_percpu_insn() \u0026\u0026\n+\t\t insn-\u003ecode == (BPF_LD \| BPF_IMM \| BPF_DW) \u0026\u0026\n+\t\t (insn-\u003esrc_reg == BPF_PSEUDO_MAP_VALUE \|\|\n+\t\t insn-\u003esrc_reg == BPF_PSEUDO_MAP_IDX_VALUE)) {\n+\t\t\tstruct bpf_map map;\n+\n+\t\t\taux = \u0026env-\u003einsn_aux_data[i + delta];\n+\t\t\tmap = env-\u003eused_maps[aux-\u003emap_index];\n+\t\t\tif (map-\u003emap_type != BPF_MAP_TYPE_PERCPU_ARRAY)\n+\t\t\t\tgoto next_insn;\n+\n+\t\t\tprog-\u003ejit_required = true;\n+\n+\t\t\t/\n+\t\t\t * We are skipping first half of ld_imm64 insn\n+\t\t\t * with 'i++;', patching over second half of it\n+\t\t\t * with that same half + mov64_percpu_reg insn.\n+\t\t\t * All because bpf_patch_insn_data() can only\n+\t\t\t * replace one 8-byte insn, which does not work\n+\t\t\t * well for ld_imm64 insn.\n+\t\t\t /\n+\n+\t\t\tinsn_buf[0] = insn[1];\n+\t\t\tinsn_buf[1] = BPF_MOV64_PERCPU_REG(insn-\u003edst_reg, insn-\u003edst_reg);\n+\t\t\tcnt = 2;\n+\n+\t\t\ti++;\n+\t\t\tnew_prog = bpf_patch_insn_data(env, i + delta, insn_buf, cnt);\n+\t\t\tif (!new_prog)\n+\t\t\t\treturn -ENOMEM;\n+\n+\t\t\tdelta += cnt - 1;\n+\t\t\tenv-\u003eprog = prog = new_prog;\n+\t\t\tinsn = new_prog-\u003einsnsi + i + delta;\n+\t\t\tgoto next_insn;\n+\t\t}\n+\n \t\tif (insn-\u003ecode != (BPF_JMP \| BPF_CALL))\n \t\t\tgoto next_insn;\n \t\tif (insn-\u003esrc_reg == BPF_PSEUDO_CALL)\ndiff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c\nindex 9f394e1aa2e85..d40cb5dd446ca 100644\n--- a/kernel/bpf/hashtab.c\n+++ b/kernel/bpf/hashtab.c\n@@ -998,7 +998,6 @@ static void dec_elem_count(struct bpf_htab htab)\n \t\tatomic_dec(\u0026htab-\u003ecount);\n }\n \n-\n static void free_htab_elem(struct bpf_htab htab, struct htab_elem l)\n {\n \thtab_put_fd_value(htab, l);\n@@ -2970,7 +2969,6 @@ static int rhtab_delete_elem(struct bpf_rhtab rhtab, struct rhtab_elem elem, v\n \treturn 0;\n }\n \n-\n static long rhtab_map_delete_elem(struct bpf_map map, void key)\n {\n \tstruct bpf_rhtab rhtab = container_of(map, struct bpf_rhtab, map);\ndiff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c\nindex 6388b6b23e490..45e2f19387b24 100644\n--- a/kernel/bpf/helpers.c\n+++ b/kernel/bpf/helpers.c\n@@ -4871,7 +4871,6 @@ static const struct btf_kfunc_id_set generic_kfunc_set = {\n \t.set = \u0026generic_btf_ids,\n };\n \n-\n BTF_ID_LIST(generic_dtor_ids)\n BTF_ID(struct, task_struct)\n BTF_ID(func, bpf_task_release_dtor)\ndiff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c\nindex 1c997aeba6fa5..74fc4b3f80d6e 100644\n--- a/kernel/bpf/liveness.c\n+++ b/kernel/bpf/liveness.c\n@@ -269,7 +269,6 @@ bpf_insn_successors(struct bpf_verifier_env env, u32 idx)\n \n __diag_pop();\n \n-\n static inline bool update_insn(struct bpf_verifier_env env,\n \t\t\t struct func_instance instance, u32 frame, u32 insn_idx)\n {\n@@ -1862,7 +1861,6 @@ static int analyze_subprog(struct bpf_verifier_env env,\n \tif (need_resched())\n \t\tcond_resched();\n \n-\n \t/\n \t * When an instance is reused (must_write_initialized == true),\n \t * record into a fresh instance and merge afterward. This avoids\ndiff --git a/kernel/bpf/queue_stack_maps.c b/kernel/bpf/queue_stack_maps.c\nindex c1c9dee4dcdd0..6e8b18c32a106 100644\n--- a/kernel/bpf/queue_stack_maps.c\n+++ b/kernel/bpf/queue_stack_maps.c\n@@ -123,7 +123,6 @@ static long __queue_map_get(struct bpf_map map, void value, bool delete)\n \treturn err;\n }\n \n-\n static long __stack_map_get(struct bpf_map map, void value, bool delete)\n {\n \tstruct bpf_queue_stack qs = bpf_queue_stack(map);\ndiff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c\nindex 8d111da886553..7d8c3e8e6d62e 100644\n--- a/kernel/bpf/syscall.c\n+++ b/kernel/bpf/syscall.c\n@@ -636,7 +636,6 @@ int bpf_map_alloc_pages(const struct bpf_map map, int nid,\n \treturn ret;\n }\n \n-\n static int btf_field_cmp(const void a, const void b)\n {\n \tconst struct btf_field f1 = a, f2 = b;\n@@ -1830,7 +1829,6 @@ static int map_lookup_elem(union bpf_attr attr)\n \treturn err;\n }\n \n-\n #define BPF_MAP_UPDATE_ELEM_LAST_FIELD flags\n \n static int map_update_elem(union bpf_attr attr, bpfptr_t uattr)\n@@ -3497,7 +3495,6 @@ int bpf_link_prime(struct bpf_link link, struct bpf_link_primer primer)\n \tif (fd \u003c 0)\n \t\treturn fd;\n \n-\n \tid = bpf_link_alloc_id(link);\n \tif (id \u003c 0) {\n \t\tput_unused_fd(fd);\n@@ -5505,7 +5502,6 @@ static int bpf_link_get_info_by_fd(struct file file,\n \treturn 0;\n }\n \n-\n static int token_get_info_by_fd(struct file file,\n \t\t\t\tstruct bpf_token token,\n \t\t\t\tconst union bpf_attr attr,\n@@ -6507,7 +6503,6 @@ BPF_CALL_3(bpf_sys_bpf, int, cmd, union bpf_attr , attr, u32, attr_size)\n \treturn __sys_bpf(cmd, KERNEL_BPFPTR(attr), attr_size, KERNEL_BPFPTR(NULL), 0);\n }\n \n-\n / To shut up -Wmissing-prototypes.\n * This function is used by the kernel light skeleton\n * to load bpf programs when modules are loaded or during kernel boot.\ndiff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c\nindex 164d16c243ca6..6ac1afced20bf 100644\n--- a/kernel/bpf/verifier.c\n+++ b/kernel/bpf/verifier.c\n@@ -635,7 +635,6 @@ static void __mark_dynptr_reg(struct bpf_reg_state reg,\n \t\t\t enum bpf_dynptr_type type,\n \t\t\t bool first_slot, int id, int parent_id);\n \n-\n static void mark_dynptr_stack_regs(struct bpf_verifier_env env,\n \t\t\t\t struct bpf_reg_state sreg1,\n \t\t\t\t struct bpf_reg_state sreg2,\n@@ -1674,7 +1673,6 @@ static bool same_callsites(struct bpf_verifier_state a, struct bpf_verifier_sta\n \treturn true;\n }\n \n-\n void bpf_free_backedges(struct bpf_scc_visit visit)\n {\n \tstruct bpf_scc_backedge backedge, next;\n@@ -2291,7 +2289,6 @@ static struct bpf_verifier_state push_async_cb(struct bpf_verifier_env env,\n \treturn \u0026elem-\u003est;\n }\n \n-\n static int cmp_subprogs(const void a, const void b)\n {\n \treturn ((struct bpf_subprog_info )a)-\u003estart -\n@@ -3969,7 +3966,6 @@ static int check_stack_read(struct bpf_verifier_env env,\n \treturn err;\n }\n \n-\n /* check_stack_write dispatches to check_stack_write_fixed_off or\n * check_stack_write_var_off.\n \n@@ -4767,7 +4763,6 @@ static int check_sock_access(struct bpf_verifier_env env, int insn_idx,\n \t\tvalid = false;\n \t}\n \n-\n \tif (valid) {\n \t\tenv-\u003einsn_aux_data[insn_idx].ctx_field_size =\n \t\t\tinfo.ctx_field_size;\n@@ -5587,6 +5582,8 @@ int bpf_map_direct_read(struct bpf_map map, int off, int size, u64 val,\n \tu64 addr;\n \tint err;\n \n+\tif (map-\u003emap_type == BPF_MAP_TYPE_INSN_ARRAY \|\| map-\u003emap_type == BPF_MAP_TYPE_PERCPU_ARRAY)\n+\t\treturn -EINVAL;\n \terr = map-\u003eops-\u003emap_direct_value_addr(map, \u0026addr, off);\n \tif (err)\n \t\treturn err;\n@@ -6083,6 +6080,51 @@ static void add_scalar_to_reg(struct bpf_reg_state dst_reg, s64 val)\n \treg_bounds_sync(dst_reg);\n }\n \n+static int check_map_mem_read(struct bpf_verifier_env env, struct bpf_reg_state reg, int off,\n+\t\t\t int bpf_size, int value_regno, bool is_ldsx)\n+{\n+\tstruct bpf_reg_state regs = cur_regs(env);\n+\tint size = bpf_size_to_bytes(bpf_size);\n+\tstruct bpf_map map = reg-\u003emap_ptr;\n+\n+\tswitch (map-\u003emap_type) {\n+\tcase BPF_MAP_TYPE_INSN_ARRAY:\n+\t\tif (bpf_size != BPF_DW) {\n+\t\t\tverbose(env, \"Invalid read of %d bytes from insn_array\\n\", size);\n+\t\t\treturn -EACCES;\n+\t\t}\n+\t\tregs[value_regno] = reg;\n+\t\tadd_scalar_to_reg(\u0026regs[value_regno], off);\n+\t\tregs[value_regno].type = PTR_TO_INSN;\n+\t\treturn 0;\n+\tcase BPF_MAP_TYPE_PERCPU_ARRAY:\n+\t\tgoto reg_unknown;\n+\tdefault:\n+\t\tbreak;\n+\t}\n+\n+\t/* If map is read-only, track its contents as scalars. /\n+\tif (tnum_is_const(reg-\u003evar_off) \u0026\u0026\n+\t bpf_map_is_rdonly(map) \u0026\u0026\n+\t map-\u003eops-\u003emap_direct_value_addr) {\n+\t\tint map_off = off + reg-\u003evar_off.value;\n+\t\tu64 val = 0;\n+\t\tint err;\n+\n+\t\terr = bpf_map_direct_read(map, map_off, size, \u0026val, is_ldsx);\n+\t\tif (err)\n+\t\t\treturn err;\n+\n+\t\tregs[value_regno].type = SCALAR_VALUE;\n+\t\t__mark_reg_known(\u0026regs[value_regno], val);\n+\t\treturn 0;\n+\t}\n+\n+reg_unknown:\n+\tmark_reg_unknown(env, regs, value_regno);\n+\treturn 0;\n+}\n+\n / check whether memory at (regno + off) is accessible for t = (read \| write)\n * if t==write, value_regno is a register which value is stored into memory\n * if t==read, value_regno is a register which will receive the value from memory\n@@ -6137,38 +6179,7 @@ static int check_mem_access(struct bpf_verifier_env env, int insn_idx, struct b\n \t\tif (kptr_field) {\n \t\t\terr = check_map_kptr_access(env, value_regno, insn_idx, kptr_field);\n \t\t} else if (t == BPF_READ \u0026\u0026 value_regno \u003e= 0) {\n-\t\t\tstruct bpf_map map = reg-\u003emap_ptr;\n-\n-\t\t\t/\n-\t\t\t If map is read-only, track its contents as scalars,\n-\t\t\t * unless it is an insn array (see the special case below)\n-\t\t\t /\n-\t\t\tif (tnum_is_const(reg-\u003evar_off) \u0026\u0026\n-\t\t\t bpf_map_is_rdonly(map) \u0026\u0026\n-\t\t\t map-\u003eops-\u003emap_direct_value_addr \u0026\u0026\n-\t\t\t map-\u003emap_type != BPF_MAP_TYPE_INSN_ARRAY) {\n-\t\t\t\tint map_off = off + reg-\u003evar_off.value;\n-\t\t\t\tu64 val = 0;\n-\n-\t\t\t\terr = bpf_map_direct_read(map, map_off, size,\n-\t\t\t\t\t\t\t \u0026val, is_ldsx);\n-\t\t\t\tif (err)\n-\t\t\t\t\treturn err;\n-\n-\t\t\t\tregs[value_regno].type = SCALAR_VALUE;\n-\t\t\t\t__mark_reg_known(\u0026regs[value_regno], val);\n-\t\t\t} else if (map-\u003emap_type == BPF_MAP_TYPE_INSN_ARRAY) {\n-\t\t\t\tif (bpf_size != BPF_DW) {\n-\t\t\t\t\tverbose(env, \"Invalid read of %d bytes from insn_array\\n\",\n-\t\t\t\t\t\t size);\n-\t\t\t\t\treturn -EACCES;\n-\t\t\t\t}\n-\t\t\t\tregs[value_regno] = reg;\n-\t\t\t\tadd_scalar_to_reg(\u0026regs[value_regno], off);\n-\t\t\t\tregs[value_regno].type = PTR_TO_INSN;\n-\t\t\t} else {\n-\t\t\t\tmark_reg_unknown(env, regs, value_regno);\n-\t\t\t}\n+\t\t\terr = check_map_mem_read(env, reg, off, bpf_size, value_regno, is_ldsx);\n \t\t}\n \t} else if (base_type(reg-\u003etype) == PTR_TO_MEM) {\n \t\tbool rdonly_mem = type_is_rdonly_mem(reg-\u003etype);\n@@ -6635,7 +6646,6 @@ static int check_stack_range_initialized(\n \tif (err)\n \t\treturn err;\n \n-\n \tif (tnum_is_const(reg-\u003evar_off)) {\n \t\tmin_off = max_off = reg-\u003evar_off.value + off;\n \t} else {\n@@ -7347,7 +7357,6 @@ static bool is_iter_new_kfunc(struct bpf_call_arg_meta meta)\n \treturn meta-\u003ekfunc_flags \u0026 KF_ITER_NEW;\n }\n \n-\n static bool is_iter_destroy_kfunc(struct bpf_call_arg_meta meta)\n {\n \treturn meta-\u003ekfunc_flags \u0026 KF_ITER_DESTROY;\n@@ -8125,6 +8134,12 @@ static int check_arg_const_str(struct bpf_verifier_env env,\n \t\treturn -EACCES;\n \t}\n \n+\tif (map-\u003emap_type == BPF_MAP_TYPE_PERCPU_ARRAY) {\n+\t\tverbose(env, \"%s points to percpu_array map which cannot be used as const string\\n\",\n+\t\t\treg_arg_name(env, argno));\n+\t\treturn -EACCES;\n+\t}\n+\n \tif (!bpf_map_is_rdonly(map)) {\n \t\tverbose(env, \"%s does not point to a readonly map'\\n\", reg_arg_name(env, argno));\n \t\treturn -EACCES;\n@@ -11607,7 +11622,6 @@ static int process_irq_flag(struct bpf_verifier_env env, struct bpf_reg_state \n \treturn 0;\n }\n \n-\n static int ref_set_non_owning(struct bpf_verifier_env env, struct bpf_reg_state reg)\n {\n \tstruct btf_record rec = reg_btf_record(reg);\n@@ -16412,7 +16426,6 @@ static int check_ld_abs(struct bpf_verifier_env env, struct bpf_insn insn)\n \treturn 0;\n }\n \n-\n static bool return_retval_range(struct bpf_verifier_env env, struct bpf_retval_range range)\n {\n \tenum bpf_prog_type prog_type = resolve_prog_type(env-\u003eprog);\n@@ -18361,8 +18374,6 @@ static void release_insn_arrays(struct bpf_verifier_env env)\n \t\tbpf_insn_array_release(env-\u003einsn_array_maps[i]);\n }\n \n-\n-\n / The verifier does more data flow analysis than llvm and will not\n * explore branches that are dead at run time. Malicious programs can\n * have dead code too. Therefore replace all dead at-run-time code\n@@ -18390,8 +18401,6 @@ static void sanitize_dead_code(struct bpf_verifier_env env)\n \t}\n }\n \n-\n-\n static void free_states(struct bpf_verifier_env env)\n {\n \tstruct bpf_verifier_state_list sl;\n@@ -18678,7 +18687,6 @@ static int do_check_main(struct bpf_verifier_env env)\n \treturn ret;\n }\n \n-\n static void print_verification_stats(struct bpf_verifier_env env)\n {\n \t/ Skip over hidden subprogs which are not verified. /\ndiff --git a/tools/bpf/bpftool/gen.c b/tools/bpf/bpftool/gen.c\nindex a01d06d22d1a3..a50540ef6521c 100644\n--- a/tools/bpf/bpftool/gen.c\n+++ b/tools/bpf/bpftool/gen.c\n@@ -101,6 +101,12 @@ static bool get_map_ident(const struct bpf_map map, char buf, size_t buf_sz)\n \t\treturn true;\n \t}\n \n+\tif (bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY) {\n+\t\tsnprintf(buf, buf_sz, \"%s\", name + 1);\n+\t\tsanitize_identifier(buf);\n+\t\treturn true;\n+\t}\n+\n \tfor (i = 0, n = ARRAY_SIZE(sfxs); i \u003c n; i++) {\n \t\tconst char sfx = sfxs[i], p;\n \n@@ -117,7 +123,7 @@ static bool get_map_ident(const struct bpf_map map, char buf, size_t buf_sz)\n \n static bool get_datasec_ident(const char sec_name, char buf, size_t buf_sz)\n {\n-\tstatic const char pfxs[] = { \".data\", \".rodata\", \".bss\", \".kconfig\" };\n+\tstatic const char pfxs[] = { \".data\", \".rodata\", \".bss\", \".percpu\", \".kconfig\" };\n \tint i, n;\n \n \t/ recognize hard coded LLVM section name /\n@@ -254,7 +260,7 @@ static const struct btf_type find_type_for_map(struct btf btf, const char map\n \treturn NULL;\n }\n \n-static bool is_mmapable_map(const struct bpf_map map, char buf, size_t sz)\n+static bool is_skel_data(const struct bpf_map map, char buf, size_t sz)\n {\n \tsize_t tmp_sz;\n \n@@ -263,13 +269,24 @@ static bool is_mmapable_map(const struct bpf_map map, char buf, size_t sz)\n \t\treturn true;\n \t}\n \n-\tif (!bpf_map__is_internal(map) \|\| !(bpf_map__map_flags(map) \u0026 BPF_F_MMAPABLE))\n+\tif (!bpf_map__is_internal(map))\n \t\treturn false;\n \n \tif (!get_map_ident(map, buf, sz))\n \t\treturn false;\n \n-\treturn true;\n+\tif (bpf_map__map_flags(map) \u0026 BPF_F_MMAPABLE)\n+\t\treturn true;\n+\n+\tif (bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY)\n+\t\treturn bpf_map__btf_value_type_id(map) != 0;\n+\n+\treturn false;\n+}\n+\n+static bool is_mmapable_map(const struct bpf_map map, char buf, size_t sz)\n+{\n+\treturn is_skel_data(map, buf, sz) \u0026\u0026 bpf_map__type(map) != BPF_MAP_TYPE_PERCPU_ARRAY;\n }\n \n static int codegen_datasecs(struct bpf_object obj, const char obj_name)\n@@ -287,7 +304,7 @@ static int codegen_datasecs(struct bpf_object obj, const char obj_name)\n \n \tbpf_object__for_each_map(map, obj) {\n \t\t/* only generate definitions for memory-mapped internal maps /\n-\t\tif (!is_mmapable_map(map, map_ident, sizeof(map_ident)))\n+\t\tif (!is_skel_data(map, map_ident, sizeof(map_ident)))\n \t\t\tcontinue;\n \n \t\tsec = find_type_for_map(btf, map_ident);\n@@ -517,7 +534,7 @@ static void codegen_asserts(struct bpf_object obj, const char obj_name)\n \t\t\", obj_name);\n \n \tbpf_object__for_each_map(map, obj) {\n-\t\tif (!is_mmapable_map(map, map_ident, sizeof(map_ident)))\n+\t\tif (!is_skel_data(map, map_ident, sizeof(map_ident)))\n \t\t\tcontinue;\n \n \t\tsec = find_type_for_map(btf, map_ident);\n@@ -668,8 +685,7 @@ static void codegen_destroy(struct bpf_object obj, const char obj_name)\n \tbpf_object__for_each_map(map, obj) {\n \t\tif (!get_map_ident(map, ident, sizeof(ident)))\n \t\t\tcontinue;\n-\t\tif (bpf_map__is_internal(map) \u0026\u0026\n-\t\t (bpf_map__map_flags(map) \u0026 BPF_F_MMAPABLE))\n+\t\tif (is_skel_data(map, ident, sizeof(ident)))\n \t\t\tprintf(\"\\tskel_free_map_data(skel-\u003e%1$s, skel-\u003emaps.%1$s.initial_value, %2$zu);\\n\",\n \t\t\t ident, bpf_map_mmap_sz(map));\n \t\tcodegen(\"\\\n@@ -741,7 +757,7 @@ static int gen_trace(struct bpf_object obj, const char obj_name, const char h\n \t\tconst void mmap_data = NULL;\n \t\tsize_t mmap_size = 0;\n \n-\t\tif (!is_mmapable_map(map, ident, sizeof(ident)))\n+\t\tif (!is_skel_data(map, ident, sizeof(ident)))\n \t\t\tcontinue;\n \n \t\tcodegen(\"\\\n@@ -849,9 +865,23 @@ static int gen_trace(struct bpf_object obj, const char obj_name, const char h\n \tbpf_object__for_each_map(map, obj) {\n \t\tconst char mmap_flags;\n \n-\t\tif (!is_mmapable_map(map, ident, sizeof(ident)))\n+\t\tif (!is_skel_data(map, ident, sizeof(ident)))\n \t\t\tcontinue;\n \n+\t\tif (bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY) {\n+\t\t\tcodegen(\"\\\n+\t\t\\n\\\n+\t\t\terr = skel_protect_map_data(skel-\u003e%1$s, \u0026skel-\u003emaps.%1$s.initial_value, %2$zd);\\n\\\n+\t\t\tif (err)\t\t\t\t\t \\n\\\n+\t\t\t\treturn err;\t\t\t\t \\n\\\n+\t\t#ifdef __KERNEL__\t\t\t\t\t \\n\\\n+\t\t\tskel-\u003e%1$s = NULL;\t\t\t\t \\n\\\n+\t\t#endif\t\t\t\t\t\t\t \\n\\\n+\t\t\t\",\n+\t\t\tident, bpf_map_mmap_sz(map));\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\tif (bpf_map__map_flags(map) \u0026 BPF_F_RDONLY_PROG)\n \t\t\tmmap_flags = \"PROT_READ\";\n \t\telse\n@@ -955,8 +985,7 @@ codegen_maps_skeleton(struct bpf_object obj, size_t map_cnt, bool mmaped, bool\n \t\t\t\tmap-\u003emap = \u0026obj-\u003emaps.%s;\t \\n\\\n \t\t\t\",\n \t\t\ti, bpf_map__name(map), ident);\n-\t\t/* memory-mapped internal maps /\n-\t\tif (mmaped \u0026\u0026 is_mmapable_map(map, ident, sizeof(ident))) {\n+\t\tif (mmaped \u0026\u0026 is_skel_data(map, ident, sizeof(ident))) {\n \t\t\tprintf(\"\\tmap-\u003emmaped = (void )\u0026obj-\u003e%s;\\n\", ident);\n \t\t}\n \ndiff --git a/tools/lib/bpf/bpf_gen_internal.h b/tools/lib/bpf/bpf_gen_internal.h\nindex 0425691877521..6c5ad6c55e8a6 100644\n--- a/tools/lib/bpf/bpf_gen_internal.h\n+++ b/tools/lib/bpf/bpf_gen_internal.h\n@@ -65,7 +65,8 @@ void bpf_gen__prog_load(struct bpf_gen gen,\n \t\t\tenum bpf_prog_type prog_type, const char prog_name,\n \t\t\tconst char license, struct bpf_insn insns, size_t insn_cnt,\n \t\t\tstruct bpf_prog_load_opts load_attr, int prog_idx);\n-void bpf_gen__map_update_elem(struct bpf_gen gen, int map_idx, void value, __u32 value_size);\n+void bpf_gen__map_update_elem(struct bpf_gen gen, int map_idx, void value, __u32 value_size,\n+\t\t\t __u64 flags);\n void bpf_gen__map_freeze(struct bpf_gen gen, int map_idx);\n void bpf_gen__record_attach_target(struct bpf_gen gen, const char name, enum bpf_attach_type type);\n void bpf_gen__record_extern(struct bpf_gen gen, const char name, bool is_weak,\ndiff --git a/tools/lib/bpf/features.c b/tools/lib/bpf/features.c\nindex b7e388f99d0bb..ef9581c113035 100644\n--- a/tools/lib/bpf/features.c\n+++ b/tools/lib/bpf/features.c\n@@ -620,6 +620,38 @@ static int probe_bpf_syscall_common_attrs(int token_fd)\n \treturn probe_sys_bpf_ext();\n }\n \n+static int probe_kern_percpu_data(int token_fd)\n+{\n+\tstruct bpf_insn insns[] = {\n+\t\tBPF_LD_MAP_VALUE(BPF_REG_1, 0, 0),\n+\t\tBPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, 0),\n+\t\tBPF_EXIT_INSN(),\n+\t};\n+\tLIBBPF_OPTS(bpf_map_create_opts, map_opts,\n+\t\t.token_fd = token_fd,\n+\t\t.map_flags = token_fd ? BPF_F_TOKEN_FD : 0,\n+\t);\n+\tLIBBPF_OPTS(bpf_prog_load_opts, prog_opts,\n+\t\t.token_fd = token_fd,\n+\t\t.prog_flags = token_fd ? BPF_F_TOKEN_FD : 0,\n+\t);\n+\tint ret, map, insn_cnt = ARRAY_SIZE(insns);\n+\n+\tmap = bpf_map_create(BPF_MAP_TYPE_PERCPU_ARRAY, \"libbpf_percpu\", sizeof(int), 8, 1,\n+\t\t\t \u0026map_opts);\n+\tif (map \u003c 0) {\n+\t\tpr_warn(\"Error in %s(): %s. Couldn't create simple percpu_array map.\\n\",\n+\t\t\t__func__, errstr(map));\n+\t\treturn map;\n+\t}\n+\n+\tinsns[0].imm = map;\n+\n+\tret = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, NULL, \"GPL\", insns, insn_cnt, \u0026prog_opts);\n+\tclose(map);\n+\treturn probe_fd(ret);\n+}\n+\n typedef int (feature_probe_fn)(int /* token_fd /);\n \n static struct kern_feature_cache feature_cache;\n@@ -707,6 +739,9 @@ static struct kern_feature_desc {\n \t[FEAT_BPF_SYSCALL_COMMON_ATTRS] = {\n \t\t\"BPF syscall common attributes support\", probe_bpf_syscall_common_attrs,\n \t},\n+\t[FEAT_PERCPU_DATA] = {\n+\t\t\"kernel supports percpu data\", probe_kern_percpu_data,\n+\t},\n };\n \n bool feat_supported(struct kern_feature_cache cache, enum kern_feature_id feat_id)\ndiff --git a/tools/lib/bpf/gen_loader.c b/tools/lib/bpf/gen_loader.c\nindex 6e3dd52427618..af3a04f161ac1 100644\n--- a/tools/lib/bpf/gen_loader.c\n+++ b/tools/lib/bpf/gen_loader.c\n@@ -1128,7 +1128,7 @@ void bpf_gen__prog_load(struct bpf_gen gen,\n }\n \n void bpf_gen__map_update_elem(struct bpf_gen gen, int map_idx, void pvalue,\n-\t\t\t __u32 value_size)\n+\t\t\t __u32 value_size, __u64 flags)\n {\n \tint attr_size = offsetofend(union bpf_attr, flags);\n \tint map_update_attr, value, key;\n@@ -1136,6 +1136,7 @@ void bpf_gen__map_update_elem(struct bpf_gen gen, int map_idx, void pvalue,\n \tint zero = 0;\n \n \tmemset(\u0026attr, 0, attr_size);\n+\tattr.flags = tgt_endian(flags);\n \n \tvalue = add_data(gen, pvalue, value_size);\n \tkey = add_data(gen, \u0026zero, sizeof(zero));\ndiff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c\nindex 514e4e9daa825..e574870fb7169 100644\n--- a/tools/lib/bpf/libbpf.c\n+++ b/tools/lib/bpf/libbpf.c\n@@ -541,6 +541,7 @@ struct bpf_struct_ops {\n };\n \n #define DATA_SEC \".data\"\n+#define PERCPU_SEC \".percpu\"\n #define BSS_SEC \".bss\"\n #define RODATA_SEC \".rodata\"\n #define KCONFIG_SEC \".kconfig\"\n@@ -555,6 +556,7 @@ enum libbpf_map_type {\n \tLIBBPF_MAP_BSS,\n \tLIBBPF_MAP_RODATA,\n \tLIBBPF_MAP_KCONFIG,\n+\tLIBBPF_MAP_PERCPU,\n };\n \n struct bpf_map_def {\n@@ -666,6 +668,7 @@ enum sec_type {\n \tSEC_DATA,\n \tSEC_RODATA,\n \tSEC_ST_OPS,\n+\tSEC_PERCPU,\n };\n \n struct elf_sec_desc {\n@@ -1839,6 +1842,8 @@ static size_t bpf_map_mmap_sz(const struct bpf_map map)\n \tswitch (map-\u003edef.type) {\n \tcase BPF_MAP_TYPE_ARRAY:\n \t\treturn array_map_mmap_sz(map-\u003edef.value_size, map-\u003edef.max_entries);\n+\tcase BPF_MAP_TYPE_PERCPU_ARRAY:\n+\t\treturn map-\u003edef.value_size;\n \tcase BPF_MAP_TYPE_ARENA:\n \t\treturn page_sz * map-\u003edef.max_entries;\n \tdefault:\n@@ -1866,7 +1871,8 @@ static int bpf_map_mmap_resize(struct bpf_map map, size_t old_sz, size_t new_sz\n \treturn 0;\n }\n \n-static char internal_map_name(struct bpf_object obj, const char real_name)\n+static char internal_map_name(struct bpf_object obj, const char real_name,\n+\t\t\t enum libbpf_map_type type)\n {\n \tchar map_name[BPF_OBJ_NAME_LEN], p;\n \tint pfx_len, sfx_len = max((size_t)7, strlen(real_name));\n@@ -1907,8 +1913,11 @@ static char internal_map_name(struct bpf_object obj, const char real_name)\n \tif (sfx_len \u003e= BPF_OBJ_NAME_LEN)\n \t\tsfx_len = BPF_OBJ_NAME_LEN - 1;\n \n-\t/ if there are two or more dots in map name, it's a custom dot map /\n-\tif (strchr(real_name + 1, '.') != NULL)\n+\t/\n+\t * Don't prefix the bpf_object name if this is a custom dot map\n+\t * (containing two or more dots) or a percpu data map.\n+\t /\n+\tif (strchr(real_name + 1, '.') != NULL \|\| type == LIBBPF_MAP_PERCPU)\n \t\tpfx_len = 0;\n \telse\n \t\tpfx_len = min((size_t)BPF_OBJ_NAME_LEN - sfx_len - 1, strlen(obj-\u003ename));\n@@ -1941,6 +1950,13 @@ static bool map_is_mmapable(struct bpf_object obj, struct bpf_map map)\n \tif (!map-\u003ebtf_value_type_id)\n \t\treturn false;\n \n+\t/\n+\t * The internal PERCPU maps are not mmapble because the underlying\n+\t * percpu_array maps do not have mmap support.\n+\t /\n+\tif (map-\u003elibbpf_type == LIBBPF_MAP_PERCPU)\n+\t\treturn false;\n+\n \tt = btf__type_by_id(obj-\u003ebtf, map-\u003ebtf_value_type_id);\n \tif (!btf_is_datasec(t))\n \t\treturn false;\n@@ -1962,6 +1978,7 @@ static int\n bpf_object__init_internal_map(struct bpf_object obj, enum libbpf_map_type type,\n \t\t\t const char real_name, int sec_idx, void data, size_t data_sz)\n {\n+\tbool is_percpu = type == LIBBPF_MAP_PERCPU;\n \tstruct bpf_map_def def;\n \tstruct bpf_map map;\n \tsize_t mmap_sz;\n@@ -1975,7 +1992,7 @@ bpf_object__init_internal_map(struct bpf_object obj, enum libbpf_map_type type,\n \tmap-\u003esec_idx = sec_idx;\n \tmap-\u003esec_offset = 0;\n \tmap-\u003ereal_name = strdup(real_name);\n-\tmap-\u003ename = internal_map_name(obj, real_name);\n+\tmap-\u003ename = internal_map_name(obj, real_name, type);\n \tif (!map-\u003ereal_name \|\| !map-\u003ename) {\n \t\tzfree(\u0026map-\u003ereal_name);\n \t\tzfree(\u0026map-\u003ename);\n@@ -1983,7 +2000,7 @@ bpf_object__init_internal_map(struct bpf_object obj, enum libbpf_map_type type,\n \t}\n \n \tdef = \u0026map-\u003edef;\n-\tdef-\u003etype = BPF_MAP_TYPE_ARRAY;\n+\tdef-\u003etype = is_percpu ? BPF_MAP_TYPE_PERCPU_ARRAY : BPF_MAP_TYPE_ARRAY;\n \tdef-\u003ekey_size = sizeof(int);\n \tdef-\u003evalue_size = data_sz;\n \tdef-\u003emax_entries = 1;\n@@ -1996,8 +2013,9 @@ bpf_object__init_internal_map(struct bpf_object obj, enum libbpf_map_type type,\n \tif (map_is_mmapable(obj, map))\n \t\tdef-\u003emap_flags \|= BPF_F_MMAPABLE;\n \n-\tpr_debug(\"map '%s' (global data): at sec_idx %d, offset %zu, flags %x.\\n\",\n-\t\t map-\u003ename, map-\u003esec_idx, map-\u003esec_offset, def-\u003emap_flags);\n+\tpr_debug(\"map '%s' (global %sdata): at sec_idx %d, offset %zu, flags %x.\\n\",\n+\t\t map-\u003ename, is_percpu ? \"percpu \" : \"\", map-\u003esec_idx,\n+\t\t map-\u003esec_offset, def-\u003emap_flags);\n \n \tmmap_sz = bpf_map_mmap_sz(map);\n \tmap-\u003emmaped = mmap(NULL, mmap_sz, PROT_READ \| PROT_WRITE,\n@@ -2057,6 +2075,13 @@ static int bpf_object__init_global_data_maps(struct bpf_object obj)\n \t\t\t\t\t\t\t NULL,\n \t\t\t\t\t\t\t sec_desc-\u003edata-\u003ed_size);\n \t\t\tbreak;\n+\t\tcase SEC_PERCPU:\n+\t\t\tsec_name = elf_sec_name(obj, elf_sec_by_idx(obj, sec_idx));\n+\t\t\terr = bpf_object__init_internal_map(obj, LIBBPF_MAP_PERCPU,\n+\t\t\t\t\t\t\t sec_name, sec_idx,\n+\t\t\t\t\t\t\t sec_desc-\u003edata-\u003ed_buf,\n+\t\t\t\t\t\t\t sec_desc-\u003edata-\u003ed_size);\n+\t\t\tbreak;\n \t\tdefault:\n \t\t\t/* skip /\n \t\t\tbreak;\n@@ -4016,6 +4041,11 @@ static int bpf_object__elf_collect(struct bpf_object obj)\n \t\t\t\tsec_desc-\u003esec_type = SEC_RODATA;\n \t\t\t\tsec_desc-\u003eshdr = sh;\n \t\t\t\tsec_desc-\u003edata = data;\n+\t\t\t} else if (strcmp(name, PERCPU_SEC) == 0 \|\|\n+\t\t\t\t str_has_pfx(name, PERCPU_SEC \".\")) {\n+\t\t\t\tsec_desc-\u003esec_type = SEC_PERCPU;\n+\t\t\t\tsec_desc-\u003eshdr = sh;\n+\t\t\t\tsec_desc-\u003edata = data;\n \t\t\t} else if (strcmp(name, STRUCT_OPS_SEC) == 0 \|\|\n \t\t\t\t strcmp(name, STRUCT_OPS_LINK_SEC) == 0 \|\|\n \t\t\t\t strcmp(name, \"?\" STRUCT_OPS_SEC) == 0 \|\|\n@@ -4544,6 +4574,7 @@ static bool bpf_object__shndx_is_data(const struct bpf_object obj,\n \tcase SEC_BSS:\n \tcase SEC_DATA:\n \tcase SEC_RODATA:\n+\tcase SEC_PERCPU:\n \t\treturn true;\n \tdefault:\n \t\treturn false;\n@@ -4569,6 +4600,8 @@ bpf_object__section_to_libbpf_map_type(const struct bpf_object obj, int shndx)\n \t\treturn LIBBPF_MAP_DATA;\n \tcase SEC_RODATA:\n \t\treturn LIBBPF_MAP_RODATA;\n+\tcase SEC_PERCPU:\n+\t\treturn LIBBPF_MAP_PERCPU;\n \tdefault:\n \t\treturn LIBBPF_MAP_UNSPEC;\n \t}\n@@ -4944,7 +4977,7 @@ static int map_fill_btf_type_info(struct bpf_object obj, struct bpf_map map)\n \n \t/\n \t LLVM annotates global data differently in BTF, that is,\n-\t * only as '.data', '.bss' or '.rodata'.\n+\t * only as '.data', '.bss', '.percpu' or '.rodata'.\n \t /\n \tif (!bpf_map__is_internal(map))\n \t\treturn -ENOENT;\n@@ -5293,18 +5326,20 @@ static int\n bpf_object__populate_internal_map(struct bpf_object obj, struct bpf_map map)\n {\n \tenum libbpf_map_type map_type = map-\u003elibbpf_type;\n+\tbool is_percpu = map_type == LIBBPF_MAP_PERCPU;\n+\tconst __u64 update_flags = is_percpu ? BPF_F_ALL_CPUS : 0;\n \tint err, zero = 0;\n \tsize_t mmap_sz;\n \n \tif (obj-\u003egen_loader) {\n \t\tbpf_gen__map_update_elem(obj-\u003egen_loader, map - obj-\u003emaps,\n-\t\t\t\t\t map-\u003emmaped, map-\u003edef.value_size);\n+\t\t\t\t\t map-\u003emmaped, map-\u003edef.value_size, update_flags);\n \t\tif (map_type == LIBBPF_MAP_RODATA \|\| map_type == LIBBPF_MAP_KCONFIG)\n \t\t\tbpf_gen__map_freeze(obj-\u003egen_loader, map - obj-\u003emaps);\n \t\treturn 0;\n \t}\n \n-\terr = bpf_map_update_elem(map-\u003efd, \u0026zero, map-\u003emmaped, 0);\n+\terr = bpf_map_update_elem(map-\u003efd, \u0026zero, map-\u003emmaped, update_flags);\n \tif (err) {\n \t\terr = -errno;\n \t\tpr_warn(\"map '%s': failed to set initial contents: %s\\n\",\n@@ -5349,6 +5384,13 @@ bpf_object__populate_internal_map(struct bpf_object obj, struct bpf_map map)\n \t\t\treturn err;\n \t\t}\n \t\tmap-\u003emmaped = mmaped;\n+\t} else if (is_percpu) {\n+\t\tif (mprotect(map-\u003emmaped, mmap_sz, PROT_READ)) {\n+\t\t\terr = -errno;\n+\t\t\tpr_warn(\"map '%s': failed to mprotect() contents: %s\\n\",\n+\t\t\t\tbpf_map__name(map), errstr(err));\n+\t\t\treturn err;\n+\t\t}\n \t} else if (map-\u003emmaped) {\n \t\tmunmap(map-\u003emmaped, mmap_sz);\n \t\tmap-\u003emmaped = NULL;\n@@ -5624,9 +5666,16 @@ bpf_object__create_maps(struct bpf_object obj)\n \t\t * runtime due to bpf_program__set_autoload(prog, false),\n \t\t * bpf_object loading will succeed just fine even on old\n \t\t * kernels.\n+\t\t * Same skipping applies to percpu data.\n \t\t /\n-\t\tif (bpf_map__is_internal(map) \u0026\u0026 !kernel_supports(obj, FEAT_GLOBAL_DATA))\n-\t\t\tmap-\u003eautocreate = false;\n+\t\tif (bpf_map__is_internal(map)) {\n+\t\t\tbool is_percpu = map-\u003elibbpf_type == LIBBPF_MAP_PERCPU;\n+\t\t\tenum kern_feature_id feat_id;\n+\n+\t\t\tfeat_id = is_percpu ? FEAT_PERCPU_DATA : FEAT_GLOBAL_DATA;\n+\t\t\tif (!kernel_supports(obj, feat_id))\n+\t\t\t\tmap-\u003eautocreate = false;\n+\t\t}\n \n \t\tif (!map-\u003eautocreate) {\n \t\t\tpr_debug(\"map '%s': skipped auto-creating...\\n\", map-\u003ename);\n@@ -10807,11 +10856,16 @@ static bool map_uses_real_name(const struct bpf_map map)\n \t * such map's corresponding ELF section name as a map name.\n \t * This check distinguishes .data/.rodata from .data.* and .rodata.\n \t maps to know which name has to be returned to the user.\n+\t * Map name of the custom .percpu.* maps might be truncated to\n+\t * BPF_OBJ_NAME_LEN-1 chars in internal_map_name(). Hence, percpu data\n+\t * maps must use real name for their user-visible name.\n \t /\n \tif (map-\u003elibbpf_type == LIBBPF_MAP_DATA \u0026\u0026 strcmp(map-\u003ereal_name, DATA_SEC) != 0)\n \t\treturn true;\n \tif (map-\u003elibbpf_type == LIBBPF_MAP_RODATA \u0026\u0026 strcmp(map-\u003ereal_name, RODATA_SEC) != 0)\n \t\treturn true;\n+\tif (map-\u003elibbpf_type == LIBBPF_MAP_PERCPU)\n+\t\treturn true;\n \treturn false;\n }\n \n@@ -10976,7 +11030,8 @@ int bpf_map__set_value_size(struct bpf_map map, __u32 size)\n \t\tsize_t mmap_old_sz, mmap_new_sz;\n \t\tint err;\n \n-\t\tif (map-\u003edef.type != BPF_MAP_TYPE_ARRAY)\n+\t\tif (map-\u003edef.type != BPF_MAP_TYPE_ARRAY \u0026\u0026\n+\t\t map-\u003edef.type != BPF_MAP_TYPE_PERCPU_ARRAY)\n \t\t\treturn libbpf_err(-EOPNOTSUPP);\n \n \t\tmmap_old_sz = bpf_map_mmap_sz(map);\ndiff --git a/tools/lib/bpf/libbpf_internal.h b/tools/lib/bpf/libbpf_internal.h\nindex 7a74abb904f8b..4c46d34fc055d 100644\n--- a/tools/lib/bpf/libbpf_internal.h\n+++ b/tools/lib/bpf/libbpf_internal.h\n@@ -401,6 +401,8 @@ enum kern_feature_id {\n \tFEAT_BTF_LAYOUT,\n \t/* Kernel supports BPF syscall common attributes /\n \tFEAT_BPF_SYSCALL_COMMON_ATTRS,\n+\t/ Kernel supports percpu data /\n+\tFEAT_PERCPU_DATA,\n \t__FEAT_CNT,\n };\n \ndiff --git a/tools/lib/bpf/skel_internal.h b/tools/lib/bpf/skel_internal.h\nindex 53fee53d36d51..1f3f332dffbe9 100644\n--- a/tools/lib/bpf/skel_internal.h\n+++ b/tools/lib/bpf/skel_internal.h\n@@ -131,8 +131,10 @@ static inline void skel_free_map_data(void p, __u64 addr, size_t sz)\n {\n \tif (addr != ~0ULL)\n \t\tkvfree(p);\n-\t/* When addr == ~0ULL the 'p' points to\n-\t * ((struct bpf_array )map)-\u003evalue. See skel_finalize_map_data.\n+\t/\n+\t * When addr == ~0ULL the init buffer has already been released.\n+\t * For skel_finalize_map_data(), 'p' points to\n+\t * ((struct bpf_array )map)-\u003evalue.\n \t /\n }\n \n@@ -170,6 +172,15 @@ static inline void skel_finalize_map_data(__u64 init_val, size_t mmap_sz, int\n \treturn addr;\n }\n \n+static inline int skel_protect_map_data(void p, __u64 init_val, size_t sz)\n+{\n+\t(void)sz;\n+\n+\tkvfree(p);\n+\tinit_val = ~0ULL;\n+\treturn 0;\n+}\n+\n #else\n \n static inline void skel_alloc(size_t size)\n@@ -208,6 +219,15 @@ static inline void skel_finalize_map_data(__u64 init_val, size_t mmap_sz, int\n \t\treturn NULL;\n \treturn addr;\n }\n+\n+static inline int skel_protect_map_data(void p, __u64 init_val, size_t sz)\n+{\n+\t(void)init_val;\n+\n+\tif (mprotect(p, sz, PROT_READ))\n+\t\treturn -errno;\n+\treturn 0;\n+}\n #endif\n \n static inline int skel_closenz(int fd)\ndiff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile\nindex d3655a7064826..560ce4016fbff 100644\n--- a/tools/testing/selftests/bpf/Makefile\n+++ b/tools/testing/selftests/bpf/Makefile\n@@ -531,7 +531,7 @@ LSKELS_SIGNED := fentry_test.c fexit_test.c atomics.c\n \n # Generate both light skeleton and libbpf skeleton for these\n LSKELS_EXTRA := test_ksyms_module.c test_ksyms_weak.c kfunc_call_test.c \\\n-\tkfunc_call_test_subprog.c\n+\tkfunc_call_test_subprog.c test_global_percpu_data.c\n SKEL_BLACKLIST += $$(LSKELS) $$(LSKELS_SIGNED)\n \n test_static_linked.skel.h-deps := test_static_linked1.bpf.o test_static_linked2.bpf.o\ndiff --git a/tools/testing/selftests/bpf/prog_tests/global_data_init.c b/tools/testing/selftests/bpf/prog_tests/global_data_init.c\nindex 8466332d7406f..7d6bda9092954 100644\n--- a/tools/testing/selftests/bpf/prog_tests/global_data_init.c\n+++ b/tools/testing/selftests/bpf/prog_tests/global_data_init.c\n@@ -1,5 +1,8 @@\n // SPDX-License-Identifier: GPL-2.0\n #include \u003ctest_progs.h\u003e\n+#include \"bpf/libbpf_internal.h\"\n+#include \"test_global_percpu_data.skel.h\"\n+#include \"test_global_percpu_data.lskel.h\"\n \n void test_global_data_init(void)\n {\n@@ -60,3 +63,336 @@ void test_global_data_init(void)\n \tfree(newval);\n \tbpf_object__close(obj);\n }\n+\n+static void test_percpu_data_on_cpus(struct bpf_map map, int map_fd, int prog_fd, int runp)\n+{\n+\tstruct test_global_percpu_data__percpu data = NULL;\n+\tint i, err, key = 0, num_online, run = 0;\n+\t__u64 args[2] = {0x1234ULL, 0x5678ULL};\n+\tsize_t data_sz;\n+\tbool online;\n+\tLIBBPF_OPTS(bpf_test_run_opts, topts,\n+\t\t .ctx_in = args,\n+\t\t .ctx_size_in = sizeof(args),\n+\t\t .flags = BPF_F_TEST_RUN_ON_CPU,\n+\t);\n+\n+\terr = parse_cpu_mask_file(\"/sys/devices/system/cpu/online\", \u0026online, \u0026num_online);\n+\tif (!ASSERT_OK(err, \"parse_cpu_mask_file\"))\n+\t\treturn;\n+\n+\tdata_sz = map ? bpf_map__value_size(map) : sizeof(data);\n+\tdata = calloc(1, data_sz);\n+\tif (!ASSERT_OK_PTR(data, \"calloc percpu data\"))\n+\t\tgoto out;\n+\n+\t/ run on every online-CPU /\n+\tfor (i = 0; i \u003c num_online; i++) {\n+\t\t__u64 flags;\n+\n+\t\tif (!online[i])\n+\t\t\tcontinue;\n+\n+\t\ttopts.cpu = i;\n+\t\ttopts.retval = -1;\n+\t\terr = bpf_prog_test_run_opts(prog_fd, \u0026topts);\n+\t\tASSERT_OK(err, \"bpf_prog_test_run_opts\");\n+\t\tASSERT_EQ(topts.retval, 0, \"bpf_prog_test_run_opts retval\");\n+\n+\t\tmemset(data, 0, data_sz);\n+\t\tflags = ((__u64) i \u003c\u003c 32) \| BPF_F_CPU;\n+\t\tif (map)\n+\t\t\terr = bpf_map__lookup_elem(map, \u0026key, sizeof(key), data, data_sz, flags);\n+\t\telse\n+\t\t\terr = bpf_map_lookup_elem_flags(map_fd, \u0026key, data, flags);\n+\t\tif (!ASSERT_OK(err, \"lookup_elem on cpu\"))\n+\t\t\tbreak;\n+\n+\t\tASSERT_EQ(runp, ++run, \"run\");\n+\t\tASSERT_EQ(data-\u003ecpu_id[0], i, \"cpu_id\");\n+\t\tASSERT_EQ(data-\u003edata, 1, \"data\");\n+\t\tASSERT_TRUE(data-\u003eset, \"set\");\n+\t\tASSERT_EQ(data-\u003enums[6], 0xc0de, \"nums[6]\");\n+\t\tASSERT_EQ(data-\u003estruct_data.i, 1, \"struct_data.i\");\n+\t\tASSERT_TRUE(data-\u003estruct_data.set, \"struct_data.set\");\n+\t\tASSERT_EQ(data-\u003estruct_data.nums[6], 0xc0de, \"struct_data.nums[6]\");\n+\t}\n+\n+out:\n+\tfree(data);\n+\tfree(online);\n+}\n+\n+static void test_global_percpu_data_init(void)\n+{\n+\tstruct test_global_percpu_data__percpu init_value = {};\n+\tstruct test_global_percpu_data__percpu init_data;\n+\tconst __u32 desired_sz = sysconf(_SC_PAGE_SIZE);\n+\tstruct test_global_percpu_data skel = NULL;\n+\tsize_t init_data_sz;\n+\tstruct bpf_map map;\n+\tint prog_fd, err;\n+\n+\tskel = test_global_percpu_data__open();\n+\tif (!ASSERT_OK_PTR(skel, \"test_global_percpu_data__open\"))\n+\t\tgoto out;\n+\tif (!ASSERT_OK_PTR(skel-\u003epercpu, \"skel-\u003epercpu\"))\n+\t\tgoto out;\n+\tif (!ASSERT_OK_PTR(skel-\u003edata_percpu, \"skel-\u003edata_percpu\"))\n+\t\tgoto out;\n+\tif (!ASSERT_OK_PTR(skel-\u003epercpu_data, \"skel-\u003epercpu_data\"))\n+\t\tgoto out;\n+\tif (!ASSERT_OK_PTR(skel-\u003epercpu_looooooooong, \"skel-\u003epercpu_looooooooong\"))\n+\t\tgoto out;\n+\n+\tASSERT_STREQ(bpf_map__name(skel-\u003emaps.percpu_data), \".percpu.data\",\n+\t\t \".percpu.data map name\");\n+\tASSERT_STREQ(bpf_map__name(skel-\u003emaps.data_percpu), \".data.percpu\",\n+\t\t \".data.percpu map name\");\n+\tASSERT_STREQ(bpf_map__name(skel-\u003emaps.percpu_looooooooong), \".percpu.looooooooong\",\n+\t\t \"long map name\");\n+\tASSERT_STREQ(bpf_map__name(skel-\u003emaps.percpu), \".percpu\", \"map name\");\n+\tASSERT_EQ(skel-\u003epercpu-\u003edata, -1, \"skel-\u003epercpu-\u003edata\");\n+\tASSERT_FALSE(skel-\u003epercpu-\u003eset, \"skel-\u003epercpu-\u003eset\");\n+\tASSERT_EQ(skel-\u003epercpu-\u003enums[6], 0, \"skel-\u003epercpu-\u003enums[6]\");\n+\tASSERT_EQ(skel-\u003epercpu-\u003estruct_data.i, -1, \"struct_data.i\");\n+\tASSERT_FALSE(skel-\u003epercpu-\u003estruct_data.set, \"struct_data.set\");\n+\tASSERT_EQ(skel-\u003epercpu-\u003estruct_data.nums[6], 0, \"struct_data.nums[6]\");\n+\n+\tmap = skel-\u003emaps.percpu;\n+\tif (!ASSERT_EQ(bpf_map__type(map), BPF_MAP_TYPE_PERCPU_ARRAY, \"bpf_map__type\"))\n+\t\tgoto out;\n+\n+\tinit_value.data = 2;\n+\tinit_value.nums[6] = -1;\n+\tinit_value.struct_data.i = 2;\n+\tinit_value.struct_data.nums[6] = -1;\n+\terr = bpf_map__set_initial_value(map, \u0026init_value, sizeof(init_value));\n+\tif (!ASSERT_OK(err, \"bpf_map__set_initial_value\"))\n+\t\tgoto out;\n+\n+\tinit_data = bpf_map__initial_value(map, \u0026init_data_sz);\n+\tif (!ASSERT_OK_PTR(init_data, \"bpf_map__initial_value\"))\n+\t\tgoto out;\n+\n+\tASSERT_EQ(init_data-\u003edata, init_value.data, \"init_value data\");\n+\tASSERT_EQ(init_data-\u003eset, init_value.set, \"init_value set\");\n+\tASSERT_EQ(init_data-\u003estruct_data.i, init_value.struct_data.i, \"init_value struct_data.i\");\n+\tASSERT_EQ(init_data-\u003estruct_data.nums[6], init_value.struct_data.nums[6],\n+\t\t \"init_value struct_data.nums[6]\");\n+\tASSERT_EQ(init_data_sz, sizeof(init_value), \"init_value size\");\n+\tASSERT_EQ((void ) init_data, (void ) skel-\u003epercpu, \"skel-\u003epercpu eq init_data\");\n+\tASSERT_EQ(skel-\u003epercpu-\u003edata, init_value.data, \"skel-\u003epercpu-\u003edata\");\n+\tASSERT_EQ(skel-\u003epercpu-\u003eset, init_value.set, \"skel-\u003epercpu-\u003eset\");\n+\tASSERT_EQ(skel-\u003epercpu-\u003estruct_data.i, init_value.struct_data.i,\n+\t\t \"skel-\u003epercpu-\u003estruct_data.i\");\n+\tASSERT_EQ(skel-\u003epercpu-\u003estruct_data.nums[6], init_value.struct_data.nums[6],\n+\t\t \"skel-\u003epercpu-\u003estruct_data.nums[6]\");\n+\n+\tASSERT_GT(desired_sz, sizeof(init_value), \"desired_sz\");\n+\terr = bpf_map__set_value_size(map, desired_sz);\n+\tif (!ASSERT_OK(err, \"bpf_map__set_value_size\"))\n+\t\tgoto out;\n+\tif (!ASSERT_EQ(bpf_map__value_size(map), desired_sz, \"percpu value size\"))\n+\t\tgoto out;\n+\tif (!ASSERT_NEQ(bpf_map__btf_value_type_id(map), 0, \"percpu BTF value type\"))\n+\t\tgoto out;\n+\n+\tinit_data = bpf_map__initial_value(map, \u0026init_data_sz);\n+\tif (!ASSERT_OK_PTR(init_data, \"resized bpf_map__initial_value\"))\n+\t\tgoto out;\n+\tif (!ASSERT_EQ(init_data_sz, desired_sz, \"resized initial value size\"))\n+\t\tgoto out;\n+\tif (!ASSERT_EQ(init_data-\u003edata, init_value.data, \"resized initial value data\"))\n+\t\tgoto out;\n+\n+\terr = test_global_percpu_data__load(skel);\n+\tif (!ASSERT_OK(err, \"test_global_percpu_data__load\"))\n+\t\tgoto out;\n+\n+\tASSERT_OK_PTR(skel-\u003epercpu, \"skel-\u003epercpu\");\n+\n+\tprog_fd = bpf_program__fd(skel-\u003eprogs.update_percpu_data);\n+\ttest_percpu_data_on_cpus(map, bpf_map__fd(map), prog_fd, \u0026skel-\u003ebss-\u003erun);\n+\n+out:\n+\ttest_global_percpu_data__destroy(skel);\n+}\n+\n+static void test_global_percpu_data_lskel(void)\n+{\n+\tstruct test_global_percpu_data_lskel lskel = NULL;\n+\tint prog_fd, map_fd;\n+\n+\tlskel = test_global_percpu_data_lskel__open_and_load();\n+\tif (!ASSERT_OK_PTR(lskel, \"test_global_percpu_data_lskel__open_and_load\"))\n+\t\tgoto out;\n+\n+\tmap_fd = lskel-\u003emaps.percpu.map_fd;\n+\tprog_fd = lskel-\u003eprogs.update_percpu_data.prog_fd;\n+\ttest_percpu_data_on_cpus(NULL, map_fd, prog_fd, \u0026lskel-\u003ebss-\u003erun);\n+\n+out:\n+\ttest_global_percpu_data_lskel__destroy(lskel);\n+}\n+\n+static int create_rdonly_percpu_array(void)\n+{\n+\tLIBBPF_OPTS(bpf_map_create_opts, map_opts,\n+\t\t .map_flags = BPF_F_RDONLY_PROG,\n+\t);\n+\tint key = 0, map_fd, err;\n+\t__u64 value = 0;\n+\n+\tmap_fd = bpf_map_create(BPF_MAP_TYPE_PERCPU_ARRAY, \"percpu_ro_map\", sizeof(int),\n+\t\t\t\tsizeof(__u64), 1, \u0026map_opts);\n+\tif (!ASSERT_GE(map_fd, 0, \"bpf_map_create\"))\n+\t\treturn -1;\n+\n+\terr = bpf_map_update_elem(map_fd, \u0026key, \u0026value, BPF_F_ALL_CPUS);\n+\tif (!ASSERT_OK(err, \"bpf_map_update_elem\"))\n+\t\tgoto out;\n+\n+\terr = bpf_map_freeze(map_fd);\n+\tif (!ASSERT_OK(err, \"bpf_map_freeze\"))\n+\t\tgoto out;\n+\n+\treturn map_fd;\n+\n+out:\n+\tclose(map_fd);\n+\treturn -1;\n+}\n+\n+static void test_global_percpu_data_rdonly_direct_read(void)\n+{\n+\t/\n+\t Raw instructions with manually prepared rdonly percpu_array map\n+\t * for testing direct-read global percpu data, because libbpf\n+\t * doesn't have rdonly internal percpu_array map support for\n+\t * global percpu data.\n+\t /\n+\tstruct bpf_insn insns[] = {\n+\t\tBPF_LD_MAP_VALUE(BPF_REG_1, 0, 0),\n+\t\tBPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, 0),\n+\t\tBPF_EXIT_INSN(),\n+\t};\n+\tint map_fd, prog_fd;\n+\n+\tmap_fd = create_rdonly_percpu_array();\n+\tif (map_fd \u003c 0)\n+\t\treturn;\n+\n+\tinsns[0].imm = map_fd;\n+\tprog_fd = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, \"percpu_ro_prog\", \"GPL\", insns,\n+\t\t\t\tARRAY_SIZE(insns), NULL);\n+\tif (ASSERT_GE(prog_fd, 0, \"bpf_prog_load\"))\n+\t\tclose(prog_fd);\n+\tclose(map_fd);\n+}\n+\n+static void test_global_percpu_data_rdonly_direct_write(void)\n+{\n+\tLIBBPF_OPTS(bpf_prog_load_opts, prog_opts);\n+\t/ See the comment in test_global_percpu_data_rdonly_direct_read() /\n+\tstruct bpf_insn insns[] = {\n+\t\tBPF_LD_MAP_VALUE(BPF_REG_1, 0, 0),\n+\t\tBPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, 0),\n+\t\tBPF_ST_MEM(BPF_DW, BPF_REG_1, 0, 0),\n+\t\tBPF_EXIT_INSN(),\n+\t};\n+\tchar log_buf[256] = {};\n+\tint map_fd, prog_fd;\n+\n+\tprog_opts.log_buf = log_buf;\n+\tprog_opts.log_size = sizeof(log_buf);\n+\tprog_opts.log_level = 1;\n+\n+\tmap_fd = create_rdonly_percpu_array();\n+\tif (map_fd \u003c 0)\n+\t\treturn;\n+\n+\tinsns[0].imm = map_fd;\n+\tprog_fd = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, \"percpu_ro_prog\", \"GPL\", insns,\n+\t\t\t\tARRAY_SIZE(insns), \u0026prog_opts);\n+\tif (!ASSERT_LT(prog_fd, 0, \"bpf_prog_load\"))\n+\t\tclose(prog_fd);\n+\telse\n+\t\tASSERT_HAS_SUBSTR(log_buf, \"write into map forbidden\", \"verifier log\");\n+\tclose(map_fd);\n+}\n+\n+static void test_global_percpu_data_verifier_log(void)\n+{\n+\tRUN_TESTS(test_global_percpu_data);\n+}\n+\n+static void test_global_percpu_data_iter(void)\n+{\n+\tDECLARE_LIBBPF_OPTS(bpf_iter_attach_opts, opts);\n+\tstruct test_global_percpu_data skel;\n+\tunion bpf_iter_link_info linfo = {};\n+\tstruct bpf_link link = NULL;\n+\tint fd, num_cpus, len, err;\n+\tchar buf[16];\n+\n+\tnum_cpus = libbpf_num_possible_cpus();\n+\tif (!ASSERT_GT(num_cpus, 0, \"libbpf_num_possible_cpus\"))\n+\t\treturn;\n+\n+\tskel = test_global_percpu_data__open();\n+\tif (!ASSERT_OK_PTR(skel, \"test_global_percpu_data__open\"))\n+\t\treturn;\n+\n+\tskel-\u003erodata-\u003enum_cpus = num_cpus;\n+\tskel-\u003erodata-\u003eoffsetof_num = offsetof(struct test_global_percpu_data__percpu, struct_data);\n+\tskel-\u003erodata-\u003eoffsetof_num += sizeof(skel-\u003epercpu-\u003estruct_data) - sizeof(int);\n+\tskel-\u003erodata-\u003eelem_sz = roundup(sizeof(struct test_global_percpu_data__percpu), 8);\n+\tskel-\u003epercpu-\u003estruct_data.nums[6] = 0xc0de;\n+\n+\terr = test_global_percpu_data__load(skel);\n+\tif (!ASSERT_OK(err, \"test_global_percpu_data__load\"))\n+\t\tgoto out;\n+\n+\tlinfo.map.map_fd = bpf_map__fd(skel-\u003emaps.percpu);\n+\topts.link_info = \u0026linfo;\n+\topts.link_info_len = sizeof(linfo);\n+\tlink = bpf_program__attach_iter(skel-\u003eprogs.dump_percpu_data, \u0026opts);\n+\tif (!ASSERT_OK_PTR(link, \"bpf_program__attach_iter\"))\n+\t\tgoto out;\n+\n+\tfd = bpf_iter_create(bpf_link__fd(link));\n+\tif (!ASSERT_GE(fd, 0, \"bpf_iter_create\"))\n+\t\tgoto out;\n+\n+\twhile ((len = read(fd, buf, sizeof(buf))) \u003e 0)\n+\t\tdo { } while (0);\n+\tASSERT_EQ(len, 0, \"read iter\");\n+\tASSERT_TRUE(skel-\u003ebss-\u003erun_iter, \"run_iter\");\n+\tASSERT_EQ(skel-\u003ebss-\u003epercpu_data_sum, 0xc0de num_cpus, \"percpu_data_sum\");\n+\n+\tclose(fd);\n+out:\n+\tbpf_link__destroy(link);\n+\ttest_global_percpu_data__destroy(skel);\n+}\n+\n+void test_global_percpu_data(void)\n+{\n+\tif (!feat_supported(NULL, FEAT_PERCPU_DATA)) {\n+\t\ttest__skip();\n+\t\treturn;\n+\t}\n+\n+\tif (test__start_subtest(\"init\"))\n+\t\ttest_global_percpu_data_init();\n+\tif (test__start_subtest(\"lskel\"))\n+\t\ttest_global_percpu_data_lskel();\n+\tif (test__start_subtest(\"rdonly_direct_read\"))\n+\t\ttest_global_percpu_data_rdonly_direct_read();\n+\tif (test__start_subtest(\"rdonly_direct_write\"))\n+\t\ttest_global_percpu_data_rdonly_direct_write();\n+\ttest_global_percpu_data_verifier_log();\n+\tif (test__start_subtest(\"iter\"))\n+\t\ttest_global_percpu_data_iter();\n+}\ndiff --git a/tools/testing/selftests/bpf/progs/test_global_percpu_data.c b/tools/testing/selftests/bpf/progs/test_global_percpu_data.c\nnew file mode 100644\nindex 0000000000000..71ff8d1bf49eb\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/progs/test_global_percpu_data.c\n@@ -0,0 +1,89 @@\n+// SPDX-License-Identifier: GPL-2.0\n+#include \u003cvmlinux.h\u003e\n+#include \u003cbpf/bpf_helpers.h\u003e\n+#include \"bpf_misc.h\"\n+\n+/* Used for testing map name. /\n+int loong SEC(\".percpu.looooooooong\");\n+int data3 SEC(\".data.percpu\");\n+int data2 SEC(\".percpu.data\");\n+\n+int run;\n+/ cpu_id as array to verify map value resizing. /\n+int cpu_id[1] SEC(\".percpu\");\n+int data SEC(\".percpu\") = -1;\n+int nums[7] SEC(\".percpu\");\n+bool set SEC(\".percpu\") = false;\n+struct {\n+\tchar set;\n+\tint i;\n+\tint nums[7];\n+} struct_data SEC(\".percpu\") = {\n+\t.set = 0,\n+\t.i = -1,\n+};\n+\n+SEC(\"raw_tp/task_rename\")\n+__auxiliary\n+int update_percpu_data(void ctx)\n+{\n+\tstruct_data.nums[6] = 0xc0de;\n+\tstruct_data.set = 1;\n+\tstruct_data.i = 1;\n+\tnums[6] = 0xc0de;\n+\tdata = 1;\n+\trun++;\n+\tset = true;\n+\tcpu_id[0] = bpf_get_smp_processor_id();\n+\treturn 0;\n+}\n+\n+static const char fmt[] SEC(\".percpu.fmt\") = \"data %d\\n\";\n+\n+SEC(\"?kprobe\")\n+__failure __msg(\"R{{[0-9]+}} points to percpu_array map which cannot be used as const string\")\n+int verifier_strncmp(void ctx)\n+{\n+\treturn bpf_strncmp(\"test\", 5, fmt);\n+}\n+\n+SEC(\"?kprobe\")\n+__failure __msg(\"R{{[0-9]+}} points to percpu_array map which cannot be used as const string\")\n+int verifier_snprintf(void ctx)\n+{\n+\tu64 args[] = { data };\n+\tchar buf[128];\n+\tint len;\n+\n+\tlen = bpf_snprintf(buf, sizeof(buf), fmt, args, sizeof(args));\n+\tif (len \u003e 0)\n+\t\tbpf_printk(\"snprintf: %s\\n\", buf);\n+\treturn 0;\n+}\n+\n+volatile const __u32 num_cpus = 0;\n+volatile const int offsetof_num;\n+volatile const int elem_sz;\n+__u32 percpu_data_sum = 0;\n+bool run_iter = false;\n+\n+SEC(\"iter/bpf_map_elem\")\n+__auxiliary\n+int dump_percpu_data(struct bpf_iter__bpf_map_elem ctx)\n+{\n+\tvoid pptr = ctx-\u003evalue;\n+\tint i;\n+\n+\tif (!pptr)\n+\t\treturn 0;\n+\n+\trun_iter = true;\n+\n+\tfor (i = 0; i \u003c num_cpus; i++) {\n+\t\tpercpu_data_sum += (int ) (pptr + offsetof_num);\n+\t\tpptr += elem_sz;\n+\t}\n+\treturn 0;\n+}\n+\n+char _license[] SEC(\"license\") = \"GPL\";\n" }
2/1	2026/08/13 16:51	agent	patch-evaluator	1m Model: best-expensive Args: null Results: { "EnableConfigs": [ "BPF_SYSCALL" ], "FocusSymbols": [ "bpf_do_misc_fixups", "bpf_map_direct_read", "percpu_array_map_direct_value_addr", "percpu_array_map_direct_value_meta" ], "Reasoning": "The patch introduces support for global percpu data in BPF programs by allowing `.percpu` sections to be backed by `BPF_MAP_TYPE_PERCPU_ARRAY`. It modifies the BPF verifier to handle direct memory reads from percpu arrays and updates JIT fixups to support direct loading of percpu variables. These changes affect core BPF verifier logic and map operations, which are reachable from user space via the `bpf()` syscall when loading and executing BPF programs.", "WorthFuzzing": true } Instruction: You are an expert Linux kernel maintainer. Your job is to review a provided patch series and determine if it makes functional changes to the kernel that should be fuzzed. IMPORTANT: The changes have ALREADY been applied and committed as the HEAD commit in your workspace. Do NOT rely on your internal knowledge of the kernel. You must actively use your code access tools to examine the actual source code and confirm any assumptions. Return WorthFuzzing=false if the patch only contains: - Modifications to Documentation/, Kconfig files, or code comments. - Purely decorative changes, such as logging (e.g., pr_err, printk) or tracepoints. - Changes to numeric constants or macros that do not functionally alter execution flow. - Code paths that are impossible to reach in virtualized environments like GCE or QEMU, even when utilizing software-emulated hardware (e.g., usb gadget, mac80211_hwsim). - Code in vendor-specific PCIe switch, SmartNIC, or GPU drivers (e.g., mlxsw, pds_core, qed, ionic, amdgpu) that require physical PCIe hardware cards not emulated in standard QEMU. - Driver .remove, .shutdown, or pci_unregister_driver teardown callbacks (e.g., igb_remove) that are executed only during PCI hot-unplug or sysfs driver unbind operations. If it modifies reachable core kernel logic, drivers, or architectures, use your code search tools to verify the code can be executed, then return WorthFuzzing=true. When returning WorthFuzzing=true, you MUST ALSO: 1. Extract any specific kernel functions that should be heavily fuzzed into FocusSymbols. Avoid listing generic hot-path functions to prevent skewed test distributions. Prefer non-static, non-inlined API entrypoint functions over internal static helper functions (which are inlined by the compiler and do not have distinct symbol addresses). 2. Identify any specific CONFIG_ options required to properly test this new/modified feature. Go and look into the Kconfig files and check for ifdefs around the code, do not make assumptions. Also check "depends on" lines in Kconfig to include any non-standard parent subsystem configs needed for Kbuild to compile the code statically into vmlinux. List them in the EnableConfigs output array, and DO NOT add a 'CONFIG_' prefix (e.g., return "NET_IPV4" instead of "CONFIG_NET_IPV4"). Prefer calling several tools at the same time to save round-trips. Use set-results tool to provide results of the analysis. It must be called exactly once before the final reply. Ignore results of this tool. Prompt: For your convenience, here is the diff of the changes: commit 3b76f69b75942f2595d46bc813574f1caa6e529f Author: syz-cluster <triage@syzkaller.com> Date: Thu Aug 13 16:51:17 2026 +0000 syz-cluster: applied patch under review diff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c index 248b4818178cd..34865701f7f75 100644 --- a/kernel/bpf/arraymap.c +++ b/kernel/bpf/arraymap.c @@ -259,6 +259,37 @@ static void percpu_array_map_lookup_elem(struct bpf_map map, void key) return this_cpu_ptr(array->pptrs[index & array->index_mask]); } +static int percpu_array_map_direct_value_addr(const struct bpf_map map, u64 imm, u32 off) +{ + struct bpf_array array = container_of(map, struct bpf_array, map); + + if (!bpf_jit_supports_percpu_insn()) + return -EOPNOTSUPP; + if (map->max_entries != 1) + return -EOPNOTSUPP; + if (off >= map->value_size) + return -EINVAL; + + imm = (u64)(__force unsigned long) array->pptrs[0]; + return 0; +} + +static int percpu_array_map_direct_value_meta(const struct bpf_map map, u64 imm, u32 off) +{ + struct bpf_array array = container_of(map, struct bpf_array, map); + u64 base = (u64)(__force unsigned long) array->pptrs[0]; + + if (!bpf_jit_supports_percpu_insn()) + return -EOPNOTSUPP; + if (map->max_entries != 1) + return -EOPNOTSUPP; + if (imm < base \|\| imm >= base + array->elem_size) + return -ENOENT; + + off = imm - base; + return 0; +} + / emit BPF instructions equivalent to C code of percpu_array_map_lookup_elem() / static int percpu_array_map_gen_lookup(struct bpf_map map, struct bpf_insn insn_buf) { @@ -551,9 +582,10 @@ static int array_map_check_btf(struct bpf_map map, const struct btf_type key_type, const struct btf_type value_type) { - /* One exception for keyless BTF: .bss/.data/.rodata map / + / One exception for keyless BTF: .bss/.data/.rodata/.percpu map / if (btf_type_is_void(key_type)) { - if (map->map_type != BPF_MAP_TYPE_ARRAY \|\| + if ((map->map_type != BPF_MAP_TYPE_ARRAY && + map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY) \|\| map->max_entries != 1) return -EINVAL; @@ -832,6 +864,8 @@ const struct bpf_map_ops percpu_array_map_ops = { .map_get_next_key = bpf_array_get_next_key, .map_lookup_elem = percpu_array_map_lookup_elem, .map_gen_lookup = percpu_array_map_gen_lookup, + .map_direct_value_addr = percpu_array_map_direct_value_addr, + .map_direct_value_meta = percpu_array_map_direct_value_meta, .map_update_elem = array_map_update_elem, .map_delete_elem = array_map_delete_elem, .map_lookup_percpu_elem = percpu_array_map_lookup_percpu_elem, diff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c index 40bd04421a991..a2b18a9f1694c 100644 --- a/kernel/bpf/backtrack.c +++ b/kernel/bpf/backtrack.c @@ -214,7 +214,6 @@ static inline bool bt_is_reg_set(struct backtrack_state bt, u32 reg) return bt->reg_masks[bt->frame] & (1 << reg); } - /* format registers bitmask, e.g., "r0,r2,r4" for 0x15 mask / static void fmt_reg_mask(char buf, ssize_t buf_sz, u32 reg_mask) { @@ -254,7 +253,6 @@ void bpf_fmt_stack_mask(char buf, ssize_t buf_sz, u64 stack_mask) } } - / For given verifier state backtrack_insn() is called from the last insn to * the first insn. Its purpose is to compute a bitmask of registers and * stack slots that needs precision in the parent verifier state. diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index 6606187ed4f43..87ffde865a503 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -2534,7 +2534,6 @@ static void btf_bitfield_show(void data, u8 bits_offset, btf_int128_print(show, print_num); } - static void btf_int_bits_show(const struct btf btf, const struct btf_type t, void data, u8 bits_offset, diff --git a/kernel/bpf/cfg.c b/kernel/bpf/cfg.c index db3416a7c9047..818f7afac83a5 100644 --- a/kernel/bpf/cfg.c +++ b/kernel/bpf/cfg.c @@ -47,7 +47,6 @@ enum { BRANCH = 2, }; - static void mark_subprog_changes_pkt_data(struct bpf_verifier_env env, int off) { struct bpf_subprog_info subprog; diff --git a/kernel/bpf/const_fold.c b/kernel/bpf/const_fold.c index 4cf120c7b2cb4..7f1b30059cc87 100644 --- a/kernel/bpf/const_fold.c +++ b/kernel/bpf/const_fold.c @@ -182,7 +182,6 @@ static void const_reg_xfer(struct bpf_verifier_env env, struct const_arg_info u64 val = 0; if (!bpf_map_is_rdonly(map) \|\| !map->ops->map_direct_value_addr \|\| - map->map_type == BPF_MAP_TYPE_INSN_ARRAY \|\| off < 0 \|\| off + size > map->value_size \|\| bpf_map_direct_read(map, off, size, &val, is_ldsx)) { dst = unknown; diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c index 2417a3461652d..177a3fcbb63ac 100644 --- a/kernel/bpf/fixups.c +++ b/kernel/bpf/fixups.c @@ -1466,7 +1466,6 @@ int bpf_fixup_call_args(struct bpf_verifier_env env) return err; } - /* The function requires that first instruction in 'patch' is insnsi[prog->len - 1] / static int add_hidden_subprog(struct bpf_verifier_env env, struct bpf_insn patch, int len) { @@ -1835,6 +1834,43 @@ int bpf_do_misc_fixups(struct bpf_verifier_env env) goto next_insn; } + if (bpf_jit_supports_percpu_insn() && + insn->code == (BPF_LD \| BPF_IMM \| BPF_DW) && + (insn->src_reg == BPF_PSEUDO_MAP_VALUE \|\| + insn->src_reg == BPF_PSEUDO_MAP_IDX_VALUE)) { + struct bpf_map map; + + aux = &env->insn_aux_data[i + delta]; + map = env->used_maps[aux->map_index]; + if (map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY) + goto next_insn; + + prog->jit_required = true; + + / + * We are skipping first half of ld_imm64 insn + * with 'i++;', patching over second half of it + * with that same half + mov64_percpu_reg insn. + * All because bpf_patch_insn_data() can only + * replace one 8-byte insn, which does not work + * well for ld_imm64 insn. + / + + insn_buf[0] = insn[1]; + insn_buf[1] = BPF_MOV64_PERCPU_REG(insn->dst_reg, insn->dst_reg); + cnt = 2; + + i++; + new_prog = bpf_patch_insn_data(env, i + delta, insn_buf, cnt); + if (!new_prog) + return -ENOMEM; + + delta += cnt - 1; + env->prog = prog = new_prog; + insn = new_prog->insnsi + i + delta; + goto next_insn; + } + if (insn->code != (BPF_JMP \| BPF_CALL)) goto next_insn; if (insn->src_reg == BPF_PSEUDO_CALL) diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c index 9f394e1aa2e85..d40cb5dd446ca 100644 --- a/kernel/bpf/hashtab.c +++ b/kernel/bpf/hashtab.c @@ -998,7 +998,6 @@ static void dec_elem_count(struct bpf_htab htab) atomic_dec(&htab->count); } - static void free_htab_elem(struct bpf_htab htab, struct htab_elem l) { htab_put_fd_value(htab, l); @@ -2970,7 +2969,6 @@ static int rhtab_delete_elem(struct bpf_rhtab rhtab, struct rhtab_elem elem, v return 0; } - static long rhtab_map_delete_elem(struct bpf_map map, void key) { struct bpf_rhtab rhtab = container_of(map, struct bpf_rhtab, map); diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index 6388b6b23e490..45e2f19387b24 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -4871,7 +4871,6 @@ static const struct btf_kfunc_id_set generic_kfunc_set = { .set = &generic_btf_ids, }; - BTF_ID_LIST(generic_dtor_ids) BTF_ID(struct, task_struct) BTF_ID(func, bpf_task_release_dtor) diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c index 1c997aeba6fa5..74fc4b3f80d6e 100644 --- a/kernel/bpf/liveness.c +++ b/kernel/bpf/liveness.c @@ -269,7 +269,6 @@ bpf_insn_successors(struct bpf_verifier_env env, u32 idx) __diag_pop(); - static inline bool update_insn(struct bpf_verifier_env env, struct func_instance instance, u32 frame, u32 insn_idx) { @@ -1862,7 +1861,6 @@ static int analyze_subprog(struct bpf_verifier_env env, if (need_resched()) cond_resched(); - / * When an instance is reused (must_write_initialized == true), * record into a fresh instance and merge afterward. This avoids diff --git a/kernel/bpf/queue_stack_maps.c b/kernel/bpf/queue_stack_maps.c index c1c9dee4dcdd0..6e8b18c32a106 100644 --- a/kernel/bpf/queue_stack_maps.c +++ b/kernel/bpf/queue_stack_maps.c @@ -123,7 +123,6 @@ static long __queue_map_get(struct bpf_map map, void value, bool delete) return err; } - static long __stack_map_get(struct bpf_map map, void value, bool delete) { struct bpf_queue_stack qs = bpf_queue_stack(map); diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 8d111da886553..7d8c3e8e6d62e 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -636,7 +636,6 @@ int bpf_map_alloc_pages(const struct bpf_map map, int nid, return ret; } - static int btf_field_cmp(const void a, const void b) { const struct btf_field f1 = a, f2 = b; @@ -1830,7 +1829,6 @@ static int map_lookup_elem(union bpf_attr attr) return err; } - #define BPF_MAP_UPDATE_ELEM_LAST_FIELD flags static int map_update_elem(union bpf_attr attr, bpfptr_t uattr) @@ -3497,7 +3495,6 @@ int bpf_link_prime(struct bpf_link link, struct bpf_link_primer primer) if (fd < 0) return fd; - id = bpf_link_alloc_id(link); if (id < 0) { put_unused_fd(fd); @@ -5505,7 +5502,6 @@ static int bpf_link_get_info_by_fd(struct file file, return 0; } - static int token_get_info_by_fd(struct file file, struct bpf_token token, const union bpf_attr attr, @@ -6507,7 +6503,6 @@ BPF_CALL_3(bpf_sys_bpf, int, cmd, union bpf_attr , attr, u32, attr_size) return __sys_bpf(cmd, KERNEL_BPFPTR(attr), attr_size, KERNEL_BPFPTR(NULL), 0); } - / To shut up -Wmissing-prototypes. * This function is used by the kernel light skeleton * to load bpf programs when modules are loaded or during kernel boot. diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 164d16c243ca6..6ac1afced20bf 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -635,7 +635,6 @@ static void __mark_dynptr_reg(struct bpf_reg_state reg, enum bpf_dynptr_type type, bool first_slot, int id, int parent_id); - static void mark_dynptr_stack_regs(struct bpf_verifier_env env, struct bpf_reg_state sreg1, struct bpf_reg_state sreg2, @@ -1674,7 +1673,6 @@ static bool same_callsites(struct bpf_verifier_state a, struct bpf_verifier_sta return true; } - void bpf_free_backedges(struct bpf_scc_visit visit) { struct bpf_scc_backedge backedge, next; @@ -2291,7 +2289,6 @@ static struct bpf_verifier_state push_async_cb(struct bpf_verifier_env env, return &elem->st; } - static int cmp_subprogs(const void a, const void b) { return ((struct bpf_subprog_info )a)->start - @@ -3969,7 +3966,6 @@ static int check_stack_read(struct bpf_verifier_env env, return err; } - /* check_stack_write dispatches to check_stack_write_fixed_off or * check_stack_write_var_off. * @@ -4767,7 +4763,6 @@ static int check_sock_access(struct bpf_verifier_env env, int insn_idx, valid = false; } - if (valid) { env->insn_aux_data[insn_idx].ctx_field_size = info.ctx_field_size; @@ -5587,6 +5582,8 @@ int bpf_map_direct_read(struct bpf_map map, int off, int size, u64 val, u64 addr; int err; + if (map->map_type == BPF_MAP_TYPE_INSN_ARRAY \|\| map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY) + return -EINVAL; err = map->ops->map_direct_value_addr(map, &addr, off); if (err) return err; @@ -6083,6 +6080,51 @@ static void add_scalar_to_reg(struct bpf_reg_state dst_reg, s64 val) reg_bounds_sync(dst_reg); } +static int check_map_mem_read(struct bpf_verifier_env env, struct bpf_reg_state reg, int off, + int bpf_size, int value_regno, bool is_ldsx) +{ + struct bpf_reg_state regs = cur_regs(env); + int size = bpf_size_to_bytes(bpf_size); + struct bpf_map map = reg->map_ptr; + + switch (map->map_type) { + case BPF_MAP_TYPE_INSN_ARRAY: + if (bpf_size != BPF_DW) { + verbose(env, "Invalid read of %d bytes from insn_array\n", size); + return -EACCES; + } + regs[value_regno] = reg; + add_scalar_to_reg(&regs[value_regno], off); + regs[value_regno].type = PTR_TO_INSN; + return 0; + case BPF_MAP_TYPE_PERCPU_ARRAY: + goto reg_unknown; + default: + break; + } + + / If map is read-only, track its contents as scalars. / + if (tnum_is_const(reg->var_off) && + bpf_map_is_rdonly(map) && + map->ops->map_direct_value_addr) { + int map_off = off + reg->var_off.value; + u64 val = 0; + int err; + + err = bpf_map_direct_read(map, map_off, size, &val, is_ldsx); + if (err) + return err; + + regs[value_regno].type = SCALAR_VALUE; + __mark_reg_known(&regs[value_regno], val); + return 0; + } + +reg_unknown: + mark_reg_unknown(env, regs, value_regno); + return 0; +} + / check whether memory at (regno + off) is accessible for t = (read \| write) * if t==write, value_regno is a register which value is stored into memory * if t==read, value_regno is a register which will receive the value from memory @@ -6137,38 +6179,7 @@ static int check_mem_access(struct bpf_verifier_env env, int insn_idx, struct b if (kptr_field) { err = check_map_kptr_access(env, value_regno, insn_idx, kptr_field); } else if (t == BPF_READ && value_regno >= 0) { - struct bpf_map map = reg->map_ptr; - - /* - * If map is read-only, track its contents as scalars, - * unless it is an insn array (see the special case below) - / - if (tnum_is_const(reg->var_off) && - bpf_map_is_rdonly(map) && - map->ops->map_direct_value_addr && - map->map_type != BPF_MAP_TYPE_INSN_ARRAY) { - int map_off = off + reg->var_off.value; - u64 val = 0; - - err = bpf_map_direct_read(map, map_off, size, - &val, is_ldsx); - if (err) - return err; - - regs[value_regno].type = SCALAR_VALUE; - __mark_reg_known(&regs[value_regno], val); - } else if (map->map_type == BPF_MAP_TYPE_INSN_ARRAY) { - if (bpf_size != BPF_DW) { - verbose(env, "Invalid read of %d bytes from insn_array\n", - size); - return -EACCES; - } - regs[value_regno] = reg; - add_scalar_to_reg(&regs[value_regno], off); - regs[value_regno].type = PTR_TO_INSN; - } else { - mark_reg_unknown(env, regs, value_regno); - } + err = check_map_mem_read(env, reg, off, bpf_size, value_regno, is_ldsx); } } else if (base_type(reg->type) == PTR_TO_MEM) { bool rdonly_mem = type_is_rdonly_mem(reg->type); @@ -6635,7 +6646,6 @@ static int check_stack_range_initialized( if (err) return err; - if (tnum_is_const(reg->var_off)) { min_off = max_off = reg->var_off.value + off; } else { @@ -7347,7 +7357,6 @@ static bool is_iter_new_kfunc(struct bpf_call_arg_meta meta) return meta->kfunc_flags & KF_ITER_NEW; } - static bool is_iter_destroy_kfunc(struct bpf_call_arg_meta meta) { return meta->kfunc_flags & KF_ITER_DESTROY; @@ -8125,6 +8134,12 @@ static int check_arg_const_str(struct bpf_verifier_env env, return -EACCES; } + if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY) { + verbose(env, "%s points to percpu_array map which cannot be used as const string\n", + reg_arg_name(env, argno)); + return -EACCES; + } + if (!bpf_map_is_rdonly(map)) { verbose(env, "%s does not point to a readonly map'\n", reg_arg_name(env, argno)); return -EACCES; @@ -11607,7 +11622,6 @@ static int process_irq_flag(struct bpf_verifier_env env, struct bpf_reg_state * return 0; } - static int ref_set_non_owning(struct bpf_verifier_env env, struct bpf_reg_state reg) { struct btf_record rec = reg_btf_record(reg); @@ -16412,7 +16426,6 @@ static int check_ld_abs(struct bpf_verifier_env env, struct bpf_insn insn) return 0; } - static bool return_retval_range(struct bpf_verifier_env env, struct bpf_retval_range range) { enum bpf_prog_type prog_type = resolve_prog_type(env->prog); @@ -18361,8 +18374,6 @@ static void release_insn_arrays(struct bpf_verifier_env env) bpf_insn_array_release(env->insn_array_maps[i]); } - - /* The verifier does more data flow analysis than llvm and will not * explore branches that are dead at run time. Malicious programs can * have dead code too. Therefore replace all dead at-run-time code @@ -18390,8 +18401,6 @@ static void sanitize_dead_code(struct bpf_verifier_env env) } } - - static void free_states(struct bpf_verifier_env env) { struct bpf_verifier_state_list sl; @@ -18678,7 +18687,6 @@ static int do_check_main(struct bpf_verifier_env env) return ret; } - static void print_verification_stats(struct bpf_verifier_env env) { / Skip over hidden subprogs which are not verified. / diff --git a/tools/bpf/bpftool/gen.c b/tools/bpf/bpftool/gen.c index a01d06d22d1a3..a50540ef6521c 100644 --- a/tools/bpf/bpftool/gen.c +++ b/tools/bpf/bpftool/gen.c @@ -101,6 +101,12 @@ static bool get_map_ident(const struct bpf_map map, char buf, size_t buf_sz) return true; } + if (bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY) { + snprintf(buf, buf_sz, "%s", name + 1); + sanitize_identifier(buf); + return true; + } + for (i = 0, n = ARRAY_SIZE(sfxs); i < n; i++) { const char sfx = sfxs[i], p; @@ -117,7 +123,7 @@ static bool get_map_ident(const struct bpf_map map, char buf, size_t buf_sz) static bool get_datasec_ident(const char sec_name, char buf, size_t buf_sz) { - static const char pfxs[] = { ".data", ".rodata", ".bss", ".kconfig" }; + static const char pfxs[] = { ".data", ".rodata", ".bss", ".percpu", ".kconfig" }; int i, n; / recognize hard coded LLVM section name / @@ -254,7 +260,7 @@ static const struct btf_type find_type_for_map(struct btf btf, const char map return NULL; } -static bool is_mmapable_map(const struct bpf_map map, char buf, size_t sz) +static bool is_skel_data(const struct bpf_map map, char buf, size_t sz) { size_t tmp_sz; @@ -263,13 +269,24 @@ static bool is_mmapable_map(const struct bpf_map map, char buf, size_t sz) return true; } - if (!bpf_map__is_internal(map) \|\| !(bpf_map__map_flags(map) & BPF_F_MMAPABLE)) + if (!bpf_map__is_internal(map)) return false; if (!get_map_ident(map, buf, sz)) return false; - return true; + if (bpf_map__map_flags(map) & BPF_F_MMAPABLE) + return true; + + if (bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY) + return bpf_map__btf_value_type_id(map) != 0; + + return false; +} + +static bool is_mmapable_map(const struct bpf_map map, char buf, size_t sz) +{ + return is_skel_data(map, buf, sz) && bpf_map__type(map) != BPF_MAP_TYPE_PERCPU_ARRAY; } static int codegen_datasecs(struct bpf_object obj, const char obj_name) @@ -287,7 +304,7 @@ static int codegen_datasecs(struct bpf_object obj, const char obj_name) bpf_object__for_each_map(map, obj) { /* only generate definitions for memory-mapped internal maps / - if (!is_mmapable_map(map, map_ident, sizeof(map_ident))) + if (!is_skel_data(map, map_ident, sizeof(map_ident))) continue; sec = find_type_for_map(btf, map_ident); @@ -517,7 +534,7 @@ static void codegen_asserts(struct bpf_object obj, const char obj_name) ", obj_name); bpf_object__for_each_map(map, obj) { - if (!is_mmapable_map(map, map_ident, sizeof(map_ident))) + if (!is_skel_data(map, map_ident, sizeof(map_ident))) continue; sec = find_type_for_map(btf, map_ident); @@ -668,8 +685,7 @@ static void codegen_destroy(struct bpf_object obj, const char obj_name) bpf_object__for_each_map(map, obj) { if (!get_map_ident(map, ident, sizeof(ident))) continue; - if (bpf_map__is_internal(map) && - (bpf_map__map_flags(map) & BPF_F_MMAPABLE)) + if (is_skel_data(map, ident, sizeof(ident))) printf("\tskel_free_map_data(skel->%1$s, skel->maps.%1$s.initial_value, %2$zu);\n", ident, bpf_map_mmap_sz(map)); codegen("\ @@ -741,7 +757,7 @@ static int gen_trace(struct bpf_object obj, const char obj_name, const char h const void mmap_data = NULL; size_t mmap_size = 0; - if (!is_mmapable_map(map, ident, sizeof(ident))) + if (!is_skel_data(map, ident, sizeof(ident))) continue; codegen("\ @@ -849,9 +865,23 @@ static int gen_trace(struct bpf_object obj, const char obj_name, const char h bpf_object__for_each_map(map, obj) { const char mmap_flags; - if (!is_mmapable_map(map, ident, sizeof(ident))) + if (!is_skel_data(map, ident, sizeof(ident))) continue; + if (bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY) { + codegen("\ + \n\ + err = skel_protect_map_data(skel->%1$s, &skel->maps.%1$s.initial_value, %2$zd);\n\ + if (err) \n\ + return err; \n\ + #ifdef __KERNEL__ \n\ + skel->%1$s = NULL; \n\ + #endif \n\ + ", + ident, bpf_map_mmap_sz(map)); + continue; + } + if (bpf_map__map_flags(map) & BPF_F_RDONLY_PROG) mmap_flags = "PROT_READ"; else @@ -955,8 +985,7 @@ codegen_maps_skeleton(struct bpf_object obj, size_t map_cnt, bool mmaped, bool map->map = &obj->maps.%s; \n\ ", i, bpf_map__name(map), ident); - /* memory-mapped internal maps / - if (mmaped && is_mmapable_map(map, ident, sizeof(ident))) { + if (mmaped && is_skel_data(map, ident, sizeof(ident))) { printf("\tmap->mmaped = (void )&obj->%s;\n", ident); } diff --git a/tools/lib/bpf/bpf_gen_internal.h b/tools/lib/bpf/bpf_gen_internal.h index 0425691877521..6c5ad6c55e8a6 100644 --- a/tools/lib/bpf/bpf_gen_internal.h +++ b/tools/lib/bpf/bpf_gen_internal.h @@ -65,7 +65,8 @@ void bpf_gen__prog_load(struct bpf_gen gen, enum bpf_prog_type prog_type, const char prog_name, const char license, struct bpf_insn insns, size_t insn_cnt, struct bpf_prog_load_opts load_attr, int prog_idx); -void bpf_gen__map_update_elem(struct bpf_gen gen, int map_idx, void value, __u32 value_size); +void bpf_gen__map_update_elem(struct bpf_gen gen, int map_idx, void value, __u32 value_size, + __u64 flags); void bpf_gen__map_freeze(struct bpf_gen gen, int map_idx); void bpf_gen__record_attach_target(struct bpf_gen gen, const char name, enum bpf_attach_type type); void bpf_gen__record_extern(struct bpf_gen gen, const char name, bool is_weak, diff --git a/tools/lib/bpf/features.c b/tools/lib/bpf/features.c index b7e388f99d0bb..ef9581c113035 100644 --- a/tools/lib/bpf/features.c +++ b/tools/lib/bpf/features.c @@ -620,6 +620,38 @@ static int probe_bpf_syscall_common_attrs(int token_fd) return probe_sys_bpf_ext(); } +static int probe_kern_percpu_data(int token_fd) +{ + struct bpf_insn insns[] = { + BPF_LD_MAP_VALUE(BPF_REG_1, 0, 0), + BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, 0), + BPF_EXIT_INSN(), + }; + LIBBPF_OPTS(bpf_map_create_opts, map_opts, + .token_fd = token_fd, + .map_flags = token_fd ? BPF_F_TOKEN_FD : 0, + ); + LIBBPF_OPTS(bpf_prog_load_opts, prog_opts, + .token_fd = token_fd, + .prog_flags = token_fd ? BPF_F_TOKEN_FD : 0, + ); + int ret, map, insn_cnt = ARRAY_SIZE(insns); + + map = bpf_map_create(BPF_MAP_TYPE_PERCPU_ARRAY, "libbpf_percpu", sizeof(int), 8, 1, + &map_opts); + if (map < 0) { + pr_warn("Error in %s(): %s. Couldn't create simple percpu_array map.\n", + __func__, errstr(map)); + return map; + } + + insns[0].imm = map; + + ret = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, NULL, "GPL", insns, insn_cnt, &prog_opts); + close(map); + return probe_fd(ret); +} + typedef int (feature_probe_fn)(int /* token_fd /); static struct kern_feature_cache feature_cache; @@ -707,6 +739,9 @@ static struct kern_feature_desc { [FEAT_BPF_SYSCALL_COMMON_ATTRS] = { "BPF syscall common attributes support", probe_bpf_syscall_common_attrs, }, + [FEAT_PERCPU_DATA] = { + "kernel supports percpu data", probe_kern_percpu_data, + }, }; bool feat_supported(struct kern_feature_cache cache, enum kern_feature_id feat_id) diff --git a/tools/lib/bpf/gen_loader.c b/tools/lib/bpf/gen_loader.c index 6e3dd52427618..af3a04f161ac1 100644 --- a/tools/lib/bpf/gen_loader.c +++ b/tools/lib/bpf/gen_loader.c @@ -1128,7 +1128,7 @@ void bpf_gen__prog_load(struct bpf_gen gen, } void bpf_gen__map_update_elem(struct bpf_gen gen, int map_idx, void pvalue, - __u32 value_size) + __u32 value_size, __u64 flags) { int attr_size = offsetofend(union bpf_attr, flags); int map_update_attr, value, key; @@ -1136,6 +1136,7 @@ void bpf_gen__map_update_elem(struct bpf_gen gen, int map_idx, void pvalue, int zero = 0; memset(&attr, 0, attr_size); + attr.flags = tgt_endian(flags); value = add_data(gen, pvalue, value_size); key = add_data(gen, &zero, sizeof(zero)); diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index 514e4e9daa825..e574870fb7169 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -541,6 +541,7 @@ struct bpf_struct_ops { }; #define DATA_SEC ".data" +#define PERCPU_SEC ".percpu" #define BSS_SEC ".bss" #define RODATA_SEC ".rodata" #define KCONFIG_SEC ".kconfig" @@ -555,6 +556,7 @@ enum libbpf_map_type { LIBBPF_MAP_BSS, LIBBPF_MAP_RODATA, LIBBPF_MAP_KCONFIG, + LIBBPF_MAP_PERCPU, }; struct bpf_map_def { @@ -666,6 +668,7 @@ enum sec_type { SEC_DATA, SEC_RODATA, SEC_ST_OPS, + SEC_PERCPU, }; struct elf_sec_desc { @@ -1839,6 +1842,8 @@ static size_t bpf_map_mmap_sz(const struct bpf_map map) switch (map->def.type) { case BPF_MAP_TYPE_ARRAY: return array_map_mmap_sz(map->def.value_size, map->def.max_entries); + case BPF_MAP_TYPE_PERCPU_ARRAY: + return map->def.value_size; case BPF_MAP_TYPE_ARENA: return page_sz * map->def.max_entries; default: @@ -1866,7 +1871,8 @@ static int bpf_map_mmap_resize(struct bpf_map map, size_t old_sz, size_t new_sz return 0; } -static char internal_map_name(struct bpf_object obj, const char real_name) +static char internal_map_name(struct bpf_object obj, const char real_name, + enum libbpf_map_type type) { char map_name[BPF_OBJ_NAME_LEN], p; int pfx_len, sfx_len = max((size_t)7, strlen(real_name)); @@ -1907,8 +1913,11 @@ static char internal_map_name(struct bpf_object obj, const char real_name) if (sfx_len >= BPF_OBJ_NAME_LEN) sfx_len = BPF_OBJ_NAME_LEN - 1; - / if there are two or more dots in map name, it's a custom dot map / - if (strchr(real_name + 1, '.') != NULL) + / + * Don't prefix the bpf_object name if this is a custom dot map + * (containing two or more dots) or a percpu data map. + / + if (strchr(real_name + 1, '.') != NULL \|\| type == LIBBPF_MAP_PERCPU) pfx_len = 0; else pfx_len = min((size_t)BPF_OBJ_NAME_LEN - sfx_len - 1, strlen(obj->name)); @@ -1941,6 +1950,13 @@ static bool map_is_mmapable(struct bpf_object obj, struct bpf_map map) if (!map->btf_value_type_id) return false; + / + * The internal PERCPU maps are not mmapble because the underlying + * percpu_array maps do not have mmap support. + / + if (map->libbpf_type == LIBBPF_MAP_PERCPU) + return false; + t = btf__type_by_id(obj->btf, map->btf_value_type_id); if (!btf_is_datasec(t)) return false; @@ -1962,6 +1978,7 @@ static int bpf_object__init_internal_map(struct bpf_object obj, enum libbpf_map_type type, const char real_name, int sec_idx, void data, size_t data_sz) { + bool is_percpu = type == LIBBPF_MAP_PERCPU; struct bpf_map_def def; struct bpf_map map; size_t mmap_sz; @@ -1975,7 +1992,7 @@ bpf_object__init_internal_map(struct bpf_object obj, enum libbpf_map_type type, map->sec_idx = sec_idx; map->sec_offset = 0; map->real_name = strdup(real_name); - map->name = internal_map_name(obj, real_name); + map->name = internal_map_name(obj, real_name, type); if (!map->real_name \|\| !map->name) { zfree(&map->real_name); zfree(&map->name); @@ -1983,7 +2000,7 @@ bpf_object__init_internal_map(struct bpf_object obj, enum libbpf_map_type type, } def = &map->def; - def->type = BPF_MAP_TYPE_ARRAY; + def->type = is_percpu ? BPF_MAP_TYPE_PERCPU_ARRAY : BPF_MAP_TYPE_ARRAY; def->key_size = sizeof(int); def->value_size = data_sz; def->max_entries = 1; @@ -1996,8 +2013,9 @@ bpf_object__init_internal_map(struct bpf_object obj, enum libbpf_map_type type, if (map_is_mmapable(obj, map)) def->map_flags \|= BPF_F_MMAPABLE; - pr_debug("map '%s' (global data): at sec_idx %d, offset %zu, flags %x.\n", - map->name, map->sec_idx, map->sec_offset, def->map_flags); + pr_debug("map '%s' (global %sdata): at sec_idx %d, offset %zu, flags %x.\n", + map->name, is_percpu ? "percpu " : "", map->sec_idx, + map->sec_offset, def->map_flags); mmap_sz = bpf_map_mmap_sz(map); map->mmaped = mmap(NULL, mmap_sz, PROT_READ \| PROT_WRITE, @@ -2057,6 +2075,13 @@ static int bpf_object__init_global_data_maps(struct bpf_object obj) NULL, sec_desc->data->d_size); break; + case SEC_PERCPU: + sec_name = elf_sec_name(obj, elf_sec_by_idx(obj, sec_idx)); + err = bpf_object__init_internal_map(obj, LIBBPF_MAP_PERCPU, + sec_name, sec_idx, + sec_desc->data->d_buf, + sec_desc->data->d_size); + break; default: /* skip / break; @@ -4016,6 +4041,11 @@ static int bpf_object__elf_collect(struct bpf_object obj) sec_desc->sec_type = SEC_RODATA; sec_desc->shdr = sh; sec_desc->data = data; + } else if (strcmp(name, PERCPU_SEC) == 0 \|\| + str_has_pfx(name, PERCPU_SEC ".")) { + sec_desc->sec_type = SEC_PERCPU; + sec_desc->shdr = sh; + sec_desc->data = data; } else if (strcmp(name, STRUCT_OPS_SEC) == 0 \|\| strcmp(name, STRUCT_OPS_LINK_SEC) == 0 \|\| strcmp(name, "?" STRUCT_OPS_SEC) == 0 \|\| @@ -4544,6 +4574,7 @@ static bool bpf_object__shndx_is_data(const struct bpf_object obj, case SEC_BSS: case SEC_DATA: case SEC_RODATA: + case SEC_PERCPU: return true; default: return false; @@ -4569,6 +4600,8 @@ bpf_object__section_to_libbpf_map_type(const struct bpf_object obj, int shndx) return LIBBPF_MAP_DATA; case SEC_RODATA: return LIBBPF_MAP_RODATA; + case SEC_PERCPU: + return LIBBPF_MAP_PERCPU; default: return LIBBPF_MAP_UNSPEC; } @@ -4944,7 +4977,7 @@ static int map_fill_btf_type_info(struct bpf_object obj, struct bpf_map map) /* * LLVM annotates global data differently in BTF, that is, - * only as '.data', '.bss' or '.rodata'. + * only as '.data', '.bss', '.percpu' or '.rodata'. / if (!bpf_map__is_internal(map)) return -ENOENT; @@ -5293,18 +5326,20 @@ static int bpf_object__populate_internal_map(struct bpf_object obj, struct bpf_map map) { enum libbpf_map_type map_type = map->libbpf_type; + bool is_percpu = map_type == LIBBPF_MAP_PERCPU; + const __u64 update_flags = is_percpu ? BPF_F_ALL_CPUS : 0; int err, zero = 0; size_t mmap_sz; if (obj->gen_loader) { bpf_gen__map_update_elem(obj->gen_loader, map - obj->maps, - map->mmaped, map->def.value_size); + map->mmaped, map->def.value_size, update_flags); if (map_type == LIBBPF_MAP_RODATA \|\| map_type == LIBBPF_MAP_KCONFIG) bpf_gen__map_freeze(obj->gen_loader, map - obj->maps); return 0; } - err = bpf_map_update_elem(map->fd, &zero, map->mmaped, 0); + err = bpf_map_update_elem(map->fd, &zero, map->mmaped, update_flags); if (err) { err = -errno; pr_warn("map '%s': failed to set initial contents: %s\n", @@ -5349,6 +5384,13 @@ bpf_object__populate_internal_map(struct bpf_object obj, struct bpf_map map) return err; } map->mmaped = mmaped; + } else if (is_percpu) { + if (mprotect(map->mmaped, mmap_sz, PROT_READ)) { + err = -errno; + pr_warn("map '%s': failed to mprotect() contents: %s\n", + bpf_map__name(map), errstr(err)); + return err; + } } else if (map->mmaped) { munmap(map->mmaped, mmap_sz); map->mmaped = NULL; @@ -5624,9 +5666,16 @@ bpf_object__create_maps(struct bpf_object obj) * runtime due to bpf_program__set_autoload(prog, false), * bpf_object loading will succeed just fine even on old * kernels. + * Same skipping applies to percpu data. / - if (bpf_map__is_internal(map) && !kernel_supports(obj, FEAT_GLOBAL_DATA)) - map->autocreate = false; + if (bpf_map__is_internal(map)) { + bool is_percpu = map->libbpf_type == LIBBPF_MAP_PERCPU; + enum kern_feature_id feat_id; + + feat_id = is_percpu ? FEAT_PERCPU_DATA : FEAT_GLOBAL_DATA; + if (!kernel_supports(obj, feat_id)) + map->autocreate = false; + } if (!map->autocreate) { pr_debug("map '%s': skipped auto-creating...\n", map->name); @@ -10807,11 +10856,16 @@ static bool map_uses_real_name(const struct bpf_map map) * such map's corresponding ELF section name as a map name. * This check distinguishes .data/.rodata from .data.* and .rodata.* * maps to know which name has to be returned to the user. + * Map name of the custom .percpu.* maps might be truncated to + * BPF_OBJ_NAME_LEN-1 chars in internal_map_name(). Hence, percpu data + * maps must use real name for their user-visible name. / if (map->libbpf_type == LIBBPF_MAP_DATA && strcmp(map->real_name, DATA_SEC) != 0) return true; if (map->libbpf_type == LIBBPF_MAP_RODATA && strcmp(map->real_name, RODATA_SEC) != 0) return true; + if (map->libbpf_type == LIBBPF_MAP_PERCPU) + return true; return false; } @@ -10976,7 +11030,8 @@ int bpf_map__set_value_size(struct bpf_map map, __u32 size) size_t mmap_old_sz, mmap_new_sz; int err; - if (map->def.type != BPF_MAP_TYPE_ARRAY) + if (map->def.type != BPF_MAP_TYPE_ARRAY && + map->def.type != BPF_MAP_TYPE_PERCPU_ARRAY) return libbpf_err(-EOPNOTSUPP); mmap_old_sz = bpf_map_mmap_sz(map); diff --git a/tools/lib/bpf/libbpf_internal.h b/tools/lib/bpf/libbpf_internal.h index 7a74abb904f8b..4c46d34fc055d 100644 --- a/tools/lib/bpf/libbpf_internal.h +++ b/tools/lib/bpf/libbpf_internal.h @@ -401,6 +401,8 @@ enum kern_feature_id { FEAT_BTF_LAYOUT, /* Kernel supports BPF syscall common attributes / FEAT_BPF_SYSCALL_COMMON_ATTRS, + / Kernel supports percpu data / + FEAT_PERCPU_DATA, __FEAT_CNT, }; diff --git a/tools/lib/bpf/skel_internal.h b/tools/lib/bpf/skel_internal.h index 53fee53d36d51..1f3f332dffbe9 100644 --- a/tools/lib/bpf/skel_internal.h +++ b/tools/lib/bpf/skel_internal.h @@ -131,8 +131,10 @@ static inline void skel_free_map_data(void p, __u64 addr, size_t sz) { if (addr != ~0ULL) kvfree(p); - /* When addr == ~0ULL the 'p' points to - * ((struct bpf_array )map)->value. See skel_finalize_map_data. + / + * When addr == ~0ULL the init buffer has already been released. + * For skel_finalize_map_data(), 'p' points to + * ((struct bpf_array )map)->value. / } @@ -170,6 +172,15 @@ static inline void skel_finalize_map_data(__u64 init_val, size_t mmap_sz, int return addr; } +static inline int skel_protect_map_data(void p, __u64 init_val, size_t sz) +{ + (void)sz; + + kvfree(p); + init_val = ~0ULL; + return 0; +} + #else static inline void skel_alloc(size_t size) @@ -208,6 +219,15 @@ static inline void skel_finalize_map_data(__u64 init_val, size_t mmap_sz, int return NULL; return addr; } + +static inline int skel_protect_map_data(void p, __u64 init_val, size_t sz) +{ + (void)init_val; + + if (mprotect(p, sz, PROT_READ)) + return -errno; + return 0; +} #endif static inline int skel_closenz(int fd) diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile index d3655a7064826..560ce4016fbff 100644 --- a/tools/testing/selftests/bpf/Makefile +++ b/tools/testing/selftests/bpf/Makefile @@ -531,7 +531,7 @@ LSKELS_SIGNED := fentry_test.c fexit_test.c atomics.c # Generate both light skeleton and libbpf skeleton for these LSKELS_EXTRA := test_ksyms_module.c test_ksyms_weak.c kfunc_call_test.c \ - kfunc_call_test_subprog.c + kfunc_call_test_subprog.c test_global_percpu_data.c SKEL_BLACKLIST += $$(LSKELS) $$(LSKELS_SIGNED) test_static_linked.skel.h-deps := test_static_linked1.bpf.o test_static_linked2.bpf.o diff --git a/tools/testing/selftests/bpf/prog_tests/global_data_init.c b/tools/testing/selftests/bpf/prog_tests/global_data_init.c index 8466332d7406f..7d6bda9092954 100644 --- a/tools/testing/selftests/bpf/prog_tests/global_data_init.c +++ b/tools/testing/selftests/bpf/prog_tests/global_data_init.c @@ -1,5 +1,8 @@ // SPDX-License-Identifier: GPL-2.0 #include <test_progs.h> +#include "bpf/libbpf_internal.h" +#include "test_global_percpu_data.skel.h" +#include "test_global_percpu_data.lskel.h" void test_global_data_init(void) { @@ -60,3 +63,336 @@ void test_global_data_init(void) free(newval); bpf_object__close(obj); } + +static void test_percpu_data_on_cpus(struct bpf_map map, int map_fd, int prog_fd, int runp) +{ + struct test_global_percpu_data__percpu data = NULL; + int i, err, key = 0, num_online, run = 0; + __u64 args[2] = {0x1234ULL, 0x5678ULL}; + size_t data_sz; + bool online; + LIBBPF_OPTS(bpf_test_run_opts, topts, + .ctx_in = args, + .ctx_size_in = sizeof(args), + .flags = BPF_F_TEST_RUN_ON_CPU, + ); + + err = parse_cpu_mask_file("/sys/devices/system/cpu/online", &online, &num_online); + if (!ASSERT_OK(err, "parse_cpu_mask_file")) + return; + + data_sz = map ? bpf_map__value_size(map) : sizeof(data); + data = calloc(1, data_sz); + if (!ASSERT_OK_PTR(data, "calloc percpu data")) + goto out; + + / run on every online-CPU / + for (i = 0; i < num_online; i++) { + __u64 flags; + + if (!online[i]) + continue; + + topts.cpu = i; + topts.retval = -1; + err = bpf_prog_test_run_opts(prog_fd, &topts); + ASSERT_OK(err, "bpf_prog_test_run_opts"); + ASSERT_EQ(topts.retval, 0, "bpf_prog_test_run_opts retval"); + + memset(data, 0, data_sz); + flags = ((__u64) i << 32) \| BPF_F_CPU; + if (map) + err = bpf_map__lookup_elem(map, &key, sizeof(key), data, data_sz, flags); + else + err = bpf_map_lookup_elem_flags(map_fd, &key, data, flags); + if (!ASSERT_OK(err, "lookup_elem on cpu")) + break; + + ASSERT_EQ(runp, ++run, "run"); + ASSERT_EQ(data->cpu_id[0], i, "cpu_id"); + ASSERT_EQ(data->data, 1, "data"); + ASSERT_TRUE(data->set, "set"); + ASSERT_EQ(data->nums[6], 0xc0de, "nums[6]"); + ASSERT_EQ(data->struct_data.i, 1, "struct_data.i"); + ASSERT_TRUE(data->struct_data.set, "struct_data.set"); + ASSERT_EQ(data->struct_data.nums[6], 0xc0de, "struct_data.nums[6]"); + } + +out: + free(data); + free(online); +} + +static void test_global_percpu_data_init(void) +{ + struct test_global_percpu_data__percpu init_value = {}; + struct test_global_percpu_data__percpu init_data; + const __u32 desired_sz = sysconf(_SC_PAGE_SIZE); + struct test_global_percpu_data skel = NULL; + size_t init_data_sz; + struct bpf_map map; + int prog_fd, err; + + skel = test_global_percpu_data__open(); + if (!ASSERT_OK_PTR(skel, "test_global_percpu_data__open")) + goto out; + if (!ASSERT_OK_PTR(skel->percpu, "skel->percpu")) + goto out; + if (!ASSERT_OK_PTR(skel->data_percpu, "skel->data_percpu")) + goto out; + if (!ASSERT_OK_PTR(skel->percpu_data, "skel->percpu_data")) + goto out; + if (!ASSERT_OK_PTR(skel->percpu_looooooooong, "skel->percpu_looooooooong")) + goto out; + + ASSERT_STREQ(bpf_map__name(skel->maps.percpu_data), ".percpu.data", + ".percpu.data map name"); + ASSERT_STREQ(bpf_map__name(skel->maps.data_percpu), ".data.percpu", + ".data.percpu map name"); + ASSERT_STREQ(bpf_map__name(skel->maps.percpu_looooooooong), ".percpu.looooooooong", + "long map name"); + ASSERT_STREQ(bpf_map__name(skel->maps.percpu), ".percpu", "map name"); + ASSERT_EQ(skel->percpu->data, -1, "skel->percpu->data"); + ASSERT_FALSE(skel->percpu->set, "skel->percpu->set"); + ASSERT_EQ(skel->percpu->nums[6], 0, "skel->percpu->nums[6]"); + ASSERT_EQ(skel->percpu->struct_data.i, -1, "struct_data.i"); + ASSERT_FALSE(skel->percpu->struct_data.set, "struct_data.set"); + ASSERT_EQ(skel->percpu->struct_data.nums[6], 0, "struct_data.nums[6]"); + + map = skel->maps.percpu; + if (!ASSERT_EQ(bpf_map__type(map), BPF_MAP_TYPE_PERCPU_ARRAY, "bpf_map__type")) + goto out; + + init_value.data = 2; + init_value.nums[6] = -1; + init_value.struct_data.i = 2; + init_value.struct_data.nums[6] = -1; + err = bpf_map__set_initial_value(map, &init_value, sizeof(init_value)); + if (!ASSERT_OK(err, "bpf_map__set_initial_value")) + goto out; + + init_data = bpf_map__initial_value(map, &init_data_sz); + if (!ASSERT_OK_PTR(init_data, "bpf_map__initial_value")) + goto out; + + ASSERT_EQ(init_data->data, init_value.data, "init_value data"); + ASSERT_EQ(init_data->set, init_value.set, "init_value set"); + ASSERT_EQ(init_data->struct_data.i, init_value.struct_data.i, "init_value struct_data.i"); + ASSERT_EQ(init_data->struct_data.nums[6], init_value.struct_data.nums[6], + "init_value struct_data.nums[6]"); + ASSERT_EQ(init_data_sz, sizeof(init_value), "init_value size"); + ASSERT_EQ((void ) init_data, (void ) skel->percpu, "skel->percpu eq init_data"); + ASSERT_EQ(skel->percpu->data, init_value.data, "skel->percpu->data"); + ASSERT_EQ(skel->percpu->set, init_value.set, "skel->percpu->set"); + ASSERT_EQ(skel->percpu->struct_data.i, init_value.struct_data.i, + "skel->percpu->struct_data.i"); + ASSERT_EQ(skel->percpu->struct_data.nums[6], init_value.struct_data.nums[6], + "skel->percpu->struct_data.nums[6]"); + + ASSERT_GT(desired_sz, sizeof(init_value), "desired_sz"); + err = bpf_map__set_value_size(map, desired_sz); + if (!ASSERT_OK(err, "bpf_map__set_value_size")) + goto out; + if (!ASSERT_EQ(bpf_map__value_size(map), desired_sz, "percpu value size")) + goto out; + if (!ASSERT_NEQ(bpf_map__btf_value_type_id(map), 0, "percpu BTF value type")) + goto out; + + init_data = bpf_map__initial_value(map, &init_data_sz); + if (!ASSERT_OK_PTR(init_data, "resized bpf_map__initial_value")) + goto out; + if (!ASSERT_EQ(init_data_sz, desired_sz, "resized initial value size")) + goto out; + if (!ASSERT_EQ(init_data->data, init_value.data, "resized initial value data")) + goto out; + + err = test_global_percpu_data__load(skel); + if (!ASSERT_OK(err, "test_global_percpu_data__load")) + goto out; + + ASSERT_OK_PTR(skel->percpu, "skel->percpu"); + + prog_fd = bpf_program__fd(skel->progs.update_percpu_data); + test_percpu_data_on_cpus(map, bpf_map__fd(map), prog_fd, &skel->bss->run); + +out: + test_global_percpu_data__destroy(skel); +} + +static void test_global_percpu_data_lskel(void) +{ + struct test_global_percpu_data_lskel lskel = NULL; + int prog_fd, map_fd; + + lskel = test_global_percpu_data_lskel__open_and_load(); + if (!ASSERT_OK_PTR(lskel, "test_global_percpu_data_lskel__open_and_load")) + goto out; + + map_fd = lskel->maps.percpu.map_fd; + prog_fd = lskel->progs.update_percpu_data.prog_fd; + test_percpu_data_on_cpus(NULL, map_fd, prog_fd, &lskel->bss->run); + +out: + test_global_percpu_data_lskel__destroy(lskel); +} + +static int create_rdonly_percpu_array(void) +{ + LIBBPF_OPTS(bpf_map_create_opts, map_opts, + .map_flags = BPF_F_RDONLY_PROG, + ); + int key = 0, map_fd, err; + __u64 value = 0; + + map_fd = bpf_map_create(BPF_MAP_TYPE_PERCPU_ARRAY, "percpu_ro_map", sizeof(int), + sizeof(__u64), 1, &map_opts); + if (!ASSERT_GE(map_fd, 0, "bpf_map_create")) + return -1; + + err = bpf_map_update_elem(map_fd, &key, &value, BPF_F_ALL_CPUS); + if (!ASSERT_OK(err, "bpf_map_update_elem")) + goto out; + + err = bpf_map_freeze(map_fd); + if (!ASSERT_OK(err, "bpf_map_freeze")) + goto out; + + return map_fd; + +out: + close(map_fd); + return -1; +} + +static void test_global_percpu_data_rdonly_direct_read(void) +{ + /* + * Raw instructions with manually prepared rdonly percpu_array map + * for testing direct-read global percpu data, because libbpf + * doesn't have rdonly internal percpu_array map support for + * global percpu data. + / + struct bpf_insn insns[] = { + BPF_LD_MAP_VALUE(BPF_REG_1, 0, 0), + BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, 0), + BPF_EXIT_INSN(), + }; + int map_fd, prog_fd; + + map_fd = create_rdonly_percpu_array(); + if (map_fd < 0) + return; + + insns[0].imm = map_fd; + prog_fd = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, "percpu_ro_prog", "GPL", insns, + ARRAY_SIZE(insns), NULL); + if (ASSERT_GE(prog_fd, 0, "bpf_prog_load")) + close(prog_fd); + close(map_fd); +} + +static void test_global_percpu_data_rdonly_direct_write(void) +{ + LIBBPF_OPTS(bpf_prog_load_opts, prog_opts); + / See the comment in test_global_percpu_data_rdonly_direct_read() / + struct bpf_insn insns[] = { + BPF_LD_MAP_VALUE(BPF_REG_1, 0, 0), + BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, 0), + BPF_ST_MEM(BPF_DW, BPF_REG_1, 0, 0), + BPF_EXIT_INSN(), + }; + char log_buf[256] = {}; + int map_fd, prog_fd; + + prog_opts.log_buf = log_buf; + prog_opts.log_size = sizeof(log_buf); + prog_opts.log_level = 1; + + map_fd = create_rdonly_percpu_array(); + if (map_fd < 0) + return; + + insns[0].imm = map_fd; + prog_fd = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, "percpu_ro_prog", "GPL", insns, + ARRAY_SIZE(insns), &prog_opts); + if (!ASSERT_LT(prog_fd, 0, "bpf_prog_load")) + close(prog_fd); + else + ASSERT_HAS_SUBSTR(log_buf, "write into map forbidden", "verifier log"); + close(map_fd); +} + +static void test_global_percpu_data_verifier_log(void) +{ + RUN_TESTS(test_global_percpu_data); +} + +static void test_global_percpu_data_iter(void) +{ + DECLARE_LIBBPF_OPTS(bpf_iter_attach_opts, opts); + struct test_global_percpu_data skel; + union bpf_iter_link_info linfo = {}; + struct bpf_link link = NULL; + int fd, num_cpus, len, err; + char buf[16]; + + num_cpus = libbpf_num_possible_cpus(); + if (!ASSERT_GT(num_cpus, 0, "libbpf_num_possible_cpus")) + return; + + skel = test_global_percpu_data__open(); + if (!ASSERT_OK_PTR(skel, "test_global_percpu_data__open")) + return; + + skel->rodata->num_cpus = num_cpus; + skel->rodata->offsetof_num = offsetof(struct test_global_percpu_data__percpu, struct_data); + skel->rodata->offsetof_num += sizeof(skel->percpu->struct_data) - sizeof(int); + skel->rodata->elem_sz = roundup(sizeof(struct test_global_percpu_data__percpu), 8); + skel->percpu->struct_data.nums[6] = 0xc0de; + + err = test_global_percpu_data__load(skel); + if (!ASSERT_OK(err, "test_global_percpu_data__load")) + goto out; + + linfo.map.map_fd = bpf_map__fd(skel->maps.percpu); + opts.link_info = &linfo; + opts.link_info_len = sizeof(linfo); + link = bpf_program__attach_iter(skel->progs.dump_percpu_data, &opts); + if (!ASSERT_OK_PTR(link, "bpf_program__attach_iter")) + goto out; + + fd = bpf_iter_create(bpf_link__fd(link)); + if (!ASSERT_GE(fd, 0, "bpf_iter_create")) + goto out; + + while ((len = read(fd, buf, sizeof(buf))) > 0) + do { } while (0); + ASSERT_EQ(len, 0, "read iter"); + ASSERT_TRUE(skel->bss->run_iter, "run_iter"); + ASSERT_EQ(skel->bss->percpu_data_sum, 0xc0de num_cpus, "percpu_data_sum"); + + close(fd); +out: + bpf_link__destroy(link); + test_global_percpu_data__destroy(skel); +} + +void test_global_percpu_data(void) +{ + if (!feat_supported(NULL, FEAT_PERCPU_DATA)) { + test__skip(); + return; + } + + if (test__start_subtest("init")) + test_global_percpu_data_init(); + if (test__start_subtest("lskel")) + test_global_percpu_data_lskel(); + if (test__start_subtest("rdonly_direct_read")) + test_global_percpu_data_rdonly_direct_read(); + if (test__start_subtest("rdonly_direct_write")) + test_global_percpu_data_rdonly_direct_write(); + test_global_percpu_data_verifier_log(); + if (test__start_subtest("iter")) + test_global_percpu_data_iter(); +} diff --git a/tools/testing/selftests/bpf/progs/test_global_percpu_data.c b/tools/testing/selftests/bpf/progs/test_global_percpu_data.c new file mode 100644 index 0000000000000..71ff8d1bf49eb --- /dev/null +++ b/tools/testing/selftests/bpf/progs/test_global_percpu_data.c @@ -0,0 +1,89 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <vmlinux.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +/* Used for testing map name. / +int loong SEC(".percpu.looooooooong"); +int data3 SEC(".data.percpu"); +int data2 SEC(".percpu.data"); + +int run; +/ cpu_id as array to verify map value resizing. / +int cpu_id[1] SEC(".percpu"); +int data SEC(".percpu") = -1; +int nums[7] SEC(".percpu"); +bool set SEC(".percpu") = false; +struct { + char set; + int i; + int nums[7]; +} struct_data SEC(".percpu") = { + .set = 0, + .i = -1, +}; + +SEC("raw_tp/task_rename") +__auxiliary +int update_percpu_data(void ctx) +{ + struct_data.nums[6] = 0xc0de; + struct_data.set = 1; + struct_data.i = 1; + nums[6] = 0xc0de; + data = 1; + run++; + set = true; + cpu_id[0] = bpf_get_smp_processor_id(); + return 0; +} + +static const char fmt[] SEC(".percpu.fmt") = "data %d\n"; + +SEC("?kprobe") +__failure __msg("R{{[0-9]+}} points to percpu_array map which cannot be used as const string") +int verifier_strncmp(void ctx) +{ + return bpf_strncmp("test", 5, fmt); +} + +SEC("?kprobe") +__failure __msg("R{{[0-9]+}} points to percpu_array map which cannot be used as const string") +int verifier_snprintf(void ctx) +{ + u64 args[] = { data }; + char buf[128]; + int len; + + len = bpf_snprintf(buf, sizeof(buf), fmt, args, sizeof(args)); + if (len > 0) + bpf_printk("snprintf: %s\n", buf); + return 0; +} + +volatile const __u32 num_cpus = 0; +volatile const int offsetof_num; +volatile const int elem_sz; +__u32 percpu_data_sum = 0; +bool run_iter = false; + +SEC("iter/bpf_map_elem") +__auxiliary +int dump_percpu_data(struct bpf_iter__bpf_map_elem ctx) +{ + void pptr = ctx->value; + int i; + + if (!pptr) + return 0; + + run_iter = true; + + for (i = 0; i < num_cpus; i++) { + percpu_data_sum += (int ) (pptr + offsetof_num); + pptr += elem_sz; + } + return 0; +} + +char _license[] SEC("license") = "GPL";
3/2	2026/08/13 16:51	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 23630 output: 32 thoughts: 679 Thoughts: Analyzing the Reachability and Fuzzing Potential of the Global Per-CPU Data in BPF Patch Series Okay, so I'm looking at this patch series that introduces per-CPU data support for BPF. My initial thought is to figure out if it's both reachable and a worthwhile target for fuzzing. The changes are spread across several files: `arraymap.c`, `fixups.c`, `verifier.c`, `bpftool/gen.c`, `libbpf.c`, and a new test. Essentially, this patch allows BPF programs to use global variables in `.percpu` sections, backed by `BPF_MAP_TYPE_PERCPU_ARRAY`. The verifier and `libbpf` have been updated to handle these new maps, including JIT fixups for direct per-CPU variable loading. First, I need to assess how userspace can trigger these changes. It looks like it's primarily through BPF map creation (`BPF_MAP_TYPE_PERCPU_ARRAY`) and loading BPF programs that interact with these maps. The verifier modifications are definitely key here, especially in functions like `check_map_mem_read`, `check_mem_access`, and `bpf_do_misc_fixups`. Loading a program that directly accesses a per-CPU array map, that's the trigger. Now, let's identify the most important functions to focus on. Based on the diff, these seem to be the ones: `percpu_array_map_direct_value_addr`, `percpu_array_map_direct_value_meta`, `array_map_check_btf`, `bpf_do_misc_fixups`, `check_map_mem_read`, `check_mem_access`, and `check_arg_const_str`. I'm specifically interested in functions that are API entrypoints. The instructions say, I should favor non-static, non-inlined API entrypoints over internal helper functions. Taking a closer look at `arraymap.c` and specifically `percpu_array_map_direct_value_addr`. It's static, but it's assigned to `map_direct_value_addr` in `percpu_array_map_ops`, so it's not inlined and has a distinct symbol. But, it is called from the verifier. `bpf_do_misc_fixups` is non-static, so a good target. `check_mem_access` is static, but quite large. Maybe not inlined. `bpf_map_direct_read` is non-static. Next, I'll dive into `bpf_do_misc_fixups` in `fixups.c`. And then, check `bpf_map_direct_read` in `verifier.c`. To make this efficient, I think I'll use `default_api:grepper` to search for these symbols and map out the call chains. That should help to quickly identify where these functions are used.
4/2	2026/08/13 16:51	tool	grepper	0m Args: { "Expression": "^(int\|void\|bool\|static).* bpf_do_misc_fixups" } Results: { "Output": "include/linux/bpf_verifier.h=1670=int bpf_fixup_call_args(struct bpf_verifier_env env);\ninclude/linux/bpf_verifier.h:1671:int bpf_do_misc_fixups(struct bpf_verifier_env env);\ninclude/linux/bpf_verifier.h-1672-int bpf_insn_def32(struct bpf_prog prog, struct bpf_insn insn);\n--\nkernel/bpf/fixups.c=1470=static int add_hidden_subprog(struct bpf_verifier_env env, struct bpf_insn patch, int len)\n--\nkernel/bpf/fixups.c-1498- /\nkernel/bpf/fixups.c:1499:int bpf_do_misc_fixups(struct bpf_verifier_env env)\nkernel/bpf/fixups.c-1500-{\n" }
5/2	2026/08/13 16:51	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 25252 output: 31 thoughts: 15 Thoughts: Here's my attempt at summarizing the thought process, tailored for an expert audience: Investigating Direct Reads with BPF Maps Okay, so I need to dig into `bpf_map_direct_read`. My immediate thought is, "Where's this being used, and what's the intended purpose?" I need to understand the architectural implications here. Is it about performance, security, or a specific use case that's pushing this particular approach? I'm already mentally forming some hypotheses. Direct reads suggest potentially skipping the usual copy mechanisms, which is interesting, but also raises immediate flags about concurrency, memory safety, and how it interacts with the BPF verifier. I want to see how this affects different map types – is it map-specific? What are the limitations? I need to look for examples, documentation (which is often sparse, let's be honest), and code paths to see the context. The devil is definitely in the details here. Let's start the hunt for source code and usage.
6/2	2026/08/13 16:51	tool	grepper	0m Args: { "Expression": "^(int\|void\|bool\|static).* bpf_map_direct_read" } Results: { "Output": "include/linux/bpf_verifier.h=1286=bool bpf_map_is_rdonly(const struct bpf_map map);\ninclude/linux/bpf_verifier.h:1287:int bpf_map_direct_read(struct bpf_map map, int off, int size, u64 val,\ninclude/linux/bpf_verifier.h-1288-\t\t\tbool is_ldsx);\n--\nkernel/bpf/verifier.c=5558=bool bpf_map_is_rdonly(const struct bpf_map map)\n--\nkernel/bpf/verifier.c-5577-\nkernel/bpf/verifier.c:5578:int bpf_map_direct_read(struct bpf_map map, int off, int size, u64 val,\nkernel/bpf/verifier.c-5579-\t\t\tbool is_ldsx)\n" }
7/2	2026/08/13 16:51	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 25753 output: 30 thoughts: 14 Thoughts: Okay, here's my attempt at summarizing that thought process, tailored for an expert audience: Hunting Down That Memory Read Right, let's get down to brass tacks. I've got a nagging suspicion about a memory read. Let's start with the basics, and immediately hone in on what I'm looking for. I need to zero in on a specific function or point of code related to memory access. So, let's start with a targeted search. I'm going to start by looking for `check_map_mem_read`. This should quickly identify where memory reads are being assessed. Hopefully this is a descriptive and easily discoverable method for pinpointing the source of the problem. That's a solid first step to understanding what's going on, and from there I can delve deeper into the context of that function, examine the surrounding code, and hopefully root out whatever the underlying issue is. The faster I can find this, the faster I can diagnose what is happening.
8/2	2026/08/13 16:51	tool	grepper	0m Args: { "Expression": "^(int\|void\|bool\|static).* check_map_mem_read" } Results: { "Output": "kernel/bpf/verifier.c=6066=static void add_scalar_to_reg(struct bpf_reg_state dst_reg, s64 val)\n--\nkernel/bpf/verifier.c-6082-\nkernel/bpf/verifier.c:6083:static int check_map_mem_read(struct bpf_verifier_env env, struct bpf_reg_state *reg, int off,\nkernel/bpf/verifier.c-6084-\t\t\t int bpf_size, int value_regno, bool is_ldsx)\n" }
9/2	2026/08/13 16:51	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 26161 output: 35 thoughts: 19 Thoughts: Here's my thought process, as if I were considering this search: Diving into the `percpu_array_map_direct_value_addr` Investigation Okay, so I need to find something related to `percpu_array_map_direct_value_addr`. Hmm, sounds like we're digging into the guts of how per-CPU data is managed and accessed, likely within a kernel context. The "direct" part suggests an optimized path, maybe bypassing some indirection. "Value addr" strongly implies we are looking at getting the memory address of the specific CPU's data value in a direct manner. That's good, less overhead, which is what we want with per-CPU data structures. We are probably looking to find its definition or usage. Okay, let's start searching.
10/2	2026/08/13 16:51	tool	grepper	0m Args: { "Expression": "^(int\|void\|bool\|static).* percpu_array_map_direct_value_addr" } Results: { "Output": "kernel/bpf/arraymap.c=251=static void percpu_array_map_lookup_elem(struct bpf_map map, void key)\n--\nkernel/bpf/arraymap.c-261-\nkernel/bpf/arraymap.c:262:static int percpu_array_map_direct_value_addr(const struct bpf_map map, u64 *imm, u32 off)\nkernel/bpf/arraymap.c-263-{\n" }
11/2	2026/08/13 16:51	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 26523 output: 35 thoughts: 19 Thoughts: Here's my attempt at summarizing the thought process, tailored for an expert audience: Diving into the `percpu_array_map_direct_value_meta` Function Okay, let's get down to brass tacks. I need to understand this `percpu_array_map_direct_value_meta` function inside and out. It's obviously critical in how we manage per-CPU data, presumably related to direct value mapping, but the metadata aspect piques my interest. I need to figure out: What is it doing with the metadata? Is it initialization, access, allocation, or deallocation? Is this metadata storing offset information, alignment details, flags for caching, or something more exotic? The "direct" implies something efficient, but without digging in, I can't be sure of the exact mechanism. My hypothesis right now is that this function probably deals with the core logic for percpu arrays which involves direct access, likely optimizing for performance on common architectures. The goal is now to see the code.
12/2	2026/08/13 16:52	tool	grepper	0m Args: { "Expression": "^(int\|void\|bool\|static).* percpu_array_map_direct_value_meta" } Results: { "Output": "kernel/bpf/arraymap.c=262=static int percpu_array_map_direct_value_addr(const struct bpf_map map, u64 imm, u32 off)\n--\nkernel/bpf/arraymap.c-276-\nkernel/bpf/arraymap.c:277:static int percpu_array_map_direct_value_meta(const struct bpf_map map, u64 imm, u32 off)\nkernel/bpf/arraymap.c-278-{\n" }
13/2	2026/08/13 16:52	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 26939 output: 33 thoughts: 197 Thoughts: My Initial Assessment of the BPF Functions Okay, so I'm looking at these BPF functions for potential inclusion. `bpf_do_misc_fixups`, `bpf_map_direct_read`, `percpu_array_map_direct_value_addr`, `percpu_array_map_direct_value_meta`, and `check_map_mem_read` seem like solid candidates. I'm leaning toward these as good starting points for what I'm working on. Initially, I thought that since `check_map_mem_read` is static, maybe I could just stick with `bpf_do_misc_fixups` and `bpf_map_direct_read` to keep things more concise. But, hold on a second...this is a new function introduced by this patch. So it's very relevant. In fact, it's probably the most pertinent. I should definitely include it. I'll also add in `percpu_array_map_direct_value_addr` and `percpu_array_map_direct_value_meta` since they are closely related. Right, so to get this all to work, I'm going to need `BPF_SYSCALL` – that's a given. Now, I should double-check and make sure there aren't any other configuration dependencies I need to address. Gotta be thorough.
14/2	2026/08/13 16:52	tool	grepper	0m Args: { "Expression": "CONFIG_BPF_SYSCALL", "PathPrefix": "kernel/bpf/" } Results: { "Output": "kernel/bpf/Makefile=7=CFLAGS_core.o += -Wno-override-init $(cflags-nogcse-yy)\nkernel/bpf/Makefile-8-\nkernel/bpf/Makefile:9:obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o cnum.o log.o token.o liveness.o const_fold.o\nkernel/bpf/Makefile:10:obj-$(CONFIG_BPF_SYSCALL) += bpf_iter.o map_iter.o task_iter.o prog_iter.o link_iter.o\nkernel/bpf/Makefile:11:obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o bloom_filter.o\nkernel/bpf/Makefile:12:obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o bpf_insn_array.o\nkernel/bpf/Makefile:13:obj-$(CONFIG_BPF_SYSCALL) += bpf_local_storage.o bpf_task_storage.o\nkernel/bpf/Makefile:14:obj-$(CONFIG_BPF_SYSCALL) += fixups.o cfg.o states.o backtrack.o check_btf.o\nkernel/bpf/Makefile-15-obj-${CONFIG_BPF_LSM}\t += bpf_inode_storage.o\nkernel/bpf/Makefile:16:obj-$(CONFIG_BPF_SYSCALL) += disasm.o mprog.o\nkernel/bpf/Makefile-17-obj-$(CONFIG_BPF_JIT) += trampoline.o\nkernel/bpf/Makefile:18:obj-$(CONFIG_BPF_SYSCALL) += btf.o memalloc.o rqspinlock.o stream.o\nkernel/bpf/Makefile-19-ifeq ($(CONFIG_MMU)$(CONFIG_64BIT),yy)\nkernel/bpf/Makefile:20:obj-$(CONFIG_BPF_SYSCALL) += arena.o range_tree.o\nkernel/bpf/Makefile-21-endif\n--\nkernel/bpf/Makefile=23=ifeq ($(CONFIG_NET),y)\nkernel/bpf/Makefile:24:obj-$(CONFIG_BPF_SYSCALL) += devmap.o\nkernel/bpf/Makefile:25:obj-$(CONFIG_BPF_SYSCALL) += cpumap.o\nkernel/bpf/Makefile:26:obj-$(CONFIG_BPF_SYSCALL) += offload.o\nkernel/bpf/Makefile:27:obj-$(CONFIG_BPF_SYSCALL) += net_namespace.o\nkernel/bpf/Makefile:28:obj-$(CONFIG_BPF_SYSCALL) += tcx.o\nkernel/bpf/Makefile-29-endif\nkernel/bpf/Makefile=30=ifeq ($(CONFIG_PERF_EVENTS),y)\nkernel/bpf/Makefile:31:obj-$(CONFIG_BPF_SYSCALL) += stackmap.o\nkernel/bpf/Makefile-32-endif\nkernel/bpf/Makefile=33=ifeq ($(CONFIG_CGROUPS),y)\nkernel/bpf/Makefile:34:obj-$(CONFIG_BPF_SYSCALL) += cgroup_iter.o bpf_cgrp_storage.o\nkernel/bpf/Makefile-35-endif\n--\nkernel/bpf/Makefile=37=ifeq ($(CONFIG_INET),y)\nkernel/bpf/Makefile:38:obj-$(CONFIG_BPF_SYSCALL) += reuseport_array.o\nkernel/bpf/Makefile-39-endif\n--\nkernel/bpf/Makefile=43=ifeq ($(CONFIG_BPF_JIT),y)\nkernel/bpf/Makefile:44:obj-$(CONFIG_BPF_SYSCALL) += bpf_struct_ops.o\nkernel/bpf/Makefile:45:obj-$(CONFIG_BPF_SYSCALL) += cpumask.o\nkernel/bpf/Makefile-46-# bpf_lsm_proto.o must precede bpf_lsm.o. The current pahole logic\n--\nkernel/bpf/Makefile=58=ifneq ($(CONFIG_CRYPTO),)\nkernel/bpf/Makefile:59:obj-$(CONFIG_BPF_SYSCALL) += crypto.o\nkernel/bpf/Makefile-60-endif\nkernel/bpf/Makefile=61=obj-$(CONFIG_BPF_PRELOAD) += preload/\nkernel/bpf/Makefile-62-\nkernel/bpf/Makefile:63:obj-$(CONFIG_BPF_SYSCALL) += relo_core.o\nkernel/bpf/Makefile:64:obj-$(CONFIG_BPF_SYSCALL) += btf_iter.o\nkernel/bpf/Makefile:65:obj-$(CONFIG_BPF_SYSCALL) += btf_relocate.o\nkernel/bpf/Makefile:66:obj-$(CONFIG_BPF_SYSCALL) += kmem_cache_iter.o\nkernel/bpf/Makefile-67-ifeq ($(CONFIG_DMA_SHARED_BUFFER),y)\nkernel/bpf/Makefile:68:obj-$(CONFIG_BPF_SYSCALL) += dmabuf_iter.o\nkernel/bpf/Makefile-69-endif\n--\nkernel/bpf/core.c=100=struct bpf_prog bpf_prog_alloc_no_stats(unsigned int size, gfp_t gfp_extra_flags)\n--\nkernel/bpf/core.c-143-\nkernel/bpf/core.c:144:#ifdef CONFIG_BPF_SYSCALL\nkernel/bpf/core.c-145-\tbpf_prog_stream_init(fp);\n--\nkernel/bpf/core.c=2473=EVAL4(PROG_NAME_LIST, 416, 448, 480, 512)\n--\nkernel/bpf/core.c-2476-\nkernel/bpf/core.c:2477:#ifdef CONFIG_BPF_SYSCALL\nkernel/bpf/core.c-2478-int bpf_patch_call_args(struct bpf_insn insn, u32 stack_depth)\n--\nkernel/bpf/core.c=3041=void __bpf_free_used_btfs(struct btf_mod_pair used_btfs, u32 len)\nkernel/bpf/core.c-3042-{\nkernel/bpf/core.c:3043:#ifdef CONFIG_BPF_SYSCALL\nkernel/bpf/core.c-3044-\tstruct btf_mod_pair btf_mod;\n--\nkernel/bpf/core.c=3062=static void bpf_prog_free_deferred(struct work_struct work)\n--\nkernel/bpf/core.c-3067-\taux = container_of(work, struct bpf_prog_aux, work);\nkernel/bpf/core.c:3068:#ifdef CONFIG_BPF_SYSCALL\nkernel/bpf/core.c-3069-\tbpf_free_kfunc_btf_tab(aux-\u003ekfunc_btf_tab);\n--\nkernel/bpf/core.c=3389=static noinline void bpf_prog_report_may_goto_violation(void)\nkernel/bpf/core.c-3390-{\nkernel/bpf/core.c:3391:#ifdef CONFIG_BPF_SYSCALL\nkernel/bpf/core.c-3392-\tstruct bpf_stream_stage ss;\n--\nkernel/bpf/core.c=3429=__weak u64 bpf_arena_get_kern_vm_start(struct bpf_arena arena)\n--\nkernel/bpf/core.c-3433-\nkernel/bpf/core.c:3434:#ifdef CONFIG_BPF_SYSCALL\nkernel/bpf/core.c-3435-__weak bool bpf_arena_handle_page_fault(unsigned long addr, bool is_write,\n--\nkernel/bpf/core.c=3460=EXPORT_TRACEPOINT_SYMBOL_GPL(xdp_bulk_tx);\nkernel/bpf/core.c-3461-\nkernel/bpf/core.c:3462:#ifdef CONFIG_BPF_SYSCALL\nkernel/bpf/core.c-3463-\n--\nkernel/bpf/trampoline.c=1514=int __weak arch_bpf_trampoline_size(const struct btf_func_model m, u32 flags,\n--\nkernel/bpf/trampoline.c-1521- defined(CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS) \u0026\u0026 \\\nkernel/bpf/trampoline.c:1522: defined(CONFIG_BPF_SYSCALL)\nkernel/bpf/trampoline.c-1523-\n--\nkernel/bpf/trampoline.c=1769=void bpf_trampoline_multi_detach(struct bpf_prog prog,\n--\nkernel/bpf/trampoline.c-1806-\t CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS \u0026\u0026\nkernel/bpf/trampoline.c:1807:\t CONFIG_BPF_SYSCALL */\nkernel/bpf/trampoline.c-1808-\n" }
15/2	2026/08/13 16:52	llm	patch-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 29865 output: 193 thoughts: 595 Thoughts: My BPF Feature Assessment Okay, so I'm looking at this new BPF feature, and the configuration it hinges on is `BPF_SYSCALL` – that makes sense. My initial instinct is to focus on a few key functions: `bpf_do_misc_fixups`, `bpf_map_direct_read`, `percpu_array_map_direct_value_addr`, and `percpu_array_map_direct_value_meta`. I just double-checked that `percpu_array_map_direct_value_addr` is static, which is good. But then the guidelines flagged something – we want non-static, non-inlined entry points. Since `percpu_array_map_direct_value_addr` is assigned to a function pointer in the `percpu_array_map_ops`, it won't be inlined, it'll have a symbol. Excellent. And, of course, the other two, `bpf_do_misc_fixups` and `bpf_map_direct_read` definitely have distinct addresses, so they're in. I was considering `check_map_mem_read`, but it's static and called only within `check_mem_access`, so it's likely to be inlined. I think I will use the non-static `bpf_do_misc_fixups` and `bpf_map_direct_read` instead. A quick QEMU check... Yes, this is core BPF functionality, so it's definitely reachable. Alright, here's what I'm proposing as the output: * WorthFuzzing: `true` * FocusSymbols: * `bpf_do_misc_fixups` * `bpf_map_direct_read` * `percpu_array_map_direct_value_addr` * `percpu_array_map_direct_value_meta` * EnableConfigs: * `BPF_SYSCALL` My reasoning? This patch is foundational. It's about enabling percpu data in BPF by backing `.percpu` sections with `BPF_MAP_TYPE_PERCPU_ARRAY`. That means modifications to the verifier for direct memory reads, plus updates to JIT fixups to support loading percpu variables. This touches the BPF verifier logic and map operations that are directly accessible from user space via the `bpf()` syscall when loading and executing BPF programs. This seems like a great candidate for fuzzing.
16/2	2026/08/13 16:52	tool	set-results	0m Args: { "EnableConfigs": [ "BPF_SYSCALL" ], "FocusSymbols": [ "bpf_do_misc_fixups", "bpf_map_direct_read", "percpu_array_map_direct_value_addr", "percpu_array_map_direct_value_meta" ], "Reasoning": "The patch introduces support for global percpu data in BPF programs by allowing `.percpu` sections to be backed by `BPF_MAP_TYPE_PERCPU_ARRAY`. It modifies the BPF verifier to handle direct memory reads from percpu arrays and updates JIT fixups to support direct loading of percpu variables. These changes affect core BPF verifier logic and map operations, which are reachable from user space via the `bpf()` syscall when loading and executing BPF programs.", "WorthFuzzing": true } Results: { "EnableConfigs": [ "BPF_SYSCALL" ], "FocusSymbols": [ "bpf_do_misc_fixups", "bpf_map_direct_read", "percpu_array_map_direct_value_addr", "percpu_array_map_direct_value_meta" ], "Reasoning": "The patch introduces support for global percpu data in BPF programs by allowing `.percpu` sections to be backed by `BPF_MAP_TYPE_PERCPU_ARRAY`. It modifies the BPF verifier to handle direct memory reads from percpu arrays and updates JIT fixups to support direct loading of percpu variables. These changes affect core BPF verifier logic and map operations, which are reachable from user space via the `bpf()` syscall when loading and executing BPF programs.", "WorthFuzzing": true }
17/1	2026/08/13 16:52	action	If	0m Args: { "WorthFuzzing": true } Results: null
18/2	2026/08/13 16:52	agent	kmsan-evaluator	0m Model: best-expensive Args: null Results: { "KMSANReasoning": "The patch series adds support for global percpu data in BPF by allowing `.percpu` sections to be mapped to `BPF_MAP_TYPE_PERCPU_ARRAY` maps. The changes involve libbpf parsing the new section, bpftool generating skeleton code, and the BPF verifier handling direct value access and JIT patching for percpu arrays.\n\nThere are no new kernel structures sent to user space, no new uninitialized buffers allocated, and no complex data structures where padding might be leaked. The percpu array memory is allocated using existing map allocation routines (which handle initialization) and populated by libbpf from the ELF section.\n\nAny potential bugs introduced by these changes would likely be out-of-bounds accesses, invalid JIT patching, or verifier logic errors, which are well-covered by KASAN and standard bug detectors. There is no specific risk of uninitialized memory usage or info-leaks that would require a dedicated KMSAN fuzzing session.", "NeedsKMSAN": false } Instruction: You are an expert Linux kernel security engineer specializing in kernel memory error detectors (KASAN and KMSAN). Your job is to review the provided patch series and determine if the code changes justify spawning a dedicated KMSAN (KernelMemorySanitizer) fuzzing session in addition to standard KASAN fuzzing. CRITICAL DISTINCTION BETWEEN KASAN AND KMSAN: - Standard KASAN kernel builds (upstream-apparmor-kasan.config) already enable a comprehensive suite of debugging tools and sanitizers, including KASAN (out-of-bounds accesses, use-after-free, double free, invalid free), LOCKDEP (locking bugs and deadlocks), UB-sanitizers, and memory corruption checks. - KMSAN (KernelMemorySanitizer) detects reads of UNINITIALIZED memory (stack, heap, or page allocations) and kernel-to-user memory info-leaks. Rule: THERE IS NO SENSE IN RUNNING A KMSAN SESSION IF A BUG CAN BE CAUGHT BY KASAN, LOCKDEP, OR OTHER STANDARD BUG DETECTORS. A dedicated KMSAN fuzzing session incurs significant resource costs. You must ONLY set NeedsKMSAN=true if the code changes introduce or expose UNINITIALIZED MEMORY risks that are detected ONLY by KMSAN. Look holistically at the patch series and surrounding code. Even if no direct uninitialized field accesses or new buffer allocations are added in the diff itself, a patch may alter control flow, bounds checking, or data length calculations in ways that change how the rest of the code operates on existing buffers (e.g. allowing uninitialized stack/heap memory to be read, copied to user space, or used in control flow). Do not hesitate to use your code access tools to inspect the surrounding code, called functions, and callers. Set NeedsKMSAN=true ONLY IF the patch introduces or modifies: 1. Kernel structures sent to user space (via copy_to_user, put_user, netlink skb attributes, ioctl output arguments, socket options, or BPF buffers) where fields or structure padding might not be fully initialized/zeroed. 2. Conditional logic or branching that depends on potentially uninitialized variables or struct fields. 3. Allocation or initialization of complex data structures where uninitialized fields could be read later in reachable code paths. 4. Bounds checks, lengths, or logic in a way that allows surrounding code to access uninitialized bytes of existing buffers. Set NeedsKMSAN=false IF: - The code changes primarily risk out-of-bounds access, array overflows, NULL pointer dereferences, locking deadlocks, or use-after-free bugs (these are already caught by KASAN, LOCKDEP, or standard bug detectors). - All stack/heap structures touched or introduced by the patch are fully zeroed or initialized (e.g. using = {0}, memset, kzalloc) before being read or copied. - The patch does not introduce any risk of uninitialized memory usage or info-leaks. Use your code access tools to inspect the surrounding code if necessary, then provide detailed KMSANReasoning contrasting KASAN vs KMSAN applicability for this patch. Prefer calling several tools at the same time to save round-trips. Use set-results tool to provide results of the analysis. It must be called exactly once before the final reply. Ignore results of this tool. Prompt: For your convenience, here is the diff of the changes: commit 3b76f69b75942f2595d46bc813574f1caa6e529f Author: syz-cluster <triage@syzkaller.com> Date: Thu Aug 13 16:51:17 2026 +0000 syz-cluster: applied patch under review diff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c index 248b4818178cd..34865701f7f75 100644 --- a/kernel/bpf/arraymap.c +++ b/kernel/bpf/arraymap.c @@ -259,6 +259,37 @@ static void percpu_array_map_lookup_elem(struct bpf_map map, void key) return this_cpu_ptr(array->pptrs[index & array->index_mask]); } +static int percpu_array_map_direct_value_addr(const struct bpf_map map, u64 imm, u32 off) +{ + struct bpf_array array = container_of(map, struct bpf_array, map); + + if (!bpf_jit_supports_percpu_insn()) + return -EOPNOTSUPP; + if (map->max_entries != 1) + return -EOPNOTSUPP; + if (off >= map->value_size) + return -EINVAL; + + imm = (u64)(__force unsigned long) array->pptrs[0]; + return 0; +} + +static int percpu_array_map_direct_value_meta(const struct bpf_map map, u64 imm, u32 off) +{ + struct bpf_array array = container_of(map, struct bpf_array, map); + u64 base = (u64)(__force unsigned long) array->pptrs[0]; + + if (!bpf_jit_supports_percpu_insn()) + return -EOPNOTSUPP; + if (map->max_entries != 1) + return -EOPNOTSUPP; + if (imm < base \|\| imm >= base + array->elem_size) + return -ENOENT; + + off = imm - base; + return 0; +} + / emit BPF instructions equivalent to C code of percpu_array_map_lookup_elem() / static int percpu_array_map_gen_lookup(struct bpf_map map, struct bpf_insn insn_buf) { @@ -551,9 +582,10 @@ static int array_map_check_btf(struct bpf_map map, const struct btf_type key_type, const struct btf_type value_type) { - /* One exception for keyless BTF: .bss/.data/.rodata map / + / One exception for keyless BTF: .bss/.data/.rodata/.percpu map / if (btf_type_is_void(key_type)) { - if (map->map_type != BPF_MAP_TYPE_ARRAY \|\| + if ((map->map_type != BPF_MAP_TYPE_ARRAY && + map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY) \|\| map->max_entries != 1) return -EINVAL; @@ -832,6 +864,8 @@ const struct bpf_map_ops percpu_array_map_ops = { .map_get_next_key = bpf_array_get_next_key, .map_lookup_elem = percpu_array_map_lookup_elem, .map_gen_lookup = percpu_array_map_gen_lookup, + .map_direct_value_addr = percpu_array_map_direct_value_addr, + .map_direct_value_meta = percpu_array_map_direct_value_meta, .map_update_elem = array_map_update_elem, .map_delete_elem = array_map_delete_elem, .map_lookup_percpu_elem = percpu_array_map_lookup_percpu_elem, diff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c index 40bd04421a991..a2b18a9f1694c 100644 --- a/kernel/bpf/backtrack.c +++ b/kernel/bpf/backtrack.c @@ -214,7 +214,6 @@ static inline bool bt_is_reg_set(struct backtrack_state bt, u32 reg) return bt->reg_masks[bt->frame] & (1 << reg); } - /* format registers bitmask, e.g., "r0,r2,r4" for 0x15 mask / static void fmt_reg_mask(char buf, ssize_t buf_sz, u32 reg_mask) { @@ -254,7 +253,6 @@ void bpf_fmt_stack_mask(char buf, ssize_t buf_sz, u64 stack_mask) } } - / For given verifier state backtrack_insn() is called from the last insn to * the first insn. Its purpose is to compute a bitmask of registers and * stack slots that needs precision in the parent verifier state. diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index 6606187ed4f43..87ffde865a503 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -2534,7 +2534,6 @@ static void btf_bitfield_show(void data, u8 bits_offset, btf_int128_print(show, print_num); } - static void btf_int_bits_show(const struct btf btf, const struct btf_type t, void data, u8 bits_offset, diff --git a/kernel/bpf/cfg.c b/kernel/bpf/cfg.c index db3416a7c9047..818f7afac83a5 100644 --- a/kernel/bpf/cfg.c +++ b/kernel/bpf/cfg.c @@ -47,7 +47,6 @@ enum { BRANCH = 2, }; - static void mark_subprog_changes_pkt_data(struct bpf_verifier_env env, int off) { struct bpf_subprog_info subprog; diff --git a/kernel/bpf/const_fold.c b/kernel/bpf/const_fold.c index 4cf120c7b2cb4..7f1b30059cc87 100644 --- a/kernel/bpf/const_fold.c +++ b/kernel/bpf/const_fold.c @@ -182,7 +182,6 @@ static void const_reg_xfer(struct bpf_verifier_env env, struct const_arg_info u64 val = 0; if (!bpf_map_is_rdonly(map) \|\| !map->ops->map_direct_value_addr \|\| - map->map_type == BPF_MAP_TYPE_INSN_ARRAY \|\| off < 0 \|\| off + size > map->value_size \|\| bpf_map_direct_read(map, off, size, &val, is_ldsx)) { dst = unknown; diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c index 2417a3461652d..177a3fcbb63ac 100644 --- a/kernel/bpf/fixups.c +++ b/kernel/bpf/fixups.c @@ -1466,7 +1466,6 @@ int bpf_fixup_call_args(struct bpf_verifier_env env) return err; } - /* The function requires that first instruction in 'patch' is insnsi[prog->len - 1] / static int add_hidden_subprog(struct bpf_verifier_env env, struct bpf_insn patch, int len) { @@ -1835,6 +1834,43 @@ int bpf_do_misc_fixups(struct bpf_verifier_env env) goto next_insn; } + if (bpf_jit_supports_percpu_insn() && + insn->code == (BPF_LD \| BPF_IMM \| BPF_DW) && + (insn->src_reg == BPF_PSEUDO_MAP_VALUE \|\| + insn->src_reg == BPF_PSEUDO_MAP_IDX_VALUE)) { + struct bpf_map map; + + aux = &env->insn_aux_data[i + delta]; + map = env->used_maps[aux->map_index]; + if (map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY) + goto next_insn; + + prog->jit_required = true; + + / + * We are skipping first half of ld_imm64 insn + * with 'i++;', patching over second half of it + * with that same half + mov64_percpu_reg insn. + * All because bpf_patch_insn_data() can only + * replace one 8-byte insn, which does not work + * well for ld_imm64 insn. + / + + insn_buf[0] = insn[1]; + insn_buf[1] = BPF_MOV64_PERCPU_REG(insn->dst_reg, insn->dst_reg); + cnt = 2; + + i++; + new_prog = bpf_patch_insn_data(env, i + delta, insn_buf, cnt); + if (!new_prog) + return -ENOMEM; + + delta += cnt - 1; + env->prog = prog = new_prog; + insn = new_prog->insnsi + i + delta; + goto next_insn; + } + if (insn->code != (BPF_JMP \| BPF_CALL)) goto next_insn; if (insn->src_reg == BPF_PSEUDO_CALL) diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c index 9f394e1aa2e85..d40cb5dd446ca 100644 --- a/kernel/bpf/hashtab.c +++ b/kernel/bpf/hashtab.c @@ -998,7 +998,6 @@ static void dec_elem_count(struct bpf_htab htab) atomic_dec(&htab->count); } - static void free_htab_elem(struct bpf_htab htab, struct htab_elem l) { htab_put_fd_value(htab, l); @@ -2970,7 +2969,6 @@ static int rhtab_delete_elem(struct bpf_rhtab rhtab, struct rhtab_elem elem, v return 0; } - static long rhtab_map_delete_elem(struct bpf_map map, void key) { struct bpf_rhtab rhtab = container_of(map, struct bpf_rhtab, map); diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index 6388b6b23e490..45e2f19387b24 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -4871,7 +4871,6 @@ static const struct btf_kfunc_id_set generic_kfunc_set = { .set = &generic_btf_ids, }; - BTF_ID_LIST(generic_dtor_ids) BTF_ID(struct, task_struct) BTF_ID(func, bpf_task_release_dtor) diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c index 1c997aeba6fa5..74fc4b3f80d6e 100644 --- a/kernel/bpf/liveness.c +++ b/kernel/bpf/liveness.c @@ -269,7 +269,6 @@ bpf_insn_successors(struct bpf_verifier_env env, u32 idx) __diag_pop(); - static inline bool update_insn(struct bpf_verifier_env env, struct func_instance instance, u32 frame, u32 insn_idx) { @@ -1862,7 +1861,6 @@ static int analyze_subprog(struct bpf_verifier_env env, if (need_resched()) cond_resched(); - / * When an instance is reused (must_write_initialized == true), * record into a fresh instance and merge afterward. This avoids diff --git a/kernel/bpf/queue_stack_maps.c b/kernel/bpf/queue_stack_maps.c index c1c9dee4dcdd0..6e8b18c32a106 100644 --- a/kernel/bpf/queue_stack_maps.c +++ b/kernel/bpf/queue_stack_maps.c @@ -123,7 +123,6 @@ static long __queue_map_get(struct bpf_map map, void value, bool delete) return err; } - static long __stack_map_get(struct bpf_map map, void value, bool delete) { struct bpf_queue_stack qs = bpf_queue_stack(map); diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 8d111da886553..7d8c3e8e6d62e 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -636,7 +636,6 @@ int bpf_map_alloc_pages(const struct bpf_map map, int nid, return ret; } - static int btf_field_cmp(const void a, const void b) { const struct btf_field f1 = a, f2 = b; @@ -1830,7 +1829,6 @@ static int map_lookup_elem(union bpf_attr attr) return err; } - #define BPF_MAP_UPDATE_ELEM_LAST_FIELD flags static int map_update_elem(union bpf_attr attr, bpfptr_t uattr) @@ -3497,7 +3495,6 @@ int bpf_link_prime(struct bpf_link link, struct bpf_link_primer primer) if (fd < 0) return fd; - id = bpf_link_alloc_id(link); if (id < 0) { put_unused_fd(fd); @@ -5505,7 +5502,6 @@ static int bpf_link_get_info_by_fd(struct file file, return 0; } - static int token_get_info_by_fd(struct file file, struct bpf_token token, const union bpf_attr attr, @@ -6507,7 +6503,6 @@ BPF_CALL_3(bpf_sys_bpf, int, cmd, union bpf_attr , attr, u32, attr_size) return __sys_bpf(cmd, KERNEL_BPFPTR(attr), attr_size, KERNEL_BPFPTR(NULL), 0); } - / To shut up -Wmissing-prototypes. * This function is used by the kernel light skeleton * to load bpf programs when modules are loaded or during kernel boot. diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 164d16c243ca6..6ac1afced20bf 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -635,7 +635,6 @@ static void __mark_dynptr_reg(struct bpf_reg_state reg, enum bpf_dynptr_type type, bool first_slot, int id, int parent_id); - static void mark_dynptr_stack_regs(struct bpf_verifier_env env, struct bpf_reg_state sreg1, struct bpf_reg_state sreg2, @@ -1674,7 +1673,6 @@ static bool same_callsites(struct bpf_verifier_state a, struct bpf_verifier_sta return true; } - void bpf_free_backedges(struct bpf_scc_visit visit) { struct bpf_scc_backedge backedge, next; @@ -2291,7 +2289,6 @@ static struct bpf_verifier_state push_async_cb(struct bpf_verifier_env env, return &elem->st; } - static int cmp_subprogs(const void a, const void b) { return ((struct bpf_subprog_info )a)->start - @@ -3969,7 +3966,6 @@ static int check_stack_read(struct bpf_verifier_env env, return err; } - /* check_stack_write dispatches to check_stack_write_fixed_off or * check_stack_write_var_off. * @@ -4767,7 +4763,6 @@ static int check_sock_access(struct bpf_verifier_env env, int insn_idx, valid = false; } - if (valid) { env->insn_aux_data[insn_idx].ctx_field_size = info.ctx_field_size; @@ -5587,6 +5582,8 @@ int bpf_map_direct_read(struct bpf_map map, int off, int size, u64 val, u64 addr; int err; + if (map->map_type == BPF_MAP_TYPE_INSN_ARRAY \|\| map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY) + return -EINVAL; err = map->ops->map_direct_value_addr(map, &addr, off); if (err) return err; @@ -6083,6 +6080,51 @@ static void add_scalar_to_reg(struct bpf_reg_state dst_reg, s64 val) reg_bounds_sync(dst_reg); } +static int check_map_mem_read(struct bpf_verifier_env env, struct bpf_reg_state reg, int off, + int bpf_size, int value_regno, bool is_ldsx) +{ + struct bpf_reg_state regs = cur_regs(env); + int size = bpf_size_to_bytes(bpf_size); + struct bpf_map map = reg->map_ptr; + + switch (map->map_type) { + case BPF_MAP_TYPE_INSN_ARRAY: + if (bpf_size != BPF_DW) { + verbose(env, "Invalid read of %d bytes from insn_array\n", size); + return -EACCES; + } + regs[value_regno] = reg; + add_scalar_to_reg(&regs[value_regno], off); + regs[value_regno].type = PTR_TO_INSN; + return 0; + case BPF_MAP_TYPE_PERCPU_ARRAY: + goto reg_unknown; + default: + break; + } + + / If map is read-only, track its contents as scalars. / + if (tnum_is_const(reg->var_off) && + bpf_map_is_rdonly(map) && + map->ops->map_direct_value_addr) { + int map_off = off + reg->var_off.value; + u64 val = 0; + int err; + + err = bpf_map_direct_read(map, map_off, size, &val, is_ldsx); + if (err) + return err; + + regs[value_regno].type = SCALAR_VALUE; + __mark_reg_known(&regs[value_regno], val); + return 0; + } + +reg_unknown: + mark_reg_unknown(env, regs, value_regno); + return 0; +} + / check whether memory at (regno + off) is accessible for t = (read \| write) * if t==write, value_regno is a register which value is stored into memory * if t==read, value_regno is a register which will receive the value from memory @@ -6137,38 +6179,7 @@ static int check_mem_access(struct bpf_verifier_env env, int insn_idx, struct b if (kptr_field) { err = check_map_kptr_access(env, value_regno, insn_idx, kptr_field); } else if (t == BPF_READ && value_regno >= 0) { - struct bpf_map map = reg->map_ptr; - - /* - * If map is read-only, track its contents as scalars, - * unless it is an insn array (see the special case below) - / - if (tnum_is_const(reg->var_off) && - bpf_map_is_rdonly(map) && - map->ops->map_direct_value_addr && - map->map_type != BPF_MAP_TYPE_INSN_ARRAY) { - int map_off = off + reg->var_off.value; - u64 val = 0; - - err = bpf_map_direct_read(map, map_off, size, - &val, is_ldsx); - if (err) - return err; - - regs[value_regno].type = SCALAR_VALUE; - __mark_reg_known(&regs[value_regno], val); - } else if (map->map_type == BPF_MAP_TYPE_INSN_ARRAY) { - if (bpf_size != BPF_DW) { - verbose(env, "Invalid read of %d bytes from insn_array\n", - size); - return -EACCES; - } - regs[value_regno] = reg; - add_scalar_to_reg(&regs[value_regno], off); - regs[value_regno].type = PTR_TO_INSN; - } else { - mark_reg_unknown(env, regs, value_regno); - } + err = check_map_mem_read(env, reg, off, bpf_size, value_regno, is_ldsx); } } else if (base_type(reg->type) == PTR_TO_MEM) { bool rdonly_mem = type_is_rdonly_mem(reg->type); @@ -6635,7 +6646,6 @@ static int check_stack_range_initialized( if (err) return err; - if (tnum_is_const(reg->var_off)) { min_off = max_off = reg->var_off.value + off; } else { @@ -7347,7 +7357,6 @@ static bool is_iter_new_kfunc(struct bpf_call_arg_meta meta) return meta->kfunc_flags & KF_ITER_NEW; } - static bool is_iter_destroy_kfunc(struct bpf_call_arg_meta meta) { return meta->kfunc_flags & KF_ITER_DESTROY; @@ -8125,6 +8134,12 @@ static int check_arg_const_str(struct bpf_verifier_env env, return -EACCES; } + if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY) { + verbose(env, "%s points to percpu_array map which cannot be used as const string\n", + reg_arg_name(env, argno)); + return -EACCES; + } + if (!bpf_map_is_rdonly(map)) { verbose(env, "%s does not point to a readonly map'\n", reg_arg_name(env, argno)); return -EACCES; @@ -11607,7 +11622,6 @@ static int process_irq_flag(struct bpf_verifier_env env, struct bpf_reg_state * return 0; } - static int ref_set_non_owning(struct bpf_verifier_env env, struct bpf_reg_state reg) { struct btf_record rec = reg_btf_record(reg); @@ -16412,7 +16426,6 @@ static int check_ld_abs(struct bpf_verifier_env env, struct bpf_insn insn) return 0; } - static bool return_retval_range(struct bpf_verifier_env env, struct bpf_retval_range range) { enum bpf_prog_type prog_type = resolve_prog_type(env->prog); @@ -18361,8 +18374,6 @@ static void release_insn_arrays(struct bpf_verifier_env env) bpf_insn_array_release(env->insn_array_maps[i]); } - - /* The verifier does more data flow analysis than llvm and will not * explore branches that are dead at run time. Malicious programs can * have dead code too. Therefore replace all dead at-run-time code @@ -18390,8 +18401,6 @@ static void sanitize_dead_code(struct bpf_verifier_env env) } } - - static void free_states(struct bpf_verifier_env env) { struct bpf_verifier_state_list sl; @@ -18678,7 +18687,6 @@ static int do_check_main(struct bpf_verifier_env env) return ret; } - static void print_verification_stats(struct bpf_verifier_env env) { / Skip over hidden subprogs which are not verified. / diff --git a/tools/bpf/bpftool/gen.c b/tools/bpf/bpftool/gen.c index a01d06d22d1a3..a50540ef6521c 100644 --- a/tools/bpf/bpftool/gen.c +++ b/tools/bpf/bpftool/gen.c @@ -101,6 +101,12 @@ static bool get_map_ident(const struct bpf_map map, char buf, size_t buf_sz) return true; } + if (bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY) { + snprintf(buf, buf_sz, "%s", name + 1); + sanitize_identifier(buf); + return true; + } + for (i = 0, n = ARRAY_SIZE(sfxs); i < n; i++) { const char sfx = sfxs[i], p; @@ -117,7 +123,7 @@ static bool get_map_ident(const struct bpf_map map, char buf, size_t buf_sz) static bool get_datasec_ident(const char sec_name, char buf, size_t buf_sz) { - static const char pfxs[] = { ".data", ".rodata", ".bss", ".kconfig" }; + static const char pfxs[] = { ".data", ".rodata", ".bss", ".percpu", ".kconfig" }; int i, n; / recognize hard coded LLVM section name / @@ -254,7 +260,7 @@ static const struct btf_type find_type_for_map(struct btf btf, const char map return NULL; } -static bool is_mmapable_map(const struct bpf_map map, char buf, size_t sz) +static bool is_skel_data(const struct bpf_map map, char buf, size_t sz) { size_t tmp_sz; @@ -263,13 +269,24 @@ static bool is_mmapable_map(const struct bpf_map map, char buf, size_t sz) return true; } - if (!bpf_map__is_internal(map) \|\| !(bpf_map__map_flags(map) & BPF_F_MMAPABLE)) + if (!bpf_map__is_internal(map)) return false; if (!get_map_ident(map, buf, sz)) return false; - return true; + if (bpf_map__map_flags(map) & BPF_F_MMAPABLE) + return true; + + if (bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY) + return bpf_map__btf_value_type_id(map) != 0; + + return false; +} + +static bool is_mmapable_map(const struct bpf_map map, char buf, size_t sz) +{ + return is_skel_data(map, buf, sz) && bpf_map__type(map) != BPF_MAP_TYPE_PERCPU_ARRAY; } static int codegen_datasecs(struct bpf_object obj, const char obj_name) @@ -287,7 +304,7 @@ static int codegen_datasecs(struct bpf_object obj, const char obj_name) bpf_object__for_each_map(map, obj) { /* only generate definitions for memory-mapped internal maps / - if (!is_mmapable_map(map, map_ident, sizeof(map_ident))) + if (!is_skel_data(map, map_ident, sizeof(map_ident))) continue; sec = find_type_for_map(btf, map_ident); @@ -517,7 +534,7 @@ static void codegen_asserts(struct bpf_object obj, const char obj_name) ", obj_name); bpf_object__for_each_map(map, obj) { - if (!is_mmapable_map(map, map_ident, sizeof(map_ident))) + if (!is_skel_data(map, map_ident, sizeof(map_ident))) continue; sec = find_type_for_map(btf, map_ident); @@ -668,8 +685,7 @@ static void codegen_destroy(struct bpf_object obj, const char obj_name) bpf_object__for_each_map(map, obj) { if (!get_map_ident(map, ident, sizeof(ident))) continue; - if (bpf_map__is_internal(map) && - (bpf_map__map_flags(map) & BPF_F_MMAPABLE)) + if (is_skel_data(map, ident, sizeof(ident))) printf("\tskel_free_map_data(skel->%1$s, skel->maps.%1$s.initial_value, %2$zu);\n", ident, bpf_map_mmap_sz(map)); codegen("\ @@ -741,7 +757,7 @@ static int gen_trace(struct bpf_object obj, const char obj_name, const char h const void mmap_data = NULL; size_t mmap_size = 0; - if (!is_mmapable_map(map, ident, sizeof(ident))) + if (!is_skel_data(map, ident, sizeof(ident))) continue; codegen("\ @@ -849,9 +865,23 @@ static int gen_trace(struct bpf_object obj, const char obj_name, const char h bpf_object__for_each_map(map, obj) { const char mmap_flags; - if (!is_mmapable_map(map, ident, sizeof(ident))) + if (!is_skel_data(map, ident, sizeof(ident))) continue; + if (bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY) { + codegen("\ + \n\ + err = skel_protect_map_data(skel->%1$s, &skel->maps.%1$s.initial_value, %2$zd);\n\ + if (err) \n\ + return err; \n\ + #ifdef __KERNEL__ \n\ + skel->%1$s = NULL; \n\ + #endif \n\ + ", + ident, bpf_map_mmap_sz(map)); + continue; + } + if (bpf_map__map_flags(map) & BPF_F_RDONLY_PROG) mmap_flags = "PROT_READ"; else @@ -955,8 +985,7 @@ codegen_maps_skeleton(struct bpf_object obj, size_t map_cnt, bool mmaped, bool map->map = &obj->maps.%s; \n\ ", i, bpf_map__name(map), ident); - /* memory-mapped internal maps / - if (mmaped && is_mmapable_map(map, ident, sizeof(ident))) { + if (mmaped && is_skel_data(map, ident, sizeof(ident))) { printf("\tmap->mmaped = (void )&obj->%s;\n", ident); } diff --git a/tools/lib/bpf/bpf_gen_internal.h b/tools/lib/bpf/bpf_gen_internal.h index 0425691877521..6c5ad6c55e8a6 100644 --- a/tools/lib/bpf/bpf_gen_internal.h +++ b/tools/lib/bpf/bpf_gen_internal.h @@ -65,7 +65,8 @@ void bpf_gen__prog_load(struct bpf_gen gen, enum bpf_prog_type prog_type, const char prog_name, const char license, struct bpf_insn insns, size_t insn_cnt, struct bpf_prog_load_opts load_attr, int prog_idx); -void bpf_gen__map_update_elem(struct bpf_gen gen, int map_idx, void value, __u32 value_size); +void bpf_gen__map_update_elem(struct bpf_gen gen, int map_idx, void value, __u32 value_size, + __u64 flags); void bpf_gen__map_freeze(struct bpf_gen gen, int map_idx); void bpf_gen__record_attach_target(struct bpf_gen gen, const char name, enum bpf_attach_type type); void bpf_gen__record_extern(struct bpf_gen gen, const char name, bool is_weak, diff --git a/tools/lib/bpf/features.c b/tools/lib/bpf/features.c index b7e388f99d0bb..ef9581c113035 100644 --- a/tools/lib/bpf/features.c +++ b/tools/lib/bpf/features.c @@ -620,6 +620,38 @@ static int probe_bpf_syscall_common_attrs(int token_fd) return probe_sys_bpf_ext(); } +static int probe_kern_percpu_data(int token_fd) +{ + struct bpf_insn insns[] = { + BPF_LD_MAP_VALUE(BPF_REG_1, 0, 0), + BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, 0), + BPF_EXIT_INSN(), + }; + LIBBPF_OPTS(bpf_map_create_opts, map_opts, + .token_fd = token_fd, + .map_flags = token_fd ? BPF_F_TOKEN_FD : 0, + ); + LIBBPF_OPTS(bpf_prog_load_opts, prog_opts, + .token_fd = token_fd, + .prog_flags = token_fd ? BPF_F_TOKEN_FD : 0, + ); + int ret, map, insn_cnt = ARRAY_SIZE(insns); + + map = bpf_map_create(BPF_MAP_TYPE_PERCPU_ARRAY, "libbpf_percpu", sizeof(int), 8, 1, + &map_opts); + if (map < 0) { + pr_warn("Error in %s(): %s. Couldn't create simple percpu_array map.\n", + __func__, errstr(map)); + return map; + } + + insns[0].imm = map; + + ret = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, NULL, "GPL", insns, insn_cnt, &prog_opts); + close(map); + return probe_fd(ret); +} + typedef int (feature_probe_fn)(int /* token_fd /); static struct kern_feature_cache feature_cache; @@ -707,6 +739,9 @@ static struct kern_feature_desc { [FEAT_BPF_SYSCALL_COMMON_ATTRS] = { "BPF syscall common attributes support", probe_bpf_syscall_common_attrs, }, + [FEAT_PERCPU_DATA] = { + "kernel supports percpu data", probe_kern_percpu_data, + }, }; bool feat_supported(struct kern_feature_cache cache, enum kern_feature_id feat_id) diff --git a/tools/lib/bpf/gen_loader.c b/tools/lib/bpf/gen_loader.c index 6e3dd52427618..af3a04f161ac1 100644 --- a/tools/lib/bpf/gen_loader.c +++ b/tools/lib/bpf/gen_loader.c @@ -1128,7 +1128,7 @@ void bpf_gen__prog_load(struct bpf_gen gen, } void bpf_gen__map_update_elem(struct bpf_gen gen, int map_idx, void pvalue, - __u32 value_size) + __u32 value_size, __u64 flags) { int attr_size = offsetofend(union bpf_attr, flags); int map_update_attr, value, key; @@ -1136,6 +1136,7 @@ void bpf_gen__map_update_elem(struct bpf_gen gen, int map_idx, void pvalue, int zero = 0; memset(&attr, 0, attr_size); + attr.flags = tgt_endian(flags); value = add_data(gen, pvalue, value_size); key = add_data(gen, &zero, sizeof(zero)); diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index 514e4e9daa825..e574870fb7169 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -541,6 +541,7 @@ struct bpf_struct_ops { }; #define DATA_SEC ".data" +#define PERCPU_SEC ".percpu" #define BSS_SEC ".bss" #define RODATA_SEC ".rodata" #define KCONFIG_SEC ".kconfig" @@ -555,6 +556,7 @@ enum libbpf_map_type { LIBBPF_MAP_BSS, LIBBPF_MAP_RODATA, LIBBPF_MAP_KCONFIG, + LIBBPF_MAP_PERCPU, }; struct bpf_map_def { @@ -666,6 +668,7 @@ enum sec_type { SEC_DATA, SEC_RODATA, SEC_ST_OPS, + SEC_PERCPU, }; struct elf_sec_desc { @@ -1839,6 +1842,8 @@ static size_t bpf_map_mmap_sz(const struct bpf_map map) switch (map->def.type) { case BPF_MAP_TYPE_ARRAY: return array_map_mmap_sz(map->def.value_size, map->def.max_entries); + case BPF_MAP_TYPE_PERCPU_ARRAY: + return map->def.value_size; case BPF_MAP_TYPE_ARENA: return page_sz * map->def.max_entries; default: @@ -1866,7 +1871,8 @@ static int bpf_map_mmap_resize(struct bpf_map map, size_t old_sz, size_t new_sz return 0; } -static char internal_map_name(struct bpf_object obj, const char real_name) +static char internal_map_name(struct bpf_object obj, const char real_name, + enum libbpf_map_type type) { char map_name[BPF_OBJ_NAME_LEN], p; int pfx_len, sfx_len = max((size_t)7, strlen(real_name)); @@ -1907,8 +1913,11 @@ static char internal_map_name(struct bpf_object obj, const char real_name) if (sfx_len >= BPF_OBJ_NAME_LEN) sfx_len = BPF_OBJ_NAME_LEN - 1; - / if there are two or more dots in map name, it's a custom dot map / - if (strchr(real_name + 1, '.') != NULL) + / + * Don't prefix the bpf_object name if this is a custom dot map + * (containing two or more dots) or a percpu data map. + / + if (strchr(real_name + 1, '.') != NULL \|\| type == LIBBPF_MAP_PERCPU) pfx_len = 0; else pfx_len = min((size_t)BPF_OBJ_NAME_LEN - sfx_len - 1, strlen(obj->name)); @@ -1941,6 +1950,13 @@ static bool map_is_mmapable(struct bpf_object obj, struct bpf_map map) if (!map->btf_value_type_id) return false; + / + * The internal PERCPU maps are not mmapble because the underlying + * percpu_array maps do not have mmap support. + / + if (map->libbpf_type == LIBBPF_MAP_PERCPU) + return false; + t = btf__type_by_id(obj->btf, map->btf_value_type_id); if (!btf_is_datasec(t)) return false; @@ -1962,6 +1978,7 @@ static int bpf_object__init_internal_map(struct bpf_object obj, enum libbpf_map_type type, const char real_name, int sec_idx, void data, size_t data_sz) { + bool is_percpu = type == LIBBPF_MAP_PERCPU; struct bpf_map_def def; struct bpf_map map; size_t mmap_sz; @@ -1975,7 +1992,7 @@ bpf_object__init_internal_map(struct bpf_object obj, enum libbpf_map_type type, map->sec_idx = sec_idx; map->sec_offset = 0; map->real_name = strdup(real_name); - map->name = internal_map_name(obj, real_name); + map->name = internal_map_name(obj, real_name, type); if (!map->real_name \|\| !map->name) { zfree(&map->real_name); zfree(&map->name); @@ -1983,7 +2000,7 @@ bpf_object__init_internal_map(struct bpf_object obj, enum libbpf_map_type type, } def = &map->def; - def->type = BPF_MAP_TYPE_ARRAY; + def->type = is_percpu ? BPF_MAP_TYPE_PERCPU_ARRAY : BPF_MAP_TYPE_ARRAY; def->key_size = sizeof(int); def->value_size = data_sz; def->max_entries = 1; @@ -1996,8 +2013,9 @@ bpf_object__init_internal_map(struct bpf_object obj, enum libbpf_map_type type, if (map_is_mmapable(obj, map)) def->map_flags \|= BPF_F_MMAPABLE; - pr_debug("map '%s' (global data): at sec_idx %d, offset %zu, flags %x.\n", - map->name, map->sec_idx, map->sec_offset, def->map_flags); + pr_debug("map '%s' (global %sdata): at sec_idx %d, offset %zu, flags %x.\n", + map->name, is_percpu ? "percpu " : "", map->sec_idx, + map->sec_offset, def->map_flags); mmap_sz = bpf_map_mmap_sz(map); map->mmaped = mmap(NULL, mmap_sz, PROT_READ \| PROT_WRITE, @@ -2057,6 +2075,13 @@ static int bpf_object__init_global_data_maps(struct bpf_object obj) NULL, sec_desc->data->d_size); break; + case SEC_PERCPU: + sec_name = elf_sec_name(obj, elf_sec_by_idx(obj, sec_idx)); + err = bpf_object__init_internal_map(obj, LIBBPF_MAP_PERCPU, + sec_name, sec_idx, + sec_desc->data->d_buf, + sec_desc->data->d_size); + break; default: /* skip / break; @@ -4016,6 +4041,11 @@ static int bpf_object__elf_collect(struct bpf_object obj) sec_desc->sec_type = SEC_RODATA; sec_desc->shdr = sh; sec_desc->data = data; + } else if (strcmp(name, PERCPU_SEC) == 0 \|\| + str_has_pfx(name, PERCPU_SEC ".")) { + sec_desc->sec_type = SEC_PERCPU; + sec_desc->shdr = sh; + sec_desc->data = data; } else if (strcmp(name, STRUCT_OPS_SEC) == 0 \|\| strcmp(name, STRUCT_OPS_LINK_SEC) == 0 \|\| strcmp(name, "?" STRUCT_OPS_SEC) == 0 \|\| @@ -4544,6 +4574,7 @@ static bool bpf_object__shndx_is_data(const struct bpf_object obj, case SEC_BSS: case SEC_DATA: case SEC_RODATA: + case SEC_PERCPU: return true; default: return false; @@ -4569,6 +4600,8 @@ bpf_object__section_to_libbpf_map_type(const struct bpf_object obj, int shndx) return LIBBPF_MAP_DATA; case SEC_RODATA: return LIBBPF_MAP_RODATA; + case SEC_PERCPU: + return LIBBPF_MAP_PERCPU; default: return LIBBPF_MAP_UNSPEC; } @@ -4944,7 +4977,7 @@ static int map_fill_btf_type_info(struct bpf_object obj, struct bpf_map map) /* * LLVM annotates global data differently in BTF, that is, - * only as '.data', '.bss' or '.rodata'. + * only as '.data', '.bss', '.percpu' or '.rodata'. / if (!bpf_map__is_internal(map)) return -ENOENT; @@ -5293,18 +5326,20 @@ static int bpf_object__populate_internal_map(struct bpf_object obj, struct bpf_map map) { enum libbpf_map_type map_type = map->libbpf_type; + bool is_percpu = map_type == LIBBPF_MAP_PERCPU; + const __u64 update_flags = is_percpu ? BPF_F_ALL_CPUS : 0; int err, zero = 0; size_t mmap_sz; if (obj->gen_loader) { bpf_gen__map_update_elem(obj->gen_loader, map - obj->maps, - map->mmaped, map->def.value_size); + map->mmaped, map->def.value_size, update_flags); if (map_type == LIBBPF_MAP_RODATA \|\| map_type == LIBBPF_MAP_KCONFIG) bpf_gen__map_freeze(obj->gen_loader, map - obj->maps); return 0; } - err = bpf_map_update_elem(map->fd, &zero, map->mmaped, 0); + err = bpf_map_update_elem(map->fd, &zero, map->mmaped, update_flags); if (err) { err = -errno; pr_warn("map '%s': failed to set initial contents: %s\n", @@ -5349,6 +5384,13 @@ bpf_object__populate_internal_map(struct bpf_object obj, struct bpf_map map) return err; } map->mmaped = mmaped; + } else if (is_percpu) { + if (mprotect(map->mmaped, mmap_sz, PROT_READ)) { + err = -errno; + pr_warn("map '%s': failed to mprotect() contents: %s\n", + bpf_map__name(map), errstr(err)); + return err; + } } else if (map->mmaped) { munmap(map->mmaped, mmap_sz); map->mmaped = NULL; @@ -5624,9 +5666,16 @@ bpf_object__create_maps(struct bpf_object obj) * runtime due to bpf_program__set_autoload(prog, false), * bpf_object loading will succeed just fine even on old * kernels. + * Same skipping applies to percpu data. / - if (bpf_map__is_internal(map) && !kernel_supports(obj, FEAT_GLOBAL_DATA)) - map->autocreate = false; + if (bpf_map__is_internal(map)) { + bool is_percpu = map->libbpf_type == LIBBPF_MAP_PERCPU; + enum kern_feature_id feat_id; + + feat_id = is_percpu ? FEAT_PERCPU_DATA : FEAT_GLOBAL_DATA; + if (!kernel_supports(obj, feat_id)) + map->autocreate = false; + } if (!map->autocreate) { pr_debug("map '%s': skipped auto-creating...\n", map->name); @@ -10807,11 +10856,16 @@ static bool map_uses_real_name(const struct bpf_map map) * such map's corresponding ELF section name as a map name. * This check distinguishes .data/.rodata from .data.* and .rodata.* * maps to know which name has to be returned to the user. + * Map name of the custom .percpu.* maps might be truncated to + * BPF_OBJ_NAME_LEN-1 chars in internal_map_name(). Hence, percpu data + * maps must use real name for their user-visible name. / if (map->libbpf_type == LIBBPF_MAP_DATA && strcmp(map->real_name, DATA_SEC) != 0) return true; if (map->libbpf_type == LIBBPF_MAP_RODATA && strcmp(map->real_name, RODATA_SEC) != 0) return true; + if (map->libbpf_type == LIBBPF_MAP_PERCPU) + return true; return false; } @@ -10976,7 +11030,8 @@ int bpf_map__set_value_size(struct bpf_map map, __u32 size) size_t mmap_old_sz, mmap_new_sz; int err; - if (map->def.type != BPF_MAP_TYPE_ARRAY) + if (map->def.type != BPF_MAP_TYPE_ARRAY && + map->def.type != BPF_MAP_TYPE_PERCPU_ARRAY) return libbpf_err(-EOPNOTSUPP); mmap_old_sz = bpf_map_mmap_sz(map); diff --git a/tools/lib/bpf/libbpf_internal.h b/tools/lib/bpf/libbpf_internal.h index 7a74abb904f8b..4c46d34fc055d 100644 --- a/tools/lib/bpf/libbpf_internal.h +++ b/tools/lib/bpf/libbpf_internal.h @@ -401,6 +401,8 @@ enum kern_feature_id { FEAT_BTF_LAYOUT, /* Kernel supports BPF syscall common attributes / FEAT_BPF_SYSCALL_COMMON_ATTRS, + / Kernel supports percpu data / + FEAT_PERCPU_DATA, __FEAT_CNT, }; diff --git a/tools/lib/bpf/skel_internal.h b/tools/lib/bpf/skel_internal.h index 53fee53d36d51..1f3f332dffbe9 100644 --- a/tools/lib/bpf/skel_internal.h +++ b/tools/lib/bpf/skel_internal.h @@ -131,8 +131,10 @@ static inline void skel_free_map_data(void p, __u64 addr, size_t sz) { if (addr != ~0ULL) kvfree(p); - /* When addr == ~0ULL the 'p' points to - * ((struct bpf_array )map)->value. See skel_finalize_map_data. + / + * When addr == ~0ULL the init buffer has already been released. + * For skel_finalize_map_data(), 'p' points to + * ((struct bpf_array )map)->value. / } @@ -170,6 +172,15 @@ static inline void skel_finalize_map_data(__u64 init_val, size_t mmap_sz, int return addr; } +static inline int skel_protect_map_data(void p, __u64 init_val, size_t sz) +{ + (void)sz; + + kvfree(p); + init_val = ~0ULL; + return 0; +} + #else static inline void skel_alloc(size_t size) @@ -208,6 +219,15 @@ static inline void skel_finalize_map_data(__u64 init_val, size_t mmap_sz, int return NULL; return addr; } + +static inline int skel_protect_map_data(void p, __u64 init_val, size_t sz) +{ + (void)init_val; + + if (mprotect(p, sz, PROT_READ)) + return -errno; + return 0; +} #endif static inline int skel_closenz(int fd) diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile index d3655a7064826..560ce4016fbff 100644 --- a/tools/testing/selftests/bpf/Makefile +++ b/tools/testing/selftests/bpf/Makefile @@ -531,7 +531,7 @@ LSKELS_SIGNED := fentry_test.c fexit_test.c atomics.c # Generate both light skeleton and libbpf skeleton for these LSKELS_EXTRA := test_ksyms_module.c test_ksyms_weak.c kfunc_call_test.c \ - kfunc_call_test_subprog.c + kfunc_call_test_subprog.c test_global_percpu_data.c SKEL_BLACKLIST += $$(LSKELS) $$(LSKELS_SIGNED) test_static_linked.skel.h-deps := test_static_linked1.bpf.o test_static_linked2.bpf.o diff --git a/tools/testing/selftests/bpf/prog_tests/global_data_init.c b/tools/testing/selftests/bpf/prog_tests/global_data_init.c index 8466332d7406f..7d6bda9092954 100644 --- a/tools/testing/selftests/bpf/prog_tests/global_data_init.c +++ b/tools/testing/selftests/bpf/prog_tests/global_data_init.c @@ -1,5 +1,8 @@ // SPDX-License-Identifier: GPL-2.0 #include <test_progs.h> +#include "bpf/libbpf_internal.h" +#include "test_global_percpu_data.skel.h" +#include "test_global_percpu_data.lskel.h" void test_global_data_init(void) { @@ -60,3 +63,336 @@ void test_global_data_init(void) free(newval); bpf_object__close(obj); } + +static void test_percpu_data_on_cpus(struct bpf_map map, int map_fd, int prog_fd, int runp) +{ + struct test_global_percpu_data__percpu data = NULL; + int i, err, key = 0, num_online, run = 0; + __u64 args[2] = {0x1234ULL, 0x5678ULL}; + size_t data_sz; + bool online; + LIBBPF_OPTS(bpf_test_run_opts, topts, + .ctx_in = args, + .ctx_size_in = sizeof(args), + .flags = BPF_F_TEST_RUN_ON_CPU, + ); + + err = parse_cpu_mask_file("/sys/devices/system/cpu/online", &online, &num_online); + if (!ASSERT_OK(err, "parse_cpu_mask_file")) + return; + + data_sz = map ? bpf_map__value_size(map) : sizeof(data); + data = calloc(1, data_sz); + if (!ASSERT_OK_PTR(data, "calloc percpu data")) + goto out; + + / run on every online-CPU / + for (i = 0; i < num_online; i++) { + __u64 flags; + + if (!online[i]) + continue; + + topts.cpu = i; + topts.retval = -1; + err = bpf_prog_test_run_opts(prog_fd, &topts); + ASSERT_OK(err, "bpf_prog_test_run_opts"); + ASSERT_EQ(topts.retval, 0, "bpf_prog_test_run_opts retval"); + + memset(data, 0, data_sz); + flags = ((__u64) i << 32) \| BPF_F_CPU; + if (map) + err = bpf_map__lookup_elem(map, &key, sizeof(key), data, data_sz, flags); + else + err = bpf_map_lookup_elem_flags(map_fd, &key, data, flags); + if (!ASSERT_OK(err, "lookup_elem on cpu")) + break; + + ASSERT_EQ(runp, ++run, "run"); + ASSERT_EQ(data->cpu_id[0], i, "cpu_id"); + ASSERT_EQ(data->data, 1, "data"); + ASSERT_TRUE(data->set, "set"); + ASSERT_EQ(data->nums[6], 0xc0de, "nums[6]"); + ASSERT_EQ(data->struct_data.i, 1, "struct_data.i"); + ASSERT_TRUE(data->struct_data.set, "struct_data.set"); + ASSERT_EQ(data->struct_data.nums[6], 0xc0de, "struct_data.nums[6]"); + } + +out: + free(data); + free(online); +} + +static void test_global_percpu_data_init(void) +{ + struct test_global_percpu_data__percpu init_value = {}; + struct test_global_percpu_data__percpu init_data; + const __u32 desired_sz = sysconf(_SC_PAGE_SIZE); + struct test_global_percpu_data skel = NULL; + size_t init_data_sz; + struct bpf_map map; + int prog_fd, err; + + skel = test_global_percpu_data__open(); + if (!ASSERT_OK_PTR(skel, "test_global_percpu_data__open")) + goto out; + if (!ASSERT_OK_PTR(skel->percpu, "skel->percpu")) + goto out; + if (!ASSERT_OK_PTR(skel->data_percpu, "skel->data_percpu")) + goto out; + if (!ASSERT_OK_PTR(skel->percpu_data, "skel->percpu_data")) + goto out; + if (!ASSERT_OK_PTR(skel->percpu_looooooooong, "skel->percpu_looooooooong")) + goto out; + + ASSERT_STREQ(bpf_map__name(skel->maps.percpu_data), ".percpu.data", + ".percpu.data map name"); + ASSERT_STREQ(bpf_map__name(skel->maps.data_percpu), ".data.percpu", + ".data.percpu map name"); + ASSERT_STREQ(bpf_map__name(skel->maps.percpu_looooooooong), ".percpu.looooooooong", + "long map name"); + ASSERT_STREQ(bpf_map__name(skel->maps.percpu), ".percpu", "map name"); + ASSERT_EQ(skel->percpu->data, -1, "skel->percpu->data"); + ASSERT_FALSE(skel->percpu->set, "skel->percpu->set"); + ASSERT_EQ(skel->percpu->nums[6], 0, "skel->percpu->nums[6]"); + ASSERT_EQ(skel->percpu->struct_data.i, -1, "struct_data.i"); + ASSERT_FALSE(skel->percpu->struct_data.set, "struct_data.set"); + ASSERT_EQ(skel->percpu->struct_data.nums[6], 0, "struct_data.nums[6]"); + + map = skel->maps.percpu; + if (!ASSERT_EQ(bpf_map__type(map), BPF_MAP_TYPE_PERCPU_ARRAY, "bpf_map__type")) + goto out; + + init_value.data = 2; + init_value.nums[6] = -1; + init_value.struct_data.i = 2; + init_value.struct_data.nums[6] = -1; + err = bpf_map__set_initial_value(map, &init_value, sizeof(init_value)); + if (!ASSERT_OK(err, "bpf_map__set_initial_value")) + goto out; + + init_data = bpf_map__initial_value(map, &init_data_sz); + if (!ASSERT_OK_PTR(init_data, "bpf_map__initial_value")) + goto out; + + ASSERT_EQ(init_data->data, init_value.data, "init_value data"); + ASSERT_EQ(init_data->set, init_value.set, "init_value set"); + ASSERT_EQ(init_data->struct_data.i, init_value.struct_data.i, "init_value struct_data.i"); + ASSERT_EQ(init_data->struct_data.nums[6], init_value.struct_data.nums[6], + "init_value struct_data.nums[6]"); + ASSERT_EQ(init_data_sz, sizeof(init_value), "init_value size"); + ASSERT_EQ((void ) init_data, (void ) skel->percpu, "skel->percpu eq init_data"); + ASSERT_EQ(skel->percpu->data, init_value.data, "skel->percpu->data"); + ASSERT_EQ(skel->percpu->set, init_value.set, "skel->percpu->set"); + ASSERT_EQ(skel->percpu->struct_data.i, init_value.struct_data.i, + "skel->percpu->struct_data.i"); + ASSERT_EQ(skel->percpu->struct_data.nums[6], init_value.struct_data.nums[6], + "skel->percpu->struct_data.nums[6]"); + + ASSERT_GT(desired_sz, sizeof(init_value), "desired_sz"); + err = bpf_map__set_value_size(map, desired_sz); + if (!ASSERT_OK(err, "bpf_map__set_value_size")) + goto out; + if (!ASSERT_EQ(bpf_map__value_size(map), desired_sz, "percpu value size")) + goto out; + if (!ASSERT_NEQ(bpf_map__btf_value_type_id(map), 0, "percpu BTF value type")) + goto out; + + init_data = bpf_map__initial_value(map, &init_data_sz); + if (!ASSERT_OK_PTR(init_data, "resized bpf_map__initial_value")) + goto out; + if (!ASSERT_EQ(init_data_sz, desired_sz, "resized initial value size")) + goto out; + if (!ASSERT_EQ(init_data->data, init_value.data, "resized initial value data")) + goto out; + + err = test_global_percpu_data__load(skel); + if (!ASSERT_OK(err, "test_global_percpu_data__load")) + goto out; + + ASSERT_OK_PTR(skel->percpu, "skel->percpu"); + + prog_fd = bpf_program__fd(skel->progs.update_percpu_data); + test_percpu_data_on_cpus(map, bpf_map__fd(map), prog_fd, &skel->bss->run); + +out: + test_global_percpu_data__destroy(skel); +} + +static void test_global_percpu_data_lskel(void) +{ + struct test_global_percpu_data_lskel lskel = NULL; + int prog_fd, map_fd; + + lskel = test_global_percpu_data_lskel__open_and_load(); + if (!ASSERT_OK_PTR(lskel, "test_global_percpu_data_lskel__open_and_load")) + goto out; + + map_fd = lskel->maps.percpu.map_fd; + prog_fd = lskel->progs.update_percpu_data.prog_fd; + test_percpu_data_on_cpus(NULL, map_fd, prog_fd, &lskel->bss->run); + +out: + test_global_percpu_data_lskel__destroy(lskel); +} + +static int create_rdonly_percpu_array(void) +{ + LIBBPF_OPTS(bpf_map_create_opts, map_opts, + .map_flags = BPF_F_RDONLY_PROG, + ); + int key = 0, map_fd, err; + __u64 value = 0; + + map_fd = bpf_map_create(BPF_MAP_TYPE_PERCPU_ARRAY, "percpu_ro_map", sizeof(int), + sizeof(__u64), 1, &map_opts); + if (!ASSERT_GE(map_fd, 0, "bpf_map_create")) + return -1; + + err = bpf_map_update_elem(map_fd, &key, &value, BPF_F_ALL_CPUS); + if (!ASSERT_OK(err, "bpf_map_update_elem")) + goto out; + + err = bpf_map_freeze(map_fd); + if (!ASSERT_OK(err, "bpf_map_freeze")) + goto out; + + return map_fd; + +out: + close(map_fd); + return -1; +} + +static void test_global_percpu_data_rdonly_direct_read(void) +{ + /* + * Raw instructions with manually prepared rdonly percpu_array map + * for testing direct-read global percpu data, because libbpf + * doesn't have rdonly internal percpu_array map support for + * global percpu data. + / + struct bpf_insn insns[] = { + BPF_LD_MAP_VALUE(BPF_REG_1, 0, 0), + BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, 0), + BPF_EXIT_INSN(), + }; + int map_fd, prog_fd; + + map_fd = create_rdonly_percpu_array(); + if (map_fd < 0) + return; + + insns[0].imm = map_fd; + prog_fd = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, "percpu_ro_prog", "GPL", insns, + ARRAY_SIZE(insns), NULL); + if (ASSERT_GE(prog_fd, 0, "bpf_prog_load")) + close(prog_fd); + close(map_fd); +} + +static void test_global_percpu_data_rdonly_direct_write(void) +{ + LIBBPF_OPTS(bpf_prog_load_opts, prog_opts); + / See the comment in test_global_percpu_data_rdonly_direct_read() / + struct bpf_insn insns[] = { + BPF_LD_MAP_VALUE(BPF_REG_1, 0, 0), + BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, 0), + BPF_ST_MEM(BPF_DW, BPF_REG_1, 0, 0), + BPF_EXIT_INSN(), + }; + char log_buf[256] = {}; + int map_fd, prog_fd; + + prog_opts.log_buf = log_buf; + prog_opts.log_size = sizeof(log_buf); + prog_opts.log_level = 1; + + map_fd = create_rdonly_percpu_array(); + if (map_fd < 0) + return; + + insns[0].imm = map_fd; + prog_fd = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, "percpu_ro_prog", "GPL", insns, + ARRAY_SIZE(insns), &prog_opts); + if (!ASSERT_LT(prog_fd, 0, "bpf_prog_load")) + close(prog_fd); + else + ASSERT_HAS_SUBSTR(log_buf, "write into map forbidden", "verifier log"); + close(map_fd); +} + +static void test_global_percpu_data_verifier_log(void) +{ + RUN_TESTS(test_global_percpu_data); +} + +static void test_global_percpu_data_iter(void) +{ + DECLARE_LIBBPF_OPTS(bpf_iter_attach_opts, opts); + struct test_global_percpu_data skel; + union bpf_iter_link_info linfo = {}; + struct bpf_link link = NULL; + int fd, num_cpus, len, err; + char buf[16]; + + num_cpus = libbpf_num_possible_cpus(); + if (!ASSERT_GT(num_cpus, 0, "libbpf_num_possible_cpus")) + return; + + skel = test_global_percpu_data__open(); + if (!ASSERT_OK_PTR(skel, "test_global_percpu_data__open")) + return; + + skel->rodata->num_cpus = num_cpus; + skel->rodata->offsetof_num = offsetof(struct test_global_percpu_data__percpu, struct_data); + skel->rodata->offsetof_num += sizeof(skel->percpu->struct_data) - sizeof(int); + skel->rodata->elem_sz = roundup(sizeof(struct test_global_percpu_data__percpu), 8); + skel->percpu->struct_data.nums[6] = 0xc0de; + + err = test_global_percpu_data__load(skel); + if (!ASSERT_OK(err, "test_global_percpu_data__load")) + goto out; + + linfo.map.map_fd = bpf_map__fd(skel->maps.percpu); + opts.link_info = &linfo; + opts.link_info_len = sizeof(linfo); + link = bpf_program__attach_iter(skel->progs.dump_percpu_data, &opts); + if (!ASSERT_OK_PTR(link, "bpf_program__attach_iter")) + goto out; + + fd = bpf_iter_create(bpf_link__fd(link)); + if (!ASSERT_GE(fd, 0, "bpf_iter_create")) + goto out; + + while ((len = read(fd, buf, sizeof(buf))) > 0) + do { } while (0); + ASSERT_EQ(len, 0, "read iter"); + ASSERT_TRUE(skel->bss->run_iter, "run_iter"); + ASSERT_EQ(skel->bss->percpu_data_sum, 0xc0de num_cpus, "percpu_data_sum"); + + close(fd); +out: + bpf_link__destroy(link); + test_global_percpu_data__destroy(skel); +} + +void test_global_percpu_data(void) +{ + if (!feat_supported(NULL, FEAT_PERCPU_DATA)) { + test__skip(); + return; + } + + if (test__start_subtest("init")) + test_global_percpu_data_init(); + if (test__start_subtest("lskel")) + test_global_percpu_data_lskel(); + if (test__start_subtest("rdonly_direct_read")) + test_global_percpu_data_rdonly_direct_read(); + if (test__start_subtest("rdonly_direct_write")) + test_global_percpu_data_rdonly_direct_write(); + test_global_percpu_data_verifier_log(); + if (test__start_subtest("iter")) + test_global_percpu_data_iter(); +} diff --git a/tools/testing/selftests/bpf/progs/test_global_percpu_data.c b/tools/testing/selftests/bpf/progs/test_global_percpu_data.c new file mode 100644 index 0000000000000..71ff8d1bf49eb --- /dev/null +++ b/tools/testing/selftests/bpf/progs/test_global_percpu_data.c @@ -0,0 +1,89 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <vmlinux.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +/* Used for testing map name. / +int loong SEC(".percpu.looooooooong"); +int data3 SEC(".data.percpu"); +int data2 SEC(".percpu.data"); + +int run; +/ cpu_id as array to verify map value resizing. / +int cpu_id[1] SEC(".percpu"); +int data SEC(".percpu") = -1; +int nums[7] SEC(".percpu"); +bool set SEC(".percpu") = false; +struct { + char set; + int i; + int nums[7]; +} struct_data SEC(".percpu") = { + .set = 0, + .i = -1, +}; + +SEC("raw_tp/task_rename") +__auxiliary +int update_percpu_data(void ctx) +{ + struct_data.nums[6] = 0xc0de; + struct_data.set = 1; + struct_data.i = 1; + nums[6] = 0xc0de; + data = 1; + run++; + set = true; + cpu_id[0] = bpf_get_smp_processor_id(); + return 0; +} + +static const char fmt[] SEC(".percpu.fmt") = "data %d\n"; + +SEC("?kprobe") +__failure __msg("R{{[0-9]+}} points to percpu_array map which cannot be used as const string") +int verifier_strncmp(void ctx) +{ + return bpf_strncmp("test", 5, fmt); +} + +SEC("?kprobe") +__failure __msg("R{{[0-9]+}} points to percpu_array map which cannot be used as const string") +int verifier_snprintf(void ctx) +{ + u64 args[] = { data }; + char buf[128]; + int len; + + len = bpf_snprintf(buf, sizeof(buf), fmt, args, sizeof(args)); + if (len > 0) + bpf_printk("snprintf: %s\n", buf); + return 0; +} + +volatile const __u32 num_cpus = 0; +volatile const int offsetof_num; +volatile const int elem_sz; +__u32 percpu_data_sum = 0; +bool run_iter = false; + +SEC("iter/bpf_map_elem") +__auxiliary +int dump_percpu_data(struct bpf_iter__bpf_map_elem ctx) +{ + void pptr = ctx->value; + int i; + + if (!pptr) + return 0; + + run_iter = true; + + for (i = 0; i < num_cpus; i++) { + percpu_data_sum += (int ) (pptr + offsetof_num); + pptr += elem_sz; + } + return 0; +} + +char _license[] SEC("license") = "GPL";
19/3	2026/08/13 16:52	llm	kmsan-evaluator	0m Model: gemini-3.1-pro-preview Args: null Results: null Tokens: input: 23627 output: 229 thoughts: 1769 Thoughts: KMSAN Assessment: Global Per-CPU Data in BPF Okay, so I've been digging into this patch series that introduces support for global percpu data within BPF. It's essentially adding the ability to define `.percpu` sections in BPF programs, which are then mapped to `BPF_MAP_TYPE_PERCPU_ARRAY` maps. My focus here is ensuring this doesn't create any new avenues for uninitialized memory issues, leaks, or other subtle vulnerabilities that could slip through. The patch series touches several areas. I'm seeing modifications in `arraymap.c` for handling these percpu array map operations, changes in the verifier (`bpf/verifier.c`) to correctly process `BPF_MAP_TYPE_PERCPU_ARRAY` during memory reads, and updates in `bpftool/gen.c` and `libbpf` to support the `.percpu` section and the necessary map types. Tests have also been added, of course. The core idea is that libbpf creates a `BPF_MAP_TYPE_PERCPU_ARRAY` map for each `.percpu` section. The contents of this section are then used to initialize the map. The verifier marks the registers as "unknown" when encountering percpu array, meaning it's not directly readable. However, I've noticed a modification to `bpf_map_direct_read`, which explicitly disables direct reads for percpu arrays. So, that's good; it eliminates a potential risk. I was curious about `percpu_array_map_direct_value_addr`. It's used by the verifier during JIT compilation. This is just for BPF JIT instruction patching. The memory itself is initialized using `bpf_map_update_elem` with `BPF_F_ALL_CPUS`. The memory is handled by either `bpf_map_alloc_pages` or the standard percpu allocator, neither of which is new here. Thinking through potential uninitialized memory risks, I'm not seeing any. There are no new kernel structures exposed to userspace. Initialization happens either through zeroing during allocation or by data provided via `bpf_map_update_elem`. The verifier changes largely just adapt to the existing `BPF_MAP_TYPE_PERCPU_ARRAY` used as global data. Since user space is providing the initial data from the ELF, it's their responsibility to provide the right data, and the kernel side zeros-out memory. Let me break down my KMSAN concerns point-by-point: * Copying Uninitialized Data to Userspace: Nope, doesn't happen. * Using Uninitialized Data in Branching: No, that's not occurring. * Exposing Uninitialized Memory: Again, no. Therefore, this doesn't warrant a dedicated KMSAN fuzzing campaign. KASAN should be sufficient. I believe that KASAN will catch any out-of-bounds, use-after-free, or other memory corruption issues arising from JIT patching or verifier errors. So, in summary, this patch series enhances BPF by allowing global percpu data using `.percpu` sections. The memory initialization process is safe, leveraging existing map allocation routines and libbpf's interaction with ELF data. There is no new uninitialized buffers, or structures. The core logic handles JIT compilation correctly to ensure there aren't any potential information leaks. While introducing new code always brings risk, any bugs are most likely to be detectable by standard memory safety tools like KASAN, and this should not require a dedicated KMSAN focus.
20/3	2026/08/13 16:52	tool	set-results	0m Args: { "KMSANReasoning": "The patch series adds support for global percpu data in BPF by allowing `.percpu` sections to be mapped to `BPF_MAP_TYPE_PERCPU_ARRAY` maps. The changes involve libbpf parsing the new section, bpftool generating skeleton code, and the BPF verifier handling direct value access and JIT patching for percpu arrays.\n\nThere are no new kernel structures sent to user space, no new uninitialized buffers allocated, and no complex data structures where padding might be leaked. The percpu array memory is allocated using existing map allocation routines (which handle initialization) and populated by libbpf from the ELF section.\n\nAny potential bugs introduced by these changes would likely be out-of-bounds accesses, invalid JIT patching, or verifier logic errors, which are well-covered by KASAN and standard bug detectors. There is no specific risk of uninitialized memory usage or info-leaks that would require a dedicated KMSAN fuzzing session.", "NeedsKMSAN": false } Results: { "KMSANReasoning": "The patch series adds support for global percpu data in BPF by allowing `.percpu` sections to be mapped to `BPF_MAP_TYPE_PERCPU_ARRAY` maps. The changes involve libbpf parsing the new section, bpftool generating skeleton code, and the BPF verifier handling direct value access and JIT patching for percpu arrays.\n\nThere are no new kernel structures sent to user space, no new uninitialized buffers allocated, and no complex data structures where padding might be leaked. The percpu array memory is allocated using existing map allocation routines (which handle initialization) and populated by libbpf from the ELF section.\n\nAny potential bugs introduced by these changes would likely be out-of-bounds accesses, invalid JIT patching, or verifier logic errors, which are well-covered by KASAN and standard bug detectors. There is no specific risk of uninitialized memory usage or info-leaks that would require a dedicated KMSAN fuzzing session.", "NeedsKMSAN": false }

{
  "EnableConfigs": [
    "BPF_SYSCALL"
  ],
  "FocusSymbols": [
    "bpf_do_misc_fixups",
    "bpf_map_direct_read",
    "percpu_array_map_direct_value_addr",
    "percpu_array_map_direct_value_meta"
  ],
  "KMSANReasoning": "The patch series adds support for global percpu data in BPF by allowing `.percpu` sections to be mapped to `BPF_MAP_TYPE_PERCPU_ARRAY` maps. The changes involve libbpf parsing the new section, bpftool generating skeleton code, and the BPF verifier handling direct value access and JIT patching for percpu arrays.\n\nThere are no new kernel structures sent to user space, no new uninitialized buffers allocated, and no complex data structures where padding might be leaked. The percpu array memory is allocated using existing map allocation routines (which handle initialization) and populated by libbpf from the ELF section.\n\nAny potential bugs introduced by these changes would likely be out-of-bounds accesses, invalid JIT patching, or verifier logic errors, which are well-covered by KASAN and standard bug detectors. There is no specific risk of uninitialized memory usage or info-leaks that would require a dedicated KMSAN fuzzing session.",
  "NeedsKMSAN": false,
  "Reasoning": "The patch introduces support for global percpu data in BPF programs by allowing `.percpu` sections to be backed by `BPF_MAP_TYPE_PERCPU_ARRAY`. It modifies the BPF verifier to handle direct memory reads from percpu arrays and updates JIT fixups to support direct loading of percpu variables. These changes affect core BPF verifier logic and map operations, which are reachable from user space via the `bpf()` syscall when loading and executing BPF programs.",
  "WorthFuzzing": true
}

{
  "PatchDiff": "commit 3b76f69b75942f2595d46bc813574f1caa6e529f\nAuthor: syz-cluster \u003ctriage@syzkaller.com\u003e\nDate:   Thu Aug 13 16:51:17 2026 +0000\n\n    syz-cluster: applied patch under review\n\ndiff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c\nindex 248b4818178cd..34865701f7f75 100644\n--- a/kernel/bpf/arraymap.c\n+++ b/kernel/bpf/arraymap.c\n@@ -259,6 +259,37 @@ static void *percpu_array_map_lookup_elem(struct bpf_map *map, void *key)\n \treturn this_cpu_ptr(array-\u003epptrs[index \u0026 array-\u003eindex_mask]);\n }\n \n+static int percpu_array_map_direct_value_addr(const struct bpf_map *map, u64 *imm, u32 off)\n+{\n+\tstruct bpf_array *array = container_of(map, struct bpf_array, map);\n+\n+\tif (!bpf_jit_supports_percpu_insn())\n+\t\treturn -EOPNOTSUPP;\n+\tif (map-\u003emax_entries != 1)\n+\t\treturn -EOPNOTSUPP;\n+\tif (off \u003e= map-\u003evalue_size)\n+\t\treturn -EINVAL;\n+\n+\t*imm = (u64)(__force unsigned long) array-\u003epptrs[0];\n+\treturn 0;\n+}\n+\n+static int percpu_array_map_direct_value_meta(const struct bpf_map *map, u64 imm, u32 *off)\n+{\n+\tstruct bpf_array *array = container_of(map, struct bpf_array, map);\n+\tu64 base = (u64)(__force unsigned long) array-\u003epptrs[0];\n+\n+\tif (!bpf_jit_supports_percpu_insn())\n+\t\treturn -EOPNOTSUPP;\n+\tif (map-\u003emax_entries != 1)\n+\t\treturn -EOPNOTSUPP;\n+\tif (imm \u003c base || imm \u003e= base + array-\u003eelem_size)\n+\t\treturn -ENOENT;\n+\n+\t*off = imm - base;\n+\treturn 0;\n+}\n+\n /* emit BPF instructions equivalent to C code of percpu_array_map_lookup_elem() */\n static int percpu_array_map_gen_lookup(struct bpf_map *map, struct bpf_insn *insn_buf)\n {\n@@ -551,9 +582,10 @@ static int array_map_check_btf(struct bpf_map *map,\n \t\t\t       const struct btf_type *key_type,\n \t\t\t       const struct btf_type *value_type)\n {\n-\t/* One exception for keyless BTF: .bss/.data/.rodata map */\n+\t/* One exception for keyless BTF: .bss/.data/.rodata/.percpu map */\n \tif (btf_type_is_void(key_type)) {\n-\t\tif (map-\u003emap_type != BPF_MAP_TYPE_ARRAY ||\n+\t\tif ((map-\u003emap_type != BPF_MAP_TYPE_ARRAY \u0026\u0026\n+\t\t     map-\u003emap_type != BPF_MAP_TYPE_PERCPU_ARRAY) ||\n \t\t    map-\u003emax_entries != 1)\n \t\t\treturn -EINVAL;\n \n@@ -832,6 +864,8 @@ const struct bpf_map_ops percpu_array_map_ops = {\n \t.map_get_next_key = bpf_array_get_next_key,\n \t.map_lookup_elem = percpu_array_map_lookup_elem,\n \t.map_gen_lookup = percpu_array_map_gen_lookup,\n+\t.map_direct_value_addr = percpu_array_map_direct_value_addr,\n+\t.map_direct_value_meta = percpu_array_map_direct_value_meta,\n \t.map_update_elem = array_map_update_elem,\n \t.map_delete_elem = array_map_delete_elem,\n \t.map_lookup_percpu_elem = percpu_array_map_lookup_percpu_elem,\ndiff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c\nindex 40bd04421a991..a2b18a9f1694c 100644\n--- a/kernel/bpf/backtrack.c\n+++ b/kernel/bpf/backtrack.c\n@@ -214,7 +214,6 @@ static inline bool bt_is_reg_set(struct backtrack_state *bt, u32 reg)\n \treturn bt-\u003ereg_masks[bt-\u003eframe] \u0026 (1 \u003c\u003c reg);\n }\n \n-\n /* format registers bitmask, e.g., \"r0,r2,r4\" for 0x15 mask */\n static void fmt_reg_mask(char *buf, ssize_t buf_sz, u32 reg_mask)\n {\n@@ -254,7 +253,6 @@ void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask)\n \t}\n }\n \n-\n /* For given verifier state backtrack_insn() is called from the last insn to\n  * the first insn. Its purpose is to compute a bitmask of registers and\n  * stack slots that needs precision in the parent verifier state.\ndiff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c\nindex 6606187ed4f43..87ffde865a503 100644\n--- a/kernel/bpf/btf.c\n+++ b/kernel/bpf/btf.c\n@@ -2534,7 +2534,6 @@ static void btf_bitfield_show(void *data, u8 bits_offset,\n \tbtf_int128_print(show, print_num);\n }\n \n-\n static void btf_int_bits_show(const struct btf *btf,\n \t\t\t      const struct btf_type *t,\n \t\t\t      void *data, u8 bits_offset,\ndiff --git a/kernel/bpf/cfg.c b/kernel/bpf/cfg.c\nindex db3416a7c9047..818f7afac83a5 100644\n--- a/kernel/bpf/cfg.c\n+++ b/kernel/bpf/cfg.c\n@@ -47,7 +47,6 @@ enum {\n \tBRANCH = 2,\n };\n \n-\n static void mark_subprog_changes_pkt_data(struct bpf_verifier_env *env, int off)\n {\n \tstruct bpf_subprog_info *subprog;\ndiff --git a/kernel/bpf/const_fold.c b/kernel/bpf/const_fold.c\nindex 4cf120c7b2cb4..7f1b30059cc87 100644\n--- a/kernel/bpf/const_fold.c\n+++ b/kernel/bpf/const_fold.c\n@@ -182,7 +182,6 @@ static void const_reg_xfer(struct bpf_verifier_env *env, struct const_arg_info *\n \t\tu64 val = 0;\n \n \t\tif (!bpf_map_is_rdonly(map) || !map-\u003eops-\u003emap_direct_value_addr ||\n-\t\t    map-\u003emap_type == BPF_MAP_TYPE_INSN_ARRAY ||\n \t\t    off \u003c 0 || off + size \u003e map-\u003evalue_size ||\n \t\t    bpf_map_direct_read(map, off, size, \u0026val, is_ldsx)) {\n \t\t\t*dst = unknown;\ndiff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c\nindex 2417a3461652d..177a3fcbb63ac 100644\n--- a/kernel/bpf/fixups.c\n+++ b/kernel/bpf/fixups.c\n@@ -1466,7 +1466,6 @@ int bpf_fixup_call_args(struct bpf_verifier_env *env)\n \treturn err;\n }\n \n-\n /* The function requires that first instruction in 'patch' is insnsi[prog-\u003elen - 1] */\n static int add_hidden_subprog(struct bpf_verifier_env *env, struct bpf_insn *patch, int len)\n {\n@@ -1835,6 +1834,43 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)\n \t\t\tgoto next_insn;\n \t\t}\n \n+\t\tif (bpf_jit_supports_percpu_insn() \u0026\u0026\n+\t\t    insn-\u003ecode == (BPF_LD | BPF_IMM | BPF_DW) \u0026\u0026\n+\t\t    (insn-\u003esrc_reg == BPF_PSEUDO_MAP_VALUE ||\n+\t\t     insn-\u003esrc_reg == BPF_PSEUDO_MAP_IDX_VALUE)) {\n+\t\t\tstruct bpf_map *map;\n+\n+\t\t\taux = \u0026env-\u003einsn_aux_data[i + delta];\n+\t\t\tmap = env-\u003eused_maps[aux-\u003emap_index];\n+\t\t\tif (map-\u003emap_type != BPF_MAP_TYPE_PERCPU_ARRAY)\n+\t\t\t\tgoto next_insn;\n+\n+\t\t\tprog-\u003ejit_required = true;\n+\n+\t\t\t/*\n+\t\t\t * We are *skipping* first half of ld_imm64 insn\n+\t\t\t * with 'i++;', patching over second half of it\n+\t\t\t * with that same half + mov64_percpu_reg insn.\n+\t\t\t * All because bpf_patch_insn_data() can only\n+\t\t\t * replace one 8-byte insn, which does not work\n+\t\t\t * well for ld_imm64 insn.\n+\t\t\t */\n+\n+\t\t\tinsn_buf[0] = insn[1];\n+\t\t\tinsn_buf[1] = BPF_MOV64_PERCPU_REG(insn-\u003edst_reg, insn-\u003edst_reg);\n+\t\t\tcnt = 2;\n+\n+\t\t\ti++;\n+\t\t\tnew_prog = bpf_patch_insn_data(env, i + delta, insn_buf, cnt);\n+\t\t\tif (!new_prog)\n+\t\t\t\treturn -ENOMEM;\n+\n+\t\t\tdelta    += cnt - 1;\n+\t\t\tenv-\u003eprog = prog = new_prog;\n+\t\t\tinsn      = new_prog-\u003einsnsi + i + delta;\n+\t\t\tgoto next_insn;\n+\t\t}\n+\n \t\tif (insn-\u003ecode != (BPF_JMP | BPF_CALL))\n \t\t\tgoto next_insn;\n \t\tif (insn-\u003esrc_reg == BPF_PSEUDO_CALL)\ndiff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c\nindex 9f394e1aa2e85..d40cb5dd446ca 100644\n--- a/kernel/bpf/hashtab.c\n+++ b/kernel/bpf/hashtab.c\n@@ -998,7 +998,6 @@ static void dec_elem_count(struct bpf_htab *htab)\n \t\tatomic_dec(\u0026htab-\u003ecount);\n }\n \n-\n static void free_htab_elem(struct bpf_htab *htab, struct htab_elem *l)\n {\n \thtab_put_fd_value(htab, l);\n@@ -2970,7 +2969,6 @@ static int rhtab_delete_elem(struct bpf_rhtab *rhtab, struct rhtab_elem *elem, v\n \treturn 0;\n }\n \n-\n static long rhtab_map_delete_elem(struct bpf_map *map, void *key)\n {\n \tstruct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map);\ndiff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c\nindex 6388b6b23e490..45e2f19387b24 100644\n--- a/kernel/bpf/helpers.c\n+++ b/kernel/bpf/helpers.c\n@@ -4871,7 +4871,6 @@ static const struct btf_kfunc_id_set generic_kfunc_set = {\n \t.set   = \u0026generic_btf_ids,\n };\n \n-\n BTF_ID_LIST(generic_dtor_ids)\n BTF_ID(struct, task_struct)\n BTF_ID(func, bpf_task_release_dtor)\ndiff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c\nindex 1c997aeba6fa5..74fc4b3f80d6e 100644\n--- a/kernel/bpf/liveness.c\n+++ b/kernel/bpf/liveness.c\n@@ -269,7 +269,6 @@ bpf_insn_successors(struct bpf_verifier_env *env, u32 idx)\n \n __diag_pop();\n \n-\n static inline bool update_insn(struct bpf_verifier_env *env,\n \t\t\t       struct func_instance *instance, u32 frame, u32 insn_idx)\n {\n@@ -1862,7 +1861,6 @@ static int analyze_subprog(struct bpf_verifier_env *env,\n \tif (need_resched())\n \t\tcond_resched();\n \n-\n \t/*\n \t * When an instance is reused (must_write_initialized == true),\n \t * record into a fresh instance and merge afterward.  This avoids\ndiff --git a/kernel/bpf/queue_stack_maps.c b/kernel/bpf/queue_stack_maps.c\nindex c1c9dee4dcdd0..6e8b18c32a106 100644\n--- a/kernel/bpf/queue_stack_maps.c\n+++ b/kernel/bpf/queue_stack_maps.c\n@@ -123,7 +123,6 @@ static long __queue_map_get(struct bpf_map *map, void *value, bool delete)\n \treturn err;\n }\n \n-\n static long __stack_map_get(struct bpf_map *map, void *value, bool delete)\n {\n \tstruct bpf_queue_stack *qs = bpf_queue_stack(map);\ndiff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c\nindex 8d111da886553..7d8c3e8e6d62e 100644\n--- a/kernel/bpf/syscall.c\n+++ b/kernel/bpf/syscall.c\n@@ -636,7 +636,6 @@ int bpf_map_alloc_pages(const struct bpf_map *map, int nid,\n \treturn ret;\n }\n \n-\n static int btf_field_cmp(const void *a, const void *b)\n {\n \tconst struct btf_field *f1 = a, *f2 = b;\n@@ -1830,7 +1829,6 @@ static int map_lookup_elem(union bpf_attr *attr)\n \treturn err;\n }\n \n-\n #define BPF_MAP_UPDATE_ELEM_LAST_FIELD flags\n \n static int map_update_elem(union bpf_attr *attr, bpfptr_t uattr)\n@@ -3497,7 +3495,6 @@ int bpf_link_prime(struct bpf_link *link, struct bpf_link_primer *primer)\n \tif (fd \u003c 0)\n \t\treturn fd;\n \n-\n \tid = bpf_link_alloc_id(link);\n \tif (id \u003c 0) {\n \t\tput_unused_fd(fd);\n@@ -5505,7 +5502,6 @@ static int bpf_link_get_info_by_fd(struct file *file,\n \treturn 0;\n }\n \n-\n static int token_get_info_by_fd(struct file *file,\n \t\t\t\tstruct bpf_token *token,\n \t\t\t\tconst union bpf_attr *attr,\n@@ -6507,7 +6503,6 @@ BPF_CALL_3(bpf_sys_bpf, int, cmd, union bpf_attr *, attr, u32, attr_size)\n \treturn __sys_bpf(cmd, KERNEL_BPFPTR(attr), attr_size, KERNEL_BPFPTR(NULL), 0);\n }\n \n-\n /* To shut up -Wmissing-prototypes.\n  * This function is used by the kernel light skeleton\n  * to load bpf programs when modules are loaded or during kernel boot.\ndiff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c\nindex 164d16c243ca6..6ac1afced20bf 100644\n--- a/kernel/bpf/verifier.c\n+++ b/kernel/bpf/verifier.c\n@@ -635,7 +635,6 @@ static void __mark_dynptr_reg(struct bpf_reg_state *reg,\n \t\t\t      enum bpf_dynptr_type type,\n \t\t\t      bool first_slot, int id, int parent_id);\n \n-\n static void mark_dynptr_stack_regs(struct bpf_verifier_env *env,\n \t\t\t\t   struct bpf_reg_state *sreg1,\n \t\t\t\t   struct bpf_reg_state *sreg2,\n@@ -1674,7 +1673,6 @@ static bool same_callsites(struct bpf_verifier_state *a, struct bpf_verifier_sta\n \treturn true;\n }\n \n-\n void bpf_free_backedges(struct bpf_scc_visit *visit)\n {\n \tstruct bpf_scc_backedge *backedge, *next;\n@@ -2291,7 +2289,6 @@ static struct bpf_verifier_state *push_async_cb(struct bpf_verifier_env *env,\n \treturn \u0026elem-\u003est;\n }\n \n-\n static int cmp_subprogs(const void *a, const void *b)\n {\n \treturn ((struct bpf_subprog_info *)a)-\u003estart -\n@@ -3969,7 +3966,6 @@ static int check_stack_read(struct bpf_verifier_env *env,\n \treturn err;\n }\n \n-\n /* check_stack_write dispatches to check_stack_write_fixed_off or\n  * check_stack_write_var_off.\n  *\n@@ -4767,7 +4763,6 @@ static int check_sock_access(struct bpf_verifier_env *env, int insn_idx,\n \t\tvalid = false;\n \t}\n \n-\n \tif (valid) {\n \t\tenv-\u003einsn_aux_data[insn_idx].ctx_field_size =\n \t\t\tinfo.ctx_field_size;\n@@ -5587,6 +5582,8 @@ int bpf_map_direct_read(struct bpf_map *map, int off, int size, u64 *val,\n \tu64 addr;\n \tint err;\n \n+\tif (map-\u003emap_type == BPF_MAP_TYPE_INSN_ARRAY || map-\u003emap_type == BPF_MAP_TYPE_PERCPU_ARRAY)\n+\t\treturn -EINVAL;\n \terr = map-\u003eops-\u003emap_direct_value_addr(map, \u0026addr, off);\n \tif (err)\n \t\treturn err;\n@@ -6083,6 +6080,51 @@ static void add_scalar_to_reg(struct bpf_reg_state *dst_reg, s64 val)\n \treg_bounds_sync(dst_reg);\n }\n \n+static int check_map_mem_read(struct bpf_verifier_env *env, struct bpf_reg_state *reg, int off,\n+\t\t\t      int bpf_size, int value_regno, bool is_ldsx)\n+{\n+\tstruct bpf_reg_state *regs = cur_regs(env);\n+\tint size = bpf_size_to_bytes(bpf_size);\n+\tstruct bpf_map *map = reg-\u003emap_ptr;\n+\n+\tswitch (map-\u003emap_type) {\n+\tcase BPF_MAP_TYPE_INSN_ARRAY:\n+\t\tif (bpf_size != BPF_DW) {\n+\t\t\tverbose(env, \"Invalid read of %d bytes from insn_array\\n\", size);\n+\t\t\treturn -EACCES;\n+\t\t}\n+\t\tregs[value_regno] = *reg;\n+\t\tadd_scalar_to_reg(\u0026regs[value_regno], off);\n+\t\tregs[value_regno].type = PTR_TO_INSN;\n+\t\treturn 0;\n+\tcase BPF_MAP_TYPE_PERCPU_ARRAY:\n+\t\tgoto reg_unknown;\n+\tdefault:\n+\t\tbreak;\n+\t}\n+\n+\t/* If map is read-only, track its contents as scalars. */\n+\tif (tnum_is_const(reg-\u003evar_off) \u0026\u0026\n+\t    bpf_map_is_rdonly(map) \u0026\u0026\n+\t    map-\u003eops-\u003emap_direct_value_addr) {\n+\t\tint map_off = off + reg-\u003evar_off.value;\n+\t\tu64 val = 0;\n+\t\tint err;\n+\n+\t\terr = bpf_map_direct_read(map, map_off, size, \u0026val, is_ldsx);\n+\t\tif (err)\n+\t\t\treturn err;\n+\n+\t\tregs[value_regno].type = SCALAR_VALUE;\n+\t\t__mark_reg_known(\u0026regs[value_regno], val);\n+\t\treturn 0;\n+\t}\n+\n+reg_unknown:\n+\tmark_reg_unknown(env, regs, value_regno);\n+\treturn 0;\n+}\n+\n /* check whether memory at (regno + off) is accessible for t = (read | write)\n  * if t==write, value_regno is a register which value is stored into memory\n  * if t==read, value_regno is a register which will receive the value from memory\n@@ -6137,38 +6179,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, struct b\n \t\tif (kptr_field) {\n \t\t\terr = check_map_kptr_access(env, value_regno, insn_idx, kptr_field);\n \t\t} else if (t == BPF_READ \u0026\u0026 value_regno \u003e= 0) {\n-\t\t\tstruct bpf_map *map = reg-\u003emap_ptr;\n-\n-\t\t\t/*\n-\t\t\t * If map is read-only, track its contents as scalars,\n-\t\t\t * unless it is an insn array (see the special case below)\n-\t\t\t */\n-\t\t\tif (tnum_is_const(reg-\u003evar_off) \u0026\u0026\n-\t\t\t    bpf_map_is_rdonly(map) \u0026\u0026\n-\t\t\t    map-\u003eops-\u003emap_direct_value_addr \u0026\u0026\n-\t\t\t    map-\u003emap_type != BPF_MAP_TYPE_INSN_ARRAY) {\n-\t\t\t\tint map_off = off + reg-\u003evar_off.value;\n-\t\t\t\tu64 val = 0;\n-\n-\t\t\t\terr = bpf_map_direct_read(map, map_off, size,\n-\t\t\t\t\t\t\t  \u0026val, is_ldsx);\n-\t\t\t\tif (err)\n-\t\t\t\t\treturn err;\n-\n-\t\t\t\tregs[value_regno].type = SCALAR_VALUE;\n-\t\t\t\t__mark_reg_known(\u0026regs[value_regno], val);\n-\t\t\t} else if (map-\u003emap_type == BPF_MAP_TYPE_INSN_ARRAY) {\n-\t\t\t\tif (bpf_size != BPF_DW) {\n-\t\t\t\t\tverbose(env, \"Invalid read of %d bytes from insn_array\\n\",\n-\t\t\t\t\t\t     size);\n-\t\t\t\t\treturn -EACCES;\n-\t\t\t\t}\n-\t\t\t\tregs[value_regno] = *reg;\n-\t\t\t\tadd_scalar_to_reg(\u0026regs[value_regno], off);\n-\t\t\t\tregs[value_regno].type = PTR_TO_INSN;\n-\t\t\t} else {\n-\t\t\t\tmark_reg_unknown(env, regs, value_regno);\n-\t\t\t}\n+\t\t\terr = check_map_mem_read(env, reg, off, bpf_size, value_regno, is_ldsx);\n \t\t}\n \t} else if (base_type(reg-\u003etype) == PTR_TO_MEM) {\n \t\tbool rdonly_mem = type_is_rdonly_mem(reg-\u003etype);\n@@ -6635,7 +6646,6 @@ static int check_stack_range_initialized(\n \tif (err)\n \t\treturn err;\n \n-\n \tif (tnum_is_const(reg-\u003evar_off)) {\n \t\tmin_off = max_off = reg-\u003evar_off.value + off;\n \t} else {\n@@ -7347,7 +7357,6 @@ static bool is_iter_new_kfunc(struct bpf_call_arg_meta *meta)\n \treturn meta-\u003ekfunc_flags \u0026 KF_ITER_NEW;\n }\n \n-\n static bool is_iter_destroy_kfunc(struct bpf_call_arg_meta *meta)\n {\n \treturn meta-\u003ekfunc_flags \u0026 KF_ITER_DESTROY;\n@@ -8125,6 +8134,12 @@ static int check_arg_const_str(struct bpf_verifier_env *env,\n \t\treturn -EACCES;\n \t}\n \n+\tif (map-\u003emap_type == BPF_MAP_TYPE_PERCPU_ARRAY) {\n+\t\tverbose(env, \"%s points to percpu_array map which cannot be used as const string\\n\",\n+\t\t\treg_arg_name(env, argno));\n+\t\treturn -EACCES;\n+\t}\n+\n \tif (!bpf_map_is_rdonly(map)) {\n \t\tverbose(env, \"%s does not point to a readonly map'\\n\", reg_arg_name(env, argno));\n \t\treturn -EACCES;\n@@ -11607,7 +11622,6 @@ static int process_irq_flag(struct bpf_verifier_env *env, struct bpf_reg_state *\n \treturn 0;\n }\n \n-\n static int ref_set_non_owning(struct bpf_verifier_env *env, struct bpf_reg_state *reg)\n {\n \tstruct btf_record *rec = reg_btf_record(reg);\n@@ -16412,7 +16426,6 @@ static int check_ld_abs(struct bpf_verifier_env *env, struct bpf_insn *insn)\n \treturn 0;\n }\n \n-\n static bool return_retval_range(struct bpf_verifier_env *env, struct bpf_retval_range *range)\n {\n \tenum bpf_prog_type prog_type = resolve_prog_type(env-\u003eprog);\n@@ -18361,8 +18374,6 @@ static void release_insn_arrays(struct bpf_verifier_env *env)\n \t\tbpf_insn_array_release(env-\u003einsn_array_maps[i]);\n }\n \n-\n-\n /* The verifier does more data flow analysis than llvm and will not\n  * explore branches that are dead at run time. Malicious programs can\n  * have dead code too. Therefore replace all dead at-run-time code\n@@ -18390,8 +18401,6 @@ static void sanitize_dead_code(struct bpf_verifier_env *env)\n \t}\n }\n \n-\n-\n static void free_states(struct bpf_verifier_env *env)\n {\n \tstruct bpf_verifier_state_list *sl;\n@@ -18678,7 +18687,6 @@ static int do_check_main(struct bpf_verifier_env *env)\n \treturn ret;\n }\n \n-\n static void print_verification_stats(struct bpf_verifier_env *env)\n {\n \t/* Skip over hidden subprogs which are not verified. */\ndiff --git a/tools/bpf/bpftool/gen.c b/tools/bpf/bpftool/gen.c\nindex a01d06d22d1a3..a50540ef6521c 100644\n--- a/tools/bpf/bpftool/gen.c\n+++ b/tools/bpf/bpftool/gen.c\n@@ -101,6 +101,12 @@ static bool get_map_ident(const struct bpf_map *map, char *buf, size_t buf_sz)\n \t\treturn true;\n \t}\n \n+\tif (bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY) {\n+\t\tsnprintf(buf, buf_sz, \"%s\", name + 1);\n+\t\tsanitize_identifier(buf);\n+\t\treturn true;\n+\t}\n+\n \tfor  (i = 0, n = ARRAY_SIZE(sfxs); i \u003c n; i++) {\n \t\tconst char *sfx = sfxs[i], *p;\n \n@@ -117,7 +123,7 @@ static bool get_map_ident(const struct bpf_map *map, char *buf, size_t buf_sz)\n \n static bool get_datasec_ident(const char *sec_name, char *buf, size_t buf_sz)\n {\n-\tstatic const char *pfxs[] = { \".data\", \".rodata\", \".bss\", \".kconfig\" };\n+\tstatic const char *pfxs[] = { \".data\", \".rodata\", \".bss\", \".percpu\", \".kconfig\" };\n \tint i, n;\n \n \t/* recognize hard coded LLVM section name */\n@@ -254,7 +260,7 @@ static const struct btf_type *find_type_for_map(struct btf *btf, const char *map\n \treturn NULL;\n }\n \n-static bool is_mmapable_map(const struct bpf_map *map, char *buf, size_t sz)\n+static bool is_skel_data(const struct bpf_map *map, char *buf, size_t sz)\n {\n \tsize_t tmp_sz;\n \n@@ -263,13 +269,24 @@ static bool is_mmapable_map(const struct bpf_map *map, char *buf, size_t sz)\n \t\treturn true;\n \t}\n \n-\tif (!bpf_map__is_internal(map) || !(bpf_map__map_flags(map) \u0026 BPF_F_MMAPABLE))\n+\tif (!bpf_map__is_internal(map))\n \t\treturn false;\n \n \tif (!get_map_ident(map, buf, sz))\n \t\treturn false;\n \n-\treturn true;\n+\tif (bpf_map__map_flags(map) \u0026 BPF_F_MMAPABLE)\n+\t\treturn true;\n+\n+\tif (bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY)\n+\t\treturn bpf_map__btf_value_type_id(map) != 0;\n+\n+\treturn false;\n+}\n+\n+static bool is_mmapable_map(const struct bpf_map *map, char *buf, size_t sz)\n+{\n+\treturn is_skel_data(map, buf, sz) \u0026\u0026 bpf_map__type(map) != BPF_MAP_TYPE_PERCPU_ARRAY;\n }\n \n static int codegen_datasecs(struct bpf_object *obj, const char *obj_name)\n@@ -287,7 +304,7 @@ static int codegen_datasecs(struct bpf_object *obj, const char *obj_name)\n \n \tbpf_object__for_each_map(map, obj) {\n \t\t/* only generate definitions for memory-mapped internal maps */\n-\t\tif (!is_mmapable_map(map, map_ident, sizeof(map_ident)))\n+\t\tif (!is_skel_data(map, map_ident, sizeof(map_ident)))\n \t\t\tcontinue;\n \n \t\tsec = find_type_for_map(btf, map_ident);\n@@ -517,7 +534,7 @@ static void codegen_asserts(struct bpf_object *obj, const char *obj_name)\n \t\t\", obj_name);\n \n \tbpf_object__for_each_map(map, obj) {\n-\t\tif (!is_mmapable_map(map, map_ident, sizeof(map_ident)))\n+\t\tif (!is_skel_data(map, map_ident, sizeof(map_ident)))\n \t\t\tcontinue;\n \n \t\tsec = find_type_for_map(btf, map_ident);\n@@ -668,8 +685,7 @@ static void codegen_destroy(struct bpf_object *obj, const char *obj_name)\n \tbpf_object__for_each_map(map, obj) {\n \t\tif (!get_map_ident(map, ident, sizeof(ident)))\n \t\t\tcontinue;\n-\t\tif (bpf_map__is_internal(map) \u0026\u0026\n-\t\t    (bpf_map__map_flags(map) \u0026 BPF_F_MMAPABLE))\n+\t\tif (is_skel_data(map, ident, sizeof(ident)))\n \t\t\tprintf(\"\\tskel_free_map_data(skel-\u003e%1$s, skel-\u003emaps.%1$s.initial_value, %2$zu);\\n\",\n \t\t\t       ident, bpf_map_mmap_sz(map));\n \t\tcodegen(\"\\\n@@ -741,7 +757,7 @@ static int gen_trace(struct bpf_object *obj, const char *obj_name, const char *h\n \t\tconst void *mmap_data = NULL;\n \t\tsize_t mmap_size = 0;\n \n-\t\tif (!is_mmapable_map(map, ident, sizeof(ident)))\n+\t\tif (!is_skel_data(map, ident, sizeof(ident)))\n \t\t\tcontinue;\n \n \t\tcodegen(\"\\\n@@ -849,9 +865,23 @@ static int gen_trace(struct bpf_object *obj, const char *obj_name, const char *h\n \tbpf_object__for_each_map(map, obj) {\n \t\tconst char *mmap_flags;\n \n-\t\tif (!is_mmapable_map(map, ident, sizeof(ident)))\n+\t\tif (!is_skel_data(map, ident, sizeof(ident)))\n \t\t\tcontinue;\n \n+\t\tif (bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY) {\n+\t\t\tcodegen(\"\\\n+\t\t\\n\\\n+\t\t\terr = skel_protect_map_data(skel-\u003e%1$s, \u0026skel-\u003emaps.%1$s.initial_value, %2$zd);\\n\\\n+\t\t\tif (err)\t\t\t\t\t    \\n\\\n+\t\t\t\treturn err;\t\t\t\t    \\n\\\n+\t\t#ifdef __KERNEL__\t\t\t\t\t    \\n\\\n+\t\t\tskel-\u003e%1$s = NULL;\t\t\t\t    \\n\\\n+\t\t#endif\t\t\t\t\t\t\t    \\n\\\n+\t\t\t\",\n+\t\t\tident, bpf_map_mmap_sz(map));\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\tif (bpf_map__map_flags(map) \u0026 BPF_F_RDONLY_PROG)\n \t\t\tmmap_flags = \"PROT_READ\";\n \t\telse\n@@ -955,8 +985,7 @@ codegen_maps_skeleton(struct bpf_object *obj, size_t map_cnt, bool mmaped, bool\n \t\t\t\tmap-\u003emap = \u0026obj-\u003emaps.%s;\t    \\n\\\n \t\t\t\",\n \t\t\ti, bpf_map__name(map), ident);\n-\t\t/* memory-mapped internal maps */\n-\t\tif (mmaped \u0026\u0026 is_mmapable_map(map, ident, sizeof(ident))) {\n+\t\tif (mmaped \u0026\u0026 is_skel_data(map, ident, sizeof(ident))) {\n \t\t\tprintf(\"\\tmap-\u003emmaped = (void **)\u0026obj-\u003e%s;\\n\", ident);\n \t\t}\n \ndiff --git a/tools/lib/bpf/bpf_gen_internal.h b/tools/lib/bpf/bpf_gen_internal.h\nindex 0425691877521..6c5ad6c55e8a6 100644\n--- a/tools/lib/bpf/bpf_gen_internal.h\n+++ b/tools/lib/bpf/bpf_gen_internal.h\n@@ -65,7 +65,8 @@ void bpf_gen__prog_load(struct bpf_gen *gen,\n \t\t\tenum bpf_prog_type prog_type, const char *prog_name,\n \t\t\tconst char *license, struct bpf_insn *insns, size_t insn_cnt,\n \t\t\tstruct bpf_prog_load_opts *load_attr, int prog_idx);\n-void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *value, __u32 value_size);\n+void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *value, __u32 value_size,\n+\t\t\t      __u64 flags);\n void bpf_gen__map_freeze(struct bpf_gen *gen, int map_idx);\n void bpf_gen__record_attach_target(struct bpf_gen *gen, const char *name, enum bpf_attach_type type);\n void bpf_gen__record_extern(struct bpf_gen *gen, const char *name, bool is_weak,\ndiff --git a/tools/lib/bpf/features.c b/tools/lib/bpf/features.c\nindex b7e388f99d0bb..ef9581c113035 100644\n--- a/tools/lib/bpf/features.c\n+++ b/tools/lib/bpf/features.c\n@@ -620,6 +620,38 @@ static int probe_bpf_syscall_common_attrs(int token_fd)\n \treturn probe_sys_bpf_ext();\n }\n \n+static int probe_kern_percpu_data(int token_fd)\n+{\n+\tstruct bpf_insn insns[] = {\n+\t\tBPF_LD_MAP_VALUE(BPF_REG_1, 0, 0),\n+\t\tBPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, 0),\n+\t\tBPF_EXIT_INSN(),\n+\t};\n+\tLIBBPF_OPTS(bpf_map_create_opts, map_opts,\n+\t\t.token_fd = token_fd,\n+\t\t.map_flags = token_fd ? BPF_F_TOKEN_FD : 0,\n+\t);\n+\tLIBBPF_OPTS(bpf_prog_load_opts, prog_opts,\n+\t\t.token_fd = token_fd,\n+\t\t.prog_flags = token_fd ? BPF_F_TOKEN_FD : 0,\n+\t);\n+\tint ret, map, insn_cnt = ARRAY_SIZE(insns);\n+\n+\tmap = bpf_map_create(BPF_MAP_TYPE_PERCPU_ARRAY, \"libbpf_percpu\", sizeof(int), 8, 1,\n+\t\t\t     \u0026map_opts);\n+\tif (map \u003c 0) {\n+\t\tpr_warn(\"Error in %s(): %s. Couldn't create simple percpu_array map.\\n\",\n+\t\t\t__func__, errstr(map));\n+\t\treturn map;\n+\t}\n+\n+\tinsns[0].imm = map;\n+\n+\tret = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, NULL, \"GPL\", insns, insn_cnt, \u0026prog_opts);\n+\tclose(map);\n+\treturn probe_fd(ret);\n+}\n+\n typedef int (*feature_probe_fn)(int /* token_fd */);\n \n static struct kern_feature_cache feature_cache;\n@@ -707,6 +739,9 @@ static struct kern_feature_desc {\n \t[FEAT_BPF_SYSCALL_COMMON_ATTRS] = {\n \t\t\"BPF syscall common attributes support\", probe_bpf_syscall_common_attrs,\n \t},\n+\t[FEAT_PERCPU_DATA] = {\n+\t\t\"kernel supports percpu data\", probe_kern_percpu_data,\n+\t},\n };\n \n bool feat_supported(struct kern_feature_cache *cache, enum kern_feature_id feat_id)\ndiff --git a/tools/lib/bpf/gen_loader.c b/tools/lib/bpf/gen_loader.c\nindex 6e3dd52427618..af3a04f161ac1 100644\n--- a/tools/lib/bpf/gen_loader.c\n+++ b/tools/lib/bpf/gen_loader.c\n@@ -1128,7 +1128,7 @@ void bpf_gen__prog_load(struct bpf_gen *gen,\n }\n \n void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *pvalue,\n-\t\t\t      __u32 value_size)\n+\t\t\t      __u32 value_size, __u64 flags)\n {\n \tint attr_size = offsetofend(union bpf_attr, flags);\n \tint map_update_attr, value, key;\n@@ -1136,6 +1136,7 @@ void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *pvalue,\n \tint zero = 0;\n \n \tmemset(\u0026attr, 0, attr_size);\n+\tattr.flags = tgt_endian(flags);\n \n \tvalue = add_data(gen, pvalue, value_size);\n \tkey = add_data(gen, \u0026zero, sizeof(zero));\ndiff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c\nindex 514e4e9daa825..e574870fb7169 100644\n--- a/tools/lib/bpf/libbpf.c\n+++ b/tools/lib/bpf/libbpf.c\n@@ -541,6 +541,7 @@ struct bpf_struct_ops {\n };\n \n #define DATA_SEC \".data\"\n+#define PERCPU_SEC \".percpu\"\n #define BSS_SEC \".bss\"\n #define RODATA_SEC \".rodata\"\n #define KCONFIG_SEC \".kconfig\"\n@@ -555,6 +556,7 @@ enum libbpf_map_type {\n \tLIBBPF_MAP_BSS,\n \tLIBBPF_MAP_RODATA,\n \tLIBBPF_MAP_KCONFIG,\n+\tLIBBPF_MAP_PERCPU,\n };\n \n struct bpf_map_def {\n@@ -666,6 +668,7 @@ enum sec_type {\n \tSEC_DATA,\n \tSEC_RODATA,\n \tSEC_ST_OPS,\n+\tSEC_PERCPU,\n };\n \n struct elf_sec_desc {\n@@ -1839,6 +1842,8 @@ static size_t bpf_map_mmap_sz(const struct bpf_map *map)\n \tswitch (map-\u003edef.type) {\n \tcase BPF_MAP_TYPE_ARRAY:\n \t\treturn array_map_mmap_sz(map-\u003edef.value_size, map-\u003edef.max_entries);\n+\tcase BPF_MAP_TYPE_PERCPU_ARRAY:\n+\t\treturn map-\u003edef.value_size;\n \tcase BPF_MAP_TYPE_ARENA:\n \t\treturn page_sz * map-\u003edef.max_entries;\n \tdefault:\n@@ -1866,7 +1871,8 @@ static int bpf_map_mmap_resize(struct bpf_map *map, size_t old_sz, size_t new_sz\n \treturn 0;\n }\n \n-static char *internal_map_name(struct bpf_object *obj, const char *real_name)\n+static char *internal_map_name(struct bpf_object *obj, const char *real_name,\n+\t\t\t       enum libbpf_map_type type)\n {\n \tchar map_name[BPF_OBJ_NAME_LEN], *p;\n \tint pfx_len, sfx_len = max((size_t)7, strlen(real_name));\n@@ -1907,8 +1913,11 @@ static char *internal_map_name(struct bpf_object *obj, const char *real_name)\n \tif (sfx_len \u003e= BPF_OBJ_NAME_LEN)\n \t\tsfx_len = BPF_OBJ_NAME_LEN - 1;\n \n-\t/* if there are two or more dots in map name, it's a custom dot map */\n-\tif (strchr(real_name + 1, '.') != NULL)\n+\t/*\n+\t * Don't prefix the bpf_object name if this is a custom dot map\n+\t * (containing two or more dots) or a percpu data map.\n+\t */\n+\tif (strchr(real_name + 1, '.') != NULL || type == LIBBPF_MAP_PERCPU)\n \t\tpfx_len = 0;\n \telse\n \t\tpfx_len = min((size_t)BPF_OBJ_NAME_LEN - sfx_len - 1, strlen(obj-\u003ename));\n@@ -1941,6 +1950,13 @@ static bool map_is_mmapable(struct bpf_object *obj, struct bpf_map *map)\n \tif (!map-\u003ebtf_value_type_id)\n \t\treturn false;\n \n+\t/*\n+\t * The internal PERCPU maps are not mmapble because the underlying\n+\t * percpu_array maps do not have mmap support.\n+\t */\n+\tif (map-\u003elibbpf_type == LIBBPF_MAP_PERCPU)\n+\t\treturn false;\n+\n \tt = btf__type_by_id(obj-\u003ebtf, map-\u003ebtf_value_type_id);\n \tif (!btf_is_datasec(t))\n \t\treturn false;\n@@ -1962,6 +1978,7 @@ static int\n bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type,\n \t\t\t      const char *real_name, int sec_idx, void *data, size_t data_sz)\n {\n+\tbool is_percpu = type == LIBBPF_MAP_PERCPU;\n \tstruct bpf_map_def *def;\n \tstruct bpf_map *map;\n \tsize_t mmap_sz;\n@@ -1975,7 +1992,7 @@ bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type,\n \tmap-\u003esec_idx = sec_idx;\n \tmap-\u003esec_offset = 0;\n \tmap-\u003ereal_name = strdup(real_name);\n-\tmap-\u003ename = internal_map_name(obj, real_name);\n+\tmap-\u003ename = internal_map_name(obj, real_name, type);\n \tif (!map-\u003ereal_name || !map-\u003ename) {\n \t\tzfree(\u0026map-\u003ereal_name);\n \t\tzfree(\u0026map-\u003ename);\n@@ -1983,7 +2000,7 @@ bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type,\n \t}\n \n \tdef = \u0026map-\u003edef;\n-\tdef-\u003etype = BPF_MAP_TYPE_ARRAY;\n+\tdef-\u003etype = is_percpu ? BPF_MAP_TYPE_PERCPU_ARRAY : BPF_MAP_TYPE_ARRAY;\n \tdef-\u003ekey_size = sizeof(int);\n \tdef-\u003evalue_size = data_sz;\n \tdef-\u003emax_entries = 1;\n@@ -1996,8 +2013,9 @@ bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type,\n \tif (map_is_mmapable(obj, map))\n \t\tdef-\u003emap_flags |= BPF_F_MMAPABLE;\n \n-\tpr_debug(\"map '%s' (global data): at sec_idx %d, offset %zu, flags %x.\\n\",\n-\t\t map-\u003ename, map-\u003esec_idx, map-\u003esec_offset, def-\u003emap_flags);\n+\tpr_debug(\"map '%s' (global %sdata): at sec_idx %d, offset %zu, flags %x.\\n\",\n+\t\t map-\u003ename, is_percpu ? \"percpu \" : \"\", map-\u003esec_idx,\n+\t\t map-\u003esec_offset, def-\u003emap_flags);\n \n \tmmap_sz = bpf_map_mmap_sz(map);\n \tmap-\u003emmaped = mmap(NULL, mmap_sz, PROT_READ | PROT_WRITE,\n@@ -2057,6 +2075,13 @@ static int bpf_object__init_global_data_maps(struct bpf_object *obj)\n \t\t\t\t\t\t\t    NULL,\n \t\t\t\t\t\t\t    sec_desc-\u003edata-\u003ed_size);\n \t\t\tbreak;\n+\t\tcase SEC_PERCPU:\n+\t\t\tsec_name = elf_sec_name(obj, elf_sec_by_idx(obj, sec_idx));\n+\t\t\terr = bpf_object__init_internal_map(obj, LIBBPF_MAP_PERCPU,\n+\t\t\t\t\t\t\t    sec_name, sec_idx,\n+\t\t\t\t\t\t\t    sec_desc-\u003edata-\u003ed_buf,\n+\t\t\t\t\t\t\t    sec_desc-\u003edata-\u003ed_size);\n+\t\t\tbreak;\n \t\tdefault:\n \t\t\t/* skip */\n \t\t\tbreak;\n@@ -4016,6 +4041,11 @@ static int bpf_object__elf_collect(struct bpf_object *obj)\n \t\t\t\tsec_desc-\u003esec_type = SEC_RODATA;\n \t\t\t\tsec_desc-\u003eshdr = sh;\n \t\t\t\tsec_desc-\u003edata = data;\n+\t\t\t} else if (strcmp(name, PERCPU_SEC) == 0 ||\n+\t\t\t\t   str_has_pfx(name, PERCPU_SEC \".\")) {\n+\t\t\t\tsec_desc-\u003esec_type = SEC_PERCPU;\n+\t\t\t\tsec_desc-\u003eshdr = sh;\n+\t\t\t\tsec_desc-\u003edata = data;\n \t\t\t} else if (strcmp(name, STRUCT_OPS_SEC) == 0 ||\n \t\t\t\t   strcmp(name, STRUCT_OPS_LINK_SEC) == 0 ||\n \t\t\t\t   strcmp(name, \"?\" STRUCT_OPS_SEC) == 0 ||\n@@ -4544,6 +4574,7 @@ static bool bpf_object__shndx_is_data(const struct bpf_object *obj,\n \tcase SEC_BSS:\n \tcase SEC_DATA:\n \tcase SEC_RODATA:\n+\tcase SEC_PERCPU:\n \t\treturn true;\n \tdefault:\n \t\treturn false;\n@@ -4569,6 +4600,8 @@ bpf_object__section_to_libbpf_map_type(const struct bpf_object *obj, int shndx)\n \t\treturn LIBBPF_MAP_DATA;\n \tcase SEC_RODATA:\n \t\treturn LIBBPF_MAP_RODATA;\n+\tcase SEC_PERCPU:\n+\t\treturn LIBBPF_MAP_PERCPU;\n \tdefault:\n \t\treturn LIBBPF_MAP_UNSPEC;\n \t}\n@@ -4944,7 +4977,7 @@ static int map_fill_btf_type_info(struct bpf_object *obj, struct bpf_map *map)\n \n \t/*\n \t * LLVM annotates global data differently in BTF, that is,\n-\t * only as '.data', '.bss' or '.rodata'.\n+\t * only as '.data', '.bss', '.percpu' or '.rodata'.\n \t */\n \tif (!bpf_map__is_internal(map))\n \t\treturn -ENOENT;\n@@ -5293,18 +5326,20 @@ static int\n bpf_object__populate_internal_map(struct bpf_object *obj, struct bpf_map *map)\n {\n \tenum libbpf_map_type map_type = map-\u003elibbpf_type;\n+\tbool is_percpu = map_type == LIBBPF_MAP_PERCPU;\n+\tconst __u64 update_flags = is_percpu ? BPF_F_ALL_CPUS : 0;\n \tint err, zero = 0;\n \tsize_t mmap_sz;\n \n \tif (obj-\u003egen_loader) {\n \t\tbpf_gen__map_update_elem(obj-\u003egen_loader, map - obj-\u003emaps,\n-\t\t\t\t\t map-\u003emmaped, map-\u003edef.value_size);\n+\t\t\t\t\t map-\u003emmaped, map-\u003edef.value_size, update_flags);\n \t\tif (map_type == LIBBPF_MAP_RODATA || map_type == LIBBPF_MAP_KCONFIG)\n \t\t\tbpf_gen__map_freeze(obj-\u003egen_loader, map - obj-\u003emaps);\n \t\treturn 0;\n \t}\n \n-\terr = bpf_map_update_elem(map-\u003efd, \u0026zero, map-\u003emmaped, 0);\n+\terr = bpf_map_update_elem(map-\u003efd, \u0026zero, map-\u003emmaped, update_flags);\n \tif (err) {\n \t\terr = -errno;\n \t\tpr_warn(\"map '%s': failed to set initial contents: %s\\n\",\n@@ -5349,6 +5384,13 @@ bpf_object__populate_internal_map(struct bpf_object *obj, struct bpf_map *map)\n \t\t\treturn err;\n \t\t}\n \t\tmap-\u003emmaped = mmaped;\n+\t} else if (is_percpu) {\n+\t\tif (mprotect(map-\u003emmaped, mmap_sz, PROT_READ)) {\n+\t\t\terr = -errno;\n+\t\t\tpr_warn(\"map '%s': failed to mprotect() contents: %s\\n\",\n+\t\t\t\tbpf_map__name(map), errstr(err));\n+\t\t\treturn err;\n+\t\t}\n \t} else if (map-\u003emmaped) {\n \t\tmunmap(map-\u003emmaped, mmap_sz);\n \t\tmap-\u003emmaped = NULL;\n@@ -5624,9 +5666,16 @@ bpf_object__create_maps(struct bpf_object *obj)\n \t\t * runtime due to bpf_program__set_autoload(prog, false),\n \t\t * bpf_object loading will succeed just fine even on old\n \t\t * kernels.\n+\t\t * Same skipping applies to percpu data.\n \t\t */\n-\t\tif (bpf_map__is_internal(map) \u0026\u0026 !kernel_supports(obj, FEAT_GLOBAL_DATA))\n-\t\t\tmap-\u003eautocreate = false;\n+\t\tif (bpf_map__is_internal(map)) {\n+\t\t\tbool is_percpu = map-\u003elibbpf_type == LIBBPF_MAP_PERCPU;\n+\t\t\tenum kern_feature_id feat_id;\n+\n+\t\t\tfeat_id = is_percpu ? FEAT_PERCPU_DATA : FEAT_GLOBAL_DATA;\n+\t\t\tif (!kernel_supports(obj, feat_id))\n+\t\t\t\tmap-\u003eautocreate = false;\n+\t\t}\n \n \t\tif (!map-\u003eautocreate) {\n \t\t\tpr_debug(\"map '%s': skipped auto-creating...\\n\", map-\u003ename);\n@@ -10807,11 +10856,16 @@ static bool map_uses_real_name(const struct bpf_map *map)\n \t * such map's corresponding ELF section name as a map name.\n \t * This check distinguishes .data/.rodata from .data.* and .rodata.*\n \t * maps to know which name has to be returned to the user.\n+\t * Map name of the custom .percpu.* maps might be truncated to\n+\t * BPF_OBJ_NAME_LEN-1 chars in internal_map_name(). Hence, percpu data\n+\t * maps must use real name for their user-visible name.\n \t */\n \tif (map-\u003elibbpf_type == LIBBPF_MAP_DATA \u0026\u0026 strcmp(map-\u003ereal_name, DATA_SEC) != 0)\n \t\treturn true;\n \tif (map-\u003elibbpf_type == LIBBPF_MAP_RODATA \u0026\u0026 strcmp(map-\u003ereal_name, RODATA_SEC) != 0)\n \t\treturn true;\n+\tif (map-\u003elibbpf_type == LIBBPF_MAP_PERCPU)\n+\t\treturn true;\n \treturn false;\n }\n \n@@ -10976,7 +11030,8 @@ int bpf_map__set_value_size(struct bpf_map *map, __u32 size)\n \t\tsize_t mmap_old_sz, mmap_new_sz;\n \t\tint err;\n \n-\t\tif (map-\u003edef.type != BPF_MAP_TYPE_ARRAY)\n+\t\tif (map-\u003edef.type != BPF_MAP_TYPE_ARRAY \u0026\u0026\n+\t\t    map-\u003edef.type != BPF_MAP_TYPE_PERCPU_ARRAY)\n \t\t\treturn libbpf_err(-EOPNOTSUPP);\n \n \t\tmmap_old_sz = bpf_map_mmap_sz(map);\ndiff --git a/tools/lib/bpf/libbpf_internal.h b/tools/lib/bpf/libbpf_internal.h\nindex 7a74abb904f8b..4c46d34fc055d 100644\n--- a/tools/lib/bpf/libbpf_internal.h\n+++ b/tools/lib/bpf/libbpf_internal.h\n@@ -401,6 +401,8 @@ enum kern_feature_id {\n \tFEAT_BTF_LAYOUT,\n \t/* Kernel supports BPF syscall common attributes */\n \tFEAT_BPF_SYSCALL_COMMON_ATTRS,\n+\t/* Kernel supports percpu data */\n+\tFEAT_PERCPU_DATA,\n \t__FEAT_CNT,\n };\n \ndiff --git a/tools/lib/bpf/skel_internal.h b/tools/lib/bpf/skel_internal.h\nindex 53fee53d36d51..1f3f332dffbe9 100644\n--- a/tools/lib/bpf/skel_internal.h\n+++ b/tools/lib/bpf/skel_internal.h\n@@ -131,8 +131,10 @@ static inline void skel_free_map_data(void *p, __u64 addr, size_t sz)\n {\n \tif (addr != ~0ULL)\n \t\tkvfree(p);\n-\t/* When addr == ~0ULL the 'p' points to\n-\t * ((struct bpf_array *)map)-\u003evalue. See skel_finalize_map_data.\n+\t/*\n+\t * When addr == ~0ULL the init buffer has already been released.\n+\t * For skel_finalize_map_data(), 'p' points to\n+\t * ((struct bpf_array *)map)-\u003evalue.\n \t */\n }\n \n@@ -170,6 +172,15 @@ static inline void *skel_finalize_map_data(__u64 *init_val, size_t mmap_sz, int\n \treturn addr;\n }\n \n+static inline int skel_protect_map_data(void *p, __u64 *init_val, size_t sz)\n+{\n+\t(void)sz;\n+\n+\tkvfree(p);\n+\t*init_val = ~0ULL;\n+\treturn 0;\n+}\n+\n #else\n \n static inline void *skel_alloc(size_t size)\n@@ -208,6 +219,15 @@ static inline void *skel_finalize_map_data(__u64 *init_val, size_t mmap_sz, int\n \t\treturn NULL;\n \treturn addr;\n }\n+\n+static inline int skel_protect_map_data(void *p, __u64 *init_val, size_t sz)\n+{\n+\t(void)init_val;\n+\n+\tif (mprotect(p, sz, PROT_READ))\n+\t\treturn -errno;\n+\treturn 0;\n+}\n #endif\n \n static inline int skel_closenz(int fd)\ndiff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile\nindex d3655a7064826..560ce4016fbff 100644\n--- a/tools/testing/selftests/bpf/Makefile\n+++ b/tools/testing/selftests/bpf/Makefile\n@@ -531,7 +531,7 @@ LSKELS_SIGNED := fentry_test.c fexit_test.c atomics.c\n \n # Generate both light skeleton and libbpf skeleton for these\n LSKELS_EXTRA := test_ksyms_module.c test_ksyms_weak.c kfunc_call_test.c \\\n-\tkfunc_call_test_subprog.c\n+\tkfunc_call_test_subprog.c test_global_percpu_data.c\n SKEL_BLACKLIST += $$(LSKELS) $$(LSKELS_SIGNED)\n \n test_static_linked.skel.h-deps := test_static_linked1.bpf.o test_static_linked2.bpf.o\ndiff --git a/tools/testing/selftests/bpf/prog_tests/global_data_init.c b/tools/testing/selftests/bpf/prog_tests/global_data_init.c\nindex 8466332d7406f..7d6bda9092954 100644\n--- a/tools/testing/selftests/bpf/prog_tests/global_data_init.c\n+++ b/tools/testing/selftests/bpf/prog_tests/global_data_init.c\n@@ -1,5 +1,8 @@\n // SPDX-License-Identifier: GPL-2.0\n #include \u003ctest_progs.h\u003e\n+#include \"bpf/libbpf_internal.h\"\n+#include \"test_global_percpu_data.skel.h\"\n+#include \"test_global_percpu_data.lskel.h\"\n \n void test_global_data_init(void)\n {\n@@ -60,3 +63,336 @@ void test_global_data_init(void)\n \tfree(newval);\n \tbpf_object__close(obj);\n }\n+\n+static void test_percpu_data_on_cpus(struct bpf_map *map, int map_fd, int prog_fd, int *runp)\n+{\n+\tstruct test_global_percpu_data__percpu *data = NULL;\n+\tint i, err, key = 0, num_online, run = 0;\n+\t__u64 args[2] = {0x1234ULL, 0x5678ULL};\n+\tsize_t data_sz;\n+\tbool *online;\n+\tLIBBPF_OPTS(bpf_test_run_opts, topts,\n+\t\t    .ctx_in = args,\n+\t\t    .ctx_size_in = sizeof(args),\n+\t\t    .flags = BPF_F_TEST_RUN_ON_CPU,\n+\t);\n+\n+\terr = parse_cpu_mask_file(\"/sys/devices/system/cpu/online\", \u0026online, \u0026num_online);\n+\tif (!ASSERT_OK(err, \"parse_cpu_mask_file\"))\n+\t\treturn;\n+\n+\tdata_sz = map ? bpf_map__value_size(map) : sizeof(*data);\n+\tdata = calloc(1, data_sz);\n+\tif (!ASSERT_OK_PTR(data, \"calloc percpu data\"))\n+\t\tgoto out;\n+\n+\t/* run on every online-CPU */\n+\tfor (i = 0; i \u003c num_online; i++) {\n+\t\t__u64 flags;\n+\n+\t\tif (!online[i])\n+\t\t\tcontinue;\n+\n+\t\ttopts.cpu = i;\n+\t\ttopts.retval = -1;\n+\t\terr = bpf_prog_test_run_opts(prog_fd, \u0026topts);\n+\t\tASSERT_OK(err, \"bpf_prog_test_run_opts\");\n+\t\tASSERT_EQ(topts.retval, 0, \"bpf_prog_test_run_opts retval\");\n+\n+\t\tmemset(data, 0, data_sz);\n+\t\tflags = ((__u64) i \u003c\u003c 32) | BPF_F_CPU;\n+\t\tif (map)\n+\t\t\terr = bpf_map__lookup_elem(map, \u0026key, sizeof(key), data, data_sz, flags);\n+\t\telse\n+\t\t\terr = bpf_map_lookup_elem_flags(map_fd, \u0026key, data, flags);\n+\t\tif (!ASSERT_OK(err, \"lookup_elem on cpu\"))\n+\t\t\tbreak;\n+\n+\t\tASSERT_EQ(*runp, ++run, \"run\");\n+\t\tASSERT_EQ(data-\u003ecpu_id[0], i, \"cpu_id\");\n+\t\tASSERT_EQ(data-\u003edata, 1, \"data\");\n+\t\tASSERT_TRUE(data-\u003eset, \"set\");\n+\t\tASSERT_EQ(data-\u003enums[6], 0xc0de, \"nums[6]\");\n+\t\tASSERT_EQ(data-\u003estruct_data.i, 1, \"struct_data.i\");\n+\t\tASSERT_TRUE(data-\u003estruct_data.set, \"struct_data.set\");\n+\t\tASSERT_EQ(data-\u003estruct_data.nums[6], 0xc0de, \"struct_data.nums[6]\");\n+\t}\n+\n+out:\n+\tfree(data);\n+\tfree(online);\n+}\n+\n+static void test_global_percpu_data_init(void)\n+{\n+\tstruct test_global_percpu_data__percpu init_value = {};\n+\tstruct test_global_percpu_data__percpu *init_data;\n+\tconst __u32 desired_sz = sysconf(_SC_PAGE_SIZE);\n+\tstruct test_global_percpu_data *skel = NULL;\n+\tsize_t init_data_sz;\n+\tstruct bpf_map *map;\n+\tint prog_fd, err;\n+\n+\tskel = test_global_percpu_data__open();\n+\tif (!ASSERT_OK_PTR(skel, \"test_global_percpu_data__open\"))\n+\t\tgoto out;\n+\tif (!ASSERT_OK_PTR(skel-\u003epercpu, \"skel-\u003epercpu\"))\n+\t\tgoto out;\n+\tif (!ASSERT_OK_PTR(skel-\u003edata_percpu, \"skel-\u003edata_percpu\"))\n+\t\tgoto out;\n+\tif (!ASSERT_OK_PTR(skel-\u003epercpu_data, \"skel-\u003epercpu_data\"))\n+\t\tgoto out;\n+\tif (!ASSERT_OK_PTR(skel-\u003epercpu_looooooooong, \"skel-\u003epercpu_looooooooong\"))\n+\t\tgoto out;\n+\n+\tASSERT_STREQ(bpf_map__name(skel-\u003emaps.percpu_data), \".percpu.data\",\n+\t\t     \".percpu.data map name\");\n+\tASSERT_STREQ(bpf_map__name(skel-\u003emaps.data_percpu), \".data.percpu\",\n+\t\t     \".data.percpu map name\");\n+\tASSERT_STREQ(bpf_map__name(skel-\u003emaps.percpu_looooooooong), \".percpu.looooooooong\",\n+\t\t     \"long map name\");\n+\tASSERT_STREQ(bpf_map__name(skel-\u003emaps.percpu), \".percpu\", \"map name\");\n+\tASSERT_EQ(skel-\u003epercpu-\u003edata, -1, \"skel-\u003epercpu-\u003edata\");\n+\tASSERT_FALSE(skel-\u003epercpu-\u003eset, \"skel-\u003epercpu-\u003eset\");\n+\tASSERT_EQ(skel-\u003epercpu-\u003enums[6], 0, \"skel-\u003epercpu-\u003enums[6]\");\n+\tASSERT_EQ(skel-\u003epercpu-\u003estruct_data.i, -1, \"struct_data.i\");\n+\tASSERT_FALSE(skel-\u003epercpu-\u003estruct_data.set, \"struct_data.set\");\n+\tASSERT_EQ(skel-\u003epercpu-\u003estruct_data.nums[6], 0, \"struct_data.nums[6]\");\n+\n+\tmap = skel-\u003emaps.percpu;\n+\tif (!ASSERT_EQ(bpf_map__type(map), BPF_MAP_TYPE_PERCPU_ARRAY, \"bpf_map__type\"))\n+\t\tgoto out;\n+\n+\tinit_value.data = 2;\n+\tinit_value.nums[6] = -1;\n+\tinit_value.struct_data.i = 2;\n+\tinit_value.struct_data.nums[6] = -1;\n+\terr = bpf_map__set_initial_value(map, \u0026init_value, sizeof(init_value));\n+\tif (!ASSERT_OK(err, \"bpf_map__set_initial_value\"))\n+\t\tgoto out;\n+\n+\tinit_data = bpf_map__initial_value(map, \u0026init_data_sz);\n+\tif (!ASSERT_OK_PTR(init_data, \"bpf_map__initial_value\"))\n+\t\tgoto out;\n+\n+\tASSERT_EQ(init_data-\u003edata, init_value.data, \"init_value data\");\n+\tASSERT_EQ(init_data-\u003eset, init_value.set, \"init_value set\");\n+\tASSERT_EQ(init_data-\u003estruct_data.i, init_value.struct_data.i, \"init_value struct_data.i\");\n+\tASSERT_EQ(init_data-\u003estruct_data.nums[6], init_value.struct_data.nums[6],\n+\t\t  \"init_value struct_data.nums[6]\");\n+\tASSERT_EQ(init_data_sz, sizeof(init_value), \"init_value size\");\n+\tASSERT_EQ((void *) init_data, (void *) skel-\u003epercpu, \"skel-\u003epercpu eq init_data\");\n+\tASSERT_EQ(skel-\u003epercpu-\u003edata, init_value.data, \"skel-\u003epercpu-\u003edata\");\n+\tASSERT_EQ(skel-\u003epercpu-\u003eset, init_value.set, \"skel-\u003epercpu-\u003eset\");\n+\tASSERT_EQ(skel-\u003epercpu-\u003estruct_data.i, init_value.struct_data.i,\n+\t\t  \"skel-\u003epercpu-\u003estruct_data.i\");\n+\tASSERT_EQ(skel-\u003epercpu-\u003estruct_data.nums[6], init_value.struct_data.nums[6],\n+\t\t  \"skel-\u003epercpu-\u003estruct_data.nums[6]\");\n+\n+\tASSERT_GT(desired_sz, sizeof(init_value), \"desired_sz\");\n+\terr = bpf_map__set_value_size(map, desired_sz);\n+\tif (!ASSERT_OK(err, \"bpf_map__set_value_size\"))\n+\t\tgoto out;\n+\tif (!ASSERT_EQ(bpf_map__value_size(map), desired_sz, \"percpu value size\"))\n+\t\tgoto out;\n+\tif (!ASSERT_NEQ(bpf_map__btf_value_type_id(map), 0, \"percpu BTF value type\"))\n+\t\tgoto out;\n+\n+\tinit_data = bpf_map__initial_value(map, \u0026init_data_sz);\n+\tif (!ASSERT_OK_PTR(init_data, \"resized bpf_map__initial_value\"))\n+\t\tgoto out;\n+\tif (!ASSERT_EQ(init_data_sz, desired_sz, \"resized initial value size\"))\n+\t\tgoto out;\n+\tif (!ASSERT_EQ(init_data-\u003edata, init_value.data, \"resized initial value data\"))\n+\t\tgoto out;\n+\n+\terr = test_global_percpu_data__load(skel);\n+\tif (!ASSERT_OK(err, \"test_global_percpu_data__load\"))\n+\t\tgoto out;\n+\n+\tASSERT_OK_PTR(skel-\u003epercpu, \"skel-\u003epercpu\");\n+\n+\tprog_fd = bpf_program__fd(skel-\u003eprogs.update_percpu_data);\n+\ttest_percpu_data_on_cpus(map, bpf_map__fd(map), prog_fd, \u0026skel-\u003ebss-\u003erun);\n+\n+out:\n+\ttest_global_percpu_data__destroy(skel);\n+}\n+\n+static void test_global_percpu_data_lskel(void)\n+{\n+\tstruct test_global_percpu_data_lskel *lskel = NULL;\n+\tint prog_fd, map_fd;\n+\n+\tlskel = test_global_percpu_data_lskel__open_and_load();\n+\tif (!ASSERT_OK_PTR(lskel, \"test_global_percpu_data_lskel__open_and_load\"))\n+\t\tgoto out;\n+\n+\tmap_fd = lskel-\u003emaps.percpu.map_fd;\n+\tprog_fd = lskel-\u003eprogs.update_percpu_data.prog_fd;\n+\ttest_percpu_data_on_cpus(NULL, map_fd, prog_fd, \u0026lskel-\u003ebss-\u003erun);\n+\n+out:\n+\ttest_global_percpu_data_lskel__destroy(lskel);\n+}\n+\n+static int create_rdonly_percpu_array(void)\n+{\n+\tLIBBPF_OPTS(bpf_map_create_opts, map_opts,\n+\t\t    .map_flags = BPF_F_RDONLY_PROG,\n+\t);\n+\tint key = 0, map_fd, err;\n+\t__u64 value = 0;\n+\n+\tmap_fd = bpf_map_create(BPF_MAP_TYPE_PERCPU_ARRAY, \"percpu_ro_map\", sizeof(int),\n+\t\t\t\tsizeof(__u64), 1, \u0026map_opts);\n+\tif (!ASSERT_GE(map_fd, 0, \"bpf_map_create\"))\n+\t\treturn -1;\n+\n+\terr = bpf_map_update_elem(map_fd, \u0026key, \u0026value, BPF_F_ALL_CPUS);\n+\tif (!ASSERT_OK(err, \"bpf_map_update_elem\"))\n+\t\tgoto out;\n+\n+\terr = bpf_map_freeze(map_fd);\n+\tif (!ASSERT_OK(err, \"bpf_map_freeze\"))\n+\t\tgoto out;\n+\n+\treturn map_fd;\n+\n+out:\n+\tclose(map_fd);\n+\treturn -1;\n+}\n+\n+static void test_global_percpu_data_rdonly_direct_read(void)\n+{\n+\t/*\n+\t * Raw instructions with manually prepared rdonly percpu_array map\n+\t * for testing direct-read global percpu data, because libbpf\n+\t * doesn't have rdonly internal percpu_array map support for\n+\t * global percpu data.\n+\t */\n+\tstruct bpf_insn insns[] = {\n+\t\tBPF_LD_MAP_VALUE(BPF_REG_1, 0, 0),\n+\t\tBPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, 0),\n+\t\tBPF_EXIT_INSN(),\n+\t};\n+\tint map_fd, prog_fd;\n+\n+\tmap_fd = create_rdonly_percpu_array();\n+\tif (map_fd \u003c 0)\n+\t\treturn;\n+\n+\tinsns[0].imm = map_fd;\n+\tprog_fd = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, \"percpu_ro_prog\", \"GPL\", insns,\n+\t\t\t\tARRAY_SIZE(insns), NULL);\n+\tif (ASSERT_GE(prog_fd, 0, \"bpf_prog_load\"))\n+\t\tclose(prog_fd);\n+\tclose(map_fd);\n+}\n+\n+static void test_global_percpu_data_rdonly_direct_write(void)\n+{\n+\tLIBBPF_OPTS(bpf_prog_load_opts, prog_opts);\n+\t/* See the comment in test_global_percpu_data_rdonly_direct_read() */\n+\tstruct bpf_insn insns[] = {\n+\t\tBPF_LD_MAP_VALUE(BPF_REG_1, 0, 0),\n+\t\tBPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, 0),\n+\t\tBPF_ST_MEM(BPF_DW, BPF_REG_1, 0, 0),\n+\t\tBPF_EXIT_INSN(),\n+\t};\n+\tchar log_buf[256] = {};\n+\tint map_fd, prog_fd;\n+\n+\tprog_opts.log_buf = log_buf;\n+\tprog_opts.log_size = sizeof(log_buf);\n+\tprog_opts.log_level = 1;\n+\n+\tmap_fd = create_rdonly_percpu_array();\n+\tif (map_fd \u003c 0)\n+\t\treturn;\n+\n+\tinsns[0].imm = map_fd;\n+\tprog_fd = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, \"percpu_ro_prog\", \"GPL\", insns,\n+\t\t\t\tARRAY_SIZE(insns), \u0026prog_opts);\n+\tif (!ASSERT_LT(prog_fd, 0, \"bpf_prog_load\"))\n+\t\tclose(prog_fd);\n+\telse\n+\t\tASSERT_HAS_SUBSTR(log_buf, \"write into map forbidden\", \"verifier log\");\n+\tclose(map_fd);\n+}\n+\n+static void test_global_percpu_data_verifier_log(void)\n+{\n+\tRUN_TESTS(test_global_percpu_data);\n+}\n+\n+static void test_global_percpu_data_iter(void)\n+{\n+\tDECLARE_LIBBPF_OPTS(bpf_iter_attach_opts, opts);\n+\tstruct test_global_percpu_data *skel;\n+\tunion bpf_iter_link_info linfo = {};\n+\tstruct bpf_link *link = NULL;\n+\tint fd, num_cpus, len, err;\n+\tchar buf[16];\n+\n+\tnum_cpus = libbpf_num_possible_cpus();\n+\tif (!ASSERT_GT(num_cpus, 0, \"libbpf_num_possible_cpus\"))\n+\t\treturn;\n+\n+\tskel = test_global_percpu_data__open();\n+\tif (!ASSERT_OK_PTR(skel, \"test_global_percpu_data__open\"))\n+\t\treturn;\n+\n+\tskel-\u003erodata-\u003enum_cpus = num_cpus;\n+\tskel-\u003erodata-\u003eoffsetof_num = offsetof(struct test_global_percpu_data__percpu, struct_data);\n+\tskel-\u003erodata-\u003eoffsetof_num += sizeof(skel-\u003epercpu-\u003estruct_data) - sizeof(int);\n+\tskel-\u003erodata-\u003eelem_sz = roundup(sizeof(struct test_global_percpu_data__percpu), 8);\n+\tskel-\u003epercpu-\u003estruct_data.nums[6] = 0xc0de;\n+\n+\terr = test_global_percpu_data__load(skel);\n+\tif (!ASSERT_OK(err, \"test_global_percpu_data__load\"))\n+\t\tgoto out;\n+\n+\tlinfo.map.map_fd = bpf_map__fd(skel-\u003emaps.percpu);\n+\topts.link_info = \u0026linfo;\n+\topts.link_info_len = sizeof(linfo);\n+\tlink = bpf_program__attach_iter(skel-\u003eprogs.dump_percpu_data, \u0026opts);\n+\tif (!ASSERT_OK_PTR(link, \"bpf_program__attach_iter\"))\n+\t\tgoto out;\n+\n+\tfd = bpf_iter_create(bpf_link__fd(link));\n+\tif (!ASSERT_GE(fd, 0, \"bpf_iter_create\"))\n+\t\tgoto out;\n+\n+\twhile ((len = read(fd, buf, sizeof(buf))) \u003e 0)\n+\t\tdo { } while (0);\n+\tASSERT_EQ(len, 0, \"read iter\");\n+\tASSERT_TRUE(skel-\u003ebss-\u003erun_iter, \"run_iter\");\n+\tASSERT_EQ(skel-\u003ebss-\u003epercpu_data_sum, 0xc0de * num_cpus, \"percpu_data_sum\");\n+\n+\tclose(fd);\n+out:\n+\tbpf_link__destroy(link);\n+\ttest_global_percpu_data__destroy(skel);\n+}\n+\n+void test_global_percpu_data(void)\n+{\n+\tif (!feat_supported(NULL, FEAT_PERCPU_DATA)) {\n+\t\ttest__skip();\n+\t\treturn;\n+\t}\n+\n+\tif (test__start_subtest(\"init\"))\n+\t\ttest_global_percpu_data_init();\n+\tif (test__start_subtest(\"lskel\"))\n+\t\ttest_global_percpu_data_lskel();\n+\tif (test__start_subtest(\"rdonly_direct_read\"))\n+\t\ttest_global_percpu_data_rdonly_direct_read();\n+\tif (test__start_subtest(\"rdonly_direct_write\"))\n+\t\ttest_global_percpu_data_rdonly_direct_write();\n+\ttest_global_percpu_data_verifier_log();\n+\tif (test__start_subtest(\"iter\"))\n+\t\ttest_global_percpu_data_iter();\n+}\ndiff --git a/tools/testing/selftests/bpf/progs/test_global_percpu_data.c b/tools/testing/selftests/bpf/progs/test_global_percpu_data.c\nnew file mode 100644\nindex 0000000000000..71ff8d1bf49eb\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/progs/test_global_percpu_data.c\n@@ -0,0 +1,89 @@\n+// SPDX-License-Identifier: GPL-2.0\n+#include \u003cvmlinux.h\u003e\n+#include \u003cbpf/bpf_helpers.h\u003e\n+#include \"bpf_misc.h\"\n+\n+/* Used for testing map name. */\n+int loong SEC(\".percpu.looooooooong\");\n+int data3 SEC(\".data.percpu\");\n+int data2 SEC(\".percpu.data\");\n+\n+int run;\n+/* cpu_id as array to verify map value resizing. */\n+int cpu_id[1] SEC(\".percpu\");\n+int data SEC(\".percpu\") = -1;\n+int nums[7] SEC(\".percpu\");\n+bool set SEC(\".percpu\") = false;\n+struct {\n+\tchar set;\n+\tint i;\n+\tint nums[7];\n+} struct_data SEC(\".percpu\") = {\n+\t.set = 0,\n+\t.i = -1,\n+};\n+\n+SEC(\"raw_tp/task_rename\")\n+__auxiliary\n+int update_percpu_data(void *ctx)\n+{\n+\tstruct_data.nums[6] = 0xc0de;\n+\tstruct_data.set = 1;\n+\tstruct_data.i = 1;\n+\tnums[6] = 0xc0de;\n+\tdata = 1;\n+\trun++;\n+\tset = true;\n+\tcpu_id[0] = bpf_get_smp_processor_id();\n+\treturn 0;\n+}\n+\n+static const char fmt[] SEC(\".percpu.fmt\") = \"data %d\\n\";\n+\n+SEC(\"?kprobe\")\n+__failure __msg(\"R{{[0-9]+}} points to percpu_array map which cannot be used as const string\")\n+int verifier_strncmp(void *ctx)\n+{\n+\treturn bpf_strncmp(\"test\", 5, fmt);\n+}\n+\n+SEC(\"?kprobe\")\n+__failure __msg(\"R{{[0-9]+}} points to percpu_array map which cannot be used as const string\")\n+int verifier_snprintf(void *ctx)\n+{\n+\tu64 args[] = { data };\n+\tchar buf[128];\n+\tint len;\n+\n+\tlen = bpf_snprintf(buf, sizeof(buf), fmt, args, sizeof(args));\n+\tif (len \u003e 0)\n+\t\tbpf_printk(\"snprintf: %s\\n\", buf);\n+\treturn 0;\n+}\n+\n+volatile const __u32 num_cpus = 0;\n+volatile const int offsetof_num;\n+volatile const int elem_sz;\n+__u32 percpu_data_sum = 0;\n+bool run_iter = false;\n+\n+SEC(\"iter/bpf_map_elem\")\n+__auxiliary\n+int dump_percpu_data(struct bpf_iter__bpf_map_elem *ctx)\n+{\n+\tvoid *pptr = ctx-\u003evalue;\n+\tint i;\n+\n+\tif (!pptr)\n+\t\treturn 0;\n+\n+\trun_iter = true;\n+\n+\tfor (i = 0; i \u003c num_cpus; i++) {\n+\t\tpercpu_data_sum += *(int *) (pptr + offsetof_num);\n+\t\tpptr += elem_sz;\n+\t}\n+\treturn 0;\n+}\n+\n+char _license[] SEC(\"license\") = \"GPL\";\n"
}

{
  "EnableConfigs": [
    "BPF_SYSCALL"
  ],
  "FocusSymbols": [
    "bpf_do_misc_fixups",
    "bpf_map_direct_read",
    "percpu_array_map_direct_value_addr",
    "percpu_array_map_direct_value_meta"
  ],
  "Reasoning": "The patch introduces support for global percpu data in BPF programs by allowing `.percpu` sections to be backed by `BPF_MAP_TYPE_PERCPU_ARRAY`. It modifies the BPF verifier to handle direct memory reads from percpu arrays and updates JIT fixups to support direct loading of percpu variables. These changes affect core BPF verifier logic and map operations, which are reachable from user space via the `bpf()` syscall when loading and executing BPF programs.",
  "WorthFuzzing": true
}

You are an expert Linux kernel maintainer.
Your job is to review a provided patch series and determine
if it makes functional changes to the kernel that should be fuzzed.

IMPORTANT: The changes have ALREADY been applied and committed as the HEAD commit in
your workspace. Do NOT rely on your internal knowledge of the kernel. You must actively
use your code access tools to examine the actual source code and confirm any assumptions.

Return WorthFuzzing=false if the patch only contains:
- Modifications to Documentation/, Kconfig files, or code comments.
- Purely decorative changes, such as logging (e.g., pr_err, printk) or tracepoints.
- Changes to numeric constants or macros that do not functionally alter execution flow.
- Code paths that are impossible to reach in virtualized environments like GCE or QEMU,
  even when utilizing software-emulated hardware (e.g., usb gadget, mac80211_hwsim).
- Code in vendor-specific PCIe switch, SmartNIC, or GPU drivers (e.g., mlxsw, pds_core, qed,
  ionic, amdgpu) that require physical PCIe hardware cards not emulated in standard QEMU.
- Driver .remove, .shutdown, or pci_unregister_driver teardown callbacks (e.g., igb_remove)
  that are executed only during PCI hot-unplug or sysfs driver unbind operations.

If it modifies reachable core kernel logic, drivers, or architectures, use your code search
tools to verify the code can be executed, then return WorthFuzzing=true.

When returning WorthFuzzing=true, you MUST ALSO:
1. Extract any specific kernel functions that should be heavily fuzzed into FocusSymbols.
   Avoid listing generic hot-path functions to prevent skewed test distributions.
   Prefer non-static, non-inlined API entrypoint functions over internal static helper functions
   (which are inlined by the compiler and do not have distinct symbol addresses).
2. Identify any specific CONFIG_ options required to properly test this new/modified feature.
   Go and look into the Kconfig files and check for ifdefs around the code, do not make assumptions.
   Also check "depends on" lines in Kconfig to include any non-standard parent subsystem configs
   needed for Kbuild to compile the code statically into vmlinux. List them in the EnableConfigs
   output array, and DO NOT add a 'CONFIG_' prefix (e.g., return "NET_IPV4" instead of "CONFIG_NET_IPV4").
Prefer calling several tools at the same time to save round-trips.


Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.

For your convenience, here is the diff of the changes:
commit 3b76f69b75942f2595d46bc813574f1caa6e529f
Author: syz-cluster <triage@syzkaller.com>
Date:   Thu Aug 13 16:51:17 2026 +0000

    syz-cluster: applied patch under review

diff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c
index 248b4818178cd..34865701f7f75 100644
--- a/kernel/bpf/arraymap.c
+++ b/kernel/bpf/arraymap.c
@@ -259,6 +259,37 @@ static void *percpu_array_map_lookup_elem(struct bpf_map *map, void *key)
 	return this_cpu_ptr(array->pptrs[index & array->index_mask]);
 }
 
+static int percpu_array_map_direct_value_addr(const struct bpf_map *map, u64 *imm, u32 off)
+{
+	struct bpf_array *array = container_of(map, struct bpf_array, map);
+
+	if (!bpf_jit_supports_percpu_insn())
+		return -EOPNOTSUPP;
+	if (map->max_entries != 1)
+		return -EOPNOTSUPP;
+	if (off >= map->value_size)
+		return -EINVAL;
+
+	*imm = (u64)(__force unsigned long) array->pptrs[0];
+	return 0;
+}
+
+static int percpu_array_map_direct_value_meta(const struct bpf_map *map, u64 imm, u32 *off)
+{
+	struct bpf_array *array = container_of(map, struct bpf_array, map);
+	u64 base = (u64)(__force unsigned long) array->pptrs[0];
+
+	if (!bpf_jit_supports_percpu_insn())
+		return -EOPNOTSUPP;
+	if (map->max_entries != 1)
+		return -EOPNOTSUPP;
+	if (imm < base || imm >= base + array->elem_size)
+		return -ENOENT;
+
+	*off = imm - base;
+	return 0;
+}
+
 /* emit BPF instructions equivalent to C code of percpu_array_map_lookup_elem() */
 static int percpu_array_map_gen_lookup(struct bpf_map *map, struct bpf_insn *insn_buf)
 {
@@ -551,9 +582,10 @@ static int array_map_check_btf(struct bpf_map *map,
 			       const struct btf_type *key_type,
 			       const struct btf_type *value_type)
 {
-	/* One exception for keyless BTF: .bss/.data/.rodata map */
+	/* One exception for keyless BTF: .bss/.data/.rodata/.percpu map */
 	if (btf_type_is_void(key_type)) {
-		if (map->map_type != BPF_MAP_TYPE_ARRAY ||
+		if ((map->map_type != BPF_MAP_TYPE_ARRAY &&
+		     map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY) ||
 		    map->max_entries != 1)
 			return -EINVAL;
 
@@ -832,6 +864,8 @@ const struct bpf_map_ops percpu_array_map_ops = {
 	.map_get_next_key = bpf_array_get_next_key,
 	.map_lookup_elem = percpu_array_map_lookup_elem,
 	.map_gen_lookup = percpu_array_map_gen_lookup,
+	.map_direct_value_addr = percpu_array_map_direct_value_addr,
+	.map_direct_value_meta = percpu_array_map_direct_value_meta,
 	.map_update_elem = array_map_update_elem,
 	.map_delete_elem = array_map_delete_elem,
 	.map_lookup_percpu_elem = percpu_array_map_lookup_percpu_elem,
diff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c
index 40bd04421a991..a2b18a9f1694c 100644
--- a/kernel/bpf/backtrack.c
+++ b/kernel/bpf/backtrack.c
@@ -214,7 +214,6 @@ static inline bool bt_is_reg_set(struct backtrack_state *bt, u32 reg)
 	return bt->reg_masks[bt->frame] & (1 << reg);
 }
 
-
 /* format registers bitmask, e.g., "r0,r2,r4" for 0x15 mask */
 static void fmt_reg_mask(char *buf, ssize_t buf_sz, u32 reg_mask)
 {
@@ -254,7 +253,6 @@ void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask)
 	}
 }
 
-
 /* For given verifier state backtrack_insn() is called from the last insn to
  * the first insn. Its purpose is to compute a bitmask of registers and
  * stack slots that needs precision in the parent verifier state.
diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index 6606187ed4f43..87ffde865a503 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -2534,7 +2534,6 @@ static void btf_bitfield_show(void *data, u8 bits_offset,
 	btf_int128_print(show, print_num);
 }
 
-
 static void btf_int_bits_show(const struct btf *btf,
 			      const struct btf_type *t,
 			      void *data, u8 bits_offset,
diff --git a/kernel/bpf/cfg.c b/kernel/bpf/cfg.c
index db3416a7c9047..818f7afac83a5 100644
--- a/kernel/bpf/cfg.c
+++ b/kernel/bpf/cfg.c
@@ -47,7 +47,6 @@ enum {
 	BRANCH = 2,
 };
 
-
 static void mark_subprog_changes_pkt_data(struct bpf_verifier_env *env, int off)
 {
 	struct bpf_subprog_info *subprog;
diff --git a/kernel/bpf/const_fold.c b/kernel/bpf/const_fold.c
index 4cf120c7b2cb4..7f1b30059cc87 100644
--- a/kernel/bpf/const_fold.c
+++ b/kernel/bpf/const_fold.c
@@ -182,7 +182,6 @@ static void const_reg_xfer(struct bpf_verifier_env *env, struct const_arg_info *
 		u64 val = 0;
 
 		if (!bpf_map_is_rdonly(map) || !map->ops->map_direct_value_addr ||
-		    map->map_type == BPF_MAP_TYPE_INSN_ARRAY ||
 		    off < 0 || off + size > map->value_size ||
 		    bpf_map_direct_read(map, off, size, &val, is_ldsx)) {
 			*dst = unknown;
diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
index 2417a3461652d..177a3fcbb63ac 100644
--- a/kernel/bpf/fixups.c
+++ b/kernel/bpf/fixups.c
@@ -1466,7 +1466,6 @@ int bpf_fixup_call_args(struct bpf_verifier_env *env)
 	return err;
 }
 
-
 /* The function requires that first instruction in 'patch' is insnsi[prog->len - 1] */
 static int add_hidden_subprog(struct bpf_verifier_env *env, struct bpf_insn *patch, int len)
 {
@@ -1835,6 +1834,43 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)
 			goto next_insn;
 		}
 
+		if (bpf_jit_supports_percpu_insn() &&
+		    insn->code == (BPF_LD | BPF_IMM | BPF_DW) &&
+		    (insn->src_reg == BPF_PSEUDO_MAP_VALUE ||
+		     insn->src_reg == BPF_PSEUDO_MAP_IDX_VALUE)) {
+			struct bpf_map *map;
+
+			aux = &env->insn_aux_data[i + delta];
+			map = env->used_maps[aux->map_index];
+			if (map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY)
+				goto next_insn;
+
+			prog->jit_required = true;
+
+			/*
+			 * We are *skipping* first half of ld_imm64 insn
+			 * with 'i++;', patching over second half of it
+			 * with that same half + mov64_percpu_reg insn.
+			 * All because bpf_patch_insn_data() can only
+			 * replace one 8-byte insn, which does not work
+			 * well for ld_imm64 insn.
+			 */
+
+			insn_buf[0] = insn[1];
+			insn_buf[1] = BPF_MOV64_PERCPU_REG(insn->dst_reg, insn->dst_reg);
+			cnt = 2;
+
+			i++;
+			new_prog = bpf_patch_insn_data(env, i + delta, insn_buf, cnt);
+			if (!new_prog)
+				return -ENOMEM;
+
+			delta    += cnt - 1;
+			env->prog = prog = new_prog;
+			insn      = new_prog->insnsi + i + delta;
+			goto next_insn;
+		}
+
 		if (insn->code != (BPF_JMP | BPF_CALL))
 			goto next_insn;
 		if (insn->src_reg == BPF_PSEUDO_CALL)
diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c
index 9f394e1aa2e85..d40cb5dd446ca 100644
--- a/kernel/bpf/hashtab.c
+++ b/kernel/bpf/hashtab.c
@@ -998,7 +998,6 @@ static void dec_elem_count(struct bpf_htab *htab)
 		atomic_dec(&htab->count);
 }
 
-
 static void free_htab_elem(struct bpf_htab *htab, struct htab_elem *l)
 {
 	htab_put_fd_value(htab, l);
@@ -2970,7 +2969,6 @@ static int rhtab_delete_elem(struct bpf_rhtab *rhtab, struct rhtab_elem *elem, v
 	return 0;
 }
 
-
 static long rhtab_map_delete_elem(struct bpf_map *map, void *key)
 {
 	struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map);
diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c
index 6388b6b23e490..45e2f19387b24 100644
--- a/kernel/bpf/helpers.c
+++ b/kernel/bpf/helpers.c
@@ -4871,7 +4871,6 @@ static const struct btf_kfunc_id_set generic_kfunc_set = {
 	.set   = &generic_btf_ids,
 };
 
-
 BTF_ID_LIST(generic_dtor_ids)
 BTF_ID(struct, task_struct)
 BTF_ID(func, bpf_task_release_dtor)
diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index 1c997aeba6fa5..74fc4b3f80d6e 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -269,7 +269,6 @@ bpf_insn_successors(struct bpf_verifier_env *env, u32 idx)
 
 __diag_pop();
 
-
 static inline bool update_insn(struct bpf_verifier_env *env,
 			       struct func_instance *instance, u32 frame, u32 insn_idx)
 {
@@ -1862,7 +1861,6 @@ static int analyze_subprog(struct bpf_verifier_env *env,
 	if (need_resched())
 		cond_resched();
 
-
 	/*
 	 * When an instance is reused (must_write_initialized == true),
 	 * record into a fresh instance and merge afterward.  This avoids
diff --git a/kernel/bpf/queue_stack_maps.c b/kernel/bpf/queue_stack_maps.c
index c1c9dee4dcdd0..6e8b18c32a106 100644
--- a/kernel/bpf/queue_stack_maps.c
+++ b/kernel/bpf/queue_stack_maps.c
@@ -123,7 +123,6 @@ static long __queue_map_get(struct bpf_map *map, void *value, bool delete)
 	return err;
 }
 
-
 static long __stack_map_get(struct bpf_map *map, void *value, bool delete)
 {
 	struct bpf_queue_stack *qs = bpf_queue_stack(map);
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 8d111da886553..7d8c3e8e6d62e 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -636,7 +636,6 @@ int bpf_map_alloc_pages(const struct bpf_map *map, int nid,
 	return ret;
 }
 
-
 static int btf_field_cmp(const void *a, const void *b)
 {
 	const struct btf_field *f1 = a, *f2 = b;
@@ -1830,7 +1829,6 @@ static int map_lookup_elem(union bpf_attr *attr)
 	return err;
 }
 
-
 #define BPF_MAP_UPDATE_ELEM_LAST_FIELD flags
 
 static int map_update_elem(union bpf_attr *attr, bpfptr_t uattr)
@@ -3497,7 +3495,6 @@ int bpf_link_prime(struct bpf_link *link, struct bpf_link_primer *primer)
 	if (fd < 0)
 		return fd;
 
-
 	id = bpf_link_alloc_id(link);
 	if (id < 0) {
 		put_unused_fd(fd);
@@ -5505,7 +5502,6 @@ static int bpf_link_get_info_by_fd(struct file *file,
 	return 0;
 }
 
-
 static int token_get_info_by_fd(struct file *file,
 				struct bpf_token *token,
 				const union bpf_attr *attr,
@@ -6507,7 +6503,6 @@ BPF_CALL_3(bpf_sys_bpf, int, cmd, union bpf_attr *, attr, u32, attr_size)
 	return __sys_bpf(cmd, KERNEL_BPFPTR(attr), attr_size, KERNEL_BPFPTR(NULL), 0);
 }
 
-
 /* To shut up -Wmissing-prototypes.
  * This function is used by the kernel light skeleton
  * to load bpf programs when modules are loaded or during kernel boot.
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 164d16c243ca6..6ac1afced20bf 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -635,7 +635,6 @@ static void __mark_dynptr_reg(struct bpf_reg_state *reg,
 			      enum bpf_dynptr_type type,
 			      bool first_slot, int id, int parent_id);
 
-
 static void mark_dynptr_stack_regs(struct bpf_verifier_env *env,
 				   struct bpf_reg_state *sreg1,
 				   struct bpf_reg_state *sreg2,
@@ -1674,7 +1673,6 @@ static bool same_callsites(struct bpf_verifier_state *a, struct bpf_verifier_sta
 	return true;
 }
 
-
 void bpf_free_backedges(struct bpf_scc_visit *visit)
 {
 	struct bpf_scc_backedge *backedge, *next;
@@ -2291,7 +2289,6 @@ static struct bpf_verifier_state *push_async_cb(struct bpf_verifier_env *env,
 	return &elem->st;
 }
 
-
 static int cmp_subprogs(const void *a, const void *b)
 {
 	return ((struct bpf_subprog_info *)a)->start -
@@ -3969,7 +3966,6 @@ static int check_stack_read(struct bpf_verifier_env *env,
 	return err;
 }
 
-
 /* check_stack_write dispatches to check_stack_write_fixed_off or
  * check_stack_write_var_off.
  *
@@ -4767,7 +4763,6 @@ static int check_sock_access(struct bpf_verifier_env *env, int insn_idx,
 		valid = false;
 	}
 
-
 	if (valid) {
 		env->insn_aux_data[insn_idx].ctx_field_size =
 			info.ctx_field_size;
@@ -5587,6 +5582,8 @@ int bpf_map_direct_read(struct bpf_map *map, int off, int size, u64 *val,
 	u64 addr;
 	int err;
 
+	if (map->map_type == BPF_MAP_TYPE_INSN_ARRAY || map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY)
+		return -EINVAL;
 	err = map->ops->map_direct_value_addr(map, &addr, off);
 	if (err)
 		return err;
@@ -6083,6 +6080,51 @@ static void add_scalar_to_reg(struct bpf_reg_state *dst_reg, s64 val)
 	reg_bounds_sync(dst_reg);
 }
 
+static int check_map_mem_read(struct bpf_verifier_env *env, struct bpf_reg_state *reg, int off,
+			      int bpf_size, int value_regno, bool is_ldsx)
+{
+	struct bpf_reg_state *regs = cur_regs(env);
+	int size = bpf_size_to_bytes(bpf_size);
+	struct bpf_map *map = reg->map_ptr;
+
+	switch (map->map_type) {
+	case BPF_MAP_TYPE_INSN_ARRAY:
+		if (bpf_size != BPF_DW) {
+			verbose(env, "Invalid read of %d bytes from insn_array\n", size);
+			return -EACCES;
+		}
+		regs[value_regno] = *reg;
+		add_scalar_to_reg(&regs[value_regno], off);
+		regs[value_regno].type = PTR_TO_INSN;
+		return 0;
+	case BPF_MAP_TYPE_PERCPU_ARRAY:
+		goto reg_unknown;
+	default:
+		break;
+	}
+
+	/* If map is read-only, track its contents as scalars. */
+	if (tnum_is_const(reg->var_off) &&
+	    bpf_map_is_rdonly(map) &&
+	    map->ops->map_direct_value_addr) {
+		int map_off = off + reg->var_off.value;
+		u64 val = 0;
+		int err;
+
+		err = bpf_map_direct_read(map, map_off, size, &val, is_ldsx);
+		if (err)
+			return err;
+
+		regs[value_regno].type = SCALAR_VALUE;
+		__mark_reg_known(&regs[value_regno], val);
+		return 0;
+	}
+
+reg_unknown:
+	mark_reg_unknown(env, regs, value_regno);
+	return 0;
+}
+
 /* check whether memory at (regno + off) is accessible for t = (read | write)
  * if t==write, value_regno is a register which value is stored into memory
  * if t==read, value_regno is a register which will receive the value from memory
@@ -6137,38 +6179,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, struct b
 		if (kptr_field) {
 			err = check_map_kptr_access(env, value_regno, insn_idx, kptr_field);
 		} else if (t == BPF_READ && value_regno >= 0) {
-			struct bpf_map *map = reg->map_ptr;
-
-			/*
-			 * If map is read-only, track its contents as scalars,
-			 * unless it is an insn array (see the special case below)
-			 */
-			if (tnum_is_const(reg->var_off) &&
-			    bpf_map_is_rdonly(map) &&
-			    map->ops->map_direct_value_addr &&
-			    map->map_type != BPF_MAP_TYPE_INSN_ARRAY) {
-				int map_off = off + reg->var_off.value;
-				u64 val = 0;
-
-				err = bpf_map_direct_read(map, map_off, size,
-							  &val, is_ldsx);
-				if (err)
-					return err;
-
-				regs[value_regno].type = SCALAR_VALUE;
-				__mark_reg_known(&regs[value_regno], val);
-			} else if (map->map_type == BPF_MAP_TYPE_INSN_ARRAY) {
-				if (bpf_size != BPF_DW) {
-					verbose(env, "Invalid read of %d bytes from insn_array\n",
-						     size);
-					return -EACCES;
-				}
-				regs[value_regno] = *reg;
-				add_scalar_to_reg(&regs[value_regno], off);
-				regs[value_regno].type = PTR_TO_INSN;
-			} else {
-				mark_reg_unknown(env, regs, value_regno);
-			}
+			err = check_map_mem_read(env, reg, off, bpf_size, value_regno, is_ldsx);
 		}
 	} else if (base_type(reg->type) == PTR_TO_MEM) {
 		bool rdonly_mem = type_is_rdonly_mem(reg->type);
@@ -6635,7 +6646,6 @@ static int check_stack_range_initialized(
 	if (err)
 		return err;
 
-
 	if (tnum_is_const(reg->var_off)) {
 		min_off = max_off = reg->var_off.value + off;
 	} else {
@@ -7347,7 +7357,6 @@ static bool is_iter_new_kfunc(struct bpf_call_arg_meta *meta)
 	return meta->kfunc_flags & KF_ITER_NEW;
 }
 
-
 static bool is_iter_destroy_kfunc(struct bpf_call_arg_meta *meta)
 {
 	return meta->kfunc_flags & KF_ITER_DESTROY;
@@ -8125,6 +8134,12 @@ static int check_arg_const_str(struct bpf_verifier_env *env,
 		return -EACCES;
 	}
 
+	if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY) {
+		verbose(env, "%s points to percpu_array map which cannot be used as const string\n",
+			reg_arg_name(env, argno));
+		return -EACCES;
+	}
+
 	if (!bpf_map_is_rdonly(map)) {
 		verbose(env, "%s does not point to a readonly map'\n", reg_arg_name(env, argno));
 		return -EACCES;
@@ -11607,7 +11622,6 @@ static int process_irq_flag(struct bpf_verifier_env *env, struct bpf_reg_state *
 	return 0;
 }
 
-
 static int ref_set_non_owning(struct bpf_verifier_env *env, struct bpf_reg_state *reg)
 {
 	struct btf_record *rec = reg_btf_record(reg);
@@ -16412,7 +16426,6 @@ static int check_ld_abs(struct bpf_verifier_env *env, struct bpf_insn *insn)
 	return 0;
 }
 
-
 static bool return_retval_range(struct bpf_verifier_env *env, struct bpf_retval_range *range)
 {
 	enum bpf_prog_type prog_type = resolve_prog_type(env->prog);
@@ -18361,8 +18374,6 @@ static void release_insn_arrays(struct bpf_verifier_env *env)
 		bpf_insn_array_release(env->insn_array_maps[i]);
 }
 
-
-
 /* The verifier does more data flow analysis than llvm and will not
  * explore branches that are dead at run time. Malicious programs can
  * have dead code too. Therefore replace all dead at-run-time code
@@ -18390,8 +18401,6 @@ static void sanitize_dead_code(struct bpf_verifier_env *env)
 	}
 }
 
-
-
 static void free_states(struct bpf_verifier_env *env)
 {
 	struct bpf_verifier_state_list *sl;
@@ -18678,7 +18687,6 @@ static int do_check_main(struct bpf_verifier_env *env)
 	return ret;
 }
 
-
 static void print_verification_stats(struct bpf_verifier_env *env)
 {
 	/* Skip over hidden subprogs which are not verified. */
diff --git a/tools/bpf/bpftool/gen.c b/tools/bpf/bpftool/gen.c
index a01d06d22d1a3..a50540ef6521c 100644
--- a/tools/bpf/bpftool/gen.c
+++ b/tools/bpf/bpftool/gen.c
@@ -101,6 +101,12 @@ static bool get_map_ident(const struct bpf_map *map, char *buf, size_t buf_sz)
 		return true;
 	}
 
+	if (bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY) {
+		snprintf(buf, buf_sz, "%s", name + 1);
+		sanitize_identifier(buf);
+		return true;
+	}
+
 	for  (i = 0, n = ARRAY_SIZE(sfxs); i < n; i++) {
 		const char *sfx = sfxs[i], *p;
 
@@ -117,7 +123,7 @@ static bool get_map_ident(const struct bpf_map *map, char *buf, size_t buf_sz)
 
 static bool get_datasec_ident(const char *sec_name, char *buf, size_t buf_sz)
 {
-	static const char *pfxs[] = { ".data", ".rodata", ".bss", ".kconfig" };
+	static const char *pfxs[] = { ".data", ".rodata", ".bss", ".percpu", ".kconfig" };
 	int i, n;
 
 	/* recognize hard coded LLVM section name */
@@ -254,7 +260,7 @@ static const struct btf_type *find_type_for_map(struct btf *btf, const char *map
 	return NULL;
 }
 
-static bool is_mmapable_map(const struct bpf_map *map, char *buf, size_t sz)
+static bool is_skel_data(const struct bpf_map *map, char *buf, size_t sz)
 {
 	size_t tmp_sz;
 
@@ -263,13 +269,24 @@ static bool is_mmapable_map(const struct bpf_map *map, char *buf, size_t sz)
 		return true;
 	}
 
-	if (!bpf_map__is_internal(map) || !(bpf_map__map_flags(map) & BPF_F_MMAPABLE))
+	if (!bpf_map__is_internal(map))
 		return false;
 
 	if (!get_map_ident(map, buf, sz))
 		return false;
 
-	return true;
+	if (bpf_map__map_flags(map) & BPF_F_MMAPABLE)
+		return true;
+
+	if (bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY)
+		return bpf_map__btf_value_type_id(map) != 0;
+
+	return false;
+}
+
+static bool is_mmapable_map(const struct bpf_map *map, char *buf, size_t sz)
+{
+	return is_skel_data(map, buf, sz) && bpf_map__type(map) != BPF_MAP_TYPE_PERCPU_ARRAY;
 }
 
 static int codegen_datasecs(struct bpf_object *obj, const char *obj_name)
@@ -287,7 +304,7 @@ static int codegen_datasecs(struct bpf_object *obj, const char *obj_name)
 
 	bpf_object__for_each_map(map, obj) {
 		/* only generate definitions for memory-mapped internal maps */
-		if (!is_mmapable_map(map, map_ident, sizeof(map_ident)))
+		if (!is_skel_data(map, map_ident, sizeof(map_ident)))
 			continue;
 
 		sec = find_type_for_map(btf, map_ident);
@@ -517,7 +534,7 @@ static void codegen_asserts(struct bpf_object *obj, const char *obj_name)
 		", obj_name);
 
 	bpf_object__for_each_map(map, obj) {
-		if (!is_mmapable_map(map, map_ident, sizeof(map_ident)))
+		if (!is_skel_data(map, map_ident, sizeof(map_ident)))
 			continue;
 
 		sec = find_type_for_map(btf, map_ident);
@@ -668,8 +685,7 @@ static void codegen_destroy(struct bpf_object *obj, const char *obj_name)
 	bpf_object__for_each_map(map, obj) {
 		if (!get_map_ident(map, ident, sizeof(ident)))
 			continue;
-		if (bpf_map__is_internal(map) &&
-		    (bpf_map__map_flags(map) & BPF_F_MMAPABLE))
+		if (is_skel_data(map, ident, sizeof(ident)))
 			printf("\tskel_free_map_data(skel->%1$s, skel->maps.%1$s.initial_value, %2$zu);\n",
 			       ident, bpf_map_mmap_sz(map));
 		codegen("\
@@ -741,7 +757,7 @@ static int gen_trace(struct bpf_object *obj, const char *obj_name, const char *h
 		const void *mmap_data = NULL;
 		size_t mmap_size = 0;
 
-		if (!is_mmapable_map(map, ident, sizeof(ident)))
+		if (!is_skel_data(map, ident, sizeof(ident)))
 			continue;
 
 		codegen("\
@@ -849,9 +865,23 @@ static int gen_trace(struct bpf_object *obj, const char *obj_name, const char *h
 	bpf_object__for_each_map(map, obj) {
 		const char *mmap_flags;
 
-		if (!is_mmapable_map(map, ident, sizeof(ident)))
+		if (!is_skel_data(map, ident, sizeof(ident)))
 			continue;
 
+		if (bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY) {
+			codegen("\
+		\n\
+			err = skel_protect_map_data(skel->%1$s, &skel->maps.%1$s.initial_value, %2$zd);\n\
+			if (err)					    \n\
+				return err;				    \n\
+		#ifdef __KERNEL__					    \n\
+			skel->%1$s = NULL;				    \n\
+		#endif							    \n\
+			",
+			ident, bpf_map_mmap_sz(map));
+			continue;
+		}
+
 		if (bpf_map__map_flags(map) & BPF_F_RDONLY_PROG)
 			mmap_flags = "PROT_READ";
 		else
@@ -955,8 +985,7 @@ codegen_maps_skeleton(struct bpf_object *obj, size_t map_cnt, bool mmaped, bool
 				map->map = &obj->maps.%s;	    \n\
 			",
 			i, bpf_map__name(map), ident);
-		/* memory-mapped internal maps */
-		if (mmaped && is_mmapable_map(map, ident, sizeof(ident))) {
+		if (mmaped && is_skel_data(map, ident, sizeof(ident))) {
 			printf("\tmap->mmaped = (void **)&obj->%s;\n", ident);
 		}
 
diff --git a/tools/lib/bpf/bpf_gen_internal.h b/tools/lib/bpf/bpf_gen_internal.h
index 0425691877521..6c5ad6c55e8a6 100644
--- a/tools/lib/bpf/bpf_gen_internal.h
+++ b/tools/lib/bpf/bpf_gen_internal.h
@@ -65,7 +65,8 @@ void bpf_gen__prog_load(struct bpf_gen *gen,
 			enum bpf_prog_type prog_type, const char *prog_name,
 			const char *license, struct bpf_insn *insns, size_t insn_cnt,
 			struct bpf_prog_load_opts *load_attr, int prog_idx);
-void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *value, __u32 value_size);
+void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *value, __u32 value_size,
+			      __u64 flags);
 void bpf_gen__map_freeze(struct bpf_gen *gen, int map_idx);
 void bpf_gen__record_attach_target(struct bpf_gen *gen, const char *name, enum bpf_attach_type type);
 void bpf_gen__record_extern(struct bpf_gen *gen, const char *name, bool is_weak,
diff --git a/tools/lib/bpf/features.c b/tools/lib/bpf/features.c
index b7e388f99d0bb..ef9581c113035 100644
--- a/tools/lib/bpf/features.c
+++ b/tools/lib/bpf/features.c
@@ -620,6 +620,38 @@ static int probe_bpf_syscall_common_attrs(int token_fd)
 	return probe_sys_bpf_ext();
 }
 
+static int probe_kern_percpu_data(int token_fd)
+{
+	struct bpf_insn insns[] = {
+		BPF_LD_MAP_VALUE(BPF_REG_1, 0, 0),
+		BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, 0),
+		BPF_EXIT_INSN(),
+	};
+	LIBBPF_OPTS(bpf_map_create_opts, map_opts,
+		.token_fd = token_fd,
+		.map_flags = token_fd ? BPF_F_TOKEN_FD : 0,
+	);
+	LIBBPF_OPTS(bpf_prog_load_opts, prog_opts,
+		.token_fd = token_fd,
+		.prog_flags = token_fd ? BPF_F_TOKEN_FD : 0,
+	);
+	int ret, map, insn_cnt = ARRAY_SIZE(insns);
+
+	map = bpf_map_create(BPF_MAP_TYPE_PERCPU_ARRAY, "libbpf_percpu", sizeof(int), 8, 1,
+			     &map_opts);
+	if (map < 0) {
+		pr_warn("Error in %s(): %s. Couldn't create simple percpu_array map.\n",
+			__func__, errstr(map));
+		return map;
+	}
+
+	insns[0].imm = map;
+
+	ret = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, NULL, "GPL", insns, insn_cnt, &prog_opts);
+	close(map);
+	return probe_fd(ret);
+}
+
 typedef int (*feature_probe_fn)(int /* token_fd */);
 
 static struct kern_feature_cache feature_cache;
@@ -707,6 +739,9 @@ static struct kern_feature_desc {
 	[FEAT_BPF_SYSCALL_COMMON_ATTRS] = {
 		"BPF syscall common attributes support", probe_bpf_syscall_common_attrs,
 	},
+	[FEAT_PERCPU_DATA] = {
+		"kernel supports percpu data", probe_kern_percpu_data,
+	},
 };
 
 bool feat_supported(struct kern_feature_cache *cache, enum kern_feature_id feat_id)
diff --git a/tools/lib/bpf/gen_loader.c b/tools/lib/bpf/gen_loader.c
index 6e3dd52427618..af3a04f161ac1 100644
--- a/tools/lib/bpf/gen_loader.c
+++ b/tools/lib/bpf/gen_loader.c
@@ -1128,7 +1128,7 @@ void bpf_gen__prog_load(struct bpf_gen *gen,
 }
 
 void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *pvalue,
-			      __u32 value_size)
+			      __u32 value_size, __u64 flags)
 {
 	int attr_size = offsetofend(union bpf_attr, flags);
 	int map_update_attr, value, key;
@@ -1136,6 +1136,7 @@ void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *pvalue,
 	int zero = 0;
 
 	memset(&attr, 0, attr_size);
+	attr.flags = tgt_endian(flags);
 
 	value = add_data(gen, pvalue, value_size);
 	key = add_data(gen, &zero, sizeof(zero));
diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
index 514e4e9daa825..e574870fb7169 100644
--- a/tools/lib/bpf/libbpf.c
+++ b/tools/lib/bpf/libbpf.c
@@ -541,6 +541,7 @@ struct bpf_struct_ops {
 };
 
 #define DATA_SEC ".data"
+#define PERCPU_SEC ".percpu"
 #define BSS_SEC ".bss"
 #define RODATA_SEC ".rodata"
 #define KCONFIG_SEC ".kconfig"
@@ -555,6 +556,7 @@ enum libbpf_map_type {
 	LIBBPF_MAP_BSS,
 	LIBBPF_MAP_RODATA,
 	LIBBPF_MAP_KCONFIG,
+	LIBBPF_MAP_PERCPU,
 };
 
 struct bpf_map_def {
@@ -666,6 +668,7 @@ enum sec_type {
 	SEC_DATA,
 	SEC_RODATA,
 	SEC_ST_OPS,
+	SEC_PERCPU,
 };
 
 struct elf_sec_desc {
@@ -1839,6 +1842,8 @@ static size_t bpf_map_mmap_sz(const struct bpf_map *map)
 	switch (map->def.type) {
 	case BPF_MAP_TYPE_ARRAY:
 		return array_map_mmap_sz(map->def.value_size, map->def.max_entries);
+	case BPF_MAP_TYPE_PERCPU_ARRAY:
+		return map->def.value_size;
 	case BPF_MAP_TYPE_ARENA:
 		return page_sz * map->def.max_entries;
 	default:
@@ -1866,7 +1871,8 @@ static int bpf_map_mmap_resize(struct bpf_map *map, size_t old_sz, size_t new_sz
 	return 0;
 }
 
-static char *internal_map_name(struct bpf_object *obj, const char *real_name)
+static char *internal_map_name(struct bpf_object *obj, const char *real_name,
+			       enum libbpf_map_type type)
 {
 	char map_name[BPF_OBJ_NAME_LEN], *p;
 	int pfx_len, sfx_len = max((size_t)7, strlen(real_name));
@@ -1907,8 +1913,11 @@ static char *internal_map_name(struct bpf_object *obj, const char *real_name)
 	if (sfx_len >= BPF_OBJ_NAME_LEN)
 		sfx_len = BPF_OBJ_NAME_LEN - 1;
 
-	/* if there are two or more dots in map name, it's a custom dot map */
-	if (strchr(real_name + 1, '.') != NULL)
+	/*
+	 * Don't prefix the bpf_object name if this is a custom dot map
+	 * (containing two or more dots) or a percpu data map.
+	 */
+	if (strchr(real_name + 1, '.') != NULL || type == LIBBPF_MAP_PERCPU)
 		pfx_len = 0;
 	else
 		pfx_len = min((size_t)BPF_OBJ_NAME_LEN - sfx_len - 1, strlen(obj->name));
@@ -1941,6 +1950,13 @@ static bool map_is_mmapable(struct bpf_object *obj, struct bpf_map *map)
 	if (!map->btf_value_type_id)
 		return false;
 
+	/*
+	 * The internal PERCPU maps are not mmapble because the underlying
+	 * percpu_array maps do not have mmap support.
+	 */
+	if (map->libbpf_type == LIBBPF_MAP_PERCPU)
+		return false;
+
 	t = btf__type_by_id(obj->btf, map->btf_value_type_id);
 	if (!btf_is_datasec(t))
 		return false;
@@ -1962,6 +1978,7 @@ static int
 bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type,
 			      const char *real_name, int sec_idx, void *data, size_t data_sz)
 {
+	bool is_percpu = type == LIBBPF_MAP_PERCPU;
 	struct bpf_map_def *def;
 	struct bpf_map *map;
 	size_t mmap_sz;
@@ -1975,7 +1992,7 @@ bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type,
 	map->sec_idx = sec_idx;
 	map->sec_offset = 0;
 	map->real_name = strdup(real_name);
-	map->name = internal_map_name(obj, real_name);
+	map->name = internal_map_name(obj, real_name, type);
 	if (!map->real_name || !map->name) {
 		zfree(&map->real_name);
 		zfree(&map->name);
@@ -1983,7 +2000,7 @@ bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type,
 	}
 
 	def = &map->def;
-	def->type = BPF_MAP_TYPE_ARRAY;
+	def->type = is_percpu ? BPF_MAP_TYPE_PERCPU_ARRAY : BPF_MAP_TYPE_ARRAY;
 	def->key_size = sizeof(int);
 	def->value_size = data_sz;
 	def->max_entries = 1;
@@ -1996,8 +2013,9 @@ bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type,
 	if (map_is_mmapable(obj, map))
 		def->map_flags |= BPF_F_MMAPABLE;
 
-	pr_debug("map '%s' (global data): at sec_idx %d, offset %zu, flags %x.\n",
-		 map->name, map->sec_idx, map->sec_offset, def->map_flags);
+	pr_debug("map '%s' (global %sdata): at sec_idx %d, offset %zu, flags %x.\n",
+		 map->name, is_percpu ? "percpu " : "", map->sec_idx,
+		 map->sec_offset, def->map_flags);
 
 	mmap_sz = bpf_map_mmap_sz(map);
 	map->mmaped = mmap(NULL, mmap_sz, PROT_READ | PROT_WRITE,
@@ -2057,6 +2075,13 @@ static int bpf_object__init_global_data_maps(struct bpf_object *obj)
 							    NULL,
 							    sec_desc->data->d_size);
 			break;
+		case SEC_PERCPU:
+			sec_name = elf_sec_name(obj, elf_sec_by_idx(obj, sec_idx));
+			err = bpf_object__init_internal_map(obj, LIBBPF_MAP_PERCPU,
+							    sec_name, sec_idx,
+							    sec_desc->data->d_buf,
+							    sec_desc->data->d_size);
+			break;
 		default:
 			/* skip */
 			break;
@@ -4016,6 +4041,11 @@ static int bpf_object__elf_collect(struct bpf_object *obj)
 				sec_desc->sec_type = SEC_RODATA;
 				sec_desc->shdr = sh;
 				sec_desc->data = data;
+			} else if (strcmp(name, PERCPU_SEC) == 0 ||
+				   str_has_pfx(name, PERCPU_SEC ".")) {
+				sec_desc->sec_type = SEC_PERCPU;
+				sec_desc->shdr = sh;
+				sec_desc->data = data;
 			} else if (strcmp(name, STRUCT_OPS_SEC) == 0 ||
 				   strcmp(name, STRUCT_OPS_LINK_SEC) == 0 ||
 				   strcmp(name, "?" STRUCT_OPS_SEC) == 0 ||
@@ -4544,6 +4574,7 @@ static bool bpf_object__shndx_is_data(const struct bpf_object *obj,
 	case SEC_BSS:
 	case SEC_DATA:
 	case SEC_RODATA:
+	case SEC_PERCPU:
 		return true;
 	default:
 		return false;
@@ -4569,6 +4600,8 @@ bpf_object__section_to_libbpf_map_type(const struct bpf_object *obj, int shndx)
 		return LIBBPF_MAP_DATA;
 	case SEC_RODATA:
 		return LIBBPF_MAP_RODATA;
+	case SEC_PERCPU:
+		return LIBBPF_MAP_PERCPU;
 	default:
 		return LIBBPF_MAP_UNSPEC;
 	}
@@ -4944,7 +4977,7 @@ static int map_fill_btf_type_info(struct bpf_object *obj, struct bpf_map *map)
 
 	/*
 	 * LLVM annotates global data differently in BTF, that is,
-	 * only as '.data', '.bss' or '.rodata'.
+	 * only as '.data', '.bss', '.percpu' or '.rodata'.
 	 */
 	if (!bpf_map__is_internal(map))
 		return -ENOENT;
@@ -5293,18 +5326,20 @@ static int
 bpf_object__populate_internal_map(struct bpf_object *obj, struct bpf_map *map)
 {
 	enum libbpf_map_type map_type = map->libbpf_type;
+	bool is_percpu = map_type == LIBBPF_MAP_PERCPU;
+	const __u64 update_flags = is_percpu ? BPF_F_ALL_CPUS : 0;
 	int err, zero = 0;
 	size_t mmap_sz;
 
 	if (obj->gen_loader) {
 		bpf_gen__map_update_elem(obj->gen_loader, map - obj->maps,
-					 map->mmaped, map->def.value_size);
+					 map->mmaped, map->def.value_size, update_flags);
 		if (map_type == LIBBPF_MAP_RODATA || map_type == LIBBPF_MAP_KCONFIG)
 			bpf_gen__map_freeze(obj->gen_loader, map - obj->maps);
 		return 0;
 	}
 
-	err = bpf_map_update_elem(map->fd, &zero, map->mmaped, 0);
+	err = bpf_map_update_elem(map->fd, &zero, map->mmaped, update_flags);
 	if (err) {
 		err = -errno;
 		pr_warn("map '%s': failed to set initial contents: %s\n",
@@ -5349,6 +5384,13 @@ bpf_object__populate_internal_map(struct bpf_object *obj, struct bpf_map *map)
 			return err;
 		}
 		map->mmaped = mmaped;
+	} else if (is_percpu) {
+		if (mprotect(map->mmaped, mmap_sz, PROT_READ)) {
+			err = -errno;
+			pr_warn("map '%s': failed to mprotect() contents: %s\n",
+				bpf_map__name(map), errstr(err));
+			return err;
+		}
 	} else if (map->mmaped) {
 		munmap(map->mmaped, mmap_sz);
 		map->mmaped = NULL;
@@ -5624,9 +5666,16 @@ bpf_object__create_maps(struct bpf_object *obj)
 		 * runtime due to bpf_program__set_autoload(prog, false),
 		 * bpf_object loading will succeed just fine even on old
 		 * kernels.
+		 * Same skipping applies to percpu data.
 		 */
-		if (bpf_map__is_internal(map) && !kernel_supports(obj, FEAT_GLOBAL_DATA))
-			map->autocreate = false;
+		if (bpf_map__is_internal(map)) {
+			bool is_percpu = map->libbpf_type == LIBBPF_MAP_PERCPU;
+			enum kern_feature_id feat_id;
+
+			feat_id = is_percpu ? FEAT_PERCPU_DATA : FEAT_GLOBAL_DATA;
+			if (!kernel_supports(obj, feat_id))
+				map->autocreate = false;
+		}
 
 		if (!map->autocreate) {
 			pr_debug("map '%s': skipped auto-creating...\n", map->name);
@@ -10807,11 +10856,16 @@ static bool map_uses_real_name(const struct bpf_map *map)
 	 * such map's corresponding ELF section name as a map name.
 	 * This check distinguishes .data/.rodata from .data.* and .rodata.*
 	 * maps to know which name has to be returned to the user.
+	 * Map name of the custom .percpu.* maps might be truncated to
+	 * BPF_OBJ_NAME_LEN-1 chars in internal_map_name(). Hence, percpu data
+	 * maps must use real name for their user-visible name.
 	 */
 	if (map->libbpf_type == LIBBPF_MAP_DATA && strcmp(map->real_name, DATA_SEC) != 0)
 		return true;
 	if (map->libbpf_type == LIBBPF_MAP_RODATA && strcmp(map->real_name, RODATA_SEC) != 0)
 		return true;
+	if (map->libbpf_type == LIBBPF_MAP_PERCPU)
+		return true;
 	return false;
 }
 
@@ -10976,7 +11030,8 @@ int bpf_map__set_value_size(struct bpf_map *map, __u32 size)
 		size_t mmap_old_sz, mmap_new_sz;
 		int err;
 
-		if (map->def.type != BPF_MAP_TYPE_ARRAY)
+		if (map->def.type != BPF_MAP_TYPE_ARRAY &&
+		    map->def.type != BPF_MAP_TYPE_PERCPU_ARRAY)
 			return libbpf_err(-EOPNOTSUPP);
 
 		mmap_old_sz = bpf_map_mmap_sz(map);
diff --git a/tools/lib/bpf/libbpf_internal.h b/tools/lib/bpf/libbpf_internal.h
index 7a74abb904f8b..4c46d34fc055d 100644
--- a/tools/lib/bpf/libbpf_internal.h
+++ b/tools/lib/bpf/libbpf_internal.h
@@ -401,6 +401,8 @@ enum kern_feature_id {
 	FEAT_BTF_LAYOUT,
 	/* Kernel supports BPF syscall common attributes */
 	FEAT_BPF_SYSCALL_COMMON_ATTRS,
+	/* Kernel supports percpu data */
+	FEAT_PERCPU_DATA,
 	__FEAT_CNT,
 };
 
diff --git a/tools/lib/bpf/skel_internal.h b/tools/lib/bpf/skel_internal.h
index 53fee53d36d51..1f3f332dffbe9 100644
--- a/tools/lib/bpf/skel_internal.h
+++ b/tools/lib/bpf/skel_internal.h
@@ -131,8 +131,10 @@ static inline void skel_free_map_data(void *p, __u64 addr, size_t sz)
 {
 	if (addr != ~0ULL)
 		kvfree(p);
-	/* When addr == ~0ULL the 'p' points to
-	 * ((struct bpf_array *)map)->value. See skel_finalize_map_data.
+	/*
+	 * When addr == ~0ULL the init buffer has already been released.
+	 * For skel_finalize_map_data(), 'p' points to
+	 * ((struct bpf_array *)map)->value.
 	 */
 }
 
@@ -170,6 +172,15 @@ static inline void *skel_finalize_map_data(__u64 *init_val, size_t mmap_sz, int
 	return addr;
 }
 
+static inline int skel_protect_map_data(void *p, __u64 *init_val, size_t sz)
+{
+	(void)sz;
+
+	kvfree(p);
+	*init_val = ~0ULL;
+	return 0;
+}
+
 #else
 
 static inline void *skel_alloc(size_t size)
@@ -208,6 +219,15 @@ static inline void *skel_finalize_map_data(__u64 *init_val, size_t mmap_sz, int
 		return NULL;
 	return addr;
 }
+
+static inline int skel_protect_map_data(void *p, __u64 *init_val, size_t sz)
+{
+	(void)init_val;
+
+	if (mprotect(p, sz, PROT_READ))
+		return -errno;
+	return 0;
+}
 #endif
 
 static inline int skel_closenz(int fd)
diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile
index d3655a7064826..560ce4016fbff 100644
--- a/tools/testing/selftests/bpf/Makefile
+++ b/tools/testing/selftests/bpf/Makefile
@@ -531,7 +531,7 @@ LSKELS_SIGNED := fentry_test.c fexit_test.c atomics.c
 
 # Generate both light skeleton and libbpf skeleton for these
 LSKELS_EXTRA := test_ksyms_module.c test_ksyms_weak.c kfunc_call_test.c \
-	kfunc_call_test_subprog.c
+	kfunc_call_test_subprog.c test_global_percpu_data.c
 SKEL_BLACKLIST += $$(LSKELS) $$(LSKELS_SIGNED)
 
 test_static_linked.skel.h-deps := test_static_linked1.bpf.o test_static_linked2.bpf.o
diff --git a/tools/testing/selftests/bpf/prog_tests/global_data_init.c b/tools/testing/selftests/bpf/prog_tests/global_data_init.c
index 8466332d7406f..7d6bda9092954 100644
--- a/tools/testing/selftests/bpf/prog_tests/global_data_init.c
+++ b/tools/testing/selftests/bpf/prog_tests/global_data_init.c
@@ -1,5 +1,8 @@
 // SPDX-License-Identifier: GPL-2.0
 #include <test_progs.h>
+#include "bpf/libbpf_internal.h"
+#include "test_global_percpu_data.skel.h"
+#include "test_global_percpu_data.lskel.h"
 
 void test_global_data_init(void)
 {
@@ -60,3 +63,336 @@ void test_global_data_init(void)
 	free(newval);
 	bpf_object__close(obj);
 }
+
+static void test_percpu_data_on_cpus(struct bpf_map *map, int map_fd, int prog_fd, int *runp)
+{
+	struct test_global_percpu_data__percpu *data = NULL;
+	int i, err, key = 0, num_online, run = 0;
+	__u64 args[2] = {0x1234ULL, 0x5678ULL};
+	size_t data_sz;
+	bool *online;
+	LIBBPF_OPTS(bpf_test_run_opts, topts,
+		    .ctx_in = args,
+		    .ctx_size_in = sizeof(args),
+		    .flags = BPF_F_TEST_RUN_ON_CPU,
+	);
+
+	err = parse_cpu_mask_file("/sys/devices/system/cpu/online", &online, &num_online);
+	if (!ASSERT_OK(err, "parse_cpu_mask_file"))
+		return;
+
+	data_sz = map ? bpf_map__value_size(map) : sizeof(*data);
+	data = calloc(1, data_sz);
+	if (!ASSERT_OK_PTR(data, "calloc percpu data"))
+		goto out;
+
+	/* run on every online-CPU */
+	for (i = 0; i < num_online; i++) {
+		__u64 flags;
+
+		if (!online[i])
+			continue;
+
+		topts.cpu = i;
+		topts.retval = -1;
+		err = bpf_prog_test_run_opts(prog_fd, &topts);
+		ASSERT_OK(err, "bpf_prog_test_run_opts");
+		ASSERT_EQ(topts.retval, 0, "bpf_prog_test_run_opts retval");
+
+		memset(data, 0, data_sz);
+		flags = ((__u64) i << 32) | BPF_F_CPU;
+		if (map)
+			err = bpf_map__lookup_elem(map, &key, sizeof(key), data, data_sz, flags);
+		else
+			err = bpf_map_lookup_elem_flags(map_fd, &key, data, flags);
+		if (!ASSERT_OK(err, "lookup_elem on cpu"))
+			break;
+
+		ASSERT_EQ(*runp, ++run, "run");
+		ASSERT_EQ(data->cpu_id[0], i, "cpu_id");
+		ASSERT_EQ(data->data, 1, "data");
+		ASSERT_TRUE(data->set, "set");
+		ASSERT_EQ(data->nums[6], 0xc0de, "nums[6]");
+		ASSERT_EQ(data->struct_data.i, 1, "struct_data.i");
+		ASSERT_TRUE(data->struct_data.set, "struct_data.set");
+		ASSERT_EQ(data->struct_data.nums[6], 0xc0de, "struct_data.nums[6]");
+	}
+
+out:
+	free(data);
+	free(online);
+}
+
+static void test_global_percpu_data_init(void)
+{
+	struct test_global_percpu_data__percpu init_value = {};
+	struct test_global_percpu_data__percpu *init_data;
+	const __u32 desired_sz = sysconf(_SC_PAGE_SIZE);
+	struct test_global_percpu_data *skel = NULL;
+	size_t init_data_sz;
+	struct bpf_map *map;
+	int prog_fd, err;
+
+	skel = test_global_percpu_data__open();
+	if (!ASSERT_OK_PTR(skel, "test_global_percpu_data__open"))
+		goto out;
+	if (!ASSERT_OK_PTR(skel->percpu, "skel->percpu"))
+		goto out;
+	if (!ASSERT_OK_PTR(skel->data_percpu, "skel->data_percpu"))
+		goto out;
+	if (!ASSERT_OK_PTR(skel->percpu_data, "skel->percpu_data"))
+		goto out;
+	if (!ASSERT_OK_PTR(skel->percpu_looooooooong, "skel->percpu_looooooooong"))
+		goto out;
+
+	ASSERT_STREQ(bpf_map__name(skel->maps.percpu_data), ".percpu.data",
+		     ".percpu.data map name");
+	ASSERT_STREQ(bpf_map__name(skel->maps.data_percpu), ".data.percpu",
+		     ".data.percpu map name");
+	ASSERT_STREQ(bpf_map__name(skel->maps.percpu_looooooooong), ".percpu.looooooooong",
+		     "long map name");
+	ASSERT_STREQ(bpf_map__name(skel->maps.percpu), ".percpu", "map name");
+	ASSERT_EQ(skel->percpu->data, -1, "skel->percpu->data");
+	ASSERT_FALSE(skel->percpu->set, "skel->percpu->set");
+	ASSERT_EQ(skel->percpu->nums[6], 0, "skel->percpu->nums[6]");
+	ASSERT_EQ(skel->percpu->struct_data.i, -1, "struct_data.i");
+	ASSERT_FALSE(skel->percpu->struct_data.set, "struct_data.set");
+	ASSERT_EQ(skel->percpu->struct_data.nums[6], 0, "struct_data.nums[6]");
+
+	map = skel->maps.percpu;
+	if (!ASSERT_EQ(bpf_map__type(map), BPF_MAP_TYPE_PERCPU_ARRAY, "bpf_map__type"))
+		goto out;
+
+	init_value.data = 2;
+	init_value.nums[6] = -1;
+	init_value.struct_data.i = 2;
+	init_value.struct_data.nums[6] = -1;
+	err = bpf_map__set_initial_value(map, &init_value, sizeof(init_value));
+	if (!ASSERT_OK(err, "bpf_map__set_initial_value"))
+		goto out;
+
+	init_data = bpf_map__initial_value(map, &init_data_sz);
+	if (!ASSERT_OK_PTR(init_data, "bpf_map__initial_value"))
+		goto out;
+
+	ASSERT_EQ(init_data->data, init_value.data, "init_value data");
+	ASSERT_EQ(init_data->set, init_value.set, "init_value set");
+	ASSERT_EQ(init_data->struct_data.i, init_value.struct_data.i, "init_value struct_data.i");
+	ASSERT_EQ(init_data->struct_data.nums[6], init_value.struct_data.nums[6],
+		  "init_value struct_data.nums[6]");
+	ASSERT_EQ(init_data_sz, sizeof(init_value), "init_value size");
+	ASSERT_EQ((void *) init_data, (void *) skel->percpu, "skel->percpu eq init_data");
+	ASSERT_EQ(skel->percpu->data, init_value.data, "skel->percpu->data");
+	ASSERT_EQ(skel->percpu->set, init_value.set, "skel->percpu->set");
+	ASSERT_EQ(skel->percpu->struct_data.i, init_value.struct_data.i,
+		  "skel->percpu->struct_data.i");
+	ASSERT_EQ(skel->percpu->struct_data.nums[6], init_value.struct_data.nums[6],
+		  "skel->percpu->struct_data.nums[6]");
+
+	ASSERT_GT(desired_sz, sizeof(init_value), "desired_sz");
+	err = bpf_map__set_value_size(map, desired_sz);
+	if (!ASSERT_OK(err, "bpf_map__set_value_size"))
+		goto out;
+	if (!ASSERT_EQ(bpf_map__value_size(map), desired_sz, "percpu value size"))
+		goto out;
+	if (!ASSERT_NEQ(bpf_map__btf_value_type_id(map), 0, "percpu BTF value type"))
+		goto out;
+
+	init_data = bpf_map__initial_value(map, &init_data_sz);
+	if (!ASSERT_OK_PTR(init_data, "resized bpf_map__initial_value"))
+		goto out;
+	if (!ASSERT_EQ(init_data_sz, desired_sz, "resized initial value size"))
+		goto out;
+	if (!ASSERT_EQ(init_data->data, init_value.data, "resized initial value data"))
+		goto out;
+
+	err = test_global_percpu_data__load(skel);
+	if (!ASSERT_OK(err, "test_global_percpu_data__load"))
+		goto out;
+
+	ASSERT_OK_PTR(skel->percpu, "skel->percpu");
+
+	prog_fd = bpf_program__fd(skel->progs.update_percpu_data);
+	test_percpu_data_on_cpus(map, bpf_map__fd(map), prog_fd, &skel->bss->run);
+
+out:
+	test_global_percpu_data__destroy(skel);
+}
+
+static void test_global_percpu_data_lskel(void)
+{
+	struct test_global_percpu_data_lskel *lskel = NULL;
+	int prog_fd, map_fd;
+
+	lskel = test_global_percpu_data_lskel__open_and_load();
+	if (!ASSERT_OK_PTR(lskel, "test_global_percpu_data_lskel__open_and_load"))
+		goto out;
+
+	map_fd = lskel->maps.percpu.map_fd;
+	prog_fd = lskel->progs.update_percpu_data.prog_fd;
+	test_percpu_data_on_cpus(NULL, map_fd, prog_fd, &lskel->bss->run);
+
+out:
+	test_global_percpu_data_lskel__destroy(lskel);
+}
+
+static int create_rdonly_percpu_array(void)
+{
+	LIBBPF_OPTS(bpf_map_create_opts, map_opts,
+		    .map_flags = BPF_F_RDONLY_PROG,
+	);
+	int key = 0, map_fd, err;
+	__u64 value = 0;
+
+	map_fd = bpf_map_create(BPF_MAP_TYPE_PERCPU_ARRAY, "percpu_ro_map", sizeof(int),
+				sizeof(__u64), 1, &map_opts);
+	if (!ASSERT_GE(map_fd, 0, "bpf_map_create"))
+		return -1;
+
+	err = bpf_map_update_elem(map_fd, &key, &value, BPF_F_ALL_CPUS);
+	if (!ASSERT_OK(err, "bpf_map_update_elem"))
+		goto out;
+
+	err = bpf_map_freeze(map_fd);
+	if (!ASSERT_OK(err, "bpf_map_freeze"))
+		goto out;
+
+	return map_fd;
+
+out:
+	close(map_fd);
+	return -1;
+}
+
+static void test_global_percpu_data_rdonly_direct_read(void)
+{
+	/*
+	 * Raw instructions with manually prepared rdonly percpu_array map
+	 * for testing direct-read global percpu data, because libbpf
+	 * doesn't have rdonly internal percpu_array map support for
+	 * global percpu data.
+	 */
+	struct bpf_insn insns[] = {
+		BPF_LD_MAP_VALUE(BPF_REG_1, 0, 0),
+		BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, 0),
+		BPF_EXIT_INSN(),
+	};
+	int map_fd, prog_fd;
+
+	map_fd = create_rdonly_percpu_array();
+	if (map_fd < 0)
+		return;
+
+	insns[0].imm = map_fd;
+	prog_fd = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, "percpu_ro_prog", "GPL", insns,
+				ARRAY_SIZE(insns), NULL);
+	if (ASSERT_GE(prog_fd, 0, "bpf_prog_load"))
+		close(prog_fd);
+	close(map_fd);
+}
+
+static void test_global_percpu_data_rdonly_direct_write(void)
+{
+	LIBBPF_OPTS(bpf_prog_load_opts, prog_opts);
+	/* See the comment in test_global_percpu_data_rdonly_direct_read() */
+	struct bpf_insn insns[] = {
+		BPF_LD_MAP_VALUE(BPF_REG_1, 0, 0),
+		BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, 0),
+		BPF_ST_MEM(BPF_DW, BPF_REG_1, 0, 0),
+		BPF_EXIT_INSN(),
+	};
+	char log_buf[256] = {};
+	int map_fd, prog_fd;
+
+	prog_opts.log_buf = log_buf;
+	prog_opts.log_size = sizeof(log_buf);
+	prog_opts.log_level = 1;
+
+	map_fd = create_rdonly_percpu_array();
+	if (map_fd < 0)
+		return;
+
+	insns[0].imm = map_fd;
+	prog_fd = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, "percpu_ro_prog", "GPL", insns,
+				ARRAY_SIZE(insns), &prog_opts);
+	if (!ASSERT_LT(prog_fd, 0, "bpf_prog_load"))
+		close(prog_fd);
+	else
+		ASSERT_HAS_SUBSTR(log_buf, "write into map forbidden", "verifier log");
+	close(map_fd);
+}
+
+static void test_global_percpu_data_verifier_log(void)
+{
+	RUN_TESTS(test_global_percpu_data);
+}
+
+static void test_global_percpu_data_iter(void)
+{
+	DECLARE_LIBBPF_OPTS(bpf_iter_attach_opts, opts);
+	struct test_global_percpu_data *skel;
+	union bpf_iter_link_info linfo = {};
+	struct bpf_link *link = NULL;
+	int fd, num_cpus, len, err;
+	char buf[16];
+
+	num_cpus = libbpf_num_possible_cpus();
+	if (!ASSERT_GT(num_cpus, 0, "libbpf_num_possible_cpus"))
+		return;
+
+	skel = test_global_percpu_data__open();
+	if (!ASSERT_OK_PTR(skel, "test_global_percpu_data__open"))
+		return;
+
+	skel->rodata->num_cpus = num_cpus;
+	skel->rodata->offsetof_num = offsetof(struct test_global_percpu_data__percpu, struct_data);
+	skel->rodata->offsetof_num += sizeof(skel->percpu->struct_data) - sizeof(int);
+	skel->rodata->elem_sz = roundup(sizeof(struct test_global_percpu_data__percpu), 8);
+	skel->percpu->struct_data.nums[6] = 0xc0de;
+
+	err = test_global_percpu_data__load(skel);
+	if (!ASSERT_OK(err, "test_global_percpu_data__load"))
+		goto out;
+
+	linfo.map.map_fd = bpf_map__fd(skel->maps.percpu);
+	opts.link_info = &linfo;
+	opts.link_info_len = sizeof(linfo);
+	link = bpf_program__attach_iter(skel->progs.dump_percpu_data, &opts);
+	if (!ASSERT_OK_PTR(link, "bpf_program__attach_iter"))
+		goto out;
+
+	fd = bpf_iter_create(bpf_link__fd(link));
+	if (!ASSERT_GE(fd, 0, "bpf_iter_create"))
+		goto out;
+
+	while ((len = read(fd, buf, sizeof(buf))) > 0)
+		do { } while (0);
+	ASSERT_EQ(len, 0, "read iter");
+	ASSERT_TRUE(skel->bss->run_iter, "run_iter");
+	ASSERT_EQ(skel->bss->percpu_data_sum, 0xc0de * num_cpus, "percpu_data_sum");
+
+	close(fd);
+out:
+	bpf_link__destroy(link);
+	test_global_percpu_data__destroy(skel);
+}
+
+void test_global_percpu_data(void)
+{
+	if (!feat_supported(NULL, FEAT_PERCPU_DATA)) {
+		test__skip();
+		return;
+	}
+
+	if (test__start_subtest("init"))
+		test_global_percpu_data_init();
+	if (test__start_subtest("lskel"))
+		test_global_percpu_data_lskel();
+	if (test__start_subtest("rdonly_direct_read"))
+		test_global_percpu_data_rdonly_direct_read();
+	if (test__start_subtest("rdonly_direct_write"))
+		test_global_percpu_data_rdonly_direct_write();
+	test_global_percpu_data_verifier_log();
+	if (test__start_subtest("iter"))
+		test_global_percpu_data_iter();
+}
diff --git a/tools/testing/selftests/bpf/progs/test_global_percpu_data.c b/tools/testing/selftests/bpf/progs/test_global_percpu_data.c
new file mode 100644
index 0000000000000..71ff8d1bf49eb
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/test_global_percpu_data.c
@@ -0,0 +1,89 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+/* Used for testing map name. */
+int loong SEC(".percpu.looooooooong");
+int data3 SEC(".data.percpu");
+int data2 SEC(".percpu.data");
+
+int run;
+/* cpu_id as array to verify map value resizing. */
+int cpu_id[1] SEC(".percpu");
+int data SEC(".percpu") = -1;
+int nums[7] SEC(".percpu");
+bool set SEC(".percpu") = false;
+struct {
+	char set;
+	int i;
+	int nums[7];
+} struct_data SEC(".percpu") = {
+	.set = 0,
+	.i = -1,
+};
+
+SEC("raw_tp/task_rename")
+__auxiliary
+int update_percpu_data(void *ctx)
+{
+	struct_data.nums[6] = 0xc0de;
+	struct_data.set = 1;
+	struct_data.i = 1;
+	nums[6] = 0xc0de;
+	data = 1;
+	run++;
+	set = true;
+	cpu_id[0] = bpf_get_smp_processor_id();
+	return 0;
+}
+
+static const char fmt[] SEC(".percpu.fmt") = "data %d\n";
+
+SEC("?kprobe")
+__failure __msg("R{{[0-9]+}} points to percpu_array map which cannot be used as const string")
+int verifier_strncmp(void *ctx)
+{
+	return bpf_strncmp("test", 5, fmt);
+}
+
+SEC("?kprobe")
+__failure __msg("R{{[0-9]+}} points to percpu_array map which cannot be used as const string")
+int verifier_snprintf(void *ctx)
+{
+	u64 args[] = { data };
+	char buf[128];
+	int len;
+
+	len = bpf_snprintf(buf, sizeof(buf), fmt, args, sizeof(args));
+	if (len > 0)
+		bpf_printk("snprintf: %s\n", buf);
+	return 0;
+}
+
+volatile const __u32 num_cpus = 0;
+volatile const int offsetof_num;
+volatile const int elem_sz;
+__u32 percpu_data_sum = 0;
+bool run_iter = false;
+
+SEC("iter/bpf_map_elem")
+__auxiliary
+int dump_percpu_data(struct bpf_iter__bpf_map_elem *ctx)
+{
+	void *pptr = ctx->value;
+	int i;
+
+	if (!pptr)
+		return 0;
+
+	run_iter = true;
+
+	for (i = 0; i < num_cpus; i++) {
+		percpu_data_sum += *(int *) (pptr + offsetof_num);
+		pptr += elem_sz;
+	}
+	return 0;
+}
+
+char _license[] SEC("license") = "GPL";

gemini-3.1-pro-preview

							input: 23630
							output: 32
							thoughts: 679

**Analyzing the Reachability and Fuzzing Potential of the Global Per-CPU Data in BPF Patch Series**

Okay, so I'm looking at this patch series that introduces per-CPU data support for BPF. My initial thought is to figure out if it's both reachable and a worthwhile target for fuzzing. The changes are spread across several files: `arraymap.c`, `fixups.c`, `verifier.c`, `bpftool/gen.c`, `libbpf.c`, and a new test. Essentially, this patch allows BPF programs to use global variables in `.percpu` sections, backed by `BPF_MAP_TYPE_PERCPU_ARRAY`. The verifier and `libbpf` have been updated to handle these new maps, including JIT fixups for direct per-CPU variable loading.

First, I need to assess how userspace can trigger these changes. It looks like it's primarily through BPF map creation (`BPF_MAP_TYPE_PERCPU_ARRAY`) and loading BPF programs that interact with these maps. The verifier modifications are definitely key here, especially in functions like `check_map_mem_read`, `check_mem_access`, and `bpf_do_misc_fixups`. Loading a program that directly accesses a per-CPU array map, that's the trigger.

Now, let's identify the most important functions to focus on. Based on the diff, these seem to be the ones: `percpu_array_map_direct_value_addr`, `percpu_array_map_direct_value_meta`, `array_map_check_btf`, `bpf_do_misc_fixups`, `check_map_mem_read`, `check_mem_access`, and `check_arg_const_str`.

I'm specifically interested in functions that are API entrypoints. The instructions say, I should favor non-static, non-inlined API entrypoints over internal helper functions.

Taking a closer look at `arraymap.c` and specifically `percpu_array_map_direct_value_addr`. It's static, but it's assigned to `map_direct_value_addr` in `percpu_array_map_ops`, so it's not inlined and has a distinct symbol. But, it is called from the verifier. `bpf_do_misc_fixups` is non-static, so a good target. `check_mem_access` is static, but quite large. Maybe not inlined. `bpf_map_direct_read` is non-static.

Next, I'll dive into `bpf_do_misc_fixups` in `fixups.c`. And then, check `bpf_map_direct_read` in `verifier.c`.

To make this efficient, I think I'll use `default_api:grepper` to search for these symbols and map out the call chains. That should help to quickly identify where these functions are used.

{
  "Expression": "^(int|void|bool|static).* bpf_do_misc_fixups"
}

{
  "Output": "include/linux/bpf_verifier.h=1670=int bpf_fixup_call_args(struct bpf_verifier_env *env);\ninclude/linux/bpf_verifier.h:1671:int bpf_do_misc_fixups(struct bpf_verifier_env *env);\ninclude/linux/bpf_verifier.h-1672-int bpf_insn_def32(struct bpf_prog *prog, struct bpf_insn *insn);\n--\nkernel/bpf/fixups.c=1470=static int add_hidden_subprog(struct bpf_verifier_env *env, struct bpf_insn *patch, int len)\n--\nkernel/bpf/fixups.c-1498- */\nkernel/bpf/fixups.c:1499:int bpf_do_misc_fixups(struct bpf_verifier_env *env)\nkernel/bpf/fixups.c-1500-{\n"
}

gemini-3.1-pro-preview

							input: 25252
							output: 31
							thoughts: 15

Here's my attempt at summarizing the thought process, tailored for an expert audience:

**Investigating Direct Reads with BPF Maps**

Okay, so I need to dig into `bpf_map_direct_read`. My immediate thought is, "Where's this being used, and what's the intended purpose?" I need to understand the architectural implications here. Is it about performance, security, or a specific use case that's pushing this particular approach? I'm already mentally forming some hypotheses. Direct reads suggest potentially skipping the usual copy mechanisms, which is interesting, but also raises immediate flags about concurrency, memory safety, and how it interacts with the BPF verifier. I want to see how this affects different map types – is it map-specific? What are the limitations? I need to look for examples, documentation (which is often sparse, let's be honest), and code paths to see the context. The devil is definitely in the details here. Let's start the hunt for source code and usage.

{
  "Expression": "^(int|void|bool|static).* bpf_map_direct_read"
}

{
  "Output": "include/linux/bpf_verifier.h=1286=bool bpf_map_is_rdonly(const struct bpf_map *map);\ninclude/linux/bpf_verifier.h:1287:int bpf_map_direct_read(struct bpf_map *map, int off, int size, u64 *val,\ninclude/linux/bpf_verifier.h-1288-\t\t\tbool is_ldsx);\n--\nkernel/bpf/verifier.c=5558=bool bpf_map_is_rdonly(const struct bpf_map *map)\n--\nkernel/bpf/verifier.c-5577-\nkernel/bpf/verifier.c:5578:int bpf_map_direct_read(struct bpf_map *map, int off, int size, u64 *val,\nkernel/bpf/verifier.c-5579-\t\t\tbool is_ldsx)\n"
}

gemini-3.1-pro-preview

							input: 25753
							output: 30
							thoughts: 14

Okay, here's my attempt at summarizing that thought process, tailored for an expert audience:

**Hunting Down That Memory Read**

Right, let's get down to brass tacks. I've got a nagging suspicion about a memory read. Let's start with the basics, and immediately hone in on what I'm looking for. I need to zero in on a specific function or point of code related to memory access. So, let's start with a targeted search. I'm going to start by looking for `check_map_mem_read`. This should quickly identify where memory reads are being assessed. Hopefully this is a descriptive and easily discoverable method for pinpointing the source of the problem. That's a solid first step to understanding what's going on, and from there I can delve deeper into the context of that function, examine the surrounding code, and hopefully root out whatever the underlying issue is. The faster I can find this, the faster I can diagnose what is happening.

{
  "Expression": "^(int|void|bool|static).* check_map_mem_read"
}

{
  "Output": "kernel/bpf/verifier.c=6066=static void add_scalar_to_reg(struct bpf_reg_state *dst_reg, s64 val)\n--\nkernel/bpf/verifier.c-6082-\nkernel/bpf/verifier.c:6083:static int check_map_mem_read(struct bpf_verifier_env *env, struct bpf_reg_state *reg, int off,\nkernel/bpf/verifier.c-6084-\t\t\t      int bpf_size, int value_regno, bool is_ldsx)\n"
}

gemini-3.1-pro-preview

							input: 26161
							output: 35
							thoughts: 19

Here's my thought process, as if I were considering this search:

**Diving into the `percpu_array_map_direct_value_addr` Investigation**

Okay, so I need to find something related to `percpu_array_map_direct_value_addr`. Hmm, sounds like we're digging into the guts of how per-CPU data is managed and accessed, likely within a kernel context. The "direct" part suggests an optimized path, maybe bypassing some indirection. "Value addr" strongly implies we are looking at getting the memory address of the specific CPU's data value in a direct manner. That's good, less overhead, which is what we want with per-CPU data structures. We are probably looking to find its definition or usage. Okay, let's start searching.

{
  "Expression": "^(int|void|bool|static).* percpu_array_map_direct_value_addr"
}

{
  "Output": "kernel/bpf/arraymap.c=251=static void *percpu_array_map_lookup_elem(struct bpf_map *map, void *key)\n--\nkernel/bpf/arraymap.c-261-\nkernel/bpf/arraymap.c:262:static int percpu_array_map_direct_value_addr(const struct bpf_map *map, u64 *imm, u32 off)\nkernel/bpf/arraymap.c-263-{\n"
}

gemini-3.1-pro-preview

							input: 26523
							output: 35
							thoughts: 19

Here's my attempt at summarizing the thought process, tailored for an expert audience:

**Diving into the `percpu_array_map_direct_value_meta` Function**

Okay, let's get down to brass tacks. I need to understand this `percpu_array_map_direct_value_meta` function inside and out. It's obviously critical in how we manage per-CPU data, presumably related to direct value mapping, but the metadata aspect piques my interest. I need to figure out: What is it *doing* with the metadata? Is it initialization, access, allocation, or deallocation? Is this metadata storing offset information, alignment details, flags for caching, or something more exotic? The "direct" implies something efficient, but without digging in, I can't be sure of the exact mechanism. My hypothesis right now is that this function probably deals with the core logic for percpu arrays which involves direct access, likely optimizing for performance on common architectures. The goal is now to see the code.

{
  "Expression": "^(int|void|bool|static).* percpu_array_map_direct_value_meta"
}

{
  "Output": "kernel/bpf/arraymap.c=262=static int percpu_array_map_direct_value_addr(const struct bpf_map *map, u64 *imm, u32 off)\n--\nkernel/bpf/arraymap.c-276-\nkernel/bpf/arraymap.c:277:static int percpu_array_map_direct_value_meta(const struct bpf_map *map, u64 imm, u32 *off)\nkernel/bpf/arraymap.c-278-{\n"
}

gemini-3.1-pro-preview

							input: 26939
							output: 33
							thoughts: 197

**My Initial Assessment of the BPF Functions**

Okay, so I'm looking at these BPF functions for potential inclusion. `bpf_do_misc_fixups`, `bpf_map_direct_read`, `percpu_array_map_direct_value_addr`, `percpu_array_map_direct_value_meta`, and `check_map_mem_read` seem like solid candidates. I'm leaning toward these as good starting points for what I'm working on. Initially, I thought that since `check_map_mem_read` is static, maybe I could just stick with `bpf_do_misc_fixups` and `bpf_map_direct_read` to keep things more concise. But, hold on a second...this is a *new* function introduced by this patch. So it's very relevant. In fact, it's probably the most pertinent. I should definitely include it. I'll also add in `percpu_array_map_direct_value_addr` and `percpu_array_map_direct_value_meta` since they are closely related.

Right, so to get this all to work, I'm going to need `BPF_SYSCALL` – that's a given. Now, I should double-check and make sure there aren't any other configuration dependencies I need to address. Gotta be thorough.

{
  "Expression": "CONFIG_BPF_SYSCALL",
  "PathPrefix": "kernel/bpf/"
}

{
  "Output": "kernel/bpf/Makefile=7=CFLAGS_core.o += -Wno-override-init $(cflags-nogcse-yy)\nkernel/bpf/Makefile-8-\nkernel/bpf/Makefile:9:obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o cnum.o log.o token.o liveness.o const_fold.o\nkernel/bpf/Makefile:10:obj-$(CONFIG_BPF_SYSCALL) += bpf_iter.o map_iter.o task_iter.o prog_iter.o link_iter.o\nkernel/bpf/Makefile:11:obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o bloom_filter.o\nkernel/bpf/Makefile:12:obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o bpf_insn_array.o\nkernel/bpf/Makefile:13:obj-$(CONFIG_BPF_SYSCALL) += bpf_local_storage.o bpf_task_storage.o\nkernel/bpf/Makefile:14:obj-$(CONFIG_BPF_SYSCALL) += fixups.o cfg.o states.o backtrack.o check_btf.o\nkernel/bpf/Makefile-15-obj-${CONFIG_BPF_LSM}\t  += bpf_inode_storage.o\nkernel/bpf/Makefile:16:obj-$(CONFIG_BPF_SYSCALL) += disasm.o mprog.o\nkernel/bpf/Makefile-17-obj-$(CONFIG_BPF_JIT) += trampoline.o\nkernel/bpf/Makefile:18:obj-$(CONFIG_BPF_SYSCALL) += btf.o memalloc.o rqspinlock.o stream.o\nkernel/bpf/Makefile-19-ifeq ($(CONFIG_MMU)$(CONFIG_64BIT),yy)\nkernel/bpf/Makefile:20:obj-$(CONFIG_BPF_SYSCALL) += arena.o range_tree.o\nkernel/bpf/Makefile-21-endif\n--\nkernel/bpf/Makefile=23=ifeq ($(CONFIG_NET),y)\nkernel/bpf/Makefile:24:obj-$(CONFIG_BPF_SYSCALL) += devmap.o\nkernel/bpf/Makefile:25:obj-$(CONFIG_BPF_SYSCALL) += cpumap.o\nkernel/bpf/Makefile:26:obj-$(CONFIG_BPF_SYSCALL) += offload.o\nkernel/bpf/Makefile:27:obj-$(CONFIG_BPF_SYSCALL) += net_namespace.o\nkernel/bpf/Makefile:28:obj-$(CONFIG_BPF_SYSCALL) += tcx.o\nkernel/bpf/Makefile-29-endif\nkernel/bpf/Makefile=30=ifeq ($(CONFIG_PERF_EVENTS),y)\nkernel/bpf/Makefile:31:obj-$(CONFIG_BPF_SYSCALL) += stackmap.o\nkernel/bpf/Makefile-32-endif\nkernel/bpf/Makefile=33=ifeq ($(CONFIG_CGROUPS),y)\nkernel/bpf/Makefile:34:obj-$(CONFIG_BPF_SYSCALL) += cgroup_iter.o bpf_cgrp_storage.o\nkernel/bpf/Makefile-35-endif\n--\nkernel/bpf/Makefile=37=ifeq ($(CONFIG_INET),y)\nkernel/bpf/Makefile:38:obj-$(CONFIG_BPF_SYSCALL) += reuseport_array.o\nkernel/bpf/Makefile-39-endif\n--\nkernel/bpf/Makefile=43=ifeq ($(CONFIG_BPF_JIT),y)\nkernel/bpf/Makefile:44:obj-$(CONFIG_BPF_SYSCALL) += bpf_struct_ops.o\nkernel/bpf/Makefile:45:obj-$(CONFIG_BPF_SYSCALL) += cpumask.o\nkernel/bpf/Makefile-46-# bpf_lsm_proto.o must precede bpf_lsm.o. The current pahole logic\n--\nkernel/bpf/Makefile=58=ifneq ($(CONFIG_CRYPTO),)\nkernel/bpf/Makefile:59:obj-$(CONFIG_BPF_SYSCALL) += crypto.o\nkernel/bpf/Makefile-60-endif\nkernel/bpf/Makefile=61=obj-$(CONFIG_BPF_PRELOAD) += preload/\nkernel/bpf/Makefile-62-\nkernel/bpf/Makefile:63:obj-$(CONFIG_BPF_SYSCALL) += relo_core.o\nkernel/bpf/Makefile:64:obj-$(CONFIG_BPF_SYSCALL) += btf_iter.o\nkernel/bpf/Makefile:65:obj-$(CONFIG_BPF_SYSCALL) += btf_relocate.o\nkernel/bpf/Makefile:66:obj-$(CONFIG_BPF_SYSCALL) += kmem_cache_iter.o\nkernel/bpf/Makefile-67-ifeq ($(CONFIG_DMA_SHARED_BUFFER),y)\nkernel/bpf/Makefile:68:obj-$(CONFIG_BPF_SYSCALL) += dmabuf_iter.o\nkernel/bpf/Makefile-69-endif\n--\nkernel/bpf/core.c=100=struct bpf_prog *bpf_prog_alloc_no_stats(unsigned int size, gfp_t gfp_extra_flags)\n--\nkernel/bpf/core.c-143-\nkernel/bpf/core.c:144:#ifdef CONFIG_BPF_SYSCALL\nkernel/bpf/core.c-145-\tbpf_prog_stream_init(fp);\n--\nkernel/bpf/core.c=2473=EVAL4(PROG_NAME_LIST, 416, 448, 480, 512)\n--\nkernel/bpf/core.c-2476-\nkernel/bpf/core.c:2477:#ifdef CONFIG_BPF_SYSCALL\nkernel/bpf/core.c-2478-int bpf_patch_call_args(struct bpf_insn *insn, u32 stack_depth)\n--\nkernel/bpf/core.c=3041=void __bpf_free_used_btfs(struct btf_mod_pair *used_btfs, u32 len)\nkernel/bpf/core.c-3042-{\nkernel/bpf/core.c:3043:#ifdef CONFIG_BPF_SYSCALL\nkernel/bpf/core.c-3044-\tstruct btf_mod_pair *btf_mod;\n--\nkernel/bpf/core.c=3062=static void bpf_prog_free_deferred(struct work_struct *work)\n--\nkernel/bpf/core.c-3067-\taux = container_of(work, struct bpf_prog_aux, work);\nkernel/bpf/core.c:3068:#ifdef CONFIG_BPF_SYSCALL\nkernel/bpf/core.c-3069-\tbpf_free_kfunc_btf_tab(aux-\u003ekfunc_btf_tab);\n--\nkernel/bpf/core.c=3389=static noinline void bpf_prog_report_may_goto_violation(void)\nkernel/bpf/core.c-3390-{\nkernel/bpf/core.c:3391:#ifdef CONFIG_BPF_SYSCALL\nkernel/bpf/core.c-3392-\tstruct bpf_stream_stage ss;\n--\nkernel/bpf/core.c=3429=__weak u64 bpf_arena_get_kern_vm_start(struct bpf_arena *arena)\n--\nkernel/bpf/core.c-3433-\nkernel/bpf/core.c:3434:#ifdef CONFIG_BPF_SYSCALL\nkernel/bpf/core.c-3435-__weak bool bpf_arena_handle_page_fault(unsigned long addr, bool is_write,\n--\nkernel/bpf/core.c=3460=EXPORT_TRACEPOINT_SYMBOL_GPL(xdp_bulk_tx);\nkernel/bpf/core.c-3461-\nkernel/bpf/core.c:3462:#ifdef CONFIG_BPF_SYSCALL\nkernel/bpf/core.c-3463-\n--\nkernel/bpf/trampoline.c=1514=int __weak arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,\n--\nkernel/bpf/trampoline.c-1521-    defined(CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS) \u0026\u0026 \\\nkernel/bpf/trampoline.c:1522:    defined(CONFIG_BPF_SYSCALL)\nkernel/bpf/trampoline.c-1523-\n--\nkernel/bpf/trampoline.c=1769=void bpf_trampoline_multi_detach(struct bpf_prog *prog,\n--\nkernel/bpf/trampoline.c-1806-\t  CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS \u0026\u0026\nkernel/bpf/trampoline.c:1807:\t  CONFIG_BPF_SYSCALL */\nkernel/bpf/trampoline.c-1808-\n"
}

gemini-3.1-pro-preview

							input: 29865
							output: 193
							thoughts: 595

**My BPF Feature Assessment**

Okay, so I'm looking at this new BPF feature, and the configuration it hinges on is `BPF_SYSCALL` – that makes sense. My initial instinct is to focus on a few key functions: `bpf_do_misc_fixups`, `bpf_map_direct_read`, `percpu_array_map_direct_value_addr`, and `percpu_array_map_direct_value_meta`.

I just double-checked that `percpu_array_map_direct_value_addr` is static, which is good. But then the guidelines flagged something – we want non-static, non-inlined entry points. Since `percpu_array_map_direct_value_addr` is assigned to a function pointer in the `percpu_array_map_ops`, it won't be inlined, it'll have a symbol. Excellent. And, of course, the other two, `bpf_do_misc_fixups` and `bpf_map_direct_read` definitely have distinct addresses, so they're in.

I was considering `check_map_mem_read`, but it's static and called only within `check_mem_access`, so it's likely to be inlined. I think I will use the non-static `bpf_do_misc_fixups` and `bpf_map_direct_read` instead.

A quick QEMU check... Yes, this is core BPF functionality, so it's definitely reachable.

Alright, here's what I'm proposing as the output:

*   **WorthFuzzing**: `true`
*   **FocusSymbols**:
    *   `bpf_do_misc_fixups`
    *   `bpf_map_direct_read`
    *   `percpu_array_map_direct_value_addr`
    *   `percpu_array_map_direct_value_meta`
*   **EnableConfigs**:
    *   `BPF_SYSCALL`

My reasoning? This patch is foundational. It's about enabling percpu data in BPF by backing `.percpu` sections with `BPF_MAP_TYPE_PERCPU_ARRAY`. That means modifications to the verifier for direct memory reads, plus updates to JIT fixups to support loading percpu variables. This touches the BPF verifier logic and map operations that are directly accessible from user space via the `bpf()` syscall when loading and executing BPF programs. This seems like a great candidate for fuzzing.

{
  "EnableConfigs": [
    "BPF_SYSCALL"
  ],
  "FocusSymbols": [
    "bpf_do_misc_fixups",
    "bpf_map_direct_read",
    "percpu_array_map_direct_value_addr",
    "percpu_array_map_direct_value_meta"
  ],
  "Reasoning": "The patch introduces support for global percpu data in BPF programs by allowing `.percpu` sections to be backed by `BPF_MAP_TYPE_PERCPU_ARRAY`. It modifies the BPF verifier to handle direct memory reads from percpu arrays and updates JIT fixups to support direct loading of percpu variables. These changes affect core BPF verifier logic and map operations, which are reachable from user space via the `bpf()` syscall when loading and executing BPF programs.",
  "WorthFuzzing": true
}

{
  "EnableConfigs": [
    "BPF_SYSCALL"
  ],
  "FocusSymbols": [
    "bpf_do_misc_fixups",
    "bpf_map_direct_read",
    "percpu_array_map_direct_value_addr",
    "percpu_array_map_direct_value_meta"
  ],
  "Reasoning": "The patch introduces support for global percpu data in BPF programs by allowing `.percpu` sections to be backed by `BPF_MAP_TYPE_PERCPU_ARRAY`. It modifies the BPF verifier to handle direct memory reads from percpu arrays and updates JIT fixups to support direct loading of percpu variables. These changes affect core BPF verifier logic and map operations, which are reachable from user space via the `bpf()` syscall when loading and executing BPF programs.",
  "WorthFuzzing": true
}

{
  "WorthFuzzing": true
}

{
  "KMSANReasoning": "The patch series adds support for global percpu data in BPF by allowing `.percpu` sections to be mapped to `BPF_MAP_TYPE_PERCPU_ARRAY` maps. The changes involve libbpf parsing the new section, bpftool generating skeleton code, and the BPF verifier handling direct value access and JIT patching for percpu arrays.\n\nThere are no new kernel structures sent to user space, no new uninitialized buffers allocated, and no complex data structures where padding might be leaked. The percpu array memory is allocated using existing map allocation routines (which handle initialization) and populated by libbpf from the ELF section.\n\nAny potential bugs introduced by these changes would likely be out-of-bounds accesses, invalid JIT patching, or verifier logic errors, which are well-covered by KASAN and standard bug detectors. There is no specific risk of uninitialized memory usage or info-leaks that would require a dedicated KMSAN fuzzing session.",
  "NeedsKMSAN": false
}

You are an expert Linux kernel security engineer specializing in kernel memory
error detectors (KASAN and KMSAN). Your job is to review the provided patch series and
determine if the code changes justify spawning a dedicated KMSAN (KernelMemorySanitizer)
fuzzing session in addition to standard KASAN fuzzing.

CRITICAL DISTINCTION BETWEEN KASAN AND KMSAN:
- Standard KASAN kernel builds (upstream-apparmor-kasan.config) already enable
  a comprehensive suite of debugging tools and sanitizers, including KASAN
  (out-of-bounds accesses, use-after-free, double free, invalid free), LOCKDEP
  (locking bugs and deadlocks), UB-sanitizers, and memory corruption checks.
- KMSAN (KernelMemorySanitizer) detects reads of UNINITIALIZED memory (stack, heap,
  or page allocations) and kernel-to-user memory info-leaks.

Rule: THERE IS NO SENSE IN RUNNING A KMSAN SESSION IF A BUG CAN BE CAUGHT BY KASAN,
LOCKDEP, OR OTHER STANDARD BUG DETECTORS.
A dedicated KMSAN fuzzing session incurs significant resource costs. You must ONLY
set NeedsKMSAN=true if the code changes introduce or expose UNINITIALIZED MEMORY risks
that are detected ONLY by KMSAN.

Look holistically at the patch series and surrounding code. Even if no direct
uninitialized field accesses or new buffer allocations are added in the diff itself,
a patch may alter control flow, bounds checking, or data length calculations in ways
that change how the rest of the code operates on existing buffers (e.g. allowing
uninitialized stack/heap memory to be read, copied to user space, or used in control
flow). Do not hesitate to use your code access tools to inspect the surrounding code,
called functions, and callers.

Set NeedsKMSAN=true ONLY IF the patch introduces or modifies:
1. Kernel structures sent to user space (via copy_to_user, put_user, netlink skb
   attributes, ioctl output arguments, socket options, or BPF buffers) where fields
   or structure padding might not be fully initialized/zeroed.
2. Conditional logic or branching that depends on potentially uninitialized variables
   or struct fields.
3. Allocation or initialization of complex data structures where uninitialized fields
   could be read later in reachable code paths.
4. Bounds checks, lengths, or logic in a way that allows surrounding code to access
   uninitialized bytes of existing buffers.

Set NeedsKMSAN=false IF:
- The code changes primarily risk out-of-bounds access, array overflows, NULL pointer
  dereferences, locking deadlocks, or use-after-free bugs (these are already caught
  by KASAN, LOCKDEP, or standard bug detectors).
- All stack/heap structures touched or introduced by the patch are fully zeroed
  or initialized (e.g. using = {0}, memset, kzalloc) before being read or copied.
- The patch does not introduce any risk of uninitialized memory usage or info-leaks.

Use your code access tools to inspect the surrounding code if necessary, then provide
detailed KMSANReasoning contrasting KASAN vs KMSAN applicability for this patch.
Prefer calling several tools at the same time to save round-trips.


Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.

For your convenience, here is the diff of the changes:
commit 3b76f69b75942f2595d46bc813574f1caa6e529f
Author: syz-cluster <triage@syzkaller.com>
Date:   Thu Aug 13 16:51:17 2026 +0000

    syz-cluster: applied patch under review

diff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c
index 248b4818178cd..34865701f7f75 100644
--- a/kernel/bpf/arraymap.c
+++ b/kernel/bpf/arraymap.c
@@ -259,6 +259,37 @@ static void *percpu_array_map_lookup_elem(struct bpf_map *map, void *key)
 	return this_cpu_ptr(array->pptrs[index & array->index_mask]);
 }
 
+static int percpu_array_map_direct_value_addr(const struct bpf_map *map, u64 *imm, u32 off)
+{
+	struct bpf_array *array = container_of(map, struct bpf_array, map);
+
+	if (!bpf_jit_supports_percpu_insn())
+		return -EOPNOTSUPP;
+	if (map->max_entries != 1)
+		return -EOPNOTSUPP;
+	if (off >= map->value_size)
+		return -EINVAL;
+
+	*imm = (u64)(__force unsigned long) array->pptrs[0];
+	return 0;
+}
+
+static int percpu_array_map_direct_value_meta(const struct bpf_map *map, u64 imm, u32 *off)
+{
+	struct bpf_array *array = container_of(map, struct bpf_array, map);
+	u64 base = (u64)(__force unsigned long) array->pptrs[0];
+
+	if (!bpf_jit_supports_percpu_insn())
+		return -EOPNOTSUPP;
+	if (map->max_entries != 1)
+		return -EOPNOTSUPP;
+	if (imm < base || imm >= base + array->elem_size)
+		return -ENOENT;
+
+	*off = imm - base;
+	return 0;
+}
+
 /* emit BPF instructions equivalent to C code of percpu_array_map_lookup_elem() */
 static int percpu_array_map_gen_lookup(struct bpf_map *map, struct bpf_insn *insn_buf)
 {
@@ -551,9 +582,10 @@ static int array_map_check_btf(struct bpf_map *map,
 			       const struct btf_type *key_type,
 			       const struct btf_type *value_type)
 {
-	/* One exception for keyless BTF: .bss/.data/.rodata map */
+	/* One exception for keyless BTF: .bss/.data/.rodata/.percpu map */
 	if (btf_type_is_void(key_type)) {
-		if (map->map_type != BPF_MAP_TYPE_ARRAY ||
+		if ((map->map_type != BPF_MAP_TYPE_ARRAY &&
+		     map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY) ||
 		    map->max_entries != 1)
 			return -EINVAL;
 
@@ -832,6 +864,8 @@ const struct bpf_map_ops percpu_array_map_ops = {
 	.map_get_next_key = bpf_array_get_next_key,
 	.map_lookup_elem = percpu_array_map_lookup_elem,
 	.map_gen_lookup = percpu_array_map_gen_lookup,
+	.map_direct_value_addr = percpu_array_map_direct_value_addr,
+	.map_direct_value_meta = percpu_array_map_direct_value_meta,
 	.map_update_elem = array_map_update_elem,
 	.map_delete_elem = array_map_delete_elem,
 	.map_lookup_percpu_elem = percpu_array_map_lookup_percpu_elem,
diff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c
index 40bd04421a991..a2b18a9f1694c 100644
--- a/kernel/bpf/backtrack.c
+++ b/kernel/bpf/backtrack.c
@@ -214,7 +214,6 @@ static inline bool bt_is_reg_set(struct backtrack_state *bt, u32 reg)
 	return bt->reg_masks[bt->frame] & (1 << reg);
 }
 
-
 /* format registers bitmask, e.g., "r0,r2,r4" for 0x15 mask */
 static void fmt_reg_mask(char *buf, ssize_t buf_sz, u32 reg_mask)
 {
@@ -254,7 +253,6 @@ void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask)
 	}
 }
 
-
 /* For given verifier state backtrack_insn() is called from the last insn to
  * the first insn. Its purpose is to compute a bitmask of registers and
  * stack slots that needs precision in the parent verifier state.
diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index 6606187ed4f43..87ffde865a503 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -2534,7 +2534,6 @@ static void btf_bitfield_show(void *data, u8 bits_offset,
 	btf_int128_print(show, print_num);
 }
 
-
 static void btf_int_bits_show(const struct btf *btf,
 			      const struct btf_type *t,
 			      void *data, u8 bits_offset,
diff --git a/kernel/bpf/cfg.c b/kernel/bpf/cfg.c
index db3416a7c9047..818f7afac83a5 100644
--- a/kernel/bpf/cfg.c
+++ b/kernel/bpf/cfg.c
@@ -47,7 +47,6 @@ enum {
 	BRANCH = 2,
 };
 
-
 static void mark_subprog_changes_pkt_data(struct bpf_verifier_env *env, int off)
 {
 	struct bpf_subprog_info *subprog;
diff --git a/kernel/bpf/const_fold.c b/kernel/bpf/const_fold.c
index 4cf120c7b2cb4..7f1b30059cc87 100644
--- a/kernel/bpf/const_fold.c
+++ b/kernel/bpf/const_fold.c
@@ -182,7 +182,6 @@ static void const_reg_xfer(struct bpf_verifier_env *env, struct const_arg_info *
 		u64 val = 0;
 
 		if (!bpf_map_is_rdonly(map) || !map->ops->map_direct_value_addr ||
-		    map->map_type == BPF_MAP_TYPE_INSN_ARRAY ||
 		    off < 0 || off + size > map->value_size ||
 		    bpf_map_direct_read(map, off, size, &val, is_ldsx)) {
 			*dst = unknown;
diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
index 2417a3461652d..177a3fcbb63ac 100644
--- a/kernel/bpf/fixups.c
+++ b/kernel/bpf/fixups.c
@@ -1466,7 +1466,6 @@ int bpf_fixup_call_args(struct bpf_verifier_env *env)
 	return err;
 }
 
-
 /* The function requires that first instruction in 'patch' is insnsi[prog->len - 1] */
 static int add_hidden_subprog(struct bpf_verifier_env *env, struct bpf_insn *patch, int len)
 {
@@ -1835,6 +1834,43 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)
 			goto next_insn;
 		}
 
+		if (bpf_jit_supports_percpu_insn() &&
+		    insn->code == (BPF_LD | BPF_IMM | BPF_DW) &&
+		    (insn->src_reg == BPF_PSEUDO_MAP_VALUE ||
+		     insn->src_reg == BPF_PSEUDO_MAP_IDX_VALUE)) {
+			struct bpf_map *map;
+
+			aux = &env->insn_aux_data[i + delta];
+			map = env->used_maps[aux->map_index];
+			if (map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY)
+				goto next_insn;
+
+			prog->jit_required = true;
+
+			/*
+			 * We are *skipping* first half of ld_imm64 insn
+			 * with 'i++;', patching over second half of it
+			 * with that same half + mov64_percpu_reg insn.
+			 * All because bpf_patch_insn_data() can only
+			 * replace one 8-byte insn, which does not work
+			 * well for ld_imm64 insn.
+			 */
+
+			insn_buf[0] = insn[1];
+			insn_buf[1] = BPF_MOV64_PERCPU_REG(insn->dst_reg, insn->dst_reg);
+			cnt = 2;
+
+			i++;
+			new_prog = bpf_patch_insn_data(env, i + delta, insn_buf, cnt);
+			if (!new_prog)
+				return -ENOMEM;
+
+			delta    += cnt - 1;
+			env->prog = prog = new_prog;
+			insn      = new_prog->insnsi + i + delta;
+			goto next_insn;
+		}
+
 		if (insn->code != (BPF_JMP | BPF_CALL))
 			goto next_insn;
 		if (insn->src_reg == BPF_PSEUDO_CALL)
diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c
index 9f394e1aa2e85..d40cb5dd446ca 100644
--- a/kernel/bpf/hashtab.c
+++ b/kernel/bpf/hashtab.c
@@ -998,7 +998,6 @@ static void dec_elem_count(struct bpf_htab *htab)
 		atomic_dec(&htab->count);
 }
 
-
 static void free_htab_elem(struct bpf_htab *htab, struct htab_elem *l)
 {
 	htab_put_fd_value(htab, l);
@@ -2970,7 +2969,6 @@ static int rhtab_delete_elem(struct bpf_rhtab *rhtab, struct rhtab_elem *elem, v
 	return 0;
 }
 
-
 static long rhtab_map_delete_elem(struct bpf_map *map, void *key)
 {
 	struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map);
diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c
index 6388b6b23e490..45e2f19387b24 100644
--- a/kernel/bpf/helpers.c
+++ b/kernel/bpf/helpers.c
@@ -4871,7 +4871,6 @@ static const struct btf_kfunc_id_set generic_kfunc_set = {
 	.set   = &generic_btf_ids,
 };
 
-
 BTF_ID_LIST(generic_dtor_ids)
 BTF_ID(struct, task_struct)
 BTF_ID(func, bpf_task_release_dtor)
diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index 1c997aeba6fa5..74fc4b3f80d6e 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -269,7 +269,6 @@ bpf_insn_successors(struct bpf_verifier_env *env, u32 idx)
 
 __diag_pop();
 
-
 static inline bool update_insn(struct bpf_verifier_env *env,
 			       struct func_instance *instance, u32 frame, u32 insn_idx)
 {
@@ -1862,7 +1861,6 @@ static int analyze_subprog(struct bpf_verifier_env *env,
 	if (need_resched())
 		cond_resched();
 
-
 	/*
 	 * When an instance is reused (must_write_initialized == true),
 	 * record into a fresh instance and merge afterward.  This avoids
diff --git a/kernel/bpf/queue_stack_maps.c b/kernel/bpf/queue_stack_maps.c
index c1c9dee4dcdd0..6e8b18c32a106 100644
--- a/kernel/bpf/queue_stack_maps.c
+++ b/kernel/bpf/queue_stack_maps.c
@@ -123,7 +123,6 @@ static long __queue_map_get(struct bpf_map *map, void *value, bool delete)
 	return err;
 }
 
-
 static long __stack_map_get(struct bpf_map *map, void *value, bool delete)
 {
 	struct bpf_queue_stack *qs = bpf_queue_stack(map);
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 8d111da886553..7d8c3e8e6d62e 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -636,7 +636,6 @@ int bpf_map_alloc_pages(const struct bpf_map *map, int nid,
 	return ret;
 }
 
-
 static int btf_field_cmp(const void *a, const void *b)
 {
 	const struct btf_field *f1 = a, *f2 = b;
@@ -1830,7 +1829,6 @@ static int map_lookup_elem(union bpf_attr *attr)
 	return err;
 }
 
-
 #define BPF_MAP_UPDATE_ELEM_LAST_FIELD flags
 
 static int map_update_elem(union bpf_attr *attr, bpfptr_t uattr)
@@ -3497,7 +3495,6 @@ int bpf_link_prime(struct bpf_link *link, struct bpf_link_primer *primer)
 	if (fd < 0)
 		return fd;
 
-
 	id = bpf_link_alloc_id(link);
 	if (id < 0) {
 		put_unused_fd(fd);
@@ -5505,7 +5502,6 @@ static int bpf_link_get_info_by_fd(struct file *file,
 	return 0;
 }
 
-
 static int token_get_info_by_fd(struct file *file,
 				struct bpf_token *token,
 				const union bpf_attr *attr,
@@ -6507,7 +6503,6 @@ BPF_CALL_3(bpf_sys_bpf, int, cmd, union bpf_attr *, attr, u32, attr_size)
 	return __sys_bpf(cmd, KERNEL_BPFPTR(attr), attr_size, KERNEL_BPFPTR(NULL), 0);
 }
 
-
 /* To shut up -Wmissing-prototypes.
  * This function is used by the kernel light skeleton
  * to load bpf programs when modules are loaded or during kernel boot.
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 164d16c243ca6..6ac1afced20bf 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -635,7 +635,6 @@ static void __mark_dynptr_reg(struct bpf_reg_state *reg,
 			      enum bpf_dynptr_type type,
 			      bool first_slot, int id, int parent_id);
 
-
 static void mark_dynptr_stack_regs(struct bpf_verifier_env *env,
 				   struct bpf_reg_state *sreg1,
 				   struct bpf_reg_state *sreg2,
@@ -1674,7 +1673,6 @@ static bool same_callsites(struct bpf_verifier_state *a, struct bpf_verifier_sta
 	return true;
 }
 
-
 void bpf_free_backedges(struct bpf_scc_visit *visit)
 {
 	struct bpf_scc_backedge *backedge, *next;
@@ -2291,7 +2289,6 @@ static struct bpf_verifier_state *push_async_cb(struct bpf_verifier_env *env,
 	return &elem->st;
 }
 
-
 static int cmp_subprogs(const void *a, const void *b)
 {
 	return ((struct bpf_subprog_info *)a)->start -
@@ -3969,7 +3966,6 @@ static int check_stack_read(struct bpf_verifier_env *env,
 	return err;
 }
 
-
 /* check_stack_write dispatches to check_stack_write_fixed_off or
  * check_stack_write_var_off.
  *
@@ -4767,7 +4763,6 @@ static int check_sock_access(struct bpf_verifier_env *env, int insn_idx,
 		valid = false;
 	}
 
-
 	if (valid) {
 		env->insn_aux_data[insn_idx].ctx_field_size =
 			info.ctx_field_size;
@@ -5587,6 +5582,8 @@ int bpf_map_direct_read(struct bpf_map *map, int off, int size, u64 *val,
 	u64 addr;
 	int err;
 
+	if (map->map_type == BPF_MAP_TYPE_INSN_ARRAY || map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY)
+		return -EINVAL;
 	err = map->ops->map_direct_value_addr(map, &addr, off);
 	if (err)
 		return err;
@@ -6083,6 +6080,51 @@ static void add_scalar_to_reg(struct bpf_reg_state *dst_reg, s64 val)
 	reg_bounds_sync(dst_reg);
 }
 
+static int check_map_mem_read(struct bpf_verifier_env *env, struct bpf_reg_state *reg, int off,
+			      int bpf_size, int value_regno, bool is_ldsx)
+{
+	struct bpf_reg_state *regs = cur_regs(env);
+	int size = bpf_size_to_bytes(bpf_size);
+	struct bpf_map *map = reg->map_ptr;
+
+	switch (map->map_type) {
+	case BPF_MAP_TYPE_INSN_ARRAY:
+		if (bpf_size != BPF_DW) {
+			verbose(env, "Invalid read of %d bytes from insn_array\n", size);
+			return -EACCES;
+		}
+		regs[value_regno] = *reg;
+		add_scalar_to_reg(&regs[value_regno], off);
+		regs[value_regno].type = PTR_TO_INSN;
+		return 0;
+	case BPF_MAP_TYPE_PERCPU_ARRAY:
+		goto reg_unknown;
+	default:
+		break;
+	}
+
+	/* If map is read-only, track its contents as scalars. */
+	if (tnum_is_const(reg->var_off) &&
+	    bpf_map_is_rdonly(map) &&
+	    map->ops->map_direct_value_addr) {
+		int map_off = off + reg->var_off.value;
+		u64 val = 0;
+		int err;
+
+		err = bpf_map_direct_read(map, map_off, size, &val, is_ldsx);
+		if (err)
+			return err;
+
+		regs[value_regno].type = SCALAR_VALUE;
+		__mark_reg_known(&regs[value_regno], val);
+		return 0;
+	}
+
+reg_unknown:
+	mark_reg_unknown(env, regs, value_regno);
+	return 0;
+}
+
 /* check whether memory at (regno + off) is accessible for t = (read | write)
  * if t==write, value_regno is a register which value is stored into memory
  * if t==read, value_regno is a register which will receive the value from memory
@@ -6137,38 +6179,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, struct b
 		if (kptr_field) {
 			err = check_map_kptr_access(env, value_regno, insn_idx, kptr_field);
 		} else if (t == BPF_READ && value_regno >= 0) {
-			struct bpf_map *map = reg->map_ptr;
-
-			/*
-			 * If map is read-only, track its contents as scalars,
-			 * unless it is an insn array (see the special case below)
-			 */
-			if (tnum_is_const(reg->var_off) &&
-			    bpf_map_is_rdonly(map) &&
-			    map->ops->map_direct_value_addr &&
-			    map->map_type != BPF_MAP_TYPE_INSN_ARRAY) {
-				int map_off = off + reg->var_off.value;
-				u64 val = 0;
-
-				err = bpf_map_direct_read(map, map_off, size,
-							  &val, is_ldsx);
-				if (err)
-					return err;
-
-				regs[value_regno].type = SCALAR_VALUE;
-				__mark_reg_known(&regs[value_regno], val);
-			} else if (map->map_type == BPF_MAP_TYPE_INSN_ARRAY) {
-				if (bpf_size != BPF_DW) {
-					verbose(env, "Invalid read of %d bytes from insn_array\n",
-						     size);
-					return -EACCES;
-				}
-				regs[value_regno] = *reg;
-				add_scalar_to_reg(&regs[value_regno], off);
-				regs[value_regno].type = PTR_TO_INSN;
-			} else {
-				mark_reg_unknown(env, regs, value_regno);
-			}
+			err = check_map_mem_read(env, reg, off, bpf_size, value_regno, is_ldsx);
 		}
 	} else if (base_type(reg->type) == PTR_TO_MEM) {
 		bool rdonly_mem = type_is_rdonly_mem(reg->type);
@@ -6635,7 +6646,6 @@ static int check_stack_range_initialized(
 	if (err)
 		return err;
 
-
 	if (tnum_is_const(reg->var_off)) {
 		min_off = max_off = reg->var_off.value + off;
 	} else {
@@ -7347,7 +7357,6 @@ static bool is_iter_new_kfunc(struct bpf_call_arg_meta *meta)
 	return meta->kfunc_flags & KF_ITER_NEW;
 }
 
-
 static bool is_iter_destroy_kfunc(struct bpf_call_arg_meta *meta)
 {
 	return meta->kfunc_flags & KF_ITER_DESTROY;
@@ -8125,6 +8134,12 @@ static int check_arg_const_str(struct bpf_verifier_env *env,
 		return -EACCES;
 	}
 
+	if (map->map_type == BPF_MAP_TYPE_PERCPU_ARRAY) {
+		verbose(env, "%s points to percpu_array map which cannot be used as const string\n",
+			reg_arg_name(env, argno));
+		return -EACCES;
+	}
+
 	if (!bpf_map_is_rdonly(map)) {
 		verbose(env, "%s does not point to a readonly map'\n", reg_arg_name(env, argno));
 		return -EACCES;
@@ -11607,7 +11622,6 @@ static int process_irq_flag(struct bpf_verifier_env *env, struct bpf_reg_state *
 	return 0;
 }
 
-
 static int ref_set_non_owning(struct bpf_verifier_env *env, struct bpf_reg_state *reg)
 {
 	struct btf_record *rec = reg_btf_record(reg);
@@ -16412,7 +16426,6 @@ static int check_ld_abs(struct bpf_verifier_env *env, struct bpf_insn *insn)
 	return 0;
 }
 
-
 static bool return_retval_range(struct bpf_verifier_env *env, struct bpf_retval_range *range)
 {
 	enum bpf_prog_type prog_type = resolve_prog_type(env->prog);
@@ -18361,8 +18374,6 @@ static void release_insn_arrays(struct bpf_verifier_env *env)
 		bpf_insn_array_release(env->insn_array_maps[i]);
 }
 
-
-
 /* The verifier does more data flow analysis than llvm and will not
  * explore branches that are dead at run time. Malicious programs can
  * have dead code too. Therefore replace all dead at-run-time code
@@ -18390,8 +18401,6 @@ static void sanitize_dead_code(struct bpf_verifier_env *env)
 	}
 }
 
-
-
 static void free_states(struct bpf_verifier_env *env)
 {
 	struct bpf_verifier_state_list *sl;
@@ -18678,7 +18687,6 @@ static int do_check_main(struct bpf_verifier_env *env)
 	return ret;
 }
 
-
 static void print_verification_stats(struct bpf_verifier_env *env)
 {
 	/* Skip over hidden subprogs which are not verified. */
diff --git a/tools/bpf/bpftool/gen.c b/tools/bpf/bpftool/gen.c
index a01d06d22d1a3..a50540ef6521c 100644
--- a/tools/bpf/bpftool/gen.c
+++ b/tools/bpf/bpftool/gen.c
@@ -101,6 +101,12 @@ static bool get_map_ident(const struct bpf_map *map, char *buf, size_t buf_sz)
 		return true;
 	}
 
+	if (bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY) {
+		snprintf(buf, buf_sz, "%s", name + 1);
+		sanitize_identifier(buf);
+		return true;
+	}
+
 	for  (i = 0, n = ARRAY_SIZE(sfxs); i < n; i++) {
 		const char *sfx = sfxs[i], *p;
 
@@ -117,7 +123,7 @@ static bool get_map_ident(const struct bpf_map *map, char *buf, size_t buf_sz)
 
 static bool get_datasec_ident(const char *sec_name, char *buf, size_t buf_sz)
 {
-	static const char *pfxs[] = { ".data", ".rodata", ".bss", ".kconfig" };
+	static const char *pfxs[] = { ".data", ".rodata", ".bss", ".percpu", ".kconfig" };
 	int i, n;
 
 	/* recognize hard coded LLVM section name */
@@ -254,7 +260,7 @@ static const struct btf_type *find_type_for_map(struct btf *btf, const char *map
 	return NULL;
 }
 
-static bool is_mmapable_map(const struct bpf_map *map, char *buf, size_t sz)
+static bool is_skel_data(const struct bpf_map *map, char *buf, size_t sz)
 {
 	size_t tmp_sz;
 
@@ -263,13 +269,24 @@ static bool is_mmapable_map(const struct bpf_map *map, char *buf, size_t sz)
 		return true;
 	}
 
-	if (!bpf_map__is_internal(map) || !(bpf_map__map_flags(map) & BPF_F_MMAPABLE))
+	if (!bpf_map__is_internal(map))
 		return false;
 
 	if (!get_map_ident(map, buf, sz))
 		return false;
 
-	return true;
+	if (bpf_map__map_flags(map) & BPF_F_MMAPABLE)
+		return true;
+
+	if (bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY)
+		return bpf_map__btf_value_type_id(map) != 0;
+
+	return false;
+}
+
+static bool is_mmapable_map(const struct bpf_map *map, char *buf, size_t sz)
+{
+	return is_skel_data(map, buf, sz) && bpf_map__type(map) != BPF_MAP_TYPE_PERCPU_ARRAY;
 }
 
 static int codegen_datasecs(struct bpf_object *obj, const char *obj_name)
@@ -287,7 +304,7 @@ static int codegen_datasecs(struct bpf_object *obj, const char *obj_name)
 
 	bpf_object__for_each_map(map, obj) {
 		/* only generate definitions for memory-mapped internal maps */
-		if (!is_mmapable_map(map, map_ident, sizeof(map_ident)))
+		if (!is_skel_data(map, map_ident, sizeof(map_ident)))
 			continue;
 
 		sec = find_type_for_map(btf, map_ident);
@@ -517,7 +534,7 @@ static void codegen_asserts(struct bpf_object *obj, const char *obj_name)
 		", obj_name);
 
 	bpf_object__for_each_map(map, obj) {
-		if (!is_mmapable_map(map, map_ident, sizeof(map_ident)))
+		if (!is_skel_data(map, map_ident, sizeof(map_ident)))
 			continue;
 
 		sec = find_type_for_map(btf, map_ident);
@@ -668,8 +685,7 @@ static void codegen_destroy(struct bpf_object *obj, const char *obj_name)
 	bpf_object__for_each_map(map, obj) {
 		if (!get_map_ident(map, ident, sizeof(ident)))
 			continue;
-		if (bpf_map__is_internal(map) &&
-		    (bpf_map__map_flags(map) & BPF_F_MMAPABLE))
+		if (is_skel_data(map, ident, sizeof(ident)))
 			printf("\tskel_free_map_data(skel->%1$s, skel->maps.%1$s.initial_value, %2$zu);\n",
 			       ident, bpf_map_mmap_sz(map));
 		codegen("\
@@ -741,7 +757,7 @@ static int gen_trace(struct bpf_object *obj, const char *obj_name, const char *h
 		const void *mmap_data = NULL;
 		size_t mmap_size = 0;
 
-		if (!is_mmapable_map(map, ident, sizeof(ident)))
+		if (!is_skel_data(map, ident, sizeof(ident)))
 			continue;
 
 		codegen("\
@@ -849,9 +865,23 @@ static int gen_trace(struct bpf_object *obj, const char *obj_name, const char *h
 	bpf_object__for_each_map(map, obj) {
 		const char *mmap_flags;
 
-		if (!is_mmapable_map(map, ident, sizeof(ident)))
+		if (!is_skel_data(map, ident, sizeof(ident)))
 			continue;
 
+		if (bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY) {
+			codegen("\
+		\n\
+			err = skel_protect_map_data(skel->%1$s, &skel->maps.%1$s.initial_value, %2$zd);\n\
+			if (err)					    \n\
+				return err;				    \n\
+		#ifdef __KERNEL__					    \n\
+			skel->%1$s = NULL;				    \n\
+		#endif							    \n\
+			",
+			ident, bpf_map_mmap_sz(map));
+			continue;
+		}
+
 		if (bpf_map__map_flags(map) & BPF_F_RDONLY_PROG)
 			mmap_flags = "PROT_READ";
 		else
@@ -955,8 +985,7 @@ codegen_maps_skeleton(struct bpf_object *obj, size_t map_cnt, bool mmaped, bool
 				map->map = &obj->maps.%s;	    \n\
 			",
 			i, bpf_map__name(map), ident);
-		/* memory-mapped internal maps */
-		if (mmaped && is_mmapable_map(map, ident, sizeof(ident))) {
+		if (mmaped && is_skel_data(map, ident, sizeof(ident))) {
 			printf("\tmap->mmaped = (void **)&obj->%s;\n", ident);
 		}
 
diff --git a/tools/lib/bpf/bpf_gen_internal.h b/tools/lib/bpf/bpf_gen_internal.h
index 0425691877521..6c5ad6c55e8a6 100644
--- a/tools/lib/bpf/bpf_gen_internal.h
+++ b/tools/lib/bpf/bpf_gen_internal.h
@@ -65,7 +65,8 @@ void bpf_gen__prog_load(struct bpf_gen *gen,
 			enum bpf_prog_type prog_type, const char *prog_name,
 			const char *license, struct bpf_insn *insns, size_t insn_cnt,
 			struct bpf_prog_load_opts *load_attr, int prog_idx);
-void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *value, __u32 value_size);
+void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *value, __u32 value_size,
+			      __u64 flags);
 void bpf_gen__map_freeze(struct bpf_gen *gen, int map_idx);
 void bpf_gen__record_attach_target(struct bpf_gen *gen, const char *name, enum bpf_attach_type type);
 void bpf_gen__record_extern(struct bpf_gen *gen, const char *name, bool is_weak,
diff --git a/tools/lib/bpf/features.c b/tools/lib/bpf/features.c
index b7e388f99d0bb..ef9581c113035 100644
--- a/tools/lib/bpf/features.c
+++ b/tools/lib/bpf/features.c
@@ -620,6 +620,38 @@ static int probe_bpf_syscall_common_attrs(int token_fd)
 	return probe_sys_bpf_ext();
 }
 
+static int probe_kern_percpu_data(int token_fd)
+{
+	struct bpf_insn insns[] = {
+		BPF_LD_MAP_VALUE(BPF_REG_1, 0, 0),
+		BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, 0),
+		BPF_EXIT_INSN(),
+	};
+	LIBBPF_OPTS(bpf_map_create_opts, map_opts,
+		.token_fd = token_fd,
+		.map_flags = token_fd ? BPF_F_TOKEN_FD : 0,
+	);
+	LIBBPF_OPTS(bpf_prog_load_opts, prog_opts,
+		.token_fd = token_fd,
+		.prog_flags = token_fd ? BPF_F_TOKEN_FD : 0,
+	);
+	int ret, map, insn_cnt = ARRAY_SIZE(insns);
+
+	map = bpf_map_create(BPF_MAP_TYPE_PERCPU_ARRAY, "libbpf_percpu", sizeof(int), 8, 1,
+			     &map_opts);
+	if (map < 0) {
+		pr_warn("Error in %s(): %s. Couldn't create simple percpu_array map.\n",
+			__func__, errstr(map));
+		return map;
+	}
+
+	insns[0].imm = map;
+
+	ret = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, NULL, "GPL", insns, insn_cnt, &prog_opts);
+	close(map);
+	return probe_fd(ret);
+}
+
 typedef int (*feature_probe_fn)(int /* token_fd */);
 
 static struct kern_feature_cache feature_cache;
@@ -707,6 +739,9 @@ static struct kern_feature_desc {
 	[FEAT_BPF_SYSCALL_COMMON_ATTRS] = {
 		"BPF syscall common attributes support", probe_bpf_syscall_common_attrs,
 	},
+	[FEAT_PERCPU_DATA] = {
+		"kernel supports percpu data", probe_kern_percpu_data,
+	},
 };
 
 bool feat_supported(struct kern_feature_cache *cache, enum kern_feature_id feat_id)
diff --git a/tools/lib/bpf/gen_loader.c b/tools/lib/bpf/gen_loader.c
index 6e3dd52427618..af3a04f161ac1 100644
--- a/tools/lib/bpf/gen_loader.c
+++ b/tools/lib/bpf/gen_loader.c
@@ -1128,7 +1128,7 @@ void bpf_gen__prog_load(struct bpf_gen *gen,
 }
 
 void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *pvalue,
-			      __u32 value_size)
+			      __u32 value_size, __u64 flags)
 {
 	int attr_size = offsetofend(union bpf_attr, flags);
 	int map_update_attr, value, key;
@@ -1136,6 +1136,7 @@ void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *pvalue,
 	int zero = 0;
 
 	memset(&attr, 0, attr_size);
+	attr.flags = tgt_endian(flags);
 
 	value = add_data(gen, pvalue, value_size);
 	key = add_data(gen, &zero, sizeof(zero));
diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
index 514e4e9daa825..e574870fb7169 100644
--- a/tools/lib/bpf/libbpf.c
+++ b/tools/lib/bpf/libbpf.c
@@ -541,6 +541,7 @@ struct bpf_struct_ops {
 };
 
 #define DATA_SEC ".data"
+#define PERCPU_SEC ".percpu"
 #define BSS_SEC ".bss"
 #define RODATA_SEC ".rodata"
 #define KCONFIG_SEC ".kconfig"
@@ -555,6 +556,7 @@ enum libbpf_map_type {
 	LIBBPF_MAP_BSS,
 	LIBBPF_MAP_RODATA,
 	LIBBPF_MAP_KCONFIG,
+	LIBBPF_MAP_PERCPU,
 };
 
 struct bpf_map_def {
@@ -666,6 +668,7 @@ enum sec_type {
 	SEC_DATA,
 	SEC_RODATA,
 	SEC_ST_OPS,
+	SEC_PERCPU,
 };
 
 struct elf_sec_desc {
@@ -1839,6 +1842,8 @@ static size_t bpf_map_mmap_sz(const struct bpf_map *map)
 	switch (map->def.type) {
 	case BPF_MAP_TYPE_ARRAY:
 		return array_map_mmap_sz(map->def.value_size, map->def.max_entries);
+	case BPF_MAP_TYPE_PERCPU_ARRAY:
+		return map->def.value_size;
 	case BPF_MAP_TYPE_ARENA:
 		return page_sz * map->def.max_entries;
 	default:
@@ -1866,7 +1871,8 @@ static int bpf_map_mmap_resize(struct bpf_map *map, size_t old_sz, size_t new_sz
 	return 0;
 }
 
-static char *internal_map_name(struct bpf_object *obj, const char *real_name)
+static char *internal_map_name(struct bpf_object *obj, const char *real_name,
+			       enum libbpf_map_type type)
 {
 	char map_name[BPF_OBJ_NAME_LEN], *p;
 	int pfx_len, sfx_len = max((size_t)7, strlen(real_name));
@@ -1907,8 +1913,11 @@ static char *internal_map_name(struct bpf_object *obj, const char *real_name)
 	if (sfx_len >= BPF_OBJ_NAME_LEN)
 		sfx_len = BPF_OBJ_NAME_LEN - 1;
 
-	/* if there are two or more dots in map name, it's a custom dot map */
-	if (strchr(real_name + 1, '.') != NULL)
+	/*
+	 * Don't prefix the bpf_object name if this is a custom dot map
+	 * (containing two or more dots) or a percpu data map.
+	 */
+	if (strchr(real_name + 1, '.') != NULL || type == LIBBPF_MAP_PERCPU)
 		pfx_len = 0;
 	else
 		pfx_len = min((size_t)BPF_OBJ_NAME_LEN - sfx_len - 1, strlen(obj->name));
@@ -1941,6 +1950,13 @@ static bool map_is_mmapable(struct bpf_object *obj, struct bpf_map *map)
 	if (!map->btf_value_type_id)
 		return false;
 
+	/*
+	 * The internal PERCPU maps are not mmapble because the underlying
+	 * percpu_array maps do not have mmap support.
+	 */
+	if (map->libbpf_type == LIBBPF_MAP_PERCPU)
+		return false;
+
 	t = btf__type_by_id(obj->btf, map->btf_value_type_id);
 	if (!btf_is_datasec(t))
 		return false;
@@ -1962,6 +1978,7 @@ static int
 bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type,
 			      const char *real_name, int sec_idx, void *data, size_t data_sz)
 {
+	bool is_percpu = type == LIBBPF_MAP_PERCPU;
 	struct bpf_map_def *def;
 	struct bpf_map *map;
 	size_t mmap_sz;
@@ -1975,7 +1992,7 @@ bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type,
 	map->sec_idx = sec_idx;
 	map->sec_offset = 0;
 	map->real_name = strdup(real_name);
-	map->name = internal_map_name(obj, real_name);
+	map->name = internal_map_name(obj, real_name, type);
 	if (!map->real_name || !map->name) {
 		zfree(&map->real_name);
 		zfree(&map->name);
@@ -1983,7 +2000,7 @@ bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type,
 	}
 
 	def = &map->def;
-	def->type = BPF_MAP_TYPE_ARRAY;
+	def->type = is_percpu ? BPF_MAP_TYPE_PERCPU_ARRAY : BPF_MAP_TYPE_ARRAY;
 	def->key_size = sizeof(int);
 	def->value_size = data_sz;
 	def->max_entries = 1;
@@ -1996,8 +2013,9 @@ bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type,
 	if (map_is_mmapable(obj, map))
 		def->map_flags |= BPF_F_MMAPABLE;
 
-	pr_debug("map '%s' (global data): at sec_idx %d, offset %zu, flags %x.\n",
-		 map->name, map->sec_idx, map->sec_offset, def->map_flags);
+	pr_debug("map '%s' (global %sdata): at sec_idx %d, offset %zu, flags %x.\n",
+		 map->name, is_percpu ? "percpu " : "", map->sec_idx,
+		 map->sec_offset, def->map_flags);
 
 	mmap_sz = bpf_map_mmap_sz(map);
 	map->mmaped = mmap(NULL, mmap_sz, PROT_READ | PROT_WRITE,
@@ -2057,6 +2075,13 @@ static int bpf_object__init_global_data_maps(struct bpf_object *obj)
 							    NULL,
 							    sec_desc->data->d_size);
 			break;
+		case SEC_PERCPU:
+			sec_name = elf_sec_name(obj, elf_sec_by_idx(obj, sec_idx));
+			err = bpf_object__init_internal_map(obj, LIBBPF_MAP_PERCPU,
+							    sec_name, sec_idx,
+							    sec_desc->data->d_buf,
+							    sec_desc->data->d_size);
+			break;
 		default:
 			/* skip */
 			break;
@@ -4016,6 +4041,11 @@ static int bpf_object__elf_collect(struct bpf_object *obj)
 				sec_desc->sec_type = SEC_RODATA;
 				sec_desc->shdr = sh;
 				sec_desc->data = data;
+			} else if (strcmp(name, PERCPU_SEC) == 0 ||
+				   str_has_pfx(name, PERCPU_SEC ".")) {
+				sec_desc->sec_type = SEC_PERCPU;
+				sec_desc->shdr = sh;
+				sec_desc->data = data;
 			} else if (strcmp(name, STRUCT_OPS_SEC) == 0 ||
 				   strcmp(name, STRUCT_OPS_LINK_SEC) == 0 ||
 				   strcmp(name, "?" STRUCT_OPS_SEC) == 0 ||
@@ -4544,6 +4574,7 @@ static bool bpf_object__shndx_is_data(const struct bpf_object *obj,
 	case SEC_BSS:
 	case SEC_DATA:
 	case SEC_RODATA:
+	case SEC_PERCPU:
 		return true;
 	default:
 		return false;
@@ -4569,6 +4600,8 @@ bpf_object__section_to_libbpf_map_type(const struct bpf_object *obj, int shndx)
 		return LIBBPF_MAP_DATA;
 	case SEC_RODATA:
 		return LIBBPF_MAP_RODATA;
+	case SEC_PERCPU:
+		return LIBBPF_MAP_PERCPU;
 	default:
 		return LIBBPF_MAP_UNSPEC;
 	}
@@ -4944,7 +4977,7 @@ static int map_fill_btf_type_info(struct bpf_object *obj, struct bpf_map *map)
 
 	/*
 	 * LLVM annotates global data differently in BTF, that is,
-	 * only as '.data', '.bss' or '.rodata'.
+	 * only as '.data', '.bss', '.percpu' or '.rodata'.
 	 */
 	if (!bpf_map__is_internal(map))
 		return -ENOENT;
@@ -5293,18 +5326,20 @@ static int
 bpf_object__populate_internal_map(struct bpf_object *obj, struct bpf_map *map)
 {
 	enum libbpf_map_type map_type = map->libbpf_type;
+	bool is_percpu = map_type == LIBBPF_MAP_PERCPU;
+	const __u64 update_flags = is_percpu ? BPF_F_ALL_CPUS : 0;
 	int err, zero = 0;
 	size_t mmap_sz;
 
 	if (obj->gen_loader) {
 		bpf_gen__map_update_elem(obj->gen_loader, map - obj->maps,
-					 map->mmaped, map->def.value_size);
+					 map->mmaped, map->def.value_size, update_flags);
 		if (map_type == LIBBPF_MAP_RODATA || map_type == LIBBPF_MAP_KCONFIG)
 			bpf_gen__map_freeze(obj->gen_loader, map - obj->maps);
 		return 0;
 	}
 
-	err = bpf_map_update_elem(map->fd, &zero, map->mmaped, 0);
+	err = bpf_map_update_elem(map->fd, &zero, map->mmaped, update_flags);
 	if (err) {
 		err = -errno;
 		pr_warn("map '%s': failed to set initial contents: %s\n",
@@ -5349,6 +5384,13 @@ bpf_object__populate_internal_map(struct bpf_object *obj, struct bpf_map *map)
 			return err;
 		}
 		map->mmaped = mmaped;
+	} else if (is_percpu) {
+		if (mprotect(map->mmaped, mmap_sz, PROT_READ)) {
+			err = -errno;
+			pr_warn("map '%s': failed to mprotect() contents: %s\n",
+				bpf_map__name(map), errstr(err));
+			return err;
+		}
 	} else if (map->mmaped) {
 		munmap(map->mmaped, mmap_sz);
 		map->mmaped = NULL;
@@ -5624,9 +5666,16 @@ bpf_object__create_maps(struct bpf_object *obj)
 		 * runtime due to bpf_program__set_autoload(prog, false),
 		 * bpf_object loading will succeed just fine even on old
 		 * kernels.
+		 * Same skipping applies to percpu data.
 		 */
-		if (bpf_map__is_internal(map) && !kernel_supports(obj, FEAT_GLOBAL_DATA))
-			map->autocreate = false;
+		if (bpf_map__is_internal(map)) {
+			bool is_percpu = map->libbpf_type == LIBBPF_MAP_PERCPU;
+			enum kern_feature_id feat_id;
+
+			feat_id = is_percpu ? FEAT_PERCPU_DATA : FEAT_GLOBAL_DATA;
+			if (!kernel_supports(obj, feat_id))
+				map->autocreate = false;
+		}
 
 		if (!map->autocreate) {
 			pr_debug("map '%s': skipped auto-creating...\n", map->name);
@@ -10807,11 +10856,16 @@ static bool map_uses_real_name(const struct bpf_map *map)
 	 * such map's corresponding ELF section name as a map name.
 	 * This check distinguishes .data/.rodata from .data.* and .rodata.*
 	 * maps to know which name has to be returned to the user.
+	 * Map name of the custom .percpu.* maps might be truncated to
+	 * BPF_OBJ_NAME_LEN-1 chars in internal_map_name(). Hence, percpu data
+	 * maps must use real name for their user-visible name.
 	 */
 	if (map->libbpf_type == LIBBPF_MAP_DATA && strcmp(map->real_name, DATA_SEC) != 0)
 		return true;
 	if (map->libbpf_type == LIBBPF_MAP_RODATA && strcmp(map->real_name, RODATA_SEC) != 0)
 		return true;
+	if (map->libbpf_type == LIBBPF_MAP_PERCPU)
+		return true;
 	return false;
 }
 
@@ -10976,7 +11030,8 @@ int bpf_map__set_value_size(struct bpf_map *map, __u32 size)
 		size_t mmap_old_sz, mmap_new_sz;
 		int err;
 
-		if (map->def.type != BPF_MAP_TYPE_ARRAY)
+		if (map->def.type != BPF_MAP_TYPE_ARRAY &&
+		    map->def.type != BPF_MAP_TYPE_PERCPU_ARRAY)
 			return libbpf_err(-EOPNOTSUPP);
 
 		mmap_old_sz = bpf_map_mmap_sz(map);
diff --git a/tools/lib/bpf/libbpf_internal.h b/tools/lib/bpf/libbpf_internal.h
index 7a74abb904f8b..4c46d34fc055d 100644
--- a/tools/lib/bpf/libbpf_internal.h
+++ b/tools/lib/bpf/libbpf_internal.h
@@ -401,6 +401,8 @@ enum kern_feature_id {
 	FEAT_BTF_LAYOUT,
 	/* Kernel supports BPF syscall common attributes */
 	FEAT_BPF_SYSCALL_COMMON_ATTRS,
+	/* Kernel supports percpu data */
+	FEAT_PERCPU_DATA,
 	__FEAT_CNT,
 };
 
diff --git a/tools/lib/bpf/skel_internal.h b/tools/lib/bpf/skel_internal.h
index 53fee53d36d51..1f3f332dffbe9 100644
--- a/tools/lib/bpf/skel_internal.h
+++ b/tools/lib/bpf/skel_internal.h
@@ -131,8 +131,10 @@ static inline void skel_free_map_data(void *p, __u64 addr, size_t sz)
 {
 	if (addr != ~0ULL)
 		kvfree(p);
-	/* When addr == ~0ULL the 'p' points to
-	 * ((struct bpf_array *)map)->value. See skel_finalize_map_data.
+	/*
+	 * When addr == ~0ULL the init buffer has already been released.
+	 * For skel_finalize_map_data(), 'p' points to
+	 * ((struct bpf_array *)map)->value.
 	 */
 }
 
@@ -170,6 +172,15 @@ static inline void *skel_finalize_map_data(__u64 *init_val, size_t mmap_sz, int
 	return addr;
 }
 
+static inline int skel_protect_map_data(void *p, __u64 *init_val, size_t sz)
+{
+	(void)sz;
+
+	kvfree(p);
+	*init_val = ~0ULL;
+	return 0;
+}
+
 #else
 
 static inline void *skel_alloc(size_t size)
@@ -208,6 +219,15 @@ static inline void *skel_finalize_map_data(__u64 *init_val, size_t mmap_sz, int
 		return NULL;
 	return addr;
 }
+
+static inline int skel_protect_map_data(void *p, __u64 *init_val, size_t sz)
+{
+	(void)init_val;
+
+	if (mprotect(p, sz, PROT_READ))
+		return -errno;
+	return 0;
+}
 #endif
 
 static inline int skel_closenz(int fd)
diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile
index d3655a7064826..560ce4016fbff 100644
--- a/tools/testing/selftests/bpf/Makefile
+++ b/tools/testing/selftests/bpf/Makefile
@@ -531,7 +531,7 @@ LSKELS_SIGNED := fentry_test.c fexit_test.c atomics.c
 
 # Generate both light skeleton and libbpf skeleton for these
 LSKELS_EXTRA := test_ksyms_module.c test_ksyms_weak.c kfunc_call_test.c \
-	kfunc_call_test_subprog.c
+	kfunc_call_test_subprog.c test_global_percpu_data.c
 SKEL_BLACKLIST += $$(LSKELS) $$(LSKELS_SIGNED)
 
 test_static_linked.skel.h-deps := test_static_linked1.bpf.o test_static_linked2.bpf.o
diff --git a/tools/testing/selftests/bpf/prog_tests/global_data_init.c b/tools/testing/selftests/bpf/prog_tests/global_data_init.c
index 8466332d7406f..7d6bda9092954 100644
--- a/tools/testing/selftests/bpf/prog_tests/global_data_init.c
+++ b/tools/testing/selftests/bpf/prog_tests/global_data_init.c
@@ -1,5 +1,8 @@
 // SPDX-License-Identifier: GPL-2.0
 #include <test_progs.h>
+#include "bpf/libbpf_internal.h"
+#include "test_global_percpu_data.skel.h"
+#include "test_global_percpu_data.lskel.h"
 
 void test_global_data_init(void)
 {
@@ -60,3 +63,336 @@ void test_global_data_init(void)
 	free(newval);
 	bpf_object__close(obj);
 }
+
+static void test_percpu_data_on_cpus(struct bpf_map *map, int map_fd, int prog_fd, int *runp)
+{
+	struct test_global_percpu_data__percpu *data = NULL;
+	int i, err, key = 0, num_online, run = 0;
+	__u64 args[2] = {0x1234ULL, 0x5678ULL};
+	size_t data_sz;
+	bool *online;
+	LIBBPF_OPTS(bpf_test_run_opts, topts,
+		    .ctx_in = args,
+		    .ctx_size_in = sizeof(args),
+		    .flags = BPF_F_TEST_RUN_ON_CPU,
+	);
+
+	err = parse_cpu_mask_file("/sys/devices/system/cpu/online", &online, &num_online);
+	if (!ASSERT_OK(err, "parse_cpu_mask_file"))
+		return;
+
+	data_sz = map ? bpf_map__value_size(map) : sizeof(*data);
+	data = calloc(1, data_sz);
+	if (!ASSERT_OK_PTR(data, "calloc percpu data"))
+		goto out;
+
+	/* run on every online-CPU */
+	for (i = 0; i < num_online; i++) {
+		__u64 flags;
+
+		if (!online[i])
+			continue;
+
+		topts.cpu = i;
+		topts.retval = -1;
+		err = bpf_prog_test_run_opts(prog_fd, &topts);
+		ASSERT_OK(err, "bpf_prog_test_run_opts");
+		ASSERT_EQ(topts.retval, 0, "bpf_prog_test_run_opts retval");
+
+		memset(data, 0, data_sz);
+		flags = ((__u64) i << 32) | BPF_F_CPU;
+		if (map)
+			err = bpf_map__lookup_elem(map, &key, sizeof(key), data, data_sz, flags);
+		else
+			err = bpf_map_lookup_elem_flags(map_fd, &key, data, flags);
+		if (!ASSERT_OK(err, "lookup_elem on cpu"))
+			break;
+
+		ASSERT_EQ(*runp, ++run, "run");
+		ASSERT_EQ(data->cpu_id[0], i, "cpu_id");
+		ASSERT_EQ(data->data, 1, "data");
+		ASSERT_TRUE(data->set, "set");
+		ASSERT_EQ(data->nums[6], 0xc0de, "nums[6]");
+		ASSERT_EQ(data->struct_data.i, 1, "struct_data.i");
+		ASSERT_TRUE(data->struct_data.set, "struct_data.set");
+		ASSERT_EQ(data->struct_data.nums[6], 0xc0de, "struct_data.nums[6]");
+	}
+
+out:
+	free(data);
+	free(online);
+}
+
+static void test_global_percpu_data_init(void)
+{
+	struct test_global_percpu_data__percpu init_value = {};
+	struct test_global_percpu_data__percpu *init_data;
+	const __u32 desired_sz = sysconf(_SC_PAGE_SIZE);
+	struct test_global_percpu_data *skel = NULL;
+	size_t init_data_sz;
+	struct bpf_map *map;
+	int prog_fd, err;
+
+	skel = test_global_percpu_data__open();
+	if (!ASSERT_OK_PTR(skel, "test_global_percpu_data__open"))
+		goto out;
+	if (!ASSERT_OK_PTR(skel->percpu, "skel->percpu"))
+		goto out;
+	if (!ASSERT_OK_PTR(skel->data_percpu, "skel->data_percpu"))
+		goto out;
+	if (!ASSERT_OK_PTR(skel->percpu_data, "skel->percpu_data"))
+		goto out;
+	if (!ASSERT_OK_PTR(skel->percpu_looooooooong, "skel->percpu_looooooooong"))
+		goto out;
+
+	ASSERT_STREQ(bpf_map__name(skel->maps.percpu_data), ".percpu.data",
+		     ".percpu.data map name");
+	ASSERT_STREQ(bpf_map__name(skel->maps.data_percpu), ".data.percpu",
+		     ".data.percpu map name");
+	ASSERT_STREQ(bpf_map__name(skel->maps.percpu_looooooooong), ".percpu.looooooooong",
+		     "long map name");
+	ASSERT_STREQ(bpf_map__name(skel->maps.percpu), ".percpu", "map name");
+	ASSERT_EQ(skel->percpu->data, -1, "skel->percpu->data");
+	ASSERT_FALSE(skel->percpu->set, "skel->percpu->set");
+	ASSERT_EQ(skel->percpu->nums[6], 0, "skel->percpu->nums[6]");
+	ASSERT_EQ(skel->percpu->struct_data.i, -1, "struct_data.i");
+	ASSERT_FALSE(skel->percpu->struct_data.set, "struct_data.set");
+	ASSERT_EQ(skel->percpu->struct_data.nums[6], 0, "struct_data.nums[6]");
+
+	map = skel->maps.percpu;
+	if (!ASSERT_EQ(bpf_map__type(map), BPF_MAP_TYPE_PERCPU_ARRAY, "bpf_map__type"))
+		goto out;
+
+	init_value.data = 2;
+	init_value.nums[6] = -1;
+	init_value.struct_data.i = 2;
+	init_value.struct_data.nums[6] = -1;
+	err = bpf_map__set_initial_value(map, &init_value, sizeof(init_value));
+	if (!ASSERT_OK(err, "bpf_map__set_initial_value"))
+		goto out;
+
+	init_data = bpf_map__initial_value(map, &init_data_sz);
+	if (!ASSERT_OK_PTR(init_data, "bpf_map__initial_value"))
+		goto out;
+
+	ASSERT_EQ(init_data->data, init_value.data, "init_value data");
+	ASSERT_EQ(init_data->set, init_value.set, "init_value set");
+	ASSERT_EQ(init_data->struct_data.i, init_value.struct_data.i, "init_value struct_data.i");
+	ASSERT_EQ(init_data->struct_data.nums[6], init_value.struct_data.nums[6],
+		  "init_value struct_data.nums[6]");
+	ASSERT_EQ(init_data_sz, sizeof(init_value), "init_value size");
+	ASSERT_EQ((void *) init_data, (void *) skel->percpu, "skel->percpu eq init_data");
+	ASSERT_EQ(skel->percpu->data, init_value.data, "skel->percpu->data");
+	ASSERT_EQ(skel->percpu->set, init_value.set, "skel->percpu->set");
+	ASSERT_EQ(skel->percpu->struct_data.i, init_value.struct_data.i,
+		  "skel->percpu->struct_data.i");
+	ASSERT_EQ(skel->percpu->struct_data.nums[6], init_value.struct_data.nums[6],
+		  "skel->percpu->struct_data.nums[6]");
+
+	ASSERT_GT(desired_sz, sizeof(init_value), "desired_sz");
+	err = bpf_map__set_value_size(map, desired_sz);
+	if (!ASSERT_OK(err, "bpf_map__set_value_size"))
+		goto out;
+	if (!ASSERT_EQ(bpf_map__value_size(map), desired_sz, "percpu value size"))
+		goto out;
+	if (!ASSERT_NEQ(bpf_map__btf_value_type_id(map), 0, "percpu BTF value type"))
+		goto out;
+
+	init_data = bpf_map__initial_value(map, &init_data_sz);
+	if (!ASSERT_OK_PTR(init_data, "resized bpf_map__initial_value"))
+		goto out;
+	if (!ASSERT_EQ(init_data_sz, desired_sz, "resized initial value size"))
+		goto out;
+	if (!ASSERT_EQ(init_data->data, init_value.data, "resized initial value data"))
+		goto out;
+
+	err = test_global_percpu_data__load(skel);
+	if (!ASSERT_OK(err, "test_global_percpu_data__load"))
+		goto out;
+
+	ASSERT_OK_PTR(skel->percpu, "skel->percpu");
+
+	prog_fd = bpf_program__fd(skel->progs.update_percpu_data);
+	test_percpu_data_on_cpus(map, bpf_map__fd(map), prog_fd, &skel->bss->run);
+
+out:
+	test_global_percpu_data__destroy(skel);
+}
+
+static void test_global_percpu_data_lskel(void)
+{
+	struct test_global_percpu_data_lskel *lskel = NULL;
+	int prog_fd, map_fd;
+
+	lskel = test_global_percpu_data_lskel__open_and_load();
+	if (!ASSERT_OK_PTR(lskel, "test_global_percpu_data_lskel__open_and_load"))
+		goto out;
+
+	map_fd = lskel->maps.percpu.map_fd;
+	prog_fd = lskel->progs.update_percpu_data.prog_fd;
+	test_percpu_data_on_cpus(NULL, map_fd, prog_fd, &lskel->bss->run);
+
+out:
+	test_global_percpu_data_lskel__destroy(lskel);
+}
+
+static int create_rdonly_percpu_array(void)
+{
+	LIBBPF_OPTS(bpf_map_create_opts, map_opts,
+		    .map_flags = BPF_F_RDONLY_PROG,
+	);
+	int key = 0, map_fd, err;
+	__u64 value = 0;
+
+	map_fd = bpf_map_create(BPF_MAP_TYPE_PERCPU_ARRAY, "percpu_ro_map", sizeof(int),
+				sizeof(__u64), 1, &map_opts);
+	if (!ASSERT_GE(map_fd, 0, "bpf_map_create"))
+		return -1;
+
+	err = bpf_map_update_elem(map_fd, &key, &value, BPF_F_ALL_CPUS);
+	if (!ASSERT_OK(err, "bpf_map_update_elem"))
+		goto out;
+
+	err = bpf_map_freeze(map_fd);
+	if (!ASSERT_OK(err, "bpf_map_freeze"))
+		goto out;
+
+	return map_fd;
+
+out:
+	close(map_fd);
+	return -1;
+}
+
+static void test_global_percpu_data_rdonly_direct_read(void)
+{
+	/*
+	 * Raw instructions with manually prepared rdonly percpu_array map
+	 * for testing direct-read global percpu data, because libbpf
+	 * doesn't have rdonly internal percpu_array map support for
+	 * global percpu data.
+	 */
+	struct bpf_insn insns[] = {
+		BPF_LD_MAP_VALUE(BPF_REG_1, 0, 0),
+		BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, 0),
+		BPF_EXIT_INSN(),
+	};
+	int map_fd, prog_fd;
+
+	map_fd = create_rdonly_percpu_array();
+	if (map_fd < 0)
+		return;
+
+	insns[0].imm = map_fd;
+	prog_fd = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, "percpu_ro_prog", "GPL", insns,
+				ARRAY_SIZE(insns), NULL);
+	if (ASSERT_GE(prog_fd, 0, "bpf_prog_load"))
+		close(prog_fd);
+	close(map_fd);
+}
+
+static void test_global_percpu_data_rdonly_direct_write(void)
+{
+	LIBBPF_OPTS(bpf_prog_load_opts, prog_opts);
+	/* See the comment in test_global_percpu_data_rdonly_direct_read() */
+	struct bpf_insn insns[] = {
+		BPF_LD_MAP_VALUE(BPF_REG_1, 0, 0),
+		BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, 0),
+		BPF_ST_MEM(BPF_DW, BPF_REG_1, 0, 0),
+		BPF_EXIT_INSN(),
+	};
+	char log_buf[256] = {};
+	int map_fd, prog_fd;
+
+	prog_opts.log_buf = log_buf;
+	prog_opts.log_size = sizeof(log_buf);
+	prog_opts.log_level = 1;
+
+	map_fd = create_rdonly_percpu_array();
+	if (map_fd < 0)
+		return;
+
+	insns[0].imm = map_fd;
+	prog_fd = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, "percpu_ro_prog", "GPL", insns,
+				ARRAY_SIZE(insns), &prog_opts);
+	if (!ASSERT_LT(prog_fd, 0, "bpf_prog_load"))
+		close(prog_fd);
+	else
+		ASSERT_HAS_SUBSTR(log_buf, "write into map forbidden", "verifier log");
+	close(map_fd);
+}
+
+static void test_global_percpu_data_verifier_log(void)
+{
+	RUN_TESTS(test_global_percpu_data);
+}
+
+static void test_global_percpu_data_iter(void)
+{
+	DECLARE_LIBBPF_OPTS(bpf_iter_attach_opts, opts);
+	struct test_global_percpu_data *skel;
+	union bpf_iter_link_info linfo = {};
+	struct bpf_link *link = NULL;
+	int fd, num_cpus, len, err;
+	char buf[16];
+
+	num_cpus = libbpf_num_possible_cpus();
+	if (!ASSERT_GT(num_cpus, 0, "libbpf_num_possible_cpus"))
+		return;
+
+	skel = test_global_percpu_data__open();
+	if (!ASSERT_OK_PTR(skel, "test_global_percpu_data__open"))
+		return;
+
+	skel->rodata->num_cpus = num_cpus;
+	skel->rodata->offsetof_num = offsetof(struct test_global_percpu_data__percpu, struct_data);
+	skel->rodata->offsetof_num += sizeof(skel->percpu->struct_data) - sizeof(int);
+	skel->rodata->elem_sz = roundup(sizeof(struct test_global_percpu_data__percpu), 8);
+	skel->percpu->struct_data.nums[6] = 0xc0de;
+
+	err = test_global_percpu_data__load(skel);
+	if (!ASSERT_OK(err, "test_global_percpu_data__load"))
+		goto out;
+
+	linfo.map.map_fd = bpf_map__fd(skel->maps.percpu);
+	opts.link_info = &linfo;
+	opts.link_info_len = sizeof(linfo);
+	link = bpf_program__attach_iter(skel->progs.dump_percpu_data, &opts);
+	if (!ASSERT_OK_PTR(link, "bpf_program__attach_iter"))
+		goto out;
+
+	fd = bpf_iter_create(bpf_link__fd(link));
+	if (!ASSERT_GE(fd, 0, "bpf_iter_create"))
+		goto out;
+
+	while ((len = read(fd, buf, sizeof(buf))) > 0)
+		do { } while (0);
+	ASSERT_EQ(len, 0, "read iter");
+	ASSERT_TRUE(skel->bss->run_iter, "run_iter");
+	ASSERT_EQ(skel->bss->percpu_data_sum, 0xc0de * num_cpus, "percpu_data_sum");
+
+	close(fd);
+out:
+	bpf_link__destroy(link);
+	test_global_percpu_data__destroy(skel);
+}
+
+void test_global_percpu_data(void)
+{
+	if (!feat_supported(NULL, FEAT_PERCPU_DATA)) {
+		test__skip();
+		return;
+	}
+
+	if (test__start_subtest("init"))
+		test_global_percpu_data_init();
+	if (test__start_subtest("lskel"))
+		test_global_percpu_data_lskel();
+	if (test__start_subtest("rdonly_direct_read"))
+		test_global_percpu_data_rdonly_direct_read();
+	if (test__start_subtest("rdonly_direct_write"))
+		test_global_percpu_data_rdonly_direct_write();
+	test_global_percpu_data_verifier_log();
+	if (test__start_subtest("iter"))
+		test_global_percpu_data_iter();
+}
diff --git a/tools/testing/selftests/bpf/progs/test_global_percpu_data.c b/tools/testing/selftests/bpf/progs/test_global_percpu_data.c
new file mode 100644
index 0000000000000..71ff8d1bf49eb
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/test_global_percpu_data.c
@@ -0,0 +1,89 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+/* Used for testing map name. */
+int loong SEC(".percpu.looooooooong");
+int data3 SEC(".data.percpu");
+int data2 SEC(".percpu.data");
+
+int run;
+/* cpu_id as array to verify map value resizing. */
+int cpu_id[1] SEC(".percpu");
+int data SEC(".percpu") = -1;
+int nums[7] SEC(".percpu");
+bool set SEC(".percpu") = false;
+struct {
+	char set;
+	int i;
+	int nums[7];
+} struct_data SEC(".percpu") = {
+	.set = 0,
+	.i = -1,
+};
+
+SEC("raw_tp/task_rename")
+__auxiliary
+int update_percpu_data(void *ctx)
+{
+	struct_data.nums[6] = 0xc0de;
+	struct_data.set = 1;
+	struct_data.i = 1;
+	nums[6] = 0xc0de;
+	data = 1;
+	run++;
+	set = true;
+	cpu_id[0] = bpf_get_smp_processor_id();
+	return 0;
+}
+
+static const char fmt[] SEC(".percpu.fmt") = "data %d\n";
+
+SEC("?kprobe")
+__failure __msg("R{{[0-9]+}} points to percpu_array map which cannot be used as const string")
+int verifier_strncmp(void *ctx)
+{
+	return bpf_strncmp("test", 5, fmt);
+}
+
+SEC("?kprobe")
+__failure __msg("R{{[0-9]+}} points to percpu_array map which cannot be used as const string")
+int verifier_snprintf(void *ctx)
+{
+	u64 args[] = { data };
+	char buf[128];
+	int len;
+
+	len = bpf_snprintf(buf, sizeof(buf), fmt, args, sizeof(args));
+	if (len > 0)
+		bpf_printk("snprintf: %s\n", buf);
+	return 0;
+}
+
+volatile const __u32 num_cpus = 0;
+volatile const int offsetof_num;
+volatile const int elem_sz;
+__u32 percpu_data_sum = 0;
+bool run_iter = false;
+
+SEC("iter/bpf_map_elem")
+__auxiliary
+int dump_percpu_data(struct bpf_iter__bpf_map_elem *ctx)
+{
+	void *pptr = ctx->value;
+	int i;
+
+	if (!pptr)
+		return 0;
+
+	run_iter = true;
+
+	for (i = 0; i < num_cpus; i++) {
+		percpu_data_sum += *(int *) (pptr + offsetof_num);
+		pptr += elem_sz;
+	}
+	return 0;
+}
+
+char _license[] SEC("license") = "GPL";

gemini-3.1-pro-preview

							input: 23627
							output: 229
							thoughts: 1769

**KMSAN Assessment: Global Per-CPU Data in BPF**

Okay, so I've been digging into this patch series that introduces support for global percpu data within BPF. It's essentially adding the ability to define `.percpu` sections in BPF programs, which are then mapped to `BPF_MAP_TYPE_PERCPU_ARRAY` maps. My focus here is ensuring this doesn't create any new avenues for uninitialized memory issues, leaks, or other subtle vulnerabilities that could slip through.

The patch series touches several areas. I'm seeing modifications in `arraymap.c` for handling these percpu array map operations, changes in the verifier (`bpf/verifier.c`) to correctly process `BPF_MAP_TYPE_PERCPU_ARRAY` during memory reads, and updates in `bpftool/gen.c` and `libbpf` to support the `.percpu` section and the necessary map types. Tests have also been added, of course.

The core idea is that libbpf creates a `BPF_MAP_TYPE_PERCPU_ARRAY` map for each `.percpu` section. The contents of this section are then used to initialize the map. The verifier marks the registers as "unknown" when encountering percpu array, meaning it's not directly readable. However, I've noticed a modification to `bpf_map_direct_read`, which explicitly *disables* direct reads for percpu arrays. So, that's good; it eliminates a potential risk.

I was curious about `percpu_array_map_direct_value_addr`. It's used by the verifier during JIT compilation. This is just for BPF JIT instruction patching. The memory itself is initialized using `bpf_map_update_elem` with `BPF_F_ALL_CPUS`. The memory is handled by either `bpf_map_alloc_pages` or the standard percpu allocator, neither of which is new here.

Thinking through potential uninitialized memory risks, I'm not seeing any. There are no new kernel structures exposed to userspace. Initialization happens either through zeroing during allocation or by data provided via `bpf_map_update_elem`. The verifier changes largely just adapt to the existing `BPF_MAP_TYPE_PERCPU_ARRAY` used as global data. Since user space is providing the initial data from the ELF, it's their responsibility to provide the right data, and the kernel side zeros-out memory.

Let me break down my KMSAN concerns point-by-point:

* **Copying Uninitialized Data to Userspace:** Nope, doesn't happen.
* **Using Uninitialized Data in Branching:** No, that's not occurring.
* **Exposing Uninitialized Memory:** Again, no.

Therefore, this *doesn't* warrant a dedicated KMSAN fuzzing campaign. KASAN should be sufficient. I believe that KASAN will catch any out-of-bounds, use-after-free, or other memory corruption issues arising from JIT patching or verifier errors.

So, in summary, this patch series enhances BPF by allowing global percpu data using `.percpu` sections. The memory initialization process is safe, leveraging existing map allocation routines and libbpf's interaction with ELF data. There is no new uninitialized buffers, or structures. The core logic handles JIT compilation correctly to ensure there aren't any potential information leaks. While introducing new code always brings risk, any bugs are most likely to be detectable by standard memory safety tools like KASAN, and this should not require a dedicated KMSAN focus.

{
  "KMSANReasoning": "The patch series adds support for global percpu data in BPF by allowing `.percpu` sections to be mapped to `BPF_MAP_TYPE_PERCPU_ARRAY` maps. The changes involve libbpf parsing the new section, bpftool generating skeleton code, and the BPF verifier handling direct value access and JIT patching for percpu arrays.\n\nThere are no new kernel structures sent to user space, no new uninitialized buffers allocated, and no complex data structures where padding might be leaked. The percpu array memory is allocated using existing map allocation routines (which handle initialization) and populated by libbpf from the ELF section.\n\nAny potential bugs introduced by these changes would likely be out-of-bounds accesses, invalid JIT patching, or verifier logic errors, which are well-covered by KASAN and standard bug detectors. There is no specific risk of uninitialized memory usage or info-leaks that would require a dedicated KMSAN fuzzing session.",
  "NeedsKMSAN": false
}

{
  "KMSANReasoning": "The patch series adds support for global percpu data in BPF by allowing `.percpu` sections to be mapped to `BPF_MAP_TYPE_PERCPU_ARRAY` maps. The changes involve libbpf parsing the new section, bpftool generating skeleton code, and the BPF verifier handling direct value access and JIT patching for percpu arrays.\n\nThere are no new kernel structures sent to user space, no new uninitialized buffers allocated, and no complex data structures where padding might be leaked. The percpu array memory is allocated using existing map allocation routines (which handle initialization) and populated by libbpf from the ELF section.\n\nAny potential bugs introduced by these changes would likely be out-of-bounds accesses, invalid JIT patching, or verifier logic errors, which are well-covered by KASAN and standard bug detectors. There is no specific risk of uninitialized memory usage or info-leaks that would require a dedicated KMSAN fuzzing session.",
  "NeedsKMSAN": false
}

LLM Calls Summary:
Total Calls	Total Tokens	Avg Tokens	Total Duration (Seconds)	Avg Duration (Seconds)

Tool Calls Summary:
Total Calls	Total Duration (Seconds)	Avg Duration (Seconds)