| Seq | Timestamp | Type | Name | Duration |
|---|---|---|---|---|
| 0/0 | 2026/07/04 05:24 | flow | patch-triage |
0mArgs:null Results: {
"EnableConfigs": null,
"FocusSymbols": [],
"Reasoning": "The patch only adds and modifies BPF selftests in tools/testing/selftests/bpf/. It does not make any functional changes to the core kernel, drivers, or architectures.",
"WorthFuzzing": false
} |
| 1/1 | 2026/07/04 05:24 | action | read-patch-diff |
0mArgs:null Results: {
"PatchDiff": "commit cd2b01880376591d96e530b1c5e11beef185c315\nAuthor: syz-cluster \u003ctriage@syzkaller.com\u003e\nDate: Sat Jul 4 05:24:42 2026 +0000\n\n syz-cluster: applied patch under review\n\ndiff --git a/tools/testing/selftests/bpf/cgroup_helpers.c b/tools/testing/selftests/bpf/cgroup_helpers.c\nindex 45cd0b479fe357..fe8ec07c610052 100644\n--- a/tools/testing/selftests/bpf/cgroup_helpers.c\n+++ b/tools/testing/selftests/bpf/cgroup_helpers.c\n@@ -188,6 +188,52 @@ int write_cgroup_file_parent(const char *relative_path, const char *file,\n \treturn __write_cgroup_file(cgroup_path, file, buf);\n }\n \n+static int __read_cgroup_file(const char *cgroup_path, const char *file,\n+\t\t\t char *buf, size_t buf_size)\n+{\n+\tchar file_path[PATH_MAX + 1];\n+\tssize_t len;\n+\tint fd;\n+\n+\tsnprintf(file_path, sizeof(file_path), \"%s/%s\", cgroup_path, file);\n+\tfd = open(file_path, O_RDONLY);\n+\tif (fd \u003c 0) {\n+\t\tlog_err(\"Opening %s\", file_path);\n+\t\treturn 1;\n+\t}\n+\n+\tlen = read(fd, buf, buf_size - 1);\n+\tclose(fd);\n+\tif (len \u003c 0) {\n+\t\tlog_err(\"Reading %s\", file_path);\n+\t\treturn 1;\n+\t}\n+\tbuf[len] = '\\0';\n+\treturn 0;\n+}\n+\n+/**\n+ * read_cgroup_file() - Read from a cgroup file\n+ * @relative_path: The cgroup path, relative to the workdir\n+ * @file: The name of the file in cgroupfs to read from\n+ * @buf: Buffer to read into; NUL-terminated on success\n+ * @buf_size: Size of @buf; at most @buf_size - 1 bytes are read\n+ *\n+ * Read from a file in the given cgroup's directory. As with reading any\n+ * cgroupfs control/stat file, @buf should be large enough to hold the whole\n+ * value in a single read().\n+ *\n+ * If successful, 0 is returned.\n+ */\n+int read_cgroup_file(const char *relative_path, const char *file,\n+\t\t char *buf, size_t buf_size)\n+{\n+\tchar cgroup_path[PATH_MAX - 24];\n+\n+\tformat_cgroup_path(cgroup_path, relative_path);\n+\treturn __read_cgroup_file(cgroup_path, file, buf, buf_size);\n+}\n+\n /**\n * setup_cgroup_environment() - Setup the cgroup environment\n *\ndiff --git a/tools/testing/selftests/bpf/cgroup_helpers.h b/tools/testing/selftests/bpf/cgroup_helpers.h\nindex 3857304be87410..1ed76dd3a1da27 100644\n--- a/tools/testing/selftests/bpf/cgroup_helpers.h\n+++ b/tools/testing/selftests/bpf/cgroup_helpers.h\n@@ -15,6 +15,8 @@ int write_cgroup_file(const char *relative_path, const char *file,\n \t\t const char *buf);\n int write_cgroup_file_parent(const char *relative_path, const char *file,\n \t\t\t const char *buf);\n+int read_cgroup_file(const char *relative_path, const char *file,\n+\t\t char *buf, size_t buf_size);\n int cgroup_setup_and_join(const char *relative_path);\n int get_root_cgroup(void);\n int create_and_get_cgroup(const char *relative_path);\ndiff --git a/tools/testing/selftests/bpf/config b/tools/testing/selftests/bpf/config\nindex adb25146e88ccf..4e75b4ea8649bf 100644\n--- a/tools/testing/selftests/bpf/config\n+++ b/tools/testing/selftests/bpf/config\n@@ -55,6 +55,7 @@ CONFIG_LIRC=y\n CONFIG_LIVEPATCH=y\n CONFIG_LWTUNNEL=y\n CONFIG_LWTUNNEL_BPF=y\n+CONFIG_MEMCG=y\n CONFIG_MODULE_SIG=y\n CONFIG_MODULE_SRCVERSION_ALL=y\n CONFIG_MODULE_UNLOAD=y\ndiff --git a/tools/testing/selftests/bpf/memcg_stat_reader.h b/tools/testing/selftests/bpf/memcg_stat_reader.h\nnew file mode 100644\nindex 00000000000000..72afebe95ccb82\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/memcg_stat_reader.h\n@@ -0,0 +1,35 @@\n+/* SPDX-License-Identifier: GPL-2.0 */\n+/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */\n+#ifndef __MEMCG_STAT_READER_H\n+#define __MEMCG_STAT_READER_H\n+\n+/*\n+ * One per-cgroup snapshot, produced by the BPF cgroup iterator and read back\n+ * from a BPF hash map keyed by cgroup id. The \"matched\" subset is always\n+ * populated so it can be compared field-by-field against what userspace parses\n+ * out of memory.stat / memory.current / memory.max. The \"full\" fold is only\n+ * populated when collect_full is set and exists to (a) force the extra kfunc\n+ * reads to happen (so the full-vs-matched timing is honest) and (b) give a\n+ * coarse, informational signal of how many fields the full path touched.\n+ */\n+struct memcg_stat_snapshot {\n+\t__u64 cgroup_id;\n+\n+\t/* Matched subset. Page-state values are in bytes (already unit-scaled\n+\t * by the kernel), so they compare directly against memory.stat.\n+\t */\n+\t__u64 anon;\t\t/* NR_ANON_MAPPED, bytes */\n+\t__u64 file;\t\t/* NR_FILE_PAGES, bytes */\n+\t__u64 shmem;\t\t/* NR_SHMEM, bytes */\n+\t__u64 file_mapped;\t/* NR_FILE_MAPPED, bytes */\n+\t__u64 pgfault;\t\t/* PGFAULT, count */\n+\t__u64 usage_pages;\t/* page_counter memory.usage, in PAGES */\n+\t__u64 max_pages;\t/* page_counter memory.max, in PAGES */\n+\n+\t/* Full-mode fold: sum and count of every field the full path read. */\n+\t__u64 full_sum;\n+\t__u32 full_fields;\n+\t__u32 pad;\n+};\n+\n+#endif /* __MEMCG_STAT_READER_H */\ndiff --git a/tools/testing/selftests/bpf/prog_tests/memcg_stat_churn.c b/tools/testing/selftests/bpf/prog_tests/memcg_stat_churn.c\nnew file mode 100644\nindex 00000000000000..3e386d0b4c033b\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/prog_tests/memcg_stat_churn.c\n@@ -0,0 +1,716 @@\n+// SPDX-License-Identifier: GPL-2.0\n+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */\n+\n+/*\n+ * memcg_stat_churn\n+ * ================\n+ * A load variant of the memcg_stat_reader benchmark. Where memcg_stat_reader\n+ * charges a quiescent tree once and then measures both readers against static\n+ * stats, this test keeps the memory-cgroup rstat perpetually DIRTY while it\n+ * measures:\n+ *\n+ * - Build a synthetic cgroup subtree (fanout x depth), same as the reader.\n+ * - Fork one \"churner\" process per selected leaf. Each churner joins its\n+ * leaf, pins a small resident anon set (so tree anon stays \u003e 0), then loops\n+ * mmap()+memset()+munmap() for the whole measurement window. The constant\n+ * charge/uncharge traffic keeps every touched memcg's per-cpu stats dirty,\n+ * so each reader pays a realistic flush/read cost instead of a warm no-op.\n+ * - While the churn runs, the parent repeatedly SAMPLES both readers:\n+ * (A) traditional: open/read/parse memory.stat (+current/+max) for every\n+ * cgroup from userspace;\n+ * (B) BPF: one SEC(\"iter.s/cgroup\") walk over the subtree calling the\n+ * memcg kfuncs into a hash map, drained once per sample.\n+ * Before each timed read the parent idles for a fixed gap (untimed) so the\n+ * tree re-accumulates a roughly fixed amount of dirty rstat; every read\n+ * (file/BPF x matched/full) therefore starts from approximately the same\n+ * state and pays its own rstat flush inside the timed region. The\n+ * file-vs-BPF order is also alternated across samples so residual jitter\n+ * doesn't systematically favour whichever reader runs first.\n+ * - Times are averaged over all samples and the file/BPF speedup ratio is\n+ * reported. The gap is the \"staleness / poll-interval\" knob: a larger gap\n+ * means a larger flush that both paths pay, so the ratio is more\n+ * conservative (see CHURN_GAP_US).\n+ *\n+ * The BPF program, its hash map and the snapshot struct are REUSED verbatim\n+ * from memcg_stat_reader (progs/memcg_stat_reader.c + memcg_stat_reader.h); only\n+ * the userspace load model and sampling loop are new here.\n+ *\n+ * Under churn the stats are a moving target, so this test does NOT do a\n+ * field-by-field BPF-vs-file equality check (that is memcg_stat_reader's job).\n+ * Pass/fail gates only on structural sanity -- the iterator visited every\n+ * cgroup and the tree carries some anon charge. The timing table and final\n+ * RATIO line are informational diagnostics, printed like any other test output\n+ * (i.e. under -v or on failure, never on a normal PASS).\n+ */\n+#include \u003ctest_progs.h\u003e\n+#include \u003cbpf/libbpf.h\u003e\n+#include \u003cbpf/btf.h\u003e\n+#include \u003cstdlib.h\u003e\n+#include \u003cstring.h\u003e\n+#include \u003ctime.h\u003e\n+#include \u003cunistd.h\u003e\n+#include \u003csys/mman.h\u003e\n+#include \u003csys/wait.h\u003e\n+#include \"cgroup_helpers.h\"\n+#include \"memcg_stat_reader.h\"\n+#include \"memcg_stat_reader.skel.h\"\n+\n+#define SUBTREE_ROOT\t\"/mcg_churn\"\n+\n+#define WARMUP_ITERS\t2\n+\n+struct cg_node {\n+\tchar rel[128];\n+\t__u64 id;\n+\tbool is_leaf;\n+};\n+\n+/* Field subset parsed from memory.stat (mirrors memcg_stat_reader). */\n+struct file_snap {\n+\t__u64 anon, file, shmem, file_mapped, pgfault;\n+\t__u64 current;\t\t/* memory.current, bytes */\n+\t__u64 max;\t\t/* memory.max, bytes (valid unless max_is_max) */\n+\t__u64 full_sum;\n+\t__u32 full_fields;\n+\tbool max_is_max;\n+};\n+\n+static volatile __u64 sink;\t/* keep the optimizer from eliding reads */\n+\n+static long long now_ns(void)\n+{\n+\tstruct timespec t;\n+\n+\tclock_gettime(CLOCK_MONOTONIC, \u0026t);\n+\treturn (long long)t.tv_sec * 1000000000LL + t.tv_nsec;\n+}\n+\n+/* ---- tree construction (same shape as memcg_stat_reader) --------------- */\n+\n+static struct cg_node *nodes;\n+static int n_nodes;\n+static int n_leaves;\n+\n+static int add_node(const char *rel, bool is_leaf, int *keep_fd)\n+{\n+\tint fd;\n+\n+\tfd = create_and_get_cgroup(rel);\n+\tif (fd \u003c 0)\n+\t\treturn -1;\n+\tif (keep_fd)\n+\t\t*keep_fd = fd;\n+\telse\n+\t\tclose(fd);\n+\n+\tstrncpy(nodes[n_nodes].rel, rel, sizeof(nodes[n_nodes].rel) - 1);\n+\tnodes[n_nodes].rel[sizeof(nodes[n_nodes].rel) - 1] = '\\0';\n+\tnodes[n_nodes].id = get_cgroup_id(rel);\n+\tnodes[n_nodes].is_leaf = is_leaf;\n+\tif (is_leaf)\n+\t\tn_leaves++;\n+\tn_nodes++;\n+\treturn 0;\n+}\n+\n+/* Recursively create children of @rel. @rel must already exist and be recorded. */\n+static int build_children(const char *rel, int fanout, int depth)\n+{\n+\tchar child[128];\n+\tint i;\n+\n+\tif (depth == 0)\n+\t\treturn 0;\n+\n+\t/* Enable memory on this interior node so its children get a memcg. */\n+\tif (enable_controllers(rel, \"memory\"))\n+\t\treturn -1;\n+\n+\tfor (i = 0; i \u003c fanout; i++) {\n+\t\tsnprintf(child, sizeof(child), \"%s/c%d\", rel, i);\n+\t\tif (add_node(child, depth == 1, NULL))\n+\t\t\treturn -1;\n+\t\tif (build_children(child, fanout, depth - 1))\n+\t\t\treturn -1;\n+\t}\n+\treturn 0;\n+}\n+\n+static size_t tree_capacity(int fanout, int depth)\n+{\n+\tsize_t total = 1, level = 1;\n+\tint d;\n+\n+\tfor (d = 0; d \u003c depth; d++) {\n+\t\tlevel *= fanout;\n+\t\ttotal += level;\n+\t}\n+\treturn total;\n+}\n+\n+static int build_tree(int fanout, int depth, int *root_fd)\n+{\n+\tn_nodes = 0;\n+\tn_leaves = 0;\n+\tnodes = calloc(tree_capacity(fanout, depth), sizeof(*nodes));\n+\tif (!nodes)\n+\t\treturn -1;\n+\n+\tif (add_node(SUBTREE_ROOT, depth == 0, root_fd))\n+\t\treturn -1;\n+\treturn build_children(SUBTREE_ROOT, fanout, depth);\n+}\n+\n+/* ---- churn load -------------------------------------------------------- */\n+\n+/*\n+ * Shared control block, mmap'd MAP_SHARED before the forks so the parent can\n+ * signal all churners to stop with a single write.\n+ */\n+struct churn_ctl {\n+\tvolatile int stop;\n+};\n+\n+static struct churn_ctl *ctl;\n+static pid_t *churn_pids;\n+static int n_churners;\n+static int churn_ready[2] = { -1, -1 };\t/* churner -\u003e parent \"ready\" barrier */\n+\n+/*\n+ * One churner process. Joins its leaf, pins a resident anon set so the tree\n+ * always carries some charge, signals readiness, then continuously faults in\n+ * and frees a private anon region until told to stop. Never returns.\n+ */\n+static void churner_child(const struct cg_node *leaf, size_t region_bytes,\n+\t\t\t size_t resident_bytes)\n+{\n+\tvoid *resident;\n+\n+\tclose(churn_ready[0]);\n+\n+\t/*\n+\t * cgroup_helpers builds paths from getpid(); in this forked child that\n+\t * differs from the parent that built the tree, so use the _parent\n+\t * (getppid()) variant to resolve the leaf under the parent's work dir.\n+\t */\n+\tif (join_parent_cgroup(leaf-\u003erel))\n+\t\t_exit(1);\n+\n+\tresident = mmap(NULL, resident_bytes, PROT_READ | PROT_WRITE,\n+\t\t\tMAP_ANONYMOUS | MAP_PRIVATE, -1, 0);\n+\tif (resident == MAP_FAILED)\n+\t\t_exit(2);\n+\tmemset(resident, 1, resident_bytes);\t/* fault in, keep mapped */\n+\n+\tif (write(churn_ready[1], \"x\", 1) != 1)\n+\t\t_exit(3);\n+\tclose(churn_ready[1]);\t/* so a sibling's early death yields EOF, not a parent hang */\n+\n+\twhile (!ctl-\u003estop) {\n+\t\tvoid *p = mmap(NULL, region_bytes, PROT_READ | PROT_WRITE,\n+\t\t\t MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);\n+\n+\t\tif (p == MAP_FAILED)\n+\t\t\tcontinue;\n+\t\tmemset(p, 1, region_bytes);\t/* fault in -\u003e anon charge */\n+\t\tmunmap(p, region_bytes);\t/* free -\u003e uncharge (keeps rstat dirty) */\n+\t}\n+\tmunmap(resident, resident_bytes);\n+\t_exit(0);\n+}\n+\n+/*\n+ * Fork one churner per @charge_fraction-th leaf. Returns 0 once every churner\n+ * has joined its leaf and pinned its resident set (so measurement starts under\n+ * steady-state load). On failure the caller's cleanup path calls\n+ * stop_churners() to reap whatever was started.\n+ */\n+static int start_churners(size_t region_bytes, size_t resident_bytes,\n+\t\t\t int charge_fraction)\n+{\n+\tint mod = charge_fraction \u003e 0 ? charge_fraction : 1;\n+\tint leaf_idx = 0;\n+\tint i;\n+\n+\tctl = mmap(NULL, sizeof(*ctl), PROT_READ | PROT_WRITE,\n+\t\t MAP_SHARED | MAP_ANONYMOUS, -1, 0);\n+\tif (!ASSERT_NEQ(ctl, MAP_FAILED, \"mmap churn_ctl\")) {\n+\t\tctl = NULL;\n+\t\treturn -1;\n+\t}\n+\tctl-\u003estop = 0;\n+\n+\tif (!ASSERT_OK(pipe(churn_ready), \"pipe churn_ready\"))\n+\t\treturn -1;\n+\n+\tchurn_pids = calloc(n_leaves, sizeof(*churn_pids));\n+\tif (!ASSERT_OK_PTR(churn_pids, \"calloc churn_pids\"))\n+\t\treturn -1;\n+\n+\tfor (i = 0; i \u003c n_nodes; i++) {\n+\t\tpid_t pid;\n+\n+\t\tif (!nodes[i].is_leaf)\n+\t\t\tcontinue;\n+\t\tif ((leaf_idx++ % mod) != 0)\n+\t\t\tcontinue;\n+\n+\t\tpid = fork();\n+\t\tif (pid \u003c 0) {\n+\t\t\tASSERT_GE(pid, 0, \"fork churner\");\n+\t\t\treturn -1;\n+\t\t}\n+\t\tif (pid == 0)\n+\t\t\tchurner_child(\u0026nodes[i], region_bytes, resident_bytes);\n+\n+\t\tchurn_pids[n_churners++] = pid;\n+\t}\n+\n+\t/* parent: this end is only for the children to signal on */\n+\tclose(churn_ready[1]);\n+\tchurn_ready[1] = -1;\n+\n+\t/* wait until every churner has joined + pinned its resident set */\n+\tfor (i = 0; i \u003c n_churners; i++) {\n+\t\tchar c;\n+\t\tssize_t r = read(churn_ready[0], \u0026c, 1);\n+\n+\t\tif (r == 0)\n+\t\t\tfprintf(stderr,\n+\t\t\t\t\"a churner exited before signaling ready (join_parent_cgroup/mmap failure?)\\n\");\n+\t\tif (!ASSERT_EQ(r, 1, \"churner ready\"))\n+\t\t\treturn -1;\n+\t}\n+\treturn 0;\n+}\n+\n+static void stop_churners(void)\n+{\n+\tint i, status;\n+\n+\tif (ctl)\n+\t\tctl-\u003estop = 1;\t\t\t/* release all churn loops */\n+\n+\tif (churn_ready[1] \u003e= 0) {\n+\t\tclose(churn_ready[1]);\n+\t\tchurn_ready[1] = -1;\n+\t}\n+\tif (churn_ready[0] \u003e= 0) {\n+\t\tclose(churn_ready[0]);\n+\t\tchurn_ready[0] = -1;\n+\t}\n+\n+\tfor (i = 0; i \u003c n_churners; i++) {\n+\t\tif (!churn_pids || churn_pids[i] \u003c= 0)\n+\t\t\tcontinue;\n+\t\tif (waitpid(churn_pids[i], \u0026status, 0) == churn_pids[i] \u0026\u0026\n+\t\t (!WIFEXITED(status) || WEXITSTATUS(status) != 0))\n+\t\t\tfprintf(stderr,\n+\t\t\t\t\"churner %d exited abnormally (status=0x%x)\\n\",\n+\t\t\t\tchurn_pids[i], status);\n+\t}\n+\n+\tfree(churn_pids);\n+\tchurn_pids = NULL;\n+\tn_churners = 0;\n+\n+\tif (ctl) {\n+\t\tmunmap((void *)ctl, sizeof(*ctl));\n+\t\tctl = NULL;\n+\t}\n+}\n+\n+/* ---- file (traditional) reader ----------------------------------------- */\n+\n+static void parse_stat(char *buf, struct file_snap *o)\n+{\n+\tchar *save, *line;\n+\n+\tfor (line = strtok_r(buf, \"\\n\", \u0026save); line;\n+\t line = strtok_r(NULL, \"\\n\", \u0026save)) {\n+\t\tunsigned long long val;\n+\t\tchar name[64];\n+\n+\t\tif (sscanf(line, \"%63s %llu\", name, \u0026val) != 2)\n+\t\t\tcontinue;\n+\t\to-\u003efull_sum += val;\n+\t\to-\u003efull_fields++;\n+\t\tif (!strcmp(name, \"anon\"))\n+\t\t\to-\u003eanon = val;\n+\t\telse if (!strcmp(name, \"file\"))\n+\t\t\to-\u003efile = val;\n+\t\telse if (!strcmp(name, \"shmem\"))\n+\t\t\to-\u003eshmem = val;\n+\t\telse if (!strcmp(name, \"file_mapped\"))\n+\t\t\to-\u003efile_mapped = val;\n+\t\telse if (!strcmp(name, \"pgfault\"))\n+\t\t\to-\u003epgfault = val;\n+\t}\n+}\n+\n+static int file_read_node(const char *rel, struct file_snap *o)\n+{\n+\tchar buf[8192];\n+\n+\tmemset(o, 0, sizeof(*o));\n+\n+\tif (read_cgroup_file(rel, \"memory.stat\", buf, sizeof(buf)))\n+\t\treturn -1;\n+\tparse_stat(buf, o);\n+\n+\tif (!read_cgroup_file(rel, \"memory.current\", buf, sizeof(buf)))\n+\t\to-\u003ecurrent = strtoull(buf, NULL, 10);\n+\tif (!read_cgroup_file(rel, \"memory.max\", buf, sizeof(buf))) {\n+\t\tif (!strncmp(buf, \"max\", 3))\n+\t\t\to-\u003emax_is_max = true;\n+\t\telse\n+\t\t\to-\u003emax = strtoull(buf, NULL, 10);\n+\t}\n+\treturn 0;\n+}\n+\n+/*\n+ * One timed traditional pass over the whole tree; returns nanoseconds.\n+ * @gap_us idles (untimed) before the pass so the tree re-accumulates a roughly\n+ * fixed amount of churn first; the resulting rstat flush is then paid inside the\n+ * timed region, giving every read approximately the same start state.\n+ */\n+static long long file_pass(int gap_us)\n+{\n+\tstruct file_snap s;\n+\tlong long t0;\n+\tint i;\n+\n+\tif (gap_us)\n+\t\tusleep(gap_us);\n+\tt0 = now_ns();\n+\tfor (i = 0; i \u003c n_nodes; i++) {\n+\t\tfile_read_node(nodes[i].rel, \u0026s);\n+\t\tsink += s.anon + s.full_sum;\n+\t}\n+\treturn now_ns() - t0;\n+}\n+\n+/* ---- BPF reader -------------------------------------------------------- */\n+\n+static int bpf_walk_once(struct bpf_link *link)\n+{\n+\tchar buf[4096];\n+\tssize_t r;\n+\tint fd;\n+\n+\tfd = bpf_iter_create(bpf_link__fd(link));\n+\tif (fd \u003c 0)\n+\t\treturn -1;\n+\twhile ((r = read(fd, buf, sizeof(buf))) \u003e 0)\n+\t\t;\n+\tclose(fd);\n+\treturn r == 0 ? 0 : -1;\n+}\n+\n+static int drain_map(int mfd, struct memcg_stat_snapshot *out, int max)\n+{\n+\t__u64 key = 0, next;\n+\tint n = 0, err;\n+\n+\terr = bpf_map_get_next_key(mfd, NULL, \u0026next);\n+\twhile (err == 0) {\n+\t\tif (n \u003c max \u0026\u0026 !bpf_map_lookup_elem(mfd, \u0026next, \u0026out[n])) {\n+\t\t\tsink += out[n].anon + out[n].full_sum;\n+\t\t\tn++;\n+\t\t}\n+\t\tkey = next;\n+\t\terr = bpf_map_get_next_key(mfd, \u0026key, \u0026next);\n+\t}\n+\treturn n;\n+}\n+\n+/*\n+ * One timed BPF pass: kernel walk (ro) + map drain into userspace. Returns the\n+ * total nanoseconds; *ro_ns gets the walk-only time, *got the entries drained.\n+ * @gap_us idles (untimed) before the walk, exactly as in file_pass(), so the\n+ * per-node rstat flush the walk pays reflects the same accumulated churn.\n+ */\n+static long long bpf_pass(struct bpf_link *link, struct memcg_stat_reader *skel,\n+\t\t\t struct memcg_stat_snapshot *tmp,\n+\t\t\t long long *ro_ns, int *got, int *werr, int gap_us)\n+{\n+\tint mfd = bpf_map__fd(skel-\u003emaps.results);\n+\tlong long t0, t1, t2;\n+\tint err;\n+\n+\tskel-\u003ebss-\u003ecollect_full = 1;\n+\n+\tif (gap_us)\n+\t\tusleep(gap_us);\n+\tt0 = now_ns();\n+\terr = bpf_walk_once(link);\n+\tt1 = now_ns();\n+\t*got = drain_map(mfd, tmp, n_nodes + 8);\n+\tt2 = now_ns();\n+\n+\tif (werr)\n+\t\t*werr = err;\n+\t*ro_ns = t1 - t0;\n+\treturn t2 - t0;\n+}\n+\n+/* ---- structural sanity (no field-by-field check under churn) ------------ */\n+\n+static void check_structural(struct bpf_link *link,\n+\t\t\t struct memcg_stat_reader *skel)\n+{\n+\tint mfd = bpf_map__fd(skel-\u003emaps.results);\n+\t__u64 total_anon = 0;\n+\tint i, missing = 0;\n+\n+\tskel-\u003ebss-\u003ecollect_full = 0;\n+\tif (!ASSERT_OK(bpf_walk_once(link), \"bpf walk\"))\n+\t\treturn;\n+\n+\tfor (i = 0; i \u003c n_nodes; i++) {\n+\t\tstruct memcg_stat_snapshot b;\n+\n+\t\tif (bpf_map_lookup_elem(mfd, \u0026nodes[i].id, \u0026b)) {\n+\t\t\tmissing++;\n+\t\t\tcontinue;\n+\t\t}\n+\t\ttotal_anon += b.anon;\n+\t}\n+\n+\tASSERT_EQ(missing, 0, \"all cgroups present in map\");\n+\t/*\n+\t * The churners pin a resident anon set for the whole window, so with no\n+\t * swap and no ancestor memory.max forcing reclaim (the base selftest\n+\t * config sets neither), the tree always carries anon while churn runs.\n+\t */\n+\tASSERT_GT(total_anon, 0, \"tree carries anon under churn\");\n+}\n+\n+/* ---- one case ---------------------------------------------------------- */\n+\n+struct sample_acc {\n+\tlong long file_ns;\n+\tlong long bpf_ns, bpf_ro_ns;\n+\tint last_got;\n+};\n+\n+struct testcase {\n+\tconst char *name;\n+\tint fanout;\n+\tint depth;\n+\tint churn_fraction;\t/* one churner per Nth leaf; 1 = all */\n+\tsize_t region_bytes;\t/* per-iteration churn region */\n+\tsize_t resident_bytes;\t/* pinned resident set per churner */\n+\tint samples;\n+\tint gap_us;\t\t/* idle before EACH read: the \"staleness\" knob (see cases[]) */\n+};\n+\n+static void run_case(const struct testcase *tc)\n+{\n+\tstruct memcg_stat_snapshot *tmp = NULL;\n+\tstruct memcg_stat_reader *skel = NULL;\n+\tstruct bpf_link *link = NULL;\n+\tstruct sample_acc acc = {};\n+\tdouble f, b, bro;\n+\tint root_fd = -1;\n+\tint churners = 0;\n+\tint bad_walks = 0;\n+\tint s, w;\n+\n+\tif (!ASSERT_OK(build_tree(tc-\u003efanout, tc-\u003edepth, \u0026root_fd), \"build tree\"))\n+\t\tgoto out;\n+\n+\tif (start_churners(tc-\u003eregion_bytes, tc-\u003eresident_bytes,\n+\t\t\t tc-\u003echurn_fraction))\n+\t\tgoto out;\n+\tchurners = n_churners;\n+\n+\tskel = memcg_stat_reader__open();\n+\tif (!ASSERT_OK_PTR(skel, \"skel open\"))\n+\t\tgoto out;\n+\tif (!ASSERT_OK(bpf_map__set_max_entries(skel-\u003emaps.results, n_nodes + 8),\n+\t\t \"set max_entries\"))\n+\t\tgoto out;\n+\tif (!ASSERT_OK(memcg_stat_reader__load(skel), \"skel load\"))\n+\t\tgoto out;\n+\n+\tDECLARE_LIBBPF_OPTS(bpf_iter_attach_opts, opts);\n+\tunion bpf_iter_link_info linfo = {};\n+\n+\tlinfo.cgroup.cgroup_fd = root_fd;\n+\tlinfo.cgroup.order = BPF_CGROUP_ITER_DESCENDANTS_PRE;\n+\topts.link_info = \u0026linfo;\n+\topts.link_info_len = sizeof(linfo);\n+\n+\tlink = bpf_program__attach_iter(skel-\u003eprogs.cgroup_memcg_stat_reader,\n+\t\t\t\t\t\u0026opts);\n+\tif (!ASSERT_OK_PTR(link, \"attach iter\"))\n+\t\tgoto out;\n+\n+\ttmp = calloc(n_nodes + 8, sizeof(*tmp));\n+\tif (!ASSERT_OK_PTR(tmp, \"calloc tmp\"))\n+\t\tgoto out;\n+\n+\t/*\n+\t * Authoritative completeness/correctness gate: run once on the freshly\n+\t * loaded (still empty) map, so missing==0 proves this walk visited every\n+\t * cgroup. The map is not cleared between the later timed walks, so the\n+\t * end-of-loop count is only a weaker, informational cross-check.\n+\t */\n+\tcheck_structural(link, skel);\n+\n+\t/* warm caches/vmstats for both paths symmetrically (same gap regime) */\n+\tfor (w = 0; w \u003c WARMUP_ITERS; w++) {\n+\t\tlong long ro;\n+\t\tint got;\n+\n+\t\tfile_pass(tc-\u003egap_us);\n+\t\tbpf_pass(link, skel, tmp, \u0026ro, \u0026got, NULL, tc-\u003egap_us);\n+\t}\n+\n+\t/*\n+\t * Timed samples. Every read idles tc-\u003egap_us (untimed) first, so the\n+\t * tree re-accumulates a roughly fixed amount of churn and each read\n+\t * starts from approximately the same state, paying its own rstat flush\n+\t * inside the timed region. The file/bpf order is flipped on odd samples\n+\t * so any residual jitter doesn't systematically favour whichever reader\n+\t * runs first.\n+\t */\n+\tfor (s = 0; s \u003c tc-\u003esamples; s++) {\n+\t\tlong long ro;\n+\t\tint got, werr;\n+\n+\t\tif (s \u0026 1) {\n+\t\t\tacc.bpf_ns += bpf_pass(link, skel, tmp, \u0026ro, \u0026got, \u0026werr, tc-\u003egap_us);\n+\t\t\tacc.bpf_ro_ns += ro;\n+\t\t\tacc.last_got = got;\n+\t\t\tbad_walks += !!werr;\n+\t\t\tacc.file_ns += file_pass(tc-\u003egap_us);\n+\t\t} else {\n+\t\t\tacc.file_ns += file_pass(tc-\u003egap_us);\n+\t\t\tacc.bpf_ns += bpf_pass(link, skel, tmp, \u0026ro, \u0026got, \u0026werr, tc-\u003egap_us);\n+\t\t\tacc.bpf_ro_ns += ro;\n+\t\t\tacc.last_got = got;\n+\t\t\tbad_walks += !!werr;\n+\t\t}\n+\t}\n+\n+\tf = (double)acc.file_ns / tc-\u003esamples / 1000.0;\n+\tb = (double)acc.bpf_ns / tc-\u003esamples / 1000.0;\n+\tbro = (double)acc.bpf_ro_ns / tc-\u003esamples / 1000.0;\n+\n+\t/*\n+\t * Informational timing diagnostic (captured like any test output: shown\n+\t * under -v or on failure, not on a normal PASS). The pass/fail verdict\n+\t * comes solely from the structural checks above.\n+\t */\n+\tprintf(\"\\n==== memcg_stat_churn: %s ====\\n\", tc-\u003ename);\n+\tprintf(\"tree: nodes=%d leaves=%d churners=%d fanout=%d depth=%d region=%zuKB resident=%zuKB samples=%d gap=%dms\\n\",\n+\t n_nodes, n_leaves, churners, tc-\u003efanout, tc-\u003edepth,\n+\t tc-\u003eregion_bytes \u003e\u003e 10, tc-\u003eresident_bytes \u003e\u003e 10, tc-\u003esamples,\n+\t tc-\u003egap_us / 1000);\n+\tprintf(\"all times in us (average per full-tree read under churn, full memory.stat field set); ratio = file/bpf; ro = bpf read()-only (no map drain)\\n\");\n+\tprintf(\"each read idles gap=%dms first so every read starts from ~the same accumulated churn; the rstat flush is counted in the read\\n\",\n+\t tc-\u003egap_us / 1000);\n+\tprintf(\"file_avg=%.1f bpf_avg=%.1f bpf_ro=%.1f ratio(file/bpf)=%.2fx\\n\",\n+\t f, b, bro, b \u003e 0 ? f / b : 0.0);\n+\tprintf(\"per-cgroup: file avg=%.0f ns bpf avg=%.0f ns\\n\",\n+\t f * 1000.0 / n_nodes, b * 1000.0 / n_nodes);\n+\tprintf(\"bpf entries produced: %d (expected %d)\\n\", acc.last_got, n_nodes);\n+\tprintf(\"RATIO (under churn): file/bpf = %.2fx\\n\", b \u003e 0 ? f / b : 0.0);\n+\n+\tASSERT_EQ(bad_walks, 0, \"all timed bpf walks completed\");\n+\tASSERT_EQ(acc.last_got, n_nodes, \"bpf visited whole subtree under churn\");\n+\n+out:\n+\tfree(tmp);\n+\tbpf_link__destroy(link);\n+\tmemcg_stat_reader__destroy(skel);\n+\tif (root_fd \u003e= 0)\n+\t\tclose(root_fd);\n+\tstop_churners();\t/* reap churners so leaves become removable */\n+\n+\t/*\n+\t * Remove the subtree in reverse creation order. Nodes are recorded in\n+\t * DFS pre-order (a parent precedes all its descendants), so iterating\n+\t * backwards removes every child before its parent.\n+\t */\n+\tif (nodes) {\n+\t\tint i;\n+\n+\t\tfor (i = n_nodes - 1; i \u003e= 0; i--)\n+\t\t\tremove_cgroup(nodes[i].rel);\n+\t\tfree(nodes);\n+\t\tnodes = NULL;\n+\t}\n+}\n+\n+/*\n+ * gap_us: idle time inserted (untimed) before every read so the tree\n+ * re-accumulates a roughly fixed amount of dirty rstat first; the read then\n+ * pays that flush inside its timed region. This gives all four reads\n+ * (file/bpf x matched/full) approximately the same start state and folds the\n+ * flush cost into the measured time. It is the \"staleness / poll-interval\"\n+ * knob: larger gap -\u003e larger common flush -\u003e the file/bpf ratio compresses.\n+ * Pick it past the point where the flush cost saturates; validate by checking\n+ * that bpf matched \u003c= bpf full is restored and that doubling it barely moves\n+ * the numbers. 50 ms is a reasonable default here.\n+ */\n+#define CHURN_GAP_US\t(50 * 1000)\n+\n+static const struct testcase cases[] = {\n+\t/*\n+\t * Both cases use a large (1111-cgroup) tree, where the whole-tree read is\n+\t * big enough that its cost dominates the rstat flush and scheduler jitter,\n+\t * so the reported ratios are reproducible run to run; on a small (tens of\n+\t * cgroups) tree the sub-millisecond BPF read is swamped by that noise and\n+\t * the ratio bounces. They differ only in churn density -- large_dense\n+\t * churns one leaf in 4, large_sparse one in 8 -- which changes how much of\n+\t * the tree the shared flush has to touch. samples are kept even so the\n+\t * file/bpf order-alternation (s \u0026 1) cancels residual first-mover bias.\n+\t */\n+\t/* name fan dep frac region resident samp gap */\n+\t{ \"large_dense\", 10, 3, 4, 256 \u003c\u003c 10, 128 \u003c\u003c 10, 8, CHURN_GAP_US },\n+\t{ \"large_sparse\", 10, 3, 8, 256 \u003c\u003c 10, 128 \u003c\u003c 10, 8, CHURN_GAP_US },\n+};\n+\n+/*\n+ * The memcg kfuncs the reused BPF program relies on (bpf_get_mem_cgroup et al.)\n+ * are built only with CONFIG_MEMCG (mm/bpf_memcontrol.c). On a kernel without\n+ * it they are absent from vmlinux BTF and the program fails to load, so probe\n+ * for one and skip cleanly rather than reporting a spurious failure.\n+ */\n+static bool memcg_kfuncs_available(void)\n+{\n+\tstruct btf *btf;\n+\tbool ok;\n+\n+\tbtf = btf__load_vmlinux_btf();\n+\tif (!btf)\n+\t\treturn false;\n+\tok = btf__find_by_name_kind(btf, \"bpf_get_mem_cgroup\", BTF_KIND_FUNC) \u003e 0;\n+\tbtf__free(btf);\n+\treturn ok;\n+}\n+\n+void serial_test_memcg_stat_churn(void)\n+{\n+\tint i;\n+\n+\tif (!memcg_kfuncs_available()) {\n+\t\ttest__skip();\n+\t\treturn;\n+\t}\n+\n+\tif (!ASSERT_OK(setup_cgroup_environment(), \"setup cgroup env\"))\n+\t\treturn;\n+\n+\tfor (i = 0; i \u003c ARRAY_SIZE(cases); i++) {\n+\t\tif (!test__start_subtest(cases[i].name))\n+\t\t\tcontinue;\n+\t\trun_case(\u0026cases[i]);\n+\t}\n+\n+\tcleanup_cgroup_environment();\n+}\ndiff --git a/tools/testing/selftests/bpf/prog_tests/memcg_stat_churn_percpu.c b/tools/testing/selftests/bpf/prog_tests/memcg_stat_churn_percpu.c\nnew file mode 100644\nindex 00000000000000..16c3f261c87802\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/prog_tests/memcg_stat_churn_percpu.c\n@@ -0,0 +1,902 @@\n+// SPDX-License-Identifier: GPL-2.0\n+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */\n+\n+/*\n+ * memcg_stat_churn_percpu\n+ * =======================\n+ * A CPU-spread variant of memcg_stat_churn. It measures the same two whole-tree\n+ * memcg-stat readers -- the traditional per-cgroup memory.stat parse and a single\n+ * SEC(\"iter.s/cgroup\") BPF walk -- under continuous allocation churn, but this\n+ * time each churned cgroup is deliberately dirtied on MANY CPUs at once:\n+ *\n+ * - Build a synthetic cgroup subtree (fanout x depth), same as the reader.\n+ * - Reserve one CPU for the reader and pin the reader (this parent) to it, so\n+ * the timed reads run on a churn-free CPU and are not preempted by the load\n+ * (measuring on a contended CPU swamps a short walk with scheduler latency).\n+ * - Fork one \"churner\" process per hot leaf (tc-\u003echurn_leaves; 0 = all leaves,\n+ * spread evenly across the tree). Each churner joins its leaf and then, in a\n+ * loop, migrates its own affinity round-robin across K CPUs\n+ * (tc-\u003ecpus_per_leaf; 0 = all churner CPUs) doing one mmap()+memset()+\n+ * munmap() on each. The charge/uncharge happens on whatever CPU the task is\n+ * currently running on, so cycling the affinity queues this leaf's rstat\n+ * dirty on all K CPUs; those per-cpu entries persist until flushed, so\n+ * between two reads the leaf ends up dirty on all K CPUs.\n+ * - While the churn runs, the parent repeatedly SAMPLES both readers exactly as\n+ * in memcg_stat_churn: settle_flush() then an untimed gap before each read so\n+ * each read starts from exactly gap-worth of churn and pays its own flush\n+ * inside the timed region; the file/BPF order is alternated.\n+ *\n+ * Why this matters: both readers flush rstat through the same\n+ * mem_cgroup_flush_stats() path, and the cost of that flush grows with the\n+ * number of (cgroup, cpu) pairs that have pending updates. Where memcg_stat_churn\n+ * dirties each cgroup on essentially one CPU (a read flushes one per-cpu tree per\n+ * cgroup), this test makes K a first-class knob: a read of a cgroup dirtied on K\n+ * CPUs must visit K per-cpu trees. Sweeping K (see the narrow/wide/widest cases)\n+ * drives the shared flush cost F up and compresses the file/BPF ratio, isolating\n+ * the effect of per-cgroup cross-CPU fan-out. Because the reader runs on a\n+ * reserved, churn-free CPU, the flush cost -- not scheduler jitter -- is what the\n+ * timed reads capture; the ratio is robust because the flush hits both readers\n+ * equally.\n+ *\n+ * The BPF program, its hash map and the snapshot struct are REUSED verbatim from\n+ * memcg_stat_reader (progs/memcg_stat_reader.c + memcg_stat_reader.h); only the\n+ * userspace load model (CPU-pinned churners) and sampling loop are new here.\n+ *\n+ * Under churn the stats are a moving target, so this test does NOT do a\n+ * field-by-field BPF-vs-file equality check (that is memcg_stat_reader's job).\n+ * Pass/fail gates only on structural sanity -- the iterator visited every cgroup\n+ * and the tree carries some anon charge. The timing table and final RATIO line\n+ * are informational diagnostics, printed like any other test output (i.e. under\n+ * -v or on failure, never on a normal PASS).\n+ */\n+#ifndef _GNU_SOURCE\n+#define _GNU_SOURCE\t\t/* sched_setaffinity(), CPU_SET() (lib.mk also -D's it) */\n+#endif\n+#include \u003ctest_progs.h\u003e\n+#include \u003cbpf/libbpf.h\u003e\n+#include \u003cbpf/btf.h\u003e\n+#include \u003csched.h\u003e\n+#include \u003cstdlib.h\u003e\n+#include \u003cstring.h\u003e\n+#include \u003ctime.h\u003e\n+#include \u003cunistd.h\u003e\n+#include \u003csys/mman.h\u003e\n+#include \u003csys/wait.h\u003e\n+#include \"cgroup_helpers.h\"\n+#include \"memcg_stat_reader.h\"\n+#include \"memcg_stat_reader.skel.h\"\n+\n+#define SUBTREE_ROOT\t\"/mcg_pcpu\"\n+\n+#define WARMUP_ITERS\t2\n+\n+struct cg_node {\n+\tchar rel[128];\n+\t__u64 id;\n+\tbool is_leaf;\n+};\n+\n+/* Field subset parsed from memory.stat (mirrors memcg_stat_reader). */\n+struct file_snap {\n+\t__u64 anon, file, shmem, file_mapped, pgfault;\n+\t__u64 current;\t\t/* memory.current, bytes */\n+\t__u64 max;\t\t/* memory.max, bytes (valid unless max_is_max) */\n+\t__u64 full_sum;\n+\t__u32 full_fields;\n+\tbool max_is_max;\n+};\n+\n+static volatile __u64 sink;\t/* keep the optimizer from eliding reads */\n+\n+static long long now_ns(void)\n+{\n+\tstruct timespec t;\n+\n+\tclock_gettime(CLOCK_MONOTONIC, \u0026t);\n+\treturn (long long)t.tv_sec * 1000000000LL + t.tv_nsec;\n+}\n+\n+/* ---- allowed CPU set --------------------------------------------------- */\n+\n+static int *cpu_list;\t\t/* ids of the CPUs this process may run on */\n+static int n_cpu;\t\t/* number of such CPUs */\n+static int n_reserved;\t\t/* CPUs held out for the reader (0 or 1) */\n+static int reader_cpu = -1;\t/* CPU the reader is pinned to, or -1 */\n+static cpu_set_t orig_affinity;\t/* parent's affinity, restored on exit */\n+\n+/*\n+ * Collect the CPUs the test is allowed to run on (respecting any cpuset the\n+ * harness put us in). We spawn one pinned churner per usable CPU minus the one\n+ * reserved for the reader. Bounded by CPU_SETSIZE (1024); machines wider than\n+ * that would use only the first CPU_SETSIZE CPUs, which is fine for a diagnostic.\n+ */\n+static int collect_cpus(void)\n+{\n+\tcpu_set_t set;\n+\tint i, want, n = 0;\n+\n+\tCPU_ZERO(\u0026set);\n+\tif (sched_getaffinity(0, sizeof(set), \u0026set))\n+\t\treturn -1;\n+\torig_affinity = set;\t\t/* restored in test teardown */\n+\twant = CPU_COUNT(\u0026set);\n+\tif (want \u003c= 0)\n+\t\treturn -1;\n+\tcpu_list = calloc(want, sizeof(*cpu_list));\n+\tif (!cpu_list)\n+\t\treturn -1;\n+\tfor (i = 0; i \u003c CPU_SETSIZE \u0026\u0026 n \u003c want; i++)\n+\t\tif (CPU_ISSET(i, \u0026set))\n+\t\t\tcpu_list[n++] = i;\n+\tn_cpu = n;\n+\treturn 0;\n+}\n+\n+/* ---- tree construction (same shape as memcg_stat_reader) --------------- */\n+\n+static struct cg_node *nodes;\n+static int n_nodes;\n+static int n_leaves;\n+\n+static int add_node(const char *rel, bool is_leaf, int *keep_fd)\n+{\n+\tint fd;\n+\n+\tfd = create_and_get_cgroup(rel);\n+\tif (fd \u003c 0)\n+\t\treturn -1;\n+\tif (keep_fd)\n+\t\t*keep_fd = fd;\n+\telse\n+\t\tclose(fd);\n+\n+\tstrncpy(nodes[n_nodes].rel, rel, sizeof(nodes[n_nodes].rel) - 1);\n+\tnodes[n_nodes].rel[sizeof(nodes[n_nodes].rel) - 1] = '\\0';\n+\tnodes[n_nodes].id = get_cgroup_id(rel);\n+\tnodes[n_nodes].is_leaf = is_leaf;\n+\tif (is_leaf)\n+\t\tn_leaves++;\n+\tn_nodes++;\n+\treturn 0;\n+}\n+\n+/* Recursively create children of @rel. @rel must already exist and be recorded. */\n+static int build_children(const char *rel, int fanout, int depth)\n+{\n+\tchar child[128];\n+\tint i;\n+\n+\tif (depth == 0)\n+\t\treturn 0;\n+\n+\t/* Enable memory on this interior node so its children get a memcg. */\n+\tif (enable_controllers(rel, \"memory\"))\n+\t\treturn -1;\n+\n+\tfor (i = 0; i \u003c fanout; i++) {\n+\t\tsnprintf(child, sizeof(child), \"%s/c%d\", rel, i);\n+\t\tif (add_node(child, depth == 1, NULL))\n+\t\t\treturn -1;\n+\t\tif (build_children(child, fanout, depth - 1))\n+\t\t\treturn -1;\n+\t}\n+\treturn 0;\n+}\n+\n+static size_t tree_capacity(int fanout, int depth)\n+{\n+\tsize_t total = 1, level = 1;\n+\tint d;\n+\n+\tfor (d = 0; d \u003c depth; d++) {\n+\t\tlevel *= fanout;\n+\t\ttotal += level;\n+\t}\n+\treturn total;\n+}\n+\n+static int build_tree(int fanout, int depth, int *root_fd)\n+{\n+\tn_nodes = 0;\n+\tn_leaves = 0;\n+\tnodes = calloc(tree_capacity(fanout, depth), sizeof(*nodes));\n+\tif (!nodes)\n+\t\treturn -1;\n+\n+\tif (add_node(SUBTREE_ROOT, depth == 0, root_fd))\n+\t\treturn -1;\n+\treturn build_children(SUBTREE_ROOT, fanout, depth);\n+}\n+\n+/* ---- churn load (migrating churners, K CPUs per hot cgroup) ------------- */\n+\n+/*\n+ * Shared control block, mmap'd MAP_SHARED before the forks so the parent can\n+ * signal all churners to stop with a single write.\n+ */\n+struct churn_ctl {\n+\tvolatile int stop;\n+};\n+\n+static struct churn_ctl *ctl;\n+static pid_t *churn_pids;\n+static int n_churners;\n+static int n_hot_leaves;\t\t/* distinct cgroups (leaves) churned */\n+static int n_cpus_per_leaf;\t\t/* K: CPUs each hot cgroup is dirtied on */\n+static int churn_ready[2] = { -1, -1 };\t/* churner -\u003e parent \"ready\" barrier */\n+\n+/* Pin the calling task to a single CPU. */\n+static int pin_cpu(int cpu)\n+{\n+\tcpu_set_t set;\n+\n+\tCPU_ZERO(\u0026set);\n+\tCPU_SET(cpu, \u0026set);\n+\treturn sched_setaffinity(0, sizeof(set), \u0026set);\n+}\n+\n+/*\n+ * One churner process, dedicated to a single hot leaf but spread over K CPUs.\n+ * It joins its leaf, pins a resident anon set so the tree always carries some\n+ * charge, signals readiness, then loops: migrate to the next of its K CPUs and\n+ * do one mmap()+memset()+munmap() there. The charge/uncharge happens on\n+ * whatever CPU the task currently runs on, so cycling the affinity queues this\n+ * leaf's rstat dirty on all K CPUs; those per-cpu entries persist until flushed,\n+ * so between two reads the leaf ends up dirty on all K CPUs and a reader's flush\n+ * of the subtree must visit K per-cpu trees for this one cgroup. Never returns.\n+ *\n+ * @base is this churner's starting index into the churner CPU pool\n+ * (cpu_list[n_reserved ..]); its K CPUs are (base + 0..K-1) mod pool size.\n+ */\n+static void churner_child(const struct cg_node *leaf, int base, int k,\n+\t\t\t size_t region_bytes, size_t resident_bytes)\n+{\n+\tint c_pool = n_cpu - n_reserved;\n+\tvoid *resident;\n+\tint j = 0;\n+\n+\tclose(churn_ready[0]);\n+\n+\t/*\n+\t * Move onto our first CPU before charging. Children inherit the reader's\n+\t * reserved-CPU affinity from the parent, so without this the resident set\n+\t * would be charged on the reader's CPU.\n+\t */\n+\tif (pin_cpu(cpu_list[n_reserved + base % c_pool]))\n+\t\t_exit(4);\n+\n+\t/*\n+\t * cgroup_helpers builds paths from getpid(); in this forked child that\n+\t * differs from the parent that built the tree, so use the _parent\n+\t * (getppid()) variant to resolve the leaf under the parent's work dir.\n+\t */\n+\tif (join_parent_cgroup(leaf-\u003erel))\n+\t\t_exit(1);\n+\n+\tresident = mmap(NULL, resident_bytes, PROT_READ | PROT_WRITE,\n+\t\t\tMAP_ANONYMOUS | MAP_PRIVATE, -1, 0);\n+\tif (resident == MAP_FAILED)\n+\t\t_exit(2);\n+\tmemset(resident, 1, resident_bytes);\t/* fault in, keep mapped */\n+\n+\tif (write(churn_ready[1], \"x\", 1) != 1)\n+\t\t_exit(3);\n+\tclose(churn_ready[1]);\t/* so a sibling's early death yields EOF, not a parent hang */\n+\n+\twhile (!ctl-\u003estop) {\n+\t\tvoid *p;\n+\n+\t\t/* migrate to the next of our K CPUs, then dirty the leaf there */\n+\t\tpin_cpu(cpu_list[n_reserved + (base + j) % c_pool]);\n+\t\tif (++j == k)\n+\t\t\tj = 0;\n+\n+\t\tp = mmap(NULL, region_bytes, PROT_READ | PROT_WRITE,\n+\t\t\t MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);\n+\t\tif (p == MAP_FAILED)\n+\t\t\tcontinue;\n+\t\tmemset(p, 1, region_bytes);\t/* fault in -\u003e anon charge */\n+\t\tmunmap(p, region_bytes);\t/* free -\u003e uncharge (keeps rstat dirty) */\n+\t}\n+\tmunmap(resident, resident_bytes);\n+\t_exit(0);\n+}\n+\n+/*\n+ * Fork one migrating churner per hot leaf (H = @churn_leaves, 0 = all leaves),\n+ * each spread over K CPUs (@cpus_per_leaf, 0 or \u003e pool =\u003e all churner CPUs).\n+ * Hot leaves are chosen evenly across the tree. Returns 0 once every churner\n+ * has joined its leaf and pinned its resident set (so measurement starts under\n+ * steady-state load). On failure the caller's cleanup path calls\n+ * stop_churners() to reap whatever was started.\n+ */\n+static int start_churners(size_t region_bytes, size_t resident_bytes,\n+\t\t\t int churn_leaves, int cpus_per_leaf)\n+{\n+\tint *leaf_idx = NULL, *pool = NULL;\n+\tint n_leaf_idx = 0, pool_n = 0;\n+\tint c_pool = n_cpu - n_reserved;\t/* CPUs available to churners */\n+\tint k_eff, i, h, ret = -1;\n+\n+\t/* K: CPUs each hot cgroup is dirtied on (0 or too big =\u003e all churner CPUs) */\n+\tk_eff = (cpus_per_leaf \u003e 0 \u0026\u0026 cpus_per_leaf \u003c c_pool) ? cpus_per_leaf\n+\t\t\t\t\t\t\t : c_pool;\n+\tn_cpus_per_leaf = k_eff;\n+\n+\t/* gather all leaf node indices in creation order */\n+\tleaf_idx = calloc(n_leaves, sizeof(*leaf_idx));\n+\tif (!ASSERT_OK_PTR(leaf_idx, \"calloc leaf_idx\"))\n+\t\treturn -1;\n+\tfor (i = 0; i \u003c n_nodes; i++)\n+\t\tif (nodes[i].is_leaf)\n+\t\t\tleaf_idx[n_leaf_idx++] = i;\n+\n+\t/*\n+\t * H hot cgroups (one migrating churner each), spread evenly across all\n+\t * leaves; churn_leaves \u003c= 0 or \u003e= n_leaves =\u003e every leaf is hot.\n+\t */\n+\tpool_n = (churn_leaves \u003e 0 \u0026\u0026 churn_leaves \u003c n_leaf_idx) ? churn_leaves\n+\t\t\t\t\t\t\t\t: n_leaf_idx;\n+\tpool = calloc(pool_n, sizeof(*pool));\n+\tif (!ASSERT_OK_PTR(pool, \"calloc pool\")) {\n+\t\tfree(leaf_idx);\n+\t\treturn -1;\n+\t}\n+\tfor (i = 0; i \u003c pool_n; i++)\n+\t\tpool[i] = leaf_idx[(int)((long long)i * n_leaf_idx / pool_n)];\n+\tfree(leaf_idx);\n+\tn_hot_leaves = pool_n;\n+\n+\tctl = mmap(NULL, sizeof(*ctl), PROT_READ | PROT_WRITE,\n+\t\t MAP_SHARED | MAP_ANONYMOUS, -1, 0);\n+\tif (!ASSERT_NEQ(ctl, MAP_FAILED, \"mmap churn_ctl\")) {\n+\t\tctl = NULL;\n+\t\tgoto out;\n+\t}\n+\tctl-\u003estop = 0;\n+\n+\tif (!ASSERT_OK(pipe(churn_ready), \"pipe churn_ready\"))\n+\t\tgoto out;\n+\n+\tchurn_pids = calloc(pool_n, sizeof(*churn_pids));\n+\tif (!ASSERT_OK_PTR(churn_pids, \"calloc churn_pids\"))\n+\t\tgoto out;\n+\n+\t/*\n+\t * One migrating churner per hot leaf. Churner h spans the K CPUs\n+\t * (h*K + 0..K-1) mod c_pool of the churner pool (cpu_list[n_reserved ..]),\n+\t * so different churners start on different CPUs and cpu_list[0] (the\n+\t * reader's reserved CPU) is never used by a churner.\n+\t */\n+\tfor (h = 0; h \u003c pool_n; h++) {\n+\t\tconst struct cg_node *leaf = \u0026nodes[pool[h]];\n+\t\tpid_t pid = fork();\n+\n+\t\tif (pid \u003c 0) {\n+\t\t\tASSERT_GE(pid, 0, \"fork churner\");\n+\t\t\tgoto out;\n+\t\t}\n+\t\tif (pid == 0)\n+\t\t\tchurner_child(leaf, h * k_eff, k_eff,\n+\t\t\t\t region_bytes, resident_bytes);\n+\n+\t\tchurn_pids[n_churners++] = pid;\n+\t}\n+\n+\t/* parent: this end is only for the children to signal on */\n+\tclose(churn_ready[1]);\n+\tchurn_ready[1] = -1;\n+\n+\t/* wait until every churner has joined + pinned its resident set */\n+\tfor (h = 0; h \u003c n_churners; h++) {\n+\t\tchar c;\n+\t\tssize_t r = read(churn_ready[0], \u0026c, 1);\n+\n+\t\tif (r == 0)\n+\t\t\tfprintf(stderr,\n+\t\t\t\t\"a churner exited before signaling ready (affinity/join/mmap failure?)\\n\");\n+\t\tif (!ASSERT_EQ(r, 1, \"churner ready\"))\n+\t\t\tgoto out;\n+\t}\n+\tret = 0;\n+out:\n+\tfree(pool);\n+\treturn ret;\n+}\n+\n+static void stop_churners(void)\n+{\n+\tint i, status;\n+\n+\tif (ctl)\n+\t\tctl-\u003estop = 1;\t\t\t/* release all churn loops */\n+\n+\tif (churn_ready[1] \u003e= 0) {\n+\t\tclose(churn_ready[1]);\n+\t\tchurn_ready[1] = -1;\n+\t}\n+\tif (churn_ready[0] \u003e= 0) {\n+\t\tclose(churn_ready[0]);\n+\t\tchurn_ready[0] = -1;\n+\t}\n+\n+\tfor (i = 0; i \u003c n_churners; i++) {\n+\t\tif (!churn_pids || churn_pids[i] \u003c= 0)\n+\t\t\tcontinue;\n+\t\tif (waitpid(churn_pids[i], \u0026status, 0) == churn_pids[i] \u0026\u0026\n+\t\t (!WIFEXITED(status) || WEXITSTATUS(status) != 0))\n+\t\t\tfprintf(stderr,\n+\t\t\t\t\"churner %d exited abnormally (status=0x%x)\\n\",\n+\t\t\t\tchurn_pids[i], status);\n+\t}\n+\n+\tfree(churn_pids);\n+\tchurn_pids = NULL;\n+\tn_churners = 0;\n+\n+\tif (ctl) {\n+\t\tmunmap((void *)ctl, sizeof(*ctl));\n+\t\tctl = NULL;\n+\t}\n+}\n+\n+/* ---- file (traditional) reader ----------------------------------------- */\n+\n+static void parse_stat(char *buf, struct file_snap *o)\n+{\n+\tchar *save, *line;\n+\n+\tfor (line = strtok_r(buf, \"\\n\", \u0026save); line;\n+\t line = strtok_r(NULL, \"\\n\", \u0026save)) {\n+\t\tunsigned long long val;\n+\t\tchar name[64];\n+\n+\t\tif (sscanf(line, \"%63s %llu\", name, \u0026val) != 2)\n+\t\t\tcontinue;\n+\t\to-\u003efull_sum += val;\n+\t\to-\u003efull_fields++;\n+\t\tif (!strcmp(name, \"anon\"))\n+\t\t\to-\u003eanon = val;\n+\t\telse if (!strcmp(name, \"file\"))\n+\t\t\to-\u003efile = val;\n+\t\telse if (!strcmp(name, \"shmem\"))\n+\t\t\to-\u003eshmem = val;\n+\t\telse if (!strcmp(name, \"file_mapped\"))\n+\t\t\to-\u003efile_mapped = val;\n+\t\telse if (!strcmp(name, \"pgfault\"))\n+\t\t\to-\u003epgfault = val;\n+\t}\n+}\n+\n+static int file_read_node(const char *rel, struct file_snap *o)\n+{\n+\tchar buf[8192];\n+\n+\tmemset(o, 0, sizeof(*o));\n+\n+\tif (read_cgroup_file(rel, \"memory.stat\", buf, sizeof(buf)))\n+\t\treturn -1;\n+\tparse_stat(buf, o);\n+\n+\tif (!read_cgroup_file(rel, \"memory.current\", buf, sizeof(buf)))\n+\t\to-\u003ecurrent = strtoull(buf, NULL, 10);\n+\tif (!read_cgroup_file(rel, \"memory.max\", buf, sizeof(buf))) {\n+\t\tif (!strncmp(buf, \"max\", 3))\n+\t\t\to-\u003emax_is_max = true;\n+\t\telse\n+\t\t\to-\u003emax = strtoull(buf, NULL, 10);\n+\t}\n+\treturn 0;\n+}\n+\n+/*\n+ * Untimed whole-subtree flush used to normalise the pre-read state. Reading the\n+ * subtree root's memory.stat flushes the entire subtree's rstat, so the\n+ * usleep(gap) that follows re-accumulates *exactly* gap-worth of churn no matter\n+ * what the previous timed read was. Without this reset the accumulation window\n+ * would be (previous_read_duration + gap), and since a file pass is ~20x longer\n+ * than a BPF walk that made the flush a BPF read pays depend on ordering -- a\n+ * ~15% window asymmetry, enough to invert bpf_matched vs bpf_full on\n+ * flush-dominated cases (e.g. \"hot\": few cgroups churned from many CPUs).\n+ * nodes[0] is SUBTREE_ROOT (added first in build_tree), whose memcg covers the\n+ * whole tree, so one read here flushes every node the timed reads care about.\n+ */\n+static void settle_flush(void)\n+{\n+\tchar buf[8192];\n+\n+\tif (nodes \u0026\u0026 n_nodes \u003e 0)\n+\t\tread_cgroup_file(nodes[0].rel, \"memory.stat\", buf, sizeof(buf));\n+}\n+\n+/*\n+ * One timed traditional pass over the whole tree; returns nanoseconds. A\n+ * settle_flush() then an untimed @gap_us idle precede the pass so the tree\n+ * re-accumulates exactly gap-worth of churn first; the resulting rstat flush is\n+ * then paid inside the timed region, giving every read the same start state.\n+ */\n+static long long file_pass(int gap_us)\n+{\n+\tstruct file_snap s;\n+\tlong long t0;\n+\tint i;\n+\n+\tsettle_flush();\n+\tif (gap_us)\n+\t\tusleep(gap_us);\n+\tt0 = now_ns();\n+\tfor (i = 0; i \u003c n_nodes; i++) {\n+\t\tfile_read_node(nodes[i].rel, \u0026s);\n+\t\tsink += s.anon + s.full_sum;\n+\t}\n+\treturn now_ns() - t0;\n+}\n+\n+/* ---- BPF reader -------------------------------------------------------- */\n+\n+static int bpf_walk_once(struct bpf_link *link)\n+{\n+\tchar buf[4096];\n+\tssize_t r;\n+\tint fd;\n+\n+\tfd = bpf_iter_create(bpf_link__fd(link));\n+\tif (fd \u003c 0)\n+\t\treturn -1;\n+\twhile ((r = read(fd, buf, sizeof(buf))) \u003e 0)\n+\t\t;\n+\tclose(fd);\n+\treturn r == 0 ? 0 : -1;\n+}\n+\n+static int drain_map(int mfd, struct memcg_stat_snapshot *out, int max)\n+{\n+\t__u64 key = 0, next;\n+\tint n = 0, err;\n+\n+\terr = bpf_map_get_next_key(mfd, NULL, \u0026next);\n+\twhile (err == 0) {\n+\t\tif (n \u003c max \u0026\u0026 !bpf_map_lookup_elem(mfd, \u0026next, \u0026out[n])) {\n+\t\t\tsink += out[n].anon + out[n].full_sum;\n+\t\t\tn++;\n+\t\t}\n+\t\tkey = next;\n+\t\terr = bpf_map_get_next_key(mfd, \u0026key, \u0026next);\n+\t}\n+\treturn n;\n+}\n+\n+/*\n+ * One timed BPF pass: kernel walk (ro) + map drain into userspace. Returns the\n+ * total nanoseconds; *ro_ns gets the walk-only time, *got the entries drained.\n+ * A settle_flush() then an untimed @gap_us idle precede the walk, exactly as in\n+ * file_pass(), so the rstat flush the walk pays reflects the same gap-worth of\n+ * accumulated churn regardless of read ordering.\n+ */\n+static long long bpf_pass(struct bpf_link *link, struct memcg_stat_reader *skel,\n+\t\t\t struct memcg_stat_snapshot *tmp,\n+\t\t\t long long *ro_ns, int *got, int *werr, int gap_us)\n+{\n+\tint mfd = bpf_map__fd(skel-\u003emaps.results);\n+\tlong long t0, t1, t2;\n+\tint err;\n+\n+\tskel-\u003ebss-\u003ecollect_full = 1;\n+\n+\tsettle_flush();\n+\tif (gap_us)\n+\t\tusleep(gap_us);\n+\tt0 = now_ns();\n+\terr = bpf_walk_once(link);\n+\tt1 = now_ns();\n+\t*got = drain_map(mfd, tmp, n_nodes + 8);\n+\tt2 = now_ns();\n+\n+\tif (werr)\n+\t\t*werr = err;\n+\t*ro_ns = t1 - t0;\n+\treturn t2 - t0;\n+}\n+\n+/* ---- structural sanity (no field-by-field check under churn) ------------ */\n+\n+static void check_structural(struct bpf_link *link,\n+\t\t\t struct memcg_stat_reader *skel)\n+{\n+\tint mfd = bpf_map__fd(skel-\u003emaps.results);\n+\t__u64 total_anon = 0;\n+\tint i, missing = 0;\n+\n+\tskel-\u003ebss-\u003ecollect_full = 0;\n+\tif (!ASSERT_OK(bpf_walk_once(link), \"bpf walk\"))\n+\t\treturn;\n+\n+\tfor (i = 0; i \u003c n_nodes; i++) {\n+\t\tstruct memcg_stat_snapshot b;\n+\n+\t\tif (bpf_map_lookup_elem(mfd, \u0026nodes[i].id, \u0026b)) {\n+\t\t\tmissing++;\n+\t\t\tcontinue;\n+\t\t}\n+\t\ttotal_anon += b.anon;\n+\t}\n+\n+\tASSERT_EQ(missing, 0, \"all cgroups present in map\");\n+\t/*\n+\t * The churners pin a resident anon set for the whole window, so with no\n+\t * swap and no ancestor memory.max forcing reclaim (the base selftest\n+\t * config sets neither), the tree always carries anon while churn runs.\n+\t */\n+\tASSERT_GT(total_anon, 0, \"tree carries anon under churn\");\n+}\n+\n+/* ---- one case ---------------------------------------------------------- */\n+\n+struct sample_acc {\n+\tlong long file_ns;\n+\tlong long bpf_ns, bpf_ro_ns;\n+\tint last_got;\n+};\n+\n+struct testcase {\n+\tconst char *name;\n+\tint fanout;\n+\tint depth;\n+\tint churn_leaves;\t/* H: # hot cgroups (one migrating churner each); 0 = all leaves */\n+\tint cpus_per_leaf;\t/* K: CPUs each hot cgroup is dirtied on; 0 = all churner CPUs */\n+\tsize_t region_bytes;\t/* per-iteration churn region */\n+\tsize_t resident_bytes;\t/* pinned resident set per churner */\n+\tint samples;\n+\tint gap_us;\t\t/* idle before EACH read: the \"staleness\" knob (see cases[]) */\n+};\n+\n+static void run_case(const struct testcase *tc)\n+{\n+\tstruct memcg_stat_snapshot *tmp = NULL;\n+\tstruct memcg_stat_reader *skel = NULL;\n+\tstruct bpf_link *link = NULL;\n+\tstruct sample_acc acc = {};\n+\tdouble f, b, bro;\n+\tint root_fd = -1;\n+\tint churners = 0;\n+\tint bad_walks = 0;\n+\tint s, w;\n+\n+\tif (!ASSERT_OK(build_tree(tc-\u003efanout, tc-\u003edepth, \u0026root_fd), \"build tree\"))\n+\t\tgoto out;\n+\n+\tif (start_churners(tc-\u003eregion_bytes, tc-\u003eresident_bytes,\n+\t\t\t tc-\u003echurn_leaves, tc-\u003ecpus_per_leaf))\n+\t\tgoto out;\n+\tchurners = n_churners;\n+\n+\tskel = memcg_stat_reader__open();\n+\tif (!ASSERT_OK_PTR(skel, \"skel open\"))\n+\t\tgoto out;\n+\tif (!ASSERT_OK(bpf_map__set_max_entries(skel-\u003emaps.results, n_nodes + 8),\n+\t\t \"set max_entries\"))\n+\t\tgoto out;\n+\tif (!ASSERT_OK(memcg_stat_reader__load(skel), \"skel load\"))\n+\t\tgoto out;\n+\n+\tDECLARE_LIBBPF_OPTS(bpf_iter_attach_opts, opts);\n+\tunion bpf_iter_link_info linfo = {};\n+\n+\tlinfo.cgroup.cgroup_fd = root_fd;\n+\tlinfo.cgroup.order = BPF_CGROUP_ITER_DESCENDANTS_PRE;\n+\topts.link_info = \u0026linfo;\n+\topts.link_info_len = sizeof(linfo);\n+\n+\tlink = bpf_program__attach_iter(skel-\u003eprogs.cgroup_memcg_stat_reader,\n+\t\t\t\t\t\u0026opts);\n+\tif (!ASSERT_OK_PTR(link, \"attach iter\"))\n+\t\tgoto out;\n+\n+\ttmp = calloc(n_nodes + 8, sizeof(*tmp));\n+\tif (!ASSERT_OK_PTR(tmp, \"calloc tmp\"))\n+\t\tgoto out;\n+\n+\t/*\n+\t * Authoritative completeness/correctness gate: run once on the freshly\n+\t * loaded (still empty) map, so missing==0 proves this walk visited every\n+\t * cgroup. The map is not cleared between the later timed walks, so the\n+\t * end-of-loop count is only a weaker, informational cross-check.\n+\t */\n+\tcheck_structural(link, skel);\n+\n+\t/* warm caches/vmstats for both paths symmetrically (same gap regime) */\n+\tfor (w = 0; w \u003c WARMUP_ITERS; w++) {\n+\t\tlong long ro;\n+\t\tint got;\n+\n+\t\tfile_pass(tc-\u003egap_us);\n+\t\tbpf_pass(link, skel, tmp, \u0026ro, \u0026got, NULL, tc-\u003egap_us);\n+\t}\n+\n+\t/*\n+\t * Timed samples. Every read settle_flush()es then idles tc-\u003egap_us\n+\t * (untimed) first, so the tree re-accumulates exactly gap-worth of churn\n+\t * and each read pays its own rstat flush inside the timed region. The\n+\t * file/bpf order is flipped on odd samples so any residual jitter doesn't\n+\t * systematically favour whichever reader runs first.\n+\t */\n+\tfor (s = 0; s \u003c tc-\u003esamples; s++) {\n+\t\tlong long ro;\n+\t\tint got, werr;\n+\n+\t\tif (s \u0026 1) {\n+\t\t\tacc.bpf_ns += bpf_pass(link, skel, tmp, \u0026ro, \u0026got, \u0026werr, tc-\u003egap_us);\n+\t\t\tacc.bpf_ro_ns += ro;\n+\t\t\tacc.last_got = got;\n+\t\t\tbad_walks += !!werr;\n+\t\t\tacc.file_ns += file_pass(tc-\u003egap_us);\n+\t\t} else {\n+\t\t\tacc.file_ns += file_pass(tc-\u003egap_us);\n+\t\t\tacc.bpf_ns += bpf_pass(link, skel, tmp, \u0026ro, \u0026got, \u0026werr, tc-\u003egap_us);\n+\t\t\tacc.bpf_ro_ns += ro;\n+\t\t\tacc.last_got = got;\n+\t\t\tbad_walks += !!werr;\n+\t\t}\n+\t}\n+\n+\tf = (double)acc.file_ns / tc-\u003esamples / 1000.0;\n+\tb = (double)acc.bpf_ns / tc-\u003esamples / 1000.0;\n+\tbro = (double)acc.bpf_ro_ns / tc-\u003esamples / 1000.0;\n+\n+\t/*\n+\t * Informational timing diagnostic (captured like any test output: shown\n+\t * under -v or on failure, not on a normal PASS). The pass/fail verdict\n+\t * comes solely from the structural checks above.\n+\t */\n+\tprintf(\"\\n==== memcg_stat_churn_percpu: %s ====\\n\", tc-\u003ename);\n+\tprintf(\"tree: nodes=%d leaves=%d hot_leaves=%d cpus_per_leaf=%d dirty_pairs=%d cpus=%d reserved=%d churners=%d fanout=%d depth=%d region=%zuKB resident=%zuKB samples=%d gap=%dms\\n\",\n+\t n_nodes, n_leaves, n_hot_leaves, n_cpus_per_leaf,\n+\t n_hot_leaves * n_cpus_per_leaf, n_cpu, n_reserved, churners,\n+\t tc-\u003efanout, tc-\u003edepth, tc-\u003eregion_bytes \u003e\u003e 10,\n+\t tc-\u003eresident_bytes \u003e\u003e 10, tc-\u003esamples, tc-\u003egap_us / 1000);\n+\tprintf(\"each hot cgroup churned across %d CPUs (migrating churner) so a reader flush visits ~%d per-cpu trees per hot cgroup; reader pinned to reserved CPU %d\\n\",\n+\t n_cpus_per_leaf, n_cpus_per_leaf, reader_cpu);\n+\tprintf(\"all times in us (average per full-tree read under churn, full memory.stat field set); ratio = file/bpf; ro = bpf read()-only (no map drain)\\n\");\n+\tprintf(\"each read flushes then idles gap=%dms so every read starts from exactly gap-worth of churn; the rstat flush is counted in the read\\n\",\n+\t tc-\u003egap_us / 1000);\n+\tprintf(\"file_avg=%.1f bpf_avg=%.1f bpf_ro=%.1f ratio(file/bpf)=%.2fx\\n\",\n+\t f, b, bro, b \u003e 0 ? f / b : 0.0);\n+\tprintf(\"per-cgroup: file avg=%.0f ns bpf avg=%.0f ns\\n\",\n+\t f * 1000.0 / n_nodes, b * 1000.0 / n_nodes);\n+\tprintf(\"bpf entries produced: %d (expected %d)\\n\", acc.last_got, n_nodes);\n+\tprintf(\"RATIO (%d CPUs/cgroup): file/bpf = %.2fx\\n\",\n+\t n_cpus_per_leaf, b \u003e 0 ? f / b : 0.0);\n+\n+\tASSERT_EQ(bad_walks, 0, \"all timed bpf walks completed\");\n+\tASSERT_EQ(acc.last_got, n_nodes, \"bpf visited whole subtree under churn\");\n+\n+out:\n+\tfree(tmp);\n+\tbpf_link__destroy(link);\n+\tmemcg_stat_reader__destroy(skel);\n+\tif (root_fd \u003e= 0)\n+\t\tclose(root_fd);\n+\tstop_churners();\t/* reap churners so leaves become removable */\n+\n+\t/*\n+\t * Remove the subtree in reverse creation order. Nodes are recorded in\n+\t * DFS pre-order (a parent precedes all its descendants), so iterating\n+\t * backwards removes every child before its parent.\n+\t */\n+\tif (nodes) {\n+\t\tint i;\n+\n+\t\tfor (i = n_nodes - 1; i \u003e= 0; i--)\n+\t\t\tremove_cgroup(nodes[i].rel);\n+\t\tfree(nodes);\n+\t\tnodes = NULL;\n+\t}\n+}\n+\n+/*\n+ * gap_us: idle time inserted (untimed) before every read so the tree\n+ * re-accumulates a roughly fixed amount of dirty rstat first; the read then\n+ * pays that flush inside its timed region. This gives all four reads\n+ * (file/bpf x matched/full) approximately the same start state and folds the\n+ * flush cost into the measured time. It is the \"staleness / poll-interval\"\n+ * knob: larger gap -\u003e larger common flush -\u003e the file/bpf ratio compresses.\n+ * See memcg_stat_churn for the full rationale; 50 ms is a reasonable default.\n+ */\n+#define CHURN_GAP_US\t(50 * 1000)\n+\n+static const struct testcase cases[] = {\n+\t/*\n+\t * The narrow/wide/widest trio runs on a large (1111-cgroup) tree and\n+\t * churns a fixed set of 64 hot leaves, sweeping only K, the number of CPUs\n+\t * each hot cgroup is dirtied on, to isolate per-cgroup cross-CPU flush\n+\t * fan-out:\n+\t *\n+\t * narrow - K=1 : each hot cgroup dirty on 1 CPU (64 x 1 dirty pairs).\n+\t * wide - K=8 : each hot cgroup dirty on 8 CPUs (64 x 8 dirty pairs).\n+\t * widest - K=all: each hot cgroup dirty on every churner CPU (64 x cpus).\n+\t *\n+\t * The hot-cgroup count (64) is identical across all three, so only the\n+\t * per-cgroup CPU fan-out changes. As K grows the shared rstat flush F grows\n+\t * (more per-cpu trees to visit), so both readers' cost rises and the\n+\t * file/bpf ratio compresses toward 1; widest is the most conservative\n+\t * regime. The large tree keeps the whole-tree read cost dominant over the\n+\t * flush/scheduler jitter, so the ratios are reproducible (a small tree makes\n+\t * the sub-millisecond BPF read too noisy to compare). samples are kept even\n+\t * so the file/bpf order-alternation (s \u0026 1) cancels first-mover bias; widest\n+\t * gets more samples as its bigger flush has more variance.\n+\t */\n+\t/* name fan dep H K region resident samp gap */\n+\t{ \"narrow\", 10, 3, 64, 1, 256 \u003c\u003c 10, 128 \u003c\u003c 10, 8, CHURN_GAP_US },\n+\t{ \"wide\", 10, 3, 64, 8, 256 \u003c\u003c 10, 128 \u003c\u003c 10, 8, CHURN_GAP_US },\n+\t{ \"widest\", 10, 3, 64, 0, 256 \u003c\u003c 10, 128 \u003c\u003c 10, 10, CHURN_GAP_US },\n+};\n+\n+/*\n+ * The memcg kfuncs the reused BPF program relies on (bpf_get_mem_cgroup et al.)\n+ * are built only with CONFIG_MEMCG (mm/bpf_memcontrol.c). On a kernel without\n+ * it they are absent from vmlinux BTF and the program fails to load, so probe\n+ * for one and skip cleanly rather than reporting a spurious failure.\n+ */\n+static bool memcg_kfuncs_available(void)\n+{\n+\tstruct btf *btf;\n+\tbool ok;\n+\n+\tbtf = btf__load_vmlinux_btf();\n+\tif (!btf)\n+\t\treturn false;\n+\tok = btf__find_by_name_kind(btf, \"bpf_get_mem_cgroup\", BTF_KIND_FUNC) \u003e 0;\n+\tbtf__free(btf);\n+\treturn ok;\n+}\n+\n+/*\n+ * Reserve one CPU for the reader (this parent) and pin the parent to it, so its\n+ * timed reads run on a CPU that carries no churner. Needs at least 2 CPUs; on a\n+ * single-CPU host we skip reserving and the one churner shares the CPU with the\n+ * reader (noisier, but 1-CPU hosts are not the target). Best-effort: if the\n+ * parent cannot be pinned we carry on without a reservation.\n+ */\n+static void reserve_reader_cpu(void)\n+{\n+\tif (n_cpu \u003c 2)\n+\t\treturn;\n+\tif (pin_cpu(cpu_list[0]))\n+\t\treturn;\n+\tn_reserved = 1;\n+\treader_cpu = cpu_list[0];\n+}\n+\n+void serial_test_memcg_stat_churn_percpu(void)\n+{\n+\tint i;\n+\n+\tif (!memcg_kfuncs_available()) {\n+\t\ttest__skip();\n+\t\treturn;\n+\t}\n+\n+\tif (!ASSERT_OK(collect_cpus(), \"collect cpus\"))\n+\t\treturn;\n+\n+\treserve_reader_cpu();\n+\n+\tif (!ASSERT_OK(setup_cgroup_environment(), \"setup cgroup env\"))\n+\t\tgoto restore;\n+\n+\tfor (i = 0; i \u003c ARRAY_SIZE(cases); i++) {\n+\t\tif (!test__start_subtest(cases[i].name))\n+\t\t\tcontinue;\n+\t\trun_case(\u0026cases[i]);\n+\t}\n+\n+\tcleanup_cgroup_environment();\n+restore:\n+\t/* undo reserve_reader_cpu() so later test_progs tests keep full affinity */\n+\tsched_setaffinity(0, sizeof(orig_affinity), \u0026orig_affinity);\n+\tfree(cpu_list);\n+\tcpu_list = NULL;\n+\tn_cpu = 0;\n+\tn_reserved = 0;\n+\treader_cpu = -1;\n+}\ndiff --git a/tools/testing/selftests/bpf/prog_tests/memcg_stat_reader.c b/tools/testing/selftests/bpf/prog_tests/memcg_stat_reader.c\nnew file mode 100644\nindex 00000000000000..b1e631b1520a0e\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/prog_tests/memcg_stat_reader.c\n@@ -0,0 +1,617 @@\n+// SPDX-License-Identifier: GPL-2.0\n+/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */\n+\n+/*\n+ * memcg_stat_reader\n+ * =================\n+ * Read memory-cgroup statistics for a whole synthetic cgroup subtree TWO ways\n+ * and compare them:\n+ *\n+ * (A) traditional: open+read+parse memory.stat / memory.current / memory.max\n+ * for every cgroup, in userspace;\n+ * (B) BPF: a single SEC(\"iter.s/cgroup\") program walked over the subtree in\n+ * DESCENDANTS_PRE order, calling the memcg kfuncs per cgroup and stashing\n+ * the results in a hash map keyed by cgroup id, drained once afterwards.\n+ *\n+ * The test (a) asserts the BPF path agrees with the file path for a checked\n+ * field subset (correctness) and (b) reports the wall-clock cost of each path\n+ * reading the full ~memory.stat field set, across cgroup trees of increasing\n+ * size and load.\n+ *\n+ * The pass/fail result depends only on the correctness checks; the timing table\n+ * is an informational diagnostic captured like any other test output, i.e. shown\n+ * only under -v (or when the test fails), never on a normal PASS.\n+ */\n+#include \u003ctest_progs.h\u003e\n+#include \u003cbpf/libbpf.h\u003e\n+#include \u003cbpf/btf.h\u003e\n+#include \u003cstdlib.h\u003e\n+#include \u003cstring.h\u003e\n+#include \u003ctime.h\u003e\n+#include \u003cunistd.h\u003e\n+#include \u003csys/mman.h\u003e\n+#include \u003csys/wait.h\u003e\n+#include \"cgroup_helpers.h\"\n+#include \"memcg_stat_reader.h\"\n+#include \"memcg_stat_reader.skel.h\"\n+\n+#define SUBTREE_ROOT\t\"/mcg_stat\"\n+\n+#define WARMUP_ITERS\t3\n+\n+struct cg_node {\n+\tchar rel[128];\n+\t__u64 id;\n+\tbool is_leaf;\n+};\n+\n+/* Field subset the BPF prog reads matched against memory.stat by hand. */\n+struct file_snap {\n+\t__u64 anon, file, shmem, file_mapped, pgfault;\n+\t__u64 current;\t\t/* memory.current, bytes */\n+\t__u64 max;\t\t/* memory.max, bytes (valid unless max_is_max) */\n+\t__u64 full_sum;\n+\t__u32 full_fields;\n+\tbool max_is_max;\n+};\n+\n+struct timing {\n+\tdouble avg_us;\t\t/* average per full-tree pass */\n+\tdouble ro_avg_us;\t/* BPF read()-only average (no map drain); 0 for file */\n+\tint nodes_seen;\t\t/* entries produced (BPF) */\n+\t__u32 fields;\t\t/* fields/cgroup touched (informational) */\n+};\n+\n+static volatile __u64 sink;\t/* keep the optimizer from eliding reads */\n+static long page_size;\n+\n+static long long now_ns(void)\n+{\n+\tstruct timespec t;\n+\n+\tclock_gettime(CLOCK_MONOTONIC, \u0026t);\n+\treturn (long long)t.tv_sec * 1000000000LL + t.tv_nsec;\n+}\n+\n+/* ---- tree construction ------------------------------------------------- */\n+\n+static struct cg_node *nodes;\n+static int n_nodes;\n+static int n_leaves;\n+\n+static int add_node(const char *rel, bool is_leaf, int *keep_fd)\n+{\n+\tint fd;\n+\n+\tfd = create_and_get_cgroup(rel);\n+\tif (fd \u003c 0)\n+\t\treturn -1;\n+\tif (keep_fd)\n+\t\t*keep_fd = fd;\n+\telse\n+\t\tclose(fd);\n+\n+\tstrncpy(nodes[n_nodes].rel, rel, sizeof(nodes[n_nodes].rel) - 1);\n+\tnodes[n_nodes].rel[sizeof(nodes[n_nodes].rel) - 1] = '\\0';\n+\tnodes[n_nodes].id = get_cgroup_id(rel);\n+\tnodes[n_nodes].is_leaf = is_leaf;\n+\tif (is_leaf)\n+\t\tn_leaves++;\n+\tn_nodes++;\n+\treturn 0;\n+}\n+\n+/* Recursively create children of @rel. @rel must already exist and be recorded. */\n+static int build_children(const char *rel, int fanout, int depth)\n+{\n+\t/* size 128 should be enough for file path with max depth 3 is the test*/\n+\tchar child[128];\n+\tint i;\n+\n+\tif (depth == 0)\n+\t\treturn 0;\n+\n+\t/* Enable memory on this interior node so its children get memory. */\n+\tif (enable_controllers(rel, \"memory\"))\n+\t\treturn -1;\n+\n+\tfor (i = 0; i \u003c fanout; i++) {\n+\t\tsnprintf(child, sizeof(child), \"%s/c%d\", rel, i);\n+\t\tif (add_node(child, depth == 1, NULL))\n+\t\t\treturn -1;\n+\t\tif (build_children(child, fanout, depth - 1))\n+\t\t\treturn -1;\n+\t}\n+\treturn 0;\n+}\n+\n+static size_t tree_capacity(int fanout, int depth)\n+{\n+\tsize_t total = 1, level = 1;\n+\tint d;\n+\n+\tfor (d = 0; d \u003c depth; d++) {\n+\t\tlevel *= fanout;\n+\t\ttotal += level;\n+\t}\n+\treturn total;\n+}\n+\n+static int build_tree(int fanout, int depth, int *root_fd)\n+{\n+\tn_nodes = 0;\n+\tn_leaves = 0;\n+\tnodes = calloc(tree_capacity(fanout, depth), sizeof(*nodes));\n+\tif (!nodes)\n+\t\treturn -1;\n+\n+\t/* special handle for the leaf (0 depth) */\n+\tif (add_node(SUBTREE_ROOT, depth == 0, root_fd))\n+\t\treturn -1;\n+\treturn build_children(SUBTREE_ROOT, fanout, depth);\n+}\n+\n+/* ---- charging ---------------------------------------------------------- */\n+\n+/*\n+ * A forked child walks the leaves, joining each and faulting in a private anon\n+ * region so the charge lands on that leaf, then keeps every region mapped and\n+ * blocks. Interior nodes accumulate the charge hierarchically. The child is\n+ * left stopped (blocked on the control pipe) so the stats are static while the\n+ * parent measures.\n+ */\n+static pid_t charger_pid = -1;\n+static int charger_ctrl[2] = { -1, -1 };\n+\n+static int start_charger(size_t charge_bytes, int charge_fraction)\n+{\n+\tint ready[2];\n+\tpid_t pid;\n+\tint i, mod;\n+\tchar c;\n+\n+\tif (!ASSERT_OK(pipe(ready), \"pipe ready\"))\n+\t\treturn -1;\n+\tif (!ASSERT_OK(pipe(charger_ctrl), \"pipe ctrl\")) {\n+\t\tclose(ready[0]);\n+\t\tclose(ready[1]);\n+\t\treturn -1;\n+\t}\n+\n+\tpid = fork();\n+\tif (pid \u003c 0) {\n+\t\tASSERT_GE(pid, 0, \"fork charger\");\n+\t\tclose(ready[0]);\n+\t\tclose(ready[1]);\n+\t\tclose(charger_ctrl[0]);\n+\t\tclose(charger_ctrl[1]);\n+\t\tcharger_ctrl[0] = charger_ctrl[1] = -1;\n+\t\treturn -1;\n+\t}\n+\n+\tif (pid == 0) {\n+\t\t/* child (assert only in the parent so it isn't printed twice) */\n+\t\tint leaf_idx = 0;\n+\n+\t\tclose(ready[0]);\n+\t\tclose(charger_ctrl[1]);\n+\n+\t\tmod = charge_fraction \u003e 0 ? charge_fraction : 1;\n+\t\tfor (i = 0; i \u003c n_nodes; i++) {\n+\t\t\tvoid *p;\n+\n+\t\t\tif (!nodes[i].is_leaf)\n+\t\t\t\tcontinue;\n+\t\t\tif ((leaf_idx++ % mod) != 0)\n+\t\t\t\tcontinue;\n+\t\t\t/*\n+\t\t\t * cgroup_helpers builds paths from getpid(); in this\n+\t\t\t * forked child that differs from the parent that built\n+\t\t\t * the tree, so use the _parent (getppid()) variant to\n+\t\t\t * resolve the leaf under the parent's work dir.\n+\t\t\t */\n+\t\t\tif (join_parent_cgroup(nodes[i].rel))\n+\t\t\t\t_exit(1);\n+\t\t\tp = mmap(NULL, charge_bytes, PROT_READ | PROT_WRITE,\n+\t\t\t\t MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);\n+\t\t\tif (p == MAP_FAILED)\n+\t\t\t\t_exit(2);\n+\t\t\tmemset(p, 1, charge_bytes);\n+\t\t\t/* keep p mapped so the charge persists */\n+\t\t}\n+\t\t/* signal ready, then block until the parent closes the pipe */\n+\t\tif (write(ready[1], \"x\", 1) != 1)\n+\t\t\t_exit(3);\n+\t\twhile (read(charger_ctrl[0], \u0026c, 1) \u003e 0)\n+\t\t\t;\n+\t\t_exit(0);\n+\t}\n+\n+\t/* parent */\n+\tcharger_pid = pid;\n+\tclose(ready[1]);\n+\tclose(charger_ctrl[0]);\n+\tcharger_ctrl[0] = -1;\n+\n+\t/* wait until the child has charged every leaf */\n+\tif (!ASSERT_EQ(read(ready[0], \u0026c, 1), 1, \"charger ready\")) {\n+\t\tclose(ready[0]);\n+\t\treturn -1;\n+\t}\n+\tclose(ready[0]);\n+\treturn 0;\n+}\n+\n+static void stop_charger(void)\n+{\n+\tint status;\n+\n+\tif (charger_ctrl[1] \u003e= 0) {\n+\t\tclose(charger_ctrl[1]);\t/* unblock the child -\u003e it exits */\n+\t\tcharger_ctrl[1] = -1;\n+\t}\n+\tif (charger_pid \u003e 0) {\n+\t\tif (waitpid(charger_pid, \u0026status, 0) == charger_pid \u0026\u0026\n+\t\t (!WIFEXITED(status) || WEXITSTATUS(status) != 0))\n+\t\t\tfprintf(stderr,\n+\t\t\t\t\"charger child exited abnormally (status=0x%x)\\n\",\n+\t\t\t\tstatus);\n+\t\tcharger_pid = -1;\n+\t}\n+}\n+\n+/* ---- file (traditional) reader ----------------------------------------- */\n+\n+static void parse_stat(char *buf, struct file_snap *o)\n+{\n+\tchar *save, *line;\n+\n+\tfor (line = strtok_r(buf, \"\\n\", \u0026save); line;\n+\t line = strtok_r(NULL, \"\\n\", \u0026save)) {\n+\t\tunsigned long long val;\n+\t\tchar name[64];\n+\n+\t\tif (sscanf(line, \"%63s %llu\", name, \u0026val) != 2)\n+\t\t\tcontinue;\n+\t\to-\u003efull_sum += val;\n+\t\to-\u003efull_fields++;\n+\t\tif (!strcmp(name, \"anon\"))\n+\t\t\to-\u003eanon = val;\n+\t\telse if (!strcmp(name, \"file\"))\n+\t\t\to-\u003efile = val;\n+\t\telse if (!strcmp(name, \"shmem\"))\n+\t\t\to-\u003eshmem = val;\n+\t\telse if (!strcmp(name, \"file_mapped\"))\n+\t\t\to-\u003efile_mapped = val;\n+\t\telse if (!strcmp(name, \"pgfault\"))\n+\t\t\to-\u003epgfault = val;\n+\t}\n+}\n+\n+static int file_read_node(const char *rel, struct file_snap *o)\n+{\n+\tchar buf[8192];\n+\n+\tmemset(o, 0, sizeof(*o));\n+\n+\tif (read_cgroup_file(rel, \"memory.stat\", buf, sizeof(buf)))\n+\t\treturn -1;\n+\tparse_stat(buf, o);\n+\n+\tif (!read_cgroup_file(rel, \"memory.current\", buf, sizeof(buf)))\n+\t\to-\u003ecurrent = strtoull(buf, NULL, 10);\n+\tif (!read_cgroup_file(rel, \"memory.max\", buf, sizeof(buf))) {\n+\t\tif (!strncmp(buf, \"max\", 3))\n+\t\t\to-\u003emax_is_max = true;\n+\t\telse\n+\t\t\to-\u003emax = strtoull(buf, NULL, 10);\n+\t}\n+\treturn 0;\n+}\n+\n+static void time_file(int iters, struct timing *res)\n+{\n+\tlong long total = 0;\n+\tstruct file_snap s;\n+\tint it, i;\n+\n+\tfor (it = 0; it \u003c WARMUP_ITERS; it++)\n+\t\tfor (i = 0; i \u003c n_nodes; i++)\n+\t\t\tfile_read_node(nodes[i].rel, \u0026s);\n+\n+\tfor (it = 0; it \u003c iters; it++) {\n+\t\tlong long t0 = now_ns();\n+\n+\t\tfor (i = 0; i \u003c n_nodes; i++) {\n+\t\t\tfile_read_node(nodes[i].rel, \u0026s);\n+\t\t\tsink += s.anon + s.full_sum;\n+\t\t}\n+\t\ttotal += now_ns() - t0;\n+\t}\n+\tres-\u003eavg_us = (double)total / iters / 1000.0;\n+\tres-\u003efields = s.full_fields;\n+}\n+\n+/* ---- BPF reader -------------------------------------------------------- */\n+\n+static int bpf_walk_once(struct bpf_link *link)\n+{\n+\tchar buf[4096];\n+\tssize_t r;\n+\tint fd;\n+\n+\tfd = bpf_iter_create(bpf_link__fd(link));\n+\tif (fd \u003c 0)\n+\t\treturn -1;\n+\twhile ((r = read(fd, buf, sizeof(buf))) \u003e 0)\n+\t\t;\n+\tclose(fd);\n+\treturn r == 0 ? 0 : -1;\n+}\n+\n+static int drain_map(int mfd, struct memcg_stat_snapshot *out, int max)\n+{\n+\t__u64 key = 0, next;\n+\tint n = 0, err;\n+\n+\terr = bpf_map_get_next_key(mfd, NULL, \u0026next);\n+\twhile (err == 0) {\n+\t\tif (n \u003c max \u0026\u0026 !bpf_map_lookup_elem(mfd, \u0026next, \u0026out[n])) {\n+\t\t\tsink += out[n].anon + out[n].full_sum;\n+\t\t\tn++;\n+\t\t}\n+\t\tkey = next;\n+\t\terr = bpf_map_get_next_key(mfd, \u0026key, \u0026next);\n+\t}\n+\treturn n;\n+}\n+\n+static void time_bpf(struct bpf_link *link, struct memcg_stat_reader *skel,\n+\t\t int iters, struct timing *res)\n+{\n+\tstruct memcg_stat_snapshot *tmp;\n+\tlong long total = 0, ro_total = 0;\n+\tint mfd = bpf_map__fd(skel-\u003emaps.results);\n+\tint it, got = 0;\n+\n+\ttmp = calloc(n_nodes + 8, sizeof(*tmp));\n+\tif (!ASSERT_OK_PTR(tmp, \"calloc tmp\"))\n+\t\treturn;\n+\n+\tskel-\u003ebss-\u003ecollect_full = 1;\n+\n+\tfor (it = 0; it \u003c WARMUP_ITERS; it++) {\n+\t\tbpf_walk_once(link);\n+\t\tdrain_map(mfd, tmp, n_nodes + 8);\n+\t}\n+\n+\tfor (it = 0; it \u003c iters; it++) {\n+\t\tlong long t0, t1, t2;\n+\n+\t\tt0 = now_ns();\n+\t\tbpf_walk_once(link);\n+\t\tt1 = now_ns();\n+\t\tgot = drain_map(mfd, tmp, n_nodes + 8);\n+\t\tt2 = now_ns();\n+\n+\t\ttotal += t2 - t0;\n+\t\tro_total += t1 - t0;\n+\t}\n+\n+\tres-\u003eavg_us = (double)total / iters / 1000.0;\n+\tres-\u003ero_avg_us = (double)ro_total / iters / 1000.0;\n+\tres-\u003enodes_seen = got;\n+\tres-\u003efields = tmp[0].full_fields;\n+\tfree(tmp);\n+}\n+\n+/* ---- correctness ------------------------------------------------------- */\n+\n+static void check_correctness(struct bpf_link *link,\n+\t\t\t struct memcg_stat_reader *skel)\n+{\n+\tint mfd = bpf_map__fd(skel-\u003emaps.results);\n+\t__u64 total_anon = 0, worst_cur_drift = 0;\n+\t__u64 anon_tol = 4 * page_size;\n+\tint i, anon_mism = 0, missing = 0;\n+\n+\tskel-\u003ebss-\u003ecollect_full = 0;\n+\tif (!ASSERT_OK(bpf_walk_once(link), \"bpf walk\"))\n+\t\treturn;\n+\n+\tfor (i = 0; i \u003c n_nodes; i++) {\n+\t\tstruct memcg_stat_snapshot b;\n+\t\t__u64 cur, drift;\n+\t\tstruct file_snap f;\n+\n+\t\tif (bpf_map_lookup_elem(mfd, \u0026nodes[i].id, \u0026b)) {\n+\t\t\tmissing++;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (file_read_node(nodes[i].rel, \u0026f)) {\n+\t\t\tmissing++;\n+\t\t\tcontinue;\n+\t\t}\n+\t\ttotal_anon += b.anon;\n+\n+\t\t/*\n+\t\t * anon (NR_ANON_MAPPED) is rstat-flushed and, with the charger\n+\t\t * stopped, deterministic: BPF and memory.stat must agree. The\n+\t\t * tolerance is far tighter than a units error (bytes vs pages\n+\t\t * differ by PAGE_SIZE), so a wrong-unit/wrong-field bug trips it.\n+\t\t */\n+\t\tif ((b.anon \u003e f.anon ? b.anon - f.anon : f.anon - b.anon) \u003e anon_tol) {\n+\t\t\tanon_mism++;\n+\t\t\tif (anon_mism \u003c= 5)\n+\t\t\t\tfprintf(stderr,\n+\t\t\t\t\t\"anon mismatch %s: bpf=%llu file=%llu\\n\",\n+\t\t\t\t\tnodes[i].rel, b.anon, f.anon);\n+\t\t}\n+\n+\t\t/*\n+\t\t * memory.current is the LIVE page_counter. Both sides read the\n+\t\t * same counter, but the BPF values are captured in one fast walk\n+\t\t * while the files are read across the whole (much longer) loop,\n+\t\t * so any difference is time skew on a moving counter, not a BPF\n+\t\t * bug -- track it as informational only.\n+\t\t */\n+\t\tcur = b.usage_pages * page_size;\n+\t\tdrift = cur \u003e f.current ? cur - f.current : f.current - cur;\n+\t\tif (drift \u003e worst_cur_drift)\n+\t\t\tworst_cur_drift = drift;\n+\t}\n+\n+\tASSERT_EQ(missing, 0, \"all cgroups present in map\");\n+\tASSERT_EQ(anon_mism, 0, \"bpf vs file anon (rstat-flushed)\");\n+\tASSERT_GT(total_anon, 0, \"tree charged some anon\");\n+\tprintf(\"max memory.current drift bpf-vs-file: %llu bytes (live counter, read across the walk window)\\n\",\n+\t worst_cur_drift);\n+}\n+\n+/* ---- one case ---------------------------------------------------------- */\n+\n+struct testcase {\n+\tconst char *name;\n+\tint fanout;\n+\tint depth;\n+\tsize_t charge_bytes;\n+\tint charge_fraction;\t/* charge every Nth leaf; 1 = all */\n+\tint iters;\n+};\n+\n+static void run_case(const struct testcase *tc)\n+{\n+\tstruct timing f = {}, b = {};\n+\tstruct memcg_stat_reader *skel = NULL;\n+\tstruct bpf_link *link = NULL;\n+\tint root_fd = -1;\n+\tint charged;\n+\n+\tif (!ASSERT_OK(build_tree(tc-\u003efanout, tc-\u003edepth, \u0026root_fd), \"build tree\"))\n+\t\tgoto out;\n+\n+\tif (start_charger(tc-\u003echarge_bytes, tc-\u003echarge_fraction))\n+\t\tgoto out;\n+\n+\tskel = memcg_stat_reader__open();\n+\tif (!ASSERT_OK_PTR(skel, \"skel open\"))\n+\t\tgoto out;\n+\tif (!ASSERT_OK(bpf_map__set_max_entries(skel-\u003emaps.results, n_nodes + 8),\n+\t\t \"set max_entries\"))\n+\t\tgoto out;\n+\tif (!ASSERT_OK(memcg_stat_reader__load(skel), \"skel load\"))\n+\t\tgoto out;\n+\n+\tDECLARE_LIBBPF_OPTS(bpf_iter_attach_opts, opts);\n+\tunion bpf_iter_link_info linfo = {};\n+\n+\tlinfo.cgroup.cgroup_fd = root_fd;\n+\tlinfo.cgroup.order = BPF_CGROUP_ITER_DESCENDANTS_PRE;\n+\topts.link_info = \u0026linfo;\n+\topts.link_info_len = sizeof(linfo);\n+\n+\tlink = bpf_program__attach_iter(skel-\u003eprogs.cgroup_memcg_stat_reader,\n+\t\t\t\t\t\u0026opts);\n+\tif (!ASSERT_OK_PTR(link, \"attach iter\"))\n+\t\tgoto out;\n+\n+\tcheck_correctness(link, skel);\n+\n+\ttime_file(tc-\u003eiters, \u0026f);\n+\ttime_bpf(link, skel, tc-\u003eiters, \u0026b);\n+\n+\tcharged = tc-\u003echarge_fraction \u003e 0 ?\n+\t\t (n_leaves + tc-\u003echarge_fraction - 1) / tc-\u003echarge_fraction :\n+\t\t n_leaves;\n+\n+\t/*\n+\t * Informational timing diagnostic: captured like any test output, so it\n+\t * is shown under -v or on failure but not on a normal PASS. The pass/fail\n+\t * verdict is decided solely by the correctness checks, never by these\n+\t * numbers.\n+\t */\n+\tprintf(\"\\n==== memcg_stat_reader: %s ====\\n\", tc-\u003ename);\n+\tprintf(\"tree: nodes=%d leaves=%d charged_leaves=%d fanout=%d depth=%d charge=%zuKB/leaf iters=%d\\n\",\n+\t n_nodes, n_leaves, charged, tc-\u003efanout, tc-\u003edepth,\n+\t tc-\u003echarge_bytes \u003e\u003e 10, tc-\u003eiters);\n+\tprintf(\"all times in us (average per full-tree pass, full memory.stat field set); ro = bpf read()-only (no map drain)\\n\");\n+\tprintf(\"file_avg=%.1f bpf_avg=%.1f bpf_ro=%.1f speedup(file/bpf)=%.2fx\\n\",\n+\t f.avg_us, b.avg_us, b.ro_avg_us,\n+\t b.avg_us \u003e 0 ? f.avg_us / b.avg_us : 0.0);\n+\tprintf(\"per-cgroup: file avg=%.0f ns bpf avg=%.0f ns\\n\",\n+\t f.avg_us * 1000.0 / n_nodes, b.avg_us * 1000.0 / n_nodes);\n+\tprintf(\"fields/cgroup: bpf=%u | file stat lines=%u\\n\", b.fields, f.fields);\n+\tprintf(\"bpf entries produced: %d (expected %d)\\n\", b.nodes_seen, n_nodes);\n+\n+\tASSERT_EQ(b.nodes_seen, n_nodes, \"bpf visited whole subtree\");\n+\n+out:\n+\tbpf_link__destroy(link);\n+\tmemcg_stat_reader__destroy(skel);\n+\tif (root_fd \u003e= 0)\n+\t\tclose(root_fd);\n+\tstop_charger();\t\t/* reap charger so leaves become empty */\n+\n+\t/*\n+\t * Remove the subtree in reverse creation order. Nodes are recorded in\n+\t * DFS pre-order (a parent precedes all its descendants), so iterating\n+\t * backwards removes every child before its parent.\n+\t */\n+\tif (nodes) {\n+\t\tint i;\n+\n+\t\tfor (i = n_nodes - 1; i \u003e= 0; i--)\n+\t\t\tremove_cgroup(nodes[i].rel);\n+\t\tfree(nodes);\n+\t\tnodes = NULL;\n+\t}\n+}\n+\n+static const struct testcase cases[] = {\n+\t{ \"small\", 4, 2, 256 \u003c\u003c 10, 1, 200 },\n+\t{ \"medium\", 10, 2, 256 \u003c\u003c 10, 1, 50 },\n+\t{ \"large\", 10, 3, 256 \u003c\u003c 10, 1, 10 },\n+\t{ \"large_sparse\", 10, 3, 256 \u003c\u003c 10, 8, 10 },\n+};\n+\n+/*\n+ * The memcg kfuncs the BPF program relies on (bpf_get_mem_cgroup et al.) are\n+ * built only with CONFIG_MEMCG (mm/bpf_memcontrol.c). On a kernel without it\n+ * they are absent from vmlinux BTF and the program fails to load, so probe for\n+ * one of them and skip cleanly rather than reporting a spurious failure.\n+ */\n+static bool memcg_kfuncs_available(void)\n+{\n+\tstruct btf *btf;\n+\tbool ok;\n+\n+\tbtf = btf__load_vmlinux_btf();\n+\tif (!btf)\n+\t\treturn false;\n+\tok = btf__find_by_name_kind(btf, \"bpf_get_mem_cgroup\", BTF_KIND_FUNC) \u003e 0;\n+\tbtf__free(btf);\n+\treturn ok;\n+}\n+\n+void test_memcg_stat_reader(void)\n+{\n+\tint i;\n+\n+\tif (!memcg_kfuncs_available()) {\n+\t\ttest__skip();\n+\t\treturn;\n+\t}\n+\n+\tpage_size = sysconf(_SC_PAGESIZE);\n+\n+\tif (!ASSERT_OK(setup_cgroup_environment(), \"setup cgroup env\"))\n+\t\treturn;\n+\n+\tfor (i = 0; i \u003c ARRAY_SIZE(cases); i++) {\n+\t\tif (!test__start_subtest(cases[i].name))\n+\t\t\tcontinue;\n+\t\trun_case(\u0026cases[i]);\n+\t}\n+\n+\tcleanup_cgroup_environment();\n+}\ndiff --git a/tools/testing/selftests/bpf/progs/memcg_stat_reader.c b/tools/testing/selftests/bpf/progs/memcg_stat_reader.c\nnew file mode 100644\nindex 00000000000000..a2c1b1b483646e\n--- /dev/null\n+++ b/tools/testing/selftests/bpf/progs/memcg_stat_reader.c\n@@ -0,0 +1,181 @@\n+// SPDX-License-Identifier: GPL-2.0\n+/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */\n+#include \u003cvmlinux.h\u003e\n+#include \u003cbpf/bpf_helpers.h\u003e\n+#include \u003cbpf/bpf_core_read.h\u003e\n+#include \"memcg_stat_reader.h\"\n+\n+char _license[] SEC(\"license\") = \"GPL\";\n+\n+/*\n+ * Flipped by userspace between timed runs (a plain .bss global, writable at\n+ * runtime through the skeleton mmap):\n+ * 0 - collect only the matched subset (a handful of kfunc calls)\n+ * 1 - additionally fold in the full memory.stat field set (many kfunc calls)\n+ */\n+int collect_full;\n+\n+/*\n+ * Per-cgroup results, keyed by cgroup id. The BPF-side id (cgrp-\u003ekn-\u003eid)\n+ * equals the userspace get_cgroup_id() value, so the test can correlate map\n+ * entries back to the cgroups it created. max_entries is resized by userspace\n+ * (bpf_map__set_max_entries) to the size of the subtree before load.\n+ */\n+struct {\n+\t__uint(type, BPF_MAP_TYPE_HASH);\n+\t__uint(max_entries, 1);\n+\t__type(key, __u64);\n+\t__type(value, struct memcg_stat_snapshot);\n+} results SEC(\".maps\");\n+\n+/*\n+ * Accumulate one page-state / vm-event read. Each enumerator is guarded by\n+ * bpf_core_enum_value_exists(): the full field set below spans counters that are\n+ * config- or version-gated (e.g. NR_SECONDARY_PAGETABLE, PGDEMOTE_KHUGEPAGED,\n+ * MEMCG_PERCPU_B), so on a kernel whose BTF lacks one, the bpf_core_enum_value()\n+ * relocation would otherwise poison the instruction and fail the *entire*\n+ * program load. With the _exists guard the missing enumerator relocates to a\n+ * compile-time-false branch that the verifier drops as dead code, so the fold is\n+ * simply skipped and the rest of the program (including the matched/correctness\n+ * path) still loads.\n+ */\n+#define FOLD_PS(ENUM, NAME) do {\t\t\t\t\t\t\\\n+\tif (bpf_core_enum_value_exists(enum ENUM, NAME)) {\t\t\t\\\n+\t\t__u64 __v = bpf_mem_cgroup_page_state(memcg,\t\t\t\\\n+\t\t\t\tbpf_core_enum_value(enum ENUM, NAME));\t\t\\\n+\t\tif (__v != (__u64)-1) {\t\t\t\t\t\t\\\n+\t\t\tsum += __v;\t\t\t\t\t\t\\\n+\t\t\tnr++;\t\t\t\t\t\t\t\\\n+\t\t}\t\t\t\t\t\t\t\t\\\n+\t}\t\t\t\t\t\t\t\t\t\\\n+} while (0)\n+\n+#define FOLD_EV(NAME) do {\t\t\t\t\t\t\t\\\n+\tif (bpf_core_enum_value_exists(enum vm_event_item, NAME)) {\t\t\\\n+\t\t__u64 __v = bpf_mem_cgroup_vm_events(memcg,\t\t\t\\\n+\t\t\t\tbpf_core_enum_value(enum vm_event_item, NAME));\t\\\n+\t\tif (__v != (__u64)-1) {\t\t\t\t\t\t\\\n+\t\t\tsum += __v;\t\t\t\t\t\t\\\n+\t\t\tnr++;\t\t\t\t\t\t\\\n+\t\t}\t\t\t\t\t\t\t\\\n+\t}\t\t\t\t\t\t\t\t\\\n+} while (0)\n+\n+/*\n+ * Read a broad memory.stat field set so the timed \"full\" run pays the realistic\n+ * per-field kfunc cost. Enumerators absent from the running kernel's BTF are\n+ * skipped (see the _exists guard in FOLD_PS/FOLD_EV), so this stays loadable\n+ * across kernel configs/versions. __always_inline so the acquired memcg\n+ * reference stays in the main frame (no cross-subprog reference tracking); the\n+ * runtime collect_full branch keeps it off the matched path.\n+ */\n+static __always_inline void collect_full_stats(struct mem_cgroup *memcg,\n+\t\t\t\t\t struct memcg_stat_snapshot *snap)\n+{\n+\t__u64 sum = 0;\n+\t__u32 nr = 0;\n+\n+\t/* node_stat_item: size + event counters that memory.stat prints */\n+\tFOLD_PS(node_stat_item, NR_ANON_MAPPED);\n+\tFOLD_PS(node_stat_item, NR_FILE_PAGES);\n+\tFOLD_PS(node_stat_item, NR_FILE_MAPPED);\n+\tFOLD_PS(node_stat_item, NR_FILE_DIRTY);\n+\tFOLD_PS(node_stat_item, NR_WRITEBACK);\n+\tFOLD_PS(node_stat_item, NR_SHMEM);\n+\tFOLD_PS(node_stat_item, NR_INACTIVE_ANON);\n+\tFOLD_PS(node_stat_item, NR_ACTIVE_ANON);\n+\tFOLD_PS(node_stat_item, NR_INACTIVE_FILE);\n+\tFOLD_PS(node_stat_item, NR_ACTIVE_FILE);\n+\tFOLD_PS(node_stat_item, NR_UNEVICTABLE);\n+\tFOLD_PS(node_stat_item, NR_SLAB_RECLAIMABLE_B);\n+\tFOLD_PS(node_stat_item, NR_SLAB_UNRECLAIMABLE_B);\n+\tFOLD_PS(node_stat_item, NR_KERNEL_STACK_KB);\n+\tFOLD_PS(node_stat_item, NR_PAGETABLE);\n+\tFOLD_PS(node_stat_item, NR_SECONDARY_PAGETABLE);\n+\tFOLD_PS(node_stat_item, NR_VMALLOC);\n+\tFOLD_PS(node_stat_item, WORKINGSET_REFAULT_ANON);\n+\tFOLD_PS(node_stat_item, WORKINGSET_REFAULT_FILE);\n+\tFOLD_PS(node_stat_item, WORKINGSET_ACTIVATE_ANON);\n+\tFOLD_PS(node_stat_item, WORKINGSET_ACTIVATE_FILE);\n+\tFOLD_PS(node_stat_item, WORKINGSET_RESTORE_ANON);\n+\tFOLD_PS(node_stat_item, WORKINGSET_RESTORE_FILE);\n+\tFOLD_PS(node_stat_item, WORKINGSET_NODERECLAIM);\n+\tFOLD_PS(node_stat_item, PGDEMOTE_KSWAPD);\n+\tFOLD_PS(node_stat_item, PGDEMOTE_DIRECT);\n+\tFOLD_PS(node_stat_item, PGDEMOTE_KHUGEPAGED);\n+\tFOLD_PS(node_stat_item, PGSTEAL_KSWAPD);\n+\tFOLD_PS(node_stat_item, PGSTEAL_DIRECT);\n+\tFOLD_PS(node_stat_item, PGSTEAL_KHUGEPAGED);\n+\tFOLD_PS(node_stat_item, PGSCAN_KSWAPD);\n+\tFOLD_PS(node_stat_item, PGSCAN_DIRECT);\n+\tFOLD_PS(node_stat_item, PGSCAN_KHUGEPAGED);\n+\tFOLD_PS(node_stat_item, PGREFILL);\n+\n+\t/* memcg_stat_item: numbered past NR_VM_NODE_STAT_ITEMS */\n+\tFOLD_PS(memcg_stat_item, MEMCG_KMEM);\n+\tFOLD_PS(memcg_stat_item, MEMCG_SOCK);\n+\tFOLD_PS(memcg_stat_item, MEMCG_PERCPU_B);\n+\n+\t/* vm_event_item: the raw-count tail of memory.stat */\n+\tFOLD_EV(PGFAULT);\n+\tFOLD_EV(PGMAJFAULT);\n+\tFOLD_EV(PGACTIVATE);\n+\tFOLD_EV(PGDEACTIVATE);\n+\tFOLD_EV(PGLAZYFREE);\n+\tFOLD_EV(PGLAZYFREED);\n+\n+\tsnap-\u003efull_sum = sum;\n+\tsnap-\u003efull_fields = nr;\n+}\n+\n+SEC(\"iter.s/cgroup\")\n+int cgroup_memcg_stat_reader(struct bpf_iter__cgroup *ctx)\n+{\n+\tstruct cgroup *cgrp = ctx-\u003ecgroup;\n+\tstruct memcg_stat_snapshot snap = {};\n+\tstruct cgroup_subsys_state *css;\n+\tstruct mem_cgroup *memcg;\n+\t__u64 cg_id;\n+\n+\t/*\n+\t * DESCENDANTS_PRE ends with a terminal element where cgroup == NULL.\n+\t * Return 0 (not 1) so the walk runs to completion.\n+\t */\n+\tif (!cgrp)\n+\t\treturn 0;\n+\n+\tcss = \u0026cgrp-\u003eself;\n+\tmemcg = bpf_get_mem_cgroup(css);\n+\tif (!memcg)\n+\t\treturn 0;\n+\n+\t/* Bring this memcg's rstat up to date before reading it. */\n+\tbpf_mem_cgroup_flush_stats(memcg);\n+\n+\tcg_id = BPF_CORE_READ(cgrp, kn, id);\n+\tsnap.cgroup_id = cg_id;\n+\n+\t/* Matched subset: always collected so correctness holds in both modes. */\n+\tsnap.anon = bpf_mem_cgroup_page_state(memcg,\n+\t\t\tbpf_core_enum_value(enum node_stat_item, NR_ANON_MAPPED));\n+\tsnap.file = bpf_mem_cgroup_page_state(memcg,\n+\t\t\tbpf_core_enum_value(enum node_stat_item, NR_FILE_PAGES));\n+\tsnap.shmem = bpf_mem_cgroup_page_state(memcg,\n+\t\t\tbpf_core_enum_value(enum node_stat_item, NR_SHMEM));\n+\tsnap.file_mapped = bpf_mem_cgroup_page_state(memcg,\n+\t\t\tbpf_core_enum_value(enum node_stat_item, NR_FILE_MAPPED));\n+\tsnap.pgfault = bpf_mem_cgroup_vm_events(memcg,\n+\t\t\tbpf_core_enum_value(enum vm_event_item, PGFAULT));\n+\n+\t/* page_counter fields need no kfunc; read them off the trusted ptr. */\n+\tsnap.usage_pages = BPF_CORE_READ(memcg, memory.usage.counter);\n+\tsnap.max_pages = BPF_CORE_READ(memcg, memory.max);\n+\n+\tif (collect_full)\n+\t\tcollect_full_stats(memcg, \u0026snap);\n+\n+\tbpf_map_update_elem(\u0026results, \u0026cg_id, \u0026snap, BPF_ANY);\n+\n+\tbpf_put_mem_cgroup(memcg);\n+\treturn 0;\n+}\n"
} |
| 2/1 | 2026/07/04 05:24 | agent | patch-evaluator |
0mModel:gemini-3.1-pro-preview Args: null Results: {
"EnableConfigs": null,
"FocusSymbols": [],
"Reasoning": "The patch only adds and modifies BPF selftests in tools/testing/selftests/bpf/. It does not make any functional changes to the core kernel, drivers, or architectures.",
"WorthFuzzing": false
}Instruction: You are an expert Linux kernel maintainer. Your job is to review a provided patch series and determine if it makes functional changes to the kernel that should be fuzzed. IMPORTANT: The changes have ALREADY been applied and committed as the HEAD commit in your workspace. Do NOT rely on your internal knowledge of the kernel. You must actively use your code access tools to examine the actual source code and confirm any assumptions. Return WorthFuzzing=false if the patch only contains: - Modifications to Documentation/, Kconfig files, or code comments. - Purely decorative changes, such as logging (e.g., pr_err, printk) or tracepoints. - Changes to numeric constants or macros that do not functionally alter execution flow. - Code paths that are impossible to reach in virtualized environments like GCE or QEMU, even when utilizing software-emulated hardware (e.g., usb gadget, mac80211_hwsim). If it modifies reachable core kernel logic, drivers, or architectures, use your code search tools to verify the code can be executed, then return WorthFuzzing=true. When returning WorthFuzzing=true, you MUST ALSO: 1. Extract any specific kernel functions that should be heavily fuzzed into FocusSymbols. Avoid listing generic hot-path functions to prevent skewed test distributions. 2. Identify any specific CONFIG_ options required to properly test this new/modified feature. Go and look into the Kconfig files and check for ifdefs around the code, do not make assumptions. Do not list too generic configs (we already have them enabled). Only list those that specifically cover the modified code. List them in the EnableConfigs output array, and DO NOT add a 'CONFIG_' prefix (e.g., return "NET_IPV4" instead of "CONFIG_NET_IPV4"). Prefer calling several tools at the same time to save round-trips. Use set-results tool to provide results of the analysis. It must be called exactly once before the final reply. Ignore results of this tool. Prompt: For your convenience, here is the diff of the changes:
commit cd2b01880376591d96e530b1c5e11beef185c315
Author: syz-cluster <triage@syzkaller.com>
Date: Sat Jul 4 05:24:42 2026 +0000
syz-cluster: applied patch under review
diff --git a/tools/testing/selftests/bpf/cgroup_helpers.c b/tools/testing/selftests/bpf/cgroup_helpers.c
index 45cd0b479fe357..fe8ec07c610052 100644
--- a/tools/testing/selftests/bpf/cgroup_helpers.c
+++ b/tools/testing/selftests/bpf/cgroup_helpers.c
@@ -188,6 +188,52 @@ int write_cgroup_file_parent(const char *relative_path, const char *file,
return __write_cgroup_file(cgroup_path, file, buf);
}
+static int __read_cgroup_file(const char *cgroup_path, const char *file,
+ char *buf, size_t buf_size)
+{
+ char file_path[PATH_MAX + 1];
+ ssize_t len;
+ int fd;
+
+ snprintf(file_path, sizeof(file_path), "%s/%s", cgroup_path, file);
+ fd = open(file_path, O_RDONLY);
+ if (fd < 0) {
+ log_err("Opening %s", file_path);
+ return 1;
+ }
+
+ len = read(fd, buf, buf_size - 1);
+ close(fd);
+ if (len < 0) {
+ log_err("Reading %s", file_path);
+ return 1;
+ }
+ buf[len] = '\0';
+ return 0;
+}
+
+/**
+ * read_cgroup_file() - Read from a cgroup file
+ * @relative_path: The cgroup path, relative to the workdir
+ * @file: The name of the file in cgroupfs to read from
+ * @buf: Buffer to read into; NUL-terminated on success
+ * @buf_size: Size of @buf; at most @buf_size - 1 bytes are read
+ *
+ * Read from a file in the given cgroup's directory. As with reading any
+ * cgroupfs control/stat file, @buf should be large enough to hold the whole
+ * value in a single read().
+ *
+ * If successful, 0 is returned.
+ */
+int read_cgroup_file(const char *relative_path, const char *file,
+ char *buf, size_t buf_size)
+{
+ char cgroup_path[PATH_MAX - 24];
+
+ format_cgroup_path(cgroup_path, relative_path);
+ return __read_cgroup_file(cgroup_path, file, buf, buf_size);
+}
+
/**
* setup_cgroup_environment() - Setup the cgroup environment
*
diff --git a/tools/testing/selftests/bpf/cgroup_helpers.h b/tools/testing/selftests/bpf/cgroup_helpers.h
index 3857304be87410..1ed76dd3a1da27 100644
--- a/tools/testing/selftests/bpf/cgroup_helpers.h
+++ b/tools/testing/selftests/bpf/cgroup_helpers.h
@@ -15,6 +15,8 @@ int write_cgroup_file(const char *relative_path, const char *file,
const char *buf);
int write_cgroup_file_parent(const char *relative_path, const char *file,
const char *buf);
+int read_cgroup_file(const char *relative_path, const char *file,
+ char *buf, size_t buf_size);
int cgroup_setup_and_join(const char *relative_path);
int get_root_cgroup(void);
int create_and_get_cgroup(const char *relative_path);
diff --git a/tools/testing/selftests/bpf/config b/tools/testing/selftests/bpf/config
index adb25146e88ccf..4e75b4ea8649bf 100644
--- a/tools/testing/selftests/bpf/config
+++ b/tools/testing/selftests/bpf/config
@@ -55,6 +55,7 @@ CONFIG_LIRC=y
CONFIG_LIVEPATCH=y
CONFIG_LWTUNNEL=y
CONFIG_LWTUNNEL_BPF=y
+CONFIG_MEMCG=y
CONFIG_MODULE_SIG=y
CONFIG_MODULE_SRCVERSION_ALL=y
CONFIG_MODULE_UNLOAD=y
diff --git a/tools/testing/selftests/bpf/memcg_stat_reader.h b/tools/testing/selftests/bpf/memcg_stat_reader.h
new file mode 100644
index 00000000000000..72afebe95ccb82
--- /dev/null
+++ b/tools/testing/selftests/bpf/memcg_stat_reader.h
@@ -0,0 +1,35 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */
+#ifndef __MEMCG_STAT_READER_H
+#define __MEMCG_STAT_READER_H
+
+/*
+ * One per-cgroup snapshot, produced by the BPF cgroup iterator and read back
+ * from a BPF hash map keyed by cgroup id. The "matched" subset is always
+ * populated so it can be compared field-by-field against what userspace parses
+ * out of memory.stat / memory.current / memory.max. The "full" fold is only
+ * populated when collect_full is set and exists to (a) force the extra kfunc
+ * reads to happen (so the full-vs-matched timing is honest) and (b) give a
+ * coarse, informational signal of how many fields the full path touched.
+ */
+struct memcg_stat_snapshot {
+ __u64 cgroup_id;
+
+ /* Matched subset. Page-state values are in bytes (already unit-scaled
+ * by the kernel), so they compare directly against memory.stat.
+ */
+ __u64 anon; /* NR_ANON_MAPPED, bytes */
+ __u64 file; /* NR_FILE_PAGES, bytes */
+ __u64 shmem; /* NR_SHMEM, bytes */
+ __u64 file_mapped; /* NR_FILE_MAPPED, bytes */
+ __u64 pgfault; /* PGFAULT, count */
+ __u64 usage_pages; /* page_counter memory.usage, in PAGES */
+ __u64 max_pages; /* page_counter memory.max, in PAGES */
+
+ /* Full-mode fold: sum and count of every field the full path read. */
+ __u64 full_sum;
+ __u32 full_fields;
+ __u32 pad;
+};
+
+#endif /* __MEMCG_STAT_READER_H */
diff --git a/tools/testing/selftests/bpf/prog_tests/memcg_stat_churn.c b/tools/testing/selftests/bpf/prog_tests/memcg_stat_churn.c
new file mode 100644
index 00000000000000..3e386d0b4c033b
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/memcg_stat_churn.c
@@ -0,0 +1,716 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+/*
+ * memcg_stat_churn
+ * ================
+ * A load variant of the memcg_stat_reader benchmark. Where memcg_stat_reader
+ * charges a quiescent tree once and then measures both readers against static
+ * stats, this test keeps the memory-cgroup rstat perpetually DIRTY while it
+ * measures:
+ *
+ * - Build a synthetic cgroup subtree (fanout x depth), same as the reader.
+ * - Fork one "churner" process per selected leaf. Each churner joins its
+ * leaf, pins a small resident anon set (so tree anon stays > 0), then loops
+ * mmap()+memset()+munmap() for the whole measurement window. The constant
+ * charge/uncharge traffic keeps every touched memcg's per-cpu stats dirty,
+ * so each reader pays a realistic flush/read cost instead of a warm no-op.
+ * - While the churn runs, the parent repeatedly SAMPLES both readers:
+ * (A) traditional: open/read/parse memory.stat (+current/+max) for every
+ * cgroup from userspace;
+ * (B) BPF: one SEC("iter.s/cgroup") walk over the subtree calling the
+ * memcg kfuncs into a hash map, drained once per sample.
+ * Before each timed read the parent idles for a fixed gap (untimed) so the
+ * tree re-accumulates a roughly fixed amount of dirty rstat; every read
+ * (file/BPF x matched/full) therefore starts from approximately the same
+ * state and pays its own rstat flush inside the timed region. The
+ * file-vs-BPF order is also alternated across samples so residual jitter
+ * doesn't systematically favour whichever reader runs first.
+ * - Times are averaged over all samples and the file/BPF speedup ratio is
+ * reported. The gap is the "staleness / poll-interval" knob: a larger gap
+ * means a larger flush that both paths pay, so the ratio is more
+ * conservative (see CHURN_GAP_US).
+ *
+ * The BPF program, its hash map and the snapshot struct are REUSED verbatim
+ * from memcg_stat_reader (progs/memcg_stat_reader.c + memcg_stat_reader.h); only
+ * the userspace load model and sampling loop are new here.
+ *
+ * Under churn the stats are a moving target, so this test does NOT do a
+ * field-by-field BPF-vs-file equality check (that is memcg_stat_reader's job).
+ * Pass/fail gates only on structural sanity -- the iterator visited every
+ * cgroup and the tree carries some anon charge. The timing table and final
+ * RATIO line are informational diagnostics, printed like any other test output
+ * (i.e. under -v or on failure, never on a normal PASS).
+ */
+#include <test_progs.h>
+#include <bpf/libbpf.h>
+#include <bpf/btf.h>
+#include <stdlib.h>
+#include <string.h>
+#include <time.h>
+#include <unistd.h>
+#include <sys/mman.h>
+#include <sys/wait.h>
+#include "cgroup_helpers.h"
+#include "memcg_stat_reader.h"
+#include "memcg_stat_reader.skel.h"
+
+#define SUBTREE_ROOT "/mcg_churn"
+
+#define WARMUP_ITERS 2
+
+struct cg_node {
+ char rel[128];
+ __u64 id;
+ bool is_leaf;
+};
+
+/* Field subset parsed from memory.stat (mirrors memcg_stat_reader). */
+struct file_snap {
+ __u64 anon, file, shmem, file_mapped, pgfault;
+ __u64 current; /* memory.current, bytes */
+ __u64 max; /* memory.max, bytes (valid unless max_is_max) */
+ __u64 full_sum;
+ __u32 full_fields;
+ bool max_is_max;
+};
+
+static volatile __u64 sink; /* keep the optimizer from eliding reads */
+
+static long long now_ns(void)
+{
+ struct timespec t;
+
+ clock_gettime(CLOCK_MONOTONIC, &t);
+ return (long long)t.tv_sec * 1000000000LL + t.tv_nsec;
+}
+
+/* ---- tree construction (same shape as memcg_stat_reader) --------------- */
+
+static struct cg_node *nodes;
+static int n_nodes;
+static int n_leaves;
+
+static int add_node(const char *rel, bool is_leaf, int *keep_fd)
+{
+ int fd;
+
+ fd = create_and_get_cgroup(rel);
+ if (fd < 0)
+ return -1;
+ if (keep_fd)
+ *keep_fd = fd;
+ else
+ close(fd);
+
+ strncpy(nodes[n_nodes].rel, rel, sizeof(nodes[n_nodes].rel) - 1);
+ nodes[n_nodes].rel[sizeof(nodes[n_nodes].rel) - 1] = '\0';
+ nodes[n_nodes].id = get_cgroup_id(rel);
+ nodes[n_nodes].is_leaf = is_leaf;
+ if (is_leaf)
+ n_leaves++;
+ n_nodes++;
+ return 0;
+}
+
+/* Recursively create children of @rel. @rel must already exist and be recorded. */
+static int build_children(const char *rel, int fanout, int depth)
+{
+ char child[128];
+ int i;
+
+ if (depth == 0)
+ return 0;
+
+ /* Enable memory on this interior node so its children get a memcg. */
+ if (enable_controllers(rel, "memory"))
+ return -1;
+
+ for (i = 0; i < fanout; i++) {
+ snprintf(child, sizeof(child), "%s/c%d", rel, i);
+ if (add_node(child, depth == 1, NULL))
+ return -1;
+ if (build_children(child, fanout, depth - 1))
+ return -1;
+ }
+ return 0;
+}
+
+static size_t tree_capacity(int fanout, int depth)
+{
+ size_t total = 1, level = 1;
+ int d;
+
+ for (d = 0; d < depth; d++) {
+ level *= fanout;
+ total += level;
+ }
+ return total;
+}
+
+static int build_tree(int fanout, int depth, int *root_fd)
+{
+ n_nodes = 0;
+ n_leaves = 0;
+ nodes = calloc(tree_capacity(fanout, depth), sizeof(*nodes));
+ if (!nodes)
+ return -1;
+
+ if (add_node(SUBTREE_ROOT, depth == 0, root_fd))
+ return -1;
+ return build_children(SUBTREE_ROOT, fanout, depth);
+}
+
+/* ---- churn load -------------------------------------------------------- */
+
+/*
+ * Shared control block, mmap'd MAP_SHARED before the forks so the parent can
+ * signal all churners to stop with a single write.
+ */
+struct churn_ctl {
+ volatile int stop;
+};
+
+static struct churn_ctl *ctl;
+static pid_t *churn_pids;
+static int n_churners;
+static int churn_ready[2] = { -1, -1 }; /* churner -> parent "ready" barrier */
+
+/*
+ * One churner process. Joins its leaf, pins a resident anon set so the tree
+ * always carries some charge, signals readiness, then continuously faults in
+ * and frees a private anon region until told to stop. Never returns.
+ */
+static void churner_child(const struct cg_node *leaf, size_t region_bytes,
+ size_t resident_bytes)
+{
+ void *resident;
+
+ close(churn_ready[0]);
+
+ /*
+ * cgroup_helpers builds paths from getpid(); in this forked child that
+ * differs from the parent that built the tree, so use the _parent
+ * (getppid()) variant to resolve the leaf under the parent's work dir.
+ */
+ if (join_parent_cgroup(leaf->rel))
+ _exit(1);
+
+ resident = mmap(NULL, resident_bytes, PROT_READ | PROT_WRITE,
+ MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
+ if (resident == MAP_FAILED)
+ _exit(2);
+ memset(resident, 1, resident_bytes); /* fault in, keep mapped */
+
+ if (write(churn_ready[1], "x", 1) != 1)
+ _exit(3);
+ close(churn_ready[1]); /* so a sibling's early death yields EOF, not a parent hang */
+
+ while (!ctl->stop) {
+ void *p = mmap(NULL, region_bytes, PROT_READ | PROT_WRITE,
+ MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
+
+ if (p == MAP_FAILED)
+ continue;
+ memset(p, 1, region_bytes); /* fault in -> anon charge */
+ munmap(p, region_bytes); /* free -> uncharge (keeps rstat dirty) */
+ }
+ munmap(resident, resident_bytes);
+ _exit(0);
+}
+
+/*
+ * Fork one churner per @charge_fraction-th leaf. Returns 0 once every churner
+ * has joined its leaf and pinned its resident set (so measurement starts under
+ * steady-state load). On failure the caller's cleanup path calls
+ * stop_churners() to reap whatever was started.
+ */
+static int start_churners(size_t region_bytes, size_t resident_bytes,
+ int charge_fraction)
+{
+ int mod = charge_fraction > 0 ? charge_fraction : 1;
+ int leaf_idx = 0;
+ int i;
+
+ ctl = mmap(NULL, sizeof(*ctl), PROT_READ | PROT_WRITE,
+ MAP_SHARED | MAP_ANONYMOUS, -1, 0);
+ if (!ASSERT_NEQ(ctl, MAP_FAILED, "mmap churn_ctl")) {
+ ctl = NULL;
+ return -1;
+ }
+ ctl->stop = 0;
+
+ if (!ASSERT_OK(pipe(churn_ready), "pipe churn_ready"))
+ return -1;
+
+ churn_pids = calloc(n_leaves, sizeof(*churn_pids));
+ if (!ASSERT_OK_PTR(churn_pids, "calloc churn_pids"))
+ return -1;
+
+ for (i = 0; i < n_nodes; i++) {
+ pid_t pid;
+
+ if (!nodes[i].is_leaf)
+ continue;
+ if ((leaf_idx++ % mod) != 0)
+ continue;
+
+ pid = fork();
+ if (pid < 0) {
+ ASSERT_GE(pid, 0, "fork churner");
+ return -1;
+ }
+ if (pid == 0)
+ churner_child(&nodes[i], region_bytes, resident_bytes);
+
+ churn_pids[n_churners++] = pid;
+ }
+
+ /* parent: this end is only for the children to signal on */
+ close(churn_ready[1]);
+ churn_ready[1] = -1;
+
+ /* wait until every churner has joined + pinned its resident set */
+ for (i = 0; i < n_churners; i++) {
+ char c;
+ ssize_t r = read(churn_ready[0], &c, 1);
+
+ if (r == 0)
+ fprintf(stderr,
+ "a churner exited before signaling ready (join_parent_cgroup/mmap failure?)\n");
+ if (!ASSERT_EQ(r, 1, "churner ready"))
+ return -1;
+ }
+ return 0;
+}
+
+static void stop_churners(void)
+{
+ int i, status;
+
+ if (ctl)
+ ctl->stop = 1; /* release all churn loops */
+
+ if (churn_ready[1] >= 0) {
+ close(churn_ready[1]);
+ churn_ready[1] = -1;
+ }
+ if (churn_ready[0] >= 0) {
+ close(churn_ready[0]);
+ churn_ready[0] = -1;
+ }
+
+ for (i = 0; i < n_churners; i++) {
+ if (!churn_pids || churn_pids[i] <= 0)
+ continue;
+ if (waitpid(churn_pids[i], &status, 0) == churn_pids[i] &&
+ (!WIFEXITED(status) || WEXITSTATUS(status) != 0))
+ fprintf(stderr,
+ "churner %d exited abnormally (status=0x%x)\n",
+ churn_pids[i], status);
+ }
+
+ free(churn_pids);
+ churn_pids = NULL;
+ n_churners = 0;
+
+ if (ctl) {
+ munmap((void *)ctl, sizeof(*ctl));
+ ctl = NULL;
+ }
+}
+
+/* ---- file (traditional) reader ----------------------------------------- */
+
+static void parse_stat(char *buf, struct file_snap *o)
+{
+ char *save, *line;
+
+ for (line = strtok_r(buf, "\n", &save); line;
+ line = strtok_r(NULL, "\n", &save)) {
+ unsigned long long val;
+ char name[64];
+
+ if (sscanf(line, "%63s %llu", name, &val) != 2)
+ continue;
+ o->full_sum += val;
+ o->full_fields++;
+ if (!strcmp(name, "anon"))
+ o->anon = val;
+ else if (!strcmp(name, "file"))
+ o->file = val;
+ else if (!strcmp(name, "shmem"))
+ o->shmem = val;
+ else if (!strcmp(name, "file_mapped"))
+ o->file_mapped = val;
+ else if (!strcmp(name, "pgfault"))
+ o->pgfault = val;
+ }
+}
+
+static int file_read_node(const char *rel, struct file_snap *o)
+{
+ char buf[8192];
+
+ memset(o, 0, sizeof(*o));
+
+ if (read_cgroup_file(rel, "memory.stat", buf, sizeof(buf)))
+ return -1;
+ parse_stat(buf, o);
+
+ if (!read_cgroup_file(rel, "memory.current", buf, sizeof(buf)))
+ o->current = strtoull(buf, NULL, 10);
+ if (!read_cgroup_file(rel, "memory.max", buf, sizeof(buf))) {
+ if (!strncmp(buf, "max", 3))
+ o->max_is_max = true;
+ else
+ o->max = strtoull(buf, NULL, 10);
+ }
+ return 0;
+}
+
+/*
+ * One timed traditional pass over the whole tree; returns nanoseconds.
+ * @gap_us idles (untimed) before the pass so the tree re-accumulates a roughly
+ * fixed amount of churn first; the resulting rstat flush is then paid inside the
+ * timed region, giving every read approximately the same start state.
+ */
+static long long file_pass(int gap_us)
+{
+ struct file_snap s;
+ long long t0;
+ int i;
+
+ if (gap_us)
+ usleep(gap_us);
+ t0 = now_ns();
+ for (i = 0; i < n_nodes; i++) {
+ file_read_node(nodes[i].rel, &s);
+ sink += s.anon + s.full_sum;
+ }
+ return now_ns() - t0;
+}
+
+/* ---- BPF reader -------------------------------------------------------- */
+
+static int bpf_walk_once(struct bpf_link *link)
+{
+ char buf[4096];
+ ssize_t r;
+ int fd;
+
+ fd = bpf_iter_create(bpf_link__fd(link));
+ if (fd < 0)
+ return -1;
+ while ((r = read(fd, buf, sizeof(buf))) > 0)
+ ;
+ close(fd);
+ return r == 0 ? 0 : -1;
+}
+
+static int drain_map(int mfd, struct memcg_stat_snapshot *out, int max)
+{
+ __u64 key = 0, next;
+ int n = 0, err;
+
+ err = bpf_map_get_next_key(mfd, NULL, &next);
+ while (err == 0) {
+ if (n < max && !bpf_map_lookup_elem(mfd, &next, &out[n])) {
+ sink += out[n].anon + out[n].full_sum;
+ n++;
+ }
+ key = next;
+ err = bpf_map_get_next_key(mfd, &key, &next);
+ }
+ return n;
+}
+
+/*
+ * One timed BPF pass: kernel walk (ro) + map drain into userspace. Returns the
+ * total nanoseconds; *ro_ns gets the walk-only time, *got the entries drained.
+ * @gap_us idles (untimed) before the walk, exactly as in file_pass(), so the
+ * per-node rstat flush the walk pays reflects the same accumulated churn.
+ */
+static long long bpf_pass(struct bpf_link *link, struct memcg_stat_reader *skel,
+ struct memcg_stat_snapshot *tmp,
+ long long *ro_ns, int *got, int *werr, int gap_us)
+{
+ int mfd = bpf_map__fd(skel->maps.results);
+ long long t0, t1, t2;
+ int err;
+
+ skel->bss->collect_full = 1;
+
+ if (gap_us)
+ usleep(gap_us);
+ t0 = now_ns();
+ err = bpf_walk_once(link);
+ t1 = now_ns();
+ *got = drain_map(mfd, tmp, n_nodes + 8);
+ t2 = now_ns();
+
+ if (werr)
+ *werr = err;
+ *ro_ns = t1 - t0;
+ return t2 - t0;
+}
+
+/* ---- structural sanity (no field-by-field check under churn) ------------ */
+
+static void check_structural(struct bpf_link *link,
+ struct memcg_stat_reader *skel)
+{
+ int mfd = bpf_map__fd(skel->maps.results);
+ __u64 total_anon = 0;
+ int i, missing = 0;
+
+ skel->bss->collect_full = 0;
+ if (!ASSERT_OK(bpf_walk_once(link), "bpf walk"))
+ return;
+
+ for (i = 0; i < n_nodes; i++) {
+ struct memcg_stat_snapshot b;
+
+ if (bpf_map_lookup_elem(mfd, &nodes[i].id, &b)) {
+ missing++;
+ continue;
+ }
+ total_anon += b.anon;
+ }
+
+ ASSERT_EQ(missing, 0, "all cgroups present in map");
+ /*
+ * The churners pin a resident anon set for the whole window, so with no
+ * swap and no ancestor memory.max forcing reclaim (the base selftest
+ * config sets neither), the tree always carries anon while churn runs.
+ */
+ ASSERT_GT(total_anon, 0, "tree carries anon under churn");
+}
+
+/* ---- one case ---------------------------------------------------------- */
+
+struct sample_acc {
+ long long file_ns;
+ long long bpf_ns, bpf_ro_ns;
+ int last_got;
+};
+
+struct testcase {
+ const char *name;
+ int fanout;
+ int depth;
+ int churn_fraction; /* one churner per Nth leaf; 1 = all */
+ size_t region_bytes; /* per-iteration churn region */
+ size_t resident_bytes; /* pinned resident set per churner */
+ int samples;
+ int gap_us; /* idle before EACH read: the "staleness" knob (see cases[]) */
+};
+
+static void run_case(const struct testcase *tc)
+{
+ struct memcg_stat_snapshot *tmp = NULL;
+ struct memcg_stat_reader *skel = NULL;
+ struct bpf_link *link = NULL;
+ struct sample_acc acc = {};
+ double f, b, bro;
+ int root_fd = -1;
+ int churners = 0;
+ int bad_walks = 0;
+ int s, w;
+
+ if (!ASSERT_OK(build_tree(tc->fanout, tc->depth, &root_fd), "build tree"))
+ goto out;
+
+ if (start_churners(tc->region_bytes, tc->resident_bytes,
+ tc->churn_fraction))
+ goto out;
+ churners = n_churners;
+
+ skel = memcg_stat_reader__open();
+ if (!ASSERT_OK_PTR(skel, "skel open"))
+ goto out;
+ if (!ASSERT_OK(bpf_map__set_max_entries(skel->maps.results, n_nodes + 8),
+ "set max_entries"))
+ goto out;
+ if (!ASSERT_OK(memcg_stat_reader__load(skel), "skel load"))
+ goto out;
+
+ DECLARE_LIBBPF_OPTS(bpf_iter_attach_opts, opts);
+ union bpf_iter_link_info linfo = {};
+
+ linfo.cgroup.cgroup_fd = root_fd;
+ linfo.cgroup.order = BPF_CGROUP_ITER_DESCENDANTS_PRE;
+ opts.link_info = &linfo;
+ opts.link_info_len = sizeof(linfo);
+
+ link = bpf_program__attach_iter(skel->progs.cgroup_memcg_stat_reader,
+ &opts);
+ if (!ASSERT_OK_PTR(link, "attach iter"))
+ goto out;
+
+ tmp = calloc(n_nodes + 8, sizeof(*tmp));
+ if (!ASSERT_OK_PTR(tmp, "calloc tmp"))
+ goto out;
+
+ /*
+ * Authoritative completeness/correctness gate: run once on the freshly
+ * loaded (still empty) map, so missing==0 proves this walk visited every
+ * cgroup. The map is not cleared between the later timed walks, so the
+ * end-of-loop count is only a weaker, informational cross-check.
+ */
+ check_structural(link, skel);
+
+ /* warm caches/vmstats for both paths symmetrically (same gap regime) */
+ for (w = 0; w < WARMUP_ITERS; w++) {
+ long long ro;
+ int got;
+
+ file_pass(tc->gap_us);
+ bpf_pass(link, skel, tmp, &ro, &got, NULL, tc->gap_us);
+ }
+
+ /*
+ * Timed samples. Every read idles tc->gap_us (untimed) first, so the
+ * tree re-accumulates a roughly fixed amount of churn and each read
+ * starts from approximately the same state, paying its own rstat flush
+ * inside the timed region. The file/bpf order is flipped on odd samples
+ * so any residual jitter doesn't systematically favour whichever reader
+ * runs first.
+ */
+ for (s = 0; s < tc->samples; s++) {
+ long long ro;
+ int got, werr;
+
+ if (s & 1) {
+ acc.bpf_ns += bpf_pass(link, skel, tmp, &ro, &got, &werr, tc->gap_us);
+ acc.bpf_ro_ns += ro;
+ acc.last_got = got;
+ bad_walks += !!werr;
+ acc.file_ns += file_pass(tc->gap_us);
+ } else {
+ acc.file_ns += file_pass(tc->gap_us);
+ acc.bpf_ns += bpf_pass(link, skel, tmp, &ro, &got, &werr, tc->gap_us);
+ acc.bpf_ro_ns += ro;
+ acc.last_got = got;
+ bad_walks += !!werr;
+ }
+ }
+
+ f = (double)acc.file_ns / tc->samples / 1000.0;
+ b = (double)acc.bpf_ns / tc->samples / 1000.0;
+ bro = (double)acc.bpf_ro_ns / tc->samples / 1000.0;
+
+ /*
+ * Informational timing diagnostic (captured like any test output: shown
+ * under -v or on failure, not on a normal PASS). The pass/fail verdict
+ * comes solely from the structural checks above.
+ */
+ printf("\n==== memcg_stat_churn: %s ====\n", tc->name);
+ printf("tree: nodes=%d leaves=%d churners=%d fanout=%d depth=%d region=%zuKB resident=%zuKB samples=%d gap=%dms\n",
+ n_nodes, n_leaves, churners, tc->fanout, tc->depth,
+ tc->region_bytes >> 10, tc->resident_bytes >> 10, tc->samples,
+ tc->gap_us / 1000);
+ printf("all times in us (average per full-tree read under churn, full memory.stat field set); ratio = file/bpf; ro = bpf read()-only (no map drain)\n");
+ printf("each read idles gap=%dms first so every read starts from ~the same accumulated churn; the rstat flush is counted in the read\n",
+ tc->gap_us / 1000);
+ printf("file_avg=%.1f bpf_avg=%.1f bpf_ro=%.1f ratio(file/bpf)=%.2fx\n",
+ f, b, bro, b > 0 ? f / b : 0.0);
+ printf("per-cgroup: file avg=%.0f ns bpf avg=%.0f ns\n",
+ f * 1000.0 / n_nodes, b * 1000.0 / n_nodes);
+ printf("bpf entries produced: %d (expected %d)\n", acc.last_got, n_nodes);
+ printf("RATIO (under churn): file/bpf = %.2fx\n", b > 0 ? f / b : 0.0);
+
+ ASSERT_EQ(bad_walks, 0, "all timed bpf walks completed");
+ ASSERT_EQ(acc.last_got, n_nodes, "bpf visited whole subtree under churn");
+
+out:
+ free(tmp);
+ bpf_link__destroy(link);
+ memcg_stat_reader__destroy(skel);
+ if (root_fd >= 0)
+ close(root_fd);
+ stop_churners(); /* reap churners so leaves become removable */
+
+ /*
+ * Remove the subtree in reverse creation order. Nodes are recorded in
+ * DFS pre-order (a parent precedes all its descendants), so iterating
+ * backwards removes every child before its parent.
+ */
+ if (nodes) {
+ int i;
+
+ for (i = n_nodes - 1; i >= 0; i--)
+ remove_cgroup(nodes[i].rel);
+ free(nodes);
+ nodes = NULL;
+ }
+}
+
+/*
+ * gap_us: idle time inserted (untimed) before every read so the tree
+ * re-accumulates a roughly fixed amount of dirty rstat first; the read then
+ * pays that flush inside its timed region. This gives all four reads
+ * (file/bpf x matched/full) approximately the same start state and folds the
+ * flush cost into the measured time. It is the "staleness / poll-interval"
+ * knob: larger gap -> larger common flush -> the file/bpf ratio compresses.
+ * Pick it past the point where the flush cost saturates; validate by checking
+ * that bpf matched <= bpf full is restored and that doubling it barely moves
+ * the numbers. 50 ms is a reasonable default here.
+ */
+#define CHURN_GAP_US (50 * 1000)
+
+static const struct testcase cases[] = {
+ /*
+ * Both cases use a large (1111-cgroup) tree, where the whole-tree read is
+ * big enough that its cost dominates the rstat flush and scheduler jitter,
+ * so the reported ratios are reproducible run to run; on a small (tens of
+ * cgroups) tree the sub-millisecond BPF read is swamped by that noise and
+ * the ratio bounces. They differ only in churn density -- large_dense
+ * churns one leaf in 4, large_sparse one in 8 -- which changes how much of
+ * the tree the shared flush has to touch. samples are kept even so the
+ * file/bpf order-alternation (s & 1) cancels residual first-mover bias.
+ */
+ /* name fan dep frac region resident samp gap */
+ { "large_dense", 10, 3, 4, 256 << 10, 128 << 10, 8, CHURN_GAP_US },
+ { "large_sparse", 10, 3, 8, 256 << 10, 128 << 10, 8, CHURN_GAP_US },
+};
+
+/*
+ * The memcg kfuncs the reused BPF program relies on (bpf_get_mem_cgroup et al.)
+ * are built only with CONFIG_MEMCG (mm/bpf_memcontrol.c). On a kernel without
+ * it they are absent from vmlinux BTF and the program fails to load, so probe
+ * for one and skip cleanly rather than reporting a spurious failure.
+ */
+static bool memcg_kfuncs_available(void)
+{
+ struct btf *btf;
+ bool ok;
+
+ btf = btf__load_vmlinux_btf();
+ if (!btf)
+ return false;
+ ok = btf__find_by_name_kind(btf, "bpf_get_mem_cgroup", BTF_KIND_FUNC) > 0;
+ btf__free(btf);
+ return ok;
+}
+
+void serial_test_memcg_stat_churn(void)
+{
+ int i;
+
+ if (!memcg_kfuncs_available()) {
+ test__skip();
+ return;
+ }
+
+ if (!ASSERT_OK(setup_cgroup_environment(), "setup cgroup env"))
+ return;
+
+ for (i = 0; i < ARRAY_SIZE(cases); i++) {
+ if (!test__start_subtest(cases[i].name))
+ continue;
+ run_case(&cases[i]);
+ }
+
+ cleanup_cgroup_environment();
+}
diff --git a/tools/testing/selftests/bpf/prog_tests/memcg_stat_churn_percpu.c b/tools/testing/selftests/bpf/prog_tests/memcg_stat_churn_percpu.c
new file mode 100644
index 00000000000000..16c3f261c87802
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/memcg_stat_churn_percpu.c
@@ -0,0 +1,902 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+/*
+ * memcg_stat_churn_percpu
+ * =======================
+ * A CPU-spread variant of memcg_stat_churn. It measures the same two whole-tree
+ * memcg-stat readers -- the traditional per-cgroup memory.stat parse and a single
+ * SEC("iter.s/cgroup") BPF walk -- under continuous allocation churn, but this
+ * time each churned cgroup is deliberately dirtied on MANY CPUs at once:
+ *
+ * - Build a synthetic cgroup subtree (fanout x depth), same as the reader.
+ * - Reserve one CPU for the reader and pin the reader (this parent) to it, so
+ * the timed reads run on a churn-free CPU and are not preempted by the load
+ * (measuring on a contended CPU swamps a short walk with scheduler latency).
+ * - Fork one "churner" process per hot leaf (tc->churn_leaves; 0 = all leaves,
+ * spread evenly across the tree). Each churner joins its leaf and then, in a
+ * loop, migrates its own affinity round-robin across K CPUs
+ * (tc->cpus_per_leaf; 0 = all churner CPUs) doing one mmap()+memset()+
+ * munmap() on each. The charge/uncharge happens on whatever CPU the task is
+ * currently running on, so cycling the affinity queues this leaf's rstat
+ * dirty on all K CPUs; those per-cpu entries persist until flushed, so
+ * between two reads the leaf ends up dirty on all K CPUs.
+ * - While the churn runs, the parent repeatedly SAMPLES both readers exactly as
+ * in memcg_stat_churn: settle_flush() then an untimed gap before each read so
+ * each read starts from exactly gap-worth of churn and pays its own flush
+ * inside the timed region; the file/BPF order is alternated.
+ *
+ * Why this matters: both readers flush rstat through the same
+ * mem_cgroup_flush_stats() path, and the cost of that flush grows with the
+ * number of (cgroup, cpu) pairs that have pending updates. Where memcg_stat_churn
+ * dirties each cgroup on essentially one CPU (a read flushes one per-cpu tree per
+ * cgroup), this test makes K a first-class knob: a read of a cgroup dirtied on K
+ * CPUs must visit K per-cpu trees. Sweeping K (see the narrow/wide/widest cases)
+ * drives the shared flush cost F up and compresses the file/BPF ratio, isolating
+ * the effect of per-cgroup cross-CPU fan-out. Because the reader runs on a
+ * reserved, churn-free CPU, the flush cost -- not scheduler jitter -- is what the
+ * timed reads capture; the ratio is robust because the flush hits both readers
+ * equally.
+ *
+ * The BPF program, its hash map and the snapshot struct are REUSED verbatim from
+ * memcg_stat_reader (progs/memcg_stat_reader.c + memcg_stat_reader.h); only the
+ * userspace load model (CPU-pinned churners) and sampling loop are new here.
+ *
+ * Under churn the stats are a moving target, so this test does NOT do a
+ * field-by-field BPF-vs-file equality check (that is memcg_stat_reader's job).
+ * Pass/fail gates only on structural sanity -- the iterator visited every cgroup
+ * and the tree carries some anon charge. The timing table and final RATIO line
+ * are informational diagnostics, printed like any other test output (i.e. under
+ * -v or on failure, never on a normal PASS).
+ */
+#ifndef _GNU_SOURCE
+#define _GNU_SOURCE /* sched_setaffinity(), CPU_SET() (lib.mk also -D's it) */
+#endif
+#include <test_progs.h>
+#include <bpf/libbpf.h>
+#include <bpf/btf.h>
+#include <sched.h>
+#include <stdlib.h>
+#include <string.h>
+#include <time.h>
+#include <unistd.h>
+#include <sys/mman.h>
+#include <sys/wait.h>
+#include "cgroup_helpers.h"
+#include "memcg_stat_reader.h"
+#include "memcg_stat_reader.skel.h"
+
+#define SUBTREE_ROOT "/mcg_pcpu"
+
+#define WARMUP_ITERS 2
+
+struct cg_node {
+ char rel[128];
+ __u64 id;
+ bool is_leaf;
+};
+
+/* Field subset parsed from memory.stat (mirrors memcg_stat_reader). */
+struct file_snap {
+ __u64 anon, file, shmem, file_mapped, pgfault;
+ __u64 current; /* memory.current, bytes */
+ __u64 max; /* memory.max, bytes (valid unless max_is_max) */
+ __u64 full_sum;
+ __u32 full_fields;
+ bool max_is_max;
+};
+
+static volatile __u64 sink; /* keep the optimizer from eliding reads */
+
+static long long now_ns(void)
+{
+ struct timespec t;
+
+ clock_gettime(CLOCK_MONOTONIC, &t);
+ return (long long)t.tv_sec * 1000000000LL + t.tv_nsec;
+}
+
+/* ---- allowed CPU set --------------------------------------------------- */
+
+static int *cpu_list; /* ids of the CPUs this process may run on */
+static int n_cpu; /* number of such CPUs */
+static int n_reserved; /* CPUs held out for the reader (0 or 1) */
+static int reader_cpu = -1; /* CPU the reader is pinned to, or -1 */
+static cpu_set_t orig_affinity; /* parent's affinity, restored on exit */
+
+/*
+ * Collect the CPUs the test is allowed to run on (respecting any cpuset the
+ * harness put us in). We spawn one pinned churner per usable CPU minus the one
+ * reserved for the reader. Bounded by CPU_SETSIZE (1024); machines wider than
+ * that would use only the first CPU_SETSIZE CPUs, which is fine for a diagnostic.
+ */
+static int collect_cpus(void)
+{
+ cpu_set_t set;
+ int i, want, n = 0;
+
+ CPU_ZERO(&set);
+ if (sched_getaffinity(0, sizeof(set), &set))
+ return -1;
+ orig_affinity = set; /* restored in test teardown */
+ want = CPU_COUNT(&set);
+ if (want <= 0)
+ return -1;
+ cpu_list = calloc(want, sizeof(*cpu_list));
+ if (!cpu_list)
+ return -1;
+ for (i = 0; i < CPU_SETSIZE && n < want; i++)
+ if (CPU_ISSET(i, &set))
+ cpu_list[n++] = i;
+ n_cpu = n;
+ return 0;
+}
+
+/* ---- tree construction (same shape as memcg_stat_reader) --------------- */
+
+static struct cg_node *nodes;
+static int n_nodes;
+static int n_leaves;
+
+static int add_node(const char *rel, bool is_leaf, int *keep_fd)
+{
+ int fd;
+
+ fd = create_and_get_cgroup(rel);
+ if (fd < 0)
+ return -1;
+ if (keep_fd)
+ *keep_fd = fd;
+ else
+ close(fd);
+
+ strncpy(nodes[n_nodes].rel, rel, sizeof(nodes[n_nodes].rel) - 1);
+ nodes[n_nodes].rel[sizeof(nodes[n_nodes].rel) - 1] = '\0';
+ nodes[n_nodes].id = get_cgroup_id(rel);
+ nodes[n_nodes].is_leaf = is_leaf;
+ if (is_leaf)
+ n_leaves++;
+ n_nodes++;
+ return 0;
+}
+
+/* Recursively create children of @rel. @rel must already exist and be recorded. */
+static int build_children(const char *rel, int fanout, int depth)
+{
+ char child[128];
+ int i;
+
+ if (depth == 0)
+ return 0;
+
+ /* Enable memory on this interior node so its children get a memcg. */
+ if (enable_controllers(rel, "memory"))
+ return -1;
+
+ for (i = 0; i < fanout; i++) {
+ snprintf(child, sizeof(child), "%s/c%d", rel, i);
+ if (add_node(child, depth == 1, NULL))
+ return -1;
+ if (build_children(child, fanout, depth - 1))
+ return -1;
+ }
+ return 0;
+}
+
+static size_t tree_capacity(int fanout, int depth)
+{
+ size_t total = 1, level = 1;
+ int d;
+
+ for (d = 0; d < depth; d++) {
+ level *= fanout;
+ total += level;
+ }
+ return total;
+}
+
+static int build_tree(int fanout, int depth, int *root_fd)
+{
+ n_nodes = 0;
+ n_leaves = 0;
+ nodes = calloc(tree_capacity(fanout, depth), sizeof(*nodes));
+ if (!nodes)
+ return -1;
+
+ if (add_node(SUBTREE_ROOT, depth == 0, root_fd))
+ return -1;
+ return build_children(SUBTREE_ROOT, fanout, depth);
+}
+
+/* ---- churn load (migrating churners, K CPUs per hot cgroup) ------------- */
+
+/*
+ * Shared control block, mmap'd MAP_SHARED before the forks so the parent can
+ * signal all churners to stop with a single write.
+ */
+struct churn_ctl {
+ volatile int stop;
+};
+
+static struct churn_ctl *ctl;
+static pid_t *churn_pids;
+static int n_churners;
+static int n_hot_leaves; /* distinct cgroups (leaves) churned */
+static int n_cpus_per_leaf; /* K: CPUs each hot cgroup is dirtied on */
+static int churn_ready[2] = { -1, -1 }; /* churner -> parent "ready" barrier */
+
+/* Pin the calling task to a single CPU. */
+static int pin_cpu(int cpu)
+{
+ cpu_set_t set;
+
+ CPU_ZERO(&set);
+ CPU_SET(cpu, &set);
+ return sched_setaffinity(0, sizeof(set), &set);
+}
+
+/*
+ * One churner process, dedicated to a single hot leaf but spread over K CPUs.
+ * It joins its leaf, pins a resident anon set so the tree always carries some
+ * charge, signals readiness, then loops: migrate to the next of its K CPUs and
+ * do one mmap()+memset()+munmap() there. The charge/uncharge happens on
+ * whatever CPU the task currently runs on, so cycling the affinity queues this
+ * leaf's rstat dirty on all K CPUs; those per-cpu entries persist until flushed,
+ * so between two reads the leaf ends up dirty on all K CPUs and a reader's flush
+ * of the subtree must visit K per-cpu trees for this one cgroup. Never returns.
+ *
+ * @base is this churner's starting index into the churner CPU pool
+ * (cpu_list[n_reserved ..]); its K CPUs are (base + 0..K-1) mod pool size.
+ */
+static void churner_child(const struct cg_node *leaf, int base, int k,
+ size_t region_bytes, size_t resident_bytes)
+{
+ int c_pool = n_cpu - n_reserved;
+ void *resident;
+ int j = 0;
+
+ close(churn_ready[0]);
+
+ /*
+ * Move onto our first CPU before charging. Children inherit the reader's
+ * reserved-CPU affinity from the parent, so without this the resident set
+ * would be charged on the reader's CPU.
+ */
+ if (pin_cpu(cpu_list[n_reserved + base % c_pool]))
+ _exit(4);
+
+ /*
+ * cgroup_helpers builds paths from getpid(); in this forked child that
+ * differs from the parent that built the tree, so use the _parent
+ * (getppid()) variant to resolve the leaf under the parent's work dir.
+ */
+ if (join_parent_cgroup(leaf->rel))
+ _exit(1);
+
+ resident = mmap(NULL, resident_bytes, PROT_READ | PROT_WRITE,
+ MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
+ if (resident == MAP_FAILED)
+ _exit(2);
+ memset(resident, 1, resident_bytes); /* fault in, keep mapped */
+
+ if (write(churn_ready[1], "x", 1) != 1)
+ _exit(3);
+ close(churn_ready[1]); /* so a sibling's early death yields EOF, not a parent hang */
+
+ while (!ctl->stop) {
+ void *p;
+
+ /* migrate to the next of our K CPUs, then dirty the leaf there */
+ pin_cpu(cpu_list[n_reserved + (base + j) % c_pool]);
+ if (++j == k)
+ j = 0;
+
+ p = mmap(NULL, region_bytes, PROT_READ | PROT_WRITE,
+ MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
+ if (p == MAP_FAILED)
+ continue;
+ memset(p, 1, region_bytes); /* fault in -> anon charge */
+ munmap(p, region_bytes); /* free -> uncharge (keeps rstat dirty) */
+ }
+ munmap(resident, resident_bytes);
+ _exit(0);
+}
+
+/*
+ * Fork one migrating churner per hot leaf (H = @churn_leaves, 0 = all leaves),
+ * each spread over K CPUs (@cpus_per_leaf, 0 or > pool => all churner CPUs).
+ * Hot leaves are chosen evenly across the tree. Returns 0 once every churner
+ * has joined its leaf and pinned its resident set (so measurement starts under
+ * steady-state load). On failure the caller's cleanup path calls
+ * stop_churners() to reap whatever was started.
+ */
+static int start_churners(size_t region_bytes, size_t resident_bytes,
+ int churn_leaves, int cpus_per_leaf)
+{
+ int *leaf_idx = NULL, *pool = NULL;
+ int n_leaf_idx = 0, pool_n = 0;
+ int c_pool = n_cpu - n_reserved; /* CPUs available to churners */
+ int k_eff, i, h, ret = -1;
+
+ /* K: CPUs each hot cgroup is dirtied on (0 or too big => all churner CPUs) */
+ k_eff = (cpus_per_leaf > 0 && cpus_per_leaf < c_pool) ? cpus_per_leaf
+ : c_pool;
+ n_cpus_per_leaf = k_eff;
+
+ /* gather all leaf node indices in creation order */
+ leaf_idx = calloc(n_leaves, sizeof(*leaf_idx));
+ if (!ASSERT_OK_PTR(leaf_idx, "calloc leaf_idx"))
+ return -1;
+ for (i = 0; i < n_nodes; i++)
+ if (nodes[i].is_leaf)
+ leaf_idx[n_leaf_idx++] = i;
+
+ /*
+ * H hot cgroups (one migrating churner each), spread evenly across all
+ * leaves; churn_leaves <= 0 or >= n_leaves => every leaf is hot.
+ */
+ pool_n = (churn_leaves > 0 && churn_leaves < n_leaf_idx) ? churn_leaves
+ : n_leaf_idx;
+ pool = calloc(pool_n, sizeof(*pool));
+ if (!ASSERT_OK_PTR(pool, "calloc pool")) {
+ free(leaf_idx);
+ return -1;
+ }
+ for (i = 0; i < pool_n; i++)
+ pool[i] = leaf_idx[(int)((long long)i * n_leaf_idx / pool_n)];
+ free(leaf_idx);
+ n_hot_leaves = pool_n;
+
+ ctl = mmap(NULL, sizeof(*ctl), PROT_READ | PROT_WRITE,
+ MAP_SHARED | MAP_ANONYMOUS, -1, 0);
+ if (!ASSERT_NEQ(ctl, MAP_FAILED, "mmap churn_ctl")) {
+ ctl = NULL;
+ goto out;
+ }
+ ctl->stop = 0;
+
+ if (!ASSERT_OK(pipe(churn_ready), "pipe churn_ready"))
+ goto out;
+
+ churn_pids = calloc(pool_n, sizeof(*churn_pids));
+ if (!ASSERT_OK_PTR(churn_pids, "calloc churn_pids"))
+ goto out;
+
+ /*
+ * One migrating churner per hot leaf. Churner h spans the K CPUs
+ * (h*K + 0..K-1) mod c_pool of the churner pool (cpu_list[n_reserved ..]),
+ * so different churners start on different CPUs and cpu_list[0] (the
+ * reader's reserved CPU) is never used by a churner.
+ */
+ for (h = 0; h < pool_n; h++) {
+ const struct cg_node *leaf = &nodes[pool[h]];
+ pid_t pid = fork();
+
+ if (pid < 0) {
+ ASSERT_GE(pid, 0, "fork churner");
+ goto out;
+ }
+ if (pid == 0)
+ churner_child(leaf, h * k_eff, k_eff,
+ region_bytes, resident_bytes);
+
+ churn_pids[n_churners++] = pid;
+ }
+
+ /* parent: this end is only for the children to signal on */
+ close(churn_ready[1]);
+ churn_ready[1] = -1;
+
+ /* wait until every churner has joined + pinned its resident set */
+ for (h = 0; h < n_churners; h++) {
+ char c;
+ ssize_t r = read(churn_ready[0], &c, 1);
+
+ if (r == 0)
+ fprintf(stderr,
+ "a churner exited before signaling ready (affinity/join/mmap failure?)\n");
+ if (!ASSERT_EQ(r, 1, "churner ready"))
+ goto out;
+ }
+ ret = 0;
+out:
+ free(pool);
+ return ret;
+}
+
+static void stop_churners(void)
+{
+ int i, status;
+
+ if (ctl)
+ ctl->stop = 1; /* release all churn loops */
+
+ if (churn_ready[1] >= 0) {
+ close(churn_ready[1]);
+ churn_ready[1] = -1;
+ }
+ if (churn_ready[0] >= 0) {
+ close(churn_ready[0]);
+ churn_ready[0] = -1;
+ }
+
+ for (i = 0; i < n_churners; i++) {
+ if (!churn_pids || churn_pids[i] <= 0)
+ continue;
+ if (waitpid(churn_pids[i], &status, 0) == churn_pids[i] &&
+ (!WIFEXITED(status) || WEXITSTATUS(status) != 0))
+ fprintf(stderr,
+ "churner %d exited abnormally (status=0x%x)\n",
+ churn_pids[i], status);
+ }
+
+ free(churn_pids);
+ churn_pids = NULL;
+ n_churners = 0;
+
+ if (ctl) {
+ munmap((void *)ctl, sizeof(*ctl));
+ ctl = NULL;
+ }
+}
+
+/* ---- file (traditional) reader ----------------------------------------- */
+
+static void parse_stat(char *buf, struct file_snap *o)
+{
+ char *save, *line;
+
+ for (line = strtok_r(buf, "\n", &save); line;
+ line = strtok_r(NULL, "\n", &save)) {
+ unsigned long long val;
+ char name[64];
+
+ if (sscanf(line, "%63s %llu", name, &val) != 2)
+ continue;
+ o->full_sum += val;
+ o->full_fields++;
+ if (!strcmp(name, "anon"))
+ o->anon = val;
+ else if (!strcmp(name, "file"))
+ o->file = val;
+ else if (!strcmp(name, "shmem"))
+ o->shmem = val;
+ else if (!strcmp(name, "file_mapped"))
+ o->file_mapped = val;
+ else if (!strcmp(name, "pgfault"))
+ o->pgfault = val;
+ }
+}
+
+static int file_read_node(const char *rel, struct file_snap *o)
+{
+ char buf[8192];
+
+ memset(o, 0, sizeof(*o));
+
+ if (read_cgroup_file(rel, "memory.stat", buf, sizeof(buf)))
+ return -1;
+ parse_stat(buf, o);
+
+ if (!read_cgroup_file(rel, "memory.current", buf, sizeof(buf)))
+ o->current = strtoull(buf, NULL, 10);
+ if (!read_cgroup_file(rel, "memory.max", buf, sizeof(buf))) {
+ if (!strncmp(buf, "max", 3))
+ o->max_is_max = true;
+ else
+ o->max = strtoull(buf, NULL, 10);
+ }
+ return 0;
+}
+
+/*
+ * Untimed whole-subtree flush used to normalise the pre-read state. Reading the
+ * subtree root's memory.stat flushes the entire subtree's rstat, so the
+ * usleep(gap) that follows re-accumulates *exactly* gap-worth of churn no matter
+ * what the previous timed read was. Without this reset the accumulation window
+ * would be (previous_read_duration + gap), and since a file pass is ~20x longer
+ * than a BPF walk that made the flush a BPF read pays depend on ordering -- a
+ * ~15% window asymmetry, enough to invert bpf_matched vs bpf_full on
+ * flush-dominated cases (e.g. "hot": few cgroups churned from many CPUs).
+ * nodes[0] is SUBTREE_ROOT (added first in build_tree), whose memcg covers the
+ * whole tree, so one read here flushes every node the timed reads care about.
+ */
+static void settle_flush(void)
+{
+ char buf[8192];
+
+ if (nodes && n_nodes > 0)
+ read_cgroup_file(nodes[0].rel, "memory.stat", buf, sizeof(buf));
+}
+
+/*
+ * One timed traditional pass over the whole tree; returns nanoseconds. A
+ * settle_flush() then an untimed @gap_us idle precede the pass so the tree
+ * re-accumulates exactly gap-worth of churn first; the resulting rstat flush is
+ * then paid inside the timed region, giving every read the same start state.
+ */
+static long long file_pass(int gap_us)
+{
+ struct file_snap s;
+ long long t0;
+ int i;
+
+ settle_flush();
+ if (gap_us)
+ usleep(gap_us);
+ t0 = now_ns();
+ for (i = 0; i < n_nodes; i++) {
+ file_read_node(nodes[i].rel, &s);
+ sink += s.anon + s.full_sum;
+ }
+ return now_ns() - t0;
+}
+
+/* ---- BPF reader -------------------------------------------------------- */
+
+static int bpf_walk_once(struct bpf_link *link)
+{
+ char buf[4096];
+ ssize_t r;
+ int fd;
+
+ fd = bpf_iter_create(bpf_link__fd(link));
+ if (fd < 0)
+ return -1;
+ while ((r = read(fd, buf, sizeof(buf))) > 0)
+ ;
+ close(fd);
+ return r == 0 ? 0 : -1;
+}
+
+static int drain_map(int mfd, struct memcg_stat_snapshot *out, int max)
+{
+ __u64 key = 0, next;
+ int n = 0, err;
+
+ err = bpf_map_get_next_key(mfd, NULL, &next);
+ while (err == 0) {
+ if (n < max && !bpf_map_lookup_elem(mfd, &next, &out[n])) {
+ sink += out[n].anon + out[n].full_sum;
+ n++;
+ }
+ key = next;
+ err = bpf_map_get_next_key(mfd, &key, &next);
+ }
+ return n;
+}
+
+/*
+ * One timed BPF pass: kernel walk (ro) + map drain into userspace. Returns the
+ * total nanoseconds; *ro_ns gets the walk-only time, *got the entries drained.
+ * A settle_flush() then an untimed @gap_us idle precede the walk, exactly as in
+ * file_pass(), so the rstat flush the walk pays reflects the same gap-worth of
+ * accumulated churn regardless of read ordering.
+ */
+static long long bpf_pass(struct bpf_link *link, struct memcg_stat_reader *skel,
+ struct memcg_stat_snapshot *tmp,
+ long long *ro_ns, int *got, int *werr, int gap_us)
+{
+ int mfd = bpf_map__fd(skel->maps.results);
+ long long t0, t1, t2;
+ int err;
+
+ skel->bss->collect_full = 1;
+
+ settle_flush();
+ if (gap_us)
+ usleep(gap_us);
+ t0 = now_ns();
+ err = bpf_walk_once(link);
+ t1 = now_ns();
+ *got = drain_map(mfd, tmp, n_nodes + 8);
+ t2 = now_ns();
+
+ if (werr)
+ *werr = err;
+ *ro_ns = t1 - t0;
+ return t2 - t0;
+}
+
+/* ---- structural sanity (no field-by-field check under churn) ------------ */
+
+static void check_structural(struct bpf_link *link,
+ struct memcg_stat_reader *skel)
+{
+ int mfd = bpf_map__fd(skel->maps.results);
+ __u64 total_anon = 0;
+ int i, missing = 0;
+
+ skel->bss->collect_full = 0;
+ if (!ASSERT_OK(bpf_walk_once(link), "bpf walk"))
+ return;
+
+ for (i = 0; i < n_nodes; i++) {
+ struct memcg_stat_snapshot b;
+
+ if (bpf_map_lookup_elem(mfd, &nodes[i].id, &b)) {
+ missing++;
+ continue;
+ }
+ total_anon += b.anon;
+ }
+
+ ASSERT_EQ(missing, 0, "all cgroups present in map");
+ /*
+ * The churners pin a resident anon set for the whole window, so with no
+ * swap and no ancestor memory.max forcing reclaim (the base selftest
+ * config sets neither), the tree always carries anon while churn runs.
+ */
+ ASSERT_GT(total_anon, 0, "tree carries anon under churn");
+}
+
+/* ---- one case ---------------------------------------------------------- */
+
+struct sample_acc {
+ long long file_ns;
+ long long bpf_ns, bpf_ro_ns;
+ int last_got;
+};
+
+struct testcase {
+ const char *name;
+ int fanout;
+ int depth;
+ int churn_leaves; /* H: # hot cgroups (one migrating churner each); 0 = all leaves */
+ int cpus_per_leaf; /* K: CPUs each hot cgroup is dirtied on; 0 = all churner CPUs */
+ size_t region_bytes; /* per-iteration churn region */
+ size_t resident_bytes; /* pinned resident set per churner */
+ int samples;
+ int gap_us; /* idle before EACH read: the "staleness" knob (see cases[]) */
+};
+
+static void run_case(const struct testcase *tc)
+{
+ struct memcg_stat_snapshot *tmp = NULL;
+ struct memcg_stat_reader *skel = NULL;
+ struct bpf_link *link = NULL;
+ struct sample_acc acc = {};
+ double f, b, bro;
+ int root_fd = -1;
+ int churners = 0;
+ int bad_walks = 0;
+ int s, w;
+
+ if (!ASSERT_OK(build_tree(tc->fanout, tc->depth, &root_fd), "build tree"))
+ goto out;
+
+ if (start_churners(tc->region_bytes, tc->resident_bytes,
+ tc->churn_leaves, tc->cpus_per_leaf))
+ goto out;
+ churners = n_churners;
+
+ skel = memcg_stat_reader__open();
+ if (!ASSERT_OK_PTR(skel, "skel open"))
+ goto out;
+ if (!ASSERT_OK(bpf_map__set_max_entries(skel->maps.results, n_nodes + 8),
+ "set max_entries"))
+ goto out;
+ if (!ASSERT_OK(memcg_stat_reader__load(skel), "skel load"))
+ goto out;
+
+ DECLARE_LIBBPF_OPTS(bpf_iter_attach_opts, opts);
+ union bpf_iter_link_info linfo = {};
+
+ linfo.cgroup.cgroup_fd = root_fd;
+ linfo.cgroup.order = BPF_CGROUP_ITER_DESCENDANTS_PRE;
+ opts.link_info = &linfo;
+ opts.link_info_len = sizeof(linfo);
+
+ link = bpf_program__attach_iter(skel->progs.cgroup_memcg_stat_reader,
+ &opts);
+ if (!ASSERT_OK_PTR(link, "attach iter"))
+ goto out;
+
+ tmp = calloc(n_nodes + 8, sizeof(*tmp));
+ if (!ASSERT_OK_PTR(tmp, "calloc tmp"))
+ goto out;
+
+ /*
+ * Authoritative completeness/correctness gate: run once on the freshly
+ * loaded (still empty) map, so missing==0 proves this walk visited every
+ * cgroup. The map is not cleared between the later timed walks, so the
+ * end-of-loop count is only a weaker, informational cross-check.
+ */
+ check_structural(link, skel);
+
+ /* warm caches/vmstats for both paths symmetrically (same gap regime) */
+ for (w = 0; w < WARMUP_ITERS; w++) {
+ long long ro;
+ int got;
+
+ file_pass(tc->gap_us);
+ bpf_pass(link, skel, tmp, &ro, &got, NULL, tc->gap_us);
+ }
+
+ /*
+ * Timed samples. Every read settle_flush()es then idles tc->gap_us
+ * (untimed) first, so the tree re-accumulates exactly gap-worth of churn
+ * and each read pays its own rstat flush inside the timed region. The
+ * file/bpf order is flipped on odd samples so any residual jitter doesn't
+ * systematically favour whichever reader runs first.
+ */
+ for (s = 0; s < tc->samples; s++) {
+ long long ro;
+ int got, werr;
+
+ if (s & 1) {
+ acc.bpf_ns += bpf_pass(link, skel, tmp, &ro, &got, &werr, tc->gap_us);
+ acc.bpf_ro_ns += ro;
+ acc.last_got = got;
+ bad_walks += !!werr;
+ acc.file_ns += file_pass(tc->gap_us);
+ } else {
+ acc.file_ns += file_pass(tc->gap_us);
+ acc.bpf_ns += bpf_pass(link, skel, tmp, &ro, &got, &werr, tc->gap_us);
+ acc.bpf_ro_ns += ro;
+ acc.last_got = got;
+ bad_walks += !!werr;
+ }
+ }
+
+ f = (double)acc.file_ns / tc->samples / 1000.0;
+ b = (double)acc.bpf_ns / tc->samples / 1000.0;
+ bro = (double)acc.bpf_ro_ns / tc->samples / 1000.0;
+
+ /*
+ * Informational timing diagnostic (captured like any test output: shown
+ * under -v or on failure, not on a normal PASS). The pass/fail verdict
+ * comes solely from the structural checks above.
+ */
+ printf("\n==== memcg_stat_churn_percpu: %s ====\n", tc->name);
+ printf("tree: nodes=%d leaves=%d hot_leaves=%d cpus_per_leaf=%d dirty_pairs=%d cpus=%d reserved=%d churners=%d fanout=%d depth=%d region=%zuKB resident=%zuKB samples=%d gap=%dms\n",
+ n_nodes, n_leaves, n_hot_leaves, n_cpus_per_leaf,
+ n_hot_leaves * n_cpus_per_leaf, n_cpu, n_reserved, churners,
+ tc->fanout, tc->depth, tc->region_bytes >> 10,
+ tc->resident_bytes >> 10, tc->samples, tc->gap_us / 1000);
+ printf("each hot cgroup churned across %d CPUs (migrating churner) so a reader flush visits ~%d per-cpu trees per hot cgroup; reader pinned to reserved CPU %d\n",
+ n_cpus_per_leaf, n_cpus_per_leaf, reader_cpu);
+ printf("all times in us (average per full-tree read under churn, full memory.stat field set); ratio = file/bpf; ro = bpf read()-only (no map drain)\n");
+ printf("each read flushes then idles gap=%dms so every read starts from exactly gap-worth of churn; the rstat flush is counted in the read\n",
+ tc->gap_us / 1000);
+ printf("file_avg=%.1f bpf_avg=%.1f bpf_ro=%.1f ratio(file/bpf)=%.2fx\n",
+ f, b, bro, b > 0 ? f / b : 0.0);
+ printf("per-cgroup: file avg=%.0f ns bpf avg=%.0f ns\n",
+ f * 1000.0 / n_nodes, b * 1000.0 / n_nodes);
+ printf("bpf entries produced: %d (expected %d)\n", acc.last_got, n_nodes);
+ printf("RATIO (%d CPUs/cgroup): file/bpf = %.2fx\n",
+ n_cpus_per_leaf, b > 0 ? f / b : 0.0);
+
+ ASSERT_EQ(bad_walks, 0, "all timed bpf walks completed");
+ ASSERT_EQ(acc.last_got, n_nodes, "bpf visited whole subtree under churn");
+
+out:
+ free(tmp);
+ bpf_link__destroy(link);
+ memcg_stat_reader__destroy(skel);
+ if (root_fd >= 0)
+ close(root_fd);
+ stop_churners(); /* reap churners so leaves become removable */
+
+ /*
+ * Remove the subtree in reverse creation order. Nodes are recorded in
+ * DFS pre-order (a parent precedes all its descendants), so iterating
+ * backwards removes every child before its parent.
+ */
+ if (nodes) {
+ int i;
+
+ for (i = n_nodes - 1; i >= 0; i--)
+ remove_cgroup(nodes[i].rel);
+ free(nodes);
+ nodes = NULL;
+ }
+}
+
+/*
+ * gap_us: idle time inserted (untimed) before every read so the tree
+ * re-accumulates a roughly fixed amount of dirty rstat first; the read then
+ * pays that flush inside its timed region. This gives all four reads
+ * (file/bpf x matched/full) approximately the same start state and folds the
+ * flush cost into the measured time. It is the "staleness / poll-interval"
+ * knob: larger gap -> larger common flush -> the file/bpf ratio compresses.
+ * See memcg_stat_churn for the full rationale; 50 ms is a reasonable default.
+ */
+#define CHURN_GAP_US (50 * 1000)
+
+static const struct testcase cases[] = {
+ /*
+ * The narrow/wide/widest trio runs on a large (1111-cgroup) tree and
+ * churns a fixed set of 64 hot leaves, sweeping only K, the number of CPUs
+ * each hot cgroup is dirtied on, to isolate per-cgroup cross-CPU flush
+ * fan-out:
+ *
+ * narrow - K=1 : each hot cgroup dirty on 1 CPU (64 x 1 dirty pairs).
+ * wide - K=8 : each hot cgroup dirty on 8 CPUs (64 x 8 dirty pairs).
+ * widest - K=all: each hot cgroup dirty on every churner CPU (64 x cpus).
+ *
+ * The hot-cgroup count (64) is identical across all three, so only the
+ * per-cgroup CPU fan-out changes. As K grows the shared rstat flush F grows
+ * (more per-cpu trees to visit), so both readers' cost rises and the
+ * file/bpf ratio compresses toward 1; widest is the most conservative
+ * regime. The large tree keeps the whole-tree read cost dominant over the
+ * flush/scheduler jitter, so the ratios are reproducible (a small tree makes
+ * the sub-millisecond BPF read too noisy to compare). samples are kept even
+ * so the file/bpf order-alternation (s & 1) cancels first-mover bias; widest
+ * gets more samples as its bigger flush has more variance.
+ */
+ /* name fan dep H K region resident samp gap */
+ { "narrow", 10, 3, 64, 1, 256 << 10, 128 << 10, 8, CHURN_GAP_US },
+ { "wide", 10, 3, 64, 8, 256 << 10, 128 << 10, 8, CHURN_GAP_US },
+ { "widest", 10, 3, 64, 0, 256 << 10, 128 << 10, 10, CHURN_GAP_US },
+};
+
+/*
+ * The memcg kfuncs the reused BPF program relies on (bpf_get_mem_cgroup et al.)
+ * are built only with CONFIG_MEMCG (mm/bpf_memcontrol.c). On a kernel without
+ * it they are absent from vmlinux BTF and the program fails to load, so probe
+ * for one and skip cleanly rather than reporting a spurious failure.
+ */
+static bool memcg_kfuncs_available(void)
+{
+ struct btf *btf;
+ bool ok;
+
+ btf = btf__load_vmlinux_btf();
+ if (!btf)
+ return false;
+ ok = btf__find_by_name_kind(btf, "bpf_get_mem_cgroup", BTF_KIND_FUNC) > 0;
+ btf__free(btf);
+ return ok;
+}
+
+/*
+ * Reserve one CPU for the reader (this parent) and pin the parent to it, so its
+ * timed reads run on a CPU that carries no churner. Needs at least 2 CPUs; on a
+ * single-CPU host we skip reserving and the one churner shares the CPU with the
+ * reader (noisier, but 1-CPU hosts are not the target). Best-effort: if the
+ * parent cannot be pinned we carry on without a reservation.
+ */
+static void reserve_reader_cpu(void)
+{
+ if (n_cpu < 2)
+ return;
+ if (pin_cpu(cpu_list[0]))
+ return;
+ n_reserved = 1;
+ reader_cpu = cpu_list[0];
+}
+
+void serial_test_memcg_stat_churn_percpu(void)
+{
+ int i;
+
+ if (!memcg_kfuncs_available()) {
+ test__skip();
+ return;
+ }
+
+ if (!ASSERT_OK(collect_cpus(), "collect cpus"))
+ return;
+
+ reserve_reader_cpu();
+
+ if (!ASSERT_OK(setup_cgroup_environment(), "setup cgroup env"))
+ goto restore;
+
+ for (i = 0; i < ARRAY_SIZE(cases); i++) {
+ if (!test__start_subtest(cases[i].name))
+ continue;
+ run_case(&cases[i]);
+ }
+
+ cleanup_cgroup_environment();
+restore:
+ /* undo reserve_reader_cpu() so later test_progs tests keep full affinity */
+ sched_setaffinity(0, sizeof(orig_affinity), &orig_affinity);
+ free(cpu_list);
+ cpu_list = NULL;
+ n_cpu = 0;
+ n_reserved = 0;
+ reader_cpu = -1;
+}
diff --git a/tools/testing/selftests/bpf/prog_tests/memcg_stat_reader.c b/tools/testing/selftests/bpf/prog_tests/memcg_stat_reader.c
new file mode 100644
index 00000000000000..b1e631b1520a0e
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/memcg_stat_reader.c
@@ -0,0 +1,617 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */
+
+/*
+ * memcg_stat_reader
+ * =================
+ * Read memory-cgroup statistics for a whole synthetic cgroup subtree TWO ways
+ * and compare them:
+ *
+ * (A) traditional: open+read+parse memory.stat / memory.current / memory.max
+ * for every cgroup, in userspace;
+ * (B) BPF: a single SEC("iter.s/cgroup") program walked over the subtree in
+ * DESCENDANTS_PRE order, calling the memcg kfuncs per cgroup and stashing
+ * the results in a hash map keyed by cgroup id, drained once afterwards.
+ *
+ * The test (a) asserts the BPF path agrees with the file path for a checked
+ * field subset (correctness) and (b) reports the wall-clock cost of each path
+ * reading the full ~memory.stat field set, across cgroup trees of increasing
+ * size and load.
+ *
+ * The pass/fail result depends only on the correctness checks; the timing table
+ * is an informational diagnostic captured like any other test output, i.e. shown
+ * only under -v (or when the test fails), never on a normal PASS.
+ */
+#include <test_progs.h>
+#include <bpf/libbpf.h>
+#include <bpf/btf.h>
+#include <stdlib.h>
+#include <string.h>
+#include <time.h>
+#include <unistd.h>
+#include <sys/mman.h>
+#include <sys/wait.h>
+#include "cgroup_helpers.h"
+#include "memcg_stat_reader.h"
+#include "memcg_stat_reader.skel.h"
+
+#define SUBTREE_ROOT "/mcg_stat"
+
+#define WARMUP_ITERS 3
+
+struct cg_node {
+ char rel[128];
+ __u64 id;
+ bool is_leaf;
+};
+
+/* Field subset the BPF prog reads matched against memory.stat by hand. */
+struct file_snap {
+ __u64 anon, file, shmem, file_mapped, pgfault;
+ __u64 current; /* memory.current, bytes */
+ __u64 max; /* memory.max, bytes (valid unless max_is_max) */
+ __u64 full_sum;
+ __u32 full_fields;
+ bool max_is_max;
+};
+
+struct timing {
+ double avg_us; /* average per full-tree pass */
+ double ro_avg_us; /* BPF read()-only average (no map drain); 0 for file */
+ int nodes_seen; /* entries produced (BPF) */
+ __u32 fields; /* fields/cgroup touched (informational) */
+};
+
+static volatile __u64 sink; /* keep the optimizer from eliding reads */
+static long page_size;
+
+static long long now_ns(void)
+{
+ struct timespec t;
+
+ clock_gettime(CLOCK_MONOTONIC, &t);
+ return (long long)t.tv_sec * 1000000000LL + t.tv_nsec;
+}
+
+/* ---- tree construction ------------------------------------------------- */
+
+static struct cg_node *nodes;
+static int n_nodes;
+static int n_leaves;
+
+static int add_node(const char *rel, bool is_leaf, int *keep_fd)
+{
+ int fd;
+
+ fd = create_and_get_cgroup(rel);
+ if (fd < 0)
+ return -1;
+ if (keep_fd)
+ *keep_fd = fd;
+ else
+ close(fd);
+
+ strncpy(nodes[n_nodes].rel, rel, sizeof(nodes[n_nodes].rel) - 1);
+ nodes[n_nodes].rel[sizeof(nodes[n_nodes].rel) - 1] = '\0';
+ nodes[n_nodes].id = get_cgroup_id(rel);
+ nodes[n_nodes].is_leaf = is_leaf;
+ if (is_leaf)
+ n_leaves++;
+ n_nodes++;
+ return 0;
+}
+
+/* Recursively create children of @rel. @rel must already exist and be recorded. */
+static int build_children(const char *rel, int fanout, int depth)
+{
+ /* size 128 should be enough for file path with max depth 3 is the test*/
+ char child[128];
+ int i;
+
+ if (depth == 0)
+ return 0;
+
+ /* Enable memory on this interior node so its children get memory. */
+ if (enable_controllers(rel, "memory"))
+ return -1;
+
+ for (i = 0; i < fanout; i++) {
+ snprintf(child, sizeof(child), "%s/c%d", rel, i);
+ if (add_node(child, depth == 1, NULL))
+ return -1;
+ if (build_children(child, fanout, depth - 1))
+ return -1;
+ }
+ return 0;
+}
+
+static size_t tree_capacity(int fanout, int depth)
+{
+ size_t total = 1, level = 1;
+ int d;
+
+ for (d = 0; d < depth; d++) {
+ level *= fanout;
+ total += level;
+ }
+ return total;
+}
+
+static int build_tree(int fanout, int depth, int *root_fd)
+{
+ n_nodes = 0;
+ n_leaves = 0;
+ nodes = calloc(tree_capacity(fanout, depth), sizeof(*nodes));
+ if (!nodes)
+ return -1;
+
+ /* special handle for the leaf (0 depth) */
+ if (add_node(SUBTREE_ROOT, depth == 0, root_fd))
+ return -1;
+ return build_children(SUBTREE_ROOT, fanout, depth);
+}
+
+/* ---- charging ---------------------------------------------------------- */
+
+/*
+ * A forked child walks the leaves, joining each and faulting in a private anon
+ * region so the charge lands on that leaf, then keeps every region mapped and
+ * blocks. Interior nodes accumulate the charge hierarchically. The child is
+ * left stopped (blocked on the control pipe) so the stats are static while the
+ * parent measures.
+ */
+static pid_t charger_pid = -1;
+static int charger_ctrl[2] = { -1, -1 };
+
+static int start_charger(size_t charge_bytes, int charge_fraction)
+{
+ int ready[2];
+ pid_t pid;
+ int i, mod;
+ char c;
+
+ if (!ASSERT_OK(pipe(ready), "pipe ready"))
+ return -1;
+ if (!ASSERT_OK(pipe(charger_ctrl), "pipe ctrl")) {
+ close(ready[0]);
+ close(ready[1]);
+ return -1;
+ }
+
+ pid = fork();
+ if (pid < 0) {
+ ASSERT_GE(pid, 0, "fork charger");
+ close(ready[0]);
+ close(ready[1]);
+ close(charger_ctrl[0]);
+ close(charger_ctrl[1]);
+ charger_ctrl[0] = charger_ctrl[1] = -1;
+ return -1;
+ }
+
+ if (pid == 0) {
+ /* child (assert only in the parent so it isn't printed twice) */
+ int leaf_idx = 0;
+
+ close(ready[0]);
+ close(charger_ctrl[1]);
+
+ mod = charge_fraction > 0 ? charge_fraction : 1;
+ for (i = 0; i < n_nodes; i++) {
+ void *p;
+
+ if (!nodes[i].is_leaf)
+ continue;
+ if ((leaf_idx++ % mod) != 0)
+ continue;
+ /*
+ * cgroup_helpers builds paths from getpid(); in this
+ * forked child that differs from the parent that built
+ * the tree, so use the _parent (getppid()) variant to
+ * resolve the leaf under the parent's work dir.
+ */
+ if (join_parent_cgroup(nodes[i].rel))
+ _exit(1);
+ p = mmap(NULL, charge_bytes, PROT_READ | PROT_WRITE,
+ MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
+ if (p == MAP_FAILED)
+ _exit(2);
+ memset(p, 1, charge_bytes);
+ /* keep p mapped so the charge persists */
+ }
+ /* signal ready, then block until the parent closes the pipe */
+ if (write(ready[1], "x", 1) != 1)
+ _exit(3);
+ while (read(charger_ctrl[0], &c, 1) > 0)
+ ;
+ _exit(0);
+ }
+
+ /* parent */
+ charger_pid = pid;
+ close(ready[1]);
+ close(charger_ctrl[0]);
+ charger_ctrl[0] = -1;
+
+ /* wait until the child has charged every leaf */
+ if (!ASSERT_EQ(read(ready[0], &c, 1), 1, "charger ready")) {
+ close(ready[0]);
+ return -1;
+ }
+ close(ready[0]);
+ return 0;
+}
+
+static void stop_charger(void)
+{
+ int status;
+
+ if (charger_ctrl[1] >= 0) {
+ close(charger_ctrl[1]); /* unblock the child -> it exits */
+ charger_ctrl[1] = -1;
+ }
+ if (charger_pid > 0) {
+ if (waitpid(charger_pid, &status, 0) == charger_pid &&
+ (!WIFEXITED(status) || WEXITSTATUS(status) != 0))
+ fprintf(stderr,
+ "charger child exited abnormally (status=0x%x)\n",
+ status);
+ charger_pid = -1;
+ }
+}
+
+/* ---- file (traditional) reader ----------------------------------------- */
+
+static void parse_stat(char *buf, struct file_snap *o)
+{
+ char *save, *line;
+
+ for (line = strtok_r(buf, "\n", &save); line;
+ line = strtok_r(NULL, "\n", &save)) {
+ unsigned long long val;
+ char name[64];
+
+ if (sscanf(line, "%63s %llu", name, &val) != 2)
+ continue;
+ o->full_sum += val;
+ o->full_fields++;
+ if (!strcmp(name, "anon"))
+ o->anon = val;
+ else if (!strcmp(name, "file"))
+ o->file = val;
+ else if (!strcmp(name, "shmem"))
+ o->shmem = val;
+ else if (!strcmp(name, "file_mapped"))
+ o->file_mapped = val;
+ else if (!strcmp(name, "pgfault"))
+ o->pgfault = val;
+ }
+}
+
+static int file_read_node(const char *rel, struct file_snap *o)
+{
+ char buf[8192];
+
+ memset(o, 0, sizeof(*o));
+
+ if (read_cgroup_file(rel, "memory.stat", buf, sizeof(buf)))
+ return -1;
+ parse_stat(buf, o);
+
+ if (!read_cgroup_file(rel, "memory.current", buf, sizeof(buf)))
+ o->current = strtoull(buf, NULL, 10);
+ if (!read_cgroup_file(rel, "memory.max", buf, sizeof(buf))) {
+ if (!strncmp(buf, "max", 3))
+ o->max_is_max = true;
+ else
+ o->max = strtoull(buf, NULL, 10);
+ }
+ return 0;
+}
+
+static void time_file(int iters, struct timing *res)
+{
+ long long total = 0;
+ struct file_snap s;
+ int it, i;
+
+ for (it = 0; it < WARMUP_ITERS; it++)
+ for (i = 0; i < n_nodes; i++)
+ file_read_node(nodes[i].rel, &s);
+
+ for (it = 0; it < iters; it++) {
+ long long t0 = now_ns();
+
+ for (i = 0; i < n_nodes; i++) {
+ file_read_node(nodes[i].rel, &s);
+ sink += s.anon + s.full_sum;
+ }
+ total += now_ns() - t0;
+ }
+ res->avg_us = (double)total / iters / 1000.0;
+ res->fields = s.full_fields;
+}
+
+/* ---- BPF reader -------------------------------------------------------- */
+
+static int bpf_walk_once(struct bpf_link *link)
+{
+ char buf[4096];
+ ssize_t r;
+ int fd;
+
+ fd = bpf_iter_create(bpf_link__fd(link));
+ if (fd < 0)
+ return -1;
+ while ((r = read(fd, buf, sizeof(buf))) > 0)
+ ;
+ close(fd);
+ return r == 0 ? 0 : -1;
+}
+
+static int drain_map(int mfd, struct memcg_stat_snapshot *out, int max)
+{
+ __u64 key = 0, next;
+ int n = 0, err;
+
+ err = bpf_map_get_next_key(mfd, NULL, &next);
+ while (err == 0) {
+ if (n < max && !bpf_map_lookup_elem(mfd, &next, &out[n])) {
+ sink += out[n].anon + out[n].full_sum;
+ n++;
+ }
+ key = next;
+ err = bpf_map_get_next_key(mfd, &key, &next);
+ }
+ return n;
+}
+
+static void time_bpf(struct bpf_link *link, struct memcg_stat_reader *skel,
+ int iters, struct timing *res)
+{
+ struct memcg_stat_snapshot *tmp;
+ long long total = 0, ro_total = 0;
+ int mfd = bpf_map__fd(skel->maps.results);
+ int it, got = 0;
+
+ tmp = calloc(n_nodes + 8, sizeof(*tmp));
+ if (!ASSERT_OK_PTR(tmp, "calloc tmp"))
+ return;
+
+ skel->bss->collect_full = 1;
+
+ for (it = 0; it < WARMUP_ITERS; it++) {
+ bpf_walk_once(link);
+ drain_map(mfd, tmp, n_nodes + 8);
+ }
+
+ for (it = 0; it < iters; it++) {
+ long long t0, t1, t2;
+
+ t0 = now_ns();
+ bpf_walk_once(link);
+ t1 = now_ns();
+ got = drain_map(mfd, tmp, n_nodes + 8);
+ t2 = now_ns();
+
+ total += t2 - t0;
+ ro_total += t1 - t0;
+ }
+
+ res->avg_us = (double)total / iters / 1000.0;
+ res->ro_avg_us = (double)ro_total / iters / 1000.0;
+ res->nodes_seen = got;
+ res->fields = tmp[0].full_fields;
+ free(tmp);
+}
+
+/* ---- correctness ------------------------------------------------------- */
+
+static void check_correctness(struct bpf_link *link,
+ struct memcg_stat_reader *skel)
+{
+ int mfd = bpf_map__fd(skel->maps.results);
+ __u64 total_anon = 0, worst_cur_drift = 0;
+ __u64 anon_tol = 4 * page_size;
+ int i, anon_mism = 0, missing = 0;
+
+ skel->bss->collect_full = 0;
+ if (!ASSERT_OK(bpf_walk_once(link), "bpf walk"))
+ return;
+
+ for (i = 0; i < n_nodes; i++) {
+ struct memcg_stat_snapshot b;
+ __u64 cur, drift;
+ struct file_snap f;
+
+ if (bpf_map_lookup_elem(mfd, &nodes[i].id, &b)) {
+ missing++;
+ continue;
+ }
+ if (file_read_node(nodes[i].rel, &f)) {
+ missing++;
+ continue;
+ }
+ total_anon += b.anon;
+
+ /*
+ * anon (NR_ANON_MAPPED) is rstat-flushed and, with the charger
+ * stopped, deterministic: BPF and memory.stat must agree. The
+ * tolerance is far tighter than a units error (bytes vs pages
+ * differ by PAGE_SIZE), so a wrong-unit/wrong-field bug trips it.
+ */
+ if ((b.anon > f.anon ? b.anon - f.anon : f.anon - b.anon) > anon_tol) {
+ anon_mism++;
+ if (anon_mism <= 5)
+ fprintf(stderr,
+ "anon mismatch %s: bpf=%llu file=%llu\n",
+ nodes[i].rel, b.anon, f.anon);
+ }
+
+ /*
+ * memory.current is the LIVE page_counter. Both sides read the
+ * same counter, but the BPF values are captured in one fast walk
+ * while the files are read across the whole (much longer) loop,
+ * so any difference is time skew on a moving counter, not a BPF
+ * bug -- track it as informational only.
+ */
+ cur = b.usage_pages * page_size;
+ drift = cur > f.current ? cur - f.current : f.current - cur;
+ if (drift > worst_cur_drift)
+ worst_cur_drift = drift;
+ }
+
+ ASSERT_EQ(missing, 0, "all cgroups present in map");
+ ASSERT_EQ(anon_mism, 0, "bpf vs file anon (rstat-flushed)");
+ ASSERT_GT(total_anon, 0, "tree charged some anon");
+ printf("max memory.current drift bpf-vs-file: %llu bytes (live counter, read across the walk window)\n",
+ worst_cur_drift);
+}
+
+/* ---- one case ---------------------------------------------------------- */
+
+struct testcase {
+ const char *name;
+ int fanout;
+ int depth;
+ size_t charge_bytes;
+ int charge_fraction; /* charge every Nth leaf; 1 = all */
+ int iters;
+};
+
+static void run_case(const struct testcase *tc)
+{
+ struct timing f = {}, b = {};
+ struct memcg_stat_reader *skel = NULL;
+ struct bpf_link *link = NULL;
+ int root_fd = -1;
+ int charged;
+
+ if (!ASSERT_OK(build_tree(tc->fanout, tc->depth, &root_fd), "build tree"))
+ goto out;
+
+ if (start_charger(tc->charge_bytes, tc->charge_fraction))
+ goto out;
+
+ skel = memcg_stat_reader__open();
+ if (!ASSERT_OK_PTR(skel, "skel open"))
+ goto out;
+ if (!ASSERT_OK(bpf_map__set_max_entries(skel->maps.results, n_nodes + 8),
+ "set max_entries"))
+ goto out;
+ if (!ASSERT_OK(memcg_stat_reader__load(skel), "skel load"))
+ goto out;
+
+ DECLARE_LIBBPF_OPTS(bpf_iter_attach_opts, opts);
+ union bpf_iter_link_info linfo = {};
+
+ linfo.cgroup.cgroup_fd = root_fd;
+ linfo.cgroup.order = BPF_CGROUP_ITER_DESCENDANTS_PRE;
+ opts.link_info = &linfo;
+ opts.link_info_len = sizeof(linfo);
+
+ link = bpf_program__attach_iter(skel->progs.cgroup_memcg_stat_reader,
+ &opts);
+ if (!ASSERT_OK_PTR(link, "attach iter"))
+ goto out;
+
+ check_correctness(link, skel);
+
+ time_file(tc->iters, &f);
+ time_bpf(link, skel, tc->iters, &b);
+
+ charged = tc->charge_fraction > 0 ?
+ (n_leaves + tc->charge_fraction - 1) / tc->charge_fraction :
+ n_leaves;
+
+ /*
+ * Informational timing diagnostic: captured like any test output, so it
+ * is shown under -v or on failure but not on a normal PASS. The pass/fail
+ * verdict is decided solely by the correctness checks, never by these
+ * numbers.
+ */
+ printf("\n==== memcg_stat_reader: %s ====\n", tc->name);
+ printf("tree: nodes=%d leaves=%d charged_leaves=%d fanout=%d depth=%d charge=%zuKB/leaf iters=%d\n",
+ n_nodes, n_leaves, charged, tc->fanout, tc->depth,
+ tc->charge_bytes >> 10, tc->iters);
+ printf("all times in us (average per full-tree pass, full memory.stat field set); ro = bpf read()-only (no map drain)\n");
+ printf("file_avg=%.1f bpf_avg=%.1f bpf_ro=%.1f speedup(file/bpf)=%.2fx\n",
+ f.avg_us, b.avg_us, b.ro_avg_us,
+ b.avg_us > 0 ? f.avg_us / b.avg_us : 0.0);
+ printf("per-cgroup: file avg=%.0f ns bpf avg=%.0f ns\n",
+ f.avg_us * 1000.0 / n_nodes, b.avg_us * 1000.0 / n_nodes);
+ printf("fields/cgroup: bpf=%u | file stat lines=%u\n", b.fields, f.fields);
+ printf("bpf entries produced: %d (expected %d)\n", b.nodes_seen, n_nodes);
+
+ ASSERT_EQ(b.nodes_seen, n_nodes, "bpf visited whole subtree");
+
+out:
+ bpf_link__destroy(link);
+ memcg_stat_reader__destroy(skel);
+ if (root_fd >= 0)
+ close(root_fd);
+ stop_charger(); /* reap charger so leaves become empty */
+
+ /*
+ * Remove the subtree in reverse creation order. Nodes are recorded in
+ * DFS pre-order (a parent precedes all its descendants), so iterating
+ * backwards removes every child before its parent.
+ */
+ if (nodes) {
+ int i;
+
+ for (i = n_nodes - 1; i >= 0; i--)
+ remove_cgroup(nodes[i].rel);
+ free(nodes);
+ nodes = NULL;
+ }
+}
+
+static const struct testcase cases[] = {
+ { "small", 4, 2, 256 << 10, 1, 200 },
+ { "medium", 10, 2, 256 << 10, 1, 50 },
+ { "large", 10, 3, 256 << 10, 1, 10 },
+ { "large_sparse", 10, 3, 256 << 10, 8, 10 },
+};
+
+/*
+ * The memcg kfuncs the BPF program relies on (bpf_get_mem_cgroup et al.) are
+ * built only with CONFIG_MEMCG (mm/bpf_memcontrol.c). On a kernel without it
+ * they are absent from vmlinux BTF and the program fails to load, so probe for
+ * one of them and skip cleanly rather than reporting a spurious failure.
+ */
+static bool memcg_kfuncs_available(void)
+{
+ struct btf *btf;
+ bool ok;
+
+ btf = btf__load_vmlinux_btf();
+ if (!btf)
+ return false;
+ ok = btf__find_by_name_kind(btf, "bpf_get_mem_cgroup", BTF_KIND_FUNC) > 0;
+ btf__free(btf);
+ return ok;
+}
+
+void test_memcg_stat_reader(void)
+{
+ int i;
+
+ if (!memcg_kfuncs_available()) {
+ test__skip();
+ return;
+ }
+
+ page_size = sysconf(_SC_PAGESIZE);
+
+ if (!ASSERT_OK(setup_cgroup_environment(), "setup cgroup env"))
+ return;
+
+ for (i = 0; i < ARRAY_SIZE(cases); i++) {
+ if (!test__start_subtest(cases[i].name))
+ continue;
+ run_case(&cases[i]);
+ }
+
+ cleanup_cgroup_environment();
+}
diff --git a/tools/testing/selftests/bpf/progs/memcg_stat_reader.c b/tools/testing/selftests/bpf/progs/memcg_stat_reader.c
new file mode 100644
index 00000000000000..a2c1b1b483646e
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/memcg_stat_reader.c
@@ -0,0 +1,181 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2025 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_core_read.h>
+#include "memcg_stat_reader.h"
+
+char _license[] SEC("license") = "GPL";
+
+/*
+ * Flipped by userspace between timed runs (a plain .bss global, writable at
+ * runtime through the skeleton mmap):
+ * 0 - collect only the matched subset (a handful of kfunc calls)
+ * 1 - additionally fold in the full memory.stat field set (many kfunc calls)
+ */
+int collect_full;
+
+/*
+ * Per-cgroup results, keyed by cgroup id. The BPF-side id (cgrp->kn->id)
+ * equals the userspace get_cgroup_id() value, so the test can correlate map
+ * entries back to the cgroups it created. max_entries is resized by userspace
+ * (bpf_map__set_max_entries) to the size of the subtree before load.
+ */
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __uint(max_entries, 1);
+ __type(key, __u64);
+ __type(value, struct memcg_stat_snapshot);
+} results SEC(".maps");
+
+/*
+ * Accumulate one page-state / vm-event read. Each enumerator is guarded by
+ * bpf_core_enum_value_exists(): the full field set below spans counters that are
+ * config- or version-gated (e.g. NR_SECONDARY_PAGETABLE, PGDEMOTE_KHUGEPAGED,
+ * MEMCG_PERCPU_B), so on a kernel whose BTF lacks one, the bpf_core_enum_value()
+ * relocation would otherwise poison the instruction and fail the *entire*
+ * program load. With the _exists guard the missing enumerator relocates to a
+ * compile-time-false branch that the verifier drops as dead code, so the fold is
+ * simply skipped and the rest of the program (including the matched/correctness
+ * path) still loads.
+ */
+#define FOLD_PS(ENUM, NAME) do { \
+ if (bpf_core_enum_value_exists(enum ENUM, NAME)) { \
+ __u64 __v = bpf_mem_cgroup_page_state(memcg, \
+ bpf_core_enum_value(enum ENUM, NAME)); \
+ if (__v != (__u64)-1) { \
+ sum += __v; \
+ nr++; \
+ } \
+ } \
+} while (0)
+
+#define FOLD_EV(NAME) do { \
+ if (bpf_core_enum_value_exists(enum vm_event_item, NAME)) { \
+ __u64 __v = bpf_mem_cgroup_vm_events(memcg, \
+ bpf_core_enum_value(enum vm_event_item, NAME)); \
+ if (__v != (__u64)-1) { \
+ sum += __v; \
+ nr++; \
+ } \
+ } \
+} while (0)
+
+/*
+ * Read a broad memory.stat field set so the timed "full" run pays the realistic
+ * per-field kfunc cost. Enumerators absent from the running kernel's BTF are
+ * skipped (see the _exists guard in FOLD_PS/FOLD_EV), so this stays loadable
+ * across kernel configs/versions. __always_inline so the acquired memcg
+ * reference stays in the main frame (no cross-subprog reference tracking); the
+ * runtime collect_full branch keeps it off the matched path.
+ */
+static __always_inline void collect_full_stats(struct mem_cgroup *memcg,
+ struct memcg_stat_snapshot *snap)
+{
+ __u64 sum = 0;
+ __u32 nr = 0;
+
+ /* node_stat_item: size + event counters that memory.stat prints */
+ FOLD_PS(node_stat_item, NR_ANON_MAPPED);
+ FOLD_PS(node_stat_item, NR_FILE_PAGES);
+ FOLD_PS(node_stat_item, NR_FILE_MAPPED);
+ FOLD_PS(node_stat_item, NR_FILE_DIRTY);
+ FOLD_PS(node_stat_item, NR_WRITEBACK);
+ FOLD_PS(node_stat_item, NR_SHMEM);
+ FOLD_PS(node_stat_item, NR_INACTIVE_ANON);
+ FOLD_PS(node_stat_item, NR_ACTIVE_ANON);
+ FOLD_PS(node_stat_item, NR_INACTIVE_FILE);
+ FOLD_PS(node_stat_item, NR_ACTIVE_FILE);
+ FOLD_PS(node_stat_item, NR_UNEVICTABLE);
+ FOLD_PS(node_stat_item, NR_SLAB_RECLAIMABLE_B);
+ FOLD_PS(node_stat_item, NR_SLAB_UNRECLAIMABLE_B);
+ FOLD_PS(node_stat_item, NR_KERNEL_STACK_KB);
+ FOLD_PS(node_stat_item, NR_PAGETABLE);
+ FOLD_PS(node_stat_item, NR_SECONDARY_PAGETABLE);
+ FOLD_PS(node_stat_item, NR_VMALLOC);
+ FOLD_PS(node_stat_item, WORKINGSET_REFAULT_ANON);
+ FOLD_PS(node_stat_item, WORKINGSET_REFAULT_FILE);
+ FOLD_PS(node_stat_item, WORKINGSET_ACTIVATE_ANON);
+ FOLD_PS(node_stat_item, WORKINGSET_ACTIVATE_FILE);
+ FOLD_PS(node_stat_item, WORKINGSET_RESTORE_ANON);
+ FOLD_PS(node_stat_item, WORKINGSET_RESTORE_FILE);
+ FOLD_PS(node_stat_item, WORKINGSET_NODERECLAIM);
+ FOLD_PS(node_stat_item, PGDEMOTE_KSWAPD);
+ FOLD_PS(node_stat_item, PGDEMOTE_DIRECT);
+ FOLD_PS(node_stat_item, PGDEMOTE_KHUGEPAGED);
+ FOLD_PS(node_stat_item, PGSTEAL_KSWAPD);
+ FOLD_PS(node_stat_item, PGSTEAL_DIRECT);
+ FOLD_PS(node_stat_item, PGSTEAL_KHUGEPAGED);
+ FOLD_PS(node_stat_item, PGSCAN_KSWAPD);
+ FOLD_PS(node_stat_item, PGSCAN_DIRECT);
+ FOLD_PS(node_stat_item, PGSCAN_KHUGEPAGED);
+ FOLD_PS(node_stat_item, PGREFILL);
+
+ /* memcg_stat_item: numbered past NR_VM_NODE_STAT_ITEMS */
+ FOLD_PS(memcg_stat_item, MEMCG_KMEM);
+ FOLD_PS(memcg_stat_item, MEMCG_SOCK);
+ FOLD_PS(memcg_stat_item, MEMCG_PERCPU_B);
+
+ /* vm_event_item: the raw-count tail of memory.stat */
+ FOLD_EV(PGFAULT);
+ FOLD_EV(PGMAJFAULT);
+ FOLD_EV(PGACTIVATE);
+ FOLD_EV(PGDEACTIVATE);
+ FOLD_EV(PGLAZYFREE);
+ FOLD_EV(PGLAZYFREED);
+
+ snap->full_sum = sum;
+ snap->full_fields = nr;
+}
+
+SEC("iter.s/cgroup")
+int cgroup_memcg_stat_reader(struct bpf_iter__cgroup *ctx)
+{
+ struct cgroup *cgrp = ctx->cgroup;
+ struct memcg_stat_snapshot snap = {};
+ struct cgroup_subsys_state *css;
+ struct mem_cgroup *memcg;
+ __u64 cg_id;
+
+ /*
+ * DESCENDANTS_PRE ends with a terminal element where cgroup == NULL.
+ * Return 0 (not 1) so the walk runs to completion.
+ */
+ if (!cgrp)
+ return 0;
+
+ css = &cgrp->self;
+ memcg = bpf_get_mem_cgroup(css);
+ if (!memcg)
+ return 0;
+
+ /* Bring this memcg's rstat up to date before reading it. */
+ bpf_mem_cgroup_flush_stats(memcg);
+
+ cg_id = BPF_CORE_READ(cgrp, kn, id);
+ snap.cgroup_id = cg_id;
+
+ /* Matched subset: always collected so correctness holds in both modes. */
+ snap.anon = bpf_mem_cgroup_page_state(memcg,
+ bpf_core_enum_value(enum node_stat_item, NR_ANON_MAPPED));
+ snap.file = bpf_mem_cgroup_page_state(memcg,
+ bpf_core_enum_value(enum node_stat_item, NR_FILE_PAGES));
+ snap.shmem = bpf_mem_cgroup_page_state(memcg,
+ bpf_core_enum_value(enum node_stat_item, NR_SHMEM));
+ snap.file_mapped = bpf_mem_cgroup_page_state(memcg,
+ bpf_core_enum_value(enum node_stat_item, NR_FILE_MAPPED));
+ snap.pgfault = bpf_mem_cgroup_vm_events(memcg,
+ bpf_core_enum_value(enum vm_event_item, PGFAULT));
+
+ /* page_counter fields need no kfunc; read them off the trusted ptr. */
+ snap.usage_pages = BPF_CORE_READ(memcg, memory.usage.counter);
+ snap.max_pages = BPF_CORE_READ(memcg, memory.max);
+
+ if (collect_full)
+ collect_full_stats(memcg, &snap);
+
+ bpf_map_update_elem(&results, &cg_id, &snap, BPF_ANY);
+
+ bpf_put_mem_cgroup(memcg);
+ return 0;
+}
|
| 3/2 | 2026/07/04 05:24 | llm | patch-evaluator |
0mModel:gemini-3.1-pro-preview Args: null Results: null Tokens: input: 33354 output: 71 thoughts: 659 Thoughts: **WorthFuzzing=false: BPF Selftest Addition** Okay, so I'm looking at this patch and immediately, I see it's all contained within `tools/testing/selftests/bpf/`. We're talking about new BPF selftests here: specifically for `memcg_stat_reader`, `memcg_stat_churn`, and `memcg_stat_churn_percpu`. The initial glance confirms it's all within that directory, but I always triple-check. Alright, let's look at the diff. I see: `cgroup_helpers.c`, `cgroup_helpers.h`, `config`, `memcg_stat_reader.h`, `prog_tests/memcg_stat_churn.c`, `prog_tests/memcg_stat_churn_percpu.c`, `prog_tests/memcg_stat_reader.c`, and `progs/memcg_stat_reader.c`. All within the `bpf/` subdirectory. Good. So far, so good. No core kernel code, no drivers, nothing that would even *potentially* be fuzzed, or that I could see a fuzz target being created. Now, I remember the rules. "Return WorthFuzzing=false if the patch only contains... modifications to Documentation/, Kconfig files, or code comments... Purely decorative changes... Changes to numeric constants or macros that do not functionally alter execution flow... Code paths that are impossible to reach in virtualized environments..." Wait a minute... selftests. Where do selftests fit into that? Selftests aren't core kernel logic or drivers, right? They're userspace programs. They are explicitly *not* core kernel logic, drivers, or architectures. The selftests are not being modified to target a part of the kernel which is reachable and fuzzed. So, this boils down to: is it modifying any reachable core kernel logic, drivers, or architectures? Nope. This is just adding tests. Let's verify again, just to be absolutely sure. The diff in the prompt is the whole thing, the entire commit. I can confirm this using `default_api:grepper` if I really want to be sure. It's only `tools/testing/selftests/bpf/*`. No surprises there. Therefore, this is a clear case. This patch *only* adds/modifies selftests. No functional changes to code that *would* be fuzzed. So, I'm returning `WorthFuzzing=false`. The reasoning is straightforward: no changes to the kernel's fuzzable attack surface. |
| 4/2 | 2026/07/04 05:24 | tool | set-results |
0mArgs:{
"EnableConfigs": [],
"FocusSymbols": [],
"Reasoning": "The patch only adds and modifies BPF selftests in tools/testing/selftests/bpf/. It does not make any functional changes to the core kernel, drivers, or architectures.",
"WorthFuzzing": false
}Results: {
"EnableConfigs": null,
"FocusSymbols": [],
"Reasoning": "The patch only adds and modifies BPF selftests in tools/testing/selftests/bpf/. It does not make any functional changes to the core kernel, drivers, or architectures.",
"WorthFuzzing": false
} |
| Total Calls | Total Tokens | Avg Tokens | Total Duration (Seconds) | Avg Duration (Seconds) |
|---|
| Total Calls | Total Duration (Seconds) | Avg Duration (Seconds) |
|---|