| Seq | Timestamp | Type | Name | Duration |
|---|---|---|---|---|
| 0/0 | 2026/11/03 12:05 | flow | patch-triage |
33mArgs:null Results: {
"EnableConfigs": [
"SMP"
],
"FocusSymbols": [
"percpu_counter_tree_activate_many",
"percpu_counter_tree_add",
"percpu_counter_tree_precise_sum",
"percpu_counter_tree_destroy_many",
"percpu_counter_tree_precise_compare",
"percpu_counter_tree_approximate_compare",
"percpu_counter_tree_set"
],
"KMSANReasoning": "The patch series introduces Hierarchical Per-CPU Counters (HPCC, implemented via `struct percpu_counter_tree`) and replaces `struct percpu_counter` with HPCC for memory management RSS statistics (`mm-\u003erss_stat`).\n\nKMSAN vs KASAN Applicability Analysis:\n1. Memory Allocation and Initialization:\n - Dynamic allocations for tree items use `kzalloc()` or `kmem_cache_zalloc()`, ensuring full zero-initialization of tree buffers.\n - Per-CPU allocations (`__alloc_percpu_gfp()`) in Linux are zeroed upon allocation.\n - Struct fields for `percpu_counter_tree` are systematically initialized in `__percpu_counter_tree_init()` (and lazy initialization explicitly sets pointers to NULL and biases to 0).\n - `struct mm_struct` additions (`rss_stat`, `rss_owner`, `rss_stat_mutex`) are initialized in `mm_init()` via `memset` and mutex initialization.\n - No complex data structures are left uninitialized or partially initialized.\n\n2. User-space Boundary and Info-leaks:\n - No raw kernel structures or buffers with potential padding bytes are copied to user space (`copy_to_user`, ioctl, netlink, BPF, etc.).\n - The RSS values derived from the tree counters are plain numerical values formatted into strings (e.g., in `/proc/\u003cpid\u003e/stat` via `seq_put_decimal_ull()`) or logged in trace events.\n\n3. Bug Profiles:\n - The primary risks of this patch series involve concurrency, memory ordering (store-release/load-acquire barriers between levels and during activation), synchronization (`rss_stat_mutex`), arithmetic carry logic, or potential out-of-bounds indexing in tree item strides.\n - Out-of-bounds indexing and use-after-free bugs are detected by KASAN. Locking deadlocks are caught by LOCKDEP. KMSAN is specifically designed to detect reads of uninitialized memory and information leaks, neither of which are risks introduced by this patch.\n\nTherefore, a dedicated KMSAN fuzzing session is not warranted.",
"NeedsKMSAN": false,
"Reasoning": "The patch series implements Hierarchical Per-CPU Counters (HPCC) and transitions the kernel's mm RSS statistics (mm-\u003erss_stat) to use this new data structure. This modifies core subsystems (mm, fork, exec, exit) and introduces lockless tree propagation, atomic carry cmpxchg loops, and lazy activation when processes create threads with CLONE_VM. The modified code is directly reachable by user space via common syscalls (clone, fork, exit, execve, mmap, munmap, page faults, and /proc/\u003cpid\u003e/statm reading) and is an excellent target for fuzzing race conditions and concurrency issues. Ubiquitous single-threaded hot-paths (such as percpu_counter_tree_add_owner and percpu_counter_tree_init_many_lazy) are excluded from FocusSymbols to prevent coverage dilution.",
"WorthFuzzing": true
} |
| 1/1 | 2026/11/03 12:05 | action | read-patch-diff |
0mArgs:null Results: {
"PatchDiff": "commit d5aca8d49d662c298b87359f285e9556d3b2d61a\nAuthor: syz-cluster \u003ctriage@syzkaller.com\u003e\nDate: Tue Nov 3 12:05:07 2026 +0000\n\n syz-cluster: applied patch under review\n\ndiff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt\nindex 0b3c140babd944..ba2bdeb1480b7f 100644\n--- a/Documentation/admin-guide/kernel-parameters.txt\n+++ b/Documentation/admin-guide/kernel-parameters.txt\n@@ -5342,6 +5342,20 @@ Kernel parameters\n \t\t\tallocator. This parameter is primarily\tfor debugging\n \t\t\tand performance comparison.\n \n+\tpercpu_counter_tree.arity=\t[KNL,SMP]\n+\t\t\tFormat: \u003cfan-out\u003e[,\u003cfan-out\u003e...]\n+\t\t\tOverride the shape of the hierarchical per-cpu\n+\t\t\tcounter trees (used for the RSS counters), listing\n+\t\t\tthe fan-out of each tree level from the per-cpu\n+\t\t\tcounters to the root. Each fan-out is a power of 2\n+\t\t\tand their product must be nr_cpu_ids (the highest\n+\t\t\tpossible CPU number plus one) rounded up to a power\n+\t\t\tof 2, e.g. \"16\" (flat) or \"4,4\" when nr_cpu_ids\n+\t\t\tis 16. An invalid shape is ignored with a warning.\n+\t\t\tThe shape in use is shown in\n+\t\t\t/sys/module/percpu_counter_tree/parameters/arity.\n+\t\t\tThis parameter is meant for performance comparison.\n+\n \tpirq=\t\t[SMP,APIC] Manual mp-table setup\n \t\t\tSee Documentation/arch/x86/i386/IO-APIC.rst.\n \n@@ -6778,6 +6792,16 @@ Kernel parameters\n \t\t\tdebug mode. Defaults to CONFIG_RSEQ_DEBUG_DEFAULT_ENABLE.\n \t\t\tFormat: \u003cbool\u003e\n \n+\trss_stat_batch=\t[KNL,SMP]\n+\t\t\tFormat: \u003cinteger\u003e\n+\t\t\tBatch size of the per-mm RSS counters: the number of\n+\t\t\tpages a CPU accumulates before propagating them into\n+\t\t\tthe counter tree. A power of 2 between 2 and 128.\n+\t\t\tA smaller batch improves the accuracy of the\n+\t\t\tapproximate RSS, a larger batch reduces the update\n+\t\t\tcost. Default: 32.\n+\t\t\tThis parameter is meant for performance comparison.\n+\n \trt_group_sched=\t[KNL] Enable or disable SCHED_RR/FIFO group scheduling\n \t\t\twhen CONFIG_RT_GROUP_SCHED=y. Defaults to\n \t\t\t!CONFIG_RT_GROUP_SCHED_DEFAULT_DISABLED.\ndiff --git a/Documentation/core-api/index.rst b/Documentation/core-api/index.rst\nindex 92f91c6a0d79d6..fe03cefc17348a 100644\n--- a/Documentation/core-api/index.rst\n+++ b/Documentation/core-api/index.rst\n@@ -48,6 +48,7 @@ Library functionality that is used throughout the kernel.\n generic-radix-tree\n packing\n this_cpu_ops\n+ percpu-counter-tree\n timekeeping\n errseq\n wrappers/atomic_t\ndiff --git a/Documentation/core-api/percpu-counter-tree.rst b/Documentation/core-api/percpu-counter-tree.rst\nnew file mode 100644\nindex 00000000000000..8aebf91aa133d7\n--- /dev/null\n+++ b/Documentation/core-api/percpu-counter-tree.rst\n@@ -0,0 +1,145 @@\n+.. SPDX-License-Identifier: GPL-2.0+ OR MIT\n+\n+========================================\n+The Hierarchical Per-CPU Counters (HPCC)\n+========================================\n+\n+:Author: Mathieu Desnoyers\n+\n+Introduction\n+============\n+\n+Counters come in many varieties, each with their own trade offs:\n+\n+ * A global atomic counter provides a fast read access to the current\n+ sum, at the expense of cache-line bouncing on updates. This leads to\n+ poor performance of frequent updates from various cores on large SMP\n+ systems.\n+\n+ * A per-cpu split counter provides fast updates to per-cpu counters,\n+ at the expense of a slower aggregation (sum). The sum operation needs\n+ to iterate over all per-cpu counters to calculate the current total.\n+\n+The hierarchical per-cpu counters attempt to provide the best of both\n+worlds (fast updates, and fast sum) by relaxing requirements on the sum\n+accuracy. It allows quickly querying an approximated sum value, along\n+with the possible min/max ranges of the associated precise sum. The\n+exact precise sum can still be calculated with an iteration on all\n+per-cpu counters, but the availability of an approximated sum value with\n+possible precise sum min/max ranges allows eliminating candidates which\n+are certainly outside of a known target range without the overhead of\n+precise sums.\n+\n+Overview\n+========\n+\n+The hierarchical per-cpu counters are organized as a tree with the\n+per-cpu counters as its first level and the tree root as its last\n+level.\n+\n+The intermediate tree levels contain carry propagation counters. When\n+reaching a threshold (batch size), the carry is propagated up the\n+tree.\n+\n+Each counter below the root, from the per-cpu counters up, keeps the\n+residual it has not propagated yet, and only propagates once this\n+residual reaches its batch size in either direction. This hysteresis\n+prevents updates going back and forth around a batch size boundary from\n+propagating further up the tree each time.\n+\n+This allows reading an approximated value at the root, which has a\n+bounded accuracy determined by the tree topology and the batch size:\n+the precise sum is within the same distance of the approximation in\n+both directions. This bound grows as O(nr_cpus * log(nr_cpus)).\n+\n+The per-cpu counters are bytes, which limits the batch size to 128. The\n+tree positions are cache line aligned to prevent false sharing between\n+CPUs updating different branches of the tree. A tree position holds the\n+items of all the counter trees initialized together by\n+percpu_counter_tree_init_many(), so a set of counters updated together,\n+such as the RSS counters of an mm, shares cache lines.\n+percpu_counter_tree_init_many() allocates the per-cpu counters, one per\n+possible CPU, and the tree items, whose number depends on nr_cpu_ids, so\n+struct percpu_counter_tree has a fixed size.\n+\n+The fan-out of each tree level is selected at boot from nr_cpu_ids, the\n+highest possible CPU number plus one. It can be overridden with the\n+percpu_counter_tree.arity= boot parameter for performance comparison.\n+\n+The bounded accuracy is a property of the settled tree. While updates are\n+in flight the transient error can exceed it, bounded by the number of\n+updaters and their increment size.\n+\n+Likewise, the precise sum is exact when no update runs concurrently. It\n+takes no lock: an update which carries from the per-cpu counter to the\n+upper levels can be counted in part, the residual it left below read and\n+its carry missed on the way up.\n+\n+Lazy Tree Activation\n+====================\n+\n+Counters initialized with percpu_counter_tree_init_many_lazy() start\n+without their per-cpu counters and tree items, for counters which may\n+never need the scalability of the tree. Until\n+percpu_counter_tree_activate_many() allocates and publishes the tree,\n+the counters are exact:\n+\n+ * percpu_counter_tree_add_owner() adds to the bias of the counter,\n+ written only by its owner, a single context chosen by the user of the\n+ counters, such as a task, without atomic instruction.\n+\n+ * percpu_counter_tree_add() adds to the atomic bias of the counter, for\n+ any other updater.\n+\n+The precise and approximate sums read both, in O(1), and count each\n+update whole. The activation does not move these values: the sums keep\n+reading them along with the tree, so updates racing with the activation\n+are not lost, whichever side of it they land on. The bias only has the\n+owner as writer, and the atomic bias only receives atomic additions,\n+before and after the activation. After activation, both update functions\n+add to the tree, and the approximation has the accuracy of the tree.\n+\n+The activations of the same counters must be serialized by the caller:\n+a concurrent activation would publish a second tree and lose the\n+updates made to the first.\n+\n+The RSS counters of an mm use this: the task owning the mm updates the\n+plain counter, and other tasks, such as reclaim, update the atomic\n+counter. The tree is activated when a second thread shares the mm,\n+under mm-\u003erss_stat_mutex, which serializes the activations.\n+A single-threaded process therefore has exact RSS counters, cheap\n+precise sums, and neither per-cpu memory nor tree items for them.\n+\n+Use Cases\n+=========\n+\n+Use cases HPCC is meant to handle involve tracking resources which are\n+used across many CPUs to quickly sum as feedback for decision making to\n+apply throttling, quota limits, sort tasks, and perform memory or task\n+migration decisions. When considering approximated sums within the\n+accuracy range of the decision threshold, the user can either:\n+\n+ * Be conservative and fast: Consider that the sum has reached the\n+ limit as soon as the given limit is within the approximation range.\n+\n+ * Be aggressive and fast: Consider that the sum is over the\n+ limit only when the approximation range is over the given limit.\n+\n+ * Be precise and slow: Do a precise comparison with the limit, which\n+ requires a precise sum when the limit is within the approximated\n+ range.\n+\n+One use-case for these hierarchical counters is to implement a two-pass\n+algorithm to speed up picking a maximum/minimum sum value from\n+a set. A first pass compares the approximated values, and then a second\n+pass only needs the precise sum for counter trees which are within the\n+possible precise sum range of the counter tree chosen by the first pass.\n+\n+Because the comparison is on the wrapped difference, it is not transitive:\n+ordering is undefined over a set of counters spanning more than LONG_MAX.\n+\n+Functions and structures\n+========================\n+\n+.. kernel-doc:: include/linux/percpu_counter_tree.h\n+.. kernel-doc:: lib/percpu_counter_tree.c\ndiff --git a/MAINTAINERS b/MAINTAINERS\nindex 93abbb315065de..3878203de1c30f 100644\n--- a/MAINTAINERS\n+++ b/MAINTAINERS\n@@ -11947,6 +11947,18 @@ L:\tlinux-input@vger.kernel.org\n S:\tMaintained\n F:\tdrivers/hid/hid-logitech-hidpp.c\n \n+HIERARCHICAL PER-CPU COUNTERS (HPCC)\n+M:\tMathieu Desnoyers \u003cmathieu.desnoyers@efficios.com\u003e\n+L:\tlinux-mm@kvack.org\n+L:\tlinux-kernel@vger.kernel.org\n+S:\tSupported\n+F:\tDocumentation/core-api/percpu-counter-tree.rst\n+F:\tinclude/linux/percpu_counter_tree.h\n+F:\tlib/percpu_counter_tree.c\n+F:\tlib/tests/percpu_counter_tree_bench.c\n+F:\tlib/tests/percpu_counter_tree_kunit.c\n+F:\ttools/testing/hpcc/\n+\n HIGH-RESOLUTION TIMERS, TIMER WHEEL, CLOCKEVENTS\n M:\tAnna-Maria Behnsen \u003canna-maria@linutronix.de\u003e\n M:\tFrederic Weisbecker \u003cfrederic@kernel.org\u003e\ndiff --git a/fs/exec.c b/fs/exec.c\nindex 8d2b14e1d455cf..7a6f07a6d482fc 100644\n--- a/fs/exec.c\n+++ b/fs/exec.c\n@@ -875,6 +875,8 @@ static int exec_mmap(struct linux_binprm *bprm)\n \t\t}\n \t}\n \n+\t/* This task owns the RSS counters of the new mm, see __add_mm_counter(). */\n+\tWRITE_ONCE(mm-\u003erss_owner, tsk);\n \ttask_lock(tsk);\n \tmembarrier_exec_mmap(mm);\n \n@@ -903,6 +905,9 @@ static int exec_mmap(struct linux_binprm *bprm)\n \tif (old_mm) {\n \t\tmmap_read_unlock(old_mm);\n \t\tBUG_ON(active_mm != old_mm);\n+\t\t/* Updates of the old mm by this task are now atomic. */\n+\t\tif (READ_ONCE(old_mm-\u003erss_owner) == tsk)\n+\t\t\tWRITE_ONCE(old_mm-\u003erss_owner, NULL);\n \t\t/* Defer teardown to setup_new_exec(), outside the exec locks. */\n \t\tbprm-\u003eold_mm = old_mm;\n \t} else {\ndiff --git a/include/linux/mm.h b/include/linux/mm.h\nindex 9f167950926a15..4d740c0c5f5481 100644\n--- a/include/linux/mm.h\n+++ b/include/linux/mm.h\n@@ -3607,33 +3607,51 @@ static inline bool get_user_page_fast_only(unsigned long addr,\n */\n static inline unsigned long get_mm_counter(struct mm_struct *mm, int member)\n {\n-\treturn percpu_counter_read_positive(\u0026mm-\u003erss_stat[member]);\n+\treturn percpu_counter_tree_approximate_sum_positive(\u0026mm-\u003erss_stat[member]);\n }\n \n static inline unsigned long get_mm_counter_sum(struct mm_struct *mm, int member)\n {\n-\treturn percpu_counter_sum_positive(\u0026mm-\u003erss_stat[member]);\n+\treturn percpu_counter_tree_precise_sum_positive(\u0026mm-\u003erss_stat[member]);\n }\n \n void mm_trace_rss_stat(struct mm_struct *mm, int member);\n \n+/*\n+ * Until a second thread shares the mm, the RSS counters are exact single\n+ * counters: the task which owns the mm updates a plain counter, and other\n+ * tasks (reclaim, khugepaged, the OOM reaper, remote GUP, ...) an atomic\n+ * one. See copy_mm(). Once the tree is active, all updates go to it:\n+ * rss_owner is not read, so the updates of the threads of a process do not\n+ * touch its cache line.\n+ */\n+static inline void __add_mm_counter(struct mm_struct *mm, int member, long value)\n+{\n+\tstruct percpu_counter_tree *counter = \u0026mm-\u003erss_stat[member];\n+\n+\tif (!percpu_counter_tree_active(counter) \u0026\u0026 current == READ_ONCE(mm-\u003erss_owner))\n+\t\tpercpu_counter_tree_add_owner(counter, value);\n+\telse\n+\t\tpercpu_counter_tree_add(counter, value);\n+}\n+\n static inline void add_mm_counter(struct mm_struct *mm, int member, long value)\n {\n-\tpercpu_counter_add(\u0026mm-\u003erss_stat[member], value);\n+\t__add_mm_counter(mm, member, value);\n \n \tmm_trace_rss_stat(mm, member);\n }\n \n static inline void inc_mm_counter(struct mm_struct *mm, int member)\n {\n-\tpercpu_counter_inc(\u0026mm-\u003erss_stat[member]);\n+\t__add_mm_counter(mm, member, 1);\n \n \tmm_trace_rss_stat(mm, member);\n }\n \n static inline void dec_mm_counter(struct mm_struct *mm, int member)\n {\n-\tpercpu_counter_dec(\u0026mm-\u003erss_stat[member]);\n+\t__add_mm_counter(mm, member, -1);\n \n \tmm_trace_rss_stat(mm, member);\n }\ndiff --git a/include/linux/mm_types.h b/include/linux/mm_types.h\nindex 1ed35d258d2e36..f0d315d72adb02 100644\n--- a/include/linux/mm_types.h\n+++ b/include/linux/mm_types.h\n@@ -19,6 +19,7 @@\n #include \u003clinux/workqueue.h\u003e\n #include \u003clinux/seqlock.h\u003e\n #include \u003clinux/percpu_counter.h\u003e\n+#include \u003clinux/percpu_counter_tree.h\u003e\n #include \u003clinux/types.h\u003e\n #include \u003clinux/futex_types.h\u003e\n #include \u003clinux/rseq_types.h\u003e\n@@ -1350,7 +1351,23 @@ struct mm_struct {\n \t\tunsigned long saved_e_flags;\n #endif\n \n-\t\tstruct percpu_counter rss_stat[NR_MM_COUNTERS];\n+\t\tstruct percpu_counter_tree rss_stat[NR_MM_COUNTERS];\n+\t\t/*\n+\t\t * Task updating rss_stat with percpu_counter_tree_add_owner()\n+\t\t * until the counter tree is activated by a second thread: the\n+\t\t * task running on this mm, set by dup_mm() and exec_mmap(),\n+\t\t * cleared when that task exits or execs, NULL for the other\n+\t\t * users of mm_alloc(). Only compared with current, never\n+\t\t * dereferenced. Not mm-\u003eowner, which depends on CONFIG_MEMCG\n+\t\t * and which mm_update_next_owner() hands over to another task\n+\t\t * while a vfork() child may still update the counters.\n+\t\t */\n+\t\tstruct task_struct *rss_owner;\n+\t\t/*\n+\t\t * Serializes the activations of the rss_stat counter tree,\n+\t\t * until the tree is active. See mm_activate_rss_stat().\n+\t\t */\n+\t\tstruct mutex rss_stat_mutex;\n \n \t\tstruct linux_binfmt *binfmt;\n \ndiff --git a/include/linux/percpu_counter_tree.h b/include/linux/percpu_counter_tree.h\nnew file mode 100644\nindex 00000000000000..60ef1da22bd3fd\n--- /dev/null\n+++ b/include/linux/percpu_counter_tree.h\n@@ -0,0 +1,365 @@\n+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */\n+/* SPDX-FileCopyrightText: 2025-2026 Mathieu Desnoyers \u003cmathieu.desnoyers@efficios.com\u003e */\n+\n+#ifndef _PERCPU_COUNTER_TREE_H\n+#define _PERCPU_COUNTER_TREE_H\n+\n+#include \u003clinux/bug.h\u003e\n+#include \u003clinux/atomic.h\u003e\n+#include \u003clinux/limits.h\u003e\n+#include \u003clinux/percpu.h\u003e\n+\n+/* Largest batch size: level 0 residuals are within (-batch, +batch) of a signed byte. */\n+#define PERCPU_COUNTER_TREE_MAX_BATCH\t128\n+static_assert(PERCPU_COUNTER_TREE_MAX_BATCH - 1 \u003c= S8_MAX);\n+\n+#ifdef CONFIG_SMP\n+\n+struct percpu_counter_tree {\n+\t/*\n+\t * Per-CPU split counters (tree level 0): residuals not propagated\n+\t * yet, within (-batch, +batch).\n+\t */\n+\ts8 __percpu *level0;\n+\t/*\n+\t * The batch size is the increment step at level 0 which triggers a\n+\t * carry propagation: a power of 2 between 2 and\n+\t * PERCPU_COUNTER_TREE_MAX_BATCH. It is stored next to level0, the\n+\t * other field read by the update fast path, so both usually share a\n+\t * cache line: this leaves a hole before items_stride.\n+\t */\n+\tunsigned char batch_order;\n+\tunsigned int items_stride;\t/* Distance between this counter's consecutive items. */\n+\t/*\n+\t * Approximate sum: this counter's item of the root, in the tree\n+\t * items allocated with the per-CPU counters. NULL until the tree of a\n+\t * counter initialized by percpu_counter_tree_init_many_lazy() is\n+\t * activated.\n+\t */\n+\tatomic_long_t *approx_sum;\n+\t/*\n+\t * Bias to apply to counter precise and approximate values. Before the\n+\t * tree is activated, it also accumulates the updates of\n+\t * percpu_counter_tree_add_owner(), written only by the owner.\n+\t */\n+\tlong bias;\n+\t/*\n+\t * Updates of percpu_counter_tree_add() made before the tree is\n+\t * activated, also applied to the precise and approximate values.\n+\t */\n+\tatomic_long_t atomic_bias;\n+};\n+\n+int percpu_counter_tree_init_many(struct percpu_counter_tree *counters,\n+\t\t\t\t unsigned int nr_counters, unsigned long batch_size,\n+\t\t\t\t gfp_t gfp_flags);\n+int percpu_counter_tree_init(struct percpu_counter_tree *counter,\n+\t\t\t unsigned long batch_size, gfp_t gfp_flags);\n+int percpu_counter_tree_init_many_lazy(struct percpu_counter_tree *counters,\n+\t\t\t\t unsigned int nr_counters, unsigned long batch_size);\n+int percpu_counter_tree_activate_many(struct percpu_counter_tree *counters,\n+\t\t\t\t unsigned int nr_counters, gfp_t gfp_flags);\n+void percpu_counter_tree_destroy_many(struct percpu_counter_tree *counters,\n+\t\t\t\t unsigned int nr_counters);\n+void percpu_counter_tree_destroy(struct percpu_counter_tree *counter);\n+void percpu_counter_tree_add(struct percpu_counter_tree *counter, long inc);\n+void percpu_counter_tree_add_owner(struct percpu_counter_tree *counter, long inc);\n+long percpu_counter_tree_precise_sum(struct percpu_counter_tree *counter);\n+int percpu_counter_tree_approximate_compare(struct percpu_counter_tree *a,\n+\t\t\t\t\t struct percpu_counter_tree *b);\n+int percpu_counter_tree_approximate_compare_value(struct percpu_counter_tree *counter, long v);\n+int percpu_counter_tree_precise_compare(struct percpu_counter_tree *a,\n+\t\t\t\t\tstruct percpu_counter_tree *b);\n+int percpu_counter_tree_precise_compare_value(struct percpu_counter_tree *counter, long v);\n+void percpu_counter_tree_set(struct percpu_counter_tree *counter, long v);\n+int percpu_counter_tree_subsystem_init(void);\n+\n+/**\n+ * percpu_counter_tree_active() - Query whether the tree of a counter is active.\n+ * @counter: The counter to query.\n+ *\n+ * A counter initialized by percpu_counter_tree_init_many() is active. A\n+ * counter initialized by percpu_counter_tree_init_many_lazy() becomes\n+ * active when percpu_counter_tree_activate_many() publishes its tree, and\n+ * stays active. Once active, percpu_counter_tree_add() and\n+ * percpu_counter_tree_add_owner() both add to the tree, so a user can skip\n+ * selecting between them.\n+ *\n+ * Return: true if the updates of @counter go to its tree.\n+ */\n+static inline\n+bool percpu_counter_tree_active(struct percpu_counter_tree *counter)\n+{\n+\t/* No ordering: the update functions and the sums read level0 again. */\n+\treturn !!READ_ONCE(counter-\u003elevel0);\n+}\n+\n+/**\n+ * percpu_counter_tree_approximate_sum() - Return approximate counter sum.\n+ * @counter: The counter to sum.\n+ *\n+ * Querying the approximate sum is fast, but it is only accurate within\n+ * the bound returned by percpu_counter_tree_approximate_accuracy().\n+ * This is meant to be used when speed is preferred over accuracy.\n+ *\n+ * Return: The current approximate counter sum.\n+ */\n+static inline\n+long percpu_counter_tree_approximate_sum(struct percpu_counter_tree *counter)\n+{\n+\t/*\n+\t * Pairs with the release of approx_sum in\n+\t * percpu_counter_tree_activate_many(): the root is zeroed before its\n+\t * pointer, and the address dependency orders the read of the root\n+\t * after it, as for rcu_dereference().\n+\t */\n+\tatomic_long_t *approx_sum = READ_ONCE(counter-\u003eapprox_sum);\n+\tunsigned long v = (unsigned long)READ_ONCE(counter-\u003ebias) +\n+\t\t\t (unsigned long)atomic_long_read(\u0026counter-\u003eatomic_bias);\n+\n+\tif (approx_sum)\n+\t\tv += (unsigned long)atomic_long_read(approx_sum);\n+\treturn (long)v;\n+}\n+\n+unsigned long percpu_counter_tree_approximate_accuracy(struct percpu_counter_tree *counter);\n+\n+#if IS_ENABLED(CONFIG_KUNIT)\n+unsigned int percpu_counter_tree_check_config(void);\n+unsigned int percpu_counter_tree_check_residuals(struct percpu_counter_tree *counter);\n+void percpu_counter_tree_copy_state(struct percpu_counter_tree *dst,\n+\t\t\t\t struct percpu_counter_tree *src);\n+#endif\n+\n+#else\t/* !CONFIG_SMP */\n+\n+struct percpu_counter_tree {\n+\tatomic_long_t count;\n+};\n+\n+static inline\n+int percpu_counter_tree_init_many(struct percpu_counter_tree *counters,\n+\t\t\t\t unsigned int nr_counters, unsigned long batch_size,\n+\t\t\t\t gfp_t gfp_flags)\n+{\n+\t/*\n+\t * There is no tree to batch into on UP, so @batch_size has no\n+\t * effect. Validate it anyway so callers get the same result as\n+\t * on SMP.\n+\t */\n+\tif (WARN_ON(batch_size \u003c= 1 || (batch_size \u0026 (batch_size - 1)) ||\n+\t\t batch_size \u003e PERCPU_COUNTER_TREE_MAX_BATCH))\n+\t\treturn -EINVAL;\n+\tfor (unsigned int i = 0; i \u003c nr_counters; i++)\n+\t\tatomic_long_set(\u0026counters[i].count, 0);\n+\treturn 0;\n+}\n+\n+static inline\n+int percpu_counter_tree_init(struct percpu_counter_tree *counter,\n+\t\t\t unsigned long batch_size, gfp_t gfp_flags)\n+{\n+\treturn percpu_counter_tree_init_many(counter, 1, batch_size, gfp_flags);\n+}\n+\n+static inline\n+int percpu_counter_tree_init_many_lazy(struct percpu_counter_tree *counters,\n+\t\t\t\t unsigned int nr_counters, unsigned long batch_size)\n+{\n+\treturn percpu_counter_tree_init_many(counters, nr_counters, batch_size, 0);\n+}\n+\n+static inline\n+int percpu_counter_tree_activate_many(struct percpu_counter_tree *counters,\n+\t\t\t\t unsigned int nr_counters, gfp_t gfp_flags)\n+{\n+\treturn 0;\n+}\n+\n+static inline\n+bool percpu_counter_tree_active(struct percpu_counter_tree *counter)\n+{\n+\treturn true;\n+}\n+\n+static inline\n+void percpu_counter_tree_destroy_many(struct percpu_counter_tree *counters,\n+\t\t\t\t unsigned int nr_counters)\n+{\n+}\n+\n+static inline\n+void percpu_counter_tree_destroy(struct percpu_counter_tree *counter)\n+{\n+}\n+\n+static inline\n+long percpu_counter_tree_precise_sum(struct percpu_counter_tree *counter)\n+{\n+\treturn atomic_long_read(\u0026counter-\u003ecount);\n+}\n+\n+static inline\n+int percpu_counter_tree_precise_compare(struct percpu_counter_tree *a,\n+\t\t\t\t\tstruct percpu_counter_tree *b)\n+{\n+\t/* Compare on the wrapped difference, as on SMP. */\n+\tlong delta = (long)((unsigned long)percpu_counter_tree_precise_sum(a) -\n+\t\t\t (unsigned long)percpu_counter_tree_precise_sum(b));\n+\n+\tif (delta \u003c 0)\n+\t\treturn -1;\n+\tif (delta \u003e 0)\n+\t\treturn 1;\n+\treturn 0;\n+}\n+\n+static inline\n+int percpu_counter_tree_precise_compare_value(struct percpu_counter_tree *counter, long v)\n+{\n+\t/* Compare on the wrapped difference, as on SMP. */\n+\tlong delta = (long)((unsigned long)v -\n+\t\t\t (unsigned long)percpu_counter_tree_precise_sum(counter));\n+\n+\tif (delta \u003c 0)\n+\t\treturn -1;\n+\tif (delta \u003e 0)\n+\t\treturn 1;\n+\treturn 0;\n+}\n+\n+static inline\n+int percpu_counter_tree_approximate_compare(struct percpu_counter_tree *a,\n+\t\t\t\t\t struct percpu_counter_tree *b)\n+{\n+\treturn percpu_counter_tree_precise_compare(a, b);\n+}\n+\n+static inline\n+int percpu_counter_tree_approximate_compare_value(struct percpu_counter_tree *counter, long v)\n+{\n+\treturn percpu_counter_tree_precise_compare_value(counter, v);\n+}\n+\n+static inline\n+void percpu_counter_tree_set(struct percpu_counter_tree *counter, long v)\n+{\n+\tatomic_long_set(\u0026counter-\u003ecount, v);\n+}\n+\n+static inline\n+unsigned long percpu_counter_tree_approximate_accuracy(struct percpu_counter_tree *counter)\n+{\n+\treturn 0;\n+}\n+\n+static inline\n+void percpu_counter_tree_add(struct percpu_counter_tree *counter, long inc)\n+{\n+\tatomic_long_add(inc, \u0026counter-\u003ecount);\n+}\n+\n+static inline\n+void percpu_counter_tree_add_owner(struct percpu_counter_tree *counter, long inc)\n+{\n+\tpercpu_counter_tree_add(counter, inc);\n+}\n+\n+static inline\n+long percpu_counter_tree_approximate_sum(struct percpu_counter_tree *counter)\n+{\n+\treturn percpu_counter_tree_precise_sum(counter);\n+}\n+\n+static inline\n+int percpu_counter_tree_subsystem_init(void)\n+{\n+\treturn 0;\n+}\n+\n+#endif\t/* CONFIG_SMP */\n+\n+/**\n+ * percpu_counter_tree_approximate_sum_positive() - Return a positive approximate counter sum.\n+ * @counter: The counter to sum.\n+ *\n+ * Return an approximate counter sum which is guaranteed to be greater\n+ * or equal to 0.\n+ *\n+ * Return: The current positive approximate counter sum.\n+ */\n+static inline\n+long percpu_counter_tree_approximate_sum_positive(struct percpu_counter_tree *counter)\n+{\n+\tlong v = percpu_counter_tree_approximate_sum(counter);\n+\n+\treturn v \u003e 0 ? v : 0;\n+}\n+\n+/**\n+ * percpu_counter_tree_precise_sum_positive() - Return a positive precise counter sum.\n+ * @counter: The counter to sum.\n+ *\n+ * Return a precise counter sum which is guaranteed to be greater\n+ * or equal to 0.\n+ *\n+ * Return: The current positive precise counter sum.\n+ */\n+static inline\n+long percpu_counter_tree_precise_sum_positive(struct percpu_counter_tree *counter)\n+{\n+\tlong v = percpu_counter_tree_precise_sum(counter);\n+\n+\treturn v \u003e 0 ? v : 0;\n+}\n+\n+/**\n+ * percpu_counter_tree_approximate_min_max_range() - Return the min and max precise values.\n+ * @approx_sum: Approximated sum.\n+ * @accuracy: Tree accuracy, from percpu_counter_tree_approximate_accuracy().\n+ * @precise_min: Minimum possible value for precise sum (output).\n+ * @precise_max: Maximum possible value for precise sum (output).\n+ *\n+ * Calculate the minimum and maximum precise values for a given approximation\n+ * and accuracy. The accuracy is a bound on a settled tree: one with no update\n+ * in progress.\n+ *\n+ * The approximation is within the accuracy of the precise sum, in both\n+ * directions:\n+ *\n+ * precise_sum - accuracy \u003c= approx_sum \u003c= precise_sum + accuracy\n+ *\n+ * Therefore, the precise sum is within the accuracy of the approximation:\n+ *\n+ * approx_sum - accuracy \u003c= precise_sum \u003c= approx_sum + accuracy\n+ */\n+static inline\n+void percpu_counter_tree_approximate_min_max_range(long approx_sum, unsigned long accuracy,\n+\t\t\t\t\t\t long *precise_min, long *precise_max)\n+{\n+\t*precise_min = approx_sum - accuracy;\n+\t*precise_max = approx_sum + accuracy;\n+}\n+\n+/**\n+ * percpu_counter_tree_approximate_min_max() - Return the approximation, min and max precise values.\n+ * @counter: The counter to sum.\n+ * @approx_sum: Approximate sum (output).\n+ * @precise_min: Minimum possible value for precise sum (output).\n+ * @precise_max: Maximum possible value for precise sum (output).\n+ *\n+ * Return the approximate sum, minimum and maximum precise values for\n+ * a counter.\n+ */\n+static inline\n+void percpu_counter_tree_approximate_min_max(struct percpu_counter_tree *counter,\n+\t\t\t\t\t long *approx_sum, long *precise_min, long *precise_max)\n+{\n+\tunsigned long accuracy = percpu_counter_tree_approximate_accuracy(counter);\n+\tlong v = percpu_counter_tree_approximate_sum(counter);\n+\n+\tpercpu_counter_tree_approximate_min_max_range(v, accuracy, precise_min, precise_max);\n+\t*approx_sum = v;\n+}\n+\n+#endif /* _PERCPU_COUNTER_TREE_H */\ndiff --git a/include/trace/events/kmem.h b/include/trace/events/kmem.h\nindex cd7920c81f85ba..8ee6d5df596105 100644\n--- a/include/trace/events/kmem.h\n+++ b/include/trace/events/kmem.h\n@@ -448,8 +448,7 @@ TRACE_EVENT(rss_stat,\n \t\t */\n \t\t__entry-\u003ecurr = current-\u003emm == mm \u0026\u0026 !(current-\u003eflags \u0026 PF_KTHREAD);\n \t\t__entry-\u003emember = member;\n-\t\t__entry-\u003esize = (percpu_counter_sum_positive(\u0026mm-\u003erss_stat[member])\n-\t\t\t\t\t\t\t \u003c\u003c PAGE_SHIFT);\n+\t\t__entry-\u003esize = get_mm_counter(mm, member) \u003c\u003c PAGE_SHIFT;\n \t),\n \n \tTP_printk(\"mm_id=%u curr=%d type=%s size=%ldB\",\ndiff --git a/init/main.c b/init/main.c\nindex 7b0c8c34e6a911..d9614cc5bd63d2 100644\n--- a/init/main.c\n+++ b/init/main.c\n@@ -108,6 +108,7 @@\n #include \u003clinux/time_namespace.h\u003e\n #include \u003clinux/unaligned.h\u003e\n #include \u003clinux/vdso_datastore.h\u003e\n+#include \u003clinux/percpu_counter_tree.h\u003e\n #include \u003cnet/net_namespace.h\u003e\n \n #include \u003casm/io.h\u003e\n@@ -1104,6 +1105,8 @@ void start_kernel(void)\n \tsort_main_extable();\n \ttrap_init();\n \tmm_core_init();\n+\t/* After parsing its boot parameters and setting up slab, before the first counter. */\n+\tpercpu_counter_tree_subsystem_init();\n \tmaple_tree_init();\n \tpoking_init();\n \tftrace_init();\ndiff --git a/kernel/exit.c b/kernel/exit.c\nindex d5e7bc2af804a6..c153cd064987c6 100644\n--- a/kernel/exit.c\n+++ b/kernel/exit.c\n@@ -582,6 +582,9 @@ static void exit_mm(void)\n \tlocal_irq_enable();\n \ttask_unlock(current);\n \tmmap_read_unlock(mm);\n+\t/* From now on, this task updates the RSS counters of @mm atomically. */\n+\tif (READ_ONCE(mm-\u003erss_owner) == current)\n+\t\tWRITE_ONCE(mm-\u003erss_owner, NULL);\n \tmm_update_next_owner(mm);\n \tmmput(mm);\n \tif (test_thread_flag(TIF_MEMDIE))\ndiff --git a/kernel/fork.c b/kernel/fork.c\nindex 8b108ed371529c..d0641700cd9c65 100644\n--- a/kernel/fork.c\n+++ b/kernel/fork.c\n@@ -137,6 +137,28 @@\n */\n #define MAX_THREADS FUTEX_TID_MASK\n \n+/*\n+ * Batch size of rss stat approximation, tunable with the rss_stat_batch=\n+ * boot parameter to compare batch sizes on a given machine.\n+ */\n+static unsigned int rss_stat_batch_size __ro_after_init = 32;\n+\n+static int __init setup_rss_stat_batch(char *str)\n+{\n+\tunsigned int batch;\n+\n+\tif (kstrtouint(str, 0, \u0026batch) || batch \u003c 2 || !is_power_of_2(batch) ||\n+\t batch \u003e PERCPU_COUNTER_TREE_MAX_BATCH) {\n+\t\tpr_warn(\"rss_stat_batch: expecting a power of 2 between 2 and %u\\n\",\n+\t\t\tPERCPU_COUNTER_TREE_MAX_BATCH);\n+\t\treturn 1;\n+\t}\n+\trss_stat_batch_size = batch;\n+\tpr_info(\"rss_stat_batch: %u\\n\", batch);\n+\treturn 1;\n+}\n+__setup(\"rss_stat_batch=\", setup_rss_stat_batch);\n+\n /*\n * Protected counters by write_lock_irq(\u0026tasklist_lock)\n */\n@@ -633,7 +655,7 @@ static void check_mm(struct mm_struct *mm)\n \t\t\t \"Please make sure 'struct resident_page_types[]' is updated as well\");\n \n \tfor (i = 0; i \u003c NR_MM_COUNTERS; i++) {\n-\t\tlong x = percpu_counter_sum(\u0026mm-\u003erss_stat[i]);\n+\t\tlong x = percpu_counter_tree_precise_sum(\u0026mm-\u003erss_stat[i]);\n \n \t\tif (unlikely(x)) {\n \t\t\tpr_alert(\"BUG: Bad rss-counter state mm:%p type:%s val:%ld Comm:%s Pid:%d\\n\",\n@@ -738,7 +760,7 @@ void __mmdrop(struct mm_struct *mm)\n \tcheck_mm(mm);\n \tmm_pasid_drop(mm);\n \tmm_destroy_cid(mm);\n-\tpercpu_counter_destroy_many(mm-\u003erss_stat, NR_MM_COUNTERS);\n+\tpercpu_counter_tree_destroy_many(mm-\u003erss_stat, NR_MM_COUNTERS);\n \n \tfree_mm(mm);\n }\n@@ -1103,6 +1125,7 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p)\n \tmemset(\u0026mm-\u003erss_stat, 0, sizeof(mm-\u003erss_stat));\n \tspin_lock_init(\u0026mm-\u003epage_table_lock);\n \tspin_lock_init(\u0026mm-\u003earg_lock);\n+\tmutex_init(\u0026mm-\u003erss_stat_mutex);\n \tmm_init_cpumask(mm);\n \tmm_init_aio(mm);\n \tmm_init_owner(mm, p);\n@@ -1143,8 +1166,7 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p)\n \tif (mm_alloc_sched(mm))\n \t\tgoto fail_sched;\n \n-\tif (percpu_counter_init_many(mm-\u003erss_stat, 0, GFP_KERNEL_ACCOUNT,\n-\t\t\t\t NR_MM_COUNTERS))\n+\tif (percpu_counter_tree_init_many_lazy(mm-\u003erss_stat, NR_MM_COUNTERS, rss_stat_batch_size))\n \t\tgoto fail_pcpu;\n \n \tlru_gen_init_mm(mm);\n@@ -1537,6 +1559,8 @@ static struct mm_struct *dup_mm(struct task_struct *tsk,\n \n \tif (!mm_init(mm, tsk))\n \t\tgoto fail_nomem;\n+\t/* The child owns its RSS counters, see __add_mm_counter(). */\n+\tmm-\u003erss_owner = tsk;\n \n \tuprobe_start_dup_mmap();\n \terr = dup_mmap(mm, oldmm);\n@@ -1564,6 +1588,23 @@ static struct mm_struct *dup_mm(struct task_struct *tsk,\n \treturn NULL;\n }\n \n+/*\n+ * Serialize the activations of the RSS counter tree, whichever tasks create\n+ * threads. Once the last counter is active, the activation is complete and\n+ * thread creation skips the mutex. That check needs no acquire: the updates\n+ * and the sums of the counters order their own accesses to the tree. Not\n+ * observing the other counters active yet only delays their updates going\n+ * to the tree.\n+ */\n+static int mm_activate_rss_stat(struct mm_struct *mm)\n+{\n+\tif (percpu_counter_tree_active(\u0026mm-\u003erss_stat[NR_MM_COUNTERS - 1]))\n+\t\treturn 0;\n+\tguard(mutex)(\u0026mm-\u003erss_stat_mutex);\n+\treturn percpu_counter_tree_activate_many(mm-\u003erss_stat, NR_MM_COUNTERS,\n+\t\t\t\t\t\t GFP_KERNEL_ACCOUNT);\n+}\n+\n static int copy_mm(u64 clone_flags, struct task_struct *tsk)\n {\n \tstruct mm_struct *mm, *oldmm;\n@@ -1588,6 +1629,18 @@ static int copy_mm(u64 clone_flags, struct task_struct *tsk)\n \t\treturn 0;\n \n \tif (clone_flags \u0026 CLONE_VM) {\n+\t\tint err = 0;\n+\n+\t\t/*\n+\t\t * A second thread shares the mm: activate the RSS counter\n+\t\t * tree. A vfork child shares the mm while the parent waits,\n+\t\t * so the single counters still fit. The tree stays active if\n+\t\t * the clone fails later, which is harmless.\n+\t\t */\n+\t\tif (!(clone_flags \u0026 CLONE_VFORK))\n+\t\t\terr = mm_activate_rss_stat(oldmm);\n+\t\tif (err)\n+\t\t\treturn err;\n \t\tmmget(oldmm);\n \t\tmm = oldmm;\n \t} else {\ndiff --git a/lib/Kconfig b/lib/Kconfig\nindex 1e42f5167c68bc..6ab9644a4cb830 100644\n--- a/lib/Kconfig\n+++ b/lib/Kconfig\n@@ -41,6 +41,33 @@ config PACKING_KUNIT_TEST\n \n \t When in doubt, say N.\n \n+config PERCPU_COUNTER_TREE_KUNIT_TEST\n+\ttristate \"KUnit tests for hierarchical per-cpu counters\" if !KUNIT_ALL_TESTS\n+\tdepends on KUNIT\n+\tdefault KUNIT_ALL_TESTS\n+\thelp\n+\t This builds Kunit tests for the hierarchical per-cpu counters.\n+\n+\t For more information on KUnit and unit tests in general,\n+\t please refer to the KUnit documentation in Documentation/dev-tools/kunit/.\n+\n+\t When in doubt, say N.\n+\n+config PERCPU_COUNTER_TREE_BENCH\n+\ttristate \"Benchmark of hierarchical per-cpu counters\"\n+\tdepends on KUNIT \u0026\u0026 SMP\n+\thelp\n+\t This builds a KUnit benchmark comparing the hierarchical per-cpu\n+\t counters with struct percpu_counter: update cost with all online\n+\t CPUs updating shared and private counters, and init/destroy cost.\n+\t It runs for a few minutes on large machines.\n+\n+\t The batch size and the tree shape can be changed with the\n+\t percpu_counter_tree_bench.batch= and percpu_counter_tree.arity=\n+\t parameters, see lib/tests/percpu_counter_tree_bench.c.\n+\n+\t When in doubt, say N.\n+\n config BITREVERSE\n \ttristate\n \tselect GENERIC_BITREVERSE if !HAVE_ARCH_BITREVERSE\ndiff --git a/lib/Makefile b/lib/Makefile\nindex a2afe954805cb8..07410b1d20a379 100644\n--- a/lib/Makefile\n+++ b/lib/Makefile\n@@ -182,6 +182,7 @@ obj-$(CONFIG_TEXTSEARCH_KMP) += ts_kmp.o\n obj-$(CONFIG_TEXTSEARCH_BM) += ts_bm.o\n obj-$(CONFIG_TEXTSEARCH_FSM) += ts_fsm.o\n obj-$(CONFIG_SMP) += percpu_counter.o\n+obj-$(CONFIG_SMP) += percpu_counter_tree.o\n obj-$(CONFIG_AUDIT_GENERIC) += audit.o\n obj-$(CONFIG_AUDIT_COMPAT_GENERIC) += compat_audit.o\n \ndiff --git a/lib/percpu_counter_tree.c b/lib/percpu_counter_tree.c\nnew file mode 100644\nindex 00000000000000..491675897e41c2\n--- /dev/null\n+++ b/lib/percpu_counter_tree.c\n@@ -0,0 +1,1450 @@\n+// SPDX-License-Identifier: GPL-2.0+ OR MIT\n+// SPDX-FileCopyrightText: 2025-2026 Mathieu Desnoyers \u003cmathieu.desnoyers@efficios.com\u003e\n+\n+/*\n+ * Split Counters With Tree Approximation Propagation\n+ *\n+ * * Overview\n+ *\n+ * Each counter is a tree of counters. A CPU updates its own level 0\n+ * counter, a signed byte. Each counter keeps the residual it has not\n+ * propagated yet, and propagates a carry to its parent on the next level\n+ * once that residual reaches the batch size of its level, up to the root.\n+ * Most updates therefore stay on the updating CPU. The root holds the\n+ * approximate sum, read in O(1), and the precise sum adds the residuals of\n+ * all levels to it.\n+ *\n+ * * Tree shape\n+ *\n+ * Illustration with 8 CPUs and a fan-out of 2 at each level, which makes\n+ * 3 levels of counters below the root. This is not an actual\n+ * configuration: at 8 CPUs, the default tree is flat.\n+ *\n+ * Level 3: 0 (root)\n+ * | \\\n+ * | \\\n+ * | \\\n+ * Level 2: 0 1\n+ * | \\ | \\\n+ * | \\ | \\\n+ * | \\ | \\\n+ * Level 1: 0 2 1 3\n+ * | \\ | \\ | \\ | \\\n+ * | \\ | \\ | \\ | \\\n+ * | \\ | \\ | \\ | \\\n+ * Level 0: 0 4 2 6 1 5 3 7\n+ *\n+ * CPU c propagates into item (c \u0026 (L - 1)) of a level holding L items,\n+ * so the CPUs sharing an item are not adjacent: CPUs 0 and 4 share\n+ * level 1 item 0, and CPUs 0, 2, 4, 6 share level 2 item 0. This mapping\n+ * uses every item of each level and gives each item the same number of\n+ * CPUs, give or take one, even when nr_cpu_ids is not a power of 2. The\n+ * diagram lists the CPUs in bit-reversed order to avoid crossing lines.\n+ *\n+ * The actual fan-out of each level depends on the topology (see\n+ * per_nr_cpu_order_config): up to 32 CPUs, the tree is flat, and the\n+ * level 0 counters propagate directly to the root.\n+ *\n+ * * Batch sizes and residuals\n+ *\n+ * The batch size of a level, BATCH(level N), is the batch size of the\n+ * level below multiplied by the fan-out of that level. Example for\n+ * BATCH(level 0) = 32, with the tree of the illustration above:\n+ *\n+ * BATCH(level 0) = 32\n+ * BATCH(level 1) = 64\n+ * BATCH(level 2) = 128\n+ *\n+ * Once the residual of a counter reaches +/- BATCH(level N), it propagates\n+ * its multiples of BATCH(level N), truncated toward zero, and subtracts\n+ * them. This hysteresis prevents updates going back and forth around a\n+ * BATCH(level N) boundary from propagating up the tree each time.\n+ *\n+ * The residual of a settled counter therefore stays within\n+ * (-BATCH(level N), +BATCH(level N)), and moves in steps of the counter's\n+ * input: level 0 receives the updates, in steps of 1, while a counter\n+ * above level 0 only receives the carries of the level below, in steps of\n+ * BATCH(level N - 1). The largest residual of a counter is the largest\n+ * multiple of its input step below its batch size: BATCH(level 0) - 1 at\n+ * level 0, and BATCH(level N) - BATCH(level N - 1) above:\n+ *\n+ * input residual worst case worst case\n+ * step per counter below precise above precise\n+ * Level 0: 1 [-31 .. 31] 248 (8 * 31) 248 (8 * 31)\n+ * Level 1: 32 {-32, 0, 32} 128 (4 * 32) 128 (4 * 32)\n+ * Level 2: 64 {-64, 0, 64} 128 (2 * 64) 128 (2 * 64)\n+ * Total: 504 504\n+ *\n+ * * Approximation accuracy\n+ *\n+ * The approximate sum is the precise sum minus the residuals, so a\n+ * settled counter (no update in progress) is within:\n+ *\n+ * precise - 504 \u003c= approx \u003c= precise + 504\n+ *\n+ * The accuracy returned by percpu_counter_tree_approximate_accuracy() is\n+ * exact. It adds the largest residuals of all the counters:\n+ * BATCH(level 0) - 1 for each of the nr_cpu_ids level 0 counters, and\n+ * BATCH(level N) - BATCH(level N - 1) for each counter of level N above.\n+ * Counting each level 0 counter as one BATCH(level 0), minus its input\n+ * step of 1:\n+ *\n+ * accuracy_multiplier = nr_cpu_ids + sum over levels N \u003e= 1 of\n+ * (number of counters at level N) *\n+ * (BATCH(level N) - BATCH(level N - 1)) / BATCH(level 0)\n+ *\n+ * accuracy = BATCH(level 0) * accuracy_multiplier - nr_cpu_ids\n+ *\n+ * In the example above, accuracy_multiplier = 8 + 4 * 1 + 2 * 2 = 16, and:\n+ *\n+ * accuracy = 32 * 16 - 8 = 504\n+ *\n+ * Every counter above level 0 is reachable from a CPU below nr_cpu_ids,\n+ * even when nr_cpu_ids is not a power of 2, so both bounds can be reached\n+ * when the possible CPUs are contiguous. When they are sparse, the bound\n+ * also counts the level 0 residuals of the missing CPUs: it holds, but\n+ * cannot be reached.\n+ *\n+ * * Carry propagation\n+ *\n+ * The items at each tree position of the counters initialized together\n+ * share cache lines, which are not shared with other tree positions to\n+ * prevent false sharing between CPUs.\n+ *\n+ * Example with expanded values, for the tree and batch sizes above:\n+ *\n+ * counter_add(counter, inc):\n+ *\n+ * if (!inc)\n+ * return;\n+ *\n+ * // Level 0: per-CPU residual within (-32, +32).\n+ * old = percpu_read(counter @ Level 0);\n+ * do {\n+ * v = old + inc;\n+ * out = v \u003c 0 ? -(-v \u0026 ~0b00011111) : (v \u0026 ~0b00011111);\n+ * } while (!percpu_try_cmpxchg(counter @ Level 0, \u0026old, v - out));\n+ * inc = out;\n+ * if (!inc)\n+ * return;\n+ *\n+ * // Levels 1 and 2: add the carry, then extract the multiples\n+ * // of the level batch held by the counter, truncated toward\n+ * // zero. A failed cmpxchg updates res to the current residual.\n+ * res = atomic_long_add_return(counter @ Level 1, inc);\n+ * do {\n+ * inc = res \u003c 0 ? -(-res \u0026 ~0b00111111) : (res \u0026 ~0b00111111);\n+ * if (!inc)\n+ * return;\n+ * } while (!atomic_long_try_cmpxchg(counter @ Level 1, \u0026res, res - inc));\n+ *\n+ * res = atomic_long_add_return(counter @ Level 2, inc);\n+ * do {\n+ * inc = res \u003c 0 ? -(-res \u0026 ~0b01111111) : (res \u0026 ~0b01111111);\n+ * if (!inc)\n+ * return;\n+ * } while (!atomic_long_try_cmpxchg(counter @ Level 2, \u0026res, res - inc));\n+ *\n+ * atomic_long_add(counter @ Level 3, inc);\n+ */\n+\n+#define pr_fmt(fmt) \"percpu_counter_tree: \" fmt\n+\n+#include \u003clinux/percpu_counter_tree.h\u003e\n+#include \u003clinux/moduleparam.h\u003e\n+#include \u003clinux/align.h\u003e\n+#include \u003clinux/cache.h\u003e\n+#include \u003clinux/cpumask.h\u003e\n+#include \u003clinux/atomic.h\u003e\n+#include \u003clinux/export.h\u003e\n+#include \u003clinux/percpu.h\u003e\n+#include \u003clinux/errno.h\u003e\n+#include \u003clinux/slab.h\u003e\n+#include \u003clinux/math.h\u003e\n+#include \u003clinux/log2.h\u003e\n+#include \u003clinux/args.h\u003e\n+#include \u003clinux/string.h\u003e\n+#include \u003ckunit/visibility.h\u003e\n+\n+#define MAX_NR_LEVELS 7\n+\n+/*\n+ * The counter configuration is selected at boot time based on the\n+ * hardware topology.\n+ */\n+struct counter_config {\n+\t/*\n+\t * nr_levels is the number of hierarchical counter tree levels. It\n+\t * excludes the final level (approximate sum).\n+\t */\n+\tunsigned char nr_levels;\n+\t/* n-arity of each level from 0 to (nr_levels - 1). */\n+\tunsigned char n_arity_order[MAX_NR_LEVELS];\n+};\n+\n+/*\n+ * The fan-out of each level of the tree is chosen with a heuristic trying to\n+ * minimize both the tree depth and the fan-out.\n+ */\n+#define PCT_COUNTER_CONFIG(order, ...)\t\t\t\t\t\t\\\n+\t[order] = {\t\t\t\t\t\t\t\t\\\n+\t\t.nr_levels = COUNT_ARGS(__VA_ARGS__),\t\t\t\t\\\n+\t\t.n_arity_order = { __VA_ARGS__ },\t\t\t\t\\\n+\t}\n+\n+/*\n+ * Up to 32 CPUs, the tree is flat: the level 0 counters propagate directly\n+ * to the root, like struct percpu_counter. From 33 to 64 CPUs, the level 0\n+ * counters propagate to 4 intermediate counters, each shared by up to 16\n+ * CPUs, so the carries of all CPUs do not contend on the root. Beyond, each\n+ * level has a fan-out of 8. When the order is not a multiple of 3, a\n+ * remaining order of 1 gives the root a fan-out of 16, and a remaining\n+ * order of 2 adds a level with a fan-out of 4 below the root: a root\n+ * fan-out of 16 updates about as fast as a fan-out of 2 below the root,\n+ * with one level less, which shrinks the accuracy bound and the tree\n+ * items, while a root fan-out of 32 is slower than a fan-out of 4 below\n+ * it. This keeps the carry rate reaching each tree item low with the RSS\n+ * batch size of 32, and the approximation error grows as\n+ * O(nr_cpus * log(nr_cpus)), compared to O(nr_cpus^2) for struct\n+ * percpu_counter whose batch size grows with the number of CPUs.\n+ *\n+ * The tree is shaped for nr_cpu_ids rounded up to a power of 2, and each\n+ * tree position takes a cache line. A set of 4 counters initialized together,\n+ * such as the RSS counters of an mm, uses less memory than 4 struct\n+ * percpu_counter from a few CPUs up to 64, and beyond when nr_cpu_ids is a\n+ * power of 2 (at most 0.89 times as much). Between powers of 2, fewer CPUs\n+ * share the cost of the tree items: up to 1.1 times at 3 * 2^n CPUs, and\n+ * up to 1.48 times just above a power of 2. A lone counter uses one item\n+ * of each cache line: 3.0 to 3.5 times the memory of a struct\n+ * percpu_counter when nr_cpu_ids is a power of 2 beyond 64, and up to 5.7\n+ * times just above.\n+ */\n+static const struct counter_config per_nr_cpu_order_config[] = {\n+\tPCT_COUNTER_CONFIG(0),\n+\tPCT_COUNTER_CONFIG(1, 1),\n+\tPCT_COUNTER_CONFIG(2, 2),\n+\tPCT_COUNTER_CONFIG(3, 3),\n+\tPCT_COUNTER_CONFIG(4, 4),\n+\tPCT_COUNTER_CONFIG(5, 5),\n+\tPCT_COUNTER_CONFIG(6, 4, 2),\n+\tPCT_COUNTER_CONFIG(7, 3, 4),\n+\tPCT_COUNTER_CONFIG(8, 3, 2, 3),\n+\tPCT_COUNTER_CONFIG(9, 3, 3, 3),\n+\tPCT_COUNTER_CONFIG(10, 3, 3, 4),\n+\tPCT_COUNTER_CONFIG(11, 3, 3, 2, 3),\n+\tPCT_COUNTER_CONFIG(12, 3, 3, 3, 3),\n+\tPCT_COUNTER_CONFIG(13, 3, 3, 3, 4),\n+\tPCT_COUNTER_CONFIG(14, 3, 3, 3, 2, 3),\n+\tPCT_COUNTER_CONFIG(15, 3, 3, 3, 3, 3),\n+\tPCT_COUNTER_CONFIG(16, 3, 3, 3, 3, 4),\n+\tPCT_COUNTER_CONFIG(17, 3, 3, 3, 3, 2, 3),\n+\tPCT_COUNTER_CONFIG(18, 3, 3, 3, 3, 3, 3),\n+\tPCT_COUNTER_CONFIG(19, 3, 3, 3, 3, 3, 4),\n+\tPCT_COUNTER_CONFIG(20, 3, 3, 3, 3, 3, 2, 3),\n+};\n+\n+#undef PCT_COUNTER_CONFIG\n+\n+/* The configuration table covers every possible number of CPUs. */\n+#define MAX_NR_CPUS_ORDER\t20\n+#if NR_CPUS \u003e (1UL \u003c\u003c MAX_NR_CPUS_ORDER)\n+#error \"NR_CPUS exceeds the hierarchical counter configuration table\"\n+#endif\n+static_assert(ARRAY_SIZE(per_nr_cpu_order_config) == MAX_NR_CPUS_ORDER + 1);\n+\n+/* Tree topology, selected at boot and shared by all the counters. */\n+static struct {\n+\t/* Hierarchical counter configuration for the hardware topology. */\n+\tconst struct counter_config *config;\n+\t/* Tree shape selected with the \"arity\" boot parameter. */\n+\tstruct counter_config boot_config;\n+\t/* Order of nr_cpu_ids. */\n+\tunsigned int nr_cpus_order;\n+\t/*\n+\t * Tree items per counter, including the root. UINT_MAX catches early\n+\t * boot misuses.\n+\t */\n+\tunsigned int nr_items;\n+\t/*\n+\t * Largest residuals of all the counters, in units of the level 0\n+\t * batch size: see percpu_counter_tree_accuracy().\n+\t */\n+\tunsigned long accuracy_multiplier;\n+\t/* Tree items when those of a tree position fit in one cache line. */\n+\tstruct kmem_cache *items_cachep;\n+} tree_topology __ro_after_init = {\n+\t.nr_items = UINT_MAX,\n+};\n+\n+/* Value of the \"arity\" boot parameter, parsed at subsystem initialization. */\n+static const char *arity_param;\n+\n+/*\n+ * Size of the items of one tree position for the counters initialized\n+ * together by percpu_counter_tree_init_many(), packed in cache lines which\n+ * are not shared with other tree positions. Each item is the residual of\n+ * the carries received and not propagated yet, within (-batch, +batch) of\n+ * its tree level when settled. The items of a tree position are updated\n+ * by the same CPUs, so packing them does not add contention between\n+ * unrelated CPUs.\n+ */\n+static\n+size_t percpu_counter_tree_position_size(unsigned int nr_counters)\n+{\n+\treturn ALIGN(nr_counters * sizeof(atomic_long_t), SMP_CACHE_BYTES);\n+}\n+\n+/* Distance between the consecutive items of each of @nr_counters counters. */\n+static inline\n+unsigned int percpu_counter_tree_items_stride(unsigned int nr_counters)\n+{\n+\treturn percpu_counter_tree_position_size(nr_counters) / sizeof(atomic_long_t);\n+}\n+\n+/* Allocate the zeroed tree items of @nr_counters counters, the root last. */\n+static\n+atomic_long_t *percpu_counter_tree_alloc_items(unsigned int nr_counters, gfp_t gfp_flags)\n+{\n+\tsize_t position_size = percpu_counter_tree_position_size(nr_counters);\n+\n+\tif (position_size == SMP_CACHE_BYTES)\n+\t\treturn kmem_cache_zalloc(tree_topology.items_cachep, gfp_flags);\n+\t/* kmalloc() aligns power of 2 sizes on their size. */\n+\treturn kzalloc(roundup_pow_of_two(tree_topology.nr_items * position_size),\n+\t\t gfp_flags);\n+}\n+\n+static\n+void percpu_counter_tree_free_items(atomic_long_t *items, unsigned int nr_counters)\n+{\n+\tif (percpu_counter_tree_position_size(nr_counters) == SMP_CACHE_BYTES)\n+\t\tkmem_cache_free(tree_topology.items_cachep, items);\n+\telse\n+\t\tkfree(items);\n+}\n+\n+static\n+int __percpu_counter_tree_init(struct percpu_counter_tree *counter,\n+\t\t\t unsigned long batch_size,\n+\t\t\t s8 __percpu *level0,\n+\t\t\t atomic_long_t *items, unsigned int items_stride)\n+{\n+\t/* Batch size must be greater than 1, a power of 2, and fit in a byte. */\n+\tif (WARN_ON(batch_size \u003c= 1 || (batch_size \u0026 (batch_size - 1)) ||\n+\t\t batch_size \u003e PERCPU_COUNTER_TREE_MAX_BATCH))\n+\t\treturn -EINVAL;\n+\tcounter-\u003ebatch_order = ilog2(batch_size);\n+\tcounter-\u003ebias = 0;\n+\tatomic_long_set(\u0026counter-\u003eatomic_bias, 0);\n+\tcounter-\u003elevel0 = level0;\n+\tcounter-\u003eitems_stride = items_stride;\n+\t/* The root is the last item, allocated by the activation if lazy. */\n+\tcounter-\u003eapprox_sum = items ?\n+\t\t\u0026items[(tree_topology.nr_items - 1) * items_stride] : NULL;\n+\treturn 0;\n+}\n+\n+static inline\n+unsigned long percpu_counter_tree_batch_size(struct percpu_counter_tree *counter)\n+{\n+\treturn 1UL \u003c\u003c counter-\u003ebatch_order;\n+}\n+\n+/* This counter's first item. */\n+static inline\n+atomic_long_t *percpu_counter_tree_items(struct percpu_counter_tree *counter)\n+{\n+\treturn counter-\u003eapprox_sum - (tree_topology.nr_items - 1) * counter-\u003eitems_stride;\n+}\n+\n+/*\n+ * The residuals of all levels are within (-batch, +batch) of their level,\n+ * so the approximation is within the same distance of the precise sum in\n+ * both directions: the largest residuals of all the counters, which\n+ * accuracy_multiplier counts in units of the level 0 batch size, minus the\n+ * input step of 1 of each level 0 counter. See \"Approximation accuracy\" at\n+ * the top of this file.\n+ */\n+static inline\n+unsigned long percpu_counter_tree_accuracy(struct percpu_counter_tree *counter)\n+{\n+\tunsigned long batch_size = percpu_counter_tree_batch_size(counter);\n+\n+\treturn batch_size * tree_topology.accuracy_multiplier - nr_cpu_ids;\n+}\n+\n+/* Initialize @nr_counters counters, with their per-CPU counters and tree items, or NULL if lazy. */\n+static\n+int __percpu_counter_tree_init_many(struct percpu_counter_tree *counters,\n+\t\t\t\t unsigned int nr_counters, unsigned long batch_size,\n+\t\t\t\t s8 __percpu *level0, atomic_long_t *items)\n+{\n+\tunsigned int items_stride = percpu_counter_tree_items_stride(nr_counters);\n+\tunsigned int i;\n+\tint ret;\n+\n+\tfor (i = 0; i \u003c nr_counters; i++) {\n+\t\tret = __percpu_counter_tree_init(\u0026counters[i], batch_size,\n+\t\t\t\t\t\t level0 ? level0 + i : NULL,\n+\t\t\t\t\t\t items ? items + i : NULL, items_stride);\n+\t\tif (ret)\n+\t\t\treturn ret;\n+\t}\n+\treturn 0;\n+}\n+\n+/**\n+ * percpu_counter_tree_init_many() - Initialize many per-CPU counter trees.\n+ * @counters: An array of @nr_counters counters to initialize.\n+ *\t Their memory is provided by the caller.\n+ * @nr_counters: The number of counter trees to initialize\n+ * @batch_size: The batch size is the increment step at level 0 which triggers a\n+ *\t\tcarry propagation.\n+ *\t\tThe batch size is required to be greater than 1, a power of 2, and at\n+ *\t\tmost PERCPU_COUNTER_TREE_MAX_BATCH.\n+ * @gfp_flags: gfp flags to pass to the per-CPU and tree item allocators.\n+ *\n+ * Initialize many per-CPU counter trees, with a single allocation of\n+ * per-CPU counters and a single allocation of tree items for the\n+ * @nr_counters counters. The items at each tree position of the\n+ * @nr_counters counters share cache lines. When they fit in one cache\n+ * line (up to 8 counters on 64-bit with 64-byte cache lines), the tree\n+ * items come from a slab cache; beyond, their allocation is\n+ * rounded up to a power of 2, which can double it.\n+ *\n+ * Return:\n+ * * %0: Success\n+ * * %-EINVAL:\t\t- Invalid @batch_size argument\n+ * * %-ENOMEM:\t\t- Out of memory\n+ */\n+int percpu_counter_tree_init_many(struct percpu_counter_tree *counters,\n+\t\t\t\t unsigned int nr_counters, unsigned long batch_size,\n+\t\t\t\t gfp_t gfp_flags)\n+{\n+\ts8 __percpu *level0;\n+\tatomic_long_t *items;\n+\tint ret;\n+\n+\t/* The tree shape is selected by percpu_counter_tree_subsystem_init() at boot. */\n+\tif (WARN_ON_ONCE(tree_topology.nr_items == UINT_MAX))\n+\t\treturn -EINVAL;\n+\tlevel0 = __alloc_percpu_gfp(nr_counters * sizeof(*level0),\n+\t\t\t\t __alignof__(*counters-\u003elevel0), gfp_flags);\n+\tif (!level0)\n+\t\treturn -ENOMEM;\n+\titems = percpu_counter_tree_alloc_items(nr_counters, gfp_flags);\n+\tif (!items) {\n+\t\tret = -ENOMEM;\n+\t\tgoto free_level0;\n+\t}\n+\tret = __percpu_counter_tree_init_many(counters, nr_counters, batch_size, level0, items);\n+\tif (!ret)\n+\t\treturn 0;\n+\tpercpu_counter_tree_free_items(items, nr_counters);\n+free_level0:\n+\tfree_percpu(level0);\n+\treturn ret;\n+}\n+EXPORT_SYMBOL_GPL(percpu_counter_tree_init_many);\n+\n+/**\n+ * percpu_counter_tree_init() - Initialize one per-CPU counter tree.\n+ * @counter: Counter to initialize.\n+ *\t Its memory is provided by the caller.\n+ * @batch_size: The batch size is the increment step at level 0 which triggers a\n+ *\t\tcarry propagation.\n+ *\t\tThe batch size is required to be greater than 1, a power of 2, and at\n+ *\t\tmost PERCPU_COUNTER_TREE_MAX_BATCH.\n+ * @gfp_flags: gfp flags to pass to the per-CPU and tree item allocators.\n+ *\n+ * Initialize one per-CPU counter tree.\n+ *\n+ * Return:\n+ * * %0: Success\n+ * * %-EINVAL:\t\t- Invalid @batch_size argument\n+ * * %-ENOMEM:\t\t- Out of memory\n+ */\n+int percpu_counter_tree_init(struct percpu_counter_tree *counter,\n+\t\t\t unsigned long batch_size, gfp_t gfp_flags)\n+{\n+\treturn percpu_counter_tree_init_many(counter, 1, batch_size, gfp_flags);\n+}\n+EXPORT_SYMBOL_GPL(percpu_counter_tree_init);\n+\n+/**\n+ * percpu_counter_tree_init_many_lazy() - Initialize many counters, without their tree.\n+ * @counters: An array of @nr_counters counters to initialize.\n+ *\t Their memory is provided by the caller.\n+ * @nr_counters: The number of counter trees to initialize\n+ * @batch_size: The batch size, as for percpu_counter_tree_init_many().\n+ *\n+ * Initialize counters without allocating their per-CPU counters and tree\n+ * items, for counters which may never need the scalability of the tree,\n+ * such as the RSS counters of a single-threaded process. Until\n+ * percpu_counter_tree_activate_many() allocates the tree, the counters\n+ * are exact: percpu_counter_tree_add_owner() adds to the bias, written\n+ * only by the owner, percpu_counter_tree_add() adds to the atomic bias,\n+ * and the precise and approximate sums read both in O(1).\n+ * percpu_counter_tree_destroy_many() and percpu_counter_tree_set() work on\n+ * counters which were never activated. The accuracy range of a lazily\n+ * initialized counter is the range of its tree, even before activation,\n+ * which keeps the comparison functions correct across an activation.\n+ * The batch size is fixed here: the activation only publishes the per-CPU\n+ * counters and the tree, so updaters read the batch size without ordering.\n+ *\n+ * Return:\n+ * * %0: Success\n+ * * %-EINVAL:\t\t- Invalid @batch_size argument\n+ */\n+int percpu_counter_tree_init_many_lazy(struct percpu_counter_tree *counters,\n+\t\t\t\t unsigned int nr_counters, unsigned long batch_size)\n+{\n+\t/* The tree shape is selected by percpu_counter_tree_subsystem_init() at boot. */\n+\tif (WARN_ON_ONCE(tree_topology.nr_items == UINT_MAX))\n+\t\treturn -EINVAL;\n+\treturn __percpu_counter_tree_init_many(counters, nr_counters, batch_size, NULL, NULL);\n+}\n+EXPORT_SYMBOL_GPL(percpu_counter_tree_init_many_lazy);\n+\n+/*\n+ * Publication of the tree\n+ *\n+ * The updaters and the readers of a counter take no lock, and may run\n+ * while percpu_counter_tree_activate_many() publishes its tree. The\n+ * activation allocates the per-CPU counters and the tree items zeroed,\n+ * then publishes two pointers per counter, each with a store-release:\n+ *\n+ * - the release of approx_sum: the root is zeroed before its pointer,\n+ * - the release of level0, after it: the per-CPU counters, the tree items\n+ * and approx_sum are set before level0.\n+ *\n+ * level0 opens the tree to updates: until it is published, the tree is\n+ * empty. Each load of a published pointer pairs with one of the releases:\n+ *\n+ * - Update fast path, percpu_counter_tree_add() and\n+ * percpu_counter_tree_add_owner(): READ_ONCE(level0), then the per-CPU\n+ * counters through it. The address dependency pairs with the release of\n+ * level0, as for rcu_dereference().\n+ * - Carry slow path: smp_rmb() after the read of level0, then\n+ * READ_ONCE(approx_sum). Pairs with the release of level0, which\n+ * follows the store of approx_sum.\n+ * - Precise sum: smp_load_acquire(level0), then approx_sum, the tree items\n+ * and the per-CPU counters. Pairs with the release of level0.\n+ * - Approximate sum: READ_ONCE(approx_sum), then the root through it. The\n+ * address dependency pairs with the release of approx_sum.\n+ * - percpu_counter_tree_active(): READ_ONCE(level0), without ordering. It\n+ * only selects the update function, which reads level0 again, or lets\n+ * the caller skip an activation which is complete.\n+ * - Activations: serialized by the caller, for instance with a mutex,\n+ * which orders a complete activation before the check of level0 at the\n+ * start of the next one.\n+ *\n+ * The update fast path has no acquire. smp_load_acquire() is free on\n+ * strongly ordered architectures such as x86, but on weakly ordered ones,\n+ * such as arm64, powerpc and RISC-V, it costs an ordering load or a barrier\n+ * on every update, to order a publication which happens at most once in\n+ * the life of a counter. The address dependency orders the per-CPU\n+ * counters, reached through level0, at no cost. It does not order\n+ * approx_sum, a separate field: the carry slow path, taken only when an\n+ * update carries, pays an smp_rmb() to read it, and the precise sum, which\n+ * reads the whole tree, reads level0 with an acquire.\n+ *\n+ * Missing the publication is harmless. An updater which reads level0 as\n+ * NULL adds to the bias (owner) or to the atomic bias (others), and a\n+ * reader which reads a NULL pointer sums the bias and the atomic bias:\n+ * the sums keep reading both after the activation, and the tree receives\n+ * no update before level0 is published.\n+ *\n+ * The read barriers between the levels of the precise sum are unrelated\n+ * to the publication: they order carries, see\n+ * percpu_counter_tree_precise_sum_unbiased().\n+ */\n+\n+/**\n+ * percpu_counter_tree_activate_many() - Activate the tree of lazily initialized counters.\n+ * @counters: The array of @nr_counters counters passed to\n+ *\t percpu_counter_tree_init_many_lazy().\n+ * @nr_counters: The number of counters.\n+ * @gfp_flags: gfp flags to pass to the per-CPU and tree item allocators.\n+ *\n+ * Allocate the per-CPU counters and the tree items, and publish them:\n+ * later updates go to the tree, and the approximation then has the\n+ * accuracy of the tree. The updates made before activation stay in the\n+ * bias and the atomic bias, which the sums keep reading, so updates racing\n+ * with the activation are not lost: the bias only has the owner as writer,\n+ * and the atomic bias only receives atomic additions. Activating active\n+ * counters does nothing. @counters and @nr_counters must be those passed\n+ * to percpu_counter_tree_init_many_lazy().\n+ *\n+ * The activations of the same counters must be serialized by the caller,\n+ * for instance with a mutex: a concurrent activation would publish a second\n+ * tree and lose the updates made to the first. The counters are published\n+ * in order: once percpu_counter_tree_active() reports the last one active,\n+ * the activation is complete, and a caller can skip it and its\n+ * serialization. The caller may not observe the other counters active yet,\n+ * which only delays their updates going to the tree.\n+ *\n+ * Return:\n+ * * %0: Success\n+ * * %-EINVAL:\t\t- @nr_counters differs from percpu_counter_tree_init_many_lazy()\n+ * * %-ENOMEM:\t\t- Out of memory\n+ */\n+int percpu_counter_tree_activate_many(struct percpu_counter_tree *counters,\n+\t\t\t\t unsigned int nr_counters, gfp_t gfp_flags)\n+{\n+\ts8 __percpu *level0;\n+\tatomic_long_t *items, *root;\n+\tunsigned int i;\n+\n+\t/* Serialized by the caller: a previous activation is complete. */\n+\tif (counters[0].level0)\n+\t\treturn 0;\n+\t/* The tree items are sized for @nr_counters, at the stride set by the initialization. */\n+\tif (WARN_ON_ONCE(counters[0].items_stride !=\n+\t\t\t percpu_counter_tree_items_stride(nr_counters)))\n+\t\treturn -EINVAL;\n+\tlevel0 = __alloc_percpu_gfp(nr_counters * sizeof(*level0),\n+\t\t\t\t __alignof__(*counters-\u003elevel0), gfp_flags);\n+\tif (!level0)\n+\t\treturn -ENOMEM;\n+\titems = percpu_counter_tree_alloc_items(nr_counters, gfp_flags);\n+\tif (!items) {\n+\t\tfree_percpu(level0);\n+\t\treturn -ENOMEM;\n+\t}\n+\troot = \u0026items[(tree_topology.nr_items - 1) * counters[0].items_stride];\n+\t/* See \"Publication of the tree\" above. */\n+\tfor (i = 0; i \u003c nr_counters; i++) {\n+\t\t/* Release of approx_sum: the root is zeroed before its pointer. */\n+\t\tsmp_store_release(\u0026counters[i].approx_sum, root + i);\n+\t\t/*\n+\t\t * Release of level0: the per-CPU counters, the tree items and\n+\t\t * approx_sum are set before level0, which opens the tree to\n+\t\t * updates.\n+\t\t */\n+\t\tsmp_store_release(\u0026counters[i].level0, level0 + i);\n+\t}\n+\treturn 0;\n+}\n+EXPORT_SYMBOL_GPL(percpu_counter_tree_activate_many);\n+\n+/**\n+ * percpu_counter_tree_destroy_many() - Destroy many per-CPU counter trees.\n+ * @counters: Array of counters trees to destroy.\n+ * @nr_counters: The number of counter trees to destroy.\n+ *\n+ * Release internal resources allocated for @nr_counters per-CPU counter trees.\n+ * @counters and @nr_counters must be those passed to\n+ * percpu_counter_tree_init_many() or percpu_counter_tree_init_many_lazy().\n+ */\n+void percpu_counter_tree_destroy_many(struct percpu_counter_tree *counters,\n+\t\t\t\t unsigned int nr_counters)\n+{\n+\tfree_percpu(counters-\u003elevel0);\n+\t/*\n+\t * @nr_counters selects the allocator of the tree items: on a mismatch\n+\t * with the initialization, leak them rather than corrupt the allocator.\n+\t */\n+\tif (WARN_ON_ONCE(counters-\u003eitems_stride !=\n+\t\t\t percpu_counter_tree_items_stride(nr_counters)))\n+\t\treturn;\n+\t/* The first item of the first counter is the base of the tree items, if allocated. */\n+\tif (counters-\u003eapprox_sum)\n+\t\tpercpu_counter_tree_free_items(percpu_counter_tree_items(counters), nr_counters);\n+}\n+EXPORT_SYMBOL_GPL(percpu_counter_tree_destroy_many);\n+\n+/**\n+ * percpu_counter_tree_destroy() - Destroy one per-CPU counter tree.\n+ * @counter: Counter to destroy.\n+ *\n+ * Release internal resources allocated for one per-CPU counter tree.\n+ */\n+void percpu_counter_tree_destroy(struct percpu_counter_tree *counter)\n+{\n+\treturn percpu_counter_tree_destroy_many(counter, 1);\n+}\n+EXPORT_SYMBOL_GPL(percpu_counter_tree_destroy);\n+\n+/*\n+ * Amount propagated by a counter holding @v: the multiples of @bit_mask,\n+ * truncated toward zero.\n+ */\n+static inline\n+long percpu_counter_tree_item_carry(long v, unsigned long bit_mask)\n+{\n+\tif (v \u003c 0)\n+\t\treturn -(long)(-(unsigned long)v \u0026 ~(bit_mask - 1));\n+\treturn (long)((unsigned long)v \u0026 ~(bit_mask - 1));\n+}\n+\n+/*\n+ * It does not matter through which path the carry propagates up the\n+ * tree, therefore there is no need to disable preemption because the\n+ * cpu number is only used to favor cache locality.\n+ */\n+static\n+void percpu_counter_tree_add_slowpath(struct percpu_counter_tree *counter, long inc)\n+{\n+\tunsigned int level_items, nr_levels = tree_topology.config-\u003enr_levels,\n+\t\t level, n_arity_order;\n+\tunsigned long bit_mask;\n+\tunsigned int stride = counter-\u003eitems_stride;\n+\tunsigned int cpu = raw_smp_processor_id();\n+\tatomic_long_t *root, *item;\n+\n+\t/*\n+\t * Pairs with the release of level0 in\n+\t * percpu_counter_tree_activate_many(), which follows the store of\n+\t * approx_sum. The caller read level0, but approx_sum is a separate\n+\t * field, which the address dependency of that read does not order:\n+\t * read it after level0.\n+\t */\n+\tsmp_rmb();\n+\troot = READ_ONCE(counter-\u003eapprox_sum);\n+\titem = root - (tree_topology.nr_items - 1) * stride;\n+\tn_arity_order = tree_topology.config-\u003en_arity_order[0];\n+\tbit_mask = percpu_counter_tree_batch_size(counter) \u003c\u003c n_arity_order;\n+\tlevel_items = 1U \u003c\u003c (tree_topology.nr_cpus_order - n_arity_order);\n+\n+\tfor (level = 1; level \u003c nr_levels; level++) {\n+\t\t/*\n+\t\t * A multiple of bit_mask would leave the residual of this\n+\t\t * level unchanged: propagate it to the next level directly.\n+\t\t * This trades the hysteresis of this level for not touching\n+\t\t * its cache line.\n+\t\t */\n+\t\tif (abs(inc) \u0026 (bit_mask - 1)) {\n+\t\t\tatomic_long_t *count = \u0026item[(cpu \u0026 (level_items - 1)) * stride];\n+\t\t\tlong res;\n+\n+\t\t\t/*\n+\t\t\t * Add the carry to the residual of this level, then\n+\t\t\t * extract the multiples of bit_mask of the residual,\n+\t\t\t * truncated toward zero, and propagate them to the\n+\t\t\t * next level, which leaves the residual within\n+\t\t\t * (-bit_mask, +bit_mask).\n+\t\t\t *\n+\t\t\t * The extraction is a cmpxchg on the value returned by\n+\t\t\t * the addition, whose cache line is already exclusive,\n+\t\t\t * so it usually succeeds at once. When a concurrent\n+\t\t\t * updater changed the residual in between, the cmpxchg\n+\t\t\t * fails and the carry is recomputed from the current\n+\t\t\t * value, which is 0 if the other updater extracted it.\n+\t\t\t * Each carry is thus extracted once, from an exact\n+\t\t\t * value: the residual only exceeds its range by the\n+\t\t\t * additions in flight, and is within range once all\n+\t\t\t * the updaters are done. An addition which does not\n+\t\t\t * carry stays a single atomic operation, which cannot\n+\t\t\t * fail.\n+\t\t\t *\n+\t\t\t * Relaxed ordering is sufficient for the residual\n+\t\t\t * itself: the reasoning above only involves this\n+\t\t\t * atomic, whose modifications all CPUs see in the same\n+\t\t\t * order, each read-modify-write reading the value\n+\t\t\t * written by the one just before it, whatever its\n+\t\t\t * ordering. Ordering only matters relative to the other\n+\t\t\t * levels, for the precise sum: the release ordering of\n+\t\t\t * the addition orders the departure of the carry from\n+\t\t\t * the level below before its arrival here.\n+\t\t\t */\n+\t\t\tres = atomic_long_add_return_release(inc, count);\n+\t\t\tdo {\n+\t\t\t\tinc = percpu_counter_tree_item_carry(res, bit_mask);\n+\t\t\t\tif (likely(!inc))\n+\t\t\t\t\treturn;\n+\t\t\t} while (!atomic_long_try_cmpxchg_relaxed(count, \u0026res, res - inc));\n+\t\t}\n+\t\titem += level_items * stride;\n+\t\tn_arity_order = tree_topology.config-\u003en_arity_order[level];\n+\t\tlevel_items \u003e\u003e= n_arity_order;\n+\t\tbit_mask \u003c\u003c= n_arity_order;\n+\t}\n+\t/* The atomic API has no add with release ordering without a return value. */\n+\tatomic_long_add_return_release(inc, root);\n+}\n+\n+/* Add @inc to the per-CPU counters @level0 of an activated tree. */\n+static inline\n+void percpu_counter_tree_add_tree(struct percpu_counter_tree *counter,\n+\t\t\t\t s8 __percpu *level0, long inc)\n+{\n+\tunsigned long bit_mask = percpu_counter_tree_batch_size(counter);\n+\ts8 old, new;\n+\tlong v, carry;\n+\n+\t/*\n+\t * Keep the residual within (-batch, +batch) and propagate its\n+\t * multiples of the batch size, truncated toward zero. This\n+\t * hysteresis prevents updates going back and forth around a batch\n+\t * boundary from propagating each time.\n+\t *\n+\t * Preemption stays enabled: the update may go to the level 0\n+\t * counter of any CPU, and the cmpxchg validates the value read.\n+\t * After a migration or an interrupt between the read and the\n+\t * cmpxchg, the cmpxchg either fails and retries with the current\n+\t * residual of the current CPU, or succeeds on a residual equal to\n+\t * the value read, which the new value correctly replaces. Each\n+\t * this_cpu operation is itself safe against preemption and\n+\t * interrupts, and the carry may propagate from any CPU, see\n+\t * percpu_counter_tree_add_slowpath().\n+\t */\n+\told = this_cpu_read(*level0);\n+\tdo {\n+\t\tv = old + inc;\n+\t\tcarry = percpu_counter_tree_item_carry(v, bit_mask);\n+\t\tnew = (s8)(v - carry);\n+\t} while (!this_cpu_try_cmpxchg(*level0, \u0026old, new));\n+\tif (likely(!carry))\n+\t\treturn;\n+\tpercpu_counter_tree_add_slowpath(counter, carry);\n+}\n+\n+/**\n+ * percpu_counter_tree_add() - Add to a per-CPU counter tree.\n+ * @counter: Counter added to.\n+ * @inc: Increment value (either positive or negative).\n+ *\n+ * Add @inc to a per-CPU counter tree. This is a fast-path which will\n+ * typically increment per-CPU counters as long as there is no carry\n+ * greater or equal to the counter tree batch size. Before the tree is\n+ * activated, add to the atomic bias.\n+ */\n+void percpu_counter_tree_add(struct percpu_counter_tree *counter, long inc)\n+{\n+\t/*\n+\t * Pairs with the release of level0 in\n+\t * percpu_counter_tree_activate_many(): the per-CPU counters are only\n+\t * reached through this pointer, and the address dependency orders\n+\t * their initialization before their use, as for rcu_dereference().\n+\t * No acquire, see \"Publication of the tree\".\n+\t */\n+\ts8 __percpu *level0 = READ_ONCE(counter-\u003elevel0);\n+\n+\tif (!level0) {\n+\t\tatomic_long_add(inc, \u0026counter-\u003eatomic_bias);\n+\t\treturn;\n+\t}\n+\tpercpu_counter_tree_add_tree(counter, level0, inc);\n+}\n+EXPORT_SYMBOL_GPL(percpu_counter_tree_add);\n+\n+/**\n+ * percpu_counter_tree_add_owner() - Add to a per-CPU counter tree as its owner.\n+ * @counter: Counter added to.\n+ * @inc: Increment value (either positive or negative).\n+ *\n+ * Like percpu_counter_tree_add(), but before the tree is activated, add\n+ * to the bias without atomic instruction. The calls for a given counter\n+ * must not run concurrently with each other: they come from a single\n+ * context, the owner chosen by the user of the counter, such as a task.\n+ */\n+void percpu_counter_tree_add_owner(struct percpu_counter_tree *counter, long inc)\n+{\n+\t/* Address dependency, as in percpu_counter_tree_add(). */\n+\ts8 __percpu *level0 = READ_ONCE(counter-\u003elevel0);\n+\n+\tif (!level0) {\n+\t\tWRITE_ONCE(counter-\u003ebias, (long)((unsigned long)READ_ONCE(counter-\u003ebias) + inc));\n+\t\treturn;\n+\t}\n+\tpercpu_counter_tree_add_tree(counter, level0, inc);\n+}\n+EXPORT_SYMBOL_GPL(percpu_counter_tree_add_owner);\n+\n+/*\n+ * The precise sum is the sum of the residuals of all tree levels. A carry\n+ * leaves a level before reaching a higher level, and the release ordering\n+ * of its arrival orders its departure before it. Reading the levels from\n+ * the root down, separated by read barriers, therefore never counts a carry\n+ * twice. A carry in flight during the sum may be missed: the result is\n+ * precise for a counter without concurrent updates.\n+ */\n+static\n+long percpu_counter_tree_precise_sum_unbiased(struct percpu_counter_tree *counter)\n+{\n+\tunsigned int nr_levels = tree_topology.config-\u003enr_levels, i,\n+\t\t level_items[MAX_NR_LEVELS], stride = counter-\u003eitems_stride;\n+\t/*\n+\t * Pairs with the release of level0 in\n+\t * percpu_counter_tree_activate_many(): approx_sum, the tree items and\n+\t * the per-CPU counters read below are set before level0.\n+\t */\n+\ts8 __percpu *level0 = smp_load_acquire(\u0026counter-\u003elevel0);\n+\tunsigned long sum = atomic_long_read(\u0026counter-\u003eatomic_bias);\n+\tatomic_long_t *item;\n+\tint level, cpu;\n+\n+\t/* Before activation, the atomic bias holds the whole unbiased sum. */\n+\tif (!level0)\n+\t\treturn (long)sum;\n+\t/* Number of items of each level above level 0. */\n+\tlevel_items[0] = 1U \u003c\u003c tree_topology.nr_cpus_order;\n+\tfor (level = 1; level \u003c (int)nr_levels; level++)\n+\t\tlevel_items[level] = level_items[level - 1] \u003e\u003e\n+\t\t\t\t tree_topology.config-\u003en_arity_order[level - 1];\n+\t/* Published before level0, so ordered by its acquire. */\n+\titem = READ_ONCE(counter-\u003eapprox_sum);\n+\tsum += atomic_long_read(item);\n+\tsmp_rmb();\t/* Read each level before the level below it. */\n+\t/* The items of each level precede those of the level above it. */\n+\tfor (level = (int)nr_levels - 1; level \u003e= 1; level--) {\n+\t\titem -= level_items[level] * stride;\n+\t\tfor (i = 0; i \u003c level_items[level]; i++)\n+\t\t\tsum += atomic_long_read(\u0026item[i * stride]);\n+\t\tsmp_rmb();\t/* Read this level before the level below it. */\n+\t}\n+\tfor_each_possible_cpu(cpu)\n+\t\tsum += (long)READ_ONCE(*per_cpu_ptr(level0, cpu));\n+\treturn (long)sum;\n+}\n+\n+/**\n+ * percpu_counter_tree_precise_sum() - Return precise counter sum.\n+ * @counter: The counter to sum.\n+ *\n+ * Querying the precise sum is relatively expensive because it needs to\n+ * iterate over all CPUs.\n+ * This is meant to be used when accuracy is preferred over speed.\n+ *\n+ * The sum is exact when no update runs concurrently. It takes no lock, so\n+ * a concurrent update which carries from level 0 to the levels above can\n+ * be counted in part: the sum may read the residual it left below and miss\n+ * its carry on the way up, an error in either direction of at most the\n+ * carries in flight. A carry is never counted twice.\n+ *\n+ * Return: The current precise counter sum.\n+ */\n+long percpu_counter_tree_precise_sum(struct percpu_counter_tree *counter)\n+{\n+\treturn percpu_counter_tree_precise_sum_unbiased(counter) + READ_ONCE(counter-\u003ebias);\n+}\n+EXPORT_SYMBOL_GPL(percpu_counter_tree_precise_sum);\n+\n+/*\n+ * The comparison between counters is on the wrapped difference. It is valid\n+ * only when the two counters differ by less than LONG_MAX, and results are\n+ * undefined beyond that.\n+ */\n+static inline\n+long wrapped_delta(long a, long b)\n+{\n+\treturn (long)((unsigned long)a - (unsigned long)b);\n+}\n+\n+/*\n+ * Each counter's approximation lies within its accuracy of its precise sum,\n+ * in both directions. The comparison only observes the approximations, so\n+ * it must reason about the possible precise values around each observed\n+ * approximation, which lie within [approx - accuracy, approx + accuracy]:\n+ *\n+ * precise_a range\n+ * \u003c-----------+-----------\u003e\n+ * accuracy_a | accuracy_a\n+ * approx_a\n+ * | precise_b range\n+ * | \u003c-----------+-----------\u003e\n+ * | accuracy_b | accuracy_b\n+ * | approx_b\n+ * | |\n+ * |\u003c------------ gap -------------\u003e|\n+ * ----------------------+--------------------------------+-------------\u003e values\n+ *\n+ * The two precise ranges are disjoint, and the comparison can return a\n+ * definitive answer, only once the gap between approximations exceeds the\n+ * sum of the two accuracies:\n+ *\n+ * gap \u003e accuracy_a + accuracy_b\n+ *\n+ * When comparing against a plain value, the value has zero width, so the\n+ * margin reduces to the accuracy of the counter: the value is above the\n+ * counter once it exceeds approx + accuracy, and below it once it is less\n+ * than approx - accuracy.\n+ */\n+static\n+int compare_delta(long delta, unsigned long accuracy)\n+{\n+\tif (delta \u003e= 0) {\n+\t\tif (delta \u003c= accuracy)\n+\t\t\treturn 0;\n+\t\telse\n+\t\t\treturn 1;\n+\t} else {\n+\t\tif (-delta \u003c= accuracy)\n+\t\t\treturn 0;\n+\t\telse\n+\t\t\treturn -1;\n+\t}\n+}\n+\n+/**\n+ * percpu_counter_tree_approximate_compare - Approximated comparison of two counter trees.\n+ * @a: First counter to compare.\n+ * @b: Second counter to compare.\n+ *\n+ * Evaluate an approximate comparison of two counter trees. This approximation\n+ * comparison is fast, and provides an accurate answer if the counters are\n+ * found to be either less than or greater than the other. However, if the\n+ * approximated comparison returns 0, the counters respective sums are found to\n+ * be within the two counters accuracy range. The two counters are read\n+ * independently; the result is not an atomic snapshot of both.\n+ * The comparison between counters is on the wrapped difference. It is valid\n+ * only when the two counters differ by less than LONG_MAX, and results are\n+ * undefined beyond that.\n+ *\n+ * Return:\n+ * * %0\t\t- Counters @a and @b do not differ by more than the sum of their respective\n+ * accuracy ranges.\n+ * * %-1\t- Counter @a less than counter @b.\n+ * * %1\t\t- Counter @a is greater than counter @b.\n+ */\n+int percpu_counter_tree_approximate_compare(struct percpu_counter_tree *a,\n+\t\t\t\t\t struct percpu_counter_tree *b)\n+{\n+\tunsigned long accuracy = percpu_counter_tree_accuracy(a) + percpu_counter_tree_accuracy(b);\n+\n+\t/* See the range geometry above compare_delta(). */\n+\treturn compare_delta(wrapped_delta(percpu_counter_tree_approximate_sum(a),\n+\t\t\t\t\t percpu_counter_tree_approximate_sum(b)),\n+\t\t\t accuracy);\n+}\n+EXPORT_SYMBOL_GPL(percpu_counter_tree_approximate_compare);\n+\n+/**\n+ * percpu_counter_tree_approximate_compare_value - Approximated comparison of a counter tree\n+ *\t\t\t\t\t\t against a given value.\n+ * @counter: Counter to compare.\n+ * @v: Value to compare.\n+ *\n+ * Evaluate an approximate comparison of a counter tree against a given value.\n+ * This approximation comparison is fast, and provides an accurate\n+ * answer if the counter is found to be either less than or greater\n+ * than the value. However, if the approximated comparison returns\n+ * 0, the value is within the counter accuracy range.\n+ * The comparison between counters is on the wrapped difference. It is valid\n+ * only when @v and the counter differ by less than LONG_MAX, and results are\n+ * undefined beyond that.\n+ *\n+ * Return:\n+ * * %0\t\t- The value @v is within the accuracy range of the counter.\n+ * * %-1\t- The value @v is less than the counter.\n+ * * %1\t\t- The value @v is greater than the counter.\n+ */\n+int percpu_counter_tree_approximate_compare_value(struct percpu_counter_tree *counter, long v)\n+{\n+\tunsigned long accuracy = percpu_counter_tree_accuracy(counter);\n+\n+\t/* See the range geometry above compare_delta(). */\n+\treturn compare_delta(wrapped_delta(v, percpu_counter_tree_approximate_sum(counter)),\n+\t\t\t accuracy);\n+}\n+EXPORT_SYMBOL_GPL(percpu_counter_tree_approximate_compare_value);\n+\n+/**\n+ * percpu_counter_tree_precise_compare - Precise comparison of two counter trees.\n+ * @a: First counter to compare.\n+ * @b: Second counter to compare.\n+ *\n+ * Evaluate a precise comparison of two counter trees. As an optimization, it\n+ * uses the approximate counter comparison to quickly compare counters which\n+ * are far apart. Only cases where counter sums are within the accuracy range\n+ * require precise counter sums. The two counters are read independently; the\n+ * result is not an atomic snapshot of both. With concurrent updates, the\n+ * precise sums can count updates in part, see\n+ * percpu_counter_tree_precise_sum().\n+ * The comparison between counters is on the wrapped difference. It is valid\n+ * only when the two counters differ by less than LONG_MAX, and results are\n+ * undefined beyond that.\n+ *\n+ * Return:\n+ * * %0\t\t- Counters are equal.\n+ * * %-1\t- Counter @a less than counter @b.\n+ * * %1\t\t- Counter @a is greater than counter @b.\n+ */\n+int percpu_counter_tree_precise_compare(struct percpu_counter_tree *a,\n+\t\t\t\t\tstruct percpu_counter_tree *b)\n+{\n+\tlong count_a = percpu_counter_tree_approximate_sum(a),\n+\t count_b = percpu_counter_tree_approximate_sum(b);\n+\tunsigned long accuracy_a = percpu_counter_tree_accuracy(a),\n+\t\t accuracy_b = percpu_counter_tree_accuracy(b);\n+\tlong delta = wrapped_delta(count_a, count_b);\n+\tint res;\n+\n+\t/* See the range geometry above compare_delta(). */\n+\tres = compare_delta(delta, accuracy_a + accuracy_b);\n+\t/* The values are distanced enough for an accurate approximated comparison. */\n+\tif (res)\n+\t\treturn res;\n+\n+\t/*\n+\t * The approximated comparison is within the accuracy range, therefore at least one\n+\t * precise sum is needed. Sum the counter which has the largest accuracy first.\n+\t */\n+\tif (accuracy_b \u003c accuracy_a) {\n+\t\tcount_a = percpu_counter_tree_precise_sum(a);\n+\t\t/* precise_a - approx_b: only b's precise range remains. */\n+\t\tres = compare_delta(wrapped_delta(count_a, count_b), accuracy_b);\n+\t\tif (res)\n+\t\t\treturn res;\n+\t\t/* Precise sum of second counter is required. */\n+\t\tcount_b = percpu_counter_tree_precise_sum(b);\n+\t} else {\n+\t\tcount_b = percpu_counter_tree_precise_sum(b);\n+\t\t/* approx_a - precise_b: only a's precise range remains. */\n+\t\tres = compare_delta(wrapped_delta(count_a, count_b), accuracy_a);\n+\t\tif (res)\n+\t\t\treturn res;\n+\t\t/* Precise sum of second counter is required. */\n+\t\tcount_a = percpu_counter_tree_precise_sum(a);\n+\t}\n+\tdelta = wrapped_delta(count_a, count_b);\n+\tif (delta \u003c 0)\n+\t\treturn -1;\n+\tif (delta \u003e 0)\n+\t\treturn 1;\n+\treturn 0;\n+}\n+EXPORT_SYMBOL_GPL(percpu_counter_tree_precise_compare);\n+\n+/**\n+ * percpu_counter_tree_precise_compare_value - Precise comparison of a counter tree against\n+ *\t\t\t\t\t a given value.\n+ * @counter: Counter to compare.\n+ * @v: Value to compare.\n+ *\n+ * Evaluate a precise comparison of a counter tree against a given value. As\n+ * an optimization, it uses the approximate counter comparison to quickly\n+ * identify whether the counter and value are far apart. Only cases where the\n+ * value is within the counter accuracy range require a precise counter sum.\n+ * With concurrent updates, the precise sum can count updates in part, see\n+ * percpu_counter_tree_precise_sum().\n+ * The comparison between counters is on the wrapped difference. It is valid\n+ * only when @v and the counter differ by less than LONG_MAX, and results are\n+ * undefined beyond that.\n+ *\n+ * Return:\n+ * * %0\t\t- The value @v is equal to the counter.\n+ * * %-1\t- The value @v is less than the counter.\n+ * * %1\t\t- The value @v is greater than the counter.\n+ */\n+int percpu_counter_tree_precise_compare_value(struct percpu_counter_tree *counter, long v)\n+{\n+\tlong count = percpu_counter_tree_approximate_sum(counter), delta;\n+\tunsigned long accuracy = percpu_counter_tree_accuracy(counter);\n+\tint res;\n+\n+\t/* See the range geometry above compare_delta(). */\n+\tres = compare_delta(wrapped_delta(v, count), accuracy);\n+\t/* The values are distanced enough for an accurate approximated comparison. */\n+\tif (res)\n+\t\treturn res;\n+\n+\t/* Precise sum is required. */\n+\tcount = percpu_counter_tree_precise_sum(counter);\n+\tdelta = wrapped_delta(v, count);\n+\tif (delta \u003c 0)\n+\t\treturn -1;\n+\tif (delta \u003e 0)\n+\t\treturn 1;\n+\treturn 0;\n+}\n+EXPORT_SYMBOL_GPL(percpu_counter_tree_precise_compare_value);\n+\n+/**\n+ * percpu_counter_tree_set - Set the counter tree sum to a given value.\n+ * @counter: Counter to set.\n+ * @v: Value to set.\n+ *\n+ * Set the counter sum to a given value. It can be useful for instance\n+ * to reset the counter sum to 0. Note that even after setting the\n+ * counter sum to a given value, the counter sum approximation can\n+ * return any value within the accuracy range around that value.\n+ *\n+ * The caller is responsible for preventing concurrent updates and\n+ * precise sums of @counter while it is being set.\n+ */\n+void percpu_counter_tree_set(struct percpu_counter_tree *counter, long v)\n+{\n+\tWRITE_ONCE(counter-\u003ebias, v - percpu_counter_tree_precise_sum_unbiased(counter));\n+}\n+EXPORT_SYMBOL_GPL(percpu_counter_tree_set);\n+\n+/**\n+ * percpu_counter_tree_approximate_accuracy - Query the accuracy of a counter tree.\n+ * @counter: Counter to query.\n+ *\n+ * The approximation differs from the precise sum by the residuals not yet propagated\n+ * up the tree. The residuals of all levels are symmetric, so the approximation is\n+ * within the same distance of the precise sum in both directions.\n+ * The accuracy is derived from the hardware topology and the counter tree batch size.\n+ * It is invariant for a given counter tree.\n+ * The accuracy describes a settled counter: one with no update in progress.\n+ * An update in flight between its per-CPU update and its carry propagation is not covered.\n+ * Using this function should not be typically required, see the following functions instead:\n+ * * percpu_counter_tree_approximate_compare(),\n+ * * percpu_counter_tree_approximate_compare_value(),\n+ * * percpu_counter_tree_precise_compare(),\n+ * * percpu_counter_tree_precise_compare_value().\n+ *\n+ * Return: The accuracy of the counter: the approximate sum of a settled counter\n+ * is within [precise sum - accuracy, precise sum + accuracy].\n+ */\n+unsigned long percpu_counter_tree_approximate_accuracy(struct percpu_counter_tree *counter)\n+{\n+\treturn percpu_counter_tree_accuracy(counter);\n+}\n+EXPORT_SYMBOL_GPL(percpu_counter_tree_approximate_accuracy);\n+\n+/* Calculate the number of tree items and the accuracy multiplier. */\n+static\n+void __init calculate_topology(void)\n+{\n+\tunsigned int nr_levels = tree_topology.config-\u003enr_levels, level;\n+\tunsigned int level_items = 1U \u003c\u003c tree_topology.nr_cpus_order;\n+\tunsigned long batch_size = 1;\n+\n+\t/* The root, and the items of each level above level 0. */\n+\ttree_topology.nr_items = 1;\n+\t/* One batch per level 0 counter: see percpu_counter_tree_accuracy(). */\n+\ttree_topology.accuracy_multiplier = nr_cpu_ids;\n+\n+\t/*\n+\t * The multiplier is derived from a batch size of 1 to speed up\n+\t * calculating the accuracy at tree initialization. A level N\n+\t * counter holds at most BATCH(level N) - BATCH(level N - 1) in\n+\t * either direction.\n+\t */\n+\tfor (level = 1; level \u003c nr_levels; level++) {\n+\t\tunsigned int n_arity_order = tree_topology.config-\u003en_arity_order[level - 1];\n+\n+\t\tlevel_items \u003e\u003e= n_arity_order;\n+\t\ttree_topology.nr_items += level_items;\n+\t\ttree_topology.accuracy_multiplier +=\n+\t\t\t((batch_size \u003c\u003c n_arity_order) - batch_size) * level_items;\n+\t\tbatch_size \u003c\u003c= n_arity_order;\n+\t}\n+}\n+\n+#if IS_ENABLED(CONFIG_KUNIT)\n+/*\n+ * Validate the configuration table: the n-arity orders of each entry must\n+ * sum to its order.\n+ *\n+ * Return: The number of invalid entries.\n+ */\n+unsigned int percpu_counter_tree_check_config(void)\n+{\n+\tunsigned int order, nr_errors = 0;\n+\n+\tfor (order = 0; order \u003c ARRAY_SIZE(per_nr_cpu_order_config); order++) {\n+\t\tconst struct counter_config *c = \u0026per_nr_cpu_order_config[order];\n+\t\tunsigned int level, shift = 0;\n+\n+\t\tfor (level = 0; level \u003c c-\u003enr_levels; level++)\n+\t\t\tshift += c-\u003en_arity_order[level];\n+\t\tif (shift != order) {\n+\t\t\tpr_err(\"order %u: arity sum %u\\n\", order, shift);\n+\t\t\tnr_errors++;\n+\t\t}\n+\t}\n+\treturn nr_errors;\n+}\n+EXPORT_SYMBOL_IF_KUNIT(percpu_counter_tree_check_config);\n+\n+/*\n+ * Check the invariants of a settled counter (no update in progress):\n+ * - a level 0 counter is within (-BATCH(level 0), +BATCH(level 0)),\n+ * - a counter at level N, above level 0, holds a multiple of\n+ * BATCH(level N - 1) within (-BATCH(level N), +BATCH(level N)),\n+ * - the root holds a multiple of the batch of the last level.\n+ * Before activation, a lazily initialized counter has no tree to check.\n+ *\n+ * Return: The number of violated invariants.\n+ */\n+unsigned int percpu_counter_tree_check_residuals(struct percpu_counter_tree *counter)\n+{\n+\tunsigned int nr_levels = tree_topology.config-\u003enr_levels, level, level_items,\n+\t\t nr_errors = 0, stride = counter-\u003eitems_stride;\n+\tunsigned long lo = percpu_counter_tree_batch_size(counter);\n+\tatomic_long_t *item;\n+\tlong root;\n+\tint cpu;\n+\n+\tif (!counter-\u003elevel0)\n+\t\treturn 0;\n+\titem = percpu_counter_tree_items(counter);\n+\tfor_each_possible_cpu(cpu) {\n+\t\tlong v = READ_ONCE(*per_cpu_ptr(counter-\u003elevel0, cpu));\n+\n+\t\tif (v \u003c= -(long)lo || v \u003e= (long)lo) {\n+\t\t\tpr_err(\"cpu %d: level 0 residual %ld, batch %lu\\n\", cpu, v, lo);\n+\t\t\tnr_errors++;\n+\t\t}\n+\t}\n+\tlevel_items = 1U \u003c\u003c tree_topology.nr_cpus_order;\n+\tfor (level = 1; level \u003c nr_levels; level++) {\n+\t\tunsigned int n_arity_order = tree_topology.config-\u003en_arity_order[level - 1], i;\n+\t\tunsigned long hi = lo \u003c\u003c n_arity_order;\n+\n+\t\tlevel_items \u003e\u003e= n_arity_order;\n+\t\tfor (i = 0; i \u003c level_items; i++) {\n+\t\t\tlong v = atomic_long_read(\u0026item[i * stride]);\n+\n+\t\t\tif (v \u003c= -(long)hi || v \u003e= (long)hi || ((unsigned long)v \u0026 (lo - 1))) {\n+\t\t\t\tpr_err(\"level %u item %u: residual %ld, batch %lu\\n\",\n+\t\t\t\t level, i, v, hi);\n+\t\t\t\tnr_errors++;\n+\t\t\t}\n+\t\t}\n+\t\titem += level_items * stride;\n+\t\tlo = hi;\n+\t}\n+\troot = atomic_long_read(counter-\u003eapprox_sum);\n+\tif ((unsigned long)root \u0026 (lo - 1)) {\n+\t\tpr_err(\"root %ld: not a multiple of %lu\\n\", root, lo);\n+\t\tnr_errors++;\n+\t}\n+\treturn nr_errors;\n+}\n+EXPORT_SYMBOL_IF_KUNIT(percpu_counter_tree_check_residuals);\n+\n+/*\n+ * Copy the state of counter @src into counter @dst, initialized with the\n+ * same batch size and without concurrent updates: their per-CPU counters,\n+ * tree items and biases. Both trees are active.\n+ */\n+void percpu_counter_tree_copy_state(struct percpu_counter_tree *dst,\n+\t\t\t\t struct percpu_counter_tree *src)\n+{\n+\tatomic_long_t *dst_items = percpu_counter_tree_items(dst),\n+\t\t *src_items = percpu_counter_tree_items(src);\n+\tunsigned int pos;\n+\tint cpu;\n+\n+\tfor_each_possible_cpu(cpu)\n+\t\t*per_cpu_ptr(dst-\u003elevel0, cpu) = *per_cpu_ptr(src-\u003elevel0, cpu);\n+\tfor (pos = 0; pos \u003c tree_topology.nr_items; pos++)\n+\t\tatomic_long_set(\u0026dst_items[pos * dst-\u003eitems_stride],\n+\t\t\t\tatomic_long_read(\u0026src_items[pos * src-\u003eitems_stride]));\n+\tdst-\u003ebias = src-\u003ebias;\n+\tatomic_long_set(\u0026dst-\u003eatomic_bias, atomic_long_read(\u0026src-\u003eatomic_bias));\n+}\n+EXPORT_SYMBOL_IF_KUNIT(percpu_counter_tree_copy_state);\n+#endif\n+\n+/* Room for the fan-out of each level, up to 7 digits and a comma, and a NUL. */\n+#define SHAPE_BUF_SIZE\t(8 * MAX_NR_LEVELS + 1)\n+\n+/* Format the fan-out of each level, from level 0 to the root, e.g. \"8,4,8\". */\n+static\n+int format_shape(char *buf, size_t size, const struct counter_config *config)\n+{\n+\tunsigned int level;\n+\tint len = 0;\n+\n+\tfor (level = 0; level \u003c config-\u003enr_levels; level++)\n+\t\tlen += scnprintf(buf + len, size - len, \"%s%u\", level ? \",\" : \"\",\n+\t\t\t\t 1U \u003c\u003c config-\u003en_arity_order[level]);\n+\treturn len;\n+}\n+\n+static\n+int arity_param_set(const char *val, const struct kernel_param *kp)\n+{\n+\t/* Applied once by percpu_counter_tree_subsystem_init(). */\n+\tif (tree_topology.config)\n+\t\treturn -EBUSY;\n+\tarity_param = val;\n+\treturn 0;\n+}\n+\n+static\n+int arity_param_get(char *buffer, const struct kernel_param *kp)\n+{\n+\tint len;\n+\n+\tif (!tree_topology.config)\n+\t\treturn sprintf(buffer, \"%s\\n\", arity_param ? : \"\");\n+\t/* The buffer is a page, larger than any shape and its newline. */\n+\tlen = format_shape(buffer, SHAPE_BUF_SIZE, tree_topology.config);\n+\tbuffer[len++] = '\\n';\n+\treturn len;\n+}\n+\n+static const struct kernel_param_ops arity_param_ops = {\n+\t.set = arity_param_set,\n+\t.get = arity_param_get,\n+};\n+\n+/*\n+ * percpu_counter_tree.arity=\u003cfan-out\u003e[,\u003cfan-out\u003e...] overrides the tree\n+ * shape chosen for nr_cpu_ids, to compare shapes on a given machine.\n+ * Reading the parameter shows the shape in use.\n+ */\n+module_param_cb(arity, \u0026arity_param_ops, NULL, 0444);\n+MODULE_PARM_DESC(arity, \"Fan-out of each tree level from level 0 to the root, e.g. 8,4,8\");\n+\n+/*\n+ * Parse a list of fan-outs, from level 0 to the root, into @config. Each\n+ * fan-out is a power of 2 of at least 2, and their product is nr_cpu_ids\n+ * rounded up to a power of 2.\n+ *\n+ * Return: 0 on success, -EINVAL if the list does not describe a tree\n+ * covering the CPU numbers below nr_cpu_ids.\n+ */\n+static\n+int __init parse_arity(const char *str, struct counter_config *config)\n+{\n+\tint ints[MAX_NR_LEVELS + 2];\t/* Count, levels, and one extra to detect overflow. */\n+\tunsigned int level, order = 0;\n+\n+\tif (*get_options(str, ARRAY_SIZE(ints), ints) || ints[0] \u003e MAX_NR_LEVELS)\n+\t\treturn -EINVAL;\n+\tconfig-\u003enr_levels = ints[0];\n+\tfor (level = 0; level \u003c config-\u003enr_levels; level++) {\n+\t\tint fan_out = ints[level + 1];\n+\n+\t\tif (fan_out \u003c 2 || !is_power_of_2(fan_out))\n+\t\t\treturn -EINVAL;\n+\t\tconfig-\u003en_arity_order[level] = ilog2(fan_out);\n+\t\torder += config-\u003en_arity_order[level];\n+\t\tif (order \u003e tree_topology.nr_cpus_order)\n+\t\t\treturn -EINVAL;\n+\t}\n+\tif (order != tree_topology.nr_cpus_order)\n+\t\treturn -EINVAL;\n+\treturn 0;\n+}\n+\n+int __init percpu_counter_tree_subsystem_init(void)\n+{\n+\tconst char *origin = \"default\";\n+\tchar shape[SHAPE_BUF_SIZE] = \"\";\n+\n+\ttree_topology.nr_cpus_order = get_count_order(nr_cpu_ids);\n+\tif (tree_topology.nr_cpus_order \u003e= ARRAY_SIZE(per_nr_cpu_order_config)) {\n+\t\tpr_err(\"Unsupported number of CPUs (%u)\\n\", nr_cpu_ids);\n+\t\treturn -EINVAL;\n+\t}\n+\ttree_topology.config = \u0026per_nr_cpu_order_config[tree_topology.nr_cpus_order];\n+\tif (arity_param \u0026\u0026 *arity_param) {\n+\t\tif (parse_arity(arity_param, \u0026tree_topology.boot_config)) {\n+\t\t\tpr_warn(\"Invalid arity \\\"%s\\\" for %u CPUs, using the default\\n\",\n+\t\t\t\tarity_param, nr_cpu_ids);\n+\t\t} else {\n+\t\t\ttree_topology.config = \u0026tree_topology.boot_config;\n+\t\t\torigin = \"boot parameter\";\n+\t\t}\n+\t}\n+\tcalculate_topology();\n+\ttree_topology.items_cachep =\n+\t\tkmem_cache_create(\"percpu_counter_tree_items\",\n+\t\t\t\t tree_topology.nr_items * SMP_CACHE_BYTES,\n+\t\t\t\t SMP_CACHE_BYTES, SLAB_PANIC, NULL);\n+\tformat_shape(shape, sizeof(shape), tree_topology.config);\n+\tpr_info(\"%u CPUs, fan-out %s (%s), %u tree items\\n\",\n+\t\tnr_cpu_ids, tree_topology.nr_cpus_order ? shape : \"none\", origin,\n+\t\ttree_topology.nr_items);\n+\treturn 0;\n+}\ndiff --git a/lib/tests/Makefile b/lib/tests/Makefile\nindex df0537e1b91faf..d158a8be8a1944 100644\n--- a/lib/tests/Makefile\n+++ b/lib/tests/Makefile\n@@ -59,4 +59,7 @@ obj-$(CONFIG_UTIL_MACROS_KUNIT) += util_macros_kunit.o\n obj-$(CONFIG_RATELIMIT_KUNIT_TEST) += test_ratelimit.o\n obj-$(CONFIG_UUID_KUNIT_TEST) += uuid_kunit.o\n \n+obj-$(CONFIG_PERCPU_COUNTER_TREE_KUNIT_TEST) += percpu_counter_tree_kunit.o\n+obj-$(CONFIG_PERCPU_COUNTER_TREE_BENCH) += percpu_counter_tree_bench.o\n+\n obj-$(CONFIG_TEST_RUNTIME_MODULE)\t\t+= module/\ndiff --git a/lib/tests/percpu_counter_tree_bench.c b/lib/tests/percpu_counter_tree_bench.c\nnew file mode 100644\nindex 00000000000000..bf260bef196a58\n--- /dev/null\n+++ b/lib/tests/percpu_counter_tree_bench.c\n@@ -0,0 +1,604 @@\n+// SPDX-License-Identifier: GPL-2.0+ OR MIT\n+/*\n+ * Benchmark of hierarchical per-CPU counters against struct percpu_counter.\n+ *\n+ * update: ns per percpu_counter_tree_add() per thread, one kthread pinned\n+ * on each online CPU, on one shared counter, on one counter per\n+ * thread, or on the counters of a few mm-like groups of 4 counters\n+ * initialized together. The \"single\" variant gives each thread a\n+ * lazily initialized counter updated with\n+ * percpu_counter_tree_add_owner(), like a single-threaded process.\n+ * The \"percpu_counter\" variants use percpu_counter_add() with the\n+ * default batch, like the RSS counters before hierarchical counters.\n+ * lifecycle: ns per init_many(4) + destroy_many(4), like fork + exit of an\n+ * mm, from one thread or from all online CPUs at once; \"lazy\" uses\n+ * init_many_lazy(4), like a single-threaded process.\n+ * churn: like lifecycle with a few updates, keeping one in @pin_period sets\n+ * of counters alive until the end, like long-lived processes.\n+ *\n+ * Each result ends with the implementation measured: \"tree\" for the\n+ * hierarchical counters, \"percpu_counter\" for struct percpu_counter.\n+ *\n+ * The knobs to try on a given machine are:\n+ * - the batch size: percpu_counter_tree_bench.batch=\u003c2..128\u003e (power of 2),\n+ * or \"modprobe percpu_counter_tree_bench batch=\u003cn\u003e\",\n+ * - the tree shape: percpu_counter_tree.arity=\u003cfan-out\u003e,... on the kernel\n+ * command line, from level 0 to the root. The product of the fan-outs\n+ * is nr_cpu_ids rounded up to a power of 2, e.g. \"16\" (flat) or \"4,4\"\n+ * when nr_cpu_ids is 16. The shape in use is shown in\n+ * /sys/module/percpu_counter_tree/parameters/arity.\n+ *\n+ * For example, in a VM:\n+ * ./tools/testing/kunit/kunit.py run --arch=x86_64 \\\n+ *\t--kconfig_add CONFIG_PERCPU_COUNTER_TREE_BENCH=y \\\n+ *\t--kconfig_add CONFIG_NR_CPUS=256 --qemu_args=\"-smp 64 -m 8192\" \\\n+ *\t--kernel_args percpu_counter_tree.arity=8,8 \\\n+ *\t--kernel_args percpu_counter_tree_bench.batch=16 \\\n+ *\t--timeout 3000 percpu_counter_tree_bench\n+ */\n+#include \u003ckunit/test.h\u003e\n+#include \u003clinux/percpu_counter_tree.h\u003e\n+#include \u003clinux/percpu_counter.h\u003e\n+#include \u003clinux/kthread.h\u003e\n+#include \u003clinux/random.h\u003e\n+#include \u003clinux/slab.h\u003e\n+#include \u003clinux/delay.h\u003e\n+#include \u003clinux/ktime.h\u003e\n+#include \u003clinux/math64.h\u003e\n+#include \u003clinux/sched/task.h\u003e\n+#include \u003clinux/cpuhplock.h\u003e\n+#include \u003clinux/moduleparam.h\u003e\n+\n+/* Like the RSS counters by default. */\n+static unsigned int batch = 32;\n+module_param(batch, uint, 0444);\n+MODULE_PARM_DESC(batch, \"Batch size of the hierarchical counters (power of 2, 2 to 128)\");\n+\n+#define BENCH_MS\t300\n+#define LIFECYCLE_NR\t4\t/* Counters per mm. */\n+#define LIFECYCLE_ITER\t2000\n+#define CHURN_ITER\t4096\n+\n+enum { W_MONO, W_BOUNCE, W_RAND32, W_RAND512, NR_W };\n+static const char * const wname[] = { \"monotonic +1\", \"bounce +-1\", \"random +-32\", \"random +-512\" };\n+\n+enum bench_impl { IMPL_TREE, IMPL_PERCPU_COUNTER };\n+static const char * const impl_name[] = { \"tree\", \"percpu_counter\" };\n+\n+struct bench_shared {\n+\tatomic_t ready;\n+\tint wl;\n+\tunsigned int pin_period;\t/* churn: keep one in pin_period sets, 0: none. */\n+\tenum bench_impl impl;\n+\tbool go, stop;\n+\tbool owner;\t\t/* update: percpu_counter_tree_add_owner(). */\n+\tbool lifecycle;\n+\tbool lazy;\t\t/* lifecycle: percpu_counter_tree_init_many_lazy(). */\n+};\n+\n+struct bench_thread {\n+\tu64 ns;\n+\tunsigned long ops;\n+\tstruct bench_shared *sh;\n+\tstruct percpu_counter_tree *c;\n+\tstruct percpu_counter *pc;\t/* update: IMPL_PERCPU_COUNTER. */\n+\tstruct percpu_counter_tree *pinned;\t/* churn: IMPL_TREE sets kept alive. */\n+\tstruct percpu_counter *pinned_pc;\t/* churn: IMPL_PERCPU_COUNTER sets kept alive. */\n+\tstruct task_struct *task;\n+\tunsigned int nr_pinned;\n+\tu32 seed;\n+} ____cacheline_aligned_in_smp;\n+\n+/*\n+ * Buffers are allocated through KUnit, which frees them if an assertion\n+ * aborts the test, and are freed explicitly otherwise, since the benchmark\n+ * runs many steps within one test.\n+ */\n+KUNIT_DEFINE_ACTION_WRAPPER(bench_kvfree, kvfree, const void *);\n+\n+static void *bench_kvzalloc(struct kunit *test, size_t size)\n+{\n+\tvoid *p = kvzalloc(size, GFP_KERNEL);\n+\n+\tif (p \u0026\u0026 kunit_add_action_or_reset(test, bench_kvfree, p))\n+\t\treturn NULL;\n+\treturn p;\n+}\n+\n+static void bench_kvfree_now(struct kunit *test, void *p)\n+{\n+\tkunit_release_action(test, bench_kvfree, p);\n+}\n+\n+/* Wait for the start signal; return true if the benchmark is cancelled. */\n+static bool bench_wait_go(struct bench_shared *sh)\n+{\n+\tatomic_inc(\u0026sh-\u003eready);\n+\t/* Pairs with bench_start(): a cancellation sets stop before go. */\n+\twhile (!smp_load_acquire(\u0026sh-\u003ego))\n+\t\tcond_resched();\n+\treturn READ_ONCE(sh-\u003estop);\n+}\n+\n+static int bench_update_fn(void *data)\n+{\n+\tstruct bench_thread *t = data;\n+\tstruct bench_shared *sh = t-\u003esh;\n+\tunsigned long ops = 0;\n+\tlong sign = -1;\n+\tu32 x = t-\u003eseed;\n+\tint i;\n+\n+\tif (bench_wait_go(sh))\n+\t\treturn 0;\n+\twhile (!READ_ONCE(sh-\u003estop)) {\n+\t\tfor (i = 0; i \u003c 1024; i++) {\n+\t\t\tlong inc;\n+\n+\t\t\tswitch (sh-\u003ewl) {\n+\t\t\tcase W_MONO:\n+\t\t\t\tinc = 1;\n+\t\t\t\tbreak;\n+\t\t\tcase W_BOUNCE:\n+\t\t\t\tinc = sign;\n+\t\t\t\tsign = -sign;\n+\t\t\t\tbreak;\n+\t\t\tdefault:\n+\t\t\t\tx ^= x \u003c\u003c 13;\n+\t\t\t\tx ^= x \u003e\u003e 17;\n+\t\t\t\tx ^= x \u003c\u003c 5;\n+\t\t\t\tinc = sh-\u003ewl == W_RAND32 ? 32 : 512;\n+\t\t\t\tif (x \u0026 1)\n+\t\t\t\t\tinc = -inc;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\tif (sh-\u003eimpl == IMPL_PERCPU_COUNTER)\n+\t\t\t\tpercpu_counter_add(t-\u003epc, inc);\n+\t\t\telse if (sh-\u003eowner)\n+\t\t\t\tpercpu_counter_tree_add_owner(t-\u003ec, inc);\n+\t\t\telse\n+\t\t\t\tpercpu_counter_tree_add(t-\u003ec, inc);\n+\t\t}\n+\t\tops += 1024;\n+\t\tcond_resched();\n+\t}\n+\tt-\u003eops = ops;\n+\treturn 0;\n+}\n+\n+static int bench_lifecycle_fn(void *data)\n+{\n+\tstruct bench_thread *t = data;\n+\tstruct percpu_counter_tree c[LIFECYCLE_NR];\n+\tstruct percpu_counter pc[LIFECYCLE_NR];\n+\tu64 t0;\n+\tint i;\n+\n+\tif (bench_wait_go(t-\u003esh))\n+\t\treturn 0;\n+\tt0 = ktime_get_ns();\n+\tfor (i = 0; t-\u003esh-\u003eimpl == IMPL_PERCPU_COUNTER \u0026\u0026 i \u003c LIFECYCLE_ITER; i++) {\n+\t\tif (percpu_counter_init_many(pc, 0, GFP_KERNEL, LIFECYCLE_NR))\n+\t\t\tbreak;\n+\t\tpercpu_counter_add(\u0026pc[0], 1);\n+\t\tpercpu_counter_add(\u0026pc[0], -1);\n+\t\tpercpu_counter_destroy_many(pc, LIFECYCLE_NR);\n+\t\tcond_resched();\n+\t}\n+\tfor (; t-\u003esh-\u003eimpl == IMPL_TREE \u0026\u0026 t-\u003esh-\u003elazy \u0026\u0026 i \u003c LIFECYCLE_ITER; i++) {\n+\t\tif (percpu_counter_tree_init_many_lazy(c, LIFECYCLE_NR, batch))\n+\t\t\tbreak;\n+\t\tpercpu_counter_tree_add_owner(\u0026c[0], 1);\n+\t\tpercpu_counter_tree_add_owner(\u0026c[0], -1);\n+\t\tpercpu_counter_tree_destroy_many(c, LIFECYCLE_NR);\n+\t\tcond_resched();\n+\t}\n+\tfor (; t-\u003esh-\u003eimpl == IMPL_TREE \u0026\u0026 !t-\u003esh-\u003elazy \u0026\u0026 i \u003c LIFECYCLE_ITER; i++) {\n+\t\tif (percpu_counter_tree_init_many(c, LIFECYCLE_NR, batch, GFP_KERNEL))\n+\t\t\tbreak;\n+\t\tpercpu_counter_tree_add(\u0026c[0], 1);\n+\t\tpercpu_counter_tree_add(\u0026c[0], -1);\n+\t\tpercpu_counter_tree_destroy_many(c, LIFECYCLE_NR);\n+\t\tcond_resched();\n+\t}\n+\tt-\u003ens = ktime_get_ns() - t0;\n+\tt-\u003eops = i;\n+\treturn 0;\n+}\n+\n+static int bench_churn_fn(void *data)\n+{\n+\tstruct bench_thread *t = data;\n+\tunsigned int pin_period = t-\u003esh-\u003epin_period;\n+\tstruct percpu_counter_tree tmp[LIFECYCLE_NR], *c;\n+\tstruct percpu_counter tmp_pc[LIFECYCLE_NR], *pc;\n+\tu64 t0;\n+\tint i;\n+\n+\tif (bench_wait_go(t-\u003esh))\n+\t\treturn 0;\n+\tt0 = ktime_get_ns();\n+\tfor (i = 0; t-\u003esh-\u003eimpl == IMPL_PERCPU_COUNTER \u0026\u0026 i \u003c CHURN_ITER; i++) {\n+\t\tbool pin = pin_period \u0026\u0026 !(i % pin_period);\n+\n+\t\tpc = pin ? \u0026t-\u003epinned_pc[t-\u003enr_pinned * LIFECYCLE_NR] : tmp_pc;\n+\t\tif (percpu_counter_init_many(pc, 0, GFP_KERNEL, LIFECYCLE_NR))\n+\t\t\tbreak;\n+\t\tpercpu_counter_add(\u0026pc[0], 100);\n+\t\tpercpu_counter_add(\u0026pc[1], -77);\n+\t\tpercpu_counter_add(\u0026pc[2], 1);\n+\t\tif (pin)\n+\t\t\tt-\u003enr_pinned++;\n+\t\telse\n+\t\t\tpercpu_counter_destroy_many(pc, LIFECYCLE_NR);\n+\t\tcond_resched();\n+\t}\n+\tfor (; t-\u003esh-\u003eimpl == IMPL_TREE \u0026\u0026 i \u003c CHURN_ITER; i++) {\n+\t\tbool pin = pin_period \u0026\u0026 !(i % pin_period);\n+\n+\t\tc = pin ? \u0026t-\u003epinned[t-\u003enr_pinned * LIFECYCLE_NR] : tmp;\n+\t\tif (percpu_counter_tree_init_many(c, LIFECYCLE_NR, batch, GFP_KERNEL))\n+\t\t\tbreak;\n+\t\tpercpu_counter_tree_add(\u0026c[0], 100);\n+\t\tpercpu_counter_tree_add(\u0026c[1], -77);\n+\t\tpercpu_counter_tree_add(\u0026c[2], 1);\n+\t\tif (pin)\n+\t\t\tt-\u003enr_pinned++;\n+\t\telse\n+\t\t\tpercpu_counter_tree_destroy_many(c, LIFECYCLE_NR);\n+\t\tcond_resched();\n+\t}\n+\tt-\u003ens = ktime_get_ns() - t0;\n+\tt-\u003eops = i;\n+\treturn 0;\n+}\n+\n+/* List at most @max online CPUs in @cpus, and return their number. */\n+static unsigned int bench_cpus(int *cpus, unsigned int max)\n+{\n+\tunsigned int n = 0;\n+\tint cpu;\n+\n+\tfor_each_online_cpu(cpu) {\n+\t\tif (n == max)\n+\t\t\tbreak;\n+\t\tcpus[n++] = cpu;\n+\t}\n+\treturn n;\n+}\n+\n+static void bench_stop(struct bench_thread *t, unsigned int n)\n+{\n+\tunsigned int i;\n+\n+\tfor (i = 0; i \u003c n; i++)\n+\t\tkthread_stop_put(t[i].task);\n+}\n+\n+/*\n+ * Start a thread on each of the @n CPUs of @cpus, waiting for the start\n+ * signal. On failure, cancel and join the threads already started.\n+ */\n+static int bench_start(struct bench_shared *sh, struct bench_thread *t, const int *cpus,\n+\t\t unsigned int n)\n+{\n+\tunsigned int i;\n+\n+\tfor (i = 0; i \u003c n; i++) {\n+\t\tt[i].sh = sh;\n+\t\tt[i].task = kthread_create_on_cpu(sh-\u003epin_period != UINT_MAX ? bench_churn_fn :\n+\t\t\t\t\t\t sh-\u003elifecycle ? bench_lifecycle_fn :\n+\t\t\t\t\t\t bench_update_fn, \u0026t[i], cpus[i], \"hpcc_bench/%u\");\n+\t\tif (IS_ERR(t[i].task)) {\n+\t\t\tint ret = PTR_ERR(t[i].task);\n+\n+\t\t\tWRITE_ONCE(sh-\u003estop, true);\n+\t\t\t/* Pairs with bench_wait_go(): the threads see stop with go. */\n+\t\t\tsmp_store_release(\u0026sh-\u003ego, true);\n+\t\t\tbench_stop(t, i);\n+\t\t\treturn ret;\n+\t\t}\n+\t\tget_task_struct(t[i].task);\n+\t\twake_up_process(t[i].task);\n+\t}\n+\twhile (atomic_read(\u0026sh-\u003eready) \u003c n)\n+\t\tusleep_range(1000, 2000);\n+\treturn 0;\n+}\n+\n+enum bench_mode { BENCH_SHARED, BENCH_PRIVATE, BENCH_MM, BENCH_SINGLE };\n+static const char * const mode_name[] = { \"shared\", \"private\", \"mm\", \"single\" };\n+\n+/*\n+ * BENCH_SHARED: all threads update one counter.\n+ * BENCH_PRIVATE: each thread updates its own counter (a separate mm).\n+ * BENCH_MM: blocks of n / LIFECYCLE_NR threads each update one counter of\n+ * one mm. Its counters share the cache lines of their tree items, and\n+ * CPUs sharing an item are strided, so each line is updated for every\n+ * counter.\n+ * BENCH_SINGLE: each thread owns a lazily initialized counter, like a\n+ * single-threaded process, never activated. Its IMPL_PERCPU_COUNTER\n+ * comparison is BENCH_PRIVATE.\n+ */\n+/* Counters @i of bench_update(), @c_stride and @pc_stride bytes apart. */\n+#define BENCH_C(i)\t((struct percpu_counter_tree *)((char *)c + (i) * c_stride))\n+#define BENCH_PC(i)\t((struct percpu_counter *)((char *)pc + (i) * pc_stride))\n+\n+static void bench_update(struct kunit *test, enum bench_mode mode, int wl,\n+\t\t\t enum bench_impl impl)\n+{\n+\tbool shared = mode == BENCH_SHARED || mode == BENCH_MM;\n+\tunsigned int n, i, nr_counters, nr_init = 0;\n+\tstruct bench_shared sh = { .wl = wl, .pin_period = UINT_MAX, .impl = impl,\n+\t\t\t\t .owner = mode == BENCH_SINGLE };\n+\tstruct percpu_counter_tree *c;\n+\tstruct percpu_counter *pc;\n+\tstruct bench_thread *t;\n+\tunsigned long ops = 0;\n+\tunsigned int *perm;\n+\tsize_t c_stride, pc_stride;\n+\tvoid *c_mem, *pc_mem;\n+\tu64 t0, t1;\n+\tint *cpus;\n+\tint ret;\n+\n+\tcpus = kunit_kcalloc(test, nr_cpu_ids, sizeof(*cpus), GFP_KERNEL);\n+\tt = bench_kvzalloc(test, array_size(nr_cpu_ids, sizeof(*t)));\n+\tperm = kunit_kcalloc(test, nr_cpu_ids, sizeof(*perm), GFP_KERNEL);\n+\tKUNIT_ASSERT_NOT_NULL(test, cpus);\n+\tKUNIT_ASSERT_NOT_NULL(test, t);\n+\tKUNIT_ASSERT_NOT_NULL(test, perm);\n+\t/*\n+\t * Set up the counters for the CPUs online now, before taking\n+\t * cpus_read_lock(), which a failed assertion would leave held. The\n+\t * threads then run on the CPUs online under the lock, at most as many.\n+\t */\n+\tn = num_online_cpus();\n+\tnr_counters = mode == BENCH_SHARED ? 1 : mode == BENCH_MM ? LIFECYCLE_NR : n;\n+\t/*\n+\t * Each private counter is written by its own thread: give each its\n+\t * own cache line, as the RSS counters of separate mms, to avoid false\n+\t * sharing. The counters of one mm stay packed, as in mm_struct.\n+\t */\n+\tc_stride = shared ? sizeof(*c) : ALIGN(sizeof(*c), SMP_CACHE_BYTES);\n+\tpc_stride = shared ? sizeof(*pc) : ALIGN(sizeof(*pc), SMP_CACHE_BYTES);\n+\tc_mem = kunit_kzalloc(test, nr_counters * c_stride + SMP_CACHE_BYTES, GFP_KERNEL);\n+\tKUNIT_ASSERT_NOT_NULL(test, c_mem);\n+\tc = PTR_ALIGN(c_mem, SMP_CACHE_BYTES);\n+\tpc_mem = kunit_kzalloc(test, nr_counters * pc_stride + SMP_CACHE_BYTES, GFP_KERNEL);\n+\tKUNIT_ASSERT_NOT_NULL(test, pc_mem);\n+\tpc = PTR_ALIGN(pc_mem, SMP_CACHE_BYTES);\n+\t/*\n+\t * Each private counter stands for a separate mm: initialize them\n+\t * separately so they do not share the cache lines of their tree items.\n+\t */\n+\tif (impl == IMPL_PERCPU_COUNTER \u0026\u0026 mode == BENCH_MM) {\n+\t\tif (!percpu_counter_init_many(pc, 0, GFP_KERNEL, nr_counters))\n+\t\t\tnr_init = nr_counters;\n+\t} else if (impl == IMPL_PERCPU_COUNTER) {\n+\t\tfor (; nr_init \u003c nr_counters; nr_init++)\n+\t\t\tif (percpu_counter_init(BENCH_PC(nr_init), 0, GFP_KERNEL))\n+\t\t\t\tbreak;\n+\t} else if (mode == BENCH_MM) {\n+\t\tif (!percpu_counter_tree_init_many(c, nr_counters, batch, GFP_KERNEL))\n+\t\t\tnr_init = nr_counters;\n+\t} else {\n+\t\tfor (; nr_init \u003c nr_counters; nr_init++) {\n+\t\t\tif (mode == BENCH_SINGLE ?\n+\t\t\t percpu_counter_tree_init_many_lazy(BENCH_C(nr_init), 1, batch) :\n+\t\t\t percpu_counter_tree_init(BENCH_C(nr_init), batch, GFP_KERNEL))\n+\t\t\t\tbreak;\n+\t\t}\n+\t}\n+\tif (nr_init \u003c nr_counters) {\n+\t\tKUNIT_FAIL(test, \"cannot initialize %u counters\", nr_counters);\n+\t\tgoto destroy;\n+\t}\n+\tcpus_read_lock();\n+\tn = bench_cpus(cpus, n);\n+\t/* Random placement of the private counters. */\n+\tfor (i = 0; i \u003c n; i++)\n+\t\tperm[i] = mode == BENCH_SHARED ? 0 : mode == BENCH_MM ? i * LIFECYCLE_NR / n : i;\n+\tfor (i = n - 1; !shared \u0026\u0026 i \u003e 0; i--)\n+\t\tswap(perm[i], perm[get_random_u32_below(i + 1)]);\n+\tfor (i = 0; i \u003c n; i++) {\n+\t\tt[i].c = BENCH_C(perm[i]);\n+\t\tt[i].pc = BENCH_PC(perm[i]);\n+\t\tt[i].seed = get_random_u32() | 1;\n+\t}\n+\tret = bench_start(\u0026sh, t, cpus, n);\n+\tif (ret) {\n+\t\tcpus_read_unlock();\n+\t\tKUNIT_FAIL(test, \"cannot start the benchmark threads: %d\", ret);\n+\t\tgoto destroy;\n+\t}\n+\tt0 = ktime_get_ns();\n+\tWRITE_ONCE(sh.go, true);\n+\tmsleep(BENCH_MS);\n+\tWRITE_ONCE(sh.stop, true);\n+\tt1 = ktime_get_ns();\n+\tbench_stop(t, n);\n+\tcpus_read_unlock();\n+\tfor (i = 0; i \u003c n; i++)\n+\t\tops += t[i].ops;\n+\tfor (i = 0; impl == IMPL_TREE \u0026\u0026 i \u003c nr_counters; i++)\n+\t\tKUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_check_residuals(BENCH_C(i)));\n+\tkunit_info(test, \"update %-8s %-14s %3u thr %10llu ns/op/thr %s\\n\",\n+\t\t mode_name[mode], wname[wl], n,\n+\t\t ops ? div64_ul((t1 - t0) * n, ops) : 0, impl_name[impl]);\n+destroy:\n+\tif (impl == IMPL_PERCPU_COUNTER \u0026\u0026 mode == BENCH_MM) {\n+\t\tif (nr_init)\n+\t\t\tpercpu_counter_destroy_many(pc, nr_counters);\n+\t} else if (impl == IMPL_PERCPU_COUNTER) {\n+\t\tfor (i = 0; i \u003c nr_init; i++)\n+\t\t\tpercpu_counter_destroy(BENCH_PC(i));\n+\t} else if (mode == BENCH_MM) {\n+\t\tif (nr_init)\n+\t\t\tpercpu_counter_tree_destroy_many(c, nr_counters);\n+\t} else {\n+\t\tfor (i = 0; i \u003c nr_init; i++)\n+\t\t\tpercpu_counter_tree_destroy(BENCH_C(i));\n+\t}\n+\tkunit_kfree(test, pc_mem);\n+\tkunit_kfree(test, c_mem);\n+\tkunit_kfree(test, perm);\n+\tbench_kvfree_now(test, t);\n+\tkunit_kfree(test, cpus);\n+}\n+\n+static void bench_lifecycle(struct kunit *test, bool all, enum bench_impl impl, bool lazy)\n+{\n+\tstruct bench_shared sh = { .lifecycle = true, .pin_period = UINT_MAX,\n+\t\t\t\t .impl = impl, .lazy = lazy };\n+\tunsigned int n, i;\n+\tstruct bench_thread *t;\n+\tunsigned long ops = 0;\n+\tu64 ns = 0;\n+\tint *cpus;\n+\tint ret;\n+\n+\tcpus = kunit_kcalloc(test, nr_cpu_ids, sizeof(*cpus), GFP_KERNEL);\n+\tt = bench_kvzalloc(test, array_size(nr_cpu_ids, sizeof(*t)));\n+\tKUNIT_ASSERT_NOT_NULL(test, cpus);\n+\tKUNIT_ASSERT_NOT_NULL(test, t);\n+\tcpus_read_lock();\n+\tn = bench_cpus(cpus, all ? num_online_cpus() : 1);\n+\tret = bench_start(\u0026sh, t, cpus, n);\n+\tif (ret) {\n+\t\tcpus_read_unlock();\n+\t\tKUNIT_FAIL(test, \"cannot start the benchmark threads: %d\", ret);\n+\t\tgoto free;\n+\t}\n+\tWRITE_ONCE(sh.go, true);\n+\tbench_stop(t, n);\n+\tcpus_read_unlock();\n+\tfor (i = 0; i \u003c n; i++) {\n+\t\tops += t[i].ops;\n+\t\tns += t[i].ns;\n+\t}\n+\tKUNIT_EXPECT_EQ(test, ops, (unsigned long)n * LIFECYCLE_ITER);\n+\tkunit_info(test, \"lifecycle init_many%s(%d)+destroy_many %3u thr %10llu ns/op %s\\n\",\n+\t\t lazy ? \"_lazy\" : \"\", LIFECYCLE_NR, n, ops ? div64_ul(ns, ops) : 0,\n+\t\t impl_name[impl]);\n+free:\n+\tbench_kvfree_now(test, t);\n+\tkunit_kfree(test, cpus);\n+}\n+\n+static void bench_churn(struct kunit *test, bool all, unsigned int pin_period,\n+\t\t\tenum bench_impl impl)\n+{\n+\tstruct bench_shared sh = { .pin_period = pin_period, .impl = impl };\n+\tunsigned int n, i, j, max_pinned = pin_period ? DIV_ROUND_UP(CHURN_ITER, pin_period) : 0;\n+\tunsigned long ops = 0, live = 0;\n+\tunsigned int nr_alloc;\n+\tstruct bench_thread *t;\n+\tu64 ns = 0;\n+\tint *cpus;\n+\tint ret;\n+\n+\tcpus = kunit_kcalloc(test, nr_cpu_ids, sizeof(*cpus), GFP_KERNEL);\n+\tt = bench_kvzalloc(test, array_size(nr_cpu_ids, sizeof(*t)));\n+\tKUNIT_ASSERT_NOT_NULL(test, cpus);\n+\tKUNIT_ASSERT_NOT_NULL(test, t);\n+\t/* Allocate before taking cpus_read_lock(), as in bench_update(). */\n+\tnr_alloc = all ? num_online_cpus() : 1;\n+\tfor (i = 0; i \u003c nr_alloc; i++) {\n+\t\tt[i].pinned = bench_kvzalloc(test, array_size(max_pinned * LIFECYCLE_NR + 1,\n+\t\t\t\t\t\t\t sizeof(*t[i].pinned)));\n+\t\tt[i].pinned_pc = bench_kvzalloc(test, array_size(max_pinned * LIFECYCLE_NR + 1,\n+\t\t\t\t\t\t\t\t sizeof(*t[i].pinned_pc)));\n+\t\tKUNIT_ASSERT_NOT_NULL(test, t[i].pinned_pc);\n+\t\tKUNIT_ASSERT_NOT_NULL(test, t[i].pinned);\n+\t}\n+\tcpus_read_lock();\n+\tn = bench_cpus(cpus, nr_alloc);\n+\tret = bench_start(\u0026sh, t, cpus, n);\n+\tif (ret) {\n+\t\tcpus_read_unlock();\n+\t\tKUNIT_FAIL(test, \"cannot start the benchmark threads: %d\", ret);\n+\t\tgoto destroy;\n+\t}\n+\tWRITE_ONCE(sh.go, true);\n+\tbench_stop(t, n);\n+\tcpus_read_unlock();\n+\tfor (i = 0; i \u003c n; i++) {\n+\t\tops += t[i].ops;\n+\t\tns += t[i].ns;\n+\t\tlive += t[i].nr_pinned * LIFECYCLE_NR;\n+\t}\n+\tKUNIT_EXPECT_EQ(test, ops, (unsigned long)n * CHURN_ITER);\n+\tkunit_info(test, \"churn pin 1/%-4u %3u thr %8llu ns/op live %6lu %s\\n\",\n+\t\t pin_period, n, ops ? div64_ul(ns, ops) : 0, live, impl_name[impl]);\n+destroy:\n+\tfor (i = 0; i \u003c nr_alloc; i++) {\n+\t\tfor (j = 0; j \u003c t[i].nr_pinned; j++) {\n+\t\t\tif (impl == IMPL_PERCPU_COUNTER)\n+\t\t\t\tpercpu_counter_destroy_many(\u0026t[i].pinned_pc[j * LIFECYCLE_NR],\n+\t\t\t\t\t\t\t LIFECYCLE_NR);\n+\t\t\telse\n+\t\t\t\tpercpu_counter_tree_destroy_many(\u0026t[i].pinned[j * LIFECYCLE_NR],\n+\t\t\t\t\t\t\t\t LIFECYCLE_NR);\n+\t\t}\n+\t\tbench_kvfree_now(test, t[i].pinned_pc);\n+\t\tbench_kvfree_now(test, t[i].pinned);\n+\t}\n+\tbench_kvfree_now(test, t);\n+\tkunit_kfree(test, cpus);\n+}\n+\n+static void hpcc_bench(struct kunit *test)\n+{\n+\tstatic const unsigned int pin_periods[] = { 0, 256, 64, 16, 4 };\n+\tstruct percpu_counter_tree probe;\n+\tunsigned int p;\n+\tint wl;\n+\n+\tif (batch \u003c 2 || !is_power_of_2(batch) || batch \u003e PERCPU_COUNTER_TREE_MAX_BATCH)\n+\t\tkunit_skip(test, \"invalid batch size %u\", batch);\n+\tKUNIT_ASSERT_EQ(test, 0, percpu_counter_tree_init(\u0026probe, batch, GFP_KERNEL));\n+\tkunit_info(test, \"%u CPUs online, nr_cpu_ids %u\\n\", num_online_cpus(), nr_cpu_ids);\n+\tkunit_info(test, \"%s batch %u, accuracy +-%lu\\n\", impl_name[IMPL_TREE], batch,\n+\t\t percpu_counter_tree_approximate_accuracy(\u0026probe));\n+\tkunit_info(test, \"%s batch %d, accuracy +-%lu\\n\", impl_name[IMPL_PERCPU_COUNTER],\n+\t\t percpu_counter_batch, (unsigned long)nr_cpu_ids * (percpu_counter_batch - 1));\n+\tpercpu_counter_tree_destroy(\u0026probe);\n+\tbench_lifecycle(test, false, IMPL_TREE, false);\n+\tbench_lifecycle(test, false, IMPL_TREE, true);\n+\tbench_lifecycle(test, false, IMPL_PERCPU_COUNTER, false);\n+\tbench_lifecycle(test, true, IMPL_TREE, false);\n+\tbench_lifecycle(test, true, IMPL_TREE, true);\n+\tbench_lifecycle(test, true, IMPL_PERCPU_COUNTER, false);\n+\tfor (p = 0; p \u003c ARRAY_SIZE(pin_periods); p++) {\n+\t\tbench_churn(test, false, pin_periods[p], IMPL_TREE);\n+\t\tbench_churn(test, false, pin_periods[p], IMPL_PERCPU_COUNTER);\n+\t\tbench_churn(test, true, pin_periods[p], IMPL_TREE);\n+\t\tbench_churn(test, true, pin_periods[p], IMPL_PERCPU_COUNTER);\n+\t}\n+\tfor (wl = 0; wl \u003c NR_W; wl++) {\n+\t\tbench_update(test, BENCH_SHARED, wl, IMPL_TREE);\n+\t\tbench_update(test, BENCH_SHARED, wl, IMPL_PERCPU_COUNTER);\n+\t\tbench_update(test, BENCH_PRIVATE, wl, IMPL_TREE);\n+\t\tbench_update(test, BENCH_PRIVATE, wl, IMPL_PERCPU_COUNTER);\n+\t\tbench_update(test, BENCH_MM, wl, IMPL_TREE);\n+\t\tbench_update(test, BENCH_MM, wl, IMPL_PERCPU_COUNTER);\n+\t\tbench_update(test, BENCH_SINGLE, wl, IMPL_TREE);\n+\t}\n+}\n+\n+static struct kunit_case hpcc_bench_cases[] = {\n+\tKUNIT_CASE_SLOW(hpcc_bench),\n+\t{}\n+};\n+\n+static struct kunit_suite hpcc_bench_suite = {\n+\t.name = \"percpu_counter_tree_bench\",\n+\t.test_cases = hpcc_bench_cases,\n+};\n+\n+kunit_test_suite(hpcc_bench_suite);\n+\n+MODULE_DESCRIPTION(\"Benchmark for hierarchical per-CPU counters\");\n+MODULE_LICENSE(\"Dual MIT/GPL\");\n+MODULE_IMPORT_NS(\"EXPORTED_FOR_KUNIT_TESTING\");\ndiff --git a/lib/tests/percpu_counter_tree_kunit.c b/lib/tests/percpu_counter_tree_kunit.c\nnew file mode 100644\nindex 00000000000000..2f5852b1f52140\n--- /dev/null\n+++ b/lib/tests/percpu_counter_tree_kunit.c\n@@ -0,0 +1,1050 @@\n+// SPDX-License-Identifier: GPL-2.0+ OR MIT\n+// SPDX-FileCopyrightText: 2026 Mathieu Desnoyers \u003cmathieu.desnoyers@efficios.com\u003e\n+\n+#include \u003ckunit/test.h\u003e\n+#include \u003clinux/delay.h\u003e\n+#include \u003clinux/mutex.h\u003e\n+#include \u003clinux/percpu_counter_tree.h\u003e\n+#include \u003clinux/kthread.h\u003e\n+#include \u003clinux/wait.h\u003e\n+#include \u003clinux/random.h\u003e\n+#include \u003clinux/cpuhplock.h\u003e\n+#include \u003clinux/slab.h\u003e\n+#include \u003clinux/sched/task.h\u003e\n+#include \u003clinux/workqueue.h\u003e\n+\n+struct multi_thread_test_data {\n+\tstruct work_struct work;\n+\tlong increment;\n+\tunsigned int nr_inc;\n+\tint counter_index;\n+};\n+\n+#define NR_COUNTERS\t2\n+\n+/* Hierarchical per-CPU counter instances. */\n+static struct percpu_counter_tree counter[NR_COUNTERS];\n+\n+/* Global atomic counters for validation. */\n+static atomic_long_t global_counter[NR_COUNTERS];\n+\n+static DECLARE_WAIT_QUEUE_HEAD(kernel_threads_wq);\n+static atomic_t kernel_threads_to_run;\n+\n+static void complete_work(void)\n+{\n+\tif (atomic_dec_and_test(\u0026kernel_threads_to_run))\n+\t\twake_up(\u0026kernel_threads_wq);\n+}\n+\n+/* Random tasks test. */\n+#define NR_RAND_TASKS\t1000\n+\n+static void hpcc_print_info(struct kunit *test)\n+{\n+\tkunit_info(test, \"Running test with %d CPUs\\n\", num_online_cpus());\n+}\n+\n+static void add_to_counter(int counter_index, unsigned int nr_inc, long increment)\n+{\n+\tunsigned int i;\n+\n+\tfor (i = 0; i \u003c nr_inc; i++) {\n+\t\tpercpu_counter_tree_add(\u0026counter[counter_index], increment);\n+\t\tatomic_long_add(increment, \u0026global_counter[counter_index]);\n+\t}\n+}\n+\n+/* Check the residuals and the carries of a settled counter. */\n+static void check_residuals(struct kunit *test, struct percpu_counter_tree *c)\n+{\n+#ifdef CONFIG_SMP\n+\tKUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_check_residuals(c));\n+#endif\n+}\n+\n+static void check_counters(struct kunit *test)\n+{\n+\tint counter_index;\n+\n+\t/* Compare each counter with its global counter. */\n+\tfor (counter_index = 0; counter_index \u003c NR_COUNTERS; counter_index++) {\n+\t\tstruct percpu_counter_tree *c = \u0026counter[counter_index];\n+\t\tlong v = atomic_long_read(\u0026global_counter[counter_index]);\n+\t\tlong approx_sum = percpu_counter_tree_approximate_sum(c);\n+\t\tunsigned long accuracy;\n+\t\tlong precise_min, precise_max;\n+\n+\t\t/* Precise comparison. */\n+\t\tKUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(c), v);\n+\t\tcheck_residuals(test, c);\n+\t\tKUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_precise_compare_value(c, v));\n+\n+\t\t/* Approximate comparison. */\n+\t\tKUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare_value(c, v));\n+\n+\t\t/* Accuracy limits checks. */\n+\t\taccuracy = percpu_counter_tree_approximate_accuracy(c);\n+\n+\t\t/*\n+\t\t * The accuracy is an unsigned long, so the inner expression\n+\t\t * promotes to unsigned. Compute the difference in unsigned and\n+\t\t * cast to signed for the comparison against 0.\n+\t\t */\n+\t\tKUNIT_EXPECT_GE(test, (long)(approx_sum - (v - accuracy)), 0);\n+\t\tKUNIT_EXPECT_LE(test, (long)(approx_sum - (v + accuracy)), 0);\n+\n+\t\t/* Precise min/max range check. */\n+\t\tpercpu_counter_tree_approximate_min_max_range(approx_sum, accuracy,\n+\t\t\t\t\t\t\t \u0026precise_min, \u0026precise_max);\n+\n+\t\tKUNIT_EXPECT_GE(test, v - precise_min, 0);\n+\t\tKUNIT_EXPECT_LE(test, v - precise_max, 0);\n+\t}\n+\t/* Compare each counter with the second counter. */\n+\tKUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(\u0026counter[0]),\n+\t\t\tpercpu_counter_tree_precise_sum(\u0026counter[1]));\n+\tKUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_precise_compare(\u0026counter[0], \u0026counter[1]));\n+\tKUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare(\u0026counter[0], \u0026counter[1]));\n+}\n+\n+/*\n+ * Workqueue running the updates of the multi-threaded tests. Its workers\n+ * are reused across rounds, which keeps the tests fast on machines where\n+ * creating threads on remote CPUs is slow. Work items are CPU intensive,\n+ * so several of them queued on one CPU run concurrently in separate\n+ * workers, and can preempt each other in the middle of an update.\n+ */\n+static struct workqueue_struct *hpcc_test_wq;\n+\n+static void multi_thread_work_fn(struct work_struct *work)\n+{\n+\tstruct multi_thread_test_data *td =\n+\t\tcontainer_of(work, struct multi_thread_test_data, work);\n+\n+\tadd_to_counter(td-\u003ecounter_index, td-\u003enr_inc, td-\u003eincrement);\n+\tkfree(td);\n+\tcomplete_work();\n+}\n+\n+/* Queue an update of a counter on @target_cpu. Call with the CPU hotplug lock held. */\n+static int test_queue_on_cpu(int target_cpu, int counter_index, unsigned int nr_inc,\n+\t\t\t long increment)\n+{\n+\tstruct multi_thread_test_data *td = kzalloc(sizeof(*td), GFP_KERNEL);\n+\n+\tif (!td)\n+\t\treturn -ENOMEM;\n+\tINIT_WORK(\u0026td-\u003ework, multi_thread_work_fn);\n+\ttd-\u003eincrement = increment;\n+\ttd-\u003enr_inc = nr_inc;\n+\ttd-\u003ecounter_index = counter_index;\n+\tatomic_inc(\u0026kernel_threads_to_run);\n+\tqueue_work_on(target_cpu, hpcc_test_wq, \u0026td-\u003ework);\n+\treturn 0;\n+}\n+\n+static void init_kthreads(void)\n+{\n+\tatomic_set(\u0026kernel_threads_to_run, 1);\n+}\n+\n+/*\n+ * The caller needs to wait for work completion _before_ stopping kthreads.\n+ */\n+static void fini_kthreads(void)\n+{\n+\t/* Release our own reference. */\n+\tcomplete_work();\n+\t/* Wait for all others threads to run. */\n+\twait_event(kernel_threads_wq, (atomic_read(\u0026kernel_threads_to_run) == 0));\n+}\n+\n+static int init_counters(unsigned long batch_size)\n+{\n+\tint i, ret;\n+\n+\tret = percpu_counter_tree_init_many(counter, NR_COUNTERS, batch_size, GFP_KERNEL);\n+\tif (ret)\n+\t\treturn ret;\n+\tfor (i = 0; i \u003c NR_COUNTERS; i++)\n+\t\tatomic_long_set(\u0026global_counter[i], 0);\n+\treturn 0;\n+}\n+\n+static void fini_counters(void)\n+{\n+\tpercpu_counter_tree_destroy_many(counter, NR_COUNTERS);\n+}\n+\n+enum up_test_inc_type {\n+\tINC_ONE,\n+\tINC_MINUS_ONE,\n+\tINC_RANDOM,\n+};\n+\n+/*\n+ * Single-threaded tests. Those run the updates on various CPUs, one at a\n+ * time: work_on_cpu() waits for each update to complete before the next,\n+ * effectively making sure there are no concurrent updates. Each test\n+ * stops early after SINGLE_THREAD_MAX_SECONDS, because cross-CPU wakeups\n+ * can be slow on large virtual machines.\n+ */\n+#define SINGLE_THREAD_MAX_SECONDS\t10\n+\n+struct single_thread_update {\n+\tlong increment;\n+\tint counter_index;\n+};\n+\n+static long single_thread_update_fn(void *arg)\n+{\n+\tstruct single_thread_update *u = arg;\n+\n+\tadd_to_counter(u-\u003ecounter_index, 1, u-\u003eincrement);\n+\treturn 0;\n+}\n+\n+static void do_hpcc_test_single_thread(struct kunit *test, int _cpu0, int _cpu1,\n+\t\t\t\t enum up_test_inc_type type)\n+{\n+\tunsigned long batch_size_order = 5;\n+\tint cpu0 = _cpu0;\n+\tint cpu1 = _cpu1;\n+\tunsigned long deadline = jiffies + SINGLE_THREAD_MAX_SECONDS * HZ;\n+\tint ret, i;\n+\n+\tret = init_counters(1UL \u003c\u003c batch_size_order);\n+\tKUNIT_EXPECT_EQ(test, ret, 0);\n+\tif (ret)\n+\t\treturn;\n+\tcpus_read_lock();\n+\tfor (i = 0; i \u003c 10000 \u0026\u0026 time_before(jiffies, deadline); i++) {\n+\t\tstruct single_thread_update u0 = { .counter_index = 0 };\n+\t\tstruct single_thread_update u1 = { .counter_index = 1 };\n+\t\tlong increment;\n+\n+\t\tswitch (type) {\n+\t\tcase INC_ONE:\n+\t\t\tincrement = 1;\n+\t\t\tbreak;\n+\t\tcase INC_MINUS_ONE:\n+\t\t\tincrement = -1;\n+\t\t\tbreak;\n+\t\tcase INC_RANDOM:\n+\t\t\tincrement = (long)get_random_long() % 50000;\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\tWARN_ON_ONCE(1);\n+\t\t\tincrement = 0;\n+\t\t}\n+\t\tif (_cpu0 \u003c 0)\n+\t\t\tcpu0 = cpumask_any_distribute(cpu_online_mask);\n+\t\tif (_cpu1 \u003c 0)\n+\t\t\tcpu1 = cpumask_any_distribute(cpu_online_mask);\n+\t\tu0.increment = increment;\n+\t\tu1.increment = increment;\n+\t\twork_on_cpu(cpu0, single_thread_update_fn, \u0026u0);\n+\t\twork_on_cpu(cpu1, single_thread_update_fn, \u0026u1);\n+\t\tcheck_counters(test);\n+\t}\n+\tcpus_read_unlock();\n+\tfini_counters();\n+}\n+\n+static void hpcc_test_single_thread_first(struct kunit *test)\n+{\n+\tint cpu = cpumask_first(cpu_online_mask);\n+\n+\tdo_hpcc_test_single_thread(test, cpu, cpu, INC_ONE);\n+\tdo_hpcc_test_single_thread(test, cpu, cpu, INC_MINUS_ONE);\n+\tdo_hpcc_test_single_thread(test, cpu, cpu, INC_RANDOM);\n+}\n+\n+static void hpcc_test_single_thread_first_random(struct kunit *test)\n+{\n+\tint cpu = cpumask_first(cpu_online_mask);\n+\n+\tdo_hpcc_test_single_thread(test, cpu, -1, INC_ONE);\n+\tdo_hpcc_test_single_thread(test, cpu, -1, INC_MINUS_ONE);\n+\tdo_hpcc_test_single_thread(test, cpu, -1, INC_RANDOM);\n+}\n+\n+static void hpcc_test_single_thread_random(struct kunit *test)\n+{\n+\tdo_hpcc_test_single_thread(test, -1, -1, INC_ONE);\n+\tdo_hpcc_test_single_thread(test, -1, -1, INC_MINUS_ONE);\n+\tdo_hpcc_test_single_thread(test, -1, -1, INC_RANDOM);\n+}\n+\n+/* Multi-threaded SMP tests. */\n+\n+/*\n+ * Bound the number of CPUs running workers in the multi-threaded tests,\n+ * so their run time does not grow with the size of the machine. Beyond\n+ * this bound, the workers run on a subset spread evenly across the\n+ * online CPUs, so they still reach every branch of the tree.\n+ */\n+#define MAX_WORKER_CPUS\t64\n+\n+/*\n+ * Each multi-threaded test stops early after MULTI_THREAD_MAX_SECONDS,\n+ * because cross-CPU wakeups can be slow on large virtual machines.\n+ */\n+#define MULTI_THREAD_MAX_SECONDS\t60\n+\n+static struct cpumask worker_cpus;\n+\n+/* Select the CPUs running workers. Call with the CPU hotplug lock held. */\n+static const struct cpumask *select_worker_cpus(void)\n+{\n+\tunsigned int nr_online = num_online_cpus(), i = 0;\n+\tint cpu;\n+\n+\tcpumask_clear(\u0026worker_cpus);\n+\tfor_each_online_cpu(cpu) {\n+\t\t/* Pick MAX_WORKER_CPUS of the nr_online CPUs, evenly spaced. */\n+\t\tif ((i + 1) * MAX_WORKER_CPUS / nr_online !=\n+\t\t i * MAX_WORKER_CPUS / nr_online)\n+\t\t\tcpumask_set_cpu(cpu, \u0026worker_cpus);\n+\t\ti++;\n+\t}\n+\treturn \u0026worker_cpus;\n+}\n+\n+/*\n+ * CPU selection for the second counter's thread, relative to the CPU\n+ * running the first counter's thread.\n+ */\n+enum mt_cpu_select {\n+\tMT_SAME_CPU,\t\t/* Same CPU as the first counter. */\n+\tMT_EVEN_CPU,\t\t/* Funnel CPU pairs (0, 1), (2, 3), ... to 0, 2, ... */\n+\tMT_FIRST_CPU,\t\t/* First online CPU. */\n+\tMT_RANDOM_CPU,\t\t/* Any online CPU. */\n+};\n+\n+/* Return the CPU for the second counter's thread, or -1 to skip @cpu. */\n+static int mt_select_cpu(enum mt_cpu_select select, int cpu)\n+{\n+\tint even_cpu;\n+\n+\tswitch (select) {\n+\tcase MT_SAME_CPU:\n+\t\treturn cpu;\n+\tcase MT_EVEN_CPU:\n+\t\teven_cpu = cpu \u0026 ~1;\n+\t\treturn cpumask_test_cpu(even_cpu, cpu_online_mask) ? even_cpu : -1;\n+\tcase MT_FIRST_CPU:\n+\t\treturn cpumask_first(cpu_online_mask);\n+\tcase MT_RANDOM_CPU:\n+\t\treturn cpumask_any_distribute(cpu_online_mask);\n+\t}\n+\tWARN_ON_ONCE(1);\n+\treturn -1;\n+}\n+\n+static void do_hpcc_multi_thread_increment(struct kunit *test, enum mt_cpu_select select,\n+\t\t\t\t\t unsigned long batch_size, unsigned int nr_inc,\n+\t\t\t\t\t long increment)\n+{\n+\tconst struct cpumask *cpus;\n+\tint ret, cpu;\n+\n+\tret = init_counters(batch_size);\n+\tKUNIT_EXPECT_EQ(test, ret, 0);\n+\tif (ret)\n+\t\treturn;\n+\tinit_kthreads();\n+\tcpus_read_lock();\n+\tcpus = select_worker_cpus();\n+\tfor_each_cpu(cpu, cpus) {\n+\t\tint cpu1 = mt_select_cpu(select, cpu);\n+\n+\t\tif (cpu1 \u003c 0)\n+\t\t\tcontinue;\n+\t\tret = test_queue_on_cpu(cpu, 0, nr_inc, increment);\n+\t\tKUNIT_EXPECT_EQ(test, ret, 0);\n+\t\tif (ret)\n+\t\t\tbreak;\n+\t\tret = test_queue_on_cpu(cpu1, 1, nr_inc, increment);\n+\t\tKUNIT_EXPECT_EQ(test, ret, 0);\n+\t\tif (ret)\n+\t\t\tbreak;\n+\t}\n+\tfini_kthreads();\n+\tcpus_read_unlock();\n+\tcheck_counters(test);\n+\tfini_counters();\n+}\n+\n+static void hpcc_test_multi_thread_batch_increment(struct kunit *test)\n+{\n+\tunsigned long deadline = jiffies + MULTI_THREAD_MAX_SECONDS * HZ;\n+\tunsigned long batch_size_order;\n+\n+\tfor (batch_size_order = 2; batch_size_order \u003c= 7; batch_size_order++) {\n+\t\tunsigned long batch_size = 1UL \u003c\u003c batch_size_order;\n+\t\tunsigned int nr_inc;\n+\n+\t\tfor (nr_inc = 1; nr_inc \u003c 1024; nr_inc *= 2) {\n+\t\t\tlong increment;\n+\n+\t\t\tfor (increment = 1; increment \u003c 100000; increment *= 10) {\n+\t\t\t\tenum mt_cpu_select select;\n+\n+\t\t\t\tif (time_after(jiffies, deadline))\n+\t\t\t\t\treturn;\n+\t\t\t\tfor (select = MT_SAME_CPU; select \u003c= MT_RANDOM_CPU; select++)\n+\t\t\t\t\tdo_hpcc_multi_thread_increment(test, select, batch_size,\n+\t\t\t\t\t\t\t\t nr_inc, increment);\n+\t\t\t}\n+\t\t}\n+\t}\n+}\n+\n+static void hpcc_test_multi_thread_random_walk(struct kunit *test)\n+{\n+\tunsigned long deadline = jiffies + MULTI_THREAD_MAX_SECONDS * HZ;\n+\tunsigned long batch_size_order = 5;\n+\tint loop;\n+\n+\tfor (loop = 0; loop \u003c 100 \u0026\u0026 time_before(jiffies, deadline); loop++) {\n+\t\tint ret, i;\n+\n+\t\tret = init_counters(1UL \u003c\u003c batch_size_order);\n+\t\tKUNIT_EXPECT_EQ(test, ret, 0);\n+\t\tif (ret)\n+\t\t\treturn;\n+\t\tinit_kthreads();\n+\t\tcpus_read_lock();\n+\t\tfor (i = 0; i \u003c NR_RAND_TASKS; i++) {\n+\t\t\tlong increment = (long)get_random_long() % 512;\n+\t\t\tunsigned int nr_inc = ((unsigned long)get_random_long()) % 1024;\n+\n+\t\t\tret = test_queue_on_cpu(cpumask_any_distribute(cpu_online_mask),\n+\t\t\t\t\t\t0, nr_inc, increment);\n+\t\t\tKUNIT_EXPECT_EQ(test, ret, 0);\n+\t\t\tif (ret)\n+\t\t\t\tbreak;\n+\t\t\tret = test_queue_on_cpu(cpumask_any_distribute(cpu_online_mask),\n+\t\t\t\t\t\t1, nr_inc, increment);\n+\t\t\tKUNIT_EXPECT_EQ(test, ret, 0);\n+\t\t\tif (ret)\n+\t\t\t\tbreak;\n+\t\t}\n+\t\tfini_kthreads();\n+\t\tcpus_read_unlock();\n+\t\tcheck_counters(test);\n+\t\tfini_counters();\n+\t}\n+}\n+\n+#define CONTENTION_NR_ROUNDS\t20000\n+#define CONTENTION_NR_INC\t8\n+\n+struct contention_shared {\n+\tunsigned long batch_size;\n+\tatomic_t round;\t\t\t/* Round started by the coordinator. */\n+\tatomic_t nr_done;\t\t/* Rounds completed by the other workers. */\n+\tunsigned int nr_workers;\n+\tunsigned int nr_errors;\t\t/* Rounds ending with invalid residuals. */\n+\tbool started;\n+};\n+\n+struct contention_data {\n+\tstruct contention_shared *sh;\n+\tlong sum;\n+\tu32 seed;\n+\tbool coordinator;\n+} ____cacheline_aligned_in_smp;\n+\n+static DECLARE_WAIT_QUEUE_HEAD(contention_wq);\n+\n+/*\n+ * Add random multiples of the batch size, of either sign, so each\n+ * addition carries into the counters above level 0. Increments of\n+ * opposite signs and of more than one batch size exercise concurrent\n+ * carry extractions retrying after a concurrent update of the residual.\n+ */\n+static void contention_add(struct contention_data *cd)\n+{\n+\tunsigned int i;\n+\n+\tfor (i = 0; i \u003c CONTENTION_NR_INC; i++) {\n+\t\tu32 x = cd-\u003eseed;\n+\t\tlong inc;\n+\n+\t\t/* xorshift32: get_random_u32() would dominate the loop. */\n+\t\tx ^= x \u003c\u003c 13;\n+\t\tx ^= x \u003e\u003e 17;\n+\t\tx ^= x \u003c\u003c 5;\n+\t\tcd-\u003eseed = x;\n+\t\tinc = (long)((x \u0026 15) + 1) * cd-\u003esh-\u003ebatch_size;\n+\t\tif (x \u0026 16)\n+\t\t\tinc = -inc;\n+\t\tpercpu_counter_tree_add(\u0026counter[0], inc);\n+\t\tcd-\u003esum += inc;\n+\t}\n+}\n+\n+/*\n+ * The workers run rounds of a few additions each, all starting together.\n+ * Between rounds, the coordinator checks the residuals of the settled\n+ * counter: an invalid residual left by the last updaters of a round would\n+ * otherwise be repaired by the next additions.\n+ */\n+static int contention_worker_fn(void *data)\n+{\n+\tstruct contention_data *cd = data;\n+\tstruct contention_shared *sh = cd-\u003esh;\n+\tint round;\n+\n+\twait_event(contention_wq, READ_ONCE(sh-\u003estarted));\n+\tfor (round = 1; round \u003c= CONTENTION_NR_ROUNDS; round++) {\n+\t\tif (cd-\u003ecoordinator) {\n+\t\t\tatomic_set_release(\u0026sh-\u003eround, round);\n+\t\t\tcontention_add(cd);\n+\t\t\twhile (atomic_read_acquire(\u0026sh-\u003enr_done) \u003c round * (sh-\u003enr_workers - 1))\n+\t\t\t\tcond_resched();\n+#ifdef CONFIG_SMP\n+\t\t\tif (percpu_counter_tree_check_residuals(\u0026counter[0]))\n+\t\t\t\tsh-\u003enr_errors++;\n+#endif\n+\t\t} else {\n+\t\t\twhile (atomic_read_acquire(\u0026sh-\u003eround) \u003c round)\n+\t\t\t\tcond_resched();\n+\t\t\tcontention_add(cd);\n+\t\t\t/* Order the additions before the coordinator check. */\n+\t\t\tatomic_inc_return_release(\u0026sh-\u003enr_done);\n+\t\t}\n+\t}\n+\tcomplete_work();\n+\treturn 0;\n+}\n+\n+static void do_hpcc_test_contended_carries(struct kunit *test, unsigned long batch_size)\n+{\n+\tstruct percpu_counter_tree *c = \u0026counter[0];\n+\tstruct contention_shared sh = { .batch_size = batch_size };\n+\tstruct contention_data *cd;\n+\tstruct task_struct **task;\n+\tconst struct cpumask *cpus;\n+\tunsigned long accuracy;\n+\tlong expected = 0, approx;\n+\tbool first = true;\n+\tint ret, cpu;\n+\n+\tcd = kvcalloc(nr_cpu_ids, sizeof(*cd), GFP_KERNEL);\n+\ttask = kvcalloc(nr_cpu_ids, sizeof(*task), GFP_KERNEL);\n+\tKUNIT_EXPECT_NOT_NULL(test, cd);\n+\tKUNIT_EXPECT_NOT_NULL(test, task);\n+\tif (!cd || !task)\n+\t\tgoto free;\n+\tret = init_counters(batch_size);\n+\tKUNIT_EXPECT_EQ(test, ret, 0);\n+\tif (ret)\n+\t\tgoto free;\n+\tinit_kthreads();\n+\tcpus_read_lock();\n+\tcpus = select_worker_cpus();\n+\tfor_each_cpu(cpu, cpus) {\n+\t\tcd[cpu].sh = \u0026sh;\n+\t\tcd[cpu].coordinator = first;\n+\t\tcd[cpu].seed = get_random_u32() | 1;\n+\t\ttask[cpu] = kthread_create_on_cpu(contention_worker_fn, \u0026cd[cpu], cpu,\n+\t\t\t\t\t\t \"kunit_hpcc_contention/%u\");\n+\t\tKUNIT_EXPECT_FALSE(test, IS_ERR(task[cpu]));\n+\t\tif (IS_ERR(task[cpu]))\n+\t\t\tbreak;\n+\t\tatomic_inc(\u0026kernel_threads_to_run);\n+\t\tget_task_struct(task[cpu]);\n+\t\twake_up_process(task[cpu]);\n+\t\tsh.nr_workers++;\n+\t\tfirst = false;\n+\t}\n+\t/* Start the workers once their number is known. */\n+\tWRITE_ONCE(sh.started, true);\n+\twake_up_all(\u0026contention_wq);\n+\tfini_kthreads();\n+\tfor_each_cpu(cpu, cpus) {\n+\t\tif (IS_ERR_OR_NULL(task[cpu]))\n+\t\t\tcontinue;\n+\t\tkthread_stop_put(task[cpu]);\n+\t\texpected += cd[cpu].sum;\n+\t}\n+\tcpus_read_unlock();\n+\n+\tkunit_info(test, \"batch_size=%lu workers=%u rounds with invalid residuals=%u\\n\",\n+\t\t batch_size, sh.nr_workers, sh.nr_errors);\n+\tKUNIT_EXPECT_EQ(test, sh.nr_errors, 0);\n+\tKUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(c), expected);\n+\tcheck_residuals(test, c);\n+\tapprox = percpu_counter_tree_approximate_sum(c);\n+\taccuracy = percpu_counter_tree_approximate_accuracy(c);\n+\tKUNIT_EXPECT_GE(test, approx, expected - (long)accuracy);\n+\tKUNIT_EXPECT_LE(test, approx, expected + (long)accuracy);\n+\tfini_counters();\n+free:\n+\tkvfree(task);\n+\tkvfree(cd);\n+}\n+\n+/*\n+ * Concurrent carries from the worker CPUs into the counters above level 0\n+ * which they share, with a check that the residuals are back within range\n+ * and that no carry was lost or duplicated after each round.\n+ */\n+static void hpcc_test_contended_carries(struct kunit *test)\n+{\n+\tunsigned long batch_size_order;\n+\n+\tif (num_online_cpus() \u003c 2)\n+\t\tkunit_skip(test, \"needs at least 2 online CPUs\");\n+\tfor (batch_size_order = 1; batch_size_order \u003c= 5; batch_size_order += 2)\n+\t\tdo_hpcc_test_contended_carries(test, 1UL \u003c\u003c batch_size_order);\n+}\n+\n+static void hpcc_test_init_one(struct kunit *test)\n+{\n+\tstruct percpu_counter_tree pct;\n+\tint ret;\n+\n+\tret = percpu_counter_tree_init(\u0026pct, 32, GFP_KERNEL);\n+\tKUNIT_EXPECT_EQ(test, ret, 0);\n+\tif (ret)\n+\t\treturn;\n+\n+\tpercpu_counter_tree_destroy(\u0026pct);\n+}\n+\n+static void hpcc_test_set(struct kunit *test)\n+{\n+\tstatic long values[] = {\n+\t\t5, 100, 127, 128, 255, 256, 4095, 4096, 500000, 0,\n+\t\t-5, -100, -127, -128, -255, -256, -4095, -4096, -500000,\n+\t};\n+\tstruct percpu_counter_tree pct;\n+\tint i, ret;\n+\n+\tret = percpu_counter_tree_init(\u0026pct, 32, GFP_KERNEL);\n+\tKUNIT_EXPECT_EQ(test, ret, 0);\n+\tif (ret)\n+\t\treturn;\n+\n+\tfor (i = 0; i \u003c ARRAY_SIZE(values); i++) {\n+\t\tlong v = values[i];\n+\n+\t\tpercpu_counter_tree_set(\u0026pct, v);\n+\t\tKUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(\u0026pct), v);\n+\t\tKUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare_value(\u0026pct, v));\n+\n+\t\tpercpu_counter_tree_add(\u0026pct, v);\n+\t\tKUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(\u0026pct), 2 * v);\n+\t\tKUNIT_EXPECT_EQ(test, 0,\n+\t\t\t\tpercpu_counter_tree_approximate_compare_value(\u0026pct, 2 * v));\n+\n+\t\tpercpu_counter_tree_add(\u0026pct, -2 * v);\n+\t\tKUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(\u0026pct), 0);\n+\t\tKUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare_value(\u0026pct, 0));\n+\t}\n+\n+\tpercpu_counter_tree_destroy(\u0026pct);\n+}\n+\n+struct saturate_data {\n+\tstruct percpu_counter_tree *c, *probe;\n+\tlong inc;\n+\tunsigned long max_steps;\n+\tbool overrun;\n+};\n+\n+/* Copy the state of counter @src into counter @dst, of the same topology. */\n+static void copy_counter_state(struct percpu_counter_tree *dst,\n+\t\t\t struct percpu_counter_tree *src)\n+{\n+#ifdef CONFIG_SMP\n+\tpercpu_counter_tree_copy_state(dst, src);\n+#else\n+\tatomic_long_set(\u0026dst-\u003ecount, atomic_long_read(\u0026src-\u003ecount));\n+#endif\n+}\n+\n+/*\n+ * Add inc (1 or -1) until the next addition would propagate a carry to the\n+ * approximate sum. This leaves every tree item on the path from this CPU to\n+ * the root holding its largest residual in the direction of inc which has\n+ * not been propagated yet, without knowledge of the tree topology. Each\n+ * addition is first tried on the probe counter, which holds the same state:\n+ * reverting an addition which reached the approximate sum would not restore\n+ * the residuals subtracted on the way. The path from a CPU to the root\n+ * holds less than the accuracy bound: more additions mean that a carry was\n+ * lost.\n+ */\n+static long saturate_worker_fn(void *data)\n+{\n+\tstruct saturate_data *sd = data;\n+\tunsigned long steps;\n+\n+\tfor (steps = 0; steps \u003c sd-\u003emax_steps; steps++) {\n+\t\tlong approx = percpu_counter_tree_approximate_sum(sd-\u003eprobe);\n+\n+\t\tpercpu_counter_tree_add(sd-\u003eprobe, sd-\u003einc);\n+\t\tif (percpu_counter_tree_approximate_sum(sd-\u003eprobe) != approx) {\n+\t\t\tcopy_counter_state(sd-\u003eprobe, sd-\u003ec);\n+\t\t\tbreak;\n+\t\t}\n+\t\tpercpu_counter_tree_add(sd-\u003ec, sd-\u003einc);\n+\t}\n+\tif (steps == sd-\u003emax_steps)\n+\t\tsd-\u003eoverrun = true;\n+\treturn 0;\n+}\n+\n+static void do_hpcc_test_saturated_compare(struct kunit *test, unsigned long batch_size,\n+\t\t\t\t\t long inc)\n+{\n+\tstruct percpu_counter_tree *c = \u0026counter[0], *exact = \u0026counter[1], probe;\n+\tstruct saturate_data sd = { .c = c, .probe = \u0026probe, .inc = inc };\n+\tunsigned long accuracy;\n+\tlong approx, precise, precise_min, precise_max;\n+\tint ret, cpu;\n+\n+\tret = percpu_counter_tree_init(\u0026probe, batch_size, GFP_KERNEL);\n+\tKUNIT_EXPECT_EQ(test, ret, 0);\n+\tif (ret)\n+\t\treturn;\n+\tret = init_counters(batch_size);\n+\tKUNIT_EXPECT_EQ(test, ret, 0);\n+\tif (ret)\n+\t\tgoto destroy_probe;\n+\taccuracy = percpu_counter_tree_approximate_accuracy(c);\n+\tsd.max_steps = batch_size + accuracy;\n+\tcpus_read_lock();\n+\tfor_each_online_cpu(cpu) {\n+\t\twork_on_cpu(cpu, saturate_worker_fn, \u0026sd);\n+\t\tKUNIT_EXPECT_FALSE(test, sd.overrun);\n+\t\tif (sd.overrun)\n+\t\t\tbreak;\n+\t}\n+\tcpus_read_unlock();\n+\n+\tprecise = percpu_counter_tree_precise_sum(c);\n+\tapprox = percpu_counter_tree_approximate_sum(c);\n+\tkunit_info(test, \"batch_size=%lu inc=%ld precise=%ld approx=%ld accuracy=%lu\\n\",\n+\t\t batch_size, inc, precise, approx, accuracy);\n+\n+\tcheck_residuals(test, c);\n+\tcheck_residuals(test, \u0026probe);\n+\n+\t/* Accuracy contract. */\n+\tKUNIT_EXPECT_GE(test, approx, precise - (long)accuracy);\n+\tKUNIT_EXPECT_LE(test, approx, precise + (long)accuracy);\n+\t/* Saturating every possible CPU reaches the accuracy exactly. */\n+\tif (num_online_cpus() == nr_cpu_ids)\n+\t\tKUNIT_EXPECT_EQ(test, (precise - approx) * inc, (long)accuracy);\n+\tpercpu_counter_tree_approximate_min_max_range(approx, accuracy,\n+\t\t\t\t\t\t \u0026precise_min, \u0026precise_max);\n+\tKUNIT_EXPECT_GE(test, precise, precise_min);\n+\tKUNIT_EXPECT_LE(test, precise, precise_max);\n+\n+\t/* Comparisons against a value. */\n+\tKUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare_value(c, precise));\n+\tKUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_precise_compare_value(c, precise));\n+\tKUNIT_EXPECT_EQ(test, -1, percpu_counter_tree_precise_compare_value(c, precise - 1));\n+\tKUNIT_EXPECT_EQ(test, 1, percpu_counter_tree_precise_compare_value(c, precise + 1));\n+\n+\t/* Comparisons against a counter with the same precise sum, exactly approximated. */\n+\tpercpu_counter_tree_set(exact, precise);\n+\tKUNIT_EXPECT_EQ(test, percpu_counter_tree_approximate_sum(exact), precise);\n+\tKUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare(c, exact));\n+\tKUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare(exact, c));\n+\tKUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_precise_compare(c, exact));\n+\tKUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_precise_compare(exact, c));\n+\n+\tfini_counters();\n+destroy_probe:\n+\tpercpu_counter_tree_destroy(\u0026probe);\n+}\n+\n+/*\n+ * Saturate the unpropagated residuals of every tree item reachable from\n+ * online CPUs, upward and downward, which yields the largest distance\n+ * between the precise and approximate sums of a settled counter in each\n+ * direction, and check that the comparison\n+ * functions do not return a wrong definitive answer. Small batch sizes\n+ * are the most likely to expose margin errors, because the fan-out of\n+ * the tree levels can be larger than the batch size.\n+ */\n+static void hpcc_test_saturated_compare(struct kunit *test)\n+{\n+\tunsigned long batch_size_order;\n+\n+\tfor (batch_size_order = 1; batch_size_order \u003c= 5; batch_size_order++) {\n+\t\tdo_hpcc_test_saturated_compare(test, 1UL \u003c\u003c batch_size_order, 1);\n+\t\tdo_hpcc_test_saturated_compare(test, 1UL \u003c\u003c batch_size_order, -1);\n+\t}\n+}\n+\n+/* Check a counter against its expected sum: exact before activation, within range after. */\n+static void check_lazy_counter(struct kunit *test, struct percpu_counter_tree *c, long v)\n+{\n+\tlong approx_sum = percpu_counter_tree_approximate_sum(c);\n+\tunsigned long accuracy;\n+\n+\tKUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(c), v);\n+\tKUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_precise_compare_value(c, v));\n+\tKUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare_value(c, v));\n+\tcheck_residuals(test, c);\n+#ifdef CONFIG_SMP\n+\tif (!READ_ONCE(c-\u003elevel0)) {\n+\t\tKUNIT_EXPECT_EQ(test, approx_sum, v);\n+\t\treturn;\n+\t}\n+#endif\n+\taccuracy = percpu_counter_tree_approximate_accuracy(c);\n+\tKUNIT_EXPECT_GE(test, (long)(approx_sum - (v - accuracy)), 0);\n+\tKUNIT_EXPECT_LE(test, (long)(approx_sum - (v + accuracy)), 0);\n+}\n+\n+/*\n+ * Lazily initialized counters: exact sums before the tree is activated,\n+ * with owner and atomic updates, set(), idempotent activation, updates\n+ * after activation, and destruction of counters never activated.\n+ */\n+static void hpcc_test_lazy(struct kunit *test)\n+{\n+\tstruct percpu_counter_tree pct[2];\n+\tlong expect[2] = { 0, 0 };\n+\tint i, j;\n+\n+\tKUNIT_ASSERT_EQ(test, 0, percpu_counter_tree_init_many_lazy(pct, 2, 32));\n+#ifdef CONFIG_SMP\n+\tKUNIT_EXPECT_FALSE(test, percpu_counter_tree_active(\u0026pct[0]));\n+\tKUNIT_EXPECT_FALSE(test, percpu_counter_tree_active(\u0026pct[1]));\n+\t/* The level 0 counters are __percpu pointers: compare them as booleans. */\n+\tKUNIT_EXPECT_TRUE(test, !pct[0].level0);\n+\tKUNIT_EXPECT_TRUE(test, !pct[1].level0);\n+\tKUNIT_EXPECT_NULL(test, pct[0].approx_sum);\n+\tKUNIT_EXPECT_NULL(test, pct[1].approx_sum);\n+#endif\n+\tfor (j = 0; j \u003c 2; j++) {\n+\t\t/* j == 0: before activation; j == 1: after activation. */\n+\t\tfor (i = 0; i \u003c 10000; i++) {\n+\t\t\tlong inc = (long)(get_random_u32() % 1025) - 512;\n+\n+\t\t\tif (get_random_u32() \u0026 1)\n+\t\t\t\tpercpu_counter_tree_add_owner(\u0026pct[0], inc);\n+\t\t\telse\n+\t\t\t\tpercpu_counter_tree_add(\u0026pct[0], inc);\n+\t\t\texpect[0] += inc;\n+\t\t\tpercpu_counter_tree_add(\u0026pct[1], -inc);\n+\t\t\texpect[1] -= inc;\n+\t\t}\n+\t\tcheck_lazy_counter(test, \u0026pct[0], expect[0]);\n+\t\tcheck_lazy_counter(test, \u0026pct[1], expect[1]);\n+\t\tpercpu_counter_tree_set(\u0026pct[1], 12345);\n+\t\texpect[1] = 12345;\n+\t\tcheck_lazy_counter(test, \u0026pct[1], expect[1]);\n+\t\tif (j)\n+\t\t\tbreak;\n+\t\tKUNIT_ASSERT_EQ(test, 0, percpu_counter_tree_activate_many(pct, 2, GFP_KERNEL));\n+\t\tKUNIT_EXPECT_TRUE(test, percpu_counter_tree_active(\u0026pct[0]));\n+\t\tKUNIT_EXPECT_TRUE(test, percpu_counter_tree_active(\u0026pct[1]));\n+#ifdef CONFIG_SMP\n+\t\t{\n+\t\t\ts8 __percpu *level0 = pct[0].level0;\n+\n+\t\t\tKUNIT_EXPECT_TRUE(test, level0);\n+\t\t\tKUNIT_EXPECT_TRUE(test, pct[1].level0 == level0 + 1);\n+\t\t\tKUNIT_EXPECT_NOT_NULL(test, pct[0].approx_sum);\n+\t\t\tKUNIT_EXPECT_PTR_EQ(test, pct[1].approx_sum, pct[0].approx_sum + 1);\n+\t\t\tKUNIT_EXPECT_EQ(test, 0,\n+\t\t\t\t\tpercpu_counter_tree_activate_many(pct, 2, GFP_KERNEL));\n+\t\t\tKUNIT_EXPECT_TRUE(test, pct[0].level0 == level0);\n+\t\t}\n+#endif\n+\t\t/* The sums carry the updates made before activation. */\n+\t\tcheck_lazy_counter(test, \u0026pct[0], expect[0]);\n+\t\tcheck_lazy_counter(test, \u0026pct[1], expect[1]);\n+\t}\n+\tpercpu_counter_tree_destroy_many(pct, 2);\n+\n+\t/* Destroy counters which were never activated. */\n+\tKUNIT_ASSERT_EQ(test, 0, percpu_counter_tree_init_many_lazy(pct, 2, 32));\n+\tpercpu_counter_tree_add_owner(\u0026pct[0], 7);\n+\tpercpu_counter_tree_add(\u0026pct[1], -7);\n+\tcheck_lazy_counter(test, \u0026pct[0], 7);\n+\tcheck_lazy_counter(test, \u0026pct[1], -7);\n+\tpercpu_counter_tree_destroy_many(pct, 2);\n+}\n+\n+/*\n+ * Activation racing with updates: one owner and updaters on other CPUs add\n+ * to a lazily initialized counter. Each worker makes a first part of its\n+ * updates, reports their sum and waits. The test checks that the counter\n+ * is exact, then releases the workers and activates the tree while they\n+ * make the rest of their updates. The remote updaters also activate the\n+ * tree when released, serialized with the test and with each other by a\n+ * mutex, and check that the tree is active once their activation returns.\n+ * No update may be lost, whichever side of the activation it lands on.\n+ */\n+#define LAZY_RACE_NR_INC\t200000\n+#define LAZY_RACE_NR_FIRST\t(LAZY_RACE_NR_INC / 8)\n+#define LAZY_RACE_MAX_REMOTE\t8\n+\n+struct lazy_race_shared {\n+\tstruct percpu_counter_tree *c;\n+\tatomic_long_t first_sum;\t/* Sum of the updates made before the release. */\n+\tatomic_long_t expect;\t\t/* Sum of all updates. */\n+\tstruct mutex activate_lock;\t/* Serializes the activations. */\n+\tatomic_t paused;\t\t/* Workers waiting for the release. */\n+\tatomic_t activate_errors;\t/* Failed activations of the workers. */\n+\tbool go;\t\t\t/* Releases the workers. */\n+};\n+\n+struct lazy_race_work {\n+\tstruct work_struct work;\n+\tstruct lazy_race_shared *sh;\n+\tbool owner;\n+};\n+\n+static void lazy_race_fn(struct work_struct *work)\n+{\n+\tstruct lazy_race_work *w = container_of(work, struct lazy_race_work, work);\n+\tstruct lazy_race_shared *sh = w-\u003esh;\n+\tlong sum = 0;\n+\tint i;\n+\n+\tfor (i = 0; i \u003c LAZY_RACE_NR_INC; i++) {\n+\t\tlong inc = (long)(get_random_u32() % 65) - 32;\n+\n+\t\tif (i == LAZY_RACE_NR_FIRST) {\n+\t\t\t/* Fully ordered: the first updates are visible before the pause. */\n+\t\t\tatomic_long_add_return(sum, \u0026sh-\u003efirst_sum);\n+\t\t\tatomic_inc_return(\u0026sh-\u003epaused);\n+\t\t\twhile (!READ_ONCE(sh-\u003ego))\n+\t\t\t\tcond_resched();\n+\t\t\tif (!w-\u003eowner) {\n+\t\t\t\tmutex_lock(\u0026sh-\u003eactivate_lock);\n+\t\t\t\tif (percpu_counter_tree_activate_many(sh-\u003ec, 1, GFP_KERNEL) ||\n+\t\t\t\t !percpu_counter_tree_active(sh-\u003ec))\n+\t\t\t\t\tatomic_inc(\u0026sh-\u003eactivate_errors);\n+\t\t\t\tmutex_unlock(\u0026sh-\u003eactivate_lock);\n+\t\t\t}\n+\t\t}\n+\t\tif (w-\u003eowner)\n+\t\t\tpercpu_counter_tree_add_owner(sh-\u003ec, inc);\n+\t\telse\n+\t\t\tpercpu_counter_tree_add(sh-\u003ec, inc);\n+\t\tsum += inc;\n+\t\tif (!(i \u0026 1023))\n+\t\t\tcond_resched();\n+\t}\n+\tatomic_long_add(sum, \u0026sh-\u003eexpect);\n+\tkfree(w);\n+\tcomplete_work();\n+}\n+\n+static void hpcc_test_lazy_activation_race(struct kunit *test)\n+{\n+\tstruct percpu_counter_tree pct;\n+\tint round;\n+\n+\tif (num_online_cpus() \u003c 2)\n+\t\tkunit_skip(test, \"needs at least 2 online CPUs\");\n+\tfor (round = 0; round \u003c 4; round++) {\n+\t\tstruct lazy_race_shared sh = { .c = \u0026pct };\n+\t\tunsigned long timeout = jiffies + 10 * HZ;\n+\t\tint cpu, nr_workers = 0;\n+\n+\t\tKUNIT_ASSERT_EQ(test, 0, percpu_counter_tree_init_many_lazy(\u0026pct, 1, 32));\n+\t\tmutex_init(\u0026sh.activate_lock);\n+\t\tinit_kthreads();\n+\t\tcpus_read_lock();\n+\t\tfor_each_online_cpu(cpu) {\n+\t\t\tstruct lazy_race_work *w;\n+\n+\t\t\tif (nr_workers \u003e LAZY_RACE_MAX_REMOTE)\n+\t\t\t\tbreak;\n+\t\t\tw = kzalloc_obj(*w);\n+\t\t\tKUNIT_EXPECT_NOT_NULL(test, w);\n+\t\t\tif (!w)\n+\t\t\t\tbreak;\n+\t\t\tINIT_WORK(\u0026w-\u003ework, lazy_race_fn);\n+\t\t\tw-\u003esh = \u0026sh;\n+\t\t\t/* The first worker is the owner, the others update remotely. */\n+\t\t\tw-\u003eowner = !nr_workers;\n+\t\t\tatomic_inc(\u0026kernel_threads_to_run);\n+\t\t\tqueue_work_on(cpu, hpcc_test_wq, \u0026w-\u003ework);\n+\t\t\tnr_workers++;\n+\t\t}\n+\t\tcpus_read_unlock();\n+\t\twhile (atomic_read(\u0026sh.paused) \u003c nr_workers \u0026\u0026 time_before(jiffies, timeout))\n+\t\t\tusleep_range(100, 200);\n+\t\tsmp_mb();\t/* Read the counter after the first updates of the workers. */\n+\t\t/* Every worker, the owner included, has updates left for after the activation. */\n+\t\tKUNIT_EXPECT_EQ(test, atomic_read(\u0026sh.paused), nr_workers);\n+\t\t/* Before activation, the owner and atomic updates are exact. */\n+\t\tcheck_lazy_counter(test, \u0026pct, atomic_long_read(\u0026sh.first_sum));\n+\t\tWRITE_ONCE(sh.go, true);\n+\t\tmutex_lock(\u0026sh.activate_lock);\n+\t\tKUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_activate_many(\u0026pct, 1, GFP_KERNEL));\n+\t\tKUNIT_EXPECT_TRUE(test, percpu_counter_tree_active(\u0026pct));\n+\t\tmutex_unlock(\u0026sh.activate_lock);\n+\t\tfini_kthreads();\n+\t\tmutex_destroy(\u0026sh.activate_lock);\n+\t\tKUNIT_EXPECT_EQ(test, 0, atomic_read(\u0026sh.activate_errors));\n+\t\tcheck_lazy_counter(test, \u0026pct, atomic_long_read(\u0026sh.expect));\n+\t\tpercpu_counter_tree_destroy_many(\u0026pct, 1);\n+\t}\n+}\n+\n+static void hpcc_test_check_config(struct kunit *test)\n+{\n+#ifdef CONFIG_SMP\n+\tKUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_check_config());\n+#else\n+\tkunit_skip(test, \"The configuration table requires CONFIG_SMP\");\n+#endif\n+}\n+\n+static struct kunit_case hpcc_test_cases[] = {\n+\tKUNIT_CASE(hpcc_test_check_config),\n+\tKUNIT_CASE(hpcc_print_info),\n+\tKUNIT_CASE(hpcc_test_single_thread_first),\n+\tKUNIT_CASE(hpcc_test_single_thread_first_random),\n+\tKUNIT_CASE_SLOW(hpcc_test_single_thread_random),\n+\tKUNIT_CASE(hpcc_test_multi_thread_batch_increment),\n+\tKUNIT_CASE_SLOW(hpcc_test_multi_thread_random_walk),\n+\tKUNIT_CASE(hpcc_test_init_one),\n+\tKUNIT_CASE(hpcc_test_set),\n+\tKUNIT_CASE(hpcc_test_saturated_compare),\n+\tKUNIT_CASE(hpcc_test_contended_carries),\n+\tKUNIT_CASE(hpcc_test_lazy),\n+\tKUNIT_CASE(hpcc_test_lazy_activation_race),\n+\t{}\n+};\n+\n+static int hpcc_test_suite_init(struct kunit_suite *suite)\n+{\n+\thpcc_test_wq = alloc_workqueue(\"hpcc_test\", WQ_PERCPU | WQ_CPU_INTENSIVE, 0);\n+\treturn hpcc_test_wq ? 0 : -ENOMEM;\n+}\n+\n+static void hpcc_test_suite_exit(struct kunit_suite *suite)\n+{\n+\tdestroy_workqueue(hpcc_test_wq);\n+}\n+\n+static struct kunit_suite hpcc_test_suite = {\n+\t.name = \"percpu_counter_tree\",\n+\t.suite_init = hpcc_test_suite_init,\n+\t.suite_exit = hpcc_test_suite_exit,\n+\t.test_cases = hpcc_test_cases,\n+};\n+\n+kunit_test_suite(hpcc_test_suite);\n+\n+MODULE_DESCRIPTION(\"Test cases for hierarchical per-CPU counters\");\n+MODULE_LICENSE(\"Dual MIT/GPL\");\n+MODULE_IMPORT_NS(\"EXPORTED_FOR_KUNIT_TESTING\");\ndiff --git a/tools/testing/hpcc/.gitignore b/tools/testing/hpcc/.gitignore\nnew file mode 100644\nindex 00000000000000..f362a8a46868a5\n--- /dev/null\n+++ b/tools/testing/hpcc/.gitignore\n@@ -0,0 +1,2 @@\n+# SPDX-License-Identifier: GPL-2.0+ OR MIT\n+hpcc\ndiff --git a/tools/testing/hpcc/Makefile b/tools/testing/hpcc/Makefile\nnew file mode 100644\nindex 00000000000000..e4758643a3f0e5\n--- /dev/null\n+++ b/tools/testing/hpcc/Makefile\n@@ -0,0 +1,18 @@\n+# SPDX-License-Identifier: GPL-2.0+ OR MIT\n+\n+.PHONY: default clean\n+\n+default: hpcc\n+\n+CFLAGS += -O2 -g -Wall -I./include -I../../include -fno-strict-overflow -fno-strict-aliasing\n+CFLAGS += -DCONFIG_SMP -DCONFIG_KUNIT=1 -D'NR_CPUS=(1U \u003c\u003c 20)'\n+\n+DEPS = ../../../lib/percpu_counter_tree.c \\\n+\t../../../include/linux/percpu_counter_tree.h \\\n+\tinclude/hpcc_shim.h\n+\n+hpcc: main.c $(DEPS)\n+\t$(CC) $(CFLAGS) -o $@ main.c $(LDLIBS)\n+\n+clean:\n+\t$(RM) hpcc\ndiff --git a/tools/testing/hpcc/include/hpcc_shim.h b/tools/testing/hpcc/include/hpcc_shim.h\nnew file mode 100644\nindex 00000000000000..eaac72eab2d172\n--- /dev/null\n+++ b/tools/testing/hpcc/include/hpcc_shim.h\n@@ -0,0 +1,203 @@\n+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */\n+/*\n+ * Minimal userspace environment to build lib/percpu_counter_tree.c.\n+ *\n+ * The test is single-threaded: per-CPU data is an array indexed by the\n+ * simulated current CPU, and atomic operations are plain operations.\n+ */\n+#ifndef _HPCC_SHIM_H\n+#define _HPCC_SHIM_H\n+\n+#include \u003cstdbool.h\u003e\n+#include \u003cstddef.h\u003e\n+#include \u003cstdint.h\u003e\n+#include \u003cstdio.h\u003e\n+#include \u003cstdlib.h\u003e\n+#include \u003cstring.h\u003e\n+#include \u003cerrno.h\u003e\n+\n+/* From tools/include. */\n+#include \u003clinux/compiler.h\u003e\n+#include \u003clinux/gfp_types.h\u003e\n+#include \u003clinux/init.h\u003e\n+#include \u003clinux/kconfig.h\u003e\n+#include \u003clinux/kernel.h\u003e\n+#include \u003clinux/types.h\u003e\n+\n+#define __ro_after_init\n+#define __percpu\n+#define SMP_CACHE_BYTES\t\t\t64\n+#define ____cacheline_aligned_in_smp\t__aligned(SMP_CACHE_BYTES)\n+#define S8_MAX\t\t\t\tINT8_MAX\n+\n+#undef abs\n+#define abs(x) ({ typeof(x) __x = (x); __x \u003c 0 ? -__x : __x; })\n+\n+#define pr_err(...)\t\tfprintf(stderr, __VA_ARGS__)\n+#define pr_warn(...)\t\tfprintf(stderr, __VA_ARGS__)\n+#define pr_info(...)\t\tdo { if (0) printf(__VA_ARGS__); } while (0)\n+\n+/* Implemented in main.c. */\n+char *get_options(const char *str, int nints, int *ints);\n+\n+extern unsigned long hpcc_nr_warnings;\n+\n+#define WARN(cond, ...) ({\t\t\t\t\t\t\\\n+\tint __ret_warn = !!(cond);\t\t\t\t\t\\\n+\tif (__ret_warn) {\t\t\t\t\t\t\\\n+\t\thpcc_nr_warnings++;\t\t\t\t\t\\\n+\t\tfprintf(stderr, \"WARNING at %s:%d: \", __FILE__, __LINE__); \\\n+\t\tfprintf(stderr, __VA_ARGS__);\t\t\t\t\\\n+\t}\t\t\t\t\t\t\t\t\\\n+\t__ret_warn;\t\t\t\t\t\t\t\\\n+})\n+#define WARN_ON(cond)\t\tWARN(cond, \"%s\\n\", #cond)\n+#define WARN_ON_ONCE(cond)\tWARN_ON(cond)\n+\n+static inline int get_count_order(unsigned int count)\n+{\n+\tint order;\n+\n+\tif (!count)\n+\t\treturn -1;\n+\torder = 31 - __builtin_clz(count);\n+\tif (count \u0026 (count - 1))\n+\t\torder++;\n+\treturn order;\n+}\n+\n+#define atomic_long_read(v)\t\t\tREAD_ONCE((v)-\u003ecounter)\n+#define atomic_long_set(v, i)\t\t\tWRITE_ONCE((v)-\u003ecounter, (i))\n+#define atomic_long_add(i, v)\t\t\t((void)((v)-\u003ecounter += (i)))\n+#define atomic_long_add_return_relaxed(i, v)\t((v)-\u003ecounter += (i))\n+#define atomic_long_add_return_release(i, v)\t((v)-\u003ecounter += (i))\n+#define smp_rmb()\t\t\t\tbarrier()\n+/* The harness runs on one thread: ordering primitives are plain accesses. */\n+#define smp_load_acquire(p)\t\t\tREAD_ONCE(*(p))\n+#define smp_store_release(p, v)\t\t\tWRITE_ONCE(*(p), (v))\n+\n+static inline bool atomic_long_try_cmpxchg_relaxed(atomic_long_t *v, long *old, long new)\n+{\n+\tif (v-\u003ecounter == *old) {\n+\t\tv-\u003ecounter = new;\n+\t\treturn true;\n+\t}\n+\t*old = v-\u003ecounter;\n+\treturn false;\n+}\n+\n+/* Simulated topology, and CPU on which the current code runs. */\n+extern unsigned int nr_cpu_ids;\n+extern unsigned int hpcc_current_cpu;\n+\n+#define raw_smp_processor_id()\t\thpcc_current_cpu\n+#define for_each_possible_cpu(cpu)\tfor ((cpu) = 0; (cpu) \u003c (int)nr_cpu_ids; (cpu)++)\n+\n+/*\n+ * Each per-CPU allocation reserves HPCC_PCPU_UNIT bytes per CPU, so\n+ * per_cpu_ptr() can be applied to a pointer within an allocation.\n+ */\n+#define HPCC_PCPU_UNIT\t64\n+\n+/* Outstanding allocations, to check for leaks. */\n+extern long hpcc_nr_allocs;\n+\n+static inline void *hpcc_alloc_account(void *p)\n+{\n+\tif (p)\n+\t\thpcc_nr_allocs++;\n+\treturn p;\n+}\n+\n+static inline void hpcc_free_account(void *p)\n+{\n+\tif (p)\n+\t\thpcc_nr_allocs--;\n+}\n+\n+static inline void *__alloc_percpu_gfp(size_t size, size_t align, gfp_t gfp)\n+{\n+\tif (size \u003e HPCC_PCPU_UNIT || align \u003e HPCC_PCPU_UNIT)\n+\t\treturn NULL;\n+\treturn hpcc_alloc_account(calloc(nr_cpu_ids, HPCC_PCPU_UNIT));\n+}\n+\n+static inline void free_percpu(void *ptr)\n+{\n+\thpcc_free_account(ptr);\n+\tfree(ptr);\n+}\n+\n+#define SLAB_PANIC\t0\n+\n+struct kmem_cache {\n+\tsize_t size;\n+\tsize_t align;\n+};\n+\n+static inline struct kmem_cache *kmem_cache_create(const char *name, unsigned int size,\n+\t\t\t\t\t\t unsigned int align, unsigned int flags,\n+\t\t\t\t\t\t void (*ctor)(void *))\n+{\n+\tstruct kmem_cache *s = malloc(sizeof(*s));\n+\n+\tif (!s)\n+\t\tabort();\n+\ts-\u003esize = size;\n+\ts-\u003ealign = align;\n+\treturn s;\n+}\n+\n+static inline void kmem_cache_destroy(struct kmem_cache *s)\n+{\n+\tfree(s);\n+}\n+\n+static inline void *kmem_cache_zalloc(struct kmem_cache *s, gfp_t gfp)\n+{\n+\t/* The allocation size needs to be a multiple of the alignment. */\n+\tvoid *p = aligned_alloc(s-\u003ealign, (s-\u003esize + s-\u003ealign - 1) / s-\u003ealign * s-\u003ealign);\n+\n+\tif (p)\n+\t\tmemset(p, 0, s-\u003esize);\n+\treturn hpcc_alloc_account(p);\n+}\n+\n+static inline void kmem_cache_free(struct kmem_cache *s, void *p)\n+{\n+\thpcc_free_account(p);\n+\tfree(p);\n+}\n+\n+static inline void *kzalloc(size_t size, gfp_t gfp)\n+{\n+\treturn hpcc_alloc_account(calloc(1, size));\n+}\n+\n+static inline void kfree(void *p)\n+{\n+\thpcc_free_account(p);\n+\tfree(p);\n+}\n+\n+#define per_cpu_ptr(ptr, cpu)\t\\\n+\t((typeof(ptr))((char *)(ptr) + (size_t)(cpu) * HPCC_PCPU_UNIT))\n+\n+/* Level 0 counters are the only per-CPU data: s8. */\n+static inline bool __hpcc_this_cpu_try_cmpxchg(s8 *pcp, s8 *old, s8 new)\n+{\n+\ts8 *p = per_cpu_ptr(pcp, hpcc_current_cpu);\n+\n+\tif (*p != *old) {\n+\t\t*old = *p;\n+\t\treturn false;\n+\t}\n+\t*p = new;\n+\treturn true;\n+}\n+\n+#define this_cpu_read(pcp)\t(*per_cpu_ptr(\u0026(pcp), hpcc_current_cpu))\n+#define this_cpu_try_cmpxchg(pcp, ovalp, nval)\t\\\n+\t__hpcc_this_cpu_try_cmpxchg(\u0026(pcp), (ovalp), (nval))\n+\n+#endif /* _HPCC_SHIM_H */\ndiff --git a/tools/testing/hpcc/include/kunit/visibility.h b/tools/testing/hpcc/include/kunit/visibility.h\nnew file mode 100644\nindex 00000000000000..dfcbfd01a4151f\n--- /dev/null\n+++ b/tools/testing/hpcc/include/kunit/visibility.h\n@@ -0,0 +1,2 @@\n+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */\n+#define EXPORT_SYMBOL_IF_KUNIT(symbol)\ndiff --git a/tools/testing/hpcc/include/linux/args.h b/tools/testing/hpcc/include/linux/args.h\nnew file mode 100644\nindex 00000000000000..0c00a85fdd213d\n--- /dev/null\n+++ b/tools/testing/hpcc/include/linux/args.h\n@@ -0,0 +1,4 @@\n+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */\n+/* Also defined by the C library \u003csys/cdefs.h\u003e. */\n+#undef __CONCAT\n+#include \"../../../../../include/linux/args.h\"\ndiff --git a/tools/testing/hpcc/include/linux/atomic.h b/tools/testing/hpcc/include/linux/atomic.h\nnew file mode 100644\nindex 00000000000000..b2aa26415cca18\n--- /dev/null\n+++ b/tools/testing/hpcc/include/linux/atomic.h\n@@ -0,0 +1,2 @@\n+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */\n+#include \"../hpcc_shim.h\"\ndiff --git a/tools/testing/hpcc/include/linux/bug.h b/tools/testing/hpcc/include/linux/bug.h\nnew file mode 100644\nindex 00000000000000..b2aa26415cca18\n--- /dev/null\n+++ b/tools/testing/hpcc/include/linux/bug.h\n@@ -0,0 +1,2 @@\n+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */\n+#include \"../hpcc_shim.h\"\ndiff --git a/tools/testing/hpcc/include/linux/cache.h b/tools/testing/hpcc/include/linux/cache.h\nnew file mode 100644\nindex 00000000000000..b2aa26415cca18\n--- /dev/null\n+++ b/tools/testing/hpcc/include/linux/cache.h\n@@ -0,0 +1,2 @@\n+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */\n+#include \"../hpcc_shim.h\"\ndiff --git a/tools/testing/hpcc/include/linux/cpumask.h b/tools/testing/hpcc/include/linux/cpumask.h\nnew file mode 100644\nindex 00000000000000..b2aa26415cca18\n--- /dev/null\n+++ b/tools/testing/hpcc/include/linux/cpumask.h\n@@ -0,0 +1,2 @@\n+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */\n+#include \"../hpcc_shim.h\"\ndiff --git a/tools/testing/hpcc/include/linux/errno.h b/tools/testing/hpcc/include/linux/errno.h\nnew file mode 100644\nindex 00000000000000..45fdd60261e717\n--- /dev/null\n+++ b/tools/testing/hpcc/include/linux/errno.h\n@@ -0,0 +1,3 @@\n+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */\n+/* Also included by the C library \u003cerrno.h\u003e: only provide the error numbers. */\n+#include \u003casm/errno.h\u003e\ndiff --git a/tools/testing/hpcc/include/linux/limits.h b/tools/testing/hpcc/include/linux/limits.h\nnew file mode 100644\nindex 00000000000000..b2aa26415cca18\n--- /dev/null\n+++ b/tools/testing/hpcc/include/linux/limits.h\n@@ -0,0 +1,2 @@\n+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */\n+#include \"../hpcc_shim.h\"\ndiff --git a/tools/testing/hpcc/include/linux/math.h b/tools/testing/hpcc/include/linux/math.h\nnew file mode 100644\nindex 00000000000000..b2aa26415cca18\n--- /dev/null\n+++ b/tools/testing/hpcc/include/linux/math.h\n@@ -0,0 +1,2 @@\n+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */\n+#include \"../hpcc_shim.h\"\ndiff --git a/tools/testing/hpcc/include/linux/moduleparam.h b/tools/testing/hpcc/include/linux/moduleparam.h\nnew file mode 100644\nindex 00000000000000..7e40df4295b2f5\n--- /dev/null\n+++ b/tools/testing/hpcc/include/linux/moduleparam.h\n@@ -0,0 +1,12 @@\n+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */\n+#include \"../hpcc_shim.h\"\n+\n+struct kernel_param;\n+\n+struct kernel_param_ops {\n+\tint (*set)(const char *val, const struct kernel_param *kp);\n+\tint (*get)(char *buffer, const struct kernel_param *kp);\n+};\n+\n+#define module_param_cb(name, ops, arg, perm)\n+#define MODULE_PARM_DESC(parm, desc)\ndiff --git a/tools/testing/hpcc/include/linux/percpu.h b/tools/testing/hpcc/include/linux/percpu.h\nnew file mode 100644\nindex 00000000000000..b2aa26415cca18\n--- /dev/null\n+++ b/tools/testing/hpcc/include/linux/percpu.h\n@@ -0,0 +1,2 @@\n+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */\n+#include \"../hpcc_shim.h\"\ndiff --git a/tools/testing/hpcc/include/linux/percpu_counter_tree.h b/tools/testing/hpcc/include/linux/percpu_counter_tree.h\nnew file mode 100644\nindex 00000000000000..065a2aca4d5b8b\n--- /dev/null\n+++ b/tools/testing/hpcc/include/linux/percpu_counter_tree.h\n@@ -0,0 +1,2 @@\n+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */\n+#include \"../../../../../include/linux/percpu_counter_tree.h\"\ndiff --git a/tools/testing/hpcc/include/linux/preempt.h b/tools/testing/hpcc/include/linux/preempt.h\nnew file mode 100644\nindex 00000000000000..b2aa26415cca18\n--- /dev/null\n+++ b/tools/testing/hpcc/include/linux/preempt.h\n@@ -0,0 +1,2 @@\n+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */\n+#include \"../hpcc_shim.h\"\ndiff --git a/tools/testing/hpcc/include/linux/slab.h b/tools/testing/hpcc/include/linux/slab.h\nnew file mode 100644\nindex 00000000000000..b2aa26415cca18\n--- /dev/null\n+++ b/tools/testing/hpcc/include/linux/slab.h\n@@ -0,0 +1,2 @@\n+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */\n+#include \"../hpcc_shim.h\"\ndiff --git a/tools/testing/hpcc/main.c b/tools/testing/hpcc/main.c\nnew file mode 100644\nindex 00000000000000..be17b887312631\n--- /dev/null\n+++ b/tools/testing/hpcc/main.c\n@@ -0,0 +1,536 @@\n+// SPDX-License-Identifier: GPL-2.0+ OR MIT\n+/*\n+ * Userspace test of the hierarchical per-CPU counters (lib/percpu_counter_tree.c).\n+ *\n+ * The library is built with a simulated topology, which allows testing\n+ * every supported tree configuration on a single machine, whereas the\n+ * KUnit test only covers the topology of the machine it runs on.\n+ *\n+ * For each simulated number of CPUs and batch size:\n+ *\n+ * - Saturate the counter upward and downward: leave every tree item with\n+ * its largest positive (or negative) residual which has not been\n+ * propagated yet, check that the distance between the precise and\n+ * approximate sums matches the residual capacity of the tree in that\n+ * direction, and check the accuracy range contract and the comparison\n+ * functions in these worst-case states.\n+ *\n+ * - Random walk: apply random increments from random CPUs to two counters,\n+ * and check the accuracy range contract and the comparison functions\n+ * against shadow sums after each increment.\n+ *\n+ * - Lazy counter: random owner and atomic updates of a lazily initialized\n+ * counter, exact before its tree is activated halfway, with the same\n+ * checks after each increment.\n+ *\n+ * The test is single-threaded, so it only covers settled counters.\n+ */\n+\n+#include \u003cstdarg.h\u003e\n+#include \"hpcc_shim.h\"\n+\n+unsigned int nr_cpu_ids;\n+unsigned int hpcc_current_cpu;\n+unsigned long hpcc_nr_warnings;\n+long hpcc_nr_allocs;\n+\n+#include \"../../../lib/percpu_counter_tree.c\"\n+\n+#define MAX_REPORTED_FAILURES\t10\n+\n+static unsigned long nr_case_failures;\n+\n+#define CHECK(cond) do {\t\t\t\t\t\t\t\\\n+\tif (!(cond)) {\t\t\t\t\t\t\t\t\\\n+\t\tif (nr_case_failures++ \u003c MAX_REPORTED_FAILURES)\t\t\t\\\n+\t\t\tprintf(\"# %s:%d: check failed: %s\\n\",\t\t\t\\\n+\t\t\t __FILE__, __LINE__, #cond);\t\t\t\\\n+\t}\t\t\t\t\t\t\t\t\t\\\n+} while (0)\n+\n+static uint64_t rng_state = 88172645463325252ULL;\n+\n+static uint64_t rnd(void)\n+{\n+\trng_state ^= rng_state \u003c\u003c 13;\n+\trng_state ^= rng_state \u003e\u003e 7;\n+\trng_state ^= rng_state \u003c\u003c 17;\n+\treturn rng_state;\n+}\n+\n+static int sign(long v)\n+{\n+\treturn (v \u003e 0) - (v \u003c 0);\n+}\n+\n+struct test_counter {\n+\tstruct percpu_counter_tree tree;\n+\tlong shadow;\t/* Expected precise sum. */\n+};\n+\n+int scnprintf(char *buf, size_t size, const char *fmt, ...)\n+{\n+\tva_list args;\n+\tint len;\n+\n+\tif (!size)\n+\t\treturn 0;\n+\tva_start(args, fmt);\n+\tlen = vsnprintf(buf, size, fmt, args);\n+\tva_end(args);\n+\treturn len \u003c (int)size ? len : (int)size - 1;\n+}\n+\n+/* Comma-separated integers, without the ranges of lib/cmdline.c. */\n+char *get_options(const char *str, int nints, int *ints)\n+{\n+\tchar *end;\n+\tint i = 0;\n+\n+\twhile (i \u003c nints - 1 \u0026\u0026 *str) {\n+\t\tlong v = strtol(str, \u0026end, 0);\n+\n+\t\tif (end == str)\n+\t\t\tbreak;\n+\t\tints[++i] = v;\n+\t\tstr = end;\n+\t\tif (*str != ',')\n+\t\t\tbreak;\n+\t\tstr++;\n+\t}\n+\tints[0] = i;\n+\treturn (char *)str;\n+}\n+\n+static void hpcc_init_topology(unsigned int nr_cpus)\n+{\n+\tnr_cpu_ids = nr_cpus;\n+\t/* Reset the state accumulated by a prior subsystem initialization. */\n+\tkmem_cache_destroy(tree_topology.items_cachep);\n+\tmemset(\u0026tree_topology, 0, sizeof(tree_topology));\n+\ttree_topology.nr_items = UINT_MAX;\n+\tCHECK(!percpu_counter_tree_subsystem_init());\n+}\n+\n+static int test_counter_init(struct test_counter *c, unsigned long batch_size)\n+{\n+\tc-\u003eshadow = 0;\n+\treturn percpu_counter_tree_init(\u0026c-\u003etree, batch_size, GFP_KERNEL);\n+}\n+\n+static void test_counter_destroy(struct test_counter *c)\n+{\n+\tpercpu_counter_tree_destroy(\u0026c-\u003etree);\n+}\n+\n+static void test_counter_add(struct test_counter *c, unsigned int cpu, long inc)\n+{\n+\thpcc_current_cpu = cpu;\n+\tpercpu_counter_tree_add(\u0026c-\u003etree, inc);\n+\tc-\u003eshadow += inc;\n+}\n+\n+static void test_counter_set(struct test_counter *c, long v)\n+{\n+\tpercpu_counter_tree_set(\u0026c-\u003etree, v);\n+\tc-\u003eshadow = v;\n+}\n+\n+/*\n+ * Residual capacity of the tree items above level 0, in either direction:\n+ * sum of the largest residual which each item can hold without propagating\n+ * a carry. A level L item only receives multiples of the level L - 1 carry,\n+ * and propagates once its residual reaches the level L carry, so it holds\n+ * at most bit(L) - bit(L - 1) in either direction.\n+ */\n+static unsigned long tree_item_capacity(unsigned long batch_size)\n+{\n+\tunsigned long capacity = 0, lo = batch_size, hi;\n+\tunsigned int level, level_items;\n+\n+\tlevel_items = 1U \u003c\u003c tree_topology.nr_cpus_order;\n+\tfor (level = 1; level \u003c tree_topology.config-\u003enr_levels; level++) {\n+\t\tunsigned int n_arity_order = tree_topology.config-\u003en_arity_order[level - 1];\n+\n+\t\tlevel_items \u003e\u003e= n_arity_order;\n+\t\thi = lo \u003c\u003c n_arity_order;\n+\t\tcapacity += level_items * (hi - lo);\n+\t\tlo = hi;\n+\t}\n+\treturn capacity;\n+}\n+\n+/*\n+ * Residual capacity of the tree, in either direction: a level 0 counter\n+ * holds at most batch_size - 1 on top of the item capacity.\n+ */\n+static unsigned long tree_residual_capacity(unsigned long batch_size)\n+{\n+\treturn nr_cpu_ids * (batch_size - 1) + tree_item_capacity(batch_size);\n+}\n+\n+/* Copy the state of counter @src into counter @dst, of the same topology. */\n+static void test_counter_copy(struct test_counter *dst, struct test_counter *src)\n+{\n+\tpercpu_counter_tree_copy_state(\u0026dst-\u003etree, \u0026src-\u003etree);\n+\tdst-\u003eshadow = src-\u003eshadow;\n+}\n+\n+/*\n+ * Check the accuracy range contract, and the comparison of the counter\n+ * against values around its precise sum and its possible precise range.\n+ */\n+static void check_counter(struct test_counter *c)\n+{\n+\tstruct percpu_counter_tree *t = \u0026c-\u003etree;\n+\tlong approx = percpu_counter_tree_approximate_sum(t);\n+\tlong precise = c-\u003eshadow, precise_min, precise_max;\n+\tunsigned long accuracy;\n+\tlong values[7];\n+\tunsigned int i;\n+\n+\tCHECK(percpu_counter_tree_precise_sum(t) == precise);\n+\tCHECK(!percpu_counter_tree_check_residuals(t));\n+\n+\t/* precise - accuracy \u003c= approx \u003c= precise + accuracy */\n+\taccuracy = percpu_counter_tree_approximate_accuracy(t);\n+\tCHECK(approx \u003e= precise - (long)accuracy);\n+\tCHECK(approx \u003c= precise + (long)accuracy);\n+\n+\tpercpu_counter_tree_approximate_min_max_range(approx, accuracy,\n+\t\t\t\t\t\t \u0026precise_min, \u0026precise_max);\n+\tCHECK(precise \u003e= precise_min);\n+\tCHECK(precise \u003c= precise_max);\n+\n+\t/* Approximate comparison is definitive exactly outside of the precise range. */\n+\tCHECK(percpu_counter_tree_approximate_compare_value(t, precise_min - 1) == -1);\n+\tCHECK(percpu_counter_tree_approximate_compare_value(t, precise_min) == 0);\n+\tCHECK(percpu_counter_tree_approximate_compare_value(t, precise_max) == 0);\n+\tCHECK(percpu_counter_tree_approximate_compare_value(t, precise_max + 1) == 1);\n+\n+\tvalues[0] = precise - 1;\n+\tvalues[1] = precise;\n+\tvalues[2] = precise + 1;\n+\tvalues[3] = precise_min - 1;\n+\tvalues[4] = precise_min;\n+\tvalues[5] = precise_max;\n+\tvalues[6] = precise_max + 1;\n+\tfor (i = 0; i \u003c ARRAY_SIZE(values); i++) {\n+\t\tlong v = values[i];\n+\t\tint expect = sign(v - precise);\n+\t\tint res = percpu_counter_tree_approximate_compare_value(t, v);\n+\n+\t\tCHECK(res == 0 || res == expect);\n+\t\tCHECK(percpu_counter_tree_precise_compare_value(t, v) == expect);\n+\t}\n+}\n+\n+/* Check the comparison of two counters against their shadow sums. */\n+static void check_counter_pair(struct test_counter *a, struct test_counter *b)\n+{\n+\tint expect = sign(a-\u003eshadow - b-\u003eshadow);\n+\tint res = percpu_counter_tree_approximate_compare(\u0026a-\u003etree, \u0026b-\u003etree);\n+\n+\tCHECK(res == 0 || res == expect);\n+\tCHECK(percpu_counter_tree_precise_compare(\u0026a-\u003etree, \u0026b-\u003etree) == expect);\n+\tres = percpu_counter_tree_approximate_compare(\u0026b-\u003etree, \u0026a-\u003etree);\n+\tCHECK(res == 0 || res == -expect);\n+\tCHECK(percpu_counter_tree_precise_compare(\u0026b-\u003etree, \u0026a-\u003etree) == -expect);\n+}\n+\n+/*\n+ * On each CPU, add @inc (1 or -1) until the next addition would propagate a\n+ * carry to the approximate sum. This leaves every tree item on the path from\n+ * this CPU to the root holding its largest residual in the direction of @inc.\n+ * Each addition is first tried on @probe, which holds the same state as @c:\n+ * reverting an addition which reached the approximate sum would not restore\n+ * the residuals subtracted on the way. The path from a CPU to the root\n+ * holds less than the accuracy bound: more additions mean that a carry was\n+ * lost.\n+ */\n+static void saturate(struct test_counter *c, struct test_counter *probe, long inc)\n+{\n+\tunsigned long max_steps;\n+\tunsigned int cpu;\n+\n+\tmax_steps = percpu_counter_tree_approximate_accuracy(\u0026c-\u003etree) +\n+\t\t (1UL \u003c\u003c c-\u003etree.batch_order);\n+\tfor (cpu = 0; cpu \u003c nr_cpu_ids; cpu++) {\n+\t\tunsigned long steps;\n+\n+\t\tfor (steps = 0; steps \u003c max_steps; steps++) {\n+\t\t\tlong approx = percpu_counter_tree_approximate_sum(\u0026probe-\u003etree);\n+\n+\t\t\ttest_counter_add(probe, cpu, inc);\n+\t\t\tif (percpu_counter_tree_approximate_sum(\u0026probe-\u003etree) != approx) {\n+\t\t\t\ttest_counter_copy(probe, c);\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\ttest_counter_add(c, cpu, inc);\n+\t\t}\n+\t\tif (steps == max_steps) {\n+\t\t\tCHECK(0);\n+\t\t\treturn;\n+\t\t}\n+\t}\n+}\n+\n+static void test_saturated(unsigned long batch_size, long inc)\n+{\n+\tstruct test_counter c, probe, exact;\n+\tunsigned long accuracy;\n+\tlong approx, delta;\n+\n+\tif (test_counter_init(\u0026c, batch_size) || test_counter_init(\u0026probe, batch_size) ||\n+\t test_counter_init(\u0026exact, batch_size)) {\n+\t\tCHECK(0);\n+\t\treturn;\n+\t}\n+\tsaturate(\u0026c, \u0026probe, inc);\n+\tapprox = percpu_counter_tree_approximate_sum(\u0026c.tree);\n+\taccuracy = percpu_counter_tree_approximate_accuracy(\u0026c.tree);\n+\t/* Symmetric residuals: the accuracy is reached in both directions. */\n+\tCHECK((c.shadow - approx) * inc == (long)tree_residual_capacity(batch_size));\n+\tCHECK((c.shadow - approx) * inc == (long)accuracy);\n+\tcheck_counter(\u0026c);\n+\n+\t/* Compare with exactly approximated counters around the saturated precise sum. */\n+\tfor (delta = -1; delta \u003c= 1; delta++) {\n+\t\ttest_counter_set(\u0026exact, c.shadow + delta);\n+\t\tCHECK(percpu_counter_tree_approximate_sum(\u0026exact.tree) == exact.shadow);\n+\t\tcheck_counter_pair(\u0026c, \u0026exact);\n+\t}\n+\ttest_counter_destroy(\u0026c);\n+\ttest_counter_destroy(\u0026probe);\n+\ttest_counter_destroy(\u0026exact);\n+}\n+\n+static long random_increment(unsigned long batch_size)\n+{\n+\tlong v;\n+\n+\tswitch (rnd() % 4) {\n+\tcase 0:\n+\t\tv = rnd() % 4;\n+\t\tbreak;\n+\tcase 1:\n+\t\tv = rnd() % (4 * batch_size);\n+\t\tbreak;\n+\tcase 2:\n+\t\tv = rnd() % (64 * batch_size);\n+\t\tbreak;\n+\tdefault:\n+\t\tv = rnd() % 1000000;\n+\t\tbreak;\n+\t}\n+\treturn rnd() \u0026 1 ? v : -v;\n+}\n+\n+static void test_random_walk(unsigned long batch_size, unsigned long nr_steps)\n+{\n+\tstruct test_counter c[2];\n+\tunsigned long step;\n+\n+\tif (test_counter_init(\u0026c[0], batch_size) || test_counter_init(\u0026c[1], batch_size)) {\n+\t\tCHECK(0);\n+\t\treturn;\n+\t}\n+\tfor (step = 0; step \u003c nr_steps; step++) {\n+\t\tstruct test_counter *t = \u0026c[rnd() \u0026 1];\n+\n+\t\ttest_counter_add(t, rnd() % nr_cpu_ids, random_increment(batch_size));\n+\t\t/* Bring the counters close to each other to exercise the precise comparison. */\n+\t\tif (!(rnd() % 8)) {\n+\t\t\tlong delta = c[0].shadow - c[1].shadow;\n+\t\t\tunsigned long accuracy;\n+\n+\t\t\taccuracy = percpu_counter_tree_approximate_accuracy(\u0026c[0].tree);\n+\t\t\tdelta += (long)(rnd() % (2 * accuracy + 1)) - (long)accuracy;\n+\t\t\ttest_counter_add(\u0026c[1], rnd() % nr_cpu_ids, delta);\n+\t\t}\n+\t\tcheck_counter(t);\n+\t\tcheck_counter_pair(\u0026c[0], \u0026c[1]);\n+\t}\n+\ttest_counter_destroy(\u0026c[0]);\n+\ttest_counter_destroy(\u0026c[1]);\n+}\n+\n+static void test_lazy(unsigned long batch_size, unsigned long nr_steps)\n+{\n+\tlong nr_allocs = hpcc_nr_allocs;\n+\tstruct test_counter c;\n+\tunsigned long step;\n+\n+\tif (percpu_counter_tree_init_many_lazy(\u0026c.tree, 1, batch_size)) {\n+\t\tCHECK(0);\n+\t\treturn;\n+\t}\n+\tc.shadow = 0;\n+\tCHECK(!c.tree.level0);\n+\tCHECK(!c.tree.approx_sum);\n+\tCHECK(!percpu_counter_tree_active(\u0026c.tree));\n+\tfor (step = 0; step \u003c nr_steps; step++) {\n+\t\tlong inc = random_increment(batch_size);\n+\n+\t\thpcc_current_cpu = rnd() % nr_cpu_ids;\n+\t\tif (rnd() \u0026 1)\n+\t\t\tpercpu_counter_tree_add_owner(\u0026c.tree, inc);\n+\t\telse\n+\t\t\tpercpu_counter_tree_add(\u0026c.tree, inc);\n+\t\tc.shadow += inc;\n+\t\tif (!c.tree.level0)\n+\t\t\tCHECK(percpu_counter_tree_approximate_sum(\u0026c.tree) == c.shadow);\n+\t\tcheck_counter(\u0026c);\n+\t\tif (step == nr_steps / 2) {\n+\t\t\tCHECK(!percpu_counter_tree_activate_many(\u0026c.tree, 1, GFP_KERNEL));\n+\t\t\tCHECK(c.tree.level0);\n+\t\t\tCHECK(c.tree.approx_sum);\n+\t\t\tCHECK(percpu_counter_tree_active(\u0026c.tree));\n+\t\t\t/* One tree: its per-CPU counters and its tree items. */\n+\t\t\tCHECK(hpcc_nr_allocs == nr_allocs + 2);\n+\t\t\tCHECK(!percpu_counter_tree_activate_many(\u0026c.tree, 1, GFP_KERNEL));\n+\t\t\tCHECK(hpcc_nr_allocs == nr_allocs + 2);\n+\t\t}\n+\t}\n+\ttest_counter_destroy(\u0026c);\n+\tCHECK(hpcc_nr_allocs == nr_allocs);\n+}\n+\n+static const unsigned int test_nr_cpus[] = {\n+\t1, 2, 3, 4, 5, 8, 9, 16, 17, 32, 33, 64, 65, 100, 128, 256, 512,\n+\t1000, 1024, 2048, 4096, 8192, 16384, 32768, 65536,\n+};\n+\n+/* Exhaustive mode: also cover the largest supported configurations. */\n+static const unsigned int test_nr_cpus_large[] = {\n+\t1U \u003c\u003c 17, 1U \u003c\u003c 18, 1U \u003c\u003c 19, 1U \u003c\u003c 20,\n+};\n+\n+static const unsigned long test_batch_sizes[] = { 2, 4, 8, 16, 32, 64, 128 };\n+\n+static int test_num;\n+\n+static int run_case(unsigned int nr_cpus, unsigned long batch_size)\n+{\n+\t/* Each check iterates over all CPUs: bound the random walk cost. */\n+\tunsigned long nr_steps = 10000000 / nr_cpus;\n+\n+\tif (nr_steps \u003c 100)\n+\t\tnr_steps = 100;\n+\tif (nr_steps \u003e 50000)\n+\t\tnr_steps = 50000;\n+\n+\tnr_case_failures = 0;\n+\thpcc_nr_warnings = 0;\n+\thpcc_init_topology(nr_cpus);\n+\ttest_saturated(batch_size, 1);\n+\ttest_saturated(batch_size, -1);\n+\ttest_random_walk(batch_size, nr_steps);\n+\ttest_lazy(batch_size, nr_steps / 2);\n+\tCHECK(!hpcc_nr_warnings);\n+\tprintf(\"%sok %d - nr_cpus=%u batch_size=%lu\\n\", nr_case_failures ? \"not \" : \"\",\n+\t ++test_num, nr_cpus, batch_size);\n+\tif (nr_case_failures \u003e MAX_REPORTED_FAILURES)\n+\t\tprintf(\"# ... %lu failed checks in total\\n\", nr_case_failures);\n+\treturn nr_case_failures ? 1 : 0;\n+}\n+\n+/* Tree shapes given with the arity boot parameter. */\n+static const struct {\n+\tunsigned int nr_cpus;\n+\tconst char *arity;\n+\tconst char *shape;\t/* Shape in use, NULL if the arity is invalid. */\n+} test_arity[] = {\n+\t{ 16, \"16\", \"16\" },\n+\t{ 16, \"4,4\", \"4,4\" },\n+\t{ 16, \"2,2,2,2\", \"2,2,2,2\" },\n+\t{ 16, \"4,2\", NULL },\t\t/* Covers 8 CPUs. */\n+\t{ 16, \"4,8\", NULL },\t\t/* Covers 32 CPUs. */\n+\t{ 16, \"4,4,1\", NULL },\t\t/* Fan-out of 1. */\n+\t{ 16, \"4,3\", NULL },\t\t/* Not a power of 2. */\n+\t{ 16, \"4,4x\", NULL },\t\t/* Trailing characters. */\n+\t{ 16, \"-4,-4\", NULL },\n+\t{ 17, \"8,4\", \"8,4\" },\n+\t{ 100, \"128\", \"128\" },\n+\t{ 192, \"8,4,8\", \"8,4,8\" },\n+\t{ 192, \"2,2,2,2,2,2,2,2\", NULL },\t/* More than MAX_NR_LEVELS. */\n+\t{ 1024, \"32,32\", \"32,32\" },\n+\t{ 1, \"2\", NULL },\n+};\n+\n+static int run_arity_case(unsigned int i)\n+{\n+\tconst char *expect = test_arity[i].shape;\n+\tchar buf[SHAPE_BUF_SIZE];\n+\tint len;\n+\n+\tnr_case_failures = 0;\n+\thpcc_nr_warnings = 0;\n+\ttree_topology.config = NULL;\n+\tCHECK(!arity_param_set(test_arity[i].arity, NULL));\n+\thpcc_init_topology(test_arity[i].nr_cpus);\n+\tarity_param = NULL;\n+\tCHECK(arity_param_set(\"16\", NULL) == -EBUSY);\n+\tCHECK((tree_topology.config == \u0026tree_topology.boot_config) == !!expect);\n+\tlen = arity_param_get(buf, NULL);\n+\tCHECK(len \u003e 0 \u0026\u0026 buf[len - 1] == '\\n');\n+\tbuf[len - 1] = '\\0';\n+\tif (expect) {\n+\t\tCHECK(!strcmp(buf, expect));\n+\t\ttest_saturated(2, 1);\n+\t\ttest_saturated(32, -1);\n+\t\ttest_random_walk(32, 20000);\n+\t}\n+\tCHECK(!hpcc_nr_warnings);\n+\tprintf(\"%sok %d - arity nr_cpus=%u \\\"%s\\\" -\u003e \\\"%s\\\"\\n\", nr_case_failures ? \"not \" : \"\",\n+\t ++test_num, test_arity[i].nr_cpus, test_arity[i].arity, buf);\n+\treturn nr_case_failures ? 1 : 0;\n+}\n+\n+static int run_check_config(void)\n+{\n+\tunsigned int nr_errors = percpu_counter_tree_check_config();\n+\n+\tprintf(\"%sok %d - configuration table\\n\", nr_errors ? \"not \" : \"\", ++test_num);\n+\treturn nr_errors ? 1 : 0;\n+}\n+\n+static void usage(const char *name)\n+{\n+\tfprintf(stderr, \"Usage: %s [-a]\\n\", name);\n+\tfprintf(stderr, \" -a Also test the largest supported numbers of CPUs (slow).\\n\");\n+}\n+\n+int main(int argc, char **argv)\n+{\n+\tunsigned int i, j, nr_tests;\n+\tbool large = false;\n+\tint failed = 0;\n+\n+\tif (argc \u003e 2 || (argc == 2 \u0026\u0026 strcmp(argv[1], \"-a\"))) {\n+\t\tusage(argv[0]);\n+\t\treturn 2;\n+\t}\n+\tif (argc == 2)\n+\t\tlarge = true;\n+\n+\tnr_tests = 1 + ARRAY_SIZE(test_arity) +\n+\t\t ARRAY_SIZE(test_nr_cpus) * ARRAY_SIZE(test_batch_sizes);\n+\tif (large)\n+\t\tnr_tests += ARRAY_SIZE(test_nr_cpus_large) * ARRAY_SIZE(test_batch_sizes);\n+\tprintf(\"1..%u\\n\", nr_tests);\n+\n+\tfailed += run_check_config();\n+\tfor (i = 0; i \u003c ARRAY_SIZE(test_arity); i++)\n+\t\tfailed += run_arity_case(i);\n+\tfor (i = 0; i \u003c ARRAY_SIZE(test_nr_cpus); i++)\n+\t\tfor (j = 0; j \u003c ARRAY_SIZE(test_batch_sizes); j++)\n+\t\t\tfailed += run_case(test_nr_cpus[i], test_batch_sizes[j]);\n+\tif (large) {\n+\t\tfor (i = 0; i \u003c ARRAY_SIZE(test_nr_cpus_large); i++)\n+\t\t\tfor (j = 0; j \u003c ARRAY_SIZE(test_batch_sizes); j++)\n+\t\t\t\tfailed += run_case(test_nr_cpus_large[i], test_batch_sizes[j]);\n+\t}\n+\tprintf(\"# %d/%u test cases failed\\n\", failed, nr_tests);\n+\treturn failed ? 1 : 0;\n+}\n"
} |
| 2/1 | 2026/11/03 12:05 | agent | patch-evaluator |
10mModel:core Args: null Results: {
"EnableConfigs": [
"SMP"
],
"FocusSymbols": [
"percpu_counter_tree_activate_many",
"percpu_counter_tree_add",
"percpu_counter_tree_precise_sum",
"percpu_counter_tree_destroy_many",
"percpu_counter_tree_precise_compare",
"percpu_counter_tree_approximate_compare",
"percpu_counter_tree_set"
],
"Reasoning": "The patch series implements Hierarchical Per-CPU Counters (HPCC) and transitions the kernel's mm RSS statistics (mm-\u003erss_stat) to use this new data structure. This modifies core subsystems (mm, fork, exec, exit) and introduces lockless tree propagation, atomic carry cmpxchg loops, and lazy activation when processes create threads with CLONE_VM. The modified code is directly reachable by user space via common syscalls (clone, fork, exit, execve, mmap, munmap, page faults, and /proc/\u003cpid\u003e/statm reading) and is an excellent target for fuzzing race conditions and concurrency issues. Ubiquitous single-threaded hot-paths (such as percpu_counter_tree_add_owner and percpu_counter_tree_init_many_lazy) are excluded from FocusSymbols to prevent coverage dilution.",
"WorthFuzzing": true
}Instruction: You are an expert Linux kernel maintainer and security engineer.
Your job is to review a provided patch series and evaluate whether it warrants fuzzing with syzkaller.
IMPORTANT: The changes have ALREADY been applied and committed as the HEAD commit in
your workspace. Do NOT rely on internal assumptions. You must actively use your code access
tools to inspect the actual source code, callers, and surrounding context.
================================================================================
1. CORE TRIAGE PHILOSOPHY
================================================================================
The goal of patch fuzzing is to discover crashes, regressions, exposed latent bugs,
and newly triggered assertions introduced by the patch series.
- REACHABILITY IS THE PRIMARY GATE:
Fuzzing can only discover bugs in code that can actually execute in standard virtualized
environments (GCE or QEMU, utilizing software-emulated devices like USB gadgets, netdev, tun/tap).
If the modified code is structurally unreachable (see Section 2), it MUST NOT be fuzzed,
regardless of whether it adds assertions or complex logic.
- DO NOT BLINDLY TRUST "NO FUNCTIONAL CHANGE" (NFCI) OR "REFACTORING" CLAIMS:
Patch authors routinely label changes as "cleanups", "refactorings", or state
"No functional change intended". Do NOT take these claims at face value.
Code refactorings that rearrange logic, introduce helper functions, or alter state management
in core subsystems frequently introduce subtle semantic shifts or uncover latent kernel bugs.
If reachable executable code is modified or refactored, it MUST be fuzzed.
- NEW OR MODIFIED ASSERTIONS IN REACHABLE CODE MUST BE FUZZED:
When a patch introduces or modifies runtime checks or assertions (e.g., WARN_ON*, VM_WARN_ON*,
BUG_ON*, lockdep_assert*) in reachable code paths, it enforces new or stricter invariants.
Even if the author believes the invariant always holds, fuzzing is essential to verify whether
an unusual sequence of operations can violate it.
================================================================================
2. WHEN TO RETURN WorthFuzzing=false (NEGATIVE CRITERIA)
================================================================================
Return WorthFuzzing=false ONLY IF all modified code falls strictly into one or more of these categories:
- Non-kernel and non-executable changes:
* Modifications to Documentation/, comments, or spelling fixes.
* User-space directories, self-tests, samples, or scripts (e.g., tools/, samples/, scripts/, usr/)
that do not affect the compiled kernel image (vmlinux) or kernel modules.
* Purely decorative logging (e.g., message strings in pr_err, printk, dev_info) or tracepoints
that do not alter control flow or data structures.
* Build system or Kconfig changes that do not alter compiled C logic.
- Structurally unreachable hardware:
* Vendor-specific PCIe switches, SmartNICs, or GPU drivers (e.g., mlxsw, pds_core, qed,
ionic, amdgpu) requiring physical ASIC/PCIe cards not emulated in standard QEMU.
- Unreachable execution paths:
* Driver teardown callbacks (.remove, .shutdown, pci_unregister_driver) executed only during
physical PCI hot-unplug or manual sysfs driver unbinding.
* Code paths exclusive to architectures other than the target architecture.
================================================================================
3. WHEN TO RETURN WorthFuzzing=true (POSITIVE CRITERIA)
================================================================================
Return WorthFuzzing=true whenever the patch touches reachable executable code, including:
- Core Subsystems:
* Any logic modifications in memory management (mm/), synchronization/locking (kernel/locking/),
BPF, scheduler, core networking, VFS, or syscall handling.
- Refactorings and Code Cleanups:
* Any restructuring of reachable data structures, helper abstractions, or algorithm flows.
- Runtime Assertions and Defensive Checks:
* Any introduction or alteration of assertions (WARN_ON*, VM_WARN_ON*, BUG_ON*, etc.) in reachable paths.
- Reachable Drivers and Protocols:
* Drivers accessible via virtual buses (virtio, USB gadget, loopback, netlink, binder, sockets, etc.).
================================================================================
4. EXTRACTING FocusSymbols (PREVENTING DILUTION)
================================================================================
When WorthFuzzing=true, you must extract specific kernel functions into FocusSymbols to guide the fuzzer:
- AVOID UBIQUITOUS LIFECYCLE HOT-PATHS:
Do NOT list generic, ubiquitous functions called by almost every program in the corpus
(including, but not limited to: general memory allocators and deallocators, page fault
and trap handlers, or core synchronization primitives; this is not an exhaustive list).
Listing ubiquitous functions causes the fuzzer to classify thousands of unrelated tests as "focused",
which severely dilutes fuzzing effort away from the actual changes.
- TARGET SPECIFIC FEATURE LOGIC AND ENTRYPOINTS:
List functions that specifically implement the logic being added or altered, or direct API entrypoints
for the subsystem feature under review.
- HANDLING STATIC INLINE FUNCTIONS IN HEADERS (.h):
Compiler-inlined static functions (such as static inlines in mm/*.h or include/linux/*.h) lack
distinct symbol addresses in vmlinux and cannot be targeted directly by symbol coverage filters.
If the changes are primarily in static inline helpers, identify non-static, feature-specific caller
functions in .c files that exercise them (avoiding ubiquitous lifecycle wrappers).
================================================================================
5. IDENTIFYING EnableConfigs
================================================================================
Identify any specific CONFIG_ options required to properly compile and reach the modified code:
- Inspect Kconfig files and #ifdef guards; do not make assumptions.
- Check "depends on" lines in Kconfig to include any non-standard parent subsystem configs needed.
- Strip any 'CONFIG_' prefix (e.g., return "NET_IPV4" instead of "CONFIG_NET_IPV4").
Prefer calling several tools at the same time to save round-trips.
Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.
Prompt: Target architecture: amd64
For your convenience, here is the diff of the changes:
commit d5aca8d49d662c298b87359f285e9556d3b2d61a
Author: syz-cluster <triage@syzkaller.com>
Date: Tue Nov 3 12:05:07 2026 +0000
syz-cluster: applied patch under review
diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 0b3c140babd944..ba2bdeb1480b7f 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -5342,6 +5342,20 @@ Kernel parameters
allocator. This parameter is primarily for debugging
and performance comparison.
+ percpu_counter_tree.arity= [KNL,SMP]
+ Format: <fan-out>[,<fan-out>...]
+ Override the shape of the hierarchical per-cpu
+ counter trees (used for the RSS counters), listing
+ the fan-out of each tree level from the per-cpu
+ counters to the root. Each fan-out is a power of 2
+ and their product must be nr_cpu_ids (the highest
+ possible CPU number plus one) rounded up to a power
+ of 2, e.g. "16" (flat) or "4,4" when nr_cpu_ids
+ is 16. An invalid shape is ignored with a warning.
+ The shape in use is shown in
+ /sys/module/percpu_counter_tree/parameters/arity.
+ This parameter is meant for performance comparison.
+
pirq= [SMP,APIC] Manual mp-table setup
See Documentation/arch/x86/i386/IO-APIC.rst.
@@ -6778,6 +6792,16 @@ Kernel parameters
debug mode. Defaults to CONFIG_RSEQ_DEBUG_DEFAULT_ENABLE.
Format: <bool>
+ rss_stat_batch= [KNL,SMP]
+ Format: <integer>
+ Batch size of the per-mm RSS counters: the number of
+ pages a CPU accumulates before propagating them into
+ the counter tree. A power of 2 between 2 and 128.
+ A smaller batch improves the accuracy of the
+ approximate RSS, a larger batch reduces the update
+ cost. Default: 32.
+ This parameter is meant for performance comparison.
+
rt_group_sched= [KNL] Enable or disable SCHED_RR/FIFO group scheduling
when CONFIG_RT_GROUP_SCHED=y. Defaults to
!CONFIG_RT_GROUP_SCHED_DEFAULT_DISABLED.
diff --git a/Documentation/core-api/index.rst b/Documentation/core-api/index.rst
index 92f91c6a0d79d6..fe03cefc17348a 100644
--- a/Documentation/core-api/index.rst
+++ b/Documentation/core-api/index.rst
@@ -48,6 +48,7 @@ Library functionality that is used throughout the kernel.
generic-radix-tree
packing
this_cpu_ops
+ percpu-counter-tree
timekeeping
errseq
wrappers/atomic_t
diff --git a/Documentation/core-api/percpu-counter-tree.rst b/Documentation/core-api/percpu-counter-tree.rst
new file mode 100644
index 00000000000000..8aebf91aa133d7
--- /dev/null
+++ b/Documentation/core-api/percpu-counter-tree.rst
@@ -0,0 +1,145 @@
+.. SPDX-License-Identifier: GPL-2.0+ OR MIT
+
+========================================
+The Hierarchical Per-CPU Counters (HPCC)
+========================================
+
+:Author: Mathieu Desnoyers
+
+Introduction
+============
+
+Counters come in many varieties, each with their own trade offs:
+
+ * A global atomic counter provides a fast read access to the current
+ sum, at the expense of cache-line bouncing on updates. This leads to
+ poor performance of frequent updates from various cores on large SMP
+ systems.
+
+ * A per-cpu split counter provides fast updates to per-cpu counters,
+ at the expense of a slower aggregation (sum). The sum operation needs
+ to iterate over all per-cpu counters to calculate the current total.
+
+The hierarchical per-cpu counters attempt to provide the best of both
+worlds (fast updates, and fast sum) by relaxing requirements on the sum
+accuracy. It allows quickly querying an approximated sum value, along
+with the possible min/max ranges of the associated precise sum. The
+exact precise sum can still be calculated with an iteration on all
+per-cpu counters, but the availability of an approximated sum value with
+possible precise sum min/max ranges allows eliminating candidates which
+are certainly outside of a known target range without the overhead of
+precise sums.
+
+Overview
+========
+
+The hierarchical per-cpu counters are organized as a tree with the
+per-cpu counters as its first level and the tree root as its last
+level.
+
+The intermediate tree levels contain carry propagation counters. When
+reaching a threshold (batch size), the carry is propagated up the
+tree.
+
+Each counter below the root, from the per-cpu counters up, keeps the
+residual it has not propagated yet, and only propagates once this
+residual reaches its batch size in either direction. This hysteresis
+prevents updates going back and forth around a batch size boundary from
+propagating further up the tree each time.
+
+This allows reading an approximated value at the root, which has a
+bounded accuracy determined by the tree topology and the batch size:
+the precise sum is within the same distance of the approximation in
+both directions. This bound grows as O(nr_cpus * log(nr_cpus)).
+
+The per-cpu counters are bytes, which limits the batch size to 128. The
+tree positions are cache line aligned to prevent false sharing between
+CPUs updating different branches of the tree. A tree position holds the
+items of all the counter trees initialized together by
+percpu_counter_tree_init_many(), so a set of counters updated together,
+such as the RSS counters of an mm, shares cache lines.
+percpu_counter_tree_init_many() allocates the per-cpu counters, one per
+possible CPU, and the tree items, whose number depends on nr_cpu_ids, so
+struct percpu_counter_tree has a fixed size.
+
+The fan-out of each tree level is selected at boot from nr_cpu_ids, the
+highest possible CPU number plus one. It can be overridden with the
+percpu_counter_tree.arity= boot parameter for performance comparison.
+
+The bounded accuracy is a property of the settled tree. While updates are
+in flight the transient error can exceed it, bounded by the number of
+updaters and their increment size.
+
+Likewise, the precise sum is exact when no update runs concurrently. It
+takes no lock: an update which carries from the per-cpu counter to the
+upper levels can be counted in part, the residual it left below read and
+its carry missed on the way up.
+
+Lazy Tree Activation
+====================
+
+Counters initialized with percpu_counter_tree_init_many_lazy() start
+without their per-cpu counters and tree items, for counters which may
+never need the scalability of the tree. Until
+percpu_counter_tree_activate_many() allocates and publishes the tree,
+the counters are exact:
+
+ * percpu_counter_tree_add_owner() adds to the bias of the counter,
+ written only by its owner, a single context chosen by the user of the
+ counters, such as a task, without atomic instruction.
+
+ * percpu_counter_tree_add() adds to the atomic bias of the counter, for
+ any other updater.
+
+The precise and approximate sums read both, in O(1), and count each
+update whole. The activation does not move these values: the sums keep
+reading them along with the tree, so updates racing with the activation
+are not lost, whichever side of it they land on. The bias only has the
+owner as writer, and the atomic bias only receives atomic additions,
+before and after the activation. After activation, both update functions
+add to the tree, and the approximation has the accuracy of the tree.
+
+The activations of the same counters must be serialized by the caller:
+a concurrent activation would publish a second tree and lose the
+updates made to the first.
+
+The RSS counters of an mm use this: the task owning the mm updates the
+plain counter, and other tasks, such as reclaim, update the atomic
+counter. The tree is activated when a second thread shares the mm,
+under mm->rss_stat_mutex, which serializes the activations.
+A single-threaded process therefore has exact RSS counters, cheap
+precise sums, and neither per-cpu memory nor tree items for them.
+
+Use Cases
+=========
+
+Use cases HPCC is meant to handle involve tracking resources which are
+used across many CPUs to quickly sum as feedback for decision making to
+apply throttling, quota limits, sort tasks, and perform memory or task
+migration decisions. When considering approximated sums within the
+accuracy range of the decision threshold, the user can either:
+
+ * Be conservative and fast: Consider that the sum has reached the
+ limit as soon as the given limit is within the approximation range.
+
+ * Be aggressive and fast: Consider that the sum is over the
+ limit only when the approximation range is over the given limit.
+
+ * Be precise and slow: Do a precise comparison with the limit, which
+ requires a precise sum when the limit is within the approximated
+ range.
+
+One use-case for these hierarchical counters is to implement a two-pass
+algorithm to speed up picking a maximum/minimum sum value from
+a set. A first pass compares the approximated values, and then a second
+pass only needs the precise sum for counter trees which are within the
+possible precise sum range of the counter tree chosen by the first pass.
+
+Because the comparison is on the wrapped difference, it is not transitive:
+ordering is undefined over a set of counters spanning more than LONG_MAX.
+
+Functions and structures
+========================
+
+.. kernel-doc:: include/linux/percpu_counter_tree.h
+.. kernel-doc:: lib/percpu_counter_tree.c
diff --git a/MAINTAINERS b/MAINTAINERS
index 93abbb315065de..3878203de1c30f 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -11947,6 +11947,18 @@ L: linux-input@vger.kernel.org
S: Maintained
F: drivers/hid/hid-logitech-hidpp.c
+HIERARCHICAL PER-CPU COUNTERS (HPCC)
+M: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
+L: linux-mm@kvack.org
+L: linux-kernel@vger.kernel.org
+S: Supported
+F: Documentation/core-api/percpu-counter-tree.rst
+F: include/linux/percpu_counter_tree.h
+F: lib/percpu_counter_tree.c
+F: lib/tests/percpu_counter_tree_bench.c
+F: lib/tests/percpu_counter_tree_kunit.c
+F: tools/testing/hpcc/
+
HIGH-RESOLUTION TIMERS, TIMER WHEEL, CLOCKEVENTS
M: Anna-Maria Behnsen <anna-maria@linutronix.de>
M: Frederic Weisbecker <frederic@kernel.org>
diff --git a/fs/exec.c b/fs/exec.c
index 8d2b14e1d455cf..7a6f07a6d482fc 100644
--- a/fs/exec.c
+++ b/fs/exec.c
@@ -875,6 +875,8 @@ static int exec_mmap(struct linux_binprm *bprm)
}
}
+ /* This task owns the RSS counters of the new mm, see __add_mm_counter(). */
+ WRITE_ONCE(mm->rss_owner, tsk);
task_lock(tsk);
membarrier_exec_mmap(mm);
@@ -903,6 +905,9 @@ static int exec_mmap(struct linux_binprm *bprm)
if (old_mm) {
mmap_read_unlock(old_mm);
BUG_ON(active_mm != old_mm);
+ /* Updates of the old mm by this task are now atomic. */
+ if (READ_ONCE(old_mm->rss_owner) == tsk)
+ WRITE_ONCE(old_mm->rss_owner, NULL);
/* Defer teardown to setup_new_exec(), outside the exec locks. */
bprm->old_mm = old_mm;
} else {
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 9f167950926a15..4d740c0c5f5481 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -3607,33 +3607,51 @@ static inline bool get_user_page_fast_only(unsigned long addr,
*/
static inline unsigned long get_mm_counter(struct mm_struct *mm, int member)
{
- return percpu_counter_read_positive(&mm->rss_stat[member]);
+ return percpu_counter_tree_approximate_sum_positive(&mm->rss_stat[member]);
}
static inline unsigned long get_mm_counter_sum(struct mm_struct *mm, int member)
{
- return percpu_counter_sum_positive(&mm->rss_stat[member]);
+ return percpu_counter_tree_precise_sum_positive(&mm->rss_stat[member]);
}
void mm_trace_rss_stat(struct mm_struct *mm, int member);
+/*
+ * Until a second thread shares the mm, the RSS counters are exact single
+ * counters: the task which owns the mm updates a plain counter, and other
+ * tasks (reclaim, khugepaged, the OOM reaper, remote GUP, ...) an atomic
+ * one. See copy_mm(). Once the tree is active, all updates go to it:
+ * rss_owner is not read, so the updates of the threads of a process do not
+ * touch its cache line.
+ */
+static inline void __add_mm_counter(struct mm_struct *mm, int member, long value)
+{
+ struct percpu_counter_tree *counter = &mm->rss_stat[member];
+
+ if (!percpu_counter_tree_active(counter) && current == READ_ONCE(mm->rss_owner))
+ percpu_counter_tree_add_owner(counter, value);
+ else
+ percpu_counter_tree_add(counter, value);
+}
+
static inline void add_mm_counter(struct mm_struct *mm, int member, long value)
{
- percpu_counter_add(&mm->rss_stat[member], value);
+ __add_mm_counter(mm, member, value);
mm_trace_rss_stat(mm, member);
}
static inline void inc_mm_counter(struct mm_struct *mm, int member)
{
- percpu_counter_inc(&mm->rss_stat[member]);
+ __add_mm_counter(mm, member, 1);
mm_trace_rss_stat(mm, member);
}
static inline void dec_mm_counter(struct mm_struct *mm, int member)
{
- percpu_counter_dec(&mm->rss_stat[member]);
+ __add_mm_counter(mm, member, -1);
mm_trace_rss_stat(mm, member);
}
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index 1ed35d258d2e36..f0d315d72adb02 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -19,6 +19,7 @@
#include <linux/workqueue.h>
#include <linux/seqlock.h>
#include <linux/percpu_counter.h>
+#include <linux/percpu_counter_tree.h>
#include <linux/types.h>
#include <linux/futex_types.h>
#include <linux/rseq_types.h>
@@ -1350,7 +1351,23 @@ struct mm_struct {
unsigned long saved_e_flags;
#endif
- struct percpu_counter rss_stat[NR_MM_COUNTERS];
+ struct percpu_counter_tree rss_stat[NR_MM_COUNTERS];
+ /*
+ * Task updating rss_stat with percpu_counter_tree_add_owner()
+ * until the counter tree is activated by a second thread: the
+ * task running on this mm, set by dup_mm() and exec_mmap(),
+ * cleared when that task exits or execs, NULL for the other
+ * users of mm_alloc(). Only compared with current, never
+ * dereferenced. Not mm->owner, which depends on CONFIG_MEMCG
+ * and which mm_update_next_owner() hands over to another task
+ * while a vfork() child may still update the counters.
+ */
+ struct task_struct *rss_owner;
+ /*
+ * Serializes the activations of the rss_stat counter tree,
+ * until the tree is active. See mm_activate_rss_stat().
+ */
+ struct mutex rss_stat_mutex;
struct linux_binfmt *binfmt;
diff --git a/include/linux/percpu_counter_tree.h b/include/linux/percpu_counter_tree.h
new file mode 100644
index 00000000000000..60ef1da22bd3fd
--- /dev/null
+++ b/include/linux/percpu_counter_tree.h
@@ -0,0 +1,365 @@
+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */
+/* SPDX-FileCopyrightText: 2025-2026 Mathieu Desnoyers <mathieu.desnoyers@efficios.com> */
+
+#ifndef _PERCPU_COUNTER_TREE_H
+#define _PERCPU_COUNTER_TREE_H
+
+#include <linux/bug.h>
+#include <linux/atomic.h>
+#include <linux/limits.h>
+#include <linux/percpu.h>
+
+/* Largest batch size: level 0 residuals are within (-batch, +batch) of a signed byte. */
+#define PERCPU_COUNTER_TREE_MAX_BATCH 128
+static_assert(PERCPU_COUNTER_TREE_MAX_BATCH - 1 <= S8_MAX);
+
+#ifdef CONFIG_SMP
+
+struct percpu_counter_tree {
+ /*
+ * Per-CPU split counters (tree level 0): residuals not propagated
+ * yet, within (-batch, +batch).
+ */
+ s8 __percpu *level0;
+ /*
+ * The batch size is the increment step at level 0 which triggers a
+ * carry propagation: a power of 2 between 2 and
+ * PERCPU_COUNTER_TREE_MAX_BATCH. It is stored next to level0, the
+ * other field read by the update fast path, so both usually share a
+ * cache line: this leaves a hole before items_stride.
+ */
+ unsigned char batch_order;
+ unsigned int items_stride; /* Distance between this counter's consecutive items. */
+ /*
+ * Approximate sum: this counter's item of the root, in the tree
+ * items allocated with the per-CPU counters. NULL until the tree of a
+ * counter initialized by percpu_counter_tree_init_many_lazy() is
+ * activated.
+ */
+ atomic_long_t *approx_sum;
+ /*
+ * Bias to apply to counter precise and approximate values. Before the
+ * tree is activated, it also accumulates the updates of
+ * percpu_counter_tree_add_owner(), written only by the owner.
+ */
+ long bias;
+ /*
+ * Updates of percpu_counter_tree_add() made before the tree is
+ * activated, also applied to the precise and approximate values.
+ */
+ atomic_long_t atomic_bias;
+};
+
+int percpu_counter_tree_init_many(struct percpu_counter_tree *counters,
+ unsigned int nr_counters, unsigned long batch_size,
+ gfp_t gfp_flags);
+int percpu_counter_tree_init(struct percpu_counter_tree *counter,
+ unsigned long batch_size, gfp_t gfp_flags);
+int percpu_counter_tree_init_many_lazy(struct percpu_counter_tree *counters,
+ unsigned int nr_counters, unsigned long batch_size);
+int percpu_counter_tree_activate_many(struct percpu_counter_tree *counters,
+ unsigned int nr_counters, gfp_t gfp_flags);
+void percpu_counter_tree_destroy_many(struct percpu_counter_tree *counters,
+ unsigned int nr_counters);
+void percpu_counter_tree_destroy(struct percpu_counter_tree *counter);
+void percpu_counter_tree_add(struct percpu_counter_tree *counter, long inc);
+void percpu_counter_tree_add_owner(struct percpu_counter_tree *counter, long inc);
+long percpu_counter_tree_precise_sum(struct percpu_counter_tree *counter);
+int percpu_counter_tree_approximate_compare(struct percpu_counter_tree *a,
+ struct percpu_counter_tree *b);
+int percpu_counter_tree_approximate_compare_value(struct percpu_counter_tree *counter, long v);
+int percpu_counter_tree_precise_compare(struct percpu_counter_tree *a,
+ struct percpu_counter_tree *b);
+int percpu_counter_tree_precise_compare_value(struct percpu_counter_tree *counter, long v);
+void percpu_counter_tree_set(struct percpu_counter_tree *counter, long v);
+int percpu_counter_tree_subsystem_init(void);
+
+/**
+ * percpu_counter_tree_active() - Query whether the tree of a counter is active.
+ * @counter: The counter to query.
+ *
+ * A counter initialized by percpu_counter_tree_init_many() is active. A
+ * counter initialized by percpu_counter_tree_init_many_lazy() becomes
+ * active when percpu_counter_tree_activate_many() publishes its tree, and
+ * stays active. Once active, percpu_counter_tree_add() and
+ * percpu_counter_tree_add_owner() both add to the tree, so a user can skip
+ * selecting between them.
+ *
+ * Return: true if the updates of @counter go to its tree.
+ */
+static inline
+bool percpu_counter_tree_active(struct percpu_counter_tree *counter)
+{
+ /* No ordering: the update functions and the sums read level0 again. */
+ return !!READ_ONCE(counter->level0);
+}
+
+/**
+ * percpu_counter_tree_approximate_sum() - Return approximate counter sum.
+ * @counter: The counter to sum.
+ *
+ * Querying the approximate sum is fast, but it is only accurate within
+ * the bound returned by percpu_counter_tree_approximate_accuracy().
+ * This is meant to be used when speed is preferred over accuracy.
+ *
+ * Return: The current approximate counter sum.
+ */
+static inline
+long percpu_counter_tree_approximate_sum(struct percpu_counter_tree *counter)
+{
+ /*
+ * Pairs with the release of approx_sum in
+ * percpu_counter_tree_activate_many(): the root is zeroed before its
+ * pointer, and the address dependency orders the read of the root
+ * after it, as for rcu_dereference().
+ */
+ atomic_long_t *approx_sum = READ_ONCE(counter->approx_sum);
+ unsigned long v = (unsigned long)READ_ONCE(counter->bias) +
+ (unsigned long)atomic_long_read(&counter->atomic_bias);
+
+ if (approx_sum)
+ v += (unsigned long)atomic_long_read(approx_sum);
+ return (long)v;
+}
+
+unsigned long percpu_counter_tree_approximate_accuracy(struct percpu_counter_tree *counter);
+
+#if IS_ENABLED(CONFIG_KUNIT)
+unsigned int percpu_counter_tree_check_config(void);
+unsigned int percpu_counter_tree_check_residuals(struct percpu_counter_tree *counter);
+void percpu_counter_tree_copy_state(struct percpu_counter_tree *dst,
+ struct percpu_counter_tree *src);
+#endif
+
+#else /* !CONFIG_SMP */
+
+struct percpu_counter_tree {
+ atomic_long_t count;
+};
+
+static inline
+int percpu_counter_tree_init_many(struct percpu_counter_tree *counters,
+ unsigned int nr_counters, unsigned long batch_size,
+ gfp_t gfp_flags)
+{
+ /*
+ * There is no tree to batch into on UP, so @batch_size has no
+ * effect. Validate it anyway so callers get the same result as
+ * on SMP.
+ */
+ if (WARN_ON(batch_size <= 1 || (batch_size & (batch_size - 1)) ||
+ batch_size > PERCPU_COUNTER_TREE_MAX_BATCH))
+ return -EINVAL;
+ for (unsigned int i = 0; i < nr_counters; i++)
+ atomic_long_set(&counters[i].count, 0);
+ return 0;
+}
+
+static inline
+int percpu_counter_tree_init(struct percpu_counter_tree *counter,
+ unsigned long batch_size, gfp_t gfp_flags)
+{
+ return percpu_counter_tree_init_many(counter, 1, batch_size, gfp_flags);
+}
+
+static inline
+int percpu_counter_tree_init_many_lazy(struct percpu_counter_tree *counters,
+ unsigned int nr_counters, unsigned long batch_size)
+{
+ return percpu_counter_tree_init_many(counters, nr_counters, batch_size, 0);
+}
+
+static inline
+int percpu_counter_tree_activate_many(struct percpu_counter_tree *counters,
+ unsigned int nr_counters, gfp_t gfp_flags)
+{
+ return 0;
+}
+
+static inline
+bool percpu_counter_tree_active(struct percpu_counter_tree *counter)
+{
+ return true;
+}
+
+static inline
+void percpu_counter_tree_destroy_many(struct percpu_counter_tree *counters,
+ unsigned int nr_counters)
+{
+}
+
+static inline
+void percpu_counter_tree_destroy(struct percpu_counter_tree *counter)
+{
+}
+
+static inline
+long percpu_counter_tree_precise_sum(struct percpu_counter_tree *counter)
+{
+ return atomic_long_read(&counter->count);
+}
+
+static inline
+int percpu_counter_tree_precise_compare(struct percpu_counter_tree *a,
+ struct percpu_counter_tree *b)
+{
+ /* Compare on the wrapped difference, as on SMP. */
+ long delta = (long)((unsigned long)percpu_counter_tree_precise_sum(a) -
+ (unsigned long)percpu_counter_tree_precise_sum(b));
+
+ if (delta < 0)
+ return -1;
+ if (delta > 0)
+ return 1;
+ return 0;
+}
+
+static inline
+int percpu_counter_tree_precise_compare_value(struct percpu_counter_tree *counter, long v)
+{
+ /* Compare on the wrapped difference, as on SMP. */
+ long delta = (long)((unsigned long)v -
+ (unsigned long)percpu_counter_tree_precise_sum(counter));
+
+ if (delta < 0)
+ return -1;
+ if (delta > 0)
+ return 1;
+ return 0;
+}
+
+static inline
+int percpu_counter_tree_approximate_compare(struct percpu_counter_tree *a,
+ struct percpu_counter_tree *b)
+{
+ return percpu_counter_tree_precise_compare(a, b);
+}
+
+static inline
+int percpu_counter_tree_approximate_compare_value(struct percpu_counter_tree *counter, long v)
+{
+ return percpu_counter_tree_precise_compare_value(counter, v);
+}
+
+static inline
+void percpu_counter_tree_set(struct percpu_counter_tree *counter, long v)
+{
+ atomic_long_set(&counter->count, v);
+}
+
+static inline
+unsigned long percpu_counter_tree_approximate_accuracy(struct percpu_counter_tree *counter)
+{
+ return 0;
+}
+
+static inline
+void percpu_counter_tree_add(struct percpu_counter_tree *counter, long inc)
+{
+ atomic_long_add(inc, &counter->count);
+}
+
+static inline
+void percpu_counter_tree_add_owner(struct percpu_counter_tree *counter, long inc)
+{
+ percpu_counter_tree_add(counter, inc);
+}
+
+static inline
+long percpu_counter_tree_approximate_sum(struct percpu_counter_tree *counter)
+{
+ return percpu_counter_tree_precise_sum(counter);
+}
+
+static inline
+int percpu_counter_tree_subsystem_init(void)
+{
+ return 0;
+}
+
+#endif /* CONFIG_SMP */
+
+/**
+ * percpu_counter_tree_approximate_sum_positive() - Return a positive approximate counter sum.
+ * @counter: The counter to sum.
+ *
+ * Return an approximate counter sum which is guaranteed to be greater
+ * or equal to 0.
+ *
+ * Return: The current positive approximate counter sum.
+ */
+static inline
+long percpu_counter_tree_approximate_sum_positive(struct percpu_counter_tree *counter)
+{
+ long v = percpu_counter_tree_approximate_sum(counter);
+
+ return v > 0 ? v : 0;
+}
+
+/**
+ * percpu_counter_tree_precise_sum_positive() - Return a positive precise counter sum.
+ * @counter: The counter to sum.
+ *
+ * Return a precise counter sum which is guaranteed to be greater
+ * or equal to 0.
+ *
+ * Return: The current positive precise counter sum.
+ */
+static inline
+long percpu_counter_tree_precise_sum_positive(struct percpu_counter_tree *counter)
+{
+ long v = percpu_counter_tree_precise_sum(counter);
+
+ return v > 0 ? v : 0;
+}
+
+/**
+ * percpu_counter_tree_approximate_min_max_range() - Return the min and max precise values.
+ * @approx_sum: Approximated sum.
+ * @accuracy: Tree accuracy, from percpu_counter_tree_approximate_accuracy().
+ * @precise_min: Minimum possible value for precise sum (output).
+ * @precise_max: Maximum possible value for precise sum (output).
+ *
+ * Calculate the minimum and maximum precise values for a given approximation
+ * and accuracy. The accuracy is a bound on a settled tree: one with no update
+ * in progress.
+ *
+ * The approximation is within the accuracy of the precise sum, in both
+ * directions:
+ *
+ * precise_sum - accuracy <= approx_sum <= precise_sum + accuracy
+ *
+ * Therefore, the precise sum is within the accuracy of the approximation:
+ *
+ * approx_sum - accuracy <= precise_sum <= approx_sum + accuracy
+ */
+static inline
+void percpu_counter_tree_approximate_min_max_range(long approx_sum, unsigned long accuracy,
+ long *precise_min, long *precise_max)
+{
+ *precise_min = approx_sum - accuracy;
+ *precise_max = approx_sum + accuracy;
+}
+
+/**
+ * percpu_counter_tree_approximate_min_max() - Return the approximation, min and max precise values.
+ * @counter: The counter to sum.
+ * @approx_sum: Approximate sum (output).
+ * @precise_min: Minimum possible value for precise sum (output).
+ * @precise_max: Maximum possible value for precise sum (output).
+ *
+ * Return the approximate sum, minimum and maximum precise values for
+ * a counter.
+ */
+static inline
+void percpu_counter_tree_approximate_min_max(struct percpu_counter_tree *counter,
+ long *approx_sum, long *precise_min, long *precise_max)
+{
+ unsigned long accuracy = percpu_counter_tree_approximate_accuracy(counter);
+ long v = percpu_counter_tree_approximate_sum(counter);
+
+ percpu_counter_tree_approximate_min_max_range(v, accuracy, precise_min, precise_max);
+ *approx_sum = v;
+}
+
+#endif /* _PERCPU_COUNTER_TREE_H */
diff --git a/include/trace/events/kmem.h b/include/trace/events/kmem.h
index cd7920c81f85ba..8ee6d5df596105 100644
--- a/include/trace/events/kmem.h
+++ b/include/trace/events/kmem.h
@@ -448,8 +448,7 @@ TRACE_EVENT(rss_stat,
*/
__entry->curr = current->mm == mm && !(current->flags & PF_KTHREAD);
__entry->member = member;
- __entry->size = (percpu_counter_sum_positive(&mm->rss_stat[member])
- << PAGE_SHIFT);
+ __entry->size = get_mm_counter(mm, member) << PAGE_SHIFT;
),
TP_printk("mm_id=%u curr=%d type=%s size=%ldB",
diff --git a/init/main.c b/init/main.c
index 7b0c8c34e6a911..d9614cc5bd63d2 100644
--- a/init/main.c
+++ b/init/main.c
@@ -108,6 +108,7 @@
#include <linux/time_namespace.h>
#include <linux/unaligned.h>
#include <linux/vdso_datastore.h>
+#include <linux/percpu_counter_tree.h>
#include <net/net_namespace.h>
#include <asm/io.h>
@@ -1104,6 +1105,8 @@ void start_kernel(void)
sort_main_extable();
trap_init();
mm_core_init();
+ /* After parsing its boot parameters and setting up slab, before the first counter. */
+ percpu_counter_tree_subsystem_init();
maple_tree_init();
poking_init();
ftrace_init();
diff --git a/kernel/exit.c b/kernel/exit.c
index d5e7bc2af804a6..c153cd064987c6 100644
--- a/kernel/exit.c
+++ b/kernel/exit.c
@@ -582,6 +582,9 @@ static void exit_mm(void)
local_irq_enable();
task_unlock(current);
mmap_read_unlock(mm);
+ /* From now on, this task updates the RSS counters of @mm atomically. */
+ if (READ_ONCE(mm->rss_owner) == current)
+ WRITE_ONCE(mm->rss_owner, NULL);
mm_update_next_owner(mm);
mmput(mm);
if (test_thread_flag(TIF_MEMDIE))
diff --git a/kernel/fork.c b/kernel/fork.c
index 8b108ed371529c..d0641700cd9c65 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -137,6 +137,28 @@
*/
#define MAX_THREADS FUTEX_TID_MASK
+/*
+ * Batch size of rss stat approximation, tunable with the rss_stat_batch=
+ * boot parameter to compare batch sizes on a given machine.
+ */
+static unsigned int rss_stat_batch_size __ro_after_init = 32;
+
+static int __init setup_rss_stat_batch(char *str)
+{
+ unsigned int batch;
+
+ if (kstrtouint(str, 0, &batch) || batch < 2 || !is_power_of_2(batch) ||
+ batch > PERCPU_COUNTER_TREE_MAX_BATCH) {
+ pr_warn("rss_stat_batch: expecting a power of 2 between 2 and %u\n",
+ PERCPU_COUNTER_TREE_MAX_BATCH);
+ return 1;
+ }
+ rss_stat_batch_size = batch;
+ pr_info("rss_stat_batch: %u\n", batch);
+ return 1;
+}
+__setup("rss_stat_batch=", setup_rss_stat_batch);
+
/*
* Protected counters by write_lock_irq(&tasklist_lock)
*/
@@ -633,7 +655,7 @@ static void check_mm(struct mm_struct *mm)
"Please make sure 'struct resident_page_types[]' is updated as well");
for (i = 0; i < NR_MM_COUNTERS; i++) {
- long x = percpu_counter_sum(&mm->rss_stat[i]);
+ long x = percpu_counter_tree_precise_sum(&mm->rss_stat[i]);
if (unlikely(x)) {
pr_alert("BUG: Bad rss-counter state mm:%p type:%s val:%ld Comm:%s Pid:%d\n",
@@ -738,7 +760,7 @@ void __mmdrop(struct mm_struct *mm)
check_mm(mm);
mm_pasid_drop(mm);
mm_destroy_cid(mm);
- percpu_counter_destroy_many(mm->rss_stat, NR_MM_COUNTERS);
+ percpu_counter_tree_destroy_many(mm->rss_stat, NR_MM_COUNTERS);
free_mm(mm);
}
@@ -1103,6 +1125,7 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p)
memset(&mm->rss_stat, 0, sizeof(mm->rss_stat));
spin_lock_init(&mm->page_table_lock);
spin_lock_init(&mm->arg_lock);
+ mutex_init(&mm->rss_stat_mutex);
mm_init_cpumask(mm);
mm_init_aio(mm);
mm_init_owner(mm, p);
@@ -1143,8 +1166,7 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p)
if (mm_alloc_sched(mm))
goto fail_sched;
- if (percpu_counter_init_many(mm->rss_stat, 0, GFP_KERNEL_ACCOUNT,
- NR_MM_COUNTERS))
+ if (percpu_counter_tree_init_many_lazy(mm->rss_stat, NR_MM_COUNTERS, rss_stat_batch_size))
goto fail_pcpu;
lru_gen_init_mm(mm);
@@ -1537,6 +1559,8 @@ static struct mm_struct *dup_mm(struct task_struct *tsk,
if (!mm_init(mm, tsk))
goto fail_nomem;
+ /* The child owns its RSS counters, see __add_mm_counter(). */
+ mm->rss_owner = tsk;
uprobe_start_dup_mmap();
err = dup_mmap(mm, oldmm);
@@ -1564,6 +1588,23 @@ static struct mm_struct *dup_mm(struct task_struct *tsk,
return NULL;
}
+/*
+ * Serialize the activations of the RSS counter tree, whichever tasks create
+ * threads. Once the last counter is active, the activation is complete and
+ * thread creation skips the mutex. That check needs no acquire: the updates
+ * and the sums of the counters order their own accesses to the tree. Not
+ * observing the other counters active yet only delays their updates going
+ * to the tree.
+ */
+static int mm_activate_rss_stat(struct mm_struct *mm)
+{
+ if (percpu_counter_tree_active(&mm->rss_stat[NR_MM_COUNTERS - 1]))
+ return 0;
+ guard(mutex)(&mm->rss_stat_mutex);
+ return percpu_counter_tree_activate_many(mm->rss_stat, NR_MM_COUNTERS,
+ GFP_KERNEL_ACCOUNT);
+}
+
static int copy_mm(u64 clone_flags, struct task_struct *tsk)
{
struct mm_struct *mm, *oldmm;
@@ -1588,6 +1629,18 @@ static int copy_mm(u64 clone_flags, struct task_struct *tsk)
return 0;
if (clone_flags & CLONE_VM) {
+ int err = 0;
+
+ /*
+ * A second thread shares the mm: activate the RSS counter
+ * tree. A vfork child shares the mm while the parent waits,
+ * so the single counters still fit. The tree stays active if
+ * the clone fails later, which is harmless.
+ */
+ if (!(clone_flags & CLONE_VFORK))
+ err = mm_activate_rss_stat(oldmm);
+ if (err)
+ return err;
mmget(oldmm);
mm = oldmm;
} else {
diff --git a/lib/Kconfig b/lib/Kconfig
index 1e42f5167c68bc..6ab9644a4cb830 100644
--- a/lib/Kconfig
+++ b/lib/Kconfig
@@ -41,6 +41,33 @@ config PACKING_KUNIT_TEST
When in doubt, say N.
+config PERCPU_COUNTER_TREE_KUNIT_TEST
+ tristate "KUnit tests for hierarchical per-cpu counters" if !KUNIT_ALL_TESTS
+ depends on KUNIT
+ default KUNIT_ALL_TESTS
+ help
+ This builds Kunit tests for the hierarchical per-cpu counters.
+
+ For more information on KUnit and unit tests in general,
+ please refer to the KUnit documentation in Documentation/dev-tools/kunit/.
+
+ When in doubt, say N.
+
+config PERCPU_COUNTER_TREE_BENCH
+ tristate "Benchmark of hierarchical per-cpu counters"
+ depends on KUNIT && SMP
+ help
+ This builds a KUnit benchmark comparing the hierarchical per-cpu
+ counters with struct percpu_counter: update cost with all online
+ CPUs updating shared and private counters, and init/destroy cost.
+ It runs for a few minutes on large machines.
+
+ The batch size and the tree shape can be changed with the
+ percpu_counter_tree_bench.batch= and percpu_counter_tree.arity=
+ parameters, see lib/tests/percpu_counter_tree_bench.c.
+
+ When in doubt, say N.
+
config BITREVERSE
tristate
select GENERIC_BITREVERSE if !HAVE_ARCH_BITREVERSE
diff --git a/lib/Makefile b/lib/Makefile
index a2afe954805cb8..07410b1d20a379 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -182,6 +182,7 @@ obj-$(CONFIG_TEXTSEARCH_KMP) += ts_kmp.o
obj-$(CONFIG_TEXTSEARCH_BM) += ts_bm.o
obj-$(CONFIG_TEXTSEARCH_FSM) += ts_fsm.o
obj-$(CONFIG_SMP) += percpu_counter.o
+obj-$(CONFIG_SMP) += percpu_counter_tree.o
obj-$(CONFIG_AUDIT_GENERIC) += audit.o
obj-$(CONFIG_AUDIT_COMPAT_GENERIC) += compat_audit.o
diff --git a/lib/percpu_counter_tree.c b/lib/percpu_counter_tree.c
new file mode 100644
index 00000000000000..491675897e41c2
--- /dev/null
+++ b/lib/percpu_counter_tree.c
@@ -0,0 +1,1450 @@
+// SPDX-License-Identifier: GPL-2.0+ OR MIT
+// SPDX-FileCopyrightText: 2025-2026 Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
+
+/*
+ * Split Counters With Tree Approximation Propagation
+ *
+ * * Overview
+ *
+ * Each counter is a tree of counters. A CPU updates its own level 0
+ * counter, a signed byte. Each counter keeps the residual it has not
+ * propagated yet, and propagates a carry to its parent on the next level
+ * once that residual reaches the batch size of its level, up to the root.
+ * Most updates therefore stay on the updating CPU. The root holds the
+ * approximate sum, read in O(1), and the precise sum adds the residuals of
+ * all levels to it.
+ *
+ * * Tree shape
+ *
+ * Illustration with 8 CPUs and a fan-out of 2 at each level, which makes
+ * 3 levels of counters below the root. This is not an actual
+ * configuration: at 8 CPUs, the default tree is flat.
+ *
+ * Level 3: 0 (root)
+ * | \
+ * | \
+ * | \
+ * Level 2: 0 1
+ * | \ | \
+ * | \ | \
+ * | \ | \
+ * Level 1: 0 2 1 3
+ * | \ | \ | \ | \
+ * | \ | \ | \ | \
+ * | \ | \ | \ | \
+ * Level 0: 0 4 2 6 1 5 3 7
+ *
+ * CPU c propagates into item (c & (L - 1)) of a level holding L items,
+ * so the CPUs sharing an item are not adjacent: CPUs 0 and 4 share
+ * level 1 item 0, and CPUs 0, 2, 4, 6 share level 2 item 0. This mapping
+ * uses every item of each level and gives each item the same number of
+ * CPUs, give or take one, even when nr_cpu_ids is not a power of 2. The
+ * diagram lists the CPUs in bit-reversed order to avoid crossing lines.
+ *
+ * The actual fan-out of each level depends on the topology (see
+ * per_nr_cpu_order_config): up to 32 CPUs, the tree is flat, and the
+ * level 0 counters propagate directly to the root.
+ *
+ * * Batch sizes and residuals
+ *
+ * The batch size of a level, BATCH(level N), is the batch size of the
+ * level below multiplied by the fan-out of that level. Example for
+ * BATCH(level 0) = 32, with the tree of the illustration above:
+ *
+ * BATCH(level 0) = 32
+ * BATCH(level 1) = 64
+ * BATCH(level 2) = 128
+ *
+ * Once the residual of a counter reaches +/- BATCH(level N), it propagates
+ * its multiples of BATCH(level N), truncated toward zero, and subtracts
+ * them. This hysteresis prevents updates going back and forth around a
+ * BATCH(level N) boundary from propagating up the tree each time.
+ *
+ * The residual of a settled counter therefore stays within
+ * (-BATCH(level N), +BATCH(level N)), and moves in steps of the counter's
+ * input: level 0 receives the updates, in steps of 1, while a counter
+ * above level 0 only receives the carries of the level below, in steps of
+ * BATCH(level N - 1). The largest residual of a counter is the largest
+ * multiple of its input step below its batch size: BATCH(level 0) - 1 at
+ * level 0, and BATCH(level N) - BATCH(level N - 1) above:
+ *
+ * input residual worst case worst case
+ * step per counter below precise above precise
+ * Level 0: 1 [-31 .. 31] 248 (8 * 31) 248 (8 * 31)
+ * Level 1: 32 {-32, 0, 32} 128 (4 * 32) 128 (4 * 32)
+ * Level 2: 64 {-64, 0, 64} 128 (2 * 64) 128 (2 * 64)
+ * Total: 504 504
+ *
+ * * Approximation accuracy
+ *
+ * The approximate sum is the precise sum minus the residuals, so a
+ * settled counter (no update in progress) is within:
+ *
+ * precise - 504 <= approx <= precise + 504
+ *
+ * The accuracy returned by percpu_counter_tree_approximate_accuracy() is
+ * exact. It adds the largest residuals of all the counters:
+ * BATCH(level 0) - 1 for each of the nr_cpu_ids level 0 counters, and
+ * BATCH(level N) - BATCH(level N - 1) for each counter of level N above.
+ * Counting each level 0 counter as one BATCH(level 0), minus its input
+ * step of 1:
+ *
+ * accuracy_multiplier = nr_cpu_ids + sum over levels N >= 1 of
+ * (number of counters at level N) *
+ * (BATCH(level N) - BATCH(level N - 1)) / BATCH(level 0)
+ *
+ * accuracy = BATCH(level 0) * accuracy_multiplier - nr_cpu_ids
+ *
+ * In the example above, accuracy_multiplier = 8 + 4 * 1 + 2 * 2 = 16, and:
+ *
+ * accuracy = 32 * 16 - 8 = 504
+ *
+ * Every counter above level 0 is reachable from a CPU below nr_cpu_ids,
+ * even when nr_cpu_ids is not a power of 2, so both bounds can be reached
+ * when the possible CPUs are contiguous. When they are sparse, the bound
+ * also counts the level 0 residuals of the missing CPUs: it holds, but
+ * cannot be reached.
+ *
+ * * Carry propagation
+ *
+ * The items at each tree position of the counters initialized together
+ * share cache lines, which are not shared with other tree positions to
+ * prevent false sharing between CPUs.
+ *
+ * Example with expanded values, for the tree and batch sizes above:
+ *
+ * counter_add(counter, inc):
+ *
+ * if (!inc)
+ * return;
+ *
+ * // Level 0: per-CPU residual within (-32, +32).
+ * old = percpu_read(counter @ Level 0);
+ * do {
+ * v = old + inc;
+ * out = v < 0 ? -(-v & ~0b00011111) : (v & ~0b00011111);
+ * } while (!percpu_try_cmpxchg(counter @ Level 0, &old, v - out));
+ * inc = out;
+ * if (!inc)
+ * return;
+ *
+ * // Levels 1 and 2: add the carry, then extract the multiples
+ * // of the level batch held by the counter, truncated toward
+ * // zero. A failed cmpxchg updates res to the current residual.
+ * res = atomic_long_add_return(counter @ Level 1, inc);
+ * do {
+ * inc = res < 0 ? -(-res & ~0b00111111) : (res & ~0b00111111);
+ * if (!inc)
+ * return;
+ * } while (!atomic_long_try_cmpxchg(counter @ Level 1, &res, res - inc));
+ *
+ * res = atomic_long_add_return(counter @ Level 2, inc);
+ * do {
+ * inc = res < 0 ? -(-res & ~0b01111111) : (res & ~0b01111111);
+ * if (!inc)
+ * return;
+ * } while (!atomic_long_try_cmpxchg(counter @ Level 2, &res, res - inc));
+ *
+ * atomic_long_add(counter @ Level 3, inc);
+ */
+
+#define pr_fmt(fmt) "percpu_counter_tree: " fmt
+
+#include <linux/percpu_counter_tree.h>
+#include <linux/moduleparam.h>
+#include <linux/align.h>
+#include <linux/cache.h>
+#include <linux/cpumask.h>
+#include <linux/atomic.h>
+#include <linux/export.h>
+#include <linux/percpu.h>
+#include <linux/errno.h>
+#include <linux/slab.h>
+#include <linux/math.h>
+#include <linux/log2.h>
+#include <linux/args.h>
+#include <linux/string.h>
+#include <kunit/visibility.h>
+
+#define MAX_NR_LEVELS 7
+
+/*
+ * The counter configuration is selected at boot time based on the
+ * hardware topology.
+ */
+struct counter_config {
+ /*
+ * nr_levels is the number of hierarchical counter tree levels. It
+ * excludes the final level (approximate sum).
+ */
+ unsigned char nr_levels;
+ /* n-arity of each level from 0 to (nr_levels - 1). */
+ unsigned char n_arity_order[MAX_NR_LEVELS];
+};
+
+/*
+ * The fan-out of each level of the tree is chosen with a heuristic trying to
+ * minimize both the tree depth and the fan-out.
+ */
+#define PCT_COUNTER_CONFIG(order, ...) \
+ [order] = { \
+ .nr_levels = COUNT_ARGS(__VA_ARGS__), \
+ .n_arity_order = { __VA_ARGS__ }, \
+ }
+
+/*
+ * Up to 32 CPUs, the tree is flat: the level 0 counters propagate directly
+ * to the root, like struct percpu_counter. From 33 to 64 CPUs, the level 0
+ * counters propagate to 4 intermediate counters, each shared by up to 16
+ * CPUs, so the carries of all CPUs do not contend on the root. Beyond, each
+ * level has a fan-out of 8. When the order is not a multiple of 3, a
+ * remaining order of 1 gives the root a fan-out of 16, and a remaining
+ * order of 2 adds a level with a fan-out of 4 below the root: a root
+ * fan-out of 16 updates about as fast as a fan-out of 2 below the root,
+ * with one level less, which shrinks the accuracy bound and the tree
+ * items, while a root fan-out of 32 is slower than a fan-out of 4 below
+ * it. This keeps the carry rate reaching each tree item low with the RSS
+ * batch size of 32, and the approximation error grows as
+ * O(nr_cpus * log(nr_cpus)), compared to O(nr_cpus^2) for struct
+ * percpu_counter whose batch size grows with the number of CPUs.
+ *
+ * The tree is shaped for nr_cpu_ids rounded up to a power of 2, and each
+ * tree position takes a cache line. A set of 4 counters initialized together,
+ * such as the RSS counters of an mm, uses less memory than 4 struct
+ * percpu_counter from a few CPUs up to 64, and beyond when nr_cpu_ids is a
+ * power of 2 (at most 0.89 times as much). Between powers of 2, fewer CPUs
+ * share the cost of the tree items: up to 1.1 times at 3 * 2^n CPUs, and
+ * up to 1.48 times just above a power of 2. A lone counter uses one item
+ * of each cache line: 3.0 to 3.5 times the memory of a struct
+ * percpu_counter when nr_cpu_ids is a power of 2 beyond 64, and up to 5.7
+ * times just above.
+ */
+static const struct counter_config per_nr_cpu_order_config[] = {
+ PCT_COUNTER_CONFIG(0),
+ PCT_COUNTER_CONFIG(1, 1),
+ PCT_COUNTER_CONFIG(2, 2),
+ PCT_COUNTER_CONFIG(3, 3),
+ PCT_COUNTER_CONFIG(4, 4),
+ PCT_COUNTER_CONFIG(5, 5),
+ PCT_COUNTER_CONFIG(6, 4, 2),
+ PCT_COUNTER_CONFIG(7, 3, 4),
+ PCT_COUNTER_CONFIG(8, 3, 2, 3),
+ PCT_COUNTER_CONFIG(9, 3, 3, 3),
+ PCT_COUNTER_CONFIG(10, 3, 3, 4),
+ PCT_COUNTER_CONFIG(11, 3, 3, 2, 3),
+ PCT_COUNTER_CONFIG(12, 3, 3, 3, 3),
+ PCT_COUNTER_CONFIG(13, 3, 3, 3, 4),
+ PCT_COUNTER_CONFIG(14, 3, 3, 3, 2, 3),
+ PCT_COUNTER_CONFIG(15, 3, 3, 3, 3, 3),
+ PCT_COUNTER_CONFIG(16, 3, 3, 3, 3, 4),
+ PCT_COUNTER_CONFIG(17, 3, 3, 3, 3, 2, 3),
+ PCT_COUNTER_CONFIG(18, 3, 3, 3, 3, 3, 3),
+ PCT_COUNTER_CONFIG(19, 3, 3, 3, 3, 3, 4),
+ PCT_COUNTER_CONFIG(20, 3, 3, 3, 3, 3, 2, 3),
+};
+
+#undef PCT_COUNTER_CONFIG
+
+/* The configuration table covers every possible number of CPUs. */
+#define MAX_NR_CPUS_ORDER 20
+#if NR_CPUS > (1UL << MAX_NR_CPUS_ORDER)
+#error "NR_CPUS exceeds the hierarchical counter configuration table"
+#endif
+static_assert(ARRAY_SIZE(per_nr_cpu_order_config) == MAX_NR_CPUS_ORDER + 1);
+
+/* Tree topology, selected at boot and shared by all the counters. */
+static struct {
+ /* Hierarchical counter configuration for the hardware topology. */
+ const struct counter_config *config;
+ /* Tree shape selected with the "arity" boot parameter. */
+ struct counter_config boot_config;
+ /* Order of nr_cpu_ids. */
+ unsigned int nr_cpus_order;
+ /*
+ * Tree items per counter, including the root. UINT_MAX catches early
+ * boot misuses.
+ */
+ unsigned int nr_items;
+ /*
+ * Largest residuals of all the counters, in units of the level 0
+ * batch size: see percpu_counter_tree_accuracy().
+ */
+ unsigned long accuracy_multiplier;
+ /* Tree items when those of a tree position fit in one cache line. */
+ struct kmem_cache *items_cachep;
+} tree_topology __ro_after_init = {
+ .nr_items = UINT_MAX,
+};
+
+/* Value of the "arity" boot parameter, parsed at subsystem initialization. */
+static const char *arity_param;
+
+/*
+ * Size of the items of one tree position for the counters initialized
+ * together by percpu_counter_tree_init_many(), packed in cache lines which
+ * are not shared with other tree positions. Each item is the residual of
+ * the carries received and not propagated yet, within (-batch, +batch) of
+ * its tree level when settled. The items of a tree position are updated
+ * by the same CPUs, so packing them does not add contention between
+ * unrelated CPUs.
+ */
+static
+size_t percpu_counter_tree_position_size(unsigned int nr_counters)
+{
+ return ALIGN(nr_counters * sizeof(atomic_long_t), SMP_CACHE_BYTES);
+}
+
+/* Distance between the consecutive items of each of @nr_counters counters. */
+static inline
+unsigned int percpu_counter_tree_items_stride(unsigned int nr_counters)
+{
+ return percpu_counter_tree_position_size(nr_counters) / sizeof(atomic_long_t);
+}
+
+/* Allocate the zeroed tree items of @nr_counters counters, the root last. */
+static
+atomic_long_t *percpu_counter_tree_alloc_items(unsigned int nr_counters, gfp_t gfp_flags)
+{
+ size_t position_size = percpu_counter_tree_position_size(nr_counters);
+
+ if (position_size == SMP_CACHE_BYTES)
+ return kmem_cache_zalloc(tree_topology.items_cachep, gfp_flags);
+ /* kmalloc() aligns power of 2 sizes on their size. */
+ return kzalloc(roundup_pow_of_two(tree_topology.nr_items * position_size),
+ gfp_flags);
+}
+
+static
+void percpu_counter_tree_free_items(atomic_long_t *items, unsigned int nr_counters)
+{
+ if (percpu_counter_tree_position_size(nr_counters) == SMP_CACHE_BYTES)
+ kmem_cache_free(tree_topology.items_cachep, items);
+ else
+ kfree(items);
+}
+
+static
+int __percpu_counter_tree_init(struct percpu_counter_tree *counter,
+ unsigned long batch_size,
+ s8 __percpu *level0,
+ atomic_long_t *items, unsigned int items_stride)
+{
+ /* Batch size must be greater than 1, a power of 2, and fit in a byte. */
+ if (WARN_ON(batch_size <= 1 || (batch_size & (batch_size - 1)) ||
+ batch_size > PERCPU_COUNTER_TREE_MAX_BATCH))
+ return -EINVAL;
+ counter->batch_order = ilog2(batch_size);
+ counter->bias = 0;
+ atomic_long_set(&counter->atomic_bias, 0);
+ counter->level0 = level0;
+ counter->items_stride = items_stride;
+ /* The root is the last item, allocated by the activation if lazy. */
+ counter->approx_sum = items ?
+ &items[(tree_topology.nr_items - 1) * items_stride] : NULL;
+ return 0;
+}
+
+static inline
+unsigned long percpu_counter_tree_batch_size(struct percpu_counter_tree *counter)
+{
+ return 1UL << counter->batch_order;
+}
+
+/* This counter's first item. */
+static inline
+atomic_long_t *percpu_counter_tree_items(struct percpu_counter_tree *counter)
+{
+ return counter->approx_sum - (tree_topology.nr_items - 1) * counter->items_stride;
+}
+
+/*
+ * The residuals of all levels are within (-batch, +batch) of their level,
+ * so the approximation is within the same distance of the precise sum in
+ * both directions: the largest residuals of all the counters, which
+ * accuracy_multiplier counts in units of the level 0 batch size, minus the
+ * input step of 1 of each level 0 counter. See "Approximation accuracy" at
+ * the top of this file.
+ */
+static inline
+unsigned long percpu_counter_tree_accuracy(struct percpu_counter_tree *counter)
+{
+ unsigned long batch_size = percpu_counter_tree_batch_size(counter);
+
+ return batch_size * tree_topology.accuracy_multiplier - nr_cpu_ids;
+}
+
+/* Initialize @nr_counters counters, with their per-CPU counters and tree items, or NULL if lazy. */
+static
+int __percpu_counter_tree_init_many(struct percpu_counter_tree *counters,
+ unsigned int nr_counters, unsigned long batch_size,
+ s8 __percpu *level0, atomic_long_t *items)
+{
+ unsigned int items_stride = percpu_counter_tree_items_stride(nr_counters);
+ unsigned int i;
+ int ret;
+
+ for (i = 0; i < nr_counters; i++) {
+ ret = __percpu_counter_tree_init(&counters[i], batch_size,
+ level0 ? level0 + i : NULL,
+ items ? items + i : NULL, items_stride);
+ if (ret)
+ return ret;
+ }
+ return 0;
+}
+
+/**
+ * percpu_counter_tree_init_many() - Initialize many per-CPU counter trees.
+ * @counters: An array of @nr_counters counters to initialize.
+ * Their memory is provided by the caller.
+ * @nr_counters: The number of counter trees to initialize
+ * @batch_size: The batch size is the increment step at level 0 which triggers a
+ * carry propagation.
+ * The batch size is required to be greater than 1, a power of 2, and at
+ * most PERCPU_COUNTER_TREE_MAX_BATCH.
+ * @gfp_flags: gfp flags to pass to the per-CPU and tree item allocators.
+ *
+ * Initialize many per-CPU counter trees, with a single allocation of
+ * per-CPU counters and a single allocation of tree items for the
+ * @nr_counters counters. The items at each tree position of the
+ * @nr_counters counters share cache lines. When they fit in one cache
+ * line (up to 8 counters on 64-bit with 64-byte cache lines), the tree
+ * items come from a slab cache; beyond, their allocation is
+ * rounded up to a power of 2, which can double it.
+ *
+ * Return:
+ * * %0: Success
+ * * %-EINVAL: - Invalid @batch_size argument
+ * * %-ENOMEM: - Out of memory
+ */
+int percpu_counter_tree_init_many(struct percpu_counter_tree *counters,
+ unsigned int nr_counters, unsigned long batch_size,
+ gfp_t gfp_flags)
+{
+ s8 __percpu *level0;
+ atomic_long_t *items;
+ int ret;
+
+ /* The tree shape is selected by percpu_counter_tree_subsystem_init() at boot. */
+ if (WARN_ON_ONCE(tree_topology.nr_items == UINT_MAX))
+ return -EINVAL;
+ level0 = __alloc_percpu_gfp(nr_counters * sizeof(*level0),
+ __alignof__(*counters->level0), gfp_flags);
+ if (!level0)
+ return -ENOMEM;
+ items = percpu_counter_tree_alloc_items(nr_counters, gfp_flags);
+ if (!items) {
+ ret = -ENOMEM;
+ goto free_level0;
+ }
+ ret = __percpu_counter_tree_init_many(counters, nr_counters, batch_size, level0, items);
+ if (!ret)
+ return 0;
+ percpu_counter_tree_free_items(items, nr_counters);
+free_level0:
+ free_percpu(level0);
+ return ret;
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_init_many);
+
+/**
+ * percpu_counter_tree_init() - Initialize one per-CPU counter tree.
+ * @counter: Counter to initialize.
+ * Its memory is provided by the caller.
+ * @batch_size: The batch size is the increment step at level 0 which triggers a
+ * carry propagation.
+ * The batch size is required to be greater than 1, a power of 2, and at
+ * most PERCPU_COUNTER_TREE_MAX_BATCH.
+ * @gfp_flags: gfp flags to pass to the per-CPU and tree item allocators.
+ *
+ * Initialize one per-CPU counter tree.
+ *
+ * Return:
+ * * %0: Success
+ * * %-EINVAL: - Invalid @batch_size argument
+ * * %-ENOMEM: - Out of memory
+ */
+int percpu_counter_tree_init(struct percpu_counter_tree *counter,
+ unsigned long batch_size, gfp_t gfp_flags)
+{
+ return percpu_counter_tree_init_many(counter, 1, batch_size, gfp_flags);
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_init);
+
+/**
+ * percpu_counter_tree_init_many_lazy() - Initialize many counters, without their tree.
+ * @counters: An array of @nr_counters counters to initialize.
+ * Their memory is provided by the caller.
+ * @nr_counters: The number of counter trees to initialize
+ * @batch_size: The batch size, as for percpu_counter_tree_init_many().
+ *
+ * Initialize counters without allocating their per-CPU counters and tree
+ * items, for counters which may never need the scalability of the tree,
+ * such as the RSS counters of a single-threaded process. Until
+ * percpu_counter_tree_activate_many() allocates the tree, the counters
+ * are exact: percpu_counter_tree_add_owner() adds to the bias, written
+ * only by the owner, percpu_counter_tree_add() adds to the atomic bias,
+ * and the precise and approximate sums read both in O(1).
+ * percpu_counter_tree_destroy_many() and percpu_counter_tree_set() work on
+ * counters which were never activated. The accuracy range of a lazily
+ * initialized counter is the range of its tree, even before activation,
+ * which keeps the comparison functions correct across an activation.
+ * The batch size is fixed here: the activation only publishes the per-CPU
+ * counters and the tree, so updaters read the batch size without ordering.
+ *
+ * Return:
+ * * %0: Success
+ * * %-EINVAL: - Invalid @batch_size argument
+ */
+int percpu_counter_tree_init_many_lazy(struct percpu_counter_tree *counters,
+ unsigned int nr_counters, unsigned long batch_size)
+{
+ /* The tree shape is selected by percpu_counter_tree_subsystem_init() at boot. */
+ if (WARN_ON_ONCE(tree_topology.nr_items == UINT_MAX))
+ return -EINVAL;
+ return __percpu_counter_tree_init_many(counters, nr_counters, batch_size, NULL, NULL);
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_init_many_lazy);
+
+/*
+ * Publication of the tree
+ *
+ * The updaters and the readers of a counter take no lock, and may run
+ * while percpu_counter_tree_activate_many() publishes its tree. The
+ * activation allocates the per-CPU counters and the tree items zeroed,
+ * then publishes two pointers per counter, each with a store-release:
+ *
+ * - the release of approx_sum: the root is zeroed before its pointer,
+ * - the release of level0, after it: the per-CPU counters, the tree items
+ * and approx_sum are set before level0.
+ *
+ * level0 opens the tree to updates: until it is published, the tree is
+ * empty. Each load of a published pointer pairs with one of the releases:
+ *
+ * - Update fast path, percpu_counter_tree_add() and
+ * percpu_counter_tree_add_owner(): READ_ONCE(level0), then the per-CPU
+ * counters through it. The address dependency pairs with the release of
+ * level0, as for rcu_dereference().
+ * - Carry slow path: smp_rmb() after the read of level0, then
+ * READ_ONCE(approx_sum). Pairs with the release of level0, which
+ * follows the store of approx_sum.
+ * - Precise sum: smp_load_acquire(level0), then approx_sum, the tree items
+ * and the per-CPU counters. Pairs with the release of level0.
+ * - Approximate sum: READ_ONCE(approx_sum), then the root through it. The
+ * address dependency pairs with the release of approx_sum.
+ * - percpu_counter_tree_active(): READ_ONCE(level0), without ordering. It
+ * only selects the update function, which reads level0 again, or lets
+ * the caller skip an activation which is complete.
+ * - Activations: serialized by the caller, for instance with a mutex,
+ * which orders a complete activation before the check of level0 at the
+ * start of the next one.
+ *
+ * The update fast path has no acquire. smp_load_acquire() is free on
+ * strongly ordered architectures such as x86, but on weakly ordered ones,
+ * such as arm64, powerpc and RISC-V, it costs an ordering load or a barrier
+ * on every update, to order a publication which happens at most once in
+ * the life of a counter. The address dependency orders the per-CPU
+ * counters, reached through level0, at no cost. It does not order
+ * approx_sum, a separate field: the carry slow path, taken only when an
+ * update carries, pays an smp_rmb() to read it, and the precise sum, which
+ * reads the whole tree, reads level0 with an acquire.
+ *
+ * Missing the publication is harmless. An updater which reads level0 as
+ * NULL adds to the bias (owner) or to the atomic bias (others), and a
+ * reader which reads a NULL pointer sums the bias and the atomic bias:
+ * the sums keep reading both after the activation, and the tree receives
+ * no update before level0 is published.
+ *
+ * The read barriers between the levels of the precise sum are unrelated
+ * to the publication: they order carries, see
+ * percpu_counter_tree_precise_sum_unbiased().
+ */
+
+/**
+ * percpu_counter_tree_activate_many() - Activate the tree of lazily initialized counters.
+ * @counters: The array of @nr_counters counters passed to
+ * percpu_counter_tree_init_many_lazy().
+ * @nr_counters: The number of counters.
+ * @gfp_flags: gfp flags to pass to the per-CPU and tree item allocators.
+ *
+ * Allocate the per-CPU counters and the tree items, and publish them:
+ * later updates go to the tree, and the approximation then has the
+ * accuracy of the tree. The updates made before activation stay in the
+ * bias and the atomic bias, which the sums keep reading, so updates racing
+ * with the activation are not lost: the bias only has the owner as writer,
+ * and the atomic bias only receives atomic additions. Activating active
+ * counters does nothing. @counters and @nr_counters must be those passed
+ * to percpu_counter_tree_init_many_lazy().
+ *
+ * The activations of the same counters must be serialized by the caller,
+ * for instance with a mutex: a concurrent activation would publish a second
+ * tree and lose the updates made to the first. The counters are published
+ * in order: once percpu_counter_tree_active() reports the last one active,
+ * the activation is complete, and a caller can skip it and its
+ * serialization. The caller may not observe the other counters active yet,
+ * which only delays their updates going to the tree.
+ *
+ * Return:
+ * * %0: Success
+ * * %-EINVAL: - @nr_counters differs from percpu_counter_tree_init_many_lazy()
+ * * %-ENOMEM: - Out of memory
+ */
+int percpu_counter_tree_activate_many(struct percpu_counter_tree *counters,
+ unsigned int nr_counters, gfp_t gfp_flags)
+{
+ s8 __percpu *level0;
+ atomic_long_t *items, *root;
+ unsigned int i;
+
+ /* Serialized by the caller: a previous activation is complete. */
+ if (counters[0].level0)
+ return 0;
+ /* The tree items are sized for @nr_counters, at the stride set by the initialization. */
+ if (WARN_ON_ONCE(counters[0].items_stride !=
+ percpu_counter_tree_items_stride(nr_counters)))
+ return -EINVAL;
+ level0 = __alloc_percpu_gfp(nr_counters * sizeof(*level0),
+ __alignof__(*counters->level0), gfp_flags);
+ if (!level0)
+ return -ENOMEM;
+ items = percpu_counter_tree_alloc_items(nr_counters, gfp_flags);
+ if (!items) {
+ free_percpu(level0);
+ return -ENOMEM;
+ }
+ root = &items[(tree_topology.nr_items - 1) * counters[0].items_stride];
+ /* See "Publication of the tree" above. */
+ for (i = 0; i < nr_counters; i++) {
+ /* Release of approx_sum: the root is zeroed before its pointer. */
+ smp_store_release(&counters[i].approx_sum, root + i);
+ /*
+ * Release of level0: the per-CPU counters, the tree items and
+ * approx_sum are set before level0, which opens the tree to
+ * updates.
+ */
+ smp_store_release(&counters[i].level0, level0 + i);
+ }
+ return 0;
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_activate_many);
+
+/**
+ * percpu_counter_tree_destroy_many() - Destroy many per-CPU counter trees.
+ * @counters: Array of counters trees to destroy.
+ * @nr_counters: The number of counter trees to destroy.
+ *
+ * Release internal resources allocated for @nr_counters per-CPU counter trees.
+ * @counters and @nr_counters must be those passed to
+ * percpu_counter_tree_init_many() or percpu_counter_tree_init_many_lazy().
+ */
+void percpu_counter_tree_destroy_many(struct percpu_counter_tree *counters,
+ unsigned int nr_counters)
+{
+ free_percpu(counters->level0);
+ /*
+ * @nr_counters selects the allocator of the tree items: on a mismatch
+ * with the initialization, leak them rather than corrupt the allocator.
+ */
+ if (WARN_ON_ONCE(counters->items_stride !=
+ percpu_counter_tree_items_stride(nr_counters)))
+ return;
+ /* The first item of the first counter is the base of the tree items, if allocated. */
+ if (counters->approx_sum)
+ percpu_counter_tree_free_items(percpu_counter_tree_items(counters), nr_counters);
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_destroy_many);
+
+/**
+ * percpu_counter_tree_destroy() - Destroy one per-CPU counter tree.
+ * @counter: Counter to destroy.
+ *
+ * Release internal resources allocated for one per-CPU counter tree.
+ */
+void percpu_counter_tree_destroy(struct percpu_counter_tree *counter)
+{
+ return percpu_counter_tree_destroy_many(counter, 1);
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_destroy);
+
+/*
+ * Amount propagated by a counter holding @v: the multiples of @bit_mask,
+ * truncated toward zero.
+ */
+static inline
+long percpu_counter_tree_item_carry(long v, unsigned long bit_mask)
+{
+ if (v < 0)
+ return -(long)(-(unsigned long)v & ~(bit_mask - 1));
+ return (long)((unsigned long)v & ~(bit_mask - 1));
+}
+
+/*
+ * It does not matter through which path the carry propagates up the
+ * tree, therefore there is no need to disable preemption because the
+ * cpu number is only used to favor cache locality.
+ */
+static
+void percpu_counter_tree_add_slowpath(struct percpu_counter_tree *counter, long inc)
+{
+ unsigned int level_items, nr_levels = tree_topology.config->nr_levels,
+ level, n_arity_order;
+ unsigned long bit_mask;
+ unsigned int stride = counter->items_stride;
+ unsigned int cpu = raw_smp_processor_id();
+ atomic_long_t *root, *item;
+
+ /*
+ * Pairs with the release of level0 in
+ * percpu_counter_tree_activate_many(), which follows the store of
+ * approx_sum. The caller read level0, but approx_sum is a separate
+ * field, which the address dependency of that read does not order:
+ * read it after level0.
+ */
+ smp_rmb();
+ root = READ_ONCE(counter->approx_sum);
+ item = root - (tree_topology.nr_items - 1) * stride;
+ n_arity_order = tree_topology.config->n_arity_order[0];
+ bit_mask = percpu_counter_tree_batch_size(counter) << n_arity_order;
+ level_items = 1U << (tree_topology.nr_cpus_order - n_arity_order);
+
+ for (level = 1; level < nr_levels; level++) {
+ /*
+ * A multiple of bit_mask would leave the residual of this
+ * level unchanged: propagate it to the next level directly.
+ * This trades the hysteresis of this level for not touching
+ * its cache line.
+ */
+ if (abs(inc) & (bit_mask - 1)) {
+ atomic_long_t *count = &item[(cpu & (level_items - 1)) * stride];
+ long res;
+
+ /*
+ * Add the carry to the residual of this level, then
+ * extract the multiples of bit_mask of the residual,
+ * truncated toward zero, and propagate them to the
+ * next level, which leaves the residual within
+ * (-bit_mask, +bit_mask).
+ *
+ * The extraction is a cmpxchg on the value returned by
+ * the addition, whose cache line is already exclusive,
+ * so it usually succeeds at once. When a concurrent
+ * updater changed the residual in between, the cmpxchg
+ * fails and the carry is recomputed from the current
+ * value, which is 0 if the other updater extracted it.
+ * Each carry is thus extracted once, from an exact
+ * value: the residual only exceeds its range by the
+ * additions in flight, and is within range once all
+ * the updaters are done. An addition which does not
+ * carry stays a single atomic operation, which cannot
+ * fail.
+ *
+ * Relaxed ordering is sufficient for the residual
+ * itself: the reasoning above only involves this
+ * atomic, whose modifications all CPUs see in the same
+ * order, each read-modify-write reading the value
+ * written by the one just before it, whatever its
+ * ordering. Ordering only matters relative to the other
+ * levels, for the precise sum: the release ordering of
+ * the addition orders the departure of the carry from
+ * the level below before its arrival here.
+ */
+ res = atomic_long_add_return_release(inc, count);
+ do {
+ inc = percpu_counter_tree_item_carry(res, bit_mask);
+ if (likely(!inc))
+ return;
+ } while (!atomic_long_try_cmpxchg_relaxed(count, &res, res - inc));
+ }
+ item += level_items * stride;
+ n_arity_order = tree_topology.config->n_arity_order[level];
+ level_items >>= n_arity_order;
+ bit_mask <<= n_arity_order;
+ }
+ /* The atomic API has no add with release ordering without a return value. */
+ atomic_long_add_return_release(inc, root);
+}
+
+/* Add @inc to the per-CPU counters @level0 of an activated tree. */
+static inline
+void percpu_counter_tree_add_tree(struct percpu_counter_tree *counter,
+ s8 __percpu *level0, long inc)
+{
+ unsigned long bit_mask = percpu_counter_tree_batch_size(counter);
+ s8 old, new;
+ long v, carry;
+
+ /*
+ * Keep the residual within (-batch, +batch) and propagate its
+ * multiples of the batch size, truncated toward zero. This
+ * hysteresis prevents updates going back and forth around a batch
+ * boundary from propagating each time.
+ *
+ * Preemption stays enabled: the update may go to the level 0
+ * counter of any CPU, and the cmpxchg validates the value read.
+ * After a migration or an interrupt between the read and the
+ * cmpxchg, the cmpxchg either fails and retries with the current
+ * residual of the current CPU, or succeeds on a residual equal to
+ * the value read, which the new value correctly replaces. Each
+ * this_cpu operation is itself safe against preemption and
+ * interrupts, and the carry may propagate from any CPU, see
+ * percpu_counter_tree_add_slowpath().
+ */
+ old = this_cpu_read(*level0);
+ do {
+ v = old + inc;
+ carry = percpu_counter_tree_item_carry(v, bit_mask);
+ new = (s8)(v - carry);
+ } while (!this_cpu_try_cmpxchg(*level0, &old, new));
+ if (likely(!carry))
+ return;
+ percpu_counter_tree_add_slowpath(counter, carry);
+}
+
+/**
+ * percpu_counter_tree_add() - Add to a per-CPU counter tree.
+ * @counter: Counter added to.
+ * @inc: Increment value (either positive or negative).
+ *
+ * Add @inc to a per-CPU counter tree. This is a fast-path which will
+ * typically increment per-CPU counters as long as there is no carry
+ * greater or equal to the counter tree batch size. Before the tree is
+ * activated, add to the atomic bias.
+ */
+void percpu_counter_tree_add(struct percpu_counter_tree *counter, long inc)
+{
+ /*
+ * Pairs with the release of level0 in
+ * percpu_counter_tree_activate_many(): the per-CPU counters are only
+ * reached through this pointer, and the address dependency orders
+ * their initialization before their use, as for rcu_dereference().
+ * No acquire, see "Publication of the tree".
+ */
+ s8 __percpu *level0 = READ_ONCE(counter->level0);
+
+ if (!level0) {
+ atomic_long_add(inc, &counter->atomic_bias);
+ return;
+ }
+ percpu_counter_tree_add_tree(counter, level0, inc);
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_add);
+
+/**
+ * percpu_counter_tree_add_owner() - Add to a per-CPU counter tree as its owner.
+ * @counter: Counter added to.
+ * @inc: Increment value (either positive or negative).
+ *
+ * Like percpu_counter_tree_add(), but before the tree is activated, add
+ * to the bias without atomic instruction. The calls for a given counter
+ * must not run concurrently with each other: they come from a single
+ * context, the owner chosen by the user of the counter, such as a task.
+ */
+void percpu_counter_tree_add_owner(struct percpu_counter_tree *counter, long inc)
+{
+ /* Address dependency, as in percpu_counter_tree_add(). */
+ s8 __percpu *level0 = READ_ONCE(counter->level0);
+
+ if (!level0) {
+ WRITE_ONCE(counter->bias, (long)((unsigned long)READ_ONCE(counter->bias) + inc));
+ return;
+ }
+ percpu_counter_tree_add_tree(counter, level0, inc);
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_add_owner);
+
+/*
+ * The precise sum is the sum of the residuals of all tree levels. A carry
+ * leaves a level before reaching a higher level, and the release ordering
+ * of its arrival orders its departure before it. Reading the levels from
+ * the root down, separated by read barriers, therefore never counts a carry
+ * twice. A carry in flight during the sum may be missed: the result is
+ * precise for a counter without concurrent updates.
+ */
+static
+long percpu_counter_tree_precise_sum_unbiased(struct percpu_counter_tree *counter)
+{
+ unsigned int nr_levels = tree_topology.config->nr_levels, i,
+ level_items[MAX_NR_LEVELS], stride = counter->items_stride;
+ /*
+ * Pairs with the release of level0 in
+ * percpu_counter_tree_activate_many(): approx_sum, the tree items and
+ * the per-CPU counters read below are set before level0.
+ */
+ s8 __percpu *level0 = smp_load_acquire(&counter->level0);
+ unsigned long sum = atomic_long_read(&counter->atomic_bias);
+ atomic_long_t *item;
+ int level, cpu;
+
+ /* Before activation, the atomic bias holds the whole unbiased sum. */
+ if (!level0)
+ return (long)sum;
+ /* Number of items of each level above level 0. */
+ level_items[0] = 1U << tree_topology.nr_cpus_order;
+ for (level = 1; level < (int)nr_levels; level++)
+ level_items[level] = level_items[level - 1] >>
+ tree_topology.config->n_arity_order[level - 1];
+ /* Published before level0, so ordered by its acquire. */
+ item = READ_ONCE(counter->approx_sum);
+ sum += atomic_long_read(item);
+ smp_rmb(); /* Read each level before the level below it. */
+ /* The items of each level precede those of the level above it. */
+ for (level = (int)nr_levels - 1; level >= 1; level--) {
+ item -= level_items[level] * stride;
+ for (i = 0; i < level_items[level]; i++)
+ sum += atomic_long_read(&item[i * stride]);
+ smp_rmb(); /* Read this level before the level below it. */
+ }
+ for_each_possible_cpu(cpu)
+ sum += (long)READ_ONCE(*per_cpu_ptr(level0, cpu));
+ return (long)sum;
+}
+
+/**
+ * percpu_counter_tree_precise_sum() - Return precise counter sum.
+ * @counter: The counter to sum.
+ *
+ * Querying the precise sum is relatively expensive because it needs to
+ * iterate over all CPUs.
+ * This is meant to be used when accuracy is preferred over speed.
+ *
+ * The sum is exact when no update runs concurrently. It takes no lock, so
+ * a concurrent update which carries from level 0 to the levels above can
+ * be counted in part: the sum may read the residual it left below and miss
+ * its carry on the way up, an error in either direction of at most the
+ * carries in flight. A carry is never counted twice.
+ *
+ * Return: The current precise counter sum.
+ */
+long percpu_counter_tree_precise_sum(struct percpu_counter_tree *counter)
+{
+ return percpu_counter_tree_precise_sum_unbiased(counter) + READ_ONCE(counter->bias);
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_precise_sum);
+
+/*
+ * The comparison between counters is on the wrapped difference. It is valid
+ * only when the two counters differ by less than LONG_MAX, and results are
+ * undefined beyond that.
+ */
+static inline
+long wrapped_delta(long a, long b)
+{
+ return (long)((unsigned long)a - (unsigned long)b);
+}
+
+/*
+ * Each counter's approximation lies within its accuracy of its precise sum,
+ * in both directions. The comparison only observes the approximations, so
+ * it must reason about the possible precise values around each observed
+ * approximation, which lie within [approx - accuracy, approx + accuracy]:
+ *
+ * precise_a range
+ * <-----------+----------->
+ * accuracy_a | accuracy_a
+ * approx_a
+ * | precise_b range
+ * | <-----------+----------->
+ * | accuracy_b | accuracy_b
+ * | approx_b
+ * | |
+ * |<------------ gap ------------->|
+ * ----------------------+--------------------------------+-------------> values
+ *
+ * The two precise ranges are disjoint, and the comparison can return a
+ * definitive answer, only once the gap between approximations exceeds the
+ * sum of the two accuracies:
+ *
+ * gap > accuracy_a + accuracy_b
+ *
+ * When comparing against a plain value, the value has zero width, so the
+ * margin reduces to the accuracy of the counter: the value is above the
+ * counter once it exceeds approx + accuracy, and below it once it is less
+ * than approx - accuracy.
+ */
+static
+int compare_delta(long delta, unsigned long accuracy)
+{
+ if (delta >= 0) {
+ if (delta <= accuracy)
+ return 0;
+ else
+ return 1;
+ } else {
+ if (-delta <= accuracy)
+ return 0;
+ else
+ return -1;
+ }
+}
+
+/**
+ * percpu_counter_tree_approximate_compare - Approximated comparison of two counter trees.
+ * @a: First counter to compare.
+ * @b: Second counter to compare.
+ *
+ * Evaluate an approximate comparison of two counter trees. This approximation
+ * comparison is fast, and provides an accurate answer if the counters are
+ * found to be either less than or greater than the other. However, if the
+ * approximated comparison returns 0, the counters respective sums are found to
+ * be within the two counters accuracy range. The two counters are read
+ * independently; the result is not an atomic snapshot of both.
+ * The comparison between counters is on the wrapped difference. It is valid
+ * only when the two counters differ by less than LONG_MAX, and results are
+ * undefined beyond that.
+ *
+ * Return:
+ * * %0 - Counters @a and @b do not differ by more than the sum of their respective
+ * accuracy ranges.
+ * * %-1 - Counter @a less than counter @b.
+ * * %1 - Counter @a is greater than counter @b.
+ */
+int percpu_counter_tree_approximate_compare(struct percpu_counter_tree *a,
+ struct percpu_counter_tree *b)
+{
+ unsigned long accuracy = percpu_counter_tree_accuracy(a) + percpu_counter_tree_accuracy(b);
+
+ /* See the range geometry above compare_delta(). */
+ return compare_delta(wrapped_delta(percpu_counter_tree_approximate_sum(a),
+ percpu_counter_tree_approximate_sum(b)),
+ accuracy);
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_approximate_compare);
+
+/**
+ * percpu_counter_tree_approximate_compare_value - Approximated comparison of a counter tree
+ * against a given value.
+ * @counter: Counter to compare.
+ * @v: Value to compare.
+ *
+ * Evaluate an approximate comparison of a counter tree against a given value.
+ * This approximation comparison is fast, and provides an accurate
+ * answer if the counter is found to be either less than or greater
+ * than the value. However, if the approximated comparison returns
+ * 0, the value is within the counter accuracy range.
+ * The comparison between counters is on the wrapped difference. It is valid
+ * only when @v and the counter differ by less than LONG_MAX, and results are
+ * undefined beyond that.
+ *
+ * Return:
+ * * %0 - The value @v is within the accuracy range of the counter.
+ * * %-1 - The value @v is less than the counter.
+ * * %1 - The value @v is greater than the counter.
+ */
+int percpu_counter_tree_approximate_compare_value(struct percpu_counter_tree *counter, long v)
+{
+ unsigned long accuracy = percpu_counter_tree_accuracy(counter);
+
+ /* See the range geometry above compare_delta(). */
+ return compare_delta(wrapped_delta(v, percpu_counter_tree_approximate_sum(counter)),
+ accuracy);
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_approximate_compare_value);
+
+/**
+ * percpu_counter_tree_precise_compare - Precise comparison of two counter trees.
+ * @a: First counter to compare.
+ * @b: Second counter to compare.
+ *
+ * Evaluate a precise comparison of two counter trees. As an optimization, it
+ * uses the approximate counter comparison to quickly compare counters which
+ * are far apart. Only cases where counter sums are within the accuracy range
+ * require precise counter sums. The two counters are read independently; the
+ * result is not an atomic snapshot of both. With concurrent updates, the
+ * precise sums can count updates in part, see
+ * percpu_counter_tree_precise_sum().
+ * The comparison between counters is on the wrapped difference. It is valid
+ * only when the two counters differ by less than LONG_MAX, and results are
+ * undefined beyond that.
+ *
+ * Return:
+ * * %0 - Counters are equal.
+ * * %-1 - Counter @a less than counter @b.
+ * * %1 - Counter @a is greater than counter @b.
+ */
+int percpu_counter_tree_precise_compare(struct percpu_counter_tree *a,
+ struct percpu_counter_tree *b)
+{
+ long count_a = percpu_counter_tree_approximate_sum(a),
+ count_b = percpu_counter_tree_approximate_sum(b);
+ unsigned long accuracy_a = percpu_counter_tree_accuracy(a),
+ accuracy_b = percpu_counter_tree_accuracy(b);
+ long delta = wrapped_delta(count_a, count_b);
+ int res;
+
+ /* See the range geometry above compare_delta(). */
+ res = compare_delta(delta, accuracy_a + accuracy_b);
+ /* The values are distanced enough for an accurate approximated comparison. */
+ if (res)
+ return res;
+
+ /*
+ * The approximated comparison is within the accuracy range, therefore at least one
+ * precise sum is needed. Sum the counter which has the largest accuracy first.
+ */
+ if (accuracy_b < accuracy_a) {
+ count_a = percpu_counter_tree_precise_sum(a);
+ /* precise_a - approx_b: only b's precise range remains. */
+ res = compare_delta(wrapped_delta(count_a, count_b), accuracy_b);
+ if (res)
+ return res;
+ /* Precise sum of second counter is required. */
+ count_b = percpu_counter_tree_precise_sum(b);
+ } else {
+ count_b = percpu_counter_tree_precise_sum(b);
+ /* approx_a - precise_b: only a's precise range remains. */
+ res = compare_delta(wrapped_delta(count_a, count_b), accuracy_a);
+ if (res)
+ return res;
+ /* Precise sum of second counter is required. */
+ count_a = percpu_counter_tree_precise_sum(a);
+ }
+ delta = wrapped_delta(count_a, count_b);
+ if (delta < 0)
+ return -1;
+ if (delta > 0)
+ return 1;
+ return 0;
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_precise_compare);
+
+/**
+ * percpu_counter_tree_precise_compare_value - Precise comparison of a counter tree against
+ * a given value.
+ * @counter: Counter to compare.
+ * @v: Value to compare.
+ *
+ * Evaluate a precise comparison of a counter tree against a given value. As
+ * an optimization, it uses the approximate counter comparison to quickly
+ * identify whether the counter and value are far apart. Only cases where the
+ * value is within the counter accuracy range require a precise counter sum.
+ * With concurrent updates, the precise sum can count updates in part, see
+ * percpu_counter_tree_precise_sum().
+ * The comparison between counters is on the wrapped difference. It is valid
+ * only when @v and the counter differ by less than LONG_MAX, and results are
+ * undefined beyond that.
+ *
+ * Return:
+ * * %0 - The value @v is equal to the counter.
+ * * %-1 - The value @v is less than the counter.
+ * * %1 - The value @v is greater than the counter.
+ */
+int percpu_counter_tree_precise_compare_value(struct percpu_counter_tree *counter, long v)
+{
+ long count = percpu_counter_tree_approximate_sum(counter), delta;
+ unsigned long accuracy = percpu_counter_tree_accuracy(counter);
+ int res;
+
+ /* See the range geometry above compare_delta(). */
+ res = compare_delta(wrapped_delta(v, count), accuracy);
+ /* The values are distanced enough for an accurate approximated comparison. */
+ if (res)
+ return res;
+
+ /* Precise sum is required. */
+ count = percpu_counter_tree_precise_sum(counter);
+ delta = wrapped_delta(v, count);
+ if (delta < 0)
+ return -1;
+ if (delta > 0)
+ return 1;
+ return 0;
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_precise_compare_value);
+
+/**
+ * percpu_counter_tree_set - Set the counter tree sum to a given value.
+ * @counter: Counter to set.
+ * @v: Value to set.
+ *
+ * Set the counter sum to a given value. It can be useful for instance
+ * to reset the counter sum to 0. Note that even after setting the
+ * counter sum to a given value, the counter sum approximation can
+ * return any value within the accuracy range around that value.
+ *
+ * The caller is responsible for preventing concurrent updates and
+ * precise sums of @counter while it is being set.
+ */
+void percpu_counter_tree_set(struct percpu_counter_tree *counter, long v)
+{
+ WRITE_ONCE(counter->bias, v - percpu_counter_tree_precise_sum_unbiased(counter));
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_set);
+
+/**
+ * percpu_counter_tree_approximate_accuracy - Query the accuracy of a counter tree.
+ * @counter: Counter to query.
+ *
+ * The approximation differs from the precise sum by the residuals not yet propagated
+ * up the tree. The residuals of all levels are symmetric, so the approximation is
+ * within the same distance of the precise sum in both directions.
+ * The accuracy is derived from the hardware topology and the counter tree batch size.
+ * It is invariant for a given counter tree.
+ * The accuracy describes a settled counter: one with no update in progress.
+ * An update in flight between its per-CPU update and its carry propagation is not covered.
+ * Using this function should not be typically required, see the following functions instead:
+ * * percpu_counter_tree_approximate_compare(),
+ * * percpu_counter_tree_approximate_compare_value(),
+ * * percpu_counter_tree_precise_compare(),
+ * * percpu_counter_tree_precise_compare_value().
+ *
+ * Return: The accuracy of the counter: the approximate sum of a settled counter
+ * is within [precise sum - accuracy, precise sum + accuracy].
+ */
+unsigned long percpu_counter_tree_approximate_accuracy(struct percpu_counter_tree *counter)
+{
+ return percpu_counter_tree_accuracy(counter);
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_approximate_accuracy);
+
+/* Calculate the number of tree items and the accuracy multiplier. */
+static
+void __init calculate_topology(void)
+{
+ unsigned int nr_levels = tree_topology.config->nr_levels, level;
+ unsigned int level_items = 1U << tree_topology.nr_cpus_order;
+ unsigned long batch_size = 1;
+
+ /* The root, and the items of each level above level 0. */
+ tree_topology.nr_items = 1;
+ /* One batch per level 0 counter: see percpu_counter_tree_accuracy(). */
+ tree_topology.accuracy_multiplier = nr_cpu_ids;
+
+ /*
+ * The multiplier is derived from a batch size of 1 to speed up
+ * calculating the accuracy at tree initialization. A level N
+ * counter holds at most BATCH(level N) - BATCH(level N - 1) in
+ * either direction.
+ */
+ for (level = 1; level < nr_levels; level++) {
+ unsigned int n_arity_order = tree_topology.config->n_arity_order[level - 1];
+
+ level_items >>= n_arity_order;
+ tree_topology.nr_items += level_items;
+ tree_topology.accuracy_multiplier +=
+ ((batch_size << n_arity_order) - batch_size) * level_items;
+ batch_size <<= n_arity_order;
+ }
+}
+
+#if IS_ENABLED(CONFIG_KUNIT)
+/*
+ * Validate the configuration table: the n-arity orders of each entry must
+ * sum to its order.
+ *
+ * Return: The number of invalid entries.
+ */
+unsigned int percpu_counter_tree_check_config(void)
+{
+ unsigned int order, nr_errors = 0;
+
+ for (order = 0; order < ARRAY_SIZE(per_nr_cpu_order_config); order++) {
+ const struct counter_config *c = &per_nr_cpu_order_config[order];
+ unsigned int level, shift = 0;
+
+ for (level = 0; level < c->nr_levels; level++)
+ shift += c->n_arity_order[level];
+ if (shift != order) {
+ pr_err("order %u: arity sum %u\n", order, shift);
+ nr_errors++;
+ }
+ }
+ return nr_errors;
+}
+EXPORT_SYMBOL_IF_KUNIT(percpu_counter_tree_check_config);
+
+/*
+ * Check the invariants of a settled counter (no update in progress):
+ * - a level 0 counter is within (-BATCH(level 0), +BATCH(level 0)),
+ * - a counter at level N, above level 0, holds a multiple of
+ * BATCH(level N - 1) within (-BATCH(level N), +BATCH(level N)),
+ * - the root holds a multiple of the batch of the last level.
+ * Before activation, a lazily initialized counter has no tree to check.
+ *
+ * Return: The number of violated invariants.
+ */
+unsigned int percpu_counter_tree_check_residuals(struct percpu_counter_tree *counter)
+{
+ unsigned int nr_levels = tree_topology.config->nr_levels, level, level_items,
+ nr_errors = 0, stride = counter->items_stride;
+ unsigned long lo = percpu_counter_tree_batch_size(counter);
+ atomic_long_t *item;
+ long root;
+ int cpu;
+
+ if (!counter->level0)
+ return 0;
+ item = percpu_counter_tree_items(counter);
+ for_each_possible_cpu(cpu) {
+ long v = READ_ONCE(*per_cpu_ptr(counter->level0, cpu));
+
+ if (v <= -(long)lo || v >= (long)lo) {
+ pr_err("cpu %d: level 0 residual %ld, batch %lu\n", cpu, v, lo);
+ nr_errors++;
+ }
+ }
+ level_items = 1U << tree_topology.nr_cpus_order;
+ for (level = 1; level < nr_levels; level++) {
+ unsigned int n_arity_order = tree_topology.config->n_arity_order[level - 1], i;
+ unsigned long hi = lo << n_arity_order;
+
+ level_items >>= n_arity_order;
+ for (i = 0; i < level_items; i++) {
+ long v = atomic_long_read(&item[i * stride]);
+
+ if (v <= -(long)hi || v >= (long)hi || ((unsigned long)v & (lo - 1))) {
+ pr_err("level %u item %u: residual %ld, batch %lu\n",
+ level, i, v, hi);
+ nr_errors++;
+ }
+ }
+ item += level_items * stride;
+ lo = hi;
+ }
+ root = atomic_long_read(counter->approx_sum);
+ if ((unsigned long)root & (lo - 1)) {
+ pr_err("root %ld: not a multiple of %lu\n", root, lo);
+ nr_errors++;
+ }
+ return nr_errors;
+}
+EXPORT_SYMBOL_IF_KUNIT(percpu_counter_tree_check_residuals);
+
+/*
+ * Copy the state of counter @src into counter @dst, initialized with the
+ * same batch size and without concurrent updates: their per-CPU counters,
+ * tree items and biases. Both trees are active.
+ */
+void percpu_counter_tree_copy_state(struct percpu_counter_tree *dst,
+ struct percpu_counter_tree *src)
+{
+ atomic_long_t *dst_items = percpu_counter_tree_items(dst),
+ *src_items = percpu_counter_tree_items(src);
+ unsigned int pos;
+ int cpu;
+
+ for_each_possible_cpu(cpu)
+ *per_cpu_ptr(dst->level0, cpu) = *per_cpu_ptr(src->level0, cpu);
+ for (pos = 0; pos < tree_topology.nr_items; pos++)
+ atomic_long_set(&dst_items[pos * dst->items_stride],
+ atomic_long_read(&src_items[pos * src->items_stride]));
+ dst->bias = src->bias;
+ atomic_long_set(&dst->atomic_bias, atomic_long_read(&src->atomic_bias));
+}
+EXPORT_SYMBOL_IF_KUNIT(percpu_counter_tree_copy_state);
+#endif
+
+/* Room for the fan-out of each level, up to 7 digits and a comma, and a NUL. */
+#define SHAPE_BUF_SIZE (8 * MAX_NR_LEVELS + 1)
+
+/* Format the fan-out of each level, from level 0 to the root, e.g. "8,4,8". */
+static
+int format_shape(char *buf, size_t size, const struct counter_config *config)
+{
+ unsigned int level;
+ int len = 0;
+
+ for (level = 0; level < config->nr_levels; level++)
+ len += scnprintf(buf + len, size - len, "%s%u", level ? "," : "",
+ 1U << config->n_arity_order[level]);
+ return len;
+}
+
+static
+int arity_param_set(const char *val, const struct kernel_param *kp)
+{
+ /* Applied once by percpu_counter_tree_subsystem_init(). */
+ if (tree_topology.config)
+ return -EBUSY;
+ arity_param = val;
+ return 0;
+}
+
+static
+int arity_param_get(char *buffer, const struct kernel_param *kp)
+{
+ int len;
+
+ if (!tree_topology.config)
+ return sprintf(buffer, "%s\n", arity_param ? : "");
+ /* The buffer is a page, larger than any shape and its newline. */
+ len = format_shape(buffer, SHAPE_BUF_SIZE, tree_topology.config);
+ buffer[len++] = '\n';
+ return len;
+}
+
+static const struct kernel_param_ops arity_param_ops = {
+ .set = arity_param_set,
+ .get = arity_param_get,
+};
+
+/*
+ * percpu_counter_tree.arity=<fan-out>[,<fan-out>...] overrides the tree
+ * shape chosen for nr_cpu_ids, to compare shapes on a given machine.
+ * Reading the parameter shows the shape in use.
+ */
+module_param_cb(arity, &arity_param_ops, NULL, 0444);
+MODULE_PARM_DESC(arity, "Fan-out of each tree level from level 0 to the root, e.g. 8,4,8");
+
+/*
+ * Parse a list of fan-outs, from level 0 to the root, into @config. Each
+ * fan-out is a power of 2 of at least 2, and their product is nr_cpu_ids
+ * rounded up to a power of 2.
+ *
+ * Return: 0 on success, -EINVAL if the list does not describe a tree
+ * covering the CPU numbers below nr_cpu_ids.
+ */
+static
+int __init parse_arity(const char *str, struct counter_config *config)
+{
+ int ints[MAX_NR_LEVELS + 2]; /* Count, levels, and one extra to detect overflow. */
+ unsigned int level, order = 0;
+
+ if (*get_options(str, ARRAY_SIZE(ints), ints) || ints[0] > MAX_NR_LEVELS)
+ return -EINVAL;
+ config->nr_levels = ints[0];
+ for (level = 0; level < config->nr_levels; level++) {
+ int fan_out = ints[level + 1];
+
+ if (fan_out < 2 || !is_power_of_2(fan_out))
+ return -EINVAL;
+ config->n_arity_order[level] = ilog2(fan_out);
+ order += config->n_arity_order[level];
+ if (order > tree_topology.nr_cpus_order)
+ return -EINVAL;
+ }
+ if (order != tree_topology.nr_cpus_order)
+ return -EINVAL;
+ return 0;
+}
+
+int __init percpu_counter_tree_subsystem_init(void)
+{
+ const char *origin = "default";
+ char shape[SHAPE_BUF_SIZE] = "";
+
+ tree_topology.nr_cpus_order = get_count_order(nr_cpu_ids);
+ if (tree_topology.nr_cpus_order >= ARRAY_SIZE(per_nr_cpu_order_config)) {
+ pr_err("Unsupported number of CPUs (%u)\n", nr_cpu_ids);
+ return -EINVAL;
+ }
+ tree_topology.config = &per_nr_cpu_order_config[tree_topology.nr_cpus_order];
+ if (arity_param && *arity_param) {
+ if (parse_arity(arity_param, &tree_topology.boot_config)) {
+ pr_warn("Invalid arity \"%s\" for %u CPUs, using the default\n",
+ arity_param, nr_cpu_ids);
+ } else {
+ tree_topology.config = &tree_topology.boot_config;
+ origin = "boot parameter";
+ }
+ }
+ calculate_topology();
+ tree_topology.items_cachep =
+ kmem_cache_create("percpu_counter_tree_items",
+ tree_topology.nr_items * SMP_CACHE_BYTES,
+ SMP_CACHE_BYTES, SLAB_PANIC, NULL);
+ format_shape(shape, sizeof(shape), tree_topology.config);
+ pr_info("%u CPUs, fan-out %s (%s), %u tree items\n",
+ nr_cpu_ids, tree_topology.nr_cpus_order ? shape : "none", origin,
+ tree_topology.nr_items);
+ return 0;
+}
diff --git a/lib/tests/Makefile b/lib/tests/Makefile
index df0537e1b91faf..d158a8be8a1944 100644
--- a/lib/tests/Makefile
+++ b/lib/tests/Makefile
@@ -59,4 +59,7 @@ obj-$(CONFIG_UTIL_MACROS_KUNIT) += util_macros_kunit.o
obj-$(CONFIG_RATELIMIT_KUNIT_TEST) += test_ratelimit.o
obj-$(CONFIG_UUID_KUNIT_TEST) += uuid_kunit.o
+obj-$(CONFIG_PERCPU_COUNTER_TREE_KUNIT_TEST) += percpu_counter_tree_kunit.o
+obj-$(CONFIG_PERCPU_COUNTER_TREE_BENCH) += percpu_counter_tree_bench.o
+
obj-$(CONFIG_TEST_RUNTIME_MODULE) += module/
diff --git a/lib/tests/percpu_counter_tree_bench.c b/lib/tests/percpu_counter_tree_bench.c
new file mode 100644
index 00000000000000..bf260bef196a58
--- /dev/null
+++ b/lib/tests/percpu_counter_tree_bench.c
@@ -0,0 +1,604 @@
+// SPDX-License-Identifier: GPL-2.0+ OR MIT
+/*
+ * Benchmark of hierarchical per-CPU counters against struct percpu_counter.
+ *
+ * update: ns per percpu_counter_tree_add() per thread, one kthread pinned
+ * on each online CPU, on one shared counter, on one counter per
+ * thread, or on the counters of a few mm-like groups of 4 counters
+ * initialized together. The "single" variant gives each thread a
+ * lazily initialized counter updated with
+ * percpu_counter_tree_add_owner(), like a single-threaded process.
+ * The "percpu_counter" variants use percpu_counter_add() with the
+ * default batch, like the RSS counters before hierarchical counters.
+ * lifecycle: ns per init_many(4) + destroy_many(4), like fork + exit of an
+ * mm, from one thread or from all online CPUs at once; "lazy" uses
+ * init_many_lazy(4), like a single-threaded process.
+ * churn: like lifecycle with a few updates, keeping one in @pin_period sets
+ * of counters alive until the end, like long-lived processes.
+ *
+ * Each result ends with the implementation measured: "tree" for the
+ * hierarchical counters, "percpu_counter" for struct percpu_counter.
+ *
+ * The knobs to try on a given machine are:
+ * - the batch size: percpu_counter_tree_bench.batch=<2..128> (power of 2),
+ * or "modprobe percpu_counter_tree_bench batch=<n>",
+ * - the tree shape: percpu_counter_tree.arity=<fan-out>,... on the kernel
+ * command line, from level 0 to the root. The product of the fan-outs
+ * is nr_cpu_ids rounded up to a power of 2, e.g. "16" (flat) or "4,4"
+ * when nr_cpu_ids is 16. The shape in use is shown in
+ * /sys/module/percpu_counter_tree/parameters/arity.
+ *
+ * For example, in a VM:
+ * ./tools/testing/kunit/kunit.py run --arch=x86_64 \
+ * --kconfig_add CONFIG_PERCPU_COUNTER_TREE_BENCH=y \
+ * --kconfig_add CONFIG_NR_CPUS=256 --qemu_args="-smp 64 -m 8192" \
+ * --kernel_args percpu_counter_tree.arity=8,8 \
+ * --kernel_args percpu_counter_tree_bench.batch=16 \
+ * --timeout 3000 percpu_counter_tree_bench
+ */
+#include <kunit/test.h>
+#include <linux/percpu_counter_tree.h>
+#include <linux/percpu_counter.h>
+#include <linux/kthread.h>
+#include <linux/random.h>
+#include <linux/slab.h>
+#include <linux/delay.h>
+#include <linux/ktime.h>
+#include <linux/math64.h>
+#include <linux/sched/task.h>
+#include <linux/cpuhplock.h>
+#include <linux/moduleparam.h>
+
+/* Like the RSS counters by default. */
+static unsigned int batch = 32;
+module_param(batch, uint, 0444);
+MODULE_PARM_DESC(batch, "Batch size of the hierarchical counters (power of 2, 2 to 128)");
+
+#define BENCH_MS 300
+#define LIFECYCLE_NR 4 /* Counters per mm. */
+#define LIFECYCLE_ITER 2000
+#define CHURN_ITER 4096
+
+enum { W_MONO, W_BOUNCE, W_RAND32, W_RAND512, NR_W };
+static const char * const wname[] = { "monotonic +1", "bounce +-1", "random +-32", "random +-512" };
+
+enum bench_impl { IMPL_TREE, IMPL_PERCPU_COUNTER };
+static const char * const impl_name[] = { "tree", "percpu_counter" };
+
+struct bench_shared {
+ atomic_t ready;
+ int wl;
+ unsigned int pin_period; /* churn: keep one in pin_period sets, 0: none. */
+ enum bench_impl impl;
+ bool go, stop;
+ bool owner; /* update: percpu_counter_tree_add_owner(). */
+ bool lifecycle;
+ bool lazy; /* lifecycle: percpu_counter_tree_init_many_lazy(). */
+};
+
+struct bench_thread {
+ u64 ns;
+ unsigned long ops;
+ struct bench_shared *sh;
+ struct percpu_counter_tree *c;
+ struct percpu_counter *pc; /* update: IMPL_PERCPU_COUNTER. */
+ struct percpu_counter_tree *pinned; /* churn: IMPL_TREE sets kept alive. */
+ struct percpu_counter *pinned_pc; /* churn: IMPL_PERCPU_COUNTER sets kept alive. */
+ struct task_struct *task;
+ unsigned int nr_pinned;
+ u32 seed;
+} ____cacheline_aligned_in_smp;
+
+/*
+ * Buffers are allocated through KUnit, which frees them if an assertion
+ * aborts the test, and are freed explicitly otherwise, since the benchmark
+ * runs many steps within one test.
+ */
+KUNIT_DEFINE_ACTION_WRAPPER(bench_kvfree, kvfree, const void *);
+
+static void *bench_kvzalloc(struct kunit *test, size_t size)
+{
+ void *p = kvzalloc(size, GFP_KERNEL);
+
+ if (p && kunit_add_action_or_reset(test, bench_kvfree, p))
+ return NULL;
+ return p;
+}
+
+static void bench_kvfree_now(struct kunit *test, void *p)
+{
+ kunit_release_action(test, bench_kvfree, p);
+}
+
+/* Wait for the start signal; return true if the benchmark is cancelled. */
+static bool bench_wait_go(struct bench_shared *sh)
+{
+ atomic_inc(&sh->ready);
+ /* Pairs with bench_start(): a cancellation sets stop before go. */
+ while (!smp_load_acquire(&sh->go))
+ cond_resched();
+ return READ_ONCE(sh->stop);
+}
+
+static int bench_update_fn(void *data)
+{
+ struct bench_thread *t = data;
+ struct bench_shared *sh = t->sh;
+ unsigned long ops = 0;
+ long sign = -1;
+ u32 x = t->seed;
+ int i;
+
+ if (bench_wait_go(sh))
+ return 0;
+ while (!READ_ONCE(sh->stop)) {
+ for (i = 0; i < 1024; i++) {
+ long inc;
+
+ switch (sh->wl) {
+ case W_MONO:
+ inc = 1;
+ break;
+ case W_BOUNCE:
+ inc = sign;
+ sign = -sign;
+ break;
+ default:
+ x ^= x << 13;
+ x ^= x >> 17;
+ x ^= x << 5;
+ inc = sh->wl == W_RAND32 ? 32 : 512;
+ if (x & 1)
+ inc = -inc;
+ break;
+ }
+ if (sh->impl == IMPL_PERCPU_COUNTER)
+ percpu_counter_add(t->pc, inc);
+ else if (sh->owner)
+ percpu_counter_tree_add_owner(t->c, inc);
+ else
+ percpu_counter_tree_add(t->c, inc);
+ }
+ ops += 1024;
+ cond_resched();
+ }
+ t->ops = ops;
+ return 0;
+}
+
+static int bench_lifecycle_fn(void *data)
+{
+ struct bench_thread *t = data;
+ struct percpu_counter_tree c[LIFECYCLE_NR];
+ struct percpu_counter pc[LIFECYCLE_NR];
+ u64 t0;
+ int i;
+
+ if (bench_wait_go(t->sh))
+ return 0;
+ t0 = ktime_get_ns();
+ for (i = 0; t->sh->impl == IMPL_PERCPU_COUNTER && i < LIFECYCLE_ITER; i++) {
+ if (percpu_counter_init_many(pc, 0, GFP_KERNEL, LIFECYCLE_NR))
+ break;
+ percpu_counter_add(&pc[0], 1);
+ percpu_counter_add(&pc[0], -1);
+ percpu_counter_destroy_many(pc, LIFECYCLE_NR);
+ cond_resched();
+ }
+ for (; t->sh->impl == IMPL_TREE && t->sh->lazy && i < LIFECYCLE_ITER; i++) {
+ if (percpu_counter_tree_init_many_lazy(c, LIFECYCLE_NR, batch))
+ break;
+ percpu_counter_tree_add_owner(&c[0], 1);
+ percpu_counter_tree_add_owner(&c[0], -1);
+ percpu_counter_tree_destroy_many(c, LIFECYCLE_NR);
+ cond_resched();
+ }
+ for (; t->sh->impl == IMPL_TREE && !t->sh->lazy && i < LIFECYCLE_ITER; i++) {
+ if (percpu_counter_tree_init_many(c, LIFECYCLE_NR, batch, GFP_KERNEL))
+ break;
+ percpu_counter_tree_add(&c[0], 1);
+ percpu_counter_tree_add(&c[0], -1);
+ percpu_counter_tree_destroy_many(c, LIFECYCLE_NR);
+ cond_resched();
+ }
+ t->ns = ktime_get_ns() - t0;
+ t->ops = i;
+ return 0;
+}
+
+static int bench_churn_fn(void *data)
+{
+ struct bench_thread *t = data;
+ unsigned int pin_period = t->sh->pin_period;
+ struct percpu_counter_tree tmp[LIFECYCLE_NR], *c;
+ struct percpu_counter tmp_pc[LIFECYCLE_NR], *pc;
+ u64 t0;
+ int i;
+
+ if (bench_wait_go(t->sh))
+ return 0;
+ t0 = ktime_get_ns();
+ for (i = 0; t->sh->impl == IMPL_PERCPU_COUNTER && i < CHURN_ITER; i++) {
+ bool pin = pin_period && !(i % pin_period);
+
+ pc = pin ? &t->pinned_pc[t->nr_pinned * LIFECYCLE_NR] : tmp_pc;
+ if (percpu_counter_init_many(pc, 0, GFP_KERNEL, LIFECYCLE_NR))
+ break;
+ percpu_counter_add(&pc[0], 100);
+ percpu_counter_add(&pc[1], -77);
+ percpu_counter_add(&pc[2], 1);
+ if (pin)
+ t->nr_pinned++;
+ else
+ percpu_counter_destroy_many(pc, LIFECYCLE_NR);
+ cond_resched();
+ }
+ for (; t->sh->impl == IMPL_TREE && i < CHURN_ITER; i++) {
+ bool pin = pin_period && !(i % pin_period);
+
+ c = pin ? &t->pinned[t->nr_pinned * LIFECYCLE_NR] : tmp;
+ if (percpu_counter_tree_init_many(c, LIFECYCLE_NR, batch, GFP_KERNEL))
+ break;
+ percpu_counter_tree_add(&c[0], 100);
+ percpu_counter_tree_add(&c[1], -77);
+ percpu_counter_tree_add(&c[2], 1);
+ if (pin)
+ t->nr_pinned++;
+ else
+ percpu_counter_tree_destroy_many(c, LIFECYCLE_NR);
+ cond_resched();
+ }
+ t->ns = ktime_get_ns() - t0;
+ t->ops = i;
+ return 0;
+}
+
+/* List at most @max online CPUs in @cpus, and return their number. */
+static unsigned int bench_cpus(int *cpus, unsigned int max)
+{
+ unsigned int n = 0;
+ int cpu;
+
+ for_each_online_cpu(cpu) {
+ if (n == max)
+ break;
+ cpus[n++] = cpu;
+ }
+ return n;
+}
+
+static void bench_stop(struct bench_thread *t, unsigned int n)
+{
+ unsigned int i;
+
+ for (i = 0; i < n; i++)
+ kthread_stop_put(t[i].task);
+}
+
+/*
+ * Start a thread on each of the @n CPUs of @cpus, waiting for the start
+ * signal. On failure, cancel and join the threads already started.
+ */
+static int bench_start(struct bench_shared *sh, struct bench_thread *t, const int *cpus,
+ unsigned int n)
+{
+ unsigned int i;
+
+ for (i = 0; i < n; i++) {
+ t[i].sh = sh;
+ t[i].task = kthread_create_on_cpu(sh->pin_period != UINT_MAX ? bench_churn_fn :
+ sh->lifecycle ? bench_lifecycle_fn :
+ bench_update_fn, &t[i], cpus[i], "hpcc_bench/%u");
+ if (IS_ERR(t[i].task)) {
+ int ret = PTR_ERR(t[i].task);
+
+ WRITE_ONCE(sh->stop, true);
+ /* Pairs with bench_wait_go(): the threads see stop with go. */
+ smp_store_release(&sh->go, true);
+ bench_stop(t, i);
+ return ret;
+ }
+ get_task_struct(t[i].task);
+ wake_up_process(t[i].task);
+ }
+ while (atomic_read(&sh->ready) < n)
+ usleep_range(1000, 2000);
+ return 0;
+}
+
+enum bench_mode { BENCH_SHARED, BENCH_PRIVATE, BENCH_MM, BENCH_SINGLE };
+static const char * const mode_name[] = { "shared", "private", "mm", "single" };
+
+/*
+ * BENCH_SHARED: all threads update one counter.
+ * BENCH_PRIVATE: each thread updates its own counter (a separate mm).
+ * BENCH_MM: blocks of n / LIFECYCLE_NR threads each update one counter of
+ * one mm. Its counters share the cache lines of their tree items, and
+ * CPUs sharing an item are strided, so each line is updated for every
+ * counter.
+ * BENCH_SINGLE: each thread owns a lazily initialized counter, like a
+ * single-threaded process, never activated. Its IMPL_PERCPU_COUNTER
+ * comparison is BENCH_PRIVATE.
+ */
+/* Counters @i of bench_update(), @c_stride and @pc_stride bytes apart. */
+#define BENCH_C(i) ((struct percpu_counter_tree *)((char *)c + (i) * c_stride))
+#define BENCH_PC(i) ((struct percpu_counter *)((char *)pc + (i) * pc_stride))
+
+static void bench_update(struct kunit *test, enum bench_mode mode, int wl,
+ enum bench_impl impl)
+{
+ bool shared = mode == BENCH_SHARED || mode == BENCH_MM;
+ unsigned int n, i, nr_counters, nr_init = 0;
+ struct bench_shared sh = { .wl = wl, .pin_period = UINT_MAX, .impl = impl,
+ .owner = mode == BENCH_SINGLE };
+ struct percpu_counter_tree *c;
+ struct percpu_counter *pc;
+ struct bench_thread *t;
+ unsigned long ops = 0;
+ unsigned int *perm;
+ size_t c_stride, pc_stride;
+ void *c_mem, *pc_mem;
+ u64 t0, t1;
+ int *cpus;
+ int ret;
+
+ cpus = kunit_kcalloc(test, nr_cpu_ids, sizeof(*cpus), GFP_KERNEL);
+ t = bench_kvzalloc(test, array_size(nr_cpu_ids, sizeof(*t)));
+ perm = kunit_kcalloc(test, nr_cpu_ids, sizeof(*perm), GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, cpus);
+ KUNIT_ASSERT_NOT_NULL(test, t);
+ KUNIT_ASSERT_NOT_NULL(test, perm);
+ /*
+ * Set up the counters for the CPUs online now, before taking
+ * cpus_read_lock(), which a failed assertion would leave held. The
+ * threads then run on the CPUs online under the lock, at most as many.
+ */
+ n = num_online_cpus();
+ nr_counters = mode == BENCH_SHARED ? 1 : mode == BENCH_MM ? LIFECYCLE_NR : n;
+ /*
+ * Each private counter is written by its own thread: give each its
+ * own cache line, as the RSS counters of separate mms, to avoid false
+ * sharing. The counters of one mm stay packed, as in mm_struct.
+ */
+ c_stride = shared ? sizeof(*c) : ALIGN(sizeof(*c), SMP_CACHE_BYTES);
+ pc_stride = shared ? sizeof(*pc) : ALIGN(sizeof(*pc), SMP_CACHE_BYTES);
+ c_mem = kunit_kzalloc(test, nr_counters * c_stride + SMP_CACHE_BYTES, GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, c_mem);
+ c = PTR_ALIGN(c_mem, SMP_CACHE_BYTES);
+ pc_mem = kunit_kzalloc(test, nr_counters * pc_stride + SMP_CACHE_BYTES, GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, pc_mem);
+ pc = PTR_ALIGN(pc_mem, SMP_CACHE_BYTES);
+ /*
+ * Each private counter stands for a separate mm: initialize them
+ * separately so they do not share the cache lines of their tree items.
+ */
+ if (impl == IMPL_PERCPU_COUNTER && mode == BENCH_MM) {
+ if (!percpu_counter_init_many(pc, 0, GFP_KERNEL, nr_counters))
+ nr_init = nr_counters;
+ } else if (impl == IMPL_PERCPU_COUNTER) {
+ for (; nr_init < nr_counters; nr_init++)
+ if (percpu_counter_init(BENCH_PC(nr_init), 0, GFP_KERNEL))
+ break;
+ } else if (mode == BENCH_MM) {
+ if (!percpu_counter_tree_init_many(c, nr_counters, batch, GFP_KERNEL))
+ nr_init = nr_counters;
+ } else {
+ for (; nr_init < nr_counters; nr_init++) {
+ if (mode == BENCH_SINGLE ?
+ percpu_counter_tree_init_many_lazy(BENCH_C(nr_init), 1, batch) :
+ percpu_counter_tree_init(BENCH_C(nr_init), batch, GFP_KERNEL))
+ break;
+ }
+ }
+ if (nr_init < nr_counters) {
+ KUNIT_FAIL(test, "cannot initialize %u counters", nr_counters);
+ goto destroy;
+ }
+ cpus_read_lock();
+ n = bench_cpus(cpus, n);
+ /* Random placement of the private counters. */
+ for (i = 0; i < n; i++)
+ perm[i] = mode == BENCH_SHARED ? 0 : mode == BENCH_MM ? i * LIFECYCLE_NR / n : i;
+ for (i = n - 1; !shared && i > 0; i--)
+ swap(perm[i], perm[get_random_u32_below(i + 1)]);
+ for (i = 0; i < n; i++) {
+ t[i].c = BENCH_C(perm[i]);
+ t[i].pc = BENCH_PC(perm[i]);
+ t[i].seed = get_random_u32() | 1;
+ }
+ ret = bench_start(&sh, t, cpus, n);
+ if (ret) {
+ cpus_read_unlock();
+ KUNIT_FAIL(test, "cannot start the benchmark threads: %d", ret);
+ goto destroy;
+ }
+ t0 = ktime_get_ns();
+ WRITE_ONCE(sh.go, true);
+ msleep(BENCH_MS);
+ WRITE_ONCE(sh.stop, true);
+ t1 = ktime_get_ns();
+ bench_stop(t, n);
+ cpus_read_unlock();
+ for (i = 0; i < n; i++)
+ ops += t[i].ops;
+ for (i = 0; impl == IMPL_TREE && i < nr_counters; i++)
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_check_residuals(BENCH_C(i)));
+ kunit_info(test, "update %-8s %-14s %3u thr %10llu ns/op/thr %s\n",
+ mode_name[mode], wname[wl], n,
+ ops ? div64_ul((t1 - t0) * n, ops) : 0, impl_name[impl]);
+destroy:
+ if (impl == IMPL_PERCPU_COUNTER && mode == BENCH_MM) {
+ if (nr_init)
+ percpu_counter_destroy_many(pc, nr_counters);
+ } else if (impl == IMPL_PERCPU_COUNTER) {
+ for (i = 0; i < nr_init; i++)
+ percpu_counter_destroy(BENCH_PC(i));
+ } else if (mode == BENCH_MM) {
+ if (nr_init)
+ percpu_counter_tree_destroy_many(c, nr_counters);
+ } else {
+ for (i = 0; i < nr_init; i++)
+ percpu_counter_tree_destroy(BENCH_C(i));
+ }
+ kunit_kfree(test, pc_mem);
+ kunit_kfree(test, c_mem);
+ kunit_kfree(test, perm);
+ bench_kvfree_now(test, t);
+ kunit_kfree(test, cpus);
+}
+
+static void bench_lifecycle(struct kunit *test, bool all, enum bench_impl impl, bool lazy)
+{
+ struct bench_shared sh = { .lifecycle = true, .pin_period = UINT_MAX,
+ .impl = impl, .lazy = lazy };
+ unsigned int n, i;
+ struct bench_thread *t;
+ unsigned long ops = 0;
+ u64 ns = 0;
+ int *cpus;
+ int ret;
+
+ cpus = kunit_kcalloc(test, nr_cpu_ids, sizeof(*cpus), GFP_KERNEL);
+ t = bench_kvzalloc(test, array_size(nr_cpu_ids, sizeof(*t)));
+ KUNIT_ASSERT_NOT_NULL(test, cpus);
+ KUNIT_ASSERT_NOT_NULL(test, t);
+ cpus_read_lock();
+ n = bench_cpus(cpus, all ? num_online_cpus() : 1);
+ ret = bench_start(&sh, t, cpus, n);
+ if (ret) {
+ cpus_read_unlock();
+ KUNIT_FAIL(test, "cannot start the benchmark threads: %d", ret);
+ goto free;
+ }
+ WRITE_ONCE(sh.go, true);
+ bench_stop(t, n);
+ cpus_read_unlock();
+ for (i = 0; i < n; i++) {
+ ops += t[i].ops;
+ ns += t[i].ns;
+ }
+ KUNIT_EXPECT_EQ(test, ops, (unsigned long)n * LIFECYCLE_ITER);
+ kunit_info(test, "lifecycle init_many%s(%d)+destroy_many %3u thr %10llu ns/op %s\n",
+ lazy ? "_lazy" : "", LIFECYCLE_NR, n, ops ? div64_ul(ns, ops) : 0,
+ impl_name[impl]);
+free:
+ bench_kvfree_now(test, t);
+ kunit_kfree(test, cpus);
+}
+
+static void bench_churn(struct kunit *test, bool all, unsigned int pin_period,
+ enum bench_impl impl)
+{
+ struct bench_shared sh = { .pin_period = pin_period, .impl = impl };
+ unsigned int n, i, j, max_pinned = pin_period ? DIV_ROUND_UP(CHURN_ITER, pin_period) : 0;
+ unsigned long ops = 0, live = 0;
+ unsigned int nr_alloc;
+ struct bench_thread *t;
+ u64 ns = 0;
+ int *cpus;
+ int ret;
+
+ cpus = kunit_kcalloc(test, nr_cpu_ids, sizeof(*cpus), GFP_KERNEL);
+ t = bench_kvzalloc(test, array_size(nr_cpu_ids, sizeof(*t)));
+ KUNIT_ASSERT_NOT_NULL(test, cpus);
+ KUNIT_ASSERT_NOT_NULL(test, t);
+ /* Allocate before taking cpus_read_lock(), as in bench_update(). */
+ nr_alloc = all ? num_online_cpus() : 1;
+ for (i = 0; i < nr_alloc; i++) {
+ t[i].pinned = bench_kvzalloc(test, array_size(max_pinned * LIFECYCLE_NR + 1,
+ sizeof(*t[i].pinned)));
+ t[i].pinned_pc = bench_kvzalloc(test, array_size(max_pinned * LIFECYCLE_NR + 1,
+ sizeof(*t[i].pinned_pc)));
+ KUNIT_ASSERT_NOT_NULL(test, t[i].pinned_pc);
+ KUNIT_ASSERT_NOT_NULL(test, t[i].pinned);
+ }
+ cpus_read_lock();
+ n = bench_cpus(cpus, nr_alloc);
+ ret = bench_start(&sh, t, cpus, n);
+ if (ret) {
+ cpus_read_unlock();
+ KUNIT_FAIL(test, "cannot start the benchmark threads: %d", ret);
+ goto destroy;
+ }
+ WRITE_ONCE(sh.go, true);
+ bench_stop(t, n);
+ cpus_read_unlock();
+ for (i = 0; i < n; i++) {
+ ops += t[i].ops;
+ ns += t[i].ns;
+ live += t[i].nr_pinned * LIFECYCLE_NR;
+ }
+ KUNIT_EXPECT_EQ(test, ops, (unsigned long)n * CHURN_ITER);
+ kunit_info(test, "churn pin 1/%-4u %3u thr %8llu ns/op live %6lu %s\n",
+ pin_period, n, ops ? div64_ul(ns, ops) : 0, live, impl_name[impl]);
+destroy:
+ for (i = 0; i < nr_alloc; i++) {
+ for (j = 0; j < t[i].nr_pinned; j++) {
+ if (impl == IMPL_PERCPU_COUNTER)
+ percpu_counter_destroy_many(&t[i].pinned_pc[j * LIFECYCLE_NR],
+ LIFECYCLE_NR);
+ else
+ percpu_counter_tree_destroy_many(&t[i].pinned[j * LIFECYCLE_NR],
+ LIFECYCLE_NR);
+ }
+ bench_kvfree_now(test, t[i].pinned_pc);
+ bench_kvfree_now(test, t[i].pinned);
+ }
+ bench_kvfree_now(test, t);
+ kunit_kfree(test, cpus);
+}
+
+static void hpcc_bench(struct kunit *test)
+{
+ static const unsigned int pin_periods[] = { 0, 256, 64, 16, 4 };
+ struct percpu_counter_tree probe;
+ unsigned int p;
+ int wl;
+
+ if (batch < 2 || !is_power_of_2(batch) || batch > PERCPU_COUNTER_TREE_MAX_BATCH)
+ kunit_skip(test, "invalid batch size %u", batch);
+ KUNIT_ASSERT_EQ(test, 0, percpu_counter_tree_init(&probe, batch, GFP_KERNEL));
+ kunit_info(test, "%u CPUs online, nr_cpu_ids %u\n", num_online_cpus(), nr_cpu_ids);
+ kunit_info(test, "%s batch %u, accuracy +-%lu\n", impl_name[IMPL_TREE], batch,
+ percpu_counter_tree_approximate_accuracy(&probe));
+ kunit_info(test, "%s batch %d, accuracy +-%lu\n", impl_name[IMPL_PERCPU_COUNTER],
+ percpu_counter_batch, (unsigned long)nr_cpu_ids * (percpu_counter_batch - 1));
+ percpu_counter_tree_destroy(&probe);
+ bench_lifecycle(test, false, IMPL_TREE, false);
+ bench_lifecycle(test, false, IMPL_TREE, true);
+ bench_lifecycle(test, false, IMPL_PERCPU_COUNTER, false);
+ bench_lifecycle(test, true, IMPL_TREE, false);
+ bench_lifecycle(test, true, IMPL_TREE, true);
+ bench_lifecycle(test, true, IMPL_PERCPU_COUNTER, false);
+ for (p = 0; p < ARRAY_SIZE(pin_periods); p++) {
+ bench_churn(test, false, pin_periods[p], IMPL_TREE);
+ bench_churn(test, false, pin_periods[p], IMPL_PERCPU_COUNTER);
+ bench_churn(test, true, pin_periods[p], IMPL_TREE);
+ bench_churn(test, true, pin_periods[p], IMPL_PERCPU_COUNTER);
+ }
+ for (wl = 0; wl < NR_W; wl++) {
+ bench_update(test, BENCH_SHARED, wl, IMPL_TREE);
+ bench_update(test, BENCH_SHARED, wl, IMPL_PERCPU_COUNTER);
+ bench_update(test, BENCH_PRIVATE, wl, IMPL_TREE);
+ bench_update(test, BENCH_PRIVATE, wl, IMPL_PERCPU_COUNTER);
+ bench_update(test, BENCH_MM, wl, IMPL_TREE);
+ bench_update(test, BENCH_MM, wl, IMPL_PERCPU_COUNTER);
+ bench_update(test, BENCH_SINGLE, wl, IMPL_TREE);
+ }
+}
+
+static struct kunit_case hpcc_bench_cases[] = {
+ KUNIT_CASE_SLOW(hpcc_bench),
+ {}
+};
+
+static struct kunit_suite hpcc_bench_suite = {
+ .name = "percpu_counter_tree_bench",
+ .test_cases = hpcc_bench_cases,
+};
+
+kunit_test_suite(hpcc_bench_suite);
+
+MODULE_DESCRIPTION("Benchmark for hierarchical per-CPU counters");
+MODULE_LICENSE("Dual MIT/GPL");
+MODULE_IMPORT_NS("EXPORTED_FOR_KUNIT_TESTING");
diff --git a/lib/tests/percpu_counter_tree_kunit.c b/lib/tests/percpu_counter_tree_kunit.c
new file mode 100644
index 00000000000000..2f5852b1f52140
--- /dev/null
+++ b/lib/tests/percpu_counter_tree_kunit.c
@@ -0,0 +1,1050 @@
+// SPDX-License-Identifier: GPL-2.0+ OR MIT
+// SPDX-FileCopyrightText: 2026 Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
+
+#include <kunit/test.h>
+#include <linux/delay.h>
+#include <linux/mutex.h>
+#include <linux/percpu_counter_tree.h>
+#include <linux/kthread.h>
+#include <linux/wait.h>
+#include <linux/random.h>
+#include <linux/cpuhplock.h>
+#include <linux/slab.h>
+#include <linux/sched/task.h>
+#include <linux/workqueue.h>
+
+struct multi_thread_test_data {
+ struct work_struct work;
+ long increment;
+ unsigned int nr_inc;
+ int counter_index;
+};
+
+#define NR_COUNTERS 2
+
+/* Hierarchical per-CPU counter instances. */
+static struct percpu_counter_tree counter[NR_COUNTERS];
+
+/* Global atomic counters for validation. */
+static atomic_long_t global_counter[NR_COUNTERS];
+
+static DECLARE_WAIT_QUEUE_HEAD(kernel_threads_wq);
+static atomic_t kernel_threads_to_run;
+
+static void complete_work(void)
+{
+ if (atomic_dec_and_test(&kernel_threads_to_run))
+ wake_up(&kernel_threads_wq);
+}
+
+/* Random tasks test. */
+#define NR_RAND_TASKS 1000
+
+static void hpcc_print_info(struct kunit *test)
+{
+ kunit_info(test, "Running test with %d CPUs\n", num_online_cpus());
+}
+
+static void add_to_counter(int counter_index, unsigned int nr_inc, long increment)
+{
+ unsigned int i;
+
+ for (i = 0; i < nr_inc; i++) {
+ percpu_counter_tree_add(&counter[counter_index], increment);
+ atomic_long_add(increment, &global_counter[counter_index]);
+ }
+}
+
+/* Check the residuals and the carries of a settled counter. */
+static void check_residuals(struct kunit *test, struct percpu_counter_tree *c)
+{
+#ifdef CONFIG_SMP
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_check_residuals(c));
+#endif
+}
+
+static void check_counters(struct kunit *test)
+{
+ int counter_index;
+
+ /* Compare each counter with its global counter. */
+ for (counter_index = 0; counter_index < NR_COUNTERS; counter_index++) {
+ struct percpu_counter_tree *c = &counter[counter_index];
+ long v = atomic_long_read(&global_counter[counter_index]);
+ long approx_sum = percpu_counter_tree_approximate_sum(c);
+ unsigned long accuracy;
+ long precise_min, precise_max;
+
+ /* Precise comparison. */
+ KUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(c), v);
+ check_residuals(test, c);
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_precise_compare_value(c, v));
+
+ /* Approximate comparison. */
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare_value(c, v));
+
+ /* Accuracy limits checks. */
+ accuracy = percpu_counter_tree_approximate_accuracy(c);
+
+ /*
+ * The accuracy is an unsigned long, so the inner expression
+ * promotes to unsigned. Compute the difference in unsigned and
+ * cast to signed for the comparison against 0.
+ */
+ KUNIT_EXPECT_GE(test, (long)(approx_sum - (v - accuracy)), 0);
+ KUNIT_EXPECT_LE(test, (long)(approx_sum - (v + accuracy)), 0);
+
+ /* Precise min/max range check. */
+ percpu_counter_tree_approximate_min_max_range(approx_sum, accuracy,
+ &precise_min, &precise_max);
+
+ KUNIT_EXPECT_GE(test, v - precise_min, 0);
+ KUNIT_EXPECT_LE(test, v - precise_max, 0);
+ }
+ /* Compare each counter with the second counter. */
+ KUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(&counter[0]),
+ percpu_counter_tree_precise_sum(&counter[1]));
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_precise_compare(&counter[0], &counter[1]));
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare(&counter[0], &counter[1]));
+}
+
+/*
+ * Workqueue running the updates of the multi-threaded tests. Its workers
+ * are reused across rounds, which keeps the tests fast on machines where
+ * creating threads on remote CPUs is slow. Work items are CPU intensive,
+ * so several of them queued on one CPU run concurrently in separate
+ * workers, and can preempt each other in the middle of an update.
+ */
+static struct workqueue_struct *hpcc_test_wq;
+
+static void multi_thread_work_fn(struct work_struct *work)
+{
+ struct multi_thread_test_data *td =
+ container_of(work, struct multi_thread_test_data, work);
+
+ add_to_counter(td->counter_index, td->nr_inc, td->increment);
+ kfree(td);
+ complete_work();
+}
+
+/* Queue an update of a counter on @target_cpu. Call with the CPU hotplug lock held. */
+static int test_queue_on_cpu(int target_cpu, int counter_index, unsigned int nr_inc,
+ long increment)
+{
+ struct multi_thread_test_data *td = kzalloc(sizeof(*td), GFP_KERNEL);
+
+ if (!td)
+ return -ENOMEM;
+ INIT_WORK(&td->work, multi_thread_work_fn);
+ td->increment = increment;
+ td->nr_inc = nr_inc;
+ td->counter_index = counter_index;
+ atomic_inc(&kernel_threads_to_run);
+ queue_work_on(target_cpu, hpcc_test_wq, &td->work);
+ return 0;
+}
+
+static void init_kthreads(void)
+{
+ atomic_set(&kernel_threads_to_run, 1);
+}
+
+/*
+ * The caller needs to wait for work completion _before_ stopping kthreads.
+ */
+static void fini_kthreads(void)
+{
+ /* Release our own reference. */
+ complete_work();
+ /* Wait for all others threads to run. */
+ wait_event(kernel_threads_wq, (atomic_read(&kernel_threads_to_run) == 0));
+}
+
+static int init_counters(unsigned long batch_size)
+{
+ int i, ret;
+
+ ret = percpu_counter_tree_init_many(counter, NR_COUNTERS, batch_size, GFP_KERNEL);
+ if (ret)
+ return ret;
+ for (i = 0; i < NR_COUNTERS; i++)
+ atomic_long_set(&global_counter[i], 0);
+ return 0;
+}
+
+static void fini_counters(void)
+{
+ percpu_counter_tree_destroy_many(counter, NR_COUNTERS);
+}
+
+enum up_test_inc_type {
+ INC_ONE,
+ INC_MINUS_ONE,
+ INC_RANDOM,
+};
+
+/*
+ * Single-threaded tests. Those run the updates on various CPUs, one at a
+ * time: work_on_cpu() waits for each update to complete before the next,
+ * effectively making sure there are no concurrent updates. Each test
+ * stops early after SINGLE_THREAD_MAX_SECONDS, because cross-CPU wakeups
+ * can be slow on large virtual machines.
+ */
+#define SINGLE_THREAD_MAX_SECONDS 10
+
+struct single_thread_update {
+ long increment;
+ int counter_index;
+};
+
+static long single_thread_update_fn(void *arg)
+{
+ struct single_thread_update *u = arg;
+
+ add_to_counter(u->counter_index, 1, u->increment);
+ return 0;
+}
+
+static void do_hpcc_test_single_thread(struct kunit *test, int _cpu0, int _cpu1,
+ enum up_test_inc_type type)
+{
+ unsigned long batch_size_order = 5;
+ int cpu0 = _cpu0;
+ int cpu1 = _cpu1;
+ unsigned long deadline = jiffies + SINGLE_THREAD_MAX_SECONDS * HZ;
+ int ret, i;
+
+ ret = init_counters(1UL << batch_size_order);
+ KUNIT_EXPECT_EQ(test, ret, 0);
+ if (ret)
+ return;
+ cpus_read_lock();
+ for (i = 0; i < 10000 && time_before(jiffies, deadline); i++) {
+ struct single_thread_update u0 = { .counter_index = 0 };
+ struct single_thread_update u1 = { .counter_index = 1 };
+ long increment;
+
+ switch (type) {
+ case INC_ONE:
+ increment = 1;
+ break;
+ case INC_MINUS_ONE:
+ increment = -1;
+ break;
+ case INC_RANDOM:
+ increment = (long)get_random_long() % 50000;
+ break;
+ default:
+ WARN_ON_ONCE(1);
+ increment = 0;
+ }
+ if (_cpu0 < 0)
+ cpu0 = cpumask_any_distribute(cpu_online_mask);
+ if (_cpu1 < 0)
+ cpu1 = cpumask_any_distribute(cpu_online_mask);
+ u0.increment = increment;
+ u1.increment = increment;
+ work_on_cpu(cpu0, single_thread_update_fn, &u0);
+ work_on_cpu(cpu1, single_thread_update_fn, &u1);
+ check_counters(test);
+ }
+ cpus_read_unlock();
+ fini_counters();
+}
+
+static void hpcc_test_single_thread_first(struct kunit *test)
+{
+ int cpu = cpumask_first(cpu_online_mask);
+
+ do_hpcc_test_single_thread(test, cpu, cpu, INC_ONE);
+ do_hpcc_test_single_thread(test, cpu, cpu, INC_MINUS_ONE);
+ do_hpcc_test_single_thread(test, cpu, cpu, INC_RANDOM);
+}
+
+static void hpcc_test_single_thread_first_random(struct kunit *test)
+{
+ int cpu = cpumask_first(cpu_online_mask);
+
+ do_hpcc_test_single_thread(test, cpu, -1, INC_ONE);
+ do_hpcc_test_single_thread(test, cpu, -1, INC_MINUS_ONE);
+ do_hpcc_test_single_thread(test, cpu, -1, INC_RANDOM);
+}
+
+static void hpcc_test_single_thread_random(struct kunit *test)
+{
+ do_hpcc_test_single_thread(test, -1, -1, INC_ONE);
+ do_hpcc_test_single_thread(test, -1, -1, INC_MINUS_ONE);
+ do_hpcc_test_single_thread(test, -1, -1, INC_RANDOM);
+}
+
+/* Multi-threaded SMP tests. */
+
+/*
+ * Bound the number of CPUs running workers in the multi-threaded tests,
+ * so their run time does not grow with the size of the machine. Beyond
+ * this bound, the workers run on a subset spread evenly across the
+ * online CPUs, so they still reach every branch of the tree.
+ */
+#define MAX_WORKER_CPUS 64
+
+/*
+ * Each multi-threaded test stops early after MULTI_THREAD_MAX_SECONDS,
+ * because cross-CPU wakeups can be slow on large virtual machines.
+ */
+#define MULTI_THREAD_MAX_SECONDS 60
+
+static struct cpumask worker_cpus;
+
+/* Select the CPUs running workers. Call with the CPU hotplug lock held. */
+static const struct cpumask *select_worker_cpus(void)
+{
+ unsigned int nr_online = num_online_cpus(), i = 0;
+ int cpu;
+
+ cpumask_clear(&worker_cpus);
+ for_each_online_cpu(cpu) {
+ /* Pick MAX_WORKER_CPUS of the nr_online CPUs, evenly spaced. */
+ if ((i + 1) * MAX_WORKER_CPUS / nr_online !=
+ i * MAX_WORKER_CPUS / nr_online)
+ cpumask_set_cpu(cpu, &worker_cpus);
+ i++;
+ }
+ return &worker_cpus;
+}
+
+/*
+ * CPU selection for the second counter's thread, relative to the CPU
+ * running the first counter's thread.
+ */
+enum mt_cpu_select {
+ MT_SAME_CPU, /* Same CPU as the first counter. */
+ MT_EVEN_CPU, /* Funnel CPU pairs (0, 1), (2, 3), ... to 0, 2, ... */
+ MT_FIRST_CPU, /* First online CPU. */
+ MT_RANDOM_CPU, /* Any online CPU. */
+};
+
+/* Return the CPU for the second counter's thread, or -1 to skip @cpu. */
+static int mt_select_cpu(enum mt_cpu_select select, int cpu)
+{
+ int even_cpu;
+
+ switch (select) {
+ case MT_SAME_CPU:
+ return cpu;
+ case MT_EVEN_CPU:
+ even_cpu = cpu & ~1;
+ return cpumask_test_cpu(even_cpu, cpu_online_mask) ? even_cpu : -1;
+ case MT_FIRST_CPU:
+ return cpumask_first(cpu_online_mask);
+ case MT_RANDOM_CPU:
+ return cpumask_any_distribute(cpu_online_mask);
+ }
+ WARN_ON_ONCE(1);
+ return -1;
+}
+
+static void do_hpcc_multi_thread_increment(struct kunit *test, enum mt_cpu_select select,
+ unsigned long batch_size, unsigned int nr_inc,
+ long increment)
+{
+ const struct cpumask *cpus;
+ int ret, cpu;
+
+ ret = init_counters(batch_size);
+ KUNIT_EXPECT_EQ(test, ret, 0);
+ if (ret)
+ return;
+ init_kthreads();
+ cpus_read_lock();
+ cpus = select_worker_cpus();
+ for_each_cpu(cpu, cpus) {
+ int cpu1 = mt_select_cpu(select, cpu);
+
+ if (cpu1 < 0)
+ continue;
+ ret = test_queue_on_cpu(cpu, 0, nr_inc, increment);
+ KUNIT_EXPECT_EQ(test, ret, 0);
+ if (ret)
+ break;
+ ret = test_queue_on_cpu(cpu1, 1, nr_inc, increment);
+ KUNIT_EXPECT_EQ(test, ret, 0);
+ if (ret)
+ break;
+ }
+ fini_kthreads();
+ cpus_read_unlock();
+ check_counters(test);
+ fini_counters();
+}
+
+static void hpcc_test_multi_thread_batch_increment(struct kunit *test)
+{
+ unsigned long deadline = jiffies + MULTI_THREAD_MAX_SECONDS * HZ;
+ unsigned long batch_size_order;
+
+ for (batch_size_order = 2; batch_size_order <= 7; batch_size_order++) {
+ unsigned long batch_size = 1UL << batch_size_order;
+ unsigned int nr_inc;
+
+ for (nr_inc = 1; nr_inc < 1024; nr_inc *= 2) {
+ long increment;
+
+ for (increment = 1; increment < 100000; increment *= 10) {
+ enum mt_cpu_select select;
+
+ if (time_after(jiffies, deadline))
+ return;
+ for (select = MT_SAME_CPU; select <= MT_RANDOM_CPU; select++)
+ do_hpcc_multi_thread_increment(test, select, batch_size,
+ nr_inc, increment);
+ }
+ }
+ }
+}
+
+static void hpcc_test_multi_thread_random_walk(struct kunit *test)
+{
+ unsigned long deadline = jiffies + MULTI_THREAD_MAX_SECONDS * HZ;
+ unsigned long batch_size_order = 5;
+ int loop;
+
+ for (loop = 0; loop < 100 && time_before(jiffies, deadline); loop++) {
+ int ret, i;
+
+ ret = init_counters(1UL << batch_size_order);
+ KUNIT_EXPECT_EQ(test, ret, 0);
+ if (ret)
+ return;
+ init_kthreads();
+ cpus_read_lock();
+ for (i = 0; i < NR_RAND_TASKS; i++) {
+ long increment = (long)get_random_long() % 512;
+ unsigned int nr_inc = ((unsigned long)get_random_long()) % 1024;
+
+ ret = test_queue_on_cpu(cpumask_any_distribute(cpu_online_mask),
+ 0, nr_inc, increment);
+ KUNIT_EXPECT_EQ(test, ret, 0);
+ if (ret)
+ break;
+ ret = test_queue_on_cpu(cpumask_any_distribute(cpu_online_mask),
+ 1, nr_inc, increment);
+ KUNIT_EXPECT_EQ(test, ret, 0);
+ if (ret)
+ break;
+ }
+ fini_kthreads();
+ cpus_read_unlock();
+ check_counters(test);
+ fini_counters();
+ }
+}
+
+#define CONTENTION_NR_ROUNDS 20000
+#define CONTENTION_NR_INC 8
+
+struct contention_shared {
+ unsigned long batch_size;
+ atomic_t round; /* Round started by the coordinator. */
+ atomic_t nr_done; /* Rounds completed by the other workers. */
+ unsigned int nr_workers;
+ unsigned int nr_errors; /* Rounds ending with invalid residuals. */
+ bool started;
+};
+
+struct contention_data {
+ struct contention_shared *sh;
+ long sum;
+ u32 seed;
+ bool coordinator;
+} ____cacheline_aligned_in_smp;
+
+static DECLARE_WAIT_QUEUE_HEAD(contention_wq);
+
+/*
+ * Add random multiples of the batch size, of either sign, so each
+ * addition carries into the counters above level 0. Increments of
+ * opposite signs and of more than one batch size exercise concurrent
+ * carry extractions retrying after a concurrent update of the residual.
+ */
+static void contention_add(struct contention_data *cd)
+{
+ unsigned int i;
+
+ for (i = 0; i < CONTENTION_NR_INC; i++) {
+ u32 x = cd->seed;
+ long inc;
+
+ /* xorshift32: get_random_u32() would dominate the loop. */
+ x ^= x << 13;
+ x ^= x >> 17;
+ x ^= x << 5;
+ cd->seed = x;
+ inc = (long)((x & 15) + 1) * cd->sh->batch_size;
+ if (x & 16)
+ inc = -inc;
+ percpu_counter_tree_add(&counter[0], inc);
+ cd->sum += inc;
+ }
+}
+
+/*
+ * The workers run rounds of a few additions each, all starting together.
+ * Between rounds, the coordinator checks the residuals of the settled
+ * counter: an invalid residual left by the last updaters of a round would
+ * otherwise be repaired by the next additions.
+ */
+static int contention_worker_fn(void *data)
+{
+ struct contention_data *cd = data;
+ struct contention_shared *sh = cd->sh;
+ int round;
+
+ wait_event(contention_wq, READ_ONCE(sh->started));
+ for (round = 1; round <= CONTENTION_NR_ROUNDS; round++) {
+ if (cd->coordinator) {
+ atomic_set_release(&sh->round, round);
+ contention_add(cd);
+ while (atomic_read_acquire(&sh->nr_done) < round * (sh->nr_workers - 1))
+ cond_resched();
+#ifdef CONFIG_SMP
+ if (percpu_counter_tree_check_residuals(&counter[0]))
+ sh->nr_errors++;
+#endif
+ } else {
+ while (atomic_read_acquire(&sh->round) < round)
+ cond_resched();
+ contention_add(cd);
+ /* Order the additions before the coordinator check. */
+ atomic_inc_return_release(&sh->nr_done);
+ }
+ }
+ complete_work();
+ return 0;
+}
+
+static void do_hpcc_test_contended_carries(struct kunit *test, unsigned long batch_size)
+{
+ struct percpu_counter_tree *c = &counter[0];
+ struct contention_shared sh = { .batch_size = batch_size };
+ struct contention_data *cd;
+ struct task_struct **task;
+ const struct cpumask *cpus;
+ unsigned long accuracy;
+ long expected = 0, approx;
+ bool first = true;
+ int ret, cpu;
+
+ cd = kvcalloc(nr_cpu_ids, sizeof(*cd), GFP_KERNEL);
+ task = kvcalloc(nr_cpu_ids, sizeof(*task), GFP_KERNEL);
+ KUNIT_EXPECT_NOT_NULL(test, cd);
+ KUNIT_EXPECT_NOT_NULL(test, task);
+ if (!cd || !task)
+ goto free;
+ ret = init_counters(batch_size);
+ KUNIT_EXPECT_EQ(test, ret, 0);
+ if (ret)
+ goto free;
+ init_kthreads();
+ cpus_read_lock();
+ cpus = select_worker_cpus();
+ for_each_cpu(cpu, cpus) {
+ cd[cpu].sh = &sh;
+ cd[cpu].coordinator = first;
+ cd[cpu].seed = get_random_u32() | 1;
+ task[cpu] = kthread_create_on_cpu(contention_worker_fn, &cd[cpu], cpu,
+ "kunit_hpcc_contention/%u");
+ KUNIT_EXPECT_FALSE(test, IS_ERR(task[cpu]));
+ if (IS_ERR(task[cpu]))
+ break;
+ atomic_inc(&kernel_threads_to_run);
+ get_task_struct(task[cpu]);
+ wake_up_process(task[cpu]);
+ sh.nr_workers++;
+ first = false;
+ }
+ /* Start the workers once their number is known. */
+ WRITE_ONCE(sh.started, true);
+ wake_up_all(&contention_wq);
+ fini_kthreads();
+ for_each_cpu(cpu, cpus) {
+ if (IS_ERR_OR_NULL(task[cpu]))
+ continue;
+ kthread_stop_put(task[cpu]);
+ expected += cd[cpu].sum;
+ }
+ cpus_read_unlock();
+
+ kunit_info(test, "batch_size=%lu workers=%u rounds with invalid residuals=%u\n",
+ batch_size, sh.nr_workers, sh.nr_errors);
+ KUNIT_EXPECT_EQ(test, sh.nr_errors, 0);
+ KUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(c), expected);
+ check_residuals(test, c);
+ approx = percpu_counter_tree_approximate_sum(c);
+ accuracy = percpu_counter_tree_approximate_accuracy(c);
+ KUNIT_EXPECT_GE(test, approx, expected - (long)accuracy);
+ KUNIT_EXPECT_LE(test, approx, expected + (long)accuracy);
+ fini_counters();
+free:
+ kvfree(task);
+ kvfree(cd);
+}
+
+/*
+ * Concurrent carries from the worker CPUs into the counters above level 0
+ * which they share, with a check that the residuals are back within range
+ * and that no carry was lost or duplicated after each round.
+ */
+static void hpcc_test_contended_carries(struct kunit *test)
+{
+ unsigned long batch_size_order;
+
+ if (num_online_cpus() < 2)
+ kunit_skip(test, "needs at least 2 online CPUs");
+ for (batch_size_order = 1; batch_size_order <= 5; batch_size_order += 2)
+ do_hpcc_test_contended_carries(test, 1UL << batch_size_order);
+}
+
+static void hpcc_test_init_one(struct kunit *test)
+{
+ struct percpu_counter_tree pct;
+ int ret;
+
+ ret = percpu_counter_tree_init(&pct, 32, GFP_KERNEL);
+ KUNIT_EXPECT_EQ(test, ret, 0);
+ if (ret)
+ return;
+
+ percpu_counter_tree_destroy(&pct);
+}
+
+static void hpcc_test_set(struct kunit *test)
+{
+ static long values[] = {
+ 5, 100, 127, 128, 255, 256, 4095, 4096, 500000, 0,
+ -5, -100, -127, -128, -255, -256, -4095, -4096, -500000,
+ };
+ struct percpu_counter_tree pct;
+ int i, ret;
+
+ ret = percpu_counter_tree_init(&pct, 32, GFP_KERNEL);
+ KUNIT_EXPECT_EQ(test, ret, 0);
+ if (ret)
+ return;
+
+ for (i = 0; i < ARRAY_SIZE(values); i++) {
+ long v = values[i];
+
+ percpu_counter_tree_set(&pct, v);
+ KUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(&pct), v);
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare_value(&pct, v));
+
+ percpu_counter_tree_add(&pct, v);
+ KUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(&pct), 2 * v);
+ KUNIT_EXPECT_EQ(test, 0,
+ percpu_counter_tree_approximate_compare_value(&pct, 2 * v));
+
+ percpu_counter_tree_add(&pct, -2 * v);
+ KUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(&pct), 0);
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare_value(&pct, 0));
+ }
+
+ percpu_counter_tree_destroy(&pct);
+}
+
+struct saturate_data {
+ struct percpu_counter_tree *c, *probe;
+ long inc;
+ unsigned long max_steps;
+ bool overrun;
+};
+
+/* Copy the state of counter @src into counter @dst, of the same topology. */
+static void copy_counter_state(struct percpu_counter_tree *dst,
+ struct percpu_counter_tree *src)
+{
+#ifdef CONFIG_SMP
+ percpu_counter_tree_copy_state(dst, src);
+#else
+ atomic_long_set(&dst->count, atomic_long_read(&src->count));
+#endif
+}
+
+/*
+ * Add inc (1 or -1) until the next addition would propagate a carry to the
+ * approximate sum. This leaves every tree item on the path from this CPU to
+ * the root holding its largest residual in the direction of inc which has
+ * not been propagated yet, without knowledge of the tree topology. Each
+ * addition is first tried on the probe counter, which holds the same state:
+ * reverting an addition which reached the approximate sum would not restore
+ * the residuals subtracted on the way. The path from a CPU to the root
+ * holds less than the accuracy bound: more additions mean that a carry was
+ * lost.
+ */
+static long saturate_worker_fn(void *data)
+{
+ struct saturate_data *sd = data;
+ unsigned long steps;
+
+ for (steps = 0; steps < sd->max_steps; steps++) {
+ long approx = percpu_counter_tree_approximate_sum(sd->probe);
+
+ percpu_counter_tree_add(sd->probe, sd->inc);
+ if (percpu_counter_tree_approximate_sum(sd->probe) != approx) {
+ copy_counter_state(sd->probe, sd->c);
+ break;
+ }
+ percpu_counter_tree_add(sd->c, sd->inc);
+ }
+ if (steps == sd->max_steps)
+ sd->overrun = true;
+ return 0;
+}
+
+static void do_hpcc_test_saturated_compare(struct kunit *test, unsigned long batch_size,
+ long inc)
+{
+ struct percpu_counter_tree *c = &counter[0], *exact = &counter[1], probe;
+ struct saturate_data sd = { .c = c, .probe = &probe, .inc = inc };
+ unsigned long accuracy;
+ long approx, precise, precise_min, precise_max;
+ int ret, cpu;
+
+ ret = percpu_counter_tree_init(&probe, batch_size, GFP_KERNEL);
+ KUNIT_EXPECT_EQ(test, ret, 0);
+ if (ret)
+ return;
+ ret = init_counters(batch_size);
+ KUNIT_EXPECT_EQ(test, ret, 0);
+ if (ret)
+ goto destroy_probe;
+ accuracy = percpu_counter_tree_approximate_accuracy(c);
+ sd.max_steps = batch_size + accuracy;
+ cpus_read_lock();
+ for_each_online_cpu(cpu) {
+ work_on_cpu(cpu, saturate_worker_fn, &sd);
+ KUNIT_EXPECT_FALSE(test, sd.overrun);
+ if (sd.overrun)
+ break;
+ }
+ cpus_read_unlock();
+
+ precise = percpu_counter_tree_precise_sum(c);
+ approx = percpu_counter_tree_approximate_sum(c);
+ kunit_info(test, "batch_size=%lu inc=%ld precise=%ld approx=%ld accuracy=%lu\n",
+ batch_size, inc, precise, approx, accuracy);
+
+ check_residuals(test, c);
+ check_residuals(test, &probe);
+
+ /* Accuracy contract. */
+ KUNIT_EXPECT_GE(test, approx, precise - (long)accuracy);
+ KUNIT_EXPECT_LE(test, approx, precise + (long)accuracy);
+ /* Saturating every possible CPU reaches the accuracy exactly. */
+ if (num_online_cpus() == nr_cpu_ids)
+ KUNIT_EXPECT_EQ(test, (precise - approx) * inc, (long)accuracy);
+ percpu_counter_tree_approximate_min_max_range(approx, accuracy,
+ &precise_min, &precise_max);
+ KUNIT_EXPECT_GE(test, precise, precise_min);
+ KUNIT_EXPECT_LE(test, precise, precise_max);
+
+ /* Comparisons against a value. */
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare_value(c, precise));
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_precise_compare_value(c, precise));
+ KUNIT_EXPECT_EQ(test, -1, percpu_counter_tree_precise_compare_value(c, precise - 1));
+ KUNIT_EXPECT_EQ(test, 1, percpu_counter_tree_precise_compare_value(c, precise + 1));
+
+ /* Comparisons against a counter with the same precise sum, exactly approximated. */
+ percpu_counter_tree_set(exact, precise);
+ KUNIT_EXPECT_EQ(test, percpu_counter_tree_approximate_sum(exact), precise);
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare(c, exact));
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare(exact, c));
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_precise_compare(c, exact));
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_precise_compare(exact, c));
+
+ fini_counters();
+destroy_probe:
+ percpu_counter_tree_destroy(&probe);
+}
+
+/*
+ * Saturate the unpropagated residuals of every tree item reachable from
+ * online CPUs, upward and downward, which yields the largest distance
+ * between the precise and approximate sums of a settled counter in each
+ * direction, and check that the comparison
+ * functions do not return a wrong definitive answer. Small batch sizes
+ * are the most likely to expose margin errors, because the fan-out of
+ * the tree levels can be larger than the batch size.
+ */
+static void hpcc_test_saturated_compare(struct kunit *test)
+{
+ unsigned long batch_size_order;
+
+ for (batch_size_order = 1; batch_size_order <= 5; batch_size_order++) {
+ do_hpcc_test_saturated_compare(test, 1UL << batch_size_order, 1);
+ do_hpcc_test_saturated_compare(test, 1UL << batch_size_order, -1);
+ }
+}
+
+/* Check a counter against its expected sum: exact before activation, within range after. */
+static void check_lazy_counter(struct kunit *test, struct percpu_counter_tree *c, long v)
+{
+ long approx_sum = percpu_counter_tree_approximate_sum(c);
+ unsigned long accuracy;
+
+ KUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(c), v);
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_precise_compare_value(c, v));
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare_value(c, v));
+ check_residuals(test, c);
+#ifdef CONFIG_SMP
+ if (!READ_ONCE(c->level0)) {
+ KUNIT_EXPECT_EQ(test, approx_sum, v);
+ return;
+ }
+#endif
+ accuracy = percpu_counter_tree_approximate_accuracy(c);
+ KUNIT_EXPECT_GE(test, (long)(approx_sum - (v - accuracy)), 0);
+ KUNIT_EXPECT_LE(test, (long)(approx_sum - (v + accuracy)), 0);
+}
+
+/*
+ * Lazily initialized counters: exact sums before the tree is activated,
+ * with owner and atomic updates, set(), idempotent activation, updates
+ * after activation, and destruction of counters never activated.
+ */
+static void hpcc_test_lazy(struct kunit *test)
+{
+ struct percpu_counter_tree pct[2];
+ long expect[2] = { 0, 0 };
+ int i, j;
+
+ KUNIT_ASSERT_EQ(test, 0, percpu_counter_tree_init_many_lazy(pct, 2, 32));
+#ifdef CONFIG_SMP
+ KUNIT_EXPECT_FALSE(test, percpu_counter_tree_active(&pct[0]));
+ KUNIT_EXPECT_FALSE(test, percpu_counter_tree_active(&pct[1]));
+ /* The level 0 counters are __percpu pointers: compare them as booleans. */
+ KUNIT_EXPECT_TRUE(test, !pct[0].level0);
+ KUNIT_EXPECT_TRUE(test, !pct[1].level0);
+ KUNIT_EXPECT_NULL(test, pct[0].approx_sum);
+ KUNIT_EXPECT_NULL(test, pct[1].approx_sum);
+#endif
+ for (j = 0; j < 2; j++) {
+ /* j == 0: before activation; j == 1: after activation. */
+ for (i = 0; i < 10000; i++) {
+ long inc = (long)(get_random_u32() % 1025) - 512;
+
+ if (get_random_u32() & 1)
+ percpu_counter_tree_add_owner(&pct[0], inc);
+ else
+ percpu_counter_tree_add(&pct[0], inc);
+ expect[0] += inc;
+ percpu_counter_tree_add(&pct[1], -inc);
+ expect[1] -= inc;
+ }
+ check_lazy_counter(test, &pct[0], expect[0]);
+ check_lazy_counter(test, &pct[1], expect[1]);
+ percpu_counter_tree_set(&pct[1], 12345);
+ expect[1] = 12345;
+ check_lazy_counter(test, &pct[1], expect[1]);
+ if (j)
+ break;
+ KUNIT_ASSERT_EQ(test, 0, percpu_counter_tree_activate_many(pct, 2, GFP_KERNEL));
+ KUNIT_EXPECT_TRUE(test, percpu_counter_tree_active(&pct[0]));
+ KUNIT_EXPECT_TRUE(test, percpu_counter_tree_active(&pct[1]));
+#ifdef CONFIG_SMP
+ {
+ s8 __percpu *level0 = pct[0].level0;
+
+ KUNIT_EXPECT_TRUE(test, level0);
+ KUNIT_EXPECT_TRUE(test, pct[1].level0 == level0 + 1);
+ KUNIT_EXPECT_NOT_NULL(test, pct[0].approx_sum);
+ KUNIT_EXPECT_PTR_EQ(test, pct[1].approx_sum, pct[0].approx_sum + 1);
+ KUNIT_EXPECT_EQ(test, 0,
+ percpu_counter_tree_activate_many(pct, 2, GFP_KERNEL));
+ KUNIT_EXPECT_TRUE(test, pct[0].level0 == level0);
+ }
+#endif
+ /* The sums carry the updates made before activation. */
+ check_lazy_counter(test, &pct[0], expect[0]);
+ check_lazy_counter(test, &pct[1], expect[1]);
+ }
+ percpu_counter_tree_destroy_many(pct, 2);
+
+ /* Destroy counters which were never activated. */
+ KUNIT_ASSERT_EQ(test, 0, percpu_counter_tree_init_many_lazy(pct, 2, 32));
+ percpu_counter_tree_add_owner(&pct[0], 7);
+ percpu_counter_tree_add(&pct[1], -7);
+ check_lazy_counter(test, &pct[0], 7);
+ check_lazy_counter(test, &pct[1], -7);
+ percpu_counter_tree_destroy_many(pct, 2);
+}
+
+/*
+ * Activation racing with updates: one owner and updaters on other CPUs add
+ * to a lazily initialized counter. Each worker makes a first part of its
+ * updates, reports their sum and waits. The test checks that the counter
+ * is exact, then releases the workers and activates the tree while they
+ * make the rest of their updates. The remote updaters also activate the
+ * tree when released, serialized with the test and with each other by a
+ * mutex, and check that the tree is active once their activation returns.
+ * No update may be lost, whichever side of the activation it lands on.
+ */
+#define LAZY_RACE_NR_INC 200000
+#define LAZY_RACE_NR_FIRST (LAZY_RACE_NR_INC / 8)
+#define LAZY_RACE_MAX_REMOTE 8
+
+struct lazy_race_shared {
+ struct percpu_counter_tree *c;
+ atomic_long_t first_sum; /* Sum of the updates made before the release. */
+ atomic_long_t expect; /* Sum of all updates. */
+ struct mutex activate_lock; /* Serializes the activations. */
+ atomic_t paused; /* Workers waiting for the release. */
+ atomic_t activate_errors; /* Failed activations of the workers. */
+ bool go; /* Releases the workers. */
+};
+
+struct lazy_race_work {
+ struct work_struct work;
+ struct lazy_race_shared *sh;
+ bool owner;
+};
+
+static void lazy_race_fn(struct work_struct *work)
+{
+ struct lazy_race_work *w = container_of(work, struct lazy_race_work, work);
+ struct lazy_race_shared *sh = w->sh;
+ long sum = 0;
+ int i;
+
+ for (i = 0; i < LAZY_RACE_NR_INC; i++) {
+ long inc = (long)(get_random_u32() % 65) - 32;
+
+ if (i == LAZY_RACE_NR_FIRST) {
+ /* Fully ordered: the first updates are visible before the pause. */
+ atomic_long_add_return(sum, &sh->first_sum);
+ atomic_inc_return(&sh->paused);
+ while (!READ_ONCE(sh->go))
+ cond_resched();
+ if (!w->owner) {
+ mutex_lock(&sh->activate_lock);
+ if (percpu_counter_tree_activate_many(sh->c, 1, GFP_KERNEL) ||
+ !percpu_counter_tree_active(sh->c))
+ atomic_inc(&sh->activate_errors);
+ mutex_unlock(&sh->activate_lock);
+ }
+ }
+ if (w->owner)
+ percpu_counter_tree_add_owner(sh->c, inc);
+ else
+ percpu_counter_tree_add(sh->c, inc);
+ sum += inc;
+ if (!(i & 1023))
+ cond_resched();
+ }
+ atomic_long_add(sum, &sh->expect);
+ kfree(w);
+ complete_work();
+}
+
+static void hpcc_test_lazy_activation_race(struct kunit *test)
+{
+ struct percpu_counter_tree pct;
+ int round;
+
+ if (num_online_cpus() < 2)
+ kunit_skip(test, "needs at least 2 online CPUs");
+ for (round = 0; round < 4; round++) {
+ struct lazy_race_shared sh = { .c = &pct };
+ unsigned long timeout = jiffies + 10 * HZ;
+ int cpu, nr_workers = 0;
+
+ KUNIT_ASSERT_EQ(test, 0, percpu_counter_tree_init_many_lazy(&pct, 1, 32));
+ mutex_init(&sh.activate_lock);
+ init_kthreads();
+ cpus_read_lock();
+ for_each_online_cpu(cpu) {
+ struct lazy_race_work *w;
+
+ if (nr_workers > LAZY_RACE_MAX_REMOTE)
+ break;
+ w = kzalloc_obj(*w);
+ KUNIT_EXPECT_NOT_NULL(test, w);
+ if (!w)
+ break;
+ INIT_WORK(&w->work, lazy_race_fn);
+ w->sh = &sh;
+ /* The first worker is the owner, the others update remotely. */
+ w->owner = !nr_workers;
+ atomic_inc(&kernel_threads_to_run);
+ queue_work_on(cpu, hpcc_test_wq, &w->work);
+ nr_workers++;
+ }
+ cpus_read_unlock();
+ while (atomic_read(&sh.paused) < nr_workers && time_before(jiffies, timeout))
+ usleep_range(100, 200);
+ smp_mb(); /* Read the counter after the first updates of the workers. */
+ /* Every worker, the owner included, has updates left for after the activation. */
+ KUNIT_EXPECT_EQ(test, atomic_read(&sh.paused), nr_workers);
+ /* Before activation, the owner and atomic updates are exact. */
+ check_lazy_counter(test, &pct, atomic_long_read(&sh.first_sum));
+ WRITE_ONCE(sh.go, true);
+ mutex_lock(&sh.activate_lock);
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_activate_many(&pct, 1, GFP_KERNEL));
+ KUNIT_EXPECT_TRUE(test, percpu_counter_tree_active(&pct));
+ mutex_unlock(&sh.activate_lock);
+ fini_kthreads();
+ mutex_destroy(&sh.activate_lock);
+ KUNIT_EXPECT_EQ(test, 0, atomic_read(&sh.activate_errors));
+ check_lazy_counter(test, &pct, atomic_long_read(&sh.expect));
+ percpu_counter_tree_destroy_many(&pct, 1);
+ }
+}
+
+static void hpcc_test_check_config(struct kunit *test)
+{
+#ifdef CONFIG_SMP
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_check_config());
+#else
+ kunit_skip(test, "The configuration table requires CONFIG_SMP");
+#endif
+}
+
+static struct kunit_case hpcc_test_cases[] = {
+ KUNIT_CASE(hpcc_test_check_config),
+ KUNIT_CASE(hpcc_print_info),
+ KUNIT_CASE(hpcc_test_single_thread_first),
+ KUNIT_CASE(hpcc_test_single_thread_first_random),
+ KUNIT_CASE_SLOW(hpcc_test_single_thread_random),
+ KUNIT_CASE(hpcc_test_multi_thread_batch_increment),
+ KUNIT_CASE_SLOW(hpcc_test_multi_thread_random_walk),
+ KUNIT_CASE(hpcc_test_init_one),
+ KUNIT_CASE(hpcc_test_set),
+ KUNIT_CASE(hpcc_test_saturated_compare),
+ KUNIT_CASE(hpcc_test_contended_carries),
+ KUNIT_CASE(hpcc_test_lazy),
+ KUNIT_CASE(hpcc_test_lazy_activation_race),
+ {}
+};
+
+static int hpcc_test_suite_init(struct kunit_suite *suite)
+{
+ hpcc_test_wq = alloc_workqueue("hpcc_test", WQ_PERCPU | WQ_CPU_INTENSIVE, 0);
+ return hpcc_test_wq ? 0 : -ENOMEM;
+}
+
+static void hpcc_test_suite_exit(struct kunit_suite *suite)
+{
+ destroy_workqueue(hpcc_test_wq);
+}
+
+static struct kunit_suite hpcc_test_suite = {
+ .name = "percpu_counter_tree",
+ .suite_init = hpcc_test_suite_init,
+ .suite_exit = hpcc_test_suite_exit,
+ .test_cases = hpcc_test_cases,
+};
+
+kunit_test_suite(hpcc_test_suite);
+
+MODULE_DESCRIPTION("Test cases for hierarchical per-CPU counters");
+MODULE_LICENSE("Dual MIT/GPL");
+MODULE_IMPORT_NS("EXPORTED_FOR_KUNIT_TESTING");
diff --git a/tools/testing/hpcc/.gitignore b/tools/testing/hpcc/.gitignore
new file mode 100644
index 00000000000000..f362a8a46868a5
--- /dev/null
+++ b/tools/testing/hpcc/.gitignore
@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: GPL-2.0+ OR MIT
+hpcc
diff --git a/tools/testing/hpcc/Makefile b/tools/testing/hpcc/Makefile
new file mode 100644
index 00000000000000..e4758643a3f0e5
--- /dev/null
+++ b/tools/testing/hpcc/Makefile
@@ -0,0 +1,18 @@
+# SPDX-License-Identifier: GPL-2.0+ OR MIT
+
+.PHONY: default clean
+
+default: hpcc
+
+CFLAGS += -O2 -g -Wall -I./include -I../../include -fno-strict-overflow -fno-strict-aliasing
+CFLAGS += -DCONFIG_SMP -DCONFIG_KUNIT=1 -D'NR_CPUS=(1U << 20)'
+
+DEPS = ../../../lib/percpu_counter_tree.c \
+ ../../../include/linux/percpu_counter_tree.h \
+ include/hpcc_shim.h
+
+hpcc: main.c $(DEPS)
+ $(CC) $(CFLAGS) -o $@ main.c $(LDLIBS)
+
+clean:
+ $(RM) hpcc
diff --git a/tools/testing/hpcc/include/hpcc_shim.h b/tools/testing/hpcc/include/hpcc_shim.h
new file mode 100644
index 00000000000000..eaac72eab2d172
--- /dev/null
+++ b/tools/testing/hpcc/include/hpcc_shim.h
@@ -0,0 +1,203 @@
+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */
+/*
+ * Minimal userspace environment to build lib/percpu_counter_tree.c.
+ *
+ * The test is single-threaded: per-CPU data is an array indexed by the
+ * simulated current CPU, and atomic operations are plain operations.
+ */
+#ifndef _HPCC_SHIM_H
+#define _HPCC_SHIM_H
+
+#include <stdbool.h>
+#include <stddef.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <errno.h>
+
+/* From tools/include. */
+#include <linux/compiler.h>
+#include <linux/gfp_types.h>
+#include <linux/init.h>
+#include <linux/kconfig.h>
+#include <linux/kernel.h>
+#include <linux/types.h>
+
+#define __ro_after_init
+#define __percpu
+#define SMP_CACHE_BYTES 64
+#define ____cacheline_aligned_in_smp __aligned(SMP_CACHE_BYTES)
+#define S8_MAX INT8_MAX
+
+#undef abs
+#define abs(x) ({ typeof(x) __x = (x); __x < 0 ? -__x : __x; })
+
+#define pr_err(...) fprintf(stderr, __VA_ARGS__)
+#define pr_warn(...) fprintf(stderr, __VA_ARGS__)
+#define pr_info(...) do { if (0) printf(__VA_ARGS__); } while (0)
+
+/* Implemented in main.c. */
+char *get_options(const char *str, int nints, int *ints);
+
+extern unsigned long hpcc_nr_warnings;
+
+#define WARN(cond, ...) ({ \
+ int __ret_warn = !!(cond); \
+ if (__ret_warn) { \
+ hpcc_nr_warnings++; \
+ fprintf(stderr, "WARNING at %s:%d: ", __FILE__, __LINE__); \
+ fprintf(stderr, __VA_ARGS__); \
+ } \
+ __ret_warn; \
+})
+#define WARN_ON(cond) WARN(cond, "%s\n", #cond)
+#define WARN_ON_ONCE(cond) WARN_ON(cond)
+
+static inline int get_count_order(unsigned int count)
+{
+ int order;
+
+ if (!count)
+ return -1;
+ order = 31 - __builtin_clz(count);
+ if (count & (count - 1))
+ order++;
+ return order;
+}
+
+#define atomic_long_read(v) READ_ONCE((v)->counter)
+#define atomic_long_set(v, i) WRITE_ONCE((v)->counter, (i))
+#define atomic_long_add(i, v) ((void)((v)->counter += (i)))
+#define atomic_long_add_return_relaxed(i, v) ((v)->counter += (i))
+#define atomic_long_add_return_release(i, v) ((v)->counter += (i))
+#define smp_rmb() barrier()
+/* The harness runs on one thread: ordering primitives are plain accesses. */
+#define smp_load_acquire(p) READ_ONCE(*(p))
+#define smp_store_release(p, v) WRITE_ONCE(*(p), (v))
+
+static inline bool atomic_long_try_cmpxchg_relaxed(atomic_long_t *v, long *old, long new)
+{
+ if (v->counter == *old) {
+ v->counter = new;
+ return true;
+ }
+ *old = v->counter;
+ return false;
+}
+
+/* Simulated topology, and CPU on which the current code runs. */
+extern unsigned int nr_cpu_ids;
+extern unsigned int hpcc_current_cpu;
+
+#define raw_smp_processor_id() hpcc_current_cpu
+#define for_each_possible_cpu(cpu) for ((cpu) = 0; (cpu) < (int)nr_cpu_ids; (cpu)++)
+
+/*
+ * Each per-CPU allocation reserves HPCC_PCPU_UNIT bytes per CPU, so
+ * per_cpu_ptr() can be applied to a pointer within an allocation.
+ */
+#define HPCC_PCPU_UNIT 64
+
+/* Outstanding allocations, to check for leaks. */
+extern long hpcc_nr_allocs;
+
+static inline void *hpcc_alloc_account(void *p)
+{
+ if (p)
+ hpcc_nr_allocs++;
+ return p;
+}
+
+static inline void hpcc_free_account(void *p)
+{
+ if (p)
+ hpcc_nr_allocs--;
+}
+
+static inline void *__alloc_percpu_gfp(size_t size, size_t align, gfp_t gfp)
+{
+ if (size > HPCC_PCPU_UNIT || align > HPCC_PCPU_UNIT)
+ return NULL;
+ return hpcc_alloc_account(calloc(nr_cpu_ids, HPCC_PCPU_UNIT));
+}
+
+static inline void free_percpu(void *ptr)
+{
+ hpcc_free_account(ptr);
+ free(ptr);
+}
+
+#define SLAB_PANIC 0
+
+struct kmem_cache {
+ size_t size;
+ size_t align;
+};
+
+static inline struct kmem_cache *kmem_cache_create(const char *name, unsigned int size,
+ unsigned int align, unsigned int flags,
+ void (*ctor)(void *))
+{
+ struct kmem_cache *s = malloc(sizeof(*s));
+
+ if (!s)
+ abort();
+ s->size = size;
+ s->align = align;
+ return s;
+}
+
+static inline void kmem_cache_destroy(struct kmem_cache *s)
+{
+ free(s);
+}
+
+static inline void *kmem_cache_zalloc(struct kmem_cache *s, gfp_t gfp)
+{
+ /* The allocation size needs to be a multiple of the alignment. */
+ void *p = aligned_alloc(s->align, (s->size + s->align - 1) / s->align * s->align);
+
+ if (p)
+ memset(p, 0, s->size);
+ return hpcc_alloc_account(p);
+}
+
+static inline void kmem_cache_free(struct kmem_cache *s, void *p)
+{
+ hpcc_free_account(p);
+ free(p);
+}
+
+static inline void *kzalloc(size_t size, gfp_t gfp)
+{
+ return hpcc_alloc_account(calloc(1, size));
+}
+
+static inline void kfree(void *p)
+{
+ hpcc_free_account(p);
+ free(p);
+}
+
+#define per_cpu_ptr(ptr, cpu) \
+ ((typeof(ptr))((char *)(ptr) + (size_t)(cpu) * HPCC_PCPU_UNIT))
+
+/* Level 0 counters are the only per-CPU data: s8. */
+static inline bool __hpcc_this_cpu_try_cmpxchg(s8 *pcp, s8 *old, s8 new)
+{
+ s8 *p = per_cpu_ptr(pcp, hpcc_current_cpu);
+
+ if (*p != *old) {
+ *old = *p;
+ return false;
+ }
+ *p = new;
+ return true;
+}
+
+#define this_cpu_read(pcp) (*per_cpu_ptr(&(pcp), hpcc_current_cpu))
+#define this_cpu_try_cmpxchg(pcp, ovalp, nval) \
+ __hpcc_this_cpu_try_cmpxchg(&(pcp), (ovalp), (nval))
+
+#endif /* _HPCC_SHIM_H */
diff --git a/tools/testing/hpcc/include/kunit/visibility.h b/tools/testing/hpcc/include/kunit/visibility.h
new file mode 100644
index 00000000000000..dfcbfd01a4151f
--- /dev/null
+++ b/tools/testing/hpcc/include/kunit/visibility.h
@@ -0,0 +1,2 @@
+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */
+#define EXPORT_SYMBOL_IF_KUNIT(symbol)
diff --git a/tools/testing/hpcc/include/linux/args.h b/tools/testing/hpcc/include/linux/args.h
new file mode 100644
index 00000000000000..0c00a85fdd213d
--- /dev/null
+++ b/tools/testing/hpcc/include/linux/args.h
@@ -0,0 +1,4 @@
+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */
+/* Also defined by the C library <sys/cdefs.h>. */
+#undef __CONCAT
+#include "../../../../../include/linux/args.h"
diff --git a/tools/testing/hpcc/include/linux/atomic.h b/tools/testing/hpcc/include/linux/atomic.h
new file mode 100644
index 00000000000000..b2aa26415cca18
--- /dev/null
+++ b/tools/testing/hpcc/include/linux/atomic.h
@@ -0,0 +1,2 @@
+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */
+#include "../hpcc_shim.h"
diff --git a/tools/testing/hpcc/include/linux/bug.h b/tools/testing/hpcc/include/linux/bug.h
new file mode 100644
index 00000000000000..b2aa26415cca18
--- /dev/null
+++ b/tools/testing/hpcc/include/linux/bug.h
@@ -0,0 +1,2 @@
+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */
+#include "../hpcc_shim.h"
diff --git a/tools/testing/hpcc/include/linux/cache.h b/tools/testing/hpcc/include/linux/cache.h
new file mode 100644
index 00000000000000..b2aa26415cca18
--- /dev/null
+++ b/tools/testing/hpcc/include/linux/cache.h
@@ -0,0 +1,2 @@
+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */
+#include "../hpcc_shim.h"
diff --git a/tools/testing/hpcc/include/linux/cpumask.h b/tools/testing/hpcc/include/linux/cpumask.h
new file mode 100644
index 00000000000000..b2aa26415cca18
--- /dev/null
+++ b/tools/testing/hpcc/include/linux/cpumask.h
@@ -0,0 +1,2 @@
+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */
+#include "../hpcc_shim.h"
diff --git a/tools/testing/hpcc/include/linux/errno.h b/tools/testing/hpcc/include/linux/errno.h
new file mode 100644
index 00000000000000..45fdd60261e717
--- /dev/null
+++ b/tools/testing/hpcc/include/linux/errno.h
@@ -0,0 +1,3 @@
+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */
+/* Also included by the C library <errno.h>: only provide the error numbers. */
+#include <asm/errno.h>
diff --git a/tools/testing/hpcc/include/linux/limits.h b/tools/testing/hpcc/include/linux/limits.h
new file mode 100644
index 00000000000000..b2aa26415cca18
--- /dev/null
+++ b/tools/testing/hpcc/include/linux/limits.h
@@ -0,0 +1,2 @@
+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */
+#include "../hpcc_shim.h"
diff --git a/tools/testing/hpcc/include/linux/math.h b/tools/testing/hpcc/include/linux/math.h
new file mode 100644
index 00000000000000..b2aa26415cca18
--- /dev/null
+++ b/tools/testing/hpcc/include/linux/math.h
@@ -0,0 +1,2 @@
+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */
+#include "../hpcc_shim.h"
diff --git a/tools/testing/hpcc/include/linux/moduleparam.h b/tools/testing/hpcc/include/linux/moduleparam.h
new file mode 100644
index 00000000000000..7e40df4295b2f5
--- /dev/null
+++ b/tools/testing/hpcc/include/linux/moduleparam.h
@@ -0,0 +1,12 @@
+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */
+#include "../hpcc_shim.h"
+
+struct kernel_param;
+
+struct kernel_param_ops {
+ int (*set)(const char *val, const struct kernel_param *kp);
+ int (*get)(char *buffer, const struct kernel_param *kp);
+};
+
+#define module_param_cb(name, ops, arg, perm)
+#define MODULE_PARM_DESC(parm, desc)
diff --git a/tools/testing/hpcc/include/linux/percpu.h b/tools/testing/hpcc/include/linux/percpu.h
new file mode 100644
index 00000000000000..b2aa26415cca18
--- /dev/null
+++ b/tools/testing/hpcc/include/linux/percpu.h
@@ -0,0 +1,2 @@
+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */
+#include "../hpcc_shim.h"
diff --git a/tools/testing/hpcc/include/linux/percpu_counter_tree.h b/tools/testing/hpcc/include/linux/percpu_counter_tree.h
new file mode 100644
index 00000000000000..065a2aca4d5b8b
--- /dev/null
+++ b/tools/testing/hpcc/include/linux/percpu_counter_tree.h
@@ -0,0 +1,2 @@
+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */
+#include "../../../../../include/linux/percpu_counter_tree.h"
diff --git a/tools/testing/hpcc/include/linux/preempt.h b/tools/testing/hpcc/include/linux/preempt.h
new file mode 100644
index 00000000000000..b2aa26415cca18
--- /dev/null
+++ b/tools/testing/hpcc/include/linux/preempt.h
@@ -0,0 +1,2 @@
+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */
+#include "../hpcc_shim.h"
diff --git a/tools/testing/hpcc/include/linux/slab.h b/tools/testing/hpcc/include/linux/slab.h
new file mode 100644
index 00000000000000..b2aa26415cca18
--- /dev/null
+++ b/tools/testing/hpcc/include/linux/slab.h
@@ -0,0 +1,2 @@
+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */
+#include "../hpcc_shim.h"
diff --git a/tools/testing/hpcc/main.c b/tools/testing/hpcc/main.c
new file mode 100644
index 00000000000000..be17b887312631
--- /dev/null
+++ b/tools/testing/hpcc/main.c
@@ -0,0 +1,536 @@
+// SPDX-License-Identifier: GPL-2.0+ OR MIT
+/*
+ * Userspace test of the hierarchical per-CPU counters (lib/percpu_counter_tree.c).
+ *
+ * The library is built with a simulated topology, which allows testing
+ * every supported tree configuration on a single machine, whereas the
+ * KUnit test only covers the topology of the machine it runs on.
+ *
+ * For each simulated number of CPUs and batch size:
+ *
+ * - Saturate the counter upward and downward: leave every tree item with
+ * its largest positive (or negative) residual which has not been
+ * propagated yet, check that the distance between the precise and
+ * approximate sums matches the residual capacity of the tree in that
+ * direction, and check the accuracy range contract and the comparison
+ * functions in these worst-case states.
+ *
+ * - Random walk: apply random increments from random CPUs to two counters,
+ * and check the accuracy range contract and the comparison functions
+ * against shadow sums after each increment.
+ *
+ * - Lazy counter: random owner and atomic updates of a lazily initialized
+ * counter, exact before its tree is activated halfway, with the same
+ * checks after each increment.
+ *
+ * The test is single-threaded, so it only covers settled counters.
+ */
+
+#include <stdarg.h>
+#include "hpcc_shim.h"
+
+unsigned int nr_cpu_ids;
+unsigned int hpcc_current_cpu;
+unsigned long hpcc_nr_warnings;
+long hpcc_nr_allocs;
+
+#include "../../../lib/percpu_counter_tree.c"
+
+#define MAX_REPORTED_FAILURES 10
+
+static unsigned long nr_case_failures;
+
+#define CHECK(cond) do { \
+ if (!(cond)) { \
+ if (nr_case_failures++ < MAX_REPORTED_FAILURES) \
+ printf("# %s:%d: check failed: %s\n", \
+ __FILE__, __LINE__, #cond); \
+ } \
+} while (0)
+
+static uint64_t rng_state = 88172645463325252ULL;
+
+static uint64_t rnd(void)
+{
+ rng_state ^= rng_state << 13;
+ rng_state ^= rng_state >> 7;
+ rng_state ^= rng_state << 17;
+ return rng_state;
+}
+
+static int sign(long v)
+{
+ return (v > 0) - (v < 0);
+}
+
+struct test_counter {
+ struct percpu_counter_tree tree;
+ long shadow; /* Expected precise sum. */
+};
+
+int scnprintf(char *buf, size_t size, const char *fmt, ...)
+{
+ va_list args;
+ int len;
+
+ if (!size)
+ return 0;
+ va_start(args, fmt);
+ len = vsnprintf(buf, size, fmt, args);
+ va_end(args);
+ return len < (int)size ? len : (int)size - 1;
+}
+
+/* Comma-separated integers, without the ranges of lib/cmdline.c. */
+char *get_options(const char *str, int nints, int *ints)
+{
+ char *end;
+ int i = 0;
+
+ while (i < nints - 1 && *str) {
+ long v = strtol(str, &end, 0);
+
+ if (end == str)
+ break;
+ ints[++i] = v;
+ str = end;
+ if (*str != ',')
+ break;
+ str++;
+ }
+ ints[0] = i;
+ return (char *)str;
+}
+
+static void hpcc_init_topology(unsigned int nr_cpus)
+{
+ nr_cpu_ids = nr_cpus;
+ /* Reset the state accumulated by a prior subsystem initialization. */
+ kmem_cache_destroy(tree_topology.items_cachep);
+ memset(&tree_topology, 0, sizeof(tree_topology));
+ tree_topology.nr_items = UINT_MAX;
+ CHECK(!percpu_counter_tree_subsystem_init());
+}
+
+static int test_counter_init(struct test_counter *c, unsigned long batch_size)
+{
+ c->shadow = 0;
+ return percpu_counter_tree_init(&c->tree, batch_size, GFP_KERNEL);
+}
+
+static void test_counter_destroy(struct test_counter *c)
+{
+ percpu_counter_tree_destroy(&c->tree);
+}
+
+static void test_counter_add(struct test_counter *c, unsigned int cpu, long inc)
+{
+ hpcc_current_cpu = cpu;
+ percpu_counter_tree_add(&c->tree, inc);
+ c->shadow += inc;
+}
+
+static void test_counter_set(struct test_counter *c, long v)
+{
+ percpu_counter_tree_set(&c->tree, v);
+ c->shadow = v;
+}
+
+/*
+ * Residual capacity of the tree items above level 0, in either direction:
+ * sum of the largest residual which each item can hold without propagating
+ * a carry. A level L item only receives multiples of the level L - 1 carry,
+ * and propagates once its residual reaches the level L carry, so it holds
+ * at most bit(L) - bit(L - 1) in either direction.
+ */
+static unsigned long tree_item_capacity(unsigned long batch_size)
+{
+ unsigned long capacity = 0, lo = batch_size, hi;
+ unsigned int level, level_items;
+
+ level_items = 1U << tree_topology.nr_cpus_order;
+ for (level = 1; level < tree_topology.config->nr_levels; level++) {
+ unsigned int n_arity_order = tree_topology.config->n_arity_order[level - 1];
+
+ level_items >>= n_arity_order;
+ hi = lo << n_arity_order;
+ capacity += level_items * (hi - lo);
+ lo = hi;
+ }
+ return capacity;
+}
+
+/*
+ * Residual capacity of the tree, in either direction: a level 0 counter
+ * holds at most batch_size - 1 on top of the item capacity.
+ */
+static unsigned long tree_residual_capacity(unsigned long batch_size)
+{
+ return nr_cpu_ids * (batch_size - 1) + tree_item_capacity(batch_size);
+}
+
+/* Copy the state of counter @src into counter @dst, of the same topology. */
+static void test_counter_copy(struct test_counter *dst, struct test_counter *src)
+{
+ percpu_counter_tree_copy_state(&dst->tree, &src->tree);
+ dst->shadow = src->shadow;
+}
+
+/*
+ * Check the accuracy range contract, and the comparison of the counter
+ * against values around its precise sum and its possible precise range.
+ */
+static void check_counter(struct test_counter *c)
+{
+ struct percpu_counter_tree *t = &c->tree;
+ long approx = percpu_counter_tree_approximate_sum(t);
+ long precise = c->shadow, precise_min, precise_max;
+ unsigned long accuracy;
+ long values[7];
+ unsigned int i;
+
+ CHECK(percpu_counter_tree_precise_sum(t) == precise);
+ CHECK(!percpu_counter_tree_check_residuals(t));
+
+ /* precise - accuracy <= approx <= precise + accuracy */
+ accuracy = percpu_counter_tree_approximate_accuracy(t);
+ CHECK(approx >= precise - (long)accuracy);
+ CHECK(approx <= precise + (long)accuracy);
+
+ percpu_counter_tree_approximate_min_max_range(approx, accuracy,
+ &precise_min, &precise_max);
+ CHECK(precise >= precise_min);
+ CHECK(precise <= precise_max);
+
+ /* Approximate comparison is definitive exactly outside of the precise range. */
+ CHECK(percpu_counter_tree_approximate_compare_value(t, precise_min - 1) == -1);
+ CHECK(percpu_counter_tree_approximate_compare_value(t, precise_min) == 0);
+ CHECK(percpu_counter_tree_approximate_compare_value(t, precise_max) == 0);
+ CHECK(percpu_counter_tree_approximate_compare_value(t, precise_max + 1) == 1);
+
+ values[0] = precise - 1;
+ values[1] = precise;
+ values[2] = precise + 1;
+ values[3] = precise_min - 1;
+ values[4] = precise_min;
+ values[5] = precise_max;
+ values[6] = precise_max + 1;
+ for (i = 0; i < ARRAY_SIZE(values); i++) {
+ long v = values[i];
+ int expect = sign(v - precise);
+ int res = percpu_counter_tree_approximate_compare_value(t, v);
+
+ CHECK(res == 0 || res == expect);
+ CHECK(percpu_counter_tree_precise_compare_value(t, v) == expect);
+ }
+}
+
+/* Check the comparison of two counters against their shadow sums. */
+static void check_counter_pair(struct test_counter *a, struct test_counter *b)
+{
+ int expect = sign(a->shadow - b->shadow);
+ int res = percpu_counter_tree_approximate_compare(&a->tree, &b->tree);
+
+ CHECK(res == 0 || res == expect);
+ CHECK(percpu_counter_tree_precise_compare(&a->tree, &b->tree) == expect);
+ res = percpu_counter_tree_approximate_compare(&b->tree, &a->tree);
+ CHECK(res == 0 || res == -expect);
+ CHECK(percpu_counter_tree_precise_compare(&b->tree, &a->tree) == -expect);
+}
+
+/*
+ * On each CPU, add @inc (1 or -1) until the next addition would propagate a
+ * carry to the approximate sum. This leaves every tree item on the path from
+ * this CPU to the root holding its largest residual in the direction of @inc.
+ * Each addition is first tried on @probe, which holds the same state as @c:
+ * reverting an addition which reached the approximate sum would not restore
+ * the residuals subtracted on the way. The path from a CPU to the root
+ * holds less than the accuracy bound: more additions mean that a carry was
+ * lost.
+ */
+static void saturate(struct test_counter *c, struct test_counter *probe, long inc)
+{
+ unsigned long max_steps;
+ unsigned int cpu;
+
+ max_steps = percpu_counter_tree_approximate_accuracy(&c->tree) +
+ (1UL << c->tree.batch_order);
+ for (cpu = 0; cpu < nr_cpu_ids; cpu++) {
+ unsigned long steps;
+
+ for (steps = 0; steps < max_steps; steps++) {
+ long approx = percpu_counter_tree_approximate_sum(&probe->tree);
+
+ test_counter_add(probe, cpu, inc);
+ if (percpu_counter_tree_approximate_sum(&probe->tree) != approx) {
+ test_counter_copy(probe, c);
+ break;
+ }
+ test_counter_add(c, cpu, inc);
+ }
+ if (steps == max_steps) {
+ CHECK(0);
+ return;
+ }
+ }
+}
+
+static void test_saturated(unsigned long batch_size, long inc)
+{
+ struct test_counter c, probe, exact;
+ unsigned long accuracy;
+ long approx, delta;
+
+ if (test_counter_init(&c, batch_size) || test_counter_init(&probe, batch_size) ||
+ test_counter_init(&exact, batch_size)) {
+ CHECK(0);
+ return;
+ }
+ saturate(&c, &probe, inc);
+ approx = percpu_counter_tree_approximate_sum(&c.tree);
+ accuracy = percpu_counter_tree_approximate_accuracy(&c.tree);
+ /* Symmetric residuals: the accuracy is reached in both directions. */
+ CHECK((c.shadow - approx) * inc == (long)tree_residual_capacity(batch_size));
+ CHECK((c.shadow - approx) * inc == (long)accuracy);
+ check_counter(&c);
+
+ /* Compare with exactly approximated counters around the saturated precise sum. */
+ for (delta = -1; delta <= 1; delta++) {
+ test_counter_set(&exact, c.shadow + delta);
+ CHECK(percpu_counter_tree_approximate_sum(&exact.tree) == exact.shadow);
+ check_counter_pair(&c, &exact);
+ }
+ test_counter_destroy(&c);
+ test_counter_destroy(&probe);
+ test_counter_destroy(&exact);
+}
+
+static long random_increment(unsigned long batch_size)
+{
+ long v;
+
+ switch (rnd() % 4) {
+ case 0:
+ v = rnd() % 4;
+ break;
+ case 1:
+ v = rnd() % (4 * batch_size);
+ break;
+ case 2:
+ v = rnd() % (64 * batch_size);
+ break;
+ default:
+ v = rnd() % 1000000;
+ break;
+ }
+ return rnd() & 1 ? v : -v;
+}
+
+static void test_random_walk(unsigned long batch_size, unsigned long nr_steps)
+{
+ struct test_counter c[2];
+ unsigned long step;
+
+ if (test_counter_init(&c[0], batch_size) || test_counter_init(&c[1], batch_size)) {
+ CHECK(0);
+ return;
+ }
+ for (step = 0; step < nr_steps; step++) {
+ struct test_counter *t = &c[rnd() & 1];
+
+ test_counter_add(t, rnd() % nr_cpu_ids, random_increment(batch_size));
+ /* Bring the counters close to each other to exercise the precise comparison. */
+ if (!(rnd() % 8)) {
+ long delta = c[0].shadow - c[1].shadow;
+ unsigned long accuracy;
+
+ accuracy = percpu_counter_tree_approximate_accuracy(&c[0].tree);
+ delta += (long)(rnd() % (2 * accuracy + 1)) - (long)accuracy;
+ test_counter_add(&c[1], rnd() % nr_cpu_ids, delta);
+ }
+ check_counter(t);
+ check_counter_pair(&c[0], &c[1]);
+ }
+ test_counter_destroy(&c[0]);
+ test_counter_destroy(&c[1]);
+}
+
+static void test_lazy(unsigned long batch_size, unsigned long nr_steps)
+{
+ long nr_allocs = hpcc_nr_allocs;
+ struct test_counter c;
+ unsigned long step;
+
+ if (percpu_counter_tree_init_many_lazy(&c.tree, 1, batch_size)) {
+ CHECK(0);
+ return;
+ }
+ c.shadow = 0;
+ CHECK(!c.tree.level0);
+ CHECK(!c.tree.approx_sum);
+ CHECK(!percpu_counter_tree_active(&c.tree));
+ for (step = 0; step < nr_steps; step++) {
+ long inc = random_increment(batch_size);
+
+ hpcc_current_cpu = rnd() % nr_cpu_ids;
+ if (rnd() & 1)
+ percpu_counter_tree_add_owner(&c.tree, inc);
+ else
+ percpu_counter_tree_add(&c.tree, inc);
+ c.shadow += inc;
+ if (!c.tree.level0)
+ CHECK(percpu_counter_tree_approximate_sum(&c.tree) == c.shadow);
+ check_counter(&c);
+ if (step == nr_steps / 2) {
+ CHECK(!percpu_counter_tree_activate_many(&c.tree, 1, GFP_KERNEL));
+ CHECK(c.tree.level0);
+ CHECK(c.tree.approx_sum);
+ CHECK(percpu_counter_tree_active(&c.tree));
+ /* One tree: its per-CPU counters and its tree items. */
+ CHECK(hpcc_nr_allocs == nr_allocs + 2);
+ CHECK(!percpu_counter_tree_activate_many(&c.tree, 1, GFP_KERNEL));
+ CHECK(hpcc_nr_allocs == nr_allocs + 2);
+ }
+ }
+ test_counter_destroy(&c);
+ CHECK(hpcc_nr_allocs == nr_allocs);
+}
+
+static const unsigned int test_nr_cpus[] = {
+ 1, 2, 3, 4, 5, 8, 9, 16, 17, 32, 33, 64, 65, 100, 128, 256, 512,
+ 1000, 1024, 2048, 4096, 8192, 16384, 32768, 65536,
+};
+
+/* Exhaustive mode: also cover the largest supported configurations. */
+static const unsigned int test_nr_cpus_large[] = {
+ 1U << 17, 1U << 18, 1U << 19, 1U << 20,
+};
+
+static const unsigned long test_batch_sizes[] = { 2, 4, 8, 16, 32, 64, 128 };
+
+static int test_num;
+
+static int run_case(unsigned int nr_cpus, unsigned long batch_size)
+{
+ /* Each check iterates over all CPUs: bound the random walk cost. */
+ unsigned long nr_steps = 10000000 / nr_cpus;
+
+ if (nr_steps < 100)
+ nr_steps = 100;
+ if (nr_steps > 50000)
+ nr_steps = 50000;
+
+ nr_case_failures = 0;
+ hpcc_nr_warnings = 0;
+ hpcc_init_topology(nr_cpus);
+ test_saturated(batch_size, 1);
+ test_saturated(batch_size, -1);
+ test_random_walk(batch_size, nr_steps);
+ test_lazy(batch_size, nr_steps / 2);
+ CHECK(!hpcc_nr_warnings);
+ printf("%sok %d - nr_cpus=%u batch_size=%lu\n", nr_case_failures ? "not " : "",
+ ++test_num, nr_cpus, batch_size);
+ if (nr_case_failures > MAX_REPORTED_FAILURES)
+ printf("# ... %lu failed checks in total\n", nr_case_failures);
+ return nr_case_failures ? 1 : 0;
+}
+
+/* Tree shapes given with the arity boot parameter. */
+static const struct {
+ unsigned int nr_cpus;
+ const char *arity;
+ const char *shape; /* Shape in use, NULL if the arity is invalid. */
+} test_arity[] = {
+ { 16, "16", "16" },
+ { 16, "4,4", "4,4" },
+ { 16, "2,2,2,2", "2,2,2,2" },
+ { 16, "4,2", NULL }, /* Covers 8 CPUs. */
+ { 16, "4,8", NULL }, /* Covers 32 CPUs. */
+ { 16, "4,4,1", NULL }, /* Fan-out of 1. */
+ { 16, "4,3", NULL }, /* Not a power of 2. */
+ { 16, "4,4x", NULL }, /* Trailing characters. */
+ { 16, "-4,-4", NULL },
+ { 17, "8,4", "8,4" },
+ { 100, "128", "128" },
+ { 192, "8,4,8", "8,4,8" },
+ { 192, "2,2,2,2,2,2,2,2", NULL }, /* More than MAX_NR_LEVELS. */
+ { 1024, "32,32", "32,32" },
+ { 1, "2", NULL },
+};
+
+static int run_arity_case(unsigned int i)
+{
+ const char *expect = test_arity[i].shape;
+ char buf[SHAPE_BUF_SIZE];
+ int len;
+
+ nr_case_failures = 0;
+ hpcc_nr_warnings = 0;
+ tree_topology.config = NULL;
+ CHECK(!arity_param_set(test_arity[i].arity, NULL));
+ hpcc_init_topology(test_arity[i].nr_cpus);
+ arity_param = NULL;
+ CHECK(arity_param_set("16", NULL) == -EBUSY);
+ CHECK((tree_topology.config == &tree_topology.boot_config) == !!expect);
+ len = arity_param_get(buf, NULL);
+ CHECK(len > 0 && buf[len - 1] == '\n');
+ buf[len - 1] = '\0';
+ if (expect) {
+ CHECK(!strcmp(buf, expect));
+ test_saturated(2, 1);
+ test_saturated(32, -1);
+ test_random_walk(32, 20000);
+ }
+ CHECK(!hpcc_nr_warnings);
+ printf("%sok %d - arity nr_cpus=%u \"%s\" -> \"%s\"\n", nr_case_failures ? "not " : "",
+ ++test_num, test_arity[i].nr_cpus, test_arity[i].arity, buf);
+ return nr_case_failures ? 1 : 0;
+}
+
+static int run_check_config(void)
+{
+ unsigned int nr_errors = percpu_counter_tree_check_config();
+
+ printf("%sok %d - configuration table\n", nr_errors ? "not " : "", ++test_num);
+ return nr_errors ? 1 : 0;
+}
+
+static void usage(const char *name)
+{
+ fprintf(stderr, "Usage: %s [-a]\n", name);
+ fprintf(stderr, " -a Also test the largest supported numbers of CPUs (slow).\n");
+}
+
+int main(int argc, char **argv)
+{
+ unsigned int i, j, nr_tests;
+ bool large = false;
+ int failed = 0;
+
+ if (argc > 2 || (argc == 2 && strcmp(argv[1], "-a"))) {
+ usage(argv[0]);
+ return 2;
+ }
+ if (argc == 2)
+ large = true;
+
+ nr_tests = 1 + ARRAY_SIZE(test_arity) +
+ ARRAY_SIZE(test_nr_cpus) * ARRAY_SIZE(test_batch_sizes);
+ if (large)
+ nr_tests += ARRAY_SIZE(test_nr_cpus_large) * ARRAY_SIZE(test_batch_sizes);
+ printf("1..%u\n", nr_tests);
+
+ failed += run_check_config();
+ for (i = 0; i < ARRAY_SIZE(test_arity); i++)
+ failed += run_arity_case(i);
+ for (i = 0; i < ARRAY_SIZE(test_nr_cpus); i++)
+ for (j = 0; j < ARRAY_SIZE(test_batch_sizes); j++)
+ failed += run_case(test_nr_cpus[i], test_batch_sizes[j]);
+ if (large) {
+ for (i = 0; i < ARRAY_SIZE(test_nr_cpus_large); i++)
+ for (j = 0; j < ARRAY_SIZE(test_batch_sizes); j++)
+ failed += run_case(test_nr_cpus_large[i], test_batch_sizes[j]);
+ }
+ printf("# %d/%u test cases failed\n", failed, nr_tests);
+ return failed ? 1 : 0;
+}
|
| 3/2 | 2026/11/03 12:05 | llm | patch-evaluator |
1mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 66567 output: 20 thoughts: 1647 Thoughts: **Analyzing the Hierarchical Per-CPU Counter Patch Series** Okay, so the task at hand is to evaluate this patch series applied as HEAD in the kernel workspace. My initial thought is to dive straight into what this series actually *does*. I need to get a clear picture before I can even begin to think about the fuzzing implications. From the diff, I can see that this is a significant undertaking. The series is introducing "Hierarchical Per-CPU Counters" (HPCC), which sounds like a fundamental change to how the kernel tracks per-CPU statistics. My initial impression is that this touches very core kernel subsystems. The files being modified really underscore this; documentation, MAINTAINERS, and core memory management code are all present. A quick review of the changes reveals a lot of focus on `mm->rss_stat`, which appears to be the resident set size (RSS) counters for memory management. The changes in `include/linux/mm_types.h` confirm this - `struct percpu_counter rss_stat[NR_MM_COUNTERS];` is being replaced by `struct percpu_counter_tree rss_stat[NR_MM_COUNTERS];`. This looks like a complete overhaul of how we track RSS, moving from a standard per-CPU counter to a hierarchical tree structure. This also introduces a few new members to the `mm_struct`, including a mutex to protect the new data structure. The diff also points to changes in `fs/exec.c`, `kernel/exit.c`, and `kernel/fork.c`, which tells me this impacts fundamental process lifecycle operations. There are also changes in the counter APIs: `get_mm_counter()`, `get_mm_counter_sum()`, and the introduction of new functions using the `percpu_counter_tree_*` prefix. I can see the new `__add_mm_counter()` function for updating these counters. Looking at the files affected, I'm already getting a good sense of how deep this goes. We are talking about critical path code that is executed on every page allocation, page fault, mmap, unmap, reclaim, exit, fork, exec, etc. This is literally at the heart of the memory management subsystem. This is a big deal! The series introduces lazy activation, lockless updates, batching, and carry propagation using atomic `cmpxchg`, and so on. The main question is, is this code reachable? The answer is a resounding YES! Any userspace process performing standard operations like mmap, page faults, fork, clone, exec, or exit will directly execute this code. This means any syzkaller fuzzer running standard syscalls will exercise the new functionality. This patch series looks like a prime candidate for fuzzing. My next step is to nail down the necessary configuration. Based on `lib/Makefile`, the `CONFIG_SMP` option is required. If `CONFIG_SMP` is not enabled, the code for the hierarchical per-CPU counters isn't even compiled. On UP systems, the `percpu_counter_tree` structure would simply be an `atomic_long_t`. This implies that `CONFIG_SMP` is the most critical configuration for testing the new code path. I need to grep for `percpu_counter_tree` and check `Kconfig` for other potential dependencies. So, in conclusion, `WorthFuzzing = true`. |
| 4/2 | 2026/11/03 12:06 | tool | grepper |
0mArgs:{
"Expression": "percpu_counter_tree"
}Results: {
"Output": "\nFull output is too long, showing 200 out of 1481 lines.\nUse more precise expression or PathPrefixes if possible.\n\nNumber of matching lines per file (18 files in total):\nDocumentation/admin-guide/kernel-parameters.txt:2\nDocumentation/core-api/percpu-counter-tree.rst:10\nMAINTAINERS:4\ninclude/linux/mm.h:6\ninclude/linux/mm_types.h:3\ninclude/linux/percpu_counter_tree.h:81\ninit/main.c:2\nkernel/fork.c:5\nlib/Kconfig:2\nlib/Makefile:1\nlib/percpu_counter_tree.c:152\nlib/tests/Makefile:2\nlib/tests/percpu_counter_tree_bench.c:45\nlib/tests/percpu_counter_tree_kunit.c:99\ntools/testing/hpcc/Makefile:2\ntools/testing/hpcc/include/hpcc_shim.h:1\ntools/testing/hpcc/include/linux/percpu_counter_tree.h:1\ntools/testing/hpcc/main.c:41\n\nDocumentation/admin-guide/kernel-parameters.txt=98=Kernel parameters\n--\nDocumentation/admin-guide/kernel-parameters.txt-5344-\nDocumentation/admin-guide/kernel-parameters.txt:5345:\tpercpu_counter_tree.arity=\t[KNL,SMP]\nDocumentation/admin-guide/kernel-parameters.txt-5346-\t\t\tFormat: \u003cfan-out\u003e[,\u003cfan-out\u003e...]\n--\nDocumentation/admin-guide/kernel-parameters.txt-5355-\t\t\tThe shape in use is shown in\nDocumentation/admin-guide/kernel-parameters.txt:5356:\t\t\t/sys/module/percpu_counter_tree/parameters/arity.\nDocumentation/admin-guide/kernel-parameters.txt-5357-\t\t\tThis parameter is meant for performance comparison.\n--\nDocumentation/core-api/percpu-counter-tree.rst=58=items of all the counter trees initialized together by\nDocumentation/core-api/percpu-counter-tree.rst:59:percpu_counter_tree_init_many(), so a set of counters updated together,\nDocumentation/core-api/percpu-counter-tree.rst-60-such as the RSS counters of an mm, shares cache lines.\nDocumentation/core-api/percpu-counter-tree.rst:61:percpu_counter_tree_init_many() allocates the per-cpu counters, one per\nDocumentation/core-api/percpu-counter-tree.rst-62-possible CPU, and the tree items, whose number depends on nr_cpu_ids, so\nDocumentation/core-api/percpu-counter-tree.rst:63:struct percpu_counter_tree has a fixed size.\nDocumentation/core-api/percpu-counter-tree.rst-64-\n--\nDocumentation/core-api/percpu-counter-tree.rst=66=highest possible CPU number plus one. It can be overridden with the\nDocumentation/core-api/percpu-counter-tree.rst:67:percpu_counter_tree.arity= boot parameter for performance comparison.\nDocumentation/core-api/percpu-counter-tree.rst-68-\n--\nDocumentation/core-api/percpu-counter-tree.rst=78=Lazy Tree Activation\n--\nDocumentation/core-api/percpu-counter-tree.rst-80-\nDocumentation/core-api/percpu-counter-tree.rst:81:Counters initialized with percpu_counter_tree_init_many_lazy() start\nDocumentation/core-api/percpu-counter-tree.rst-82-without their per-cpu counters and tree items, for counters which may\nDocumentation/core-api/percpu-counter-tree.rst=83=never need the scalability of the tree. Until\nDocumentation/core-api/percpu-counter-tree.rst:84:percpu_counter_tree_activate_many() allocates and publishes the tree,\nDocumentation/core-api/percpu-counter-tree.rst-85-the counters are exact:\nDocumentation/core-api/percpu-counter-tree.rst-86-\nDocumentation/core-api/percpu-counter-tree.rst:87: * percpu_counter_tree_add_owner() adds to the bias of the counter,\nDocumentation/core-api/percpu-counter-tree.rst-88- written only by its owner, a single context chosen by the user of the\n--\nDocumentation/core-api/percpu-counter-tree.rst-90-\nDocumentation/core-api/percpu-counter-tree.rst:91: * percpu_counter_tree_add() adds to the atomic bias of the counter, for\nDocumentation/core-api/percpu-counter-tree.rst-92- any other updater.\n--\nDocumentation/core-api/percpu-counter-tree.rst=141=Functions and structures\n--\nDocumentation/core-api/percpu-counter-tree.rst-143-\nDocumentation/core-api/percpu-counter-tree.rst:144:.. kernel-doc:: include/linux/percpu_counter_tree.h\nDocumentation/core-api/percpu-counter-tree.rst:145:.. kernel-doc:: lib/percpu_counter_tree.c\n--\nMAINTAINERS=11955=F:\tDocumentation/core-api/percpu-counter-tree.rst\nMAINTAINERS:11956:F:\tinclude/linux/percpu_counter_tree.h\nMAINTAINERS:11957:F:\tlib/percpu_counter_tree.c\nMAINTAINERS:11958:F:\tlib/tests/percpu_counter_tree_bench.c\nMAINTAINERS:11959:F:\tlib/tests/percpu_counter_tree_kunit.c\nMAINTAINERS-11960-F:\ttools/testing/hpcc/\n--\ninclude/linux/mm.h=3608=static inline unsigned long get_mm_counter(struct mm_struct *mm, int member)\ninclude/linux/mm.h-3609-{\ninclude/linux/mm.h:3610:\treturn percpu_counter_tree_approximate_sum_positive(\u0026mm-\u003erss_stat[member]);\ninclude/linux/mm.h-3611-}\n--\ninclude/linux/mm.h=3613=static inline unsigned long get_mm_counter_sum(struct mm_struct *mm, int member)\ninclude/linux/mm.h-3614-{\ninclude/linux/mm.h:3615:\treturn percpu_counter_tree_precise_sum_positive(\u0026mm-\u003erss_stat[member]);\ninclude/linux/mm.h-3616-}\n--\ninclude/linux/mm.h=3628=static inline void __add_mm_counter(struct mm_struct *mm, int member, long value)\ninclude/linux/mm.h-3629-{\ninclude/linux/mm.h:3630:\tstruct percpu_counter_tree *counter = \u0026mm-\u003erss_stat[member];\ninclude/linux/mm.h-3631-\ninclude/linux/mm.h:3632:\tif (!percpu_counter_tree_active(counter) \u0026\u0026 current == READ_ONCE(mm-\u003erss_owner))\ninclude/linux/mm.h:3633:\t\tpercpu_counter_tree_add_owner(counter, value);\ninclude/linux/mm.h-3634-\telse\ninclude/linux/mm.h:3635:\t\tpercpu_counter_tree_add(counter, value);\ninclude/linux/mm.h-3636-}\n--\ninclude/linux/mm_types.h-21-#include \u003clinux/percpu_counter.h\u003e\ninclude/linux/mm_types.h:22:#include \u003clinux/percpu_counter_tree.h\u003e\ninclude/linux/mm_types.h-23-#include \u003clinux/types.h\u003e\n--\ninclude/linux/mm_types.h=1215=struct mm_struct {\n--\ninclude/linux/mm_types.h-1353-\ninclude/linux/mm_types.h:1354:\t\tstruct percpu_counter_tree rss_stat[NR_MM_COUNTERS];\ninclude/linux/mm_types.h-1355-\t\t/*\ninclude/linux/mm_types.h:1356:\t\t * Task updating rss_stat with percpu_counter_tree_add_owner()\ninclude/linux/mm_types.h-1357-\t\t * until the counter tree is activated by a second thread: the\n--\ninclude/linux/percpu_counter_tree.h=14=static_assert(PERCPU_COUNTER_TREE_MAX_BATCH - 1 \u003c= S8_MAX);\n--\ninclude/linux/percpu_counter_tree.h-17-\ninclude/linux/percpu_counter_tree.h:18:struct percpu_counter_tree {\ninclude/linux/percpu_counter_tree.h-19-\t/*\n--\ninclude/linux/percpu_counter_tree.h-35-\t * items allocated with the per-CPU counters. NULL until the tree of a\ninclude/linux/percpu_counter_tree.h:36:\t * counter initialized by percpu_counter_tree_init_many_lazy() is\ninclude/linux/percpu_counter_tree.h-37-\t * activated.\n--\ninclude/linux/percpu_counter_tree.h-42-\t * tree is activated, it also accumulates the updates of\ninclude/linux/percpu_counter_tree.h:43:\t * percpu_counter_tree_add_owner(), written only by the owner.\ninclude/linux/percpu_counter_tree.h-44-\t */\n--\ninclude/linux/percpu_counter_tree.h-46-\t/*\ninclude/linux/percpu_counter_tree.h:47:\t * Updates of percpu_counter_tree_add() made before the tree is\ninclude/linux/percpu_counter_tree.h-48-\t * activated, also applied to the precise and approximate values.\n--\ninclude/linux/percpu_counter_tree.h-52-\ninclude/linux/percpu_counter_tree.h:53:int percpu_counter_tree_init_many(struct percpu_counter_tree *counters,\ninclude/linux/percpu_counter_tree.h-54-\t\t\t\t unsigned int nr_counters, unsigned long batch_size,\ninclude/linux/percpu_counter_tree.h-55-\t\t\t\t gfp_t gfp_flags);\ninclude/linux/percpu_counter_tree.h:56:int percpu_counter_tree_init(struct percpu_counter_tree *counter,\ninclude/linux/percpu_counter_tree.h-57-\t\t\t unsigned long batch_size, gfp_t gfp_flags);\ninclude/linux/percpu_counter_tree.h:58:int percpu_counter_tree_init_many_lazy(struct percpu_counter_tree *counters,\ninclude/linux/percpu_counter_tree.h-59-\t\t\t\t unsigned int nr_counters, unsigned long batch_size);\ninclude/linux/percpu_counter_tree.h:60:int percpu_counter_tree_activate_many(struct percpu_counter_tree *counters,\ninclude/linux/percpu_counter_tree.h-61-\t\t\t\t unsigned int nr_counters, gfp_t gfp_flags);\ninclude/linux/percpu_counter_tree.h:62:void percpu_counter_tree_destroy_many(struct percpu_counter_tree *counters,\ninclude/linux/percpu_counter_tree.h-63-\t\t\t\t unsigned int nr_counters);\ninclude/linux/percpu_counter_tree.h:64:void percpu_counter_tree_destroy(struct percpu_counter_tree *counter);\ninclude/linux/percpu_counter_tree.h:65:void percpu_counter_tree_add(struct percpu_counter_tree *counter, long inc);\ninclude/linux/percpu_counter_tree.h:66:void percpu_counter_tree_add_owner(struct percpu_counter_tree *counter, long inc);\ninclude/linux/percpu_counter_tree.h:67:long percpu_counter_tree_precise_sum(struct percpu_counter_tree *counter);\ninclude/linux/percpu_counter_tree.h:68:int percpu_counter_tree_approximate_compare(struct percpu_counter_tree *a,\ninclude/linux/percpu_counter_tree.h:69:\t\t\t\t\t struct percpu_counter_tree *b);\ninclude/linux/percpu_counter_tree.h:70:int percpu_counter_tree_approximate_compare_value(struct percpu_counter_tree *counter, long v);\ninclude/linux/percpu_counter_tree.h:71:int percpu_counter_tree_precise_compare(struct percpu_counter_tree *a,\ninclude/linux/percpu_counter_tree.h:72:\t\t\t\t\tstruct percpu_counter_tree *b);\ninclude/linux/percpu_counter_tree.h:73:int percpu_counter_tree_precise_compare_value(struct percpu_counter_tree *counter, long v);\ninclude/linux/percpu_counter_tree.h:74:void percpu_counter_tree_set(struct percpu_counter_tree *counter, long v);\ninclude/linux/percpu_counter_tree.h:75:int percpu_counter_tree_subsystem_init(void);\ninclude/linux/percpu_counter_tree.h-76-\ninclude/linux/percpu_counter_tree.h-77-/**\ninclude/linux/percpu_counter_tree.h:78: * percpu_counter_tree_active() - Query whether the tree of a counter is active.\ninclude/linux/percpu_counter_tree.h-79- * @counter: The counter to query.\ninclude/linux/percpu_counter_tree.h-80- *\ninclude/linux/percpu_counter_tree.h:81: * A counter initialized by percpu_counter_tree_init_many() is active. A\ninclude/linux/percpu_counter_tree.h:82: * counter initialized by percpu_counter_tree_init_many_lazy() becomes\ninclude/linux/percpu_counter_tree.h:83: * active when percpu_counter_tree_activate_many() publishes its tree, and\ninclude/linux/percpu_counter_tree.h:84: * stays active. Once active, percpu_counter_tree_add() and\ninclude/linux/percpu_counter_tree.h:85: * percpu_counter_tree_add_owner() both add to the tree, so a user can skip\ninclude/linux/percpu_counter_tree.h-86- * selecting between them.\n--\ninclude/linux/percpu_counter_tree.h=90=static inline\ninclude/linux/percpu_counter_tree.h:91:bool percpu_counter_tree_active(struct percpu_counter_tree *counter)\ninclude/linux/percpu_counter_tree.h-92-{\n--\ninclude/linux/percpu_counter_tree.h-97-/**\ninclude/linux/percpu_counter_tree.h:98: * percpu_counter_tree_approximate_sum() - Return approximate counter sum.\ninclude/linux/percpu_counter_tree.h-99- * @counter: The counter to sum.\n--\ninclude/linux/percpu_counter_tree.h-101- * Querying the approximate sum is fast, but it is only accurate within\ninclude/linux/percpu_counter_tree.h:102: * the bound returned by percpu_counter_tree_approximate_accuracy().\ninclude/linux/percpu_counter_tree.h-103- * This is meant to be used when speed is preferred over accuracy.\n--\ninclude/linux/percpu_counter_tree.h=107=static inline\ninclude/linux/percpu_counter_tree.h:108:long percpu_counter_tree_approximate_sum(struct percpu_counter_tree *counter)\ninclude/linux/percpu_counter_tree.h-109-{\n--\ninclude/linux/percpu_counter_tree.h-111-\t * Pairs with the release of approx_sum in\ninclude/linux/percpu_counter_tree.h:112:\t * percpu_counter_tree_activate_many(): the root is zeroed before its\ninclude/linux/percpu_counter_tree.h-113-\t * pointer, and the address dependency orders the read of the root\n--\ninclude/linux/percpu_counter_tree.h-124-\ninclude/linux/percpu_counter_tree.h:125:unsigned long percpu_counter_tree_approximate_accuracy(struct percpu_counter_tree *counter);\ninclude/linux/percpu_counter_tree.h-126-\ninclude/linux/percpu_counter_tree.h-127-#if IS_ENABLED(CONFIG_KUNIT)\ninclude/linux/percpu_counter_tree.h:128:unsigned int percpu_counter_tree_check_config(void);\ninclude/linux/percpu_counter_tree.h:129:unsigned int percpu_counter_tree_check_residuals(struct percpu_counter_tree *counter);\ninclude/linux/percpu_counter_tree.h:130:void percpu_counter_tree_copy_state(struct percpu_counter_tree *dst,\ninclude/linux/percpu_counter_tree.h:131:\t\t\t\t struct percpu_counter_tree *src);\ninclude/linux/percpu_counter_tree.h-132-#endif\n--\ninclude/linux/percpu_counter_tree.h-135-\ninclude/linux/percpu_counter_tree.h:136:struct percpu_counter_tree {\ninclude/linux/percpu_counter_tree.h-137-\tatomic_long_t count;\n--\ninclude/linux/percpu_counter_tree.h=140=static inline\ninclude/linux/percpu_counter_tree.h:141:int percpu_counter_tree_init_many(struct percpu_counter_tree *counters,\ninclude/linux/percpu_counter_tree.h-142-\t\t\t\t unsigned int nr_counters, unsigned long batch_size,\n--\ninclude/linux/percpu_counter_tree.h=158=static inline\ninclude/linux/percpu_counter_tree.h:159:int percpu_counter_tree_init(struct percpu_counter_tree *counter,\ninclude/linux/percpu_counter_tree.h-160-\t\t\t unsigned long batch_size, gfp_t gfp_flags)\ninclude/linux/percpu_counter_tree.h-161-{\ninclude/linux/percpu_counter_tree.h:162:\treturn percpu_counter_tree_init_many(counter, 1, batch_size, gfp_flags);\ninclude/linux/percpu_counter_tree.h-163-}\n--\ninclude/linux/percpu_counter_tree.h=165=static inline\ninclude/linux/percpu_counter_tree.h:166:int percpu_counter_tree_init_many_lazy(struct percpu_counter_tree *counters,\ninclude/linux/percpu_counter_tree.h-167-\t\t\t\t unsigned int nr_counters, unsigned long batch_size)\ninclude/linux/percpu_counter_tree.h-168-{\ninclude/linux/percpu_counter_tree.h:169:\treturn percpu_counter_tree_init_many(counters, nr_counters, batch_size, 0);\ninclude/linux/percpu_counter_tree.h-170-}\n--\ninclude/linux/percpu_counter_tree.h=172=static inline\ninclude/linux/percpu_counter_tree.h:173:int percpu_counter_tree_activate_many(struct percpu_counter_tree *counters,\ninclude/linux/percpu_counter_tree.h-174-\t\t\t\t unsigned int nr_counters, gfp_t gfp_flags)\n--\ninclude/linux/percpu_counter_tree.h=179=static inline\ninclude/linux/percpu_counter_tree.h:180:bool percpu_counter_tree_active(struct percpu_counter_tree *counter)\ninclude/linux/percpu_counter_tree.h-181-{\n--\ninclude/linux/percpu_counter_tree.h=185=static inline\ninclude/linux/percpu_counter_tree.h:186:void percpu_counter_tree_destroy_many(struct percpu_counter_tree *counters,\ninclude/linux/percpu_counter_tree.h-187-\t\t\t\t unsigned int nr_counters)\n\n"
} |
| 5/2 | 2026/11/03 12:06 | llm | patch-evaluator |
1mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 74122 output: 77 thoughts: 45 Thoughts: Okay, let's dive into `lib/percpu_counter_tree.c`. My immediate goal is to understand the scope of this file. I need a quick overview of its role within the kernel. **Understanding the `percpu_counter_tree.c` Functions** First, I'm going to quickly identify which functions are exported and therefore potentially used by other parts of the kernel. Understanding the interface it presents to the rest of the system is paramount. Then, I need to know which functions are internal, marked `static`, as they are the implementation details and how they interact. This will give me a solid foundation for understanding the data structures and overall design employed in this per-CPU counter tree implementation. With this basic knowledge, I can then drill down into specific details as necessary, depending on the larger context. |
| 6/2 | 2026/11/03 12:08 | tool | grepper |
0mArgs:{
"Expression": "^([a-zA-Z0-9_]+ +)*percpu_counter_tree_[a-zA-Z0-9_]+",
"PathPrefixes": [
"lib/percpu_counter_tree.c",
"include/linux/percpu_counter_tree.h"
]
}Results: {
"Output": "\nFull output is too long, showing 200 out of 278 lines.\nUse more precise expression or PathPrefixes if possible.\n\nNumber of matching lines per file (2 files in total):\ninclude/linux/percpu_counter_tree.h:43\nlib/percpu_counter_tree.c:28\n\ninclude/linux/percpu_counter_tree.h=18=struct percpu_counter_tree {\n--\ninclude/linux/percpu_counter_tree.h-52-\ninclude/linux/percpu_counter_tree.h:53:int percpu_counter_tree_init_many(struct percpu_counter_tree *counters,\ninclude/linux/percpu_counter_tree.h-54-\t\t\t\t unsigned int nr_counters, unsigned long batch_size,\ninclude/linux/percpu_counter_tree.h-55-\t\t\t\t gfp_t gfp_flags);\ninclude/linux/percpu_counter_tree.h:56:int percpu_counter_tree_init(struct percpu_counter_tree *counter,\ninclude/linux/percpu_counter_tree.h-57-\t\t\t unsigned long batch_size, gfp_t gfp_flags);\ninclude/linux/percpu_counter_tree.h:58:int percpu_counter_tree_init_many_lazy(struct percpu_counter_tree *counters,\ninclude/linux/percpu_counter_tree.h-59-\t\t\t\t unsigned int nr_counters, unsigned long batch_size);\ninclude/linux/percpu_counter_tree.h:60:int percpu_counter_tree_activate_many(struct percpu_counter_tree *counters,\ninclude/linux/percpu_counter_tree.h-61-\t\t\t\t unsigned int nr_counters, gfp_t gfp_flags);\ninclude/linux/percpu_counter_tree.h:62:void percpu_counter_tree_destroy_many(struct percpu_counter_tree *counters,\ninclude/linux/percpu_counter_tree.h-63-\t\t\t\t unsigned int nr_counters);\ninclude/linux/percpu_counter_tree.h:64:void percpu_counter_tree_destroy(struct percpu_counter_tree *counter);\ninclude/linux/percpu_counter_tree.h:65:void percpu_counter_tree_add(struct percpu_counter_tree *counter, long inc);\ninclude/linux/percpu_counter_tree.h:66:void percpu_counter_tree_add_owner(struct percpu_counter_tree *counter, long inc);\ninclude/linux/percpu_counter_tree.h:67:long percpu_counter_tree_precise_sum(struct percpu_counter_tree *counter);\ninclude/linux/percpu_counter_tree.h:68:int percpu_counter_tree_approximate_compare(struct percpu_counter_tree *a,\ninclude/linux/percpu_counter_tree.h-69-\t\t\t\t\t struct percpu_counter_tree *b);\ninclude/linux/percpu_counter_tree.h:70:int percpu_counter_tree_approximate_compare_value(struct percpu_counter_tree *counter, long v);\ninclude/linux/percpu_counter_tree.h:71:int percpu_counter_tree_precise_compare(struct percpu_counter_tree *a,\ninclude/linux/percpu_counter_tree.h-72-\t\t\t\t\tstruct percpu_counter_tree *b);\ninclude/linux/percpu_counter_tree.h:73:int percpu_counter_tree_precise_compare_value(struct percpu_counter_tree *counter, long v);\ninclude/linux/percpu_counter_tree.h:74:void percpu_counter_tree_set(struct percpu_counter_tree *counter, long v);\ninclude/linux/percpu_counter_tree.h:75:int percpu_counter_tree_subsystem_init(void);\ninclude/linux/percpu_counter_tree.h-76-\n--\ninclude/linux/percpu_counter_tree.h=90=static inline\ninclude/linux/percpu_counter_tree.h:91:bool percpu_counter_tree_active(struct percpu_counter_tree *counter)\ninclude/linux/percpu_counter_tree.h-92-{\n--\ninclude/linux/percpu_counter_tree.h=107=static inline\ninclude/linux/percpu_counter_tree.h:108:long percpu_counter_tree_approximate_sum(struct percpu_counter_tree *counter)\ninclude/linux/percpu_counter_tree.h-109-{\n--\ninclude/linux/percpu_counter_tree.h-124-\ninclude/linux/percpu_counter_tree.h:125:unsigned long percpu_counter_tree_approximate_accuracy(struct percpu_counter_tree *counter);\ninclude/linux/percpu_counter_tree.h-126-\ninclude/linux/percpu_counter_tree.h-127-#if IS_ENABLED(CONFIG_KUNIT)\ninclude/linux/percpu_counter_tree.h:128:unsigned int percpu_counter_tree_check_config(void);\ninclude/linux/percpu_counter_tree.h:129:unsigned int percpu_counter_tree_check_residuals(struct percpu_counter_tree *counter);\ninclude/linux/percpu_counter_tree.h:130:void percpu_counter_tree_copy_state(struct percpu_counter_tree *dst,\ninclude/linux/percpu_counter_tree.h-131-\t\t\t\t struct percpu_counter_tree *src);\n--\ninclude/linux/percpu_counter_tree.h=140=static inline\ninclude/linux/percpu_counter_tree.h:141:int percpu_counter_tree_init_many(struct percpu_counter_tree *counters,\ninclude/linux/percpu_counter_tree.h-142-\t\t\t\t unsigned int nr_counters, unsigned long batch_size,\n--\ninclude/linux/percpu_counter_tree.h=158=static inline\ninclude/linux/percpu_counter_tree.h:159:int percpu_counter_tree_init(struct percpu_counter_tree *counter,\ninclude/linux/percpu_counter_tree.h-160-\t\t\t unsigned long batch_size, gfp_t gfp_flags)\n--\ninclude/linux/percpu_counter_tree.h=165=static inline\ninclude/linux/percpu_counter_tree.h:166:int percpu_counter_tree_init_many_lazy(struct percpu_counter_tree *counters,\ninclude/linux/percpu_counter_tree.h-167-\t\t\t\t unsigned int nr_counters, unsigned long batch_size)\n--\ninclude/linux/percpu_counter_tree.h=172=static inline\ninclude/linux/percpu_counter_tree.h:173:int percpu_counter_tree_activate_many(struct percpu_counter_tree *counters,\ninclude/linux/percpu_counter_tree.h-174-\t\t\t\t unsigned int nr_counters, gfp_t gfp_flags)\n--\ninclude/linux/percpu_counter_tree.h=179=static inline\ninclude/linux/percpu_counter_tree.h:180:bool percpu_counter_tree_active(struct percpu_counter_tree *counter)\ninclude/linux/percpu_counter_tree.h-181-{\n--\ninclude/linux/percpu_counter_tree.h=185=static inline\ninclude/linux/percpu_counter_tree.h:186:void percpu_counter_tree_destroy_many(struct percpu_counter_tree *counters,\ninclude/linux/percpu_counter_tree.h-187-\t\t\t\t unsigned int nr_counters)\n--\ninclude/linux/percpu_counter_tree.h=191=static inline\ninclude/linux/percpu_counter_tree.h:192:void percpu_counter_tree_destroy(struct percpu_counter_tree *counter)\ninclude/linux/percpu_counter_tree.h-193-{\n--\ninclude/linux/percpu_counter_tree.h=196=static inline\ninclude/linux/percpu_counter_tree.h:197:long percpu_counter_tree_precise_sum(struct percpu_counter_tree *counter)\ninclude/linux/percpu_counter_tree.h-198-{\n--\ninclude/linux/percpu_counter_tree.h=202=static inline\ninclude/linux/percpu_counter_tree.h:203:int percpu_counter_tree_precise_compare(struct percpu_counter_tree *a,\ninclude/linux/percpu_counter_tree.h-204-\t\t\t\t\tstruct percpu_counter_tree *b)\n--\ninclude/linux/percpu_counter_tree.h=217=static inline\ninclude/linux/percpu_counter_tree.h:218:int percpu_counter_tree_precise_compare_value(struct percpu_counter_tree *counter, long v)\ninclude/linux/percpu_counter_tree.h-219-{\n--\ninclude/linux/percpu_counter_tree.h=231=static inline\ninclude/linux/percpu_counter_tree.h:232:int percpu_counter_tree_approximate_compare(struct percpu_counter_tree *a,\ninclude/linux/percpu_counter_tree.h-233-\t\t\t\t\t struct percpu_counter_tree *b)\n--\ninclude/linux/percpu_counter_tree.h=238=static inline\ninclude/linux/percpu_counter_tree.h:239:int percpu_counter_tree_approximate_compare_value(struct percpu_counter_tree *counter, long v)\ninclude/linux/percpu_counter_tree.h-240-{\n--\ninclude/linux/percpu_counter_tree.h=244=static inline\ninclude/linux/percpu_counter_tree.h:245:void percpu_counter_tree_set(struct percpu_counter_tree *counter, long v)\ninclude/linux/percpu_counter_tree.h-246-{\n--\ninclude/linux/percpu_counter_tree.h=250=static inline\ninclude/linux/percpu_counter_tree.h:251:unsigned long percpu_counter_tree_approximate_accuracy(struct percpu_counter_tree *counter)\ninclude/linux/percpu_counter_tree.h-252-{\n--\ninclude/linux/percpu_counter_tree.h=256=static inline\ninclude/linux/percpu_counter_tree.h:257:void percpu_counter_tree_add(struct percpu_counter_tree *counter, long inc)\ninclude/linux/percpu_counter_tree.h-258-{\n--\ninclude/linux/percpu_counter_tree.h=262=static inline\ninclude/linux/percpu_counter_tree.h:263:void percpu_counter_tree_add_owner(struct percpu_counter_tree *counter, long inc)\ninclude/linux/percpu_counter_tree.h-264-{\n--\ninclude/linux/percpu_counter_tree.h=268=static inline\ninclude/linux/percpu_counter_tree.h:269:long percpu_counter_tree_approximate_sum(struct percpu_counter_tree *counter)\ninclude/linux/percpu_counter_tree.h-270-{\n--\ninclude/linux/percpu_counter_tree.h=274=static inline\ninclude/linux/percpu_counter_tree.h:275:int percpu_counter_tree_subsystem_init(void)\ninclude/linux/percpu_counter_tree.h-276-{\n--\ninclude/linux/percpu_counter_tree.h=291=static inline\ninclude/linux/percpu_counter_tree.h:292:long percpu_counter_tree_approximate_sum_positive(struct percpu_counter_tree *counter)\ninclude/linux/percpu_counter_tree.h-293-{\n--\ninclude/linux/percpu_counter_tree.h=308=static inline\ninclude/linux/percpu_counter_tree.h:309:long percpu_counter_tree_precise_sum_positive(struct percpu_counter_tree *counter)\ninclude/linux/percpu_counter_tree.h-310-{\n--\ninclude/linux/percpu_counter_tree.h=336=static inline\ninclude/linux/percpu_counter_tree.h:337:void percpu_counter_tree_approximate_min_max_range(long approx_sum, unsigned long accuracy,\ninclude/linux/percpu_counter_tree.h-338-\t\t\t\t\t\t long *precise_min, long *precise_max)\n--\ninclude/linux/percpu_counter_tree.h=354=static inline\ninclude/linux/percpu_counter_tree.h:355:void percpu_counter_tree_approximate_min_max(struct percpu_counter_tree *counter,\ninclude/linux/percpu_counter_tree.h-356-\t\t\t\t\t long *approx_sum, long *precise_min, long *precise_max)\n--\nlib/percpu_counter_tree.c=291=static\nlib/percpu_counter_tree.c:292:size_t percpu_counter_tree_position_size(unsigned int nr_counters)\nlib/percpu_counter_tree.c-293-{\n--\nlib/percpu_counter_tree.c=298=static inline\nlib/percpu_counter_tree.c:299:unsigned int percpu_counter_tree_items_stride(unsigned int nr_counters)\nlib/percpu_counter_tree.c-300-{\n--\nlib/percpu_counter_tree.c=317=static\nlib/percpu_counter_tree.c:318:void percpu_counter_tree_free_items(atomic_long_t *items, unsigned int nr_counters)\nlib/percpu_counter_tree.c-319-{\n--\nlib/percpu_counter_tree.c=347=static inline\nlib/percpu_counter_tree.c:348:unsigned long percpu_counter_tree_batch_size(struct percpu_counter_tree *counter)\nlib/percpu_counter_tree.c-349-{\n--\nlib/percpu_counter_tree.c=368=static inline\nlib/percpu_counter_tree.c:369:unsigned long percpu_counter_tree_accuracy(struct percpu_counter_tree *counter)\nlib/percpu_counter_tree.c-370-{\n--\nlib/percpu_counter_tree.c=378=int __percpu_counter_tree_init_many(struct percpu_counter_tree *counters,\n--\nlib/percpu_counter_tree.c-419- */\nlib/percpu_counter_tree.c:420:int percpu_counter_tree_init_many(struct percpu_counter_tree *counters,\nlib/percpu_counter_tree.c-421-\t\t\t\t unsigned int nr_counters, unsigned long batch_size,\n--\nlib/percpu_counter_tree.c=448=EXPORT_SYMBOL_GPL(percpu_counter_tree_init_many);\n--\nlib/percpu_counter_tree.c-466- */\nlib/percpu_counter_tree.c:467:int percpu_counter_tree_init(struct percpu_counter_tree *counter,\nlib/percpu_counter_tree.c-468-\t\t\t unsigned long batch_size, gfp_t gfp_flags)\n--\nlib/percpu_counter_tree.c=472=EXPORT_SYMBOL_GPL(percpu_counter_tree_init);\n--\nlib/percpu_counter_tree.c-498- */\nlib/percpu_counter_tree.c:499:int percpu_counter_tree_init_many_lazy(struct percpu_counter_tree *counters,\nlib/percpu_counter_tree.c-500-\t\t\t\t unsigned int nr_counters, unsigned long batch_size)\n--\nlib/percpu_counter_tree.c=507=EXPORT_SYMBOL_GPL(percpu_counter_tree_init_many_lazy);\n--\nlib/percpu_counter_tree.c-591- */\nlib/percpu_counter_tree.c:592:int percpu_counter_tree_activate_many(struct percpu_counter_tree *counters,\nlib/percpu_counter_tree.c-593-\t\t\t\t unsigned int nr_counters, gfp_t gfp_flags)\n--\nlib/percpu_counter_tree.c=629=EXPORT_SYMBOL_GPL(percpu_counter_tree_activate_many);\n--\nlib/percpu_counter_tree.c-639- */\nlib/percpu_counter_tree.c:640:void percpu_counter_tree_destroy_many(struct percpu_counter_tree *counters,\nlib/percpu_counter_tree.c-641-\t\t\t\t unsigned int nr_counters)\n--\nlib/percpu_counter_tree.c=655=EXPORT_SYMBOL_GPL(percpu_counter_tree_destroy_many);\n--\nlib/percpu_counter_tree.c-662- */\nlib/percpu_counter_tree.c:663:void percpu_counter_tree_destroy(struct percpu_counter_tree *counter)\nlib/percpu_counter_tree.c-664-{\n--\nlib/percpu_counter_tree.c=673=static inline\nlib/percpu_counter_tree.c:674:long percpu_counter_tree_item_carry(long v, unsigned long bit_mask)\nlib/percpu_counter_tree.c-675-{\n--\nlib/percpu_counter_tree.c=686=static\nlib/percpu_counter_tree.c:687:void percpu_counter_tree_add_slowpath(struct percpu_counter_tree *counter, long inc)\nlib/percpu_counter_tree.c-688-{\n--\nlib/percpu_counter_tree.c=768=static inline\nlib/percpu_counter_tree.c:769:void percpu_counter_tree_add_tree(struct percpu_counter_tree *counter,\nlib/percpu_counter_tree.c-770-\t\t\t\t s8 __percpu *level0, long inc)\n\n"
} |
| 7/2 | 2026/11/03 12:08 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 79046 output: 34 thoughts: 46 Thoughts: Okay, let's dive into this. Here's my thought process on examining `lib/percpu_counter_tree.c`: **Analyzing `percpu_counter_tree.c` Implementation** Right, let me get a feel for this. My initial instinct is to see what else is happening within this file. Specifically, I need to understand the full context of the code. Let's start with a comprehensive sweep of the file, starting from line 770 and going through to the end. I want to see how these functions interact and fit within the broader structure, especially those related to per-CPU counters and tree management. Understanding the surrounding code will be vital to understand the design. |
| 8/2 | 2026/11/03 12:09 | tool | grepper |
0mArgs:{
"Expression": "EXPORT_SYMBOL",
"PathPrefixes": [
"lib/percpu_counter_tree.c"
]
}Results: {
"Output": "lib/percpu_counter_tree.c=420=int percpu_counter_tree_init_many(struct percpu_counter_tree *counters,\n--\nlib/percpu_counter_tree.c-447-}\nlib/percpu_counter_tree.c:448:EXPORT_SYMBOL_GPL(percpu_counter_tree_init_many);\nlib/percpu_counter_tree.c-449-\n--\nlib/percpu_counter_tree.c=467=int percpu_counter_tree_init(struct percpu_counter_tree *counter,\n--\nlib/percpu_counter_tree.c-471-}\nlib/percpu_counter_tree.c:472:EXPORT_SYMBOL_GPL(percpu_counter_tree_init);\nlib/percpu_counter_tree.c-473-\n--\nlib/percpu_counter_tree.c=499=int percpu_counter_tree_init_many_lazy(struct percpu_counter_tree *counters,\n--\nlib/percpu_counter_tree.c-506-}\nlib/percpu_counter_tree.c:507:EXPORT_SYMBOL_GPL(percpu_counter_tree_init_many_lazy);\nlib/percpu_counter_tree.c-508-\n--\nlib/percpu_counter_tree.c=592=int percpu_counter_tree_activate_many(struct percpu_counter_tree *counters,\n--\nlib/percpu_counter_tree.c-628-}\nlib/percpu_counter_tree.c:629:EXPORT_SYMBOL_GPL(percpu_counter_tree_activate_many);\nlib/percpu_counter_tree.c-630-\n--\nlib/percpu_counter_tree.c=640=void percpu_counter_tree_destroy_many(struct percpu_counter_tree *counters,\n--\nlib/percpu_counter_tree.c-654-}\nlib/percpu_counter_tree.c:655:EXPORT_SYMBOL_GPL(percpu_counter_tree_destroy_many);\nlib/percpu_counter_tree.c-656-\n--\nlib/percpu_counter_tree.c=663=void percpu_counter_tree_destroy(struct percpu_counter_tree *counter)\n--\nlib/percpu_counter_tree.c-666-}\nlib/percpu_counter_tree.c:667:EXPORT_SYMBOL_GPL(percpu_counter_tree_destroy);\nlib/percpu_counter_tree.c-668-\n--\nlib/percpu_counter_tree.c=813=void percpu_counter_tree_add(struct percpu_counter_tree *counter, long inc)\n--\nlib/percpu_counter_tree.c-829-}\nlib/percpu_counter_tree.c:830:EXPORT_SYMBOL_GPL(percpu_counter_tree_add);\nlib/percpu_counter_tree.c-831-\n--\nlib/percpu_counter_tree.c=842=void percpu_counter_tree_add_owner(struct percpu_counter_tree *counter, long inc)\n--\nlib/percpu_counter_tree.c-852-}\nlib/percpu_counter_tree.c:853:EXPORT_SYMBOL_GPL(percpu_counter_tree_add_owner);\nlib/percpu_counter_tree.c-854-\n--\nlib/percpu_counter_tree.c=918=long percpu_counter_tree_precise_sum(struct percpu_counter_tree *counter)\n--\nlib/percpu_counter_tree.c-921-}\nlib/percpu_counter_tree.c:922:EXPORT_SYMBOL_GPL(percpu_counter_tree_precise_sum);\nlib/percpu_counter_tree.c-923-\n--\nlib/percpu_counter_tree.c=1001=int percpu_counter_tree_approximate_compare(struct percpu_counter_tree *a,\n--\nlib/percpu_counter_tree.c-1010-}\nlib/percpu_counter_tree.c:1011:EXPORT_SYMBOL_GPL(percpu_counter_tree_approximate_compare);\nlib/percpu_counter_tree.c-1012-\n--\nlib/percpu_counter_tree.c=1033=int percpu_counter_tree_approximate_compare_value(struct percpu_counter_tree *counter, long v)\n--\nlib/percpu_counter_tree.c-1040-}\nlib/percpu_counter_tree.c:1041:EXPORT_SYMBOL_GPL(percpu_counter_tree_approximate_compare_value);\nlib/percpu_counter_tree.c-1042-\n--\nlib/percpu_counter_tree.c=1064=int percpu_counter_tree_precise_compare(struct percpu_counter_tree *a,\n--\nlib/percpu_counter_tree.c-1107-}\nlib/percpu_counter_tree.c:1108:EXPORT_SYMBOL_GPL(percpu_counter_tree_precise_compare);\nlib/percpu_counter_tree.c-1109-\n--\nlib/percpu_counter_tree.c=1131=int percpu_counter_tree_precise_compare_value(struct percpu_counter_tree *counter, long v)\n--\nlib/percpu_counter_tree.c-1151-}\nlib/percpu_counter_tree.c:1152:EXPORT_SYMBOL_GPL(percpu_counter_tree_precise_compare_value);\nlib/percpu_counter_tree.c-1153-\n--\nlib/percpu_counter_tree.c=1167=void percpu_counter_tree_set(struct percpu_counter_tree *counter, long v)\n--\nlib/percpu_counter_tree.c-1170-}\nlib/percpu_counter_tree.c:1171:EXPORT_SYMBOL_GPL(percpu_counter_tree_set);\nlib/percpu_counter_tree.c-1172-\n--\nlib/percpu_counter_tree.c=1193=unsigned long percpu_counter_tree_approximate_accuracy(struct percpu_counter_tree *counter)\n--\nlib/percpu_counter_tree.c-1196-}\nlib/percpu_counter_tree.c:1197:EXPORT_SYMBOL_GPL(percpu_counter_tree_approximate_accuracy);\nlib/percpu_counter_tree.c-1198-\n--\nlib/percpu_counter_tree.c=1236=unsigned int percpu_counter_tree_check_config(void)\n--\nlib/percpu_counter_tree.c-1252-}\nlib/percpu_counter_tree.c:1253:EXPORT_SYMBOL_IF_KUNIT(percpu_counter_tree_check_config);\nlib/percpu_counter_tree.c-1254-\n--\nlib/percpu_counter_tree.c=1265=unsigned int percpu_counter_tree_check_residuals(struct percpu_counter_tree *counter)\n--\nlib/percpu_counter_tree.c-1309-}\nlib/percpu_counter_tree.c:1310:EXPORT_SYMBOL_IF_KUNIT(percpu_counter_tree_check_residuals);\nlib/percpu_counter_tree.c-1311-\n--\nlib/percpu_counter_tree.c=1317=void percpu_counter_tree_copy_state(struct percpu_counter_tree *dst,\n--\nlib/percpu_counter_tree.c-1332-}\nlib/percpu_counter_tree.c:1333:EXPORT_SYMBOL_IF_KUNIT(percpu_counter_tree_copy_state);\nlib/percpu_counter_tree.c-1334-#endif\n"
} |
| 9/2 | 2026/11/03 12:09 | llm | patch-evaluator |
1mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 81283 output: 38 thoughts: 1593 Thoughts: **Analyzing RSS Stat Interaction with Memory Management**
Okay, let's dive into how RSS statistics interact with memory management, specifically focusing on the `percpu_counter_tree` implementation. My initial thought is to trace the call sites for key functions: `__add_mm_counter`, `add_mm_counter`, `inc_mm_counter`, and `dec_mm_counter`. It seems like these are ubiquitous, called from within the memory management subsystem itself: page faults, `alloc_pages`, `unmap`, `do_anonymous_page`, `zap_pte_range`, and so on.
Now, how is `percpu_counter_tree_activate_many` called? I've found it in `kernel/fork.c`, within the `mm_activate_rss_stat` function. This activation happens when a second thread shares the memory map (`CLONE_VM`) or during a `vfork` operation. The code's interesting, there is some locking here.
Further exploration of `kernel/exit.c` reveals that `exit_mm` checks `mm->rss_owner`, potentially setting it to `NULL`. I see a similar pattern in `fs/exec.c` with `exec_mmap` -- `mm->rss_owner` is set to the current task. `kernel/fork.c`'s `dup_mm` also sets `mm->rss_owner`. And, of course, `__mmdrop` calls `percpu_counter_tree_destroy_many`, which is expected. `check_mm` calls `percpu_counter_tree_precise_sum` for RSS accounting, which gives me an idea of how this is consumed by user space. Tracing also leads me to a call from `get_mm_counter()` in `trace/events/kmem.h`.
Let's summarize the key functions and how they're utilized:
1. `percpu_counter_tree_init_many_lazy` is called during `mm_init()` (triggered by `fork`, `exec`, etc.).
2. In a single-threaded process, `__add_mm_counter()` calls `percpu_counter_tree_add_owner()` (or `percpu_counter_tree_add()` if it's not the owner).
3. When a thread is created (`clone` with `CLONE_VM`), `mm_activate_rss_stat()` calls `percpu_counter_tree_activate_many()`.
4. Subsequent calls to `percpu_counter_tree_add()` happen when `percpu_counter_tree_active()` returns true.
5. Inside `percpu_counter_tree_add()`, `percpu_counter_tree_add_tree()` is called, which in turn invokes `percpu_counter_tree_add_slowpath()` when the carry threshold is reached.
6. `percpu_counter_tree_destroy_many()` is called in `exit_mm()` and `__mmdrop()`. Also, `percpu_counter_tree_precise_sum()` is used by `check_mm()`.
7. `proc`, `sysfs`, or page fault reporting can call `get_mm_counter()`, which calls `percpu_counter_tree_approximate_sum_positive()` or `percpu_counter_tree_approximate_sum()`.
I should also note that `percpu_counter_tree_subsystem_init()` is called in `start_kernel()`.
Now, I need to keep in mind the focus symbols guidelines. Let's not include generic functions like memory allocators/deallocators, page fault handlers. Let's instead focus on the core logic and API entry points.
I will focus on:
* Functions in `lib/percpu_counter_tree.c`:
* `percpu_counter_tree_init_many`
* `percpu_counter_tree_init`
* `percpu_counter_tree_init_many_lazy`
* `percpu_counter_tree_activate_many`
* `percpu_counter_tree_destroy_many`
* `percpu_counter_tree_destroy`
* `percpu_counter_tree_add`
* `percpu_counter_tree_add_owner`
* `percpu_counter_tree_precise_sum`
* `percpu_counter_tree_approximate_compare`
* `percpu_counter_tree_approximate_compare_value`
* `percpu_counter_tree_
|
| 10/2 | 2026/11/03 12:10 | tool | read-file |
0mArgs:{
"File": "lib/percpu_counter_tree.c",
"FirstLine": 680,
"LineCount": 40
}Results: {
"Contents": " 680:\t\n 681:\t/*\n 682:\t * It does not matter through which path the carry propagates up the\n 683:\t * tree, therefore there is no need to disable preemption because the\n 684:\t * cpu number is only used to favor cache locality.\n 685:\t */\n 686:\tstatic\n 687:\tvoid percpu_counter_tree_add_slowpath(struct percpu_counter_tree *counter, long inc)\n 688:\t{\n 689:\t\tunsigned int level_items, nr_levels = tree_topology.config-\u003enr_levels,\n 690:\t\t\t level, n_arity_order;\n 691:\t\tunsigned long bit_mask;\n 692:\t\tunsigned int stride = counter-\u003eitems_stride;\n 693:\t\tunsigned int cpu = raw_smp_processor_id();\n 694:\t\tatomic_long_t *root, *item;\n 695:\t\n 696:\t\t/*\n 697:\t\t * Pairs with the release of level0 in\n 698:\t\t * percpu_counter_tree_activate_many(), which follows the store of\n 699:\t\t * approx_sum. The caller read level0, but approx_sum is a separate\n 700:\t\t * field, which the address dependency of that read does not order:\n 701:\t\t * read it after level0.\n 702:\t\t */\n 703:\t\tsmp_rmb();\n 704:\t\troot = READ_ONCE(counter-\u003eapprox_sum);\n 705:\t\titem = root - (tree_topology.nr_items - 1) * stride;\n 706:\t\tn_arity_order = tree_topology.config-\u003en_arity_order[0];\n 707:\t\tbit_mask = percpu_counter_tree_batch_size(counter) \u003c\u003c n_arity_order;\n 708:\t\tlevel_items = 1U \u003c\u003c (tree_topology.nr_cpus_order - n_arity_order);\n 709:\t\n 710:\t\tfor (level = 1; level \u003c nr_levels; level++) {\n 711:\t\t\t/*\n 712:\t\t\t * A multiple of bit_mask would leave the residual of this\n 713:\t\t\t * level unchanged: propagate it to the next level directly.\n 714:\t\t\t * This trades the hysteresis of this level for not touching\n 715:\t\t\t * its cache line.\n 716:\t\t\t */\n 717:\t\t\tif (abs(inc) \u0026 (bit_mask - 1)) {\n 718:\t\t\t\tatomic_long_t *count = \u0026item[(cpu \u0026 (level_items - 1)) * stride];\n 719:\t\t\t\tlong res;\n"
} |
| 11/2 | 2026/11/03 12:10 | llm | patch-evaluator |
1mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 84646 output: 24 thoughts: 2594 Thoughts: **Focusing on the Per-CPU Counter Tree: Prioritizing Targeted Fuzzing** Alright, let's break this down. My initial thought was to list all the functions in `lib/percpu_counter_tree.c`, but quickly realized that would be a mistake. We're aiming to use this for fuzzing, and the goal here is to focus the fuzzer on the new logic being added, not dilute it with ubiquitous functions. `percpu_counter_tree_add_slowpath` is static, so it's irrelevant. But the non-static functions, the ones exposed, initially look like candidates: `percpu_counter_tree_activate_many`, `percpu_counter_tree_init_many_lazy`, `percpu_counter_tree_destroy_many`, `percpu_counter_tree_add`, `percpu_counter_tree_add_owner`, `percpu_counter_tree_precise_sum`, `percpu_counter_tree_init_many`, `percpu_counter_tree_init`, `percpu_counter_tree_destroy`, `percpu_counter_tree_set`, `percpu_counter_tree_approximate_compare_value`, `percpu_counter_tree_precise_compare_value`, and `percpu_counter_tree_approximate_accuracy`. I need to filter out the common stuff. `percpu_counter_tree_add` and `percpu_counter_tree_add_owner` are called when memory counters are updated, which happens on *every* program, so those are out. The goal is to identify specific feature logic or entrypoints for this *new* per-cpu counter tree subsystem, and avoid listing generic functions. The key entrypoint is `percpu_counter_tree_activate_many` because it activates the counter tree and is called during `clone(CLONE_VM)` without `CLONE_VFORK`, when threads are created by `copy_mm()`. That makes it *very* interesting. `percpu_counter_tree_precise_sum` is only called when precise comparison is needed. `percpu_counter_tree_init_many_lazy` is called in `mm_init()`, but that happens on every `fork` or `execve`, so it is too generic and not useful. The critical distinction is this: `percpu_counter_tree_add` and `percpu_counter_tree_add_owner` are called with `__add_mm_counter`, which happens on almost all page faults and memory allocations. But *before* activation (single-threaded), `percpu_counter_tree_add_owner()` is called when the thread has the `mm->rss_owner` role, and after (multi-threaded), it calls `percpu_counter_tree_add()`. The former is also almost ubiquitous. However, `percpu_counter_tree_add` becomes relevant in multi-threaded scenarios. So, the focus is really on: * `percpu_counter_tree_activate_many`: *This is a core entrypoint*. Multi-threading and the activation of the RSS counter tree. This is *not* ubiquitous. This is the entrypoint to the new logic. This needs to be a FocusSymbol. * `percpu_counter_tree_precise_sum`: *This is called when `check_mm` happens, or when a precise comparison is needed*. * `percpu_counter_tree_approximate_compare`, `percpu_counter_tree_precise_compare`: Comparison API entrypoints. * `percpu_counter_tree_add`: *Only called if RSS is modified by other threads or there is a multithreaded test case*. * `percpu_counter_tree_add_owner`: *Almost ubiquitous because of memory allocation, but not always, and only on single-threaded syzkaller programs*. And let's confirm the list of all exported non-static functions in `lib/percpu_counter_tree.c`. It's essential to not miss any of the key entrypoints: `percpu_counter_tree_activate_many`, `percpu_counter_tree_precise_sum`, `percpu_counter_tree_approximate_compare_value`, `percpu_counter_tree_precise_compare_value`, `percpu_counter_tree_set`, `percpu_counter_tree_approximate_accuracy`, `percpu_counter |
| 12/2 | 2026/11/03 12:11 | tool | grepper |
0mArgs:{
"Expression": "percpu_counter_tree_precise_sum"
}Results: {
"Output": "include/linux/mm.h=3613=static inline unsigned long get_mm_counter_sum(struct mm_struct *mm, int member)\ninclude/linux/mm.h-3614-{\ninclude/linux/mm.h:3615:\treturn percpu_counter_tree_precise_sum_positive(\u0026mm-\u003erss_stat[member]);\ninclude/linux/mm.h-3616-}\n--\ninclude/linux/percpu_counter_tree.h=66=void percpu_counter_tree_add_owner(struct percpu_counter_tree *counter, long inc);\ninclude/linux/percpu_counter_tree.h:67:long percpu_counter_tree_precise_sum(struct percpu_counter_tree *counter);\ninclude/linux/percpu_counter_tree.h-68-int percpu_counter_tree_approximate_compare(struct percpu_counter_tree *a,\n--\ninclude/linux/percpu_counter_tree.h=196=static inline\ninclude/linux/percpu_counter_tree.h:197:long percpu_counter_tree_precise_sum(struct percpu_counter_tree *counter)\ninclude/linux/percpu_counter_tree.h-198-{\n--\ninclude/linux/percpu_counter_tree.h=203=int percpu_counter_tree_precise_compare(struct percpu_counter_tree *a,\n--\ninclude/linux/percpu_counter_tree.h-206-\t/* Compare on the wrapped difference, as on SMP. */\ninclude/linux/percpu_counter_tree.h:207:\tlong delta = (long)((unsigned long)percpu_counter_tree_precise_sum(a) -\ninclude/linux/percpu_counter_tree.h:208:\t\t\t (unsigned long)percpu_counter_tree_precise_sum(b));\ninclude/linux/percpu_counter_tree.h-209-\n--\ninclude/linux/percpu_counter_tree.h=218=int percpu_counter_tree_precise_compare_value(struct percpu_counter_tree *counter, long v)\n--\ninclude/linux/percpu_counter_tree.h-221-\tlong delta = (long)((unsigned long)v -\ninclude/linux/percpu_counter_tree.h:222:\t\t\t (unsigned long)percpu_counter_tree_precise_sum(counter));\ninclude/linux/percpu_counter_tree.h-223-\n--\ninclude/linux/percpu_counter_tree.h=269=long percpu_counter_tree_approximate_sum(struct percpu_counter_tree *counter)\ninclude/linux/percpu_counter_tree.h-270-{\ninclude/linux/percpu_counter_tree.h:271:\treturn percpu_counter_tree_precise_sum(counter);\ninclude/linux/percpu_counter_tree.h-272-}\n--\ninclude/linux/percpu_counter_tree.h=292=long percpu_counter_tree_approximate_sum_positive(struct percpu_counter_tree *counter)\n--\ninclude/linux/percpu_counter_tree.h-299-/**\ninclude/linux/percpu_counter_tree.h:300: * percpu_counter_tree_precise_sum_positive() - Return a positive precise counter sum.\ninclude/linux/percpu_counter_tree.h-301- * @counter: The counter to sum.\n--\ninclude/linux/percpu_counter_tree.h=308=static inline\ninclude/linux/percpu_counter_tree.h:309:long percpu_counter_tree_precise_sum_positive(struct percpu_counter_tree *counter)\ninclude/linux/percpu_counter_tree.h-310-{\ninclude/linux/percpu_counter_tree.h:311:\tlong v = percpu_counter_tree_precise_sum(counter);\ninclude/linux/percpu_counter_tree.h-312-\n--\nkernel/fork.c=650=static void check_mm(struct mm_struct *mm)\n--\nkernel/fork.c-657-\tfor (i = 0; i \u003c NR_MM_COUNTERS; i++) {\nkernel/fork.c:658:\t\tlong x = percpu_counter_tree_precise_sum(\u0026mm-\u003erss_stat[i]);\nkernel/fork.c-659-\n--\nlib/percpu_counter_tree.c=507=EXPORT_SYMBOL_GPL(percpu_counter_tree_init_many_lazy);\n--\nlib/percpu_counter_tree.c-559- * to the publication: they order carries, see\nlib/percpu_counter_tree.c:560: * percpu_counter_tree_precise_sum_unbiased().\nlib/percpu_counter_tree.c-561- */\n--\nlib/percpu_counter_tree.c=863=static\nlib/percpu_counter_tree.c:864:long percpu_counter_tree_precise_sum_unbiased(struct percpu_counter_tree *counter)\nlib/percpu_counter_tree.c-865-{\n--\nlib/percpu_counter_tree.c-902-/**\nlib/percpu_counter_tree.c:903: * percpu_counter_tree_precise_sum() - Return precise counter sum.\nlib/percpu_counter_tree.c-904- * @counter: The counter to sum.\n--\nlib/percpu_counter_tree.c-917- */\nlib/percpu_counter_tree.c:918:long percpu_counter_tree_precise_sum(struct percpu_counter_tree *counter)\nlib/percpu_counter_tree.c-919-{\nlib/percpu_counter_tree.c:920:\treturn percpu_counter_tree_precise_sum_unbiased(counter) + READ_ONCE(counter-\u003ebias);\nlib/percpu_counter_tree.c-921-}\nlib/percpu_counter_tree.c:922:EXPORT_SYMBOL_GPL(percpu_counter_tree_precise_sum);\nlib/percpu_counter_tree.c-923-\n--\nlib/percpu_counter_tree.c=1041=EXPORT_SYMBOL_GPL(percpu_counter_tree_approximate_compare_value);\n--\nlib/percpu_counter_tree.c-1053- * precise sums can count updates in part, see\nlib/percpu_counter_tree.c:1054: * percpu_counter_tree_precise_sum().\nlib/percpu_counter_tree.c-1055- * The comparison between counters is on the wrapped difference. It is valid\n--\nlib/percpu_counter_tree.c=1064=int percpu_counter_tree_precise_compare(struct percpu_counter_tree *a,\n--\nlib/percpu_counter_tree.c-1084-\tif (accuracy_b \u003c accuracy_a) {\nlib/percpu_counter_tree.c:1085:\t\tcount_a = percpu_counter_tree_precise_sum(a);\nlib/percpu_counter_tree.c-1086-\t\t/* precise_a - approx_b: only b's precise range remains. */\n--\nlib/percpu_counter_tree.c-1090-\t\t/* Precise sum of second counter is required. */\nlib/percpu_counter_tree.c:1091:\t\tcount_b = percpu_counter_tree_precise_sum(b);\nlib/percpu_counter_tree.c-1092-\t} else {\nlib/percpu_counter_tree.c:1093:\t\tcount_b = percpu_counter_tree_precise_sum(b);\nlib/percpu_counter_tree.c-1094-\t\t/* approx_a - precise_b: only a's precise range remains. */\n--\nlib/percpu_counter_tree.c-1098-\t\t/* Precise sum of second counter is required. */\nlib/percpu_counter_tree.c:1099:\t\tcount_a = percpu_counter_tree_precise_sum(a);\nlib/percpu_counter_tree.c-1100-\t}\n--\nlib/percpu_counter_tree.c=1108=EXPORT_SYMBOL_GPL(percpu_counter_tree_precise_compare);\n--\nlib/percpu_counter_tree.c-1120- * With concurrent updates, the precise sum can count updates in part, see\nlib/percpu_counter_tree.c:1121: * percpu_counter_tree_precise_sum().\nlib/percpu_counter_tree.c-1122- * The comparison between counters is on the wrapped difference. It is valid\n--\nlib/percpu_counter_tree.c=1131=int percpu_counter_tree_precise_compare_value(struct percpu_counter_tree *counter, long v)\n--\nlib/percpu_counter_tree.c-1143-\t/* Precise sum is required. */\nlib/percpu_counter_tree.c:1144:\tcount = percpu_counter_tree_precise_sum(counter);\nlib/percpu_counter_tree.c-1145-\tdelta = wrapped_delta(v, count);\n--\nlib/percpu_counter_tree.c=1167=void percpu_counter_tree_set(struct percpu_counter_tree *counter, long v)\nlib/percpu_counter_tree.c-1168-{\nlib/percpu_counter_tree.c:1169:\tWRITE_ONCE(counter-\u003ebias, v - percpu_counter_tree_precise_sum_unbiased(counter));\nlib/percpu_counter_tree.c-1170-}\n--\nlib/tests/percpu_counter_tree_kunit.c=66=static void check_counters(struct kunit *test)\n--\nlib/tests/percpu_counter_tree_kunit.c-78-\t\t/* Precise comparison. */\nlib/tests/percpu_counter_tree_kunit.c:79:\t\tKUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(c), v);\nlib/tests/percpu_counter_tree_kunit.c-80-\t\tcheck_residuals(test, c);\n--\nlib/tests/percpu_counter_tree_kunit.c-104-\t/* Compare each counter with the second counter. */\nlib/tests/percpu_counter_tree_kunit.c:105:\tKUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(\u0026counter[0]),\nlib/tests/percpu_counter_tree_kunit.c:106:\t\t\tpercpu_counter_tree_precise_sum(\u0026counter[1]));\nlib/tests/percpu_counter_tree_kunit.c-107-\tKUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_precise_compare(\u0026counter[0], \u0026counter[1]));\n--\nlib/tests/percpu_counter_tree_kunit.c=525=static void do_hpcc_test_contended_carries(struct kunit *test, unsigned long batch_size)\n--\nlib/tests/percpu_counter_tree_kunit.c-579-\tKUNIT_EXPECT_EQ(test, sh.nr_errors, 0);\nlib/tests/percpu_counter_tree_kunit.c:580:\tKUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(c), expected);\nlib/tests/percpu_counter_tree_kunit.c-581-\tcheck_residuals(test, c);\n--\nlib/tests/percpu_counter_tree_kunit.c=620=static void hpcc_test_set(struct kunit *test)\n--\nlib/tests/percpu_counter_tree_kunit.c-637-\t\tpercpu_counter_tree_set(\u0026pct, v);\nlib/tests/percpu_counter_tree_kunit.c:638:\t\tKUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(\u0026pct), v);\nlib/tests/percpu_counter_tree_kunit.c-639-\t\tKUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare_value(\u0026pct, v));\n--\nlib/tests/percpu_counter_tree_kunit.c-641-\t\tpercpu_counter_tree_add(\u0026pct, v);\nlib/tests/percpu_counter_tree_kunit.c:642:\t\tKUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(\u0026pct), 2 * v);\nlib/tests/percpu_counter_tree_kunit.c-643-\t\tKUNIT_EXPECT_EQ(test, 0,\n--\nlib/tests/percpu_counter_tree_kunit.c-646-\t\tpercpu_counter_tree_add(\u0026pct, -2 * v);\nlib/tests/percpu_counter_tree_kunit.c:647:\t\tKUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(\u0026pct), 0);\nlib/tests/percpu_counter_tree_kunit.c-648-\t\tKUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare_value(\u0026pct, 0));\n--\nlib/tests/percpu_counter_tree_kunit.c=703=static void do_hpcc_test_saturated_compare(struct kunit *test, unsigned long batch_size,\n--\nlib/tests/percpu_counter_tree_kunit.c-730-\nlib/tests/percpu_counter_tree_kunit.c:731:\tprecise = percpu_counter_tree_precise_sum(c);\nlib/tests/percpu_counter_tree_kunit.c-732-\tapprox = percpu_counter_tree_approximate_sum(c);\n--\nlib/tests/percpu_counter_tree_kunit.c=789=static void check_lazy_counter(struct kunit *test, struct percpu_counter_tree *c, long v)\n--\nlib/tests/percpu_counter_tree_kunit.c-793-\nlib/tests/percpu_counter_tree_kunit.c:794:\tKUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(c), v);\nlib/tests/percpu_counter_tree_kunit.c-795-\tKUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_precise_compare_value(c, v));\n--\ntools/testing/hpcc/main.c=183=static void check_counter(struct test_counter *c)\n--\ntools/testing/hpcc/main.c-191-\ntools/testing/hpcc/main.c:192:\tCHECK(percpu_counter_tree_precise_sum(t) == precise);\ntools/testing/hpcc/main.c-193-\tCHECK(!percpu_counter_tree_check_residuals(t));\n"
} |
| 13/2 | 2026/11/03 12:11 | llm | patch-evaluator |
1mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 92035 output: 19 thoughts: 28 Thoughts: Okay, here's my attempt at summarizing that thought process, keeping in mind the user is a kernel expert: **Investigating Memory Management Counter Usage** Alright, let's dive into this. I need to understand the call sites for `get_mm_counter` and `get_mm_counter_sum` within the kernel. It’s a pretty standard approach when debugging memory management issues or trying to understand how specific counters are being used to track memory utilization. I need to find all the places in the kernel code where these functions are actually invoked. This is crucial for tracing the flow of information and pinpointing where these counters are being incremented or read. Depending on the goal, I'm probably looking for possible bottlenecks, unexpected counter changes, or just looking to better understand a specific subsystem's memory management strategy. It’s always the first step for this kind of investigation. Time to get my hands dirty and start searching the source tree! |
| 14/2 | 2026/11/03 12:12 | tool | grepper |
0mArgs:{
"Expression": "get_mm_counter"
}Results: {
"Output": "fs/proc/task_mmu.c=37=void task_mem(struct seq_file *m, struct mm_struct *mm)\n--\nfs/proc/task_mmu.c-41-\nfs/proc/task_mmu.c:42:\tanon = get_mm_counter_sum(mm, MM_ANONPAGES);\nfs/proc/task_mmu.c:43:\tfile = get_mm_counter_sum(mm, MM_FILEPAGES);\nfs/proc/task_mmu.c:44:\tshmem = get_mm_counter_sum(mm, MM_SHMEMPAGES);\nfs/proc/task_mmu.c-45-\n--\nfs/proc/task_mmu.c-64-\nfs/proc/task_mmu.c:65:\tswap = get_mm_counter_sum(mm, MM_SWAPENTS);\nfs/proc/task_mmu.c-66-\tSEQ_PUT_DEC(\"VmPeak:\\t\", hiwater_vm);\n--\nfs/proc/task_mmu.c=94=unsigned long task_statm(struct mm_struct *mm,\n--\nfs/proc/task_mmu.c-97-{\nfs/proc/task_mmu.c:98:\t*shared = get_mm_counter_sum(mm, MM_FILEPAGES) +\nfs/proc/task_mmu.c:99:\t\t\tget_mm_counter_sum(mm, MM_SHMEMPAGES);\nfs/proc/task_mmu.c-100-\t*text = (PAGE_ALIGN(mm-\u003eend_code) - (mm-\u003estart_code \u0026 PAGE_MASK))\n--\nfs/proc/task_mmu.c-102-\t*data = mm-\u003edata_vm + mm-\u003estack_vm;\nfs/proc/task_mmu.c:103:\t*resident = *shared + get_mm_counter_sum(mm, MM_ANONPAGES);\nfs/proc/task_mmu.c-104-\treturn mm-\u003etotal_vm;\n--\ninclude/linux/mm.h=3600=static inline bool get_user_page_fast_only(unsigned long addr,\n--\ninclude/linux/mm.h-3607- */\ninclude/linux/mm.h:3608:static inline unsigned long get_mm_counter(struct mm_struct *mm, int member)\ninclude/linux/mm.h-3609-{\n--\ninclude/linux/mm.h-3612-\ninclude/linux/mm.h:3613:static inline unsigned long get_mm_counter_sum(struct mm_struct *mm, int member)\ninclude/linux/mm.h-3614-{\n--\ninclude/linux/mm.h=3674=static inline unsigned long get_mm_rss(struct mm_struct *mm)\ninclude/linux/mm.h-3675-{\ninclude/linux/mm.h:3676:\treturn get_mm_counter(mm, MM_FILEPAGES) +\ninclude/linux/mm.h:3677:\t\tget_mm_counter(mm, MM_ANONPAGES) +\ninclude/linux/mm.h:3678:\t\tget_mm_counter(mm, MM_SHMEMPAGES);\ninclude/linux/mm.h-3679-}\n--\ninclude/linux/mm.h=3681=static inline unsigned long get_mm_rss_sum(struct mm_struct *mm)\ninclude/linux/mm.h-3682-{\ninclude/linux/mm.h:3683:\treturn get_mm_counter_sum(mm, MM_FILEPAGES) +\ninclude/linux/mm.h:3684:\t\tget_mm_counter_sum(mm, MM_ANONPAGES) +\ninclude/linux/mm.h:3685:\t\tget_mm_counter_sum(mm, MM_SHMEMPAGES);\ninclude/linux/mm.h-3686-}\n--\ninclude/trace/events/kmem.h=427=TRACE_EVENT(rss_stat,\n--\ninclude/trace/events/kmem.h-450-\t\t__entry-\u003emember = member;\ninclude/trace/events/kmem.h:451:\t\t__entry-\u003esize = get_mm_counter(mm, member) \u003c\u003c PAGE_SHIFT;\ninclude/trace/events/kmem.h-452-\t),\n--\ninclude/trace/events/oom.h=76=TRACE_EVENT(mark_victim,\n--\ninclude/trace/events/oom.h-96-\t\t__entry-\u003etotal_vm = PG_COUNT_TO_KB(task-\u003emm-\u003etotal_vm);\ninclude/trace/events/oom.h:97:\t\t__entry-\u003eanon_rss = PG_COUNT_TO_KB(get_mm_counter(task-\u003emm, MM_ANONPAGES));\ninclude/trace/events/oom.h:98:\t\t__entry-\u003efile_rss = PG_COUNT_TO_KB(get_mm_counter(task-\u003emm, MM_FILEPAGES));\ninclude/trace/events/oom.h:99:\t\t__entry-\u003eshmem_rss = PG_COUNT_TO_KB(get_mm_counter(task-\u003emm, MM_SHMEMPAGES));\ninclude/trace/events/oom.h-100-\t\t__entry-\u003euid = uid;\n--\nmm/oom_kill.c=199=long oom_badness(struct task_struct *p, unsigned long totalpages)\n--\nmm/oom_kill.c-227-\t */\nmm/oom_kill.c:228:\tpoints = get_mm_rss_sum(p-\u003emm) + get_mm_counter_sum(p-\u003emm, MM_SWAPENTS) +\nmm/oom_kill.c-229-\t\tmm_pgtables_bytes(p-\u003emm) / PAGE_SIZE;\n--\nmm/oom_kill.c=372=static int dump_task(struct task_struct *p, void *arg)\n--\nmm/oom_kill.c-395-\t\ttask-\u003etgid, task-\u003emm-\u003etotal_vm, get_mm_rss_sum(task-\u003emm),\nmm/oom_kill.c:396:\t\tget_mm_counter_sum(task-\u003emm, MM_ANONPAGES), get_mm_counter_sum(task-\u003emm, MM_FILEPAGES),\nmm/oom_kill.c:397:\t\tget_mm_counter_sum(task-\u003emm, MM_SHMEMPAGES), mm_pgtables_bytes(task-\u003emm),\nmm/oom_kill.c:398:\t\tget_mm_counter_sum(task-\u003emm, MM_SWAPENTS),\nmm/oom_kill.c-399-\t\ttask-\u003esignal-\u003eoom_score_adj, task-\u003ecomm);\n--\nmm/oom_kill.c=555=static bool oom_reap_task_mm(struct task_struct *tsk, struct mm_struct *mm)\n--\nmm/oom_kill.c-583-\t\t\ttask_pid_nr(tsk), tsk-\u003ecomm,\nmm/oom_kill.c:584:\t\t\tK(get_mm_counter_sum(mm, MM_ANONPAGES)),\nmm/oom_kill.c:585:\t\t\tK(get_mm_counter_sum(mm, MM_FILEPAGES)),\nmm/oom_kill.c:586:\t\t\tK(get_mm_counter_sum(mm, MM_SHMEMPAGES)));\nmm/oom_kill.c-587-out_finish:\n--\nmm/oom_kill.c=905=static void __oom_kill_process(struct task_struct *victim, const char *message)\n--\nmm/oom_kill.c-939-\t\tmessage, task_pid_nr(victim), victim-\u003ecomm, K(mm-\u003etotal_vm),\nmm/oom_kill.c:940:\t\tK(get_mm_counter_sum(mm, MM_ANONPAGES)),\nmm/oom_kill.c:941:\t\tK(get_mm_counter_sum(mm, MM_FILEPAGES)),\nmm/oom_kill.c:942:\t\tK(get_mm_counter_sum(mm, MM_SHMEMPAGES)),\nmm/oom_kill.c-943-\t\tfrom_kuid(\u0026init_user_ns, task_uid(victim)),\n"
} |
| 15/2 | 2026/11/03 12:12 | llm | patch-evaluator |
1mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 94362 output: 31 thoughts: 1895 Thoughts: **Focusing on Per-CPU Counter Trees** Okay, so I'm looking at how this per-cpu counter tree stuff works, and a few things are jumping out. First, it's clear that `get_mm_counter_sum()` leads to a call chain that includes `percpu_counter_tree_precise_sum_positive()` and ultimately, `percpu_counter_tree_precise_sum()`. This is also triggered by reading `/proc/<pid>/status` or `/proc/<pid>/statm` which call `task_mem` and `task_statm` which in turn call `get_mm_counter_sum()`! Not to mention, `check_mm()` in `kernel/fork.c` also hits `percpu_counter_tree_precise_sum()`. So, that precise sum is definitely a hot spot. Now, let's figure out which functions are *actually* important to focus on for fuzzing. I need to avoid the generic, ubiquitous lifecycle hot-paths. Here's the breakdown of the functions I've examined: 1. `percpu_counter_tree_activate_many`: This one's interesting, because it's only called when creating threads via `copy_mm()`. It handles allocations, memory barriers, and the initial publication of the tree. This includes allocations, memory barriers, and the act of *publishing* the data. 2. `percpu_counter_tree_add`: This updates the tree, including the logic for carry propagation (`percpu_counter_tree_add_slowpath`), which involves cmpxchg loops and all that fun. 3. `percpu_counter_tree_precise_sum`: I've already noted this one. It's used in `/proc/pid/statm`, `/proc/pid/status`, and the OOM killer, and it reads across memory barriers. 4. `percpu_counter_tree_approximate_compare`: An approximate comparison for the counter trees. 5. `percpu_counter_tree_precise_compare`: A more precise comparison approach. 6. `percpu_counter_tree_set`: Sets a counter tree's value. 7. `percpu_counter_tree_destroy_many`: This is the cleanup function that frees percpu and tree items. Now, hold on. What about `percpu_counter_tree_init_many_lazy`? Ah, right, that's the constructor for *lazy* counter trees. And is `percpu_counter_tree_add_owner` something to include? It is, but it appears in a lot of code, and that's precisely the sort of ubiquitous function I want to avoid. It seems to be called on every single page fault/mmap/unmap for single-threaded processes. The instruction's advice rings true: if it's called on every program in syzkaller, the fuzzer gets diluted. The actual code within `percpu_counter_tree_add_owner` is very simple, a potential fast path, but *still* ubiquitous. So, *really*, if I'm trying to find more unique bugs, I'd exclude it. But `percpu_counter_tree_activate_many` and `percpu_counter_tree_add` are the money shots! `percpu_counter_tree_activate_many` only happens with thread creation. Subsequent updates then go through the `percpu_counter_tree_add()`, which then goes through the hierarchy, `percpu_counter_tree_add_tree()` and `percpu_counter_tree_add_slowpath()`. Also, I have to remember that `percpu_counter_tree_init_many_lazy` is called during `mm_init()`. But wait, doesn't `mm_init()` run for every process creation, either through `fork()` (via `dup_mm` or `copy_mm`), or `execve()` (via `exec_mmap`)? So, `percpu_counter_tree_init_many_lazy` is called on almost *every* process! It's initialization though - it's trivial: it just sets values to zero/null. The areas most likely to have bugs are: 1. `percpu_counter_tree_activate_many`: Race conditions between updaters and activators. 2. `percpu_counter_tree_add`: Tree propagation, cmpxchg loop, atomic operations. 3. `percpu_counter_tree_precise_sum`: Memory barriers and iteration across the tree. 4. `percpu_counter_tree_destroy_many`: Freeing per |
| 16/2 | 2026/11/03 12:14 | tool | grepper |
0mArgs:{
"Expression": "PERCPU_COUNTER_TREE",
"PathPrefixes": [
"lib/Kconfig"
]
}Results: {
"Output": "lib/Kconfig=32=config PACKING_KUNIT_TEST\n--\nlib/Kconfig-43-\nlib/Kconfig:44:config PERCPU_COUNTER_TREE_KUNIT_TEST\nlib/Kconfig-45-\ttristate \"KUnit tests for hierarchical per-cpu counters\" if !KUNIT_ALL_TESTS\n--\nlib/Kconfig-55-\nlib/Kconfig:56:config PERCPU_COUNTER_TREE_BENCH\nlib/Kconfig-57-\ttristate \"Benchmark of hierarchical per-cpu counters\"\n"
} |
| 17/2 | 2026/11/03 12:14 | llm | patch-evaluator |
1mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 97459 output: 301 thoughts: 872 Thoughts: **Fuzzing Strategy for Hierarchical Per-CPU Counters (HPCC)**
Okay, let's think this through. `PERCPU_COUNTER_TREE_KUNIT_TEST` and `PERCPU_COUNTER_TREE_BENCH` are just testing and benchmarking tools. The real meat is in `lib/percpu_counter_tree.o`, compiled when `CONFIG_SMP` is enabled. So, `SMP` is definitely the key configuration option here.
Now, let's break down the fuzzing strategy systematically.
1. **Is it Worth Fuzzing?** Absolutely! This patch series introduces a new synchronization data structure, HPCC, and fundamentally changes how mm RSS counters are handled. This touches core memory management, fork/exec/exit lifecycle, task creation, lockless memory ordering, and concurrent atomic cmpxchg operations across CPUs. It's directly linked to userspace syscalls like `clone`, `fork`, `execve`, `exit`, `mmap`, `munmap`, page faults, and reading `/proc/$PID/status`. This is prime territory for syzkaller, especially given its strength in multi-threaded workloads. Concurrency bugs, race conditions in lazy tree activation, carry calculation issues, and invariants/assertions in `check_mm` are all likely candidates for discovery.
2. **EnableConfigs:** Clearly, we need `SMP`. `lib/percpu_counter_tree.c` is only compiled if `CONFIG_SMP` is set; otherwise, it degrades to a simple atomic integer.
3. **FocusSymbols:** Okay, let's identify the core functions for fuzzing. I'm going to select the key, less ubiquitous entrypoints and logic functions:
* `percpu_counter_tree_activate_many`: Lazy activation of the counter tree when threads share an mm.
* `percpu_counter_tree_add`: Updates the counter tree with carry propagation and cmpxchg across hierarchy levels.
* `percpu_counter_tree_precise_sum`: Reads tree levels and per-CPU data with memory barriers.
* `percpu_counter_tree_destroy_many`: Tears down tree items and per-CPU allocations.
* `percpu_counter_tree_precise_compare`: Executes a two-pass approximate/precise comparison algorithm.
* `percpu_counter_tree_approximate_compare`: Approximate comparison of counter trees.
* `percpu_counter_tree_set`: Resets counter tree biased values.
I'm purposefully omitting `percpu_counter_tree_init_many_lazy` and `percpu_counter_tree_add_owner` because those are called on virtually every single-threaded process fork/exec and memory fault. That would dilute the fuzzing across the entire corpus. Also skipping `percpu_counter_tree_subsystem_init` because it's only called once at boot time.
Let me just double-check those symbol names in `lib/percpu_counter_tree.c` to be sure:
* `percpu_counter_tree_activate_many` (line 592)
* `percpu_counter_tree_add` (line 813)
* `percpu_counter_tree_precise_sum` (line 918)
* `percpu_counter_tree_destroy_many` (line 640)
* `percpu_counter_tree_precise_compare` (line 1064)
* `percpu_counter_tree_approximate_compare` (line 1001)
* `percpu_counter_tree_set` (line 1167)
Are these symbols exported without static qualification? Yes, they're all `EXPORT_SYMBOL_GPL(...)`.
Okay, I think I've got a solid plan. Let's configure the `set-results` tool and start fuzzing.
|
| 18/2 | 2026/11/03 12:15 | tool | set-results |
0mArgs:{
"EnableConfigs": [
"SMP"
],
"FocusSymbols": [
"percpu_counter_tree_activate_many",
"percpu_counter_tree_add",
"percpu_counter_tree_precise_sum",
"percpu_counter_tree_destroy_many",
"percpu_counter_tree_precise_compare",
"percpu_counter_tree_approximate_compare",
"percpu_counter_tree_set"
],
"Reasoning": "The patch series implements Hierarchical Per-CPU Counters (HPCC) and transitions the kernel's mm RSS statistics (mm-\u003erss_stat) to use this new data structure. This modifies core subsystems (mm, fork, exec, exit) and introduces lockless tree propagation, atomic carry cmpxchg loops, and lazy activation when processes create threads with CLONE_VM. The modified code is directly reachable by user space via common syscalls (clone, fork, exit, execve, mmap, munmap, page faults, and /proc/\u003cpid\u003e/statm reading) and is an excellent target for fuzzing race conditions and concurrency issues. Ubiquitous single-threaded hot-paths (such as percpu_counter_tree_add_owner and percpu_counter_tree_init_many_lazy) are excluded from FocusSymbols to prevent coverage dilution.",
"WorthFuzzing": true
}Results: {
"EnableConfigs": [
"SMP"
],
"FocusSymbols": [
"percpu_counter_tree_activate_many",
"percpu_counter_tree_add",
"percpu_counter_tree_precise_sum",
"percpu_counter_tree_destroy_many",
"percpu_counter_tree_precise_compare",
"percpu_counter_tree_approximate_compare",
"percpu_counter_tree_set"
],
"Reasoning": "The patch series implements Hierarchical Per-CPU Counters (HPCC) and transitions the kernel's mm RSS statistics (mm-\u003erss_stat) to use this new data structure. This modifies core subsystems (mm, fork, exec, exit) and introduces lockless tree propagation, atomic carry cmpxchg loops, and lazy activation when processes create threads with CLONE_VM. The modified code is directly reachable by user space via common syscalls (clone, fork, exit, execve, mmap, munmap, page faults, and /proc/\u003cpid\u003e/statm reading) and is an excellent target for fuzzing race conditions and concurrency issues. Ubiquitous single-threaded hot-paths (such as percpu_counter_tree_add_owner and percpu_counter_tree_init_many_lazy) are excluded from FocusSymbols to prevent coverage dilution.",
"WorthFuzzing": true
} |
| 19/1 | 2026/11/03 12:15 | action | If |
22mArgs:{
"WorthFuzzing": true
}Results: null |
| 20/2 | 2026/11/03 12:15 | agent | kmsan-evaluator |
22mModel:core Args: null Results: {
"KMSANReasoning": "The patch series introduces Hierarchical Per-CPU Counters (HPCC, implemented via `struct percpu_counter_tree`) and replaces `struct percpu_counter` with HPCC for memory management RSS statistics (`mm-\u003erss_stat`).\n\nKMSAN vs KASAN Applicability Analysis:\n1. Memory Allocation and Initialization:\n - Dynamic allocations for tree items use `kzalloc()` or `kmem_cache_zalloc()`, ensuring full zero-initialization of tree buffers.\n - Per-CPU allocations (`__alloc_percpu_gfp()`) in Linux are zeroed upon allocation.\n - Struct fields for `percpu_counter_tree` are systematically initialized in `__percpu_counter_tree_init()` (and lazy initialization explicitly sets pointers to NULL and biases to 0).\n - `struct mm_struct` additions (`rss_stat`, `rss_owner`, `rss_stat_mutex`) are initialized in `mm_init()` via `memset` and mutex initialization.\n - No complex data structures are left uninitialized or partially initialized.\n\n2. User-space Boundary and Info-leaks:\n - No raw kernel structures or buffers with potential padding bytes are copied to user space (`copy_to_user`, ioctl, netlink, BPF, etc.).\n - The RSS values derived from the tree counters are plain numerical values formatted into strings (e.g., in `/proc/\u003cpid\u003e/stat` via `seq_put_decimal_ull()`) or logged in trace events.\n\n3. Bug Profiles:\n - The primary risks of this patch series involve concurrency, memory ordering (store-release/load-acquire barriers between levels and during activation), synchronization (`rss_stat_mutex`), arithmetic carry logic, or potential out-of-bounds indexing in tree item strides.\n - Out-of-bounds indexing and use-after-free bugs are detected by KASAN. Locking deadlocks are caught by LOCKDEP. KMSAN is specifically designed to detect reads of uninitialized memory and information leaks, neither of which are risks introduced by this patch.\n\nTherefore, a dedicated KMSAN fuzzing session is not warranted.",
"NeedsKMSAN": false
}Instruction: You are an expert Linux kernel security engineer specializing in kernel memory
error detectors (KASAN and KMSAN). Your job is to review the provided patch series and
determine if the code changes justify spawning a dedicated KMSAN (KernelMemorySanitizer)
fuzzing session in addition to standard KASAN fuzzing.
CRITICAL DISTINCTION BETWEEN KASAN AND KMSAN:
- Standard KASAN kernel builds (upstream-apparmor-kasan.config) already enable
a comprehensive suite of debugging tools and sanitizers, including KASAN
(out-of-bounds accesses, use-after-free, double free, invalid free), LOCKDEP
(locking bugs and deadlocks), UB-sanitizers, and memory corruption checks.
- KMSAN (KernelMemorySanitizer) detects reads of UNINITIALIZED memory (stack, heap,
or page allocations) and kernel-to-user memory info-leaks.
Rule: THERE IS NO SENSE IN RUNNING A KMSAN SESSION IF A BUG CAN BE CAUGHT BY KASAN,
LOCKDEP, OR OTHER STANDARD BUG DETECTORS.
A dedicated KMSAN fuzzing session incurs significant resource costs. You must ONLY
set NeedsKMSAN=true if the code changes introduce or expose UNINITIALIZED MEMORY risks
that are detected ONLY by KMSAN.
Look holistically at the patch series and surrounding code. Even if no direct
uninitialized field accesses or new buffer allocations are added in the diff itself,
a patch may alter control flow, bounds checking, or data length calculations in ways
that change how the rest of the code operates on existing buffers (e.g. allowing
uninitialized stack/heap memory to be read, copied to user space, or used in control
flow). Do not hesitate to use your code access tools to inspect the surrounding code,
called functions, and callers.
Set NeedsKMSAN=true ONLY IF the patch introduces or modifies:
1. Kernel structures sent to user space (via copy_to_user, put_user, netlink skb
attributes, ioctl output arguments, socket options, or BPF buffers) where fields
or structure padding might not be fully initialized/zeroed.
2. Conditional logic or branching that depends on potentially uninitialized variables
or struct fields.
3. Allocation or initialization of complex data structures where uninitialized fields
could be read later in reachable code paths.
4. Bounds checks, lengths, or logic in a way that allows surrounding code to access
uninitialized bytes of existing buffers.
Set NeedsKMSAN=false IF:
- The code changes primarily risk out-of-bounds access, array overflows, NULL pointer
dereferences, locking deadlocks, or use-after-free bugs (these are already caught
by KASAN, LOCKDEP, or standard bug detectors).
- All stack/heap structures touched or introduced by the patch are fully zeroed
or initialized (e.g. using = {0}, memset, kzalloc) before being read or copied.
- The patch does not introduce any risk of uninitialized memory usage or info-leaks.
Use your code access tools to inspect the surrounding code if necessary, then provide
detailed KMSANReasoning contrasting KASAN vs KMSAN applicability for this patch.
Prefer calling several tools at the same time to save round-trips.
Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.
Prompt: Target architecture: amd64
For your convenience, here is the diff of the changes:
commit d5aca8d49d662c298b87359f285e9556d3b2d61a
Author: syz-cluster <triage@syzkaller.com>
Date: Tue Nov 3 12:05:07 2026 +0000
syz-cluster: applied patch under review
diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 0b3c140babd944..ba2bdeb1480b7f 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -5342,6 +5342,20 @@ Kernel parameters
allocator. This parameter is primarily for debugging
and performance comparison.
+ percpu_counter_tree.arity= [KNL,SMP]
+ Format: <fan-out>[,<fan-out>...]
+ Override the shape of the hierarchical per-cpu
+ counter trees (used for the RSS counters), listing
+ the fan-out of each tree level from the per-cpu
+ counters to the root. Each fan-out is a power of 2
+ and their product must be nr_cpu_ids (the highest
+ possible CPU number plus one) rounded up to a power
+ of 2, e.g. "16" (flat) or "4,4" when nr_cpu_ids
+ is 16. An invalid shape is ignored with a warning.
+ The shape in use is shown in
+ /sys/module/percpu_counter_tree/parameters/arity.
+ This parameter is meant for performance comparison.
+
pirq= [SMP,APIC] Manual mp-table setup
See Documentation/arch/x86/i386/IO-APIC.rst.
@@ -6778,6 +6792,16 @@ Kernel parameters
debug mode. Defaults to CONFIG_RSEQ_DEBUG_DEFAULT_ENABLE.
Format: <bool>
+ rss_stat_batch= [KNL,SMP]
+ Format: <integer>
+ Batch size of the per-mm RSS counters: the number of
+ pages a CPU accumulates before propagating them into
+ the counter tree. A power of 2 between 2 and 128.
+ A smaller batch improves the accuracy of the
+ approximate RSS, a larger batch reduces the update
+ cost. Default: 32.
+ This parameter is meant for performance comparison.
+
rt_group_sched= [KNL] Enable or disable SCHED_RR/FIFO group scheduling
when CONFIG_RT_GROUP_SCHED=y. Defaults to
!CONFIG_RT_GROUP_SCHED_DEFAULT_DISABLED.
diff --git a/Documentation/core-api/index.rst b/Documentation/core-api/index.rst
index 92f91c6a0d79d6..fe03cefc17348a 100644
--- a/Documentation/core-api/index.rst
+++ b/Documentation/core-api/index.rst
@@ -48,6 +48,7 @@ Library functionality that is used throughout the kernel.
generic-radix-tree
packing
this_cpu_ops
+ percpu-counter-tree
timekeeping
errseq
wrappers/atomic_t
diff --git a/Documentation/core-api/percpu-counter-tree.rst b/Documentation/core-api/percpu-counter-tree.rst
new file mode 100644
index 00000000000000..8aebf91aa133d7
--- /dev/null
+++ b/Documentation/core-api/percpu-counter-tree.rst
@@ -0,0 +1,145 @@
+.. SPDX-License-Identifier: GPL-2.0+ OR MIT
+
+========================================
+The Hierarchical Per-CPU Counters (HPCC)
+========================================
+
+:Author: Mathieu Desnoyers
+
+Introduction
+============
+
+Counters come in many varieties, each with their own trade offs:
+
+ * A global atomic counter provides a fast read access to the current
+ sum, at the expense of cache-line bouncing on updates. This leads to
+ poor performance of frequent updates from various cores on large SMP
+ systems.
+
+ * A per-cpu split counter provides fast updates to per-cpu counters,
+ at the expense of a slower aggregation (sum). The sum operation needs
+ to iterate over all per-cpu counters to calculate the current total.
+
+The hierarchical per-cpu counters attempt to provide the best of both
+worlds (fast updates, and fast sum) by relaxing requirements on the sum
+accuracy. It allows quickly querying an approximated sum value, along
+with the possible min/max ranges of the associated precise sum. The
+exact precise sum can still be calculated with an iteration on all
+per-cpu counters, but the availability of an approximated sum value with
+possible precise sum min/max ranges allows eliminating candidates which
+are certainly outside of a known target range without the overhead of
+precise sums.
+
+Overview
+========
+
+The hierarchical per-cpu counters are organized as a tree with the
+per-cpu counters as its first level and the tree root as its last
+level.
+
+The intermediate tree levels contain carry propagation counters. When
+reaching a threshold (batch size), the carry is propagated up the
+tree.
+
+Each counter below the root, from the per-cpu counters up, keeps the
+residual it has not propagated yet, and only propagates once this
+residual reaches its batch size in either direction. This hysteresis
+prevents updates going back and forth around a batch size boundary from
+propagating further up the tree each time.
+
+This allows reading an approximated value at the root, which has a
+bounded accuracy determined by the tree topology and the batch size:
+the precise sum is within the same distance of the approximation in
+both directions. This bound grows as O(nr_cpus * log(nr_cpus)).
+
+The per-cpu counters are bytes, which limits the batch size to 128. The
+tree positions are cache line aligned to prevent false sharing between
+CPUs updating different branches of the tree. A tree position holds the
+items of all the counter trees initialized together by
+percpu_counter_tree_init_many(), so a set of counters updated together,
+such as the RSS counters of an mm, shares cache lines.
+percpu_counter_tree_init_many() allocates the per-cpu counters, one per
+possible CPU, and the tree items, whose number depends on nr_cpu_ids, so
+struct percpu_counter_tree has a fixed size.
+
+The fan-out of each tree level is selected at boot from nr_cpu_ids, the
+highest possible CPU number plus one. It can be overridden with the
+percpu_counter_tree.arity= boot parameter for performance comparison.
+
+The bounded accuracy is a property of the settled tree. While updates are
+in flight the transient error can exceed it, bounded by the number of
+updaters and their increment size.
+
+Likewise, the precise sum is exact when no update runs concurrently. It
+takes no lock: an update which carries from the per-cpu counter to the
+upper levels can be counted in part, the residual it left below read and
+its carry missed on the way up.
+
+Lazy Tree Activation
+====================
+
+Counters initialized with percpu_counter_tree_init_many_lazy() start
+without their per-cpu counters and tree items, for counters which may
+never need the scalability of the tree. Until
+percpu_counter_tree_activate_many() allocates and publishes the tree,
+the counters are exact:
+
+ * percpu_counter_tree_add_owner() adds to the bias of the counter,
+ written only by its owner, a single context chosen by the user of the
+ counters, such as a task, without atomic instruction.
+
+ * percpu_counter_tree_add() adds to the atomic bias of the counter, for
+ any other updater.
+
+The precise and approximate sums read both, in O(1), and count each
+update whole. The activation does not move these values: the sums keep
+reading them along with the tree, so updates racing with the activation
+are not lost, whichever side of it they land on. The bias only has the
+owner as writer, and the atomic bias only receives atomic additions,
+before and after the activation. After activation, both update functions
+add to the tree, and the approximation has the accuracy of the tree.
+
+The activations of the same counters must be serialized by the caller:
+a concurrent activation would publish a second tree and lose the
+updates made to the first.
+
+The RSS counters of an mm use this: the task owning the mm updates the
+plain counter, and other tasks, such as reclaim, update the atomic
+counter. The tree is activated when a second thread shares the mm,
+under mm->rss_stat_mutex, which serializes the activations.
+A single-threaded process therefore has exact RSS counters, cheap
+precise sums, and neither per-cpu memory nor tree items for them.
+
+Use Cases
+=========
+
+Use cases HPCC is meant to handle involve tracking resources which are
+used across many CPUs to quickly sum as feedback for decision making to
+apply throttling, quota limits, sort tasks, and perform memory or task
+migration decisions. When considering approximated sums within the
+accuracy range of the decision threshold, the user can either:
+
+ * Be conservative and fast: Consider that the sum has reached the
+ limit as soon as the given limit is within the approximation range.
+
+ * Be aggressive and fast: Consider that the sum is over the
+ limit only when the approximation range is over the given limit.
+
+ * Be precise and slow: Do a precise comparison with the limit, which
+ requires a precise sum when the limit is within the approximated
+ range.
+
+One use-case for these hierarchical counters is to implement a two-pass
+algorithm to speed up picking a maximum/minimum sum value from
+a set. A first pass compares the approximated values, and then a second
+pass only needs the precise sum for counter trees which are within the
+possible precise sum range of the counter tree chosen by the first pass.
+
+Because the comparison is on the wrapped difference, it is not transitive:
+ordering is undefined over a set of counters spanning more than LONG_MAX.
+
+Functions and structures
+========================
+
+.. kernel-doc:: include/linux/percpu_counter_tree.h
+.. kernel-doc:: lib/percpu_counter_tree.c
diff --git a/MAINTAINERS b/MAINTAINERS
index 93abbb315065de..3878203de1c30f 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -11947,6 +11947,18 @@ L: linux-input@vger.kernel.org
S: Maintained
F: drivers/hid/hid-logitech-hidpp.c
+HIERARCHICAL PER-CPU COUNTERS (HPCC)
+M: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
+L: linux-mm@kvack.org
+L: linux-kernel@vger.kernel.org
+S: Supported
+F: Documentation/core-api/percpu-counter-tree.rst
+F: include/linux/percpu_counter_tree.h
+F: lib/percpu_counter_tree.c
+F: lib/tests/percpu_counter_tree_bench.c
+F: lib/tests/percpu_counter_tree_kunit.c
+F: tools/testing/hpcc/
+
HIGH-RESOLUTION TIMERS, TIMER WHEEL, CLOCKEVENTS
M: Anna-Maria Behnsen <anna-maria@linutronix.de>
M: Frederic Weisbecker <frederic@kernel.org>
diff --git a/fs/exec.c b/fs/exec.c
index 8d2b14e1d455cf..7a6f07a6d482fc 100644
--- a/fs/exec.c
+++ b/fs/exec.c
@@ -875,6 +875,8 @@ static int exec_mmap(struct linux_binprm *bprm)
}
}
+ /* This task owns the RSS counters of the new mm, see __add_mm_counter(). */
+ WRITE_ONCE(mm->rss_owner, tsk);
task_lock(tsk);
membarrier_exec_mmap(mm);
@@ -903,6 +905,9 @@ static int exec_mmap(struct linux_binprm *bprm)
if (old_mm) {
mmap_read_unlock(old_mm);
BUG_ON(active_mm != old_mm);
+ /* Updates of the old mm by this task are now atomic. */
+ if (READ_ONCE(old_mm->rss_owner) == tsk)
+ WRITE_ONCE(old_mm->rss_owner, NULL);
/* Defer teardown to setup_new_exec(), outside the exec locks. */
bprm->old_mm = old_mm;
} else {
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 9f167950926a15..4d740c0c5f5481 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -3607,33 +3607,51 @@ static inline bool get_user_page_fast_only(unsigned long addr,
*/
static inline unsigned long get_mm_counter(struct mm_struct *mm, int member)
{
- return percpu_counter_read_positive(&mm->rss_stat[member]);
+ return percpu_counter_tree_approximate_sum_positive(&mm->rss_stat[member]);
}
static inline unsigned long get_mm_counter_sum(struct mm_struct *mm, int member)
{
- return percpu_counter_sum_positive(&mm->rss_stat[member]);
+ return percpu_counter_tree_precise_sum_positive(&mm->rss_stat[member]);
}
void mm_trace_rss_stat(struct mm_struct *mm, int member);
+/*
+ * Until a second thread shares the mm, the RSS counters are exact single
+ * counters: the task which owns the mm updates a plain counter, and other
+ * tasks (reclaim, khugepaged, the OOM reaper, remote GUP, ...) an atomic
+ * one. See copy_mm(). Once the tree is active, all updates go to it:
+ * rss_owner is not read, so the updates of the threads of a process do not
+ * touch its cache line.
+ */
+static inline void __add_mm_counter(struct mm_struct *mm, int member, long value)
+{
+ struct percpu_counter_tree *counter = &mm->rss_stat[member];
+
+ if (!percpu_counter_tree_active(counter) && current == READ_ONCE(mm->rss_owner))
+ percpu_counter_tree_add_owner(counter, value);
+ else
+ percpu_counter_tree_add(counter, value);
+}
+
static inline void add_mm_counter(struct mm_struct *mm, int member, long value)
{
- percpu_counter_add(&mm->rss_stat[member], value);
+ __add_mm_counter(mm, member, value);
mm_trace_rss_stat(mm, member);
}
static inline void inc_mm_counter(struct mm_struct *mm, int member)
{
- percpu_counter_inc(&mm->rss_stat[member]);
+ __add_mm_counter(mm, member, 1);
mm_trace_rss_stat(mm, member);
}
static inline void dec_mm_counter(struct mm_struct *mm, int member)
{
- percpu_counter_dec(&mm->rss_stat[member]);
+ __add_mm_counter(mm, member, -1);
mm_trace_rss_stat(mm, member);
}
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index 1ed35d258d2e36..f0d315d72adb02 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -19,6 +19,7 @@
#include <linux/workqueue.h>
#include <linux/seqlock.h>
#include <linux/percpu_counter.h>
+#include <linux/percpu_counter_tree.h>
#include <linux/types.h>
#include <linux/futex_types.h>
#include <linux/rseq_types.h>
@@ -1350,7 +1351,23 @@ struct mm_struct {
unsigned long saved_e_flags;
#endif
- struct percpu_counter rss_stat[NR_MM_COUNTERS];
+ struct percpu_counter_tree rss_stat[NR_MM_COUNTERS];
+ /*
+ * Task updating rss_stat with percpu_counter_tree_add_owner()
+ * until the counter tree is activated by a second thread: the
+ * task running on this mm, set by dup_mm() and exec_mmap(),
+ * cleared when that task exits or execs, NULL for the other
+ * users of mm_alloc(). Only compared with current, never
+ * dereferenced. Not mm->owner, which depends on CONFIG_MEMCG
+ * and which mm_update_next_owner() hands over to another task
+ * while a vfork() child may still update the counters.
+ */
+ struct task_struct *rss_owner;
+ /*
+ * Serializes the activations of the rss_stat counter tree,
+ * until the tree is active. See mm_activate_rss_stat().
+ */
+ struct mutex rss_stat_mutex;
struct linux_binfmt *binfmt;
diff --git a/include/linux/percpu_counter_tree.h b/include/linux/percpu_counter_tree.h
new file mode 100644
index 00000000000000..60ef1da22bd3fd
--- /dev/null
+++ b/include/linux/percpu_counter_tree.h
@@ -0,0 +1,365 @@
+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */
+/* SPDX-FileCopyrightText: 2025-2026 Mathieu Desnoyers <mathieu.desnoyers@efficios.com> */
+
+#ifndef _PERCPU_COUNTER_TREE_H
+#define _PERCPU_COUNTER_TREE_H
+
+#include <linux/bug.h>
+#include <linux/atomic.h>
+#include <linux/limits.h>
+#include <linux/percpu.h>
+
+/* Largest batch size: level 0 residuals are within (-batch, +batch) of a signed byte. */
+#define PERCPU_COUNTER_TREE_MAX_BATCH 128
+static_assert(PERCPU_COUNTER_TREE_MAX_BATCH - 1 <= S8_MAX);
+
+#ifdef CONFIG_SMP
+
+struct percpu_counter_tree {
+ /*
+ * Per-CPU split counters (tree level 0): residuals not propagated
+ * yet, within (-batch, +batch).
+ */
+ s8 __percpu *level0;
+ /*
+ * The batch size is the increment step at level 0 which triggers a
+ * carry propagation: a power of 2 between 2 and
+ * PERCPU_COUNTER_TREE_MAX_BATCH. It is stored next to level0, the
+ * other field read by the update fast path, so both usually share a
+ * cache line: this leaves a hole before items_stride.
+ */
+ unsigned char batch_order;
+ unsigned int items_stride; /* Distance between this counter's consecutive items. */
+ /*
+ * Approximate sum: this counter's item of the root, in the tree
+ * items allocated with the per-CPU counters. NULL until the tree of a
+ * counter initialized by percpu_counter_tree_init_many_lazy() is
+ * activated.
+ */
+ atomic_long_t *approx_sum;
+ /*
+ * Bias to apply to counter precise and approximate values. Before the
+ * tree is activated, it also accumulates the updates of
+ * percpu_counter_tree_add_owner(), written only by the owner.
+ */
+ long bias;
+ /*
+ * Updates of percpu_counter_tree_add() made before the tree is
+ * activated, also applied to the precise and approximate values.
+ */
+ atomic_long_t atomic_bias;
+};
+
+int percpu_counter_tree_init_many(struct percpu_counter_tree *counters,
+ unsigned int nr_counters, unsigned long batch_size,
+ gfp_t gfp_flags);
+int percpu_counter_tree_init(struct percpu_counter_tree *counter,
+ unsigned long batch_size, gfp_t gfp_flags);
+int percpu_counter_tree_init_many_lazy(struct percpu_counter_tree *counters,
+ unsigned int nr_counters, unsigned long batch_size);
+int percpu_counter_tree_activate_many(struct percpu_counter_tree *counters,
+ unsigned int nr_counters, gfp_t gfp_flags);
+void percpu_counter_tree_destroy_many(struct percpu_counter_tree *counters,
+ unsigned int nr_counters);
+void percpu_counter_tree_destroy(struct percpu_counter_tree *counter);
+void percpu_counter_tree_add(struct percpu_counter_tree *counter, long inc);
+void percpu_counter_tree_add_owner(struct percpu_counter_tree *counter, long inc);
+long percpu_counter_tree_precise_sum(struct percpu_counter_tree *counter);
+int percpu_counter_tree_approximate_compare(struct percpu_counter_tree *a,
+ struct percpu_counter_tree *b);
+int percpu_counter_tree_approximate_compare_value(struct percpu_counter_tree *counter, long v);
+int percpu_counter_tree_precise_compare(struct percpu_counter_tree *a,
+ struct percpu_counter_tree *b);
+int percpu_counter_tree_precise_compare_value(struct percpu_counter_tree *counter, long v);
+void percpu_counter_tree_set(struct percpu_counter_tree *counter, long v);
+int percpu_counter_tree_subsystem_init(void);
+
+/**
+ * percpu_counter_tree_active() - Query whether the tree of a counter is active.
+ * @counter: The counter to query.
+ *
+ * A counter initialized by percpu_counter_tree_init_many() is active. A
+ * counter initialized by percpu_counter_tree_init_many_lazy() becomes
+ * active when percpu_counter_tree_activate_many() publishes its tree, and
+ * stays active. Once active, percpu_counter_tree_add() and
+ * percpu_counter_tree_add_owner() both add to the tree, so a user can skip
+ * selecting between them.
+ *
+ * Return: true if the updates of @counter go to its tree.
+ */
+static inline
+bool percpu_counter_tree_active(struct percpu_counter_tree *counter)
+{
+ /* No ordering: the update functions and the sums read level0 again. */
+ return !!READ_ONCE(counter->level0);
+}
+
+/**
+ * percpu_counter_tree_approximate_sum() - Return approximate counter sum.
+ * @counter: The counter to sum.
+ *
+ * Querying the approximate sum is fast, but it is only accurate within
+ * the bound returned by percpu_counter_tree_approximate_accuracy().
+ * This is meant to be used when speed is preferred over accuracy.
+ *
+ * Return: The current approximate counter sum.
+ */
+static inline
+long percpu_counter_tree_approximate_sum(struct percpu_counter_tree *counter)
+{
+ /*
+ * Pairs with the release of approx_sum in
+ * percpu_counter_tree_activate_many(): the root is zeroed before its
+ * pointer, and the address dependency orders the read of the root
+ * after it, as for rcu_dereference().
+ */
+ atomic_long_t *approx_sum = READ_ONCE(counter->approx_sum);
+ unsigned long v = (unsigned long)READ_ONCE(counter->bias) +
+ (unsigned long)atomic_long_read(&counter->atomic_bias);
+
+ if (approx_sum)
+ v += (unsigned long)atomic_long_read(approx_sum);
+ return (long)v;
+}
+
+unsigned long percpu_counter_tree_approximate_accuracy(struct percpu_counter_tree *counter);
+
+#if IS_ENABLED(CONFIG_KUNIT)
+unsigned int percpu_counter_tree_check_config(void);
+unsigned int percpu_counter_tree_check_residuals(struct percpu_counter_tree *counter);
+void percpu_counter_tree_copy_state(struct percpu_counter_tree *dst,
+ struct percpu_counter_tree *src);
+#endif
+
+#else /* !CONFIG_SMP */
+
+struct percpu_counter_tree {
+ atomic_long_t count;
+};
+
+static inline
+int percpu_counter_tree_init_many(struct percpu_counter_tree *counters,
+ unsigned int nr_counters, unsigned long batch_size,
+ gfp_t gfp_flags)
+{
+ /*
+ * There is no tree to batch into on UP, so @batch_size has no
+ * effect. Validate it anyway so callers get the same result as
+ * on SMP.
+ */
+ if (WARN_ON(batch_size <= 1 || (batch_size & (batch_size - 1)) ||
+ batch_size > PERCPU_COUNTER_TREE_MAX_BATCH))
+ return -EINVAL;
+ for (unsigned int i = 0; i < nr_counters; i++)
+ atomic_long_set(&counters[i].count, 0);
+ return 0;
+}
+
+static inline
+int percpu_counter_tree_init(struct percpu_counter_tree *counter,
+ unsigned long batch_size, gfp_t gfp_flags)
+{
+ return percpu_counter_tree_init_many(counter, 1, batch_size, gfp_flags);
+}
+
+static inline
+int percpu_counter_tree_init_many_lazy(struct percpu_counter_tree *counters,
+ unsigned int nr_counters, unsigned long batch_size)
+{
+ return percpu_counter_tree_init_many(counters, nr_counters, batch_size, 0);
+}
+
+static inline
+int percpu_counter_tree_activate_many(struct percpu_counter_tree *counters,
+ unsigned int nr_counters, gfp_t gfp_flags)
+{
+ return 0;
+}
+
+static inline
+bool percpu_counter_tree_active(struct percpu_counter_tree *counter)
+{
+ return true;
+}
+
+static inline
+void percpu_counter_tree_destroy_many(struct percpu_counter_tree *counters,
+ unsigned int nr_counters)
+{
+}
+
+static inline
+void percpu_counter_tree_destroy(struct percpu_counter_tree *counter)
+{
+}
+
+static inline
+long percpu_counter_tree_precise_sum(struct percpu_counter_tree *counter)
+{
+ return atomic_long_read(&counter->count);
+}
+
+static inline
+int percpu_counter_tree_precise_compare(struct percpu_counter_tree *a,
+ struct percpu_counter_tree *b)
+{
+ /* Compare on the wrapped difference, as on SMP. */
+ long delta = (long)((unsigned long)percpu_counter_tree_precise_sum(a) -
+ (unsigned long)percpu_counter_tree_precise_sum(b));
+
+ if (delta < 0)
+ return -1;
+ if (delta > 0)
+ return 1;
+ return 0;
+}
+
+static inline
+int percpu_counter_tree_precise_compare_value(struct percpu_counter_tree *counter, long v)
+{
+ /* Compare on the wrapped difference, as on SMP. */
+ long delta = (long)((unsigned long)v -
+ (unsigned long)percpu_counter_tree_precise_sum(counter));
+
+ if (delta < 0)
+ return -1;
+ if (delta > 0)
+ return 1;
+ return 0;
+}
+
+static inline
+int percpu_counter_tree_approximate_compare(struct percpu_counter_tree *a,
+ struct percpu_counter_tree *b)
+{
+ return percpu_counter_tree_precise_compare(a, b);
+}
+
+static inline
+int percpu_counter_tree_approximate_compare_value(struct percpu_counter_tree *counter, long v)
+{
+ return percpu_counter_tree_precise_compare_value(counter, v);
+}
+
+static inline
+void percpu_counter_tree_set(struct percpu_counter_tree *counter, long v)
+{
+ atomic_long_set(&counter->count, v);
+}
+
+static inline
+unsigned long percpu_counter_tree_approximate_accuracy(struct percpu_counter_tree *counter)
+{
+ return 0;
+}
+
+static inline
+void percpu_counter_tree_add(struct percpu_counter_tree *counter, long inc)
+{
+ atomic_long_add(inc, &counter->count);
+}
+
+static inline
+void percpu_counter_tree_add_owner(struct percpu_counter_tree *counter, long inc)
+{
+ percpu_counter_tree_add(counter, inc);
+}
+
+static inline
+long percpu_counter_tree_approximate_sum(struct percpu_counter_tree *counter)
+{
+ return percpu_counter_tree_precise_sum(counter);
+}
+
+static inline
+int percpu_counter_tree_subsystem_init(void)
+{
+ return 0;
+}
+
+#endif /* CONFIG_SMP */
+
+/**
+ * percpu_counter_tree_approximate_sum_positive() - Return a positive approximate counter sum.
+ * @counter: The counter to sum.
+ *
+ * Return an approximate counter sum which is guaranteed to be greater
+ * or equal to 0.
+ *
+ * Return: The current positive approximate counter sum.
+ */
+static inline
+long percpu_counter_tree_approximate_sum_positive(struct percpu_counter_tree *counter)
+{
+ long v = percpu_counter_tree_approximate_sum(counter);
+
+ return v > 0 ? v : 0;
+}
+
+/**
+ * percpu_counter_tree_precise_sum_positive() - Return a positive precise counter sum.
+ * @counter: The counter to sum.
+ *
+ * Return a precise counter sum which is guaranteed to be greater
+ * or equal to 0.
+ *
+ * Return: The current positive precise counter sum.
+ */
+static inline
+long percpu_counter_tree_precise_sum_positive(struct percpu_counter_tree *counter)
+{
+ long v = percpu_counter_tree_precise_sum(counter);
+
+ return v > 0 ? v : 0;
+}
+
+/**
+ * percpu_counter_tree_approximate_min_max_range() - Return the min and max precise values.
+ * @approx_sum: Approximated sum.
+ * @accuracy: Tree accuracy, from percpu_counter_tree_approximate_accuracy().
+ * @precise_min: Minimum possible value for precise sum (output).
+ * @precise_max: Maximum possible value for precise sum (output).
+ *
+ * Calculate the minimum and maximum precise values for a given approximation
+ * and accuracy. The accuracy is a bound on a settled tree: one with no update
+ * in progress.
+ *
+ * The approximation is within the accuracy of the precise sum, in both
+ * directions:
+ *
+ * precise_sum - accuracy <= approx_sum <= precise_sum + accuracy
+ *
+ * Therefore, the precise sum is within the accuracy of the approximation:
+ *
+ * approx_sum - accuracy <= precise_sum <= approx_sum + accuracy
+ */
+static inline
+void percpu_counter_tree_approximate_min_max_range(long approx_sum, unsigned long accuracy,
+ long *precise_min, long *precise_max)
+{
+ *precise_min = approx_sum - accuracy;
+ *precise_max = approx_sum + accuracy;
+}
+
+/**
+ * percpu_counter_tree_approximate_min_max() - Return the approximation, min and max precise values.
+ * @counter: The counter to sum.
+ * @approx_sum: Approximate sum (output).
+ * @precise_min: Minimum possible value for precise sum (output).
+ * @precise_max: Maximum possible value for precise sum (output).
+ *
+ * Return the approximate sum, minimum and maximum precise values for
+ * a counter.
+ */
+static inline
+void percpu_counter_tree_approximate_min_max(struct percpu_counter_tree *counter,
+ long *approx_sum, long *precise_min, long *precise_max)
+{
+ unsigned long accuracy = percpu_counter_tree_approximate_accuracy(counter);
+ long v = percpu_counter_tree_approximate_sum(counter);
+
+ percpu_counter_tree_approximate_min_max_range(v, accuracy, precise_min, precise_max);
+ *approx_sum = v;
+}
+
+#endif /* _PERCPU_COUNTER_TREE_H */
diff --git a/include/trace/events/kmem.h b/include/trace/events/kmem.h
index cd7920c81f85ba..8ee6d5df596105 100644
--- a/include/trace/events/kmem.h
+++ b/include/trace/events/kmem.h
@@ -448,8 +448,7 @@ TRACE_EVENT(rss_stat,
*/
__entry->curr = current->mm == mm && !(current->flags & PF_KTHREAD);
__entry->member = member;
- __entry->size = (percpu_counter_sum_positive(&mm->rss_stat[member])
- << PAGE_SHIFT);
+ __entry->size = get_mm_counter(mm, member) << PAGE_SHIFT;
),
TP_printk("mm_id=%u curr=%d type=%s size=%ldB",
diff --git a/init/main.c b/init/main.c
index 7b0c8c34e6a911..d9614cc5bd63d2 100644
--- a/init/main.c
+++ b/init/main.c
@@ -108,6 +108,7 @@
#include <linux/time_namespace.h>
#include <linux/unaligned.h>
#include <linux/vdso_datastore.h>
+#include <linux/percpu_counter_tree.h>
#include <net/net_namespace.h>
#include <asm/io.h>
@@ -1104,6 +1105,8 @@ void start_kernel(void)
sort_main_extable();
trap_init();
mm_core_init();
+ /* After parsing its boot parameters and setting up slab, before the first counter. */
+ percpu_counter_tree_subsystem_init();
maple_tree_init();
poking_init();
ftrace_init();
diff --git a/kernel/exit.c b/kernel/exit.c
index d5e7bc2af804a6..c153cd064987c6 100644
--- a/kernel/exit.c
+++ b/kernel/exit.c
@@ -582,6 +582,9 @@ static void exit_mm(void)
local_irq_enable();
task_unlock(current);
mmap_read_unlock(mm);
+ /* From now on, this task updates the RSS counters of @mm atomically. */
+ if (READ_ONCE(mm->rss_owner) == current)
+ WRITE_ONCE(mm->rss_owner, NULL);
mm_update_next_owner(mm);
mmput(mm);
if (test_thread_flag(TIF_MEMDIE))
diff --git a/kernel/fork.c b/kernel/fork.c
index 8b108ed371529c..d0641700cd9c65 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -137,6 +137,28 @@
*/
#define MAX_THREADS FUTEX_TID_MASK
+/*
+ * Batch size of rss stat approximation, tunable with the rss_stat_batch=
+ * boot parameter to compare batch sizes on a given machine.
+ */
+static unsigned int rss_stat_batch_size __ro_after_init = 32;
+
+static int __init setup_rss_stat_batch(char *str)
+{
+ unsigned int batch;
+
+ if (kstrtouint(str, 0, &batch) || batch < 2 || !is_power_of_2(batch) ||
+ batch > PERCPU_COUNTER_TREE_MAX_BATCH) {
+ pr_warn("rss_stat_batch: expecting a power of 2 between 2 and %u\n",
+ PERCPU_COUNTER_TREE_MAX_BATCH);
+ return 1;
+ }
+ rss_stat_batch_size = batch;
+ pr_info("rss_stat_batch: %u\n", batch);
+ return 1;
+}
+__setup("rss_stat_batch=", setup_rss_stat_batch);
+
/*
* Protected counters by write_lock_irq(&tasklist_lock)
*/
@@ -633,7 +655,7 @@ static void check_mm(struct mm_struct *mm)
"Please make sure 'struct resident_page_types[]' is updated as well");
for (i = 0; i < NR_MM_COUNTERS; i++) {
- long x = percpu_counter_sum(&mm->rss_stat[i]);
+ long x = percpu_counter_tree_precise_sum(&mm->rss_stat[i]);
if (unlikely(x)) {
pr_alert("BUG: Bad rss-counter state mm:%p type:%s val:%ld Comm:%s Pid:%d\n",
@@ -738,7 +760,7 @@ void __mmdrop(struct mm_struct *mm)
check_mm(mm);
mm_pasid_drop(mm);
mm_destroy_cid(mm);
- percpu_counter_destroy_many(mm->rss_stat, NR_MM_COUNTERS);
+ percpu_counter_tree_destroy_many(mm->rss_stat, NR_MM_COUNTERS);
free_mm(mm);
}
@@ -1103,6 +1125,7 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p)
memset(&mm->rss_stat, 0, sizeof(mm->rss_stat));
spin_lock_init(&mm->page_table_lock);
spin_lock_init(&mm->arg_lock);
+ mutex_init(&mm->rss_stat_mutex);
mm_init_cpumask(mm);
mm_init_aio(mm);
mm_init_owner(mm, p);
@@ -1143,8 +1166,7 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p)
if (mm_alloc_sched(mm))
goto fail_sched;
- if (percpu_counter_init_many(mm->rss_stat, 0, GFP_KERNEL_ACCOUNT,
- NR_MM_COUNTERS))
+ if (percpu_counter_tree_init_many_lazy(mm->rss_stat, NR_MM_COUNTERS, rss_stat_batch_size))
goto fail_pcpu;
lru_gen_init_mm(mm);
@@ -1537,6 +1559,8 @@ static struct mm_struct *dup_mm(struct task_struct *tsk,
if (!mm_init(mm, tsk))
goto fail_nomem;
+ /* The child owns its RSS counters, see __add_mm_counter(). */
+ mm->rss_owner = tsk;
uprobe_start_dup_mmap();
err = dup_mmap(mm, oldmm);
@@ -1564,6 +1588,23 @@ static struct mm_struct *dup_mm(struct task_struct *tsk,
return NULL;
}
+/*
+ * Serialize the activations of the RSS counter tree, whichever tasks create
+ * threads. Once the last counter is active, the activation is complete and
+ * thread creation skips the mutex. That check needs no acquire: the updates
+ * and the sums of the counters order their own accesses to the tree. Not
+ * observing the other counters active yet only delays their updates going
+ * to the tree.
+ */
+static int mm_activate_rss_stat(struct mm_struct *mm)
+{
+ if (percpu_counter_tree_active(&mm->rss_stat[NR_MM_COUNTERS - 1]))
+ return 0;
+ guard(mutex)(&mm->rss_stat_mutex);
+ return percpu_counter_tree_activate_many(mm->rss_stat, NR_MM_COUNTERS,
+ GFP_KERNEL_ACCOUNT);
+}
+
static int copy_mm(u64 clone_flags, struct task_struct *tsk)
{
struct mm_struct *mm, *oldmm;
@@ -1588,6 +1629,18 @@ static int copy_mm(u64 clone_flags, struct task_struct *tsk)
return 0;
if (clone_flags & CLONE_VM) {
+ int err = 0;
+
+ /*
+ * A second thread shares the mm: activate the RSS counter
+ * tree. A vfork child shares the mm while the parent waits,
+ * so the single counters still fit. The tree stays active if
+ * the clone fails later, which is harmless.
+ */
+ if (!(clone_flags & CLONE_VFORK))
+ err = mm_activate_rss_stat(oldmm);
+ if (err)
+ return err;
mmget(oldmm);
mm = oldmm;
} else {
diff --git a/lib/Kconfig b/lib/Kconfig
index 1e42f5167c68bc..6ab9644a4cb830 100644
--- a/lib/Kconfig
+++ b/lib/Kconfig
@@ -41,6 +41,33 @@ config PACKING_KUNIT_TEST
When in doubt, say N.
+config PERCPU_COUNTER_TREE_KUNIT_TEST
+ tristate "KUnit tests for hierarchical per-cpu counters" if !KUNIT_ALL_TESTS
+ depends on KUNIT
+ default KUNIT_ALL_TESTS
+ help
+ This builds Kunit tests for the hierarchical per-cpu counters.
+
+ For more information on KUnit and unit tests in general,
+ please refer to the KUnit documentation in Documentation/dev-tools/kunit/.
+
+ When in doubt, say N.
+
+config PERCPU_COUNTER_TREE_BENCH
+ tristate "Benchmark of hierarchical per-cpu counters"
+ depends on KUNIT && SMP
+ help
+ This builds a KUnit benchmark comparing the hierarchical per-cpu
+ counters with struct percpu_counter: update cost with all online
+ CPUs updating shared and private counters, and init/destroy cost.
+ It runs for a few minutes on large machines.
+
+ The batch size and the tree shape can be changed with the
+ percpu_counter_tree_bench.batch= and percpu_counter_tree.arity=
+ parameters, see lib/tests/percpu_counter_tree_bench.c.
+
+ When in doubt, say N.
+
config BITREVERSE
tristate
select GENERIC_BITREVERSE if !HAVE_ARCH_BITREVERSE
diff --git a/lib/Makefile b/lib/Makefile
index a2afe954805cb8..07410b1d20a379 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -182,6 +182,7 @@ obj-$(CONFIG_TEXTSEARCH_KMP) += ts_kmp.o
obj-$(CONFIG_TEXTSEARCH_BM) += ts_bm.o
obj-$(CONFIG_TEXTSEARCH_FSM) += ts_fsm.o
obj-$(CONFIG_SMP) += percpu_counter.o
+obj-$(CONFIG_SMP) += percpu_counter_tree.o
obj-$(CONFIG_AUDIT_GENERIC) += audit.o
obj-$(CONFIG_AUDIT_COMPAT_GENERIC) += compat_audit.o
diff --git a/lib/percpu_counter_tree.c b/lib/percpu_counter_tree.c
new file mode 100644
index 00000000000000..491675897e41c2
--- /dev/null
+++ b/lib/percpu_counter_tree.c
@@ -0,0 +1,1450 @@
+// SPDX-License-Identifier: GPL-2.0+ OR MIT
+// SPDX-FileCopyrightText: 2025-2026 Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
+
+/*
+ * Split Counters With Tree Approximation Propagation
+ *
+ * * Overview
+ *
+ * Each counter is a tree of counters. A CPU updates its own level 0
+ * counter, a signed byte. Each counter keeps the residual it has not
+ * propagated yet, and propagates a carry to its parent on the next level
+ * once that residual reaches the batch size of its level, up to the root.
+ * Most updates therefore stay on the updating CPU. The root holds the
+ * approximate sum, read in O(1), and the precise sum adds the residuals of
+ * all levels to it.
+ *
+ * * Tree shape
+ *
+ * Illustration with 8 CPUs and a fan-out of 2 at each level, which makes
+ * 3 levels of counters below the root. This is not an actual
+ * configuration: at 8 CPUs, the default tree is flat.
+ *
+ * Level 3: 0 (root)
+ * | \
+ * | \
+ * | \
+ * Level 2: 0 1
+ * | \ | \
+ * | \ | \
+ * | \ | \
+ * Level 1: 0 2 1 3
+ * | \ | \ | \ | \
+ * | \ | \ | \ | \
+ * | \ | \ | \ | \
+ * Level 0: 0 4 2 6 1 5 3 7
+ *
+ * CPU c propagates into item (c & (L - 1)) of a level holding L items,
+ * so the CPUs sharing an item are not adjacent: CPUs 0 and 4 share
+ * level 1 item 0, and CPUs 0, 2, 4, 6 share level 2 item 0. This mapping
+ * uses every item of each level and gives each item the same number of
+ * CPUs, give or take one, even when nr_cpu_ids is not a power of 2. The
+ * diagram lists the CPUs in bit-reversed order to avoid crossing lines.
+ *
+ * The actual fan-out of each level depends on the topology (see
+ * per_nr_cpu_order_config): up to 32 CPUs, the tree is flat, and the
+ * level 0 counters propagate directly to the root.
+ *
+ * * Batch sizes and residuals
+ *
+ * The batch size of a level, BATCH(level N), is the batch size of the
+ * level below multiplied by the fan-out of that level. Example for
+ * BATCH(level 0) = 32, with the tree of the illustration above:
+ *
+ * BATCH(level 0) = 32
+ * BATCH(level 1) = 64
+ * BATCH(level 2) = 128
+ *
+ * Once the residual of a counter reaches +/- BATCH(level N), it propagates
+ * its multiples of BATCH(level N), truncated toward zero, and subtracts
+ * them. This hysteresis prevents updates going back and forth around a
+ * BATCH(level N) boundary from propagating up the tree each time.
+ *
+ * The residual of a settled counter therefore stays within
+ * (-BATCH(level N), +BATCH(level N)), and moves in steps of the counter's
+ * input: level 0 receives the updates, in steps of 1, while a counter
+ * above level 0 only receives the carries of the level below, in steps of
+ * BATCH(level N - 1). The largest residual of a counter is the largest
+ * multiple of its input step below its batch size: BATCH(level 0) - 1 at
+ * level 0, and BATCH(level N) - BATCH(level N - 1) above:
+ *
+ * input residual worst case worst case
+ * step per counter below precise above precise
+ * Level 0: 1 [-31 .. 31] 248 (8 * 31) 248 (8 * 31)
+ * Level 1: 32 {-32, 0, 32} 128 (4 * 32) 128 (4 * 32)
+ * Level 2: 64 {-64, 0, 64} 128 (2 * 64) 128 (2 * 64)
+ * Total: 504 504
+ *
+ * * Approximation accuracy
+ *
+ * The approximate sum is the precise sum minus the residuals, so a
+ * settled counter (no update in progress) is within:
+ *
+ * precise - 504 <= approx <= precise + 504
+ *
+ * The accuracy returned by percpu_counter_tree_approximate_accuracy() is
+ * exact. It adds the largest residuals of all the counters:
+ * BATCH(level 0) - 1 for each of the nr_cpu_ids level 0 counters, and
+ * BATCH(level N) - BATCH(level N - 1) for each counter of level N above.
+ * Counting each level 0 counter as one BATCH(level 0), minus its input
+ * step of 1:
+ *
+ * accuracy_multiplier = nr_cpu_ids + sum over levels N >= 1 of
+ * (number of counters at level N) *
+ * (BATCH(level N) - BATCH(level N - 1)) / BATCH(level 0)
+ *
+ * accuracy = BATCH(level 0) * accuracy_multiplier - nr_cpu_ids
+ *
+ * In the example above, accuracy_multiplier = 8 + 4 * 1 + 2 * 2 = 16, and:
+ *
+ * accuracy = 32 * 16 - 8 = 504
+ *
+ * Every counter above level 0 is reachable from a CPU below nr_cpu_ids,
+ * even when nr_cpu_ids is not a power of 2, so both bounds can be reached
+ * when the possible CPUs are contiguous. When they are sparse, the bound
+ * also counts the level 0 residuals of the missing CPUs: it holds, but
+ * cannot be reached.
+ *
+ * * Carry propagation
+ *
+ * The items at each tree position of the counters initialized together
+ * share cache lines, which are not shared with other tree positions to
+ * prevent false sharing between CPUs.
+ *
+ * Example with expanded values, for the tree and batch sizes above:
+ *
+ * counter_add(counter, inc):
+ *
+ * if (!inc)
+ * return;
+ *
+ * // Level 0: per-CPU residual within (-32, +32).
+ * old = percpu_read(counter @ Level 0);
+ * do {
+ * v = old + inc;
+ * out = v < 0 ? -(-v & ~0b00011111) : (v & ~0b00011111);
+ * } while (!percpu_try_cmpxchg(counter @ Level 0, &old, v - out));
+ * inc = out;
+ * if (!inc)
+ * return;
+ *
+ * // Levels 1 and 2: add the carry, then extract the multiples
+ * // of the level batch held by the counter, truncated toward
+ * // zero. A failed cmpxchg updates res to the current residual.
+ * res = atomic_long_add_return(counter @ Level 1, inc);
+ * do {
+ * inc = res < 0 ? -(-res & ~0b00111111) : (res & ~0b00111111);
+ * if (!inc)
+ * return;
+ * } while (!atomic_long_try_cmpxchg(counter @ Level 1, &res, res - inc));
+ *
+ * res = atomic_long_add_return(counter @ Level 2, inc);
+ * do {
+ * inc = res < 0 ? -(-res & ~0b01111111) : (res & ~0b01111111);
+ * if (!inc)
+ * return;
+ * } while (!atomic_long_try_cmpxchg(counter @ Level 2, &res, res - inc));
+ *
+ * atomic_long_add(counter @ Level 3, inc);
+ */
+
+#define pr_fmt(fmt) "percpu_counter_tree: " fmt
+
+#include <linux/percpu_counter_tree.h>
+#include <linux/moduleparam.h>
+#include <linux/align.h>
+#include <linux/cache.h>
+#include <linux/cpumask.h>
+#include <linux/atomic.h>
+#include <linux/export.h>
+#include <linux/percpu.h>
+#include <linux/errno.h>
+#include <linux/slab.h>
+#include <linux/math.h>
+#include <linux/log2.h>
+#include <linux/args.h>
+#include <linux/string.h>
+#include <kunit/visibility.h>
+
+#define MAX_NR_LEVELS 7
+
+/*
+ * The counter configuration is selected at boot time based on the
+ * hardware topology.
+ */
+struct counter_config {
+ /*
+ * nr_levels is the number of hierarchical counter tree levels. It
+ * excludes the final level (approximate sum).
+ */
+ unsigned char nr_levels;
+ /* n-arity of each level from 0 to (nr_levels - 1). */
+ unsigned char n_arity_order[MAX_NR_LEVELS];
+};
+
+/*
+ * The fan-out of each level of the tree is chosen with a heuristic trying to
+ * minimize both the tree depth and the fan-out.
+ */
+#define PCT_COUNTER_CONFIG(order, ...) \
+ [order] = { \
+ .nr_levels = COUNT_ARGS(__VA_ARGS__), \
+ .n_arity_order = { __VA_ARGS__ }, \
+ }
+
+/*
+ * Up to 32 CPUs, the tree is flat: the level 0 counters propagate directly
+ * to the root, like struct percpu_counter. From 33 to 64 CPUs, the level 0
+ * counters propagate to 4 intermediate counters, each shared by up to 16
+ * CPUs, so the carries of all CPUs do not contend on the root. Beyond, each
+ * level has a fan-out of 8. When the order is not a multiple of 3, a
+ * remaining order of 1 gives the root a fan-out of 16, and a remaining
+ * order of 2 adds a level with a fan-out of 4 below the root: a root
+ * fan-out of 16 updates about as fast as a fan-out of 2 below the root,
+ * with one level less, which shrinks the accuracy bound and the tree
+ * items, while a root fan-out of 32 is slower than a fan-out of 4 below
+ * it. This keeps the carry rate reaching each tree item low with the RSS
+ * batch size of 32, and the approximation error grows as
+ * O(nr_cpus * log(nr_cpus)), compared to O(nr_cpus^2) for struct
+ * percpu_counter whose batch size grows with the number of CPUs.
+ *
+ * The tree is shaped for nr_cpu_ids rounded up to a power of 2, and each
+ * tree position takes a cache line. A set of 4 counters initialized together,
+ * such as the RSS counters of an mm, uses less memory than 4 struct
+ * percpu_counter from a few CPUs up to 64, and beyond when nr_cpu_ids is a
+ * power of 2 (at most 0.89 times as much). Between powers of 2, fewer CPUs
+ * share the cost of the tree items: up to 1.1 times at 3 * 2^n CPUs, and
+ * up to 1.48 times just above a power of 2. A lone counter uses one item
+ * of each cache line: 3.0 to 3.5 times the memory of a struct
+ * percpu_counter when nr_cpu_ids is a power of 2 beyond 64, and up to 5.7
+ * times just above.
+ */
+static const struct counter_config per_nr_cpu_order_config[] = {
+ PCT_COUNTER_CONFIG(0),
+ PCT_COUNTER_CONFIG(1, 1),
+ PCT_COUNTER_CONFIG(2, 2),
+ PCT_COUNTER_CONFIG(3, 3),
+ PCT_COUNTER_CONFIG(4, 4),
+ PCT_COUNTER_CONFIG(5, 5),
+ PCT_COUNTER_CONFIG(6, 4, 2),
+ PCT_COUNTER_CONFIG(7, 3, 4),
+ PCT_COUNTER_CONFIG(8, 3, 2, 3),
+ PCT_COUNTER_CONFIG(9, 3, 3, 3),
+ PCT_COUNTER_CONFIG(10, 3, 3, 4),
+ PCT_COUNTER_CONFIG(11, 3, 3, 2, 3),
+ PCT_COUNTER_CONFIG(12, 3, 3, 3, 3),
+ PCT_COUNTER_CONFIG(13, 3, 3, 3, 4),
+ PCT_COUNTER_CONFIG(14, 3, 3, 3, 2, 3),
+ PCT_COUNTER_CONFIG(15, 3, 3, 3, 3, 3),
+ PCT_COUNTER_CONFIG(16, 3, 3, 3, 3, 4),
+ PCT_COUNTER_CONFIG(17, 3, 3, 3, 3, 2, 3),
+ PCT_COUNTER_CONFIG(18, 3, 3, 3, 3, 3, 3),
+ PCT_COUNTER_CONFIG(19, 3, 3, 3, 3, 3, 4),
+ PCT_COUNTER_CONFIG(20, 3, 3, 3, 3, 3, 2, 3),
+};
+
+#undef PCT_COUNTER_CONFIG
+
+/* The configuration table covers every possible number of CPUs. */
+#define MAX_NR_CPUS_ORDER 20
+#if NR_CPUS > (1UL << MAX_NR_CPUS_ORDER)
+#error "NR_CPUS exceeds the hierarchical counter configuration table"
+#endif
+static_assert(ARRAY_SIZE(per_nr_cpu_order_config) == MAX_NR_CPUS_ORDER + 1);
+
+/* Tree topology, selected at boot and shared by all the counters. */
+static struct {
+ /* Hierarchical counter configuration for the hardware topology. */
+ const struct counter_config *config;
+ /* Tree shape selected with the "arity" boot parameter. */
+ struct counter_config boot_config;
+ /* Order of nr_cpu_ids. */
+ unsigned int nr_cpus_order;
+ /*
+ * Tree items per counter, including the root. UINT_MAX catches early
+ * boot misuses.
+ */
+ unsigned int nr_items;
+ /*
+ * Largest residuals of all the counters, in units of the level 0
+ * batch size: see percpu_counter_tree_accuracy().
+ */
+ unsigned long accuracy_multiplier;
+ /* Tree items when those of a tree position fit in one cache line. */
+ struct kmem_cache *items_cachep;
+} tree_topology __ro_after_init = {
+ .nr_items = UINT_MAX,
+};
+
+/* Value of the "arity" boot parameter, parsed at subsystem initialization. */
+static const char *arity_param;
+
+/*
+ * Size of the items of one tree position for the counters initialized
+ * together by percpu_counter_tree_init_many(), packed in cache lines which
+ * are not shared with other tree positions. Each item is the residual of
+ * the carries received and not propagated yet, within (-batch, +batch) of
+ * its tree level when settled. The items of a tree position are updated
+ * by the same CPUs, so packing them does not add contention between
+ * unrelated CPUs.
+ */
+static
+size_t percpu_counter_tree_position_size(unsigned int nr_counters)
+{
+ return ALIGN(nr_counters * sizeof(atomic_long_t), SMP_CACHE_BYTES);
+}
+
+/* Distance between the consecutive items of each of @nr_counters counters. */
+static inline
+unsigned int percpu_counter_tree_items_stride(unsigned int nr_counters)
+{
+ return percpu_counter_tree_position_size(nr_counters) / sizeof(atomic_long_t);
+}
+
+/* Allocate the zeroed tree items of @nr_counters counters, the root last. */
+static
+atomic_long_t *percpu_counter_tree_alloc_items(unsigned int nr_counters, gfp_t gfp_flags)
+{
+ size_t position_size = percpu_counter_tree_position_size(nr_counters);
+
+ if (position_size == SMP_CACHE_BYTES)
+ return kmem_cache_zalloc(tree_topology.items_cachep, gfp_flags);
+ /* kmalloc() aligns power of 2 sizes on their size. */
+ return kzalloc(roundup_pow_of_two(tree_topology.nr_items * position_size),
+ gfp_flags);
+}
+
+static
+void percpu_counter_tree_free_items(atomic_long_t *items, unsigned int nr_counters)
+{
+ if (percpu_counter_tree_position_size(nr_counters) == SMP_CACHE_BYTES)
+ kmem_cache_free(tree_topology.items_cachep, items);
+ else
+ kfree(items);
+}
+
+static
+int __percpu_counter_tree_init(struct percpu_counter_tree *counter,
+ unsigned long batch_size,
+ s8 __percpu *level0,
+ atomic_long_t *items, unsigned int items_stride)
+{
+ /* Batch size must be greater than 1, a power of 2, and fit in a byte. */
+ if (WARN_ON(batch_size <= 1 || (batch_size & (batch_size - 1)) ||
+ batch_size > PERCPU_COUNTER_TREE_MAX_BATCH))
+ return -EINVAL;
+ counter->batch_order = ilog2(batch_size);
+ counter->bias = 0;
+ atomic_long_set(&counter->atomic_bias, 0);
+ counter->level0 = level0;
+ counter->items_stride = items_stride;
+ /* The root is the last item, allocated by the activation if lazy. */
+ counter->approx_sum = items ?
+ &items[(tree_topology.nr_items - 1) * items_stride] : NULL;
+ return 0;
+}
+
+static inline
+unsigned long percpu_counter_tree_batch_size(struct percpu_counter_tree *counter)
+{
+ return 1UL << counter->batch_order;
+}
+
+/* This counter's first item. */
+static inline
+atomic_long_t *percpu_counter_tree_items(struct percpu_counter_tree *counter)
+{
+ return counter->approx_sum - (tree_topology.nr_items - 1) * counter->items_stride;
+}
+
+/*
+ * The residuals of all levels are within (-batch, +batch) of their level,
+ * so the approximation is within the same distance of the precise sum in
+ * both directions: the largest residuals of all the counters, which
+ * accuracy_multiplier counts in units of the level 0 batch size, minus the
+ * input step of 1 of each level 0 counter. See "Approximation accuracy" at
+ * the top of this file.
+ */
+static inline
+unsigned long percpu_counter_tree_accuracy(struct percpu_counter_tree *counter)
+{
+ unsigned long batch_size = percpu_counter_tree_batch_size(counter);
+
+ return batch_size * tree_topology.accuracy_multiplier - nr_cpu_ids;
+}
+
+/* Initialize @nr_counters counters, with their per-CPU counters and tree items, or NULL if lazy. */
+static
+int __percpu_counter_tree_init_many(struct percpu_counter_tree *counters,
+ unsigned int nr_counters, unsigned long batch_size,
+ s8 __percpu *level0, atomic_long_t *items)
+{
+ unsigned int items_stride = percpu_counter_tree_items_stride(nr_counters);
+ unsigned int i;
+ int ret;
+
+ for (i = 0; i < nr_counters; i++) {
+ ret = __percpu_counter_tree_init(&counters[i], batch_size,
+ level0 ? level0 + i : NULL,
+ items ? items + i : NULL, items_stride);
+ if (ret)
+ return ret;
+ }
+ return 0;
+}
+
+/**
+ * percpu_counter_tree_init_many() - Initialize many per-CPU counter trees.
+ * @counters: An array of @nr_counters counters to initialize.
+ * Their memory is provided by the caller.
+ * @nr_counters: The number of counter trees to initialize
+ * @batch_size: The batch size is the increment step at level 0 which triggers a
+ * carry propagation.
+ * The batch size is required to be greater than 1, a power of 2, and at
+ * most PERCPU_COUNTER_TREE_MAX_BATCH.
+ * @gfp_flags: gfp flags to pass to the per-CPU and tree item allocators.
+ *
+ * Initialize many per-CPU counter trees, with a single allocation of
+ * per-CPU counters and a single allocation of tree items for the
+ * @nr_counters counters. The items at each tree position of the
+ * @nr_counters counters share cache lines. When they fit in one cache
+ * line (up to 8 counters on 64-bit with 64-byte cache lines), the tree
+ * items come from a slab cache; beyond, their allocation is
+ * rounded up to a power of 2, which can double it.
+ *
+ * Return:
+ * * %0: Success
+ * * %-EINVAL: - Invalid @batch_size argument
+ * * %-ENOMEM: - Out of memory
+ */
+int percpu_counter_tree_init_many(struct percpu_counter_tree *counters,
+ unsigned int nr_counters, unsigned long batch_size,
+ gfp_t gfp_flags)
+{
+ s8 __percpu *level0;
+ atomic_long_t *items;
+ int ret;
+
+ /* The tree shape is selected by percpu_counter_tree_subsystem_init() at boot. */
+ if (WARN_ON_ONCE(tree_topology.nr_items == UINT_MAX))
+ return -EINVAL;
+ level0 = __alloc_percpu_gfp(nr_counters * sizeof(*level0),
+ __alignof__(*counters->level0), gfp_flags);
+ if (!level0)
+ return -ENOMEM;
+ items = percpu_counter_tree_alloc_items(nr_counters, gfp_flags);
+ if (!items) {
+ ret = -ENOMEM;
+ goto free_level0;
+ }
+ ret = __percpu_counter_tree_init_many(counters, nr_counters, batch_size, level0, items);
+ if (!ret)
+ return 0;
+ percpu_counter_tree_free_items(items, nr_counters);
+free_level0:
+ free_percpu(level0);
+ return ret;
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_init_many);
+
+/**
+ * percpu_counter_tree_init() - Initialize one per-CPU counter tree.
+ * @counter: Counter to initialize.
+ * Its memory is provided by the caller.
+ * @batch_size: The batch size is the increment step at level 0 which triggers a
+ * carry propagation.
+ * The batch size is required to be greater than 1, a power of 2, and at
+ * most PERCPU_COUNTER_TREE_MAX_BATCH.
+ * @gfp_flags: gfp flags to pass to the per-CPU and tree item allocators.
+ *
+ * Initialize one per-CPU counter tree.
+ *
+ * Return:
+ * * %0: Success
+ * * %-EINVAL: - Invalid @batch_size argument
+ * * %-ENOMEM: - Out of memory
+ */
+int percpu_counter_tree_init(struct percpu_counter_tree *counter,
+ unsigned long batch_size, gfp_t gfp_flags)
+{
+ return percpu_counter_tree_init_many(counter, 1, batch_size, gfp_flags);
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_init);
+
+/**
+ * percpu_counter_tree_init_many_lazy() - Initialize many counters, without their tree.
+ * @counters: An array of @nr_counters counters to initialize.
+ * Their memory is provided by the caller.
+ * @nr_counters: The number of counter trees to initialize
+ * @batch_size: The batch size, as for percpu_counter_tree_init_many().
+ *
+ * Initialize counters without allocating their per-CPU counters and tree
+ * items, for counters which may never need the scalability of the tree,
+ * such as the RSS counters of a single-threaded process. Until
+ * percpu_counter_tree_activate_many() allocates the tree, the counters
+ * are exact: percpu_counter_tree_add_owner() adds to the bias, written
+ * only by the owner, percpu_counter_tree_add() adds to the atomic bias,
+ * and the precise and approximate sums read both in O(1).
+ * percpu_counter_tree_destroy_many() and percpu_counter_tree_set() work on
+ * counters which were never activated. The accuracy range of a lazily
+ * initialized counter is the range of its tree, even before activation,
+ * which keeps the comparison functions correct across an activation.
+ * The batch size is fixed here: the activation only publishes the per-CPU
+ * counters and the tree, so updaters read the batch size without ordering.
+ *
+ * Return:
+ * * %0: Success
+ * * %-EINVAL: - Invalid @batch_size argument
+ */
+int percpu_counter_tree_init_many_lazy(struct percpu_counter_tree *counters,
+ unsigned int nr_counters, unsigned long batch_size)
+{
+ /* The tree shape is selected by percpu_counter_tree_subsystem_init() at boot. */
+ if (WARN_ON_ONCE(tree_topology.nr_items == UINT_MAX))
+ return -EINVAL;
+ return __percpu_counter_tree_init_many(counters, nr_counters, batch_size, NULL, NULL);
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_init_many_lazy);
+
+/*
+ * Publication of the tree
+ *
+ * The updaters and the readers of a counter take no lock, and may run
+ * while percpu_counter_tree_activate_many() publishes its tree. The
+ * activation allocates the per-CPU counters and the tree items zeroed,
+ * then publishes two pointers per counter, each with a store-release:
+ *
+ * - the release of approx_sum: the root is zeroed before its pointer,
+ * - the release of level0, after it: the per-CPU counters, the tree items
+ * and approx_sum are set before level0.
+ *
+ * level0 opens the tree to updates: until it is published, the tree is
+ * empty. Each load of a published pointer pairs with one of the releases:
+ *
+ * - Update fast path, percpu_counter_tree_add() and
+ * percpu_counter_tree_add_owner(): READ_ONCE(level0), then the per-CPU
+ * counters through it. The address dependency pairs with the release of
+ * level0, as for rcu_dereference().
+ * - Carry slow path: smp_rmb() after the read of level0, then
+ * READ_ONCE(approx_sum). Pairs with the release of level0, which
+ * follows the store of approx_sum.
+ * - Precise sum: smp_load_acquire(level0), then approx_sum, the tree items
+ * and the per-CPU counters. Pairs with the release of level0.
+ * - Approximate sum: READ_ONCE(approx_sum), then the root through it. The
+ * address dependency pairs with the release of approx_sum.
+ * - percpu_counter_tree_active(): READ_ONCE(level0), without ordering. It
+ * only selects the update function, which reads level0 again, or lets
+ * the caller skip an activation which is complete.
+ * - Activations: serialized by the caller, for instance with a mutex,
+ * which orders a complete activation before the check of level0 at the
+ * start of the next one.
+ *
+ * The update fast path has no acquire. smp_load_acquire() is free on
+ * strongly ordered architectures such as x86, but on weakly ordered ones,
+ * such as arm64, powerpc and RISC-V, it costs an ordering load or a barrier
+ * on every update, to order a publication which happens at most once in
+ * the life of a counter. The address dependency orders the per-CPU
+ * counters, reached through level0, at no cost. It does not order
+ * approx_sum, a separate field: the carry slow path, taken only when an
+ * update carries, pays an smp_rmb() to read it, and the precise sum, which
+ * reads the whole tree, reads level0 with an acquire.
+ *
+ * Missing the publication is harmless. An updater which reads level0 as
+ * NULL adds to the bias (owner) or to the atomic bias (others), and a
+ * reader which reads a NULL pointer sums the bias and the atomic bias:
+ * the sums keep reading both after the activation, and the tree receives
+ * no update before level0 is published.
+ *
+ * The read barriers between the levels of the precise sum are unrelated
+ * to the publication: they order carries, see
+ * percpu_counter_tree_precise_sum_unbiased().
+ */
+
+/**
+ * percpu_counter_tree_activate_many() - Activate the tree of lazily initialized counters.
+ * @counters: The array of @nr_counters counters passed to
+ * percpu_counter_tree_init_many_lazy().
+ * @nr_counters: The number of counters.
+ * @gfp_flags: gfp flags to pass to the per-CPU and tree item allocators.
+ *
+ * Allocate the per-CPU counters and the tree items, and publish them:
+ * later updates go to the tree, and the approximation then has the
+ * accuracy of the tree. The updates made before activation stay in the
+ * bias and the atomic bias, which the sums keep reading, so updates racing
+ * with the activation are not lost: the bias only has the owner as writer,
+ * and the atomic bias only receives atomic additions. Activating active
+ * counters does nothing. @counters and @nr_counters must be those passed
+ * to percpu_counter_tree_init_many_lazy().
+ *
+ * The activations of the same counters must be serialized by the caller,
+ * for instance with a mutex: a concurrent activation would publish a second
+ * tree and lose the updates made to the first. The counters are published
+ * in order: once percpu_counter_tree_active() reports the last one active,
+ * the activation is complete, and a caller can skip it and its
+ * serialization. The caller may not observe the other counters active yet,
+ * which only delays their updates going to the tree.
+ *
+ * Return:
+ * * %0: Success
+ * * %-EINVAL: - @nr_counters differs from percpu_counter_tree_init_many_lazy()
+ * * %-ENOMEM: - Out of memory
+ */
+int percpu_counter_tree_activate_many(struct percpu_counter_tree *counters,
+ unsigned int nr_counters, gfp_t gfp_flags)
+{
+ s8 __percpu *level0;
+ atomic_long_t *items, *root;
+ unsigned int i;
+
+ /* Serialized by the caller: a previous activation is complete. */
+ if (counters[0].level0)
+ return 0;
+ /* The tree items are sized for @nr_counters, at the stride set by the initialization. */
+ if (WARN_ON_ONCE(counters[0].items_stride !=
+ percpu_counter_tree_items_stride(nr_counters)))
+ return -EINVAL;
+ level0 = __alloc_percpu_gfp(nr_counters * sizeof(*level0),
+ __alignof__(*counters->level0), gfp_flags);
+ if (!level0)
+ return -ENOMEM;
+ items = percpu_counter_tree_alloc_items(nr_counters, gfp_flags);
+ if (!items) {
+ free_percpu(level0);
+ return -ENOMEM;
+ }
+ root = &items[(tree_topology.nr_items - 1) * counters[0].items_stride];
+ /* See "Publication of the tree" above. */
+ for (i = 0; i < nr_counters; i++) {
+ /* Release of approx_sum: the root is zeroed before its pointer. */
+ smp_store_release(&counters[i].approx_sum, root + i);
+ /*
+ * Release of level0: the per-CPU counters, the tree items and
+ * approx_sum are set before level0, which opens the tree to
+ * updates.
+ */
+ smp_store_release(&counters[i].level0, level0 + i);
+ }
+ return 0;
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_activate_many);
+
+/**
+ * percpu_counter_tree_destroy_many() - Destroy many per-CPU counter trees.
+ * @counters: Array of counters trees to destroy.
+ * @nr_counters: The number of counter trees to destroy.
+ *
+ * Release internal resources allocated for @nr_counters per-CPU counter trees.
+ * @counters and @nr_counters must be those passed to
+ * percpu_counter_tree_init_many() or percpu_counter_tree_init_many_lazy().
+ */
+void percpu_counter_tree_destroy_many(struct percpu_counter_tree *counters,
+ unsigned int nr_counters)
+{
+ free_percpu(counters->level0);
+ /*
+ * @nr_counters selects the allocator of the tree items: on a mismatch
+ * with the initialization, leak them rather than corrupt the allocator.
+ */
+ if (WARN_ON_ONCE(counters->items_stride !=
+ percpu_counter_tree_items_stride(nr_counters)))
+ return;
+ /* The first item of the first counter is the base of the tree items, if allocated. */
+ if (counters->approx_sum)
+ percpu_counter_tree_free_items(percpu_counter_tree_items(counters), nr_counters);
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_destroy_many);
+
+/**
+ * percpu_counter_tree_destroy() - Destroy one per-CPU counter tree.
+ * @counter: Counter to destroy.
+ *
+ * Release internal resources allocated for one per-CPU counter tree.
+ */
+void percpu_counter_tree_destroy(struct percpu_counter_tree *counter)
+{
+ return percpu_counter_tree_destroy_many(counter, 1);
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_destroy);
+
+/*
+ * Amount propagated by a counter holding @v: the multiples of @bit_mask,
+ * truncated toward zero.
+ */
+static inline
+long percpu_counter_tree_item_carry(long v, unsigned long bit_mask)
+{
+ if (v < 0)
+ return -(long)(-(unsigned long)v & ~(bit_mask - 1));
+ return (long)((unsigned long)v & ~(bit_mask - 1));
+}
+
+/*
+ * It does not matter through which path the carry propagates up the
+ * tree, therefore there is no need to disable preemption because the
+ * cpu number is only used to favor cache locality.
+ */
+static
+void percpu_counter_tree_add_slowpath(struct percpu_counter_tree *counter, long inc)
+{
+ unsigned int level_items, nr_levels = tree_topology.config->nr_levels,
+ level, n_arity_order;
+ unsigned long bit_mask;
+ unsigned int stride = counter->items_stride;
+ unsigned int cpu = raw_smp_processor_id();
+ atomic_long_t *root, *item;
+
+ /*
+ * Pairs with the release of level0 in
+ * percpu_counter_tree_activate_many(), which follows the store of
+ * approx_sum. The caller read level0, but approx_sum is a separate
+ * field, which the address dependency of that read does not order:
+ * read it after level0.
+ */
+ smp_rmb();
+ root = READ_ONCE(counter->approx_sum);
+ item = root - (tree_topology.nr_items - 1) * stride;
+ n_arity_order = tree_topology.config->n_arity_order[0];
+ bit_mask = percpu_counter_tree_batch_size(counter) << n_arity_order;
+ level_items = 1U << (tree_topology.nr_cpus_order - n_arity_order);
+
+ for (level = 1; level < nr_levels; level++) {
+ /*
+ * A multiple of bit_mask would leave the residual of this
+ * level unchanged: propagate it to the next level directly.
+ * This trades the hysteresis of this level for not touching
+ * its cache line.
+ */
+ if (abs(inc) & (bit_mask - 1)) {
+ atomic_long_t *count = &item[(cpu & (level_items - 1)) * stride];
+ long res;
+
+ /*
+ * Add the carry to the residual of this level, then
+ * extract the multiples of bit_mask of the residual,
+ * truncated toward zero, and propagate them to the
+ * next level, which leaves the residual within
+ * (-bit_mask, +bit_mask).
+ *
+ * The extraction is a cmpxchg on the value returned by
+ * the addition, whose cache line is already exclusive,
+ * so it usually succeeds at once. When a concurrent
+ * updater changed the residual in between, the cmpxchg
+ * fails and the carry is recomputed from the current
+ * value, which is 0 if the other updater extracted it.
+ * Each carry is thus extracted once, from an exact
+ * value: the residual only exceeds its range by the
+ * additions in flight, and is within range once all
+ * the updaters are done. An addition which does not
+ * carry stays a single atomic operation, which cannot
+ * fail.
+ *
+ * Relaxed ordering is sufficient for the residual
+ * itself: the reasoning above only involves this
+ * atomic, whose modifications all CPUs see in the same
+ * order, each read-modify-write reading the value
+ * written by the one just before it, whatever its
+ * ordering. Ordering only matters relative to the other
+ * levels, for the precise sum: the release ordering of
+ * the addition orders the departure of the carry from
+ * the level below before its arrival here.
+ */
+ res = atomic_long_add_return_release(inc, count);
+ do {
+ inc = percpu_counter_tree_item_carry(res, bit_mask);
+ if (likely(!inc))
+ return;
+ } while (!atomic_long_try_cmpxchg_relaxed(count, &res, res - inc));
+ }
+ item += level_items * stride;
+ n_arity_order = tree_topology.config->n_arity_order[level];
+ level_items >>= n_arity_order;
+ bit_mask <<= n_arity_order;
+ }
+ /* The atomic API has no add with release ordering without a return value. */
+ atomic_long_add_return_release(inc, root);
+}
+
+/* Add @inc to the per-CPU counters @level0 of an activated tree. */
+static inline
+void percpu_counter_tree_add_tree(struct percpu_counter_tree *counter,
+ s8 __percpu *level0, long inc)
+{
+ unsigned long bit_mask = percpu_counter_tree_batch_size(counter);
+ s8 old, new;
+ long v, carry;
+
+ /*
+ * Keep the residual within (-batch, +batch) and propagate its
+ * multiples of the batch size, truncated toward zero. This
+ * hysteresis prevents updates going back and forth around a batch
+ * boundary from propagating each time.
+ *
+ * Preemption stays enabled: the update may go to the level 0
+ * counter of any CPU, and the cmpxchg validates the value read.
+ * After a migration or an interrupt between the read and the
+ * cmpxchg, the cmpxchg either fails and retries with the current
+ * residual of the current CPU, or succeeds on a residual equal to
+ * the value read, which the new value correctly replaces. Each
+ * this_cpu operation is itself safe against preemption and
+ * interrupts, and the carry may propagate from any CPU, see
+ * percpu_counter_tree_add_slowpath().
+ */
+ old = this_cpu_read(*level0);
+ do {
+ v = old + inc;
+ carry = percpu_counter_tree_item_carry(v, bit_mask);
+ new = (s8)(v - carry);
+ } while (!this_cpu_try_cmpxchg(*level0, &old, new));
+ if (likely(!carry))
+ return;
+ percpu_counter_tree_add_slowpath(counter, carry);
+}
+
+/**
+ * percpu_counter_tree_add() - Add to a per-CPU counter tree.
+ * @counter: Counter added to.
+ * @inc: Increment value (either positive or negative).
+ *
+ * Add @inc to a per-CPU counter tree. This is a fast-path which will
+ * typically increment per-CPU counters as long as there is no carry
+ * greater or equal to the counter tree batch size. Before the tree is
+ * activated, add to the atomic bias.
+ */
+void percpu_counter_tree_add(struct percpu_counter_tree *counter, long inc)
+{
+ /*
+ * Pairs with the release of level0 in
+ * percpu_counter_tree_activate_many(): the per-CPU counters are only
+ * reached through this pointer, and the address dependency orders
+ * their initialization before their use, as for rcu_dereference().
+ * No acquire, see "Publication of the tree".
+ */
+ s8 __percpu *level0 = READ_ONCE(counter->level0);
+
+ if (!level0) {
+ atomic_long_add(inc, &counter->atomic_bias);
+ return;
+ }
+ percpu_counter_tree_add_tree(counter, level0, inc);
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_add);
+
+/**
+ * percpu_counter_tree_add_owner() - Add to a per-CPU counter tree as its owner.
+ * @counter: Counter added to.
+ * @inc: Increment value (either positive or negative).
+ *
+ * Like percpu_counter_tree_add(), but before the tree is activated, add
+ * to the bias without atomic instruction. The calls for a given counter
+ * must not run concurrently with each other: they come from a single
+ * context, the owner chosen by the user of the counter, such as a task.
+ */
+void percpu_counter_tree_add_owner(struct percpu_counter_tree *counter, long inc)
+{
+ /* Address dependency, as in percpu_counter_tree_add(). */
+ s8 __percpu *level0 = READ_ONCE(counter->level0);
+
+ if (!level0) {
+ WRITE_ONCE(counter->bias, (long)((unsigned long)READ_ONCE(counter->bias) + inc));
+ return;
+ }
+ percpu_counter_tree_add_tree(counter, level0, inc);
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_add_owner);
+
+/*
+ * The precise sum is the sum of the residuals of all tree levels. A carry
+ * leaves a level before reaching a higher level, and the release ordering
+ * of its arrival orders its departure before it. Reading the levels from
+ * the root down, separated by read barriers, therefore never counts a carry
+ * twice. A carry in flight during the sum may be missed: the result is
+ * precise for a counter without concurrent updates.
+ */
+static
+long percpu_counter_tree_precise_sum_unbiased(struct percpu_counter_tree *counter)
+{
+ unsigned int nr_levels = tree_topology.config->nr_levels, i,
+ level_items[MAX_NR_LEVELS], stride = counter->items_stride;
+ /*
+ * Pairs with the release of level0 in
+ * percpu_counter_tree_activate_many(): approx_sum, the tree items and
+ * the per-CPU counters read below are set before level0.
+ */
+ s8 __percpu *level0 = smp_load_acquire(&counter->level0);
+ unsigned long sum = atomic_long_read(&counter->atomic_bias);
+ atomic_long_t *item;
+ int level, cpu;
+
+ /* Before activation, the atomic bias holds the whole unbiased sum. */
+ if (!level0)
+ return (long)sum;
+ /* Number of items of each level above level 0. */
+ level_items[0] = 1U << tree_topology.nr_cpus_order;
+ for (level = 1; level < (int)nr_levels; level++)
+ level_items[level] = level_items[level - 1] >>
+ tree_topology.config->n_arity_order[level - 1];
+ /* Published before level0, so ordered by its acquire. */
+ item = READ_ONCE(counter->approx_sum);
+ sum += atomic_long_read(item);
+ smp_rmb(); /* Read each level before the level below it. */
+ /* The items of each level precede those of the level above it. */
+ for (level = (int)nr_levels - 1; level >= 1; level--) {
+ item -= level_items[level] * stride;
+ for (i = 0; i < level_items[level]; i++)
+ sum += atomic_long_read(&item[i * stride]);
+ smp_rmb(); /* Read this level before the level below it. */
+ }
+ for_each_possible_cpu(cpu)
+ sum += (long)READ_ONCE(*per_cpu_ptr(level0, cpu));
+ return (long)sum;
+}
+
+/**
+ * percpu_counter_tree_precise_sum() - Return precise counter sum.
+ * @counter: The counter to sum.
+ *
+ * Querying the precise sum is relatively expensive because it needs to
+ * iterate over all CPUs.
+ * This is meant to be used when accuracy is preferred over speed.
+ *
+ * The sum is exact when no update runs concurrently. It takes no lock, so
+ * a concurrent update which carries from level 0 to the levels above can
+ * be counted in part: the sum may read the residual it left below and miss
+ * its carry on the way up, an error in either direction of at most the
+ * carries in flight. A carry is never counted twice.
+ *
+ * Return: The current precise counter sum.
+ */
+long percpu_counter_tree_precise_sum(struct percpu_counter_tree *counter)
+{
+ return percpu_counter_tree_precise_sum_unbiased(counter) + READ_ONCE(counter->bias);
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_precise_sum);
+
+/*
+ * The comparison between counters is on the wrapped difference. It is valid
+ * only when the two counters differ by less than LONG_MAX, and results are
+ * undefined beyond that.
+ */
+static inline
+long wrapped_delta(long a, long b)
+{
+ return (long)((unsigned long)a - (unsigned long)b);
+}
+
+/*
+ * Each counter's approximation lies within its accuracy of its precise sum,
+ * in both directions. The comparison only observes the approximations, so
+ * it must reason about the possible precise values around each observed
+ * approximation, which lie within [approx - accuracy, approx + accuracy]:
+ *
+ * precise_a range
+ * <-----------+----------->
+ * accuracy_a | accuracy_a
+ * approx_a
+ * | precise_b range
+ * | <-----------+----------->
+ * | accuracy_b | accuracy_b
+ * | approx_b
+ * | |
+ * |<------------ gap ------------->|
+ * ----------------------+--------------------------------+-------------> values
+ *
+ * The two precise ranges are disjoint, and the comparison can return a
+ * definitive answer, only once the gap between approximations exceeds the
+ * sum of the two accuracies:
+ *
+ * gap > accuracy_a + accuracy_b
+ *
+ * When comparing against a plain value, the value has zero width, so the
+ * margin reduces to the accuracy of the counter: the value is above the
+ * counter once it exceeds approx + accuracy, and below it once it is less
+ * than approx - accuracy.
+ */
+static
+int compare_delta(long delta, unsigned long accuracy)
+{
+ if (delta >= 0) {
+ if (delta <= accuracy)
+ return 0;
+ else
+ return 1;
+ } else {
+ if (-delta <= accuracy)
+ return 0;
+ else
+ return -1;
+ }
+}
+
+/**
+ * percpu_counter_tree_approximate_compare - Approximated comparison of two counter trees.
+ * @a: First counter to compare.
+ * @b: Second counter to compare.
+ *
+ * Evaluate an approximate comparison of two counter trees. This approximation
+ * comparison is fast, and provides an accurate answer if the counters are
+ * found to be either less than or greater than the other. However, if the
+ * approximated comparison returns 0, the counters respective sums are found to
+ * be within the two counters accuracy range. The two counters are read
+ * independently; the result is not an atomic snapshot of both.
+ * The comparison between counters is on the wrapped difference. It is valid
+ * only when the two counters differ by less than LONG_MAX, and results are
+ * undefined beyond that.
+ *
+ * Return:
+ * * %0 - Counters @a and @b do not differ by more than the sum of their respective
+ * accuracy ranges.
+ * * %-1 - Counter @a less than counter @b.
+ * * %1 - Counter @a is greater than counter @b.
+ */
+int percpu_counter_tree_approximate_compare(struct percpu_counter_tree *a,
+ struct percpu_counter_tree *b)
+{
+ unsigned long accuracy = percpu_counter_tree_accuracy(a) + percpu_counter_tree_accuracy(b);
+
+ /* See the range geometry above compare_delta(). */
+ return compare_delta(wrapped_delta(percpu_counter_tree_approximate_sum(a),
+ percpu_counter_tree_approximate_sum(b)),
+ accuracy);
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_approximate_compare);
+
+/**
+ * percpu_counter_tree_approximate_compare_value - Approximated comparison of a counter tree
+ * against a given value.
+ * @counter: Counter to compare.
+ * @v: Value to compare.
+ *
+ * Evaluate an approximate comparison of a counter tree against a given value.
+ * This approximation comparison is fast, and provides an accurate
+ * answer if the counter is found to be either less than or greater
+ * than the value. However, if the approximated comparison returns
+ * 0, the value is within the counter accuracy range.
+ * The comparison between counters is on the wrapped difference. It is valid
+ * only when @v and the counter differ by less than LONG_MAX, and results are
+ * undefined beyond that.
+ *
+ * Return:
+ * * %0 - The value @v is within the accuracy range of the counter.
+ * * %-1 - The value @v is less than the counter.
+ * * %1 - The value @v is greater than the counter.
+ */
+int percpu_counter_tree_approximate_compare_value(struct percpu_counter_tree *counter, long v)
+{
+ unsigned long accuracy = percpu_counter_tree_accuracy(counter);
+
+ /* See the range geometry above compare_delta(). */
+ return compare_delta(wrapped_delta(v, percpu_counter_tree_approximate_sum(counter)),
+ accuracy);
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_approximate_compare_value);
+
+/**
+ * percpu_counter_tree_precise_compare - Precise comparison of two counter trees.
+ * @a: First counter to compare.
+ * @b: Second counter to compare.
+ *
+ * Evaluate a precise comparison of two counter trees. As an optimization, it
+ * uses the approximate counter comparison to quickly compare counters which
+ * are far apart. Only cases where counter sums are within the accuracy range
+ * require precise counter sums. The two counters are read independently; the
+ * result is not an atomic snapshot of both. With concurrent updates, the
+ * precise sums can count updates in part, see
+ * percpu_counter_tree_precise_sum().
+ * The comparison between counters is on the wrapped difference. It is valid
+ * only when the two counters differ by less than LONG_MAX, and results are
+ * undefined beyond that.
+ *
+ * Return:
+ * * %0 - Counters are equal.
+ * * %-1 - Counter @a less than counter @b.
+ * * %1 - Counter @a is greater than counter @b.
+ */
+int percpu_counter_tree_precise_compare(struct percpu_counter_tree *a,
+ struct percpu_counter_tree *b)
+{
+ long count_a = percpu_counter_tree_approximate_sum(a),
+ count_b = percpu_counter_tree_approximate_sum(b);
+ unsigned long accuracy_a = percpu_counter_tree_accuracy(a),
+ accuracy_b = percpu_counter_tree_accuracy(b);
+ long delta = wrapped_delta(count_a, count_b);
+ int res;
+
+ /* See the range geometry above compare_delta(). */
+ res = compare_delta(delta, accuracy_a + accuracy_b);
+ /* The values are distanced enough for an accurate approximated comparison. */
+ if (res)
+ return res;
+
+ /*
+ * The approximated comparison is within the accuracy range, therefore at least one
+ * precise sum is needed. Sum the counter which has the largest accuracy first.
+ */
+ if (accuracy_b < accuracy_a) {
+ count_a = percpu_counter_tree_precise_sum(a);
+ /* precise_a - approx_b: only b's precise range remains. */
+ res = compare_delta(wrapped_delta(count_a, count_b), accuracy_b);
+ if (res)
+ return res;
+ /* Precise sum of second counter is required. */
+ count_b = percpu_counter_tree_precise_sum(b);
+ } else {
+ count_b = percpu_counter_tree_precise_sum(b);
+ /* approx_a - precise_b: only a's precise range remains. */
+ res = compare_delta(wrapped_delta(count_a, count_b), accuracy_a);
+ if (res)
+ return res;
+ /* Precise sum of second counter is required. */
+ count_a = percpu_counter_tree_precise_sum(a);
+ }
+ delta = wrapped_delta(count_a, count_b);
+ if (delta < 0)
+ return -1;
+ if (delta > 0)
+ return 1;
+ return 0;
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_precise_compare);
+
+/**
+ * percpu_counter_tree_precise_compare_value - Precise comparison of a counter tree against
+ * a given value.
+ * @counter: Counter to compare.
+ * @v: Value to compare.
+ *
+ * Evaluate a precise comparison of a counter tree against a given value. As
+ * an optimization, it uses the approximate counter comparison to quickly
+ * identify whether the counter and value are far apart. Only cases where the
+ * value is within the counter accuracy range require a precise counter sum.
+ * With concurrent updates, the precise sum can count updates in part, see
+ * percpu_counter_tree_precise_sum().
+ * The comparison between counters is on the wrapped difference. It is valid
+ * only when @v and the counter differ by less than LONG_MAX, and results are
+ * undefined beyond that.
+ *
+ * Return:
+ * * %0 - The value @v is equal to the counter.
+ * * %-1 - The value @v is less than the counter.
+ * * %1 - The value @v is greater than the counter.
+ */
+int percpu_counter_tree_precise_compare_value(struct percpu_counter_tree *counter, long v)
+{
+ long count = percpu_counter_tree_approximate_sum(counter), delta;
+ unsigned long accuracy = percpu_counter_tree_accuracy(counter);
+ int res;
+
+ /* See the range geometry above compare_delta(). */
+ res = compare_delta(wrapped_delta(v, count), accuracy);
+ /* The values are distanced enough for an accurate approximated comparison. */
+ if (res)
+ return res;
+
+ /* Precise sum is required. */
+ count = percpu_counter_tree_precise_sum(counter);
+ delta = wrapped_delta(v, count);
+ if (delta < 0)
+ return -1;
+ if (delta > 0)
+ return 1;
+ return 0;
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_precise_compare_value);
+
+/**
+ * percpu_counter_tree_set - Set the counter tree sum to a given value.
+ * @counter: Counter to set.
+ * @v: Value to set.
+ *
+ * Set the counter sum to a given value. It can be useful for instance
+ * to reset the counter sum to 0. Note that even after setting the
+ * counter sum to a given value, the counter sum approximation can
+ * return any value within the accuracy range around that value.
+ *
+ * The caller is responsible for preventing concurrent updates and
+ * precise sums of @counter while it is being set.
+ */
+void percpu_counter_tree_set(struct percpu_counter_tree *counter, long v)
+{
+ WRITE_ONCE(counter->bias, v - percpu_counter_tree_precise_sum_unbiased(counter));
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_set);
+
+/**
+ * percpu_counter_tree_approximate_accuracy - Query the accuracy of a counter tree.
+ * @counter: Counter to query.
+ *
+ * The approximation differs from the precise sum by the residuals not yet propagated
+ * up the tree. The residuals of all levels are symmetric, so the approximation is
+ * within the same distance of the precise sum in both directions.
+ * The accuracy is derived from the hardware topology and the counter tree batch size.
+ * It is invariant for a given counter tree.
+ * The accuracy describes a settled counter: one with no update in progress.
+ * An update in flight between its per-CPU update and its carry propagation is not covered.
+ * Using this function should not be typically required, see the following functions instead:
+ * * percpu_counter_tree_approximate_compare(),
+ * * percpu_counter_tree_approximate_compare_value(),
+ * * percpu_counter_tree_precise_compare(),
+ * * percpu_counter_tree_precise_compare_value().
+ *
+ * Return: The accuracy of the counter: the approximate sum of a settled counter
+ * is within [precise sum - accuracy, precise sum + accuracy].
+ */
+unsigned long percpu_counter_tree_approximate_accuracy(struct percpu_counter_tree *counter)
+{
+ return percpu_counter_tree_accuracy(counter);
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_approximate_accuracy);
+
+/* Calculate the number of tree items and the accuracy multiplier. */
+static
+void __init calculate_topology(void)
+{
+ unsigned int nr_levels = tree_topology.config->nr_levels, level;
+ unsigned int level_items = 1U << tree_topology.nr_cpus_order;
+ unsigned long batch_size = 1;
+
+ /* The root, and the items of each level above level 0. */
+ tree_topology.nr_items = 1;
+ /* One batch per level 0 counter: see percpu_counter_tree_accuracy(). */
+ tree_topology.accuracy_multiplier = nr_cpu_ids;
+
+ /*
+ * The multiplier is derived from a batch size of 1 to speed up
+ * calculating the accuracy at tree initialization. A level N
+ * counter holds at most BATCH(level N) - BATCH(level N - 1) in
+ * either direction.
+ */
+ for (level = 1; level < nr_levels; level++) {
+ unsigned int n_arity_order = tree_topology.config->n_arity_order[level - 1];
+
+ level_items >>= n_arity_order;
+ tree_topology.nr_items += level_items;
+ tree_topology.accuracy_multiplier +=
+ ((batch_size << n_arity_order) - batch_size) * level_items;
+ batch_size <<= n_arity_order;
+ }
+}
+
+#if IS_ENABLED(CONFIG_KUNIT)
+/*
+ * Validate the configuration table: the n-arity orders of each entry must
+ * sum to its order.
+ *
+ * Return: The number of invalid entries.
+ */
+unsigned int percpu_counter_tree_check_config(void)
+{
+ unsigned int order, nr_errors = 0;
+
+ for (order = 0; order < ARRAY_SIZE(per_nr_cpu_order_config); order++) {
+ const struct counter_config *c = &per_nr_cpu_order_config[order];
+ unsigned int level, shift = 0;
+
+ for (level = 0; level < c->nr_levels; level++)
+ shift += c->n_arity_order[level];
+ if (shift != order) {
+ pr_err("order %u: arity sum %u\n", order, shift);
+ nr_errors++;
+ }
+ }
+ return nr_errors;
+}
+EXPORT_SYMBOL_IF_KUNIT(percpu_counter_tree_check_config);
+
+/*
+ * Check the invariants of a settled counter (no update in progress):
+ * - a level 0 counter is within (-BATCH(level 0), +BATCH(level 0)),
+ * - a counter at level N, above level 0, holds a multiple of
+ * BATCH(level N - 1) within (-BATCH(level N), +BATCH(level N)),
+ * - the root holds a multiple of the batch of the last level.
+ * Before activation, a lazily initialized counter has no tree to check.
+ *
+ * Return: The number of violated invariants.
+ */
+unsigned int percpu_counter_tree_check_residuals(struct percpu_counter_tree *counter)
+{
+ unsigned int nr_levels = tree_topology.config->nr_levels, level, level_items,
+ nr_errors = 0, stride = counter->items_stride;
+ unsigned long lo = percpu_counter_tree_batch_size(counter);
+ atomic_long_t *item;
+ long root;
+ int cpu;
+
+ if (!counter->level0)
+ return 0;
+ item = percpu_counter_tree_items(counter);
+ for_each_possible_cpu(cpu) {
+ long v = READ_ONCE(*per_cpu_ptr(counter->level0, cpu));
+
+ if (v <= -(long)lo || v >= (long)lo) {
+ pr_err("cpu %d: level 0 residual %ld, batch %lu\n", cpu, v, lo);
+ nr_errors++;
+ }
+ }
+ level_items = 1U << tree_topology.nr_cpus_order;
+ for (level = 1; level < nr_levels; level++) {
+ unsigned int n_arity_order = tree_topology.config->n_arity_order[level - 1], i;
+ unsigned long hi = lo << n_arity_order;
+
+ level_items >>= n_arity_order;
+ for (i = 0; i < level_items; i++) {
+ long v = atomic_long_read(&item[i * stride]);
+
+ if (v <= -(long)hi || v >= (long)hi || ((unsigned long)v & (lo - 1))) {
+ pr_err("level %u item %u: residual %ld, batch %lu\n",
+ level, i, v, hi);
+ nr_errors++;
+ }
+ }
+ item += level_items * stride;
+ lo = hi;
+ }
+ root = atomic_long_read(counter->approx_sum);
+ if ((unsigned long)root & (lo - 1)) {
+ pr_err("root %ld: not a multiple of %lu\n", root, lo);
+ nr_errors++;
+ }
+ return nr_errors;
+}
+EXPORT_SYMBOL_IF_KUNIT(percpu_counter_tree_check_residuals);
+
+/*
+ * Copy the state of counter @src into counter @dst, initialized with the
+ * same batch size and without concurrent updates: their per-CPU counters,
+ * tree items and biases. Both trees are active.
+ */
+void percpu_counter_tree_copy_state(struct percpu_counter_tree *dst,
+ struct percpu_counter_tree *src)
+{
+ atomic_long_t *dst_items = percpu_counter_tree_items(dst),
+ *src_items = percpu_counter_tree_items(src);
+ unsigned int pos;
+ int cpu;
+
+ for_each_possible_cpu(cpu)
+ *per_cpu_ptr(dst->level0, cpu) = *per_cpu_ptr(src->level0, cpu);
+ for (pos = 0; pos < tree_topology.nr_items; pos++)
+ atomic_long_set(&dst_items[pos * dst->items_stride],
+ atomic_long_read(&src_items[pos * src->items_stride]));
+ dst->bias = src->bias;
+ atomic_long_set(&dst->atomic_bias, atomic_long_read(&src->atomic_bias));
+}
+EXPORT_SYMBOL_IF_KUNIT(percpu_counter_tree_copy_state);
+#endif
+
+/* Room for the fan-out of each level, up to 7 digits and a comma, and a NUL. */
+#define SHAPE_BUF_SIZE (8 * MAX_NR_LEVELS + 1)
+
+/* Format the fan-out of each level, from level 0 to the root, e.g. "8,4,8". */
+static
+int format_shape(char *buf, size_t size, const struct counter_config *config)
+{
+ unsigned int level;
+ int len = 0;
+
+ for (level = 0; level < config->nr_levels; level++)
+ len += scnprintf(buf + len, size - len, "%s%u", level ? "," : "",
+ 1U << config->n_arity_order[level]);
+ return len;
+}
+
+static
+int arity_param_set(const char *val, const struct kernel_param *kp)
+{
+ /* Applied once by percpu_counter_tree_subsystem_init(). */
+ if (tree_topology.config)
+ return -EBUSY;
+ arity_param = val;
+ return 0;
+}
+
+static
+int arity_param_get(char *buffer, const struct kernel_param *kp)
+{
+ int len;
+
+ if (!tree_topology.config)
+ return sprintf(buffer, "%s\n", arity_param ? : "");
+ /* The buffer is a page, larger than any shape and its newline. */
+ len = format_shape(buffer, SHAPE_BUF_SIZE, tree_topology.config);
+ buffer[len++] = '\n';
+ return len;
+}
+
+static const struct kernel_param_ops arity_param_ops = {
+ .set = arity_param_set,
+ .get = arity_param_get,
+};
+
+/*
+ * percpu_counter_tree.arity=<fan-out>[,<fan-out>...] overrides the tree
+ * shape chosen for nr_cpu_ids, to compare shapes on a given machine.
+ * Reading the parameter shows the shape in use.
+ */
+module_param_cb(arity, &arity_param_ops, NULL, 0444);
+MODULE_PARM_DESC(arity, "Fan-out of each tree level from level 0 to the root, e.g. 8,4,8");
+
+/*
+ * Parse a list of fan-outs, from level 0 to the root, into @config. Each
+ * fan-out is a power of 2 of at least 2, and their product is nr_cpu_ids
+ * rounded up to a power of 2.
+ *
+ * Return: 0 on success, -EINVAL if the list does not describe a tree
+ * covering the CPU numbers below nr_cpu_ids.
+ */
+static
+int __init parse_arity(const char *str, struct counter_config *config)
+{
+ int ints[MAX_NR_LEVELS + 2]; /* Count, levels, and one extra to detect overflow. */
+ unsigned int level, order = 0;
+
+ if (*get_options(str, ARRAY_SIZE(ints), ints) || ints[0] > MAX_NR_LEVELS)
+ return -EINVAL;
+ config->nr_levels = ints[0];
+ for (level = 0; level < config->nr_levels; level++) {
+ int fan_out = ints[level + 1];
+
+ if (fan_out < 2 || !is_power_of_2(fan_out))
+ return -EINVAL;
+ config->n_arity_order[level] = ilog2(fan_out);
+ order += config->n_arity_order[level];
+ if (order > tree_topology.nr_cpus_order)
+ return -EINVAL;
+ }
+ if (order != tree_topology.nr_cpus_order)
+ return -EINVAL;
+ return 0;
+}
+
+int __init percpu_counter_tree_subsystem_init(void)
+{
+ const char *origin = "default";
+ char shape[SHAPE_BUF_SIZE] = "";
+
+ tree_topology.nr_cpus_order = get_count_order(nr_cpu_ids);
+ if (tree_topology.nr_cpus_order >= ARRAY_SIZE(per_nr_cpu_order_config)) {
+ pr_err("Unsupported number of CPUs (%u)\n", nr_cpu_ids);
+ return -EINVAL;
+ }
+ tree_topology.config = &per_nr_cpu_order_config[tree_topology.nr_cpus_order];
+ if (arity_param && *arity_param) {
+ if (parse_arity(arity_param, &tree_topology.boot_config)) {
+ pr_warn("Invalid arity \"%s\" for %u CPUs, using the default\n",
+ arity_param, nr_cpu_ids);
+ } else {
+ tree_topology.config = &tree_topology.boot_config;
+ origin = "boot parameter";
+ }
+ }
+ calculate_topology();
+ tree_topology.items_cachep =
+ kmem_cache_create("percpu_counter_tree_items",
+ tree_topology.nr_items * SMP_CACHE_BYTES,
+ SMP_CACHE_BYTES, SLAB_PANIC, NULL);
+ format_shape(shape, sizeof(shape), tree_topology.config);
+ pr_info("%u CPUs, fan-out %s (%s), %u tree items\n",
+ nr_cpu_ids, tree_topology.nr_cpus_order ? shape : "none", origin,
+ tree_topology.nr_items);
+ return 0;
+}
diff --git a/lib/tests/Makefile b/lib/tests/Makefile
index df0537e1b91faf..d158a8be8a1944 100644
--- a/lib/tests/Makefile
+++ b/lib/tests/Makefile
@@ -59,4 +59,7 @@ obj-$(CONFIG_UTIL_MACROS_KUNIT) += util_macros_kunit.o
obj-$(CONFIG_RATELIMIT_KUNIT_TEST) += test_ratelimit.o
obj-$(CONFIG_UUID_KUNIT_TEST) += uuid_kunit.o
+obj-$(CONFIG_PERCPU_COUNTER_TREE_KUNIT_TEST) += percpu_counter_tree_kunit.o
+obj-$(CONFIG_PERCPU_COUNTER_TREE_BENCH) += percpu_counter_tree_bench.o
+
obj-$(CONFIG_TEST_RUNTIME_MODULE) += module/
diff --git a/lib/tests/percpu_counter_tree_bench.c b/lib/tests/percpu_counter_tree_bench.c
new file mode 100644
index 00000000000000..bf260bef196a58
--- /dev/null
+++ b/lib/tests/percpu_counter_tree_bench.c
@@ -0,0 +1,604 @@
+// SPDX-License-Identifier: GPL-2.0+ OR MIT
+/*
+ * Benchmark of hierarchical per-CPU counters against struct percpu_counter.
+ *
+ * update: ns per percpu_counter_tree_add() per thread, one kthread pinned
+ * on each online CPU, on one shared counter, on one counter per
+ * thread, or on the counters of a few mm-like groups of 4 counters
+ * initialized together. The "single" variant gives each thread a
+ * lazily initialized counter updated with
+ * percpu_counter_tree_add_owner(), like a single-threaded process.
+ * The "percpu_counter" variants use percpu_counter_add() with the
+ * default batch, like the RSS counters before hierarchical counters.
+ * lifecycle: ns per init_many(4) + destroy_many(4), like fork + exit of an
+ * mm, from one thread or from all online CPUs at once; "lazy" uses
+ * init_many_lazy(4), like a single-threaded process.
+ * churn: like lifecycle with a few updates, keeping one in @pin_period sets
+ * of counters alive until the end, like long-lived processes.
+ *
+ * Each result ends with the implementation measured: "tree" for the
+ * hierarchical counters, "percpu_counter" for struct percpu_counter.
+ *
+ * The knobs to try on a given machine are:
+ * - the batch size: percpu_counter_tree_bench.batch=<2..128> (power of 2),
+ * or "modprobe percpu_counter_tree_bench batch=<n>",
+ * - the tree shape: percpu_counter_tree.arity=<fan-out>,... on the kernel
+ * command line, from level 0 to the root. The product of the fan-outs
+ * is nr_cpu_ids rounded up to a power of 2, e.g. "16" (flat) or "4,4"
+ * when nr_cpu_ids is 16. The shape in use is shown in
+ * /sys/module/percpu_counter_tree/parameters/arity.
+ *
+ * For example, in a VM:
+ * ./tools/testing/kunit/kunit.py run --arch=x86_64 \
+ * --kconfig_add CONFIG_PERCPU_COUNTER_TREE_BENCH=y \
+ * --kconfig_add CONFIG_NR_CPUS=256 --qemu_args="-smp 64 -m 8192" \
+ * --kernel_args percpu_counter_tree.arity=8,8 \
+ * --kernel_args percpu_counter_tree_bench.batch=16 \
+ * --timeout 3000 percpu_counter_tree_bench
+ */
+#include <kunit/test.h>
+#include <linux/percpu_counter_tree.h>
+#include <linux/percpu_counter.h>
+#include <linux/kthread.h>
+#include <linux/random.h>
+#include <linux/slab.h>
+#include <linux/delay.h>
+#include <linux/ktime.h>
+#include <linux/math64.h>
+#include <linux/sched/task.h>
+#include <linux/cpuhplock.h>
+#include <linux/moduleparam.h>
+
+/* Like the RSS counters by default. */
+static unsigned int batch = 32;
+module_param(batch, uint, 0444);
+MODULE_PARM_DESC(batch, "Batch size of the hierarchical counters (power of 2, 2 to 128)");
+
+#define BENCH_MS 300
+#define LIFECYCLE_NR 4 /* Counters per mm. */
+#define LIFECYCLE_ITER 2000
+#define CHURN_ITER 4096
+
+enum { W_MONO, W_BOUNCE, W_RAND32, W_RAND512, NR_W };
+static const char * const wname[] = { "monotonic +1", "bounce +-1", "random +-32", "random +-512" };
+
+enum bench_impl { IMPL_TREE, IMPL_PERCPU_COUNTER };
+static const char * const impl_name[] = { "tree", "percpu_counter" };
+
+struct bench_shared {
+ atomic_t ready;
+ int wl;
+ unsigned int pin_period; /* churn: keep one in pin_period sets, 0: none. */
+ enum bench_impl impl;
+ bool go, stop;
+ bool owner; /* update: percpu_counter_tree_add_owner(). */
+ bool lifecycle;
+ bool lazy; /* lifecycle: percpu_counter_tree_init_many_lazy(). */
+};
+
+struct bench_thread {
+ u64 ns;
+ unsigned long ops;
+ struct bench_shared *sh;
+ struct percpu_counter_tree *c;
+ struct percpu_counter *pc; /* update: IMPL_PERCPU_COUNTER. */
+ struct percpu_counter_tree *pinned; /* churn: IMPL_TREE sets kept alive. */
+ struct percpu_counter *pinned_pc; /* churn: IMPL_PERCPU_COUNTER sets kept alive. */
+ struct task_struct *task;
+ unsigned int nr_pinned;
+ u32 seed;
+} ____cacheline_aligned_in_smp;
+
+/*
+ * Buffers are allocated through KUnit, which frees them if an assertion
+ * aborts the test, and are freed explicitly otherwise, since the benchmark
+ * runs many steps within one test.
+ */
+KUNIT_DEFINE_ACTION_WRAPPER(bench_kvfree, kvfree, const void *);
+
+static void *bench_kvzalloc(struct kunit *test, size_t size)
+{
+ void *p = kvzalloc(size, GFP_KERNEL);
+
+ if (p && kunit_add_action_or_reset(test, bench_kvfree, p))
+ return NULL;
+ return p;
+}
+
+static void bench_kvfree_now(struct kunit *test, void *p)
+{
+ kunit_release_action(test, bench_kvfree, p);
+}
+
+/* Wait for the start signal; return true if the benchmark is cancelled. */
+static bool bench_wait_go(struct bench_shared *sh)
+{
+ atomic_inc(&sh->ready);
+ /* Pairs with bench_start(): a cancellation sets stop before go. */
+ while (!smp_load_acquire(&sh->go))
+ cond_resched();
+ return READ_ONCE(sh->stop);
+}
+
+static int bench_update_fn(void *data)
+{
+ struct bench_thread *t = data;
+ struct bench_shared *sh = t->sh;
+ unsigned long ops = 0;
+ long sign = -1;
+ u32 x = t->seed;
+ int i;
+
+ if (bench_wait_go(sh))
+ return 0;
+ while (!READ_ONCE(sh->stop)) {
+ for (i = 0; i < 1024; i++) {
+ long inc;
+
+ switch (sh->wl) {
+ case W_MONO:
+ inc = 1;
+ break;
+ case W_BOUNCE:
+ inc = sign;
+ sign = -sign;
+ break;
+ default:
+ x ^= x << 13;
+ x ^= x >> 17;
+ x ^= x << 5;
+ inc = sh->wl == W_RAND32 ? 32 : 512;
+ if (x & 1)
+ inc = -inc;
+ break;
+ }
+ if (sh->impl == IMPL_PERCPU_COUNTER)
+ percpu_counter_add(t->pc, inc);
+ else if (sh->owner)
+ percpu_counter_tree_add_owner(t->c, inc);
+ else
+ percpu_counter_tree_add(t->c, inc);
+ }
+ ops += 1024;
+ cond_resched();
+ }
+ t->ops = ops;
+ return 0;
+}
+
+static int bench_lifecycle_fn(void *data)
+{
+ struct bench_thread *t = data;
+ struct percpu_counter_tree c[LIFECYCLE_NR];
+ struct percpu_counter pc[LIFECYCLE_NR];
+ u64 t0;
+ int i;
+
+ if (bench_wait_go(t->sh))
+ return 0;
+ t0 = ktime_get_ns();
+ for (i = 0; t->sh->impl == IMPL_PERCPU_COUNTER && i < LIFECYCLE_ITER; i++) {
+ if (percpu_counter_init_many(pc, 0, GFP_KERNEL, LIFECYCLE_NR))
+ break;
+ percpu_counter_add(&pc[0], 1);
+ percpu_counter_add(&pc[0], -1);
+ percpu_counter_destroy_many(pc, LIFECYCLE_NR);
+ cond_resched();
+ }
+ for (; t->sh->impl == IMPL_TREE && t->sh->lazy && i < LIFECYCLE_ITER; i++) {
+ if (percpu_counter_tree_init_many_lazy(c, LIFECYCLE_NR, batch))
+ break;
+ percpu_counter_tree_add_owner(&c[0], 1);
+ percpu_counter_tree_add_owner(&c[0], -1);
+ percpu_counter_tree_destroy_many(c, LIFECYCLE_NR);
+ cond_resched();
+ }
+ for (; t->sh->impl == IMPL_TREE && !t->sh->lazy && i < LIFECYCLE_ITER; i++) {
+ if (percpu_counter_tree_init_many(c, LIFECYCLE_NR, batch, GFP_KERNEL))
+ break;
+ percpu_counter_tree_add(&c[0], 1);
+ percpu_counter_tree_add(&c[0], -1);
+ percpu_counter_tree_destroy_many(c, LIFECYCLE_NR);
+ cond_resched();
+ }
+ t->ns = ktime_get_ns() - t0;
+ t->ops = i;
+ return 0;
+}
+
+static int bench_churn_fn(void *data)
+{
+ struct bench_thread *t = data;
+ unsigned int pin_period = t->sh->pin_period;
+ struct percpu_counter_tree tmp[LIFECYCLE_NR], *c;
+ struct percpu_counter tmp_pc[LIFECYCLE_NR], *pc;
+ u64 t0;
+ int i;
+
+ if (bench_wait_go(t->sh))
+ return 0;
+ t0 = ktime_get_ns();
+ for (i = 0; t->sh->impl == IMPL_PERCPU_COUNTER && i < CHURN_ITER; i++) {
+ bool pin = pin_period && !(i % pin_period);
+
+ pc = pin ? &t->pinned_pc[t->nr_pinned * LIFECYCLE_NR] : tmp_pc;
+ if (percpu_counter_init_many(pc, 0, GFP_KERNEL, LIFECYCLE_NR))
+ break;
+ percpu_counter_add(&pc[0], 100);
+ percpu_counter_add(&pc[1], -77);
+ percpu_counter_add(&pc[2], 1);
+ if (pin)
+ t->nr_pinned++;
+ else
+ percpu_counter_destroy_many(pc, LIFECYCLE_NR);
+ cond_resched();
+ }
+ for (; t->sh->impl == IMPL_TREE && i < CHURN_ITER; i++) {
+ bool pin = pin_period && !(i % pin_period);
+
+ c = pin ? &t->pinned[t->nr_pinned * LIFECYCLE_NR] : tmp;
+ if (percpu_counter_tree_init_many(c, LIFECYCLE_NR, batch, GFP_KERNEL))
+ break;
+ percpu_counter_tree_add(&c[0], 100);
+ percpu_counter_tree_add(&c[1], -77);
+ percpu_counter_tree_add(&c[2], 1);
+ if (pin)
+ t->nr_pinned++;
+ else
+ percpu_counter_tree_destroy_many(c, LIFECYCLE_NR);
+ cond_resched();
+ }
+ t->ns = ktime_get_ns() - t0;
+ t->ops = i;
+ return 0;
+}
+
+/* List at most @max online CPUs in @cpus, and return their number. */
+static unsigned int bench_cpus(int *cpus, unsigned int max)
+{
+ unsigned int n = 0;
+ int cpu;
+
+ for_each_online_cpu(cpu) {
+ if (n == max)
+ break;
+ cpus[n++] = cpu;
+ }
+ return n;
+}
+
+static void bench_stop(struct bench_thread *t, unsigned int n)
+{
+ unsigned int i;
+
+ for (i = 0; i < n; i++)
+ kthread_stop_put(t[i].task);
+}
+
+/*
+ * Start a thread on each of the @n CPUs of @cpus, waiting for the start
+ * signal. On failure, cancel and join the threads already started.
+ */
+static int bench_start(struct bench_shared *sh, struct bench_thread *t, const int *cpus,
+ unsigned int n)
+{
+ unsigned int i;
+
+ for (i = 0; i < n; i++) {
+ t[i].sh = sh;
+ t[i].task = kthread_create_on_cpu(sh->pin_period != UINT_MAX ? bench_churn_fn :
+ sh->lifecycle ? bench_lifecycle_fn :
+ bench_update_fn, &t[i], cpus[i], "hpcc_bench/%u");
+ if (IS_ERR(t[i].task)) {
+ int ret = PTR_ERR(t[i].task);
+
+ WRITE_ONCE(sh->stop, true);
+ /* Pairs with bench_wait_go(): the threads see stop with go. */
+ smp_store_release(&sh->go, true);
+ bench_stop(t, i);
+ return ret;
+ }
+ get_task_struct(t[i].task);
+ wake_up_process(t[i].task);
+ }
+ while (atomic_read(&sh->ready) < n)
+ usleep_range(1000, 2000);
+ return 0;
+}
+
+enum bench_mode { BENCH_SHARED, BENCH_PRIVATE, BENCH_MM, BENCH_SINGLE };
+static const char * const mode_name[] = { "shared", "private", "mm", "single" };
+
+/*
+ * BENCH_SHARED: all threads update one counter.
+ * BENCH_PRIVATE: each thread updates its own counter (a separate mm).
+ * BENCH_MM: blocks of n / LIFECYCLE_NR threads each update one counter of
+ * one mm. Its counters share the cache lines of their tree items, and
+ * CPUs sharing an item are strided, so each line is updated for every
+ * counter.
+ * BENCH_SINGLE: each thread owns a lazily initialized counter, like a
+ * single-threaded process, never activated. Its IMPL_PERCPU_COUNTER
+ * comparison is BENCH_PRIVATE.
+ */
+/* Counters @i of bench_update(), @c_stride and @pc_stride bytes apart. */
+#define BENCH_C(i) ((struct percpu_counter_tree *)((char *)c + (i) * c_stride))
+#define BENCH_PC(i) ((struct percpu_counter *)((char *)pc + (i) * pc_stride))
+
+static void bench_update(struct kunit *test, enum bench_mode mode, int wl,
+ enum bench_impl impl)
+{
+ bool shared = mode == BENCH_SHARED || mode == BENCH_MM;
+ unsigned int n, i, nr_counters, nr_init = 0;
+ struct bench_shared sh = { .wl = wl, .pin_period = UINT_MAX, .impl = impl,
+ .owner = mode == BENCH_SINGLE };
+ struct percpu_counter_tree *c;
+ struct percpu_counter *pc;
+ struct bench_thread *t;
+ unsigned long ops = 0;
+ unsigned int *perm;
+ size_t c_stride, pc_stride;
+ void *c_mem, *pc_mem;
+ u64 t0, t1;
+ int *cpus;
+ int ret;
+
+ cpus = kunit_kcalloc(test, nr_cpu_ids, sizeof(*cpus), GFP_KERNEL);
+ t = bench_kvzalloc(test, array_size(nr_cpu_ids, sizeof(*t)));
+ perm = kunit_kcalloc(test, nr_cpu_ids, sizeof(*perm), GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, cpus);
+ KUNIT_ASSERT_NOT_NULL(test, t);
+ KUNIT_ASSERT_NOT_NULL(test, perm);
+ /*
+ * Set up the counters for the CPUs online now, before taking
+ * cpus_read_lock(), which a failed assertion would leave held. The
+ * threads then run on the CPUs online under the lock, at most as many.
+ */
+ n = num_online_cpus();
+ nr_counters = mode == BENCH_SHARED ? 1 : mode == BENCH_MM ? LIFECYCLE_NR : n;
+ /*
+ * Each private counter is written by its own thread: give each its
+ * own cache line, as the RSS counters of separate mms, to avoid false
+ * sharing. The counters of one mm stay packed, as in mm_struct.
+ */
+ c_stride = shared ? sizeof(*c) : ALIGN(sizeof(*c), SMP_CACHE_BYTES);
+ pc_stride = shared ? sizeof(*pc) : ALIGN(sizeof(*pc), SMP_CACHE_BYTES);
+ c_mem = kunit_kzalloc(test, nr_counters * c_stride + SMP_CACHE_BYTES, GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, c_mem);
+ c = PTR_ALIGN(c_mem, SMP_CACHE_BYTES);
+ pc_mem = kunit_kzalloc(test, nr_counters * pc_stride + SMP_CACHE_BYTES, GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, pc_mem);
+ pc = PTR_ALIGN(pc_mem, SMP_CACHE_BYTES);
+ /*
+ * Each private counter stands for a separate mm: initialize them
+ * separately so they do not share the cache lines of their tree items.
+ */
+ if (impl == IMPL_PERCPU_COUNTER && mode == BENCH_MM) {
+ if (!percpu_counter_init_many(pc, 0, GFP_KERNEL, nr_counters))
+ nr_init = nr_counters;
+ } else if (impl == IMPL_PERCPU_COUNTER) {
+ for (; nr_init < nr_counters; nr_init++)
+ if (percpu_counter_init(BENCH_PC(nr_init), 0, GFP_KERNEL))
+ break;
+ } else if (mode == BENCH_MM) {
+ if (!percpu_counter_tree_init_many(c, nr_counters, batch, GFP_KERNEL))
+ nr_init = nr_counters;
+ } else {
+ for (; nr_init < nr_counters; nr_init++) {
+ if (mode == BENCH_SINGLE ?
+ percpu_counter_tree_init_many_lazy(BENCH_C(nr_init), 1, batch) :
+ percpu_counter_tree_init(BENCH_C(nr_init), batch, GFP_KERNEL))
+ break;
+ }
+ }
+ if (nr_init < nr_counters) {
+ KUNIT_FAIL(test, "cannot initialize %u counters", nr_counters);
+ goto destroy;
+ }
+ cpus_read_lock();
+ n = bench_cpus(cpus, n);
+ /* Random placement of the private counters. */
+ for (i = 0; i < n; i++)
+ perm[i] = mode == BENCH_SHARED ? 0 : mode == BENCH_MM ? i * LIFECYCLE_NR / n : i;
+ for (i = n - 1; !shared && i > 0; i--)
+ swap(perm[i], perm[get_random_u32_below(i + 1)]);
+ for (i = 0; i < n; i++) {
+ t[i].c = BENCH_C(perm[i]);
+ t[i].pc = BENCH_PC(perm[i]);
+ t[i].seed = get_random_u32() | 1;
+ }
+ ret = bench_start(&sh, t, cpus, n);
+ if (ret) {
+ cpus_read_unlock();
+ KUNIT_FAIL(test, "cannot start the benchmark threads: %d", ret);
+ goto destroy;
+ }
+ t0 = ktime_get_ns();
+ WRITE_ONCE(sh.go, true);
+ msleep(BENCH_MS);
+ WRITE_ONCE(sh.stop, true);
+ t1 = ktime_get_ns();
+ bench_stop(t, n);
+ cpus_read_unlock();
+ for (i = 0; i < n; i++)
+ ops += t[i].ops;
+ for (i = 0; impl == IMPL_TREE && i < nr_counters; i++)
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_check_residuals(BENCH_C(i)));
+ kunit_info(test, "update %-8s %-14s %3u thr %10llu ns/op/thr %s\n",
+ mode_name[mode], wname[wl], n,
+ ops ? div64_ul((t1 - t0) * n, ops) : 0, impl_name[impl]);
+destroy:
+ if (impl == IMPL_PERCPU_COUNTER && mode == BENCH_MM) {
+ if (nr_init)
+ percpu_counter_destroy_many(pc, nr_counters);
+ } else if (impl == IMPL_PERCPU_COUNTER) {
+ for (i = 0; i < nr_init; i++)
+ percpu_counter_destroy(BENCH_PC(i));
+ } else if (mode == BENCH_MM) {
+ if (nr_init)
+ percpu_counter_tree_destroy_many(c, nr_counters);
+ } else {
+ for (i = 0; i < nr_init; i++)
+ percpu_counter_tree_destroy(BENCH_C(i));
+ }
+ kunit_kfree(test, pc_mem);
+ kunit_kfree(test, c_mem);
+ kunit_kfree(test, perm);
+ bench_kvfree_now(test, t);
+ kunit_kfree(test, cpus);
+}
+
+static void bench_lifecycle(struct kunit *test, bool all, enum bench_impl impl, bool lazy)
+{
+ struct bench_shared sh = { .lifecycle = true, .pin_period = UINT_MAX,
+ .impl = impl, .lazy = lazy };
+ unsigned int n, i;
+ struct bench_thread *t;
+ unsigned long ops = 0;
+ u64 ns = 0;
+ int *cpus;
+ int ret;
+
+ cpus = kunit_kcalloc(test, nr_cpu_ids, sizeof(*cpus), GFP_KERNEL);
+ t = bench_kvzalloc(test, array_size(nr_cpu_ids, sizeof(*t)));
+ KUNIT_ASSERT_NOT_NULL(test, cpus);
+ KUNIT_ASSERT_NOT_NULL(test, t);
+ cpus_read_lock();
+ n = bench_cpus(cpus, all ? num_online_cpus() : 1);
+ ret = bench_start(&sh, t, cpus, n);
+ if (ret) {
+ cpus_read_unlock();
+ KUNIT_FAIL(test, "cannot start the benchmark threads: %d", ret);
+ goto free;
+ }
+ WRITE_ONCE(sh.go, true);
+ bench_stop(t, n);
+ cpus_read_unlock();
+ for (i = 0; i < n; i++) {
+ ops += t[i].ops;
+ ns += t[i].ns;
+ }
+ KUNIT_EXPECT_EQ(test, ops, (unsigned long)n * LIFECYCLE_ITER);
+ kunit_info(test, "lifecycle init_many%s(%d)+destroy_many %3u thr %10llu ns/op %s\n",
+ lazy ? "_lazy" : "", LIFECYCLE_NR, n, ops ? div64_ul(ns, ops) : 0,
+ impl_name[impl]);
+free:
+ bench_kvfree_now(test, t);
+ kunit_kfree(test, cpus);
+}
+
+static void bench_churn(struct kunit *test, bool all, unsigned int pin_period,
+ enum bench_impl impl)
+{
+ struct bench_shared sh = { .pin_period = pin_period, .impl = impl };
+ unsigned int n, i, j, max_pinned = pin_period ? DIV_ROUND_UP(CHURN_ITER, pin_period) : 0;
+ unsigned long ops = 0, live = 0;
+ unsigned int nr_alloc;
+ struct bench_thread *t;
+ u64 ns = 0;
+ int *cpus;
+ int ret;
+
+ cpus = kunit_kcalloc(test, nr_cpu_ids, sizeof(*cpus), GFP_KERNEL);
+ t = bench_kvzalloc(test, array_size(nr_cpu_ids, sizeof(*t)));
+ KUNIT_ASSERT_NOT_NULL(test, cpus);
+ KUNIT_ASSERT_NOT_NULL(test, t);
+ /* Allocate before taking cpus_read_lock(), as in bench_update(). */
+ nr_alloc = all ? num_online_cpus() : 1;
+ for (i = 0; i < nr_alloc; i++) {
+ t[i].pinned = bench_kvzalloc(test, array_size(max_pinned * LIFECYCLE_NR + 1,
+ sizeof(*t[i].pinned)));
+ t[i].pinned_pc = bench_kvzalloc(test, array_size(max_pinned * LIFECYCLE_NR + 1,
+ sizeof(*t[i].pinned_pc)));
+ KUNIT_ASSERT_NOT_NULL(test, t[i].pinned_pc);
+ KUNIT_ASSERT_NOT_NULL(test, t[i].pinned);
+ }
+ cpus_read_lock();
+ n = bench_cpus(cpus, nr_alloc);
+ ret = bench_start(&sh, t, cpus, n);
+ if (ret) {
+ cpus_read_unlock();
+ KUNIT_FAIL(test, "cannot start the benchmark threads: %d", ret);
+ goto destroy;
+ }
+ WRITE_ONCE(sh.go, true);
+ bench_stop(t, n);
+ cpus_read_unlock();
+ for (i = 0; i < n; i++) {
+ ops += t[i].ops;
+ ns += t[i].ns;
+ live += t[i].nr_pinned * LIFECYCLE_NR;
+ }
+ KUNIT_EXPECT_EQ(test, ops, (unsigned long)n * CHURN_ITER);
+ kunit_info(test, "churn pin 1/%-4u %3u thr %8llu ns/op live %6lu %s\n",
+ pin_period, n, ops ? div64_ul(ns, ops) : 0, live, impl_name[impl]);
+destroy:
+ for (i = 0; i < nr_alloc; i++) {
+ for (j = 0; j < t[i].nr_pinned; j++) {
+ if (impl == IMPL_PERCPU_COUNTER)
+ percpu_counter_destroy_many(&t[i].pinned_pc[j * LIFECYCLE_NR],
+ LIFECYCLE_NR);
+ else
+ percpu_counter_tree_destroy_many(&t[i].pinned[j * LIFECYCLE_NR],
+ LIFECYCLE_NR);
+ }
+ bench_kvfree_now(test, t[i].pinned_pc);
+ bench_kvfree_now(test, t[i].pinned);
+ }
+ bench_kvfree_now(test, t);
+ kunit_kfree(test, cpus);
+}
+
+static void hpcc_bench(struct kunit *test)
+{
+ static const unsigned int pin_periods[] = { 0, 256, 64, 16, 4 };
+ struct percpu_counter_tree probe;
+ unsigned int p;
+ int wl;
+
+ if (batch < 2 || !is_power_of_2(batch) || batch > PERCPU_COUNTER_TREE_MAX_BATCH)
+ kunit_skip(test, "invalid batch size %u", batch);
+ KUNIT_ASSERT_EQ(test, 0, percpu_counter_tree_init(&probe, batch, GFP_KERNEL));
+ kunit_info(test, "%u CPUs online, nr_cpu_ids %u\n", num_online_cpus(), nr_cpu_ids);
+ kunit_info(test, "%s batch %u, accuracy +-%lu\n", impl_name[IMPL_TREE], batch,
+ percpu_counter_tree_approximate_accuracy(&probe));
+ kunit_info(test, "%s batch %d, accuracy +-%lu\n", impl_name[IMPL_PERCPU_COUNTER],
+ percpu_counter_batch, (unsigned long)nr_cpu_ids * (percpu_counter_batch - 1));
+ percpu_counter_tree_destroy(&probe);
+ bench_lifecycle(test, false, IMPL_TREE, false);
+ bench_lifecycle(test, false, IMPL_TREE, true);
+ bench_lifecycle(test, false, IMPL_PERCPU_COUNTER, false);
+ bench_lifecycle(test, true, IMPL_TREE, false);
+ bench_lifecycle(test, true, IMPL_TREE, true);
+ bench_lifecycle(test, true, IMPL_PERCPU_COUNTER, false);
+ for (p = 0; p < ARRAY_SIZE(pin_periods); p++) {
+ bench_churn(test, false, pin_periods[p], IMPL_TREE);
+ bench_churn(test, false, pin_periods[p], IMPL_PERCPU_COUNTER);
+ bench_churn(test, true, pin_periods[p], IMPL_TREE);
+ bench_churn(test, true, pin_periods[p], IMPL_PERCPU_COUNTER);
+ }
+ for (wl = 0; wl < NR_W; wl++) {
+ bench_update(test, BENCH_SHARED, wl, IMPL_TREE);
+ bench_update(test, BENCH_SHARED, wl, IMPL_PERCPU_COUNTER);
+ bench_update(test, BENCH_PRIVATE, wl, IMPL_TREE);
+ bench_update(test, BENCH_PRIVATE, wl, IMPL_PERCPU_COUNTER);
+ bench_update(test, BENCH_MM, wl, IMPL_TREE);
+ bench_update(test, BENCH_MM, wl, IMPL_PERCPU_COUNTER);
+ bench_update(test, BENCH_SINGLE, wl, IMPL_TREE);
+ }
+}
+
+static struct kunit_case hpcc_bench_cases[] = {
+ KUNIT_CASE_SLOW(hpcc_bench),
+ {}
+};
+
+static struct kunit_suite hpcc_bench_suite = {
+ .name = "percpu_counter_tree_bench",
+ .test_cases = hpcc_bench_cases,
+};
+
+kunit_test_suite(hpcc_bench_suite);
+
+MODULE_DESCRIPTION("Benchmark for hierarchical per-CPU counters");
+MODULE_LICENSE("Dual MIT/GPL");
+MODULE_IMPORT_NS("EXPORTED_FOR_KUNIT_TESTING");
diff --git a/lib/tests/percpu_counter_tree_kunit.c b/lib/tests/percpu_counter_tree_kunit.c
new file mode 100644
index 00000000000000..2f5852b1f52140
--- /dev/null
+++ b/lib/tests/percpu_counter_tree_kunit.c
@@ -0,0 +1,1050 @@
+// SPDX-License-Identifier: GPL-2.0+ OR MIT
+// SPDX-FileCopyrightText: 2026 Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
+
+#include <kunit/test.h>
+#include <linux/delay.h>
+#include <linux/mutex.h>
+#include <linux/percpu_counter_tree.h>
+#include <linux/kthread.h>
+#include <linux/wait.h>
+#include <linux/random.h>
+#include <linux/cpuhplock.h>
+#include <linux/slab.h>
+#include <linux/sched/task.h>
+#include <linux/workqueue.h>
+
+struct multi_thread_test_data {
+ struct work_struct work;
+ long increment;
+ unsigned int nr_inc;
+ int counter_index;
+};
+
+#define NR_COUNTERS 2
+
+/* Hierarchical per-CPU counter instances. */
+static struct percpu_counter_tree counter[NR_COUNTERS];
+
+/* Global atomic counters for validation. */
+static atomic_long_t global_counter[NR_COUNTERS];
+
+static DECLARE_WAIT_QUEUE_HEAD(kernel_threads_wq);
+static atomic_t kernel_threads_to_run;
+
+static void complete_work(void)
+{
+ if (atomic_dec_and_test(&kernel_threads_to_run))
+ wake_up(&kernel_threads_wq);
+}
+
+/* Random tasks test. */
+#define NR_RAND_TASKS 1000
+
+static void hpcc_print_info(struct kunit *test)
+{
+ kunit_info(test, "Running test with %d CPUs\n", num_online_cpus());
+}
+
+static void add_to_counter(int counter_index, unsigned int nr_inc, long increment)
+{
+ unsigned int i;
+
+ for (i = 0; i < nr_inc; i++) {
+ percpu_counter_tree_add(&counter[counter_index], increment);
+ atomic_long_add(increment, &global_counter[counter_index]);
+ }
+}
+
+/* Check the residuals and the carries of a settled counter. */
+static void check_residuals(struct kunit *test, struct percpu_counter_tree *c)
+{
+#ifdef CONFIG_SMP
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_check_residuals(c));
+#endif
+}
+
+static void check_counters(struct kunit *test)
+{
+ int counter_index;
+
+ /* Compare each counter with its global counter. */
+ for (counter_index = 0; counter_index < NR_COUNTERS; counter_index++) {
+ struct percpu_counter_tree *c = &counter[counter_index];
+ long v = atomic_long_read(&global_counter[counter_index]);
+ long approx_sum = percpu_counter_tree_approximate_sum(c);
+ unsigned long accuracy;
+ long precise_min, precise_max;
+
+ /* Precise comparison. */
+ KUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(c), v);
+ check_residuals(test, c);
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_precise_compare_value(c, v));
+
+ /* Approximate comparison. */
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare_value(c, v));
+
+ /* Accuracy limits checks. */
+ accuracy = percpu_counter_tree_approximate_accuracy(c);
+
+ /*
+ * The accuracy is an unsigned long, so the inner expression
+ * promotes to unsigned. Compute the difference in unsigned and
+ * cast to signed for the comparison against 0.
+ */
+ KUNIT_EXPECT_GE(test, (long)(approx_sum - (v - accuracy)), 0);
+ KUNIT_EXPECT_LE(test, (long)(approx_sum - (v + accuracy)), 0);
+
+ /* Precise min/max range check. */
+ percpu_counter_tree_approximate_min_max_range(approx_sum, accuracy,
+ &precise_min, &precise_max);
+
+ KUNIT_EXPECT_GE(test, v - precise_min, 0);
+ KUNIT_EXPECT_LE(test, v - precise_max, 0);
+ }
+ /* Compare each counter with the second counter. */
+ KUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(&counter[0]),
+ percpu_counter_tree_precise_sum(&counter[1]));
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_precise_compare(&counter[0], &counter[1]));
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare(&counter[0], &counter[1]));
+}
+
+/*
+ * Workqueue running the updates of the multi-threaded tests. Its workers
+ * are reused across rounds, which keeps the tests fast on machines where
+ * creating threads on remote CPUs is slow. Work items are CPU intensive,
+ * so several of them queued on one CPU run concurrently in separate
+ * workers, and can preempt each other in the middle of an update.
+ */
+static struct workqueue_struct *hpcc_test_wq;
+
+static void multi_thread_work_fn(struct work_struct *work)
+{
+ struct multi_thread_test_data *td =
+ container_of(work, struct multi_thread_test_data, work);
+
+ add_to_counter(td->counter_index, td->nr_inc, td->increment);
+ kfree(td);
+ complete_work();
+}
+
+/* Queue an update of a counter on @target_cpu. Call with the CPU hotplug lock held. */
+static int test_queue_on_cpu(int target_cpu, int counter_index, unsigned int nr_inc,
+ long increment)
+{
+ struct multi_thread_test_data *td = kzalloc(sizeof(*td), GFP_KERNEL);
+
+ if (!td)
+ return -ENOMEM;
+ INIT_WORK(&td->work, multi_thread_work_fn);
+ td->increment = increment;
+ td->nr_inc = nr_inc;
+ td->counter_index = counter_index;
+ atomic_inc(&kernel_threads_to_run);
+ queue_work_on(target_cpu, hpcc_test_wq, &td->work);
+ return 0;
+}
+
+static void init_kthreads(void)
+{
+ atomic_set(&kernel_threads_to_run, 1);
+}
+
+/*
+ * The caller needs to wait for work completion _before_ stopping kthreads.
+ */
+static void fini_kthreads(void)
+{
+ /* Release our own reference. */
+ complete_work();
+ /* Wait for all others threads to run. */
+ wait_event(kernel_threads_wq, (atomic_read(&kernel_threads_to_run) == 0));
+}
+
+static int init_counters(unsigned long batch_size)
+{
+ int i, ret;
+
+ ret = percpu_counter_tree_init_many(counter, NR_COUNTERS, batch_size, GFP_KERNEL);
+ if (ret)
+ return ret;
+ for (i = 0; i < NR_COUNTERS; i++)
+ atomic_long_set(&global_counter[i], 0);
+ return 0;
+}
+
+static void fini_counters(void)
+{
+ percpu_counter_tree_destroy_many(counter, NR_COUNTERS);
+}
+
+enum up_test_inc_type {
+ INC_ONE,
+ INC_MINUS_ONE,
+ INC_RANDOM,
+};
+
+/*
+ * Single-threaded tests. Those run the updates on various CPUs, one at a
+ * time: work_on_cpu() waits for each update to complete before the next,
+ * effectively making sure there are no concurrent updates. Each test
+ * stops early after SINGLE_THREAD_MAX_SECONDS, because cross-CPU wakeups
+ * can be slow on large virtual machines.
+ */
+#define SINGLE_THREAD_MAX_SECONDS 10
+
+struct single_thread_update {
+ long increment;
+ int counter_index;
+};
+
+static long single_thread_update_fn(void *arg)
+{
+ struct single_thread_update *u = arg;
+
+ add_to_counter(u->counter_index, 1, u->increment);
+ return 0;
+}
+
+static void do_hpcc_test_single_thread(struct kunit *test, int _cpu0, int _cpu1,
+ enum up_test_inc_type type)
+{
+ unsigned long batch_size_order = 5;
+ int cpu0 = _cpu0;
+ int cpu1 = _cpu1;
+ unsigned long deadline = jiffies + SINGLE_THREAD_MAX_SECONDS * HZ;
+ int ret, i;
+
+ ret = init_counters(1UL << batch_size_order);
+ KUNIT_EXPECT_EQ(test, ret, 0);
+ if (ret)
+ return;
+ cpus_read_lock();
+ for (i = 0; i < 10000 && time_before(jiffies, deadline); i++) {
+ struct single_thread_update u0 = { .counter_index = 0 };
+ struct single_thread_update u1 = { .counter_index = 1 };
+ long increment;
+
+ switch (type) {
+ case INC_ONE:
+ increment = 1;
+ break;
+ case INC_MINUS_ONE:
+ increment = -1;
+ break;
+ case INC_RANDOM:
+ increment = (long)get_random_long() % 50000;
+ break;
+ default:
+ WARN_ON_ONCE(1);
+ increment = 0;
+ }
+ if (_cpu0 < 0)
+ cpu0 = cpumask_any_distribute(cpu_online_mask);
+ if (_cpu1 < 0)
+ cpu1 = cpumask_any_distribute(cpu_online_mask);
+ u0.increment = increment;
+ u1.increment = increment;
+ work_on_cpu(cpu0, single_thread_update_fn, &u0);
+ work_on_cpu(cpu1, single_thread_update_fn, &u1);
+ check_counters(test);
+ }
+ cpus_read_unlock();
+ fini_counters();
+}
+
+static void hpcc_test_single_thread_first(struct kunit *test)
+{
+ int cpu = cpumask_first(cpu_online_mask);
+
+ do_hpcc_test_single_thread(test, cpu, cpu, INC_ONE);
+ do_hpcc_test_single_thread(test, cpu, cpu, INC_MINUS_ONE);
+ do_hpcc_test_single_thread(test, cpu, cpu, INC_RANDOM);
+}
+
+static void hpcc_test_single_thread_first_random(struct kunit *test)
+{
+ int cpu = cpumask_first(cpu_online_mask);
+
+ do_hpcc_test_single_thread(test, cpu, -1, INC_ONE);
+ do_hpcc_test_single_thread(test, cpu, -1, INC_MINUS_ONE);
+ do_hpcc_test_single_thread(test, cpu, -1, INC_RANDOM);
+}
+
+static void hpcc_test_single_thread_random(struct kunit *test)
+{
+ do_hpcc_test_single_thread(test, -1, -1, INC_ONE);
+ do_hpcc_test_single_thread(test, -1, -1, INC_MINUS_ONE);
+ do_hpcc_test_single_thread(test, -1, -1, INC_RANDOM);
+}
+
+/* Multi-threaded SMP tests. */
+
+/*
+ * Bound the number of CPUs running workers in the multi-threaded tests,
+ * so their run time does not grow with the size of the machine. Beyond
+ * this bound, the workers run on a subset spread evenly across the
+ * online CPUs, so they still reach every branch of the tree.
+ */
+#define MAX_WORKER_CPUS 64
+
+/*
+ * Each multi-threaded test stops early after MULTI_THREAD_MAX_SECONDS,
+ * because cross-CPU wakeups can be slow on large virtual machines.
+ */
+#define MULTI_THREAD_MAX_SECONDS 60
+
+static struct cpumask worker_cpus;
+
+/* Select the CPUs running workers. Call with the CPU hotplug lock held. */
+static const struct cpumask *select_worker_cpus(void)
+{
+ unsigned int nr_online = num_online_cpus(), i = 0;
+ int cpu;
+
+ cpumask_clear(&worker_cpus);
+ for_each_online_cpu(cpu) {
+ /* Pick MAX_WORKER_CPUS of the nr_online CPUs, evenly spaced. */
+ if ((i + 1) * MAX_WORKER_CPUS / nr_online !=
+ i * MAX_WORKER_CPUS / nr_online)
+ cpumask_set_cpu(cpu, &worker_cpus);
+ i++;
+ }
+ return &worker_cpus;
+}
+
+/*
+ * CPU selection for the second counter's thread, relative to the CPU
+ * running the first counter's thread.
+ */
+enum mt_cpu_select {
+ MT_SAME_CPU, /* Same CPU as the first counter. */
+ MT_EVEN_CPU, /* Funnel CPU pairs (0, 1), (2, 3), ... to 0, 2, ... */
+ MT_FIRST_CPU, /* First online CPU. */
+ MT_RANDOM_CPU, /* Any online CPU. */
+};
+
+/* Return the CPU for the second counter's thread, or -1 to skip @cpu. */
+static int mt_select_cpu(enum mt_cpu_select select, int cpu)
+{
+ int even_cpu;
+
+ switch (select) {
+ case MT_SAME_CPU:
+ return cpu;
+ case MT_EVEN_CPU:
+ even_cpu = cpu & ~1;
+ return cpumask_test_cpu(even_cpu, cpu_online_mask) ? even_cpu : -1;
+ case MT_FIRST_CPU:
+ return cpumask_first(cpu_online_mask);
+ case MT_RANDOM_CPU:
+ return cpumask_any_distribute(cpu_online_mask);
+ }
+ WARN_ON_ONCE(1);
+ return -1;
+}
+
+static void do_hpcc_multi_thread_increment(struct kunit *test, enum mt_cpu_select select,
+ unsigned long batch_size, unsigned int nr_inc,
+ long increment)
+{
+ const struct cpumask *cpus;
+ int ret, cpu;
+
+ ret = init_counters(batch_size);
+ KUNIT_EXPECT_EQ(test, ret, 0);
+ if (ret)
+ return;
+ init_kthreads();
+ cpus_read_lock();
+ cpus = select_worker_cpus();
+ for_each_cpu(cpu, cpus) {
+ int cpu1 = mt_select_cpu(select, cpu);
+
+ if (cpu1 < 0)
+ continue;
+ ret = test_queue_on_cpu(cpu, 0, nr_inc, increment);
+ KUNIT_EXPECT_EQ(test, ret, 0);
+ if (ret)
+ break;
+ ret = test_queue_on_cpu(cpu1, 1, nr_inc, increment);
+ KUNIT_EXPECT_EQ(test, ret, 0);
+ if (ret)
+ break;
+ }
+ fini_kthreads();
+ cpus_read_unlock();
+ check_counters(test);
+ fini_counters();
+}
+
+static void hpcc_test_multi_thread_batch_increment(struct kunit *test)
+{
+ unsigned long deadline = jiffies + MULTI_THREAD_MAX_SECONDS * HZ;
+ unsigned long batch_size_order;
+
+ for (batch_size_order = 2; batch_size_order <= 7; batch_size_order++) {
+ unsigned long batch_size = 1UL << batch_size_order;
+ unsigned int nr_inc;
+
+ for (nr_inc = 1; nr_inc < 1024; nr_inc *= 2) {
+ long increment;
+
+ for (increment = 1; increment < 100000; increment *= 10) {
+ enum mt_cpu_select select;
+
+ if (time_after(jiffies, deadline))
+ return;
+ for (select = MT_SAME_CPU; select <= MT_RANDOM_CPU; select++)
+ do_hpcc_multi_thread_increment(test, select, batch_size,
+ nr_inc, increment);
+ }
+ }
+ }
+}
+
+static void hpcc_test_multi_thread_random_walk(struct kunit *test)
+{
+ unsigned long deadline = jiffies + MULTI_THREAD_MAX_SECONDS * HZ;
+ unsigned long batch_size_order = 5;
+ int loop;
+
+ for (loop = 0; loop < 100 && time_before(jiffies, deadline); loop++) {
+ int ret, i;
+
+ ret = init_counters(1UL << batch_size_order);
+ KUNIT_EXPECT_EQ(test, ret, 0);
+ if (ret)
+ return;
+ init_kthreads();
+ cpus_read_lock();
+ for (i = 0; i < NR_RAND_TASKS; i++) {
+ long increment = (long)get_random_long() % 512;
+ unsigned int nr_inc = ((unsigned long)get_random_long()) % 1024;
+
+ ret = test_queue_on_cpu(cpumask_any_distribute(cpu_online_mask),
+ 0, nr_inc, increment);
+ KUNIT_EXPECT_EQ(test, ret, 0);
+ if (ret)
+ break;
+ ret = test_queue_on_cpu(cpumask_any_distribute(cpu_online_mask),
+ 1, nr_inc, increment);
+ KUNIT_EXPECT_EQ(test, ret, 0);
+ if (ret)
+ break;
+ }
+ fini_kthreads();
+ cpus_read_unlock();
+ check_counters(test);
+ fini_counters();
+ }
+}
+
+#define CONTENTION_NR_ROUNDS 20000
+#define CONTENTION_NR_INC 8
+
+struct contention_shared {
+ unsigned long batch_size;
+ atomic_t round; /* Round started by the coordinator. */
+ atomic_t nr_done; /* Rounds completed by the other workers. */
+ unsigned int nr_workers;
+ unsigned int nr_errors; /* Rounds ending with invalid residuals. */
+ bool started;
+};
+
+struct contention_data {
+ struct contention_shared *sh;
+ long sum;
+ u32 seed;
+ bool coordinator;
+} ____cacheline_aligned_in_smp;
+
+static DECLARE_WAIT_QUEUE_HEAD(contention_wq);
+
+/*
+ * Add random multiples of the batch size, of either sign, so each
+ * addition carries into the counters above level 0. Increments of
+ * opposite signs and of more than one batch size exercise concurrent
+ * carry extractions retrying after a concurrent update of the residual.
+ */
+static void contention_add(struct contention_data *cd)
+{
+ unsigned int i;
+
+ for (i = 0; i < CONTENTION_NR_INC; i++) {
+ u32 x = cd->seed;
+ long inc;
+
+ /* xorshift32: get_random_u32() would dominate the loop. */
+ x ^= x << 13;
+ x ^= x >> 17;
+ x ^= x << 5;
+ cd->seed = x;
+ inc = (long)((x & 15) + 1) * cd->sh->batch_size;
+ if (x & 16)
+ inc = -inc;
+ percpu_counter_tree_add(&counter[0], inc);
+ cd->sum += inc;
+ }
+}
+
+/*
+ * The workers run rounds of a few additions each, all starting together.
+ * Between rounds, the coordinator checks the residuals of the settled
+ * counter: an invalid residual left by the last updaters of a round would
+ * otherwise be repaired by the next additions.
+ */
+static int contention_worker_fn(void *data)
+{
+ struct contention_data *cd = data;
+ struct contention_shared *sh = cd->sh;
+ int round;
+
+ wait_event(contention_wq, READ_ONCE(sh->started));
+ for (round = 1; round <= CONTENTION_NR_ROUNDS; round++) {
+ if (cd->coordinator) {
+ atomic_set_release(&sh->round, round);
+ contention_add(cd);
+ while (atomic_read_acquire(&sh->nr_done) < round * (sh->nr_workers - 1))
+ cond_resched();
+#ifdef CONFIG_SMP
+ if (percpu_counter_tree_check_residuals(&counter[0]))
+ sh->nr_errors++;
+#endif
+ } else {
+ while (atomic_read_acquire(&sh->round) < round)
+ cond_resched();
+ contention_add(cd);
+ /* Order the additions before the coordinator check. */
+ atomic_inc_return_release(&sh->nr_done);
+ }
+ }
+ complete_work();
+ return 0;
+}
+
+static void do_hpcc_test_contended_carries(struct kunit *test, unsigned long batch_size)
+{
+ struct percpu_counter_tree *c = &counter[0];
+ struct contention_shared sh = { .batch_size = batch_size };
+ struct contention_data *cd;
+ struct task_struct **task;
+ const struct cpumask *cpus;
+ unsigned long accuracy;
+ long expected = 0, approx;
+ bool first = true;
+ int ret, cpu;
+
+ cd = kvcalloc(nr_cpu_ids, sizeof(*cd), GFP_KERNEL);
+ task = kvcalloc(nr_cpu_ids, sizeof(*task), GFP_KERNEL);
+ KUNIT_EXPECT_NOT_NULL(test, cd);
+ KUNIT_EXPECT_NOT_NULL(test, task);
+ if (!cd || !task)
+ goto free;
+ ret = init_counters(batch_size);
+ KUNIT_EXPECT_EQ(test, ret, 0);
+ if (ret)
+ goto free;
+ init_kthreads();
+ cpus_read_lock();
+ cpus = select_worker_cpus();
+ for_each_cpu(cpu, cpus) {
+ cd[cpu].sh = &sh;
+ cd[cpu].coordinator = first;
+ cd[cpu].seed = get_random_u32() | 1;
+ task[cpu] = kthread_create_on_cpu(contention_worker_fn, &cd[cpu], cpu,
+ "kunit_hpcc_contention/%u");
+ KUNIT_EXPECT_FALSE(test, IS_ERR(task[cpu]));
+ if (IS_ERR(task[cpu]))
+ break;
+ atomic_inc(&kernel_threads_to_run);
+ get_task_struct(task[cpu]);
+ wake_up_process(task[cpu]);
+ sh.nr_workers++;
+ first = false;
+ }
+ /* Start the workers once their number is known. */
+ WRITE_ONCE(sh.started, true);
+ wake_up_all(&contention_wq);
+ fini_kthreads();
+ for_each_cpu(cpu, cpus) {
+ if (IS_ERR_OR_NULL(task[cpu]))
+ continue;
+ kthread_stop_put(task[cpu]);
+ expected += cd[cpu].sum;
+ }
+ cpus_read_unlock();
+
+ kunit_info(test, "batch_size=%lu workers=%u rounds with invalid residuals=%u\n",
+ batch_size, sh.nr_workers, sh.nr_errors);
+ KUNIT_EXPECT_EQ(test, sh.nr_errors, 0);
+ KUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(c), expected);
+ check_residuals(test, c);
+ approx = percpu_counter_tree_approximate_sum(c);
+ accuracy = percpu_counter_tree_approximate_accuracy(c);
+ KUNIT_EXPECT_GE(test, approx, expected - (long)accuracy);
+ KUNIT_EXPECT_LE(test, approx, expected + (long)accuracy);
+ fini_counters();
+free:
+ kvfree(task);
+ kvfree(cd);
+}
+
+/*
+ * Concurrent carries from the worker CPUs into the counters above level 0
+ * which they share, with a check that the residuals are back within range
+ * and that no carry was lost or duplicated after each round.
+ */
+static void hpcc_test_contended_carries(struct kunit *test)
+{
+ unsigned long batch_size_order;
+
+ if (num_online_cpus() < 2)
+ kunit_skip(test, "needs at least 2 online CPUs");
+ for (batch_size_order = 1; batch_size_order <= 5; batch_size_order += 2)
+ do_hpcc_test_contended_carries(test, 1UL << batch_size_order);
+}
+
+static void hpcc_test_init_one(struct kunit *test)
+{
+ struct percpu_counter_tree pct;
+ int ret;
+
+ ret = percpu_counter_tree_init(&pct, 32, GFP_KERNEL);
+ KUNIT_EXPECT_EQ(test, ret, 0);
+ if (ret)
+ return;
+
+ percpu_counter_tree_destroy(&pct);
+}
+
+static void hpcc_test_set(struct kunit *test)
+{
+ static long values[] = {
+ 5, 100, 127, 128, 255, 256, 4095, 4096, 500000, 0,
+ -5, -100, -127, -128, -255, -256, -4095, -4096, -500000,
+ };
+ struct percpu_counter_tree pct;
+ int i, ret;
+
+ ret = percpu_counter_tree_init(&pct, 32, GFP_KERNEL);
+ KUNIT_EXPECT_EQ(test, ret, 0);
+ if (ret)
+ return;
+
+ for (i = 0; i < ARRAY_SIZE(values); i++) {
+ long v = values[i];
+
+ percpu_counter_tree_set(&pct, v);
+ KUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(&pct), v);
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare_value(&pct, v));
+
+ percpu_counter_tree_add(&pct, v);
+ KUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(&pct), 2 * v);
+ KUNIT_EXPECT_EQ(test, 0,
+ percpu_counter_tree_approximate_compare_value(&pct, 2 * v));
+
+ percpu_counter_tree_add(&pct, -2 * v);
+ KUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(&pct), 0);
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare_value(&pct, 0));
+ }
+
+ percpu_counter_tree_destroy(&pct);
+}
+
+struct saturate_data {
+ struct percpu_counter_tree *c, *probe;
+ long inc;
+ unsigned long max_steps;
+ bool overrun;
+};
+
+/* Copy the state of counter @src into counter @dst, of the same topology. */
+static void copy_counter_state(struct percpu_counter_tree *dst,
+ struct percpu_counter_tree *src)
+{
+#ifdef CONFIG_SMP
+ percpu_counter_tree_copy_state(dst, src);
+#else
+ atomic_long_set(&dst->count, atomic_long_read(&src->count));
+#endif
+}
+
+/*
+ * Add inc (1 or -1) until the next addition would propagate a carry to the
+ * approximate sum. This leaves every tree item on the path from this CPU to
+ * the root holding its largest residual in the direction of inc which has
+ * not been propagated yet, without knowledge of the tree topology. Each
+ * addition is first tried on the probe counter, which holds the same state:
+ * reverting an addition which reached the approximate sum would not restore
+ * the residuals subtracted on the way. The path from a CPU to the root
+ * holds less than the accuracy bound: more additions mean that a carry was
+ * lost.
+ */
+static long saturate_worker_fn(void *data)
+{
+ struct saturate_data *sd = data;
+ unsigned long steps;
+
+ for (steps = 0; steps < sd->max_steps; steps++) {
+ long approx = percpu_counter_tree_approximate_sum(sd->probe);
+
+ percpu_counter_tree_add(sd->probe, sd->inc);
+ if (percpu_counter_tree_approximate_sum(sd->probe) != approx) {
+ copy_counter_state(sd->probe, sd->c);
+ break;
+ }
+ percpu_counter_tree_add(sd->c, sd->inc);
+ }
+ if (steps == sd->max_steps)
+ sd->overrun = true;
+ return 0;
+}
+
+static void do_hpcc_test_saturated_compare(struct kunit *test, unsigned long batch_size,
+ long inc)
+{
+ struct percpu_counter_tree *c = &counter[0], *exact = &counter[1], probe;
+ struct saturate_data sd = { .c = c, .probe = &probe, .inc = inc };
+ unsigned long accuracy;
+ long approx, precise, precise_min, precise_max;
+ int ret, cpu;
+
+ ret = percpu_counter_tree_init(&probe, batch_size, GFP_KERNEL);
+ KUNIT_EXPECT_EQ(test, ret, 0);
+ if (ret)
+ return;
+ ret = init_counters(batch_size);
+ KUNIT_EXPECT_EQ(test, ret, 0);
+ if (ret)
+ goto destroy_probe;
+ accuracy = percpu_counter_tree_approximate_accuracy(c);
+ sd.max_steps = batch_size + accuracy;
+ cpus_read_lock();
+ for_each_online_cpu(cpu) {
+ work_on_cpu(cpu, saturate_worker_fn, &sd);
+ KUNIT_EXPECT_FALSE(test, sd.overrun);
+ if (sd.overrun)
+ break;
+ }
+ cpus_read_unlock();
+
+ precise = percpu_counter_tree_precise_sum(c);
+ approx = percpu_counter_tree_approximate_sum(c);
+ kunit_info(test, "batch_size=%lu inc=%ld precise=%ld approx=%ld accuracy=%lu\n",
+ batch_size, inc, precise, approx, accuracy);
+
+ check_residuals(test, c);
+ check_residuals(test, &probe);
+
+ /* Accuracy contract. */
+ KUNIT_EXPECT_GE(test, approx, precise - (long)accuracy);
+ KUNIT_EXPECT_LE(test, approx, precise + (long)accuracy);
+ /* Saturating every possible CPU reaches the accuracy exactly. */
+ if (num_online_cpus() == nr_cpu_ids)
+ KUNIT_EXPECT_EQ(test, (precise - approx) * inc, (long)accuracy);
+ percpu_counter_tree_approximate_min_max_range(approx, accuracy,
+ &precise_min, &precise_max);
+ KUNIT_EXPECT_GE(test, precise, precise_min);
+ KUNIT_EXPECT_LE(test, precise, precise_max);
+
+ /* Comparisons against a value. */
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare_value(c, precise));
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_precise_compare_value(c, precise));
+ KUNIT_EXPECT_EQ(test, -1, percpu_counter_tree_precise_compare_value(c, precise - 1));
+ KUNIT_EXPECT_EQ(test, 1, percpu_counter_tree_precise_compare_value(c, precise + 1));
+
+ /* Comparisons against a counter with the same precise sum, exactly approximated. */
+ percpu_counter_tree_set(exact, precise);
+ KUNIT_EXPECT_EQ(test, percpu_counter_tree_approximate_sum(exact), precise);
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare(c, exact));
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare(exact, c));
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_precise_compare(c, exact));
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_precise_compare(exact, c));
+
+ fini_counters();
+destroy_probe:
+ percpu_counter_tree_destroy(&probe);
+}
+
+/*
+ * Saturate the unpropagated residuals of every tree item reachable from
+ * online CPUs, upward and downward, which yields the largest distance
+ * between the precise and approximate sums of a settled counter in each
+ * direction, and check that the comparison
+ * functions do not return a wrong definitive answer. Small batch sizes
+ * are the most likely to expose margin errors, because the fan-out of
+ * the tree levels can be larger than the batch size.
+ */
+static void hpcc_test_saturated_compare(struct kunit *test)
+{
+ unsigned long batch_size_order;
+
+ for (batch_size_order = 1; batch_size_order <= 5; batch_size_order++) {
+ do_hpcc_test_saturated_compare(test, 1UL << batch_size_order, 1);
+ do_hpcc_test_saturated_compare(test, 1UL << batch_size_order, -1);
+ }
+}
+
+/* Check a counter against its expected sum: exact before activation, within range after. */
+static void check_lazy_counter(struct kunit *test, struct percpu_counter_tree *c, long v)
+{
+ long approx_sum = percpu_counter_tree_approximate_sum(c);
+ unsigned long accuracy;
+
+ KUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(c), v);
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_precise_compare_value(c, v));
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare_value(c, v));
+ check_residuals(test, c);
+#ifdef CONFIG_SMP
+ if (!READ_ONCE(c->level0)) {
+ KUNIT_EXPECT_EQ(test, approx_sum, v);
+ return;
+ }
+#endif
+ accuracy = percpu_counter_tree_approximate_accuracy(c);
+ KUNIT_EXPECT_GE(test, (long)(approx_sum - (v - accuracy)), 0);
+ KUNIT_EXPECT_LE(test, (long)(approx_sum - (v + accuracy)), 0);
+}
+
+/*
+ * Lazily initialized counters: exact sums before the tree is activated,
+ * with owner and atomic updates, set(), idempotent activation, updates
+ * after activation, and destruction of counters never activated.
+ */
+static void hpcc_test_lazy(struct kunit *test)
+{
+ struct percpu_counter_tree pct[2];
+ long expect[2] = { 0, 0 };
+ int i, j;
+
+ KUNIT_ASSERT_EQ(test, 0, percpu_counter_tree_init_many_lazy(pct, 2, 32));
+#ifdef CONFIG_SMP
+ KUNIT_EXPECT_FALSE(test, percpu_counter_tree_active(&pct[0]));
+ KUNIT_EXPECT_FALSE(test, percpu_counter_tree_active(&pct[1]));
+ /* The level 0 counters are __percpu pointers: compare them as booleans. */
+ KUNIT_EXPECT_TRUE(test, !pct[0].level0);
+ KUNIT_EXPECT_TRUE(test, !pct[1].level0);
+ KUNIT_EXPECT_NULL(test, pct[0].approx_sum);
+ KUNIT_EXPECT_NULL(test, pct[1].approx_sum);
+#endif
+ for (j = 0; j < 2; j++) {
+ /* j == 0: before activation; j == 1: after activation. */
+ for (i = 0; i < 10000; i++) {
+ long inc = (long)(get_random_u32() % 1025) - 512;
+
+ if (get_random_u32() & 1)
+ percpu_counter_tree_add_owner(&pct[0], inc);
+ else
+ percpu_counter_tree_add(&pct[0], inc);
+ expect[0] += inc;
+ percpu_counter_tree_add(&pct[1], -inc);
+ expect[1] -= inc;
+ }
+ check_lazy_counter(test, &pct[0], expect[0]);
+ check_lazy_counter(test, &pct[1], expect[1]);
+ percpu_counter_tree_set(&pct[1], 12345);
+ expect[1] = 12345;
+ check_lazy_counter(test, &pct[1], expect[1]);
+ if (j)
+ break;
+ KUNIT_ASSERT_EQ(test, 0, percpu_counter_tree_activate_many(pct, 2, GFP_KERNEL));
+ KUNIT_EXPECT_TRUE(test, percpu_counter_tree_active(&pct[0]));
+ KUNIT_EXPECT_TRUE(test, percpu_counter_tree_active(&pct[1]));
+#ifdef CONFIG_SMP
+ {
+ s8 __percpu *level0 = pct[0].level0;
+
+ KUNIT_EXPECT_TRUE(test, level0);
+ KUNIT_EXPECT_TRUE(test, pct[1].level0 == level0 + 1);
+ KUNIT_EXPECT_NOT_NULL(test, pct[0].approx_sum);
+ KUNIT_EXPECT_PTR_EQ(test, pct[1].approx_sum, pct[0].approx_sum + 1);
+ KUNIT_EXPECT_EQ(test, 0,
+ percpu_counter_tree_activate_many(pct, 2, GFP_KERNEL));
+ KUNIT_EXPECT_TRUE(test, pct[0].level0 == level0);
+ }
+#endif
+ /* The sums carry the updates made before activation. */
+ check_lazy_counter(test, &pct[0], expect[0]);
+ check_lazy_counter(test, &pct[1], expect[1]);
+ }
+ percpu_counter_tree_destroy_many(pct, 2);
+
+ /* Destroy counters which were never activated. */
+ KUNIT_ASSERT_EQ(test, 0, percpu_counter_tree_init_many_lazy(pct, 2, 32));
+ percpu_counter_tree_add_owner(&pct[0], 7);
+ percpu_counter_tree_add(&pct[1], -7);
+ check_lazy_counter(test, &pct[0], 7);
+ check_lazy_counter(test, &pct[1], -7);
+ percpu_counter_tree_destroy_many(pct, 2);
+}
+
+/*
+ * Activation racing with updates: one owner and updaters on other CPUs add
+ * to a lazily initialized counter. Each worker makes a first part of its
+ * updates, reports their sum and waits. The test checks that the counter
+ * is exact, then releases the workers and activates the tree while they
+ * make the rest of their updates. The remote updaters also activate the
+ * tree when released, serialized with the test and with each other by a
+ * mutex, and check that the tree is active once their activation returns.
+ * No update may be lost, whichever side of the activation it lands on.
+ */
+#define LAZY_RACE_NR_INC 200000
+#define LAZY_RACE_NR_FIRST (LAZY_RACE_NR_INC / 8)
+#define LAZY_RACE_MAX_REMOTE 8
+
+struct lazy_race_shared {
+ struct percpu_counter_tree *c;
+ atomic_long_t first_sum; /* Sum of the updates made before the release. */
+ atomic_long_t expect; /* Sum of all updates. */
+ struct mutex activate_lock; /* Serializes the activations. */
+ atomic_t paused; /* Workers waiting for the release. */
+ atomic_t activate_errors; /* Failed activations of the workers. */
+ bool go; /* Releases the workers. */
+};
+
+struct lazy_race_work {
+ struct work_struct work;
+ struct lazy_race_shared *sh;
+ bool owner;
+};
+
+static void lazy_race_fn(struct work_struct *work)
+{
+ struct lazy_race_work *w = container_of(work, struct lazy_race_work, work);
+ struct lazy_race_shared *sh = w->sh;
+ long sum = 0;
+ int i;
+
+ for (i = 0; i < LAZY_RACE_NR_INC; i++) {
+ long inc = (long)(get_random_u32() % 65) - 32;
+
+ if (i == LAZY_RACE_NR_FIRST) {
+ /* Fully ordered: the first updates are visible before the pause. */
+ atomic_long_add_return(sum, &sh->first_sum);
+ atomic_inc_return(&sh->paused);
+ while (!READ_ONCE(sh->go))
+ cond_resched();
+ if (!w->owner) {
+ mutex_lock(&sh->activate_lock);
+ if (percpu_counter_tree_activate_many(sh->c, 1, GFP_KERNEL) ||
+ !percpu_counter_tree_active(sh->c))
+ atomic_inc(&sh->activate_errors);
+ mutex_unlock(&sh->activate_lock);
+ }
+ }
+ if (w->owner)
+ percpu_counter_tree_add_owner(sh->c, inc);
+ else
+ percpu_counter_tree_add(sh->c, inc);
+ sum += inc;
+ if (!(i & 1023))
+ cond_resched();
+ }
+ atomic_long_add(sum, &sh->expect);
+ kfree(w);
+ complete_work();
+}
+
+static void hpcc_test_lazy_activation_race(struct kunit *test)
+{
+ struct percpu_counter_tree pct;
+ int round;
+
+ if (num_online_cpus() < 2)
+ kunit_skip(test, "needs at least 2 online CPUs");
+ for (round = 0; round < 4; round++) {
+ struct lazy_race_shared sh = { .c = &pct };
+ unsigned long timeout = jiffies + 10 * HZ;
+ int cpu, nr_workers = 0;
+
+ KUNIT_ASSERT_EQ(test, 0, percpu_counter_tree_init_many_lazy(&pct, 1, 32));
+ mutex_init(&sh.activate_lock);
+ init_kthreads();
+ cpus_read_lock();
+ for_each_online_cpu(cpu) {
+ struct lazy_race_work *w;
+
+ if (nr_workers > LAZY_RACE_MAX_REMOTE)
+ break;
+ w = kzalloc_obj(*w);
+ KUNIT_EXPECT_NOT_NULL(test, w);
+ if (!w)
+ break;
+ INIT_WORK(&w->work, lazy_race_fn);
+ w->sh = &sh;
+ /* The first worker is the owner, the others update remotely. */
+ w->owner = !nr_workers;
+ atomic_inc(&kernel_threads_to_run);
+ queue_work_on(cpu, hpcc_test_wq, &w->work);
+ nr_workers++;
+ }
+ cpus_read_unlock();
+ while (atomic_read(&sh.paused) < nr_workers && time_before(jiffies, timeout))
+ usleep_range(100, 200);
+ smp_mb(); /* Read the counter after the first updates of the workers. */
+ /* Every worker, the owner included, has updates left for after the activation. */
+ KUNIT_EXPECT_EQ(test, atomic_read(&sh.paused), nr_workers);
+ /* Before activation, the owner and atomic updates are exact. */
+ check_lazy_counter(test, &pct, atomic_long_read(&sh.first_sum));
+ WRITE_ONCE(sh.go, true);
+ mutex_lock(&sh.activate_lock);
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_activate_many(&pct, 1, GFP_KERNEL));
+ KUNIT_EXPECT_TRUE(test, percpu_counter_tree_active(&pct));
+ mutex_unlock(&sh.activate_lock);
+ fini_kthreads();
+ mutex_destroy(&sh.activate_lock);
+ KUNIT_EXPECT_EQ(test, 0, atomic_read(&sh.activate_errors));
+ check_lazy_counter(test, &pct, atomic_long_read(&sh.expect));
+ percpu_counter_tree_destroy_many(&pct, 1);
+ }
+}
+
+static void hpcc_test_check_config(struct kunit *test)
+{
+#ifdef CONFIG_SMP
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_check_config());
+#else
+ kunit_skip(test, "The configuration table requires CONFIG_SMP");
+#endif
+}
+
+static struct kunit_case hpcc_test_cases[] = {
+ KUNIT_CASE(hpcc_test_check_config),
+ KUNIT_CASE(hpcc_print_info),
+ KUNIT_CASE(hpcc_test_single_thread_first),
+ KUNIT_CASE(hpcc_test_single_thread_first_random),
+ KUNIT_CASE_SLOW(hpcc_test_single_thread_random),
+ KUNIT_CASE(hpcc_test_multi_thread_batch_increment),
+ KUNIT_CASE_SLOW(hpcc_test_multi_thread_random_walk),
+ KUNIT_CASE(hpcc_test_init_one),
+ KUNIT_CASE(hpcc_test_set),
+ KUNIT_CASE(hpcc_test_saturated_compare),
+ KUNIT_CASE(hpcc_test_contended_carries),
+ KUNIT_CASE(hpcc_test_lazy),
+ KUNIT_CASE(hpcc_test_lazy_activation_race),
+ {}
+};
+
+static int hpcc_test_suite_init(struct kunit_suite *suite)
+{
+ hpcc_test_wq = alloc_workqueue("hpcc_test", WQ_PERCPU | WQ_CPU_INTENSIVE, 0);
+ return hpcc_test_wq ? 0 : -ENOMEM;
+}
+
+static void hpcc_test_suite_exit(struct kunit_suite *suite)
+{
+ destroy_workqueue(hpcc_test_wq);
+}
+
+static struct kunit_suite hpcc_test_suite = {
+ .name = "percpu_counter_tree",
+ .suite_init = hpcc_test_suite_init,
+ .suite_exit = hpcc_test_suite_exit,
+ .test_cases = hpcc_test_cases,
+};
+
+kunit_test_suite(hpcc_test_suite);
+
+MODULE_DESCRIPTION("Test cases for hierarchical per-CPU counters");
+MODULE_LICENSE("Dual MIT/GPL");
+MODULE_IMPORT_NS("EXPORTED_FOR_KUNIT_TESTING");
diff --git a/tools/testing/hpcc/.gitignore b/tools/testing/hpcc/.gitignore
new file mode 100644
index 00000000000000..f362a8a46868a5
--- /dev/null
+++ b/tools/testing/hpcc/.gitignore
@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: GPL-2.0+ OR MIT
+hpcc
diff --git a/tools/testing/hpcc/Makefile b/tools/testing/hpcc/Makefile
new file mode 100644
index 00000000000000..e4758643a3f0e5
--- /dev/null
+++ b/tools/testing/hpcc/Makefile
@@ -0,0 +1,18 @@
+# SPDX-License-Identifier: GPL-2.0+ OR MIT
+
+.PHONY: default clean
+
+default: hpcc
+
+CFLAGS += -O2 -g -Wall -I./include -I../../include -fno-strict-overflow -fno-strict-aliasing
+CFLAGS += -DCONFIG_SMP -DCONFIG_KUNIT=1 -D'NR_CPUS=(1U << 20)'
+
+DEPS = ../../../lib/percpu_counter_tree.c \
+ ../../../include/linux/percpu_counter_tree.h \
+ include/hpcc_shim.h
+
+hpcc: main.c $(DEPS)
+ $(CC) $(CFLAGS) -o $@ main.c $(LDLIBS)
+
+clean:
+ $(RM) hpcc
diff --git a/tools/testing/hpcc/include/hpcc_shim.h b/tools/testing/hpcc/include/hpcc_shim.h
new file mode 100644
index 00000000000000..eaac72eab2d172
--- /dev/null
+++ b/tools/testing/hpcc/include/hpcc_shim.h
@@ -0,0 +1,203 @@
+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */
+/*
+ * Minimal userspace environment to build lib/percpu_counter_tree.c.
+ *
+ * The test is single-threaded: per-CPU data is an array indexed by the
+ * simulated current CPU, and atomic operations are plain operations.
+ */
+#ifndef _HPCC_SHIM_H
+#define _HPCC_SHIM_H
+
+#include <stdbool.h>
+#include <stddef.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <errno.h>
+
+/* From tools/include. */
+#include <linux/compiler.h>
+#include <linux/gfp_types.h>
+#include <linux/init.h>
+#include <linux/kconfig.h>
+#include <linux/kernel.h>
+#include <linux/types.h>
+
+#define __ro_after_init
+#define __percpu
+#define SMP_CACHE_BYTES 64
+#define ____cacheline_aligned_in_smp __aligned(SMP_CACHE_BYTES)
+#define S8_MAX INT8_MAX
+
+#undef abs
+#define abs(x) ({ typeof(x) __x = (x); __x < 0 ? -__x : __x; })
+
+#define pr_err(...) fprintf(stderr, __VA_ARGS__)
+#define pr_warn(...) fprintf(stderr, __VA_ARGS__)
+#define pr_info(...) do { if (0) printf(__VA_ARGS__); } while (0)
+
+/* Implemented in main.c. */
+char *get_options(const char *str, int nints, int *ints);
+
+extern unsigned long hpcc_nr_warnings;
+
+#define WARN(cond, ...) ({ \
+ int __ret_warn = !!(cond); \
+ if (__ret_warn) { \
+ hpcc_nr_warnings++; \
+ fprintf(stderr, "WARNING at %s:%d: ", __FILE__, __LINE__); \
+ fprintf(stderr, __VA_ARGS__); \
+ } \
+ __ret_warn; \
+})
+#define WARN_ON(cond) WARN(cond, "%s\n", #cond)
+#define WARN_ON_ONCE(cond) WARN_ON(cond)
+
+static inline int get_count_order(unsigned int count)
+{
+ int order;
+
+ if (!count)
+ return -1;
+ order = 31 - __builtin_clz(count);
+ if (count & (count - 1))
+ order++;
+ return order;
+}
+
+#define atomic_long_read(v) READ_ONCE((v)->counter)
+#define atomic_long_set(v, i) WRITE_ONCE((v)->counter, (i))
+#define atomic_long_add(i, v) ((void)((v)->counter += (i)))
+#define atomic_long_add_return_relaxed(i, v) ((v)->counter += (i))
+#define atomic_long_add_return_release(i, v) ((v)->counter += (i))
+#define smp_rmb() barrier()
+/* The harness runs on one thread: ordering primitives are plain accesses. */
+#define smp_load_acquire(p) READ_ONCE(*(p))
+#define smp_store_release(p, v) WRITE_ONCE(*(p), (v))
+
+static inline bool atomic_long_try_cmpxchg_relaxed(atomic_long_t *v, long *old, long new)
+{
+ if (v->counter == *old) {
+ v->counter = new;
+ return true;
+ }
+ *old = v->counter;
+ return false;
+}
+
+/* Simulated topology, and CPU on which the current code runs. */
+extern unsigned int nr_cpu_ids;
+extern unsigned int hpcc_current_cpu;
+
+#define raw_smp_processor_id() hpcc_current_cpu
+#define for_each_possible_cpu(cpu) for ((cpu) = 0; (cpu) < (int)nr_cpu_ids; (cpu)++)
+
+/*
+ * Each per-CPU allocation reserves HPCC_PCPU_UNIT bytes per CPU, so
+ * per_cpu_ptr() can be applied to a pointer within an allocation.
+ */
+#define HPCC_PCPU_UNIT 64
+
+/* Outstanding allocations, to check for leaks. */
+extern long hpcc_nr_allocs;
+
+static inline void *hpcc_alloc_account(void *p)
+{
+ if (p)
+ hpcc_nr_allocs++;
+ return p;
+}
+
+static inline void hpcc_free_account(void *p)
+{
+ if (p)
+ hpcc_nr_allocs--;
+}
+
+static inline void *__alloc_percpu_gfp(size_t size, size_t align, gfp_t gfp)
+{
+ if (size > HPCC_PCPU_UNIT || align > HPCC_PCPU_UNIT)
+ return NULL;
+ return hpcc_alloc_account(calloc(nr_cpu_ids, HPCC_PCPU_UNIT));
+}
+
+static inline void free_percpu(void *ptr)
+{
+ hpcc_free_account(ptr);
+ free(ptr);
+}
+
+#define SLAB_PANIC 0
+
+struct kmem_cache {
+ size_t size;
+ size_t align;
+};
+
+static inline struct kmem_cache *kmem_cache_create(const char *name, unsigned int size,
+ unsigned int align, unsigned int flags,
+ void (*ctor)(void *))
+{
+ struct kmem_cache *s = malloc(sizeof(*s));
+
+ if (!s)
+ abort();
+ s->size = size;
+ s->align = align;
+ return s;
+}
+
+static inline void kmem_cache_destroy(struct kmem_cache *s)
+{
+ free(s);
+}
+
+static inline void *kmem_cache_zalloc(struct kmem_cache *s, gfp_t gfp)
+{
+ /* The allocation size needs to be a multiple of the alignment. */
+ void *p = aligned_alloc(s->align, (s->size + s->align - 1) / s->align * s->align);
+
+ if (p)
+ memset(p, 0, s->size);
+ return hpcc_alloc_account(p);
+}
+
+static inline void kmem_cache_free(struct kmem_cache *s, void *p)
+{
+ hpcc_free_account(p);
+ free(p);
+}
+
+static inline void *kzalloc(size_t size, gfp_t gfp)
+{
+ return hpcc_alloc_account(calloc(1, size));
+}
+
+static inline void kfree(void *p)
+{
+ hpcc_free_account(p);
+ free(p);
+}
+
+#define per_cpu_ptr(ptr, cpu) \
+ ((typeof(ptr))((char *)(ptr) + (size_t)(cpu) * HPCC_PCPU_UNIT))
+
+/* Level 0 counters are the only per-CPU data: s8. */
+static inline bool __hpcc_this_cpu_try_cmpxchg(s8 *pcp, s8 *old, s8 new)
+{
+ s8 *p = per_cpu_ptr(pcp, hpcc_current_cpu);
+
+ if (*p != *old) {
+ *old = *p;
+ return false;
+ }
+ *p = new;
+ return true;
+}
+
+#define this_cpu_read(pcp) (*per_cpu_ptr(&(pcp), hpcc_current_cpu))
+#define this_cpu_try_cmpxchg(pcp, ovalp, nval) \
+ __hpcc_this_cpu_try_cmpxchg(&(pcp), (ovalp), (nval))
+
+#endif /* _HPCC_SHIM_H */
diff --git a/tools/testing/hpcc/include/kunit/visibility.h b/tools/testing/hpcc/include/kunit/visibility.h
new file mode 100644
index 00000000000000..dfcbfd01a4151f
--- /dev/null
+++ b/tools/testing/hpcc/include/kunit/visibility.h
@@ -0,0 +1,2 @@
+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */
+#define EXPORT_SYMBOL_IF_KUNIT(symbol)
diff --git a/tools/testing/hpcc/include/linux/args.h b/tools/testing/hpcc/include/linux/args.h
new file mode 100644
index 00000000000000..0c00a85fdd213d
--- /dev/null
+++ b/tools/testing/hpcc/include/linux/args.h
@@ -0,0 +1,4 @@
+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */
+/* Also defined by the C library <sys/cdefs.h>. */
+#undef __CONCAT
+#include "../../../../../include/linux/args.h"
diff --git a/tools/testing/hpcc/include/linux/atomic.h b/tools/testing/hpcc/include/linux/atomic.h
new file mode 100644
index 00000000000000..b2aa26415cca18
--- /dev/null
+++ b/tools/testing/hpcc/include/linux/atomic.h
@@ -0,0 +1,2 @@
+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */
+#include "../hpcc_shim.h"
diff --git a/tools/testing/hpcc/include/linux/bug.h b/tools/testing/hpcc/include/linux/bug.h
new file mode 100644
index 00000000000000..b2aa26415cca18
--- /dev/null
+++ b/tools/testing/hpcc/include/linux/bug.h
@@ -0,0 +1,2 @@
+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */
+#include "../hpcc_shim.h"
diff --git a/tools/testing/hpcc/include/linux/cache.h b/tools/testing/hpcc/include/linux/cache.h
new file mode 100644
index 00000000000000..b2aa26415cca18
--- /dev/null
+++ b/tools/testing/hpcc/include/linux/cache.h
@@ -0,0 +1,2 @@
+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */
+#include "../hpcc_shim.h"
diff --git a/tools/testing/hpcc/include/linux/cpumask.h b/tools/testing/hpcc/include/linux/cpumask.h
new file mode 100644
index 00000000000000..b2aa26415cca18
--- /dev/null
+++ b/tools/testing/hpcc/include/linux/cpumask.h
@@ -0,0 +1,2 @@
+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */
+#include "../hpcc_shim.h"
diff --git a/tools/testing/hpcc/include/linux/errno.h b/tools/testing/hpcc/include/linux/errno.h
new file mode 100644
index 00000000000000..45fdd60261e717
--- /dev/null
+++ b/tools/testing/hpcc/include/linux/errno.h
@@ -0,0 +1,3 @@
+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */
+/* Also included by the C library <errno.h>: only provide the error numbers. */
+#include <asm/errno.h>
diff --git a/tools/testing/hpcc/include/linux/limits.h b/tools/testing/hpcc/include/linux/limits.h
new file mode 100644
index 00000000000000..b2aa26415cca18
--- /dev/null
+++ b/tools/testing/hpcc/include/linux/limits.h
@@ -0,0 +1,2 @@
+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */
+#include "../hpcc_shim.h"
diff --git a/tools/testing/hpcc/include/linux/math.h b/tools/testing/hpcc/include/linux/math.h
new file mode 100644
index 00000000000000..b2aa26415cca18
--- /dev/null
+++ b/tools/testing/hpcc/include/linux/math.h
@@ -0,0 +1,2 @@
+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */
+#include "../hpcc_shim.h"
diff --git a/tools/testing/hpcc/include/linux/moduleparam.h b/tools/testing/hpcc/include/linux/moduleparam.h
new file mode 100644
index 00000000000000..7e40df4295b2f5
--- /dev/null
+++ b/tools/testing/hpcc/include/linux/moduleparam.h
@@ -0,0 +1,12 @@
+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */
+#include "../hpcc_shim.h"
+
+struct kernel_param;
+
+struct kernel_param_ops {
+ int (*set)(const char *val, const struct kernel_param *kp);
+ int (*get)(char *buffer, const struct kernel_param *kp);
+};
+
+#define module_param_cb(name, ops, arg, perm)
+#define MODULE_PARM_DESC(parm, desc)
diff --git a/tools/testing/hpcc/include/linux/percpu.h b/tools/testing/hpcc/include/linux/percpu.h
new file mode 100644
index 00000000000000..b2aa26415cca18
--- /dev/null
+++ b/tools/testing/hpcc/include/linux/percpu.h
@@ -0,0 +1,2 @@
+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */
+#include "../hpcc_shim.h"
diff --git a/tools/testing/hpcc/include/linux/percpu_counter_tree.h b/tools/testing/hpcc/include/linux/percpu_counter_tree.h
new file mode 100644
index 00000000000000..065a2aca4d5b8b
--- /dev/null
+++ b/tools/testing/hpcc/include/linux/percpu_counter_tree.h
@@ -0,0 +1,2 @@
+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */
+#include "../../../../../include/linux/percpu_counter_tree.h"
diff --git a/tools/testing/hpcc/include/linux/preempt.h b/tools/testing/hpcc/include/linux/preempt.h
new file mode 100644
index 00000000000000..b2aa26415cca18
--- /dev/null
+++ b/tools/testing/hpcc/include/linux/preempt.h
@@ -0,0 +1,2 @@
+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */
+#include "../hpcc_shim.h"
diff --git a/tools/testing/hpcc/include/linux/slab.h b/tools/testing/hpcc/include/linux/slab.h
new file mode 100644
index 00000000000000..b2aa26415cca18
--- /dev/null
+++ b/tools/testing/hpcc/include/linux/slab.h
@@ -0,0 +1,2 @@
+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */
+#include "../hpcc_shim.h"
diff --git a/tools/testing/hpcc/main.c b/tools/testing/hpcc/main.c
new file mode 100644
index 00000000000000..be17b887312631
--- /dev/null
+++ b/tools/testing/hpcc/main.c
@@ -0,0 +1,536 @@
+// SPDX-License-Identifier: GPL-2.0+ OR MIT
+/*
+ * Userspace test of the hierarchical per-CPU counters (lib/percpu_counter_tree.c).
+ *
+ * The library is built with a simulated topology, which allows testing
+ * every supported tree configuration on a single machine, whereas the
+ * KUnit test only covers the topology of the machine it runs on.
+ *
+ * For each simulated number of CPUs and batch size:
+ *
+ * - Saturate the counter upward and downward: leave every tree item with
+ * its largest positive (or negative) residual which has not been
+ * propagated yet, check that the distance between the precise and
+ * approximate sums matches the residual capacity of the tree in that
+ * direction, and check the accuracy range contract and the comparison
+ * functions in these worst-case states.
+ *
+ * - Random walk: apply random increments from random CPUs to two counters,
+ * and check the accuracy range contract and the comparison functions
+ * against shadow sums after each increment.
+ *
+ * - Lazy counter: random owner and atomic updates of a lazily initialized
+ * counter, exact before its tree is activated halfway, with the same
+ * checks after each increment.
+ *
+ * The test is single-threaded, so it only covers settled counters.
+ */
+
+#include <stdarg.h>
+#include "hpcc_shim.h"
+
+unsigned int nr_cpu_ids;
+unsigned int hpcc_current_cpu;
+unsigned long hpcc_nr_warnings;
+long hpcc_nr_allocs;
+
+#include "../../../lib/percpu_counter_tree.c"
+
+#define MAX_REPORTED_FAILURES 10
+
+static unsigned long nr_case_failures;
+
+#define CHECK(cond) do { \
+ if (!(cond)) { \
+ if (nr_case_failures++ < MAX_REPORTED_FAILURES) \
+ printf("# %s:%d: check failed: %s\n", \
+ __FILE__, __LINE__, #cond); \
+ } \
+} while (0)
+
+static uint64_t rng_state = 88172645463325252ULL;
+
+static uint64_t rnd(void)
+{
+ rng_state ^= rng_state << 13;
+ rng_state ^= rng_state >> 7;
+ rng_state ^= rng_state << 17;
+ return rng_state;
+}
+
+static int sign(long v)
+{
+ return (v > 0) - (v < 0);
+}
+
+struct test_counter {
+ struct percpu_counter_tree tree;
+ long shadow; /* Expected precise sum. */
+};
+
+int scnprintf(char *buf, size_t size, const char *fmt, ...)
+{
+ va_list args;
+ int len;
+
+ if (!size)
+ return 0;
+ va_start(args, fmt);
+ len = vsnprintf(buf, size, fmt, args);
+ va_end(args);
+ return len < (int)size ? len : (int)size - 1;
+}
+
+/* Comma-separated integers, without the ranges of lib/cmdline.c. */
+char *get_options(const char *str, int nints, int *ints)
+{
+ char *end;
+ int i = 0;
+
+ while (i < nints - 1 && *str) {
+ long v = strtol(str, &end, 0);
+
+ if (end == str)
+ break;
+ ints[++i] = v;
+ str = end;
+ if (*str != ',')
+ break;
+ str++;
+ }
+ ints[0] = i;
+ return (char *)str;
+}
+
+static void hpcc_init_topology(unsigned int nr_cpus)
+{
+ nr_cpu_ids = nr_cpus;
+ /* Reset the state accumulated by a prior subsystem initialization. */
+ kmem_cache_destroy(tree_topology.items_cachep);
+ memset(&tree_topology, 0, sizeof(tree_topology));
+ tree_topology.nr_items = UINT_MAX;
+ CHECK(!percpu_counter_tree_subsystem_init());
+}
+
+static int test_counter_init(struct test_counter *c, unsigned long batch_size)
+{
+ c->shadow = 0;
+ return percpu_counter_tree_init(&c->tree, batch_size, GFP_KERNEL);
+}
+
+static void test_counter_destroy(struct test_counter *c)
+{
+ percpu_counter_tree_destroy(&c->tree);
+}
+
+static void test_counter_add(struct test_counter *c, unsigned int cpu, long inc)
+{
+ hpcc_current_cpu = cpu;
+ percpu_counter_tree_add(&c->tree, inc);
+ c->shadow += inc;
+}
+
+static void test_counter_set(struct test_counter *c, long v)
+{
+ percpu_counter_tree_set(&c->tree, v);
+ c->shadow = v;
+}
+
+/*
+ * Residual capacity of the tree items above level 0, in either direction:
+ * sum of the largest residual which each item can hold without propagating
+ * a carry. A level L item only receives multiples of the level L - 1 carry,
+ * and propagates once its residual reaches the level L carry, so it holds
+ * at most bit(L) - bit(L - 1) in either direction.
+ */
+static unsigned long tree_item_capacity(unsigned long batch_size)
+{
+ unsigned long capacity = 0, lo = batch_size, hi;
+ unsigned int level, level_items;
+
+ level_items = 1U << tree_topology.nr_cpus_order;
+ for (level = 1; level < tree_topology.config->nr_levels; level++) {
+ unsigned int n_arity_order = tree_topology.config->n_arity_order[level - 1];
+
+ level_items >>= n_arity_order;
+ hi = lo << n_arity_order;
+ capacity += level_items * (hi - lo);
+ lo = hi;
+ }
+ return capacity;
+}
+
+/*
+ * Residual capacity of the tree, in either direction: a level 0 counter
+ * holds at most batch_size - 1 on top of the item capacity.
+ */
+static unsigned long tree_residual_capacity(unsigned long batch_size)
+{
+ return nr_cpu_ids * (batch_size - 1) + tree_item_capacity(batch_size);
+}
+
+/* Copy the state of counter @src into counter @dst, of the same topology. */
+static void test_counter_copy(struct test_counter *dst, struct test_counter *src)
+{
+ percpu_counter_tree_copy_state(&dst->tree, &src->tree);
+ dst->shadow = src->shadow;
+}
+
+/*
+ * Check the accuracy range contract, and the comparison of the counter
+ * against values around its precise sum and its possible precise range.
+ */
+static void check_counter(struct test_counter *c)
+{
+ struct percpu_counter_tree *t = &c->tree;
+ long approx = percpu_counter_tree_approximate_sum(t);
+ long precise = c->shadow, precise_min, precise_max;
+ unsigned long accuracy;
+ long values[7];
+ unsigned int i;
+
+ CHECK(percpu_counter_tree_precise_sum(t) == precise);
+ CHECK(!percpu_counter_tree_check_residuals(t));
+
+ /* precise - accuracy <= approx <= precise + accuracy */
+ accuracy = percpu_counter_tree_approximate_accuracy(t);
+ CHECK(approx >= precise - (long)accuracy);
+ CHECK(approx <= precise + (long)accuracy);
+
+ percpu_counter_tree_approximate_min_max_range(approx, accuracy,
+ &precise_min, &precise_max);
+ CHECK(precise >= precise_min);
+ CHECK(precise <= precise_max);
+
+ /* Approximate comparison is definitive exactly outside of the precise range. */
+ CHECK(percpu_counter_tree_approximate_compare_value(t, precise_min - 1) == -1);
+ CHECK(percpu_counter_tree_approximate_compare_value(t, precise_min) == 0);
+ CHECK(percpu_counter_tree_approximate_compare_value(t, precise_max) == 0);
+ CHECK(percpu_counter_tree_approximate_compare_value(t, precise_max + 1) == 1);
+
+ values[0] = precise - 1;
+ values[1] = precise;
+ values[2] = precise + 1;
+ values[3] = precise_min - 1;
+ values[4] = precise_min;
+ values[5] = precise_max;
+ values[6] = precise_max + 1;
+ for (i = 0; i < ARRAY_SIZE(values); i++) {
+ long v = values[i];
+ int expect = sign(v - precise);
+ int res = percpu_counter_tree_approximate_compare_value(t, v);
+
+ CHECK(res == 0 || res == expect);
+ CHECK(percpu_counter_tree_precise_compare_value(t, v) == expect);
+ }
+}
+
+/* Check the comparison of two counters against their shadow sums. */
+static void check_counter_pair(struct test_counter *a, struct test_counter *b)
+{
+ int expect = sign(a->shadow - b->shadow);
+ int res = percpu_counter_tree_approximate_compare(&a->tree, &b->tree);
+
+ CHECK(res == 0 || res == expect);
+ CHECK(percpu_counter_tree_precise_compare(&a->tree, &b->tree) == expect);
+ res = percpu_counter_tree_approximate_compare(&b->tree, &a->tree);
+ CHECK(res == 0 || res == -expect);
+ CHECK(percpu_counter_tree_precise_compare(&b->tree, &a->tree) == -expect);
+}
+
+/*
+ * On each CPU, add @inc (1 or -1) until the next addition would propagate a
+ * carry to the approximate sum. This leaves every tree item on the path from
+ * this CPU to the root holding its largest residual in the direction of @inc.
+ * Each addition is first tried on @probe, which holds the same state as @c:
+ * reverting an addition which reached the approximate sum would not restore
+ * the residuals subtracted on the way. The path from a CPU to the root
+ * holds less than the accuracy bound: more additions mean that a carry was
+ * lost.
+ */
+static void saturate(struct test_counter *c, struct test_counter *probe, long inc)
+{
+ unsigned long max_steps;
+ unsigned int cpu;
+
+ max_steps = percpu_counter_tree_approximate_accuracy(&c->tree) +
+ (1UL << c->tree.batch_order);
+ for (cpu = 0; cpu < nr_cpu_ids; cpu++) {
+ unsigned long steps;
+
+ for (steps = 0; steps < max_steps; steps++) {
+ long approx = percpu_counter_tree_approximate_sum(&probe->tree);
+
+ test_counter_add(probe, cpu, inc);
+ if (percpu_counter_tree_approximate_sum(&probe->tree) != approx) {
+ test_counter_copy(probe, c);
+ break;
+ }
+ test_counter_add(c, cpu, inc);
+ }
+ if (steps == max_steps) {
+ CHECK(0);
+ return;
+ }
+ }
+}
+
+static void test_saturated(unsigned long batch_size, long inc)
+{
+ struct test_counter c, probe, exact;
+ unsigned long accuracy;
+ long approx, delta;
+
+ if (test_counter_init(&c, batch_size) || test_counter_init(&probe, batch_size) ||
+ test_counter_init(&exact, batch_size)) {
+ CHECK(0);
+ return;
+ }
+ saturate(&c, &probe, inc);
+ approx = percpu_counter_tree_approximate_sum(&c.tree);
+ accuracy = percpu_counter_tree_approximate_accuracy(&c.tree);
+ /* Symmetric residuals: the accuracy is reached in both directions. */
+ CHECK((c.shadow - approx) * inc == (long)tree_residual_capacity(batch_size));
+ CHECK((c.shadow - approx) * inc == (long)accuracy);
+ check_counter(&c);
+
+ /* Compare with exactly approximated counters around the saturated precise sum. */
+ for (delta = -1; delta <= 1; delta++) {
+ test_counter_set(&exact, c.shadow + delta);
+ CHECK(percpu_counter_tree_approximate_sum(&exact.tree) == exact.shadow);
+ check_counter_pair(&c, &exact);
+ }
+ test_counter_destroy(&c);
+ test_counter_destroy(&probe);
+ test_counter_destroy(&exact);
+}
+
+static long random_increment(unsigned long batch_size)
+{
+ long v;
+
+ switch (rnd() % 4) {
+ case 0:
+ v = rnd() % 4;
+ break;
+ case 1:
+ v = rnd() % (4 * batch_size);
+ break;
+ case 2:
+ v = rnd() % (64 * batch_size);
+ break;
+ default:
+ v = rnd() % 1000000;
+ break;
+ }
+ return rnd() & 1 ? v : -v;
+}
+
+static void test_random_walk(unsigned long batch_size, unsigned long nr_steps)
+{
+ struct test_counter c[2];
+ unsigned long step;
+
+ if (test_counter_init(&c[0], batch_size) || test_counter_init(&c[1], batch_size)) {
+ CHECK(0);
+ return;
+ }
+ for (step = 0; step < nr_steps; step++) {
+ struct test_counter *t = &c[rnd() & 1];
+
+ test_counter_add(t, rnd() % nr_cpu_ids, random_increment(batch_size));
+ /* Bring the counters close to each other to exercise the precise comparison. */
+ if (!(rnd() % 8)) {
+ long delta = c[0].shadow - c[1].shadow;
+ unsigned long accuracy;
+
+ accuracy = percpu_counter_tree_approximate_accuracy(&c[0].tree);
+ delta += (long)(rnd() % (2 * accuracy + 1)) - (long)accuracy;
+ test_counter_add(&c[1], rnd() % nr_cpu_ids, delta);
+ }
+ check_counter(t);
+ check_counter_pair(&c[0], &c[1]);
+ }
+ test_counter_destroy(&c[0]);
+ test_counter_destroy(&c[1]);
+}
+
+static void test_lazy(unsigned long batch_size, unsigned long nr_steps)
+{
+ long nr_allocs = hpcc_nr_allocs;
+ struct test_counter c;
+ unsigned long step;
+
+ if (percpu_counter_tree_init_many_lazy(&c.tree, 1, batch_size)) {
+ CHECK(0);
+ return;
+ }
+ c.shadow = 0;
+ CHECK(!c.tree.level0);
+ CHECK(!c.tree.approx_sum);
+ CHECK(!percpu_counter_tree_active(&c.tree));
+ for (step = 0; step < nr_steps; step++) {
+ long inc = random_increment(batch_size);
+
+ hpcc_current_cpu = rnd() % nr_cpu_ids;
+ if (rnd() & 1)
+ percpu_counter_tree_add_owner(&c.tree, inc);
+ else
+ percpu_counter_tree_add(&c.tree, inc);
+ c.shadow += inc;
+ if (!c.tree.level0)
+ CHECK(percpu_counter_tree_approximate_sum(&c.tree) == c.shadow);
+ check_counter(&c);
+ if (step == nr_steps / 2) {
+ CHECK(!percpu_counter_tree_activate_many(&c.tree, 1, GFP_KERNEL));
+ CHECK(c.tree.level0);
+ CHECK(c.tree.approx_sum);
+ CHECK(percpu_counter_tree_active(&c.tree));
+ /* One tree: its per-CPU counters and its tree items. */
+ CHECK(hpcc_nr_allocs == nr_allocs + 2);
+ CHECK(!percpu_counter_tree_activate_many(&c.tree, 1, GFP_KERNEL));
+ CHECK(hpcc_nr_allocs == nr_allocs + 2);
+ }
+ }
+ test_counter_destroy(&c);
+ CHECK(hpcc_nr_allocs == nr_allocs);
+}
+
+static const unsigned int test_nr_cpus[] = {
+ 1, 2, 3, 4, 5, 8, 9, 16, 17, 32, 33, 64, 65, 100, 128, 256, 512,
+ 1000, 1024, 2048, 4096, 8192, 16384, 32768, 65536,
+};
+
+/* Exhaustive mode: also cover the largest supported configurations. */
+static const unsigned int test_nr_cpus_large[] = {
+ 1U << 17, 1U << 18, 1U << 19, 1U << 20,
+};
+
+static const unsigned long test_batch_sizes[] = { 2, 4, 8, 16, 32, 64, 128 };
+
+static int test_num;
+
+static int run_case(unsigned int nr_cpus, unsigned long batch_size)
+{
+ /* Each check iterates over all CPUs: bound the random walk cost. */
+ unsigned long nr_steps = 10000000 / nr_cpus;
+
+ if (nr_steps < 100)
+ nr_steps = 100;
+ if (nr_steps > 50000)
+ nr_steps = 50000;
+
+ nr_case_failures = 0;
+ hpcc_nr_warnings = 0;
+ hpcc_init_topology(nr_cpus);
+ test_saturated(batch_size, 1);
+ test_saturated(batch_size, -1);
+ test_random_walk(batch_size, nr_steps);
+ test_lazy(batch_size, nr_steps / 2);
+ CHECK(!hpcc_nr_warnings);
+ printf("%sok %d - nr_cpus=%u batch_size=%lu\n", nr_case_failures ? "not " : "",
+ ++test_num, nr_cpus, batch_size);
+ if (nr_case_failures > MAX_REPORTED_FAILURES)
+ printf("# ... %lu failed checks in total\n", nr_case_failures);
+ return nr_case_failures ? 1 : 0;
+}
+
+/* Tree shapes given with the arity boot parameter. */
+static const struct {
+ unsigned int nr_cpus;
+ const char *arity;
+ const char *shape; /* Shape in use, NULL if the arity is invalid. */
+} test_arity[] = {
+ { 16, "16", "16" },
+ { 16, "4,4", "4,4" },
+ { 16, "2,2,2,2", "2,2,2,2" },
+ { 16, "4,2", NULL }, /* Covers 8 CPUs. */
+ { 16, "4,8", NULL }, /* Covers 32 CPUs. */
+ { 16, "4,4,1", NULL }, /* Fan-out of 1. */
+ { 16, "4,3", NULL }, /* Not a power of 2. */
+ { 16, "4,4x", NULL }, /* Trailing characters. */
+ { 16, "-4,-4", NULL },
+ { 17, "8,4", "8,4" },
+ { 100, "128", "128" },
+ { 192, "8,4,8", "8,4,8" },
+ { 192, "2,2,2,2,2,2,2,2", NULL }, /* More than MAX_NR_LEVELS. */
+ { 1024, "32,32", "32,32" },
+ { 1, "2", NULL },
+};
+
+static int run_arity_case(unsigned int i)
+{
+ const char *expect = test_arity[i].shape;
+ char buf[SHAPE_BUF_SIZE];
+ int len;
+
+ nr_case_failures = 0;
+ hpcc_nr_warnings = 0;
+ tree_topology.config = NULL;
+ CHECK(!arity_param_set(test_arity[i].arity, NULL));
+ hpcc_init_topology(test_arity[i].nr_cpus);
+ arity_param = NULL;
+ CHECK(arity_param_set("16", NULL) == -EBUSY);
+ CHECK((tree_topology.config == &tree_topology.boot_config) == !!expect);
+ len = arity_param_get(buf, NULL);
+ CHECK(len > 0 && buf[len - 1] == '\n');
+ buf[len - 1] = '\0';
+ if (expect) {
+ CHECK(!strcmp(buf, expect));
+ test_saturated(2, 1);
+ test_saturated(32, -1);
+ test_random_walk(32, 20000);
+ }
+ CHECK(!hpcc_nr_warnings);
+ printf("%sok %d - arity nr_cpus=%u \"%s\" -> \"%s\"\n", nr_case_failures ? "not " : "",
+ ++test_num, test_arity[i].nr_cpus, test_arity[i].arity, buf);
+ return nr_case_failures ? 1 : 0;
+}
+
+static int run_check_config(void)
+{
+ unsigned int nr_errors = percpu_counter_tree_check_config();
+
+ printf("%sok %d - configuration table\n", nr_errors ? "not " : "", ++test_num);
+ return nr_errors ? 1 : 0;
+}
+
+static void usage(const char *name)
+{
+ fprintf(stderr, "Usage: %s [-a]\n", name);
+ fprintf(stderr, " -a Also test the largest supported numbers of CPUs (slow).\n");
+}
+
+int main(int argc, char **argv)
+{
+ unsigned int i, j, nr_tests;
+ bool large = false;
+ int failed = 0;
+
+ if (argc > 2 || (argc == 2 && strcmp(argv[1], "-a"))) {
+ usage(argv[0]);
+ return 2;
+ }
+ if (argc == 2)
+ large = true;
+
+ nr_tests = 1 + ARRAY_SIZE(test_arity) +
+ ARRAY_SIZE(test_nr_cpus) * ARRAY_SIZE(test_batch_sizes);
+ if (large)
+ nr_tests += ARRAY_SIZE(test_nr_cpus_large) * ARRAY_SIZE(test_batch_sizes);
+ printf("1..%u\n", nr_tests);
+
+ failed += run_check_config();
+ for (i = 0; i < ARRAY_SIZE(test_arity); i++)
+ failed += run_arity_case(i);
+ for (i = 0; i < ARRAY_SIZE(test_nr_cpus); i++)
+ for (j = 0; j < ARRAY_SIZE(test_batch_sizes); j++)
+ failed += run_case(test_nr_cpus[i], test_batch_sizes[j]);
+ if (large) {
+ for (i = 0; i < ARRAY_SIZE(test_nr_cpus_large); i++)
+ for (j = 0; j < ARRAY_SIZE(test_batch_sizes); j++)
+ failed += run_case(test_nr_cpus_large[i], test_batch_sizes[j]);
+ }
+ printf("# %d/%u test cases failed\n", failed, nr_tests);
+ return failed ? 1 : 0;
+}
|
| 21/3 | 2026/11/03 12:15 | llm | kmsan-evaluator |
1mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 65779 output: 23 thoughts: 1643 Thoughts: **KMSAN Fuzzing Session Justification: Hierarchical Per-CPU Counters (HPCC)**
Okay, let's break this down. My goal is to determine if this "HPCC" patch, which replaces the existing `percpu_counter` with `percpu_counter_tree` for `mm->rss_stat`, warrants a dedicated KMSAN fuzzing session. I need to follow the rules: does this patch introduce risks of uninitialized memory usage or data leaks that *only* KMSAN would catch, and are not already handled by KASAN, LOCKDEP, UBSAN, etc.?
First, I review the general instructions. KMSAN is for detecting reads of uninitialized memory, particularly when standard bug detectors are insufficient. It's resource-intensive, so I only use it if the patch specifically creates an uninitialized memory risk. I focus on these areas: kernel structures to user space, conditional logic based on potentially uninitialized variables, complex data structure allocation/initialization where uninitialized fields could be read later, and bounds checks exposing uninitialized data. If the changes are mainly related to out-of-bounds, overflows, or other bugs that KASAN catches, it's a "NeedsKMSAN=false" case.
Now, let's examine the patch. It's a significant change, introducing a new hierarchical counter. I will assess the code areas to determine if this new hierarchical structure poses a risk:
1. **Documentation/admin-guide/kernel-parameters.txt, Documentation/core-api/index.rst, Documentation/core-api/percpu-counter-tree.rst:** I will review these documentation changes to understand the scope and intended use of the new counter.
2. **MAINTAINERS:** I'll check if the relevant maintainers are listed.
3. **fs/exec.c, include/linux/mm.h, include/linux/mm\_types.h:** These files show where the new counter is being used and how it interacts with memory management structures. I look closely at `mm->rss_stat`, `rss_owner`, and `rss_stat_mutex`, and `get_mm_counter`, `get_mm_counter_sum`, and `__add_mm_counter`. I need to see how these are initialized and whether there are any conditional branches depending on potentially uninitialized values. I verify the struct `percpu_counter_tree` and associated members and functions.
4. **include/linux/percpu\_counter\_tree.h:** I'll analyze the functions related to counter initialization, addition, and destruction. I focus on anything that could potentially lead to uninitialized memory being read.
5. **include/trace/events/kmem.h:** I need to determine if memory counters are being written in any of these functions to tracepoints.
6. **init/main.c:** I will verify the initialization function.
7. **kernel/exit.c, kernel/fork.c:** These are key areas, especially concerning `mm` structures and the lifecycle of these counters during process creation, destruction, and memory management. I look carefully at `mm_init`, `dup_mm`, `copy_mm`, `exit_mm`, and `__mmdrop`.
* In `kernel/fork.c`'s `mm_init`, I see `memset(&mm->rss_stat, 0, sizeof(mm->rss_stat));` and `percpu_counter_tree_init_many_lazy`. This is good: initialization is explicitly zeroing the memory. There is also a `mutex_init`.
* I see a lot of function calls, but the `__mmdrop` has `percpu_counter_tree_destroy_many`, which will clean up memory.
8. **lib/percpu\_counter\_tree.c:** Now the crucial part. This is the implementation. I need to scrutinize how the memory is allocated and handled within the `percpu_counter_tree` functions. The `alloc_items` function, uses `kmem_cache_zalloc` and `kzalloc`, both of which zero out memory, so that is promising. The `alloc_percpu` function uses `__alloc_percpu_gfp`. I will verify that the memory allocated by `__alloc_percpu_gfp` is zeroed. I will confirm this zeroing behavior using grepper or by reading the file.
If all memory allocations and accesses within the `percpu_counter_tree` structure are consistently zeroed or initialized, and there are no direct transfers of potentially uninitialized memory to user space, then a dedicated KMSAN fuzzing session is likely *not* justified. However, if there are any points where uninitialized memory could be read
|
| 22/3 | 2026/11/03 12:16 | tool | grepper |
0mArgs:{
"Expression": "define __alloc_percpu_gfp"
}Results: {
"Output": "include/linux/percpu.h=137=extern void __percpu *pcpu_alloc_noprof(size_t size, size_t align, bool reserved,\n--\ninclude/linux/percpu.h-139-\ninclude/linux/percpu.h:140:#define __alloc_percpu_gfp(_size, _align, _gfp)\t\t\t\t\\\ninclude/linux/percpu.h-141-\talloc_hooks(pcpu_alloc_noprof(_size, _align, false, _gfp))\n"
} |
| 23/3 | 2026/11/03 12:16 | llm | kmsan-evaluator |
2mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 68614 output: 35 thoughts: 26 Thoughts: Here's my thought process, as requested: **Memory Allocation and Initialization - A Quick Check** Okay, so I'm looking at memory allocation routines, specifically `pcpu_alloc_noprof` or `pcpu_alloc`. My immediate concern, as always, is memory hygiene. I need to quickly ascertain whether these functions initialize the allocated memory to zero. This is crucial for several reasons: it prevents information leaks from previously used memory, it simplifies debugging by providing a known initial state, and it often avoids subtle, hard-to-track bugs that can arise from uninitialized data. A quick confirmation on the zeroing behavior is critical before proceeding with any assumptions. |
| 24/3 | 2026/11/03 12:19 | tool | grepper |
0mArgs:{
"Expression": "pcpu_alloc_noprof",
"PathPrefixes": [
"mm/percpu.c"
]
}Results: {
"Output": "mm/percpu.c=1715=static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t size)\n--\nmm/percpu.c-1733- */\nmm/percpu.c:1734:void __percpu *pcpu_alloc_noprof(size_t size, size_t align, bool reserved,\nmm/percpu.c-1735-\t\t\t\t gfp_t gfp)\n--\nmm/percpu.c-1944-}\nmm/percpu.c:1945:EXPORT_SYMBOL_GPL(pcpu_alloc_noprof);\nmm/percpu.c-1946-\n"
} |
| 25/3 | 2026/11/03 12:19 | llm | kmsan-evaluator |
1mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 69004 output: 35 thoughts: 19 Thoughts: Okay, let's dive into this. I'm going to pull up `mm/percpu.c` and focus around line 1734. I'm likely interested in something specific in this section, either performance, memory management, or maybe a potential bug that was mentioned elsewhere. Whatever the context, I want to refresh my memory of the surrounding code and identify the exact area of the percpu implementation that's being used. Now to read that section and identify what I need to see. |
| 26/3 | 2026/11/03 12:20 | tool | read-file |
0mArgs:{
"File": "mm/percpu.c",
"FirstLine": 1880,
"LineCount": 50
}Results: {
"Contents": "1880:\t\t\tfor_each_clear_bitrange_from(rs, re, chunk-\u003epopulated, page_end) {\n1881:\t\t\t\tWARN_ON(chunk-\u003eimmutable);\n1882:\t\n1883:\t\t\t\tret = pcpu_populate_chunk(chunk, rs, re, pcpu_gfp);\n1884:\t\n1885:\t\t\t\tspin_lock_irqsave(\u0026pcpu_lock, flags);\n1886:\t\t\t\tif (ret) {\n1887:\t\t\t\t\tpcpu_free_area(chunk, off);\n1888:\t\t\t\t\terr = \"failed to populate\";\n1889:\t\t\t\t\tgoto fail_unlock;\n1890:\t\t\t\t}\n1891:\t\t\t\tpcpu_chunk_populated(chunk, rs, re);\n1892:\t\t\t\tspin_unlock_irqrestore(\u0026pcpu_lock, flags);\n1893:\t\t\t}\n1894:\t\n1895:\t\t\tmutex_unlock(\u0026pcpu_alloc_mutex);\n1896:\t\t}\n1897:\t\n1898:\t\t/* clear the areas and return address relative to base address */\n1899:\t\tfor_each_possible_cpu(cpu)\n1900:\t\t\tmemset((void *)pcpu_chunk_addr(chunk, cpu, 0) + off, 0, size);\n1901:\t\n1902:\t\tptr = __addr_to_pcpu_ptr(chunk-\u003ebase_addr + off);\n1903:\t\tkmemleak_alloc_percpu(ptr, size, gfp);\n1904:\t\n1905:\t\ttrace_percpu_alloc_percpu(_RET_IP_, reserved, is_atomic, size, align,\n1906:\t\t\t\t\t chunk-\u003ebase_addr, off, ptr,\n1907:\t\t\t\t\t pcpu_obj_full_size(size), gfp);\n1908:\t\n1909:\t\tpcpu_memcg_post_alloc_hook(objcg, chunk, off, size);\n1910:\t\n1911:\t\tpcpu_alloc_tag_alloc_hook(chunk, off, size);\n1912:\t\n1913:\t\treturn ptr;\n1914:\t\n1915:\tfail_unlock:\n1916:\t\tspin_unlock_irqrestore(\u0026pcpu_lock, flags);\n1917:\tfail:\n1918:\t\ttrace_percpu_alloc_percpu_fail(reserved, is_atomic, size, align);\n1919:\t\n1920:\t\tif (do_warn) {\n1921:\t\t\tint remaining = atomic_dec_if_positive(\u0026warn_limit);\n1922:\t\n1923:\t\t\tif (remaining \u003e= 0) {\n1924:\t\t\t\tpr_warn(\"allocation failed, size=%zu align=%zu atomic=%d, %s\\n\",\n1925:\t\t\t\t\tsize, align, is_atomic, err);\n1926:\t\t\t\tif (!is_atomic)\n1927:\t\t\t\t\tdump_stack();\n1928:\t\t\t\tif (remaining == 0)\n1929:\t\t\t\t\tpr_info(\"limit reached, disable warning\\n\");\n"
} |
| 27/3 | 2026/11/03 12:20 | llm | kmsan-evaluator |
1mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 69974 output: 17 thoughts: 801 Thoughts: **Analyzing Memory Initialization in the Kernel** Okay, let's break down this code. It seems we're looking at memory initialization patterns across the kernel, specifically focusing on per-CPU and process memory. My initial observation is on lines 1899-1900, where a `for_each_possible_cpu` loop calls `memset` to zero out per-CPU memory using `pcpu_chunk_addr`. This is a straightforward, direct approach to ensure clean memory. Now, let's consider the allocation of `items`, likely related to counters. The `percpu_counter_tree_alloc_items` function uses either `kmem_cache_zalloc` or `kzalloc`. Both these functions are important, because, as you know, both are designed to return zeroed memory. That's a key point: zeroing is baked into the allocation. Next, I'm examining `__percpu_counter_tree_init`. This function initializes the `struct percpu_counter_tree`. Here, we see several fields being explicitly set: `batch_order` (derived from `batch_size`), `bias`, `atomic_bias`, `level0`, `items_stride`, and `approx_sum`. Importantly, `bias` and `atomic_bias` are initialized to 0, which is important for any counter implementation, as it represents the offset. `approx_sum` is initialized based on whether `items` is non-null, again likely tied to counter activation. This indicates that the core counter structure is getting a defined, initialized state. Finally, the code in `kernel/fork.c` is on my radar. We see `memset` being used to zero out `mm->rss_stat`. A mutex is initialized and then `percpu_counter_tree_init_many_lazy` is called to init the rss counters. Also `mm->rss_owner` being initialized to NULL in `mm_init` (or inherited/set in dup_mm / exec_mmap) means that the memory management is handled and initialized safely. I need to drill down and confirm how the `mm` structure itself is allocated. Is it `kzalloc` or `memset` within `mm_alloc`, so the entire memory segment starts clean? This initial zeroing is critical for avoiding garbage data and potential security issues, especially when dealing with process memory. |
| 28/3 | 2026/11/03 12:22 | tool | grepper |
0mArgs:{
"Expression": "allocate_mm"
}Results: {
"Output": "drivers/gpu/drm/hyperv/hyperv_drm_drv.c=71=static int hv_drm_setup_vram(struct hv_drm_device *hv,\n--\ndrivers/gpu/drm/hyperv/hyperv_drm_drv.c-78-\ndrivers/gpu/drm/hyperv/hyperv_drm_drv.c:79:\tret = vmbus_allocate_mmio(\u0026hv-\u003emem, hdev, 0, -1, hv-\u003efb_size, 0x100000,\ndrivers/gpu/drm/hyperv/hyperv_drm_drv.c-80-\t\t\t\t true);\n--\ndrivers/hv/vmbus_drv.c=2374=static void __maybe_unused vmbus_reserve_fb(void)\n--\ndrivers/hv/vmbus_drv.c-2440-/**\ndrivers/hv/vmbus_drv.c:2441: * vmbus_allocate_mmio() - Pick a memory-mapped I/O range.\ndrivers/hv/vmbus_drv.c-2442- * @new:\t\tIf successful, supplied a pointer to the\n--\ndrivers/hv/vmbus_drv.c-2462- */\ndrivers/hv/vmbus_drv.c:2463:int vmbus_allocate_mmio(struct resource **new, struct hv_device *device_obj,\ndrivers/hv/vmbus_drv.c-2464-\t\t\tresource_size_t min, resource_size_t max,\n--\ndrivers/hv/vmbus_drv.c-2531-}\ndrivers/hv/vmbus_drv.c:2532:EXPORT_SYMBOL_GPL(vmbus_allocate_mmio);\ndrivers/hv/vmbus_drv.c-2533-\n--\ndrivers/hv/vmbus_drv.c=2542=void vmbus_free_mmio(resource_size_t start, resource_size_t size)\n--\ndrivers/hv/vmbus_drv.c-2551-\t * region since no corresponding __request_region() was done\ndrivers/hv/vmbus_drv.c:2552:\t * in vmbus_allocate_mmio().\ndrivers/hv/vmbus_drv.c-2553-\t */\n--\ndrivers/pci/controller/pci-hyperv.c=3321=static void hv_pci_free_bridge_windows(struct hv_pcibus_device *hbus)\n--\ndrivers/pci/controller/pci-hyperv.c-3345- *\ndrivers/pci/controller/pci-hyperv.c:3346: * This function calls vmbus_allocate_mmio(), which is itself a\ndrivers/pci/controller/pci-hyperv.c-3347- * bit of a compromise. Ideally, we might change the pnp layer\n--\ndrivers/pci/controller/pci-hyperv.c=3366=static int hv_pci_allocate_bridge_windows(struct hv_pcibus_device *hbus)\n--\ndrivers/pci/controller/pci-hyperv.c-3372-\t\talign = 1ULL \u003c\u003c (63 - __builtin_clzll(hbus-\u003elow_mmio_space));\ndrivers/pci/controller/pci-hyperv.c:3373:\t\tret = vmbus_allocate_mmio(\u0026hbus-\u003elow_mmio_res, hbus-\u003ehdev, 0,\ndrivers/pci/controller/pci-hyperv.c-3374-\t\t\t\t\t (u64)(u32)0xffffffff,\n--\ndrivers/pci/controller/pci-hyperv.c-3391-\t\talign = 1ULL \u003c\u003c (63 - __builtin_clzll(hbus-\u003ehigh_mmio_space));\ndrivers/pci/controller/pci-hyperv.c:3392:\t\tret = vmbus_allocate_mmio(\u0026hbus-\u003ehigh_mmio_res, hbus-\u003ehdev,\ndrivers/pci/controller/pci-hyperv.c-3393-\t\t\t\t\t 0x100000000, -1,\n--\ndrivers/pci/controller/pci-hyperv.c=3429=static int hv_allocate_config_window(struct hv_pcibus_device *hbus)\n--\ndrivers/pci/controller/pci-hyperv.c-3436-\t */\ndrivers/pci/controller/pci-hyperv.c:3437:\tret = vmbus_allocate_mmio(\u0026hbus-\u003emem_config, hbus-\u003ehdev, 0, -1,\ndrivers/pci/controller/pci-hyperv.c-3438-\t\t\t\t PCI_CONFIG_MMIO_LENGTH, 0x1000, false);\n--\ndrivers/pci/controller/pci-hyperv.c-3442-\t/*\ndrivers/pci/controller/pci-hyperv.c:3443:\t * vmbus_allocate_mmio() gets used for allocating both device endpoint\ndrivers/pci/controller/pci-hyperv.c-3444-\t * resource claims (those which cannot be overlapped) and the ranges\n--\ninclude/linux/hyperv.h=1382=void vmbus_hvsock_device_unregister(struct vmbus_channel *channel);\ninclude/linux/hyperv.h-1383-\ninclude/linux/hyperv.h:1384:int vmbus_allocate_mmio(struct resource **new, struct hv_device *device_obj,\ninclude/linux/hyperv.h-1385-\t\t\tresource_size_t min, resource_size_t max,\n--\nkernel/fork.c=650=static void check_mm(struct mm_struct *mm)\n--\nkernel/fork.c-676-\nkernel/fork.c:677:#define allocate_mm()\t(kmem_cache_alloc(mm_cachep, GFP_KERNEL))\nkernel/fork.c-678-#define free_mm(mm)\t(kmem_cache_free(mm_cachep, (mm)))\n--\nkernel/fork.c=1193=struct mm_struct *mm_alloc(void)\n--\nkernel/fork.c-1196-\nkernel/fork.c:1197:\tmm = allocate_mm();\nkernel/fork.c-1198-\tif (!mm)\n--\nkernel/fork.c=1548=static struct mm_struct *dup_mm(struct task_struct *tsk,\n--\nkernel/fork.c-1553-\nkernel/fork.c:1554:\tmm = allocate_mm();\nkernel/fork.c-1555-\tif (!mm)\n"
} |
| 29/3 | 2026/11/03 12:22 | llm | kmsan-evaluator |
1mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 73007 output: 34 thoughts: 23 Thoughts: Here's my thought process, as if I were considering those lines of code: **Diving into the Memory Management of `fork()`** Okay, time to get my hands dirty with `kernel/fork.c`. Specifically, I want to refresh my memory on the memory management setup during a `fork()` call. I'm focusing on where the memory structures are allocated, and how they get initialized for the new child process. Let's start by zeroing in on `allocate_mm`. I know that's where the initial `mm_struct` is likely allocated for the new process. Is it a copy, a new allocation, or something more clever using COW (Copy-on-Write) principles? I need to understand how the MMU is brought into the picture, and what happens to the page tables. Then I'll jump to `mm_init`. This is where the virtual memory space gets set up, but what exactly gets copied and when? Do they share any memory initially before a write, or am I missing something? Knowing the interaction between these two functions will be the key to understanding how `fork()` efficiently creates a new process image. From the architecture point of view, how much of this is architecture dependent? |
| 30/3 | 2026/11/03 12:24 | tool | read-file |
0mArgs:{
"File": "kernel/fork.c",
"FirstLine": 1100,
"LineCount": 40
}Results: {
"Contents": "1100:\t#ifdef CONFIG_UPROBES\n1101:\t\tmm-\u003euprobes_state.xol_area = NULL;\n1102:\t#endif\n1103:\t}\n1104:\t\n1105:\tstatic void mmap_init_lock(struct mm_struct *mm)\n1106:\t{\n1107:\t\tinit_rwsem(\u0026mm-\u003emmap_lock);\n1108:\t\tmm_lock_seqcount_init(mm);\n1109:\t\trcuwait_init(\u0026mm-\u003evma_writer_wait);\n1110:\t}\n1111:\t\n1112:\tstatic struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p)\n1113:\t{\n1114:\t\tmt_init_flags(\u0026mm-\u003emm_mt, MM_MT_FLAGS);\n1115:\t\tmt_set_external_lock(\u0026mm-\u003emm_mt, \u0026mm-\u003emmap_lock);\n1116:\t\tatomic_set(\u0026mm-\u003emm_users, 1);\n1117:\t\tatomic_set(\u0026mm-\u003emm_count, 1);\n1118:\t\tseqcount_init(\u0026mm-\u003ewrite_protect_seq);\n1119:\t\tmmap_init_lock(mm);\n1120:\t\tINIT_LIST_HEAD(\u0026mm-\u003emmlist);\n1121:\t\tmm_pgtables_bytes_init(mm);\n1122:\t\tmm-\u003emap_count = 0;\n1123:\t\tmm-\u003elocked_vm = 0;\n1124:\t\tatomic64_set(\u0026mm-\u003epinned_vm, 0);\n1125:\t\tmemset(\u0026mm-\u003erss_stat, 0, sizeof(mm-\u003erss_stat));\n1126:\t\tspin_lock_init(\u0026mm-\u003epage_table_lock);\n1127:\t\tspin_lock_init(\u0026mm-\u003earg_lock);\n1128:\t\tmutex_init(\u0026mm-\u003erss_stat_mutex);\n1129:\t\tmm_init_cpumask(mm);\n1130:\t\tmm_init_aio(mm);\n1131:\t\tmm_init_owner(mm, p);\n1132:\t\tmm_pasid_init(mm);\n1133:\t\tRCU_INIT_POINTER(mm-\u003eexe_file, NULL);\n1134:\t\tmmu_notifier_subscriptions_init(mm);\n1135:\t\tinit_tlb_flush_pending(mm);\n1136:\t#if defined(CONFIG_TRANSPARENT_HUGEPAGE) \u0026\u0026 !defined(CONFIG_SPLIT_PMD_PTLOCKS)\n1137:\t\tmm-\u003epmd_huge_pte = NULL;\n1138:\t#endif\n1139:\t\tmm_init_uprobes_state(mm);\n"
} |
| 31/3 | 2026/11/03 12:24 | llm | kmsan-evaluator |
1mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 74053 output: 34 thoughts: 249 Thoughts: **Investigating Memory Management Initialization**
Okay, hold on a second! I need to trace the initialization of the memory management structure (`mm_struct`). I'm looking specifically at `mm_init` – line 1125, there's a `memset` call: `memset(&mm->rss_stat, 0, sizeof(mm->rss_stat));`. But what about `mm->rss_owner`? Where is *that* initialized? Is it handled within `mm_alloc()` or perhaps within `mm_init` itself?
Right, let's dive deeper. I'm going to follow the trail. `mm_alloc()` is called, and I see it, from `kernel/fork.c`. It calls `mm_alloc()`:
`struct mm_struct *mm_alloc(void)`
`{`
` struct mm_struct *mm;`
` mm = allocate_mm();`
` if (!mm)`
` return NULL;`
` memset(mm, 0, sizeof(*mm));`
` return mm_init(mm, current);`
`}`
Aha! I see it *does* `memset` the entire structure to zero *before* calling `mm_init`. Good. But I need to be absolutely certain this is consistent. I have to verify if `memset(mm, 0, sizeof(*mm))` is also called in `dup_mm()`. Let me check lines 1190-1210 and 1550-1570 of `kernel/fork.c` to be sure. Consistency is key here.
|
| 32/3 | 2026/11/03 12:25 | tool | read-file |
0mArgs:{
"File": "kernel/fork.c",
"FirstLine": 1190,
"LineCount": 20
}Results: {
"Contents": "1190:\t/*\n1191:\t * Allocate and initialize an mm_struct.\n1192:\t */\n1193:\tstruct mm_struct *mm_alloc(void)\n1194:\t{\n1195:\t\tstruct mm_struct *mm;\n1196:\t\n1197:\t\tmm = allocate_mm();\n1198:\t\tif (!mm)\n1199:\t\t\treturn NULL;\n1200:\t\n1201:\t\tmemset(mm, 0, sizeof(*mm));\n1202:\t\treturn mm_init(mm, current);\n1203:\t}\n1204:\tEXPORT_SYMBOL_IF_KUNIT(mm_alloc);\n1205:\t\n1206:\tstatic inline void __mmput(struct mm_struct *mm)\n1207:\t{\n1208:\t\tVM_BUG_ON(atomic_read(\u0026mm-\u003emm_users));\n1209:\t\n"
} |
| 33/3 | 2026/11/03 12:25 | llm | kmsan-evaluator |
1mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 74954 output: 34 thoughts: 9 Thoughts: Okay, here's my attempt at summarizing the thought process, tailored for an expert audience, and written in the first person: **Analyzing the `dup_mm` Function** Alright, let's dive into this `dup_mm` function. My initial thought process revolves around understanding its purpose and potential pitfalls. Given its naming, I'm already anticipating that it has something to do with memory management and, most likely, duplicating or allocating memory. The "dup" prefix immediately flags a potential area for concern: are we talking about shallow copies, deep copies, or something else entirely? The "mm" suffix hints at a lower-level memory management interaction, perhaps dealing directly with pointers or regions of memory. I need to confirm the exact nature of the duplication. Is this a simple bitwise copy, or does it require more sophisticated data structure awareness? For example, is it responsible for duplicating all associated data structures, if it's operating on a complex object? Are there any pointer manipulations within the function itself? I need to know if the duplication process is correct, especially when dealing with nested structures, circular references, or dynamically allocated memory. Next, I'll need to assess the function's memory safety. Is it correctly handling allocation and deallocation? Are there any potential memory leaks or double-frees? Is error handling robust, especially regarding scenarios where memory allocation fails? If it's dealing with copying a section of memory into a new location, I need to check if the allocation target is large enough to contain the source, and to ensure there's no buffer overflow or underrun. I'll need to check the inputs and preconditions, and the memory footprint it has, and assess if it can introduce any performance bottlenecks. Overall, I need a detailed walkthrough to ensure the function behaves as expected under all reasonable conditions. |
| 34/3 | 2026/11/03 12:27 | tool | read-file |
0mArgs:{
"File": "kernel/fork.c",
"FirstLine": 1550,
"LineCount": 25
}Results: {
"Contents": "1550:\t{\n1551:\t\tstruct mm_struct *mm;\n1552:\t\tint err;\n1553:\t\n1554:\t\tmm = allocate_mm();\n1555:\t\tif (!mm)\n1556:\t\t\tgoto fail_nomem;\n1557:\t\n1558:\t\tmemcpy(mm, oldmm, sizeof(*mm));\n1559:\t\n1560:\t\tif (!mm_init(mm, tsk))\n1561:\t\t\tgoto fail_nomem;\n1562:\t\t/* The child owns its RSS counters, see __add_mm_counter(). */\n1563:\t\tmm-\u003erss_owner = tsk;\n1564:\t\n1565:\t\tuprobe_start_dup_mmap();\n1566:\t\terr = dup_mmap(mm, oldmm);\n1567:\t\tif (err)\n1568:\t\t\tgoto free_pt;\n1569:\t\tuprobe_end_dup_mmap();\n1570:\t\n1571:\t\tmm-\u003ehiwater_rss = get_mm_rss(mm);\n1572:\t\tmm-\u003ehiwater_vm = mm-\u003etotal_vm;\n1573:\t\n1574:\t\tif (mm-\u003ebinfmt \u0026\u0026 !try_module_get(mm-\u003ebinfmt-\u003emodule))\n"
} |
| 35/3 | 2026/11/03 12:27 | llm | kmsan-evaluator |
2mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 75735 output: 31 thoughts: 158 Thoughts: **Memory Management Conundrum** Okay, so I'm looking at this `dup_mm` function, and the first thing that jumps out is that `memcpy`. It copies `oldmm` into `mm`, which is standard. But then, `mm_init(mm, tsk)` gets called. That's where things get interesting. My immediate thought is: does `mm_init` zero out `rss_owner`? Because if so, the previous `memcpy` is pretty pointless in respect of `rss_owner`. Hold on... I'm getting ahead of myself. Right, re-reading `dup_mm`, I see `mm->rss_owner = tsk;` explicitly sets that field *after* the `memcpy`. That clears things up a bit. But wait, what about the memory allocation itself? If `mm_alloc()` is called, doesn't that initialize the memory with `memset(mm, 0, sizeof(*mm))`? In that case, wouldn't `rss_owner` be initialized to NULL? That makes a difference to how we design around this. And what about `init_mm`? That's a key initialization function. I need to make sure I understand that. Right, I'm going to quickly check `init_mm` in `mm/init-mm.c`. I need to see if it's been modified, or if `rss_stat` is involved. Understanding how `init_mm` works will shed a lot of light on how all these other functions interact and, therefore, the optimal way to write code that interacts with memory management. |
| 36/3 | 2026/11/03 12:29 | tool | grepper |
0mArgs:{
"Expression": "init_mm",
"PathPrefixes": [
"mm/init-mm.c"
]
}Results: {
"Output": "mm/init-mm.c=20=const struct vm_operations_struct vma_dummy_vm_ops;\n--\nmm/init-mm.c-28- *\nmm/init-mm.c:29: * Since there is only one init_mm in the entire system, keep it simple\nmm/init-mm.c-30- * and size this cpu_bitmask to NR_CPUS.\nmm/init-mm.c-31- */\nmm/init-mm.c:32:struct mm_struct init_mm = {\nmm/init-mm.c:33:\t.mm_mt\t\t= MTREE_INIT_EXT(mm_mt, MM_MT_FLAGS, init_mm.mmap_lock),\nmm/init-mm.c-34-\t.pgd\t\t= swapper_pg_dir,\n--\nmm/init-mm.c-36-\t.mm_count\t= ATOMIC_INIT(1),\nmm/init-mm.c:37:\t.write_protect_seq = SEQCNT_ZERO(init_mm.write_protect_seq),\nmm/init-mm.c:38:\tMMAP_LOCK_INITIALIZER(init_mm)\nmm/init-mm.c:39:\t.page_table_lock = __SPIN_LOCK_UNLOCKED(init_mm.page_table_lock),\nmm/init-mm.c:40:\t.arg_lock\t= __SPIN_LOCK_UNLOCKED(init_mm.arg_lock),\nmm/init-mm.c:41:\t.mmlist\t\t= LIST_HEAD_INIT(init_mm.mmlist),\nmm/init-mm.c:42:\t.vma_writer_wait = __RCUWAIT_INITIALIZER(init_mm.vma_writer_wait),\nmm/init-mm.c:43:\t.mm_lock_seq\t= SEQCNT_ZERO(init_mm.mm_lock_seq),\nmm/init-mm.c-44-#ifdef CONFIG_SCHED_MM_CID\nmm/init-mm.c:45:\t.mm_cid.lock = __RAW_SPIN_LOCK_UNLOCKED(init_mm.mm_cid.lock),\nmm/init-mm.c-46-#endif\nmm/init-mm.c-47-\t.flexible_array\t= MM_STRUCT_FLEXIBLE_ARRAY_INIT,\nmm/init-mm.c:48:\tINIT_MM_CONTEXT(init_mm)\nmm/init-mm.c-49-};\nmm/init-mm.c-50-\nmm/init-mm.c:51:void setup_initial_init_mm(void *start_code, void *end_code,\nmm/init-mm.c-52-\t\t\t void *end_data, void *brk)\nmm/init-mm.c-53-{\nmm/init-mm.c:54:\tinit_mm.start_code = (unsigned long)start_code;\nmm/init-mm.c:55:\tinit_mm.end_code = (unsigned long)end_code;\nmm/init-mm.c:56:\tinit_mm.end_data = (unsigned long)end_data;\nmm/init-mm.c:57:\tinit_mm.brk = (unsigned long)brk;\nmm/init-mm.c-58-}\n"
} |
| 37/3 | 2026/11/03 12:29 | llm | kmsan-evaluator |
1mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 77114 output: 27 thoughts: 87 Thoughts: Okay, let me lay out my thinking here. **Deconstructing Memory Counter Implementation** Right, `init_mm` is static, meaning its BSS and data sections are initialized to zero. I need to keep that in mind. However, that immediately makes me think – what about `percpu_counter_tree`? That data structure is used throughout the kernel, so how does it interact with this? And specifically, how are the RSS counters in a given memory management structure actually read and interpreted? I need to understand the mechanics of that read path. To that end, where are functions like `get_mm_counter` or `get_mm_counter_sum` even *called*? That’s the key. Let's see if I can trace the calls to `get_mm_counter` or `get_mm_rss` in the kernel. I'll fire up a grep command and get a feel for the places where these are needed. |
| 38/3 | 2026/11/03 12:31 | tool | grepper |
0mArgs:{
"Expression": "get_mm_counter\\(|get_mm_rss\\("
}Results: {
"Output": "arch/sparc/mm/fault_64.c=267=asmlinkage void __kprobes do_sparc64_fault(struct pt_regs *regs)\n--\narch/sparc/mm/fault_64.c-465-lock_released:\narch/sparc/mm/fault_64.c:466:\tmm_rss = get_mm_rss(mm);\narch/sparc/mm/fault_64.c-467-#if defined(CONFIG_TRANSPARENT_HUGEPAGE)\n--\narch/sparc/mm/tsb.c=536=int init_new_context(struct task_struct *tsk, struct mm_struct *mm)\narch/sparc/mm/tsb.c-537-{\narch/sparc/mm/tsb.c:538:\tunsigned long mm_rss = get_mm_rss(mm);\narch/sparc/mm/tsb.c-539-#if defined(CONFIG_HUGETLB_PAGE) || defined(CONFIG_TRANSPARENT_HUGEPAGE)\n--\nfs/exec.c=117=bool path_noexec(const struct path *path)\n--\nfs/exec.c-129- * use a lot of memory, account these pages in current-\u003emm temporary\nfs/exec.c:130: * for oom_badness()-\u003eget_mm_rss(). Once exec succeeds or fails, we\nfs/exec.c-131- * change the counter back via acct_arg_size(0).\n--\nfs/proc/array.c=466=static int do_task_stat(struct seq_file *m, struct pid_namespace *ns,\n--\nfs/proc/array.c-612-\tseq_put_decimal_ull(m, \" \", vsize);\nfs/proc/array.c:613:\tseq_put_decimal_ull(m, \" \", mm ? get_mm_rss(mm) : 0);\nfs/proc/array.c-614-\tseq_put_decimal_ull(m, \" \", rsslim);\n--\ninclude/linux/mm.h=3600=static inline bool get_user_page_fast_only(unsigned long addr,\n--\ninclude/linux/mm.h-3607- */\ninclude/linux/mm.h:3608:static inline unsigned long get_mm_counter(struct mm_struct *mm, int member)\ninclude/linux/mm.h-3609-{\n--\ninclude/linux/mm.h=3667=static inline int mm_counter(struct folio *folio)\n--\ninclude/linux/mm.h-3673-\ninclude/linux/mm.h:3674:static inline unsigned long get_mm_rss(struct mm_struct *mm)\ninclude/linux/mm.h-3675-{\ninclude/linux/mm.h:3676:\treturn get_mm_counter(mm, MM_FILEPAGES) +\ninclude/linux/mm.h:3677:\t\tget_mm_counter(mm, MM_ANONPAGES) +\ninclude/linux/mm.h:3678:\t\tget_mm_counter(mm, MM_SHMEMPAGES);\ninclude/linux/mm.h-3679-}\n--\ninclude/linux/mm.h=3688=static inline unsigned long get_mm_hiwater_rss(struct mm_struct *mm)\ninclude/linux/mm.h-3689-{\ninclude/linux/mm.h:3690:\treturn max(mm-\u003ehiwater_rss, get_mm_rss(mm));\ninclude/linux/mm.h-3691-}\n--\ninclude/linux/mm.h=3698=static inline void update_hiwater_rss(struct mm_struct *mm)\ninclude/linux/mm.h-3699-{\ninclude/linux/mm.h:3700:\tunsigned long _rss = get_mm_rss(mm);\ninclude/linux/mm.h-3701-\n--\ninclude/linux/mm.h=3712=static inline void reset_mm_hiwater_rss(struct mm_struct *mm)\ninclude/linux/mm.h-3713-{\ninclude/linux/mm.h:3714:\tmm-\u003ehiwater_rss = get_mm_rss(mm);\ninclude/linux/mm.h-3715-}\n--\ninclude/trace/events/kmem.h=427=TRACE_EVENT(rss_stat,\n--\ninclude/trace/events/kmem.h-450-\t\t__entry-\u003emember = member;\ninclude/trace/events/kmem.h:451:\t\t__entry-\u003esize = get_mm_counter(mm, member) \u003c\u003c PAGE_SHIFT;\ninclude/trace/events/kmem.h-452-\t),\n--\ninclude/trace/events/oom.h=76=TRACE_EVENT(mark_victim,\n--\ninclude/trace/events/oom.h-96-\t\t__entry-\u003etotal_vm = PG_COUNT_TO_KB(task-\u003emm-\u003etotal_vm);\ninclude/trace/events/oom.h:97:\t\t__entry-\u003eanon_rss = PG_COUNT_TO_KB(get_mm_counter(task-\u003emm, MM_ANONPAGES));\ninclude/trace/events/oom.h:98:\t\t__entry-\u003efile_rss = PG_COUNT_TO_KB(get_mm_counter(task-\u003emm, MM_FILEPAGES));\ninclude/trace/events/oom.h:99:\t\t__entry-\u003eshmem_rss = PG_COUNT_TO_KB(get_mm_counter(task-\u003emm, MM_SHMEMPAGES));\ninclude/trace/events/oom.h-100-\t\t__entry-\u003euid = uid;\n--\nkernel/fork.c=1548=static struct mm_struct *dup_mm(struct task_struct *tsk,\n--\nkernel/fork.c-1570-\nkernel/fork.c:1571:\tmm-\u003ehiwater_rss = get_mm_rss(mm);\nkernel/fork.c-1572-\tmm-\u003ehiwater_vm = mm-\u003etotal_vm;\n--\nkernel/sched/fair.c=2446=static unsigned int task_nr_scan_windows(struct task_struct *p)\n--\nkernel/sched/fair.c-2456-\tnr_scan_pages = MB_TO_PAGES(sysctl_numa_balancing_scan_size);\nkernel/sched/fair.c:2457:\trss = get_mm_rss(p-\u003emm);\nkernel/sched/fair.c-2458-\tif (!rss)\n--\nkernel/tsacct.c=123=static void __acct_update_integrals(struct task_struct *tsk,\n--\nkernel/tsacct.c-142-\t */\nkernel/tsacct.c:143:\ttsk-\u003eacct_rss_mem1 += delta * get_mm_rss(tsk-\u003emm) \u003e\u003e 10;\nkernel/tsacct.c-144-\ttsk-\u003eacct_vm_mem1 += delta * READ_ONCE(tsk-\u003emm-\u003etotal_vm) \u003e\u003e 10;\n"
} |
| 39/3 | 2026/11/03 12:31 | llm | kmsan-evaluator |
1mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 79218 output: 19 thoughts: 1224 Thoughts: **Analyzing Potential Uninitialized Memory Access in `percpu_counter_tree`** Okay, so I'm digging into this `percpu_counter_tree` and trying to be *very* careful about potential uninitialized memory reads or logic based on uninitialized variables. First, let me recall that `get_mm_counter` calculates an `unsigned long` value, which is then printed as a decimal in `/proc/<pid>/stat`. There's no direct raw struct or byte dump to user space from that particular function, so that avenue is less concerning. My attention is now on the sysfs parameter for `arity`. The module parameter callback, `arity_param_get`, is intriguing. It seems to potentially write to a user-provided buffer. It calls `format_shape`, which is the key here. It looks like `format_shape` formats a `counter_config` into a provided buffer using `scnprintf`, guaranteeing null termination, and appends a newline. It also returns the number of bytes written. This sysfs handler then outputs exactly this many bytes. That seems safe, as the `format_shape` function *does* seem to initialize the buffer contents. Next, I need to look into `lib/percpu_counter_tree.c`. Let's step through it slowly, with extra focus. The initialization function, `percpu_counter_tree_subsystem_init`, looks like a good place to start. It initializes `tree_topology`. It calls `get_count_order` to determine the number of CPUs. A key bit is the parsing of the `arity` parameter if provided by the bootloader. The `parse_arity` function in particular is the next point of interest. Here, an array `ints` is used to store values parsed from the input string, which determines the arity of the counter tree. This is where I have a tiny bit of concern as it involves the use of the `get_options` function. Let's see how the `get_options` function in `lib/cmdline.c` works. It's crucial that `ints` is properly initialized *before* being used. |
| 40/3 | 2026/11/03 12:33 | tool | grepper |
0mArgs:{
"Expression": "char \\*get_options"
}Results: {
"Output": "include/linux/string.h=329=extern int get_option(char **str, int *pint);\ninclude/linux/string.h:330:extern char *get_options(const char *str, int nints, int *ints);\ninclude/linux/string.h-331-extern unsigned long long memparse(const char *ptr, char **retptr);\n--\nlib/cmdline.c=83=EXPORT_SYMBOL(get_option);\n--\nlib/cmdline.c-109-\nlib/cmdline.c:110:char *get_options(const char *str, int nints, int *ints)\nlib/cmdline.c-111-{\n--\ntools/testing/hpcc/include/hpcc_shim.h-40-/* Implemented in main.c. */\ntools/testing/hpcc/include/hpcc_shim.h:41:char *get_options(const char *str, int nints, int *ints);\ntools/testing/hpcc/include/hpcc_shim.h-42-\n--\ntools/testing/hpcc/main.c=71=int scnprintf(char *buf, size_t size, const char *fmt, ...)\n--\ntools/testing/hpcc/main.c-84-/* Comma-separated integers, without the ranges of lib/cmdline.c. */\ntools/testing/hpcc/main.c:85:char *get_options(const char *str, int nints, int *ints)\ntools/testing/hpcc/main.c-86-{\n"
} |
| 41/3 | 2026/11/03 12:33 | llm | kmsan-evaluator |
1mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 81311 output: 34 thoughts: 22 Thoughts: Here's a first-person summary, tailored for an expert audience, as requested: **Diving into the Command-Line Interface** Okay, time to get my hands dirty. Let's see what's happening under the hood with the command-line parsing. I'm going to jump straight into `lib/cmdline.c`, specifically around lines 110-140. That's usually where the meat of the argument handling resides, where the application is interpreting what the user typed in and translating it into internal actions. I expect to see the main `getopt()` loop there, perhaps some custom option parsing functions, or maybe even a state machine for handling more complex argument interactions. I'll be looking for how they handle positional arguments, flag handling, and whether they're using a library like `getopt` or if they have their own, custom implementation. Knowing how they handle the argument parsing will give me a good understanding of how flexible the command line interface will be, and how easy it'll be to extend or modify in the future. I want to see how robust the error handling is too; a poorly implemented command-line interface is a usability nightmare! |
| 42/3 | 2026/11/03 12:34 | tool | read-file |
0mArgs:{
"File": "lib/cmdline.c",
"FirstLine": 110,
"LineCount": 35
}Results: {
"Contents": " 110:\tchar *get_options(const char *str, int nints, int *ints)\n 111:\t{\n 112:\t\tbool validate = (nints == 0);\n 113:\t\tint res, i = 1;\n 114:\t\n 115:\t\twhile (i \u003c nints || validate) {\n 116:\t\t\tint *pint = validate ? ints : ints + i;\n 117:\t\n 118:\t\t\tres = get_option((char **)\u0026str, pint);\n 119:\t\t\tif (res == 0)\n 120:\t\t\t\tbreak;\n 121:\t\t\tif (res == 3) {\n 122:\t\t\t\tint n = validate ? 0 : nints - i;\n 123:\t\t\t\tint range_nums;\n 124:\t\n 125:\t\t\t\trange_nums = get_range((char **)\u0026str, pint, n);\n 126:\t\t\t\tif (range_nums \u003c 0)\n 127:\t\t\t\t\tbreak;\n 128:\t\t\t\t/* The range didn't fit, so the array is full */\n 129:\t\t\t\tif (!validate \u0026\u0026 range_nums \u003e n) {\n 130:\t\t\t\t\ti = nints;\n 131:\t\t\t\t\tbreak;\n 132:\t\t\t\t}\n 133:\t\t\t\t/* Leave room for the upper number of the range */\n 134:\t\t\t\tif (range_nums \u003e= INT_MAX - i)\n 135:\t\t\t\t\tbreak;\n 136:\t\t\t\t/*\n 137:\t\t\t\t * Decrement the result by one to leave out the\n 138:\t\t\t\t * last number in the range. The next iteration\n 139:\t\t\t\t * will handle the upper number in the range\n 140:\t\t\t\t */\n 141:\t\t\t\ti += (range_nums - 1);\n 142:\t\t\t}\n 143:\t\t\ti++;\n 144:\t\t\tif (res == 1)\n"
} |
| 43/3 | 2026/11/03 12:34 | llm | kmsan-evaluator |
2mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 82121 output: 34 thoughts: 22 Thoughts: Here's my take on those lines... **Diving into `lib/cmdline.c`** Alright, let's zero in on lines 140 to 160 within `lib/cmdline.c`. I need to refresh my memory on the argument parsing logic there. I suspect there might be something interesting happening in that section. It’s been a while since I've been deep in this code, so I'm just quickly scanning my mental map of the file, getting my bearings, thinking about the probable parsing implementation. Considering I'm familiar with the overall structure, I'm expecting to see calls to `getopt()` or something similarly robust at play. I anticipate there's probably a loop, handling different flags and their corresponding arguments. I am thinking, what are the chances the issue I'm debugging is in how the options are being interpreted. I can't be sure, but it will be a good idea to refresh my memory here. Knowing the nuances of how the command-line options are handled is crucial for this investigation. Right now, I will be looking out for any potential vulnerabilities, edge case handling, or logic errors that could be leading to this observed behavior. |
| 44/3 | 2026/11/03 12:36 | tool | read-file |
0mArgs:{
"File": "lib/cmdline.c",
"FirstLine": 140,
"LineCount": 25
}Results: {
"Contents": " 140:\t\t\t\t */\n 141:\t\t\t\ti += (range_nums - 1);\n 142:\t\t\t}\n 143:\t\t\ti++;\n 144:\t\t\tif (res == 1)\n 145:\t\t\t\tbreak;\n 146:\t\t}\n 147:\t\tints[0] = i - 1;\n 148:\t\treturn (char *)str;\n 149:\t}\n 150:\tEXPORT_SYMBOL(get_options);\n 151:\t\n 152:\t/**\n 153:\t *\tmemparse - parse a string with mem suffixes into a number\n 154:\t *\t@ptr: Where parse begins\n 155:\t *\t@retptr: (output) Optional pointer to next char after parse completes\n 156:\t *\n 157:\t *\tParses a string into a number. The number stored at @ptr is\n 158:\t *\tpotentially suffixed with K, M, G, T, P, E.\n 159:\t *\n 160:\t *\tReturn: The value as recognized by simple_strtoull() multiplied\n 161:\t *\tby the value as specified by suffix, if any.\n 162:\t */\n 163:\t\n 164:\tunsigned long long memparse(const char *ptr, char **retptr)\n"
} |
| 45/3 | 2026/11/03 12:36 | llm | kmsan-evaluator |
1mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 82792 output: 487 thoughts: 2463 Thoughts: **Analysis of the Hierarchical Per-CPU Counter Patch** Okay, let's break this down. First, I see that `ints[0]` holds the count of parsed integers, and the subsequent `ints` array stores the parsed values. In `parse_arity`, everything looks good: the `get_options` call is safe, and the array access `ints[0]` is checked against `MAX_NR_LEVELS`, ensuring bounds. The `config->nr_levels` and subsequent `fan_out` assignments are fully initialized. Now, into the `percpu_counter_tree_add_slowpath` function. The code initializes several variables: `level_items`, `nr_levels`, `stride`, etc. The pointers all point to allocated, zeroed items. There's a `READ_ONCE` on `counter->approx_sum` and the pointer arithmetic calculates `item` correctly, accessing the correct elements using `stride`. The loop iterates through the levels, checks for carry propagation (using `percpu_counter_tree_item_carry`), and updates the atomic counter. Crucially, the bitwise operations and shifts are all consistent and, based on the `kzalloc` or `kmem_cache_zalloc` allocations, we know the items are zeroed. Moving on to `percpu_counter_tree_precise_sum_unbiased`. It uses `level_items` and, there was a minor concern but `level_items` *is* fully initialized before its read in the loop, so there is no undefined behavior here. The code reads the levels in reverse order and sums the items. The `READ_ONCE` on the `level0` and `approx_sum` and the atomic read on the atomic bias are correct. The level iteration is handled carefully, and the code reads each level before the level below it, with `smp_rmb`. Also, all the counters and items are zeroed. Then there's `percpu_counter_tree_approximate_min_max_range`, which just does some basic arithmetic, writing to outputs. No issues there. Now, let's think about potential security or memory safety vulnerabilities. I don't see any of the classic KMSAN concerns: 1. **No user-space data exposure:** The code doesn't modify kernel structures that are sent to user space directly. The values are exposed to the user as formatted strings in procfs or via tracepoints. No raw structs or buffers are exposed directly. 2. **No conditional uninit reads:** There's no branching or logic that depends on potentially uninitialized variables. All structures are properly initialized during allocation. 3. **No uninitialized allocations:** Allocations are using `kzalloc` or `kmem_cache_zalloc`, ensuring zero-initialization. Per-CPU allocations are inherently zeroed. Lazy counter initialization *explicitly* zeroes. 4. **No bounds check issues:** I haven't seen anything that alters bounds checks or lengths in a way that would allow reading uninitialized memory. So, what kinds of bugs *could* exist? I can see: * Synchronization/memory ordering bugs. * Deadlocks with `rss_stat_mutex` or `task_lock`. * Use-after-free or leaks during destruction. * Arithmetic or logical errors in the counter logic. * Out-of-bounds accesses. But, critically, these are bugs that are addressed by standard mechanisms: KASAN (for OOB), LOCKDEP (for locking issues), and potentially KMSAN *if* there was an uninitialized read *exposed*. There is no uninitialized read exposed to the userspace in these patches. KASAN is sufficient. Therefore, `NeedsKMSAN` should be `false`. Now, let's call `set-results`. |
| 46/3 | 2026/11/03 12:38 | tool | set-results |
0mArgs:{
"KMSANReasoning": "The patch series introduces Hierarchical Per-CPU Counters (HPCC, implemented via `struct percpu_counter_tree`) and replaces `struct percpu_counter` with HPCC for memory management RSS statistics (`mm-\u003erss_stat`).\n\nKMSAN vs KASAN Applicability Analysis:\n1. Memory Allocation and Initialization:\n - Dynamic allocations for tree items use `kzalloc()` or `kmem_cache_zalloc()`, ensuring full zero-initialization of tree buffers.\n - Per-CPU allocations (`__alloc_percpu_gfp()`) in Linux are zeroed upon allocation.\n - Struct fields for `percpu_counter_tree` are systematically initialized in `__percpu_counter_tree_init()` (and lazy initialization explicitly sets pointers to NULL and biases to 0).\n - `struct mm_struct` additions (`rss_stat`, `rss_owner`, `rss_stat_mutex`) are initialized in `mm_init()` via `memset` and mutex initialization.\n - No complex data structures are left uninitialized or partially initialized.\n\n2. User-space Boundary and Info-leaks:\n - No raw kernel structures or buffers with potential padding bytes are copied to user space (`copy_to_user`, ioctl, netlink, BPF, etc.).\n - The RSS values derived from the tree counters are plain numerical values formatted into strings (e.g., in `/proc/\u003cpid\u003e/stat` via `seq_put_decimal_ull()`) or logged in trace events.\n\n3. Bug Profiles:\n - The primary risks of this patch series involve concurrency, memory ordering (store-release/load-acquire barriers between levels and during activation), synchronization (`rss_stat_mutex`), arithmetic carry logic, or potential out-of-bounds indexing in tree item strides.\n - Out-of-bounds indexing and use-after-free bugs are detected by KASAN. Locking deadlocks are caught by LOCKDEP. KMSAN is specifically designed to detect reads of uninitialized memory and information leaks, neither of which are risks introduced by this patch.\n\nTherefore, a dedicated KMSAN fuzzing session is not warranted.",
"NeedsKMSAN": false
}Results: {
"KMSANReasoning": "The patch series introduces Hierarchical Per-CPU Counters (HPCC, implemented via `struct percpu_counter_tree`) and replaces `struct percpu_counter` with HPCC for memory management RSS statistics (`mm-\u003erss_stat`).\n\nKMSAN vs KASAN Applicability Analysis:\n1. Memory Allocation and Initialization:\n - Dynamic allocations for tree items use `kzalloc()` or `kmem_cache_zalloc()`, ensuring full zero-initialization of tree buffers.\n - Per-CPU allocations (`__alloc_percpu_gfp()`) in Linux are zeroed upon allocation.\n - Struct fields for `percpu_counter_tree` are systematically initialized in `__percpu_counter_tree_init()` (and lazy initialization explicitly sets pointers to NULL and biases to 0).\n - `struct mm_struct` additions (`rss_stat`, `rss_owner`, `rss_stat_mutex`) are initialized in `mm_init()` via `memset` and mutex initialization.\n - No complex data structures are left uninitialized or partially initialized.\n\n2. User-space Boundary and Info-leaks:\n - No raw kernel structures or buffers with potential padding bytes are copied to user space (`copy_to_user`, ioctl, netlink, BPF, etc.).\n - The RSS values derived from the tree counters are plain numerical values formatted into strings (e.g., in `/proc/\u003cpid\u003e/stat` via `seq_put_decimal_ull()`) or logged in trace events.\n\n3. Bug Profiles:\n - The primary risks of this patch series involve concurrency, memory ordering (store-release/load-acquire barriers between levels and during activation), synchronization (`rss_stat_mutex`), arithmetic carry logic, or potential out-of-bounds indexing in tree item strides.\n - Out-of-bounds indexing and use-after-free bugs are detected by KASAN. Locking deadlocks are caught by LOCKDEP. KMSAN is specifically designed to detect reads of uninitialized memory and information leaks, neither of which are risks introduced by this patch.\n\nTherefore, a dedicated KMSAN fuzzing session is not warranted.",
"NeedsKMSAN": false
} |
| Total Calls | Total Tokens | Avg Tokens | Total Duration (Seconds) | Avg Duration (Seconds) |
|---|
| Total Calls | Total Duration (Seconds) | Avg Duration (Seconds) |
|---|