| Seq | Timestamp | Type | Name | Duration |
|---|---|---|---|---|
| 0/0 | 2026/09/06 04:16 | flow | patch-triage |
0mArgs:null Results: {
"EnableConfigs": [
"SMP"
],
"FocusSymbols": [
"percpu_counter_tree_add",
"percpu_counter_tree_approximate_compare",
"percpu_counter_tree_approximate_compare_value",
"percpu_counter_tree_init_many",
"percpu_counter_tree_precise_compare",
"percpu_counter_tree_precise_compare_value",
"percpu_counter_tree_precise_sum",
"percpu_counter_tree_set"
],
"KMSANReasoning": "The patch series introduces Hierarchical Per-CPU Counters (HPCC) and transitions mm_struct RSS accounting (rss_stat) to use HPCC instead of standard percpu counters.\n\nEvaluation of uninitialized memory risk:\n1. All allocated structures and tree items are fully zero-initialized before use:\n - In `percpu_counter_tree_init_many()`, `items` are explicitly zeroed via `memset()`, and `level0` per-CPU memory is allocated via `__alloc_percpu_gfp()` which returns zero-filled memory. All struct fields of `struct percpu_counter_tree` are explicitly initialized.\n - For `init_mm`, the statically allocated flexible array containing tree items is zero-initialized via `MM_STRUCT_FLEXIBLE_ARRAY_INIT`.\n - In `mm_init()`, RSS counters are initialized via `percpu_counter_tree_init_many()` with `get_rss_stat_items(mm)`.\n2. No kernel structures or buffers with uninitialized padding or fields are exposed or copied to user space (counter values are read as scalar `long` / `unsigned long` values and formatted into strings or proc/sysfs seq_files).\n3. No branching or calculations depend on uninitialized variables or memory.\n4. Any potential issues in this patch series (such as indexing errors in the tree hierarchy, arithmetic/carry calculation bugs, race conditions, or out-of-bounds accesses) fall squarely under KASAN, LOCKDEP, and KUnit testing rather than uninitialized memory detectors.\n\nTherefore, dedicated KMSAN fuzzing is not required (NeedsKMSAN=false).",
"NeedsKMSAN": false,
"Reasoning": "The patch introduces Hierarchical Per-CPU Counters (HPCC) and converts core mm RSS counters (struct mm_struct -\u003e rss_stat) to use HPCC. This modifies core memory management accounting, process creation/destruction, and introduces new carry propagation and comparison algorithms in reachable kernel space.",
"WorthFuzzing": true
} |
| 1/1 | 2026/09/06 04:16 | action | read-patch-diff |
0mArgs:null Results: {
"PatchDiff": "commit dde82b75d01b1954902ba04c172efd2ae7beff50\nAuthor: syz-cluster \u003ctriage@syzkaller.com\u003e\nDate: Sun Sep 6 04:16:19 2026 +0000\n\n syz-cluster: applied patch under review\n\ndiff --git a/Documentation/core-api/percpu-counter-tree.rst b/Documentation/core-api/percpu-counter-tree.rst\nnew file mode 100644\nindex 0000000000000..878cbd85304cf\n--- /dev/null\n+++ b/Documentation/core-api/percpu-counter-tree.rst\n@@ -0,0 +1,82 @@\n+========================================\n+The Hierarchical Per-CPU Counters (HPCC)\n+========================================\n+\n+:Author: Mathieu Desnoyers\n+\n+Introduction\n+============\n+\n+Counters come in many varieties, each with their own trade offs:\n+\n+ * A global atomic counter provides a fast read access to the current\n+ sum, at the expense of cache-line bouncing on updates. This leads to\n+ poor performance of frequent updates from various cores on large SMP\n+ systems.\n+\n+ * A per-cpu split counter provides fast updates to per-cpu counters,\n+ at the expense of a slower aggregation (sum). The sum operation needs\n+ to iterate over all per-cpu counters to calculate the current total.\n+\n+The hierarchical per-cpu counters attempt to provide the best of both\n+worlds (fast updates, and fast sum) by relaxing requirements on the sum\n+accuracy. It allows quickly querying an approximated sum value, along\n+with the possible min/max ranges of the associated precise sum. The\n+exact precise sum can still be calculated with an iteration on all\n+per-cpu counter, but the availability of an approximated sum value with\n+possible precise sum min/max ranges allows eliminating candidates which\n+are certainly outside of a known target range without the overhead of\n+precise sums.\n+\n+Overview\n+========\n+\n+The herarchical per-cpu counters are organized as a tree with the tree\n+root at the bottom (last level) and the first level of the tree\n+consisting of per-cpu counters.\n+\n+The intermediate tree levels contain carry propagation counters. When\n+reaching a threshold (batch size), the carry is propagated down the\n+tree.\n+\n+This allows reading an approximated value at the root, which has a\n+bounded accuracy (minimum/maximum possible precise sum range) determined\n+by the tree topology.\n+\n+The bounded accuracy is a property of the settled tree. While updates are\n+in flight the transient error can exceed it, bounded by the number of\n+updaters and their increment size.\n+\n+Use Cases\n+=========\n+\n+Use cases HPCC is meant to handle invove tracking resources which are\n+used across many CPUs to quickly sum as feedback for decision making to\n+apply throttling, quota limits, sort tasks, and perform memory or task\n+migration decisions. When considering approximated sums within the\n+accuracy range of the decision threshold, the user can either:\n+\n+ * Be conservative and fast: Consider that the sum has reached the\n+ limit as soon as the given limit is within the approximation range.\n+\n+ * Be aggressive and fast: Consider that the sum is over the\n+ limit only when the approximation range is over the given limit.\n+\n+ * Be precise and slow: Do a precise comparison with the limit, which\n+ requires a precise sum when the limit is within the approximated\n+ range.\n+\n+One use-case for these hierarchical counters is to implement a two-pass\n+algorithm to speed up sorting picking a maximum/minimunm sum value from\n+a set. A first pass compares the approximated values, and then a second\n+pass only needs the precise sum for counter trees which are within the\n+possible precise sum range of the counter tree chosen by the first pass.\n+\n+Because the comparison is on the wrapped difference, it is not transitive:\n+ordering is undefined over a set of counters spanning more than LONG_MAX.\n+\n+Functions and structures\n+========================\n+\n+.. kernel-doc:: include/linux/percpu_counter_tree.h\n+.. kernel-doc:: lib/percpu_counter_tree.c\ndiff --git a/include/linux/mm.h b/include/linux/mm.h\nindex dd09c438fa23e..ffbcd18c4ecb3 100644\n--- a/include/linux/mm.h\n+++ b/include/linux/mm.h\n@@ -3447,38 +3447,48 @@ static inline bool get_user_page_fast_only(unsigned long addr,\n {\n \treturn get_user_pages_fast_only(addr, 1, gup_flags, pagep) == 1;\n }\n+\n+static inline struct percpu_counter_tree_level_item *get_rss_stat_items(struct mm_struct *mm)\n+{\n+\tunsigned long ptr = (unsigned long)mm;\n+\n+\tptr += offsetof(struct mm_struct, flexible_array);\n+\tptr += get_rss_stat_items_offset();\n+\treturn (struct percpu_counter_tree_level_item *)ptr;\n+}\n+\n /*\n * per-process(per-mm_struct) statistics.\n */\n static inline unsigned long get_mm_counter(struct mm_struct *mm, int member)\n {\n-\treturn percpu_counter_read_positive(\u0026mm-\u003erss_stat[member]);\n+\treturn percpu_counter_tree_approximate_sum_positive(\u0026mm-\u003erss_stat[member]);\n }\n \n static inline unsigned long get_mm_counter_sum(struct mm_struct *mm, int member)\n {\n-\treturn percpu_counter_sum_positive(\u0026mm-\u003erss_stat[member]);\n+\treturn percpu_counter_tree_precise_sum_positive(\u0026mm-\u003erss_stat[member]);\n }\n \n void mm_trace_rss_stat(struct mm_struct *mm, int member);\n \n static inline void add_mm_counter(struct mm_struct *mm, int member, long value)\n {\n-\tpercpu_counter_add(\u0026mm-\u003erss_stat[member], value);\n+\tpercpu_counter_tree_add(\u0026mm-\u003erss_stat[member], value);\n \n \tmm_trace_rss_stat(mm, member);\n }\n \n static inline void inc_mm_counter(struct mm_struct *mm, int member)\n {\n-\tpercpu_counter_inc(\u0026mm-\u003erss_stat[member]);\n+\tpercpu_counter_tree_add(\u0026mm-\u003erss_stat[member], 1);\n \n \tmm_trace_rss_stat(mm, member);\n }\n \n static inline void dec_mm_counter(struct mm_struct *mm, int member)\n {\n-\tpercpu_counter_dec(\u0026mm-\u003erss_stat[member]);\n+\tpercpu_counter_tree_add(\u0026mm-\u003erss_stat[member], -1);\n \n \tmm_trace_rss_stat(mm, member);\n }\ndiff --git a/include/linux/mm_types.h b/include/linux/mm_types.h\nindex 6d815f6440c94..b26160b85f93a 100644\n--- a/include/linux/mm_types.h\n+++ b/include/linux/mm_types.h\n@@ -18,7 +18,7 @@\n #include \u003clinux/page-flags-layout.h\u003e\n #include \u003clinux/workqueue.h\u003e\n #include \u003clinux/seqlock.h\u003e\n-#include \u003clinux/percpu_counter.h\u003e\n+#include \u003clinux/percpu_counter_tree.h\u003e\n #include \u003clinux/types.h\u003e\n #include \u003clinux/futex_types.h\u003e\n #include \u003clinux/rseq_types.h\u003e\n@@ -1170,6 +1170,19 @@ typedef struct {\n \tDECLARE_BITMAP(__mm_flags, NUM_MM_FLAG_BITS);\n } __private mm_flags_t;\n \n+/*\n+ * The alignment of the mm_struct flexible array is based on the largest\n+ * alignment of its content:\n+ * __alignof__(struct percpu_counter_tree_level_item) provides a\n+ * cacheline aligned alignment on SMP systems, else alignment on\n+ * unsigned long on UP systems.\n+ */\n+#ifdef CONFIG_SMP\n+# define __mm_struct_flexible_array_aligned\t__aligned(__alignof__(struct percpu_counter_tree_level_item))\n+#else\n+# define __mm_struct_flexible_array_aligned\t__aligned(__alignof__(unsigned long))\n+#endif\n+\n struct kioctx_table;\n struct iommu_mm_data;\n struct mm_struct {\n@@ -1312,7 +1325,7 @@ struct mm_struct {\n \t\tunsigned long saved_e_flags;\n #endif\n \n-\t\tstruct percpu_counter rss_stat[NR_MM_COUNTERS];\n+\t\tstruct percpu_counter_tree rss_stat[NR_MM_COUNTERS];\n \n \t\tstruct linux_binfmt *binfmt;\n \n@@ -1422,10 +1435,14 @@ struct mm_struct {\n \t} __randomize_layout;\n \n \t/*\n-\t * The mm_cpumask needs to be at the end of mm_struct, because it\n-\t * is dynamically sized based on nr_cpu_ids.\n+\t * The mm_cpumask, mm_cid masks, and rss hierarchical counter\n+\t * items need to be at the end of mm_struct, because they are\n+\t * dynamically sized based on nr_cpu_ids.\n+\t * The HPCC counter tree items are placed last and aligned\n+\t * with PERCPU_COUNTER_TREE_ITEMS_ALIGN to satisfy their\n+\t * cacheline alignment requirement.\n \t */\n-\tchar flexible_array[] __aligned(__alignof__(unsigned long));\n+\tchar flexible_array[] __mm_struct_flexible_array_aligned;\n };\n \n /* Copy value to the first system word of mm flags, non-atomically. */\n@@ -1462,18 +1479,12 @@ static inline void __mm_flags_set_mask_bits_word(struct mm_struct *mm,\n \t\t\t MT_FLAGS_USE_RCU)\n extern struct mm_struct init_mm;\n \n-#define MM_STRUCT_FLEXIBLE_ARRAY_INIT\t\t\t\t\\\n-{\t\t\t\t\t\t\t\t\\\n-\t[0 ... sizeof(cpumask_t) + MM_CID_STATIC_SIZE - 1] = 0\t\\\n-}\n-\n-/* Pointer magic because the dynamic array size confuses some compilers. */\n-static inline void mm_init_cpumask(struct mm_struct *mm)\n-{\n-\tunsigned long cpu_bitmap = (unsigned long)mm;\n-\n-\tcpu_bitmap += offsetof(struct mm_struct, flexible_array);\n-\tcpumask_clear((struct cpumask *)cpu_bitmap);\n+#define MM_STRUCT_FLEXIBLE_ARRAY_INIT\t\t\t\t\t\t\\\n+{\t\t\t\t\t\t\t\t\t\t\\\n+\t[0 ... PERCPU_COUNTER_TREE_ITEMS_ALIGN(\t\t\t\t\t\\\n+\t\t\tsizeof(cpumask_t) + MM_CID_STATIC_SIZE)\t\t\t\\\n+\t + (PERCPU_COUNTER_TREE_ITEMS_STATIC_SIZE * NR_MM_COUNTERS)\t\\\n+\t - 1] = 0\t\t\t\t\t\t\t\t\\\n }\n \n /* Future-safe accessor for struct mm_struct's cpu_vm_mask. */\n@@ -1482,6 +1493,11 @@ static inline cpumask_t *mm_cpumask(struct mm_struct *mm)\n \treturn (struct cpumask *)\u0026mm-\u003eflexible_array;\n }\n \n+static inline void mm_init_cpumask(struct mm_struct *mm)\n+{\n+\tcpumask_clear((struct cpumask *)mm_cpumask(mm));\n+}\n+\n #ifdef CONFIG_LRU_GEN\n \n struct lru_gen_mm_list {\n@@ -1653,6 +1669,21 @@ static inline void mm_destroy_sched(struct mm_struct *mm) { }\n \n #endif /* CONFIG_SCHED_CACHE */\n \n+static inline size_t get_rss_stat_items_size(void)\n+{\n+\treturn percpu_counter_tree_items_size() * NR_MM_COUNTERS;\n+}\n+\n+/*\n+ * Return the offset of the RSS stat HPCC items within the mm_struct\n+ * flexible array. The items are placed after the cpumask and mm_cid,\n+ * aligned to the cacheline boundary required by the tree level items.\n+ */\n+static inline size_t get_rss_stat_items_offset(void)\n+{\n+\treturn PERCPU_COUNTER_TREE_ITEMS_ALIGN(cpumask_size() + mm_cid_size());\n+}\n+\n struct mmu_gather;\n extern void tlb_gather_mmu(struct mmu_gather *tlb, struct mm_struct *mm);\n extern void tlb_gather_mmu_fullmm(struct mmu_gather *tlb, struct mm_struct *mm);\ndiff --git a/include/linux/percpu_counter_tree.h b/include/linux/percpu_counter_tree.h\nnew file mode 100644\nindex 0000000000000..d593806b4ab26\n--- /dev/null\n+++ b/include/linux/percpu_counter_tree.h\n@@ -0,0 +1,391 @@\n+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */\n+/* SPDX-FileCopyrightText: 2025 Mathieu Desnoyers \u003cmathieu.desnoyers@efficios.com\u003e */\n+\n+#ifndef _PERCPU_COUNTER_TREE_H\n+#define _PERCPU_COUNTER_TREE_H\n+\n+#include \u003clinux/preempt.h\u003e\n+#include \u003clinux/atomic.h\u003e\n+#include \u003clinux/percpu.h\u003e\n+\n+#ifdef CONFIG_SMP\n+\n+#if NR_CPUS == (1U \u003c\u003c 0)\n+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS\t0\n+#elif NR_CPUS \u003c= (1U \u003c\u003c 1)\n+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS\t1\n+#elif NR_CPUS \u003c= (1U \u003c\u003c 2)\n+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS\t3\n+#elif NR_CPUS \u003c= (1U \u003c\u003c 3)\n+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS\t7\n+#elif NR_CPUS \u003c= (1U \u003c\u003c 4)\n+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS\t7\n+#elif NR_CPUS \u003c= (1U \u003c\u003c 5)\n+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS\t11\n+#elif NR_CPUS \u003c= (1U \u003c\u003c 6)\n+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS\t21\n+#elif NR_CPUS \u003c= (1U \u003c\u003c 7)\n+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS\t21\n+#elif NR_CPUS \u003c= (1U \u003c\u003c 8)\n+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS\t37\n+#elif NR_CPUS \u003c= (1U \u003c\u003c 9)\n+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS\t73\n+#elif NR_CPUS \u003c= (1U \u003c\u003c 10)\n+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS\t149\n+#elif NR_CPUS \u003c= (1U \u003c\u003c 11)\n+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS\t293\n+#elif NR_CPUS \u003c= (1U \u003c\u003c 12)\n+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS\t585\n+#elif NR_CPUS \u003c= (1U \u003c\u003c 13)\n+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS\t1173\n+#elif NR_CPUS \u003c= (1U \u003c\u003c 14)\n+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS\t2341\n+#elif NR_CPUS \u003c= (1U \u003c\u003c 15)\n+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS\t4681\n+#elif NR_CPUS \u003c= (1U \u003c\u003c 16)\n+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS\t4681\n+#elif NR_CPUS \u003c= (1U \u003c\u003c 17)\n+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS\t8777\n+#elif NR_CPUS \u003c= (1U \u003c\u003c 18)\n+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS\t17481\n+#elif NR_CPUS \u003c= (1U \u003c\u003c 19)\n+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS\t34953\n+#elif NR_CPUS \u003c= (1U \u003c\u003c 20)\n+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS\t69905\n+#else\n+# error \"Unsupported number of CPUs.\"\n+#endif\n+\n+struct percpu_counter_tree_level_item {\n+\tatomic_long_t count;\t\t/*\n+\t\t\t\t\t * Count the number of carry for this tree item.\n+\t\t\t\t\t * The carry counter is kept at the order of the\n+\t\t\t\t\t * carry accounted for at this tree level.\n+\t\t\t\t\t */\n+} ____cacheline_aligned_in_smp;\n+\n+#define PERCPU_COUNTER_TREE_ITEMS_STATIC_SIZE\t\\\n+\t(PERCPU_COUNTER_TREE_STATIC_NR_ITEMS * sizeof(struct percpu_counter_tree_level_item))\n+#define PERCPU_COUNTER_TREE_ITEMS_ALIGN(offset)\t\\\n+\tALIGN((offset), __alignof__(struct percpu_counter_tree_level_item))\n+\n+struct percpu_counter_tree {\n+\t/* Fast-path fields. */\n+\tunsigned long __percpu *level0;\t/* Pointer to per-CPU split counters (tree level 0). */\n+\tunsigned long level0_bit_mask;\t/* Bit mask to apply to detect carry propagation from tree level 0. */\n+\tunion {\n+\t\tunsigned long *i;\t/* Approximate sum for single-CPU topology. */\n+\t\tatomic_long_t *a;\t/* Approximate sum for SMP topology. */\n+\t} approx_sum;\n+\tlong bias;\t\t\t/* Bias to apply to counter precise and approximate values. */\n+\n+\t/* Slow-path fields. */\n+\tstruct percpu_counter_tree_level_item *items;\t/* Array of tree items for levels 1 to N. */\n+\tunsigned long batch_size;\t/*\n+\t\t\t\t\t * The batch size is the increment step at level 0 which\n+\t\t\t\t\t * triggers a carry propagation. The batch size is required\n+\t\t\t\t\t * to be greater than 1, and a power of 2.\n+\t\t\t\t\t */\n+\t/*\n+\t * The tree approximate sum is guaranteed to be within this accuracy range:\n+\t * (precise_sum - approx_accuracy_range.under) \u003c= approx_sum \u003c= (precise_sum + approx_accuracy_range.over).\n+\t * This accuracy is derived from the hardware topology and the tree batch_size.\n+\t * The \"under\" accuracy is larger than the \"over\" accuracy because the negative range of a\n+\t * two's complement signed integer is one unit larger than the positive range. This delta\n+\t * is summed for each tree item, which leads to a significantly larger \"under\" accuracy range\n+\t * compared to the \"over\" accuracy range. The accuracy range is a bound on a settled counter:\n+\t * one with no update in progress.\n+\t */\n+\tstruct {\n+\t\tunsigned long under;\n+\t\tunsigned long over;\n+\t} approx_accuracy_range;\n+};\n+\n+extern size_t __percpu_counter_tree_items_size __ro_after_init;\n+\n+/*\n+ * percpu_counter_tree_items_size - Query the size required for counter tree items.\n+ *\n+ * Query the size of the memory area required to hold the counter tree\n+ * items. This depends on the hardware topology and is invariant after\n+ * boot.\n+ *\n+ * Return: Size required to hold tree items.\n+ */\n+static inline\n+size_t percpu_counter_tree_items_size(void)\n+{\n+\tWARN_ON_ONCE(__percpu_counter_tree_items_size == SIZE_MAX);\n+\treturn __percpu_counter_tree_items_size;\n+}\n+\n+int percpu_counter_tree_init_many(struct percpu_counter_tree *counters, struct percpu_counter_tree_level_item *items,\n+\t\t\t\t unsigned int nr_counters, unsigned long batch_size, gfp_t gfp_flags);\n+int percpu_counter_tree_init(struct percpu_counter_tree *counter, struct percpu_counter_tree_level_item *items,\n+\t\t\t unsigned long batch_size, gfp_t gfp_flags);\n+void percpu_counter_tree_destroy_many(struct percpu_counter_tree *counter, unsigned int nr_counters);\n+void percpu_counter_tree_destroy(struct percpu_counter_tree *counter);\n+void percpu_counter_tree_add(struct percpu_counter_tree *counter, long inc);\n+long percpu_counter_tree_precise_sum(struct percpu_counter_tree *counter);\n+int percpu_counter_tree_approximate_compare(struct percpu_counter_tree *a, struct percpu_counter_tree *b);\n+int percpu_counter_tree_approximate_compare_value(struct percpu_counter_tree *counter, long v);\n+int percpu_counter_tree_precise_compare(struct percpu_counter_tree *a, struct percpu_counter_tree *b);\n+int percpu_counter_tree_precise_compare_value(struct percpu_counter_tree *counter, long v);\n+void percpu_counter_tree_set(struct percpu_counter_tree *counter, long v);\n+int percpu_counter_tree_subsystem_init(void);\n+\n+/**\n+ * percpu_counter_tree_approximate_sum() - Return approximate counter sum.\n+ * @counter: The counter to sum.\n+ *\n+ * Querying the approximate sum is fast, but it is only accurate within\n+ * the bounds delimited by percpu_counter_tree_approximate_accuracy_range().\n+ * This is meant to be used when speed is preferred over accuracy.\n+ *\n+ * Return: The current approximate counter sum.\n+ */\n+static inline\n+long percpu_counter_tree_approximate_sum(struct percpu_counter_tree *counter)\n+{\n+\tunsigned long v;\n+\n+\tif (!counter-\u003elevel0_bit_mask)\n+\t\tv = READ_ONCE(*counter-\u003eapprox_sum.i);\n+\telse\n+\t\tv = atomic_long_read(counter-\u003eapprox_sum.a);\n+\treturn (long) (v + (unsigned long)READ_ONCE(counter-\u003ebias));\n+}\n+\n+/**\n+ * percpu_counter_tree_approximate_accuracy_range - Query the accuracy range for a counter tree.\n+ * @counter: Counter to query.\n+ * @under: Pointer to a variable to be incremented of the approximation\n+ * accuracy range below the precise sum.\n+ * @over: Pointer to a variable to be incremented of the approximation\n+ * accuracy range above the precise sum.\n+ *\n+ * Query the accuracy range limits for the counter.\n+ * Because of two's complement binary representation, the \"under\" range is typically\n+ * slightly larger than the \"over\" range.\n+ * Those values are derived from the hardware topology and the counter tree batch size.\n+ * They are invariant for a given counter tree.\n+ * The bounds describe a settled counter: one with no update in progress.\n+ * An update in flight between its per-CPU increment and its carry propagation is not covered.\n+ * Using this function should not be typically required, see the following functions instead:\n+ * * percpu_counter_tree_approximate_compare(),\n+ * * percpu_counter_tree_approximate_compare_value(),\n+ * * percpu_counter_tree_precise_compare(),\n+ * * percpu_counter_tree_precise_compare_value().\n+ */\n+static inline\n+void percpu_counter_tree_approximate_accuracy_range(struct percpu_counter_tree *counter,\n+\t\t\t\t\t\t unsigned long *under, unsigned long *over)\n+{\n+\t*under += counter-\u003eapprox_accuracy_range.under;\n+\t*over += counter-\u003eapprox_accuracy_range.over;\n+}\n+\n+#else\t/* !CONFIG_SMP */\n+\n+#define PERCPU_COUNTER_TREE_ITEMS_STATIC_SIZE\t0\n+#define PERCPU_COUNTER_TREE_ITEMS_ALIGN(offset)\t(offset)\n+\n+struct percpu_counter_tree_level_item;\n+\n+struct percpu_counter_tree {\n+\tatomic_long_t count;\n+};\n+\n+static inline\n+size_t percpu_counter_tree_items_size(void)\n+{\n+\treturn 0;\n+}\n+\n+static inline\n+int percpu_counter_tree_init_many(struct percpu_counter_tree *counters, struct percpu_counter_tree_level_item *items,\n+\t\t\t\t unsigned int nr_counters, unsigned long batch_size, gfp_t gfp_flags)\n+{\n+\tfor (unsigned int i = 0; i \u003c nr_counters; i++)\n+\t\tatomic_long_set(\u0026counters[i].count, 0);\n+\treturn 0;\n+}\n+\n+static inline\n+int percpu_counter_tree_init(struct percpu_counter_tree *counter, struct percpu_counter_tree_level_item *items,\n+\t\t\t unsigned long batch_size, gfp_t gfp_flags)\n+{\n+\treturn percpu_counter_tree_init_many(counter, items, 1, batch_size, gfp_flags);\n+}\n+\n+static inline\n+void percpu_counter_tree_destroy_many(struct percpu_counter_tree *counter, unsigned int nr_counters)\n+{\n+}\n+\n+static inline\n+void percpu_counter_tree_destroy(struct percpu_counter_tree *counter)\n+{\n+}\n+\n+static inline\n+long percpu_counter_tree_precise_sum(struct percpu_counter_tree *counter)\n+{\n+\treturn atomic_long_read(\u0026counter-\u003ecount);\n+}\n+\n+static inline\n+int percpu_counter_tree_precise_compare(struct percpu_counter_tree *a, struct percpu_counter_tree *b)\n+{\n+\tlong count_a = percpu_counter_tree_precise_sum(a),\n+\t count_b = percpu_counter_tree_precise_sum(b);\n+\n+\tif (count_a == count_b)\n+\t\treturn 0;\n+\tif (count_a \u003c count_b)\n+\t\treturn -1;\n+\treturn 1;\n+}\n+\n+static inline\n+int percpu_counter_tree_precise_compare_value(struct percpu_counter_tree *counter, long v)\n+{\n+\tlong count = percpu_counter_tree_precise_sum(counter);\n+\n+\tif (count == v)\n+\t\treturn 0;\n+\tif (count \u003c v)\n+\t\treturn -1;\n+\treturn 1;\n+}\n+\n+static inline\n+int percpu_counter_tree_approximate_compare(struct percpu_counter_tree *a, struct percpu_counter_tree *b)\n+{\n+\treturn percpu_counter_tree_precise_compare(a, b);\n+}\n+\n+static inline\n+int percpu_counter_tree_approximate_compare_value(struct percpu_counter_tree *counter, long v)\n+{\n+\treturn percpu_counter_tree_precise_compare_value(counter, v);\n+}\n+\n+static inline\n+void percpu_counter_tree_set(struct percpu_counter_tree *counter, long v)\n+{\n+\tatomic_long_set(\u0026counter-\u003ecount, v);\n+}\n+\n+static inline\n+void percpu_counter_tree_approximate_accuracy_range(struct percpu_counter_tree *counter,\n+\t\t\t\t\t\t unsigned long *under, unsigned long *over)\n+{\n+}\n+\n+static inline\n+void percpu_counter_tree_add(struct percpu_counter_tree *counter, long inc)\n+{\n+\tatomic_long_add(inc, \u0026counter-\u003ecount);\n+}\n+\n+static inline\n+long percpu_counter_tree_approximate_sum(struct percpu_counter_tree *counter)\n+{\n+\treturn percpu_counter_tree_precise_sum(counter);\n+}\n+\n+static inline\n+int percpu_counter_tree_subsystem_init(void)\n+{\n+\treturn 0;\n+}\n+\n+#endif\t/* CONFIG_SMP */\n+\n+/**\n+ * percpu_counter_tree_approximate_sum_positive() - Return a positive approximate counter sum.\n+ * @counter: The counter to sum.\n+ *\n+ * Return an approximate counter sum which is guaranteed to be greater\n+ * or equal to 0.\n+ *\n+ * Return: The current positive approximate counter sum.\n+ */\n+static inline\n+long percpu_counter_tree_approximate_sum_positive(struct percpu_counter_tree *counter)\n+{\n+\tlong v = percpu_counter_tree_approximate_sum(counter);\n+\treturn v \u003e 0 ? v : 0;\n+}\n+\n+/**\n+ * percpu_counter_tree_precise_sum_positive() - Return a positive precise counter sum.\n+ * @counter: The counter to sum.\n+ *\n+ * Return a precise counter sum which is guaranteed to be greater\n+ * or equal to 0.\n+ *\n+ * Return: The current positive precise counter sum.\n+ */\n+static inline\n+long percpu_counter_tree_precise_sum_positive(struct percpu_counter_tree *counter)\n+{\n+\tlong v = percpu_counter_tree_precise_sum(counter);\n+\treturn v \u003e 0 ? v : 0;\n+}\n+\n+/**\n+ * percpu_counter_tree_approximate_min_max_range() - Return the approximation min and max precise values.\n+ * @approx_sum: Approximated sum.\n+ * @under: Tree accuracy range (under).\n+ * @over: Tree accuracy range (over).\n+ * @precise_min: Minimum possible value for precise sum (output).\n+ * @precise_max: Maximum possible value for precise sum (output).\n+ *\n+ * Calculate the minimum and maximum precise values for a given approximation\n+ * and (under, over) accuracy range. The accuracy range is a bound on a settled\n+ * tree: one with no update in progress.\n+ *\n+ * The range of the approximation as a function of the precise sum is expressed as:\n+ *\n+ * approx_sum \u003e= precise_sum - approx_accuracy_range.under\n+ * approx_sum \u003c= precise_sum + approx_accuracy_range.over\n+ *\n+ * Therefore, the range of the precise sum as a function of the approximation is expressed as:\n+ *\n+ * precise_sum \u003c= approx_sum + approx_accuracy_range.under\n+ * precise_sum \u003e= approx_sum - approx_accuracy_range.over\n+ */\n+static inline\n+void percpu_counter_tree_approximate_min_max_range(long approx_sum, unsigned long under, unsigned long over,\n+\t\t\t\t\t\t long *precise_min, long *precise_max)\n+{\n+\t*precise_min = approx_sum - over;\n+\t*precise_max = approx_sum + under;\n+}\n+\n+/**\n+ * percpu_counter_tree_approximate_min_max() - Return the tree approximation, min and max possible precise values.\n+ * @counter: The counter to sum.\n+ * @approx_sum: Approximate sum (output).\n+ * @precise_min: Minimum possible value for precise sum (output).\n+ * @precise_max: Maximum possible value for precise sum (output).\n+ *\n+ * Return the approximate sum, minimum and maximum precise values for\n+ * a counter.\n+ */\n+static inline\n+void percpu_counter_tree_approximate_min_max(struct percpu_counter_tree *counter,\n+\t\t\t\t\t long *approx_sum, long *precise_min, long *precise_max)\n+{\n+\tunsigned long under = 0, over = 0;\n+\tlong v = percpu_counter_tree_approximate_sum(counter);\n+\n+\tpercpu_counter_tree_approximate_accuracy_range(counter, \u0026under, \u0026over);\n+\tpercpu_counter_tree_approximate_min_max_range(v, under, over, precise_min, precise_max);\n+\t*approx_sum = v;\n+}\n+\n+#endif /* _PERCPU_COUNTER_TREE_H */\ndiff --git a/include/trace/events/kmem.h b/include/trace/events/kmem.h\nindex cd7920c81f85b..290ccb9fd25dd 100644\n--- a/include/trace/events/kmem.h\n+++ b/include/trace/events/kmem.h\n@@ -448,7 +448,7 @@ TRACE_EVENT(rss_stat,\n \t\t */\n \t\t__entry-\u003ecurr = current-\u003emm == mm \u0026\u0026 !(current-\u003eflags \u0026 PF_KTHREAD);\n \t\t__entry-\u003emember = member;\n-\t\t__entry-\u003esize = (percpu_counter_sum_positive(\u0026mm-\u003erss_stat[member])\n+\t\t__entry-\u003esize = (percpu_counter_tree_approximate_sum_positive(\u0026mm-\u003erss_stat[member])\n \t\t\t\t\t\t\t \u003c\u003c PAGE_SHIFT);\n \t),\n \ndiff --git a/init/main.c b/init/main.c\nindex 2613d3f9b3ce9..845cb57e90965 100644\n--- a/init/main.c\n+++ b/init/main.c\n@@ -108,6 +108,7 @@\n #include \u003clinux/time_namespace.h\u003e\n #include \u003clinux/unaligned.h\u003e\n #include \u003clinux/vdso_datastore.h\u003e\n+#include \u003clinux/percpu_counter_tree.h\u003e\n #include \u003cnet/net_namespace.h\u003e\n \n #include \u003casm/io.h\u003e\n@@ -1010,6 +1011,7 @@ void start_kernel(void)\n \tsetup_boot_config();\n \tsetup_command_line(command_line);\n \tsetup_nr_cpu_ids();\n+\tpercpu_counter_tree_subsystem_init();\n \tsetup_per_cpu_areas();\n \tsmp_prepare_boot_cpu();\t/* arch-specific boot-cpu hooks */\n \tearly_numa_node_init();\ndiff --git a/kernel/fork.c b/kernel/fork.c\nindex 416758c8a3d43..db398b6c52886 100644\n--- a/kernel/fork.c\n+++ b/kernel/fork.c\n@@ -137,6 +137,11 @@\n */\n #define MAX_THREADS FUTEX_TID_MASK\n \n+/*\n+ * Batch size of rss stat approximation\n+ */\n+#define RSS_STAT_BATCH_SIZE\t32\n+\n /*\n * Protected counters by write_lock_irq(\u0026tasklist_lock)\n */\n@@ -633,14 +638,12 @@ static void check_mm(struct mm_struct *mm)\n \t\t\t \"Please make sure 'struct resident_page_types[]' is updated as well\");\n \n \tfor (i = 0; i \u003c NR_MM_COUNTERS; i++) {\n-\t\tlong x = percpu_counter_sum(\u0026mm-\u003erss_stat[i]);\n-\n-\t\tif (unlikely(x)) {\n+\t\tif (unlikely(percpu_counter_tree_precise_compare_value(\u0026mm-\u003erss_stat[i], 0) != 0))\n \t\t\tpr_alert(\"BUG: Bad rss-counter state mm:%p type:%s val:%ld Comm:%s Pid:%d\\n\",\n-\t\t\t\t mm, resident_page_types[i], x,\n+\t\t\t\t mm, resident_page_types[i],\n+\t\t\t\t percpu_counter_tree_precise_sum(\u0026mm-\u003erss_stat[i]),\n \t\t\t\t current-\u003ecomm,\n \t\t\t\t task_pid_nr(current));\n-\t\t}\n \t}\n \n \tif (mm_pgtables_bytes(mm))\n@@ -738,7 +741,7 @@ void __mmdrop(struct mm_struct *mm)\n \tcheck_mm(mm);\n \tmm_pasid_drop(mm);\n \tmm_destroy_cid(mm);\n-\tpercpu_counter_destroy_many(mm-\u003erss_stat, NR_MM_COUNTERS);\n+\tpercpu_counter_tree_destroy_many(mm-\u003erss_stat, NR_MM_COUNTERS);\n \n \tfree_mm(mm);\n }\n@@ -1144,8 +1147,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p)\n \tif (mm_alloc_sched(mm))\n \t\tgoto fail_sched;\n \n-\tif (percpu_counter_init_many(mm-\u003erss_stat, 0, GFP_KERNEL_ACCOUNT,\n-\t\t\t\t NR_MM_COUNTERS))\n+\tif (percpu_counter_tree_init_many(mm-\u003erss_stat, get_rss_stat_items(mm),\n+\t\t\t\t\t NR_MM_COUNTERS, RSS_STAT_BATCH_SIZE,\n+\t\t\t\t\t GFP_KERNEL_ACCOUNT))\n \t\tgoto fail_pcpu;\n \n \tlru_gen_init_mm(mm);\n@@ -3122,7 +3126,9 @@ void __init mm_cache_init(void)\n \t * dynamically sized based on the maximum CPU number this system\n \t * can have, taking hotplug into account (nr_cpu_ids).\n \t */\n-\tmm_size = sizeof(struct mm_struct) + cpumask_size() + mm_cid_size();\n+\tmm_size = sizeof(struct mm_struct) +\n+\t\tPERCPU_COUNTER_TREE_ITEMS_ALIGN(cpumask_size() + mm_cid_size()) +\n+\t\tget_rss_stat_items_size();\n \n \tmm_cachep = kmem_cache_create_usercopy(\"mm_struct\",\n \t\t\tmm_size, ARCH_MIN_MMSTRUCT_ALIGN,\ndiff --git a/lib/Kconfig b/lib/Kconfig\nindex 4e6b34c3346d5..c0524258b2058 100644\n--- a/lib/Kconfig\n+++ b/lib/Kconfig\n@@ -41,6 +41,18 @@ config PACKING_KUNIT_TEST\n \n \t When in doubt, say N.\n \n+config PERCPU_COUNTER_TREE_KUNIT_TEST\n+\ttristate \"KUnit tests for hierarchical per-cpu counters\" if !KUNIT_ALL_TESTS\n+\tdepends on KUNIT\n+\tdefault KUNIT_ALL_TESTS\n+\thelp\n+\t This builds Kunit tests for the hierarchical per-cpu counters.\n+\n+\t For more information on KUnit and unit tests in general,\n+\t please refer to the KUnit documentation in Documentation/dev-tools/kunit/.\n+\n+\t When in doubt, say N.\n+\n config BITREVERSE\n \ttristate\n \tselect GENERIC_BITREVERSE if !HAVE_ARCH_BITREVERSE\ndiff --git a/lib/Makefile b/lib/Makefile\nindex dfab958327c5c..f1d0cb831630b 100644\n--- a/lib/Makefile\n+++ b/lib/Makefile\n@@ -182,6 +182,7 @@ obj-$(CONFIG_TEXTSEARCH_KMP) += ts_kmp.o\n obj-$(CONFIG_TEXTSEARCH_BM) += ts_bm.o\n obj-$(CONFIG_TEXTSEARCH_FSM) += ts_fsm.o\n obj-$(CONFIG_SMP) += percpu_counter.o\n+obj-$(CONFIG_SMP) += percpu_counter_tree.o\n obj-$(CONFIG_AUDIT_GENERIC) += audit.o\n obj-$(CONFIG_AUDIT_COMPAT_GENERIC) += compat_audit.o\n \ndiff --git a/lib/percpu_counter_tree.c b/lib/percpu_counter_tree.c\nnew file mode 100644\nindex 0000000000000..26bfe183a882d\n--- /dev/null\n+++ b/lib/percpu_counter_tree.c\n@@ -0,0 +1,747 @@\n+// SPDX-License-Identifier: GPL-2.0+ OR MIT\n+// SPDX-FileCopyrightText: 2025 Mathieu Desnoyers \u003cmathieu.desnoyers@efficios.com\u003e\n+\n+/*\n+ * Split Counters With Tree Approximation Propagation\n+ *\n+ * * Propagation diagram when reaching batch size thresholds (± batch size):\n+ *\n+ * Example diagram for 8 CPUs:\n+ *\n+ * log2(8) = 3 levels\n+ *\n+ * At each level, each pair propagates its values to the next level when\n+ * reaching the batch size thresholds.\n+ *\n+ * Counters at levels 0, 1, 2 can be kept on a single byte ([-128 .. +127] range),\n+ * although it may be relevant to keep them on \"long\" counters for\n+ * simplicity. (complexity vs memory footprint tradeoff)\n+ *\n+ * Counter at level 3 can be kept on a \"long\" counter.\n+ *\n+ * Level 0: 0 1 2 3 4 5 6 7\n+ * | / | / | / | /\n+ * | / | / | / | /\n+ * | / | / | / | /\n+ * Level 1: 0 1 2 3\n+ * | / | /\n+ * | / | /\n+ * | / | /\n+ * Level 2: 0 1\n+ * | /\n+ * | /\n+ * | /\n+ * Level 3: 0\n+ *\n+ * * Approximation accuracy:\n+ *\n+ * BATCH(level N): Level N batch size.\n+ *\n+ * Example for BATCH(level 0) = 32.\n+ *\n+ * BATCH(level 0) = 32\n+ * BATCH(level 1) = 64\n+ * BATCH(level 2) = 128\n+ * BATCH(level N) = BATCH(level 0) * 2^N\n+ *\n+ * per-counter global\n+ * accuracy accuracy\n+ * Level 0: [ -32 .. +31] ±256 (8 * 32)\n+ * Level 1: [ -64 .. +63] ±256 (4 * 64)\n+ * Level 2: [-128 .. +127] ±256 (2 * 128)\n+ * Total: ------ ±768 (log2(nr_cpu_ids) * BATCH(level 0) * nr_cpu_ids)\n+ *\n+ * Note that the global accuracy can be calculated more precisely\n+ * by taking into account that the positive accuracy range is\n+ * 31 rather than 32.\n+ *\n+ * -----\n+ *\n+ * Approximate Sum Carry Propagation\n+ *\n+ * Let's define a number of counter bits for each level, e.g.:\n+ *\n+ * log2(BATCH(level 0)) = log2(32) = 5\n+ * Let's assume, for this example, a 32-bit architecture (sizeof(long) == 4).\n+ *\n+ * nr_bit value_mask range\n+ * Level 0: 5 bits v 0 .. +31\n+ * Level 1: 1 bit (v \u0026 ~((1UL \u003c\u003c 5) - 1)) 0 .. +63\n+ * Level 2: 1 bit (v \u0026 ~((1UL \u003c\u003c 6) - 1)) 0 .. +127\n+ * Level 3: 25 bits (v \u0026 ~((1UL \u003c\u003c 7) - 1)) 0 .. 2^32-1\n+ *\n+ * Note: Use a \"long\" per-cpu counter at level 0 to allow precise sum.\n+ *\n+ * Note: Use cacheline aligned counters at levels above 0 to prevent false sharing.\n+ * If memory footprint is an issue, a specialized allocator could be used\n+ * to eliminate padding.\n+ *\n+ * Example with expanded values:\n+ *\n+ * counter_add(counter, inc):\n+ *\n+ * if (!inc)\n+ * return;\n+ *\n+ * res = percpu_add_return(counter @ Level 0, inc);\n+ * orig = res - inc;\n+ * if (inc \u003c 0) {\n+ * inc = -(-inc \u0026 ~0b00011111); // Clear used bits\n+ * // xor bit 5: underflow\n+ * if ((inc ^ orig ^ res) \u0026 0b00100000)\n+ * inc -= 0b00100000;\n+ * } else {\n+ * inc \u0026= ~0b00011111; // Clear used bits\n+ * // xor bit 5: overflow\n+ * if ((inc ^ orig ^ res) \u0026 0b00100000)\n+ * inc += 0b00100000;\n+ * }\n+ * if (!inc)\n+ * return;\n+ *\n+ * res = atomic_long_add_return(counter @ Level 1, inc);\n+ * orig = res - inc;\n+ * if (inc \u003c 0) {\n+ * inc = -(-inc \u0026 ~0b00111111); // Clear used bits\n+ * // xor bit 6: underflow\n+ * if ((inc ^ orig ^ res) \u0026 0b01000000)\n+ * inc -= 0b01000000;\n+ * } else {\n+ * inc \u0026= ~0b00111111; // Clear used bits\n+ * // xor bit 6: overflow\n+ * if ((inc ^ orig ^ res) \u0026 0b01000000)\n+ * inc += 0b01000000;\n+ * }\n+ * if (!inc)\n+ * return;\n+ *\n+ * res = atomic_long_add_return(counter @ Level 2, inc);\n+ * orig = res - inc;\n+ * if (inc \u003c 0) {\n+ * inc = -(-inc \u0026 ~0b01111111); // Clear used bits\n+ * // xor bit 7: underflow\n+ * if ((inc ^ orig ^ res) \u0026 0b10000000)\n+ * inc -= 0b10000000;\n+ * } else {\n+ * inc \u0026= ~0b01111111; // Clear used bits\n+ * // xor bit 7: overflow\n+ * if ((inc ^ orig ^ res) \u0026 0b10000000)\n+ * inc += 0b10000000;\n+ * }\n+ * if (!inc)\n+ * return;\n+ *\n+ * atomic_long_add(counter @ Level 3, inc);\n+ */\n+\n+#include \u003clinux/percpu_counter_tree.h\u003e\n+#include \u003clinux/cpumask.h\u003e\n+#include \u003clinux/atomic.h\u003e\n+#include \u003clinux/export.h\u003e\n+#include \u003clinux/percpu.h\u003e\n+#include \u003clinux/errno.h\u003e\n+#include \u003clinux/slab.h\u003e\n+#include \u003clinux/math.h\u003e\n+\n+#define MAX_NR_LEVELS 5\n+\n+/*\n+ * The counter configuration is selected at boot time based on the\n+ * hardware topology.\n+ */\n+struct counter_config {\n+\tunsigned int nr_items;\t\t\t\t/*\n+\t\t\t\t\t\t\t * nr_items is the number of items in the tree for levels 1\n+\t\t\t\t\t\t\t * up to and including the final level (approximate sum).\n+\t\t\t\t\t\t\t * It excludes the level 0 per-CPU counters.\n+\t\t\t\t\t\t\t */\n+\tunsigned char nr_levels;\t\t\t/*\n+\t\t\t\t\t\t\t * nr_levels is the number of hierarchical counter tree levels.\n+\t\t\t\t\t\t\t * It excludes the final level (approximate sum).\n+\t\t\t\t\t\t\t */\n+\tunsigned char n_arity_order[MAX_NR_LEVELS];\t/*\n+\t\t\t\t\t\t\t * n-arity of tree nodes for each level from\n+\t\t\t\t\t\t\t * 0 to (nr_levels - 1).\n+\t\t\t\t\t\t\t */\n+};\n+\n+static const struct counter_config per_nr_cpu_order_config[] = {\n+\t[0] =\t{ .nr_items = 0,\t.nr_levels = 0,\t\t.n_arity_order = { 0 } },\n+\t[1] =\t{ .nr_items = 1,\t.nr_levels = 1,\t\t.n_arity_order = { 1 } },\n+\t[2] =\t{ .nr_items = 3,\t.nr_levels = 2,\t\t.n_arity_order = { 1, 1 } },\n+\t[3] =\t{ .nr_items = 7,\t.nr_levels = 3,\t\t.n_arity_order = { 1, 1, 1 } },\n+\t[4] =\t{ .nr_items = 7,\t.nr_levels = 3,\t\t.n_arity_order = { 2, 1, 1 } },\n+\t[5] =\t{ .nr_items = 11,\t.nr_levels = 3,\t\t.n_arity_order = { 2, 2, 1 } },\n+\t[6] =\t{ .nr_items = 21,\t.nr_levels = 3,\t\t.n_arity_order = { 2, 2, 2 } },\n+\t[7] =\t{ .nr_items = 21,\t.nr_levels = 3,\t\t.n_arity_order = { 3, 2, 2 } },\n+\t[8] =\t{ .nr_items = 37,\t.nr_levels = 3,\t\t.n_arity_order = { 3, 3, 2 } },\n+\t[9] =\t{ .nr_items = 73,\t.nr_levels = 3,\t\t.n_arity_order = { 3, 3, 3 } },\n+\t[10] =\t{ .nr_items = 149,\t.nr_levels = 4,\t\t.n_arity_order = { 3, 3, 2, 2 } },\n+\t[11] =\t{ .nr_items = 293,\t.nr_levels = 4,\t\t.n_arity_order = { 3, 3, 3, 2 } },\n+\t[12] =\t{ .nr_items = 585,\t.nr_levels = 4,\t\t.n_arity_order = { 3, 3, 3, 3 } },\n+\t[13] =\t{ .nr_items = 1173,\t.nr_levels = 5,\t\t.n_arity_order = { 3, 3, 3, 2, 2 } },\n+\t[14] =\t{ .nr_items = 2341,\t.nr_levels = 5,\t\t.n_arity_order = { 3, 3, 3, 3, 2 } },\n+\t[15] =\t{ .nr_items = 4681,\t.nr_levels = 5,\t\t.n_arity_order = { 3, 3, 3, 3, 3 } },\n+\t[16] =\t{ .nr_items = 4681,\t.nr_levels = 5,\t\t.n_arity_order = { 4, 3, 3, 3, 3 } },\n+\t[17] =\t{ .nr_items = 8777,\t.nr_levels = 5,\t\t.n_arity_order = { 4, 4, 3, 3, 3 } },\n+\t[18] =\t{ .nr_items = 17481,\t.nr_levels = 5,\t\t.n_arity_order = { 4, 4, 4, 3, 3 } },\n+\t[19] =\t{ .nr_items = 34953,\t.nr_levels = 5,\t\t.n_arity_order = { 4, 4, 4, 4, 3 } },\n+\t[20] =\t{ .nr_items = 69905,\t.nr_levels = 5,\t\t.n_arity_order = { 4, 4, 4, 4, 4 } },\n+};\n+\n+static const struct counter_config *counter_config;\t/* Hierarchical counter configuration for the hardware topology. */\n+static unsigned int nr_cpus_order;\t\t\t/* Order of nr_cpu_ids. */\n+static unsigned long accuracy_multiplier;\t\t/* Calculate accuracy for a given batch size (multiplication factor). */\n+\n+static\n+int __percpu_counter_tree_init(struct percpu_counter_tree *counter,\n+\t\t\t unsigned long batch_size, gfp_t gfp_flags,\n+\t\t\t unsigned long __percpu *level0,\n+\t\t\t struct percpu_counter_tree_level_item *items)\n+{\n+\t/* Batch size must be greater than 1, and a power of 2. */\n+\tif (WARN_ON(batch_size \u003c= 1 || (batch_size \u0026 (batch_size - 1))))\n+\t\treturn -EINVAL;\n+\tcounter-\u003ebatch_size = batch_size;\n+\tcounter-\u003ebias = 0;\n+\tcounter-\u003elevel0 = level0;\n+\tcounter-\u003eitems = items;\n+\tif (!nr_cpus_order) {\n+\t\tcounter-\u003eapprox_sum.i = per_cpu_ptr(counter-\u003elevel0, 0);\n+\t\tcounter-\u003elevel0_bit_mask = 0;\n+\t} else {\n+\t\tcounter-\u003eapprox_sum.a = \u0026counter-\u003eitems[counter_config-\u003enr_items - 1].count;\n+\t\tcounter-\u003elevel0_bit_mask = 1UL \u003c\u003c get_count_order(batch_size);\n+\t}\n+\t/*\n+\t * Each tree item signed integer has a negative range which is\n+\t * one unit greater than the positive range.\n+\t */\n+\tcounter-\u003eapprox_accuracy_range.under = batch_size * accuracy_multiplier;\n+\tcounter-\u003eapprox_accuracy_range.over = (batch_size - 1) * accuracy_multiplier;\n+\treturn 0;\n+}\n+\n+/**\n+ * percpu_counter_tree_init_many() - Initialize many per-CPU counter trees.\n+ * @counters: An array of @nr_counters counters to initialize.\n+ *\t Their memory is provided by the caller.\n+ * @items: Pointer to memory area where to store tree items.\n+ *\t This memory is provided by the caller.\n+ *\t Its size needs to be at least @nr_counters * percpu_counter_tree_items_size().\n+ * @nr_counters: The number of counter trees to initialize\n+ * @batch_size: The batch size is the increment step at level 0 which triggers a\n+ * \t\tcarry propagation.\n+ *\t\tThe batch size is required to be greater than 1, and a power of 2.\n+ * @gfp_flags: gfp flags to pass to the per-CPU allocator.\n+ *\n+ * Initialize many per-CPU counter trees using a single per-CPU\n+ * allocator invocation for @nr_counters counters.\n+ *\n+ * Return:\n+ * * %0: Success\n+ * * %-EINVAL:\t\t- Invalid @batch_size argument\n+ * * %-ENOMEM:\t\t- Out of memory\n+ */\n+int percpu_counter_tree_init_many(struct percpu_counter_tree *counters, struct percpu_counter_tree_level_item *items,\n+\t\t\t\t unsigned int nr_counters, unsigned long batch_size, gfp_t gfp_flags)\n+{\n+\tvoid __percpu *level0, *level0_iter;\n+\tsize_t counter_size = sizeof(*counters-\u003elevel0),\n+\t items_size = percpu_counter_tree_items_size();\n+\tvoid *items_iter;\n+\tunsigned int i;\n+\tint ret;\n+\n+\tmemset(items, 0, items_size * nr_counters);\n+\tlevel0 = __alloc_percpu_gfp(nr_counters * counter_size,\n+\t\t\t\t __alignof__(*counters-\u003elevel0), gfp_flags);\n+\tif (!level0)\n+\t\treturn -ENOMEM;\n+\tlevel0_iter = level0;\n+\titems_iter = items;\n+\tfor (i = 0; i \u003c nr_counters; i++) {\n+\t\tret = __percpu_counter_tree_init(\u0026counters[i], batch_size, gfp_flags, level0_iter, items_iter);\n+\t\tif (ret)\n+\t\t\tgoto free_level0;\n+\t\tlevel0_iter += counter_size;\n+\t\titems_iter += items_size;\n+\t}\n+\treturn 0;\n+\n+free_level0:\n+\tfree_percpu(level0);\n+\treturn ret;\n+}\n+EXPORT_SYMBOL_GPL(percpu_counter_tree_init_many);\n+\n+/**\n+ * percpu_counter_tree_init() - Initialize one per-CPU counter tree.\n+ * @counter: Counter to initialize.\n+ *\t Its memory is provided by the caller.\n+ * @items: Pointer to memory area where to store tree items.\n+ *\t This memory is provided by the caller.\n+ *\t Its size needs to be at least percpu_counter_tree_items_size().\n+ * @batch_size: The batch size is the increment step at level 0 which triggers a\n+ * \t\tcarry propagation.\n+ *\t\tThe batch size is required to be greater than 1, and a power of 2.\n+ * @gfp_flags: gfp flags to pass to the per-CPU allocator.\n+ *\n+ * Initialize one per-CPU counter tree.\n+ *\n+ * Return:\n+ * * %0: Success\n+ * * %-EINVAL:\t\t- Invalid @batch_size argument\n+ * * %-ENOMEM:\t\t- Out of memory\n+ */\n+int percpu_counter_tree_init(struct percpu_counter_tree *counter, struct percpu_counter_tree_level_item *items,\n+\t\t\t unsigned long batch_size, gfp_t gfp_flags)\n+{\n+\treturn percpu_counter_tree_init_many(counter, items, 1, batch_size, gfp_flags);\n+}\n+EXPORT_SYMBOL_GPL(percpu_counter_tree_init);\n+\n+/**\n+ * percpu_counter_tree_destroy_many() - Destroy many per-CPU counter trees.\n+ * @counters: Array of counters trees to destroy.\n+ * @nr_counters: The number of counter trees to destroy.\n+ *\n+ * Release internal resources allocated for @nr_counters per-CPU counter trees.\n+ */\n+\n+void percpu_counter_tree_destroy_many(struct percpu_counter_tree *counters, unsigned int nr_counters)\n+{\n+\tfree_percpu(counters-\u003elevel0);\n+}\n+EXPORT_SYMBOL_GPL(percpu_counter_tree_destroy_many);\n+\n+/**\n+ * percpu_counter_tree_destroy() - Destroy one per-CPU counter tree.\n+ * @counter: Counter to destroy.\n+ *\n+ * Release internal resources allocated for one per-CPU counter tree.\n+ */\n+void percpu_counter_tree_destroy(struct percpu_counter_tree *counter)\n+{\n+\treturn percpu_counter_tree_destroy_many(counter, 1);\n+}\n+EXPORT_SYMBOL_GPL(percpu_counter_tree_destroy);\n+\n+static\n+long percpu_counter_tree_carry(long orig, long res, long inc, unsigned long bit_mask)\n+{\n+\tif (inc \u003c 0) {\n+\t\tinc = -(-inc \u0026 ~(bit_mask - 1));\n+\t\t/*\n+\t\t * xor bit_mask: underflow.\n+\t\t *\n+\t\t * If inc has bit set, decrement an additional bit if\n+\t\t * there is _no_ bit transition between orig and res.\n+\t\t * Else, inc has bit cleared, decrement an additional\n+\t\t * bit if there is a bit transition between orig and\n+\t\t * res.\n+\t\t */\n+\t\tif ((inc ^ orig ^ res) \u0026 bit_mask)\n+\t\t\tinc -= bit_mask;\n+\t} else {\n+\t\tinc \u0026= ~(bit_mask - 1);\n+\t\t/*\n+\t\t * xor bit_mask: overflow.\n+\t\t *\n+\t\t * If inc has bit set, increment an additional bit if\n+\t\t * there is _no_ bit transition between orig and res.\n+\t\t * Else, inc has bit cleared, increment an additional\n+\t\t * bit if there is a bit transition between orig and\n+\t\t * res.\n+\t\t */\n+\t\tif ((inc ^ orig ^ res) \u0026 bit_mask)\n+\t\t\tinc += bit_mask;\n+\t}\n+\treturn inc;\n+}\n+\n+/*\n+ * It does not matter through which path the carry propagates up the\n+ * tree, therefore there is no need to disable preemption because the\n+ * cpu number is only used to favor cache locality.\n+ */\n+static\n+void percpu_counter_tree_add_slowpath(struct percpu_counter_tree *counter, long inc)\n+{\n+\tunsigned int level_items, nr_levels = counter_config-\u003enr_levels,\n+\t\t level, n_arity_order;\n+\tunsigned long bit_mask;\n+\tstruct percpu_counter_tree_level_item *item = counter-\u003eitems;\n+\tunsigned int cpu = raw_smp_processor_id();\n+\n+\tWARN_ON_ONCE(!nr_cpus_order);\t/* Should never be called for 1 cpu. */\n+\n+\tn_arity_order = counter_config-\u003en_arity_order[0];\n+\tbit_mask = counter-\u003elevel0_bit_mask \u003c\u003c n_arity_order;\n+\tlevel_items = 1U \u003c\u003c (nr_cpus_order - n_arity_order);\n+\n+\tfor (level = 1; level \u003c nr_levels; level++) {\n+\t\t/*\n+\t\t * For the purpose of carry propagation, the\n+\t\t * intermediate level counters only need to keep track\n+\t\t * of the bits relevant for carry propagation. We\n+\t\t * therefore don't care about higher order bits.\n+\t\t * Note that this optimization is unwanted if the\n+\t\t * intended use is to track counters within intermediate\n+\t\t * levels of the topology.\n+\t\t */\n+\t\tif (abs(inc) \u0026 (bit_mask - 1)) {\n+\t\t\tatomic_long_t *count = \u0026item[cpu \u0026 (level_items - 1)].count;\n+\t\t\tunsigned long orig, res;\n+\n+\t\t\tres = atomic_long_add_return_relaxed(inc, count);\n+\t\t\torig = res - inc;\n+\t\t\tinc = percpu_counter_tree_carry(orig, res, inc, bit_mask);\n+\t\t\tif (likely(!inc))\n+\t\t\t\treturn;\n+\t\t}\n+\t\titem += level_items;\n+\t\tn_arity_order = counter_config-\u003en_arity_order[level];\n+\t\tlevel_items \u003e\u003e= n_arity_order;\n+\t\tbit_mask \u003c\u003c= n_arity_order;\n+\t}\n+\tatomic_long_add(inc, counter-\u003eapprox_sum.a);\n+}\n+\n+/**\n+ * percpu_counter_tree_add() - Add to a per-CPU counter tree.\n+ * @counter: Counter added to.\n+ * @inc: Increment value (either positive or negative).\n+ *\n+ * Add @inc to a per-CPU counter tree. This is a fast-path which will\n+ * typically increment per-CPU counters as long as there is no carry\n+ * greater or equal to the counter tree batch size.\n+ */\n+void percpu_counter_tree_add(struct percpu_counter_tree *counter, long inc)\n+{\n+\tunsigned long bit_mask = counter-\u003elevel0_bit_mask, orig, res;\n+\n+\tres = this_cpu_add_return(*counter-\u003elevel0, inc);\n+\torig = res - inc;\n+\tinc = percpu_counter_tree_carry(orig, res, inc, bit_mask);\n+\tif (likely(!inc))\n+\t\treturn;\n+\tpercpu_counter_tree_add_slowpath(counter, inc);\n+}\n+EXPORT_SYMBOL_GPL(percpu_counter_tree_add);\n+\n+static\n+long percpu_counter_tree_precise_sum_unbiased(struct percpu_counter_tree *counter)\n+{\n+\tunsigned long sum = 0;\n+\tint cpu;\n+\n+\tfor_each_possible_cpu(cpu)\n+\t\tsum += *per_cpu_ptr(counter-\u003elevel0, cpu);\n+\treturn (long) sum;\n+}\n+\n+/**\n+ * percpu_counter_tree_precise_sum() - Return precise counter sum.\n+ * @counter: The counter to sum.\n+ *\n+ * Querying the precise sum is relatively expensive because it needs to\n+ * iterate over all CPUs.\n+ * This is meant to be used when accuracy is preferred over speed.\n+ *\n+ * Return: The current precise counter sum.\n+ */\n+long percpu_counter_tree_precise_sum(struct percpu_counter_tree *counter)\n+{\n+\treturn percpu_counter_tree_precise_sum_unbiased(counter) + READ_ONCE(counter-\u003ebias);\n+}\n+EXPORT_SYMBOL_GPL(percpu_counter_tree_precise_sum);\n+\n+/*\n+ * The comparison between counters is on the wrapped difference. It is valid\n+ * only when the two counters differ by less than LONG_MAX, and results are\n+ * undefined beyond that.\n+ */\n+static inline\n+long wrapped_delta(long a, long b)\n+{\n+\treturn (long)((unsigned long)a - (unsigned long)b);\n+}\n+\n+/*\n+ * Each counter's approximation lies within [precise - under, precise + over]:\n+ *\n+ * approx_a range\n+ * \u003c-----------+-----------\u003e\n+ * under_a | over_a\n+ * precise_a\n+ * | approx_b range\n+ * | \u003c-----------+-----------\u003e\n+ * | under_b | over_b\n+ * | precise_b\n+ * | |\n+ * |\u003c------------ gap -------------\u003e|\n+ * ----------------------+--------------------------------+-------------\u003e values\n+ *\n+ * The two ranges are disjoint, and the comparison can return a definitive\n+ * answer, only once the gap exceeds the two facing half-widths:\n+ *\n+ * a below b: gap \u003e over_a + under_b (a's upper edge, b's lower edge)\n+ * a above b: gap \u003e under_a + over_b (a's lower edge, b's upper edge)\n+ *\n+ * When comparing against a plain value, the value has zero width, so the\n+ * margins reduce to that single counter's under and over.\n+ *\n+ * accuracy_neg is used when delta \u003c 0, and accuracy_pos is used when delta \u003e= 0.\n+ */\n+static\n+int compare_delta(long delta, unsigned long accuracy_neg, unsigned long accuracy_pos)\n+{\n+\tif (delta \u003e= 0) {\n+\t\tif (delta \u003c= accuracy_pos)\n+\t\t\treturn 0;\n+\t\telse\n+\t\t\treturn 1;\n+\t} else {\n+\t\tif (-delta \u003c= accuracy_neg)\n+\t\t\treturn 0;\n+\t\telse\n+\t\t\treturn -1;\n+\t}\n+}\n+\n+/**\n+ * percpu_counter_tree_approximate_compare - Approximated comparison of two counter trees.\n+ * @a: First counter to compare.\n+ * @b: Second counter to compare.\n+ *\n+ * Evaluate an approximate comparison of two counter trees. This approximation\n+ * comparison is fast, and provides an accurate answer if the counters are\n+ * found to be either less than or greater than the other. However, if the\n+ * approximated comparison returns 0, the counters respective sums are found to\n+ * be within the two counters accuracy range. The two counters are read\n+ * independently; the result is not an atomic snapshot of both.\n+ * The comparison between counters is on the wrapped difference. It is valid\n+ * only when the two counters differ by less than LONG_MAX, and results are\n+ * undefined beyond that.\n+ *\n+ * Return:\n+ * * %0\t\t- Counters @a and @b do not differ by more than the sum of their respective\n+ * accuracy ranges.\n+ * * %-1\t- Counter @a less than counter @b.\n+ * * %1\t\t- Counter @a is greater than counter @b.\n+ */\n+int percpu_counter_tree_approximate_compare(struct percpu_counter_tree *a, struct percpu_counter_tree *b)\n+{\n+\t/* See the range geometry above compare_delta(). */\n+\treturn compare_delta(wrapped_delta(percpu_counter_tree_approximate_sum(a),\n+\t\t\t\t\t percpu_counter_tree_approximate_sum(b)),\n+\t\t\t a-\u003eapprox_accuracy_range.over + b-\u003eapprox_accuracy_range.under,\n+\t\t\t a-\u003eapprox_accuracy_range.under + b-\u003eapprox_accuracy_range.over);\n+}\n+EXPORT_SYMBOL_GPL(percpu_counter_tree_approximate_compare);\n+\n+/**\n+ * percpu_counter_tree_approximate_compare_value - Approximated comparison of a counter tree against a given value.\n+ * @counter: Counter to compare.\n+ * @v: Value to compare.\n+ *\n+ * Evaluate an approximate comparison of a counter tree against a given value.\n+ * This approximation comparison is fast, and provides an accurate\n+ * answer if the counter is found to be either less than or greater\n+ * than the value. However, if the approximated comparison returns\n+ * 0, the value is within the counter accuracy range.\n+ * The comparison between counters is on the wrapped difference. It is valid\n+ * only when @v and the counter differ by less than LONG_MAX, and results are\n+ * undefined beyond that.\n+ *\n+ * Return:\n+ * * %0\t\t- The value @v is within the accuracy range of the counter.\n+ * * %-1\t- The value @v is less than the counter.\n+ * * %1\t\t- The value @v is greater than the counter.\n+ */\n+int percpu_counter_tree_approximate_compare_value(struct percpu_counter_tree *counter, long v)\n+{\n+\treturn compare_delta(wrapped_delta(v, percpu_counter_tree_approximate_sum(counter)),\n+\t\t\t counter-\u003eapprox_accuracy_range.under,\n+\t\t\t counter-\u003eapprox_accuracy_range.over);\n+}\n+EXPORT_SYMBOL_GPL(percpu_counter_tree_approximate_compare_value);\n+\n+/**\n+ * percpu_counter_tree_precise_compare - Precise comparison of two counter trees.\n+ * @a: First counter to compare.\n+ * @b: Second counter to compare.\n+ *\n+ * Evaluate a precise comparison of two counter trees. As an optimization, it\n+ * uses the approximate counter comparison to quickly compare counters which\n+ * are far apart. Only cases where counter sums are within the accuracy range\n+ * require precise counter sums. The two counters are read independently; the\n+ * result is not an atomic snapshot of both.\n+ * The comparison between counters is on the wrapped difference. It is valid\n+ * only when the two counters differ by less than LONG_MAX, and results are\n+ * undefined beyond that.\n+ *\n+ * Return:\n+ * * %0\t\t- Counters are equal.\n+ * * %-1\t- Counter @a less than counter @b.\n+ * * %1\t\t- Counter @a is greater than counter @b.\n+ */\n+int percpu_counter_tree_precise_compare(struct percpu_counter_tree *a, struct percpu_counter_tree *b)\n+{\n+\tlong count_a = percpu_counter_tree_approximate_sum(a),\n+\t count_b = percpu_counter_tree_approximate_sum(b);\n+\tunsigned long accuracy_a, accuracy_b;\n+\tlong delta = wrapped_delta(count_a, count_b);\n+\tint res;\n+\n+\t/* See the range geometry above compare_delta(). */\n+\tres = compare_delta(delta,\n+\t\t\t a-\u003eapprox_accuracy_range.over + b-\u003eapprox_accuracy_range.under,\n+\t\t\t a-\u003eapprox_accuracy_range.under + b-\u003eapprox_accuracy_range.over);\n+\t/* The values are distanced enough for an accurate approximated comparison. */\n+\tif (res)\n+\t\treturn res;\n+\n+\t/*\n+\t * The approximated comparison is within the accuracy range, therefore at least one\n+\t * precise sum is needed. Sum the counter which has the largest accuracy first.\n+\t */\n+\tif (delta \u003e= 0) {\n+\t\taccuracy_a = a-\u003eapprox_accuracy_range.under;\n+\t\taccuracy_b = b-\u003eapprox_accuracy_range.over;\n+\t} else {\n+\t\taccuracy_a = a-\u003eapprox_accuracy_range.over;\n+\t\taccuracy_b = b-\u003eapprox_accuracy_range.under;\n+\t}\n+\tif (accuracy_b \u003c accuracy_a) {\n+\t\tcount_a = percpu_counter_tree_precise_sum(a);\n+\t\tres = compare_delta(wrapped_delta(count_a, count_b),\n+\t\t\t\t b-\u003eapprox_accuracy_range.under,\n+\t\t\t\t b-\u003eapprox_accuracy_range.over);\n+\t\tif (res)\n+\t\t\treturn res;\n+\t\t/* Precise sum of second counter is required. */\n+\t\tcount_b = percpu_counter_tree_precise_sum(b);\n+\t} else {\n+\t\tcount_b = percpu_counter_tree_precise_sum(b);\n+\t\tres = compare_delta(wrapped_delta(count_a, count_b),\n+\t\t\t\t a-\u003eapprox_accuracy_range.over,\n+\t\t\t\t a-\u003eapprox_accuracy_range.under);\n+\t\tif (res)\n+\t\t\treturn res;\n+\t\t/* Precise sum of second counter is required. */\n+\t\tcount_a = percpu_counter_tree_precise_sum(a);\n+\t}\n+\tdelta = wrapped_delta(count_a, count_b);\n+\tif (delta \u003c 0)\n+\t\treturn -1;\n+\tif (delta \u003e 0)\n+\t\treturn 1;\n+\treturn 0;\n+}\n+EXPORT_SYMBOL_GPL(percpu_counter_tree_precise_compare);\n+\n+/**\n+ * percpu_counter_tree_precise_compare_value - Precise comparison of a counter tree against a given value.\n+ * @counter: Counter to compare.\n+ * @v: Value to compare.\n+ *\n+ * Evaluate a precise comparison of a counter tree against a given value. As\n+ * an optimization, it uses the approximate counter comparison to quickly\n+ * identify whether the counter and value are far apart. Only cases where the\n+ * value is within the counter accuracy range require a precise counter sum.\n+ * The comparison between counters is on the wrapped difference. It is valid\n+ * only when @v and the counter differ by less than LONG_MAX, and results are\n+ * undefined beyond that.\n+ *\n+ * Return:\n+ * * %0\t\t- The value @v is equal to the counter.\n+ * * %-1\t- The value @v is less than the counter.\n+ * * %1\t\t- The value @v is greater than the counter.\n+ */\n+int percpu_counter_tree_precise_compare_value(struct percpu_counter_tree *counter, long v)\n+{\n+\tlong count = percpu_counter_tree_approximate_sum(counter), delta;\n+\tint res;\n+\n+\tres = compare_delta(wrapped_delta(v, count),\n+\t\t\t counter-\u003eapprox_accuracy_range.under,\n+\t\t\t counter-\u003eapprox_accuracy_range.over);\n+\t/* The values are distanced enough for an accurate approximated comparison. */\n+\tif (res)\n+\t\treturn res;\n+\n+\t/* Precise sum is required. */\n+\tcount = percpu_counter_tree_precise_sum(counter);\n+\tdelta = wrapped_delta(v, count);\n+\tif (delta \u003c 0)\n+\t\treturn -1;\n+\tif (delta \u003e 0)\n+\t\treturn 1;\n+\treturn 0;\n+}\n+EXPORT_SYMBOL_GPL(percpu_counter_tree_precise_compare_value);\n+\n+static\n+void percpu_counter_tree_set_bias(struct percpu_counter_tree *counter, long bias)\n+{\n+\tWRITE_ONCE(counter-\u003ebias, bias);\n+}\n+\n+/**\n+ * percpu_counter_tree_set - Set the counter tree sum to a given value.\n+ * @counter: Counter to set.\n+ * @v: Value to set.\n+ *\n+ * Set the counter sum to a given value. It can be useful for instance\n+ * to reset the counter sum to 0. Note that even after setting the\n+ * counter sum to a given value, the counter sum approximation can\n+ * return any value within the accuracy range around that value.\n+ */\n+void percpu_counter_tree_set(struct percpu_counter_tree *counter, long v)\n+{\n+\tpercpu_counter_tree_set_bias(counter,\n+\t\t\t\t v - percpu_counter_tree_precise_sum_unbiased(counter));\n+}\n+EXPORT_SYMBOL_GPL(percpu_counter_tree_set);\n+\n+/* Initialize to SIZE_MAX to catch early boot misuses. */\n+size_t __percpu_counter_tree_items_size __ro_after_init = SIZE_MAX;\n+EXPORT_SYMBOL_GPL(__percpu_counter_tree_items_size);\n+\n+static void __init calculate_accuracy_topology(void)\n+{\n+\tunsigned int nr_levels = counter_config-\u003enr_levels, level;\n+\tunsigned int level_items = 1U \u003c\u003c nr_cpus_order;\n+\tunsigned long batch_size = 1;\n+\n+\tfor (level = 0; level \u003c nr_levels; level++) {\n+\t\tunsigned int n_arity_order = counter_config-\u003en_arity_order[level];\n+\n+\t\t/*\n+\t\t * The accuracy multiplier is derived from a batch size of 1\n+\t\t * to speed up calculating the accuracy at tree initialization.\n+\t\t */\n+\t\taccuracy_multiplier += batch_size * level_items;\n+\t\tbatch_size \u003c\u003c= n_arity_order;\n+\t\tlevel_items \u003e\u003e= n_arity_order;\n+\t}\n+}\n+\n+int __init percpu_counter_tree_subsystem_init(void)\n+{\n+\tnr_cpus_order = get_count_order(nr_cpu_ids);\n+\tif (WARN_ON_ONCE(nr_cpus_order \u003e= ARRAY_SIZE(per_nr_cpu_order_config))) {\n+\t\tprintk(KERN_ERR \"Unsupported number of CPUs (%u)\\n\", nr_cpu_ids);\n+\t\treturn -1;\n+\t}\n+\tcounter_config = \u0026per_nr_cpu_order_config[nr_cpus_order];\n+\tif (!nr_cpus_order)\n+\t\t__percpu_counter_tree_items_size = 0;\n+\telse\n+\t\t__percpu_counter_tree_items_size = counter_config-\u003enr_items *\n+\t\t\tsizeof(struct percpu_counter_tree_level_item);\n+\tcalculate_accuracy_topology();\n+\treturn 0;\n+}\ndiff --git a/lib/tests/Makefile b/lib/tests/Makefile\nindex 3cac3b63a7522..051e73705977b 100644\n--- a/lib/tests/Makefile\n+++ b/lib/tests/Makefile\n@@ -58,4 +58,6 @@ obj-$(CONFIG_UTIL_MACROS_KUNIT) += util_macros_kunit.o\n obj-$(CONFIG_RATELIMIT_KUNIT_TEST) += test_ratelimit.o\n obj-$(CONFIG_UUID_KUNIT_TEST) += uuid_kunit.o\n \n+obj-$(CONFIG_PERCPU_COUNTER_TREE_KUNIT_TEST) += percpu_counter_tree_kunit.o\n+\n obj-$(CONFIG_TEST_RUNTIME_MODULE)\t\t+= module/\ndiff --git a/lib/tests/percpu_counter_tree_kunit.c b/lib/tests/percpu_counter_tree_kunit.c\nnew file mode 100644\nindex 0000000000000..8af8d2659b57b\n--- /dev/null\n+++ b/lib/tests/percpu_counter_tree_kunit.c\n@@ -0,0 +1,583 @@\n+// SPDX-License-Identifier: GPL-2.0+ OR MIT\n+// SPDX-FileCopyrightText: 2026 Mathieu Desnoyers \u003cmathieu.desnoyers@efficios.com\u003e\n+\n+#include \u003ckunit/test.h\u003e\n+#include \u003clinux/percpu_counter_tree.h\u003e\n+#include \u003clinux/kthread.h\u003e\n+#include \u003clinux/wait.h\u003e\n+#include \u003clinux/random.h\u003e\n+#include \u003clinux/cpuhplock.h\u003e\n+#include \u003clinux/slab.h\u003e\n+#include \u003clinux/sched/task.h\u003e\n+\n+struct multi_thread_test_data {\n+\tlong increment;\n+\tunsigned int nr_inc;\n+\tint counter_index;\n+};\n+\n+#define NR_COUNTERS\t2\n+\n+/* Hierarchical per-CPU counter instances. */\n+static struct percpu_counter_tree counter[NR_COUNTERS];\n+static struct percpu_counter_tree_level_item *items;\n+\n+/* Global atomic counters for validation. */\n+static atomic_long_t global_counter[NR_COUNTERS];\n+\n+static DECLARE_WAIT_QUEUE_HEAD(kernel_threads_wq);\n+static atomic_t kernel_threads_to_run;\n+\n+static void complete_work(void)\n+{\n+\tif (atomic_dec_and_test(\u0026kernel_threads_to_run))\n+\t\twake_up(\u0026kernel_threads_wq);\n+}\n+\n+/* Random tasks test. */\n+#define NR_RAND_TASKS\t1000\n+\n+static void hpcc_print_info(struct kunit *test)\n+{\n+\tkunit_info(test, \"Running test with %d CPUs\\n\", num_online_cpus());\n+}\n+\n+static void add_to_counter(int counter_index, unsigned int nr_inc, long increment)\n+{\n+\tunsigned int i;\n+\n+\tfor (i = 0; i \u003c nr_inc; i++) {\n+\t\tpercpu_counter_tree_add(\u0026counter[counter_index], increment);\n+\t\tatomic_long_add(increment, \u0026global_counter[counter_index]);\n+\t}\n+}\n+\n+static void check_counters(struct kunit *test)\n+{\n+\tint counter_index;\n+\n+\t/* Compare each counter with its global counter. */\n+\tfor (counter_index = 0; counter_index \u003c NR_COUNTERS; counter_index++) {\n+\t\tlong v = atomic_long_read(\u0026global_counter[counter_index]);\n+\t\tlong approx_sum = percpu_counter_tree_approximate_sum(\u0026counter[counter_index]);\n+\t\tunsigned long under_accuracy = 0, over_accuracy = 0;\n+\t\tlong precise_min, precise_max;\n+\n+\t\t/* Precise comparison. */\n+\t\tKUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(\u0026counter[counter_index]), v);\n+\t\tKUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_precise_compare_value(\u0026counter[counter_index], v));\n+\n+\t\t/* Approximate comparison. */\n+\t\tKUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare_value(\u0026counter[counter_index], v));\n+\n+\t\t/* Accuracy limits checks. */\n+\t\tpercpu_counter_tree_approximate_accuracy_range(\u0026counter[counter_index], \u0026under_accuracy, \u0026over_accuracy);\n+\n+\t\t/*\n+\t\t * under_accuracy and over_accuracy are unsigned long, so the\n+\t\t * inner expression promotes to unsigned. Compute the difference\n+\t\t * in unsigned and cast to signed for the comparison against 0.\n+\t\t */\n+\t\tKUNIT_EXPECT_GE(test, (long)(approx_sum - (v - under_accuracy)), 0);\n+\t\tKUNIT_EXPECT_LE(test, (long)(approx_sum - (v + over_accuracy)), 0);\n+\n+\t\t/* Precise min/max range check. */\n+\t\tpercpu_counter_tree_approximate_min_max_range(approx_sum, under_accuracy, over_accuracy, \u0026precise_min, \u0026precise_max);\n+\n+\t\tKUNIT_EXPECT_GE(test, v - precise_min, 0);\n+\t\tKUNIT_EXPECT_LE(test, v - precise_max, 0);\n+\t}\n+\t/* Compare each counter with the second counter. */\n+\tKUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(\u0026counter[0]), percpu_counter_tree_precise_sum(\u0026counter[1]));\n+\tKUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_precise_compare(\u0026counter[0], \u0026counter[1]));\n+\tKUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare(\u0026counter[0], \u0026counter[1]));\n+}\n+\n+static int multi_thread_worker_fn(void *data)\n+{\n+\tstruct multi_thread_test_data *td = data;\n+\n+\tadd_to_counter(td-\u003ecounter_index, td-\u003enr_inc, td-\u003eincrement);\n+\tcomplete_work();\n+\tkfree(td);\n+\treturn 0;\n+}\n+\n+static struct task_struct *test_run_on_specific_cpu(int target_cpu, int counter_index, unsigned int nr_inc, long increment)\n+{\n+\tstruct multi_thread_test_data *td = kzalloc(sizeof(struct multi_thread_test_data), GFP_KERNEL);\n+\tstruct task_struct *task;\n+\n+\tif (!td)\n+\t\treturn NULL;\n+\ttd-\u003eincrement = increment;\n+\ttd-\u003enr_inc = nr_inc;\n+\ttd-\u003ecounter_index = counter_index;\n+\tatomic_inc(\u0026kernel_threads_to_run);\n+\ttask = kthread_create_on_cpu(multi_thread_worker_fn, td, target_cpu, \"kunit_multi_thread_worker\");\n+\tif (IS_ERR_OR_NULL(task)) {\n+\t\tcomplete_work();\n+\t\tkfree(td);\n+\t\treturn task;\n+\t}\n+\tget_task_struct(task);\n+\twake_up_process(task);\n+\treturn task;\n+}\n+\n+static void init_kthreads(void)\n+{\n+\tatomic_set(\u0026kernel_threads_to_run, 1);\n+}\n+\n+/*\n+ * The caller needs to wait for work completion _before_ stopping kthreads.\n+ */\n+static void fini_kthreads(void)\n+{\n+\t/* Release our own reference. */\n+\tcomplete_work();\n+\t/* Wait for all others threads to run. */\n+\twait_event(kernel_threads_wq, (atomic_read(\u0026kernel_threads_to_run) == 0));\n+}\n+\n+static void test_sync_kthreads(void)\n+{\n+\tfini_kthreads();\n+\tinit_kthreads();\n+}\n+\n+static int init_counters(unsigned long batch_size)\n+{\n+\tint i, ret;\n+\n+\titems = kvcalloc(NR_COUNTERS, percpu_counter_tree_items_size(), GFP_KERNEL);\n+\tif (!items)\n+\t\treturn -ENOMEM;\n+\tret = percpu_counter_tree_init_many(counter, items, NR_COUNTERS, batch_size, GFP_KERNEL);\n+\tif (ret) {\n+\t\tkvfree(items);\n+\t\treturn ret;\n+\t}\n+\tfor (i = 0; i \u003c NR_COUNTERS; i++)\n+\t\tatomic_long_set(\u0026global_counter[i], 0);\n+\treturn 0;\n+}\n+\n+static void fini_counters(void)\n+{\n+\tpercpu_counter_tree_destroy_many(counter, NR_COUNTERS);\n+\tkvfree(items);\n+}\n+\n+enum up_test_inc_type {\n+\tINC_ONE,\n+\tINC_MINUS_ONE,\n+\tINC_RANDOM,\n+};\n+\n+/*\n+ * Single-threaded tests. Those use many threads to run on various CPUs,\n+ * but synchronize for completion of each thread before running the\n+ * next, effectively making sure there are no concurrent updates.\n+ */\n+static void do_hpcc_test_single_thread(struct kunit *test, int _cpu0, int _cpu1, enum up_test_inc_type type)\n+{\n+\tunsigned long batch_size_order = 5;\n+\tint cpu0 = _cpu0;\n+\tint cpu1 = _cpu1;\n+\tint ret, i;\n+\n+\tret = init_counters(1UL \u003c\u003c batch_size_order);\n+\tKUNIT_EXPECT_EQ(test, ret, 0);\n+\tif (ret)\n+\t\treturn;\n+\tinit_kthreads();\n+\tcpus_read_lock();\n+\tfor (i = 0; i \u003c 10000; i++) {\n+\t\tstruct task_struct *task;\n+\t\tlong increment;\n+\n+\t\tswitch (type) {\n+\t\tcase INC_ONE:\n+\t\t\tincrement = 1;\n+\t\t\tbreak;\n+\t\tcase INC_MINUS_ONE:\n+\t\t\tincrement = -1;\n+\t\t\tbreak;\n+\t\tcase INC_RANDOM:\n+\t\t\tincrement = (long) get_random_long() % 50000;\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\tWARN_ON_ONCE(1);\n+\t\t\tincrement = 0;\n+\t\t}\n+\t\tif (_cpu0 \u003c 0)\n+\t\t\tcpu0 = cpumask_any_distribute(cpu_online_mask);\n+\t\tif (_cpu1 \u003c 0)\n+\t\t\tcpu1 = cpumask_any_distribute(cpu_online_mask);\n+\t\ttask = test_run_on_specific_cpu(cpu0, 0, 1, increment);\n+\t\tKUNIT_EXPECT_NOT_ERR_OR_NULL(test, task);\n+\t\tif (IS_ERR_OR_NULL(task)) {\n+\t\t\tfini_kthreads();\n+\t\t\tgoto end;\n+\t\t}\n+\t\ttest_sync_kthreads();\n+\t\tkthread_stop_put(task);\n+\t\ttask = test_run_on_specific_cpu(cpu1, 1, 1, increment);\n+\t\tKUNIT_EXPECT_NOT_ERR_OR_NULL(test, task);\n+\t\tif (IS_ERR_OR_NULL(task)) {\n+\t\t\tfini_kthreads();\n+\t\t\tgoto end;\n+\t\t}\n+\t\ttest_sync_kthreads();\n+\t\tkthread_stop_put(task);\n+\t\tcheck_counters(test);\n+\t}\n+end:\n+\tcpus_read_unlock();\n+\tfini_counters();\n+}\n+\n+static void hpcc_test_single_thread_first(struct kunit *test)\n+{\n+\tint cpu = cpumask_first(cpu_online_mask);\n+\n+\tdo_hpcc_test_single_thread(test, cpu, cpu, INC_ONE);\n+\tdo_hpcc_test_single_thread(test, cpu, cpu, INC_MINUS_ONE);\n+\tdo_hpcc_test_single_thread(test, cpu, cpu, INC_RANDOM);\n+}\n+\n+static void hpcc_test_single_thread_first_random(struct kunit *test)\n+{\n+\tint cpu = cpumask_first(cpu_online_mask);\n+\n+\tdo_hpcc_test_single_thread(test, cpu, -1, INC_ONE);\n+\tdo_hpcc_test_single_thread(test, cpu, -1, INC_MINUS_ONE);\n+\tdo_hpcc_test_single_thread(test, cpu, -1, INC_RANDOM);\n+}\n+\n+static void hpcc_test_single_thread_random(struct kunit *test)\n+{\n+\tdo_hpcc_test_single_thread(test, -1, -1, INC_ONE);\n+\tdo_hpcc_test_single_thread(test, -1, -1, INC_MINUS_ONE);\n+\tdo_hpcc_test_single_thread(test, -1, -1, INC_RANDOM);\n+}\n+\n+/* Multi-threaded SMP tests. */\n+\n+static void do_hpcc_multi_thread_increment_each_cpu(struct kunit *test, unsigned long batch_size, unsigned int nr_inc, long increment)\n+{\n+\tstruct task_struct *(*task)[NR_COUNTERS];\n+\tint ret, cpu;\n+\n+\ttask = kvcalloc(nr_cpu_ids, sizeof(*task), GFP_KERNEL);\n+\tKUNIT_EXPECT_NOT_NULL(test, task);\n+\tif (!task)\n+\t\treturn;\n+\tret = init_counters(batch_size);\n+\tKUNIT_EXPECT_EQ(test, ret, 0);\n+\tif (ret)\n+\t\tgoto end;\n+\tinit_kthreads();\n+\tcpus_read_lock();\n+\tfor_each_online_cpu(cpu) {\n+\t\ttask[cpu][0] = test_run_on_specific_cpu(cpu, 0, nr_inc, increment);\n+\t\tKUNIT_EXPECT_NOT_ERR_OR_NULL(test, task[cpu][0]);\n+\t\tif (IS_ERR_OR_NULL(task[cpu][0]))\n+\t\t\tbreak;\n+\t\ttask[cpu][1] = test_run_on_specific_cpu(cpu, 1, nr_inc, increment);\n+\t\tKUNIT_EXPECT_NOT_ERR_OR_NULL(test, task[cpu][1]);\n+\t\tif (IS_ERR_OR_NULL(task[cpu][1]))\n+\t\t\tbreak;\n+\t}\n+\tfini_kthreads();\n+\tfor_each_online_cpu(cpu) {\n+\t\tif (!IS_ERR_OR_NULL(task[cpu][0]))\n+\t\t\tkthread_stop_put(task[cpu][0]);\n+\t\tif (!IS_ERR_OR_NULL(task[cpu][1]))\n+\t\t\tkthread_stop_put(task[cpu][1]);\n+\t}\n+\tcpus_read_unlock();\n+\tcheck_counters(test);\n+\tfini_counters();\n+end:\n+\tkvfree(task);\n+}\n+\n+static void do_hpcc_multi_thread_increment_even_cpus(struct kunit *test, unsigned long batch_size, unsigned int nr_inc, long increment)\n+{\n+\tstruct task_struct *(*task)[NR_COUNTERS];\n+\tint ret, cpu;\n+\n+\ttask = kvcalloc(nr_cpu_ids, sizeof(*task), GFP_KERNEL);\n+\tKUNIT_EXPECT_NOT_NULL(test, task);\n+\tif (!task)\n+\t\treturn;\n+\tret = init_counters(batch_size);\n+\tKUNIT_EXPECT_EQ(test, ret, 0);\n+\tif (ret)\n+\t\tgoto end;\n+\tinit_kthreads();\n+\tcpus_read_lock();\n+\tfor_each_online_cpu(cpu) {\n+\t\t/*\n+\t\t * Funnel pairs of CPU indexes to their associated even index.\n+\t\t *\n+\t\t * Map 0 and 1 -\u003e index 0\n+\t\t * Map 2 and 3 -\u003e index 2 and so on.\n+\t\t */\n+\t\tint even_cpu = cpu \u0026 ~1;\n+\n+\t\tif (!cpumask_test_cpu(even_cpu, cpu_online_mask))\n+\t\t\tcontinue;\n+\t\ttask[cpu][0] = test_run_on_specific_cpu(cpu, 0, nr_inc, increment);\n+\t\tKUNIT_EXPECT_NOT_ERR_OR_NULL(test, task[cpu][0]);\n+\t\tif (IS_ERR_OR_NULL(task[cpu][0]))\n+\t\t\tbreak;\n+\t\ttask[cpu][1] = test_run_on_specific_cpu(even_cpu, 1, nr_inc, increment);\n+\t\tKUNIT_EXPECT_NOT_ERR_OR_NULL(test, task[cpu][1]);\n+\t\tif (IS_ERR_OR_NULL(task[cpu][1]))\n+\t\t\tbreak;\n+\t}\n+\tfini_kthreads();\n+\tfor_each_online_cpu(cpu) {\n+\t\tif (!IS_ERR_OR_NULL(task[cpu][0]))\n+\t\t\tkthread_stop_put(task[cpu][0]);\n+\t\tif (!IS_ERR_OR_NULL(task[cpu][1]))\n+\t\t\tkthread_stop_put(task[cpu][1]);\n+\t}\n+\tcpus_read_unlock();\n+\tcheck_counters(test);\n+\tfini_counters();\n+end:\n+\tkvfree(task);\n+}\n+\n+static void do_hpcc_multi_thread_increment_single_cpu(struct kunit *test, unsigned long batch_size, unsigned int nr_inc, long increment)\n+{\n+\tstruct task_struct *(*task)[NR_COUNTERS];\n+\tint ret, cpu;\n+\n+\ttask = kvcalloc(nr_cpu_ids, sizeof(*task), GFP_KERNEL);\n+\tKUNIT_EXPECT_NOT_NULL(test, task);\n+\tif (!task)\n+\t\treturn;\n+\tret = init_counters(batch_size);\n+\tKUNIT_EXPECT_EQ(test, ret, 0);\n+\tif (ret)\n+\t\tgoto end;\n+\tinit_kthreads();\n+\tcpus_read_lock();\n+\tfor_each_online_cpu(cpu) {\n+\t\ttask[cpu][0] = test_run_on_specific_cpu(cpu, 0, nr_inc, increment);\n+\t\tKUNIT_EXPECT_NOT_ERR_OR_NULL(test, task[cpu][0]);\n+\t\tif (IS_ERR_OR_NULL(task[cpu][0]))\n+\t\t\tbreak;\n+\t\ttask[cpu][1] = test_run_on_specific_cpu(cpumask_first(cpu_online_mask), 1, nr_inc, increment);\n+\t\tKUNIT_EXPECT_NOT_ERR_OR_NULL(test, task[cpu][1]);\n+\t\tif (IS_ERR_OR_NULL(task[cpu][1]))\n+\t\t\tbreak;\n+\t}\n+\tfini_kthreads();\n+\tfor_each_online_cpu(cpu) {\n+\t\tif (!IS_ERR_OR_NULL(task[cpu][0]))\n+\t\t\tkthread_stop_put(task[cpu][0]);\n+\t\tif (!IS_ERR_OR_NULL(task[cpu][1]))\n+\t\t\tkthread_stop_put(task[cpu][1]);\n+\t}\n+\tcpus_read_unlock();\n+\tcheck_counters(test);\n+\tfini_counters();\n+end:\n+\tkvfree(task);\n+}\n+\n+static void do_hpcc_multi_thread_increment_random_cpu(struct kunit *test, unsigned long batch_size, unsigned int nr_inc, long increment)\n+{\n+\tstruct task_struct *(*task)[NR_COUNTERS];\n+\tint ret, cpu;\n+\n+\ttask = kvcalloc(nr_cpu_ids, sizeof(*task), GFP_KERNEL);\n+\tKUNIT_EXPECT_NOT_NULL(test, task);\n+\tif (!task)\n+\t\treturn;\n+\tret = init_counters(batch_size);\n+\tKUNIT_EXPECT_EQ(test, ret, 0);\n+\tif (ret)\n+\t\tgoto end;\n+\tinit_kthreads();\n+\tcpus_read_lock();\n+\tfor_each_online_cpu(cpu) {\n+\t\ttask[cpu][0] = test_run_on_specific_cpu(cpu, 0, nr_inc, increment);\n+\t\tKUNIT_EXPECT_NOT_ERR_OR_NULL(test, task[cpu][0]);\n+\t\tif (IS_ERR_OR_NULL(task[cpu][0]))\n+\t\t\tbreak;\n+\t\ttask[cpu][1] = test_run_on_specific_cpu(cpumask_any_distribute(cpu_online_mask), 1, nr_inc, increment);\n+\t\tKUNIT_EXPECT_NOT_ERR_OR_NULL(test, task[cpu][1]);\n+\t\tif (IS_ERR_OR_NULL(task[cpu][1]))\n+\t\t\tbreak;\n+\t}\n+\tfini_kthreads();\n+\tfor_each_online_cpu(cpu) {\n+\t\tif (!IS_ERR_OR_NULL(task[cpu][0]))\n+\t\t\tkthread_stop_put(task[cpu][0]);\n+\t\tif (!IS_ERR_OR_NULL(task[cpu][1]))\n+\t\t\tkthread_stop_put(task[cpu][1]);\n+\t}\n+\tcpus_read_unlock();\n+\tcheck_counters(test);\n+\tfini_counters();\n+end:\n+\tkvfree(task);\n+}\n+\n+static void hpcc_test_multi_thread_batch_increment(struct kunit *test)\n+{\n+\tunsigned long batch_size_order;\n+\n+\tfor (batch_size_order = 2; batch_size_order \u003c 10; batch_size_order++) {\n+\t\tunsigned int nr_inc;\n+\n+\t\tfor (nr_inc = 1; nr_inc \u003c 1024; nr_inc *= 2) {\n+\t\t\tlong increment;\n+\n+\t\t\tfor (increment = 1; increment \u003c 100000; increment *= 10) {\n+\t\t\t\tdo_hpcc_multi_thread_increment_each_cpu(test, 1UL \u003c\u003c batch_size_order, nr_inc, increment);\n+\t\t\t\tdo_hpcc_multi_thread_increment_even_cpus(test, 1UL \u003c\u003c batch_size_order, nr_inc, increment);\n+\t\t\t\tdo_hpcc_multi_thread_increment_single_cpu(test, 1UL \u003c\u003c batch_size_order, nr_inc, increment);\n+\t\t\t\tdo_hpcc_multi_thread_increment_random_cpu(test, 1UL \u003c\u003c batch_size_order, nr_inc, increment);\n+\t\t\t}\n+\t\t}\n+\t}\n+}\n+\n+static void hpcc_test_multi_thread_random_walk(struct kunit *test)\n+{\n+\tstruct task_struct *(*task)[NR_COUNTERS];\n+\tunsigned long batch_size_order = 5;\n+\tint loop;\n+\n+\ttask = kvcalloc(NR_RAND_TASKS, sizeof(*task), GFP_KERNEL);\n+\tKUNIT_EXPECT_NOT_NULL(test, task);\n+\tif (!task)\n+\t\treturn;\n+\tfor (loop = 0; loop \u003c 100; loop++) {\n+\t\tint ret, i;\n+\n+\t\tmemset(task, 0, NR_RAND_TASKS * sizeof(*task));\n+\t\tret = init_counters(1UL \u003c\u003c batch_size_order);\n+\t\tKUNIT_EXPECT_EQ(test, ret, 0);\n+\t\tif (ret)\n+\t\t\tgoto end;\n+\t\tinit_kthreads();\n+\t\tcpus_read_lock();\n+\t\tfor (i = 0; i \u003c NR_RAND_TASKS; i++) {\n+\t\t\tlong increment = (long) get_random_long() % 512;\n+\t\t\tunsigned int nr_inc = ((unsigned long) get_random_long()) % 1024;\n+\n+\t\t\ttask[i][0] = test_run_on_specific_cpu(cpumask_any_distribute(cpu_online_mask), 0, nr_inc, increment);\n+\t\t\tKUNIT_EXPECT_NOT_ERR_OR_NULL(test, task[i][0]);\n+\t\t\tif (IS_ERR_OR_NULL(task[i][0]))\n+\t\t\t\tbreak;\n+\t\t\ttask[i][1] = test_run_on_specific_cpu(cpumask_any_distribute(cpu_online_mask), 1, nr_inc, increment);\n+\t\t\tKUNIT_EXPECT_NOT_ERR_OR_NULL(test, task[i][1]);\n+\t\t\tif (IS_ERR_OR_NULL(task[i][1]))\n+\t\t\t\tbreak;\n+\t\t}\n+\t\tfini_kthreads();\n+\t\tfor (i = 0; i \u003c NR_RAND_TASKS; i++) {\n+\t\t\tif (!IS_ERR_OR_NULL(task[i][0]))\n+\t\t\t\tkthread_stop_put(task[i][0]);\n+\t\t\tif (!IS_ERR_OR_NULL(task[i][1]))\n+\t\t\t\tkthread_stop_put(task[i][1]);\n+\t\t}\n+\t\tcpus_read_unlock();\n+\t\tcheck_counters(test);\n+\t\tfini_counters();\n+\t}\n+end:\n+\tkvfree(task);\n+}\n+\n+static void hpcc_test_init_one(struct kunit *test)\n+{\n+\tstruct percpu_counter_tree pct;\n+\tstruct percpu_counter_tree_level_item *counter_items;\n+\tint ret;\n+\n+\tcounter_items = kzalloc(percpu_counter_tree_items_size(), GFP_KERNEL);\n+\tKUNIT_EXPECT_NOT_NULL(test, counter_items);\n+\tif (!counter_items)\n+\t\treturn;\n+\tret = percpu_counter_tree_init(\u0026pct, counter_items, 32, GFP_KERNEL);\n+\tKUNIT_EXPECT_EQ(test, ret, 0);\n+\tif (ret)\n+\t\tgoto end;\n+\n+\tpercpu_counter_tree_destroy(\u0026pct);\n+end:\n+\tkfree(counter_items);\n+}\n+\n+static void hpcc_test_set(struct kunit *test)\n+{\n+\tstatic long values[] = {\n+\t\t5, 100, 127, 128, 255, 256, 4095, 4096, 500000, 0,\n+\t\t-5, -100, -127, -128, -255, -256, -4095, -4096, -500000,\n+\t};\n+\tstruct percpu_counter_tree pct;\n+\tstruct percpu_counter_tree_level_item *counter_items;\n+\tint i, ret;\n+\n+\tcounter_items = kzalloc(percpu_counter_tree_items_size(), GFP_KERNEL);\n+\tKUNIT_EXPECT_NOT_NULL(test, counter_items);\n+\tif (!counter_items)\n+\t\treturn;\n+\tret = percpu_counter_tree_init(\u0026pct, counter_items, 32, GFP_KERNEL);\n+\tKUNIT_EXPECT_EQ(test, ret, 0);\n+\tif (ret)\n+\t\tgoto end;\n+\n+\tfor (i = 0; i \u003c ARRAY_SIZE(values); i++) {\n+\t\tlong v = values[i];\n+\n+\t\tpercpu_counter_tree_set(\u0026pct, v);\n+\t\tKUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(\u0026pct), v);\n+\t\tKUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare_value(\u0026pct, v));\n+\n+\t\tpercpu_counter_tree_add(\u0026pct, v);\n+\t\tKUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(\u0026pct), 2 * v);\n+\t\tKUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare_value(\u0026pct, 2 * v));\n+\n+\t\tpercpu_counter_tree_add(\u0026pct, -2 * v);\n+\t\tKUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(\u0026pct), 0);\n+\t\tKUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare_value(\u0026pct, 0));\n+\t}\n+\n+\tpercpu_counter_tree_destroy(\u0026pct);\n+end:\n+\tkfree(counter_items);\n+}\n+\n+static struct kunit_case hpcc_test_cases[] = {\n+\tKUNIT_CASE(hpcc_print_info),\n+\tKUNIT_CASE(hpcc_test_single_thread_first),\n+\tKUNIT_CASE(hpcc_test_single_thread_first_random),\n+\tKUNIT_CASE(hpcc_test_single_thread_random),\n+\tKUNIT_CASE(hpcc_test_multi_thread_batch_increment),\n+\tKUNIT_CASE(hpcc_test_multi_thread_random_walk),\n+\tKUNIT_CASE(hpcc_test_init_one),\n+\tKUNIT_CASE(hpcc_test_set),\n+\t{}\n+};\n+\n+static struct kunit_suite hpcc_test_suite = {\n+\t.name = \"percpu_counter_tree\",\n+\t.test_cases = hpcc_test_cases,\n+};\n+\n+kunit_test_suite(hpcc_test_suite);\n+\n+MODULE_DESCRIPTION(\"Test cases for hierarchical per-CPU counters\");\n+MODULE_LICENSE(\"Dual MIT/GPL\");\n"
} |
| 2/1 | 2026/09/06 04:16 | agent | patch-evaluator |
0mModel:core Args: null Results: {
"EnableConfigs": [
"SMP"
],
"FocusSymbols": [
"percpu_counter_tree_add",
"percpu_counter_tree_approximate_compare",
"percpu_counter_tree_approximate_compare_value",
"percpu_counter_tree_init_many",
"percpu_counter_tree_precise_compare",
"percpu_counter_tree_precise_compare_value",
"percpu_counter_tree_precise_sum",
"percpu_counter_tree_set"
],
"Reasoning": "The patch introduces Hierarchical Per-CPU Counters (HPCC) and converts core mm RSS counters (struct mm_struct -\u003e rss_stat) to use HPCC. This modifies core memory management accounting, process creation/destruction, and introduces new carry propagation and comparison algorithms in reachable kernel space.",
"WorthFuzzing": true
}Instruction: You are an expert Linux kernel maintainer and security engineer.
Your job is to review a provided patch series and evaluate whether it warrants fuzzing with syzkaller.
IMPORTANT: The changes have ALREADY been applied and committed as the HEAD commit in
your workspace. Do NOT rely on internal assumptions. You must actively use your code access
tools to inspect the actual source code, callers, and surrounding context.
================================================================================
1. CORE TRIAGE PHILOSOPHY
================================================================================
The goal of patch fuzzing is to discover crashes, regressions, exposed latent bugs,
and newly triggered assertions introduced by the patch series.
- REACHABILITY IS THE PRIMARY GATE:
Fuzzing can only discover bugs in code that can actually execute in standard virtualized
environments (GCE or QEMU, utilizing software-emulated devices like USB gadgets, netdev, tun/tap).
If the modified code is structurally unreachable (see Section 2), it MUST NOT be fuzzed,
regardless of whether it adds assertions or complex logic.
- DO NOT BLINDLY TRUST "NO FUNCTIONAL CHANGE" (NFCI) OR "REFACTORING" CLAIMS:
Patch authors routinely label changes as "cleanups", "refactorings", or state
"No functional change intended". Do NOT take these claims at face value.
Code refactorings that rearrange logic, introduce helper functions, or alter state management
in core subsystems frequently introduce subtle semantic shifts or uncover latent kernel bugs.
If reachable executable code is modified or refactored, it MUST be fuzzed.
- NEW OR MODIFIED ASSERTIONS IN REACHABLE CODE MUST BE FUZZED:
When a patch introduces or modifies runtime checks or assertions (e.g., WARN_ON*, VM_WARN_ON*,
BUG_ON*, lockdep_assert*) in reachable code paths, it enforces new or stricter invariants.
Even if the author believes the invariant always holds, fuzzing is essential to verify whether
an unusual sequence of operations can violate it.
================================================================================
2. WHEN TO RETURN WorthFuzzing=false (NEGATIVE CRITERIA)
================================================================================
Return WorthFuzzing=false ONLY IF all modified code falls strictly into one or more of these categories:
- Non-kernel and non-executable changes:
* Modifications to Documentation/, comments, or spelling fixes.
* User-space directories, self-tests, samples, or scripts (e.g., tools/, samples/, scripts/, usr/)
that do not affect the compiled kernel image (vmlinux) or kernel modules.
* Purely decorative logging (e.g., message strings in pr_err, printk, dev_info) or tracepoints
that do not alter control flow or data structures.
* Build system or Kconfig changes that do not alter compiled C logic.
- Structurally unreachable hardware:
* Vendor-specific PCIe switches, SmartNICs, or GPU drivers (e.g., mlxsw, pds_core, qed,
ionic, amdgpu) requiring physical ASIC/PCIe cards not emulated in standard QEMU.
- Unreachable execution paths:
* Driver teardown callbacks (.remove, .shutdown, pci_unregister_driver) executed only during
physical PCI hot-unplug or manual sysfs driver unbinding.
* Code paths exclusive to architectures other than the target architecture.
================================================================================
3. WHEN TO RETURN WorthFuzzing=true (POSITIVE CRITERIA)
================================================================================
Return WorthFuzzing=true whenever the patch touches reachable executable code, including:
- Core Subsystems:
* Any logic modifications in memory management (mm/), synchronization/locking (kernel/locking/),
BPF, scheduler, core networking, VFS, or syscall handling.
- Refactorings and Code Cleanups:
* Any restructuring of reachable data structures, helper abstractions, or algorithm flows.
- Runtime Assertions and Defensive Checks:
* Any introduction or alteration of assertions (WARN_ON*, VM_WARN_ON*, BUG_ON*, etc.) in reachable paths.
- Reachable Drivers and Protocols:
* Drivers accessible via virtual buses (virtio, USB gadget, loopback, netlink, binder, sockets, etc.).
================================================================================
4. EXTRACTING FocusSymbols (PREVENTING DILUTION)
================================================================================
When WorthFuzzing=true, you must extract specific kernel functions into FocusSymbols to guide the fuzzer:
- AVOID UBIQUITOUS LIFECYCLE HOT-PATHS:
Do NOT list generic, ubiquitous functions called by almost every program in the corpus
(including, but not limited to: general memory allocators and deallocators, page fault
and trap handlers, or core synchronization primitives; this is not an exhaustive list).
Listing ubiquitous functions causes the fuzzer to classify thousands of unrelated tests as "focused",
which severely dilutes fuzzing effort away from the actual changes.
- TARGET SPECIFIC FEATURE LOGIC AND ENTRYPOINTS:
List functions that specifically implement the logic being added or altered, or direct API entrypoints
for the subsystem feature under review.
- HANDLING STATIC INLINE FUNCTIONS IN HEADERS (.h):
Compiler-inlined static functions (such as static inlines in mm/*.h or include/linux/*.h) lack
distinct symbol addresses in vmlinux and cannot be targeted directly by symbol coverage filters.
If the changes are primarily in static inline helpers, identify non-static, feature-specific caller
functions in .c files that exercise them (avoiding ubiquitous lifecycle wrappers).
================================================================================
5. IDENTIFYING EnableConfigs
================================================================================
Identify any specific CONFIG_ options required to properly compile and reach the modified code:
- Inspect Kconfig files and #ifdef guards; do not make assumptions.
- Check "depends on" lines in Kconfig to include any non-standard parent subsystem configs needed.
- Strip any 'CONFIG_' prefix (e.g., return "NET_IPV4" instead of "CONFIG_NET_IPV4").
Prefer calling several tools at the same time to save round-trips.
Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.
Prompt: Target architecture: amd64
For your convenience, here is the diff of the changes:
commit dde82b75d01b1954902ba04c172efd2ae7beff50
Author: syz-cluster <triage@syzkaller.com>
Date: Sun Sep 6 04:16:19 2026 +0000
syz-cluster: applied patch under review
diff --git a/Documentation/core-api/percpu-counter-tree.rst b/Documentation/core-api/percpu-counter-tree.rst
new file mode 100644
index 0000000000000..878cbd85304cf
--- /dev/null
+++ b/Documentation/core-api/percpu-counter-tree.rst
@@ -0,0 +1,82 @@
+========================================
+The Hierarchical Per-CPU Counters (HPCC)
+========================================
+
+:Author: Mathieu Desnoyers
+
+Introduction
+============
+
+Counters come in many varieties, each with their own trade offs:
+
+ * A global atomic counter provides a fast read access to the current
+ sum, at the expense of cache-line bouncing on updates. This leads to
+ poor performance of frequent updates from various cores on large SMP
+ systems.
+
+ * A per-cpu split counter provides fast updates to per-cpu counters,
+ at the expense of a slower aggregation (sum). The sum operation needs
+ to iterate over all per-cpu counters to calculate the current total.
+
+The hierarchical per-cpu counters attempt to provide the best of both
+worlds (fast updates, and fast sum) by relaxing requirements on the sum
+accuracy. It allows quickly querying an approximated sum value, along
+with the possible min/max ranges of the associated precise sum. The
+exact precise sum can still be calculated with an iteration on all
+per-cpu counter, but the availability of an approximated sum value with
+possible precise sum min/max ranges allows eliminating candidates which
+are certainly outside of a known target range without the overhead of
+precise sums.
+
+Overview
+========
+
+The herarchical per-cpu counters are organized as a tree with the tree
+root at the bottom (last level) and the first level of the tree
+consisting of per-cpu counters.
+
+The intermediate tree levels contain carry propagation counters. When
+reaching a threshold (batch size), the carry is propagated down the
+tree.
+
+This allows reading an approximated value at the root, which has a
+bounded accuracy (minimum/maximum possible precise sum range) determined
+by the tree topology.
+
+The bounded accuracy is a property of the settled tree. While updates are
+in flight the transient error can exceed it, bounded by the number of
+updaters and their increment size.
+
+Use Cases
+=========
+
+Use cases HPCC is meant to handle invove tracking resources which are
+used across many CPUs to quickly sum as feedback for decision making to
+apply throttling, quota limits, sort tasks, and perform memory or task
+migration decisions. When considering approximated sums within the
+accuracy range of the decision threshold, the user can either:
+
+ * Be conservative and fast: Consider that the sum has reached the
+ limit as soon as the given limit is within the approximation range.
+
+ * Be aggressive and fast: Consider that the sum is over the
+ limit only when the approximation range is over the given limit.
+
+ * Be precise and slow: Do a precise comparison with the limit, which
+ requires a precise sum when the limit is within the approximated
+ range.
+
+One use-case for these hierarchical counters is to implement a two-pass
+algorithm to speed up sorting picking a maximum/minimunm sum value from
+a set. A first pass compares the approximated values, and then a second
+pass only needs the precise sum for counter trees which are within the
+possible precise sum range of the counter tree chosen by the first pass.
+
+Because the comparison is on the wrapped difference, it is not transitive:
+ordering is undefined over a set of counters spanning more than LONG_MAX.
+
+Functions and structures
+========================
+
+.. kernel-doc:: include/linux/percpu_counter_tree.h
+.. kernel-doc:: lib/percpu_counter_tree.c
diff --git a/include/linux/mm.h b/include/linux/mm.h
index dd09c438fa23e..ffbcd18c4ecb3 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -3447,38 +3447,48 @@ static inline bool get_user_page_fast_only(unsigned long addr,
{
return get_user_pages_fast_only(addr, 1, gup_flags, pagep) == 1;
}
+
+static inline struct percpu_counter_tree_level_item *get_rss_stat_items(struct mm_struct *mm)
+{
+ unsigned long ptr = (unsigned long)mm;
+
+ ptr += offsetof(struct mm_struct, flexible_array);
+ ptr += get_rss_stat_items_offset();
+ return (struct percpu_counter_tree_level_item *)ptr;
+}
+
/*
* per-process(per-mm_struct) statistics.
*/
static inline unsigned long get_mm_counter(struct mm_struct *mm, int member)
{
- return percpu_counter_read_positive(&mm->rss_stat[member]);
+ return percpu_counter_tree_approximate_sum_positive(&mm->rss_stat[member]);
}
static inline unsigned long get_mm_counter_sum(struct mm_struct *mm, int member)
{
- return percpu_counter_sum_positive(&mm->rss_stat[member]);
+ return percpu_counter_tree_precise_sum_positive(&mm->rss_stat[member]);
}
void mm_trace_rss_stat(struct mm_struct *mm, int member);
static inline void add_mm_counter(struct mm_struct *mm, int member, long value)
{
- percpu_counter_add(&mm->rss_stat[member], value);
+ percpu_counter_tree_add(&mm->rss_stat[member], value);
mm_trace_rss_stat(mm, member);
}
static inline void inc_mm_counter(struct mm_struct *mm, int member)
{
- percpu_counter_inc(&mm->rss_stat[member]);
+ percpu_counter_tree_add(&mm->rss_stat[member], 1);
mm_trace_rss_stat(mm, member);
}
static inline void dec_mm_counter(struct mm_struct *mm, int member)
{
- percpu_counter_dec(&mm->rss_stat[member]);
+ percpu_counter_tree_add(&mm->rss_stat[member], -1);
mm_trace_rss_stat(mm, member);
}
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index 6d815f6440c94..b26160b85f93a 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -18,7 +18,7 @@
#include <linux/page-flags-layout.h>
#include <linux/workqueue.h>
#include <linux/seqlock.h>
-#include <linux/percpu_counter.h>
+#include <linux/percpu_counter_tree.h>
#include <linux/types.h>
#include <linux/futex_types.h>
#include <linux/rseq_types.h>
@@ -1170,6 +1170,19 @@ typedef struct {
DECLARE_BITMAP(__mm_flags, NUM_MM_FLAG_BITS);
} __private mm_flags_t;
+/*
+ * The alignment of the mm_struct flexible array is based on the largest
+ * alignment of its content:
+ * __alignof__(struct percpu_counter_tree_level_item) provides a
+ * cacheline aligned alignment on SMP systems, else alignment on
+ * unsigned long on UP systems.
+ */
+#ifdef CONFIG_SMP
+# define __mm_struct_flexible_array_aligned __aligned(__alignof__(struct percpu_counter_tree_level_item))
+#else
+# define __mm_struct_flexible_array_aligned __aligned(__alignof__(unsigned long))
+#endif
+
struct kioctx_table;
struct iommu_mm_data;
struct mm_struct {
@@ -1312,7 +1325,7 @@ struct mm_struct {
unsigned long saved_e_flags;
#endif
- struct percpu_counter rss_stat[NR_MM_COUNTERS];
+ struct percpu_counter_tree rss_stat[NR_MM_COUNTERS];
struct linux_binfmt *binfmt;
@@ -1422,10 +1435,14 @@ struct mm_struct {
} __randomize_layout;
/*
- * The mm_cpumask needs to be at the end of mm_struct, because it
- * is dynamically sized based on nr_cpu_ids.
+ * The mm_cpumask, mm_cid masks, and rss hierarchical counter
+ * items need to be at the end of mm_struct, because they are
+ * dynamically sized based on nr_cpu_ids.
+ * The HPCC counter tree items are placed last and aligned
+ * with PERCPU_COUNTER_TREE_ITEMS_ALIGN to satisfy their
+ * cacheline alignment requirement.
*/
- char flexible_array[] __aligned(__alignof__(unsigned long));
+ char flexible_array[] __mm_struct_flexible_array_aligned;
};
/* Copy value to the first system word of mm flags, non-atomically. */
@@ -1462,18 +1479,12 @@ static inline void __mm_flags_set_mask_bits_word(struct mm_struct *mm,
MT_FLAGS_USE_RCU)
extern struct mm_struct init_mm;
-#define MM_STRUCT_FLEXIBLE_ARRAY_INIT \
-{ \
- [0 ... sizeof(cpumask_t) + MM_CID_STATIC_SIZE - 1] = 0 \
-}
-
-/* Pointer magic because the dynamic array size confuses some compilers. */
-static inline void mm_init_cpumask(struct mm_struct *mm)
-{
- unsigned long cpu_bitmap = (unsigned long)mm;
-
- cpu_bitmap += offsetof(struct mm_struct, flexible_array);
- cpumask_clear((struct cpumask *)cpu_bitmap);
+#define MM_STRUCT_FLEXIBLE_ARRAY_INIT \
+{ \
+ [0 ... PERCPU_COUNTER_TREE_ITEMS_ALIGN( \
+ sizeof(cpumask_t) + MM_CID_STATIC_SIZE) \
+ + (PERCPU_COUNTER_TREE_ITEMS_STATIC_SIZE * NR_MM_COUNTERS) \
+ - 1] = 0 \
}
/* Future-safe accessor for struct mm_struct's cpu_vm_mask. */
@@ -1482,6 +1493,11 @@ static inline cpumask_t *mm_cpumask(struct mm_struct *mm)
return (struct cpumask *)&mm->flexible_array;
}
+static inline void mm_init_cpumask(struct mm_struct *mm)
+{
+ cpumask_clear((struct cpumask *)mm_cpumask(mm));
+}
+
#ifdef CONFIG_LRU_GEN
struct lru_gen_mm_list {
@@ -1653,6 +1669,21 @@ static inline void mm_destroy_sched(struct mm_struct *mm) { }
#endif /* CONFIG_SCHED_CACHE */
+static inline size_t get_rss_stat_items_size(void)
+{
+ return percpu_counter_tree_items_size() * NR_MM_COUNTERS;
+}
+
+/*
+ * Return the offset of the RSS stat HPCC items within the mm_struct
+ * flexible array. The items are placed after the cpumask and mm_cid,
+ * aligned to the cacheline boundary required by the tree level items.
+ */
+static inline size_t get_rss_stat_items_offset(void)
+{
+ return PERCPU_COUNTER_TREE_ITEMS_ALIGN(cpumask_size() + mm_cid_size());
+}
+
struct mmu_gather;
extern void tlb_gather_mmu(struct mmu_gather *tlb, struct mm_struct *mm);
extern void tlb_gather_mmu_fullmm(struct mmu_gather *tlb, struct mm_struct *mm);
diff --git a/include/linux/percpu_counter_tree.h b/include/linux/percpu_counter_tree.h
new file mode 100644
index 0000000000000..d593806b4ab26
--- /dev/null
+++ b/include/linux/percpu_counter_tree.h
@@ -0,0 +1,391 @@
+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */
+/* SPDX-FileCopyrightText: 2025 Mathieu Desnoyers <mathieu.desnoyers@efficios.com> */
+
+#ifndef _PERCPU_COUNTER_TREE_H
+#define _PERCPU_COUNTER_TREE_H
+
+#include <linux/preempt.h>
+#include <linux/atomic.h>
+#include <linux/percpu.h>
+
+#ifdef CONFIG_SMP
+
+#if NR_CPUS == (1U << 0)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 0
+#elif NR_CPUS <= (1U << 1)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 1
+#elif NR_CPUS <= (1U << 2)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 3
+#elif NR_CPUS <= (1U << 3)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 7
+#elif NR_CPUS <= (1U << 4)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 7
+#elif NR_CPUS <= (1U << 5)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 11
+#elif NR_CPUS <= (1U << 6)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 21
+#elif NR_CPUS <= (1U << 7)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 21
+#elif NR_CPUS <= (1U << 8)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 37
+#elif NR_CPUS <= (1U << 9)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 73
+#elif NR_CPUS <= (1U << 10)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 149
+#elif NR_CPUS <= (1U << 11)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 293
+#elif NR_CPUS <= (1U << 12)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 585
+#elif NR_CPUS <= (1U << 13)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 1173
+#elif NR_CPUS <= (1U << 14)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 2341
+#elif NR_CPUS <= (1U << 15)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 4681
+#elif NR_CPUS <= (1U << 16)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 4681
+#elif NR_CPUS <= (1U << 17)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 8777
+#elif NR_CPUS <= (1U << 18)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 17481
+#elif NR_CPUS <= (1U << 19)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 34953
+#elif NR_CPUS <= (1U << 20)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 69905
+#else
+# error "Unsupported number of CPUs."
+#endif
+
+struct percpu_counter_tree_level_item {
+ atomic_long_t count; /*
+ * Count the number of carry for this tree item.
+ * The carry counter is kept at the order of the
+ * carry accounted for at this tree level.
+ */
+} ____cacheline_aligned_in_smp;
+
+#define PERCPU_COUNTER_TREE_ITEMS_STATIC_SIZE \
+ (PERCPU_COUNTER_TREE_STATIC_NR_ITEMS * sizeof(struct percpu_counter_tree_level_item))
+#define PERCPU_COUNTER_TREE_ITEMS_ALIGN(offset) \
+ ALIGN((offset), __alignof__(struct percpu_counter_tree_level_item))
+
+struct percpu_counter_tree {
+ /* Fast-path fields. */
+ unsigned long __percpu *level0; /* Pointer to per-CPU split counters (tree level 0). */
+ unsigned long level0_bit_mask; /* Bit mask to apply to detect carry propagation from tree level 0. */
+ union {
+ unsigned long *i; /* Approximate sum for single-CPU topology. */
+ atomic_long_t *a; /* Approximate sum for SMP topology. */
+ } approx_sum;
+ long bias; /* Bias to apply to counter precise and approximate values. */
+
+ /* Slow-path fields. */
+ struct percpu_counter_tree_level_item *items; /* Array of tree items for levels 1 to N. */
+ unsigned long batch_size; /*
+ * The batch size is the increment step at level 0 which
+ * triggers a carry propagation. The batch size is required
+ * to be greater than 1, and a power of 2.
+ */
+ /*
+ * The tree approximate sum is guaranteed to be within this accuracy range:
+ * (precise_sum - approx_accuracy_range.under) <= approx_sum <= (precise_sum + approx_accuracy_range.over).
+ * This accuracy is derived from the hardware topology and the tree batch_size.
+ * The "under" accuracy is larger than the "over" accuracy because the negative range of a
+ * two's complement signed integer is one unit larger than the positive range. This delta
+ * is summed for each tree item, which leads to a significantly larger "under" accuracy range
+ * compared to the "over" accuracy range. The accuracy range is a bound on a settled counter:
+ * one with no update in progress.
+ */
+ struct {
+ unsigned long under;
+ unsigned long over;
+ } approx_accuracy_range;
+};
+
+extern size_t __percpu_counter_tree_items_size __ro_after_init;
+
+/*
+ * percpu_counter_tree_items_size - Query the size required for counter tree items.
+ *
+ * Query the size of the memory area required to hold the counter tree
+ * items. This depends on the hardware topology and is invariant after
+ * boot.
+ *
+ * Return: Size required to hold tree items.
+ */
+static inline
+size_t percpu_counter_tree_items_size(void)
+{
+ WARN_ON_ONCE(__percpu_counter_tree_items_size == SIZE_MAX);
+ return __percpu_counter_tree_items_size;
+}
+
+int percpu_counter_tree_init_many(struct percpu_counter_tree *counters, struct percpu_counter_tree_level_item *items,
+ unsigned int nr_counters, unsigned long batch_size, gfp_t gfp_flags);
+int percpu_counter_tree_init(struct percpu_counter_tree *counter, struct percpu_counter_tree_level_item *items,
+ unsigned long batch_size, gfp_t gfp_flags);
+void percpu_counter_tree_destroy_many(struct percpu_counter_tree *counter, unsigned int nr_counters);
+void percpu_counter_tree_destroy(struct percpu_counter_tree *counter);
+void percpu_counter_tree_add(struct percpu_counter_tree *counter, long inc);
+long percpu_counter_tree_precise_sum(struct percpu_counter_tree *counter);
+int percpu_counter_tree_approximate_compare(struct percpu_counter_tree *a, struct percpu_counter_tree *b);
+int percpu_counter_tree_approximate_compare_value(struct percpu_counter_tree *counter, long v);
+int percpu_counter_tree_precise_compare(struct percpu_counter_tree *a, struct percpu_counter_tree *b);
+int percpu_counter_tree_precise_compare_value(struct percpu_counter_tree *counter, long v);
+void percpu_counter_tree_set(struct percpu_counter_tree *counter, long v);
+int percpu_counter_tree_subsystem_init(void);
+
+/**
+ * percpu_counter_tree_approximate_sum() - Return approximate counter sum.
+ * @counter: The counter to sum.
+ *
+ * Querying the approximate sum is fast, but it is only accurate within
+ * the bounds delimited by percpu_counter_tree_approximate_accuracy_range().
+ * This is meant to be used when speed is preferred over accuracy.
+ *
+ * Return: The current approximate counter sum.
+ */
+static inline
+long percpu_counter_tree_approximate_sum(struct percpu_counter_tree *counter)
+{
+ unsigned long v;
+
+ if (!counter->level0_bit_mask)
+ v = READ_ONCE(*counter->approx_sum.i);
+ else
+ v = atomic_long_read(counter->approx_sum.a);
+ return (long) (v + (unsigned long)READ_ONCE(counter->bias));
+}
+
+/**
+ * percpu_counter_tree_approximate_accuracy_range - Query the accuracy range for a counter tree.
+ * @counter: Counter to query.
+ * @under: Pointer to a variable to be incremented of the approximation
+ * accuracy range below the precise sum.
+ * @over: Pointer to a variable to be incremented of the approximation
+ * accuracy range above the precise sum.
+ *
+ * Query the accuracy range limits for the counter.
+ * Because of two's complement binary representation, the "under" range is typically
+ * slightly larger than the "over" range.
+ * Those values are derived from the hardware topology and the counter tree batch size.
+ * They are invariant for a given counter tree.
+ * The bounds describe a settled counter: one with no update in progress.
+ * An update in flight between its per-CPU increment and its carry propagation is not covered.
+ * Using this function should not be typically required, see the following functions instead:
+ * * percpu_counter_tree_approximate_compare(),
+ * * percpu_counter_tree_approximate_compare_value(),
+ * * percpu_counter_tree_precise_compare(),
+ * * percpu_counter_tree_precise_compare_value().
+ */
+static inline
+void percpu_counter_tree_approximate_accuracy_range(struct percpu_counter_tree *counter,
+ unsigned long *under, unsigned long *over)
+{
+ *under += counter->approx_accuracy_range.under;
+ *over += counter->approx_accuracy_range.over;
+}
+
+#else /* !CONFIG_SMP */
+
+#define PERCPU_COUNTER_TREE_ITEMS_STATIC_SIZE 0
+#define PERCPU_COUNTER_TREE_ITEMS_ALIGN(offset) (offset)
+
+struct percpu_counter_tree_level_item;
+
+struct percpu_counter_tree {
+ atomic_long_t count;
+};
+
+static inline
+size_t percpu_counter_tree_items_size(void)
+{
+ return 0;
+}
+
+static inline
+int percpu_counter_tree_init_many(struct percpu_counter_tree *counters, struct percpu_counter_tree_level_item *items,
+ unsigned int nr_counters, unsigned long batch_size, gfp_t gfp_flags)
+{
+ for (unsigned int i = 0; i < nr_counters; i++)
+ atomic_long_set(&counters[i].count, 0);
+ return 0;
+}
+
+static inline
+int percpu_counter_tree_init(struct percpu_counter_tree *counter, struct percpu_counter_tree_level_item *items,
+ unsigned long batch_size, gfp_t gfp_flags)
+{
+ return percpu_counter_tree_init_many(counter, items, 1, batch_size, gfp_flags);
+}
+
+static inline
+void percpu_counter_tree_destroy_many(struct percpu_counter_tree *counter, unsigned int nr_counters)
+{
+}
+
+static inline
+void percpu_counter_tree_destroy(struct percpu_counter_tree *counter)
+{
+}
+
+static inline
+long percpu_counter_tree_precise_sum(struct percpu_counter_tree *counter)
+{
+ return atomic_long_read(&counter->count);
+}
+
+static inline
+int percpu_counter_tree_precise_compare(struct percpu_counter_tree *a, struct percpu_counter_tree *b)
+{
+ long count_a = percpu_counter_tree_precise_sum(a),
+ count_b = percpu_counter_tree_precise_sum(b);
+
+ if (count_a == count_b)
+ return 0;
+ if (count_a < count_b)
+ return -1;
+ return 1;
+}
+
+static inline
+int percpu_counter_tree_precise_compare_value(struct percpu_counter_tree *counter, long v)
+{
+ long count = percpu_counter_tree_precise_sum(counter);
+
+ if (count == v)
+ return 0;
+ if (count < v)
+ return -1;
+ return 1;
+}
+
+static inline
+int percpu_counter_tree_approximate_compare(struct percpu_counter_tree *a, struct percpu_counter_tree *b)
+{
+ return percpu_counter_tree_precise_compare(a, b);
+}
+
+static inline
+int percpu_counter_tree_approximate_compare_value(struct percpu_counter_tree *counter, long v)
+{
+ return percpu_counter_tree_precise_compare_value(counter, v);
+}
+
+static inline
+void percpu_counter_tree_set(struct percpu_counter_tree *counter, long v)
+{
+ atomic_long_set(&counter->count, v);
+}
+
+static inline
+void percpu_counter_tree_approximate_accuracy_range(struct percpu_counter_tree *counter,
+ unsigned long *under, unsigned long *over)
+{
+}
+
+static inline
+void percpu_counter_tree_add(struct percpu_counter_tree *counter, long inc)
+{
+ atomic_long_add(inc, &counter->count);
+}
+
+static inline
+long percpu_counter_tree_approximate_sum(struct percpu_counter_tree *counter)
+{
+ return percpu_counter_tree_precise_sum(counter);
+}
+
+static inline
+int percpu_counter_tree_subsystem_init(void)
+{
+ return 0;
+}
+
+#endif /* CONFIG_SMP */
+
+/**
+ * percpu_counter_tree_approximate_sum_positive() - Return a positive approximate counter sum.
+ * @counter: The counter to sum.
+ *
+ * Return an approximate counter sum which is guaranteed to be greater
+ * or equal to 0.
+ *
+ * Return: The current positive approximate counter sum.
+ */
+static inline
+long percpu_counter_tree_approximate_sum_positive(struct percpu_counter_tree *counter)
+{
+ long v = percpu_counter_tree_approximate_sum(counter);
+ return v > 0 ? v : 0;
+}
+
+/**
+ * percpu_counter_tree_precise_sum_positive() - Return a positive precise counter sum.
+ * @counter: The counter to sum.
+ *
+ * Return a precise counter sum which is guaranteed to be greater
+ * or equal to 0.
+ *
+ * Return: The current positive precise counter sum.
+ */
+static inline
+long percpu_counter_tree_precise_sum_positive(struct percpu_counter_tree *counter)
+{
+ long v = percpu_counter_tree_precise_sum(counter);
+ return v > 0 ? v : 0;
+}
+
+/**
+ * percpu_counter_tree_approximate_min_max_range() - Return the approximation min and max precise values.
+ * @approx_sum: Approximated sum.
+ * @under: Tree accuracy range (under).
+ * @over: Tree accuracy range (over).
+ * @precise_min: Minimum possible value for precise sum (output).
+ * @precise_max: Maximum possible value for precise sum (output).
+ *
+ * Calculate the minimum and maximum precise values for a given approximation
+ * and (under, over) accuracy range. The accuracy range is a bound on a settled
+ * tree: one with no update in progress.
+ *
+ * The range of the approximation as a function of the precise sum is expressed as:
+ *
+ * approx_sum >= precise_sum - approx_accuracy_range.under
+ * approx_sum <= precise_sum + approx_accuracy_range.over
+ *
+ * Therefore, the range of the precise sum as a function of the approximation is expressed as:
+ *
+ * precise_sum <= approx_sum + approx_accuracy_range.under
+ * precise_sum >= approx_sum - approx_accuracy_range.over
+ */
+static inline
+void percpu_counter_tree_approximate_min_max_range(long approx_sum, unsigned long under, unsigned long over,
+ long *precise_min, long *precise_max)
+{
+ *precise_min = approx_sum - over;
+ *precise_max = approx_sum + under;
+}
+
+/**
+ * percpu_counter_tree_approximate_min_max() - Return the tree approximation, min and max possible precise values.
+ * @counter: The counter to sum.
+ * @approx_sum: Approximate sum (output).
+ * @precise_min: Minimum possible value for precise sum (output).
+ * @precise_max: Maximum possible value for precise sum (output).
+ *
+ * Return the approximate sum, minimum and maximum precise values for
+ * a counter.
+ */
+static inline
+void percpu_counter_tree_approximate_min_max(struct percpu_counter_tree *counter,
+ long *approx_sum, long *precise_min, long *precise_max)
+{
+ unsigned long under = 0, over = 0;
+ long v = percpu_counter_tree_approximate_sum(counter);
+
+ percpu_counter_tree_approximate_accuracy_range(counter, &under, &over);
+ percpu_counter_tree_approximate_min_max_range(v, under, over, precise_min, precise_max);
+ *approx_sum = v;
+}
+
+#endif /* _PERCPU_COUNTER_TREE_H */
diff --git a/include/trace/events/kmem.h b/include/trace/events/kmem.h
index cd7920c81f85b..290ccb9fd25dd 100644
--- a/include/trace/events/kmem.h
+++ b/include/trace/events/kmem.h
@@ -448,7 +448,7 @@ TRACE_EVENT(rss_stat,
*/
__entry->curr = current->mm == mm && !(current->flags & PF_KTHREAD);
__entry->member = member;
- __entry->size = (percpu_counter_sum_positive(&mm->rss_stat[member])
+ __entry->size = (percpu_counter_tree_approximate_sum_positive(&mm->rss_stat[member])
<< PAGE_SHIFT);
),
diff --git a/init/main.c b/init/main.c
index 2613d3f9b3ce9..845cb57e90965 100644
--- a/init/main.c
+++ b/init/main.c
@@ -108,6 +108,7 @@
#include <linux/time_namespace.h>
#include <linux/unaligned.h>
#include <linux/vdso_datastore.h>
+#include <linux/percpu_counter_tree.h>
#include <net/net_namespace.h>
#include <asm/io.h>
@@ -1010,6 +1011,7 @@ void start_kernel(void)
setup_boot_config();
setup_command_line(command_line);
setup_nr_cpu_ids();
+ percpu_counter_tree_subsystem_init();
setup_per_cpu_areas();
smp_prepare_boot_cpu(); /* arch-specific boot-cpu hooks */
early_numa_node_init();
diff --git a/kernel/fork.c b/kernel/fork.c
index 416758c8a3d43..db398b6c52886 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -137,6 +137,11 @@
*/
#define MAX_THREADS FUTEX_TID_MASK
+/*
+ * Batch size of rss stat approximation
+ */
+#define RSS_STAT_BATCH_SIZE 32
+
/*
* Protected counters by write_lock_irq(&tasklist_lock)
*/
@@ -633,14 +638,12 @@ static void check_mm(struct mm_struct *mm)
"Please make sure 'struct resident_page_types[]' is updated as well");
for (i = 0; i < NR_MM_COUNTERS; i++) {
- long x = percpu_counter_sum(&mm->rss_stat[i]);
-
- if (unlikely(x)) {
+ if (unlikely(percpu_counter_tree_precise_compare_value(&mm->rss_stat[i], 0) != 0))
pr_alert("BUG: Bad rss-counter state mm:%p type:%s val:%ld Comm:%s Pid:%d\n",
- mm, resident_page_types[i], x,
+ mm, resident_page_types[i],
+ percpu_counter_tree_precise_sum(&mm->rss_stat[i]),
current->comm,
task_pid_nr(current));
- }
}
if (mm_pgtables_bytes(mm))
@@ -738,7 +741,7 @@ void __mmdrop(struct mm_struct *mm)
check_mm(mm);
mm_pasid_drop(mm);
mm_destroy_cid(mm);
- percpu_counter_destroy_many(mm->rss_stat, NR_MM_COUNTERS);
+ percpu_counter_tree_destroy_many(mm->rss_stat, NR_MM_COUNTERS);
free_mm(mm);
}
@@ -1144,8 +1147,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p)
if (mm_alloc_sched(mm))
goto fail_sched;
- if (percpu_counter_init_many(mm->rss_stat, 0, GFP_KERNEL_ACCOUNT,
- NR_MM_COUNTERS))
+ if (percpu_counter_tree_init_many(mm->rss_stat, get_rss_stat_items(mm),
+ NR_MM_COUNTERS, RSS_STAT_BATCH_SIZE,
+ GFP_KERNEL_ACCOUNT))
goto fail_pcpu;
lru_gen_init_mm(mm);
@@ -3122,7 +3126,9 @@ void __init mm_cache_init(void)
* dynamically sized based on the maximum CPU number this system
* can have, taking hotplug into account (nr_cpu_ids).
*/
- mm_size = sizeof(struct mm_struct) + cpumask_size() + mm_cid_size();
+ mm_size = sizeof(struct mm_struct) +
+ PERCPU_COUNTER_TREE_ITEMS_ALIGN(cpumask_size() + mm_cid_size()) +
+ get_rss_stat_items_size();
mm_cachep = kmem_cache_create_usercopy("mm_struct",
mm_size, ARCH_MIN_MMSTRUCT_ALIGN,
diff --git a/lib/Kconfig b/lib/Kconfig
index 4e6b34c3346d5..c0524258b2058 100644
--- a/lib/Kconfig
+++ b/lib/Kconfig
@@ -41,6 +41,18 @@ config PACKING_KUNIT_TEST
When in doubt, say N.
+config PERCPU_COUNTER_TREE_KUNIT_TEST
+ tristate "KUnit tests for hierarchical per-cpu counters" if !KUNIT_ALL_TESTS
+ depends on KUNIT
+ default KUNIT_ALL_TESTS
+ help
+ This builds Kunit tests for the hierarchical per-cpu counters.
+
+ For more information on KUnit and unit tests in general,
+ please refer to the KUnit documentation in Documentation/dev-tools/kunit/.
+
+ When in doubt, say N.
+
config BITREVERSE
tristate
select GENERIC_BITREVERSE if !HAVE_ARCH_BITREVERSE
diff --git a/lib/Makefile b/lib/Makefile
index dfab958327c5c..f1d0cb831630b 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -182,6 +182,7 @@ obj-$(CONFIG_TEXTSEARCH_KMP) += ts_kmp.o
obj-$(CONFIG_TEXTSEARCH_BM) += ts_bm.o
obj-$(CONFIG_TEXTSEARCH_FSM) += ts_fsm.o
obj-$(CONFIG_SMP) += percpu_counter.o
+obj-$(CONFIG_SMP) += percpu_counter_tree.o
obj-$(CONFIG_AUDIT_GENERIC) += audit.o
obj-$(CONFIG_AUDIT_COMPAT_GENERIC) += compat_audit.o
diff --git a/lib/percpu_counter_tree.c b/lib/percpu_counter_tree.c
new file mode 100644
index 0000000000000..26bfe183a882d
--- /dev/null
+++ b/lib/percpu_counter_tree.c
@@ -0,0 +1,747 @@
+// SPDX-License-Identifier: GPL-2.0+ OR MIT
+// SPDX-FileCopyrightText: 2025 Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
+
+/*
+ * Split Counters With Tree Approximation Propagation
+ *
+ * * Propagation diagram when reaching batch size thresholds (± batch size):
+ *
+ * Example diagram for 8 CPUs:
+ *
+ * log2(8) = 3 levels
+ *
+ * At each level, each pair propagates its values to the next level when
+ * reaching the batch size thresholds.
+ *
+ * Counters at levels 0, 1, 2 can be kept on a single byte ([-128 .. +127] range),
+ * although it may be relevant to keep them on "long" counters for
+ * simplicity. (complexity vs memory footprint tradeoff)
+ *
+ * Counter at level 3 can be kept on a "long" counter.
+ *
+ * Level 0: 0 1 2 3 4 5 6 7
+ * | / | / | / | /
+ * | / | / | / | /
+ * | / | / | / | /
+ * Level 1: 0 1 2 3
+ * | / | /
+ * | / | /
+ * | / | /
+ * Level 2: 0 1
+ * | /
+ * | /
+ * | /
+ * Level 3: 0
+ *
+ * * Approximation accuracy:
+ *
+ * BATCH(level N): Level N batch size.
+ *
+ * Example for BATCH(level 0) = 32.
+ *
+ * BATCH(level 0) = 32
+ * BATCH(level 1) = 64
+ * BATCH(level 2) = 128
+ * BATCH(level N) = BATCH(level 0) * 2^N
+ *
+ * per-counter global
+ * accuracy accuracy
+ * Level 0: [ -32 .. +31] ±256 (8 * 32)
+ * Level 1: [ -64 .. +63] ±256 (4 * 64)
+ * Level 2: [-128 .. +127] ±256 (2 * 128)
+ * Total: ------ ±768 (log2(nr_cpu_ids) * BATCH(level 0) * nr_cpu_ids)
+ *
+ * Note that the global accuracy can be calculated more precisely
+ * by taking into account that the positive accuracy range is
+ * 31 rather than 32.
+ *
+ * -----
+ *
+ * Approximate Sum Carry Propagation
+ *
+ * Let's define a number of counter bits for each level, e.g.:
+ *
+ * log2(BATCH(level 0)) = log2(32) = 5
+ * Let's assume, for this example, a 32-bit architecture (sizeof(long) == 4).
+ *
+ * nr_bit value_mask range
+ * Level 0: 5 bits v 0 .. +31
+ * Level 1: 1 bit (v & ~((1UL << 5) - 1)) 0 .. +63
+ * Level 2: 1 bit (v & ~((1UL << 6) - 1)) 0 .. +127
+ * Level 3: 25 bits (v & ~((1UL << 7) - 1)) 0 .. 2^32-1
+ *
+ * Note: Use a "long" per-cpu counter at level 0 to allow precise sum.
+ *
+ * Note: Use cacheline aligned counters at levels above 0 to prevent false sharing.
+ * If memory footprint is an issue, a specialized allocator could be used
+ * to eliminate padding.
+ *
+ * Example with expanded values:
+ *
+ * counter_add(counter, inc):
+ *
+ * if (!inc)
+ * return;
+ *
+ * res = percpu_add_return(counter @ Level 0, inc);
+ * orig = res - inc;
+ * if (inc < 0) {
+ * inc = -(-inc & ~0b00011111); // Clear used bits
+ * // xor bit 5: underflow
+ * if ((inc ^ orig ^ res) & 0b00100000)
+ * inc -= 0b00100000;
+ * } else {
+ * inc &= ~0b00011111; // Clear used bits
+ * // xor bit 5: overflow
+ * if ((inc ^ orig ^ res) & 0b00100000)
+ * inc += 0b00100000;
+ * }
+ * if (!inc)
+ * return;
+ *
+ * res = atomic_long_add_return(counter @ Level 1, inc);
+ * orig = res - inc;
+ * if (inc < 0) {
+ * inc = -(-inc & ~0b00111111); // Clear used bits
+ * // xor bit 6: underflow
+ * if ((inc ^ orig ^ res) & 0b01000000)
+ * inc -= 0b01000000;
+ * } else {
+ * inc &= ~0b00111111; // Clear used bits
+ * // xor bit 6: overflow
+ * if ((inc ^ orig ^ res) & 0b01000000)
+ * inc += 0b01000000;
+ * }
+ * if (!inc)
+ * return;
+ *
+ * res = atomic_long_add_return(counter @ Level 2, inc);
+ * orig = res - inc;
+ * if (inc < 0) {
+ * inc = -(-inc & ~0b01111111); // Clear used bits
+ * // xor bit 7: underflow
+ * if ((inc ^ orig ^ res) & 0b10000000)
+ * inc -= 0b10000000;
+ * } else {
+ * inc &= ~0b01111111; // Clear used bits
+ * // xor bit 7: overflow
+ * if ((inc ^ orig ^ res) & 0b10000000)
+ * inc += 0b10000000;
+ * }
+ * if (!inc)
+ * return;
+ *
+ * atomic_long_add(counter @ Level 3, inc);
+ */
+
+#include <linux/percpu_counter_tree.h>
+#include <linux/cpumask.h>
+#include <linux/atomic.h>
+#include <linux/export.h>
+#include <linux/percpu.h>
+#include <linux/errno.h>
+#include <linux/slab.h>
+#include <linux/math.h>
+
+#define MAX_NR_LEVELS 5
+
+/*
+ * The counter configuration is selected at boot time based on the
+ * hardware topology.
+ */
+struct counter_config {
+ unsigned int nr_items; /*
+ * nr_items is the number of items in the tree for levels 1
+ * up to and including the final level (approximate sum).
+ * It excludes the level 0 per-CPU counters.
+ */
+ unsigned char nr_levels; /*
+ * nr_levels is the number of hierarchical counter tree levels.
+ * It excludes the final level (approximate sum).
+ */
+ unsigned char n_arity_order[MAX_NR_LEVELS]; /*
+ * n-arity of tree nodes for each level from
+ * 0 to (nr_levels - 1).
+ */
+};
+
+static const struct counter_config per_nr_cpu_order_config[] = {
+ [0] = { .nr_items = 0, .nr_levels = 0, .n_arity_order = { 0 } },
+ [1] = { .nr_items = 1, .nr_levels = 1, .n_arity_order = { 1 } },
+ [2] = { .nr_items = 3, .nr_levels = 2, .n_arity_order = { 1, 1 } },
+ [3] = { .nr_items = 7, .nr_levels = 3, .n_arity_order = { 1, 1, 1 } },
+ [4] = { .nr_items = 7, .nr_levels = 3, .n_arity_order = { 2, 1, 1 } },
+ [5] = { .nr_items = 11, .nr_levels = 3, .n_arity_order = { 2, 2, 1 } },
+ [6] = { .nr_items = 21, .nr_levels = 3, .n_arity_order = { 2, 2, 2 } },
+ [7] = { .nr_items = 21, .nr_levels = 3, .n_arity_order = { 3, 2, 2 } },
+ [8] = { .nr_items = 37, .nr_levels = 3, .n_arity_order = { 3, 3, 2 } },
+ [9] = { .nr_items = 73, .nr_levels = 3, .n_arity_order = { 3, 3, 3 } },
+ [10] = { .nr_items = 149, .nr_levels = 4, .n_arity_order = { 3, 3, 2, 2 } },
+ [11] = { .nr_items = 293, .nr_levels = 4, .n_arity_order = { 3, 3, 3, 2 } },
+ [12] = { .nr_items = 585, .nr_levels = 4, .n_arity_order = { 3, 3, 3, 3 } },
+ [13] = { .nr_items = 1173, .nr_levels = 5, .n_arity_order = { 3, 3, 3, 2, 2 } },
+ [14] = { .nr_items = 2341, .nr_levels = 5, .n_arity_order = { 3, 3, 3, 3, 2 } },
+ [15] = { .nr_items = 4681, .nr_levels = 5, .n_arity_order = { 3, 3, 3, 3, 3 } },
+ [16] = { .nr_items = 4681, .nr_levels = 5, .n_arity_order = { 4, 3, 3, 3, 3 } },
+ [17] = { .nr_items = 8777, .nr_levels = 5, .n_arity_order = { 4, 4, 3, 3, 3 } },
+ [18] = { .nr_items = 17481, .nr_levels = 5, .n_arity_order = { 4, 4, 4, 3, 3 } },
+ [19] = { .nr_items = 34953, .nr_levels = 5, .n_arity_order = { 4, 4, 4, 4, 3 } },
+ [20] = { .nr_items = 69905, .nr_levels = 5, .n_arity_order = { 4, 4, 4, 4, 4 } },
+};
+
+static const struct counter_config *counter_config; /* Hierarchical counter configuration for the hardware topology. */
+static unsigned int nr_cpus_order; /* Order of nr_cpu_ids. */
+static unsigned long accuracy_multiplier; /* Calculate accuracy for a given batch size (multiplication factor). */
+
+static
+int __percpu_counter_tree_init(struct percpu_counter_tree *counter,
+ unsigned long batch_size, gfp_t gfp_flags,
+ unsigned long __percpu *level0,
+ struct percpu_counter_tree_level_item *items)
+{
+ /* Batch size must be greater than 1, and a power of 2. */
+ if (WARN_ON(batch_size <= 1 || (batch_size & (batch_size - 1))))
+ return -EINVAL;
+ counter->batch_size = batch_size;
+ counter->bias = 0;
+ counter->level0 = level0;
+ counter->items = items;
+ if (!nr_cpus_order) {
+ counter->approx_sum.i = per_cpu_ptr(counter->level0, 0);
+ counter->level0_bit_mask = 0;
+ } else {
+ counter->approx_sum.a = &counter->items[counter_config->nr_items - 1].count;
+ counter->level0_bit_mask = 1UL << get_count_order(batch_size);
+ }
+ /*
+ * Each tree item signed integer has a negative range which is
+ * one unit greater than the positive range.
+ */
+ counter->approx_accuracy_range.under = batch_size * accuracy_multiplier;
+ counter->approx_accuracy_range.over = (batch_size - 1) * accuracy_multiplier;
+ return 0;
+}
+
+/**
+ * percpu_counter_tree_init_many() - Initialize many per-CPU counter trees.
+ * @counters: An array of @nr_counters counters to initialize.
+ * Their memory is provided by the caller.
+ * @items: Pointer to memory area where to store tree items.
+ * This memory is provided by the caller.
+ * Its size needs to be at least @nr_counters * percpu_counter_tree_items_size().
+ * @nr_counters: The number of counter trees to initialize
+ * @batch_size: The batch size is the increment step at level 0 which triggers a
+ * carry propagation.
+ * The batch size is required to be greater than 1, and a power of 2.
+ * @gfp_flags: gfp flags to pass to the per-CPU allocator.
+ *
+ * Initialize many per-CPU counter trees using a single per-CPU
+ * allocator invocation for @nr_counters counters.
+ *
+ * Return:
+ * * %0: Success
+ * * %-EINVAL: - Invalid @batch_size argument
+ * * %-ENOMEM: - Out of memory
+ */
+int percpu_counter_tree_init_many(struct percpu_counter_tree *counters, struct percpu_counter_tree_level_item *items,
+ unsigned int nr_counters, unsigned long batch_size, gfp_t gfp_flags)
+{
+ void __percpu *level0, *level0_iter;
+ size_t counter_size = sizeof(*counters->level0),
+ items_size = percpu_counter_tree_items_size();
+ void *items_iter;
+ unsigned int i;
+ int ret;
+
+ memset(items, 0, items_size * nr_counters);
+ level0 = __alloc_percpu_gfp(nr_counters * counter_size,
+ __alignof__(*counters->level0), gfp_flags);
+ if (!level0)
+ return -ENOMEM;
+ level0_iter = level0;
+ items_iter = items;
+ for (i = 0; i < nr_counters; i++) {
+ ret = __percpu_counter_tree_init(&counters[i], batch_size, gfp_flags, level0_iter, items_iter);
+ if (ret)
+ goto free_level0;
+ level0_iter += counter_size;
+ items_iter += items_size;
+ }
+ return 0;
+
+free_level0:
+ free_percpu(level0);
+ return ret;
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_init_many);
+
+/**
+ * percpu_counter_tree_init() - Initialize one per-CPU counter tree.
+ * @counter: Counter to initialize.
+ * Its memory is provided by the caller.
+ * @items: Pointer to memory area where to store tree items.
+ * This memory is provided by the caller.
+ * Its size needs to be at least percpu_counter_tree_items_size().
+ * @batch_size: The batch size is the increment step at level 0 which triggers a
+ * carry propagation.
+ * The batch size is required to be greater than 1, and a power of 2.
+ * @gfp_flags: gfp flags to pass to the per-CPU allocator.
+ *
+ * Initialize one per-CPU counter tree.
+ *
+ * Return:
+ * * %0: Success
+ * * %-EINVAL: - Invalid @batch_size argument
+ * * %-ENOMEM: - Out of memory
+ */
+int percpu_counter_tree_init(struct percpu_counter_tree *counter, struct percpu_counter_tree_level_item *items,
+ unsigned long batch_size, gfp_t gfp_flags)
+{
+ return percpu_counter_tree_init_many(counter, items, 1, batch_size, gfp_flags);
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_init);
+
+/**
+ * percpu_counter_tree_destroy_many() - Destroy many per-CPU counter trees.
+ * @counters: Array of counters trees to destroy.
+ * @nr_counters: The number of counter trees to destroy.
+ *
+ * Release internal resources allocated for @nr_counters per-CPU counter trees.
+ */
+
+void percpu_counter_tree_destroy_many(struct percpu_counter_tree *counters, unsigned int nr_counters)
+{
+ free_percpu(counters->level0);
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_destroy_many);
+
+/**
+ * percpu_counter_tree_destroy() - Destroy one per-CPU counter tree.
+ * @counter: Counter to destroy.
+ *
+ * Release internal resources allocated for one per-CPU counter tree.
+ */
+void percpu_counter_tree_destroy(struct percpu_counter_tree *counter)
+{
+ return percpu_counter_tree_destroy_many(counter, 1);
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_destroy);
+
+static
+long percpu_counter_tree_carry(long orig, long res, long inc, unsigned long bit_mask)
+{
+ if (inc < 0) {
+ inc = -(-inc & ~(bit_mask - 1));
+ /*
+ * xor bit_mask: underflow.
+ *
+ * If inc has bit set, decrement an additional bit if
+ * there is _no_ bit transition between orig and res.
+ * Else, inc has bit cleared, decrement an additional
+ * bit if there is a bit transition between orig and
+ * res.
+ */
+ if ((inc ^ orig ^ res) & bit_mask)
+ inc -= bit_mask;
+ } else {
+ inc &= ~(bit_mask - 1);
+ /*
+ * xor bit_mask: overflow.
+ *
+ * If inc has bit set, increment an additional bit if
+ * there is _no_ bit transition between orig and res.
+ * Else, inc has bit cleared, increment an additional
+ * bit if there is a bit transition between orig and
+ * res.
+ */
+ if ((inc ^ orig ^ res) & bit_mask)
+ inc += bit_mask;
+ }
+ return inc;
+}
+
+/*
+ * It does not matter through which path the carry propagates up the
+ * tree, therefore there is no need to disable preemption because the
+ * cpu number is only used to favor cache locality.
+ */
+static
+void percpu_counter_tree_add_slowpath(struct percpu_counter_tree *counter, long inc)
+{
+ unsigned int level_items, nr_levels = counter_config->nr_levels,
+ level, n_arity_order;
+ unsigned long bit_mask;
+ struct percpu_counter_tree_level_item *item = counter->items;
+ unsigned int cpu = raw_smp_processor_id();
+
+ WARN_ON_ONCE(!nr_cpus_order); /* Should never be called for 1 cpu. */
+
+ n_arity_order = counter_config->n_arity_order[0];
+ bit_mask = counter->level0_bit_mask << n_arity_order;
+ level_items = 1U << (nr_cpus_order - n_arity_order);
+
+ for (level = 1; level < nr_levels; level++) {
+ /*
+ * For the purpose of carry propagation, the
+ * intermediate level counters only need to keep track
+ * of the bits relevant for carry propagation. We
+ * therefore don't care about higher order bits.
+ * Note that this optimization is unwanted if the
+ * intended use is to track counters within intermediate
+ * levels of the topology.
+ */
+ if (abs(inc) & (bit_mask - 1)) {
+ atomic_long_t *count = &item[cpu & (level_items - 1)].count;
+ unsigned long orig, res;
+
+ res = atomic_long_add_return_relaxed(inc, count);
+ orig = res - inc;
+ inc = percpu_counter_tree_carry(orig, res, inc, bit_mask);
+ if (likely(!inc))
+ return;
+ }
+ item += level_items;
+ n_arity_order = counter_config->n_arity_order[level];
+ level_items >>= n_arity_order;
+ bit_mask <<= n_arity_order;
+ }
+ atomic_long_add(inc, counter->approx_sum.a);
+}
+
+/**
+ * percpu_counter_tree_add() - Add to a per-CPU counter tree.
+ * @counter: Counter added to.
+ * @inc: Increment value (either positive or negative).
+ *
+ * Add @inc to a per-CPU counter tree. This is a fast-path which will
+ * typically increment per-CPU counters as long as there is no carry
+ * greater or equal to the counter tree batch size.
+ */
+void percpu_counter_tree_add(struct percpu_counter_tree *counter, long inc)
+{
+ unsigned long bit_mask = counter->level0_bit_mask, orig, res;
+
+ res = this_cpu_add_return(*counter->level0, inc);
+ orig = res - inc;
+ inc = percpu_counter_tree_carry(orig, res, inc, bit_mask);
+ if (likely(!inc))
+ return;
+ percpu_counter_tree_add_slowpath(counter, inc);
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_add);
+
+static
+long percpu_counter_tree_precise_sum_unbiased(struct percpu_counter_tree *counter)
+{
+ unsigned long sum = 0;
+ int cpu;
+
+ for_each_possible_cpu(cpu)
+ sum += *per_cpu_ptr(counter->level0, cpu);
+ return (long) sum;
+}
+
+/**
+ * percpu_counter_tree_precise_sum() - Return precise counter sum.
+ * @counter: The counter to sum.
+ *
+ * Querying the precise sum is relatively expensive because it needs to
+ * iterate over all CPUs.
+ * This is meant to be used when accuracy is preferred over speed.
+ *
+ * Return: The current precise counter sum.
+ */
+long percpu_counter_tree_precise_sum(struct percpu_counter_tree *counter)
+{
+ return percpu_counter_tree_precise_sum_unbiased(counter) + READ_ONCE(counter->bias);
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_precise_sum);
+
+/*
+ * The comparison between counters is on the wrapped difference. It is valid
+ * only when the two counters differ by less than LONG_MAX, and results are
+ * undefined beyond that.
+ */
+static inline
+long wrapped_delta(long a, long b)
+{
+ return (long)((unsigned long)a - (unsigned long)b);
+}
+
+/*
+ * Each counter's approximation lies within [precise - under, precise + over]:
+ *
+ * approx_a range
+ * <-----------+----------->
+ * under_a | over_a
+ * precise_a
+ * | approx_b range
+ * | <-----------+----------->
+ * | under_b | over_b
+ * | precise_b
+ * | |
+ * |<------------ gap ------------->|
+ * ----------------------+--------------------------------+-------------> values
+ *
+ * The two ranges are disjoint, and the comparison can return a definitive
+ * answer, only once the gap exceeds the two facing half-widths:
+ *
+ * a below b: gap > over_a + under_b (a's upper edge, b's lower edge)
+ * a above b: gap > under_a + over_b (a's lower edge, b's upper edge)
+ *
+ * When comparing against a plain value, the value has zero width, so the
+ * margins reduce to that single counter's under and over.
+ *
+ * accuracy_neg is used when delta < 0, and accuracy_pos is used when delta >= 0.
+ */
+static
+int compare_delta(long delta, unsigned long accuracy_neg, unsigned long accuracy_pos)
+{
+ if (delta >= 0) {
+ if (delta <= accuracy_pos)
+ return 0;
+ else
+ return 1;
+ } else {
+ if (-delta <= accuracy_neg)
+ return 0;
+ else
+ return -1;
+ }
+}
+
+/**
+ * percpu_counter_tree_approximate_compare - Approximated comparison of two counter trees.
+ * @a: First counter to compare.
+ * @b: Second counter to compare.
+ *
+ * Evaluate an approximate comparison of two counter trees. This approximation
+ * comparison is fast, and provides an accurate answer if the counters are
+ * found to be either less than or greater than the other. However, if the
+ * approximated comparison returns 0, the counters respective sums are found to
+ * be within the two counters accuracy range. The two counters are read
+ * independently; the result is not an atomic snapshot of both.
+ * The comparison between counters is on the wrapped difference. It is valid
+ * only when the two counters differ by less than LONG_MAX, and results are
+ * undefined beyond that.
+ *
+ * Return:
+ * * %0 - Counters @a and @b do not differ by more than the sum of their respective
+ * accuracy ranges.
+ * * %-1 - Counter @a less than counter @b.
+ * * %1 - Counter @a is greater than counter @b.
+ */
+int percpu_counter_tree_approximate_compare(struct percpu_counter_tree *a, struct percpu_counter_tree *b)
+{
+ /* See the range geometry above compare_delta(). */
+ return compare_delta(wrapped_delta(percpu_counter_tree_approximate_sum(a),
+ percpu_counter_tree_approximate_sum(b)),
+ a->approx_accuracy_range.over + b->approx_accuracy_range.under,
+ a->approx_accuracy_range.under + b->approx_accuracy_range.over);
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_approximate_compare);
+
+/**
+ * percpu_counter_tree_approximate_compare_value - Approximated comparison of a counter tree against a given value.
+ * @counter: Counter to compare.
+ * @v: Value to compare.
+ *
+ * Evaluate an approximate comparison of a counter tree against a given value.
+ * This approximation comparison is fast, and provides an accurate
+ * answer if the counter is found to be either less than or greater
+ * than the value. However, if the approximated comparison returns
+ * 0, the value is within the counter accuracy range.
+ * The comparison between counters is on the wrapped difference. It is valid
+ * only when @v and the counter differ by less than LONG_MAX, and results are
+ * undefined beyond that.
+ *
+ * Return:
+ * * %0 - The value @v is within the accuracy range of the counter.
+ * * %-1 - The value @v is less than the counter.
+ * * %1 - The value @v is greater than the counter.
+ */
+int percpu_counter_tree_approximate_compare_value(struct percpu_counter_tree *counter, long v)
+{
+ return compare_delta(wrapped_delta(v, percpu_counter_tree_approximate_sum(counter)),
+ counter->approx_accuracy_range.under,
+ counter->approx_accuracy_range.over);
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_approximate_compare_value);
+
+/**
+ * percpu_counter_tree_precise_compare - Precise comparison of two counter trees.
+ * @a: First counter to compare.
+ * @b: Second counter to compare.
+ *
+ * Evaluate a precise comparison of two counter trees. As an optimization, it
+ * uses the approximate counter comparison to quickly compare counters which
+ * are far apart. Only cases where counter sums are within the accuracy range
+ * require precise counter sums. The two counters are read independently; the
+ * result is not an atomic snapshot of both.
+ * The comparison between counters is on the wrapped difference. It is valid
+ * only when the two counters differ by less than LONG_MAX, and results are
+ * undefined beyond that.
+ *
+ * Return:
+ * * %0 - Counters are equal.
+ * * %-1 - Counter @a less than counter @b.
+ * * %1 - Counter @a is greater than counter @b.
+ */
+int percpu_counter_tree_precise_compare(struct percpu_counter_tree *a, struct percpu_counter_tree *b)
+{
+ long count_a = percpu_counter_tree_approximate_sum(a),
+ count_b = percpu_counter_tree_approximate_sum(b);
+ unsigned long accuracy_a, accuracy_b;
+ long delta = wrapped_delta(count_a, count_b);
+ int res;
+
+ /* See the range geometry above compare_delta(). */
+ res = compare_delta(delta,
+ a->approx_accuracy_range.over + b->approx_accuracy_range.under,
+ a->approx_accuracy_range.under + b->approx_accuracy_range.over);
+ /* The values are distanced enough for an accurate approximated comparison. */
+ if (res)
+ return res;
+
+ /*
+ * The approximated comparison is within the accuracy range, therefore at least one
+ * precise sum is needed. Sum the counter which has the largest accuracy first.
+ */
+ if (delta >= 0) {
+ accuracy_a = a->approx_accuracy_range.under;
+ accuracy_b = b->approx_accuracy_range.over;
+ } else {
+ accuracy_a = a->approx_accuracy_range.over;
+ accuracy_b = b->approx_accuracy_range.under;
+ }
+ if (accuracy_b < accuracy_a) {
+ count_a = percpu_counter_tree_precise_sum(a);
+ res = compare_delta(wrapped_delta(count_a, count_b),
+ b->approx_accuracy_range.under,
+ b->approx_accuracy_range.over);
+ if (res)
+ return res;
+ /* Precise sum of second counter is required. */
+ count_b = percpu_counter_tree_precise_sum(b);
+ } else {
+ count_b = percpu_counter_tree_precise_sum(b);
+ res = compare_delta(wrapped_delta(count_a, count_b),
+ a->approx_accuracy_range.over,
+ a->approx_accuracy_range.under);
+ if (res)
+ return res;
+ /* Precise sum of second counter is required. */
+ count_a = percpu_counter_tree_precise_sum(a);
+ }
+ delta = wrapped_delta(count_a, count_b);
+ if (delta < 0)
+ return -1;
+ if (delta > 0)
+ return 1;
+ return 0;
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_precise_compare);
+
+/**
+ * percpu_counter_tree_precise_compare_value - Precise comparison of a counter tree against a given value.
+ * @counter: Counter to compare.
+ * @v: Value to compare.
+ *
+ * Evaluate a precise comparison of a counter tree against a given value. As
+ * an optimization, it uses the approximate counter comparison to quickly
+ * identify whether the counter and value are far apart. Only cases where the
+ * value is within the counter accuracy range require a precise counter sum.
+ * The comparison between counters is on the wrapped difference. It is valid
+ * only when @v and the counter differ by less than LONG_MAX, and results are
+ * undefined beyond that.
+ *
+ * Return:
+ * * %0 - The value @v is equal to the counter.
+ * * %-1 - The value @v is less than the counter.
+ * * %1 - The value @v is greater than the counter.
+ */
+int percpu_counter_tree_precise_compare_value(struct percpu_counter_tree *counter, long v)
+{
+ long count = percpu_counter_tree_approximate_sum(counter), delta;
+ int res;
+
+ res = compare_delta(wrapped_delta(v, count),
+ counter->approx_accuracy_range.under,
+ counter->approx_accuracy_range.over);
+ /* The values are distanced enough for an accurate approximated comparison. */
+ if (res)
+ return res;
+
+ /* Precise sum is required. */
+ count = percpu_counter_tree_precise_sum(counter);
+ delta = wrapped_delta(v, count);
+ if (delta < 0)
+ return -1;
+ if (delta > 0)
+ return 1;
+ return 0;
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_precise_compare_value);
+
+static
+void percpu_counter_tree_set_bias(struct percpu_counter_tree *counter, long bias)
+{
+ WRITE_ONCE(counter->bias, bias);
+}
+
+/**
+ * percpu_counter_tree_set - Set the counter tree sum to a given value.
+ * @counter: Counter to set.
+ * @v: Value to set.
+ *
+ * Set the counter sum to a given value. It can be useful for instance
+ * to reset the counter sum to 0. Note that even after setting the
+ * counter sum to a given value, the counter sum approximation can
+ * return any value within the accuracy range around that value.
+ */
+void percpu_counter_tree_set(struct percpu_counter_tree *counter, long v)
+{
+ percpu_counter_tree_set_bias(counter,
+ v - percpu_counter_tree_precise_sum_unbiased(counter));
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_set);
+
+/* Initialize to SIZE_MAX to catch early boot misuses. */
+size_t __percpu_counter_tree_items_size __ro_after_init = SIZE_MAX;
+EXPORT_SYMBOL_GPL(__percpu_counter_tree_items_size);
+
+static void __init calculate_accuracy_topology(void)
+{
+ unsigned int nr_levels = counter_config->nr_levels, level;
+ unsigned int level_items = 1U << nr_cpus_order;
+ unsigned long batch_size = 1;
+
+ for (level = 0; level < nr_levels; level++) {
+ unsigned int n_arity_order = counter_config->n_arity_order[level];
+
+ /*
+ * The accuracy multiplier is derived from a batch size of 1
+ * to speed up calculating the accuracy at tree initialization.
+ */
+ accuracy_multiplier += batch_size * level_items;
+ batch_size <<= n_arity_order;
+ level_items >>= n_arity_order;
+ }
+}
+
+int __init percpu_counter_tree_subsystem_init(void)
+{
+ nr_cpus_order = get_count_order(nr_cpu_ids);
+ if (WARN_ON_ONCE(nr_cpus_order >= ARRAY_SIZE(per_nr_cpu_order_config))) {
+ printk(KERN_ERR "Unsupported number of CPUs (%u)\n", nr_cpu_ids);
+ return -1;
+ }
+ counter_config = &per_nr_cpu_order_config[nr_cpus_order];
+ if (!nr_cpus_order)
+ __percpu_counter_tree_items_size = 0;
+ else
+ __percpu_counter_tree_items_size = counter_config->nr_items *
+ sizeof(struct percpu_counter_tree_level_item);
+ calculate_accuracy_topology();
+ return 0;
+}
diff --git a/lib/tests/Makefile b/lib/tests/Makefile
index 3cac3b63a7522..051e73705977b 100644
--- a/lib/tests/Makefile
+++ b/lib/tests/Makefile
@@ -58,4 +58,6 @@ obj-$(CONFIG_UTIL_MACROS_KUNIT) += util_macros_kunit.o
obj-$(CONFIG_RATELIMIT_KUNIT_TEST) += test_ratelimit.o
obj-$(CONFIG_UUID_KUNIT_TEST) += uuid_kunit.o
+obj-$(CONFIG_PERCPU_COUNTER_TREE_KUNIT_TEST) += percpu_counter_tree_kunit.o
+
obj-$(CONFIG_TEST_RUNTIME_MODULE) += module/
diff --git a/lib/tests/percpu_counter_tree_kunit.c b/lib/tests/percpu_counter_tree_kunit.c
new file mode 100644
index 0000000000000..8af8d2659b57b
--- /dev/null
+++ b/lib/tests/percpu_counter_tree_kunit.c
@@ -0,0 +1,583 @@
+// SPDX-License-Identifier: GPL-2.0+ OR MIT
+// SPDX-FileCopyrightText: 2026 Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
+
+#include <kunit/test.h>
+#include <linux/percpu_counter_tree.h>
+#include <linux/kthread.h>
+#include <linux/wait.h>
+#include <linux/random.h>
+#include <linux/cpuhplock.h>
+#include <linux/slab.h>
+#include <linux/sched/task.h>
+
+struct multi_thread_test_data {
+ long increment;
+ unsigned int nr_inc;
+ int counter_index;
+};
+
+#define NR_COUNTERS 2
+
+/* Hierarchical per-CPU counter instances. */
+static struct percpu_counter_tree counter[NR_COUNTERS];
+static struct percpu_counter_tree_level_item *items;
+
+/* Global atomic counters for validation. */
+static atomic_long_t global_counter[NR_COUNTERS];
+
+static DECLARE_WAIT_QUEUE_HEAD(kernel_threads_wq);
+static atomic_t kernel_threads_to_run;
+
+static void complete_work(void)
+{
+ if (atomic_dec_and_test(&kernel_threads_to_run))
+ wake_up(&kernel_threads_wq);
+}
+
+/* Random tasks test. */
+#define NR_RAND_TASKS 1000
+
+static void hpcc_print_info(struct kunit *test)
+{
+ kunit_info(test, "Running test with %d CPUs\n", num_online_cpus());
+}
+
+static void add_to_counter(int counter_index, unsigned int nr_inc, long increment)
+{
+ unsigned int i;
+
+ for (i = 0; i < nr_inc; i++) {
+ percpu_counter_tree_add(&counter[counter_index], increment);
+ atomic_long_add(increment, &global_counter[counter_index]);
+ }
+}
+
+static void check_counters(struct kunit *test)
+{
+ int counter_index;
+
+ /* Compare each counter with its global counter. */
+ for (counter_index = 0; counter_index < NR_COUNTERS; counter_index++) {
+ long v = atomic_long_read(&global_counter[counter_index]);
+ long approx_sum = percpu_counter_tree_approximate_sum(&counter[counter_index]);
+ unsigned long under_accuracy = 0, over_accuracy = 0;
+ long precise_min, precise_max;
+
+ /* Precise comparison. */
+ KUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(&counter[counter_index]), v);
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_precise_compare_value(&counter[counter_index], v));
+
+ /* Approximate comparison. */
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare_value(&counter[counter_index], v));
+
+ /* Accuracy limits checks. */
+ percpu_counter_tree_approximate_accuracy_range(&counter[counter_index], &under_accuracy, &over_accuracy);
+
+ /*
+ * under_accuracy and over_accuracy are unsigned long, so the
+ * inner expression promotes to unsigned. Compute the difference
+ * in unsigned and cast to signed for the comparison against 0.
+ */
+ KUNIT_EXPECT_GE(test, (long)(approx_sum - (v - under_accuracy)), 0);
+ KUNIT_EXPECT_LE(test, (long)(approx_sum - (v + over_accuracy)), 0);
+
+ /* Precise min/max range check. */
+ percpu_counter_tree_approximate_min_max_range(approx_sum, under_accuracy, over_accuracy, &precise_min, &precise_max);
+
+ KUNIT_EXPECT_GE(test, v - precise_min, 0);
+ KUNIT_EXPECT_LE(test, v - precise_max, 0);
+ }
+ /* Compare each counter with the second counter. */
+ KUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(&counter[0]), percpu_counter_tree_precise_sum(&counter[1]));
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_precise_compare(&counter[0], &counter[1]));
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare(&counter[0], &counter[1]));
+}
+
+static int multi_thread_worker_fn(void *data)
+{
+ struct multi_thread_test_data *td = data;
+
+ add_to_counter(td->counter_index, td->nr_inc, td->increment);
+ complete_work();
+ kfree(td);
+ return 0;
+}
+
+static struct task_struct *test_run_on_specific_cpu(int target_cpu, int counter_index, unsigned int nr_inc, long increment)
+{
+ struct multi_thread_test_data *td = kzalloc(sizeof(struct multi_thread_test_data), GFP_KERNEL);
+ struct task_struct *task;
+
+ if (!td)
+ return NULL;
+ td->increment = increment;
+ td->nr_inc = nr_inc;
+ td->counter_index = counter_index;
+ atomic_inc(&kernel_threads_to_run);
+ task = kthread_create_on_cpu(multi_thread_worker_fn, td, target_cpu, "kunit_multi_thread_worker");
+ if (IS_ERR_OR_NULL(task)) {
+ complete_work();
+ kfree(td);
+ return task;
+ }
+ get_task_struct(task);
+ wake_up_process(task);
+ return task;
+}
+
+static void init_kthreads(void)
+{
+ atomic_set(&kernel_threads_to_run, 1);
+}
+
+/*
+ * The caller needs to wait for work completion _before_ stopping kthreads.
+ */
+static void fini_kthreads(void)
+{
+ /* Release our own reference. */
+ complete_work();
+ /* Wait for all others threads to run. */
+ wait_event(kernel_threads_wq, (atomic_read(&kernel_threads_to_run) == 0));
+}
+
+static void test_sync_kthreads(void)
+{
+ fini_kthreads();
+ init_kthreads();
+}
+
+static int init_counters(unsigned long batch_size)
+{
+ int i, ret;
+
+ items = kvcalloc(NR_COUNTERS, percpu_counter_tree_items_size(), GFP_KERNEL);
+ if (!items)
+ return -ENOMEM;
+ ret = percpu_counter_tree_init_many(counter, items, NR_COUNTERS, batch_size, GFP_KERNEL);
+ if (ret) {
+ kvfree(items);
+ return ret;
+ }
+ for (i = 0; i < NR_COUNTERS; i++)
+ atomic_long_set(&global_counter[i], 0);
+ return 0;
+}
+
+static void fini_counters(void)
+{
+ percpu_counter_tree_destroy_many(counter, NR_COUNTERS);
+ kvfree(items);
+}
+
+enum up_test_inc_type {
+ INC_ONE,
+ INC_MINUS_ONE,
+ INC_RANDOM,
+};
+
+/*
+ * Single-threaded tests. Those use many threads to run on various CPUs,
+ * but synchronize for completion of each thread before running the
+ * next, effectively making sure there are no concurrent updates.
+ */
+static void do_hpcc_test_single_thread(struct kunit *test, int _cpu0, int _cpu1, enum up_test_inc_type type)
+{
+ unsigned long batch_size_order = 5;
+ int cpu0 = _cpu0;
+ int cpu1 = _cpu1;
+ int ret, i;
+
+ ret = init_counters(1UL << batch_size_order);
+ KUNIT_EXPECT_EQ(test, ret, 0);
+ if (ret)
+ return;
+ init_kthreads();
+ cpus_read_lock();
+ for (i = 0; i < 10000; i++) {
+ struct task_struct *task;
+ long increment;
+
+ switch (type) {
+ case INC_ONE:
+ increment = 1;
+ break;
+ case INC_MINUS_ONE:
+ increment = -1;
+ break;
+ case INC_RANDOM:
+ increment = (long) get_random_long() % 50000;
+ break;
+ default:
+ WARN_ON_ONCE(1);
+ increment = 0;
+ }
+ if (_cpu0 < 0)
+ cpu0 = cpumask_any_distribute(cpu_online_mask);
+ if (_cpu1 < 0)
+ cpu1 = cpumask_any_distribute(cpu_online_mask);
+ task = test_run_on_specific_cpu(cpu0, 0, 1, increment);
+ KUNIT_EXPECT_NOT_ERR_OR_NULL(test, task);
+ if (IS_ERR_OR_NULL(task)) {
+ fini_kthreads();
+ goto end;
+ }
+ test_sync_kthreads();
+ kthread_stop_put(task);
+ task = test_run_on_specific_cpu(cpu1, 1, 1, increment);
+ KUNIT_EXPECT_NOT_ERR_OR_NULL(test, task);
+ if (IS_ERR_OR_NULL(task)) {
+ fini_kthreads();
+ goto end;
+ }
+ test_sync_kthreads();
+ kthread_stop_put(task);
+ check_counters(test);
+ }
+end:
+ cpus_read_unlock();
+ fini_counters();
+}
+
+static void hpcc_test_single_thread_first(struct kunit *test)
+{
+ int cpu = cpumask_first(cpu_online_mask);
+
+ do_hpcc_test_single_thread(test, cpu, cpu, INC_ONE);
+ do_hpcc_test_single_thread(test, cpu, cpu, INC_MINUS_ONE);
+ do_hpcc_test_single_thread(test, cpu, cpu, INC_RANDOM);
+}
+
+static void hpcc_test_single_thread_first_random(struct kunit *test)
+{
+ int cpu = cpumask_first(cpu_online_mask);
+
+ do_hpcc_test_single_thread(test, cpu, -1, INC_ONE);
+ do_hpcc_test_single_thread(test, cpu, -1, INC_MINUS_ONE);
+ do_hpcc_test_single_thread(test, cpu, -1, INC_RANDOM);
+}
+
+static void hpcc_test_single_thread_random(struct kunit *test)
+{
+ do_hpcc_test_single_thread(test, -1, -1, INC_ONE);
+ do_hpcc_test_single_thread(test, -1, -1, INC_MINUS_ONE);
+ do_hpcc_test_single_thread(test, -1, -1, INC_RANDOM);
+}
+
+/* Multi-threaded SMP tests. */
+
+static void do_hpcc_multi_thread_increment_each_cpu(struct kunit *test, unsigned long batch_size, unsigned int nr_inc, long increment)
+{
+ struct task_struct *(*task)[NR_COUNTERS];
+ int ret, cpu;
+
+ task = kvcalloc(nr_cpu_ids, sizeof(*task), GFP_KERNEL);
+ KUNIT_EXPECT_NOT_NULL(test, task);
+ if (!task)
+ return;
+ ret = init_counters(batch_size);
+ KUNIT_EXPECT_EQ(test, ret, 0);
+ if (ret)
+ goto end;
+ init_kthreads();
+ cpus_read_lock();
+ for_each_online_cpu(cpu) {
+ task[cpu][0] = test_run_on_specific_cpu(cpu, 0, nr_inc, increment);
+ KUNIT_EXPECT_NOT_ERR_OR_NULL(test, task[cpu][0]);
+ if (IS_ERR_OR_NULL(task[cpu][0]))
+ break;
+ task[cpu][1] = test_run_on_specific_cpu(cpu, 1, nr_inc, increment);
+ KUNIT_EXPECT_NOT_ERR_OR_NULL(test, task[cpu][1]);
+ if (IS_ERR_OR_NULL(task[cpu][1]))
+ break;
+ }
+ fini_kthreads();
+ for_each_online_cpu(cpu) {
+ if (!IS_ERR_OR_NULL(task[cpu][0]))
+ kthread_stop_put(task[cpu][0]);
+ if (!IS_ERR_OR_NULL(task[cpu][1]))
+ kthread_stop_put(task[cpu][1]);
+ }
+ cpus_read_unlock();
+ check_counters(test);
+ fini_counters();
+end:
+ kvfree(task);
+}
+
+static void do_hpcc_multi_thread_increment_even_cpus(struct kunit *test, unsigned long batch_size, unsigned int nr_inc, long increment)
+{
+ struct task_struct *(*task)[NR_COUNTERS];
+ int ret, cpu;
+
+ task = kvcalloc(nr_cpu_ids, sizeof(*task), GFP_KERNEL);
+ KUNIT_EXPECT_NOT_NULL(test, task);
+ if (!task)
+ return;
+ ret = init_counters(batch_size);
+ KUNIT_EXPECT_EQ(test, ret, 0);
+ if (ret)
+ goto end;
+ init_kthreads();
+ cpus_read_lock();
+ for_each_online_cpu(cpu) {
+ /*
+ * Funnel pairs of CPU indexes to their associated even index.
+ *
+ * Map 0 and 1 -> index 0
+ * Map 2 and 3 -> index 2 and so on.
+ */
+ int even_cpu = cpu & ~1;
+
+ if (!cpumask_test_cpu(even_cpu, cpu_online_mask))
+ continue;
+ task[cpu][0] = test_run_on_specific_cpu(cpu, 0, nr_inc, increment);
+ KUNIT_EXPECT_NOT_ERR_OR_NULL(test, task[cpu][0]);
+ if (IS_ERR_OR_NULL(task[cpu][0]))
+ break;
+ task[cpu][1] = test_run_on_specific_cpu(even_cpu, 1, nr_inc, increment);
+ KUNIT_EXPECT_NOT_ERR_OR_NULL(test, task[cpu][1]);
+ if (IS_ERR_OR_NULL(task[cpu][1]))
+ break;
+ }
+ fini_kthreads();
+ for_each_online_cpu(cpu) {
+ if (!IS_ERR_OR_NULL(task[cpu][0]))
+ kthread_stop_put(task[cpu][0]);
+ if (!IS_ERR_OR_NULL(task[cpu][1]))
+ kthread_stop_put(task[cpu][1]);
+ }
+ cpus_read_unlock();
+ check_counters(test);
+ fini_counters();
+end:
+ kvfree(task);
+}
+
+static void do_hpcc_multi_thread_increment_single_cpu(struct kunit *test, unsigned long batch_size, unsigned int nr_inc, long increment)
+{
+ struct task_struct *(*task)[NR_COUNTERS];
+ int ret, cpu;
+
+ task = kvcalloc(nr_cpu_ids, sizeof(*task), GFP_KERNEL);
+ KUNIT_EXPECT_NOT_NULL(test, task);
+ if (!task)
+ return;
+ ret = init_counters(batch_size);
+ KUNIT_EXPECT_EQ(test, ret, 0);
+ if (ret)
+ goto end;
+ init_kthreads();
+ cpus_read_lock();
+ for_each_online_cpu(cpu) {
+ task[cpu][0] = test_run_on_specific_cpu(cpu, 0, nr_inc, increment);
+ KUNIT_EXPECT_NOT_ERR_OR_NULL(test, task[cpu][0]);
+ if (IS_ERR_OR_NULL(task[cpu][0]))
+ break;
+ task[cpu][1] = test_run_on_specific_cpu(cpumask_first(cpu_online_mask), 1, nr_inc, increment);
+ KUNIT_EXPECT_NOT_ERR_OR_NULL(test, task[cpu][1]);
+ if (IS_ERR_OR_NULL(task[cpu][1]))
+ break;
+ }
+ fini_kthreads();
+ for_each_online_cpu(cpu) {
+ if (!IS_ERR_OR_NULL(task[cpu][0]))
+ kthread_stop_put(task[cpu][0]);
+ if (!IS_ERR_OR_NULL(task[cpu][1]))
+ kthread_stop_put(task[cpu][1]);
+ }
+ cpus_read_unlock();
+ check_counters(test);
+ fini_counters();
+end:
+ kvfree(task);
+}
+
+static void do_hpcc_multi_thread_increment_random_cpu(struct kunit *test, unsigned long batch_size, unsigned int nr_inc, long increment)
+{
+ struct task_struct *(*task)[NR_COUNTERS];
+ int ret, cpu;
+
+ task = kvcalloc(nr_cpu_ids, sizeof(*task), GFP_KERNEL);
+ KUNIT_EXPECT_NOT_NULL(test, task);
+ if (!task)
+ return;
+ ret = init_counters(batch_size);
+ KUNIT_EXPECT_EQ(test, ret, 0);
+ if (ret)
+ goto end;
+ init_kthreads();
+ cpus_read_lock();
+ for_each_online_cpu(cpu) {
+ task[cpu][0] = test_run_on_specific_cpu(cpu, 0, nr_inc, increment);
+ KUNIT_EXPECT_NOT_ERR_OR_NULL(test, task[cpu][0]);
+ if (IS_ERR_OR_NULL(task[cpu][0]))
+ break;
+ task[cpu][1] = test_run_on_specific_cpu(cpumask_any_distribute(cpu_online_mask), 1, nr_inc, increment);
+ KUNIT_EXPECT_NOT_ERR_OR_NULL(test, task[cpu][1]);
+ if (IS_ERR_OR_NULL(task[cpu][1]))
+ break;
+ }
+ fini_kthreads();
+ for_each_online_cpu(cpu) {
+ if (!IS_ERR_OR_NULL(task[cpu][0]))
+ kthread_stop_put(task[cpu][0]);
+ if (!IS_ERR_OR_NULL(task[cpu][1]))
+ kthread_stop_put(task[cpu][1]);
+ }
+ cpus_read_unlock();
+ check_counters(test);
+ fini_counters();
+end:
+ kvfree(task);
+}
+
+static void hpcc_test_multi_thread_batch_increment(struct kunit *test)
+{
+ unsigned long batch_size_order;
+
+ for (batch_size_order = 2; batch_size_order < 10; batch_size_order++) {
+ unsigned int nr_inc;
+
+ for (nr_inc = 1; nr_inc < 1024; nr_inc *= 2) {
+ long increment;
+
+ for (increment = 1; increment < 100000; increment *= 10) {
+ do_hpcc_multi_thread_increment_each_cpu(test, 1UL << batch_size_order, nr_inc, increment);
+ do_hpcc_multi_thread_increment_even_cpus(test, 1UL << batch_size_order, nr_inc, increment);
+ do_hpcc_multi_thread_increment_single_cpu(test, 1UL << batch_size_order, nr_inc, increment);
+ do_hpcc_multi_thread_increment_random_cpu(test, 1UL << batch_size_order, nr_inc, increment);
+ }
+ }
+ }
+}
+
+static void hpcc_test_multi_thread_random_walk(struct kunit *test)
+{
+ struct task_struct *(*task)[NR_COUNTERS];
+ unsigned long batch_size_order = 5;
+ int loop;
+
+ task = kvcalloc(NR_RAND_TASKS, sizeof(*task), GFP_KERNEL);
+ KUNIT_EXPECT_NOT_NULL(test, task);
+ if (!task)
+ return;
+ for (loop = 0; loop < 100; loop++) {
+ int ret, i;
+
+ memset(task, 0, NR_RAND_TASKS * sizeof(*task));
+ ret = init_counters(1UL << batch_size_order);
+ KUNIT_EXPECT_EQ(test, ret, 0);
+ if (ret)
+ goto end;
+ init_kthreads();
+ cpus_read_lock();
+ for (i = 0; i < NR_RAND_TASKS; i++) {
+ long increment = (long) get_random_long() % 512;
+ unsigned int nr_inc = ((unsigned long) get_random_long()) % 1024;
+
+ task[i][0] = test_run_on_specific_cpu(cpumask_any_distribute(cpu_online_mask), 0, nr_inc, increment);
+ KUNIT_EXPECT_NOT_ERR_OR_NULL(test, task[i][0]);
+ if (IS_ERR_OR_NULL(task[i][0]))
+ break;
+ task[i][1] = test_run_on_specific_cpu(cpumask_any_distribute(cpu_online_mask), 1, nr_inc, increment);
+ KUNIT_EXPECT_NOT_ERR_OR_NULL(test, task[i][1]);
+ if (IS_ERR_OR_NULL(task[i][1]))
+ break;
+ }
+ fini_kthreads();
+ for (i = 0; i < NR_RAND_TASKS; i++) {
+ if (!IS_ERR_OR_NULL(task[i][0]))
+ kthread_stop_put(task[i][0]);
+ if (!IS_ERR_OR_NULL(task[i][1]))
+ kthread_stop_put(task[i][1]);
+ }
+ cpus_read_unlock();
+ check_counters(test);
+ fini_counters();
+ }
+end:
+ kvfree(task);
+}
+
+static void hpcc_test_init_one(struct kunit *test)
+{
+ struct percpu_counter_tree pct;
+ struct percpu_counter_tree_level_item *counter_items;
+ int ret;
+
+ counter_items = kzalloc(percpu_counter_tree_items_size(), GFP_KERNEL);
+ KUNIT_EXPECT_NOT_NULL(test, counter_items);
+ if (!counter_items)
+ return;
+ ret = percpu_counter_tree_init(&pct, counter_items, 32, GFP_KERNEL);
+ KUNIT_EXPECT_EQ(test, ret, 0);
+ if (ret)
+ goto end;
+
+ percpu_counter_tree_destroy(&pct);
+end:
+ kfree(counter_items);
+}
+
+static void hpcc_test_set(struct kunit *test)
+{
+ static long values[] = {
+ 5, 100, 127, 128, 255, 256, 4095, 4096, 500000, 0,
+ -5, -100, -127, -128, -255, -256, -4095, -4096, -500000,
+ };
+ struct percpu_counter_tree pct;
+ struct percpu_counter_tree_level_item *counter_items;
+ int i, ret;
+
+ counter_items = kzalloc(percpu_counter_tree_items_size(), GFP_KERNEL);
+ KUNIT_EXPECT_NOT_NULL(test, counter_items);
+ if (!counter_items)
+ return;
+ ret = percpu_counter_tree_init(&pct, counter_items, 32, GFP_KERNEL);
+ KUNIT_EXPECT_EQ(test, ret, 0);
+ if (ret)
+ goto end;
+
+ for (i = 0; i < ARRAY_SIZE(values); i++) {
+ long v = values[i];
+
+ percpu_counter_tree_set(&pct, v);
+ KUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(&pct), v);
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare_value(&pct, v));
+
+ percpu_counter_tree_add(&pct, v);
+ KUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(&pct), 2 * v);
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare_value(&pct, 2 * v));
+
+ percpu_counter_tree_add(&pct, -2 * v);
+ KUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(&pct), 0);
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare_value(&pct, 0));
+ }
+
+ percpu_counter_tree_destroy(&pct);
+end:
+ kfree(counter_items);
+}
+
+static struct kunit_case hpcc_test_cases[] = {
+ KUNIT_CASE(hpcc_print_info),
+ KUNIT_CASE(hpcc_test_single_thread_first),
+ KUNIT_CASE(hpcc_test_single_thread_first_random),
+ KUNIT_CASE(hpcc_test_single_thread_random),
+ KUNIT_CASE(hpcc_test_multi_thread_batch_increment),
+ KUNIT_CASE(hpcc_test_multi_thread_random_walk),
+ KUNIT_CASE(hpcc_test_init_one),
+ KUNIT_CASE(hpcc_test_set),
+ {}
+};
+
+static struct kunit_suite hpcc_test_suite = {
+ .name = "percpu_counter_tree",
+ .test_cases = hpcc_test_cases,
+};
+
+kunit_test_suite(hpcc_test_suite);
+
+MODULE_DESCRIPTION("Test cases for hierarchical per-CPU counters");
+MODULE_LICENSE("Dual MIT/GPL");
|
| 3/2 | 2026/09/06 04:16 | llm | patch-evaluator |
0mModel:gemini-3.7-flash Args: null Results: null Tokens: input: 29848 output: 20 thoughts: 739 Thoughts: **Fuzzing Evaluation: Hierarchical Per-CPU Counters (HPCC) Patch Series** Okay, let's break this down. My initial assessment is this *definitely* warrants fuzzing with syzkaller. The patch series is adding a new "Hierarchical Per-CPU Counters (HPCC)" data structure and subsystem, which isn't just a minor addition; it's integrating this HPCC directly into *core memory management (mm)* statistics. This goes way beyond isolated library code. Specifically, it's replacing the existing `struct percpu_counter rss_stat` within `struct mm_struct` with the `percpu_counter_tree` structure. And that touches a *lot* of things. The changes ripple through `mm.h`, `mm_types.h`, `kernel/fork.c`, `init/main.c`, and even the tracepoints. This impacts how the kernel tracks RSS (resident set size), and that includes key operations like process creation (`fork`), execution (`exec`), memory mapping and unmapping, and page faults. That's *highly* executable code, and *very* frequently exercised. Every single process fork, exec, memory mapping, unmapping, and page fault will hit this code. Plus the fork/exit/mm init related functions. So, according to my "WorthFuzzing" criteria, this is a clear "yes." It hits core subsystems like memory management and the `kernel/fork.c` code, and it introduces new library components. It's prime syzkaller territory. Now, I need to verify Kconfig dependencies. The `lib/Makefile` indicates that this is only enabled under `CONFIG_SMP`, but it has inline stubs in the header. That’s standard for this type of feature, but I need to make sure there aren't any other obscure configurations that could influence its behavior. I'll need to double-check `lib/Kconfig` and possibly do some grepping to ensure there are no other prerequisites or dependencies that might change the fuzzer's approach. |
| 4/2 | 2026/09/06 04:16 | tool | grepper |
0mArgs:{
"Expression": "percpu_counter_tree"
}Results: {
"Output": "\nFull output is too long, showing 500 out of 643 lines.\nUse more precise expression if possible.\n\nDocumentation/core-api/percpu-counter-tree.rst=78=Functions and structures\n--\nDocumentation/core-api/percpu-counter-tree.rst-80-\nDocumentation/core-api/percpu-counter-tree.rst:81:.. kernel-doc:: include/linux/percpu_counter_tree.h\nDocumentation/core-api/percpu-counter-tree.rst:82:.. kernel-doc:: lib/percpu_counter_tree.c\n--\ninclude/linux/mm.h=3445=static inline bool get_user_page_fast_only(unsigned long addr,\n--\ninclude/linux/mm.h-3450-\ninclude/linux/mm.h:3451:static inline struct percpu_counter_tree_level_item *get_rss_stat_items(struct mm_struct *mm)\ninclude/linux/mm.h-3452-{\n--\ninclude/linux/mm.h-3456-\tptr += get_rss_stat_items_offset();\ninclude/linux/mm.h:3457:\treturn (struct percpu_counter_tree_level_item *)ptr;\ninclude/linux/mm.h-3458-}\n--\ninclude/linux/mm.h=3463=static inline unsigned long get_mm_counter(struct mm_struct *mm, int member)\ninclude/linux/mm.h-3464-{\ninclude/linux/mm.h:3465:\treturn percpu_counter_tree_approximate_sum_positive(\u0026mm-\u003erss_stat[member]);\ninclude/linux/mm.h-3466-}\n--\ninclude/linux/mm.h=3468=static inline unsigned long get_mm_counter_sum(struct mm_struct *mm, int member)\ninclude/linux/mm.h-3469-{\ninclude/linux/mm.h:3470:\treturn percpu_counter_tree_precise_sum_positive(\u0026mm-\u003erss_stat[member]);\ninclude/linux/mm.h-3471-}\n--\ninclude/linux/mm.h=3475=static inline void add_mm_counter(struct mm_struct *mm, int member, long value)\ninclude/linux/mm.h-3476-{\ninclude/linux/mm.h:3477:\tpercpu_counter_tree_add(\u0026mm-\u003erss_stat[member], value);\ninclude/linux/mm.h-3478-\n--\ninclude/linux/mm.h=3482=static inline void inc_mm_counter(struct mm_struct *mm, int member)\ninclude/linux/mm.h-3483-{\ninclude/linux/mm.h:3484:\tpercpu_counter_tree_add(\u0026mm-\u003erss_stat[member], 1);\ninclude/linux/mm.h-3485-\n--\ninclude/linux/mm.h=3489=static inline void dec_mm_counter(struct mm_struct *mm, int member)\ninclude/linux/mm.h-3490-{\ninclude/linux/mm.h:3491:\tpercpu_counter_tree_add(\u0026mm-\u003erss_stat[member], -1);\ninclude/linux/mm.h-3492-\n--\ninclude/linux/mm_types.h-20-#include \u003clinux/seqlock.h\u003e\ninclude/linux/mm_types.h:21:#include \u003clinux/percpu_counter_tree.h\u003e\ninclude/linux/mm_types.h-22-#include \u003clinux/types.h\u003e\n--\ninclude/linux/mm_types.h=1169=typedef struct {\n--\ninclude/linux/mm_types.h-1175- * alignment of its content:\ninclude/linux/mm_types.h:1176: * __alignof__(struct percpu_counter_tree_level_item) provides a\ninclude/linux/mm_types.h-1177- * cacheline aligned alignment on SMP systems, else alignment on\n--\ninclude/linux/mm_types.h-1180-#ifdef CONFIG_SMP\ninclude/linux/mm_types.h:1181:# define __mm_struct_flexible_array_aligned\t__aligned(__alignof__(struct percpu_counter_tree_level_item))\ninclude/linux/mm_types.h-1182-#else\n--\ninclude/linux/mm_types.h=1188=struct mm_struct {\n--\ninclude/linux/mm_types.h-1327-\ninclude/linux/mm_types.h:1328:\t\tstruct percpu_counter_tree rss_stat[NR_MM_COUNTERS];\ninclude/linux/mm_types.h-1329-\n--\ninclude/linux/mm_types.h=1672=static inline size_t get_rss_stat_items_size(void)\ninclude/linux/mm_types.h-1673-{\ninclude/linux/mm_types.h:1674:\treturn percpu_counter_tree_items_size() * NR_MM_COUNTERS;\ninclude/linux/mm_types.h-1675-}\n--\ninclude/linux/percpu_counter_tree.h-58-\ninclude/linux/percpu_counter_tree.h:59:struct percpu_counter_tree_level_item {\ninclude/linux/percpu_counter_tree.h-60-\tatomic_long_t count;\t\t/*\n--\ninclude/linux/percpu_counter_tree.h-67-#define PERCPU_COUNTER_TREE_ITEMS_STATIC_SIZE\t\\\ninclude/linux/percpu_counter_tree.h:68:\t(PERCPU_COUNTER_TREE_STATIC_NR_ITEMS * sizeof(struct percpu_counter_tree_level_item))\ninclude/linux/percpu_counter_tree.h-69-#define PERCPU_COUNTER_TREE_ITEMS_ALIGN(offset)\t\\\ninclude/linux/percpu_counter_tree.h:70:\tALIGN((offset), __alignof__(struct percpu_counter_tree_level_item))\ninclude/linux/percpu_counter_tree.h-71-\ninclude/linux/percpu_counter_tree.h:72:struct percpu_counter_tree {\ninclude/linux/percpu_counter_tree.h-73-\t/* Fast-path fields. */\n--\ninclude/linux/percpu_counter_tree.h-82-\t/* Slow-path fields. */\ninclude/linux/percpu_counter_tree.h:83:\tstruct percpu_counter_tree_level_item *items;\t/* Array of tree items for levels 1 to N. */\ninclude/linux/percpu_counter_tree.h-84-\tunsigned long batch_size;\t/*\n--\ninclude/linux/percpu_counter_tree.h-104-\ninclude/linux/percpu_counter_tree.h:105:extern size_t __percpu_counter_tree_items_size __ro_after_init;\ninclude/linux/percpu_counter_tree.h-106-\ninclude/linux/percpu_counter_tree.h-107-/*\ninclude/linux/percpu_counter_tree.h:108: * percpu_counter_tree_items_size - Query the size required for counter tree items.\ninclude/linux/percpu_counter_tree.h-109- *\n--\ninclude/linux/percpu_counter_tree.h=116=static inline\ninclude/linux/percpu_counter_tree.h:117:size_t percpu_counter_tree_items_size(void)\ninclude/linux/percpu_counter_tree.h-118-{\ninclude/linux/percpu_counter_tree.h:119:\tWARN_ON_ONCE(__percpu_counter_tree_items_size == SIZE_MAX);\ninclude/linux/percpu_counter_tree.h:120:\treturn __percpu_counter_tree_items_size;\ninclude/linux/percpu_counter_tree.h-121-}\ninclude/linux/percpu_counter_tree.h-122-\ninclude/linux/percpu_counter_tree.h:123:int percpu_counter_tree_init_many(struct percpu_counter_tree *counters, struct percpu_counter_tree_level_item *items,\ninclude/linux/percpu_counter_tree.h-124-\t\t\t\t unsigned int nr_counters, unsigned long batch_size, gfp_t gfp_flags);\ninclude/linux/percpu_counter_tree.h:125:int percpu_counter_tree_init(struct percpu_counter_tree *counter, struct percpu_counter_tree_level_item *items,\ninclude/linux/percpu_counter_tree.h-126-\t\t\t unsigned long batch_size, gfp_t gfp_flags);\ninclude/linux/percpu_counter_tree.h:127:void percpu_counter_tree_destroy_many(struct percpu_counter_tree *counter, unsigned int nr_counters);\ninclude/linux/percpu_counter_tree.h:128:void percpu_counter_tree_destroy(struct percpu_counter_tree *counter);\ninclude/linux/percpu_counter_tree.h:129:void percpu_counter_tree_add(struct percpu_counter_tree *counter, long inc);\ninclude/linux/percpu_counter_tree.h:130:long percpu_counter_tree_precise_sum(struct percpu_counter_tree *counter);\ninclude/linux/percpu_counter_tree.h:131:int percpu_counter_tree_approximate_compare(struct percpu_counter_tree *a, struct percpu_counter_tree *b);\ninclude/linux/percpu_counter_tree.h:132:int percpu_counter_tree_approximate_compare_value(struct percpu_counter_tree *counter, long v);\ninclude/linux/percpu_counter_tree.h:133:int percpu_counter_tree_precise_compare(struct percpu_counter_tree *a, struct percpu_counter_tree *b);\ninclude/linux/percpu_counter_tree.h:134:int percpu_counter_tree_precise_compare_value(struct percpu_counter_tree *counter, long v);\ninclude/linux/percpu_counter_tree.h:135:void percpu_counter_tree_set(struct percpu_counter_tree *counter, long v);\ninclude/linux/percpu_counter_tree.h:136:int percpu_counter_tree_subsystem_init(void);\ninclude/linux/percpu_counter_tree.h-137-\ninclude/linux/percpu_counter_tree.h-138-/**\ninclude/linux/percpu_counter_tree.h:139: * percpu_counter_tree_approximate_sum() - Return approximate counter sum.\ninclude/linux/percpu_counter_tree.h-140- * @counter: The counter to sum.\n--\ninclude/linux/percpu_counter_tree.h-142- * Querying the approximate sum is fast, but it is only accurate within\ninclude/linux/percpu_counter_tree.h:143: * the bounds delimited by percpu_counter_tree_approximate_accuracy_range().\ninclude/linux/percpu_counter_tree.h-144- * This is meant to be used when speed is preferred over accuracy.\n--\ninclude/linux/percpu_counter_tree.h=148=static inline\ninclude/linux/percpu_counter_tree.h:149:long percpu_counter_tree_approximate_sum(struct percpu_counter_tree *counter)\ninclude/linux/percpu_counter_tree.h-150-{\n--\ninclude/linux/percpu_counter_tree.h-160-/**\ninclude/linux/percpu_counter_tree.h:161: * percpu_counter_tree_approximate_accuracy_range - Query the accuracy range for a counter tree.\ninclude/linux/percpu_counter_tree.h-162- * @counter: Counter to query.\n--\ninclude/linux/percpu_counter_tree.h-175- * Using this function should not be typically required, see the following functions instead:\ninclude/linux/percpu_counter_tree.h:176: * * percpu_counter_tree_approximate_compare(),\ninclude/linux/percpu_counter_tree.h:177: * * percpu_counter_tree_approximate_compare_value(),\ninclude/linux/percpu_counter_tree.h:178: * * percpu_counter_tree_precise_compare(),\ninclude/linux/percpu_counter_tree.h:179: * * percpu_counter_tree_precise_compare_value().\ninclude/linux/percpu_counter_tree.h-180- */\ninclude/linux/percpu_counter_tree.h=181=static inline\ninclude/linux/percpu_counter_tree.h:182:void percpu_counter_tree_approximate_accuracy_range(struct percpu_counter_tree *counter,\ninclude/linux/percpu_counter_tree.h-183-\t\t\t\t\t\t unsigned long *under, unsigned long *over)\n--\ninclude/linux/percpu_counter_tree.h-193-\ninclude/linux/percpu_counter_tree.h:194:struct percpu_counter_tree_level_item;\ninclude/linux/percpu_counter_tree.h-195-\ninclude/linux/percpu_counter_tree.h:196:struct percpu_counter_tree {\ninclude/linux/percpu_counter_tree.h-197-\tatomic_long_t count;\n--\ninclude/linux/percpu_counter_tree.h=200=static inline\ninclude/linux/percpu_counter_tree.h:201:size_t percpu_counter_tree_items_size(void)\ninclude/linux/percpu_counter_tree.h-202-{\n--\ninclude/linux/percpu_counter_tree.h=206=static inline\ninclude/linux/percpu_counter_tree.h:207:int percpu_counter_tree_init_many(struct percpu_counter_tree *counters, struct percpu_counter_tree_level_item *items,\ninclude/linux/percpu_counter_tree.h-208-\t\t\t\t unsigned int nr_counters, unsigned long batch_size, gfp_t gfp_flags)\n--\ninclude/linux/percpu_counter_tree.h=215=static inline\ninclude/linux/percpu_counter_tree.h:216:int percpu_counter_tree_init(struct percpu_counter_tree *counter, struct percpu_counter_tree_level_item *items,\ninclude/linux/percpu_counter_tree.h-217-\t\t\t unsigned long batch_size, gfp_t gfp_flags)\ninclude/linux/percpu_counter_tree.h-218-{\ninclude/linux/percpu_counter_tree.h:219:\treturn percpu_counter_tree_init_many(counter, items, 1, batch_size, gfp_flags);\ninclude/linux/percpu_counter_tree.h-220-}\n--\ninclude/linux/percpu_counter_tree.h=222=static inline\ninclude/linux/percpu_counter_tree.h:223:void percpu_counter_tree_destroy_many(struct percpu_counter_tree *counter, unsigned int nr_counters)\ninclude/linux/percpu_counter_tree.h-224-{\n--\ninclude/linux/percpu_counter_tree.h=227=static inline\ninclude/linux/percpu_counter_tree.h:228:void percpu_counter_tree_destroy(struct percpu_counter_tree *counter)\ninclude/linux/percpu_counter_tree.h-229-{\n--\ninclude/linux/percpu_counter_tree.h=232=static inline\ninclude/linux/percpu_counter_tree.h:233:long percpu_counter_tree_precise_sum(struct percpu_counter_tree *counter)\ninclude/linux/percpu_counter_tree.h-234-{\n--\ninclude/linux/percpu_counter_tree.h=238=static inline\ninclude/linux/percpu_counter_tree.h:239:int percpu_counter_tree_precise_compare(struct percpu_counter_tree *a, struct percpu_counter_tree *b)\ninclude/linux/percpu_counter_tree.h-240-{\ninclude/linux/percpu_counter_tree.h:241:\tlong count_a = percpu_counter_tree_precise_sum(a),\ninclude/linux/percpu_counter_tree.h:242:\t count_b = percpu_counter_tree_precise_sum(b);\ninclude/linux/percpu_counter_tree.h-243-\n--\ninclude/linux/percpu_counter_tree.h=251=static inline\ninclude/linux/percpu_counter_tree.h:252:int percpu_counter_tree_precise_compare_value(struct percpu_counter_tree *counter, long v)\ninclude/linux/percpu_counter_tree.h-253-{\ninclude/linux/percpu_counter_tree.h:254:\tlong count = percpu_counter_tree_precise_sum(counter);\ninclude/linux/percpu_counter_tree.h-255-\n--\ninclude/linux/percpu_counter_tree.h=263=static inline\ninclude/linux/percpu_counter_tree.h:264:int percpu_counter_tree_approximate_compare(struct percpu_counter_tree *a, struct percpu_counter_tree *b)\ninclude/linux/percpu_counter_tree.h-265-{\ninclude/linux/percpu_counter_tree.h:266:\treturn percpu_counter_tree_precise_compare(a, b);\ninclude/linux/percpu_counter_tree.h-267-}\n--\ninclude/linux/percpu_counter_tree.h=269=static inline\ninclude/linux/percpu_counter_tree.h:270:int percpu_counter_tree_approximate_compare_value(struct percpu_counter_tree *counter, long v)\ninclude/linux/percpu_counter_tree.h-271-{\ninclude/linux/percpu_counter_tree.h:272:\treturn percpu_counter_tree_precise_compare_value(counter, v);\ninclude/linux/percpu_counter_tree.h-273-}\n--\ninclude/linux/percpu_counter_tree.h=275=static inline\ninclude/linux/percpu_counter_tree.h:276:void percpu_counter_tree_set(struct percpu_counter_tree *counter, long v)\ninclude/linux/percpu_counter_tree.h-277-{\n--\ninclude/linux/percpu_counter_tree.h=281=static inline\ninclude/linux/percpu_counter_tree.h:282:void percpu_counter_tree_approximate_accuracy_range(struct percpu_counter_tree *counter,\ninclude/linux/percpu_counter_tree.h-283-\t\t\t\t\t\t unsigned long *under, unsigned long *over)\n--\ninclude/linux/percpu_counter_tree.h=287=static inline\ninclude/linux/percpu_counter_tree.h:288:void percpu_counter_tree_add(struct percpu_counter_tree *counter, long inc)\ninclude/linux/percpu_counter_tree.h-289-{\n--\ninclude/linux/percpu_counter_tree.h=293=static inline\ninclude/linux/percpu_counter_tree.h:294:long percpu_counter_tree_approximate_sum(struct percpu_counter_tree *counter)\ninclude/linux/percpu_counter_tree.h-295-{\ninclude/linux/percpu_counter_tree.h:296:\treturn percpu_counter_tree_precise_sum(counter);\ninclude/linux/percpu_counter_tree.h-297-}\n--\ninclude/linux/percpu_counter_tree.h=299=static inline\ninclude/linux/percpu_counter_tree.h:300:int percpu_counter_tree_subsystem_init(void)\ninclude/linux/percpu_counter_tree.h-301-{\n--\ninclude/linux/percpu_counter_tree.h-307-/**\ninclude/linux/percpu_counter_tree.h:308: * percpu_counter_tree_approximate_sum_positive() - Return a positive approximate counter sum.\ninclude/linux/percpu_counter_tree.h-309- * @counter: The counter to sum.\n--\ninclude/linux/percpu_counter_tree.h=316=static inline\ninclude/linux/percpu_counter_tree.h:317:long percpu_counter_tree_approximate_sum_positive(struct percpu_counter_tree *counter)\ninclude/linux/percpu_counter_tree.h-318-{\ninclude/linux/percpu_counter_tree.h:319:\tlong v = percpu_counter_tree_approximate_sum(counter);\ninclude/linux/percpu_counter_tree.h-320-\treturn v \u003e 0 ? v : 0;\n--\ninclude/linux/percpu_counter_tree.h-323-/**\ninclude/linux/percpu_counter_tree.h:324: * percpu_counter_tree_precise_sum_positive() - Return a positive precise counter sum.\ninclude/linux/percpu_counter_tree.h-325- * @counter: The counter to sum.\n--\ninclude/linux/percpu_counter_tree.h=332=static inline\ninclude/linux/percpu_counter_tree.h:333:long percpu_counter_tree_precise_sum_positive(struct percpu_counter_tree *counter)\ninclude/linux/percpu_counter_tree.h-334-{\ninclude/linux/percpu_counter_tree.h:335:\tlong v = percpu_counter_tree_precise_sum(counter);\ninclude/linux/percpu_counter_tree.h-336-\treturn v \u003e 0 ? v : 0;\n--\ninclude/linux/percpu_counter_tree.h-339-/**\ninclude/linux/percpu_counter_tree.h:340: * percpu_counter_tree_approximate_min_max_range() - Return the approximation min and max precise values.\ninclude/linux/percpu_counter_tree.h-341- * @approx_sum: Approximated sum.\n--\ninclude/linux/percpu_counter_tree.h=361=static inline\ninclude/linux/percpu_counter_tree.h:362:void percpu_counter_tree_approximate_min_max_range(long approx_sum, unsigned long under, unsigned long over,\ninclude/linux/percpu_counter_tree.h-363-\t\t\t\t\t\t long *precise_min, long *precise_max)\n--\ninclude/linux/percpu_counter_tree.h-369-/**\ninclude/linux/percpu_counter_tree.h:370: * percpu_counter_tree_approximate_min_max() - Return the tree approximation, min and max possible precise values.\ninclude/linux/percpu_counter_tree.h-371- * @counter: The counter to sum.\n--\ninclude/linux/percpu_counter_tree.h=379=static inline\ninclude/linux/percpu_counter_tree.h:380:void percpu_counter_tree_approximate_min_max(struct percpu_counter_tree *counter,\ninclude/linux/percpu_counter_tree.h-381-\t\t\t\t\t long *approx_sum, long *precise_min, long *precise_max)\n--\ninclude/linux/percpu_counter_tree.h-383-\tunsigned long under = 0, over = 0;\ninclude/linux/percpu_counter_tree.h:384:\tlong v = percpu_counter_tree_approximate_sum(counter);\ninclude/linux/percpu_counter_tree.h-385-\ninclude/linux/percpu_counter_tree.h:386:\tpercpu_counter_tree_approximate_accuracy_range(counter, \u0026under, \u0026over);\ninclude/linux/percpu_counter_tree.h:387:\tpercpu_counter_tree_approximate_min_max_range(v, under, over, precise_min, precise_max);\ninclude/linux/percpu_counter_tree.h-388-\t*approx_sum = v;\n--\ninclude/trace/events/kmem.h=427=TRACE_EVENT(rss_stat,\n--\ninclude/trace/events/kmem.h-450-\t\t__entry-\u003emember = member;\ninclude/trace/events/kmem.h:451:\t\t__entry-\u003esize = (percpu_counter_tree_approximate_sum_positive(\u0026mm-\u003erss_stat[member])\ninclude/trace/events/kmem.h-452-\t\t\t\t\t\t\t \u003c\u003c PAGE_SHIFT);\n--\ninit/main.c-110-#include \u003clinux/vdso_datastore.h\u003e\ninit/main.c:111:#include \u003clinux/percpu_counter_tree.h\u003e\ninit/main.c-112-#include \u003cnet/net_namespace.h\u003e\n--\ninit/main.c=983=void start_kernel(void)\n--\ninit/main.c-1013-\tsetup_nr_cpu_ids();\ninit/main.c:1014:\tpercpu_counter_tree_subsystem_init();\ninit/main.c-1015-\tsetup_per_cpu_areas();\n--\nkernel/fork.c=633=static void check_mm(struct mm_struct *mm)\n--\nkernel/fork.c-640-\tfor (i = 0; i \u003c NR_MM_COUNTERS; i++) {\nkernel/fork.c:641:\t\tif (unlikely(percpu_counter_tree_precise_compare_value(\u0026mm-\u003erss_stat[i], 0) != 0))\nkernel/fork.c-642-\t\t\tpr_alert(\"BUG: Bad rss-counter state mm:%p type:%s val:%ld Comm:%s Pid:%d\\n\",\nkernel/fork.c-643-\t\t\t\t mm, resident_page_types[i],\nkernel/fork.c:644:\t\t\t\t percpu_counter_tree_precise_sum(\u0026mm-\u003erss_stat[i]),\nkernel/fork.c-645-\t\t\t\t current-\u003ecomm,\n--\nkernel/fork.c=727=void __mmdrop(struct mm_struct *mm)\n--\nkernel/fork.c-743-\tmm_destroy_cid(mm);\nkernel/fork.c:744:\tpercpu_counter_tree_destroy_many(mm-\u003erss_stat, NR_MM_COUNTERS);\nkernel/fork.c-745-\n--\nkernel/fork.c=1094=static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p)\n--\nkernel/fork.c-1149-\nkernel/fork.c:1150:\tif (percpu_counter_tree_init_many(mm-\u003erss_stat, get_rss_stat_items(mm),\nkernel/fork.c-1151-\t\t\t\t\t NR_MM_COUNTERS, RSS_STAT_BATCH_SIZE,\n--\nlib/Makefile=184=obj-$(CONFIG_SMP) += percpu_counter.o\nlib/Makefile:185:obj-$(CONFIG_SMP) += percpu_counter_tree.o\nlib/Makefile-186-obj-$(CONFIG_AUDIT_GENERIC) += audit.o\n--\nlib/percpu_counter_tree.c-136-\nlib/percpu_counter_tree.c:137:#include \u003clinux/percpu_counter_tree.h\u003e\nlib/percpu_counter_tree.c-138-#include \u003clinux/cpumask.h\u003e\n--\nlib/percpu_counter_tree.c=196=static\nlib/percpu_counter_tree.c:197:int __percpu_counter_tree_init(struct percpu_counter_tree *counter,\nlib/percpu_counter_tree.c-198-\t\t\t unsigned long batch_size, gfp_t gfp_flags,\nlib/percpu_counter_tree.c-199-\t\t\t unsigned long __percpu *level0,\nlib/percpu_counter_tree.c:200:\t\t\t struct percpu_counter_tree_level_item *items)\nlib/percpu_counter_tree.c-201-{\n--\nlib/percpu_counter_tree.c-225-/**\nlib/percpu_counter_tree.c:226: * percpu_counter_tree_init_many() - Initialize many per-CPU counter trees.\nlib/percpu_counter_tree.c-227- * @counters: An array of @nr_counters counters to initialize.\n--\nlib/percpu_counter_tree.c-230- *\t This memory is provided by the caller.\nlib/percpu_counter_tree.c:231: *\t Its size needs to be at least @nr_counters * percpu_counter_tree_items_size().\nlib/percpu_counter_tree.c-232- * @nr_counters: The number of counter trees to initialize\n--\nlib/percpu_counter_tree.c-245- */\nlib/percpu_counter_tree.c:246:int percpu_counter_tree_init_many(struct percpu_counter_tree *counters, struct percpu_counter_tree_level_item *items,\nlib/percpu_counter_tree.c-247-\t\t\t\t unsigned int nr_counters, unsigned long batch_size, gfp_t gfp_flags)\n--\nlib/percpu_counter_tree.c-250-\tsize_t counter_size = sizeof(*counters-\u003elevel0),\nlib/percpu_counter_tree.c:251:\t items_size = percpu_counter_tree_items_size();\nlib/percpu_counter_tree.c-252-\tvoid *items_iter;\n--\nlib/percpu_counter_tree.c-263-\tfor (i = 0; i \u003c nr_counters; i++) {\nlib/percpu_counter_tree.c:264:\t\tret = __percpu_counter_tree_init(\u0026counters[i], batch_size, gfp_flags, level0_iter, items_iter);\nlib/percpu_counter_tree.c-265-\t\tif (ret)\n--\nlib/percpu_counter_tree.c-275-}\nlib/percpu_counter_tree.c:276:EXPORT_SYMBOL_GPL(percpu_counter_tree_init_many);\nlib/percpu_counter_tree.c-277-\nlib/percpu_counter_tree.c-278-/**\nlib/percpu_counter_tree.c:279: * percpu_counter_tree_init() - Initialize one per-CPU counter tree.\nlib/percpu_counter_tree.c-280- * @counter: Counter to initialize.\n--\nlib/percpu_counter_tree.c-283- *\t This memory is provided by the caller.\nlib/percpu_counter_tree.c:284: *\t Its size needs to be at least percpu_counter_tree_items_size().\nlib/percpu_counter_tree.c-285- * @batch_size: The batch size is the increment step at level 0 which triggers a\n--\nlib/percpu_counter_tree.c-296- */\nlib/percpu_counter_tree.c:297:int percpu_counter_tree_init(struct percpu_counter_tree *counter, struct percpu_counter_tree_level_item *items,\nlib/percpu_counter_tree.c-298-\t\t\t unsigned long batch_size, gfp_t gfp_flags)\nlib/percpu_counter_tree.c-299-{\nlib/percpu_counter_tree.c:300:\treturn percpu_counter_tree_init_many(counter, items, 1, batch_size, gfp_flags);\nlib/percpu_counter_tree.c-301-}\nlib/percpu_counter_tree.c:302:EXPORT_SYMBOL_GPL(percpu_counter_tree_init);\nlib/percpu_counter_tree.c-303-\nlib/percpu_counter_tree.c-304-/**\nlib/percpu_counter_tree.c:305: * percpu_counter_tree_destroy_many() - Destroy many per-CPU counter trees.\nlib/percpu_counter_tree.c-306- * @counters: Array of counters trees to destroy.\n--\nlib/percpu_counter_tree.c-311-\nlib/percpu_counter_tree.c:312:void percpu_counter_tree_destroy_many(struct percpu_counter_tree *counters, unsigned int nr_counters)\nlib/percpu_counter_tree.c-313-{\n--\nlib/percpu_counter_tree.c-315-}\nlib/percpu_counter_tree.c:316:EXPORT_SYMBOL_GPL(percpu_counter_tree_destroy_many);\nlib/percpu_counter_tree.c-317-\nlib/percpu_counter_tree.c-318-/**\nlib/percpu_counter_tree.c:319: * percpu_counter_tree_destroy() - Destroy one per-CPU counter tree.\nlib/percpu_counter_tree.c-320- * @counter: Counter to destroy.\n--\nlib/percpu_counter_tree.c-323- */\nlib/percpu_counter_tree.c:324:void percpu_counter_tree_destroy(struct percpu_counter_tree *counter)\nlib/percpu_counter_tree.c-325-{\nlib/percpu_counter_tree.c:326:\treturn percpu_counter_tree_destroy_many(counter, 1);\nlib/percpu_counter_tree.c-327-}\nlib/percpu_counter_tree.c:328:EXPORT_SYMBOL_GPL(percpu_counter_tree_destroy);\nlib/percpu_counter_tree.c-329-\nlib/percpu_counter_tree.c=330=static\nlib/percpu_counter_tree.c:331:long percpu_counter_tree_carry(long orig, long res, long inc, unsigned long bit_mask)\nlib/percpu_counter_tree.c-332-{\n--\nlib/percpu_counter_tree.c=368=static\nlib/percpu_counter_tree.c:369:void percpu_counter_tree_add_slowpath(struct percpu_counter_tree *counter, long inc)\nlib/percpu_counter_tree.c-370-{\n--\nlib/percpu_counter_tree.c-373-\tunsigned long bit_mask;\nlib/percpu_counter_tree.c:374:\tstruct percpu_counter_tree_level_item *item = counter-\u003eitems;\nlib/percpu_counter_tree.c-375-\tunsigned int cpu = raw_smp_processor_id();\n--\nlib/percpu_counter_tree.c-398-\t\t\torig = res - inc;\nlib/percpu_counter_tree.c:399:\t\t\tinc = percpu_counter_tree_carry(orig, res, inc, bit_mask);\nlib/percpu_counter_tree.c-400-\t\t\tif (likely(!inc))\n--\nlib/percpu_counter_tree.c-411-/**\nlib/percpu_counter_tree.c:412: * percpu_counter_tree_add() - Add to a per-CPU counter tree.\nlib/percpu_counter_tree.c-413- * @counter: Counter added to.\n--\nlib/percpu_counter_tree.c-419- */\nlib/percpu_counter_tree.c:420:void percpu_counter_tree_add(struct percpu_counter_tree *counter, long inc)\nlib/percpu_counter_tree.c-421-{\n--\nlib/percpu_counter_tree.c-425-\torig = res - inc;\nlib/percpu_counter_tree.c:426:\tinc = percpu_counter_tree_carry(orig, res, inc, bit_mask);\nlib/percpu_counter_tree.c-427-\tif (likely(!inc))\nlib/percpu_counter_tree.c-428-\t\treturn;\nlib/percpu_counter_tree.c:429:\tpercpu_counter_tree_add_slowpath(counter, inc);\nlib/percpu_counter_tree.c-430-}\nlib/percpu_counter_tree.c:431:EXPORT_SYMBOL_GPL(percpu_counter_tree_add);\nlib/percpu_counter_tree.c-432-\nlib/percpu_counter_tree.c=433=static\nlib/percpu_counter_tree.c:434:long percpu_counter_tree_precise_sum_unbiased(struct percpu_counter_tree *counter)\nlib/percpu_counter_tree.c-435-{\n--\nlib/percpu_counter_tree.c-444-/**\nlib/percpu_counter_tree.c:445: * percpu_counter_tree_precise_sum() - Return precise counter sum.\nlib/percpu_counter_tree.c-446- * @counter: The counter to sum.\n--\nlib/percpu_counter_tree.c-453- */\nlib/percpu_counter_tree.c:454:long percpu_counter_tree_precise_sum(struct percpu_counter_tree *counter)\nlib/percpu_counter_tree.c-455-{\nlib/percpu_counter_tree.c:456:\treturn percpu_counter_tree_precise_sum_unbiased(counter) + READ_ONCE(counter-\u003ebias);\nlib/percpu_counter_tree.c-457-}\nlib/percpu_counter_tree.c:458:EXPORT_SYMBOL_GPL(percpu_counter_tree_precise_sum);\nlib/percpu_counter_tree.c-459-\n--\nlib/percpu_counter_tree.c=498=int compare_delta(long delta, unsigned long accuracy_neg, unsigned long accuracy_pos)\n--\nlib/percpu_counter_tree.c-513-/**\nlib/percpu_counter_tree.c:514: * percpu_counter_tree_approximate_compare - Approximated comparison of two counter trees.\nlib/percpu_counter_tree.c-515- * @a: First counter to compare.\n--\nlib/percpu_counter_tree.c-533- */\nlib/percpu_counter_tree.c:534:int percpu_counter_tree_approximate_compare(struct percpu_counter_tree *a, struct percpu_counter_tree *b)\nlib/percpu_counter_tree.c-535-{\nlib/percpu_counter_tree.c-536-\t/* See the range geometry above compare_delta(). */\nlib/percpu_counter_tree.c:537:\treturn compare_delta(wrapped_delta(percpu_counter_tree_approximate_sum(a),\nlib/percpu_counter_tree.c:538:\t\t\t\t\t percpu_counter_tree_approximate_sum(b)),\nlib/percpu_counter_tree.c-539-\t\t\t a-\u003eapprox_accuracy_range.over + b-\u003eapprox_accuracy_range.under,\n--\nlib/percpu_counter_tree.c-541-}\nlib/percpu_counter_tree.c:542:EXPORT_SYMBOL_GPL(percpu_counter_tree_approximate_compare);\nlib/percpu_counter_tree.c-543-\nlib/percpu_counter_tree.c-544-/**\nlib/percpu_counter_tree.c:545: * percpu_counter_tree_approximate_compare_value - Approximated comparison of a counter tree against a given value.\nlib/percpu_counter_tree.c-546- * @counter: Counter to compare.\n--\nlib/percpu_counter_tree.c-562- */\nlib/percpu_counter_tree.c:563:int percpu_counter_tree_approximate_compare_value(struct percpu_counter_tree *counter, long v)\nlib/percpu_counter_tree.c-564-{\nlib/percpu_counter_tree.c:565:\treturn compare_delta(wrapped_delta(v, percpu_counter_tree_approximate_sum(counter)),\nlib/percpu_counter_tree.c-566-\t\t\t counter-\u003eapprox_accuracy_range.under,\n--\nlib/percpu_counter_tree.c-568-}\nlib/percpu_counter_tree.c:569:EXPORT_SYMBOL_GPL(percpu_counter_tree_approximate_compare_value);\nlib/percpu_counter_tree.c-570-\nlib/percpu_counter_tree.c-571-/**\nlib/percpu_counter_tree.c:572: * percpu_counter_tree_precise_compare - Precise comparison of two counter trees.\nlib/percpu_counter_tree.c-573- * @a: First counter to compare.\n--\nlib/percpu_counter_tree.c-589- */\nlib/percpu_counter_tree.c:590:int percpu_counter_tree_precise_compare(struct percpu_counter_tree *a, struct percpu_counter_tree *b)\nlib/percpu_counter_tree.c-591-{\nlib/percpu_counter_tree.c:592:\tlong count_a = percpu_counter_tree_approximate_sum(a),\nlib/percpu_counter_tree.c:593:\t count_b = percpu_counter_tree_approximate_sum(b);\nlib/percpu_counter_tree.c-594-\tunsigned long accuracy_a, accuracy_b;\n--\nlib/percpu_counter_tree.c-617-\tif (accuracy_b \u003c accuracy_a) {\nlib/percpu_counter_tree.c:618:\t\tcount_a = percpu_counter_tree_precise_sum(a);\nlib/percpu_counter_tree.c-619-\t\tres = compare_delta(wrapped_delta(count_a, count_b),\n--\nlib/percpu_counter_tree.c-624-\t\t/* Precise sum of second counter is required. */\nlib/percpu_counter_tree.c:625:\t\tcount_b = percpu_counter_tree_precise_sum(b);\nlib/percpu_counter_tree.c-626-\t} else {\nlib/percpu_counter_tree.c:627:\t\tcount_b = percpu_counter_tree_precise_sum(b);\nlib/percpu_counter_tree.c-628-\t\tres = compare_delta(wrapped_delta(count_a, count_b),\n--\nlib/percpu_counter_tree.c-633-\t\t/* Precise sum of second counter is required. */\nlib/percpu_counter_tree.c:634:\t\tcount_a = percpu_counter_tree_precise_sum(a);\nlib/percpu_counter_tree.c-635-\t}\n--\nlib/percpu_counter_tree.c-642-}\nlib/percpu_counter_tree.c:643:EXPORT_SYMBOL_GPL(percpu_counter_tree_precise_compare);\nlib/percpu_counter_tree.c-644-\nlib/percpu_counter_tree.c-645-/**\nlib/percpu_counter_tree.c:646: * percpu_counter_tree_precise_compare_value - Precise comparison of a counter tree against a given value.\nlib/percpu_counter_tree.c-647- * @counter: Counter to compare.\n--\nlib/percpu_counter_tree.c-662- */\nlib/percpu_counter_tree.c:663:int percpu_counter_tree_precise_compare_value(struct percpu_counter_tree *counter, long v)\nlib/percpu_counter_tree.c-664-{\nlib/percpu_counter_tree.c:665:\tlong count = percpu_counter_tree_approximate_sum(counter), delta;\nlib/percpu_counter_tree.c-666-\tint res;\n--\nlib/percpu_counter_tree.c-675-\t/* Precise sum is required. */\nlib/percpu_counter_tree.c:676:\tcount = percpu_counter_tree_precise_sum(counter);\nlib/percpu_counter_tree.c-677-\tdelta = wrapped_delta(v, count);\n--\nlib/percpu_counter_tree.c-683-}\nlib/percpu_counter_tree.c:684:EXPORT_SYMBOL_GPL(percpu_counter_tree_precise_compare_value);\nlib/percpu_counter_tree.c-685-\nlib/percpu_counter_tree.c=686=static\nlib/percpu_counter_tree.c:687:void percpu_counter_tree_set_bias(struct percpu_counter_tree *counter, long bias)\nlib/percpu_counter_tree.c-688-{\n--\n\n"
} |
| 5/2 | 2026/09/06 04:16 | llm | patch-evaluator |
0mModel:gemini-3.7-flash Args: null Results: null Tokens: input: 42522 output: 198 thoughts: 874 Thoughts: **Analyzing percpu_counter_tree symbols for fuzzing potential** Okay, let's break this down systematically. My immediate goal is to identify and characterize the fuzzing potential of the `percpu_counter_tree` functionality. I'm starting by identifying non-ubiquitous and non-inlined symbols from `lib/percpu_counter_tree.c`. I've quickly listed the functions: `percpu_counter_tree_init_many`, `percpu_counter_tree_init`, `percpu_counter_tree_destroy_many`, `percpu_counter_tree_destroy`, `percpu_counter_tree_add`, `percpu_counter_tree_precise_sum`, and so on. Now, let's zoom in on `percpu_counter_tree_add`. It seems like this function is central to the patch's implementation, and while used in mm RSS counter updates, it's not universally called everywhere, making it a good candidate. I'm also including `percpu_counter_tree_init_many`, `percpu_counter_tree_precise_sum`, `percpu_counter_tree_precise_compare_value`, `percpu_counter_tree_approximate_compare_value`, `percpu_counter_tree_approximate_compare`, `percpu_counter_tree_precise_compare`, and `percpu_counter_tree_set` in the list of interesting symbols. Next, I need to check for dependencies. A quick scan of `lib/Makefile` confirms that `CONFIG_SMP` is required to compile `percpu_counter_tree.c`. Since SMP is standard on amd64 kernels, that shouldn't be a hurdle. Thus, `EnableConfigs` will contain only `"SMP"`. I've double-checked my `FocusSymbols` list, ensuring that all the functions are included. Finally, I confirmed that `percpu_counter_tree_add` is *not* inlined. It's declared as a standard C function in `include/linux/percpu_counter_tree.h` and defined in `lib/percpu_counter_tree.c`. This is exactly what I was hoping to see. Now, reviewing the overall assessment: The code is reachable within core subsystems (mm, fork, memory accounting), and thus reachable in standard QEMU/GCE amd64 kernels. We've confirmed `WorthFuzzing: true`, and the `FocusSymbols`, `EnableConfigs`, and the reasoning is all concise. We have established that the code has some fuzzing value. Alright, with all that information in hand, I'm ready to invoke `set-results`. |
| 6/2 | 2026/09/06 04:16 | tool | set-results |
0mArgs:{
"EnableConfigs": [
"SMP"
],
"FocusSymbols": [
"percpu_counter_tree_add",
"percpu_counter_tree_approximate_compare",
"percpu_counter_tree_approximate_compare_value",
"percpu_counter_tree_init_many",
"percpu_counter_tree_precise_compare",
"percpu_counter_tree_precise_compare_value",
"percpu_counter_tree_precise_sum",
"percpu_counter_tree_set"
],
"Reasoning": "The patch introduces Hierarchical Per-CPU Counters (HPCC) and converts core mm RSS counters (struct mm_struct -\u003e rss_stat) to use HPCC. This modifies core memory management accounting, process creation/destruction, and introduces new carry propagation and comparison algorithms in reachable kernel space.",
"WorthFuzzing": true
}Results: {
"EnableConfigs": [
"SMP"
],
"FocusSymbols": [
"percpu_counter_tree_add",
"percpu_counter_tree_approximate_compare",
"percpu_counter_tree_approximate_compare_value",
"percpu_counter_tree_init_many",
"percpu_counter_tree_precise_compare",
"percpu_counter_tree_precise_compare_value",
"percpu_counter_tree_precise_sum",
"percpu_counter_tree_set"
],
"Reasoning": "The patch introduces Hierarchical Per-CPU Counters (HPCC) and converts core mm RSS counters (struct mm_struct -\u003e rss_stat) to use HPCC. This modifies core memory management accounting, process creation/destruction, and introduces new carry propagation and comparison algorithms in reachable kernel space.",
"WorthFuzzing": true
} |
| 7/1 | 2026/09/06 04:16 | action | If |
0mArgs:{
"WorthFuzzing": true
}Results: null |
| 8/2 | 2026/09/06 04:16 | agent | kmsan-evaluator |
0mModel:core Args: null Results: {
"KMSANReasoning": "The patch series introduces Hierarchical Per-CPU Counters (HPCC) and transitions mm_struct RSS accounting (rss_stat) to use HPCC instead of standard percpu counters.\n\nEvaluation of uninitialized memory risk:\n1. All allocated structures and tree items are fully zero-initialized before use:\n - In `percpu_counter_tree_init_many()`, `items` are explicitly zeroed via `memset()`, and `level0` per-CPU memory is allocated via `__alloc_percpu_gfp()` which returns zero-filled memory. All struct fields of `struct percpu_counter_tree` are explicitly initialized.\n - For `init_mm`, the statically allocated flexible array containing tree items is zero-initialized via `MM_STRUCT_FLEXIBLE_ARRAY_INIT`.\n - In `mm_init()`, RSS counters are initialized via `percpu_counter_tree_init_many()` with `get_rss_stat_items(mm)`.\n2. No kernel structures or buffers with uninitialized padding or fields are exposed or copied to user space (counter values are read as scalar `long` / `unsigned long` values and formatted into strings or proc/sysfs seq_files).\n3. No branching or calculations depend on uninitialized variables or memory.\n4. Any potential issues in this patch series (such as indexing errors in the tree hierarchy, arithmetic/carry calculation bugs, race conditions, or out-of-bounds accesses) fall squarely under KASAN, LOCKDEP, and KUnit testing rather than uninitialized memory detectors.\n\nTherefore, dedicated KMSAN fuzzing is not required (NeedsKMSAN=false).",
"NeedsKMSAN": false
}Instruction: You are an expert Linux kernel security engineer specializing in kernel memory
error detectors (KASAN and KMSAN). Your job is to review the provided patch series and
determine if the code changes justify spawning a dedicated KMSAN (KernelMemorySanitizer)
fuzzing session in addition to standard KASAN fuzzing.
CRITICAL DISTINCTION BETWEEN KASAN AND KMSAN:
- Standard KASAN kernel builds (upstream-apparmor-kasan.config) already enable
a comprehensive suite of debugging tools and sanitizers, including KASAN
(out-of-bounds accesses, use-after-free, double free, invalid free), LOCKDEP
(locking bugs and deadlocks), UB-sanitizers, and memory corruption checks.
- KMSAN (KernelMemorySanitizer) detects reads of UNINITIALIZED memory (stack, heap,
or page allocations) and kernel-to-user memory info-leaks.
Rule: THERE IS NO SENSE IN RUNNING A KMSAN SESSION IF A BUG CAN BE CAUGHT BY KASAN,
LOCKDEP, OR OTHER STANDARD BUG DETECTORS.
A dedicated KMSAN fuzzing session incurs significant resource costs. You must ONLY
set NeedsKMSAN=true if the code changes introduce or expose UNINITIALIZED MEMORY risks
that are detected ONLY by KMSAN.
Look holistically at the patch series and surrounding code. Even if no direct
uninitialized field accesses or new buffer allocations are added in the diff itself,
a patch may alter control flow, bounds checking, or data length calculations in ways
that change how the rest of the code operates on existing buffers (e.g. allowing
uninitialized stack/heap memory to be read, copied to user space, or used in control
flow). Do not hesitate to use your code access tools to inspect the surrounding code,
called functions, and callers.
Set NeedsKMSAN=true ONLY IF the patch introduces or modifies:
1. Kernel structures sent to user space (via copy_to_user, put_user, netlink skb
attributes, ioctl output arguments, socket options, or BPF buffers) where fields
or structure padding might not be fully initialized/zeroed.
2. Conditional logic or branching that depends on potentially uninitialized variables
or struct fields.
3. Allocation or initialization of complex data structures where uninitialized fields
could be read later in reachable code paths.
4. Bounds checks, lengths, or logic in a way that allows surrounding code to access
uninitialized bytes of existing buffers.
Set NeedsKMSAN=false IF:
- The code changes primarily risk out-of-bounds access, array overflows, NULL pointer
dereferences, locking deadlocks, or use-after-free bugs (these are already caught
by KASAN, LOCKDEP, or standard bug detectors).
- All stack/heap structures touched or introduced by the patch are fully zeroed
or initialized (e.g. using = {0}, memset, kzalloc) before being read or copied.
- The patch does not introduce any risk of uninitialized memory usage or info-leaks.
Use your code access tools to inspect the surrounding code if necessary, then provide
detailed KMSANReasoning contrasting KASAN vs KMSAN applicability for this patch.
Prefer calling several tools at the same time to save round-trips.
Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.
Prompt: Target architecture: amd64
For your convenience, here is the diff of the changes:
commit dde82b75d01b1954902ba04c172efd2ae7beff50
Author: syz-cluster <triage@syzkaller.com>
Date: Sun Sep 6 04:16:19 2026 +0000
syz-cluster: applied patch under review
diff --git a/Documentation/core-api/percpu-counter-tree.rst b/Documentation/core-api/percpu-counter-tree.rst
new file mode 100644
index 0000000000000..878cbd85304cf
--- /dev/null
+++ b/Documentation/core-api/percpu-counter-tree.rst
@@ -0,0 +1,82 @@
+========================================
+The Hierarchical Per-CPU Counters (HPCC)
+========================================
+
+:Author: Mathieu Desnoyers
+
+Introduction
+============
+
+Counters come in many varieties, each with their own trade offs:
+
+ * A global atomic counter provides a fast read access to the current
+ sum, at the expense of cache-line bouncing on updates. This leads to
+ poor performance of frequent updates from various cores on large SMP
+ systems.
+
+ * A per-cpu split counter provides fast updates to per-cpu counters,
+ at the expense of a slower aggregation (sum). The sum operation needs
+ to iterate over all per-cpu counters to calculate the current total.
+
+The hierarchical per-cpu counters attempt to provide the best of both
+worlds (fast updates, and fast sum) by relaxing requirements on the sum
+accuracy. It allows quickly querying an approximated sum value, along
+with the possible min/max ranges of the associated precise sum. The
+exact precise sum can still be calculated with an iteration on all
+per-cpu counter, but the availability of an approximated sum value with
+possible precise sum min/max ranges allows eliminating candidates which
+are certainly outside of a known target range without the overhead of
+precise sums.
+
+Overview
+========
+
+The herarchical per-cpu counters are organized as a tree with the tree
+root at the bottom (last level) and the first level of the tree
+consisting of per-cpu counters.
+
+The intermediate tree levels contain carry propagation counters. When
+reaching a threshold (batch size), the carry is propagated down the
+tree.
+
+This allows reading an approximated value at the root, which has a
+bounded accuracy (minimum/maximum possible precise sum range) determined
+by the tree topology.
+
+The bounded accuracy is a property of the settled tree. While updates are
+in flight the transient error can exceed it, bounded by the number of
+updaters and their increment size.
+
+Use Cases
+=========
+
+Use cases HPCC is meant to handle invove tracking resources which are
+used across many CPUs to quickly sum as feedback for decision making to
+apply throttling, quota limits, sort tasks, and perform memory or task
+migration decisions. When considering approximated sums within the
+accuracy range of the decision threshold, the user can either:
+
+ * Be conservative and fast: Consider that the sum has reached the
+ limit as soon as the given limit is within the approximation range.
+
+ * Be aggressive and fast: Consider that the sum is over the
+ limit only when the approximation range is over the given limit.
+
+ * Be precise and slow: Do a precise comparison with the limit, which
+ requires a precise sum when the limit is within the approximated
+ range.
+
+One use-case for these hierarchical counters is to implement a two-pass
+algorithm to speed up sorting picking a maximum/minimunm sum value from
+a set. A first pass compares the approximated values, and then a second
+pass only needs the precise sum for counter trees which are within the
+possible precise sum range of the counter tree chosen by the first pass.
+
+Because the comparison is on the wrapped difference, it is not transitive:
+ordering is undefined over a set of counters spanning more than LONG_MAX.
+
+Functions and structures
+========================
+
+.. kernel-doc:: include/linux/percpu_counter_tree.h
+.. kernel-doc:: lib/percpu_counter_tree.c
diff --git a/include/linux/mm.h b/include/linux/mm.h
index dd09c438fa23e..ffbcd18c4ecb3 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -3447,38 +3447,48 @@ static inline bool get_user_page_fast_only(unsigned long addr,
{
return get_user_pages_fast_only(addr, 1, gup_flags, pagep) == 1;
}
+
+static inline struct percpu_counter_tree_level_item *get_rss_stat_items(struct mm_struct *mm)
+{
+ unsigned long ptr = (unsigned long)mm;
+
+ ptr += offsetof(struct mm_struct, flexible_array);
+ ptr += get_rss_stat_items_offset();
+ return (struct percpu_counter_tree_level_item *)ptr;
+}
+
/*
* per-process(per-mm_struct) statistics.
*/
static inline unsigned long get_mm_counter(struct mm_struct *mm, int member)
{
- return percpu_counter_read_positive(&mm->rss_stat[member]);
+ return percpu_counter_tree_approximate_sum_positive(&mm->rss_stat[member]);
}
static inline unsigned long get_mm_counter_sum(struct mm_struct *mm, int member)
{
- return percpu_counter_sum_positive(&mm->rss_stat[member]);
+ return percpu_counter_tree_precise_sum_positive(&mm->rss_stat[member]);
}
void mm_trace_rss_stat(struct mm_struct *mm, int member);
static inline void add_mm_counter(struct mm_struct *mm, int member, long value)
{
- percpu_counter_add(&mm->rss_stat[member], value);
+ percpu_counter_tree_add(&mm->rss_stat[member], value);
mm_trace_rss_stat(mm, member);
}
static inline void inc_mm_counter(struct mm_struct *mm, int member)
{
- percpu_counter_inc(&mm->rss_stat[member]);
+ percpu_counter_tree_add(&mm->rss_stat[member], 1);
mm_trace_rss_stat(mm, member);
}
static inline void dec_mm_counter(struct mm_struct *mm, int member)
{
- percpu_counter_dec(&mm->rss_stat[member]);
+ percpu_counter_tree_add(&mm->rss_stat[member], -1);
mm_trace_rss_stat(mm, member);
}
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index 6d815f6440c94..b26160b85f93a 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -18,7 +18,7 @@
#include <linux/page-flags-layout.h>
#include <linux/workqueue.h>
#include <linux/seqlock.h>
-#include <linux/percpu_counter.h>
+#include <linux/percpu_counter_tree.h>
#include <linux/types.h>
#include <linux/futex_types.h>
#include <linux/rseq_types.h>
@@ -1170,6 +1170,19 @@ typedef struct {
DECLARE_BITMAP(__mm_flags, NUM_MM_FLAG_BITS);
} __private mm_flags_t;
+/*
+ * The alignment of the mm_struct flexible array is based on the largest
+ * alignment of its content:
+ * __alignof__(struct percpu_counter_tree_level_item) provides a
+ * cacheline aligned alignment on SMP systems, else alignment on
+ * unsigned long on UP systems.
+ */
+#ifdef CONFIG_SMP
+# define __mm_struct_flexible_array_aligned __aligned(__alignof__(struct percpu_counter_tree_level_item))
+#else
+# define __mm_struct_flexible_array_aligned __aligned(__alignof__(unsigned long))
+#endif
+
struct kioctx_table;
struct iommu_mm_data;
struct mm_struct {
@@ -1312,7 +1325,7 @@ struct mm_struct {
unsigned long saved_e_flags;
#endif
- struct percpu_counter rss_stat[NR_MM_COUNTERS];
+ struct percpu_counter_tree rss_stat[NR_MM_COUNTERS];
struct linux_binfmt *binfmt;
@@ -1422,10 +1435,14 @@ struct mm_struct {
} __randomize_layout;
/*
- * The mm_cpumask needs to be at the end of mm_struct, because it
- * is dynamically sized based on nr_cpu_ids.
+ * The mm_cpumask, mm_cid masks, and rss hierarchical counter
+ * items need to be at the end of mm_struct, because they are
+ * dynamically sized based on nr_cpu_ids.
+ * The HPCC counter tree items are placed last and aligned
+ * with PERCPU_COUNTER_TREE_ITEMS_ALIGN to satisfy their
+ * cacheline alignment requirement.
*/
- char flexible_array[] __aligned(__alignof__(unsigned long));
+ char flexible_array[] __mm_struct_flexible_array_aligned;
};
/* Copy value to the first system word of mm flags, non-atomically. */
@@ -1462,18 +1479,12 @@ static inline void __mm_flags_set_mask_bits_word(struct mm_struct *mm,
MT_FLAGS_USE_RCU)
extern struct mm_struct init_mm;
-#define MM_STRUCT_FLEXIBLE_ARRAY_INIT \
-{ \
- [0 ... sizeof(cpumask_t) + MM_CID_STATIC_SIZE - 1] = 0 \
-}
-
-/* Pointer magic because the dynamic array size confuses some compilers. */
-static inline void mm_init_cpumask(struct mm_struct *mm)
-{
- unsigned long cpu_bitmap = (unsigned long)mm;
-
- cpu_bitmap += offsetof(struct mm_struct, flexible_array);
- cpumask_clear((struct cpumask *)cpu_bitmap);
+#define MM_STRUCT_FLEXIBLE_ARRAY_INIT \
+{ \
+ [0 ... PERCPU_COUNTER_TREE_ITEMS_ALIGN( \
+ sizeof(cpumask_t) + MM_CID_STATIC_SIZE) \
+ + (PERCPU_COUNTER_TREE_ITEMS_STATIC_SIZE * NR_MM_COUNTERS) \
+ - 1] = 0 \
}
/* Future-safe accessor for struct mm_struct's cpu_vm_mask. */
@@ -1482,6 +1493,11 @@ static inline cpumask_t *mm_cpumask(struct mm_struct *mm)
return (struct cpumask *)&mm->flexible_array;
}
+static inline void mm_init_cpumask(struct mm_struct *mm)
+{
+ cpumask_clear((struct cpumask *)mm_cpumask(mm));
+}
+
#ifdef CONFIG_LRU_GEN
struct lru_gen_mm_list {
@@ -1653,6 +1669,21 @@ static inline void mm_destroy_sched(struct mm_struct *mm) { }
#endif /* CONFIG_SCHED_CACHE */
+static inline size_t get_rss_stat_items_size(void)
+{
+ return percpu_counter_tree_items_size() * NR_MM_COUNTERS;
+}
+
+/*
+ * Return the offset of the RSS stat HPCC items within the mm_struct
+ * flexible array. The items are placed after the cpumask and mm_cid,
+ * aligned to the cacheline boundary required by the tree level items.
+ */
+static inline size_t get_rss_stat_items_offset(void)
+{
+ return PERCPU_COUNTER_TREE_ITEMS_ALIGN(cpumask_size() + mm_cid_size());
+}
+
struct mmu_gather;
extern void tlb_gather_mmu(struct mmu_gather *tlb, struct mm_struct *mm);
extern void tlb_gather_mmu_fullmm(struct mmu_gather *tlb, struct mm_struct *mm);
diff --git a/include/linux/percpu_counter_tree.h b/include/linux/percpu_counter_tree.h
new file mode 100644
index 0000000000000..d593806b4ab26
--- /dev/null
+++ b/include/linux/percpu_counter_tree.h
@@ -0,0 +1,391 @@
+/* SPDX-License-Identifier: GPL-2.0+ OR MIT */
+/* SPDX-FileCopyrightText: 2025 Mathieu Desnoyers <mathieu.desnoyers@efficios.com> */
+
+#ifndef _PERCPU_COUNTER_TREE_H
+#define _PERCPU_COUNTER_TREE_H
+
+#include <linux/preempt.h>
+#include <linux/atomic.h>
+#include <linux/percpu.h>
+
+#ifdef CONFIG_SMP
+
+#if NR_CPUS == (1U << 0)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 0
+#elif NR_CPUS <= (1U << 1)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 1
+#elif NR_CPUS <= (1U << 2)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 3
+#elif NR_CPUS <= (1U << 3)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 7
+#elif NR_CPUS <= (1U << 4)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 7
+#elif NR_CPUS <= (1U << 5)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 11
+#elif NR_CPUS <= (1U << 6)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 21
+#elif NR_CPUS <= (1U << 7)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 21
+#elif NR_CPUS <= (1U << 8)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 37
+#elif NR_CPUS <= (1U << 9)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 73
+#elif NR_CPUS <= (1U << 10)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 149
+#elif NR_CPUS <= (1U << 11)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 293
+#elif NR_CPUS <= (1U << 12)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 585
+#elif NR_CPUS <= (1U << 13)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 1173
+#elif NR_CPUS <= (1U << 14)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 2341
+#elif NR_CPUS <= (1U << 15)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 4681
+#elif NR_CPUS <= (1U << 16)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 4681
+#elif NR_CPUS <= (1U << 17)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 8777
+#elif NR_CPUS <= (1U << 18)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 17481
+#elif NR_CPUS <= (1U << 19)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 34953
+#elif NR_CPUS <= (1U << 20)
+# define PERCPU_COUNTER_TREE_STATIC_NR_ITEMS 69905
+#else
+# error "Unsupported number of CPUs."
+#endif
+
+struct percpu_counter_tree_level_item {
+ atomic_long_t count; /*
+ * Count the number of carry for this tree item.
+ * The carry counter is kept at the order of the
+ * carry accounted for at this tree level.
+ */
+} ____cacheline_aligned_in_smp;
+
+#define PERCPU_COUNTER_TREE_ITEMS_STATIC_SIZE \
+ (PERCPU_COUNTER_TREE_STATIC_NR_ITEMS * sizeof(struct percpu_counter_tree_level_item))
+#define PERCPU_COUNTER_TREE_ITEMS_ALIGN(offset) \
+ ALIGN((offset), __alignof__(struct percpu_counter_tree_level_item))
+
+struct percpu_counter_tree {
+ /* Fast-path fields. */
+ unsigned long __percpu *level0; /* Pointer to per-CPU split counters (tree level 0). */
+ unsigned long level0_bit_mask; /* Bit mask to apply to detect carry propagation from tree level 0. */
+ union {
+ unsigned long *i; /* Approximate sum for single-CPU topology. */
+ atomic_long_t *a; /* Approximate sum for SMP topology. */
+ } approx_sum;
+ long bias; /* Bias to apply to counter precise and approximate values. */
+
+ /* Slow-path fields. */
+ struct percpu_counter_tree_level_item *items; /* Array of tree items for levels 1 to N. */
+ unsigned long batch_size; /*
+ * The batch size is the increment step at level 0 which
+ * triggers a carry propagation. The batch size is required
+ * to be greater than 1, and a power of 2.
+ */
+ /*
+ * The tree approximate sum is guaranteed to be within this accuracy range:
+ * (precise_sum - approx_accuracy_range.under) <= approx_sum <= (precise_sum + approx_accuracy_range.over).
+ * This accuracy is derived from the hardware topology and the tree batch_size.
+ * The "under" accuracy is larger than the "over" accuracy because the negative range of a
+ * two's complement signed integer is one unit larger than the positive range. This delta
+ * is summed for each tree item, which leads to a significantly larger "under" accuracy range
+ * compared to the "over" accuracy range. The accuracy range is a bound on a settled counter:
+ * one with no update in progress.
+ */
+ struct {
+ unsigned long under;
+ unsigned long over;
+ } approx_accuracy_range;
+};
+
+extern size_t __percpu_counter_tree_items_size __ro_after_init;
+
+/*
+ * percpu_counter_tree_items_size - Query the size required for counter tree items.
+ *
+ * Query the size of the memory area required to hold the counter tree
+ * items. This depends on the hardware topology and is invariant after
+ * boot.
+ *
+ * Return: Size required to hold tree items.
+ */
+static inline
+size_t percpu_counter_tree_items_size(void)
+{
+ WARN_ON_ONCE(__percpu_counter_tree_items_size == SIZE_MAX);
+ return __percpu_counter_tree_items_size;
+}
+
+int percpu_counter_tree_init_many(struct percpu_counter_tree *counters, struct percpu_counter_tree_level_item *items,
+ unsigned int nr_counters, unsigned long batch_size, gfp_t gfp_flags);
+int percpu_counter_tree_init(struct percpu_counter_tree *counter, struct percpu_counter_tree_level_item *items,
+ unsigned long batch_size, gfp_t gfp_flags);
+void percpu_counter_tree_destroy_many(struct percpu_counter_tree *counter, unsigned int nr_counters);
+void percpu_counter_tree_destroy(struct percpu_counter_tree *counter);
+void percpu_counter_tree_add(struct percpu_counter_tree *counter, long inc);
+long percpu_counter_tree_precise_sum(struct percpu_counter_tree *counter);
+int percpu_counter_tree_approximate_compare(struct percpu_counter_tree *a, struct percpu_counter_tree *b);
+int percpu_counter_tree_approximate_compare_value(struct percpu_counter_tree *counter, long v);
+int percpu_counter_tree_precise_compare(struct percpu_counter_tree *a, struct percpu_counter_tree *b);
+int percpu_counter_tree_precise_compare_value(struct percpu_counter_tree *counter, long v);
+void percpu_counter_tree_set(struct percpu_counter_tree *counter, long v);
+int percpu_counter_tree_subsystem_init(void);
+
+/**
+ * percpu_counter_tree_approximate_sum() - Return approximate counter sum.
+ * @counter: The counter to sum.
+ *
+ * Querying the approximate sum is fast, but it is only accurate within
+ * the bounds delimited by percpu_counter_tree_approximate_accuracy_range().
+ * This is meant to be used when speed is preferred over accuracy.
+ *
+ * Return: The current approximate counter sum.
+ */
+static inline
+long percpu_counter_tree_approximate_sum(struct percpu_counter_tree *counter)
+{
+ unsigned long v;
+
+ if (!counter->level0_bit_mask)
+ v = READ_ONCE(*counter->approx_sum.i);
+ else
+ v = atomic_long_read(counter->approx_sum.a);
+ return (long) (v + (unsigned long)READ_ONCE(counter->bias));
+}
+
+/**
+ * percpu_counter_tree_approximate_accuracy_range - Query the accuracy range for a counter tree.
+ * @counter: Counter to query.
+ * @under: Pointer to a variable to be incremented of the approximation
+ * accuracy range below the precise sum.
+ * @over: Pointer to a variable to be incremented of the approximation
+ * accuracy range above the precise sum.
+ *
+ * Query the accuracy range limits for the counter.
+ * Because of two's complement binary representation, the "under" range is typically
+ * slightly larger than the "over" range.
+ * Those values are derived from the hardware topology and the counter tree batch size.
+ * They are invariant for a given counter tree.
+ * The bounds describe a settled counter: one with no update in progress.
+ * An update in flight between its per-CPU increment and its carry propagation is not covered.
+ * Using this function should not be typically required, see the following functions instead:
+ * * percpu_counter_tree_approximate_compare(),
+ * * percpu_counter_tree_approximate_compare_value(),
+ * * percpu_counter_tree_precise_compare(),
+ * * percpu_counter_tree_precise_compare_value().
+ */
+static inline
+void percpu_counter_tree_approximate_accuracy_range(struct percpu_counter_tree *counter,
+ unsigned long *under, unsigned long *over)
+{
+ *under += counter->approx_accuracy_range.under;
+ *over += counter->approx_accuracy_range.over;
+}
+
+#else /* !CONFIG_SMP */
+
+#define PERCPU_COUNTER_TREE_ITEMS_STATIC_SIZE 0
+#define PERCPU_COUNTER_TREE_ITEMS_ALIGN(offset) (offset)
+
+struct percpu_counter_tree_level_item;
+
+struct percpu_counter_tree {
+ atomic_long_t count;
+};
+
+static inline
+size_t percpu_counter_tree_items_size(void)
+{
+ return 0;
+}
+
+static inline
+int percpu_counter_tree_init_many(struct percpu_counter_tree *counters, struct percpu_counter_tree_level_item *items,
+ unsigned int nr_counters, unsigned long batch_size, gfp_t gfp_flags)
+{
+ for (unsigned int i = 0; i < nr_counters; i++)
+ atomic_long_set(&counters[i].count, 0);
+ return 0;
+}
+
+static inline
+int percpu_counter_tree_init(struct percpu_counter_tree *counter, struct percpu_counter_tree_level_item *items,
+ unsigned long batch_size, gfp_t gfp_flags)
+{
+ return percpu_counter_tree_init_many(counter, items, 1, batch_size, gfp_flags);
+}
+
+static inline
+void percpu_counter_tree_destroy_many(struct percpu_counter_tree *counter, unsigned int nr_counters)
+{
+}
+
+static inline
+void percpu_counter_tree_destroy(struct percpu_counter_tree *counter)
+{
+}
+
+static inline
+long percpu_counter_tree_precise_sum(struct percpu_counter_tree *counter)
+{
+ return atomic_long_read(&counter->count);
+}
+
+static inline
+int percpu_counter_tree_precise_compare(struct percpu_counter_tree *a, struct percpu_counter_tree *b)
+{
+ long count_a = percpu_counter_tree_precise_sum(a),
+ count_b = percpu_counter_tree_precise_sum(b);
+
+ if (count_a == count_b)
+ return 0;
+ if (count_a < count_b)
+ return -1;
+ return 1;
+}
+
+static inline
+int percpu_counter_tree_precise_compare_value(struct percpu_counter_tree *counter, long v)
+{
+ long count = percpu_counter_tree_precise_sum(counter);
+
+ if (count == v)
+ return 0;
+ if (count < v)
+ return -1;
+ return 1;
+}
+
+static inline
+int percpu_counter_tree_approximate_compare(struct percpu_counter_tree *a, struct percpu_counter_tree *b)
+{
+ return percpu_counter_tree_precise_compare(a, b);
+}
+
+static inline
+int percpu_counter_tree_approximate_compare_value(struct percpu_counter_tree *counter, long v)
+{
+ return percpu_counter_tree_precise_compare_value(counter, v);
+}
+
+static inline
+void percpu_counter_tree_set(struct percpu_counter_tree *counter, long v)
+{
+ atomic_long_set(&counter->count, v);
+}
+
+static inline
+void percpu_counter_tree_approximate_accuracy_range(struct percpu_counter_tree *counter,
+ unsigned long *under, unsigned long *over)
+{
+}
+
+static inline
+void percpu_counter_tree_add(struct percpu_counter_tree *counter, long inc)
+{
+ atomic_long_add(inc, &counter->count);
+}
+
+static inline
+long percpu_counter_tree_approximate_sum(struct percpu_counter_tree *counter)
+{
+ return percpu_counter_tree_precise_sum(counter);
+}
+
+static inline
+int percpu_counter_tree_subsystem_init(void)
+{
+ return 0;
+}
+
+#endif /* CONFIG_SMP */
+
+/**
+ * percpu_counter_tree_approximate_sum_positive() - Return a positive approximate counter sum.
+ * @counter: The counter to sum.
+ *
+ * Return an approximate counter sum which is guaranteed to be greater
+ * or equal to 0.
+ *
+ * Return: The current positive approximate counter sum.
+ */
+static inline
+long percpu_counter_tree_approximate_sum_positive(struct percpu_counter_tree *counter)
+{
+ long v = percpu_counter_tree_approximate_sum(counter);
+ return v > 0 ? v : 0;
+}
+
+/**
+ * percpu_counter_tree_precise_sum_positive() - Return a positive precise counter sum.
+ * @counter: The counter to sum.
+ *
+ * Return a precise counter sum which is guaranteed to be greater
+ * or equal to 0.
+ *
+ * Return: The current positive precise counter sum.
+ */
+static inline
+long percpu_counter_tree_precise_sum_positive(struct percpu_counter_tree *counter)
+{
+ long v = percpu_counter_tree_precise_sum(counter);
+ return v > 0 ? v : 0;
+}
+
+/**
+ * percpu_counter_tree_approximate_min_max_range() - Return the approximation min and max precise values.
+ * @approx_sum: Approximated sum.
+ * @under: Tree accuracy range (under).
+ * @over: Tree accuracy range (over).
+ * @precise_min: Minimum possible value for precise sum (output).
+ * @precise_max: Maximum possible value for precise sum (output).
+ *
+ * Calculate the minimum and maximum precise values for a given approximation
+ * and (under, over) accuracy range. The accuracy range is a bound on a settled
+ * tree: one with no update in progress.
+ *
+ * The range of the approximation as a function of the precise sum is expressed as:
+ *
+ * approx_sum >= precise_sum - approx_accuracy_range.under
+ * approx_sum <= precise_sum + approx_accuracy_range.over
+ *
+ * Therefore, the range of the precise sum as a function of the approximation is expressed as:
+ *
+ * precise_sum <= approx_sum + approx_accuracy_range.under
+ * precise_sum >= approx_sum - approx_accuracy_range.over
+ */
+static inline
+void percpu_counter_tree_approximate_min_max_range(long approx_sum, unsigned long under, unsigned long over,
+ long *precise_min, long *precise_max)
+{
+ *precise_min = approx_sum - over;
+ *precise_max = approx_sum + under;
+}
+
+/**
+ * percpu_counter_tree_approximate_min_max() - Return the tree approximation, min and max possible precise values.
+ * @counter: The counter to sum.
+ * @approx_sum: Approximate sum (output).
+ * @precise_min: Minimum possible value for precise sum (output).
+ * @precise_max: Maximum possible value for precise sum (output).
+ *
+ * Return the approximate sum, minimum and maximum precise values for
+ * a counter.
+ */
+static inline
+void percpu_counter_tree_approximate_min_max(struct percpu_counter_tree *counter,
+ long *approx_sum, long *precise_min, long *precise_max)
+{
+ unsigned long under = 0, over = 0;
+ long v = percpu_counter_tree_approximate_sum(counter);
+
+ percpu_counter_tree_approximate_accuracy_range(counter, &under, &over);
+ percpu_counter_tree_approximate_min_max_range(v, under, over, precise_min, precise_max);
+ *approx_sum = v;
+}
+
+#endif /* _PERCPU_COUNTER_TREE_H */
diff --git a/include/trace/events/kmem.h b/include/trace/events/kmem.h
index cd7920c81f85b..290ccb9fd25dd 100644
--- a/include/trace/events/kmem.h
+++ b/include/trace/events/kmem.h
@@ -448,7 +448,7 @@ TRACE_EVENT(rss_stat,
*/
__entry->curr = current->mm == mm && !(current->flags & PF_KTHREAD);
__entry->member = member;
- __entry->size = (percpu_counter_sum_positive(&mm->rss_stat[member])
+ __entry->size = (percpu_counter_tree_approximate_sum_positive(&mm->rss_stat[member])
<< PAGE_SHIFT);
),
diff --git a/init/main.c b/init/main.c
index 2613d3f9b3ce9..845cb57e90965 100644
--- a/init/main.c
+++ b/init/main.c
@@ -108,6 +108,7 @@
#include <linux/time_namespace.h>
#include <linux/unaligned.h>
#include <linux/vdso_datastore.h>
+#include <linux/percpu_counter_tree.h>
#include <net/net_namespace.h>
#include <asm/io.h>
@@ -1010,6 +1011,7 @@ void start_kernel(void)
setup_boot_config();
setup_command_line(command_line);
setup_nr_cpu_ids();
+ percpu_counter_tree_subsystem_init();
setup_per_cpu_areas();
smp_prepare_boot_cpu(); /* arch-specific boot-cpu hooks */
early_numa_node_init();
diff --git a/kernel/fork.c b/kernel/fork.c
index 416758c8a3d43..db398b6c52886 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -137,6 +137,11 @@
*/
#define MAX_THREADS FUTEX_TID_MASK
+/*
+ * Batch size of rss stat approximation
+ */
+#define RSS_STAT_BATCH_SIZE 32
+
/*
* Protected counters by write_lock_irq(&tasklist_lock)
*/
@@ -633,14 +638,12 @@ static void check_mm(struct mm_struct *mm)
"Please make sure 'struct resident_page_types[]' is updated as well");
for (i = 0; i < NR_MM_COUNTERS; i++) {
- long x = percpu_counter_sum(&mm->rss_stat[i]);
-
- if (unlikely(x)) {
+ if (unlikely(percpu_counter_tree_precise_compare_value(&mm->rss_stat[i], 0) != 0))
pr_alert("BUG: Bad rss-counter state mm:%p type:%s val:%ld Comm:%s Pid:%d\n",
- mm, resident_page_types[i], x,
+ mm, resident_page_types[i],
+ percpu_counter_tree_precise_sum(&mm->rss_stat[i]),
current->comm,
task_pid_nr(current));
- }
}
if (mm_pgtables_bytes(mm))
@@ -738,7 +741,7 @@ void __mmdrop(struct mm_struct *mm)
check_mm(mm);
mm_pasid_drop(mm);
mm_destroy_cid(mm);
- percpu_counter_destroy_many(mm->rss_stat, NR_MM_COUNTERS);
+ percpu_counter_tree_destroy_many(mm->rss_stat, NR_MM_COUNTERS);
free_mm(mm);
}
@@ -1144,8 +1147,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p)
if (mm_alloc_sched(mm))
goto fail_sched;
- if (percpu_counter_init_many(mm->rss_stat, 0, GFP_KERNEL_ACCOUNT,
- NR_MM_COUNTERS))
+ if (percpu_counter_tree_init_many(mm->rss_stat, get_rss_stat_items(mm),
+ NR_MM_COUNTERS, RSS_STAT_BATCH_SIZE,
+ GFP_KERNEL_ACCOUNT))
goto fail_pcpu;
lru_gen_init_mm(mm);
@@ -3122,7 +3126,9 @@ void __init mm_cache_init(void)
* dynamically sized based on the maximum CPU number this system
* can have, taking hotplug into account (nr_cpu_ids).
*/
- mm_size = sizeof(struct mm_struct) + cpumask_size() + mm_cid_size();
+ mm_size = sizeof(struct mm_struct) +
+ PERCPU_COUNTER_TREE_ITEMS_ALIGN(cpumask_size() + mm_cid_size()) +
+ get_rss_stat_items_size();
mm_cachep = kmem_cache_create_usercopy("mm_struct",
mm_size, ARCH_MIN_MMSTRUCT_ALIGN,
diff --git a/lib/Kconfig b/lib/Kconfig
index 4e6b34c3346d5..c0524258b2058 100644
--- a/lib/Kconfig
+++ b/lib/Kconfig
@@ -41,6 +41,18 @@ config PACKING_KUNIT_TEST
When in doubt, say N.
+config PERCPU_COUNTER_TREE_KUNIT_TEST
+ tristate "KUnit tests for hierarchical per-cpu counters" if !KUNIT_ALL_TESTS
+ depends on KUNIT
+ default KUNIT_ALL_TESTS
+ help
+ This builds Kunit tests for the hierarchical per-cpu counters.
+
+ For more information on KUnit and unit tests in general,
+ please refer to the KUnit documentation in Documentation/dev-tools/kunit/.
+
+ When in doubt, say N.
+
config BITREVERSE
tristate
select GENERIC_BITREVERSE if !HAVE_ARCH_BITREVERSE
diff --git a/lib/Makefile b/lib/Makefile
index dfab958327c5c..f1d0cb831630b 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -182,6 +182,7 @@ obj-$(CONFIG_TEXTSEARCH_KMP) += ts_kmp.o
obj-$(CONFIG_TEXTSEARCH_BM) += ts_bm.o
obj-$(CONFIG_TEXTSEARCH_FSM) += ts_fsm.o
obj-$(CONFIG_SMP) += percpu_counter.o
+obj-$(CONFIG_SMP) += percpu_counter_tree.o
obj-$(CONFIG_AUDIT_GENERIC) += audit.o
obj-$(CONFIG_AUDIT_COMPAT_GENERIC) += compat_audit.o
diff --git a/lib/percpu_counter_tree.c b/lib/percpu_counter_tree.c
new file mode 100644
index 0000000000000..26bfe183a882d
--- /dev/null
+++ b/lib/percpu_counter_tree.c
@@ -0,0 +1,747 @@
+// SPDX-License-Identifier: GPL-2.0+ OR MIT
+// SPDX-FileCopyrightText: 2025 Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
+
+/*
+ * Split Counters With Tree Approximation Propagation
+ *
+ * * Propagation diagram when reaching batch size thresholds (± batch size):
+ *
+ * Example diagram for 8 CPUs:
+ *
+ * log2(8) = 3 levels
+ *
+ * At each level, each pair propagates its values to the next level when
+ * reaching the batch size thresholds.
+ *
+ * Counters at levels 0, 1, 2 can be kept on a single byte ([-128 .. +127] range),
+ * although it may be relevant to keep them on "long" counters for
+ * simplicity. (complexity vs memory footprint tradeoff)
+ *
+ * Counter at level 3 can be kept on a "long" counter.
+ *
+ * Level 0: 0 1 2 3 4 5 6 7
+ * | / | / | / | /
+ * | / | / | / | /
+ * | / | / | / | /
+ * Level 1: 0 1 2 3
+ * | / | /
+ * | / | /
+ * | / | /
+ * Level 2: 0 1
+ * | /
+ * | /
+ * | /
+ * Level 3: 0
+ *
+ * * Approximation accuracy:
+ *
+ * BATCH(level N): Level N batch size.
+ *
+ * Example for BATCH(level 0) = 32.
+ *
+ * BATCH(level 0) = 32
+ * BATCH(level 1) = 64
+ * BATCH(level 2) = 128
+ * BATCH(level N) = BATCH(level 0) * 2^N
+ *
+ * per-counter global
+ * accuracy accuracy
+ * Level 0: [ -32 .. +31] ±256 (8 * 32)
+ * Level 1: [ -64 .. +63] ±256 (4 * 64)
+ * Level 2: [-128 .. +127] ±256 (2 * 128)
+ * Total: ------ ±768 (log2(nr_cpu_ids) * BATCH(level 0) * nr_cpu_ids)
+ *
+ * Note that the global accuracy can be calculated more precisely
+ * by taking into account that the positive accuracy range is
+ * 31 rather than 32.
+ *
+ * -----
+ *
+ * Approximate Sum Carry Propagation
+ *
+ * Let's define a number of counter bits for each level, e.g.:
+ *
+ * log2(BATCH(level 0)) = log2(32) = 5
+ * Let's assume, for this example, a 32-bit architecture (sizeof(long) == 4).
+ *
+ * nr_bit value_mask range
+ * Level 0: 5 bits v 0 .. +31
+ * Level 1: 1 bit (v & ~((1UL << 5) - 1)) 0 .. +63
+ * Level 2: 1 bit (v & ~((1UL << 6) - 1)) 0 .. +127
+ * Level 3: 25 bits (v & ~((1UL << 7) - 1)) 0 .. 2^32-1
+ *
+ * Note: Use a "long" per-cpu counter at level 0 to allow precise sum.
+ *
+ * Note: Use cacheline aligned counters at levels above 0 to prevent false sharing.
+ * If memory footprint is an issue, a specialized allocator could be used
+ * to eliminate padding.
+ *
+ * Example with expanded values:
+ *
+ * counter_add(counter, inc):
+ *
+ * if (!inc)
+ * return;
+ *
+ * res = percpu_add_return(counter @ Level 0, inc);
+ * orig = res - inc;
+ * if (inc < 0) {
+ * inc = -(-inc & ~0b00011111); // Clear used bits
+ * // xor bit 5: underflow
+ * if ((inc ^ orig ^ res) & 0b00100000)
+ * inc -= 0b00100000;
+ * } else {
+ * inc &= ~0b00011111; // Clear used bits
+ * // xor bit 5: overflow
+ * if ((inc ^ orig ^ res) & 0b00100000)
+ * inc += 0b00100000;
+ * }
+ * if (!inc)
+ * return;
+ *
+ * res = atomic_long_add_return(counter @ Level 1, inc);
+ * orig = res - inc;
+ * if (inc < 0) {
+ * inc = -(-inc & ~0b00111111); // Clear used bits
+ * // xor bit 6: underflow
+ * if ((inc ^ orig ^ res) & 0b01000000)
+ * inc -= 0b01000000;
+ * } else {
+ * inc &= ~0b00111111; // Clear used bits
+ * // xor bit 6: overflow
+ * if ((inc ^ orig ^ res) & 0b01000000)
+ * inc += 0b01000000;
+ * }
+ * if (!inc)
+ * return;
+ *
+ * res = atomic_long_add_return(counter @ Level 2, inc);
+ * orig = res - inc;
+ * if (inc < 0) {
+ * inc = -(-inc & ~0b01111111); // Clear used bits
+ * // xor bit 7: underflow
+ * if ((inc ^ orig ^ res) & 0b10000000)
+ * inc -= 0b10000000;
+ * } else {
+ * inc &= ~0b01111111; // Clear used bits
+ * // xor bit 7: overflow
+ * if ((inc ^ orig ^ res) & 0b10000000)
+ * inc += 0b10000000;
+ * }
+ * if (!inc)
+ * return;
+ *
+ * atomic_long_add(counter @ Level 3, inc);
+ */
+
+#include <linux/percpu_counter_tree.h>
+#include <linux/cpumask.h>
+#include <linux/atomic.h>
+#include <linux/export.h>
+#include <linux/percpu.h>
+#include <linux/errno.h>
+#include <linux/slab.h>
+#include <linux/math.h>
+
+#define MAX_NR_LEVELS 5
+
+/*
+ * The counter configuration is selected at boot time based on the
+ * hardware topology.
+ */
+struct counter_config {
+ unsigned int nr_items; /*
+ * nr_items is the number of items in the tree for levels 1
+ * up to and including the final level (approximate sum).
+ * It excludes the level 0 per-CPU counters.
+ */
+ unsigned char nr_levels; /*
+ * nr_levels is the number of hierarchical counter tree levels.
+ * It excludes the final level (approximate sum).
+ */
+ unsigned char n_arity_order[MAX_NR_LEVELS]; /*
+ * n-arity of tree nodes for each level from
+ * 0 to (nr_levels - 1).
+ */
+};
+
+static const struct counter_config per_nr_cpu_order_config[] = {
+ [0] = { .nr_items = 0, .nr_levels = 0, .n_arity_order = { 0 } },
+ [1] = { .nr_items = 1, .nr_levels = 1, .n_arity_order = { 1 } },
+ [2] = { .nr_items = 3, .nr_levels = 2, .n_arity_order = { 1, 1 } },
+ [3] = { .nr_items = 7, .nr_levels = 3, .n_arity_order = { 1, 1, 1 } },
+ [4] = { .nr_items = 7, .nr_levels = 3, .n_arity_order = { 2, 1, 1 } },
+ [5] = { .nr_items = 11, .nr_levels = 3, .n_arity_order = { 2, 2, 1 } },
+ [6] = { .nr_items = 21, .nr_levels = 3, .n_arity_order = { 2, 2, 2 } },
+ [7] = { .nr_items = 21, .nr_levels = 3, .n_arity_order = { 3, 2, 2 } },
+ [8] = { .nr_items = 37, .nr_levels = 3, .n_arity_order = { 3, 3, 2 } },
+ [9] = { .nr_items = 73, .nr_levels = 3, .n_arity_order = { 3, 3, 3 } },
+ [10] = { .nr_items = 149, .nr_levels = 4, .n_arity_order = { 3, 3, 2, 2 } },
+ [11] = { .nr_items = 293, .nr_levels = 4, .n_arity_order = { 3, 3, 3, 2 } },
+ [12] = { .nr_items = 585, .nr_levels = 4, .n_arity_order = { 3, 3, 3, 3 } },
+ [13] = { .nr_items = 1173, .nr_levels = 5, .n_arity_order = { 3, 3, 3, 2, 2 } },
+ [14] = { .nr_items = 2341, .nr_levels = 5, .n_arity_order = { 3, 3, 3, 3, 2 } },
+ [15] = { .nr_items = 4681, .nr_levels = 5, .n_arity_order = { 3, 3, 3, 3, 3 } },
+ [16] = { .nr_items = 4681, .nr_levels = 5, .n_arity_order = { 4, 3, 3, 3, 3 } },
+ [17] = { .nr_items = 8777, .nr_levels = 5, .n_arity_order = { 4, 4, 3, 3, 3 } },
+ [18] = { .nr_items = 17481, .nr_levels = 5, .n_arity_order = { 4, 4, 4, 3, 3 } },
+ [19] = { .nr_items = 34953, .nr_levels = 5, .n_arity_order = { 4, 4, 4, 4, 3 } },
+ [20] = { .nr_items = 69905, .nr_levels = 5, .n_arity_order = { 4, 4, 4, 4, 4 } },
+};
+
+static const struct counter_config *counter_config; /* Hierarchical counter configuration for the hardware topology. */
+static unsigned int nr_cpus_order; /* Order of nr_cpu_ids. */
+static unsigned long accuracy_multiplier; /* Calculate accuracy for a given batch size (multiplication factor). */
+
+static
+int __percpu_counter_tree_init(struct percpu_counter_tree *counter,
+ unsigned long batch_size, gfp_t gfp_flags,
+ unsigned long __percpu *level0,
+ struct percpu_counter_tree_level_item *items)
+{
+ /* Batch size must be greater than 1, and a power of 2. */
+ if (WARN_ON(batch_size <= 1 || (batch_size & (batch_size - 1))))
+ return -EINVAL;
+ counter->batch_size = batch_size;
+ counter->bias = 0;
+ counter->level0 = level0;
+ counter->items = items;
+ if (!nr_cpus_order) {
+ counter->approx_sum.i = per_cpu_ptr(counter->level0, 0);
+ counter->level0_bit_mask = 0;
+ } else {
+ counter->approx_sum.a = &counter->items[counter_config->nr_items - 1].count;
+ counter->level0_bit_mask = 1UL << get_count_order(batch_size);
+ }
+ /*
+ * Each tree item signed integer has a negative range which is
+ * one unit greater than the positive range.
+ */
+ counter->approx_accuracy_range.under = batch_size * accuracy_multiplier;
+ counter->approx_accuracy_range.over = (batch_size - 1) * accuracy_multiplier;
+ return 0;
+}
+
+/**
+ * percpu_counter_tree_init_many() - Initialize many per-CPU counter trees.
+ * @counters: An array of @nr_counters counters to initialize.
+ * Their memory is provided by the caller.
+ * @items: Pointer to memory area where to store tree items.
+ * This memory is provided by the caller.
+ * Its size needs to be at least @nr_counters * percpu_counter_tree_items_size().
+ * @nr_counters: The number of counter trees to initialize
+ * @batch_size: The batch size is the increment step at level 0 which triggers a
+ * carry propagation.
+ * The batch size is required to be greater than 1, and a power of 2.
+ * @gfp_flags: gfp flags to pass to the per-CPU allocator.
+ *
+ * Initialize many per-CPU counter trees using a single per-CPU
+ * allocator invocation for @nr_counters counters.
+ *
+ * Return:
+ * * %0: Success
+ * * %-EINVAL: - Invalid @batch_size argument
+ * * %-ENOMEM: - Out of memory
+ */
+int percpu_counter_tree_init_many(struct percpu_counter_tree *counters, struct percpu_counter_tree_level_item *items,
+ unsigned int nr_counters, unsigned long batch_size, gfp_t gfp_flags)
+{
+ void __percpu *level0, *level0_iter;
+ size_t counter_size = sizeof(*counters->level0),
+ items_size = percpu_counter_tree_items_size();
+ void *items_iter;
+ unsigned int i;
+ int ret;
+
+ memset(items, 0, items_size * nr_counters);
+ level0 = __alloc_percpu_gfp(nr_counters * counter_size,
+ __alignof__(*counters->level0), gfp_flags);
+ if (!level0)
+ return -ENOMEM;
+ level0_iter = level0;
+ items_iter = items;
+ for (i = 0; i < nr_counters; i++) {
+ ret = __percpu_counter_tree_init(&counters[i], batch_size, gfp_flags, level0_iter, items_iter);
+ if (ret)
+ goto free_level0;
+ level0_iter += counter_size;
+ items_iter += items_size;
+ }
+ return 0;
+
+free_level0:
+ free_percpu(level0);
+ return ret;
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_init_many);
+
+/**
+ * percpu_counter_tree_init() - Initialize one per-CPU counter tree.
+ * @counter: Counter to initialize.
+ * Its memory is provided by the caller.
+ * @items: Pointer to memory area where to store tree items.
+ * This memory is provided by the caller.
+ * Its size needs to be at least percpu_counter_tree_items_size().
+ * @batch_size: The batch size is the increment step at level 0 which triggers a
+ * carry propagation.
+ * The batch size is required to be greater than 1, and a power of 2.
+ * @gfp_flags: gfp flags to pass to the per-CPU allocator.
+ *
+ * Initialize one per-CPU counter tree.
+ *
+ * Return:
+ * * %0: Success
+ * * %-EINVAL: - Invalid @batch_size argument
+ * * %-ENOMEM: - Out of memory
+ */
+int percpu_counter_tree_init(struct percpu_counter_tree *counter, struct percpu_counter_tree_level_item *items,
+ unsigned long batch_size, gfp_t gfp_flags)
+{
+ return percpu_counter_tree_init_many(counter, items, 1, batch_size, gfp_flags);
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_init);
+
+/**
+ * percpu_counter_tree_destroy_many() - Destroy many per-CPU counter trees.
+ * @counters: Array of counters trees to destroy.
+ * @nr_counters: The number of counter trees to destroy.
+ *
+ * Release internal resources allocated for @nr_counters per-CPU counter trees.
+ */
+
+void percpu_counter_tree_destroy_many(struct percpu_counter_tree *counters, unsigned int nr_counters)
+{
+ free_percpu(counters->level0);
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_destroy_many);
+
+/**
+ * percpu_counter_tree_destroy() - Destroy one per-CPU counter tree.
+ * @counter: Counter to destroy.
+ *
+ * Release internal resources allocated for one per-CPU counter tree.
+ */
+void percpu_counter_tree_destroy(struct percpu_counter_tree *counter)
+{
+ return percpu_counter_tree_destroy_many(counter, 1);
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_destroy);
+
+static
+long percpu_counter_tree_carry(long orig, long res, long inc, unsigned long bit_mask)
+{
+ if (inc < 0) {
+ inc = -(-inc & ~(bit_mask - 1));
+ /*
+ * xor bit_mask: underflow.
+ *
+ * If inc has bit set, decrement an additional bit if
+ * there is _no_ bit transition between orig and res.
+ * Else, inc has bit cleared, decrement an additional
+ * bit if there is a bit transition between orig and
+ * res.
+ */
+ if ((inc ^ orig ^ res) & bit_mask)
+ inc -= bit_mask;
+ } else {
+ inc &= ~(bit_mask - 1);
+ /*
+ * xor bit_mask: overflow.
+ *
+ * If inc has bit set, increment an additional bit if
+ * there is _no_ bit transition between orig and res.
+ * Else, inc has bit cleared, increment an additional
+ * bit if there is a bit transition between orig and
+ * res.
+ */
+ if ((inc ^ orig ^ res) & bit_mask)
+ inc += bit_mask;
+ }
+ return inc;
+}
+
+/*
+ * It does not matter through which path the carry propagates up the
+ * tree, therefore there is no need to disable preemption because the
+ * cpu number is only used to favor cache locality.
+ */
+static
+void percpu_counter_tree_add_slowpath(struct percpu_counter_tree *counter, long inc)
+{
+ unsigned int level_items, nr_levels = counter_config->nr_levels,
+ level, n_arity_order;
+ unsigned long bit_mask;
+ struct percpu_counter_tree_level_item *item = counter->items;
+ unsigned int cpu = raw_smp_processor_id();
+
+ WARN_ON_ONCE(!nr_cpus_order); /* Should never be called for 1 cpu. */
+
+ n_arity_order = counter_config->n_arity_order[0];
+ bit_mask = counter->level0_bit_mask << n_arity_order;
+ level_items = 1U << (nr_cpus_order - n_arity_order);
+
+ for (level = 1; level < nr_levels; level++) {
+ /*
+ * For the purpose of carry propagation, the
+ * intermediate level counters only need to keep track
+ * of the bits relevant for carry propagation. We
+ * therefore don't care about higher order bits.
+ * Note that this optimization is unwanted if the
+ * intended use is to track counters within intermediate
+ * levels of the topology.
+ */
+ if (abs(inc) & (bit_mask - 1)) {
+ atomic_long_t *count = &item[cpu & (level_items - 1)].count;
+ unsigned long orig, res;
+
+ res = atomic_long_add_return_relaxed(inc, count);
+ orig = res - inc;
+ inc = percpu_counter_tree_carry(orig, res, inc, bit_mask);
+ if (likely(!inc))
+ return;
+ }
+ item += level_items;
+ n_arity_order = counter_config->n_arity_order[level];
+ level_items >>= n_arity_order;
+ bit_mask <<= n_arity_order;
+ }
+ atomic_long_add(inc, counter->approx_sum.a);
+}
+
+/**
+ * percpu_counter_tree_add() - Add to a per-CPU counter tree.
+ * @counter: Counter added to.
+ * @inc: Increment value (either positive or negative).
+ *
+ * Add @inc to a per-CPU counter tree. This is a fast-path which will
+ * typically increment per-CPU counters as long as there is no carry
+ * greater or equal to the counter tree batch size.
+ */
+void percpu_counter_tree_add(struct percpu_counter_tree *counter, long inc)
+{
+ unsigned long bit_mask = counter->level0_bit_mask, orig, res;
+
+ res = this_cpu_add_return(*counter->level0, inc);
+ orig = res - inc;
+ inc = percpu_counter_tree_carry(orig, res, inc, bit_mask);
+ if (likely(!inc))
+ return;
+ percpu_counter_tree_add_slowpath(counter, inc);
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_add);
+
+static
+long percpu_counter_tree_precise_sum_unbiased(struct percpu_counter_tree *counter)
+{
+ unsigned long sum = 0;
+ int cpu;
+
+ for_each_possible_cpu(cpu)
+ sum += *per_cpu_ptr(counter->level0, cpu);
+ return (long) sum;
+}
+
+/**
+ * percpu_counter_tree_precise_sum() - Return precise counter sum.
+ * @counter: The counter to sum.
+ *
+ * Querying the precise sum is relatively expensive because it needs to
+ * iterate over all CPUs.
+ * This is meant to be used when accuracy is preferred over speed.
+ *
+ * Return: The current precise counter sum.
+ */
+long percpu_counter_tree_precise_sum(struct percpu_counter_tree *counter)
+{
+ return percpu_counter_tree_precise_sum_unbiased(counter) + READ_ONCE(counter->bias);
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_precise_sum);
+
+/*
+ * The comparison between counters is on the wrapped difference. It is valid
+ * only when the two counters differ by less than LONG_MAX, and results are
+ * undefined beyond that.
+ */
+static inline
+long wrapped_delta(long a, long b)
+{
+ return (long)((unsigned long)a - (unsigned long)b);
+}
+
+/*
+ * Each counter's approximation lies within [precise - under, precise + over]:
+ *
+ * approx_a range
+ * <-----------+----------->
+ * under_a | over_a
+ * precise_a
+ * | approx_b range
+ * | <-----------+----------->
+ * | under_b | over_b
+ * | precise_b
+ * | |
+ * |<------------ gap ------------->|
+ * ----------------------+--------------------------------+-------------> values
+ *
+ * The two ranges are disjoint, and the comparison can return a definitive
+ * answer, only once the gap exceeds the two facing half-widths:
+ *
+ * a below b: gap > over_a + under_b (a's upper edge, b's lower edge)
+ * a above b: gap > under_a + over_b (a's lower edge, b's upper edge)
+ *
+ * When comparing against a plain value, the value has zero width, so the
+ * margins reduce to that single counter's under and over.
+ *
+ * accuracy_neg is used when delta < 0, and accuracy_pos is used when delta >= 0.
+ */
+static
+int compare_delta(long delta, unsigned long accuracy_neg, unsigned long accuracy_pos)
+{
+ if (delta >= 0) {
+ if (delta <= accuracy_pos)
+ return 0;
+ else
+ return 1;
+ } else {
+ if (-delta <= accuracy_neg)
+ return 0;
+ else
+ return -1;
+ }
+}
+
+/**
+ * percpu_counter_tree_approximate_compare - Approximated comparison of two counter trees.
+ * @a: First counter to compare.
+ * @b: Second counter to compare.
+ *
+ * Evaluate an approximate comparison of two counter trees. This approximation
+ * comparison is fast, and provides an accurate answer if the counters are
+ * found to be either less than or greater than the other. However, if the
+ * approximated comparison returns 0, the counters respective sums are found to
+ * be within the two counters accuracy range. The two counters are read
+ * independently; the result is not an atomic snapshot of both.
+ * The comparison between counters is on the wrapped difference. It is valid
+ * only when the two counters differ by less than LONG_MAX, and results are
+ * undefined beyond that.
+ *
+ * Return:
+ * * %0 - Counters @a and @b do not differ by more than the sum of their respective
+ * accuracy ranges.
+ * * %-1 - Counter @a less than counter @b.
+ * * %1 - Counter @a is greater than counter @b.
+ */
+int percpu_counter_tree_approximate_compare(struct percpu_counter_tree *a, struct percpu_counter_tree *b)
+{
+ /* See the range geometry above compare_delta(). */
+ return compare_delta(wrapped_delta(percpu_counter_tree_approximate_sum(a),
+ percpu_counter_tree_approximate_sum(b)),
+ a->approx_accuracy_range.over + b->approx_accuracy_range.under,
+ a->approx_accuracy_range.under + b->approx_accuracy_range.over);
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_approximate_compare);
+
+/**
+ * percpu_counter_tree_approximate_compare_value - Approximated comparison of a counter tree against a given value.
+ * @counter: Counter to compare.
+ * @v: Value to compare.
+ *
+ * Evaluate an approximate comparison of a counter tree against a given value.
+ * This approximation comparison is fast, and provides an accurate
+ * answer if the counter is found to be either less than or greater
+ * than the value. However, if the approximated comparison returns
+ * 0, the value is within the counter accuracy range.
+ * The comparison between counters is on the wrapped difference. It is valid
+ * only when @v and the counter differ by less than LONG_MAX, and results are
+ * undefined beyond that.
+ *
+ * Return:
+ * * %0 - The value @v is within the accuracy range of the counter.
+ * * %-1 - The value @v is less than the counter.
+ * * %1 - The value @v is greater than the counter.
+ */
+int percpu_counter_tree_approximate_compare_value(struct percpu_counter_tree *counter, long v)
+{
+ return compare_delta(wrapped_delta(v, percpu_counter_tree_approximate_sum(counter)),
+ counter->approx_accuracy_range.under,
+ counter->approx_accuracy_range.over);
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_approximate_compare_value);
+
+/**
+ * percpu_counter_tree_precise_compare - Precise comparison of two counter trees.
+ * @a: First counter to compare.
+ * @b: Second counter to compare.
+ *
+ * Evaluate a precise comparison of two counter trees. As an optimization, it
+ * uses the approximate counter comparison to quickly compare counters which
+ * are far apart. Only cases where counter sums are within the accuracy range
+ * require precise counter sums. The two counters are read independently; the
+ * result is not an atomic snapshot of both.
+ * The comparison between counters is on the wrapped difference. It is valid
+ * only when the two counters differ by less than LONG_MAX, and results are
+ * undefined beyond that.
+ *
+ * Return:
+ * * %0 - Counters are equal.
+ * * %-1 - Counter @a less than counter @b.
+ * * %1 - Counter @a is greater than counter @b.
+ */
+int percpu_counter_tree_precise_compare(struct percpu_counter_tree *a, struct percpu_counter_tree *b)
+{
+ long count_a = percpu_counter_tree_approximate_sum(a),
+ count_b = percpu_counter_tree_approximate_sum(b);
+ unsigned long accuracy_a, accuracy_b;
+ long delta = wrapped_delta(count_a, count_b);
+ int res;
+
+ /* See the range geometry above compare_delta(). */
+ res = compare_delta(delta,
+ a->approx_accuracy_range.over + b->approx_accuracy_range.under,
+ a->approx_accuracy_range.under + b->approx_accuracy_range.over);
+ /* The values are distanced enough for an accurate approximated comparison. */
+ if (res)
+ return res;
+
+ /*
+ * The approximated comparison is within the accuracy range, therefore at least one
+ * precise sum is needed. Sum the counter which has the largest accuracy first.
+ */
+ if (delta >= 0) {
+ accuracy_a = a->approx_accuracy_range.under;
+ accuracy_b = b->approx_accuracy_range.over;
+ } else {
+ accuracy_a = a->approx_accuracy_range.over;
+ accuracy_b = b->approx_accuracy_range.under;
+ }
+ if (accuracy_b < accuracy_a) {
+ count_a = percpu_counter_tree_precise_sum(a);
+ res = compare_delta(wrapped_delta(count_a, count_b),
+ b->approx_accuracy_range.under,
+ b->approx_accuracy_range.over);
+ if (res)
+ return res;
+ /* Precise sum of second counter is required. */
+ count_b = percpu_counter_tree_precise_sum(b);
+ } else {
+ count_b = percpu_counter_tree_precise_sum(b);
+ res = compare_delta(wrapped_delta(count_a, count_b),
+ a->approx_accuracy_range.over,
+ a->approx_accuracy_range.under);
+ if (res)
+ return res;
+ /* Precise sum of second counter is required. */
+ count_a = percpu_counter_tree_precise_sum(a);
+ }
+ delta = wrapped_delta(count_a, count_b);
+ if (delta < 0)
+ return -1;
+ if (delta > 0)
+ return 1;
+ return 0;
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_precise_compare);
+
+/**
+ * percpu_counter_tree_precise_compare_value - Precise comparison of a counter tree against a given value.
+ * @counter: Counter to compare.
+ * @v: Value to compare.
+ *
+ * Evaluate a precise comparison of a counter tree against a given value. As
+ * an optimization, it uses the approximate counter comparison to quickly
+ * identify whether the counter and value are far apart. Only cases where the
+ * value is within the counter accuracy range require a precise counter sum.
+ * The comparison between counters is on the wrapped difference. It is valid
+ * only when @v and the counter differ by less than LONG_MAX, and results are
+ * undefined beyond that.
+ *
+ * Return:
+ * * %0 - The value @v is equal to the counter.
+ * * %-1 - The value @v is less than the counter.
+ * * %1 - The value @v is greater than the counter.
+ */
+int percpu_counter_tree_precise_compare_value(struct percpu_counter_tree *counter, long v)
+{
+ long count = percpu_counter_tree_approximate_sum(counter), delta;
+ int res;
+
+ res = compare_delta(wrapped_delta(v, count),
+ counter->approx_accuracy_range.under,
+ counter->approx_accuracy_range.over);
+ /* The values are distanced enough for an accurate approximated comparison. */
+ if (res)
+ return res;
+
+ /* Precise sum is required. */
+ count = percpu_counter_tree_precise_sum(counter);
+ delta = wrapped_delta(v, count);
+ if (delta < 0)
+ return -1;
+ if (delta > 0)
+ return 1;
+ return 0;
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_precise_compare_value);
+
+static
+void percpu_counter_tree_set_bias(struct percpu_counter_tree *counter, long bias)
+{
+ WRITE_ONCE(counter->bias, bias);
+}
+
+/**
+ * percpu_counter_tree_set - Set the counter tree sum to a given value.
+ * @counter: Counter to set.
+ * @v: Value to set.
+ *
+ * Set the counter sum to a given value. It can be useful for instance
+ * to reset the counter sum to 0. Note that even after setting the
+ * counter sum to a given value, the counter sum approximation can
+ * return any value within the accuracy range around that value.
+ */
+void percpu_counter_tree_set(struct percpu_counter_tree *counter, long v)
+{
+ percpu_counter_tree_set_bias(counter,
+ v - percpu_counter_tree_precise_sum_unbiased(counter));
+}
+EXPORT_SYMBOL_GPL(percpu_counter_tree_set);
+
+/* Initialize to SIZE_MAX to catch early boot misuses. */
+size_t __percpu_counter_tree_items_size __ro_after_init = SIZE_MAX;
+EXPORT_SYMBOL_GPL(__percpu_counter_tree_items_size);
+
+static void __init calculate_accuracy_topology(void)
+{
+ unsigned int nr_levels = counter_config->nr_levels, level;
+ unsigned int level_items = 1U << nr_cpus_order;
+ unsigned long batch_size = 1;
+
+ for (level = 0; level < nr_levels; level++) {
+ unsigned int n_arity_order = counter_config->n_arity_order[level];
+
+ /*
+ * The accuracy multiplier is derived from a batch size of 1
+ * to speed up calculating the accuracy at tree initialization.
+ */
+ accuracy_multiplier += batch_size * level_items;
+ batch_size <<= n_arity_order;
+ level_items >>= n_arity_order;
+ }
+}
+
+int __init percpu_counter_tree_subsystem_init(void)
+{
+ nr_cpus_order = get_count_order(nr_cpu_ids);
+ if (WARN_ON_ONCE(nr_cpus_order >= ARRAY_SIZE(per_nr_cpu_order_config))) {
+ printk(KERN_ERR "Unsupported number of CPUs (%u)\n", nr_cpu_ids);
+ return -1;
+ }
+ counter_config = &per_nr_cpu_order_config[nr_cpus_order];
+ if (!nr_cpus_order)
+ __percpu_counter_tree_items_size = 0;
+ else
+ __percpu_counter_tree_items_size = counter_config->nr_items *
+ sizeof(struct percpu_counter_tree_level_item);
+ calculate_accuracy_topology();
+ return 0;
+}
diff --git a/lib/tests/Makefile b/lib/tests/Makefile
index 3cac3b63a7522..051e73705977b 100644
--- a/lib/tests/Makefile
+++ b/lib/tests/Makefile
@@ -58,4 +58,6 @@ obj-$(CONFIG_UTIL_MACROS_KUNIT) += util_macros_kunit.o
obj-$(CONFIG_RATELIMIT_KUNIT_TEST) += test_ratelimit.o
obj-$(CONFIG_UUID_KUNIT_TEST) += uuid_kunit.o
+obj-$(CONFIG_PERCPU_COUNTER_TREE_KUNIT_TEST) += percpu_counter_tree_kunit.o
+
obj-$(CONFIG_TEST_RUNTIME_MODULE) += module/
diff --git a/lib/tests/percpu_counter_tree_kunit.c b/lib/tests/percpu_counter_tree_kunit.c
new file mode 100644
index 0000000000000..8af8d2659b57b
--- /dev/null
+++ b/lib/tests/percpu_counter_tree_kunit.c
@@ -0,0 +1,583 @@
+// SPDX-License-Identifier: GPL-2.0+ OR MIT
+// SPDX-FileCopyrightText: 2026 Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
+
+#include <kunit/test.h>
+#include <linux/percpu_counter_tree.h>
+#include <linux/kthread.h>
+#include <linux/wait.h>
+#include <linux/random.h>
+#include <linux/cpuhplock.h>
+#include <linux/slab.h>
+#include <linux/sched/task.h>
+
+struct multi_thread_test_data {
+ long increment;
+ unsigned int nr_inc;
+ int counter_index;
+};
+
+#define NR_COUNTERS 2
+
+/* Hierarchical per-CPU counter instances. */
+static struct percpu_counter_tree counter[NR_COUNTERS];
+static struct percpu_counter_tree_level_item *items;
+
+/* Global atomic counters for validation. */
+static atomic_long_t global_counter[NR_COUNTERS];
+
+static DECLARE_WAIT_QUEUE_HEAD(kernel_threads_wq);
+static atomic_t kernel_threads_to_run;
+
+static void complete_work(void)
+{
+ if (atomic_dec_and_test(&kernel_threads_to_run))
+ wake_up(&kernel_threads_wq);
+}
+
+/* Random tasks test. */
+#define NR_RAND_TASKS 1000
+
+static void hpcc_print_info(struct kunit *test)
+{
+ kunit_info(test, "Running test with %d CPUs\n", num_online_cpus());
+}
+
+static void add_to_counter(int counter_index, unsigned int nr_inc, long increment)
+{
+ unsigned int i;
+
+ for (i = 0; i < nr_inc; i++) {
+ percpu_counter_tree_add(&counter[counter_index], increment);
+ atomic_long_add(increment, &global_counter[counter_index]);
+ }
+}
+
+static void check_counters(struct kunit *test)
+{
+ int counter_index;
+
+ /* Compare each counter with its global counter. */
+ for (counter_index = 0; counter_index < NR_COUNTERS; counter_index++) {
+ long v = atomic_long_read(&global_counter[counter_index]);
+ long approx_sum = percpu_counter_tree_approximate_sum(&counter[counter_index]);
+ unsigned long under_accuracy = 0, over_accuracy = 0;
+ long precise_min, precise_max;
+
+ /* Precise comparison. */
+ KUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(&counter[counter_index]), v);
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_precise_compare_value(&counter[counter_index], v));
+
+ /* Approximate comparison. */
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare_value(&counter[counter_index], v));
+
+ /* Accuracy limits checks. */
+ percpu_counter_tree_approximate_accuracy_range(&counter[counter_index], &under_accuracy, &over_accuracy);
+
+ /*
+ * under_accuracy and over_accuracy are unsigned long, so the
+ * inner expression promotes to unsigned. Compute the difference
+ * in unsigned and cast to signed for the comparison against 0.
+ */
+ KUNIT_EXPECT_GE(test, (long)(approx_sum - (v - under_accuracy)), 0);
+ KUNIT_EXPECT_LE(test, (long)(approx_sum - (v + over_accuracy)), 0);
+
+ /* Precise min/max range check. */
+ percpu_counter_tree_approximate_min_max_range(approx_sum, under_accuracy, over_accuracy, &precise_min, &precise_max);
+
+ KUNIT_EXPECT_GE(test, v - precise_min, 0);
+ KUNIT_EXPECT_LE(test, v - precise_max, 0);
+ }
+ /* Compare each counter with the second counter. */
+ KUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(&counter[0]), percpu_counter_tree_precise_sum(&counter[1]));
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_precise_compare(&counter[0], &counter[1]));
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare(&counter[0], &counter[1]));
+}
+
+static int multi_thread_worker_fn(void *data)
+{
+ struct multi_thread_test_data *td = data;
+
+ add_to_counter(td->counter_index, td->nr_inc, td->increment);
+ complete_work();
+ kfree(td);
+ return 0;
+}
+
+static struct task_struct *test_run_on_specific_cpu(int target_cpu, int counter_index, unsigned int nr_inc, long increment)
+{
+ struct multi_thread_test_data *td = kzalloc(sizeof(struct multi_thread_test_data), GFP_KERNEL);
+ struct task_struct *task;
+
+ if (!td)
+ return NULL;
+ td->increment = increment;
+ td->nr_inc = nr_inc;
+ td->counter_index = counter_index;
+ atomic_inc(&kernel_threads_to_run);
+ task = kthread_create_on_cpu(multi_thread_worker_fn, td, target_cpu, "kunit_multi_thread_worker");
+ if (IS_ERR_OR_NULL(task)) {
+ complete_work();
+ kfree(td);
+ return task;
+ }
+ get_task_struct(task);
+ wake_up_process(task);
+ return task;
+}
+
+static void init_kthreads(void)
+{
+ atomic_set(&kernel_threads_to_run, 1);
+}
+
+/*
+ * The caller needs to wait for work completion _before_ stopping kthreads.
+ */
+static void fini_kthreads(void)
+{
+ /* Release our own reference. */
+ complete_work();
+ /* Wait for all others threads to run. */
+ wait_event(kernel_threads_wq, (atomic_read(&kernel_threads_to_run) == 0));
+}
+
+static void test_sync_kthreads(void)
+{
+ fini_kthreads();
+ init_kthreads();
+}
+
+static int init_counters(unsigned long batch_size)
+{
+ int i, ret;
+
+ items = kvcalloc(NR_COUNTERS, percpu_counter_tree_items_size(), GFP_KERNEL);
+ if (!items)
+ return -ENOMEM;
+ ret = percpu_counter_tree_init_many(counter, items, NR_COUNTERS, batch_size, GFP_KERNEL);
+ if (ret) {
+ kvfree(items);
+ return ret;
+ }
+ for (i = 0; i < NR_COUNTERS; i++)
+ atomic_long_set(&global_counter[i], 0);
+ return 0;
+}
+
+static void fini_counters(void)
+{
+ percpu_counter_tree_destroy_many(counter, NR_COUNTERS);
+ kvfree(items);
+}
+
+enum up_test_inc_type {
+ INC_ONE,
+ INC_MINUS_ONE,
+ INC_RANDOM,
+};
+
+/*
+ * Single-threaded tests. Those use many threads to run on various CPUs,
+ * but synchronize for completion of each thread before running the
+ * next, effectively making sure there are no concurrent updates.
+ */
+static void do_hpcc_test_single_thread(struct kunit *test, int _cpu0, int _cpu1, enum up_test_inc_type type)
+{
+ unsigned long batch_size_order = 5;
+ int cpu0 = _cpu0;
+ int cpu1 = _cpu1;
+ int ret, i;
+
+ ret = init_counters(1UL << batch_size_order);
+ KUNIT_EXPECT_EQ(test, ret, 0);
+ if (ret)
+ return;
+ init_kthreads();
+ cpus_read_lock();
+ for (i = 0; i < 10000; i++) {
+ struct task_struct *task;
+ long increment;
+
+ switch (type) {
+ case INC_ONE:
+ increment = 1;
+ break;
+ case INC_MINUS_ONE:
+ increment = -1;
+ break;
+ case INC_RANDOM:
+ increment = (long) get_random_long() % 50000;
+ break;
+ default:
+ WARN_ON_ONCE(1);
+ increment = 0;
+ }
+ if (_cpu0 < 0)
+ cpu0 = cpumask_any_distribute(cpu_online_mask);
+ if (_cpu1 < 0)
+ cpu1 = cpumask_any_distribute(cpu_online_mask);
+ task = test_run_on_specific_cpu(cpu0, 0, 1, increment);
+ KUNIT_EXPECT_NOT_ERR_OR_NULL(test, task);
+ if (IS_ERR_OR_NULL(task)) {
+ fini_kthreads();
+ goto end;
+ }
+ test_sync_kthreads();
+ kthread_stop_put(task);
+ task = test_run_on_specific_cpu(cpu1, 1, 1, increment);
+ KUNIT_EXPECT_NOT_ERR_OR_NULL(test, task);
+ if (IS_ERR_OR_NULL(task)) {
+ fini_kthreads();
+ goto end;
+ }
+ test_sync_kthreads();
+ kthread_stop_put(task);
+ check_counters(test);
+ }
+end:
+ cpus_read_unlock();
+ fini_counters();
+}
+
+static void hpcc_test_single_thread_first(struct kunit *test)
+{
+ int cpu = cpumask_first(cpu_online_mask);
+
+ do_hpcc_test_single_thread(test, cpu, cpu, INC_ONE);
+ do_hpcc_test_single_thread(test, cpu, cpu, INC_MINUS_ONE);
+ do_hpcc_test_single_thread(test, cpu, cpu, INC_RANDOM);
+}
+
+static void hpcc_test_single_thread_first_random(struct kunit *test)
+{
+ int cpu = cpumask_first(cpu_online_mask);
+
+ do_hpcc_test_single_thread(test, cpu, -1, INC_ONE);
+ do_hpcc_test_single_thread(test, cpu, -1, INC_MINUS_ONE);
+ do_hpcc_test_single_thread(test, cpu, -1, INC_RANDOM);
+}
+
+static void hpcc_test_single_thread_random(struct kunit *test)
+{
+ do_hpcc_test_single_thread(test, -1, -1, INC_ONE);
+ do_hpcc_test_single_thread(test, -1, -1, INC_MINUS_ONE);
+ do_hpcc_test_single_thread(test, -1, -1, INC_RANDOM);
+}
+
+/* Multi-threaded SMP tests. */
+
+static void do_hpcc_multi_thread_increment_each_cpu(struct kunit *test, unsigned long batch_size, unsigned int nr_inc, long increment)
+{
+ struct task_struct *(*task)[NR_COUNTERS];
+ int ret, cpu;
+
+ task = kvcalloc(nr_cpu_ids, sizeof(*task), GFP_KERNEL);
+ KUNIT_EXPECT_NOT_NULL(test, task);
+ if (!task)
+ return;
+ ret = init_counters(batch_size);
+ KUNIT_EXPECT_EQ(test, ret, 0);
+ if (ret)
+ goto end;
+ init_kthreads();
+ cpus_read_lock();
+ for_each_online_cpu(cpu) {
+ task[cpu][0] = test_run_on_specific_cpu(cpu, 0, nr_inc, increment);
+ KUNIT_EXPECT_NOT_ERR_OR_NULL(test, task[cpu][0]);
+ if (IS_ERR_OR_NULL(task[cpu][0]))
+ break;
+ task[cpu][1] = test_run_on_specific_cpu(cpu, 1, nr_inc, increment);
+ KUNIT_EXPECT_NOT_ERR_OR_NULL(test, task[cpu][1]);
+ if (IS_ERR_OR_NULL(task[cpu][1]))
+ break;
+ }
+ fini_kthreads();
+ for_each_online_cpu(cpu) {
+ if (!IS_ERR_OR_NULL(task[cpu][0]))
+ kthread_stop_put(task[cpu][0]);
+ if (!IS_ERR_OR_NULL(task[cpu][1]))
+ kthread_stop_put(task[cpu][1]);
+ }
+ cpus_read_unlock();
+ check_counters(test);
+ fini_counters();
+end:
+ kvfree(task);
+}
+
+static void do_hpcc_multi_thread_increment_even_cpus(struct kunit *test, unsigned long batch_size, unsigned int nr_inc, long increment)
+{
+ struct task_struct *(*task)[NR_COUNTERS];
+ int ret, cpu;
+
+ task = kvcalloc(nr_cpu_ids, sizeof(*task), GFP_KERNEL);
+ KUNIT_EXPECT_NOT_NULL(test, task);
+ if (!task)
+ return;
+ ret = init_counters(batch_size);
+ KUNIT_EXPECT_EQ(test, ret, 0);
+ if (ret)
+ goto end;
+ init_kthreads();
+ cpus_read_lock();
+ for_each_online_cpu(cpu) {
+ /*
+ * Funnel pairs of CPU indexes to their associated even index.
+ *
+ * Map 0 and 1 -> index 0
+ * Map 2 and 3 -> index 2 and so on.
+ */
+ int even_cpu = cpu & ~1;
+
+ if (!cpumask_test_cpu(even_cpu, cpu_online_mask))
+ continue;
+ task[cpu][0] = test_run_on_specific_cpu(cpu, 0, nr_inc, increment);
+ KUNIT_EXPECT_NOT_ERR_OR_NULL(test, task[cpu][0]);
+ if (IS_ERR_OR_NULL(task[cpu][0]))
+ break;
+ task[cpu][1] = test_run_on_specific_cpu(even_cpu, 1, nr_inc, increment);
+ KUNIT_EXPECT_NOT_ERR_OR_NULL(test, task[cpu][1]);
+ if (IS_ERR_OR_NULL(task[cpu][1]))
+ break;
+ }
+ fini_kthreads();
+ for_each_online_cpu(cpu) {
+ if (!IS_ERR_OR_NULL(task[cpu][0]))
+ kthread_stop_put(task[cpu][0]);
+ if (!IS_ERR_OR_NULL(task[cpu][1]))
+ kthread_stop_put(task[cpu][1]);
+ }
+ cpus_read_unlock();
+ check_counters(test);
+ fini_counters();
+end:
+ kvfree(task);
+}
+
+static void do_hpcc_multi_thread_increment_single_cpu(struct kunit *test, unsigned long batch_size, unsigned int nr_inc, long increment)
+{
+ struct task_struct *(*task)[NR_COUNTERS];
+ int ret, cpu;
+
+ task = kvcalloc(nr_cpu_ids, sizeof(*task), GFP_KERNEL);
+ KUNIT_EXPECT_NOT_NULL(test, task);
+ if (!task)
+ return;
+ ret = init_counters(batch_size);
+ KUNIT_EXPECT_EQ(test, ret, 0);
+ if (ret)
+ goto end;
+ init_kthreads();
+ cpus_read_lock();
+ for_each_online_cpu(cpu) {
+ task[cpu][0] = test_run_on_specific_cpu(cpu, 0, nr_inc, increment);
+ KUNIT_EXPECT_NOT_ERR_OR_NULL(test, task[cpu][0]);
+ if (IS_ERR_OR_NULL(task[cpu][0]))
+ break;
+ task[cpu][1] = test_run_on_specific_cpu(cpumask_first(cpu_online_mask), 1, nr_inc, increment);
+ KUNIT_EXPECT_NOT_ERR_OR_NULL(test, task[cpu][1]);
+ if (IS_ERR_OR_NULL(task[cpu][1]))
+ break;
+ }
+ fini_kthreads();
+ for_each_online_cpu(cpu) {
+ if (!IS_ERR_OR_NULL(task[cpu][0]))
+ kthread_stop_put(task[cpu][0]);
+ if (!IS_ERR_OR_NULL(task[cpu][1]))
+ kthread_stop_put(task[cpu][1]);
+ }
+ cpus_read_unlock();
+ check_counters(test);
+ fini_counters();
+end:
+ kvfree(task);
+}
+
+static void do_hpcc_multi_thread_increment_random_cpu(struct kunit *test, unsigned long batch_size, unsigned int nr_inc, long increment)
+{
+ struct task_struct *(*task)[NR_COUNTERS];
+ int ret, cpu;
+
+ task = kvcalloc(nr_cpu_ids, sizeof(*task), GFP_KERNEL);
+ KUNIT_EXPECT_NOT_NULL(test, task);
+ if (!task)
+ return;
+ ret = init_counters(batch_size);
+ KUNIT_EXPECT_EQ(test, ret, 0);
+ if (ret)
+ goto end;
+ init_kthreads();
+ cpus_read_lock();
+ for_each_online_cpu(cpu) {
+ task[cpu][0] = test_run_on_specific_cpu(cpu, 0, nr_inc, increment);
+ KUNIT_EXPECT_NOT_ERR_OR_NULL(test, task[cpu][0]);
+ if (IS_ERR_OR_NULL(task[cpu][0]))
+ break;
+ task[cpu][1] = test_run_on_specific_cpu(cpumask_any_distribute(cpu_online_mask), 1, nr_inc, increment);
+ KUNIT_EXPECT_NOT_ERR_OR_NULL(test, task[cpu][1]);
+ if (IS_ERR_OR_NULL(task[cpu][1]))
+ break;
+ }
+ fini_kthreads();
+ for_each_online_cpu(cpu) {
+ if (!IS_ERR_OR_NULL(task[cpu][0]))
+ kthread_stop_put(task[cpu][0]);
+ if (!IS_ERR_OR_NULL(task[cpu][1]))
+ kthread_stop_put(task[cpu][1]);
+ }
+ cpus_read_unlock();
+ check_counters(test);
+ fini_counters();
+end:
+ kvfree(task);
+}
+
+static void hpcc_test_multi_thread_batch_increment(struct kunit *test)
+{
+ unsigned long batch_size_order;
+
+ for (batch_size_order = 2; batch_size_order < 10; batch_size_order++) {
+ unsigned int nr_inc;
+
+ for (nr_inc = 1; nr_inc < 1024; nr_inc *= 2) {
+ long increment;
+
+ for (increment = 1; increment < 100000; increment *= 10) {
+ do_hpcc_multi_thread_increment_each_cpu(test, 1UL << batch_size_order, nr_inc, increment);
+ do_hpcc_multi_thread_increment_even_cpus(test, 1UL << batch_size_order, nr_inc, increment);
+ do_hpcc_multi_thread_increment_single_cpu(test, 1UL << batch_size_order, nr_inc, increment);
+ do_hpcc_multi_thread_increment_random_cpu(test, 1UL << batch_size_order, nr_inc, increment);
+ }
+ }
+ }
+}
+
+static void hpcc_test_multi_thread_random_walk(struct kunit *test)
+{
+ struct task_struct *(*task)[NR_COUNTERS];
+ unsigned long batch_size_order = 5;
+ int loop;
+
+ task = kvcalloc(NR_RAND_TASKS, sizeof(*task), GFP_KERNEL);
+ KUNIT_EXPECT_NOT_NULL(test, task);
+ if (!task)
+ return;
+ for (loop = 0; loop < 100; loop++) {
+ int ret, i;
+
+ memset(task, 0, NR_RAND_TASKS * sizeof(*task));
+ ret = init_counters(1UL << batch_size_order);
+ KUNIT_EXPECT_EQ(test, ret, 0);
+ if (ret)
+ goto end;
+ init_kthreads();
+ cpus_read_lock();
+ for (i = 0; i < NR_RAND_TASKS; i++) {
+ long increment = (long) get_random_long() % 512;
+ unsigned int nr_inc = ((unsigned long) get_random_long()) % 1024;
+
+ task[i][0] = test_run_on_specific_cpu(cpumask_any_distribute(cpu_online_mask), 0, nr_inc, increment);
+ KUNIT_EXPECT_NOT_ERR_OR_NULL(test, task[i][0]);
+ if (IS_ERR_OR_NULL(task[i][0]))
+ break;
+ task[i][1] = test_run_on_specific_cpu(cpumask_any_distribute(cpu_online_mask), 1, nr_inc, increment);
+ KUNIT_EXPECT_NOT_ERR_OR_NULL(test, task[i][1]);
+ if (IS_ERR_OR_NULL(task[i][1]))
+ break;
+ }
+ fini_kthreads();
+ for (i = 0; i < NR_RAND_TASKS; i++) {
+ if (!IS_ERR_OR_NULL(task[i][0]))
+ kthread_stop_put(task[i][0]);
+ if (!IS_ERR_OR_NULL(task[i][1]))
+ kthread_stop_put(task[i][1]);
+ }
+ cpus_read_unlock();
+ check_counters(test);
+ fini_counters();
+ }
+end:
+ kvfree(task);
+}
+
+static void hpcc_test_init_one(struct kunit *test)
+{
+ struct percpu_counter_tree pct;
+ struct percpu_counter_tree_level_item *counter_items;
+ int ret;
+
+ counter_items = kzalloc(percpu_counter_tree_items_size(), GFP_KERNEL);
+ KUNIT_EXPECT_NOT_NULL(test, counter_items);
+ if (!counter_items)
+ return;
+ ret = percpu_counter_tree_init(&pct, counter_items, 32, GFP_KERNEL);
+ KUNIT_EXPECT_EQ(test, ret, 0);
+ if (ret)
+ goto end;
+
+ percpu_counter_tree_destroy(&pct);
+end:
+ kfree(counter_items);
+}
+
+static void hpcc_test_set(struct kunit *test)
+{
+ static long values[] = {
+ 5, 100, 127, 128, 255, 256, 4095, 4096, 500000, 0,
+ -5, -100, -127, -128, -255, -256, -4095, -4096, -500000,
+ };
+ struct percpu_counter_tree pct;
+ struct percpu_counter_tree_level_item *counter_items;
+ int i, ret;
+
+ counter_items = kzalloc(percpu_counter_tree_items_size(), GFP_KERNEL);
+ KUNIT_EXPECT_NOT_NULL(test, counter_items);
+ if (!counter_items)
+ return;
+ ret = percpu_counter_tree_init(&pct, counter_items, 32, GFP_KERNEL);
+ KUNIT_EXPECT_EQ(test, ret, 0);
+ if (ret)
+ goto end;
+
+ for (i = 0; i < ARRAY_SIZE(values); i++) {
+ long v = values[i];
+
+ percpu_counter_tree_set(&pct, v);
+ KUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(&pct), v);
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare_value(&pct, v));
+
+ percpu_counter_tree_add(&pct, v);
+ KUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(&pct), 2 * v);
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare_value(&pct, 2 * v));
+
+ percpu_counter_tree_add(&pct, -2 * v);
+ KUNIT_EXPECT_EQ(test, percpu_counter_tree_precise_sum(&pct), 0);
+ KUNIT_EXPECT_EQ(test, 0, percpu_counter_tree_approximate_compare_value(&pct, 0));
+ }
+
+ percpu_counter_tree_destroy(&pct);
+end:
+ kfree(counter_items);
+}
+
+static struct kunit_case hpcc_test_cases[] = {
+ KUNIT_CASE(hpcc_print_info),
+ KUNIT_CASE(hpcc_test_single_thread_first),
+ KUNIT_CASE(hpcc_test_single_thread_first_random),
+ KUNIT_CASE(hpcc_test_single_thread_random),
+ KUNIT_CASE(hpcc_test_multi_thread_batch_increment),
+ KUNIT_CASE(hpcc_test_multi_thread_random_walk),
+ KUNIT_CASE(hpcc_test_init_one),
+ KUNIT_CASE(hpcc_test_set),
+ {}
+};
+
+static struct kunit_suite hpcc_test_suite = {
+ .name = "percpu_counter_tree",
+ .test_cases = hpcc_test_cases,
+};
+
+kunit_test_suite(hpcc_test_suite);
+
+MODULE_DESCRIPTION("Test cases for hierarchical per-CPU counters");
+MODULE_LICENSE("Dual MIT/GPL");
|
| 9/3 | 2026/09/06 04:16 | llm | kmsan-evaluator |
0mModel:gemini-3.7-flash Args: null Results: null Tokens: input: 29060 output: 17 thoughts: 815 Thoughts: **Analyzing the Per-CPU Counter Tree Patch** Okay, let's dive into this patch. My initial focus is on potential memory safety issues, specifically uninitialized memory risks that KMSAN would highlight. I'll be looking to see if KASAN or LOCKDEP could be triggered as well, but uninitialized memory is priority one. First, I see the addition of documentation: `Documentation/core-api/percpu-counter-tree.rst`. Good, we'll need that. Now, let's get into the code changes. I note that the `include/linux/mm.h` header has had `get_rss_stat_items(struct mm_struct *mm)` added. The patch replaces direct `percpu_counter` function calls for `mm->rss_stat` with `percpu_counter_tree_*` functions. This is the core shift. `include/linux/mm_types.h` shows a few key struct changes. `struct percpu_counter_tree_level_item` gets an alignment update, and the `rss_stat` is now a `struct percpu_counter_tree` instead of the old `struct percpu_counter`. There are also some updates to the flexible array initialization macros and functions: `get_rss_stat_items_size()` and `get_rss_stat_items_offset()`. The `include/linux/percpu_counter_tree.h` header introduces the new structures: `struct percpu_counter_tree` and `struct percpu_counter_tree_level_item`. It also includes those inline functions for approximate and precise sums and comparisons, that's important for the efficiency. `include/trace/events/kmem.h` uses the `percpu_counter_tree_approximate_sum_positive` in the trace event. The system is being initialized during boot, in `init/main.c`, through a call to `percpu_counter_tree_subsystem_init()`. Looking at `kernel/fork.c`, I see a wholesale replacement of the old `rss_stat` initialization, destruction, and checking with the new `percpu_counter_tree_*` functions. `mm_init` now calls `percpu_counter_tree_init_many(mm->rss_stat, get_rss_stat_items(mm), ...)`. There's also some adjustment to `mm_size` calculation for the `mm_struct` kmem_cache during `mm_cache_init`. The `lib` directory shows the addition of `percpu_counter_tree.o` to the Makefile, and KUnit tests. In `lib/percpu_counter_tree.c`, I see the implementation of the `percpu_counter_tree` logic. Importantly, `percpu_counter_tree_init_many` uses `memset(items, 0, items_size * nr_counters)` to zero the items. This is good, reduces the likelihood of uninitialized memory issues. The `level0` is allocated with `__alloc_percpu_gfp`. I want to double-check that `__alloc_percpu_gfp` zeroes the memory. Let's see... looking at the code... yes, it allocates zeroed per-cpu memory. This means allocations via `__alloc_percpu_gfp` are safe from uninitialized memory reads. Finally, `lib/tests/percpu_counter_tree_kunit.c` gives us KUnit tests, which is essential. Now, let's step back and consider the allocation of the `mm_struct` itself. When `mm_struct` is allocated in `dup_mm` or `mm_alloc`, does `allocate_mm()` use `kmem_cache_alloc` or `kmem_cache_zalloc`? I need to search `kernel/fork.c` to be sure. That would also provide memory safety. |
| 10/3 | 2026/09/06 04:16 | tool | grepper |
0mArgs:{
"Expression": "allocate_mm"
}Results: {
"Output": "drivers/gpu/drm/hyperv/hyperv_drm_drv.c=71=static int hv_drm_setup_vram(struct hv_drm_device *hv,\n--\ndrivers/gpu/drm/hyperv/hyperv_drm_drv.c-78-\ndrivers/gpu/drm/hyperv/hyperv_drm_drv.c:79:\tret = vmbus_allocate_mmio(\u0026hv-\u003emem, hdev, 0, -1, hv-\u003efb_size, 0x100000,\ndrivers/gpu/drm/hyperv/hyperv_drm_drv.c-80-\t\t\t\t true);\n--\ndrivers/hv/vmbus_drv.c=2373=static void __maybe_unused vmbus_reserve_fb(void)\n--\ndrivers/hv/vmbus_drv.c-2439-/**\ndrivers/hv/vmbus_drv.c:2440: * vmbus_allocate_mmio() - Pick a memory-mapped I/O range.\ndrivers/hv/vmbus_drv.c-2441- * @new:\t\tIf successful, supplied a pointer to the\n--\ndrivers/hv/vmbus_drv.c-2461- */\ndrivers/hv/vmbus_drv.c:2462:int vmbus_allocate_mmio(struct resource **new, struct hv_device *device_obj,\ndrivers/hv/vmbus_drv.c-2463-\t\t\tresource_size_t min, resource_size_t max,\n--\ndrivers/hv/vmbus_drv.c-2530-}\ndrivers/hv/vmbus_drv.c:2531:EXPORT_SYMBOL_GPL(vmbus_allocate_mmio);\ndrivers/hv/vmbus_drv.c-2532-\n--\ndrivers/hv/vmbus_drv.c=2541=void vmbus_free_mmio(resource_size_t start, resource_size_t size)\n--\ndrivers/hv/vmbus_drv.c-2550-\t * region since no corresponding __request_region() was done\ndrivers/hv/vmbus_drv.c:2551:\t * in vmbus_allocate_mmio().\ndrivers/hv/vmbus_drv.c-2552-\t */\n--\ndrivers/pci/controller/pci-hyperv.c=3301=static void hv_pci_free_bridge_windows(struct hv_pcibus_device *hbus)\n--\ndrivers/pci/controller/pci-hyperv.c-3325- *\ndrivers/pci/controller/pci-hyperv.c:3326: * This function calls vmbus_allocate_mmio(), which is itself a\ndrivers/pci/controller/pci-hyperv.c-3327- * bit of a compromise. Ideally, we might change the pnp layer\n--\ndrivers/pci/controller/pci-hyperv.c=3346=static int hv_pci_allocate_bridge_windows(struct hv_pcibus_device *hbus)\n--\ndrivers/pci/controller/pci-hyperv.c-3352-\t\talign = 1ULL \u003c\u003c (63 - __builtin_clzll(hbus-\u003elow_mmio_space));\ndrivers/pci/controller/pci-hyperv.c:3353:\t\tret = vmbus_allocate_mmio(\u0026hbus-\u003elow_mmio_res, hbus-\u003ehdev, 0,\ndrivers/pci/controller/pci-hyperv.c-3354-\t\t\t\t\t (u64)(u32)0xffffffff,\n--\ndrivers/pci/controller/pci-hyperv.c-3371-\t\talign = 1ULL \u003c\u003c (63 - __builtin_clzll(hbus-\u003ehigh_mmio_space));\ndrivers/pci/controller/pci-hyperv.c:3372:\t\tret = vmbus_allocate_mmio(\u0026hbus-\u003ehigh_mmio_res, hbus-\u003ehdev,\ndrivers/pci/controller/pci-hyperv.c-3373-\t\t\t\t\t 0x100000000, -1,\n--\ndrivers/pci/controller/pci-hyperv.c=3409=static int hv_allocate_config_window(struct hv_pcibus_device *hbus)\n--\ndrivers/pci/controller/pci-hyperv.c-3416-\t */\ndrivers/pci/controller/pci-hyperv.c:3417:\tret = vmbus_allocate_mmio(\u0026hbus-\u003emem_config, hbus-\u003ehdev, 0, -1,\ndrivers/pci/controller/pci-hyperv.c-3418-\t\t\t\t PCI_CONFIG_MMIO_LENGTH, 0x1000, false);\n--\ndrivers/pci/controller/pci-hyperv.c-3422-\t/*\ndrivers/pci/controller/pci-hyperv.c:3423:\t * vmbus_allocate_mmio() gets used for allocating both device endpoint\ndrivers/pci/controller/pci-hyperv.c-3424-\t * resource claims (those which cannot be overlapped) and the ranges\n--\ninclude/linux/hyperv.h=1342=void vmbus_hvsock_device_unregister(struct vmbus_channel *channel);\ninclude/linux/hyperv.h-1343-\ninclude/linux/hyperv.h:1344:int vmbus_allocate_mmio(struct resource **new, struct hv_device *device_obj,\ninclude/linux/hyperv.h-1345-\t\t\tresource_size_t min, resource_size_t max,\n--\nkernel/fork.c=633=static void check_mm(struct mm_struct *mm)\n--\nkernel/fork.c-657-\nkernel/fork.c:658:#define allocate_mm()\t(kmem_cache_alloc(mm_cachep, GFP_KERNEL))\nkernel/fork.c-659-#define free_mm(mm)\t(kmem_cache_free(mm_cachep, (mm)))\n--\nkernel/fork.c=1176=struct mm_struct *mm_alloc(void)\n--\nkernel/fork.c-1179-\nkernel/fork.c:1180:\tmm = allocate_mm();\nkernel/fork.c-1181-\tif (!mm)\n--\nkernel/fork.c=1531=static struct mm_struct *dup_mm(struct task_struct *tsk,\n--\nkernel/fork.c-1536-\nkernel/fork.c:1537:\tmm = allocate_mm();\nkernel/fork.c-1538-\tif (!mm)\n"
} |
| 11/3 | 2026/09/06 04:16 | llm | kmsan-evaluator |
0mModel:gemini-3.7-flash Args: null Results: null Tokens: input: 32452 output: 34 thoughts: 42 Thoughts: Here's my analysis of the provided text, summarized in the first person, assuming I'm an expert in kernel internals: **Diving into `mm_init` in `kernel/fork.c`** Alright, let's get down to brass tacks. I'm going to quickly refresh my memory on the `mm_init` function's behavior within the `kernel/fork.c` context. Specifically, I'm focusing on lines 1140 through 1175. That should be the core of what I'm looking for - likely the memory management initialization steps after a fork, before the child's context becomes fully operational. I expect to see the basic memory descriptor set up, perhaps memory map copying or reference counting adjustments. Let's see how they handle the inherited memory layout from the parent process. This section should offer insight into the critical memory allocation and setup stages for the newly forked process. |
| 12/3 | 2026/09/06 04:16 | tool | read-file |
0mArgs:{
"File": "kernel/fork.c",
"FirstLine": 1140,
"LineCount": 40
}Results: {
"Contents": "1140:\t\n1141:\t\tif (init_new_context(p, mm))\n1142:\t\t\tgoto fail_nocontext;\n1143:\t\n1144:\t\tif (mm_alloc_cid(mm, p))\n1145:\t\t\tgoto fail_cid;\n1146:\t\n1147:\t\tif (mm_alloc_sched(mm))\n1148:\t\t\tgoto fail_sched;\n1149:\t\n1150:\t\tif (percpu_counter_tree_init_many(mm-\u003erss_stat, get_rss_stat_items(mm),\n1151:\t\t\t\t\t\t NR_MM_COUNTERS, RSS_STAT_BATCH_SIZE,\n1152:\t\t\t\t\t\t GFP_KERNEL_ACCOUNT))\n1153:\t\t\tgoto fail_pcpu;\n1154:\t\n1155:\t\tlru_gen_init_mm(mm);\n1156:\t\treturn mm;\n1157:\t\n1158:\tfail_pcpu:\n1159:\t\tmm_destroy_sched(mm);\n1160:\tfail_sched:\n1161:\t\tmm_destroy_cid(mm);\n1162:\tfail_cid:\n1163:\t\tdestroy_context(mm);\n1164:\tfail_nocontext:\n1165:\t\tmm_free_id(mm);\n1166:\tfail_noid:\n1167:\t\tmm_free_pgd(mm);\n1168:\tfail_mm_init:\n1169:\t\tfree_mm(mm);\n1170:\t\treturn NULL;\n1171:\t}\n1172:\t\n1173:\t/*\n1174:\t * Allocate and initialize an mm_struct.\n1175:\t */\n1176:\tstruct mm_struct *mm_alloc(void)\n1177:\t{\n1178:\t\tstruct mm_struct *mm;\n1179:\t\n"
} |
| 13/3 | 2026/09/06 04:16 | llm | kmsan-evaluator |
0mModel:gemini-3.7-flash Args: null Results: null Tokens: input: 33252 output: 25 thoughts: 205 Thoughts: **My Initial Assessment of Memory Initialization** Okay, let's break this down systematically. From what I see in `mm_init_cpumask` and `percpu_counter_tree_init_many`, the initial memory setup looks sound. I'm noting the `memset` call within `percpu_counter_tree_init_many`; that takes care of zeroing out the memory. Specifically, it's initializing the `items` array. The assignments to `counter->bias`, `counter->batch_size`, `counter->level0`, `counter->items`, `counter->approx_sum`, `counter->level0_bit_mask`, and `counter->approx_accuracy_range` are all solid indicators of correct initialization of the per-CPU counters. The key data members of the counter structure appear to be receiving their required initialization as a part of the percpu counter initialization. Now, I need to shift my focus to `init_mm`. I see it's employing `MM_STRUCT_FLEXIBLE_ARRAY_INIT` for its flexible array. I'll need to double-check the definition of `init_mm` itself and, importantly, what happens during the initialization of its `rss_stat` member. Making sure the RSS stat is properly initialized is the next crucial step. This is where I'll expect to see if there are any lingering issues. |
| 14/3 | 2026/09/06 04:16 | tool | grepper |
0mArgs:{
"Expression": "init_mm",
"PathPrefix": "mm/"
}Results: {
"Output": "\nFull output is too long, showing 500 out of 543 lines.\nUse more precise expression if possible.\n\nmm/highmem.c=197=static void flush_all_zero_pkmaps(void)\n--\nmm/highmem.c-229-\t\tpage = pte_page(ptent);\nmm/highmem.c:230:\t\tpte_clear(\u0026init_mm, PKMAP_ADDR(i), \u0026pkmap_page_table[i]);\nmm/highmem.c-231-\n--\nmm/highmem.c=246=static inline unsigned long map_new_virtual(struct page *page)\n--\nmm/highmem.c-290-\tvaddr = PKMAP_ADDR(last_pkmap_nr);\nmm/highmem.c:291:\tset_pte_at(\u0026init_mm, vaddr,\nmm/highmem.c-292-\t\t \u0026(pkmap_page_table[last_pkmap_nr]), mk_pte(page, kmap_prot));\n--\nmm/highmem.c=550=void *__kmap_local_pfn_prot(unsigned long pfn, pgprot_t prot)\n--\nmm/highmem.c-566-\tpteval = pfn_pte(pfn, prot);\nmm/highmem.c:567:\tarch_kmap_local_set_pte(\u0026init_mm, vaddr, kmap_pte, pteval);\nmm/highmem.c-568-\tarch_kmap_local_post_map(vaddr, pteval);\n--\nmm/highmem.c=597=void kunmap_local_indexed(const void *vaddr)\n--\nmm/highmem.c-626-\tarch_kmap_local_pre_unmap(addr);\nmm/highmem.c:627:\tpte_clear(\u0026init_mm, addr, kmap_pte);\nmm/highmem.c-628-\tarch_kmap_local_post_unmap(addr);\n--\nmm/highmem.c=646=void __kmap_local_sched_out(void)\n--\nmm/highmem.c-677-\t\tarch_kmap_local_pre_unmap(addr);\nmm/highmem.c:678:\t\tpte_clear(\u0026init_mm, addr, kmap_pte);\nmm/highmem.c-679-\t\tarch_kmap_local_post_unmap(addr);\n--\nmm/highmem.c=683=void __kmap_local_sched_in(void)\n--\nmm/highmem.c-706-\t\tkmap_pte = kmap_get_pte(addr, idx);\nmm/highmem.c:707:\t\tset_pte_at(\u0026init_mm, addr, kmap_pte, pteval);\nmm/highmem.c-708-\t\tarch_kmap_local_post_map(addr, pteval);\n--\nmm/hugetlb_vmemmap.c=53=static int vmemmap_split_pmd(pmd_t *pmd, struct page *head, unsigned long start,\n--\nmm/hugetlb_vmemmap.c-60-\nmm/hugetlb_vmemmap.c:61:\tpgtable = pte_alloc_one_kernel(\u0026init_mm);\nmm/hugetlb_vmemmap.c-62-\tif (!pgtable)\n--\nmm/hugetlb_vmemmap.c-64-\nmm/hugetlb_vmemmap.c:65:\tpmd_populate_kernel(\u0026init_mm, \u0026__pmd, pgtable);\nmm/hugetlb_vmemmap.c-66-\n--\nmm/hugetlb_vmemmap.c-72-\t\tpte = pte_offset_kernel(\u0026__pmd, addr);\nmm/hugetlb_vmemmap.c:73:\t\tset_pte_at(\u0026init_mm, addr, pte, entry);\nmm/hugetlb_vmemmap.c-74-\t}\nmm/hugetlb_vmemmap.c-75-\nmm/hugetlb_vmemmap.c:76:\tspin_lock(\u0026init_mm.page_table_lock);\nmm/hugetlb_vmemmap.c-77-\tif (likely(pmd_leaf(*pmd))) {\n--\nmm/hugetlb_vmemmap.c-87-\t\tsmp_wmb();\nmm/hugetlb_vmemmap.c:88:\t\tpmd_populate_kernel(\u0026init_mm, pmd, pgtable);\nmm/hugetlb_vmemmap.c-89-\t\tif (!(walk-\u003eflags \u0026 VMEMMAP_SPLIT_NO_TLB_FLUSH))\n--\nmm/hugetlb_vmemmap.c-91-\t} else {\nmm/hugetlb_vmemmap.c:92:\t\tpte_free_kernel(\u0026init_mm, pgtable);\nmm/hugetlb_vmemmap.c-93-\t}\nmm/hugetlb_vmemmap.c:94:\tspin_unlock(\u0026init_mm.page_table_lock);\nmm/hugetlb_vmemmap.c-95-\n--\nmm/hugetlb_vmemmap.c=99=static int vmemmap_pmd_entry(pmd_t *pmd, unsigned long addr,\n--\nmm/hugetlb_vmemmap.c-109-\nmm/hugetlb_vmemmap.c:110:\tspin_lock(\u0026init_mm.page_table_lock);\nmm/hugetlb_vmemmap.c-111-\thead = pmd_leaf(*pmd) ? pmd_page(*pmd) : NULL;\n--\nmm/hugetlb_vmemmap.c-132-\t}\nmm/hugetlb_vmemmap.c:133:\tspin_unlock(\u0026init_mm.page_table_lock);\nmm/hugetlb_vmemmap.c-134-\tif (!head || ret)\n--\nmm/hugetlb_vmemmap.c=156=static int vmemmap_remap_range(unsigned long start, unsigned long end,\n--\nmm/hugetlb_vmemmap.c-162-\nmm/hugetlb_vmemmap.c:163:\tmmap_read_lock(\u0026init_mm);\nmm/hugetlb_vmemmap.c-164-\tret = walk_kernel_page_table_range(start, end, \u0026vmemmap_remap_ops,\nmm/hugetlb_vmemmap.c-165-\t\t\t\t NULL, walk);\nmm/hugetlb_vmemmap.c:166:\tmmap_read_unlock(\u0026init_mm);\nmm/hugetlb_vmemmap.c-167-\tif (ret)\n--\nmm/hugetlb_vmemmap.c=202=static void vmemmap_remap_pte(pte_t *pte, unsigned long addr,\n--\nmm/hugetlb_vmemmap.c-232-\tlist_add(\u0026page-\u003elru, walk-\u003evmemmap_pages);\nmm/hugetlb_vmemmap.c:233:\tset_pte_at(\u0026init_mm, addr, pte, entry);\nmm/hugetlb_vmemmap.c-234-}\n--\nmm/hugetlb_vmemmap.c=236=static void vmemmap_restore_pte(pte_t *pte, unsigned long addr,\n--\nmm/hugetlb_vmemmap.c-259-\tsmp_wmb();\nmm/hugetlb_vmemmap.c:260:\tset_pte_at(\u0026init_mm, addr, pte, mk_pte(dst, PAGE_KERNEL));\nmm/hugetlb_vmemmap.c-261-}\n--\nmm/init-mm.c=20=const struct vm_operations_struct vma_dummy_vm_ops;\n--\nmm/init-mm.c-28- *\nmm/init-mm.c:29: * Since there is only one init_mm in the entire system, keep it simple\nmm/init-mm.c-30- * and size this cpu_bitmask to NR_CPUS.\nmm/init-mm.c-31- */\nmm/init-mm.c:32:struct mm_struct init_mm = {\nmm/init-mm.c:33:\t.mm_mt\t\t= MTREE_INIT_EXT(mm_mt, MM_MT_FLAGS, init_mm.mmap_lock),\nmm/init-mm.c-34-\t.pgd\t\t= swapper_pg_dir,\n--\nmm/init-mm.c-36-\t.mm_count\t= ATOMIC_INIT(1),\nmm/init-mm.c:37:\t.write_protect_seq = SEQCNT_ZERO(init_mm.write_protect_seq),\nmm/init-mm.c:38:\tMMAP_LOCK_INITIALIZER(init_mm)\nmm/init-mm.c:39:\t.page_table_lock = __SPIN_LOCK_UNLOCKED(init_mm.page_table_lock),\nmm/init-mm.c:40:\t.arg_lock\t= __SPIN_LOCK_UNLOCKED(init_mm.arg_lock),\nmm/init-mm.c:41:\t.mmlist\t\t= LIST_HEAD_INIT(init_mm.mmlist),\nmm/init-mm.c-42-#ifdef CONFIG_PER_VMA_LOCK\nmm/init-mm.c:43:\t.vma_writer_wait = __RCUWAIT_INITIALIZER(init_mm.vma_writer_wait),\nmm/init-mm.c:44:\t.mm_lock_seq\t= SEQCNT_ZERO(init_mm.mm_lock_seq),\nmm/init-mm.c-45-#endif\nmm/init-mm.c-46-#ifdef CONFIG_SCHED_MM_CID\nmm/init-mm.c:47:\t.mm_cid.lock = __RAW_SPIN_LOCK_UNLOCKED(init_mm.mm_cid.lock),\nmm/init-mm.c-48-#endif\nmm/init-mm.c-49-\t.flexible_array\t= MM_STRUCT_FLEXIBLE_ARRAY_INIT,\nmm/init-mm.c:50:\tINIT_MM_CONTEXT(init_mm)\nmm/init-mm.c-51-};\nmm/init-mm.c-52-\nmm/init-mm.c:53:void setup_initial_init_mm(void *start_code, void *end_code,\nmm/init-mm.c-54-\t\t\t void *end_data, void *brk)\nmm/init-mm.c-55-{\nmm/init-mm.c:56:\tinit_mm.start_code = (unsigned long)start_code;\nmm/init-mm.c:57:\tinit_mm.end_code = (unsigned long)end_code;\nmm/init-mm.c:58:\tinit_mm.end_data = (unsigned long)end_data;\nmm/init-mm.c:59:\tinit_mm.brk = (unsigned long)brk;\nmm/init-mm.c-60-}\n--\nmm/internal.h=1619=static inline void mm_prepare_for_swap_entries(struct mm_struct *mm)\n--\nmm/internal.h-1623-\t\tif (list_empty(\u0026mm-\u003emmlist))\nmm/internal.h:1624:\t\t\tlist_add(\u0026mm-\u003emmlist, \u0026init_mm.mmlist);\nmm/internal.h-1625-\t\tspin_unlock(\u0026mmlist_lock);\n--\nmm/kasan/init.c=92=static void __ref zero_pte_populate(pmd_t *pmd, unsigned long addr,\n--\nmm/kasan/init.c-102-\twhile (addr + PAGE_SIZE \u003c= end) {\nmm/kasan/init.c:103:\t\tset_pte_at(\u0026init_mm, addr, pte, zero_pte);\nmm/kasan/init.c-104-\t\taddr += PAGE_SIZE;\n--\nmm/kasan/init.c=109=static int __ref zero_pmd_populate(pud_t *pud, unsigned long addr,\n--\nmm/kasan/init.c-118-\t\tif (IS_ALIGNED(addr, PMD_SIZE) \u0026\u0026 end - addr \u003e= PMD_SIZE) {\nmm/kasan/init.c:119:\t\t\tpmd_populate_kernel(\u0026init_mm, pmd,\nmm/kasan/init.c-120-\t\t\t\t\tlm_alias(kasan_early_shadow_pte));\n--\nmm/kasan/init.c-127-\t\t\tif (slab_is_available())\nmm/kasan/init.c:128:\t\t\t\tp = pte_alloc_one_kernel(\u0026init_mm);\nmm/kasan/init.c-129-\t\t\telse {\n--\nmm/kasan/init.c-135-\nmm/kasan/init.c:136:\t\t\tpmd_populate_kernel(\u0026init_mm, pmd, p);\nmm/kasan/init.c-137-\t\t}\n--\nmm/kasan/init.c=144=static int __ref zero_pud_populate(p4d_t *p4d, unsigned long addr,\n--\nmm/kasan/init.c-154-\nmm/kasan/init.c:155:\t\t\tpud_populate(\u0026init_mm, pud,\nmm/kasan/init.c-156-\t\t\t\t\tlm_alias(kasan_early_shadow_pmd));\nmm/kasan/init.c-157-\t\t\tpmd = pmd_offset(pud, addr);\nmm/kasan/init.c:158:\t\t\tpmd_populate_kernel(\u0026init_mm, pmd,\nmm/kasan/init.c-159-\t\t\t\t\tlm_alias(kasan_early_shadow_pte));\n--\nmm/kasan/init.c-166-\t\t\tif (slab_is_available()) {\nmm/kasan/init.c:167:\t\t\t\tp = pmd_alloc(\u0026init_mm, pud, addr);\nmm/kasan/init.c-168-\t\t\t\tif (!p)\n--\nmm/kasan/init.c-172-\t\t\t\tpmd_init(p);\nmm/kasan/init.c:173:\t\t\t\tpud_populate(\u0026init_mm, pud, p);\nmm/kasan/init.c-174-\t\t\t}\n--\nmm/kasan/init.c=182=static int __ref zero_p4d_populate(pgd_t *pgd, unsigned long addr,\n--\nmm/kasan/init.c-196-\t\t\tpud = pud_offset(p4d, addr);\nmm/kasan/init.c:197:\t\t\tpud_populate(\u0026init_mm, pud,\nmm/kasan/init.c-198-\t\t\t\t\tlm_alias(kasan_early_shadow_pmd));\nmm/kasan/init.c-199-\t\t\tpmd = pmd_offset(pud, addr);\nmm/kasan/init.c:200:\t\t\tpmd_populate_kernel(\u0026init_mm, pmd,\nmm/kasan/init.c-201-\t\t\t\t\tlm_alias(kasan_early_shadow_pte));\n--\nmm/kasan/init.c-208-\t\t\tif (slab_is_available()) {\nmm/kasan/init.c:209:\t\t\t\tp = pud_alloc(\u0026init_mm, p4d, addr);\nmm/kasan/init.c-210-\t\t\t\tif (!p)\n--\nmm/kasan/init.c=230=int __ref kasan_populate_early_shadow(const void *shadow_start,\n--\nmm/kasan/init.c-259-\t\t\tpud = pud_offset(p4d, addr);\nmm/kasan/init.c:260:\t\t\tpud_populate(\u0026init_mm, pud,\nmm/kasan/init.c-261-\t\t\t\t\tlm_alias(kasan_early_shadow_pmd));\nmm/kasan/init.c-262-\t\t\tpmd = pmd_offset(pud, addr);\nmm/kasan/init.c:263:\t\t\tpmd_populate_kernel(\u0026init_mm, pmd,\nmm/kasan/init.c-264-\t\t\t\t\tlm_alias(kasan_early_shadow_pte));\n--\nmm/kasan/init.c-270-\t\t\tif (slab_is_available()) {\nmm/kasan/init.c:271:\t\t\t\tif (!p4d_alloc(\u0026init_mm, pgd, addr))\nmm/kasan/init.c-272-\t\t\t\t\treturn -ENOMEM;\n--\nmm/kasan/init.c=284=static void kasan_free_pte(pte_t *pte_start, pmd_t *pmd)\n--\nmm/kasan/init.c-294-\nmm/kasan/init.c:295:\tpte_free_kernel(\u0026init_mm, pte_start);\nmm/kasan/init.c-296-\tpmd_clear(pmd);\n--\nmm/kasan/init.c=299=static void kasan_free_pmd(pmd_t *pmd_start, pud_t *pud)\n--\nmm/kasan/init.c-309-\nmm/kasan/init.c:310:\tpmd_free(\u0026init_mm, pmd_start);\nmm/kasan/init.c-311-\tpud_clear(pud);\n--\nmm/kasan/init.c=314=static void kasan_free_pud(pud_t *pud_start, p4d_t *p4d)\n--\nmm/kasan/init.c-324-\nmm/kasan/init.c:325:\tpud_free(\u0026init_mm, pud_start);\nmm/kasan/init.c-326-\tp4d_clear(p4d);\n--\nmm/kasan/init.c=329=static void kasan_free_p4d(p4d_t *p4d_start, pgd_t *pgd)\n--\nmm/kasan/init.c-339-\nmm/kasan/init.c:340:\tp4d_free(\u0026init_mm, p4d_start);\nmm/kasan/init.c-341-\tpgd_clear(pgd);\n--\nmm/kasan/init.c=344=static void kasan_remove_pte_table(pte_t *pte, unsigned long addr,\n--\nmm/kasan/init.c-361-\t\t\tcontinue;\nmm/kasan/init.c:362:\t\tpte_clear(\u0026init_mm, addr, pte);\nmm/kasan/init.c-363-\t}\n--\nmm/kasan/shadow.c=300=static int kasan_populate_vmalloc_pte(pte_t *ptep, unsigned long addr,\n--\nmm/kasan/shadow.c-314-\nmm/kasan/shadow.c:315:\tspin_lock(\u0026init_mm.page_table_lock);\nmm/kasan/shadow.c-316-\tif (likely(pte_none(ptep_get(ptep)))) {\nmm/kasan/shadow.c:317:\t\tset_pte_at(\u0026init_mm, addr, ptep, pte);\nmm/kasan/shadow.c-318-\t\tdata-\u003epages[index] = NULL;\nmm/kasan/shadow.c-319-\t}\nmm/kasan/shadow.c:320:\tspin_unlock(\u0026init_mm.page_table_lock);\nmm/kasan/shadow.c-321-\n--\nmm/kasan/shadow.c=357=static int __kasan_populate_vmalloc_do(unsigned long start, unsigned long end, gfp_t gfp_mask)\n--\nmm/kasan/shadow.c-380-\t\tflags = memalloc_apply_gfp_scope(gfp_mask);\nmm/kasan/shadow.c:381:\t\tret = apply_to_page_range(\u0026init_mm, start, nr_pages * PAGE_SIZE,\nmm/kasan/shadow.c-382-\t\t\t\t\t kasan_populate_vmalloc_pte, \u0026data);\n--\nmm/kasan/shadow.c=468=static int kasan_depopulate_vmalloc_pte(pte_t *ptep, unsigned long addr,\n--\nmm/kasan/shadow.c-475-\nmm/kasan/shadow.c:476:\tspin_lock(\u0026init_mm.page_table_lock);\nmm/kasan/shadow.c-477-\tpte = ptep_get(ptep);\n--\nmm/kasan/shadow.c-479-\tif (likely(!none))\nmm/kasan/shadow.c:480:\t\tpte_clear(\u0026init_mm, addr, ptep);\nmm/kasan/shadow.c:481:\tspin_unlock(\u0026init_mm.page_table_lock);\nmm/kasan/shadow.c-482-\n--\nmm/kasan/shadow.c=566=void __kasan_release_vmalloc(unsigned long start, unsigned long end,\n--\nmm/kasan/shadow.c-601-\t\tif (flags \u0026 KASAN_VMALLOC_PAGE_RANGE)\nmm/kasan/shadow.c:602:\t\t\tapply_to_existing_page_range(\u0026init_mm,\nmm/kasan/shadow.c-603-\t\t\t\t\t (unsigned long)shadow_start,\n--\nmm/memory.c=128=static const struct ctl_table mmu_sysctl_table[] = {\n--\nmm/memory.c-137-\nmm/memory.c:138:static int __init init_mm_sysctl(void)\nmm/memory.c-139-{\n--\nmm/memory.c-143-\nmm/memory.c:144:subsys_initcall(init_mm_sysctl);\nmm/memory.c-145-\n--\nmm/memory.c=463=int __pte_alloc_kernel(pmd_t *pmd)\nmm/memory.c-464-{\nmm/memory.c:465:\tpte_t *new = pte_alloc_one_kernel(\u0026init_mm);\nmm/memory.c-466-\tif (!new)\n--\nmm/memory.c-468-\nmm/memory.c:469:\tspin_lock(\u0026init_mm.page_table_lock);\nmm/memory.c-470-\tif (likely(pmd_none(*pmd))) {\t/* Has another populated it ? */\nmm/memory.c-471-\t\tsmp_wmb(); /* See comment in pmd_install() */\nmm/memory.c:472:\t\tpmd_populate_kernel(\u0026init_mm, pmd, new);\nmm/memory.c-473-\t\tnew = NULL;\nmm/memory.c-474-\t}\nmm/memory.c:475:\tspin_unlock(\u0026init_mm.page_table_lock);\nmm/memory.c-476-\tif (new)\nmm/memory.c:477:\t\tpte_free_kernel(\u0026init_mm, new);\nmm/memory.c-478-\treturn 0;\n--\nmm/memory.c=3442=static int apply_to_pte_range(struct mm_struct *mm, pmd_t *pmd,\n--\nmm/memory.c-3451-\tif (create) {\nmm/memory.c:3452:\t\tmapped_pte = pte = (mm == \u0026init_mm) ?\nmm/memory.c-3453-\t\t\tpte_alloc_kernel_track(pmd, addr, mask) :\n--\nmm/memory.c-3457-\t} else {\nmm/memory.c:3458:\t\tmapped_pte = pte = (mm == \u0026init_mm) ?\nmm/memory.c-3459-\t\t\tpte_offset_kernel(pmd, addr) :\n--\nmm/memory.c-3479-\nmm/memory.c:3480:\tif (mm != \u0026init_mm)\nmm/memory.c-3481-\t\tpte_unmap_unlock(mapped_pte, ptl);\n--\nmm/page_table_check.c=148=void __page_table_check_pte_clear(struct mm_struct *mm, unsigned long addr,\n--\nmm/page_table_check.c-150-{\nmm/page_table_check.c:151:\tif (\u0026init_mm == mm)\nmm/page_table_check.c-152-\t\treturn;\n--\nmm/page_table_check.c=169=void __page_table_check_pmd_clear(struct mm_struct *mm, unsigned long addr,\n--\nmm/page_table_check.c-171-{\nmm/page_table_check.c:172:\tif (\u0026init_mm == mm)\nmm/page_table_check.c-173-\t\treturn;\n--\nmm/page_table_check.c=181=void __page_table_check_pud_clear(struct mm_struct *mm, unsigned long addr,\n--\nmm/page_table_check.c-183-{\nmm/page_table_check.c:184:\tif (\u0026init_mm == mm)\nmm/page_table_check.c-185-\t\treturn;\n--\nmm/page_table_check.c=210=void __page_table_check_ptes_set(struct mm_struct *mm, unsigned long addr,\n--\nmm/page_table_check.c-214-\nmm/page_table_check.c:215:\tif (\u0026init_mm == mm)\nmm/page_table_check.c-216-\t\treturn;\n--\nmm/page_table_check.c=239=void __page_table_check_pmds_set(struct mm_struct *mm, unsigned long addr,\n--\nmm/page_table_check.c-244-\nmm/page_table_check.c:245:\tif (\u0026init_mm == mm)\nmm/page_table_check.c-246-\t\treturn;\n--\nmm/page_table_check.c=258=void __page_table_check_puds_set(struct mm_struct *mm, unsigned long addr,\n--\nmm/page_table_check.c-263-\nmm/page_table_check.c:264:\tif (\u0026init_mm == mm)\nmm/page_table_check.c-265-\t\treturn;\n--\nmm/page_table_check.c=274=void __page_table_check_pte_clear_range(struct mm_struct *mm,\n--\nmm/page_table_check.c-277-{\nmm/page_table_check.c:278:\tif (\u0026init_mm == mm)\nmm/page_table_check.c-279-\t\treturn;\n--\nmm/pagewalk.c=61=static int walk_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,\n--\nmm/pagewalk.c-74-\t\t */\nmm/pagewalk.c:75:\t\tif (walk-\u003emm == \u0026init_mm || addr \u003e= TASK_SIZE)\nmm/pagewalk.c-76-\t\t\tpte = pte_offset_kernel(pmd, addr);\n--\nmm/pagewalk.c-80-\t\t\terr = walk_pte_range_inner(pte, addr, end, walk);\nmm/pagewalk.c:81:\t\t\tif (walk-\u003emm != \u0026init_mm \u0026\u0026 addr \u003c TASK_SIZE)\nmm/pagewalk.c-82-\t\t\t\tpte_unmap(pte);\n--\nmm/pagewalk.c=623=int walk_kernel_page_table_range(unsigned long start, unsigned long end,\n--\nmm/pagewalk.c-633-\t */\nmm/pagewalk.c:634:\tmmap_assert_locked(\u0026init_mm);\nmm/pagewalk.c-635-\n--\nmm/pagewalk.c=646=int walk_kernel_page_table_range_lockless(unsigned long start, unsigned long end,\n--\nmm/pagewalk.c-650-\t\t.ops\t\t= ops,\nmm/pagewalk.c:651:\t\t.mm\t\t= \u0026init_mm,\nmm/pagewalk.c-652-\t\t.pgd\t\t= pgd,\n--\nmm/pagewalk.c=682=int walk_page_range_debug(struct mm_struct *mm, unsigned long start,\n--\nmm/pagewalk.c-706-\t *\nmm/pagewalk.c:707:\t * However, they both must also hold the init_mm lock to account for\nmm/pagewalk.c-708-\t * concurrent kernel page table freeing.\nmm/pagewalk.c-709-\t */\nmm/pagewalk.c:710:\tmmap_assert_write_locked(\u0026init_mm);\nmm/pagewalk.c-711-\n--\nmm/percpu.c=3153=void __init __weak pcpu_populate_pte(unsigned long addr)\n--\nmm/percpu.c-3173-\t\tpmd = memblock_alloc_or_panic(PMD_TABLE_SIZE, PMD_TABLE_SIZE);\nmm/percpu.c:3174:\t\tpud_populate(\u0026init_mm, pud, pmd);\nmm/percpu.c-3175-\t}\n--\nmm/percpu.c-3181-\t\tnew = memblock_alloc_or_panic(PTE_TABLE_SIZE, PTE_TABLE_SIZE);\nmm/percpu.c:3182:\t\tpmd_populate_kernel(\u0026init_mm, pmd, new);\nmm/percpu.c-3183-\t}\n--\nmm/ptdump.c=175=void ptdump_walk_pgd(struct ptdump_state *st, struct mm_struct *mm, pgd_t *pgd)\n--\nmm/ptdump.c-180-\tmmap_write_lock(mm);\nmm/ptdump.c:181:\t/* To stabilise kernel page tables we must hold the init_mm lock too. */\nmm/ptdump.c:182:\tif (mm != \u0026init_mm)\nmm/ptdump.c:183:\t\tmmap_write_lock_nested(\u0026init_mm, SINGLE_DEPTH_NESTING);\nmm/ptdump.c-184-\n--\nmm/ptdump.c-190-\nmm/ptdump.c:191:\tif (mm != \u0026init_mm)\nmm/ptdump.c:192:\t\tmmap_write_unlock(\u0026init_mm);\nmm/ptdump.c-193-\tmmap_write_unlock(mm);\n--\nmm/sparse-vmemmap.c=241=static pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node,\n--\nmm/sparse-vmemmap.c-258-\t\t\t/*\nmm/sparse-vmemmap.c:259:\t\t\t * When a PTE/PMD entry is freed from the init_mm\nmm/sparse-vmemmap.c-260-\t\t\t * there's a free_pages() call to this page allocated\n--\nmm/sparse-vmemmap.c-270-\t\tentry = pfn_pte(ptpfn, PAGE_KERNEL);\nmm/sparse-vmemmap.c:271:\t\tset_pte_at(\u0026init_mm, addr, pte, entry);\nmm/sparse-vmemmap.c-272-\t} else if (WARN_ON_ONCE(vmemmap_optimizable_pfn(pfn)))\n--\nmm/sparse-vmemmap.c=277=static pmd_t * __meminit vmemmap_pmd_populate(pud_t *pud, unsigned long addr, int node)\n--\nmm/sparse-vmemmap.c-284-\t\tkernel_pte_init(p);\nmm/sparse-vmemmap.c:285:\t\tpmd_populate_kernel(\u0026init_mm, pmd, p);\nmm/sparse-vmemmap.c-286-\t}\n--\nmm/sparse-vmemmap.c=290=static pud_t * __meminit vmemmap_pud_populate(p4d_t *p4d, unsigned long addr, int node)\n--\nmm/sparse-vmemmap.c-297-\t\tpmd_init(p);\nmm/sparse-vmemmap.c:298:\t\tpud_populate(\u0026init_mm, pud, p);\nmm/sparse-vmemmap.c-299-\t}\n--\nmm/sparse-vmemmap.c=395=void vmemmap_wrprotect_hvo(unsigned long addr, unsigned long end,\n--\nmm/sparse-vmemmap.c-402-\t\tpte = virt_to_kpte(maddr);\nmm/sparse-vmemmap.c:403:\t\tptep_set_wrprotect(\u0026init_mm, maddr, pte);\nmm/sparse-vmemmap.c-404-\t}\n--\nmm/swapfile.c=2736=static int try_to_unuse(unsigned int type)\n--\nmm/swapfile.c-2754-\nmm/swapfile.c:2755:\tprev_mm = \u0026init_mm;\nmm/swapfile.c-2756-\tmmget(prev_mm);\n--\nmm/swapfile.c-2758-\tspin_lock(\u0026mmlist_lock);\nmm/swapfile.c:2759:\tp = \u0026init_mm.mmlist;\nmm/swapfile.c-2760-\twhile (swap_usage_in_pages(si) \u0026\u0026\nmm/swapfile.c-2761-\t !signal_pending(current) \u0026\u0026\nmm/swapfile.c:2762:\t (p = p-\u003enext) != \u0026init_mm.mmlist) {\nmm/swapfile.c-2763-\n--\nmm/swapfile.c=2844=static void drain_mmlist(void)\n--\nmm/swapfile.c-2852-\tspin_lock(\u0026mmlist_lock);\nmm/swapfile.c:2853:\tlist_for_each_safe(p, next, \u0026init_mm.mmlist)\nmm/swapfile.c-2854-\t\tlist_del_init(p);\n--\nmm/vmalloc.c=96=static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,\n--\nmm/vmalloc.c-129-\t\t\tentry = arch_make_huge_pte(entry, ilog2(size), 0);\nmm/vmalloc.c:130:\t\t\tset_huge_pte_at(\u0026init_mm, addr, pte, entry, size);\nmm/vmalloc.c-131-\t\t\tpfn += PFN_DOWN(size);\n--\nmm/vmalloc.c-134-#endif\nmm/vmalloc.c:135:\t\tset_pte_at(\u0026init_mm, addr, pte, pfn_pte(pfn, prot));\nmm/vmalloc.c-136-\t\tpfn++;\n--\nmm/vmalloc.c=144=static int vmap_try_huge_pmd(pmd_t *pmd, unsigned long addr, unsigned long end,\n--\nmm/vmalloc.c-172-\t */\nmm/vmalloc.c:173:\tscoped_cond_guard(mmap_read_lock_try, return 0, \u0026init_mm) {\nmm/vmalloc.c-174-\t\tif (!pmd_free_pte_page(pmd, addr))\n--\nmm/vmalloc.c=180=static int vmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end,\n--\nmm/vmalloc.c-187-\nmm/vmalloc.c:188:\tpmd = pmd_alloc_track(\u0026init_mm, pud, addr, mask);\nmm/vmalloc.c-189-\tif (!pmd)\n--\nmm/vmalloc.c=207=static int vmap_try_huge_pud(pud_t *pud, unsigned long addr, unsigned long end,\n--\nmm/vmalloc.c-229-\t/* See comment in vmap_try_huge_pmd(). */\nmm/vmalloc.c:230:\tscoped_cond_guard(mmap_read_lock_try, return 0, \u0026init_mm) {\nmm/vmalloc.c-231-\t\tif (!pud_free_pmd_page(pud, addr))\n--\nmm/vmalloc.c=237=static int vmap_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end,\n--\nmm/vmalloc.c-244-\nmm/vmalloc.c:245:\tpud = pud_alloc_track(\u0026init_mm, p4d, addr, mask);\nmm/vmalloc.c-246-\tif (!pud)\n--\nmm/vmalloc.c=264=static int vmap_try_huge_p4d(p4d_t *p4d, unsigned long addr, unsigned long end,\n--\nmm/vmalloc.c-286-\t/* See comment in vmap_try_huge_pmd(). */\nmm/vmalloc.c:287:\tscoped_cond_guard(mmap_read_lock_try, return 0, \u0026init_mm) {\nmm/vmalloc.c-288-\t\tif (!p4d_free_pud_page(p4d, addr))\n--\nmm/vmalloc.c=294=static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end,\n--\nmm/vmalloc.c-301-\nmm/vmalloc.c:302:\tp4d = p4d_alloc_track(\u0026init_mm, pgd, addr, mask);\nmm/vmalloc.c-303-\tif (!p4d)\n--\nmm/vmalloc.c=389=static void vunmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,\n--\nmm/vmalloc.c-406-\t\t\t}\nmm/vmalloc.c:407:\t\t\tptent = huge_ptep_get_and_clear(\u0026init_mm, addr, pte, size);\nmm/vmalloc.c-408-\t\t\tif (WARN_ON(end - addr \u003c size))\n--\nmm/vmalloc.c-411-#endif\n\n"
} |
| 15/3 | 2026/09/06 04:16 | llm | kmsan-evaluator |
0mModel:gemini-3.7-flash Args: null Results: null Tokens: input: 43671 output: 380 thoughts: 995 Thoughts: **KMSAN Assessment: No Uninitialized Memory Issues Detected** Alright, let's break this down. First, the `init_mm` structure's flexible array is initialized with `MM_STRUCT_FLEXIBLE_ARRAY_INIT`, which effectively zeroes it. Importantly, the `rss_stat` within `init_mm` isn't relevant for RSS accounting in kernel threads/`init_mm`, so that's a non-issue. My main concern is always uninitialized data, so I'm focusing on that. I've gone through the code, focusing on the following areas: 1. `percpu_counter_tree_init_many`: This function meticulously zeros the memory for its items and allocates zeroed percpu memory via `__alloc_percpu_gfp`. Everything's initialized: `batch_size`, `bias`, `level0`, `items`, and the sum approximation fields, all initialized. Even when the counter configuration is such that there is no level0, it is properly handled. 2. `percpu_counter_tree_add`: Operates on initialized counters (atomic or percpu). 3. `percpu_counter_tree_approximate_sum`, `percpu_counter_tree_precise_sum`, and any comparisons: Reads from initialized counters. 4. `get_mm_counter`, `get_mm_counter_sum`, `inc_mm_counter`, etc.: These are memory management related and all operate on the existing structures. 5. I checked explicitly for any potential `copy_to_user` calls with uninitialized data. Nope, the counters are numerical values (longs) and any outputs (procfs, sysfs, statm) format numbers using `sprintf` or `seq_printf`, so no risk of uninitialized memory making it to userspace. No uninitialized struct padding is sent to user space, and no uninitialized memory. 6. The code appears to be sound in terms of uninitialized memory. Potential bugs are more in the areas of concurrency, algorithmic correctness (tree carry propagation, overflow), and out-of-bounds access within tree levels, but these would be identified by KASAN/LOCKDEP/KUnit tests, not through KMSAN. The KMSAN criteria are: 1. Kernel structures sent to userspace with uninitialized fields/padding: Not applicable. 2. Conditional logic based on potentially uninitialized variables: Not applicable. 3. Complex data structures with uninitialized fields read later: Not applicable. 4. Uninitialized bytes accessed through bounds/lengths: Not applicable. Since none of the KMSAN checks apply, and all allocated structs and buffers are explicitly zeroed, it's safe to say: `NeedsKMSAN = false`. I am confident in this assessment. This patch implements a new Hierarchical Per-CPU Counters (HPCC) infrastructure for scalable RSS statistics in `mm_struct`, and there are no apparent uninitialized memory concerns. Now I will use `set-results` with this reasoning. |
| 16/3 | 2026/09/06 04:16 | tool | set-results |
0mArgs:{
"KMSANReasoning": "The patch series introduces Hierarchical Per-CPU Counters (HPCC) and transitions mm_struct RSS accounting (rss_stat) to use HPCC instead of standard percpu counters.\n\nEvaluation of uninitialized memory risk:\n1. All allocated structures and tree items are fully zero-initialized before use:\n - In `percpu_counter_tree_init_many()`, `items` are explicitly zeroed via `memset()`, and `level0` per-CPU memory is allocated via `__alloc_percpu_gfp()` which returns zero-filled memory. All struct fields of `struct percpu_counter_tree` are explicitly initialized.\n - For `init_mm`, the statically allocated flexible array containing tree items is zero-initialized via `MM_STRUCT_FLEXIBLE_ARRAY_INIT`.\n - In `mm_init()`, RSS counters are initialized via `percpu_counter_tree_init_many()` with `get_rss_stat_items(mm)`.\n2. No kernel structures or buffers with uninitialized padding or fields are exposed or copied to user space (counter values are read as scalar `long` / `unsigned long` values and formatted into strings or proc/sysfs seq_files).\n3. No branching or calculations depend on uninitialized variables or memory.\n4. Any potential issues in this patch series (such as indexing errors in the tree hierarchy, arithmetic/carry calculation bugs, race conditions, or out-of-bounds accesses) fall squarely under KASAN, LOCKDEP, and KUnit testing rather than uninitialized memory detectors.\n\nTherefore, dedicated KMSAN fuzzing is not required (NeedsKMSAN=false).",
"NeedsKMSAN": false
}Results: {
"KMSANReasoning": "The patch series introduces Hierarchical Per-CPU Counters (HPCC) and transitions mm_struct RSS accounting (rss_stat) to use HPCC instead of standard percpu counters.\n\nEvaluation of uninitialized memory risk:\n1. All allocated structures and tree items are fully zero-initialized before use:\n - In `percpu_counter_tree_init_many()`, `items` are explicitly zeroed via `memset()`, and `level0` per-CPU memory is allocated via `__alloc_percpu_gfp()` which returns zero-filled memory. All struct fields of `struct percpu_counter_tree` are explicitly initialized.\n - For `init_mm`, the statically allocated flexible array containing tree items is zero-initialized via `MM_STRUCT_FLEXIBLE_ARRAY_INIT`.\n - In `mm_init()`, RSS counters are initialized via `percpu_counter_tree_init_many()` with `get_rss_stat_items(mm)`.\n2. No kernel structures or buffers with uninitialized padding or fields are exposed or copied to user space (counter values are read as scalar `long` / `unsigned long` values and formatted into strings or proc/sysfs seq_files).\n3. No branching or calculations depend on uninitialized variables or memory.\n4. Any potential issues in this patch series (such as indexing errors in the tree hierarchy, arithmetic/carry calculation bugs, race conditions, or out-of-bounds accesses) fall squarely under KASAN, LOCKDEP, and KUnit testing rather than uninitialized memory detectors.\n\nTherefore, dedicated KMSAN fuzzing is not required (NeedsKMSAN=false).",
"NeedsKMSAN": false
} |
| Total Calls | Total Tokens | Avg Tokens | Total Duration (Seconds) | Avg Duration (Seconds) |
|---|
| Total Calls | Total Duration (Seconds) | Avg Duration (Seconds) |
|---|