| Seq | Timestamp | Type | Name | Duration |
|---|---|---|---|---|
| 0/0 | 2026/08/17 12:57 | flow | patch-triage |
2mArgs:null Results: {
"EnableConfigs": [
"PAGE_OWNER",
"DEBUG_KMEMLEAK",
"KUNIT",
"STACKDEPOT_KUNIT_TEST"
],
"FocusSymbols": [
"stack_depot_fetch_into",
"__stack_depot_get_stack_record",
"stack_depot_snprint",
"stack_depot_print"
],
"KMSANReasoning": "The patch series introduces a new trie-based storage backend for `stackdepot` to compress and store stack traces more efficiently. It involves custom memory sub-allocation (`trie_pool_alloc`) from page-sized pools. \n\nWhile the memory returned by `trie_pool_alloc` is reused without zeroing, the code carefully and fully initializes all relevant fields of the allocated structures (`trie_node_init`, `trie_children_init`, `trie_children_insert`) before they are published via RCU. Any padding bytes within the structures (e.g., in `struct stack_depot_trie_node`) are never read or exposed.\n\nThe patch also changes the fetch API to `stack_depot_fetch_into`, requiring callers to provide their own buffers. This function explicitly unpoisons the written memory using `kmsan_unpoison_memory`. This is safe and correct because the trie traversal logic guarantees writing exactly the calculated number of frames, even in the presence of concurrent reparenting or splitting.\n\nThere are no structures copied to user space, no parsing of untrusted input, and no paths where uninitialized memory could affect control flow. Any bugs in the trie logic or memory management would primarily manifest as logic errors (wrong stack traces), out-of-bounds accesses, or use-after-free bugs. These are adequately covered by standard KASAN, LOCKDEP, and the comprehensive KUnit tests included in the patch. Therefore, a dedicated KMSAN fuzzing session is not justified.",
"NeedsKMSAN": false,
"Reasoning": "The patch introduces a new trie-based storage backend for the stack depot, a core kernel library used by memory debugging and tracking tools (e.g., KASAN, KMSAN, page_owner, kmemleak). It adds a new API `stack_depot_fetch_into` and modifies existing APIs to support the new backend and a new `STACK_DEPOT_FLAG_COUNTABLE` flag. These changes significantly alter the internal logic of stack trace storage and retrieval, making it worth fuzzing to ensure memory safety and correctness.",
"WorthFuzzing": true
} |
| 1/1 | 2026/08/17 12:57 | action | read-patch-diff |
0mArgs:null Results: {
"PatchDiff": "commit 1d27bb5b64bebf701b469bdea8a91ce99a84492b\nAuthor: syz-cluster \u003ctriage@syzkaller.com\u003e\nDate: Mon Aug 17 12:57:47 2026 +0000\n\n syz-cluster: applied patch under review\n\ndiff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt\nindex 1af62cd16c9de..ebb7b7e1867f6 100644\n--- a/Documentation/admin-guide/kernel-parameters.txt\n+++ b/Documentation/admin-guide/kernel-parameters.txt\n@@ -7387,6 +7387,13 @@ Kernel parameters\n \t\t\tstack traces. Pools are allocated on-demand up to this\n \t\t\tlimit. Default value is 8191 pools.\n \n+\tstackdepot.trie_enabled= [KNL]\n+\t\t\tFormat: \u003cbool\u003e\n+\t\t\tEnable trie storage for persistent, non-refcounted\n+\t\t\tstack depot records at boot. Disabled by default.\n+\t\t\tstack_depot_max_pools must leave unused pool-index\n+\t\t\tvalues for trie handles.\n+\n \tstacktrace\t[FTRACE]\n \t\t\tEnable the stack tracer on boot up.\n \ndiff --git a/arch/arm64/include/asm/stackdepot.h b/arch/arm64/include/asm/stackdepot.h\nnew file mode 100644\nindex 0000000000000..df8959d593366\n--- /dev/null\n+++ b/arch/arm64/include/asm/stackdepot.h\n@@ -0,0 +1,42 @@\n+/* SPDX-License-Identifier: GPL-2.0 */\n+#ifndef __ASM_STACKDEPOT_H\n+#define __ASM_STACKDEPOT_H\n+\n+#include \u003clinux/types.h\u003e\n+#include \u003casm/sections.h\u003e\n+\n+/*\n+ * Modules are allocated inside a 2 GB relocation window containing the\n+ * kernel image. Store a signed 32-bit offset from _text so compression is\n+ * independent of 4 GB high-bit boundaries crossed by that window.\n+ */\n+static inline unsigned long arch_stack_depot_frame_from_payload(u32 payload)\n+{\n+\tlong offset;\n+\n+\toffset = (s32)payload;\n+\tif (offset \u003c 0)\n+\t\treturn (unsigned long)_text - (unsigned long)(-offset);\n+\treturn (unsigned long)_text + (unsigned long)offset;\n+}\n+\n+static inline bool\n+arch_stack_depot_frame_try_compress(unsigned long frame, u32 *payload)\n+{\n+\tu32 candidate;\n+\n+\tcandidate = (u32)(frame - (unsigned long)_text);\n+\tif (arch_stack_depot_frame_from_payload(candidate) != frame)\n+\t\treturn false;\n+\n+\t*payload = candidate;\n+\treturn true;\n+}\n+\n+static inline void\n+arch_stack_depot_frame_decompress(u32 payload, unsigned long *frame)\n+{\n+\t*frame = arch_stack_depot_frame_from_payload(payload);\n+}\n+\n+#endif /* __ASM_STACKDEPOT_H */\ndiff --git a/arch/um/include/asm/Kbuild b/arch/um/include/asm/Kbuild\nindex 8fdc0bd9ab6fb..14778d2457d79 100644\n--- a/arch/um/include/asm/Kbuild\n+++ b/arch/um/include/asm/Kbuild\n@@ -21,6 +21,7 @@ generic-y += preempt.h\n generic-y += ring_buffer.h\n generic-y += runtime-const.h\n generic-y += softirq_stack.h\n+generic-y += stackdepot.h\n generic-y += switch_to.h\n generic-y += topology.h\n generic-y += trace_clock.h\ndiff --git a/arch/x86/include/asm/stackdepot.h b/arch/x86/include/asm/stackdepot.h\nnew file mode 100644\nindex 0000000000000..9a8d04fa8c1c8\n--- /dev/null\n+++ b/arch/x86/include/asm/stackdepot.h\n@@ -0,0 +1,37 @@\n+/* SPDX-License-Identifier: GPL-2.0 */\n+#ifndef _ASM_X86_STACKDEPOT_H\n+#define _ASM_X86_STACKDEPOT_H\n+\n+#include \u003clinux/types.h\u003e\n+\n+#ifdef CONFIG_X86_64\n+/*\n+ * Compress canonical kernel text/module addresses whose upper 32 bits are all\n+ * ones. Other kernel virtual addresses stay raw, so decompression reconstructs\n+ * the original frame by restoring this prefix.\n+ */\n+#define STACK_DEPOT_X86_64_FRAME_PREFIX\t0xffffffff00000000UL\n+#define STACK_DEPOT_X86_64_FRAME_LOW_MASK\t0x00000000ffffffffUL\n+\n+static inline bool\n+arch_stack_depot_frame_try_compress(unsigned long frame, u32 *low)\n+{\n+\tif ((frame \u0026 ~STACK_DEPOT_X86_64_FRAME_LOW_MASK) !=\n+\t STACK_DEPOT_X86_64_FRAME_PREFIX)\n+\t\treturn false;\n+\n+\t*low = (u32)frame;\n+\treturn true;\n+}\n+\n+static inline void\n+arch_stack_depot_frame_decompress(u32 low, unsigned long *frame)\n+{\n+\t*frame = STACK_DEPOT_X86_64_FRAME_PREFIX | low;\n+}\n+\n+#else\n+#include \u003casm-generic/stackdepot.h\u003e\n+#endif /* CONFIG_X86_64 */\n+\n+#endif /* _ASM_X86_STACKDEPOT_H */\ndiff --git a/drivers/gpu/drm/drm_modeset_lock.c b/drivers/gpu/drm/drm_modeset_lock.c\nindex 2c806b0146d67..a2ddb02b2aea5 100644\n--- a/drivers/gpu/drm/drm_modeset_lock.c\n+++ b/drivers/gpu/drm/drm_modeset_lock.c\n@@ -94,16 +94,13 @@ static noinline depot_stack_handle_t __drm_stack_depot_save(void)\n static void __drm_stack_depot_print(depot_stack_handle_t stack_depot)\n {\n \tstruct drm_printer p = drm_dbg_printer(NULL, DRM_UT_KMS, \"drm_modeset_lock\");\n-\tunsigned long *entries;\n-\tunsigned int nr_entries;\n \tchar *buf;\n \n \tbuf = kmalloc(PAGE_SIZE, GFP_NOWAIT | __GFP_NOWARN);\n \tif (!buf)\n \t\treturn;\n \n-\tnr_entries = stack_depot_fetch(stack_depot, \u0026entries);\n-\tstack_trace_snprint(buf, PAGE_SIZE, entries, nr_entries, 2);\n+\tstack_depot_snprint(stack_depot, buf, PAGE_SIZE, 2);\n \n \tdrm_printf(\u0026p, \"attempting to lock a contended lock without backoff:\\n%s\", buf);\n \ndiff --git a/include/asm-generic/Kbuild b/include/asm-generic/Kbuild\nindex 15df9dcb42a5b..ac178162fa114 100644\n--- a/include/asm-generic/Kbuild\n+++ b/include/asm-generic/Kbuild\n@@ -55,6 +55,7 @@ mandatory-y += serial.h\n mandatory-y += shmparam.h\n mandatory-y += simd.h\n mandatory-y += softirq_stack.h\n+mandatory-y += stackdepot.h\n mandatory-y += switch_to.h\n mandatory-y += timex.h\n mandatory-y += tlbflush.h\ndiff --git a/include/asm-generic/stackdepot.h b/include/asm-generic/stackdepot.h\nnew file mode 100644\nindex 0000000000000..846975767bdd4\n--- /dev/null\n+++ b/include/asm-generic/stackdepot.h\n@@ -0,0 +1,19 @@\n+/* SPDX-License-Identifier: GPL-2.0 */\n+#ifndef __ASM_GENERIC_STACKDEPOT_H\n+#define __ASM_GENERIC_STACKDEPOT_H\n+\n+#include \u003clinux/types.h\u003e\n+\n+static inline bool\n+arch_stack_depot_frame_try_compress(unsigned long frame, u32 *low)\n+{\n+\treturn false;\n+}\n+\n+static inline void\n+arch_stack_depot_frame_decompress(u32 low, unsigned long *frame)\n+{\n+\t/* Generic code never compresses frames, so this hook is unreachable. */\n+}\n+\n+#endif /* __ASM_GENERIC_STACKDEPOT_H */\ndiff --git a/include/linux/stackdepot.h b/include/linux/stackdepot.h\nindex 2cc21ffcdaf9e..788737eb0c4a2 100644\n--- a/include/linux/stackdepot.h\n+++ b/include/linux/stackdepot.h\n@@ -53,7 +53,8 @@ union handle_parts {\n struct stack_record {\n \tstruct list_head hash_list;\t/* Links in the hash table */\n \tu32 hash;\t\t\t/* Hash in hash table */\n-\tu32 size;\t\t\t/* Number of stored frames */\n+\tu16 size;\t\t\t/* Number of stored frames */\n+\tu16 flags;\n \tunion handle_parts handle;\t/* Constant after initialization */\n \trefcount_t count;\n \tunion {\n@@ -84,8 +85,9 @@ typedef u32 depot_flags_t;\n */\n #define STACK_DEPOT_FLAG_CAN_ALLOC\t((depot_flags_t)0x0001)\n #define STACK_DEPOT_FLAG_GET\t\t((depot_flags_t)0x0002)\n+#define STACK_DEPOT_FLAG_COUNTABLE\t((depot_flags_t)0x0004)\n \n-#define STACK_DEPOT_FLAGS_NUM\t2\n+#define STACK_DEPOT_FLAGS_NUM\t3\n #define STACK_DEPOT_FLAGS_MASK\t((depot_flags_t)((1 \u003c\u003c STACK_DEPOT_FLAGS_NUM) - 1))\n \n /*\n@@ -144,6 +146,15 @@ static inline int stack_depot_early_init(void)\t{ return 0; }\n * Users of this flag must also call stack_depot_put() when keeping the stack\n * trace is no longer required to avoid overflowing the refcount.\n *\n+ * If STACK_DEPOT_FLAG_COUNTABLE is set in @depot_flags, stack depot stores the\n+ * stack in hash-backed storage for callers that need direct stack_record count\n+ * access. This flag does not imply %STACK_DEPOT_FLAG_CAN_ALLOC and is mutually\n+ * exclusive with %STACK_DEPOT_FLAG_GET.\n+ *\n+ * When trie storage is enabled, persistent non-refcounted saves use trie\n+ * storage. Constrained callers only look up existing stacks; they do not insert\n+ * a missing stack. Trie failures do not fall back to hash storage.\n+ *\n * If the provided stack trace comes from the interrupt context, only the part\n * up to the interrupt entry is saved.\n *\n@@ -152,7 +163,7 @@ static inline int stack_depot_early_init(void)\t{ return 0; }\n * this is the case for contexts where neither %GFP_ATOMIC nor\n * %GFP_NOWAIT can be used (NMI, raw_spin_lock).\n *\n- * Return: Handle of the stack struct stored in depot, 0 on failure\n+ * Return: Handle of the stack trace stored in depot, 0 on failure\n */\n depot_stack_handle_t stack_depot_save_flags(unsigned long *entries,\n \t\t\t\t\t unsigned int nr_entries,\n@@ -169,6 +180,10 @@ depot_stack_handle_t stack_depot_save_flags(unsigned long *entries,\n * Does not increment the refcount on the saved stack trace; see\n * stack_depot_save_flags() for more details.\n *\n+ * When trie storage is enabled, this can return trie-backed handles. Use\n+ * stack_depot_fetch_into(), stack_depot_print(), or stack_depot_snprint() for\n+ * backend-independent access to the stack contents.\n+ *\n * Context: Contexts where allocations via alloc_pages() are allowed;\n * see stack_depot_save_flags() for more details.\n *\n@@ -178,11 +193,12 @@ depot_stack_handle_t stack_depot_save(unsigned long *entries,\n \t\t\t\t unsigned int nr_entries, gfp_t alloc_flags);\n \n /**\n- * __stack_depot_get_stack_record - Get a pointer to a stack_record struct\n+ * __stack_depot_get_stack_record - Get a hash-backed stack record\n *\n * @handle: Stack depot handle\n *\n- * This function is only for internal purposes.\n+ * This function is only for internal purposes. @handle must have been saved\n+ * with %STACK_DEPOT_FLAG_COUNTABLE.\n *\n * Return: Returns a pointer to a stack_record struct\n */\n@@ -191,14 +207,55 @@ struct stack_record *__stack_depot_get_stack_record(depot_stack_handle_t handle)\n /**\n * stack_depot_fetch - Fetch a stack trace from stack depot\n *\n- * @handle:\tStack depot handle returned from stack_depot_save()\n+ * @handle:\tHash-backed stack depot handle\n * @entries:\tPointer to store the address of the stack trace\n *\n+ * This helper returns a pointer to stackdepot-owned contiguous storage for\n+ * legacy hash-backed handles. Callers that need backend-independent access to\n+ * stack contents should use stack_depot_fetch_into(), stack_depot_print(), or\n+ * stack_depot_snprint(). Passing a trie-backed handle is invalid and may WARN.\n+ *\n * Return: Number of frames for the fetched stack\n */\n unsigned int stack_depot_fetch(depot_stack_handle_t handle,\n \t\t\t unsigned long **entries);\n \n+/**\n+ * stack_depot_fetch_into - Fetch a stack trace into caller-owned storage\n+ *\n+ * @handle:\tStack depot handle\n+ * @entries:\tCaller-owned buffer to copy the stack trace into\n+ * @max_entries:\tNumber of frames that fit in @entries\n+ *\n+ * Copies the stored frames into caller-owned @entries. If fewer frames are\n+ * stored than @max_entries, only the stored frames are written and their count\n+ * is returned. If more frames are stored than @max_entries, the copy is skipped\n+ * entirely and 0 is returned.\n+ *\n+ * Passing a NULL @entries buffer or zero @max_entries for a valid @handle is\n+ * invalid. Callers must provide storage for @max_entries frames.\n+ *\n+ * Callers should size @entries to match the save-side stack depth cap (for\n+ * example, %CONFIG_STACKDEPOT_MAX_FRAMES or the local stack_trace_save() limit)\n+ * when losing diagnostics on an undersized buffer would be surprising.\n+ *\n+ * A non-zero invalid @handle, including a post-put handle, may WARN. Its return\n+ * value and copied contents are undefined because the record may have been\n+ * reused for another stack.\n+ *\n+ * Callers must ensure @handle remains valid for the duration of this call.\n+ * Persistent handles saved without %STACK_DEPOT_FLAG_GET require no extra\n+ * reference; handles saved with %STACK_DEPOT_FLAG_GET require a held reference.\n+ * Callers must not call stack_depot_put() on persistent handles.\n+ * Racing this helper with stack_depot_put() on the same handle is invalid.\n+ *\n+ * Return: Number of frames copied, 0 if @handle is 0, stack depot is disabled,\n+ * or @max_entries is less than the number of stored frames.\n+ */\n+unsigned int stack_depot_fetch_into(depot_stack_handle_t handle,\n+\t\t\t\t unsigned long *entries,\n+\t\t\t\t unsigned int max_entries);\n+\n /**\n * stack_depot_print - Print a stack trace from stack depot\n *\n@@ -224,10 +281,14 @@ int stack_depot_snprint(depot_stack_handle_t handle, char *buf, size_t size,\n *\n * @handle:\tStack depot handle returned from stack_depot_save()\n *\n- * The stack trace is evicted from stack depot once all references to it have\n- * been dropped (once the number of stack_depot_evict() calls matches the\n- * number of stack_depot_save_flags() calls with STACK_DEPOT_FLAG_GET set for\n- * this stack trace).\n+ * Drop a reference acquired by stack_depot_save_flags() with\n+ * %STACK_DEPOT_FLAG_GET. Calling this for a handle saved without\n+ * %STACK_DEPOT_FLAG_GET is invalid; persistent handles, including trie-backed\n+ * handles, are owned by stack depot for the lifetime of the system.\n+ *\n+ * The stack trace is evicted once the number of stack_depot_put() calls matches\n+ * the number of successful stack_depot_save_flags() calls with\n+ * %STACK_DEPOT_FLAG_GET for this stack trace.\n */\n void stack_depot_put(depot_stack_handle_t handle);\n \ndiff --git a/lib/Kconfig.debug b/lib/Kconfig.debug\nindex 00921b1676e87..af238949fb7a0 100644\n--- a/lib/Kconfig.debug\n+++ b/lib/Kconfig.debug\n@@ -2771,6 +2771,23 @@ config RESOURCE_KUNIT_TEST\n \n \t If unsure, say N.\n \n+config STACKDEPOT_KUNIT_TEST\n+\tbool \"KUnit test for stack depot\" if !KUNIT_ALL_TESTS\n+\tdepends on KUNIT=y \u0026\u0026 STACKDEPOT\n+\tdepends on STACKDEPOT_MAX_FRAMES \u003e= 3\n+\tdefault KUNIT_ALL_TESTS\n+\thelp\n+\t Enable this option to test stack depot API behavior at boot.\n+\t This test is built in because it exercises internal, non-exported\n+\t stack depot helpers, so KUNIT must also be built in.\n+\n+\t KUnit tests run during boot and output the results to the debug log\n+\t in TAP format (https://testanything.org/). Only useful for kernel\n+\t developers running the KUnit test harness, and not intended for\n+\t inclusion into a production build.\n+\n+\t If unsure, say N.\n+\n config SYSCTL_KUNIT_TEST\n \ttristate \"KUnit test for sysctl\" if !KUNIT_ALL_TESTS\n \tdepends on KUNIT\ndiff --git a/lib/stackdepot.c b/lib/stackdepot.c\nindex dd2717ff94bff..1a002063a9487 100644\n--- a/lib/stackdepot.c\n+++ b/lib/stackdepot.c\n@@ -2,9 +2,11 @@\n /*\n * Stack depot - a stack trace storage that avoids duplication.\n *\n- * Internally, stack depot maintains a hash table of unique stacktraces. The\n- * stack traces themselves are stored contiguously one after another in a set\n- * of separate page allocations.\n+ * Internally, stack depot has two storage backends. Refcounted entries and\n+ * callers that request STACK_DEPOT_FLAG_COUNTABLE use the legacy hash table with\n+ * contiguous stack records in stack pools. Persistent non-refcounted entries\n+ * can use trie storage when enabled; trie nodes share common frame prefixes and\n+ * are published through RCU children containers.\n *\n * Author: Alexander Potapenko \u003cglider@google.com\u003e\n * Copyright (C) 2016 Google, Inc.\n@@ -14,13 +16,19 @@\n \n #define pr_fmt(fmt) \"stackdepot: \" fmt\n \n+#include \u003clinux/bitmap.h\u003e\n+#include \u003clinux/build_bug.h\u003e\n #include \u003clinux/debugfs.h\u003e\n+#include \u003clinux/errno.h\u003e\n #include \u003clinux/gfp.h\u003e\n #include \u003clinux/jhash.h\u003e\n+#include \u003clinux/jump_label.h\u003e\n #include \u003clinux/kernel.h\u003e\n+#include \u003clinux/log2.h\u003e\n #include \u003clinux/kmsan.h\u003e\n #include \u003clinux/list.h\u003e\n #include \u003clinux/mm.h\u003e\n+#include \u003clinux/moduleparam.h\u003e\n #include \u003clinux/mutex.h\u003e\n #include \u003clinux/poison.h\u003e\n #include \u003clinux/printk.h\u003e\n@@ -36,9 +44,12 @@\n #include \u003clinux/memblock.h\u003e\n #include \u003clinux/kasan-enabled.h\u003e\n \n+#include \u003casm/stackdepot.h\u003e\n+\n /*\n * The pool_index is offset by 1 so the first record does not have a 0 handle.\n */\n+/* Parsed before mm_core_init(); trie handle decoding assumes this is then fixed. */\n static unsigned int stack_max_pools __read_mostly =\n \tMIN((1LL \u003c\u003c DEPOT_POOL_INDEX_BITS) - 1, 8192);\n \n@@ -63,18 +74,18 @@ static unsigned int stack_hash_mask;\n \n /* The lock must be held when performing pool or freelist modifications. */\n static DEFINE_RAW_SPINLOCK(pool_lock);\n-/* Array of memory regions that store stack records. */\n+/* Array of memory regions used by both stack depot backends. */\n static void **stack_pools __pt_guarded_by(\u0026pool_lock);\n /* Newly allocated pool that is not yet added to stack_pools. */\n static void *new_pool;\n /* Number of pools in stack_pools. */\n static int pools_num;\n-/* Offset to the unused space in the currently used pool. */\n+/* Offset to unused hash storage in the current pool. */\n static size_t pool_offset __guarded_by(\u0026pool_lock) = DEPOT_POOL_SIZE;\n /* Freelist of stack records within stack_pools. */\n static __guarded_by(\u0026pool_lock) LIST_HEAD(free_stacks);\n \n-/* Statistics counters for debugfs. */\n+/* Hash-backend statistics counters for debugfs. */\n enum depot_counter_id {\n \tDEPOT_COUNTER_REFD_ALLOCS,\n \tDEPOT_COUNTER_REFD_FREES,\n@@ -90,11 +101,631 @@ static const char *const counter_names[] = {\n \t[DEPOT_COUNTER_REFD_FREES]\t= \"refcounted_frees\",\n \t[DEPOT_COUNTER_REFD_INUSE]\t= \"refcounted_in_use\",\n \t[DEPOT_COUNTER_FREELIST_SIZE]\t= \"freelist_size\",\n-\t[DEPOT_COUNTER_PERSIST_COUNT]\t= \"persistent_count\",\n-\t[DEPOT_COUNTER_PERSIST_BYTES]\t= \"persistent_bytes\",\n+\t[DEPOT_COUNTER_PERSIST_COUNT]\t= \"hash_persistent_count\",\n+\t[DEPOT_COUNTER_PERSIST_BYTES]\t= \"hash_persistent_bytes\",\n };\n static_assert(ARRAY_SIZE(counter_names) == DEPOT_COUNTER_COUNT);\n \n+enum stack_depot_frame_mode {\n+\tSTACK_DEPOT_FRAME_RAW,\n+\tSTACK_DEPOT_FRAME_COMPRESSED,\n+};\n+\n+/*\n+ * A trie node stores one run of frames that all use the same payload format.\n+ * Architectures may compress some frames to 32-bit payloads; mixed raw and\n+ * compressed input is split across multiple trie nodes so each node has one\n+ * decoding mode.\n+ */\n+struct stack_depot_frame_run {\n+\tu16 nr_entries;\n+\tu8 mode;\n+};\n+\n+static_assert(CONFIG_STACKDEPOT_MAX_FRAMES \u003c= U16_MAX);\n+\n+struct stack_depot_trie_children;\n+\n+struct stack_depot_trie_node {\n+\t/* Parent links let fetch rebuild a full stack from a node to the root. */\n+\tconst struct stack_depot_trie_node __rcu *parent;\n+\t/* Children are RCU-published containers. */\n+\tconst struct stack_depot_trie_children __rcu *children;\n+\t/* Non-zero when a stored stack ends at this node. */\n+\tu32 stack_id;\n+\tstruct stack_depot_frame_run run;\n+\tunsigned char data[];\n+};\n+\n+/*\n+ * Child nodes are sorted by first frame and searched by insertion position.\n+ * Existing child pointers are immutable. Writers may publish into unused tail\n+ * capacity; other updates publish a replacement container.\n+ */\n+struct stack_depot_trie_children {\n+\tunsigned int nr_children;\n+\tunsigned int capacity;\n+\tconst struct stack_depot_trie_node __rcu *nodes[];\n+};\n+\n+/* Retired children carry an optional node through their RCU grace period. */\n+struct stack_depot_trie_retired_children {\n+\tstruct list_head list;\n+\tunsigned long rcu_state;\n+\tconst struct stack_depot_trie_node *pending_node;\n+\tunsigned char data[];\n+};\n+\n+static_assert(IS_ALIGNED(offsetof(struct stack_depot_trie_retired_children, data),\n+\t\t\t 1UL \u003c\u003c DEPOT_STACK_ALIGN));\n+\n+#define STACK_DEPOT_TRIE_SLOT_SIZE BIT(DEPOT_STACK_ALIGN)\n+#define STACK_DEPOT_TRIE_POOL_SLOTS \\\n+\t(DEPOT_POOL_SIZE / STACK_DEPOT_TRIE_SLOT_SIZE)\n+\n+struct stack_depot_trie_pool {\n+\tstruct list_head list;\n+\tunsigned int free_slots;\n+\tDECLARE_BITMAP(used, STACK_DEPOT_TRIE_POOL_SLOTS);\n+};\n+\n+#define STACK_DEPOT_TRIE_POOL_FIRST_SLOT \\\n+\tDIV_ROUND_UP(sizeof(struct stack_depot_trie_pool), \\\n+\t\t STACK_DEPOT_TRIE_SLOT_SIZE)\n+#define STACK_DEPOT_TRIE_POOL_USABLE_SIZE \\\n+\t((STACK_DEPOT_TRIE_POOL_SLOTS - STACK_DEPOT_TRIE_POOL_FIRST_SLOT) * \\\n+\t STACK_DEPOT_TRIE_SLOT_SIZE)\n+\n+static_assert(STACK_DEPOT_TRIE_POOL_FIRST_SLOT \u003c STACK_DEPOT_TRIE_POOL_SLOTS);\n+\n+static DEFINE_STATIC_KEY_FALSE(stack_depot_trie_enabled);\n+static const struct stack_depot_trie_children __rcu *stack_depot_trie_root;\n+static DEFINE_RAW_SPINLOCK(stack_depot_trie_writer_lock);\n+static bool stack_depot_trie_requested;\n+\n+module_param_named(trie_enabled, stack_depot_trie_requested, bool, 0);\n+MODULE_PARM_DESC(trie_enabled, \"Enable stack depot trie storage at boot\");\n+\n+#define DEPOT_POOL_INDEX_MASK ((1U \u003c\u003c DEPOT_POOL_INDEX_BITS) - 1)\n+#define DEPOT_OFFSET_MASK ((1U \u003c\u003c DEPOT_OFFSET_BITS) - 1)\n+\n+/* Retired fixed-size slots remain reserved until their RCU grace period ends. */\n+static LIST_HEAD(stack_depot_trie_pools);\n+static LIST_HEAD(pending_trie_children);\n+\n+/*\n+ * stack_max_pools is the split point between hash and trie handle encodings.\n+ * A handle with pool_index_plus_1 in 1..stack_max_pools names a hash-backed\n+ * stack pool. Larger pool-index values cannot refer to hash pools, so trie\n+ * storage uses that handle space to encode a dense stack ID. The side table\n+ * maps each stack ID to its trie node.\n+ */\n+static inline u32 trie_max_stack_id(void)\n+{\n+\treturn (DEPOT_POOL_INDEX_MASK - stack_max_pools) \u003c\u003c\n+\t\tDEPOT_OFFSET_BITS;\n+}\n+\n+static depot_stack_handle_t trie_handle(u32 stack_id)\n+{\n+\tunion handle_parts parts = {};\n+\tu64 pool_index_plus_1;\n+\tu32 pool_delta;\n+\tu32 index;\n+\n+\tindex = stack_id - 1;\n+\tpool_delta = index \u003e\u003e DEPOT_OFFSET_BITS;\n+\tpool_index_plus_1 = (u64)stack_max_pools + 1 + pool_delta;\n+\n+\tparts.pool_index_plus_1 = pool_index_plus_1;\n+\tparts.offset = index \u0026 DEPOT_OFFSET_MASK;\n+\treturn parts.handle;\n+}\n+\n+static inline bool stack_depot_handle_is_trie(depot_stack_handle_t handle)\n+{\n+\tunion handle_parts parts = { .handle = handle };\n+\n+\treturn parts.pool_index_plus_1 \u003e stack_max_pools;\n+}\n+\n+static u32 trie_stack_id(depot_stack_handle_t handle)\n+{\n+\tunion handle_parts parts = { .handle = handle };\n+\tu32 pool_delta;\n+\n+\tpool_delta = parts.pool_index_plus_1 - stack_max_pools - 1;\n+\treturn (pool_delta \u003c\u003c DEPOT_OFFSET_BITS) + parts.offset + 1;\n+}\n+\n+/*\n+ * Trie handles encode a dense stack ID. The side table maps that ID to a node\n+ * pointer for lockless fetch and print paths, which can run from diagnostic\n+ * contexts where taking a lock would be unsafe. Initialization installs the\n+ * root; early initialization also installs the first directory and chunk.\n+ * Additional directories and chunks are published lazily as stack IDs grow.\n+ */\n+#define STACK_DEPOT_TRIE_SIDE_TABLE_CHUNK_SIZE \\\n+\t(PAGE_SIZE / sizeof(struct stack_depot_trie_node *))\n+#define STACK_DEPOT_TRIE_SIDE_TABLE_DIR_SIZE \\\n+\t(PAGE_SIZE / sizeof(struct stack_depot_trie_node **))\n+\n+struct stack_depot_trie_side_dir {\n+\t/* Both the chunk pointer and each node pointer in it are RCU-published. */\n+\tconst struct stack_depot_trie_node __rcu * __rcu *\n+\t\tchunks[STACK_DEPOT_TRIE_SIDE_TABLE_DIR_SIZE];\n+};\n+\n+struct stack_depot_trie_side_root {\n+\tunsigned int dir_capacity;\n+\tstruct stack_depot_trie_side_dir __rcu *dirs[];\n+};\n+\n+struct stack_depot_trie_side_prealloc {\n+\t/* Preallocated side-table directory page for sparse growth. */\n+\tstruct stack_depot_trie_side_dir *dir;\n+\t/* Preallocated side-table pointer chunk for sparse growth. */\n+\tconst struct stack_depot_trie_node __rcu **chunk;\n+};\n+\n+static struct stack_depot_trie_side_root *trie_side_table_root;\n+static DEFINE_RAW_SPINLOCK(trie_side_table_cache_lock);\n+/* Zeroed unpublished pages; get/put transfer ownership under the cache lock. */\n+static struct stack_depot_trie_side_prealloc trie_side_table_cache;\n+static u32 trie_side_table_last_stack_id;\n+\n+/* Lock order: writer_lock -\u003e pool_lock. The cache lock is never nested. */\n+\n+static inline size_t stack_depot_frame_run_entry_bytes(enum stack_depot_frame_mode mode)\n+{\n+\tif (mode == STACK_DEPOT_FRAME_COMPRESSED)\n+\t\treturn sizeof(u32);\n+\treturn sizeof(unsigned long);\n+}\n+\n+static inline size_t stack_depot_frame_run_bytes(const struct stack_depot_frame_run *run)\n+{\n+\treturn run-\u003enr_entries * stack_depot_frame_run_entry_bytes(run-\u003emode);\n+}\n+\n+static inline size_t trie_node_bytes(const struct stack_depot_frame_run *run)\n+{\n+\treturn ALIGN(offsetof(struct stack_depot_trie_node, data) +\n+\t\t stack_depot_frame_run_bytes(run), sizeof(unsigned long));\n+}\n+\n+static size_t trie_children_alloc_size(unsigned int capacity)\n+{\n+\tsize_t size;\n+\n+\tsize = struct_size_t(struct stack_depot_trie_children, nodes,\n+\t\t\t capacity);\n+\treturn offsetof(struct stack_depot_trie_retired_children, data) +\n+\t\tALIGN(size, sizeof(unsigned long));\n+}\n+\n+static inline unsigned int trie_side_table_root_index(u32 id)\n+{\n+\treturn ((id - 1) / STACK_DEPOT_TRIE_SIDE_TABLE_CHUNK_SIZE) /\n+\t\tSTACK_DEPOT_TRIE_SIDE_TABLE_DIR_SIZE;\n+}\n+\n+static inline unsigned int trie_side_table_dir_index(u32 id)\n+{\n+\treturn ((id - 1) / STACK_DEPOT_TRIE_SIDE_TABLE_CHUNK_SIZE) %\n+\t\tSTACK_DEPOT_TRIE_SIDE_TABLE_DIR_SIZE;\n+}\n+\n+static inline unsigned int trie_side_table_slot_index(u32 id)\n+{\n+\treturn (id - 1) % STACK_DEPOT_TRIE_SIDE_TABLE_CHUNK_SIZE;\n+}\n+\n+static struct stack_depot_trie_side_dir *trie_side_table_load_dir(unsigned int root)\n+{\n+\tstruct stack_depot_trie_side_root *root_vec;\n+\n+\troot_vec = trie_side_table_root;\n+\tif (!root_vec || root \u003e= root_vec-\u003edir_capacity)\n+\t\treturn NULL;\n+\t/* Pairs with side-table directory rcu_assign_pointer(). */\n+\treturn rcu_dereference_check(root_vec-\u003edirs[root],\n+\t\t\t\t lockdep_is_held(\u0026stack_depot_trie_writer_lock) ||\n+\t\t\t\t rcu_read_lock_sched_held());\n+}\n+\n+static inline const struct stack_depot_trie_node __rcu **\n+trie_side_table_dir_load_chunk(struct stack_depot_trie_side_dir *dir,\n+\t\t\t unsigned int idx)\n+{\n+\t/* Pairs with the chunk rcu_assign_pointer() in stack ID preparation. */\n+\treturn rcu_dereference_check(dir-\u003echunks[idx],\n+\t\t\t\t lockdep_is_held(\u0026stack_depot_trie_writer_lock) ||\n+\t\t\t\t rcu_read_lock_sched_held());\n+}\n+\n+static u32\n+trie_side_table_prepare_stack_slot(struct stack_depot_trie_side_prealloc *prealloc)\n+{\n+\tconst struct stack_depot_trie_node __rcu **chunk;\n+\tstruct stack_depot_trie_side_dir *dir;\n+\tstruct stack_depot_trie_side_root *root_vec;\n+\tunsigned int root;\n+\tunsigned int idx;\n+\tu32 id;\n+\n+\tlockdep_assert_held(\u0026stack_depot_trie_writer_lock);\n+\n+\tid = trie_side_table_last_stack_id + 1;\n+\tif (id \u003e trie_max_stack_id())\n+\t\treturn 0;\n+\n+\troot_vec = trie_side_table_root;\n+\troot = trie_side_table_root_index(id);\n+\tdir = trie_side_table_load_dir(root);\n+\tif (!dir) {\n+\t\tdir = prealloc-\u003edir;\n+\t\tprealloc-\u003edir = NULL;\n+\t\t/* Publish the zeroed directory before readers can load it locklessly. */\n+\t\trcu_assign_pointer(root_vec-\u003edirs[root], dir);\n+\t}\n+\n+\tidx = trie_side_table_dir_index(id);\n+\tchunk = trie_side_table_dir_load_chunk(dir, idx);\n+\tif (!chunk) {\n+\t\tchunk = prealloc-\u003echunk;\n+\t\tprealloc-\u003echunk = NULL;\n+\t\trcu_assign_pointer(dir-\u003echunks[idx], chunk);\n+\t}\n+\n+\treturn id;\n+}\n+\n+static inline unsigned int trie_side_table_root_size_for_max_id(u32 max_stack_id)\n+{\n+\tunsigned int top_size;\n+\n+\ttop_size = DIV_ROUND_UP(max_stack_id,\n+\t\t\t\tSTACK_DEPOT_TRIE_SIDE_TABLE_CHUNK_SIZE);\n+\treturn DIV_ROUND_UP(top_size, STACK_DEPOT_TRIE_SIDE_TABLE_DIR_SIZE);\n+}\n+\n+static int __init stack_depot_trie_init_memblock(void)\n+{\n+\tstruct stack_depot_trie_side_root *root_vec;\n+\tstruct stack_depot_trie_side_dir *first_dir;\n+\tconst struct stack_depot_trie_node __rcu **first_chunk;\n+\tsize_t root_bytes;\n+\tu32 max_stack_id;\n+\tunsigned int root_size;\n+\n+\tmax_stack_id = trie_max_stack_id();\n+\tif (!max_stack_id)\n+\t\treturn -EINVAL;\n+\troot_size = trie_side_table_root_size_for_max_id(max_stack_id);\n+\troot_bytes = struct_size_t(struct stack_depot_trie_side_root, dirs,\n+\t\t\t\t root_size);\n+\n+\troot_vec = memblock_alloc(root_bytes, __alignof__(*root_vec));\n+\tif (!root_vec)\n+\t\treturn -ENOMEM;\n+\tfirst_dir = memblock_alloc(PAGE_SIZE, PAGE_SIZE);\n+\tif (!first_dir) {\n+\t\tmemblock_free(root_vec, root_bytes);\n+\t\treturn -ENOMEM;\n+\t}\n+\tfirst_chunk = memblock_alloc(PAGE_SIZE, PAGE_SIZE);\n+\tif (!first_chunk) {\n+\t\tmemblock_free(first_dir, PAGE_SIZE);\n+\t\tmemblock_free(root_vec, root_bytes);\n+\t\treturn -ENOMEM;\n+\t}\n+\n+\troot_vec-\u003edir_capacity = root_size;\n+\tRCU_INIT_POINTER(root_vec-\u003edirs[0], first_dir);\n+\tRCU_INIT_POINTER(first_dir-\u003echunks[0], first_chunk);\n+\ttrie_side_table_root = root_vec;\n+\tstatic_branch_enable(\u0026stack_depot_trie_enabled);\n+\treturn 0;\n+}\n+\n+static int stack_depot_trie_init(void)\n+{\n+\tstruct stack_depot_trie_side_root *root_vec;\n+\tunsigned int root_size;\n+\tu32 max_stack_id;\n+\n+\tmax_stack_id = trie_max_stack_id();\n+\tif (!max_stack_id)\n+\t\treturn -EINVAL;\n+\n+\troot_size = trie_side_table_root_size_for_max_id(max_stack_id);\n+\troot_vec = kvzalloc_flex(*root_vec, dirs, root_size);\n+\tif (!root_vec)\n+\t\treturn -ENOMEM;\n+\n+\troot_vec-\u003edir_capacity = root_size;\n+\ttrie_side_table_root = root_vec;\n+\tstatic_branch_enable(\u0026stack_depot_trie_enabled);\n+\treturn 0;\n+}\n+\n+static int trie_side_table_get_prealloc(gfp_t gfp_flags,\n+\t\t\t\t\tstruct stack_depot_trie_side_prealloc *prealloc)\n+{\n+\tunsigned long flags;\n+\n+\tgfp_flags = gfp_nested_mask(gfp_flags);\n+\traw_spin_lock_irqsave(\u0026trie_side_table_cache_lock, flags);\n+\tprealloc-\u003edir = trie_side_table_cache.dir;\n+\tprealloc-\u003echunk = trie_side_table_cache.chunk;\n+\ttrie_side_table_cache.dir = NULL;\n+\ttrie_side_table_cache.chunk = NULL;\n+\traw_spin_unlock_irqrestore(\u0026trie_side_table_cache_lock, flags);\n+\n+\tif (!prealloc-\u003edir) {\n+\t\tprealloc-\u003edir = (void *)get_zeroed_page(gfp_flags);\n+\t\tif (!prealloc-\u003edir)\n+\t\t\treturn -ENOMEM;\n+\t}\n+\tif (!prealloc-\u003echunk) {\n+\t\tprealloc-\u003echunk = (void *)get_zeroed_page(gfp_flags);\n+\t\tif (!prealloc-\u003echunk)\n+\t\t\treturn -ENOMEM;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static void trie_side_table_put_prealloc(struct stack_depot_trie_side_prealloc *prealloc)\n+{\n+\tunsigned long flags;\n+\n+\traw_spin_lock_irqsave(\u0026trie_side_table_cache_lock, flags);\n+\tif (!trie_side_table_cache.dir) {\n+\t\ttrie_side_table_cache.dir = prealloc-\u003edir;\n+\t\tprealloc-\u003edir = NULL;\n+\t}\n+\tif (!trie_side_table_cache.chunk) {\n+\t\ttrie_side_table_cache.chunk = prealloc-\u003echunk;\n+\t\tprealloc-\u003echunk = NULL;\n+\t}\n+\traw_spin_unlock_irqrestore(\u0026trie_side_table_cache_lock, flags);\n+\n+\tif (prealloc-\u003edir)\n+\t\tfree_page((unsigned long)prealloc-\u003edir);\n+\tif (prealloc-\u003echunk)\n+\t\tfree_page((unsigned long)prealloc-\u003echunk);\n+}\n+\n+static const struct stack_depot_trie_node *trie_side_table_lookup(u32 id)\n+{\n+\tconst struct stack_depot_trie_node __rcu **chunk;\n+\tstruct stack_depot_trie_side_dir *dir;\n+\tunsigned int root;\n+\n+\troot = trie_side_table_root_index(id);\n+\tdir = trie_side_table_load_dir(root);\n+\tif (!dir)\n+\t\treturn NULL;\n+\tchunk = trie_side_table_dir_load_chunk(dir, trie_side_table_dir_index(id));\n+\tif (!chunk)\n+\t\treturn NULL;\n+\n+\t/* Pairs with side-table node publication. */\n+\treturn rcu_dereference_check(chunk[trie_side_table_slot_index(id)],\n+\t\t\t\t lockdep_is_held(\u0026stack_depot_trie_writer_lock) ||\n+\t\t\t\t rcu_read_lock_sched_held());\n+}\n+\n+static inline struct stack_depot_trie_retired_children *\n+trie_retired_children(const void *ptr)\n+{\n+\treturn container_of(ptr, struct stack_depot_trie_retired_children, data);\n+}\n+\n+static bool depot_init_pool(void **prealloc);\n+\n+static unsigned int trie_pool_reserve_slots(struct stack_depot_trie_pool *pool,\n+\t\t\t\t\t unsigned int nr_slots)\n+{\n+\tunsigned int run = 0;\n+\tunsigned int i;\n+\tunsigned int slot;\n+\n+\tif (pool-\u003efree_slots \u003c nr_slots)\n+\t\treturn STACK_DEPOT_TRIE_POOL_SLOTS;\n+\n+\t/* A free run can cross any previous allocation position. */\n+\tfor (slot = STACK_DEPOT_TRIE_POOL_FIRST_SLOT;\n+\t slot \u003c STACK_DEPOT_TRIE_POOL_SLOTS; slot++) {\n+\t\tif (pool-\u003eused[slot / BITS_PER_LONG] \u0026\n+\t\t BIT(slot % BITS_PER_LONG)) {\n+\t\t\trun = 0;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (++run != nr_slots)\n+\t\t\tcontinue;\n+\n+\t\tfor (i = slot + 1 - nr_slots; i \u003c= slot; i++)\n+\t\t\tpool-\u003eused[i / BITS_PER_LONG] |= BIT(i % BITS_PER_LONG);\n+\t\tpool-\u003efree_slots -= nr_slots;\n+\t\treturn slot + 1 - nr_slots;\n+\t}\n+\n+\treturn STACK_DEPOT_TRIE_POOL_SLOTS;\n+}\n+\n+/* Allocate at least @size bytes from one contiguous trie-pool slot run. */\n+static void *trie_pool_alloc(size_t size, void **prealloc)\n+{\n+\tstruct stack_depot_trie_pool *pool;\n+\tunsigned int nr_slots;\n+\tunsigned int slot;\n+\n+\tlockdep_assert_held(\u0026pool_lock);\n+\n+\tif (size \u003e STACK_DEPOT_TRIE_POOL_USABLE_SIZE)\n+\t\treturn NULL;\n+\tnr_slots = DIV_ROUND_UP(size, STACK_DEPOT_TRIE_SLOT_SIZE);\n+\tlist_for_each_entry_reverse(pool, \u0026stack_depot_trie_pools, list) {\n+\t\tslot = trie_pool_reserve_slots(pool, nr_slots);\n+\t\tif (slot != STACK_DEPOT_TRIE_POOL_SLOTS)\n+\t\t\treturn (char *)pool + slot * STACK_DEPOT_TRIE_SLOT_SIZE;\n+\t}\n+\n+\tif (!depot_init_pool(prealloc))\n+\t\treturn NULL;\n+\tpool = stack_pools[pools_num - 1];\n+\t/* Keep hash records out of this bitmap-owned pool. */\n+\tpool_offset = DEPOT_POOL_SIZE;\n+\tmemset(pool, 0, sizeof(*pool));\n+\tpool-\u003efree_slots = STACK_DEPOT_TRIE_POOL_SLOTS -\n+\t\t\t STACK_DEPOT_TRIE_POOL_FIRST_SLOT;\n+\tlist_add_tail(\u0026pool-\u003elist, \u0026stack_depot_trie_pools);\n+\n+\tslot = trie_pool_reserve_slots(pool, nr_slots);\n+\treturn (char *)pool + slot * STACK_DEPOT_TRIE_SLOT_SIZE;\n+}\n+\n+/* Release the slots for the byte count originally passed to allocation. */\n+static void trie_pool_release(const void *ptr, size_t size)\n+{\n+\tstruct stack_depot_trie_pool *pool;\n+\tunsigned long pfn;\n+\tunsigned int nr_slots;\n+\tunsigned int slot;\n+\tunsigned int i;\n+\n+\tlockdep_assert_held(\u0026pool_lock);\n+\n+\tpfn = page_to_pfn(virt_to_page(ptr));\n+\tpfn \u0026= ~(BIT(DEPOT_POOL_ORDER) - 1);\n+\tpool = page_address(pfn_to_page(pfn));\n+\tslot = ((unsigned long)ptr - (unsigned long)pool) \u003e\u003e DEPOT_STACK_ALIGN;\n+\tnr_slots = DIV_ROUND_UP(size, STACK_DEPOT_TRIE_SLOT_SIZE);\n+\tfor (i = slot; i \u003c slot + nr_slots; i++)\n+\t\tpool-\u003eused[i / BITS_PER_LONG] \u0026= ~BIT(i % BITS_PER_LONG);\n+\tpool-\u003efree_slots += nr_slots;\n+}\n+\n+static struct stack_depot_trie_children *\n+trie_pool_alloc_children(unsigned int capacity, void **prealloc)\n+{\n+\tstruct stack_depot_trie_retired_children *retired;\n+\tstruct stack_depot_trie_children *children;\n+\n+\t/* Capacity counts child-pointer entries; allocation includes RCU metadata. */\n+\tretired = trie_pool_alloc(trie_children_alloc_size(capacity), prealloc);\n+\tif (!retired)\n+\t\treturn NULL;\n+\n+\tchildren = (void *)retired-\u003edata;\n+\tchildren-\u003enr_children = 0;\n+\tchildren-\u003ecapacity = capacity;\n+\treturn children;\n+}\n+\n+static void\n+trie_pool_release_children(const struct stack_depot_trie_children *children)\n+{\n+\t/* Capacity is immutable and therefore recovers the allocation byte size. */\n+\ttrie_pool_release(trie_retired_children(children),\n+\t\t\t trie_children_alloc_size(children-\u003ecapacity));\n+}\n+\n+/*\n+ * Return RCU-ready objects before allocating. Pending children are FIFO, so\n+ * stop at the first incomplete grace period. A replaced node shares the same\n+ * retirement cookie and is released with its former children container.\n+ */\n+static void trie_drain_pending_children(void)\n+{\n+\tstruct stack_depot_trie_retired_children *retired;\n+\tstruct stack_depot_trie_retired_children *tmp;\n+\tstruct stack_depot_trie_children *children;\n+\n+\tlockdep_assert_held(\u0026pool_lock);\n+\n+\tlist_for_each_entry_safe(retired, tmp, \u0026pending_trie_children, list) {\n+\t\tif (!poll_state_synchronize_rcu(retired-\u003ercu_state))\n+\t\t\tbreak;\n+\t\tchildren = (void *)retired-\u003edata;\n+\t\tlist_del(\u0026retired-\u003elist);\n+\t\tif (retired-\u003epending_node)\n+\t\t\ttrie_pool_release(retired-\u003epending_node,\n+\t\t\t\t\t trie_node_bytes(\u0026retired-\u003epending_node-\u003erun));\n+\t\ttrie_pool_release_children(children);\n+\t}\n+}\n+\n+static void trie_retire_children(const struct stack_depot_trie_children *children)\n+{\n+\tstruct stack_depot_trie_retired_children *retired;\n+\n+\tlockdep_assert_held(\u0026pool_lock);\n+\n+\tretired = trie_retired_children(children);\n+\tretired-\u003epending_node = NULL;\n+\tretired-\u003ercu_state = get_state_synchronize_rcu();\n+\tlist_add_tail(\u0026retired-\u003elist, \u0026pending_trie_children);\n+}\n+\n+static void\n+trie_retire_children_with_node(const struct stack_depot_trie_children *children,\n+\t\t\t const struct stack_depot_trie_node *node)\n+{\n+\tstruct stack_depot_trie_retired_children *retired;\n+\n+\tlockdep_assert_held(\u0026stack_depot_trie_writer_lock);\n+\traw_spin_lock(\u0026pool_lock);\n+\ttrie_retire_children(children);\n+\tretired = trie_retired_children(children);\n+\tretired-\u003epending_node = node;\n+\traw_spin_unlock(\u0026pool_lock);\n+}\n+\n+static const struct stack_depot_trie_node *\n+stack_depot_trie_lookup(const unsigned long *entries, unsigned int nr_entries);\n+\n+static depot_stack_handle_t\n+trie_find_handle(const unsigned long *entries, unsigned int nr_entries)\n+{\n+\tdepot_stack_handle_t handle = 0;\n+\tconst struct stack_depot_trie_node *node;\n+\n+\trcu_read_lock_sched_notrace();\n+\tnode = stack_depot_trie_lookup(entries, nr_entries);\n+\tif (node)\n+\t\thandle = trie_handle(node-\u003estack_id);\n+\trcu_read_unlock_sched_notrace();\n+\n+\treturn handle;\n+}\n+\n+/*\n+ * Publish only after the node and its path are fully initialized and all\n+ * fallible allocation is complete. Publication commits the path, so it cannot\n+ * then be rolled back. Side-table mappings must precede trie topology\n+ * publication that makes new or remapped nodes reachable from lookup.\n+ * Published storage remains valid until RCU retirement; only descendant parent\n+ * links may change meanwhile.\n+ */\n+static void trie_side_table_publish(const struct stack_depot_trie_node *node)\n+{\n+\tconst struct stack_depot_trie_node __rcu **chunk;\n+\tstruct stack_depot_trie_side_dir *dir;\n+\tu32 stack_id = node-\u003estack_id;\n+\n+\tlockdep_assert_held(\u0026stack_depot_trie_writer_lock);\n+\n+\tdir = trie_side_table_load_dir(trie_side_table_root_index(stack_id));\n+\tchunk = trie_side_table_dir_load_chunk(dir,\n+\t\t\t\t\t trie_side_table_dir_index(stack_id));\n+\t/* Pairs with trie_side_table_lookup(). */\n+\trcu_assign_pointer(chunk[trie_side_table_slot_index(stack_id)], node);\n+}\n+\n static int __init disable_stack_depot(char *str)\n {\n \treturn kstrtobool(str, \u0026stack_depot_disabled);\n@@ -146,7 +777,7 @@ static void init_stack_table(unsigned long entries)\n \t\tINIT_LIST_HEAD(\u0026stack_table[i]);\n }\n \n-/* Allocates a hash table via memblock. Can only be used during early boot. */\n+/* Initializes hash and optional trie storage during early boot. */\n int __init stack_depot_early_init(void)\n {\n \tunsigned long entries = 0;\n@@ -220,11 +851,15 @@ int __init stack_depot_early_init(void)\n \t\tstack_depot_disabled = true;\n \t\treturn -ENOMEM;\n \t}\n+\tif (stack_depot_trie_requested \u0026\u0026 stack_depot_trie_init_memblock()) {\n+\t\tpr_warn(\"trie storage initialization failed, disabling trie storage\\n\");\n+\t\tstack_depot_trie_requested = false;\n+\t}\n \n \treturn 0;\n }\n \n-/* Allocates a hash table via kvcalloc. Can be used after boot. */\n+/* Initializes hash and optional trie storage after boot. */\n int stack_depot_init(void)\n {\n \tstatic DEFINE_MUTEX(stack_depot_init_mutex);\n@@ -278,6 +913,15 @@ int stack_depot_init(void)\n \t\tkvfree(stack_table);\n \t\tstack_depot_disabled = true;\n \t\tret = -ENOMEM;\n+\t\tgoto out_unlock;\n+\t}\n+\tif (stack_depot_trie_requested) {\n+\t\tret = stack_depot_trie_init();\n+\t\tif (ret) {\n+\t\t\tpr_warn(\"trie storage initialization failed, disabling trie storage\\n\");\n+\t\t\tstack_depot_trie_requested = false;\n+\t\t\tret = 0;\n+\t\t}\n \t}\n \n out_unlock:\n@@ -323,7 +967,7 @@ static bool depot_init_pool(void **prealloc)\n \t * NULL; do not reset to NULL if we have reached the maximum number of\n \t * pools.\n \t */\n-\tif (pools_num \u003c stack_max_pools)\n+\tif (pools_num + 1 \u003c stack_max_pools)\n \t\tWRITE_ONCE(new_pool, NULL);\n \telse\n \t\tWRITE_ONCE(new_pool, STACK_DEPOT_POISON);\n@@ -467,6 +1111,7 @@ depot_alloc_stack(unsigned long *entries, unsigned int nr_entries, u32 hash, dep\n \t/* Save the stack trace. */\n \tstack-\u003ehash = hash;\n \tstack-\u003esize = nr_entries;\n+\tstack-\u003eflags = flags \u0026 STACK_DEPOT_FLAG_COUNTABLE;\n \t/* stack-\u003ehandle is already filled in by depot_pop_free_pool(). */\n \tmemcpy(stack-\u003eentries, entries, flex_array_size(stack, entries, nr_entries));\n \n@@ -609,6 +1254,9 @@ static inline struct stack_record *find_stack(struct list_head *bucket,\n \tlist_for_each_entry_rcu(stack, bucket, hash_list) {\n \t\tif (stack-\u003ehash != hash || stack-\u003esize != size)\n \t\t\tcontinue;\n+\t\t/* Page owner countable records have a distinct count lifetime. */\n+\t\tif ((stack-\u003eflags ^ flags) \u0026 STACK_DEPOT_FLAG_COUNTABLE)\n+\t\t\tcontinue;\n \n \t\t/*\n \t\t * This may race with depot_free_stack() accessing the freelist\n@@ -638,6 +1286,63 @@ static inline struct stack_record *find_stack(struct list_head *bucket,\n \treturn ret;\n }\n \n+static u32\n+stack_depot_trie_insert(const unsigned long *entries,\n+\t\t\tunsigned int nr_entries, void **pool_prealloc,\n+\t\t\tstruct stack_depot_trie_side_prealloc *side_prealloc);\n+\n+static depot_stack_handle_t\n+stack_depot_trie_save(unsigned long *entries, unsigned int nr_entries,\n+\t\t gfp_t alloc_flags)\n+{\n+\tunsigned int attempt;\n+\n+\t/* Allow one stale pool hint before the two pools a largest insert needs. */\n+\tfor (attempt = 0; attempt \u003c 3; attempt++) {\n+\t\tstruct stack_depot_trie_side_prealloc side_prealloc = {};\n+\t\tvoid *pool_prealloc = NULL;\n+\t\tdepot_stack_handle_t handle;\n+\t\tunsigned long flags;\n+\t\tstruct page *page;\n+\t\tu32 stack_id = 0;\n+\n+\t\thandle = trie_find_handle(entries, nr_entries);\n+\t\tif (handle)\n+\t\t\treturn handle;\n+\n+\t\tif (trie_side_table_get_prealloc(alloc_flags, \u0026side_prealloc)) {\n+\t\t\ttrie_side_table_put_prealloc(\u0026side_prealloc);\n+\t\t\treturn 0;\n+\t\t}\n+\n+\t\t/* The hint may race; a missing page is recovered by the retry. */\n+\t\tif (!READ_ONCE(new_pool)) {\n+\t\t\tpage = alloc_pages(gfp_nested_mask(alloc_flags),\n+\t\t\t\t\t DEPOT_POOL_ORDER);\n+\t\t\tif (page)\n+\t\t\t\tpool_prealloc = page_address(page);\n+\t\t}\n+\n+\t\traw_spin_lock_irqsave(\u0026stack_depot_trie_writer_lock, flags);\n+\t\tstack_id = stack_depot_trie_insert(entries, nr_entries,\n+\t\t\t\t\t\t \u0026pool_prealloc, \u0026side_prealloc);\n+\t\traw_spin_unlock_irqrestore(\u0026stack_depot_trie_writer_lock, flags);\n+\n+\t\tif (pool_prealloc) {\n+\t\t\traw_spin_lock_irqsave(\u0026pool_lock, flags);\n+\t\t\tdepot_keep_new_pool(\u0026pool_prealloc);\n+\t\t\traw_spin_unlock_irqrestore(\u0026pool_lock, flags);\n+\t\t}\n+\t\tif (pool_prealloc)\n+\t\t\tfree_pages((unsigned long)pool_prealloc, DEPOT_POOL_ORDER);\n+\t\ttrie_side_table_put_prealloc(\u0026side_prealloc);\n+\t\tif (stack_id)\n+\t\t\treturn trie_handle(stack_id);\n+\t}\n+\n+\treturn 0;\n+}\n+\n depot_stack_handle_t stack_depot_save_flags(unsigned long *entries,\n \t\t\t\t\t unsigned int nr_entries,\n \t\t\t\t\t gfp_t alloc_flags,\n@@ -655,6 +1360,9 @@ depot_stack_handle_t stack_depot_save_flags(unsigned long *entries,\n \n \tif (WARN_ON(depot_flags \u0026 ~STACK_DEPOT_FLAGS_MASK))\n \t\treturn 0;\n+\tif (WARN_ON_ONCE((depot_flags \u0026 STACK_DEPOT_FLAG_GET) \u0026\u0026\n+\t\t\t (depot_flags \u0026 STACK_DEPOT_FLAG_COUNTABLE)))\n+\t\treturn 0;\n \n \t/*\n \t * If this stack trace is from an interrupt, including anything before\n@@ -669,6 +1377,17 @@ depot_stack_handle_t stack_depot_save_flags(unsigned long *entries,\n \tif (unlikely(nr_entries == 0) || stack_depot_disabled)\n \t\treturn 0;\n \n+\tif (!(depot_flags \u0026 (STACK_DEPOT_FLAG_GET | STACK_DEPOT_FLAG_COUNTABLE)) \u0026\u0026\n+\t static_branch_unlikely(\u0026stack_depot_trie_enabled)) {\n+\t\tif (nr_entries \u003e CONFIG_STACKDEPOT_MAX_FRAMES)\n+\t\t\tnr_entries = CONFIG_STACKDEPOT_MAX_FRAMES;\n+\t\tif (in_nmi() || !can_alloc) {\n+\t\t\tWARN_ON_ONCE(can_alloc);\n+\t\t\treturn trie_find_handle(entries, nr_entries);\n+\t\t}\n+\t\treturn stack_depot_trie_save(entries, nr_entries, alloc_flags);\n+\t}\n+\n \thash = hash_stack(entries, nr_entries);\n \tbucket = \u0026stack_table[hash \u0026 stack_hash_mask];\n \n@@ -751,10 +1470,697 @@ EXPORT_SYMBOL_GPL(stack_depot_save);\n \n struct stack_record *__stack_depot_get_stack_record(depot_stack_handle_t handle)\n {\n+\tstruct stack_record *stack;\n+\n \tif (!handle)\n \t\treturn NULL;\n+\tif (WARN_ON_ONCE(stack_depot_handle_is_trie(handle)))\n+\t\treturn NULL;\n+\n+\tstack = depot_fetch_stack(handle);\n+\tif (!stack)\n+\t\treturn NULL;\n+\tif (WARN_ON_ONCE(!(stack-\u003eflags \u0026 STACK_DEPOT_FLAG_COUNTABLE)))\n+\t\treturn NULL;\n+\n+\treturn stack;\n+}\n+\n+static void frame_run_init(const unsigned long *entries,\n+\t\t\t unsigned int nr_entries,\n+\t\t\t struct stack_depot_frame_run *run)\n+{\n+\tu32 payload;\n+\tunsigned int i;\n+\tbool compressed;\n+\n+\tcompressed = arch_stack_depot_frame_try_compress(entries[0], \u0026payload);\n+\tfor (i = 1; i \u003c nr_entries; i++) {\n+\t\tbool next;\n+\n+\t\tnext = arch_stack_depot_frame_try_compress(entries[i], \u0026payload);\n+\t\tif (next != compressed)\n+\t\t\tbreak;\n+\t}\n \n-\treturn depot_fetch_stack(handle);\n+\t/* @i is the first non-matching frame, or @nr_entries if all matched. */\n+\trun-\u003emode = compressed ? STACK_DEPOT_FRAME_COMPRESSED : STACK_DEPOT_FRAME_RAW;\n+\trun-\u003enr_entries = i;\n+}\n+\n+static void\n+stack_depot_trie_node_frame(const struct stack_depot_trie_node *node,\n+\t\t\t unsigned int index, unsigned long *frame)\n+{\n+\tu32 payload;\n+\n+\tif (node-\u003erun.mode == STACK_DEPOT_FRAME_RAW) {\n+\t\tmemcpy(frame, node-\u003edata + index * sizeof(*frame),\n+\t\t sizeof(*frame));\n+\t\treturn;\n+\t}\n+\n+\tmemcpy(\u0026payload, node-\u003edata + index * sizeof(payload), sizeof(payload));\n+\tarch_stack_depot_frame_decompress(payload, frame);\n+}\n+\n+static void trie_node_init(struct stack_depot_trie_node *node,\n+\t\t\t const struct stack_depot_trie_node *parent, u32 stack_id,\n+\t\t\t const unsigned long *entries,\n+\t\t\t const struct stack_depot_frame_run *run)\n+{\n+\tif (run-\u003emode == STACK_DEPOT_FRAME_COMPRESSED) {\n+\t\tunsigned int i;\n+\n+\t\tfor (i = 0; i \u003c run-\u003enr_entries; i++) {\n+\t\t\tu32 payload;\n+\n+\t\t\tarch_stack_depot_frame_try_compress(entries[i], \u0026payload);\n+\t\t\tmemcpy(node-\u003edata + i * sizeof(payload), \u0026payload,\n+\t\t\t sizeof(payload));\n+\t\t}\n+\t} else {\n+\t\tmemcpy(node-\u003edata, entries, stack_depot_frame_run_bytes(run));\n+\t}\n+\n+\tRCU_INIT_POINTER(node-\u003eparent, parent);\n+\tRCU_INIT_POINTER(node-\u003echildren, NULL);\n+\tnode-\u003estack_id = stack_id;\n+\tnode-\u003erun = *run;\n+}\n+\n+static void trie_node_init_slice(struct stack_depot_trie_node *node,\n+\t\t\t\t const struct stack_depot_trie_node *parent, u32 stack_id,\n+\t\t\t\t const struct stack_depot_trie_node *src_node,\n+\t\t\t\t unsigned int start, unsigned int nr_entries)\n+{\n+\tstruct stack_depot_frame_run run;\n+\tsize_t entry_bytes;\n+\n+\trun = src_node-\u003erun;\n+\trun.nr_entries = nr_entries;\n+\n+\tentry_bytes = stack_depot_frame_run_entry_bytes(src_node-\u003erun.mode);\n+\tmemcpy(node-\u003edata, src_node-\u003edata + start * entry_bytes,\n+\t stack_depot_frame_run_bytes(\u0026run));\n+\tRCU_INIT_POINTER(node-\u003eparent, parent);\n+\tRCU_INIT_POINTER(node-\u003echildren, NULL);\n+\tnode-\u003estack_id = stack_id;\n+\tnode-\u003erun = run;\n+}\n+\n+static unsigned int trie_node_match(const struct stack_depot_trie_node *node,\n+\t\t\t\t const unsigned long *entries,\n+\t\t\t\t unsigned int nr_entries)\n+{\n+\tunsigned int limit;\n+\tunsigned int i;\n+\n+\tlimit = min(node-\u003erun.nr_entries, nr_entries);\n+\tif (node-\u003erun.mode == STACK_DEPOT_FRAME_RAW) {\n+\t\tfor (i = 0; i \u003c limit; i++) {\n+\t\t\tunsigned long frame;\n+\n+\t\t\tmemcpy(\u0026frame, node-\u003edata + i * sizeof(frame), sizeof(frame));\n+\t\t\tif (frame != entries[i])\n+\t\t\t\tbreak;\n+\t\t}\n+\n+\t\treturn i;\n+\t}\n+\n+\tfor (i = 0; i \u003c limit; i++) {\n+\t\tunsigned long frame;\n+\n+\t\tstack_depot_trie_node_frame(node, i, \u0026frame);\n+\t\tif (frame != entries[i])\n+\t\t\tbreak;\n+\t}\n+\n+\treturn i;\n+}\n+\n+static inline const struct stack_depot_trie_node *\n+trie_load_parent(const struct stack_depot_trie_node *node)\n+{\n+\treturn rcu_dereference_check(node-\u003eparent,\n+\t\t\t\t lockdep_is_held(\u0026stack_depot_trie_writer_lock) ||\n+\t\t\t\t rcu_read_lock_sched_held());\n+}\n+\n+static inline const struct stack_depot_trie_children *\n+trie_load_children(const struct stack_depot_trie_children __rcu * const *slot)\n+{\n+\treturn rcu_dereference_check(*slot,\n+\t\t\t\t lockdep_is_held(\u0026stack_depot_trie_writer_lock) ||\n+\t\t\t\t rcu_read_lock_sched_held());\n+}\n+\n+static inline const struct stack_depot_trie_node *\n+trie_children_load_child(const struct stack_depot_trie_children *children,\n+\t\t\t unsigned int pos)\n+{\n+\treturn rcu_dereference_check(children-\u003enodes[pos],\n+\t\t\t\t lockdep_is_held(\u0026stack_depot_trie_writer_lock) ||\n+\t\t\t\t rcu_read_lock_sched_held());\n+}\n+\n+static bool\n+trie_children_find_position(const struct stack_depot_trie_children *children,\n+\t\t\t unsigned long frame, unsigned int *pos)\n+{\n+\tunsigned int left = 0;\n+\tunsigned int right;\n+\n+\tright = READ_ONCE(children-\u003enr_children);\n+\twhile (left \u003c right) {\n+\t\tunsigned int mid = left + (right - left) / 2;\n+\t\tconst struct stack_depot_trie_node *node;\n+\t\tunsigned long mid_frame;\n+\n+\t\tnode = trie_children_load_child(children, mid);\n+\t\tif (!node) {\n+\t\t\t/* Tail append may produce a transient lockless lookup miss. */\n+\t\t\tright = mid;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tstack_depot_trie_node_frame(node, 0, \u0026mid_frame);\n+\t\tif (mid_frame \u003c frame) {\n+\t\t\tleft = mid + 1;\n+\t\t} else if (mid_frame \u003e frame) {\n+\t\t\tright = mid;\n+\t\t} else {\n+\t\t\t*pos = mid;\n+\t\t\treturn true;\n+\t\t}\n+\t}\n+\n+\t*pos = left;\n+\treturn false;\n+}\n+\n+/* Initialize an unpublished container from a stable published prefix. */\n+static void trie_children_init(const struct stack_depot_trie_children *old,\n+\t\t\t struct stack_depot_trie_children *new)\n+{\n+\tunsigned int nr_old = old-\u003enr_children;\n+\tunsigned int i;\n+\n+\tnew-\u003enr_children = nr_old;\n+\tfor (i = 0; i \u003c nr_old; i++)\n+\t\tRCU_INIT_POINTER(new-\u003enodes[i], trie_children_load_child(old, i));\n+\tfor (i = nr_old; i \u003c new-\u003ecapacity; i++)\n+\t\tRCU_INIT_POINTER(new-\u003enodes[i], NULL);\n+}\n+\n+static void trie_children_insert(struct stack_depot_trie_children *children,\n+\t\t\t\t const struct stack_depot_trie_node *node,\n+\t\t\t\t unsigned int pos)\n+{\n+\tunsigned int i;\n+\n+\tfor (i = children-\u003enr_children; i \u003e pos; i--)\n+\t\tRCU_INIT_POINTER(children-\u003enodes[i],\n+\t\t\t\t trie_children_load_child(children, i - 1));\n+\tRCU_INIT_POINTER(children-\u003enodes[pos], node);\n+\tchildren-\u003enr_children++;\n+}\n+\n+static void trie_reparent_children(struct stack_depot_trie_node *parent)\n+{\n+\tconst struct stack_depot_trie_children *children;\n+\tunsigned int i;\n+\n+\tlockdep_assert_held(\u0026stack_depot_trie_writer_lock);\n+\n+\tchildren = trie_load_children(\u0026parent-\u003echildren);\n+\tif (!children)\n+\t\treturn;\n+\t/*\n+\t * Replacement nodes reuse unchanged descendant subtrees. Repoint their\n+\t * parent links before retiring the old parent so fetch never follows a freed\n+\t * node. Lockless fetches may see the new parent before publication, but the\n+\t * old and new parent chains contain the same frames and remain RCU-live.\n+\t */\n+\tfor (i = 0; i \u003c children-\u003enr_children; i++) {\n+\t\tstruct stack_depot_trie_node *child;\n+\n+\t\tchild = (struct stack_depot_trie_node *)trie_children_load_child(children, i);\n+\t\trcu_assign_pointer(child-\u003eparent, parent);\n+\t}\n+}\n+\n+/*\n+ * Split entries into runs, allocate and initialize each node once, and link\n+ * adjacent nodes through singleton children. Both trie locks must be held.\n+ * Failure walks the unpublished parent chain and releases local ownership.\n+ */\n+static const struct stack_depot_trie_node *\n+trie_path_alloc(const struct stack_depot_trie_node *parent, u32 stack_id,\n+\t\tconst unsigned long *entries, unsigned int nr_entries,\n+\t\tvoid **pool_prealloc,\n+\t\tconst struct stack_depot_trie_node **node_out)\n+{\n+\tstruct stack_depot_trie_children *path_children = NULL;\n+\tconst struct stack_depot_trie_node *path_root = NULL;\n+\tconst struct stack_depot_trie_node *last_node = parent;\n+\tunsigned int entry = 0;\n+\n+\tlockdep_assert_held(\u0026pool_lock);\n+\tlockdep_assert_held(\u0026stack_depot_trie_writer_lock);\n+\n+\twhile (entry \u003c nr_entries) {\n+\t\tstruct stack_depot_frame_run run;\n+\t\tstruct stack_depot_trie_node *node;\n+\n+\t\tframe_run_init(\u0026entries[entry], nr_entries - entry, \u0026run);\n+\t\tnode = trie_pool_alloc(trie_node_bytes(\u0026run), pool_prealloc);\n+\t\tif (!node)\n+\t\t\tgoto err_release;\n+\n+\t\ttrie_node_init(node, last_node,\n+\t\t\t entry + run.nr_entries == nr_entries ? stack_id : 0,\n+\t\t\t \u0026entries[entry], \u0026run);\n+\t\tentry += run.nr_entries;\n+\t\tlast_node = node;\n+\t\tif (!path_root)\n+\t\t\tpath_root = node;\n+\n+\t\tif (path_children)\n+\t\t\ttrie_children_insert(path_children, last_node, 0);\n+\t\tif (entry \u003c nr_entries) {\n+\t\t\tpath_children = trie_pool_alloc_children(1, pool_prealloc);\n+\t\t\tif (!path_children)\n+\t\t\t\tgoto err_release;\n+\t\t\tRCU_INIT_POINTER(node-\u003echildren, path_children);\n+\t\t}\n+\t}\n+\n+\t*node_out = last_node;\n+\treturn path_root;\n+\n+err_release:\n+\twhile (last_node != parent) {\n+\t\tconst struct stack_depot_trie_children *node_children;\n+\t\tconst struct stack_depot_trie_node *node = last_node;\n+\n+\t\tlast_node = trie_load_parent(node);\n+\t\tnode_children = trie_load_children(\u0026node-\u003echildren);\n+\t\tif (node_children)\n+\t\t\ttrie_pool_release_children(node_children);\n+\t\ttrie_pool_release(node, trie_node_bytes(\u0026node-\u003erun));\n+\t}\n+\treturn NULL;\n+}\n+\n+static const struct stack_depot_trie_node *\n+stack_depot_trie_lookup(const unsigned long *entries, unsigned int nr_entries)\n+{\n+\tconst struct stack_depot_trie_children *children;\n+\tunsigned int entry = 0;\n+\n+\tchildren = trie_load_children(\u0026stack_depot_trie_root);\n+\n+\twhile (entry \u003c nr_entries) {\n+\t\tconst struct stack_depot_trie_node *node;\n+\t\tunsigned int remaining = nr_entries - entry;\n+\t\tunsigned int matched;\n+\t\tunsigned int pos;\n+\n+\t\tif (!children)\n+\t\t\treturn NULL;\n+\t\tif (!trie_children_find_position(children, entries[entry], \u0026pos))\n+\t\t\treturn NULL;\n+\n+\t\tnode = trie_children_load_child(children, pos);\n+\t\tmatched = trie_node_match(node, \u0026entries[entry], remaining);\n+\t\tif (matched \u003c node-\u003erun.nr_entries)\n+\t\t\treturn NULL;\n+\t\tentry += matched;\n+\t\tif (entry == nr_entries)\n+\t\t\treturn node-\u003estack_id ? node : NULL;\n+\n+\t\tchildren = trie_load_children(\u0026node-\u003echildren);\n+\t}\n+\n+\treturn NULL;\n+}\n+\n+static u32\n+trie_insert_path(const struct stack_depot_trie_children __rcu **slot,\n+\t\t struct stack_depot_trie_node *parent,\n+\t\t const struct stack_depot_trie_children *children,\n+\t\t unsigned int pos, const unsigned long *entries,\n+\t\t unsigned int nr_entries, void **pool_prealloc,\n+\t\t struct stack_depot_trie_side_prealloc *side_prealloc)\n+{\n+\tstruct stack_depot_trie_children *new_children = NULL;\n+\tconst struct stack_depot_trie_node *path_root;\n+\tconst struct stack_depot_trie_node *node;\n+\tunsigned int capacity = 1;\n+\tu32 new_stack_id;\n+\tbool tail_append = false;\n+\n+\t/*\n+\t * Reuse spare capacity only for a sorted tail append. Other insertions\n+\t * replace the children container without modifying visible pointers.\n+\t */\n+\tif (children) {\n+\t\tcapacity = roundup_pow_of_two(children-\u003enr_children + 1);\n+\t\ttail_append = pos == children-\u003enr_children \u0026\u0026\n+\t\t\tchildren-\u003enr_children \u003c children-\u003ecapacity;\n+\t}\n+\tif (!tail_append \u0026\u0026 trie_children_alloc_size(capacity) \u003e\n+\t STACK_DEPOT_TRIE_POOL_USABLE_SIZE)\n+\t\treturn 0;\n+\n+\tnew_stack_id = trie_side_table_prepare_stack_slot(side_prealloc);\n+\tif (!new_stack_id)\n+\t\treturn 0;\n+\n+\traw_spin_lock(\u0026pool_lock);\n+\tprintk_deferred_enter();\n+\ttrie_drain_pending_children();\n+\n+\t/* Reserve replacement topology before the path, the final fallible step. */\n+\tif (!tail_append) {\n+\t\tnew_children = trie_pool_alloc_children(capacity, pool_prealloc);\n+\t\tif (!new_children)\n+\t\t\tgoto err_release;\n+\t}\n+\tpath_root = trie_path_alloc(parent, new_stack_id, entries, nr_entries,\n+\t\t\t\t pool_prealloc, \u0026node);\n+\tif (!path_root)\n+\t\tgoto err_release;\n+\n+\t/* Commit the stack ID before making the path reachable from the trie. */\n+\ttrie_side_table_publish(node);\n+\tif (tail_append) {\n+\t\tstruct stack_depot_trie_children *tail_children =\n+\t\t\t(struct stack_depot_trie_children *)children;\n+\n+\t\t/*\n+\t\t * Publish the node before the visible count. Readers may transiently\n+\t\t * see NULL and miss; the writer-lock recheck prevents duplicates.\n+\t\t */\n+\t\trcu_assign_pointer(tail_children-\u003enodes[pos], path_root);\n+\t\tWRITE_ONCE(tail_children-\u003enr_children, pos + 1);\n+\t} else {\n+\t\tif (children)\n+\t\t\ttrie_children_init(children, new_children);\n+\t\ttrie_children_insert(new_children, path_root, pos);\n+\t\trcu_assign_pointer(*slot, new_children);\n+\t\tif (children)\n+\t\t\ttrie_retire_children(children);\n+\t}\n+\n+\tprintk_deferred_exit();\n+\traw_spin_unlock(\u0026pool_lock);\n+\treturn new_stack_id;\n+\n+err_release:\n+\tif (new_children)\n+\t\ttrie_pool_release_children(new_children);\n+\tprintk_deferred_exit();\n+\traw_spin_unlock(\u0026pool_lock);\n+\treturn 0;\n+}\n+\n+static u32\n+trie_split_child(const struct stack_depot_trie_children __rcu **slot,\n+\t\t const struct stack_depot_trie_children *children,\n+\t\t const struct stack_depot_trie_node *child,\n+\t\t unsigned int pos, unsigned int matched,\n+\t\t const unsigned long *entries, unsigned int nr_entries,\n+\t\t void **pool_prealloc,\n+\t\t struct stack_depot_trie_side_prealloc *side_prealloc)\n+{\n+\tstruct stack_depot_trie_children *prefix_children = NULL;\n+\tstruct stack_depot_trie_children *new_children = NULL;\n+\tconst struct stack_depot_trie_node *new_node;\n+\tconst struct stack_depot_trie_node *suffix_roots[2];\n+\tstruct stack_depot_frame_run run;\n+\tstruct stack_depot_trie_node *split_prefix = NULL;\n+\tstruct stack_depot_trie_node *old_suffix = NULL;\n+\tunsigned int nr_suffix_roots;\n+\tunsigned int old_suffix_len;\n+\tunsigned int i;\n+\tsize_t split_prefix_size;\n+\tsize_t old_suffix_size;\n+\tu32 new_stack_id;\n+\tbool has_new_suffix;\n+\n+\tnew_stack_id = trie_side_table_prepare_stack_slot(side_prealloc);\n+\tif (!new_stack_id)\n+\t\treturn 0;\n+\n+\t/* Rebuild the child's run as newly allocated prefix and old suffix nodes. */\n+\trun = child-\u003erun;\n+\trun.nr_entries = matched;\n+\tsplit_prefix_size = trie_node_bytes(\u0026run);\n+\told_suffix_len = child-\u003erun.nr_entries - matched;\n+\trun.nr_entries = old_suffix_len;\n+\told_suffix_size = trie_node_bytes(\u0026run);\n+\thas_new_suffix = matched \u003c nr_entries;\n+\tnr_suffix_roots = has_new_suffix ? 2 : 1;\n+\n+\traw_spin_lock(\u0026pool_lock);\n+\tprintk_deferred_enter();\n+\ttrie_drain_pending_children();\n+\n+\t/* Reserve fixed split topology before the optional new suffix path. */\n+\tsplit_prefix = trie_pool_alloc(split_prefix_size, pool_prealloc);\n+\tif (!split_prefix)\n+\t\tgoto err_release;\n+\told_suffix = trie_pool_alloc(old_suffix_size, pool_prealloc);\n+\tif (!old_suffix)\n+\t\tgoto err_release;\n+\tnew_children = trie_pool_alloc_children(children-\u003ecapacity, pool_prealloc);\n+\tif (!new_children)\n+\t\tgoto err_release;\n+\tprefix_children = trie_pool_alloc_children(nr_suffix_roots, pool_prealloc);\n+\tif (!prefix_children)\n+\t\tgoto err_release;\n+\n+\tif (has_new_suffix) {\n+\t\tconst struct stack_depot_trie_node *new_suffix;\n+\t\tunsigned long old_suffix_frame;\n+\n+\t\tnew_suffix = trie_path_alloc(split_prefix, new_stack_id,\n+\t\t\t\t\t \u0026entries[matched], nr_entries - matched,\n+\t\t\t\t\t pool_prealloc, \u0026new_node);\n+\t\tif (!new_suffix)\n+\t\t\tgoto err_release;\n+\t\tstack_depot_trie_node_frame(child, matched, \u0026old_suffix_frame);\n+\t\t/* Children remain sorted by the first frame of each suffix. */\n+\t\tif (old_suffix_frame \u003c entries[matched]) {\n+\t\t\tsuffix_roots[0] = old_suffix;\n+\t\t\tsuffix_roots[1] = new_suffix;\n+\t\t} else {\n+\t\t\tsuffix_roots[0] = new_suffix;\n+\t\t\tsuffix_roots[1] = old_suffix;\n+\t\t}\n+\t} else {\n+\t\tnew_node = split_prefix;\n+\t\tsuffix_roots[0] = old_suffix;\n+\t}\n+\n+\tprintk_deferred_exit();\n+\traw_spin_unlock(\u0026pool_lock);\n+\n+\t/* Rebuild the old path as prefix -\u003e old suffix and attach suffix roots. */\n+\ttrie_node_init_slice(split_prefix, trie_load_parent(child),\n+\t\t\t has_new_suffix ? 0 : new_stack_id, child, 0, matched);\n+\ttrie_node_init_slice(old_suffix, split_prefix, child-\u003estack_id, child,\n+\t\t\t matched, old_suffix_len);\n+\tfor (i = 0; i \u003c nr_suffix_roots; i++)\n+\t\ttrie_children_insert(prefix_children, suffix_roots[i], i);\n+\tRCU_INIT_POINTER(old_suffix-\u003echildren,\n+\t\t\t trie_load_children(\u0026child-\u003echildren));\n+\tRCU_INIT_POINTER(split_prefix-\u003echildren, prefix_children);\n+\n+\t/* Publish IDs, reparent descendants, then replace and retire topology. */\n+\tif (child-\u003estack_id)\n+\t\ttrie_side_table_publish(old_suffix);\n+\ttrie_side_table_publish(new_node);\n+\t/* Old and replacement chains contain identical frames during transition. */\n+\ttrie_children_init(children, new_children);\n+\tRCU_INIT_POINTER(new_children-\u003enodes[pos], split_prefix);\n+\ttrie_reparent_children(old_suffix);\n+\trcu_assign_pointer(*slot, new_children);\n+\ttrie_retire_children_with_node(children, child);\n+\n+\treturn new_stack_id;\n+\n+err_release:\n+\tif (split_prefix)\n+\t\ttrie_pool_release(split_prefix, split_prefix_size);\n+\tif (old_suffix)\n+\t\ttrie_pool_release(old_suffix, old_suffix_size);\n+\tif (prefix_children)\n+\t\ttrie_pool_release_children(prefix_children);\n+\tif (new_children)\n+\t\ttrie_pool_release_children(new_children);\n+\tprintk_deferred_exit();\n+\traw_spin_unlock(\u0026pool_lock);\n+\treturn 0;\n+}\n+\n+static u32\n+trie_promote_child(const struct stack_depot_trie_children __rcu **slot,\n+\t\t const struct stack_depot_trie_children *children,\n+\t\t const struct stack_depot_trie_node *child,\n+\t\t unsigned int pos, void **pool_prealloc,\n+\t\t struct stack_depot_trie_side_prealloc *side_prealloc)\n+{\n+\tstruct stack_depot_trie_children *new_children;\n+\tstruct stack_depot_trie_node *promoted_node;\n+\tsize_t node_size;\n+\tu32 new_stack_id;\n+\n+\tnew_stack_id = trie_side_table_prepare_stack_slot(side_prealloc);\n+\tif (!new_stack_id)\n+\t\treturn 0;\n+\tnode_size = trie_node_bytes(\u0026child-\u003erun);\n+\n+\t/* Reserve a clone and replacement children container before publication. */\n+\traw_spin_lock(\u0026pool_lock);\n+\tprintk_deferred_enter();\n+\ttrie_drain_pending_children();\n+\tpromoted_node = trie_pool_alloc(node_size, pool_prealloc);\n+\tif (!promoted_node)\n+\t\tgoto out_unlock;\n+\tnew_children = trie_pool_alloc_children(children-\u003ecapacity, pool_prealloc);\n+\tif (!new_children)\n+\t\tgoto out_release_node;\n+\tprintk_deferred_exit();\n+\traw_spin_unlock(\u0026pool_lock);\n+\n+\t/* Add the stack ID through a clone, then reparent before retirement. */\n+\tmemcpy(promoted_node, child, node_size);\n+\tpromoted_node-\u003estack_id = new_stack_id;\n+\ttrie_side_table_publish(promoted_node);\n+\ttrie_children_init(children, new_children);\n+\tRCU_INIT_POINTER(new_children-\u003enodes[pos], promoted_node);\n+\ttrie_reparent_children(promoted_node);\n+\trcu_assign_pointer(*slot, new_children);\n+\ttrie_retire_children_with_node(children, child);\n+\n+\treturn new_stack_id;\n+\n+out_release_node:\n+\ttrie_pool_release(promoted_node, node_size);\n+out_unlock:\n+\tprintk_deferred_exit();\n+\traw_spin_unlock(\u0026pool_lock);\n+\treturn 0;\n+}\n+\n+static u32\n+stack_depot_trie_insert(const unsigned long *entries,\n+\t\t\tunsigned int nr_entries, void **pool_prealloc,\n+\t\t\tstruct stack_depot_trie_side_prealloc *side_prealloc)\n+{\n+\tconst struct stack_depot_trie_children *children;\n+\tconst struct stack_depot_trie_children __rcu **slot =\n+\t\t\u0026stack_depot_trie_root;\n+\tconst struct stack_depot_trie_node *child;\n+\tstruct stack_depot_trie_node *parent = NULL;\n+\tunsigned int matched;\n+\tunsigned int pos;\n+\tu32 stack_id;\n+\n+\tlockdep_assert_held(\u0026stack_depot_trie_writer_lock);\n+\n+\tfor (;;) {\n+\t\tpos = 0;\n+\t\tchildren = trie_load_children(slot);\n+\t\t/* No matching child: attach the remaining path. */\n+\t\tif (!children ||\n+\t\t !trie_children_find_position(children, entries[0], \u0026pos)) {\n+\t\t\tstack_id = trie_insert_path(slot, parent, children, pos,\n+\t\t\t\t\t\t entries, nr_entries, pool_prealloc,\n+\t\t\t\t\t\t side_prealloc);\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tchild = trie_children_load_child(children, pos);\n+\t\tmatched = trie_node_match(child, entries, nr_entries);\n+\t\t/* A partial child match requires a prefix/suffix split. */\n+\t\tif (matched \u003c child-\u003erun.nr_entries) {\n+\t\t\tstack_id = trie_split_child(slot, children, child, pos,\n+\t\t\t\t\t\t matched, entries, nr_entries,\n+\t\t\t\t\t\t pool_prealloc, side_prealloc);\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\t/* The input ends here: reuse a stack node or promote an internal one. */\n+\t\tif (matched == nr_entries) {\n+\t\t\tif (child-\u003estack_id)\n+\t\t\t\treturn child-\u003estack_id;\n+\t\t\tstack_id = trie_promote_child(slot, children, child, pos,\n+\t\t\t\t\t\t pool_prealloc, side_prealloc);\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\t/* The child matched completely; continue with the remaining frames. */\n+\t\tparent = (struct stack_depot_trie_node *)child;\n+\t\tslot = \u0026parent-\u003echildren;\n+\t\tentries += matched;\n+\t\tnr_entries -= matched;\n+\t}\n+\n+\tif (stack_id)\n+\t\ttrie_side_table_last_stack_id = stack_id;\n+\treturn stack_id;\n+}\n+\n+static unsigned int trie_fetch_into(const struct stack_depot_trie_node *node,\n+\t\t\t\t unsigned long *entries,\n+\t\t\t\t unsigned int max_entries)\n+{\n+\tconst struct stack_depot_trie_node *cur;\n+\tunsigned int total;\n+\tunsigned int pos;\n+\tunsigned int i;\n+\n+\ttotal = 0;\n+\tfor (cur = node; cur; cur = trie_load_parent(cur))\n+\t\ttotal += cur-\u003erun.nr_entries;\n+\tif (max_entries \u003c total)\n+\t\treturn 0;\n+\n+\tpos = total;\n+\tfor (cur = node; cur; cur = trie_load_parent(cur)) {\n+\t\tpos -= cur-\u003erun.nr_entries;\n+\t\tfor (i = 0; i \u003c cur-\u003erun.nr_entries; i++)\n+\t\t\tstack_depot_trie_node_frame(cur, i, \u0026entries[pos + i]);\n+\t}\n+\n+\treturn total;\n+}\n+\n+static unsigned int trie_fetch_handle_into(depot_stack_handle_t handle,\n+\t\t\t\t\t unsigned long *entries,\n+\t\t\t\t\t unsigned int max_entries)\n+{\n+\tconst struct stack_depot_trie_node *node;\n+\tu32 stack_id;\n+\tunsigned int nr_entries;\n+\n+\tstack_id = trie_stack_id(handle);\n+\trcu_read_lock_sched_notrace();\n+\tnode = trie_side_table_lookup(stack_id);\n+\tif (WARN_ONCE(!node, \"corrupt trie handle %08x\\n\", handle)) {\n+\t\trcu_read_unlock_sched_notrace();\n+\t\treturn 0;\n+\t}\n+\tnr_entries = trie_fetch_into(node, entries, max_entries);\n+\trcu_read_unlock_sched_notrace();\n+\tif (nr_entries)\n+\t\tkmsan_unpoison_memory(entries, nr_entries * sizeof(*entries));\n+\n+\treturn nr_entries;\n }\n \n unsigned int stack_depot_fetch(depot_stack_handle_t handle,\n@@ -771,6 +2177,8 @@ unsigned int stack_depot_fetch(depot_stack_handle_t handle,\n \n \tif (!handle || stack_depot_disabled)\n \t\treturn 0;\n+\tif (WARN_ON_ONCE(stack_depot_handle_is_trie(handle)))\n+\t\treturn 0;\n \n \tstack = depot_fetch_stack(handle);\n \t/*\n@@ -785,12 +2193,44 @@ unsigned int stack_depot_fetch(depot_stack_handle_t handle,\n }\n EXPORT_SYMBOL_GPL(stack_depot_fetch);\n \n+unsigned int stack_depot_fetch_into(depot_stack_handle_t handle,\n+\t\t\t\t unsigned long *entries,\n+\t\t\t\t unsigned int max_entries)\n+{\n+\tstruct stack_record *stack;\n+\tunsigned int nr_entries;\n+\n+\tif (!handle)\n+\t\treturn 0;\n+\tif (stack_depot_disabled)\n+\t\treturn 0;\n+\tWARN_ON_ONCE(!entries || !max_entries);\n+\tif (stack_depot_handle_is_trie(handle))\n+\t\treturn trie_fetch_handle_into(handle, entries, max_entries);\n+\n+\tstack = depot_fetch_stack(handle);\n+\tif (!stack)\n+\t\treturn 0;\n+\tnr_entries = stack-\u003esize;\n+\tif (WARN_ON_ONCE(!nr_entries))\n+\t\treturn 0;\n+\tif (nr_entries \u003e max_entries)\n+\t\treturn 0;\n+\n+\tmemcpy(entries, stack-\u003eentries, nr_entries * sizeof(*entries));\n+\tkmsan_unpoison_memory(entries, nr_entries * sizeof(*entries));\n+\treturn nr_entries;\n+}\n+EXPORT_SYMBOL_GPL(stack_depot_fetch_into);\n+\n void stack_depot_put(depot_stack_handle_t handle)\n {\n \tstruct stack_record *stack;\n \n \tif (!handle || stack_depot_disabled)\n \t\treturn;\n+\tif (WARN_ON_ONCE(stack_depot_handle_is_trie(handle)))\n+\t\treturn;\n \n \tstack = depot_fetch_stack(handle);\n \t/*\n@@ -800,6 +2240,8 @@ void stack_depot_put(depot_stack_handle_t handle)\n \tif (WARN(!stack, \"corrupt handle or unbalanced stack_depot_put()\"))\n \t\treturn;\n \n+\tif (WARN_ON_ONCE(stack-\u003eflags \u0026 STACK_DEPOT_FLAG_COUNTABLE))\n+\t\treturn;\n \tif (refcount_dec_and_test(\u0026stack-\u003ecount))\n \t\tdepot_free_stack(stack);\n }\n@@ -810,6 +2252,15 @@ void stack_depot_print(depot_stack_handle_t stack)\n \tunsigned long *entries;\n \tunsigned int nr_entries;\n \n+\tif (stack_depot_handle_is_trie(stack)) {\n+\t\tunsigned long trie_entries[CONFIG_STACKDEPOT_MAX_FRAMES];\n+\n+\t\tnr_entries = trie_fetch_handle_into(stack, trie_entries,\n+\t\t\t\t\t\t ARRAY_SIZE(trie_entries));\n+\t\tstack_trace_print(trie_entries, nr_entries, 0);\n+\t\treturn;\n+\t}\n+\n \tnr_entries = stack_depot_fetch(stack, \u0026entries);\n \tif (nr_entries \u003e 0)\n \t\tstack_trace_print(entries, nr_entries, 0);\n@@ -822,6 +2273,15 @@ int stack_depot_snprint(depot_stack_handle_t handle, char *buf, size_t size,\n \tunsigned long *entries;\n \tunsigned int nr_entries;\n \n+\tif (stack_depot_handle_is_trie(handle)) {\n+\t\tunsigned long trie_entries[CONFIG_STACKDEPOT_MAX_FRAMES];\n+\n+\t\tnr_entries = trie_fetch_handle_into(handle, trie_entries,\n+\t\t\t\t\t\t ARRAY_SIZE(trie_entries));\n+\t\treturn stack_trace_snprint(buf, size, trie_entries, nr_entries,\n+\t\t\t\t\t spaces);\n+\t}\n+\n \tnr_entries = stack_depot_fetch(handle, \u0026entries);\n \treturn nr_entries ? stack_trace_snprint(buf, size, entries, nr_entries,\n \t\t\t\t\t\tspaces) : 0;\ndiff --git a/lib/tests/Makefile b/lib/tests/Makefile\nindex 4ead57602eac4..2d40bd21a8ef9 100644\n--- a/lib/tests/Makefile\n+++ b/lib/tests/Makefile\n@@ -49,6 +49,7 @@ obj-$(CONFIG_SCANF_KUNIT_TEST) += scanf_kunit.o\n obj-$(CONFIG_SEQ_BUF_KUNIT_TEST) += seq_buf_kunit.o\n obj-$(CONFIG_SIPHASH_KUNIT_TEST) += siphash_kunit.o\n obj-$(CONFIG_SLUB_KUNIT_TEST) += slub_kunit.o\n+obj-$(CONFIG_STACKDEPOT_KUNIT_TEST) += stackdepot_kunit.o\n obj-$(CONFIG_TEST_SORT) += test_sort.o\n CFLAGS_stackinit_kunit.o += $(call cc-disable-warning, switch-unreachable)\n obj-$(CONFIG_STACKINIT_KUNIT_TEST) += stackinit_kunit.o\ndiff --git a/lib/tests/stackdepot_kunit.c b/lib/tests/stackdepot_kunit.c\nnew file mode 100644\nindex 0000000000000..be14cae98fcfb\n--- /dev/null\n+++ b/lib/tests/stackdepot_kunit.c\n@@ -0,0 +1,473 @@\n+// SPDX-License-Identifier: GPL-2.0-only\n+\n+#include \u003ckunit/test.h\u003e\n+#include \u003clinux/array_size.h\u003e\n+#include \u003clinux/gfp.h\u003e\n+#include \u003clinux/limits.h\u003e\n+#include \u003clinux/moduleparam.h\u003e\n+#include \u003clinux/stackdepot.h\u003e\n+#include \u003clinux/stacktrace.h\u003e\n+#include \u003clinux/string.h\u003e\n+\n+#include \u003casm/stackdepot.h\u003e\n+\n+static int expected_trie_pool_limit = -1;\n+module_param_named(trie_pool_limit, expected_trie_pool_limit, int, 0);\n+MODULE_PARM_DESC(trie_pool_limit, \"Expected stackdepot hash/trie pool split\");\n+\n+#ifdef CONFIG_ARM64\n+#include \u003casm/sections.h\u003e\n+\n+static inline unsigned long stackdepot_arm64_frame(long offset)\n+{\n+\treturn (unsigned long)((long)_text + offset);\n+}\n+#endif\n+\n+static void stackdepot_trie_max_path_roundtrip(struct kunit *test)\n+{\n+\tunion handle_parts parts;\n+\tunsigned long *entries;\n+\tunsigned long *fetched;\n+\tdepot_stack_handle_t handle;\n+\tsize_t size = CONFIG_STACKDEPOT_MAX_FRAMES * sizeof(*entries);\n+\tu32 pool_index_plus_1;\n+\tunsigned int i;\n+\n+\tif (expected_trie_pool_limit \u003c 0)\n+\t\tkunit_skip(test, \"trie pool limit was not provided\");\n+\tKUNIT_ASSERT_EQ(test, stack_depot_init(), 0);\n+\tentries = kunit_kcalloc(test, CONFIG_STACKDEPOT_MAX_FRAMES,\n+\t\t\t\tsizeof(*entries), GFP_KERNEL);\n+\tKUNIT_ASSERT_NOT_NULL(test, entries);\n+\tfetched = kunit_kcalloc(test, CONFIG_STACKDEPOT_MAX_FRAMES,\n+\t\t\t\tsizeof(*fetched), GFP_KERNEL);\n+\tKUNIT_ASSERT_NOT_NULL(test, fetched);\n+\tfor (i = 0; i \u003c CONFIG_STACKDEPOT_MAX_FRAMES; i++) {\n+#ifdef CONFIG_ARM64\n+\t\tentries[i] = i \u0026 1 ? 0x1000UL + i * 0x1000UL :\n+\t\t\tstackdepot_arm64_frame(i * 4);\n+#elif defined(CONFIG_X86_64) \u0026\u0026 !defined(CONFIG_UML)\n+\t\tentries[i] = i \u0026 1 ? 0xffff888000000000UL + i * 0x1000UL :\n+\t\t\t0xffffffff10000000UL + i * 0x10UL;\n+#else\n+\t\tentries[i] = 0x1000UL + i * 0x1000UL;\n+#endif\n+\t}\n+\n+\thandle = stack_depot_save(entries, CONFIG_STACKDEPOT_MAX_FRAMES,\n+\t\t\t\t GFP_KERNEL);\n+\tKUNIT_ASSERT_NE(test, handle, (depot_stack_handle_t)0);\n+\tparts.handle = handle;\n+\tpool_index_plus_1 = parts.pool_index_plus_1;\n+\tKUNIT_EXPECT_GT(test, pool_index_plus_1, (u32)expected_trie_pool_limit);\n+\tKUNIT_EXPECT_EQ(test,\n+\t\t\tstack_depot_fetch_into(handle, fetched,\n+\t\t\t\t\t CONFIG_STACKDEPOT_MAX_FRAMES),\n+\t\t\t(unsigned int)CONFIG_STACKDEPOT_MAX_FRAMES);\n+\tKUNIT_EXPECT_MEMEQ(test, fetched, entries, size);\n+\tKUNIT_EXPECT_EQ(test,\n+\t\t\tstack_depot_save(entries, CONFIG_STACKDEPOT_MAX_FRAMES,\n+\t\t\t\t\t GFP_KERNEL),\n+\t\t\thandle);\n+}\n+\n+static void stackdepot_save_flags_public(struct kunit *test)\n+{\n+\tunsigned long entries[] = { 0x501000UL, 0x502000UL, 0x503000UL };\n+\tunsigned long get_entries[] = { 0x601000UL, 0x602000UL };\n+\tunsigned long missing_entries[] = { 0x701000UL, 0x702000UL };\n+\tunsigned long fetched[ARRAY_SIZE(entries)] = {};\n+\tdepot_stack_handle_t noalloc_handle;\n+\tdepot_stack_handle_t overlong_handle;\n+\tdepot_stack_handle_t plain_handle;\n+\tdepot_stack_handle_t get_handle;\n+\tdepot_stack_handle_t again;\n+\tdepot_stack_handle_t extra;\n+\tgfp_t no_spin = GFP_NOWAIT \u0026 ~__GFP_RECLAIM;\n+\tunsigned long *overlong_fetched;\n+\tunsigned long *overlong_entries;\n+\tunsigned int overlong_nr = CONFIG_STACKDEPOT_MAX_FRAMES + 1;\n+\tunsigned int nr_entries;\n+\tsize_t overlong_size;\n+\tunsigned int i;\n+\n+\tKUNIT_ASSERT_EQ(test, stack_depot_init(), 0);\n+\toverlong_entries = kunit_kcalloc(test, overlong_nr,\n+\t\t\t\t\t sizeof(*overlong_entries), GFP_KERNEL);\n+\tKUNIT_ASSERT_NOT_NULL(test, overlong_entries);\n+\toverlong_fetched = kunit_kcalloc(test, CONFIG_STACKDEPOT_MAX_FRAMES,\n+\t\t\t\t\t sizeof(*overlong_fetched), GFP_KERNEL);\n+\tKUNIT_ASSERT_NOT_NULL(test, overlong_fetched);\n+\tfor (i = 0; i \u003c overlong_nr; i++)\n+\t\toverlong_entries[i] = 0x800000UL + i * 0x1000UL;\n+\n+\tplain_handle = stack_depot_save(entries, ARRAY_SIZE(entries), GFP_KERNEL);\n+\tKUNIT_ASSERT_NE(test, plain_handle, (depot_stack_handle_t)0);\n+\tagain = stack_depot_save(entries, ARRAY_SIZE(entries), GFP_KERNEL);\n+\tKUNIT_EXPECT_EQ(test, again, plain_handle);\n+\n+\tnr_entries = stack_depot_fetch_into(plain_handle, fetched,\n+\t\t\t\t\t ARRAY_SIZE(fetched));\n+\tKUNIT_EXPECT_EQ(test, nr_entries, (unsigned int)ARRAY_SIZE(entries));\n+\tKUNIT_EXPECT_MEMEQ(test, fetched, entries, sizeof(entries));\n+\n+\tnoalloc_handle = stack_depot_save_flags(entries, ARRAY_SIZE(entries), no_spin, 0);\n+\tKUNIT_EXPECT_EQ(test, noalloc_handle, plain_handle);\n+\tif (expected_trie_pool_limit \u003e= 0) {\n+\t\tnoalloc_handle =\n+\t\t\tstack_depot_save_flags(missing_entries,\n+\t\t\t\t\t ARRAY_SIZE(missing_entries),\n+\t\t\t\t\t no_spin, 0);\n+\t\tKUNIT_EXPECT_EQ(test, noalloc_handle, (depot_stack_handle_t)0);\n+\t}\n+\n+\tget_handle = stack_depot_save_flags(get_entries, ARRAY_SIZE(get_entries),\n+\t\t\t\t\t GFP_KERNEL,\n+\t\t\t\t\t STACK_DEPOT_FLAG_CAN_ALLOC |\n+\t\t\t\t\t STACK_DEPOT_FLAG_GET);\n+\tKUNIT_ASSERT_NE(test, get_handle, (depot_stack_handle_t)0);\n+\tstack_depot_put(get_handle);\n+\n+\toverlong_handle = stack_depot_save(overlong_entries, overlong_nr,\n+\t\t\t\t\t GFP_KERNEL);\n+\tKUNIT_ASSERT_NE(test, overlong_handle, (depot_stack_handle_t)0);\n+\tnr_entries = stack_depot_fetch_into(overlong_handle, overlong_fetched,\n+\t\t\t\t\t CONFIG_STACKDEPOT_MAX_FRAMES);\n+\tKUNIT_EXPECT_EQ(test, nr_entries, (unsigned int)CONFIG_STACKDEPOT_MAX_FRAMES);\n+\toverlong_size = CONFIG_STACKDEPOT_MAX_FRAMES * sizeof(*overlong_entries);\n+\tKUNIT_EXPECT_MEMEQ(test, overlong_fetched, overlong_entries, overlong_size);\n+\n+\textra = stack_depot_set_extra_bits(plain_handle, 7);\n+\tKUNIT_ASSERT_NE(test, extra, (depot_stack_handle_t)0);\n+\tKUNIT_EXPECT_EQ(test, stack_depot_get_extra_bits(extra), 7U);\n+\tmemset(fetched, 0, sizeof(fetched));\n+\tnr_entries = stack_depot_fetch_into(extra, fetched, ARRAY_SIZE(fetched));\n+\tKUNIT_EXPECT_EQ(test, nr_entries, (unsigned int)ARRAY_SIZE(entries));\n+\tKUNIT_EXPECT_MEMEQ(test, fetched, entries, sizeof(entries));\n+}\n+\n+static void stackdepot_snprint_public(struct kunit *test)\n+{\n+\tunsigned long entries[] = { 0x1000UL, 0x2000UL, 0x3000UL };\n+\tchar expected[256];\n+\tchar actual[256];\n+\tdepot_stack_handle_t handle;\n+\tunsigned int expected_len;\n+\tint actual_len;\n+\n+\tKUNIT_ASSERT_EQ(test, stack_depot_init(), 0);\n+\thandle = stack_depot_save(entries, ARRAY_SIZE(entries), GFP_KERNEL);\n+\tKUNIT_ASSERT_NE(test, handle, (depot_stack_handle_t)0);\n+\n+\texpected_len = stack_trace_snprint(expected, sizeof(expected), entries,\n+\t\t\t\t\t ARRAY_SIZE(entries), 2);\n+\tactual_len = stack_depot_snprint(handle, actual, sizeof(actual), 2);\n+\tKUNIT_EXPECT_EQ(test, actual_len, (int)expected_len);\n+\tKUNIT_EXPECT_STREQ(test, actual, expected);\n+}\n+\n+static void stackdepot_countable_public(struct kunit *test)\n+{\n+\tunsigned long plain_entries[] = {\n+\t\t0x141000UL,\n+\t\t0x142000UL,\n+\t\t0x143000UL,\n+\t};\n+\tunsigned long get_entries[] = {\n+\t\t0x151000UL,\n+\t\t0x152000UL,\n+\t\t0x153000UL,\n+\t};\n+\tunsigned long fetched[ARRAY_SIZE(plain_entries)] = {};\n+\tdepot_flags_t countable = STACK_DEPOT_FLAG_CAN_ALLOC |\n+\t\t\t\t STACK_DEPOT_FLAG_COUNTABLE;\n+\tstruct stack_record *record;\n+\tdepot_stack_handle_t count_handle;\n+\tdepot_stack_handle_t plain_handle;\n+\tdepot_stack_handle_t get_handle;\n+\tunsigned int get_nr = ARRAY_SIZE(get_entries);\n+\tunsigned int plain_nr = ARRAY_SIZE(plain_entries);\n+\tunsigned int nr_entries;\n+\n+\tKUNIT_ASSERT_EQ(test, stack_depot_init(), 0);\n+\n+\tplain_handle = stack_depot_save(plain_entries, plain_nr, GFP_KERNEL);\n+\tKUNIT_ASSERT_NE(test, plain_handle, (depot_stack_handle_t)0);\n+\tcount_handle = stack_depot_save_flags(plain_entries, plain_nr, GFP_KERNEL,\n+\t\t\t\t\t countable);\n+\tKUNIT_ASSERT_NE(test, count_handle, (depot_stack_handle_t)0);\n+\trecord = __stack_depot_get_stack_record(count_handle);\n+\tKUNIT_ASSERT_NOT_NULL(test, record);\n+\tKUNIT_EXPECT_EQ(test, record-\u003esize, (u16)plain_nr);\n+\tKUNIT_EXPECT_MEMEQ(test, record-\u003eentries, plain_entries,\n+\t\t\t sizeof(plain_entries));\n+\tnr_entries = stack_depot_fetch_into(count_handle, fetched,\n+\t\t\t\t\t ARRAY_SIZE(fetched));\n+\tKUNIT_EXPECT_EQ(test, nr_entries, plain_nr);\n+\tKUNIT_EXPECT_MEMEQ(test, fetched, plain_entries, sizeof(plain_entries));\n+\n+\tget_handle = stack_depot_save_flags(get_entries, get_nr, GFP_KERNEL,\n+\t\t\t\t\t STACK_DEPOT_FLAG_CAN_ALLOC |\n+\t\t\t\t\t STACK_DEPOT_FLAG_GET);\n+\tKUNIT_ASSERT_NE(test, get_handle, (depot_stack_handle_t)0);\n+\tcount_handle = stack_depot_save_flags(get_entries, get_nr, GFP_KERNEL,\n+\t\t\t\t\t countable);\n+\tKUNIT_ASSERT_NE(test, count_handle, (depot_stack_handle_t)0);\n+\trecord = __stack_depot_get_stack_record(count_handle);\n+\tKUNIT_ASSERT_NOT_NULL(test, record);\n+\tKUNIT_EXPECT_MEMEQ(test, record-\u003eentries, get_entries, sizeof(get_entries));\n+\n+\tstack_depot_put(get_handle);\n+}\n+\n+static void stackdepot_fetch_into_roundtrip(struct kunit *test)\n+{\n+\tunsigned long entries[] = {\n+\t\t0x101000UL,\n+\t\t0x102000UL,\n+\t\t0x103000UL,\n+\t};\n+\tunsigned long exact[ARRAY_SIZE(entries)] = {};\n+\tunsigned long fetched[ARRAY_SIZE(entries) + 1] = {\n+\t\t[ARRAY_SIZE(entries)] = 0xa5a5a5a5UL,\n+\t};\n+\tunsigned long expected_tail = fetched[ARRAY_SIZE(entries)];\n+\tdepot_stack_handle_t handle;\n+\tunsigned int nr_entries;\n+\n+\tKUNIT_ASSERT_EQ(test, stack_depot_init(), 0);\n+\n+\thandle = stack_depot_save(entries, ARRAY_SIZE(entries), GFP_KERNEL);\n+\tKUNIT_ASSERT_NE(test, handle, (depot_stack_handle_t)0);\n+\n+\tnr_entries = stack_depot_fetch_into(handle, exact, ARRAY_SIZE(exact));\n+\tKUNIT_EXPECT_EQ(test, nr_entries, (unsigned int)ARRAY_SIZE(entries));\n+\tKUNIT_EXPECT_MEMEQ(test, exact, entries, sizeof(entries));\n+\n+\tnr_entries = stack_depot_fetch_into(handle, fetched, ARRAY_SIZE(fetched));\n+\tKUNIT_EXPECT_EQ(test, nr_entries, (unsigned int)ARRAY_SIZE(entries));\n+\tKUNIT_EXPECT_MEMEQ(test, fetched, entries, sizeof(entries));\n+\tKUNIT_EXPECT_EQ(test, fetched[ARRAY_SIZE(entries)], expected_tail);\n+}\n+\n+static void stackdepot_fetch_into_rejects_missing_or_short_stack(struct kunit *test)\n+{\n+\tunsigned long entries[] = {\n+\t\t0x111000UL,\n+\t\t0x112000UL,\n+\t\t0x113000UL,\n+\t};\n+\tunsigned long fetched[ARRAY_SIZE(entries)] = {\n+\t\t0xa1a1a1a1UL,\n+\t\t0xb2b2b2b2UL,\n+\t\t0xc3c3c3c3UL,\n+\t};\n+\tunsigned long expected[ARRAY_SIZE(fetched)];\n+\tdepot_stack_handle_t handle;\n+\tunsigned int nr_entries;\n+\n+\tKUNIT_ASSERT_EQ(test, stack_depot_init(), 0);\n+\n+\thandle = stack_depot_save(entries, ARRAY_SIZE(entries), GFP_KERNEL);\n+\tKUNIT_ASSERT_NE(test, handle, (depot_stack_handle_t)0);\n+\tmemcpy(expected, fetched, sizeof(expected));\n+\n+\tnr_entries = stack_depot_fetch_into(0, fetched, ARRAY_SIZE(fetched));\n+\tKUNIT_EXPECT_EQ(test, nr_entries, 0U);\n+\tKUNIT_EXPECT_MEMEQ(test, fetched, expected, sizeof(expected));\n+\n+\tnr_entries = stack_depot_fetch_into(0, NULL, 0);\n+\tKUNIT_EXPECT_EQ(test, nr_entries, 0U);\n+\n+\tnr_entries = stack_depot_fetch_into(handle, fetched,\n+\t\t\t\t\t ARRAY_SIZE(fetched) - 1);\n+\tKUNIT_EXPECT_EQ(test, nr_entries, 0U);\n+\tKUNIT_EXPECT_MEMEQ(test, fetched, expected, sizeof(expected));\n+}\n+\n+static void stackdepot_trie_topology_roundtrip(struct kunit *test)\n+{\n+\tunion handle_parts parts;\n+\tunsigned long stacks[][3] = {\n+\t\t{ 0x201000UL, 0x202000UL },\n+\t\t{ 0x201000UL, 0x203000UL },\n+\t\t{ 0x201000UL },\n+\t\t{ 0x201000UL, 0x203000UL, 0x204000UL },\n+\t\t{ 0x201000UL, 0x205000UL },\n+\t\t{ 0x201000UL, 0x204000UL },\n+\t\t{ 0x201000UL, 0x206000UL },\n+\t\t{ 0x201000UL, 0x207000UL },\n+\t\t{ 0x301000UL, 0x302000UL },\n+\t\t{ 0x301000UL, 0x302000UL, 0x303000UL },\n+\t\t{ 0x301000UL, 0x304000UL },\n+\t\t{ 0x401000UL, 0x402000UL, 0x403000UL },\n+\t\t{ 0x401000UL, 0x402000UL },\n+\t};\n+\tunsigned int nr_entries[] = { 2, 2, 1, 3, 2, 2, 2, 2, 2, 3, 2, 3, 2 };\n+\tdepot_stack_handle_t handles[ARRAY_SIZE(stacks)];\n+\tunsigned long fetched[ARRAY_SIZE(stacks[0])];\n+\tu32 pool_index_plus_1;\n+\tunsigned int i;\n+\n+\tif (expected_trie_pool_limit \u003c 0)\n+\t\tkunit_skip(test, \"trie pool limit was not provided\");\n+\tKUNIT_ASSERT_EQ(test, stack_depot_init(), 0);\n+\n+\tfor (i = 0; i \u003c ARRAY_SIZE(stacks); i++) {\n+\t\thandles[i] = stack_depot_save(stacks[i], nr_entries[i], GFP_KERNEL);\n+\t\tKUNIT_ASSERT_NE(test, handles[i], (depot_stack_handle_t)0);\n+\t}\n+\tparts.handle = handles[0];\n+\tpool_index_plus_1 = parts.pool_index_plus_1;\n+\tKUNIT_ASSERT_GT(test, pool_index_plus_1,\n+\t\t\t(u32)expected_trie_pool_limit);\n+\n+\tfor (i = 0; i \u003c ARRAY_SIZE(stacks); i++) {\n+\t\tmemset(fetched, 0, sizeof(fetched));\n+\t\tKUNIT_EXPECT_EQ(test,\n+\t\t\t\tstack_depot_fetch_into(handles[i], fetched,\n+\t\t\t\t\t\t ARRAY_SIZE(fetched)),\n+\t\t\t\tnr_entries[i]);\n+\t\tKUNIT_EXPECT_MEMEQ(test, fetched, stacks[i],\n+\t\t\t\t nr_entries[i] * sizeof(fetched[0]));\n+\t\tKUNIT_EXPECT_EQ(test,\n+\t\t\t\tstack_depot_save(stacks[i], nr_entries[i], GFP_KERNEL),\n+\t\t\t\thandles[i]);\n+\t}\n+}\n+\n+static void stackdepot_frame_storage_roundtrip(struct kunit *test)\n+{\n+\tunion handle_parts parts;\n+\tunsigned long fetched[3] = {};\n+\tdepot_stack_handle_t handle;\n+\tu32 pool_index_plus_1;\n+\tunsigned int nr_entries;\n+#if defined(CONFIG_ARM64)\n+\tunsigned long entries[] = {\n+\t\tstackdepot_arm64_frame(S32_MIN),\n+\t\t0x1000UL,\n+\t\tstackdepot_arm64_frame(S32_MAX),\n+\t};\n+#elif defined(CONFIG_X86_64)\n+\tunsigned long entries[] = {\n+\t\t0xffffffff10001000UL,\n+\t\t0xffff888000001000UL,\n+\t\t0xffffffff20002000UL,\n+\t};\n+#else\n+\tunsigned long entries[] = { 0x301000UL, 0x302000UL, 0x303000UL };\n+#endif\n+\n+\tif (expected_trie_pool_limit \u003c 0)\n+\t\tkunit_skip(test, \"trie pool limit was not provided\");\n+\tKUNIT_ASSERT_EQ(test, stack_depot_init(), 0);\n+\thandle = stack_depot_save(entries, ARRAY_SIZE(entries), GFP_KERNEL);\n+\tKUNIT_ASSERT_NE(test, handle, (depot_stack_handle_t)0);\n+\tparts.handle = handle;\n+\tpool_index_plus_1 = parts.pool_index_plus_1;\n+\tKUNIT_ASSERT_GT(test, pool_index_plus_1,\n+\t\t\t(u32)expected_trie_pool_limit);\n+\n+\tnr_entries = stack_depot_fetch_into(handle, fetched, ARRAY_SIZE(fetched));\n+\tKUNIT_EXPECT_EQ(test, nr_entries, (unsigned int)ARRAY_SIZE(entries));\n+\tKUNIT_EXPECT_MEMEQ(test, fetched, entries, sizeof(entries));\n+}\n+\n+static void stackdepot_frame_raw_fallback(struct kunit *test)\n+{\n+\tunsigned long frame = 0x1000UL;\n+\tbool compressed;\n+\tu32 payload;\n+\n+#ifdef CONFIG_ARM64\n+\tframe = (unsigned long)_text + (unsigned long)S32_MAX + 1UL;\n+#endif\n+\n+\tcompressed = arch_stack_depot_frame_try_compress(frame, \u0026payload);\n+\tKUNIT_EXPECT_FALSE(test, compressed);\n+}\n+\n+#if defined(CONFIG_X86_64) \u0026\u0026 !defined(CONFIG_UML)\n+static void stackdepot_frame_x86_64(struct kunit *test)\n+{\n+\tunsigned long direct_map = 0xffff888000001000UL;\n+\tunsigned long frame = 0xffffffff81234567UL;\n+\tunsigned long out;\n+\tbool compressed;\n+\tu32 low;\n+\n+\tcompressed = arch_stack_depot_frame_try_compress(frame, \u0026low);\n+\tKUNIT_EXPECT_TRUE(test, compressed);\n+\tKUNIT_EXPECT_EQ(test, low, (u32)0x81234567);\n+\tarch_stack_depot_frame_decompress(low, \u0026out);\n+\tKUNIT_EXPECT_EQ(test, out, frame);\n+\n+\tcompressed = arch_stack_depot_frame_try_compress(direct_map, \u0026low);\n+\tKUNIT_EXPECT_FALSE(test, compressed);\n+}\n+#endif /* CONFIG_X86_64 \u0026\u0026 !CONFIG_UML */\n+\n+#ifdef CONFIG_ARM64\n+static void stackdepot_frame_arm64(struct kunit *test)\n+{\n+\tlong negative_offset = S32_MIN;\n+\tlong positive_offset = S32_MAX;\n+\tlong offset = 0x123456;\n+\tunsigned long frame = stackdepot_arm64_frame(offset);\n+\tunsigned long out;\n+\tbool compressed;\n+\tu32 payload;\n+\n+\tcompressed = arch_stack_depot_frame_try_compress(frame, \u0026payload);\n+\tKUNIT_EXPECT_TRUE(test, compressed);\n+\tKUNIT_EXPECT_EQ(test, payload, (u32)(s32)offset);\n+\tarch_stack_depot_frame_decompress(payload, \u0026out);\n+\tKUNIT_EXPECT_EQ(test, out, frame);\n+\n+\tframe = stackdepot_arm64_frame(negative_offset);\n+\tcompressed = arch_stack_depot_frame_try_compress(frame, \u0026payload);\n+\tKUNIT_EXPECT_TRUE(test, compressed);\n+\tKUNIT_EXPECT_EQ(test, payload, (u32)(s32)negative_offset);\n+\tarch_stack_depot_frame_decompress(payload, \u0026out);\n+\tKUNIT_EXPECT_EQ(test, out, frame);\n+\n+\tframe = stackdepot_arm64_frame(positive_offset);\n+\tcompressed = arch_stack_depot_frame_try_compress(frame, \u0026payload);\n+\tKUNIT_EXPECT_TRUE(test, compressed);\n+\tKUNIT_EXPECT_EQ(test, payload, (u32)(s32)positive_offset);\n+\tarch_stack_depot_frame_decompress(payload, \u0026out);\n+\tKUNIT_EXPECT_EQ(test, out, frame);\n+}\n+#endif /* CONFIG_ARM64 */\n+\n+static struct kunit_case stackdepot_test_cases[] = {\n+\tKUNIT_CASE(stackdepot_trie_max_path_roundtrip),\n+\tKUNIT_CASE(stackdepot_save_flags_public),\n+\tKUNIT_CASE(stackdepot_snprint_public),\n+\tKUNIT_CASE(stackdepot_countable_public),\n+\tKUNIT_CASE(stackdepot_fetch_into_roundtrip),\n+\tKUNIT_CASE(stackdepot_fetch_into_rejects_missing_or_short_stack),\n+\tKUNIT_CASE(stackdepot_trie_topology_roundtrip),\n+\tKUNIT_CASE(stackdepot_frame_storage_roundtrip),\n+\tKUNIT_CASE(stackdepot_frame_raw_fallback),\n+#if defined(CONFIG_X86_64) \u0026\u0026 !defined(CONFIG_UML)\n+\tKUNIT_CASE(stackdepot_frame_x86_64),\n+#endif\n+#ifdef CONFIG_ARM64\n+\tKUNIT_CASE(stackdepot_frame_arm64),\n+#endif\n+\t{}\n+};\n+\n+static struct kunit_suite stackdepot_test_suite = {\n+\t.name = \"stackdepot\",\n+\t.test_cases = stackdepot_test_cases,\n+};\n+\n+kunit_test_suite(stackdepot_test_suite);\n+\n+MODULE_DESCRIPTION(\"KUnit tests for stack depot\");\n+MODULE_AUTHOR(\"Caleb Kan \u003cckan@cloudflare.com\u003e\");\n+MODULE_LICENSE(\"GPL\");\ndiff --git a/mm/kmemleak.c b/mm/kmemleak.c\nindex 8fa409a4f9fb2..c42741a88bd42 100644\n--- a/mm/kmemleak.c\n+++ b/mm/kmemleak.c\n@@ -378,10 +378,10 @@ static void __print_unreferenced(struct seq_file *seq,\n \t\t\t\t bool hex_dump)\n {\n \tint i;\n-\tunsigned long *entries;\n+\tunsigned long entries[MAX_TRACE];\n \tunsigned int nr_entries;\n \n-\tnr_entries = stack_depot_fetch(object-\u003etrace_handle, \u0026entries);\n+\tnr_entries = stack_depot_fetch_into(object-\u003etrace_handle, entries, ARRAY_SIZE(entries));\n \twarn_or_seq_printf(seq, \"unreferenced object%s 0x%08lx (size %zu):\\n\",\n \t\t\t __object_type_str(object),\n \t\t\t object-\u003epointer, object-\u003esize);\ndiff --git a/mm/kmsan/kmsan_test.c b/mm/kmsan/kmsan_test.c\nindex 31f47cc4dab40..7c04e4b21873d 100644\n--- a/mm/kmsan/kmsan_test.c\n+++ b/mm/kmsan/kmsan_test.c\n@@ -669,7 +669,7 @@ static void test_long_origin_chain(struct kunit *test)\n */\n static void test_stackdepot_roundtrip(struct kunit *test)\n {\n-\tunsigned long src_entries[16], *dst_entries;\n+\tunsigned long src_entries[16], dst_entries[16];\n \tunsigned int src_nentries, dst_nentries;\n \tEXPECTATION_NO_REPORT(expect);\n \tdepot_stack_handle_t handle;\n@@ -680,7 +680,7 @@ static void test_stackdepot_roundtrip(struct kunit *test)\n \t\tstack_trace_save(src_entries, ARRAY_SIZE(src_entries), 1);\n \thandle = stack_depot_save(src_entries, src_nentries, GFP_KERNEL);\n \tstack_depot_print(handle);\n-\tdst_nentries = stack_depot_fetch(handle, \u0026dst_entries);\n+\tdst_nentries = stack_depot_fetch_into(handle, dst_entries, ARRAY_SIZE(dst_entries));\n \tKUNIT_EXPECT_TRUE(test, src_nentries == dst_nentries);\n \n \tkmsan_check_memory((void *)dst_entries,\ndiff --git a/mm/kmsan/report.c b/mm/kmsan/report.c\nindex d6853ce089541..c20c24cffde55 100644\n--- a/mm/kmsan/report.c\n+++ b/mm/kmsan/report.c\n@@ -85,7 +85,7 @@ static char *pretty_descr(char *descr)\n \n void kmsan_print_origin(depot_stack_handle_t origin)\n {\n-\tunsigned long *entries = NULL, *chained_entries = NULL;\n+\tunsigned long entries[KMSAN_STACK_DEPTH];\n \tunsigned int nr_entries, chained_nr_entries, skipnr;\n \tvoid *pc1 = NULL, *pc2 = NULL;\n \tdepot_stack_handle_t head;\n@@ -97,7 +97,8 @@ void kmsan_print_origin(depot_stack_handle_t origin)\n \t\treturn;\n \n \twhile (true) {\n-\t\tnr_entries = stack_depot_fetch(origin, \u0026entries);\n+\t\tnr_entries =\n+\t\t\tstack_depot_fetch_into(origin, entries, ARRAY_SIZE(entries));\n \t\tdepth = kmsan_depth_from_eb(stack_depot_get_extra_bits(origin));\n \t\tmagic = nr_entries ? entries[0] : 0;\n \t\tif ((nr_entries == 4) \u0026\u0026 (magic == KMSAN_ALLOCA_MAGIC_ORIGIN)) {\n@@ -123,14 +124,10 @@ void kmsan_print_origin(depot_stack_handle_t origin)\n \t\t\torigin = entries[2];\n \t\t\tpr_err(\"Uninit was stored to memory at:\\n\");\n \t\t\tchained_nr_entries =\n-\t\t\t\tstack_depot_fetch(head, \u0026chained_entries);\n-\t\t\tkmsan_internal_unpoison_memory(\n-\t\t\t\tchained_entries,\n-\t\t\t\tchained_nr_entries * sizeof(*chained_entries),\n-\t\t\t\t/*checked*/ false);\n-\t\t\tskipnr = get_stack_skipnr(chained_entries,\n-\t\t\t\t\t\t chained_nr_entries);\n-\t\t\tstack_trace_print(chained_entries + skipnr,\n+\t\t\t\tstack_depot_fetch_into(head, entries,\n+\t\t\t\t\t\t ARRAY_SIZE(entries));\n+\t\t\tskipnr = get_stack_skipnr(entries, chained_nr_entries);\n+\t\t\tstack_trace_print(entries + skipnr,\n \t\t\t\t\t chained_nr_entries - skipnr, 0);\n \t\t\tpr_err(\"\\n\");\n \t\t\tcontinue;\ndiff --git a/mm/page_owner.c b/mm/page_owner.c\nindex fbbda7ba914ba..af37532729b02 100644\n--- a/mm/page_owner.c\n+++ b/mm/page_owner.c\n@@ -119,7 +119,8 @@ static __always_inline depot_stack_handle_t create_dummy_stack(void)\n \tunsigned int nr_entries;\n \n \tnr_entries = stack_trace_save(entries, ARRAY_SIZE(entries), 0);\n-\treturn stack_depot_save(entries, nr_entries, GFP_KERNEL);\n+\treturn stack_depot_save_flags(entries, nr_entries, GFP_KERNEL,\n+\t\t\t\t STACK_DEPOT_FLAG_CAN_ALLOC | STACK_DEPOT_FLAG_COUNTABLE);\n }\n \n static noinline void register_dummy_stack(void)\n@@ -181,7 +182,8 @@ static noinline depot_stack_handle_t save_stack(gfp_t flags)\n \n \tset_current_in_page_owner();\n \tnr_entries = stack_trace_save(entries, ARRAY_SIZE(entries), 2);\n-\thandle = stack_depot_save(entries, nr_entries, flags);\n+\thandle = stack_depot_save_flags(entries, nr_entries, flags,\n+\t\t\t\t\tSTACK_DEPOT_FLAG_CAN_ALLOC | STACK_DEPOT_FLAG_COUNTABLE);\n \tif (!handle)\n \t\thandle = failure_handle;\n \tunset_current_in_page_owner();\ndiff --git a/mm/slub.c b/mm/slub.c\nindex 422bc3e12c02c..138c3bc473c9c 100644\n--- a/mm/slub.c\n+++ b/mm/slub.c\n@@ -8093,12 +8093,12 @@ void __kmem_obj_info(struct kmem_obj_info *kpp, void *object, struct slab *slab)\n #ifdef CONFIG_STACKDEPOT\n \t{\n \t\tdepot_stack_handle_t handle;\n-\t\tunsigned long *entries;\n+\t\tunsigned long entries[TRACK_ADDRS_COUNT];\n \t\tunsigned int nr_entries;\n \n \t\thandle = READ_ONCE(trackp-\u003ehandle);\n \t\tif (handle) {\n-\t\t\tnr_entries = stack_depot_fetch(handle, \u0026entries);\n+\t\t\tnr_entries = stack_depot_fetch_into(handle, entries, ARRAY_SIZE(entries));\n \t\t\tfor (i = 0; i \u003c KS_ADDRS_COUNT \u0026\u0026 i \u003c nr_entries; i++)\n \t\t\t\tkpp-\u003ekp_stack[i] = (void *)entries[i];\n \t\t}\n@@ -8106,7 +8106,7 @@ void __kmem_obj_info(struct kmem_obj_info *kpp, void *object, struct slab *slab)\n \t\ttrackp = get_track(s, objp, TRACK_FREE);\n \t\thandle = READ_ONCE(trackp-\u003ehandle);\n \t\tif (handle) {\n-\t\t\tnr_entries = stack_depot_fetch(handle, \u0026entries);\n+\t\t\tnr_entries = stack_depot_fetch_into(handle, entries, ARRAY_SIZE(entries));\n \t\t\tfor (i = 0; i \u003c KS_ADDRS_COUNT \u0026\u0026 i \u003c nr_entries; i++)\n \t\t\t\tkpp-\u003ekp_free_stack[i] = (void *)entries[i];\n \t\t}\n@@ -9815,12 +9815,14 @@ static int slab_debugfs_show(struct seq_file *seq, void *v)\n #ifdef CONFIG_STACKDEPOT\n \t\t{\n \t\t\tdepot_stack_handle_t handle;\n-\t\t\tunsigned long *entries;\n+\t\t\tunsigned long entries[TRACK_ADDRS_COUNT];\n \t\t\tunsigned int nr_entries, j;\n \n \t\t\thandle = READ_ONCE(l-\u003ehandle);\n \t\t\tif (handle) {\n-\t\t\t\tnr_entries = stack_depot_fetch(handle, \u0026entries);\n+\t\t\t\tnr_entries =\n+\t\t\t\t\tstack_depot_fetch_into(handle, entries,\n+\t\t\t\t\t\t\t ARRAY_SIZE(entries));\n \t\t\t\tseq_puts(seq, \"\\n\");\n \t\t\t\tfor (j = 0; j \u003c nr_entries; j++)\n \t\t\t\t\tseq_printf(seq, \" %pS\\n\", (void *)entries[j]);\ndiff --git a/scripts/gdb/linux/stackdepot.py b/scripts/gdb/linux/stackdepot.py\nindex 37313a5a51a0d..82aeb9f532c3d 100644\n--- a/scripts/gdb/linux/stackdepot.py\n+++ b/scripts/gdb/linux/stackdepot.py\n@@ -37,6 +37,10 @@ def stack_depot_fetch(handle):\n if handle == 0:\n raise gdb.GdbError(\"handle is 0\\n\")\n \n+ stack_max_pools = gdb.parse_and_eval('stack_max_pools')\n+ if parts['pool_index_plus_1'] \u003e stack_max_pools:\n+ raise gdb.GdbError(\"trie-backed stack depot handles are not supported\\n\")\n+\n pool_index = parts['pool_index_plus_1'] - 1\n if pool_index \u003e= pools_num:\n gdb.write(\"pool index %d out of bounds (%d) for stack id 0x%08x\\n\" % (parts['pool_index'], pools_num, handle))\n"
} |
| 2/1 | 2026/08/17 12:57 | agent | patch-evaluator |
0mModel:best-expensive Args: null Results: {
"EnableConfigs": [
"PAGE_OWNER",
"DEBUG_KMEMLEAK",
"KUNIT",
"STACKDEPOT_KUNIT_TEST"
],
"FocusSymbols": [
"stack_depot_fetch_into",
"__stack_depot_get_stack_record",
"stack_depot_snprint",
"stack_depot_print"
],
"Reasoning": "The patch introduces a new trie-based storage backend for the stack depot, a core kernel library used by memory debugging and tracking tools (e.g., KASAN, KMSAN, page_owner, kmemleak). It adds a new API `stack_depot_fetch_into` and modifies existing APIs to support the new backend and a new `STACK_DEPOT_FLAG_COUNTABLE` flag. These changes significantly alter the internal logic of stack trace storage and retrieval, making it worth fuzzing to ensure memory safety and correctness.",
"WorthFuzzing": true
}Instruction: You are an expert Linux kernel maintainer. Your job is to review a provided patch series and determine if it makes functional changes to the kernel that should be fuzzed. IMPORTANT: The changes have ALREADY been applied and committed as the HEAD commit in your workspace. Do NOT rely on your internal knowledge of the kernel. You must actively use your code access tools to examine the actual source code and confirm any assumptions. Return WorthFuzzing=false if the patch only contains: - Modifications to Documentation/, Kconfig files, or code comments. - Purely decorative changes, such as logging (e.g., pr_err, printk) or tracepoints. - Changes to numeric constants or macros that do not functionally alter execution flow. - Code paths that are impossible to reach in virtualized environments like GCE or QEMU, even when utilizing software-emulated hardware (e.g., usb gadget, mac80211_hwsim). - Code in vendor-specific PCIe switch, SmartNIC, or GPU drivers (e.g., mlxsw, pds_core, qed, ionic, amdgpu) that require physical PCIe hardware cards not emulated in standard QEMU. - Driver .remove, .shutdown, or pci_unregister_driver teardown callbacks (e.g., igb_remove) that are executed only during PCI hot-unplug or sysfs driver unbind operations. If it modifies reachable core kernel logic, drivers, or architectures, use your code search tools to verify the code can be executed, then return WorthFuzzing=true. When returning WorthFuzzing=true, you MUST ALSO: 1. Extract any specific kernel functions that should be heavily fuzzed into FocusSymbols. Avoid listing generic hot-path functions to prevent skewed test distributions. Prefer non-static, non-inlined API entrypoint functions over internal static helper functions (which are inlined by the compiler and do not have distinct symbol addresses). 2. Identify any specific CONFIG_ options required to properly test this new/modified feature. Go and look into the Kconfig files and check for ifdefs around the code, do not make assumptions. Also check "depends on" lines in Kconfig to include any non-standard parent subsystem configs needed for Kbuild to compile the code statically into vmlinux. List them in the EnableConfigs output array, and DO NOT add a 'CONFIG_' prefix (e.g., return "NET_IPV4" instead of "CONFIG_NET_IPV4"). Prefer calling several tools at the same time to save round-trips. Use set-results tool to provide results of the analysis. It must be called exactly once before the final reply. Ignore results of this tool. Prompt: For your convenience, here is the diff of the changes:
commit 1d27bb5b64bebf701b469bdea8a91ce99a84492b
Author: syz-cluster <triage@syzkaller.com>
Date: Mon Aug 17 12:57:47 2026 +0000
syz-cluster: applied patch under review
diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 1af62cd16c9de..ebb7b7e1867f6 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -7387,6 +7387,13 @@ Kernel parameters
stack traces. Pools are allocated on-demand up to this
limit. Default value is 8191 pools.
+ stackdepot.trie_enabled= [KNL]
+ Format: <bool>
+ Enable trie storage for persistent, non-refcounted
+ stack depot records at boot. Disabled by default.
+ stack_depot_max_pools must leave unused pool-index
+ values for trie handles.
+
stacktrace [FTRACE]
Enable the stack tracer on boot up.
diff --git a/arch/arm64/include/asm/stackdepot.h b/arch/arm64/include/asm/stackdepot.h
new file mode 100644
index 0000000000000..df8959d593366
--- /dev/null
+++ b/arch/arm64/include/asm/stackdepot.h
@@ -0,0 +1,42 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __ASM_STACKDEPOT_H
+#define __ASM_STACKDEPOT_H
+
+#include <linux/types.h>
+#include <asm/sections.h>
+
+/*
+ * Modules are allocated inside a 2 GB relocation window containing the
+ * kernel image. Store a signed 32-bit offset from _text so compression is
+ * independent of 4 GB high-bit boundaries crossed by that window.
+ */
+static inline unsigned long arch_stack_depot_frame_from_payload(u32 payload)
+{
+ long offset;
+
+ offset = (s32)payload;
+ if (offset < 0)
+ return (unsigned long)_text - (unsigned long)(-offset);
+ return (unsigned long)_text + (unsigned long)offset;
+}
+
+static inline bool
+arch_stack_depot_frame_try_compress(unsigned long frame, u32 *payload)
+{
+ u32 candidate;
+
+ candidate = (u32)(frame - (unsigned long)_text);
+ if (arch_stack_depot_frame_from_payload(candidate) != frame)
+ return false;
+
+ *payload = candidate;
+ return true;
+}
+
+static inline void
+arch_stack_depot_frame_decompress(u32 payload, unsigned long *frame)
+{
+ *frame = arch_stack_depot_frame_from_payload(payload);
+}
+
+#endif /* __ASM_STACKDEPOT_H */
diff --git a/arch/um/include/asm/Kbuild b/arch/um/include/asm/Kbuild
index 8fdc0bd9ab6fb..14778d2457d79 100644
--- a/arch/um/include/asm/Kbuild
+++ b/arch/um/include/asm/Kbuild
@@ -21,6 +21,7 @@ generic-y += preempt.h
generic-y += ring_buffer.h
generic-y += runtime-const.h
generic-y += softirq_stack.h
+generic-y += stackdepot.h
generic-y += switch_to.h
generic-y += topology.h
generic-y += trace_clock.h
diff --git a/arch/x86/include/asm/stackdepot.h b/arch/x86/include/asm/stackdepot.h
new file mode 100644
index 0000000000000..9a8d04fa8c1c8
--- /dev/null
+++ b/arch/x86/include/asm/stackdepot.h
@@ -0,0 +1,37 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_X86_STACKDEPOT_H
+#define _ASM_X86_STACKDEPOT_H
+
+#include <linux/types.h>
+
+#ifdef CONFIG_X86_64
+/*
+ * Compress canonical kernel text/module addresses whose upper 32 bits are all
+ * ones. Other kernel virtual addresses stay raw, so decompression reconstructs
+ * the original frame by restoring this prefix.
+ */
+#define STACK_DEPOT_X86_64_FRAME_PREFIX 0xffffffff00000000UL
+#define STACK_DEPOT_X86_64_FRAME_LOW_MASK 0x00000000ffffffffUL
+
+static inline bool
+arch_stack_depot_frame_try_compress(unsigned long frame, u32 *low)
+{
+ if ((frame & ~STACK_DEPOT_X86_64_FRAME_LOW_MASK) !=
+ STACK_DEPOT_X86_64_FRAME_PREFIX)
+ return false;
+
+ *low = (u32)frame;
+ return true;
+}
+
+static inline void
+arch_stack_depot_frame_decompress(u32 low, unsigned long *frame)
+{
+ *frame = STACK_DEPOT_X86_64_FRAME_PREFIX | low;
+}
+
+#else
+#include <asm-generic/stackdepot.h>
+#endif /* CONFIG_X86_64 */
+
+#endif /* _ASM_X86_STACKDEPOT_H */
diff --git a/drivers/gpu/drm/drm_modeset_lock.c b/drivers/gpu/drm/drm_modeset_lock.c
index 2c806b0146d67..a2ddb02b2aea5 100644
--- a/drivers/gpu/drm/drm_modeset_lock.c
+++ b/drivers/gpu/drm/drm_modeset_lock.c
@@ -94,16 +94,13 @@ static noinline depot_stack_handle_t __drm_stack_depot_save(void)
static void __drm_stack_depot_print(depot_stack_handle_t stack_depot)
{
struct drm_printer p = drm_dbg_printer(NULL, DRM_UT_KMS, "drm_modeset_lock");
- unsigned long *entries;
- unsigned int nr_entries;
char *buf;
buf = kmalloc(PAGE_SIZE, GFP_NOWAIT | __GFP_NOWARN);
if (!buf)
return;
- nr_entries = stack_depot_fetch(stack_depot, &entries);
- stack_trace_snprint(buf, PAGE_SIZE, entries, nr_entries, 2);
+ stack_depot_snprint(stack_depot, buf, PAGE_SIZE, 2);
drm_printf(&p, "attempting to lock a contended lock without backoff:\n%s", buf);
diff --git a/include/asm-generic/Kbuild b/include/asm-generic/Kbuild
index 15df9dcb42a5b..ac178162fa114 100644
--- a/include/asm-generic/Kbuild
+++ b/include/asm-generic/Kbuild
@@ -55,6 +55,7 @@ mandatory-y += serial.h
mandatory-y += shmparam.h
mandatory-y += simd.h
mandatory-y += softirq_stack.h
+mandatory-y += stackdepot.h
mandatory-y += switch_to.h
mandatory-y += timex.h
mandatory-y += tlbflush.h
diff --git a/include/asm-generic/stackdepot.h b/include/asm-generic/stackdepot.h
new file mode 100644
index 0000000000000..846975767bdd4
--- /dev/null
+++ b/include/asm-generic/stackdepot.h
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __ASM_GENERIC_STACKDEPOT_H
+#define __ASM_GENERIC_STACKDEPOT_H
+
+#include <linux/types.h>
+
+static inline bool
+arch_stack_depot_frame_try_compress(unsigned long frame, u32 *low)
+{
+ return false;
+}
+
+static inline void
+arch_stack_depot_frame_decompress(u32 low, unsigned long *frame)
+{
+ /* Generic code never compresses frames, so this hook is unreachable. */
+}
+
+#endif /* __ASM_GENERIC_STACKDEPOT_H */
diff --git a/include/linux/stackdepot.h b/include/linux/stackdepot.h
index 2cc21ffcdaf9e..788737eb0c4a2 100644
--- a/include/linux/stackdepot.h
+++ b/include/linux/stackdepot.h
@@ -53,7 +53,8 @@ union handle_parts {
struct stack_record {
struct list_head hash_list; /* Links in the hash table */
u32 hash; /* Hash in hash table */
- u32 size; /* Number of stored frames */
+ u16 size; /* Number of stored frames */
+ u16 flags;
union handle_parts handle; /* Constant after initialization */
refcount_t count;
union {
@@ -84,8 +85,9 @@ typedef u32 depot_flags_t;
*/
#define STACK_DEPOT_FLAG_CAN_ALLOC ((depot_flags_t)0x0001)
#define STACK_DEPOT_FLAG_GET ((depot_flags_t)0x0002)
+#define STACK_DEPOT_FLAG_COUNTABLE ((depot_flags_t)0x0004)
-#define STACK_DEPOT_FLAGS_NUM 2
+#define STACK_DEPOT_FLAGS_NUM 3
#define STACK_DEPOT_FLAGS_MASK ((depot_flags_t)((1 << STACK_DEPOT_FLAGS_NUM) - 1))
/*
@@ -144,6 +146,15 @@ static inline int stack_depot_early_init(void) { return 0; }
* Users of this flag must also call stack_depot_put() when keeping the stack
* trace is no longer required to avoid overflowing the refcount.
*
+ * If STACK_DEPOT_FLAG_COUNTABLE is set in @depot_flags, stack depot stores the
+ * stack in hash-backed storage for callers that need direct stack_record count
+ * access. This flag does not imply %STACK_DEPOT_FLAG_CAN_ALLOC and is mutually
+ * exclusive with %STACK_DEPOT_FLAG_GET.
+ *
+ * When trie storage is enabled, persistent non-refcounted saves use trie
+ * storage. Constrained callers only look up existing stacks; they do not insert
+ * a missing stack. Trie failures do not fall back to hash storage.
+ *
* If the provided stack trace comes from the interrupt context, only the part
* up to the interrupt entry is saved.
*
@@ -152,7 +163,7 @@ static inline int stack_depot_early_init(void) { return 0; }
* this is the case for contexts where neither %GFP_ATOMIC nor
* %GFP_NOWAIT can be used (NMI, raw_spin_lock).
*
- * Return: Handle of the stack struct stored in depot, 0 on failure
+ * Return: Handle of the stack trace stored in depot, 0 on failure
*/
depot_stack_handle_t stack_depot_save_flags(unsigned long *entries,
unsigned int nr_entries,
@@ -169,6 +180,10 @@ depot_stack_handle_t stack_depot_save_flags(unsigned long *entries,
* Does not increment the refcount on the saved stack trace; see
* stack_depot_save_flags() for more details.
*
+ * When trie storage is enabled, this can return trie-backed handles. Use
+ * stack_depot_fetch_into(), stack_depot_print(), or stack_depot_snprint() for
+ * backend-independent access to the stack contents.
+ *
* Context: Contexts where allocations via alloc_pages() are allowed;
* see stack_depot_save_flags() for more details.
*
@@ -178,11 +193,12 @@ depot_stack_handle_t stack_depot_save(unsigned long *entries,
unsigned int nr_entries, gfp_t alloc_flags);
/**
- * __stack_depot_get_stack_record - Get a pointer to a stack_record struct
+ * __stack_depot_get_stack_record - Get a hash-backed stack record
*
* @handle: Stack depot handle
*
- * This function is only for internal purposes.
+ * This function is only for internal purposes. @handle must have been saved
+ * with %STACK_DEPOT_FLAG_COUNTABLE.
*
* Return: Returns a pointer to a stack_record struct
*/
@@ -191,14 +207,55 @@ struct stack_record *__stack_depot_get_stack_record(depot_stack_handle_t handle)
/**
* stack_depot_fetch - Fetch a stack trace from stack depot
*
- * @handle: Stack depot handle returned from stack_depot_save()
+ * @handle: Hash-backed stack depot handle
* @entries: Pointer to store the address of the stack trace
*
+ * This helper returns a pointer to stackdepot-owned contiguous storage for
+ * legacy hash-backed handles. Callers that need backend-independent access to
+ * stack contents should use stack_depot_fetch_into(), stack_depot_print(), or
+ * stack_depot_snprint(). Passing a trie-backed handle is invalid and may WARN.
+ *
* Return: Number of frames for the fetched stack
*/
unsigned int stack_depot_fetch(depot_stack_handle_t handle,
unsigned long **entries);
+/**
+ * stack_depot_fetch_into - Fetch a stack trace into caller-owned storage
+ *
+ * @handle: Stack depot handle
+ * @entries: Caller-owned buffer to copy the stack trace into
+ * @max_entries: Number of frames that fit in @entries
+ *
+ * Copies the stored frames into caller-owned @entries. If fewer frames are
+ * stored than @max_entries, only the stored frames are written and their count
+ * is returned. If more frames are stored than @max_entries, the copy is skipped
+ * entirely and 0 is returned.
+ *
+ * Passing a NULL @entries buffer or zero @max_entries for a valid @handle is
+ * invalid. Callers must provide storage for @max_entries frames.
+ *
+ * Callers should size @entries to match the save-side stack depth cap (for
+ * example, %CONFIG_STACKDEPOT_MAX_FRAMES or the local stack_trace_save() limit)
+ * when losing diagnostics on an undersized buffer would be surprising.
+ *
+ * A non-zero invalid @handle, including a post-put handle, may WARN. Its return
+ * value and copied contents are undefined because the record may have been
+ * reused for another stack.
+ *
+ * Callers must ensure @handle remains valid for the duration of this call.
+ * Persistent handles saved without %STACK_DEPOT_FLAG_GET require no extra
+ * reference; handles saved with %STACK_DEPOT_FLAG_GET require a held reference.
+ * Callers must not call stack_depot_put() on persistent handles.
+ * Racing this helper with stack_depot_put() on the same handle is invalid.
+ *
+ * Return: Number of frames copied, 0 if @handle is 0, stack depot is disabled,
+ * or @max_entries is less than the number of stored frames.
+ */
+unsigned int stack_depot_fetch_into(depot_stack_handle_t handle,
+ unsigned long *entries,
+ unsigned int max_entries);
+
/**
* stack_depot_print - Print a stack trace from stack depot
*
@@ -224,10 +281,14 @@ int stack_depot_snprint(depot_stack_handle_t handle, char *buf, size_t size,
*
* @handle: Stack depot handle returned from stack_depot_save()
*
- * The stack trace is evicted from stack depot once all references to it have
- * been dropped (once the number of stack_depot_evict() calls matches the
- * number of stack_depot_save_flags() calls with STACK_DEPOT_FLAG_GET set for
- * this stack trace).
+ * Drop a reference acquired by stack_depot_save_flags() with
+ * %STACK_DEPOT_FLAG_GET. Calling this for a handle saved without
+ * %STACK_DEPOT_FLAG_GET is invalid; persistent handles, including trie-backed
+ * handles, are owned by stack depot for the lifetime of the system.
+ *
+ * The stack trace is evicted once the number of stack_depot_put() calls matches
+ * the number of successful stack_depot_save_flags() calls with
+ * %STACK_DEPOT_FLAG_GET for this stack trace.
*/
void stack_depot_put(depot_stack_handle_t handle);
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index 00921b1676e87..af238949fb7a0 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -2771,6 +2771,23 @@ config RESOURCE_KUNIT_TEST
If unsure, say N.
+config STACKDEPOT_KUNIT_TEST
+ bool "KUnit test for stack depot" if !KUNIT_ALL_TESTS
+ depends on KUNIT=y && STACKDEPOT
+ depends on STACKDEPOT_MAX_FRAMES >= 3
+ default KUNIT_ALL_TESTS
+ help
+ Enable this option to test stack depot API behavior at boot.
+ This test is built in because it exercises internal, non-exported
+ stack depot helpers, so KUNIT must also be built in.
+
+ KUnit tests run during boot and output the results to the debug log
+ in TAP format (https://testanything.org/). Only useful for kernel
+ developers running the KUnit test harness, and not intended for
+ inclusion into a production build.
+
+ If unsure, say N.
+
config SYSCTL_KUNIT_TEST
tristate "KUnit test for sysctl" if !KUNIT_ALL_TESTS
depends on KUNIT
diff --git a/lib/stackdepot.c b/lib/stackdepot.c
index dd2717ff94bff..1a002063a9487 100644
--- a/lib/stackdepot.c
+++ b/lib/stackdepot.c
@@ -2,9 +2,11 @@
/*
* Stack depot - a stack trace storage that avoids duplication.
*
- * Internally, stack depot maintains a hash table of unique stacktraces. The
- * stack traces themselves are stored contiguously one after another in a set
- * of separate page allocations.
+ * Internally, stack depot has two storage backends. Refcounted entries and
+ * callers that request STACK_DEPOT_FLAG_COUNTABLE use the legacy hash table with
+ * contiguous stack records in stack pools. Persistent non-refcounted entries
+ * can use trie storage when enabled; trie nodes share common frame prefixes and
+ * are published through RCU children containers.
*
* Author: Alexander Potapenko <glider@google.com>
* Copyright (C) 2016 Google, Inc.
@@ -14,13 +16,19 @@
#define pr_fmt(fmt) "stackdepot: " fmt
+#include <linux/bitmap.h>
+#include <linux/build_bug.h>
#include <linux/debugfs.h>
+#include <linux/errno.h>
#include <linux/gfp.h>
#include <linux/jhash.h>
+#include <linux/jump_label.h>
#include <linux/kernel.h>
+#include <linux/log2.h>
#include <linux/kmsan.h>
#include <linux/list.h>
#include <linux/mm.h>
+#include <linux/moduleparam.h>
#include <linux/mutex.h>
#include <linux/poison.h>
#include <linux/printk.h>
@@ -36,9 +44,12 @@
#include <linux/memblock.h>
#include <linux/kasan-enabled.h>
+#include <asm/stackdepot.h>
+
/*
* The pool_index is offset by 1 so the first record does not have a 0 handle.
*/
+/* Parsed before mm_core_init(); trie handle decoding assumes this is then fixed. */
static unsigned int stack_max_pools __read_mostly =
MIN((1LL << DEPOT_POOL_INDEX_BITS) - 1, 8192);
@@ -63,18 +74,18 @@ static unsigned int stack_hash_mask;
/* The lock must be held when performing pool or freelist modifications. */
static DEFINE_RAW_SPINLOCK(pool_lock);
-/* Array of memory regions that store stack records. */
+/* Array of memory regions used by both stack depot backends. */
static void **stack_pools __pt_guarded_by(&pool_lock);
/* Newly allocated pool that is not yet added to stack_pools. */
static void *new_pool;
/* Number of pools in stack_pools. */
static int pools_num;
-/* Offset to the unused space in the currently used pool. */
+/* Offset to unused hash storage in the current pool. */
static size_t pool_offset __guarded_by(&pool_lock) = DEPOT_POOL_SIZE;
/* Freelist of stack records within stack_pools. */
static __guarded_by(&pool_lock) LIST_HEAD(free_stacks);
-/* Statistics counters for debugfs. */
+/* Hash-backend statistics counters for debugfs. */
enum depot_counter_id {
DEPOT_COUNTER_REFD_ALLOCS,
DEPOT_COUNTER_REFD_FREES,
@@ -90,11 +101,631 @@ static const char *const counter_names[] = {
[DEPOT_COUNTER_REFD_FREES] = "refcounted_frees",
[DEPOT_COUNTER_REFD_INUSE] = "refcounted_in_use",
[DEPOT_COUNTER_FREELIST_SIZE] = "freelist_size",
- [DEPOT_COUNTER_PERSIST_COUNT] = "persistent_count",
- [DEPOT_COUNTER_PERSIST_BYTES] = "persistent_bytes",
+ [DEPOT_COUNTER_PERSIST_COUNT] = "hash_persistent_count",
+ [DEPOT_COUNTER_PERSIST_BYTES] = "hash_persistent_bytes",
};
static_assert(ARRAY_SIZE(counter_names) == DEPOT_COUNTER_COUNT);
+enum stack_depot_frame_mode {
+ STACK_DEPOT_FRAME_RAW,
+ STACK_DEPOT_FRAME_COMPRESSED,
+};
+
+/*
+ * A trie node stores one run of frames that all use the same payload format.
+ * Architectures may compress some frames to 32-bit payloads; mixed raw and
+ * compressed input is split across multiple trie nodes so each node has one
+ * decoding mode.
+ */
+struct stack_depot_frame_run {
+ u16 nr_entries;
+ u8 mode;
+};
+
+static_assert(CONFIG_STACKDEPOT_MAX_FRAMES <= U16_MAX);
+
+struct stack_depot_trie_children;
+
+struct stack_depot_trie_node {
+ /* Parent links let fetch rebuild a full stack from a node to the root. */
+ const struct stack_depot_trie_node __rcu *parent;
+ /* Children are RCU-published containers. */
+ const struct stack_depot_trie_children __rcu *children;
+ /* Non-zero when a stored stack ends at this node. */
+ u32 stack_id;
+ struct stack_depot_frame_run run;
+ unsigned char data[];
+};
+
+/*
+ * Child nodes are sorted by first frame and searched by insertion position.
+ * Existing child pointers are immutable. Writers may publish into unused tail
+ * capacity; other updates publish a replacement container.
+ */
+struct stack_depot_trie_children {
+ unsigned int nr_children;
+ unsigned int capacity;
+ const struct stack_depot_trie_node __rcu *nodes[];
+};
+
+/* Retired children carry an optional node through their RCU grace period. */
+struct stack_depot_trie_retired_children {
+ struct list_head list;
+ unsigned long rcu_state;
+ const struct stack_depot_trie_node *pending_node;
+ unsigned char data[];
+};
+
+static_assert(IS_ALIGNED(offsetof(struct stack_depot_trie_retired_children, data),
+ 1UL << DEPOT_STACK_ALIGN));
+
+#define STACK_DEPOT_TRIE_SLOT_SIZE BIT(DEPOT_STACK_ALIGN)
+#define STACK_DEPOT_TRIE_POOL_SLOTS \
+ (DEPOT_POOL_SIZE / STACK_DEPOT_TRIE_SLOT_SIZE)
+
+struct stack_depot_trie_pool {
+ struct list_head list;
+ unsigned int free_slots;
+ DECLARE_BITMAP(used, STACK_DEPOT_TRIE_POOL_SLOTS);
+};
+
+#define STACK_DEPOT_TRIE_POOL_FIRST_SLOT \
+ DIV_ROUND_UP(sizeof(struct stack_depot_trie_pool), \
+ STACK_DEPOT_TRIE_SLOT_SIZE)
+#define STACK_DEPOT_TRIE_POOL_USABLE_SIZE \
+ ((STACK_DEPOT_TRIE_POOL_SLOTS - STACK_DEPOT_TRIE_POOL_FIRST_SLOT) * \
+ STACK_DEPOT_TRIE_SLOT_SIZE)
+
+static_assert(STACK_DEPOT_TRIE_POOL_FIRST_SLOT < STACK_DEPOT_TRIE_POOL_SLOTS);
+
+static DEFINE_STATIC_KEY_FALSE(stack_depot_trie_enabled);
+static const struct stack_depot_trie_children __rcu *stack_depot_trie_root;
+static DEFINE_RAW_SPINLOCK(stack_depot_trie_writer_lock);
+static bool stack_depot_trie_requested;
+
+module_param_named(trie_enabled, stack_depot_trie_requested, bool, 0);
+MODULE_PARM_DESC(trie_enabled, "Enable stack depot trie storage at boot");
+
+#define DEPOT_POOL_INDEX_MASK ((1U << DEPOT_POOL_INDEX_BITS) - 1)
+#define DEPOT_OFFSET_MASK ((1U << DEPOT_OFFSET_BITS) - 1)
+
+/* Retired fixed-size slots remain reserved until their RCU grace period ends. */
+static LIST_HEAD(stack_depot_trie_pools);
+static LIST_HEAD(pending_trie_children);
+
+/*
+ * stack_max_pools is the split point between hash and trie handle encodings.
+ * A handle with pool_index_plus_1 in 1..stack_max_pools names a hash-backed
+ * stack pool. Larger pool-index values cannot refer to hash pools, so trie
+ * storage uses that handle space to encode a dense stack ID. The side table
+ * maps each stack ID to its trie node.
+ */
+static inline u32 trie_max_stack_id(void)
+{
+ return (DEPOT_POOL_INDEX_MASK - stack_max_pools) <<
+ DEPOT_OFFSET_BITS;
+}
+
+static depot_stack_handle_t trie_handle(u32 stack_id)
+{
+ union handle_parts parts = {};
+ u64 pool_index_plus_1;
+ u32 pool_delta;
+ u32 index;
+
+ index = stack_id - 1;
+ pool_delta = index >> DEPOT_OFFSET_BITS;
+ pool_index_plus_1 = (u64)stack_max_pools + 1 + pool_delta;
+
+ parts.pool_index_plus_1 = pool_index_plus_1;
+ parts.offset = index & DEPOT_OFFSET_MASK;
+ return parts.handle;
+}
+
+static inline bool stack_depot_handle_is_trie(depot_stack_handle_t handle)
+{
+ union handle_parts parts = { .handle = handle };
+
+ return parts.pool_index_plus_1 > stack_max_pools;
+}
+
+static u32 trie_stack_id(depot_stack_handle_t handle)
+{
+ union handle_parts parts = { .handle = handle };
+ u32 pool_delta;
+
+ pool_delta = parts.pool_index_plus_1 - stack_max_pools - 1;
+ return (pool_delta << DEPOT_OFFSET_BITS) + parts.offset + 1;
+}
+
+/*
+ * Trie handles encode a dense stack ID. The side table maps that ID to a node
+ * pointer for lockless fetch and print paths, which can run from diagnostic
+ * contexts where taking a lock would be unsafe. Initialization installs the
+ * root; early initialization also installs the first directory and chunk.
+ * Additional directories and chunks are published lazily as stack IDs grow.
+ */
+#define STACK_DEPOT_TRIE_SIDE_TABLE_CHUNK_SIZE \
+ (PAGE_SIZE / sizeof(struct stack_depot_trie_node *))
+#define STACK_DEPOT_TRIE_SIDE_TABLE_DIR_SIZE \
+ (PAGE_SIZE / sizeof(struct stack_depot_trie_node **))
+
+struct stack_depot_trie_side_dir {
+ /* Both the chunk pointer and each node pointer in it are RCU-published. */
+ const struct stack_depot_trie_node __rcu * __rcu *
+ chunks[STACK_DEPOT_TRIE_SIDE_TABLE_DIR_SIZE];
+};
+
+struct stack_depot_trie_side_root {
+ unsigned int dir_capacity;
+ struct stack_depot_trie_side_dir __rcu *dirs[];
+};
+
+struct stack_depot_trie_side_prealloc {
+ /* Preallocated side-table directory page for sparse growth. */
+ struct stack_depot_trie_side_dir *dir;
+ /* Preallocated side-table pointer chunk for sparse growth. */
+ const struct stack_depot_trie_node __rcu **chunk;
+};
+
+static struct stack_depot_trie_side_root *trie_side_table_root;
+static DEFINE_RAW_SPINLOCK(trie_side_table_cache_lock);
+/* Zeroed unpublished pages; get/put transfer ownership under the cache lock. */
+static struct stack_depot_trie_side_prealloc trie_side_table_cache;
+static u32 trie_side_table_last_stack_id;
+
+/* Lock order: writer_lock -> pool_lock. The cache lock is never nested. */
+
+static inline size_t stack_depot_frame_run_entry_bytes(enum stack_depot_frame_mode mode)
+{
+ if (mode == STACK_DEPOT_FRAME_COMPRESSED)
+ return sizeof(u32);
+ return sizeof(unsigned long);
+}
+
+static inline size_t stack_depot_frame_run_bytes(const struct stack_depot_frame_run *run)
+{
+ return run->nr_entries * stack_depot_frame_run_entry_bytes(run->mode);
+}
+
+static inline size_t trie_node_bytes(const struct stack_depot_frame_run *run)
+{
+ return ALIGN(offsetof(struct stack_depot_trie_node, data) +
+ stack_depot_frame_run_bytes(run), sizeof(unsigned long));
+}
+
+static size_t trie_children_alloc_size(unsigned int capacity)
+{
+ size_t size;
+
+ size = struct_size_t(struct stack_depot_trie_children, nodes,
+ capacity);
+ return offsetof(struct stack_depot_trie_retired_children, data) +
+ ALIGN(size, sizeof(unsigned long));
+}
+
+static inline unsigned int trie_side_table_root_index(u32 id)
+{
+ return ((id - 1) / STACK_DEPOT_TRIE_SIDE_TABLE_CHUNK_SIZE) /
+ STACK_DEPOT_TRIE_SIDE_TABLE_DIR_SIZE;
+}
+
+static inline unsigned int trie_side_table_dir_index(u32 id)
+{
+ return ((id - 1) / STACK_DEPOT_TRIE_SIDE_TABLE_CHUNK_SIZE) %
+ STACK_DEPOT_TRIE_SIDE_TABLE_DIR_SIZE;
+}
+
+static inline unsigned int trie_side_table_slot_index(u32 id)
+{
+ return (id - 1) % STACK_DEPOT_TRIE_SIDE_TABLE_CHUNK_SIZE;
+}
+
+static struct stack_depot_trie_side_dir *trie_side_table_load_dir(unsigned int root)
+{
+ struct stack_depot_trie_side_root *root_vec;
+
+ root_vec = trie_side_table_root;
+ if (!root_vec || root >= root_vec->dir_capacity)
+ return NULL;
+ /* Pairs with side-table directory rcu_assign_pointer(). */
+ return rcu_dereference_check(root_vec->dirs[root],
+ lockdep_is_held(&stack_depot_trie_writer_lock) ||
+ rcu_read_lock_sched_held());
+}
+
+static inline const struct stack_depot_trie_node __rcu **
+trie_side_table_dir_load_chunk(struct stack_depot_trie_side_dir *dir,
+ unsigned int idx)
+{
+ /* Pairs with the chunk rcu_assign_pointer() in stack ID preparation. */
+ return rcu_dereference_check(dir->chunks[idx],
+ lockdep_is_held(&stack_depot_trie_writer_lock) ||
+ rcu_read_lock_sched_held());
+}
+
+static u32
+trie_side_table_prepare_stack_slot(struct stack_depot_trie_side_prealloc *prealloc)
+{
+ const struct stack_depot_trie_node __rcu **chunk;
+ struct stack_depot_trie_side_dir *dir;
+ struct stack_depot_trie_side_root *root_vec;
+ unsigned int root;
+ unsigned int idx;
+ u32 id;
+
+ lockdep_assert_held(&stack_depot_trie_writer_lock);
+
+ id = trie_side_table_last_stack_id + 1;
+ if (id > trie_max_stack_id())
+ return 0;
+
+ root_vec = trie_side_table_root;
+ root = trie_side_table_root_index(id);
+ dir = trie_side_table_load_dir(root);
+ if (!dir) {
+ dir = prealloc->dir;
+ prealloc->dir = NULL;
+ /* Publish the zeroed directory before readers can load it locklessly. */
+ rcu_assign_pointer(root_vec->dirs[root], dir);
+ }
+
+ idx = trie_side_table_dir_index(id);
+ chunk = trie_side_table_dir_load_chunk(dir, idx);
+ if (!chunk) {
+ chunk = prealloc->chunk;
+ prealloc->chunk = NULL;
+ rcu_assign_pointer(dir->chunks[idx], chunk);
+ }
+
+ return id;
+}
+
+static inline unsigned int trie_side_table_root_size_for_max_id(u32 max_stack_id)
+{
+ unsigned int top_size;
+
+ top_size = DIV_ROUND_UP(max_stack_id,
+ STACK_DEPOT_TRIE_SIDE_TABLE_CHUNK_SIZE);
+ return DIV_ROUND_UP(top_size, STACK_DEPOT_TRIE_SIDE_TABLE_DIR_SIZE);
+}
+
+static int __init stack_depot_trie_init_memblock(void)
+{
+ struct stack_depot_trie_side_root *root_vec;
+ struct stack_depot_trie_side_dir *first_dir;
+ const struct stack_depot_trie_node __rcu **first_chunk;
+ size_t root_bytes;
+ u32 max_stack_id;
+ unsigned int root_size;
+
+ max_stack_id = trie_max_stack_id();
+ if (!max_stack_id)
+ return -EINVAL;
+ root_size = trie_side_table_root_size_for_max_id(max_stack_id);
+ root_bytes = struct_size_t(struct stack_depot_trie_side_root, dirs,
+ root_size);
+
+ root_vec = memblock_alloc(root_bytes, __alignof__(*root_vec));
+ if (!root_vec)
+ return -ENOMEM;
+ first_dir = memblock_alloc(PAGE_SIZE, PAGE_SIZE);
+ if (!first_dir) {
+ memblock_free(root_vec, root_bytes);
+ return -ENOMEM;
+ }
+ first_chunk = memblock_alloc(PAGE_SIZE, PAGE_SIZE);
+ if (!first_chunk) {
+ memblock_free(first_dir, PAGE_SIZE);
+ memblock_free(root_vec, root_bytes);
+ return -ENOMEM;
+ }
+
+ root_vec->dir_capacity = root_size;
+ RCU_INIT_POINTER(root_vec->dirs[0], first_dir);
+ RCU_INIT_POINTER(first_dir->chunks[0], first_chunk);
+ trie_side_table_root = root_vec;
+ static_branch_enable(&stack_depot_trie_enabled);
+ return 0;
+}
+
+static int stack_depot_trie_init(void)
+{
+ struct stack_depot_trie_side_root *root_vec;
+ unsigned int root_size;
+ u32 max_stack_id;
+
+ max_stack_id = trie_max_stack_id();
+ if (!max_stack_id)
+ return -EINVAL;
+
+ root_size = trie_side_table_root_size_for_max_id(max_stack_id);
+ root_vec = kvzalloc_flex(*root_vec, dirs, root_size);
+ if (!root_vec)
+ return -ENOMEM;
+
+ root_vec->dir_capacity = root_size;
+ trie_side_table_root = root_vec;
+ static_branch_enable(&stack_depot_trie_enabled);
+ return 0;
+}
+
+static int trie_side_table_get_prealloc(gfp_t gfp_flags,
+ struct stack_depot_trie_side_prealloc *prealloc)
+{
+ unsigned long flags;
+
+ gfp_flags = gfp_nested_mask(gfp_flags);
+ raw_spin_lock_irqsave(&trie_side_table_cache_lock, flags);
+ prealloc->dir = trie_side_table_cache.dir;
+ prealloc->chunk = trie_side_table_cache.chunk;
+ trie_side_table_cache.dir = NULL;
+ trie_side_table_cache.chunk = NULL;
+ raw_spin_unlock_irqrestore(&trie_side_table_cache_lock, flags);
+
+ if (!prealloc->dir) {
+ prealloc->dir = (void *)get_zeroed_page(gfp_flags);
+ if (!prealloc->dir)
+ return -ENOMEM;
+ }
+ if (!prealloc->chunk) {
+ prealloc->chunk = (void *)get_zeroed_page(gfp_flags);
+ if (!prealloc->chunk)
+ return -ENOMEM;
+ }
+
+ return 0;
+}
+
+static void trie_side_table_put_prealloc(struct stack_depot_trie_side_prealloc *prealloc)
+{
+ unsigned long flags;
+
+ raw_spin_lock_irqsave(&trie_side_table_cache_lock, flags);
+ if (!trie_side_table_cache.dir) {
+ trie_side_table_cache.dir = prealloc->dir;
+ prealloc->dir = NULL;
+ }
+ if (!trie_side_table_cache.chunk) {
+ trie_side_table_cache.chunk = prealloc->chunk;
+ prealloc->chunk = NULL;
+ }
+ raw_spin_unlock_irqrestore(&trie_side_table_cache_lock, flags);
+
+ if (prealloc->dir)
+ free_page((unsigned long)prealloc->dir);
+ if (prealloc->chunk)
+ free_page((unsigned long)prealloc->chunk);
+}
+
+static const struct stack_depot_trie_node *trie_side_table_lookup(u32 id)
+{
+ const struct stack_depot_trie_node __rcu **chunk;
+ struct stack_depot_trie_side_dir *dir;
+ unsigned int root;
+
+ root = trie_side_table_root_index(id);
+ dir = trie_side_table_load_dir(root);
+ if (!dir)
+ return NULL;
+ chunk = trie_side_table_dir_load_chunk(dir, trie_side_table_dir_index(id));
+ if (!chunk)
+ return NULL;
+
+ /* Pairs with side-table node publication. */
+ return rcu_dereference_check(chunk[trie_side_table_slot_index(id)],
+ lockdep_is_held(&stack_depot_trie_writer_lock) ||
+ rcu_read_lock_sched_held());
+}
+
+static inline struct stack_depot_trie_retired_children *
+trie_retired_children(const void *ptr)
+{
+ return container_of(ptr, struct stack_depot_trie_retired_children, data);
+}
+
+static bool depot_init_pool(void **prealloc);
+
+static unsigned int trie_pool_reserve_slots(struct stack_depot_trie_pool *pool,
+ unsigned int nr_slots)
+{
+ unsigned int run = 0;
+ unsigned int i;
+ unsigned int slot;
+
+ if (pool->free_slots < nr_slots)
+ return STACK_DEPOT_TRIE_POOL_SLOTS;
+
+ /* A free run can cross any previous allocation position. */
+ for (slot = STACK_DEPOT_TRIE_POOL_FIRST_SLOT;
+ slot < STACK_DEPOT_TRIE_POOL_SLOTS; slot++) {
+ if (pool->used[slot / BITS_PER_LONG] &
+ BIT(slot % BITS_PER_LONG)) {
+ run = 0;
+ continue;
+ }
+ if (++run != nr_slots)
+ continue;
+
+ for (i = slot + 1 - nr_slots; i <= slot; i++)
+ pool->used[i / BITS_PER_LONG] |= BIT(i % BITS_PER_LONG);
+ pool->free_slots -= nr_slots;
+ return slot + 1 - nr_slots;
+ }
+
+ return STACK_DEPOT_TRIE_POOL_SLOTS;
+}
+
+/* Allocate at least @size bytes from one contiguous trie-pool slot run. */
+static void *trie_pool_alloc(size_t size, void **prealloc)
+{
+ struct stack_depot_trie_pool *pool;
+ unsigned int nr_slots;
+ unsigned int slot;
+
+ lockdep_assert_held(&pool_lock);
+
+ if (size > STACK_DEPOT_TRIE_POOL_USABLE_SIZE)
+ return NULL;
+ nr_slots = DIV_ROUND_UP(size, STACK_DEPOT_TRIE_SLOT_SIZE);
+ list_for_each_entry_reverse(pool, &stack_depot_trie_pools, list) {
+ slot = trie_pool_reserve_slots(pool, nr_slots);
+ if (slot != STACK_DEPOT_TRIE_POOL_SLOTS)
+ return (char *)pool + slot * STACK_DEPOT_TRIE_SLOT_SIZE;
+ }
+
+ if (!depot_init_pool(prealloc))
+ return NULL;
+ pool = stack_pools[pools_num - 1];
+ /* Keep hash records out of this bitmap-owned pool. */
+ pool_offset = DEPOT_POOL_SIZE;
+ memset(pool, 0, sizeof(*pool));
+ pool->free_slots = STACK_DEPOT_TRIE_POOL_SLOTS -
+ STACK_DEPOT_TRIE_POOL_FIRST_SLOT;
+ list_add_tail(&pool->list, &stack_depot_trie_pools);
+
+ slot = trie_pool_reserve_slots(pool, nr_slots);
+ return (char *)pool + slot * STACK_DEPOT_TRIE_SLOT_SIZE;
+}
+
+/* Release the slots for the byte count originally passed to allocation. */
+static void trie_pool_release(const void *ptr, size_t size)
+{
+ struct stack_depot_trie_pool *pool;
+ unsigned long pfn;
+ unsigned int nr_slots;
+ unsigned int slot;
+ unsigned int i;
+
+ lockdep_assert_held(&pool_lock);
+
+ pfn = page_to_pfn(virt_to_page(ptr));
+ pfn &= ~(BIT(DEPOT_POOL_ORDER) - 1);
+ pool = page_address(pfn_to_page(pfn));
+ slot = ((unsigned long)ptr - (unsigned long)pool) >> DEPOT_STACK_ALIGN;
+ nr_slots = DIV_ROUND_UP(size, STACK_DEPOT_TRIE_SLOT_SIZE);
+ for (i = slot; i < slot + nr_slots; i++)
+ pool->used[i / BITS_PER_LONG] &= ~BIT(i % BITS_PER_LONG);
+ pool->free_slots += nr_slots;
+}
+
+static struct stack_depot_trie_children *
+trie_pool_alloc_children(unsigned int capacity, void **prealloc)
+{
+ struct stack_depot_trie_retired_children *retired;
+ struct stack_depot_trie_children *children;
+
+ /* Capacity counts child-pointer entries; allocation includes RCU metadata. */
+ retired = trie_pool_alloc(trie_children_alloc_size(capacity), prealloc);
+ if (!retired)
+ return NULL;
+
+ children = (void *)retired->data;
+ children->nr_children = 0;
+ children->capacity = capacity;
+ return children;
+}
+
+static void
+trie_pool_release_children(const struct stack_depot_trie_children *children)
+{
+ /* Capacity is immutable and therefore recovers the allocation byte size. */
+ trie_pool_release(trie_retired_children(children),
+ trie_children_alloc_size(children->capacity));
+}
+
+/*
+ * Return RCU-ready objects before allocating. Pending children are FIFO, so
+ * stop at the first incomplete grace period. A replaced node shares the same
+ * retirement cookie and is released with its former children container.
+ */
+static void trie_drain_pending_children(void)
+{
+ struct stack_depot_trie_retired_children *retired;
+ struct stack_depot_trie_retired_children *tmp;
+ struct stack_depot_trie_children *children;
+
+ lockdep_assert_held(&pool_lock);
+
+ list_for_each_entry_safe(retired, tmp, &pending_trie_children, list) {
+ if (!poll_state_synchronize_rcu(retired->rcu_state))
+ break;
+ children = (void *)retired->data;
+ list_del(&retired->list);
+ if (retired->pending_node)
+ trie_pool_release(retired->pending_node,
+ trie_node_bytes(&retired->pending_node->run));
+ trie_pool_release_children(children);
+ }
+}
+
+static void trie_retire_children(const struct stack_depot_trie_children *children)
+{
+ struct stack_depot_trie_retired_children *retired;
+
+ lockdep_assert_held(&pool_lock);
+
+ retired = trie_retired_children(children);
+ retired->pending_node = NULL;
+ retired->rcu_state = get_state_synchronize_rcu();
+ list_add_tail(&retired->list, &pending_trie_children);
+}
+
+static void
+trie_retire_children_with_node(const struct stack_depot_trie_children *children,
+ const struct stack_depot_trie_node *node)
+{
+ struct stack_depot_trie_retired_children *retired;
+
+ lockdep_assert_held(&stack_depot_trie_writer_lock);
+ raw_spin_lock(&pool_lock);
+ trie_retire_children(children);
+ retired = trie_retired_children(children);
+ retired->pending_node = node;
+ raw_spin_unlock(&pool_lock);
+}
+
+static const struct stack_depot_trie_node *
+stack_depot_trie_lookup(const unsigned long *entries, unsigned int nr_entries);
+
+static depot_stack_handle_t
+trie_find_handle(const unsigned long *entries, unsigned int nr_entries)
+{
+ depot_stack_handle_t handle = 0;
+ const struct stack_depot_trie_node *node;
+
+ rcu_read_lock_sched_notrace();
+ node = stack_depot_trie_lookup(entries, nr_entries);
+ if (node)
+ handle = trie_handle(node->stack_id);
+ rcu_read_unlock_sched_notrace();
+
+ return handle;
+}
+
+/*
+ * Publish only after the node and its path are fully initialized and all
+ * fallible allocation is complete. Publication commits the path, so it cannot
+ * then be rolled back. Side-table mappings must precede trie topology
+ * publication that makes new or remapped nodes reachable from lookup.
+ * Published storage remains valid until RCU retirement; only descendant parent
+ * links may change meanwhile.
+ */
+static void trie_side_table_publish(const struct stack_depot_trie_node *node)
+{
+ const struct stack_depot_trie_node __rcu **chunk;
+ struct stack_depot_trie_side_dir *dir;
+ u32 stack_id = node->stack_id;
+
+ lockdep_assert_held(&stack_depot_trie_writer_lock);
+
+ dir = trie_side_table_load_dir(trie_side_table_root_index(stack_id));
+ chunk = trie_side_table_dir_load_chunk(dir,
+ trie_side_table_dir_index(stack_id));
+ /* Pairs with trie_side_table_lookup(). */
+ rcu_assign_pointer(chunk[trie_side_table_slot_index(stack_id)], node);
+}
+
static int __init disable_stack_depot(char *str)
{
return kstrtobool(str, &stack_depot_disabled);
@@ -146,7 +777,7 @@ static void init_stack_table(unsigned long entries)
INIT_LIST_HEAD(&stack_table[i]);
}
-/* Allocates a hash table via memblock. Can only be used during early boot. */
+/* Initializes hash and optional trie storage during early boot. */
int __init stack_depot_early_init(void)
{
unsigned long entries = 0;
@@ -220,11 +851,15 @@ int __init stack_depot_early_init(void)
stack_depot_disabled = true;
return -ENOMEM;
}
+ if (stack_depot_trie_requested && stack_depot_trie_init_memblock()) {
+ pr_warn("trie storage initialization failed, disabling trie storage\n");
+ stack_depot_trie_requested = false;
+ }
return 0;
}
-/* Allocates a hash table via kvcalloc. Can be used after boot. */
+/* Initializes hash and optional trie storage after boot. */
int stack_depot_init(void)
{
static DEFINE_MUTEX(stack_depot_init_mutex);
@@ -278,6 +913,15 @@ int stack_depot_init(void)
kvfree(stack_table);
stack_depot_disabled = true;
ret = -ENOMEM;
+ goto out_unlock;
+ }
+ if (stack_depot_trie_requested) {
+ ret = stack_depot_trie_init();
+ if (ret) {
+ pr_warn("trie storage initialization failed, disabling trie storage\n");
+ stack_depot_trie_requested = false;
+ ret = 0;
+ }
}
out_unlock:
@@ -323,7 +967,7 @@ static bool depot_init_pool(void **prealloc)
* NULL; do not reset to NULL if we have reached the maximum number of
* pools.
*/
- if (pools_num < stack_max_pools)
+ if (pools_num + 1 < stack_max_pools)
WRITE_ONCE(new_pool, NULL);
else
WRITE_ONCE(new_pool, STACK_DEPOT_POISON);
@@ -467,6 +1111,7 @@ depot_alloc_stack(unsigned long *entries, unsigned int nr_entries, u32 hash, dep
/* Save the stack trace. */
stack->hash = hash;
stack->size = nr_entries;
+ stack->flags = flags & STACK_DEPOT_FLAG_COUNTABLE;
/* stack->handle is already filled in by depot_pop_free_pool(). */
memcpy(stack->entries, entries, flex_array_size(stack, entries, nr_entries));
@@ -609,6 +1254,9 @@ static inline struct stack_record *find_stack(struct list_head *bucket,
list_for_each_entry_rcu(stack, bucket, hash_list) {
if (stack->hash != hash || stack->size != size)
continue;
+ /* Page owner countable records have a distinct count lifetime. */
+ if ((stack->flags ^ flags) & STACK_DEPOT_FLAG_COUNTABLE)
+ continue;
/*
* This may race with depot_free_stack() accessing the freelist
@@ -638,6 +1286,63 @@ static inline struct stack_record *find_stack(struct list_head *bucket,
return ret;
}
+static u32
+stack_depot_trie_insert(const unsigned long *entries,
+ unsigned int nr_entries, void **pool_prealloc,
+ struct stack_depot_trie_side_prealloc *side_prealloc);
+
+static depot_stack_handle_t
+stack_depot_trie_save(unsigned long *entries, unsigned int nr_entries,
+ gfp_t alloc_flags)
+{
+ unsigned int attempt;
+
+ /* Allow one stale pool hint before the two pools a largest insert needs. */
+ for (attempt = 0; attempt < 3; attempt++) {
+ struct stack_depot_trie_side_prealloc side_prealloc = {};
+ void *pool_prealloc = NULL;
+ depot_stack_handle_t handle;
+ unsigned long flags;
+ struct page *page;
+ u32 stack_id = 0;
+
+ handle = trie_find_handle(entries, nr_entries);
+ if (handle)
+ return handle;
+
+ if (trie_side_table_get_prealloc(alloc_flags, &side_prealloc)) {
+ trie_side_table_put_prealloc(&side_prealloc);
+ return 0;
+ }
+
+ /* The hint may race; a missing page is recovered by the retry. */
+ if (!READ_ONCE(new_pool)) {
+ page = alloc_pages(gfp_nested_mask(alloc_flags),
+ DEPOT_POOL_ORDER);
+ if (page)
+ pool_prealloc = page_address(page);
+ }
+
+ raw_spin_lock_irqsave(&stack_depot_trie_writer_lock, flags);
+ stack_id = stack_depot_trie_insert(entries, nr_entries,
+ &pool_prealloc, &side_prealloc);
+ raw_spin_unlock_irqrestore(&stack_depot_trie_writer_lock, flags);
+
+ if (pool_prealloc) {
+ raw_spin_lock_irqsave(&pool_lock, flags);
+ depot_keep_new_pool(&pool_prealloc);
+ raw_spin_unlock_irqrestore(&pool_lock, flags);
+ }
+ if (pool_prealloc)
+ free_pages((unsigned long)pool_prealloc, DEPOT_POOL_ORDER);
+ trie_side_table_put_prealloc(&side_prealloc);
+ if (stack_id)
+ return trie_handle(stack_id);
+ }
+
+ return 0;
+}
+
depot_stack_handle_t stack_depot_save_flags(unsigned long *entries,
unsigned int nr_entries,
gfp_t alloc_flags,
@@ -655,6 +1360,9 @@ depot_stack_handle_t stack_depot_save_flags(unsigned long *entries,
if (WARN_ON(depot_flags & ~STACK_DEPOT_FLAGS_MASK))
return 0;
+ if (WARN_ON_ONCE((depot_flags & STACK_DEPOT_FLAG_GET) &&
+ (depot_flags & STACK_DEPOT_FLAG_COUNTABLE)))
+ return 0;
/*
* If this stack trace is from an interrupt, including anything before
@@ -669,6 +1377,17 @@ depot_stack_handle_t stack_depot_save_flags(unsigned long *entries,
if (unlikely(nr_entries == 0) || stack_depot_disabled)
return 0;
+ if (!(depot_flags & (STACK_DEPOT_FLAG_GET | STACK_DEPOT_FLAG_COUNTABLE)) &&
+ static_branch_unlikely(&stack_depot_trie_enabled)) {
+ if (nr_entries > CONFIG_STACKDEPOT_MAX_FRAMES)
+ nr_entries = CONFIG_STACKDEPOT_MAX_FRAMES;
+ if (in_nmi() || !can_alloc) {
+ WARN_ON_ONCE(can_alloc);
+ return trie_find_handle(entries, nr_entries);
+ }
+ return stack_depot_trie_save(entries, nr_entries, alloc_flags);
+ }
+
hash = hash_stack(entries, nr_entries);
bucket = &stack_table[hash & stack_hash_mask];
@@ -751,10 +1470,697 @@ EXPORT_SYMBOL_GPL(stack_depot_save);
struct stack_record *__stack_depot_get_stack_record(depot_stack_handle_t handle)
{
+ struct stack_record *stack;
+
if (!handle)
return NULL;
+ if (WARN_ON_ONCE(stack_depot_handle_is_trie(handle)))
+ return NULL;
+
+ stack = depot_fetch_stack(handle);
+ if (!stack)
+ return NULL;
+ if (WARN_ON_ONCE(!(stack->flags & STACK_DEPOT_FLAG_COUNTABLE)))
+ return NULL;
+
+ return stack;
+}
+
+static void frame_run_init(const unsigned long *entries,
+ unsigned int nr_entries,
+ struct stack_depot_frame_run *run)
+{
+ u32 payload;
+ unsigned int i;
+ bool compressed;
+
+ compressed = arch_stack_depot_frame_try_compress(entries[0], &payload);
+ for (i = 1; i < nr_entries; i++) {
+ bool next;
+
+ next = arch_stack_depot_frame_try_compress(entries[i], &payload);
+ if (next != compressed)
+ break;
+ }
- return depot_fetch_stack(handle);
+ /* @i is the first non-matching frame, or @nr_entries if all matched. */
+ run->mode = compressed ? STACK_DEPOT_FRAME_COMPRESSED : STACK_DEPOT_FRAME_RAW;
+ run->nr_entries = i;
+}
+
+static void
+stack_depot_trie_node_frame(const struct stack_depot_trie_node *node,
+ unsigned int index, unsigned long *frame)
+{
+ u32 payload;
+
+ if (node->run.mode == STACK_DEPOT_FRAME_RAW) {
+ memcpy(frame, node->data + index * sizeof(*frame),
+ sizeof(*frame));
+ return;
+ }
+
+ memcpy(&payload, node->data + index * sizeof(payload), sizeof(payload));
+ arch_stack_depot_frame_decompress(payload, frame);
+}
+
+static void trie_node_init(struct stack_depot_trie_node *node,
+ const struct stack_depot_trie_node *parent, u32 stack_id,
+ const unsigned long *entries,
+ const struct stack_depot_frame_run *run)
+{
+ if (run->mode == STACK_DEPOT_FRAME_COMPRESSED) {
+ unsigned int i;
+
+ for (i = 0; i < run->nr_entries; i++) {
+ u32 payload;
+
+ arch_stack_depot_frame_try_compress(entries[i], &payload);
+ memcpy(node->data + i * sizeof(payload), &payload,
+ sizeof(payload));
+ }
+ } else {
+ memcpy(node->data, entries, stack_depot_frame_run_bytes(run));
+ }
+
+ RCU_INIT_POINTER(node->parent, parent);
+ RCU_INIT_POINTER(node->children, NULL);
+ node->stack_id = stack_id;
+ node->run = *run;
+}
+
+static void trie_node_init_slice(struct stack_depot_trie_node *node,
+ const struct stack_depot_trie_node *parent, u32 stack_id,
+ const struct stack_depot_trie_node *src_node,
+ unsigned int start, unsigned int nr_entries)
+{
+ struct stack_depot_frame_run run;
+ size_t entry_bytes;
+
+ run = src_node->run;
+ run.nr_entries = nr_entries;
+
+ entry_bytes = stack_depot_frame_run_entry_bytes(src_node->run.mode);
+ memcpy(node->data, src_node->data + start * entry_bytes,
+ stack_depot_frame_run_bytes(&run));
+ RCU_INIT_POINTER(node->parent, parent);
+ RCU_INIT_POINTER(node->children, NULL);
+ node->stack_id = stack_id;
+ node->run = run;
+}
+
+static unsigned int trie_node_match(const struct stack_depot_trie_node *node,
+ const unsigned long *entries,
+ unsigned int nr_entries)
+{
+ unsigned int limit;
+ unsigned int i;
+
+ limit = min(node->run.nr_entries, nr_entries);
+ if (node->run.mode == STACK_DEPOT_FRAME_RAW) {
+ for (i = 0; i < limit; i++) {
+ unsigned long frame;
+
+ memcpy(&frame, node->data + i * sizeof(frame), sizeof(frame));
+ if (frame != entries[i])
+ break;
+ }
+
+ return i;
+ }
+
+ for (i = 0; i < limit; i++) {
+ unsigned long frame;
+
+ stack_depot_trie_node_frame(node, i, &frame);
+ if (frame != entries[i])
+ break;
+ }
+
+ return i;
+}
+
+static inline const struct stack_depot_trie_node *
+trie_load_parent(const struct stack_depot_trie_node *node)
+{
+ return rcu_dereference_check(node->parent,
+ lockdep_is_held(&stack_depot_trie_writer_lock) ||
+ rcu_read_lock_sched_held());
+}
+
+static inline const struct stack_depot_trie_children *
+trie_load_children(const struct stack_depot_trie_children __rcu * const *slot)
+{
+ return rcu_dereference_check(*slot,
+ lockdep_is_held(&stack_depot_trie_writer_lock) ||
+ rcu_read_lock_sched_held());
+}
+
+static inline const struct stack_depot_trie_node *
+trie_children_load_child(const struct stack_depot_trie_children *children,
+ unsigned int pos)
+{
+ return rcu_dereference_check(children->nodes[pos],
+ lockdep_is_held(&stack_depot_trie_writer_lock) ||
+ rcu_read_lock_sched_held());
+}
+
+static bool
+trie_children_find_position(const struct stack_depot_trie_children *children,
+ unsigned long frame, unsigned int *pos)
+{
+ unsigned int left = 0;
+ unsigned int right;
+
+ right = READ_ONCE(children->nr_children);
+ while (left < right) {
+ unsigned int mid = left + (right - left) / 2;
+ const struct stack_depot_trie_node *node;
+ unsigned long mid_frame;
+
+ node = trie_children_load_child(children, mid);
+ if (!node) {
+ /* Tail append may produce a transient lockless lookup miss. */
+ right = mid;
+ continue;
+ }
+ stack_depot_trie_node_frame(node, 0, &mid_frame);
+ if (mid_frame < frame) {
+ left = mid + 1;
+ } else if (mid_frame > frame) {
+ right = mid;
+ } else {
+ *pos = mid;
+ return true;
+ }
+ }
+
+ *pos = left;
+ return false;
+}
+
+/* Initialize an unpublished container from a stable published prefix. */
+static void trie_children_init(const struct stack_depot_trie_children *old,
+ struct stack_depot_trie_children *new)
+{
+ unsigned int nr_old = old->nr_children;
+ unsigned int i;
+
+ new->nr_children = nr_old;
+ for (i = 0; i < nr_old; i++)
+ RCU_INIT_POINTER(new->nodes[i], trie_children_load_child(old, i));
+ for (i = nr_old; i < new->capacity; i++)
+ RCU_INIT_POINTER(new->nodes[i], NULL);
+}
+
+static void trie_children_insert(struct stack_depot_trie_children *children,
+ const struct stack_depot_trie_node *node,
+ unsigned int pos)
+{
+ unsigned int i;
+
+ for (i = children->nr_children; i > pos; i--)
+ RCU_INIT_POINTER(children->nodes[i],
+ trie_children_load_child(children, i - 1));
+ RCU_INIT_POINTER(children->nodes[pos], node);
+ children->nr_children++;
+}
+
+static void trie_reparent_children(struct stack_depot_trie_node *parent)
+{
+ const struct stack_depot_trie_children *children;
+ unsigned int i;
+
+ lockdep_assert_held(&stack_depot_trie_writer_lock);
+
+ children = trie_load_children(&parent->children);
+ if (!children)
+ return;
+ /*
+ * Replacement nodes reuse unchanged descendant subtrees. Repoint their
+ * parent links before retiring the old parent so fetch never follows a freed
+ * node. Lockless fetches may see the new parent before publication, but the
+ * old and new parent chains contain the same frames and remain RCU-live.
+ */
+ for (i = 0; i < children->nr_children; i++) {
+ struct stack_depot_trie_node *child;
+
+ child = (struct stack_depot_trie_node *)trie_children_load_child(children, i);
+ rcu_assign_pointer(child->parent, parent);
+ }
+}
+
+/*
+ * Split entries into runs, allocate and initialize each node once, and link
+ * adjacent nodes through singleton children. Both trie locks must be held.
+ * Failure walks the unpublished parent chain and releases local ownership.
+ */
+static const struct stack_depot_trie_node *
+trie_path_alloc(const struct stack_depot_trie_node *parent, u32 stack_id,
+ const unsigned long *entries, unsigned int nr_entries,
+ void **pool_prealloc,
+ const struct stack_depot_trie_node **node_out)
+{
+ struct stack_depot_trie_children *path_children = NULL;
+ const struct stack_depot_trie_node *path_root = NULL;
+ const struct stack_depot_trie_node *last_node = parent;
+ unsigned int entry = 0;
+
+ lockdep_assert_held(&pool_lock);
+ lockdep_assert_held(&stack_depot_trie_writer_lock);
+
+ while (entry < nr_entries) {
+ struct stack_depot_frame_run run;
+ struct stack_depot_trie_node *node;
+
+ frame_run_init(&entries[entry], nr_entries - entry, &run);
+ node = trie_pool_alloc(trie_node_bytes(&run), pool_prealloc);
+ if (!node)
+ goto err_release;
+
+ trie_node_init(node, last_node,
+ entry + run.nr_entries == nr_entries ? stack_id : 0,
+ &entries[entry], &run);
+ entry += run.nr_entries;
+ last_node = node;
+ if (!path_root)
+ path_root = node;
+
+ if (path_children)
+ trie_children_insert(path_children, last_node, 0);
+ if (entry < nr_entries) {
+ path_children = trie_pool_alloc_children(1, pool_prealloc);
+ if (!path_children)
+ goto err_release;
+ RCU_INIT_POINTER(node->children, path_children);
+ }
+ }
+
+ *node_out = last_node;
+ return path_root;
+
+err_release:
+ while (last_node != parent) {
+ const struct stack_depot_trie_children *node_children;
+ const struct stack_depot_trie_node *node = last_node;
+
+ last_node = trie_load_parent(node);
+ node_children = trie_load_children(&node->children);
+ if (node_children)
+ trie_pool_release_children(node_children);
+ trie_pool_release(node, trie_node_bytes(&node->run));
+ }
+ return NULL;
+}
+
+static const struct stack_depot_trie_node *
+stack_depot_trie_lookup(const unsigned long *entries, unsigned int nr_entries)
+{
+ const struct stack_depot_trie_children *children;
+ unsigned int entry = 0;
+
+ children = trie_load_children(&stack_depot_trie_root);
+
+ while (entry < nr_entries) {
+ const struct stack_depot_trie_node *node;
+ unsigned int remaining = nr_entries - entry;
+ unsigned int matched;
+ unsigned int pos;
+
+ if (!children)
+ return NULL;
+ if (!trie_children_find_position(children, entries[entry], &pos))
+ return NULL;
+
+ node = trie_children_load_child(children, pos);
+ matched = trie_node_match(node, &entries[entry], remaining);
+ if (matched < node->run.nr_entries)
+ return NULL;
+ entry += matched;
+ if (entry == nr_entries)
+ return node->stack_id ? node : NULL;
+
+ children = trie_load_children(&node->children);
+ }
+
+ return NULL;
+}
+
+static u32
+trie_insert_path(const struct stack_depot_trie_children __rcu **slot,
+ struct stack_depot_trie_node *parent,
+ const struct stack_depot_trie_children *children,
+ unsigned int pos, const unsigned long *entries,
+ unsigned int nr_entries, void **pool_prealloc,
+ struct stack_depot_trie_side_prealloc *side_prealloc)
+{
+ struct stack_depot_trie_children *new_children = NULL;
+ const struct stack_depot_trie_node *path_root;
+ const struct stack_depot_trie_node *node;
+ unsigned int capacity = 1;
+ u32 new_stack_id;
+ bool tail_append = false;
+
+ /*
+ * Reuse spare capacity only for a sorted tail append. Other insertions
+ * replace the children container without modifying visible pointers.
+ */
+ if (children) {
+ capacity = roundup_pow_of_two(children->nr_children + 1);
+ tail_append = pos == children->nr_children &&
+ children->nr_children < children->capacity;
+ }
+ if (!tail_append && trie_children_alloc_size(capacity) >
+ STACK_DEPOT_TRIE_POOL_USABLE_SIZE)
+ return 0;
+
+ new_stack_id = trie_side_table_prepare_stack_slot(side_prealloc);
+ if (!new_stack_id)
+ return 0;
+
+ raw_spin_lock(&pool_lock);
+ printk_deferred_enter();
+ trie_drain_pending_children();
+
+ /* Reserve replacement topology before the path, the final fallible step. */
+ if (!tail_append) {
+ new_children = trie_pool_alloc_children(capacity, pool_prealloc);
+ if (!new_children)
+ goto err_release;
+ }
+ path_root = trie_path_alloc(parent, new_stack_id, entries, nr_entries,
+ pool_prealloc, &node);
+ if (!path_root)
+ goto err_release;
+
+ /* Commit the stack ID before making the path reachable from the trie. */
+ trie_side_table_publish(node);
+ if (tail_append) {
+ struct stack_depot_trie_children *tail_children =
+ (struct stack_depot_trie_children *)children;
+
+ /*
+ * Publish the node before the visible count. Readers may transiently
+ * see NULL and miss; the writer-lock recheck prevents duplicates.
+ */
+ rcu_assign_pointer(tail_children->nodes[pos], path_root);
+ WRITE_ONCE(tail_children->nr_children, pos + 1);
+ } else {
+ if (children)
+ trie_children_init(children, new_children);
+ trie_children_insert(new_children, path_root, pos);
+ rcu_assign_pointer(*slot, new_children);
+ if (children)
+ trie_retire_children(children);
+ }
+
+ printk_deferred_exit();
+ raw_spin_unlock(&pool_lock);
+ return new_stack_id;
+
+err_release:
+ if (new_children)
+ trie_pool_release_children(new_children);
+ printk_deferred_exit();
+ raw_spin_unlock(&pool_lock);
+ return 0;
+}
+
+static u32
+trie_split_child(const struct stack_depot_trie_children __rcu **slot,
+ const struct stack_depot_trie_children *children,
+ const struct stack_depot_trie_node *child,
+ unsigned int pos, unsigned int matched,
+ const unsigned long *entries, unsigned int nr_entries,
+ void **pool_prealloc,
+ struct stack_depot_trie_side_prealloc *side_prealloc)
+{
+ struct stack_depot_trie_children *prefix_children = NULL;
+ struct stack_depot_trie_children *new_children = NULL;
+ const struct stack_depot_trie_node *new_node;
+ const struct stack_depot_trie_node *suffix_roots[2];
+ struct stack_depot_frame_run run;
+ struct stack_depot_trie_node *split_prefix = NULL;
+ struct stack_depot_trie_node *old_suffix = NULL;
+ unsigned int nr_suffix_roots;
+ unsigned int old_suffix_len;
+ unsigned int i;
+ size_t split_prefix_size;
+ size_t old_suffix_size;
+ u32 new_stack_id;
+ bool has_new_suffix;
+
+ new_stack_id = trie_side_table_prepare_stack_slot(side_prealloc);
+ if (!new_stack_id)
+ return 0;
+
+ /* Rebuild the child's run as newly allocated prefix and old suffix nodes. */
+ run = child->run;
+ run.nr_entries = matched;
+ split_prefix_size = trie_node_bytes(&run);
+ old_suffix_len = child->run.nr_entries - matched;
+ run.nr_entries = old_suffix_len;
+ old_suffix_size = trie_node_bytes(&run);
+ has_new_suffix = matched < nr_entries;
+ nr_suffix_roots = has_new_suffix ? 2 : 1;
+
+ raw_spin_lock(&pool_lock);
+ printk_deferred_enter();
+ trie_drain_pending_children();
+
+ /* Reserve fixed split topology before the optional new suffix path. */
+ split_prefix = trie_pool_alloc(split_prefix_size, pool_prealloc);
+ if (!split_prefix)
+ goto err_release;
+ old_suffix = trie_pool_alloc(old_suffix_size, pool_prealloc);
+ if (!old_suffix)
+ goto err_release;
+ new_children = trie_pool_alloc_children(children->capacity, pool_prealloc);
+ if (!new_children)
+ goto err_release;
+ prefix_children = trie_pool_alloc_children(nr_suffix_roots, pool_prealloc);
+ if (!prefix_children)
+ goto err_release;
+
+ if (has_new_suffix) {
+ const struct stack_depot_trie_node *new_suffix;
+ unsigned long old_suffix_frame;
+
+ new_suffix = trie_path_alloc(split_prefix, new_stack_id,
+ &entries[matched], nr_entries - matched,
+ pool_prealloc, &new_node);
+ if (!new_suffix)
+ goto err_release;
+ stack_depot_trie_node_frame(child, matched, &old_suffix_frame);
+ /* Children remain sorted by the first frame of each suffix. */
+ if (old_suffix_frame < entries[matched]) {
+ suffix_roots[0] = old_suffix;
+ suffix_roots[1] = new_suffix;
+ } else {
+ suffix_roots[0] = new_suffix;
+ suffix_roots[1] = old_suffix;
+ }
+ } else {
+ new_node = split_prefix;
+ suffix_roots[0] = old_suffix;
+ }
+
+ printk_deferred_exit();
+ raw_spin_unlock(&pool_lock);
+
+ /* Rebuild the old path as prefix -> old suffix and attach suffix roots. */
+ trie_node_init_slice(split_prefix, trie_load_parent(child),
+ has_new_suffix ? 0 : new_stack_id, child, 0, matched);
+ trie_node_init_slice(old_suffix, split_prefix, child->stack_id, child,
+ matched, old_suffix_len);
+ for (i = 0; i < nr_suffix_roots; i++)
+ trie_children_insert(prefix_children, suffix_roots[i], i);
+ RCU_INIT_POINTER(old_suffix->children,
+ trie_load_children(&child->children));
+ RCU_INIT_POINTER(split_prefix->children, prefix_children);
+
+ /* Publish IDs, reparent descendants, then replace and retire topology. */
+ if (child->stack_id)
+ trie_side_table_publish(old_suffix);
+ trie_side_table_publish(new_node);
+ /* Old and replacement chains contain identical frames during transition. */
+ trie_children_init(children, new_children);
+ RCU_INIT_POINTER(new_children->nodes[pos], split_prefix);
+ trie_reparent_children(old_suffix);
+ rcu_assign_pointer(*slot, new_children);
+ trie_retire_children_with_node(children, child);
+
+ return new_stack_id;
+
+err_release:
+ if (split_prefix)
+ trie_pool_release(split_prefix, split_prefix_size);
+ if (old_suffix)
+ trie_pool_release(old_suffix, old_suffix_size);
+ if (prefix_children)
+ trie_pool_release_children(prefix_children);
+ if (new_children)
+ trie_pool_release_children(new_children);
+ printk_deferred_exit();
+ raw_spin_unlock(&pool_lock);
+ return 0;
+}
+
+static u32
+trie_promote_child(const struct stack_depot_trie_children __rcu **slot,
+ const struct stack_depot_trie_children *children,
+ const struct stack_depot_trie_node *child,
+ unsigned int pos, void **pool_prealloc,
+ struct stack_depot_trie_side_prealloc *side_prealloc)
+{
+ struct stack_depot_trie_children *new_children;
+ struct stack_depot_trie_node *promoted_node;
+ size_t node_size;
+ u32 new_stack_id;
+
+ new_stack_id = trie_side_table_prepare_stack_slot(side_prealloc);
+ if (!new_stack_id)
+ return 0;
+ node_size = trie_node_bytes(&child->run);
+
+ /* Reserve a clone and replacement children container before publication. */
+ raw_spin_lock(&pool_lock);
+ printk_deferred_enter();
+ trie_drain_pending_children();
+ promoted_node = trie_pool_alloc(node_size, pool_prealloc);
+ if (!promoted_node)
+ goto out_unlock;
+ new_children = trie_pool_alloc_children(children->capacity, pool_prealloc);
+ if (!new_children)
+ goto out_release_node;
+ printk_deferred_exit();
+ raw_spin_unlock(&pool_lock);
+
+ /* Add the stack ID through a clone, then reparent before retirement. */
+ memcpy(promoted_node, child, node_size);
+ promoted_node->stack_id = new_stack_id;
+ trie_side_table_publish(promoted_node);
+ trie_children_init(children, new_children);
+ RCU_INIT_POINTER(new_children->nodes[pos], promoted_node);
+ trie_reparent_children(promoted_node);
+ rcu_assign_pointer(*slot, new_children);
+ trie_retire_children_with_node(children, child);
+
+ return new_stack_id;
+
+out_release_node:
+ trie_pool_release(promoted_node, node_size);
+out_unlock:
+ printk_deferred_exit();
+ raw_spin_unlock(&pool_lock);
+ return 0;
+}
+
+static u32
+stack_depot_trie_insert(const unsigned long *entries,
+ unsigned int nr_entries, void **pool_prealloc,
+ struct stack_depot_trie_side_prealloc *side_prealloc)
+{
+ const struct stack_depot_trie_children *children;
+ const struct stack_depot_trie_children __rcu **slot =
+ &stack_depot_trie_root;
+ const struct stack_depot_trie_node *child;
+ struct stack_depot_trie_node *parent = NULL;
+ unsigned int matched;
+ unsigned int pos;
+ u32 stack_id;
+
+ lockdep_assert_held(&stack_depot_trie_writer_lock);
+
+ for (;;) {
+ pos = 0;
+ children = trie_load_children(slot);
+ /* No matching child: attach the remaining path. */
+ if (!children ||
+ !trie_children_find_position(children, entries[0], &pos)) {
+ stack_id = trie_insert_path(slot, parent, children, pos,
+ entries, nr_entries, pool_prealloc,
+ side_prealloc);
+ break;
+ }
+
+ child = trie_children_load_child(children, pos);
+ matched = trie_node_match(child, entries, nr_entries);
+ /* A partial child match requires a prefix/suffix split. */
+ if (matched < child->run.nr_entries) {
+ stack_id = trie_split_child(slot, children, child, pos,
+ matched, entries, nr_entries,
+ pool_prealloc, side_prealloc);
+ break;
+ }
+
+ /* The input ends here: reuse a stack node or promote an internal one. */
+ if (matched == nr_entries) {
+ if (child->stack_id)
+ return child->stack_id;
+ stack_id = trie_promote_child(slot, children, child, pos,
+ pool_prealloc, side_prealloc);
+ break;
+ }
+
+ /* The child matched completely; continue with the remaining frames. */
+ parent = (struct stack_depot_trie_node *)child;
+ slot = &parent->children;
+ entries += matched;
+ nr_entries -= matched;
+ }
+
+ if (stack_id)
+ trie_side_table_last_stack_id = stack_id;
+ return stack_id;
+}
+
+static unsigned int trie_fetch_into(const struct stack_depot_trie_node *node,
+ unsigned long *entries,
+ unsigned int max_entries)
+{
+ const struct stack_depot_trie_node *cur;
+ unsigned int total;
+ unsigned int pos;
+ unsigned int i;
+
+ total = 0;
+ for (cur = node; cur; cur = trie_load_parent(cur))
+ total += cur->run.nr_entries;
+ if (max_entries < total)
+ return 0;
+
+ pos = total;
+ for (cur = node; cur; cur = trie_load_parent(cur)) {
+ pos -= cur->run.nr_entries;
+ for (i = 0; i < cur->run.nr_entries; i++)
+ stack_depot_trie_node_frame(cur, i, &entries[pos + i]);
+ }
+
+ return total;
+}
+
+static unsigned int trie_fetch_handle_into(depot_stack_handle_t handle,
+ unsigned long *entries,
+ unsigned int max_entries)
+{
+ const struct stack_depot_trie_node *node;
+ u32 stack_id;
+ unsigned int nr_entries;
+
+ stack_id = trie_stack_id(handle);
+ rcu_read_lock_sched_notrace();
+ node = trie_side_table_lookup(stack_id);
+ if (WARN_ONCE(!node, "corrupt trie handle %08x\n", handle)) {
+ rcu_read_unlock_sched_notrace();
+ return 0;
+ }
+ nr_entries = trie_fetch_into(node, entries, max_entries);
+ rcu_read_unlock_sched_notrace();
+ if (nr_entries)
+ kmsan_unpoison_memory(entries, nr_entries * sizeof(*entries));
+
+ return nr_entries;
}
unsigned int stack_depot_fetch(depot_stack_handle_t handle,
@@ -771,6 +2177,8 @@ unsigned int stack_depot_fetch(depot_stack_handle_t handle,
if (!handle || stack_depot_disabled)
return 0;
+ if (WARN_ON_ONCE(stack_depot_handle_is_trie(handle)))
+ return 0;
stack = depot_fetch_stack(handle);
/*
@@ -785,12 +2193,44 @@ unsigned int stack_depot_fetch(depot_stack_handle_t handle,
}
EXPORT_SYMBOL_GPL(stack_depot_fetch);
+unsigned int stack_depot_fetch_into(depot_stack_handle_t handle,
+ unsigned long *entries,
+ unsigned int max_entries)
+{
+ struct stack_record *stack;
+ unsigned int nr_entries;
+
+ if (!handle)
+ return 0;
+ if (stack_depot_disabled)
+ return 0;
+ WARN_ON_ONCE(!entries || !max_entries);
+ if (stack_depot_handle_is_trie(handle))
+ return trie_fetch_handle_into(handle, entries, max_entries);
+
+ stack = depot_fetch_stack(handle);
+ if (!stack)
+ return 0;
+ nr_entries = stack->size;
+ if (WARN_ON_ONCE(!nr_entries))
+ return 0;
+ if (nr_entries > max_entries)
+ return 0;
+
+ memcpy(entries, stack->entries, nr_entries * sizeof(*entries));
+ kmsan_unpoison_memory(entries, nr_entries * sizeof(*entries));
+ return nr_entries;
+}
+EXPORT_SYMBOL_GPL(stack_depot_fetch_into);
+
void stack_depot_put(depot_stack_handle_t handle)
{
struct stack_record *stack;
if (!handle || stack_depot_disabled)
return;
+ if (WARN_ON_ONCE(stack_depot_handle_is_trie(handle)))
+ return;
stack = depot_fetch_stack(handle);
/*
@@ -800,6 +2240,8 @@ void stack_depot_put(depot_stack_handle_t handle)
if (WARN(!stack, "corrupt handle or unbalanced stack_depot_put()"))
return;
+ if (WARN_ON_ONCE(stack->flags & STACK_DEPOT_FLAG_COUNTABLE))
+ return;
if (refcount_dec_and_test(&stack->count))
depot_free_stack(stack);
}
@@ -810,6 +2252,15 @@ void stack_depot_print(depot_stack_handle_t stack)
unsigned long *entries;
unsigned int nr_entries;
+ if (stack_depot_handle_is_trie(stack)) {
+ unsigned long trie_entries[CONFIG_STACKDEPOT_MAX_FRAMES];
+
+ nr_entries = trie_fetch_handle_into(stack, trie_entries,
+ ARRAY_SIZE(trie_entries));
+ stack_trace_print(trie_entries, nr_entries, 0);
+ return;
+ }
+
nr_entries = stack_depot_fetch(stack, &entries);
if (nr_entries > 0)
stack_trace_print(entries, nr_entries, 0);
@@ -822,6 +2273,15 @@ int stack_depot_snprint(depot_stack_handle_t handle, char *buf, size_t size,
unsigned long *entries;
unsigned int nr_entries;
+ if (stack_depot_handle_is_trie(handle)) {
+ unsigned long trie_entries[CONFIG_STACKDEPOT_MAX_FRAMES];
+
+ nr_entries = trie_fetch_handle_into(handle, trie_entries,
+ ARRAY_SIZE(trie_entries));
+ return stack_trace_snprint(buf, size, trie_entries, nr_entries,
+ spaces);
+ }
+
nr_entries = stack_depot_fetch(handle, &entries);
return nr_entries ? stack_trace_snprint(buf, size, entries, nr_entries,
spaces) : 0;
diff --git a/lib/tests/Makefile b/lib/tests/Makefile
index 4ead57602eac4..2d40bd21a8ef9 100644
--- a/lib/tests/Makefile
+++ b/lib/tests/Makefile
@@ -49,6 +49,7 @@ obj-$(CONFIG_SCANF_KUNIT_TEST) += scanf_kunit.o
obj-$(CONFIG_SEQ_BUF_KUNIT_TEST) += seq_buf_kunit.o
obj-$(CONFIG_SIPHASH_KUNIT_TEST) += siphash_kunit.o
obj-$(CONFIG_SLUB_KUNIT_TEST) += slub_kunit.o
+obj-$(CONFIG_STACKDEPOT_KUNIT_TEST) += stackdepot_kunit.o
obj-$(CONFIG_TEST_SORT) += test_sort.o
CFLAGS_stackinit_kunit.o += $(call cc-disable-warning, switch-unreachable)
obj-$(CONFIG_STACKINIT_KUNIT_TEST) += stackinit_kunit.o
diff --git a/lib/tests/stackdepot_kunit.c b/lib/tests/stackdepot_kunit.c
new file mode 100644
index 0000000000000..be14cae98fcfb
--- /dev/null
+++ b/lib/tests/stackdepot_kunit.c
@@ -0,0 +1,473 @@
+// SPDX-License-Identifier: GPL-2.0-only
+
+#include <kunit/test.h>
+#include <linux/array_size.h>
+#include <linux/gfp.h>
+#include <linux/limits.h>
+#include <linux/moduleparam.h>
+#include <linux/stackdepot.h>
+#include <linux/stacktrace.h>
+#include <linux/string.h>
+
+#include <asm/stackdepot.h>
+
+static int expected_trie_pool_limit = -1;
+module_param_named(trie_pool_limit, expected_trie_pool_limit, int, 0);
+MODULE_PARM_DESC(trie_pool_limit, "Expected stackdepot hash/trie pool split");
+
+#ifdef CONFIG_ARM64
+#include <asm/sections.h>
+
+static inline unsigned long stackdepot_arm64_frame(long offset)
+{
+ return (unsigned long)((long)_text + offset);
+}
+#endif
+
+static void stackdepot_trie_max_path_roundtrip(struct kunit *test)
+{
+ union handle_parts parts;
+ unsigned long *entries;
+ unsigned long *fetched;
+ depot_stack_handle_t handle;
+ size_t size = CONFIG_STACKDEPOT_MAX_FRAMES * sizeof(*entries);
+ u32 pool_index_plus_1;
+ unsigned int i;
+
+ if (expected_trie_pool_limit < 0)
+ kunit_skip(test, "trie pool limit was not provided");
+ KUNIT_ASSERT_EQ(test, stack_depot_init(), 0);
+ entries = kunit_kcalloc(test, CONFIG_STACKDEPOT_MAX_FRAMES,
+ sizeof(*entries), GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, entries);
+ fetched = kunit_kcalloc(test, CONFIG_STACKDEPOT_MAX_FRAMES,
+ sizeof(*fetched), GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, fetched);
+ for (i = 0; i < CONFIG_STACKDEPOT_MAX_FRAMES; i++) {
+#ifdef CONFIG_ARM64
+ entries[i] = i & 1 ? 0x1000UL + i * 0x1000UL :
+ stackdepot_arm64_frame(i * 4);
+#elif defined(CONFIG_X86_64) && !defined(CONFIG_UML)
+ entries[i] = i & 1 ? 0xffff888000000000UL + i * 0x1000UL :
+ 0xffffffff10000000UL + i * 0x10UL;
+#else
+ entries[i] = 0x1000UL + i * 0x1000UL;
+#endif
+ }
+
+ handle = stack_depot_save(entries, CONFIG_STACKDEPOT_MAX_FRAMES,
+ GFP_KERNEL);
+ KUNIT_ASSERT_NE(test, handle, (depot_stack_handle_t)0);
+ parts.handle = handle;
+ pool_index_plus_1 = parts.pool_index_plus_1;
+ KUNIT_EXPECT_GT(test, pool_index_plus_1, (u32)expected_trie_pool_limit);
+ KUNIT_EXPECT_EQ(test,
+ stack_depot_fetch_into(handle, fetched,
+ CONFIG_STACKDEPOT_MAX_FRAMES),
+ (unsigned int)CONFIG_STACKDEPOT_MAX_FRAMES);
+ KUNIT_EXPECT_MEMEQ(test, fetched, entries, size);
+ KUNIT_EXPECT_EQ(test,
+ stack_depot_save(entries, CONFIG_STACKDEPOT_MAX_FRAMES,
+ GFP_KERNEL),
+ handle);
+}
+
+static void stackdepot_save_flags_public(struct kunit *test)
+{
+ unsigned long entries[] = { 0x501000UL, 0x502000UL, 0x503000UL };
+ unsigned long get_entries[] = { 0x601000UL, 0x602000UL };
+ unsigned long missing_entries[] = { 0x701000UL, 0x702000UL };
+ unsigned long fetched[ARRAY_SIZE(entries)] = {};
+ depot_stack_handle_t noalloc_handle;
+ depot_stack_handle_t overlong_handle;
+ depot_stack_handle_t plain_handle;
+ depot_stack_handle_t get_handle;
+ depot_stack_handle_t again;
+ depot_stack_handle_t extra;
+ gfp_t no_spin = GFP_NOWAIT & ~__GFP_RECLAIM;
+ unsigned long *overlong_fetched;
+ unsigned long *overlong_entries;
+ unsigned int overlong_nr = CONFIG_STACKDEPOT_MAX_FRAMES + 1;
+ unsigned int nr_entries;
+ size_t overlong_size;
+ unsigned int i;
+
+ KUNIT_ASSERT_EQ(test, stack_depot_init(), 0);
+ overlong_entries = kunit_kcalloc(test, overlong_nr,
+ sizeof(*overlong_entries), GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, overlong_entries);
+ overlong_fetched = kunit_kcalloc(test, CONFIG_STACKDEPOT_MAX_FRAMES,
+ sizeof(*overlong_fetched), GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, overlong_fetched);
+ for (i = 0; i < overlong_nr; i++)
+ overlong_entries[i] = 0x800000UL + i * 0x1000UL;
+
+ plain_handle = stack_depot_save(entries, ARRAY_SIZE(entries), GFP_KERNEL);
+ KUNIT_ASSERT_NE(test, plain_handle, (depot_stack_handle_t)0);
+ again = stack_depot_save(entries, ARRAY_SIZE(entries), GFP_KERNEL);
+ KUNIT_EXPECT_EQ(test, again, plain_handle);
+
+ nr_entries = stack_depot_fetch_into(plain_handle, fetched,
+ ARRAY_SIZE(fetched));
+ KUNIT_EXPECT_EQ(test, nr_entries, (unsigned int)ARRAY_SIZE(entries));
+ KUNIT_EXPECT_MEMEQ(test, fetched, entries, sizeof(entries));
+
+ noalloc_handle = stack_depot_save_flags(entries, ARRAY_SIZE(entries), no_spin, 0);
+ KUNIT_EXPECT_EQ(test, noalloc_handle, plain_handle);
+ if (expected_trie_pool_limit >= 0) {
+ noalloc_handle =
+ stack_depot_save_flags(missing_entries,
+ ARRAY_SIZE(missing_entries),
+ no_spin, 0);
+ KUNIT_EXPECT_EQ(test, noalloc_handle, (depot_stack_handle_t)0);
+ }
+
+ get_handle = stack_depot_save_flags(get_entries, ARRAY_SIZE(get_entries),
+ GFP_KERNEL,
+ STACK_DEPOT_FLAG_CAN_ALLOC |
+ STACK_DEPOT_FLAG_GET);
+ KUNIT_ASSERT_NE(test, get_handle, (depot_stack_handle_t)0);
+ stack_depot_put(get_handle);
+
+ overlong_handle = stack_depot_save(overlong_entries, overlong_nr,
+ GFP_KERNEL);
+ KUNIT_ASSERT_NE(test, overlong_handle, (depot_stack_handle_t)0);
+ nr_entries = stack_depot_fetch_into(overlong_handle, overlong_fetched,
+ CONFIG_STACKDEPOT_MAX_FRAMES);
+ KUNIT_EXPECT_EQ(test, nr_entries, (unsigned int)CONFIG_STACKDEPOT_MAX_FRAMES);
+ overlong_size = CONFIG_STACKDEPOT_MAX_FRAMES * sizeof(*overlong_entries);
+ KUNIT_EXPECT_MEMEQ(test, overlong_fetched, overlong_entries, overlong_size);
+
+ extra = stack_depot_set_extra_bits(plain_handle, 7);
+ KUNIT_ASSERT_NE(test, extra, (depot_stack_handle_t)0);
+ KUNIT_EXPECT_EQ(test, stack_depot_get_extra_bits(extra), 7U);
+ memset(fetched, 0, sizeof(fetched));
+ nr_entries = stack_depot_fetch_into(extra, fetched, ARRAY_SIZE(fetched));
+ KUNIT_EXPECT_EQ(test, nr_entries, (unsigned int)ARRAY_SIZE(entries));
+ KUNIT_EXPECT_MEMEQ(test, fetched, entries, sizeof(entries));
+}
+
+static void stackdepot_snprint_public(struct kunit *test)
+{
+ unsigned long entries[] = { 0x1000UL, 0x2000UL, 0x3000UL };
+ char expected[256];
+ char actual[256];
+ depot_stack_handle_t handle;
+ unsigned int expected_len;
+ int actual_len;
+
+ KUNIT_ASSERT_EQ(test, stack_depot_init(), 0);
+ handle = stack_depot_save(entries, ARRAY_SIZE(entries), GFP_KERNEL);
+ KUNIT_ASSERT_NE(test, handle, (depot_stack_handle_t)0);
+
+ expected_len = stack_trace_snprint(expected, sizeof(expected), entries,
+ ARRAY_SIZE(entries), 2);
+ actual_len = stack_depot_snprint(handle, actual, sizeof(actual), 2);
+ KUNIT_EXPECT_EQ(test, actual_len, (int)expected_len);
+ KUNIT_EXPECT_STREQ(test, actual, expected);
+}
+
+static void stackdepot_countable_public(struct kunit *test)
+{
+ unsigned long plain_entries[] = {
+ 0x141000UL,
+ 0x142000UL,
+ 0x143000UL,
+ };
+ unsigned long get_entries[] = {
+ 0x151000UL,
+ 0x152000UL,
+ 0x153000UL,
+ };
+ unsigned long fetched[ARRAY_SIZE(plain_entries)] = {};
+ depot_flags_t countable = STACK_DEPOT_FLAG_CAN_ALLOC |
+ STACK_DEPOT_FLAG_COUNTABLE;
+ struct stack_record *record;
+ depot_stack_handle_t count_handle;
+ depot_stack_handle_t plain_handle;
+ depot_stack_handle_t get_handle;
+ unsigned int get_nr = ARRAY_SIZE(get_entries);
+ unsigned int plain_nr = ARRAY_SIZE(plain_entries);
+ unsigned int nr_entries;
+
+ KUNIT_ASSERT_EQ(test, stack_depot_init(), 0);
+
+ plain_handle = stack_depot_save(plain_entries, plain_nr, GFP_KERNEL);
+ KUNIT_ASSERT_NE(test, plain_handle, (depot_stack_handle_t)0);
+ count_handle = stack_depot_save_flags(plain_entries, plain_nr, GFP_KERNEL,
+ countable);
+ KUNIT_ASSERT_NE(test, count_handle, (depot_stack_handle_t)0);
+ record = __stack_depot_get_stack_record(count_handle);
+ KUNIT_ASSERT_NOT_NULL(test, record);
+ KUNIT_EXPECT_EQ(test, record->size, (u16)plain_nr);
+ KUNIT_EXPECT_MEMEQ(test, record->entries, plain_entries,
+ sizeof(plain_entries));
+ nr_entries = stack_depot_fetch_into(count_handle, fetched,
+ ARRAY_SIZE(fetched));
+ KUNIT_EXPECT_EQ(test, nr_entries, plain_nr);
+ KUNIT_EXPECT_MEMEQ(test, fetched, plain_entries, sizeof(plain_entries));
+
+ get_handle = stack_depot_save_flags(get_entries, get_nr, GFP_KERNEL,
+ STACK_DEPOT_FLAG_CAN_ALLOC |
+ STACK_DEPOT_FLAG_GET);
+ KUNIT_ASSERT_NE(test, get_handle, (depot_stack_handle_t)0);
+ count_handle = stack_depot_save_flags(get_entries, get_nr, GFP_KERNEL,
+ countable);
+ KUNIT_ASSERT_NE(test, count_handle, (depot_stack_handle_t)0);
+ record = __stack_depot_get_stack_record(count_handle);
+ KUNIT_ASSERT_NOT_NULL(test, record);
+ KUNIT_EXPECT_MEMEQ(test, record->entries, get_entries, sizeof(get_entries));
+
+ stack_depot_put(get_handle);
+}
+
+static void stackdepot_fetch_into_roundtrip(struct kunit *test)
+{
+ unsigned long entries[] = {
+ 0x101000UL,
+ 0x102000UL,
+ 0x103000UL,
+ };
+ unsigned long exact[ARRAY_SIZE(entries)] = {};
+ unsigned long fetched[ARRAY_SIZE(entries) + 1] = {
+ [ARRAY_SIZE(entries)] = 0xa5a5a5a5UL,
+ };
+ unsigned long expected_tail = fetched[ARRAY_SIZE(entries)];
+ depot_stack_handle_t handle;
+ unsigned int nr_entries;
+
+ KUNIT_ASSERT_EQ(test, stack_depot_init(), 0);
+
+ handle = stack_depot_save(entries, ARRAY_SIZE(entries), GFP_KERNEL);
+ KUNIT_ASSERT_NE(test, handle, (depot_stack_handle_t)0);
+
+ nr_entries = stack_depot_fetch_into(handle, exact, ARRAY_SIZE(exact));
+ KUNIT_EXPECT_EQ(test, nr_entries, (unsigned int)ARRAY_SIZE(entries));
+ KUNIT_EXPECT_MEMEQ(test, exact, entries, sizeof(entries));
+
+ nr_entries = stack_depot_fetch_into(handle, fetched, ARRAY_SIZE(fetched));
+ KUNIT_EXPECT_EQ(test, nr_entries, (unsigned int)ARRAY_SIZE(entries));
+ KUNIT_EXPECT_MEMEQ(test, fetched, entries, sizeof(entries));
+ KUNIT_EXPECT_EQ(test, fetched[ARRAY_SIZE(entries)], expected_tail);
+}
+
+static void stackdepot_fetch_into_rejects_missing_or_short_stack(struct kunit *test)
+{
+ unsigned long entries[] = {
+ 0x111000UL,
+ 0x112000UL,
+ 0x113000UL,
+ };
+ unsigned long fetched[ARRAY_SIZE(entries)] = {
+ 0xa1a1a1a1UL,
+ 0xb2b2b2b2UL,
+ 0xc3c3c3c3UL,
+ };
+ unsigned long expected[ARRAY_SIZE(fetched)];
+ depot_stack_handle_t handle;
+ unsigned int nr_entries;
+
+ KUNIT_ASSERT_EQ(test, stack_depot_init(), 0);
+
+ handle = stack_depot_save(entries, ARRAY_SIZE(entries), GFP_KERNEL);
+ KUNIT_ASSERT_NE(test, handle, (depot_stack_handle_t)0);
+ memcpy(expected, fetched, sizeof(expected));
+
+ nr_entries = stack_depot_fetch_into(0, fetched, ARRAY_SIZE(fetched));
+ KUNIT_EXPECT_EQ(test, nr_entries, 0U);
+ KUNIT_EXPECT_MEMEQ(test, fetched, expected, sizeof(expected));
+
+ nr_entries = stack_depot_fetch_into(0, NULL, 0);
+ KUNIT_EXPECT_EQ(test, nr_entries, 0U);
+
+ nr_entries = stack_depot_fetch_into(handle, fetched,
+ ARRAY_SIZE(fetched) - 1);
+ KUNIT_EXPECT_EQ(test, nr_entries, 0U);
+ KUNIT_EXPECT_MEMEQ(test, fetched, expected, sizeof(expected));
+}
+
+static void stackdepot_trie_topology_roundtrip(struct kunit *test)
+{
+ union handle_parts parts;
+ unsigned long stacks[][3] = {
+ { 0x201000UL, 0x202000UL },
+ { 0x201000UL, 0x203000UL },
+ { 0x201000UL },
+ { 0x201000UL, 0x203000UL, 0x204000UL },
+ { 0x201000UL, 0x205000UL },
+ { 0x201000UL, 0x204000UL },
+ { 0x201000UL, 0x206000UL },
+ { 0x201000UL, 0x207000UL },
+ { 0x301000UL, 0x302000UL },
+ { 0x301000UL, 0x302000UL, 0x303000UL },
+ { 0x301000UL, 0x304000UL },
+ { 0x401000UL, 0x402000UL, 0x403000UL },
+ { 0x401000UL, 0x402000UL },
+ };
+ unsigned int nr_entries[] = { 2, 2, 1, 3, 2, 2, 2, 2, 2, 3, 2, 3, 2 };
+ depot_stack_handle_t handles[ARRAY_SIZE(stacks)];
+ unsigned long fetched[ARRAY_SIZE(stacks[0])];
+ u32 pool_index_plus_1;
+ unsigned int i;
+
+ if (expected_trie_pool_limit < 0)
+ kunit_skip(test, "trie pool limit was not provided");
+ KUNIT_ASSERT_EQ(test, stack_depot_init(), 0);
+
+ for (i = 0; i < ARRAY_SIZE(stacks); i++) {
+ handles[i] = stack_depot_save(stacks[i], nr_entries[i], GFP_KERNEL);
+ KUNIT_ASSERT_NE(test, handles[i], (depot_stack_handle_t)0);
+ }
+ parts.handle = handles[0];
+ pool_index_plus_1 = parts.pool_index_plus_1;
+ KUNIT_ASSERT_GT(test, pool_index_plus_1,
+ (u32)expected_trie_pool_limit);
+
+ for (i = 0; i < ARRAY_SIZE(stacks); i++) {
+ memset(fetched, 0, sizeof(fetched));
+ KUNIT_EXPECT_EQ(test,
+ stack_depot_fetch_into(handles[i], fetched,
+ ARRAY_SIZE(fetched)),
+ nr_entries[i]);
+ KUNIT_EXPECT_MEMEQ(test, fetched, stacks[i],
+ nr_entries[i] * sizeof(fetched[0]));
+ KUNIT_EXPECT_EQ(test,
+ stack_depot_save(stacks[i], nr_entries[i], GFP_KERNEL),
+ handles[i]);
+ }
+}
+
+static void stackdepot_frame_storage_roundtrip(struct kunit *test)
+{
+ union handle_parts parts;
+ unsigned long fetched[3] = {};
+ depot_stack_handle_t handle;
+ u32 pool_index_plus_1;
+ unsigned int nr_entries;
+#if defined(CONFIG_ARM64)
+ unsigned long entries[] = {
+ stackdepot_arm64_frame(S32_MIN),
+ 0x1000UL,
+ stackdepot_arm64_frame(S32_MAX),
+ };
+#elif defined(CONFIG_X86_64)
+ unsigned long entries[] = {
+ 0xffffffff10001000UL,
+ 0xffff888000001000UL,
+ 0xffffffff20002000UL,
+ };
+#else
+ unsigned long entries[] = { 0x301000UL, 0x302000UL, 0x303000UL };
+#endif
+
+ if (expected_trie_pool_limit < 0)
+ kunit_skip(test, "trie pool limit was not provided");
+ KUNIT_ASSERT_EQ(test, stack_depot_init(), 0);
+ handle = stack_depot_save(entries, ARRAY_SIZE(entries), GFP_KERNEL);
+ KUNIT_ASSERT_NE(test, handle, (depot_stack_handle_t)0);
+ parts.handle = handle;
+ pool_index_plus_1 = parts.pool_index_plus_1;
+ KUNIT_ASSERT_GT(test, pool_index_plus_1,
+ (u32)expected_trie_pool_limit);
+
+ nr_entries = stack_depot_fetch_into(handle, fetched, ARRAY_SIZE(fetched));
+ KUNIT_EXPECT_EQ(test, nr_entries, (unsigned int)ARRAY_SIZE(entries));
+ KUNIT_EXPECT_MEMEQ(test, fetched, entries, sizeof(entries));
+}
+
+static void stackdepot_frame_raw_fallback(struct kunit *test)
+{
+ unsigned long frame = 0x1000UL;
+ bool compressed;
+ u32 payload;
+
+#ifdef CONFIG_ARM64
+ frame = (unsigned long)_text + (unsigned long)S32_MAX + 1UL;
+#endif
+
+ compressed = arch_stack_depot_frame_try_compress(frame, &payload);
+ KUNIT_EXPECT_FALSE(test, compressed);
+}
+
+#if defined(CONFIG_X86_64) && !defined(CONFIG_UML)
+static void stackdepot_frame_x86_64(struct kunit *test)
+{
+ unsigned long direct_map = 0xffff888000001000UL;
+ unsigned long frame = 0xffffffff81234567UL;
+ unsigned long out;
+ bool compressed;
+ u32 low;
+
+ compressed = arch_stack_depot_frame_try_compress(frame, &low);
+ KUNIT_EXPECT_TRUE(test, compressed);
+ KUNIT_EXPECT_EQ(test, low, (u32)0x81234567);
+ arch_stack_depot_frame_decompress(low, &out);
+ KUNIT_EXPECT_EQ(test, out, frame);
+
+ compressed = arch_stack_depot_frame_try_compress(direct_map, &low);
+ KUNIT_EXPECT_FALSE(test, compressed);
+}
+#endif /* CONFIG_X86_64 && !CONFIG_UML */
+
+#ifdef CONFIG_ARM64
+static void stackdepot_frame_arm64(struct kunit *test)
+{
+ long negative_offset = S32_MIN;
+ long positive_offset = S32_MAX;
+ long offset = 0x123456;
+ unsigned long frame = stackdepot_arm64_frame(offset);
+ unsigned long out;
+ bool compressed;
+ u32 payload;
+
+ compressed = arch_stack_depot_frame_try_compress(frame, &payload);
+ KUNIT_EXPECT_TRUE(test, compressed);
+ KUNIT_EXPECT_EQ(test, payload, (u32)(s32)offset);
+ arch_stack_depot_frame_decompress(payload, &out);
+ KUNIT_EXPECT_EQ(test, out, frame);
+
+ frame = stackdepot_arm64_frame(negative_offset);
+ compressed = arch_stack_depot_frame_try_compress(frame, &payload);
+ KUNIT_EXPECT_TRUE(test, compressed);
+ KUNIT_EXPECT_EQ(test, payload, (u32)(s32)negative_offset);
+ arch_stack_depot_frame_decompress(payload, &out);
+ KUNIT_EXPECT_EQ(test, out, frame);
+
+ frame = stackdepot_arm64_frame(positive_offset);
+ compressed = arch_stack_depot_frame_try_compress(frame, &payload);
+ KUNIT_EXPECT_TRUE(test, compressed);
+ KUNIT_EXPECT_EQ(test, payload, (u32)(s32)positive_offset);
+ arch_stack_depot_frame_decompress(payload, &out);
+ KUNIT_EXPECT_EQ(test, out, frame);
+}
+#endif /* CONFIG_ARM64 */
+
+static struct kunit_case stackdepot_test_cases[] = {
+ KUNIT_CASE(stackdepot_trie_max_path_roundtrip),
+ KUNIT_CASE(stackdepot_save_flags_public),
+ KUNIT_CASE(stackdepot_snprint_public),
+ KUNIT_CASE(stackdepot_countable_public),
+ KUNIT_CASE(stackdepot_fetch_into_roundtrip),
+ KUNIT_CASE(stackdepot_fetch_into_rejects_missing_or_short_stack),
+ KUNIT_CASE(stackdepot_trie_topology_roundtrip),
+ KUNIT_CASE(stackdepot_frame_storage_roundtrip),
+ KUNIT_CASE(stackdepot_frame_raw_fallback),
+#if defined(CONFIG_X86_64) && !defined(CONFIG_UML)
+ KUNIT_CASE(stackdepot_frame_x86_64),
+#endif
+#ifdef CONFIG_ARM64
+ KUNIT_CASE(stackdepot_frame_arm64),
+#endif
+ {}
+};
+
+static struct kunit_suite stackdepot_test_suite = {
+ .name = "stackdepot",
+ .test_cases = stackdepot_test_cases,
+};
+
+kunit_test_suite(stackdepot_test_suite);
+
+MODULE_DESCRIPTION("KUnit tests for stack depot");
+MODULE_AUTHOR("Caleb Kan <ckan@cloudflare.com>");
+MODULE_LICENSE("GPL");
diff --git a/mm/kmemleak.c b/mm/kmemleak.c
index 8fa409a4f9fb2..c42741a88bd42 100644
--- a/mm/kmemleak.c
+++ b/mm/kmemleak.c
@@ -378,10 +378,10 @@ static void __print_unreferenced(struct seq_file *seq,
bool hex_dump)
{
int i;
- unsigned long *entries;
+ unsigned long entries[MAX_TRACE];
unsigned int nr_entries;
- nr_entries = stack_depot_fetch(object->trace_handle, &entries);
+ nr_entries = stack_depot_fetch_into(object->trace_handle, entries, ARRAY_SIZE(entries));
warn_or_seq_printf(seq, "unreferenced object%s 0x%08lx (size %zu):\n",
__object_type_str(object),
object->pointer, object->size);
diff --git a/mm/kmsan/kmsan_test.c b/mm/kmsan/kmsan_test.c
index 31f47cc4dab40..7c04e4b21873d 100644
--- a/mm/kmsan/kmsan_test.c
+++ b/mm/kmsan/kmsan_test.c
@@ -669,7 +669,7 @@ static void test_long_origin_chain(struct kunit *test)
*/
static void test_stackdepot_roundtrip(struct kunit *test)
{
- unsigned long src_entries[16], *dst_entries;
+ unsigned long src_entries[16], dst_entries[16];
unsigned int src_nentries, dst_nentries;
EXPECTATION_NO_REPORT(expect);
depot_stack_handle_t handle;
@@ -680,7 +680,7 @@ static void test_stackdepot_roundtrip(struct kunit *test)
stack_trace_save(src_entries, ARRAY_SIZE(src_entries), 1);
handle = stack_depot_save(src_entries, src_nentries, GFP_KERNEL);
stack_depot_print(handle);
- dst_nentries = stack_depot_fetch(handle, &dst_entries);
+ dst_nentries = stack_depot_fetch_into(handle, dst_entries, ARRAY_SIZE(dst_entries));
KUNIT_EXPECT_TRUE(test, src_nentries == dst_nentries);
kmsan_check_memory((void *)dst_entries,
diff --git a/mm/kmsan/report.c b/mm/kmsan/report.c
index d6853ce089541..c20c24cffde55 100644
--- a/mm/kmsan/report.c
+++ b/mm/kmsan/report.c
@@ -85,7 +85,7 @@ static char *pretty_descr(char *descr)
void kmsan_print_origin(depot_stack_handle_t origin)
{
- unsigned long *entries = NULL, *chained_entries = NULL;
+ unsigned long entries[KMSAN_STACK_DEPTH];
unsigned int nr_entries, chained_nr_entries, skipnr;
void *pc1 = NULL, *pc2 = NULL;
depot_stack_handle_t head;
@@ -97,7 +97,8 @@ void kmsan_print_origin(depot_stack_handle_t origin)
return;
while (true) {
- nr_entries = stack_depot_fetch(origin, &entries);
+ nr_entries =
+ stack_depot_fetch_into(origin, entries, ARRAY_SIZE(entries));
depth = kmsan_depth_from_eb(stack_depot_get_extra_bits(origin));
magic = nr_entries ? entries[0] : 0;
if ((nr_entries == 4) && (magic == KMSAN_ALLOCA_MAGIC_ORIGIN)) {
@@ -123,14 +124,10 @@ void kmsan_print_origin(depot_stack_handle_t origin)
origin = entries[2];
pr_err("Uninit was stored to memory at:\n");
chained_nr_entries =
- stack_depot_fetch(head, &chained_entries);
- kmsan_internal_unpoison_memory(
- chained_entries,
- chained_nr_entries * sizeof(*chained_entries),
- /*checked*/ false);
- skipnr = get_stack_skipnr(chained_entries,
- chained_nr_entries);
- stack_trace_print(chained_entries + skipnr,
+ stack_depot_fetch_into(head, entries,
+ ARRAY_SIZE(entries));
+ skipnr = get_stack_skipnr(entries, chained_nr_entries);
+ stack_trace_print(entries + skipnr,
chained_nr_entries - skipnr, 0);
pr_err("\n");
continue;
diff --git a/mm/page_owner.c b/mm/page_owner.c
index fbbda7ba914ba..af37532729b02 100644
--- a/mm/page_owner.c
+++ b/mm/page_owner.c
@@ -119,7 +119,8 @@ static __always_inline depot_stack_handle_t create_dummy_stack(void)
unsigned int nr_entries;
nr_entries = stack_trace_save(entries, ARRAY_SIZE(entries), 0);
- return stack_depot_save(entries, nr_entries, GFP_KERNEL);
+ return stack_depot_save_flags(entries, nr_entries, GFP_KERNEL,
+ STACK_DEPOT_FLAG_CAN_ALLOC | STACK_DEPOT_FLAG_COUNTABLE);
}
static noinline void register_dummy_stack(void)
@@ -181,7 +182,8 @@ static noinline depot_stack_handle_t save_stack(gfp_t flags)
set_current_in_page_owner();
nr_entries = stack_trace_save(entries, ARRAY_SIZE(entries), 2);
- handle = stack_depot_save(entries, nr_entries, flags);
+ handle = stack_depot_save_flags(entries, nr_entries, flags,
+ STACK_DEPOT_FLAG_CAN_ALLOC | STACK_DEPOT_FLAG_COUNTABLE);
if (!handle)
handle = failure_handle;
unset_current_in_page_owner();
diff --git a/mm/slub.c b/mm/slub.c
index 422bc3e12c02c..138c3bc473c9c 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -8093,12 +8093,12 @@ void __kmem_obj_info(struct kmem_obj_info *kpp, void *object, struct slab *slab)
#ifdef CONFIG_STACKDEPOT
{
depot_stack_handle_t handle;
- unsigned long *entries;
+ unsigned long entries[TRACK_ADDRS_COUNT];
unsigned int nr_entries;
handle = READ_ONCE(trackp->handle);
if (handle) {
- nr_entries = stack_depot_fetch(handle, &entries);
+ nr_entries = stack_depot_fetch_into(handle, entries, ARRAY_SIZE(entries));
for (i = 0; i < KS_ADDRS_COUNT && i < nr_entries; i++)
kpp->kp_stack[i] = (void *)entries[i];
}
@@ -8106,7 +8106,7 @@ void __kmem_obj_info(struct kmem_obj_info *kpp, void *object, struct slab *slab)
trackp = get_track(s, objp, TRACK_FREE);
handle = READ_ONCE(trackp->handle);
if (handle) {
- nr_entries = stack_depot_fetch(handle, &entries);
+ nr_entries = stack_depot_fetch_into(handle, entries, ARRAY_SIZE(entries));
for (i = 0; i < KS_ADDRS_COUNT && i < nr_entries; i++)
kpp->kp_free_stack[i] = (void *)entries[i];
}
@@ -9815,12 +9815,14 @@ static int slab_debugfs_show(struct seq_file *seq, void *v)
#ifdef CONFIG_STACKDEPOT
{
depot_stack_handle_t handle;
- unsigned long *entries;
+ unsigned long entries[TRACK_ADDRS_COUNT];
unsigned int nr_entries, j;
handle = READ_ONCE(l->handle);
if (handle) {
- nr_entries = stack_depot_fetch(handle, &entries);
+ nr_entries =
+ stack_depot_fetch_into(handle, entries,
+ ARRAY_SIZE(entries));
seq_puts(seq, "\n");
for (j = 0; j < nr_entries; j++)
seq_printf(seq, " %pS\n", (void *)entries[j]);
diff --git a/scripts/gdb/linux/stackdepot.py b/scripts/gdb/linux/stackdepot.py
index 37313a5a51a0d..82aeb9f532c3d 100644
--- a/scripts/gdb/linux/stackdepot.py
+++ b/scripts/gdb/linux/stackdepot.py
@@ -37,6 +37,10 @@ def stack_depot_fetch(handle):
if handle == 0:
raise gdb.GdbError("handle is 0\n")
+ stack_max_pools = gdb.parse_and_eval('stack_max_pools')
+ if parts['pool_index_plus_1'] > stack_max_pools:
+ raise gdb.GdbError("trie-backed stack depot handles are not supported\n")
+
pool_index = parts['pool_index_plus_1'] - 1
if pool_index >= pools_num:
gdb.write("pool index %d out of bounds (%d) for stack id 0x%08x\n" % (parts['pool_index'], pools_num, handle))
|
| 3/2 | 2026/08/17 12:57 | llm | patch-evaluator |
0mModel:gemini-3.1-pro-preview Args: null Results: null Tokens: input: 36840 output: 27 thoughts: 607 Thoughts: **Fuzzing the Stack Depot Trie Implementation** Okay, so I've got this patch series for the Linux kernel that introduces a new "trie" backend for the `stackdepot` subsystem. The big question is: does this warrant fuzzing? Absolutely, yes. `lib/stackdepot.c` is a core library, critical for tools like KASAN, KMSAN, page_owner, and SLUB debugging. The changes are pretty extensive and touch on fundamental functionality. My immediate thought is to zero in on specific functions within `lib/stackdepot.c`. Let's see: `stack_depot_save_flags`, `stack_depot_save`, `stack_depot_fetch`, `stack_depot_fetch_into`, `stack_depot_print`, `stack_depot_snprint`, and `stack_depot_put`. `stack_depot_save` and `stack_depot_save_flags` look like good entry points for fuzzing, given that they're the points of entry into the system. Now, about this trie enablement. It's triggered by the `stackdepot.trie_enabled=1` boot parameter. But the fuzzing prompt asks for `EnableConfigs`, right? The patch also includes a KUnit test, `STACKDEPOT_KUNIT_TEST`, which is interesting. So, how do we *enable* `stackdepot` in the first place? I know it's often indirectly selected by options like `CONFIG_KASAN`, `CONFIG_PAGE_OWNER`, or `CONFIG_DEBUG_KMEMLEAK`. I'll need to check `lib/Kconfig` or `lib/Kconfig.debug` to confirm. The KUnit test is worth considering, though. Normally, you wouldn't run a KUnit test directly with a fuzzer, but defining it in `EnableConfigs` might compile the code we need. A quick check of `lib/Kconfig` shows that `STACKDEPOT` is usually a hidden configuration. So, enabling the KUnit test seems like the key here. |
| 4/2 | 2026/08/17 12:57 | tool | grepper |
0mArgs:{
"Expression": "config STACKDEPOT",
"PathPrefix": "lib/"
}Results: {
"Output": "lib/Kconfig=557=config ARCH_STACKWALK\n--\nlib/Kconfig-559-\nlib/Kconfig:560:config STACKDEPOT\nlib/Kconfig-561-\tbool\n--\nlib/Kconfig-565-\nlib/Kconfig:566:config STACKDEPOT_ALWAYS_INIT\nlib/Kconfig-567-\tbool\n--\nlib/Kconfig-571-\nlib/Kconfig:572:config STACKDEPOT_MAX_FRAMES\nlib/Kconfig-573-\tint \"Maximum number of frames in trace saved in stack depot\"\n--\nlib/Kconfig.debug=2761=config RESOURCE_KUNIT_TEST\n--\nlib/Kconfig.debug-2773-\nlib/Kconfig.debug:2774:config STACKDEPOT_KUNIT_TEST\nlib/Kconfig.debug-2775-\tbool \"KUnit test for stack depot\" if !KUNIT_ALL_TESTS\n"
} |
| 5/2 | 2026/08/17 12:57 | llm | patch-evaluator |
0mModel:gemini-3.1-pro-preview Args: null Results: null Tokens: input: 38191 output: 25 thoughts: 32 Thoughts: Okay, here's my interpretation of that thought process, as if it were my own: **Investigating the `STACKDEPOT` Configuration** Right, so `STACKDEPOT` is a boolean configuration option. It's automatically enabled, and doesn't seem to have a direct user prompt associated with it. That means it's likely being enabled based on the settings of other configurations. My immediate thought is, let's trace its dependencies. I need to understand *why* `STACKDEPOT` is being chosen. What other settings are triggering this one? Understanding that will give me the context I need to figure out its purpose and whether its selection is correct in the broader system. Time to delve into the configuration dependencies and figure out the upstream logic. |
| 6/2 | 2026/08/17 12:58 | tool | grepper |
0mArgs:{
"Expression": "select STACKDEPOT",
"PathPrefix": ""
}Results: {
"Output": "drivers/gpu/drm/Kconfig=143=config DRM_DEBUG_DP_MST_TOPOLOGY_REFS\n--\ndrivers/gpu/drm/Kconfig-145-\tdepends on STACKTRACE_SUPPORT\ndrivers/gpu/drm/Kconfig:146: select STACKDEPOT\ndrivers/gpu/drm/Kconfig-147- select DRM_KMS_HELPER\n--\ndrivers/gpu/drm/Kconfig=158=config DRM_DEBUG_MODESET_LOCK\n--\ndrivers/gpu/drm/Kconfig-162-\tdepends on EXPERT\ndrivers/gpu/drm/Kconfig:163:\tselect STACKDEPOT\ndrivers/gpu/drm/Kconfig-164-\tdefault y if DEBUG_WW_MUTEX_SLOWPATH\n--\ndrivers/gpu/drm/Kconfig.debug=39=config DRM_DEBUG_MM\n--\ndrivers/gpu/drm/Kconfig.debug-43-\tdepends on STACKTRACE_SUPPORT\ndrivers/gpu/drm/Kconfig.debug:44:\tselect STACKDEPOT\ndrivers/gpu/drm/Kconfig.debug-45-\thelp\n--\ndrivers/gpu/drm/i915/Kconfig.debug=36=config DRM_I915_DEBUG\n--\ndrivers/gpu/drm/i915/Kconfig.debug-44-\tselect REF_TRACKER\ndrivers/gpu/drm/i915/Kconfig.debug:45:\tselect STACKDEPOT\ndrivers/gpu/drm/i915/Kconfig.debug-46-\tselect STACKTRACE\n--\ndrivers/gpu/drm/i915/Kconfig.debug=178=config DRM_I915_DEBUG_GUC\n--\ndrivers/gpu/drm/i915/Kconfig.debug-180-\tdepends on DRM_I915\ndrivers/gpu/drm/i915/Kconfig.debug:181:\tselect STACKDEPOT\ndrivers/gpu/drm/i915/Kconfig.debug-182-\tdefault n\n--\ndrivers/gpu/drm/i915/Kconfig.debug=251=config DRM_I915_DEBUG_RUNTIME_PM\n--\ndrivers/gpu/drm/i915/Kconfig.debug-255-\tselect REF_TRACKER\ndrivers/gpu/drm/i915/Kconfig.debug:256:\tselect STACKDEPOT\ndrivers/gpu/drm/i915/Kconfig.debug-257-\tselect STACKTRACE\n--\ndrivers/gpu/drm/i915/Kconfig.debug=267=config DRM_I915_DEBUG_WAKEREF\n--\ndrivers/gpu/drm/i915/Kconfig.debug-270-\tselect REF_TRACKER\ndrivers/gpu/drm/i915/Kconfig.debug:271:\tselect STACKDEPOT\ndrivers/gpu/drm/i915/Kconfig.debug-272-\tselect STACKTRACE\n--\ndrivers/gpu/drm/xe/Kconfig.debug=89=config DRM_XE_DEBUG_GUC\n--\ndrivers/gpu/drm/xe/Kconfig.debug-92- default n\ndrivers/gpu/drm/xe/Kconfig.debug:93: select STACKDEPOT\ndrivers/gpu/drm/xe/Kconfig.debug-94- help\n--\nlib/Kconfig=566=config STACKDEPOT_ALWAYS_INIT\nlib/Kconfig-567-\tbool\nlib/Kconfig:568:\tselect STACKDEPOT\nlib/Kconfig-569-\thelp\n--\nlib/Kconfig=578=config REF_TRACKER\n--\nlib/Kconfig-580-\tdepends on STACKTRACE_SUPPORT\nlib/Kconfig:581:\tselect STACKDEPOT\nlib/Kconfig-582-\n--\nlib/Kconfig.kasan=46=menuconfig KASAN\n--\nlib/Kconfig.kasan-52-\tdepends on SYSFS \u0026\u0026 !SLUB_TINY\nlib/Kconfig.kasan:53:\tselect STACKDEPOT_ALWAYS_INIT\nlib/Kconfig.kasan-54-\thelp\n--\nlib/Kconfig.kmsan=8=config KMSAN\n--\nlib/Kconfig.kmsan-12-\tdepends on !PREEMPT_RT\nlib/Kconfig.kmsan:13:\tselect STACKDEPOT\nlib/Kconfig.kmsan:14:\tselect STACKDEPOT_ALWAYS_INIT\nlib/Kconfig.kmsan-15-\thelp\n--\nmm/Kconfig.debug=48=config SLUB_DEBUG\n--\nmm/Kconfig.debug-51-\tdepends on SYSFS \u0026\u0026 !SLUB_TINY\nmm/Kconfig.debug:52:\tselect STACKDEPOT if STACKTRACE_SUPPORT\nmm/Kconfig.debug-53-\thelp\n--\nmm/Kconfig.debug=59=config SLUB_DEBUG_ON\n--\nmm/Kconfig.debug-61-\tdepends on SLUB_DEBUG\nmm/Kconfig.debug:62:\tselect STACKDEPOT_ALWAYS_INIT if STACKTRACE_SUPPORT\nmm/Kconfig.debug-63-\tdefault n\n--\nmm/Kconfig.debug=105=config PAGE_OWNER\n--\nmm/Kconfig.debug-109-\tselect STACKTRACE\nmm/Kconfig.debug:110:\tselect STACKDEPOT\nmm/Kconfig.debug-111-\tselect PAGE_EXTENSION\n--\nmm/Kconfig.debug=240=config DEBUG_KMEMLEAK\n--\nmm/Kconfig.debug-246-\tselect CRC32\nmm/Kconfig.debug:247:\tselect STACKDEPOT\nmm/Kconfig.debug:248:\tselect STACKDEPOT_ALWAYS_INIT if !DEBUG_KMEMLEAK_DEFAULT_OFF\nmm/Kconfig.debug-249-\thelp\n"
} |
| 7/2 | 2026/08/17 12:58 | llm | patch-evaluator |
0mModel:gemini-3.1-pro-preview Args: null Results: null Tokens: input: 39887 output: 36 thoughts: 340 Thoughts: **My Configuration Considerations for Stack Depot** Okay, so I'm looking at how `STACKDEPOT` gets enabled in the kernel. Seems like `PAGE_OWNER`, `DEBUG_KMEMLEAK`, `KASAN`, `KMSAN`, and `SLUB_DEBUG` all indirectly select it. My initial thought was to simply enable one of those to force compilation. `STACKDEPOT_KUNIT_TEST` looked promising, as that would compile the test code along with `STACKDEPOT` but then I remembered, `STACKDEPOT_KUNIT_TEST` *depends* on both `KUNIT` and `STACKDEPOT` themselves. So, I'd have to ensure both of those are set. Then I realized I can't select `STACKDEPOT` directly; it's an internal dependency. So, I need to pick something that *selects* it. `PAGE_OWNER` or `DEBUG_KMEMLEAK` will do the trick and I think I'll go with both to be sure. Now, this patch introduces `stack_depot_trie_enabled`, which is controlled by the boot parameter `stackdepot.trie_enabled=1`. However, the prompt is asking for "EnableConfigs", which means kernel configurations. I can't specify boot parameters directly. Hmm, but the fuzzer *might* still hit the code if the feature is enabled by default or if the fuzzer somehow provides its own boot parameters. That boot parameter, `stackdepot.trie_enabled`, is a problem. I'll need to check the code itself for any other way to enable it. I'll go look in `lib/stackdepot.c` for `stack_depot_trie_requested`. |
| 8/2 | 2026/08/17 12:58 | tool | grepper |
0mArgs:{
"Expression": "stack_depot_trie_requested",
"PathPrefix": "lib/stackdepot.c"
}Results: {
"Output": "lib/stackdepot.c=183=static DEFINE_RAW_SPINLOCK(stack_depot_trie_writer_lock);\nlib/stackdepot.c:184:static bool stack_depot_trie_requested;\nlib/stackdepot.c-185-\nlib/stackdepot.c:186:module_param_named(trie_enabled, stack_depot_trie_requested, bool, 0);\nlib/stackdepot.c-187-MODULE_PARM_DESC(trie_enabled, \"Enable stack depot trie storage at boot\");\n--\nlib/stackdepot.c=781=int __init stack_depot_early_init(void)\n--\nlib/stackdepot.c-853-\t}\nlib/stackdepot.c:854:\tif (stack_depot_trie_requested \u0026\u0026 stack_depot_trie_init_memblock()) {\nlib/stackdepot.c-855-\t\tpr_warn(\"trie storage initialization failed, disabling trie storage\\n\");\nlib/stackdepot.c:856:\t\tstack_depot_trie_requested = false;\nlib/stackdepot.c-857-\t}\n--\nlib/stackdepot.c=863=int stack_depot_init(void)\n--\nlib/stackdepot.c-917-\t}\nlib/stackdepot.c:918:\tif (stack_depot_trie_requested) {\nlib/stackdepot.c-919-\t\tret = stack_depot_trie_init();\n--\nlib/stackdepot.c-921-\t\t\tpr_warn(\"trie storage initialization failed, disabling trie storage\\n\");\nlib/stackdepot.c:922:\t\t\tstack_depot_trie_requested = false;\nlib/stackdepot.c-923-\t\t\tret = 0;\n"
} |
| 9/2 | 2026/08/17 12:58 | llm | patch-evaluator |
0mModel:gemini-3.1-pro-preview Args: null Results: null Tokens: input: 41136 output: 35 thoughts: 251 Thoughts: **My Initial Assessment of the Stack Depot Implementation** Okay, so we have this trie functionality, which is controlled by a module parameter, `stackdepot.trie_enabled`, allowing us to enable it right from the kernel command line. Good, straightforward. And the KUnit tests cover the trie storage when `expected_trie_pool_limit` is set, so that's a positive sign of test coverage. Now, let's zero in on the key functions we're dealing with: `stack_depot_save`, `stack_depot_save_flags`, `stack_depot_fetch`, `stack_depot_fetch_into`, `stack_depot_print`, `stack_depot_snprint`, and `stack_depot_put`. All of these are exported or global functions within `lib/stackdepot.c`. This suggests a clear API, which is what we would expect. My immediate concern is performance, of course. `stack_depot_save` is called on every memory allocation if KASAN or page_owner are enabled. That's a potential hot-path right there, which could be problematic. I think we need to carefully evaluate its performance impact. However, `stack_depot_fetch_into` seems like the more interesting function to analyze at this point. It's newer, and primarily used in reporting paths, like kmemleak, slub debug, and kmsan reports. That's where I will start, focusing on its usage context and performance implications within those specific debugging and reporting scenarios. |
| 10/2 | 2026/08/17 12:58 | tool | grepper |
0mArgs:{
"Expression": "stack_depot_fetch_into",
"PathPrefix": "lib/stackdepot.c"
}Results: {
"Output": "lib/stackdepot.c=2194=EXPORT_SYMBOL_GPL(stack_depot_fetch);\nlib/stackdepot.c-2195-\nlib/stackdepot.c:2196:unsigned int stack_depot_fetch_into(depot_stack_handle_t handle,\nlib/stackdepot.c-2197-\t\t\t\t unsigned long *entries,\n--\nlib/stackdepot.c-2223-}\nlib/stackdepot.c:2224:EXPORT_SYMBOL_GPL(stack_depot_fetch_into);\nlib/stackdepot.c-2225-\n"
} |
| 11/2 | 2026/08/17 12:58 | llm | patch-evaluator |
0mModel:gemini-3.1-pro-preview Args: null Results: null Tokens: input: 41930 output: 38 thoughts: 50 Thoughts: Okay, here's my take on those thoughts, framed as my own: **Initial Assessment of Stack Depot Changes** Right, so we've got a couple of key changes here. First, there's `stack_depot_fetch_into`, which, if I'm understanding this correctly, looks like a brand-new API. That immediately piques my interest; a new API always implies potentially significant architectural shifts, or at least a new way of interacting with the stack depot. I need to understand its design goals and how it fits into the overall system. And then, there's `stack_depot_save_flags`, and that's been modified, which is also significant. Modifying a function like this could introduce backward compatibility issues or change the fundamental behaviour of saving data. I need to understand that. Now, given these changes, my primary focus needs to be on how they impact the core logic, so let's check `__stack_depot_get_stack_record`. The impact of a new API and a modified one would be most apparent by its use of the core data-retrieval function. Let's start there. |
| 12/2 | 2026/08/17 12:58 | tool | grepper |
0mArgs:{
"Expression": "__stack_depot_get_stack_record",
"PathPrefix": "lib/stackdepot.c"
}Results: {
"Output": "lib/stackdepot.c=1469=EXPORT_SYMBOL_GPL(stack_depot_save);\nlib/stackdepot.c-1470-\nlib/stackdepot.c:1471:struct stack_record *__stack_depot_get_stack_record(depot_stack_handle_t handle)\nlib/stackdepot.c-1472-{\n"
} |
| 13/2 | 2026/08/17 12:58 | llm | patch-evaluator |
0mModel:gemini-3.1-pro-preview Args: null Results: null Tokens: input: 42372 output: 218 thoughts: 1093 Thoughts: **Analyzing the Stack Depot for Focused Fuzzing** Okay, so this looks like a solid target for some fuzzing work. I'm focusing on the stack depot, specifically the new trie-based storage backend. I'm thinking about which functions would be the best to target. Let's see... the `stack_depot_trie_*` functions are all static, so those are out. I need to list the non-static functions. Right, let's see: `stack_depot_save_flags`, `stack_depot_save`, `__stack_depot_get_stack_record`, `stack_depot_fetch`, `stack_depot_fetch_into`, `stack_depot_print`, `stack_depot_snprint`, and `stack_depot_put`. The prompt says to avoid hot paths, so `stack_depot_save` and `stack_depot_save_flags` are probably off the list because of their use with debug tools like `PAGE_OWNER` or `KASAN`. `stack_depot_fetch_into`, `stack_depot_fetch`, `stack_depot_print`, `stack_depot_snprint`, and `__stack_depot_get_stack_record` seem better since they are primarily for reporting or debug use cases. `stack_depot_put` is called during object freeing for `page_owner` or DRM, so it's a maybe. So I'm leaning toward including `stack_depot_fetch_into`, `__stack_depot_get_stack_record`, `stack_depot_snprint`, `stack_depot_print`, and maybe `stack_depot_put`. Let's check `stack_depot_fetch_into` and `__stack_depot_get_stack_record` directly. Now, what about the configuration options? I should enable a few things to make this fuzzing as effective as possible. `PAGE_OWNER`, `DEBUG_KMEMLEAK`, `STACKDEPOT_KUNIT_TEST`, and `KUNIT` seem like a good place to start. Let's make sure I have the dependencies right for `STACKDEPOT_KUNIT_TEST`. It depends on `KUNIT=y && STACKDEPOT` and `STACKDEPOT_MAX_FRAMES >= 3`. That's good, I already have `KUNIT` and `PAGE_OWNER` which selects `STACKDEPOT`. Just to be sure, let me verify that `PAGE_OWNER` and `DEBUG_KMEMLEAK` do select `STACKDEPOT`. Yep, both have the `select STACKDEPOT` in the Kconfig files. So the EnableConfigs are: `PAGE_OWNER`, `DEBUG_KMEMLEAK`, `KUNIT`, and `STACKDEPOT_KUNIT_TEST`. And the FocusSymbols: `stack_depot_fetch_into`, `__stack_depot_get_stack_record`, `stack_depot_snprint`, and `stack_depot_print`. Okay, let me quickly review the entire lib/stackdepot.c to see if there are any other relevant non-static functions that were added or modified. Alright: `stack_depot_early_init`, `stack_depot_init`, `stack_depot_save_flags`, `stack_depot_save`, `__stack_depot_get_stack_record`, `stack_depot_fetch`, `stack_depot_fetch_into`, `stack_depot_put`, `stack_depot_print`, `stack_depot_snprint`. The arch-specific code adds `arch_stack_depot_frame_try_compress` and `arch_stack_depot_frame_decompress`, but those are static inline. So, the focus symbols are good. The reasoning here is that the patch fundamentally changes how stack traces are stored and retrieved, which is a core part of the kernel used by debugging and tracking tools. The changes introduce new APIs, new flags, and a different internal implementation, making it a good target for fuzzing to ensure memory safety and correctness. Now, let's call `set-results`. |
| 14/2 | 2026/08/17 12:58 | tool | set-results |
0mArgs:{
"EnableConfigs": [
"PAGE_OWNER",
"DEBUG_KMEMLEAK",
"KUNIT",
"STACKDEPOT_KUNIT_TEST"
],
"FocusSymbols": [
"stack_depot_fetch_into",
"__stack_depot_get_stack_record",
"stack_depot_snprint",
"stack_depot_print"
],
"Reasoning": "The patch introduces a new trie-based storage backend for the stack depot, a core kernel library used by memory debugging and tracking tools (e.g., KASAN, KMSAN, page_owner, kmemleak). It adds a new API `stack_depot_fetch_into` and modifies existing APIs to support the new backend and a new `STACK_DEPOT_FLAG_COUNTABLE` flag. These changes significantly alter the internal logic of stack trace storage and retrieval, making it worth fuzzing to ensure memory safety and correctness.",
"WorthFuzzing": true
}Results: {
"EnableConfigs": [
"PAGE_OWNER",
"DEBUG_KMEMLEAK",
"KUNIT",
"STACKDEPOT_KUNIT_TEST"
],
"FocusSymbols": [
"stack_depot_fetch_into",
"__stack_depot_get_stack_record",
"stack_depot_snprint",
"stack_depot_print"
],
"Reasoning": "The patch introduces a new trie-based storage backend for the stack depot, a core kernel library used by memory debugging and tracking tools (e.g., KASAN, KMSAN, page_owner, kmemleak). It adds a new API `stack_depot_fetch_into` and modifies existing APIs to support the new backend and a new `STACK_DEPOT_FLAG_COUNTABLE` flag. These changes significantly alter the internal logic of stack trace storage and retrieval, making it worth fuzzing to ensure memory safety and correctness.",
"WorthFuzzing": true
} |
| 15/1 | 2026/08/17 12:58 | action | If |
1mArgs:{
"WorthFuzzing": true
}Results: null |
| 16/2 | 2026/08/17 12:58 | agent | kmsan-evaluator |
1mModel:best-expensive Args: null Results: {
"KMSANReasoning": "The patch series introduces a new trie-based storage backend for `stackdepot` to compress and store stack traces more efficiently. It involves custom memory sub-allocation (`trie_pool_alloc`) from page-sized pools. \n\nWhile the memory returned by `trie_pool_alloc` is reused without zeroing, the code carefully and fully initializes all relevant fields of the allocated structures (`trie_node_init`, `trie_children_init`, `trie_children_insert`) before they are published via RCU. Any padding bytes within the structures (e.g., in `struct stack_depot_trie_node`) are never read or exposed.\n\nThe patch also changes the fetch API to `stack_depot_fetch_into`, requiring callers to provide their own buffers. This function explicitly unpoisons the written memory using `kmsan_unpoison_memory`. This is safe and correct because the trie traversal logic guarantees writing exactly the calculated number of frames, even in the presence of concurrent reparenting or splitting.\n\nThere are no structures copied to user space, no parsing of untrusted input, and no paths where uninitialized memory could affect control flow. Any bugs in the trie logic or memory management would primarily manifest as logic errors (wrong stack traces), out-of-bounds accesses, or use-after-free bugs. These are adequately covered by standard KASAN, LOCKDEP, and the comprehensive KUnit tests included in the patch. Therefore, a dedicated KMSAN fuzzing session is not justified.",
"NeedsKMSAN": false
}Instruction: You are an expert Linux kernel security engineer specializing in kernel memory
error detectors (KASAN and KMSAN). Your job is to review the provided patch series and
determine if the code changes justify spawning a dedicated KMSAN (KernelMemorySanitizer)
fuzzing session in addition to standard KASAN fuzzing.
CRITICAL DISTINCTION BETWEEN KASAN AND KMSAN:
- Standard KASAN kernel builds (upstream-apparmor-kasan.config) already enable
a comprehensive suite of debugging tools and sanitizers, including KASAN
(out-of-bounds accesses, use-after-free, double free, invalid free), LOCKDEP
(locking bugs and deadlocks), UB-sanitizers, and memory corruption checks.
- KMSAN (KernelMemorySanitizer) detects reads of UNINITIALIZED memory (stack, heap,
or page allocations) and kernel-to-user memory info-leaks.
Rule: THERE IS NO SENSE IN RUNNING A KMSAN SESSION IF A BUG CAN BE CAUGHT BY KASAN,
LOCKDEP, OR OTHER STANDARD BUG DETECTORS.
A dedicated KMSAN fuzzing session incurs significant resource costs. You must ONLY
set NeedsKMSAN=true if the code changes introduce or expose UNINITIALIZED MEMORY risks
that are detected ONLY by KMSAN.
Look holistically at the patch series and surrounding code. Even if no direct
uninitialized field accesses or new buffer allocations are added in the diff itself,
a patch may alter control flow, bounds checking, or data length calculations in ways
that change how the rest of the code operates on existing buffers (e.g. allowing
uninitialized stack/heap memory to be read, copied to user space, or used in control
flow). Do not hesitate to use your code access tools to inspect the surrounding code,
called functions, and callers.
Set NeedsKMSAN=true ONLY IF the patch introduces or modifies:
1. Kernel structures sent to user space (via copy_to_user, put_user, netlink skb
attributes, ioctl output arguments, socket options, or BPF buffers) where fields
or structure padding might not be fully initialized/zeroed.
2. Conditional logic or branching that depends on potentially uninitialized variables
or struct fields.
3. Allocation or initialization of complex data structures where uninitialized fields
could be read later in reachable code paths.
4. Bounds checks, lengths, or logic in a way that allows surrounding code to access
uninitialized bytes of existing buffers.
Set NeedsKMSAN=false IF:
- The code changes primarily risk out-of-bounds access, array overflows, NULL pointer
dereferences, locking deadlocks, or use-after-free bugs (these are already caught
by KASAN, LOCKDEP, or standard bug detectors).
- All stack/heap structures touched or introduced by the patch are fully zeroed
or initialized (e.g. using = {0}, memset, kzalloc) before being read or copied.
- The patch does not introduce any risk of uninitialized memory usage or info-leaks.
Use your code access tools to inspect the surrounding code if necessary, then provide
detailed KMSANReasoning contrasting KASAN vs KMSAN applicability for this patch.
Prefer calling several tools at the same time to save round-trips.
Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.
Prompt: For your convenience, here is the diff of the changes:
commit 1d27bb5b64bebf701b469bdea8a91ce99a84492b
Author: syz-cluster <triage@syzkaller.com>
Date: Mon Aug 17 12:57:47 2026 +0000
syz-cluster: applied patch under review
diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 1af62cd16c9de..ebb7b7e1867f6 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -7387,6 +7387,13 @@ Kernel parameters
stack traces. Pools are allocated on-demand up to this
limit. Default value is 8191 pools.
+ stackdepot.trie_enabled= [KNL]
+ Format: <bool>
+ Enable trie storage for persistent, non-refcounted
+ stack depot records at boot. Disabled by default.
+ stack_depot_max_pools must leave unused pool-index
+ values for trie handles.
+
stacktrace [FTRACE]
Enable the stack tracer on boot up.
diff --git a/arch/arm64/include/asm/stackdepot.h b/arch/arm64/include/asm/stackdepot.h
new file mode 100644
index 0000000000000..df8959d593366
--- /dev/null
+++ b/arch/arm64/include/asm/stackdepot.h
@@ -0,0 +1,42 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __ASM_STACKDEPOT_H
+#define __ASM_STACKDEPOT_H
+
+#include <linux/types.h>
+#include <asm/sections.h>
+
+/*
+ * Modules are allocated inside a 2 GB relocation window containing the
+ * kernel image. Store a signed 32-bit offset from _text so compression is
+ * independent of 4 GB high-bit boundaries crossed by that window.
+ */
+static inline unsigned long arch_stack_depot_frame_from_payload(u32 payload)
+{
+ long offset;
+
+ offset = (s32)payload;
+ if (offset < 0)
+ return (unsigned long)_text - (unsigned long)(-offset);
+ return (unsigned long)_text + (unsigned long)offset;
+}
+
+static inline bool
+arch_stack_depot_frame_try_compress(unsigned long frame, u32 *payload)
+{
+ u32 candidate;
+
+ candidate = (u32)(frame - (unsigned long)_text);
+ if (arch_stack_depot_frame_from_payload(candidate) != frame)
+ return false;
+
+ *payload = candidate;
+ return true;
+}
+
+static inline void
+arch_stack_depot_frame_decompress(u32 payload, unsigned long *frame)
+{
+ *frame = arch_stack_depot_frame_from_payload(payload);
+}
+
+#endif /* __ASM_STACKDEPOT_H */
diff --git a/arch/um/include/asm/Kbuild b/arch/um/include/asm/Kbuild
index 8fdc0bd9ab6fb..14778d2457d79 100644
--- a/arch/um/include/asm/Kbuild
+++ b/arch/um/include/asm/Kbuild
@@ -21,6 +21,7 @@ generic-y += preempt.h
generic-y += ring_buffer.h
generic-y += runtime-const.h
generic-y += softirq_stack.h
+generic-y += stackdepot.h
generic-y += switch_to.h
generic-y += topology.h
generic-y += trace_clock.h
diff --git a/arch/x86/include/asm/stackdepot.h b/arch/x86/include/asm/stackdepot.h
new file mode 100644
index 0000000000000..9a8d04fa8c1c8
--- /dev/null
+++ b/arch/x86/include/asm/stackdepot.h
@@ -0,0 +1,37 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_X86_STACKDEPOT_H
+#define _ASM_X86_STACKDEPOT_H
+
+#include <linux/types.h>
+
+#ifdef CONFIG_X86_64
+/*
+ * Compress canonical kernel text/module addresses whose upper 32 bits are all
+ * ones. Other kernel virtual addresses stay raw, so decompression reconstructs
+ * the original frame by restoring this prefix.
+ */
+#define STACK_DEPOT_X86_64_FRAME_PREFIX 0xffffffff00000000UL
+#define STACK_DEPOT_X86_64_FRAME_LOW_MASK 0x00000000ffffffffUL
+
+static inline bool
+arch_stack_depot_frame_try_compress(unsigned long frame, u32 *low)
+{
+ if ((frame & ~STACK_DEPOT_X86_64_FRAME_LOW_MASK) !=
+ STACK_DEPOT_X86_64_FRAME_PREFIX)
+ return false;
+
+ *low = (u32)frame;
+ return true;
+}
+
+static inline void
+arch_stack_depot_frame_decompress(u32 low, unsigned long *frame)
+{
+ *frame = STACK_DEPOT_X86_64_FRAME_PREFIX | low;
+}
+
+#else
+#include <asm-generic/stackdepot.h>
+#endif /* CONFIG_X86_64 */
+
+#endif /* _ASM_X86_STACKDEPOT_H */
diff --git a/drivers/gpu/drm/drm_modeset_lock.c b/drivers/gpu/drm/drm_modeset_lock.c
index 2c806b0146d67..a2ddb02b2aea5 100644
--- a/drivers/gpu/drm/drm_modeset_lock.c
+++ b/drivers/gpu/drm/drm_modeset_lock.c
@@ -94,16 +94,13 @@ static noinline depot_stack_handle_t __drm_stack_depot_save(void)
static void __drm_stack_depot_print(depot_stack_handle_t stack_depot)
{
struct drm_printer p = drm_dbg_printer(NULL, DRM_UT_KMS, "drm_modeset_lock");
- unsigned long *entries;
- unsigned int nr_entries;
char *buf;
buf = kmalloc(PAGE_SIZE, GFP_NOWAIT | __GFP_NOWARN);
if (!buf)
return;
- nr_entries = stack_depot_fetch(stack_depot, &entries);
- stack_trace_snprint(buf, PAGE_SIZE, entries, nr_entries, 2);
+ stack_depot_snprint(stack_depot, buf, PAGE_SIZE, 2);
drm_printf(&p, "attempting to lock a contended lock without backoff:\n%s", buf);
diff --git a/include/asm-generic/Kbuild b/include/asm-generic/Kbuild
index 15df9dcb42a5b..ac178162fa114 100644
--- a/include/asm-generic/Kbuild
+++ b/include/asm-generic/Kbuild
@@ -55,6 +55,7 @@ mandatory-y += serial.h
mandatory-y += shmparam.h
mandatory-y += simd.h
mandatory-y += softirq_stack.h
+mandatory-y += stackdepot.h
mandatory-y += switch_to.h
mandatory-y += timex.h
mandatory-y += tlbflush.h
diff --git a/include/asm-generic/stackdepot.h b/include/asm-generic/stackdepot.h
new file mode 100644
index 0000000000000..846975767bdd4
--- /dev/null
+++ b/include/asm-generic/stackdepot.h
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __ASM_GENERIC_STACKDEPOT_H
+#define __ASM_GENERIC_STACKDEPOT_H
+
+#include <linux/types.h>
+
+static inline bool
+arch_stack_depot_frame_try_compress(unsigned long frame, u32 *low)
+{
+ return false;
+}
+
+static inline void
+arch_stack_depot_frame_decompress(u32 low, unsigned long *frame)
+{
+ /* Generic code never compresses frames, so this hook is unreachable. */
+}
+
+#endif /* __ASM_GENERIC_STACKDEPOT_H */
diff --git a/include/linux/stackdepot.h b/include/linux/stackdepot.h
index 2cc21ffcdaf9e..788737eb0c4a2 100644
--- a/include/linux/stackdepot.h
+++ b/include/linux/stackdepot.h
@@ -53,7 +53,8 @@ union handle_parts {
struct stack_record {
struct list_head hash_list; /* Links in the hash table */
u32 hash; /* Hash in hash table */
- u32 size; /* Number of stored frames */
+ u16 size; /* Number of stored frames */
+ u16 flags;
union handle_parts handle; /* Constant after initialization */
refcount_t count;
union {
@@ -84,8 +85,9 @@ typedef u32 depot_flags_t;
*/
#define STACK_DEPOT_FLAG_CAN_ALLOC ((depot_flags_t)0x0001)
#define STACK_DEPOT_FLAG_GET ((depot_flags_t)0x0002)
+#define STACK_DEPOT_FLAG_COUNTABLE ((depot_flags_t)0x0004)
-#define STACK_DEPOT_FLAGS_NUM 2
+#define STACK_DEPOT_FLAGS_NUM 3
#define STACK_DEPOT_FLAGS_MASK ((depot_flags_t)((1 << STACK_DEPOT_FLAGS_NUM) - 1))
/*
@@ -144,6 +146,15 @@ static inline int stack_depot_early_init(void) { return 0; }
* Users of this flag must also call stack_depot_put() when keeping the stack
* trace is no longer required to avoid overflowing the refcount.
*
+ * If STACK_DEPOT_FLAG_COUNTABLE is set in @depot_flags, stack depot stores the
+ * stack in hash-backed storage for callers that need direct stack_record count
+ * access. This flag does not imply %STACK_DEPOT_FLAG_CAN_ALLOC and is mutually
+ * exclusive with %STACK_DEPOT_FLAG_GET.
+ *
+ * When trie storage is enabled, persistent non-refcounted saves use trie
+ * storage. Constrained callers only look up existing stacks; they do not insert
+ * a missing stack. Trie failures do not fall back to hash storage.
+ *
* If the provided stack trace comes from the interrupt context, only the part
* up to the interrupt entry is saved.
*
@@ -152,7 +163,7 @@ static inline int stack_depot_early_init(void) { return 0; }
* this is the case for contexts where neither %GFP_ATOMIC nor
* %GFP_NOWAIT can be used (NMI, raw_spin_lock).
*
- * Return: Handle of the stack struct stored in depot, 0 on failure
+ * Return: Handle of the stack trace stored in depot, 0 on failure
*/
depot_stack_handle_t stack_depot_save_flags(unsigned long *entries,
unsigned int nr_entries,
@@ -169,6 +180,10 @@ depot_stack_handle_t stack_depot_save_flags(unsigned long *entries,
* Does not increment the refcount on the saved stack trace; see
* stack_depot_save_flags() for more details.
*
+ * When trie storage is enabled, this can return trie-backed handles. Use
+ * stack_depot_fetch_into(), stack_depot_print(), or stack_depot_snprint() for
+ * backend-independent access to the stack contents.
+ *
* Context: Contexts where allocations via alloc_pages() are allowed;
* see stack_depot_save_flags() for more details.
*
@@ -178,11 +193,12 @@ depot_stack_handle_t stack_depot_save(unsigned long *entries,
unsigned int nr_entries, gfp_t alloc_flags);
/**
- * __stack_depot_get_stack_record - Get a pointer to a stack_record struct
+ * __stack_depot_get_stack_record - Get a hash-backed stack record
*
* @handle: Stack depot handle
*
- * This function is only for internal purposes.
+ * This function is only for internal purposes. @handle must have been saved
+ * with %STACK_DEPOT_FLAG_COUNTABLE.
*
* Return: Returns a pointer to a stack_record struct
*/
@@ -191,14 +207,55 @@ struct stack_record *__stack_depot_get_stack_record(depot_stack_handle_t handle)
/**
* stack_depot_fetch - Fetch a stack trace from stack depot
*
- * @handle: Stack depot handle returned from stack_depot_save()
+ * @handle: Hash-backed stack depot handle
* @entries: Pointer to store the address of the stack trace
*
+ * This helper returns a pointer to stackdepot-owned contiguous storage for
+ * legacy hash-backed handles. Callers that need backend-independent access to
+ * stack contents should use stack_depot_fetch_into(), stack_depot_print(), or
+ * stack_depot_snprint(). Passing a trie-backed handle is invalid and may WARN.
+ *
* Return: Number of frames for the fetched stack
*/
unsigned int stack_depot_fetch(depot_stack_handle_t handle,
unsigned long **entries);
+/**
+ * stack_depot_fetch_into - Fetch a stack trace into caller-owned storage
+ *
+ * @handle: Stack depot handle
+ * @entries: Caller-owned buffer to copy the stack trace into
+ * @max_entries: Number of frames that fit in @entries
+ *
+ * Copies the stored frames into caller-owned @entries. If fewer frames are
+ * stored than @max_entries, only the stored frames are written and their count
+ * is returned. If more frames are stored than @max_entries, the copy is skipped
+ * entirely and 0 is returned.
+ *
+ * Passing a NULL @entries buffer or zero @max_entries for a valid @handle is
+ * invalid. Callers must provide storage for @max_entries frames.
+ *
+ * Callers should size @entries to match the save-side stack depth cap (for
+ * example, %CONFIG_STACKDEPOT_MAX_FRAMES or the local stack_trace_save() limit)
+ * when losing diagnostics on an undersized buffer would be surprising.
+ *
+ * A non-zero invalid @handle, including a post-put handle, may WARN. Its return
+ * value and copied contents are undefined because the record may have been
+ * reused for another stack.
+ *
+ * Callers must ensure @handle remains valid for the duration of this call.
+ * Persistent handles saved without %STACK_DEPOT_FLAG_GET require no extra
+ * reference; handles saved with %STACK_DEPOT_FLAG_GET require a held reference.
+ * Callers must not call stack_depot_put() on persistent handles.
+ * Racing this helper with stack_depot_put() on the same handle is invalid.
+ *
+ * Return: Number of frames copied, 0 if @handle is 0, stack depot is disabled,
+ * or @max_entries is less than the number of stored frames.
+ */
+unsigned int stack_depot_fetch_into(depot_stack_handle_t handle,
+ unsigned long *entries,
+ unsigned int max_entries);
+
/**
* stack_depot_print - Print a stack trace from stack depot
*
@@ -224,10 +281,14 @@ int stack_depot_snprint(depot_stack_handle_t handle, char *buf, size_t size,
*
* @handle: Stack depot handle returned from stack_depot_save()
*
- * The stack trace is evicted from stack depot once all references to it have
- * been dropped (once the number of stack_depot_evict() calls matches the
- * number of stack_depot_save_flags() calls with STACK_DEPOT_FLAG_GET set for
- * this stack trace).
+ * Drop a reference acquired by stack_depot_save_flags() with
+ * %STACK_DEPOT_FLAG_GET. Calling this for a handle saved without
+ * %STACK_DEPOT_FLAG_GET is invalid; persistent handles, including trie-backed
+ * handles, are owned by stack depot for the lifetime of the system.
+ *
+ * The stack trace is evicted once the number of stack_depot_put() calls matches
+ * the number of successful stack_depot_save_flags() calls with
+ * %STACK_DEPOT_FLAG_GET for this stack trace.
*/
void stack_depot_put(depot_stack_handle_t handle);
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index 00921b1676e87..af238949fb7a0 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -2771,6 +2771,23 @@ config RESOURCE_KUNIT_TEST
If unsure, say N.
+config STACKDEPOT_KUNIT_TEST
+ bool "KUnit test for stack depot" if !KUNIT_ALL_TESTS
+ depends on KUNIT=y && STACKDEPOT
+ depends on STACKDEPOT_MAX_FRAMES >= 3
+ default KUNIT_ALL_TESTS
+ help
+ Enable this option to test stack depot API behavior at boot.
+ This test is built in because it exercises internal, non-exported
+ stack depot helpers, so KUNIT must also be built in.
+
+ KUnit tests run during boot and output the results to the debug log
+ in TAP format (https://testanything.org/). Only useful for kernel
+ developers running the KUnit test harness, and not intended for
+ inclusion into a production build.
+
+ If unsure, say N.
+
config SYSCTL_KUNIT_TEST
tristate "KUnit test for sysctl" if !KUNIT_ALL_TESTS
depends on KUNIT
diff --git a/lib/stackdepot.c b/lib/stackdepot.c
index dd2717ff94bff..1a002063a9487 100644
--- a/lib/stackdepot.c
+++ b/lib/stackdepot.c
@@ -2,9 +2,11 @@
/*
* Stack depot - a stack trace storage that avoids duplication.
*
- * Internally, stack depot maintains a hash table of unique stacktraces. The
- * stack traces themselves are stored contiguously one after another in a set
- * of separate page allocations.
+ * Internally, stack depot has two storage backends. Refcounted entries and
+ * callers that request STACK_DEPOT_FLAG_COUNTABLE use the legacy hash table with
+ * contiguous stack records in stack pools. Persistent non-refcounted entries
+ * can use trie storage when enabled; trie nodes share common frame prefixes and
+ * are published through RCU children containers.
*
* Author: Alexander Potapenko <glider@google.com>
* Copyright (C) 2016 Google, Inc.
@@ -14,13 +16,19 @@
#define pr_fmt(fmt) "stackdepot: " fmt
+#include <linux/bitmap.h>
+#include <linux/build_bug.h>
#include <linux/debugfs.h>
+#include <linux/errno.h>
#include <linux/gfp.h>
#include <linux/jhash.h>
+#include <linux/jump_label.h>
#include <linux/kernel.h>
+#include <linux/log2.h>
#include <linux/kmsan.h>
#include <linux/list.h>
#include <linux/mm.h>
+#include <linux/moduleparam.h>
#include <linux/mutex.h>
#include <linux/poison.h>
#include <linux/printk.h>
@@ -36,9 +44,12 @@
#include <linux/memblock.h>
#include <linux/kasan-enabled.h>
+#include <asm/stackdepot.h>
+
/*
* The pool_index is offset by 1 so the first record does not have a 0 handle.
*/
+/* Parsed before mm_core_init(); trie handle decoding assumes this is then fixed. */
static unsigned int stack_max_pools __read_mostly =
MIN((1LL << DEPOT_POOL_INDEX_BITS) - 1, 8192);
@@ -63,18 +74,18 @@ static unsigned int stack_hash_mask;
/* The lock must be held when performing pool or freelist modifications. */
static DEFINE_RAW_SPINLOCK(pool_lock);
-/* Array of memory regions that store stack records. */
+/* Array of memory regions used by both stack depot backends. */
static void **stack_pools __pt_guarded_by(&pool_lock);
/* Newly allocated pool that is not yet added to stack_pools. */
static void *new_pool;
/* Number of pools in stack_pools. */
static int pools_num;
-/* Offset to the unused space in the currently used pool. */
+/* Offset to unused hash storage in the current pool. */
static size_t pool_offset __guarded_by(&pool_lock) = DEPOT_POOL_SIZE;
/* Freelist of stack records within stack_pools. */
static __guarded_by(&pool_lock) LIST_HEAD(free_stacks);
-/* Statistics counters for debugfs. */
+/* Hash-backend statistics counters for debugfs. */
enum depot_counter_id {
DEPOT_COUNTER_REFD_ALLOCS,
DEPOT_COUNTER_REFD_FREES,
@@ -90,11 +101,631 @@ static const char *const counter_names[] = {
[DEPOT_COUNTER_REFD_FREES] = "refcounted_frees",
[DEPOT_COUNTER_REFD_INUSE] = "refcounted_in_use",
[DEPOT_COUNTER_FREELIST_SIZE] = "freelist_size",
- [DEPOT_COUNTER_PERSIST_COUNT] = "persistent_count",
- [DEPOT_COUNTER_PERSIST_BYTES] = "persistent_bytes",
+ [DEPOT_COUNTER_PERSIST_COUNT] = "hash_persistent_count",
+ [DEPOT_COUNTER_PERSIST_BYTES] = "hash_persistent_bytes",
};
static_assert(ARRAY_SIZE(counter_names) == DEPOT_COUNTER_COUNT);
+enum stack_depot_frame_mode {
+ STACK_DEPOT_FRAME_RAW,
+ STACK_DEPOT_FRAME_COMPRESSED,
+};
+
+/*
+ * A trie node stores one run of frames that all use the same payload format.
+ * Architectures may compress some frames to 32-bit payloads; mixed raw and
+ * compressed input is split across multiple trie nodes so each node has one
+ * decoding mode.
+ */
+struct stack_depot_frame_run {
+ u16 nr_entries;
+ u8 mode;
+};
+
+static_assert(CONFIG_STACKDEPOT_MAX_FRAMES <= U16_MAX);
+
+struct stack_depot_trie_children;
+
+struct stack_depot_trie_node {
+ /* Parent links let fetch rebuild a full stack from a node to the root. */
+ const struct stack_depot_trie_node __rcu *parent;
+ /* Children are RCU-published containers. */
+ const struct stack_depot_trie_children __rcu *children;
+ /* Non-zero when a stored stack ends at this node. */
+ u32 stack_id;
+ struct stack_depot_frame_run run;
+ unsigned char data[];
+};
+
+/*
+ * Child nodes are sorted by first frame and searched by insertion position.
+ * Existing child pointers are immutable. Writers may publish into unused tail
+ * capacity; other updates publish a replacement container.
+ */
+struct stack_depot_trie_children {
+ unsigned int nr_children;
+ unsigned int capacity;
+ const struct stack_depot_trie_node __rcu *nodes[];
+};
+
+/* Retired children carry an optional node through their RCU grace period. */
+struct stack_depot_trie_retired_children {
+ struct list_head list;
+ unsigned long rcu_state;
+ const struct stack_depot_trie_node *pending_node;
+ unsigned char data[];
+};
+
+static_assert(IS_ALIGNED(offsetof(struct stack_depot_trie_retired_children, data),
+ 1UL << DEPOT_STACK_ALIGN));
+
+#define STACK_DEPOT_TRIE_SLOT_SIZE BIT(DEPOT_STACK_ALIGN)
+#define STACK_DEPOT_TRIE_POOL_SLOTS \
+ (DEPOT_POOL_SIZE / STACK_DEPOT_TRIE_SLOT_SIZE)
+
+struct stack_depot_trie_pool {
+ struct list_head list;
+ unsigned int free_slots;
+ DECLARE_BITMAP(used, STACK_DEPOT_TRIE_POOL_SLOTS);
+};
+
+#define STACK_DEPOT_TRIE_POOL_FIRST_SLOT \
+ DIV_ROUND_UP(sizeof(struct stack_depot_trie_pool), \
+ STACK_DEPOT_TRIE_SLOT_SIZE)
+#define STACK_DEPOT_TRIE_POOL_USABLE_SIZE \
+ ((STACK_DEPOT_TRIE_POOL_SLOTS - STACK_DEPOT_TRIE_POOL_FIRST_SLOT) * \
+ STACK_DEPOT_TRIE_SLOT_SIZE)
+
+static_assert(STACK_DEPOT_TRIE_POOL_FIRST_SLOT < STACK_DEPOT_TRIE_POOL_SLOTS);
+
+static DEFINE_STATIC_KEY_FALSE(stack_depot_trie_enabled);
+static const struct stack_depot_trie_children __rcu *stack_depot_trie_root;
+static DEFINE_RAW_SPINLOCK(stack_depot_trie_writer_lock);
+static bool stack_depot_trie_requested;
+
+module_param_named(trie_enabled, stack_depot_trie_requested, bool, 0);
+MODULE_PARM_DESC(trie_enabled, "Enable stack depot trie storage at boot");
+
+#define DEPOT_POOL_INDEX_MASK ((1U << DEPOT_POOL_INDEX_BITS) - 1)
+#define DEPOT_OFFSET_MASK ((1U << DEPOT_OFFSET_BITS) - 1)
+
+/* Retired fixed-size slots remain reserved until their RCU grace period ends. */
+static LIST_HEAD(stack_depot_trie_pools);
+static LIST_HEAD(pending_trie_children);
+
+/*
+ * stack_max_pools is the split point between hash and trie handle encodings.
+ * A handle with pool_index_plus_1 in 1..stack_max_pools names a hash-backed
+ * stack pool. Larger pool-index values cannot refer to hash pools, so trie
+ * storage uses that handle space to encode a dense stack ID. The side table
+ * maps each stack ID to its trie node.
+ */
+static inline u32 trie_max_stack_id(void)
+{
+ return (DEPOT_POOL_INDEX_MASK - stack_max_pools) <<
+ DEPOT_OFFSET_BITS;
+}
+
+static depot_stack_handle_t trie_handle(u32 stack_id)
+{
+ union handle_parts parts = {};
+ u64 pool_index_plus_1;
+ u32 pool_delta;
+ u32 index;
+
+ index = stack_id - 1;
+ pool_delta = index >> DEPOT_OFFSET_BITS;
+ pool_index_plus_1 = (u64)stack_max_pools + 1 + pool_delta;
+
+ parts.pool_index_plus_1 = pool_index_plus_1;
+ parts.offset = index & DEPOT_OFFSET_MASK;
+ return parts.handle;
+}
+
+static inline bool stack_depot_handle_is_trie(depot_stack_handle_t handle)
+{
+ union handle_parts parts = { .handle = handle };
+
+ return parts.pool_index_plus_1 > stack_max_pools;
+}
+
+static u32 trie_stack_id(depot_stack_handle_t handle)
+{
+ union handle_parts parts = { .handle = handle };
+ u32 pool_delta;
+
+ pool_delta = parts.pool_index_plus_1 - stack_max_pools - 1;
+ return (pool_delta << DEPOT_OFFSET_BITS) + parts.offset + 1;
+}
+
+/*
+ * Trie handles encode a dense stack ID. The side table maps that ID to a node
+ * pointer for lockless fetch and print paths, which can run from diagnostic
+ * contexts where taking a lock would be unsafe. Initialization installs the
+ * root; early initialization also installs the first directory and chunk.
+ * Additional directories and chunks are published lazily as stack IDs grow.
+ */
+#define STACK_DEPOT_TRIE_SIDE_TABLE_CHUNK_SIZE \
+ (PAGE_SIZE / sizeof(struct stack_depot_trie_node *))
+#define STACK_DEPOT_TRIE_SIDE_TABLE_DIR_SIZE \
+ (PAGE_SIZE / sizeof(struct stack_depot_trie_node **))
+
+struct stack_depot_trie_side_dir {
+ /* Both the chunk pointer and each node pointer in it are RCU-published. */
+ const struct stack_depot_trie_node __rcu * __rcu *
+ chunks[STACK_DEPOT_TRIE_SIDE_TABLE_DIR_SIZE];
+};
+
+struct stack_depot_trie_side_root {
+ unsigned int dir_capacity;
+ struct stack_depot_trie_side_dir __rcu *dirs[];
+};
+
+struct stack_depot_trie_side_prealloc {
+ /* Preallocated side-table directory page for sparse growth. */
+ struct stack_depot_trie_side_dir *dir;
+ /* Preallocated side-table pointer chunk for sparse growth. */
+ const struct stack_depot_trie_node __rcu **chunk;
+};
+
+static struct stack_depot_trie_side_root *trie_side_table_root;
+static DEFINE_RAW_SPINLOCK(trie_side_table_cache_lock);
+/* Zeroed unpublished pages; get/put transfer ownership under the cache lock. */
+static struct stack_depot_trie_side_prealloc trie_side_table_cache;
+static u32 trie_side_table_last_stack_id;
+
+/* Lock order: writer_lock -> pool_lock. The cache lock is never nested. */
+
+static inline size_t stack_depot_frame_run_entry_bytes(enum stack_depot_frame_mode mode)
+{
+ if (mode == STACK_DEPOT_FRAME_COMPRESSED)
+ return sizeof(u32);
+ return sizeof(unsigned long);
+}
+
+static inline size_t stack_depot_frame_run_bytes(const struct stack_depot_frame_run *run)
+{
+ return run->nr_entries * stack_depot_frame_run_entry_bytes(run->mode);
+}
+
+static inline size_t trie_node_bytes(const struct stack_depot_frame_run *run)
+{
+ return ALIGN(offsetof(struct stack_depot_trie_node, data) +
+ stack_depot_frame_run_bytes(run), sizeof(unsigned long));
+}
+
+static size_t trie_children_alloc_size(unsigned int capacity)
+{
+ size_t size;
+
+ size = struct_size_t(struct stack_depot_trie_children, nodes,
+ capacity);
+ return offsetof(struct stack_depot_trie_retired_children, data) +
+ ALIGN(size, sizeof(unsigned long));
+}
+
+static inline unsigned int trie_side_table_root_index(u32 id)
+{
+ return ((id - 1) / STACK_DEPOT_TRIE_SIDE_TABLE_CHUNK_SIZE) /
+ STACK_DEPOT_TRIE_SIDE_TABLE_DIR_SIZE;
+}
+
+static inline unsigned int trie_side_table_dir_index(u32 id)
+{
+ return ((id - 1) / STACK_DEPOT_TRIE_SIDE_TABLE_CHUNK_SIZE) %
+ STACK_DEPOT_TRIE_SIDE_TABLE_DIR_SIZE;
+}
+
+static inline unsigned int trie_side_table_slot_index(u32 id)
+{
+ return (id - 1) % STACK_DEPOT_TRIE_SIDE_TABLE_CHUNK_SIZE;
+}
+
+static struct stack_depot_trie_side_dir *trie_side_table_load_dir(unsigned int root)
+{
+ struct stack_depot_trie_side_root *root_vec;
+
+ root_vec = trie_side_table_root;
+ if (!root_vec || root >= root_vec->dir_capacity)
+ return NULL;
+ /* Pairs with side-table directory rcu_assign_pointer(). */
+ return rcu_dereference_check(root_vec->dirs[root],
+ lockdep_is_held(&stack_depot_trie_writer_lock) ||
+ rcu_read_lock_sched_held());
+}
+
+static inline const struct stack_depot_trie_node __rcu **
+trie_side_table_dir_load_chunk(struct stack_depot_trie_side_dir *dir,
+ unsigned int idx)
+{
+ /* Pairs with the chunk rcu_assign_pointer() in stack ID preparation. */
+ return rcu_dereference_check(dir->chunks[idx],
+ lockdep_is_held(&stack_depot_trie_writer_lock) ||
+ rcu_read_lock_sched_held());
+}
+
+static u32
+trie_side_table_prepare_stack_slot(struct stack_depot_trie_side_prealloc *prealloc)
+{
+ const struct stack_depot_trie_node __rcu **chunk;
+ struct stack_depot_trie_side_dir *dir;
+ struct stack_depot_trie_side_root *root_vec;
+ unsigned int root;
+ unsigned int idx;
+ u32 id;
+
+ lockdep_assert_held(&stack_depot_trie_writer_lock);
+
+ id = trie_side_table_last_stack_id + 1;
+ if (id > trie_max_stack_id())
+ return 0;
+
+ root_vec = trie_side_table_root;
+ root = trie_side_table_root_index(id);
+ dir = trie_side_table_load_dir(root);
+ if (!dir) {
+ dir = prealloc->dir;
+ prealloc->dir = NULL;
+ /* Publish the zeroed directory before readers can load it locklessly. */
+ rcu_assign_pointer(root_vec->dirs[root], dir);
+ }
+
+ idx = trie_side_table_dir_index(id);
+ chunk = trie_side_table_dir_load_chunk(dir, idx);
+ if (!chunk) {
+ chunk = prealloc->chunk;
+ prealloc->chunk = NULL;
+ rcu_assign_pointer(dir->chunks[idx], chunk);
+ }
+
+ return id;
+}
+
+static inline unsigned int trie_side_table_root_size_for_max_id(u32 max_stack_id)
+{
+ unsigned int top_size;
+
+ top_size = DIV_ROUND_UP(max_stack_id,
+ STACK_DEPOT_TRIE_SIDE_TABLE_CHUNK_SIZE);
+ return DIV_ROUND_UP(top_size, STACK_DEPOT_TRIE_SIDE_TABLE_DIR_SIZE);
+}
+
+static int __init stack_depot_trie_init_memblock(void)
+{
+ struct stack_depot_trie_side_root *root_vec;
+ struct stack_depot_trie_side_dir *first_dir;
+ const struct stack_depot_trie_node __rcu **first_chunk;
+ size_t root_bytes;
+ u32 max_stack_id;
+ unsigned int root_size;
+
+ max_stack_id = trie_max_stack_id();
+ if (!max_stack_id)
+ return -EINVAL;
+ root_size = trie_side_table_root_size_for_max_id(max_stack_id);
+ root_bytes = struct_size_t(struct stack_depot_trie_side_root, dirs,
+ root_size);
+
+ root_vec = memblock_alloc(root_bytes, __alignof__(*root_vec));
+ if (!root_vec)
+ return -ENOMEM;
+ first_dir = memblock_alloc(PAGE_SIZE, PAGE_SIZE);
+ if (!first_dir) {
+ memblock_free(root_vec, root_bytes);
+ return -ENOMEM;
+ }
+ first_chunk = memblock_alloc(PAGE_SIZE, PAGE_SIZE);
+ if (!first_chunk) {
+ memblock_free(first_dir, PAGE_SIZE);
+ memblock_free(root_vec, root_bytes);
+ return -ENOMEM;
+ }
+
+ root_vec->dir_capacity = root_size;
+ RCU_INIT_POINTER(root_vec->dirs[0], first_dir);
+ RCU_INIT_POINTER(first_dir->chunks[0], first_chunk);
+ trie_side_table_root = root_vec;
+ static_branch_enable(&stack_depot_trie_enabled);
+ return 0;
+}
+
+static int stack_depot_trie_init(void)
+{
+ struct stack_depot_trie_side_root *root_vec;
+ unsigned int root_size;
+ u32 max_stack_id;
+
+ max_stack_id = trie_max_stack_id();
+ if (!max_stack_id)
+ return -EINVAL;
+
+ root_size = trie_side_table_root_size_for_max_id(max_stack_id);
+ root_vec = kvzalloc_flex(*root_vec, dirs, root_size);
+ if (!root_vec)
+ return -ENOMEM;
+
+ root_vec->dir_capacity = root_size;
+ trie_side_table_root = root_vec;
+ static_branch_enable(&stack_depot_trie_enabled);
+ return 0;
+}
+
+static int trie_side_table_get_prealloc(gfp_t gfp_flags,
+ struct stack_depot_trie_side_prealloc *prealloc)
+{
+ unsigned long flags;
+
+ gfp_flags = gfp_nested_mask(gfp_flags);
+ raw_spin_lock_irqsave(&trie_side_table_cache_lock, flags);
+ prealloc->dir = trie_side_table_cache.dir;
+ prealloc->chunk = trie_side_table_cache.chunk;
+ trie_side_table_cache.dir = NULL;
+ trie_side_table_cache.chunk = NULL;
+ raw_spin_unlock_irqrestore(&trie_side_table_cache_lock, flags);
+
+ if (!prealloc->dir) {
+ prealloc->dir = (void *)get_zeroed_page(gfp_flags);
+ if (!prealloc->dir)
+ return -ENOMEM;
+ }
+ if (!prealloc->chunk) {
+ prealloc->chunk = (void *)get_zeroed_page(gfp_flags);
+ if (!prealloc->chunk)
+ return -ENOMEM;
+ }
+
+ return 0;
+}
+
+static void trie_side_table_put_prealloc(struct stack_depot_trie_side_prealloc *prealloc)
+{
+ unsigned long flags;
+
+ raw_spin_lock_irqsave(&trie_side_table_cache_lock, flags);
+ if (!trie_side_table_cache.dir) {
+ trie_side_table_cache.dir = prealloc->dir;
+ prealloc->dir = NULL;
+ }
+ if (!trie_side_table_cache.chunk) {
+ trie_side_table_cache.chunk = prealloc->chunk;
+ prealloc->chunk = NULL;
+ }
+ raw_spin_unlock_irqrestore(&trie_side_table_cache_lock, flags);
+
+ if (prealloc->dir)
+ free_page((unsigned long)prealloc->dir);
+ if (prealloc->chunk)
+ free_page((unsigned long)prealloc->chunk);
+}
+
+static const struct stack_depot_trie_node *trie_side_table_lookup(u32 id)
+{
+ const struct stack_depot_trie_node __rcu **chunk;
+ struct stack_depot_trie_side_dir *dir;
+ unsigned int root;
+
+ root = trie_side_table_root_index(id);
+ dir = trie_side_table_load_dir(root);
+ if (!dir)
+ return NULL;
+ chunk = trie_side_table_dir_load_chunk(dir, trie_side_table_dir_index(id));
+ if (!chunk)
+ return NULL;
+
+ /* Pairs with side-table node publication. */
+ return rcu_dereference_check(chunk[trie_side_table_slot_index(id)],
+ lockdep_is_held(&stack_depot_trie_writer_lock) ||
+ rcu_read_lock_sched_held());
+}
+
+static inline struct stack_depot_trie_retired_children *
+trie_retired_children(const void *ptr)
+{
+ return container_of(ptr, struct stack_depot_trie_retired_children, data);
+}
+
+static bool depot_init_pool(void **prealloc);
+
+static unsigned int trie_pool_reserve_slots(struct stack_depot_trie_pool *pool,
+ unsigned int nr_slots)
+{
+ unsigned int run = 0;
+ unsigned int i;
+ unsigned int slot;
+
+ if (pool->free_slots < nr_slots)
+ return STACK_DEPOT_TRIE_POOL_SLOTS;
+
+ /* A free run can cross any previous allocation position. */
+ for (slot = STACK_DEPOT_TRIE_POOL_FIRST_SLOT;
+ slot < STACK_DEPOT_TRIE_POOL_SLOTS; slot++) {
+ if (pool->used[slot / BITS_PER_LONG] &
+ BIT(slot % BITS_PER_LONG)) {
+ run = 0;
+ continue;
+ }
+ if (++run != nr_slots)
+ continue;
+
+ for (i = slot + 1 - nr_slots; i <= slot; i++)
+ pool->used[i / BITS_PER_LONG] |= BIT(i % BITS_PER_LONG);
+ pool->free_slots -= nr_slots;
+ return slot + 1 - nr_slots;
+ }
+
+ return STACK_DEPOT_TRIE_POOL_SLOTS;
+}
+
+/* Allocate at least @size bytes from one contiguous trie-pool slot run. */
+static void *trie_pool_alloc(size_t size, void **prealloc)
+{
+ struct stack_depot_trie_pool *pool;
+ unsigned int nr_slots;
+ unsigned int slot;
+
+ lockdep_assert_held(&pool_lock);
+
+ if (size > STACK_DEPOT_TRIE_POOL_USABLE_SIZE)
+ return NULL;
+ nr_slots = DIV_ROUND_UP(size, STACK_DEPOT_TRIE_SLOT_SIZE);
+ list_for_each_entry_reverse(pool, &stack_depot_trie_pools, list) {
+ slot = trie_pool_reserve_slots(pool, nr_slots);
+ if (slot != STACK_DEPOT_TRIE_POOL_SLOTS)
+ return (char *)pool + slot * STACK_DEPOT_TRIE_SLOT_SIZE;
+ }
+
+ if (!depot_init_pool(prealloc))
+ return NULL;
+ pool = stack_pools[pools_num - 1];
+ /* Keep hash records out of this bitmap-owned pool. */
+ pool_offset = DEPOT_POOL_SIZE;
+ memset(pool, 0, sizeof(*pool));
+ pool->free_slots = STACK_DEPOT_TRIE_POOL_SLOTS -
+ STACK_DEPOT_TRIE_POOL_FIRST_SLOT;
+ list_add_tail(&pool->list, &stack_depot_trie_pools);
+
+ slot = trie_pool_reserve_slots(pool, nr_slots);
+ return (char *)pool + slot * STACK_DEPOT_TRIE_SLOT_SIZE;
+}
+
+/* Release the slots for the byte count originally passed to allocation. */
+static void trie_pool_release(const void *ptr, size_t size)
+{
+ struct stack_depot_trie_pool *pool;
+ unsigned long pfn;
+ unsigned int nr_slots;
+ unsigned int slot;
+ unsigned int i;
+
+ lockdep_assert_held(&pool_lock);
+
+ pfn = page_to_pfn(virt_to_page(ptr));
+ pfn &= ~(BIT(DEPOT_POOL_ORDER) - 1);
+ pool = page_address(pfn_to_page(pfn));
+ slot = ((unsigned long)ptr - (unsigned long)pool) >> DEPOT_STACK_ALIGN;
+ nr_slots = DIV_ROUND_UP(size, STACK_DEPOT_TRIE_SLOT_SIZE);
+ for (i = slot; i < slot + nr_slots; i++)
+ pool->used[i / BITS_PER_LONG] &= ~BIT(i % BITS_PER_LONG);
+ pool->free_slots += nr_slots;
+}
+
+static struct stack_depot_trie_children *
+trie_pool_alloc_children(unsigned int capacity, void **prealloc)
+{
+ struct stack_depot_trie_retired_children *retired;
+ struct stack_depot_trie_children *children;
+
+ /* Capacity counts child-pointer entries; allocation includes RCU metadata. */
+ retired = trie_pool_alloc(trie_children_alloc_size(capacity), prealloc);
+ if (!retired)
+ return NULL;
+
+ children = (void *)retired->data;
+ children->nr_children = 0;
+ children->capacity = capacity;
+ return children;
+}
+
+static void
+trie_pool_release_children(const struct stack_depot_trie_children *children)
+{
+ /* Capacity is immutable and therefore recovers the allocation byte size. */
+ trie_pool_release(trie_retired_children(children),
+ trie_children_alloc_size(children->capacity));
+}
+
+/*
+ * Return RCU-ready objects before allocating. Pending children are FIFO, so
+ * stop at the first incomplete grace period. A replaced node shares the same
+ * retirement cookie and is released with its former children container.
+ */
+static void trie_drain_pending_children(void)
+{
+ struct stack_depot_trie_retired_children *retired;
+ struct stack_depot_trie_retired_children *tmp;
+ struct stack_depot_trie_children *children;
+
+ lockdep_assert_held(&pool_lock);
+
+ list_for_each_entry_safe(retired, tmp, &pending_trie_children, list) {
+ if (!poll_state_synchronize_rcu(retired->rcu_state))
+ break;
+ children = (void *)retired->data;
+ list_del(&retired->list);
+ if (retired->pending_node)
+ trie_pool_release(retired->pending_node,
+ trie_node_bytes(&retired->pending_node->run));
+ trie_pool_release_children(children);
+ }
+}
+
+static void trie_retire_children(const struct stack_depot_trie_children *children)
+{
+ struct stack_depot_trie_retired_children *retired;
+
+ lockdep_assert_held(&pool_lock);
+
+ retired = trie_retired_children(children);
+ retired->pending_node = NULL;
+ retired->rcu_state = get_state_synchronize_rcu();
+ list_add_tail(&retired->list, &pending_trie_children);
+}
+
+static void
+trie_retire_children_with_node(const struct stack_depot_trie_children *children,
+ const struct stack_depot_trie_node *node)
+{
+ struct stack_depot_trie_retired_children *retired;
+
+ lockdep_assert_held(&stack_depot_trie_writer_lock);
+ raw_spin_lock(&pool_lock);
+ trie_retire_children(children);
+ retired = trie_retired_children(children);
+ retired->pending_node = node;
+ raw_spin_unlock(&pool_lock);
+}
+
+static const struct stack_depot_trie_node *
+stack_depot_trie_lookup(const unsigned long *entries, unsigned int nr_entries);
+
+static depot_stack_handle_t
+trie_find_handle(const unsigned long *entries, unsigned int nr_entries)
+{
+ depot_stack_handle_t handle = 0;
+ const struct stack_depot_trie_node *node;
+
+ rcu_read_lock_sched_notrace();
+ node = stack_depot_trie_lookup(entries, nr_entries);
+ if (node)
+ handle = trie_handle(node->stack_id);
+ rcu_read_unlock_sched_notrace();
+
+ return handle;
+}
+
+/*
+ * Publish only after the node and its path are fully initialized and all
+ * fallible allocation is complete. Publication commits the path, so it cannot
+ * then be rolled back. Side-table mappings must precede trie topology
+ * publication that makes new or remapped nodes reachable from lookup.
+ * Published storage remains valid until RCU retirement; only descendant parent
+ * links may change meanwhile.
+ */
+static void trie_side_table_publish(const struct stack_depot_trie_node *node)
+{
+ const struct stack_depot_trie_node __rcu **chunk;
+ struct stack_depot_trie_side_dir *dir;
+ u32 stack_id = node->stack_id;
+
+ lockdep_assert_held(&stack_depot_trie_writer_lock);
+
+ dir = trie_side_table_load_dir(trie_side_table_root_index(stack_id));
+ chunk = trie_side_table_dir_load_chunk(dir,
+ trie_side_table_dir_index(stack_id));
+ /* Pairs with trie_side_table_lookup(). */
+ rcu_assign_pointer(chunk[trie_side_table_slot_index(stack_id)], node);
+}
+
static int __init disable_stack_depot(char *str)
{
return kstrtobool(str, &stack_depot_disabled);
@@ -146,7 +777,7 @@ static void init_stack_table(unsigned long entries)
INIT_LIST_HEAD(&stack_table[i]);
}
-/* Allocates a hash table via memblock. Can only be used during early boot. */
+/* Initializes hash and optional trie storage during early boot. */
int __init stack_depot_early_init(void)
{
unsigned long entries = 0;
@@ -220,11 +851,15 @@ int __init stack_depot_early_init(void)
stack_depot_disabled = true;
return -ENOMEM;
}
+ if (stack_depot_trie_requested && stack_depot_trie_init_memblock()) {
+ pr_warn("trie storage initialization failed, disabling trie storage\n");
+ stack_depot_trie_requested = false;
+ }
return 0;
}
-/* Allocates a hash table via kvcalloc. Can be used after boot. */
+/* Initializes hash and optional trie storage after boot. */
int stack_depot_init(void)
{
static DEFINE_MUTEX(stack_depot_init_mutex);
@@ -278,6 +913,15 @@ int stack_depot_init(void)
kvfree(stack_table);
stack_depot_disabled = true;
ret = -ENOMEM;
+ goto out_unlock;
+ }
+ if (stack_depot_trie_requested) {
+ ret = stack_depot_trie_init();
+ if (ret) {
+ pr_warn("trie storage initialization failed, disabling trie storage\n");
+ stack_depot_trie_requested = false;
+ ret = 0;
+ }
}
out_unlock:
@@ -323,7 +967,7 @@ static bool depot_init_pool(void **prealloc)
* NULL; do not reset to NULL if we have reached the maximum number of
* pools.
*/
- if (pools_num < stack_max_pools)
+ if (pools_num + 1 < stack_max_pools)
WRITE_ONCE(new_pool, NULL);
else
WRITE_ONCE(new_pool, STACK_DEPOT_POISON);
@@ -467,6 +1111,7 @@ depot_alloc_stack(unsigned long *entries, unsigned int nr_entries, u32 hash, dep
/* Save the stack trace. */
stack->hash = hash;
stack->size = nr_entries;
+ stack->flags = flags & STACK_DEPOT_FLAG_COUNTABLE;
/* stack->handle is already filled in by depot_pop_free_pool(). */
memcpy(stack->entries, entries, flex_array_size(stack, entries, nr_entries));
@@ -609,6 +1254,9 @@ static inline struct stack_record *find_stack(struct list_head *bucket,
list_for_each_entry_rcu(stack, bucket, hash_list) {
if (stack->hash != hash || stack->size != size)
continue;
+ /* Page owner countable records have a distinct count lifetime. */
+ if ((stack->flags ^ flags) & STACK_DEPOT_FLAG_COUNTABLE)
+ continue;
/*
* This may race with depot_free_stack() accessing the freelist
@@ -638,6 +1286,63 @@ static inline struct stack_record *find_stack(struct list_head *bucket,
return ret;
}
+static u32
+stack_depot_trie_insert(const unsigned long *entries,
+ unsigned int nr_entries, void **pool_prealloc,
+ struct stack_depot_trie_side_prealloc *side_prealloc);
+
+static depot_stack_handle_t
+stack_depot_trie_save(unsigned long *entries, unsigned int nr_entries,
+ gfp_t alloc_flags)
+{
+ unsigned int attempt;
+
+ /* Allow one stale pool hint before the two pools a largest insert needs. */
+ for (attempt = 0; attempt < 3; attempt++) {
+ struct stack_depot_trie_side_prealloc side_prealloc = {};
+ void *pool_prealloc = NULL;
+ depot_stack_handle_t handle;
+ unsigned long flags;
+ struct page *page;
+ u32 stack_id = 0;
+
+ handle = trie_find_handle(entries, nr_entries);
+ if (handle)
+ return handle;
+
+ if (trie_side_table_get_prealloc(alloc_flags, &side_prealloc)) {
+ trie_side_table_put_prealloc(&side_prealloc);
+ return 0;
+ }
+
+ /* The hint may race; a missing page is recovered by the retry. */
+ if (!READ_ONCE(new_pool)) {
+ page = alloc_pages(gfp_nested_mask(alloc_flags),
+ DEPOT_POOL_ORDER);
+ if (page)
+ pool_prealloc = page_address(page);
+ }
+
+ raw_spin_lock_irqsave(&stack_depot_trie_writer_lock, flags);
+ stack_id = stack_depot_trie_insert(entries, nr_entries,
+ &pool_prealloc, &side_prealloc);
+ raw_spin_unlock_irqrestore(&stack_depot_trie_writer_lock, flags);
+
+ if (pool_prealloc) {
+ raw_spin_lock_irqsave(&pool_lock, flags);
+ depot_keep_new_pool(&pool_prealloc);
+ raw_spin_unlock_irqrestore(&pool_lock, flags);
+ }
+ if (pool_prealloc)
+ free_pages((unsigned long)pool_prealloc, DEPOT_POOL_ORDER);
+ trie_side_table_put_prealloc(&side_prealloc);
+ if (stack_id)
+ return trie_handle(stack_id);
+ }
+
+ return 0;
+}
+
depot_stack_handle_t stack_depot_save_flags(unsigned long *entries,
unsigned int nr_entries,
gfp_t alloc_flags,
@@ -655,6 +1360,9 @@ depot_stack_handle_t stack_depot_save_flags(unsigned long *entries,
if (WARN_ON(depot_flags & ~STACK_DEPOT_FLAGS_MASK))
return 0;
+ if (WARN_ON_ONCE((depot_flags & STACK_DEPOT_FLAG_GET) &&
+ (depot_flags & STACK_DEPOT_FLAG_COUNTABLE)))
+ return 0;
/*
* If this stack trace is from an interrupt, including anything before
@@ -669,6 +1377,17 @@ depot_stack_handle_t stack_depot_save_flags(unsigned long *entries,
if (unlikely(nr_entries == 0) || stack_depot_disabled)
return 0;
+ if (!(depot_flags & (STACK_DEPOT_FLAG_GET | STACK_DEPOT_FLAG_COUNTABLE)) &&
+ static_branch_unlikely(&stack_depot_trie_enabled)) {
+ if (nr_entries > CONFIG_STACKDEPOT_MAX_FRAMES)
+ nr_entries = CONFIG_STACKDEPOT_MAX_FRAMES;
+ if (in_nmi() || !can_alloc) {
+ WARN_ON_ONCE(can_alloc);
+ return trie_find_handle(entries, nr_entries);
+ }
+ return stack_depot_trie_save(entries, nr_entries, alloc_flags);
+ }
+
hash = hash_stack(entries, nr_entries);
bucket = &stack_table[hash & stack_hash_mask];
@@ -751,10 +1470,697 @@ EXPORT_SYMBOL_GPL(stack_depot_save);
struct stack_record *__stack_depot_get_stack_record(depot_stack_handle_t handle)
{
+ struct stack_record *stack;
+
if (!handle)
return NULL;
+ if (WARN_ON_ONCE(stack_depot_handle_is_trie(handle)))
+ return NULL;
+
+ stack = depot_fetch_stack(handle);
+ if (!stack)
+ return NULL;
+ if (WARN_ON_ONCE(!(stack->flags & STACK_DEPOT_FLAG_COUNTABLE)))
+ return NULL;
+
+ return stack;
+}
+
+static void frame_run_init(const unsigned long *entries,
+ unsigned int nr_entries,
+ struct stack_depot_frame_run *run)
+{
+ u32 payload;
+ unsigned int i;
+ bool compressed;
+
+ compressed = arch_stack_depot_frame_try_compress(entries[0], &payload);
+ for (i = 1; i < nr_entries; i++) {
+ bool next;
+
+ next = arch_stack_depot_frame_try_compress(entries[i], &payload);
+ if (next != compressed)
+ break;
+ }
- return depot_fetch_stack(handle);
+ /* @i is the first non-matching frame, or @nr_entries if all matched. */
+ run->mode = compressed ? STACK_DEPOT_FRAME_COMPRESSED : STACK_DEPOT_FRAME_RAW;
+ run->nr_entries = i;
+}
+
+static void
+stack_depot_trie_node_frame(const struct stack_depot_trie_node *node,
+ unsigned int index, unsigned long *frame)
+{
+ u32 payload;
+
+ if (node->run.mode == STACK_DEPOT_FRAME_RAW) {
+ memcpy(frame, node->data + index * sizeof(*frame),
+ sizeof(*frame));
+ return;
+ }
+
+ memcpy(&payload, node->data + index * sizeof(payload), sizeof(payload));
+ arch_stack_depot_frame_decompress(payload, frame);
+}
+
+static void trie_node_init(struct stack_depot_trie_node *node,
+ const struct stack_depot_trie_node *parent, u32 stack_id,
+ const unsigned long *entries,
+ const struct stack_depot_frame_run *run)
+{
+ if (run->mode == STACK_DEPOT_FRAME_COMPRESSED) {
+ unsigned int i;
+
+ for (i = 0; i < run->nr_entries; i++) {
+ u32 payload;
+
+ arch_stack_depot_frame_try_compress(entries[i], &payload);
+ memcpy(node->data + i * sizeof(payload), &payload,
+ sizeof(payload));
+ }
+ } else {
+ memcpy(node->data, entries, stack_depot_frame_run_bytes(run));
+ }
+
+ RCU_INIT_POINTER(node->parent, parent);
+ RCU_INIT_POINTER(node->children, NULL);
+ node->stack_id = stack_id;
+ node->run = *run;
+}
+
+static void trie_node_init_slice(struct stack_depot_trie_node *node,
+ const struct stack_depot_trie_node *parent, u32 stack_id,
+ const struct stack_depot_trie_node *src_node,
+ unsigned int start, unsigned int nr_entries)
+{
+ struct stack_depot_frame_run run;
+ size_t entry_bytes;
+
+ run = src_node->run;
+ run.nr_entries = nr_entries;
+
+ entry_bytes = stack_depot_frame_run_entry_bytes(src_node->run.mode);
+ memcpy(node->data, src_node->data + start * entry_bytes,
+ stack_depot_frame_run_bytes(&run));
+ RCU_INIT_POINTER(node->parent, parent);
+ RCU_INIT_POINTER(node->children, NULL);
+ node->stack_id = stack_id;
+ node->run = run;
+}
+
+static unsigned int trie_node_match(const struct stack_depot_trie_node *node,
+ const unsigned long *entries,
+ unsigned int nr_entries)
+{
+ unsigned int limit;
+ unsigned int i;
+
+ limit = min(node->run.nr_entries, nr_entries);
+ if (node->run.mode == STACK_DEPOT_FRAME_RAW) {
+ for (i = 0; i < limit; i++) {
+ unsigned long frame;
+
+ memcpy(&frame, node->data + i * sizeof(frame), sizeof(frame));
+ if (frame != entries[i])
+ break;
+ }
+
+ return i;
+ }
+
+ for (i = 0; i < limit; i++) {
+ unsigned long frame;
+
+ stack_depot_trie_node_frame(node, i, &frame);
+ if (frame != entries[i])
+ break;
+ }
+
+ return i;
+}
+
+static inline const struct stack_depot_trie_node *
+trie_load_parent(const struct stack_depot_trie_node *node)
+{
+ return rcu_dereference_check(node->parent,
+ lockdep_is_held(&stack_depot_trie_writer_lock) ||
+ rcu_read_lock_sched_held());
+}
+
+static inline const struct stack_depot_trie_children *
+trie_load_children(const struct stack_depot_trie_children __rcu * const *slot)
+{
+ return rcu_dereference_check(*slot,
+ lockdep_is_held(&stack_depot_trie_writer_lock) ||
+ rcu_read_lock_sched_held());
+}
+
+static inline const struct stack_depot_trie_node *
+trie_children_load_child(const struct stack_depot_trie_children *children,
+ unsigned int pos)
+{
+ return rcu_dereference_check(children->nodes[pos],
+ lockdep_is_held(&stack_depot_trie_writer_lock) ||
+ rcu_read_lock_sched_held());
+}
+
+static bool
+trie_children_find_position(const struct stack_depot_trie_children *children,
+ unsigned long frame, unsigned int *pos)
+{
+ unsigned int left = 0;
+ unsigned int right;
+
+ right = READ_ONCE(children->nr_children);
+ while (left < right) {
+ unsigned int mid = left + (right - left) / 2;
+ const struct stack_depot_trie_node *node;
+ unsigned long mid_frame;
+
+ node = trie_children_load_child(children, mid);
+ if (!node) {
+ /* Tail append may produce a transient lockless lookup miss. */
+ right = mid;
+ continue;
+ }
+ stack_depot_trie_node_frame(node, 0, &mid_frame);
+ if (mid_frame < frame) {
+ left = mid + 1;
+ } else if (mid_frame > frame) {
+ right = mid;
+ } else {
+ *pos = mid;
+ return true;
+ }
+ }
+
+ *pos = left;
+ return false;
+}
+
+/* Initialize an unpublished container from a stable published prefix. */
+static void trie_children_init(const struct stack_depot_trie_children *old,
+ struct stack_depot_trie_children *new)
+{
+ unsigned int nr_old = old->nr_children;
+ unsigned int i;
+
+ new->nr_children = nr_old;
+ for (i = 0; i < nr_old; i++)
+ RCU_INIT_POINTER(new->nodes[i], trie_children_load_child(old, i));
+ for (i = nr_old; i < new->capacity; i++)
+ RCU_INIT_POINTER(new->nodes[i], NULL);
+}
+
+static void trie_children_insert(struct stack_depot_trie_children *children,
+ const struct stack_depot_trie_node *node,
+ unsigned int pos)
+{
+ unsigned int i;
+
+ for (i = children->nr_children; i > pos; i--)
+ RCU_INIT_POINTER(children->nodes[i],
+ trie_children_load_child(children, i - 1));
+ RCU_INIT_POINTER(children->nodes[pos], node);
+ children->nr_children++;
+}
+
+static void trie_reparent_children(struct stack_depot_trie_node *parent)
+{
+ const struct stack_depot_trie_children *children;
+ unsigned int i;
+
+ lockdep_assert_held(&stack_depot_trie_writer_lock);
+
+ children = trie_load_children(&parent->children);
+ if (!children)
+ return;
+ /*
+ * Replacement nodes reuse unchanged descendant subtrees. Repoint their
+ * parent links before retiring the old parent so fetch never follows a freed
+ * node. Lockless fetches may see the new parent before publication, but the
+ * old and new parent chains contain the same frames and remain RCU-live.
+ */
+ for (i = 0; i < children->nr_children; i++) {
+ struct stack_depot_trie_node *child;
+
+ child = (struct stack_depot_trie_node *)trie_children_load_child(children, i);
+ rcu_assign_pointer(child->parent, parent);
+ }
+}
+
+/*
+ * Split entries into runs, allocate and initialize each node once, and link
+ * adjacent nodes through singleton children. Both trie locks must be held.
+ * Failure walks the unpublished parent chain and releases local ownership.
+ */
+static const struct stack_depot_trie_node *
+trie_path_alloc(const struct stack_depot_trie_node *parent, u32 stack_id,
+ const unsigned long *entries, unsigned int nr_entries,
+ void **pool_prealloc,
+ const struct stack_depot_trie_node **node_out)
+{
+ struct stack_depot_trie_children *path_children = NULL;
+ const struct stack_depot_trie_node *path_root = NULL;
+ const struct stack_depot_trie_node *last_node = parent;
+ unsigned int entry = 0;
+
+ lockdep_assert_held(&pool_lock);
+ lockdep_assert_held(&stack_depot_trie_writer_lock);
+
+ while (entry < nr_entries) {
+ struct stack_depot_frame_run run;
+ struct stack_depot_trie_node *node;
+
+ frame_run_init(&entries[entry], nr_entries - entry, &run);
+ node = trie_pool_alloc(trie_node_bytes(&run), pool_prealloc);
+ if (!node)
+ goto err_release;
+
+ trie_node_init(node, last_node,
+ entry + run.nr_entries == nr_entries ? stack_id : 0,
+ &entries[entry], &run);
+ entry += run.nr_entries;
+ last_node = node;
+ if (!path_root)
+ path_root = node;
+
+ if (path_children)
+ trie_children_insert(path_children, last_node, 0);
+ if (entry < nr_entries) {
+ path_children = trie_pool_alloc_children(1, pool_prealloc);
+ if (!path_children)
+ goto err_release;
+ RCU_INIT_POINTER(node->children, path_children);
+ }
+ }
+
+ *node_out = last_node;
+ return path_root;
+
+err_release:
+ while (last_node != parent) {
+ const struct stack_depot_trie_children *node_children;
+ const struct stack_depot_trie_node *node = last_node;
+
+ last_node = trie_load_parent(node);
+ node_children = trie_load_children(&node->children);
+ if (node_children)
+ trie_pool_release_children(node_children);
+ trie_pool_release(node, trie_node_bytes(&node->run));
+ }
+ return NULL;
+}
+
+static const struct stack_depot_trie_node *
+stack_depot_trie_lookup(const unsigned long *entries, unsigned int nr_entries)
+{
+ const struct stack_depot_trie_children *children;
+ unsigned int entry = 0;
+
+ children = trie_load_children(&stack_depot_trie_root);
+
+ while (entry < nr_entries) {
+ const struct stack_depot_trie_node *node;
+ unsigned int remaining = nr_entries - entry;
+ unsigned int matched;
+ unsigned int pos;
+
+ if (!children)
+ return NULL;
+ if (!trie_children_find_position(children, entries[entry], &pos))
+ return NULL;
+
+ node = trie_children_load_child(children, pos);
+ matched = trie_node_match(node, &entries[entry], remaining);
+ if (matched < node->run.nr_entries)
+ return NULL;
+ entry += matched;
+ if (entry == nr_entries)
+ return node->stack_id ? node : NULL;
+
+ children = trie_load_children(&node->children);
+ }
+
+ return NULL;
+}
+
+static u32
+trie_insert_path(const struct stack_depot_trie_children __rcu **slot,
+ struct stack_depot_trie_node *parent,
+ const struct stack_depot_trie_children *children,
+ unsigned int pos, const unsigned long *entries,
+ unsigned int nr_entries, void **pool_prealloc,
+ struct stack_depot_trie_side_prealloc *side_prealloc)
+{
+ struct stack_depot_trie_children *new_children = NULL;
+ const struct stack_depot_trie_node *path_root;
+ const struct stack_depot_trie_node *node;
+ unsigned int capacity = 1;
+ u32 new_stack_id;
+ bool tail_append = false;
+
+ /*
+ * Reuse spare capacity only for a sorted tail append. Other insertions
+ * replace the children container without modifying visible pointers.
+ */
+ if (children) {
+ capacity = roundup_pow_of_two(children->nr_children + 1);
+ tail_append = pos == children->nr_children &&
+ children->nr_children < children->capacity;
+ }
+ if (!tail_append && trie_children_alloc_size(capacity) >
+ STACK_DEPOT_TRIE_POOL_USABLE_SIZE)
+ return 0;
+
+ new_stack_id = trie_side_table_prepare_stack_slot(side_prealloc);
+ if (!new_stack_id)
+ return 0;
+
+ raw_spin_lock(&pool_lock);
+ printk_deferred_enter();
+ trie_drain_pending_children();
+
+ /* Reserve replacement topology before the path, the final fallible step. */
+ if (!tail_append) {
+ new_children = trie_pool_alloc_children(capacity, pool_prealloc);
+ if (!new_children)
+ goto err_release;
+ }
+ path_root = trie_path_alloc(parent, new_stack_id, entries, nr_entries,
+ pool_prealloc, &node);
+ if (!path_root)
+ goto err_release;
+
+ /* Commit the stack ID before making the path reachable from the trie. */
+ trie_side_table_publish(node);
+ if (tail_append) {
+ struct stack_depot_trie_children *tail_children =
+ (struct stack_depot_trie_children *)children;
+
+ /*
+ * Publish the node before the visible count. Readers may transiently
+ * see NULL and miss; the writer-lock recheck prevents duplicates.
+ */
+ rcu_assign_pointer(tail_children->nodes[pos], path_root);
+ WRITE_ONCE(tail_children->nr_children, pos + 1);
+ } else {
+ if (children)
+ trie_children_init(children, new_children);
+ trie_children_insert(new_children, path_root, pos);
+ rcu_assign_pointer(*slot, new_children);
+ if (children)
+ trie_retire_children(children);
+ }
+
+ printk_deferred_exit();
+ raw_spin_unlock(&pool_lock);
+ return new_stack_id;
+
+err_release:
+ if (new_children)
+ trie_pool_release_children(new_children);
+ printk_deferred_exit();
+ raw_spin_unlock(&pool_lock);
+ return 0;
+}
+
+static u32
+trie_split_child(const struct stack_depot_trie_children __rcu **slot,
+ const struct stack_depot_trie_children *children,
+ const struct stack_depot_trie_node *child,
+ unsigned int pos, unsigned int matched,
+ const unsigned long *entries, unsigned int nr_entries,
+ void **pool_prealloc,
+ struct stack_depot_trie_side_prealloc *side_prealloc)
+{
+ struct stack_depot_trie_children *prefix_children = NULL;
+ struct stack_depot_trie_children *new_children = NULL;
+ const struct stack_depot_trie_node *new_node;
+ const struct stack_depot_trie_node *suffix_roots[2];
+ struct stack_depot_frame_run run;
+ struct stack_depot_trie_node *split_prefix = NULL;
+ struct stack_depot_trie_node *old_suffix = NULL;
+ unsigned int nr_suffix_roots;
+ unsigned int old_suffix_len;
+ unsigned int i;
+ size_t split_prefix_size;
+ size_t old_suffix_size;
+ u32 new_stack_id;
+ bool has_new_suffix;
+
+ new_stack_id = trie_side_table_prepare_stack_slot(side_prealloc);
+ if (!new_stack_id)
+ return 0;
+
+ /* Rebuild the child's run as newly allocated prefix and old suffix nodes. */
+ run = child->run;
+ run.nr_entries = matched;
+ split_prefix_size = trie_node_bytes(&run);
+ old_suffix_len = child->run.nr_entries - matched;
+ run.nr_entries = old_suffix_len;
+ old_suffix_size = trie_node_bytes(&run);
+ has_new_suffix = matched < nr_entries;
+ nr_suffix_roots = has_new_suffix ? 2 : 1;
+
+ raw_spin_lock(&pool_lock);
+ printk_deferred_enter();
+ trie_drain_pending_children();
+
+ /* Reserve fixed split topology before the optional new suffix path. */
+ split_prefix = trie_pool_alloc(split_prefix_size, pool_prealloc);
+ if (!split_prefix)
+ goto err_release;
+ old_suffix = trie_pool_alloc(old_suffix_size, pool_prealloc);
+ if (!old_suffix)
+ goto err_release;
+ new_children = trie_pool_alloc_children(children->capacity, pool_prealloc);
+ if (!new_children)
+ goto err_release;
+ prefix_children = trie_pool_alloc_children(nr_suffix_roots, pool_prealloc);
+ if (!prefix_children)
+ goto err_release;
+
+ if (has_new_suffix) {
+ const struct stack_depot_trie_node *new_suffix;
+ unsigned long old_suffix_frame;
+
+ new_suffix = trie_path_alloc(split_prefix, new_stack_id,
+ &entries[matched], nr_entries - matched,
+ pool_prealloc, &new_node);
+ if (!new_suffix)
+ goto err_release;
+ stack_depot_trie_node_frame(child, matched, &old_suffix_frame);
+ /* Children remain sorted by the first frame of each suffix. */
+ if (old_suffix_frame < entries[matched]) {
+ suffix_roots[0] = old_suffix;
+ suffix_roots[1] = new_suffix;
+ } else {
+ suffix_roots[0] = new_suffix;
+ suffix_roots[1] = old_suffix;
+ }
+ } else {
+ new_node = split_prefix;
+ suffix_roots[0] = old_suffix;
+ }
+
+ printk_deferred_exit();
+ raw_spin_unlock(&pool_lock);
+
+ /* Rebuild the old path as prefix -> old suffix and attach suffix roots. */
+ trie_node_init_slice(split_prefix, trie_load_parent(child),
+ has_new_suffix ? 0 : new_stack_id, child, 0, matched);
+ trie_node_init_slice(old_suffix, split_prefix, child->stack_id, child,
+ matched, old_suffix_len);
+ for (i = 0; i < nr_suffix_roots; i++)
+ trie_children_insert(prefix_children, suffix_roots[i], i);
+ RCU_INIT_POINTER(old_suffix->children,
+ trie_load_children(&child->children));
+ RCU_INIT_POINTER(split_prefix->children, prefix_children);
+
+ /* Publish IDs, reparent descendants, then replace and retire topology. */
+ if (child->stack_id)
+ trie_side_table_publish(old_suffix);
+ trie_side_table_publish(new_node);
+ /* Old and replacement chains contain identical frames during transition. */
+ trie_children_init(children, new_children);
+ RCU_INIT_POINTER(new_children->nodes[pos], split_prefix);
+ trie_reparent_children(old_suffix);
+ rcu_assign_pointer(*slot, new_children);
+ trie_retire_children_with_node(children, child);
+
+ return new_stack_id;
+
+err_release:
+ if (split_prefix)
+ trie_pool_release(split_prefix, split_prefix_size);
+ if (old_suffix)
+ trie_pool_release(old_suffix, old_suffix_size);
+ if (prefix_children)
+ trie_pool_release_children(prefix_children);
+ if (new_children)
+ trie_pool_release_children(new_children);
+ printk_deferred_exit();
+ raw_spin_unlock(&pool_lock);
+ return 0;
+}
+
+static u32
+trie_promote_child(const struct stack_depot_trie_children __rcu **slot,
+ const struct stack_depot_trie_children *children,
+ const struct stack_depot_trie_node *child,
+ unsigned int pos, void **pool_prealloc,
+ struct stack_depot_trie_side_prealloc *side_prealloc)
+{
+ struct stack_depot_trie_children *new_children;
+ struct stack_depot_trie_node *promoted_node;
+ size_t node_size;
+ u32 new_stack_id;
+
+ new_stack_id = trie_side_table_prepare_stack_slot(side_prealloc);
+ if (!new_stack_id)
+ return 0;
+ node_size = trie_node_bytes(&child->run);
+
+ /* Reserve a clone and replacement children container before publication. */
+ raw_spin_lock(&pool_lock);
+ printk_deferred_enter();
+ trie_drain_pending_children();
+ promoted_node = trie_pool_alloc(node_size, pool_prealloc);
+ if (!promoted_node)
+ goto out_unlock;
+ new_children = trie_pool_alloc_children(children->capacity, pool_prealloc);
+ if (!new_children)
+ goto out_release_node;
+ printk_deferred_exit();
+ raw_spin_unlock(&pool_lock);
+
+ /* Add the stack ID through a clone, then reparent before retirement. */
+ memcpy(promoted_node, child, node_size);
+ promoted_node->stack_id = new_stack_id;
+ trie_side_table_publish(promoted_node);
+ trie_children_init(children, new_children);
+ RCU_INIT_POINTER(new_children->nodes[pos], promoted_node);
+ trie_reparent_children(promoted_node);
+ rcu_assign_pointer(*slot, new_children);
+ trie_retire_children_with_node(children, child);
+
+ return new_stack_id;
+
+out_release_node:
+ trie_pool_release(promoted_node, node_size);
+out_unlock:
+ printk_deferred_exit();
+ raw_spin_unlock(&pool_lock);
+ return 0;
+}
+
+static u32
+stack_depot_trie_insert(const unsigned long *entries,
+ unsigned int nr_entries, void **pool_prealloc,
+ struct stack_depot_trie_side_prealloc *side_prealloc)
+{
+ const struct stack_depot_trie_children *children;
+ const struct stack_depot_trie_children __rcu **slot =
+ &stack_depot_trie_root;
+ const struct stack_depot_trie_node *child;
+ struct stack_depot_trie_node *parent = NULL;
+ unsigned int matched;
+ unsigned int pos;
+ u32 stack_id;
+
+ lockdep_assert_held(&stack_depot_trie_writer_lock);
+
+ for (;;) {
+ pos = 0;
+ children = trie_load_children(slot);
+ /* No matching child: attach the remaining path. */
+ if (!children ||
+ !trie_children_find_position(children, entries[0], &pos)) {
+ stack_id = trie_insert_path(slot, parent, children, pos,
+ entries, nr_entries, pool_prealloc,
+ side_prealloc);
+ break;
+ }
+
+ child = trie_children_load_child(children, pos);
+ matched = trie_node_match(child, entries, nr_entries);
+ /* A partial child match requires a prefix/suffix split. */
+ if (matched < child->run.nr_entries) {
+ stack_id = trie_split_child(slot, children, child, pos,
+ matched, entries, nr_entries,
+ pool_prealloc, side_prealloc);
+ break;
+ }
+
+ /* The input ends here: reuse a stack node or promote an internal one. */
+ if (matched == nr_entries) {
+ if (child->stack_id)
+ return child->stack_id;
+ stack_id = trie_promote_child(slot, children, child, pos,
+ pool_prealloc, side_prealloc);
+ break;
+ }
+
+ /* The child matched completely; continue with the remaining frames. */
+ parent = (struct stack_depot_trie_node *)child;
+ slot = &parent->children;
+ entries += matched;
+ nr_entries -= matched;
+ }
+
+ if (stack_id)
+ trie_side_table_last_stack_id = stack_id;
+ return stack_id;
+}
+
+static unsigned int trie_fetch_into(const struct stack_depot_trie_node *node,
+ unsigned long *entries,
+ unsigned int max_entries)
+{
+ const struct stack_depot_trie_node *cur;
+ unsigned int total;
+ unsigned int pos;
+ unsigned int i;
+
+ total = 0;
+ for (cur = node; cur; cur = trie_load_parent(cur))
+ total += cur->run.nr_entries;
+ if (max_entries < total)
+ return 0;
+
+ pos = total;
+ for (cur = node; cur; cur = trie_load_parent(cur)) {
+ pos -= cur->run.nr_entries;
+ for (i = 0; i < cur->run.nr_entries; i++)
+ stack_depot_trie_node_frame(cur, i, &entries[pos + i]);
+ }
+
+ return total;
+}
+
+static unsigned int trie_fetch_handle_into(depot_stack_handle_t handle,
+ unsigned long *entries,
+ unsigned int max_entries)
+{
+ const struct stack_depot_trie_node *node;
+ u32 stack_id;
+ unsigned int nr_entries;
+
+ stack_id = trie_stack_id(handle);
+ rcu_read_lock_sched_notrace();
+ node = trie_side_table_lookup(stack_id);
+ if (WARN_ONCE(!node, "corrupt trie handle %08x\n", handle)) {
+ rcu_read_unlock_sched_notrace();
+ return 0;
+ }
+ nr_entries = trie_fetch_into(node, entries, max_entries);
+ rcu_read_unlock_sched_notrace();
+ if (nr_entries)
+ kmsan_unpoison_memory(entries, nr_entries * sizeof(*entries));
+
+ return nr_entries;
}
unsigned int stack_depot_fetch(depot_stack_handle_t handle,
@@ -771,6 +2177,8 @@ unsigned int stack_depot_fetch(depot_stack_handle_t handle,
if (!handle || stack_depot_disabled)
return 0;
+ if (WARN_ON_ONCE(stack_depot_handle_is_trie(handle)))
+ return 0;
stack = depot_fetch_stack(handle);
/*
@@ -785,12 +2193,44 @@ unsigned int stack_depot_fetch(depot_stack_handle_t handle,
}
EXPORT_SYMBOL_GPL(stack_depot_fetch);
+unsigned int stack_depot_fetch_into(depot_stack_handle_t handle,
+ unsigned long *entries,
+ unsigned int max_entries)
+{
+ struct stack_record *stack;
+ unsigned int nr_entries;
+
+ if (!handle)
+ return 0;
+ if (stack_depot_disabled)
+ return 0;
+ WARN_ON_ONCE(!entries || !max_entries);
+ if (stack_depot_handle_is_trie(handle))
+ return trie_fetch_handle_into(handle, entries, max_entries);
+
+ stack = depot_fetch_stack(handle);
+ if (!stack)
+ return 0;
+ nr_entries = stack->size;
+ if (WARN_ON_ONCE(!nr_entries))
+ return 0;
+ if (nr_entries > max_entries)
+ return 0;
+
+ memcpy(entries, stack->entries, nr_entries * sizeof(*entries));
+ kmsan_unpoison_memory(entries, nr_entries * sizeof(*entries));
+ return nr_entries;
+}
+EXPORT_SYMBOL_GPL(stack_depot_fetch_into);
+
void stack_depot_put(depot_stack_handle_t handle)
{
struct stack_record *stack;
if (!handle || stack_depot_disabled)
return;
+ if (WARN_ON_ONCE(stack_depot_handle_is_trie(handle)))
+ return;
stack = depot_fetch_stack(handle);
/*
@@ -800,6 +2240,8 @@ void stack_depot_put(depot_stack_handle_t handle)
if (WARN(!stack, "corrupt handle or unbalanced stack_depot_put()"))
return;
+ if (WARN_ON_ONCE(stack->flags & STACK_DEPOT_FLAG_COUNTABLE))
+ return;
if (refcount_dec_and_test(&stack->count))
depot_free_stack(stack);
}
@@ -810,6 +2252,15 @@ void stack_depot_print(depot_stack_handle_t stack)
unsigned long *entries;
unsigned int nr_entries;
+ if (stack_depot_handle_is_trie(stack)) {
+ unsigned long trie_entries[CONFIG_STACKDEPOT_MAX_FRAMES];
+
+ nr_entries = trie_fetch_handle_into(stack, trie_entries,
+ ARRAY_SIZE(trie_entries));
+ stack_trace_print(trie_entries, nr_entries, 0);
+ return;
+ }
+
nr_entries = stack_depot_fetch(stack, &entries);
if (nr_entries > 0)
stack_trace_print(entries, nr_entries, 0);
@@ -822,6 +2273,15 @@ int stack_depot_snprint(depot_stack_handle_t handle, char *buf, size_t size,
unsigned long *entries;
unsigned int nr_entries;
+ if (stack_depot_handle_is_trie(handle)) {
+ unsigned long trie_entries[CONFIG_STACKDEPOT_MAX_FRAMES];
+
+ nr_entries = trie_fetch_handle_into(handle, trie_entries,
+ ARRAY_SIZE(trie_entries));
+ return stack_trace_snprint(buf, size, trie_entries, nr_entries,
+ spaces);
+ }
+
nr_entries = stack_depot_fetch(handle, &entries);
return nr_entries ? stack_trace_snprint(buf, size, entries, nr_entries,
spaces) : 0;
diff --git a/lib/tests/Makefile b/lib/tests/Makefile
index 4ead57602eac4..2d40bd21a8ef9 100644
--- a/lib/tests/Makefile
+++ b/lib/tests/Makefile
@@ -49,6 +49,7 @@ obj-$(CONFIG_SCANF_KUNIT_TEST) += scanf_kunit.o
obj-$(CONFIG_SEQ_BUF_KUNIT_TEST) += seq_buf_kunit.o
obj-$(CONFIG_SIPHASH_KUNIT_TEST) += siphash_kunit.o
obj-$(CONFIG_SLUB_KUNIT_TEST) += slub_kunit.o
+obj-$(CONFIG_STACKDEPOT_KUNIT_TEST) += stackdepot_kunit.o
obj-$(CONFIG_TEST_SORT) += test_sort.o
CFLAGS_stackinit_kunit.o += $(call cc-disable-warning, switch-unreachable)
obj-$(CONFIG_STACKINIT_KUNIT_TEST) += stackinit_kunit.o
diff --git a/lib/tests/stackdepot_kunit.c b/lib/tests/stackdepot_kunit.c
new file mode 100644
index 0000000000000..be14cae98fcfb
--- /dev/null
+++ b/lib/tests/stackdepot_kunit.c
@@ -0,0 +1,473 @@
+// SPDX-License-Identifier: GPL-2.0-only
+
+#include <kunit/test.h>
+#include <linux/array_size.h>
+#include <linux/gfp.h>
+#include <linux/limits.h>
+#include <linux/moduleparam.h>
+#include <linux/stackdepot.h>
+#include <linux/stacktrace.h>
+#include <linux/string.h>
+
+#include <asm/stackdepot.h>
+
+static int expected_trie_pool_limit = -1;
+module_param_named(trie_pool_limit, expected_trie_pool_limit, int, 0);
+MODULE_PARM_DESC(trie_pool_limit, "Expected stackdepot hash/trie pool split");
+
+#ifdef CONFIG_ARM64
+#include <asm/sections.h>
+
+static inline unsigned long stackdepot_arm64_frame(long offset)
+{
+ return (unsigned long)((long)_text + offset);
+}
+#endif
+
+static void stackdepot_trie_max_path_roundtrip(struct kunit *test)
+{
+ union handle_parts parts;
+ unsigned long *entries;
+ unsigned long *fetched;
+ depot_stack_handle_t handle;
+ size_t size = CONFIG_STACKDEPOT_MAX_FRAMES * sizeof(*entries);
+ u32 pool_index_plus_1;
+ unsigned int i;
+
+ if (expected_trie_pool_limit < 0)
+ kunit_skip(test, "trie pool limit was not provided");
+ KUNIT_ASSERT_EQ(test, stack_depot_init(), 0);
+ entries = kunit_kcalloc(test, CONFIG_STACKDEPOT_MAX_FRAMES,
+ sizeof(*entries), GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, entries);
+ fetched = kunit_kcalloc(test, CONFIG_STACKDEPOT_MAX_FRAMES,
+ sizeof(*fetched), GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, fetched);
+ for (i = 0; i < CONFIG_STACKDEPOT_MAX_FRAMES; i++) {
+#ifdef CONFIG_ARM64
+ entries[i] = i & 1 ? 0x1000UL + i * 0x1000UL :
+ stackdepot_arm64_frame(i * 4);
+#elif defined(CONFIG_X86_64) && !defined(CONFIG_UML)
+ entries[i] = i & 1 ? 0xffff888000000000UL + i * 0x1000UL :
+ 0xffffffff10000000UL + i * 0x10UL;
+#else
+ entries[i] = 0x1000UL + i * 0x1000UL;
+#endif
+ }
+
+ handle = stack_depot_save(entries, CONFIG_STACKDEPOT_MAX_FRAMES,
+ GFP_KERNEL);
+ KUNIT_ASSERT_NE(test, handle, (depot_stack_handle_t)0);
+ parts.handle = handle;
+ pool_index_plus_1 = parts.pool_index_plus_1;
+ KUNIT_EXPECT_GT(test, pool_index_plus_1, (u32)expected_trie_pool_limit);
+ KUNIT_EXPECT_EQ(test,
+ stack_depot_fetch_into(handle, fetched,
+ CONFIG_STACKDEPOT_MAX_FRAMES),
+ (unsigned int)CONFIG_STACKDEPOT_MAX_FRAMES);
+ KUNIT_EXPECT_MEMEQ(test, fetched, entries, size);
+ KUNIT_EXPECT_EQ(test,
+ stack_depot_save(entries, CONFIG_STACKDEPOT_MAX_FRAMES,
+ GFP_KERNEL),
+ handle);
+}
+
+static void stackdepot_save_flags_public(struct kunit *test)
+{
+ unsigned long entries[] = { 0x501000UL, 0x502000UL, 0x503000UL };
+ unsigned long get_entries[] = { 0x601000UL, 0x602000UL };
+ unsigned long missing_entries[] = { 0x701000UL, 0x702000UL };
+ unsigned long fetched[ARRAY_SIZE(entries)] = {};
+ depot_stack_handle_t noalloc_handle;
+ depot_stack_handle_t overlong_handle;
+ depot_stack_handle_t plain_handle;
+ depot_stack_handle_t get_handle;
+ depot_stack_handle_t again;
+ depot_stack_handle_t extra;
+ gfp_t no_spin = GFP_NOWAIT & ~__GFP_RECLAIM;
+ unsigned long *overlong_fetched;
+ unsigned long *overlong_entries;
+ unsigned int overlong_nr = CONFIG_STACKDEPOT_MAX_FRAMES + 1;
+ unsigned int nr_entries;
+ size_t overlong_size;
+ unsigned int i;
+
+ KUNIT_ASSERT_EQ(test, stack_depot_init(), 0);
+ overlong_entries = kunit_kcalloc(test, overlong_nr,
+ sizeof(*overlong_entries), GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, overlong_entries);
+ overlong_fetched = kunit_kcalloc(test, CONFIG_STACKDEPOT_MAX_FRAMES,
+ sizeof(*overlong_fetched), GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, overlong_fetched);
+ for (i = 0; i < overlong_nr; i++)
+ overlong_entries[i] = 0x800000UL + i * 0x1000UL;
+
+ plain_handle = stack_depot_save(entries, ARRAY_SIZE(entries), GFP_KERNEL);
+ KUNIT_ASSERT_NE(test, plain_handle, (depot_stack_handle_t)0);
+ again = stack_depot_save(entries, ARRAY_SIZE(entries), GFP_KERNEL);
+ KUNIT_EXPECT_EQ(test, again, plain_handle);
+
+ nr_entries = stack_depot_fetch_into(plain_handle, fetched,
+ ARRAY_SIZE(fetched));
+ KUNIT_EXPECT_EQ(test, nr_entries, (unsigned int)ARRAY_SIZE(entries));
+ KUNIT_EXPECT_MEMEQ(test, fetched, entries, sizeof(entries));
+
+ noalloc_handle = stack_depot_save_flags(entries, ARRAY_SIZE(entries), no_spin, 0);
+ KUNIT_EXPECT_EQ(test, noalloc_handle, plain_handle);
+ if (expected_trie_pool_limit >= 0) {
+ noalloc_handle =
+ stack_depot_save_flags(missing_entries,
+ ARRAY_SIZE(missing_entries),
+ no_spin, 0);
+ KUNIT_EXPECT_EQ(test, noalloc_handle, (depot_stack_handle_t)0);
+ }
+
+ get_handle = stack_depot_save_flags(get_entries, ARRAY_SIZE(get_entries),
+ GFP_KERNEL,
+ STACK_DEPOT_FLAG_CAN_ALLOC |
+ STACK_DEPOT_FLAG_GET);
+ KUNIT_ASSERT_NE(test, get_handle, (depot_stack_handle_t)0);
+ stack_depot_put(get_handle);
+
+ overlong_handle = stack_depot_save(overlong_entries, overlong_nr,
+ GFP_KERNEL);
+ KUNIT_ASSERT_NE(test, overlong_handle, (depot_stack_handle_t)0);
+ nr_entries = stack_depot_fetch_into(overlong_handle, overlong_fetched,
+ CONFIG_STACKDEPOT_MAX_FRAMES);
+ KUNIT_EXPECT_EQ(test, nr_entries, (unsigned int)CONFIG_STACKDEPOT_MAX_FRAMES);
+ overlong_size = CONFIG_STACKDEPOT_MAX_FRAMES * sizeof(*overlong_entries);
+ KUNIT_EXPECT_MEMEQ(test, overlong_fetched, overlong_entries, overlong_size);
+
+ extra = stack_depot_set_extra_bits(plain_handle, 7);
+ KUNIT_ASSERT_NE(test, extra, (depot_stack_handle_t)0);
+ KUNIT_EXPECT_EQ(test, stack_depot_get_extra_bits(extra), 7U);
+ memset(fetched, 0, sizeof(fetched));
+ nr_entries = stack_depot_fetch_into(extra, fetched, ARRAY_SIZE(fetched));
+ KUNIT_EXPECT_EQ(test, nr_entries, (unsigned int)ARRAY_SIZE(entries));
+ KUNIT_EXPECT_MEMEQ(test, fetched, entries, sizeof(entries));
+}
+
+static void stackdepot_snprint_public(struct kunit *test)
+{
+ unsigned long entries[] = { 0x1000UL, 0x2000UL, 0x3000UL };
+ char expected[256];
+ char actual[256];
+ depot_stack_handle_t handle;
+ unsigned int expected_len;
+ int actual_len;
+
+ KUNIT_ASSERT_EQ(test, stack_depot_init(), 0);
+ handle = stack_depot_save(entries, ARRAY_SIZE(entries), GFP_KERNEL);
+ KUNIT_ASSERT_NE(test, handle, (depot_stack_handle_t)0);
+
+ expected_len = stack_trace_snprint(expected, sizeof(expected), entries,
+ ARRAY_SIZE(entries), 2);
+ actual_len = stack_depot_snprint(handle, actual, sizeof(actual), 2);
+ KUNIT_EXPECT_EQ(test, actual_len, (int)expected_len);
+ KUNIT_EXPECT_STREQ(test, actual, expected);
+}
+
+static void stackdepot_countable_public(struct kunit *test)
+{
+ unsigned long plain_entries[] = {
+ 0x141000UL,
+ 0x142000UL,
+ 0x143000UL,
+ };
+ unsigned long get_entries[] = {
+ 0x151000UL,
+ 0x152000UL,
+ 0x153000UL,
+ };
+ unsigned long fetched[ARRAY_SIZE(plain_entries)] = {};
+ depot_flags_t countable = STACK_DEPOT_FLAG_CAN_ALLOC |
+ STACK_DEPOT_FLAG_COUNTABLE;
+ struct stack_record *record;
+ depot_stack_handle_t count_handle;
+ depot_stack_handle_t plain_handle;
+ depot_stack_handle_t get_handle;
+ unsigned int get_nr = ARRAY_SIZE(get_entries);
+ unsigned int plain_nr = ARRAY_SIZE(plain_entries);
+ unsigned int nr_entries;
+
+ KUNIT_ASSERT_EQ(test, stack_depot_init(), 0);
+
+ plain_handle = stack_depot_save(plain_entries, plain_nr, GFP_KERNEL);
+ KUNIT_ASSERT_NE(test, plain_handle, (depot_stack_handle_t)0);
+ count_handle = stack_depot_save_flags(plain_entries, plain_nr, GFP_KERNEL,
+ countable);
+ KUNIT_ASSERT_NE(test, count_handle, (depot_stack_handle_t)0);
+ record = __stack_depot_get_stack_record(count_handle);
+ KUNIT_ASSERT_NOT_NULL(test, record);
+ KUNIT_EXPECT_EQ(test, record->size, (u16)plain_nr);
+ KUNIT_EXPECT_MEMEQ(test, record->entries, plain_entries,
+ sizeof(plain_entries));
+ nr_entries = stack_depot_fetch_into(count_handle, fetched,
+ ARRAY_SIZE(fetched));
+ KUNIT_EXPECT_EQ(test, nr_entries, plain_nr);
+ KUNIT_EXPECT_MEMEQ(test, fetched, plain_entries, sizeof(plain_entries));
+
+ get_handle = stack_depot_save_flags(get_entries, get_nr, GFP_KERNEL,
+ STACK_DEPOT_FLAG_CAN_ALLOC |
+ STACK_DEPOT_FLAG_GET);
+ KUNIT_ASSERT_NE(test, get_handle, (depot_stack_handle_t)0);
+ count_handle = stack_depot_save_flags(get_entries, get_nr, GFP_KERNEL,
+ countable);
+ KUNIT_ASSERT_NE(test, count_handle, (depot_stack_handle_t)0);
+ record = __stack_depot_get_stack_record(count_handle);
+ KUNIT_ASSERT_NOT_NULL(test, record);
+ KUNIT_EXPECT_MEMEQ(test, record->entries, get_entries, sizeof(get_entries));
+
+ stack_depot_put(get_handle);
+}
+
+static void stackdepot_fetch_into_roundtrip(struct kunit *test)
+{
+ unsigned long entries[] = {
+ 0x101000UL,
+ 0x102000UL,
+ 0x103000UL,
+ };
+ unsigned long exact[ARRAY_SIZE(entries)] = {};
+ unsigned long fetched[ARRAY_SIZE(entries) + 1] = {
+ [ARRAY_SIZE(entries)] = 0xa5a5a5a5UL,
+ };
+ unsigned long expected_tail = fetched[ARRAY_SIZE(entries)];
+ depot_stack_handle_t handle;
+ unsigned int nr_entries;
+
+ KUNIT_ASSERT_EQ(test, stack_depot_init(), 0);
+
+ handle = stack_depot_save(entries, ARRAY_SIZE(entries), GFP_KERNEL);
+ KUNIT_ASSERT_NE(test, handle, (depot_stack_handle_t)0);
+
+ nr_entries = stack_depot_fetch_into(handle, exact, ARRAY_SIZE(exact));
+ KUNIT_EXPECT_EQ(test, nr_entries, (unsigned int)ARRAY_SIZE(entries));
+ KUNIT_EXPECT_MEMEQ(test, exact, entries, sizeof(entries));
+
+ nr_entries = stack_depot_fetch_into(handle, fetched, ARRAY_SIZE(fetched));
+ KUNIT_EXPECT_EQ(test, nr_entries, (unsigned int)ARRAY_SIZE(entries));
+ KUNIT_EXPECT_MEMEQ(test, fetched, entries, sizeof(entries));
+ KUNIT_EXPECT_EQ(test, fetched[ARRAY_SIZE(entries)], expected_tail);
+}
+
+static void stackdepot_fetch_into_rejects_missing_or_short_stack(struct kunit *test)
+{
+ unsigned long entries[] = {
+ 0x111000UL,
+ 0x112000UL,
+ 0x113000UL,
+ };
+ unsigned long fetched[ARRAY_SIZE(entries)] = {
+ 0xa1a1a1a1UL,
+ 0xb2b2b2b2UL,
+ 0xc3c3c3c3UL,
+ };
+ unsigned long expected[ARRAY_SIZE(fetched)];
+ depot_stack_handle_t handle;
+ unsigned int nr_entries;
+
+ KUNIT_ASSERT_EQ(test, stack_depot_init(), 0);
+
+ handle = stack_depot_save(entries, ARRAY_SIZE(entries), GFP_KERNEL);
+ KUNIT_ASSERT_NE(test, handle, (depot_stack_handle_t)0);
+ memcpy(expected, fetched, sizeof(expected));
+
+ nr_entries = stack_depot_fetch_into(0, fetched, ARRAY_SIZE(fetched));
+ KUNIT_EXPECT_EQ(test, nr_entries, 0U);
+ KUNIT_EXPECT_MEMEQ(test, fetched, expected, sizeof(expected));
+
+ nr_entries = stack_depot_fetch_into(0, NULL, 0);
+ KUNIT_EXPECT_EQ(test, nr_entries, 0U);
+
+ nr_entries = stack_depot_fetch_into(handle, fetched,
+ ARRAY_SIZE(fetched) - 1);
+ KUNIT_EXPECT_EQ(test, nr_entries, 0U);
+ KUNIT_EXPECT_MEMEQ(test, fetched, expected, sizeof(expected));
+}
+
+static void stackdepot_trie_topology_roundtrip(struct kunit *test)
+{
+ union handle_parts parts;
+ unsigned long stacks[][3] = {
+ { 0x201000UL, 0x202000UL },
+ { 0x201000UL, 0x203000UL },
+ { 0x201000UL },
+ { 0x201000UL, 0x203000UL, 0x204000UL },
+ { 0x201000UL, 0x205000UL },
+ { 0x201000UL, 0x204000UL },
+ { 0x201000UL, 0x206000UL },
+ { 0x201000UL, 0x207000UL },
+ { 0x301000UL, 0x302000UL },
+ { 0x301000UL, 0x302000UL, 0x303000UL },
+ { 0x301000UL, 0x304000UL },
+ { 0x401000UL, 0x402000UL, 0x403000UL },
+ { 0x401000UL, 0x402000UL },
+ };
+ unsigned int nr_entries[] = { 2, 2, 1, 3, 2, 2, 2, 2, 2, 3, 2, 3, 2 };
+ depot_stack_handle_t handles[ARRAY_SIZE(stacks)];
+ unsigned long fetched[ARRAY_SIZE(stacks[0])];
+ u32 pool_index_plus_1;
+ unsigned int i;
+
+ if (expected_trie_pool_limit < 0)
+ kunit_skip(test, "trie pool limit was not provided");
+ KUNIT_ASSERT_EQ(test, stack_depot_init(), 0);
+
+ for (i = 0; i < ARRAY_SIZE(stacks); i++) {
+ handles[i] = stack_depot_save(stacks[i], nr_entries[i], GFP_KERNEL);
+ KUNIT_ASSERT_NE(test, handles[i], (depot_stack_handle_t)0);
+ }
+ parts.handle = handles[0];
+ pool_index_plus_1 = parts.pool_index_plus_1;
+ KUNIT_ASSERT_GT(test, pool_index_plus_1,
+ (u32)expected_trie_pool_limit);
+
+ for (i = 0; i < ARRAY_SIZE(stacks); i++) {
+ memset(fetched, 0, sizeof(fetched));
+ KUNIT_EXPECT_EQ(test,
+ stack_depot_fetch_into(handles[i], fetched,
+ ARRAY_SIZE(fetched)),
+ nr_entries[i]);
+ KUNIT_EXPECT_MEMEQ(test, fetched, stacks[i],
+ nr_entries[i] * sizeof(fetched[0]));
+ KUNIT_EXPECT_EQ(test,
+ stack_depot_save(stacks[i], nr_entries[i], GFP_KERNEL),
+ handles[i]);
+ }
+}
+
+static void stackdepot_frame_storage_roundtrip(struct kunit *test)
+{
+ union handle_parts parts;
+ unsigned long fetched[3] = {};
+ depot_stack_handle_t handle;
+ u32 pool_index_plus_1;
+ unsigned int nr_entries;
+#if defined(CONFIG_ARM64)
+ unsigned long entries[] = {
+ stackdepot_arm64_frame(S32_MIN),
+ 0x1000UL,
+ stackdepot_arm64_frame(S32_MAX),
+ };
+#elif defined(CONFIG_X86_64)
+ unsigned long entries[] = {
+ 0xffffffff10001000UL,
+ 0xffff888000001000UL,
+ 0xffffffff20002000UL,
+ };
+#else
+ unsigned long entries[] = { 0x301000UL, 0x302000UL, 0x303000UL };
+#endif
+
+ if (expected_trie_pool_limit < 0)
+ kunit_skip(test, "trie pool limit was not provided");
+ KUNIT_ASSERT_EQ(test, stack_depot_init(), 0);
+ handle = stack_depot_save(entries, ARRAY_SIZE(entries), GFP_KERNEL);
+ KUNIT_ASSERT_NE(test, handle, (depot_stack_handle_t)0);
+ parts.handle = handle;
+ pool_index_plus_1 = parts.pool_index_plus_1;
+ KUNIT_ASSERT_GT(test, pool_index_plus_1,
+ (u32)expected_trie_pool_limit);
+
+ nr_entries = stack_depot_fetch_into(handle, fetched, ARRAY_SIZE(fetched));
+ KUNIT_EXPECT_EQ(test, nr_entries, (unsigned int)ARRAY_SIZE(entries));
+ KUNIT_EXPECT_MEMEQ(test, fetched, entries, sizeof(entries));
+}
+
+static void stackdepot_frame_raw_fallback(struct kunit *test)
+{
+ unsigned long frame = 0x1000UL;
+ bool compressed;
+ u32 payload;
+
+#ifdef CONFIG_ARM64
+ frame = (unsigned long)_text + (unsigned long)S32_MAX + 1UL;
+#endif
+
+ compressed = arch_stack_depot_frame_try_compress(frame, &payload);
+ KUNIT_EXPECT_FALSE(test, compressed);
+}
+
+#if defined(CONFIG_X86_64) && !defined(CONFIG_UML)
+static void stackdepot_frame_x86_64(struct kunit *test)
+{
+ unsigned long direct_map = 0xffff888000001000UL;
+ unsigned long frame = 0xffffffff81234567UL;
+ unsigned long out;
+ bool compressed;
+ u32 low;
+
+ compressed = arch_stack_depot_frame_try_compress(frame, &low);
+ KUNIT_EXPECT_TRUE(test, compressed);
+ KUNIT_EXPECT_EQ(test, low, (u32)0x81234567);
+ arch_stack_depot_frame_decompress(low, &out);
+ KUNIT_EXPECT_EQ(test, out, frame);
+
+ compressed = arch_stack_depot_frame_try_compress(direct_map, &low);
+ KUNIT_EXPECT_FALSE(test, compressed);
+}
+#endif /* CONFIG_X86_64 && !CONFIG_UML */
+
+#ifdef CONFIG_ARM64
+static void stackdepot_frame_arm64(struct kunit *test)
+{
+ long negative_offset = S32_MIN;
+ long positive_offset = S32_MAX;
+ long offset = 0x123456;
+ unsigned long frame = stackdepot_arm64_frame(offset);
+ unsigned long out;
+ bool compressed;
+ u32 payload;
+
+ compressed = arch_stack_depot_frame_try_compress(frame, &payload);
+ KUNIT_EXPECT_TRUE(test, compressed);
+ KUNIT_EXPECT_EQ(test, payload, (u32)(s32)offset);
+ arch_stack_depot_frame_decompress(payload, &out);
+ KUNIT_EXPECT_EQ(test, out, frame);
+
+ frame = stackdepot_arm64_frame(negative_offset);
+ compressed = arch_stack_depot_frame_try_compress(frame, &payload);
+ KUNIT_EXPECT_TRUE(test, compressed);
+ KUNIT_EXPECT_EQ(test, payload, (u32)(s32)negative_offset);
+ arch_stack_depot_frame_decompress(payload, &out);
+ KUNIT_EXPECT_EQ(test, out, frame);
+
+ frame = stackdepot_arm64_frame(positive_offset);
+ compressed = arch_stack_depot_frame_try_compress(frame, &payload);
+ KUNIT_EXPECT_TRUE(test, compressed);
+ KUNIT_EXPECT_EQ(test, payload, (u32)(s32)positive_offset);
+ arch_stack_depot_frame_decompress(payload, &out);
+ KUNIT_EXPECT_EQ(test, out, frame);
+}
+#endif /* CONFIG_ARM64 */
+
+static struct kunit_case stackdepot_test_cases[] = {
+ KUNIT_CASE(stackdepot_trie_max_path_roundtrip),
+ KUNIT_CASE(stackdepot_save_flags_public),
+ KUNIT_CASE(stackdepot_snprint_public),
+ KUNIT_CASE(stackdepot_countable_public),
+ KUNIT_CASE(stackdepot_fetch_into_roundtrip),
+ KUNIT_CASE(stackdepot_fetch_into_rejects_missing_or_short_stack),
+ KUNIT_CASE(stackdepot_trie_topology_roundtrip),
+ KUNIT_CASE(stackdepot_frame_storage_roundtrip),
+ KUNIT_CASE(stackdepot_frame_raw_fallback),
+#if defined(CONFIG_X86_64) && !defined(CONFIG_UML)
+ KUNIT_CASE(stackdepot_frame_x86_64),
+#endif
+#ifdef CONFIG_ARM64
+ KUNIT_CASE(stackdepot_frame_arm64),
+#endif
+ {}
+};
+
+static struct kunit_suite stackdepot_test_suite = {
+ .name = "stackdepot",
+ .test_cases = stackdepot_test_cases,
+};
+
+kunit_test_suite(stackdepot_test_suite);
+
+MODULE_DESCRIPTION("KUnit tests for stack depot");
+MODULE_AUTHOR("Caleb Kan <ckan@cloudflare.com>");
+MODULE_LICENSE("GPL");
diff --git a/mm/kmemleak.c b/mm/kmemleak.c
index 8fa409a4f9fb2..c42741a88bd42 100644
--- a/mm/kmemleak.c
+++ b/mm/kmemleak.c
@@ -378,10 +378,10 @@ static void __print_unreferenced(struct seq_file *seq,
bool hex_dump)
{
int i;
- unsigned long *entries;
+ unsigned long entries[MAX_TRACE];
unsigned int nr_entries;
- nr_entries = stack_depot_fetch(object->trace_handle, &entries);
+ nr_entries = stack_depot_fetch_into(object->trace_handle, entries, ARRAY_SIZE(entries));
warn_or_seq_printf(seq, "unreferenced object%s 0x%08lx (size %zu):\n",
__object_type_str(object),
object->pointer, object->size);
diff --git a/mm/kmsan/kmsan_test.c b/mm/kmsan/kmsan_test.c
index 31f47cc4dab40..7c04e4b21873d 100644
--- a/mm/kmsan/kmsan_test.c
+++ b/mm/kmsan/kmsan_test.c
@@ -669,7 +669,7 @@ static void test_long_origin_chain(struct kunit *test)
*/
static void test_stackdepot_roundtrip(struct kunit *test)
{
- unsigned long src_entries[16], *dst_entries;
+ unsigned long src_entries[16], dst_entries[16];
unsigned int src_nentries, dst_nentries;
EXPECTATION_NO_REPORT(expect);
depot_stack_handle_t handle;
@@ -680,7 +680,7 @@ static void test_stackdepot_roundtrip(struct kunit *test)
stack_trace_save(src_entries, ARRAY_SIZE(src_entries), 1);
handle = stack_depot_save(src_entries, src_nentries, GFP_KERNEL);
stack_depot_print(handle);
- dst_nentries = stack_depot_fetch(handle, &dst_entries);
+ dst_nentries = stack_depot_fetch_into(handle, dst_entries, ARRAY_SIZE(dst_entries));
KUNIT_EXPECT_TRUE(test, src_nentries == dst_nentries);
kmsan_check_memory((void *)dst_entries,
diff --git a/mm/kmsan/report.c b/mm/kmsan/report.c
index d6853ce089541..c20c24cffde55 100644
--- a/mm/kmsan/report.c
+++ b/mm/kmsan/report.c
@@ -85,7 +85,7 @@ static char *pretty_descr(char *descr)
void kmsan_print_origin(depot_stack_handle_t origin)
{
- unsigned long *entries = NULL, *chained_entries = NULL;
+ unsigned long entries[KMSAN_STACK_DEPTH];
unsigned int nr_entries, chained_nr_entries, skipnr;
void *pc1 = NULL, *pc2 = NULL;
depot_stack_handle_t head;
@@ -97,7 +97,8 @@ void kmsan_print_origin(depot_stack_handle_t origin)
return;
while (true) {
- nr_entries = stack_depot_fetch(origin, &entries);
+ nr_entries =
+ stack_depot_fetch_into(origin, entries, ARRAY_SIZE(entries));
depth = kmsan_depth_from_eb(stack_depot_get_extra_bits(origin));
magic = nr_entries ? entries[0] : 0;
if ((nr_entries == 4) && (magic == KMSAN_ALLOCA_MAGIC_ORIGIN)) {
@@ -123,14 +124,10 @@ void kmsan_print_origin(depot_stack_handle_t origin)
origin = entries[2];
pr_err("Uninit was stored to memory at:\n");
chained_nr_entries =
- stack_depot_fetch(head, &chained_entries);
- kmsan_internal_unpoison_memory(
- chained_entries,
- chained_nr_entries * sizeof(*chained_entries),
- /*checked*/ false);
- skipnr = get_stack_skipnr(chained_entries,
- chained_nr_entries);
- stack_trace_print(chained_entries + skipnr,
+ stack_depot_fetch_into(head, entries,
+ ARRAY_SIZE(entries));
+ skipnr = get_stack_skipnr(entries, chained_nr_entries);
+ stack_trace_print(entries + skipnr,
chained_nr_entries - skipnr, 0);
pr_err("\n");
continue;
diff --git a/mm/page_owner.c b/mm/page_owner.c
index fbbda7ba914ba..af37532729b02 100644
--- a/mm/page_owner.c
+++ b/mm/page_owner.c
@@ -119,7 +119,8 @@ static __always_inline depot_stack_handle_t create_dummy_stack(void)
unsigned int nr_entries;
nr_entries = stack_trace_save(entries, ARRAY_SIZE(entries), 0);
- return stack_depot_save(entries, nr_entries, GFP_KERNEL);
+ return stack_depot_save_flags(entries, nr_entries, GFP_KERNEL,
+ STACK_DEPOT_FLAG_CAN_ALLOC | STACK_DEPOT_FLAG_COUNTABLE);
}
static noinline void register_dummy_stack(void)
@@ -181,7 +182,8 @@ static noinline depot_stack_handle_t save_stack(gfp_t flags)
set_current_in_page_owner();
nr_entries = stack_trace_save(entries, ARRAY_SIZE(entries), 2);
- handle = stack_depot_save(entries, nr_entries, flags);
+ handle = stack_depot_save_flags(entries, nr_entries, flags,
+ STACK_DEPOT_FLAG_CAN_ALLOC | STACK_DEPOT_FLAG_COUNTABLE);
if (!handle)
handle = failure_handle;
unset_current_in_page_owner();
diff --git a/mm/slub.c b/mm/slub.c
index 422bc3e12c02c..138c3bc473c9c 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -8093,12 +8093,12 @@ void __kmem_obj_info(struct kmem_obj_info *kpp, void *object, struct slab *slab)
#ifdef CONFIG_STACKDEPOT
{
depot_stack_handle_t handle;
- unsigned long *entries;
+ unsigned long entries[TRACK_ADDRS_COUNT];
unsigned int nr_entries;
handle = READ_ONCE(trackp->handle);
if (handle) {
- nr_entries = stack_depot_fetch(handle, &entries);
+ nr_entries = stack_depot_fetch_into(handle, entries, ARRAY_SIZE(entries));
for (i = 0; i < KS_ADDRS_COUNT && i < nr_entries; i++)
kpp->kp_stack[i] = (void *)entries[i];
}
@@ -8106,7 +8106,7 @@ void __kmem_obj_info(struct kmem_obj_info *kpp, void *object, struct slab *slab)
trackp = get_track(s, objp, TRACK_FREE);
handle = READ_ONCE(trackp->handle);
if (handle) {
- nr_entries = stack_depot_fetch(handle, &entries);
+ nr_entries = stack_depot_fetch_into(handle, entries, ARRAY_SIZE(entries));
for (i = 0; i < KS_ADDRS_COUNT && i < nr_entries; i++)
kpp->kp_free_stack[i] = (void *)entries[i];
}
@@ -9815,12 +9815,14 @@ static int slab_debugfs_show(struct seq_file *seq, void *v)
#ifdef CONFIG_STACKDEPOT
{
depot_stack_handle_t handle;
- unsigned long *entries;
+ unsigned long entries[TRACK_ADDRS_COUNT];
unsigned int nr_entries, j;
handle = READ_ONCE(l->handle);
if (handle) {
- nr_entries = stack_depot_fetch(handle, &entries);
+ nr_entries =
+ stack_depot_fetch_into(handle, entries,
+ ARRAY_SIZE(entries));
seq_puts(seq, "\n");
for (j = 0; j < nr_entries; j++)
seq_printf(seq, " %pS\n", (void *)entries[j]);
diff --git a/scripts/gdb/linux/stackdepot.py b/scripts/gdb/linux/stackdepot.py
index 37313a5a51a0d..82aeb9f532c3d 100644
--- a/scripts/gdb/linux/stackdepot.py
+++ b/scripts/gdb/linux/stackdepot.py
@@ -37,6 +37,10 @@ def stack_depot_fetch(handle):
if handle == 0:
raise gdb.GdbError("handle is 0\n")
+ stack_max_pools = gdb.parse_and_eval('stack_max_pools')
+ if parts['pool_index_plus_1'] > stack_max_pools:
+ raise gdb.GdbError("trie-backed stack depot handles are not supported\n")
+
pool_index = parts['pool_index_plus_1'] - 1
if pool_index >= pools_num:
gdb.write("pool index %d out of bounds (%d) for stack id 0x%08x\n" % (parts['pool_index'], pools_num, handle))
|
| 17/3 | 2026/08/17 12:58 | llm | kmsan-evaluator |
1mModel:gemini-3.1-pro-preview Args: null Results: null Tokens: input: 36837 output: 350 thoughts: 11994 Thoughts: **KMSAN Evaluation of Stack Depot Trie Storage Patch** Okay, so I've been asked to assess whether this stack depot patch warrants dedicated KMSAN fuzzing. Let me break down my thinking here. First, I need to understand what this patch is doing: it's introducing a new trie-based storage backend for stack depot, along with a kernel parameter and modifications to existing functions like `stack_depot_fetch` and its variant `stack_depot_fetch_into`. This new backend is all about compressing stack frames to save memory by using architecture specific hooks for compression/decompression. It also reworks how some core users of `stack_depot_fetch`, like `kmemleak` and KMSAN itself, fetch stack traces, changing them to use a caller-provided buffer. It also includes unit tests. Now, the crucial question: does this patch introduce any new risks that KMSAN is uniquely positioned to catch? KMSAN is all about detecting uses of uninitialized memory. Given the complexity of the trie data structures – nodes, children containers, side tables, and pools – I immediately start thinking about allocations and initialization. The code uses `memblock_alloc`, `kvzalloc_flex`, `get_zeroed_page`, and `alloc_pages` – all of which *should* give us zero-initialized memory in most cases, but I need to be sure the logic around memory sub-allocation, particularly with those memory pools, is sound. I'm looking for spots where memory is reused but not fully re-initialized. I see that `trie_pool_alloc` hands out memory from a pool. The pools are zeroed initially during allocation. Furthermore, `trie_node_init` and `trie_children_init` are responsible for initializing the trie nodes. I carefully analyze the code, especially `trie_node_init`, and check for any potential padding bytes that might be left uninitialized. In the case of `trie_node`'s `run` member, which contains the `nr_entries`, it is copied from the stack, so the padding in `run` might be uninitialized. But, since it's an internal data structure, and the padding is never read, this is harmless. Now, the changes to `stack_depot_fetch_into` are interesting, especially the introduction of caller-provided buffers. If I understand correctly, callers allocate a buffer on the stack and call `stack_depot_fetch_into` to fill it. Here's where the explicit calls to `kmsan_unpoison_memory` become a key consideration. This unpoisoning *should* be safe because the function is designed to write exactly what it reports, up to `nr_entries`, but any bug in that logic that would cause the unpoisoning of memory that isn't really initialized is something that needs careful consideration. If there's a bug in the code where it fails to write to the whole array, but it still calls `kmsan_unpoison_memory`, KMSAN wouldn't catch the uninitialized read. A similar thing could happen with the `trie_fetch_into` function in the trie backend. The code explicitly unpoisons the entries based on the total. So, there is a risk that uninitialized memory is effectively hidden from KMSAN. I carefully check that the total written to buffer is correct. I've also got to examine the calls to `arch_stack_depot_frame_try_compress`, which can return whether or not the frame can be compressed. If compression fails, does it correctly handle the uninitialized payload? If the code assumes compression works but it fails, the `payload` variable could be uninitialized. But, if a `memcpy` with an uninitialized source is done, KMSAN *would* complain. And if it's read and used in an `arch_stack_depot_frame_decompress` (doing arithmetic), KMSAN would definitely catch it. KMSAN would be extremely useful if a control flow branch relied on an uninitialized value! To summarize, this patch does involve a lot of memory management, but the core issue is whether or not there are uninitialized reads or uses. I am not seeing anything other than harmless, unread, padding bytes. I've walked the code very carefully to try to find something. I am sure that the explicit calls to `kmsan_unpoison_memory` are correct. It's a key question. Given all of this, I don't see any significant, high-probability risks of *uniquely* KMSAN-detectable bugs. Any issues here would probably manifest as memory corruption or logic errors that KASAN and the KUnit tests would likely catch. KMSAN isn't the silver bullet here; it is important to test the code. Therefore |
| 18/3 | 2026/08/17 13:00 | tool | set-results |
0mArgs:{
"KMSANReasoning": "The patch series introduces a new trie-based storage backend for `stackdepot` to compress and store stack traces more efficiently. It involves custom memory sub-allocation (`trie_pool_alloc`) from page-sized pools. \n\nWhile the memory returned by `trie_pool_alloc` is reused without zeroing, the code carefully and fully initializes all relevant fields of the allocated structures (`trie_node_init`, `trie_children_init`, `trie_children_insert`) before they are published via RCU. Any padding bytes within the structures (e.g., in `struct stack_depot_trie_node`) are never read or exposed.\n\nThe patch also changes the fetch API to `stack_depot_fetch_into`, requiring callers to provide their own buffers. This function explicitly unpoisons the written memory using `kmsan_unpoison_memory`. This is safe and correct because the trie traversal logic guarantees writing exactly the calculated number of frames, even in the presence of concurrent reparenting or splitting.\n\nThere are no structures copied to user space, no parsing of untrusted input, and no paths where uninitialized memory could affect control flow. Any bugs in the trie logic or memory management would primarily manifest as logic errors (wrong stack traces), out-of-bounds accesses, or use-after-free bugs. These are adequately covered by standard KASAN, LOCKDEP, and the comprehensive KUnit tests included in the patch. Therefore, a dedicated KMSAN fuzzing session is not justified.",
"NeedsKMSAN": false
}Results: {
"KMSANReasoning": "The patch series introduces a new trie-based storage backend for `stackdepot` to compress and store stack traces more efficiently. It involves custom memory sub-allocation (`trie_pool_alloc`) from page-sized pools. \n\nWhile the memory returned by `trie_pool_alloc` is reused without zeroing, the code carefully and fully initializes all relevant fields of the allocated structures (`trie_node_init`, `trie_children_init`, `trie_children_insert`) before they are published via RCU. Any padding bytes within the structures (e.g., in `struct stack_depot_trie_node`) are never read or exposed.\n\nThe patch also changes the fetch API to `stack_depot_fetch_into`, requiring callers to provide their own buffers. This function explicitly unpoisons the written memory using `kmsan_unpoison_memory`. This is safe and correct because the trie traversal logic guarantees writing exactly the calculated number of frames, even in the presence of concurrent reparenting or splitting.\n\nThere are no structures copied to user space, no parsing of untrusted input, and no paths where uninitialized memory could affect control flow. Any bugs in the trie logic or memory management would primarily manifest as logic errors (wrong stack traces), out-of-bounds accesses, or use-after-free bugs. These are adequately covered by standard KASAN, LOCKDEP, and the comprehensive KUnit tests included in the patch. Therefore, a dedicated KMSAN fuzzing session is not justified.",
"NeedsKMSAN": false
} |
| Total Calls | Total Tokens | Avg Tokens | Total Duration (Seconds) | Avg Duration (Seconds) |
|---|
| Total Calls | Total Duration (Seconds) | Avg Duration (Seconds) |
|---|