| Seq | Timestamp | Type | Name | Duration |
|---|---|---|---|---|
| 0/0 | 2026/09/08 15:27 | flow | patch-triage |
0mArgs:null Results: {
"EnableConfigs": [
"STACKDEPOT",
"STACKDEPOT_KUNIT_TEST",
"PAGE_OWNER",
"DEBUG_KMEMLEAK",
"SLUB_DEBUG"
],
"FocusSymbols": [
"stack_depot_fetch_into",
"stack_depot_trie_save",
"stack_depot_trie_save_constrained",
"stack_depot_trie_insert",
"stack_depot_trie_lookup",
"trie_insert_path",
"trie_split_child",
"trie_promote_child",
"trie_fetch_handle_into",
"trie_fetch_handle_range",
"trie_pool_alloc",
"trie_pool_reserve_slots",
"trie_drain_pending_children",
"trie_reparent_children",
"trie_path_alloc",
"trie_side_table_publish",
"trie_side_table_lookup",
"trie_side_table_prepare_stack_slot",
"trie_print",
"trie_snprint",
"kmsan_print_origin",
"__drm_stack_depot_print",
"slab_debugfs_show"
],
"KMSANReasoning": "The patch series implements a trie storage backend and address compression for the stack depot subsystem (lib/stackdepot.c), updates stack depot consumers (such as kmemleak, SLUB debug, DRM, and KMSAN's origin printer) to use the new `stack_depot_fetch_into()` / `stack_depot_snprint()` APIs, and adds KUnit tests.\n\n1. KASAN / Lockdep / Standard Debuggers Applicability:\n- The changes heavily involve internal kernel data structure management, including RCU-protected pointer updates, bitmap slot reservations within memory pools, spinlocks, and buffer copies for stack frames.\n- Potential defects such as out-of-bounds indexing in child arrays/bitmaps, memory corruption, use-after-free during node splitting/reparenting, and concurrency/locking deadlocks are fully covered by KASAN, KCSAN, and LOCKDEP.\n\n2. KMSAN Applicability:\n- No kernel structures are copied or exposed to userspace (no copy_to_user, put_user, netlink, ioctl, etc.).\n- All internal allocations (directory pages, side tables, node arrays, pools) are allocated using zeroing allocators (`get_zeroed_page()`, `kvzalloc()`, `memblock_alloc()`) or are explicitly populated before being read or linked into the trie.\n- Callers of `stack_depot_fetch_into()` only access entries up to the returned `nr_entries` count, and stack depot explicitly invokes `kmsan_unpoison_memory()` when populating frame entries.\n- No uninitialized variables or uninitialized branching conditions are introduced.\n\nBecause the changes present memory safety and synchronization concerns rather than uninitialized memory access or info-leak risks, a dedicated KMSAN fuzzing session is not justified.",
"NeedsKMSAN": false,
"Reasoning": "The patch series introduces a new trie-based storage backend for stack depot with arch-specific frame compression (x86-64 and arm64), RCU-based child management, slot allocation/reservation within depot pools, side-table mappings, and a new stack_depot_fetch_into() API. It also updates several callers across the kernel (SLUB, page_owner, kmemleak, KMSAN, DRM) and adds runtime checks and assertions. These changes modify core memory management infrastructure and are reachable during kernel execution in standard virtualized environments.",
"WorthFuzzing": true
} |
| 1/1 | 2026/09/08 15:27 | action | read-patch-diff |
0mArgs:null Results: {
"PatchDiff": "commit efdf021b1a377c26b41b0ed29bb41a4fbb95bac4\nAuthor: syz-cluster \u003ctriage@syzkaller.com\u003e\nDate: Tue Sep 8 15:27:13 2026 +0000\n\n syz-cluster: applied patch under review\n\ndiff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt\nindex 68647ff4bdd24..b02bcbaef5dcb 100644\n--- a/Documentation/admin-guide/kernel-parameters.txt\n+++ b/Documentation/admin-guide/kernel-parameters.txt\n@@ -7449,6 +7449,13 @@ Kernel parameters\n \t\t\tstack traces. Pools are allocated on-demand up to this\n \t\t\tlimit. Default value is 8191 pools.\n \n+\tstackdepot.trie_enabled= [KNL]\n+\t\t\tFormat: \u003cbool\u003e\n+\t\t\tEnable trie storage for persistent, non-refcounted\n+\t\t\tstack depot records at boot. Disabled by default.\n+\t\t\tstack_depot_max_pools must leave unused pool-index\n+\t\t\tvalues for trie handles.\n+\n \tstacktrace\t[FTRACE]\n \t\t\tEnable the stack tracer on boot up.\n \ndiff --git a/arch/arm64/include/asm/stackdepot.h b/arch/arm64/include/asm/stackdepot.h\nnew file mode 100644\nindex 0000000000000..df8959d593366\n--- /dev/null\n+++ b/arch/arm64/include/asm/stackdepot.h\n@@ -0,0 +1,42 @@\n+/* SPDX-License-Identifier: GPL-2.0 */\n+#ifndef __ASM_STACKDEPOT_H\n+#define __ASM_STACKDEPOT_H\n+\n+#include \u003clinux/types.h\u003e\n+#include \u003casm/sections.h\u003e\n+\n+/*\n+ * Modules are allocated inside a 2 GB relocation window containing the\n+ * kernel image. Store a signed 32-bit offset from _text so compression is\n+ * independent of 4 GB high-bit boundaries crossed by that window.\n+ */\n+static inline unsigned long arch_stack_depot_frame_from_payload(u32 payload)\n+{\n+\tlong offset;\n+\n+\toffset = (s32)payload;\n+\tif (offset \u003c 0)\n+\t\treturn (unsigned long)_text - (unsigned long)(-offset);\n+\treturn (unsigned long)_text + (unsigned long)offset;\n+}\n+\n+static inline bool\n+arch_stack_depot_frame_try_compress(unsigned long frame, u32 *payload)\n+{\n+\tu32 candidate;\n+\n+\tcandidate = (u32)(frame - (unsigned long)_text);\n+\tif (arch_stack_depot_frame_from_payload(candidate) != frame)\n+\t\treturn false;\n+\n+\t*payload = candidate;\n+\treturn true;\n+}\n+\n+static inline void\n+arch_stack_depot_frame_decompress(u32 payload, unsigned long *frame)\n+{\n+\t*frame = arch_stack_depot_frame_from_payload(payload);\n+}\n+\n+#endif /* __ASM_STACKDEPOT_H */\ndiff --git a/arch/um/include/asm/Kbuild b/arch/um/include/asm/Kbuild\nindex 8fdc0bd9ab6fb..14778d2457d79 100644\n--- a/arch/um/include/asm/Kbuild\n+++ b/arch/um/include/asm/Kbuild\n@@ -21,6 +21,7 @@ generic-y += preempt.h\n generic-y += ring_buffer.h\n generic-y += runtime-const.h\n generic-y += softirq_stack.h\n+generic-y += stackdepot.h\n generic-y += switch_to.h\n generic-y += topology.h\n generic-y += trace_clock.h\ndiff --git a/arch/x86/include/asm/stackdepot.h b/arch/x86/include/asm/stackdepot.h\nnew file mode 100644\nindex 0000000000000..9a8d04fa8c1c8\n--- /dev/null\n+++ b/arch/x86/include/asm/stackdepot.h\n@@ -0,0 +1,37 @@\n+/* SPDX-License-Identifier: GPL-2.0 */\n+#ifndef _ASM_X86_STACKDEPOT_H\n+#define _ASM_X86_STACKDEPOT_H\n+\n+#include \u003clinux/types.h\u003e\n+\n+#ifdef CONFIG_X86_64\n+/*\n+ * Compress canonical kernel text/module addresses whose upper 32 bits are all\n+ * ones. Other kernel virtual addresses stay raw, so decompression reconstructs\n+ * the original frame by restoring this prefix.\n+ */\n+#define STACK_DEPOT_X86_64_FRAME_PREFIX\t0xffffffff00000000UL\n+#define STACK_DEPOT_X86_64_FRAME_LOW_MASK\t0x00000000ffffffffUL\n+\n+static inline bool\n+arch_stack_depot_frame_try_compress(unsigned long frame, u32 *low)\n+{\n+\tif ((frame \u0026 ~STACK_DEPOT_X86_64_FRAME_LOW_MASK) !=\n+\t STACK_DEPOT_X86_64_FRAME_PREFIX)\n+\t\treturn false;\n+\n+\t*low = (u32)frame;\n+\treturn true;\n+}\n+\n+static inline void\n+arch_stack_depot_frame_decompress(u32 low, unsigned long *frame)\n+{\n+\t*frame = STACK_DEPOT_X86_64_FRAME_PREFIX | low;\n+}\n+\n+#else\n+#include \u003casm-generic/stackdepot.h\u003e\n+#endif /* CONFIG_X86_64 */\n+\n+#endif /* _ASM_X86_STACKDEPOT_H */\ndiff --git a/drivers/gpu/drm/drm_modeset_lock.c b/drivers/gpu/drm/drm_modeset_lock.c\nindex e14814c30d8c0..f48b02379b711 100644\n--- a/drivers/gpu/drm/drm_modeset_lock.c\n+++ b/drivers/gpu/drm/drm_modeset_lock.c\n@@ -94,16 +94,13 @@ static noinline depot_stack_handle_t __drm_stack_depot_save(void)\n static void __drm_stack_depot_print(depot_stack_handle_t stack_depot)\n {\n \tstruct drm_printer p = drm_dbg_printer(NULL, DRM_UT_KMS, \"drm_modeset_lock\");\n-\tunsigned long *entries;\n-\tunsigned int nr_entries;\n \tchar *buf;\n \n \tbuf = kmalloc(PAGE_SIZE, GFP_NOWAIT | __GFP_NOWARN);\n \tif (!buf)\n \t\treturn;\n \n-\tnr_entries = stack_depot_fetch(stack_depot, \u0026entries);\n-\tstack_trace_snprint(buf, PAGE_SIZE, entries, nr_entries, 2);\n+\tstack_depot_snprint(stack_depot, buf, PAGE_SIZE, 2);\n \n \tdrm_printf(\u0026p, \"attempting to lock a contended lock without backoff:\\n%s\", buf);\n \ndiff --git a/include/asm-generic/Kbuild b/include/asm-generic/Kbuild\nindex 2bc00c67dc54b..d8402a6afc703 100644\n--- a/include/asm-generic/Kbuild\n+++ b/include/asm-generic/Kbuild\n@@ -55,6 +55,7 @@ mandatory-y += serial.h\n mandatory-y += shmparam.h\n mandatory-y += simd.h\n mandatory-y += softirq_stack.h\n+mandatory-y += stackdepot.h\n mandatory-y += switch_to.h\n mandatory-y += timex.h\n mandatory-y += tlbflush.h\ndiff --git a/include/asm-generic/stackdepot.h b/include/asm-generic/stackdepot.h\nnew file mode 100644\nindex 0000000000000..846975767bdd4\n--- /dev/null\n+++ b/include/asm-generic/stackdepot.h\n@@ -0,0 +1,19 @@\n+/* SPDX-License-Identifier: GPL-2.0 */\n+#ifndef __ASM_GENERIC_STACKDEPOT_H\n+#define __ASM_GENERIC_STACKDEPOT_H\n+\n+#include \u003clinux/types.h\u003e\n+\n+static inline bool\n+arch_stack_depot_frame_try_compress(unsigned long frame, u32 *low)\n+{\n+\treturn false;\n+}\n+\n+static inline void\n+arch_stack_depot_frame_decompress(u32 low, unsigned long *frame)\n+{\n+\t/* Generic code never compresses frames, so this hook is unreachable. */\n+}\n+\n+#endif /* __ASM_GENERIC_STACKDEPOT_H */\ndiff --git a/include/linux/stackdepot.h b/include/linux/stackdepot.h\nindex 2cc21ffcdaf9e..3126d17b9265b 100644\n--- a/include/linux/stackdepot.h\n+++ b/include/linux/stackdepot.h\n@@ -53,7 +53,8 @@ union handle_parts {\n struct stack_record {\n \tstruct list_head hash_list;\t/* Links in the hash table */\n \tu32 hash;\t\t\t/* Hash in hash table */\n-\tu32 size;\t\t\t/* Number of stored frames */\n+\tu16 size;\t\t\t/* Number of stored frames */\n+\tu16 flags;\n \tunion handle_parts handle;\t/* Constant after initialization */\n \trefcount_t count;\n \tunion {\n@@ -84,8 +85,9 @@ typedef u32 depot_flags_t;\n */\n #define STACK_DEPOT_FLAG_CAN_ALLOC\t((depot_flags_t)0x0001)\n #define STACK_DEPOT_FLAG_GET\t\t((depot_flags_t)0x0002)\n+#define STACK_DEPOT_FLAG_COUNTABLE\t((depot_flags_t)0x0004)\n \n-#define STACK_DEPOT_FLAGS_NUM\t2\n+#define STACK_DEPOT_FLAGS_NUM\t3\n #define STACK_DEPOT_FLAGS_MASK\t((depot_flags_t)((1 \u003c\u003c STACK_DEPOT_FLAGS_NUM) - 1))\n \n /*\n@@ -144,6 +146,17 @@ static inline int stack_depot_early_init(void)\t{ return 0; }\n * Users of this flag must also call stack_depot_put() when keeping the stack\n * trace is no longer required to avoid overflowing the refcount.\n *\n+ * If STACK_DEPOT_FLAG_COUNTABLE is set in @depot_flags, stack depot stores the\n+ * stack in hash-backed storage for callers that need direct stack_record count\n+ * access. This flag does not imply %STACK_DEPOT_FLAG_CAN_ALLOC and is mutually\n+ * exclusive with %STACK_DEPOT_FLAG_GET.\n+ *\n+ * When trie storage is enabled, persistent non-refcounted saves use trie\n+ * storage. Constrained callers first look up an existing stack, then make one\n+ * best-effort insertion attempt without allocating. NMI callers stop after the\n+ * lookup. Other callers that cannot spin use trylocks and fail if a required\n+ * lock is unavailable. Trie failures do not fall back to hash storage.\n+ *\n * If the provided stack trace comes from the interrupt context, only the part\n * up to the interrupt entry is saved.\n *\n@@ -152,7 +165,7 @@ static inline int stack_depot_early_init(void)\t{ return 0; }\n * this is the case for contexts where neither %GFP_ATOMIC nor\n * %GFP_NOWAIT can be used (NMI, raw_spin_lock).\n *\n- * Return: Handle of the stack struct stored in depot, 0 on failure\n+ * Return: Handle of the stack trace stored in depot, 0 on failure\n */\n depot_stack_handle_t stack_depot_save_flags(unsigned long *entries,\n \t\t\t\t\t unsigned int nr_entries,\n@@ -169,6 +182,10 @@ depot_stack_handle_t stack_depot_save_flags(unsigned long *entries,\n * Does not increment the refcount on the saved stack trace; see\n * stack_depot_save_flags() for more details.\n *\n+ * When trie storage is enabled, this can return trie-backed handles. Use\n+ * stack_depot_fetch_into(), stack_depot_print(), or stack_depot_snprint() for\n+ * backend-independent access to the stack contents.\n+ *\n * Context: Contexts where allocations via alloc_pages() are allowed;\n * see stack_depot_save_flags() for more details.\n *\n@@ -178,11 +195,12 @@ depot_stack_handle_t stack_depot_save(unsigned long *entries,\n \t\t\t\t unsigned int nr_entries, gfp_t alloc_flags);\n \n /**\n- * __stack_depot_get_stack_record - Get a pointer to a stack_record struct\n+ * __stack_depot_get_stack_record - Get a hash-backed stack record\n *\n * @handle: Stack depot handle\n *\n- * This function is only for internal purposes.\n+ * This function is only for internal purposes. @handle must have been saved\n+ * with %STACK_DEPOT_FLAG_COUNTABLE.\n *\n * Return: Returns a pointer to a stack_record struct\n */\n@@ -191,14 +209,55 @@ struct stack_record *__stack_depot_get_stack_record(depot_stack_handle_t handle)\n /**\n * stack_depot_fetch - Fetch a stack trace from stack depot\n *\n- * @handle:\tStack depot handle returned from stack_depot_save()\n+ * @handle:\tHash-backed stack depot handle\n * @entries:\tPointer to store the address of the stack trace\n *\n+ * This helper returns a pointer to stackdepot-owned contiguous storage for\n+ * legacy hash-backed handles. Callers that need backend-independent access to\n+ * stack contents should use stack_depot_fetch_into(), stack_depot_print(), or\n+ * stack_depot_snprint(). Passing a trie-backed handle is invalid and may WARN.\n+ *\n * Return: Number of frames for the fetched stack\n */\n unsigned int stack_depot_fetch(depot_stack_handle_t handle,\n \t\t\t unsigned long **entries);\n \n+/**\n+ * stack_depot_fetch_into - Fetch a stack trace into caller-owned storage\n+ *\n+ * @handle:\tStack depot handle\n+ * @entries:\tCaller-owned buffer to copy the stack trace into\n+ * @max_entries:\tNumber of frames that fit in @entries\n+ *\n+ * Copies the stored frames into caller-owned @entries. If fewer frames are\n+ * stored than @max_entries, only the stored frames are written and their count\n+ * is returned. If more frames are stored than @max_entries, the copy is skipped\n+ * entirely and 0 is returned.\n+ *\n+ * Passing a NULL @entries buffer or zero @max_entries for a valid @handle is\n+ * invalid. Callers must provide storage for @max_entries frames.\n+ *\n+ * Callers should size @entries to match the save-side stack depth cap (for\n+ * example, %CONFIG_STACKDEPOT_MAX_FRAMES or the local stack_trace_save() limit)\n+ * when losing diagnostics on an undersized buffer would be surprising.\n+ *\n+ * A non-zero invalid @handle, including a post-put handle, may WARN. Its return\n+ * value and copied contents are undefined because the record may have been\n+ * reused for another stack.\n+ *\n+ * Callers must ensure @handle remains valid for the duration of this call.\n+ * Persistent handles saved without %STACK_DEPOT_FLAG_GET require no extra\n+ * reference; handles saved with %STACK_DEPOT_FLAG_GET require a held reference.\n+ * Callers must not call stack_depot_put() on persistent handles.\n+ * Racing this helper with stack_depot_put() on the same handle is invalid.\n+ *\n+ * Return: Number of frames copied, 0 if @handle is 0, stack depot is disabled,\n+ * or @max_entries is less than the number of stored frames.\n+ */\n+unsigned int stack_depot_fetch_into(depot_stack_handle_t handle,\n+\t\t\t\t unsigned long *entries,\n+\t\t\t\t unsigned int max_entries);\n+\n /**\n * stack_depot_print - Print a stack trace from stack depot\n *\n@@ -224,10 +283,14 @@ int stack_depot_snprint(depot_stack_handle_t handle, char *buf, size_t size,\n *\n * @handle:\tStack depot handle returned from stack_depot_save()\n *\n- * The stack trace is evicted from stack depot once all references to it have\n- * been dropped (once the number of stack_depot_evict() calls matches the\n- * number of stack_depot_save_flags() calls with STACK_DEPOT_FLAG_GET set for\n- * this stack trace).\n+ * Drop a reference acquired by stack_depot_save_flags() with\n+ * %STACK_DEPOT_FLAG_GET. Calling this for a handle saved without\n+ * %STACK_DEPOT_FLAG_GET is invalid; persistent handles, including trie-backed\n+ * handles, are owned by stack depot for the lifetime of the system.\n+ *\n+ * The stack trace is evicted once the number of stack_depot_put() calls matches\n+ * the number of successful stack_depot_save_flags() calls with\n+ * %STACK_DEPOT_FLAG_GET for this stack trace.\n */\n void stack_depot_put(depot_stack_handle_t handle);\n \ndiff --git a/lib/Kconfig.debug b/lib/Kconfig.debug\nindex 134b15a44625e..3a78c67b6b364 100644\n--- a/lib/Kconfig.debug\n+++ b/lib/Kconfig.debug\n@@ -2785,6 +2785,23 @@ config RESOURCE_KUNIT_TEST\n \n \t If unsure, say N.\n \n+config STACKDEPOT_KUNIT_TEST\n+\tbool \"KUnit test for stack depot\" if !KUNIT_ALL_TESTS\n+\tdepends on KUNIT=y \u0026\u0026 STACKDEPOT\n+\tdepends on STACKDEPOT_MAX_FRAMES \u003e= 3\n+\tdefault KUNIT_ALL_TESTS\n+\thelp\n+\t Enable this option to test stack depot API behavior at boot.\n+\t This test is built in because it exercises internal, non-exported\n+\t stack depot helpers, so KUNIT must also be built in.\n+\n+\t KUnit tests run during boot and output the results to the debug log\n+\t in TAP format (https://testanything.org/). Only useful for kernel\n+\t developers running the KUnit test harness, and not intended for\n+\t inclusion into a production build.\n+\n+\t If unsure, say N.\n+\n config SYSCTL_KUNIT_TEST\n \ttristate \"KUnit test for sysctl\" if !KUNIT_ALL_TESTS\n \tdepends on KUNIT\ndiff --git a/lib/stackdepot.c b/lib/stackdepot.c\nindex dd2717ff94bff..33e475d941314 100644\n--- a/lib/stackdepot.c\n+++ b/lib/stackdepot.c\n@@ -2,9 +2,11 @@\n /*\n * Stack depot - a stack trace storage that avoids duplication.\n *\n- * Internally, stack depot maintains a hash table of unique stacktraces. The\n- * stack traces themselves are stored contiguously one after another in a set\n- * of separate page allocations.\n+ * Internally, stack depot has two storage backends. Refcounted entries and\n+ * callers that request STACK_DEPOT_FLAG_COUNTABLE use the legacy hash table with\n+ * contiguous stack records in stack pools. Persistent non-refcounted entries\n+ * can use trie storage when enabled; trie nodes share common frame prefixes and\n+ * are published through RCU children containers.\n *\n * Author: Alexander Potapenko \u003cglider@google.com\u003e\n * Copyright (C) 2016 Google, Inc.\n@@ -14,13 +16,19 @@\n \n #define pr_fmt(fmt) \"stackdepot: \" fmt\n \n+#include \u003clinux/bitmap.h\u003e\n+#include \u003clinux/build_bug.h\u003e\n #include \u003clinux/debugfs.h\u003e\n+#include \u003clinux/errno.h\u003e\n #include \u003clinux/gfp.h\u003e\n #include \u003clinux/jhash.h\u003e\n+#include \u003clinux/jump_label.h\u003e\n #include \u003clinux/kernel.h\u003e\n+#include \u003clinux/log2.h\u003e\n #include \u003clinux/kmsan.h\u003e\n #include \u003clinux/list.h\u003e\n #include \u003clinux/mm.h\u003e\n+#include \u003clinux/moduleparam.h\u003e\n #include \u003clinux/mutex.h\u003e\n #include \u003clinux/poison.h\u003e\n #include \u003clinux/printk.h\u003e\n@@ -36,9 +44,12 @@\n #include \u003clinux/memblock.h\u003e\n #include \u003clinux/kasan-enabled.h\u003e\n \n+#include \u003casm/stackdepot.h\u003e\n+\n /*\n * The pool_index is offset by 1 so the first record does not have a 0 handle.\n */\n+/* Parsed before mm_core_init(); trie handle decoding assumes this is then fixed. */\n static unsigned int stack_max_pools __read_mostly =\n \tMIN((1LL \u003c\u003c DEPOT_POOL_INDEX_BITS) - 1, 8192);\n \n@@ -54,6 +65,9 @@ static bool __stack_depot_early_init_passed __initdata;\n /* Initial seed for jhash2. */\n #define STACK_HASH_SEED 0x9747b28c\n \n+/* Bound 64-bit print scratch to 128 bytes while amortizing trie walks. */\n+#define STACK_DEPOT_PRINT_CHUNK_FRAMES 16\n+\n /* Hash table of stored stack records. */\n static struct list_head *stack_table;\n /* Fixed order of the number of table buckets. Used when KASAN is enabled. */\n@@ -63,18 +77,18 @@ static unsigned int stack_hash_mask;\n \n /* The lock must be held when performing pool or freelist modifications. */\n static DEFINE_RAW_SPINLOCK(pool_lock);\n-/* Array of memory regions that store stack records. */\n+/* Array of memory regions used by both stack depot backends. */\n static void **stack_pools __pt_guarded_by(\u0026pool_lock);\n /* Newly allocated pool that is not yet added to stack_pools. */\n static void *new_pool;\n /* Number of pools in stack_pools. */\n static int pools_num;\n-/* Offset to the unused space in the currently used pool. */\n+/* Offset to unused hash storage in the current pool. */\n static size_t pool_offset __guarded_by(\u0026pool_lock) = DEPOT_POOL_SIZE;\n /* Freelist of stack records within stack_pools. */\n static __guarded_by(\u0026pool_lock) LIST_HEAD(free_stacks);\n \n-/* Statistics counters for debugfs. */\n+/* Hash-backend statistics counters for debugfs. */\n enum depot_counter_id {\n \tDEPOT_COUNTER_REFD_ALLOCS,\n \tDEPOT_COUNTER_REFD_FREES,\n@@ -90,11 +104,695 @@ static const char *const counter_names[] = {\n \t[DEPOT_COUNTER_REFD_FREES]\t= \"refcounted_frees\",\n \t[DEPOT_COUNTER_REFD_INUSE]\t= \"refcounted_in_use\",\n \t[DEPOT_COUNTER_FREELIST_SIZE]\t= \"freelist_size\",\n-\t[DEPOT_COUNTER_PERSIST_COUNT]\t= \"persistent_count\",\n-\t[DEPOT_COUNTER_PERSIST_BYTES]\t= \"persistent_bytes\",\n+\t[DEPOT_COUNTER_PERSIST_COUNT]\t= \"hash_persistent_count\",\n+\t[DEPOT_COUNTER_PERSIST_BYTES]\t= \"hash_persistent_bytes\",\n };\n static_assert(ARRAY_SIZE(counter_names) == DEPOT_COUNTER_COUNT);\n \n+enum stack_depot_frame_mode {\n+\tSTACK_DEPOT_FRAME_RAW,\n+\tSTACK_DEPOT_FRAME_COMPRESSED,\n+};\n+\n+/*\n+ * A trie node stores one run of frames that all use the same payload format.\n+ * Architectures may compress some frames to 32-bit payloads; mixed raw and\n+ * compressed input is split across multiple trie nodes so each node has one\n+ * decoding mode.\n+ */\n+struct stack_depot_frame_run {\n+\tu16 nr_entries;\n+\tu8 mode;\n+};\n+\n+static_assert(CONFIG_STACKDEPOT_MAX_FRAMES \u003c= U16_MAX);\n+\n+struct stack_depot_trie_children;\n+\n+struct stack_depot_trie_node {\n+\t/* Parent links let fetch rebuild a full stack from a node to the root. */\n+\tconst struct stack_depot_trie_node __rcu *parent;\n+\t/* Children are RCU-published containers. */\n+\tconst struct stack_depot_trie_children __rcu *children;\n+\t/* Non-zero when a stored stack ends at this node. */\n+\tu32 stack_id;\n+\tstruct stack_depot_frame_run run;\n+\tunsigned char data[];\n+};\n+\n+/*\n+ * Child nodes are sorted by first frame and searched by insertion position.\n+ * Existing child pointers are immutable. Writers may publish into unused tail\n+ * capacity; other updates publish a replacement container.\n+ */\n+struct stack_depot_trie_children {\n+\tunsigned int nr_children;\n+\tunsigned int capacity;\n+\tconst struct stack_depot_trie_node __rcu *nodes[];\n+};\n+\n+/* Retired children carry an optional node through their RCU grace period. */\n+struct stack_depot_trie_retired_children {\n+\tstruct list_head list;\n+\tunsigned long rcu_state;\n+\tconst struct stack_depot_trie_node *pending_node;\n+\tunsigned char data[];\n+};\n+\n+static_assert(IS_ALIGNED(offsetof(struct stack_depot_trie_retired_children, data),\n+\t\t\t 1UL \u003c\u003c DEPOT_STACK_ALIGN));\n+\n+#define STACK_DEPOT_TRIE_SLOT_SIZE BIT(DEPOT_STACK_ALIGN)\n+#define STACK_DEPOT_TRIE_POOL_SLOTS \\\n+\t(DEPOT_POOL_SIZE / STACK_DEPOT_TRIE_SLOT_SIZE)\n+\n+static_assert(STACK_DEPOT_TRIE_POOL_SLOTS - 1 \u003c= U16_MAX);\n+\n+struct stack_depot_trie_pool {\n+\tstruct list_head list;\n+\tunsigned int free_slots;\n+\t/* Conservative upper bound on the largest free run. */\n+\tu16 free_run_upper_bound;\n+\t/* First physical slot considered by the next reservation. */\n+\tu16 next_slot;\n+\tDECLARE_BITMAP(used, STACK_DEPOT_TRIE_POOL_SLOTS);\n+};\n+\n+#define STACK_DEPOT_TRIE_POOL_FIRST_SLOT \\\n+\tDIV_ROUND_UP(sizeof(struct stack_depot_trie_pool), \\\n+\t\t STACK_DEPOT_TRIE_SLOT_SIZE)\n+#define STACK_DEPOT_TRIE_POOL_USABLE_SIZE \\\n+\t((STACK_DEPOT_TRIE_POOL_SLOTS - STACK_DEPOT_TRIE_POOL_FIRST_SLOT) * \\\n+\t STACK_DEPOT_TRIE_SLOT_SIZE)\n+\n+static_assert(STACK_DEPOT_TRIE_POOL_FIRST_SLOT \u003c STACK_DEPOT_TRIE_POOL_SLOTS);\n+\n+static DEFINE_STATIC_KEY_FALSE(stack_depot_trie_enabled);\n+static const struct stack_depot_trie_children __rcu *stack_depot_trie_root;\n+static DEFINE_RAW_SPINLOCK(stack_depot_trie_writer_lock);\n+static bool stack_depot_trie_requested;\n+\n+module_param_named(trie_enabled, stack_depot_trie_requested, bool, 0);\n+MODULE_PARM_DESC(trie_enabled, \"Enable stack depot trie storage at boot\");\n+\n+#define DEPOT_POOL_INDEX_MASK ((1U \u003c\u003c DEPOT_POOL_INDEX_BITS) - 1)\n+#define DEPOT_OFFSET_MASK ((1U \u003c\u003c DEPOT_OFFSET_BITS) - 1)\n+\n+/* Retired fixed-size slots remain reserved until their RCU grace period ends. */\n+static LIST_HEAD(stack_depot_trie_pools);\n+static LIST_HEAD(pending_trie_children);\n+\n+/*\n+ * stack_max_pools is the split point between hash and trie handle encodings.\n+ * A handle with pool_index_plus_1 in 1..stack_max_pools names a hash-backed\n+ * stack pool. Larger pool-index values cannot refer to hash pools, so trie\n+ * storage uses that handle space to encode a dense stack ID. The side table\n+ * maps each stack ID to its trie node.\n+ */\n+static inline u32 trie_max_stack_id(void)\n+{\n+\treturn (DEPOT_POOL_INDEX_MASK - stack_max_pools) \u003c\u003c\n+\t\tDEPOT_OFFSET_BITS;\n+}\n+\n+static depot_stack_handle_t trie_handle(u32 stack_id)\n+{\n+\tunion handle_parts parts = {};\n+\tu64 pool_index_plus_1;\n+\tu32 pool_delta;\n+\tu32 index;\n+\n+\tindex = stack_id - 1;\n+\tpool_delta = index \u003e\u003e DEPOT_OFFSET_BITS;\n+\tpool_index_plus_1 = (u64)stack_max_pools + 1 + pool_delta;\n+\n+\tparts.pool_index_plus_1 = pool_index_plus_1;\n+\tparts.offset = index \u0026 DEPOT_OFFSET_MASK;\n+\treturn parts.handle;\n+}\n+\n+static inline bool stack_depot_handle_is_trie(depot_stack_handle_t handle)\n+{\n+\tunion handle_parts parts = { .handle = handle };\n+\n+\treturn parts.pool_index_plus_1 \u003e stack_max_pools;\n+}\n+\n+static u32 trie_stack_id(depot_stack_handle_t handle)\n+{\n+\tunion handle_parts parts = { .handle = handle };\n+\tu32 pool_delta;\n+\n+\tpool_delta = parts.pool_index_plus_1 - stack_max_pools - 1;\n+\treturn (pool_delta \u003c\u003c DEPOT_OFFSET_BITS) + parts.offset + 1;\n+}\n+\n+/*\n+ * Trie handles encode a dense stack ID. The side table maps that ID to a node\n+ * pointer for lockless fetch and print paths, which can run from diagnostic\n+ * contexts where taking a lock would be unsafe. Initialization installs the\n+ * root; early initialization also installs the first directory and chunk.\n+ * Additional directories and chunks are published lazily as stack IDs grow.\n+ */\n+#define STACK_DEPOT_TRIE_SIDE_TABLE_CHUNK_SIZE \\\n+\t(PAGE_SIZE / sizeof(struct stack_depot_trie_node *))\n+#define STACK_DEPOT_TRIE_SIDE_TABLE_DIR_SIZE \\\n+\t(PAGE_SIZE / sizeof(struct stack_depot_trie_node **))\n+\n+struct stack_depot_trie_side_dir {\n+\t/* Both the chunk pointer and each node pointer in it are RCU-published. */\n+\tconst struct stack_depot_trie_node __rcu * __rcu *\n+\t\tchunks[STACK_DEPOT_TRIE_SIDE_TABLE_DIR_SIZE];\n+};\n+\n+struct stack_depot_trie_side_root {\n+\tunsigned int dir_capacity;\n+\tstruct stack_depot_trie_side_dir __rcu *dirs[];\n+};\n+\n+struct stack_depot_trie_side_prealloc {\n+\t/* Preallocated side-table directory page for sparse growth. */\n+\tstruct stack_depot_trie_side_dir *dir;\n+\t/* Preallocated side-table pointer chunk for sparse growth. */\n+\tconst struct stack_depot_trie_node __rcu **chunk;\n+};\n+\n+static struct stack_depot_trie_side_root *trie_side_table_root;\n+static DEFINE_RAW_SPINLOCK(trie_side_table_cache_lock);\n+/* Zeroed unpublished pages; get/put transfer ownership under the cache lock. */\n+static struct stack_depot_trie_side_prealloc trie_side_table_cache;\n+static u32 trie_side_table_last_stack_id;\n+\n+/* Lock order: writer_lock -\u003e pool_lock -\u003e side-table cache lock. */\n+\n+static inline size_t stack_depot_frame_run_entry_bytes(enum stack_depot_frame_mode mode)\n+{\n+\tif (mode == STACK_DEPOT_FRAME_COMPRESSED)\n+\t\treturn sizeof(u32);\n+\treturn sizeof(unsigned long);\n+}\n+\n+static inline size_t stack_depot_frame_run_bytes(const struct stack_depot_frame_run *run)\n+{\n+\treturn run-\u003enr_entries * stack_depot_frame_run_entry_bytes(run-\u003emode);\n+}\n+\n+static inline size_t trie_node_bytes(const struct stack_depot_frame_run *run)\n+{\n+\treturn ALIGN(offsetof(struct stack_depot_trie_node, data) +\n+\t\t stack_depot_frame_run_bytes(run), sizeof(unsigned long));\n+}\n+\n+static size_t trie_children_alloc_size(unsigned int capacity)\n+{\n+\tsize_t size;\n+\n+\tsize = struct_size_t(struct stack_depot_trie_children, nodes,\n+\t\t\t capacity);\n+\treturn offsetof(struct stack_depot_trie_retired_children, data) +\n+\t\tALIGN(size, sizeof(unsigned long));\n+}\n+\n+static inline unsigned int trie_side_table_root_index(u32 id)\n+{\n+\treturn ((id - 1) / STACK_DEPOT_TRIE_SIDE_TABLE_CHUNK_SIZE) /\n+\t\tSTACK_DEPOT_TRIE_SIDE_TABLE_DIR_SIZE;\n+}\n+\n+static inline unsigned int trie_side_table_dir_index(u32 id)\n+{\n+\treturn ((id - 1) / STACK_DEPOT_TRIE_SIDE_TABLE_CHUNK_SIZE) %\n+\t\tSTACK_DEPOT_TRIE_SIDE_TABLE_DIR_SIZE;\n+}\n+\n+static inline unsigned int trie_side_table_slot_index(u32 id)\n+{\n+\treturn (id - 1) % STACK_DEPOT_TRIE_SIDE_TABLE_CHUNK_SIZE;\n+}\n+\n+static struct stack_depot_trie_side_dir *trie_side_table_load_dir(unsigned int root)\n+{\n+\tstruct stack_depot_trie_side_root *root_vec;\n+\n+\troot_vec = trie_side_table_root;\n+\tif (!root_vec || root \u003e= root_vec-\u003edir_capacity)\n+\t\treturn NULL;\n+\t/* Pairs with side-table directory rcu_assign_pointer(). */\n+\treturn rcu_dereference_check(root_vec-\u003edirs[root],\n+\t\t\t\t lockdep_is_held(\u0026stack_depot_trie_writer_lock) ||\n+\t\t\t\t rcu_read_lock_sched_held());\n+}\n+\n+static inline const struct stack_depot_trie_node __rcu **\n+trie_side_table_dir_load_chunk(struct stack_depot_trie_side_dir *dir,\n+\t\t\t unsigned int idx)\n+{\n+\t/* Pairs with the chunk rcu_assign_pointer() in stack ID preparation. */\n+\treturn rcu_dereference_check(dir-\u003echunks[idx],\n+\t\t\t\t lockdep_is_held(\u0026stack_depot_trie_writer_lock) ||\n+\t\t\t\t rcu_read_lock_sched_held());\n+}\n+\n+/* Published capacity remains useful if insertion fails and needs no rollback. */\n+static bool\n+trie_side_table_try_take_cache(struct stack_depot_trie_side_prealloc *prealloc,\n+\t\t\t bool need_dir)\n+{\n+\tbool taken = false;\n+\n+\tlockdep_assert_held(\u0026stack_depot_trie_writer_lock);\n+\tlockdep_assert_held(\u0026pool_lock);\n+\n+\tif (!raw_spin_trylock(\u0026trie_side_table_cache_lock))\n+\t\treturn false;\n+\tif ((!prealloc-\u003echunk \u0026\u0026 !trie_side_table_cache.chunk) ||\n+\t (need_dir \u0026\u0026 !prealloc-\u003edir \u0026\u0026 !trie_side_table_cache.dir))\n+\t\tgoto out_unlock;\n+\n+\tif (need_dir \u0026\u0026 !prealloc-\u003edir) {\n+\t\tprealloc-\u003edir = trie_side_table_cache.dir;\n+\t\ttrie_side_table_cache.dir = NULL;\n+\t}\n+\tif (!prealloc-\u003echunk) {\n+\t\tprealloc-\u003echunk = trie_side_table_cache.chunk;\n+\t\ttrie_side_table_cache.chunk = NULL;\n+\t}\n+\ttaken = true;\n+\n+out_unlock:\n+\traw_spin_unlock(\u0026trie_side_table_cache_lock);\n+\treturn taken;\n+}\n+\n+static u32\n+trie_side_table_prepare_stack_slot(struct stack_depot_trie_side_prealloc *prealloc)\n+{\n+\tconst struct stack_depot_trie_node __rcu **chunk;\n+\tstruct stack_depot_trie_side_dir *dir;\n+\tstruct stack_depot_trie_side_root *root_vec;\n+\tunsigned int root;\n+\tunsigned int idx;\n+\tu32 id;\n+\n+\tlockdep_assert_held(\u0026stack_depot_trie_writer_lock);\n+\tlockdep_assert_held(\u0026pool_lock);\n+\n+\tid = trie_side_table_last_stack_id + 1;\n+\tif (id \u003e trie_max_stack_id())\n+\t\treturn 0;\n+\n+\troot_vec = trie_side_table_root;\n+\troot = trie_side_table_root_index(id);\n+\tdir = trie_side_table_load_dir(root);\n+\tif (!dir) {\n+\t\tif ((!prealloc-\u003edir || !prealloc-\u003echunk) \u0026\u0026\n+\t\t !trie_side_table_try_take_cache(prealloc, true))\n+\t\t\treturn 0;\n+\t\tdir = prealloc-\u003edir;\n+\t\tprealloc-\u003edir = NULL;\n+\t\t/* Publish the zeroed directory before readers can load it locklessly. */\n+\t\trcu_assign_pointer(root_vec-\u003edirs[root], dir);\n+\t}\n+\n+\tidx = trie_side_table_dir_index(id);\n+\tchunk = trie_side_table_dir_load_chunk(dir, idx);\n+\tif (!chunk) {\n+\t\tif (!prealloc-\u003echunk \u0026\u0026\n+\t\t !trie_side_table_try_take_cache(prealloc, false))\n+\t\t\treturn 0;\n+\t\tchunk = prealloc-\u003echunk;\n+\t\tprealloc-\u003echunk = NULL;\n+\t\trcu_assign_pointer(dir-\u003echunks[idx], chunk);\n+\t}\n+\n+\treturn id;\n+}\n+\n+static inline unsigned int trie_side_table_root_size_for_max_id(u32 max_stack_id)\n+{\n+\tunsigned int top_size;\n+\n+\ttop_size = DIV_ROUND_UP(max_stack_id, STACK_DEPOT_TRIE_SIDE_TABLE_CHUNK_SIZE);\n+\treturn DIV_ROUND_UP(top_size, STACK_DEPOT_TRIE_SIDE_TABLE_DIR_SIZE);\n+}\n+\n+static int __init stack_depot_trie_init_memblock(void)\n+{\n+\tstruct stack_depot_trie_side_root *root_vec;\n+\tstruct stack_depot_trie_side_dir *first_dir;\n+\tconst struct stack_depot_trie_node __rcu **first_chunk;\n+\tsize_t root_bytes;\n+\tu32 max_stack_id;\n+\tunsigned int root_size;\n+\n+\tmax_stack_id = trie_max_stack_id();\n+\tif (!max_stack_id)\n+\t\treturn -EINVAL;\n+\troot_size = trie_side_table_root_size_for_max_id(max_stack_id);\n+\troot_bytes = struct_size_t(struct stack_depot_trie_side_root, dirs, root_size);\n+\n+\troot_vec = memblock_alloc(root_bytes, __alignof__(*root_vec));\n+\tif (!root_vec)\n+\t\treturn -ENOMEM;\n+\tfirst_dir = memblock_alloc(PAGE_SIZE, PAGE_SIZE);\n+\tif (!first_dir) {\n+\t\tmemblock_free(root_vec, root_bytes);\n+\t\treturn -ENOMEM;\n+\t}\n+\tfirst_chunk = memblock_alloc(PAGE_SIZE, PAGE_SIZE);\n+\tif (!first_chunk) {\n+\t\tmemblock_free(first_dir, PAGE_SIZE);\n+\t\tmemblock_free(root_vec, root_bytes);\n+\t\treturn -ENOMEM;\n+\t}\n+\n+\troot_vec-\u003edir_capacity = root_size;\n+\tRCU_INIT_POINTER(root_vec-\u003edirs[0], first_dir);\n+\tRCU_INIT_POINTER(first_dir-\u003echunks[0], first_chunk);\n+\ttrie_side_table_root = root_vec;\n+\tstatic_branch_enable(\u0026stack_depot_trie_enabled);\n+\treturn 0;\n+}\n+\n+static int stack_depot_trie_init(void)\n+{\n+\tstruct stack_depot_trie_side_root *root_vec;\n+\tunsigned int root_size;\n+\tsize_t root_bytes;\n+\tu32 max_stack_id;\n+\n+\tmax_stack_id = trie_max_stack_id();\n+\tif (!max_stack_id)\n+\t\treturn -EINVAL;\n+\n+\troot_size = trie_side_table_root_size_for_max_id(max_stack_id);\n+\troot_bytes = struct_size_t(struct stack_depot_trie_side_root, dirs, root_size);\n+\troot_vec = kvzalloc(root_bytes, GFP_KERNEL);\n+\tif (!root_vec)\n+\t\treturn -ENOMEM;\n+\n+\troot_vec-\u003edir_capacity = root_size;\n+\ttrie_side_table_root = root_vec;\n+\tstatic_branch_enable(\u0026stack_depot_trie_enabled);\n+\treturn 0;\n+}\n+\n+static int trie_side_table_get_prealloc(gfp_t gfp_flags,\n+\t\t\t\t\tstruct stack_depot_trie_side_prealloc *prealloc)\n+{\n+\tunsigned long flags;\n+\n+\tgfp_flags = gfp_nested_mask(gfp_flags);\n+\traw_spin_lock_irqsave(\u0026trie_side_table_cache_lock, flags);\n+\tprealloc-\u003edir = trie_side_table_cache.dir;\n+\tprealloc-\u003echunk = trie_side_table_cache.chunk;\n+\ttrie_side_table_cache.dir = NULL;\n+\ttrie_side_table_cache.chunk = NULL;\n+\traw_spin_unlock_irqrestore(\u0026trie_side_table_cache_lock, flags);\n+\n+\tif (!prealloc-\u003edir) {\n+\t\tprealloc-\u003edir = (void *)get_zeroed_page(gfp_flags);\n+\t\tif (!prealloc-\u003edir)\n+\t\t\treturn -ENOMEM;\n+\t}\n+\tif (!prealloc-\u003echunk) {\n+\t\tprealloc-\u003echunk = (void *)get_zeroed_page(gfp_flags);\n+\t\tif (!prealloc-\u003echunk)\n+\t\t\treturn -ENOMEM;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static void trie_side_table_put_prealloc(struct stack_depot_trie_side_prealloc *prealloc)\n+{\n+\tunsigned long flags;\n+\n+\traw_spin_lock_irqsave(\u0026trie_side_table_cache_lock, flags);\n+\tif (!trie_side_table_cache.dir) {\n+\t\ttrie_side_table_cache.dir = prealloc-\u003edir;\n+\t\tprealloc-\u003edir = NULL;\n+\t}\n+\tif (!trie_side_table_cache.chunk) {\n+\t\ttrie_side_table_cache.chunk = prealloc-\u003echunk;\n+\t\tprealloc-\u003echunk = NULL;\n+\t}\n+\traw_spin_unlock_irqrestore(\u0026trie_side_table_cache_lock, flags);\n+\n+\tif (prealloc-\u003edir)\n+\t\tfree_page((unsigned long)prealloc-\u003edir);\n+\tif (prealloc-\u003echunk)\n+\t\tfree_page((unsigned long)prealloc-\u003echunk);\n+}\n+\n+static const struct stack_depot_trie_node *trie_side_table_lookup(u32 id)\n+{\n+\tconst struct stack_depot_trie_node __rcu **chunk;\n+\tstruct stack_depot_trie_side_dir *dir;\n+\tunsigned int root;\n+\n+\troot = trie_side_table_root_index(id);\n+\tdir = trie_side_table_load_dir(root);\n+\tif (!dir)\n+\t\treturn NULL;\n+\tchunk = trie_side_table_dir_load_chunk(dir, trie_side_table_dir_index(id));\n+\tif (!chunk)\n+\t\treturn NULL;\n+\n+\t/* Pairs with side-table node publication. */\n+\treturn rcu_dereference_check(chunk[trie_side_table_slot_index(id)],\n+\t\t\t\t lockdep_is_held(\u0026stack_depot_trie_writer_lock) ||\n+\t\t\t\t rcu_read_lock_sched_held());\n+}\n+\n+static inline struct stack_depot_trie_retired_children *\n+trie_retired_children(const void *ptr)\n+{\n+\treturn container_of(ptr, struct stack_depot_trie_retired_children, data);\n+}\n+\n+static bool depot_init_pool(void **prealloc);\n+\n+static unsigned int trie_pool_reserve_slots(struct stack_depot_trie_pool *pool,\n+\t\t\t\t\t unsigned int nr_slots)\n+{\n+\tunsigned int start = pool-\u003enext_slot;\n+\tunsigned int run = 0;\n+\tunsigned int longest_run = 0;\n+\tunsigned int i;\n+\tunsigned int slot;\n+\n+scan:\n+\trun = 0;\n+\tlongest_run = 0;\n+\tfor (slot = start; slot \u003c STACK_DEPOT_TRIE_POOL_SLOTS; slot++) {\n+\t\tif (pool-\u003eused[slot / BITS_PER_LONG] \u0026\n+\t\t BIT(slot % BITS_PER_LONG)) {\n+\t\t\trun = 0;\n+\t\t\tcontinue;\n+\t\t}\n+\t\trun++;\n+\t\tlongest_run = max(longest_run, run);\n+\t\tif (run != nr_slots)\n+\t\t\tcontinue;\n+\n+\t\tfor (i = slot + 1 - nr_slots; i \u003c= slot; i++)\n+\t\t\tpool-\u003eused[i / BITS_PER_LONG] |= BIT(i % BITS_PER_LONG);\n+\t\tpool-\u003efree_slots -= nr_slots;\n+\t\tif (slot + 1 == STACK_DEPOT_TRIE_POOL_SLOTS)\n+\t\t\tpool-\u003enext_slot = STACK_DEPOT_TRIE_POOL_FIRST_SLOT;\n+\t\telse\n+\t\t\tpool-\u003enext_slot = slot + 1;\n+\t\treturn slot + 1 - nr_slots;\n+\t}\n+\n+\tif (start != STACK_DEPOT_TRIE_POOL_FIRST_SLOT) {\n+\t\t/* Keep holes and runs crossing the cursor visible. */\n+\t\tstart = STACK_DEPOT_TRIE_POOL_FIRST_SLOT;\n+\t\tgoto scan;\n+\t}\n+\n+\tpool-\u003efree_run_upper_bound = longest_run;\n+\treturn STACK_DEPOT_TRIE_POOL_SLOTS;\n+}\n+\n+/* Allocate at least @size bytes from one contiguous trie-pool slot run. */\n+static void *trie_pool_alloc(size_t size, void **prealloc)\n+{\n+\tstruct stack_depot_trie_pool *pool;\n+\tunsigned int nr_slots;\n+\tunsigned int slot;\n+\n+\tlockdep_assert_held(\u0026pool_lock);\n+\n+\tif (size \u003e STACK_DEPOT_TRIE_POOL_USABLE_SIZE)\n+\t\treturn NULL;\n+\tnr_slots = DIV_ROUND_UP(size, STACK_DEPOT_TRIE_SLOT_SIZE);\n+\tlist_for_each_entry_reverse(pool, \u0026stack_depot_trie_pools, list) {\n+\t\tif (pool-\u003efree_slots \u003c nr_slots ||\n+\t\t pool-\u003efree_run_upper_bound \u003c nr_slots)\n+\t\t\tcontinue;\n+\t\tslot = trie_pool_reserve_slots(pool, nr_slots);\n+\t\tif (slot != STACK_DEPOT_TRIE_POOL_SLOTS)\n+\t\t\treturn (char *)pool + slot * STACK_DEPOT_TRIE_SLOT_SIZE;\n+\t}\n+\n+\tif (!depot_init_pool(prealloc))\n+\t\treturn NULL;\n+\tpool = stack_pools[pools_num - 1];\n+\t/* Keep hash records out of this bitmap-owned pool. */\n+\tpool_offset = DEPOT_POOL_SIZE;\n+\tmemset(pool, 0, sizeof(*pool));\n+\tpool-\u003efree_slots = STACK_DEPOT_TRIE_POOL_SLOTS -\n+\t\t\t STACK_DEPOT_TRIE_POOL_FIRST_SLOT;\n+\tpool-\u003efree_run_upper_bound = pool-\u003efree_slots;\n+\tpool-\u003enext_slot = STACK_DEPOT_TRIE_POOL_FIRST_SLOT;\n+\tlist_add_tail(\u0026pool-\u003elist, \u0026stack_depot_trie_pools);\n+\n+\tslot = trie_pool_reserve_slots(pool, nr_slots);\n+\treturn (char *)pool + slot * STACK_DEPOT_TRIE_SLOT_SIZE;\n+}\n+\n+/* Release the slots for the byte count originally passed to allocation. */\n+static void trie_pool_release(const void *ptr, size_t size)\n+{\n+\tstruct stack_depot_trie_pool *pool;\n+\tunsigned long pfn;\n+\tunsigned int nr_slots;\n+\tunsigned int slot;\n+\tunsigned int i;\n+\n+\tlockdep_assert_held(\u0026pool_lock);\n+\n+\tpfn = page_to_pfn(virt_to_page(ptr));\n+\tpfn \u0026= ~(BIT(DEPOT_POOL_ORDER) - 1);\n+\tpool = page_address(pfn_to_page(pfn));\n+\tslot = ((unsigned long)ptr - (unsigned long)pool) \u003e\u003e DEPOT_STACK_ALIGN;\n+\tnr_slots = DIV_ROUND_UP(size, STACK_DEPOT_TRIE_SLOT_SIZE);\n+\tfor (i = slot; i \u003c slot + nr_slots; i++)\n+\t\tpool-\u003eused[i / BITS_PER_LONG] \u0026= ~BIT(i % BITS_PER_LONG);\n+\tpool-\u003efree_slots += nr_slots;\n+\t/* A release can join at most two runs bounded by the old value. */\n+\tpool-\u003efree_run_upper_bound = min(pool-\u003efree_slots,\n+\t\t\t\t\t 2 * pool-\u003efree_run_upper_bound + nr_slots);\n+}\n+\n+static struct stack_depot_trie_children *\n+trie_pool_alloc_children(unsigned int capacity, void **prealloc)\n+{\n+\tstruct stack_depot_trie_retired_children *retired;\n+\tstruct stack_depot_trie_children *children;\n+\n+\t/* Capacity counts child-pointer entries; allocation includes RCU metadata. */\n+\tretired = trie_pool_alloc(trie_children_alloc_size(capacity), prealloc);\n+\tif (!retired)\n+\t\treturn NULL;\n+\n+\tchildren = (void *)retired-\u003edata;\n+\tchildren-\u003enr_children = 0;\n+\tchildren-\u003ecapacity = capacity;\n+\treturn children;\n+}\n+\n+static void\n+trie_pool_release_children(const struct stack_depot_trie_children *children)\n+{\n+\t/* Capacity is immutable and therefore recovers the allocation byte size. */\n+\ttrie_pool_release(trie_retired_children(children),\n+\t\t\t trie_children_alloc_size(children-\u003ecapacity));\n+}\n+\n+/*\n+ * Return RCU-ready objects before allocating. Pending children are FIFO, so\n+ * stop at the first incomplete grace period. A replaced node shares the same\n+ * retirement cookie and is released with its former children container.\n+ */\n+static void trie_drain_pending_children(void)\n+{\n+\tstruct stack_depot_trie_retired_children *retired;\n+\tstruct stack_depot_trie_retired_children *tmp;\n+\tstruct stack_depot_trie_children *children;\n+\n+\tlockdep_assert_held(\u0026pool_lock);\n+\n+\tlist_for_each_entry_safe(retired, tmp, \u0026pending_trie_children, list) {\n+\t\tif (!poll_state_synchronize_rcu(retired-\u003ercu_state))\n+\t\t\tbreak;\n+\t\tchildren = (void *)retired-\u003edata;\n+\t\tlist_del(\u0026retired-\u003elist);\n+\t\tif (retired-\u003epending_node)\n+\t\t\ttrie_pool_release(retired-\u003epending_node,\n+\t\t\t\t\t trie_node_bytes(\u0026retired-\u003epending_node-\u003erun));\n+\t\ttrie_pool_release_children(children);\n+\t}\n+}\n+\n+static void trie_retire_children(const struct stack_depot_trie_children *children)\n+{\n+\tstruct stack_depot_trie_retired_children *retired;\n+\n+\tlockdep_assert_held(\u0026pool_lock);\n+\n+\tretired = trie_retired_children(children);\n+\tretired-\u003epending_node = NULL;\n+\tretired-\u003ercu_state = get_state_synchronize_rcu();\n+\tlist_add_tail(\u0026retired-\u003elist, \u0026pending_trie_children);\n+}\n+\n+static void\n+trie_retire_children_with_node(const struct stack_depot_trie_children *children,\n+\t\t\t const struct stack_depot_trie_node *node)\n+{\n+\tstruct stack_depot_trie_retired_children *retired;\n+\n+\tlockdep_assert_held(\u0026stack_depot_trie_writer_lock);\n+\tlockdep_assert_held(\u0026pool_lock);\n+\ttrie_retire_children(children);\n+\tretired = trie_retired_children(children);\n+\tretired-\u003epending_node = node;\n+}\n+\n+static const struct stack_depot_trie_node *\n+stack_depot_trie_lookup(const unsigned long *entries, unsigned int nr_entries);\n+\n+static depot_stack_handle_t\n+trie_find_handle(const unsigned long *entries, unsigned int nr_entries)\n+{\n+\tdepot_stack_handle_t handle = 0;\n+\tconst struct stack_depot_trie_node *node;\n+\n+\trcu_read_lock_sched_notrace();\n+\tnode = stack_depot_trie_lookup(entries, nr_entries);\n+\tif (node)\n+\t\thandle = trie_handle(node-\u003estack_id);\n+\trcu_read_unlock_sched_notrace();\n+\n+\treturn handle;\n+}\n+\n+/*\n+ * Publish only after the node and its path are fully initialized and all\n+ * fallible allocation is complete. Publication commits the path, so it cannot\n+ * then be rolled back. Side-table mappings must precede trie topology\n+ * publication that makes new or remapped nodes reachable from lookup.\n+ * Published storage remains valid until RCU retirement; only descendant parent\n+ * links may change meanwhile.\n+ */\n+static void trie_side_table_publish(const struct stack_depot_trie_node *node)\n+{\n+\tconst struct stack_depot_trie_node __rcu **chunk;\n+\tstruct stack_depot_trie_side_dir *dir;\n+\tu32 stack_id = node-\u003estack_id;\n+\n+\tlockdep_assert_held(\u0026stack_depot_trie_writer_lock);\n+\n+\tdir = trie_side_table_load_dir(trie_side_table_root_index(stack_id));\n+\tchunk = trie_side_table_dir_load_chunk(dir,\n+\t\t\t\t\t trie_side_table_dir_index(stack_id));\n+\t/* Pairs with trie_side_table_lookup(). */\n+\trcu_assign_pointer(chunk[trie_side_table_slot_index(stack_id)], node);\n+}\n+\n static int __init disable_stack_depot(char *str)\n {\n \treturn kstrtobool(str, \u0026stack_depot_disabled);\n@@ -146,7 +844,7 @@ static void init_stack_table(unsigned long entries)\n \t\tINIT_LIST_HEAD(\u0026stack_table[i]);\n }\n \n-/* Allocates a hash table via memblock. Can only be used during early boot. */\n+/* Initializes hash and optional trie storage during early boot. */\n int __init stack_depot_early_init(void)\n {\n \tunsigned long entries = 0;\n@@ -220,11 +918,15 @@ int __init stack_depot_early_init(void)\n \t\tstack_depot_disabled = true;\n \t\treturn -ENOMEM;\n \t}\n+\tif (stack_depot_trie_requested \u0026\u0026 stack_depot_trie_init_memblock()) {\n+\t\tpr_warn(\"trie storage initialization failed, disabling trie storage\\n\");\n+\t\tstack_depot_trie_requested = false;\n+\t}\n \n \treturn 0;\n }\n \n-/* Allocates a hash table via kvcalloc. Can be used after boot. */\n+/* Initializes hash and optional trie storage after boot. */\n int stack_depot_init(void)\n {\n \tstatic DEFINE_MUTEX(stack_depot_init_mutex);\n@@ -278,6 +980,15 @@ int stack_depot_init(void)\n \t\tkvfree(stack_table);\n \t\tstack_depot_disabled = true;\n \t\tret = -ENOMEM;\n+\t\tgoto out_unlock;\n+\t}\n+\tif (stack_depot_trie_requested) {\n+\t\tret = stack_depot_trie_init();\n+\t\tif (ret) {\n+\t\t\tpr_warn(\"trie storage initialization failed, disabling trie storage\\n\");\n+\t\t\tstack_depot_trie_requested = false;\n+\t\t\tret = 0;\n+\t\t}\n \t}\n \n out_unlock:\n@@ -323,7 +1034,7 @@ static bool depot_init_pool(void **prealloc)\n \t * NULL; do not reset to NULL if we have reached the maximum number of\n \t * pools.\n \t */\n-\tif (pools_num \u003c stack_max_pools)\n+\tif (pools_num + 1 \u003c stack_max_pools)\n \t\tWRITE_ONCE(new_pool, NULL);\n \telse\n \t\tWRITE_ONCE(new_pool, STACK_DEPOT_POISON);\n@@ -467,6 +1178,7 @@ depot_alloc_stack(unsigned long *entries, unsigned int nr_entries, u32 hash, dep\n \t/* Save the stack trace. */\n \tstack-\u003ehash = hash;\n \tstack-\u003esize = nr_entries;\n+\tstack-\u003eflags = flags \u0026 STACK_DEPOT_FLAG_COUNTABLE;\n \t/* stack-\u003ehandle is already filled in by depot_pop_free_pool(). */\n \tmemcpy(stack-\u003eentries, entries, flex_array_size(stack, entries, nr_entries));\n \n@@ -609,6 +1321,9 @@ static inline struct stack_record *find_stack(struct list_head *bucket,\n \tlist_for_each_entry_rcu(stack, bucket, hash_list) {\n \t\tif (stack-\u003ehash != hash || stack-\u003esize != size)\n \t\t\tcontinue;\n+\t\t/* Page owner countable records have a distinct count lifetime. */\n+\t\tif ((stack-\u003eflags ^ flags) \u0026 STACK_DEPOT_FLAG_COUNTABLE)\n+\t\t\tcontinue;\n \n \t\t/*\n \t\t * This may race with depot_free_stack() accessing the freelist\n@@ -638,6 +1353,101 @@ static inline struct stack_record *find_stack(struct list_head *bucket,\n \treturn ret;\n }\n \n+static u32\n+stack_depot_trie_insert(const unsigned long *entries,\n+\t\t\tunsigned int nr_entries, void **pool_prealloc,\n+\t\t\tstruct stack_depot_trie_side_prealloc *side_prealloc);\n+\n+static depot_stack_handle_t\n+stack_depot_trie_save(unsigned long *entries, unsigned int nr_entries,\n+\t\t gfp_t alloc_flags)\n+{\n+\tunsigned int attempt;\n+\n+\t/* Allow one stale pool hint before the two pools a largest insert needs. */\n+\tfor (attempt = 0; attempt \u003c 3; attempt++) {\n+\t\tstruct stack_depot_trie_side_prealloc side_prealloc = {};\n+\t\tvoid *pool_prealloc = NULL;\n+\t\tdepot_stack_handle_t handle;\n+\t\tunsigned long flags;\n+\t\tstruct page *page;\n+\t\tu32 stack_id = 0;\n+\n+\t\thandle = trie_find_handle(entries, nr_entries);\n+\t\tif (handle)\n+\t\t\treturn handle;\n+\n+\t\tif (trie_side_table_get_prealloc(alloc_flags, \u0026side_prealloc)) {\n+\t\t\ttrie_side_table_put_prealloc(\u0026side_prealloc);\n+\t\t\treturn 0;\n+\t\t}\n+\n+\t\t/* The hint may race; a missing page is recovered by the retry. */\n+\t\tif (!READ_ONCE(new_pool)) {\n+\t\t\tpage = alloc_pages(gfp_nested_mask(alloc_flags),\n+\t\t\t\t\t DEPOT_POOL_ORDER);\n+\t\t\tif (page)\n+\t\t\t\tpool_prealloc = page_address(page);\n+\t\t}\n+\n+\t\traw_spin_lock_irqsave(\u0026stack_depot_trie_writer_lock, flags);\n+\t\traw_spin_lock(\u0026pool_lock);\n+\t\tprintk_deferred_enter();\n+\t\ttrie_drain_pending_children();\n+\t\tstack_id = stack_depot_trie_insert(entries, nr_entries,\n+\t\t\t\t\t\t \u0026pool_prealloc, \u0026side_prealloc);\n+\t\tif (pool_prealloc)\n+\t\t\tdepot_keep_new_pool(\u0026pool_prealloc);\n+\t\tprintk_deferred_exit();\n+\t\traw_spin_unlock(\u0026pool_lock);\n+\t\traw_spin_unlock_irqrestore(\u0026stack_depot_trie_writer_lock, flags);\n+\n+\t\tif (pool_prealloc)\n+\t\t\tfree_pages((unsigned long)pool_prealloc, DEPOT_POOL_ORDER);\n+\t\ttrie_side_table_put_prealloc(\u0026side_prealloc);\n+\t\tif (stack_id)\n+\t\t\treturn trie_handle(stack_id);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static depot_stack_handle_t\n+stack_depot_trie_save_constrained(unsigned long *entries,\n+\t\t\t\t unsigned int nr_entries, bool trylock)\n+{\n+\tstruct stack_depot_trie_side_prealloc side_prealloc = {};\n+\tvoid *pool_prealloc = NULL;\n+\tdepot_stack_handle_t handle;\n+\tunsigned long flags;\n+\tu32 stack_id;\n+\n+\thandle = trie_find_handle(entries, nr_entries);\n+\tif (handle)\n+\t\treturn handle;\n+\n+\tif (trylock) {\n+\t\tif (!raw_spin_trylock_irqsave(\u0026stack_depot_trie_writer_lock, flags))\n+\t\t\treturn 0;\n+\t\tif (!raw_spin_trylock(\u0026pool_lock)) {\n+\t\t\traw_spin_unlock_irqrestore(\u0026stack_depot_trie_writer_lock, flags);\n+\t\t\treturn 0;\n+\t\t}\n+\t} else {\n+\t\traw_spin_lock_irqsave(\u0026stack_depot_trie_writer_lock, flags);\n+\t\traw_spin_lock(\u0026pool_lock);\n+\t}\n+\n+\tprintk_deferred_enter();\n+\tstack_id = stack_depot_trie_insert(entries, nr_entries, \u0026pool_prealloc,\n+\t\t\t\t\t \u0026side_prealloc);\n+\tprintk_deferred_exit();\n+\traw_spin_unlock(\u0026pool_lock);\n+\traw_spin_unlock_irqrestore(\u0026stack_depot_trie_writer_lock, flags);\n+\n+\treturn stack_id ? trie_handle(stack_id) : 0;\n+}\n+\n depot_stack_handle_t stack_depot_save_flags(unsigned long *entries,\n \t\t\t\t\t unsigned int nr_entries,\n \t\t\t\t\t gfp_t alloc_flags,\n@@ -655,6 +1465,9 @@ depot_stack_handle_t stack_depot_save_flags(unsigned long *entries,\n \n \tif (WARN_ON(depot_flags \u0026 ~STACK_DEPOT_FLAGS_MASK))\n \t\treturn 0;\n+\tif (WARN_ON_ONCE((depot_flags \u0026 STACK_DEPOT_FLAG_GET) \u0026\u0026\n+\t\t\t (depot_flags \u0026 STACK_DEPOT_FLAG_COUNTABLE)))\n+\t\treturn 0;\n \n \t/*\n \t * If this stack trace is from an interrupt, including anything before\n@@ -669,6 +1482,20 @@ depot_stack_handle_t stack_depot_save_flags(unsigned long *entries,\n \tif (unlikely(nr_entries == 0) || stack_depot_disabled)\n \t\treturn 0;\n \n+\tif (!(depot_flags \u0026 (STACK_DEPOT_FLAG_GET | STACK_DEPOT_FLAG_COUNTABLE)) \u0026\u0026\n+\t static_branch_unlikely(\u0026stack_depot_trie_enabled)) {\n+\t\tif (nr_entries \u003e CONFIG_STACKDEPOT_MAX_FRAMES)\n+\t\t\tnr_entries = CONFIG_STACKDEPOT_MAX_FRAMES;\n+\t\tif (in_nmi()) {\n+\t\t\tWARN_ON_ONCE(can_alloc);\n+\t\t\treturn trie_find_handle(entries, nr_entries);\n+\t\t}\n+\t\tif (!can_alloc)\n+\t\t\treturn stack_depot_trie_save_constrained(entries, nr_entries,\n+\t\t\t\t\t\t\t !allow_spin);\n+\t\treturn stack_depot_trie_save(entries, nr_entries, alloc_flags);\n+\t}\n+\n \thash = hash_stack(entries, nr_entries);\n \tbucket = \u0026stack_table[hash \u0026 stack_hash_mask];\n \n@@ -751,10 +1578,728 @@ EXPORT_SYMBOL_GPL(stack_depot_save);\n \n struct stack_record *__stack_depot_get_stack_record(depot_stack_handle_t handle)\n {\n+\tstruct stack_record *stack;\n+\n \tif (!handle)\n \t\treturn NULL;\n+\tif (WARN_ON_ONCE(stack_depot_handle_is_trie(handle)))\n+\t\treturn NULL;\n+\n+\tstack = depot_fetch_stack(handle);\n+\tif (!stack)\n+\t\treturn NULL;\n+\tif (WARN_ON_ONCE(!(stack-\u003eflags \u0026 STACK_DEPOT_FLAG_COUNTABLE)))\n+\t\treturn NULL;\n+\n+\treturn stack;\n+}\n+\n+static void frame_run_init(const unsigned long *entries,\n+\t\t\t unsigned int nr_entries,\n+\t\t\t struct stack_depot_frame_run *run)\n+{\n+\tu32 payload;\n+\tunsigned int i;\n+\tbool compressed;\n+\n+\tcompressed = arch_stack_depot_frame_try_compress(entries[0], \u0026payload);\n+\tfor (i = 1; i \u003c nr_entries; i++) {\n+\t\tbool next;\n+\n+\t\tnext = arch_stack_depot_frame_try_compress(entries[i], \u0026payload);\n+\t\tif (next != compressed)\n+\t\t\tbreak;\n+\t}\n+\n+\t/* @i is the first non-matching frame, or @nr_entries if all matched. */\n+\trun-\u003emode = compressed ? STACK_DEPOT_FRAME_COMPRESSED : STACK_DEPOT_FRAME_RAW;\n+\trun-\u003enr_entries = i;\n+}\n+\n+static void\n+stack_depot_trie_node_frame(const struct stack_depot_trie_node *node,\n+\t\t\t unsigned int index, unsigned long *frame)\n+{\n+\tu32 payload;\n+\n+\tif (node-\u003erun.mode == STACK_DEPOT_FRAME_RAW) {\n+\t\tmemcpy(frame, node-\u003edata + index * sizeof(*frame),\n+\t\t sizeof(*frame));\n+\t\treturn;\n+\t}\n+\n+\tmemcpy(\u0026payload, node-\u003edata + index * sizeof(payload), sizeof(payload));\n+\tarch_stack_depot_frame_decompress(payload, frame);\n+}\n+\n+static void trie_node_init(struct stack_depot_trie_node *node,\n+\t\t\t const struct stack_depot_trie_node *parent, u32 stack_id,\n+\t\t\t const unsigned long *entries,\n+\t\t\t const struct stack_depot_frame_run *run)\n+{\n+\tif (run-\u003emode == STACK_DEPOT_FRAME_COMPRESSED) {\n+\t\tunsigned int i;\n+\n+\t\tfor (i = 0; i \u003c run-\u003enr_entries; i++) {\n+\t\t\tu32 payload;\n+\n+\t\t\tarch_stack_depot_frame_try_compress(entries[i], \u0026payload);\n+\t\t\tmemcpy(node-\u003edata + i * sizeof(payload), \u0026payload,\n+\t\t\t sizeof(payload));\n+\t\t}\n+\t} else {\n+\t\tmemcpy(node-\u003edata, entries, stack_depot_frame_run_bytes(run));\n+\t}\n+\n+\tRCU_INIT_POINTER(node-\u003eparent, parent);\n+\tRCU_INIT_POINTER(node-\u003echildren, NULL);\n+\tnode-\u003estack_id = stack_id;\n+\tnode-\u003erun = *run;\n+}\n+\n+static void trie_node_init_slice(struct stack_depot_trie_node *node,\n+\t\t\t\t const struct stack_depot_trie_node *parent, u32 stack_id,\n+\t\t\t\t const struct stack_depot_trie_node *src_node,\n+\t\t\t\t unsigned int start, unsigned int nr_entries)\n+{\n+\tstruct stack_depot_frame_run run;\n+\tsize_t entry_bytes;\n+\n+\trun = src_node-\u003erun;\n+\trun.nr_entries = nr_entries;\n+\n+\tentry_bytes = stack_depot_frame_run_entry_bytes(src_node-\u003erun.mode);\n+\tmemcpy(node-\u003edata, src_node-\u003edata + start * entry_bytes,\n+\t stack_depot_frame_run_bytes(\u0026run));\n+\tRCU_INIT_POINTER(node-\u003eparent, parent);\n+\tRCU_INIT_POINTER(node-\u003echildren, NULL);\n+\tnode-\u003estack_id = stack_id;\n+\tnode-\u003erun = run;\n+}\n+\n+static unsigned int trie_node_match(const struct stack_depot_trie_node *node,\n+\t\t\t\t const unsigned long *entries,\n+\t\t\t\t unsigned int nr_entries)\n+{\n+\tunsigned int limit;\n+\tunsigned int i;\n+\n+\tlimit = min(node-\u003erun.nr_entries, nr_entries);\n+\tif (node-\u003erun.mode == STACK_DEPOT_FRAME_RAW) {\n+\t\tfor (i = 0; i \u003c limit; i++) {\n+\t\t\tunsigned long frame;\n+\n+\t\t\tmemcpy(\u0026frame, node-\u003edata + i * sizeof(frame), sizeof(frame));\n+\t\t\tif (frame != entries[i])\n+\t\t\t\tbreak;\n+\t\t}\n+\n+\t\treturn i;\n+\t}\n+\n+\tfor (i = 0; i \u003c limit; i++) {\n+\t\tunsigned long frame;\n+\n+\t\tstack_depot_trie_node_frame(node, i, \u0026frame);\n+\t\tif (frame != entries[i])\n+\t\t\tbreak;\n+\t}\n+\n+\treturn i;\n+}\n+\n+static inline const struct stack_depot_trie_node *\n+trie_load_parent(const struct stack_depot_trie_node *node)\n+{\n+\treturn rcu_dereference_check(node-\u003eparent,\n+\t\t\t\t lockdep_is_held(\u0026stack_depot_trie_writer_lock) ||\n+\t\t\t\t rcu_read_lock_sched_held());\n+}\n+\n+static inline const struct stack_depot_trie_children *\n+trie_load_children(const struct stack_depot_trie_children __rcu * const *slot)\n+{\n+\treturn rcu_dereference_check(*slot,\n+\t\t\t\t lockdep_is_held(\u0026stack_depot_trie_writer_lock) ||\n+\t\t\t\t rcu_read_lock_sched_held());\n+}\n+\n+static inline const struct stack_depot_trie_node *\n+trie_children_load_child(const struct stack_depot_trie_children *children,\n+\t\t\t unsigned int pos)\n+{\n+\treturn rcu_dereference_check(children-\u003enodes[pos],\n+\t\t\t\t lockdep_is_held(\u0026stack_depot_trie_writer_lock) ||\n+\t\t\t\t rcu_read_lock_sched_held());\n+}\n+\n+static bool\n+trie_children_find_position(const struct stack_depot_trie_children *children,\n+\t\t\t unsigned long frame, unsigned int *pos)\n+{\n+\tunsigned int left = 0;\n+\tunsigned int right;\n+\n+\tright = READ_ONCE(children-\u003enr_children);\n+\twhile (left \u003c right) {\n+\t\tunsigned int mid = left + (right - left) / 2;\n+\t\tconst struct stack_depot_trie_node *node;\n+\t\tunsigned long mid_frame;\n+\n+\t\tnode = trie_children_load_child(children, mid);\n+\t\tif (!node) {\n+\t\t\t/* Tail append may produce a transient lockless lookup miss. */\n+\t\t\tright = mid;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tstack_depot_trie_node_frame(node, 0, \u0026mid_frame);\n+\t\tif (mid_frame \u003c frame) {\n+\t\t\tleft = mid + 1;\n+\t\t} else if (mid_frame \u003e frame) {\n+\t\t\tright = mid;\n+\t\t} else {\n+\t\t\t*pos = mid;\n+\t\t\treturn true;\n+\t\t}\n+\t}\n+\n+\t*pos = left;\n+\treturn false;\n+}\n+\n+/* Initialize an unpublished container from a stable published prefix. */\n+static void trie_children_init(const struct stack_depot_trie_children *old,\n+\t\t\t struct stack_depot_trie_children *new)\n+{\n+\tunsigned int nr_old = old-\u003enr_children;\n+\tunsigned int i;\n+\n+\tnew-\u003enr_children = nr_old;\n+\tfor (i = 0; i \u003c nr_old; i++)\n+\t\tRCU_INIT_POINTER(new-\u003enodes[i], trie_children_load_child(old, i));\n+\tfor (i = nr_old; i \u003c new-\u003ecapacity; i++)\n+\t\tRCU_INIT_POINTER(new-\u003enodes[i], NULL);\n+}\n+\n+static void trie_children_insert(struct stack_depot_trie_children *children,\n+\t\t\t\t const struct stack_depot_trie_node *node,\n+\t\t\t\t unsigned int pos)\n+{\n+\tunsigned int i;\n+\n+\tfor (i = children-\u003enr_children; i \u003e pos; i--)\n+\t\tRCU_INIT_POINTER(children-\u003enodes[i],\n+\t\t\t\t trie_children_load_child(children, i - 1));\n+\tRCU_INIT_POINTER(children-\u003enodes[pos], node);\n+\tchildren-\u003enr_children++;\n+}\n+\n+static void trie_reparent_children(struct stack_depot_trie_node *parent)\n+{\n+\tconst struct stack_depot_trie_children *children;\n+\tunsigned int i;\n+\n+\tlockdep_assert_held(\u0026stack_depot_trie_writer_lock);\n+\n+\tchildren = trie_load_children(\u0026parent-\u003echildren);\n+\tif (!children)\n+\t\treturn;\n+\t/*\n+\t * Replacement nodes reuse unchanged descendant subtrees. Repoint their\n+\t * parent links before retiring the old parent so fetch never follows a freed\n+\t * node. Lockless fetches may see the new parent before publication, but the\n+\t * old and new parent chains contain the same frames and remain RCU-live.\n+\t */\n+\tfor (i = 0; i \u003c children-\u003enr_children; i++) {\n+\t\tstruct stack_depot_trie_node *child;\n+\n+\t\tchild = (struct stack_depot_trie_node *)trie_children_load_child(children, i);\n+\t\trcu_assign_pointer(child-\u003eparent, parent);\n+\t}\n+}\n+\n+/*\n+ * Split entries into runs, allocate and initialize each node once, and link\n+ * adjacent nodes through singleton children. Both trie locks must be held.\n+ * Failure walks the unpublished parent chain and releases local ownership.\n+ */\n+static const struct stack_depot_trie_node *\n+trie_path_alloc(const struct stack_depot_trie_node *parent, u32 stack_id,\n+\t\tconst unsigned long *entries, unsigned int nr_entries,\n+\t\tvoid **pool_prealloc,\n+\t\tconst struct stack_depot_trie_node **node_out)\n+{\n+\tstruct stack_depot_trie_children *path_children = NULL;\n+\tconst struct stack_depot_trie_node *path_root = NULL;\n+\tconst struct stack_depot_trie_node *last_node = parent;\n+\tunsigned int entry = 0;\n+\n+\tlockdep_assert_held(\u0026pool_lock);\n+\tlockdep_assert_held(\u0026stack_depot_trie_writer_lock);\n+\n+\twhile (entry \u003c nr_entries) {\n+\t\tstruct stack_depot_frame_run run;\n+\t\tstruct stack_depot_trie_node *node;\n+\n+\t\tframe_run_init(\u0026entries[entry], nr_entries - entry, \u0026run);\n+\t\tnode = trie_pool_alloc(trie_node_bytes(\u0026run), pool_prealloc);\n+\t\tif (!node)\n+\t\t\tgoto err_release;\n+\n+\t\ttrie_node_init(node, last_node,\n+\t\t\t entry + run.nr_entries == nr_entries ? stack_id : 0,\n+\t\t\t \u0026entries[entry], \u0026run);\n+\t\tentry += run.nr_entries;\n+\t\tlast_node = node;\n+\t\tif (!path_root)\n+\t\t\tpath_root = node;\n+\n+\t\tif (path_children)\n+\t\t\ttrie_children_insert(path_children, last_node, 0);\n+\t\tif (entry \u003c nr_entries) {\n+\t\t\tpath_children = trie_pool_alloc_children(1, pool_prealloc);\n+\t\t\tif (!path_children)\n+\t\t\t\tgoto err_release;\n+\t\t\tRCU_INIT_POINTER(node-\u003echildren, path_children);\n+\t\t}\n+\t}\n+\n+\t*node_out = last_node;\n+\treturn path_root;\n+\n+err_release:\n+\twhile (last_node != parent) {\n+\t\tconst struct stack_depot_trie_children *node_children;\n+\t\tconst struct stack_depot_trie_node *node = last_node;\n+\n+\t\tlast_node = trie_load_parent(node);\n+\t\tnode_children = trie_load_children(\u0026node-\u003echildren);\n+\t\tif (node_children)\n+\t\t\ttrie_pool_release_children(node_children);\n+\t\ttrie_pool_release(node, trie_node_bytes(\u0026node-\u003erun));\n+\t}\n+\treturn NULL;\n+}\n+\n+static const struct stack_depot_trie_node *\n+stack_depot_trie_lookup(const unsigned long *entries, unsigned int nr_entries)\n+{\n+\tconst struct stack_depot_trie_children *children;\n+\tunsigned int entry = 0;\n+\n+\tchildren = trie_load_children(\u0026stack_depot_trie_root);\n+\n+\twhile (entry \u003c nr_entries) {\n+\t\tconst struct stack_depot_trie_node *node;\n+\t\tunsigned int remaining = nr_entries - entry;\n+\t\tunsigned int matched;\n+\t\tunsigned int pos;\n+\n+\t\tif (!children)\n+\t\t\treturn NULL;\n+\t\tif (!trie_children_find_position(children, entries[entry], \u0026pos))\n+\t\t\treturn NULL;\n+\n+\t\tnode = trie_children_load_child(children, pos);\n+\t\tmatched = trie_node_match(node, \u0026entries[entry], remaining);\n+\t\tif (matched \u003c node-\u003erun.nr_entries)\n+\t\t\treturn NULL;\n+\t\tentry += matched;\n+\t\tif (entry == nr_entries)\n+\t\t\treturn node-\u003estack_id ? node : NULL;\n+\n+\t\tchildren = trie_load_children(\u0026node-\u003echildren);\n+\t}\n+\n+\treturn NULL;\n+}\n+\n+static u32\n+trie_insert_path(const struct stack_depot_trie_children __rcu **slot,\n+\t\t struct stack_depot_trie_node *parent,\n+\t\t const struct stack_depot_trie_children *children,\n+\t\t unsigned int pos, const unsigned long *entries,\n+\t\t unsigned int nr_entries, void **pool_prealloc,\n+\t\t struct stack_depot_trie_side_prealloc *side_prealloc)\n+{\n+\tstruct stack_depot_trie_children *new_children = NULL;\n+\tconst struct stack_depot_trie_node *path_root;\n+\tconst struct stack_depot_trie_node *node;\n+\tunsigned int capacity = 1;\n+\tu32 new_stack_id;\n+\tbool tail_append = false;\n+\n+\t/*\n+\t * Reuse spare capacity only for a sorted tail append. Other insertions\n+\t * replace the children container without modifying visible pointers.\n+\t */\n+\tif (children) {\n+\t\tcapacity = roundup_pow_of_two(children-\u003enr_children + 1);\n+\t\ttail_append = pos == children-\u003enr_children \u0026\u0026\n+\t\t\tchildren-\u003enr_children \u003c children-\u003ecapacity;\n+\t}\n+\tif (!tail_append \u0026\u0026 trie_children_alloc_size(capacity) \u003e\n+\t STACK_DEPOT_TRIE_POOL_USABLE_SIZE)\n+\t\treturn 0;\n+\n+\tnew_stack_id = trie_side_table_prepare_stack_slot(side_prealloc);\n+\tif (!new_stack_id)\n+\t\treturn 0;\n+\n+\t/* Reserve replacement topology before the path, the final fallible step. */\n+\tif (!tail_append) {\n+\t\tnew_children = trie_pool_alloc_children(capacity, pool_prealloc);\n+\t\tif (!new_children)\n+\t\t\tgoto err_release;\n+\t}\n+\tpath_root = trie_path_alloc(parent, new_stack_id, entries, nr_entries,\n+\t\t\t\t pool_prealloc, \u0026node);\n+\tif (!path_root)\n+\t\tgoto err_release;\n+\n+\t/* Commit the stack ID before making the path reachable from the trie. */\n+\ttrie_side_table_publish(node);\n+\tif (tail_append) {\n+\t\tstruct stack_depot_trie_children *tail_children =\n+\t\t\t(struct stack_depot_trie_children *)children;\n+\n+\t\t/*\n+\t\t * Publish the node before the visible count. Readers may transiently\n+\t\t * see NULL and miss; the writer-lock recheck prevents duplicates.\n+\t\t */\n+\t\trcu_assign_pointer(tail_children-\u003enodes[pos], path_root);\n+\t\tWRITE_ONCE(tail_children-\u003enr_children, pos + 1);\n+\t} else {\n+\t\tif (children)\n+\t\t\ttrie_children_init(children, new_children);\n+\t\ttrie_children_insert(new_children, path_root, pos);\n+\t\trcu_assign_pointer(*slot, new_children);\n+\t\tif (children)\n+\t\t\ttrie_retire_children(children);\n+\t}\n+\n+\treturn new_stack_id;\n+\n+err_release:\n+\tif (new_children)\n+\t\ttrie_pool_release_children(new_children);\n+\treturn 0;\n+}\n+\n+static u32\n+trie_split_child(const struct stack_depot_trie_children __rcu **slot,\n+\t\t const struct stack_depot_trie_children *children,\n+\t\t const struct stack_depot_trie_node *child,\n+\t\t unsigned int pos, unsigned int matched,\n+\t\t const unsigned long *entries, unsigned int nr_entries,\n+\t\t void **pool_prealloc,\n+\t\t struct stack_depot_trie_side_prealloc *side_prealloc)\n+{\n+\tstruct stack_depot_trie_children *prefix_children = NULL;\n+\tstruct stack_depot_trie_children *new_children = NULL;\n+\tconst struct stack_depot_trie_node *new_node;\n+\tconst struct stack_depot_trie_node *suffix_roots[2];\n+\tstruct stack_depot_frame_run run;\n+\tstruct stack_depot_trie_node *split_prefix = NULL;\n+\tstruct stack_depot_trie_node *old_suffix = NULL;\n+\tunsigned int nr_suffix_roots;\n+\tunsigned int old_suffix_len;\n+\tunsigned int i;\n+\tsize_t split_prefix_size;\n+\tsize_t old_suffix_size;\n+\tu32 new_stack_id;\n+\tbool has_new_suffix;\n+\n+\tnew_stack_id = trie_side_table_prepare_stack_slot(side_prealloc);\n+\tif (!new_stack_id)\n+\t\treturn 0;\n+\n+\t/* Rebuild the child's run as newly allocated prefix and old suffix nodes. */\n+\trun = child-\u003erun;\n+\trun.nr_entries = matched;\n+\tsplit_prefix_size = trie_node_bytes(\u0026run);\n+\told_suffix_len = child-\u003erun.nr_entries - matched;\n+\trun.nr_entries = old_suffix_len;\n+\told_suffix_size = trie_node_bytes(\u0026run);\n+\thas_new_suffix = matched \u003c nr_entries;\n+\tnr_suffix_roots = has_new_suffix ? 2 : 1;\n+\n+\t/* Reserve fixed split topology before the optional new suffix path. */\n+\tsplit_prefix = trie_pool_alloc(split_prefix_size, pool_prealloc);\n+\tif (!split_prefix)\n+\t\tgoto err_release;\n+\told_suffix = trie_pool_alloc(old_suffix_size, pool_prealloc);\n+\tif (!old_suffix)\n+\t\tgoto err_release;\n+\tnew_children = trie_pool_alloc_children(children-\u003ecapacity, pool_prealloc);\n+\tif (!new_children)\n+\t\tgoto err_release;\n+\tprefix_children = trie_pool_alloc_children(nr_suffix_roots, pool_prealloc);\n+\tif (!prefix_children)\n+\t\tgoto err_release;\n+\n+\tif (has_new_suffix) {\n+\t\tconst struct stack_depot_trie_node *new_suffix;\n+\t\tunsigned long old_suffix_frame;\n+\n+\t\tnew_suffix = trie_path_alloc(split_prefix, new_stack_id,\n+\t\t\t\t\t \u0026entries[matched], nr_entries - matched,\n+\t\t\t\t\t pool_prealloc, \u0026new_node);\n+\t\tif (!new_suffix)\n+\t\t\tgoto err_release;\n+\t\tstack_depot_trie_node_frame(child, matched, \u0026old_suffix_frame);\n+\t\t/* Children remain sorted by the first frame of each suffix. */\n+\t\tif (old_suffix_frame \u003c entries[matched]) {\n+\t\t\tsuffix_roots[0] = old_suffix;\n+\t\t\tsuffix_roots[1] = new_suffix;\n+\t\t} else {\n+\t\t\tsuffix_roots[0] = new_suffix;\n+\t\t\tsuffix_roots[1] = old_suffix;\n+\t\t}\n+\t} else {\n+\t\tnew_node = split_prefix;\n+\t\tsuffix_roots[0] = old_suffix;\n+\t}\n+\n+\t/* Rebuild the old path as prefix -\u003e old suffix and attach suffix roots. */\n+\ttrie_node_init_slice(split_prefix, trie_load_parent(child),\n+\t\t\t has_new_suffix ? 0 : new_stack_id, child, 0, matched);\n+\ttrie_node_init_slice(old_suffix, split_prefix, child-\u003estack_id, child,\n+\t\t\t matched, old_suffix_len);\n+\tfor (i = 0; i \u003c nr_suffix_roots; i++)\n+\t\ttrie_children_insert(prefix_children, suffix_roots[i], i);\n+\tRCU_INIT_POINTER(old_suffix-\u003echildren,\n+\t\t\t trie_load_children(\u0026child-\u003echildren));\n+\tRCU_INIT_POINTER(split_prefix-\u003echildren, prefix_children);\n+\n+\t/* Publish IDs, reparent descendants, then replace and retire topology. */\n+\tif (child-\u003estack_id)\n+\t\ttrie_side_table_publish(old_suffix);\n+\ttrie_side_table_publish(new_node);\n+\t/* Old and replacement chains contain identical frames during transition. */\n+\ttrie_children_init(children, new_children);\n+\tRCU_INIT_POINTER(new_children-\u003enodes[pos], split_prefix);\n+\ttrie_reparent_children(old_suffix);\n+\trcu_assign_pointer(*slot, new_children);\n+\ttrie_retire_children_with_node(children, child);\n+\n+\treturn new_stack_id;\n+\n+err_release:\n+\tif (split_prefix)\n+\t\ttrie_pool_release(split_prefix, split_prefix_size);\n+\tif (old_suffix)\n+\t\ttrie_pool_release(old_suffix, old_suffix_size);\n+\tif (prefix_children)\n+\t\ttrie_pool_release_children(prefix_children);\n+\tif (new_children)\n+\t\ttrie_pool_release_children(new_children);\n+\treturn 0;\n+}\n+\n+static u32\n+trie_promote_child(const struct stack_depot_trie_children __rcu **slot,\n+\t\t const struct stack_depot_trie_children *children,\n+\t\t const struct stack_depot_trie_node *child,\n+\t\t unsigned int pos, void **pool_prealloc,\n+\t\t struct stack_depot_trie_side_prealloc *side_prealloc)\n+{\n+\tstruct stack_depot_trie_children *new_children;\n+\tstruct stack_depot_trie_node *promoted_node;\n+\tsize_t node_size;\n+\tu32 new_stack_id;\n+\n+\tnew_stack_id = trie_side_table_prepare_stack_slot(side_prealloc);\n+\tif (!new_stack_id)\n+\t\treturn 0;\n+\tnode_size = trie_node_bytes(\u0026child-\u003erun);\n+\n+\t/* Reserve a clone and replacement children container before publication. */\n+\tpromoted_node = trie_pool_alloc(node_size, pool_prealloc);\n+\tif (!promoted_node)\n+\t\treturn 0;\n+\tnew_children = trie_pool_alloc_children(children-\u003ecapacity, pool_prealloc);\n+\tif (!new_children)\n+\t\tgoto out_release_node;\n+\n+\t/* Add the stack ID through a clone, then reparent before retirement. */\n+\tmemcpy(promoted_node, child, node_size);\n+\tpromoted_node-\u003estack_id = new_stack_id;\n+\ttrie_side_table_publish(promoted_node);\n+\ttrie_children_init(children, new_children);\n+\tRCU_INIT_POINTER(new_children-\u003enodes[pos], promoted_node);\n+\ttrie_reparent_children(promoted_node);\n+\trcu_assign_pointer(*slot, new_children);\n+\ttrie_retire_children_with_node(children, child);\n+\n+\treturn new_stack_id;\n+\n+out_release_node:\n+\ttrie_pool_release(promoted_node, node_size);\n+\treturn 0;\n+}\n+\n+static u32\n+stack_depot_trie_insert(const unsigned long *entries,\n+\t\t\tunsigned int nr_entries, void **pool_prealloc,\n+\t\t\tstruct stack_depot_trie_side_prealloc *side_prealloc)\n+{\n+\tconst struct stack_depot_trie_children *children;\n+\tconst struct stack_depot_trie_children __rcu **slot =\n+\t\t\u0026stack_depot_trie_root;\n+\tconst struct stack_depot_trie_node *child;\n+\tstruct stack_depot_trie_node *parent = NULL;\n+\tunsigned int matched;\n+\tunsigned int pos;\n+\tu32 stack_id;\n+\n+\tlockdep_assert_held(\u0026stack_depot_trie_writer_lock);\n+\tlockdep_assert_held(\u0026pool_lock);\n+\n+\tfor (;;) {\n+\t\tpos = 0;\n+\t\tchildren = trie_load_children(slot);\n+\t\t/* No matching child: attach the remaining path. */\n+\t\tif (!children ||\n+\t\t !trie_children_find_position(children, entries[0], \u0026pos)) {\n+\t\t\tstack_id = trie_insert_path(slot, parent, children, pos,\n+\t\t\t\t\t\t entries, nr_entries, pool_prealloc,\n+\t\t\t\t\t\t side_prealloc);\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tchild = trie_children_load_child(children, pos);\n+\t\tmatched = trie_node_match(child, entries, nr_entries);\n+\t\t/* A partial child match requires a prefix/suffix split. */\n+\t\tif (matched \u003c child-\u003erun.nr_entries) {\n+\t\t\tstack_id = trie_split_child(slot, children, child, pos,\n+\t\t\t\t\t\t matched, entries, nr_entries,\n+\t\t\t\t\t\t pool_prealloc, side_prealloc);\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\t/* The input ends here: reuse a stack node or promote an internal one. */\n+\t\tif (matched == nr_entries) {\n+\t\t\tif (child-\u003estack_id)\n+\t\t\t\treturn child-\u003estack_id;\n+\t\t\tstack_id = trie_promote_child(slot, children, child, pos,\n+\t\t\t\t\t\t pool_prealloc, side_prealloc);\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\t/* The child matched completely; continue with the remaining frames. */\n+\t\tparent = (struct stack_depot_trie_node *)child;\n+\t\tslot = \u0026parent-\u003echildren;\n+\t\tentries += matched;\n+\t\tnr_entries -= matched;\n+\t}\n+\n+\tif (stack_id)\n+\t\ttrie_side_table_last_stack_id = stack_id;\n+\treturn stack_id;\n+}\n+\n+static unsigned int trie_fetch_into(const struct stack_depot_trie_node *node,\n+\t\t\t\t unsigned long *entries,\n+\t\t\t\t unsigned int max_entries)\n+{\n+\tconst struct stack_depot_trie_node *cur;\n+\tunsigned int total;\n+\tunsigned int pos;\n+\tunsigned int i;\n+\n+\ttotal = 0;\n+\tfor (cur = node; cur; cur = trie_load_parent(cur))\n+\t\ttotal += cur-\u003erun.nr_entries;\n+\tif (max_entries \u003c total)\n+\t\treturn 0;\n+\n+\tpos = total;\n+\tfor (cur = node; cur; cur = trie_load_parent(cur)) {\n+\t\tpos -= cur-\u003erun.nr_entries;\n+\t\tfor (i = 0; i \u003c cur-\u003erun.nr_entries; i++)\n+\t\t\tstack_depot_trie_node_frame(cur, i, \u0026entries[pos + i]);\n+\t}\n+\n+\treturn total;\n+}\n \n-\treturn depot_fetch_stack(handle);\n+static unsigned int trie_fetch_range(const struct stack_depot_trie_node *node,\n+\t\t\t\t unsigned int offset,\n+\t\t\t\t unsigned long *entries,\n+\t\t\t\t unsigned int max_entries)\n+{\n+\tconst struct stack_depot_trie_node *cur;\n+\tunsigned int end;\n+\tunsigned int start;\n+\tunsigned int total;\n+\tunsigned int pos;\n+\tunsigned int i;\n+\n+\ttotal = 0;\n+\tfor (cur = node; cur; cur = trie_load_parent(cur))\n+\t\ttotal += cur-\u003erun.nr_entries;\n+\tif (offset \u003e= total)\n+\t\treturn 0;\n+\n+\tmax_entries = min(max_entries, total - offset);\n+\tend = offset + max_entries;\n+\tpos = total;\n+\tfor (cur = node; cur; cur = trie_load_parent(cur)) {\n+\t\tpos -= cur-\u003erun.nr_entries;\n+\t\tstart = max(pos, offset);\n+\t\tfor (i = start; i \u003c min(pos + cur-\u003erun.nr_entries, end); i++)\n+\t\t\tstack_depot_trie_node_frame(cur, i - pos, \u0026entries[i - offset]);\n+\t}\n+\n+\treturn max_entries;\n+}\n+\n+static unsigned int trie_fetch_handle_into(depot_stack_handle_t handle,\n+\t\t\t\t\t unsigned long *entries,\n+\t\t\t\t\t unsigned int max_entries)\n+{\n+\tconst struct stack_depot_trie_node *node;\n+\tu32 stack_id;\n+\tunsigned int nr_entries;\n+\n+\tstack_id = trie_stack_id(handle);\n+\trcu_read_lock_sched_notrace();\n+\tnode = trie_side_table_lookup(stack_id);\n+\tif (WARN_ONCE(!node, \"corrupt trie handle %08x\\n\", handle)) {\n+\t\trcu_read_unlock_sched_notrace();\n+\t\treturn 0;\n+\t}\n+\tnr_entries = trie_fetch_into(node, entries, max_entries);\n+\trcu_read_unlock_sched_notrace();\n+\tif (nr_entries)\n+\t\tkmsan_unpoison_memory(entries, nr_entries * sizeof(*entries));\n+\n+\treturn nr_entries;\n+}\n+\n+static unsigned int trie_fetch_handle_range(depot_stack_handle_t handle,\n+\t\t\t\t\t unsigned int offset,\n+\t\t\t\t\t unsigned long *entries,\n+\t\t\t\t\t unsigned int max_entries)\n+{\n+\tconst struct stack_depot_trie_node *node;\n+\tu32 stack_id;\n+\tunsigned int nr_entries;\n+\n+\tstack_id = trie_stack_id(handle);\n+\trcu_read_lock_sched_notrace();\n+\tnode = trie_side_table_lookup(stack_id);\n+\tif (WARN_ONCE(!node, \"corrupt trie handle %08x\\n\", handle)) {\n+\t\trcu_read_unlock_sched_notrace();\n+\t\treturn 0;\n+\t}\n+\tnr_entries = trie_fetch_range(node, offset, entries, max_entries);\n+\trcu_read_unlock_sched_notrace();\n+\tif (nr_entries)\n+\t\tkmsan_unpoison_memory(entries, nr_entries * sizeof(*entries));\n+\n+\treturn nr_entries;\n }\n \n unsigned int stack_depot_fetch(depot_stack_handle_t handle,\n@@ -771,6 +2316,8 @@ unsigned int stack_depot_fetch(depot_stack_handle_t handle,\n \n \tif (!handle || stack_depot_disabled)\n \t\treturn 0;\n+\tif (WARN_ON_ONCE(stack_depot_handle_is_trie(handle)))\n+\t\treturn 0;\n \n \tstack = depot_fetch_stack(handle);\n \t/*\n@@ -785,12 +2332,44 @@ unsigned int stack_depot_fetch(depot_stack_handle_t handle,\n }\n EXPORT_SYMBOL_GPL(stack_depot_fetch);\n \n+unsigned int stack_depot_fetch_into(depot_stack_handle_t handle,\n+\t\t\t\t unsigned long *entries,\n+\t\t\t\t unsigned int max_entries)\n+{\n+\tstruct stack_record *stack;\n+\tunsigned int nr_entries;\n+\n+\tif (!handle)\n+\t\treturn 0;\n+\tif (stack_depot_disabled)\n+\t\treturn 0;\n+\tWARN_ON_ONCE(!entries || !max_entries);\n+\tif (stack_depot_handle_is_trie(handle))\n+\t\treturn trie_fetch_handle_into(handle, entries, max_entries);\n+\n+\tstack = depot_fetch_stack(handle);\n+\tif (!stack)\n+\t\treturn 0;\n+\tnr_entries = stack-\u003esize;\n+\tif (WARN_ON_ONCE(!nr_entries))\n+\t\treturn 0;\n+\tif (nr_entries \u003e max_entries)\n+\t\treturn 0;\n+\n+\tmemcpy(entries, stack-\u003eentries, nr_entries * sizeof(*entries));\n+\tkmsan_unpoison_memory(entries, nr_entries * sizeof(*entries));\n+\treturn nr_entries;\n+}\n+EXPORT_SYMBOL_GPL(stack_depot_fetch_into);\n+\n void stack_depot_put(depot_stack_handle_t handle)\n {\n \tstruct stack_record *stack;\n \n \tif (!handle || stack_depot_disabled)\n \t\treturn;\n+\tif (WARN_ON_ONCE(stack_depot_handle_is_trie(handle)))\n+\t\treturn;\n \n \tstack = depot_fetch_stack(handle);\n \t/*\n@@ -800,16 +2379,60 @@ void stack_depot_put(depot_stack_handle_t handle)\n \tif (WARN(!stack, \"corrupt handle or unbalanced stack_depot_put()\"))\n \t\treturn;\n \n+\tif (WARN_ON_ONCE(stack-\u003eflags \u0026 STACK_DEPOT_FLAG_COUNTABLE))\n+\t\treturn;\n \tif (refcount_dec_and_test(\u0026stack-\u003ecount))\n \t\tdepot_free_stack(stack);\n }\n EXPORT_SYMBOL_GPL(stack_depot_put);\n \n+static void trie_print(depot_stack_handle_t handle)\n+{\n+\tunsigned long entries[STACK_DEPOT_PRINT_CHUNK_FRAMES];\n+\tunsigned int nr_entries;\n+\tunsigned int offset = 0;\n+\n+\twhile ((nr_entries = trie_fetch_handle_range(handle, offset, entries,\n+\t\t\t\t\t\t ARRAY_SIZE(entries)))) {\n+\t\tstack_trace_print(entries, nr_entries, 0);\n+\t\toffset += nr_entries;\n+\t}\n+}\n+\n+static int trie_snprint(depot_stack_handle_t handle, char *buf, size_t size,\n+\t\t\tint spaces)\n+{\n+\tunsigned long entries[STACK_DEPOT_PRINT_CHUNK_FRAMES];\n+\tunsigned int generated;\n+\tunsigned int nr_entries;\n+\tunsigned int offset = 0;\n+\tunsigned int total = 0;\n+\n+\twhile (size \u0026\u0026\n+\t (nr_entries = trie_fetch_handle_range(handle, offset, entries,\n+\t\t\t\t\t\t ARRAY_SIZE(entries)))) {\n+\t\tgenerated = stack_trace_snprint(buf, size, entries, nr_entries, spaces);\n+\t\ttotal += generated;\n+\t\tif (generated \u003e= size)\n+\t\t\tbreak;\n+\t\tbuf += generated;\n+\t\tsize -= generated;\n+\t\toffset += nr_entries;\n+\t}\n+\n+\treturn total;\n+}\n+\n void stack_depot_print(depot_stack_handle_t stack)\n {\n \tunsigned long *entries;\n \tunsigned int nr_entries;\n \n+\tif (stack_depot_handle_is_trie(stack)) {\n+\t\ttrie_print(stack);\n+\t\treturn;\n+\t}\n+\n \tnr_entries = stack_depot_fetch(stack, \u0026entries);\n \tif (nr_entries \u003e 0)\n \t\tstack_trace_print(entries, nr_entries, 0);\n@@ -822,6 +2445,9 @@ int stack_depot_snprint(depot_stack_handle_t handle, char *buf, size_t size,\n \tunsigned long *entries;\n \tunsigned int nr_entries;\n \n+\tif (stack_depot_handle_is_trie(handle))\n+\t\treturn trie_snprint(handle, buf, size, spaces);\n+\n \tnr_entries = stack_depot_fetch(handle, \u0026entries);\n \treturn nr_entries ? stack_trace_snprint(buf, size, entries, nr_entries,\n \t\t\t\t\t\tspaces) : 0;\ndiff --git a/lib/tests/Makefile b/lib/tests/Makefile\nindex 3cac3b63a7522..1f72191f98bbc 100644\n--- a/lib/tests/Makefile\n+++ b/lib/tests/Makefile\n@@ -48,6 +48,7 @@ obj-$(CONFIG_SCANF_KUNIT_TEST) += scanf_kunit.o\n obj-$(CONFIG_SEQ_BUF_KUNIT_TEST) += seq_buf_kunit.o\n obj-$(CONFIG_SIPHASH_KUNIT_TEST) += siphash_kunit.o\n obj-$(CONFIG_SLUB_KUNIT_TEST) += slub_kunit.o\n+obj-$(CONFIG_STACKDEPOT_KUNIT_TEST) += stackdepot_kunit.o\n obj-$(CONFIG_TEST_SORT) += test_sort.o\n CFLAGS_stackinit_kunit.o += $(call cc-disable-warning, switch-unreachable)\n obj-$(CONFIG_STACKINIT_KUNIT_TEST) += stackinit_kunit.o\ndiff --git a/lib/tests/stackdepot_kunit.c b/lib/tests/stackdepot_kunit.c\nnew file mode 100644\nindex 0000000000000..b86b84d56176e\n--- /dev/null\n+++ b/lib/tests/stackdepot_kunit.c\n@@ -0,0 +1,582 @@\n+// SPDX-License-Identifier: GPL-2.0-only\n+\n+#include \u003ckunit/test.h\u003e\n+#include \u003clinux/array_size.h\u003e\n+#include \u003clinux/gfp.h\u003e\n+#include \u003clinux/kallsyms.h\u003e\n+#include \u003clinux/limits.h\u003e\n+#include \u003clinux/moduleparam.h\u003e\n+#include \u003clinux/stackdepot.h\u003e\n+#include \u003clinux/stacktrace.h\u003e\n+#include \u003clinux/string.h\u003e\n+\n+#include \u003casm/stackdepot.h\u003e\n+\n+static int expected_trie_pool_limit = -1;\n+module_param_named(trie_pool_limit, expected_trie_pool_limit, int, 0);\n+MODULE_PARM_DESC(trie_pool_limit, \"Expected stackdepot hash/trie pool split\");\n+\n+#ifdef CONFIG_ARM64\n+#include \u003casm/sections.h\u003e\n+\n+static inline unsigned long stackdepot_arm64_frame(long offset)\n+{\n+\treturn (unsigned long)((long)_text + offset);\n+}\n+#endif\n+\n+static unsigned long stackdepot_test_frame(unsigned int i)\n+{\n+#ifdef CONFIG_ARM64\n+\treturn i \u0026 1 ? 0x1000UL + i * 0x1000UL :\n+\t\tstackdepot_arm64_frame(i * 4);\n+#elif defined(CONFIG_X86_64) \u0026\u0026 !defined(CONFIG_UML)\n+\treturn i \u0026 1 ? 0xffff888000000000UL + i * 0x1000UL :\n+\t\t0xffffffff10000000UL + i * 0x10UL;\n+#else\n+\treturn 0x1000UL + i * 0x1000UL;\n+#endif\n+}\n+\n+static void stackdepot_trie_max_path_roundtrip(struct kunit *test)\n+{\n+\tunion handle_parts parts;\n+\tunsigned long *entries;\n+\tunsigned long *fetched;\n+\tdepot_stack_handle_t handle;\n+\tsize_t size = CONFIG_STACKDEPOT_MAX_FRAMES * sizeof(*entries);\n+\tu32 pool_index_plus_1;\n+\tunsigned int i;\n+\n+\tif (expected_trie_pool_limit \u003c 0)\n+\t\tkunit_skip(test, \"trie pool limit was not provided\");\n+\tKUNIT_ASSERT_EQ(test, stack_depot_init(), 0);\n+\tentries = kunit_kcalloc(test, CONFIG_STACKDEPOT_MAX_FRAMES,\n+\t\t\t\tsizeof(*entries), GFP_KERNEL);\n+\tKUNIT_ASSERT_NOT_NULL(test, entries);\n+\tfetched = kunit_kcalloc(test, CONFIG_STACKDEPOT_MAX_FRAMES,\n+\t\t\t\tsizeof(*fetched), GFP_KERNEL);\n+\tKUNIT_ASSERT_NOT_NULL(test, fetched);\n+\tfor (i = 0; i \u003c CONFIG_STACKDEPOT_MAX_FRAMES; i++)\n+\t\tentries[i] = stackdepot_test_frame(i);\n+\n+\thandle = stack_depot_save(entries, CONFIG_STACKDEPOT_MAX_FRAMES,\n+\t\t\t\t GFP_KERNEL);\n+\tKUNIT_ASSERT_NE(test, handle, (depot_stack_handle_t)0);\n+\tparts.handle = handle;\n+\tpool_index_plus_1 = parts.pool_index_plus_1;\n+\tKUNIT_EXPECT_GT(test, pool_index_plus_1, (u32)expected_trie_pool_limit);\n+\tKUNIT_EXPECT_EQ(test,\n+\t\t\tstack_depot_fetch_into(handle, fetched,\n+\t\t\t\t\t CONFIG_STACKDEPOT_MAX_FRAMES),\n+\t\t\t(unsigned int)CONFIG_STACKDEPOT_MAX_FRAMES);\n+\tKUNIT_EXPECT_MEMEQ(test, fetched, entries, size);\n+\tKUNIT_EXPECT_EQ(test,\n+\t\t\tstack_depot_save(entries, CONFIG_STACKDEPOT_MAX_FRAMES,\n+\t\t\t\t\t GFP_KERNEL),\n+\t\t\thandle);\n+}\n+\n+static void stackdepot_save_flags_public(struct kunit *test)\n+{\n+\tunion handle_parts parts;\n+\tunsigned long entries[] = { 0x501000UL, 0x502000UL, 0x503000UL };\n+\tunsigned long get_entries[] = { 0x601000UL, 0x602000UL };\n+\tunsigned long missing_entries[] = { 0x701000UL, 0x702000UL };\n+\tunsigned long blocking_entries[] = { 0x711000UL, 0x712000UL };\n+\tunsigned long fetched[ARRAY_SIZE(entries)] = {};\n+\tdepot_stack_handle_t blocking_handle;\n+\tdepot_stack_handle_t noalloc_handle;\n+\tdepot_stack_handle_t overlong_handle;\n+\tdepot_stack_handle_t plain_handle;\n+\tdepot_stack_handle_t get_handle;\n+\tdepot_stack_handle_t again;\n+\tdepot_stack_handle_t extra;\n+\tgfp_t no_spin = GFP_NOWAIT \u0026 ~__GFP_RECLAIM;\n+\tu32 pool_index_plus_1;\n+\tunsigned long *overlong_fetched;\n+\tunsigned long *overlong_entries;\n+\tunsigned int overlong_nr = CONFIG_STACKDEPOT_MAX_FRAMES + 1;\n+\tunsigned int nr_entries;\n+\tsize_t overlong_size;\n+\tunsigned int i;\n+\n+\tKUNIT_ASSERT_EQ(test, stack_depot_init(), 0);\n+\toverlong_entries = kunit_kcalloc(test, overlong_nr,\n+\t\t\t\t\t sizeof(*overlong_entries), GFP_KERNEL);\n+\tKUNIT_ASSERT_NOT_NULL(test, overlong_entries);\n+\toverlong_fetched = kunit_kcalloc(test, CONFIG_STACKDEPOT_MAX_FRAMES,\n+\t\t\t\t\t sizeof(*overlong_fetched), GFP_KERNEL);\n+\tKUNIT_ASSERT_NOT_NULL(test, overlong_fetched);\n+\tfor (i = 0; i \u003c overlong_nr; i++)\n+\t\toverlong_entries[i] = 0x800000UL + i * 0x1000UL;\n+\n+\tplain_handle = stack_depot_save(entries, ARRAY_SIZE(entries), GFP_KERNEL);\n+\tKUNIT_ASSERT_NE(test, plain_handle, (depot_stack_handle_t)0);\n+\tagain = stack_depot_save(entries, ARRAY_SIZE(entries), GFP_KERNEL);\n+\tKUNIT_EXPECT_EQ(test, again, plain_handle);\n+\n+\tnr_entries = stack_depot_fetch_into(plain_handle, fetched,\n+\t\t\t\t\t ARRAY_SIZE(fetched));\n+\tKUNIT_EXPECT_EQ(test, nr_entries, (unsigned int)ARRAY_SIZE(entries));\n+\tKUNIT_EXPECT_MEMEQ(test, fetched, entries, sizeof(entries));\n+\n+\tnoalloc_handle = stack_depot_save_flags(entries, ARRAY_SIZE(entries), no_spin, 0);\n+\tKUNIT_EXPECT_EQ(test, noalloc_handle, plain_handle);\n+\tnoalloc_handle = stack_depot_save_flags(missing_entries,\n+\t\t\t\t\t\tARRAY_SIZE(missing_entries),\n+\t\t\t\t\t\tGFP_KERNEL, 0);\n+\tKUNIT_ASSERT_NE(test, noalloc_handle, (depot_stack_handle_t)0);\n+\tif (expected_trie_pool_limit \u003e= 0) {\n+\t\tparts.handle = noalloc_handle;\n+\t\tpool_index_plus_1 = parts.pool_index_plus_1;\n+\t\tKUNIT_EXPECT_GT(test, pool_index_plus_1,\n+\t\t\t\t(u32)expected_trie_pool_limit);\n+\t}\n+\tnr_entries = stack_depot_fetch_into(noalloc_handle, fetched,\n+\t\t\t\t\t ARRAY_SIZE(fetched));\n+\tKUNIT_EXPECT_EQ(test, nr_entries,\n+\t\t\t(unsigned int)ARRAY_SIZE(missing_entries));\n+\tKUNIT_EXPECT_MEMEQ(test, fetched, missing_entries, sizeof(missing_entries));\n+\tKUNIT_EXPECT_EQ(test,\n+\t\t\tstack_depot_save_flags(missing_entries,\n+\t\t\t\t\t ARRAY_SIZE(missing_entries),\n+\t\t\t\t\t no_spin, 0),\n+\t\t\tnoalloc_handle);\n+\n+\tblocking_handle = stack_depot_save_flags(blocking_entries,\n+\t\t\t\t\t\t ARRAY_SIZE(blocking_entries),\n+\t\t\t\t\t\t GFP_KERNEL, 0);\n+\tKUNIT_ASSERT_NE(test, blocking_handle, (depot_stack_handle_t)0);\n+\tif (expected_trie_pool_limit \u003e= 0) {\n+\t\tparts.handle = blocking_handle;\n+\t\tpool_index_plus_1 = parts.pool_index_plus_1;\n+\t\tKUNIT_EXPECT_GT(test, pool_index_plus_1,\n+\t\t\t\t(u32)expected_trie_pool_limit);\n+\t}\n+\tmemset(fetched, 0, sizeof(fetched));\n+\tnr_entries = stack_depot_fetch_into(blocking_handle, fetched,\n+\t\t\t\t\t ARRAY_SIZE(fetched));\n+\tKUNIT_EXPECT_EQ(test, nr_entries,\n+\t\t\t(unsigned int)ARRAY_SIZE(blocking_entries));\n+\tKUNIT_EXPECT_MEMEQ(test, fetched, blocking_entries,\n+\t\t\t sizeof(blocking_entries));\n+\n+\tget_handle = stack_depot_save_flags(get_entries, ARRAY_SIZE(get_entries),\n+\t\t\t\t\t GFP_KERNEL,\n+\t\t\t\t\t STACK_DEPOT_FLAG_CAN_ALLOC |\n+\t\t\t\t\t STACK_DEPOT_FLAG_GET);\n+\tKUNIT_ASSERT_NE(test, get_handle, (depot_stack_handle_t)0);\n+\tstack_depot_put(get_handle);\n+\n+\toverlong_handle = stack_depot_save(overlong_entries, overlong_nr,\n+\t\t\t\t\t GFP_KERNEL);\n+\tKUNIT_ASSERT_NE(test, overlong_handle, (depot_stack_handle_t)0);\n+\tnr_entries = stack_depot_fetch_into(overlong_handle, overlong_fetched,\n+\t\t\t\t\t CONFIG_STACKDEPOT_MAX_FRAMES);\n+\tKUNIT_EXPECT_EQ(test, nr_entries, (unsigned int)CONFIG_STACKDEPOT_MAX_FRAMES);\n+\toverlong_size = CONFIG_STACKDEPOT_MAX_FRAMES * sizeof(*overlong_entries);\n+\tKUNIT_EXPECT_MEMEQ(test, overlong_fetched, overlong_entries, overlong_size);\n+\n+\textra = stack_depot_set_extra_bits(plain_handle, 7);\n+\tKUNIT_ASSERT_NE(test, extra, (depot_stack_handle_t)0);\n+\tKUNIT_EXPECT_EQ(test, stack_depot_get_extra_bits(extra), 7U);\n+\tmemset(fetched, 0, sizeof(fetched));\n+\tnr_entries = stack_depot_fetch_into(extra, fetched, ARRAY_SIZE(fetched));\n+\tKUNIT_EXPECT_EQ(test, nr_entries, (unsigned int)ARRAY_SIZE(entries));\n+\tKUNIT_EXPECT_MEMEQ(test, fetched, entries, sizeof(entries));\n+}\n+\n+static void stackdepot_snprint_public(struct kunit *test)\n+{\n+\tconst unsigned int nr_entries = CONFIG_STACKDEPOT_MAX_FRAMES;\n+\tconst size_t buf_size = nr_entries * (KSYM_SYMBOL_LEN + 4);\n+\tunsigned long *entries;\n+\tchar *expected;\n+\tchar *actual;\n+\tdepot_stack_handle_t handle;\n+\tunsigned int expected_len;\n+\tunsigned int prefix_entries;\n+\tunsigned int prefix_len;\n+\tsize_t output_size;\n+\tunsigned int i;\n+\tint actual_len;\n+\n+\tKUNIT_ASSERT_EQ(test, stack_depot_init(), 0);\n+\tentries = kunit_kmalloc_array(test, nr_entries, sizeof(*entries),\n+\t\t\t\t GFP_KERNEL);\n+\tKUNIT_ASSERT_NOT_NULL(test, entries);\n+\texpected = kunit_kzalloc(test, buf_size, GFP_KERNEL);\n+\tKUNIT_ASSERT_NOT_NULL(test, expected);\n+\tactual = kunit_kzalloc(test, buf_size, GFP_KERNEL);\n+\tKUNIT_ASSERT_NOT_NULL(test, actual);\n+\tfor (i = 0; i \u003c nr_entries; i++)\n+\t\tentries[i] = stackdepot_test_frame(i);\n+\n+\thandle = stack_depot_save(entries, nr_entries, GFP_KERNEL);\n+\tKUNIT_ASSERT_NE(test, handle, (depot_stack_handle_t)0);\n+\tif (expected_trie_pool_limit \u003e= 0) {\n+\t\tunion handle_parts parts = { .handle = handle };\n+\n+\t\tKUNIT_EXPECT_GT(test, (u32)parts.pool_index_plus_1,\n+\t\t\t\t(u32)expected_trie_pool_limit);\n+\t}\n+\texpected_len = stack_trace_snprint(expected, buf_size, entries,\n+\t\t\t\t\t nr_entries, 2);\n+\tactual_len = stack_depot_snprint(handle, actual, buf_size, 2);\n+\tKUNIT_EXPECT_EQ(test, actual_len, (int)expected_len);\n+\tKUNIT_EXPECT_STREQ(test, actual, expected);\n+\n+\tprefix_entries = nr_entries / 2 + 1;\n+\tprefix_len = stack_trace_snprint(expected, buf_size, entries,\n+\t\t\t\t\t prefix_entries, 2);\n+\tKUNIT_ASSERT_LE(test, (size_t)prefix_len + 2, buf_size);\n+\toutput_size = prefix_len + 2;\n+\tmemset(expected, 0, buf_size);\n+\tmemset(actual, 0, buf_size);\n+\texpected_len = stack_trace_snprint(expected, output_size, entries,\n+\t\t\t\t\t nr_entries, 2);\n+\tactual_len = stack_depot_snprint(handle, actual, output_size, 2);\n+\tKUNIT_EXPECT_EQ(test, actual_len, (int)expected_len);\n+\tKUNIT_EXPECT_STREQ(test, actual, expected);\n+}\n+\n+static void stackdepot_countable_public(struct kunit *test)\n+{\n+\tunsigned long plain_entries[] = {\n+\t\t0x141000UL,\n+\t\t0x142000UL,\n+\t\t0x143000UL,\n+\t};\n+\tunsigned long get_entries[] = {\n+\t\t0x151000UL,\n+\t\t0x152000UL,\n+\t\t0x153000UL,\n+\t};\n+\tunsigned long fetched[ARRAY_SIZE(plain_entries)] = {};\n+\tdepot_flags_t countable = STACK_DEPOT_FLAG_CAN_ALLOC |\n+\t\t\t\t STACK_DEPOT_FLAG_COUNTABLE;\n+\tstruct stack_record *record;\n+\tdepot_stack_handle_t count_handle;\n+\tdepot_stack_handle_t plain_handle;\n+\tdepot_stack_handle_t get_handle;\n+\tunsigned int get_nr = ARRAY_SIZE(get_entries);\n+\tunsigned int plain_nr = ARRAY_SIZE(plain_entries);\n+\tunsigned int nr_entries;\n+\n+\tKUNIT_ASSERT_EQ(test, stack_depot_init(), 0);\n+\n+\tplain_handle = stack_depot_save(plain_entries, plain_nr, GFP_KERNEL);\n+\tKUNIT_ASSERT_NE(test, plain_handle, (depot_stack_handle_t)0);\n+\tcount_handle = stack_depot_save_flags(plain_entries, plain_nr, GFP_KERNEL,\n+\t\t\t\t\t countable);\n+\tKUNIT_ASSERT_NE(test, count_handle, (depot_stack_handle_t)0);\n+\trecord = __stack_depot_get_stack_record(count_handle);\n+\tKUNIT_ASSERT_NOT_NULL(test, record);\n+\tKUNIT_EXPECT_EQ(test, record-\u003esize, (u16)plain_nr);\n+\tKUNIT_EXPECT_MEMEQ(test, record-\u003eentries, plain_entries,\n+\t\t\t sizeof(plain_entries));\n+\tnr_entries = stack_depot_fetch_into(count_handle, fetched,\n+\t\t\t\t\t ARRAY_SIZE(fetched));\n+\tKUNIT_EXPECT_EQ(test, nr_entries, plain_nr);\n+\tKUNIT_EXPECT_MEMEQ(test, fetched, plain_entries, sizeof(plain_entries));\n+\n+\tget_handle = stack_depot_save_flags(get_entries, get_nr, GFP_KERNEL,\n+\t\t\t\t\t STACK_DEPOT_FLAG_CAN_ALLOC |\n+\t\t\t\t\t STACK_DEPOT_FLAG_GET);\n+\tKUNIT_ASSERT_NE(test, get_handle, (depot_stack_handle_t)0);\n+\tcount_handle = stack_depot_save_flags(get_entries, get_nr, GFP_KERNEL,\n+\t\t\t\t\t countable);\n+\tKUNIT_ASSERT_NE(test, count_handle, (depot_stack_handle_t)0);\n+\trecord = __stack_depot_get_stack_record(count_handle);\n+\tKUNIT_ASSERT_NOT_NULL(test, record);\n+\tKUNIT_EXPECT_MEMEQ(test, record-\u003eentries, get_entries, sizeof(get_entries));\n+\n+\tstack_depot_put(get_handle);\n+}\n+\n+static void stackdepot_fetch_into_roundtrip(struct kunit *test)\n+{\n+\tunsigned long entries[] = {\n+\t\t0x101000UL,\n+\t\t0x102000UL,\n+\t\t0x103000UL,\n+\t};\n+\tunsigned long exact[ARRAY_SIZE(entries)] = {};\n+\tunsigned long fetched[ARRAY_SIZE(entries) + 1] = {\n+\t\t[ARRAY_SIZE(entries)] = 0xa5a5a5a5UL,\n+\t};\n+\tunsigned long expected_tail = fetched[ARRAY_SIZE(entries)];\n+\tdepot_stack_handle_t handle;\n+\tunsigned int nr_entries;\n+\n+\tKUNIT_ASSERT_EQ(test, stack_depot_init(), 0);\n+\n+\thandle = stack_depot_save(entries, ARRAY_SIZE(entries), GFP_KERNEL);\n+\tKUNIT_ASSERT_NE(test, handle, (depot_stack_handle_t)0);\n+\n+\tnr_entries = stack_depot_fetch_into(handle, exact, ARRAY_SIZE(exact));\n+\tKUNIT_EXPECT_EQ(test, nr_entries, (unsigned int)ARRAY_SIZE(entries));\n+\tKUNIT_EXPECT_MEMEQ(test, exact, entries, sizeof(entries));\n+\n+\tnr_entries = stack_depot_fetch_into(handle, fetched, ARRAY_SIZE(fetched));\n+\tKUNIT_EXPECT_EQ(test, nr_entries, (unsigned int)ARRAY_SIZE(entries));\n+\tKUNIT_EXPECT_MEMEQ(test, fetched, entries, sizeof(entries));\n+\tKUNIT_EXPECT_EQ(test, fetched[ARRAY_SIZE(entries)], expected_tail);\n+}\n+\n+static void stackdepot_fetch_into_rejects_missing_or_short_stack(struct kunit *test)\n+{\n+\tunsigned long entries[] = {\n+\t\t0x111000UL,\n+\t\t0x112000UL,\n+\t\t0x113000UL,\n+\t};\n+\tunsigned long fetched[ARRAY_SIZE(entries)] = {\n+\t\t0xa1a1a1a1UL,\n+\t\t0xb2b2b2b2UL,\n+\t\t0xc3c3c3c3UL,\n+\t};\n+\tunsigned long expected[ARRAY_SIZE(fetched)];\n+\tdepot_stack_handle_t handle;\n+\tunsigned int nr_entries;\n+\n+\tKUNIT_ASSERT_EQ(test, stack_depot_init(), 0);\n+\n+\thandle = stack_depot_save(entries, ARRAY_SIZE(entries), GFP_KERNEL);\n+\tKUNIT_ASSERT_NE(test, handle, (depot_stack_handle_t)0);\n+\tmemcpy(expected, fetched, sizeof(expected));\n+\n+\tnr_entries = stack_depot_fetch_into(0, fetched, ARRAY_SIZE(fetched));\n+\tKUNIT_EXPECT_EQ(test, nr_entries, 0U);\n+\tKUNIT_EXPECT_MEMEQ(test, fetched, expected, sizeof(expected));\n+\n+\tnr_entries = stack_depot_fetch_into(0, NULL, 0);\n+\tKUNIT_EXPECT_EQ(test, nr_entries, 0U);\n+\n+\tnr_entries = stack_depot_fetch_into(handle, fetched,\n+\t\t\t\t\t ARRAY_SIZE(fetched) - 1);\n+\tKUNIT_EXPECT_EQ(test, nr_entries, 0U);\n+\tKUNIT_EXPECT_MEMEQ(test, fetched, expected, sizeof(expected));\n+}\n+\n+static void stackdepot_trie_topology_roundtrip(struct kunit *test,\n+\t\t\t\t\t bool constrained)\n+{\n+\tunion handle_parts parts;\n+\tunsigned long seed[] = { 0x191000UL, 0x192000UL };\n+\tunsigned long stacks[][3] = {\n+\t\t{ 0x201000UL, 0x202000UL },\n+\t\t{ 0x201000UL, 0x203000UL },\n+\t\t{ 0x201000UL },\n+\t\t{ 0x201000UL, 0x203000UL, 0x204000UL },\n+\t\t{ 0x201000UL, 0x205000UL },\n+\t\t{ 0x201000UL, 0x204000UL },\n+\t\t{ 0x201000UL, 0x206000UL },\n+\t\t{ 0x201000UL, 0x207000UL },\n+\t\t{ 0x301000UL, 0x302000UL },\n+\t\t{ 0x301000UL, 0x302000UL, 0x303000UL },\n+\t\t{ 0x301000UL, 0x304000UL },\n+\t\t{ 0x401000UL, 0x402000UL, 0x403000UL },\n+\t\t{ 0x401000UL, 0x402000UL },\n+\t};\n+\tunsigned int nr_entries[] = { 2, 2, 1, 3, 2, 2, 2, 2, 2, 3, 2, 3, 2 };\n+\tdepot_stack_handle_t handles[ARRAY_SIZE(stacks)];\n+\tdepot_stack_handle_t seed_handle;\n+\tunsigned long fetched[ARRAY_SIZE(stacks[0])];\n+\tgfp_t no_spin = GFP_NOWAIT \u0026 ~__GFP_RECLAIM;\n+\tu32 pool_index_plus_1;\n+\tunsigned int j;\n+\tunsigned int i;\n+\n+\tif (expected_trie_pool_limit \u003c 0)\n+\t\tkunit_skip(test, \"trie pool limit was not provided\");\n+\tKUNIT_ASSERT_EQ(test, stack_depot_init(), 0);\n+\tif (constrained) {\n+\t\tseed_handle = stack_depot_save(seed, ARRAY_SIZE(seed), GFP_KERNEL);\n+\t\tKUNIT_ASSERT_NE(test, seed_handle, (depot_stack_handle_t)0);\n+\t\tfor (i = 0; i \u003c ARRAY_SIZE(stacks); i++)\n+\t\t\tfor (j = 0; j \u003c nr_entries[i]; j++)\n+\t\t\t\tstacks[i][j] += 0x10000000UL;\n+\t}\n+\n+\tfor (i = 0; i \u003c ARRAY_SIZE(stacks); i++) {\n+\t\tif (constrained)\n+\t\t\thandles[i] = stack_depot_save_flags(stacks[i], nr_entries[i],\n+\t\t\t\t\t\t\t GFP_KERNEL, 0);\n+\t\telse\n+\t\t\thandles[i] = stack_depot_save(stacks[i], nr_entries[i],\n+\t\t\t\t\t\t GFP_KERNEL);\n+\t\tKUNIT_ASSERT_NE(test, handles[i], (depot_stack_handle_t)0);\n+\t}\n+\tparts.handle = handles[0];\n+\tpool_index_plus_1 = parts.pool_index_plus_1;\n+\tKUNIT_ASSERT_GT(test, pool_index_plus_1,\n+\t\t\t(u32)expected_trie_pool_limit);\n+\n+\tfor (i = 0; i \u003c ARRAY_SIZE(stacks); i++) {\n+\t\tmemset(fetched, 0, sizeof(fetched));\n+\t\tKUNIT_EXPECT_EQ(test,\n+\t\t\t\tstack_depot_fetch_into(handles[i], fetched,\n+\t\t\t\t\t\t ARRAY_SIZE(fetched)),\n+\t\t\t\tnr_entries[i]);\n+\t\tKUNIT_EXPECT_MEMEQ(test, fetched, stacks[i],\n+\t\t\t\t nr_entries[i] * sizeof(fetched[0]));\n+\t\tif (constrained)\n+\t\t\tKUNIT_EXPECT_EQ(test,\n+\t\t\t\t\tstack_depot_save_flags(stacks[i], nr_entries[i],\n+\t\t\t\t\t\t\t no_spin, 0),\n+\t\t\t\t\thandles[i]);\n+\t\telse\n+\t\t\tKUNIT_EXPECT_EQ(test,\n+\t\t\t\t\tstack_depot_save(stacks[i], nr_entries[i],\n+\t\t\t\t\t\t\t GFP_KERNEL),\n+\t\t\t\t\thandles[i]);\n+\t}\n+}\n+\n+static void stackdepot_trie_topology_allocating(struct kunit *test)\n+{\n+\tstackdepot_trie_topology_roundtrip(test, false);\n+}\n+\n+static void stackdepot_trie_topology_constrained(struct kunit *test)\n+{\n+\tstackdepot_trie_topology_roundtrip(test, true);\n+}\n+\n+static void stackdepot_frame_storage_roundtrip(struct kunit *test)\n+{\n+\tunion handle_parts parts;\n+\tunsigned long fetched[3] = {};\n+\tdepot_stack_handle_t handle;\n+\tu32 pool_index_plus_1;\n+\tunsigned int nr_entries;\n+#if defined(CONFIG_ARM64)\n+\tunsigned long entries[] = {\n+\t\tstackdepot_arm64_frame(S32_MIN),\n+\t\t0x1000UL,\n+\t\tstackdepot_arm64_frame(S32_MAX),\n+\t};\n+#elif defined(CONFIG_X86_64)\n+\tunsigned long entries[] = {\n+\t\t0xffffffff10001000UL,\n+\t\t0xffff888000001000UL,\n+\t\t0xffffffff20002000UL,\n+\t};\n+#else\n+\tunsigned long entries[] = { 0x301000UL, 0x302000UL, 0x303000UL };\n+#endif\n+\n+\tif (expected_trie_pool_limit \u003c 0)\n+\t\tkunit_skip(test, \"trie pool limit was not provided\");\n+\tKUNIT_ASSERT_EQ(test, stack_depot_init(), 0);\n+\thandle = stack_depot_save(entries, ARRAY_SIZE(entries), GFP_KERNEL);\n+\tKUNIT_ASSERT_NE(test, handle, (depot_stack_handle_t)0);\n+\tparts.handle = handle;\n+\tpool_index_plus_1 = parts.pool_index_plus_1;\n+\tKUNIT_ASSERT_GT(test, pool_index_plus_1,\n+\t\t\t(u32)expected_trie_pool_limit);\n+\n+\tnr_entries = stack_depot_fetch_into(handle, fetched, ARRAY_SIZE(fetched));\n+\tKUNIT_EXPECT_EQ(test, nr_entries, (unsigned int)ARRAY_SIZE(entries));\n+\tKUNIT_EXPECT_MEMEQ(test, fetched, entries, sizeof(entries));\n+}\n+\n+static void stackdepot_frame_raw_fallback(struct kunit *test)\n+{\n+\tunsigned long frame = 0x1000UL;\n+\tbool compressed;\n+\tu32 payload;\n+\n+#ifdef CONFIG_ARM64\n+\tframe = (unsigned long)_text + (unsigned long)S32_MAX + 1UL;\n+#endif\n+\n+\tcompressed = arch_stack_depot_frame_try_compress(frame, \u0026payload);\n+\tKUNIT_EXPECT_FALSE(test, compressed);\n+}\n+\n+#if defined(CONFIG_X86_64) \u0026\u0026 !defined(CONFIG_UML)\n+static void stackdepot_frame_x86_64(struct kunit *test)\n+{\n+\tunsigned long direct_map = 0xffff888000001000UL;\n+\tunsigned long frame = 0xffffffff81234567UL;\n+\tunsigned long out;\n+\tbool compressed;\n+\tu32 low;\n+\n+\tcompressed = arch_stack_depot_frame_try_compress(frame, \u0026low);\n+\tKUNIT_EXPECT_TRUE(test, compressed);\n+\tKUNIT_EXPECT_EQ(test, low, (u32)0x81234567);\n+\tarch_stack_depot_frame_decompress(low, \u0026out);\n+\tKUNIT_EXPECT_EQ(test, out, frame);\n+\n+\tcompressed = arch_stack_depot_frame_try_compress(direct_map, \u0026low);\n+\tKUNIT_EXPECT_FALSE(test, compressed);\n+}\n+#endif /* CONFIG_X86_64 \u0026\u0026 !CONFIG_UML */\n+\n+#ifdef CONFIG_ARM64\n+static void stackdepot_frame_arm64(struct kunit *test)\n+{\n+\tlong negative_offset = S32_MIN;\n+\tlong positive_offset = S32_MAX;\n+\tlong offset = 0x123456;\n+\tunsigned long frame = stackdepot_arm64_frame(offset);\n+\tunsigned long out;\n+\tbool compressed;\n+\tu32 payload;\n+\n+\tcompressed = arch_stack_depot_frame_try_compress(frame, \u0026payload);\n+\tKUNIT_EXPECT_TRUE(test, compressed);\n+\tKUNIT_EXPECT_EQ(test, payload, (u32)(s32)offset);\n+\tarch_stack_depot_frame_decompress(payload, \u0026out);\n+\tKUNIT_EXPECT_EQ(test, out, frame);\n+\n+\tframe = stackdepot_arm64_frame(negative_offset);\n+\tcompressed = arch_stack_depot_frame_try_compress(frame, \u0026payload);\n+\tKUNIT_EXPECT_TRUE(test, compressed);\n+\tKUNIT_EXPECT_EQ(test, payload, (u32)(s32)negative_offset);\n+\tarch_stack_depot_frame_decompress(payload, \u0026out);\n+\tKUNIT_EXPECT_EQ(test, out, frame);\n+\n+\tframe = stackdepot_arm64_frame(positive_offset);\n+\tcompressed = arch_stack_depot_frame_try_compress(frame, \u0026payload);\n+\tKUNIT_EXPECT_TRUE(test, compressed);\n+\tKUNIT_EXPECT_EQ(test, payload, (u32)(s32)positive_offset);\n+\tarch_stack_depot_frame_decompress(payload, \u0026out);\n+\tKUNIT_EXPECT_EQ(test, out, frame);\n+}\n+#endif /* CONFIG_ARM64 */\n+\n+static struct kunit_case stackdepot_test_cases[] = {\n+\tKUNIT_CASE(stackdepot_trie_max_path_roundtrip),\n+\tKUNIT_CASE(stackdepot_save_flags_public),\n+\tKUNIT_CASE(stackdepot_snprint_public),\n+\tKUNIT_CASE(stackdepot_countable_public),\n+\tKUNIT_CASE(stackdepot_fetch_into_roundtrip),\n+\tKUNIT_CASE(stackdepot_fetch_into_rejects_missing_or_short_stack),\n+\tKUNIT_CASE(stackdepot_trie_topology_allocating),\n+\tKUNIT_CASE(stackdepot_trie_topology_constrained),\n+\tKUNIT_CASE(stackdepot_frame_storage_roundtrip),\n+\tKUNIT_CASE(stackdepot_frame_raw_fallback),\n+#if defined(CONFIG_X86_64) \u0026\u0026 !defined(CONFIG_UML)\n+\tKUNIT_CASE(stackdepot_frame_x86_64),\n+#endif\n+#ifdef CONFIG_ARM64\n+\tKUNIT_CASE(stackdepot_frame_arm64),\n+#endif\n+\t{}\n+};\n+\n+static struct kunit_suite stackdepot_test_suite = {\n+\t.name = \"stackdepot\",\n+\t.test_cases = stackdepot_test_cases,\n+};\n+\n+kunit_test_suite(stackdepot_test_suite);\n+\n+MODULE_DESCRIPTION(\"KUnit tests for stack depot\");\n+MODULE_AUTHOR(\"Caleb Kan \u003cckan@cloudflare.com\u003e\");\n+MODULE_LICENSE(\"GPL\");\ndiff --git a/mm/kmemleak.c b/mm/kmemleak.c\nindex 8fa409a4f9fb2..c42741a88bd42 100644\n--- a/mm/kmemleak.c\n+++ b/mm/kmemleak.c\n@@ -378,10 +378,10 @@ static void __print_unreferenced(struct seq_file *seq,\n \t\t\t\t bool hex_dump)\n {\n \tint i;\n-\tunsigned long *entries;\n+\tunsigned long entries[MAX_TRACE];\n \tunsigned int nr_entries;\n \n-\tnr_entries = stack_depot_fetch(object-\u003etrace_handle, \u0026entries);\n+\tnr_entries = stack_depot_fetch_into(object-\u003etrace_handle, entries, ARRAY_SIZE(entries));\n \twarn_or_seq_printf(seq, \"unreferenced object%s 0x%08lx (size %zu):\\n\",\n \t\t\t __object_type_str(object),\n \t\t\t object-\u003epointer, object-\u003esize);\ndiff --git a/mm/kmsan/kmsan_test.c b/mm/kmsan/kmsan_test.c\nindex 31f47cc4dab40..7c04e4b21873d 100644\n--- a/mm/kmsan/kmsan_test.c\n+++ b/mm/kmsan/kmsan_test.c\n@@ -669,7 +669,7 @@ static void test_long_origin_chain(struct kunit *test)\n */\n static void test_stackdepot_roundtrip(struct kunit *test)\n {\n-\tunsigned long src_entries[16], *dst_entries;\n+\tunsigned long src_entries[16], dst_entries[16];\n \tunsigned int src_nentries, dst_nentries;\n \tEXPECTATION_NO_REPORT(expect);\n \tdepot_stack_handle_t handle;\n@@ -680,7 +680,7 @@ static void test_stackdepot_roundtrip(struct kunit *test)\n \t\tstack_trace_save(src_entries, ARRAY_SIZE(src_entries), 1);\n \thandle = stack_depot_save(src_entries, src_nentries, GFP_KERNEL);\n \tstack_depot_print(handle);\n-\tdst_nentries = stack_depot_fetch(handle, \u0026dst_entries);\n+\tdst_nentries = stack_depot_fetch_into(handle, dst_entries, ARRAY_SIZE(dst_entries));\n \tKUNIT_EXPECT_TRUE(test, src_nentries == dst_nentries);\n \n \tkmsan_check_memory((void *)dst_entries,\ndiff --git a/mm/kmsan/report.c b/mm/kmsan/report.c\nindex d6853ce089541..0770658ba932e 100644\n--- a/mm/kmsan/report.c\n+++ b/mm/kmsan/report.c\n@@ -83,9 +83,9 @@ static char *pretty_descr(char *descr)\n \treturn report_local_descr;\n }\n \n-void kmsan_print_origin(depot_stack_handle_t origin)\n+static void kmsan_print_origin_with_buf(depot_stack_handle_t origin,\n+\t\t\t\t\tunsigned long *entries)\n {\n-\tunsigned long *entries = NULL, *chained_entries = NULL;\n \tunsigned int nr_entries, chained_nr_entries, skipnr;\n \tvoid *pc1 = NULL, *pc2 = NULL;\n \tdepot_stack_handle_t head;\n@@ -97,7 +97,8 @@ void kmsan_print_origin(depot_stack_handle_t origin)\n \t\treturn;\n \n \twhile (true) {\n-\t\tnr_entries = stack_depot_fetch(origin, \u0026entries);\n+\t\tnr_entries =\n+\t\t\tstack_depot_fetch_into(origin, entries, KMSAN_STACK_DEPTH);\n \t\tdepth = kmsan_depth_from_eb(stack_depot_get_extra_bits(origin));\n \t\tmagic = nr_entries ? entries[0] : 0;\n \t\tif ((nr_entries == 4) \u0026\u0026 (magic == KMSAN_ALLOCA_MAGIC_ORIGIN)) {\n@@ -123,14 +124,10 @@ void kmsan_print_origin(depot_stack_handle_t origin)\n \t\t\torigin = entries[2];\n \t\t\tpr_err(\"Uninit was stored to memory at:\\n\");\n \t\t\tchained_nr_entries =\n-\t\t\t\tstack_depot_fetch(head, \u0026chained_entries);\n-\t\t\tkmsan_internal_unpoison_memory(\n-\t\t\t\tchained_entries,\n-\t\t\t\tchained_nr_entries * sizeof(*chained_entries),\n-\t\t\t\t/*checked*/ false);\n-\t\t\tskipnr = get_stack_skipnr(chained_entries,\n-\t\t\t\t\t\t chained_nr_entries);\n-\t\t\tstack_trace_print(chained_entries + skipnr,\n+\t\t\t\tstack_depot_fetch_into(head, entries,\n+\t\t\t\t\t\t KMSAN_STACK_DEPTH);\n+\t\t\tskipnr = get_stack_skipnr(entries, chained_nr_entries);\n+\t\t\tstack_trace_print(entries + skipnr,\n \t\t\t\t\t chained_nr_entries - skipnr, 0);\n \t\t\tpr_err(\"\\n\");\n \t\t\tcontinue;\n@@ -147,6 +144,13 @@ void kmsan_print_origin(depot_stack_handle_t origin)\n \t}\n }\n \n+void kmsan_print_origin(depot_stack_handle_t origin)\n+{\n+\tunsigned long entries[KMSAN_STACK_DEPTH];\n+\n+\tkmsan_print_origin_with_buf(origin, entries);\n+}\n+\n void kmsan_report(depot_stack_handle_t origin, void *address, int size,\n \t\t int off_first, int off_last, const void __user *user_addr,\n \t\t enum kmsan_bug_reason reason)\n@@ -193,7 +197,7 @@ void kmsan_report(depot_stack_handle_t origin, void *address, int size,\n \t\t\t 0);\n \tpr_err(\"\\n\");\n \n-\tkmsan_print_origin(origin);\n+\tkmsan_print_origin_with_buf(origin, stack_entries);\n \n \tif (size) {\n \t\tpr_err(\"\\n\");\ndiff --git a/mm/page_owner.c b/mm/page_owner.c\nindex cfc31c92d7657..1fb1998bc129e 100644\n--- a/mm/page_owner.c\n+++ b/mm/page_owner.c\n@@ -119,7 +119,8 @@ static __always_inline depot_stack_handle_t create_dummy_stack(void)\n \tunsigned int nr_entries;\n \n \tnr_entries = stack_trace_save(entries, ARRAY_SIZE(entries), 0);\n-\treturn stack_depot_save(entries, nr_entries, GFP_KERNEL);\n+\treturn stack_depot_save_flags(entries, nr_entries, GFP_KERNEL,\n+\t\t\t\t STACK_DEPOT_FLAG_CAN_ALLOC | STACK_DEPOT_FLAG_COUNTABLE);\n }\n \n static noinline void register_dummy_stack(void)\n@@ -181,7 +182,8 @@ static noinline depot_stack_handle_t save_stack(gfp_t flags)\n \n \tset_current_in_page_owner();\n \tnr_entries = stack_trace_save(entries, ARRAY_SIZE(entries), 2);\n-\thandle = stack_depot_save(entries, nr_entries, flags);\n+\thandle = stack_depot_save_flags(entries, nr_entries, flags,\n+\t\t\t\t\tSTACK_DEPOT_FLAG_CAN_ALLOC | STACK_DEPOT_FLAG_COUNTABLE);\n \tif (!handle)\n \t\thandle = failure_handle;\n \tunset_current_in_page_owner();\ndiff --git a/mm/slub.c b/mm/slub.c\nindex f9b56cb439e70..4aa1c5a457182 100644\n--- a/mm/slub.c\n+++ b/mm/slub.c\n@@ -8198,12 +8198,12 @@ void __kmem_obj_info(struct kmem_obj_info *kpp, void *object, struct slab *slab)\n #ifdef CONFIG_STACKDEPOT\n \t{\n \t\tdepot_stack_handle_t handle;\n-\t\tunsigned long *entries;\n+\t\tunsigned long entries[TRACK_ADDRS_COUNT];\n \t\tunsigned int nr_entries;\n \n \t\thandle = READ_ONCE(trackp-\u003ehandle);\n \t\tif (handle) {\n-\t\t\tnr_entries = stack_depot_fetch(handle, \u0026entries);\n+\t\t\tnr_entries = stack_depot_fetch_into(handle, entries, ARRAY_SIZE(entries));\n \t\t\tfor (i = 0; i \u003c KS_ADDRS_COUNT \u0026\u0026 i \u003c nr_entries; i++)\n \t\t\t\tkpp-\u003ekp_stack[i] = (void *)entries[i];\n \t\t}\n@@ -8211,7 +8211,7 @@ void __kmem_obj_info(struct kmem_obj_info *kpp, void *object, struct slab *slab)\n \t\ttrackp = get_track(s, objp, TRACK_FREE);\n \t\thandle = READ_ONCE(trackp-\u003ehandle);\n \t\tif (handle) {\n-\t\t\tnr_entries = stack_depot_fetch(handle, \u0026entries);\n+\t\t\tnr_entries = stack_depot_fetch_into(handle, entries, ARRAY_SIZE(entries));\n \t\t\tfor (i = 0; i \u003c KS_ADDRS_COUNT \u0026\u0026 i \u003c nr_entries; i++)\n \t\t\t\tkpp-\u003ekp_free_stack[i] = (void *)entries[i];\n \t\t}\n@@ -9946,12 +9946,14 @@ static int slab_debugfs_show(struct seq_file *seq, void *v)\n #ifdef CONFIG_STACKDEPOT\n \t\t{\n \t\t\tdepot_stack_handle_t handle;\n-\t\t\tunsigned long *entries;\n+\t\t\tunsigned long entries[TRACK_ADDRS_COUNT];\n \t\t\tunsigned int nr_entries, j;\n \n \t\t\thandle = READ_ONCE(l-\u003ehandle);\n \t\t\tif (handle) {\n-\t\t\t\tnr_entries = stack_depot_fetch(handle, \u0026entries);\n+\t\t\t\tnr_entries =\n+\t\t\t\t\tstack_depot_fetch_into(handle, entries,\n+\t\t\t\t\t\t\t ARRAY_SIZE(entries));\n \t\t\t\tseq_puts(seq, \"\\n\");\n \t\t\t\tfor (j = 0; j \u003c nr_entries; j++)\n \t\t\t\t\tseq_printf(seq, \" %pS\\n\", (void *)entries[j]);\ndiff --git a/scripts/gdb/linux/stackdepot.py b/scripts/gdb/linux/stackdepot.py\nindex 37313a5a51a0d..82aeb9f532c3d 100644\n--- a/scripts/gdb/linux/stackdepot.py\n+++ b/scripts/gdb/linux/stackdepot.py\n@@ -37,6 +37,10 @@ def stack_depot_fetch(handle):\n if handle == 0:\n raise gdb.GdbError(\"handle is 0\\n\")\n \n+ stack_max_pools = gdb.parse_and_eval('stack_max_pools')\n+ if parts['pool_index_plus_1'] \u003e stack_max_pools:\n+ raise gdb.GdbError(\"trie-backed stack depot handles are not supported\\n\")\n+\n pool_index = parts['pool_index_plus_1'] - 1\n if pool_index \u003e= pools_num:\n gdb.write(\"pool index %d out of bounds (%d) for stack id 0x%08x\\n\" % (parts['pool_index'], pools_num, handle))\n"
} |
| 2/1 | 2026/09/08 15:27 | agent | patch-evaluator |
0mModel:core Args: null Results: {
"EnableConfigs": [
"STACKDEPOT",
"STACKDEPOT_KUNIT_TEST",
"PAGE_OWNER",
"DEBUG_KMEMLEAK",
"SLUB_DEBUG"
],
"FocusSymbols": [
"stack_depot_fetch_into",
"stack_depot_trie_save",
"stack_depot_trie_save_constrained",
"stack_depot_trie_insert",
"stack_depot_trie_lookup",
"trie_insert_path",
"trie_split_child",
"trie_promote_child",
"trie_fetch_handle_into",
"trie_fetch_handle_range",
"trie_pool_alloc",
"trie_pool_reserve_slots",
"trie_drain_pending_children",
"trie_reparent_children",
"trie_path_alloc",
"trie_side_table_publish",
"trie_side_table_lookup",
"trie_side_table_prepare_stack_slot",
"trie_print",
"trie_snprint",
"kmsan_print_origin",
"__drm_stack_depot_print",
"slab_debugfs_show"
],
"Reasoning": "The patch series introduces a new trie-based storage backend for stack depot with arch-specific frame compression (x86-64 and arm64), RCU-based child management, slot allocation/reservation within depot pools, side-table mappings, and a new stack_depot_fetch_into() API. It also updates several callers across the kernel (SLUB, page_owner, kmemleak, KMSAN, DRM) and adds runtime checks and assertions. These changes modify core memory management infrastructure and are reachable during kernel execution in standard virtualized environments.",
"WorthFuzzing": true
}Instruction: You are an expert Linux kernel maintainer and security engineer.
Your job is to review a provided patch series and evaluate whether it warrants fuzzing with syzkaller.
IMPORTANT: The changes have ALREADY been applied and committed as the HEAD commit in
your workspace. Do NOT rely on internal assumptions. You must actively use your code access
tools to inspect the actual source code, callers, and surrounding context.
================================================================================
1. CORE TRIAGE PHILOSOPHY
================================================================================
The goal of patch fuzzing is to discover crashes, regressions, exposed latent bugs,
and newly triggered assertions introduced by the patch series.
- REACHABILITY IS THE PRIMARY GATE:
Fuzzing can only discover bugs in code that can actually execute in standard virtualized
environments (GCE or QEMU, utilizing software-emulated devices like USB gadgets, netdev, tun/tap).
If the modified code is structurally unreachable (see Section 2), it MUST NOT be fuzzed,
regardless of whether it adds assertions or complex logic.
- DO NOT BLINDLY TRUST "NO FUNCTIONAL CHANGE" (NFCI) OR "REFACTORING" CLAIMS:
Patch authors routinely label changes as "cleanups", "refactorings", or state
"No functional change intended". Do NOT take these claims at face value.
Code refactorings that rearrange logic, introduce helper functions, or alter state management
in core subsystems frequently introduce subtle semantic shifts or uncover latent kernel bugs.
If reachable executable code is modified or refactored, it MUST be fuzzed.
- NEW OR MODIFIED ASSERTIONS IN REACHABLE CODE MUST BE FUZZED:
When a patch introduces or modifies runtime checks or assertions (e.g., WARN_ON*, VM_WARN_ON*,
BUG_ON*, lockdep_assert*) in reachable code paths, it enforces new or stricter invariants.
Even if the author believes the invariant always holds, fuzzing is essential to verify whether
an unusual sequence of operations can violate it.
================================================================================
2. WHEN TO RETURN WorthFuzzing=false (NEGATIVE CRITERIA)
================================================================================
Return WorthFuzzing=false ONLY IF all modified code falls strictly into one or more of these categories:
- Non-kernel and non-executable changes:
* Modifications to Documentation/, comments, or spelling fixes.
* User-space directories, self-tests, samples, or scripts (e.g., tools/, samples/, scripts/, usr/)
that do not affect the compiled kernel image (vmlinux) or kernel modules.
* Purely decorative logging (e.g., message strings in pr_err, printk, dev_info) or tracepoints
that do not alter control flow or data structures.
* Build system or Kconfig changes that do not alter compiled C logic.
- Structurally unreachable hardware:
* Vendor-specific PCIe switches, SmartNICs, or GPU drivers (e.g., mlxsw, pds_core, qed,
ionic, amdgpu) requiring physical ASIC/PCIe cards not emulated in standard QEMU.
- Unreachable execution paths:
* Driver teardown callbacks (.remove, .shutdown, pci_unregister_driver) executed only during
physical PCI hot-unplug or manual sysfs driver unbinding.
* Code paths exclusive to architectures other than the target architecture.
================================================================================
3. WHEN TO RETURN WorthFuzzing=true (POSITIVE CRITERIA)
================================================================================
Return WorthFuzzing=true whenever the patch touches reachable executable code, including:
- Core Subsystems:
* Any logic modifications in memory management (mm/), synchronization/locking (kernel/locking/),
BPF, scheduler, core networking, VFS, or syscall handling.
- Refactorings and Code Cleanups:
* Any restructuring of reachable data structures, helper abstractions, or algorithm flows.
- Runtime Assertions and Defensive Checks:
* Any introduction or alteration of assertions (WARN_ON*, VM_WARN_ON*, BUG_ON*, etc.) in reachable paths.
- Reachable Drivers and Protocols:
* Drivers accessible via virtual buses (virtio, USB gadget, loopback, netlink, binder, sockets, etc.).
================================================================================
4. EXTRACTING FocusSymbols (PREVENTING DILUTION)
================================================================================
When WorthFuzzing=true, you must extract specific kernel functions into FocusSymbols to guide the fuzzer:
- AVOID UBIQUITOUS LIFECYCLE HOT-PATHS:
Do NOT list generic, ubiquitous functions called by almost every program in the corpus
(including, but not limited to: general memory allocators and deallocators, page fault
and trap handlers, or core synchronization primitives; this is not an exhaustive list).
Listing ubiquitous functions causes the fuzzer to classify thousands of unrelated tests as "focused",
which severely dilutes fuzzing effort away from the actual changes.
- TARGET SPECIFIC FEATURE LOGIC AND ENTRYPOINTS:
List functions that specifically implement the logic being added or altered, or direct API entrypoints
for the subsystem feature under review.
- HANDLING STATIC INLINE FUNCTIONS IN HEADERS (.h):
Compiler-inlined static functions (such as static inlines in mm/*.h or include/linux/*.h) lack
distinct symbol addresses in vmlinux and cannot be targeted directly by symbol coverage filters.
If the changes are primarily in static inline helpers, identify non-static, feature-specific caller
functions in .c files that exercise them (avoiding ubiquitous lifecycle wrappers).
================================================================================
5. IDENTIFYING EnableConfigs
================================================================================
Identify any specific CONFIG_ options required to properly compile and reach the modified code:
- Inspect Kconfig files and #ifdef guards; do not make assumptions.
- Check "depends on" lines in Kconfig to include any non-standard parent subsystem configs needed.
- Strip any 'CONFIG_' prefix (e.g., return "NET_IPV4" instead of "CONFIG_NET_IPV4").
Prefer calling several tools at the same time to save round-trips.
Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.
Prompt: Target architecture: amd64
For your convenience, here is the diff of the changes:
commit efdf021b1a377c26b41b0ed29bb41a4fbb95bac4
Author: syz-cluster <triage@syzkaller.com>
Date: Tue Sep 8 15:27:13 2026 +0000
syz-cluster: applied patch under review
diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 68647ff4bdd24..b02bcbaef5dcb 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -7449,6 +7449,13 @@ Kernel parameters
stack traces. Pools are allocated on-demand up to this
limit. Default value is 8191 pools.
+ stackdepot.trie_enabled= [KNL]
+ Format: <bool>
+ Enable trie storage for persistent, non-refcounted
+ stack depot records at boot. Disabled by default.
+ stack_depot_max_pools must leave unused pool-index
+ values for trie handles.
+
stacktrace [FTRACE]
Enable the stack tracer on boot up.
diff --git a/arch/arm64/include/asm/stackdepot.h b/arch/arm64/include/asm/stackdepot.h
new file mode 100644
index 0000000000000..df8959d593366
--- /dev/null
+++ b/arch/arm64/include/asm/stackdepot.h
@@ -0,0 +1,42 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __ASM_STACKDEPOT_H
+#define __ASM_STACKDEPOT_H
+
+#include <linux/types.h>
+#include <asm/sections.h>
+
+/*
+ * Modules are allocated inside a 2 GB relocation window containing the
+ * kernel image. Store a signed 32-bit offset from _text so compression is
+ * independent of 4 GB high-bit boundaries crossed by that window.
+ */
+static inline unsigned long arch_stack_depot_frame_from_payload(u32 payload)
+{
+ long offset;
+
+ offset = (s32)payload;
+ if (offset < 0)
+ return (unsigned long)_text - (unsigned long)(-offset);
+ return (unsigned long)_text + (unsigned long)offset;
+}
+
+static inline bool
+arch_stack_depot_frame_try_compress(unsigned long frame, u32 *payload)
+{
+ u32 candidate;
+
+ candidate = (u32)(frame - (unsigned long)_text);
+ if (arch_stack_depot_frame_from_payload(candidate) != frame)
+ return false;
+
+ *payload = candidate;
+ return true;
+}
+
+static inline void
+arch_stack_depot_frame_decompress(u32 payload, unsigned long *frame)
+{
+ *frame = arch_stack_depot_frame_from_payload(payload);
+}
+
+#endif /* __ASM_STACKDEPOT_H */
diff --git a/arch/um/include/asm/Kbuild b/arch/um/include/asm/Kbuild
index 8fdc0bd9ab6fb..14778d2457d79 100644
--- a/arch/um/include/asm/Kbuild
+++ b/arch/um/include/asm/Kbuild
@@ -21,6 +21,7 @@ generic-y += preempt.h
generic-y += ring_buffer.h
generic-y += runtime-const.h
generic-y += softirq_stack.h
+generic-y += stackdepot.h
generic-y += switch_to.h
generic-y += topology.h
generic-y += trace_clock.h
diff --git a/arch/x86/include/asm/stackdepot.h b/arch/x86/include/asm/stackdepot.h
new file mode 100644
index 0000000000000..9a8d04fa8c1c8
--- /dev/null
+++ b/arch/x86/include/asm/stackdepot.h
@@ -0,0 +1,37 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_X86_STACKDEPOT_H
+#define _ASM_X86_STACKDEPOT_H
+
+#include <linux/types.h>
+
+#ifdef CONFIG_X86_64
+/*
+ * Compress canonical kernel text/module addresses whose upper 32 bits are all
+ * ones. Other kernel virtual addresses stay raw, so decompression reconstructs
+ * the original frame by restoring this prefix.
+ */
+#define STACK_DEPOT_X86_64_FRAME_PREFIX 0xffffffff00000000UL
+#define STACK_DEPOT_X86_64_FRAME_LOW_MASK 0x00000000ffffffffUL
+
+static inline bool
+arch_stack_depot_frame_try_compress(unsigned long frame, u32 *low)
+{
+ if ((frame & ~STACK_DEPOT_X86_64_FRAME_LOW_MASK) !=
+ STACK_DEPOT_X86_64_FRAME_PREFIX)
+ return false;
+
+ *low = (u32)frame;
+ return true;
+}
+
+static inline void
+arch_stack_depot_frame_decompress(u32 low, unsigned long *frame)
+{
+ *frame = STACK_DEPOT_X86_64_FRAME_PREFIX | low;
+}
+
+#else
+#include <asm-generic/stackdepot.h>
+#endif /* CONFIG_X86_64 */
+
+#endif /* _ASM_X86_STACKDEPOT_H */
diff --git a/drivers/gpu/drm/drm_modeset_lock.c b/drivers/gpu/drm/drm_modeset_lock.c
index e14814c30d8c0..f48b02379b711 100644
--- a/drivers/gpu/drm/drm_modeset_lock.c
+++ b/drivers/gpu/drm/drm_modeset_lock.c
@@ -94,16 +94,13 @@ static noinline depot_stack_handle_t __drm_stack_depot_save(void)
static void __drm_stack_depot_print(depot_stack_handle_t stack_depot)
{
struct drm_printer p = drm_dbg_printer(NULL, DRM_UT_KMS, "drm_modeset_lock");
- unsigned long *entries;
- unsigned int nr_entries;
char *buf;
buf = kmalloc(PAGE_SIZE, GFP_NOWAIT | __GFP_NOWARN);
if (!buf)
return;
- nr_entries = stack_depot_fetch(stack_depot, &entries);
- stack_trace_snprint(buf, PAGE_SIZE, entries, nr_entries, 2);
+ stack_depot_snprint(stack_depot, buf, PAGE_SIZE, 2);
drm_printf(&p, "attempting to lock a contended lock without backoff:\n%s", buf);
diff --git a/include/asm-generic/Kbuild b/include/asm-generic/Kbuild
index 2bc00c67dc54b..d8402a6afc703 100644
--- a/include/asm-generic/Kbuild
+++ b/include/asm-generic/Kbuild
@@ -55,6 +55,7 @@ mandatory-y += serial.h
mandatory-y += shmparam.h
mandatory-y += simd.h
mandatory-y += softirq_stack.h
+mandatory-y += stackdepot.h
mandatory-y += switch_to.h
mandatory-y += timex.h
mandatory-y += tlbflush.h
diff --git a/include/asm-generic/stackdepot.h b/include/asm-generic/stackdepot.h
new file mode 100644
index 0000000000000..846975767bdd4
--- /dev/null
+++ b/include/asm-generic/stackdepot.h
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __ASM_GENERIC_STACKDEPOT_H
+#define __ASM_GENERIC_STACKDEPOT_H
+
+#include <linux/types.h>
+
+static inline bool
+arch_stack_depot_frame_try_compress(unsigned long frame, u32 *low)
+{
+ return false;
+}
+
+static inline void
+arch_stack_depot_frame_decompress(u32 low, unsigned long *frame)
+{
+ /* Generic code never compresses frames, so this hook is unreachable. */
+}
+
+#endif /* __ASM_GENERIC_STACKDEPOT_H */
diff --git a/include/linux/stackdepot.h b/include/linux/stackdepot.h
index 2cc21ffcdaf9e..3126d17b9265b 100644
--- a/include/linux/stackdepot.h
+++ b/include/linux/stackdepot.h
@@ -53,7 +53,8 @@ union handle_parts {
struct stack_record {
struct list_head hash_list; /* Links in the hash table */
u32 hash; /* Hash in hash table */
- u32 size; /* Number of stored frames */
+ u16 size; /* Number of stored frames */
+ u16 flags;
union handle_parts handle; /* Constant after initialization */
refcount_t count;
union {
@@ -84,8 +85,9 @@ typedef u32 depot_flags_t;
*/
#define STACK_DEPOT_FLAG_CAN_ALLOC ((depot_flags_t)0x0001)
#define STACK_DEPOT_FLAG_GET ((depot_flags_t)0x0002)
+#define STACK_DEPOT_FLAG_COUNTABLE ((depot_flags_t)0x0004)
-#define STACK_DEPOT_FLAGS_NUM 2
+#define STACK_DEPOT_FLAGS_NUM 3
#define STACK_DEPOT_FLAGS_MASK ((depot_flags_t)((1 << STACK_DEPOT_FLAGS_NUM) - 1))
/*
@@ -144,6 +146,17 @@ static inline int stack_depot_early_init(void) { return 0; }
* Users of this flag must also call stack_depot_put() when keeping the stack
* trace is no longer required to avoid overflowing the refcount.
*
+ * If STACK_DEPOT_FLAG_COUNTABLE is set in @depot_flags, stack depot stores the
+ * stack in hash-backed storage for callers that need direct stack_record count
+ * access. This flag does not imply %STACK_DEPOT_FLAG_CAN_ALLOC and is mutually
+ * exclusive with %STACK_DEPOT_FLAG_GET.
+ *
+ * When trie storage is enabled, persistent non-refcounted saves use trie
+ * storage. Constrained callers first look up an existing stack, then make one
+ * best-effort insertion attempt without allocating. NMI callers stop after the
+ * lookup. Other callers that cannot spin use trylocks and fail if a required
+ * lock is unavailable. Trie failures do not fall back to hash storage.
+ *
* If the provided stack trace comes from the interrupt context, only the part
* up to the interrupt entry is saved.
*
@@ -152,7 +165,7 @@ static inline int stack_depot_early_init(void) { return 0; }
* this is the case for contexts where neither %GFP_ATOMIC nor
* %GFP_NOWAIT can be used (NMI, raw_spin_lock).
*
- * Return: Handle of the stack struct stored in depot, 0 on failure
+ * Return: Handle of the stack trace stored in depot, 0 on failure
*/
depot_stack_handle_t stack_depot_save_flags(unsigned long *entries,
unsigned int nr_entries,
@@ -169,6 +182,10 @@ depot_stack_handle_t stack_depot_save_flags(unsigned long *entries,
* Does not increment the refcount on the saved stack trace; see
* stack_depot_save_flags() for more details.
*
+ * When trie storage is enabled, this can return trie-backed handles. Use
+ * stack_depot_fetch_into(), stack_depot_print(), or stack_depot_snprint() for
+ * backend-independent access to the stack contents.
+ *
* Context: Contexts where allocations via alloc_pages() are allowed;
* see stack_depot_save_flags() for more details.
*
@@ -178,11 +195,12 @@ depot_stack_handle_t stack_depot_save(unsigned long *entries,
unsigned int nr_entries, gfp_t alloc_flags);
/**
- * __stack_depot_get_stack_record - Get a pointer to a stack_record struct
+ * __stack_depot_get_stack_record - Get a hash-backed stack record
*
* @handle: Stack depot handle
*
- * This function is only for internal purposes.
+ * This function is only for internal purposes. @handle must have been saved
+ * with %STACK_DEPOT_FLAG_COUNTABLE.
*
* Return: Returns a pointer to a stack_record struct
*/
@@ -191,14 +209,55 @@ struct stack_record *__stack_depot_get_stack_record(depot_stack_handle_t handle)
/**
* stack_depot_fetch - Fetch a stack trace from stack depot
*
- * @handle: Stack depot handle returned from stack_depot_save()
+ * @handle: Hash-backed stack depot handle
* @entries: Pointer to store the address of the stack trace
*
+ * This helper returns a pointer to stackdepot-owned contiguous storage for
+ * legacy hash-backed handles. Callers that need backend-independent access to
+ * stack contents should use stack_depot_fetch_into(), stack_depot_print(), or
+ * stack_depot_snprint(). Passing a trie-backed handle is invalid and may WARN.
+ *
* Return: Number of frames for the fetched stack
*/
unsigned int stack_depot_fetch(depot_stack_handle_t handle,
unsigned long **entries);
+/**
+ * stack_depot_fetch_into - Fetch a stack trace into caller-owned storage
+ *
+ * @handle: Stack depot handle
+ * @entries: Caller-owned buffer to copy the stack trace into
+ * @max_entries: Number of frames that fit in @entries
+ *
+ * Copies the stored frames into caller-owned @entries. If fewer frames are
+ * stored than @max_entries, only the stored frames are written and their count
+ * is returned. If more frames are stored than @max_entries, the copy is skipped
+ * entirely and 0 is returned.
+ *
+ * Passing a NULL @entries buffer or zero @max_entries for a valid @handle is
+ * invalid. Callers must provide storage for @max_entries frames.
+ *
+ * Callers should size @entries to match the save-side stack depth cap (for
+ * example, %CONFIG_STACKDEPOT_MAX_FRAMES or the local stack_trace_save() limit)
+ * when losing diagnostics on an undersized buffer would be surprising.
+ *
+ * A non-zero invalid @handle, including a post-put handle, may WARN. Its return
+ * value and copied contents are undefined because the record may have been
+ * reused for another stack.
+ *
+ * Callers must ensure @handle remains valid for the duration of this call.
+ * Persistent handles saved without %STACK_DEPOT_FLAG_GET require no extra
+ * reference; handles saved with %STACK_DEPOT_FLAG_GET require a held reference.
+ * Callers must not call stack_depot_put() on persistent handles.
+ * Racing this helper with stack_depot_put() on the same handle is invalid.
+ *
+ * Return: Number of frames copied, 0 if @handle is 0, stack depot is disabled,
+ * or @max_entries is less than the number of stored frames.
+ */
+unsigned int stack_depot_fetch_into(depot_stack_handle_t handle,
+ unsigned long *entries,
+ unsigned int max_entries);
+
/**
* stack_depot_print - Print a stack trace from stack depot
*
@@ -224,10 +283,14 @@ int stack_depot_snprint(depot_stack_handle_t handle, char *buf, size_t size,
*
* @handle: Stack depot handle returned from stack_depot_save()
*
- * The stack trace is evicted from stack depot once all references to it have
- * been dropped (once the number of stack_depot_evict() calls matches the
- * number of stack_depot_save_flags() calls with STACK_DEPOT_FLAG_GET set for
- * this stack trace).
+ * Drop a reference acquired by stack_depot_save_flags() with
+ * %STACK_DEPOT_FLAG_GET. Calling this for a handle saved without
+ * %STACK_DEPOT_FLAG_GET is invalid; persistent handles, including trie-backed
+ * handles, are owned by stack depot for the lifetime of the system.
+ *
+ * The stack trace is evicted once the number of stack_depot_put() calls matches
+ * the number of successful stack_depot_save_flags() calls with
+ * %STACK_DEPOT_FLAG_GET for this stack trace.
*/
void stack_depot_put(depot_stack_handle_t handle);
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index 134b15a44625e..3a78c67b6b364 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -2785,6 +2785,23 @@ config RESOURCE_KUNIT_TEST
If unsure, say N.
+config STACKDEPOT_KUNIT_TEST
+ bool "KUnit test for stack depot" if !KUNIT_ALL_TESTS
+ depends on KUNIT=y && STACKDEPOT
+ depends on STACKDEPOT_MAX_FRAMES >= 3
+ default KUNIT_ALL_TESTS
+ help
+ Enable this option to test stack depot API behavior at boot.
+ This test is built in because it exercises internal, non-exported
+ stack depot helpers, so KUNIT must also be built in.
+
+ KUnit tests run during boot and output the results to the debug log
+ in TAP format (https://testanything.org/). Only useful for kernel
+ developers running the KUnit test harness, and not intended for
+ inclusion into a production build.
+
+ If unsure, say N.
+
config SYSCTL_KUNIT_TEST
tristate "KUnit test for sysctl" if !KUNIT_ALL_TESTS
depends on KUNIT
diff --git a/lib/stackdepot.c b/lib/stackdepot.c
index dd2717ff94bff..33e475d941314 100644
--- a/lib/stackdepot.c
+++ b/lib/stackdepot.c
@@ -2,9 +2,11 @@
/*
* Stack depot - a stack trace storage that avoids duplication.
*
- * Internally, stack depot maintains a hash table of unique stacktraces. The
- * stack traces themselves are stored contiguously one after another in a set
- * of separate page allocations.
+ * Internally, stack depot has two storage backends. Refcounted entries and
+ * callers that request STACK_DEPOT_FLAG_COUNTABLE use the legacy hash table with
+ * contiguous stack records in stack pools. Persistent non-refcounted entries
+ * can use trie storage when enabled; trie nodes share common frame prefixes and
+ * are published through RCU children containers.
*
* Author: Alexander Potapenko <glider@google.com>
* Copyright (C) 2016 Google, Inc.
@@ -14,13 +16,19 @@
#define pr_fmt(fmt) "stackdepot: " fmt
+#include <linux/bitmap.h>
+#include <linux/build_bug.h>
#include <linux/debugfs.h>
+#include <linux/errno.h>
#include <linux/gfp.h>
#include <linux/jhash.h>
+#include <linux/jump_label.h>
#include <linux/kernel.h>
+#include <linux/log2.h>
#include <linux/kmsan.h>
#include <linux/list.h>
#include <linux/mm.h>
+#include <linux/moduleparam.h>
#include <linux/mutex.h>
#include <linux/poison.h>
#include <linux/printk.h>
@@ -36,9 +44,12 @@
#include <linux/memblock.h>
#include <linux/kasan-enabled.h>
+#include <asm/stackdepot.h>
+
/*
* The pool_index is offset by 1 so the first record does not have a 0 handle.
*/
+/* Parsed before mm_core_init(); trie handle decoding assumes this is then fixed. */
static unsigned int stack_max_pools __read_mostly =
MIN((1LL << DEPOT_POOL_INDEX_BITS) - 1, 8192);
@@ -54,6 +65,9 @@ static bool __stack_depot_early_init_passed __initdata;
/* Initial seed for jhash2. */
#define STACK_HASH_SEED 0x9747b28c
+/* Bound 64-bit print scratch to 128 bytes while amortizing trie walks. */
+#define STACK_DEPOT_PRINT_CHUNK_FRAMES 16
+
/* Hash table of stored stack records. */
static struct list_head *stack_table;
/* Fixed order of the number of table buckets. Used when KASAN is enabled. */
@@ -63,18 +77,18 @@ static unsigned int stack_hash_mask;
/* The lock must be held when performing pool or freelist modifications. */
static DEFINE_RAW_SPINLOCK(pool_lock);
-/* Array of memory regions that store stack records. */
+/* Array of memory regions used by both stack depot backends. */
static void **stack_pools __pt_guarded_by(&pool_lock);
/* Newly allocated pool that is not yet added to stack_pools. */
static void *new_pool;
/* Number of pools in stack_pools. */
static int pools_num;
-/* Offset to the unused space in the currently used pool. */
+/* Offset to unused hash storage in the current pool. */
static size_t pool_offset __guarded_by(&pool_lock) = DEPOT_POOL_SIZE;
/* Freelist of stack records within stack_pools. */
static __guarded_by(&pool_lock) LIST_HEAD(free_stacks);
-/* Statistics counters for debugfs. */
+/* Hash-backend statistics counters for debugfs. */
enum depot_counter_id {
DEPOT_COUNTER_REFD_ALLOCS,
DEPOT_COUNTER_REFD_FREES,
@@ -90,11 +104,695 @@ static const char *const counter_names[] = {
[DEPOT_COUNTER_REFD_FREES] = "refcounted_frees",
[DEPOT_COUNTER_REFD_INUSE] = "refcounted_in_use",
[DEPOT_COUNTER_FREELIST_SIZE] = "freelist_size",
- [DEPOT_COUNTER_PERSIST_COUNT] = "persistent_count",
- [DEPOT_COUNTER_PERSIST_BYTES] = "persistent_bytes",
+ [DEPOT_COUNTER_PERSIST_COUNT] = "hash_persistent_count",
+ [DEPOT_COUNTER_PERSIST_BYTES] = "hash_persistent_bytes",
};
static_assert(ARRAY_SIZE(counter_names) == DEPOT_COUNTER_COUNT);
+enum stack_depot_frame_mode {
+ STACK_DEPOT_FRAME_RAW,
+ STACK_DEPOT_FRAME_COMPRESSED,
+};
+
+/*
+ * A trie node stores one run of frames that all use the same payload format.
+ * Architectures may compress some frames to 32-bit payloads; mixed raw and
+ * compressed input is split across multiple trie nodes so each node has one
+ * decoding mode.
+ */
+struct stack_depot_frame_run {
+ u16 nr_entries;
+ u8 mode;
+};
+
+static_assert(CONFIG_STACKDEPOT_MAX_FRAMES <= U16_MAX);
+
+struct stack_depot_trie_children;
+
+struct stack_depot_trie_node {
+ /* Parent links let fetch rebuild a full stack from a node to the root. */
+ const struct stack_depot_trie_node __rcu *parent;
+ /* Children are RCU-published containers. */
+ const struct stack_depot_trie_children __rcu *children;
+ /* Non-zero when a stored stack ends at this node. */
+ u32 stack_id;
+ struct stack_depot_frame_run run;
+ unsigned char data[];
+};
+
+/*
+ * Child nodes are sorted by first frame and searched by insertion position.
+ * Existing child pointers are immutable. Writers may publish into unused tail
+ * capacity; other updates publish a replacement container.
+ */
+struct stack_depot_trie_children {
+ unsigned int nr_children;
+ unsigned int capacity;
+ const struct stack_depot_trie_node __rcu *nodes[];
+};
+
+/* Retired children carry an optional node through their RCU grace period. */
+struct stack_depot_trie_retired_children {
+ struct list_head list;
+ unsigned long rcu_state;
+ const struct stack_depot_trie_node *pending_node;
+ unsigned char data[];
+};
+
+static_assert(IS_ALIGNED(offsetof(struct stack_depot_trie_retired_children, data),
+ 1UL << DEPOT_STACK_ALIGN));
+
+#define STACK_DEPOT_TRIE_SLOT_SIZE BIT(DEPOT_STACK_ALIGN)
+#define STACK_DEPOT_TRIE_POOL_SLOTS \
+ (DEPOT_POOL_SIZE / STACK_DEPOT_TRIE_SLOT_SIZE)
+
+static_assert(STACK_DEPOT_TRIE_POOL_SLOTS - 1 <= U16_MAX);
+
+struct stack_depot_trie_pool {
+ struct list_head list;
+ unsigned int free_slots;
+ /* Conservative upper bound on the largest free run. */
+ u16 free_run_upper_bound;
+ /* First physical slot considered by the next reservation. */
+ u16 next_slot;
+ DECLARE_BITMAP(used, STACK_DEPOT_TRIE_POOL_SLOTS);
+};
+
+#define STACK_DEPOT_TRIE_POOL_FIRST_SLOT \
+ DIV_ROUND_UP(sizeof(struct stack_depot_trie_pool), \
+ STACK_DEPOT_TRIE_SLOT_SIZE)
+#define STACK_DEPOT_TRIE_POOL_USABLE_SIZE \
+ ((STACK_DEPOT_TRIE_POOL_SLOTS - STACK_DEPOT_TRIE_POOL_FIRST_SLOT) * \
+ STACK_DEPOT_TRIE_SLOT_SIZE)
+
+static_assert(STACK_DEPOT_TRIE_POOL_FIRST_SLOT < STACK_DEPOT_TRIE_POOL_SLOTS);
+
+static DEFINE_STATIC_KEY_FALSE(stack_depot_trie_enabled);
+static const struct stack_depot_trie_children __rcu *stack_depot_trie_root;
+static DEFINE_RAW_SPINLOCK(stack_depot_trie_writer_lock);
+static bool stack_depot_trie_requested;
+
+module_param_named(trie_enabled, stack_depot_trie_requested, bool, 0);
+MODULE_PARM_DESC(trie_enabled, "Enable stack depot trie storage at boot");
+
+#define DEPOT_POOL_INDEX_MASK ((1U << DEPOT_POOL_INDEX_BITS) - 1)
+#define DEPOT_OFFSET_MASK ((1U << DEPOT_OFFSET_BITS) - 1)
+
+/* Retired fixed-size slots remain reserved until their RCU grace period ends. */
+static LIST_HEAD(stack_depot_trie_pools);
+static LIST_HEAD(pending_trie_children);
+
+/*
+ * stack_max_pools is the split point between hash and trie handle encodings.
+ * A handle with pool_index_plus_1 in 1..stack_max_pools names a hash-backed
+ * stack pool. Larger pool-index values cannot refer to hash pools, so trie
+ * storage uses that handle space to encode a dense stack ID. The side table
+ * maps each stack ID to its trie node.
+ */
+static inline u32 trie_max_stack_id(void)
+{
+ return (DEPOT_POOL_INDEX_MASK - stack_max_pools) <<
+ DEPOT_OFFSET_BITS;
+}
+
+static depot_stack_handle_t trie_handle(u32 stack_id)
+{
+ union handle_parts parts = {};
+ u64 pool_index_plus_1;
+ u32 pool_delta;
+ u32 index;
+
+ index = stack_id - 1;
+ pool_delta = index >> DEPOT_OFFSET_BITS;
+ pool_index_plus_1 = (u64)stack_max_pools + 1 + pool_delta;
+
+ parts.pool_index_plus_1 = pool_index_plus_1;
+ parts.offset = index & DEPOT_OFFSET_MASK;
+ return parts.handle;
+}
+
+static inline bool stack_depot_handle_is_trie(depot_stack_handle_t handle)
+{
+ union handle_parts parts = { .handle = handle };
+
+ return parts.pool_index_plus_1 > stack_max_pools;
+}
+
+static u32 trie_stack_id(depot_stack_handle_t handle)
+{
+ union handle_parts parts = { .handle = handle };
+ u32 pool_delta;
+
+ pool_delta = parts.pool_index_plus_1 - stack_max_pools - 1;
+ return (pool_delta << DEPOT_OFFSET_BITS) + parts.offset + 1;
+}
+
+/*
+ * Trie handles encode a dense stack ID. The side table maps that ID to a node
+ * pointer for lockless fetch and print paths, which can run from diagnostic
+ * contexts where taking a lock would be unsafe. Initialization installs the
+ * root; early initialization also installs the first directory and chunk.
+ * Additional directories and chunks are published lazily as stack IDs grow.
+ */
+#define STACK_DEPOT_TRIE_SIDE_TABLE_CHUNK_SIZE \
+ (PAGE_SIZE / sizeof(struct stack_depot_trie_node *))
+#define STACK_DEPOT_TRIE_SIDE_TABLE_DIR_SIZE \
+ (PAGE_SIZE / sizeof(struct stack_depot_trie_node **))
+
+struct stack_depot_trie_side_dir {
+ /* Both the chunk pointer and each node pointer in it are RCU-published. */
+ const struct stack_depot_trie_node __rcu * __rcu *
+ chunks[STACK_DEPOT_TRIE_SIDE_TABLE_DIR_SIZE];
+};
+
+struct stack_depot_trie_side_root {
+ unsigned int dir_capacity;
+ struct stack_depot_trie_side_dir __rcu *dirs[];
+};
+
+struct stack_depot_trie_side_prealloc {
+ /* Preallocated side-table directory page for sparse growth. */
+ struct stack_depot_trie_side_dir *dir;
+ /* Preallocated side-table pointer chunk for sparse growth. */
+ const struct stack_depot_trie_node __rcu **chunk;
+};
+
+static struct stack_depot_trie_side_root *trie_side_table_root;
+static DEFINE_RAW_SPINLOCK(trie_side_table_cache_lock);
+/* Zeroed unpublished pages; get/put transfer ownership under the cache lock. */
+static struct stack_depot_trie_side_prealloc trie_side_table_cache;
+static u32 trie_side_table_last_stack_id;
+
+/* Lock order: writer_lock -> pool_lock -> side-table cache lock. */
+
+static inline size_t stack_depot_frame_run_entry_bytes(enum stack_depot_frame_mode mode)
+{
+ if (mode == STACK_DEPOT_FRAME_COMPRESSED)
+ return sizeof(u32);
+ return sizeof(unsigned long);
+}
+
+static inline size_t stack_depot_frame_run_bytes(const struct stack_depot_frame_run *run)
+{
+ return run->nr_entries * stack_depot_frame_run_entry_bytes(run->mode);
+}
+
+static inline size_t trie_node_bytes(const struct stack_depot_frame_run *run)
+{
+ return ALIGN(offsetof(struct stack_depot_trie_node, data) +
+ stack_depot_frame_run_bytes(run), sizeof(unsigned long));
+}
+
+static size_t trie_children_alloc_size(unsigned int capacity)
+{
+ size_t size;
+
+ size = struct_size_t(struct stack_depot_trie_children, nodes,
+ capacity);
+ return offsetof(struct stack_depot_trie_retired_children, data) +
+ ALIGN(size, sizeof(unsigned long));
+}
+
+static inline unsigned int trie_side_table_root_index(u32 id)
+{
+ return ((id - 1) / STACK_DEPOT_TRIE_SIDE_TABLE_CHUNK_SIZE) /
+ STACK_DEPOT_TRIE_SIDE_TABLE_DIR_SIZE;
+}
+
+static inline unsigned int trie_side_table_dir_index(u32 id)
+{
+ return ((id - 1) / STACK_DEPOT_TRIE_SIDE_TABLE_CHUNK_SIZE) %
+ STACK_DEPOT_TRIE_SIDE_TABLE_DIR_SIZE;
+}
+
+static inline unsigned int trie_side_table_slot_index(u32 id)
+{
+ return (id - 1) % STACK_DEPOT_TRIE_SIDE_TABLE_CHUNK_SIZE;
+}
+
+static struct stack_depot_trie_side_dir *trie_side_table_load_dir(unsigned int root)
+{
+ struct stack_depot_trie_side_root *root_vec;
+
+ root_vec = trie_side_table_root;
+ if (!root_vec || root >= root_vec->dir_capacity)
+ return NULL;
+ /* Pairs with side-table directory rcu_assign_pointer(). */
+ return rcu_dereference_check(root_vec->dirs[root],
+ lockdep_is_held(&stack_depot_trie_writer_lock) ||
+ rcu_read_lock_sched_held());
+}
+
+static inline const struct stack_depot_trie_node __rcu **
+trie_side_table_dir_load_chunk(struct stack_depot_trie_side_dir *dir,
+ unsigned int idx)
+{
+ /* Pairs with the chunk rcu_assign_pointer() in stack ID preparation. */
+ return rcu_dereference_check(dir->chunks[idx],
+ lockdep_is_held(&stack_depot_trie_writer_lock) ||
+ rcu_read_lock_sched_held());
+}
+
+/* Published capacity remains useful if insertion fails and needs no rollback. */
+static bool
+trie_side_table_try_take_cache(struct stack_depot_trie_side_prealloc *prealloc,
+ bool need_dir)
+{
+ bool taken = false;
+
+ lockdep_assert_held(&stack_depot_trie_writer_lock);
+ lockdep_assert_held(&pool_lock);
+
+ if (!raw_spin_trylock(&trie_side_table_cache_lock))
+ return false;
+ if ((!prealloc->chunk && !trie_side_table_cache.chunk) ||
+ (need_dir && !prealloc->dir && !trie_side_table_cache.dir))
+ goto out_unlock;
+
+ if (need_dir && !prealloc->dir) {
+ prealloc->dir = trie_side_table_cache.dir;
+ trie_side_table_cache.dir = NULL;
+ }
+ if (!prealloc->chunk) {
+ prealloc->chunk = trie_side_table_cache.chunk;
+ trie_side_table_cache.chunk = NULL;
+ }
+ taken = true;
+
+out_unlock:
+ raw_spin_unlock(&trie_side_table_cache_lock);
+ return taken;
+}
+
+static u32
+trie_side_table_prepare_stack_slot(struct stack_depot_trie_side_prealloc *prealloc)
+{
+ const struct stack_depot_trie_node __rcu **chunk;
+ struct stack_depot_trie_side_dir *dir;
+ struct stack_depot_trie_side_root *root_vec;
+ unsigned int root;
+ unsigned int idx;
+ u32 id;
+
+ lockdep_assert_held(&stack_depot_trie_writer_lock);
+ lockdep_assert_held(&pool_lock);
+
+ id = trie_side_table_last_stack_id + 1;
+ if (id > trie_max_stack_id())
+ return 0;
+
+ root_vec = trie_side_table_root;
+ root = trie_side_table_root_index(id);
+ dir = trie_side_table_load_dir(root);
+ if (!dir) {
+ if ((!prealloc->dir || !prealloc->chunk) &&
+ !trie_side_table_try_take_cache(prealloc, true))
+ return 0;
+ dir = prealloc->dir;
+ prealloc->dir = NULL;
+ /* Publish the zeroed directory before readers can load it locklessly. */
+ rcu_assign_pointer(root_vec->dirs[root], dir);
+ }
+
+ idx = trie_side_table_dir_index(id);
+ chunk = trie_side_table_dir_load_chunk(dir, idx);
+ if (!chunk) {
+ if (!prealloc->chunk &&
+ !trie_side_table_try_take_cache(prealloc, false))
+ return 0;
+ chunk = prealloc->chunk;
+ prealloc->chunk = NULL;
+ rcu_assign_pointer(dir->chunks[idx], chunk);
+ }
+
+ return id;
+}
+
+static inline unsigned int trie_side_table_root_size_for_max_id(u32 max_stack_id)
+{
+ unsigned int top_size;
+
+ top_size = DIV_ROUND_UP(max_stack_id, STACK_DEPOT_TRIE_SIDE_TABLE_CHUNK_SIZE);
+ return DIV_ROUND_UP(top_size, STACK_DEPOT_TRIE_SIDE_TABLE_DIR_SIZE);
+}
+
+static int __init stack_depot_trie_init_memblock(void)
+{
+ struct stack_depot_trie_side_root *root_vec;
+ struct stack_depot_trie_side_dir *first_dir;
+ const struct stack_depot_trie_node __rcu **first_chunk;
+ size_t root_bytes;
+ u32 max_stack_id;
+ unsigned int root_size;
+
+ max_stack_id = trie_max_stack_id();
+ if (!max_stack_id)
+ return -EINVAL;
+ root_size = trie_side_table_root_size_for_max_id(max_stack_id);
+ root_bytes = struct_size_t(struct stack_depot_trie_side_root, dirs, root_size);
+
+ root_vec = memblock_alloc(root_bytes, __alignof__(*root_vec));
+ if (!root_vec)
+ return -ENOMEM;
+ first_dir = memblock_alloc(PAGE_SIZE, PAGE_SIZE);
+ if (!first_dir) {
+ memblock_free(root_vec, root_bytes);
+ return -ENOMEM;
+ }
+ first_chunk = memblock_alloc(PAGE_SIZE, PAGE_SIZE);
+ if (!first_chunk) {
+ memblock_free(first_dir, PAGE_SIZE);
+ memblock_free(root_vec, root_bytes);
+ return -ENOMEM;
+ }
+
+ root_vec->dir_capacity = root_size;
+ RCU_INIT_POINTER(root_vec->dirs[0], first_dir);
+ RCU_INIT_POINTER(first_dir->chunks[0], first_chunk);
+ trie_side_table_root = root_vec;
+ static_branch_enable(&stack_depot_trie_enabled);
+ return 0;
+}
+
+static int stack_depot_trie_init(void)
+{
+ struct stack_depot_trie_side_root *root_vec;
+ unsigned int root_size;
+ size_t root_bytes;
+ u32 max_stack_id;
+
+ max_stack_id = trie_max_stack_id();
+ if (!max_stack_id)
+ return -EINVAL;
+
+ root_size = trie_side_table_root_size_for_max_id(max_stack_id);
+ root_bytes = struct_size_t(struct stack_depot_trie_side_root, dirs, root_size);
+ root_vec = kvzalloc(root_bytes, GFP_KERNEL);
+ if (!root_vec)
+ return -ENOMEM;
+
+ root_vec->dir_capacity = root_size;
+ trie_side_table_root = root_vec;
+ static_branch_enable(&stack_depot_trie_enabled);
+ return 0;
+}
+
+static int trie_side_table_get_prealloc(gfp_t gfp_flags,
+ struct stack_depot_trie_side_prealloc *prealloc)
+{
+ unsigned long flags;
+
+ gfp_flags = gfp_nested_mask(gfp_flags);
+ raw_spin_lock_irqsave(&trie_side_table_cache_lock, flags);
+ prealloc->dir = trie_side_table_cache.dir;
+ prealloc->chunk = trie_side_table_cache.chunk;
+ trie_side_table_cache.dir = NULL;
+ trie_side_table_cache.chunk = NULL;
+ raw_spin_unlock_irqrestore(&trie_side_table_cache_lock, flags);
+
+ if (!prealloc->dir) {
+ prealloc->dir = (void *)get_zeroed_page(gfp_flags);
+ if (!prealloc->dir)
+ return -ENOMEM;
+ }
+ if (!prealloc->chunk) {
+ prealloc->chunk = (void *)get_zeroed_page(gfp_flags);
+ if (!prealloc->chunk)
+ return -ENOMEM;
+ }
+
+ return 0;
+}
+
+static void trie_side_table_put_prealloc(struct stack_depot_trie_side_prealloc *prealloc)
+{
+ unsigned long flags;
+
+ raw_spin_lock_irqsave(&trie_side_table_cache_lock, flags);
+ if (!trie_side_table_cache.dir) {
+ trie_side_table_cache.dir = prealloc->dir;
+ prealloc->dir = NULL;
+ }
+ if (!trie_side_table_cache.chunk) {
+ trie_side_table_cache.chunk = prealloc->chunk;
+ prealloc->chunk = NULL;
+ }
+ raw_spin_unlock_irqrestore(&trie_side_table_cache_lock, flags);
+
+ if (prealloc->dir)
+ free_page((unsigned long)prealloc->dir);
+ if (prealloc->chunk)
+ free_page((unsigned long)prealloc->chunk);
+}
+
+static const struct stack_depot_trie_node *trie_side_table_lookup(u32 id)
+{
+ const struct stack_depot_trie_node __rcu **chunk;
+ struct stack_depot_trie_side_dir *dir;
+ unsigned int root;
+
+ root = trie_side_table_root_index(id);
+ dir = trie_side_table_load_dir(root);
+ if (!dir)
+ return NULL;
+ chunk = trie_side_table_dir_load_chunk(dir, trie_side_table_dir_index(id));
+ if (!chunk)
+ return NULL;
+
+ /* Pairs with side-table node publication. */
+ return rcu_dereference_check(chunk[trie_side_table_slot_index(id)],
+ lockdep_is_held(&stack_depot_trie_writer_lock) ||
+ rcu_read_lock_sched_held());
+}
+
+static inline struct stack_depot_trie_retired_children *
+trie_retired_children(const void *ptr)
+{
+ return container_of(ptr, struct stack_depot_trie_retired_children, data);
+}
+
+static bool depot_init_pool(void **prealloc);
+
+static unsigned int trie_pool_reserve_slots(struct stack_depot_trie_pool *pool,
+ unsigned int nr_slots)
+{
+ unsigned int start = pool->next_slot;
+ unsigned int run = 0;
+ unsigned int longest_run = 0;
+ unsigned int i;
+ unsigned int slot;
+
+scan:
+ run = 0;
+ longest_run = 0;
+ for (slot = start; slot < STACK_DEPOT_TRIE_POOL_SLOTS; slot++) {
+ if (pool->used[slot / BITS_PER_LONG] &
+ BIT(slot % BITS_PER_LONG)) {
+ run = 0;
+ continue;
+ }
+ run++;
+ longest_run = max(longest_run, run);
+ if (run != nr_slots)
+ continue;
+
+ for (i = slot + 1 - nr_slots; i <= slot; i++)
+ pool->used[i / BITS_PER_LONG] |= BIT(i % BITS_PER_LONG);
+ pool->free_slots -= nr_slots;
+ if (slot + 1 == STACK_DEPOT_TRIE_POOL_SLOTS)
+ pool->next_slot = STACK_DEPOT_TRIE_POOL_FIRST_SLOT;
+ else
+ pool->next_slot = slot + 1;
+ return slot + 1 - nr_slots;
+ }
+
+ if (start != STACK_DEPOT_TRIE_POOL_FIRST_SLOT) {
+ /* Keep holes and runs crossing the cursor visible. */
+ start = STACK_DEPOT_TRIE_POOL_FIRST_SLOT;
+ goto scan;
+ }
+
+ pool->free_run_upper_bound = longest_run;
+ return STACK_DEPOT_TRIE_POOL_SLOTS;
+}
+
+/* Allocate at least @size bytes from one contiguous trie-pool slot run. */
+static void *trie_pool_alloc(size_t size, void **prealloc)
+{
+ struct stack_depot_trie_pool *pool;
+ unsigned int nr_slots;
+ unsigned int slot;
+
+ lockdep_assert_held(&pool_lock);
+
+ if (size > STACK_DEPOT_TRIE_POOL_USABLE_SIZE)
+ return NULL;
+ nr_slots = DIV_ROUND_UP(size, STACK_DEPOT_TRIE_SLOT_SIZE);
+ list_for_each_entry_reverse(pool, &stack_depot_trie_pools, list) {
+ if (pool->free_slots < nr_slots ||
+ pool->free_run_upper_bound < nr_slots)
+ continue;
+ slot = trie_pool_reserve_slots(pool, nr_slots);
+ if (slot != STACK_DEPOT_TRIE_POOL_SLOTS)
+ return (char *)pool + slot * STACK_DEPOT_TRIE_SLOT_SIZE;
+ }
+
+ if (!depot_init_pool(prealloc))
+ return NULL;
+ pool = stack_pools[pools_num - 1];
+ /* Keep hash records out of this bitmap-owned pool. */
+ pool_offset = DEPOT_POOL_SIZE;
+ memset(pool, 0, sizeof(*pool));
+ pool->free_slots = STACK_DEPOT_TRIE_POOL_SLOTS -
+ STACK_DEPOT_TRIE_POOL_FIRST_SLOT;
+ pool->free_run_upper_bound = pool->free_slots;
+ pool->next_slot = STACK_DEPOT_TRIE_POOL_FIRST_SLOT;
+ list_add_tail(&pool->list, &stack_depot_trie_pools);
+
+ slot = trie_pool_reserve_slots(pool, nr_slots);
+ return (char *)pool + slot * STACK_DEPOT_TRIE_SLOT_SIZE;
+}
+
+/* Release the slots for the byte count originally passed to allocation. */
+static void trie_pool_release(const void *ptr, size_t size)
+{
+ struct stack_depot_trie_pool *pool;
+ unsigned long pfn;
+ unsigned int nr_slots;
+ unsigned int slot;
+ unsigned int i;
+
+ lockdep_assert_held(&pool_lock);
+
+ pfn = page_to_pfn(virt_to_page(ptr));
+ pfn &= ~(BIT(DEPOT_POOL_ORDER) - 1);
+ pool = page_address(pfn_to_page(pfn));
+ slot = ((unsigned long)ptr - (unsigned long)pool) >> DEPOT_STACK_ALIGN;
+ nr_slots = DIV_ROUND_UP(size, STACK_DEPOT_TRIE_SLOT_SIZE);
+ for (i = slot; i < slot + nr_slots; i++)
+ pool->used[i / BITS_PER_LONG] &= ~BIT(i % BITS_PER_LONG);
+ pool->free_slots += nr_slots;
+ /* A release can join at most two runs bounded by the old value. */
+ pool->free_run_upper_bound = min(pool->free_slots,
+ 2 * pool->free_run_upper_bound + nr_slots);
+}
+
+static struct stack_depot_trie_children *
+trie_pool_alloc_children(unsigned int capacity, void **prealloc)
+{
+ struct stack_depot_trie_retired_children *retired;
+ struct stack_depot_trie_children *children;
+
+ /* Capacity counts child-pointer entries; allocation includes RCU metadata. */
+ retired = trie_pool_alloc(trie_children_alloc_size(capacity), prealloc);
+ if (!retired)
+ return NULL;
+
+ children = (void *)retired->data;
+ children->nr_children = 0;
+ children->capacity = capacity;
+ return children;
+}
+
+static void
+trie_pool_release_children(const struct stack_depot_trie_children *children)
+{
+ /* Capacity is immutable and therefore recovers the allocation byte size. */
+ trie_pool_release(trie_retired_children(children),
+ trie_children_alloc_size(children->capacity));
+}
+
+/*
+ * Return RCU-ready objects before allocating. Pending children are FIFO, so
+ * stop at the first incomplete grace period. A replaced node shares the same
+ * retirement cookie and is released with its former children container.
+ */
+static void trie_drain_pending_children(void)
+{
+ struct stack_depot_trie_retired_children *retired;
+ struct stack_depot_trie_retired_children *tmp;
+ struct stack_depot_trie_children *children;
+
+ lockdep_assert_held(&pool_lock);
+
+ list_for_each_entry_safe(retired, tmp, &pending_trie_children, list) {
+ if (!poll_state_synchronize_rcu(retired->rcu_state))
+ break;
+ children = (void *)retired->data;
+ list_del(&retired->list);
+ if (retired->pending_node)
+ trie_pool_release(retired->pending_node,
+ trie_node_bytes(&retired->pending_node->run));
+ trie_pool_release_children(children);
+ }
+}
+
+static void trie_retire_children(const struct stack_depot_trie_children *children)
+{
+ struct stack_depot_trie_retired_children *retired;
+
+ lockdep_assert_held(&pool_lock);
+
+ retired = trie_retired_children(children);
+ retired->pending_node = NULL;
+ retired->rcu_state = get_state_synchronize_rcu();
+ list_add_tail(&retired->list, &pending_trie_children);
+}
+
+static void
+trie_retire_children_with_node(const struct stack_depot_trie_children *children,
+ const struct stack_depot_trie_node *node)
+{
+ struct stack_depot_trie_retired_children *retired;
+
+ lockdep_assert_held(&stack_depot_trie_writer_lock);
+ lockdep_assert_held(&pool_lock);
+ trie_retire_children(children);
+ retired = trie_retired_children(children);
+ retired->pending_node = node;
+}
+
+static const struct stack_depot_trie_node *
+stack_depot_trie_lookup(const unsigned long *entries, unsigned int nr_entries);
+
+static depot_stack_handle_t
+trie_find_handle(const unsigned long *entries, unsigned int nr_entries)
+{
+ depot_stack_handle_t handle = 0;
+ const struct stack_depot_trie_node *node;
+
+ rcu_read_lock_sched_notrace();
+ node = stack_depot_trie_lookup(entries, nr_entries);
+ if (node)
+ handle = trie_handle(node->stack_id);
+ rcu_read_unlock_sched_notrace();
+
+ return handle;
+}
+
+/*
+ * Publish only after the node and its path are fully initialized and all
+ * fallible allocation is complete. Publication commits the path, so it cannot
+ * then be rolled back. Side-table mappings must precede trie topology
+ * publication that makes new or remapped nodes reachable from lookup.
+ * Published storage remains valid until RCU retirement; only descendant parent
+ * links may change meanwhile.
+ */
+static void trie_side_table_publish(const struct stack_depot_trie_node *node)
+{
+ const struct stack_depot_trie_node __rcu **chunk;
+ struct stack_depot_trie_side_dir *dir;
+ u32 stack_id = node->stack_id;
+
+ lockdep_assert_held(&stack_depot_trie_writer_lock);
+
+ dir = trie_side_table_load_dir(trie_side_table_root_index(stack_id));
+ chunk = trie_side_table_dir_load_chunk(dir,
+ trie_side_table_dir_index(stack_id));
+ /* Pairs with trie_side_table_lookup(). */
+ rcu_assign_pointer(chunk[trie_side_table_slot_index(stack_id)], node);
+}
+
static int __init disable_stack_depot(char *str)
{
return kstrtobool(str, &stack_depot_disabled);
@@ -146,7 +844,7 @@ static void init_stack_table(unsigned long entries)
INIT_LIST_HEAD(&stack_table[i]);
}
-/* Allocates a hash table via memblock. Can only be used during early boot. */
+/* Initializes hash and optional trie storage during early boot. */
int __init stack_depot_early_init(void)
{
unsigned long entries = 0;
@@ -220,11 +918,15 @@ int __init stack_depot_early_init(void)
stack_depot_disabled = true;
return -ENOMEM;
}
+ if (stack_depot_trie_requested && stack_depot_trie_init_memblock()) {
+ pr_warn("trie storage initialization failed, disabling trie storage\n");
+ stack_depot_trie_requested = false;
+ }
return 0;
}
-/* Allocates a hash table via kvcalloc. Can be used after boot. */
+/* Initializes hash and optional trie storage after boot. */
int stack_depot_init(void)
{
static DEFINE_MUTEX(stack_depot_init_mutex);
@@ -278,6 +980,15 @@ int stack_depot_init(void)
kvfree(stack_table);
stack_depot_disabled = true;
ret = -ENOMEM;
+ goto out_unlock;
+ }
+ if (stack_depot_trie_requested) {
+ ret = stack_depot_trie_init();
+ if (ret) {
+ pr_warn("trie storage initialization failed, disabling trie storage\n");
+ stack_depot_trie_requested = false;
+ ret = 0;
+ }
}
out_unlock:
@@ -323,7 +1034,7 @@ static bool depot_init_pool(void **prealloc)
* NULL; do not reset to NULL if we have reached the maximum number of
* pools.
*/
- if (pools_num < stack_max_pools)
+ if (pools_num + 1 < stack_max_pools)
WRITE_ONCE(new_pool, NULL);
else
WRITE_ONCE(new_pool, STACK_DEPOT_POISON);
@@ -467,6 +1178,7 @@ depot_alloc_stack(unsigned long *entries, unsigned int nr_entries, u32 hash, dep
/* Save the stack trace. */
stack->hash = hash;
stack->size = nr_entries;
+ stack->flags = flags & STACK_DEPOT_FLAG_COUNTABLE;
/* stack->handle is already filled in by depot_pop_free_pool(). */
memcpy(stack->entries, entries, flex_array_size(stack, entries, nr_entries));
@@ -609,6 +1321,9 @@ static inline struct stack_record *find_stack(struct list_head *bucket,
list_for_each_entry_rcu(stack, bucket, hash_list) {
if (stack->hash != hash || stack->size != size)
continue;
+ /* Page owner countable records have a distinct count lifetime. */
+ if ((stack->flags ^ flags) & STACK_DEPOT_FLAG_COUNTABLE)
+ continue;
/*
* This may race with depot_free_stack() accessing the freelist
@@ -638,6 +1353,101 @@ static inline struct stack_record *find_stack(struct list_head *bucket,
return ret;
}
+static u32
+stack_depot_trie_insert(const unsigned long *entries,
+ unsigned int nr_entries, void **pool_prealloc,
+ struct stack_depot_trie_side_prealloc *side_prealloc);
+
+static depot_stack_handle_t
+stack_depot_trie_save(unsigned long *entries, unsigned int nr_entries,
+ gfp_t alloc_flags)
+{
+ unsigned int attempt;
+
+ /* Allow one stale pool hint before the two pools a largest insert needs. */
+ for (attempt = 0; attempt < 3; attempt++) {
+ struct stack_depot_trie_side_prealloc side_prealloc = {};
+ void *pool_prealloc = NULL;
+ depot_stack_handle_t handle;
+ unsigned long flags;
+ struct page *page;
+ u32 stack_id = 0;
+
+ handle = trie_find_handle(entries, nr_entries);
+ if (handle)
+ return handle;
+
+ if (trie_side_table_get_prealloc(alloc_flags, &side_prealloc)) {
+ trie_side_table_put_prealloc(&side_prealloc);
+ return 0;
+ }
+
+ /* The hint may race; a missing page is recovered by the retry. */
+ if (!READ_ONCE(new_pool)) {
+ page = alloc_pages(gfp_nested_mask(alloc_flags),
+ DEPOT_POOL_ORDER);
+ if (page)
+ pool_prealloc = page_address(page);
+ }
+
+ raw_spin_lock_irqsave(&stack_depot_trie_writer_lock, flags);
+ raw_spin_lock(&pool_lock);
+ printk_deferred_enter();
+ trie_drain_pending_children();
+ stack_id = stack_depot_trie_insert(entries, nr_entries,
+ &pool_prealloc, &side_prealloc);
+ if (pool_prealloc)
+ depot_keep_new_pool(&pool_prealloc);
+ printk_deferred_exit();
+ raw_spin_unlock(&pool_lock);
+ raw_spin_unlock_irqrestore(&stack_depot_trie_writer_lock, flags);
+
+ if (pool_prealloc)
+ free_pages((unsigned long)pool_prealloc, DEPOT_POOL_ORDER);
+ trie_side_table_put_prealloc(&side_prealloc);
+ if (stack_id)
+ return trie_handle(stack_id);
+ }
+
+ return 0;
+}
+
+static depot_stack_handle_t
+stack_depot_trie_save_constrained(unsigned long *entries,
+ unsigned int nr_entries, bool trylock)
+{
+ struct stack_depot_trie_side_prealloc side_prealloc = {};
+ void *pool_prealloc = NULL;
+ depot_stack_handle_t handle;
+ unsigned long flags;
+ u32 stack_id;
+
+ handle = trie_find_handle(entries, nr_entries);
+ if (handle)
+ return handle;
+
+ if (trylock) {
+ if (!raw_spin_trylock_irqsave(&stack_depot_trie_writer_lock, flags))
+ return 0;
+ if (!raw_spin_trylock(&pool_lock)) {
+ raw_spin_unlock_irqrestore(&stack_depot_trie_writer_lock, flags);
+ return 0;
+ }
+ } else {
+ raw_spin_lock_irqsave(&stack_depot_trie_writer_lock, flags);
+ raw_spin_lock(&pool_lock);
+ }
+
+ printk_deferred_enter();
+ stack_id = stack_depot_trie_insert(entries, nr_entries, &pool_prealloc,
+ &side_prealloc);
+ printk_deferred_exit();
+ raw_spin_unlock(&pool_lock);
+ raw_spin_unlock_irqrestore(&stack_depot_trie_writer_lock, flags);
+
+ return stack_id ? trie_handle(stack_id) : 0;
+}
+
depot_stack_handle_t stack_depot_save_flags(unsigned long *entries,
unsigned int nr_entries,
gfp_t alloc_flags,
@@ -655,6 +1465,9 @@ depot_stack_handle_t stack_depot_save_flags(unsigned long *entries,
if (WARN_ON(depot_flags & ~STACK_DEPOT_FLAGS_MASK))
return 0;
+ if (WARN_ON_ONCE((depot_flags & STACK_DEPOT_FLAG_GET) &&
+ (depot_flags & STACK_DEPOT_FLAG_COUNTABLE)))
+ return 0;
/*
* If this stack trace is from an interrupt, including anything before
@@ -669,6 +1482,20 @@ depot_stack_handle_t stack_depot_save_flags(unsigned long *entries,
if (unlikely(nr_entries == 0) || stack_depot_disabled)
return 0;
+ if (!(depot_flags & (STACK_DEPOT_FLAG_GET | STACK_DEPOT_FLAG_COUNTABLE)) &&
+ static_branch_unlikely(&stack_depot_trie_enabled)) {
+ if (nr_entries > CONFIG_STACKDEPOT_MAX_FRAMES)
+ nr_entries = CONFIG_STACKDEPOT_MAX_FRAMES;
+ if (in_nmi()) {
+ WARN_ON_ONCE(can_alloc);
+ return trie_find_handle(entries, nr_entries);
+ }
+ if (!can_alloc)
+ return stack_depot_trie_save_constrained(entries, nr_entries,
+ !allow_spin);
+ return stack_depot_trie_save(entries, nr_entries, alloc_flags);
+ }
+
hash = hash_stack(entries, nr_entries);
bucket = &stack_table[hash & stack_hash_mask];
@@ -751,10 +1578,728 @@ EXPORT_SYMBOL_GPL(stack_depot_save);
struct stack_record *__stack_depot_get_stack_record(depot_stack_handle_t handle)
{
+ struct stack_record *stack;
+
if (!handle)
return NULL;
+ if (WARN_ON_ONCE(stack_depot_handle_is_trie(handle)))
+ return NULL;
+
+ stack = depot_fetch_stack(handle);
+ if (!stack)
+ return NULL;
+ if (WARN_ON_ONCE(!(stack->flags & STACK_DEPOT_FLAG_COUNTABLE)))
+ return NULL;
+
+ return stack;
+}
+
+static void frame_run_init(const unsigned long *entries,
+ unsigned int nr_entries,
+ struct stack_depot_frame_run *run)
+{
+ u32 payload;
+ unsigned int i;
+ bool compressed;
+
+ compressed = arch_stack_depot_frame_try_compress(entries[0], &payload);
+ for (i = 1; i < nr_entries; i++) {
+ bool next;
+
+ next = arch_stack_depot_frame_try_compress(entries[i], &payload);
+ if (next != compressed)
+ break;
+ }
+
+ /* @i is the first non-matching frame, or @nr_entries if all matched. */
+ run->mode = compressed ? STACK_DEPOT_FRAME_COMPRESSED : STACK_DEPOT_FRAME_RAW;
+ run->nr_entries = i;
+}
+
+static void
+stack_depot_trie_node_frame(const struct stack_depot_trie_node *node,
+ unsigned int index, unsigned long *frame)
+{
+ u32 payload;
+
+ if (node->run.mode == STACK_DEPOT_FRAME_RAW) {
+ memcpy(frame, node->data + index * sizeof(*frame),
+ sizeof(*frame));
+ return;
+ }
+
+ memcpy(&payload, node->data + index * sizeof(payload), sizeof(payload));
+ arch_stack_depot_frame_decompress(payload, frame);
+}
+
+static void trie_node_init(struct stack_depot_trie_node *node,
+ const struct stack_depot_trie_node *parent, u32 stack_id,
+ const unsigned long *entries,
+ const struct stack_depot_frame_run *run)
+{
+ if (run->mode == STACK_DEPOT_FRAME_COMPRESSED) {
+ unsigned int i;
+
+ for (i = 0; i < run->nr_entries; i++) {
+ u32 payload;
+
+ arch_stack_depot_frame_try_compress(entries[i], &payload);
+ memcpy(node->data + i * sizeof(payload), &payload,
+ sizeof(payload));
+ }
+ } else {
+ memcpy(node->data, entries, stack_depot_frame_run_bytes(run));
+ }
+
+ RCU_INIT_POINTER(node->parent, parent);
+ RCU_INIT_POINTER(node->children, NULL);
+ node->stack_id = stack_id;
+ node->run = *run;
+}
+
+static void trie_node_init_slice(struct stack_depot_trie_node *node,
+ const struct stack_depot_trie_node *parent, u32 stack_id,
+ const struct stack_depot_trie_node *src_node,
+ unsigned int start, unsigned int nr_entries)
+{
+ struct stack_depot_frame_run run;
+ size_t entry_bytes;
+
+ run = src_node->run;
+ run.nr_entries = nr_entries;
+
+ entry_bytes = stack_depot_frame_run_entry_bytes(src_node->run.mode);
+ memcpy(node->data, src_node->data + start * entry_bytes,
+ stack_depot_frame_run_bytes(&run));
+ RCU_INIT_POINTER(node->parent, parent);
+ RCU_INIT_POINTER(node->children, NULL);
+ node->stack_id = stack_id;
+ node->run = run;
+}
+
+static unsigned int trie_node_match(const struct stack_depot_trie_node *node,
+ const unsigned long *entries,
+ unsigned int nr_entries)
+{
+ unsigned int limit;
+ unsigned int i;
+
+ limit = min(node->run.nr_entries, nr_entries);
+ if (node->run.mode == STACK_DEPOT_FRAME_RAW) {
+ for (i = 0; i < limit; i++) {
+ unsigned long frame;
+
+ memcpy(&frame, node->data + i * sizeof(frame), sizeof(frame));
+ if (frame != entries[i])
+ break;
+ }
+
+ return i;
+ }
+
+ for (i = 0; i < limit; i++) {
+ unsigned long frame;
+
+ stack_depot_trie_node_frame(node, i, &frame);
+ if (frame != entries[i])
+ break;
+ }
+
+ return i;
+}
+
+static inline const struct stack_depot_trie_node *
+trie_load_parent(const struct stack_depot_trie_node *node)
+{
+ return rcu_dereference_check(node->parent,
+ lockdep_is_held(&stack_depot_trie_writer_lock) ||
+ rcu_read_lock_sched_held());
+}
+
+static inline const struct stack_depot_trie_children *
+trie_load_children(const struct stack_depot_trie_children __rcu * const *slot)
+{
+ return rcu_dereference_check(*slot,
+ lockdep_is_held(&stack_depot_trie_writer_lock) ||
+ rcu_read_lock_sched_held());
+}
+
+static inline const struct stack_depot_trie_node *
+trie_children_load_child(const struct stack_depot_trie_children *children,
+ unsigned int pos)
+{
+ return rcu_dereference_check(children->nodes[pos],
+ lockdep_is_held(&stack_depot_trie_writer_lock) ||
+ rcu_read_lock_sched_held());
+}
+
+static bool
+trie_children_find_position(const struct stack_depot_trie_children *children,
+ unsigned long frame, unsigned int *pos)
+{
+ unsigned int left = 0;
+ unsigned int right;
+
+ right = READ_ONCE(children->nr_children);
+ while (left < right) {
+ unsigned int mid = left + (right - left) / 2;
+ const struct stack_depot_trie_node *node;
+ unsigned long mid_frame;
+
+ node = trie_children_load_child(children, mid);
+ if (!node) {
+ /* Tail append may produce a transient lockless lookup miss. */
+ right = mid;
+ continue;
+ }
+ stack_depot_trie_node_frame(node, 0, &mid_frame);
+ if (mid_frame < frame) {
+ left = mid + 1;
+ } else if (mid_frame > frame) {
+ right = mid;
+ } else {
+ *pos = mid;
+ return true;
+ }
+ }
+
+ *pos = left;
+ return false;
+}
+
+/* Initialize an unpublished container from a stable published prefix. */
+static void trie_children_init(const struct stack_depot_trie_children *old,
+ struct stack_depot_trie_children *new)
+{
+ unsigned int nr_old = old->nr_children;
+ unsigned int i;
+
+ new->nr_children = nr_old;
+ for (i = 0; i < nr_old; i++)
+ RCU_INIT_POINTER(new->nodes[i], trie_children_load_child(old, i));
+ for (i = nr_old; i < new->capacity; i++)
+ RCU_INIT_POINTER(new->nodes[i], NULL);
+}
+
+static void trie_children_insert(struct stack_depot_trie_children *children,
+ const struct stack_depot_trie_node *node,
+ unsigned int pos)
+{
+ unsigned int i;
+
+ for (i = children->nr_children; i > pos; i--)
+ RCU_INIT_POINTER(children->nodes[i],
+ trie_children_load_child(children, i - 1));
+ RCU_INIT_POINTER(children->nodes[pos], node);
+ children->nr_children++;
+}
+
+static void trie_reparent_children(struct stack_depot_trie_node *parent)
+{
+ const struct stack_depot_trie_children *children;
+ unsigned int i;
+
+ lockdep_assert_held(&stack_depot_trie_writer_lock);
+
+ children = trie_load_children(&parent->children);
+ if (!children)
+ return;
+ /*
+ * Replacement nodes reuse unchanged descendant subtrees. Repoint their
+ * parent links before retiring the old parent so fetch never follows a freed
+ * node. Lockless fetches may see the new parent before publication, but the
+ * old and new parent chains contain the same frames and remain RCU-live.
+ */
+ for (i = 0; i < children->nr_children; i++) {
+ struct stack_depot_trie_node *child;
+
+ child = (struct stack_depot_trie_node *)trie_children_load_child(children, i);
+ rcu_assign_pointer(child->parent, parent);
+ }
+}
+
+/*
+ * Split entries into runs, allocate and initialize each node once, and link
+ * adjacent nodes through singleton children. Both trie locks must be held.
+ * Failure walks the unpublished parent chain and releases local ownership.
+ */
+static const struct stack_depot_trie_node *
+trie_path_alloc(const struct stack_depot_trie_node *parent, u32 stack_id,
+ const unsigned long *entries, unsigned int nr_entries,
+ void **pool_prealloc,
+ const struct stack_depot_trie_node **node_out)
+{
+ struct stack_depot_trie_children *path_children = NULL;
+ const struct stack_depot_trie_node *path_root = NULL;
+ const struct stack_depot_trie_node *last_node = parent;
+ unsigned int entry = 0;
+
+ lockdep_assert_held(&pool_lock);
+ lockdep_assert_held(&stack_depot_trie_writer_lock);
+
+ while (entry < nr_entries) {
+ struct stack_depot_frame_run run;
+ struct stack_depot_trie_node *node;
+
+ frame_run_init(&entries[entry], nr_entries - entry, &run);
+ node = trie_pool_alloc(trie_node_bytes(&run), pool_prealloc);
+ if (!node)
+ goto err_release;
+
+ trie_node_init(node, last_node,
+ entry + run.nr_entries == nr_entries ? stack_id : 0,
+ &entries[entry], &run);
+ entry += run.nr_entries;
+ last_node = node;
+ if (!path_root)
+ path_root = node;
+
+ if (path_children)
+ trie_children_insert(path_children, last_node, 0);
+ if (entry < nr_entries) {
+ path_children = trie_pool_alloc_children(1, pool_prealloc);
+ if (!path_children)
+ goto err_release;
+ RCU_INIT_POINTER(node->children, path_children);
+ }
+ }
+
+ *node_out = last_node;
+ return path_root;
+
+err_release:
+ while (last_node != parent) {
+ const struct stack_depot_trie_children *node_children;
+ const struct stack_depot_trie_node *node = last_node;
+
+ last_node = trie_load_parent(node);
+ node_children = trie_load_children(&node->children);
+ if (node_children)
+ trie_pool_release_children(node_children);
+ trie_pool_release(node, trie_node_bytes(&node->run));
+ }
+ return NULL;
+}
+
+static const struct stack_depot_trie_node *
+stack_depot_trie_lookup(const unsigned long *entries, unsigned int nr_entries)
+{
+ const struct stack_depot_trie_children *children;
+ unsigned int entry = 0;
+
+ children = trie_load_children(&stack_depot_trie_root);
+
+ while (entry < nr_entries) {
+ const struct stack_depot_trie_node *node;
+ unsigned int remaining = nr_entries - entry;
+ unsigned int matched;
+ unsigned int pos;
+
+ if (!children)
+ return NULL;
+ if (!trie_children_find_position(children, entries[entry], &pos))
+ return NULL;
+
+ node = trie_children_load_child(children, pos);
+ matched = trie_node_match(node, &entries[entry], remaining);
+ if (matched < node->run.nr_entries)
+ return NULL;
+ entry += matched;
+ if (entry == nr_entries)
+ return node->stack_id ? node : NULL;
+
+ children = trie_load_children(&node->children);
+ }
+
+ return NULL;
+}
+
+static u32
+trie_insert_path(const struct stack_depot_trie_children __rcu **slot,
+ struct stack_depot_trie_node *parent,
+ const struct stack_depot_trie_children *children,
+ unsigned int pos, const unsigned long *entries,
+ unsigned int nr_entries, void **pool_prealloc,
+ struct stack_depot_trie_side_prealloc *side_prealloc)
+{
+ struct stack_depot_trie_children *new_children = NULL;
+ const struct stack_depot_trie_node *path_root;
+ const struct stack_depot_trie_node *node;
+ unsigned int capacity = 1;
+ u32 new_stack_id;
+ bool tail_append = false;
+
+ /*
+ * Reuse spare capacity only for a sorted tail append. Other insertions
+ * replace the children container without modifying visible pointers.
+ */
+ if (children) {
+ capacity = roundup_pow_of_two(children->nr_children + 1);
+ tail_append = pos == children->nr_children &&
+ children->nr_children < children->capacity;
+ }
+ if (!tail_append && trie_children_alloc_size(capacity) >
+ STACK_DEPOT_TRIE_POOL_USABLE_SIZE)
+ return 0;
+
+ new_stack_id = trie_side_table_prepare_stack_slot(side_prealloc);
+ if (!new_stack_id)
+ return 0;
+
+ /* Reserve replacement topology before the path, the final fallible step. */
+ if (!tail_append) {
+ new_children = trie_pool_alloc_children(capacity, pool_prealloc);
+ if (!new_children)
+ goto err_release;
+ }
+ path_root = trie_path_alloc(parent, new_stack_id, entries, nr_entries,
+ pool_prealloc, &node);
+ if (!path_root)
+ goto err_release;
+
+ /* Commit the stack ID before making the path reachable from the trie. */
+ trie_side_table_publish(node);
+ if (tail_append) {
+ struct stack_depot_trie_children *tail_children =
+ (struct stack_depot_trie_children *)children;
+
+ /*
+ * Publish the node before the visible count. Readers may transiently
+ * see NULL and miss; the writer-lock recheck prevents duplicates.
+ */
+ rcu_assign_pointer(tail_children->nodes[pos], path_root);
+ WRITE_ONCE(tail_children->nr_children, pos + 1);
+ } else {
+ if (children)
+ trie_children_init(children, new_children);
+ trie_children_insert(new_children, path_root, pos);
+ rcu_assign_pointer(*slot, new_children);
+ if (children)
+ trie_retire_children(children);
+ }
+
+ return new_stack_id;
+
+err_release:
+ if (new_children)
+ trie_pool_release_children(new_children);
+ return 0;
+}
+
+static u32
+trie_split_child(const struct stack_depot_trie_children __rcu **slot,
+ const struct stack_depot_trie_children *children,
+ const struct stack_depot_trie_node *child,
+ unsigned int pos, unsigned int matched,
+ const unsigned long *entries, unsigned int nr_entries,
+ void **pool_prealloc,
+ struct stack_depot_trie_side_prealloc *side_prealloc)
+{
+ struct stack_depot_trie_children *prefix_children = NULL;
+ struct stack_depot_trie_children *new_children = NULL;
+ const struct stack_depot_trie_node *new_node;
+ const struct stack_depot_trie_node *suffix_roots[2];
+ struct stack_depot_frame_run run;
+ struct stack_depot_trie_node *split_prefix = NULL;
+ struct stack_depot_trie_node *old_suffix = NULL;
+ unsigned int nr_suffix_roots;
+ unsigned int old_suffix_len;
+ unsigned int i;
+ size_t split_prefix_size;
+ size_t old_suffix_size;
+ u32 new_stack_id;
+ bool has_new_suffix;
+
+ new_stack_id = trie_side_table_prepare_stack_slot(side_prealloc);
+ if (!new_stack_id)
+ return 0;
+
+ /* Rebuild the child's run as newly allocated prefix and old suffix nodes. */
+ run = child->run;
+ run.nr_entries = matched;
+ split_prefix_size = trie_node_bytes(&run);
+ old_suffix_len = child->run.nr_entries - matched;
+ run.nr_entries = old_suffix_len;
+ old_suffix_size = trie_node_bytes(&run);
+ has_new_suffix = matched < nr_entries;
+ nr_suffix_roots = has_new_suffix ? 2 : 1;
+
+ /* Reserve fixed split topology before the optional new suffix path. */
+ split_prefix = trie_pool_alloc(split_prefix_size, pool_prealloc);
+ if (!split_prefix)
+ goto err_release;
+ old_suffix = trie_pool_alloc(old_suffix_size, pool_prealloc);
+ if (!old_suffix)
+ goto err_release;
+ new_children = trie_pool_alloc_children(children->capacity, pool_prealloc);
+ if (!new_children)
+ goto err_release;
+ prefix_children = trie_pool_alloc_children(nr_suffix_roots, pool_prealloc);
+ if (!prefix_children)
+ goto err_release;
+
+ if (has_new_suffix) {
+ const struct stack_depot_trie_node *new_suffix;
+ unsigned long old_suffix_frame;
+
+ new_suffix = trie_path_alloc(split_prefix, new_stack_id,
+ &entries[matched], nr_entries - matched,
+ pool_prealloc, &new_node);
+ if (!new_suffix)
+ goto err_release;
+ stack_depot_trie_node_frame(child, matched, &old_suffix_frame);
+ /* Children remain sorted by the first frame of each suffix. */
+ if (old_suffix_frame < entries[matched]) {
+ suffix_roots[0] = old_suffix;
+ suffix_roots[1] = new_suffix;
+ } else {
+ suffix_roots[0] = new_suffix;
+ suffix_roots[1] = old_suffix;
+ }
+ } else {
+ new_node = split_prefix;
+ suffix_roots[0] = old_suffix;
+ }
+
+ /* Rebuild the old path as prefix -> old suffix and attach suffix roots. */
+ trie_node_init_slice(split_prefix, trie_load_parent(child),
+ has_new_suffix ? 0 : new_stack_id, child, 0, matched);
+ trie_node_init_slice(old_suffix, split_prefix, child->stack_id, child,
+ matched, old_suffix_len);
+ for (i = 0; i < nr_suffix_roots; i++)
+ trie_children_insert(prefix_children, suffix_roots[i], i);
+ RCU_INIT_POINTER(old_suffix->children,
+ trie_load_children(&child->children));
+ RCU_INIT_POINTER(split_prefix->children, prefix_children);
+
+ /* Publish IDs, reparent descendants, then replace and retire topology. */
+ if (child->stack_id)
+ trie_side_table_publish(old_suffix);
+ trie_side_table_publish(new_node);
+ /* Old and replacement chains contain identical frames during transition. */
+ trie_children_init(children, new_children);
+ RCU_INIT_POINTER(new_children->nodes[pos], split_prefix);
+ trie_reparent_children(old_suffix);
+ rcu_assign_pointer(*slot, new_children);
+ trie_retire_children_with_node(children, child);
+
+ return new_stack_id;
+
+err_release:
+ if (split_prefix)
+ trie_pool_release(split_prefix, split_prefix_size);
+ if (old_suffix)
+ trie_pool_release(old_suffix, old_suffix_size);
+ if (prefix_children)
+ trie_pool_release_children(prefix_children);
+ if (new_children)
+ trie_pool_release_children(new_children);
+ return 0;
+}
+
+static u32
+trie_promote_child(const struct stack_depot_trie_children __rcu **slot,
+ const struct stack_depot_trie_children *children,
+ const struct stack_depot_trie_node *child,
+ unsigned int pos, void **pool_prealloc,
+ struct stack_depot_trie_side_prealloc *side_prealloc)
+{
+ struct stack_depot_trie_children *new_children;
+ struct stack_depot_trie_node *promoted_node;
+ size_t node_size;
+ u32 new_stack_id;
+
+ new_stack_id = trie_side_table_prepare_stack_slot(side_prealloc);
+ if (!new_stack_id)
+ return 0;
+ node_size = trie_node_bytes(&child->run);
+
+ /* Reserve a clone and replacement children container before publication. */
+ promoted_node = trie_pool_alloc(node_size, pool_prealloc);
+ if (!promoted_node)
+ return 0;
+ new_children = trie_pool_alloc_children(children->capacity, pool_prealloc);
+ if (!new_children)
+ goto out_release_node;
+
+ /* Add the stack ID through a clone, then reparent before retirement. */
+ memcpy(promoted_node, child, node_size);
+ promoted_node->stack_id = new_stack_id;
+ trie_side_table_publish(promoted_node);
+ trie_children_init(children, new_children);
+ RCU_INIT_POINTER(new_children->nodes[pos], promoted_node);
+ trie_reparent_children(promoted_node);
+ rcu_assign_pointer(*slot, new_children);
+ trie_retire_children_with_node(children, child);
+
+ return new_stack_id;
+
+out_release_node:
+ trie_pool_release(promoted_node, node_size);
+ return 0;
+}
+
+static u32
+stack_depot_trie_insert(const unsigned long *entries,
+ unsigned int nr_entries, void **pool_prealloc,
+ struct stack_depot_trie_side_prealloc *side_prealloc)
+{
+ const struct stack_depot_trie_children *children;
+ const struct stack_depot_trie_children __rcu **slot =
+ &stack_depot_trie_root;
+ const struct stack_depot_trie_node *child;
+ struct stack_depot_trie_node *parent = NULL;
+ unsigned int matched;
+ unsigned int pos;
+ u32 stack_id;
+
+ lockdep_assert_held(&stack_depot_trie_writer_lock);
+ lockdep_assert_held(&pool_lock);
+
+ for (;;) {
+ pos = 0;
+ children = trie_load_children(slot);
+ /* No matching child: attach the remaining path. */
+ if (!children ||
+ !trie_children_find_position(children, entries[0], &pos)) {
+ stack_id = trie_insert_path(slot, parent, children, pos,
+ entries, nr_entries, pool_prealloc,
+ side_prealloc);
+ break;
+ }
+
+ child = trie_children_load_child(children, pos);
+ matched = trie_node_match(child, entries, nr_entries);
+ /* A partial child match requires a prefix/suffix split. */
+ if (matched < child->run.nr_entries) {
+ stack_id = trie_split_child(slot, children, child, pos,
+ matched, entries, nr_entries,
+ pool_prealloc, side_prealloc);
+ break;
+ }
+
+ /* The input ends here: reuse a stack node or promote an internal one. */
+ if (matched == nr_entries) {
+ if (child->stack_id)
+ return child->stack_id;
+ stack_id = trie_promote_child(slot, children, child, pos,
+ pool_prealloc, side_prealloc);
+ break;
+ }
+
+ /* The child matched completely; continue with the remaining frames. */
+ parent = (struct stack_depot_trie_node *)child;
+ slot = &parent->children;
+ entries += matched;
+ nr_entries -= matched;
+ }
+
+ if (stack_id)
+ trie_side_table_last_stack_id = stack_id;
+ return stack_id;
+}
+
+static unsigned int trie_fetch_into(const struct stack_depot_trie_node *node,
+ unsigned long *entries,
+ unsigned int max_entries)
+{
+ const struct stack_depot_trie_node *cur;
+ unsigned int total;
+ unsigned int pos;
+ unsigned int i;
+
+ total = 0;
+ for (cur = node; cur; cur = trie_load_parent(cur))
+ total += cur->run.nr_entries;
+ if (max_entries < total)
+ return 0;
+
+ pos = total;
+ for (cur = node; cur; cur = trie_load_parent(cur)) {
+ pos -= cur->run.nr_entries;
+ for (i = 0; i < cur->run.nr_entries; i++)
+ stack_depot_trie_node_frame(cur, i, &entries[pos + i]);
+ }
+
+ return total;
+}
- return depot_fetch_stack(handle);
+static unsigned int trie_fetch_range(const struct stack_depot_trie_node *node,
+ unsigned int offset,
+ unsigned long *entries,
+ unsigned int max_entries)
+{
+ const struct stack_depot_trie_node *cur;
+ unsigned int end;
+ unsigned int start;
+ unsigned int total;
+ unsigned int pos;
+ unsigned int i;
+
+ total = 0;
+ for (cur = node; cur; cur = trie_load_parent(cur))
+ total += cur->run.nr_entries;
+ if (offset >= total)
+ return 0;
+
+ max_entries = min(max_entries, total - offset);
+ end = offset + max_entries;
+ pos = total;
+ for (cur = node; cur; cur = trie_load_parent(cur)) {
+ pos -= cur->run.nr_entries;
+ start = max(pos, offset);
+ for (i = start; i < min(pos + cur->run.nr_entries, end); i++)
+ stack_depot_trie_node_frame(cur, i - pos, &entries[i - offset]);
+ }
+
+ return max_entries;
+}
+
+static unsigned int trie_fetch_handle_into(depot_stack_handle_t handle,
+ unsigned long *entries,
+ unsigned int max_entries)
+{
+ const struct stack_depot_trie_node *node;
+ u32 stack_id;
+ unsigned int nr_entries;
+
+ stack_id = trie_stack_id(handle);
+ rcu_read_lock_sched_notrace();
+ node = trie_side_table_lookup(stack_id);
+ if (WARN_ONCE(!node, "corrupt trie handle %08x\n", handle)) {
+ rcu_read_unlock_sched_notrace();
+ return 0;
+ }
+ nr_entries = trie_fetch_into(node, entries, max_entries);
+ rcu_read_unlock_sched_notrace();
+ if (nr_entries)
+ kmsan_unpoison_memory(entries, nr_entries * sizeof(*entries));
+
+ return nr_entries;
+}
+
+static unsigned int trie_fetch_handle_range(depot_stack_handle_t handle,
+ unsigned int offset,
+ unsigned long *entries,
+ unsigned int max_entries)
+{
+ const struct stack_depot_trie_node *node;
+ u32 stack_id;
+ unsigned int nr_entries;
+
+ stack_id = trie_stack_id(handle);
+ rcu_read_lock_sched_notrace();
+ node = trie_side_table_lookup(stack_id);
+ if (WARN_ONCE(!node, "corrupt trie handle %08x\n", handle)) {
+ rcu_read_unlock_sched_notrace();
+ return 0;
+ }
+ nr_entries = trie_fetch_range(node, offset, entries, max_entries);
+ rcu_read_unlock_sched_notrace();
+ if (nr_entries)
+ kmsan_unpoison_memory(entries, nr_entries * sizeof(*entries));
+
+ return nr_entries;
}
unsigned int stack_depot_fetch(depot_stack_handle_t handle,
@@ -771,6 +2316,8 @@ unsigned int stack_depot_fetch(depot_stack_handle_t handle,
if (!handle || stack_depot_disabled)
return 0;
+ if (WARN_ON_ONCE(stack_depot_handle_is_trie(handle)))
+ return 0;
stack = depot_fetch_stack(handle);
/*
@@ -785,12 +2332,44 @@ unsigned int stack_depot_fetch(depot_stack_handle_t handle,
}
EXPORT_SYMBOL_GPL(stack_depot_fetch);
+unsigned int stack_depot_fetch_into(depot_stack_handle_t handle,
+ unsigned long *entries,
+ unsigned int max_entries)
+{
+ struct stack_record *stack;
+ unsigned int nr_entries;
+
+ if (!handle)
+ return 0;
+ if (stack_depot_disabled)
+ return 0;
+ WARN_ON_ONCE(!entries || !max_entries);
+ if (stack_depot_handle_is_trie(handle))
+ return trie_fetch_handle_into(handle, entries, max_entries);
+
+ stack = depot_fetch_stack(handle);
+ if (!stack)
+ return 0;
+ nr_entries = stack->size;
+ if (WARN_ON_ONCE(!nr_entries))
+ return 0;
+ if (nr_entries > max_entries)
+ return 0;
+
+ memcpy(entries, stack->entries, nr_entries * sizeof(*entries));
+ kmsan_unpoison_memory(entries, nr_entries * sizeof(*entries));
+ return nr_entries;
+}
+EXPORT_SYMBOL_GPL(stack_depot_fetch_into);
+
void stack_depot_put(depot_stack_handle_t handle)
{
struct stack_record *stack;
if (!handle || stack_depot_disabled)
return;
+ if (WARN_ON_ONCE(stack_depot_handle_is_trie(handle)))
+ return;
stack = depot_fetch_stack(handle);
/*
@@ -800,16 +2379,60 @@ void stack_depot_put(depot_stack_handle_t handle)
if (WARN(!stack, "corrupt handle or unbalanced stack_depot_put()"))
return;
+ if (WARN_ON_ONCE(stack->flags & STACK_DEPOT_FLAG_COUNTABLE))
+ return;
if (refcount_dec_and_test(&stack->count))
depot_free_stack(stack);
}
EXPORT_SYMBOL_GPL(stack_depot_put);
+static void trie_print(depot_stack_handle_t handle)
+{
+ unsigned long entries[STACK_DEPOT_PRINT_CHUNK_FRAMES];
+ unsigned int nr_entries;
+ unsigned int offset = 0;
+
+ while ((nr_entries = trie_fetch_handle_range(handle, offset, entries,
+ ARRAY_SIZE(entries)))) {
+ stack_trace_print(entries, nr_entries, 0);
+ offset += nr_entries;
+ }
+}
+
+static int trie_snprint(depot_stack_handle_t handle, char *buf, size_t size,
+ int spaces)
+{
+ unsigned long entries[STACK_DEPOT_PRINT_CHUNK_FRAMES];
+ unsigned int generated;
+ unsigned int nr_entries;
+ unsigned int offset = 0;
+ unsigned int total = 0;
+
+ while (size &&
+ (nr_entries = trie_fetch_handle_range(handle, offset, entries,
+ ARRAY_SIZE(entries)))) {
+ generated = stack_trace_snprint(buf, size, entries, nr_entries, spaces);
+ total += generated;
+ if (generated >= size)
+ break;
+ buf += generated;
+ size -= generated;
+ offset += nr_entries;
+ }
+
+ return total;
+}
+
void stack_depot_print(depot_stack_handle_t stack)
{
unsigned long *entries;
unsigned int nr_entries;
+ if (stack_depot_handle_is_trie(stack)) {
+ trie_print(stack);
+ return;
+ }
+
nr_entries = stack_depot_fetch(stack, &entries);
if (nr_entries > 0)
stack_trace_print(entries, nr_entries, 0);
@@ -822,6 +2445,9 @@ int stack_depot_snprint(depot_stack_handle_t handle, char *buf, size_t size,
unsigned long *entries;
unsigned int nr_entries;
+ if (stack_depot_handle_is_trie(handle))
+ return trie_snprint(handle, buf, size, spaces);
+
nr_entries = stack_depot_fetch(handle, &entries);
return nr_entries ? stack_trace_snprint(buf, size, entries, nr_entries,
spaces) : 0;
diff --git a/lib/tests/Makefile b/lib/tests/Makefile
index 3cac3b63a7522..1f72191f98bbc 100644
--- a/lib/tests/Makefile
+++ b/lib/tests/Makefile
@@ -48,6 +48,7 @@ obj-$(CONFIG_SCANF_KUNIT_TEST) += scanf_kunit.o
obj-$(CONFIG_SEQ_BUF_KUNIT_TEST) += seq_buf_kunit.o
obj-$(CONFIG_SIPHASH_KUNIT_TEST) += siphash_kunit.o
obj-$(CONFIG_SLUB_KUNIT_TEST) += slub_kunit.o
+obj-$(CONFIG_STACKDEPOT_KUNIT_TEST) += stackdepot_kunit.o
obj-$(CONFIG_TEST_SORT) += test_sort.o
CFLAGS_stackinit_kunit.o += $(call cc-disable-warning, switch-unreachable)
obj-$(CONFIG_STACKINIT_KUNIT_TEST) += stackinit_kunit.o
diff --git a/lib/tests/stackdepot_kunit.c b/lib/tests/stackdepot_kunit.c
new file mode 100644
index 0000000000000..b86b84d56176e
--- /dev/null
+++ b/lib/tests/stackdepot_kunit.c
@@ -0,0 +1,582 @@
+// SPDX-License-Identifier: GPL-2.0-only
+
+#include <kunit/test.h>
+#include <linux/array_size.h>
+#include <linux/gfp.h>
+#include <linux/kallsyms.h>
+#include <linux/limits.h>
+#include <linux/moduleparam.h>
+#include <linux/stackdepot.h>
+#include <linux/stacktrace.h>
+#include <linux/string.h>
+
+#include <asm/stackdepot.h>
+
+static int expected_trie_pool_limit = -1;
+module_param_named(trie_pool_limit, expected_trie_pool_limit, int, 0);
+MODULE_PARM_DESC(trie_pool_limit, "Expected stackdepot hash/trie pool split");
+
+#ifdef CONFIG_ARM64
+#include <asm/sections.h>
+
+static inline unsigned long stackdepot_arm64_frame(long offset)
+{
+ return (unsigned long)((long)_text + offset);
+}
+#endif
+
+static unsigned long stackdepot_test_frame(unsigned int i)
+{
+#ifdef CONFIG_ARM64
+ return i & 1 ? 0x1000UL + i * 0x1000UL :
+ stackdepot_arm64_frame(i * 4);
+#elif defined(CONFIG_X86_64) && !defined(CONFIG_UML)
+ return i & 1 ? 0xffff888000000000UL + i * 0x1000UL :
+ 0xffffffff10000000UL + i * 0x10UL;
+#else
+ return 0x1000UL + i * 0x1000UL;
+#endif
+}
+
+static void stackdepot_trie_max_path_roundtrip(struct kunit *test)
+{
+ union handle_parts parts;
+ unsigned long *entries;
+ unsigned long *fetched;
+ depot_stack_handle_t handle;
+ size_t size = CONFIG_STACKDEPOT_MAX_FRAMES * sizeof(*entries);
+ u32 pool_index_plus_1;
+ unsigned int i;
+
+ if (expected_trie_pool_limit < 0)
+ kunit_skip(test, "trie pool limit was not provided");
+ KUNIT_ASSERT_EQ(test, stack_depot_init(), 0);
+ entries = kunit_kcalloc(test, CONFIG_STACKDEPOT_MAX_FRAMES,
+ sizeof(*entries), GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, entries);
+ fetched = kunit_kcalloc(test, CONFIG_STACKDEPOT_MAX_FRAMES,
+ sizeof(*fetched), GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, fetched);
+ for (i = 0; i < CONFIG_STACKDEPOT_MAX_FRAMES; i++)
+ entries[i] = stackdepot_test_frame(i);
+
+ handle = stack_depot_save(entries, CONFIG_STACKDEPOT_MAX_FRAMES,
+ GFP_KERNEL);
+ KUNIT_ASSERT_NE(test, handle, (depot_stack_handle_t)0);
+ parts.handle = handle;
+ pool_index_plus_1 = parts.pool_index_plus_1;
+ KUNIT_EXPECT_GT(test, pool_index_plus_1, (u32)expected_trie_pool_limit);
+ KUNIT_EXPECT_EQ(test,
+ stack_depot_fetch_into(handle, fetched,
+ CONFIG_STACKDEPOT_MAX_FRAMES),
+ (unsigned int)CONFIG_STACKDEPOT_MAX_FRAMES);
+ KUNIT_EXPECT_MEMEQ(test, fetched, entries, size);
+ KUNIT_EXPECT_EQ(test,
+ stack_depot_save(entries, CONFIG_STACKDEPOT_MAX_FRAMES,
+ GFP_KERNEL),
+ handle);
+}
+
+static void stackdepot_save_flags_public(struct kunit *test)
+{
+ union handle_parts parts;
+ unsigned long entries[] = { 0x501000UL, 0x502000UL, 0x503000UL };
+ unsigned long get_entries[] = { 0x601000UL, 0x602000UL };
+ unsigned long missing_entries[] = { 0x701000UL, 0x702000UL };
+ unsigned long blocking_entries[] = { 0x711000UL, 0x712000UL };
+ unsigned long fetched[ARRAY_SIZE(entries)] = {};
+ depot_stack_handle_t blocking_handle;
+ depot_stack_handle_t noalloc_handle;
+ depot_stack_handle_t overlong_handle;
+ depot_stack_handle_t plain_handle;
+ depot_stack_handle_t get_handle;
+ depot_stack_handle_t again;
+ depot_stack_handle_t extra;
+ gfp_t no_spin = GFP_NOWAIT & ~__GFP_RECLAIM;
+ u32 pool_index_plus_1;
+ unsigned long *overlong_fetched;
+ unsigned long *overlong_entries;
+ unsigned int overlong_nr = CONFIG_STACKDEPOT_MAX_FRAMES + 1;
+ unsigned int nr_entries;
+ size_t overlong_size;
+ unsigned int i;
+
+ KUNIT_ASSERT_EQ(test, stack_depot_init(), 0);
+ overlong_entries = kunit_kcalloc(test, overlong_nr,
+ sizeof(*overlong_entries), GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, overlong_entries);
+ overlong_fetched = kunit_kcalloc(test, CONFIG_STACKDEPOT_MAX_FRAMES,
+ sizeof(*overlong_fetched), GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, overlong_fetched);
+ for (i = 0; i < overlong_nr; i++)
+ overlong_entries[i] = 0x800000UL + i * 0x1000UL;
+
+ plain_handle = stack_depot_save(entries, ARRAY_SIZE(entries), GFP_KERNEL);
+ KUNIT_ASSERT_NE(test, plain_handle, (depot_stack_handle_t)0);
+ again = stack_depot_save(entries, ARRAY_SIZE(entries), GFP_KERNEL);
+ KUNIT_EXPECT_EQ(test, again, plain_handle);
+
+ nr_entries = stack_depot_fetch_into(plain_handle, fetched,
+ ARRAY_SIZE(fetched));
+ KUNIT_EXPECT_EQ(test, nr_entries, (unsigned int)ARRAY_SIZE(entries));
+ KUNIT_EXPECT_MEMEQ(test, fetched, entries, sizeof(entries));
+
+ noalloc_handle = stack_depot_save_flags(entries, ARRAY_SIZE(entries), no_spin, 0);
+ KUNIT_EXPECT_EQ(test, noalloc_handle, plain_handle);
+ noalloc_handle = stack_depot_save_flags(missing_entries,
+ ARRAY_SIZE(missing_entries),
+ GFP_KERNEL, 0);
+ KUNIT_ASSERT_NE(test, noalloc_handle, (depot_stack_handle_t)0);
+ if (expected_trie_pool_limit >= 0) {
+ parts.handle = noalloc_handle;
+ pool_index_plus_1 = parts.pool_index_plus_1;
+ KUNIT_EXPECT_GT(test, pool_index_plus_1,
+ (u32)expected_trie_pool_limit);
+ }
+ nr_entries = stack_depot_fetch_into(noalloc_handle, fetched,
+ ARRAY_SIZE(fetched));
+ KUNIT_EXPECT_EQ(test, nr_entries,
+ (unsigned int)ARRAY_SIZE(missing_entries));
+ KUNIT_EXPECT_MEMEQ(test, fetched, missing_entries, sizeof(missing_entries));
+ KUNIT_EXPECT_EQ(test,
+ stack_depot_save_flags(missing_entries,
+ ARRAY_SIZE(missing_entries),
+ no_spin, 0),
+ noalloc_handle);
+
+ blocking_handle = stack_depot_save_flags(blocking_entries,
+ ARRAY_SIZE(blocking_entries),
+ GFP_KERNEL, 0);
+ KUNIT_ASSERT_NE(test, blocking_handle, (depot_stack_handle_t)0);
+ if (expected_trie_pool_limit >= 0) {
+ parts.handle = blocking_handle;
+ pool_index_plus_1 = parts.pool_index_plus_1;
+ KUNIT_EXPECT_GT(test, pool_index_plus_1,
+ (u32)expected_trie_pool_limit);
+ }
+ memset(fetched, 0, sizeof(fetched));
+ nr_entries = stack_depot_fetch_into(blocking_handle, fetched,
+ ARRAY_SIZE(fetched));
+ KUNIT_EXPECT_EQ(test, nr_entries,
+ (unsigned int)ARRAY_SIZE(blocking_entries));
+ KUNIT_EXPECT_MEMEQ(test, fetched, blocking_entries,
+ sizeof(blocking_entries));
+
+ get_handle = stack_depot_save_flags(get_entries, ARRAY_SIZE(get_entries),
+ GFP_KERNEL,
+ STACK_DEPOT_FLAG_CAN_ALLOC |
+ STACK_DEPOT_FLAG_GET);
+ KUNIT_ASSERT_NE(test, get_handle, (depot_stack_handle_t)0);
+ stack_depot_put(get_handle);
+
+ overlong_handle = stack_depot_save(overlong_entries, overlong_nr,
+ GFP_KERNEL);
+ KUNIT_ASSERT_NE(test, overlong_handle, (depot_stack_handle_t)0);
+ nr_entries = stack_depot_fetch_into(overlong_handle, overlong_fetched,
+ CONFIG_STACKDEPOT_MAX_FRAMES);
+ KUNIT_EXPECT_EQ(test, nr_entries, (unsigned int)CONFIG_STACKDEPOT_MAX_FRAMES);
+ overlong_size = CONFIG_STACKDEPOT_MAX_FRAMES * sizeof(*overlong_entries);
+ KUNIT_EXPECT_MEMEQ(test, overlong_fetched, overlong_entries, overlong_size);
+
+ extra = stack_depot_set_extra_bits(plain_handle, 7);
+ KUNIT_ASSERT_NE(test, extra, (depot_stack_handle_t)0);
+ KUNIT_EXPECT_EQ(test, stack_depot_get_extra_bits(extra), 7U);
+ memset(fetched, 0, sizeof(fetched));
+ nr_entries = stack_depot_fetch_into(extra, fetched, ARRAY_SIZE(fetched));
+ KUNIT_EXPECT_EQ(test, nr_entries, (unsigned int)ARRAY_SIZE(entries));
+ KUNIT_EXPECT_MEMEQ(test, fetched, entries, sizeof(entries));
+}
+
+static void stackdepot_snprint_public(struct kunit *test)
+{
+ const unsigned int nr_entries = CONFIG_STACKDEPOT_MAX_FRAMES;
+ const size_t buf_size = nr_entries * (KSYM_SYMBOL_LEN + 4);
+ unsigned long *entries;
+ char *expected;
+ char *actual;
+ depot_stack_handle_t handle;
+ unsigned int expected_len;
+ unsigned int prefix_entries;
+ unsigned int prefix_len;
+ size_t output_size;
+ unsigned int i;
+ int actual_len;
+
+ KUNIT_ASSERT_EQ(test, stack_depot_init(), 0);
+ entries = kunit_kmalloc_array(test, nr_entries, sizeof(*entries),
+ GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, entries);
+ expected = kunit_kzalloc(test, buf_size, GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, expected);
+ actual = kunit_kzalloc(test, buf_size, GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, actual);
+ for (i = 0; i < nr_entries; i++)
+ entries[i] = stackdepot_test_frame(i);
+
+ handle = stack_depot_save(entries, nr_entries, GFP_KERNEL);
+ KUNIT_ASSERT_NE(test, handle, (depot_stack_handle_t)0);
+ if (expected_trie_pool_limit >= 0) {
+ union handle_parts parts = { .handle = handle };
+
+ KUNIT_EXPECT_GT(test, (u32)parts.pool_index_plus_1,
+ (u32)expected_trie_pool_limit);
+ }
+ expected_len = stack_trace_snprint(expected, buf_size, entries,
+ nr_entries, 2);
+ actual_len = stack_depot_snprint(handle, actual, buf_size, 2);
+ KUNIT_EXPECT_EQ(test, actual_len, (int)expected_len);
+ KUNIT_EXPECT_STREQ(test, actual, expected);
+
+ prefix_entries = nr_entries / 2 + 1;
+ prefix_len = stack_trace_snprint(expected, buf_size, entries,
+ prefix_entries, 2);
+ KUNIT_ASSERT_LE(test, (size_t)prefix_len + 2, buf_size);
+ output_size = prefix_len + 2;
+ memset(expected, 0, buf_size);
+ memset(actual, 0, buf_size);
+ expected_len = stack_trace_snprint(expected, output_size, entries,
+ nr_entries, 2);
+ actual_len = stack_depot_snprint(handle, actual, output_size, 2);
+ KUNIT_EXPECT_EQ(test, actual_len, (int)expected_len);
+ KUNIT_EXPECT_STREQ(test, actual, expected);
+}
+
+static void stackdepot_countable_public(struct kunit *test)
+{
+ unsigned long plain_entries[] = {
+ 0x141000UL,
+ 0x142000UL,
+ 0x143000UL,
+ };
+ unsigned long get_entries[] = {
+ 0x151000UL,
+ 0x152000UL,
+ 0x153000UL,
+ };
+ unsigned long fetched[ARRAY_SIZE(plain_entries)] = {};
+ depot_flags_t countable = STACK_DEPOT_FLAG_CAN_ALLOC |
+ STACK_DEPOT_FLAG_COUNTABLE;
+ struct stack_record *record;
+ depot_stack_handle_t count_handle;
+ depot_stack_handle_t plain_handle;
+ depot_stack_handle_t get_handle;
+ unsigned int get_nr = ARRAY_SIZE(get_entries);
+ unsigned int plain_nr = ARRAY_SIZE(plain_entries);
+ unsigned int nr_entries;
+
+ KUNIT_ASSERT_EQ(test, stack_depot_init(), 0);
+
+ plain_handle = stack_depot_save(plain_entries, plain_nr, GFP_KERNEL);
+ KUNIT_ASSERT_NE(test, plain_handle, (depot_stack_handle_t)0);
+ count_handle = stack_depot_save_flags(plain_entries, plain_nr, GFP_KERNEL,
+ countable);
+ KUNIT_ASSERT_NE(test, count_handle, (depot_stack_handle_t)0);
+ record = __stack_depot_get_stack_record(count_handle);
+ KUNIT_ASSERT_NOT_NULL(test, record);
+ KUNIT_EXPECT_EQ(test, record->size, (u16)plain_nr);
+ KUNIT_EXPECT_MEMEQ(test, record->entries, plain_entries,
+ sizeof(plain_entries));
+ nr_entries = stack_depot_fetch_into(count_handle, fetched,
+ ARRAY_SIZE(fetched));
+ KUNIT_EXPECT_EQ(test, nr_entries, plain_nr);
+ KUNIT_EXPECT_MEMEQ(test, fetched, plain_entries, sizeof(plain_entries));
+
+ get_handle = stack_depot_save_flags(get_entries, get_nr, GFP_KERNEL,
+ STACK_DEPOT_FLAG_CAN_ALLOC |
+ STACK_DEPOT_FLAG_GET);
+ KUNIT_ASSERT_NE(test, get_handle, (depot_stack_handle_t)0);
+ count_handle = stack_depot_save_flags(get_entries, get_nr, GFP_KERNEL,
+ countable);
+ KUNIT_ASSERT_NE(test, count_handle, (depot_stack_handle_t)0);
+ record = __stack_depot_get_stack_record(count_handle);
+ KUNIT_ASSERT_NOT_NULL(test, record);
+ KUNIT_EXPECT_MEMEQ(test, record->entries, get_entries, sizeof(get_entries));
+
+ stack_depot_put(get_handle);
+}
+
+static void stackdepot_fetch_into_roundtrip(struct kunit *test)
+{
+ unsigned long entries[] = {
+ 0x101000UL,
+ 0x102000UL,
+ 0x103000UL,
+ };
+ unsigned long exact[ARRAY_SIZE(entries)] = {};
+ unsigned long fetched[ARRAY_SIZE(entries) + 1] = {
+ [ARRAY_SIZE(entries)] = 0xa5a5a5a5UL,
+ };
+ unsigned long expected_tail = fetched[ARRAY_SIZE(entries)];
+ depot_stack_handle_t handle;
+ unsigned int nr_entries;
+
+ KUNIT_ASSERT_EQ(test, stack_depot_init(), 0);
+
+ handle = stack_depot_save(entries, ARRAY_SIZE(entries), GFP_KERNEL);
+ KUNIT_ASSERT_NE(test, handle, (depot_stack_handle_t)0);
+
+ nr_entries = stack_depot_fetch_into(handle, exact, ARRAY_SIZE(exact));
+ KUNIT_EXPECT_EQ(test, nr_entries, (unsigned int)ARRAY_SIZE(entries));
+ KUNIT_EXPECT_MEMEQ(test, exact, entries, sizeof(entries));
+
+ nr_entries = stack_depot_fetch_into(handle, fetched, ARRAY_SIZE(fetched));
+ KUNIT_EXPECT_EQ(test, nr_entries, (unsigned int)ARRAY_SIZE(entries));
+ KUNIT_EXPECT_MEMEQ(test, fetched, entries, sizeof(entries));
+ KUNIT_EXPECT_EQ(test, fetched[ARRAY_SIZE(entries)], expected_tail);
+}
+
+static void stackdepot_fetch_into_rejects_missing_or_short_stack(struct kunit *test)
+{
+ unsigned long entries[] = {
+ 0x111000UL,
+ 0x112000UL,
+ 0x113000UL,
+ };
+ unsigned long fetched[ARRAY_SIZE(entries)] = {
+ 0xa1a1a1a1UL,
+ 0xb2b2b2b2UL,
+ 0xc3c3c3c3UL,
+ };
+ unsigned long expected[ARRAY_SIZE(fetched)];
+ depot_stack_handle_t handle;
+ unsigned int nr_entries;
+
+ KUNIT_ASSERT_EQ(test, stack_depot_init(), 0);
+
+ handle = stack_depot_save(entries, ARRAY_SIZE(entries), GFP_KERNEL);
+ KUNIT_ASSERT_NE(test, handle, (depot_stack_handle_t)0);
+ memcpy(expected, fetched, sizeof(expected));
+
+ nr_entries = stack_depot_fetch_into(0, fetched, ARRAY_SIZE(fetched));
+ KUNIT_EXPECT_EQ(test, nr_entries, 0U);
+ KUNIT_EXPECT_MEMEQ(test, fetched, expected, sizeof(expected));
+
+ nr_entries = stack_depot_fetch_into(0, NULL, 0);
+ KUNIT_EXPECT_EQ(test, nr_entries, 0U);
+
+ nr_entries = stack_depot_fetch_into(handle, fetched,
+ ARRAY_SIZE(fetched) - 1);
+ KUNIT_EXPECT_EQ(test, nr_entries, 0U);
+ KUNIT_EXPECT_MEMEQ(test, fetched, expected, sizeof(expected));
+}
+
+static void stackdepot_trie_topology_roundtrip(struct kunit *test,
+ bool constrained)
+{
+ union handle_parts parts;
+ unsigned long seed[] = { 0x191000UL, 0x192000UL };
+ unsigned long stacks[][3] = {
+ { 0x201000UL, 0x202000UL },
+ { 0x201000UL, 0x203000UL },
+ { 0x201000UL },
+ { 0x201000UL, 0x203000UL, 0x204000UL },
+ { 0x201000UL, 0x205000UL },
+ { 0x201000UL, 0x204000UL },
+ { 0x201000UL, 0x206000UL },
+ { 0x201000UL, 0x207000UL },
+ { 0x301000UL, 0x302000UL },
+ { 0x301000UL, 0x302000UL, 0x303000UL },
+ { 0x301000UL, 0x304000UL },
+ { 0x401000UL, 0x402000UL, 0x403000UL },
+ { 0x401000UL, 0x402000UL },
+ };
+ unsigned int nr_entries[] = { 2, 2, 1, 3, 2, 2, 2, 2, 2, 3, 2, 3, 2 };
+ depot_stack_handle_t handles[ARRAY_SIZE(stacks)];
+ depot_stack_handle_t seed_handle;
+ unsigned long fetched[ARRAY_SIZE(stacks[0])];
+ gfp_t no_spin = GFP_NOWAIT & ~__GFP_RECLAIM;
+ u32 pool_index_plus_1;
+ unsigned int j;
+ unsigned int i;
+
+ if (expected_trie_pool_limit < 0)
+ kunit_skip(test, "trie pool limit was not provided");
+ KUNIT_ASSERT_EQ(test, stack_depot_init(), 0);
+ if (constrained) {
+ seed_handle = stack_depot_save(seed, ARRAY_SIZE(seed), GFP_KERNEL);
+ KUNIT_ASSERT_NE(test, seed_handle, (depot_stack_handle_t)0);
+ for (i = 0; i < ARRAY_SIZE(stacks); i++)
+ for (j = 0; j < nr_entries[i]; j++)
+ stacks[i][j] += 0x10000000UL;
+ }
+
+ for (i = 0; i < ARRAY_SIZE(stacks); i++) {
+ if (constrained)
+ handles[i] = stack_depot_save_flags(stacks[i], nr_entries[i],
+ GFP_KERNEL, 0);
+ else
+ handles[i] = stack_depot_save(stacks[i], nr_entries[i],
+ GFP_KERNEL);
+ KUNIT_ASSERT_NE(test, handles[i], (depot_stack_handle_t)0);
+ }
+ parts.handle = handles[0];
+ pool_index_plus_1 = parts.pool_index_plus_1;
+ KUNIT_ASSERT_GT(test, pool_index_plus_1,
+ (u32)expected_trie_pool_limit);
+
+ for (i = 0; i < ARRAY_SIZE(stacks); i++) {
+ memset(fetched, 0, sizeof(fetched));
+ KUNIT_EXPECT_EQ(test,
+ stack_depot_fetch_into(handles[i], fetched,
+ ARRAY_SIZE(fetched)),
+ nr_entries[i]);
+ KUNIT_EXPECT_MEMEQ(test, fetched, stacks[i],
+ nr_entries[i] * sizeof(fetched[0]));
+ if (constrained)
+ KUNIT_EXPECT_EQ(test,
+ stack_depot_save_flags(stacks[i], nr_entries[i],
+ no_spin, 0),
+ handles[i]);
+ else
+ KUNIT_EXPECT_EQ(test,
+ stack_depot_save(stacks[i], nr_entries[i],
+ GFP_KERNEL),
+ handles[i]);
+ }
+}
+
+static void stackdepot_trie_topology_allocating(struct kunit *test)
+{
+ stackdepot_trie_topology_roundtrip(test, false);
+}
+
+static void stackdepot_trie_topology_constrained(struct kunit *test)
+{
+ stackdepot_trie_topology_roundtrip(test, true);
+}
+
+static void stackdepot_frame_storage_roundtrip(struct kunit *test)
+{
+ union handle_parts parts;
+ unsigned long fetched[3] = {};
+ depot_stack_handle_t handle;
+ u32 pool_index_plus_1;
+ unsigned int nr_entries;
+#if defined(CONFIG_ARM64)
+ unsigned long entries[] = {
+ stackdepot_arm64_frame(S32_MIN),
+ 0x1000UL,
+ stackdepot_arm64_frame(S32_MAX),
+ };
+#elif defined(CONFIG_X86_64)
+ unsigned long entries[] = {
+ 0xffffffff10001000UL,
+ 0xffff888000001000UL,
+ 0xffffffff20002000UL,
+ };
+#else
+ unsigned long entries[] = { 0x301000UL, 0x302000UL, 0x303000UL };
+#endif
+
+ if (expected_trie_pool_limit < 0)
+ kunit_skip(test, "trie pool limit was not provided");
+ KUNIT_ASSERT_EQ(test, stack_depot_init(), 0);
+ handle = stack_depot_save(entries, ARRAY_SIZE(entries), GFP_KERNEL);
+ KUNIT_ASSERT_NE(test, handle, (depot_stack_handle_t)0);
+ parts.handle = handle;
+ pool_index_plus_1 = parts.pool_index_plus_1;
+ KUNIT_ASSERT_GT(test, pool_index_plus_1,
+ (u32)expected_trie_pool_limit);
+
+ nr_entries = stack_depot_fetch_into(handle, fetched, ARRAY_SIZE(fetched));
+ KUNIT_EXPECT_EQ(test, nr_entries, (unsigned int)ARRAY_SIZE(entries));
+ KUNIT_EXPECT_MEMEQ(test, fetched, entries, sizeof(entries));
+}
+
+static void stackdepot_frame_raw_fallback(struct kunit *test)
+{
+ unsigned long frame = 0x1000UL;
+ bool compressed;
+ u32 payload;
+
+#ifdef CONFIG_ARM64
+ frame = (unsigned long)_text + (unsigned long)S32_MAX + 1UL;
+#endif
+
+ compressed = arch_stack_depot_frame_try_compress(frame, &payload);
+ KUNIT_EXPECT_FALSE(test, compressed);
+}
+
+#if defined(CONFIG_X86_64) && !defined(CONFIG_UML)
+static void stackdepot_frame_x86_64(struct kunit *test)
+{
+ unsigned long direct_map = 0xffff888000001000UL;
+ unsigned long frame = 0xffffffff81234567UL;
+ unsigned long out;
+ bool compressed;
+ u32 low;
+
+ compressed = arch_stack_depot_frame_try_compress(frame, &low);
+ KUNIT_EXPECT_TRUE(test, compressed);
+ KUNIT_EXPECT_EQ(test, low, (u32)0x81234567);
+ arch_stack_depot_frame_decompress(low, &out);
+ KUNIT_EXPECT_EQ(test, out, frame);
+
+ compressed = arch_stack_depot_frame_try_compress(direct_map, &low);
+ KUNIT_EXPECT_FALSE(test, compressed);
+}
+#endif /* CONFIG_X86_64 && !CONFIG_UML */
+
+#ifdef CONFIG_ARM64
+static void stackdepot_frame_arm64(struct kunit *test)
+{
+ long negative_offset = S32_MIN;
+ long positive_offset = S32_MAX;
+ long offset = 0x123456;
+ unsigned long frame = stackdepot_arm64_frame(offset);
+ unsigned long out;
+ bool compressed;
+ u32 payload;
+
+ compressed = arch_stack_depot_frame_try_compress(frame, &payload);
+ KUNIT_EXPECT_TRUE(test, compressed);
+ KUNIT_EXPECT_EQ(test, payload, (u32)(s32)offset);
+ arch_stack_depot_frame_decompress(payload, &out);
+ KUNIT_EXPECT_EQ(test, out, frame);
+
+ frame = stackdepot_arm64_frame(negative_offset);
+ compressed = arch_stack_depot_frame_try_compress(frame, &payload);
+ KUNIT_EXPECT_TRUE(test, compressed);
+ KUNIT_EXPECT_EQ(test, payload, (u32)(s32)negative_offset);
+ arch_stack_depot_frame_decompress(payload, &out);
+ KUNIT_EXPECT_EQ(test, out, frame);
+
+ frame = stackdepot_arm64_frame(positive_offset);
+ compressed = arch_stack_depot_frame_try_compress(frame, &payload);
+ KUNIT_EXPECT_TRUE(test, compressed);
+ KUNIT_EXPECT_EQ(test, payload, (u32)(s32)positive_offset);
+ arch_stack_depot_frame_decompress(payload, &out);
+ KUNIT_EXPECT_EQ(test, out, frame);
+}
+#endif /* CONFIG_ARM64 */
+
+static struct kunit_case stackdepot_test_cases[] = {
+ KUNIT_CASE(stackdepot_trie_max_path_roundtrip),
+ KUNIT_CASE(stackdepot_save_flags_public),
+ KUNIT_CASE(stackdepot_snprint_public),
+ KUNIT_CASE(stackdepot_countable_public),
+ KUNIT_CASE(stackdepot_fetch_into_roundtrip),
+ KUNIT_CASE(stackdepot_fetch_into_rejects_missing_or_short_stack),
+ KUNIT_CASE(stackdepot_trie_topology_allocating),
+ KUNIT_CASE(stackdepot_trie_topology_constrained),
+ KUNIT_CASE(stackdepot_frame_storage_roundtrip),
+ KUNIT_CASE(stackdepot_frame_raw_fallback),
+#if defined(CONFIG_X86_64) && !defined(CONFIG_UML)
+ KUNIT_CASE(stackdepot_frame_x86_64),
+#endif
+#ifdef CONFIG_ARM64
+ KUNIT_CASE(stackdepot_frame_arm64),
+#endif
+ {}
+};
+
+static struct kunit_suite stackdepot_test_suite = {
+ .name = "stackdepot",
+ .test_cases = stackdepot_test_cases,
+};
+
+kunit_test_suite(stackdepot_test_suite);
+
+MODULE_DESCRIPTION("KUnit tests for stack depot");
+MODULE_AUTHOR("Caleb Kan <ckan@cloudflare.com>");
+MODULE_LICENSE("GPL");
diff --git a/mm/kmemleak.c b/mm/kmemleak.c
index 8fa409a4f9fb2..c42741a88bd42 100644
--- a/mm/kmemleak.c
+++ b/mm/kmemleak.c
@@ -378,10 +378,10 @@ static void __print_unreferenced(struct seq_file *seq,
bool hex_dump)
{
int i;
- unsigned long *entries;
+ unsigned long entries[MAX_TRACE];
unsigned int nr_entries;
- nr_entries = stack_depot_fetch(object->trace_handle, &entries);
+ nr_entries = stack_depot_fetch_into(object->trace_handle, entries, ARRAY_SIZE(entries));
warn_or_seq_printf(seq, "unreferenced object%s 0x%08lx (size %zu):\n",
__object_type_str(object),
object->pointer, object->size);
diff --git a/mm/kmsan/kmsan_test.c b/mm/kmsan/kmsan_test.c
index 31f47cc4dab40..7c04e4b21873d 100644
--- a/mm/kmsan/kmsan_test.c
+++ b/mm/kmsan/kmsan_test.c
@@ -669,7 +669,7 @@ static void test_long_origin_chain(struct kunit *test)
*/
static void test_stackdepot_roundtrip(struct kunit *test)
{
- unsigned long src_entries[16], *dst_entries;
+ unsigned long src_entries[16], dst_entries[16];
unsigned int src_nentries, dst_nentries;
EXPECTATION_NO_REPORT(expect);
depot_stack_handle_t handle;
@@ -680,7 +680,7 @@ static void test_stackdepot_roundtrip(struct kunit *test)
stack_trace_save(src_entries, ARRAY_SIZE(src_entries), 1);
handle = stack_depot_save(src_entries, src_nentries, GFP_KERNEL);
stack_depot_print(handle);
- dst_nentries = stack_depot_fetch(handle, &dst_entries);
+ dst_nentries = stack_depot_fetch_into(handle, dst_entries, ARRAY_SIZE(dst_entries));
KUNIT_EXPECT_TRUE(test, src_nentries == dst_nentries);
kmsan_check_memory((void *)dst_entries,
diff --git a/mm/kmsan/report.c b/mm/kmsan/report.c
index d6853ce089541..0770658ba932e 100644
--- a/mm/kmsan/report.c
+++ b/mm/kmsan/report.c
@@ -83,9 +83,9 @@ static char *pretty_descr(char *descr)
return report_local_descr;
}
-void kmsan_print_origin(depot_stack_handle_t origin)
+static void kmsan_print_origin_with_buf(depot_stack_handle_t origin,
+ unsigned long *entries)
{
- unsigned long *entries = NULL, *chained_entries = NULL;
unsigned int nr_entries, chained_nr_entries, skipnr;
void *pc1 = NULL, *pc2 = NULL;
depot_stack_handle_t head;
@@ -97,7 +97,8 @@ void kmsan_print_origin(depot_stack_handle_t origin)
return;
while (true) {
- nr_entries = stack_depot_fetch(origin, &entries);
+ nr_entries =
+ stack_depot_fetch_into(origin, entries, KMSAN_STACK_DEPTH);
depth = kmsan_depth_from_eb(stack_depot_get_extra_bits(origin));
magic = nr_entries ? entries[0] : 0;
if ((nr_entries == 4) && (magic == KMSAN_ALLOCA_MAGIC_ORIGIN)) {
@@ -123,14 +124,10 @@ void kmsan_print_origin(depot_stack_handle_t origin)
origin = entries[2];
pr_err("Uninit was stored to memory at:\n");
chained_nr_entries =
- stack_depot_fetch(head, &chained_entries);
- kmsan_internal_unpoison_memory(
- chained_entries,
- chained_nr_entries * sizeof(*chained_entries),
- /*checked*/ false);
- skipnr = get_stack_skipnr(chained_entries,
- chained_nr_entries);
- stack_trace_print(chained_entries + skipnr,
+ stack_depot_fetch_into(head, entries,
+ KMSAN_STACK_DEPTH);
+ skipnr = get_stack_skipnr(entries, chained_nr_entries);
+ stack_trace_print(entries + skipnr,
chained_nr_entries - skipnr, 0);
pr_err("\n");
continue;
@@ -147,6 +144,13 @@ void kmsan_print_origin(depot_stack_handle_t origin)
}
}
+void kmsan_print_origin(depot_stack_handle_t origin)
+{
+ unsigned long entries[KMSAN_STACK_DEPTH];
+
+ kmsan_print_origin_with_buf(origin, entries);
+}
+
void kmsan_report(depot_stack_handle_t origin, void *address, int size,
int off_first, int off_last, const void __user *user_addr,
enum kmsan_bug_reason reason)
@@ -193,7 +197,7 @@ void kmsan_report(depot_stack_handle_t origin, void *address, int size,
0);
pr_err("\n");
- kmsan_print_origin(origin);
+ kmsan_print_origin_with_buf(origin, stack_entries);
if (size) {
pr_err("\n");
diff --git a/mm/page_owner.c b/mm/page_owner.c
index cfc31c92d7657..1fb1998bc129e 100644
--- a/mm/page_owner.c
+++ b/mm/page_owner.c
@@ -119,7 +119,8 @@ static __always_inline depot_stack_handle_t create_dummy_stack(void)
unsigned int nr_entries;
nr_entries = stack_trace_save(entries, ARRAY_SIZE(entries), 0);
- return stack_depot_save(entries, nr_entries, GFP_KERNEL);
+ return stack_depot_save_flags(entries, nr_entries, GFP_KERNEL,
+ STACK_DEPOT_FLAG_CAN_ALLOC | STACK_DEPOT_FLAG_COUNTABLE);
}
static noinline void register_dummy_stack(void)
@@ -181,7 +182,8 @@ static noinline depot_stack_handle_t save_stack(gfp_t flags)
set_current_in_page_owner();
nr_entries = stack_trace_save(entries, ARRAY_SIZE(entries), 2);
- handle = stack_depot_save(entries, nr_entries, flags);
+ handle = stack_depot_save_flags(entries, nr_entries, flags,
+ STACK_DEPOT_FLAG_CAN_ALLOC | STACK_DEPOT_FLAG_COUNTABLE);
if (!handle)
handle = failure_handle;
unset_current_in_page_owner();
diff --git a/mm/slub.c b/mm/slub.c
index f9b56cb439e70..4aa1c5a457182 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -8198,12 +8198,12 @@ void __kmem_obj_info(struct kmem_obj_info *kpp, void *object, struct slab *slab)
#ifdef CONFIG_STACKDEPOT
{
depot_stack_handle_t handle;
- unsigned long *entries;
+ unsigned long entries[TRACK_ADDRS_COUNT];
unsigned int nr_entries;
handle = READ_ONCE(trackp->handle);
if (handle) {
- nr_entries = stack_depot_fetch(handle, &entries);
+ nr_entries = stack_depot_fetch_into(handle, entries, ARRAY_SIZE(entries));
for (i = 0; i < KS_ADDRS_COUNT && i < nr_entries; i++)
kpp->kp_stack[i] = (void *)entries[i];
}
@@ -8211,7 +8211,7 @@ void __kmem_obj_info(struct kmem_obj_info *kpp, void *object, struct slab *slab)
trackp = get_track(s, objp, TRACK_FREE);
handle = READ_ONCE(trackp->handle);
if (handle) {
- nr_entries = stack_depot_fetch(handle, &entries);
+ nr_entries = stack_depot_fetch_into(handle, entries, ARRAY_SIZE(entries));
for (i = 0; i < KS_ADDRS_COUNT && i < nr_entries; i++)
kpp->kp_free_stack[i] = (void *)entries[i];
}
@@ -9946,12 +9946,14 @@ static int slab_debugfs_show(struct seq_file *seq, void *v)
#ifdef CONFIG_STACKDEPOT
{
depot_stack_handle_t handle;
- unsigned long *entries;
+ unsigned long entries[TRACK_ADDRS_COUNT];
unsigned int nr_entries, j;
handle = READ_ONCE(l->handle);
if (handle) {
- nr_entries = stack_depot_fetch(handle, &entries);
+ nr_entries =
+ stack_depot_fetch_into(handle, entries,
+ ARRAY_SIZE(entries));
seq_puts(seq, "\n");
for (j = 0; j < nr_entries; j++)
seq_printf(seq, " %pS\n", (void *)entries[j]);
diff --git a/scripts/gdb/linux/stackdepot.py b/scripts/gdb/linux/stackdepot.py
index 37313a5a51a0d..82aeb9f532c3d 100644
--- a/scripts/gdb/linux/stackdepot.py
+++ b/scripts/gdb/linux/stackdepot.py
@@ -37,6 +37,10 @@ def stack_depot_fetch(handle):
if handle == 0:
raise gdb.GdbError("handle is 0\n")
+ stack_max_pools = gdb.parse_and_eval('stack_max_pools')
+ if parts['pool_index_plus_1'] > stack_max_pools:
+ raise gdb.GdbError("trie-backed stack depot handles are not supported\n")
+
pool_index = parts['pool_index_plus_1'] - 1
if pool_index >= pools_num:
gdb.write("pool index %d out of bounds (%d) for stack id 0x%08x\n" % (parts['pool_index'], pools_num, handle))
|
| 3/2 | 2026/09/08 15:27 | llm | patch-evaluator |
0mModel:gemini-3.7-flash Args: null Results: null Tokens: input: 41652 output: 19 thoughts: 274 Thoughts: **My Initial Assessment of This Patch** Okay, let's dive into this patch and figure out the best fuzzing strategy. My first instinct is to determine if this is even fuzz-worthy and what the key areas of focus are. First, let's quickly break down the patch's core changes. It looks like we're getting a trie-based storage implementation for persistent stack depot records in `lib/stackdepot.c`. This is also touching architecture-specific headers: `arch/x86/include/asm/stackdepot.h`, `arch/arm64/include/asm/stackdepot.h`, and the generic one `include/asm-generic/stackdepot.h`. That alone suggests we need to be mindful of architecture differences. Beyond the core storage, there's a new kernel parameter: `stackdepot.trie_enabled=<bool>`. That's a good sign – knobs are usually great for fuzzing. We can use that to flip functionality on and off. Looking further, the patch touches a bunch of callers: `drivers/gpu/drm/drm_modeset_lock.c`, `mm/kmemleak.c`, `mm/kmsan/kmsan_test.c`, `mm/kmsan/report.c`, `mm/page_owner.c`, and `mm/slub.c`. This indicates that stack depot functionality is being integrated into various parts of the kernel. This is important to note as these could be good symbols to target during fuzzing. We also have a new function, `stack_depot_fetch_into()`. Anytime a new function is introduced, it's worth a look to understand how it's used and what potential vulnerabilities could exist within that function. Finally, `STACKDEPOT_KUNIT_TEST` is introduced. This implies we have some test infrastructure. We'll need to see how we can leverage those tests. Now, let's see how `CONFIG_STACKDEPOT` is defined in Kconfig. What dependencies enable it? What other configuration options influence it? Knowing this will help me set up a good fuzzing environment and tailor the fuzzing campaign for maximum impact. We'll want to cover different configurations and combinations to maximize the chances of finding issues. |
| 4/2 | 2026/09/08 15:27 | tool | grepper |
0mArgs:{
"Expression": "config STACKDEPOT"
}Results: {
"Output": "lib/Kconfig=551=config ARCH_STACKWALK\n--\nlib/Kconfig-553-\nlib/Kconfig:554:config STACKDEPOT\nlib/Kconfig-555-\tbool\n--\nlib/Kconfig-559-\nlib/Kconfig:560:config STACKDEPOT_ALWAYS_INIT\nlib/Kconfig-561-\tbool\n--\nlib/Kconfig-565-\nlib/Kconfig:566:config STACKDEPOT_MAX_FRAMES\nlib/Kconfig-567-\tint \"Maximum number of frames in trace saved in stack depot\"\n--\nlib/Kconfig.debug=2775=config RESOURCE_KUNIT_TEST\n--\nlib/Kconfig.debug-2787-\nlib/Kconfig.debug:2788:config STACKDEPOT_KUNIT_TEST\nlib/Kconfig.debug-2789-\tbool \"KUnit test for stack depot\" if !KUNIT_ALL_TESTS\n"
} |
| 5/2 | 2026/09/08 15:27 | llm | patch-evaluator |
0mModel:gemini-3.7-flash Args: null Results: null Tokens: input: 42693 output: 32 thoughts: 17 Thoughts: Here's my attempt at summarizing the thought process, tailored for an expert audience: **Diving into the Configuration System** Okay, so I'm troubleshooting something, and my gut feeling is that it's likely a configuration issue. The architecture of this system points me towards `lib/Kconfig` as the likely culprit. Specifically, line 554 seems like a sensible place to start poking around. It's a common area for handling dependencies, option defaults, or conditional logic that could be impacting the current behavior I'm observing. Time to open that file and see what's happening. I have a feeling I can trace the problem pretty quickly from there. |
| 6/2 | 2026/09/08 15:27 | tool | read-file |
0mArgs:{
"File": "lib/Kconfig",
"FirstLine": 550,
"LineCount": 30
}Results: {
"Contents": " 550:\t# Temporary. Goes away when all archs are cleaned up\n 551:\tconfig ARCH_STACKWALK\n 552:\t bool\n 553:\t\n 554:\tconfig STACKDEPOT\n 555:\t\tbool\n 556:\t\tselect STACKTRACE\n 557:\t\thelp\n 558:\t\t Stack depot: stack trace storage that avoids duplication\n 559:\t\n 560:\tconfig STACKDEPOT_ALWAYS_INIT\n 561:\t\tbool\n 562:\t\tselect STACKDEPOT\n 563:\t\thelp\n 564:\t\t Always initialize stack depot during early boot\n 565:\t\n 566:\tconfig STACKDEPOT_MAX_FRAMES\n 567:\t\tint \"Maximum number of frames in trace saved in stack depot\"\n 568:\t\trange 1 256\n 569:\t\tdefault 64\n 570:\t\tdepends on STACKDEPOT\n 571:\t\n 572:\tconfig REF_TRACKER\n 573:\t\tbool\n 574:\t\tdepends on STACKTRACE_SUPPORT\n 575:\t\tselect STACKDEPOT\n 576:\t\n 577:\tconfig SBITMAP\n 578:\t\tbool\n 579:\t\n"
} |
| 7/2 | 2026/09/08 15:27 | llm | patch-evaluator |
0mModel:gemini-3.7-flash Args: null Results: null Tokens: input: 43233 output: 19 thoughts: 70 Thoughts: Here's my take on those thoughts: **Analyzing Kernel Memory Debugging Tool Dependencies** Okay, so STACKDEPOT has been chosen for inclusion because it's been selected by a bunch of interesting kernel components – KASAN, KFENCE, KMSAN, PAGE_OWNER, REF_TRACKER, DRM, the usual suspects in memory safety and debugging, it seems. My next step, of course, is to drill down further. I need to get a really good grasp on the extent of STACKDEPOT's usage. I'll need a better look at what's directly using it. Time to run a quick `grepper` command to scan the kernel source for all the places where `select STACKDEPOT` is declared. That should give me a solid list of the modules and subsystems that are depending on it. Gotta get the lay of the land here. |
| 8/2 | 2026/09/08 15:27 | tool | grepper |
0mArgs:{
"Expression": "select STACKDEPOT"
}Results: {
"Output": "drivers/gpu/drm/Kconfig=143=config DRM_DEBUG_DP_MST_TOPOLOGY_REFS\n--\ndrivers/gpu/drm/Kconfig-145-\tdepends on STACKTRACE_SUPPORT\ndrivers/gpu/drm/Kconfig:146: select STACKDEPOT\ndrivers/gpu/drm/Kconfig-147- select DRM_KMS_HELPER\n--\ndrivers/gpu/drm/Kconfig=158=config DRM_DEBUG_MODESET_LOCK\n--\ndrivers/gpu/drm/Kconfig-162-\tdepends on EXPERT\ndrivers/gpu/drm/Kconfig:163:\tselect STACKDEPOT\ndrivers/gpu/drm/Kconfig-164-\tdefault y if DEBUG_WW_MUTEX_SLOWPATH\n--\ndrivers/gpu/drm/Kconfig.debug=39=config DRM_DEBUG_MM\n--\ndrivers/gpu/drm/Kconfig.debug-43-\tdepends on STACKTRACE_SUPPORT\ndrivers/gpu/drm/Kconfig.debug:44:\tselect STACKDEPOT\ndrivers/gpu/drm/Kconfig.debug-45-\thelp\n--\ndrivers/gpu/drm/i915/Kconfig.debug=36=config DRM_I915_DEBUG\n--\ndrivers/gpu/drm/i915/Kconfig.debug-44-\tselect REF_TRACKER\ndrivers/gpu/drm/i915/Kconfig.debug:45:\tselect STACKDEPOT\ndrivers/gpu/drm/i915/Kconfig.debug-46-\tselect STACKTRACE\n--\ndrivers/gpu/drm/i915/Kconfig.debug=178=config DRM_I915_DEBUG_GUC\n--\ndrivers/gpu/drm/i915/Kconfig.debug-180-\tdepends on DRM_I915\ndrivers/gpu/drm/i915/Kconfig.debug:181:\tselect STACKDEPOT\ndrivers/gpu/drm/i915/Kconfig.debug-182-\tdefault n\n--\ndrivers/gpu/drm/i915/Kconfig.debug=263=config DRM_I915_DEBUG_RUNTIME_PM\n--\ndrivers/gpu/drm/i915/Kconfig.debug-267-\tselect REF_TRACKER\ndrivers/gpu/drm/i915/Kconfig.debug:268:\tselect STACKDEPOT\ndrivers/gpu/drm/i915/Kconfig.debug-269-\tselect STACKTRACE\n--\ndrivers/gpu/drm/i915/Kconfig.debug=279=config DRM_I915_DEBUG_WAKEREF\n--\ndrivers/gpu/drm/i915/Kconfig.debug-282-\tselect REF_TRACKER\ndrivers/gpu/drm/i915/Kconfig.debug:283:\tselect STACKDEPOT\ndrivers/gpu/drm/i915/Kconfig.debug-284-\tselect STACKTRACE\n--\ndrivers/gpu/drm/xe/Kconfig.debug=105=config DRM_XE_DEBUG_GUC\n--\ndrivers/gpu/drm/xe/Kconfig.debug-108- default n\ndrivers/gpu/drm/xe/Kconfig.debug:109: select STACKDEPOT\ndrivers/gpu/drm/xe/Kconfig.debug-110- help\n--\nlib/Kconfig=560=config STACKDEPOT_ALWAYS_INIT\nlib/Kconfig-561-\tbool\nlib/Kconfig:562:\tselect STACKDEPOT\nlib/Kconfig-563-\thelp\n--\nlib/Kconfig=572=config REF_TRACKER\n--\nlib/Kconfig-574-\tdepends on STACKTRACE_SUPPORT\nlib/Kconfig:575:\tselect STACKDEPOT\nlib/Kconfig-576-\n--\nlib/Kconfig.kasan=46=menuconfig KASAN\n--\nlib/Kconfig.kasan-52-\tdepends on SYSFS \u0026\u0026 !SLUB_TINY\nlib/Kconfig.kasan:53:\tselect STACKDEPOT_ALWAYS_INIT\nlib/Kconfig.kasan-54-\thelp\n--\nlib/Kconfig.kmsan=8=config KMSAN\n--\nlib/Kconfig.kmsan-12-\tdepends on !PREEMPT_RT\nlib/Kconfig.kmsan:13:\tselect STACKDEPOT\nlib/Kconfig.kmsan:14:\tselect STACKDEPOT_ALWAYS_INIT\nlib/Kconfig.kmsan-15-\thelp\n--\nmm/Kconfig.debug=48=config SLUB_DEBUG\n--\nmm/Kconfig.debug-51-\tdepends on SYSFS \u0026\u0026 !SLUB_TINY\nmm/Kconfig.debug:52:\tselect STACKDEPOT if STACKTRACE_SUPPORT\nmm/Kconfig.debug-53-\thelp\n--\nmm/Kconfig.debug=59=config SLUB_DEBUG_ON\n--\nmm/Kconfig.debug-61-\tdepends on SLUB_DEBUG\nmm/Kconfig.debug:62:\tselect STACKDEPOT_ALWAYS_INIT if STACKTRACE_SUPPORT\nmm/Kconfig.debug-63-\tdefault n\n--\nmm/Kconfig.debug=105=config PAGE_OWNER\n--\nmm/Kconfig.debug-109-\tselect STACKTRACE\nmm/Kconfig.debug:110:\tselect STACKDEPOT\nmm/Kconfig.debug-111-\tselect PAGE_EXTENSION\n--\nmm/Kconfig.debug=240=config DEBUG_KMEMLEAK\n--\nmm/Kconfig.debug-246-\tselect CRC32\nmm/Kconfig.debug:247:\tselect STACKDEPOT\nmm/Kconfig.debug:248:\tselect STACKDEPOT_ALWAYS_INIT if !DEBUG_KMEMLEAK_DEFAULT_OFF\nmm/Kconfig.debug-249-\thelp\n"
} |
| 9/2 | 2026/09/08 15:27 | llm | patch-evaluator |
0mModel:gemini-3.7-flash Args: null Results: null Tokens: input: 44992 output: 25 thoughts: 944 Thoughts: **Fuzzing Feasibility of Stack Depot KUnit Test** Okay, so there's this new `STACKDEPOT_KUNIT_TEST` KUnit test config. I need to figure out if this code path is worth fuzzing. `STACKDEPOT` itself is part of core memory management and debugging infrastructure, which is a good starting point. KASAN, SLUB_DEBUG, PAGE_OWNER, KMEMLEAK, etc. all utilize it. Let's see, does this meet the criteria for "WorthFuzzing=true"? The patch introduces this new trie storage backend in `lib/stackdepot.c`, along with frame compression for x86/arm64. It modifies a whole host of functions: `stack_depot_save`, `stack_depot_save_flags`, `stack_depot_fetch_into`, `stack_depot_print`, `stack_depot_snprint`, `__stack_depot_get_stack_record`, `stack_depot_put`, etc., and the impact extends to `mm/slub.c`, `mm/kmemleak.c`, `mm/page_owner.c`, `mm/kmsan/report.c`, and `drivers/gpu/drm/drm_modeset_lock.c`. Absolutely, this qualifies as significant modification to core subsystems and data structures, so it's definitely worth fuzzing. Now, which configs do I need to enable to actually exercise this code? I've got to figure out how `STACKDEPOT` gets enabled. It's a `bool` config without a prompt, so it's likely selected by other configs. Let's see... looking at `lib/Kconfig`, `STACKDEPOT` has no prompt and neither does `STACKDEPOT_ALWAYS_INIT`. `STACKDEPOT_KUNIT_TEST` depends on `KUNIT=y && STACKDEPOT`. Alright, what are the upstream configs that commonly enable `STACKDEPOT` or can be enabled via syzkaller? KASAN selects `STACKDEPOT_ALWAYS_INIT`, which in turn selects `STACKDEPOT`. KMSAN also selects `STACKDEPOT` and `STACKDEPOT_ALWAYS_INIT`. `DEBUG_KMEMLEAK`, `PAGE_OWNER`, `SLUB_DEBUG`, and `REF_TRACKER` all select `STACKDEPOT`. Okay, that gives me a clear path for enabling it. One more thing: how does `stackdepot.trie_enabled` work? Checking `lib/stackdepot.c`, it looks like this is exposed as a module parameter: `stackdepot.trie_enabled`, which can be set at boot with `stackdepot.trie_enabled=1`! That's excellent for testing different modes. Now, what actually compiles `lib/stackdepot.c`? A quick look at `lib/Makefile` will tell me. |
| 10/2 | 2026/09/08 15:27 | tool | grepper |
0mArgs:{
"Expression": "stackdepot",
"PathPrefix": "lib/"
}Results: {
"Output": "lib/Makefile=244=obj-$(CONFIG_IRQ_POLL) += irq_poll.o\nlib/Makefile-245-\nlib/Makefile:246:# stackdepot.c should not be instrumented or call instrumented functions.\nlib/Makefile-247-# Prevent the compiler from calling builtins like memcmp() or bcmp() from this\nlib/Makefile-248-# file.\nlib/Makefile:249:CFLAGS_stackdepot.o += -fno-builtin\nlib/Makefile:250:CONTEXT_ANALYSIS_stackdepot.o := y\nlib/Makefile:251:obj-$(CONFIG_STACKDEPOT) += stackdepot.o\nlib/Makefile:252:KASAN_SANITIZE_stackdepot.o := n\nlib/Makefile:253:# In particular, instrumenting stackdepot.c with KMSAN will result in infinite\nlib/Makefile-254-# recursion.\nlib/Makefile:255:KMSAN_SANITIZE_stackdepot.o := n\nlib/Makefile:256:KCOV_INSTRUMENT_stackdepot.o := n\nlib/Makefile-257-\n--\nlib/ref_tracker.c-9-#include \u003clinux/stacktrace.h\u003e\nlib/ref_tracker.c:10:#include \u003clinux/stackdepot.h\u003e\nlib/ref_tracker.c-11-#include \u003clinux/seq_file.h\u003e\n--\nlib/stackdepot.c-16-\nlib/stackdepot.c:17:#define pr_fmt(fmt) \"stackdepot: \" fmt\nlib/stackdepot.c-18-\n--\nlib/stackdepot.c-40-#include \u003clinux/stacktrace.h\u003e\nlib/stackdepot.c:41:#include \u003clinux/stackdepot.h\u003e\nlib/stackdepot.c-42-#include \u003clinux/string.h\u003e\n--\nlib/stackdepot.c-46-\nlib/stackdepot.c:47:#include \u003casm/stackdepot.h\u003e\nlib/stackdepot.c-48-\n--\nlib/stackdepot.c=848=int __init stack_depot_early_init(void)\n--\nlib/stackdepot.c-887-\tpr_info(\"allocating hash table via alloc_large_system_hash\\n\");\nlib/stackdepot.c:888:\tstack_table = alloc_large_system_hash(\"stackdepot\",\nlib/stackdepot.c-889-\t\t\t\t\t\tsizeof(struct list_head),\n--\nlib/stackdepot.c=1292=static inline\nlib/stackdepot.c:1293:int stackdepot_memcmp(const unsigned long *u1, const unsigned long *u2,\nlib/stackdepot.c-1294-\t\t\tunsigned int n)\n--\nlib/stackdepot.c=1304=static inline struct stack_record *find_stack(struct list_head *bucket,\n--\nlib/stackdepot.c-1332-\t\t */\nlib/stackdepot.c:1333:\t\tif (data_race(stackdepot_memcmp(entries, stack-\u003eentries, size)))\nlib/stackdepot.c-1334-\t\t\tcontinue;\n--\nlib/stackdepot.c=2493=static int depot_debugfs_init(void)\n--\nlib/stackdepot.c-2499-\nlib/stackdepot.c:2500:\tdir = debugfs_create_dir(\"stackdepot\", NULL);\nlib/stackdepot.c-2501-\tdebugfs_create_file(\"stats\", 0444, dir, NULL, \u0026stats_fops);\n--\nlib/tests/Makefile=50=obj-$(CONFIG_SLUB_KUNIT_TEST) += slub_kunit.o\nlib/tests/Makefile:51:obj-$(CONFIG_STACKDEPOT_KUNIT_TEST) += stackdepot_kunit.o\nlib/tests/Makefile-52-obj-$(CONFIG_TEST_SORT) += test_sort.o\n--\nlib/tests/stackdepot_kunit.c-8-#include \u003clinux/moduleparam.h\u003e\nlib/tests/stackdepot_kunit.c:9:#include \u003clinux/stackdepot.h\u003e\nlib/tests/stackdepot_kunit.c-10-#include \u003clinux/stacktrace.h\u003e\n--\nlib/tests/stackdepot_kunit.c-12-\nlib/tests/stackdepot_kunit.c:13:#include \u003casm/stackdepot.h\u003e\nlib/tests/stackdepot_kunit.c-14-\n--\nlib/tests/stackdepot_kunit.c=16=module_param_named(trie_pool_limit, expected_trie_pool_limit, int, 0);\nlib/tests/stackdepot_kunit.c:17:MODULE_PARM_DESC(trie_pool_limit, \"Expected stackdepot hash/trie pool split\");\nlib/tests/stackdepot_kunit.c-18-\n--\nlib/tests/stackdepot_kunit.c-21-\nlib/tests/stackdepot_kunit.c:22:static inline unsigned long stackdepot_arm64_frame(long offset)\nlib/tests/stackdepot_kunit.c-23-{\n--\nlib/tests/stackdepot_kunit.c-27-\nlib/tests/stackdepot_kunit.c:28:static unsigned long stackdepot_test_frame(unsigned int i)\nlib/tests/stackdepot_kunit.c-29-{\n--\nlib/tests/stackdepot_kunit.c-31-\treturn i \u0026 1 ? 0x1000UL + i * 0x1000UL :\nlib/tests/stackdepot_kunit.c:32:\t\tstackdepot_arm64_frame(i * 4);\nlib/tests/stackdepot_kunit.c-33-#elif defined(CONFIG_X86_64) \u0026\u0026 !defined(CONFIG_UML)\n--\nlib/tests/stackdepot_kunit.c-40-\nlib/tests/stackdepot_kunit.c:41:static void stackdepot_trie_max_path_roundtrip(struct kunit *test)\nlib/tests/stackdepot_kunit.c-42-{\n--\nlib/tests/stackdepot_kunit.c-60-\tfor (i = 0; i \u003c CONFIG_STACKDEPOT_MAX_FRAMES; i++)\nlib/tests/stackdepot_kunit.c:61:\t\tentries[i] = stackdepot_test_frame(i);\nlib/tests/stackdepot_kunit.c-62-\n--\nlib/tests/stackdepot_kunit.c-79-\nlib/tests/stackdepot_kunit.c:80:static void stackdepot_save_flags_public(struct kunit *test)\nlib/tests/stackdepot_kunit.c-81-{\n--\nlib/tests/stackdepot_kunit.c-189-\nlib/tests/stackdepot_kunit.c:190:static void stackdepot_snprint_public(struct kunit *test)\nlib/tests/stackdepot_kunit.c-191-{\n--\nlib/tests/stackdepot_kunit.c-213-\tfor (i = 0; i \u003c nr_entries; i++)\nlib/tests/stackdepot_kunit.c:214:\t\tentries[i] = stackdepot_test_frame(i);\nlib/tests/stackdepot_kunit.c-215-\n--\nlib/tests/stackdepot_kunit.c-243-\nlib/tests/stackdepot_kunit.c:244:static void stackdepot_countable_public(struct kunit *test)\nlib/tests/stackdepot_kunit.c-245-{\n--\nlib/tests/stackdepot_kunit.c-297-\nlib/tests/stackdepot_kunit.c:298:static void stackdepot_fetch_into_roundtrip(struct kunit *test)\nlib/tests/stackdepot_kunit.c-299-{\n--\nlib/tests/stackdepot_kunit.c-327-\nlib/tests/stackdepot_kunit.c:328:static void stackdepot_fetch_into_rejects_missing_or_short_stack(struct kunit *test)\nlib/tests/stackdepot_kunit.c-329-{\n--\nlib/tests/stackdepot_kunit.c-362-\nlib/tests/stackdepot_kunit.c:363:static void stackdepot_trie_topology_roundtrip(struct kunit *test,\nlib/tests/stackdepot_kunit.c-364-\t\t\t\t\t bool constrained)\n--\nlib/tests/stackdepot_kunit.c-437-\nlib/tests/stackdepot_kunit.c:438:static void stackdepot_trie_topology_allocating(struct kunit *test)\nlib/tests/stackdepot_kunit.c-439-{\nlib/tests/stackdepot_kunit.c:440:\tstackdepot_trie_topology_roundtrip(test, false);\nlib/tests/stackdepot_kunit.c-441-}\nlib/tests/stackdepot_kunit.c-442-\nlib/tests/stackdepot_kunit.c:443:static void stackdepot_trie_topology_constrained(struct kunit *test)\nlib/tests/stackdepot_kunit.c-444-{\nlib/tests/stackdepot_kunit.c:445:\tstackdepot_trie_topology_roundtrip(test, true);\nlib/tests/stackdepot_kunit.c-446-}\nlib/tests/stackdepot_kunit.c-447-\nlib/tests/stackdepot_kunit.c:448:static void stackdepot_frame_storage_roundtrip(struct kunit *test)\nlib/tests/stackdepot_kunit.c-449-{\n--\nlib/tests/stackdepot_kunit.c-456-\tunsigned long entries[] = {\nlib/tests/stackdepot_kunit.c:457:\t\tstackdepot_arm64_frame(S32_MIN),\nlib/tests/stackdepot_kunit.c-458-\t\t0x1000UL,\nlib/tests/stackdepot_kunit.c:459:\t\tstackdepot_arm64_frame(S32_MAX),\nlib/tests/stackdepot_kunit.c-460-\t};\n--\nlib/tests/stackdepot_kunit.c-485-\nlib/tests/stackdepot_kunit.c:486:static void stackdepot_frame_raw_fallback(struct kunit *test)\nlib/tests/stackdepot_kunit.c-487-{\n--\nlib/tests/stackdepot_kunit.c-500-#if defined(CONFIG_X86_64) \u0026\u0026 !defined(CONFIG_UML)\nlib/tests/stackdepot_kunit.c:501:static void stackdepot_frame_x86_64(struct kunit *test)\nlib/tests/stackdepot_kunit.c-502-{\n--\nlib/tests/stackdepot_kunit.c-520-#ifdef CONFIG_ARM64\nlib/tests/stackdepot_kunit.c:521:static void stackdepot_frame_arm64(struct kunit *test)\nlib/tests/stackdepot_kunit.c-522-{\n--\nlib/tests/stackdepot_kunit.c-525-\tlong offset = 0x123456;\nlib/tests/stackdepot_kunit.c:526:\tunsigned long frame = stackdepot_arm64_frame(offset);\nlib/tests/stackdepot_kunit.c-527-\tunsigned long out;\n--\nlib/tests/stackdepot_kunit.c-536-\nlib/tests/stackdepot_kunit.c:537:\tframe = stackdepot_arm64_frame(negative_offset);\nlib/tests/stackdepot_kunit.c-538-\tcompressed = arch_stack_depot_frame_try_compress(frame, \u0026payload);\n--\nlib/tests/stackdepot_kunit.c-543-\nlib/tests/stackdepot_kunit.c:544:\tframe = stackdepot_arm64_frame(positive_offset);\nlib/tests/stackdepot_kunit.c-545-\tcompressed = arch_stack_depot_frame_try_compress(frame, \u0026payload);\n--\nlib/tests/stackdepot_kunit.c-552-\nlib/tests/stackdepot_kunit.c:553:static struct kunit_case stackdepot_test_cases[] = {\nlib/tests/stackdepot_kunit.c:554:\tKUNIT_CASE(stackdepot_trie_max_path_roundtrip),\nlib/tests/stackdepot_kunit.c:555:\tKUNIT_CASE(stackdepot_save_flags_public),\nlib/tests/stackdepot_kunit.c:556:\tKUNIT_CASE(stackdepot_snprint_public),\nlib/tests/stackdepot_kunit.c:557:\tKUNIT_CASE(stackdepot_countable_public),\nlib/tests/stackdepot_kunit.c:558:\tKUNIT_CASE(stackdepot_fetch_into_roundtrip),\nlib/tests/stackdepot_kunit.c:559:\tKUNIT_CASE(stackdepot_fetch_into_rejects_missing_or_short_stack),\nlib/tests/stackdepot_kunit.c:560:\tKUNIT_CASE(stackdepot_trie_topology_allocating),\nlib/tests/stackdepot_kunit.c:561:\tKUNIT_CASE(stackdepot_trie_topology_constrained),\nlib/tests/stackdepot_kunit.c:562:\tKUNIT_CASE(stackdepot_frame_storage_roundtrip),\nlib/tests/stackdepot_kunit.c:563:\tKUNIT_CASE(stackdepot_frame_raw_fallback),\nlib/tests/stackdepot_kunit.c-564-#if defined(CONFIG_X86_64) \u0026\u0026 !defined(CONFIG_UML)\nlib/tests/stackdepot_kunit.c:565:\tKUNIT_CASE(stackdepot_frame_x86_64),\nlib/tests/stackdepot_kunit.c-566-#endif\nlib/tests/stackdepot_kunit.c-567-#ifdef CONFIG_ARM64\nlib/tests/stackdepot_kunit.c:568:\tKUNIT_CASE(stackdepot_frame_arm64),\nlib/tests/stackdepot_kunit.c-569-#endif\n--\nlib/tests/stackdepot_kunit.c-572-\nlib/tests/stackdepot_kunit.c:573:static struct kunit_suite stackdepot_test_suite = {\nlib/tests/stackdepot_kunit.c:574:\t.name = \"stackdepot\",\nlib/tests/stackdepot_kunit.c:575:\t.test_cases = stackdepot_test_cases,\nlib/tests/stackdepot_kunit.c-576-};\nlib/tests/stackdepot_kunit.c-577-\nlib/tests/stackdepot_kunit.c:578:kunit_test_suite(stackdepot_test_suite);\nlib/tests/stackdepot_kunit.c-579-\n"
} |
| 11/2 | 2026/09/08 15:27 | llm | patch-evaluator |
0mModel:gemini-3.7-flash Args: null Results: null Tokens: input: 50614 output: 31 thoughts: 1744 Thoughts: **Analyzing Configuration and Focus Symbols for Stack Depot Implementation** Okay, let's break this down. My goal is to determine the correct configuration options and focus symbols for fuzzing this stack depot implementation. First, I need to figure out which kernel configurations should be enabled in `EnableConfigs`. The primary candidate is `CONFIG_STACKDEPOT`, or perhaps `STACKDEPOT_ALWAYS_INIT` or `PAGE_OWNER` or one of the other options that enable stack depot functionality. I need to be sure to include `STACKDEPOT` or `STACKDEPOT_ALWAYS_INIT`. I'll consider `STACKDEPOT_KUNIT_TEST` separately, ensuring I don't enable it unless necessary. `STACKDEPOT` itself is the standard, primary config. Next, I need to list the configs: `STACKDEPOT`. Now, the crucial step: identifying the `FocusSymbols`. Following the rules for identifying `FocusSymbols`, I have to avoid ubiquitous lifecycle hot-paths and instead target specific feature logic and entry points. I should also be mindful of static inline functions in headers. I'll start by examining non-static functions and key internal functions in `lib/stackdepot.c`. I have a list of all non-static and modified functions: `stack_depot_save`, `stack_depot_save_flags`, `stack_depot_fetch`, `stack_depot_fetch_into`, `stack_depot_put`, `stack_depot_print`, `stack_depot_snprint`, `__stack_depot_get_stack_record`, `stack_depot_init`, `stack_depot_early_init`. There are also many internal functions, that are essential to the new trie backend such as `stack_depot_trie_save`, `stack_depot_trie_save_constrained`, and others. The concern with `stack_depot_save` and `stack_depot_fetch_into` is if they might be called on every allocation, especially if KASAN/SLUB_DEBUG/PAGE_OWNER are enabled. However, these are the *core* functions of the subsystem under test, so that's the point. I need to select the ones that implement the trie backend and new API entry points: `stack_depot_fetch_into`, `stack_depot_trie_save`, `stack_depot_trie_save_constrained`, `stack_depot_trie_insert`, `stack_depot_trie_lookup`, `trie_insert_path`, `trie_split_child`, `trie_promote_child`, `trie_fetch_handle_into`, `trie_fetch_handle_range`, `trie_pool_alloc`, `trie_pool_reserve_slots`, `trie_drain_pending_children`, `trie_reparent_children`, `trie_path_alloc`, `trie_side_table_publish`, `trie_side_table_lookup`, `trie_side_table_prepare_stack_slot`, `trie_print`, `trie_snprint`. Now I need to also check other modified files. Specifically, I'll review functions from `mm/kmsan/report.c` (`kmsan_print_origin`), `drivers/gpu/drm/drm_modeset_lock.c` (`__drm_stack_depot_print`), `mm/page_owner.c` (`create_dummy_stack`, `save_stack`), `mm/kmemleak.c` (`__print_unreferenced`) and `mm/slub.c` (`__kmem_obj_info`, `slab_debugfs_show`). One more thing: I need to consider if symbols in `lib/stackdepot.c` might be static, because it can be compiled into the kernel. The crucial piece of information is the line `KCOV_INSTRUMENT_stackdepot.o := n` in `lib/Makefile`. This means kcov instrumentation is disabled for `stackdepot.o`. This affects the coverage analysis; I need to keep this in mind. |
| 12/2 | 2026/09/08 15:27 | tool | read-file |
0mArgs:{
"File": "lib/Makefile",
"FirstLine": 244,
"LineCount": 20
}Results: {
"Contents": " 244:\tobj-$(CONFIG_IRQ_POLL) += irq_poll.o\n 245:\t\n 246:\t# stackdepot.c should not be instrumented or call instrumented functions.\n 247:\t# Prevent the compiler from calling builtins like memcmp() or bcmp() from this\n 248:\t# file.\n 249:\tCFLAGS_stackdepot.o += -fno-builtin\n 250:\tCONTEXT_ANALYSIS_stackdepot.o := y\n 251:\tobj-$(CONFIG_STACKDEPOT) += stackdepot.o\n 252:\tKASAN_SANITIZE_stackdepot.o := n\n 253:\t# In particular, instrumenting stackdepot.c with KMSAN will result in infinite\n 254:\t# recursion.\n 255:\tKMSAN_SANITIZE_stackdepot.o := n\n 256:\tKCOV_INSTRUMENT_stackdepot.o := n\n 257:\t\n 258:\tobj-$(CONFIG_REF_TRACKER) += ref_tracker.o\n 259:\t\n 260:\tlibfdt_files = fdt.o fdt_ro.o fdt_wip.o fdt_rw.o fdt_sw.o fdt_strerror.o \\\n 261:\t\t fdt_empty_tree.o fdt_addresses.o\n 262:\t$(foreach file, $(libfdt_files), \\\n 263:\t\t$(eval CFLAGS_$(file) = -I $(srctree)/scripts/dtc/libfdt))\n"
} |
| 13/2 | 2026/09/08 15:27 | llm | patch-evaluator |
0mModel:gemini-3.7-flash Args: null Results: null Tokens: input: 53716 output: 429 thoughts: 642 Thoughts: **My Assessment of the Stack Depot Instrumentation for Fuzzing** Okay, so even though we've disabled KCOV on `stackdepot.o` itself, which is standard for memory infrastructure because of the risk of recursion, we're still good to go. The callers in `mm/` and our tests *are* instrumented, and Syzkaller can still target both the caller functions and, more importantly, stackdepot functions directly. We also have Syzkaller symbol coverage filters and breakpoint-based guidance using kallsyms, which can specifically leverage stackdepot or caller functions. This is crucial for effectiveness. Therefore, the approach is still sound. It's imperative that we identify both the key StackDepot functions *and* the significant caller functions. So, let's nail down that symbol list, and let's make it comprehensive: * `stack_depot_fetch_into` * `stack_depot_trie_save` * `stack_depot_trie_save_constrained` * `stack_depot_trie_insert` * `trie_split_child` * `trie_insert_path` * `trie_promote_child` * `trie_fetch_handle_into` * `trie_fetch_handle_range` * `trie_pool_alloc` * `trie_pool_reserve_slots` * `trie_drain_pending_children` * `trie_side_table_publish` * `trie_side_table_lookup` * `trie_side_table_prepare_stack_slot` * `trie_print` * `trie_snprint` * `kmsan_print_origin` * `__drm_stack_depot_print` * `slab_debugfs_show` Now, regarding the `set-results` parameters, let's make sure we hit these correctly: * `WorthFuzzing`: True, absolutely. This is a new trie-based storage backend with frame compression and a lockless RCU reads, dynamic pool slot reservation, and it modifies several core callers (SLUB, KMSAN, kmemleak, DRM, page_owner). These are significant changes to core memory management. This is definitely worth fuzzing and also reachable in standard virtualization environments, so we have good coverage. * `Reasoning`: We need to explicitly state that the patch implements a new trie-based storage backend for stack depot with frame compression, lockless RCU reads, dynamic pool slot reservation, and updates several callers (SLUB, KMSAN, kmemleak, DRM, page_owner). These changes modify core memory management infrastructure and are reachable in standard virtualization environments. * `EnableConfigs`: `STACKDEPOT` is the primary one, of course, but let's include related options for maximum testing. We can go with `["STACKDEPOT", "STACKDEPOT_KUNIT_TEST", "PAGE_OWNER", "DEBUG_KMEMLEAK", "SLUB_DEBUG"]` or we might decide to limit to just `["STACKDEPOT", "STACKDEPOT_KUNIT_TEST"]`. * `FocusSymbols`: We must list the specific *new* or modified, non-ubiquitous functions we've identified. Okay, let's do a final check to ensure we're following the specified format: * `WorthFuzzing`: Boolean - Yes * `FocusSymbols`: List of strings - Yes, we have that list * `EnableConfigs`: List of strings without `CONFIG_` prefix - Yes, that's the intention * `Reasoning`: String - Yes, we have a clear explanation. Alright, with all that confirmed, I'm ready to call `set-results`. Let's get this fuzzing run going! |
| 14/2 | 2026/09/08 15:27 | tool | set-results |
0mArgs:{
"EnableConfigs": [
"STACKDEPOT",
"STACKDEPOT_KUNIT_TEST",
"PAGE_OWNER",
"DEBUG_KMEMLEAK",
"SLUB_DEBUG"
],
"FocusSymbols": [
"stack_depot_fetch_into",
"stack_depot_trie_save",
"stack_depot_trie_save_constrained",
"stack_depot_trie_insert",
"stack_depot_trie_lookup",
"trie_insert_path",
"trie_split_child",
"trie_promote_child",
"trie_fetch_handle_into",
"trie_fetch_handle_range",
"trie_pool_alloc",
"trie_pool_reserve_slots",
"trie_drain_pending_children",
"trie_reparent_children",
"trie_path_alloc",
"trie_side_table_publish",
"trie_side_table_lookup",
"trie_side_table_prepare_stack_slot",
"trie_print",
"trie_snprint",
"kmsan_print_origin",
"__drm_stack_depot_print",
"slab_debugfs_show"
],
"Reasoning": "The patch series introduces a new trie-based storage backend for stack depot with arch-specific frame compression (x86-64 and arm64), RCU-based child management, slot allocation/reservation within depot pools, side-table mappings, and a new stack_depot_fetch_into() API. It also updates several callers across the kernel (SLUB, page_owner, kmemleak, KMSAN, DRM) and adds runtime checks and assertions. These changes modify core memory management infrastructure and are reachable during kernel execution in standard virtualized environments.",
"WorthFuzzing": true
}Results: {
"EnableConfigs": [
"STACKDEPOT",
"STACKDEPOT_KUNIT_TEST",
"PAGE_OWNER",
"DEBUG_KMEMLEAK",
"SLUB_DEBUG"
],
"FocusSymbols": [
"stack_depot_fetch_into",
"stack_depot_trie_save",
"stack_depot_trie_save_constrained",
"stack_depot_trie_insert",
"stack_depot_trie_lookup",
"trie_insert_path",
"trie_split_child",
"trie_promote_child",
"trie_fetch_handle_into",
"trie_fetch_handle_range",
"trie_pool_alloc",
"trie_pool_reserve_slots",
"trie_drain_pending_children",
"trie_reparent_children",
"trie_path_alloc",
"trie_side_table_publish",
"trie_side_table_lookup",
"trie_side_table_prepare_stack_slot",
"trie_print",
"trie_snprint",
"kmsan_print_origin",
"__drm_stack_depot_print",
"slab_debugfs_show"
],
"Reasoning": "The patch series introduces a new trie-based storage backend for stack depot with arch-specific frame compression (x86-64 and arm64), RCU-based child management, slot allocation/reservation within depot pools, side-table mappings, and a new stack_depot_fetch_into() API. It also updates several callers across the kernel (SLUB, page_owner, kmemleak, KMSAN, DRM) and adds runtime checks and assertions. These changes modify core memory management infrastructure and are reachable during kernel execution in standard virtualized environments.",
"WorthFuzzing": true
} |
| 15/1 | 2026/09/08 15:27 | action | If |
0mArgs:{
"WorthFuzzing": true
}Results: null |
| 16/2 | 2026/09/08 15:27 | agent | kmsan-evaluator |
0mModel:core Args: null Results: {
"KMSANReasoning": "The patch series implements a trie storage backend and address compression for the stack depot subsystem (lib/stackdepot.c), updates stack depot consumers (such as kmemleak, SLUB debug, DRM, and KMSAN's origin printer) to use the new `stack_depot_fetch_into()` / `stack_depot_snprint()` APIs, and adds KUnit tests.\n\n1. KASAN / Lockdep / Standard Debuggers Applicability:\n- The changes heavily involve internal kernel data structure management, including RCU-protected pointer updates, bitmap slot reservations within memory pools, spinlocks, and buffer copies for stack frames.\n- Potential defects such as out-of-bounds indexing in child arrays/bitmaps, memory corruption, use-after-free during node splitting/reparenting, and concurrency/locking deadlocks are fully covered by KASAN, KCSAN, and LOCKDEP.\n\n2. KMSAN Applicability:\n- No kernel structures are copied or exposed to userspace (no copy_to_user, put_user, netlink, ioctl, etc.).\n- All internal allocations (directory pages, side tables, node arrays, pools) are allocated using zeroing allocators (`get_zeroed_page()`, `kvzalloc()`, `memblock_alloc()`) or are explicitly populated before being read or linked into the trie.\n- Callers of `stack_depot_fetch_into()` only access entries up to the returned `nr_entries` count, and stack depot explicitly invokes `kmsan_unpoison_memory()` when populating frame entries.\n- No uninitialized variables or uninitialized branching conditions are introduced.\n\nBecause the changes present memory safety and synchronization concerns rather than uninitialized memory access or info-leak risks, a dedicated KMSAN fuzzing session is not justified.",
"NeedsKMSAN": false
}Instruction: You are an expert Linux kernel security engineer specializing in kernel memory
error detectors (KASAN and KMSAN). Your job is to review the provided patch series and
determine if the code changes justify spawning a dedicated KMSAN (KernelMemorySanitizer)
fuzzing session in addition to standard KASAN fuzzing.
CRITICAL DISTINCTION BETWEEN KASAN AND KMSAN:
- Standard KASAN kernel builds (upstream-apparmor-kasan.config) already enable
a comprehensive suite of debugging tools and sanitizers, including KASAN
(out-of-bounds accesses, use-after-free, double free, invalid free), LOCKDEP
(locking bugs and deadlocks), UB-sanitizers, and memory corruption checks.
- KMSAN (KernelMemorySanitizer) detects reads of UNINITIALIZED memory (stack, heap,
or page allocations) and kernel-to-user memory info-leaks.
Rule: THERE IS NO SENSE IN RUNNING A KMSAN SESSION IF A BUG CAN BE CAUGHT BY KASAN,
LOCKDEP, OR OTHER STANDARD BUG DETECTORS.
A dedicated KMSAN fuzzing session incurs significant resource costs. You must ONLY
set NeedsKMSAN=true if the code changes introduce or expose UNINITIALIZED MEMORY risks
that are detected ONLY by KMSAN.
Look holistically at the patch series and surrounding code. Even if no direct
uninitialized field accesses or new buffer allocations are added in the diff itself,
a patch may alter control flow, bounds checking, or data length calculations in ways
that change how the rest of the code operates on existing buffers (e.g. allowing
uninitialized stack/heap memory to be read, copied to user space, or used in control
flow). Do not hesitate to use your code access tools to inspect the surrounding code,
called functions, and callers.
Set NeedsKMSAN=true ONLY IF the patch introduces or modifies:
1. Kernel structures sent to user space (via copy_to_user, put_user, netlink skb
attributes, ioctl output arguments, socket options, or BPF buffers) where fields
or structure padding might not be fully initialized/zeroed.
2. Conditional logic or branching that depends on potentially uninitialized variables
or struct fields.
3. Allocation or initialization of complex data structures where uninitialized fields
could be read later in reachable code paths.
4. Bounds checks, lengths, or logic in a way that allows surrounding code to access
uninitialized bytes of existing buffers.
Set NeedsKMSAN=false IF:
- The code changes primarily risk out-of-bounds access, array overflows, NULL pointer
dereferences, locking deadlocks, or use-after-free bugs (these are already caught
by KASAN, LOCKDEP, or standard bug detectors).
- All stack/heap structures touched or introduced by the patch are fully zeroed
or initialized (e.g. using = {0}, memset, kzalloc) before being read or copied.
- The patch does not introduce any risk of uninitialized memory usage or info-leaks.
Use your code access tools to inspect the surrounding code if necessary, then provide
detailed KMSANReasoning contrasting KASAN vs KMSAN applicability for this patch.
Prefer calling several tools at the same time to save round-trips.
Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.
Prompt: Target architecture: amd64
For your convenience, here is the diff of the changes:
commit efdf021b1a377c26b41b0ed29bb41a4fbb95bac4
Author: syz-cluster <triage@syzkaller.com>
Date: Tue Sep 8 15:27:13 2026 +0000
syz-cluster: applied patch under review
diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 68647ff4bdd24..b02bcbaef5dcb 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -7449,6 +7449,13 @@ Kernel parameters
stack traces. Pools are allocated on-demand up to this
limit. Default value is 8191 pools.
+ stackdepot.trie_enabled= [KNL]
+ Format: <bool>
+ Enable trie storage for persistent, non-refcounted
+ stack depot records at boot. Disabled by default.
+ stack_depot_max_pools must leave unused pool-index
+ values for trie handles.
+
stacktrace [FTRACE]
Enable the stack tracer on boot up.
diff --git a/arch/arm64/include/asm/stackdepot.h b/arch/arm64/include/asm/stackdepot.h
new file mode 100644
index 0000000000000..df8959d593366
--- /dev/null
+++ b/arch/arm64/include/asm/stackdepot.h
@@ -0,0 +1,42 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __ASM_STACKDEPOT_H
+#define __ASM_STACKDEPOT_H
+
+#include <linux/types.h>
+#include <asm/sections.h>
+
+/*
+ * Modules are allocated inside a 2 GB relocation window containing the
+ * kernel image. Store a signed 32-bit offset from _text so compression is
+ * independent of 4 GB high-bit boundaries crossed by that window.
+ */
+static inline unsigned long arch_stack_depot_frame_from_payload(u32 payload)
+{
+ long offset;
+
+ offset = (s32)payload;
+ if (offset < 0)
+ return (unsigned long)_text - (unsigned long)(-offset);
+ return (unsigned long)_text + (unsigned long)offset;
+}
+
+static inline bool
+arch_stack_depot_frame_try_compress(unsigned long frame, u32 *payload)
+{
+ u32 candidate;
+
+ candidate = (u32)(frame - (unsigned long)_text);
+ if (arch_stack_depot_frame_from_payload(candidate) != frame)
+ return false;
+
+ *payload = candidate;
+ return true;
+}
+
+static inline void
+arch_stack_depot_frame_decompress(u32 payload, unsigned long *frame)
+{
+ *frame = arch_stack_depot_frame_from_payload(payload);
+}
+
+#endif /* __ASM_STACKDEPOT_H */
diff --git a/arch/um/include/asm/Kbuild b/arch/um/include/asm/Kbuild
index 8fdc0bd9ab6fb..14778d2457d79 100644
--- a/arch/um/include/asm/Kbuild
+++ b/arch/um/include/asm/Kbuild
@@ -21,6 +21,7 @@ generic-y += preempt.h
generic-y += ring_buffer.h
generic-y += runtime-const.h
generic-y += softirq_stack.h
+generic-y += stackdepot.h
generic-y += switch_to.h
generic-y += topology.h
generic-y += trace_clock.h
diff --git a/arch/x86/include/asm/stackdepot.h b/arch/x86/include/asm/stackdepot.h
new file mode 100644
index 0000000000000..9a8d04fa8c1c8
--- /dev/null
+++ b/arch/x86/include/asm/stackdepot.h
@@ -0,0 +1,37 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_X86_STACKDEPOT_H
+#define _ASM_X86_STACKDEPOT_H
+
+#include <linux/types.h>
+
+#ifdef CONFIG_X86_64
+/*
+ * Compress canonical kernel text/module addresses whose upper 32 bits are all
+ * ones. Other kernel virtual addresses stay raw, so decompression reconstructs
+ * the original frame by restoring this prefix.
+ */
+#define STACK_DEPOT_X86_64_FRAME_PREFIX 0xffffffff00000000UL
+#define STACK_DEPOT_X86_64_FRAME_LOW_MASK 0x00000000ffffffffUL
+
+static inline bool
+arch_stack_depot_frame_try_compress(unsigned long frame, u32 *low)
+{
+ if ((frame & ~STACK_DEPOT_X86_64_FRAME_LOW_MASK) !=
+ STACK_DEPOT_X86_64_FRAME_PREFIX)
+ return false;
+
+ *low = (u32)frame;
+ return true;
+}
+
+static inline void
+arch_stack_depot_frame_decompress(u32 low, unsigned long *frame)
+{
+ *frame = STACK_DEPOT_X86_64_FRAME_PREFIX | low;
+}
+
+#else
+#include <asm-generic/stackdepot.h>
+#endif /* CONFIG_X86_64 */
+
+#endif /* _ASM_X86_STACKDEPOT_H */
diff --git a/drivers/gpu/drm/drm_modeset_lock.c b/drivers/gpu/drm/drm_modeset_lock.c
index e14814c30d8c0..f48b02379b711 100644
--- a/drivers/gpu/drm/drm_modeset_lock.c
+++ b/drivers/gpu/drm/drm_modeset_lock.c
@@ -94,16 +94,13 @@ static noinline depot_stack_handle_t __drm_stack_depot_save(void)
static void __drm_stack_depot_print(depot_stack_handle_t stack_depot)
{
struct drm_printer p = drm_dbg_printer(NULL, DRM_UT_KMS, "drm_modeset_lock");
- unsigned long *entries;
- unsigned int nr_entries;
char *buf;
buf = kmalloc(PAGE_SIZE, GFP_NOWAIT | __GFP_NOWARN);
if (!buf)
return;
- nr_entries = stack_depot_fetch(stack_depot, &entries);
- stack_trace_snprint(buf, PAGE_SIZE, entries, nr_entries, 2);
+ stack_depot_snprint(stack_depot, buf, PAGE_SIZE, 2);
drm_printf(&p, "attempting to lock a contended lock without backoff:\n%s", buf);
diff --git a/include/asm-generic/Kbuild b/include/asm-generic/Kbuild
index 2bc00c67dc54b..d8402a6afc703 100644
--- a/include/asm-generic/Kbuild
+++ b/include/asm-generic/Kbuild
@@ -55,6 +55,7 @@ mandatory-y += serial.h
mandatory-y += shmparam.h
mandatory-y += simd.h
mandatory-y += softirq_stack.h
+mandatory-y += stackdepot.h
mandatory-y += switch_to.h
mandatory-y += timex.h
mandatory-y += tlbflush.h
diff --git a/include/asm-generic/stackdepot.h b/include/asm-generic/stackdepot.h
new file mode 100644
index 0000000000000..846975767bdd4
--- /dev/null
+++ b/include/asm-generic/stackdepot.h
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __ASM_GENERIC_STACKDEPOT_H
+#define __ASM_GENERIC_STACKDEPOT_H
+
+#include <linux/types.h>
+
+static inline bool
+arch_stack_depot_frame_try_compress(unsigned long frame, u32 *low)
+{
+ return false;
+}
+
+static inline void
+arch_stack_depot_frame_decompress(u32 low, unsigned long *frame)
+{
+ /* Generic code never compresses frames, so this hook is unreachable. */
+}
+
+#endif /* __ASM_GENERIC_STACKDEPOT_H */
diff --git a/include/linux/stackdepot.h b/include/linux/stackdepot.h
index 2cc21ffcdaf9e..3126d17b9265b 100644
--- a/include/linux/stackdepot.h
+++ b/include/linux/stackdepot.h
@@ -53,7 +53,8 @@ union handle_parts {
struct stack_record {
struct list_head hash_list; /* Links in the hash table */
u32 hash; /* Hash in hash table */
- u32 size; /* Number of stored frames */
+ u16 size; /* Number of stored frames */
+ u16 flags;
union handle_parts handle; /* Constant after initialization */
refcount_t count;
union {
@@ -84,8 +85,9 @@ typedef u32 depot_flags_t;
*/
#define STACK_DEPOT_FLAG_CAN_ALLOC ((depot_flags_t)0x0001)
#define STACK_DEPOT_FLAG_GET ((depot_flags_t)0x0002)
+#define STACK_DEPOT_FLAG_COUNTABLE ((depot_flags_t)0x0004)
-#define STACK_DEPOT_FLAGS_NUM 2
+#define STACK_DEPOT_FLAGS_NUM 3
#define STACK_DEPOT_FLAGS_MASK ((depot_flags_t)((1 << STACK_DEPOT_FLAGS_NUM) - 1))
/*
@@ -144,6 +146,17 @@ static inline int stack_depot_early_init(void) { return 0; }
* Users of this flag must also call stack_depot_put() when keeping the stack
* trace is no longer required to avoid overflowing the refcount.
*
+ * If STACK_DEPOT_FLAG_COUNTABLE is set in @depot_flags, stack depot stores the
+ * stack in hash-backed storage for callers that need direct stack_record count
+ * access. This flag does not imply %STACK_DEPOT_FLAG_CAN_ALLOC and is mutually
+ * exclusive with %STACK_DEPOT_FLAG_GET.
+ *
+ * When trie storage is enabled, persistent non-refcounted saves use trie
+ * storage. Constrained callers first look up an existing stack, then make one
+ * best-effort insertion attempt without allocating. NMI callers stop after the
+ * lookup. Other callers that cannot spin use trylocks and fail if a required
+ * lock is unavailable. Trie failures do not fall back to hash storage.
+ *
* If the provided stack trace comes from the interrupt context, only the part
* up to the interrupt entry is saved.
*
@@ -152,7 +165,7 @@ static inline int stack_depot_early_init(void) { return 0; }
* this is the case for contexts where neither %GFP_ATOMIC nor
* %GFP_NOWAIT can be used (NMI, raw_spin_lock).
*
- * Return: Handle of the stack struct stored in depot, 0 on failure
+ * Return: Handle of the stack trace stored in depot, 0 on failure
*/
depot_stack_handle_t stack_depot_save_flags(unsigned long *entries,
unsigned int nr_entries,
@@ -169,6 +182,10 @@ depot_stack_handle_t stack_depot_save_flags(unsigned long *entries,
* Does not increment the refcount on the saved stack trace; see
* stack_depot_save_flags() for more details.
*
+ * When trie storage is enabled, this can return trie-backed handles. Use
+ * stack_depot_fetch_into(), stack_depot_print(), or stack_depot_snprint() for
+ * backend-independent access to the stack contents.
+ *
* Context: Contexts where allocations via alloc_pages() are allowed;
* see stack_depot_save_flags() for more details.
*
@@ -178,11 +195,12 @@ depot_stack_handle_t stack_depot_save(unsigned long *entries,
unsigned int nr_entries, gfp_t alloc_flags);
/**
- * __stack_depot_get_stack_record - Get a pointer to a stack_record struct
+ * __stack_depot_get_stack_record - Get a hash-backed stack record
*
* @handle: Stack depot handle
*
- * This function is only for internal purposes.
+ * This function is only for internal purposes. @handle must have been saved
+ * with %STACK_DEPOT_FLAG_COUNTABLE.
*
* Return: Returns a pointer to a stack_record struct
*/
@@ -191,14 +209,55 @@ struct stack_record *__stack_depot_get_stack_record(depot_stack_handle_t handle)
/**
* stack_depot_fetch - Fetch a stack trace from stack depot
*
- * @handle: Stack depot handle returned from stack_depot_save()
+ * @handle: Hash-backed stack depot handle
* @entries: Pointer to store the address of the stack trace
*
+ * This helper returns a pointer to stackdepot-owned contiguous storage for
+ * legacy hash-backed handles. Callers that need backend-independent access to
+ * stack contents should use stack_depot_fetch_into(), stack_depot_print(), or
+ * stack_depot_snprint(). Passing a trie-backed handle is invalid and may WARN.
+ *
* Return: Number of frames for the fetched stack
*/
unsigned int stack_depot_fetch(depot_stack_handle_t handle,
unsigned long **entries);
+/**
+ * stack_depot_fetch_into - Fetch a stack trace into caller-owned storage
+ *
+ * @handle: Stack depot handle
+ * @entries: Caller-owned buffer to copy the stack trace into
+ * @max_entries: Number of frames that fit in @entries
+ *
+ * Copies the stored frames into caller-owned @entries. If fewer frames are
+ * stored than @max_entries, only the stored frames are written and their count
+ * is returned. If more frames are stored than @max_entries, the copy is skipped
+ * entirely and 0 is returned.
+ *
+ * Passing a NULL @entries buffer or zero @max_entries for a valid @handle is
+ * invalid. Callers must provide storage for @max_entries frames.
+ *
+ * Callers should size @entries to match the save-side stack depth cap (for
+ * example, %CONFIG_STACKDEPOT_MAX_FRAMES or the local stack_trace_save() limit)
+ * when losing diagnostics on an undersized buffer would be surprising.
+ *
+ * A non-zero invalid @handle, including a post-put handle, may WARN. Its return
+ * value and copied contents are undefined because the record may have been
+ * reused for another stack.
+ *
+ * Callers must ensure @handle remains valid for the duration of this call.
+ * Persistent handles saved without %STACK_DEPOT_FLAG_GET require no extra
+ * reference; handles saved with %STACK_DEPOT_FLAG_GET require a held reference.
+ * Callers must not call stack_depot_put() on persistent handles.
+ * Racing this helper with stack_depot_put() on the same handle is invalid.
+ *
+ * Return: Number of frames copied, 0 if @handle is 0, stack depot is disabled,
+ * or @max_entries is less than the number of stored frames.
+ */
+unsigned int stack_depot_fetch_into(depot_stack_handle_t handle,
+ unsigned long *entries,
+ unsigned int max_entries);
+
/**
* stack_depot_print - Print a stack trace from stack depot
*
@@ -224,10 +283,14 @@ int stack_depot_snprint(depot_stack_handle_t handle, char *buf, size_t size,
*
* @handle: Stack depot handle returned from stack_depot_save()
*
- * The stack trace is evicted from stack depot once all references to it have
- * been dropped (once the number of stack_depot_evict() calls matches the
- * number of stack_depot_save_flags() calls with STACK_DEPOT_FLAG_GET set for
- * this stack trace).
+ * Drop a reference acquired by stack_depot_save_flags() with
+ * %STACK_DEPOT_FLAG_GET. Calling this for a handle saved without
+ * %STACK_DEPOT_FLAG_GET is invalid; persistent handles, including trie-backed
+ * handles, are owned by stack depot for the lifetime of the system.
+ *
+ * The stack trace is evicted once the number of stack_depot_put() calls matches
+ * the number of successful stack_depot_save_flags() calls with
+ * %STACK_DEPOT_FLAG_GET for this stack trace.
*/
void stack_depot_put(depot_stack_handle_t handle);
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index 134b15a44625e..3a78c67b6b364 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -2785,6 +2785,23 @@ config RESOURCE_KUNIT_TEST
If unsure, say N.
+config STACKDEPOT_KUNIT_TEST
+ bool "KUnit test for stack depot" if !KUNIT_ALL_TESTS
+ depends on KUNIT=y && STACKDEPOT
+ depends on STACKDEPOT_MAX_FRAMES >= 3
+ default KUNIT_ALL_TESTS
+ help
+ Enable this option to test stack depot API behavior at boot.
+ This test is built in because it exercises internal, non-exported
+ stack depot helpers, so KUNIT must also be built in.
+
+ KUnit tests run during boot and output the results to the debug log
+ in TAP format (https://testanything.org/). Only useful for kernel
+ developers running the KUnit test harness, and not intended for
+ inclusion into a production build.
+
+ If unsure, say N.
+
config SYSCTL_KUNIT_TEST
tristate "KUnit test for sysctl" if !KUNIT_ALL_TESTS
depends on KUNIT
diff --git a/lib/stackdepot.c b/lib/stackdepot.c
index dd2717ff94bff..33e475d941314 100644
--- a/lib/stackdepot.c
+++ b/lib/stackdepot.c
@@ -2,9 +2,11 @@
/*
* Stack depot - a stack trace storage that avoids duplication.
*
- * Internally, stack depot maintains a hash table of unique stacktraces. The
- * stack traces themselves are stored contiguously one after another in a set
- * of separate page allocations.
+ * Internally, stack depot has two storage backends. Refcounted entries and
+ * callers that request STACK_DEPOT_FLAG_COUNTABLE use the legacy hash table with
+ * contiguous stack records in stack pools. Persistent non-refcounted entries
+ * can use trie storage when enabled; trie nodes share common frame prefixes and
+ * are published through RCU children containers.
*
* Author: Alexander Potapenko <glider@google.com>
* Copyright (C) 2016 Google, Inc.
@@ -14,13 +16,19 @@
#define pr_fmt(fmt) "stackdepot: " fmt
+#include <linux/bitmap.h>
+#include <linux/build_bug.h>
#include <linux/debugfs.h>
+#include <linux/errno.h>
#include <linux/gfp.h>
#include <linux/jhash.h>
+#include <linux/jump_label.h>
#include <linux/kernel.h>
+#include <linux/log2.h>
#include <linux/kmsan.h>
#include <linux/list.h>
#include <linux/mm.h>
+#include <linux/moduleparam.h>
#include <linux/mutex.h>
#include <linux/poison.h>
#include <linux/printk.h>
@@ -36,9 +44,12 @@
#include <linux/memblock.h>
#include <linux/kasan-enabled.h>
+#include <asm/stackdepot.h>
+
/*
* The pool_index is offset by 1 so the first record does not have a 0 handle.
*/
+/* Parsed before mm_core_init(); trie handle decoding assumes this is then fixed. */
static unsigned int stack_max_pools __read_mostly =
MIN((1LL << DEPOT_POOL_INDEX_BITS) - 1, 8192);
@@ -54,6 +65,9 @@ static bool __stack_depot_early_init_passed __initdata;
/* Initial seed for jhash2. */
#define STACK_HASH_SEED 0x9747b28c
+/* Bound 64-bit print scratch to 128 bytes while amortizing trie walks. */
+#define STACK_DEPOT_PRINT_CHUNK_FRAMES 16
+
/* Hash table of stored stack records. */
static struct list_head *stack_table;
/* Fixed order of the number of table buckets. Used when KASAN is enabled. */
@@ -63,18 +77,18 @@ static unsigned int stack_hash_mask;
/* The lock must be held when performing pool or freelist modifications. */
static DEFINE_RAW_SPINLOCK(pool_lock);
-/* Array of memory regions that store stack records. */
+/* Array of memory regions used by both stack depot backends. */
static void **stack_pools __pt_guarded_by(&pool_lock);
/* Newly allocated pool that is not yet added to stack_pools. */
static void *new_pool;
/* Number of pools in stack_pools. */
static int pools_num;
-/* Offset to the unused space in the currently used pool. */
+/* Offset to unused hash storage in the current pool. */
static size_t pool_offset __guarded_by(&pool_lock) = DEPOT_POOL_SIZE;
/* Freelist of stack records within stack_pools. */
static __guarded_by(&pool_lock) LIST_HEAD(free_stacks);
-/* Statistics counters for debugfs. */
+/* Hash-backend statistics counters for debugfs. */
enum depot_counter_id {
DEPOT_COUNTER_REFD_ALLOCS,
DEPOT_COUNTER_REFD_FREES,
@@ -90,11 +104,695 @@ static const char *const counter_names[] = {
[DEPOT_COUNTER_REFD_FREES] = "refcounted_frees",
[DEPOT_COUNTER_REFD_INUSE] = "refcounted_in_use",
[DEPOT_COUNTER_FREELIST_SIZE] = "freelist_size",
- [DEPOT_COUNTER_PERSIST_COUNT] = "persistent_count",
- [DEPOT_COUNTER_PERSIST_BYTES] = "persistent_bytes",
+ [DEPOT_COUNTER_PERSIST_COUNT] = "hash_persistent_count",
+ [DEPOT_COUNTER_PERSIST_BYTES] = "hash_persistent_bytes",
};
static_assert(ARRAY_SIZE(counter_names) == DEPOT_COUNTER_COUNT);
+enum stack_depot_frame_mode {
+ STACK_DEPOT_FRAME_RAW,
+ STACK_DEPOT_FRAME_COMPRESSED,
+};
+
+/*
+ * A trie node stores one run of frames that all use the same payload format.
+ * Architectures may compress some frames to 32-bit payloads; mixed raw and
+ * compressed input is split across multiple trie nodes so each node has one
+ * decoding mode.
+ */
+struct stack_depot_frame_run {
+ u16 nr_entries;
+ u8 mode;
+};
+
+static_assert(CONFIG_STACKDEPOT_MAX_FRAMES <= U16_MAX);
+
+struct stack_depot_trie_children;
+
+struct stack_depot_trie_node {
+ /* Parent links let fetch rebuild a full stack from a node to the root. */
+ const struct stack_depot_trie_node __rcu *parent;
+ /* Children are RCU-published containers. */
+ const struct stack_depot_trie_children __rcu *children;
+ /* Non-zero when a stored stack ends at this node. */
+ u32 stack_id;
+ struct stack_depot_frame_run run;
+ unsigned char data[];
+};
+
+/*
+ * Child nodes are sorted by first frame and searched by insertion position.
+ * Existing child pointers are immutable. Writers may publish into unused tail
+ * capacity; other updates publish a replacement container.
+ */
+struct stack_depot_trie_children {
+ unsigned int nr_children;
+ unsigned int capacity;
+ const struct stack_depot_trie_node __rcu *nodes[];
+};
+
+/* Retired children carry an optional node through their RCU grace period. */
+struct stack_depot_trie_retired_children {
+ struct list_head list;
+ unsigned long rcu_state;
+ const struct stack_depot_trie_node *pending_node;
+ unsigned char data[];
+};
+
+static_assert(IS_ALIGNED(offsetof(struct stack_depot_trie_retired_children, data),
+ 1UL << DEPOT_STACK_ALIGN));
+
+#define STACK_DEPOT_TRIE_SLOT_SIZE BIT(DEPOT_STACK_ALIGN)
+#define STACK_DEPOT_TRIE_POOL_SLOTS \
+ (DEPOT_POOL_SIZE / STACK_DEPOT_TRIE_SLOT_SIZE)
+
+static_assert(STACK_DEPOT_TRIE_POOL_SLOTS - 1 <= U16_MAX);
+
+struct stack_depot_trie_pool {
+ struct list_head list;
+ unsigned int free_slots;
+ /* Conservative upper bound on the largest free run. */
+ u16 free_run_upper_bound;
+ /* First physical slot considered by the next reservation. */
+ u16 next_slot;
+ DECLARE_BITMAP(used, STACK_DEPOT_TRIE_POOL_SLOTS);
+};
+
+#define STACK_DEPOT_TRIE_POOL_FIRST_SLOT \
+ DIV_ROUND_UP(sizeof(struct stack_depot_trie_pool), \
+ STACK_DEPOT_TRIE_SLOT_SIZE)
+#define STACK_DEPOT_TRIE_POOL_USABLE_SIZE \
+ ((STACK_DEPOT_TRIE_POOL_SLOTS - STACK_DEPOT_TRIE_POOL_FIRST_SLOT) * \
+ STACK_DEPOT_TRIE_SLOT_SIZE)
+
+static_assert(STACK_DEPOT_TRIE_POOL_FIRST_SLOT < STACK_DEPOT_TRIE_POOL_SLOTS);
+
+static DEFINE_STATIC_KEY_FALSE(stack_depot_trie_enabled);
+static const struct stack_depot_trie_children __rcu *stack_depot_trie_root;
+static DEFINE_RAW_SPINLOCK(stack_depot_trie_writer_lock);
+static bool stack_depot_trie_requested;
+
+module_param_named(trie_enabled, stack_depot_trie_requested, bool, 0);
+MODULE_PARM_DESC(trie_enabled, "Enable stack depot trie storage at boot");
+
+#define DEPOT_POOL_INDEX_MASK ((1U << DEPOT_POOL_INDEX_BITS) - 1)
+#define DEPOT_OFFSET_MASK ((1U << DEPOT_OFFSET_BITS) - 1)
+
+/* Retired fixed-size slots remain reserved until their RCU grace period ends. */
+static LIST_HEAD(stack_depot_trie_pools);
+static LIST_HEAD(pending_trie_children);
+
+/*
+ * stack_max_pools is the split point between hash and trie handle encodings.
+ * A handle with pool_index_plus_1 in 1..stack_max_pools names a hash-backed
+ * stack pool. Larger pool-index values cannot refer to hash pools, so trie
+ * storage uses that handle space to encode a dense stack ID. The side table
+ * maps each stack ID to its trie node.
+ */
+static inline u32 trie_max_stack_id(void)
+{
+ return (DEPOT_POOL_INDEX_MASK - stack_max_pools) <<
+ DEPOT_OFFSET_BITS;
+}
+
+static depot_stack_handle_t trie_handle(u32 stack_id)
+{
+ union handle_parts parts = {};
+ u64 pool_index_plus_1;
+ u32 pool_delta;
+ u32 index;
+
+ index = stack_id - 1;
+ pool_delta = index >> DEPOT_OFFSET_BITS;
+ pool_index_plus_1 = (u64)stack_max_pools + 1 + pool_delta;
+
+ parts.pool_index_plus_1 = pool_index_plus_1;
+ parts.offset = index & DEPOT_OFFSET_MASK;
+ return parts.handle;
+}
+
+static inline bool stack_depot_handle_is_trie(depot_stack_handle_t handle)
+{
+ union handle_parts parts = { .handle = handle };
+
+ return parts.pool_index_plus_1 > stack_max_pools;
+}
+
+static u32 trie_stack_id(depot_stack_handle_t handle)
+{
+ union handle_parts parts = { .handle = handle };
+ u32 pool_delta;
+
+ pool_delta = parts.pool_index_plus_1 - stack_max_pools - 1;
+ return (pool_delta << DEPOT_OFFSET_BITS) + parts.offset + 1;
+}
+
+/*
+ * Trie handles encode a dense stack ID. The side table maps that ID to a node
+ * pointer for lockless fetch and print paths, which can run from diagnostic
+ * contexts where taking a lock would be unsafe. Initialization installs the
+ * root; early initialization also installs the first directory and chunk.
+ * Additional directories and chunks are published lazily as stack IDs grow.
+ */
+#define STACK_DEPOT_TRIE_SIDE_TABLE_CHUNK_SIZE \
+ (PAGE_SIZE / sizeof(struct stack_depot_trie_node *))
+#define STACK_DEPOT_TRIE_SIDE_TABLE_DIR_SIZE \
+ (PAGE_SIZE / sizeof(struct stack_depot_trie_node **))
+
+struct stack_depot_trie_side_dir {
+ /* Both the chunk pointer and each node pointer in it are RCU-published. */
+ const struct stack_depot_trie_node __rcu * __rcu *
+ chunks[STACK_DEPOT_TRIE_SIDE_TABLE_DIR_SIZE];
+};
+
+struct stack_depot_trie_side_root {
+ unsigned int dir_capacity;
+ struct stack_depot_trie_side_dir __rcu *dirs[];
+};
+
+struct stack_depot_trie_side_prealloc {
+ /* Preallocated side-table directory page for sparse growth. */
+ struct stack_depot_trie_side_dir *dir;
+ /* Preallocated side-table pointer chunk for sparse growth. */
+ const struct stack_depot_trie_node __rcu **chunk;
+};
+
+static struct stack_depot_trie_side_root *trie_side_table_root;
+static DEFINE_RAW_SPINLOCK(trie_side_table_cache_lock);
+/* Zeroed unpublished pages; get/put transfer ownership under the cache lock. */
+static struct stack_depot_trie_side_prealloc trie_side_table_cache;
+static u32 trie_side_table_last_stack_id;
+
+/* Lock order: writer_lock -> pool_lock -> side-table cache lock. */
+
+static inline size_t stack_depot_frame_run_entry_bytes(enum stack_depot_frame_mode mode)
+{
+ if (mode == STACK_DEPOT_FRAME_COMPRESSED)
+ return sizeof(u32);
+ return sizeof(unsigned long);
+}
+
+static inline size_t stack_depot_frame_run_bytes(const struct stack_depot_frame_run *run)
+{
+ return run->nr_entries * stack_depot_frame_run_entry_bytes(run->mode);
+}
+
+static inline size_t trie_node_bytes(const struct stack_depot_frame_run *run)
+{
+ return ALIGN(offsetof(struct stack_depot_trie_node, data) +
+ stack_depot_frame_run_bytes(run), sizeof(unsigned long));
+}
+
+static size_t trie_children_alloc_size(unsigned int capacity)
+{
+ size_t size;
+
+ size = struct_size_t(struct stack_depot_trie_children, nodes,
+ capacity);
+ return offsetof(struct stack_depot_trie_retired_children, data) +
+ ALIGN(size, sizeof(unsigned long));
+}
+
+static inline unsigned int trie_side_table_root_index(u32 id)
+{
+ return ((id - 1) / STACK_DEPOT_TRIE_SIDE_TABLE_CHUNK_SIZE) /
+ STACK_DEPOT_TRIE_SIDE_TABLE_DIR_SIZE;
+}
+
+static inline unsigned int trie_side_table_dir_index(u32 id)
+{
+ return ((id - 1) / STACK_DEPOT_TRIE_SIDE_TABLE_CHUNK_SIZE) %
+ STACK_DEPOT_TRIE_SIDE_TABLE_DIR_SIZE;
+}
+
+static inline unsigned int trie_side_table_slot_index(u32 id)
+{
+ return (id - 1) % STACK_DEPOT_TRIE_SIDE_TABLE_CHUNK_SIZE;
+}
+
+static struct stack_depot_trie_side_dir *trie_side_table_load_dir(unsigned int root)
+{
+ struct stack_depot_trie_side_root *root_vec;
+
+ root_vec = trie_side_table_root;
+ if (!root_vec || root >= root_vec->dir_capacity)
+ return NULL;
+ /* Pairs with side-table directory rcu_assign_pointer(). */
+ return rcu_dereference_check(root_vec->dirs[root],
+ lockdep_is_held(&stack_depot_trie_writer_lock) ||
+ rcu_read_lock_sched_held());
+}
+
+static inline const struct stack_depot_trie_node __rcu **
+trie_side_table_dir_load_chunk(struct stack_depot_trie_side_dir *dir,
+ unsigned int idx)
+{
+ /* Pairs with the chunk rcu_assign_pointer() in stack ID preparation. */
+ return rcu_dereference_check(dir->chunks[idx],
+ lockdep_is_held(&stack_depot_trie_writer_lock) ||
+ rcu_read_lock_sched_held());
+}
+
+/* Published capacity remains useful if insertion fails and needs no rollback. */
+static bool
+trie_side_table_try_take_cache(struct stack_depot_trie_side_prealloc *prealloc,
+ bool need_dir)
+{
+ bool taken = false;
+
+ lockdep_assert_held(&stack_depot_trie_writer_lock);
+ lockdep_assert_held(&pool_lock);
+
+ if (!raw_spin_trylock(&trie_side_table_cache_lock))
+ return false;
+ if ((!prealloc->chunk && !trie_side_table_cache.chunk) ||
+ (need_dir && !prealloc->dir && !trie_side_table_cache.dir))
+ goto out_unlock;
+
+ if (need_dir && !prealloc->dir) {
+ prealloc->dir = trie_side_table_cache.dir;
+ trie_side_table_cache.dir = NULL;
+ }
+ if (!prealloc->chunk) {
+ prealloc->chunk = trie_side_table_cache.chunk;
+ trie_side_table_cache.chunk = NULL;
+ }
+ taken = true;
+
+out_unlock:
+ raw_spin_unlock(&trie_side_table_cache_lock);
+ return taken;
+}
+
+static u32
+trie_side_table_prepare_stack_slot(struct stack_depot_trie_side_prealloc *prealloc)
+{
+ const struct stack_depot_trie_node __rcu **chunk;
+ struct stack_depot_trie_side_dir *dir;
+ struct stack_depot_trie_side_root *root_vec;
+ unsigned int root;
+ unsigned int idx;
+ u32 id;
+
+ lockdep_assert_held(&stack_depot_trie_writer_lock);
+ lockdep_assert_held(&pool_lock);
+
+ id = trie_side_table_last_stack_id + 1;
+ if (id > trie_max_stack_id())
+ return 0;
+
+ root_vec = trie_side_table_root;
+ root = trie_side_table_root_index(id);
+ dir = trie_side_table_load_dir(root);
+ if (!dir) {
+ if ((!prealloc->dir || !prealloc->chunk) &&
+ !trie_side_table_try_take_cache(prealloc, true))
+ return 0;
+ dir = prealloc->dir;
+ prealloc->dir = NULL;
+ /* Publish the zeroed directory before readers can load it locklessly. */
+ rcu_assign_pointer(root_vec->dirs[root], dir);
+ }
+
+ idx = trie_side_table_dir_index(id);
+ chunk = trie_side_table_dir_load_chunk(dir, idx);
+ if (!chunk) {
+ if (!prealloc->chunk &&
+ !trie_side_table_try_take_cache(prealloc, false))
+ return 0;
+ chunk = prealloc->chunk;
+ prealloc->chunk = NULL;
+ rcu_assign_pointer(dir->chunks[idx], chunk);
+ }
+
+ return id;
+}
+
+static inline unsigned int trie_side_table_root_size_for_max_id(u32 max_stack_id)
+{
+ unsigned int top_size;
+
+ top_size = DIV_ROUND_UP(max_stack_id, STACK_DEPOT_TRIE_SIDE_TABLE_CHUNK_SIZE);
+ return DIV_ROUND_UP(top_size, STACK_DEPOT_TRIE_SIDE_TABLE_DIR_SIZE);
+}
+
+static int __init stack_depot_trie_init_memblock(void)
+{
+ struct stack_depot_trie_side_root *root_vec;
+ struct stack_depot_trie_side_dir *first_dir;
+ const struct stack_depot_trie_node __rcu **first_chunk;
+ size_t root_bytes;
+ u32 max_stack_id;
+ unsigned int root_size;
+
+ max_stack_id = trie_max_stack_id();
+ if (!max_stack_id)
+ return -EINVAL;
+ root_size = trie_side_table_root_size_for_max_id(max_stack_id);
+ root_bytes = struct_size_t(struct stack_depot_trie_side_root, dirs, root_size);
+
+ root_vec = memblock_alloc(root_bytes, __alignof__(*root_vec));
+ if (!root_vec)
+ return -ENOMEM;
+ first_dir = memblock_alloc(PAGE_SIZE, PAGE_SIZE);
+ if (!first_dir) {
+ memblock_free(root_vec, root_bytes);
+ return -ENOMEM;
+ }
+ first_chunk = memblock_alloc(PAGE_SIZE, PAGE_SIZE);
+ if (!first_chunk) {
+ memblock_free(first_dir, PAGE_SIZE);
+ memblock_free(root_vec, root_bytes);
+ return -ENOMEM;
+ }
+
+ root_vec->dir_capacity = root_size;
+ RCU_INIT_POINTER(root_vec->dirs[0], first_dir);
+ RCU_INIT_POINTER(first_dir->chunks[0], first_chunk);
+ trie_side_table_root = root_vec;
+ static_branch_enable(&stack_depot_trie_enabled);
+ return 0;
+}
+
+static int stack_depot_trie_init(void)
+{
+ struct stack_depot_trie_side_root *root_vec;
+ unsigned int root_size;
+ size_t root_bytes;
+ u32 max_stack_id;
+
+ max_stack_id = trie_max_stack_id();
+ if (!max_stack_id)
+ return -EINVAL;
+
+ root_size = trie_side_table_root_size_for_max_id(max_stack_id);
+ root_bytes = struct_size_t(struct stack_depot_trie_side_root, dirs, root_size);
+ root_vec = kvzalloc(root_bytes, GFP_KERNEL);
+ if (!root_vec)
+ return -ENOMEM;
+
+ root_vec->dir_capacity = root_size;
+ trie_side_table_root = root_vec;
+ static_branch_enable(&stack_depot_trie_enabled);
+ return 0;
+}
+
+static int trie_side_table_get_prealloc(gfp_t gfp_flags,
+ struct stack_depot_trie_side_prealloc *prealloc)
+{
+ unsigned long flags;
+
+ gfp_flags = gfp_nested_mask(gfp_flags);
+ raw_spin_lock_irqsave(&trie_side_table_cache_lock, flags);
+ prealloc->dir = trie_side_table_cache.dir;
+ prealloc->chunk = trie_side_table_cache.chunk;
+ trie_side_table_cache.dir = NULL;
+ trie_side_table_cache.chunk = NULL;
+ raw_spin_unlock_irqrestore(&trie_side_table_cache_lock, flags);
+
+ if (!prealloc->dir) {
+ prealloc->dir = (void *)get_zeroed_page(gfp_flags);
+ if (!prealloc->dir)
+ return -ENOMEM;
+ }
+ if (!prealloc->chunk) {
+ prealloc->chunk = (void *)get_zeroed_page(gfp_flags);
+ if (!prealloc->chunk)
+ return -ENOMEM;
+ }
+
+ return 0;
+}
+
+static void trie_side_table_put_prealloc(struct stack_depot_trie_side_prealloc *prealloc)
+{
+ unsigned long flags;
+
+ raw_spin_lock_irqsave(&trie_side_table_cache_lock, flags);
+ if (!trie_side_table_cache.dir) {
+ trie_side_table_cache.dir = prealloc->dir;
+ prealloc->dir = NULL;
+ }
+ if (!trie_side_table_cache.chunk) {
+ trie_side_table_cache.chunk = prealloc->chunk;
+ prealloc->chunk = NULL;
+ }
+ raw_spin_unlock_irqrestore(&trie_side_table_cache_lock, flags);
+
+ if (prealloc->dir)
+ free_page((unsigned long)prealloc->dir);
+ if (prealloc->chunk)
+ free_page((unsigned long)prealloc->chunk);
+}
+
+static const struct stack_depot_trie_node *trie_side_table_lookup(u32 id)
+{
+ const struct stack_depot_trie_node __rcu **chunk;
+ struct stack_depot_trie_side_dir *dir;
+ unsigned int root;
+
+ root = trie_side_table_root_index(id);
+ dir = trie_side_table_load_dir(root);
+ if (!dir)
+ return NULL;
+ chunk = trie_side_table_dir_load_chunk(dir, trie_side_table_dir_index(id));
+ if (!chunk)
+ return NULL;
+
+ /* Pairs with side-table node publication. */
+ return rcu_dereference_check(chunk[trie_side_table_slot_index(id)],
+ lockdep_is_held(&stack_depot_trie_writer_lock) ||
+ rcu_read_lock_sched_held());
+}
+
+static inline struct stack_depot_trie_retired_children *
+trie_retired_children(const void *ptr)
+{
+ return container_of(ptr, struct stack_depot_trie_retired_children, data);
+}
+
+static bool depot_init_pool(void **prealloc);
+
+static unsigned int trie_pool_reserve_slots(struct stack_depot_trie_pool *pool,
+ unsigned int nr_slots)
+{
+ unsigned int start = pool->next_slot;
+ unsigned int run = 0;
+ unsigned int longest_run = 0;
+ unsigned int i;
+ unsigned int slot;
+
+scan:
+ run = 0;
+ longest_run = 0;
+ for (slot = start; slot < STACK_DEPOT_TRIE_POOL_SLOTS; slot++) {
+ if (pool->used[slot / BITS_PER_LONG] &
+ BIT(slot % BITS_PER_LONG)) {
+ run = 0;
+ continue;
+ }
+ run++;
+ longest_run = max(longest_run, run);
+ if (run != nr_slots)
+ continue;
+
+ for (i = slot + 1 - nr_slots; i <= slot; i++)
+ pool->used[i / BITS_PER_LONG] |= BIT(i % BITS_PER_LONG);
+ pool->free_slots -= nr_slots;
+ if (slot + 1 == STACK_DEPOT_TRIE_POOL_SLOTS)
+ pool->next_slot = STACK_DEPOT_TRIE_POOL_FIRST_SLOT;
+ else
+ pool->next_slot = slot + 1;
+ return slot + 1 - nr_slots;
+ }
+
+ if (start != STACK_DEPOT_TRIE_POOL_FIRST_SLOT) {
+ /* Keep holes and runs crossing the cursor visible. */
+ start = STACK_DEPOT_TRIE_POOL_FIRST_SLOT;
+ goto scan;
+ }
+
+ pool->free_run_upper_bound = longest_run;
+ return STACK_DEPOT_TRIE_POOL_SLOTS;
+}
+
+/* Allocate at least @size bytes from one contiguous trie-pool slot run. */
+static void *trie_pool_alloc(size_t size, void **prealloc)
+{
+ struct stack_depot_trie_pool *pool;
+ unsigned int nr_slots;
+ unsigned int slot;
+
+ lockdep_assert_held(&pool_lock);
+
+ if (size > STACK_DEPOT_TRIE_POOL_USABLE_SIZE)
+ return NULL;
+ nr_slots = DIV_ROUND_UP(size, STACK_DEPOT_TRIE_SLOT_SIZE);
+ list_for_each_entry_reverse(pool, &stack_depot_trie_pools, list) {
+ if (pool->free_slots < nr_slots ||
+ pool->free_run_upper_bound < nr_slots)
+ continue;
+ slot = trie_pool_reserve_slots(pool, nr_slots);
+ if (slot != STACK_DEPOT_TRIE_POOL_SLOTS)
+ return (char *)pool + slot * STACK_DEPOT_TRIE_SLOT_SIZE;
+ }
+
+ if (!depot_init_pool(prealloc))
+ return NULL;
+ pool = stack_pools[pools_num - 1];
+ /* Keep hash records out of this bitmap-owned pool. */
+ pool_offset = DEPOT_POOL_SIZE;
+ memset(pool, 0, sizeof(*pool));
+ pool->free_slots = STACK_DEPOT_TRIE_POOL_SLOTS -
+ STACK_DEPOT_TRIE_POOL_FIRST_SLOT;
+ pool->free_run_upper_bound = pool->free_slots;
+ pool->next_slot = STACK_DEPOT_TRIE_POOL_FIRST_SLOT;
+ list_add_tail(&pool->list, &stack_depot_trie_pools);
+
+ slot = trie_pool_reserve_slots(pool, nr_slots);
+ return (char *)pool + slot * STACK_DEPOT_TRIE_SLOT_SIZE;
+}
+
+/* Release the slots for the byte count originally passed to allocation. */
+static void trie_pool_release(const void *ptr, size_t size)
+{
+ struct stack_depot_trie_pool *pool;
+ unsigned long pfn;
+ unsigned int nr_slots;
+ unsigned int slot;
+ unsigned int i;
+
+ lockdep_assert_held(&pool_lock);
+
+ pfn = page_to_pfn(virt_to_page(ptr));
+ pfn &= ~(BIT(DEPOT_POOL_ORDER) - 1);
+ pool = page_address(pfn_to_page(pfn));
+ slot = ((unsigned long)ptr - (unsigned long)pool) >> DEPOT_STACK_ALIGN;
+ nr_slots = DIV_ROUND_UP(size, STACK_DEPOT_TRIE_SLOT_SIZE);
+ for (i = slot; i < slot + nr_slots; i++)
+ pool->used[i / BITS_PER_LONG] &= ~BIT(i % BITS_PER_LONG);
+ pool->free_slots += nr_slots;
+ /* A release can join at most two runs bounded by the old value. */
+ pool->free_run_upper_bound = min(pool->free_slots,
+ 2 * pool->free_run_upper_bound + nr_slots);
+}
+
+static struct stack_depot_trie_children *
+trie_pool_alloc_children(unsigned int capacity, void **prealloc)
+{
+ struct stack_depot_trie_retired_children *retired;
+ struct stack_depot_trie_children *children;
+
+ /* Capacity counts child-pointer entries; allocation includes RCU metadata. */
+ retired = trie_pool_alloc(trie_children_alloc_size(capacity), prealloc);
+ if (!retired)
+ return NULL;
+
+ children = (void *)retired->data;
+ children->nr_children = 0;
+ children->capacity = capacity;
+ return children;
+}
+
+static void
+trie_pool_release_children(const struct stack_depot_trie_children *children)
+{
+ /* Capacity is immutable and therefore recovers the allocation byte size. */
+ trie_pool_release(trie_retired_children(children),
+ trie_children_alloc_size(children->capacity));
+}
+
+/*
+ * Return RCU-ready objects before allocating. Pending children are FIFO, so
+ * stop at the first incomplete grace period. A replaced node shares the same
+ * retirement cookie and is released with its former children container.
+ */
+static void trie_drain_pending_children(void)
+{
+ struct stack_depot_trie_retired_children *retired;
+ struct stack_depot_trie_retired_children *tmp;
+ struct stack_depot_trie_children *children;
+
+ lockdep_assert_held(&pool_lock);
+
+ list_for_each_entry_safe(retired, tmp, &pending_trie_children, list) {
+ if (!poll_state_synchronize_rcu(retired->rcu_state))
+ break;
+ children = (void *)retired->data;
+ list_del(&retired->list);
+ if (retired->pending_node)
+ trie_pool_release(retired->pending_node,
+ trie_node_bytes(&retired->pending_node->run));
+ trie_pool_release_children(children);
+ }
+}
+
+static void trie_retire_children(const struct stack_depot_trie_children *children)
+{
+ struct stack_depot_trie_retired_children *retired;
+
+ lockdep_assert_held(&pool_lock);
+
+ retired = trie_retired_children(children);
+ retired->pending_node = NULL;
+ retired->rcu_state = get_state_synchronize_rcu();
+ list_add_tail(&retired->list, &pending_trie_children);
+}
+
+static void
+trie_retire_children_with_node(const struct stack_depot_trie_children *children,
+ const struct stack_depot_trie_node *node)
+{
+ struct stack_depot_trie_retired_children *retired;
+
+ lockdep_assert_held(&stack_depot_trie_writer_lock);
+ lockdep_assert_held(&pool_lock);
+ trie_retire_children(children);
+ retired = trie_retired_children(children);
+ retired->pending_node = node;
+}
+
+static const struct stack_depot_trie_node *
+stack_depot_trie_lookup(const unsigned long *entries, unsigned int nr_entries);
+
+static depot_stack_handle_t
+trie_find_handle(const unsigned long *entries, unsigned int nr_entries)
+{
+ depot_stack_handle_t handle = 0;
+ const struct stack_depot_trie_node *node;
+
+ rcu_read_lock_sched_notrace();
+ node = stack_depot_trie_lookup(entries, nr_entries);
+ if (node)
+ handle = trie_handle(node->stack_id);
+ rcu_read_unlock_sched_notrace();
+
+ return handle;
+}
+
+/*
+ * Publish only after the node and its path are fully initialized and all
+ * fallible allocation is complete. Publication commits the path, so it cannot
+ * then be rolled back. Side-table mappings must precede trie topology
+ * publication that makes new or remapped nodes reachable from lookup.
+ * Published storage remains valid until RCU retirement; only descendant parent
+ * links may change meanwhile.
+ */
+static void trie_side_table_publish(const struct stack_depot_trie_node *node)
+{
+ const struct stack_depot_trie_node __rcu **chunk;
+ struct stack_depot_trie_side_dir *dir;
+ u32 stack_id = node->stack_id;
+
+ lockdep_assert_held(&stack_depot_trie_writer_lock);
+
+ dir = trie_side_table_load_dir(trie_side_table_root_index(stack_id));
+ chunk = trie_side_table_dir_load_chunk(dir,
+ trie_side_table_dir_index(stack_id));
+ /* Pairs with trie_side_table_lookup(). */
+ rcu_assign_pointer(chunk[trie_side_table_slot_index(stack_id)], node);
+}
+
static int __init disable_stack_depot(char *str)
{
return kstrtobool(str, &stack_depot_disabled);
@@ -146,7 +844,7 @@ static void init_stack_table(unsigned long entries)
INIT_LIST_HEAD(&stack_table[i]);
}
-/* Allocates a hash table via memblock. Can only be used during early boot. */
+/* Initializes hash and optional trie storage during early boot. */
int __init stack_depot_early_init(void)
{
unsigned long entries = 0;
@@ -220,11 +918,15 @@ int __init stack_depot_early_init(void)
stack_depot_disabled = true;
return -ENOMEM;
}
+ if (stack_depot_trie_requested && stack_depot_trie_init_memblock()) {
+ pr_warn("trie storage initialization failed, disabling trie storage\n");
+ stack_depot_trie_requested = false;
+ }
return 0;
}
-/* Allocates a hash table via kvcalloc. Can be used after boot. */
+/* Initializes hash and optional trie storage after boot. */
int stack_depot_init(void)
{
static DEFINE_MUTEX(stack_depot_init_mutex);
@@ -278,6 +980,15 @@ int stack_depot_init(void)
kvfree(stack_table);
stack_depot_disabled = true;
ret = -ENOMEM;
+ goto out_unlock;
+ }
+ if (stack_depot_trie_requested) {
+ ret = stack_depot_trie_init();
+ if (ret) {
+ pr_warn("trie storage initialization failed, disabling trie storage\n");
+ stack_depot_trie_requested = false;
+ ret = 0;
+ }
}
out_unlock:
@@ -323,7 +1034,7 @@ static bool depot_init_pool(void **prealloc)
* NULL; do not reset to NULL if we have reached the maximum number of
* pools.
*/
- if (pools_num < stack_max_pools)
+ if (pools_num + 1 < stack_max_pools)
WRITE_ONCE(new_pool, NULL);
else
WRITE_ONCE(new_pool, STACK_DEPOT_POISON);
@@ -467,6 +1178,7 @@ depot_alloc_stack(unsigned long *entries, unsigned int nr_entries, u32 hash, dep
/* Save the stack trace. */
stack->hash = hash;
stack->size = nr_entries;
+ stack->flags = flags & STACK_DEPOT_FLAG_COUNTABLE;
/* stack->handle is already filled in by depot_pop_free_pool(). */
memcpy(stack->entries, entries, flex_array_size(stack, entries, nr_entries));
@@ -609,6 +1321,9 @@ static inline struct stack_record *find_stack(struct list_head *bucket,
list_for_each_entry_rcu(stack, bucket, hash_list) {
if (stack->hash != hash || stack->size != size)
continue;
+ /* Page owner countable records have a distinct count lifetime. */
+ if ((stack->flags ^ flags) & STACK_DEPOT_FLAG_COUNTABLE)
+ continue;
/*
* This may race with depot_free_stack() accessing the freelist
@@ -638,6 +1353,101 @@ static inline struct stack_record *find_stack(struct list_head *bucket,
return ret;
}
+static u32
+stack_depot_trie_insert(const unsigned long *entries,
+ unsigned int nr_entries, void **pool_prealloc,
+ struct stack_depot_trie_side_prealloc *side_prealloc);
+
+static depot_stack_handle_t
+stack_depot_trie_save(unsigned long *entries, unsigned int nr_entries,
+ gfp_t alloc_flags)
+{
+ unsigned int attempt;
+
+ /* Allow one stale pool hint before the two pools a largest insert needs. */
+ for (attempt = 0; attempt < 3; attempt++) {
+ struct stack_depot_trie_side_prealloc side_prealloc = {};
+ void *pool_prealloc = NULL;
+ depot_stack_handle_t handle;
+ unsigned long flags;
+ struct page *page;
+ u32 stack_id = 0;
+
+ handle = trie_find_handle(entries, nr_entries);
+ if (handle)
+ return handle;
+
+ if (trie_side_table_get_prealloc(alloc_flags, &side_prealloc)) {
+ trie_side_table_put_prealloc(&side_prealloc);
+ return 0;
+ }
+
+ /* The hint may race; a missing page is recovered by the retry. */
+ if (!READ_ONCE(new_pool)) {
+ page = alloc_pages(gfp_nested_mask(alloc_flags),
+ DEPOT_POOL_ORDER);
+ if (page)
+ pool_prealloc = page_address(page);
+ }
+
+ raw_spin_lock_irqsave(&stack_depot_trie_writer_lock, flags);
+ raw_spin_lock(&pool_lock);
+ printk_deferred_enter();
+ trie_drain_pending_children();
+ stack_id = stack_depot_trie_insert(entries, nr_entries,
+ &pool_prealloc, &side_prealloc);
+ if (pool_prealloc)
+ depot_keep_new_pool(&pool_prealloc);
+ printk_deferred_exit();
+ raw_spin_unlock(&pool_lock);
+ raw_spin_unlock_irqrestore(&stack_depot_trie_writer_lock, flags);
+
+ if (pool_prealloc)
+ free_pages((unsigned long)pool_prealloc, DEPOT_POOL_ORDER);
+ trie_side_table_put_prealloc(&side_prealloc);
+ if (stack_id)
+ return trie_handle(stack_id);
+ }
+
+ return 0;
+}
+
+static depot_stack_handle_t
+stack_depot_trie_save_constrained(unsigned long *entries,
+ unsigned int nr_entries, bool trylock)
+{
+ struct stack_depot_trie_side_prealloc side_prealloc = {};
+ void *pool_prealloc = NULL;
+ depot_stack_handle_t handle;
+ unsigned long flags;
+ u32 stack_id;
+
+ handle = trie_find_handle(entries, nr_entries);
+ if (handle)
+ return handle;
+
+ if (trylock) {
+ if (!raw_spin_trylock_irqsave(&stack_depot_trie_writer_lock, flags))
+ return 0;
+ if (!raw_spin_trylock(&pool_lock)) {
+ raw_spin_unlock_irqrestore(&stack_depot_trie_writer_lock, flags);
+ return 0;
+ }
+ } else {
+ raw_spin_lock_irqsave(&stack_depot_trie_writer_lock, flags);
+ raw_spin_lock(&pool_lock);
+ }
+
+ printk_deferred_enter();
+ stack_id = stack_depot_trie_insert(entries, nr_entries, &pool_prealloc,
+ &side_prealloc);
+ printk_deferred_exit();
+ raw_spin_unlock(&pool_lock);
+ raw_spin_unlock_irqrestore(&stack_depot_trie_writer_lock, flags);
+
+ return stack_id ? trie_handle(stack_id) : 0;
+}
+
depot_stack_handle_t stack_depot_save_flags(unsigned long *entries,
unsigned int nr_entries,
gfp_t alloc_flags,
@@ -655,6 +1465,9 @@ depot_stack_handle_t stack_depot_save_flags(unsigned long *entries,
if (WARN_ON(depot_flags & ~STACK_DEPOT_FLAGS_MASK))
return 0;
+ if (WARN_ON_ONCE((depot_flags & STACK_DEPOT_FLAG_GET) &&
+ (depot_flags & STACK_DEPOT_FLAG_COUNTABLE)))
+ return 0;
/*
* If this stack trace is from an interrupt, including anything before
@@ -669,6 +1482,20 @@ depot_stack_handle_t stack_depot_save_flags(unsigned long *entries,
if (unlikely(nr_entries == 0) || stack_depot_disabled)
return 0;
+ if (!(depot_flags & (STACK_DEPOT_FLAG_GET | STACK_DEPOT_FLAG_COUNTABLE)) &&
+ static_branch_unlikely(&stack_depot_trie_enabled)) {
+ if (nr_entries > CONFIG_STACKDEPOT_MAX_FRAMES)
+ nr_entries = CONFIG_STACKDEPOT_MAX_FRAMES;
+ if (in_nmi()) {
+ WARN_ON_ONCE(can_alloc);
+ return trie_find_handle(entries, nr_entries);
+ }
+ if (!can_alloc)
+ return stack_depot_trie_save_constrained(entries, nr_entries,
+ !allow_spin);
+ return stack_depot_trie_save(entries, nr_entries, alloc_flags);
+ }
+
hash = hash_stack(entries, nr_entries);
bucket = &stack_table[hash & stack_hash_mask];
@@ -751,10 +1578,728 @@ EXPORT_SYMBOL_GPL(stack_depot_save);
struct stack_record *__stack_depot_get_stack_record(depot_stack_handle_t handle)
{
+ struct stack_record *stack;
+
if (!handle)
return NULL;
+ if (WARN_ON_ONCE(stack_depot_handle_is_trie(handle)))
+ return NULL;
+
+ stack = depot_fetch_stack(handle);
+ if (!stack)
+ return NULL;
+ if (WARN_ON_ONCE(!(stack->flags & STACK_DEPOT_FLAG_COUNTABLE)))
+ return NULL;
+
+ return stack;
+}
+
+static void frame_run_init(const unsigned long *entries,
+ unsigned int nr_entries,
+ struct stack_depot_frame_run *run)
+{
+ u32 payload;
+ unsigned int i;
+ bool compressed;
+
+ compressed = arch_stack_depot_frame_try_compress(entries[0], &payload);
+ for (i = 1; i < nr_entries; i++) {
+ bool next;
+
+ next = arch_stack_depot_frame_try_compress(entries[i], &payload);
+ if (next != compressed)
+ break;
+ }
+
+ /* @i is the first non-matching frame, or @nr_entries if all matched. */
+ run->mode = compressed ? STACK_DEPOT_FRAME_COMPRESSED : STACK_DEPOT_FRAME_RAW;
+ run->nr_entries = i;
+}
+
+static void
+stack_depot_trie_node_frame(const struct stack_depot_trie_node *node,
+ unsigned int index, unsigned long *frame)
+{
+ u32 payload;
+
+ if (node->run.mode == STACK_DEPOT_FRAME_RAW) {
+ memcpy(frame, node->data + index * sizeof(*frame),
+ sizeof(*frame));
+ return;
+ }
+
+ memcpy(&payload, node->data + index * sizeof(payload), sizeof(payload));
+ arch_stack_depot_frame_decompress(payload, frame);
+}
+
+static void trie_node_init(struct stack_depot_trie_node *node,
+ const struct stack_depot_trie_node *parent, u32 stack_id,
+ const unsigned long *entries,
+ const struct stack_depot_frame_run *run)
+{
+ if (run->mode == STACK_DEPOT_FRAME_COMPRESSED) {
+ unsigned int i;
+
+ for (i = 0; i < run->nr_entries; i++) {
+ u32 payload;
+
+ arch_stack_depot_frame_try_compress(entries[i], &payload);
+ memcpy(node->data + i * sizeof(payload), &payload,
+ sizeof(payload));
+ }
+ } else {
+ memcpy(node->data, entries, stack_depot_frame_run_bytes(run));
+ }
+
+ RCU_INIT_POINTER(node->parent, parent);
+ RCU_INIT_POINTER(node->children, NULL);
+ node->stack_id = stack_id;
+ node->run = *run;
+}
+
+static void trie_node_init_slice(struct stack_depot_trie_node *node,
+ const struct stack_depot_trie_node *parent, u32 stack_id,
+ const struct stack_depot_trie_node *src_node,
+ unsigned int start, unsigned int nr_entries)
+{
+ struct stack_depot_frame_run run;
+ size_t entry_bytes;
+
+ run = src_node->run;
+ run.nr_entries = nr_entries;
+
+ entry_bytes = stack_depot_frame_run_entry_bytes(src_node->run.mode);
+ memcpy(node->data, src_node->data + start * entry_bytes,
+ stack_depot_frame_run_bytes(&run));
+ RCU_INIT_POINTER(node->parent, parent);
+ RCU_INIT_POINTER(node->children, NULL);
+ node->stack_id = stack_id;
+ node->run = run;
+}
+
+static unsigned int trie_node_match(const struct stack_depot_trie_node *node,
+ const unsigned long *entries,
+ unsigned int nr_entries)
+{
+ unsigned int limit;
+ unsigned int i;
+
+ limit = min(node->run.nr_entries, nr_entries);
+ if (node->run.mode == STACK_DEPOT_FRAME_RAW) {
+ for (i = 0; i < limit; i++) {
+ unsigned long frame;
+
+ memcpy(&frame, node->data + i * sizeof(frame), sizeof(frame));
+ if (frame != entries[i])
+ break;
+ }
+
+ return i;
+ }
+
+ for (i = 0; i < limit; i++) {
+ unsigned long frame;
+
+ stack_depot_trie_node_frame(node, i, &frame);
+ if (frame != entries[i])
+ break;
+ }
+
+ return i;
+}
+
+static inline const struct stack_depot_trie_node *
+trie_load_parent(const struct stack_depot_trie_node *node)
+{
+ return rcu_dereference_check(node->parent,
+ lockdep_is_held(&stack_depot_trie_writer_lock) ||
+ rcu_read_lock_sched_held());
+}
+
+static inline const struct stack_depot_trie_children *
+trie_load_children(const struct stack_depot_trie_children __rcu * const *slot)
+{
+ return rcu_dereference_check(*slot,
+ lockdep_is_held(&stack_depot_trie_writer_lock) ||
+ rcu_read_lock_sched_held());
+}
+
+static inline const struct stack_depot_trie_node *
+trie_children_load_child(const struct stack_depot_trie_children *children,
+ unsigned int pos)
+{
+ return rcu_dereference_check(children->nodes[pos],
+ lockdep_is_held(&stack_depot_trie_writer_lock) ||
+ rcu_read_lock_sched_held());
+}
+
+static bool
+trie_children_find_position(const struct stack_depot_trie_children *children,
+ unsigned long frame, unsigned int *pos)
+{
+ unsigned int left = 0;
+ unsigned int right;
+
+ right = READ_ONCE(children->nr_children);
+ while (left < right) {
+ unsigned int mid = left + (right - left) / 2;
+ const struct stack_depot_trie_node *node;
+ unsigned long mid_frame;
+
+ node = trie_children_load_child(children, mid);
+ if (!node) {
+ /* Tail append may produce a transient lockless lookup miss. */
+ right = mid;
+ continue;
+ }
+ stack_depot_trie_node_frame(node, 0, &mid_frame);
+ if (mid_frame < frame) {
+ left = mid + 1;
+ } else if (mid_frame > frame) {
+ right = mid;
+ } else {
+ *pos = mid;
+ return true;
+ }
+ }
+
+ *pos = left;
+ return false;
+}
+
+/* Initialize an unpublished container from a stable published prefix. */
+static void trie_children_init(const struct stack_depot_trie_children *old,
+ struct stack_depot_trie_children *new)
+{
+ unsigned int nr_old = old->nr_children;
+ unsigned int i;
+
+ new->nr_children = nr_old;
+ for (i = 0; i < nr_old; i++)
+ RCU_INIT_POINTER(new->nodes[i], trie_children_load_child(old, i));
+ for (i = nr_old; i < new->capacity; i++)
+ RCU_INIT_POINTER(new->nodes[i], NULL);
+}
+
+static void trie_children_insert(struct stack_depot_trie_children *children,
+ const struct stack_depot_trie_node *node,
+ unsigned int pos)
+{
+ unsigned int i;
+
+ for (i = children->nr_children; i > pos; i--)
+ RCU_INIT_POINTER(children->nodes[i],
+ trie_children_load_child(children, i - 1));
+ RCU_INIT_POINTER(children->nodes[pos], node);
+ children->nr_children++;
+}
+
+static void trie_reparent_children(struct stack_depot_trie_node *parent)
+{
+ const struct stack_depot_trie_children *children;
+ unsigned int i;
+
+ lockdep_assert_held(&stack_depot_trie_writer_lock);
+
+ children = trie_load_children(&parent->children);
+ if (!children)
+ return;
+ /*
+ * Replacement nodes reuse unchanged descendant subtrees. Repoint their
+ * parent links before retiring the old parent so fetch never follows a freed
+ * node. Lockless fetches may see the new parent before publication, but the
+ * old and new parent chains contain the same frames and remain RCU-live.
+ */
+ for (i = 0; i < children->nr_children; i++) {
+ struct stack_depot_trie_node *child;
+
+ child = (struct stack_depot_trie_node *)trie_children_load_child(children, i);
+ rcu_assign_pointer(child->parent, parent);
+ }
+}
+
+/*
+ * Split entries into runs, allocate and initialize each node once, and link
+ * adjacent nodes through singleton children. Both trie locks must be held.
+ * Failure walks the unpublished parent chain and releases local ownership.
+ */
+static const struct stack_depot_trie_node *
+trie_path_alloc(const struct stack_depot_trie_node *parent, u32 stack_id,
+ const unsigned long *entries, unsigned int nr_entries,
+ void **pool_prealloc,
+ const struct stack_depot_trie_node **node_out)
+{
+ struct stack_depot_trie_children *path_children = NULL;
+ const struct stack_depot_trie_node *path_root = NULL;
+ const struct stack_depot_trie_node *last_node = parent;
+ unsigned int entry = 0;
+
+ lockdep_assert_held(&pool_lock);
+ lockdep_assert_held(&stack_depot_trie_writer_lock);
+
+ while (entry < nr_entries) {
+ struct stack_depot_frame_run run;
+ struct stack_depot_trie_node *node;
+
+ frame_run_init(&entries[entry], nr_entries - entry, &run);
+ node = trie_pool_alloc(trie_node_bytes(&run), pool_prealloc);
+ if (!node)
+ goto err_release;
+
+ trie_node_init(node, last_node,
+ entry + run.nr_entries == nr_entries ? stack_id : 0,
+ &entries[entry], &run);
+ entry += run.nr_entries;
+ last_node = node;
+ if (!path_root)
+ path_root = node;
+
+ if (path_children)
+ trie_children_insert(path_children, last_node, 0);
+ if (entry < nr_entries) {
+ path_children = trie_pool_alloc_children(1, pool_prealloc);
+ if (!path_children)
+ goto err_release;
+ RCU_INIT_POINTER(node->children, path_children);
+ }
+ }
+
+ *node_out = last_node;
+ return path_root;
+
+err_release:
+ while (last_node != parent) {
+ const struct stack_depot_trie_children *node_children;
+ const struct stack_depot_trie_node *node = last_node;
+
+ last_node = trie_load_parent(node);
+ node_children = trie_load_children(&node->children);
+ if (node_children)
+ trie_pool_release_children(node_children);
+ trie_pool_release(node, trie_node_bytes(&node->run));
+ }
+ return NULL;
+}
+
+static const struct stack_depot_trie_node *
+stack_depot_trie_lookup(const unsigned long *entries, unsigned int nr_entries)
+{
+ const struct stack_depot_trie_children *children;
+ unsigned int entry = 0;
+
+ children = trie_load_children(&stack_depot_trie_root);
+
+ while (entry < nr_entries) {
+ const struct stack_depot_trie_node *node;
+ unsigned int remaining = nr_entries - entry;
+ unsigned int matched;
+ unsigned int pos;
+
+ if (!children)
+ return NULL;
+ if (!trie_children_find_position(children, entries[entry], &pos))
+ return NULL;
+
+ node = trie_children_load_child(children, pos);
+ matched = trie_node_match(node, &entries[entry], remaining);
+ if (matched < node->run.nr_entries)
+ return NULL;
+ entry += matched;
+ if (entry == nr_entries)
+ return node->stack_id ? node : NULL;
+
+ children = trie_load_children(&node->children);
+ }
+
+ return NULL;
+}
+
+static u32
+trie_insert_path(const struct stack_depot_trie_children __rcu **slot,
+ struct stack_depot_trie_node *parent,
+ const struct stack_depot_trie_children *children,
+ unsigned int pos, const unsigned long *entries,
+ unsigned int nr_entries, void **pool_prealloc,
+ struct stack_depot_trie_side_prealloc *side_prealloc)
+{
+ struct stack_depot_trie_children *new_children = NULL;
+ const struct stack_depot_trie_node *path_root;
+ const struct stack_depot_trie_node *node;
+ unsigned int capacity = 1;
+ u32 new_stack_id;
+ bool tail_append = false;
+
+ /*
+ * Reuse spare capacity only for a sorted tail append. Other insertions
+ * replace the children container without modifying visible pointers.
+ */
+ if (children) {
+ capacity = roundup_pow_of_two(children->nr_children + 1);
+ tail_append = pos == children->nr_children &&
+ children->nr_children < children->capacity;
+ }
+ if (!tail_append && trie_children_alloc_size(capacity) >
+ STACK_DEPOT_TRIE_POOL_USABLE_SIZE)
+ return 0;
+
+ new_stack_id = trie_side_table_prepare_stack_slot(side_prealloc);
+ if (!new_stack_id)
+ return 0;
+
+ /* Reserve replacement topology before the path, the final fallible step. */
+ if (!tail_append) {
+ new_children = trie_pool_alloc_children(capacity, pool_prealloc);
+ if (!new_children)
+ goto err_release;
+ }
+ path_root = trie_path_alloc(parent, new_stack_id, entries, nr_entries,
+ pool_prealloc, &node);
+ if (!path_root)
+ goto err_release;
+
+ /* Commit the stack ID before making the path reachable from the trie. */
+ trie_side_table_publish(node);
+ if (tail_append) {
+ struct stack_depot_trie_children *tail_children =
+ (struct stack_depot_trie_children *)children;
+
+ /*
+ * Publish the node before the visible count. Readers may transiently
+ * see NULL and miss; the writer-lock recheck prevents duplicates.
+ */
+ rcu_assign_pointer(tail_children->nodes[pos], path_root);
+ WRITE_ONCE(tail_children->nr_children, pos + 1);
+ } else {
+ if (children)
+ trie_children_init(children, new_children);
+ trie_children_insert(new_children, path_root, pos);
+ rcu_assign_pointer(*slot, new_children);
+ if (children)
+ trie_retire_children(children);
+ }
+
+ return new_stack_id;
+
+err_release:
+ if (new_children)
+ trie_pool_release_children(new_children);
+ return 0;
+}
+
+static u32
+trie_split_child(const struct stack_depot_trie_children __rcu **slot,
+ const struct stack_depot_trie_children *children,
+ const struct stack_depot_trie_node *child,
+ unsigned int pos, unsigned int matched,
+ const unsigned long *entries, unsigned int nr_entries,
+ void **pool_prealloc,
+ struct stack_depot_trie_side_prealloc *side_prealloc)
+{
+ struct stack_depot_trie_children *prefix_children = NULL;
+ struct stack_depot_trie_children *new_children = NULL;
+ const struct stack_depot_trie_node *new_node;
+ const struct stack_depot_trie_node *suffix_roots[2];
+ struct stack_depot_frame_run run;
+ struct stack_depot_trie_node *split_prefix = NULL;
+ struct stack_depot_trie_node *old_suffix = NULL;
+ unsigned int nr_suffix_roots;
+ unsigned int old_suffix_len;
+ unsigned int i;
+ size_t split_prefix_size;
+ size_t old_suffix_size;
+ u32 new_stack_id;
+ bool has_new_suffix;
+
+ new_stack_id = trie_side_table_prepare_stack_slot(side_prealloc);
+ if (!new_stack_id)
+ return 0;
+
+ /* Rebuild the child's run as newly allocated prefix and old suffix nodes. */
+ run = child->run;
+ run.nr_entries = matched;
+ split_prefix_size = trie_node_bytes(&run);
+ old_suffix_len = child->run.nr_entries - matched;
+ run.nr_entries = old_suffix_len;
+ old_suffix_size = trie_node_bytes(&run);
+ has_new_suffix = matched < nr_entries;
+ nr_suffix_roots = has_new_suffix ? 2 : 1;
+
+ /* Reserve fixed split topology before the optional new suffix path. */
+ split_prefix = trie_pool_alloc(split_prefix_size, pool_prealloc);
+ if (!split_prefix)
+ goto err_release;
+ old_suffix = trie_pool_alloc(old_suffix_size, pool_prealloc);
+ if (!old_suffix)
+ goto err_release;
+ new_children = trie_pool_alloc_children(children->capacity, pool_prealloc);
+ if (!new_children)
+ goto err_release;
+ prefix_children = trie_pool_alloc_children(nr_suffix_roots, pool_prealloc);
+ if (!prefix_children)
+ goto err_release;
+
+ if (has_new_suffix) {
+ const struct stack_depot_trie_node *new_suffix;
+ unsigned long old_suffix_frame;
+
+ new_suffix = trie_path_alloc(split_prefix, new_stack_id,
+ &entries[matched], nr_entries - matched,
+ pool_prealloc, &new_node);
+ if (!new_suffix)
+ goto err_release;
+ stack_depot_trie_node_frame(child, matched, &old_suffix_frame);
+ /* Children remain sorted by the first frame of each suffix. */
+ if (old_suffix_frame < entries[matched]) {
+ suffix_roots[0] = old_suffix;
+ suffix_roots[1] = new_suffix;
+ } else {
+ suffix_roots[0] = new_suffix;
+ suffix_roots[1] = old_suffix;
+ }
+ } else {
+ new_node = split_prefix;
+ suffix_roots[0] = old_suffix;
+ }
+
+ /* Rebuild the old path as prefix -> old suffix and attach suffix roots. */
+ trie_node_init_slice(split_prefix, trie_load_parent(child),
+ has_new_suffix ? 0 : new_stack_id, child, 0, matched);
+ trie_node_init_slice(old_suffix, split_prefix, child->stack_id, child,
+ matched, old_suffix_len);
+ for (i = 0; i < nr_suffix_roots; i++)
+ trie_children_insert(prefix_children, suffix_roots[i], i);
+ RCU_INIT_POINTER(old_suffix->children,
+ trie_load_children(&child->children));
+ RCU_INIT_POINTER(split_prefix->children, prefix_children);
+
+ /* Publish IDs, reparent descendants, then replace and retire topology. */
+ if (child->stack_id)
+ trie_side_table_publish(old_suffix);
+ trie_side_table_publish(new_node);
+ /* Old and replacement chains contain identical frames during transition. */
+ trie_children_init(children, new_children);
+ RCU_INIT_POINTER(new_children->nodes[pos], split_prefix);
+ trie_reparent_children(old_suffix);
+ rcu_assign_pointer(*slot, new_children);
+ trie_retire_children_with_node(children, child);
+
+ return new_stack_id;
+
+err_release:
+ if (split_prefix)
+ trie_pool_release(split_prefix, split_prefix_size);
+ if (old_suffix)
+ trie_pool_release(old_suffix, old_suffix_size);
+ if (prefix_children)
+ trie_pool_release_children(prefix_children);
+ if (new_children)
+ trie_pool_release_children(new_children);
+ return 0;
+}
+
+static u32
+trie_promote_child(const struct stack_depot_trie_children __rcu **slot,
+ const struct stack_depot_trie_children *children,
+ const struct stack_depot_trie_node *child,
+ unsigned int pos, void **pool_prealloc,
+ struct stack_depot_trie_side_prealloc *side_prealloc)
+{
+ struct stack_depot_trie_children *new_children;
+ struct stack_depot_trie_node *promoted_node;
+ size_t node_size;
+ u32 new_stack_id;
+
+ new_stack_id = trie_side_table_prepare_stack_slot(side_prealloc);
+ if (!new_stack_id)
+ return 0;
+ node_size = trie_node_bytes(&child->run);
+
+ /* Reserve a clone and replacement children container before publication. */
+ promoted_node = trie_pool_alloc(node_size, pool_prealloc);
+ if (!promoted_node)
+ return 0;
+ new_children = trie_pool_alloc_children(children->capacity, pool_prealloc);
+ if (!new_children)
+ goto out_release_node;
+
+ /* Add the stack ID through a clone, then reparent before retirement. */
+ memcpy(promoted_node, child, node_size);
+ promoted_node->stack_id = new_stack_id;
+ trie_side_table_publish(promoted_node);
+ trie_children_init(children, new_children);
+ RCU_INIT_POINTER(new_children->nodes[pos], promoted_node);
+ trie_reparent_children(promoted_node);
+ rcu_assign_pointer(*slot, new_children);
+ trie_retire_children_with_node(children, child);
+
+ return new_stack_id;
+
+out_release_node:
+ trie_pool_release(promoted_node, node_size);
+ return 0;
+}
+
+static u32
+stack_depot_trie_insert(const unsigned long *entries,
+ unsigned int nr_entries, void **pool_prealloc,
+ struct stack_depot_trie_side_prealloc *side_prealloc)
+{
+ const struct stack_depot_trie_children *children;
+ const struct stack_depot_trie_children __rcu **slot =
+ &stack_depot_trie_root;
+ const struct stack_depot_trie_node *child;
+ struct stack_depot_trie_node *parent = NULL;
+ unsigned int matched;
+ unsigned int pos;
+ u32 stack_id;
+
+ lockdep_assert_held(&stack_depot_trie_writer_lock);
+ lockdep_assert_held(&pool_lock);
+
+ for (;;) {
+ pos = 0;
+ children = trie_load_children(slot);
+ /* No matching child: attach the remaining path. */
+ if (!children ||
+ !trie_children_find_position(children, entries[0], &pos)) {
+ stack_id = trie_insert_path(slot, parent, children, pos,
+ entries, nr_entries, pool_prealloc,
+ side_prealloc);
+ break;
+ }
+
+ child = trie_children_load_child(children, pos);
+ matched = trie_node_match(child, entries, nr_entries);
+ /* A partial child match requires a prefix/suffix split. */
+ if (matched < child->run.nr_entries) {
+ stack_id = trie_split_child(slot, children, child, pos,
+ matched, entries, nr_entries,
+ pool_prealloc, side_prealloc);
+ break;
+ }
+
+ /* The input ends here: reuse a stack node or promote an internal one. */
+ if (matched == nr_entries) {
+ if (child->stack_id)
+ return child->stack_id;
+ stack_id = trie_promote_child(slot, children, child, pos,
+ pool_prealloc, side_prealloc);
+ break;
+ }
+
+ /* The child matched completely; continue with the remaining frames. */
+ parent = (struct stack_depot_trie_node *)child;
+ slot = &parent->children;
+ entries += matched;
+ nr_entries -= matched;
+ }
+
+ if (stack_id)
+ trie_side_table_last_stack_id = stack_id;
+ return stack_id;
+}
+
+static unsigned int trie_fetch_into(const struct stack_depot_trie_node *node,
+ unsigned long *entries,
+ unsigned int max_entries)
+{
+ const struct stack_depot_trie_node *cur;
+ unsigned int total;
+ unsigned int pos;
+ unsigned int i;
+
+ total = 0;
+ for (cur = node; cur; cur = trie_load_parent(cur))
+ total += cur->run.nr_entries;
+ if (max_entries < total)
+ return 0;
+
+ pos = total;
+ for (cur = node; cur; cur = trie_load_parent(cur)) {
+ pos -= cur->run.nr_entries;
+ for (i = 0; i < cur->run.nr_entries; i++)
+ stack_depot_trie_node_frame(cur, i, &entries[pos + i]);
+ }
+
+ return total;
+}
- return depot_fetch_stack(handle);
+static unsigned int trie_fetch_range(const struct stack_depot_trie_node *node,
+ unsigned int offset,
+ unsigned long *entries,
+ unsigned int max_entries)
+{
+ const struct stack_depot_trie_node *cur;
+ unsigned int end;
+ unsigned int start;
+ unsigned int total;
+ unsigned int pos;
+ unsigned int i;
+
+ total = 0;
+ for (cur = node; cur; cur = trie_load_parent(cur))
+ total += cur->run.nr_entries;
+ if (offset >= total)
+ return 0;
+
+ max_entries = min(max_entries, total - offset);
+ end = offset + max_entries;
+ pos = total;
+ for (cur = node; cur; cur = trie_load_parent(cur)) {
+ pos -= cur->run.nr_entries;
+ start = max(pos, offset);
+ for (i = start; i < min(pos + cur->run.nr_entries, end); i++)
+ stack_depot_trie_node_frame(cur, i - pos, &entries[i - offset]);
+ }
+
+ return max_entries;
+}
+
+static unsigned int trie_fetch_handle_into(depot_stack_handle_t handle,
+ unsigned long *entries,
+ unsigned int max_entries)
+{
+ const struct stack_depot_trie_node *node;
+ u32 stack_id;
+ unsigned int nr_entries;
+
+ stack_id = trie_stack_id(handle);
+ rcu_read_lock_sched_notrace();
+ node = trie_side_table_lookup(stack_id);
+ if (WARN_ONCE(!node, "corrupt trie handle %08x\n", handle)) {
+ rcu_read_unlock_sched_notrace();
+ return 0;
+ }
+ nr_entries = trie_fetch_into(node, entries, max_entries);
+ rcu_read_unlock_sched_notrace();
+ if (nr_entries)
+ kmsan_unpoison_memory(entries, nr_entries * sizeof(*entries));
+
+ return nr_entries;
+}
+
+static unsigned int trie_fetch_handle_range(depot_stack_handle_t handle,
+ unsigned int offset,
+ unsigned long *entries,
+ unsigned int max_entries)
+{
+ const struct stack_depot_trie_node *node;
+ u32 stack_id;
+ unsigned int nr_entries;
+
+ stack_id = trie_stack_id(handle);
+ rcu_read_lock_sched_notrace();
+ node = trie_side_table_lookup(stack_id);
+ if (WARN_ONCE(!node, "corrupt trie handle %08x\n", handle)) {
+ rcu_read_unlock_sched_notrace();
+ return 0;
+ }
+ nr_entries = trie_fetch_range(node, offset, entries, max_entries);
+ rcu_read_unlock_sched_notrace();
+ if (nr_entries)
+ kmsan_unpoison_memory(entries, nr_entries * sizeof(*entries));
+
+ return nr_entries;
}
unsigned int stack_depot_fetch(depot_stack_handle_t handle,
@@ -771,6 +2316,8 @@ unsigned int stack_depot_fetch(depot_stack_handle_t handle,
if (!handle || stack_depot_disabled)
return 0;
+ if (WARN_ON_ONCE(stack_depot_handle_is_trie(handle)))
+ return 0;
stack = depot_fetch_stack(handle);
/*
@@ -785,12 +2332,44 @@ unsigned int stack_depot_fetch(depot_stack_handle_t handle,
}
EXPORT_SYMBOL_GPL(stack_depot_fetch);
+unsigned int stack_depot_fetch_into(depot_stack_handle_t handle,
+ unsigned long *entries,
+ unsigned int max_entries)
+{
+ struct stack_record *stack;
+ unsigned int nr_entries;
+
+ if (!handle)
+ return 0;
+ if (stack_depot_disabled)
+ return 0;
+ WARN_ON_ONCE(!entries || !max_entries);
+ if (stack_depot_handle_is_trie(handle))
+ return trie_fetch_handle_into(handle, entries, max_entries);
+
+ stack = depot_fetch_stack(handle);
+ if (!stack)
+ return 0;
+ nr_entries = stack->size;
+ if (WARN_ON_ONCE(!nr_entries))
+ return 0;
+ if (nr_entries > max_entries)
+ return 0;
+
+ memcpy(entries, stack->entries, nr_entries * sizeof(*entries));
+ kmsan_unpoison_memory(entries, nr_entries * sizeof(*entries));
+ return nr_entries;
+}
+EXPORT_SYMBOL_GPL(stack_depot_fetch_into);
+
void stack_depot_put(depot_stack_handle_t handle)
{
struct stack_record *stack;
if (!handle || stack_depot_disabled)
return;
+ if (WARN_ON_ONCE(stack_depot_handle_is_trie(handle)))
+ return;
stack = depot_fetch_stack(handle);
/*
@@ -800,16 +2379,60 @@ void stack_depot_put(depot_stack_handle_t handle)
if (WARN(!stack, "corrupt handle or unbalanced stack_depot_put()"))
return;
+ if (WARN_ON_ONCE(stack->flags & STACK_DEPOT_FLAG_COUNTABLE))
+ return;
if (refcount_dec_and_test(&stack->count))
depot_free_stack(stack);
}
EXPORT_SYMBOL_GPL(stack_depot_put);
+static void trie_print(depot_stack_handle_t handle)
+{
+ unsigned long entries[STACK_DEPOT_PRINT_CHUNK_FRAMES];
+ unsigned int nr_entries;
+ unsigned int offset = 0;
+
+ while ((nr_entries = trie_fetch_handle_range(handle, offset, entries,
+ ARRAY_SIZE(entries)))) {
+ stack_trace_print(entries, nr_entries, 0);
+ offset += nr_entries;
+ }
+}
+
+static int trie_snprint(depot_stack_handle_t handle, char *buf, size_t size,
+ int spaces)
+{
+ unsigned long entries[STACK_DEPOT_PRINT_CHUNK_FRAMES];
+ unsigned int generated;
+ unsigned int nr_entries;
+ unsigned int offset = 0;
+ unsigned int total = 0;
+
+ while (size &&
+ (nr_entries = trie_fetch_handle_range(handle, offset, entries,
+ ARRAY_SIZE(entries)))) {
+ generated = stack_trace_snprint(buf, size, entries, nr_entries, spaces);
+ total += generated;
+ if (generated >= size)
+ break;
+ buf += generated;
+ size -= generated;
+ offset += nr_entries;
+ }
+
+ return total;
+}
+
void stack_depot_print(depot_stack_handle_t stack)
{
unsigned long *entries;
unsigned int nr_entries;
+ if (stack_depot_handle_is_trie(stack)) {
+ trie_print(stack);
+ return;
+ }
+
nr_entries = stack_depot_fetch(stack, &entries);
if (nr_entries > 0)
stack_trace_print(entries, nr_entries, 0);
@@ -822,6 +2445,9 @@ int stack_depot_snprint(depot_stack_handle_t handle, char *buf, size_t size,
unsigned long *entries;
unsigned int nr_entries;
+ if (stack_depot_handle_is_trie(handle))
+ return trie_snprint(handle, buf, size, spaces);
+
nr_entries = stack_depot_fetch(handle, &entries);
return nr_entries ? stack_trace_snprint(buf, size, entries, nr_entries,
spaces) : 0;
diff --git a/lib/tests/Makefile b/lib/tests/Makefile
index 3cac3b63a7522..1f72191f98bbc 100644
--- a/lib/tests/Makefile
+++ b/lib/tests/Makefile
@@ -48,6 +48,7 @@ obj-$(CONFIG_SCANF_KUNIT_TEST) += scanf_kunit.o
obj-$(CONFIG_SEQ_BUF_KUNIT_TEST) += seq_buf_kunit.o
obj-$(CONFIG_SIPHASH_KUNIT_TEST) += siphash_kunit.o
obj-$(CONFIG_SLUB_KUNIT_TEST) += slub_kunit.o
+obj-$(CONFIG_STACKDEPOT_KUNIT_TEST) += stackdepot_kunit.o
obj-$(CONFIG_TEST_SORT) += test_sort.o
CFLAGS_stackinit_kunit.o += $(call cc-disable-warning, switch-unreachable)
obj-$(CONFIG_STACKINIT_KUNIT_TEST) += stackinit_kunit.o
diff --git a/lib/tests/stackdepot_kunit.c b/lib/tests/stackdepot_kunit.c
new file mode 100644
index 0000000000000..b86b84d56176e
--- /dev/null
+++ b/lib/tests/stackdepot_kunit.c
@@ -0,0 +1,582 @@
+// SPDX-License-Identifier: GPL-2.0-only
+
+#include <kunit/test.h>
+#include <linux/array_size.h>
+#include <linux/gfp.h>
+#include <linux/kallsyms.h>
+#include <linux/limits.h>
+#include <linux/moduleparam.h>
+#include <linux/stackdepot.h>
+#include <linux/stacktrace.h>
+#include <linux/string.h>
+
+#include <asm/stackdepot.h>
+
+static int expected_trie_pool_limit = -1;
+module_param_named(trie_pool_limit, expected_trie_pool_limit, int, 0);
+MODULE_PARM_DESC(trie_pool_limit, "Expected stackdepot hash/trie pool split");
+
+#ifdef CONFIG_ARM64
+#include <asm/sections.h>
+
+static inline unsigned long stackdepot_arm64_frame(long offset)
+{
+ return (unsigned long)((long)_text + offset);
+}
+#endif
+
+static unsigned long stackdepot_test_frame(unsigned int i)
+{
+#ifdef CONFIG_ARM64
+ return i & 1 ? 0x1000UL + i * 0x1000UL :
+ stackdepot_arm64_frame(i * 4);
+#elif defined(CONFIG_X86_64) && !defined(CONFIG_UML)
+ return i & 1 ? 0xffff888000000000UL + i * 0x1000UL :
+ 0xffffffff10000000UL + i * 0x10UL;
+#else
+ return 0x1000UL + i * 0x1000UL;
+#endif
+}
+
+static void stackdepot_trie_max_path_roundtrip(struct kunit *test)
+{
+ union handle_parts parts;
+ unsigned long *entries;
+ unsigned long *fetched;
+ depot_stack_handle_t handle;
+ size_t size = CONFIG_STACKDEPOT_MAX_FRAMES * sizeof(*entries);
+ u32 pool_index_plus_1;
+ unsigned int i;
+
+ if (expected_trie_pool_limit < 0)
+ kunit_skip(test, "trie pool limit was not provided");
+ KUNIT_ASSERT_EQ(test, stack_depot_init(), 0);
+ entries = kunit_kcalloc(test, CONFIG_STACKDEPOT_MAX_FRAMES,
+ sizeof(*entries), GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, entries);
+ fetched = kunit_kcalloc(test, CONFIG_STACKDEPOT_MAX_FRAMES,
+ sizeof(*fetched), GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, fetched);
+ for (i = 0; i < CONFIG_STACKDEPOT_MAX_FRAMES; i++)
+ entries[i] = stackdepot_test_frame(i);
+
+ handle = stack_depot_save(entries, CONFIG_STACKDEPOT_MAX_FRAMES,
+ GFP_KERNEL);
+ KUNIT_ASSERT_NE(test, handle, (depot_stack_handle_t)0);
+ parts.handle = handle;
+ pool_index_plus_1 = parts.pool_index_plus_1;
+ KUNIT_EXPECT_GT(test, pool_index_plus_1, (u32)expected_trie_pool_limit);
+ KUNIT_EXPECT_EQ(test,
+ stack_depot_fetch_into(handle, fetched,
+ CONFIG_STACKDEPOT_MAX_FRAMES),
+ (unsigned int)CONFIG_STACKDEPOT_MAX_FRAMES);
+ KUNIT_EXPECT_MEMEQ(test, fetched, entries, size);
+ KUNIT_EXPECT_EQ(test,
+ stack_depot_save(entries, CONFIG_STACKDEPOT_MAX_FRAMES,
+ GFP_KERNEL),
+ handle);
+}
+
+static void stackdepot_save_flags_public(struct kunit *test)
+{
+ union handle_parts parts;
+ unsigned long entries[] = { 0x501000UL, 0x502000UL, 0x503000UL };
+ unsigned long get_entries[] = { 0x601000UL, 0x602000UL };
+ unsigned long missing_entries[] = { 0x701000UL, 0x702000UL };
+ unsigned long blocking_entries[] = { 0x711000UL, 0x712000UL };
+ unsigned long fetched[ARRAY_SIZE(entries)] = {};
+ depot_stack_handle_t blocking_handle;
+ depot_stack_handle_t noalloc_handle;
+ depot_stack_handle_t overlong_handle;
+ depot_stack_handle_t plain_handle;
+ depot_stack_handle_t get_handle;
+ depot_stack_handle_t again;
+ depot_stack_handle_t extra;
+ gfp_t no_spin = GFP_NOWAIT & ~__GFP_RECLAIM;
+ u32 pool_index_plus_1;
+ unsigned long *overlong_fetched;
+ unsigned long *overlong_entries;
+ unsigned int overlong_nr = CONFIG_STACKDEPOT_MAX_FRAMES + 1;
+ unsigned int nr_entries;
+ size_t overlong_size;
+ unsigned int i;
+
+ KUNIT_ASSERT_EQ(test, stack_depot_init(), 0);
+ overlong_entries = kunit_kcalloc(test, overlong_nr,
+ sizeof(*overlong_entries), GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, overlong_entries);
+ overlong_fetched = kunit_kcalloc(test, CONFIG_STACKDEPOT_MAX_FRAMES,
+ sizeof(*overlong_fetched), GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, overlong_fetched);
+ for (i = 0; i < overlong_nr; i++)
+ overlong_entries[i] = 0x800000UL + i * 0x1000UL;
+
+ plain_handle = stack_depot_save(entries, ARRAY_SIZE(entries), GFP_KERNEL);
+ KUNIT_ASSERT_NE(test, plain_handle, (depot_stack_handle_t)0);
+ again = stack_depot_save(entries, ARRAY_SIZE(entries), GFP_KERNEL);
+ KUNIT_EXPECT_EQ(test, again, plain_handle);
+
+ nr_entries = stack_depot_fetch_into(plain_handle, fetched,
+ ARRAY_SIZE(fetched));
+ KUNIT_EXPECT_EQ(test, nr_entries, (unsigned int)ARRAY_SIZE(entries));
+ KUNIT_EXPECT_MEMEQ(test, fetched, entries, sizeof(entries));
+
+ noalloc_handle = stack_depot_save_flags(entries, ARRAY_SIZE(entries), no_spin, 0);
+ KUNIT_EXPECT_EQ(test, noalloc_handle, plain_handle);
+ noalloc_handle = stack_depot_save_flags(missing_entries,
+ ARRAY_SIZE(missing_entries),
+ GFP_KERNEL, 0);
+ KUNIT_ASSERT_NE(test, noalloc_handle, (depot_stack_handle_t)0);
+ if (expected_trie_pool_limit >= 0) {
+ parts.handle = noalloc_handle;
+ pool_index_plus_1 = parts.pool_index_plus_1;
+ KUNIT_EXPECT_GT(test, pool_index_plus_1,
+ (u32)expected_trie_pool_limit);
+ }
+ nr_entries = stack_depot_fetch_into(noalloc_handle, fetched,
+ ARRAY_SIZE(fetched));
+ KUNIT_EXPECT_EQ(test, nr_entries,
+ (unsigned int)ARRAY_SIZE(missing_entries));
+ KUNIT_EXPECT_MEMEQ(test, fetched, missing_entries, sizeof(missing_entries));
+ KUNIT_EXPECT_EQ(test,
+ stack_depot_save_flags(missing_entries,
+ ARRAY_SIZE(missing_entries),
+ no_spin, 0),
+ noalloc_handle);
+
+ blocking_handle = stack_depot_save_flags(blocking_entries,
+ ARRAY_SIZE(blocking_entries),
+ GFP_KERNEL, 0);
+ KUNIT_ASSERT_NE(test, blocking_handle, (depot_stack_handle_t)0);
+ if (expected_trie_pool_limit >= 0) {
+ parts.handle = blocking_handle;
+ pool_index_plus_1 = parts.pool_index_plus_1;
+ KUNIT_EXPECT_GT(test, pool_index_plus_1,
+ (u32)expected_trie_pool_limit);
+ }
+ memset(fetched, 0, sizeof(fetched));
+ nr_entries = stack_depot_fetch_into(blocking_handle, fetched,
+ ARRAY_SIZE(fetched));
+ KUNIT_EXPECT_EQ(test, nr_entries,
+ (unsigned int)ARRAY_SIZE(blocking_entries));
+ KUNIT_EXPECT_MEMEQ(test, fetched, blocking_entries,
+ sizeof(blocking_entries));
+
+ get_handle = stack_depot_save_flags(get_entries, ARRAY_SIZE(get_entries),
+ GFP_KERNEL,
+ STACK_DEPOT_FLAG_CAN_ALLOC |
+ STACK_DEPOT_FLAG_GET);
+ KUNIT_ASSERT_NE(test, get_handle, (depot_stack_handle_t)0);
+ stack_depot_put(get_handle);
+
+ overlong_handle = stack_depot_save(overlong_entries, overlong_nr,
+ GFP_KERNEL);
+ KUNIT_ASSERT_NE(test, overlong_handle, (depot_stack_handle_t)0);
+ nr_entries = stack_depot_fetch_into(overlong_handle, overlong_fetched,
+ CONFIG_STACKDEPOT_MAX_FRAMES);
+ KUNIT_EXPECT_EQ(test, nr_entries, (unsigned int)CONFIG_STACKDEPOT_MAX_FRAMES);
+ overlong_size = CONFIG_STACKDEPOT_MAX_FRAMES * sizeof(*overlong_entries);
+ KUNIT_EXPECT_MEMEQ(test, overlong_fetched, overlong_entries, overlong_size);
+
+ extra = stack_depot_set_extra_bits(plain_handle, 7);
+ KUNIT_ASSERT_NE(test, extra, (depot_stack_handle_t)0);
+ KUNIT_EXPECT_EQ(test, stack_depot_get_extra_bits(extra), 7U);
+ memset(fetched, 0, sizeof(fetched));
+ nr_entries = stack_depot_fetch_into(extra, fetched, ARRAY_SIZE(fetched));
+ KUNIT_EXPECT_EQ(test, nr_entries, (unsigned int)ARRAY_SIZE(entries));
+ KUNIT_EXPECT_MEMEQ(test, fetched, entries, sizeof(entries));
+}
+
+static void stackdepot_snprint_public(struct kunit *test)
+{
+ const unsigned int nr_entries = CONFIG_STACKDEPOT_MAX_FRAMES;
+ const size_t buf_size = nr_entries * (KSYM_SYMBOL_LEN + 4);
+ unsigned long *entries;
+ char *expected;
+ char *actual;
+ depot_stack_handle_t handle;
+ unsigned int expected_len;
+ unsigned int prefix_entries;
+ unsigned int prefix_len;
+ size_t output_size;
+ unsigned int i;
+ int actual_len;
+
+ KUNIT_ASSERT_EQ(test, stack_depot_init(), 0);
+ entries = kunit_kmalloc_array(test, nr_entries, sizeof(*entries),
+ GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, entries);
+ expected = kunit_kzalloc(test, buf_size, GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, expected);
+ actual = kunit_kzalloc(test, buf_size, GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, actual);
+ for (i = 0; i < nr_entries; i++)
+ entries[i] = stackdepot_test_frame(i);
+
+ handle = stack_depot_save(entries, nr_entries, GFP_KERNEL);
+ KUNIT_ASSERT_NE(test, handle, (depot_stack_handle_t)0);
+ if (expected_trie_pool_limit >= 0) {
+ union handle_parts parts = { .handle = handle };
+
+ KUNIT_EXPECT_GT(test, (u32)parts.pool_index_plus_1,
+ (u32)expected_trie_pool_limit);
+ }
+ expected_len = stack_trace_snprint(expected, buf_size, entries,
+ nr_entries, 2);
+ actual_len = stack_depot_snprint(handle, actual, buf_size, 2);
+ KUNIT_EXPECT_EQ(test, actual_len, (int)expected_len);
+ KUNIT_EXPECT_STREQ(test, actual, expected);
+
+ prefix_entries = nr_entries / 2 + 1;
+ prefix_len = stack_trace_snprint(expected, buf_size, entries,
+ prefix_entries, 2);
+ KUNIT_ASSERT_LE(test, (size_t)prefix_len + 2, buf_size);
+ output_size = prefix_len + 2;
+ memset(expected, 0, buf_size);
+ memset(actual, 0, buf_size);
+ expected_len = stack_trace_snprint(expected, output_size, entries,
+ nr_entries, 2);
+ actual_len = stack_depot_snprint(handle, actual, output_size, 2);
+ KUNIT_EXPECT_EQ(test, actual_len, (int)expected_len);
+ KUNIT_EXPECT_STREQ(test, actual, expected);
+}
+
+static void stackdepot_countable_public(struct kunit *test)
+{
+ unsigned long plain_entries[] = {
+ 0x141000UL,
+ 0x142000UL,
+ 0x143000UL,
+ };
+ unsigned long get_entries[] = {
+ 0x151000UL,
+ 0x152000UL,
+ 0x153000UL,
+ };
+ unsigned long fetched[ARRAY_SIZE(plain_entries)] = {};
+ depot_flags_t countable = STACK_DEPOT_FLAG_CAN_ALLOC |
+ STACK_DEPOT_FLAG_COUNTABLE;
+ struct stack_record *record;
+ depot_stack_handle_t count_handle;
+ depot_stack_handle_t plain_handle;
+ depot_stack_handle_t get_handle;
+ unsigned int get_nr = ARRAY_SIZE(get_entries);
+ unsigned int plain_nr = ARRAY_SIZE(plain_entries);
+ unsigned int nr_entries;
+
+ KUNIT_ASSERT_EQ(test, stack_depot_init(), 0);
+
+ plain_handle = stack_depot_save(plain_entries, plain_nr, GFP_KERNEL);
+ KUNIT_ASSERT_NE(test, plain_handle, (depot_stack_handle_t)0);
+ count_handle = stack_depot_save_flags(plain_entries, plain_nr, GFP_KERNEL,
+ countable);
+ KUNIT_ASSERT_NE(test, count_handle, (depot_stack_handle_t)0);
+ record = __stack_depot_get_stack_record(count_handle);
+ KUNIT_ASSERT_NOT_NULL(test, record);
+ KUNIT_EXPECT_EQ(test, record->size, (u16)plain_nr);
+ KUNIT_EXPECT_MEMEQ(test, record->entries, plain_entries,
+ sizeof(plain_entries));
+ nr_entries = stack_depot_fetch_into(count_handle, fetched,
+ ARRAY_SIZE(fetched));
+ KUNIT_EXPECT_EQ(test, nr_entries, plain_nr);
+ KUNIT_EXPECT_MEMEQ(test, fetched, plain_entries, sizeof(plain_entries));
+
+ get_handle = stack_depot_save_flags(get_entries, get_nr, GFP_KERNEL,
+ STACK_DEPOT_FLAG_CAN_ALLOC |
+ STACK_DEPOT_FLAG_GET);
+ KUNIT_ASSERT_NE(test, get_handle, (depot_stack_handle_t)0);
+ count_handle = stack_depot_save_flags(get_entries, get_nr, GFP_KERNEL,
+ countable);
+ KUNIT_ASSERT_NE(test, count_handle, (depot_stack_handle_t)0);
+ record = __stack_depot_get_stack_record(count_handle);
+ KUNIT_ASSERT_NOT_NULL(test, record);
+ KUNIT_EXPECT_MEMEQ(test, record->entries, get_entries, sizeof(get_entries));
+
+ stack_depot_put(get_handle);
+}
+
+static void stackdepot_fetch_into_roundtrip(struct kunit *test)
+{
+ unsigned long entries[] = {
+ 0x101000UL,
+ 0x102000UL,
+ 0x103000UL,
+ };
+ unsigned long exact[ARRAY_SIZE(entries)] = {};
+ unsigned long fetched[ARRAY_SIZE(entries) + 1] = {
+ [ARRAY_SIZE(entries)] = 0xa5a5a5a5UL,
+ };
+ unsigned long expected_tail = fetched[ARRAY_SIZE(entries)];
+ depot_stack_handle_t handle;
+ unsigned int nr_entries;
+
+ KUNIT_ASSERT_EQ(test, stack_depot_init(), 0);
+
+ handle = stack_depot_save(entries, ARRAY_SIZE(entries), GFP_KERNEL);
+ KUNIT_ASSERT_NE(test, handle, (depot_stack_handle_t)0);
+
+ nr_entries = stack_depot_fetch_into(handle, exact, ARRAY_SIZE(exact));
+ KUNIT_EXPECT_EQ(test, nr_entries, (unsigned int)ARRAY_SIZE(entries));
+ KUNIT_EXPECT_MEMEQ(test, exact, entries, sizeof(entries));
+
+ nr_entries = stack_depot_fetch_into(handle, fetched, ARRAY_SIZE(fetched));
+ KUNIT_EXPECT_EQ(test, nr_entries, (unsigned int)ARRAY_SIZE(entries));
+ KUNIT_EXPECT_MEMEQ(test, fetched, entries, sizeof(entries));
+ KUNIT_EXPECT_EQ(test, fetched[ARRAY_SIZE(entries)], expected_tail);
+}
+
+static void stackdepot_fetch_into_rejects_missing_or_short_stack(struct kunit *test)
+{
+ unsigned long entries[] = {
+ 0x111000UL,
+ 0x112000UL,
+ 0x113000UL,
+ };
+ unsigned long fetched[ARRAY_SIZE(entries)] = {
+ 0xa1a1a1a1UL,
+ 0xb2b2b2b2UL,
+ 0xc3c3c3c3UL,
+ };
+ unsigned long expected[ARRAY_SIZE(fetched)];
+ depot_stack_handle_t handle;
+ unsigned int nr_entries;
+
+ KUNIT_ASSERT_EQ(test, stack_depot_init(), 0);
+
+ handle = stack_depot_save(entries, ARRAY_SIZE(entries), GFP_KERNEL);
+ KUNIT_ASSERT_NE(test, handle, (depot_stack_handle_t)0);
+ memcpy(expected, fetched, sizeof(expected));
+
+ nr_entries = stack_depot_fetch_into(0, fetched, ARRAY_SIZE(fetched));
+ KUNIT_EXPECT_EQ(test, nr_entries, 0U);
+ KUNIT_EXPECT_MEMEQ(test, fetched, expected, sizeof(expected));
+
+ nr_entries = stack_depot_fetch_into(0, NULL, 0);
+ KUNIT_EXPECT_EQ(test, nr_entries, 0U);
+
+ nr_entries = stack_depot_fetch_into(handle, fetched,
+ ARRAY_SIZE(fetched) - 1);
+ KUNIT_EXPECT_EQ(test, nr_entries, 0U);
+ KUNIT_EXPECT_MEMEQ(test, fetched, expected, sizeof(expected));
+}
+
+static void stackdepot_trie_topology_roundtrip(struct kunit *test,
+ bool constrained)
+{
+ union handle_parts parts;
+ unsigned long seed[] = { 0x191000UL, 0x192000UL };
+ unsigned long stacks[][3] = {
+ { 0x201000UL, 0x202000UL },
+ { 0x201000UL, 0x203000UL },
+ { 0x201000UL },
+ { 0x201000UL, 0x203000UL, 0x204000UL },
+ { 0x201000UL, 0x205000UL },
+ { 0x201000UL, 0x204000UL },
+ { 0x201000UL, 0x206000UL },
+ { 0x201000UL, 0x207000UL },
+ { 0x301000UL, 0x302000UL },
+ { 0x301000UL, 0x302000UL, 0x303000UL },
+ { 0x301000UL, 0x304000UL },
+ { 0x401000UL, 0x402000UL, 0x403000UL },
+ { 0x401000UL, 0x402000UL },
+ };
+ unsigned int nr_entries[] = { 2, 2, 1, 3, 2, 2, 2, 2, 2, 3, 2, 3, 2 };
+ depot_stack_handle_t handles[ARRAY_SIZE(stacks)];
+ depot_stack_handle_t seed_handle;
+ unsigned long fetched[ARRAY_SIZE(stacks[0])];
+ gfp_t no_spin = GFP_NOWAIT & ~__GFP_RECLAIM;
+ u32 pool_index_plus_1;
+ unsigned int j;
+ unsigned int i;
+
+ if (expected_trie_pool_limit < 0)
+ kunit_skip(test, "trie pool limit was not provided");
+ KUNIT_ASSERT_EQ(test, stack_depot_init(), 0);
+ if (constrained) {
+ seed_handle = stack_depot_save(seed, ARRAY_SIZE(seed), GFP_KERNEL);
+ KUNIT_ASSERT_NE(test, seed_handle, (depot_stack_handle_t)0);
+ for (i = 0; i < ARRAY_SIZE(stacks); i++)
+ for (j = 0; j < nr_entries[i]; j++)
+ stacks[i][j] += 0x10000000UL;
+ }
+
+ for (i = 0; i < ARRAY_SIZE(stacks); i++) {
+ if (constrained)
+ handles[i] = stack_depot_save_flags(stacks[i], nr_entries[i],
+ GFP_KERNEL, 0);
+ else
+ handles[i] = stack_depot_save(stacks[i], nr_entries[i],
+ GFP_KERNEL);
+ KUNIT_ASSERT_NE(test, handles[i], (depot_stack_handle_t)0);
+ }
+ parts.handle = handles[0];
+ pool_index_plus_1 = parts.pool_index_plus_1;
+ KUNIT_ASSERT_GT(test, pool_index_plus_1,
+ (u32)expected_trie_pool_limit);
+
+ for (i = 0; i < ARRAY_SIZE(stacks); i++) {
+ memset(fetched, 0, sizeof(fetched));
+ KUNIT_EXPECT_EQ(test,
+ stack_depot_fetch_into(handles[i], fetched,
+ ARRAY_SIZE(fetched)),
+ nr_entries[i]);
+ KUNIT_EXPECT_MEMEQ(test, fetched, stacks[i],
+ nr_entries[i] * sizeof(fetched[0]));
+ if (constrained)
+ KUNIT_EXPECT_EQ(test,
+ stack_depot_save_flags(stacks[i], nr_entries[i],
+ no_spin, 0),
+ handles[i]);
+ else
+ KUNIT_EXPECT_EQ(test,
+ stack_depot_save(stacks[i], nr_entries[i],
+ GFP_KERNEL),
+ handles[i]);
+ }
+}
+
+static void stackdepot_trie_topology_allocating(struct kunit *test)
+{
+ stackdepot_trie_topology_roundtrip(test, false);
+}
+
+static void stackdepot_trie_topology_constrained(struct kunit *test)
+{
+ stackdepot_trie_topology_roundtrip(test, true);
+}
+
+static void stackdepot_frame_storage_roundtrip(struct kunit *test)
+{
+ union handle_parts parts;
+ unsigned long fetched[3] = {};
+ depot_stack_handle_t handle;
+ u32 pool_index_plus_1;
+ unsigned int nr_entries;
+#if defined(CONFIG_ARM64)
+ unsigned long entries[] = {
+ stackdepot_arm64_frame(S32_MIN),
+ 0x1000UL,
+ stackdepot_arm64_frame(S32_MAX),
+ };
+#elif defined(CONFIG_X86_64)
+ unsigned long entries[] = {
+ 0xffffffff10001000UL,
+ 0xffff888000001000UL,
+ 0xffffffff20002000UL,
+ };
+#else
+ unsigned long entries[] = { 0x301000UL, 0x302000UL, 0x303000UL };
+#endif
+
+ if (expected_trie_pool_limit < 0)
+ kunit_skip(test, "trie pool limit was not provided");
+ KUNIT_ASSERT_EQ(test, stack_depot_init(), 0);
+ handle = stack_depot_save(entries, ARRAY_SIZE(entries), GFP_KERNEL);
+ KUNIT_ASSERT_NE(test, handle, (depot_stack_handle_t)0);
+ parts.handle = handle;
+ pool_index_plus_1 = parts.pool_index_plus_1;
+ KUNIT_ASSERT_GT(test, pool_index_plus_1,
+ (u32)expected_trie_pool_limit);
+
+ nr_entries = stack_depot_fetch_into(handle, fetched, ARRAY_SIZE(fetched));
+ KUNIT_EXPECT_EQ(test, nr_entries, (unsigned int)ARRAY_SIZE(entries));
+ KUNIT_EXPECT_MEMEQ(test, fetched, entries, sizeof(entries));
+}
+
+static void stackdepot_frame_raw_fallback(struct kunit *test)
+{
+ unsigned long frame = 0x1000UL;
+ bool compressed;
+ u32 payload;
+
+#ifdef CONFIG_ARM64
+ frame = (unsigned long)_text + (unsigned long)S32_MAX + 1UL;
+#endif
+
+ compressed = arch_stack_depot_frame_try_compress(frame, &payload);
+ KUNIT_EXPECT_FALSE(test, compressed);
+}
+
+#if defined(CONFIG_X86_64) && !defined(CONFIG_UML)
+static void stackdepot_frame_x86_64(struct kunit *test)
+{
+ unsigned long direct_map = 0xffff888000001000UL;
+ unsigned long frame = 0xffffffff81234567UL;
+ unsigned long out;
+ bool compressed;
+ u32 low;
+
+ compressed = arch_stack_depot_frame_try_compress(frame, &low);
+ KUNIT_EXPECT_TRUE(test, compressed);
+ KUNIT_EXPECT_EQ(test, low, (u32)0x81234567);
+ arch_stack_depot_frame_decompress(low, &out);
+ KUNIT_EXPECT_EQ(test, out, frame);
+
+ compressed = arch_stack_depot_frame_try_compress(direct_map, &low);
+ KUNIT_EXPECT_FALSE(test, compressed);
+}
+#endif /* CONFIG_X86_64 && !CONFIG_UML */
+
+#ifdef CONFIG_ARM64
+static void stackdepot_frame_arm64(struct kunit *test)
+{
+ long negative_offset = S32_MIN;
+ long positive_offset = S32_MAX;
+ long offset = 0x123456;
+ unsigned long frame = stackdepot_arm64_frame(offset);
+ unsigned long out;
+ bool compressed;
+ u32 payload;
+
+ compressed = arch_stack_depot_frame_try_compress(frame, &payload);
+ KUNIT_EXPECT_TRUE(test, compressed);
+ KUNIT_EXPECT_EQ(test, payload, (u32)(s32)offset);
+ arch_stack_depot_frame_decompress(payload, &out);
+ KUNIT_EXPECT_EQ(test, out, frame);
+
+ frame = stackdepot_arm64_frame(negative_offset);
+ compressed = arch_stack_depot_frame_try_compress(frame, &payload);
+ KUNIT_EXPECT_TRUE(test, compressed);
+ KUNIT_EXPECT_EQ(test, payload, (u32)(s32)negative_offset);
+ arch_stack_depot_frame_decompress(payload, &out);
+ KUNIT_EXPECT_EQ(test, out, frame);
+
+ frame = stackdepot_arm64_frame(positive_offset);
+ compressed = arch_stack_depot_frame_try_compress(frame, &payload);
+ KUNIT_EXPECT_TRUE(test, compressed);
+ KUNIT_EXPECT_EQ(test, payload, (u32)(s32)positive_offset);
+ arch_stack_depot_frame_decompress(payload, &out);
+ KUNIT_EXPECT_EQ(test, out, frame);
+}
+#endif /* CONFIG_ARM64 */
+
+static struct kunit_case stackdepot_test_cases[] = {
+ KUNIT_CASE(stackdepot_trie_max_path_roundtrip),
+ KUNIT_CASE(stackdepot_save_flags_public),
+ KUNIT_CASE(stackdepot_snprint_public),
+ KUNIT_CASE(stackdepot_countable_public),
+ KUNIT_CASE(stackdepot_fetch_into_roundtrip),
+ KUNIT_CASE(stackdepot_fetch_into_rejects_missing_or_short_stack),
+ KUNIT_CASE(stackdepot_trie_topology_allocating),
+ KUNIT_CASE(stackdepot_trie_topology_constrained),
+ KUNIT_CASE(stackdepot_frame_storage_roundtrip),
+ KUNIT_CASE(stackdepot_frame_raw_fallback),
+#if defined(CONFIG_X86_64) && !defined(CONFIG_UML)
+ KUNIT_CASE(stackdepot_frame_x86_64),
+#endif
+#ifdef CONFIG_ARM64
+ KUNIT_CASE(stackdepot_frame_arm64),
+#endif
+ {}
+};
+
+static struct kunit_suite stackdepot_test_suite = {
+ .name = "stackdepot",
+ .test_cases = stackdepot_test_cases,
+};
+
+kunit_test_suite(stackdepot_test_suite);
+
+MODULE_DESCRIPTION("KUnit tests for stack depot");
+MODULE_AUTHOR("Caleb Kan <ckan@cloudflare.com>");
+MODULE_LICENSE("GPL");
diff --git a/mm/kmemleak.c b/mm/kmemleak.c
index 8fa409a4f9fb2..c42741a88bd42 100644
--- a/mm/kmemleak.c
+++ b/mm/kmemleak.c
@@ -378,10 +378,10 @@ static void __print_unreferenced(struct seq_file *seq,
bool hex_dump)
{
int i;
- unsigned long *entries;
+ unsigned long entries[MAX_TRACE];
unsigned int nr_entries;
- nr_entries = stack_depot_fetch(object->trace_handle, &entries);
+ nr_entries = stack_depot_fetch_into(object->trace_handle, entries, ARRAY_SIZE(entries));
warn_or_seq_printf(seq, "unreferenced object%s 0x%08lx (size %zu):\n",
__object_type_str(object),
object->pointer, object->size);
diff --git a/mm/kmsan/kmsan_test.c b/mm/kmsan/kmsan_test.c
index 31f47cc4dab40..7c04e4b21873d 100644
--- a/mm/kmsan/kmsan_test.c
+++ b/mm/kmsan/kmsan_test.c
@@ -669,7 +669,7 @@ static void test_long_origin_chain(struct kunit *test)
*/
static void test_stackdepot_roundtrip(struct kunit *test)
{
- unsigned long src_entries[16], *dst_entries;
+ unsigned long src_entries[16], dst_entries[16];
unsigned int src_nentries, dst_nentries;
EXPECTATION_NO_REPORT(expect);
depot_stack_handle_t handle;
@@ -680,7 +680,7 @@ static void test_stackdepot_roundtrip(struct kunit *test)
stack_trace_save(src_entries, ARRAY_SIZE(src_entries), 1);
handle = stack_depot_save(src_entries, src_nentries, GFP_KERNEL);
stack_depot_print(handle);
- dst_nentries = stack_depot_fetch(handle, &dst_entries);
+ dst_nentries = stack_depot_fetch_into(handle, dst_entries, ARRAY_SIZE(dst_entries));
KUNIT_EXPECT_TRUE(test, src_nentries == dst_nentries);
kmsan_check_memory((void *)dst_entries,
diff --git a/mm/kmsan/report.c b/mm/kmsan/report.c
index d6853ce089541..0770658ba932e 100644
--- a/mm/kmsan/report.c
+++ b/mm/kmsan/report.c
@@ -83,9 +83,9 @@ static char *pretty_descr(char *descr)
return report_local_descr;
}
-void kmsan_print_origin(depot_stack_handle_t origin)
+static void kmsan_print_origin_with_buf(depot_stack_handle_t origin,
+ unsigned long *entries)
{
- unsigned long *entries = NULL, *chained_entries = NULL;
unsigned int nr_entries, chained_nr_entries, skipnr;
void *pc1 = NULL, *pc2 = NULL;
depot_stack_handle_t head;
@@ -97,7 +97,8 @@ void kmsan_print_origin(depot_stack_handle_t origin)
return;
while (true) {
- nr_entries = stack_depot_fetch(origin, &entries);
+ nr_entries =
+ stack_depot_fetch_into(origin, entries, KMSAN_STACK_DEPTH);
depth = kmsan_depth_from_eb(stack_depot_get_extra_bits(origin));
magic = nr_entries ? entries[0] : 0;
if ((nr_entries == 4) && (magic == KMSAN_ALLOCA_MAGIC_ORIGIN)) {
@@ -123,14 +124,10 @@ void kmsan_print_origin(depot_stack_handle_t origin)
origin = entries[2];
pr_err("Uninit was stored to memory at:\n");
chained_nr_entries =
- stack_depot_fetch(head, &chained_entries);
- kmsan_internal_unpoison_memory(
- chained_entries,
- chained_nr_entries * sizeof(*chained_entries),
- /*checked*/ false);
- skipnr = get_stack_skipnr(chained_entries,
- chained_nr_entries);
- stack_trace_print(chained_entries + skipnr,
+ stack_depot_fetch_into(head, entries,
+ KMSAN_STACK_DEPTH);
+ skipnr = get_stack_skipnr(entries, chained_nr_entries);
+ stack_trace_print(entries + skipnr,
chained_nr_entries - skipnr, 0);
pr_err("\n");
continue;
@@ -147,6 +144,13 @@ void kmsan_print_origin(depot_stack_handle_t origin)
}
}
+void kmsan_print_origin(depot_stack_handle_t origin)
+{
+ unsigned long entries[KMSAN_STACK_DEPTH];
+
+ kmsan_print_origin_with_buf(origin, entries);
+}
+
void kmsan_report(depot_stack_handle_t origin, void *address, int size,
int off_first, int off_last, const void __user *user_addr,
enum kmsan_bug_reason reason)
@@ -193,7 +197,7 @@ void kmsan_report(depot_stack_handle_t origin, void *address, int size,
0);
pr_err("\n");
- kmsan_print_origin(origin);
+ kmsan_print_origin_with_buf(origin, stack_entries);
if (size) {
pr_err("\n");
diff --git a/mm/page_owner.c b/mm/page_owner.c
index cfc31c92d7657..1fb1998bc129e 100644
--- a/mm/page_owner.c
+++ b/mm/page_owner.c
@@ -119,7 +119,8 @@ static __always_inline depot_stack_handle_t create_dummy_stack(void)
unsigned int nr_entries;
nr_entries = stack_trace_save(entries, ARRAY_SIZE(entries), 0);
- return stack_depot_save(entries, nr_entries, GFP_KERNEL);
+ return stack_depot_save_flags(entries, nr_entries, GFP_KERNEL,
+ STACK_DEPOT_FLAG_CAN_ALLOC | STACK_DEPOT_FLAG_COUNTABLE);
}
static noinline void register_dummy_stack(void)
@@ -181,7 +182,8 @@ static noinline depot_stack_handle_t save_stack(gfp_t flags)
set_current_in_page_owner();
nr_entries = stack_trace_save(entries, ARRAY_SIZE(entries), 2);
- handle = stack_depot_save(entries, nr_entries, flags);
+ handle = stack_depot_save_flags(entries, nr_entries, flags,
+ STACK_DEPOT_FLAG_CAN_ALLOC | STACK_DEPOT_FLAG_COUNTABLE);
if (!handle)
handle = failure_handle;
unset_current_in_page_owner();
diff --git a/mm/slub.c b/mm/slub.c
index f9b56cb439e70..4aa1c5a457182 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -8198,12 +8198,12 @@ void __kmem_obj_info(struct kmem_obj_info *kpp, void *object, struct slab *slab)
#ifdef CONFIG_STACKDEPOT
{
depot_stack_handle_t handle;
- unsigned long *entries;
+ unsigned long entries[TRACK_ADDRS_COUNT];
unsigned int nr_entries;
handle = READ_ONCE(trackp->handle);
if (handle) {
- nr_entries = stack_depot_fetch(handle, &entries);
+ nr_entries = stack_depot_fetch_into(handle, entries, ARRAY_SIZE(entries));
for (i = 0; i < KS_ADDRS_COUNT && i < nr_entries; i++)
kpp->kp_stack[i] = (void *)entries[i];
}
@@ -8211,7 +8211,7 @@ void __kmem_obj_info(struct kmem_obj_info *kpp, void *object, struct slab *slab)
trackp = get_track(s, objp, TRACK_FREE);
handle = READ_ONCE(trackp->handle);
if (handle) {
- nr_entries = stack_depot_fetch(handle, &entries);
+ nr_entries = stack_depot_fetch_into(handle, entries, ARRAY_SIZE(entries));
for (i = 0; i < KS_ADDRS_COUNT && i < nr_entries; i++)
kpp->kp_free_stack[i] = (void *)entries[i];
}
@@ -9946,12 +9946,14 @@ static int slab_debugfs_show(struct seq_file *seq, void *v)
#ifdef CONFIG_STACKDEPOT
{
depot_stack_handle_t handle;
- unsigned long *entries;
+ unsigned long entries[TRACK_ADDRS_COUNT];
unsigned int nr_entries, j;
handle = READ_ONCE(l->handle);
if (handle) {
- nr_entries = stack_depot_fetch(handle, &entries);
+ nr_entries =
+ stack_depot_fetch_into(handle, entries,
+ ARRAY_SIZE(entries));
seq_puts(seq, "\n");
for (j = 0; j < nr_entries; j++)
seq_printf(seq, " %pS\n", (void *)entries[j]);
diff --git a/scripts/gdb/linux/stackdepot.py b/scripts/gdb/linux/stackdepot.py
index 37313a5a51a0d..82aeb9f532c3d 100644
--- a/scripts/gdb/linux/stackdepot.py
+++ b/scripts/gdb/linux/stackdepot.py
@@ -37,6 +37,10 @@ def stack_depot_fetch(handle):
if handle == 0:
raise gdb.GdbError("handle is 0\n")
+ stack_max_pools = gdb.parse_and_eval('stack_max_pools')
+ if parts['pool_index_plus_1'] > stack_max_pools:
+ raise gdb.GdbError("trie-backed stack depot handles are not supported\n")
+
pool_index = parts['pool_index_plus_1'] - 1
if pool_index >= pools_num:
gdb.write("pool index %d out of bounds (%d) for stack id 0x%08x\n" % (parts['pool_index'], pools_num, handle))
|
| 17/3 | 2026/09/08 15:27 | llm | kmsan-evaluator |
0mModel:gemini-3.7-flash Args: null Results: null Tokens: input: 40864 output: 415 thoughts: 3146 Thoughts: **KMSAN Assessment: Stack Depot Trie Backend** Alright, let's break this down. We've got a patch series introducing a trie backend for `stackdepot`, and the request is whether this warrants a dedicated KMSAN fuzzing session, in addition to our standard KASAN efforts. My expert brain says, let's dive in deep. Firstly, the patch itself: It's a pretty significant internal change, a storage backend update for stack traces. It involves modifications to various files – the core `stackdepot` code, architecture-specific includes, and the KMSAN report file to utilize `stack_depot_fetch_into`. This new `stackdepot` trie implementation adds trie compression and RCU child containers and all internal memory handling. The request is specifically for the amd64 architecture. Now, KMSAN vs. KASAN is the core issue. KASAN excels at catching memory safety bugs – use-after-free, out-of-bounds reads/writes, double frees, etc. KMSAN is the uninitialized memory detective: reads from uninitialized stack, heap, and allocation pages, and also kernel-to-user memory info-leaks are within its purview. The rule of thumb I always go by is: *Don't use KMSAN if KASAN, LOCKDEP, or standard detectors will do the trick*. KMSAN is expensive in resources. It's only for the cases where *uninitialized memory risks* are specifically introduced or exposed and are missed by the other tools. So, let's see if this patch triggers any of those KMSAN-specific conditions. The four red flags: 1) Kernel structures sent to user space where uninitialized fields could leak. 2) Conditional logic/branching depending on uninitialized variables. 3) Allocation/initialization of complex data structures with uninitialized fields. 4) Bounds checks that allow access to uninitialized bytes of existing buffers. Analyzing the code changes, there are a few areas we need to review: 1. Changes to KMSAN files: `test_stackdepot_roundtrip` now uses the `fetch_into` variant. The report code now uses `stack_depot_fetch_into` function in `kmsan_print_origin` and the code explicitly unpoisons fetched memory. 2. New Data Structures: We have a bunch of trie-related structs added in `lib/stackdepot.c` like `stack_depot_trie_node`, `stack_depot_trie_children`, etc. Now we need to go deep and check for specific risk: The `trie_node_init` function, in particular, looks like it might be a risk, but it does `memcpy` with size and `arch_stack_depot_frame_try_compress`, and everything gets initialized. The `trie_children_init` looks safe. The `trie_fetch_into` and the function calling `kmsan_unpoison_memory` are correct. The KMSAN function unpoisons what we copy. Let's look at the callers: 1. `drm_modeset_lock.c`: The change just uses `stack_depot_snprint`, which will null-terminate, so no issue there. 2. `kmemleak.c`: It uses `stack_depot_fetch_into`, but the loop is range-checked with `nr_entries`! 3. `slub.c`: Same story, the usage is safely bounded by `nr_entries`. 4. `mm/kmsan/report.c`: Also the same pattern, bounded by `nr_entries`. 5. `lib/stackdepot.c`: Internal allocations zero the memory from `get_zeroed_page`, `kvzalloc`, or `memblock_alloc`. `stack_depot_trie_init` with `kvzalloc` is zeroed. So, is anything leaking to userspace? No. The trie is for internal stack trace management, not for user-space data. No data is being copied, sent, or leaked out to userspace. All allocations within `stackdepot` itself either are zeroed initially, have all fields set, or are initialized before any read. Concurrency issues, out-of-bounds, etc. are KASAN/KCSAN territory. We need to double-check: 1. No kernel structures sent to userspace with uninitialized fields. No. 2. No conditional logic based on uninitialized variables. No. 3. No allocation of structures where fields are read uninitialized. No. 4. No bounds checks that allow reading uninitialized data. No. |
| 18/3 | 2026/09/08 15:28 | tool | set-results |
0mArgs:{
"KMSANReasoning": "The patch series implements a trie storage backend and address compression for the stack depot subsystem (lib/stackdepot.c), updates stack depot consumers (such as kmemleak, SLUB debug, DRM, and KMSAN's origin printer) to use the new `stack_depot_fetch_into()` / `stack_depot_snprint()` APIs, and adds KUnit tests.\n\n1. KASAN / Lockdep / Standard Debuggers Applicability:\n- The changes heavily involve internal kernel data structure management, including RCU-protected pointer updates, bitmap slot reservations within memory pools, spinlocks, and buffer copies for stack frames.\n- Potential defects such as out-of-bounds indexing in child arrays/bitmaps, memory corruption, use-after-free during node splitting/reparenting, and concurrency/locking deadlocks are fully covered by KASAN, KCSAN, and LOCKDEP.\n\n2. KMSAN Applicability:\n- No kernel structures are copied or exposed to userspace (no copy_to_user, put_user, netlink, ioctl, etc.).\n- All internal allocations (directory pages, side tables, node arrays, pools) are allocated using zeroing allocators (`get_zeroed_page()`, `kvzalloc()`, `memblock_alloc()`) or are explicitly populated before being read or linked into the trie.\n- Callers of `stack_depot_fetch_into()` only access entries up to the returned `nr_entries` count, and stack depot explicitly invokes `kmsan_unpoison_memory()` when populating frame entries.\n- No uninitialized variables or uninitialized branching conditions are introduced.\n\nBecause the changes present memory safety and synchronization concerns rather than uninitialized memory access or info-leak risks, a dedicated KMSAN fuzzing session is not justified.",
"NeedsKMSAN": false
}Results: {
"KMSANReasoning": "The patch series implements a trie storage backend and address compression for the stack depot subsystem (lib/stackdepot.c), updates stack depot consumers (such as kmemleak, SLUB debug, DRM, and KMSAN's origin printer) to use the new `stack_depot_fetch_into()` / `stack_depot_snprint()` APIs, and adds KUnit tests.\n\n1. KASAN / Lockdep / Standard Debuggers Applicability:\n- The changes heavily involve internal kernel data structure management, including RCU-protected pointer updates, bitmap slot reservations within memory pools, spinlocks, and buffer copies for stack frames.\n- Potential defects such as out-of-bounds indexing in child arrays/bitmaps, memory corruption, use-after-free during node splitting/reparenting, and concurrency/locking deadlocks are fully covered by KASAN, KCSAN, and LOCKDEP.\n\n2. KMSAN Applicability:\n- No kernel structures are copied or exposed to userspace (no copy_to_user, put_user, netlink, ioctl, etc.).\n- All internal allocations (directory pages, side tables, node arrays, pools) are allocated using zeroing allocators (`get_zeroed_page()`, `kvzalloc()`, `memblock_alloc()`) or are explicitly populated before being read or linked into the trie.\n- Callers of `stack_depot_fetch_into()` only access entries up to the returned `nr_entries` count, and stack depot explicitly invokes `kmsan_unpoison_memory()` when populating frame entries.\n- No uninitialized variables or uninitialized branching conditions are introduced.\n\nBecause the changes present memory safety and synchronization concerns rather than uninitialized memory access or info-leak risks, a dedicated KMSAN fuzzing session is not justified.",
"NeedsKMSAN": false
} |
| Total Calls | Total Tokens | Avg Tokens | Total Duration (Seconds) | Avg Duration (Seconds) |
|---|
| Total Calls | Total Duration (Seconds) | Avg Duration (Seconds) |
|---|