| Seq | Timestamp | Type | Name | Duration |
|---|---|---|---|---|
| 0/0 | 2026/09/24 04:18 | flow | patch-triage |
4mArgs:null Results: {
"EnableConfigs": [
"TASKS_RCU_TRAMPOLINE_READERS",
"HAVE_RCU_TRAMPOLINE_READERS",
"TASKS_RCU",
"DYNAMIC_FTRACE",
"DYNAMIC_FTRACE_WITH_DIRECT_CALLS",
"KPROBES",
"OPTPROBES",
"BPF_SYSCALL",
"BPF_JIT",
"SMP",
"PREEMPTION"
],
"FocusSymbols": [
"rcu_tasks_tramp_wait_gp",
"rcu_tasks_irq_resched_enter",
"rcu_tasks_irq_resched_exit",
"rcu_tasks_note_qs",
"rcu_tasks_wait_irq_preempted",
"rcu_tasks_trampoline_text",
"arch_rcu_tasks_trampoline_text",
"raw_irqentry_exit_cond_resched",
"kprobe_in_optimized_region",
"arch_prepare_bpf_trampoline",
"register_ftrace_direct",
"kprobe_optimizer"
],
"KMSANReasoning": "The patch series implements trampoline-reader Tasks RCU (CONFIG_TASKS_RCU_TRAMPOLINE_READERS), which replaces the classic voluntary-context-switch Tasks RCU grace period mechanism with Tasks Trace RCU read-side critical sections around ftrace, BPF, and kprobe trampolines and checks on irq-exit preemption.\n\nKey observations:\n1. No uninitialized memory usage or exposure: All new/modified fields in core structures (such as `task_struct.rcu_tasks_irq_ip` and `module.ftrace_direct_tramp`) are explicitly initialized on task creation in `rcu_copy_process()` and in zero-allocated module structs. Global and per-CPU state variables are statically defined and zero-initialized.\n2. No data copied or leaked to user space: The changes do not touch `copy_to_user`, `put_user`, netlink, socket options, or ioctl outputs.\n3. Nature of bugs and applicable detectors: Any bugs introduced by this series would manifest as synchronization races, premature grace period completion leading to use-after-free of freed trampoline text or metadata, or locking/deadlock issues. These failure modes are strictly the domain of KASAN (for use-after-free and memory corruption) and LOCKDEP (for locking violations), not KMSAN.\n\nTherefore, dedicated KMSAN fuzzing is not warranted.",
"NeedsKMSAN": false,
"Reasoning": "The patch series introduces CONFIG_TASKS_RCU_TRAMPOLINE_READERS and CONFIG_HAVE_RCU_TRAMPOLINE_READERS, restructuring how Tasks RCU grace periods and quiescent states are tracked on x86_64 and arm64. It embeds Tasks Trace RCU reader state transitions inside ftrace, optprobe, and BPF trampolines, hooks into kernel irq-exit preemption paths (raw_irqentry_exit_cond_resched) to monitor tasks preempted in unmarked trampoline text, adds synchronization in kprobe_optimizer, and introduces runtime assertions like rcu_tasks_trace_assert_idle(). This fundamentally alters core concurrency and synchronization primitives in reachable paths on amd64, warranting fuzzing.",
"WorthFuzzing": true
} |
| 1/1 | 2026/09/24 04:18 | action | read-patch-diff |
0mArgs:null Results: {
"PatchDiff": "commit 671c17ef89b17f280a59ec7bb4ed9bfe9789cf90\nAuthor: syz-cluster \u003ctriage@syzkaller.com\u003e\nDate: Thu Sep 24 04:18:56 2026 +0000\n\n syz-cluster: applied patch under review\n\ndiff --git a/Documentation/RCU/Design/Requirements/Requirements.rst b/Documentation/RCU/Design/Requirements/Requirements.rst\nindex 8101fe6229d57..c059c463e083f 100644\n--- a/Documentation/RCU/Design/Requirements/Requirements.rst\n+++ b/Documentation/RCU/Design/Requirements/Requirements.rst\n@@ -2756,6 +2756,30 @@ synchronize_rcu(), and rcu_barrier(), respectively. In\n three APIs are therefore implemented by separate functions that check\n for voluntary context switches.\n \n+Architectures that select ``CONFIG_HAVE_RCU_TRAMPOLINE_READERS`` keep the\n+same three APIs but implement the grace period differently\n+(``CONFIG_TASKS_RCU_TRAMPOLINE_READERS``). There, every trampoline whose\n+lifetime Tasks RCU guards enters a Tasks Trace RCU read-side critical\n+section (rcu_read_lock_trace() or its assembly equivalent) before calling\n+out and leaves it before returning, so a task anywhere inside such a\n+call-out is an ordinary Tasks Trace reader whether or not it is\n+preempted. The few trampoline instructions outside that reader can only\n+be occupied by a task that was interrupted there, so the grace period\n+additionally waits for each CPU to pass through a context switch, and the\n+irq-exit preemption path, the only switch that can catch a task inside\n+such text (rcu_tasks_trampoline_text()), briefly makes such a task a\n+holdout until it is next seen elsewhere. On such kernels an involuntary\n+context switch outside trampoline text *is* a Tasks-RCU quiescent state,\n+a Tasks RCU grace period no longer depends on how long any task runs\n+without sleeping, cond_resched_tasks_rcu_qs() is unnecessary, and the\n+obligation moves to the trampolines: anything that relies on\n+synchronize_rcu_tasks() to protect code a task may be preempted in must\n+take the Tasks Trace reader (see register_ftrace_direct()), or, where\n+that is impossible because the code is ordinary text with no trampoline\n+of its own, make it known to rcu_tasks_trampoline_text() and wait out\n+tasks already parked there with rcu_tasks_wait_irq_preempted(), as the\n+kprobe jump optimizer does.\n+\n Tasks Rude RCU\n ~~~~~~~~~~~~~~\n \ndiff --git a/Documentation/RCU/checklist.rst b/Documentation/RCU/checklist.rst\nindex 4b30f701225fd..87dd506bffefb 100644\n--- a/Documentation/RCU/checklist.rst\n+++ b/Documentation/RCU/checklist.rst\n@@ -252,7 +252,12 @@ over a rather long period of time, but improvements are always welcome!\n \ta.\tIf the updater uses synchronize_rcu_tasks() or\n \t\tcall_rcu_tasks(), then the readers must refrain from\n \t\texecuting voluntary context switches, that is, from\n-\t\tblocking.\n+\t\tblocking. On architectures that select\n+\t\tCONFIG_HAVE_RCU_TRAMPOLINE_READERS a reader must in\n+\t\taddition be a Tasks Trace RCU reader (that is what the\n+\t\ttrampolines there do around their call-outs) or be text\n+\t\tthat rcu_tasks_trampoline_text() recognises; an arbitrary\n+\t\tstretch of preemptible kernel code is not protected.\n \n \tb.\tIf the updater uses call_rcu_tasks_trace()\n \t\tor synchronize_rcu_tasks_trace(), then the\ndiff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig\nindex b5a51b0ef9440..bf0e56006863e 100644\n--- a/arch/arm64/Kconfig\n+++ b/arch/arm64/Kconfig\n@@ -218,6 +218,7 @@ config ARM64\n \tselect HAVE_PERF_REGS\n \tselect HAVE_PERF_USER_STACK_DUMP\n \tselect HAVE_PREEMPT_DYNAMIC_KEY\n+\tselect HAVE_RCU_TRAMPOLINE_READERS if DYNAMIC_FTRACE_WITH_ARGS\n \tselect HAVE_REGS_AND_STACK_ACCESS_API\n \tselect HAVE_RELIABLE_STACKTRACE\n \tselect HAVE_POSIX_CPU_TIMERS_TASK_WORK\ndiff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c\nindex 9c853ed3ceab8..f6a8fb1f9b436 100644\n--- a/arch/arm64/kernel/asm-offsets.c\n+++ b/arch/arm64/kernel/asm-offsets.c\n@@ -10,6 +10,7 @@\n \n #include \u003clinux/arm_sdei.h\u003e\n #include \u003clinux/sched.h\u003e\n+#include \u003clinux/srcu.h\u003e\n #include \u003clinux/ftrace.h\u003e\n #include \u003clinux/kexec.h\u003e\n #include \u003clinux/mm.h\u003e\n@@ -39,6 +40,13 @@ int main(void)\n DEFINE(TSK_STACK,\t\toffsetof(struct task_struct, stack));\n #ifdef CONFIG_STACKPROTECTOR\n DEFINE(TSK_STACK_CANARY,\toffsetof(struct task_struct, stack_canary));\n+#endif\n+#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS\n+ DEFINE(TSK_TRC_READER_NESTING,\toffsetof(struct task_struct, trc_reader_nesting));\n+ DEFINE(TSK_TRC_READER_SCP,\toffsetof(struct task_struct, trc_reader_scp));\n+ DEFINE(SRCU_SRCU_CTRP,\toffsetof(struct srcu_struct, srcu_ctrp));\n+ DEFINE(SRCU_CTR_SRCU_LOCKS,\toffsetof(struct srcu_ctr, srcu_locks));\n+ DEFINE(SRCU_CTR_SRCU_UNLOCKS,\toffsetof(struct srcu_ctr, srcu_unlocks));\n #endif\n BLANK();\n DEFINE(THREAD_CPU_CONTEXT,\toffsetof(struct task_struct, thread.cpu_context));\ndiff --git a/arch/arm64/kernel/entry-ftrace.S b/arch/arm64/kernel/entry-ftrace.S\nindex 025140caafe74..fc2805eb9e15b 100644\n--- a/arch/arm64/kernel/entry-ftrace.S\n+++ b/arch/arm64/kernel/entry-ftrace.S\n@@ -14,6 +14,72 @@\n #include \u003casm/insn.h\u003e\n \n #ifdef CONFIG_DYNAMIC_FTRACE_WITH_ARGS\n+/*\n+ * Open-coded rcu_read_lock_trace() / rcu_read_unlock_trace(), see\n+ * include/linux/rcupdate_trace.h and CONFIG_HAVE_RCU_TRAMPOLINE_READERS. The\n+ * whole of ftrace_caller is treated as trampoline text by the irq-exit check\n+ * (see arch_rcu_tasks_trampoline_text()), so these only need to bracket the\n+ * call out to ops-\u003efunc; everything before the lock and after the unlock,\n+ * including the direct-call tails that carry a BPF trampoline address in x17,\n+ * is covered by that.\n+ *\n+ * The SRCU-fast per-CPU increment is done LL/SC on this CPU's counter; being\n+ * migrated between reading the per-CPU offset and the store-exclusive only\n+ * means another CPU's counter is (atomically) incremented, which SRCU sums\n+ * over anyway. Ordering between the nesting count and the scp stash only\n+ * matters against interrupts on this CPU, which observe program order.\n+ * Clobbers x12-x16 and the flags.\n+ */\n+\t.macro trace_rcu_srcu_inc, addr:req, tmp:req, wtmp2:req\n+8888:\tldxr\t\\tmp, [\\addr]\n+\tadd\t\\tmp, \\tmp, #1\n+\tstxr\t\\wtmp2, \\tmp, [\\addr]\n+\tcbnz\t\\wtmp2, 8888b\n+\t.endm\n+\n+\t.macro trace_rcu_read_lock\n+#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS\n+\tmrs\tx12, sp_el0\t\t\t\t// current\n+\tldr\tw13, [x12, #TSK_TRC_READER_NESTING]\n+\tadd\tw14, w13, #1\n+\tstr\tw14, [x12, #TSK_TRC_READER_NESTING]\n+\tcbnz\tw13, .Ltrl_nested\\@\t\t// interrupted a reader: done\n+\tldr_l\tx13, rcu_tasks_trace_srcu_struct + SRCU_SRCU_CTRP\n+\tstr\tx13, [x12, #TSK_TRC_READER_SCP]\n+\tget_this_cpu_offset x14\n+\tadd\tx14, x14, x13\n+\tadd\tx14, x14, #SRCU_CTR_SRCU_LOCKS\n+\ttrace_rcu_srcu_inc x14, x15, w16\n+#ifndef CONFIG_TASKS_TRACE_RCU_NO_MB\n+\tdmb\tish\n+#endif\n+.Ltrl_nested\\@:\n+#endif\n+\t.endm\n+\n+\t.macro trace_rcu_read_unlock\n+#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS\n+\tmrs\tx12, sp_el0\t\t\t\t// current\n+\tldr\tw13, [x12, #TSK_TRC_READER_NESTING]\n+\tsubs\tw13, w13, #1\n+\tb.ne\t.Ltru_nested\\@\n+\t/* Outermost: pick up scp before an interrupt can see nesting == 0. */\n+\tldr\tx14, [x12, #TSK_TRC_READER_SCP]\n+\tstr\twzr, [x12, #TSK_TRC_READER_NESTING]\n+#ifndef CONFIG_TASKS_TRACE_RCU_NO_MB\n+\tdmb\tish\n+#endif\n+\tget_this_cpu_offset x15\n+\tadd\tx14, x14, x15\n+\tadd\tx14, x14, #SRCU_CTR_SRCU_UNLOCKS\n+\ttrace_rcu_srcu_inc x14, x15, w16\n+\tb\t.Ltru_done\\@\n+.Ltru_nested\\@:\n+\tstr\tw13, [x12, #TSK_TRC_READER_NESTING]\n+.Ltru_done\\@:\n+#endif\n+\t.endm\n+\n /*\n * Due to -fpatchable-function-entry=2, the compiler has placed two NOPs before\n * the regular function prologue. For an enabled callsite, ftrace_init_nop() and\n@@ -94,6 +160,8 @@ SYM_CODE_START(ftrace_caller)\n \tstp\tx29, x30, [sp, #FREGS_SIZE]\n \tadd\tx29, sp, #FREGS_SIZE\n \n+\ttrace_rcu_read_lock\n+\n \t/* Prepare arguments for the tracer func */\n \tsub\tx0, x30, #AARCH64_INSN_SIZE\t\t// ip (callsite's BL insn)\n \tmov\tx1, x9\t\t\t\t\t// parent_ip (callsite's LR)\n@@ -111,6 +179,8 @@ SYM_INNER_LABEL(ftrace_call, SYM_L_GLOBAL)\n \tbl ftrace_stub\t\t\t\t// func(ip, parent_ip, op, regs)\n #endif\n \n+\ttrace_rcu_read_unlock\n+\n /*\n * At the callsite x0-x8 and x19-x30 were live. Any C code will have preserved\n * x19-x29 per the AAPCS, and we created frame records upon entry, so we need\n@@ -178,6 +248,10 @@ SYM_CODE_START(ftrace_stub_direct_tramp)\n SYM_CODE_END(ftrace_stub_direct_tramp)\n #endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */\n \n+/* End of [ftrace_caller, ...) for arch_rcu_tasks_trampoline_text(). */\n+SYM_CODE_START(ftrace_static_tramp_end)\n+SYM_CODE_END(ftrace_static_tramp_end)\n+\n #else /* CONFIG_DYNAMIC_FTRACE_WITH_ARGS */\n \n /*\ndiff --git a/arch/arm64/kernel/ftrace.c b/arch/arm64/kernel/ftrace.c\nindex e1a3c0b3a0514..5f4193f15cd9a 100644\n--- a/arch/arm64/kernel/ftrace.c\n+++ b/arch/arm64/kernel/ftrace.c\n@@ -17,6 +17,26 @@\n #include \u003casm/insn.h\u003e\n #include \u003casm/text-patching.h\u003e\n \n+#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS\n+extern void ftrace_static_tramp_end(void);\n+\n+/* The SRCU-fast increments in entry-ftrace.S are the this_cpu_inc() form. */\n+static_assert(!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE));\n+\n+/*\n+ * See rcu_tasks_trampoline_text(). ftrace_caller and ftrace_stub_direct_tramp\n+ * are core kernel text but must be treated as trampolines: a task interrupted\n+ * in them outside the Tasks Trace reader may be carrying an ops pointer (x11)\n+ * or a direct-call BPF trampoline address (x17) whose lifetime is guarded only\n+ * by Tasks RCU.\n+ */\n+bool arch_rcu_tasks_trampoline_text(unsigned long ip)\n+{\n+\treturn ip \u003e= (unsigned long)ftrace_caller \u0026\u0026\n+\t ip \u003c (unsigned long)ftrace_static_tramp_end;\n+}\n+#endif\n+\n #ifdef CONFIG_DYNAMIC_FTRACE_WITH_ARGS\n struct fregs_offset {\n \tconst char *name;\ndiff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c\nindex c18e005a41dbe..87bbcca0298fd 100644\n--- a/arch/arm64/net/bpf_jit_comp.c\n+++ b/arch/arm64/net/bpf_jit_comp.c\n@@ -14,6 +14,7 @@\n #include \u003clinux/filter.h\u003e\n #include \u003clinux/memory.h\u003e\n #include \u003clinux/printk.h\u003e\n+#include \u003clinux/rcupdate_trace.h\u003e\n #include \u003clinux/slab.h\u003e\n \n #include \u003casm/asm-extable.h\u003e\n@@ -2591,6 +2592,77 @@ static void emit_arena_arg_conv(struct jit_ctx *ctx, u8 dst, u8 src, bool nullab\n \temit(A64_SUB(0, dst, src, base_lo), ctx);\n }\n \n+/*\n+ * Open-coded rcu_read_lock_trace() / rcu_read_unlock_trace() for the\n+ * trampoline, see CONFIG_HAVE_RCU_TRAMPOLINE_READERS and the equivalent macros\n+ * in arch/arm64/kernel/entry-ftrace.S. The SRCU-fast per-CPU increment is an\n+ * LL/SC add on this CPU's counter; being migrated between reading the per-CPU\n+ * offset and the store-exclusive only means another CPU's counter is\n+ * incremented atomically instead, which SRCU sums over anyway. Uses x10-x15,\n+ * which are scratch at every emission point, and the flags.\n+ */\n+static void emit_trace_rcu_reader(struct jit_ctx *ctx, bool lock)\n+{\n+#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS\n+\tconst int nesting_off = offsetof(struct task_struct, trc_reader_nesting);\n+\tconst int scp_off = offsetof(struct task_struct, trc_reader_scp);\n+\tconst int ctr_off = lock ? offsetof(struct srcu_ctr, srcu_locks)\n+\t\t\t : offsetof(struct srcu_ctr, srcu_unlocks);\n+\tconst bool mb = !IS_ENABLED(CONFIG_TASKS_TRACE_RCU_NO_MB);\n+\tconst u8 tsk = A64_R(10), n = A64_R(11), tmp = A64_R(12);\n+\tconst u8 ctr = A64_R(13), addr = A64_R(14), val = A64_R(15);\n+\n+\t/* The plain this_cpu_inc() form of __srcu_read_lock_fast(). */\n+\tBUILD_BUG_ON(IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE));\n+\t/* LDR/STR (immediate, unsigned offset) ranges */\n+\tBUILD_BUG_ON((nesting_off \u0026 3) || nesting_off \u003e= SZ_16K ||\n+\t\t (scp_off \u0026 7) || scp_off \u003e= SZ_32K);\n+\n+\temit(A64_MRS_SP_EL0(tsk), ctx);\t\t\t/* current */\n+\temit(A64_LDR32I(n, tsk, nesting_off), ctx);\n+\tif (lock) {\n+\t\temit(A64_ADD_I(0, tmp, n, 1), ctx);\n+\t\temit(A64_STR32I(tmp, tsk, nesting_off), ctx);\n+\t\t/* interrupted a reader: done (branch to 1:) */\n+\t\temit(A64_CBNZ(0, n, 13 + mb), ctx);\n+\t\t/* scp = rcu_tasks_trace_srcu_struct.srcu_ctrp; current-\u003etrc_reader_scp = scp */\n+\t\temit_addr_mov_i64(ctr, (u64)\u0026rcu_tasks_trace_srcu_struct.srcu_ctrp, ctx);\n+\t\temit(A64_LDR64I(ctr, ctr, 0), ctx);\n+\t\temit(A64_STR64I(ctr, tsk, scp_off), ctx);\n+\t} else {\n+\t\temit(A64_SUB_I(0, n, n, 1), ctx);\n+\t\t/* still nested: just store the count (branch to 2:) */\n+\t\temit(A64_CBNZ(0, n, 11 + mb), ctx);\n+\t\t/* outermost: pick up scp before an interrupt can see nesting == 0 */\n+\t\temit(A64_LDR64I(ctr, tsk, scp_off), ctx);\n+\t\temit(A64_STR32I(A64_ZR, tsk, nesting_off), ctx);\n+\t\tif (mb)\n+\t\t\temit(A64_DMB_ISH, ctx);\n+\t}\n+\t/* this_cpu_inc(scp-\u003esrcu_locks / scp-\u003esrcu_unlocks) */\n+\tif (cpus_have_cap(ARM64_HAS_VIRT_HOST_EXTN))\n+\t\temit(A64_MRS_TPIDR_EL2(addr), ctx);\n+\telse\n+\t\temit(A64_MRS_TPIDR_EL1(addr), ctx);\n+\temit(A64_ADD(1, addr, addr, ctr), ctx);\n+\temit(A64_ADD_I(1, addr, addr, ctr_off), ctx);\n+\temit(A64_LDXR(1, val, addr), ctx);\n+\temit(A64_ADD_I(1, val, val, 1), ctx);\n+\temit(A64_STXR(1, val, addr, tmp), ctx);\n+\temit(A64_CBNZ(0, tmp, -3), ctx);\n+\tif (lock) {\n+\t\tif (mb)\n+\t\t\temit(A64_DMB_ISH, ctx);\n+\t\t/* 1: */\n+\t} else {\n+\t\temit(A64_B(2), ctx);\n+\t\t/* 2: */\n+\t\temit(A64_STR32I(n, tsk, nesting_off), ctx);\n+\t\t/* 3: */\n+\t}\n+#endif\n+}\n+\n static void save_args(struct jit_ctx *ctx, int bargs_off, int oargs_off,\n \t\t const struct btf_func_model *m, const struct arg_aux *a,\n \t\t bool for_call_origin, bool is_struct_ops, u64 arena_base)\n@@ -2854,6 +2926,16 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,\n \temit(A64_STR64I(A64_R(19), A64_SP, regs_off), ctx);\n \temit(A64_STR64I(A64_R(20), A64_SP, regs_off + 8), ctx);\n \n+\t/*\n+\t * Tasks RCU keeps this image alive only while we are a Tasks Trace\n+\t * reader; the instructions before this point (and after the final\n+\t * unlock) are covered by the irq-exit IP check. One reader spans\n+\t * __bpf_tramp_enter() and the fentry/fmod_ret progs, a second one the\n+\t * fexit progs and __bpf_tramp_exit(); the original function runs\n+\t * outside both, with the image pinned by im-\u003epcref instead.\n+\t */\n+\temit_trace_rcu_reader(ctx, true);\n+\n \tif (flags \u0026 BPF_TRAMP_F_CALL_ORIG) {\n \t\t/* for the first pass, assume the worst case */\n \t\tif (!ctx-\u003eimage)\n@@ -2898,12 +2980,20 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,\n \tif (flags \u0026 BPF_TRAMP_F_CALL_ORIG) {\n \t\t/* the original func takes kernel addresses, never converted ones */\n \t\tsave_args(ctx, bargs_off, oargs_off, m, a, true, is_struct_ops, 0);\n+\t\temit_trace_rcu_reader(ctx, false);\n \t\t/* call original func */\n \t\temit(A64_LDR64I(A64_R(10), A64_SP, retaddr_off), ctx);\n \t\temit(A64_ADR(A64_LR, AARCH64_INSN_SIZE * 2), ctx);\n \t\temit(A64_RET(A64_R(10)), ctx);\n \t\t/* store return value */\n \t\temit(A64_STR64I(A64_R(0), A64_SP, retval_off), ctx);\n+\t\t/*\n+\t\t * Second reader. Taken before ip_after_call so that the branch\n+\t\t * to the epilogue patched in at teardown is inside it too; the\n+\t\t * fmod_ret early exit lands past this still holding the first\n+\t\t * reader, so either way exactly one is held.\n+\t\t */\n+\t\temit_trace_rcu_reader(ctx, true);\n \t\t/* reserve a nop for bpf_tramp_image_put */\n \t\tim-\u003eip_after_call = ctx-\u003ero_image + ctx-\u003eidx;\n \t\temit(A64_NOP, ctx);\n@@ -2945,6 +3035,9 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,\n \tif (flags \u0026 BPF_TRAMP_F_RESTORE_REGS)\n \t\trestore_args(ctx, bargs_off, a-\u003eregs_for_args);\n \n+\t/* Remaining instructions are covered by the irq-exit IP check. */\n+\temit_trace_rcu_reader(ctx, false);\n+\n \t/* restore callee saved register x19 and x20 */\n \temit(A64_LDR64I(A64_R(19), A64_SP, regs_off), ctx);\n \temit(A64_LDR64I(A64_R(20), A64_SP, regs_off + 8), ctx);\ndiff --git a/arch/x86/Kconfig b/arch/x86/Kconfig\nindex 15fd9ec5ecacb..64c3814eb7459 100644\n--- a/arch/x86/Kconfig\n+++ b/arch/x86/Kconfig\n@@ -288,6 +288,7 @@ config X86\n \tselect MMU_GATHER_RCU_TABLE_FREE\n \tselect MMU_GATHER_MERGE_VMAS\n \tselect HAVE_POSIX_CPU_TIMERS_TASK_WORK\n+\tselect HAVE_RCU_TRAMPOLINE_READERS\tif X86_64 \u0026\u0026 SMP \u0026\u0026 DYNAMIC_FTRACE\n \tselect HAVE_REGS_AND_STACK_ACCESS_API\n \tselect HAVE_RELIABLE_STACKTRACE\t\tif UNWINDER_ORC || STACK_VALIDATION\n \tselect HAVE_FUNCTION_ARG_ACCESS_API\ndiff --git a/arch/x86/kernel/asm-offsets.c b/arch/x86/kernel/asm-offsets.c\nindex 081816888f7a0..876c3986419a2 100644\n--- a/arch/x86/kernel/asm-offsets.c\n+++ b/arch/x86/kernel/asm-offsets.c\n@@ -9,6 +9,7 @@\n #include \u003clinux/crypto.h\u003e\n #include \u003ccrypto/aria.h\u003e\n #include \u003clinux/sched.h\u003e\n+#include \u003clinux/srcu.h\u003e\n #include \u003clinux/stddef.h\u003e\n #include \u003clinux/hardirq.h\u003e\n #include \u003clinux/suspend.h\u003e\n@@ -46,6 +47,13 @@ static void __used common(void)\n #ifdef CONFIG_STACKPROTECTOR\n \tOFFSET(TASK_stack_canary, task_struct, stack_canary);\n #endif\n+#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS\n+\tOFFSET(TASK_trc_reader_nesting, task_struct, trc_reader_nesting);\n+\tOFFSET(TASK_trc_reader_scp, task_struct, trc_reader_scp);\n+\tOFFSET(SRCU_srcu_ctrp, srcu_struct, srcu_ctrp);\n+\tOFFSET(SRCU_CTR_srcu_locks, srcu_ctr, srcu_locks);\n+\tOFFSET(SRCU_CTR_srcu_unlocks, srcu_ctr, srcu_unlocks);\n+#endif\n \n \tBLANK();\n \tOFFSET(pbe_address, pbe, address);\ndiff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c\nindex 17d6edfcb7e00..9babaed483eb7 100644\n--- a/arch/x86/kernel/ftrace.c\n+++ b/arch/x86/kernel/ftrace.c\n@@ -275,6 +275,49 @@ static inline void tramp_free(void *tramp)\n \texecmem_free(tramp);\n }\n \n+#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS\n+extern void ftrace_static_tramp_end(void);\n+extern char __return_thunk_start[], __return_thunk_end[];\n+extern char __rethunk_safe_start[], __rethunk_safe_end[];\n+\n+/*\n+ * The SRCU-fast increments in TRACE_RCU_READ_LOCK/UNLOCK (ftrace_64.S) are the\n+ * this_cpu_inc() form.\n+ */\n+static_assert(!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE));\n+\n+/*\n+ * See rcu_tasks_trampoline_text(). Some core kernel text behaves like a\n+ * trampoline for Tasks RCU purposes because a task executing there outside\n+ * any Tasks Trace reader may still be about to enter a Tasks-RCU-protected\n+ * trampoline whose address it already holds:\n+ *\n+ * - the static ftrace_caller / ftrace_regs_caller / ftrace_stub_direct_tramp\n+ * stubs, which carry a direct-call target on the stack until their final\n+ * RET, and\n+ * - the return thunks that RET expands to under CONFIG_MITIGATION_RETHUNK,\n+ * which run after leaving the stubs above and before landing in that\n+ * target.\n+ */\n+bool arch_rcu_tasks_trampoline_text(unsigned long ip)\n+{\n+\tif (ip \u003e= (unsigned long)ftrace_caller \u0026\u0026\n+\t ip \u003c (unsigned long)ftrace_static_tramp_end)\n+\t\treturn true;\n+#ifdef CONFIG_MITIGATION_RETPOLINE\n+\tif (ip \u003e= (unsigned long)__return_thunk_start \u0026\u0026\n+\t ip \u003c (unsigned long)__return_thunk_end)\n+\t\treturn true;\n+#endif\n+#ifdef CONFIG_MITIGATION_SRSO\n+\tif (ip \u003e= (unsigned long)__rethunk_safe_start \u0026\u0026\n+\t ip \u003c (unsigned long)__rethunk_safe_end)\n+\t\treturn true;\n+#endif\n+\treturn false;\n+}\n+#endif /* CONFIG_TASKS_RCU_TRAMPOLINE_READERS */\n+\n /* Defined as markers to the end of the ftrace default trampolines */\n extern void ftrace_regs_caller_end(void);\n extern void ftrace_caller_end(void);\ndiff --git a/arch/x86/kernel/ftrace_64.S b/arch/x86/kernel/ftrace_64.S\nindex 62c1c93aa1c6a..5d8cb38619781 100644\n--- a/arch/x86/kernel/ftrace_64.S\n+++ b/arch/x86/kernel/ftrace_64.S\n@@ -7,6 +7,7 @@\n #include \u003clinux/cfi_types.h\u003e\n #include \u003clinux/linkage.h\u003e\n #include \u003casm/asm-offsets.h\u003e\n+#include \u003casm/percpu.h\u003e\n #include \u003casm/ptrace.h\u003e\n #include \u003casm/ftrace.h\u003e\n #include \u003casm/nospec-branch.h\u003e\n@@ -145,6 +146,53 @@ SYM_FUNC_END(ftrace_stub_graph)\n \n #ifdef CONFIG_DYNAMIC_FTRACE\n \n+/*\n+ * Open-coded rcu_read_lock_trace() / rcu_read_unlock_trace(), see\n+ * include/linux/rcupdate_trace.h and CONFIG_HAVE_RCU_TRAMPOLINE_READERS: the\n+ * trampoline and the ftrace_ops it is about to load are kept alive by Tasks\n+ * RCU only while we are inside this reader, so the lock must precede the\n+ * function_trace_op load and the unlock must follow the call. These live\n+ * inside the region copied into dynamic trampolines; the %rip-relative\n+ * references are fixed up by text_poke_apply_relocation() in\n+ * create_trampoline(). Clobbers %rax, %rcx and flags.\n+ */\n+.macro TRACE_RCU_READ_LOCK\n+#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS\n+\tmovq\tPER_CPU_VAR(current_task), %rcx\n+\tmovl\tTASK_trc_reader_nesting(%rcx), %eax\n+\tincl\tTASK_trc_reader_nesting(%rcx)\n+\ttestl\t%eax, %eax\n+\tjnz\t.Ltrl_nested_\\@\n+\tmovq\trcu_tasks_trace_srcu_struct+SRCU_srcu_ctrp(%rip), %rax\n+\tincq\t%gs:SRCU_CTR_srcu_locks(%rax)\n+\tmovq\t%rax, TASK_trc_reader_scp(%rcx)\n+#ifndef CONFIG_TASKS_TRACE_RCU_NO_MB\n+\tlock addl $0, -4(%rsp)\t\t/* smp_mb() */\n+#endif\n+.Ltrl_nested_\\@:\n+#endif\n+.endm\n+\n+.macro TRACE_RCU_READ_UNLOCK\n+#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS\n+\tmovq\tPER_CPU_VAR(current_task), %rcx\n+\tmovl\tTASK_trc_reader_nesting(%rcx), %eax\n+\tsubl\t$1, %eax\n+\tjnz\t.Ltru_nested_\\@\n+\t/* Outermost: pick up scp before an interrupt can see nesting == 0. */\n+\tmovq\tTASK_trc_reader_scp(%rcx), %rax\n+\tmovl\t$0, TASK_trc_reader_nesting(%rcx)\n+#ifndef CONFIG_TASKS_TRACE_RCU_NO_MB\n+\tlock addl $0, -4(%rsp)\t\t/* smp_mb() */\n+#endif\n+\tincq\t%gs:SRCU_CTR_srcu_unlocks(%rax)\n+\tjmp\t.Ltru_done_\\@\n+.Ltru_nested_\\@:\n+\tmovl\t%eax, TASK_trc_reader_nesting(%rcx)\n+.Ltru_done_\\@:\n+#endif\n+.endm\n+\n SYM_FUNC_START(__fentry__)\n \tANNOTATE_NOENDBR\n \tCALL_DEPTH_ACCOUNT\n@@ -163,6 +211,8 @@ SYM_FUNC_START(ftrace_caller)\n \tleaq MCOUNT_REG_SIZE+8(%rsp), %rcx\n \tmovq %rcx, RSP(%rsp)\n \n+\tTRACE_RCU_READ_LOCK\n+\n SYM_INNER_LABEL(ftrace_caller_op_ptr, SYM_L_GLOBAL)\n \tANNOTATE_NOENDBR\n \t/* Load the ftrace_ops into the 3rd parameter */\n@@ -181,6 +231,8 @@ SYM_INNER_LABEL(ftrace_call, SYM_L_GLOBAL)\n \tANNOTATE_NOENDBR\n \tcall ftrace_stub\n \n+\tTRACE_RCU_READ_UNLOCK\n+\n \t/* Handlers can change the RIP */\n \tmovq RIP(%rsp), %rax\n \tmovq %rax, MCOUNT_REG_SIZE(%rsp)\n@@ -209,6 +261,8 @@ SYM_FUNC_START(ftrace_regs_caller)\n \n \tCALL_DEPTH_ACCOUNT\n \n+\tTRACE_RCU_READ_LOCK\n+\n SYM_INNER_LABEL(ftrace_regs_caller_op_ptr, SYM_L_GLOBAL)\n \tANNOTATE_NOENDBR\n \t/* Load the ftrace_ops into the 3rd parameter */\n@@ -246,6 +300,8 @@ SYM_INNER_LABEL(ftrace_regs_call, SYM_L_GLOBAL)\n \tANNOTATE_NOENDBR\n \tcall ftrace_stub\n \n+\tTRACE_RCU_READ_UNLOCK\n+\n \t/* Copy flags back to SS, to restore them */\n \tmovq EFLAGS(%rsp), %rax\n \tmovq %rax, MCOUNT_REG_SIZE(%rsp)\n@@ -328,6 +384,19 @@ SYM_FUNC_START(ftrace_stub_direct_tramp)\n \tRET\n SYM_FUNC_END(ftrace_stub_direct_tramp)\n \n+/*\n+ * [ftrace_caller, ftrace_static_tramp_end) is treated as trampoline text by\n+ * rcu_tasks_trampoline_text(): outside TRACE_RCU_READ_LOCK/UNLOCK the stubs\n+ * may still hold a direct-call trampoline address (ORIG_RAX / the return\n+ * address they RET to) that only Tasks RCU keeps alive. With return thunks\n+ * the RET itself runs elsewhere; arch_rcu_tasks_trampoline_text() covers\n+ * those too.\n+ */\n+SYM_CODE_START_NOALIGN(ftrace_static_tramp_end)\n+\tUNWIND_HINT_UNDEFINED\n+\tANNOTATE_NOENDBR\n+SYM_CODE_END(ftrace_static_tramp_end)\n+\n #else /* ! CONFIG_DYNAMIC_FTRACE */\n \n SYM_FUNC_START(__fentry__)\ndiff --git a/arch/x86/kernel/kprobes/opt.c b/arch/x86/kernel/kprobes/opt.c\nindex 3f8fea52619f9..68a5de6cdabe7 100644\n--- a/arch/x86/kernel/kprobes/opt.c\n+++ b/arch/x86/kernel/kprobes/opt.c\n@@ -31,6 +31,7 @@\n #include \u003casm/set_memory.h\u003e\n #include \u003casm/sections.h\u003e\n #include \u003casm/nospec-branch.h\u003e\n+#include \u003casm/asm-offsets.h\u003e\n \n #include \"common.h\"\n \n@@ -101,6 +102,47 @@ static void synthesize_set_arg1(kprobe_opcode_t *addr, unsigned long val)\n \t*(unsigned long *)addr = val;\n }\n \n+/*\n+ * Open-coded rcu_read_lock_trace() / rcu_read_unlock_trace() around the call\n+ * to optimized_callback(), see CONFIG_HAVE_RCU_TRAMPOLINE_READERS and the\n+ * equivalent macros in ftrace_64.S. The template is memcpy()d into the slot\n+ * without relocation processing, so memory references must be absolute\n+ * rather than %rip-relative. %rax and %rcx are free at both points.\n+ */\n+#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS\n+#ifndef CONFIG_TASKS_TRACE_RCU_NO_MB\n+#define OPTPROBE_TRACE_RCU_MB\t\"\tlock addl $0, -4(%rsp)\\n\"\n+#else\n+#define OPTPROBE_TRACE_RCU_MB\n+#endif\n+#define OPTPROBE_TRACE_RCU_READ_LOCK\t\t\t\t\t\t\\\n+\t\t\"\tmovq %gs:current_task, %rcx\\n\"\t\t\t\t\\\n+\t\t\"\tmovl \" __stringify(TASK_trc_reader_nesting) \"(%rcx), %eax\\n\"\t\\\n+\t\t\"\tincl \" __stringify(TASK_trc_reader_nesting) \"(%rcx)\\n\"\t\\\n+\t\t\"\ttestl %eax, %eax\\n\"\t\t\t\t\t\t\\\n+\t\t\"\tjnz 1f\\n\"\t\t\t\t\t\t\t\\\n+\t\t\"\tmovq rcu_tasks_trace_srcu_struct+\" __stringify(SRCU_srcu_ctrp) \", %rax\\n\" \\\n+\t\t\"\tincq %gs:\" __stringify(SRCU_CTR_srcu_locks) \"(%rax)\\n\"\t\t\\\n+\t\t\"\tmovq %rax, \" __stringify(TASK_trc_reader_scp) \"(%rcx)\\n\"\t\\\n+\t\tOPTPROBE_TRACE_RCU_MB\t\t\t\t\t\t\\\n+\t\t\"1:\\n\"\n+#define OPTPROBE_TRACE_RCU_READ_UNLOCK\t\t\t\t\t\t\\\n+\t\t\"\tmovq %gs:current_task, %rcx\\n\"\t\t\t\t\\\n+\t\t\"\tmovl \" __stringify(TASK_trc_reader_nesting) \"(%rcx), %eax\\n\"\t\\\n+\t\t\"\tsubl $1, %eax\\n\"\t\t\t\t\t\t\\\n+\t\t\"\tjnz 2f\\n\"\t\t\t\t\t\t\t\\\n+\t\t\"\tmovq \" __stringify(TASK_trc_reader_scp) \"(%rcx), %rax\\n\"\t\\\n+\t\t\"\tmovl $0, \" __stringify(TASK_trc_reader_nesting) \"(%rcx)\\n\"\t\\\n+\t\tOPTPROBE_TRACE_RCU_MB\t\t\t\t\t\t\\\n+\t\t\"\tincq %gs:\" __stringify(SRCU_CTR_srcu_unlocks) \"(%rax)\\n\"\t\\\n+\t\t\"\tjmp 3f\\n\"\t\t\t\t\t\t\t\\\n+\t\t\"2:\tmovl %eax, \" __stringify(TASK_trc_reader_nesting) \"(%rcx)\\n\"\t\\\n+\t\t\"3:\\n\"\n+#else\n+#define OPTPROBE_TRACE_RCU_READ_LOCK\n+#define OPTPROBE_TRACE_RCU_READ_UNLOCK\n+#endif\n+\n asm (\n \t\t\t\".pushsection .rodata\\n\"\n \t\t\t\".global optprobe_template_entry\\n\"\n@@ -114,6 +156,7 @@ asm (\n \t\t\t\"optprobe_template_clac:\\n\"\n \t\t\tASM_NOP3\n \t\t\tSAVE_REGS_STRING\n+\t\t\tOPTPROBE_TRACE_RCU_READ_LOCK\n \t\t\t\"\tmovq %rsp, %rsi\\n\"\n \t\t\t\".global optprobe_template_val\\n\"\n \t\t\t\"optprobe_template_val:\\n\"\n@@ -122,6 +165,7 @@ asm (\n \t\t\t\".global optprobe_template_call\\n\"\n \t\t\t\"optprobe_template_call:\\n\"\n \t\t\tASM_NOP5\n+\t\t\tOPTPROBE_TRACE_RCU_READ_UNLOCK\n \t\t\t/* Copy 'regs-\u003eflags' into 'regs-\u003ess'. */\n \t\t\t\"\tmovq 18*8(%rsp), %rdx\\n\"\n \t\t\t\"\tmovq %rdx, 20*8(%rsp)\\n\"\ndiff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S\nindex 2438b89a4620a..e546283dc2679 100644\n--- a/arch/x86/kernel/vmlinux.lds.S\n+++ b/arch/x86/kernel/vmlinux.lds.S\n@@ -151,7 +151,9 @@ SECTIONS\n \t\t * definition.\n \t\t */\n \t\t. = srso_alias_untrain_ret | (1 \u003c\u003c 2) | (1 \u003c\u003c 8) | (1 \u003c\u003c 14) | (1 \u003c\u003c 20);\n+\t\t__rethunk_safe_start = .;\n \t\t*(.text..__x86.rethunk_safe)\n+\t\t__rethunk_safe_end = .;\n #endif\n \t\tALIGN_ENTRY_TEXT_END\n \n@@ -162,7 +164,9 @@ SECTIONS\n \t\tSOFTIRQENTRY_TEXT\n #ifdef CONFIG_MITIGATION_RETPOLINE\n \t\t*(.text..__x86.indirect_thunk)\n+\t\t__return_thunk_start = .;\n \t\t*(.text..__x86.return_thunk)\n+\t\t__return_thunk_end = .;\n #endif\n \t\tSTATIC_CALL_TEXT\n \t\t*(.gnu.warning)\ndiff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c\nindex 2853e87797a78..b77e29c9599de 100644\n--- a/arch/x86/net/bpf_jit_comp.c\n+++ b/arch/x86/net/bpf_jit_comp.c\n@@ -14,6 +14,7 @@\n #include \u003clinux/memory.h\u003e\n #include \u003clinux/sort.h\u003e\n #include \u003clinux/execmem.h\u003e\n+#include \u003clinux/rcupdate_trace.h\u003e\n #include \u003casm/extable.h\u003e\n #include \u003casm/ftrace.h\u003e\n #include \u003casm/set_memory.h\u003e\n@@ -722,6 +723,91 @@ static void emit_indirect_jump(u8 **pprog, int bpf_reg, u8 *ip)\n \t*pprog = prog;\n }\n \n+/*\n+ * Open-coded rcu_read_lock_trace() / rcu_read_unlock_trace() for the\n+ * trampoline, see CONFIG_HAVE_RCU_TRAMPOLINE_READERS and the equivalent\n+ * macros in arch/x86/kernel/ftrace_64.S. The image is not relocated, so\n+ * current_task and rcu_tasks_trace_srcu_struct are referenced by absolute\n+ * (sign-extended 32-bit) address, the form the JIT already relies on for\n+ * this_cpu_off. Uses r10 and r11, which are scratch at every emission\n+ * point, and clobbers flags.\n+ *\n+ * lock: unlock:\n+ * mov r11, gs:[current_task] mov r11, gs:[current_task]\n+ * mov r10d, [r11+nesting_off] mov r10d, [r11+nesting_off]\n+ * inc dword ptr [r11+nesting_off] sub r10d, 1\n+ * test r10d, r10d jnz 2f\n+ * jnz 1f mov r10, [r11+scp_off]\n+ * mov r10, [\u0026srcu.srcu_ctrp] mov dword ptr [r11+nesting_off], 0\n+ * inc qword ptr gs:[r10+locks_off] (smp_mb)\n+ * mov [r11+scp_off], r10 inc qword ptr gs:[r10+unlocks_off]\n+ * (smp_mb) jmp 3f\n+ * 1: 2: mov [r11+nesting_off], r10d\n+ * 3:\n+ */\n+static void emit_trace_rcu_reader(u8 **pprog, bool lock)\n+{\n+#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS\n+\tconst u32 nesting_off = offsetof(struct task_struct, trc_reader_nesting);\n+\tconst u32 scp_off = offsetof(struct task_struct, trc_reader_scp);\n+\tconst u32 locks_off = offsetof(struct srcu_ctr, srcu_locks);\n+\tconst u32 unlocks_off = offsetof(struct srcu_ctr, srcu_unlocks);\n+\tconst bool mb = !IS_ENABLED(CONFIG_TASKS_TRACE_RCU_NO_MB);\n+\tu8 *prog = *pprog;\n+\n+\t/* The plain this_cpu_inc() form of __srcu_read_lock_fast(). */\n+\tBUILD_BUG_ON(IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE));\n+\n+\t/* mov r11, gs:[abs32 current_task] */\n+\tEMIT2(0x65, 0x4C);\n+\tEMIT3(0x8B, 0x1C, 0x25);\n+\tEMIT((u32)(unsigned long)\u0026current_task, 4);\n+\t/* mov r10d, dword ptr [r11 + nesting_off] */\n+\tEMIT3_off32(0x45, 0x8B, 0x93, nesting_off);\n+\n+\tif (lock) {\n+\t\t/* inc dword ptr [r11 + nesting_off] */\n+\t\tEMIT3_off32(0x41, 0xFF, 0x83, nesting_off);\n+\t\t/* test r10d, r10d */\n+\t\tEMIT3(0x45, 0x85, 0xD2);\n+\t\t/* jnz 1f */\n+\t\tEMIT2(X86_JNE, 8 + 8 + 7 + (mb ? 6 : 0));\n+\t\t/* mov r10, qword ptr [abs32 \u0026rcu_tasks_trace_srcu_struct.srcu_ctrp] */\n+\t\tEMIT4(0x4C, 0x8B, 0x14, 0x25);\n+\t\tEMIT((u32)(unsigned long)\u0026rcu_tasks_trace_srcu_struct.srcu_ctrp, 4);\n+\t\t/* inc qword ptr gs:[r10 + locks_off] */\n+\t\tEMIT4_off32(0x65, 0x49, 0xFF, 0x82, locks_off);\n+\t\t/* mov qword ptr [r11 + scp_off], r10 */\n+\t\tEMIT3_off32(0x4D, 0x89, 0x93, scp_off);\n+\t\t/* smp_mb(): lock add dword ptr [rsp - 4], 0 */\n+\t\tif (mb)\n+\t\t\tEMIT2_off32(0xF0, 0x83, 0x00FC2444);\n+\t\t/* 1: */\n+\t} else {\n+\t\t/* sub r10d, 1 */\n+\t\tEMIT4(0x41, 0x83, 0xEA, 0x01);\n+\t\t/* jnz 2f */\n+\t\tEMIT2(X86_JNE, 7 + 11 + (mb ? 6 : 0) + 8 + 2);\n+\t\t/* mov r10, qword ptr [r11 + scp_off] */\n+\t\tEMIT3_off32(0x4D, 0x8B, 0x93, scp_off);\n+\t\t/* mov dword ptr [r11 + nesting_off], 0 */\n+\t\tEMIT3_off32(0x41, 0xC7, 0x83, nesting_off);\n+\t\tEMIT(0, 4);\n+\t\tif (mb)\n+\t\t\tEMIT2_off32(0xF0, 0x83, 0x00FC2444);\n+\t\t/* inc qword ptr gs:[r10 + unlocks_off] */\n+\t\tEMIT4_off32(0x65, 0x49, 0xFF, 0x82, unlocks_off);\n+\t\t/* jmp 3f */\n+\t\tEMIT2(0xEB, 7);\n+\t\t/* 2: mov dword ptr [r11 + nesting_off], r10d */\n+\t\tEMIT3_off32(0x45, 0x89, 0x93, nesting_off);\n+\t\t/* 3: */\n+\t}\n+\n+\t*pprog = prog;\n+#endif\n+}\n+\n static void emit_return(u8 **pprog, u8 *ip)\n {\n \tu8 *prog = *pprog;\n@@ -3610,6 +3696,16 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im\n \t/* mov QWORD PTR [rbp - rbx_off], rbx */\n \temit_stx(\u0026prog, BPF_DW, BPF_REG_FP, BPF_REG_6, -rbx_off);\n \n+\t/*\n+\t * Tasks RCU keeps this image alive only while we are a Tasks Trace\n+\t * reader; the instructions before this point (and after the final\n+\t * unlock) are covered by the irq-exit IP check. One reader spans\n+\t * __bpf_tramp_enter() and the fentry/fmod_ret progs, a second one\n+\t * the fexit progs and __bpf_tramp_exit(); the original function runs\n+\t * outside both, with the image pinned by im-\u003epcref instead.\n+\t */\n+\temit_trace_rcu_reader(\u0026prog, true);\n+\n \tfunc_meta = nr_regs;\n \t/* Store number of argument registers of the traced function */\n \temit_store_stack_imm64(\u0026prog, BPF_REG_0, -func_meta_off, func_meta);\n@@ -3660,6 +3756,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im\n \t}\n \n \tif (flags \u0026 BPF_TRAMP_F_CALL_ORIG) {\n+\t\temit_trace_rcu_reader(\u0026prog, false);\n \t\trestore_regs(m, \u0026prog, regs_off);\n \t\tsave_args(m, \u0026prog, arg_stack_off, true, flags, 0);\n \n@@ -3682,6 +3779,13 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im\n \t\t}\n \t\t/* remember return value in a stack for bpf prog to access */\n \t\temit_stx(\u0026prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8);\n+\t\t/*\n+\t\t * Second reader. Taken before ip_after_call so that the\n+\t\t * ip_after_call -\u003e ip_epilogue jump patched in at teardown is\n+\t\t * inside it too; the fmod_ret early exit jumps past this still\n+\t\t * holding the first reader, so either way exactly one is held.\n+\t\t */\n+\t\temit_trace_rcu_reader(\u0026prog, true);\n \t\tim-\u003eip_after_call = image + (prog - (u8 *)rw_image);\n \t\temit_nops(\u0026prog, X86_PATCH_SIZE);\n \t}\n@@ -3737,6 +3841,9 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im\n \t\tLOAD_TRAMP_TAIL_CALL_CNT_PTR(stack_size);\n \t}\n \n+\t/* Remaining instructions are covered by the irq-exit IP check. */\n+\temit_trace_rcu_reader(\u0026prog, false);\n+\n \t/* restore return value of orig_call or fentry prog back into RAX */\n \tif (save_ret)\n \t\temit_ldx(\u0026prog, BPF_DW, BPF_REG_0, BPF_REG_FP, -8);\ndiff --git a/arch/x86/xen/enlighten_pv.c b/arch/x86/xen/enlighten_pv.c\nindex 2c64b388f6164..3d85035f5624f 100644\n--- a/arch/x86/xen/enlighten_pv.c\n+++ b/arch/x86/xen/enlighten_pv.c\n@@ -739,7 +739,7 @@ __visible noinstr void xen_pv_evtchn_do_upcall(struct pt_regs *regs)\n \n \tinhcall = get_and_clear_inhcall();\n \tif (inhcall \u0026\u0026 !WARN_ON_ONCE(state.exit_rcu)) {\n-\t\tirqentry_exit_cond_resched();\n+\t\tirqentry_exit_cond_resched(regs);\n \t\tinstrumentation_end();\n \t\trestore_inhcall(inhcall);\n \t} else {\ndiff --git a/include/linux/irq-entry-common.h b/include/linux/irq-entry-common.h\nindex 0bb6c03481fad..d42568a488a7f 100644\n--- a/include/linux/irq-entry-common.h\n+++ b/include/linux/irq-entry-common.h\n@@ -5,6 +5,7 @@\n #include \u003clinux/context_tracking.h\u003e\n #include \u003clinux/hrtimer_rearm.h\u003e\n #include \u003clinux/kmsan.h\u003e\n+#include \u003clinux/rcupdate_trace.h\u003e\n #include \u003clinux/rseq_entry.h\u003e\n #include \u003clinux/static_call_types.h\u003e\n #include \u003clinux/syscalls.h\u003e\n@@ -214,6 +215,7 @@ static __always_inline void __exit_to_user_mode_validate(void)\n {\n \t/* Ensure that kernel state is sane for a return to userspace */\n \tkmap_assert_nomap();\n+\trcu_tasks_trace_assert_idle();\n \tlockdep_assert_irqs_disabled();\n \tlockdep_sys_exit();\n }\n@@ -343,24 +345,25 @@ typedef struct irqentry_state {\n \n /**\n * irqentry_exit_cond_resched - Conditionally reschedule on return from interrupt\n+ * @regs:\tPointer to pt_regs of interrupted context\n *\n * Conditional reschedule with additional sanity checks.\n */\n-void raw_irqentry_exit_cond_resched(void);\n+void raw_irqentry_exit_cond_resched(struct pt_regs *regs);\n \n #ifdef CONFIG_PREEMPT_DYNAMIC\n #if defined(CONFIG_HAVE_PREEMPT_DYNAMIC_CALL)\n #define irqentry_exit_cond_resched_dynamic_enabled\traw_irqentry_exit_cond_resched\n #define irqentry_exit_cond_resched_dynamic_disabled\tNULL\n DECLARE_STATIC_CALL(irqentry_exit_cond_resched, raw_irqentry_exit_cond_resched);\n-#define irqentry_exit_cond_resched()\tstatic_call(irqentry_exit_cond_resched)()\n+#define irqentry_exit_cond_resched(regs)\tstatic_call(irqentry_exit_cond_resched)(regs)\n #elif defined(CONFIG_HAVE_PREEMPT_DYNAMIC_KEY)\n DECLARE_STATIC_KEY_TRUE(sk_dynamic_irqentry_exit_cond_resched);\n-void dynamic_irqentry_exit_cond_resched(void);\n-#define irqentry_exit_cond_resched()\tdynamic_irqentry_exit_cond_resched()\n+void dynamic_irqentry_exit_cond_resched(struct pt_regs *regs);\n+#define irqentry_exit_cond_resched(regs)\tdynamic_irqentry_exit_cond_resched(regs)\n #endif\n #else /* CONFIG_PREEMPT_DYNAMIC */\n-#define irqentry_exit_cond_resched()\traw_irqentry_exit_cond_resched()\n+#define irqentry_exit_cond_resched(regs)\traw_irqentry_exit_cond_resched(regs)\n #endif /* CONFIG_PREEMPT_DYNAMIC */\n \n /**\n@@ -465,7 +468,7 @@ static inline void irqentry_exit_to_kernel_mode_preempt(struct pt_regs *regs,\n \t\treturn;\n \n \tif (IS_ENABLED(CONFIG_PREEMPTION))\n-\t\tirqentry_exit_cond_resched();\n+\t\tirqentry_exit_cond_resched(regs);\n }\n \n /**\ndiff --git a/include/linux/kprobes.h b/include/linux/kprobes.h\nindex e6de7ae55bda0..74cc48c044173 100644\n--- a/include/linux/kprobes.h\n+++ b/include/linux/kprobes.h\n@@ -530,11 +530,17 @@ static inline bool is_kprobe_insn_slot(unsigned long addr)\n }\n #endif /* !CONFIG_KPROBES */\n \n-#ifndef CONFIG_OPTPROBES\n+#ifdef CONFIG_OPTPROBES\n+bool kprobe_in_optimized_region(unsigned long addr);\n+#else /* !CONFIG_OPTPROBES */\n static inline bool is_kprobe_optinsn_slot(unsigned long addr)\n {\n \treturn false;\n }\n+static inline bool kprobe_in_optimized_region(unsigned long addr)\n+{\n+\treturn false;\n+}\n #endif /* !CONFIG_OPTPROBES */\n \n #ifdef CONFIG_KRETPROBES\ndiff --git a/include/linux/module.h b/include/linux/module.h\nindex 96cc98568eea5..82ca4f774725a 100644\n--- a/include/linux/module.h\n+++ b/include/linux/module.h\n@@ -521,6 +521,13 @@ struct module {\n \tunsigned int num_ftrace_callsites;\n \tunsigned long *ftrace_callsites;\n #endif\n+#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS\n+\t/*\n+\t * An ftrace direct-call trampoline lives in this module's text; see\n+\t * rcu_tasks_trampoline_text(). Sticky once set.\n+\t */\n+\tbool ftrace_direct_tramp;\n+#endif\n #ifdef CONFIG_KPROBES\n \tvoid *kprobes_text_start;\n \tunsigned int kprobes_text_size;\ndiff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h\nindex 44c07a66edfff..79e4b14f83b95 100644\n--- a/include/linux/rcupdate.h\n+++ b/include/linux/rcupdate.h\n@@ -50,6 +50,23 @@ token_context_lock_instance(RCU, RCU_BH);\n /* Exported common interfaces */\n void call_rcu(struct rcu_head *head, rcu_callback_t func);\n void rcu_barrier_tasks(void);\n+\n+/*\n+ * Trampoline-reader Tasks RCU (CONFIG_TASKS_RCU_TRAMPOLINE_READERS), see\n+ * kernel/rcu/tasks.h. rcu_tasks_irq_resched_enter()/_exit() bracket the\n+ * irq-exit preemption; rcu_tasks_trampoline_text() and the arch_ override\n+ * classify an interrupted IP; rcu_tasks_wait_irq_preempted() lets a caller\n+ * wait out tasks already preempted somewhere it is about to make unsafe.\n+ */\n+void rcu_tasks_irq_resched_enter(unsigned long ip);\n+void rcu_tasks_irq_resched_exit(void);\n+bool rcu_tasks_trampoline_text(unsigned long ip);\n+bool arch_rcu_tasks_trampoline_text(unsigned long ip);\n+#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS\n+void rcu_tasks_wait_irq_preempted(bool (*inside)(unsigned long ip));\n+#else\n+static inline void rcu_tasks_wait_irq_preempted(bool (*inside)(unsigned long ip)) { }\n+#endif\n void synchronize_rcu(void);\n \n /*\n@@ -180,11 +197,16 @@ static inline void rcu_nocb_flush_deferred_wakeup(void) { }\n #ifdef CONFIG_TASKS_RCU_GENERIC\n \n # ifdef CONFIG_TASKS_RCU\n-# define rcu_tasks_classic_qs(t, preempt)\t\t\t\t\\\n+# ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS\n+void rcu_tasks_note_qs(struct task_struct *t, bool preempt);\n+# define rcu_tasks_classic_qs(t, preempt) rcu_tasks_note_qs((t), (preempt))\n+# else\n+# define rcu_tasks_classic_qs(t, preempt)\t\t\t\t\\\n \tdo {\t\t\t\t\t\t\t\t\\\n \t\tif (!(preempt) \u0026\u0026 READ_ONCE((t)-\u003ercu_tasks_holdout))\t\\\n \t\t\tWRITE_ONCE((t)-\u003ercu_tasks_holdout, false);\t\\\n \t} while (0)\n+# endif\n void call_rcu_tasks(struct rcu_head *head, rcu_callback_t func);\n void synchronize_rcu_tasks(void);\n void rcu_tasks_torture_stats_print(char *tt, char *tf);\ndiff --git a/include/linux/rcupdate_trace.h b/include/linux/rcupdate_trace.h\nindex 273c59a032512..9eb7f92c710be 100644\n--- a/include/linux/rcupdate_trace.h\n+++ b/include/linux/rcupdate_trace.h\n@@ -92,6 +92,12 @@ static inline void rcu_read_unlock_tasks_trace(struct srcu_ctr __percpu *scp)\n * the all the other tasks exit their critical sections.\n *\n * For more details, please see the documentation for rcu_read_lock().\n+ *\n+ * CONFIG_HAVE_RCU_TRAMPOLINE_READERS architectures open-code this pair in\n+ * their ftrace and BPF trampolines (arch/x86/kernel/ftrace_64.S,\n+ * arch/x86/kernel/kprobes/opt.c, arch/x86/net/bpf_jit_comp.c,\n+ * arch/arm64/kernel/entry-ftrace.S, arch/arm64/net/bpf_jit_comp.c,\n+ * samples/ftrace/ftrace-direct.h); changes here need to be mirrored there.\n */\n static inline void rcu_read_lock_trace(void)\n {\n@@ -211,6 +217,12 @@ unsigned long rcu_tasks_trace_batches_completed(void);\n // Placeholders to enable stepwise transition.\n void __init rcu_tasks_trace_suppress_unused(void);\n \n+/* A task must never reach userspace inside an rcu_read_lock_trace() reader. */\n+static inline void rcu_tasks_trace_assert_idle(void)\n+{\n+\tWARN_ON_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) \u0026\u0026 READ_ONCE(current-\u003etrc_reader_nesting));\n+}\n+\n #else\n static inline unsigned long rcu_tasks_trace_batches_completed(void) { return 0; }\n /*\n@@ -220,6 +232,7 @@ static inline unsigned long rcu_tasks_trace_batches_completed(void) { return 0;\n static inline void call_rcu_tasks_trace(struct rcu_head *rhp, rcu_callback_t func) { BUG(); }\n static inline void rcu_read_lock_trace(void) { BUG(); }\n static inline void rcu_read_unlock_trace(void) { BUG(); }\n+static inline void rcu_tasks_trace_assert_idle(void) { }\n #endif /* #ifdef CONFIG_TASKS_TRACE_RCU */\n \n DEFINE_LOCK_GUARD_0(rcu_tasks_trace,\ndiff --git a/include/linux/sched.h b/include/linux/sched.h\nindex 8b3d47a325cca..15beb44caa2c8 100644\n--- a/include/linux/sched.h\n+++ b/include/linux/sched.h\n@@ -957,6 +957,7 @@ struct task_struct {\n \tu8\t\t\t\trcu_tasks_holdout;\n \tu8\t\t\t\trcu_tasks_idx;\n \tint\t\t\t\trcu_tasks_idle_cpu;\n+\tunsigned long\t\t\trcu_tasks_irq_ip;\n \tstruct list_head\t\trcu_tasks_holdout_list;\n \tint\t\t\t\trcu_tasks_exit_cpu;\n \tstruct list_head\t\trcu_tasks_exit_list;\ndiff --git a/kernel/entry/common.c b/kernel/entry/common.c\nindex e3d381fd3d251..94318519998c8 100644\n--- a/kernel/entry/common.c\n+++ b/kernel/entry/common.c\n@@ -6,6 +6,7 @@\n #include \u003clinux/jump_label.h\u003e\n #include \u003clinux/kmsan.h\u003e\n #include \u003clinux/livepatch.h\u003e\n+#include \u003clinux/rcupdate.h\u003e\n #include \u003clinux/resume_user_mode.h\u003e\n #include \u003clinux/tick.h\u003e\n \n@@ -134,15 +135,20 @@ static inline bool arch_irqentry_exit_need_resched(void);\n static inline bool arch_irqentry_exit_need_resched(void) { return true; }\n #endif\n \n-void raw_irqentry_exit_cond_resched(void)\n+void raw_irqentry_exit_cond_resched(struct pt_regs *regs)\n {\n \tif (!preempt_count()) {\n \t\t/* Sanity check RCU and thread stack */\n \t\trcu_irq_exit_check_preempt();\n \t\tif (IS_ENABLED(CONFIG_DEBUG_ENTRY))\n \t\t\tWARN_ON_ONCE(!on_thread_stack());\n-\t\tif (need_resched() \u0026\u0026 arch_irqentry_exit_need_resched())\n+\t\tif (need_resched() \u0026\u0026 arch_irqentry_exit_need_resched()) {\n+\t\t\tif (IS_ENABLED(CONFIG_TASKS_RCU_TRAMPOLINE_READERS))\n+\t\t\t\trcu_tasks_irq_resched_enter(instruction_pointer(regs));\n \t\t\tpreempt_schedule_irq();\n+\t\t\tif (IS_ENABLED(CONFIG_TASKS_RCU_TRAMPOLINE_READERS))\n+\t\t\t\trcu_tasks_irq_resched_exit();\n+\t\t}\n \t}\n }\n #ifdef CONFIG_PREEMPT_DYNAMIC\n@@ -150,11 +156,11 @@ void raw_irqentry_exit_cond_resched(void)\n DEFINE_STATIC_CALL(irqentry_exit_cond_resched, raw_irqentry_exit_cond_resched);\n #elif defined(CONFIG_HAVE_PREEMPT_DYNAMIC_KEY)\n DEFINE_STATIC_KEY_TRUE(sk_dynamic_irqentry_exit_cond_resched);\n-void dynamic_irqentry_exit_cond_resched(void)\n+void dynamic_irqentry_exit_cond_resched(struct pt_regs *regs)\n {\n \tif (!static_branch_unlikely(\u0026sk_dynamic_irqentry_exit_cond_resched))\n \t\treturn;\n-\traw_irqentry_exit_cond_resched();\n+\traw_irqentry_exit_cond_resched(regs);\n }\n #endif\n #endif\ndiff --git a/kernel/fork.c b/kernel/fork.c\nindex 416758c8a3d43..8077336bb136b 100644\n--- a/kernel/fork.c\n+++ b/kernel/fork.c\n@@ -1871,6 +1871,7 @@ static inline void rcu_copy_process(struct task_struct *p)\n \tp-\u003ercu_tasks_holdout = false;\n \tINIT_LIST_HEAD(\u0026p-\u003ercu_tasks_holdout_list);\n \tp-\u003ercu_tasks_idle_cpu = -1;\n+\tp-\u003ercu_tasks_irq_ip = 0;\n \tINIT_LIST_HEAD(\u0026p-\u003ercu_tasks_exit_list);\n #endif /* #ifdef CONFIG_TASKS_RCU */\n #ifdef CONFIG_TASKS_TRACE_RCU\ndiff --git a/kernel/kprobes.c b/kernel/kprobes.c\nindex 6337da5cab9e7..e460fba83e4ab 100644\n--- a/kernel/kprobes.c\n+++ b/kernel/kprobes.c\n@@ -511,6 +511,48 @@ static struct kprobe *get_optimized_kprobe(kprobe_opcode_t *addr)\n \treturn NULL;\n }\n \n+/*\n+ * True while kprobe_optimizer() is waiting for its Tasks RCU grace period.\n+ * Only in that window can an interruption inside an optprobe's jump region\n+ * matter to it, so kprobe_in_optimized_region() does no work otherwise.\n+ */\n+static bool kprobe_optimizer_waiting;\n+\n+/**\n+ * kprobe_in_optimized_region - Could @addr be inside bytes a jump-optimized\n+ *\tkprobe replaces?\n+ * @addr: kernel text address, typically an interrupted instruction pointer\n+ *\n+ * kprobe_optimizer() relies on synchronize_rcu_tasks() to wait for tasks that\n+ * were interrupted on an instruction boundary inside the region about to be\n+ * overwritten by the optimized jump. Where Tasks RCU is built on\n+ * reader-marked trampolines that region has no reader, so the irq-exit\n+ * quiescent-state check asks this instead (see rcu_tasks_trampoline_text()).\n+ * This is the lockless, conservative form of get_optimized_kprobe(): it does\n+ * not care whether the kprobe found is, or ever will be, optimized. May be\n+ * called from any context with preemption disabled; the kprobe hash is\n+ * RCU-protected and every free path waits for a grace period after unhashing.\n+ *\n+ * The hash walk only runs while the optimizer is actually waiting. A task\n+ * that was preempted in such a region before the flag went up is invisible\n+ * to that check, so the optimizer first waits those out by their recorded\n+ * preemption IP (rcu_tasks_wait_irq_preempted(), whose leading\n+ * synchronize_rcu() also publishes the flag to every check in flight).\n+ */\n+bool kprobe_in_optimized_region(unsigned long addr)\n+{\n+\tint i;\n+\n+\tif (!READ_ONCE(kprobe_optimizer_waiting))\n+\t\treturn false;\n+\n+\tfor (i = 1; i \u003c MAX_OPTIMIZED_LENGTH / sizeof(kprobe_opcode_t); i++)\n+\t\tif (get_kprobe((kprobe_opcode_t *)addr - i))\n+\t\t\treturn true;\n+\treturn false;\n+}\n+NOKPROBE_SYMBOL(kprobe_in_optimized_region);\n+\n /* Optimization staging list, protected by 'kprobe_mutex' */\n static LIST_HEAD(optimizing_list);\n static LIST_HEAD(unoptimizing_list);\n@@ -644,8 +686,16 @@ static void kprobe_optimizer(void)\n \t\t * to 2nd-Nth byte of jump instruction. This wait is for avoiding it.\n \t\t * Note that on non-preemptive kernel, this is transparently converted\n \t\t * to synchronoze_sched() to wait for all interrupts to have completed.\n+\t\t * kprobe_optimizer_waiting lets a reader-marked-trampoline Tasks RCU\n+\t\t * recognise tasks interrupted in such a region while we wait, and\n+\t\t * rcu_tasks_wait_irq_preempted() (a no-op elsewhere) first waits\n+\t\t * out any that were preempted there before we said so; see\n+\t\t * kprobe_in_optimized_region().\n \t\t */\n+\t\tWRITE_ONCE(kprobe_optimizer_waiting, true);\n+\t\trcu_tasks_wait_irq_preempted(kprobe_in_optimized_region);\n \t\tsynchronize_rcu_tasks();\n+\t\tWRITE_ONCE(kprobe_optimizer_waiting, false);\n \n \t\t/* Step 3: Optimize kprobes after quiesence period */\n \t\tdo_optimize_kprobes();\ndiff --git a/kernel/rcu/Kconfig b/kernel/rcu/Kconfig\nindex 332df7a7a6347..341b68b972ef1 100644\n--- a/kernel/rcu/Kconfig\n+++ b/kernel/rcu/Kconfig\n@@ -95,8 +95,10 @@ config FORCE_TASKS_RCU\n \thelp\n \t This option force-enables a task-based RCU implementation\n \t that uses only voluntary context switch (not preemption!),\n-\t idle, and user-mode execution as quiescent states. Not for\n-\t manual selection in most cases.\n+\t idle, and user-mode execution as quiescent states, or, on\n+\t HAVE_RCU_TRAMPOLINE_READERS architectures, the variant built on\n+\t Tasks Trace RCU readers in trampolines. Not for manual\n+\t selection in most cases.\n \n config NEED_TASKS_RCU\n \tbool\n@@ -107,6 +109,28 @@ config TASKS_RCU\n \tdefault NEED_TASKS_RCU \u0026\u0026 PREEMPTION\n \tselect IRQ_WORK\n \n+config HAVE_RCU_TRAMPOLINE_READERS\n+\tbool\n+\thelp\n+\t Select this if the architecture uses the generic irq entry code and\n+\t every trampoline whose lifetime Tasks RCU guards on it (ftrace\n+\t trampolines, kprobe out-of-line and optimized-probe slots, BPF\n+\t trampolines, out-of-line ftrace direct-call trampolines) enters a\n+\t Tasks Trace RCU read-side critical section before calling out of\n+\t the trampoline and leaves it before returning, and any core text\n+\t that runs on behalf of such a trampoline outside that reader is\n+\t reported by arch_rcu_tasks_trampoline_text(). The assembly readers\n+\t use the this_cpu_inc() form of SRCU-fast, hence !NEED_SRCU_NMI_SAFE.\n+\n+config TASKS_RCU_TRAMPOLINE_READERS\n+\tdef_bool TASKS_RCU \u0026\u0026 HAVE_RCU_TRAMPOLINE_READERS \u0026\u0026 GENERIC_IRQ_ENTRY \u0026\u0026 !NEED_SRCU_NMI_SAFE\n+\tselect TASKS_TRACE_RCU\n+\thelp\n+\t Implement the Tasks RCU grace period as a per-CPU pass over\n+\t context switches and irq-exit reschedules outside trampoline text\n+\t plus a Tasks Trace RCU grace period, instead of waiting for every\n+\t task to voluntarily context switch. See kernel/rcu/tasks.h.\n+\n config FORCE_TASKS_RUDE_RCU\n \tbool \"Force selection of Tasks Rude RCU\"\n \tdepends on RCU_EXPERT\ndiff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c\nindex 794937e13e7c3..ab870ef09af0a 100644\n--- a/kernel/rcu/rcutorture.c\n+++ b/kernel/rcu/rcutorture.c\n@@ -1142,13 +1142,23 @@ static struct rcu_torture_ops trivial_preempt_ops = {\n * Definitions for RCU-tasks torture testing.\n */\n \n+/*\n+ * A classic Tasks RCU reader is any stretch of kernel code that does not\n+ * voluntarily block. With CONFIG_TASKS_RCU_TRAMPOLINE_READERS a preemption\n+ * outside trampoline text also ends it, and what a trampoline does to stay\n+ * protected across its call-out is take a Tasks Trace reader, so model that.\n+ */\n static int tasks_torture_read_lock(void)\n {\n+\tif (IS_ENABLED(CONFIG_TASKS_RCU_TRAMPOLINE_READERS))\n+\t\trcu_read_lock_trace();\n \treturn 0;\n }\n \n static void tasks_torture_read_unlock(int idx)\n {\n+\tif (IS_ENABLED(CONFIG_TASKS_RCU_TRAMPOLINE_READERS))\n+\t\trcu_read_unlock_trace();\n }\n \n static void rcu_tasks_torture_deferred_free(struct rcu_torture *p)\ndiff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h\nindex 627295396cd91..42ea6e0e61cbf 100644\n--- a/kernel/rcu/tasks.h\n+++ b/kernel/rcu/tasks.h\n@@ -152,7 +152,7 @@ static struct rcu_tasks rt_name =\t\t\t\t\t\t\t\\\n \t.kname = #rt_name,\t\t\t\t\t\t\t\t\\\n }\n \n-#ifdef CONFIG_TASKS_RCU\n+#if defined(CONFIG_TASKS_RCU) \u0026\u0026 !defined(CONFIG_TASKS_RCU_TRAMPOLINE_READERS)\n \n /* Report delay of scan exiting tasklist in rcu_tasks_postscan(). */\n static void tasks_rcu_exit_stall(struct timer_list *unused);\n@@ -802,7 +802,7 @@ static void rcu_tasks_torture_stats_print_generic(struct rcu_tasks *rtp, char *t\n \n #endif // #ifndef CONFIG_TINY_RCU\n \n-#if defined(CONFIG_TASKS_RCU)\n+#if defined(CONFIG_TASKS_RCU) \u0026\u0026 !defined(CONFIG_TASKS_RCU_TRAMPOLINE_READERS)\n \n ////////////////////////////////////////////////////////////////////////\n //\n@@ -897,10 +897,461 @@ static void rcu_tasks_wait_gp(struct rcu_tasks *rtp)\n \trtp-\u003epostgp_func(rtp);\n }\n \n-#endif /* #if defined(CONFIG_TASKS_RCU) */\n+#endif /* #if defined(CONFIG_TASKS_RCU) \u0026\u0026 !defined(CONFIG_TASKS_RCU_TRAMPOLINE_READERS) */\n \n #ifdef CONFIG_TASKS_RCU\n \n+static int rcu_tasks_lazy_ms = -1;\n+module_param(rcu_tasks_lazy_ms, int, 0444);\n+\n+#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS\n+\n+////////////////////////////////////////////////////////////////////////\n+//\n+// Tasks RCU for architectures whose trampolines are Tasks Trace RCU\n+// readers (CONFIG_HAVE_RCU_TRAMPOLINE_READERS).\n+//\n+// On these architectures every piece of text whose lifetime Tasks RCU\n+// guards -- ftrace trampolines, kprobe optinsn slots, BPF trampoline\n+// images, out-of-line ftrace direct-call trampolines -- enters a Tasks\n+// Trace RCU read-side critical section before calling out of itself and\n+// leaves it before returning, so a task anywhere inside such a call-out,\n+// preempted or not, is an ordinary rcu_read_lock_trace() reader and\n+// synchronize_rcu_tasks_trace() waits for it.\n+//\n+// What that cannot cover is the handful of instructions in the trampoline\n+// before the reader is entered and after it is left, and the one user that\n+// has no trampoline at all: the bytes after a kprobe that the jump\n+// optimizer is about to overwrite. A task can only linger in such\n+// \"unmarked\" text by being interrupted there; unmarked text never calls\n+// anything that could schedule. So a context switch on a CPU tells us that\n+// whatever that CPU was running is out of unmarked text, with one\n+// exception: a preemption from the irq-exit path, which can happen at any\n+// instruction boundary. That path has the interrupted pt_regs in hand, so\n+// just before it preempts it records the IP in the task and checks it\n+// (rcu_tasks_trampoline_text()); if it is inside unmarked text the task\n+// goes on a short holdout list first, and takes itself off again at its\n+// next context switch outside such a preemption or its next irq-exit\n+// check that finds it elsewhere. With that, every pass through\n+// __schedule() is a per-CPU quiescent event, as are usermode and idle.\n+//\n+// A grace period is then:\n+//\n+// 1. Wait for every online CPU to context switch or be found in an RCU\n+// extended quiescent state (deep idle, nohz_full userspace), nudging\n+// stragglers with resched_cpu(). Afterwards no task is in the leading\n+// unmarked instructions of a dying trampoline unless it is on the\n+// holdout list.\n+// 2. Wait for the holdout list (as it stood) to drain.\n+// 3. synchronize_rcu_tasks_trace(), for everything inside the readers.\n+// 4. Repeat 1 and 2 for tasks that have since left the reader and are in\n+// the trailing unmarked instructions.\n+//\n+// which is bounded by a few jiffies plus preempt-off latency plus an SRCU\n+// grace period, independent of how long any task runs without sleeping.\n+// Unlike the classic implementation this does wait for an idle task caught\n+// in a trampoline, since an idle CPU only counts while RCU is not watching\n+// it.\n+\n+static void rcu_tasks_tramp_wait_gp(struct rcu_tasks *rtp);\n+void call_rcu_tasks(struct rcu_head *rhp, rcu_callback_t func);\n+DEFINE_RCU_TASKS(rcu_tasks, rcu_tasks_tramp_wait_gp, call_rcu_tasks, \"RCU Tasks\");\n+\n+/* Per-CPU count of Tasks RCU quiescent events, and the GP kthread's snapshot. */\n+static DEFINE_PER_CPU(unsigned long, rcu_tasks_qs_seq);\n+static DEFINE_PER_CPU(unsigned long, rcu_tasks_qs_snap);\n+\n+/*\n+ * Tasks currently switched out by an irq-exit preemption are kept, with the\n+ * interrupted IP, on the per-CPU rtp_exit_list of the CPU that preempted them\n+ * (reusing the list, lock and task_struct fields the classic flavor uses for\n+ * its exit-path bookkeeping, which this flavor does not need), so that\n+ * rcu_tasks_wait_irq_preempted() can find them without a tasklist scan and\n+ * regardless of where they are in exit.\n+ */\n+\n+/* Tasks last seen preempted inside unmarked trampoline text. */\n+static LIST_HEAD(rcu_tasks_tramp_holdouts);\n+static DEFINE_RAW_SPINLOCK(rcu_tasks_tramp_lock);\n+\n+/* CPUs / holdouts the current grace period is still waiting for. */\n+static struct cpumask rcu_tasks_pending_cpus;\n+static LIST_HEAD(rcu_tasks_gp_holdouts);\n+\n+/**\n+ * arch_rcu_tasks_trampoline_text - Does the architecture treat @ip as unmarked trampoline text?\n+ * @ip: kernel text address inside core kernel text\n+ *\n+ * See rcu_tasks_trampoline_text(). Architectures override this to flag\n+ * core text that runs on behalf of a trampoline outside its Tasks Trace\n+ * reader, e.g. static ftrace entry stubs or return thunks that hold a\n+ * trampoline address they are about to jump to.\n+ */\n+bool __weak arch_rcu_tasks_trampoline_text(unsigned long ip)\n+{\n+\treturn false;\n+}\n+\n+/**\n+ * rcu_tasks_trampoline_text - Is @ip in text Tasks RCU protects but no reader marks?\n+ * @ip: an interrupted instruction pointer\n+ *\n+ * True when a task interrupted at @ip may be executing, or about to enter\n+ * or return into, text whose lifetime depends on synchronize_rcu_tasks()\n+ * without being inside the Tasks Trace reader that text takes around its\n+ * call-outs:\n+ *\n+ * - anything outside core kernel and module text (ftrace and BPF\n+ * trampolines, kprobe slots and other dynamically allocated text; this\n+ * deliberately does not ask is_ftrace_trampoline() and friends, since\n+ * text being torn down may already be unregistered there);\n+ * - whatever the architecture adds via arch_rcu_tasks_trampoline_text();\n+ * - the text of a module that hosts an out-of-line ftrace direct-call\n+ * trampoline (see ftrace_direct_mark_module());\n+ * - the bytes after a kprobe that a pending jump optimization is about to\n+ * overwrite, the one synchronize_rcu_tasks() user with no trampoline.\n+ *\n+ * A false positive only makes the task a holdout until its next quiescent\n+ * event. Called with interrupts disabled from the irq-exit path.\n+ */\n+bool rcu_tasks_trampoline_text(unsigned long ip)\n+{\n+\tbool ret = true;\n+\n+\tif (core_kernel_text(ip))\n+\t\treturn arch_rcu_tasks_trampoline_text(ip) ||\n+\t\t kprobe_in_optimized_region(ip);\n+\n+#ifdef CONFIG_MODULES\n+\tscoped_guard(rcu) {\n+\t\tstruct module *mod = __module_text_address(ip);\n+\n+\t\tif (mod)\n+\t\t\tret = READ_ONCE(mod-\u003eftrace_direct_tramp) ||\n+\t\t\t kprobe_in_optimized_region(ip);\n+\t}\n+#endif\n+\treturn ret;\n+}\n+NOKPROBE_SYMBOL(rcu_tasks_trampoline_text);\n+\n+/* Note a Tasks RCU quiescent event on this CPU. */\n+static void rcu_tasks_qs_event(void)\n+{\n+\tunsigned long *seq;\n+\n+\tguard(preempt_notrace)();\n+\tseq = this_cpu_ptr(\u0026rcu_tasks_qs_seq);\n+\t/* Order a preceding rcu_tasks_tramp_hold() before the count. */\n+\tsmp_store_release(seq, *seq + 1);\n+}\n+\n+static void rcu_tasks_tramp_hold(struct task_struct *t)\n+{\n+\tunsigned long flags;\n+\n+\tif (t-\u003ercu_tasks_holdout)\n+\t\treturn;\n+\traw_spin_lock_irqsave(\u0026rcu_tasks_tramp_lock, flags);\n+\tlist_add_tail(\u0026t-\u003ercu_tasks_holdout_list, \u0026rcu_tasks_tramp_holdouts);\n+\tWRITE_ONCE(t-\u003ercu_tasks_holdout, true);\n+\traw_spin_unlock_irqrestore(\u0026rcu_tasks_tramp_lock, flags);\n+}\n+\n+static void rcu_tasks_tramp_release(struct task_struct *t)\n+{\n+\tunsigned long flags;\n+\n+\tif (likely(!t-\u003ercu_tasks_holdout))\n+\t\treturn;\n+\traw_spin_lock_irqsave(\u0026rcu_tasks_tramp_lock, flags);\n+\tlist_del_init(\u0026t-\u003ercu_tasks_holdout_list);\n+\tWRITE_ONCE(t-\u003ercu_tasks_holdout, false);\n+\traw_spin_unlock_irqrestore(\u0026rcu_tasks_tramp_lock, flags);\n+}\n+\n+/**\n+ * rcu_tasks_irq_resched_enter - Tasks RCU hook for the irq-exit reschedule check\n+ * @ip: instruction pointer of the interrupted (task-level) context\n+ *\n+ * Called with interrupts disabled when an interrupt returning to kernel\n+ * mode is about to preempt_schedule_irq(), the one context switch that can\n+ * catch a task inside unmarked trampoline text. Record where the task is\n+ * parked for as long as it is (rcu_tasks_wait_irq_preempted() looks at\n+ * that), and if it is inside such text make it a holdout before\n+ * __schedule() reports the quiescent event; if it is not, this is as good\n+ * as a voluntary switch for ending an earlier hold.\n+ */\n+void rcu_tasks_irq_resched_enter(unsigned long ip)\n+{\n+\tstruct task_struct *t = current;\n+\tstruct rcu_tasks_percpu *rtpcp = this_cpu_ptr(rcu_tasks.rtpcpu);\n+\n+\tlockdep_assert_irqs_disabled();\n+\tWRITE_ONCE(t-\u003ercu_tasks_irq_ip, ip);\n+\tt-\u003ercu_tasks_exit_cpu = smp_processor_id();\n+\traw_spin_lock_rcu_node(rtpcp);\n+\tlist_add(\u0026t-\u003ercu_tasks_exit_list, \u0026rtpcp-\u003ertp_exit_list);\n+\traw_spin_unlock_rcu_node(rtpcp);\n+\n+\tif (unlikely(rcu_tasks_trampoline_text(ip)))\n+\t\trcu_tasks_tramp_hold(t);\n+\telse\n+\t\trcu_tasks_tramp_release(t);\n+}\n+NOKPROBE_SYMBOL(rcu_tasks_irq_resched_enter);\n+\n+/**\n+ * rcu_tasks_irq_resched_exit - preempt_schedule_irq() has returned\n+ *\n+ * The task is running again (possibly elsewhere) and about to return to the\n+ * interrupted context; it is no longer parked anywhere.\n+ */\n+void rcu_tasks_irq_resched_exit(void)\n+{\n+\tstruct task_struct *t = current;\n+\tstruct rcu_tasks_percpu *rtpcp = per_cpu_ptr(rcu_tasks.rtpcpu, t-\u003ercu_tasks_exit_cpu);\n+\n+\tlockdep_assert_irqs_disabled();\n+\traw_spin_lock_rcu_node(rtpcp);\n+\tlist_del_init(\u0026t-\u003ercu_tasks_exit_list);\n+\traw_spin_unlock_rcu_node(rtpcp);\n+\tWRITE_ONCE(t-\u003ercu_tasks_irq_ip, 0);\n+}\n+NOKPROBE_SYMBOL(rcu_tasks_irq_resched_exit);\n+\n+/**\n+ * rcu_tasks_note_qs - Tasks RCU hook for a context switch or explicit QS\n+ * @t: current\n+ * @preempt: this is a preemption rather than a voluntary switch\n+ *\n+ * Every pass through __schedule() (and cond_resched_tasks_rcu_qs(), and a\n+ * tick from userspace or idle) is a quiescent event for this CPU: unmarked\n+ * trampoline text never calls anything that schedules, and the irq-exit\n+ * path has already made @t a holdout if it is preempting inside such text.\n+ * Any of these outside an irq-exit preemption also shows @t itself to be\n+ * outside, ending an earlier hold -- including cond_resched() under\n+ * PREEMPT_DYNAMIC's none/voluntary modes, where the irq-exit path is off.\n+ */\n+void rcu_tasks_note_qs(struct task_struct *t, bool preempt)\n+{\n+\tWARN_ON_ONCE(t != current);\n+\tif (!READ_ONCE(t-\u003ercu_tasks_irq_ip))\n+\t\trcu_tasks_tramp_release(t);\n+\trcu_tasks_qs_event();\n+}\n+EXPORT_SYMBOL_GPL(rcu_tasks_note_qs);\t/* cond_resched_tasks_rcu_qs() */\n+\n+/**\n+ * rcu_tasks_wait_irq_preempted - wait for tasks irq-preempted inside @inside\n+ * @inside: predicate on a task's recorded irq-exit preemption IP\n+ *\n+ * For a caller about to make some ordinary text unsafe to be parked in\n+ * (the kprobe jump optimizer): once the caller has arranged for\n+ * rcu_tasks_trampoline_text() to cover that text, new irq-exit preemptions\n+ * there become holdouts, but a task preempted there earlier is invisible\n+ * to the grace period. Wait until no parked task's recorded preemption IP\n+ * is inside; a following synchronize_rcu_tasks() then covers the rest.\n+ * The leading synchronize_rcu() orders the caller's arrangement against\n+ * preemptions in flight, which run with interrupts disabled.\n+ */\n+void rcu_tasks_wait_irq_preempted(bool (*inside)(unsigned long ip))\n+{\n+\tstruct task_struct *t;\n+\tunsigned long flags;\n+\tint cpu, kick;\n+\tbool found;\n+\n+\tsynchronize_rcu();\n+\tfor (;;) {\n+\t\tfound = false;\n+\t\tfor_each_possible_cpu(cpu) {\n+\t\t\tstruct rcu_tasks_percpu *rtpcp = per_cpu_ptr(rcu_tasks.rtpcpu, cpu);\n+\n+\t\t\tkick = -1;\n+\t\t\traw_spin_lock_irqsave_rcu_node(rtpcp, flags);\n+\t\t\tlist_for_each_entry(t, \u0026rtpcp-\u003ertp_exit_list, rcu_tasks_exit_list) {\n+\t\t\t\tif (inside(READ_ONCE(t-\u003ercu_tasks_irq_ip))) {\n+\t\t\t\t\tfound = true;\n+\t\t\t\t\tif (task_curr(t))\n+\t\t\t\t\t\tkick = task_cpu(t);\n+\t\t\t\t}\n+\t\t\t}\n+\t\t\traw_spin_unlock_irqrestore_rcu_node(rtpcp, flags);\n+\t\t\tif (kick \u003e= 0)\n+\t\t\t\tresched_cpu(kick);\n+\t\t}\n+\t\tif (!found)\n+\t\t\treturn;\n+\t\tschedule_timeout_uninterruptible(1);\n+\t}\n+}\n+\n+/* Has @cpu passed a quiescent event since the snapshot, or need it not? */\n+static bool rcu_tasks_cpu_quiescent(int cpu)\n+{\n+\tif (!cpu_online(cpu))\n+\t\treturn true;\n+\t/* Pairs with the release in rcu_tasks_qs_event(). */\n+\tif (smp_load_acquire(per_cpu_ptr(\u0026rcu_tasks_qs_seq, cpu)) !=\n+\t per_cpu(rcu_tasks_qs_snap, cpu))\n+\t\treturn true;\n+\t/*\n+\t * Idle or nohz_full userspace in an RCU extended quiescent state: no\n+\t * task-level kernel frames can be live in a trampoline there, and\n+\t * whatever ran before has switched out. An idle CPU that RCU is\n+\t * watching (an interrupt from idle, or the traceable part of the idle\n+\t * loop) is deliberately not let through: the idle task may be in a\n+\t * trampoline with that interrupt on top, and since it is never\n+\t * preempted from irq exit nothing else would catch it. It gets the\n+\t * resched_cpu() like anyone else and counts once the idle loop itself\n+\t * schedules, which it cannot do from inside a trampoline.\n+\t */\n+\treturn !(ct_rcu_watching_cpu(cpu) \u0026 CT_RCU_WATCHING);\n+}\n+\n+/* Rate-limited stall report; returns true if the caller should add detail. */\n+static bool rcu_tasks_tramp_stall(struct rcu_tasks *rtp, unsigned long *lastreport,\n+\t\t\t\t const char *what)\n+{\n+\tint rtst = READ_ONCE(rcu_task_stall_timeout);\n+\n+\tif (rtst \u003c= 0 || !time_after(jiffies, *lastreport + rtst))\n+\t\treturn false;\n+\t*lastreport = jiffies;\n+\tpr_err(\"INFO: %s: %s, grace period %lu is %lu jiffies old\\n\", rtp-\u003ekname,\n+\t what, rcu_seq_current(\u0026rtp-\u003etasks_gp_seq), jiffies - rtp-\u003egp_start);\n+\treturn true;\n+}\n+\n+/*\n+ * Steps 1/4: wait until every online CPU has context switched or is in an\n+ * RCU extended quiescent state. A CPU that has neither after a jiffy is\n+ * asked to switch with resched_cpu(), which takes it through\n+ * rcu_tasks_irq_resched_enter() and __schedule() (or, from userspace, a\n+ * guest or the idle loop, straight to __schedule()).\n+ */\n+static void rcu_tasks_tramp_wait_cpus(struct rcu_tasks *rtp, unsigned long *lastreport)\n+{\n+\tstruct cpumask *pending = \u0026rcu_tasks_pending_cpus;\n+\tunsigned long start;\n+\tint cpu;\n+\n+\t/*\n+\t * The quiescent events run with preemption (in practice interrupts)\n+\t * disabled, so after this any event we go on to count began after the\n+\t * caller's updates -- the unpublished trampoline, and whatever\n+\t * rcu_tasks_trampoline_text() consults -- were visible to it.\n+\t */\n+\tsynchronize_rcu();\n+\n+\tstart = jiffies;\n+\tfor_each_online_cpu(cpu) {\n+\t\tper_cpu(rcu_tasks_qs_snap, cpu) = READ_ONCE(per_cpu(rcu_tasks_qs_seq, cpu));\n+\t\t__cpumask_set_cpu(cpu, pending);\n+\t}\n+\t/* Snapshots before the checks below; pairs with rcu_tasks_qs_event(). */\n+\tsmp_mb();\n+\n+\tfor (;;) {\n+\t\tfor_each_cpu(cpu, pending)\n+\t\t\tif (rcu_tasks_cpu_quiescent(cpu))\n+\t\t\t\t__cpumask_clear_cpu(cpu, pending);\n+\t\tif (cpumask_empty(pending))\n+\t\t\tbreak;\n+\t\tif (time_after(jiffies, start)) {\n+\t\t\tfor_each_cpu(cpu, pending)\n+\t\t\t\tresched_cpu(cpu);\n+\t\t\trtp-\u003en_ipis += cpumask_weight(pending);\n+\t\t}\n+\t\tschedule_timeout_idle(1);\n+\t\tif (rcu_tasks_tramp_stall(rtp, lastreport, \"CPUs without a quiescent event\"))\n+\t\t\tpr_err(\"\\tCPUs: %*pbl\\n\", cpumask_pr_args(pending));\n+\t}\n+}\n+\n+/*\n+ * Steps 2/4: wait for the tasks that were holdouts when we looked to stop\n+ * being holdouts. They are moved to a private list so that tasks becoming\n+ * holdouts later (in live trampolines) cannot keep us here; each removes\n+ * itself via rcu_tasks_tramp_release() wherever it is queued.\n+ */\n+static void rcu_tasks_tramp_wait_holdouts(struct rcu_tasks *rtp, unsigned long *lastreport)\n+{\n+\tstruct task_struct *t;\n+\tunsigned long flags;\n+\tint cpu;\n+\n+\traw_spin_lock_irqsave(\u0026rcu_tasks_tramp_lock, flags);\n+\tlist_splice_tail_init(\u0026rcu_tasks_tramp_holdouts, \u0026rcu_tasks_gp_holdouts);\n+\traw_spin_unlock_irqrestore(\u0026rcu_tasks_tramp_lock, flags);\n+\n+\tfor (;;) {\n+\t\tstruct cpumask *kick = \u0026rcu_tasks_pending_cpus;\n+\t\tstruct task_struct *show[8];\n+\t\tint nshow = 0, i;\n+\t\tbool empty, report;\n+\n+\t\treport = rcu_tasks_tramp_stall(rtp, lastreport,\n+\t\t\t\t\t \"tasks preempted in trampoline text\");\n+\t\tcpumask_clear(kick);\n+\t\traw_spin_lock_irqsave(\u0026rcu_tasks_tramp_lock, flags);\n+\t\tempty = list_empty(\u0026rcu_tasks_gp_holdouts);\n+\t\tlist_for_each_entry(t, \u0026rcu_tasks_gp_holdouts, rcu_tasks_holdout_list) {\n+\t\t\tif (task_curr(t))\n+\t\t\t\t__cpumask_set_cpu(task_cpu(t), kick);\n+\t\t\tif (report \u0026\u0026 nshow \u003c ARRAY_SIZE(show))\n+\t\t\t\tshow[nshow++] = get_task_struct(t);\n+\t\t}\n+\t\traw_spin_unlock_irqrestore(\u0026rcu_tasks_tramp_lock, flags);\n+\t\t/* Never printk under the lock the irq-exit path takes. */\n+\t\tfor (i = 0; i \u003c nshow; i++) {\n+\t\t\tsched_show_task(show[i]);\n+\t\t\tput_task_struct(show[i]);\n+\t\t}\n+\t\tif (empty)\n+\t\t\tbreak;\n+\t\tfor_each_cpu(cpu, kick)\n+\t\t\tresched_cpu(cpu);\n+\t\trtp-\u003en_ipis += cpumask_weight(kick);\n+\t\tschedule_timeout_idle(1);\n+\t}\n+}\n+\n+/* Wait for one trampoline-reader Tasks RCU grace period. */\n+static void rcu_tasks_tramp_wait_gp(struct rcu_tasks *rtp)\n+{\n+\tunsigned long lastreport = jiffies;\n+\n+\tset_tasks_gp_state(rtp, RTGS_WAIT_SCAN_HOLDOUTS);\n+\trcu_tasks_tramp_wait_cpus(rtp, \u0026lastreport);\n+\trcu_tasks_tramp_wait_holdouts(rtp, \u0026lastreport);\n+\n+\tset_tasks_gp_state(rtp, RTGS_WAIT_READERS);\n+\tsynchronize_rcu_tasks_trace();\n+\n+\tset_tasks_gp_state(rtp, RTGS_SCAN_HOLDOUTS);\n+\trcu_tasks_tramp_wait_cpus(rtp, \u0026lastreport);\n+\trcu_tasks_tramp_wait_holdouts(rtp, \u0026lastreport);\n+\n+\tset_tasks_gp_state(rtp, RTGS_POST_GP);\n+}\n+\n+static int __init rcu_spawn_tasks_kthread(void)\n+{\n+\trcu_tasks.gp_sleep = HZ / 10;\n+\tif (rcu_tasks_lazy_ms \u003e= 0)\n+\t\trcu_tasks.lazy_jiffies = msecs_to_jiffies(rcu_tasks_lazy_ms);\n+\trcu_tasks.wait_state = TASK_IDLE;\n+\trcu_spawn_tasks_kthread_generic(\u0026rcu_tasks);\n+\treturn 0;\n+}\n+\n+void exit_tasks_rcu_start(void) { }\n+void exit_tasks_rcu_finish(void) { }\n+\n+#else /* #ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS */\n+\n ////////////////////////////////////////////////////////////////////////\n //\n // Simple variant of RCU whose quiescent states are voluntary context\n@@ -1173,6 +1624,8 @@ static void tasks_rcu_exit_stall(struct timer_list *unused)\n #endif // #ifndef CONFIG_TINY_RCU\n }\n \n+#endif /* #else #ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS */\n+\n /**\n * call_rcu_tasks() - Queue an RCU for invocation task-based grace period\n * @rhp: structure to be used for queueing the RCU updates.\n@@ -1187,6 +1640,12 @@ static void tasks_rcu_exit_stall(struct timer_list *unused)\n * primitives analogous to rcu_read_lock() and rcu_read_unlock() because\n * this primitive is intended to determine that all tasks have passed\n * through a safe state, not so much for data-structure synchronization.\n+ * On CONFIG_TASKS_RCU_TRAMPOLINE_READERS kernels a preemption outside\n+ * trampoline text also ends one, and a reader whose protected window\n+ * spans preemptible code must additionally be a Tasks Trace RCU reader\n+ * (rcu_read_lock_trace(), as the trampolines there take around their\n+ * call-outs); an arbitrary stretch of preemptible kernel code is not\n+ * protected.\n *\n * See the description of call_rcu() for more detailed information on\n * memory ordering guarantees.\n@@ -1205,7 +1664,9 @@ EXPORT_SYMBOL_GPL(call_rcu_tasks);\n * executing rcu-tasks read-side critical sections have elapsed. These\n * read-side critical sections are delimited by calls to schedule(),\n * cond_resched_tasks_rcu_qs(), idle execution, userspace execution, calls\n- * to synchronize_rcu_tasks(), and (in theory, anyway) cond_resched().\n+ * to synchronize_rcu_tasks(), and (in theory, anyway) cond_resched();\n+ * on CONFIG_TASKS_RCU_TRAMPOLINE_READERS kernels also by preemption\n+ * outside trampoline text, see call_rcu_tasks().\n *\n * This is a very specialized primitive, intended only for a few uses in\n * tracing and other situations requiring manipulation of function\n@@ -1233,9 +1694,7 @@ void rcu_barrier_tasks(void)\n }\n EXPORT_SYMBOL_GPL(rcu_barrier_tasks);\n \n-static int rcu_tasks_lazy_ms = -1;\n-module_param(rcu_tasks_lazy_ms, int, 0444);\n-\n+#ifndef CONFIG_TASKS_RCU_TRAMPOLINE_READERS\n static int __init rcu_spawn_tasks_kthread(void)\n {\n \trcu_tasks.gp_sleep = HZ / 10;\n@@ -1251,6 +1710,7 @@ static int __init rcu_spawn_tasks_kthread(void)\n \trcu_spawn_tasks_kthread_generic(\u0026rcu_tasks);\n \treturn 0;\n }\n+#endif /* #ifndef CONFIG_TASKS_RCU_TRAMPOLINE_READERS */\n \n #if !defined(CONFIG_TINY_RCU)\n void show_rcu_tasks_classic_gp_kthread(void)\n@@ -1279,6 +1739,7 @@ void rcu_tasks_get_gp_data(int *flags, unsigned long *gp_seq)\n }\n EXPORT_SYMBOL_GPL(rcu_tasks_get_gp_data);\n \n+#ifndef CONFIG_TASKS_RCU_TRAMPOLINE_READERS\n /*\n * Protect against tasklist scan blind spot while the task is exiting and\n * may be removed from the tasklist. Do this by adding the task to yet\n@@ -1322,6 +1783,7 @@ void exit_tasks_rcu_finish(void)\n \tlist_del_init(\u0026t-\u003ercu_tasks_exit_list);\n \traw_spin_unlock_irqrestore_rcu_node(rtpcp, flags);\n }\n+#endif /* #ifndef CONFIG_TASKS_RCU_TRAMPOLINE_READERS */\n \n #else /* #ifdef CONFIG_TASKS_RCU */\n void exit_tasks_rcu_start(void) { }\ndiff --git a/kernel/rcu/update.c b/kernel/rcu/update.c\nindex b62735a678842..a122b8d1effb5 100644\n--- a/kernel/rcu/update.c\n+++ b/kernel/rcu/update.c\n@@ -40,7 +40,9 @@\n #include \u003clinux/tick.h\u003e\n #include \u003clinux/rcupdate_wait.h\u003e\n #include \u003clinux/sched/isolation.h\u003e\n+#include \u003clinux/context_tracking_state.h\u003e\n #include \u003clinux/kprobes.h\u003e\n+#include \u003clinux/module.h\u003e\n #include \u003clinux/slab.h\u003e\n #include \u003clinux/irq_work.h\u003e\n #include \u003clinux/rcupdate_trace.h\u003e\ndiff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c\nindex 53d5db60bfa56..f69f715913585 100644\n--- a/kernel/trace/ftrace.c\n+++ b/kernel/trace/ftrace.c\n@@ -6076,6 +6076,29 @@ static void reset_direct(struct ftrace_ops *ops, unsigned long addr)\n \tops-\u003etrampoline = 0;\n }\n \n+/*\n+ * A direct trampoline may live in module text rather than in dynamically\n+ * allocated text that rcu_tasks_trampoline_text() recognises on its own (see\n+ * samples/ftrace/ftrace-direct*.c). The trampoline itself must be a Tasks\n+ * Trace reader across its call-out (see register_ftrace_direct()); marking the\n+ * owning module here covers the instructions before it enters that reader and\n+ * after it leaves it, where a task interrupted in the module's text must not be\n+ * counted as Tasks-RCU quiescent, so that ftrace_shutdown()'s\n+ * synchronize_rcu_tasks() still keeps the module text from being freed under\n+ * it.\n+ */\n+static void ftrace_direct_mark_module(unsigned long addr)\n+{\n+#if defined(CONFIG_MODULES) \u0026\u0026 defined(CONFIG_TASKS_RCU_TRAMPOLINE_READERS)\n+\tstruct module *mod;\n+\n+\tguard(rcu)();\n+\tmod = __module_text_address(addr);\n+\tif (mod)\n+\t\tWRITE_ONCE(mod-\u003eftrace_direct_tramp, true);\n+#endif\n+}\n+\n /**\n * register_ftrace_direct - Call a custom trampoline directly\n * for multiple functions registered in @ops\n@@ -6090,6 +6113,17 @@ static void reset_direct(struct ftrace_ops *ops, unsigned long addr)\n * and save the parameters of the function being traced, and restore them\n * (or inject new ones if needed), before returning.\n *\n+ * Nothing but Tasks RCU keeps the trampoline at @addr alive while a task is\n+ * executing it or is preempted in something it called. On architectures that\n+ * select HAVE_RCU_TRAMPOLINE_READERS, Tasks RCU only waits for such a task if\n+ * it is a Tasks Trace RCU reader, so the trampoline must enter one\n+ * (rcu_read_lock_trace() or an assembly equivalent) before calling out and\n+ * leave it before returning, just as that option requires of the in-kernel\n+ * ftrace and BPF trampolines. The few instructions before and after are\n+ * covered by the irq-exit check: automatically for trampolines outside kernel\n+ * and module text (e.g. BPF images), and via ftrace_direct_mark_module() for\n+ * trampolines in module text.\n+ *\n * Returns:\n * 0 on success\n * -EINVAL - The @ops object was already registered with this call or\n@@ -6169,6 +6203,7 @@ int register_ftrace_direct(struct ftrace_ops *ops, unsigned long addr)\n \tops-\u003eflags |= MULTI_FLAGS;\n \tops-\u003etrampoline = FTRACE_REGS_ADDR;\n \tops-\u003edirect_call = addr;\n+\tftrace_direct_mark_module(addr);\n \n \terr = register_ftrace_function_nolock(ops);\n \tif (err)\n@@ -6237,6 +6272,8 @@ __modify_ftrace_direct(struct ftrace_ops *ops, unsigned long addr)\n \n \tlockdep_assert_held_once(\u0026direct_mutex);\n \n+\tftrace_direct_mark_module(addr);\n+\n \t/* Enable the tmp_ops to have the same functions as the direct ops */\n \tftrace_ops_init(\u0026tmp_ops);\n \ttmp_ops.func_hash = ops-\u003efunc_hash;\n@@ -6419,6 +6456,7 @@ int update_ftrace_direct_add(struct ftrace_ops *ops, struct ftrace_hash *hash)\n \t\thlist_for_each_entry(entry, \u0026hash-\u003ebuckets[i], hlist) {\n \t\t\tif (__ftrace_lookup_ip(direct_functions, entry-\u003eip))\n \t\t\t\tgoto out_unlock;\n+\t\t\tftrace_direct_mark_module(entry-\u003edirect);\n \t\t}\n \t}\n \n@@ -6702,6 +6740,7 @@ int update_ftrace_direct_mod(struct ftrace_ops *ops, struct ftrace_hash *hash, b\n \t\t\ttmp = __ftrace_lookup_ip(direct_hash, entry-\u003eip);\n \t\t\tif (!tmp)\n \t\t\t\tcontinue;\n+\t\t\tftrace_direct_mark_module(entry-\u003edirect);\n \t\t\ttmp-\u003edirect = entry-\u003edirect;\n \t\t}\n \t}\ndiff --git a/samples/ftrace/ftrace-direct-modify.c b/samples/ftrace/ftrace-direct-modify.c\nindex 164d9dd6fd921..937c8d8c2a1be 100644\n--- a/samples/ftrace/ftrace-direct-modify.c\n+++ b/samples/ftrace/ftrace-direct-modify.c\n@@ -2,6 +2,7 @@\n #include \u003clinux/module.h\u003e\n #include \u003clinux/kthread.h\u003e\n #include \u003clinux/ftrace.h\u003e\n+#include \"ftrace-direct.h\"\n #if !defined(CONFIG_ARM64) \u0026\u0026 !defined(CONFIG_PPC32)\n #include \u003casm/asm-offsets.h\u003e\n #endif\n@@ -73,7 +74,9 @@ asm (\n \"\tpushq %rbp\\n\"\n \"\tmovq %rsp, %rbp\\n\"\n \tCALL_DEPTH_ACCOUNT\n+\tTRACE_RCU_READ_LOCK\n \"\tcall my_direct_func1\\n\"\n+\tTRACE_RCU_READ_UNLOCK\n \"\tleave\\n\"\n \tASM_RET\n \"\t.size\t\tmy_tramp1, .-my_tramp1\\n\"\n@@ -85,7 +88,9 @@ asm (\n \"\tpushq %rbp\\n\"\n \"\tmovq %rsp, %rbp\\n\"\n \tCALL_DEPTH_ACCOUNT\n+\tTRACE_RCU_READ_LOCK\n \"\tcall my_direct_func2\\n\"\n+\tTRACE_RCU_READ_UNLOCK\n \"\tleave\\n\"\n \tASM_RET\n \"\t.size\t\tmy_tramp2, .-my_tramp2\\n\"\n@@ -141,11 +146,13 @@ asm (\n \"\t.globl\t\tmy_tramp1\\n\"\n \" my_tramp1:\"\n \"\thint\t34\\n\" // bti\tc\n+\tTRACE_RCU_READ_LOCK\n \"\tsub\tsp, sp, #16\\n\"\n \"\tstp\tx9, x30, [sp]\\n\"\n \"\tbl\tmy_direct_func1\\n\"\n \"\tldp\tx30, x9, [sp]\\n\"\n \"\tadd\tsp, sp, #16\\n\"\n+\tTRACE_RCU_READ_UNLOCK\n \"\tret\tx9\\n\"\n \"\t.size\t\tmy_tramp1, .-my_tramp1\\n\"\n \n@@ -153,11 +160,13 @@ asm (\n \"\t.globl\t\tmy_tramp2\\n\"\n \" my_tramp2:\"\n \"\thint\t34\\n\" // bti\tc\n+\tTRACE_RCU_READ_LOCK\n \"\tsub\tsp, sp, #16\\n\"\n \"\tstp\tx9, x30, [sp]\\n\"\n \"\tbl\tmy_direct_func2\\n\"\n \"\tldp\tx30, x9, [sp]\\n\"\n \"\tadd\tsp, sp, #16\\n\"\n+\tTRACE_RCU_READ_UNLOCK\n \"\tret\tx9\\n\"\n \"\t.size\t\tmy_tramp2, .-my_tramp2\\n\"\n \"\t.popsection\\n\"\ndiff --git a/samples/ftrace/ftrace-direct-multi-modify.c b/samples/ftrace/ftrace-direct-multi-modify.c\nindex b03766c6217bc..e12e5c8b83f06 100644\n--- a/samples/ftrace/ftrace-direct-multi-modify.c\n+++ b/samples/ftrace/ftrace-direct-multi-modify.c\n@@ -2,6 +2,7 @@\n #include \u003clinux/module.h\u003e\n #include \u003clinux/kthread.h\u003e\n #include \u003clinux/ftrace.h\u003e\n+#include \"ftrace-direct.h\"\n #if !defined(CONFIG_ARM64) \u0026\u0026 !defined(CONFIG_PPC32)\n #include \u003casm/asm-offsets.h\u003e\n #endif\n@@ -77,10 +78,12 @@ asm (\n \"\tpushq %rbp\\n\"\n \"\tmovq %rsp, %rbp\\n\"\n \tCALL_DEPTH_ACCOUNT\n+\tTRACE_RCU_READ_LOCK\n \"\tpushq %rdi\\n\"\n \"\tmovq 8(%rbp), %rdi\\n\"\n \"\tcall my_direct_func1\\n\"\n \"\tpopq %rdi\\n\"\n+\tTRACE_RCU_READ_UNLOCK\n \"\tleave\\n\"\n \tASM_RET\n \"\t.size\t\tmy_tramp1, .-my_tramp1\\n\"\n@@ -92,10 +95,12 @@ asm (\n \"\tpushq %rbp\\n\"\n \"\tmovq %rsp, %rbp\\n\"\n \tCALL_DEPTH_ACCOUNT\n+\tTRACE_RCU_READ_LOCK\n \"\tpushq %rdi\\n\"\n \"\tmovq 8(%rbp), %rdi\\n\"\n \"\tcall my_direct_func2\\n\"\n \"\tpopq %rdi\\n\"\n+\tTRACE_RCU_READ_UNLOCK\n \"\tleave\\n\"\n \tASM_RET\n \"\t.size\t\tmy_tramp2, .-my_tramp2\\n\"\n@@ -154,6 +159,7 @@ asm (\n \"\t.globl\t\tmy_tramp1\\n\"\n \" my_tramp1:\"\n \"\thint\t34\\n\" // bti\tc\n+\tTRACE_RCU_READ_LOCK\n \"\tsub\tsp, sp, #32\\n\"\n \"\tstp\tx9, x30, [sp]\\n\"\n \"\tstr\tx0, [sp, #16]\\n\"\n@@ -162,6 +168,7 @@ asm (\n \"\tldp\tx30, x9, [sp]\\n\"\n \"\tldr\tx0, [sp, #16]\\n\"\n \"\tadd\tsp, sp, #32\\n\"\n+\tTRACE_RCU_READ_UNLOCK\n \"\tret\tx9\\n\"\n \"\t.size\t\tmy_tramp1, .-my_tramp1\\n\"\n \n@@ -169,6 +176,7 @@ asm (\n \"\t.globl\t\tmy_tramp2\\n\"\n \" my_tramp2:\"\n \"\thint\t34\\n\" // bti\tc\n+\tTRACE_RCU_READ_LOCK\n \"\tsub\tsp, sp, #32\\n\"\n \"\tstp\tx9, x30, [sp]\\n\"\n \"\tstr\tx0, [sp, #16]\\n\"\n@@ -177,6 +185,7 @@ asm (\n \"\tldp\tx30, x9, [sp]\\n\"\n \"\tldr\tx0, [sp, #16]\\n\"\n \"\tadd\tsp, sp, #32\\n\"\n+\tTRACE_RCU_READ_UNLOCK\n \"\tret\tx9\\n\"\n \"\t.size\t\tmy_tramp2, .-my_tramp2\\n\"\n \"\t.popsection\\n\"\ndiff --git a/samples/ftrace/ftrace-direct-multi.c b/samples/ftrace/ftrace-direct-multi.c\nindex 3fe6ddaf0b69f..a970464ed3786 100644\n--- a/samples/ftrace/ftrace-direct-multi.c\n+++ b/samples/ftrace/ftrace-direct-multi.c\n@@ -3,6 +3,7 @@\n \n #include \u003clinux/mm.h\u003e /* for handle_mm_fault() */\n #include \u003clinux/ftrace.h\u003e\n+#include \"ftrace-direct.h\"\n #include \u003clinux/sched/stat.h\u003e\n #if !defined(CONFIG_ARM64) \u0026\u0026 !defined(CONFIG_PPC32)\n #include \u003casm/asm-offsets.h\u003e\n@@ -56,10 +57,12 @@ asm (\n \"\tpushq %rbp\\n\"\n \"\tmovq %rsp, %rbp\\n\"\n \tCALL_DEPTH_ACCOUNT\n+\tTRACE_RCU_READ_LOCK\n \"\tpushq %rdi\\n\"\n \"\tmovq 8(%rbp), %rdi\\n\"\n \"\tcall my_direct_func\\n\"\n \"\tpopq %rdi\\n\"\n+\tTRACE_RCU_READ_UNLOCK\n \"\tleave\\n\"\n \tASM_RET\n \"\t.size\t\tmy_tramp, .-my_tramp\\n\"\n@@ -101,6 +104,7 @@ asm (\n \"\t.globl\t\tmy_tramp\\n\"\n \" my_tramp:\"\n \"\thint\t34\\n\" // bti\tc\n+\tTRACE_RCU_READ_LOCK\n \"\tsub\tsp, sp, #32\\n\"\n \"\tstp\tx9, x30, [sp]\\n\"\n \"\tstr\tx0, [sp, #16]\\n\"\n@@ -109,6 +113,7 @@ asm (\n \"\tldp\tx30, x9, [sp]\\n\"\n \"\tldr\tx0, [sp, #16]\\n\"\n \"\tadd\tsp, sp, #32\\n\"\n+\tTRACE_RCU_READ_UNLOCK\n \"\tret\tx9\\n\"\n \"\t.size\t\tmy_tramp, .-my_tramp\\n\"\n \"\t.popsection\\n\"\ndiff --git a/samples/ftrace/ftrace-direct-too.c b/samples/ftrace/ftrace-direct-too.c\nindex bf2411aa6fd7a..abc098c2ab7a8 100644\n--- a/samples/ftrace/ftrace-direct-too.c\n+++ b/samples/ftrace/ftrace-direct-too.c\n@@ -3,6 +3,7 @@\n \n #include \u003clinux/mm.h\u003e /* for handle_mm_fault() */\n #include \u003clinux/ftrace.h\u003e\n+#include \"ftrace-direct.h\"\n #if !defined(CONFIG_ARM64) \u0026\u0026 !defined(CONFIG_PPC32)\n #include \u003casm/asm-offsets.h\u003e\n #endif\n@@ -61,6 +62,7 @@ asm (\n \"\tpushq %rbp\\n\"\n \"\tmovq %rsp, %rbp\\n\"\n \tCALL_DEPTH_ACCOUNT\n+\tTRACE_RCU_READ_LOCK\n \"\tpushq %rdi\\n\"\n \"\tpushq %rsi\\n\"\n \"\tpushq %rdx\\n\"\n@@ -70,6 +72,7 @@ asm (\n \"\tpopq %rdx\\n\"\n \"\tpopq %rsi\\n\"\n \"\tpopq %rdi\\n\"\n+\tTRACE_RCU_READ_UNLOCK\n \"\tleave\\n\"\n \tASM_RET\n \"\t.size\t\tmy_tramp, .-my_tramp\\n\"\n@@ -110,6 +113,7 @@ asm (\n \"\t.globl\t\tmy_tramp\\n\"\n \" my_tramp:\"\n \"\thint\t34\\n\" // bti\tc\n+\tTRACE_RCU_READ_LOCK\n \"\tsub\tsp, sp, #48\\n\"\n \"\tstp\tx9, x30, [sp]\\n\"\n \"\tstp\tx0, x1, [sp, #16]\\n\"\n@@ -119,6 +123,7 @@ asm (\n \"\tldp\tx0, x1, [sp, #16]\\n\"\n \"\tldp\tx2, x3, [sp, #32]\\n\"\n \"\tadd\tsp, sp, #48\\n\"\n+\tTRACE_RCU_READ_UNLOCK\n \"\tret\tx9\\n\"\n \"\t.size\t\tmy_tramp, .-my_tramp\\n\"\n \"\t.popsection\\n\"\ndiff --git a/samples/ftrace/ftrace-direct.c b/samples/ftrace/ftrace-direct.c\nindex 5368c8c39cbb4..99b65ad2fccc0 100644\n--- a/samples/ftrace/ftrace-direct.c\n+++ b/samples/ftrace/ftrace-direct.c\n@@ -3,6 +3,7 @@\n \n #include \u003clinux/sched.h\u003e /* for wake_up_process() */\n #include \u003clinux/ftrace.h\u003e\n+#include \"ftrace-direct.h\"\n #if !defined(CONFIG_ARM64) \u0026\u0026 !defined(CONFIG_PPC32)\n #include \u003casm/asm-offsets.h\u003e\n #endif\n@@ -54,9 +55,11 @@ asm (\n \"\tpushq %rbp\\n\"\n \"\tmovq %rsp, %rbp\\n\"\n \tCALL_DEPTH_ACCOUNT\n+\tTRACE_RCU_READ_LOCK\n \"\tpushq %rdi\\n\"\n \"\tcall my_direct_func\\n\"\n \"\tpopq %rdi\\n\"\n+\tTRACE_RCU_READ_UNLOCK\n \"\tleave\\n\"\n \tASM_RET\n \"\t.size\t\tmy_tramp, .-my_tramp\\n\"\n@@ -97,6 +100,7 @@ asm (\n \"\t.globl\t\tmy_tramp\\n\"\n \" my_tramp:\"\n \"\thint\t34\\n\" // bti\tc\n+\tTRACE_RCU_READ_LOCK\n \"\tsub\tsp, sp, #32\\n\"\n \"\tstp\tx9, x30, [sp]\\n\"\n \"\tstr\tx0, [sp, #16]\\n\"\n@@ -104,6 +108,7 @@ asm (\n \"\tldp\tx30, x9, [sp]\\n\"\n \"\tldr\tx0, [sp, #16]\\n\"\n \"\tadd\tsp, sp, #32\\n\"\n+\tTRACE_RCU_READ_UNLOCK\n \"\tret\tx9\\n\"\n \"\t.size\t\tmy_tramp, .-my_tramp\\n\"\n \"\t.popsection\\n\"\ndiff --git a/samples/ftrace/ftrace-direct.h b/samples/ftrace/ftrace-direct.h\nnew file mode 100644\nindex 0000000000000..726f67048ff5c\n--- /dev/null\n+++ b/samples/ftrace/ftrace-direct.h\n@@ -0,0 +1,126 @@\n+/* SPDX-License-Identifier: GPL-2.0-only */\n+#ifndef _SAMPLES_FTRACE_DIRECT_H\n+#define _SAMPLES_FTRACE_DIRECT_H\n+\n+#include \u003clinux/stringify.h\u003e\n+\n+/*\n+ * A direct-call trampoline is entered with no lock, refcount or RCU marker\n+ * held; only Tasks RCU keeps it (and, for a module, its text) alive while a\n+ * task is inside it or preempted in something it called. On architectures\n+ * that select HAVE_RCU_TRAMPOLINE_READERS, Tasks RCU only waits for such a\n+ * task while it is a Tasks Trace RCU reader, so the trampoline must enter one\n+ * before calling out and leave it afterwards, exactly like the ftrace and BPF\n+ * trampolines do. See register_ftrace_direct(). The instructions before the\n+ * lock and after the unlock are covered by ftrace_direct_mark_module().\n+ *\n+ * These are rcu_read_lock_trace() / rcu_read_unlock_trace() open-coded as\n+ * instruction strings for use inside the samples' asm() trampolines, after\n+ * the versions in arch/x86/kernel/ftrace_64.S and\n+ * arch/arm64/kernel/entry-ftrace.S. The scratch registers are caller-saved\n+ * and not argument registers, so they are dead on entry to and exit from an\n+ * fentry trampoline; the flags are clobbered.\n+ *\n+ * The generated asm-offsets.h is only pulled in on the architectures that need\n+ * it here: it is not generally safe to include from C (e.g. PPC32's TASK_SIZE\n+ * and arm64's TRAMP_VALIAS clash with the C definitions).\n+ */\n+#if defined(CONFIG_TASKS_RCU_TRAMPOLINE_READERS) \u0026\u0026 defined(CONFIG_X86_64)\n+\n+#include \u003casm/asm-offsets.h\u003e\n+\n+#ifndef CONFIG_TASKS_TRACE_RCU_NO_MB\n+#define TRACE_RCU_MB\t\"\tlock addl $0, -4(%rsp)\\n\"\n+#else\n+#define TRACE_RCU_MB\n+#endif\n+\n+#define TRACE_RCU_READ_LOCK\t\t\t\t\t\t\t\\\n+\t\"\tmovq %gs:current_task(%rip), %r11\\n\"\t\t\t\t\t\\\n+\t\"\tmovl \" __stringify(TASK_trc_reader_nesting) \"(%r11), %r10d\\n\"\t\t\\\n+\t\"\tincl \" __stringify(TASK_trc_reader_nesting) \"(%r11)\\n\"\t\t\\\n+\t\"\ttestl %r10d, %r10d\\n\"\t\t\t\t\t\t\t\\\n+\t\"\tjnz 771f\\n\"\t\t\t\t\t\t\t\t\\\n+\t\"\tmovq rcu_tasks_trace_srcu_struct+\" __stringify(SRCU_srcu_ctrp) \"(%rip), %r10\\n\" \\\n+\t\"\tincq %gs:\" __stringify(SRCU_CTR_srcu_locks) \"(%r10)\\n\"\t\t\t\\\n+\t\"\tmovq %r10, \" __stringify(TASK_trc_reader_scp) \"(%r11)\\n\"\t\t\\\n+\tTRACE_RCU_MB\t\t\t\t\t\t\t\t\\\n+\t\"771:\\n\"\n+\n+#define TRACE_RCU_READ_UNLOCK\t\t\t\t\t\t\t\\\n+\t\"\tmovq %gs:current_task(%rip), %r11\\n\"\t\t\t\t\t\\\n+\t\"\tmovl \" __stringify(TASK_trc_reader_nesting) \"(%r11), %r10d\\n\"\t\t\\\n+\t\"\tsubl $1, %r10d\\n\"\t\t\t\t\t\t\t\\\n+\t\"\tjnz 772f\\n\"\t\t\t\t\t\t\t\t\\\n+\t\"\tmovq \" __stringify(TASK_trc_reader_scp) \"(%r11), %r10\\n\"\t\t\\\n+\t\"\tmovl $0, \" __stringify(TASK_trc_reader_nesting) \"(%r11)\\n\"\t\t\\\n+\tTRACE_RCU_MB\t\t\t\t\t\t\t\t\\\n+\t\"\tincq %gs:\" __stringify(SRCU_CTR_srcu_unlocks) \"(%r10)\\n\"\t\t\\\n+\t\"\tjmp 773f\\n\"\t\t\t\t\t\t\t\t\\\n+\t\"772:\tmovl %r10d, \" __stringify(TASK_trc_reader_nesting) \"(%r11)\\n\"\t\\\n+\t\"773:\\n\"\n+\n+#elif defined(CONFIG_TASKS_RCU_TRAMPOLINE_READERS) \u0026\u0026 defined(CONFIG_ARM64)\n+\n+#include \u003casm/alternative-macros.h\u003e\n+#include \u003casm/cpucaps.h\u003e\n+/* arm64's asm-offsets.h redefines TRAMP_VALIAS from \u003casm/fixmap.h\u003e. */\n+#pragma push_macro(\"TRAMP_VALIAS\")\n+#undef TRAMP_VALIAS\n+#include \u003casm/asm-offsets.h\u003e\n+#pragma pop_macro(\"TRAMP_VALIAS\")\n+\n+#ifndef CONFIG_TASKS_TRACE_RCU_NO_MB\n+#define TRACE_RCU_MB\t\"\tdmb\tish\\n\"\n+#else\n+#define TRACE_RCU_MB\n+#endif\n+\n+#define TRACE_RCU_SRCU_CTRP\t\"rcu_tasks_trace_srcu_struct+\" __stringify(SRCU_SRCU_CTRP)\n+\n+/* x14 = this CPU's offset; then atomically increment the long at x14 + \\areg */\n+#define TRACE_RCU_PERCPU_INC(areg)\t\t\t\t\t\t\\\n+\tALTERNATIVE(\"\tmrs\tx14, tpidr_el1\\n\", \"\tmrs\tx14, tpidr_el2\\n\",\t\t\\\n+\t\t ARM64_HAS_VIRT_HOST_EXTN)\t\t\t\t\t\\\n+\t\"\tadd\tx14, x14, \" areg \"\\n\"\t\t\t\t\t\t\\\n+\t\"778:\tldxr\tx15, [x14]\\n\"\t\t\t\t\t\t\t\\\n+\t\"\tadd\tx15, x15, #1\\n\"\t\t\t\t\t\t\t\\\n+\t\"\tstxr\tw16, x15, [x14]\\n\"\t\t\t\t\t\t\\\n+\t\"\tcbnz\tw16, 778b\\n\"\n+\n+#define TRACE_RCU_READ_LOCK\t\t\t\t\t\t\t\\\n+\t\"\tmrs\tx12, sp_el0\\n\"\t\t\t\t\t\t\t\\\n+\t\"\tldr\tw13, [x12, #\" __stringify(TSK_TRC_READER_NESTING) \"]\\n\"\t\\\n+\t\"\tadd\tw14, w13, #1\\n\"\t\t\t\t\t\t\t\\\n+\t\"\tstr\tw14, [x12, #\" __stringify(TSK_TRC_READER_NESTING) \"]\\n\"\t\\\n+\t\"\tcbnz\tw13, 771f\\n\"\t\t\t\t\t\t\t\\\n+\t\"\tadrp\tx13, \" TRACE_RCU_SRCU_CTRP \"\\n\"\t\t\t\t\t\\\n+\t\"\tldr\tx13, [x13, #:lo12:\" TRACE_RCU_SRCU_CTRP \"]\\n\"\t\t\t\\\n+\t\"\tstr\tx13, [x12, #\" __stringify(TSK_TRC_READER_SCP) \"]\\n\"\t\t\\\n+\t\"\tadd\tx13, x13, #\" __stringify(SRCU_CTR_SRCU_LOCKS) \"\\n\"\t\t\\\n+\tTRACE_RCU_PERCPU_INC(\"x13\")\t\t\t\t\t\t\\\n+\tTRACE_RCU_MB\t\t\t\t\t\t\t\t\\\n+\t\"771:\\n\"\n+\n+#define TRACE_RCU_READ_UNLOCK\t\t\t\t\t\t\t\\\n+\t\"\tmrs\tx12, sp_el0\\n\"\t\t\t\t\t\t\t\\\n+\t\"\tldr\tw13, [x12, #\" __stringify(TSK_TRC_READER_NESTING) \"]\\n\"\t\\\n+\t\"\tsubs\tw13, w13, #1\\n\"\t\t\t\t\t\t\t\\\n+\t\"\tb.ne\t772f\\n\"\t\t\t\t\t\t\t\t\\\n+\t\"\tldr\tx13, [x12, #\" __stringify(TSK_TRC_READER_SCP) \"]\\n\"\t\t\\\n+\t\"\tstr\twzr, [x12, #\" __stringify(TSK_TRC_READER_NESTING) \"]\\n\"\t\\\n+\tTRACE_RCU_MB\t\t\t\t\t\t\t\t\\\n+\t\"\tadd\tx13, x13, #\" __stringify(SRCU_CTR_SRCU_UNLOCKS) \"\\n\"\t\t\\\n+\tTRACE_RCU_PERCPU_INC(\"x13\")\t\t\t\t\t\t\\\n+\t\"\tb\t773f\\n\"\t\t\t\t\t\t\t\t\\\n+\t\"772:\tstr\tw13, [x12, #\" __stringify(TSK_TRC_READER_NESTING) \"]\\n\"\t\\\n+\t\"773:\\n\"\n+\n+#else\n+\n+#define TRACE_RCU_READ_LOCK\n+#define TRACE_RCU_READ_UNLOCK\n+\n+#endif\n+\n+#endif /* _SAMPLES_FTRACE_DIRECT_H */\n"
} |
| 2/1 | 2026/09/24 04:18 | agent | patch-evaluator |
3mModel:core Args: null Results: {
"EnableConfigs": [
"TASKS_RCU_TRAMPOLINE_READERS",
"HAVE_RCU_TRAMPOLINE_READERS",
"TASKS_RCU",
"DYNAMIC_FTRACE",
"DYNAMIC_FTRACE_WITH_DIRECT_CALLS",
"KPROBES",
"OPTPROBES",
"BPF_SYSCALL",
"BPF_JIT",
"SMP",
"PREEMPTION"
],
"FocusSymbols": [
"rcu_tasks_tramp_wait_gp",
"rcu_tasks_irq_resched_enter",
"rcu_tasks_irq_resched_exit",
"rcu_tasks_note_qs",
"rcu_tasks_wait_irq_preempted",
"rcu_tasks_trampoline_text",
"arch_rcu_tasks_trampoline_text",
"raw_irqentry_exit_cond_resched",
"kprobe_in_optimized_region",
"arch_prepare_bpf_trampoline",
"register_ftrace_direct",
"kprobe_optimizer"
],
"Reasoning": "The patch series introduces CONFIG_TASKS_RCU_TRAMPOLINE_READERS and CONFIG_HAVE_RCU_TRAMPOLINE_READERS, restructuring how Tasks RCU grace periods and quiescent states are tracked on x86_64 and arm64. It embeds Tasks Trace RCU reader state transitions inside ftrace, optprobe, and BPF trampolines, hooks into kernel irq-exit preemption paths (raw_irqentry_exit_cond_resched) to monitor tasks preempted in unmarked trampoline text, adds synchronization in kprobe_optimizer, and introduces runtime assertions like rcu_tasks_trace_assert_idle(). This fundamentally alters core concurrency and synchronization primitives in reachable paths on amd64, warranting fuzzing.",
"WorthFuzzing": true
}Instruction: You are an expert Linux kernel maintainer and security engineer.
Your job is to review a provided patch series and evaluate whether it warrants fuzzing with syzkaller.
IMPORTANT: The changes have ALREADY been applied and committed as the HEAD commit in
your workspace. Do NOT rely on internal assumptions. You must actively use your code access
tools to inspect the actual source code, callers, and surrounding context.
================================================================================
1. CORE TRIAGE PHILOSOPHY
================================================================================
The goal of patch fuzzing is to discover crashes, regressions, exposed latent bugs,
and newly triggered assertions introduced by the patch series.
- REACHABILITY IS THE PRIMARY GATE:
Fuzzing can only discover bugs in code that can actually execute in standard virtualized
environments (GCE or QEMU, utilizing software-emulated devices like USB gadgets, netdev, tun/tap).
If the modified code is structurally unreachable (see Section 2), it MUST NOT be fuzzed,
regardless of whether it adds assertions or complex logic.
- DO NOT BLINDLY TRUST "NO FUNCTIONAL CHANGE" (NFCI) OR "REFACTORING" CLAIMS:
Patch authors routinely label changes as "cleanups", "refactorings", or state
"No functional change intended". Do NOT take these claims at face value.
Code refactorings that rearrange logic, introduce helper functions, or alter state management
in core subsystems frequently introduce subtle semantic shifts or uncover latent kernel bugs.
If reachable executable code is modified or refactored, it MUST be fuzzed.
- NEW OR MODIFIED ASSERTIONS IN REACHABLE CODE MUST BE FUZZED:
When a patch introduces or modifies runtime checks or assertions (e.g., WARN_ON*, VM_WARN_ON*,
BUG_ON*, lockdep_assert*) in reachable code paths, it enforces new or stricter invariants.
Even if the author believes the invariant always holds, fuzzing is essential to verify whether
an unusual sequence of operations can violate it.
================================================================================
2. WHEN TO RETURN WorthFuzzing=false (NEGATIVE CRITERIA)
================================================================================
Return WorthFuzzing=false ONLY IF all modified code falls strictly into one or more of these categories:
- Non-kernel and non-executable changes:
* Modifications to Documentation/, comments, or spelling fixes.
* User-space directories, self-tests, samples, or scripts (e.g., tools/, samples/, scripts/, usr/)
that do not affect the compiled kernel image (vmlinux) or kernel modules.
* Purely decorative logging (e.g., message strings in pr_err, printk, dev_info) or tracepoints
that do not alter control flow or data structures.
* Build system or Kconfig changes that do not alter compiled C logic.
- Structurally unreachable hardware:
* Vendor-specific PCIe switches, SmartNICs, or GPU drivers (e.g., mlxsw, pds_core, qed,
ionic, amdgpu) requiring physical ASIC/PCIe cards not emulated in standard QEMU.
- Unreachable execution paths:
* Driver teardown callbacks (.remove, .shutdown, pci_unregister_driver) executed only during
physical PCI hot-unplug or manual sysfs driver unbinding.
* Code paths exclusive to architectures other than the target architecture.
================================================================================
3. WHEN TO RETURN WorthFuzzing=true (POSITIVE CRITERIA)
================================================================================
Return WorthFuzzing=true whenever the patch touches reachable executable code, including:
- Core Subsystems:
* Any logic modifications in memory management (mm/), synchronization/locking (kernel/locking/),
BPF, scheduler, core networking, VFS, or syscall handling.
- Refactorings and Code Cleanups:
* Any restructuring of reachable data structures, helper abstractions, or algorithm flows.
- Runtime Assertions and Defensive Checks:
* Any introduction or alteration of assertions (WARN_ON*, VM_WARN_ON*, BUG_ON*, etc.) in reachable paths.
- Reachable Drivers and Protocols:
* Drivers accessible via virtual buses (virtio, USB gadget, loopback, netlink, binder, sockets, etc.).
================================================================================
4. EXTRACTING FocusSymbols (PREVENTING DILUTION)
================================================================================
When WorthFuzzing=true, you must extract specific kernel functions into FocusSymbols to guide the fuzzer:
- AVOID UBIQUITOUS LIFECYCLE HOT-PATHS:
Do NOT list generic, ubiquitous functions called by almost every program in the corpus
(including, but not limited to: general memory allocators and deallocators, page fault
and trap handlers, or core synchronization primitives; this is not an exhaustive list).
Listing ubiquitous functions causes the fuzzer to classify thousands of unrelated tests as "focused",
which severely dilutes fuzzing effort away from the actual changes.
- TARGET SPECIFIC FEATURE LOGIC AND ENTRYPOINTS:
List functions that specifically implement the logic being added or altered, or direct API entrypoints
for the subsystem feature under review.
- HANDLING STATIC INLINE FUNCTIONS IN HEADERS (.h):
Compiler-inlined static functions (such as static inlines in mm/*.h or include/linux/*.h) lack
distinct symbol addresses in vmlinux and cannot be targeted directly by symbol coverage filters.
If the changes are primarily in static inline helpers, identify non-static, feature-specific caller
functions in .c files that exercise them (avoiding ubiquitous lifecycle wrappers).
================================================================================
5. IDENTIFYING EnableConfigs
================================================================================
Identify any specific CONFIG_ options required to properly compile and reach the modified code:
- Inspect Kconfig files and #ifdef guards; do not make assumptions.
- Check "depends on" lines in Kconfig to include any non-standard parent subsystem configs needed.
- Strip any 'CONFIG_' prefix (e.g., return "NET_IPV4" instead of "CONFIG_NET_IPV4").
Prefer calling several tools at the same time to save round-trips.
Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.
Prompt: Target architecture: amd64
For your convenience, here is the diff of the changes:
commit 671c17ef89b17f280a59ec7bb4ed9bfe9789cf90
Author: syz-cluster <triage@syzkaller.com>
Date: Thu Sep 24 04:18:56 2026 +0000
syz-cluster: applied patch under review
diff --git a/Documentation/RCU/Design/Requirements/Requirements.rst b/Documentation/RCU/Design/Requirements/Requirements.rst
index 8101fe6229d57..c059c463e083f 100644
--- a/Documentation/RCU/Design/Requirements/Requirements.rst
+++ b/Documentation/RCU/Design/Requirements/Requirements.rst
@@ -2756,6 +2756,30 @@ synchronize_rcu(), and rcu_barrier(), respectively. In
three APIs are therefore implemented by separate functions that check
for voluntary context switches.
+Architectures that select ``CONFIG_HAVE_RCU_TRAMPOLINE_READERS`` keep the
+same three APIs but implement the grace period differently
+(``CONFIG_TASKS_RCU_TRAMPOLINE_READERS``). There, every trampoline whose
+lifetime Tasks RCU guards enters a Tasks Trace RCU read-side critical
+section (rcu_read_lock_trace() or its assembly equivalent) before calling
+out and leaves it before returning, so a task anywhere inside such a
+call-out is an ordinary Tasks Trace reader whether or not it is
+preempted. The few trampoline instructions outside that reader can only
+be occupied by a task that was interrupted there, so the grace period
+additionally waits for each CPU to pass through a context switch, and the
+irq-exit preemption path, the only switch that can catch a task inside
+such text (rcu_tasks_trampoline_text()), briefly makes such a task a
+holdout until it is next seen elsewhere. On such kernels an involuntary
+context switch outside trampoline text *is* a Tasks-RCU quiescent state,
+a Tasks RCU grace period no longer depends on how long any task runs
+without sleeping, cond_resched_tasks_rcu_qs() is unnecessary, and the
+obligation moves to the trampolines: anything that relies on
+synchronize_rcu_tasks() to protect code a task may be preempted in must
+take the Tasks Trace reader (see register_ftrace_direct()), or, where
+that is impossible because the code is ordinary text with no trampoline
+of its own, make it known to rcu_tasks_trampoline_text() and wait out
+tasks already parked there with rcu_tasks_wait_irq_preempted(), as the
+kprobe jump optimizer does.
+
Tasks Rude RCU
~~~~~~~~~~~~~~
diff --git a/Documentation/RCU/checklist.rst b/Documentation/RCU/checklist.rst
index 4b30f701225fd..87dd506bffefb 100644
--- a/Documentation/RCU/checklist.rst
+++ b/Documentation/RCU/checklist.rst
@@ -252,7 +252,12 @@ over a rather long period of time, but improvements are always welcome!
a. If the updater uses synchronize_rcu_tasks() or
call_rcu_tasks(), then the readers must refrain from
executing voluntary context switches, that is, from
- blocking.
+ blocking. On architectures that select
+ CONFIG_HAVE_RCU_TRAMPOLINE_READERS a reader must in
+ addition be a Tasks Trace RCU reader (that is what the
+ trampolines there do around their call-outs) or be text
+ that rcu_tasks_trampoline_text() recognises; an arbitrary
+ stretch of preemptible kernel code is not protected.
b. If the updater uses call_rcu_tasks_trace()
or synchronize_rcu_tasks_trace(), then the
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index b5a51b0ef9440..bf0e56006863e 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -218,6 +218,7 @@ config ARM64
select HAVE_PERF_REGS
select HAVE_PERF_USER_STACK_DUMP
select HAVE_PREEMPT_DYNAMIC_KEY
+ select HAVE_RCU_TRAMPOLINE_READERS if DYNAMIC_FTRACE_WITH_ARGS
select HAVE_REGS_AND_STACK_ACCESS_API
select HAVE_RELIABLE_STACKTRACE
select HAVE_POSIX_CPU_TIMERS_TASK_WORK
diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
index 9c853ed3ceab8..f6a8fb1f9b436 100644
--- a/arch/arm64/kernel/asm-offsets.c
+++ b/arch/arm64/kernel/asm-offsets.c
@@ -10,6 +10,7 @@
#include <linux/arm_sdei.h>
#include <linux/sched.h>
+#include <linux/srcu.h>
#include <linux/ftrace.h>
#include <linux/kexec.h>
#include <linux/mm.h>
@@ -39,6 +40,13 @@ int main(void)
DEFINE(TSK_STACK, offsetof(struct task_struct, stack));
#ifdef CONFIG_STACKPROTECTOR
DEFINE(TSK_STACK_CANARY, offsetof(struct task_struct, stack_canary));
+#endif
+#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS
+ DEFINE(TSK_TRC_READER_NESTING, offsetof(struct task_struct, trc_reader_nesting));
+ DEFINE(TSK_TRC_READER_SCP, offsetof(struct task_struct, trc_reader_scp));
+ DEFINE(SRCU_SRCU_CTRP, offsetof(struct srcu_struct, srcu_ctrp));
+ DEFINE(SRCU_CTR_SRCU_LOCKS, offsetof(struct srcu_ctr, srcu_locks));
+ DEFINE(SRCU_CTR_SRCU_UNLOCKS, offsetof(struct srcu_ctr, srcu_unlocks));
#endif
BLANK();
DEFINE(THREAD_CPU_CONTEXT, offsetof(struct task_struct, thread.cpu_context));
diff --git a/arch/arm64/kernel/entry-ftrace.S b/arch/arm64/kernel/entry-ftrace.S
index 025140caafe74..fc2805eb9e15b 100644
--- a/arch/arm64/kernel/entry-ftrace.S
+++ b/arch/arm64/kernel/entry-ftrace.S
@@ -14,6 +14,72 @@
#include <asm/insn.h>
#ifdef CONFIG_DYNAMIC_FTRACE_WITH_ARGS
+/*
+ * Open-coded rcu_read_lock_trace() / rcu_read_unlock_trace(), see
+ * include/linux/rcupdate_trace.h and CONFIG_HAVE_RCU_TRAMPOLINE_READERS. The
+ * whole of ftrace_caller is treated as trampoline text by the irq-exit check
+ * (see arch_rcu_tasks_trampoline_text()), so these only need to bracket the
+ * call out to ops->func; everything before the lock and after the unlock,
+ * including the direct-call tails that carry a BPF trampoline address in x17,
+ * is covered by that.
+ *
+ * The SRCU-fast per-CPU increment is done LL/SC on this CPU's counter; being
+ * migrated between reading the per-CPU offset and the store-exclusive only
+ * means another CPU's counter is (atomically) incremented, which SRCU sums
+ * over anyway. Ordering between the nesting count and the scp stash only
+ * matters against interrupts on this CPU, which observe program order.
+ * Clobbers x12-x16 and the flags.
+ */
+ .macro trace_rcu_srcu_inc, addr:req, tmp:req, wtmp2:req
+8888: ldxr \tmp, [\addr]
+ add \tmp, \tmp, #1
+ stxr \wtmp2, \tmp, [\addr]
+ cbnz \wtmp2, 8888b
+ .endm
+
+ .macro trace_rcu_read_lock
+#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS
+ mrs x12, sp_el0 // current
+ ldr w13, [x12, #TSK_TRC_READER_NESTING]
+ add w14, w13, #1
+ str w14, [x12, #TSK_TRC_READER_NESTING]
+ cbnz w13, .Ltrl_nested\@ // interrupted a reader: done
+ ldr_l x13, rcu_tasks_trace_srcu_struct + SRCU_SRCU_CTRP
+ str x13, [x12, #TSK_TRC_READER_SCP]
+ get_this_cpu_offset x14
+ add x14, x14, x13
+ add x14, x14, #SRCU_CTR_SRCU_LOCKS
+ trace_rcu_srcu_inc x14, x15, w16
+#ifndef CONFIG_TASKS_TRACE_RCU_NO_MB
+ dmb ish
+#endif
+.Ltrl_nested\@:
+#endif
+ .endm
+
+ .macro trace_rcu_read_unlock
+#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS
+ mrs x12, sp_el0 // current
+ ldr w13, [x12, #TSK_TRC_READER_NESTING]
+ subs w13, w13, #1
+ b.ne .Ltru_nested\@
+ /* Outermost: pick up scp before an interrupt can see nesting == 0. */
+ ldr x14, [x12, #TSK_TRC_READER_SCP]
+ str wzr, [x12, #TSK_TRC_READER_NESTING]
+#ifndef CONFIG_TASKS_TRACE_RCU_NO_MB
+ dmb ish
+#endif
+ get_this_cpu_offset x15
+ add x14, x14, x15
+ add x14, x14, #SRCU_CTR_SRCU_UNLOCKS
+ trace_rcu_srcu_inc x14, x15, w16
+ b .Ltru_done\@
+.Ltru_nested\@:
+ str w13, [x12, #TSK_TRC_READER_NESTING]
+.Ltru_done\@:
+#endif
+ .endm
+
/*
* Due to -fpatchable-function-entry=2, the compiler has placed two NOPs before
* the regular function prologue. For an enabled callsite, ftrace_init_nop() and
@@ -94,6 +160,8 @@ SYM_CODE_START(ftrace_caller)
stp x29, x30, [sp, #FREGS_SIZE]
add x29, sp, #FREGS_SIZE
+ trace_rcu_read_lock
+
/* Prepare arguments for the tracer func */
sub x0, x30, #AARCH64_INSN_SIZE // ip (callsite's BL insn)
mov x1, x9 // parent_ip (callsite's LR)
@@ -111,6 +179,8 @@ SYM_INNER_LABEL(ftrace_call, SYM_L_GLOBAL)
bl ftrace_stub // func(ip, parent_ip, op, regs)
#endif
+ trace_rcu_read_unlock
+
/*
* At the callsite x0-x8 and x19-x30 were live. Any C code will have preserved
* x19-x29 per the AAPCS, and we created frame records upon entry, so we need
@@ -178,6 +248,10 @@ SYM_CODE_START(ftrace_stub_direct_tramp)
SYM_CODE_END(ftrace_stub_direct_tramp)
#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */
+/* End of [ftrace_caller, ...) for arch_rcu_tasks_trampoline_text(). */
+SYM_CODE_START(ftrace_static_tramp_end)
+SYM_CODE_END(ftrace_static_tramp_end)
+
#else /* CONFIG_DYNAMIC_FTRACE_WITH_ARGS */
/*
diff --git a/arch/arm64/kernel/ftrace.c b/arch/arm64/kernel/ftrace.c
index e1a3c0b3a0514..5f4193f15cd9a 100644
--- a/arch/arm64/kernel/ftrace.c
+++ b/arch/arm64/kernel/ftrace.c
@@ -17,6 +17,26 @@
#include <asm/insn.h>
#include <asm/text-patching.h>
+#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS
+extern void ftrace_static_tramp_end(void);
+
+/* The SRCU-fast increments in entry-ftrace.S are the this_cpu_inc() form. */
+static_assert(!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE));
+
+/*
+ * See rcu_tasks_trampoline_text(). ftrace_caller and ftrace_stub_direct_tramp
+ * are core kernel text but must be treated as trampolines: a task interrupted
+ * in them outside the Tasks Trace reader may be carrying an ops pointer (x11)
+ * or a direct-call BPF trampoline address (x17) whose lifetime is guarded only
+ * by Tasks RCU.
+ */
+bool arch_rcu_tasks_trampoline_text(unsigned long ip)
+{
+ return ip >= (unsigned long)ftrace_caller &&
+ ip < (unsigned long)ftrace_static_tramp_end;
+}
+#endif
+
#ifdef CONFIG_DYNAMIC_FTRACE_WITH_ARGS
struct fregs_offset {
const char *name;
diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
index c18e005a41dbe..87bbcca0298fd 100644
--- a/arch/arm64/net/bpf_jit_comp.c
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -14,6 +14,7 @@
#include <linux/filter.h>
#include <linux/memory.h>
#include <linux/printk.h>
+#include <linux/rcupdate_trace.h>
#include <linux/slab.h>
#include <asm/asm-extable.h>
@@ -2591,6 +2592,77 @@ static void emit_arena_arg_conv(struct jit_ctx *ctx, u8 dst, u8 src, bool nullab
emit(A64_SUB(0, dst, src, base_lo), ctx);
}
+/*
+ * Open-coded rcu_read_lock_trace() / rcu_read_unlock_trace() for the
+ * trampoline, see CONFIG_HAVE_RCU_TRAMPOLINE_READERS and the equivalent macros
+ * in arch/arm64/kernel/entry-ftrace.S. The SRCU-fast per-CPU increment is an
+ * LL/SC add on this CPU's counter; being migrated between reading the per-CPU
+ * offset and the store-exclusive only means another CPU's counter is
+ * incremented atomically instead, which SRCU sums over anyway. Uses x10-x15,
+ * which are scratch at every emission point, and the flags.
+ */
+static void emit_trace_rcu_reader(struct jit_ctx *ctx, bool lock)
+{
+#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS
+ const int nesting_off = offsetof(struct task_struct, trc_reader_nesting);
+ const int scp_off = offsetof(struct task_struct, trc_reader_scp);
+ const int ctr_off = lock ? offsetof(struct srcu_ctr, srcu_locks)
+ : offsetof(struct srcu_ctr, srcu_unlocks);
+ const bool mb = !IS_ENABLED(CONFIG_TASKS_TRACE_RCU_NO_MB);
+ const u8 tsk = A64_R(10), n = A64_R(11), tmp = A64_R(12);
+ const u8 ctr = A64_R(13), addr = A64_R(14), val = A64_R(15);
+
+ /* The plain this_cpu_inc() form of __srcu_read_lock_fast(). */
+ BUILD_BUG_ON(IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE));
+ /* LDR/STR (immediate, unsigned offset) ranges */
+ BUILD_BUG_ON((nesting_off & 3) || nesting_off >= SZ_16K ||
+ (scp_off & 7) || scp_off >= SZ_32K);
+
+ emit(A64_MRS_SP_EL0(tsk), ctx); /* current */
+ emit(A64_LDR32I(n, tsk, nesting_off), ctx);
+ if (lock) {
+ emit(A64_ADD_I(0, tmp, n, 1), ctx);
+ emit(A64_STR32I(tmp, tsk, nesting_off), ctx);
+ /* interrupted a reader: done (branch to 1:) */
+ emit(A64_CBNZ(0, n, 13 + mb), ctx);
+ /* scp = rcu_tasks_trace_srcu_struct.srcu_ctrp; current->trc_reader_scp = scp */
+ emit_addr_mov_i64(ctr, (u64)&rcu_tasks_trace_srcu_struct.srcu_ctrp, ctx);
+ emit(A64_LDR64I(ctr, ctr, 0), ctx);
+ emit(A64_STR64I(ctr, tsk, scp_off), ctx);
+ } else {
+ emit(A64_SUB_I(0, n, n, 1), ctx);
+ /* still nested: just store the count (branch to 2:) */
+ emit(A64_CBNZ(0, n, 11 + mb), ctx);
+ /* outermost: pick up scp before an interrupt can see nesting == 0 */
+ emit(A64_LDR64I(ctr, tsk, scp_off), ctx);
+ emit(A64_STR32I(A64_ZR, tsk, nesting_off), ctx);
+ if (mb)
+ emit(A64_DMB_ISH, ctx);
+ }
+ /* this_cpu_inc(scp->srcu_locks / scp->srcu_unlocks) */
+ if (cpus_have_cap(ARM64_HAS_VIRT_HOST_EXTN))
+ emit(A64_MRS_TPIDR_EL2(addr), ctx);
+ else
+ emit(A64_MRS_TPIDR_EL1(addr), ctx);
+ emit(A64_ADD(1, addr, addr, ctr), ctx);
+ emit(A64_ADD_I(1, addr, addr, ctr_off), ctx);
+ emit(A64_LDXR(1, val, addr), ctx);
+ emit(A64_ADD_I(1, val, val, 1), ctx);
+ emit(A64_STXR(1, val, addr, tmp), ctx);
+ emit(A64_CBNZ(0, tmp, -3), ctx);
+ if (lock) {
+ if (mb)
+ emit(A64_DMB_ISH, ctx);
+ /* 1: */
+ } else {
+ emit(A64_B(2), ctx);
+ /* 2: */
+ emit(A64_STR32I(n, tsk, nesting_off), ctx);
+ /* 3: */
+ }
+#endif
+}
+
static void save_args(struct jit_ctx *ctx, int bargs_off, int oargs_off,
const struct btf_func_model *m, const struct arg_aux *a,
bool for_call_origin, bool is_struct_ops, u64 arena_base)
@@ -2854,6 +2926,16 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,
emit(A64_STR64I(A64_R(19), A64_SP, regs_off), ctx);
emit(A64_STR64I(A64_R(20), A64_SP, regs_off + 8), ctx);
+ /*
+ * Tasks RCU keeps this image alive only while we are a Tasks Trace
+ * reader; the instructions before this point (and after the final
+ * unlock) are covered by the irq-exit IP check. One reader spans
+ * __bpf_tramp_enter() and the fentry/fmod_ret progs, a second one the
+ * fexit progs and __bpf_tramp_exit(); the original function runs
+ * outside both, with the image pinned by im->pcref instead.
+ */
+ emit_trace_rcu_reader(ctx, true);
+
if (flags & BPF_TRAMP_F_CALL_ORIG) {
/* for the first pass, assume the worst case */
if (!ctx->image)
@@ -2898,12 +2980,20 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,
if (flags & BPF_TRAMP_F_CALL_ORIG) {
/* the original func takes kernel addresses, never converted ones */
save_args(ctx, bargs_off, oargs_off, m, a, true, is_struct_ops, 0);
+ emit_trace_rcu_reader(ctx, false);
/* call original func */
emit(A64_LDR64I(A64_R(10), A64_SP, retaddr_off), ctx);
emit(A64_ADR(A64_LR, AARCH64_INSN_SIZE * 2), ctx);
emit(A64_RET(A64_R(10)), ctx);
/* store return value */
emit(A64_STR64I(A64_R(0), A64_SP, retval_off), ctx);
+ /*
+ * Second reader. Taken before ip_after_call so that the branch
+ * to the epilogue patched in at teardown is inside it too; the
+ * fmod_ret early exit lands past this still holding the first
+ * reader, so either way exactly one is held.
+ */
+ emit_trace_rcu_reader(ctx, true);
/* reserve a nop for bpf_tramp_image_put */
im->ip_after_call = ctx->ro_image + ctx->idx;
emit(A64_NOP, ctx);
@@ -2945,6 +3035,9 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,
if (flags & BPF_TRAMP_F_RESTORE_REGS)
restore_args(ctx, bargs_off, a->regs_for_args);
+ /* Remaining instructions are covered by the irq-exit IP check. */
+ emit_trace_rcu_reader(ctx, false);
+
/* restore callee saved register x19 and x20 */
emit(A64_LDR64I(A64_R(19), A64_SP, regs_off), ctx);
emit(A64_LDR64I(A64_R(20), A64_SP, regs_off + 8), ctx);
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 15fd9ec5ecacb..64c3814eb7459 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -288,6 +288,7 @@ config X86
select MMU_GATHER_RCU_TABLE_FREE
select MMU_GATHER_MERGE_VMAS
select HAVE_POSIX_CPU_TIMERS_TASK_WORK
+ select HAVE_RCU_TRAMPOLINE_READERS if X86_64 && SMP && DYNAMIC_FTRACE
select HAVE_REGS_AND_STACK_ACCESS_API
select HAVE_RELIABLE_STACKTRACE if UNWINDER_ORC || STACK_VALIDATION
select HAVE_FUNCTION_ARG_ACCESS_API
diff --git a/arch/x86/kernel/asm-offsets.c b/arch/x86/kernel/asm-offsets.c
index 081816888f7a0..876c3986419a2 100644
--- a/arch/x86/kernel/asm-offsets.c
+++ b/arch/x86/kernel/asm-offsets.c
@@ -9,6 +9,7 @@
#include <linux/crypto.h>
#include <crypto/aria.h>
#include <linux/sched.h>
+#include <linux/srcu.h>
#include <linux/stddef.h>
#include <linux/hardirq.h>
#include <linux/suspend.h>
@@ -46,6 +47,13 @@ static void __used common(void)
#ifdef CONFIG_STACKPROTECTOR
OFFSET(TASK_stack_canary, task_struct, stack_canary);
#endif
+#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS
+ OFFSET(TASK_trc_reader_nesting, task_struct, trc_reader_nesting);
+ OFFSET(TASK_trc_reader_scp, task_struct, trc_reader_scp);
+ OFFSET(SRCU_srcu_ctrp, srcu_struct, srcu_ctrp);
+ OFFSET(SRCU_CTR_srcu_locks, srcu_ctr, srcu_locks);
+ OFFSET(SRCU_CTR_srcu_unlocks, srcu_ctr, srcu_unlocks);
+#endif
BLANK();
OFFSET(pbe_address, pbe, address);
diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c
index 17d6edfcb7e00..9babaed483eb7 100644
--- a/arch/x86/kernel/ftrace.c
+++ b/arch/x86/kernel/ftrace.c
@@ -275,6 +275,49 @@ static inline void tramp_free(void *tramp)
execmem_free(tramp);
}
+#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS
+extern void ftrace_static_tramp_end(void);
+extern char __return_thunk_start[], __return_thunk_end[];
+extern char __rethunk_safe_start[], __rethunk_safe_end[];
+
+/*
+ * The SRCU-fast increments in TRACE_RCU_READ_LOCK/UNLOCK (ftrace_64.S) are the
+ * this_cpu_inc() form.
+ */
+static_assert(!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE));
+
+/*
+ * See rcu_tasks_trampoline_text(). Some core kernel text behaves like a
+ * trampoline for Tasks RCU purposes because a task executing there outside
+ * any Tasks Trace reader may still be about to enter a Tasks-RCU-protected
+ * trampoline whose address it already holds:
+ *
+ * - the static ftrace_caller / ftrace_regs_caller / ftrace_stub_direct_tramp
+ * stubs, which carry a direct-call target on the stack until their final
+ * RET, and
+ * - the return thunks that RET expands to under CONFIG_MITIGATION_RETHUNK,
+ * which run after leaving the stubs above and before landing in that
+ * target.
+ */
+bool arch_rcu_tasks_trampoline_text(unsigned long ip)
+{
+ if (ip >= (unsigned long)ftrace_caller &&
+ ip < (unsigned long)ftrace_static_tramp_end)
+ return true;
+#ifdef CONFIG_MITIGATION_RETPOLINE
+ if (ip >= (unsigned long)__return_thunk_start &&
+ ip < (unsigned long)__return_thunk_end)
+ return true;
+#endif
+#ifdef CONFIG_MITIGATION_SRSO
+ if (ip >= (unsigned long)__rethunk_safe_start &&
+ ip < (unsigned long)__rethunk_safe_end)
+ return true;
+#endif
+ return false;
+}
+#endif /* CONFIG_TASKS_RCU_TRAMPOLINE_READERS */
+
/* Defined as markers to the end of the ftrace default trampolines */
extern void ftrace_regs_caller_end(void);
extern void ftrace_caller_end(void);
diff --git a/arch/x86/kernel/ftrace_64.S b/arch/x86/kernel/ftrace_64.S
index 62c1c93aa1c6a..5d8cb38619781 100644
--- a/arch/x86/kernel/ftrace_64.S
+++ b/arch/x86/kernel/ftrace_64.S
@@ -7,6 +7,7 @@
#include <linux/cfi_types.h>
#include <linux/linkage.h>
#include <asm/asm-offsets.h>
+#include <asm/percpu.h>
#include <asm/ptrace.h>
#include <asm/ftrace.h>
#include <asm/nospec-branch.h>
@@ -145,6 +146,53 @@ SYM_FUNC_END(ftrace_stub_graph)
#ifdef CONFIG_DYNAMIC_FTRACE
+/*
+ * Open-coded rcu_read_lock_trace() / rcu_read_unlock_trace(), see
+ * include/linux/rcupdate_trace.h and CONFIG_HAVE_RCU_TRAMPOLINE_READERS: the
+ * trampoline and the ftrace_ops it is about to load are kept alive by Tasks
+ * RCU only while we are inside this reader, so the lock must precede the
+ * function_trace_op load and the unlock must follow the call. These live
+ * inside the region copied into dynamic trampolines; the %rip-relative
+ * references are fixed up by text_poke_apply_relocation() in
+ * create_trampoline(). Clobbers %rax, %rcx and flags.
+ */
+.macro TRACE_RCU_READ_LOCK
+#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS
+ movq PER_CPU_VAR(current_task), %rcx
+ movl TASK_trc_reader_nesting(%rcx), %eax
+ incl TASK_trc_reader_nesting(%rcx)
+ testl %eax, %eax
+ jnz .Ltrl_nested_\@
+ movq rcu_tasks_trace_srcu_struct+SRCU_srcu_ctrp(%rip), %rax
+ incq %gs:SRCU_CTR_srcu_locks(%rax)
+ movq %rax, TASK_trc_reader_scp(%rcx)
+#ifndef CONFIG_TASKS_TRACE_RCU_NO_MB
+ lock addl $0, -4(%rsp) /* smp_mb() */
+#endif
+.Ltrl_nested_\@:
+#endif
+.endm
+
+.macro TRACE_RCU_READ_UNLOCK
+#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS
+ movq PER_CPU_VAR(current_task), %rcx
+ movl TASK_trc_reader_nesting(%rcx), %eax
+ subl $1, %eax
+ jnz .Ltru_nested_\@
+ /* Outermost: pick up scp before an interrupt can see nesting == 0. */
+ movq TASK_trc_reader_scp(%rcx), %rax
+ movl $0, TASK_trc_reader_nesting(%rcx)
+#ifndef CONFIG_TASKS_TRACE_RCU_NO_MB
+ lock addl $0, -4(%rsp) /* smp_mb() */
+#endif
+ incq %gs:SRCU_CTR_srcu_unlocks(%rax)
+ jmp .Ltru_done_\@
+.Ltru_nested_\@:
+ movl %eax, TASK_trc_reader_nesting(%rcx)
+.Ltru_done_\@:
+#endif
+.endm
+
SYM_FUNC_START(__fentry__)
ANNOTATE_NOENDBR
CALL_DEPTH_ACCOUNT
@@ -163,6 +211,8 @@ SYM_FUNC_START(ftrace_caller)
leaq MCOUNT_REG_SIZE+8(%rsp), %rcx
movq %rcx, RSP(%rsp)
+ TRACE_RCU_READ_LOCK
+
SYM_INNER_LABEL(ftrace_caller_op_ptr, SYM_L_GLOBAL)
ANNOTATE_NOENDBR
/* Load the ftrace_ops into the 3rd parameter */
@@ -181,6 +231,8 @@ SYM_INNER_LABEL(ftrace_call, SYM_L_GLOBAL)
ANNOTATE_NOENDBR
call ftrace_stub
+ TRACE_RCU_READ_UNLOCK
+
/* Handlers can change the RIP */
movq RIP(%rsp), %rax
movq %rax, MCOUNT_REG_SIZE(%rsp)
@@ -209,6 +261,8 @@ SYM_FUNC_START(ftrace_regs_caller)
CALL_DEPTH_ACCOUNT
+ TRACE_RCU_READ_LOCK
+
SYM_INNER_LABEL(ftrace_regs_caller_op_ptr, SYM_L_GLOBAL)
ANNOTATE_NOENDBR
/* Load the ftrace_ops into the 3rd parameter */
@@ -246,6 +300,8 @@ SYM_INNER_LABEL(ftrace_regs_call, SYM_L_GLOBAL)
ANNOTATE_NOENDBR
call ftrace_stub
+ TRACE_RCU_READ_UNLOCK
+
/* Copy flags back to SS, to restore them */
movq EFLAGS(%rsp), %rax
movq %rax, MCOUNT_REG_SIZE(%rsp)
@@ -328,6 +384,19 @@ SYM_FUNC_START(ftrace_stub_direct_tramp)
RET
SYM_FUNC_END(ftrace_stub_direct_tramp)
+/*
+ * [ftrace_caller, ftrace_static_tramp_end) is treated as trampoline text by
+ * rcu_tasks_trampoline_text(): outside TRACE_RCU_READ_LOCK/UNLOCK the stubs
+ * may still hold a direct-call trampoline address (ORIG_RAX / the return
+ * address they RET to) that only Tasks RCU keeps alive. With return thunks
+ * the RET itself runs elsewhere; arch_rcu_tasks_trampoline_text() covers
+ * those too.
+ */
+SYM_CODE_START_NOALIGN(ftrace_static_tramp_end)
+ UNWIND_HINT_UNDEFINED
+ ANNOTATE_NOENDBR
+SYM_CODE_END(ftrace_static_tramp_end)
+
#else /* ! CONFIG_DYNAMIC_FTRACE */
SYM_FUNC_START(__fentry__)
diff --git a/arch/x86/kernel/kprobes/opt.c b/arch/x86/kernel/kprobes/opt.c
index 3f8fea52619f9..68a5de6cdabe7 100644
--- a/arch/x86/kernel/kprobes/opt.c
+++ b/arch/x86/kernel/kprobes/opt.c
@@ -31,6 +31,7 @@
#include <asm/set_memory.h>
#include <asm/sections.h>
#include <asm/nospec-branch.h>
+#include <asm/asm-offsets.h>
#include "common.h"
@@ -101,6 +102,47 @@ static void synthesize_set_arg1(kprobe_opcode_t *addr, unsigned long val)
*(unsigned long *)addr = val;
}
+/*
+ * Open-coded rcu_read_lock_trace() / rcu_read_unlock_trace() around the call
+ * to optimized_callback(), see CONFIG_HAVE_RCU_TRAMPOLINE_READERS and the
+ * equivalent macros in ftrace_64.S. The template is memcpy()d into the slot
+ * without relocation processing, so memory references must be absolute
+ * rather than %rip-relative. %rax and %rcx are free at both points.
+ */
+#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS
+#ifndef CONFIG_TASKS_TRACE_RCU_NO_MB
+#define OPTPROBE_TRACE_RCU_MB " lock addl $0, -4(%rsp)\n"
+#else
+#define OPTPROBE_TRACE_RCU_MB
+#endif
+#define OPTPROBE_TRACE_RCU_READ_LOCK \
+ " movq %gs:current_task, %rcx\n" \
+ " movl " __stringify(TASK_trc_reader_nesting) "(%rcx), %eax\n" \
+ " incl " __stringify(TASK_trc_reader_nesting) "(%rcx)\n" \
+ " testl %eax, %eax\n" \
+ " jnz 1f\n" \
+ " movq rcu_tasks_trace_srcu_struct+" __stringify(SRCU_srcu_ctrp) ", %rax\n" \
+ " incq %gs:" __stringify(SRCU_CTR_srcu_locks) "(%rax)\n" \
+ " movq %rax, " __stringify(TASK_trc_reader_scp) "(%rcx)\n" \
+ OPTPROBE_TRACE_RCU_MB \
+ "1:\n"
+#define OPTPROBE_TRACE_RCU_READ_UNLOCK \
+ " movq %gs:current_task, %rcx\n" \
+ " movl " __stringify(TASK_trc_reader_nesting) "(%rcx), %eax\n" \
+ " subl $1, %eax\n" \
+ " jnz 2f\n" \
+ " movq " __stringify(TASK_trc_reader_scp) "(%rcx), %rax\n" \
+ " movl $0, " __stringify(TASK_trc_reader_nesting) "(%rcx)\n" \
+ OPTPROBE_TRACE_RCU_MB \
+ " incq %gs:" __stringify(SRCU_CTR_srcu_unlocks) "(%rax)\n" \
+ " jmp 3f\n" \
+ "2: movl %eax, " __stringify(TASK_trc_reader_nesting) "(%rcx)\n" \
+ "3:\n"
+#else
+#define OPTPROBE_TRACE_RCU_READ_LOCK
+#define OPTPROBE_TRACE_RCU_READ_UNLOCK
+#endif
+
asm (
".pushsection .rodata\n"
".global optprobe_template_entry\n"
@@ -114,6 +156,7 @@ asm (
"optprobe_template_clac:\n"
ASM_NOP3
SAVE_REGS_STRING
+ OPTPROBE_TRACE_RCU_READ_LOCK
" movq %rsp, %rsi\n"
".global optprobe_template_val\n"
"optprobe_template_val:\n"
@@ -122,6 +165,7 @@ asm (
".global optprobe_template_call\n"
"optprobe_template_call:\n"
ASM_NOP5
+ OPTPROBE_TRACE_RCU_READ_UNLOCK
/* Copy 'regs->flags' into 'regs->ss'. */
" movq 18*8(%rsp), %rdx\n"
" movq %rdx, 20*8(%rsp)\n"
diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S
index 2438b89a4620a..e546283dc2679 100644
--- a/arch/x86/kernel/vmlinux.lds.S
+++ b/arch/x86/kernel/vmlinux.lds.S
@@ -151,7 +151,9 @@ SECTIONS
* definition.
*/
. = srso_alias_untrain_ret | (1 << 2) | (1 << 8) | (1 << 14) | (1 << 20);
+ __rethunk_safe_start = .;
*(.text..__x86.rethunk_safe)
+ __rethunk_safe_end = .;
#endif
ALIGN_ENTRY_TEXT_END
@@ -162,7 +164,9 @@ SECTIONS
SOFTIRQENTRY_TEXT
#ifdef CONFIG_MITIGATION_RETPOLINE
*(.text..__x86.indirect_thunk)
+ __return_thunk_start = .;
*(.text..__x86.return_thunk)
+ __return_thunk_end = .;
#endif
STATIC_CALL_TEXT
*(.gnu.warning)
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index 2853e87797a78..b77e29c9599de 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -14,6 +14,7 @@
#include <linux/memory.h>
#include <linux/sort.h>
#include <linux/execmem.h>
+#include <linux/rcupdate_trace.h>
#include <asm/extable.h>
#include <asm/ftrace.h>
#include <asm/set_memory.h>
@@ -722,6 +723,91 @@ static void emit_indirect_jump(u8 **pprog, int bpf_reg, u8 *ip)
*pprog = prog;
}
+/*
+ * Open-coded rcu_read_lock_trace() / rcu_read_unlock_trace() for the
+ * trampoline, see CONFIG_HAVE_RCU_TRAMPOLINE_READERS and the equivalent
+ * macros in arch/x86/kernel/ftrace_64.S. The image is not relocated, so
+ * current_task and rcu_tasks_trace_srcu_struct are referenced by absolute
+ * (sign-extended 32-bit) address, the form the JIT already relies on for
+ * this_cpu_off. Uses r10 and r11, which are scratch at every emission
+ * point, and clobbers flags.
+ *
+ * lock: unlock:
+ * mov r11, gs:[current_task] mov r11, gs:[current_task]
+ * mov r10d, [r11+nesting_off] mov r10d, [r11+nesting_off]
+ * inc dword ptr [r11+nesting_off] sub r10d, 1
+ * test r10d, r10d jnz 2f
+ * jnz 1f mov r10, [r11+scp_off]
+ * mov r10, [&srcu.srcu_ctrp] mov dword ptr [r11+nesting_off], 0
+ * inc qword ptr gs:[r10+locks_off] (smp_mb)
+ * mov [r11+scp_off], r10 inc qword ptr gs:[r10+unlocks_off]
+ * (smp_mb) jmp 3f
+ * 1: 2: mov [r11+nesting_off], r10d
+ * 3:
+ */
+static void emit_trace_rcu_reader(u8 **pprog, bool lock)
+{
+#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS
+ const u32 nesting_off = offsetof(struct task_struct, trc_reader_nesting);
+ const u32 scp_off = offsetof(struct task_struct, trc_reader_scp);
+ const u32 locks_off = offsetof(struct srcu_ctr, srcu_locks);
+ const u32 unlocks_off = offsetof(struct srcu_ctr, srcu_unlocks);
+ const bool mb = !IS_ENABLED(CONFIG_TASKS_TRACE_RCU_NO_MB);
+ u8 *prog = *pprog;
+
+ /* The plain this_cpu_inc() form of __srcu_read_lock_fast(). */
+ BUILD_BUG_ON(IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE));
+
+ /* mov r11, gs:[abs32 current_task] */
+ EMIT2(0x65, 0x4C);
+ EMIT3(0x8B, 0x1C, 0x25);
+ EMIT((u32)(unsigned long)¤t_task, 4);
+ /* mov r10d, dword ptr [r11 + nesting_off] */
+ EMIT3_off32(0x45, 0x8B, 0x93, nesting_off);
+
+ if (lock) {
+ /* inc dword ptr [r11 + nesting_off] */
+ EMIT3_off32(0x41, 0xFF, 0x83, nesting_off);
+ /* test r10d, r10d */
+ EMIT3(0x45, 0x85, 0xD2);
+ /* jnz 1f */
+ EMIT2(X86_JNE, 8 + 8 + 7 + (mb ? 6 : 0));
+ /* mov r10, qword ptr [abs32 &rcu_tasks_trace_srcu_struct.srcu_ctrp] */
+ EMIT4(0x4C, 0x8B, 0x14, 0x25);
+ EMIT((u32)(unsigned long)&rcu_tasks_trace_srcu_struct.srcu_ctrp, 4);
+ /* inc qword ptr gs:[r10 + locks_off] */
+ EMIT4_off32(0x65, 0x49, 0xFF, 0x82, locks_off);
+ /* mov qword ptr [r11 + scp_off], r10 */
+ EMIT3_off32(0x4D, 0x89, 0x93, scp_off);
+ /* smp_mb(): lock add dword ptr [rsp - 4], 0 */
+ if (mb)
+ EMIT2_off32(0xF0, 0x83, 0x00FC2444);
+ /* 1: */
+ } else {
+ /* sub r10d, 1 */
+ EMIT4(0x41, 0x83, 0xEA, 0x01);
+ /* jnz 2f */
+ EMIT2(X86_JNE, 7 + 11 + (mb ? 6 : 0) + 8 + 2);
+ /* mov r10, qword ptr [r11 + scp_off] */
+ EMIT3_off32(0x4D, 0x8B, 0x93, scp_off);
+ /* mov dword ptr [r11 + nesting_off], 0 */
+ EMIT3_off32(0x41, 0xC7, 0x83, nesting_off);
+ EMIT(0, 4);
+ if (mb)
+ EMIT2_off32(0xF0, 0x83, 0x00FC2444);
+ /* inc qword ptr gs:[r10 + unlocks_off] */
+ EMIT4_off32(0x65, 0x49, 0xFF, 0x82, unlocks_off);
+ /* jmp 3f */
+ EMIT2(0xEB, 7);
+ /* 2: mov dword ptr [r11 + nesting_off], r10d */
+ EMIT3_off32(0x45, 0x89, 0x93, nesting_off);
+ /* 3: */
+ }
+
+ *pprog = prog;
+#endif
+}
+
static void emit_return(u8 **pprog, u8 *ip)
{
u8 *prog = *pprog;
@@ -3610,6 +3696,16 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
/* mov QWORD PTR [rbp - rbx_off], rbx */
emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_6, -rbx_off);
+ /*
+ * Tasks RCU keeps this image alive only while we are a Tasks Trace
+ * reader; the instructions before this point (and after the final
+ * unlock) are covered by the irq-exit IP check. One reader spans
+ * __bpf_tramp_enter() and the fentry/fmod_ret progs, a second one
+ * the fexit progs and __bpf_tramp_exit(); the original function runs
+ * outside both, with the image pinned by im->pcref instead.
+ */
+ emit_trace_rcu_reader(&prog, true);
+
func_meta = nr_regs;
/* Store number of argument registers of the traced function */
emit_store_stack_imm64(&prog, BPF_REG_0, -func_meta_off, func_meta);
@@ -3660,6 +3756,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
}
if (flags & BPF_TRAMP_F_CALL_ORIG) {
+ emit_trace_rcu_reader(&prog, false);
restore_regs(m, &prog, regs_off);
save_args(m, &prog, arg_stack_off, true, flags, 0);
@@ -3682,6 +3779,13 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
}
/* remember return value in a stack for bpf prog to access */
emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8);
+ /*
+ * Second reader. Taken before ip_after_call so that the
+ * ip_after_call -> ip_epilogue jump patched in at teardown is
+ * inside it too; the fmod_ret early exit jumps past this still
+ * holding the first reader, so either way exactly one is held.
+ */
+ emit_trace_rcu_reader(&prog, true);
im->ip_after_call = image + (prog - (u8 *)rw_image);
emit_nops(&prog, X86_PATCH_SIZE);
}
@@ -3737,6 +3841,9 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
LOAD_TRAMP_TAIL_CALL_CNT_PTR(stack_size);
}
+ /* Remaining instructions are covered by the irq-exit IP check. */
+ emit_trace_rcu_reader(&prog, false);
+
/* restore return value of orig_call or fentry prog back into RAX */
if (save_ret)
emit_ldx(&prog, BPF_DW, BPF_REG_0, BPF_REG_FP, -8);
diff --git a/arch/x86/xen/enlighten_pv.c b/arch/x86/xen/enlighten_pv.c
index 2c64b388f6164..3d85035f5624f 100644
--- a/arch/x86/xen/enlighten_pv.c
+++ b/arch/x86/xen/enlighten_pv.c
@@ -739,7 +739,7 @@ __visible noinstr void xen_pv_evtchn_do_upcall(struct pt_regs *regs)
inhcall = get_and_clear_inhcall();
if (inhcall && !WARN_ON_ONCE(state.exit_rcu)) {
- irqentry_exit_cond_resched();
+ irqentry_exit_cond_resched(regs);
instrumentation_end();
restore_inhcall(inhcall);
} else {
diff --git a/include/linux/irq-entry-common.h b/include/linux/irq-entry-common.h
index 0bb6c03481fad..d42568a488a7f 100644
--- a/include/linux/irq-entry-common.h
+++ b/include/linux/irq-entry-common.h
@@ -5,6 +5,7 @@
#include <linux/context_tracking.h>
#include <linux/hrtimer_rearm.h>
#include <linux/kmsan.h>
+#include <linux/rcupdate_trace.h>
#include <linux/rseq_entry.h>
#include <linux/static_call_types.h>
#include <linux/syscalls.h>
@@ -214,6 +215,7 @@ static __always_inline void __exit_to_user_mode_validate(void)
{
/* Ensure that kernel state is sane for a return to userspace */
kmap_assert_nomap();
+ rcu_tasks_trace_assert_idle();
lockdep_assert_irqs_disabled();
lockdep_sys_exit();
}
@@ -343,24 +345,25 @@ typedef struct irqentry_state {
/**
* irqentry_exit_cond_resched - Conditionally reschedule on return from interrupt
+ * @regs: Pointer to pt_regs of interrupted context
*
* Conditional reschedule with additional sanity checks.
*/
-void raw_irqentry_exit_cond_resched(void);
+void raw_irqentry_exit_cond_resched(struct pt_regs *regs);
#ifdef CONFIG_PREEMPT_DYNAMIC
#if defined(CONFIG_HAVE_PREEMPT_DYNAMIC_CALL)
#define irqentry_exit_cond_resched_dynamic_enabled raw_irqentry_exit_cond_resched
#define irqentry_exit_cond_resched_dynamic_disabled NULL
DECLARE_STATIC_CALL(irqentry_exit_cond_resched, raw_irqentry_exit_cond_resched);
-#define irqentry_exit_cond_resched() static_call(irqentry_exit_cond_resched)()
+#define irqentry_exit_cond_resched(regs) static_call(irqentry_exit_cond_resched)(regs)
#elif defined(CONFIG_HAVE_PREEMPT_DYNAMIC_KEY)
DECLARE_STATIC_KEY_TRUE(sk_dynamic_irqentry_exit_cond_resched);
-void dynamic_irqentry_exit_cond_resched(void);
-#define irqentry_exit_cond_resched() dynamic_irqentry_exit_cond_resched()
+void dynamic_irqentry_exit_cond_resched(struct pt_regs *regs);
+#define irqentry_exit_cond_resched(regs) dynamic_irqentry_exit_cond_resched(regs)
#endif
#else /* CONFIG_PREEMPT_DYNAMIC */
-#define irqentry_exit_cond_resched() raw_irqentry_exit_cond_resched()
+#define irqentry_exit_cond_resched(regs) raw_irqentry_exit_cond_resched(regs)
#endif /* CONFIG_PREEMPT_DYNAMIC */
/**
@@ -465,7 +468,7 @@ static inline void irqentry_exit_to_kernel_mode_preempt(struct pt_regs *regs,
return;
if (IS_ENABLED(CONFIG_PREEMPTION))
- irqentry_exit_cond_resched();
+ irqentry_exit_cond_resched(regs);
}
/**
diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h
index e6de7ae55bda0..74cc48c044173 100644
--- a/include/linux/kprobes.h
+++ b/include/linux/kprobes.h
@@ -530,11 +530,17 @@ static inline bool is_kprobe_insn_slot(unsigned long addr)
}
#endif /* !CONFIG_KPROBES */
-#ifndef CONFIG_OPTPROBES
+#ifdef CONFIG_OPTPROBES
+bool kprobe_in_optimized_region(unsigned long addr);
+#else /* !CONFIG_OPTPROBES */
static inline bool is_kprobe_optinsn_slot(unsigned long addr)
{
return false;
}
+static inline bool kprobe_in_optimized_region(unsigned long addr)
+{
+ return false;
+}
#endif /* !CONFIG_OPTPROBES */
#ifdef CONFIG_KRETPROBES
diff --git a/include/linux/module.h b/include/linux/module.h
index 96cc98568eea5..82ca4f774725a 100644
--- a/include/linux/module.h
+++ b/include/linux/module.h
@@ -521,6 +521,13 @@ struct module {
unsigned int num_ftrace_callsites;
unsigned long *ftrace_callsites;
#endif
+#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS
+ /*
+ * An ftrace direct-call trampoline lives in this module's text; see
+ * rcu_tasks_trampoline_text(). Sticky once set.
+ */
+ bool ftrace_direct_tramp;
+#endif
#ifdef CONFIG_KPROBES
void *kprobes_text_start;
unsigned int kprobes_text_size;
diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index 44c07a66edfff..79e4b14f83b95 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -50,6 +50,23 @@ token_context_lock_instance(RCU, RCU_BH);
/* Exported common interfaces */
void call_rcu(struct rcu_head *head, rcu_callback_t func);
void rcu_barrier_tasks(void);
+
+/*
+ * Trampoline-reader Tasks RCU (CONFIG_TASKS_RCU_TRAMPOLINE_READERS), see
+ * kernel/rcu/tasks.h. rcu_tasks_irq_resched_enter()/_exit() bracket the
+ * irq-exit preemption; rcu_tasks_trampoline_text() and the arch_ override
+ * classify an interrupted IP; rcu_tasks_wait_irq_preempted() lets a caller
+ * wait out tasks already preempted somewhere it is about to make unsafe.
+ */
+void rcu_tasks_irq_resched_enter(unsigned long ip);
+void rcu_tasks_irq_resched_exit(void);
+bool rcu_tasks_trampoline_text(unsigned long ip);
+bool arch_rcu_tasks_trampoline_text(unsigned long ip);
+#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS
+void rcu_tasks_wait_irq_preempted(bool (*inside)(unsigned long ip));
+#else
+static inline void rcu_tasks_wait_irq_preempted(bool (*inside)(unsigned long ip)) { }
+#endif
void synchronize_rcu(void);
/*
@@ -180,11 +197,16 @@ static inline void rcu_nocb_flush_deferred_wakeup(void) { }
#ifdef CONFIG_TASKS_RCU_GENERIC
# ifdef CONFIG_TASKS_RCU
-# define rcu_tasks_classic_qs(t, preempt) \
+# ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS
+void rcu_tasks_note_qs(struct task_struct *t, bool preempt);
+# define rcu_tasks_classic_qs(t, preempt) rcu_tasks_note_qs((t), (preempt))
+# else
+# define rcu_tasks_classic_qs(t, preempt) \
do { \
if (!(preempt) && READ_ONCE((t)->rcu_tasks_holdout)) \
WRITE_ONCE((t)->rcu_tasks_holdout, false); \
} while (0)
+# endif
void call_rcu_tasks(struct rcu_head *head, rcu_callback_t func);
void synchronize_rcu_tasks(void);
void rcu_tasks_torture_stats_print(char *tt, char *tf);
diff --git a/include/linux/rcupdate_trace.h b/include/linux/rcupdate_trace.h
index 273c59a032512..9eb7f92c710be 100644
--- a/include/linux/rcupdate_trace.h
+++ b/include/linux/rcupdate_trace.h
@@ -92,6 +92,12 @@ static inline void rcu_read_unlock_tasks_trace(struct srcu_ctr __percpu *scp)
* the all the other tasks exit their critical sections.
*
* For more details, please see the documentation for rcu_read_lock().
+ *
+ * CONFIG_HAVE_RCU_TRAMPOLINE_READERS architectures open-code this pair in
+ * their ftrace and BPF trampolines (arch/x86/kernel/ftrace_64.S,
+ * arch/x86/kernel/kprobes/opt.c, arch/x86/net/bpf_jit_comp.c,
+ * arch/arm64/kernel/entry-ftrace.S, arch/arm64/net/bpf_jit_comp.c,
+ * samples/ftrace/ftrace-direct.h); changes here need to be mirrored there.
*/
static inline void rcu_read_lock_trace(void)
{
@@ -211,6 +217,12 @@ unsigned long rcu_tasks_trace_batches_completed(void);
// Placeholders to enable stepwise transition.
void __init rcu_tasks_trace_suppress_unused(void);
+/* A task must never reach userspace inside an rcu_read_lock_trace() reader. */
+static inline void rcu_tasks_trace_assert_idle(void)
+{
+ WARN_ON_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) && READ_ONCE(current->trc_reader_nesting));
+}
+
#else
static inline unsigned long rcu_tasks_trace_batches_completed(void) { return 0; }
/*
@@ -220,6 +232,7 @@ static inline unsigned long rcu_tasks_trace_batches_completed(void) { return 0;
static inline void call_rcu_tasks_trace(struct rcu_head *rhp, rcu_callback_t func) { BUG(); }
static inline void rcu_read_lock_trace(void) { BUG(); }
static inline void rcu_read_unlock_trace(void) { BUG(); }
+static inline void rcu_tasks_trace_assert_idle(void) { }
#endif /* #ifdef CONFIG_TASKS_TRACE_RCU */
DEFINE_LOCK_GUARD_0(rcu_tasks_trace,
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 8b3d47a325cca..15beb44caa2c8 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -957,6 +957,7 @@ struct task_struct {
u8 rcu_tasks_holdout;
u8 rcu_tasks_idx;
int rcu_tasks_idle_cpu;
+ unsigned long rcu_tasks_irq_ip;
struct list_head rcu_tasks_holdout_list;
int rcu_tasks_exit_cpu;
struct list_head rcu_tasks_exit_list;
diff --git a/kernel/entry/common.c b/kernel/entry/common.c
index e3d381fd3d251..94318519998c8 100644
--- a/kernel/entry/common.c
+++ b/kernel/entry/common.c
@@ -6,6 +6,7 @@
#include <linux/jump_label.h>
#include <linux/kmsan.h>
#include <linux/livepatch.h>
+#include <linux/rcupdate.h>
#include <linux/resume_user_mode.h>
#include <linux/tick.h>
@@ -134,15 +135,20 @@ static inline bool arch_irqentry_exit_need_resched(void);
static inline bool arch_irqentry_exit_need_resched(void) { return true; }
#endif
-void raw_irqentry_exit_cond_resched(void)
+void raw_irqentry_exit_cond_resched(struct pt_regs *regs)
{
if (!preempt_count()) {
/* Sanity check RCU and thread stack */
rcu_irq_exit_check_preempt();
if (IS_ENABLED(CONFIG_DEBUG_ENTRY))
WARN_ON_ONCE(!on_thread_stack());
- if (need_resched() && arch_irqentry_exit_need_resched())
+ if (need_resched() && arch_irqentry_exit_need_resched()) {
+ if (IS_ENABLED(CONFIG_TASKS_RCU_TRAMPOLINE_READERS))
+ rcu_tasks_irq_resched_enter(instruction_pointer(regs));
preempt_schedule_irq();
+ if (IS_ENABLED(CONFIG_TASKS_RCU_TRAMPOLINE_READERS))
+ rcu_tasks_irq_resched_exit();
+ }
}
}
#ifdef CONFIG_PREEMPT_DYNAMIC
@@ -150,11 +156,11 @@ void raw_irqentry_exit_cond_resched(void)
DEFINE_STATIC_CALL(irqentry_exit_cond_resched, raw_irqentry_exit_cond_resched);
#elif defined(CONFIG_HAVE_PREEMPT_DYNAMIC_KEY)
DEFINE_STATIC_KEY_TRUE(sk_dynamic_irqentry_exit_cond_resched);
-void dynamic_irqentry_exit_cond_resched(void)
+void dynamic_irqentry_exit_cond_resched(struct pt_regs *regs)
{
if (!static_branch_unlikely(&sk_dynamic_irqentry_exit_cond_resched))
return;
- raw_irqentry_exit_cond_resched();
+ raw_irqentry_exit_cond_resched(regs);
}
#endif
#endif
diff --git a/kernel/fork.c b/kernel/fork.c
index 416758c8a3d43..8077336bb136b 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -1871,6 +1871,7 @@ static inline void rcu_copy_process(struct task_struct *p)
p->rcu_tasks_holdout = false;
INIT_LIST_HEAD(&p->rcu_tasks_holdout_list);
p->rcu_tasks_idle_cpu = -1;
+ p->rcu_tasks_irq_ip = 0;
INIT_LIST_HEAD(&p->rcu_tasks_exit_list);
#endif /* #ifdef CONFIG_TASKS_RCU */
#ifdef CONFIG_TASKS_TRACE_RCU
diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index 6337da5cab9e7..e460fba83e4ab 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -511,6 +511,48 @@ static struct kprobe *get_optimized_kprobe(kprobe_opcode_t *addr)
return NULL;
}
+/*
+ * True while kprobe_optimizer() is waiting for its Tasks RCU grace period.
+ * Only in that window can an interruption inside an optprobe's jump region
+ * matter to it, so kprobe_in_optimized_region() does no work otherwise.
+ */
+static bool kprobe_optimizer_waiting;
+
+/**
+ * kprobe_in_optimized_region - Could @addr be inside bytes a jump-optimized
+ * kprobe replaces?
+ * @addr: kernel text address, typically an interrupted instruction pointer
+ *
+ * kprobe_optimizer() relies on synchronize_rcu_tasks() to wait for tasks that
+ * were interrupted on an instruction boundary inside the region about to be
+ * overwritten by the optimized jump. Where Tasks RCU is built on
+ * reader-marked trampolines that region has no reader, so the irq-exit
+ * quiescent-state check asks this instead (see rcu_tasks_trampoline_text()).
+ * This is the lockless, conservative form of get_optimized_kprobe(): it does
+ * not care whether the kprobe found is, or ever will be, optimized. May be
+ * called from any context with preemption disabled; the kprobe hash is
+ * RCU-protected and every free path waits for a grace period after unhashing.
+ *
+ * The hash walk only runs while the optimizer is actually waiting. A task
+ * that was preempted in such a region before the flag went up is invisible
+ * to that check, so the optimizer first waits those out by their recorded
+ * preemption IP (rcu_tasks_wait_irq_preempted(), whose leading
+ * synchronize_rcu() also publishes the flag to every check in flight).
+ */
+bool kprobe_in_optimized_region(unsigned long addr)
+{
+ int i;
+
+ if (!READ_ONCE(kprobe_optimizer_waiting))
+ return false;
+
+ for (i = 1; i < MAX_OPTIMIZED_LENGTH / sizeof(kprobe_opcode_t); i++)
+ if (get_kprobe((kprobe_opcode_t *)addr - i))
+ return true;
+ return false;
+}
+NOKPROBE_SYMBOL(kprobe_in_optimized_region);
+
/* Optimization staging list, protected by 'kprobe_mutex' */
static LIST_HEAD(optimizing_list);
static LIST_HEAD(unoptimizing_list);
@@ -644,8 +686,16 @@ static void kprobe_optimizer(void)
* to 2nd-Nth byte of jump instruction. This wait is for avoiding it.
* Note that on non-preemptive kernel, this is transparently converted
* to synchronoze_sched() to wait for all interrupts to have completed.
+ * kprobe_optimizer_waiting lets a reader-marked-trampoline Tasks RCU
+ * recognise tasks interrupted in such a region while we wait, and
+ * rcu_tasks_wait_irq_preempted() (a no-op elsewhere) first waits
+ * out any that were preempted there before we said so; see
+ * kprobe_in_optimized_region().
*/
+ WRITE_ONCE(kprobe_optimizer_waiting, true);
+ rcu_tasks_wait_irq_preempted(kprobe_in_optimized_region);
synchronize_rcu_tasks();
+ WRITE_ONCE(kprobe_optimizer_waiting, false);
/* Step 3: Optimize kprobes after quiesence period */
do_optimize_kprobes();
diff --git a/kernel/rcu/Kconfig b/kernel/rcu/Kconfig
index 332df7a7a6347..341b68b972ef1 100644
--- a/kernel/rcu/Kconfig
+++ b/kernel/rcu/Kconfig
@@ -95,8 +95,10 @@ config FORCE_TASKS_RCU
help
This option force-enables a task-based RCU implementation
that uses only voluntary context switch (not preemption!),
- idle, and user-mode execution as quiescent states. Not for
- manual selection in most cases.
+ idle, and user-mode execution as quiescent states, or, on
+ HAVE_RCU_TRAMPOLINE_READERS architectures, the variant built on
+ Tasks Trace RCU readers in trampolines. Not for manual
+ selection in most cases.
config NEED_TASKS_RCU
bool
@@ -107,6 +109,28 @@ config TASKS_RCU
default NEED_TASKS_RCU && PREEMPTION
select IRQ_WORK
+config HAVE_RCU_TRAMPOLINE_READERS
+ bool
+ help
+ Select this if the architecture uses the generic irq entry code and
+ every trampoline whose lifetime Tasks RCU guards on it (ftrace
+ trampolines, kprobe out-of-line and optimized-probe slots, BPF
+ trampolines, out-of-line ftrace direct-call trampolines) enters a
+ Tasks Trace RCU read-side critical section before calling out of
+ the trampoline and leaves it before returning, and any core text
+ that runs on behalf of such a trampoline outside that reader is
+ reported by arch_rcu_tasks_trampoline_text(). The assembly readers
+ use the this_cpu_inc() form of SRCU-fast, hence !NEED_SRCU_NMI_SAFE.
+
+config TASKS_RCU_TRAMPOLINE_READERS
+ def_bool TASKS_RCU && HAVE_RCU_TRAMPOLINE_READERS && GENERIC_IRQ_ENTRY && !NEED_SRCU_NMI_SAFE
+ select TASKS_TRACE_RCU
+ help
+ Implement the Tasks RCU grace period as a per-CPU pass over
+ context switches and irq-exit reschedules outside trampoline text
+ plus a Tasks Trace RCU grace period, instead of waiting for every
+ task to voluntarily context switch. See kernel/rcu/tasks.h.
+
config FORCE_TASKS_RUDE_RCU
bool "Force selection of Tasks Rude RCU"
depends on RCU_EXPERT
diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c
index 794937e13e7c3..ab870ef09af0a 100644
--- a/kernel/rcu/rcutorture.c
+++ b/kernel/rcu/rcutorture.c
@@ -1142,13 +1142,23 @@ static struct rcu_torture_ops trivial_preempt_ops = {
* Definitions for RCU-tasks torture testing.
*/
+/*
+ * A classic Tasks RCU reader is any stretch of kernel code that does not
+ * voluntarily block. With CONFIG_TASKS_RCU_TRAMPOLINE_READERS a preemption
+ * outside trampoline text also ends it, and what a trampoline does to stay
+ * protected across its call-out is take a Tasks Trace reader, so model that.
+ */
static int tasks_torture_read_lock(void)
{
+ if (IS_ENABLED(CONFIG_TASKS_RCU_TRAMPOLINE_READERS))
+ rcu_read_lock_trace();
return 0;
}
static void tasks_torture_read_unlock(int idx)
{
+ if (IS_ENABLED(CONFIG_TASKS_RCU_TRAMPOLINE_READERS))
+ rcu_read_unlock_trace();
}
static void rcu_tasks_torture_deferred_free(struct rcu_torture *p)
diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h
index 627295396cd91..42ea6e0e61cbf 100644
--- a/kernel/rcu/tasks.h
+++ b/kernel/rcu/tasks.h
@@ -152,7 +152,7 @@ static struct rcu_tasks rt_name = \
.kname = #rt_name, \
}
-#ifdef CONFIG_TASKS_RCU
+#if defined(CONFIG_TASKS_RCU) && !defined(CONFIG_TASKS_RCU_TRAMPOLINE_READERS)
/* Report delay of scan exiting tasklist in rcu_tasks_postscan(). */
static void tasks_rcu_exit_stall(struct timer_list *unused);
@@ -802,7 +802,7 @@ static void rcu_tasks_torture_stats_print_generic(struct rcu_tasks *rtp, char *t
#endif // #ifndef CONFIG_TINY_RCU
-#if defined(CONFIG_TASKS_RCU)
+#if defined(CONFIG_TASKS_RCU) && !defined(CONFIG_TASKS_RCU_TRAMPOLINE_READERS)
////////////////////////////////////////////////////////////////////////
//
@@ -897,10 +897,461 @@ static void rcu_tasks_wait_gp(struct rcu_tasks *rtp)
rtp->postgp_func(rtp);
}
-#endif /* #if defined(CONFIG_TASKS_RCU) */
+#endif /* #if defined(CONFIG_TASKS_RCU) && !defined(CONFIG_TASKS_RCU_TRAMPOLINE_READERS) */
#ifdef CONFIG_TASKS_RCU
+static int rcu_tasks_lazy_ms = -1;
+module_param(rcu_tasks_lazy_ms, int, 0444);
+
+#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS
+
+////////////////////////////////////////////////////////////////////////
+//
+// Tasks RCU for architectures whose trampolines are Tasks Trace RCU
+// readers (CONFIG_HAVE_RCU_TRAMPOLINE_READERS).
+//
+// On these architectures every piece of text whose lifetime Tasks RCU
+// guards -- ftrace trampolines, kprobe optinsn slots, BPF trampoline
+// images, out-of-line ftrace direct-call trampolines -- enters a Tasks
+// Trace RCU read-side critical section before calling out of itself and
+// leaves it before returning, so a task anywhere inside such a call-out,
+// preempted or not, is an ordinary rcu_read_lock_trace() reader and
+// synchronize_rcu_tasks_trace() waits for it.
+//
+// What that cannot cover is the handful of instructions in the trampoline
+// before the reader is entered and after it is left, and the one user that
+// has no trampoline at all: the bytes after a kprobe that the jump
+// optimizer is about to overwrite. A task can only linger in such
+// "unmarked" text by being interrupted there; unmarked text never calls
+// anything that could schedule. So a context switch on a CPU tells us that
+// whatever that CPU was running is out of unmarked text, with one
+// exception: a preemption from the irq-exit path, which can happen at any
+// instruction boundary. That path has the interrupted pt_regs in hand, so
+// just before it preempts it records the IP in the task and checks it
+// (rcu_tasks_trampoline_text()); if it is inside unmarked text the task
+// goes on a short holdout list first, and takes itself off again at its
+// next context switch outside such a preemption or its next irq-exit
+// check that finds it elsewhere. With that, every pass through
+// __schedule() is a per-CPU quiescent event, as are usermode and idle.
+//
+// A grace period is then:
+//
+// 1. Wait for every online CPU to context switch or be found in an RCU
+// extended quiescent state (deep idle, nohz_full userspace), nudging
+// stragglers with resched_cpu(). Afterwards no task is in the leading
+// unmarked instructions of a dying trampoline unless it is on the
+// holdout list.
+// 2. Wait for the holdout list (as it stood) to drain.
+// 3. synchronize_rcu_tasks_trace(), for everything inside the readers.
+// 4. Repeat 1 and 2 for tasks that have since left the reader and are in
+// the trailing unmarked instructions.
+//
+// which is bounded by a few jiffies plus preempt-off latency plus an SRCU
+// grace period, independent of how long any task runs without sleeping.
+// Unlike the classic implementation this does wait for an idle task caught
+// in a trampoline, since an idle CPU only counts while RCU is not watching
+// it.
+
+static void rcu_tasks_tramp_wait_gp(struct rcu_tasks *rtp);
+void call_rcu_tasks(struct rcu_head *rhp, rcu_callback_t func);
+DEFINE_RCU_TASKS(rcu_tasks, rcu_tasks_tramp_wait_gp, call_rcu_tasks, "RCU Tasks");
+
+/* Per-CPU count of Tasks RCU quiescent events, and the GP kthread's snapshot. */
+static DEFINE_PER_CPU(unsigned long, rcu_tasks_qs_seq);
+static DEFINE_PER_CPU(unsigned long, rcu_tasks_qs_snap);
+
+/*
+ * Tasks currently switched out by an irq-exit preemption are kept, with the
+ * interrupted IP, on the per-CPU rtp_exit_list of the CPU that preempted them
+ * (reusing the list, lock and task_struct fields the classic flavor uses for
+ * its exit-path bookkeeping, which this flavor does not need), so that
+ * rcu_tasks_wait_irq_preempted() can find them without a tasklist scan and
+ * regardless of where they are in exit.
+ */
+
+/* Tasks last seen preempted inside unmarked trampoline text. */
+static LIST_HEAD(rcu_tasks_tramp_holdouts);
+static DEFINE_RAW_SPINLOCK(rcu_tasks_tramp_lock);
+
+/* CPUs / holdouts the current grace period is still waiting for. */
+static struct cpumask rcu_tasks_pending_cpus;
+static LIST_HEAD(rcu_tasks_gp_holdouts);
+
+/**
+ * arch_rcu_tasks_trampoline_text - Does the architecture treat @ip as unmarked trampoline text?
+ * @ip: kernel text address inside core kernel text
+ *
+ * See rcu_tasks_trampoline_text(). Architectures override this to flag
+ * core text that runs on behalf of a trampoline outside its Tasks Trace
+ * reader, e.g. static ftrace entry stubs or return thunks that hold a
+ * trampoline address they are about to jump to.
+ */
+bool __weak arch_rcu_tasks_trampoline_text(unsigned long ip)
+{
+ return false;
+}
+
+/**
+ * rcu_tasks_trampoline_text - Is @ip in text Tasks RCU protects but no reader marks?
+ * @ip: an interrupted instruction pointer
+ *
+ * True when a task interrupted at @ip may be executing, or about to enter
+ * or return into, text whose lifetime depends on synchronize_rcu_tasks()
+ * without being inside the Tasks Trace reader that text takes around its
+ * call-outs:
+ *
+ * - anything outside core kernel and module text (ftrace and BPF
+ * trampolines, kprobe slots and other dynamically allocated text; this
+ * deliberately does not ask is_ftrace_trampoline() and friends, since
+ * text being torn down may already be unregistered there);
+ * - whatever the architecture adds via arch_rcu_tasks_trampoline_text();
+ * - the text of a module that hosts an out-of-line ftrace direct-call
+ * trampoline (see ftrace_direct_mark_module());
+ * - the bytes after a kprobe that a pending jump optimization is about to
+ * overwrite, the one synchronize_rcu_tasks() user with no trampoline.
+ *
+ * A false positive only makes the task a holdout until its next quiescent
+ * event. Called with interrupts disabled from the irq-exit path.
+ */
+bool rcu_tasks_trampoline_text(unsigned long ip)
+{
+ bool ret = true;
+
+ if (core_kernel_text(ip))
+ return arch_rcu_tasks_trampoline_text(ip) ||
+ kprobe_in_optimized_region(ip);
+
+#ifdef CONFIG_MODULES
+ scoped_guard(rcu) {
+ struct module *mod = __module_text_address(ip);
+
+ if (mod)
+ ret = READ_ONCE(mod->ftrace_direct_tramp) ||
+ kprobe_in_optimized_region(ip);
+ }
+#endif
+ return ret;
+}
+NOKPROBE_SYMBOL(rcu_tasks_trampoline_text);
+
+/* Note a Tasks RCU quiescent event on this CPU. */
+static void rcu_tasks_qs_event(void)
+{
+ unsigned long *seq;
+
+ guard(preempt_notrace)();
+ seq = this_cpu_ptr(&rcu_tasks_qs_seq);
+ /* Order a preceding rcu_tasks_tramp_hold() before the count. */
+ smp_store_release(seq, *seq + 1);
+}
+
+static void rcu_tasks_tramp_hold(struct task_struct *t)
+{
+ unsigned long flags;
+
+ if (t->rcu_tasks_holdout)
+ return;
+ raw_spin_lock_irqsave(&rcu_tasks_tramp_lock, flags);
+ list_add_tail(&t->rcu_tasks_holdout_list, &rcu_tasks_tramp_holdouts);
+ WRITE_ONCE(t->rcu_tasks_holdout, true);
+ raw_spin_unlock_irqrestore(&rcu_tasks_tramp_lock, flags);
+}
+
+static void rcu_tasks_tramp_release(struct task_struct *t)
+{
+ unsigned long flags;
+
+ if (likely(!t->rcu_tasks_holdout))
+ return;
+ raw_spin_lock_irqsave(&rcu_tasks_tramp_lock, flags);
+ list_del_init(&t->rcu_tasks_holdout_list);
+ WRITE_ONCE(t->rcu_tasks_holdout, false);
+ raw_spin_unlock_irqrestore(&rcu_tasks_tramp_lock, flags);
+}
+
+/**
+ * rcu_tasks_irq_resched_enter - Tasks RCU hook for the irq-exit reschedule check
+ * @ip: instruction pointer of the interrupted (task-level) context
+ *
+ * Called with interrupts disabled when an interrupt returning to kernel
+ * mode is about to preempt_schedule_irq(), the one context switch that can
+ * catch a task inside unmarked trampoline text. Record where the task is
+ * parked for as long as it is (rcu_tasks_wait_irq_preempted() looks at
+ * that), and if it is inside such text make it a holdout before
+ * __schedule() reports the quiescent event; if it is not, this is as good
+ * as a voluntary switch for ending an earlier hold.
+ */
+void rcu_tasks_irq_resched_enter(unsigned long ip)
+{
+ struct task_struct *t = current;
+ struct rcu_tasks_percpu *rtpcp = this_cpu_ptr(rcu_tasks.rtpcpu);
+
+ lockdep_assert_irqs_disabled();
+ WRITE_ONCE(t->rcu_tasks_irq_ip, ip);
+ t->rcu_tasks_exit_cpu = smp_processor_id();
+ raw_spin_lock_rcu_node(rtpcp);
+ list_add(&t->rcu_tasks_exit_list, &rtpcp->rtp_exit_list);
+ raw_spin_unlock_rcu_node(rtpcp);
+
+ if (unlikely(rcu_tasks_trampoline_text(ip)))
+ rcu_tasks_tramp_hold(t);
+ else
+ rcu_tasks_tramp_release(t);
+}
+NOKPROBE_SYMBOL(rcu_tasks_irq_resched_enter);
+
+/**
+ * rcu_tasks_irq_resched_exit - preempt_schedule_irq() has returned
+ *
+ * The task is running again (possibly elsewhere) and about to return to the
+ * interrupted context; it is no longer parked anywhere.
+ */
+void rcu_tasks_irq_resched_exit(void)
+{
+ struct task_struct *t = current;
+ struct rcu_tasks_percpu *rtpcp = per_cpu_ptr(rcu_tasks.rtpcpu, t->rcu_tasks_exit_cpu);
+
+ lockdep_assert_irqs_disabled();
+ raw_spin_lock_rcu_node(rtpcp);
+ list_del_init(&t->rcu_tasks_exit_list);
+ raw_spin_unlock_rcu_node(rtpcp);
+ WRITE_ONCE(t->rcu_tasks_irq_ip, 0);
+}
+NOKPROBE_SYMBOL(rcu_tasks_irq_resched_exit);
+
+/**
+ * rcu_tasks_note_qs - Tasks RCU hook for a context switch or explicit QS
+ * @t: current
+ * @preempt: this is a preemption rather than a voluntary switch
+ *
+ * Every pass through __schedule() (and cond_resched_tasks_rcu_qs(), and a
+ * tick from userspace or idle) is a quiescent event for this CPU: unmarked
+ * trampoline text never calls anything that schedules, and the irq-exit
+ * path has already made @t a holdout if it is preempting inside such text.
+ * Any of these outside an irq-exit preemption also shows @t itself to be
+ * outside, ending an earlier hold -- including cond_resched() under
+ * PREEMPT_DYNAMIC's none/voluntary modes, where the irq-exit path is off.
+ */
+void rcu_tasks_note_qs(struct task_struct *t, bool preempt)
+{
+ WARN_ON_ONCE(t != current);
+ if (!READ_ONCE(t->rcu_tasks_irq_ip))
+ rcu_tasks_tramp_release(t);
+ rcu_tasks_qs_event();
+}
+EXPORT_SYMBOL_GPL(rcu_tasks_note_qs); /* cond_resched_tasks_rcu_qs() */
+
+/**
+ * rcu_tasks_wait_irq_preempted - wait for tasks irq-preempted inside @inside
+ * @inside: predicate on a task's recorded irq-exit preemption IP
+ *
+ * For a caller about to make some ordinary text unsafe to be parked in
+ * (the kprobe jump optimizer): once the caller has arranged for
+ * rcu_tasks_trampoline_text() to cover that text, new irq-exit preemptions
+ * there become holdouts, but a task preempted there earlier is invisible
+ * to the grace period. Wait until no parked task's recorded preemption IP
+ * is inside; a following synchronize_rcu_tasks() then covers the rest.
+ * The leading synchronize_rcu() orders the caller's arrangement against
+ * preemptions in flight, which run with interrupts disabled.
+ */
+void rcu_tasks_wait_irq_preempted(bool (*inside)(unsigned long ip))
+{
+ struct task_struct *t;
+ unsigned long flags;
+ int cpu, kick;
+ bool found;
+
+ synchronize_rcu();
+ for (;;) {
+ found = false;
+ for_each_possible_cpu(cpu) {
+ struct rcu_tasks_percpu *rtpcp = per_cpu_ptr(rcu_tasks.rtpcpu, cpu);
+
+ kick = -1;
+ raw_spin_lock_irqsave_rcu_node(rtpcp, flags);
+ list_for_each_entry(t, &rtpcp->rtp_exit_list, rcu_tasks_exit_list) {
+ if (inside(READ_ONCE(t->rcu_tasks_irq_ip))) {
+ found = true;
+ if (task_curr(t))
+ kick = task_cpu(t);
+ }
+ }
+ raw_spin_unlock_irqrestore_rcu_node(rtpcp, flags);
+ if (kick >= 0)
+ resched_cpu(kick);
+ }
+ if (!found)
+ return;
+ schedule_timeout_uninterruptible(1);
+ }
+}
+
+/* Has @cpu passed a quiescent event since the snapshot, or need it not? */
+static bool rcu_tasks_cpu_quiescent(int cpu)
+{
+ if (!cpu_online(cpu))
+ return true;
+ /* Pairs with the release in rcu_tasks_qs_event(). */
+ if (smp_load_acquire(per_cpu_ptr(&rcu_tasks_qs_seq, cpu)) !=
+ per_cpu(rcu_tasks_qs_snap, cpu))
+ return true;
+ /*
+ * Idle or nohz_full userspace in an RCU extended quiescent state: no
+ * task-level kernel frames can be live in a trampoline there, and
+ * whatever ran before has switched out. An idle CPU that RCU is
+ * watching (an interrupt from idle, or the traceable part of the idle
+ * loop) is deliberately not let through: the idle task may be in a
+ * trampoline with that interrupt on top, and since it is never
+ * preempted from irq exit nothing else would catch it. It gets the
+ * resched_cpu() like anyone else and counts once the idle loop itself
+ * schedules, which it cannot do from inside a trampoline.
+ */
+ return !(ct_rcu_watching_cpu(cpu) & CT_RCU_WATCHING);
+}
+
+/* Rate-limited stall report; returns true if the caller should add detail. */
+static bool rcu_tasks_tramp_stall(struct rcu_tasks *rtp, unsigned long *lastreport,
+ const char *what)
+{
+ int rtst = READ_ONCE(rcu_task_stall_timeout);
+
+ if (rtst <= 0 || !time_after(jiffies, *lastreport + rtst))
+ return false;
+ *lastreport = jiffies;
+ pr_err("INFO: %s: %s, grace period %lu is %lu jiffies old\n", rtp->kname,
+ what, rcu_seq_current(&rtp->tasks_gp_seq), jiffies - rtp->gp_start);
+ return true;
+}
+
+/*
+ * Steps 1/4: wait until every online CPU has context switched or is in an
+ * RCU extended quiescent state. A CPU that has neither after a jiffy is
+ * asked to switch with resched_cpu(), which takes it through
+ * rcu_tasks_irq_resched_enter() and __schedule() (or, from userspace, a
+ * guest or the idle loop, straight to __schedule()).
+ */
+static void rcu_tasks_tramp_wait_cpus(struct rcu_tasks *rtp, unsigned long *lastreport)
+{
+ struct cpumask *pending = &rcu_tasks_pending_cpus;
+ unsigned long start;
+ int cpu;
+
+ /*
+ * The quiescent events run with preemption (in practice interrupts)
+ * disabled, so after this any event we go on to count began after the
+ * caller's updates -- the unpublished trampoline, and whatever
+ * rcu_tasks_trampoline_text() consults -- were visible to it.
+ */
+ synchronize_rcu();
+
+ start = jiffies;
+ for_each_online_cpu(cpu) {
+ per_cpu(rcu_tasks_qs_snap, cpu) = READ_ONCE(per_cpu(rcu_tasks_qs_seq, cpu));
+ __cpumask_set_cpu(cpu, pending);
+ }
+ /* Snapshots before the checks below; pairs with rcu_tasks_qs_event(). */
+ smp_mb();
+
+ for (;;) {
+ for_each_cpu(cpu, pending)
+ if (rcu_tasks_cpu_quiescent(cpu))
+ __cpumask_clear_cpu(cpu, pending);
+ if (cpumask_empty(pending))
+ break;
+ if (time_after(jiffies, start)) {
+ for_each_cpu(cpu, pending)
+ resched_cpu(cpu);
+ rtp->n_ipis += cpumask_weight(pending);
+ }
+ schedule_timeout_idle(1);
+ if (rcu_tasks_tramp_stall(rtp, lastreport, "CPUs without a quiescent event"))
+ pr_err("\tCPUs: %*pbl\n", cpumask_pr_args(pending));
+ }
+}
+
+/*
+ * Steps 2/4: wait for the tasks that were holdouts when we looked to stop
+ * being holdouts. They are moved to a private list so that tasks becoming
+ * holdouts later (in live trampolines) cannot keep us here; each removes
+ * itself via rcu_tasks_tramp_release() wherever it is queued.
+ */
+static void rcu_tasks_tramp_wait_holdouts(struct rcu_tasks *rtp, unsigned long *lastreport)
+{
+ struct task_struct *t;
+ unsigned long flags;
+ int cpu;
+
+ raw_spin_lock_irqsave(&rcu_tasks_tramp_lock, flags);
+ list_splice_tail_init(&rcu_tasks_tramp_holdouts, &rcu_tasks_gp_holdouts);
+ raw_spin_unlock_irqrestore(&rcu_tasks_tramp_lock, flags);
+
+ for (;;) {
+ struct cpumask *kick = &rcu_tasks_pending_cpus;
+ struct task_struct *show[8];
+ int nshow = 0, i;
+ bool empty, report;
+
+ report = rcu_tasks_tramp_stall(rtp, lastreport,
+ "tasks preempted in trampoline text");
+ cpumask_clear(kick);
+ raw_spin_lock_irqsave(&rcu_tasks_tramp_lock, flags);
+ empty = list_empty(&rcu_tasks_gp_holdouts);
+ list_for_each_entry(t, &rcu_tasks_gp_holdouts, rcu_tasks_holdout_list) {
+ if (task_curr(t))
+ __cpumask_set_cpu(task_cpu(t), kick);
+ if (report && nshow < ARRAY_SIZE(show))
+ show[nshow++] = get_task_struct(t);
+ }
+ raw_spin_unlock_irqrestore(&rcu_tasks_tramp_lock, flags);
+ /* Never printk under the lock the irq-exit path takes. */
+ for (i = 0; i < nshow; i++) {
+ sched_show_task(show[i]);
+ put_task_struct(show[i]);
+ }
+ if (empty)
+ break;
+ for_each_cpu(cpu, kick)
+ resched_cpu(cpu);
+ rtp->n_ipis += cpumask_weight(kick);
+ schedule_timeout_idle(1);
+ }
+}
+
+/* Wait for one trampoline-reader Tasks RCU grace period. */
+static void rcu_tasks_tramp_wait_gp(struct rcu_tasks *rtp)
+{
+ unsigned long lastreport = jiffies;
+
+ set_tasks_gp_state(rtp, RTGS_WAIT_SCAN_HOLDOUTS);
+ rcu_tasks_tramp_wait_cpus(rtp, &lastreport);
+ rcu_tasks_tramp_wait_holdouts(rtp, &lastreport);
+
+ set_tasks_gp_state(rtp, RTGS_WAIT_READERS);
+ synchronize_rcu_tasks_trace();
+
+ set_tasks_gp_state(rtp, RTGS_SCAN_HOLDOUTS);
+ rcu_tasks_tramp_wait_cpus(rtp, &lastreport);
+ rcu_tasks_tramp_wait_holdouts(rtp, &lastreport);
+
+ set_tasks_gp_state(rtp, RTGS_POST_GP);
+}
+
+static int __init rcu_spawn_tasks_kthread(void)
+{
+ rcu_tasks.gp_sleep = HZ / 10;
+ if (rcu_tasks_lazy_ms >= 0)
+ rcu_tasks.lazy_jiffies = msecs_to_jiffies(rcu_tasks_lazy_ms);
+ rcu_tasks.wait_state = TASK_IDLE;
+ rcu_spawn_tasks_kthread_generic(&rcu_tasks);
+ return 0;
+}
+
+void exit_tasks_rcu_start(void) { }
+void exit_tasks_rcu_finish(void) { }
+
+#else /* #ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS */
+
////////////////////////////////////////////////////////////////////////
//
// Simple variant of RCU whose quiescent states are voluntary context
@@ -1173,6 +1624,8 @@ static void tasks_rcu_exit_stall(struct timer_list *unused)
#endif // #ifndef CONFIG_TINY_RCU
}
+#endif /* #else #ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS */
+
/**
* call_rcu_tasks() - Queue an RCU for invocation task-based grace period
* @rhp: structure to be used for queueing the RCU updates.
@@ -1187,6 +1640,12 @@ static void tasks_rcu_exit_stall(struct timer_list *unused)
* primitives analogous to rcu_read_lock() and rcu_read_unlock() because
* this primitive is intended to determine that all tasks have passed
* through a safe state, not so much for data-structure synchronization.
+ * On CONFIG_TASKS_RCU_TRAMPOLINE_READERS kernels a preemption outside
+ * trampoline text also ends one, and a reader whose protected window
+ * spans preemptible code must additionally be a Tasks Trace RCU reader
+ * (rcu_read_lock_trace(), as the trampolines there take around their
+ * call-outs); an arbitrary stretch of preemptible kernel code is not
+ * protected.
*
* See the description of call_rcu() for more detailed information on
* memory ordering guarantees.
@@ -1205,7 +1664,9 @@ EXPORT_SYMBOL_GPL(call_rcu_tasks);
* executing rcu-tasks read-side critical sections have elapsed. These
* read-side critical sections are delimited by calls to schedule(),
* cond_resched_tasks_rcu_qs(), idle execution, userspace execution, calls
- * to synchronize_rcu_tasks(), and (in theory, anyway) cond_resched().
+ * to synchronize_rcu_tasks(), and (in theory, anyway) cond_resched();
+ * on CONFIG_TASKS_RCU_TRAMPOLINE_READERS kernels also by preemption
+ * outside trampoline text, see call_rcu_tasks().
*
* This is a very specialized primitive, intended only for a few uses in
* tracing and other situations requiring manipulation of function
@@ -1233,9 +1694,7 @@ void rcu_barrier_tasks(void)
}
EXPORT_SYMBOL_GPL(rcu_barrier_tasks);
-static int rcu_tasks_lazy_ms = -1;
-module_param(rcu_tasks_lazy_ms, int, 0444);
-
+#ifndef CONFIG_TASKS_RCU_TRAMPOLINE_READERS
static int __init rcu_spawn_tasks_kthread(void)
{
rcu_tasks.gp_sleep = HZ / 10;
@@ -1251,6 +1710,7 @@ static int __init rcu_spawn_tasks_kthread(void)
rcu_spawn_tasks_kthread_generic(&rcu_tasks);
return 0;
}
+#endif /* #ifndef CONFIG_TASKS_RCU_TRAMPOLINE_READERS */
#if !defined(CONFIG_TINY_RCU)
void show_rcu_tasks_classic_gp_kthread(void)
@@ -1279,6 +1739,7 @@ void rcu_tasks_get_gp_data(int *flags, unsigned long *gp_seq)
}
EXPORT_SYMBOL_GPL(rcu_tasks_get_gp_data);
+#ifndef CONFIG_TASKS_RCU_TRAMPOLINE_READERS
/*
* Protect against tasklist scan blind spot while the task is exiting and
* may be removed from the tasklist. Do this by adding the task to yet
@@ -1322,6 +1783,7 @@ void exit_tasks_rcu_finish(void)
list_del_init(&t->rcu_tasks_exit_list);
raw_spin_unlock_irqrestore_rcu_node(rtpcp, flags);
}
+#endif /* #ifndef CONFIG_TASKS_RCU_TRAMPOLINE_READERS */
#else /* #ifdef CONFIG_TASKS_RCU */
void exit_tasks_rcu_start(void) { }
diff --git a/kernel/rcu/update.c b/kernel/rcu/update.c
index b62735a678842..a122b8d1effb5 100644
--- a/kernel/rcu/update.c
+++ b/kernel/rcu/update.c
@@ -40,7 +40,9 @@
#include <linux/tick.h>
#include <linux/rcupdate_wait.h>
#include <linux/sched/isolation.h>
+#include <linux/context_tracking_state.h>
#include <linux/kprobes.h>
+#include <linux/module.h>
#include <linux/slab.h>
#include <linux/irq_work.h>
#include <linux/rcupdate_trace.h>
diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
index 53d5db60bfa56..f69f715913585 100644
--- a/kernel/trace/ftrace.c
+++ b/kernel/trace/ftrace.c
@@ -6076,6 +6076,29 @@ static void reset_direct(struct ftrace_ops *ops, unsigned long addr)
ops->trampoline = 0;
}
+/*
+ * A direct trampoline may live in module text rather than in dynamically
+ * allocated text that rcu_tasks_trampoline_text() recognises on its own (see
+ * samples/ftrace/ftrace-direct*.c). The trampoline itself must be a Tasks
+ * Trace reader across its call-out (see register_ftrace_direct()); marking the
+ * owning module here covers the instructions before it enters that reader and
+ * after it leaves it, where a task interrupted in the module's text must not be
+ * counted as Tasks-RCU quiescent, so that ftrace_shutdown()'s
+ * synchronize_rcu_tasks() still keeps the module text from being freed under
+ * it.
+ */
+static void ftrace_direct_mark_module(unsigned long addr)
+{
+#if defined(CONFIG_MODULES) && defined(CONFIG_TASKS_RCU_TRAMPOLINE_READERS)
+ struct module *mod;
+
+ guard(rcu)();
+ mod = __module_text_address(addr);
+ if (mod)
+ WRITE_ONCE(mod->ftrace_direct_tramp, true);
+#endif
+}
+
/**
* register_ftrace_direct - Call a custom trampoline directly
* for multiple functions registered in @ops
@@ -6090,6 +6113,17 @@ static void reset_direct(struct ftrace_ops *ops, unsigned long addr)
* and save the parameters of the function being traced, and restore them
* (or inject new ones if needed), before returning.
*
+ * Nothing but Tasks RCU keeps the trampoline at @addr alive while a task is
+ * executing it or is preempted in something it called. On architectures that
+ * select HAVE_RCU_TRAMPOLINE_READERS, Tasks RCU only waits for such a task if
+ * it is a Tasks Trace RCU reader, so the trampoline must enter one
+ * (rcu_read_lock_trace() or an assembly equivalent) before calling out and
+ * leave it before returning, just as that option requires of the in-kernel
+ * ftrace and BPF trampolines. The few instructions before and after are
+ * covered by the irq-exit check: automatically for trampolines outside kernel
+ * and module text (e.g. BPF images), and via ftrace_direct_mark_module() for
+ * trampolines in module text.
+ *
* Returns:
* 0 on success
* -EINVAL - The @ops object was already registered with this call or
@@ -6169,6 +6203,7 @@ int register_ftrace_direct(struct ftrace_ops *ops, unsigned long addr)
ops->flags |= MULTI_FLAGS;
ops->trampoline = FTRACE_REGS_ADDR;
ops->direct_call = addr;
+ ftrace_direct_mark_module(addr);
err = register_ftrace_function_nolock(ops);
if (err)
@@ -6237,6 +6272,8 @@ __modify_ftrace_direct(struct ftrace_ops *ops, unsigned long addr)
lockdep_assert_held_once(&direct_mutex);
+ ftrace_direct_mark_module(addr);
+
/* Enable the tmp_ops to have the same functions as the direct ops */
ftrace_ops_init(&tmp_ops);
tmp_ops.func_hash = ops->func_hash;
@@ -6419,6 +6456,7 @@ int update_ftrace_direct_add(struct ftrace_ops *ops, struct ftrace_hash *hash)
hlist_for_each_entry(entry, &hash->buckets[i], hlist) {
if (__ftrace_lookup_ip(direct_functions, entry->ip))
goto out_unlock;
+ ftrace_direct_mark_module(entry->direct);
}
}
@@ -6702,6 +6740,7 @@ int update_ftrace_direct_mod(struct ftrace_ops *ops, struct ftrace_hash *hash, b
tmp = __ftrace_lookup_ip(direct_hash, entry->ip);
if (!tmp)
continue;
+ ftrace_direct_mark_module(entry->direct);
tmp->direct = entry->direct;
}
}
diff --git a/samples/ftrace/ftrace-direct-modify.c b/samples/ftrace/ftrace-direct-modify.c
index 164d9dd6fd921..937c8d8c2a1be 100644
--- a/samples/ftrace/ftrace-direct-modify.c
+++ b/samples/ftrace/ftrace-direct-modify.c
@@ -2,6 +2,7 @@
#include <linux/module.h>
#include <linux/kthread.h>
#include <linux/ftrace.h>
+#include "ftrace-direct.h"
#if !defined(CONFIG_ARM64) && !defined(CONFIG_PPC32)
#include <asm/asm-offsets.h>
#endif
@@ -73,7 +74,9 @@ asm (
" pushq %rbp\n"
" movq %rsp, %rbp\n"
CALL_DEPTH_ACCOUNT
+ TRACE_RCU_READ_LOCK
" call my_direct_func1\n"
+ TRACE_RCU_READ_UNLOCK
" leave\n"
ASM_RET
" .size my_tramp1, .-my_tramp1\n"
@@ -85,7 +88,9 @@ asm (
" pushq %rbp\n"
" movq %rsp, %rbp\n"
CALL_DEPTH_ACCOUNT
+ TRACE_RCU_READ_LOCK
" call my_direct_func2\n"
+ TRACE_RCU_READ_UNLOCK
" leave\n"
ASM_RET
" .size my_tramp2, .-my_tramp2\n"
@@ -141,11 +146,13 @@ asm (
" .globl my_tramp1\n"
" my_tramp1:"
" hint 34\n" // bti c
+ TRACE_RCU_READ_LOCK
" sub sp, sp, #16\n"
" stp x9, x30, [sp]\n"
" bl my_direct_func1\n"
" ldp x30, x9, [sp]\n"
" add sp, sp, #16\n"
+ TRACE_RCU_READ_UNLOCK
" ret x9\n"
" .size my_tramp1, .-my_tramp1\n"
@@ -153,11 +160,13 @@ asm (
" .globl my_tramp2\n"
" my_tramp2:"
" hint 34\n" // bti c
+ TRACE_RCU_READ_LOCK
" sub sp, sp, #16\n"
" stp x9, x30, [sp]\n"
" bl my_direct_func2\n"
" ldp x30, x9, [sp]\n"
" add sp, sp, #16\n"
+ TRACE_RCU_READ_UNLOCK
" ret x9\n"
" .size my_tramp2, .-my_tramp2\n"
" .popsection\n"
diff --git a/samples/ftrace/ftrace-direct-multi-modify.c b/samples/ftrace/ftrace-direct-multi-modify.c
index b03766c6217bc..e12e5c8b83f06 100644
--- a/samples/ftrace/ftrace-direct-multi-modify.c
+++ b/samples/ftrace/ftrace-direct-multi-modify.c
@@ -2,6 +2,7 @@
#include <linux/module.h>
#include <linux/kthread.h>
#include <linux/ftrace.h>
+#include "ftrace-direct.h"
#if !defined(CONFIG_ARM64) && !defined(CONFIG_PPC32)
#include <asm/asm-offsets.h>
#endif
@@ -77,10 +78,12 @@ asm (
" pushq %rbp\n"
" movq %rsp, %rbp\n"
CALL_DEPTH_ACCOUNT
+ TRACE_RCU_READ_LOCK
" pushq %rdi\n"
" movq 8(%rbp), %rdi\n"
" call my_direct_func1\n"
" popq %rdi\n"
+ TRACE_RCU_READ_UNLOCK
" leave\n"
ASM_RET
" .size my_tramp1, .-my_tramp1\n"
@@ -92,10 +95,12 @@ asm (
" pushq %rbp\n"
" movq %rsp, %rbp\n"
CALL_DEPTH_ACCOUNT
+ TRACE_RCU_READ_LOCK
" pushq %rdi\n"
" movq 8(%rbp), %rdi\n"
" call my_direct_func2\n"
" popq %rdi\n"
+ TRACE_RCU_READ_UNLOCK
" leave\n"
ASM_RET
" .size my_tramp2, .-my_tramp2\n"
@@ -154,6 +159,7 @@ asm (
" .globl my_tramp1\n"
" my_tramp1:"
" hint 34\n" // bti c
+ TRACE_RCU_READ_LOCK
" sub sp, sp, #32\n"
" stp x9, x30, [sp]\n"
" str x0, [sp, #16]\n"
@@ -162,6 +168,7 @@ asm (
" ldp x30, x9, [sp]\n"
" ldr x0, [sp, #16]\n"
" add sp, sp, #32\n"
+ TRACE_RCU_READ_UNLOCK
" ret x9\n"
" .size my_tramp1, .-my_tramp1\n"
@@ -169,6 +176,7 @@ asm (
" .globl my_tramp2\n"
" my_tramp2:"
" hint 34\n" // bti c
+ TRACE_RCU_READ_LOCK
" sub sp, sp, #32\n"
" stp x9, x30, [sp]\n"
" str x0, [sp, #16]\n"
@@ -177,6 +185,7 @@ asm (
" ldp x30, x9, [sp]\n"
" ldr x0, [sp, #16]\n"
" add sp, sp, #32\n"
+ TRACE_RCU_READ_UNLOCK
" ret x9\n"
" .size my_tramp2, .-my_tramp2\n"
" .popsection\n"
diff --git a/samples/ftrace/ftrace-direct-multi.c b/samples/ftrace/ftrace-direct-multi.c
index 3fe6ddaf0b69f..a970464ed3786 100644
--- a/samples/ftrace/ftrace-direct-multi.c
+++ b/samples/ftrace/ftrace-direct-multi.c
@@ -3,6 +3,7 @@
#include <linux/mm.h> /* for handle_mm_fault() */
#include <linux/ftrace.h>
+#include "ftrace-direct.h"
#include <linux/sched/stat.h>
#if !defined(CONFIG_ARM64) && !defined(CONFIG_PPC32)
#include <asm/asm-offsets.h>
@@ -56,10 +57,12 @@ asm (
" pushq %rbp\n"
" movq %rsp, %rbp\n"
CALL_DEPTH_ACCOUNT
+ TRACE_RCU_READ_LOCK
" pushq %rdi\n"
" movq 8(%rbp), %rdi\n"
" call my_direct_func\n"
" popq %rdi\n"
+ TRACE_RCU_READ_UNLOCK
" leave\n"
ASM_RET
" .size my_tramp, .-my_tramp\n"
@@ -101,6 +104,7 @@ asm (
" .globl my_tramp\n"
" my_tramp:"
" hint 34\n" // bti c
+ TRACE_RCU_READ_LOCK
" sub sp, sp, #32\n"
" stp x9, x30, [sp]\n"
" str x0, [sp, #16]\n"
@@ -109,6 +113,7 @@ asm (
" ldp x30, x9, [sp]\n"
" ldr x0, [sp, #16]\n"
" add sp, sp, #32\n"
+ TRACE_RCU_READ_UNLOCK
" ret x9\n"
" .size my_tramp, .-my_tramp\n"
" .popsection\n"
diff --git a/samples/ftrace/ftrace-direct-too.c b/samples/ftrace/ftrace-direct-too.c
index bf2411aa6fd7a..abc098c2ab7a8 100644
--- a/samples/ftrace/ftrace-direct-too.c
+++ b/samples/ftrace/ftrace-direct-too.c
@@ -3,6 +3,7 @@
#include <linux/mm.h> /* for handle_mm_fault() */
#include <linux/ftrace.h>
+#include "ftrace-direct.h"
#if !defined(CONFIG_ARM64) && !defined(CONFIG_PPC32)
#include <asm/asm-offsets.h>
#endif
@@ -61,6 +62,7 @@ asm (
" pushq %rbp\n"
" movq %rsp, %rbp\n"
CALL_DEPTH_ACCOUNT
+ TRACE_RCU_READ_LOCK
" pushq %rdi\n"
" pushq %rsi\n"
" pushq %rdx\n"
@@ -70,6 +72,7 @@ asm (
" popq %rdx\n"
" popq %rsi\n"
" popq %rdi\n"
+ TRACE_RCU_READ_UNLOCK
" leave\n"
ASM_RET
" .size my_tramp, .-my_tramp\n"
@@ -110,6 +113,7 @@ asm (
" .globl my_tramp\n"
" my_tramp:"
" hint 34\n" // bti c
+ TRACE_RCU_READ_LOCK
" sub sp, sp, #48\n"
" stp x9, x30, [sp]\n"
" stp x0, x1, [sp, #16]\n"
@@ -119,6 +123,7 @@ asm (
" ldp x0, x1, [sp, #16]\n"
" ldp x2, x3, [sp, #32]\n"
" add sp, sp, #48\n"
+ TRACE_RCU_READ_UNLOCK
" ret x9\n"
" .size my_tramp, .-my_tramp\n"
" .popsection\n"
diff --git a/samples/ftrace/ftrace-direct.c b/samples/ftrace/ftrace-direct.c
index 5368c8c39cbb4..99b65ad2fccc0 100644
--- a/samples/ftrace/ftrace-direct.c
+++ b/samples/ftrace/ftrace-direct.c
@@ -3,6 +3,7 @@
#include <linux/sched.h> /* for wake_up_process() */
#include <linux/ftrace.h>
+#include "ftrace-direct.h"
#if !defined(CONFIG_ARM64) && !defined(CONFIG_PPC32)
#include <asm/asm-offsets.h>
#endif
@@ -54,9 +55,11 @@ asm (
" pushq %rbp\n"
" movq %rsp, %rbp\n"
CALL_DEPTH_ACCOUNT
+ TRACE_RCU_READ_LOCK
" pushq %rdi\n"
" call my_direct_func\n"
" popq %rdi\n"
+ TRACE_RCU_READ_UNLOCK
" leave\n"
ASM_RET
" .size my_tramp, .-my_tramp\n"
@@ -97,6 +100,7 @@ asm (
" .globl my_tramp\n"
" my_tramp:"
" hint 34\n" // bti c
+ TRACE_RCU_READ_LOCK
" sub sp, sp, #32\n"
" stp x9, x30, [sp]\n"
" str x0, [sp, #16]\n"
@@ -104,6 +108,7 @@ asm (
" ldp x30, x9, [sp]\n"
" ldr x0, [sp, #16]\n"
" add sp, sp, #32\n"
+ TRACE_RCU_READ_UNLOCK
" ret x9\n"
" .size my_tramp, .-my_tramp\n"
" .popsection\n"
diff --git a/samples/ftrace/ftrace-direct.h b/samples/ftrace/ftrace-direct.h
new file mode 100644
index 0000000000000..726f67048ff5c
--- /dev/null
+++ b/samples/ftrace/ftrace-direct.h
@@ -0,0 +1,126 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef _SAMPLES_FTRACE_DIRECT_H
+#define _SAMPLES_FTRACE_DIRECT_H
+
+#include <linux/stringify.h>
+
+/*
+ * A direct-call trampoline is entered with no lock, refcount or RCU marker
+ * held; only Tasks RCU keeps it (and, for a module, its text) alive while a
+ * task is inside it or preempted in something it called. On architectures
+ * that select HAVE_RCU_TRAMPOLINE_READERS, Tasks RCU only waits for such a
+ * task while it is a Tasks Trace RCU reader, so the trampoline must enter one
+ * before calling out and leave it afterwards, exactly like the ftrace and BPF
+ * trampolines do. See register_ftrace_direct(). The instructions before the
+ * lock and after the unlock are covered by ftrace_direct_mark_module().
+ *
+ * These are rcu_read_lock_trace() / rcu_read_unlock_trace() open-coded as
+ * instruction strings for use inside the samples' asm() trampolines, after
+ * the versions in arch/x86/kernel/ftrace_64.S and
+ * arch/arm64/kernel/entry-ftrace.S. The scratch registers are caller-saved
+ * and not argument registers, so they are dead on entry to and exit from an
+ * fentry trampoline; the flags are clobbered.
+ *
+ * The generated asm-offsets.h is only pulled in on the architectures that need
+ * it here: it is not generally safe to include from C (e.g. PPC32's TASK_SIZE
+ * and arm64's TRAMP_VALIAS clash with the C definitions).
+ */
+#if defined(CONFIG_TASKS_RCU_TRAMPOLINE_READERS) && defined(CONFIG_X86_64)
+
+#include <asm/asm-offsets.h>
+
+#ifndef CONFIG_TASKS_TRACE_RCU_NO_MB
+#define TRACE_RCU_MB " lock addl $0, -4(%rsp)\n"
+#else
+#define TRACE_RCU_MB
+#endif
+
+#define TRACE_RCU_READ_LOCK \
+ " movq %gs:current_task(%rip), %r11\n" \
+ " movl " __stringify(TASK_trc_reader_nesting) "(%r11), %r10d\n" \
+ " incl " __stringify(TASK_trc_reader_nesting) "(%r11)\n" \
+ " testl %r10d, %r10d\n" \
+ " jnz 771f\n" \
+ " movq rcu_tasks_trace_srcu_struct+" __stringify(SRCU_srcu_ctrp) "(%rip), %r10\n" \
+ " incq %gs:" __stringify(SRCU_CTR_srcu_locks) "(%r10)\n" \
+ " movq %r10, " __stringify(TASK_trc_reader_scp) "(%r11)\n" \
+ TRACE_RCU_MB \
+ "771:\n"
+
+#define TRACE_RCU_READ_UNLOCK \
+ " movq %gs:current_task(%rip), %r11\n" \
+ " movl " __stringify(TASK_trc_reader_nesting) "(%r11), %r10d\n" \
+ " subl $1, %r10d\n" \
+ " jnz 772f\n" \
+ " movq " __stringify(TASK_trc_reader_scp) "(%r11), %r10\n" \
+ " movl $0, " __stringify(TASK_trc_reader_nesting) "(%r11)\n" \
+ TRACE_RCU_MB \
+ " incq %gs:" __stringify(SRCU_CTR_srcu_unlocks) "(%r10)\n" \
+ " jmp 773f\n" \
+ "772: movl %r10d, " __stringify(TASK_trc_reader_nesting) "(%r11)\n" \
+ "773:\n"
+
+#elif defined(CONFIG_TASKS_RCU_TRAMPOLINE_READERS) && defined(CONFIG_ARM64)
+
+#include <asm/alternative-macros.h>
+#include <asm/cpucaps.h>
+/* arm64's asm-offsets.h redefines TRAMP_VALIAS from <asm/fixmap.h>. */
+#pragma push_macro("TRAMP_VALIAS")
+#undef TRAMP_VALIAS
+#include <asm/asm-offsets.h>
+#pragma pop_macro("TRAMP_VALIAS")
+
+#ifndef CONFIG_TASKS_TRACE_RCU_NO_MB
+#define TRACE_RCU_MB " dmb ish\n"
+#else
+#define TRACE_RCU_MB
+#endif
+
+#define TRACE_RCU_SRCU_CTRP "rcu_tasks_trace_srcu_struct+" __stringify(SRCU_SRCU_CTRP)
+
+/* x14 = this CPU's offset; then atomically increment the long at x14 + \areg */
+#define TRACE_RCU_PERCPU_INC(areg) \
+ ALTERNATIVE(" mrs x14, tpidr_el1\n", " mrs x14, tpidr_el2\n", \
+ ARM64_HAS_VIRT_HOST_EXTN) \
+ " add x14, x14, " areg "\n" \
+ "778: ldxr x15, [x14]\n" \
+ " add x15, x15, #1\n" \
+ " stxr w16, x15, [x14]\n" \
+ " cbnz w16, 778b\n"
+
+#define TRACE_RCU_READ_LOCK \
+ " mrs x12, sp_el0\n" \
+ " ldr w13, [x12, #" __stringify(TSK_TRC_READER_NESTING) "]\n" \
+ " add w14, w13, #1\n" \
+ " str w14, [x12, #" __stringify(TSK_TRC_READER_NESTING) "]\n" \
+ " cbnz w13, 771f\n" \
+ " adrp x13, " TRACE_RCU_SRCU_CTRP "\n" \
+ " ldr x13, [x13, #:lo12:" TRACE_RCU_SRCU_CTRP "]\n" \
+ " str x13, [x12, #" __stringify(TSK_TRC_READER_SCP) "]\n" \
+ " add x13, x13, #" __stringify(SRCU_CTR_SRCU_LOCKS) "\n" \
+ TRACE_RCU_PERCPU_INC("x13") \
+ TRACE_RCU_MB \
+ "771:\n"
+
+#define TRACE_RCU_READ_UNLOCK \
+ " mrs x12, sp_el0\n" \
+ " ldr w13, [x12, #" __stringify(TSK_TRC_READER_NESTING) "]\n" \
+ " subs w13, w13, #1\n" \
+ " b.ne 772f\n" \
+ " ldr x13, [x12, #" __stringify(TSK_TRC_READER_SCP) "]\n" \
+ " str wzr, [x12, #" __stringify(TSK_TRC_READER_NESTING) "]\n" \
+ TRACE_RCU_MB \
+ " add x13, x13, #" __stringify(SRCU_CTR_SRCU_UNLOCKS) "\n" \
+ TRACE_RCU_PERCPU_INC("x13") \
+ " b 773f\n" \
+ "772: str w13, [x12, #" __stringify(TSK_TRC_READER_NESTING) "]\n" \
+ "773:\n"
+
+#else
+
+#define TRACE_RCU_READ_LOCK
+#define TRACE_RCU_READ_UNLOCK
+
+#endif
+
+#endif /* _SAMPLES_FTRACE_DIRECT_H */
|
| 3/2 | 2026/09/24 04:18 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 37603 output: 29 thoughts: 1141 Thoughts: **Analysis of the Provided Patch Series** Okay, let's break this down. My task is to evaluate this patch series for its fuzzing potential with syzkaller, specifically targeting the amd64 architecture. I need to determine if it's worth fuzzing and identify key areas for focused fuzzing. First, I've reviewed the instructions. Reachability is paramount; it needs to run in a standard QEMU environment on amd64. The instructions highlight that modifications to memory management, synchronization/locking, BPF, scheduler, networking, VFS, and syscall handling are all prime candidates for fuzzing. Refactorings, code cleanups, and defensive checks also get a thumbs up. Importantly, I need to identify *non-ubiquitous* kernel functions to focus the fuzzing efforts. Now, let's look at the patch diff itself. I see touches in documentation (RCU, interestingly), architecture-specific code (arm64 and a lot of x86), Kconfig settings (`HAVE_RCU_TRAMPOLINE_READERS`), and a wide range of core kernel subsystems. I'm seeing changes to `arch/x86/kernel/ftrace.c`, `arch/x86/net/bpf_jit_comp.c`, `kernel/rcu/tasks.h`, `kernel/trace/ftrace.c`, `kernel/kprobes.c`, `kernel/entry/common.c`, and several include files dealing with RCU, Kprobes, and tracing. This is immediately interesting. This patch seems to be touching core RCU (crucial for concurrency), BPF JIT compilation (which can have security implications), ftrace (critical for debugging and performance analysis), kprobes (dynamic instrumentation), and entry code (how the kernel interacts with the hardware/userspace). This is exactly the kind of critical core infrastructure I want to examine. Based on this quick analysis, **WorthFuzzing = TRUE**. This code looks highly relevant for fuzzing and is clearly reachable in a standard QEMU environment on amd64. I should be able to trigger this logic with regular system calls. Now, I'm going to leverage the tools and dive into the code more deeply to confirm this assessment, check on specific configs, and begin scoping out particular functions and call paths that would be good targets for syzkaller. |
| 4/2 | 2026/09/24 04:19 | tool | grepper |
0mArgs:{
"Expression": "CONFIG_TASKS_RCU_TRAMPOLINE_READERS"
}Results: {
"Output": "Documentation/RCU/Design/Requirements/Requirements.rst=2760=same three APIs but implement the grace period differently\nDocumentation/RCU/Design/Requirements/Requirements.rst:2761:(``CONFIG_TASKS_RCU_TRAMPOLINE_READERS``). There, every trampoline whose\nDocumentation/RCU/Design/Requirements/Requirements.rst-2762-lifetime Tasks RCU guards enters a Tasks Trace RCU read-side critical\n--\narch/arm64/kernel/asm-offsets.c=28=int main(void)\n--\narch/arm64/kernel/asm-offsets.c-43-#endif\narch/arm64/kernel/asm-offsets.c:44:#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS\narch/arm64/kernel/asm-offsets.c-45- DEFINE(TSK_TRC_READER_NESTING,\toffsetof(struct task_struct, trc_reader_nesting));\n--\narch/arm64/kernel/entry-ftrace.S-40-\t.macro trace_rcu_read_lock\narch/arm64/kernel/entry-ftrace.S:41:#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS\narch/arm64/kernel/entry-ftrace.S-42-\tmrs\tx12, sp_el0\t\t\t\t// current\n--\narch/arm64/kernel/entry-ftrace.S-60-\t.macro trace_rcu_read_unlock\narch/arm64/kernel/entry-ftrace.S:61:#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS\narch/arm64/kernel/entry-ftrace.S-62-\tmrs\tx12, sp_el0\t\t\t\t// current\n--\narch/arm64/kernel/ftrace.c-19-\narch/arm64/kernel/ftrace.c:20:#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS\narch/arm64/kernel/ftrace.c-21-extern void ftrace_static_tramp_end(void);\n--\narch/arm64/net/bpf_jit_comp.c=2604=static void emit_trace_rcu_reader(struct jit_ctx *ctx, bool lock)\narch/arm64/net/bpf_jit_comp.c-2605-{\narch/arm64/net/bpf_jit_comp.c:2606:#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS\narch/arm64/net/bpf_jit_comp.c-2607-\tconst int nesting_off = offsetof(struct task_struct, trc_reader_nesting);\n--\narch/x86/kernel/asm-offsets.c=35=static void __used common(void)\n--\narch/x86/kernel/asm-offsets.c-49-#endif\narch/x86/kernel/asm-offsets.c:50:#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS\narch/x86/kernel/asm-offsets.c-51-\tOFFSET(TASK_trc_reader_nesting, task_struct, trc_reader_nesting);\n--\narch/x86/kernel/ftrace.c=273=static inline void tramp_free(void *tramp)\n--\narch/x86/kernel/ftrace.c-277-\narch/x86/kernel/ftrace.c:278:#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS\narch/x86/kernel/ftrace.c-279-extern void ftrace_static_tramp_end(void);\n--\narch/x86/kernel/ftrace.c=302=bool arch_rcu_tasks_trampoline_text(unsigned long ip)\n--\narch/x86/kernel/ftrace.c-318-}\narch/x86/kernel/ftrace.c:319:#endif /* CONFIG_TASKS_RCU_TRAMPOLINE_READERS */\narch/x86/kernel/ftrace.c-320-\n--\narch/x86/kernel/ftrace_64.S=144=SYM_FUNC_END(ftrace_stub_graph)\n--\narch/x86/kernel/ftrace_64.S-159-.macro TRACE_RCU_READ_LOCK\narch/x86/kernel/ftrace_64.S:160:#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS\narch/x86/kernel/ftrace_64.S-161-\tmovq\tPER_CPU_VAR(current_task), %rcx\n--\narch/x86/kernel/ftrace_64.S-176-.macro TRACE_RCU_READ_UNLOCK\narch/x86/kernel/ftrace_64.S:177:#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS\narch/x86/kernel/ftrace_64.S-178-\tmovq\tPER_CPU_VAR(current_task), %rcx\n--\narch/x86/kernel/kprobes/opt.c=94=static void synthesize_set_arg1(kprobe_opcode_t *addr, unsigned long val)\n--\narch/x86/kernel/kprobes/opt.c-111- */\narch/x86/kernel/kprobes/opt.c:112:#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS\narch/x86/kernel/kprobes/opt.c-113-#ifndef CONFIG_TASKS_TRACE_RCU_NO_MB\n--\narch/x86/net/bpf_jit_comp.c=748=static void emit_trace_rcu_reader(u8 **pprog, bool lock)\narch/x86/net/bpf_jit_comp.c-749-{\narch/x86/net/bpf_jit_comp.c:750:#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS\narch/x86/net/bpf_jit_comp.c-751-\tconst u32 nesting_off = offsetof(struct task_struct, trc_reader_nesting);\n--\ninclude/linux/module.h=397=struct module {\n--\ninclude/linux/module.h-523-#endif\ninclude/linux/module.h:524:#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS\ninclude/linux/module.h-525-\t/*\n--\ninclude/linux/rcupdate.h=52=void rcu_barrier_tasks(void);\n--\ninclude/linux/rcupdate.h-54-/*\ninclude/linux/rcupdate.h:55: * Trampoline-reader Tasks RCU (CONFIG_TASKS_RCU_TRAMPOLINE_READERS), see\ninclude/linux/rcupdate.h-56- * kernel/rcu/tasks.h. rcu_tasks_irq_resched_enter()/_exit() bracket the\n--\ninclude/linux/rcupdate.h=64=bool arch_rcu_tasks_trampoline_text(unsigned long ip);\ninclude/linux/rcupdate.h:65:#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS\ninclude/linux/rcupdate.h-66-void rcu_tasks_wait_irq_preempted(bool (*inside)(unsigned long ip));\n--\ninclude/linux/rcupdate.h=187=static inline void rcu_nocb_flush_deferred_wakeup(void) { }\n--\ninclude/linux/rcupdate.h-199-# ifdef CONFIG_TASKS_RCU\ninclude/linux/rcupdate.h:200:# ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS\ninclude/linux/rcupdate.h-201-void rcu_tasks_note_qs(struct task_struct *t, bool preempt);\n--\nkernel/entry/common.c=138=void raw_irqentry_exit_cond_resched(struct pt_regs *regs)\n--\nkernel/entry/common.c-145-\t\tif (need_resched() \u0026\u0026 arch_irqentry_exit_need_resched()) {\nkernel/entry/common.c:146:\t\t\tif (IS_ENABLED(CONFIG_TASKS_RCU_TRAMPOLINE_READERS))\nkernel/entry/common.c-147-\t\t\t\trcu_tasks_irq_resched_enter(instruction_pointer(regs));\nkernel/entry/common.c-148-\t\t\tpreempt_schedule_irq();\nkernel/entry/common.c:149:\t\t\tif (IS_ENABLED(CONFIG_TASKS_RCU_TRAMPOLINE_READERS))\nkernel/entry/common.c-150-\t\t\t\trcu_tasks_irq_resched_exit();\n--\nkernel/rcu/rcutorture.c=1117=static struct rcu_torture_ops trivial_preempt_ops = {\n--\nkernel/rcu/rcutorture.c-1146- * A classic Tasks RCU reader is any stretch of kernel code that does not\nkernel/rcu/rcutorture.c:1147: * voluntarily block. With CONFIG_TASKS_RCU_TRAMPOLINE_READERS a preemption\nkernel/rcu/rcutorture.c-1148- * outside trampoline text also ends it, and what a trampoline does to stay\n--\nkernel/rcu/rcutorture.c=1151=static int tasks_torture_read_lock(void)\nkernel/rcu/rcutorture.c-1152-{\nkernel/rcu/rcutorture.c:1153:\tif (IS_ENABLED(CONFIG_TASKS_RCU_TRAMPOLINE_READERS))\nkernel/rcu/rcutorture.c-1154-\t\trcu_read_lock_trace();\n--\nkernel/rcu/rcutorture.c=1158=static void tasks_torture_read_unlock(int idx)\nkernel/rcu/rcutorture.c-1159-{\nkernel/rcu/rcutorture.c:1160:\tif (IS_ENABLED(CONFIG_TASKS_RCU_TRAMPOLINE_READERS))\nkernel/rcu/rcutorture.c-1161-\t\trcu_read_unlock_trace();\n--\nkernel/rcu/tasks.h=136=static struct rcu_tasks rt_name =\t\t\t\t\t\t\t\\\n--\nkernel/rcu/tasks.h-154-\nkernel/rcu/tasks.h:155:#if defined(CONFIG_TASKS_RCU) \u0026\u0026 !defined(CONFIG_TASKS_RCU_TRAMPOLINE_READERS)\nkernel/rcu/tasks.h-156-\n--\nkernel/rcu/tasks.h=758=static void rcu_tasks_torture_stats_print_generic(struct rcu_tasks *rtp, char *tt,\n--\nkernel/rcu/tasks.h-804-\nkernel/rcu/tasks.h:805:#if defined(CONFIG_TASKS_RCU) \u0026\u0026 !defined(CONFIG_TASKS_RCU_TRAMPOLINE_READERS)\nkernel/rcu/tasks.h-806-\n--\nkernel/rcu/tasks.h=812=static void rcu_tasks_wait_gp(struct rcu_tasks *rtp)\n--\nkernel/rcu/tasks.h-899-\nkernel/rcu/tasks.h:900:#endif /* #if defined(CONFIG_TASKS_RCU) \u0026\u0026 !defined(CONFIG_TASKS_RCU_TRAMPOLINE_READERS) */\nkernel/rcu/tasks.h-901-\n--\nkernel/rcu/tasks.h=905=module_param(rcu_tasks_lazy_ms, int, 0444);\nkernel/rcu/tasks.h-906-\nkernel/rcu/tasks.h:907:#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS\nkernel/rcu/tasks.h-908-\n--\nkernel/rcu/tasks.h=1351=void exit_tasks_rcu_finish(void) { }\nkernel/rcu/tasks.h-1352-\nkernel/rcu/tasks.h:1353:#else /* #ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS */\nkernel/rcu/tasks.h-1354-\n--\nkernel/rcu/tasks.h=1612=static void tasks_rcu_exit_stall(struct timer_list *unused)\n--\nkernel/rcu/tasks.h-1626-\nkernel/rcu/tasks.h:1627:#endif /* #else #ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS */\nkernel/rcu/tasks.h-1628-\n--\nkernel/rcu/tasks.h-1642- * through a safe state, not so much for data-structure synchronization.\nkernel/rcu/tasks.h:1643: * On CONFIG_TASKS_RCU_TRAMPOLINE_READERS kernels a preemption outside\nkernel/rcu/tasks.h-1644- * trampoline text also ends one, and a reader whose protected window\n--\nkernel/rcu/tasks.h=1657=EXPORT_SYMBOL_GPL(call_rcu_tasks);\n--\nkernel/rcu/tasks.h-1667- * to synchronize_rcu_tasks(), and (in theory, anyway) cond_resched();\nkernel/rcu/tasks.h:1668: * on CONFIG_TASKS_RCU_TRAMPOLINE_READERS kernels also by preemption\nkernel/rcu/tasks.h-1669- * outside trampoline text, see call_rcu_tasks().\n--\nkernel/rcu/tasks.h=1695=EXPORT_SYMBOL_GPL(rcu_barrier_tasks);\nkernel/rcu/tasks.h-1696-\nkernel/rcu/tasks.h:1697:#ifndef CONFIG_TASKS_RCU_TRAMPOLINE_READERS\nkernel/rcu/tasks.h-1698-static int __init rcu_spawn_tasks_kthread(void)\n--\nkernel/rcu/tasks.h-1712-}\nkernel/rcu/tasks.h:1713:#endif /* #ifndef CONFIG_TASKS_RCU_TRAMPOLINE_READERS */\nkernel/rcu/tasks.h-1714-\n--\nkernel/rcu/tasks.h=1740=EXPORT_SYMBOL_GPL(rcu_tasks_get_gp_data);\nkernel/rcu/tasks.h-1741-\nkernel/rcu/tasks.h:1742:#ifndef CONFIG_TASKS_RCU_TRAMPOLINE_READERS\nkernel/rcu/tasks.h-1743-/*\n--\nkernel/rcu/tasks.h=1774=void exit_tasks_rcu_finish(void)\n--\nkernel/rcu/tasks.h-1785-}\nkernel/rcu/tasks.h:1786:#endif /* #ifndef CONFIG_TASKS_RCU_TRAMPOLINE_READERS */\nkernel/rcu/tasks.h-1787-\n--\nkernel/trace/ftrace.c=6090=static void ftrace_direct_mark_module(unsigned long addr)\nkernel/trace/ftrace.c-6091-{\nkernel/trace/ftrace.c:6092:#if defined(CONFIG_MODULES) \u0026\u0026 defined(CONFIG_TASKS_RCU_TRAMPOLINE_READERS)\nkernel/trace/ftrace.c-6093-\tstruct module *mod;\n--\nsamples/ftrace/ftrace-direct.h-27- */\nsamples/ftrace/ftrace-direct.h:28:#if defined(CONFIG_TASKS_RCU_TRAMPOLINE_READERS) \u0026\u0026 defined(CONFIG_X86_64)\nsamples/ftrace/ftrace-direct.h-29-\n--\nsamples/ftrace/ftrace-direct.h-62-\nsamples/ftrace/ftrace-direct.h:63:#elif defined(CONFIG_TASKS_RCU_TRAMPOLINE_READERS) \u0026\u0026 defined(CONFIG_ARM64)\nsamples/ftrace/ftrace-direct.h-64-\n"
} |
| 5/2 | 2026/09/24 04:19 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 43402 output: 35 thoughts: 61 Thoughts: Okay, let's dive into this. Here's how I'd approach this task, walking through my thought process: **Diving Deep into RCU and Trampolines** Alright, I need to get a handle on `HAVE_RCU_TRAMPOLINE_READERS` and `TASKS_RCU_TRAMPOLINE_READERS`. Immediately, my mental roadmap points to the Kconfig definition. I know that's where the heart of the configuration lies. I'm going to jump straight into `kernel/rcu/Kconfig`. I'm specifically targeting around lines 105 to 135. That's usually where these kinds of feature toggles get defined within a subsystem's Kconfig file. I'm expecting to find the conditions under which these options are enabled, what they *do*, and potentially any dependencies they have on other kernel features or hardware capabilities. Understanding these definitions is absolutely crucial before I try to integrate or modify any RCU (Read-Copy-Update) or trampoline code. I need to be sure I understand the precise context. |
| 6/2 | 2026/09/24 04:19 | tool | read-file |
0mArgs:{
"File": "kernel/rcu/Kconfig",
"FirstLine": 100,
"LineCount": 40
}Results: {
"Contents": " 100:\t\t Tasks Trace RCU readers in trampolines. Not for manual\n 101:\t\t selection in most cases.\n 102:\t\n 103:\tconfig NEED_TASKS_RCU\n 104:\t\tbool\n 105:\t\tdefault n\n 106:\t\n 107:\tconfig TASKS_RCU\n 108:\t\tbool\n 109:\t\tdefault NEED_TASKS_RCU \u0026\u0026 PREEMPTION\n 110:\t\tselect IRQ_WORK\n 111:\t\n 112:\tconfig HAVE_RCU_TRAMPOLINE_READERS\n 113:\t\tbool\n 114:\t\thelp\n 115:\t\t Select this if the architecture uses the generic irq entry code and\n 116:\t\t every trampoline whose lifetime Tasks RCU guards on it (ftrace\n 117:\t\t trampolines, kprobe out-of-line and optimized-probe slots, BPF\n 118:\t\t trampolines, out-of-line ftrace direct-call trampolines) enters a\n 119:\t\t Tasks Trace RCU read-side critical section before calling out of\n 120:\t\t the trampoline and leaves it before returning, and any core text\n 121:\t\t that runs on behalf of such a trampoline outside that reader is\n 122:\t\t reported by arch_rcu_tasks_trampoline_text(). The assembly readers\n 123:\t\t use the this_cpu_inc() form of SRCU-fast, hence !NEED_SRCU_NMI_SAFE.\n 124:\t\n 125:\tconfig TASKS_RCU_TRAMPOLINE_READERS\n 126:\t\tdef_bool TASKS_RCU \u0026\u0026 HAVE_RCU_TRAMPOLINE_READERS \u0026\u0026 GENERIC_IRQ_ENTRY \u0026\u0026 !NEED_SRCU_NMI_SAFE\n 127:\t\tselect TASKS_TRACE_RCU\n 128:\t\thelp\n 129:\t\t Implement the Tasks RCU grace period as a per-CPU pass over\n 130:\t\t context switches and irq-exit reschedules outside trampoline text\n 131:\t\t plus a Tasks Trace RCU grace period, instead of waiting for every\n 132:\t\t task to voluntarily context switch. See kernel/rcu/tasks.h.\n 133:\t\n 134:\tconfig FORCE_TASKS_RUDE_RCU\n 135:\t\tbool \"Force selection of Tasks Rude RCU\"\n 136:\t\tdepends on RCU_EXPERT\n 137:\t\tselect TASKS_RUDE_RCU\n 138:\t\tdefault n\n 139:\t\thelp\n"
} |
| 7/2 | 2026/09/24 04:19 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 44413 output: 37 thoughts: 169 Thoughts: **Investigating the Configuration Options for RCU and Trampoline Readers in the x86 Architecture** Okay, let's dive into how the `HAVE_RCU_TRAMPOLINE_READERS` option gets selected within the x86 architecture's kernel configuration. Specifically, I'm focusing on the arch/x86/Kconfig file. Ah, here it is: `select HAVE_RCU_TRAMPOLINE_READERS if X86_64 && SMP && DYNAMIC_FTRACE`. So, it's selected when we have a 64-bit x86 system, symmetric multiprocessing (SMP) is enabled, and dynamic ftrace is also in play. Makes sense, that's a reasonable set of prerequisites for wanting trampoline readers. Now, I need to understand how the `TASKS_RCU` option is enabled. Let's see... `config TASKS_RCU` is defined with a `default NEED_TASKS_RCU && PREEMPTION`. Alright, so `TASKS_RCU` is the default if `NEED_TASKS_RCU` and `PREEMPTION` are both enabled. Now I need to figure out the dependencies and how `NEED_TASKS_RCU`, `TASKS_RCU` or `FORCE_TASKS_RCU` gets triggered. Time for a quick `grep` search for all instances of `NEED_TASKS_RCU`, `TASKS_RCU`, and `FORCE_TASKS_RCU`. That should tell me which other Kconfig options are influencing these settings. |
| 8/2 | 2026/09/24 04:19 | tool | grepper |
0mArgs:{
"Expression": "(NEED_TASKS_RCU|TASKS_RCU)",
"PathPrefix": "kernel/"
}Results: {
"Output": "kernel/bpf/Kconfig=27=config BPF_SYSCALL\n--\nkernel/bpf/Kconfig-30-\tselect IRQ_WORK\nkernel/bpf/Kconfig:31:\tselect NEED_TASKS_RCU\nkernel/bpf/Kconfig-32-\tselect TASKS_TRACE_RCU\n--\nkernel/bpf/trampoline.c=612=static void bpf_tramp_image_put(struct bpf_tramp_image *im)\n--\nkernel/bpf/trampoline.c-644-\t\tWARN_ON(err);\nkernel/bpf/trampoline.c:645:\t\tif (IS_ENABLED(CONFIG_TASKS_RCU))\nkernel/bpf/trampoline.c-646-\t\t\tcall_rcu_tasks(\u0026im-\u003ercu, __bpf_tramp_image_put_rcu_tasks);\n--\nkernel/context_tracking.c=42=static __always_inline void rcu_task_exit(void)\nkernel/context_tracking.c-43-{\nkernel/context_tracking.c:44:#if defined(CONFIG_TASKS_RCU) \u0026\u0026 defined(CONFIG_NO_HZ_FULL)\nkernel/context_tracking.c-45-\tWRITE_ONCE(current-\u003ercu_tasks_idle_cpu, smp_processor_id());\nkernel/context_tracking.c:46:#endif /* #if defined(CONFIG_TASKS_RCU) \u0026\u0026 defined(CONFIG_NO_HZ_FULL) */\nkernel/context_tracking.c-47-}\n--\nkernel/context_tracking.c=50=static __always_inline void rcu_task_enter(void)\nkernel/context_tracking.c-51-{\nkernel/context_tracking.c:52:#if defined(CONFIG_TASKS_RCU) \u0026\u0026 defined(CONFIG_NO_HZ_FULL)\nkernel/context_tracking.c-53-\tWRITE_ONCE(current-\u003ercu_tasks_idle_cpu, -1);\nkernel/context_tracking.c:54:#endif /* #if defined(CONFIG_TASKS_RCU) \u0026\u0026 defined(CONFIG_NO_HZ_FULL) */\nkernel/context_tracking.c-55-}\n--\nkernel/entry/common.c=138=void raw_irqentry_exit_cond_resched(struct pt_regs *regs)\n--\nkernel/entry/common.c-145-\t\tif (need_resched() \u0026\u0026 arch_irqentry_exit_need_resched()) {\nkernel/entry/common.c:146:\t\t\tif (IS_ENABLED(CONFIG_TASKS_RCU_TRAMPOLINE_READERS))\nkernel/entry/common.c-147-\t\t\t\trcu_tasks_irq_resched_enter(instruction_pointer(regs));\nkernel/entry/common.c-148-\t\t\tpreempt_schedule_irq();\nkernel/entry/common.c:149:\t\t\tif (IS_ENABLED(CONFIG_TASKS_RCU_TRAMPOLINE_READERS))\nkernel/entry/common.c-150-\t\t\t\trcu_tasks_irq_resched_exit();\n--\nkernel/fork.c=1862=static inline void rcu_copy_process(struct task_struct *p)\n--\nkernel/fork.c-1869-#endif /* #ifdef CONFIG_PREEMPT_RCU */\nkernel/fork.c:1870:#ifdef CONFIG_TASKS_RCU\nkernel/fork.c-1871-\tp-\u003ercu_tasks_holdout = false;\n--\nkernel/fork.c-1875-\tINIT_LIST_HEAD(\u0026p-\u003ercu_tasks_exit_list);\nkernel/fork.c:1876:#endif /* #ifdef CONFIG_TASKS_RCU */\nkernel/fork.c-1877-#ifdef CONFIG_TASKS_TRACE_RCU\n--\nkernel/rcu/Kconfig=81=config NEED_SRCU_NMI_SAFE\n--\nkernel/rcu/Kconfig-83-\nkernel/rcu/Kconfig:84:config TASKS_RCU_GENERIC\nkernel/rcu/Kconfig:85:\tdef_bool TASKS_RCU || TASKS_RUDE_RCU\nkernel/rcu/Kconfig-86-\thelp\n--\nkernel/rcu/Kconfig-89-\nkernel/rcu/Kconfig:90:config FORCE_TASKS_RCU\nkernel/rcu/Kconfig:91:\tbool \"Force selection of TASKS_RCU\"\nkernel/rcu/Kconfig-92-\tdepends on RCU_EXPERT\nkernel/rcu/Kconfig:93:\tselect TASKS_RCU\nkernel/rcu/Kconfig-94-\tdefault n\n--\nkernel/rcu/Kconfig-102-\nkernel/rcu/Kconfig:103:config NEED_TASKS_RCU\nkernel/rcu/Kconfig-104-\tbool\n--\nkernel/rcu/Kconfig-106-\nkernel/rcu/Kconfig:107:config TASKS_RCU\nkernel/rcu/Kconfig-108-\tbool\nkernel/rcu/Kconfig:109:\tdefault NEED_TASKS_RCU \u0026\u0026 PREEMPTION\nkernel/rcu/Kconfig-110-\tselect IRQ_WORK\n--\nkernel/rcu/Kconfig=112=config HAVE_RCU_TRAMPOLINE_READERS\n--\nkernel/rcu/Kconfig-124-\nkernel/rcu/Kconfig:125:config TASKS_RCU_TRAMPOLINE_READERS\nkernel/rcu/Kconfig:126:\tdef_bool TASKS_RCU \u0026\u0026 HAVE_RCU_TRAMPOLINE_READERS \u0026\u0026 GENERIC_IRQ_ENTRY \u0026\u0026 !NEED_SRCU_NMI_SAFE\nkernel/rcu/Kconfig-127-\tselect TASKS_TRACE_RCU\n--\nkernel/rcu/Kconfig=199=config RCU_NEED_SEGCBLIST\nkernel/rcu/Kconfig:200:\tdef_bool ( TREE_RCU || TREE_SRCU || TASKS_RCU_GENERIC )\nkernel/rcu/Kconfig-201-\n--\nkernel/rcu/rcu.h=397=extern void rcu_init_geometry(void);\n--\nkernel/rcu/rcu.h-452-\nkernel/rcu/rcu.h:453:#if !defined(CONFIG_TINY_RCU) || defined(CONFIG_TASKS_RCU_GENERIC)\nkernel/rcu/rcu.h-454-\n--\nkernel/rcu/rcu.h=500=do {\t\t\t\t\t\t\t\t\t\\\n--\nkernel/rcu/rcu.h-525-\nkernel/rcu/rcu.h:526:#endif // #if !defined(CONFIG_TINY_RCU) || defined(CONFIG_TASKS_RCU_GENERIC)\nkernel/rcu/rcu.h-527-\n--\nkernel/rcu/rcu.h=547=bool rcu_cpu_online(int cpu);\nkernel/rcu/rcu.h:548:#ifdef CONFIG_TASKS_RCU_GENERIC\nkernel/rcu/rcu.h-549-void show_rcu_tasks_gp_kthreads(void);\nkernel/rcu/rcu.h:550:#else /* #ifdef CONFIG_TASKS_RCU_GENERIC */\nkernel/rcu/rcu.h-551-static inline void show_rcu_tasks_gp_kthreads(void) {}\nkernel/rcu/rcu.h:552:#endif /* #else #ifdef CONFIG_TASKS_RCU_GENERIC */\nkernel/rcu/rcu.h-553-#endif /* #else #ifdef CONFIG_TINY_RCU */\nkernel/rcu/rcu.h-554-\nkernel/rcu/rcu.h:555:#ifdef CONFIG_TASKS_RCU\nkernel/rcu/rcu.h-556-struct task_struct *get_rcu_tasks_gp_kthread(void);\nkernel/rcu/rcu.h=557=void rcu_tasks_get_gp_data(int *flags, unsigned long *gp_seq);\nkernel/rcu/rcu.h:558:#endif // # ifdef CONFIG_TASKS_RCU\nkernel/rcu/rcu.h-559-\n--\nkernel/rcu/rcu.h=562=void rcu_tasks_rude_get_gp_data(int *flags, unsigned long *gp_seq);\n--\nkernel/rcu/rcu.h-564-\nkernel/rcu/rcu.h:565:#ifdef CONFIG_TASKS_RCU_GENERIC\nkernel/rcu/rcu.h-566-void tasks_cblist_init_generic(void);\nkernel/rcu/rcu.h:567:#else /* #ifdef CONFIG_TASKS_RCU_GENERIC */\nkernel/rcu/rcu.h-568-static inline void tasks_cblist_init_generic(void) { }\nkernel/rcu/rcu.h:569:#endif /* #else #ifdef CONFIG_TASKS_RCU_GENERIC */\nkernel/rcu/rcu.h-570-\n--\nkernel/rcu/rcu.h=677=static inline void rcu_bind_current_to_nocb(void) { }\n--\nkernel/rcu/rcu.h-679-\nkernel/rcu/rcu.h:680:#if !defined(CONFIG_TINY_RCU) \u0026\u0026 defined(CONFIG_TASKS_RCU)\nkernel/rcu/rcu.h-681-void show_rcu_tasks_classic_gp_kthread(void);\n--\nkernel/rcu/rcuscale.c=279=static struct rcu_scale_ops srcud_ops = {\n--\nkernel/rcu/rcuscale.c-295-\nkernel/rcu/rcuscale.c:296:#ifdef CONFIG_TASKS_RCU\nkernel/rcu/rcuscale.c-297-\n--\nkernel/rcu/rcuscale.c=316=static struct rcu_scale_ops tasks_ops = {\n--\nkernel/rcu/rcuscale.c-333-\nkernel/rcu/rcuscale.c:334:#else // #ifdef CONFIG_TASKS_RCU\nkernel/rcu/rcuscale.c-335-\n--\nkernel/rcu/rcuscale.c-337-\nkernel/rcu/rcuscale.c:338:#endif // #else // #ifdef CONFIG_TASKS_RCU\nkernel/rcu/rcuscale.c-339-\n--\nkernel/rcu/rcutorture.c=1117=static struct rcu_torture_ops trivial_preempt_ops = {\n--\nkernel/rcu/rcutorture.c-1138-\nkernel/rcu/rcutorture.c:1139:#ifdef CONFIG_TASKS_RCU\nkernel/rcu/rcutorture.c-1140-\n--\nkernel/rcu/rcutorture.c-1146- * A classic Tasks RCU reader is any stretch of kernel code that does not\nkernel/rcu/rcutorture.c:1147: * voluntarily block. With CONFIG_TASKS_RCU_TRAMPOLINE_READERS a preemption\nkernel/rcu/rcutorture.c-1148- * outside trampoline text also ends it, and what a trampoline does to stay\n--\nkernel/rcu/rcutorture.c=1151=static int tasks_torture_read_lock(void)\nkernel/rcu/rcutorture.c-1152-{\nkernel/rcu/rcutorture.c:1153:\tif (IS_ENABLED(CONFIG_TASKS_RCU_TRAMPOLINE_READERS))\nkernel/rcu/rcutorture.c-1154-\t\trcu_read_lock_trace();\n--\nkernel/rcu/rcutorture.c=1158=static void tasks_torture_read_unlock(int idx)\nkernel/rcu/rcutorture.c-1159-{\nkernel/rcu/rcutorture.c:1160:\tif (IS_ENABLED(CONFIG_TASKS_RCU_TRAMPOLINE_READERS))\nkernel/rcu/rcutorture.c-1161-\t\trcu_read_unlock_trace();\n--\nkernel/rcu/rcutorture.c=1174=static struct rcu_torture_ops tasks_ops = {\n--\nkernel/rcu/rcutorture.c-1194-\nkernel/rcu/rcutorture.c:1195:#else // #ifdef CONFIG_TASKS_RCU\nkernel/rcu/rcutorture.c-1196-\n--\nkernel/rcu/rcutorture.c-1198-\nkernel/rcu/rcutorture.c:1199:#endif // #else #ifdef CONFIG_TASKS_RCU\nkernel/rcu/rcutorture.c-1200-\n--\nkernel/rcu/refscale.c=277=static const struct ref_scale_ops srcu_fast_updown_ops = {\n--\nkernel/rcu/refscale.c-283-\nkernel/rcu/refscale.c:284:#ifdef CONFIG_TASKS_RCU\nkernel/rcu/refscale.c-285-\n--\nkernel/rcu/refscale.c=304=static const struct ref_scale_ops rcu_tasks_ops = {\n--\nkernel/rcu/refscale.c-312-\nkernel/rcu/refscale.c:313:#else // #ifdef CONFIG_TASKS_RCU\nkernel/rcu/refscale.c-314-\n--\nkernel/rcu/refscale.c-316-\nkernel/rcu/refscale.c:317:#endif // #else // #ifdef CONFIG_TASKS_RCU\nkernel/rcu/refscale.c-318-\n--\nkernel/rcu/tasks.h-7-\nkernel/rcu/tasks.h:8:#ifdef CONFIG_TASKS_RCU_GENERIC\nkernel/rcu/tasks.h-9-#include \"rcu_segcblist.h\"\n--\nkernel/rcu/tasks.h=136=static struct rcu_tasks rt_name =\t\t\t\t\t\t\t\\\n--\nkernel/rcu/tasks.h-154-\nkernel/rcu/tasks.h:155:#if defined(CONFIG_TASKS_RCU) \u0026\u0026 !defined(CONFIG_TASKS_RCU_TRAMPOLINE_READERS)\nkernel/rcu/tasks.h-156-\n--\nkernel/rcu/tasks.h=695=static void __init rcu_tasks_bootup_oddness(void)\nkernel/rcu/tasks.h-696-{\nkernel/rcu/tasks.h:697:#if defined(CONFIG_TASKS_RCU) || defined(CONFIG_TASKS_TRACE_RCU)\nkernel/rcu/tasks.h-698-\tint rtsimc;\n--\nkernel/rcu/tasks.h-706-\t}\nkernel/rcu/tasks.h:707:#endif /* #ifdef CONFIG_TASKS_RCU */\nkernel/rcu/tasks.h:708:#ifdef CONFIG_TASKS_RCU\nkernel/rcu/tasks.h-709-\tpr_info(\"\\tTrampoline variant of Tasks RCU enabled.\\n\");\nkernel/rcu/tasks.h:710:#endif /* #ifdef CONFIG_TASKS_RCU */\nkernel/rcu/tasks.h-711-#ifdef CONFIG_TASKS_RUDE_RCU\n--\nkernel/rcu/tasks.h=758=static void rcu_tasks_torture_stats_print_generic(struct rcu_tasks *rtp, char *tt,\n--\nkernel/rcu/tasks.h-804-\nkernel/rcu/tasks.h:805:#if defined(CONFIG_TASKS_RCU) \u0026\u0026 !defined(CONFIG_TASKS_RCU_TRAMPOLINE_READERS)\nkernel/rcu/tasks.h-806-\n--\nkernel/rcu/tasks.h=812=static void rcu_tasks_wait_gp(struct rcu_tasks *rtp)\n--\nkernel/rcu/tasks.h-899-\nkernel/rcu/tasks.h:900:#endif /* #if defined(CONFIG_TASKS_RCU) \u0026\u0026 !defined(CONFIG_TASKS_RCU_TRAMPOLINE_READERS) */\nkernel/rcu/tasks.h-901-\nkernel/rcu/tasks.h:902:#ifdef CONFIG_TASKS_RCU\nkernel/rcu/tasks.h-903-\n--\nkernel/rcu/tasks.h=905=module_param(rcu_tasks_lazy_ms, int, 0444);\nkernel/rcu/tasks.h-906-\nkernel/rcu/tasks.h:907:#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS\nkernel/rcu/tasks.h-908-\n--\nkernel/rcu/tasks.h=1351=void exit_tasks_rcu_finish(void) { }\nkernel/rcu/tasks.h-1352-\nkernel/rcu/tasks.h:1353:#else /* #ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS */\nkernel/rcu/tasks.h-1354-\n--\nkernel/rcu/tasks.h=1612=static void tasks_rcu_exit_stall(struct timer_list *unused)\n--\nkernel/rcu/tasks.h-1626-\nkernel/rcu/tasks.h:1627:#endif /* #else #ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS */\nkernel/rcu/tasks.h-1628-\n--\nkernel/rcu/tasks.h-1642- * through a safe state, not so much for data-structure synchronization.\nkernel/rcu/tasks.h:1643: * On CONFIG_TASKS_RCU_TRAMPOLINE_READERS kernels a preemption outside\nkernel/rcu/tasks.h-1644- * trampoline text also ends one, and a reader whose protected window\n--\nkernel/rcu/tasks.h=1657=EXPORT_SYMBOL_GPL(call_rcu_tasks);\n--\nkernel/rcu/tasks.h-1667- * to synchronize_rcu_tasks(), and (in theory, anyway) cond_resched();\nkernel/rcu/tasks.h:1668: * on CONFIG_TASKS_RCU_TRAMPOLINE_READERS kernels also by preemption\nkernel/rcu/tasks.h-1669- * outside trampoline text, see call_rcu_tasks().\n--\nkernel/rcu/tasks.h=1695=EXPORT_SYMBOL_GPL(rcu_barrier_tasks);\nkernel/rcu/tasks.h-1696-\nkernel/rcu/tasks.h:1697:#ifndef CONFIG_TASKS_RCU_TRAMPOLINE_READERS\nkernel/rcu/tasks.h-1698-static int __init rcu_spawn_tasks_kthread(void)\n--\nkernel/rcu/tasks.h-1712-}\nkernel/rcu/tasks.h:1713:#endif /* #ifndef CONFIG_TASKS_RCU_TRAMPOLINE_READERS */\nkernel/rcu/tasks.h-1714-\n--\nkernel/rcu/tasks.h=1740=EXPORT_SYMBOL_GPL(rcu_tasks_get_gp_data);\nkernel/rcu/tasks.h-1741-\nkernel/rcu/tasks.h:1742:#ifndef CONFIG_TASKS_RCU_TRAMPOLINE_READERS\nkernel/rcu/tasks.h-1743-/*\n--\nkernel/rcu/tasks.h=1774=void exit_tasks_rcu_finish(void)\n--\nkernel/rcu/tasks.h-1785-}\nkernel/rcu/tasks.h:1786:#endif /* #ifndef CONFIG_TASKS_RCU_TRAMPOLINE_READERS */\nkernel/rcu/tasks.h-1787-\nkernel/rcu/tasks.h:1788:#else /* #ifdef CONFIG_TASKS_RCU */\nkernel/rcu/tasks.h-1789-void exit_tasks_rcu_start(void) { }\nkernel/rcu/tasks.h=1790=void exit_tasks_rcu_finish(void) { }\nkernel/rcu/tasks.h:1791:#endif /* #else #ifdef CONFIG_TASKS_RCU */\nkernel/rcu/tasks.h-1792-\n--\nkernel/rcu/tasks.h=1928=static struct rcu_tasks_test_desc tests[] = {\n--\nkernel/rcu/tasks.h-1931-\t\t/* If not defined, the test is skipped. */\nkernel/rcu/tasks.h:1932:\t\t.notrun = IS_ENABLED(CONFIG_TASKS_RCU),\nkernel/rcu/tasks.h-1933-\t\t/* Dump rcu tasks status, if test failed. */\n--\nkernel/rcu/tasks.h-1942-\nkernel/rcu/tasks.h:1943:#if defined(CONFIG_TASKS_RCU) || defined(CONFIG_TASKS_TRACE_RCU)\nkernel/rcu/tasks.h-1944-static void test_rcu_tasks_callback(struct rcu_head *rhp)\n--\nkernel/rcu/tasks.h-1952-}\nkernel/rcu/tasks.h:1953:#endif // #if defined(CONFIG_TASKS_RCU) || defined(CONFIG_TASKS_TRACE_RCU)\nkernel/rcu/tasks.h-1954-\nkernel/rcu/tasks.h=1955=static void rcu_tasks_initiate_self_tests(void)\nkernel/rcu/tasks.h-1956-{\nkernel/rcu/tasks.h:1957:#ifdef CONFIG_TASKS_RCU\nkernel/rcu/tasks.h-1958-\tpr_info(\"Running RCU Tasks wait API self tests\\n\");\n--\nkernel/rcu/tasks.h=2035=void __init tasks_cblist_init_generic(void)\n--\nkernel/rcu/tasks.h-2039-\nkernel/rcu/tasks.h:2040:#ifdef CONFIG_TASKS_RCU\nkernel/rcu/tasks.h-2041-\tcblist_init_generic(\u0026rcu_tasks);\n--\nkernel/rcu/tasks.h=2049=static int __init rcu_init_tasks_generic(void)\nkernel/rcu/tasks.h-2050-{\nkernel/rcu/tasks.h:2051:#ifdef CONFIG_TASKS_RCU\nkernel/rcu/tasks.h-2052-\trcu_spawn_tasks_kthread();\n--\nkernel/rcu/tasks.h=2064=core_initcall(rcu_init_tasks_generic);\nkernel/rcu/tasks.h-2065-\nkernel/rcu/tasks.h:2066:#else /* #ifdef CONFIG_TASKS_RCU_GENERIC */\nkernel/rcu/tasks.h-2067-static inline void rcu_tasks_bootup_oddness(void) {}\nkernel/rcu/tasks.h:2068:#endif /* #else #ifdef CONFIG_TASKS_RCU_GENERIC */\nkernel/rcu/tasks.h-2069-\n--\nkernel/trace/Kconfig=179=config TRACING\n--\nkernel/trace/Kconfig-187-\tselect TRACE_CLOCK\nkernel/trace/Kconfig:188:\tselect NEED_TASKS_RCU\nkernel/trace/Kconfig-189-\n--\nkernel/trace/Kconfig=235=config FUNCTION_TRACER\n--\nkernel/trace/Kconfig-241-\tselect GLOB\nkernel/trace/Kconfig:242:\tselect NEED_TASKS_RCU\nkernel/trace/Kconfig-243-\tselect TASKS_RUDE_RCU\n--\nkernel/trace/ftrace.c=6090=static void ftrace_direct_mark_module(unsigned long addr)\nkernel/trace/ftrace.c-6091-{\nkernel/trace/ftrace.c:6092:#if defined(CONFIG_MODULES) \u0026\u0026 defined(CONFIG_TASKS_RCU_TRAMPOLINE_READERS)\nkernel/trace/ftrace.c-6093-\tstruct module *mod;\n"
} |
| 9/2 | 2026/09/24 04:19 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 51221 output: 31 thoughts: 210 Thoughts: **Analyzing RCU Task Selection** Okay, let's break this down. My initial observation is focused on how `NEED_TASKS_RCU` gets triggered. I see it's explicitly tied to three key areas: `BPF_SYSCALL`, `TRACING`, and `FUNCTION_TRACER`. That's a solid starting point; these are all areas where RCU would be critical, given their respective concurrency needs. Then, there's `TASKS_RCU` itself. It seems to be the core, set either when `NEED_TASKS_RCU` is chosen in the default configuration - which requires `PREEMPTION` - or, alternatively, it is "forced" on via `FORCE_TASKS_RCU`. This suggests a preemptive kernel context necessitates the use of TASKS_RCU. That is, if preemption is active, then TASKS_RCU must be active, or forced to be so. Now, let's consider `HAVE_RCU_TRAMPOLINE_READERS`. The Kconfig reveals that it's selected under a specific x86 architecture scenario: `X86_64 && SMP && DYNAMIC_FTRACE`. This makes sense; the trampoline readers are probably useful for the increased complexity of dynamic tracing on multi-core systems. Finally, `TASKS_RCU_TRAMPOLINE_READERS` which is linked to all the previous configurations, along with: `GENERIC_IRQ_ENTRY` and negated `NEED_SRCU_NMI_SAFE`. This suggests that the trampoline readers are enabled when the kernel has a generic IRQ entry mechanism and it *doesn't* require SRCU for NMI-safe contexts. To understand the full picture, I need to check `GENERIC_IRQ_ENTRY` and `NEED_SRCU_NMI_SAFE` in the x86 architecture specifically. That's the next logical step to fully understand how these configurations come together. |
| 10/2 | 2026/09/24 04:19 | tool | grepper |
0mError:no matches Args: {
"Expression": "GENERIC_IRQ_ENTRY",
"PathPrefix": "arch/x86"
}Results: {
"error": "no matches"
} |
| 11/2 | 2026/09/24 04:19 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 51901 output: 20 thoughts: 26 Thoughts: Okay, here's my summary, tailored for an expert audience, in the first person, reflecting the specified thought process: **Diving into the GENERIC_IRQ_ENTRY Configuration** Alright, let's get down to brass tacks. I need to figure out the Kconfig dependency for `GENERIC_IRQ_ENTRY`. Where exactly does that define get pulled into the kernel configuration? I'm thinking I should start by looking for its selection in the main `Kconfig` files and, given the nature of interrupt handling, I suspect the `arch/x86/Kconfig` might be a likely candidate to find its definition. I need to understand the architectural constraints or conditions that enable or disable it. It's crucial for me to pinpoint the exact configuration option that triggers `GENERIC_IRQ_ENTRY` for the specific architecture I'm working with. This information is key for debugging or for any modifications I want to perform on the IRQ handling. Let's start the hunt... |
| 12/2 | 2026/09/24 04:19 | tool | grepper |
0mArgs:{
"Expression": "GENERIC_IRQ_ENTRY"
}Results: {
"Output": "Documentation/core-api/real-time/architecture-porting.rst=91=Lazy preemption\n--\nDocumentation/core-api/real-time/architecture-porting.rst-95- request.\nDocumentation/core-api/real-time/architecture-porting.rst:96: With CONFIG_GENERIC_IRQ_ENTRY enabled, supporting this feature requires\nDocumentation/core-api/real-time/architecture-porting.rst-97- defining a bit for TIF_NEED_RESCHED_LAZY, preferably near TIF_NEED_RESCHED.\n--\narch/Kconfig=101=config HOTPLUG_PARALLEL\n--\narch/Kconfig-104-\narch/Kconfig:105:config GENERIC_IRQ_ENTRY\narch/Kconfig-106-\tbool\n--\narch/Kconfig=108=config GENERIC_SYSCALL\narch/Kconfig-109-\tbool\narch/Kconfig:110:\tdepends on GENERIC_IRQ_ENTRY\narch/Kconfig-111-\narch/Kconfig=112=config GENERIC_ENTRY\narch/Kconfig-113-\tbool\narch/Kconfig:114:\tselect GENERIC_IRQ_ENTRY\narch/Kconfig-115-\tselect GENERIC_SYSCALL\n--\narch/arm64/Kconfig=2=config ARM64\n--\narch/arm64/Kconfig-133-\tselect GENERIC_IOREMAP\narch/arm64/Kconfig:134:\tselect GENERIC_IRQ_ENTRY\narch/arm64/Kconfig-135-\tselect GENERIC_IRQ_IPI\n--\ninclude/linux/rseq.h=12=static __always_inline bool rseq_v2(struct task_struct *t)\ninclude/linux/rseq.h-13-{\ninclude/linux/rseq.h:14:\treturn IS_ENABLED(CONFIG_GENERIC_IRQ_ENTRY) \u0026\u0026 likely(t-\u003erseq.event.has_rseq \u003e 1);\ninclude/linux/rseq.h-15-}\n--\ninclude/linux/rseq_entry.h=253=static __always_inline void rseq_note_user_irq_entry(void)\ninclude/linux/rseq_entry.h-254-{\ninclude/linux/rseq_entry.h:255:\tif (IS_ENABLED(CONFIG_GENERIC_IRQ_ENTRY))\ninclude/linux/rseq_entry.h-256-\t\tcurrent-\u003erseq.event.user_irq = true;\n--\ninclude/linux/rseq_entry.h=292=bool rseq_debug_update_user_cs(struct task_struct *t, struct pt_regs *regs,\n--\ninclude/linux/rseq_entry.h-345-\ninclude/linux/rseq_entry.h:346:\t\t/* rseq_event.user_irq is only valid if CONFIG_GENERIC_IRQ_ENTRY=y */\ninclude/linux/rseq_entry.h:347:\t\tif (IS_ENABLED(CONFIG_GENERIC_IRQ_ENTRY)) {\ninclude/linux/rseq_entry.h-348-\t\t\t/* If not in interrupt from user context, let it die */\n--\ninclude/linux/rseq_entry.h=532=static rseq_inline bool rseq_update_usr(struct task_struct *t, struct pt_regs *regs,\n--\ninclude/linux/rseq_entry.h-544-\t */\ninclude/linux/rseq_entry.h:545:\tif (IS_ENABLED(CONFIG_GENERIC_IRQ_ENTRY)) {\ninclude/linux/rseq_entry.h-546-\t\tif (!static_branch_unlikely(\u0026rseq_debug_enabled)) {\n--\ninit/Kconfig=1904=config FUTEX_ROBUST_UNLOCK\ninit/Kconfig:1905:\tdef_bool FUTEX \u0026\u0026 GENERIC_IRQ_ENTRY \u0026\u0026 RSEQ \u0026\u0026 HAVE_FUTEX_ROBUST_UNLOCK\ninit/Kconfig-1906-\n--\nkernel/entry/Makefile=13=CFLAGS_common.o\t\t+= -fno-stack-protector\nkernel/entry/Makefile-14-\nkernel/entry/Makefile:15:obj-$(CONFIG_GENERIC_IRQ_ENTRY) \t+= common.o\nkernel/entry/Makefile-16-obj-$(CONFIG_GENERIC_SYSCALL) \t\t+= syscall-common.o\n--\nkernel/rcu/Kconfig=125=config TASKS_RCU_TRAMPOLINE_READERS\nkernel/rcu/Kconfig:126:\tdef_bool TASKS_RCU \u0026\u0026 HAVE_RCU_TRAMPOLINE_READERS \u0026\u0026 GENERIC_IRQ_ENTRY \u0026\u0026 !NEED_SRCU_NMI_SAFE\nkernel/rcu/Kconfig-127-\tselect TASKS_TRACE_RCU\n--\nkernel/rseq.c=415=static long rseq_register(struct rseq __user * rseq, u32 rseq_len, int flags, u32 sig)\n--\nkernel/rseq.c-428-\t */\nkernel/rseq.c:429:\tif (IS_ENABLED(CONFIG_GENERIC_IRQ_ENTRY) \u0026\u0026 rseq_len \u003e ORIG_RSEQ_SIZE)\nkernel/rseq.c-430-\t\tversion = 2;\n--\nkernel/sched/core.c-72-#ifdef CONFIG_PREEMPT_DYNAMIC\nkernel/sched/core.c:73:# ifdef CONFIG_GENERIC_IRQ_ENTRY\nkernel/sched/core.c-74-# include \u003clinux/irq-entry-common.h\u003e\n--\nkernel/sched/core.c=7914=EXPORT_SYMBOL(__cond_resched_rwlock_write);\n--\nkernel/sched/core.c-7917-\nkernel/sched/core.c:7918:# ifdef CONFIG_GENERIC_IRQ_ENTRY\nkernel/sched/core.c-7919-# include \u003clinux/irq-entry-common.h\u003e\n"
} |
| 13/2 | 2026/09/24 04:19 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 53654 output: 29 thoughts: 33 Thoughts: Here's my take on the thought process, as requested: **Digging into the Interrupt Handling Mechanism** Okay, so `GENERIC_ENTRY` is calling `GENERIC_IRQ_ENTRY`. That's a pretty standard approach, but let's be thorough. I need to understand what's *really* going on under the hood here. My gut feeling is that it's related to the x86 architecture, so let's check `arch/x86/Kconfig`. That's where the platform-specific configurations are usually defined. I'm expecting to find some crucial details about how the generic interrupt entry points are actually implemented for x86. I'm looking for the conditions that enable or disable these specific entry points and what they represent in terms of the broader interrupt handling flow. |
| 14/2 | 2026/09/24 04:19 | tool | grepper |
0mArgs:{
"Expression": "GENERIC_ENTRY",
"PathPrefix": "arch/x86"
}Results: {
"Output": "arch/x86/Kconfig=58=config X86\n--\narch/x86/Kconfig-174-\tselect GENERIC_EARLY_IOREMAP\narch/x86/Kconfig:175:\tselect GENERIC_ENTRY\narch/x86/Kconfig-176-\tselect GENERIC_IOMAP\n"
} |
| 15/2 | 2026/09/24 04:19 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 53986 output: 23 thoughts: 49 Thoughts: Okay, here's my take on those thoughts, tailored for an expert audience: **Investigating the x86 Interrupt Entry Point Configuration** Alright, so `arch/x86` is pulling in `GENERIC_ENTRY`. No surprises there, that's expected. And that, in turn, is selecting `GENERIC_IRQ_ENTRY`. Seems standard enough for the interrupt handling architecture. But, the real question is, what's the deal with `NEED_SRCU_NMI_SAFE`? I need to track down the dependency chain and figure out why that's being pulled in. I need to understand its context within the x86 interrupt handling and the implications it has on SRCU. This is likely related to NMI safety, but I want to verify its origin and specific purpose. I'll need to dig into the relevant Kconfig files and code to see what's driving this selection. Let's see if there are any obvious clues in the configuration options or documentation. |
| 16/2 | 2026/09/24 04:19 | tool | grepper |
0mArgs:{
"Expression": "NEED_SRCU_NMI_SAFE"
}Results: {
"Output": "arch/arm64/kernel/ftrace.c=21=extern void ftrace_static_tramp_end(void);\n--\narch/arm64/kernel/ftrace.c-23-/* The SRCU-fast increments in entry-ftrace.S are the this_cpu_inc() form. */\narch/arm64/kernel/ftrace.c:24:static_assert(!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE));\narch/arm64/kernel/ftrace.c-25-\n--\narch/arm64/net/bpf_jit_comp.c=2604=static void emit_trace_rcu_reader(struct jit_ctx *ctx, bool lock)\n--\narch/arm64/net/bpf_jit_comp.c-2615-\t/* The plain this_cpu_inc() form of __srcu_read_lock_fast(). */\narch/arm64/net/bpf_jit_comp.c:2616:\tBUILD_BUG_ON(IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE));\narch/arm64/net/bpf_jit_comp.c-2617-\t/* LDR/STR (immediate, unsigned offset) ranges */\n--\narch/x86/kernel/ftrace.c=281=extern char __rethunk_safe_start[], __rethunk_safe_end[];\n--\narch/x86/kernel/ftrace.c-286- */\narch/x86/kernel/ftrace.c:287:static_assert(!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE));\narch/x86/kernel/ftrace.c-288-\n--\narch/x86/net/bpf_jit_comp.c=748=static void emit_trace_rcu_reader(u8 **pprog, bool lock)\n--\narch/x86/net/bpf_jit_comp.c-758-\t/* The plain this_cpu_inc() form of __srcu_read_lock_fast(). */\narch/x86/net/bpf_jit_comp.c:759:\tBUILD_BUG_ON(IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE));\narch/x86/net/bpf_jit_comp.c-760-\n--\ninclude/linux/srcu.h=139=static inline bool same_state_synchronize_srcu(unsigned long oldstate1, unsigned long oldstate2)\n--\ninclude/linux/srcu.h-143-\ninclude/linux/srcu.h:144:#ifdef CONFIG_NEED_SRCU_NMI_SAFE\ninclude/linux/srcu.h-145-int __srcu_read_lock_nmisafe(struct srcu_struct *ssp) __acquires_shared(ssp);\n--\ninclude/linux/srcu.h=153=static inline void __srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx)\n--\ninclude/linux/srcu.h-157-}\ninclude/linux/srcu.h:158:#endif /* CONFIG_NEED_SRCU_NMI_SAFE */\ninclude/linux/srcu.h-159-\n--\ninclude/linux/srcutree.h=289=static inline struct srcu_ctr __percpu notrace *__srcu_read_lock_fast(struct srcu_struct *ssp)\n--\ninclude/linux/srcutree.h-293-\ninclude/linux/srcutree.h:294:\tif (!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE))\ninclude/linux/srcutree.h-295-\t\tthis_cpu_inc(scp-\u003esrcu_locks.counter); // Y, and implicit RCU reader.\n--\ninclude/linux/srcutree.h=313=__srcu_read_unlock_fast(struct srcu_struct *ssp, struct srcu_ctr __percpu *scp)\n--\ninclude/linux/srcutree.h-317-\tbarrier(); /* Avoid leaking the critical section. */\ninclude/linux/srcutree.h:318:\tif (!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE))\ninclude/linux/srcutree.h-319-\t\tthis_cpu_inc(scp-\u003esrcu_unlocks.counter); // Z, and implicit RCU reader.\n--\ninclude/linux/srcutree.h=333=struct srcu_ctr __percpu notrace *__srcu_read_lock_fast_updown(struct srcu_struct *ssp)\n--\ninclude/linux/srcutree.h-337-\ninclude/linux/srcutree.h:338:\tif (!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE))\ninclude/linux/srcutree.h-339-\t\tthis_cpu_inc(scp-\u003esrcu_locks.counter); // Y, and implicit RCU reader.\n--\ninclude/linux/srcutree.h=357=__srcu_read_unlock_fast_updown(struct srcu_struct *ssp, struct srcu_ctr __percpu *scp)\n--\ninclude/linux/srcutree.h-361-\tbarrier(); /* Avoid leaking the critical section. */\ninclude/linux/srcutree.h:362:\tif (!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE))\ninclude/linux/srcutree.h-363-\t\tthis_cpu_inc(scp-\u003esrcu_unlocks.counter); // Z, and implicit RCU reader.\n--\nkernel/rcu/Kconfig=62=config TREE_SRCU\n--\nkernel/rcu/Kconfig-67-\nkernel/rcu/Kconfig:68:config FORCE_NEED_SRCU_NMI_SAFE\nkernel/rcu/Kconfig:69:\tbool \"Force selection of NEED_SRCU_NMI_SAFE\"\nkernel/rcu/Kconfig-70-\tdepends on !TINY_SRCU\n--\nkernel/rcu/Kconfig-72-\tdepends on ARCH_HAS_NMI_SAFE_THIS_CPU_OPS\nkernel/rcu/Kconfig:73:\tselect NEED_SRCU_NMI_SAFE\nkernel/rcu/Kconfig-74-\tdefault n\nkernel/rcu/Kconfig-75-\thelp\nkernel/rcu/Kconfig:76:\t This option forces selection of the NEED_SRCU_NMI_SAFE\nkernel/rcu/Kconfig-77-\t Kconfig option, allowing testing of srcu_read_lock_nmisafe()\n--\nkernel/rcu/Kconfig-80-\nkernel/rcu/Kconfig:81:config NEED_SRCU_NMI_SAFE\nkernel/rcu/Kconfig-82-\tdef_bool HAVE_NMI \u0026\u0026 !ARCH_HAS_NMI_SAFE_THIS_CPU_OPS \u0026\u0026 !TINY_SRCU\n--\nkernel/rcu/Kconfig=112=config HAVE_RCU_TRAMPOLINE_READERS\n--\nkernel/rcu/Kconfig-122-\t reported by arch_rcu_tasks_trampoline_text(). The assembly readers\nkernel/rcu/Kconfig:123:\t use the this_cpu_inc() form of SRCU-fast, hence !NEED_SRCU_NMI_SAFE.\nkernel/rcu/Kconfig-124-\nkernel/rcu/Kconfig=125=config TASKS_RCU_TRAMPOLINE_READERS\nkernel/rcu/Kconfig:126:\tdef_bool TASKS_RCU \u0026\u0026 HAVE_RCU_TRAMPOLINE_READERS \u0026\u0026 GENERIC_IRQ_ENTRY \u0026\u0026 !NEED_SRCU_NMI_SAFE\nkernel/rcu/Kconfig-127-\tselect TASKS_TRACE_RCU\n--\nkernel/rcu/srcutree.c=791=EXPORT_SYMBOL_GPL(__srcu_read_unlock);\nkernel/rcu/srcutree.c-792-\nkernel/rcu/srcutree.c:793:#ifdef CONFIG_NEED_SRCU_NMI_SAFE\nkernel/rcu/srcutree.c-794-\n--\nkernel/rcu/srcutree.c=821=EXPORT_SYMBOL_GPL(__srcu_read_unlock_nmisafe);\nkernel/rcu/srcutree.c-822-\nkernel/rcu/srcutree.c:823:#endif // CONFIG_NEED_SRCU_NMI_SAFE\nkernel/rcu/srcutree.c-824-\n--\ntools/testing/selftests/rcutorture/bin/srcu_lockdep.sh=46=do\n--\ntools/testing/selftests/rcutorture/bin/srcu_lockdep.sh-52-\t\t\tval=$((d*1000+t*10+c))\ntools/testing/selftests/rcutorture/bin/srcu_lockdep.sh:53:\t\t\ttools/testing/selftests/rcutorture/bin/kvm.sh --allcpus --duration 5s --configs \"SRCU-P\" --kconfig \"CONFIG_FORCE_NEED_SRCU_NMI_SAFE=y\" --bo...\ntools/testing/selftests/rcutorture/bin/srcu_lockdep.sh-54-\t\t\tret=$?\n--\ntools/testing/selftests/rcutorture/bin/srcu_lockdep.sh=84=do\ntools/testing/selftests/rcutorture/bin/srcu_lockdep.sh-85-\terr=\ntools/testing/selftests/rcutorture/bin/srcu_lockdep.sh:86:\ttools/testing/selftests/rcutorture/bin/kvm.sh --allcpus --duration 5s --configs \"SRCU-P\" --kconfig \"CONFIG_FORCE_NEED_SRCU_NMI_SAFE=y\" --boot...\ntools/testing/selftests/rcutorture/bin/srcu_lockdep.sh-87-\tret=$?\n"
} |
| 17/2 | 2026/09/24 04:19 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 56788 output: 45 thoughts: 2177 Thoughts: **Diving into Tasks RCU and Trampolines** Okay, let's break this down. On x86_64, because `ARCH_HAS_NMI_SAFE_THIS_CPU_OPS` is true, we don't need the NMI-safe version of `this_cpu_ops`, so `NEED_SRCU_NMI_SAFE` is false. This leads to certain configurations on amd64 (X86_64), specifically the interplay between `HAVE_RCU_TRAMPOLINE_READERS`, `TASKS_RCU_TRAMPOLINE_READERS`, and how they relate to SMP, dynamic ftrace, and various RCU and preemption-related settings. It looks like `HAVE_RCU_TRAMPOLINE_READERS` gets selected if we're on x86_64, have SMP and DYNAMIC_FTRACE. And `TASKS_RCU_TRAMPOLINE_READERS` is enabled based on `TASKS_RCU`, `HAVE_RCU_TRAMPOLINE_READERS`, `GENERIC_IRQ_ENTRY`, and the absence of `NEED_SRCU_NMI_SAFE`. `TASKS_RCU` itself depends on `NEED_TASKS_RCU`, `PREEMPTION` or `FORCE_TASKS_RCU` with `NEED_TASKS_RCU` being determined by `BPF_SYSCALL`, `TRACING`, and `FUNCTION_TRACER`. Now, the core of the patch is about how Tasks RCU handles architectures that support `HAVE_RCU_TRAMPOLINE_READERS`, like arm64 and x86_64. Previously, Tasks RCU required a wait for tasks to context switch or enter userspace or become idle. This patch introduces "trampolines" (ftrace stubs, kprobe optimizations, and BPF trampolines) that act as Tasks Trace RCU readers. They enter a Tasks Trace RCU reader (`rcu_read_lock_trace()`) before calling out and exit before returning. The key is that a task calling out from a trampoline is considered a normal Tasks Trace RCU reader. Inside the trampoline instructions, a task can only be interrupted there. The `raw_irqentry_exit_cond_resched` path now calls `rcu_tasks_irq_resched_enter` if preemption is happening. This function checks if the instruction pointer is inside the trampoline text using `rcu_tasks_trampoline_text`. If so, the task is marked as a holdout (`rcu_tasks_tramp_hold`). When preemption ends, `rcu_tasks_irq_resched_exit` is called. It also introduces `kprobe_in_optimized_region` which checks if an instruction pointer is overwritten by an optimized kprobe, and if it is, `rcu_tasks_wait_irq_preempted` is used to wait for these interrupted tasks. I need to look carefully at the code changes, specifically the newly introduced functions and the existing ones that are being modified. The new additions in `kernel/rcu/tasks.h` are extensive and include functions for trampoline waiting, tracking holdouts, interrupt handling, tracing, and CPU quiescent state. The core `arch/x86/kernel/ftrace.c` file is being expanded to identify trampoline text for x86. The `emit_trace_rcu_reader` functions in `arch/x86/net/bpf_jit_comp.c` for BPF JIT code and the templates in `arch/x86/kernel/kprobes/opt.c` are also crucial. In the general kernel, `kprobe_in_optimized_region` and `kprobe_optimizer` will also be affected. Also note modifications to the Ftrace functions. It looks like tracing functions (`rcu_tasks_trace_assert_idle`) and a number of functions related to interrupt entry and exit are also being modified. And in particular the functions `raw_irqentry_exit_cond_resched` and `dynamic_irqentry_exit_cond_resched` are key. Now, an exciting point: a new runtime assertion `rcu_tasks_trace_assert_idle()` is being added, called on every return to user space. This is a very interesting addition. It's using `WARN_ON_ONCE` to catch potential leaks of the `trc_reader_nesting` variable. If any trampoline or other path fails to properly manage `trc_reader_nesting`, a warning will be triggered. I need to make sure this is understood clearly! I also notice |
| 18/2 | 2026/09/24 04:19 | tool | read-file |
0mArgs:{
"File": "arch/x86/net/bpf_jit_comp.c",
"FirstLine": 3690,
"LineCount": 40
}Results: {
"Contents": "3690:\t\t} else {\n3691:\t\t\t/* sub rsp, stack_size */\n3692:\t\t\tEMIT4(0x48, 0x83, 0xEC, stack_size);\n3693:\t\t}\n3694:\t\tif (flags \u0026 BPF_TRAMP_F_TAIL_CALL_CTX)\n3695:\t\t\tEMIT1(0x50);\t\t/* push rax */\n3696:\t\t/* mov QWORD PTR [rbp - rbx_off], rbx */\n3697:\t\temit_stx(\u0026prog, BPF_DW, BPF_REG_FP, BPF_REG_6, -rbx_off);\n3698:\t\n3699:\t\t/*\n3700:\t\t * Tasks RCU keeps this image alive only while we are a Tasks Trace\n3701:\t\t * reader; the instructions before this point (and after the final\n3702:\t\t * unlock) are covered by the irq-exit IP check. One reader spans\n3703:\t\t * __bpf_tramp_enter() and the fentry/fmod_ret progs, a second one\n3704:\t\t * the fexit progs and __bpf_tramp_exit(); the original function runs\n3705:\t\t * outside both, with the image pinned by im-\u003epcref instead.\n3706:\t\t */\n3707:\t\temit_trace_rcu_reader(\u0026prog, true);\n3708:\t\n3709:\t\tfunc_meta = nr_regs;\n3710:\t\t/* Store number of argument registers of the traced function */\n3711:\t\temit_store_stack_imm64(\u0026prog, BPF_REG_0, -func_meta_off, func_meta);\n3712:\t\n3713:\t\tif (flags \u0026 BPF_TRAMP_F_IP_ARG) {\n3714:\t\t\t/* Store IP address of the traced function */\n3715:\t\t\temit_store_stack_imm64(\u0026prog, BPF_REG_0, -ip_off, (long)func_addr);\n3716:\t\t}\n3717:\t\n3718:\t\tsave_args(m, \u0026prog, regs_off, false, flags, arena_base);\n3719:\t\n3720:\t\tif (flags \u0026 BPF_TRAMP_F_CALL_ORIG) {\n3721:\t\t\t/* arg1: mov rdi, im */\n3722:\t\t\temit_mov_imm64(\u0026prog, BPF_REG_1, (long) im \u003e\u003e 32, (u32) (long) im);\n3723:\t\t\tif (emit_rsb_call(\u0026prog, __bpf_tramp_enter,\n3724:\t\t\t\t\t image + (prog - (u8 *)rw_image))) {\n3725:\t\t\t\tret = -EINVAL;\n3726:\t\t\t\tgoto cleanup;\n3727:\t\t\t}\n3728:\t\t}\n3729:\t\n"
} |
| 19/2 | 2026/09/24 04:19 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 60809 output: 34 thoughts: 59 Thoughts: Okay, let's dive into this. I need to understand how these BPF trampolines are set up on x86, so I'm thinking, "Right, `arch_prepare_bpf_trampoline()` is the function to trace here. I vaguely recall that function and know it's probably where the heavy lifting occurs." Now, I need to see what's actually happening so I need to find the definition. So, I need to know the next function called. Ah, `arch_prepare_bpf_trampoline()` calls `__arch_prepare_bpf_trampoline()`. That's logical, a public interface calling an internal implementation. **Dissecting BPF Trampoline Preparation on x86** Okay, so I have a mental image: `arch_prepare_bpf_trampoline()` is the entry point, and then we drop into `__arch_prepare_bpf_trampoline()`. Now, I really need to get in there and see how things are set up at this lower level. Where is this function defined? Let's get to work! I'm going to run a `grep` for `arch_prepare_bpf_trampoline` and see where this function is implemented in the source code. |
| 20/2 | 2026/09/24 04:19 | tool | grepper |
0mArgs:{
"Expression": "arch_prepare_bpf_trampoline",
"PathPrefix": "arch/x86"
}Results: {
"Output": "arch/x86/net/bpf_jit_comp.c=3305=static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog,\n--\narch/x86/net/bpf_jit_comp.c-3321-\t * bpf_tramp_run_ctx is already preserved by\narch/x86/net/bpf_jit_comp.c:3322:\t * arch_prepare_bpf_trampoline().\narch/x86/net/bpf_jit_comp.c-3323-\t *\n--\narch/x86/net/bpf_jit_comp.c=3443=static int invoke_bpf_mod_ret(const struct btf_func_model *m, u8 **pprog,\n--\narch/x86/net/bpf_jit_comp.c-3544- */\narch/x86/net/bpf_jit_comp.c:3545:static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_image,\narch/x86/net/bpf_jit_comp.c-3546-\t\t\t\t\t void *rw_image_end, void *image,\n--\narch/x86/net/bpf_jit_comp.c=3884=int arch_protect_bpf_trampoline(void *image, unsigned int size)\n--\narch/x86/net/bpf_jit_comp.c-3888-\narch/x86/net/bpf_jit_comp.c:3889:int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *image_end,\narch/x86/net/bpf_jit_comp.c-3890-\t\t\t\tconst struct btf_func_model *m, u32 flags,\n--\narch/x86/net/bpf_jit_comp.c-3904-\narch/x86/net/bpf_jit_comp.c:3905:\tret = __arch_prepare_bpf_trampoline(im, rw_image, rw_image + size, image, m,\narch/x86/net/bpf_jit_comp.c-3906-\t\t\t\t\t flags, tnodes, func_addr);\n--\narch/x86/net/bpf_jit_comp.c=3918=int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,\n--\narch/x86/net/bpf_jit_comp.c-3924-\narch/x86/net/bpf_jit_comp.c:3925:\t/* Allocate a temporary buffer for __arch_prepare_bpf_trampoline().\narch/x86/net/bpf_jit_comp.c-3926-\t *\n--\narch/x86/net/bpf_jit_comp.c-3935-\narch/x86/net/bpf_jit_comp.c:3936:\tret = __arch_prepare_bpf_trampoline(\u0026im, image, image + PAGE_SIZE, image,\narch/x86/net/bpf_jit_comp.c-3937-\t\t\t\t\t m, flags, tnodes, func_addr);\n"
} |
| 21/2 | 2026/09/24 04:19 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 62030 output: 36 thoughts: 99 Thoughts: Here's my thinking process: **Diving into BPF Trampolines and RCU Grace Periods** Okay, so I'm dealing with BPF trampoline setup – that `arch_prepare_bpf_trampoline()` function is the key starting point when attaching those fentry, fexit, fmod_ret, struct_ops, and other similar programs. I need to understand how this is handled, the low-level details are going to be critical to performance. Now, my focus shifts to the RCU grace period logic. I remember some recent changes, so I should examine `kernel/rcu/tasks.h` to refresh my knowledge of the new implementation details. Specifically, I'll hone in on lines 910-1050; I'm betting that's where the relevant logic is centered. The RCU integration and how it interacts with the BPF trampolines is what I need to fully understand. |
| 22/2 | 2026/09/24 04:19 | tool | read-file |
0mArgs:{
"File": "kernel/rcu/tasks.h",
"FirstLine": 950,
"LineCount": 80
}Results: {
"Contents": " 950:\t// which is bounded by a few jiffies plus preempt-off latency plus an SRCU\n 951:\t// grace period, independent of how long any task runs without sleeping.\n 952:\t// Unlike the classic implementation this does wait for an idle task caught\n 953:\t// in a trampoline, since an idle CPU only counts while RCU is not watching\n 954:\t// it.\n 955:\t\n 956:\tstatic void rcu_tasks_tramp_wait_gp(struct rcu_tasks *rtp);\n 957:\tvoid call_rcu_tasks(struct rcu_head *rhp, rcu_callback_t func);\n 958:\tDEFINE_RCU_TASKS(rcu_tasks, rcu_tasks_tramp_wait_gp, call_rcu_tasks, \"RCU Tasks\");\n 959:\t\n 960:\t/* Per-CPU count of Tasks RCU quiescent events, and the GP kthread's snapshot. */\n 961:\tstatic DEFINE_PER_CPU(unsigned long, rcu_tasks_qs_seq);\n 962:\tstatic DEFINE_PER_CPU(unsigned long, rcu_tasks_qs_snap);\n 963:\t\n 964:\t/*\n 965:\t * Tasks currently switched out by an irq-exit preemption are kept, with the\n 966:\t * interrupted IP, on the per-CPU rtp_exit_list of the CPU that preempted them\n 967:\t * (reusing the list, lock and task_struct fields the classic flavor uses for\n 968:\t * its exit-path bookkeeping, which this flavor does not need), so that\n 969:\t * rcu_tasks_wait_irq_preempted() can find them without a tasklist scan and\n 970:\t * regardless of where they are in exit.\n 971:\t */\n 972:\t\n 973:\t/* Tasks last seen preempted inside unmarked trampoline text. */\n 974:\tstatic LIST_HEAD(rcu_tasks_tramp_holdouts);\n 975:\tstatic DEFINE_RAW_SPINLOCK(rcu_tasks_tramp_lock);\n 976:\t\n 977:\t/* CPUs / holdouts the current grace period is still waiting for. */\n 978:\tstatic struct cpumask rcu_tasks_pending_cpus;\n 979:\tstatic LIST_HEAD(rcu_tasks_gp_holdouts);\n 980:\t\n 981:\t/**\n 982:\t * arch_rcu_tasks_trampoline_text - Does the architecture treat @ip as unmarked trampoline text?\n 983:\t * @ip: kernel text address inside core kernel text\n 984:\t *\n 985:\t * See rcu_tasks_trampoline_text(). Architectures override this to flag\n 986:\t * core text that runs on behalf of a trampoline outside its Tasks Trace\n 987:\t * reader, e.g. static ftrace entry stubs or return thunks that hold a\n 988:\t * trampoline address they are about to jump to.\n 989:\t */\n 990:\tbool __weak arch_rcu_tasks_trampoline_text(unsigned long ip)\n 991:\t{\n 992:\t\treturn false;\n 993:\t}\n 994:\t\n 995:\t/**\n 996:\t * rcu_tasks_trampoline_text - Is @ip in text Tasks RCU protects but no reader marks?\n 997:\t * @ip: an interrupted instruction pointer\n 998:\t *\n 999:\t * True when a task interrupted at @ip may be executing, or about to enter\n1000:\t * or return into, text whose lifetime depends on synchronize_rcu_tasks()\n1001:\t * without being inside the Tasks Trace reader that text takes around its\n1002:\t * call-outs:\n1003:\t *\n1004:\t * - anything outside core kernel and module text (ftrace and BPF\n1005:\t * trampolines, kprobe slots and other dynamically allocated text; this\n1006:\t * deliberately does not ask is_ftrace_trampoline() and friends, since\n1007:\t * text being torn down may already be unregistered there);\n1008:\t * - whatever the architecture adds via arch_rcu_tasks_trampoline_text();\n1009:\t * - the text of a module that hosts an out-of-line ftrace direct-call\n1010:\t * trampoline (see ftrace_direct_mark_module());\n1011:\t * - the bytes after a kprobe that a pending jump optimization is about to\n1012:\t * overwrite, the one synchronize_rcu_tasks() user with no trampoline.\n1013:\t *\n1014:\t * A false positive only makes the task a holdout until its next quiescent\n1015:\t * event. Called with interrupts disabled from the irq-exit path.\n1016:\t */\n1017:\tbool rcu_tasks_trampoline_text(unsigned long ip)\n1018:\t{\n1019:\t\tbool ret = true;\n1020:\t\n1021:\t\tif (core_kernel_text(ip))\n1022:\t\t\treturn arch_rcu_tasks_trampoline_text(ip) ||\n1023:\t\t\t kprobe_in_optimized_region(ip);\n1024:\t\n1025:\t#ifdef CONFIG_MODULES\n1026:\t\tscoped_guard(rcu) {\n1027:\t\t\tstruct module *mod = __module_text_address(ip);\n1028:\t\n1029:\t\t\tif (mod)\n"
} |
| 23/2 | 2026/09/24 04:19 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 63845 output: 37 thoughts: 17 Thoughts: Okay, here's my attempt at summarizing the thought process, tailored for an expert audience, and written in the first person: **Diving into `kernel/rcu/tasks.h`** Alright, time to dig deeper. I need to understand how the scheduler interacts with RCU in this context. Let's see what `kernel/rcu/tasks.h` has to tell us. I'm focusing on the interaction between tasks and RCU, so I'll be looking for relevant data structures and functions that tie these concepts together. Specifically, I'm expecting to find details around grace-period management, task-related callbacks within the RCU lifecycle, and any mechanisms for preventing tasks from interfering with ongoing RCU operations or vice versa. This file should be the central point of reference for task-related synchronization. Hopefully, this file offers insights into their specific strategies for handling context switches, preemption, and potentially even migration scenarios within RCU critical sections. This seems like a critical file to review thoroughly. |
| 24/2 | 2026/09/24 04:19 | tool | read-file |
0mArgs:{
"File": "kernel/rcu/tasks.h",
"FirstLine": 1030,
"LineCount": 80
}Results: {
"Contents": "1030:\t\t\t\tret = READ_ONCE(mod-\u003eftrace_direct_tramp) ||\n1031:\t\t\t\t kprobe_in_optimized_region(ip);\n1032:\t\t}\n1033:\t#endif\n1034:\t\treturn ret;\n1035:\t}\n1036:\tNOKPROBE_SYMBOL(rcu_tasks_trampoline_text);\n1037:\t\n1038:\t/* Note a Tasks RCU quiescent event on this CPU. */\n1039:\tstatic void rcu_tasks_qs_event(void)\n1040:\t{\n1041:\t\tunsigned long *seq;\n1042:\t\n1043:\t\tguard(preempt_notrace)();\n1044:\t\tseq = this_cpu_ptr(\u0026rcu_tasks_qs_seq);\n1045:\t\t/* Order a preceding rcu_tasks_tramp_hold() before the count. */\n1046:\t\tsmp_store_release(seq, *seq + 1);\n1047:\t}\n1048:\t\n1049:\tstatic void rcu_tasks_tramp_hold(struct task_struct *t)\n1050:\t{\n1051:\t\tunsigned long flags;\n1052:\t\n1053:\t\tif (t-\u003ercu_tasks_holdout)\n1054:\t\t\treturn;\n1055:\t\traw_spin_lock_irqsave(\u0026rcu_tasks_tramp_lock, flags);\n1056:\t\tlist_add_tail(\u0026t-\u003ercu_tasks_holdout_list, \u0026rcu_tasks_tramp_holdouts);\n1057:\t\tWRITE_ONCE(t-\u003ercu_tasks_holdout, true);\n1058:\t\traw_spin_unlock_irqrestore(\u0026rcu_tasks_tramp_lock, flags);\n1059:\t}\n1060:\t\n1061:\tstatic void rcu_tasks_tramp_release(struct task_struct *t)\n1062:\t{\n1063:\t\tunsigned long flags;\n1064:\t\n1065:\t\tif (likely(!t-\u003ercu_tasks_holdout))\n1066:\t\t\treturn;\n1067:\t\traw_spin_lock_irqsave(\u0026rcu_tasks_tramp_lock, flags);\n1068:\t\tlist_del_init(\u0026t-\u003ercu_tasks_holdout_list);\n1069:\t\tWRITE_ONCE(t-\u003ercu_tasks_holdout, false);\n1070:\t\traw_spin_unlock_irqrestore(\u0026rcu_tasks_tramp_lock, flags);\n1071:\t}\n1072:\t\n1073:\t/**\n1074:\t * rcu_tasks_irq_resched_enter - Tasks RCU hook for the irq-exit reschedule check\n1075:\t * @ip: instruction pointer of the interrupted (task-level) context\n1076:\t *\n1077:\t * Called with interrupts disabled when an interrupt returning to kernel\n1078:\t * mode is about to preempt_schedule_irq(), the one context switch that can\n1079:\t * catch a task inside unmarked trampoline text. Record where the task is\n1080:\t * parked for as long as it is (rcu_tasks_wait_irq_preempted() looks at\n1081:\t * that), and if it is inside such text make it a holdout before\n1082:\t * __schedule() reports the quiescent event; if it is not, this is as good\n1083:\t * as a voluntary switch for ending an earlier hold.\n1084:\t */\n1085:\tvoid rcu_tasks_irq_resched_enter(unsigned long ip)\n1086:\t{\n1087:\t\tstruct task_struct *t = current;\n1088:\t\tstruct rcu_tasks_percpu *rtpcp = this_cpu_ptr(rcu_tasks.rtpcpu);\n1089:\t\n1090:\t\tlockdep_assert_irqs_disabled();\n1091:\t\tWRITE_ONCE(t-\u003ercu_tasks_irq_ip, ip);\n1092:\t\tt-\u003ercu_tasks_exit_cpu = smp_processor_id();\n1093:\t\traw_spin_lock_rcu_node(rtpcp);\n1094:\t\tlist_add(\u0026t-\u003ercu_tasks_exit_list, \u0026rtpcp-\u003ertp_exit_list);\n1095:\t\traw_spin_unlock_rcu_node(rtpcp);\n1096:\t\n1097:\t\tif (unlikely(rcu_tasks_trampoline_text(ip)))\n1098:\t\t\trcu_tasks_tramp_hold(t);\n1099:\t\telse\n1100:\t\t\trcu_tasks_tramp_release(t);\n1101:\t}\n1102:\tNOKPROBE_SYMBOL(rcu_tasks_irq_resched_enter);\n1103:\t\n1104:\t/**\n1105:\t * rcu_tasks_irq_resched_exit - preempt_schedule_irq() has returned\n1106:\t *\n1107:\t * The task is running again (possibly elsewhere) and about to return to the\n1108:\t * interrupted context; it is no longer parked anywhere.\n1109:\t */\n"
} |
| 25/2 | 2026/09/24 04:19 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 65516 output: 37 thoughts: 27 Thoughts: Okay, here's my attempt at summarizing the thought process, as requested, aimed at an expert audience and written in the first person: **Diving Back into `kernel/rcu/tasks.h`** Alright, time to get back to it. I need to keep reading, so I'm jumping back into `kernel/rcu/tasks.h` and picking up the thread. I'm focusing on lines 1110-1200 specifically. Let's see what gems we can unearth in this section. I'm expecting this area to relate to some details of RCU's task management, perhaps how it interacts with threads or handles context switches in various scenarios. Or maybe there's some fine-grained locking or synchronization going on here to ensure correctness. I'm keen to identify the specific aspect of the RCU task management which is in focus in this section of code. Okay, let's get started and see what we can find. |
| 26/2 | 2026/09/24 04:20 | tool | read-file |
0mArgs:{
"File": "kernel/rcu/tasks.h",
"FirstLine": 1110,
"LineCount": 80
}Results: {
"Contents": "1110:\tvoid rcu_tasks_irq_resched_exit(void)\n1111:\t{\n1112:\t\tstruct task_struct *t = current;\n1113:\t\tstruct rcu_tasks_percpu *rtpcp = per_cpu_ptr(rcu_tasks.rtpcpu, t-\u003ercu_tasks_exit_cpu);\n1114:\t\n1115:\t\tlockdep_assert_irqs_disabled();\n1116:\t\traw_spin_lock_rcu_node(rtpcp);\n1117:\t\tlist_del_init(\u0026t-\u003ercu_tasks_exit_list);\n1118:\t\traw_spin_unlock_rcu_node(rtpcp);\n1119:\t\tWRITE_ONCE(t-\u003ercu_tasks_irq_ip, 0);\n1120:\t}\n1121:\tNOKPROBE_SYMBOL(rcu_tasks_irq_resched_exit);\n1122:\t\n1123:\t/**\n1124:\t * rcu_tasks_note_qs - Tasks RCU hook for a context switch or explicit QS\n1125:\t * @t: current\n1126:\t * @preempt: this is a preemption rather than a voluntary switch\n1127:\t *\n1128:\t * Every pass through __schedule() (and cond_resched_tasks_rcu_qs(), and a\n1129:\t * tick from userspace or idle) is a quiescent event for this CPU: unmarked\n1130:\t * trampoline text never calls anything that schedules, and the irq-exit\n1131:\t * path has already made @t a holdout if it is preempting inside such text.\n1132:\t * Any of these outside an irq-exit preemption also shows @t itself to be\n1133:\t * outside, ending an earlier hold -- including cond_resched() under\n1134:\t * PREEMPT_DYNAMIC's none/voluntary modes, where the irq-exit path is off.\n1135:\t */\n1136:\tvoid rcu_tasks_note_qs(struct task_struct *t, bool preempt)\n1137:\t{\n1138:\t\tWARN_ON_ONCE(t != current);\n1139:\t\tif (!READ_ONCE(t-\u003ercu_tasks_irq_ip))\n1140:\t\t\trcu_tasks_tramp_release(t);\n1141:\t\trcu_tasks_qs_event();\n1142:\t}\n1143:\tEXPORT_SYMBOL_GPL(rcu_tasks_note_qs);\t/* cond_resched_tasks_rcu_qs() */\n1144:\t\n1145:\t/**\n1146:\t * rcu_tasks_wait_irq_preempted - wait for tasks irq-preempted inside @inside\n1147:\t * @inside: predicate on a task's recorded irq-exit preemption IP\n1148:\t *\n1149:\t * For a caller about to make some ordinary text unsafe to be parked in\n1150:\t * (the kprobe jump optimizer): once the caller has arranged for\n1151:\t * rcu_tasks_trampoline_text() to cover that text, new irq-exit preemptions\n1152:\t * there become holdouts, but a task preempted there earlier is invisible\n1153:\t * to the grace period. Wait until no parked task's recorded preemption IP\n1154:\t * is inside; a following synchronize_rcu_tasks() then covers the rest.\n1155:\t * The leading synchronize_rcu() orders the caller's arrangement against\n1156:\t * preemptions in flight, which run with interrupts disabled.\n1157:\t */\n1158:\tvoid rcu_tasks_wait_irq_preempted(bool (*inside)(unsigned long ip))\n1159:\t{\n1160:\t\tstruct task_struct *t;\n1161:\t\tunsigned long flags;\n1162:\t\tint cpu, kick;\n1163:\t\tbool found;\n1164:\t\n1165:\t\tsynchronize_rcu();\n1166:\t\tfor (;;) {\n1167:\t\t\tfound = false;\n1168:\t\t\tfor_each_possible_cpu(cpu) {\n1169:\t\t\t\tstruct rcu_tasks_percpu *rtpcp = per_cpu_ptr(rcu_tasks.rtpcpu, cpu);\n1170:\t\n1171:\t\t\t\tkick = -1;\n1172:\t\t\t\traw_spin_lock_irqsave_rcu_node(rtpcp, flags);\n1173:\t\t\t\tlist_for_each_entry(t, \u0026rtpcp-\u003ertp_exit_list, rcu_tasks_exit_list) {\n1174:\t\t\t\t\tif (inside(READ_ONCE(t-\u003ercu_tasks_irq_ip))) {\n1175:\t\t\t\t\t\tfound = true;\n1176:\t\t\t\t\t\tif (task_curr(t))\n1177:\t\t\t\t\t\t\tkick = task_cpu(t);\n1178:\t\t\t\t\t}\n1179:\t\t\t\t}\n1180:\t\t\t\traw_spin_unlock_irqrestore_rcu_node(rtpcp, flags);\n1181:\t\t\t\tif (kick \u003e= 0)\n1182:\t\t\t\t\tresched_cpu(kick);\n1183:\t\t\t}\n1184:\t\t\tif (!found)\n1185:\t\t\t\treturn;\n1186:\t\t\tschedule_timeout_uninterruptible(1);\n1187:\t\t}\n1188:\t}\n1189:\t\n"
} |
| 27/2 | 2026/09/24 04:20 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 67234 output: 37 thoughts: 27 Thoughts: Okay, here's my attempt at summarizing the thought process, as requested, in the first person and targeted at an expert: **Diving Back into `kernel/rcu/tasks.h`** Alright, let's pick up where we left off, lines 1190-1280 of `kernel/rcu/tasks.h`. I'm expecting this section to deal with the more granular aspects of task-related RCU operations. Given the file and the general structure of the RCU implementation, I'm anticipating dives into tasks or thread handling within the RCU context. I will be looking out for any mentions of task lists, maybe synchronization primitives specifically used by RCU on per-task bases, or mechanisms for identifying which tasks are currently participating in grace periods or other RCU-related states. I want to have a look at how this ties in with interrupt handlers, and any specific considerations needed when a task might be pre-empted or migrated. I'll need to pay close attention to the lock hierarchies, especially if there are fine-grained locks involved here that might interact with other subsystems. Basically, I want to see how the RCU machinery specifically integrates with task lifecycle management. |
| 28/2 | 2026/09/24 04:20 | tool | read-file |
0mArgs:{
"File": "kernel/rcu/tasks.h",
"FirstLine": 1190,
"LineCount": 80
}Results: {
"Contents": "1190:\t/* Has @cpu passed a quiescent event since the snapshot, or need it not? */\n1191:\tstatic bool rcu_tasks_cpu_quiescent(int cpu)\n1192:\t{\n1193:\t\tif (!cpu_online(cpu))\n1194:\t\t\treturn true;\n1195:\t\t/* Pairs with the release in rcu_tasks_qs_event(). */\n1196:\t\tif (smp_load_acquire(per_cpu_ptr(\u0026rcu_tasks_qs_seq, cpu)) !=\n1197:\t\t per_cpu(rcu_tasks_qs_snap, cpu))\n1198:\t\t\treturn true;\n1199:\t\t/*\n1200:\t\t * Idle or nohz_full userspace in an RCU extended quiescent state: no\n1201:\t\t * task-level kernel frames can be live in a trampoline there, and\n1202:\t\t * whatever ran before has switched out. An idle CPU that RCU is\n1203:\t\t * watching (an interrupt from idle, or the traceable part of the idle\n1204:\t\t * loop) is deliberately not let through: the idle task may be in a\n1205:\t\t * trampoline with that interrupt on top, and since it is never\n1206:\t\t * preempted from irq exit nothing else would catch it. It gets the\n1207:\t\t * resched_cpu() like anyone else and counts once the idle loop itself\n1208:\t\t * schedules, which it cannot do from inside a trampoline.\n1209:\t\t */\n1210:\t\treturn !(ct_rcu_watching_cpu(cpu) \u0026 CT_RCU_WATCHING);\n1211:\t}\n1212:\t\n1213:\t/* Rate-limited stall report; returns true if the caller should add detail. */\n1214:\tstatic bool rcu_tasks_tramp_stall(struct rcu_tasks *rtp, unsigned long *lastreport,\n1215:\t\t\t\t\t const char *what)\n1216:\t{\n1217:\t\tint rtst = READ_ONCE(rcu_task_stall_timeout);\n1218:\t\n1219:\t\tif (rtst \u003c= 0 || !time_after(jiffies, *lastreport + rtst))\n1220:\t\t\treturn false;\n1221:\t\t*lastreport = jiffies;\n1222:\t\tpr_err(\"INFO: %s: %s, grace period %lu is %lu jiffies old\\n\", rtp-\u003ekname,\n1223:\t\t what, rcu_seq_current(\u0026rtp-\u003etasks_gp_seq), jiffies - rtp-\u003egp_start);\n1224:\t\treturn true;\n1225:\t}\n1226:\t\n1227:\t/*\n1228:\t * Steps 1/4: wait until every online CPU has context switched or is in an\n1229:\t * RCU extended quiescent state. A CPU that has neither after a jiffy is\n1230:\t * asked to switch with resched_cpu(), which takes it through\n1231:\t * rcu_tasks_irq_resched_enter() and __schedule() (or, from userspace, a\n1232:\t * guest or the idle loop, straight to __schedule()).\n1233:\t */\n1234:\tstatic void rcu_tasks_tramp_wait_cpus(struct rcu_tasks *rtp, unsigned long *lastreport)\n1235:\t{\n1236:\t\tstruct cpumask *pending = \u0026rcu_tasks_pending_cpus;\n1237:\t\tunsigned long start;\n1238:\t\tint cpu;\n1239:\t\n1240:\t\t/*\n1241:\t\t * The quiescent events run with preemption (in practice interrupts)\n1242:\t\t * disabled, so after this any event we go on to count began after the\n1243:\t\t * caller's updates -- the unpublished trampoline, and whatever\n1244:\t\t * rcu_tasks_trampoline_text() consults -- were visible to it.\n1245:\t\t */\n1246:\t\tsynchronize_rcu();\n1247:\t\n1248:\t\tstart = jiffies;\n1249:\t\tfor_each_online_cpu(cpu) {\n1250:\t\t\tper_cpu(rcu_tasks_qs_snap, cpu) = READ_ONCE(per_cpu(rcu_tasks_qs_seq, cpu));\n1251:\t\t\t__cpumask_set_cpu(cpu, pending);\n1252:\t\t}\n1253:\t\t/* Snapshots before the checks below; pairs with rcu_tasks_qs_event(). */\n1254:\t\tsmp_mb();\n1255:\t\n1256:\t\tfor (;;) {\n1257:\t\t\tfor_each_cpu(cpu, pending)\n1258:\t\t\t\tif (rcu_tasks_cpu_quiescent(cpu))\n1259:\t\t\t\t\t__cpumask_clear_cpu(cpu, pending);\n1260:\t\t\tif (cpumask_empty(pending))\n1261:\t\t\t\tbreak;\n1262:\t\t\tif (time_after(jiffies, start)) {\n1263:\t\t\t\tfor_each_cpu(cpu, pending)\n1264:\t\t\t\t\tresched_cpu(cpu);\n1265:\t\t\t\trtp-\u003en_ipis += cpumask_weight(pending);\n1266:\t\t\t}\n1267:\t\t\tschedule_timeout_idle(1);\n1268:\t\t\tif (rcu_tasks_tramp_stall(rtp, lastreport, \"CPUs without a quiescent event\"))\n1269:\t\t\t\tpr_err(\"\\tCPUs: %*pbl\\n\", cpumask_pr_args(pending));\n"
} |
| 29/2 | 2026/09/24 04:20 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 69034 output: 37 thoughts: 27 Thoughts: Okay, here's my attempt at summarizing the thought process, as requested, targeting an expert audience and speaking in the first person: **Diving into `kernel/rcu/tasks.h` - Lines 1270-1350** Alright, let's keep going. Time to get back into the nitty-gritty and focus on lines 1270 through 1350 of `kernel/rcu/tasks.h`. I'm expecting this section to deal with some aspect of RCU's interaction with tasks, given the file's name. This probably dives deeper into how RCU synchronization impacts or is influenced by task scheduling, preemption, or perhaps even context switches, especially given the potential for long RCU read-side critical sections and the need to prevent priority inversion. I'm keen to see if this delves into how the RCU implementation handles tasks that are in a read-side critical section when they are interrupted, especially regarding how the grace periods are managed in such scenarios. I'm hoping to clarify how they track tasks, what specific mechanisms are utilized, and if there's any clever handling of tasks that are running with RCU enabled, but aren't actively participating in an RCU read-side critical section. Hopefully it includes any considerations about how it deals with tasks exiting the read-side critical section, and how that impacts grace period advancement. I'm particularly interested in seeing how it differentiates between different task states relative to RCU's operation - are they sleeping, running, preempted? And how does the kernel track which tasks are currently in read-side critical sections? What kinds of data structures are used for this, and what implications does that have for locking and contention? Let's get reading and see what insights this section has to offer. |
| 30/2 | 2026/09/24 04:20 | tool | read-file |
0mArgs:{
"File": "kernel/rcu/tasks.h",
"FirstLine": 1270,
"LineCount": 80
}Results: {
"Contents": "1270:\t\t}\n1271:\t}\n1272:\t\n1273:\t/*\n1274:\t * Steps 2/4: wait for the tasks that were holdouts when we looked to stop\n1275:\t * being holdouts. They are moved to a private list so that tasks becoming\n1276:\t * holdouts later (in live trampolines) cannot keep us here; each removes\n1277:\t * itself via rcu_tasks_tramp_release() wherever it is queued.\n1278:\t */\n1279:\tstatic void rcu_tasks_tramp_wait_holdouts(struct rcu_tasks *rtp, unsigned long *lastreport)\n1280:\t{\n1281:\t\tstruct task_struct *t;\n1282:\t\tunsigned long flags;\n1283:\t\tint cpu;\n1284:\t\n1285:\t\traw_spin_lock_irqsave(\u0026rcu_tasks_tramp_lock, flags);\n1286:\t\tlist_splice_tail_init(\u0026rcu_tasks_tramp_holdouts, \u0026rcu_tasks_gp_holdouts);\n1287:\t\traw_spin_unlock_irqrestore(\u0026rcu_tasks_tramp_lock, flags);\n1288:\t\n1289:\t\tfor (;;) {\n1290:\t\t\tstruct cpumask *kick = \u0026rcu_tasks_pending_cpus;\n1291:\t\t\tstruct task_struct *show[8];\n1292:\t\t\tint nshow = 0, i;\n1293:\t\t\tbool empty, report;\n1294:\t\n1295:\t\t\treport = rcu_tasks_tramp_stall(rtp, lastreport,\n1296:\t\t\t\t\t\t \"tasks preempted in trampoline text\");\n1297:\t\t\tcpumask_clear(kick);\n1298:\t\t\traw_spin_lock_irqsave(\u0026rcu_tasks_tramp_lock, flags);\n1299:\t\t\tempty = list_empty(\u0026rcu_tasks_gp_holdouts);\n1300:\t\t\tlist_for_each_entry(t, \u0026rcu_tasks_gp_holdouts, rcu_tasks_holdout_list) {\n1301:\t\t\t\tif (task_curr(t))\n1302:\t\t\t\t\t__cpumask_set_cpu(task_cpu(t), kick);\n1303:\t\t\t\tif (report \u0026\u0026 nshow \u003c ARRAY_SIZE(show))\n1304:\t\t\t\t\tshow[nshow++] = get_task_struct(t);\n1305:\t\t\t}\n1306:\t\t\traw_spin_unlock_irqrestore(\u0026rcu_tasks_tramp_lock, flags);\n1307:\t\t\t/* Never printk under the lock the irq-exit path takes. */\n1308:\t\t\tfor (i = 0; i \u003c nshow; i++) {\n1309:\t\t\t\tsched_show_task(show[i]);\n1310:\t\t\t\tput_task_struct(show[i]);\n1311:\t\t\t}\n1312:\t\t\tif (empty)\n1313:\t\t\t\tbreak;\n1314:\t\t\tfor_each_cpu(cpu, kick)\n1315:\t\t\t\tresched_cpu(cpu);\n1316:\t\t\trtp-\u003en_ipis += cpumask_weight(kick);\n1317:\t\t\tschedule_timeout_idle(1);\n1318:\t\t}\n1319:\t}\n1320:\t\n1321:\t/* Wait for one trampoline-reader Tasks RCU grace period. */\n1322:\tstatic void rcu_tasks_tramp_wait_gp(struct rcu_tasks *rtp)\n1323:\t{\n1324:\t\tunsigned long lastreport = jiffies;\n1325:\t\n1326:\t\tset_tasks_gp_state(rtp, RTGS_WAIT_SCAN_HOLDOUTS);\n1327:\t\trcu_tasks_tramp_wait_cpus(rtp, \u0026lastreport);\n1328:\t\trcu_tasks_tramp_wait_holdouts(rtp, \u0026lastreport);\n1329:\t\n1330:\t\tset_tasks_gp_state(rtp, RTGS_WAIT_READERS);\n1331:\t\tsynchronize_rcu_tasks_trace();\n1332:\t\n1333:\t\tset_tasks_gp_state(rtp, RTGS_SCAN_HOLDOUTS);\n1334:\t\trcu_tasks_tramp_wait_cpus(rtp, \u0026lastreport);\n1335:\t\trcu_tasks_tramp_wait_holdouts(rtp, \u0026lastreport);\n1336:\t\n1337:\t\tset_tasks_gp_state(rtp, RTGS_POST_GP);\n1338:\t}\n1339:\t\n1340:\tstatic int __init rcu_spawn_tasks_kthread(void)\n1341:\t{\n1342:\t\trcu_tasks.gp_sleep = HZ / 10;\n1343:\t\tif (rcu_tasks_lazy_ms \u003e= 0)\n1344:\t\t\trcu_tasks.lazy_jiffies = msecs_to_jiffies(rcu_tasks_lazy_ms);\n1345:\t\trcu_tasks.wait_state = TASK_IDLE;\n1346:\t\trcu_spawn_tasks_kthread_generic(\u0026rcu_tasks);\n1347:\t\treturn 0;\n1348:\t}\n1349:\t\n"
} |
| 31/2 | 2026/09/24 04:20 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 70902 output: 43 thoughts: 448 Thoughts: **Analyzing the New RCU Tasks Grace Period Implementation** Wow, this is interesting! I'm seeing a completely new grace period implementation for Tasks RCU. Look at all these new functions: `rcu_tasks_tramp_wait_gp()`, `rcu_tasks_tramp_wait_cpus()`, `rcu_tasks_tramp_wait_holdouts()`, `rcu_tasks_irq_resched_enter()`, `rcu_tasks_irq_resched_exit()`, `rcu_tasks_note_qs()`, `rcu_tasks_wait_irq_preempted()`, `rcu_tasks_trampoline_text()`, `arch_rcu_tasks_trampoline_text()`, and `kprobe_in_optimized_region()`. That's a significant amount of new code dedicated to this. And check this out – there's a BPF trampoline JIT emitter in here too: `emit_trace_rcu_reader()`, which is called within `__arch_prepare_bpf_trampoline()`. Plus, there are new ftrace and optprobe entry macros added: `TRACE_RCU_READ_LOCK`, `TRACE_RCU_READ_UNLOCK`, `OPTPROBE_TRACE_RCU_READ_LOCK`, and `OPTPROBE_TRACE_RCU_READ_UNLOCK`. This looks like a concerted effort to provide comprehensive tracing capabilities. Now, let's take a look at how this all ties into the kprobe optimizer. I'll check in `kernel/kprobes.c`: ```c kprobe_optimizer()`: WRITE_ONCE(kprobe_optimizer_waiting, true); rcu_tasks_wait_irq_preempted(kprobe_in_optimized_region); synchronize_rcu_tasks(); WRITE_ONCE(kprobe_optimizer_waiting, false); ``` Okay, so `kprobe_optimizer()` is using `rcu_tasks_wait_irq_preempted()` and `kprobe_in_optimized_region()`. I knew it! Further, `kprobe_in_optimized_region()` is also called by `rcu_tasks_trampoline_text()`, which suggests that the kprobe optimization is integrated here. That's a clever use of the existing infrastructure. I'll also dig into `arch/x86/kernel/kprobes/opt.c` and take a look at lines 130-190. I'll need to see how they've integrated these new RCU tasks functions and data structures. |
| 32/2 | 2026/09/24 04:20 | tool | read-file |
0mArgs:{
"File": "arch/x86/kernel/kprobes/opt.c",
"FirstLine": 140,
"LineCount": 60
}Results: {
"Contents": " 140:\t\t\t\"3:\\n\"\n 141:\t#else\n 142:\t#define OPTPROBE_TRACE_RCU_READ_LOCK\n 143:\t#define OPTPROBE_TRACE_RCU_READ_UNLOCK\n 144:\t#endif\n 145:\t\n 146:\tasm (\n 147:\t\t\t\t\".pushsection .rodata\\n\"\n 148:\t\t\t\t\".global optprobe_template_entry\\n\"\n 149:\t\t\t\t\"optprobe_template_entry:\\n\"\n 150:\t#ifdef CONFIG_X86_64\n 151:\t\t\t\t\" pushq $\" __stringify(__KERNEL_DS) \"\\n\"\n 152:\t\t\t\t/* Save the 'sp - 8', this will be fixed later. */\n 153:\t\t\t\t\"\tpushq %rsp\\n\"\n 154:\t\t\t\t\"\tpushfq\\n\"\n 155:\t\t\t\t\".global optprobe_template_clac\\n\"\n 156:\t\t\t\t\"optprobe_template_clac:\\n\"\n 157:\t\t\t\tASM_NOP3\n 158:\t\t\t\tSAVE_REGS_STRING\n 159:\t\t\t\tOPTPROBE_TRACE_RCU_READ_LOCK\n 160:\t\t\t\t\"\tmovq %rsp, %rsi\\n\"\n 161:\t\t\t\t\".global optprobe_template_val\\n\"\n 162:\t\t\t\t\"optprobe_template_val:\\n\"\n 163:\t\t\t\tASM_NOP5\n 164:\t\t\t\tASM_NOP5\n 165:\t\t\t\t\".global optprobe_template_call\\n\"\n 166:\t\t\t\t\"optprobe_template_call:\\n\"\n 167:\t\t\t\tASM_NOP5\n 168:\t\t\t\tOPTPROBE_TRACE_RCU_READ_UNLOCK\n 169:\t\t\t\t/* Copy 'regs-\u003eflags' into 'regs-\u003ess'. */\n 170:\t\t\t\t\"\tmovq 18*8(%rsp), %rdx\\n\"\n 171:\t\t\t\t\"\tmovq %rdx, 20*8(%rsp)\\n\"\n 172:\t\t\t\tRESTORE_REGS_STRING\n 173:\t\t\t\t/* Skip 'regs-\u003eflags' and 'regs-\u003esp'. */\n 174:\t\t\t\t\"\taddq $16, %rsp\\n\"\n 175:\t\t\t\t/* And pop flags register from 'regs-\u003ess'. */\n 176:\t\t\t\t\"\tpopfq\\n\"\n 177:\t#else /* CONFIG_X86_32 */\n 178:\t\t\t\t\"\tpushl %ss\\n\"\n 179:\t\t\t\t/* Save the 'sp - 4', this will be fixed later. */\n 180:\t\t\t\t\"\tpushl %esp\\n\"\n 181:\t\t\t\t\"\tpushfl\\n\"\n 182:\t\t\t\t\".global optprobe_template_clac\\n\"\n 183:\t\t\t\t\"optprobe_template_clac:\\n\"\n 184:\t\t\t\tASM_NOP3\n 185:\t\t\t\tSAVE_REGS_STRING\n 186:\t\t\t\t\"\tmovl %esp, %edx\\n\"\n 187:\t\t\t\t\".global optprobe_template_val\\n\"\n 188:\t\t\t\t\"optprobe_template_val:\\n\"\n 189:\t\t\t\tASM_NOP5\n 190:\t\t\t\t\".global optprobe_template_call\\n\"\n 191:\t\t\t\t\"optprobe_template_call:\\n\"\n 192:\t\t\t\tASM_NOP5\n 193:\t\t\t\t/* Copy 'regs-\u003eflags' into 'regs-\u003ess'. */\n 194:\t\t\t\t\"\tmovl 14*4(%esp), %edx\\n\"\n 195:\t\t\t\t\"\tmovl %edx, 16*4(%esp)\\n\"\n 196:\t\t\t\tRESTORE_REGS_STRING\n 197:\t\t\t\t/* Skip 'regs-\u003eflags' and 'regs-\u003esp'. */\n 198:\t\t\t\t\"\taddl $8, %esp\\n\"\n 199:\t\t\t\t/* And pop flags register from 'regs-\u003ess'. */\n"
} |
| 33/2 | 2026/09/24 04:20 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 73000 output: 38 thoughts: 829 Thoughts: **RCU Reader Tracing in Action** Okay, here's what I'm seeing. With `CONFIG_TASKS_RCU_TRAMPOLINE_READERS` enabled, the `optprobe_template_entry` utilizes `OPTPROBE_TRACE_RCU_READ_LOCK` and `OPTPROBE_TRACE_RCU_READ_UNLOCK`. Now, I need to understand what those macros actually do. Let's dig into that. So, `OPTPROBE_TRACE_RCU_READ_LOCK`... Right. It looks like the code first retrieves the current task pointer using `%gs:current_task` into `%rcx`. Then, it increments the `TASK_trc_reader_nesting` field within the task struct. The code also manages an srcu counter to track the number of readers. And here's the corresponding code in `arch/x86/kernel/kprobes/opt.c`, lines 112-140. I can see the conditional compilation based on `CONFIG_TASKS_RCU_TRAMPOLINE_READERS` and `CONFIG_TASKS_TRACE_RCU_NO_MB`. The `OPTPROBE_TRACE_RCU_READ_LOCK` macro is essentially a sequence of assembly instructions that increment the read nesting level and, when necessary, an RCU trace counter. It's handling the nested read scenarios correctly. The `OPTPROBE_TRACE_RCU_READ_UNLOCK` then decrements the nesting level and updates the counters accordingly. Looks like the `OPTPROBE_TRACE_RCU_MB` is also a memory barrier, but only enabled if `CONFIG_TASKS_TRACE_RCU_NO_MB` is not enabled. Okay, let's switch gears and focus on the overall optimization process. I need to understand how the optimized instructions are copied and set up. Time to investigate what `copy_optimized_instructions`, `setup_optimize_kprobe`, and `arch_prepare_optimized_kprobe` do, and find usages of `optprobe_template_entry` within `arch/x86/kernel/kprobes/opt.c`. |
| 34/2 | 2026/09/24 04:20 | tool | grepper |
0mArgs:{
"Expression": "optprobe_template_entry",
"PathPrefix": "arch/x86/kernel/kprobes/"
}Results: {
"Output": "arch/x86/kernel/kprobes/opt.c=146=asm (\narch/x86/kernel/kprobes/opt.c-147-\t\t\t\".pushsection .rodata\\n\"\narch/x86/kernel/kprobes/opt.c:148:\t\t\t\".global optprobe_template_entry\\n\"\narch/x86/kernel/kprobes/opt.c:149:\t\t\t\"optprobe_template_entry:\\n\"\narch/x86/kernel/kprobes/opt.c-150-#ifdef CONFIG_X86_64\n--\narch/x86/kernel/kprobes/opt.c-206-#define TMPL_CLAC_IDX \\\narch/x86/kernel/kprobes/opt.c:207:\t((long)optprobe_template_clac - (long)optprobe_template_entry)\narch/x86/kernel/kprobes/opt.c-208-#define TMPL_MOVE_IDX \\\narch/x86/kernel/kprobes/opt.c:209:\t((long)optprobe_template_val - (long)optprobe_template_entry)\narch/x86/kernel/kprobes/opt.c-210-#define TMPL_CALL_IDX \\\narch/x86/kernel/kprobes/opt.c:211:\t((long)optprobe_template_call - (long)optprobe_template_entry)\narch/x86/kernel/kprobes/opt.c-212-#define TMPL_END_IDX \\\narch/x86/kernel/kprobes/opt.c:213:\t((long)optprobe_template_end - (long)optprobe_template_entry)\narch/x86/kernel/kprobes/opt.c-214-\n--\narch/x86/kernel/kprobes/opt.c=430=int arch_prepare_optimized_kprobe(struct optimized_kprobe *op,\n--\narch/x86/kernel/kprobes/opt.c-460-\t/* Copy arch-dep-instance from template */\narch/x86/kernel/kprobes/opt.c:461:\tmemcpy(buf, optprobe_template_entry, TMPL_END_IDX);\narch/x86/kernel/kprobes/opt.c-462-\n"
} |
| 35/2 | 2026/09/24 04:20 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 74944 output: 21 thoughts: 1328 Thoughts: **Analysis of Kernel Code for Fuzzing Potential** Okay, so I've been tracing the logic here. First, I see that `arch_prepare_optimized_kprobe` copies the template containing the newly added assembly, which is called when optimizing a kprobe, pretty straightforward. Now I need to understand what's been changed related to tracing, and how that impacts fuzzing. My attention shifts to `ftrace`. I note that in `arch/x86/kernel/ftrace_64.S`, `ftrace_caller` and `ftrace_regs_caller` now include `TRACE_RCU_READ_LOCK` and `TRACE_RCU_READ_UNLOCK`. Further, `ftrace_static_tramp_end` is added, which is interesting. In `arch/x86/kernel/ftrace.c`, there's a new function, `arch_rcu_tasks_trampoline_text(unsigned long ip)`. This function checks if an instruction pointer falls within a specific range related to the ftrace trampoline and also considers return thunk addresses. Finally, in `kernel/trace/ftrace.c`, I see that `ftrace_direct_mark_module()` is called via several `ftrace_direct` related functions. Now, a crucial question pops up in my mind: can this code be executed in common virtualization environments? I'm thinking GCE or QEMU. Good news, it appears it can! I confirm that with the configurations enabled, this is standard on QEMU amd64: SMP, X86_64, DYNAMIC_FTRACE, PREEMPTION, BPF_SYSCALL, KPROBES, and OPTPROBES. That's a strong indication this could be easily targeted by syzkaller. Next, I need to check the criteria for `WorthFuzzing`. Under Section 3: POSITIVE CRITERIA, it seems like a good fit: we're modifying logic related to synchronization, potentially impacting scheduler behavior. There's logic modification in BPF, and potentially syscall handling. Moreover, there is an addition of runtime assertions. In particular, I see `rcu_tasks_trace_assert_idle` which adds a `WARN_ON_ONCE` on `__exit_to_user_mode_validate`! And there is a `WARN_ON_ONCE(t != current)` in `rcu_tasks_note_qs`. Yes, this looks good. I assess this as `WorthFuzzing = true`. I now turn to determining the appropriate fuzzing parameters: `EnableConfigs`, `FocusSymbols`, and `Reasoning`. I need to precisely identify the kernel config flags required and the relevant kernel functions to target. Tracing the dependencies, I find that `CONFIG_TASKS_RCU_TRAMPOLINE_READERS` depends on the following: * `X86_64` and `SMP` and `DYNAMIC_FTRACE` through `HAVE_RCU_TRAMPOLINE_READERS` selected via `arch/x86/Kconfig`. * `TASKS_RCU` and `HAVE_RCU_TRAMPOLINE_READERS` and `GENERIC_IRQ_ENTRY` and `!NEED_SRCU_NMI_SAFE` via `TASKS_RCU_TRAMPOLINE_READERS` in `kernel/rcu/Kconfig`. * `TASKS_RCU` defaults to `NEED_TASKS_RCU && PREEMPTION` or forced via `FORCE_TASKS_RCU`. * `NEED_TASKS_RCU` can be selected by `BPF_SYSCALL`, `TRACING`, or `FUNCTION_TRACER`. * `GENERIC_IRQ_ENTRY` is selected by `X86`. * `!NEED_SRCU_NMI_SAFE` on x86, because `ARCH_HAS_NMI_SAFE_THIS_CPU_OPS` is set. * Therefore, `PREEMPTION` (or `PREEMPT`, or `PREEMPT_DYNAMIC`) is needed. For the specific features modified: * BPF trampolines require `BPF_SYSCALL` and `BPF_JIT`. I'll need to check how BPF trampolines are enabled on x86, which is important. |
| 36/2 | 2026/09/24 04:20 | tool | grepper |
0mError:no matches Args: {
"Expression": "BPF_TRAMPOLINE"
}Results: {
"error": "no matches"
} |
| 37/2 | 2026/09/24 04:20 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 77261 output: 23 thoughts: 27 Thoughts: Here's my thought process, as if I were thinking it: **Digging into eBPF Trampolines** Okay, so I need to understand more about how these eBPF trampolines are being set up or potentially configured. My first instinct is to dive directly into the kernel source code. I'm going to start with a targeted search. I'll look for `arch_prepare_bpf_trampoline`. That function name should give me a good entry point. If that doesn't immediately yield results, I'll broaden the scope. I know the Kconfig system is how options are defined, so I'll also check where `bpf` *trampoline* related options might be set in the Kconfig files. This should reveal the build-time configuration dependencies for the feature, and shed light on how it is architected for different platforms. This two-pronged approach -- a function name search paired with Kconfig exploration -- should quickly illuminate the relevant code paths and configuration options. From there, I can start to understand the low-level details of the trampoline mechanism. |
| 38/2 | 2026/09/24 04:20 | tool | grepper |
0mArgs:{
"Expression": "arch_prepare_bpf_trampoline"
}Results: {
"Output": "arch/arm64/net/bpf_jit_comp.c=2751=static void store_func_meta(struct jit_ctx *ctx, u64 func_meta, int func_meta_off)\n--\narch/arm64/net/bpf_jit_comp.c-2756-\narch/arm64/net/bpf_jit_comp.c:2757:/* Based on the x86's implementation of arch_prepare_bpf_trampoline().\narch/arm64/net/bpf_jit_comp.c-2758- *\n--\narch/arm64/net/bpf_jit_comp.c=3113=int arch_protect_bpf_trampoline(void *image, unsigned int size)\n--\narch/arm64/net/bpf_jit_comp.c-3117-\narch/arm64/net/bpf_jit_comp.c:3118:int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image,\narch/arm64/net/bpf_jit_comp.c-3119-\t\t\t\tvoid *ro_image_end, const struct btf_func_model *m,\n--\narch/loongarch/net/bpf_jit.c=1803=static void sign_extend(struct jit_ctx *ctx, int rd, int rj, u8 size, bool sign)\n--\narch/loongarch/net/bpf_jit.c-1830-\narch/loongarch/net/bpf_jit.c:1831:static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,\narch/loongarch/net/bpf_jit.c-1832-\t\t\t\t\t const struct btf_func_model *m, struct bpf_tramp_nodes *tnodes,\n--\narch/loongarch/net/bpf_jit.c-2137-\narch/loongarch/net/bpf_jit.c:2138:int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image,\narch/loongarch/net/bpf_jit.c-2139-\t\t\t\tvoid *ro_image_end, const struct btf_func_model *m,\n--\narch/loongarch/net/bpf_jit.c-2155-\tjit_fill_hole(image, (unsigned int)(ro_image_end - ro_image));\narch/loongarch/net/bpf_jit.c:2156:\tret = __arch_prepare_bpf_trampoline(\u0026ctx, im, m, tnodes, func_addr, flags);\narch/loongarch/net/bpf_jit.c-2157-\tif (ret \u003c 0)\n--\narch/loongarch/net/bpf_jit.c=2176=int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,\n--\narch/loongarch/net/bpf_jit.c-2185-\narch/loongarch/net/bpf_jit.c:2186:\tret = __arch_prepare_bpf_trampoline(\u0026ctx, \u0026im, m, tnodes, func_addr, flags);\narch/loongarch/net/bpf_jit.c-2187-\n--\narch/parisc/net/bpf_jit_comp64.c-10- * - check if bpf_jit_needs_zext() is needed (currently enabled)\narch/parisc/net/bpf_jit_comp64.c:11: * - implement arch_prepare_bpf_trampoline(), poke(), ...\narch/parisc/net/bpf_jit_comp64.c-12- */\n--\narch/powerpc/net/bpf_jit_comp.c=686=static int invoke_bpf_mod_ret(u32 *image, u32 *ro_image, struct codegen_context *ctx,\n--\narch/powerpc/net/bpf_jit_comp.c-723-/*\narch/powerpc/net/bpf_jit_comp.c:724: * Refer __arch_prepare_bpf_trampoline() for stack component details.\narch/powerpc/net/bpf_jit_comp.c-725- *\n--\narch/powerpc/net/bpf_jit_comp.c=797=static void bpf_trampoline_restore_args_stack(u32 *image, struct codegen_context *ctx,\n--\narch/powerpc/net/bpf_jit_comp.c-811-\narch/powerpc/net/bpf_jit_comp.c:812:static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_image,\narch/powerpc/net/bpf_jit_comp.c-813-\t\t\t\t\t void *rw_image_end, void *ro_image,\n--\narch/powerpc/net/bpf_jit_comp.c=1192=int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,\n--\narch/powerpc/net/bpf_jit_comp.c-1197-\narch/powerpc/net/bpf_jit_comp.c:1198:\tret = __arch_prepare_bpf_trampoline(\u0026im, NULL, NULL, NULL, m, flags, tnodes, func_addr);\narch/powerpc/net/bpf_jit_comp.c-1199-\treturn ret;\n--\narch/powerpc/net/bpf_jit_comp.c-1201-\narch/powerpc/net/bpf_jit_comp.c:1202:int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *image_end,\narch/powerpc/net/bpf_jit_comp.c-1203-\t\t\t\tconst struct btf_func_model *m, u32 flags,\n--\narch/powerpc/net/bpf_jit_comp.c-1218-\narch/powerpc/net/bpf_jit_comp.c:1219:\tret = __arch_prepare_bpf_trampoline(im, rw_image, rw_image + size, image, m,\narch/powerpc/net/bpf_jit_comp.c-1220-\t\t\t\t\t flags, tnodes, func_addr);\n--\narch/riscv/net/bpf_jit_comp64.c=969=static int invoke_bpf(struct bpf_tramp_nodes *tn, int args_off, int retval_off,\n--\narch/riscv/net/bpf_jit_comp64.c-991-\narch/riscv/net/bpf_jit_comp64.c:992:static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,\narch/riscv/net/bpf_jit_comp64.c-993-\t\t\t\t\t const struct btf_func_model *m,\n--\narch/riscv/net/bpf_jit_comp64.c=1299=int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,\n--\narch/riscv/net/bpf_jit_comp64.c-1308-\tctx.ro_insns = NULL;\narch/riscv/net/bpf_jit_comp64.c:1309:\tret = __arch_prepare_bpf_trampoline(\u0026im, m, tnodes, func_addr, flags, \u0026ctx);\narch/riscv/net/bpf_jit_comp64.c-1310-\n--\narch/riscv/net/bpf_jit_comp64.c=1319=void arch_free_bpf_trampoline(void *image, unsigned int size)\n--\narch/riscv/net/bpf_jit_comp64.c-1323-\narch/riscv/net/bpf_jit_comp64.c:1324:int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image,\narch/riscv/net/bpf_jit_comp64.c-1325-\t\t\t\tvoid *ro_image_end, const struct btf_func_model *m,\n--\narch/riscv/net/bpf_jit_comp64.c-1340-\tctx.ro_insns = ro_image;\narch/riscv/net/bpf_jit_comp64.c:1341:\tret = __arch_prepare_bpf_trampoline(im, m, tnodes, func_addr, flags, \u0026ctx);\narch/riscv/net/bpf_jit_comp64.c-1342-\tif (ret \u003c 0)\n--\narch/s390/net/bpf_jit_comp.c=2680=static int alloc_stack(struct bpf_tramp_jit *tjit, size_t size)\n--\narch/s390/net/bpf_jit_comp.c-2697-\narch/s390/net/bpf_jit_comp.c:2698:static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,\narch/s390/net/bpf_jit_comp.c-2699-\t\t\t\t\t struct bpf_tramp_jit *tjit,\n--\narch/s390/net/bpf_jit_comp.c=3016=int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,\n--\narch/s390/net/bpf_jit_comp.c-3024-\narch/s390/net/bpf_jit_comp.c:3025:\tret = __arch_prepare_bpf_trampoline(\u0026im, \u0026tjit, m, flags,\narch/s390/net/bpf_jit_comp.c-3026-\t\t\t\t\t tnodes, orig_call);\n--\narch/s390/net/bpf_jit_comp.c-3030-\narch/s390/net/bpf_jit_comp.c:3031:int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image,\narch/s390/net/bpf_jit_comp.c-3032-\t\t\t\tvoid *image_end, const struct btf_func_model *m,\n--\narch/s390/net/bpf_jit_comp.c-3040-\tmemset(\u0026tjit, 0, sizeof(tjit));\narch/s390/net/bpf_jit_comp.c:3041:\tret = __arch_prepare_bpf_trampoline(im, \u0026tjit, m, flags,\narch/s390/net/bpf_jit_comp.c-3042-\t\t\t\t\t tnodes, func_addr);\n--\narch/s390/net/bpf_jit_comp.c-3054-\ttjit.common.prg_buf = image;\narch/s390/net/bpf_jit_comp.c:3055:\tret = __arch_prepare_bpf_trampoline(im, \u0026tjit, m, flags,\narch/s390/net/bpf_jit_comp.c-3056-\t\t\t\t\t tnodes, func_addr);\n--\narch/x86/net/bpf_jit_comp.c=3305=static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog,\n--\narch/x86/net/bpf_jit_comp.c-3321-\t * bpf_tramp_run_ctx is already preserved by\narch/x86/net/bpf_jit_comp.c:3322:\t * arch_prepare_bpf_trampoline().\narch/x86/net/bpf_jit_comp.c-3323-\t *\n--\narch/x86/net/bpf_jit_comp.c=3443=static int invoke_bpf_mod_ret(const struct btf_func_model *m, u8 **pprog,\n--\narch/x86/net/bpf_jit_comp.c-3544- */\narch/x86/net/bpf_jit_comp.c:3545:static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_image,\narch/x86/net/bpf_jit_comp.c-3546-\t\t\t\t\t void *rw_image_end, void *image,\n--\narch/x86/net/bpf_jit_comp.c=3884=int arch_protect_bpf_trampoline(void *image, unsigned int size)\n--\narch/x86/net/bpf_jit_comp.c-3888-\narch/x86/net/bpf_jit_comp.c:3889:int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *image_end,\narch/x86/net/bpf_jit_comp.c-3890-\t\t\t\tconst struct btf_func_model *m, u32 flags,\n--\narch/x86/net/bpf_jit_comp.c-3904-\narch/x86/net/bpf_jit_comp.c:3905:\tret = __arch_prepare_bpf_trampoline(im, rw_image, rw_image + size, image, m,\narch/x86/net/bpf_jit_comp.c-3906-\t\t\t\t\t flags, tnodes, func_addr);\n--\narch/x86/net/bpf_jit_comp.c=3918=int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,\n--\narch/x86/net/bpf_jit_comp.c-3924-\narch/x86/net/bpf_jit_comp.c:3925:\t/* Allocate a temporary buffer for __arch_prepare_bpf_trampoline().\narch/x86/net/bpf_jit_comp.c-3926-\t *\n--\narch/x86/net/bpf_jit_comp.c-3935-\narch/x86/net/bpf_jit_comp.c:3936:\tret = __arch_prepare_bpf_trampoline(\u0026im, image, image + PAGE_SIZE, image,\narch/x86/net/bpf_jit_comp.c-3937-\t\t\t\t\t m, flags, tnodes, func_addr);\n--\ninclude/linux/bpf.h=1310=struct bpf_tramp_image;\ninclude/linux/bpf.h:1311:int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *image_end,\ninclude/linux/bpf.h-1312-\t\t\t\tconst struct btf_func_model *m, u32 flags,\n--\nkernel/bpf/bpf_struct_ops.c=637=int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_nodes *tnodes,\n--\nkernel/bpf/bpf_struct_ops.c-668-\nkernel/bpf/bpf_struct_ops.c:669:\tsize = arch_prepare_bpf_trampoline(NULL, image + image_off,\nkernel/bpf/bpf_struct_ops.c-670-\t\t\t\t\t image + image_off + size,\n--\nkernel/bpf/trampoline.c=710=static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mutex,\n--\nkernel/bpf/trampoline.c-779-\nkernel/bpf/trampoline.c:780:\terr = arch_prepare_bpf_trampoline(im, im-\u003eimage, im-\u003eimage + size,\nkernel/bpf/trampoline.c-781-\t\t\t\t\t \u0026tr-\u003efunc.model, tr-\u003eflags, tnodes,\n--\nkernel/bpf/trampoline.c=1478=int __weak\nkernel/bpf/trampoline.c:1479:arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *image_end,\nkernel/bpf/trampoline.c-1480-\t\t\t const struct btf_func_model *m, u32 flags,\n"
} |
| 39/2 | 2026/09/24 04:20 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 81820 output: 29 thoughts: 16 Thoughts: Okay, here's my summarized thought process, assuming I'm an expert in this area and the user is too: **Investigating the Configuration for `bpf_trampoline_update`** Alright, let's get down to brass tacks. I need to figure out exactly what configuration options are enabling `bpf_trampoline_update`. It’s not enough to know *that* it’s enabled; I need to understand *how* the system is configured to allow it. This means drilling down into the relevant configuration files and maybe even checking the running kernel parameters if necessary. This isn't just about finding a toggle; it's about seeing the context. Are we dealing with a specific kernel feature enabled in the build itself? Is this controlled via a module option? Is there a runtime setting involved? Knowing the precise mechanism is key to understanding the potential implications and how this feature might be leveraged or restricted in different environments. So, time to start digging... let's see what the configuration has to say. |
| 40/2 | 2026/09/24 04:20 | tool | grepper |
0mError:no matches Args: {
"Expression": "BPF_TRAMPOLINE",
"PathPrefix": "kernel/"
}Results: {
"error": "no matches"
} |
| 41/2 | 2026/09/24 04:20 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 82094 output: 29 thoughts: 12 Thoughts: Here's my take on those thoughts, tailored for an expert audience: **Diving into the BPF Makefile** Alright, time to get my hands dirty. My initial thought process is always a methodical one, so let's start with the `kernel/bpf/Makefile`. That's the obvious entry point. I need to understand the build system's configuration for BPF – what are the dependencies, flags, and rules that govern the compilation and linking of BPF programs and related infrastructure within the kernel? I'm thinking about the implications. What are the key variables defined? Are they leveraging any standard kernel build macros or doing something custom? Are there any obvious areas for optimization or potential conflicts with other subsystems? I want to pinpoint where the BPF compilation process intersects with the overall kernel build and understand how any changes would ripple through the build pipeline. This is where I can get a true sense of the build process and potential pitfalls. |
| 42/2 | 2026/09/24 04:20 | tool | grepper |
0mArgs:{
"Expression": "trampoline\\.o",
"PathPrefix": "kernel/bpf/"
}Results: {
"Output": "kernel/bpf/Makefile=16=obj-$(CONFIG_BPF_SYSCALL) += disasm.o mprog.o\nkernel/bpf/Makefile:17:obj-$(CONFIG_BPF_JIT) += trampoline.o\nkernel/bpf/Makefile-18-obj-$(CONFIG_BPF_SYSCALL) += btf.o memalloc.o rqspinlock.o stream.o\n"
} |
| 43/2 | 2026/09/24 04:20 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 82441 output: 28 thoughts: 47 Thoughts: Okay, here's my interpretation of those thoughts, tailored for an expert audience, as if they were my own: **Kernel Exploration: JIT, Syscalls, and Optprobes** Right, let's see, `BPF_JIT` and `BPF_SYSCALL` – those are the key BPF components for runtime code generation and direct syscall access, respectively. Always good to refresh the specifics on their interaction. Now, it's making me think about `OPTPROBES`. Where does that fit in the grand scheme? I seem to recall it having implications in this context, especially in how we gather information. Given the architecture being x86, checking `arch/x86/Kconfig` for `OPTPROBES` seems like the logical first step. I need to understand its current state and any related dependencies to get a comprehensive view of how these features interact and what potential implications they have on performance, security, and instrumentation capabilities. It will likely reveal how probes and optimization strategies are managed within the x86 architecture. This context is essential for building a complete understanding of how BPF can operate in this environment. |
| 44/2 | 2026/09/24 04:20 | tool | grepper |
0mArgs:{
"Expression": "OPTPROBES",
"PathPrefix": "arch/x86"
}Results: {
"Output": "arch/x86/Kconfig=58=config X86\n--\narch/x86/Kconfig-277-\tselect HAVE_OBJTOOL\t\t\tif X86_64\narch/x86/Kconfig:278:\tselect HAVE_OPTPROBES\narch/x86/Kconfig-279-\tselect HAVE_PAGE_SIZE_4KB\n--\narch/x86/kernel/kprobes/Makefile=6=obj-$(CONFIG_KPROBES)\t\t+= core.o\narch/x86/kernel/kprobes/Makefile:7:obj-$(CONFIG_OPTPROBES)\t\t+= opt.o\narch/x86/kernel/kprobes/Makefile-8-obj-$(CONFIG_KPROBES_ON_FTRACE)\t+= ftrace.o\n--\narch/x86/kernel/kprobes/common.h=93=extern void synthesize_relcall(void *dest, void *from, void *to);\narch/x86/kernel/kprobes/common.h-94-\narch/x86/kernel/kprobes/common.h:95:#ifdef\tCONFIG_OPTPROBES\narch/x86/kernel/kprobes/common.h-96-extern int setup_detour_execution(struct kprobe *p, struct pt_regs *regs, int reenter);\narch/x86/kernel/kprobes/common.h=97=extern unsigned long __recover_optprobed_insn(kprobe_opcode_t *buf, unsigned long addr);\narch/x86/kernel/kprobes/common.h:98:#else\t/* !CONFIG_OPTPROBES */\narch/x86/kernel/kprobes/common.h-99-static inline int setup_detour_execution(struct kprobe *p, struct pt_regs *regs, int reenter)\n"
} |
| 45/2 | 2026/09/24 04:20 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 83219 output: 18 thoughts: 47 Thoughts: Okay, here's my take on those thoughts, tailored for an expert audience: **Analyzing the Dependency Chain for Optimal Probe Configuration** Right, so I see `CONFIG_OPTPROBES` is flagged, and immediately my mind goes to the prerequisite dependencies. It's explicitly tied to both `KPROBES` and `HAVE_OPTPROBES`. Makes perfect sense. I need to confirm the underlying mechanism is present. Now, my focus shifts to the source code and configuration. I need to quickly ascertain the roles of `OPTPROBES`. I'll want to check either `kernel/kprobes.c` – that's the obvious place – or the Kconfig files related to probes, where the configuration options are managed. Checking those will let me understand how the functionality is actually used. That's the first step in assessing its validity and relevance in this context. |
| 46/2 | 2026/09/24 04:20 | tool | grepper |
0mArgs:{
"Expression": "config OPTPROBES"
}Results: {
"Output": "arch/Kconfig=176=config STATIC_CALL_SELFTEST\n--\narch/Kconfig-181-\narch/Kconfig:182:config OPTPROBES\narch/Kconfig-183-\tdef_bool y\n"
} |
| 47/2 | 2026/09/24 04:20 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 83539 output: 32 thoughts: 19 Thoughts: Here's my summary, as though I'm thinking through this process: **Diving into the OPTPROBES Dependencies** Okay, let's get down to brass tacks. I need to understand the relationships surrounding `OPTPROBES` in the `arch/Kconfig` configuration file. Where does this thing live? What other options does it lean on? And critically, what are the implications for the kernel build process if I enable or disable it? I need to quickly grasp the context, so I can predict any knock-on effects that might occur. Understanding the dependencies is the key to managing configuration complexity, and to prevent any future build surprises. It's time to dig into the dependencies! |
| 48/2 | 2026/09/24 04:21 | tool | read-file |
0mArgs:{
"File": "arch/Kconfig",
"FirstLine": 180,
"LineCount": 15
}Results: {
"Contents": " 180:\t\t Boot time self-test of the call patching code.\n 181:\t\n 182:\tconfig OPTPROBES\n 183:\t\tdef_bool y\n 184:\t\tdepends on KPROBES \u0026\u0026 HAVE_OPTPROBES\n 185:\t\tselect NEED_TASKS_RCU\n 186:\t\n 187:\tconfig KPROBES_ON_FTRACE\n 188:\t\tdef_bool y\n 189:\t\tdepends on KPROBES \u0026\u0026 HAVE_KPROBES_ON_FTRACE\n 190:\t\tdepends on DYNAMIC_FTRACE_WITH_REGS\n 191:\t\thelp\n 192:\t\t If function tracer is enabled and the arch supports full\n 193:\t\t passing of pt_regs to function tracing, then kprobes can\n 194:\t\t optimize on top of function tracing.\n"
} |
| 49/2 | 2026/09/24 04:21 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 83970 output: 25 thoughts: 303 Thoughts: **Analyzing RCU and Tracepoint Dependencies** Okay, let's break this down. Line 185 is jumping out at me: `OPTPROBES` also pulls in `NEED_TASKS_RCU`! That's the crux. So, here's how this unfolds: if `KPROBES` is on, then `OPTPROBES` is *automatically* enabled (since x86 architecture, in general, selects `HAVE_OPTPROBES`), which in turn *automatically* selects `NEED_TASKS_RCU`. Next, if we have `DYNAMIC_FTRACE` enabled, and we're on a `SMP` system with `X86_64` architecture, then we automatically get `HAVE_RCU_TRAMPOLINE_READERS` selected. This is critical. Now, if we have *either* `PREEMPTION`, or `PREEMPT` / `PREEMPT_DYNAMIC`, enabled, `TASKS_RCU` is on the list. And, here's the kicker: then we *also* get `TASKS_RCU_TRAMPOLINE_READERS` enabled! Good. Let's make sure I'm solid on this condition. Let me just double-check the enabling conditions for `TASKS_RCU_TRAMPOLINE_READERS` again: `TASKS_RCU && HAVE_RCU_TRAMPOLINE_READERS && GENERIC_IRQ_ENTRY && !NEED_SRCU_NMI_SAFE`. Yep, that aligns with what I've got so far. Alright, what about direct calls with `DYNAMIC_FTRACE`? I wonder if we need `DYNAMIC_FTRACE_WITH_DIRECT_CALLS` enabled. Let's look at `kernel/trace/ftrace.c`. The `register_ftrace_direct()` function uses `ftrace_direct_mark_module(addr)`. So, the next step is definitely to investigate the role and need for `CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS`. |
| 50/2 | 2026/09/24 04:21 | tool | grepper |
0mArgs:{
"Expression": "DYNAMIC_FTRACE_WITH_DIRECT_CALLS"
}Results: {
"Output": "arch/arm64/Kconfig=2=config ARM64\n--\narch/arm64/Kconfig-189-\t\t CLANG_SUPPORTS_DYNAMIC_FTRACE_WITH_ARGS)\narch/arm64/Kconfig:190:\tselect HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS \\\narch/arm64/Kconfig-191-\t\tif DYNAMIC_FTRACE_WITH_ARGS\n--\narch/arm64/include/asm/ftrace.h=71=struct __arch_ftrace_regs {\n--\narch/arm64/include/asm/ftrace.h-74-\narch/arm64/include/asm/ftrace.h:75:#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\narch/arm64/include/asm/ftrace.h-76-\tunsigned long direct_tramp;\n--\narch/arm64/include/asm/ftrace.h=172=void ftrace_graph_func(unsigned long ip, unsigned long parent_ip,\n--\narch/arm64/include/asm/ftrace.h-175-\narch/arm64/include/asm/ftrace.h:176:#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\narch/arm64/include/asm/ftrace.h-177-static inline void arch_ftrace_set_direct_caller(struct ftrace_regs *fregs,\n--\narch/arm64/include/asm/ftrace.h-185-}\narch/arm64/include/asm/ftrace.h:186:#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */\narch/arm64/include/asm/ftrace.h-187-\n--\narch/arm64/kernel/asm-offsets.c=28=int main(void)\n--\narch/arm64/kernel/asm-offsets.c-100- DEFINE(FREGS_PC,\t\toffsetof(struct __arch_ftrace_regs, pc));\narch/arm64/kernel/asm-offsets.c:101:#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\narch/arm64/kernel/asm-offsets.c-102- DEFINE(FREGS_DIRECT_TRAMP,\toffsetof(struct __arch_ftrace_regs, direct_tramp));\n--\narch/arm64/kernel/asm-offsets.c-189- BLANK();\narch/arm64/kernel/asm-offsets.c:190:#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\narch/arm64/kernel/asm-offsets.c-191- DEFINE(FTRACE_OPS_DIRECT_CALL,\toffsetof(struct ftrace_ops, direct_call));\n--\narch/arm64/kernel/entry-ftrace.S=102=SYM_CODE_START(ftrace_caller)\n--\narch/arm64/kernel/entry-ftrace.S-119-\narch/arm64/kernel/entry-ftrace.S:120:#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\narch/arm64/kernel/entry-ftrace.S-121-\t/*\n--\narch/arm64/kernel/entry-ftrace.S-142-\narch/arm64/kernel/entry-ftrace.S:143:#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\narch/arm64/kernel/entry-ftrace.S-144-\tstr\txzr, [sp, #FREGS_DIRECT_TRAMP]\n--\narch/arm64/kernel/entry-ftrace.S=178=SYM_INNER_LABEL(ftrace_call, SYM_L_GLOBAL)\n--\narch/arm64/kernel/entry-ftrace.S-198-\narch/arm64/kernel/entry-ftrace.S:199:#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\narch/arm64/kernel/entry-ftrace.S-200-\tldr\tx17, [sp, #FREGS_DIRECT_TRAMP]\n--\narch/arm64/kernel/entry-ftrace.S-212-\narch/arm64/kernel/entry-ftrace.S:213:#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\narch/arm64/kernel/entry-ftrace.S-214-SYM_INNER_LABEL(ftrace_caller_direct_late, SYM_L_LOCAL)\n--\narch/arm64/kernel/entry-ftrace.S=231=SYM_INNER_LABEL(ftrace_caller_direct, SYM_L_LOCAL)\n--\narch/arm64/kernel/entry-ftrace.S-238-\tbr\tx17\narch/arm64/kernel/entry-ftrace.S:239:#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */\narch/arm64/kernel/entry-ftrace.S-240-SYM_CODE_END(ftrace_caller)\narch/arm64/kernel/entry-ftrace.S-241-\narch/arm64/kernel/entry-ftrace.S:242:#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\narch/arm64/kernel/entry-ftrace.S-243-SYM_CODE_START(ftrace_stub_direct_tramp)\n--\narch/arm64/kernel/entry-ftrace.S=248=SYM_CODE_END(ftrace_stub_direct_tramp)\narch/arm64/kernel/entry-ftrace.S:249:#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */\narch/arm64/kernel/entry-ftrace.S-250-\n--\narch/arm64/kernel/ftrace.c=413=int ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr)\n--\narch/arm64/kernel/ftrace.c-432-#if defined(CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS) || \\\narch/arm64/kernel/ftrace.c:433:\tdefined(CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS)\narch/arm64/kernel/ftrace.c-434-int ftrace_modify_call(struct dyn_ftrace *rec, unsigned long old_addr,\n--\narch/loongarch/Kconfig=2=config LOONGARCH\n--\narch/loongarch/Kconfig-142-\tselect HAVE_FTRACE_REGS_HAVING_PT_REGS\narch/loongarch/Kconfig:143:\tselect HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\narch/loongarch/Kconfig-144-\tselect HAVE_DYNAMIC_FTRACE_WITH_REGS\n--\narch/loongarch/include/asm/ftrace.h=71=void ftrace_graph_func(unsigned long ip, unsigned long parent_ip,\n--\narch/loongarch/include/asm/ftrace.h-73-\narch/loongarch/include/asm/ftrace.h:74:#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\narch/loongarch/include/asm/ftrace.h-75-static inline void\narch/loongarch/include/asm/ftrace.h=76=__arch_ftrace_set_direct_caller(struct pt_regs *regs, unsigned long addr)\n--\narch/loongarch/include/asm/ftrace.h-82-\t__arch_ftrace_set_direct_caller(\u0026arch_ftrace_regs(fregs)-\u003eregs, addr)\narch/loongarch/include/asm/ftrace.h:83:#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */\narch/loongarch/include/asm/ftrace.h-84-\n--\narch/loongarch/kernel/mcount_dyn.S=162=SYM_CODE_END(return_to_handler)\n--\narch/loongarch/kernel/mcount_dyn.S-164-\narch/loongarch/kernel/mcount_dyn.S:165:#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\narch/loongarch/kernel/mcount_dyn.S-166-SYM_CODE_START(ftrace_stub_direct_tramp)\n--\narch/loongarch/kernel/mcount_dyn.S=171=SYM_CODE_END(ftrace_stub_direct_tramp)\narch/loongarch/kernel/mcount_dyn.S:172:#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */\n--\narch/powerpc/Kconfig=118=config PPC\n--\narch/powerpc/Kconfig-244-\tselect HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS if PPC_FTRACE_OUT_OF_LINE || (PPC32 \u0026\u0026 ARCH_USING_PATCHABLE_FUNCTION_ENTRY)\narch/powerpc/Kconfig:245:\tselect HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS if HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS\narch/powerpc/Kconfig-246-\tselect HAVE_DYNAMIC_FTRACE_WITH_REGS\tif ARCH_USING_PATCHABLE_FUNCTION_ENTRY || MPROFILE_KERNEL || PPC32\n--\narch/powerpc/Kconfig-289-\tselect HAVE_RUST\t\t\tif PPC64 \u0026\u0026 CPU_LITTLE_ENDIAN\narch/powerpc/Kconfig:290:\tselect HAVE_SAMPLE_FTRACE_DIRECT\tif HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\narch/powerpc/Kconfig:291:\tselect HAVE_SAMPLE_FTRACE_DIRECT_MULTI\tif HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\narch/powerpc/Kconfig-292-\tselect HAVE_SETUP_PER_CPU_AREA\t\tif PPC64\n--\narch/powerpc/include/asm/ftrace.h=157=unsigned long ftrace_call_adjust(unsigned long addr);\narch/powerpc/include/asm/ftrace.h-158-\narch/powerpc/include/asm/ftrace.h:159:#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\narch/powerpc/include/asm/ftrace.h-160-/*\n--\narch/powerpc/include/asm/ftrace.h=167=static inline void arch_ftrace_set_direct_caller(struct ftrace_regs *fregs, unsigned long addr)\n--\narch/powerpc/include/asm/ftrace.h-172-}\narch/powerpc/include/asm/ftrace.h:173:#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */\narch/powerpc/include/asm/ftrace.h-174-#else\n--\narch/powerpc/kernel/asm-offsets.c=78=int main(void)\n--\narch/powerpc/kernel/asm-offsets.c-681-\tOFFSET(FTRACE_OPS_FUNC, ftrace_ops, func);\narch/powerpc/kernel/asm-offsets.c:682:#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\narch/powerpc/kernel/asm-offsets.c-683-\tOFFSET(FTRACE_OPS_DIRECT_CALL, ftrace_ops, direct_call);\n--\narch/powerpc/kernel/trace/ftrace.c=159=static int ftrace_get_call_inst(struct dyn_ftrace *rec, unsigned long addr, ppc_inst_t *call_inst)\n--\narch/powerpc/kernel/trace/ftrace.c-171-\t\t/* This can only happen with ftrace direct */\narch/powerpc/kernel/trace/ftrace.c:172:\t\tif (!IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS)) {\narch/powerpc/kernel/trace/ftrace.c-173-\t\t\tpr_err(\"0x%lx (0x%lx): Unexpected target address 0x%lx\\n\",\n--\narch/powerpc/kernel/trace/ftrace_entry.S-48-\narch/powerpc/kernel/trace/ftrace_entry.S:49:#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\narch/powerpc/kernel/trace/ftrace_entry.S-50-\t/* Load the ftrace_op */\n--\narch/powerpc/kernel/trace/ftrace_entry.S-131-\narch/powerpc/kernel/trace/ftrace_entry.S:132:#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\narch/powerpc/kernel/trace/ftrace_entry.S-133-\t/* Clear orig_gpr3 to later detect ftrace_direct call */\n--\narch/powerpc/kernel/trace/ftrace_entry.S-181-.macro\tftrace_regs_exit allregs\narch/powerpc/kernel/trace/ftrace_entry.S:182:#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\narch/powerpc/kernel/trace/ftrace_entry.S-183-\t/* Check orig_gpr3 to detect ftrace_direct call */\n--\narch/powerpc/kernel/trace/ftrace_entry.S-198-\narch/powerpc/kernel/trace/ftrace_entry.S:199:#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\narch/powerpc/kernel/trace/ftrace_entry.S-200-\tbeq\tcr1,2f\n--\narch/powerpc/kernel/trace/ftrace_entry.S-245-#ifdef CONFIG_PPC_FTRACE_OUT_OF_LINE\narch/powerpc/kernel/trace/ftrace_entry.S:246:#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\narch/powerpc/kernel/trace/ftrace_entry.S-247-\tbnectr\tcr1\n--\narch/powerpc/kernel/trace/ftrace_entry.S=289=ftrace_no_trace:\n--\narch/powerpc/kernel/trace/ftrace_entry.S-303-\narch/powerpc/kernel/trace/ftrace_entry.S:304:#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\narch/powerpc/kernel/trace/ftrace_entry.S-305-.Lftrace_direct_call_regs:\n--\narch/riscv/Kconfig=13=config RISCV\n--\narch/riscv/Kconfig-163-\tselect FUNCTION_ALIGNMENT_4B if DYNAMIC_FTRACE \u0026\u0026 RISCV_ISA_C\narch/riscv/Kconfig:164:\tselect HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS if HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS\narch/riscv/Kconfig-165-\tselect HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS if (DYNAMIC_FTRACE_WITH_ARGS \u0026\u0026 !CFI)\n--\narch/riscv/include/asm/ftrace.h=127=struct __arch_ftrace_regs {\n--\narch/riscv/include/asm/ftrace.h-132-\tunsigned long t1;\narch/riscv/include/asm/ftrace.h:133:#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\narch/riscv/include/asm/ftrace.h-134-\tunsigned long direct_tramp;\n--\narch/riscv/include/asm/ftrace.h=225=void ftrace_graph_func(unsigned long ip, unsigned long parent_ip,\n--\narch/riscv/include/asm/ftrace.h-228-\narch/riscv/include/asm/ftrace.h:229:#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\narch/riscv/include/asm/ftrace.h-230-static inline void arch_ftrace_set_direct_caller(struct ftrace_regs *fregs, unsigned long addr)\n--\narch/riscv/include/asm/ftrace.h-233-}\narch/riscv/include/asm/ftrace.h:234:#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */\narch/riscv/include/asm/ftrace.h-235-\n--\narch/riscv/kernel/asm-offsets.c=22=void asm_offsets(void)\n--\narch/riscv/kernel/asm-offsets.c-507-\tDEFINE(FTRACE_OPS_FUNC,\t\toffsetof(struct ftrace_ops, func));\narch/riscv/kernel/asm-offsets.c:508:#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\narch/riscv/kernel/asm-offsets.c-509-\tDEFINE(FTRACE_OPS_DIRECT_CALL,\toffsetof(struct ftrace_ops, direct_call));\narch/riscv/kernel/asm-offsets.c:510:#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */\narch/riscv/kernel/asm-offsets.c-511-#endif\n--\narch/riscv/kernel/ftrace.c=208=unsigned long ftrace_call_adjust(unsigned long addr)\n--\narch/riscv/kernel/ftrace.c-213-\narch/riscv/kernel/ftrace.c:214:#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\narch/riscv/kernel/ftrace.c-215-int ftrace_modify_call(struct dyn_ftrace *rec, unsigned long old_addr,\n--\narch/riscv/kernel/mcount-dyn.S=154=SYM_FUNC_START(ftrace_caller)\n--\narch/riscv/kernel/mcount-dyn.S-175-\narch/riscv/kernel/mcount-dyn.S:176:#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\narch/riscv/kernel/mcount-dyn.S-177-\t/*\n--\narch/riscv/kernel/mcount-dyn.S=191=SYM_INNER_LABEL(ftrace_call, SYM_L_GLOBAL)\n--\narch/riscv/kernel/mcount-dyn.S-195-\tRESTORE_ABI_REGS\narch/riscv/kernel/mcount-dyn.S:196:#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\narch/riscv/kernel/mcount-dyn.S-197-\tbnez\tt1, ftrace_caller_direct\n--\narch/riscv/kernel/mcount-dyn.S-199-\tjr\tt0\narch/riscv/kernel/mcount-dyn.S:200:#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\narch/riscv/kernel/mcount-dyn.S-201-SYM_INNER_LABEL(ftrace_caller_direct, SYM_L_LOCAL)\n--\narch/riscv/kernel/mcount-dyn.S=204=SYM_FUNC_END(ftrace_caller)\narch/riscv/kernel/mcount-dyn.S-205-\narch/riscv/kernel/mcount-dyn.S:206:#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\narch/riscv/kernel/mcount-dyn.S-207-SYM_CODE_START(ftrace_stub_direct_tramp)\n--\narch/riscv/kernel/mcount-dyn.S=209=SYM_CODE_END(ftrace_stub_direct_tramp)\narch/riscv/kernel/mcount-dyn.S:210:#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */\n--\narch/s390/Kconfig=78=config S390\n--\narch/s390/Kconfig-211-\tselect HAVE_FTRACE_REGS_HAVING_PT_REGS\narch/s390/Kconfig:212:\tselect HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\narch/s390/Kconfig-213-\tselect HAVE_DYNAMIC_FTRACE_WITH_REGS\n--\narch/s390/include/asm/ftrace.h=82=ftrace_regs_get_return_address(const struct ftrace_regs *fregs)\n--\narch/s390/include/asm/ftrace.h-92-\narch/s390/include/asm/ftrace.h:93:#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\narch/s390/include/asm/ftrace.h-94-/*\n--\narch/s390/include/asm/ftrace.h=101=static inline void arch_ftrace_set_direct_caller(struct ftrace_regs *fregs, unsigned long addr)\n--\narch/s390/include/asm/ftrace.h-105-}\narch/s390/include/asm/ftrace.h:106:#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */\narch/s390/include/asm/ftrace.h-107-\n--\narch/x86/Kconfig=58=config X86\n--\narch/x86/Kconfig-233-\tselect HAVE_FTRACE_REGS_HAVING_PT_REGS\tif X86_64\narch/x86/Kconfig:234:\tselect HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\narch/x86/Kconfig-235-\tselect HAVE_DYNAMIC_FTRACE_WITH_JMP\tif X86_64\n--\narch/x86/Kconfig-342-\tselect ARCH_SUPPORTS_SYSCALL_USER_DISPATCH\narch/x86/Kconfig:343:\tselect HAVE_SINGLE_FTRACE_DIRECT_OPS\tif X86_64 \u0026\u0026 DYNAMIC_FTRACE_WITH_DIRECT_CALLS\narch/x86/Kconfig-344-\n--\narch/x86/include/asm/ftrace.h=83=void ftrace_graph_func(unsigned long ip, unsigned long parent_ip,\n--\narch/x86/include/asm/ftrace.h-88-\narch/x86/include/asm/ftrace.h:89:#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\narch/x86/include/asm/ftrace.h-90-/*\n--\narch/x86/include/asm/ftrace.h=98=__arch_ftrace_set_direct_caller(struct pt_regs *regs, unsigned long addr)\n--\narch/x86/include/asm/ftrace.h-104-\t__arch_ftrace_set_direct_caller(\u0026arch_ftrace_regs(fregs)-\u003eregs, addr)\narch/x86/include/asm/ftrace.h:105:#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */\narch/x86/include/asm/ftrace.h-106-\n--\ndrivers/hid/bpf/Kconfig=4=config HID_BPF\n--\ndrivers/hid/bpf/Kconfig-7-\tdepends on BPF_SYSCALL\ndrivers/hid/bpf/Kconfig:8:\tdepends on DYNAMIC_FTRACE_WITH_DIRECT_CALLS\ndrivers/hid/bpf/Kconfig-9-\thelp\n--\ninclude/linux/ftrace.h=449=struct ftrace_ops {\n--\ninclude/linux/ftrace.h-464-\tstruct ftrace_ops\t\t*managed;\ninclude/linux/ftrace.h:465:#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\ninclude/linux/ftrace.h-466-\tunsigned long\t\t\tdirect_call;\n--\ninclude/linux/ftrace.h=536=struct ftrace_func_entry {\n--\ninclude/linux/ftrace.h-541-\ninclude/linux/ftrace.h:542:#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\ninclude/linux/ftrace.h-543-unsigned long ftrace_find_rec_direct(unsigned long ip);\n--\ninclude/linux/ftrace.h=615=static inline void arch_ftrace_set_direct_caller(struct ftrace_regs *fregs,\ninclude/linux/ftrace.h-616-\t\t\t\t\t\t unsigned long addr) { }\ninclude/linux/ftrace.h:617:#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */\ninclude/linux/ftrace.h-618-\n--\ninclude/linux/ftrace.h=1018=extern int ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr);\n--\ninclude/linux/ftrace.h-1021-\tdefined(CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS) || \\\ninclude/linux/ftrace.h:1022:\tdefined(CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS)\ninclude/linux/ftrace.h-1023-/**\n--\nkernel/bpf/trampoline.c=62=struct bpf_trampoline_ops {\n--\nkernel/bpf/trampoline.c-68-\nkernel/bpf/trampoline.c:69:#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\nkernel/bpf/trampoline.c-70-static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mutex,\n--\nkernel/bpf/trampoline.c=210=void bpf_image_ksym_del(struct bpf_ksym *ksym)\n--\nkernel/bpf/trampoline.c-216-\nkernel/bpf/trampoline.c:217:#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\nkernel/bpf/trampoline.c-218-#ifdef CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS\n--\nkernel/bpf/trampoline.c=367=static int direct_ops_mod(struct bpf_trampoline *tr, void *ptr, bool lock_direct_mutex)\n--\nkernel/bpf/trampoline.c-370-}\nkernel/bpf/trampoline.c:371:#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */\nkernel/bpf/trampoline.c-372-\n--\nkernel/bpf/trampoline.c=710=static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mutex,\n--\nkernel/bpf/trampoline.c-743-\nkernel/bpf/trampoline.c:744:#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\nkernel/bpf/trampoline.c-745-again:\n--\nkernel/bpf/trampoline.c-796-\nkernel/bpf/trampoline.c:797:#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\nkernel/bpf/trampoline.c-798-\tif (err == -EAGAIN) {\n--\nkernel/bpf/trampoline.c=1514=int __weak arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,\n--\nkernel/bpf/trampoline.c-1519-\nkernel/bpf/trampoline.c:1520:#if defined(CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS) \u0026\u0026 \\\nkernel/bpf/trampoline.c-1521- defined(CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS) \u0026\u0026 \\\n--\nkernel/bpf/trampoline.c=1769=void bpf_trampoline_multi_detach(struct bpf_prog *prog,\n--\nkernel/bpf/trampoline.c-1804-\nkernel/bpf/trampoline.c:1805:#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS \u0026\u0026\nkernel/bpf/trampoline.c-1806-\t CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS \u0026\u0026\n--\nkernel/trace/Kconfig=47=config HAVE_DYNAMIC_FTRACE_WITH_REGS\n--\nkernel/trace/Kconfig-49-\nkernel/trace/Kconfig:50:config HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\nkernel/trace/Kconfig-51-\tbool\n--\nkernel/trace/Kconfig=324=config DYNAMIC_FTRACE_WITH_REGS\n--\nkernel/trace/Kconfig-328-\nkernel/trace/Kconfig:329:config DYNAMIC_FTRACE_WITH_DIRECT_CALLS\nkernel/trace/Kconfig-330-\tdef_bool y\nkernel/trace/Kconfig-331-\tdepends on DYNAMIC_FTRACE_WITH_REGS || DYNAMIC_FTRACE_WITH_ARGS\nkernel/trace/Kconfig:332:\tdepends on HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\nkernel/trace/Kconfig-333-\n--\nkernel/trace/Kconfig=343=config DYNAMIC_FTRACE_WITH_JMP\n--\nkernel/trace/Kconfig-345-\tdepends on DYNAMIC_FTRACE\nkernel/trace/Kconfig:346:\tdepends on DYNAMIC_FTRACE_WITH_DIRECT_CALLS\nkernel/trace/Kconfig-347-\tdepends on HAVE_DYNAMIC_FTRACE_WITH_JMP\n--\nkernel/trace/bpf_trace.c=3680=__bpf_kfunc_end_defs();\nkernel/trace/bpf_trace.c-3681-\nkernel/trace/bpf_trace.c:3682:#if defined(CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS) \u0026\u0026 \\\nkernel/trace/bpf_trace.c-3683- defined(CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS)\n--\nkernel/trace/bpf_trace.c=3930=int bpf_tracing_multi_attach(struct bpf_prog *prog, const union bpf_attr *attr)\n--\nkernel/trace/bpf_trace.c-3934-\nkernel/trace/bpf_trace.c:3935:#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS \u0026\u0026 CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS */\n--\nkernel/trace/fgraph.c=563=ftrace_push_return_trace(unsigned long ret, unsigned long func,\n--\nkernel/trace/fgraph.c-644-/* Make sure this only works without direct calls */\nkernel/trace/fgraph.c:645:# ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\nkernel/trace/fgraph.c-646-# error MCOUNT_INSN_SIZE not defined with direct calls enabled\n--\nkernel/trace/ftrace.c=2626=ftrace_find_unique_ops(struct dyn_ftrace *rec)\n--\nkernel/trace/ftrace.c-2643-\nkernel/trace/ftrace.c:2644:#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\nkernel/trace/ftrace.c-2645-/* Protected by rcu_tasks for reading, and direct_mutex for writing */\n--\nkernel/trace/ftrace.c=2665=static void call_direct_funcs(unsigned long ip, unsigned long pip,\n--\nkernel/trace/ftrace.c-2679-}\nkernel/trace/ftrace.c:2680:#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */\nkernel/trace/ftrace.c-2681-\n--\nkernel/trace/ftrace.c=6012=ftrace_set_addr(struct ftrace_ops *ops, unsigned long *ips, unsigned int cnt,\n--\nkernel/trace/ftrace.c-6017-\nkernel/trace/ftrace.c:6018:#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\nkernel/trace/ftrace.c-6019-\n--\nkernel/trace/ftrace.c=6678=int update_ftrace_direct_mod(struct ftrace_ops *ops, struct ftrace_hash *hash, bool do_direct_lock)\n--\nkernel/trace/ftrace.c-6759-\nkernel/trace/ftrace.c:6760:#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */\nkernel/trace/ftrace.c-6761-\n--\nkernel/trace/ftrace.c=7839=static int ftrace_get_trampoline_kallsym(unsigned int symnum,\n--\nkernel/trace/ftrace.c-7859-\nkernel/trace/ftrace.c:7860:#if defined(CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS) || defined(CONFIG_MODULES)\nkernel/trace/ftrace.c-7861-/*\n--\nkernel/trace/ftrace.c=9145=int ftrace_is_dead(void)\n--\nkernel/trace/ftrace.c-9149-\nkernel/trace/ftrace.c:9150:#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\nkernel/trace/ftrace.c-9151-/*\n--\nkernel/trace/ftrace.c=9212=static void cleanup_direct_functions_after_ipmodify(struct ftrace_ops *ops)\n--\nkernel/trace/ftrace.c-9252-\nkernel/trace/ftrace.c:9253:#else /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */\nkernel/trace/ftrace.c-9254-\n--\nkernel/trace/ftrace.c=9260=static void cleanup_direct_functions_after_ipmodify(struct ftrace_ops *ops)\n--\nkernel/trace/ftrace.c-9266-\nkernel/trace/ftrace.c:9267:#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */\nkernel/trace/ftrace.c-9268-\n--\nkernel/trace/trace_selftest.c=1048=static struct fgraph_ops fgraph_ops __initdata = {\n--\nkernel/trace/trace_selftest.c-1052-\nkernel/trace/trace_selftest.c:1053:#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\nkernel/trace/trace_selftest.c-1054-static struct ftrace_ops direct;\n--\nkernel/trace/trace_selftest.c=1062=trace_selftest_startup_function_graph(struct tracer *trace,\n--\nkernel/trace/trace_selftest.c-1115-\nkernel/trace/trace_selftest.c:1116:#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS\nkernel/trace/trace_selftest.c-1117-\t/*\n--\nsamples/Kconfig=32=config SAMPLE_FTRACE_DIRECT\nsamples/Kconfig-33-\ttristate \"Build register_ftrace_direct() example\"\nsamples/Kconfig:34:\tdepends on DYNAMIC_FTRACE_WITH_DIRECT_CALLS \u0026\u0026 m\nsamples/Kconfig-35-\tdepends on HAVE_SAMPLE_FTRACE_DIRECT\n--\nsamples/Kconfig=40=config SAMPLE_FTRACE_DIRECT_MULTI\nsamples/Kconfig-41-\ttristate \"Build register_ftrace_direct() on multiple ips example\"\nsamples/Kconfig:42:\tdepends on DYNAMIC_FTRACE_WITH_DIRECT_CALLS \u0026\u0026 m\nsamples/Kconfig-43-\tdepends on HAVE_SAMPLE_FTRACE_DIRECT_MULTI\n--\ntools/testing/selftests/hid/config=12=CONFIG_DEBUG_INFO_BTF=y\ntools/testing/selftests/hid/config:13:CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS=y\ntools/testing/selftests/hid/config-14-CONFIG_FPROBE=y\n"
} |
| 51/2 | 2026/09/24 04:21 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 93751 output: 21 thoughts: 1091 Thoughts: **Identifying Relevant Configurations and Focus Symbols** Okay, let's break this down systematically. My initial thought process was to enumerate all the configuration options that *seem* relevant for enabling this new functionality on x86_64. I started by listing out `SMP`, `DYNAMIC_FTRACE`, `PREEMPTION`, `BPF_SYSCALL` (or TRACING, or FUNCTION_TRACER, or KPROBES/OPTPROBES), `BPF_JIT`, `KPROBES`, `OPTPROBES`, `DYNAMIC_FTRACE_WITH_DIRECT_CALLS`, and `PROVE_RCU`. Quite the list! Then, I paused and remembered the prompt's instruction to be specific and avoid assumptions. I needed to identify the *key* configs, not just any config that *could* be related. Section 5 of the prompt is pretty explicit: "Inspect Kconfig files and `#ifdef` guards; do not make assumptions" and strip the "CONFIG_" prefix. So, I re-evaluated, focusing on the dependencies. `TASKS_RCU_TRAMPOLINE_READERS` is the target, which relies on `TASKS_RCU`, `HAVE_RCU_TRAMPOLINE_READERS`, `GENERIC_IRQ_ENTRY`, and `!NEED_SRCU_NMI_SAFE`. Focusing on x86_64, I needed to figure out what explicitly needed to be enabled, and deduced `SMP`, `DYNAMIC_FTRACE`, `PREEMPTION` (or `PREEMPT`), `BPF_SYSCALL`, `BPF_JIT`, `KPROBES`, `OPTPROBES`, `DYNAMIC_FTRACE_WITH_DIRECT_CALLS`, and `PROVE_RCU`. I then realized that `HAVE_RCU_TRAMPOLINE_READERS` is *selected* automatically in `arch/x86/Kconfig` if `X86_64`, `SMP`, and `DYNAMIC_FTRACE` are set. Which simplifies things considerably. Also, `TASKS_RCU_TRAMPOLINE_READERS` is only enabled if `TASKS_RCU` is enabled, which is enabled by `NEED_TASKS_RCU && PREEMPTION`. Now, I shifted gears to identify the `FocusSymbols`. That's where I need to be more careful. I reread Section 4 of the prompt: I need to avoid listing generic, ubiquitous functions. The goal is to target specific feature logic, API entrypoints, etc. And be aware of static inline functions, and find non-static callers that exercise those inline functions. The key is in `kernel/rcu/tasks.h`. My immediate thought was to identify the functions in this file that are non-static or have distinct symbols. I'll need to figure out which functions are non-static and not ubiquitously used, and how `tasks.h` is included in the build. I'll need to grep for `tasks.h`. |
| 52/2 | 2026/09/24 04:21 | tool | grepper |
0mArgs:{
"Expression": "#include \"tasks\\.h\""
}Results: {
"Output": "kernel/rcu/update.c=677=void rcu_early_boot_tests(void) {}\n--\nkernel/rcu/update.c-679-\nkernel/rcu/update.c:680:#include \"tasks.h\"\nkernel/rcu/update.c-681-\n"
} |
| 53/2 | 2026/09/24 04:21 | llm | patch-evaluator |
1mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 95606 output: 30 thoughts: 1183 Thoughts: **Analyzing Potential Hotpaths in RCU Tasks Implementation** Okay, so I'm looking at the inclusion of `kernel/rcu/tasks.h` in `kernel/rcu/update.c`. My immediate thought is to understand the functions it exposes, specifically under `CONFIG_TASKS_RCU_TRAMPOLINE_READERS`. I see: `rcu_tasks_tramp_wait_gp`, `rcu_tasks_tramp_wait_cpus`, `rcu_tasks_tramp_wait_holdouts`, `rcu_tasks_irq_resched_enter`, `rcu_tasks_irq_resched_exit`, `rcu_tasks_note_qs`, `rcu_tasks_wait_irq_preempted`, `rcu_tasks_trampoline_text`, and `arch_rcu_tasks_trampoline_text`. My concern immediately zeroes in on `rcu_tasks_irq_resched_enter`. It's a global function and, if it's called on a hot path, this could be an issue for the fuzzer. My question: Where is `raw_irqentry_exit_cond_resched` called? As an expert, I know that function is called on *every* interrupt return. Ah, the issue is clear. Section 4 of the document specifically cautions against listing ubiquitous functions. I immediately know the rationale: listing a function called in a ubiquitous path, such as an interrupt return, would cause the fuzzer to classify a large number of unrelated tests as "focused", which dilutes the fuzzing effort. The concern is valid: `rcu_tasks_irq_resched_enter` is called by `raw_irqentry_exit_cond_resched`, which itself is called on every interrupt exit, so only executing when preemption is scheduled on irq exit. So, it's not *every* program, but it's on a *very* hot path. I need to be careful with this function. It's only executed when `need_resched()` is true when exiting an interrupt, and the `preempt_schedule_irq()` is performed. I'm going to want to focus my attention on the specific logic in other functions that implement the feature: I also see functions related to this trampoline read-racy Tasks RCU, namely `rcu_tasks_trampoline_text` and `arch_rcu_tasks_trampoline_text`. I also want to consider the other functions, like `rcu_tasks_wait_irq_preempted`, `rcu_tasks_tramp_wait_gp`, `rcu_tasks_tramp_wait_cpus`, `rcu_tasks_tramp_wait_holdouts`. Then, for unrelated functions, I'm noting: `kprobe_in_optimized_region`, `ftrace_direct_mark_module`, `arch_prepare_bpf_trampoline` and `arch_prepare_optimized_kprobe`. Now, I'm thinking about the bigger picture. I'll need to check the call sites of `synchronize_rcu_tasks()` and `call_rcu_tasks()` to see how ubiquitous *they* are. That will need a separate investigation. |
| 54/2 | 2026/09/24 04:22 | tool | grepper |
0mArgs:{
"Expression": "(synchronize_rcu_tasks|call_rcu_tasks)"
}Results: {
"Output": "\nFull output is too long, showing 500 out of 795 lines.\nUse more precise expression if possible.\n\nDocumentation/RCU/Design/Requirements/Requirements.rst=2736=is, calls to schedule(), cond_resched(), and\nDocumentation/RCU/Design/Requirements/Requirements.rst:2737:synchronize_rcu_tasks(). In addition, transitions to and from\nDocumentation/RCU/Design/Requirements/Requirements.rst-2738-userspace execution also delimit tasks-RCU read-side critical sections.\n--\nDocumentation/RCU/Design/Requirements/Requirements.rst=2750=The tasks-RCU API is quite compact, consisting only of\nDocumentation/RCU/Design/Requirements/Requirements.rst:2751:call_rcu_tasks(), synchronize_rcu_tasks(), and\nDocumentation/RCU/Design/Requirements/Requirements.rst-2752-rcu_barrier_tasks(). In ``CONFIG_PREEMPTION=n`` kernels, trampolines\n--\nDocumentation/RCU/Design/Requirements/Requirements.rst=2775=obligation moves to the trampolines: anything that relies on\nDocumentation/RCU/Design/Requirements/Requirements.rst:2776:synchronize_rcu_tasks() to protect code a task may be preempted in must\nDocumentation/RCU/Design/Requirements/Requirements.rst-2777-take the Tasks Trace reader (see register_ftrace_direct()), or, where\n--\nDocumentation/RCU/Design/Requirements/Requirements.rst=2800=The tasks-rude-RCU API is also reader-marking-free and thus quite compact,\nDocumentation/RCU/Design/Requirements/Requirements.rst:2801:consisting solely of synchronize_rcu_tasks_rude().\nDocumentation/RCU/Design/Requirements/Requirements.rst-2802-\n--\nDocumentation/RCU/Design/Requirements/Requirements.rst=2814=Now that the implementation is based on SRCU-fast, a call\nDocumentation/RCU/Design/Requirements/Requirements.rst:2815:to synchronize_rcu_tasks_trace() implies at least one call to\nDocumentation/RCU/Design/Requirements/Requirements.rst-2816-synchronize_rcu(), that is, every Tasks Trace RCU grace period contains\nDocumentation/RCU/Design/Requirements/Requirements.rst=2817=at least one plain vanilla RCU grace period. Should there ever\nDocumentation/RCU/Design/Requirements/Requirements.rst:2818:be a synchronize_rcu_tasks_trace_expedited(), this guarantee would\nDocumentation/RCU/Design/Requirements/Requirements.rst-2819-*not* necessarily apply to this hypothetical API member.\n--\nDocumentation/RCU/Design/Requirements/Requirements.rst=2822=consisting of rcu_read_lock_trace(), rcu_read_unlock_trace(),\nDocumentation/RCU/Design/Requirements/Requirements.rst:2823:rcu_read_lock_trace_held(), call_rcu_tasks_trace(),\nDocumentation/RCU/Design/Requirements/Requirements.rst:2824:synchronize_rcu_tasks_trace(), and rcu_barrier_tasks_trace().\nDocumentation/RCU/Design/Requirements/Requirements.rst-2825-\n--\nDocumentation/RCU/checklist.rst=12=over a rather long period of time, but improvements are always welcome!\n--\nDocumentation/RCU/checklist.rst-202-\nDocumentation/RCU/checklist.rst:203:5.\tIf any of call_rcu(), call_srcu(), call_rcu_tasks(), or\nDocumentation/RCU/checklist.rst:204:\tcall_rcu_tasks_trace() is used, the callback function may be\nDocumentation/RCU/checklist.rst-205-\tinvoked from softirq context, and in any case with bottom halves\n--\nDocumentation/RCU/checklist.rst-213-\tfor synchronize_srcu(), synchronize_rcu_expedited(),\nDocumentation/RCU/checklist.rst:214:\tsynchronize_srcu_expedited(), synchronize_rcu_tasks(),\nDocumentation/RCU/checklist.rst:215:\tsynchronize_rcu_tasks_rude(), and synchronize_rcu_tasks_trace().\nDocumentation/RCU/checklist.rst-216-\n--\nDocumentation/RCU/checklist.rst-251-\nDocumentation/RCU/checklist.rst:252:\ta.\tIf the updater uses synchronize_rcu_tasks() or\nDocumentation/RCU/checklist.rst:253:\t\tcall_rcu_tasks(), then the readers must refrain from\nDocumentation/RCU/checklist.rst-254-\t\texecuting voluntary context switches, that is, from\n--\nDocumentation/RCU/checklist.rst-261-\nDocumentation/RCU/checklist.rst:262:\tb.\tIf the updater uses call_rcu_tasks_trace()\nDocumentation/RCU/checklist.rst:263:\t\tor synchronize_rcu_tasks_trace(), then the\nDocumentation/RCU/checklist.rst-264-\t\tcorresponding readers must use rcu_read_lock_trace()\n--\nDocumentation/RCU/checklist.rst-266-\nDocumentation/RCU/checklist.rst:267:\tc.\tIf an updater uses synchronize_rcu_tasks_rude(),\nDocumentation/RCU/checklist.rst-268-\t\tthen the corresponding readers must use anything that\n--\nDocumentation/RCU/checklist.rst-338-\nDocumentation/RCU/checklist.rst:339:\tThe same cautions apply to call_srcu(), call_rcu_tasks(), and\nDocumentation/RCU/checklist.rst:340:\tcall_rcu_tasks_trace(). This is why there is an srcu_barrier(),\nDocumentation/RCU/checklist.rst-341-\trcu_barrier_tasks(), and rcu_barrier_tasks_trace(), respectively.\n--\nDocumentation/RCU/checklist.rst-395-\tCPU that executed the corresponding call_rcu(), call_srcu(),\nDocumentation/RCU/checklist.rst:396:\tcall_rcu_tasks(), or call_rcu_tasks_trace(). For example, if\nDocumentation/RCU/checklist.rst-397-\ta given CPU goes offline while having an RCU callback pending,\n--\nDocumentation/RCU/checklist.rst-525-17.\tIf you pass a callback function defined within a module\nDocumentation/RCU/checklist.rst:526:\tto one of call_rcu(), call_srcu(), call_rcu_tasks(), or\nDocumentation/RCU/checklist.rst:527:\tcall_rcu_tasks_trace(), then it is necessary to wait for all\nDocumentation/RCU/checklist.rst-528-\tpending callbacks to be invoked before unloading that module.\n--\nDocumentation/RCU/checklist.rst-538-\t-\tcall_srcu() -\u003e srcu_barrier()\nDocumentation/RCU/checklist.rst:539:\t-\tcall_rcu_tasks() -\u003e rcu_barrier_tasks()\nDocumentation/RCU/checklist.rst:540:\t-\tcall_rcu_tasks_trace() -\u003e rcu_barrier_tasks_trace()\nDocumentation/RCU/checklist.rst-541-\n--\nDocumentation/RCU/checklist.rst-554-\t\ttogether with and srcu_barrier()\nDocumentation/RCU/checklist.rst:555:\t-\tsynchronize_rcu_tasks() and rcu_barrier_tasks()\nDocumentation/RCU/checklist.rst-556-\t-\tsynchronize_tasks_trace() and rcu_barrier_tasks_trace()\n--\nDocumentation/RCU/whatisRCU.rst=1170=RCU-Tasks::\n--\nDocumentation/RCU/whatisRCU.rst-1173-\nDocumentation/RCU/whatisRCU.rst:1174:\tN/A\t\t\tcall_rcu_tasks\t\t\trcu_barrier_tasks\nDocumentation/RCU/whatisRCU.rst:1175:\t\t\t\tsynchronize_rcu_tasks\nDocumentation/RCU/whatisRCU.rst-1176-\n--\nDocumentation/RCU/whatisRCU.rst=1178=RCU-Tasks-Rude::\n--\nDocumentation/RCU/whatisRCU.rst-1181-\nDocumentation/RCU/whatisRCU.rst:1182:\tN/A\t\t\tsynchronize_rcu_tasks_rude\trcu_barrier_tasks_rude\nDocumentation/RCU/whatisRCU.rst:1183:\t\t\t\tcall_rcu_tasks_rude\nDocumentation/RCU/whatisRCU.rst-1184-\n--\nDocumentation/RCU/whatisRCU.rst=1186=RCU-Tasks-Trace::\n--\nDocumentation/RCU/whatisRCU.rst-1189-\nDocumentation/RCU/whatisRCU.rst:1190:\trcu_read_lock_trace\tcall_rcu_tasks_trace\t\trcu_barrier_tasks_trace\nDocumentation/RCU/whatisRCU.rst:1191:\trcu_read_unlock_trace\tsynchronize_rcu_tasks_trace\nDocumentation/RCU/whatisRCU.rst-1192-\tguard(rcu_tasks_trace)()\n--\nDocumentation/admin-guide/kernel-parameters.txt=98=Kernel parameters\n--\nDocumentation/admin-guide/kernel-parameters.txt-6465-\t\t\tSet timeout in milliseconds RCU Tasks asynchronous\nDocumentation/admin-guide/kernel-parameters.txt:6466:\t\t\tcallback batching for call_rcu_tasks().\nDocumentation/admin-guide/kernel-parameters.txt-6467-\t\t\tA negative value will take the default. A value\nDocumentation/admin-guide/kernel-parameters.txt-6468-\t\t\tof zero will disable batching.\tBatching is\nDocumentation/admin-guide/kernel-parameters.txt:6469:\t\t\talways disabled for synchronize_rcu_tasks().\nDocumentation/admin-guide/kernel-parameters.txt-6470-\n--\narch/x86/kernel/alternative.c=2523=void *text_poke_copy_locked(void *addr, const void *opcode, size_t len,\n--\narch/x86/kernel/alternative.c-2551- * new code blocks into unused regions of RX memory. Can be used in\narch/x86/kernel/alternative.c:2552: * conjunction with synchronize_rcu_tasks() to wait for existing\narch/x86/kernel/alternative.c-2553- * execution to quiesce after having made sure no existing functions\n--\ninclude/linux/rcupdate.h=201=void rcu_tasks_note_qs(struct task_struct *t, bool preempt);\n--\ninclude/linux/rcupdate.h-209-# endif\ninclude/linux/rcupdate.h:210:void call_rcu_tasks(struct rcu_head *head, rcu_callback_t func);\ninclude/linux/rcupdate.h:211:void synchronize_rcu_tasks(void);\ninclude/linux/rcupdate.h-212-void rcu_tasks_torture_stats_print(char *tt, char *tf);\n--\ninclude/linux/rcupdate.h-214-# define rcu_tasks_classic_qs(t, preempt) do { } while (0)\ninclude/linux/rcupdate.h:215:# define call_rcu_tasks call_rcu\ninclude/linux/rcupdate.h:216:# define synchronize_rcu_tasks synchronize_rcu\ninclude/linux/rcupdate.h-217-# endif\n--\ninclude/linux/rcupdate.h-221-# ifdef CONFIG_TASKS_RUDE_RCU\ninclude/linux/rcupdate.h:222:void synchronize_rcu_tasks_rude(void);\ninclude/linux/rcupdate.h-223-void rcu_tasks_rude_torture_stats_print(char *tt, char *tf);\n--\ninclude/linux/rcupdate.h=228=void exit_tasks_rcu_finish(void);\n--\ninclude/linux/rcupdate.h-232-#define rcu_note_voluntary_context_switch(t) do { } while (0)\ninclude/linux/rcupdate.h:233:#define call_rcu_tasks call_rcu\ninclude/linux/rcupdate.h:234:#define synchronize_rcu_tasks synchronize_rcu\ninclude/linux/rcupdate.h-235-static inline void exit_tasks_rcu_start(void) { }\n--\ninclude/linux/rcupdate_trace.h=28=static inline int rcu_read_lock_trace_held(void)\n--\ninclude/linux/rcupdate_trace.h-39- *\ninclude/linux/rcupdate_trace.h:40: * When synchronize_rcu_tasks_trace() is invoked by one task, then that\ninclude/linux/rcupdate_trace.h-41- * task is guaranteed to block until all other tasks exit their read-side\n--\ninclude/linux/rcupdate_trace.h=76=static inline void rcu_read_unlock_tasks_trace(struct srcu_ctr __percpu *scp)\n--\ninclude/linux/rcupdate_trace.h-86- *\ninclude/linux/rcupdate_trace.h:87: * When synchronize_rcu_tasks_trace() is invoked by one task, then that\ninclude/linux/rcupdate_trace.h-88- * task is guaranteed to block until all other tasks exit their read-side\n--\ninclude/linux/rcupdate_trace.h=129=static inline void rcu_read_unlock_trace(void)\n--\ninclude/linux/rcupdate_trace.h-149-/**\ninclude/linux/rcupdate_trace.h:150: * call_rcu_tasks_trace() - Queue a callback trace task-based grace period\ninclude/linux/rcupdate_trace.h-151- * @rhp: structure to be used for queueing the RCU updates.\n--\ninclude/linux/rcupdate_trace.h-162- */\ninclude/linux/rcupdate_trace.h:163:static inline void call_rcu_tasks_trace(struct rcu_head *rhp, rcu_callback_t func)\ninclude/linux/rcupdate_trace.h-164-{\n--\ninclude/linux/rcupdate_trace.h-168-/**\ninclude/linux/rcupdate_trace.h:169: * synchronize_rcu_tasks_trace - wait for a trace rcu-tasks grace period\ninclude/linux/rcupdate_trace.h-170- *\n--\ninclude/linux/rcupdate_trace.h-178- * tracing and other situations requiring manipulation of function preambles\ninclude/linux/rcupdate_trace.h:179: * and profiling hooks. The synchronize_rcu_tasks_trace() function is not\ninclude/linux/rcupdate_trace.h-180- * (yet) intended for heavy use from multiple CPUs.\n--\ninclude/linux/rcupdate_trace.h-184- */\ninclude/linux/rcupdate_trace.h:185:static inline void synchronize_rcu_tasks_trace(void)\ninclude/linux/rcupdate_trace.h-186-{\n--\ninclude/linux/rcupdate_trace.h-190-/**\ninclude/linux/rcupdate_trace.h:191: * rcu_barrier_tasks_trace - Wait for in-flight call_rcu_tasks_trace() callbacks.\ninclude/linux/rcupdate_trace.h-192- *\n--\ninclude/linux/rcupdate_trace.h=227=static inline unsigned long rcu_tasks_trace_batches_completed(void) { return 0; }\n--\ninclude/linux/rcupdate_trace.h-231- */\ninclude/linux/rcupdate_trace.h:232:static inline void call_rcu_tasks_trace(struct rcu_head *rhp, rcu_callback_t func) { BUG(); }\ninclude/linux/rcupdate_trace.h-233-static inline void rcu_read_lock_trace(void) { BUG(); }\n--\ninclude/linux/rcupdate_wait.h=29=do {\t\t\t\t\t\t\t\t\t\t\t\t\\\n--\ninclude/linux/rcupdate_wait.h-42- * This macro waits concurrently for multiple types of RCU grace periods.\ninclude/linux/rcupdate_wait.h:43: * For example, synchronize_rcu_mult(call_rcu, call_rcu_tasks) would wait\ninclude/linux/rcupdate_wait.h-44- * on concurrent RCU and RCU-tasks grace periods. Waiting on a given SRCU\n--\ninclude/linux/tracepoint.h=96=void for_each_tracepoint_in_module(struct module *mod,\n--\ninclude/linux/tracepoint.h-112- * - tracepoint_is_faultable() == false: call_srcu()\ninclude/linux/tracepoint.h:113: * - tracepoint_is_faultable() == true: call_rcu_tasks_trace()\ninclude/linux/tracepoint.h-114- */\n--\ninclude/linux/tracepoint.h=117=static inline void tracepoint_synchronize_unregister(void)\ninclude/linux/tracepoint.h-118-{\ninclude/linux/tracepoint.h:119:\tsynchronize_rcu_tasks_trace();\ninclude/linux/tracepoint.h-120-\tsynchronize_srcu(\u0026tracepoint_srcu);\n--\ninclude/linux/tracepoint.h=138=static inline void call_tracepoint_unregister_syscall(struct rcu_head *rcu, rcu_callback_t func)\ninclude/linux/tracepoint.h-139-{\ninclude/linux/tracepoint.h:140:\tcall_rcu_tasks_trace(rcu, func);\ninclude/linux/tracepoint.h-141-}\n--\nkernel/bpf/bpf_local_storage.c=111=static void bpf_local_storage_free(struct bpf_local_storage *local_storage,\n--\nkernel/bpf/bpf_local_storage.c-121-\nkernel/bpf/bpf_local_storage.c:122:\tcall_rcu_tasks_trace(\u0026local_storage-\u003ercu,\nkernel/bpf/bpf_local_storage.c-123-\t\t\t bpf_local_storage_free_trace_rcu);\n--\nkernel/bpf/bpf_local_storage.c=144=void bpf_selem_free(struct bpf_local_storage_elem *selem,\n--\nkernel/bpf/bpf_local_storage.c-157-\nkernel/bpf/bpf_local_storage.c:158:\tcall_rcu_tasks_trace(\u0026selem-\u003ercu, bpf_selem_free_trace_rcu);\nkernel/bpf/bpf_local_storage.c-159-}\n--\nkernel/bpf/bpf_struct_ops.c=1027=static void bpf_struct_ops_map_free(struct bpf_map *map)\n--\nkernel/bpf/bpf_struct_ops.c-1056-\t */\nkernel/bpf/bpf_struct_ops.c:1057:\tsynchronize_rcu_mult(call_rcu, call_rcu_tasks);\nkernel/bpf/bpf_struct_ops.c-1058-\n--\nkernel/bpf/core.c=2780=void bpf_prog_array_free_sleepable(struct bpf_prog_array *progs)\n--\nkernel/bpf/core.c-2783-\t\treturn;\nkernel/bpf/core.c:2784:\tcall_rcu_tasks_trace(\u0026progs-\u003ercu, __bpf_prog_array_free_sleepable_cb);\nkernel/bpf/core.c-2785-}\n--\nkernel/bpf/helpers.c=1242=static void bpf_async_cb_rcu_free(struct rcu_head *rcu)\n--\nkernel/bpf/helpers.c-1255-\nkernel/bpf/helpers.c:1256:/* Callback from call_rcu_tasks_trace, chains to call_rcu for final free */\nkernel/bpf/helpers.c-1257-static void bpf_async_cb_rcu_tasks_trace_free(struct rcu_head *rcu)\n--\nkernel/bpf/helpers.c-1285-\t\t */\nkernel/bpf/helpers.c:1286:\t\tcall_rcu_tasks_trace(\u0026cb-\u003ercu, bpf_async_cb_rcu_tasks_trace_free);\nkernel/bpf/helpers.c-1287-\t\treturn;\n--\nkernel/bpf/helpers.c=1294=static void worker_for_call_rcu(struct irq_work *work)\n--\nkernel/bpf/helpers.c-1297-\nkernel/bpf/helpers.c:1298:\tcall_rcu_tasks_trace(\u0026cb-\u003ercu, bpf_async_cb_rcu_tasks_trace_free);\nkernel/bpf/helpers.c-1299-}\n--\nkernel/bpf/helpers.c=1301=static void bpf_async_refcount_put(struct bpf_async_cb *cb)\n--\nkernel/bpf/helpers.c-1309-\t} else {\nkernel/bpf/helpers.c:1310:\t\tcall_rcu_tasks_trace(\u0026cb-\u003ercu, bpf_async_cb_rcu_tasks_trace_free);\nkernel/bpf/helpers.c-1311-\t}\n--\nkernel/bpf/memalloc.c=297=static void do_call_rcu_ttrace(struct bpf_mem_cache *c)\n--\nkernel/bpf/memalloc.c-318-\t/*\nkernel/bpf/memalloc.c:319:\t * Use call_rcu_tasks_trace() to wait for sleepable progs to finish.\nkernel/bpf/memalloc.c-320-\t * RCU Tasks Trace grace period implies RCU grace period, so pass\n--\nkernel/bpf/memalloc.c-322-\t */\nkernel/bpf/memalloc.c:323:\tcall_rcu_tasks_trace(\u0026c-\u003ercu_ttrace, __free_rcu);\nkernel/bpf/memalloc.c-324-}\n--\nkernel/bpf/memalloc.c=686=static void free_mem_alloc(struct bpf_mem_alloc *ma)\n--\nkernel/bpf/memalloc.c-691-\t *\nkernel/bpf/memalloc.c:692:\t * rcu_barrier_tasks_trace() doesn't imply synchronize_rcu_tasks_trace(),\nkernel/bpf/memalloc.c-693-\t * but rcu_barrier_tasks_trace() and rcu_barrier() below are only used\n--\nkernel/bpf/syscall.c=152=static void maybe_wait_bpf_programs(struct bpf_map *map)\n--\nkernel/bpf/syscall.c-156-\t * programs that could be running use the new map value. For sleepable\nkernel/bpf/syscall.c:157:\t * BPF programs, synchronize_rcu_tasks_trace() should be used to wait\nkernel/bpf/syscall.c-158-\t * for the completions of these programs, but considering the waiting\n--\nkernel/bpf/syscall.c=951=void bpf_map_put(struct bpf_map *map)\n--\nkernel/bpf/syscall.c-959-\t\tif (READ_ONCE(map-\u003efree_after_mult_rcu_gp))\nkernel/bpf/syscall.c:960:\t\t\tcall_rcu_tasks_trace(\u0026map-\u003ercu, bpf_map_free_rcu_gp);\nkernel/bpf/syscall.c-961-\t\telse if (READ_ONCE(map-\u003efree_after_rcu_gp))\n--\nkernel/bpf/syscall.c=2451=static void __bpf_prog_put_noref(struct bpf_prog *prog, bool deferred)\n--\nkernel/bpf/syscall.c-2464-\t\tif (prog-\u003esleepable)\nkernel/bpf/syscall.c:2465:\t\t\tcall_rcu_tasks_trace(\u0026prog-\u003eaux-\u003ercu, __bpf_prog_put_rcu);\nkernel/bpf/syscall.c-2466-\t\telse\n--\nkernel/bpf/syscall.c=3324=static void bpf_link_free(struct bpf_link *link)\n--\nkernel/bpf/syscall.c-3348-\t\t\t/* RCU Tasks Trace grace period implies RCU grace period. */\nkernel/bpf/syscall.c:3349:\t\t\tcall_rcu_tasks_trace(\u0026link-\u003ercu, bpf_link_defer_dealloc_rcu_gp);\nkernel/bpf/syscall.c-3350-\t\t/* We need to do a SRCU grace period wait for non-faultable tracepoint BPF links. */\n--\nkernel/bpf/trampoline.c=590=static void __bpf_tramp_image_release(struct percpu_ref *pcref)\n--\nkernel/bpf/trampoline.c-594-\tim = container_of(pcref, struct bpf_tramp_image, pcref);\nkernel/bpf/trampoline.c:595:\tcall_rcu_tasks(\u0026im-\u003ercu, __bpf_tramp_image_put_rcu);\nkernel/bpf/trampoline.c-596-}\n--\nkernel/bpf/trampoline.c=599=static void __bpf_tramp_image_put_rcu_tasks(struct rcu_head *rcu)\n--\nkernel/bpf/trampoline.c-608-\t\t/* the case of fentry trampoline */\nkernel/bpf/trampoline.c:609:\t\tcall_rcu_tasks(\u0026im-\u003ercu, __bpf_tramp_image_put_rcu);\nkernel/bpf/trampoline.c-610-}\n--\nkernel/bpf/trampoline.c=612=static void bpf_tramp_image_put(struct bpf_tramp_image *im)\n--\nkernel/bpf/trampoline.c-626-\t * executing or sleeping.\nkernel/bpf/trampoline.c:627:\t * In case of CONFIG_PREEMPT=y use call_rcu_tasks() to wait on\nkernel/bpf/trampoline.c-628-\t * first few asm instructions to execute and call into\n--\nkernel/bpf/trampoline.c-631-\t * function to finish.\nkernel/bpf/trampoline.c:632:\t * Then use call_rcu_tasks() to make sure few asm insns in\nkernel/bpf/trampoline.c-633-\t * the trampoline epilogue are done as well.\n--\nkernel/bpf/trampoline.c-637-\t * percpu_ref_kill will be waiting for. Hence the first\nkernel/bpf/trampoline.c:638:\t * call_rcu_tasks() is not necessary.\nkernel/bpf/trampoline.c-639-\t */\n--\nkernel/bpf/trampoline.c-645-\t\tif (IS_ENABLED(CONFIG_TASKS_RCU))\nkernel/bpf/trampoline.c:646:\t\t\tcall_rcu_tasks(\u0026im-\u003ercu, __bpf_tramp_image_put_rcu_tasks);\nkernel/bpf/trampoline.c-647-\t\telse\n--\nkernel/bpf/trampoline.c-653-\t * function and doesn't use percpu_ref.\nkernel/bpf/trampoline.c:654:\t * Use call_rcu_tasks_trace() to wait for sleepable progs to finish.\nkernel/bpf/trampoline.c:655:\t * Then use call_rcu_tasks() to wait for the rest of trampoline asm\nkernel/bpf/trampoline.c-656-\t * and normal progs.\nkernel/bpf/trampoline.c-657-\t */\nkernel/bpf/trampoline.c:658:\tcall_rcu_tasks_trace(\u0026im-\u003ercu, __bpf_tramp_image_put_rcu_tasks);\nkernel/bpf/trampoline.c-659-}\n--\nkernel/events/uprobes.c=668=static void uprobe_free_srcu(struct rcu_head *rcu)\n--\nkernel/events/uprobes.c-671-\nkernel/events/uprobes.c:672:\tcall_rcu_tasks_trace(\u0026uprobe-\u003ercu, uprobe_free_rcu_tasks_trace);\nkernel/events/uprobes.c-673-}\n--\nkernel/events/uprobes.c=1359=void uprobe_unregister_sync(void)\n--\nkernel/events/uprobes.c-1369-\t */\nkernel/events/uprobes.c:1370:\tsynchronize_rcu_tasks_trace();\nkernel/events/uprobes.c-1371-\tsynchronize_srcu(\u0026uretprobes_srcu);\n--\nkernel/kprobes.c=519=static bool kprobe_optimizer_waiting;\n--\nkernel/kprobes.c-525- *\nkernel/kprobes.c:526: * kprobe_optimizer() relies on synchronize_rcu_tasks() to wait for tasks that\nkernel/kprobes.c-527- * were interrupted on an instruction boundary inside the region about to be\n--\nkernel/kprobes.c=668=static void kprobe_optimizer(void)\n--\nkernel/kprobes.c-696-\t\trcu_tasks_wait_irq_preempted(kprobe_in_optimized_region);\nkernel/kprobes.c:697:\t\tsynchronize_rcu_tasks();\nkernel/kprobes.c-698-\t\tWRITE_ONCE(kprobe_optimizer_waiting, false);\n--\nkernel/kprobes.c=2847=static int __init init_optprobes(void)\n--\nkernel/kprobes.c-2850-\t * Enable kprobe optimization - this kicks the optimizer which\nkernel/kprobes.c:2851:\t * depends on synchronize_rcu_tasks() and ksoftirqd, that is\nkernel/kprobes.c-2852-\t * not spawned in early initcall. So delay the optimization.\n--\nkernel/rcu/rcuscale.c=316=static struct rcu_scale_ops tasks_ops = {\n--\nkernel/rcu/rcuscale.c-322-\t.gp_diff\t= rcu_seq_diff,\nkernel/rcu/rcuscale.c:323:\t.async\t\t= call_rcu_tasks,\nkernel/rcu/rcuscale.c-324-\t.gp_barrier\t= rcu_barrier_tasks,\nkernel/rcu/rcuscale.c:325:\t.sync\t\t= synchronize_rcu_tasks,\nkernel/rcu/rcuscale.c:326:\t.exp_sync\t= synchronize_rcu_tasks,\nkernel/rcu/rcuscale.c-327-\t.rso_gp_kthread\t= get_rcu_tasks_gp_kthread,\n--\nkernel/rcu/rcuscale.c=360=static struct rcu_scale_ops tasks_rude_ops = {\n--\nkernel/rcu/rcuscale.c-366-\t.gp_diff\t= rcu_seq_diff,\nkernel/rcu/rcuscale.c:367:\t.sync\t\t= synchronize_rcu_tasks_rude,\nkernel/rcu/rcuscale.c:368:\t.exp_sync\t= synchronize_rcu_tasks_rude,\nkernel/rcu/rcuscale.c-369-\t.rso_gp_kthread\t= get_rcu_tasks_rude_gp_kthread,\n--\nkernel/rcu/rcuscale.c=399=static struct rcu_scale_ops tasks_tracing_ops = {\n--\nkernel/rcu/rcuscale.c-405-\t.gp_diff\t= rcu_seq_diff,\nkernel/rcu/rcuscale.c:406:\t.async\t\t= call_rcu_tasks_trace,\nkernel/rcu/rcuscale.c-407-\t.gp_barrier\t= rcu_barrier_tasks_trace,\nkernel/rcu/rcuscale.c:408:\t.sync\t\t= synchronize_rcu_tasks_trace,\nkernel/rcu/rcuscale.c:409:\t.exp_sync\t= synchronize_rcu_tasks_trace,\nkernel/rcu/rcuscale.c-410-\t.name\t\t= \"tasks-tracing\"\n--\nkernel/rcu/rcutorture.c=1164=static void rcu_tasks_torture_deferred_free(struct rcu_torture *p)\nkernel/rcu/rcutorture.c-1165-{\nkernel/rcu/rcutorture.c:1166:\tcall_rcu_tasks(\u0026p-\u003ertort_rcu, rcu_torture_cb);\nkernel/rcu/rcutorture.c-1167-}\n--\nkernel/rcu/rcutorture.c=1169=static void synchronize_rcu_mult_test(void)\nkernel/rcu/rcutorture.c-1170-{\nkernel/rcu/rcutorture.c:1171:\tsynchronize_rcu_mult(call_rcu_tasks, call_rcu);\nkernel/rcu/rcutorture.c-1172-}\n--\nkernel/rcu/rcutorture.c=1174=static struct rcu_torture_ops tasks_ops = {\n--\nkernel/rcu/rcutorture.c-1181-\t.deferred_free\t= rcu_tasks_torture_deferred_free,\nkernel/rcu/rcutorture.c:1182:\t.sync\t\t= synchronize_rcu_tasks,\nkernel/rcu/rcutorture.c-1183-\t.exp_sync\t= synchronize_rcu_mult_test,\nkernel/rcu/rcutorture.c:1184:\t.call\t\t= call_rcu_tasks,\nkernel/rcu/rcutorture.c-1185-\t.cb_barrier\t= rcu_barrier_tasks,\n--\nkernel/rcu/rcutorture.c=1208=static struct rcu_torture_ops tasks_rude_ops = {\n--\nkernel/rcu/rcutorture.c-1214-\t.get_gp_seq\t= rcu_no_completed,\nkernel/rcu/rcutorture.c:1215:\t.sync\t\t= synchronize_rcu_tasks_rude,\nkernel/rcu/rcutorture.c:1216:\t.exp_sync\t= synchronize_rcu_tasks_rude,\nkernel/rcu/rcutorture.c-1217-\t.gp_kthread_dbg\t= show_rcu_tasks_rude_gp_kthread,\n--\nkernel/rcu/rcutorture.c=1259=static void rcu_tasks_tracing_torture_deferred_free(struct rcu_torture *p)\nkernel/rcu/rcutorture.c-1260-{\nkernel/rcu/rcutorture.c:1261:\tcall_rcu_tasks_trace(\u0026p-\u003ertort_rcu, rcu_torture_cb);\nkernel/rcu/rcutorture.c-1262-}\n--\nkernel/rcu/rcutorture.c=1264=static struct rcu_torture_ops tasks_tracing_ops = {\n--\nkernel/rcu/rcutorture.c-1273-\t.deferred_free\t= rcu_tasks_tracing_torture_deferred_free,\nkernel/rcu/rcutorture.c:1274:\t.sync\t\t= synchronize_rcu_tasks_trace,\nkernel/rcu/rcutorture.c:1275:\t.exp_sync\t= synchronize_rcu_tasks_trace,\nkernel/rcu/rcutorture.c-1276-\t.exp_current\t= rcu_tasks_trace_expedite_current,\nkernel/rcu/rcutorture.c:1277:\t.call\t\t= call_rcu_tasks_trace,\nkernel/rcu/rcutorture.c-1278-\t.cb_barrier\t= rcu_barrier_tasks_trace,\n--\nkernel/rcu/rcutorture.c=4516=static void rcu_torture_init_srcu_lockdep(void)\n--\nkernel/rcu/rcutorture.c-4615-\t\t\tchar *fl = i == 0 ? \"rcu_read_lock_trace\" : \"srcu_read_lock\";\nkernel/rcu/rcutorture.c:4616:\t\t\tchar *fs = i == cyclelen - 1 ? \"synchronize_rcu_tasks_trace\"\nkernel/rcu/rcutorture.c-4617-\t\t\t\t\t\t : \"synchronize_srcu\";\n--\nkernel/rcu/rcutorture.c-4626-\t\t\t\tif (i == cyclelen - 1)\nkernel/rcu/rcutorture.c:4627:\t\t\t\t\tsynchronize_rcu_tasks_trace();\nkernel/rcu/rcutorture.c-4628-\t\t\t\telse\n--\nkernel/rcu/tasks.h=93=struct rcu_tasks {\n--\nkernel/rcu/tasks.h-128-\nkernel/rcu/tasks.h:129:static void call_rcu_tasks_iw_wakeup(struct irq_work *iwp);\nkernel/rcu/tasks.h-130-\n--\nkernel/rcu/tasks.h=132=static DEFINE_PER_CPU(struct rcu_tasks_percpu, rt_name ## __percpu) = {\t\t\t\\\nkernel/rcu/tasks.h-133-\t.lock = __RAW_SPIN_LOCK_UNLOCKED(rt_name ## __percpu.cbs_pcpu_lock),\t\t\\\nkernel/rcu/tasks.h:134:\t.rtp_irq_work = IRQ_WORK_INIT_HARD(call_rcu_tasks_iw_wakeup),\t\t\t\\\nkernel/rcu/tasks.h-135-};\t\t\t\t\t\t\t\t\t\t\t\\\n--\nkernel/rcu/tasks.h=302=static unsigned long rcu_tasks_lazy_time(struct rcu_tasks *rtp)\n--\nkernel/rcu/tasks.h-307-// Timer handler that unlazifies lazy callbacks.\nkernel/rcu/tasks.h:308:static void call_rcu_tasks_generic_timer(struct timer_list *tlp)\nkernel/rcu/tasks.h-309-{\n--\nkernel/rcu/tasks.h-328-\nkernel/rcu/tasks.h:329:// IRQ-work handler that does deferred wakeup for call_rcu_tasks_generic().\nkernel/rcu/tasks.h:330:static void call_rcu_tasks_iw_wakeup(struct irq_work *iwp)\nkernel/rcu/tasks.h-331-{\n--\nkernel/rcu/tasks.h-339-// Enqueue a callback for the specified flavor of Tasks RCU.\nkernel/rcu/tasks.h:340:static void call_rcu_tasks_generic(struct rcu_head *rhp, rcu_callback_t func,\nkernel/rcu/tasks.h-341-\t\t\t\t struct rcu_tasks *rtp)\n--\nkernel/rcu/tasks.h=458=static int rcu_tasks_need_gpcb(struct rcu_tasks *rtp)\n--\nkernel/rcu/tasks.h-505-\t// to CPU 0. Note the matching RCU read-side critical section in\nkernel/rcu/tasks.h:506:\t// call_rcu_tasks_generic().\nkernel/rcu/tasks.h-507-\tif (rcu_task_cb_adjust \u0026\u0026 ncbs \u003c= rcu_task_collapse_lim) {\n--\nkernel/rcu/tasks.h=632=static int __noreturn rcu_tasks_kthread(void *arg)\n--\nkernel/rcu/tasks.h-639-\nkernel/rcu/tasks.h:640:\t\ttimer_setup(\u0026rtpcp-\u003elazy_timer, call_rcu_tasks_generic_timer, 0);\nkernel/rcu/tasks.h-641-\t\trtpcp-\u003eurgent_gp = 1;\n--\nkernel/rcu/tasks.h-664-// Wait for a grace period for the specified flavor of Tasks RCU.\nkernel/rcu/tasks.h:665:static void synchronize_rcu_tasks_generic(struct rcu_tasks *rtp)\nkernel/rcu/tasks.h-666-{\n--\nkernel/rcu/tasks.h=905=module_param(rcu_tasks_lazy_ms, int, 0444);\n--\nkernel/rcu/tasks.h-919-// preempted or not, is an ordinary rcu_read_lock_trace() reader and\nkernel/rcu/tasks.h:920:// synchronize_rcu_tasks_trace() waits for it.\nkernel/rcu/tasks.h-921-//\n--\nkernel/rcu/tasks.h-945-// 2. Wait for the holdout list (as it stood) to drain.\nkernel/rcu/tasks.h:946:// 3. synchronize_rcu_tasks_trace(), for everything inside the readers.\nkernel/rcu/tasks.h-947-// 4. Repeat 1 and 2 for tasks that have since left the reader and are in\n--\nkernel/rcu/tasks.h=956=static void rcu_tasks_tramp_wait_gp(struct rcu_tasks *rtp);\nkernel/rcu/tasks.h:957:void call_rcu_tasks(struct rcu_head *rhp, rcu_callback_t func);\nkernel/rcu/tasks.h:958:DEFINE_RCU_TASKS(rcu_tasks, rcu_tasks_tramp_wait_gp, call_rcu_tasks, \"RCU Tasks\");\nkernel/rcu/tasks.h-959-\n--\nkernel/rcu/tasks.h=990=bool __weak arch_rcu_tasks_trampoline_text(unsigned long ip)\n--\nkernel/rcu/tasks.h-999- * True when a task interrupted at @ip may be executing, or about to enter\nkernel/rcu/tasks.h:1000: * or return into, text whose lifetime depends on synchronize_rcu_tasks()\nkernel/rcu/tasks.h-1001- * without being inside the Tasks Trace reader that text takes around its\n--\nkernel/rcu/tasks.h-1011- * - the bytes after a kprobe that a pending jump optimization is about to\nkernel/rcu/tasks.h:1012: * overwrite, the one synchronize_rcu_tasks() user with no trampoline.\nkernel/rcu/tasks.h-1013- *\n--\nkernel/rcu/tasks.h=1143=EXPORT_SYMBOL_GPL(rcu_tasks_note_qs);\t/* cond_resched_tasks_rcu_qs() */\n--\nkernel/rcu/tasks.h-1153- * to the grace period. Wait until no parked task's recorded preemption IP\n\n"
} |
| 55/2 | 2026/09/24 04:22 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 108550 output: 31 thoughts: 2021 Thoughts: **Focus Symbol Determination for Tasks RCU Patch** Alright, let's break this down for fuzzing purposes. My primary concern here is identifying the right `FocusSymbols` to prevent the fuzzer from being diluted by overly broad coverage. I need to zero in on the specific changes introduced or modified by this patch related to Tasks RCU and associated trampoline operations, avoiding ubiquitous kernel functions. I began by examining the callers of `call_rcu_tasks` and `synchronize_rcu_tasks` to see where the Tasks RCU functionality is being invoked. I've noted down the following functions: `bpf_tramp_image_put`, `kprobe_optimizer`, `ftrace_shutdown`, `unregister_ftrace_direct`, and `bpf_struct_ops_map_free`. Then I looked at what executes the grace period loop, which leads me to `rcu_tasks_tramp_wait_gp` and its downstream calls: `rcu_tasks_tramp_wait_cpus`, `rcu_tasks_tramp_wait_holdouts`, and `synchronize_rcu_tasks_trace`. Furthermore, I've considered the functions related to preemption: `rcu_tasks_irq_resched_enter`, `rcu_tasks_irq_resched_exit`, `rcu_tasks_trampoline_text`, `arch_rcu_tasks_trampoline_text`, `rcu_tasks_note_qs`, `rcu_tasks_wait_irq_preempted`, `kprobe_in_optimized_region`, `ftrace_direct_mark_module`, `emit_trace_rcu_reader`, and `rcu_tasks_trace_assert_idle`. Now, the `FocusSymbols` guidance is clear: target the specific feature logic and entrypoints, while avoiding ubiquitous functions. I've identified the key functions that have been added or modified: 1. `rcu_tasks_trampoline_text` 2. `arch_rcu_tasks_trampoline_text` 3. `rcu_tasks_wait_irq_preempted` 4. `rcu_tasks_note_qs` 5. `kprobe_in_optimized_region` 6. `arch_prepare_bpf_trampoline` 7. `register_ftrace_direct` 8. `update_ftrace_direct_add` 9. `update_ftrace_direct_mod` 10. `rcu_tasks_tramp_wait_gp` I noticed `rcu_tasks_irq_resched_enter` seems like a key function but it might be ubiquitous. `preempt_schedule_irq()` happens on involuntary preemption when `TIF_NEED_RESCHED` is set during an interrupt returning to kernel space. This could happen in many syscalls as kernel preempts on interrupt exits. Then I'm making sure to exclude functions like that. I am now focusing on determining which of the functions listed are non-static and non-ubiquitous: * I need to be sure about `rcu_tasks_tramp_wait_gp` `rcu_tasks_tramp_wait_cpus` `rcu_tasks_tramp_wait_holdouts`. They are static, but the address of `rcu_tasks_tramp_wait_gp` is taken, so it *won't* be inlined. This makes it a distinct symbol. * `rcu_tasks_trampoline_text`, `arch_rcu_tasks_trampoline_text`, `kprobe_in_optimized_region`, `rcu_tasks_wait_irq_preempted`, `arch_prepare_bpf_trampoline`, `register_ftrace_direct` are all global symbols. * `kprobe_optimizer` is the last one I need to be sure about. So, the focus should be on the following functions: `rcu_tasks_trampoline_text`, `arch_rcu_tasks_trampoline_text`, `rcu_tasks_wait_irq_preempted`, `kprobe_in_optimized_region`, `rcu_tasks_tramp_wait_gp`, and then the downstream calls of `rcu_tasks_tramp_wait_cpus` and `rcu_tasks_tramp_wait_holdouts` and `arch_prepare_bpf_trampoline`, `register_ftrace_direct`. I will need to verify if `kprobe_optimizer` is static or non-static. Those will be my initial `Focus |
| 56/2 | 2026/09/24 04:22 | tool | grepper |
0mArgs:{
"Expression": "kprobe_optimizer",
"PathPrefix": "kernel/kprobes.c"
}Results: {
"Output": "kernel/kprobes.c=495=static struct kprobe *get_optimized_kprobe(kprobe_opcode_t *addr)\n--\nkernel/kprobes.c-514-/*\nkernel/kprobes.c:515: * True while kprobe_optimizer() is waiting for its Tasks RCU grace period.\nkernel/kprobes.c-516- * Only in that window can an interruption inside an optprobe's jump region\n--\nkernel/kprobes.c-518- */\nkernel/kprobes.c:519:static bool kprobe_optimizer_waiting;\nkernel/kprobes.c-520-\n--\nkernel/kprobes.c-525- *\nkernel/kprobes.c:526: * kprobe_optimizer() relies on synchronize_rcu_tasks() to wait for tasks that\nkernel/kprobes.c-527- * were interrupted on an instruction boundary inside the region about to be\n--\nkernel/kprobes.c=542=bool kprobe_in_optimized_region(unsigned long addr)\n--\nkernel/kprobes.c-545-\nkernel/kprobes.c:546:\tif (!READ_ONCE(kprobe_optimizer_waiting))\nkernel/kprobes.c-547-\t\treturn false;\n--\nkernel/kprobes.c=561=static void optimize_kprobe(struct kprobe *p);\nkernel/kprobes.c:562:static struct task_struct *kprobe_optimizer_task;\nkernel/kprobes.c:563:static wait_queue_head_t kprobe_optimizer_wait;\nkernel/kprobes.c-564-static atomic_t optimizer_state;\n--\nkernel/kprobes.c=637=static void do_free_cleaned_kprobes(void)\n--\nkernel/kprobes.c-664-\nkernel/kprobes.c:665:static void kick_kprobe_optimizer(void);\nkernel/kprobes.c-666-\nkernel/kprobes.c-667-/* Kprobe jump optimizer */\nkernel/kprobes.c:668:static void kprobe_optimizer(void)\nkernel/kprobes.c-669-{\n--\nkernel/kprobes.c-688-\t\t * to synchronoze_sched() to wait for all interrupts to have completed.\nkernel/kprobes.c:689:\t\t * kprobe_optimizer_waiting lets a reader-marked-trampoline Tasks RCU\nkernel/kprobes.c-690-\t\t * recognise tasks interrupted in such a region while we wait, and\n--\nkernel/kprobes.c-694-\t\t */\nkernel/kprobes.c:695:\t\tWRITE_ONCE(kprobe_optimizer_waiting, true);\nkernel/kprobes.c-696-\t\trcu_tasks_wait_irq_preempted(kprobe_in_optimized_region);\nkernel/kprobes.c-697-\t\tsynchronize_rcu_tasks();\nkernel/kprobes.c:698:\t\tWRITE_ONCE(kprobe_optimizer_waiting, false);\nkernel/kprobes.c-699-\n--\nkernel/kprobes.c-711-\tif (!list_empty(\u0026optimizing_list) || !list_empty(\u0026unoptimizing_list))\nkernel/kprobes.c:712:\t\tkick_kprobe_optimizer();\t/*normal kick*/\nkernel/kprobes.c-713-}\nkernel/kprobes.c-714-\nkernel/kprobes.c:715:static int kprobe_optimizer_thread(void *data)\nkernel/kprobes.c-716-{\n--\nkernel/kprobes.c-718-\t\t/* To avoid hung_task, wait in interruptible state. */\nkernel/kprobes.c:719:\t\twait_event_interruptible(kprobe_optimizer_wait,\nkernel/kprobes.c-720-\t\t\t atomic_read(\u0026optimizer_state) != OPTIMIZER_ST_IDLE ||\n--\nkernel/kprobes.c-731-\t\t\twait_event_interruptible_timeout(\nkernel/kprobes.c:732:\t\t\t\tkprobe_optimizer_wait,\nkernel/kprobes.c-733-\t\t\t\tatomic_read(\u0026optimizer_state) == OPTIMIZER_ST_FLUSHING ||\n--\nkernel/kprobes.c-741-\nkernel/kprobes.c:742:\t\tkprobe_optimizer();\nkernel/kprobes.c-743-\t}\n--\nkernel/kprobes.c-747-/* Start optimizer after OPTIMIZE_DELAY passed */\nkernel/kprobes.c:748:static void kick_kprobe_optimizer(void)\nkernel/kprobes.c-749-{\n--\nkernel/kprobes.c-752-\t\tOPTIMIZER_ST_IDLE, OPTIMIZER_ST_KICKED) == OPTIMIZER_ST_IDLE)\nkernel/kprobes.c:753:\t\twake_up(\u0026kprobe_optimizer_wait);\nkernel/kprobes.c-754-}\nkernel/kprobes.c-755-\nkernel/kprobes.c:756:static void wait_for_kprobe_optimizer_locked(void)\nkernel/kprobes.c-757-{\n--\nkernel/kprobes.c-767-\t\t\tOPTIMIZER_ST_FLUSHING) != OPTIMIZER_ST_FLUSHING)\nkernel/kprobes.c:768:\t\t\twake_up(\u0026kprobe_optimizer_wait);\nkernel/kprobes.c-769-\n--\nkernel/kprobes.c-776-/* Wait for completing optimization and unoptimization */\nkernel/kprobes.c:777:void wait_for_kprobe_optimizer(void)\nkernel/kprobes.c-778-{\n--\nkernel/kprobes.c-780-\nkernel/kprobes.c:781:\twait_for_kprobe_optimizer_locked();\nkernel/kprobes.c-782-}\n--\nkernel/kprobes.c=797=static void optimize_kprobe(struct kprobe *p)\n--\nkernel/kprobes.c-833-\tlist_add(\u0026op-\u003elist, \u0026optimizing_list);\nkernel/kprobes.c:834:\tkick_kprobe_optimizer();\nkernel/kprobes.c-835-}\n--\nkernel/kprobes.c=846=static void unoptimize_kprobe(struct kprobe *p, bool force)\n--\nkernel/kprobes.c-881-\t\tlist_add(\u0026op-\u003elist, \u0026unoptimizing_list);\nkernel/kprobes.c:882:\t\tkick_kprobe_optimizer();\nkernel/kprobes.c-883-\t}\n--\nkernel/kprobes.c=908=static void kill_optimized_kprobe(struct kprobe *p)\n--\nkernel/kprobes.c-920-\t\t * Unused kprobe is on unoptimizing or freeing list. We move it\nkernel/kprobes.c:921:\t\t * to freeing_list and let the kprobe_optimizer() remove it from\nkernel/kprobes.c-922-\t\t * the kprobe hash list and free it.\n--\nkernel/kprobes.c=1024=static void unoptimize_all_kprobes(void)\n--\nkernel/kprobes.c-1045-\t/* Wait for unoptimizing completion. */\nkernel/kprobes.c:1046:\twait_for_kprobe_optimizer_locked();\nkernel/kprobes.c-1047-\tpr_info(\"kprobe jump-optimization is disabled. All kprobes are based on software breakpoint.\\n\");\n--\nkernel/kprobes.c=1124=static void __init init_optprobe(void)\n--\nkernel/kprobes.c-1130-\nkernel/kprobes.c:1131:\tinit_waitqueue_head(\u0026kprobe_optimizer_wait);\nkernel/kprobes.c-1132-\tatomic_set(\u0026optimizer_state, OPTIMIZER_ST_IDLE);\nkernel/kprobes.c:1133:\tkprobe_optimizer_task = kthread_run(kprobe_optimizer_thread, NULL,\nkernel/kprobes.c-1134-\t\t\t\t\t \"kprobe-optimizer\");\n--\nkernel/kprobes.c-1146-#define kprobe_disarmed(p)\t\t\tkprobe_disabled(p)\nkernel/kprobes.c:1147:#define wait_for_kprobe_optimizer_locked()\t\t\t\\\nkernel/kprobes.c-1148-\tlockdep_assert_held(\u0026kprobe_mutex)\n--\nkernel/kprobes.c=2719=static int kprobes_module_callback(struct notifier_block *nb,\n--\nkernel/kprobes.c-2756-\t\t\t\t * will not be touched by the delayed\nkernel/kprobes.c:2757:\t\t\t\t * kprobe_optimizer() work handler.\nkernel/kprobes.c-2758-\t\t\t\t */\n--\nkernel/kprobes.c=3031=static int disarm_all_kprobes(void)\n--\nkernel/kprobes.c-3067-\t/* Wait for disarming all kprobes by optimizer */\nkernel/kprobes.c:3068:\twait_for_kprobe_optimizer_locked();\nkernel/kprobes.c-3069-\treturn ret;\n"
} |
| 57/2 | 2026/09/24 04:22 | llm | patch-evaluator-compressor |
0mModel:gemini-3.8-flash Args: null Results: null Reply: ### 1. Approaches Tried and Results
- **Patch Diff Inspection**:
- Examined the applied patch (commit `671c17ef89b17f280a59ec7bb4ed9bfe9789cf90`), which redesigns Tasks RCU for architectures supporting trampoline readers (`CONFIG_HAVE_RCU_TRAMPOLINE_READERS` and `CONFIG_TASKS_RCU_TRAMPOLINE_READERS`).
- Architecture modifications touch `arch/arm64` and `arch/x86` (`ftrace`, BPF JIT compiler `bpf_jit_comp.c`, `kprobes/opt.c`, assembly offsets, and linker script symbols like `__rethunk_safe_start` / `__return_thunk_start`).
- Core subsystem changes touch `kernel/rcu/tasks.h`, `kernel/rcu/Kconfig`, `kernel/kprobes.c`, `kernel/entry/common.c`, `kernel/trace/ftrace.c`, and headers (`include/linux/rcupdate.h`, `include/linux/rcupdate_trace.h`, `include/linux/irq-entry-common.h`, `include/linux/kprobes.h`, `include/linux/sched.h`).
- **Kconfig Dependency Analysis on x86_64**:
- `CONFIG_HAVE_RCU_TRAMPOLINE_READERS`: selected in `arch/x86/Kconfig` if `X86_64 && SMP && DYNAMIC_FTRACE`.
- `CONFIG_TASKS_RCU_TRAMPOLINE_READERS`: `def_bool TASKS_RCU && HAVE_RCU_TRAMPOLINE_READERS && GENERIC_IRQ_ENTRY && !NEED_SRCU_NMI_SAFE`.
- `CONFIG_GENERIC_IRQ_ENTRY`: selected by `CONFIG_GENERIC_ENTRY` which is selected unconditionally on `arch/x86`.
- `CONFIG_NEED_SRCU_NMI_SAFE`: `def_bool HAVE_NMI && !ARCH_HAS_NMI_SAFE_THIS_CPU_OPS && !TINY_SRCU`. On x86_64, `ARCH_HAS_NMI_SAFE_THIS_CPU_OPS` is set, so `NEED_SRCU_NMI_SAFE` is false.
- `CONFIG_TASKS_RCU`: `default NEED_TASKS_RCU && PREEMPTION`, or selected via `CONFIG_FORCE_TASKS_RCU`.
- `CONFIG_NEED_TASKS_RCU`: selected by `CONFIG_BPF_SYSCALL`, `CONFIG_TRACING`, `CONFIG_FUNCTION_TRACER`, and `CONFIG_OPTPROBES` (`arch/Kconfig`).
- `CONFIG_OPTPROBES`: `def_bool y`, depends on `KPROBES && HAVE_OPTPROBES` (x86 selects `HAVE_OPTPROBES`).
- `CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS`: `def_bool y`, depends on `(DYNAMIC_FTRACE_WITH_REGS || DYNAMIC_FTRACE_WITH_ARGS) && HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS` (x86 selects `HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS`).
- `CONFIG_BPF_JIT`: compiles `kernel/bpf/trampoline.c` and enables BPF trampoline generation (`arch_prepare_bpf_trampoline`).
- **Reachability and Execution Assessment**:
- Code executes in standard QEMU/KVM virtualized environments on `amd64`.
- Modifies core RCU synchronization logic, interrupt exit/preemption paths, BPF trampolines, ftrace stubs, and kprobes jump optimizer.
- Adds defensive runtime assertion: `rcu_tasks_trace_assert_idle()` calling `WARN_ON_ONCE(...)` on every kernel-to-userspace exit (`__exit_to_user_mode_validate`), and `WARN_ON_ONCE(t != current)` in `rcu_tasks_note_qs()`.
---
### 2. Current Hypotheses and Conclusions
- **Evaluation Determination**:
- **`WorthFuzzing = true`**.
- Does NOT fall into negative criteria (affects core kernel, synchronization, scheduler/preemption, BPF JIT, ftrace, kprobes).
- Strongly meets positive criteria: alters core concurrency/locking protocols (Tasks RCU grace period lifecycle), modifies BPF trampoline code generation, updates interrupt exit preemption logic, and introduces runtime assertions (`rcu_tasks_trace_assert_idle`, `rcu_tasks_note_qs`).
- **Focus Target Selection Strategy**:
- Exclude ubiquitous hot paths: e.g., avoid `__schedule`, `schedule`, generic `preempt_schedule_irq`, general syscall entry/exit wrappers.
- Target subsystem entry points and specific new/modified logic:
- RCU tasks trampoline grace period routines: `rcu_tasks_tramp_wait_gp`, `rcu_tasks_irq_resched_enter`, `rcu_tasks_irq_resched_exit`, `rcu_tasks_note_qs`, `rcu_tasks_wait_irq_preempted`, `rcu_tasks_trampoline_text`, `arch_rcu_tasks_trampoline_text`.
- Interrupt exit cond-resched hook: `raw_irqentry_exit_cond_resched`.
- Kprobes optimizer: `kprobe_in_optimized_region`, `kprobe_optimizer`, `arch_prepare_optimized_kprobe`.
- BPF trampoline JIT compiler: `arch_prepare_bpf_trampoline`, `emit_trace_rcu_reader`.
- Ftrace direct registration: `register_ftrace_direct`, `ftrace_direct_mark_module`.
---
### 3. Critical Files, Symbols, and Config Options
#### Key Files
- `kernel/rcu/tasks.h`: Implementation of the trampoline-reader Tasks RCU engine (`rcu_tasks_tramp_wait_gp`, `rcu_tasks_tramp_wait_cpus`, `rcu_tasks_tramp_wait_holdouts`, `rcu_tasks_irq_resched_enter`, `rcu_tasks_irq_resched_exit`, `rcu_tasks_note_qs`, `rcu_tasks_wait_irq_preempted`, `rcu_tasks_trampoline_text`).
- `kernel/entry/common.c`: `raw_irqentry_exit_cond_resched(struct pt_regs *regs)`.
- `include/linux/irq-entry-common.h`: `__exit_to_user_mode_validate` containing `rcu_tasks_trace_assert_idle()`.
- `kernel/kprobes.c`: `kprobe_in_optimized_region()`, `kprobe_optimizer()`.
- `arch/x86/kernel/kprobes/opt.c`: `arch_prepare_optimized_kprobe`, inline assembly macros `OPTPROBE_TRACE_RCU_READ_LOCK`, `OPTPROBE_TRACE_RCU_READ_UNLOCK`.
- `arch/x86/net/bpf_jit_comp.c`: `emit_trace_rcu_reader()`, `__arch_prepare_bpf_trampoline()`, `arch_prepare_bpf_trampoline()`.
- `arch/x86/kernel/ftrace.c`: `arch_rcu_tasks_trampoline_text()`.
- `arch/x86/kernel/ftrace_64.S`: `ftrace_caller`, `ftrace_regs_caller`, `TRACE_RCU_READ_LOCK`, `TRACE_RCU_READ_UNLOCK`, `ftrace_static_tramp_end`.
- `kernel/trace/ftrace.c`: `ftrace_direct_mark_module()`, `register_ftrace_direct()`.
#### Required Configs (stripped of `CONFIG_`)
- `TASKS_RCU_TRAMPOLINE_READERS`
- `HAVE_RCU_TRAMPOLINE_READERS`
- `TASKS_RCU`
- `DYNAMIC_FTRACE`
- `DYNAMIC_FTRACE_WITH_DIRECT_CALLS`
- `KPROBES`
- `OPTPROBES`
- `BPF_SYSCALL`
- `BPF_JIT`
- `SMP`
- `PREEMPTION` (or `PREEMPT` / `PREEMPT_DYNAMIC`)
#### Candidate FocusSymbols
- `rcu_tasks_tramp_wait_gp`
- `rcu_tasks_irq_resched_enter`
- `rcu_tasks_irq_resched_exit`
- `rcu_tasks_note_qs`
- `rcu_tasks_wait_irq_preempted`
- `rcu_tasks_trampoline_text`
- `arch_rcu_tasks_trampoline_text`
- `raw_irqentry_exit_cond_resched`
- `kprobe_in_optimized_region`
- `arch_prepare_optimized_kprobe`
- `arch_prepare_bpf_trampoline`
- `register_ftrace_direct`
---
### 4. Reasoning Loops & Next Actions
- **Avoidance of Loops**: All architectural and configuration dependencies have been verified across x86 Kconfigs. No further grepping for Kconfig options is required.
- **Next Step**: When execution resumes, call `set-results` with `WorthFuzzing=true`, the verified `EnableConfigs`, `FocusSymbols`, and clear `Reasoning`.Tokens: input: 89858 output: 2241 thoughts: 0 |
| 58/2 | 2026/09/24 04:22 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 55863 output: 441 thoughts: 901 Thoughts: **Fuzzing Target Assessment: Tasks RCU with Trampoline Readers** Okay, let me quickly run through this. We need to verify this patch's fuzzing worthiness. First, let's nail down if this is indeed fuzz-worthy. 1. **WorthFuzzing**: Absolutely. We are looking at a core subsystem modification. We're talking about changes to crucial synchronization and locking primitives within `kernel/rcu/tasks.h`, scheduler and preemption hooks via `kernel/entry/common.c`, BPF related logic in `arch/x86/net/bpf_jit_comp.c`, and ftrace modifications, found in `arch/x86/kernel/ftrace.c`, `arch/x86/kernel/ftrace_64.S`, and `kernel/trace/ftrace.c`, and kprobes modifications in `kernel/kprobes.c`, and `arch/x86/kernel/kprobes/opt.c`. Essentially, we're mucking around with the Tasks RCU grace period engine and introducing a defensive runtime assertion (`rcu_tasks_trace_assert_idle`) which is a nice safety net. Plus, it's directly accessible in a standard QEMU environment on amd64/x86_64, so accessibility isn't a problem. `WorthFuzzing = true`. That's a strong start. 2. **EnableConfigs**: We need to make sure we've got the right kernel configurations enabled. Let's make a mental note of those configurations stripped of `CONFIG_`: `TASKS_RCU_TRAMPOLINE_READERS`, `HAVE_RCU_TRAMPOLINE_READERS`, `TASKS_RCU`, `DYNAMIC_FTRACE`, `DYNAMIC_FTRACE_WITH_DIRECT_CALLS`, `KPROBES`, `OPTPROBES`, `BPF_SYSCALL`, `BPF_JIT`, `SMP`, and `PREEMPTION`. Gotta make sure these are all present and accounted for to exercise the new code paths. 3. **FocusSymbols**: The goal here is to steer clear of the noise and focus on the juicy bits. We want to avoid hot-path functions, think `__schedule`, page allocators, memory allocators and generic syscall handlers, etc. We're going for targeted fuzzing here, so let's zoom in on the specific feature logic and entrypoints. Okay, here's what we need: `rcu_tasks_tramp_wait_gp` (the main grace period function for Tasks RCU trampoline readers), `rcu_tasks_irq_resched_enter` and `rcu_tasks_irq_resched_exit` (Tasks RCU hooks in irq preemption). Also: `rcu_tasks_note_qs`, `rcu_tasks_wait_irq_preempted`, `rcu_tasks_trampoline_text`, and `arch_rcu_tasks_trampoline_text`. And finally `raw_irqentry_exit_cond_resched`, `kprobe_in_optimized_region`, `arch_prepare_bpf_trampoline`, `register_ftrace_direct` and `kprobe_optimizer`. That looks like a good set of entry points. 4. **Reasoning**: Right, time to synthesize this into a compelling argument. Essentially, the patch transforms the standard Tasks RCU grace period mechanism by introducing `CONFIG_TASKS_RCU_TRAMPOLINE_READERS` on x86_64 and arm64 architectures. The code uses trampolines (ftrace, BPF, and optprobes) as Tasks Trace RCU readers. This preemption in unmarked trampoline text is tracked during irq-exit cond resched. This patch warrants fuzzing because it plays with synchronization, interrupt exit preemption paths, BPF JIT trampoline generation, ftrace stub handling, and kprobe jump optimizer synchronization, and introduces runtime assertions, such as `rcu_tasks_trace_assert_idle`. And, like I said, it is all fully reachable on standard x86_64 SMP configurations within QEMU. That is a good, complete, and correct summary. Okay, I'm satisfied. Time to run `set-results`. |
| 59/2 | 2026/09/24 04:22 | tool | set-results |
0mArgs:{
"EnableConfigs": [
"TASKS_RCU_TRAMPOLINE_READERS",
"HAVE_RCU_TRAMPOLINE_READERS",
"TASKS_RCU",
"DYNAMIC_FTRACE",
"DYNAMIC_FTRACE_WITH_DIRECT_CALLS",
"KPROBES",
"OPTPROBES",
"BPF_SYSCALL",
"BPF_JIT",
"SMP",
"PREEMPTION"
],
"FocusSymbols": [
"rcu_tasks_tramp_wait_gp",
"rcu_tasks_irq_resched_enter",
"rcu_tasks_irq_resched_exit",
"rcu_tasks_note_qs",
"rcu_tasks_wait_irq_preempted",
"rcu_tasks_trampoline_text",
"arch_rcu_tasks_trampoline_text",
"raw_irqentry_exit_cond_resched",
"kprobe_in_optimized_region",
"arch_prepare_bpf_trampoline",
"register_ftrace_direct",
"kprobe_optimizer"
],
"Reasoning": "The patch series introduces CONFIG_TASKS_RCU_TRAMPOLINE_READERS and CONFIG_HAVE_RCU_TRAMPOLINE_READERS, restructuring how Tasks RCU grace periods and quiescent states are tracked on x86_64 and arm64. It embeds Tasks Trace RCU reader state transitions inside ftrace, optprobe, and BPF trampolines, hooks into kernel irq-exit preemption paths (raw_irqentry_exit_cond_resched) to monitor tasks preempted in unmarked trampoline text, adds synchronization in kprobe_optimizer, and introduces runtime assertions like rcu_tasks_trace_assert_idle(). This fundamentally alters core concurrency and synchronization primitives in reachable paths on amd64, warranting fuzzing.",
"WorthFuzzing": true
}Results: {
"EnableConfigs": [
"TASKS_RCU_TRAMPOLINE_READERS",
"HAVE_RCU_TRAMPOLINE_READERS",
"TASKS_RCU",
"DYNAMIC_FTRACE",
"DYNAMIC_FTRACE_WITH_DIRECT_CALLS",
"KPROBES",
"OPTPROBES",
"BPF_SYSCALL",
"BPF_JIT",
"SMP",
"PREEMPTION"
],
"FocusSymbols": [
"rcu_tasks_tramp_wait_gp",
"rcu_tasks_irq_resched_enter",
"rcu_tasks_irq_resched_exit",
"rcu_tasks_note_qs",
"rcu_tasks_wait_irq_preempted",
"rcu_tasks_trampoline_text",
"arch_rcu_tasks_trampoline_text",
"raw_irqentry_exit_cond_resched",
"kprobe_in_optimized_region",
"arch_prepare_bpf_trampoline",
"register_ftrace_direct",
"kprobe_optimizer"
],
"Reasoning": "The patch series introduces CONFIG_TASKS_RCU_TRAMPOLINE_READERS and CONFIG_HAVE_RCU_TRAMPOLINE_READERS, restructuring how Tasks RCU grace periods and quiescent states are tracked on x86_64 and arm64. It embeds Tasks Trace RCU reader state transitions inside ftrace, optprobe, and BPF trampolines, hooks into kernel irq-exit preemption paths (raw_irqentry_exit_cond_resched) to monitor tasks preempted in unmarked trampoline text, adds synchronization in kprobe_optimizer, and introduces runtime assertions like rcu_tasks_trace_assert_idle(). This fundamentally alters core concurrency and synchronization primitives in reachable paths on amd64, warranting fuzzing.",
"WorthFuzzing": true
} |
| 60/1 | 2026/09/24 04:22 | action | If |
0mArgs:{
"WorthFuzzing": true
}Results: null |
| 61/2 | 2026/09/24 04:22 | agent | kmsan-evaluator |
0mModel:core Args: null Results: {
"KMSANReasoning": "The patch series implements trampoline-reader Tasks RCU (CONFIG_TASKS_RCU_TRAMPOLINE_READERS), which replaces the classic voluntary-context-switch Tasks RCU grace period mechanism with Tasks Trace RCU read-side critical sections around ftrace, BPF, and kprobe trampolines and checks on irq-exit preemption.\n\nKey observations:\n1. No uninitialized memory usage or exposure: All new/modified fields in core structures (such as `task_struct.rcu_tasks_irq_ip` and `module.ftrace_direct_tramp`) are explicitly initialized on task creation in `rcu_copy_process()` and in zero-allocated module structs. Global and per-CPU state variables are statically defined and zero-initialized.\n2. No data copied or leaked to user space: The changes do not touch `copy_to_user`, `put_user`, netlink, socket options, or ioctl outputs.\n3. Nature of bugs and applicable detectors: Any bugs introduced by this series would manifest as synchronization races, premature grace period completion leading to use-after-free of freed trampoline text or metadata, or locking/deadlock issues. These failure modes are strictly the domain of KASAN (for use-after-free and memory corruption) and LOCKDEP (for locking violations), not KMSAN.\n\nTherefore, dedicated KMSAN fuzzing is not warranted.",
"NeedsKMSAN": false
}Instruction: You are an expert Linux kernel security engineer specializing in kernel memory
error detectors (KASAN and KMSAN). Your job is to review the provided patch series and
determine if the code changes justify spawning a dedicated KMSAN (KernelMemorySanitizer)
fuzzing session in addition to standard KASAN fuzzing.
CRITICAL DISTINCTION BETWEEN KASAN AND KMSAN:
- Standard KASAN kernel builds (upstream-apparmor-kasan.config) already enable
a comprehensive suite of debugging tools and sanitizers, including KASAN
(out-of-bounds accesses, use-after-free, double free, invalid free), LOCKDEP
(locking bugs and deadlocks), UB-sanitizers, and memory corruption checks.
- KMSAN (KernelMemorySanitizer) detects reads of UNINITIALIZED memory (stack, heap,
or page allocations) and kernel-to-user memory info-leaks.
Rule: THERE IS NO SENSE IN RUNNING A KMSAN SESSION IF A BUG CAN BE CAUGHT BY KASAN,
LOCKDEP, OR OTHER STANDARD BUG DETECTORS.
A dedicated KMSAN fuzzing session incurs significant resource costs. You must ONLY
set NeedsKMSAN=true if the code changes introduce or expose UNINITIALIZED MEMORY risks
that are detected ONLY by KMSAN.
Look holistically at the patch series and surrounding code. Even if no direct
uninitialized field accesses or new buffer allocations are added in the diff itself,
a patch may alter control flow, bounds checking, or data length calculations in ways
that change how the rest of the code operates on existing buffers (e.g. allowing
uninitialized stack/heap memory to be read, copied to user space, or used in control
flow). Do not hesitate to use your code access tools to inspect the surrounding code,
called functions, and callers.
Set NeedsKMSAN=true ONLY IF the patch introduces or modifies:
1. Kernel structures sent to user space (via copy_to_user, put_user, netlink skb
attributes, ioctl output arguments, socket options, or BPF buffers) where fields
or structure padding might not be fully initialized/zeroed.
2. Conditional logic or branching that depends on potentially uninitialized variables
or struct fields.
3. Allocation or initialization of complex data structures where uninitialized fields
could be read later in reachable code paths.
4. Bounds checks, lengths, or logic in a way that allows surrounding code to access
uninitialized bytes of existing buffers.
Set NeedsKMSAN=false IF:
- The code changes primarily risk out-of-bounds access, array overflows, NULL pointer
dereferences, locking deadlocks, or use-after-free bugs (these are already caught
by KASAN, LOCKDEP, or standard bug detectors).
- All stack/heap structures touched or introduced by the patch are fully zeroed
or initialized (e.g. using = {0}, memset, kzalloc) before being read or copied.
- The patch does not introduce any risk of uninitialized memory usage or info-leaks.
Use your code access tools to inspect the surrounding code if necessary, then provide
detailed KMSANReasoning contrasting KASAN vs KMSAN applicability for this patch.
Prefer calling several tools at the same time to save round-trips.
Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.
Prompt: Target architecture: amd64
For your convenience, here is the diff of the changes:
commit 671c17ef89b17f280a59ec7bb4ed9bfe9789cf90
Author: syz-cluster <triage@syzkaller.com>
Date: Thu Sep 24 04:18:56 2026 +0000
syz-cluster: applied patch under review
diff --git a/Documentation/RCU/Design/Requirements/Requirements.rst b/Documentation/RCU/Design/Requirements/Requirements.rst
index 8101fe6229d57..c059c463e083f 100644
--- a/Documentation/RCU/Design/Requirements/Requirements.rst
+++ b/Documentation/RCU/Design/Requirements/Requirements.rst
@@ -2756,6 +2756,30 @@ synchronize_rcu(), and rcu_barrier(), respectively. In
three APIs are therefore implemented by separate functions that check
for voluntary context switches.
+Architectures that select ``CONFIG_HAVE_RCU_TRAMPOLINE_READERS`` keep the
+same three APIs but implement the grace period differently
+(``CONFIG_TASKS_RCU_TRAMPOLINE_READERS``). There, every trampoline whose
+lifetime Tasks RCU guards enters a Tasks Trace RCU read-side critical
+section (rcu_read_lock_trace() or its assembly equivalent) before calling
+out and leaves it before returning, so a task anywhere inside such a
+call-out is an ordinary Tasks Trace reader whether or not it is
+preempted. The few trampoline instructions outside that reader can only
+be occupied by a task that was interrupted there, so the grace period
+additionally waits for each CPU to pass through a context switch, and the
+irq-exit preemption path, the only switch that can catch a task inside
+such text (rcu_tasks_trampoline_text()), briefly makes such a task a
+holdout until it is next seen elsewhere. On such kernels an involuntary
+context switch outside trampoline text *is* a Tasks-RCU quiescent state,
+a Tasks RCU grace period no longer depends on how long any task runs
+without sleeping, cond_resched_tasks_rcu_qs() is unnecessary, and the
+obligation moves to the trampolines: anything that relies on
+synchronize_rcu_tasks() to protect code a task may be preempted in must
+take the Tasks Trace reader (see register_ftrace_direct()), or, where
+that is impossible because the code is ordinary text with no trampoline
+of its own, make it known to rcu_tasks_trampoline_text() and wait out
+tasks already parked there with rcu_tasks_wait_irq_preempted(), as the
+kprobe jump optimizer does.
+
Tasks Rude RCU
~~~~~~~~~~~~~~
diff --git a/Documentation/RCU/checklist.rst b/Documentation/RCU/checklist.rst
index 4b30f701225fd..87dd506bffefb 100644
--- a/Documentation/RCU/checklist.rst
+++ b/Documentation/RCU/checklist.rst
@@ -252,7 +252,12 @@ over a rather long period of time, but improvements are always welcome!
a. If the updater uses synchronize_rcu_tasks() or
call_rcu_tasks(), then the readers must refrain from
executing voluntary context switches, that is, from
- blocking.
+ blocking. On architectures that select
+ CONFIG_HAVE_RCU_TRAMPOLINE_READERS a reader must in
+ addition be a Tasks Trace RCU reader (that is what the
+ trampolines there do around their call-outs) or be text
+ that rcu_tasks_trampoline_text() recognises; an arbitrary
+ stretch of preemptible kernel code is not protected.
b. If the updater uses call_rcu_tasks_trace()
or synchronize_rcu_tasks_trace(), then the
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index b5a51b0ef9440..bf0e56006863e 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -218,6 +218,7 @@ config ARM64
select HAVE_PERF_REGS
select HAVE_PERF_USER_STACK_DUMP
select HAVE_PREEMPT_DYNAMIC_KEY
+ select HAVE_RCU_TRAMPOLINE_READERS if DYNAMIC_FTRACE_WITH_ARGS
select HAVE_REGS_AND_STACK_ACCESS_API
select HAVE_RELIABLE_STACKTRACE
select HAVE_POSIX_CPU_TIMERS_TASK_WORK
diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
index 9c853ed3ceab8..f6a8fb1f9b436 100644
--- a/arch/arm64/kernel/asm-offsets.c
+++ b/arch/arm64/kernel/asm-offsets.c
@@ -10,6 +10,7 @@
#include <linux/arm_sdei.h>
#include <linux/sched.h>
+#include <linux/srcu.h>
#include <linux/ftrace.h>
#include <linux/kexec.h>
#include <linux/mm.h>
@@ -39,6 +40,13 @@ int main(void)
DEFINE(TSK_STACK, offsetof(struct task_struct, stack));
#ifdef CONFIG_STACKPROTECTOR
DEFINE(TSK_STACK_CANARY, offsetof(struct task_struct, stack_canary));
+#endif
+#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS
+ DEFINE(TSK_TRC_READER_NESTING, offsetof(struct task_struct, trc_reader_nesting));
+ DEFINE(TSK_TRC_READER_SCP, offsetof(struct task_struct, trc_reader_scp));
+ DEFINE(SRCU_SRCU_CTRP, offsetof(struct srcu_struct, srcu_ctrp));
+ DEFINE(SRCU_CTR_SRCU_LOCKS, offsetof(struct srcu_ctr, srcu_locks));
+ DEFINE(SRCU_CTR_SRCU_UNLOCKS, offsetof(struct srcu_ctr, srcu_unlocks));
#endif
BLANK();
DEFINE(THREAD_CPU_CONTEXT, offsetof(struct task_struct, thread.cpu_context));
diff --git a/arch/arm64/kernel/entry-ftrace.S b/arch/arm64/kernel/entry-ftrace.S
index 025140caafe74..fc2805eb9e15b 100644
--- a/arch/arm64/kernel/entry-ftrace.S
+++ b/arch/arm64/kernel/entry-ftrace.S
@@ -14,6 +14,72 @@
#include <asm/insn.h>
#ifdef CONFIG_DYNAMIC_FTRACE_WITH_ARGS
+/*
+ * Open-coded rcu_read_lock_trace() / rcu_read_unlock_trace(), see
+ * include/linux/rcupdate_trace.h and CONFIG_HAVE_RCU_TRAMPOLINE_READERS. The
+ * whole of ftrace_caller is treated as trampoline text by the irq-exit check
+ * (see arch_rcu_tasks_trampoline_text()), so these only need to bracket the
+ * call out to ops->func; everything before the lock and after the unlock,
+ * including the direct-call tails that carry a BPF trampoline address in x17,
+ * is covered by that.
+ *
+ * The SRCU-fast per-CPU increment is done LL/SC on this CPU's counter; being
+ * migrated between reading the per-CPU offset and the store-exclusive only
+ * means another CPU's counter is (atomically) incremented, which SRCU sums
+ * over anyway. Ordering between the nesting count and the scp stash only
+ * matters against interrupts on this CPU, which observe program order.
+ * Clobbers x12-x16 and the flags.
+ */
+ .macro trace_rcu_srcu_inc, addr:req, tmp:req, wtmp2:req
+8888: ldxr \tmp, [\addr]
+ add \tmp, \tmp, #1
+ stxr \wtmp2, \tmp, [\addr]
+ cbnz \wtmp2, 8888b
+ .endm
+
+ .macro trace_rcu_read_lock
+#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS
+ mrs x12, sp_el0 // current
+ ldr w13, [x12, #TSK_TRC_READER_NESTING]
+ add w14, w13, #1
+ str w14, [x12, #TSK_TRC_READER_NESTING]
+ cbnz w13, .Ltrl_nested\@ // interrupted a reader: done
+ ldr_l x13, rcu_tasks_trace_srcu_struct + SRCU_SRCU_CTRP
+ str x13, [x12, #TSK_TRC_READER_SCP]
+ get_this_cpu_offset x14
+ add x14, x14, x13
+ add x14, x14, #SRCU_CTR_SRCU_LOCKS
+ trace_rcu_srcu_inc x14, x15, w16
+#ifndef CONFIG_TASKS_TRACE_RCU_NO_MB
+ dmb ish
+#endif
+.Ltrl_nested\@:
+#endif
+ .endm
+
+ .macro trace_rcu_read_unlock
+#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS
+ mrs x12, sp_el0 // current
+ ldr w13, [x12, #TSK_TRC_READER_NESTING]
+ subs w13, w13, #1
+ b.ne .Ltru_nested\@
+ /* Outermost: pick up scp before an interrupt can see nesting == 0. */
+ ldr x14, [x12, #TSK_TRC_READER_SCP]
+ str wzr, [x12, #TSK_TRC_READER_NESTING]
+#ifndef CONFIG_TASKS_TRACE_RCU_NO_MB
+ dmb ish
+#endif
+ get_this_cpu_offset x15
+ add x14, x14, x15
+ add x14, x14, #SRCU_CTR_SRCU_UNLOCKS
+ trace_rcu_srcu_inc x14, x15, w16
+ b .Ltru_done\@
+.Ltru_nested\@:
+ str w13, [x12, #TSK_TRC_READER_NESTING]
+.Ltru_done\@:
+#endif
+ .endm
+
/*
* Due to -fpatchable-function-entry=2, the compiler has placed two NOPs before
* the regular function prologue. For an enabled callsite, ftrace_init_nop() and
@@ -94,6 +160,8 @@ SYM_CODE_START(ftrace_caller)
stp x29, x30, [sp, #FREGS_SIZE]
add x29, sp, #FREGS_SIZE
+ trace_rcu_read_lock
+
/* Prepare arguments for the tracer func */
sub x0, x30, #AARCH64_INSN_SIZE // ip (callsite's BL insn)
mov x1, x9 // parent_ip (callsite's LR)
@@ -111,6 +179,8 @@ SYM_INNER_LABEL(ftrace_call, SYM_L_GLOBAL)
bl ftrace_stub // func(ip, parent_ip, op, regs)
#endif
+ trace_rcu_read_unlock
+
/*
* At the callsite x0-x8 and x19-x30 were live. Any C code will have preserved
* x19-x29 per the AAPCS, and we created frame records upon entry, so we need
@@ -178,6 +248,10 @@ SYM_CODE_START(ftrace_stub_direct_tramp)
SYM_CODE_END(ftrace_stub_direct_tramp)
#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */
+/* End of [ftrace_caller, ...) for arch_rcu_tasks_trampoline_text(). */
+SYM_CODE_START(ftrace_static_tramp_end)
+SYM_CODE_END(ftrace_static_tramp_end)
+
#else /* CONFIG_DYNAMIC_FTRACE_WITH_ARGS */
/*
diff --git a/arch/arm64/kernel/ftrace.c b/arch/arm64/kernel/ftrace.c
index e1a3c0b3a0514..5f4193f15cd9a 100644
--- a/arch/arm64/kernel/ftrace.c
+++ b/arch/arm64/kernel/ftrace.c
@@ -17,6 +17,26 @@
#include <asm/insn.h>
#include <asm/text-patching.h>
+#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS
+extern void ftrace_static_tramp_end(void);
+
+/* The SRCU-fast increments in entry-ftrace.S are the this_cpu_inc() form. */
+static_assert(!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE));
+
+/*
+ * See rcu_tasks_trampoline_text(). ftrace_caller and ftrace_stub_direct_tramp
+ * are core kernel text but must be treated as trampolines: a task interrupted
+ * in them outside the Tasks Trace reader may be carrying an ops pointer (x11)
+ * or a direct-call BPF trampoline address (x17) whose lifetime is guarded only
+ * by Tasks RCU.
+ */
+bool arch_rcu_tasks_trampoline_text(unsigned long ip)
+{
+ return ip >= (unsigned long)ftrace_caller &&
+ ip < (unsigned long)ftrace_static_tramp_end;
+}
+#endif
+
#ifdef CONFIG_DYNAMIC_FTRACE_WITH_ARGS
struct fregs_offset {
const char *name;
diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
index c18e005a41dbe..87bbcca0298fd 100644
--- a/arch/arm64/net/bpf_jit_comp.c
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -14,6 +14,7 @@
#include <linux/filter.h>
#include <linux/memory.h>
#include <linux/printk.h>
+#include <linux/rcupdate_trace.h>
#include <linux/slab.h>
#include <asm/asm-extable.h>
@@ -2591,6 +2592,77 @@ static void emit_arena_arg_conv(struct jit_ctx *ctx, u8 dst, u8 src, bool nullab
emit(A64_SUB(0, dst, src, base_lo), ctx);
}
+/*
+ * Open-coded rcu_read_lock_trace() / rcu_read_unlock_trace() for the
+ * trampoline, see CONFIG_HAVE_RCU_TRAMPOLINE_READERS and the equivalent macros
+ * in arch/arm64/kernel/entry-ftrace.S. The SRCU-fast per-CPU increment is an
+ * LL/SC add on this CPU's counter; being migrated between reading the per-CPU
+ * offset and the store-exclusive only means another CPU's counter is
+ * incremented atomically instead, which SRCU sums over anyway. Uses x10-x15,
+ * which are scratch at every emission point, and the flags.
+ */
+static void emit_trace_rcu_reader(struct jit_ctx *ctx, bool lock)
+{
+#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS
+ const int nesting_off = offsetof(struct task_struct, trc_reader_nesting);
+ const int scp_off = offsetof(struct task_struct, trc_reader_scp);
+ const int ctr_off = lock ? offsetof(struct srcu_ctr, srcu_locks)
+ : offsetof(struct srcu_ctr, srcu_unlocks);
+ const bool mb = !IS_ENABLED(CONFIG_TASKS_TRACE_RCU_NO_MB);
+ const u8 tsk = A64_R(10), n = A64_R(11), tmp = A64_R(12);
+ const u8 ctr = A64_R(13), addr = A64_R(14), val = A64_R(15);
+
+ /* The plain this_cpu_inc() form of __srcu_read_lock_fast(). */
+ BUILD_BUG_ON(IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE));
+ /* LDR/STR (immediate, unsigned offset) ranges */
+ BUILD_BUG_ON((nesting_off & 3) || nesting_off >= SZ_16K ||
+ (scp_off & 7) || scp_off >= SZ_32K);
+
+ emit(A64_MRS_SP_EL0(tsk), ctx); /* current */
+ emit(A64_LDR32I(n, tsk, nesting_off), ctx);
+ if (lock) {
+ emit(A64_ADD_I(0, tmp, n, 1), ctx);
+ emit(A64_STR32I(tmp, tsk, nesting_off), ctx);
+ /* interrupted a reader: done (branch to 1:) */
+ emit(A64_CBNZ(0, n, 13 + mb), ctx);
+ /* scp = rcu_tasks_trace_srcu_struct.srcu_ctrp; current->trc_reader_scp = scp */
+ emit_addr_mov_i64(ctr, (u64)&rcu_tasks_trace_srcu_struct.srcu_ctrp, ctx);
+ emit(A64_LDR64I(ctr, ctr, 0), ctx);
+ emit(A64_STR64I(ctr, tsk, scp_off), ctx);
+ } else {
+ emit(A64_SUB_I(0, n, n, 1), ctx);
+ /* still nested: just store the count (branch to 2:) */
+ emit(A64_CBNZ(0, n, 11 + mb), ctx);
+ /* outermost: pick up scp before an interrupt can see nesting == 0 */
+ emit(A64_LDR64I(ctr, tsk, scp_off), ctx);
+ emit(A64_STR32I(A64_ZR, tsk, nesting_off), ctx);
+ if (mb)
+ emit(A64_DMB_ISH, ctx);
+ }
+ /* this_cpu_inc(scp->srcu_locks / scp->srcu_unlocks) */
+ if (cpus_have_cap(ARM64_HAS_VIRT_HOST_EXTN))
+ emit(A64_MRS_TPIDR_EL2(addr), ctx);
+ else
+ emit(A64_MRS_TPIDR_EL1(addr), ctx);
+ emit(A64_ADD(1, addr, addr, ctr), ctx);
+ emit(A64_ADD_I(1, addr, addr, ctr_off), ctx);
+ emit(A64_LDXR(1, val, addr), ctx);
+ emit(A64_ADD_I(1, val, val, 1), ctx);
+ emit(A64_STXR(1, val, addr, tmp), ctx);
+ emit(A64_CBNZ(0, tmp, -3), ctx);
+ if (lock) {
+ if (mb)
+ emit(A64_DMB_ISH, ctx);
+ /* 1: */
+ } else {
+ emit(A64_B(2), ctx);
+ /* 2: */
+ emit(A64_STR32I(n, tsk, nesting_off), ctx);
+ /* 3: */
+ }
+#endif
+}
+
static void save_args(struct jit_ctx *ctx, int bargs_off, int oargs_off,
const struct btf_func_model *m, const struct arg_aux *a,
bool for_call_origin, bool is_struct_ops, u64 arena_base)
@@ -2854,6 +2926,16 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,
emit(A64_STR64I(A64_R(19), A64_SP, regs_off), ctx);
emit(A64_STR64I(A64_R(20), A64_SP, regs_off + 8), ctx);
+ /*
+ * Tasks RCU keeps this image alive only while we are a Tasks Trace
+ * reader; the instructions before this point (and after the final
+ * unlock) are covered by the irq-exit IP check. One reader spans
+ * __bpf_tramp_enter() and the fentry/fmod_ret progs, a second one the
+ * fexit progs and __bpf_tramp_exit(); the original function runs
+ * outside both, with the image pinned by im->pcref instead.
+ */
+ emit_trace_rcu_reader(ctx, true);
+
if (flags & BPF_TRAMP_F_CALL_ORIG) {
/* for the first pass, assume the worst case */
if (!ctx->image)
@@ -2898,12 +2980,20 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,
if (flags & BPF_TRAMP_F_CALL_ORIG) {
/* the original func takes kernel addresses, never converted ones */
save_args(ctx, bargs_off, oargs_off, m, a, true, is_struct_ops, 0);
+ emit_trace_rcu_reader(ctx, false);
/* call original func */
emit(A64_LDR64I(A64_R(10), A64_SP, retaddr_off), ctx);
emit(A64_ADR(A64_LR, AARCH64_INSN_SIZE * 2), ctx);
emit(A64_RET(A64_R(10)), ctx);
/* store return value */
emit(A64_STR64I(A64_R(0), A64_SP, retval_off), ctx);
+ /*
+ * Second reader. Taken before ip_after_call so that the branch
+ * to the epilogue patched in at teardown is inside it too; the
+ * fmod_ret early exit lands past this still holding the first
+ * reader, so either way exactly one is held.
+ */
+ emit_trace_rcu_reader(ctx, true);
/* reserve a nop for bpf_tramp_image_put */
im->ip_after_call = ctx->ro_image + ctx->idx;
emit(A64_NOP, ctx);
@@ -2945,6 +3035,9 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,
if (flags & BPF_TRAMP_F_RESTORE_REGS)
restore_args(ctx, bargs_off, a->regs_for_args);
+ /* Remaining instructions are covered by the irq-exit IP check. */
+ emit_trace_rcu_reader(ctx, false);
+
/* restore callee saved register x19 and x20 */
emit(A64_LDR64I(A64_R(19), A64_SP, regs_off), ctx);
emit(A64_LDR64I(A64_R(20), A64_SP, regs_off + 8), ctx);
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 15fd9ec5ecacb..64c3814eb7459 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -288,6 +288,7 @@ config X86
select MMU_GATHER_RCU_TABLE_FREE
select MMU_GATHER_MERGE_VMAS
select HAVE_POSIX_CPU_TIMERS_TASK_WORK
+ select HAVE_RCU_TRAMPOLINE_READERS if X86_64 && SMP && DYNAMIC_FTRACE
select HAVE_REGS_AND_STACK_ACCESS_API
select HAVE_RELIABLE_STACKTRACE if UNWINDER_ORC || STACK_VALIDATION
select HAVE_FUNCTION_ARG_ACCESS_API
diff --git a/arch/x86/kernel/asm-offsets.c b/arch/x86/kernel/asm-offsets.c
index 081816888f7a0..876c3986419a2 100644
--- a/arch/x86/kernel/asm-offsets.c
+++ b/arch/x86/kernel/asm-offsets.c
@@ -9,6 +9,7 @@
#include <linux/crypto.h>
#include <crypto/aria.h>
#include <linux/sched.h>
+#include <linux/srcu.h>
#include <linux/stddef.h>
#include <linux/hardirq.h>
#include <linux/suspend.h>
@@ -46,6 +47,13 @@ static void __used common(void)
#ifdef CONFIG_STACKPROTECTOR
OFFSET(TASK_stack_canary, task_struct, stack_canary);
#endif
+#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS
+ OFFSET(TASK_trc_reader_nesting, task_struct, trc_reader_nesting);
+ OFFSET(TASK_trc_reader_scp, task_struct, trc_reader_scp);
+ OFFSET(SRCU_srcu_ctrp, srcu_struct, srcu_ctrp);
+ OFFSET(SRCU_CTR_srcu_locks, srcu_ctr, srcu_locks);
+ OFFSET(SRCU_CTR_srcu_unlocks, srcu_ctr, srcu_unlocks);
+#endif
BLANK();
OFFSET(pbe_address, pbe, address);
diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c
index 17d6edfcb7e00..9babaed483eb7 100644
--- a/arch/x86/kernel/ftrace.c
+++ b/arch/x86/kernel/ftrace.c
@@ -275,6 +275,49 @@ static inline void tramp_free(void *tramp)
execmem_free(tramp);
}
+#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS
+extern void ftrace_static_tramp_end(void);
+extern char __return_thunk_start[], __return_thunk_end[];
+extern char __rethunk_safe_start[], __rethunk_safe_end[];
+
+/*
+ * The SRCU-fast increments in TRACE_RCU_READ_LOCK/UNLOCK (ftrace_64.S) are the
+ * this_cpu_inc() form.
+ */
+static_assert(!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE));
+
+/*
+ * See rcu_tasks_trampoline_text(). Some core kernel text behaves like a
+ * trampoline for Tasks RCU purposes because a task executing there outside
+ * any Tasks Trace reader may still be about to enter a Tasks-RCU-protected
+ * trampoline whose address it already holds:
+ *
+ * - the static ftrace_caller / ftrace_regs_caller / ftrace_stub_direct_tramp
+ * stubs, which carry a direct-call target on the stack until their final
+ * RET, and
+ * - the return thunks that RET expands to under CONFIG_MITIGATION_RETHUNK,
+ * which run after leaving the stubs above and before landing in that
+ * target.
+ */
+bool arch_rcu_tasks_trampoline_text(unsigned long ip)
+{
+ if (ip >= (unsigned long)ftrace_caller &&
+ ip < (unsigned long)ftrace_static_tramp_end)
+ return true;
+#ifdef CONFIG_MITIGATION_RETPOLINE
+ if (ip >= (unsigned long)__return_thunk_start &&
+ ip < (unsigned long)__return_thunk_end)
+ return true;
+#endif
+#ifdef CONFIG_MITIGATION_SRSO
+ if (ip >= (unsigned long)__rethunk_safe_start &&
+ ip < (unsigned long)__rethunk_safe_end)
+ return true;
+#endif
+ return false;
+}
+#endif /* CONFIG_TASKS_RCU_TRAMPOLINE_READERS */
+
/* Defined as markers to the end of the ftrace default trampolines */
extern void ftrace_regs_caller_end(void);
extern void ftrace_caller_end(void);
diff --git a/arch/x86/kernel/ftrace_64.S b/arch/x86/kernel/ftrace_64.S
index 62c1c93aa1c6a..5d8cb38619781 100644
--- a/arch/x86/kernel/ftrace_64.S
+++ b/arch/x86/kernel/ftrace_64.S
@@ -7,6 +7,7 @@
#include <linux/cfi_types.h>
#include <linux/linkage.h>
#include <asm/asm-offsets.h>
+#include <asm/percpu.h>
#include <asm/ptrace.h>
#include <asm/ftrace.h>
#include <asm/nospec-branch.h>
@@ -145,6 +146,53 @@ SYM_FUNC_END(ftrace_stub_graph)
#ifdef CONFIG_DYNAMIC_FTRACE
+/*
+ * Open-coded rcu_read_lock_trace() / rcu_read_unlock_trace(), see
+ * include/linux/rcupdate_trace.h and CONFIG_HAVE_RCU_TRAMPOLINE_READERS: the
+ * trampoline and the ftrace_ops it is about to load are kept alive by Tasks
+ * RCU only while we are inside this reader, so the lock must precede the
+ * function_trace_op load and the unlock must follow the call. These live
+ * inside the region copied into dynamic trampolines; the %rip-relative
+ * references are fixed up by text_poke_apply_relocation() in
+ * create_trampoline(). Clobbers %rax, %rcx and flags.
+ */
+.macro TRACE_RCU_READ_LOCK
+#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS
+ movq PER_CPU_VAR(current_task), %rcx
+ movl TASK_trc_reader_nesting(%rcx), %eax
+ incl TASK_trc_reader_nesting(%rcx)
+ testl %eax, %eax
+ jnz .Ltrl_nested_\@
+ movq rcu_tasks_trace_srcu_struct+SRCU_srcu_ctrp(%rip), %rax
+ incq %gs:SRCU_CTR_srcu_locks(%rax)
+ movq %rax, TASK_trc_reader_scp(%rcx)
+#ifndef CONFIG_TASKS_TRACE_RCU_NO_MB
+ lock addl $0, -4(%rsp) /* smp_mb() */
+#endif
+.Ltrl_nested_\@:
+#endif
+.endm
+
+.macro TRACE_RCU_READ_UNLOCK
+#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS
+ movq PER_CPU_VAR(current_task), %rcx
+ movl TASK_trc_reader_nesting(%rcx), %eax
+ subl $1, %eax
+ jnz .Ltru_nested_\@
+ /* Outermost: pick up scp before an interrupt can see nesting == 0. */
+ movq TASK_trc_reader_scp(%rcx), %rax
+ movl $0, TASK_trc_reader_nesting(%rcx)
+#ifndef CONFIG_TASKS_TRACE_RCU_NO_MB
+ lock addl $0, -4(%rsp) /* smp_mb() */
+#endif
+ incq %gs:SRCU_CTR_srcu_unlocks(%rax)
+ jmp .Ltru_done_\@
+.Ltru_nested_\@:
+ movl %eax, TASK_trc_reader_nesting(%rcx)
+.Ltru_done_\@:
+#endif
+.endm
+
SYM_FUNC_START(__fentry__)
ANNOTATE_NOENDBR
CALL_DEPTH_ACCOUNT
@@ -163,6 +211,8 @@ SYM_FUNC_START(ftrace_caller)
leaq MCOUNT_REG_SIZE+8(%rsp), %rcx
movq %rcx, RSP(%rsp)
+ TRACE_RCU_READ_LOCK
+
SYM_INNER_LABEL(ftrace_caller_op_ptr, SYM_L_GLOBAL)
ANNOTATE_NOENDBR
/* Load the ftrace_ops into the 3rd parameter */
@@ -181,6 +231,8 @@ SYM_INNER_LABEL(ftrace_call, SYM_L_GLOBAL)
ANNOTATE_NOENDBR
call ftrace_stub
+ TRACE_RCU_READ_UNLOCK
+
/* Handlers can change the RIP */
movq RIP(%rsp), %rax
movq %rax, MCOUNT_REG_SIZE(%rsp)
@@ -209,6 +261,8 @@ SYM_FUNC_START(ftrace_regs_caller)
CALL_DEPTH_ACCOUNT
+ TRACE_RCU_READ_LOCK
+
SYM_INNER_LABEL(ftrace_regs_caller_op_ptr, SYM_L_GLOBAL)
ANNOTATE_NOENDBR
/* Load the ftrace_ops into the 3rd parameter */
@@ -246,6 +300,8 @@ SYM_INNER_LABEL(ftrace_regs_call, SYM_L_GLOBAL)
ANNOTATE_NOENDBR
call ftrace_stub
+ TRACE_RCU_READ_UNLOCK
+
/* Copy flags back to SS, to restore them */
movq EFLAGS(%rsp), %rax
movq %rax, MCOUNT_REG_SIZE(%rsp)
@@ -328,6 +384,19 @@ SYM_FUNC_START(ftrace_stub_direct_tramp)
RET
SYM_FUNC_END(ftrace_stub_direct_tramp)
+/*
+ * [ftrace_caller, ftrace_static_tramp_end) is treated as trampoline text by
+ * rcu_tasks_trampoline_text(): outside TRACE_RCU_READ_LOCK/UNLOCK the stubs
+ * may still hold a direct-call trampoline address (ORIG_RAX / the return
+ * address they RET to) that only Tasks RCU keeps alive. With return thunks
+ * the RET itself runs elsewhere; arch_rcu_tasks_trampoline_text() covers
+ * those too.
+ */
+SYM_CODE_START_NOALIGN(ftrace_static_tramp_end)
+ UNWIND_HINT_UNDEFINED
+ ANNOTATE_NOENDBR
+SYM_CODE_END(ftrace_static_tramp_end)
+
#else /* ! CONFIG_DYNAMIC_FTRACE */
SYM_FUNC_START(__fentry__)
diff --git a/arch/x86/kernel/kprobes/opt.c b/arch/x86/kernel/kprobes/opt.c
index 3f8fea52619f9..68a5de6cdabe7 100644
--- a/arch/x86/kernel/kprobes/opt.c
+++ b/arch/x86/kernel/kprobes/opt.c
@@ -31,6 +31,7 @@
#include <asm/set_memory.h>
#include <asm/sections.h>
#include <asm/nospec-branch.h>
+#include <asm/asm-offsets.h>
#include "common.h"
@@ -101,6 +102,47 @@ static void synthesize_set_arg1(kprobe_opcode_t *addr, unsigned long val)
*(unsigned long *)addr = val;
}
+/*
+ * Open-coded rcu_read_lock_trace() / rcu_read_unlock_trace() around the call
+ * to optimized_callback(), see CONFIG_HAVE_RCU_TRAMPOLINE_READERS and the
+ * equivalent macros in ftrace_64.S. The template is memcpy()d into the slot
+ * without relocation processing, so memory references must be absolute
+ * rather than %rip-relative. %rax and %rcx are free at both points.
+ */
+#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS
+#ifndef CONFIG_TASKS_TRACE_RCU_NO_MB
+#define OPTPROBE_TRACE_RCU_MB " lock addl $0, -4(%rsp)\n"
+#else
+#define OPTPROBE_TRACE_RCU_MB
+#endif
+#define OPTPROBE_TRACE_RCU_READ_LOCK \
+ " movq %gs:current_task, %rcx\n" \
+ " movl " __stringify(TASK_trc_reader_nesting) "(%rcx), %eax\n" \
+ " incl " __stringify(TASK_trc_reader_nesting) "(%rcx)\n" \
+ " testl %eax, %eax\n" \
+ " jnz 1f\n" \
+ " movq rcu_tasks_trace_srcu_struct+" __stringify(SRCU_srcu_ctrp) ", %rax\n" \
+ " incq %gs:" __stringify(SRCU_CTR_srcu_locks) "(%rax)\n" \
+ " movq %rax, " __stringify(TASK_trc_reader_scp) "(%rcx)\n" \
+ OPTPROBE_TRACE_RCU_MB \
+ "1:\n"
+#define OPTPROBE_TRACE_RCU_READ_UNLOCK \
+ " movq %gs:current_task, %rcx\n" \
+ " movl " __stringify(TASK_trc_reader_nesting) "(%rcx), %eax\n" \
+ " subl $1, %eax\n" \
+ " jnz 2f\n" \
+ " movq " __stringify(TASK_trc_reader_scp) "(%rcx), %rax\n" \
+ " movl $0, " __stringify(TASK_trc_reader_nesting) "(%rcx)\n" \
+ OPTPROBE_TRACE_RCU_MB \
+ " incq %gs:" __stringify(SRCU_CTR_srcu_unlocks) "(%rax)\n" \
+ " jmp 3f\n" \
+ "2: movl %eax, " __stringify(TASK_trc_reader_nesting) "(%rcx)\n" \
+ "3:\n"
+#else
+#define OPTPROBE_TRACE_RCU_READ_LOCK
+#define OPTPROBE_TRACE_RCU_READ_UNLOCK
+#endif
+
asm (
".pushsection .rodata\n"
".global optprobe_template_entry\n"
@@ -114,6 +156,7 @@ asm (
"optprobe_template_clac:\n"
ASM_NOP3
SAVE_REGS_STRING
+ OPTPROBE_TRACE_RCU_READ_LOCK
" movq %rsp, %rsi\n"
".global optprobe_template_val\n"
"optprobe_template_val:\n"
@@ -122,6 +165,7 @@ asm (
".global optprobe_template_call\n"
"optprobe_template_call:\n"
ASM_NOP5
+ OPTPROBE_TRACE_RCU_READ_UNLOCK
/* Copy 'regs->flags' into 'regs->ss'. */
" movq 18*8(%rsp), %rdx\n"
" movq %rdx, 20*8(%rsp)\n"
diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S
index 2438b89a4620a..e546283dc2679 100644
--- a/arch/x86/kernel/vmlinux.lds.S
+++ b/arch/x86/kernel/vmlinux.lds.S
@@ -151,7 +151,9 @@ SECTIONS
* definition.
*/
. = srso_alias_untrain_ret | (1 << 2) | (1 << 8) | (1 << 14) | (1 << 20);
+ __rethunk_safe_start = .;
*(.text..__x86.rethunk_safe)
+ __rethunk_safe_end = .;
#endif
ALIGN_ENTRY_TEXT_END
@@ -162,7 +164,9 @@ SECTIONS
SOFTIRQENTRY_TEXT
#ifdef CONFIG_MITIGATION_RETPOLINE
*(.text..__x86.indirect_thunk)
+ __return_thunk_start = .;
*(.text..__x86.return_thunk)
+ __return_thunk_end = .;
#endif
STATIC_CALL_TEXT
*(.gnu.warning)
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index 2853e87797a78..b77e29c9599de 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -14,6 +14,7 @@
#include <linux/memory.h>
#include <linux/sort.h>
#include <linux/execmem.h>
+#include <linux/rcupdate_trace.h>
#include <asm/extable.h>
#include <asm/ftrace.h>
#include <asm/set_memory.h>
@@ -722,6 +723,91 @@ static void emit_indirect_jump(u8 **pprog, int bpf_reg, u8 *ip)
*pprog = prog;
}
+/*
+ * Open-coded rcu_read_lock_trace() / rcu_read_unlock_trace() for the
+ * trampoline, see CONFIG_HAVE_RCU_TRAMPOLINE_READERS and the equivalent
+ * macros in arch/x86/kernel/ftrace_64.S. The image is not relocated, so
+ * current_task and rcu_tasks_trace_srcu_struct are referenced by absolute
+ * (sign-extended 32-bit) address, the form the JIT already relies on for
+ * this_cpu_off. Uses r10 and r11, which are scratch at every emission
+ * point, and clobbers flags.
+ *
+ * lock: unlock:
+ * mov r11, gs:[current_task] mov r11, gs:[current_task]
+ * mov r10d, [r11+nesting_off] mov r10d, [r11+nesting_off]
+ * inc dword ptr [r11+nesting_off] sub r10d, 1
+ * test r10d, r10d jnz 2f
+ * jnz 1f mov r10, [r11+scp_off]
+ * mov r10, [&srcu.srcu_ctrp] mov dword ptr [r11+nesting_off], 0
+ * inc qword ptr gs:[r10+locks_off] (smp_mb)
+ * mov [r11+scp_off], r10 inc qword ptr gs:[r10+unlocks_off]
+ * (smp_mb) jmp 3f
+ * 1: 2: mov [r11+nesting_off], r10d
+ * 3:
+ */
+static void emit_trace_rcu_reader(u8 **pprog, bool lock)
+{
+#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS
+ const u32 nesting_off = offsetof(struct task_struct, trc_reader_nesting);
+ const u32 scp_off = offsetof(struct task_struct, trc_reader_scp);
+ const u32 locks_off = offsetof(struct srcu_ctr, srcu_locks);
+ const u32 unlocks_off = offsetof(struct srcu_ctr, srcu_unlocks);
+ const bool mb = !IS_ENABLED(CONFIG_TASKS_TRACE_RCU_NO_MB);
+ u8 *prog = *pprog;
+
+ /* The plain this_cpu_inc() form of __srcu_read_lock_fast(). */
+ BUILD_BUG_ON(IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE));
+
+ /* mov r11, gs:[abs32 current_task] */
+ EMIT2(0x65, 0x4C);
+ EMIT3(0x8B, 0x1C, 0x25);
+ EMIT((u32)(unsigned long)¤t_task, 4);
+ /* mov r10d, dword ptr [r11 + nesting_off] */
+ EMIT3_off32(0x45, 0x8B, 0x93, nesting_off);
+
+ if (lock) {
+ /* inc dword ptr [r11 + nesting_off] */
+ EMIT3_off32(0x41, 0xFF, 0x83, nesting_off);
+ /* test r10d, r10d */
+ EMIT3(0x45, 0x85, 0xD2);
+ /* jnz 1f */
+ EMIT2(X86_JNE, 8 + 8 + 7 + (mb ? 6 : 0));
+ /* mov r10, qword ptr [abs32 &rcu_tasks_trace_srcu_struct.srcu_ctrp] */
+ EMIT4(0x4C, 0x8B, 0x14, 0x25);
+ EMIT((u32)(unsigned long)&rcu_tasks_trace_srcu_struct.srcu_ctrp, 4);
+ /* inc qword ptr gs:[r10 + locks_off] */
+ EMIT4_off32(0x65, 0x49, 0xFF, 0x82, locks_off);
+ /* mov qword ptr [r11 + scp_off], r10 */
+ EMIT3_off32(0x4D, 0x89, 0x93, scp_off);
+ /* smp_mb(): lock add dword ptr [rsp - 4], 0 */
+ if (mb)
+ EMIT2_off32(0xF0, 0x83, 0x00FC2444);
+ /* 1: */
+ } else {
+ /* sub r10d, 1 */
+ EMIT4(0x41, 0x83, 0xEA, 0x01);
+ /* jnz 2f */
+ EMIT2(X86_JNE, 7 + 11 + (mb ? 6 : 0) + 8 + 2);
+ /* mov r10, qword ptr [r11 + scp_off] */
+ EMIT3_off32(0x4D, 0x8B, 0x93, scp_off);
+ /* mov dword ptr [r11 + nesting_off], 0 */
+ EMIT3_off32(0x41, 0xC7, 0x83, nesting_off);
+ EMIT(0, 4);
+ if (mb)
+ EMIT2_off32(0xF0, 0x83, 0x00FC2444);
+ /* inc qword ptr gs:[r10 + unlocks_off] */
+ EMIT4_off32(0x65, 0x49, 0xFF, 0x82, unlocks_off);
+ /* jmp 3f */
+ EMIT2(0xEB, 7);
+ /* 2: mov dword ptr [r11 + nesting_off], r10d */
+ EMIT3_off32(0x45, 0x89, 0x93, nesting_off);
+ /* 3: */
+ }
+
+ *pprog = prog;
+#endif
+}
+
static void emit_return(u8 **pprog, u8 *ip)
{
u8 *prog = *pprog;
@@ -3610,6 +3696,16 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
/* mov QWORD PTR [rbp - rbx_off], rbx */
emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_6, -rbx_off);
+ /*
+ * Tasks RCU keeps this image alive only while we are a Tasks Trace
+ * reader; the instructions before this point (and after the final
+ * unlock) are covered by the irq-exit IP check. One reader spans
+ * __bpf_tramp_enter() and the fentry/fmod_ret progs, a second one
+ * the fexit progs and __bpf_tramp_exit(); the original function runs
+ * outside both, with the image pinned by im->pcref instead.
+ */
+ emit_trace_rcu_reader(&prog, true);
+
func_meta = nr_regs;
/* Store number of argument registers of the traced function */
emit_store_stack_imm64(&prog, BPF_REG_0, -func_meta_off, func_meta);
@@ -3660,6 +3756,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
}
if (flags & BPF_TRAMP_F_CALL_ORIG) {
+ emit_trace_rcu_reader(&prog, false);
restore_regs(m, &prog, regs_off);
save_args(m, &prog, arg_stack_off, true, flags, 0);
@@ -3682,6 +3779,13 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
}
/* remember return value in a stack for bpf prog to access */
emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8);
+ /*
+ * Second reader. Taken before ip_after_call so that the
+ * ip_after_call -> ip_epilogue jump patched in at teardown is
+ * inside it too; the fmod_ret early exit jumps past this still
+ * holding the first reader, so either way exactly one is held.
+ */
+ emit_trace_rcu_reader(&prog, true);
im->ip_after_call = image + (prog - (u8 *)rw_image);
emit_nops(&prog, X86_PATCH_SIZE);
}
@@ -3737,6 +3841,9 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
LOAD_TRAMP_TAIL_CALL_CNT_PTR(stack_size);
}
+ /* Remaining instructions are covered by the irq-exit IP check. */
+ emit_trace_rcu_reader(&prog, false);
+
/* restore return value of orig_call or fentry prog back into RAX */
if (save_ret)
emit_ldx(&prog, BPF_DW, BPF_REG_0, BPF_REG_FP, -8);
diff --git a/arch/x86/xen/enlighten_pv.c b/arch/x86/xen/enlighten_pv.c
index 2c64b388f6164..3d85035f5624f 100644
--- a/arch/x86/xen/enlighten_pv.c
+++ b/arch/x86/xen/enlighten_pv.c
@@ -739,7 +739,7 @@ __visible noinstr void xen_pv_evtchn_do_upcall(struct pt_regs *regs)
inhcall = get_and_clear_inhcall();
if (inhcall && !WARN_ON_ONCE(state.exit_rcu)) {
- irqentry_exit_cond_resched();
+ irqentry_exit_cond_resched(regs);
instrumentation_end();
restore_inhcall(inhcall);
} else {
diff --git a/include/linux/irq-entry-common.h b/include/linux/irq-entry-common.h
index 0bb6c03481fad..d42568a488a7f 100644
--- a/include/linux/irq-entry-common.h
+++ b/include/linux/irq-entry-common.h
@@ -5,6 +5,7 @@
#include <linux/context_tracking.h>
#include <linux/hrtimer_rearm.h>
#include <linux/kmsan.h>
+#include <linux/rcupdate_trace.h>
#include <linux/rseq_entry.h>
#include <linux/static_call_types.h>
#include <linux/syscalls.h>
@@ -214,6 +215,7 @@ static __always_inline void __exit_to_user_mode_validate(void)
{
/* Ensure that kernel state is sane for a return to userspace */
kmap_assert_nomap();
+ rcu_tasks_trace_assert_idle();
lockdep_assert_irqs_disabled();
lockdep_sys_exit();
}
@@ -343,24 +345,25 @@ typedef struct irqentry_state {
/**
* irqentry_exit_cond_resched - Conditionally reschedule on return from interrupt
+ * @regs: Pointer to pt_regs of interrupted context
*
* Conditional reschedule with additional sanity checks.
*/
-void raw_irqentry_exit_cond_resched(void);
+void raw_irqentry_exit_cond_resched(struct pt_regs *regs);
#ifdef CONFIG_PREEMPT_DYNAMIC
#if defined(CONFIG_HAVE_PREEMPT_DYNAMIC_CALL)
#define irqentry_exit_cond_resched_dynamic_enabled raw_irqentry_exit_cond_resched
#define irqentry_exit_cond_resched_dynamic_disabled NULL
DECLARE_STATIC_CALL(irqentry_exit_cond_resched, raw_irqentry_exit_cond_resched);
-#define irqentry_exit_cond_resched() static_call(irqentry_exit_cond_resched)()
+#define irqentry_exit_cond_resched(regs) static_call(irqentry_exit_cond_resched)(regs)
#elif defined(CONFIG_HAVE_PREEMPT_DYNAMIC_KEY)
DECLARE_STATIC_KEY_TRUE(sk_dynamic_irqentry_exit_cond_resched);
-void dynamic_irqentry_exit_cond_resched(void);
-#define irqentry_exit_cond_resched() dynamic_irqentry_exit_cond_resched()
+void dynamic_irqentry_exit_cond_resched(struct pt_regs *regs);
+#define irqentry_exit_cond_resched(regs) dynamic_irqentry_exit_cond_resched(regs)
#endif
#else /* CONFIG_PREEMPT_DYNAMIC */
-#define irqentry_exit_cond_resched() raw_irqentry_exit_cond_resched()
+#define irqentry_exit_cond_resched(regs) raw_irqentry_exit_cond_resched(regs)
#endif /* CONFIG_PREEMPT_DYNAMIC */
/**
@@ -465,7 +468,7 @@ static inline void irqentry_exit_to_kernel_mode_preempt(struct pt_regs *regs,
return;
if (IS_ENABLED(CONFIG_PREEMPTION))
- irqentry_exit_cond_resched();
+ irqentry_exit_cond_resched(regs);
}
/**
diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h
index e6de7ae55bda0..74cc48c044173 100644
--- a/include/linux/kprobes.h
+++ b/include/linux/kprobes.h
@@ -530,11 +530,17 @@ static inline bool is_kprobe_insn_slot(unsigned long addr)
}
#endif /* !CONFIG_KPROBES */
-#ifndef CONFIG_OPTPROBES
+#ifdef CONFIG_OPTPROBES
+bool kprobe_in_optimized_region(unsigned long addr);
+#else /* !CONFIG_OPTPROBES */
static inline bool is_kprobe_optinsn_slot(unsigned long addr)
{
return false;
}
+static inline bool kprobe_in_optimized_region(unsigned long addr)
+{
+ return false;
+}
#endif /* !CONFIG_OPTPROBES */
#ifdef CONFIG_KRETPROBES
diff --git a/include/linux/module.h b/include/linux/module.h
index 96cc98568eea5..82ca4f774725a 100644
--- a/include/linux/module.h
+++ b/include/linux/module.h
@@ -521,6 +521,13 @@ struct module {
unsigned int num_ftrace_callsites;
unsigned long *ftrace_callsites;
#endif
+#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS
+ /*
+ * An ftrace direct-call trampoline lives in this module's text; see
+ * rcu_tasks_trampoline_text(). Sticky once set.
+ */
+ bool ftrace_direct_tramp;
+#endif
#ifdef CONFIG_KPROBES
void *kprobes_text_start;
unsigned int kprobes_text_size;
diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index 44c07a66edfff..79e4b14f83b95 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -50,6 +50,23 @@ token_context_lock_instance(RCU, RCU_BH);
/* Exported common interfaces */
void call_rcu(struct rcu_head *head, rcu_callback_t func);
void rcu_barrier_tasks(void);
+
+/*
+ * Trampoline-reader Tasks RCU (CONFIG_TASKS_RCU_TRAMPOLINE_READERS), see
+ * kernel/rcu/tasks.h. rcu_tasks_irq_resched_enter()/_exit() bracket the
+ * irq-exit preemption; rcu_tasks_trampoline_text() and the arch_ override
+ * classify an interrupted IP; rcu_tasks_wait_irq_preempted() lets a caller
+ * wait out tasks already preempted somewhere it is about to make unsafe.
+ */
+void rcu_tasks_irq_resched_enter(unsigned long ip);
+void rcu_tasks_irq_resched_exit(void);
+bool rcu_tasks_trampoline_text(unsigned long ip);
+bool arch_rcu_tasks_trampoline_text(unsigned long ip);
+#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS
+void rcu_tasks_wait_irq_preempted(bool (*inside)(unsigned long ip));
+#else
+static inline void rcu_tasks_wait_irq_preempted(bool (*inside)(unsigned long ip)) { }
+#endif
void synchronize_rcu(void);
/*
@@ -180,11 +197,16 @@ static inline void rcu_nocb_flush_deferred_wakeup(void) { }
#ifdef CONFIG_TASKS_RCU_GENERIC
# ifdef CONFIG_TASKS_RCU
-# define rcu_tasks_classic_qs(t, preempt) \
+# ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS
+void rcu_tasks_note_qs(struct task_struct *t, bool preempt);
+# define rcu_tasks_classic_qs(t, preempt) rcu_tasks_note_qs((t), (preempt))
+# else
+# define rcu_tasks_classic_qs(t, preempt) \
do { \
if (!(preempt) && READ_ONCE((t)->rcu_tasks_holdout)) \
WRITE_ONCE((t)->rcu_tasks_holdout, false); \
} while (0)
+# endif
void call_rcu_tasks(struct rcu_head *head, rcu_callback_t func);
void synchronize_rcu_tasks(void);
void rcu_tasks_torture_stats_print(char *tt, char *tf);
diff --git a/include/linux/rcupdate_trace.h b/include/linux/rcupdate_trace.h
index 273c59a032512..9eb7f92c710be 100644
--- a/include/linux/rcupdate_trace.h
+++ b/include/linux/rcupdate_trace.h
@@ -92,6 +92,12 @@ static inline void rcu_read_unlock_tasks_trace(struct srcu_ctr __percpu *scp)
* the all the other tasks exit their critical sections.
*
* For more details, please see the documentation for rcu_read_lock().
+ *
+ * CONFIG_HAVE_RCU_TRAMPOLINE_READERS architectures open-code this pair in
+ * their ftrace and BPF trampolines (arch/x86/kernel/ftrace_64.S,
+ * arch/x86/kernel/kprobes/opt.c, arch/x86/net/bpf_jit_comp.c,
+ * arch/arm64/kernel/entry-ftrace.S, arch/arm64/net/bpf_jit_comp.c,
+ * samples/ftrace/ftrace-direct.h); changes here need to be mirrored there.
*/
static inline void rcu_read_lock_trace(void)
{
@@ -211,6 +217,12 @@ unsigned long rcu_tasks_trace_batches_completed(void);
// Placeholders to enable stepwise transition.
void __init rcu_tasks_trace_suppress_unused(void);
+/* A task must never reach userspace inside an rcu_read_lock_trace() reader. */
+static inline void rcu_tasks_trace_assert_idle(void)
+{
+ WARN_ON_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) && READ_ONCE(current->trc_reader_nesting));
+}
+
#else
static inline unsigned long rcu_tasks_trace_batches_completed(void) { return 0; }
/*
@@ -220,6 +232,7 @@ static inline unsigned long rcu_tasks_trace_batches_completed(void) { return 0;
static inline void call_rcu_tasks_trace(struct rcu_head *rhp, rcu_callback_t func) { BUG(); }
static inline void rcu_read_lock_trace(void) { BUG(); }
static inline void rcu_read_unlock_trace(void) { BUG(); }
+static inline void rcu_tasks_trace_assert_idle(void) { }
#endif /* #ifdef CONFIG_TASKS_TRACE_RCU */
DEFINE_LOCK_GUARD_0(rcu_tasks_trace,
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 8b3d47a325cca..15beb44caa2c8 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -957,6 +957,7 @@ struct task_struct {
u8 rcu_tasks_holdout;
u8 rcu_tasks_idx;
int rcu_tasks_idle_cpu;
+ unsigned long rcu_tasks_irq_ip;
struct list_head rcu_tasks_holdout_list;
int rcu_tasks_exit_cpu;
struct list_head rcu_tasks_exit_list;
diff --git a/kernel/entry/common.c b/kernel/entry/common.c
index e3d381fd3d251..94318519998c8 100644
--- a/kernel/entry/common.c
+++ b/kernel/entry/common.c
@@ -6,6 +6,7 @@
#include <linux/jump_label.h>
#include <linux/kmsan.h>
#include <linux/livepatch.h>
+#include <linux/rcupdate.h>
#include <linux/resume_user_mode.h>
#include <linux/tick.h>
@@ -134,15 +135,20 @@ static inline bool arch_irqentry_exit_need_resched(void);
static inline bool arch_irqentry_exit_need_resched(void) { return true; }
#endif
-void raw_irqentry_exit_cond_resched(void)
+void raw_irqentry_exit_cond_resched(struct pt_regs *regs)
{
if (!preempt_count()) {
/* Sanity check RCU and thread stack */
rcu_irq_exit_check_preempt();
if (IS_ENABLED(CONFIG_DEBUG_ENTRY))
WARN_ON_ONCE(!on_thread_stack());
- if (need_resched() && arch_irqentry_exit_need_resched())
+ if (need_resched() && arch_irqentry_exit_need_resched()) {
+ if (IS_ENABLED(CONFIG_TASKS_RCU_TRAMPOLINE_READERS))
+ rcu_tasks_irq_resched_enter(instruction_pointer(regs));
preempt_schedule_irq();
+ if (IS_ENABLED(CONFIG_TASKS_RCU_TRAMPOLINE_READERS))
+ rcu_tasks_irq_resched_exit();
+ }
}
}
#ifdef CONFIG_PREEMPT_DYNAMIC
@@ -150,11 +156,11 @@ void raw_irqentry_exit_cond_resched(void)
DEFINE_STATIC_CALL(irqentry_exit_cond_resched, raw_irqentry_exit_cond_resched);
#elif defined(CONFIG_HAVE_PREEMPT_DYNAMIC_KEY)
DEFINE_STATIC_KEY_TRUE(sk_dynamic_irqentry_exit_cond_resched);
-void dynamic_irqentry_exit_cond_resched(void)
+void dynamic_irqentry_exit_cond_resched(struct pt_regs *regs)
{
if (!static_branch_unlikely(&sk_dynamic_irqentry_exit_cond_resched))
return;
- raw_irqentry_exit_cond_resched();
+ raw_irqentry_exit_cond_resched(regs);
}
#endif
#endif
diff --git a/kernel/fork.c b/kernel/fork.c
index 416758c8a3d43..8077336bb136b 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -1871,6 +1871,7 @@ static inline void rcu_copy_process(struct task_struct *p)
p->rcu_tasks_holdout = false;
INIT_LIST_HEAD(&p->rcu_tasks_holdout_list);
p->rcu_tasks_idle_cpu = -1;
+ p->rcu_tasks_irq_ip = 0;
INIT_LIST_HEAD(&p->rcu_tasks_exit_list);
#endif /* #ifdef CONFIG_TASKS_RCU */
#ifdef CONFIG_TASKS_TRACE_RCU
diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index 6337da5cab9e7..e460fba83e4ab 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -511,6 +511,48 @@ static struct kprobe *get_optimized_kprobe(kprobe_opcode_t *addr)
return NULL;
}
+/*
+ * True while kprobe_optimizer() is waiting for its Tasks RCU grace period.
+ * Only in that window can an interruption inside an optprobe's jump region
+ * matter to it, so kprobe_in_optimized_region() does no work otherwise.
+ */
+static bool kprobe_optimizer_waiting;
+
+/**
+ * kprobe_in_optimized_region - Could @addr be inside bytes a jump-optimized
+ * kprobe replaces?
+ * @addr: kernel text address, typically an interrupted instruction pointer
+ *
+ * kprobe_optimizer() relies on synchronize_rcu_tasks() to wait for tasks that
+ * were interrupted on an instruction boundary inside the region about to be
+ * overwritten by the optimized jump. Where Tasks RCU is built on
+ * reader-marked trampolines that region has no reader, so the irq-exit
+ * quiescent-state check asks this instead (see rcu_tasks_trampoline_text()).
+ * This is the lockless, conservative form of get_optimized_kprobe(): it does
+ * not care whether the kprobe found is, or ever will be, optimized. May be
+ * called from any context with preemption disabled; the kprobe hash is
+ * RCU-protected and every free path waits for a grace period after unhashing.
+ *
+ * The hash walk only runs while the optimizer is actually waiting. A task
+ * that was preempted in such a region before the flag went up is invisible
+ * to that check, so the optimizer first waits those out by their recorded
+ * preemption IP (rcu_tasks_wait_irq_preempted(), whose leading
+ * synchronize_rcu() also publishes the flag to every check in flight).
+ */
+bool kprobe_in_optimized_region(unsigned long addr)
+{
+ int i;
+
+ if (!READ_ONCE(kprobe_optimizer_waiting))
+ return false;
+
+ for (i = 1; i < MAX_OPTIMIZED_LENGTH / sizeof(kprobe_opcode_t); i++)
+ if (get_kprobe((kprobe_opcode_t *)addr - i))
+ return true;
+ return false;
+}
+NOKPROBE_SYMBOL(kprobe_in_optimized_region);
+
/* Optimization staging list, protected by 'kprobe_mutex' */
static LIST_HEAD(optimizing_list);
static LIST_HEAD(unoptimizing_list);
@@ -644,8 +686,16 @@ static void kprobe_optimizer(void)
* to 2nd-Nth byte of jump instruction. This wait is for avoiding it.
* Note that on non-preemptive kernel, this is transparently converted
* to synchronoze_sched() to wait for all interrupts to have completed.
+ * kprobe_optimizer_waiting lets a reader-marked-trampoline Tasks RCU
+ * recognise tasks interrupted in such a region while we wait, and
+ * rcu_tasks_wait_irq_preempted() (a no-op elsewhere) first waits
+ * out any that were preempted there before we said so; see
+ * kprobe_in_optimized_region().
*/
+ WRITE_ONCE(kprobe_optimizer_waiting, true);
+ rcu_tasks_wait_irq_preempted(kprobe_in_optimized_region);
synchronize_rcu_tasks();
+ WRITE_ONCE(kprobe_optimizer_waiting, false);
/* Step 3: Optimize kprobes after quiesence period */
do_optimize_kprobes();
diff --git a/kernel/rcu/Kconfig b/kernel/rcu/Kconfig
index 332df7a7a6347..341b68b972ef1 100644
--- a/kernel/rcu/Kconfig
+++ b/kernel/rcu/Kconfig
@@ -95,8 +95,10 @@ config FORCE_TASKS_RCU
help
This option force-enables a task-based RCU implementation
that uses only voluntary context switch (not preemption!),
- idle, and user-mode execution as quiescent states. Not for
- manual selection in most cases.
+ idle, and user-mode execution as quiescent states, or, on
+ HAVE_RCU_TRAMPOLINE_READERS architectures, the variant built on
+ Tasks Trace RCU readers in trampolines. Not for manual
+ selection in most cases.
config NEED_TASKS_RCU
bool
@@ -107,6 +109,28 @@ config TASKS_RCU
default NEED_TASKS_RCU && PREEMPTION
select IRQ_WORK
+config HAVE_RCU_TRAMPOLINE_READERS
+ bool
+ help
+ Select this if the architecture uses the generic irq entry code and
+ every trampoline whose lifetime Tasks RCU guards on it (ftrace
+ trampolines, kprobe out-of-line and optimized-probe slots, BPF
+ trampolines, out-of-line ftrace direct-call trampolines) enters a
+ Tasks Trace RCU read-side critical section before calling out of
+ the trampoline and leaves it before returning, and any core text
+ that runs on behalf of such a trampoline outside that reader is
+ reported by arch_rcu_tasks_trampoline_text(). The assembly readers
+ use the this_cpu_inc() form of SRCU-fast, hence !NEED_SRCU_NMI_SAFE.
+
+config TASKS_RCU_TRAMPOLINE_READERS
+ def_bool TASKS_RCU && HAVE_RCU_TRAMPOLINE_READERS && GENERIC_IRQ_ENTRY && !NEED_SRCU_NMI_SAFE
+ select TASKS_TRACE_RCU
+ help
+ Implement the Tasks RCU grace period as a per-CPU pass over
+ context switches and irq-exit reschedules outside trampoline text
+ plus a Tasks Trace RCU grace period, instead of waiting for every
+ task to voluntarily context switch. See kernel/rcu/tasks.h.
+
config FORCE_TASKS_RUDE_RCU
bool "Force selection of Tasks Rude RCU"
depends on RCU_EXPERT
diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c
index 794937e13e7c3..ab870ef09af0a 100644
--- a/kernel/rcu/rcutorture.c
+++ b/kernel/rcu/rcutorture.c
@@ -1142,13 +1142,23 @@ static struct rcu_torture_ops trivial_preempt_ops = {
* Definitions for RCU-tasks torture testing.
*/
+/*
+ * A classic Tasks RCU reader is any stretch of kernel code that does not
+ * voluntarily block. With CONFIG_TASKS_RCU_TRAMPOLINE_READERS a preemption
+ * outside trampoline text also ends it, and what a trampoline does to stay
+ * protected across its call-out is take a Tasks Trace reader, so model that.
+ */
static int tasks_torture_read_lock(void)
{
+ if (IS_ENABLED(CONFIG_TASKS_RCU_TRAMPOLINE_READERS))
+ rcu_read_lock_trace();
return 0;
}
static void tasks_torture_read_unlock(int idx)
{
+ if (IS_ENABLED(CONFIG_TASKS_RCU_TRAMPOLINE_READERS))
+ rcu_read_unlock_trace();
}
static void rcu_tasks_torture_deferred_free(struct rcu_torture *p)
diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h
index 627295396cd91..42ea6e0e61cbf 100644
--- a/kernel/rcu/tasks.h
+++ b/kernel/rcu/tasks.h
@@ -152,7 +152,7 @@ static struct rcu_tasks rt_name = \
.kname = #rt_name, \
}
-#ifdef CONFIG_TASKS_RCU
+#if defined(CONFIG_TASKS_RCU) && !defined(CONFIG_TASKS_RCU_TRAMPOLINE_READERS)
/* Report delay of scan exiting tasklist in rcu_tasks_postscan(). */
static void tasks_rcu_exit_stall(struct timer_list *unused);
@@ -802,7 +802,7 @@ static void rcu_tasks_torture_stats_print_generic(struct rcu_tasks *rtp, char *t
#endif // #ifndef CONFIG_TINY_RCU
-#if defined(CONFIG_TASKS_RCU)
+#if defined(CONFIG_TASKS_RCU) && !defined(CONFIG_TASKS_RCU_TRAMPOLINE_READERS)
////////////////////////////////////////////////////////////////////////
//
@@ -897,10 +897,461 @@ static void rcu_tasks_wait_gp(struct rcu_tasks *rtp)
rtp->postgp_func(rtp);
}
-#endif /* #if defined(CONFIG_TASKS_RCU) */
+#endif /* #if defined(CONFIG_TASKS_RCU) && !defined(CONFIG_TASKS_RCU_TRAMPOLINE_READERS) */
#ifdef CONFIG_TASKS_RCU
+static int rcu_tasks_lazy_ms = -1;
+module_param(rcu_tasks_lazy_ms, int, 0444);
+
+#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS
+
+////////////////////////////////////////////////////////////////////////
+//
+// Tasks RCU for architectures whose trampolines are Tasks Trace RCU
+// readers (CONFIG_HAVE_RCU_TRAMPOLINE_READERS).
+//
+// On these architectures every piece of text whose lifetime Tasks RCU
+// guards -- ftrace trampolines, kprobe optinsn slots, BPF trampoline
+// images, out-of-line ftrace direct-call trampolines -- enters a Tasks
+// Trace RCU read-side critical section before calling out of itself and
+// leaves it before returning, so a task anywhere inside such a call-out,
+// preempted or not, is an ordinary rcu_read_lock_trace() reader and
+// synchronize_rcu_tasks_trace() waits for it.
+//
+// What that cannot cover is the handful of instructions in the trampoline
+// before the reader is entered and after it is left, and the one user that
+// has no trampoline at all: the bytes after a kprobe that the jump
+// optimizer is about to overwrite. A task can only linger in such
+// "unmarked" text by being interrupted there; unmarked text never calls
+// anything that could schedule. So a context switch on a CPU tells us that
+// whatever that CPU was running is out of unmarked text, with one
+// exception: a preemption from the irq-exit path, which can happen at any
+// instruction boundary. That path has the interrupted pt_regs in hand, so
+// just before it preempts it records the IP in the task and checks it
+// (rcu_tasks_trampoline_text()); if it is inside unmarked text the task
+// goes on a short holdout list first, and takes itself off again at its
+// next context switch outside such a preemption or its next irq-exit
+// check that finds it elsewhere. With that, every pass through
+// __schedule() is a per-CPU quiescent event, as are usermode and idle.
+//
+// A grace period is then:
+//
+// 1. Wait for every online CPU to context switch or be found in an RCU
+// extended quiescent state (deep idle, nohz_full userspace), nudging
+// stragglers with resched_cpu(). Afterwards no task is in the leading
+// unmarked instructions of a dying trampoline unless it is on the
+// holdout list.
+// 2. Wait for the holdout list (as it stood) to drain.
+// 3. synchronize_rcu_tasks_trace(), for everything inside the readers.
+// 4. Repeat 1 and 2 for tasks that have since left the reader and are in
+// the trailing unmarked instructions.
+//
+// which is bounded by a few jiffies plus preempt-off latency plus an SRCU
+// grace period, independent of how long any task runs without sleeping.
+// Unlike the classic implementation this does wait for an idle task caught
+// in a trampoline, since an idle CPU only counts while RCU is not watching
+// it.
+
+static void rcu_tasks_tramp_wait_gp(struct rcu_tasks *rtp);
+void call_rcu_tasks(struct rcu_head *rhp, rcu_callback_t func);
+DEFINE_RCU_TASKS(rcu_tasks, rcu_tasks_tramp_wait_gp, call_rcu_tasks, "RCU Tasks");
+
+/* Per-CPU count of Tasks RCU quiescent events, and the GP kthread's snapshot. */
+static DEFINE_PER_CPU(unsigned long, rcu_tasks_qs_seq);
+static DEFINE_PER_CPU(unsigned long, rcu_tasks_qs_snap);
+
+/*
+ * Tasks currently switched out by an irq-exit preemption are kept, with the
+ * interrupted IP, on the per-CPU rtp_exit_list of the CPU that preempted them
+ * (reusing the list, lock and task_struct fields the classic flavor uses for
+ * its exit-path bookkeeping, which this flavor does not need), so that
+ * rcu_tasks_wait_irq_preempted() can find them without a tasklist scan and
+ * regardless of where they are in exit.
+ */
+
+/* Tasks last seen preempted inside unmarked trampoline text. */
+static LIST_HEAD(rcu_tasks_tramp_holdouts);
+static DEFINE_RAW_SPINLOCK(rcu_tasks_tramp_lock);
+
+/* CPUs / holdouts the current grace period is still waiting for. */
+static struct cpumask rcu_tasks_pending_cpus;
+static LIST_HEAD(rcu_tasks_gp_holdouts);
+
+/**
+ * arch_rcu_tasks_trampoline_text - Does the architecture treat @ip as unmarked trampoline text?
+ * @ip: kernel text address inside core kernel text
+ *
+ * See rcu_tasks_trampoline_text(). Architectures override this to flag
+ * core text that runs on behalf of a trampoline outside its Tasks Trace
+ * reader, e.g. static ftrace entry stubs or return thunks that hold a
+ * trampoline address they are about to jump to.
+ */
+bool __weak arch_rcu_tasks_trampoline_text(unsigned long ip)
+{
+ return false;
+}
+
+/**
+ * rcu_tasks_trampoline_text - Is @ip in text Tasks RCU protects but no reader marks?
+ * @ip: an interrupted instruction pointer
+ *
+ * True when a task interrupted at @ip may be executing, or about to enter
+ * or return into, text whose lifetime depends on synchronize_rcu_tasks()
+ * without being inside the Tasks Trace reader that text takes around its
+ * call-outs:
+ *
+ * - anything outside core kernel and module text (ftrace and BPF
+ * trampolines, kprobe slots and other dynamically allocated text; this
+ * deliberately does not ask is_ftrace_trampoline() and friends, since
+ * text being torn down may already be unregistered there);
+ * - whatever the architecture adds via arch_rcu_tasks_trampoline_text();
+ * - the text of a module that hosts an out-of-line ftrace direct-call
+ * trampoline (see ftrace_direct_mark_module());
+ * - the bytes after a kprobe that a pending jump optimization is about to
+ * overwrite, the one synchronize_rcu_tasks() user with no trampoline.
+ *
+ * A false positive only makes the task a holdout until its next quiescent
+ * event. Called with interrupts disabled from the irq-exit path.
+ */
+bool rcu_tasks_trampoline_text(unsigned long ip)
+{
+ bool ret = true;
+
+ if (core_kernel_text(ip))
+ return arch_rcu_tasks_trampoline_text(ip) ||
+ kprobe_in_optimized_region(ip);
+
+#ifdef CONFIG_MODULES
+ scoped_guard(rcu) {
+ struct module *mod = __module_text_address(ip);
+
+ if (mod)
+ ret = READ_ONCE(mod->ftrace_direct_tramp) ||
+ kprobe_in_optimized_region(ip);
+ }
+#endif
+ return ret;
+}
+NOKPROBE_SYMBOL(rcu_tasks_trampoline_text);
+
+/* Note a Tasks RCU quiescent event on this CPU. */
+static void rcu_tasks_qs_event(void)
+{
+ unsigned long *seq;
+
+ guard(preempt_notrace)();
+ seq = this_cpu_ptr(&rcu_tasks_qs_seq);
+ /* Order a preceding rcu_tasks_tramp_hold() before the count. */
+ smp_store_release(seq, *seq + 1);
+}
+
+static void rcu_tasks_tramp_hold(struct task_struct *t)
+{
+ unsigned long flags;
+
+ if (t->rcu_tasks_holdout)
+ return;
+ raw_spin_lock_irqsave(&rcu_tasks_tramp_lock, flags);
+ list_add_tail(&t->rcu_tasks_holdout_list, &rcu_tasks_tramp_holdouts);
+ WRITE_ONCE(t->rcu_tasks_holdout, true);
+ raw_spin_unlock_irqrestore(&rcu_tasks_tramp_lock, flags);
+}
+
+static void rcu_tasks_tramp_release(struct task_struct *t)
+{
+ unsigned long flags;
+
+ if (likely(!t->rcu_tasks_holdout))
+ return;
+ raw_spin_lock_irqsave(&rcu_tasks_tramp_lock, flags);
+ list_del_init(&t->rcu_tasks_holdout_list);
+ WRITE_ONCE(t->rcu_tasks_holdout, false);
+ raw_spin_unlock_irqrestore(&rcu_tasks_tramp_lock, flags);
+}
+
+/**
+ * rcu_tasks_irq_resched_enter - Tasks RCU hook for the irq-exit reschedule check
+ * @ip: instruction pointer of the interrupted (task-level) context
+ *
+ * Called with interrupts disabled when an interrupt returning to kernel
+ * mode is about to preempt_schedule_irq(), the one context switch that can
+ * catch a task inside unmarked trampoline text. Record where the task is
+ * parked for as long as it is (rcu_tasks_wait_irq_preempted() looks at
+ * that), and if it is inside such text make it a holdout before
+ * __schedule() reports the quiescent event; if it is not, this is as good
+ * as a voluntary switch for ending an earlier hold.
+ */
+void rcu_tasks_irq_resched_enter(unsigned long ip)
+{
+ struct task_struct *t = current;
+ struct rcu_tasks_percpu *rtpcp = this_cpu_ptr(rcu_tasks.rtpcpu);
+
+ lockdep_assert_irqs_disabled();
+ WRITE_ONCE(t->rcu_tasks_irq_ip, ip);
+ t->rcu_tasks_exit_cpu = smp_processor_id();
+ raw_spin_lock_rcu_node(rtpcp);
+ list_add(&t->rcu_tasks_exit_list, &rtpcp->rtp_exit_list);
+ raw_spin_unlock_rcu_node(rtpcp);
+
+ if (unlikely(rcu_tasks_trampoline_text(ip)))
+ rcu_tasks_tramp_hold(t);
+ else
+ rcu_tasks_tramp_release(t);
+}
+NOKPROBE_SYMBOL(rcu_tasks_irq_resched_enter);
+
+/**
+ * rcu_tasks_irq_resched_exit - preempt_schedule_irq() has returned
+ *
+ * The task is running again (possibly elsewhere) and about to return to the
+ * interrupted context; it is no longer parked anywhere.
+ */
+void rcu_tasks_irq_resched_exit(void)
+{
+ struct task_struct *t = current;
+ struct rcu_tasks_percpu *rtpcp = per_cpu_ptr(rcu_tasks.rtpcpu, t->rcu_tasks_exit_cpu);
+
+ lockdep_assert_irqs_disabled();
+ raw_spin_lock_rcu_node(rtpcp);
+ list_del_init(&t->rcu_tasks_exit_list);
+ raw_spin_unlock_rcu_node(rtpcp);
+ WRITE_ONCE(t->rcu_tasks_irq_ip, 0);
+}
+NOKPROBE_SYMBOL(rcu_tasks_irq_resched_exit);
+
+/**
+ * rcu_tasks_note_qs - Tasks RCU hook for a context switch or explicit QS
+ * @t: current
+ * @preempt: this is a preemption rather than a voluntary switch
+ *
+ * Every pass through __schedule() (and cond_resched_tasks_rcu_qs(), and a
+ * tick from userspace or idle) is a quiescent event for this CPU: unmarked
+ * trampoline text never calls anything that schedules, and the irq-exit
+ * path has already made @t a holdout if it is preempting inside such text.
+ * Any of these outside an irq-exit preemption also shows @t itself to be
+ * outside, ending an earlier hold -- including cond_resched() under
+ * PREEMPT_DYNAMIC's none/voluntary modes, where the irq-exit path is off.
+ */
+void rcu_tasks_note_qs(struct task_struct *t, bool preempt)
+{
+ WARN_ON_ONCE(t != current);
+ if (!READ_ONCE(t->rcu_tasks_irq_ip))
+ rcu_tasks_tramp_release(t);
+ rcu_tasks_qs_event();
+}
+EXPORT_SYMBOL_GPL(rcu_tasks_note_qs); /* cond_resched_tasks_rcu_qs() */
+
+/**
+ * rcu_tasks_wait_irq_preempted - wait for tasks irq-preempted inside @inside
+ * @inside: predicate on a task's recorded irq-exit preemption IP
+ *
+ * For a caller about to make some ordinary text unsafe to be parked in
+ * (the kprobe jump optimizer): once the caller has arranged for
+ * rcu_tasks_trampoline_text() to cover that text, new irq-exit preemptions
+ * there become holdouts, but a task preempted there earlier is invisible
+ * to the grace period. Wait until no parked task's recorded preemption IP
+ * is inside; a following synchronize_rcu_tasks() then covers the rest.
+ * The leading synchronize_rcu() orders the caller's arrangement against
+ * preemptions in flight, which run with interrupts disabled.
+ */
+void rcu_tasks_wait_irq_preempted(bool (*inside)(unsigned long ip))
+{
+ struct task_struct *t;
+ unsigned long flags;
+ int cpu, kick;
+ bool found;
+
+ synchronize_rcu();
+ for (;;) {
+ found = false;
+ for_each_possible_cpu(cpu) {
+ struct rcu_tasks_percpu *rtpcp = per_cpu_ptr(rcu_tasks.rtpcpu, cpu);
+
+ kick = -1;
+ raw_spin_lock_irqsave_rcu_node(rtpcp, flags);
+ list_for_each_entry(t, &rtpcp->rtp_exit_list, rcu_tasks_exit_list) {
+ if (inside(READ_ONCE(t->rcu_tasks_irq_ip))) {
+ found = true;
+ if (task_curr(t))
+ kick = task_cpu(t);
+ }
+ }
+ raw_spin_unlock_irqrestore_rcu_node(rtpcp, flags);
+ if (kick >= 0)
+ resched_cpu(kick);
+ }
+ if (!found)
+ return;
+ schedule_timeout_uninterruptible(1);
+ }
+}
+
+/* Has @cpu passed a quiescent event since the snapshot, or need it not? */
+static bool rcu_tasks_cpu_quiescent(int cpu)
+{
+ if (!cpu_online(cpu))
+ return true;
+ /* Pairs with the release in rcu_tasks_qs_event(). */
+ if (smp_load_acquire(per_cpu_ptr(&rcu_tasks_qs_seq, cpu)) !=
+ per_cpu(rcu_tasks_qs_snap, cpu))
+ return true;
+ /*
+ * Idle or nohz_full userspace in an RCU extended quiescent state: no
+ * task-level kernel frames can be live in a trampoline there, and
+ * whatever ran before has switched out. An idle CPU that RCU is
+ * watching (an interrupt from idle, or the traceable part of the idle
+ * loop) is deliberately not let through: the idle task may be in a
+ * trampoline with that interrupt on top, and since it is never
+ * preempted from irq exit nothing else would catch it. It gets the
+ * resched_cpu() like anyone else and counts once the idle loop itself
+ * schedules, which it cannot do from inside a trampoline.
+ */
+ return !(ct_rcu_watching_cpu(cpu) & CT_RCU_WATCHING);
+}
+
+/* Rate-limited stall report; returns true if the caller should add detail. */
+static bool rcu_tasks_tramp_stall(struct rcu_tasks *rtp, unsigned long *lastreport,
+ const char *what)
+{
+ int rtst = READ_ONCE(rcu_task_stall_timeout);
+
+ if (rtst <= 0 || !time_after(jiffies, *lastreport + rtst))
+ return false;
+ *lastreport = jiffies;
+ pr_err("INFO: %s: %s, grace period %lu is %lu jiffies old\n", rtp->kname,
+ what, rcu_seq_current(&rtp->tasks_gp_seq), jiffies - rtp->gp_start);
+ return true;
+}
+
+/*
+ * Steps 1/4: wait until every online CPU has context switched or is in an
+ * RCU extended quiescent state. A CPU that has neither after a jiffy is
+ * asked to switch with resched_cpu(), which takes it through
+ * rcu_tasks_irq_resched_enter() and __schedule() (or, from userspace, a
+ * guest or the idle loop, straight to __schedule()).
+ */
+static void rcu_tasks_tramp_wait_cpus(struct rcu_tasks *rtp, unsigned long *lastreport)
+{
+ struct cpumask *pending = &rcu_tasks_pending_cpus;
+ unsigned long start;
+ int cpu;
+
+ /*
+ * The quiescent events run with preemption (in practice interrupts)
+ * disabled, so after this any event we go on to count began after the
+ * caller's updates -- the unpublished trampoline, and whatever
+ * rcu_tasks_trampoline_text() consults -- were visible to it.
+ */
+ synchronize_rcu();
+
+ start = jiffies;
+ for_each_online_cpu(cpu) {
+ per_cpu(rcu_tasks_qs_snap, cpu) = READ_ONCE(per_cpu(rcu_tasks_qs_seq, cpu));
+ __cpumask_set_cpu(cpu, pending);
+ }
+ /* Snapshots before the checks below; pairs with rcu_tasks_qs_event(). */
+ smp_mb();
+
+ for (;;) {
+ for_each_cpu(cpu, pending)
+ if (rcu_tasks_cpu_quiescent(cpu))
+ __cpumask_clear_cpu(cpu, pending);
+ if (cpumask_empty(pending))
+ break;
+ if (time_after(jiffies, start)) {
+ for_each_cpu(cpu, pending)
+ resched_cpu(cpu);
+ rtp->n_ipis += cpumask_weight(pending);
+ }
+ schedule_timeout_idle(1);
+ if (rcu_tasks_tramp_stall(rtp, lastreport, "CPUs without a quiescent event"))
+ pr_err("\tCPUs: %*pbl\n", cpumask_pr_args(pending));
+ }
+}
+
+/*
+ * Steps 2/4: wait for the tasks that were holdouts when we looked to stop
+ * being holdouts. They are moved to a private list so that tasks becoming
+ * holdouts later (in live trampolines) cannot keep us here; each removes
+ * itself via rcu_tasks_tramp_release() wherever it is queued.
+ */
+static void rcu_tasks_tramp_wait_holdouts(struct rcu_tasks *rtp, unsigned long *lastreport)
+{
+ struct task_struct *t;
+ unsigned long flags;
+ int cpu;
+
+ raw_spin_lock_irqsave(&rcu_tasks_tramp_lock, flags);
+ list_splice_tail_init(&rcu_tasks_tramp_holdouts, &rcu_tasks_gp_holdouts);
+ raw_spin_unlock_irqrestore(&rcu_tasks_tramp_lock, flags);
+
+ for (;;) {
+ struct cpumask *kick = &rcu_tasks_pending_cpus;
+ struct task_struct *show[8];
+ int nshow = 0, i;
+ bool empty, report;
+
+ report = rcu_tasks_tramp_stall(rtp, lastreport,
+ "tasks preempted in trampoline text");
+ cpumask_clear(kick);
+ raw_spin_lock_irqsave(&rcu_tasks_tramp_lock, flags);
+ empty = list_empty(&rcu_tasks_gp_holdouts);
+ list_for_each_entry(t, &rcu_tasks_gp_holdouts, rcu_tasks_holdout_list) {
+ if (task_curr(t))
+ __cpumask_set_cpu(task_cpu(t), kick);
+ if (report && nshow < ARRAY_SIZE(show))
+ show[nshow++] = get_task_struct(t);
+ }
+ raw_spin_unlock_irqrestore(&rcu_tasks_tramp_lock, flags);
+ /* Never printk under the lock the irq-exit path takes. */
+ for (i = 0; i < nshow; i++) {
+ sched_show_task(show[i]);
+ put_task_struct(show[i]);
+ }
+ if (empty)
+ break;
+ for_each_cpu(cpu, kick)
+ resched_cpu(cpu);
+ rtp->n_ipis += cpumask_weight(kick);
+ schedule_timeout_idle(1);
+ }
+}
+
+/* Wait for one trampoline-reader Tasks RCU grace period. */
+static void rcu_tasks_tramp_wait_gp(struct rcu_tasks *rtp)
+{
+ unsigned long lastreport = jiffies;
+
+ set_tasks_gp_state(rtp, RTGS_WAIT_SCAN_HOLDOUTS);
+ rcu_tasks_tramp_wait_cpus(rtp, &lastreport);
+ rcu_tasks_tramp_wait_holdouts(rtp, &lastreport);
+
+ set_tasks_gp_state(rtp, RTGS_WAIT_READERS);
+ synchronize_rcu_tasks_trace();
+
+ set_tasks_gp_state(rtp, RTGS_SCAN_HOLDOUTS);
+ rcu_tasks_tramp_wait_cpus(rtp, &lastreport);
+ rcu_tasks_tramp_wait_holdouts(rtp, &lastreport);
+
+ set_tasks_gp_state(rtp, RTGS_POST_GP);
+}
+
+static int __init rcu_spawn_tasks_kthread(void)
+{
+ rcu_tasks.gp_sleep = HZ / 10;
+ if (rcu_tasks_lazy_ms >= 0)
+ rcu_tasks.lazy_jiffies = msecs_to_jiffies(rcu_tasks_lazy_ms);
+ rcu_tasks.wait_state = TASK_IDLE;
+ rcu_spawn_tasks_kthread_generic(&rcu_tasks);
+ return 0;
+}
+
+void exit_tasks_rcu_start(void) { }
+void exit_tasks_rcu_finish(void) { }
+
+#else /* #ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS */
+
////////////////////////////////////////////////////////////////////////
//
// Simple variant of RCU whose quiescent states are voluntary context
@@ -1173,6 +1624,8 @@ static void tasks_rcu_exit_stall(struct timer_list *unused)
#endif // #ifndef CONFIG_TINY_RCU
}
+#endif /* #else #ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS */
+
/**
* call_rcu_tasks() - Queue an RCU for invocation task-based grace period
* @rhp: structure to be used for queueing the RCU updates.
@@ -1187,6 +1640,12 @@ static void tasks_rcu_exit_stall(struct timer_list *unused)
* primitives analogous to rcu_read_lock() and rcu_read_unlock() because
* this primitive is intended to determine that all tasks have passed
* through a safe state, not so much for data-structure synchronization.
+ * On CONFIG_TASKS_RCU_TRAMPOLINE_READERS kernels a preemption outside
+ * trampoline text also ends one, and a reader whose protected window
+ * spans preemptible code must additionally be a Tasks Trace RCU reader
+ * (rcu_read_lock_trace(), as the trampolines there take around their
+ * call-outs); an arbitrary stretch of preemptible kernel code is not
+ * protected.
*
* See the description of call_rcu() for more detailed information on
* memory ordering guarantees.
@@ -1205,7 +1664,9 @@ EXPORT_SYMBOL_GPL(call_rcu_tasks);
* executing rcu-tasks read-side critical sections have elapsed. These
* read-side critical sections are delimited by calls to schedule(),
* cond_resched_tasks_rcu_qs(), idle execution, userspace execution, calls
- * to synchronize_rcu_tasks(), and (in theory, anyway) cond_resched().
+ * to synchronize_rcu_tasks(), and (in theory, anyway) cond_resched();
+ * on CONFIG_TASKS_RCU_TRAMPOLINE_READERS kernels also by preemption
+ * outside trampoline text, see call_rcu_tasks().
*
* This is a very specialized primitive, intended only for a few uses in
* tracing and other situations requiring manipulation of function
@@ -1233,9 +1694,7 @@ void rcu_barrier_tasks(void)
}
EXPORT_SYMBOL_GPL(rcu_barrier_tasks);
-static int rcu_tasks_lazy_ms = -1;
-module_param(rcu_tasks_lazy_ms, int, 0444);
-
+#ifndef CONFIG_TASKS_RCU_TRAMPOLINE_READERS
static int __init rcu_spawn_tasks_kthread(void)
{
rcu_tasks.gp_sleep = HZ / 10;
@@ -1251,6 +1710,7 @@ static int __init rcu_spawn_tasks_kthread(void)
rcu_spawn_tasks_kthread_generic(&rcu_tasks);
return 0;
}
+#endif /* #ifndef CONFIG_TASKS_RCU_TRAMPOLINE_READERS */
#if !defined(CONFIG_TINY_RCU)
void show_rcu_tasks_classic_gp_kthread(void)
@@ -1279,6 +1739,7 @@ void rcu_tasks_get_gp_data(int *flags, unsigned long *gp_seq)
}
EXPORT_SYMBOL_GPL(rcu_tasks_get_gp_data);
+#ifndef CONFIG_TASKS_RCU_TRAMPOLINE_READERS
/*
* Protect against tasklist scan blind spot while the task is exiting and
* may be removed from the tasklist. Do this by adding the task to yet
@@ -1322,6 +1783,7 @@ void exit_tasks_rcu_finish(void)
list_del_init(&t->rcu_tasks_exit_list);
raw_spin_unlock_irqrestore_rcu_node(rtpcp, flags);
}
+#endif /* #ifndef CONFIG_TASKS_RCU_TRAMPOLINE_READERS */
#else /* #ifdef CONFIG_TASKS_RCU */
void exit_tasks_rcu_start(void) { }
diff --git a/kernel/rcu/update.c b/kernel/rcu/update.c
index b62735a678842..a122b8d1effb5 100644
--- a/kernel/rcu/update.c
+++ b/kernel/rcu/update.c
@@ -40,7 +40,9 @@
#include <linux/tick.h>
#include <linux/rcupdate_wait.h>
#include <linux/sched/isolation.h>
+#include <linux/context_tracking_state.h>
#include <linux/kprobes.h>
+#include <linux/module.h>
#include <linux/slab.h>
#include <linux/irq_work.h>
#include <linux/rcupdate_trace.h>
diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
index 53d5db60bfa56..f69f715913585 100644
--- a/kernel/trace/ftrace.c
+++ b/kernel/trace/ftrace.c
@@ -6076,6 +6076,29 @@ static void reset_direct(struct ftrace_ops *ops, unsigned long addr)
ops->trampoline = 0;
}
+/*
+ * A direct trampoline may live in module text rather than in dynamically
+ * allocated text that rcu_tasks_trampoline_text() recognises on its own (see
+ * samples/ftrace/ftrace-direct*.c). The trampoline itself must be a Tasks
+ * Trace reader across its call-out (see register_ftrace_direct()); marking the
+ * owning module here covers the instructions before it enters that reader and
+ * after it leaves it, where a task interrupted in the module's text must not be
+ * counted as Tasks-RCU quiescent, so that ftrace_shutdown()'s
+ * synchronize_rcu_tasks() still keeps the module text from being freed under
+ * it.
+ */
+static void ftrace_direct_mark_module(unsigned long addr)
+{
+#if defined(CONFIG_MODULES) && defined(CONFIG_TASKS_RCU_TRAMPOLINE_READERS)
+ struct module *mod;
+
+ guard(rcu)();
+ mod = __module_text_address(addr);
+ if (mod)
+ WRITE_ONCE(mod->ftrace_direct_tramp, true);
+#endif
+}
+
/**
* register_ftrace_direct - Call a custom trampoline directly
* for multiple functions registered in @ops
@@ -6090,6 +6113,17 @@ static void reset_direct(struct ftrace_ops *ops, unsigned long addr)
* and save the parameters of the function being traced, and restore them
* (or inject new ones if needed), before returning.
*
+ * Nothing but Tasks RCU keeps the trampoline at @addr alive while a task is
+ * executing it or is preempted in something it called. On architectures that
+ * select HAVE_RCU_TRAMPOLINE_READERS, Tasks RCU only waits for such a task if
+ * it is a Tasks Trace RCU reader, so the trampoline must enter one
+ * (rcu_read_lock_trace() or an assembly equivalent) before calling out and
+ * leave it before returning, just as that option requires of the in-kernel
+ * ftrace and BPF trampolines. The few instructions before and after are
+ * covered by the irq-exit check: automatically for trampolines outside kernel
+ * and module text (e.g. BPF images), and via ftrace_direct_mark_module() for
+ * trampolines in module text.
+ *
* Returns:
* 0 on success
* -EINVAL - The @ops object was already registered with this call or
@@ -6169,6 +6203,7 @@ int register_ftrace_direct(struct ftrace_ops *ops, unsigned long addr)
ops->flags |= MULTI_FLAGS;
ops->trampoline = FTRACE_REGS_ADDR;
ops->direct_call = addr;
+ ftrace_direct_mark_module(addr);
err = register_ftrace_function_nolock(ops);
if (err)
@@ -6237,6 +6272,8 @@ __modify_ftrace_direct(struct ftrace_ops *ops, unsigned long addr)
lockdep_assert_held_once(&direct_mutex);
+ ftrace_direct_mark_module(addr);
+
/* Enable the tmp_ops to have the same functions as the direct ops */
ftrace_ops_init(&tmp_ops);
tmp_ops.func_hash = ops->func_hash;
@@ -6419,6 +6456,7 @@ int update_ftrace_direct_add(struct ftrace_ops *ops, struct ftrace_hash *hash)
hlist_for_each_entry(entry, &hash->buckets[i], hlist) {
if (__ftrace_lookup_ip(direct_functions, entry->ip))
goto out_unlock;
+ ftrace_direct_mark_module(entry->direct);
}
}
@@ -6702,6 +6740,7 @@ int update_ftrace_direct_mod(struct ftrace_ops *ops, struct ftrace_hash *hash, b
tmp = __ftrace_lookup_ip(direct_hash, entry->ip);
if (!tmp)
continue;
+ ftrace_direct_mark_module(entry->direct);
tmp->direct = entry->direct;
}
}
diff --git a/samples/ftrace/ftrace-direct-modify.c b/samples/ftrace/ftrace-direct-modify.c
index 164d9dd6fd921..937c8d8c2a1be 100644
--- a/samples/ftrace/ftrace-direct-modify.c
+++ b/samples/ftrace/ftrace-direct-modify.c
@@ -2,6 +2,7 @@
#include <linux/module.h>
#include <linux/kthread.h>
#include <linux/ftrace.h>
+#include "ftrace-direct.h"
#if !defined(CONFIG_ARM64) && !defined(CONFIG_PPC32)
#include <asm/asm-offsets.h>
#endif
@@ -73,7 +74,9 @@ asm (
" pushq %rbp\n"
" movq %rsp, %rbp\n"
CALL_DEPTH_ACCOUNT
+ TRACE_RCU_READ_LOCK
" call my_direct_func1\n"
+ TRACE_RCU_READ_UNLOCK
" leave\n"
ASM_RET
" .size my_tramp1, .-my_tramp1\n"
@@ -85,7 +88,9 @@ asm (
" pushq %rbp\n"
" movq %rsp, %rbp\n"
CALL_DEPTH_ACCOUNT
+ TRACE_RCU_READ_LOCK
" call my_direct_func2\n"
+ TRACE_RCU_READ_UNLOCK
" leave\n"
ASM_RET
" .size my_tramp2, .-my_tramp2\n"
@@ -141,11 +146,13 @@ asm (
" .globl my_tramp1\n"
" my_tramp1:"
" hint 34\n" // bti c
+ TRACE_RCU_READ_LOCK
" sub sp, sp, #16\n"
" stp x9, x30, [sp]\n"
" bl my_direct_func1\n"
" ldp x30, x9, [sp]\n"
" add sp, sp, #16\n"
+ TRACE_RCU_READ_UNLOCK
" ret x9\n"
" .size my_tramp1, .-my_tramp1\n"
@@ -153,11 +160,13 @@ asm (
" .globl my_tramp2\n"
" my_tramp2:"
" hint 34\n" // bti c
+ TRACE_RCU_READ_LOCK
" sub sp, sp, #16\n"
" stp x9, x30, [sp]\n"
" bl my_direct_func2\n"
" ldp x30, x9, [sp]\n"
" add sp, sp, #16\n"
+ TRACE_RCU_READ_UNLOCK
" ret x9\n"
" .size my_tramp2, .-my_tramp2\n"
" .popsection\n"
diff --git a/samples/ftrace/ftrace-direct-multi-modify.c b/samples/ftrace/ftrace-direct-multi-modify.c
index b03766c6217bc..e12e5c8b83f06 100644
--- a/samples/ftrace/ftrace-direct-multi-modify.c
+++ b/samples/ftrace/ftrace-direct-multi-modify.c
@@ -2,6 +2,7 @@
#include <linux/module.h>
#include <linux/kthread.h>
#include <linux/ftrace.h>
+#include "ftrace-direct.h"
#if !defined(CONFIG_ARM64) && !defined(CONFIG_PPC32)
#include <asm/asm-offsets.h>
#endif
@@ -77,10 +78,12 @@ asm (
" pushq %rbp\n"
" movq %rsp, %rbp\n"
CALL_DEPTH_ACCOUNT
+ TRACE_RCU_READ_LOCK
" pushq %rdi\n"
" movq 8(%rbp), %rdi\n"
" call my_direct_func1\n"
" popq %rdi\n"
+ TRACE_RCU_READ_UNLOCK
" leave\n"
ASM_RET
" .size my_tramp1, .-my_tramp1\n"
@@ -92,10 +95,12 @@ asm (
" pushq %rbp\n"
" movq %rsp, %rbp\n"
CALL_DEPTH_ACCOUNT
+ TRACE_RCU_READ_LOCK
" pushq %rdi\n"
" movq 8(%rbp), %rdi\n"
" call my_direct_func2\n"
" popq %rdi\n"
+ TRACE_RCU_READ_UNLOCK
" leave\n"
ASM_RET
" .size my_tramp2, .-my_tramp2\n"
@@ -154,6 +159,7 @@ asm (
" .globl my_tramp1\n"
" my_tramp1:"
" hint 34\n" // bti c
+ TRACE_RCU_READ_LOCK
" sub sp, sp, #32\n"
" stp x9, x30, [sp]\n"
" str x0, [sp, #16]\n"
@@ -162,6 +168,7 @@ asm (
" ldp x30, x9, [sp]\n"
" ldr x0, [sp, #16]\n"
" add sp, sp, #32\n"
+ TRACE_RCU_READ_UNLOCK
" ret x9\n"
" .size my_tramp1, .-my_tramp1\n"
@@ -169,6 +176,7 @@ asm (
" .globl my_tramp2\n"
" my_tramp2:"
" hint 34\n" // bti c
+ TRACE_RCU_READ_LOCK
" sub sp, sp, #32\n"
" stp x9, x30, [sp]\n"
" str x0, [sp, #16]\n"
@@ -177,6 +185,7 @@ asm (
" ldp x30, x9, [sp]\n"
" ldr x0, [sp, #16]\n"
" add sp, sp, #32\n"
+ TRACE_RCU_READ_UNLOCK
" ret x9\n"
" .size my_tramp2, .-my_tramp2\n"
" .popsection\n"
diff --git a/samples/ftrace/ftrace-direct-multi.c b/samples/ftrace/ftrace-direct-multi.c
index 3fe6ddaf0b69f..a970464ed3786 100644
--- a/samples/ftrace/ftrace-direct-multi.c
+++ b/samples/ftrace/ftrace-direct-multi.c
@@ -3,6 +3,7 @@
#include <linux/mm.h> /* for handle_mm_fault() */
#include <linux/ftrace.h>
+#include "ftrace-direct.h"
#include <linux/sched/stat.h>
#if !defined(CONFIG_ARM64) && !defined(CONFIG_PPC32)
#include <asm/asm-offsets.h>
@@ -56,10 +57,12 @@ asm (
" pushq %rbp\n"
" movq %rsp, %rbp\n"
CALL_DEPTH_ACCOUNT
+ TRACE_RCU_READ_LOCK
" pushq %rdi\n"
" movq 8(%rbp), %rdi\n"
" call my_direct_func\n"
" popq %rdi\n"
+ TRACE_RCU_READ_UNLOCK
" leave\n"
ASM_RET
" .size my_tramp, .-my_tramp\n"
@@ -101,6 +104,7 @@ asm (
" .globl my_tramp\n"
" my_tramp:"
" hint 34\n" // bti c
+ TRACE_RCU_READ_LOCK
" sub sp, sp, #32\n"
" stp x9, x30, [sp]\n"
" str x0, [sp, #16]\n"
@@ -109,6 +113,7 @@ asm (
" ldp x30, x9, [sp]\n"
" ldr x0, [sp, #16]\n"
" add sp, sp, #32\n"
+ TRACE_RCU_READ_UNLOCK
" ret x9\n"
" .size my_tramp, .-my_tramp\n"
" .popsection\n"
diff --git a/samples/ftrace/ftrace-direct-too.c b/samples/ftrace/ftrace-direct-too.c
index bf2411aa6fd7a..abc098c2ab7a8 100644
--- a/samples/ftrace/ftrace-direct-too.c
+++ b/samples/ftrace/ftrace-direct-too.c
@@ -3,6 +3,7 @@
#include <linux/mm.h> /* for handle_mm_fault() */
#include <linux/ftrace.h>
+#include "ftrace-direct.h"
#if !defined(CONFIG_ARM64) && !defined(CONFIG_PPC32)
#include <asm/asm-offsets.h>
#endif
@@ -61,6 +62,7 @@ asm (
" pushq %rbp\n"
" movq %rsp, %rbp\n"
CALL_DEPTH_ACCOUNT
+ TRACE_RCU_READ_LOCK
" pushq %rdi\n"
" pushq %rsi\n"
" pushq %rdx\n"
@@ -70,6 +72,7 @@ asm (
" popq %rdx\n"
" popq %rsi\n"
" popq %rdi\n"
+ TRACE_RCU_READ_UNLOCK
" leave\n"
ASM_RET
" .size my_tramp, .-my_tramp\n"
@@ -110,6 +113,7 @@ asm (
" .globl my_tramp\n"
" my_tramp:"
" hint 34\n" // bti c
+ TRACE_RCU_READ_LOCK
" sub sp, sp, #48\n"
" stp x9, x30, [sp]\n"
" stp x0, x1, [sp, #16]\n"
@@ -119,6 +123,7 @@ asm (
" ldp x0, x1, [sp, #16]\n"
" ldp x2, x3, [sp, #32]\n"
" add sp, sp, #48\n"
+ TRACE_RCU_READ_UNLOCK
" ret x9\n"
" .size my_tramp, .-my_tramp\n"
" .popsection\n"
diff --git a/samples/ftrace/ftrace-direct.c b/samples/ftrace/ftrace-direct.c
index 5368c8c39cbb4..99b65ad2fccc0 100644
--- a/samples/ftrace/ftrace-direct.c
+++ b/samples/ftrace/ftrace-direct.c
@@ -3,6 +3,7 @@
#include <linux/sched.h> /* for wake_up_process() */
#include <linux/ftrace.h>
+#include "ftrace-direct.h"
#if !defined(CONFIG_ARM64) && !defined(CONFIG_PPC32)
#include <asm/asm-offsets.h>
#endif
@@ -54,9 +55,11 @@ asm (
" pushq %rbp\n"
" movq %rsp, %rbp\n"
CALL_DEPTH_ACCOUNT
+ TRACE_RCU_READ_LOCK
" pushq %rdi\n"
" call my_direct_func\n"
" popq %rdi\n"
+ TRACE_RCU_READ_UNLOCK
" leave\n"
ASM_RET
" .size my_tramp, .-my_tramp\n"
@@ -97,6 +100,7 @@ asm (
" .globl my_tramp\n"
" my_tramp:"
" hint 34\n" // bti c
+ TRACE_RCU_READ_LOCK
" sub sp, sp, #32\n"
" stp x9, x30, [sp]\n"
" str x0, [sp, #16]\n"
@@ -104,6 +108,7 @@ asm (
" ldp x30, x9, [sp]\n"
" ldr x0, [sp, #16]\n"
" add sp, sp, #32\n"
+ TRACE_RCU_READ_UNLOCK
" ret x9\n"
" .size my_tramp, .-my_tramp\n"
" .popsection\n"
diff --git a/samples/ftrace/ftrace-direct.h b/samples/ftrace/ftrace-direct.h
new file mode 100644
index 0000000000000..726f67048ff5c
--- /dev/null
+++ b/samples/ftrace/ftrace-direct.h
@@ -0,0 +1,126 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef _SAMPLES_FTRACE_DIRECT_H
+#define _SAMPLES_FTRACE_DIRECT_H
+
+#include <linux/stringify.h>
+
+/*
+ * A direct-call trampoline is entered with no lock, refcount or RCU marker
+ * held; only Tasks RCU keeps it (and, for a module, its text) alive while a
+ * task is inside it or preempted in something it called. On architectures
+ * that select HAVE_RCU_TRAMPOLINE_READERS, Tasks RCU only waits for such a
+ * task while it is a Tasks Trace RCU reader, so the trampoline must enter one
+ * before calling out and leave it afterwards, exactly like the ftrace and BPF
+ * trampolines do. See register_ftrace_direct(). The instructions before the
+ * lock and after the unlock are covered by ftrace_direct_mark_module().
+ *
+ * These are rcu_read_lock_trace() / rcu_read_unlock_trace() open-coded as
+ * instruction strings for use inside the samples' asm() trampolines, after
+ * the versions in arch/x86/kernel/ftrace_64.S and
+ * arch/arm64/kernel/entry-ftrace.S. The scratch registers are caller-saved
+ * and not argument registers, so they are dead on entry to and exit from an
+ * fentry trampoline; the flags are clobbered.
+ *
+ * The generated asm-offsets.h is only pulled in on the architectures that need
+ * it here: it is not generally safe to include from C (e.g. PPC32's TASK_SIZE
+ * and arm64's TRAMP_VALIAS clash with the C definitions).
+ */
+#if defined(CONFIG_TASKS_RCU_TRAMPOLINE_READERS) && defined(CONFIG_X86_64)
+
+#include <asm/asm-offsets.h>
+
+#ifndef CONFIG_TASKS_TRACE_RCU_NO_MB
+#define TRACE_RCU_MB " lock addl $0, -4(%rsp)\n"
+#else
+#define TRACE_RCU_MB
+#endif
+
+#define TRACE_RCU_READ_LOCK \
+ " movq %gs:current_task(%rip), %r11\n" \
+ " movl " __stringify(TASK_trc_reader_nesting) "(%r11), %r10d\n" \
+ " incl " __stringify(TASK_trc_reader_nesting) "(%r11)\n" \
+ " testl %r10d, %r10d\n" \
+ " jnz 771f\n" \
+ " movq rcu_tasks_trace_srcu_struct+" __stringify(SRCU_srcu_ctrp) "(%rip), %r10\n" \
+ " incq %gs:" __stringify(SRCU_CTR_srcu_locks) "(%r10)\n" \
+ " movq %r10, " __stringify(TASK_trc_reader_scp) "(%r11)\n" \
+ TRACE_RCU_MB \
+ "771:\n"
+
+#define TRACE_RCU_READ_UNLOCK \
+ " movq %gs:current_task(%rip), %r11\n" \
+ " movl " __stringify(TASK_trc_reader_nesting) "(%r11), %r10d\n" \
+ " subl $1, %r10d\n" \
+ " jnz 772f\n" \
+ " movq " __stringify(TASK_trc_reader_scp) "(%r11), %r10\n" \
+ " movl $0, " __stringify(TASK_trc_reader_nesting) "(%r11)\n" \
+ TRACE_RCU_MB \
+ " incq %gs:" __stringify(SRCU_CTR_srcu_unlocks) "(%r10)\n" \
+ " jmp 773f\n" \
+ "772: movl %r10d, " __stringify(TASK_trc_reader_nesting) "(%r11)\n" \
+ "773:\n"
+
+#elif defined(CONFIG_TASKS_RCU_TRAMPOLINE_READERS) && defined(CONFIG_ARM64)
+
+#include <asm/alternative-macros.h>
+#include <asm/cpucaps.h>
+/* arm64's asm-offsets.h redefines TRAMP_VALIAS from <asm/fixmap.h>. */
+#pragma push_macro("TRAMP_VALIAS")
+#undef TRAMP_VALIAS
+#include <asm/asm-offsets.h>
+#pragma pop_macro("TRAMP_VALIAS")
+
+#ifndef CONFIG_TASKS_TRACE_RCU_NO_MB
+#define TRACE_RCU_MB " dmb ish\n"
+#else
+#define TRACE_RCU_MB
+#endif
+
+#define TRACE_RCU_SRCU_CTRP "rcu_tasks_trace_srcu_struct+" __stringify(SRCU_SRCU_CTRP)
+
+/* x14 = this CPU's offset; then atomically increment the long at x14 + \areg */
+#define TRACE_RCU_PERCPU_INC(areg) \
+ ALTERNATIVE(" mrs x14, tpidr_el1\n", " mrs x14, tpidr_el2\n", \
+ ARM64_HAS_VIRT_HOST_EXTN) \
+ " add x14, x14, " areg "\n" \
+ "778: ldxr x15, [x14]\n" \
+ " add x15, x15, #1\n" \
+ " stxr w16, x15, [x14]\n" \
+ " cbnz w16, 778b\n"
+
+#define TRACE_RCU_READ_LOCK \
+ " mrs x12, sp_el0\n" \
+ " ldr w13, [x12, #" __stringify(TSK_TRC_READER_NESTING) "]\n" \
+ " add w14, w13, #1\n" \
+ " str w14, [x12, #" __stringify(TSK_TRC_READER_NESTING) "]\n" \
+ " cbnz w13, 771f\n" \
+ " adrp x13, " TRACE_RCU_SRCU_CTRP "\n" \
+ " ldr x13, [x13, #:lo12:" TRACE_RCU_SRCU_CTRP "]\n" \
+ " str x13, [x12, #" __stringify(TSK_TRC_READER_SCP) "]\n" \
+ " add x13, x13, #" __stringify(SRCU_CTR_SRCU_LOCKS) "\n" \
+ TRACE_RCU_PERCPU_INC("x13") \
+ TRACE_RCU_MB \
+ "771:\n"
+
+#define TRACE_RCU_READ_UNLOCK \
+ " mrs x12, sp_el0\n" \
+ " ldr w13, [x12, #" __stringify(TSK_TRC_READER_NESTING) "]\n" \
+ " subs w13, w13, #1\n" \
+ " b.ne 772f\n" \
+ " ldr x13, [x12, #" __stringify(TSK_TRC_READER_SCP) "]\n" \
+ " str wzr, [x12, #" __stringify(TSK_TRC_READER_NESTING) "]\n" \
+ TRACE_RCU_MB \
+ " add x13, x13, #" __stringify(SRCU_CTR_SRCU_UNLOCKS) "\n" \
+ TRACE_RCU_PERCPU_INC("x13") \
+ " b 773f\n" \
+ "772: str w13, [x12, #" __stringify(TSK_TRC_READER_NESTING) "]\n" \
+ "773:\n"
+
+#else
+
+#define TRACE_RCU_READ_LOCK
+#define TRACE_RCU_READ_UNLOCK
+
+#endif
+
+#endif /* _SAMPLES_FTRACE_DIRECT_H */
|
| 62/3 | 2026/09/24 04:22 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 36815 output: 21 thoughts: 1580 Thoughts: **KMSAN Fuzzing Analysis: RCU Trampoline Optimization** Okay, so the task is to determine if this patch series warrants a KMSAN fuzzing session. The guiding principle is clear: if KASAN, LOCKDEP, or any other standard bug detectors can catch a bug, KMSAN is unnecessary. KMSAN is reserved for uninitialized memory reads *exclusively* revealed by it. First, let's establish the context of the patch. It's about introducing `CONFIG_HAVE_RCU_TRAMPOLINE_READERS` and `CONFIG_TASKS_RCU_TRAMPOLINE_READERS` for RCU trampoline readers, specifically optimizing Tasks RCU. This involves changes across a wide range of files: documentation, architecture-specific Kconfigs, assembly offsets, entry points, Ftrace code, BPF JIT compilation, and core kernel components related to RCU, scheduling, and kprobes. This indicates it is a relatively broad change. Now, let's drill down into the potential for uninitialized memory reads. The key change seems to be the introduction of `rcu_tasks_irq_ip` into `task_struct`, which is zero-initialized on fork in `rcu_copy_process()`. This immediately alleviates a concern, as it's properly initialized during task creation. The modules are `kzalloc`'d when loaded, which further reduces risk. The patch includes emitting assembly for `rcu_read_lock_trace()` and `rcu_read_unlock_trace()` within the BPF JIT code. That code accesses `current->trc_reader_nesting` and `current->trc_reader_scp`. Let's see if these are uninitialized. I will need to grep for these. I've also scanned the code for data copies to user space or any allocations of structs that could be read without zeroing. No explicit `copy_to_user`, `put_user`, BPF buffer outputs to userspace, netlink skb, or ioctl output arguments are present. A look at `rcu_tasks_tramp_wait_holdouts()` showed a local `show[8]` array, but `nshow` controls how many entries of `show` are read. All read entries are initialized. Finally, the `raw_irqentry_exit_cond_resched()` function receives a `struct pt_regs *regs`. However, `regs` comes from the hardware/irq entry which should preclude the possibility of uninitialized memory. From my initial review, I don't see any obvious KMSAN-specific vulnerabilities. The important structures are initialized, and there are no direct transfers to userspace. Unless the grep search on `trc_reader_nesting` reveals something unusual, I'd say that **NeedsKMSAN=false** for this patch. However, it's always good practice to re-review after the complete greps and a final assessment on the overall impact of such a broad set of changes. |
| 63/3 | 2026/09/24 04:23 | tool | grepper |
0mArgs:{
"Expression": "trc_reader_nesting"
}Results: {
"Output": "arch/arm64/kernel/asm-offsets.c=28=int main(void)\n--\narch/arm64/kernel/asm-offsets.c-44-#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS\narch/arm64/kernel/asm-offsets.c:45: DEFINE(TSK_TRC_READER_NESTING,\toffsetof(struct task_struct, trc_reader_nesting));\narch/arm64/kernel/asm-offsets.c-46- DEFINE(TSK_TRC_READER_SCP,\toffsetof(struct task_struct, trc_reader_scp));\n--\narch/arm64/net/bpf_jit_comp.c=2604=static void emit_trace_rcu_reader(struct jit_ctx *ctx, bool lock)\n--\narch/arm64/net/bpf_jit_comp.c-2606-#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS\narch/arm64/net/bpf_jit_comp.c:2607:\tconst int nesting_off = offsetof(struct task_struct, trc_reader_nesting);\narch/arm64/net/bpf_jit_comp.c-2608-\tconst int scp_off = offsetof(struct task_struct, trc_reader_scp);\n--\narch/x86/kernel/asm-offsets.c=35=static void __used common(void)\n--\narch/x86/kernel/asm-offsets.c-50-#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS\narch/x86/kernel/asm-offsets.c:51:\tOFFSET(TASK_trc_reader_nesting, task_struct, trc_reader_nesting);\narch/x86/kernel/asm-offsets.c-52-\tOFFSET(TASK_trc_reader_scp, task_struct, trc_reader_scp);\n--\narch/x86/kernel/ftrace_64.S=144=SYM_FUNC_END(ftrace_stub_graph)\n--\narch/x86/kernel/ftrace_64.S-161-\tmovq\tPER_CPU_VAR(current_task), %rcx\narch/x86/kernel/ftrace_64.S:162:\tmovl\tTASK_trc_reader_nesting(%rcx), %eax\narch/x86/kernel/ftrace_64.S:163:\tincl\tTASK_trc_reader_nesting(%rcx)\narch/x86/kernel/ftrace_64.S-164-\ttestl\t%eax, %eax\n--\narch/x86/kernel/ftrace_64.S-178-\tmovq\tPER_CPU_VAR(current_task), %rcx\narch/x86/kernel/ftrace_64.S:179:\tmovl\tTASK_trc_reader_nesting(%rcx), %eax\narch/x86/kernel/ftrace_64.S-180-\tsubl\t$1, %eax\n--\narch/x86/kernel/ftrace_64.S-183-\tmovq\tTASK_trc_reader_scp(%rcx), %rax\narch/x86/kernel/ftrace_64.S:184:\tmovl\t$0, TASK_trc_reader_nesting(%rcx)\narch/x86/kernel/ftrace_64.S-185-#ifndef CONFIG_TASKS_TRACE_RCU_NO_MB\n--\narch/x86/kernel/ftrace_64.S-190-.Ltru_nested_\\@:\narch/x86/kernel/ftrace_64.S:191:\tmovl\t%eax, TASK_trc_reader_nesting(%rcx)\narch/x86/kernel/ftrace_64.S-192-.Ltru_done_\\@:\n--\narch/x86/kernel/kprobes/opt.c=94=static void synthesize_set_arg1(kprobe_opcode_t *addr, unsigned long val)\n--\narch/x86/kernel/kprobes/opt.c-119-\t\t\"\tmovq %gs:current_task, %rcx\\n\"\t\t\t\t\\\narch/x86/kernel/kprobes/opt.c:120:\t\t\"\tmovl \" __stringify(TASK_trc_reader_nesting) \"(%rcx), %eax\\n\"\t\\\narch/x86/kernel/kprobes/opt.c:121:\t\t\"\tincl \" __stringify(TASK_trc_reader_nesting) \"(%rcx)\\n\"\t\\\narch/x86/kernel/kprobes/opt.c-122-\t\t\"\ttestl %eax, %eax\\n\"\t\t\t\t\t\t\\\n--\narch/x86/kernel/kprobes/opt.c-130-\t\t\"\tmovq %gs:current_task, %rcx\\n\"\t\t\t\t\\\narch/x86/kernel/kprobes/opt.c:131:\t\t\"\tmovl \" __stringify(TASK_trc_reader_nesting) \"(%rcx), %eax\\n\"\t\\\narch/x86/kernel/kprobes/opt.c-132-\t\t\"\tsubl $1, %eax\\n\"\t\t\t\t\t\t\\\n--\narch/x86/kernel/kprobes/opt.c-134-\t\t\"\tmovq \" __stringify(TASK_trc_reader_scp) \"(%rcx), %rax\\n\"\t\\\narch/x86/kernel/kprobes/opt.c:135:\t\t\"\tmovl $0, \" __stringify(TASK_trc_reader_nesting) \"(%rcx)\\n\"\t\\\narch/x86/kernel/kprobes/opt.c-136-\t\tOPTPROBE_TRACE_RCU_MB\t\t\t\t\t\t\\\n--\narch/x86/kernel/kprobes/opt.c-138-\t\t\"\tjmp 3f\\n\"\t\t\t\t\t\t\t\\\narch/x86/kernel/kprobes/opt.c:139:\t\t\"2:\tmovl %eax, \" __stringify(TASK_trc_reader_nesting) \"(%rcx)\\n\"\t\\\narch/x86/kernel/kprobes/opt.c-140-\t\t\"3:\\n\"\n--\narch/x86/net/bpf_jit_comp.c=748=static void emit_trace_rcu_reader(u8 **pprog, bool lock)\n--\narch/x86/net/bpf_jit_comp.c-750-#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS\narch/x86/net/bpf_jit_comp.c:751:\tconst u32 nesting_off = offsetof(struct task_struct, trc_reader_nesting);\narch/x86/net/bpf_jit_comp.c-752-\tconst u32 scp_off = offsetof(struct task_struct, trc_reader_scp);\n--\ninclude/linux/rcupdate_trace.h=102=static inline void rcu_read_lock_trace(void)\n--\ninclude/linux/rcupdate_trace.h-107-\trcu_try_lock_acquire(\u0026rcu_tasks_trace_srcu_struct.dep_map);\ninclude/linux/rcupdate_trace.h:108:\tn = READ_ONCE(t-\u003etrc_reader_nesting);\ninclude/linux/rcupdate_trace.h:109:\tWRITE_ONCE(t-\u003etrc_reader_nesting, n + 1);\ninclude/linux/rcupdate_trace.h-110-\tif (n) {\n--\ninclude/linux/rcupdate_trace.h=129=static inline void rcu_read_unlock_trace(void)\n--\ninclude/linux/rcupdate_trace.h-134-\ninclude/linux/rcupdate_trace.h:135:\tn = READ_ONCE(t-\u003etrc_reader_nesting) - 1;\ninclude/linux/rcupdate_trace.h-136-\tif (n) {\ninclude/linux/rcupdate_trace.h:137:\t\tWRITE_ONCE(t-\u003etrc_reader_nesting, n);\ninclude/linux/rcupdate_trace.h-138-\t} else {\n--\ninclude/linux/rcupdate_trace.h-140-\t\tbarrier(); // scp before nesting to protect against interrupt handler.\ninclude/linux/rcupdate_trace.h:141:\t\tWRITE_ONCE(t-\u003etrc_reader_nesting, n);\ninclude/linux/rcupdate_trace.h-142-\t\tif (!IS_ENABLED(CONFIG_TASKS_TRACE_RCU_NO_MB))\n--\ninclude/linux/rcupdate_trace.h=221=static inline void rcu_tasks_trace_assert_idle(void)\ninclude/linux/rcupdate_trace.h-222-{\ninclude/linux/rcupdate_trace.h:223:\tWARN_ON_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) \u0026\u0026 READ_ONCE(current-\u003etrc_reader_nesting));\ninclude/linux/rcupdate_trace.h-224-}\n--\ninclude/linux/sched.h=835=struct task_struct {\n--\ninclude/linux/sched.h-966-#ifdef CONFIG_TASKS_TRACE_RCU\ninclude/linux/sched.h:967:\tint\t\t\t\ttrc_reader_nesting;\ninclude/linux/sched.h-968-\tstruct srcu_ctr __percpu\t*trc_reader_scp;\n--\ninit/init_task.c=105=struct task_struct init_task __aligned(L1_CACHE_BYTES) = {\n--\ninit/init_task.c-208-#ifdef CONFIG_TASKS_TRACE_RCU\ninit/init_task.c:209:\t.trc_reader_nesting = 0,\ninit/init_task.c-210-#endif\n--\nkernel/fork.c=1862=static inline void rcu_copy_process(struct task_struct *p)\n--\nkernel/fork.c-1877-#ifdef CONFIG_TASKS_TRACE_RCU\nkernel/fork.c:1878:\tp-\u003etrc_reader_nesting = 0;\nkernel/fork.c-1879-#endif /* #ifdef CONFIG_TASKS_TRACE_RCU */\n--\nsamples/ftrace/ftrace-direct.h-39-\t\"\tmovq %gs:current_task(%rip), %r11\\n\"\t\t\t\t\t\\\nsamples/ftrace/ftrace-direct.h:40:\t\"\tmovl \" __stringify(TASK_trc_reader_nesting) \"(%r11), %r10d\\n\"\t\t\\\nsamples/ftrace/ftrace-direct.h:41:\t\"\tincl \" __stringify(TASK_trc_reader_nesting) \"(%r11)\\n\"\t\t\\\nsamples/ftrace/ftrace-direct.h-42-\t\"\ttestl %r10d, %r10d\\n\"\t\t\t\t\t\t\t\\\n--\nsamples/ftrace/ftrace-direct.h-51-\t\"\tmovq %gs:current_task(%rip), %r11\\n\"\t\t\t\t\t\\\nsamples/ftrace/ftrace-direct.h:52:\t\"\tmovl \" __stringify(TASK_trc_reader_nesting) \"(%r11), %r10d\\n\"\t\t\\\nsamples/ftrace/ftrace-direct.h-53-\t\"\tsubl $1, %r10d\\n\"\t\t\t\t\t\t\t\\\n--\nsamples/ftrace/ftrace-direct.h-55-\t\"\tmovq \" __stringify(TASK_trc_reader_scp) \"(%r11), %r10\\n\"\t\t\\\nsamples/ftrace/ftrace-direct.h:56:\t\"\tmovl $0, \" __stringify(TASK_trc_reader_nesting) \"(%r11)\\n\"\t\t\\\nsamples/ftrace/ftrace-direct.h-57-\tTRACE_RCU_MB\t\t\t\t\t\t\t\t\\\n--\nsamples/ftrace/ftrace-direct.h-59-\t\"\tjmp 773f\\n\"\t\t\t\t\t\t\t\t\\\nsamples/ftrace/ftrace-direct.h:60:\t\"772:\tmovl %r10d, \" __stringify(TASK_trc_reader_nesting) \"(%r11)\\n\"\t\\\nsamples/ftrace/ftrace-direct.h-61-\t\"773:\\n\"\n"
} |
| 64/3 | 2026/09/24 04:23 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 42258 output: 324 thoughts: 992 Thoughts: **KMSAN Assessment: Confident `NeedsKMSAN=false`** Okay, let's break this down. First, I've confirmed the initialization of `trc_reader_nesting` and `rcu_tasks_irq_ip`. `trc_reader_nesting` is part of `task_struct` and is explicitly initialized during task creation. `rcu_tasks_irq_ip` is zeroed in `fork.c`, and I can safely assume that any fields not explicitly initialized in `init_task.c` default to zero because they are static. This means no uninitialized values are floating around at the very lowest level. I've also checked the state of `kprobe_optimizer_waiting`, which is static and zero-initialized, and `ftrace_direct_tramp`, part of `struct module` which is allocated via `kzalloc`. Now, let's consider the risks. We're primarily dealing with concurrency issues, potential use-after-free scenarios during RCU grace periods, possible out-of-bounds execution due to BPF/ftrace trampolines, and the constant threat of deadlocks. KASAN is perfectly placed to catch use-after-free and out-of-bounds issues, and LOCKDEP handles the deadlock possibilities. I've carefully checked for any user-copy interfaces (none), uninitialized heap or stack variables (none), and any conditional logic depending on uninitialized variables (also none). Now, let's revisit the KMSAN criteria. 1. Kernel structures sent to user space? Nope. 2. Conditional logic on uninitialized variables? Nope. `rcu_tasks_irq_ip` is initialized to 0 in fork, then updated/cleared on IRQ entry and exit. 3. Complex data structures with uninitialized fields? Nope. 4. Bounds checks that might expose uninitialized memory? No. The patch *does* primarily pose risks that fall squarely into KASAN's domain – use-after-free, out-of-bounds, etc. These are all related to Tasks RCU grace periods, and Tasks Trace RCU readers in trampolines. On the other hand, the patch *doesn't* introduce any uninitialized memory usage, and all allocated structures are properly initialized. In short: KASAN and LOCKDEP cover the risks. My double-check confirms this. `raw_irqentry_exit_cond_resched`, the assembly macros for `ftrace`, and the various structures are all accounted for and initialized. No user space copy, no uninitialized memory usage. KASAN and LOCKDEP are fully appropriate. Therefore, I'm confident in setting `NeedsKMSAN=false`. The reasoning is clear, and the risks are firmly within the scope of KASAN's capabilities. |
| 65/3 | 2026/09/24 04:23 | tool | set-results |
0mArgs:{
"KMSANReasoning": "The patch series implements trampoline-reader Tasks RCU (CONFIG_TASKS_RCU_TRAMPOLINE_READERS), which replaces the classic voluntary-context-switch Tasks RCU grace period mechanism with Tasks Trace RCU read-side critical sections around ftrace, BPF, and kprobe trampolines and checks on irq-exit preemption.\n\nKey observations:\n1. No uninitialized memory usage or exposure: All new/modified fields in core structures (such as `task_struct.rcu_tasks_irq_ip` and `module.ftrace_direct_tramp`) are explicitly initialized on task creation in `rcu_copy_process()` and in zero-allocated module structs. Global and per-CPU state variables are statically defined and zero-initialized.\n2. No data copied or leaked to user space: The changes do not touch `copy_to_user`, `put_user`, netlink, socket options, or ioctl outputs.\n3. Nature of bugs and applicable detectors: Any bugs introduced by this series would manifest as synchronization races, premature grace period completion leading to use-after-free of freed trampoline text or metadata, or locking/deadlock issues. These failure modes are strictly the domain of KASAN (for use-after-free and memory corruption) and LOCKDEP (for locking violations), not KMSAN.\n\nTherefore, dedicated KMSAN fuzzing is not warranted.",
"NeedsKMSAN": false
}Results: {
"KMSANReasoning": "The patch series implements trampoline-reader Tasks RCU (CONFIG_TASKS_RCU_TRAMPOLINE_READERS), which replaces the classic voluntary-context-switch Tasks RCU grace period mechanism with Tasks Trace RCU read-side critical sections around ftrace, BPF, and kprobe trampolines and checks on irq-exit preemption.\n\nKey observations:\n1. No uninitialized memory usage or exposure: All new/modified fields in core structures (such as `task_struct.rcu_tasks_irq_ip` and `module.ftrace_direct_tramp`) are explicitly initialized on task creation in `rcu_copy_process()` and in zero-allocated module structs. Global and per-CPU state variables are statically defined and zero-initialized.\n2. No data copied or leaked to user space: The changes do not touch `copy_to_user`, `put_user`, netlink, socket options, or ioctl outputs.\n3. Nature of bugs and applicable detectors: Any bugs introduced by this series would manifest as synchronization races, premature grace period completion leading to use-after-free of freed trampoline text or metadata, or locking/deadlock issues. These failure modes are strictly the domain of KASAN (for use-after-free and memory corruption) and LOCKDEP (for locking violations), not KMSAN.\n\nTherefore, dedicated KMSAN fuzzing is not warranted.",
"NeedsKMSAN": false
} |
| Total Calls | Total Tokens | Avg Tokens | Total Duration (Seconds) | Avg Duration (Seconds) |
|---|
| Total Calls | Total Duration (Seconds) | Avg Duration (Seconds) |
|---|