An out-of-line direct trampoline registered with register_ftrace_direct() is kept alive only by Tasks RCU while a task executes it or is preempted in something it called; ftrace_shutdown()'s synchronize_rcu_tasks() is what stops rmmod freeing it under such a task. Once preemption becomes a Tasks RCU quiescent state, such a trampoline must hold current->rcu_tramp_nesting across its call-out like the ftrace and BPF trampolines do, so document that in register_ftrace_direct(). That still leaves the few instructions before the increment and after the decrement. For BPF images those are in dynamically allocated text that rcu_tasks_ip_in_trampoline() already treats as protected, but the in-tree samples (and any similar user) place their trampolines in module .text. Add a sticky module::ftrace_direct_tramp flag, set by every register/modify path when the direct address is module text, and have rcu_tasks_ip_in_trampoline() treat a task interrupted anywhere in such a module as a potential reader. Other modules' text is unaffected. Assisted-by: LLM Signed-off-by: Josef Bacik --- include/linux/module.h | 7 +++++++ kernel/rcu/tasks.h | 23 +++++++++++++++++++++-- kernel/trace/ftrace.c | 39 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 67 insertions(+), 2 deletions(-) diff --git a/include/linux/module.h b/include/linux/module.h index 96cc98568eea..ea4727f53fab 100644 --- a/include/linux/module.h +++ b/include/linux/module.h @@ -521,6 +521,13 @@ struct module { unsigned int num_ftrace_callsites; unsigned long *ftrace_callsites; #endif +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS + /* + * An ftrace direct-call trampoline lives in this module's text; see + * rcu_tasks_ip_in_trampoline(). Sticky once set. + */ + bool ftrace_direct_tramp; +#endif #ifdef CONFIG_KPROBES void *kprobes_text_start; unsigned int kprobes_text_size; diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 0e46d8fe4d8e..1b9fe1bfa591 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -1114,16 +1114,35 @@ bool __weak arch_rcu_tasks_ip_in_trampoline(unsigned long ip) * deliberately does not consult is_ftrace_trampoline() and friends: text * being torn down may already be unregistered there while a task still * stands on it; - * - in core text the architecture flags via arch_rcu_tasks_ip_in_trampoline(). + * - in core text the architecture flags via arch_rcu_tasks_ip_in_trampoline(); + * - in the text of a module that hosts an ftrace direct-call trampoline, + * which covers the instructions before that trampoline's increment and + * after its decrement (see ftrace_direct_mark_module()). * * A false positive only defers the quiescent state to the task's next * context switch. */ bool rcu_tasks_ip_in_trampoline(unsigned long ip) { + bool ret = true; + if (core_kernel_text(ip)) return arch_rcu_tasks_ip_in_trampoline(ip); - return !is_module_text_address(ip); + +#ifdef CONFIG_MODULES + scoped_guard(rcu) { + struct module *mod = __module_text_address(ip); + +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS + if (mod) + ret = READ_ONCE(mod->ftrace_direct_tramp); +#else + if (mod) + ret = false; +#endif + } +#endif + return ret; } NOKPROBE_SYMBOL(rcu_tasks_ip_in_trampoline); diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 53d5db60bfa5..14f27b887231 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -6076,6 +6076,29 @@ static void reset_direct(struct ftrace_ops *ops, unsigned long addr) ops->trampoline = 0; } +/* + * A direct trampoline may live in module text rather than in dynamically + * allocated text that rcu_tasks_ip_in_trampoline() recognises on its own (see + * samples/ftrace/ftrace-direct*.c). The trampoline itself must hold + * current->rcu_tramp_nesting across its call-out (see register_ftrace_direct()); + * marking the owning module here covers the instructions before that increment + * and after the decrement, where a task interrupted in the module's text must + * not be treated as Tasks-RCU quiescent, so that ftrace_shutdown()'s + * synchronize_rcu_tasks() still keeps the module text from being freed under + * it. + */ +static void ftrace_direct_mark_module(unsigned long addr) +{ +#ifdef CONFIG_MODULES + struct module *mod; + + guard(rcu)(); + mod = __module_text_address(addr); + if (mod) + WRITE_ONCE(mod->ftrace_direct_tramp, true); +#endif +} + /** * register_ftrace_direct - Call a custom trampoline directly * for multiple functions registered in @ops @@ -6090,6 +6113,17 @@ static void reset_direct(struct ftrace_ops *ops, unsigned long addr) * and save the parameters of the function being traced, and restore them * (or inject new ones if needed), before returning. * + * Nothing but Tasks RCU keeps the trampoline at @addr alive while a task is + * executing it or is preempted in something it called. On architectures that + * select ARCH_HAS_RCU_TASKS_PREEMPT_QS a preemption is a Tasks RCU quiescent + * state unless current->rcu_tramp_nesting is non-zero, so the trampoline must + * increment it before calling out and decrement it before returning, as the + * ftrace and BPF trampolines do (see rcu_tasks_trampoline_enter() and + * samples/ftrace/ftrace-direct.h). The few instructions before the increment + * and after the decrement are covered by the irq-exit IP check: automatically + * for trampolines outside kernel and module text (e.g. BPF images), and via + * ftrace_direct_mark_module() for trampolines in module text. + * * Returns: * 0 on success * -EINVAL - The @ops object was already registered with this call or @@ -6169,6 +6203,7 @@ int register_ftrace_direct(struct ftrace_ops *ops, unsigned long addr) ops->flags |= MULTI_FLAGS; ops->trampoline = FTRACE_REGS_ADDR; ops->direct_call = addr; + ftrace_direct_mark_module(addr); err = register_ftrace_function_nolock(ops); if (err) @@ -6237,6 +6272,8 @@ __modify_ftrace_direct(struct ftrace_ops *ops, unsigned long addr) lockdep_assert_held_once(&direct_mutex); + ftrace_direct_mark_module(addr); + /* Enable the tmp_ops to have the same functions as the direct ops */ ftrace_ops_init(&tmp_ops); tmp_ops.func_hash = ops->func_hash; @@ -6419,6 +6456,7 @@ int update_ftrace_direct_add(struct ftrace_ops *ops, struct ftrace_hash *hash) hlist_for_each_entry(entry, &hash->buckets[i], hlist) { if (__ftrace_lookup_ip(direct_functions, entry->ip)) goto out_unlock; + ftrace_direct_mark_module(entry->direct); } } @@ -6702,6 +6740,7 @@ int update_ftrace_direct_mod(struct ftrace_ops *ops, struct ftrace_hash *hash, b tmp = __ftrace_lookup_ip(direct_hash, entry->ip); if (!tmp) continue; + ftrace_direct_mark_module(entry->direct); tmp->direct = entry->direct; } } -- 2.55.0