A BPF kthread that services a per-CPU resource, such as a NAPI instance whose interrupt lands on a fixed CPU, wants to run where that resource lives. The program can read the thread's PID from its task and leave the placement to user space, but a program that creates threads as its load demands should not need a round trip through user space to place each one. Add bpf_kthread_bind(), which restricts the thread to one CPU through set_cpus_allowed_ptr(). A thread that has not been started wakes on that CPU; one that has moves there at its next park or return, when it enables migration, and the call waits for that. The wait runs with migration enabled, for the reason given with the stop: a bind made while pinned could be the far side of a wait for the caller's own migrate_enable(), a bind or a stop issued by the thread being bound, and the two would wait on each other forever. It also lets a thread bind itself, which the scheduler refuses with a warning for a migration-disabled task whose target is not the CPU it is on. Unlike kthread_bind() it leaves the affinity changeable from user space and subject to cpusets, like that of any other task, with one difference from a user's own affinity: the bind is not remembered as a user mask, so a cpuset.cpus write on the thread's cgroup resets the thread to the whole cpuset, as the CPU going offline does, where a user task's mask would be re-applied. The scheduler moves the thread, or places it elsewhere at its next wakeup. A change of the housekeeping CPUs resets the affinity too, as it does for every kernel thread that is not bound for good. A program that needs the binding re-applies it. The binding sticks whenever it is applied: the generic kthread prologue applies the default affinity of the thread's node when the thread first runs, which is why bpf_kthread_create() runs every thread through that prologue before it returns. Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/kthread.c | 48 +++++++++++++++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 5 ++++- 2 files changed, 52 insertions(+), 1 deletion(-) diff --git a/kernel/bpf/kthread.c b/kernel/bpf/kthread.c index d8f63461d756..444fff4bd102 100644 --- a/kernel/bpf/kthread.c +++ b/kernel/bpf/kthread.c @@ -4,6 +4,7 @@ #include #include #include +#include #include #include #include @@ -350,6 +351,52 @@ __bpf_kfunc int bpf_kthread_start(struct task_struct *task, u64 flags) return 0; } +/** + * bpf_kthread_bind - Restrict a BPF kthread to one CPU + * @task: a thread created by bpf_kthread_create() + * @cpu: the CPU the thread is to run on + * @flags: must be 0 + * + * Sets the thread's affinity to @cpu. A thread that has not been started wakes + * on @cpu; one that has moves there at its next park or return, since its + * callback runs with migration disabled, and the call waits for that with + * migration enabled, so the caller may return on another CPU and a thread may + * bind itself. Unlike kthread_bind(), this leaves the affinity changeable from + * user space and subject to cpusets, like that of any other task. The binding + * is lost when @cpu goes offline or when the cpuset of the thread's cgroup + * changes its CPUs: the thread is moved, or placed elsewhere at its next + * wakeup, and its affinity reset to the cpuset's CPUs, as a user task's is + * after an offline. It is also lost when the housekeeping CPUs change, which + * resets the affinity of every kernel thread that is not bound for good. A + * program that needs the binding re-applies it. + * + * Return: 0 on success, -EINVAL if @flags is not 0 or @cpu is not a possible + * CPU or is offline, -ESRCH if @task is not a BPF kthread or has been stopped. + */ +__bpf_kfunc int bpf_kthread_bind(struct task_struct *task, u32 cpu, u64 flags) +{ + unsigned long irqflags; + bool found; + int ret; + + if (flags) + return -EINVAL; + if (cpu >= nr_cpu_ids || !cpu_possible(cpu)) + return -EINVAL; + + raw_spin_lock_irqsave(&bpf_kthreads_lock, irqflags); + found = !!bpf_kthread_find(task); + raw_spin_unlock_irqrestore(&bpf_kthreads_lock, irqflags); + if (!found) + return -ESRCH; + + /* A running thread moves at its next park: see the comment at the top. */ + migrate_enable(); + ret = set_cpus_allowed_ptr(task, cpumask_of(cpu)); + migrate_disable(); + return ret; +} + /** * bpf_kthread_stop - Stop a BPF kthread and wait for it to exit * @task: a thread created by bpf_kthread_create() @@ -452,6 +499,7 @@ __bpf_kfunc_end_defs(); BTF_KFUNCS_START(bpf_kthread_kfunc_ids) BTF_ID_FLAGS(func, bpf_kthread_create, KF_ACQUIRE | KF_RET_NULL | KF_SLEEPABLE | KF_IMPLICIT_ARGS) BTF_ID_FLAGS(func, bpf_kthread_start, KF_RCU) +BTF_ID_FLAGS(func, bpf_kthread_bind, KF_SLEEPABLE) BTF_ID_FLAGS(func, bpf_kthread_stop, KF_SLEEPABLE) BTF_ID_FLAGS(func, bpf_kthread_should_stop) BTF_KFUNCS_END(bpf_kthread_kfunc_ids) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index ee409ac5be77..54fc44dd8fcc 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -14366,6 +14366,7 @@ enum special_kfunc_type { KF_bpf_call_rcu_tasks_trace, KF_bpf_kthread_create, KF_bpf_kthread_stop, + KF_bpf_kthread_bind, KF_bpf_arena_alloc_pages, KF_bpf_arena_free_pages, KF_bpf_session_is_return, @@ -14461,6 +14462,7 @@ BTF_ID(func, bpf_call_rcu) BTF_ID(func, bpf_call_rcu_tasks_trace) BTF_ID(func, bpf_kthread_create) BTF_ID(func, bpf_kthread_stop) +BTF_ID(func, bpf_kthread_bind) #if defined(CONFIG_MMU) && defined(CONFIG_64BIT) BTF_ID(func, bpf_arena_alloc_pages) BTF_ID(func, bpf_arena_free_pages) @@ -14507,7 +14509,8 @@ static bool is_migrating_kfunc(u32 btf_id) { return btf_id == special_kfunc_list[KF_bpf_task_park] || btf_id == special_kfunc_list[KF_bpf_kthread_create] || - btf_id == special_kfunc_list[KF_bpf_kthread_stop]; + btf_id == special_kfunc_list[KF_bpf_kthread_stop] || + btf_id == special_kfunc_list[KF_bpf_kthread_bind]; } static bool is_bpf_obj_new_kfunc(u32 func_id) -- 2.53.0