From: Caron Wills Since commit 041ee6f3727a ("kthread: Rely on HK_TYPE_DOMAIN for preferred affinity management") kthread_fetch_affinity() only removes domain isolated CPUs from a kthread's affinity, assuming that nohz_full CPUs are always domain isolated as well. Up to v6.13 every kthread inherited kthreadd's affinity to housekeeping_cpumask(HK_TYPE_KTHREAD), and from v6.14 the preferred affinity was filtered with HK_TYPE_KTHREAD, so nohz_full CPUs were always excluded. On a system that keeps its nohz_full CPUs in the scheduler domains, e.g. isolcpus=nohz,18-95,108-178 nohz_full=18-95,108-178 HK_TYPE_DOMAIN covers every CPU, and the RCU expedited grace period workers rcu_exp_par_gp_kthread_worker/N, the RCU boost kthreads rcub/N and every NUMA node affine kthread can now land on nohz_full CPUs. rcu_thread_affine_rnp() prefers the CPUs of the rcu_node, so for a leaf rcu_node made entirely of nohz_full CPUs the worker is pinned to nothing but nohz_full CPUs. Those workers are SCHED_NORMAL (CONFIG_RCU_BOOST=n leaves kthread_prio at 0), so a SCHED_FIFO userspace busy loop on such a CPU starves them indefinitely. Every synchronize_rcu_expedited() in the system then blocks in sync_rcu_exp_select_cpus() -> kthread_flush_work() until the FIFO task yields. LP: #2165410 captured exactly that on three 7.0 kernels: the cpuset_migrate_mm kworker sits in synchronize_rcu_expedited+0x1b2/0x220 lru_cache_disable+0x15/0x30 do_migrate_pages+0x4e/0x2b0 cpuset_migrate_mm_workfn+0x26/0x50 with rcu_exp_gp_kthread_worker in D state in kthread_flush_work(), and PID 1 waits behind that migration in flush_workqueue() for minutes. Intersect the affinity with the HK_TYPE_KERNEL_NOISE housekeeping mask as well and fall back to the CPUs that are housekeeping for both types. On systems where nohz_full implies domain isolation the two masks are identical and nothing changes. HK_TYPE_KERNEL_NOISE is named explicitly rather than HK_TYPE_KTHREAD: commit 8f78b749f3da ("sched/isolation: Make HK_TYPE_KTHREAD an alias of HK_TYPE_DOMAIN") (v7.1-rc3) turned the latter into a synonym of the domain mask, which is precisely the assumption being corrected here, and made a first version of this patch a no-op on 7.1+ kernels. Fixes: 041ee6f3727a ("kthread: Rely on HK_TYPE_DOMAIN for preferred affinity management") Link: https://bugs.launchpad.net/bugs/2165410 Signed-off-by: Caron Wills Signed-off-by: Matthew Ruffell --- kernel/kthread.c | 30 ++++++++++++++++++++++++++---- 1 file changed, 26 insertions(+), 4 deletions(-) diff --git a/kernel/kthread.c b/kernel/kthread.c index fdb8fb07f40b..b9d8dfe1b6ca 100644 --- a/kernel/kthread.c +++ b/kernel/kthread.c @@ -327,24 +327,46 @@ void __noreturn kthread_complete_and_exit(struct completion *comp, long code) } EXPORT_SYMBOL(kthread_complete_and_exit); +/* + * Unbound kthreads must run neither on domain isolated CPUs (isolcpus=domain, + * cpuset isolated partitions) nor on CPUs isolated from kernel noise + * (nohz_full=, isolcpus=nohz). The two sets usually coincide, but the kernel + * accepts nohz_full CPUs that remain in the scheduler domains, so both + * housekeeping types have to be honoured. + */ static void kthread_fetch_affinity(struct kthread *kthread, struct cpumask *cpumask) { - const struct cpumask *pref; + const struct cpumask *pref, *hk_domain, *hk_kthread; guard(rcu)(); + hk_domain = housekeeping_cpumask(HK_TYPE_DOMAIN); + /* + * Deliberately not HK_TYPE_KTHREAD: since v7.1 that is an alias of + * HK_TYPE_DOMAIN (commit 8f78b749f3da), which is the very assumption + * being corrected here. + */ + hk_kthread = housekeeping_cpumask(HK_TYPE_KERNEL_NOISE); + if (kthread->preferred_affinity) { pref = kthread->preferred_affinity; } else { if (kthread->node == NUMA_NO_NODE) - pref = housekeeping_cpumask(HK_TYPE_DOMAIN); + pref = hk_domain; else pref = cpumask_of_node(kthread->node); } - cpumask_and(cpumask, pref, housekeeping_cpumask(HK_TYPE_DOMAIN)); + cpumask_and(cpumask, pref, hk_domain); + cpumask_and(cpumask, cpumask, hk_kthread); + if (cpumask_empty(cpumask)) + cpumask_and(cpumask, hk_domain, hk_kthread); + /* + * Both the boot parameters and cpuset guarantee that at least one + * CPU stays housekeeping for every type, but don't rely on it. + */ if (!cpumask_intersects(cpumask, cpu_online_mask)) - cpumask_copy(cpumask, housekeeping_cpumask(HK_TYPE_DOMAIN)); + cpumask_copy(cpumask, hk_domain); } static void kthread_affine_node(void) -- 2.55.0