From: Caron Wills Commit 1c3329df409b ("sched/isolation: Remove HK_TYPE_TICK test from cpu_is_isolated()") reduced cpu_is_isolated() to a HK_TYPE_DOMAIN test on the assumption that "it doesn't make sense to use nohz_full without also isolating the related CPUs from the domain topology", i.e. that every nohz_full CPU is also domain isolated. The kernel does not enforce that assumption. "nohz_full=" and "isolcpus=nohz," are accepted and documented on their own, and deployments exist that deliberately keep the nohz_full CPUs inside the scheduler domains, for example the AWS c8a.metal-48xl systems of LP: #2165410, booted with isolcpus=nohz,18-95,108-178 nohz_full=18-95,108-178 rcu_nocbs=18-95,108-178 On such a system cpu_is_isolated() is now false for every CPU and all of its users silently lost their isolation awareness compared to v6.8, which also tested HK_TYPE_TICK: - fs/buffer.c bh_lru_install() caches buffer_heads on the nohz_full CPUs again, so has_bh_in_lru() makes __lru_add_drain_all() queue drain work on them; - mm/vmstat.c vmstat_shepherd() queues vmstat_update on them again; - mm/memcontrol.c schedule_drain_work() queues memcg stock drains on them again; - block/blk-mq.c keeps them in hctx->cpumask. Per-CPU work queued on a nohz_full CPU that runs a SCHED_FIFO busy loop never executes, and the flush_work() callers hang behind it. This is one half of the PID 1 D-state stall described in LP: #2165410. Test HK_TYPE_KERNEL_NOISE (the housekeeping type set by nohz_full= and isolcpus=nohz, of which HK_TYPE_TICK is an alias) in addition to HK_TYPE_DOMAIN, and use the housekeeping_cpu() wrapper so that systems without any CPU isolation keep paying nothing but a patched-out static branch. On systems where the nohz_full CPUs are also domain isolated nothing changes. Fixes: 1c3329df409b ("sched/isolation: Remove HK_TYPE_TICK test from cpu_is_isolated()") Link: https://bugs.launchpad.net/bugs/2165410 Signed-off-by: Caron Wills Signed-off-by: Matthew Ruffell --- include/linux/sched/isolation.h | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/include/linux/sched/isolation.h b/include/linux/sched/isolation.h index cf0fd03dd7a2..294c508b0018 100644 --- a/include/linux/sched/isolation.h +++ b/include/linux/sched/isolation.h @@ -84,9 +84,17 @@ static inline bool housekeeping_cpu(int cpu, enum hk_type type) return true; } +/* + * A CPU is isolated if it has been taken out of the scheduler domains + * (isolcpus=domain or a cpuset isolated partition) or out of kernel noise + * (nohz_full= or isolcpus=nohz). The latter does not imply the former: + * the kernel accepts nohz_full CPUs that are still part of the scheduler + * domains, and deferred per-CPU work must not target those CPUs either. + */ static inline bool cpu_is_isolated(int cpu) { - return !housekeeping_test_cpu(cpu, HK_TYPE_DOMAIN); + return !housekeeping_cpu(cpu, HK_TYPE_DOMAIN) || + !housekeeping_cpu(cpu, HK_TYPE_KERNEL_NOISE); } #endif /* _LINUX_SCHED_ISOLATION_H */ -- 2.55.0