Aging G-stage PTEs currently runs with mmu_lock held for write, taken by the common MMU notifier code. When MGLRU or kswapd ages a large range, every vCPU taking a G-stage fault blocks on the lock, and the dirty-logging read-side fast path blocks behind aging as well. The preceding patches prepare G-stage page-table walks for lockless aging by using consistent PTE snapshots, preserving concurrent Accessed-bit updates, and deferring page-table frees with RCU. Select KVM_MMU_LOCKLESS_AGING so the common code no longer takes mmu_lock for aging, and protect the G-stage walk with an RCU read-side critical section. Read the root inside the RCU read-side critical section and check the resulting snapshot instead of checking kvm->arch.pgd separately before initializing the G-stage context. This ensures that the root used by the walk remains protected until the walk completes. On QEMU TCG with 4 vCPUs, running dirty_log_perf_test -v 3 -b 256M -i 3 with MGLRU aging of the VM's cgroup forced every 100ms (86 passes): before after mmu_lock write wait, total 44.4 s 0.15 s mmu_lock write contentions 1,002,092 30,092 mmu_lock read contentions 351,937 0 guest dirty-memory time 18.6 s 16.7 s Before, 93% of write-lock waiters were kvm_mmu_notifier_clear_young(). With no aging, no meaningful difference was observed between the two kernels. Signed-off-by: SeungJu Cheon --- arch/riscv/kvm/Kconfig | 1 + arch/riscv/kvm/mmu.c | 26 +++++++++++++------------- 2 files changed, 14 insertions(+), 13 deletions(-) diff --git a/arch/riscv/kvm/Kconfig b/arch/riscv/kvm/Kconfig index ec2cee0a39e0..77898d58ff9a 100644 --- a/arch/riscv/kvm/Kconfig +++ b/arch/riscv/kvm/Kconfig @@ -32,6 +32,7 @@ config KVM select VIRT_XFER_TO_GUEST_WORK select SCHED_INFO select GUEST_PERF_EVENTS if PERF_EVENTS + select KVM_MMU_LOCKLESS_AGING help Support hosting virtualized guest machines. diff --git a/arch/riscv/kvm/mmu.c b/arch/riscv/kvm/mmu.c index 8aed69abf814..ff282bdbe492 100644 --- a/arch/riscv/kvm/mmu.c +++ b/arch/riscv/kvm/mmu.c @@ -349,30 +349,30 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range) return false; } -bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) +static bool kvm_riscv_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range, + bool test_only) { struct kvm_gstage gstage; - if (!kvm->arch.pgd) - return false; + guard(rcu)(); + lockdep_assert_not_held(&kvm->mmu_lock); kvm_riscv_gstage_init(&gstage, kvm); + if (!gstage.pgd) + return false; return kvm_riscv_gstage_age_range(&gstage, range->start << PAGE_SHIFT, - range->end << PAGE_SHIFT, false); + range->end << PAGE_SHIFT, test_only); } -bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) +bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { - struct kvm_gstage gstage; - - if (!kvm->arch.pgd) - return false; - - kvm_riscv_gstage_init(&gstage, kvm); + return kvm_riscv_age_gfn(kvm, range, false); +} - return kvm_riscv_gstage_age_range(&gstage, range->start << PAGE_SHIFT, - range->end << PAGE_SHIFT, true); +bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) +{ + return kvm_riscv_age_gfn(kvm, range, true); } static bool fault_supports_gstage_huge_mapping(struct kvm_memory_slot *memslot, -- 2.52.0