Reload the MMU (which is a nop if the MMU doesn't need to be reloaded) on
every attempt to pre-fault a guest page, i.e. when the page fault path
signals that the caller should retry. If the synchronize_srcu_expedited()
in kvm_invalidate_memslot() completes before kvm_vcpu_pre_fault_memory()
grabs SRCU, but kvm_mmu_reload() in the pre-fault path completes before
kvm_invalidate_memslot() triggers x86's "fast zap all", then the pre-fault
task will reach kvm_tdp_page_prefault() with an invalid root.
Attempting to fault-in memory with an invalid root ultimately puts
kvm_tdp_page_prefault() into an infinite (breakable) retry loop, which
manifests most obviously as a hang in the pre_fault_memory_test selftest,
but also eventually causes RCU (SRCU?) to complain.
INFO: rcu_tasks detected stalls on tasks:
000000000cda47bd: .. nvcsw: 6/6 holdout: 1 idle_cpu: -1/25
task:pre_fault_memor state:R running task stack:12696
pid:95588 tgid:95588 ppid:95584 task_flags:0x400000 flags:0x00080801
Call Trace:
lock_release+0x4e/0x320
__get_user_pages+0x546/0xcd0
up_read+0x1b/0x30
get_user_pages_unlocked+0xee/0x350
hva_to_pfn+0xd3/0x3d0 [kvm]
lock_release+0x4e/0x320
xa_load+0x5c/0x170
xa_load+0x14c/0x170
__kvm_faultin_pfn+0xd9/0x130 [kvm]
lock_acquire+0x65/0x2b0
lock_release+0x4e/0x320
kvm_mmu_faultin_pfn+0x1e1/0x690 [kvm]
gup_fast_fallback+0x63e/0xdf0
kvm_tdp_page_fault+0xeb/0x140 [kvm]
kvm_mmu_do_page_fault+0x12e/0x200 [kvm]
kvm_arch_vcpu_pre_fault_memory+0x16e/0x200 [kvm]
kvm_vcpu_pre_fault_memory+0xc1/0x1f0 [kvm]
kvm_vcpu_pre_fault_memory+0x116/0x1f0 [kvm]
kvm_vcpu_ioctl+0x3a4/0x6b0 [kvm]
clockevents_program_event+0x5d/0x170
__se_sys_ioctl+0x6d/0xb0
entry_SYSCALL_64_after_hwframe+0x4b/0x53
do_syscall_64+0x10a/0x480
__irq_exit_rcu+0x8e/0x140
entry_SYSCALL_64_after_hwframe+0x4b/0x53
Fixes: 6e01b7601dfe ("KVM: x86: Implement kvm_arch_vcpu_pre_fault_memory()")
Signed-off-by: Sean Christopherson
---
arch/x86/kvm/mmu/mmu.c | 12 ++++--------
1 file changed, 4 insertions(+), 8 deletions(-)
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index c519e8e8d646..621b0a42f2a1 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -5036,6 +5036,10 @@ static int kvm_tdp_page_prefault(struct kvm_vcpu *vcpu, gpa_t gpa,
if (kvm_check_request(KVM_REQ_VM_DEAD, vcpu))
return -EIO;
+ r = kvm_mmu_reload(vcpu);
+ if (r)
+ return r;
+
cond_resched();
r = kvm_mmu_do_page_fault(vcpu, gpa, error_code, true, NULL, level);
} while (r == RET_PF_RETRY);
@@ -5076,14 +5080,6 @@ long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu,
if (kvm_is_gfn_alias(vcpu->kvm, gpa_to_gfn(range->gpa)))
return -EINVAL;
- /*
- * reload is efficient when called repeatedly, so we can do it on
- * every iteration.
- */
- r = kvm_mmu_reload(vcpu);
- if (r)
- return r;
-
direct_bits = 0;
if (kvm_arch_has_private_mem(vcpu->kvm) &&
kvm_mem_is_private(vcpu->kvm, gpa_to_gfn(range->gpa)))
--
2.55.0.679.g6767b8d81c-goog