When loading nested state, explicitly check that the current CR3 is valid before loading any state, as a misbehaving userspace can muck with guest CPUID to make KVM think the current CR3 is invalid, and thus cause the late "load" of CR3 to fail when initializing the guest's MMU. If that happens, KVM will leave half-baked state in the vCPU, e.g. in L2 with L1's page walker, and if nested NPT is disabled, L1's PGD as well. Alternatively, KVM could try and unwind vCPU state if the late CR3 "load" fails, but that is subtly quite difficult as it would require snapshotting and restoring the previous state. Simply leaving nested on failure isn't an option either, because that too would leave the vCPU with half-baked state (in L1 with L2 state and a very stale MMU). Pre-checking CR3 will also allow hoisting the CR3 check out of nested_svm_load_cr3() entirely. WARN and bug the VM if the late CR3 load fails, as loading nested state never loads PDPTRs, i.e. failure is impossible, and KVM doesn't unwind on failure (see above). I.e. if the impossible happens, KVM needs to block all operations on the half-baked vCPU. Fixes: e1779c2714c3 ("KVM: x86: nSVM: fix potential NULL derefernce on nested migration") Cc: stable@vger.kernel.org Reported-by: Stefan Teodorescu Reported-by: Sashiko Bot Closes: https://lore.kernel.org/all/sashiko-outbox-210535@kernel.org Closes: https://lore.kernel.org/all/sashiko-outbox-205939@kernel.org Closes: https://lore.kernel.org/all/sashiko-outbox-193903@kernel.org Signed-off-by: Sean Christopherson --- arch/x86/kvm/svm/nested.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c index 07554089ec05..1e3842851dae 100644 --- a/arch/x86/kvm/svm/nested.c +++ b/arch/x86/kvm/svm/nested.c @@ -2047,6 +2047,9 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu, if (((cr0 & X86_CR0_CD) == 0) && (cr0 & X86_CR0_NW)) goto out_free; + if (!kvm_vcpu_is_legal_cr3(vcpu, vcpu->arch.cr3)) + goto out_free; + /* * Validate host state saved from before VMRUN (see * nested_svm_check_permissions). Note that the g_pat field is not @@ -2114,7 +2117,7 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu, */ ret = nested_svm_load_cr3(&svm->vcpu, vcpu->arch.cr3, nested_npt_enabled(svm), false); - if (ret) + if (KVM_BUG_ON(ret, vcpu->kvm)) goto out_free; svm->nested.force_msr_bitmap_recalc = true; -- 2.56.1.703.gf848ed807c-goog From: Paolo Bonzini nested_vmcb_check_save() checks CR3 legality only if long mode is active, but it really needs to be checked in all cases - the check uses must be zero bits of the host, and indeed nested_svm_load_cr3() later will reject illegal CR3 in every mode. Move the check earlier, and thus out of nested_svm_load_cr3(). Likewise for nested_vmx_load_cr3() and nested_vmx_check_guest_state(). Signed-off-by: Paolo Bonzini Tested-by: syzbot@syzkaller.appspotmail.com Cc: stable@vger.kernel.org [sean: tag for stable] Signed-off-by: Sean Christopherson --- arch/x86/kvm/svm/nested.c | 9 +++------ arch/x86/kvm/vmx/nested.c | 8 ++------ 2 files changed, 5 insertions(+), 12 deletions(-) diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c index 1e3842851dae..a743179b172a 100644 --- a/arch/x86/kvm/svm/nested.c +++ b/arch/x86/kvm/svm/nested.c @@ -443,7 +443,8 @@ static bool nested_vmcb_check_save(struct kvm_vcpu *vcpu, return false; if (CC((save->cr0 & X86_CR0_CD) == 0 && (save->cr0 & X86_CR0_NW)) || - CC(save->cr0 & ~0xffffffffULL)) + CC(save->cr0 & ~0xffffffffULL) || + CC(!kvm_vcpu_is_legal_cr3(vcpu, save->cr3))) return false; if (CC(!kvm_dr6_valid(save->dr6)) || CC(!kvm_dr7_valid(save->dr7))) @@ -456,8 +457,7 @@ static bool nested_vmcb_check_save(struct kvm_vcpu *vcpu, */ if ((save->efer & EFER_LME) && (save->cr0 & X86_CR0_PG)) { if (CC(!(save->cr4 & X86_CR4_PAE)) || - CC(!(save->cr0 & X86_CR0_PE)) || - CC(!kvm_vcpu_is_legal_cr3(vcpu, save->cr3))) + CC(!(save->cr0 & X86_CR0_PE))) return false; if (CC((save->cs.attrib & SVM_SELECTOR_L_MASK) && @@ -717,9 +717,6 @@ static void nested_svm_transition_tlb_flush(struct kvm_vcpu *vcpu) static int nested_svm_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3, bool nested_npt, bool reload_pdptrs) { - if (CC(!kvm_vcpu_is_legal_cr3(vcpu, cr3))) - return -EINVAL; - if (reload_pdptrs && is_pae_paging(vcpu)) { if (nested_npt) kvm_register_mark_for_reload(vcpu, VCPU_REG_PDPTR); diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c index f0d75f5a968c..ef4ca1906fb0 100644 --- a/arch/x86/kvm/vmx/nested.c +++ b/arch/x86/kvm/vmx/nested.c @@ -1235,11 +1235,6 @@ static int nested_vmx_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3, bool nested_ept, bool reload_pdptrs, enum vm_entry_failure_code *entry_failure_code) { - if (CC(!kvm_vcpu_is_legal_cr3(vcpu, cr3))) { - *entry_failure_code = ENTRY_FAIL_DEFAULT; - return -EINVAL; - } - /* * If PAE paging and EPT are both on, CR3 is not used by the CPU and * must not be dereferenced. @@ -3323,7 +3318,8 @@ static int nested_vmx_check_guest_state(struct kvm_vcpu *vcpu, *entry_failure_code = ENTRY_FAIL_DEFAULT; if (CC(!nested_guest_cr0_valid(vcpu, vmcs12->guest_cr0)) || - CC(!nested_guest_cr4_valid(vcpu, vmcs12->guest_cr4))) + CC(!nested_guest_cr4_valid(vcpu, vmcs12->guest_cr4)) || + CC(!kvm_vcpu_is_legal_cr3(vcpu, vmcs12->guest_cr3))) return -EINVAL; if (CC(vmcs12->guest_cr4 & X86_CR4_CET && !(vmcs12->guest_cr0 & X86_CR0_WP))) -- 2.56.1.703.gf848ed807c-goog Mark CR3 as dirty when loading CR3 on nested VMRUN and nested #VMEXIT, even though svm_load_mmu_pgd() unconditionally updates vmcb.save.CR3, i.e. never consumes the dirty bit. This will allow deduplicating nearly-identical code between nVMX and nSVM with respect to loading CR3. Note, VMX consumes the dirty flag in kvm_mmu_new_pgd(), i.e. having nVMX mark CR3 after the fact isn't a viable option. Marking CR3 dirty in nVMX code before invoking the to-be-common helper also isn't safe in general, as it's theoretically possible KVM could observe+consume the dirty flag before vcpu->arch.cr3 is updated (even though that's extremely unlikely to ever be the case). Cc: stable@vger.kernel.org Signed-off-by: Sean Christopherson --- arch/x86/kvm/svm/nested.c | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c index a743179b172a..32f95c9d72be 100644 --- a/arch/x86/kvm/svm/nested.c +++ b/arch/x86/kvm/svm/nested.c @@ -725,6 +725,7 @@ static int nested_svm_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3, } vcpu->arch.cr3 = cr3; + kvm_register_mark_dirty(vcpu, VCPU_REG_CR3); /* Re-initialize the MMU, e.g. to pick up CR4 MMU role changes. */ kvm_init_mmu(vcpu); -- 2.56.1.703.gf848ed807c-goog Dedup nested_{svm,vmx}_load_cr3() into kvm_nested_load_cr3(), and drop several MMU exports that existed purely to support the related vendor code. Except for slight differences around PDPTRs, the logic and reasoning for handling CR3 and PDPTRs on nested transitions is identical between Intel and AMD. Add a KVM_BUG_ON() in nested_vmx_load_cr3() to ensure that the new helper doesn't unexpectedly fail when PDPTRs aren't being reloaded. No functional change intended (the new KVM_BUG_ON() is unreachable). Cc: stable@vger.kernel.org Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 2 -- arch/x86/kvm/regs.c | 24 ++++++++++++++++++++++++ arch/x86/kvm/regs.h | 2 ++ arch/x86/kvm/svm/nested.c | 23 +++-------------------- arch/x86/kvm/vmx/nested.c | 23 ++--------------------- 5 files changed, 31 insertions(+), 43 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index f4b12d6aef55..9186e8e2ec10 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -5475,7 +5475,6 @@ void kvm_mmu_new_pgd(struct kvm_vcpu *vcpu, gpa_t new_pgd) __clear_sp_write_flooding_count(sp); } } -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_mmu_new_pgd); static bool sync_mmio_spte(struct kvm_vcpu *vcpu, u64 *sptep, gfn_t gfn, unsigned int access) @@ -6275,7 +6274,6 @@ void kvm_init_mmu(struct kvm_vcpu *vcpu) init_kvm_shadow_mmu(vcpu, cpu_role); } } -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_init_mmu); void kvm_mmu_after_set_cpuid(struct kvm_vcpu *vcpu) { diff --git a/arch/x86/kvm/regs.c b/arch/x86/kvm/regs.c index 3621f1a47d37..b43ad11f061c 100644 --- a/arch/x86/kvm/regs.c +++ b/arch/x86/kvm/regs.c @@ -201,6 +201,30 @@ int load_pdptrs(struct kvm_vcpu *vcpu, unsigned long cr3) } EXPORT_SYMBOL_FOR_KVM_INTERNAL(load_pdptrs); +int kvm_nested_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3, + bool nested_tdp, bool reload_pdptrs) +{ + /* + * If TDP is enabled, then the PDPTRs either come from the VMCS (EPT) + * or are read from memory on-demand (NPT), i.e. aren't loaded here. + */ + if (reload_pdptrs && !nested_tdp && is_pae_paging(vcpu) && + !load_pdptrs(vcpu, cr3)) + return -EINVAL; + + vcpu->arch.cr3 = cr3; + kvm_register_mark_dirty(vcpu, VCPU_REG_CR3); + + /* Re-initialize the MMU, e.g. to pick up CR4 MMU role changes. */ + kvm_init_mmu(vcpu); + + if (!nested_tdp) + kvm_mmu_new_pgd(vcpu, cr3); + + return 0; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_nested_load_cr3); + static bool kvm_is_valid_cr0(struct kvm_vcpu *vcpu, unsigned long cr0) { #ifdef CONFIG_X86_64 diff --git a/arch/x86/kvm/regs.h b/arch/x86/kvm/regs.h index 97c17b120a57..3c34ca7d8a8e 100644 --- a/arch/x86/kvm/regs.h +++ b/arch/x86/kvm/regs.h @@ -64,6 +64,8 @@ unsigned long kvm_get_dr6_fixed_1(struct kvm_vcpu *vcpu); unsigned long kvm_get_cr8(struct kvm_vcpu *vcpu); void kvm_lmsw(struct kvm_vcpu *vcpu, unsigned long msw); int load_pdptrs(struct kvm_vcpu *vcpu, unsigned long cr3); +int kvm_nested_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3, + bool nested_tdp, bool reload_pdptrs); static inline bool is_long_mode(struct kvm_vcpu *vcpu) { diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c index 32f95c9d72be..a045cb5ba9ff 100644 --- a/arch/x86/kvm/svm/nested.c +++ b/arch/x86/kvm/svm/nested.c @@ -710,30 +710,13 @@ static void nested_svm_transition_tlb_flush(struct kvm_vcpu *vcpu) kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu); } -/* - * Load guest's/host's cr3 on nested vmentry or vmexit. @nested_npt is true - * if we are emulating VM-Entry into a guest with NPT enabled. - */ static int nested_svm_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3, bool nested_npt, bool reload_pdptrs) { - if (reload_pdptrs && is_pae_paging(vcpu)) { - if (nested_npt) - kvm_register_mark_for_reload(vcpu, VCPU_REG_PDPTR); - else if (CC(!load_pdptrs(vcpu, cr3))) - return -EINVAL; - } + if (reload_pdptrs && is_pae_paging(vcpu) && nested_npt) + kvm_register_mark_for_reload(vcpu, VCPU_REG_PDPTR); - vcpu->arch.cr3 = cr3; - kvm_register_mark_dirty(vcpu, VCPU_REG_CR3); - - /* Re-initialize the MMU, e.g. to pick up CR4 MMU role changes. */ - kvm_init_mmu(vcpu); - - if (!nested_npt) - kvm_mmu_new_pgd(vcpu, cr3); - - return 0; + return kvm_nested_load_cr3(vcpu, cr3, nested_npt, reload_pdptrs); } static void nested_vmcb02_prepare_save(struct vcpu_svm *svm) diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c index ef4ca1906fb0..6adfd1fe7fa6 100644 --- a/arch/x86/kvm/vmx/nested.c +++ b/arch/x86/kvm/vmx/nested.c @@ -1225,35 +1225,16 @@ static bool nested_msr_store_list_has_msr(struct kvm_vcpu *vcpu, u32 msr_index) return false; } -/* - * Load guest's/host's cr3 at nested entry/exit. @nested_ept is true if we are - * emulating VM-Entry into a guest with EPT enabled. On failure, the expected - * Exit Qualification (for a VM-Entry consistency check VM-Exit) is assigned to - * @entry_failure_code. - */ static int nested_vmx_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3, bool nested_ept, bool reload_pdptrs, enum vm_entry_failure_code *entry_failure_code) { - /* - * If PAE paging and EPT are both on, CR3 is not used by the CPU and - * must not be dereferenced. - */ - if (reload_pdptrs && !nested_ept && is_pae_paging(vcpu) && - CC(!load_pdptrs(vcpu, cr3))) { + if (CC(kvm_nested_load_cr3(vcpu, cr3, nested_ept, reload_pdptrs))) { + KVM_BUG_ON(!reload_pdptrs, vcpu->kvm); *entry_failure_code = ENTRY_FAIL_PDPTE; return -EINVAL; } - vcpu->arch.cr3 = cr3; - kvm_register_mark_dirty(vcpu, VCPU_REG_CR3); - - /* Re-initialize the MMU, e.g. to pick up CR4 MMU role changes. */ - kvm_init_mmu(vcpu); - - if (!nested_ept) - kvm_mmu_new_pgd(vcpu, cr3); - return 0; } -- 2.56.1.703.gf848ed807c-goog Initialize the MMU and PGD (root) before loading PDPTRs when emulating a nested VM-Enter or VM-Exit to ensure the MMU is up-to-date even if loading PDPTRs fails. If a nested VM-Exit fails to load L1's PDPTEs, and L1 ran L2 without EPT/NP, KVM can run L1 on a freed MMU root, i.e. the CPU's page walk is a use-after-free, if userspace blows past the KVM_EXIT_SHUTDOWN. root_mmu vmcx01 vmcx02 ======== ====== ====== L1 runs R1 R1 VM-Enter to L2 R2 R1 R2 memslot deleted, L2 runs R3 R1 R3 R1, R2 freed VM-Exit, bad PDPTEs R3 R1 KVM_EXIT_SHUTDOWN R3 R1 KVM_RUN enters L1 R3 R1 Unconditionally initializing the MMU if KVM sets CR3 is a-ok, as the nested VM-Exit paths need exactly that behavior, emulation of VM-Enter generates a VM-Exit on both SVM and VMX (i.e. KVM is guaranteed to load L1 state and re-initialize the MMU), and RSM+KVM_SET_NESTED_STATE defer loading PDPTRs until the next entry, i.e. don't load PDPTRs and thus can't fail. Note, pre-checking the PDPTRs isn't an option because they come from guest memory when TDP is disabled, i.e. the PDPTEs in memory can legally change between the VM-Enter and VM-Exit. Fixes: 1dc35dacc16b ("KVM: nVMX: check host CR3 on vmentry and vmexit") Fixes: d82aaef9c88a ("KVM: nSVM: use nested_svm_load_cr3() on guest->host switch") Reported-by: Chris Mason Cc: stable@vger.kernel.org Signed-off-by: Sean Christopherson --- arch/x86/kvm/regs.c | 24 +++++++++++++++--------- 1 file changed, 15 insertions(+), 9 deletions(-) diff --git a/arch/x86/kvm/regs.c b/arch/x86/kvm/regs.c index b43ad11f061c..1cbc7b9f6ef4 100644 --- a/arch/x86/kvm/regs.c +++ b/arch/x86/kvm/regs.c @@ -204,23 +204,29 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(load_pdptrs); int kvm_nested_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3, bool nested_tdp, bool reload_pdptrs) { + vcpu->arch.cr3 = cr3; + kvm_register_mark_dirty(vcpu, VCPU_REG_CR3); + + /* Re-initialize the MMU, e.g. to pick up CR4 MMU role changes. */ + kvm_init_mmu(vcpu); + + if (!nested_tdp) + kvm_mmu_new_pgd(vcpu, cr3); + /* * If TDP is enabled, then the PDPTRs either come from the VMCS (EPT) * or are read from memory on-demand (NPT), i.e. aren't loaded here. + * Load PDPTRs *after* initializing the MMU and PGD, to ensure the MMU + * is initialized even if the nested transition ultimately fails. On + * an emulated VM-Exit, failure will put the vCPU in a shutdown state, + * but KVM still needs to keep the MMU up-to-date. On an emulated + * VM-Entry, failure will generate a nested VM-Exit, i.e. L1's state + * and MMU will be explicitly loaded. */ if (reload_pdptrs && !nested_tdp && is_pae_paging(vcpu) && !load_pdptrs(vcpu, cr3)) return -EINVAL; - vcpu->arch.cr3 = cr3; - kvm_register_mark_dirty(vcpu, VCPU_REG_CR3); - - /* Re-initialize the MMU, e.g. to pick up CR4 MMU role changes. */ - kvm_init_mmu(vcpu); - - if (!nested_tdp) - kvm_mmu_new_pgd(vcpu, cr3); - return 0; } EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_nested_load_cr3); -- 2.56.1.703.gf848ed807c-goog