Introduce kvm_x86_ops.split_external_spte() and wrap it in a helper function split_external_spte(). Invoke the helper function split_external_spte() in tdp_mmu_set_spte() to propagate splitting transitions from the mirror page table to the external page table under write mmu_lock. Introduce a new valid transition case for splitting and document all valid transitions of the mirror page table under write mmu_lock in tdp_mmu_set_spte(). Signed-off-by: Xiaoyao Li Signed-off-by: Isaku Yamahata Signed-off-by: Yan Zhao --- v3: - Rename split_external_spt() to split_external_spte(). - Pass in param "old_mirror_spte" to hook kvm_x86_ops.set_external_spte(). This is in aligned with the parameter change in hook kvm_x86_ops.set_external_spte() in Sean's cleanup series, and also allows future DPAMT patches to acquire guest private PFN from the old mirror spte. - Rename param "external_spt" to "new_external_spt" in hook kvm_x86_ops.set_external_spte() to indicate this is a new page table page for the external page table. - Drop declaration of get_external_spt() by moving split_external_spte() after get_external_spt() but before set_external_spte_present() and tdp_mmu_set_spte(). (Kai) - split_external_spte --> split_external_spte() (Kai) RFC v2: - Removed the KVM_BUG_ON() in split_external_spt(). (Rick) - Add a comment for the KVM_BUG_ON() in tdp_mmu_set_spte(). (Rick) - Use kvm_x86_call() instead of static_call(). (Binbin) RFC v1: - Split patch. - Dropped invoking hook zap_private_spte and kvm_flush_remote_tlbs() in KVM MMU core. --- arch/x86/include/asm/kvm-x86-ops.h | 1 + arch/x86/include/asm/kvm_host.h | 4 ++++ arch/x86/kvm/mmu/tdp_mmu.c | 29 +++++++++++++++++++++++++---- 3 files changed, 30 insertions(+), 4 deletions(-) diff --git a/arch/x86/include/asm/kvm-x86-ops.h b/arch/x86/include/asm/kvm-x86-ops.h index 58c5c9b082ca..84fa8689b45c 100644 --- a/arch/x86/include/asm/kvm-x86-ops.h +++ b/arch/x86/include/asm/kvm-x86-ops.h @@ -98,6 +98,7 @@ KVM_X86_OP_OPTIONAL(link_external_spt) KVM_X86_OP_OPTIONAL(set_external_spte) KVM_X86_OP_OPTIONAL(free_external_spt) KVM_X86_OP_OPTIONAL(remove_external_spte) +KVM_X86_OP_OPTIONAL(split_external_spte) KVM_X86_OP_OPTIONAL(alloc_external_fault_cache) KVM_X86_OP_OPTIONAL(topup_external_fault_cache) KVM_X86_OP_OPTIONAL(free_external_fault_cache) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 7818da148a8c..56089d6b9b51 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1848,6 +1848,10 @@ struct kvm_x86_ops { void (*remove_external_spte)(struct kvm *kvm, gfn_t gfn, enum pg_level level, u64 mirror_spte); + /* Split a huge mapping into smaller mappings in external page table */ + int (*split_external_spte)(struct kvm *kvm, gfn_t gfn, enum pg_level level, + u64 old_mirror_spte, void *new_external_spt); + /* Allocation a pages from the external page cache. */ void *(*alloc_external_fault_cache)(struct kvm_vcpu *vcpu); diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index dfa56554f9e0..977914b2627f 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -508,6 +508,19 @@ static void *get_external_spt(gfn_t gfn, u64 new_spte, int level) return NULL; } +static int split_external_spte(struct kvm *kvm, gfn_t gfn, u64 old_spte, + u64 new_spte, int level) +{ + void *new_external_spt = get_external_spt(gfn, new_spte, level); + int ret; + + KVM_BUG_ON(!new_external_spt, kvm); + + ret = kvm_x86_call(split_external_spte)(kvm, gfn, level, old_spte, + new_external_spt); + return ret; +} + static int __must_check set_external_spte_present(struct kvm *kvm, tdp_ptep_t sptep, gfn_t gfn, u64 old_spte, u64 new_spte, int level) @@ -758,12 +771,20 @@ static u64 tdp_mmu_set_spte(struct kvm *kvm, int as_id, tdp_ptep_t sptep, handle_changed_spte(kvm, as_id, gfn, old_spte, new_spte, level, false); /* - * Users that do non-atomic setting of PTEs don't operate on mirror - * roots, so don't handle it and bug the VM if it's seen. + * Propagate changes of SPTE to the external page table under write + * mmu_lock. + * Current valid transitions: + * - present leaf to !present. + * - present non-leaf to !present. + * - present leaf to present non-leaf (splitting) */ if (is_mirror_sptep(sptep)) { - KVM_BUG_ON(is_shadow_present_pte(new_spte), kvm); - remove_external_spte(kvm, gfn, old_spte, level); + if (!is_shadow_present_pte(new_spte)) + remove_external_spte(kvm, gfn, old_spte, level); + else if (is_last_spte(old_spte, level) && !is_last_spte(new_spte, level)) + split_external_spte(kvm, gfn, old_spte, new_spte, level); + else + KVM_BUG_ON(1, kvm); } return old_spte; -- 2.43.2