| Seq | Timestamp | Type | Name | Duration |
|---|---|---|---|---|
| 0/0 | 2026/10/12 23:48 | flow | patch-triage |
5mArgs:null Results: {
"EnableConfigs": [
"KVM",
"KVM_AMD",
"KVM_INTEL"
],
"FocusSymbols": [
"kvm_alloc_tlb_tag",
"kvm_free_tlb_tag",
"invlpga_interception",
"enter_svm_guest_mode",
"nested_svm_vmrun",
"nested_svm_vmexit",
"svm_allocate_nested",
"svm_free_nested",
"svm_leave_nested",
"svm_set_nested_state",
"svm_flush_tlb_all"
],
"KMSANReasoning": "The patch series refactors KVM's TLB tag (VPID/ASID) allocation and updates nested SVM TLB flushing and ASID management:\n1. It replaces the separate VMX VPID bitmap allocator with a generic TLB tag allocator (`kvm_init_tlb_tags`, `kvm_alloc_tlb_tag`, `kvm_free_tlb_tag`), using `bitmap_zalloc()` which guarantees zero-initialized allocations.\n2. It tracks nested ASIDs and flushes on SVM (`asid02`, `last_asid`), explicitly initializing `last_asid = 0` and properly allocating/freeing ASIDs. The parent `struct vcpu_svm` is allocated via `kmem_cache_zalloc()`.\n3. No new kernel-to-userspace copies or information leak vectors are introduced. The internal fields (`asid02`, `last_asid`, `fallback_asid`) are internal to KVM and not copied to userspace via `KVM_GET_NESTED_STATE` or any other ioctl.\n4. Surrounding buffer accesses, bounds checks, and data length calculations are unaffected.\nAny potential risks introduced by this patch series (such as bitmap indexing errors, concurrency/locking races, use-after-free or double-free of tags) are detected by KASAN, LOCKDEP, and built-in WARN_ON_ONCE checks. There are no uninitialized memory risks that would uniquely warrant KMSAN fuzzing.",
"NeedsKMSAN": false,
"Reasoning": "The patch series significantly reworks ASID and VPID management across KVM (x86 MMU, SVM, and VMX). It introduces a centralized TLB tag allocator in kvm/mmu/mmu.c, replaces the generation-based SVM ASID allocator with explicit tag tracking and fallback ASID handling, completely refactors nested SVM TLB flushes on guest mode transitions, reworks INVLPGA instruction interception emulation, and introduces the KVM_REQ_MMU_SYNC_ALL_ROOTS request. It also adds new invariant assertions (WARN_ON_ONCE, KVM_BUG_ON). This code is fully reachable in virtualized environments via /dev/kvm ioctls and nested virtualization execution.",
"WorthFuzzing": true
} |
| 1/1 | 2026/10/12 23:48 | action | read-patch-diff |
0mArgs:null Results: {
"PatchDiff": "commit 21e80c95a8d3388e9dc50d771315b61b19202b62\nAuthor: syz-cluster \u003ctriage@syzkaller.com\u003e\nDate: Mon Oct 12 23:48:32 2026 +0000\n\n syz-cluster: applied patch under review\n\ndiff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst\nindex ddc61ea36ac59..60f49b3675920 100644\n--- a/Documentation/virt/kvm/api.rst\n+++ b/Documentation/virt/kvm/api.rst\n@@ -9770,6 +9770,26 @@ the APIC ID is found in EDX for all subleaves of 0x0b and 0x1f, and in EAX\n for 0x8000001e; the latter also encodes the core id and node id in bits\n 7:0 of EBX and ECX respectively.\n \n+Number of ASIDs (nested SVM)\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+CPUID[0x8000000A]:EBX reports the number of ASIDs supported by the processor\n+on AMD CPUs. KVM's ``KVM_GET_SUPPORTED_CPUID`` hardcodes this value to 8 for\n+historical reasons. This value is arbitrary and does not represent a limit\n+on the supported number of ASIDs, or even a reasonable default value.\n+\n+KVM allows userspace to set any value (up to 0xFFFFFFFF), and emulates those\n+ASIDs by multiplexing them onto hardware ASIDs. AMD CPUs have only ever\n+supported 3 values: 0x40, 0x8000, and 0x1000 -- with modern CPUs advertising\n+0x8000.\n+\n+Advertising a small number of ASIDs could lead an L1 hypervisor to perform\n+unnecessary TLB flushes (e.g. older KVM would flush the entire TLB when running\n+out of ASIDs). Userspace should probably set the number of ASIDs to a\n+reasonably high power of 2 value to remain consistent with hardware and avoid\n+any misbehavior by L1 hypervisors, ideally one of the values advertised by\n+actual hardware.\n+\n Obsolete ioctls and capabilities\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n \ndiff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h\nindex e2eef7057bd35..1d28bb7e92c7f 100644\n--- a/arch/x86/include/asm/kvm_host.h\n+++ b/arch/x86/include/asm/kvm_host.h\n@@ -127,6 +127,7 @@\n \tKVM_ARCH_REQ_FLAGS(33, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)\n #define KVM_REQ_UPDATE_PROTECTED_GUEST_STATE \\\n \tKVM_ARCH_REQ_FLAGS(34, KVM_REQUEST_WAIT)\n+#define KVM_REQ_MMU_SYNC_ALL_ROOTS\tKVM_ARCH_REQ(35)\n \n #define INVALID_PAGE (~(hpa_t)0)\n #define VALID_PAGE(x) ((x) != INVALID_PAGE)\ndiff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c\nindex adafd6fa2c542..e45c01d6faeaf 100644\n--- a/arch/x86/kvm/cpuid.c\n+++ b/arch/x86/kvm/cpuid.c\n@@ -1846,8 +1846,8 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function)\n \t\t\tbreak;\n \t\t}\n \t\tentry-\u003eeax = 1; /* SVM revision 1 */\n-\t\tentry-\u003eebx = 8; /* Lets support 8 ASIDs in case we add proper\n-\t\t\t\t ASID emulation to nested SVM */\n+\t\tentry-\u003eebx = 8; /* Arbitrary value for historical reasons. KVM\n+\t\t\t\t practically allows and ignores all values. */\n \t\tentry-\u003eecx = 0; /* Reserved */\n \t\tcpuid_entry_override(entry, CPUID_8000_000A_EDX);\n \t\tbreak;\ndiff --git a/arch/x86/kvm/hyperv.h b/arch/x86/kvm/hyperv.h\nindex 622a6553e9ac2..b82f70e6fcea1 100644\n--- a/arch/x86/kvm/hyperv.h\n+++ b/arch/x86/kvm/hyperv.h\n@@ -312,18 +312,34 @@ static inline struct kvm_vcpu_hv_tlb_flush_fifo *kvm_hv_get_tlb_flush_fifo(struc\n \treturn \u0026hv_vcpu-\u003etlb_flush_fifo[i];\n }\n \n-static inline void kvm_hv_vcpu_purge_flush_tlb(struct kvm_vcpu *vcpu)\n+/* Purge pending TLB flushes in @fifo, or in all FIFOs if @fifo is NULL */\n+static inline void __kvm_hv_purge_tlb_flush_fifo(struct kvm_vcpu *vcpu,\n+\t\t\t\t\t\t struct kvm_vcpu_hv_tlb_flush_fifo *fifo)\n {\n-\tstruct kvm_vcpu_hv_tlb_flush_fifo *tlb_flush_fifo;\n+\tstruct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu);\n+\tint i;\n \n-\tif (!kvm_check_request(KVM_REQ_HV_TLB_FLUSH, vcpu))\n+\tif (!hv_vcpu || !kvm_check_request(KVM_REQ_HV_TLB_FLUSH, vcpu))\n \t\treturn;\n \n-\ttlb_flush_fifo = kvm_hv_get_tlb_flush_fifo(vcpu, is_guest_mode(vcpu));\n-\tif (!tlb_flush_fifo)\n-\t\treturn;\n+\tfor (i = 0; i \u003c ARRAY_SIZE(hv_vcpu-\u003etlb_flush_fifo); i++) {\n+\t\tif (!fifo || fifo == \u0026hv_vcpu-\u003etlb_flush_fifo[i])\n+\t\t\tkfifo_reset_out(\u0026hv_vcpu-\u003etlb_flush_fifo[i].entries);\n+\t}\n+}\n+\n+static inline void kvm_hv_purge_tlb_flush_fifo(struct kvm_vcpu *vcpu)\n+{\n+\tstruct kvm_vcpu_hv_tlb_flush_fifo *fifo;\n+\n+\tfifo = kvm_hv_get_tlb_flush_fifo(vcpu, is_guest_mode(vcpu));\n+\tif (fifo)\n+\t\t__kvm_hv_purge_tlb_flush_fifo(vcpu, fifo);\n+}\n \n-\tkfifo_reset_out(\u0026tlb_flush_fifo-\u003eentries);\n+static inline void kvm_hv_purge_all_tlb_flush_fifos(struct kvm_vcpu *vcpu)\n+{\n+\t__kvm_hv_purge_tlb_flush_fifo(vcpu, NULL);\n }\n \n static inline bool guest_hv_cpuid_has_l2_tlb_flush(struct kvm_vcpu *vcpu)\n@@ -396,7 +412,8 @@ static inline int kvm_hv_hypercall(struct kvm_vcpu *vcpu)\n {\n \treturn HV_STATUS_ACCESS_DENIED;\n }\n-static inline void kvm_hv_vcpu_purge_flush_tlb(struct kvm_vcpu *vcpu) {}\n+static inline void kvm_hv_purge_tlb_flush_fifo(struct kvm_vcpu *vcpu) {}\n+static inline void kvm_hv_purge_all_tlb_flush_fifos(struct kvm_vcpu *vcpu) {}\n static inline bool kvm_hv_synic_has_vector(struct kvm_vcpu *vcpu, int vector)\n {\n \treturn false;\ndiff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h\nindex 2ae7f9ed4cf86..b6ad0ff72393e 100644\n--- a/arch/x86/kvm/mmu.h\n+++ b/arch/x86/kvm/mmu.h\n@@ -170,6 +170,13 @@ void kvm_mmu_unload(struct kvm_vcpu *vcpu);\n void kvm_mmu_free_obsolete_roots(struct kvm_vcpu *vcpu);\n void kvm_mmu_sync_roots(struct kvm_vcpu *vcpu);\n void kvm_mmu_sync_prev_roots(struct kvm_vcpu *vcpu);\n+\n+static inline void kvm_mmu_sync_all_roots(struct kvm_vcpu *vcpu)\n+{\n+\tkvm_mmu_sync_roots(vcpu);\n+\tkvm_mmu_sync_prev_roots(vcpu);\n+}\n+\n void kvm_mmu_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,\n \t\t\t int bytes);\n \n@@ -410,4 +417,12 @@ static inline bool kvm_is_gfn_alias(struct kvm *kvm, gfn_t gfn)\n {\n \treturn gfn \u0026 kvm_gfn_direct_bits(kvm);\n }\n+\n+typedef unsigned int kvm_tlb_tag_t;\n+\n+int kvm_init_tlb_tags(unsigned int nr_total, unsigned int nr_reserved);\n+void kvm_destroy_tlb_tags(void);\n+kvm_tlb_tag_t kvm_alloc_tlb_tag(void);\n+void kvm_free_tlb_tag(kvm_tlb_tag_t tag);\n+\n #endif\ndiff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c\nindex aadd5bdc2c97a..c096c1a2b4684 100644\n--- a/arch/x86/kvm/mmu/mmu.c\n+++ b/arch/x86/kvm/mmu/mmu.c\n@@ -8178,6 +8178,98 @@ void kvm_mmu_pre_destroy_vm(struct kvm *kvm)\n \t\tvhost_task_stop(kvm-\u003earch.nx_huge_page_recovery_thread);\n }\n \n+static struct {\n+\tspinlock_t\tlock;\n+\tunsigned long\t*bitmap;\n+\tunsigned int\tnr;\n+\tkvm_tlb_tag_t\tstart;\n+} tlb_tags;\n+\n+/*\n+ * Initialize @nr_total TLB tags, where @nr_reserved of those tags are reserved.\n+ * Reserved tags start at 0 and are contiguous. Hence, usable tags are\n+ * [nr_reserved, nr_total). At least one reserved tag is required, as 0 is the\n+ * host TLB tag on both VMX and AMD, and is the value returned on a failed tag\n+ * allocation.\n+ */\n+int kvm_init_tlb_tags(unsigned int nr_total, unsigned int nr_reserved)\n+{\n+\t/*\n+\t * Limit the number of TLB tags to VMX's hardcoded maximum of 0x10000\n+\t * to avoid wasting memory for the bitmap in the unlikely scenario the\n+\t * CPU supports an inordinate number of ASIDs (on AMD). If userspace\n+\t * wants to concurrently run tens of thousands of vCPUs, they'll likely\n+\t * need a solution that works for both Intel and AMD.\n+\t */\n+\tconst unsigned int MAX_NR_TLB_TAGS = VMX_NR_VPIDS;\n+\tunsigned int nr;\n+\n+\tif (WARN_ON_ONCE(!nr_reserved || nr_reserved \u003e nr_total))\n+\t\treturn -EINVAL;\n+\n+\tnr = nr_total - nr_reserved;\n+\tif (!nr)\n+\t\treturn 0;\n+\n+\tif (nr \u003e MAX_NR_TLB_TAGS) {\n+\t\tpr_warn_once(\"Number of TLB tags capped (%u instead of %u)\\n\",\n+\t\t\t MAX_NR_TLB_TAGS, nr);\n+\t\tnr = MAX_NR_TLB_TAGS;\n+\t}\n+\n+\ttlb_tags.bitmap = bitmap_zalloc(nr, GFP_KERNEL);\n+\tif (!tlb_tags.bitmap)\n+\t\treturn -ENOMEM;\n+\n+\ttlb_tags.nr = nr;\n+\ttlb_tags.start = nr_reserved;\n+\tspin_lock_init(\u0026tlb_tags.lock);\n+\treturn 0;\n+}\n+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_init_tlb_tags);\n+\n+void kvm_destroy_tlb_tags(void)\n+{\n+\tbitmap_free(tlb_tags.bitmap);\n+\ttlb_tags.bitmap = NULL;\n+}\n+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_destroy_tlb_tags);\n+\n+kvm_tlb_tag_t kvm_alloc_tlb_tag(void)\n+{\n+\tunsigned int bit;\n+\n+\tif (!tlb_tags.bitmap)\n+\t\treturn 0;\n+\n+\tguard(spinlock)(\u0026tlb_tags.lock);\n+\n+\tbit = find_first_zero_bit(tlb_tags.bitmap, tlb_tags.nr);\n+\tif (bit \u003e= tlb_tags.nr)\n+\t\treturn 0;\n+\n+\t__set_bit(bit, tlb_tags.bitmap);\n+\treturn tlb_tags.start + bit;\n+}\n+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_alloc_tlb_tag);\n+\n+void kvm_free_tlb_tag(kvm_tlb_tag_t tag)\n+{\n+\tunsigned int bit;\n+\n+\tif (!tag || WARN_ON_ONCE(tag \u003c tlb_tags.start))\n+\t\treturn;\n+\n+\tbit = tag - tlb_tags.start;\n+\tif (WARN_ON_ONCE(bit \u003e= tlb_tags.nr))\n+\t\treturn;\n+\n+\tguard(spinlock)(\u0026tlb_tags.lock);\n+\n+\t__clear_bit(bit, tlb_tags.bitmap);\n+}\n+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_free_tlb_tag);\n+\n #ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES\n static bool hugepage_test_mixed(struct kvm_memory_slot *slot, gfn_t gfn,\n \t\t\t\tint level)\ndiff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c\nindex af0ef678433d7..6fe68f2ae282d 100644\n--- a/arch/x86/kvm/svm/nested.c\n+++ b/arch/x86/kvm/svm/nested.c\n@@ -556,10 +556,8 @@ void __nested_copy_vmcb_control_to_cache(struct kvm_vcpu *vcpu,\n \tto-\u003enested_cr3 = from-\u003enested_cr3;\n \tto-\u003epause_filter_count = from-\u003epause_filter_count;\n \tto-\u003epause_filter_thresh = from-\u003epause_filter_thresh;\n-\n-\t/* Copy asid here because nested_vmcb_check_controls() will check it */\n-\tto-\u003easid = from-\u003easid;\n-\tto-\u003eclean = from-\u003eclean;\n+\tto-\u003easid\t\t= from-\u003easid;\n+\tto-\u003eclean\t\t= from-\u003eclean;\n \n #ifdef CONFIG_KVM_HYPERV\n \t/* Hyper-V extensions (Enlightened VMCB) */\n@@ -688,26 +686,69 @@ static void nested_save_pending_event_to_vmcb12(struct vcpu_svm *svm,\n \tvmcb12-\u003econtrol.exit_int_info = exit_int_info;\n }\n \n-static void nested_svm_transition_tlb_flush(struct kvm_vcpu *vcpu)\n+static void nested_svm_entry_tlb_flush(struct kvm_vcpu *vcpu)\n {\n+\tstruct vcpu_svm *svm = to_svm(vcpu);\n+\tbool new_asid = false;\n+\n \t/* Handle pending Hyper-V TLB flush requests */\n \tkvm_hv_nested_transtion_tlb_flush(vcpu, npt_enabled);\n \n+\tif (svm-\u003enested.ctl.asid != svm-\u003enested.last_asid) {\n+\t\tsvm-\u003enested.last_asid = svm-\u003enested.ctl.asid;\n+\t\tnew_asid = true;\n+\t}\n+\n \t/*\n-\t * TODO: optimize unconditional TLB flush/MMU sync. A partial list of\n-\t * things to fix before this can be conditional:\n+\t * If L1 requested a TLB flush for L2, flush L2's TLB on nested entry\n+\t * and sync the nested NPT MMU, as TLB_CONTROL also flushes NPT\n+\t * guest-physical mappings. Sync all roots, since the TLB flush is not\n+\t * specific to the currently loaded nCR3.\n+\t *\n+\t * Handle L1 changing L2's ASID12 similarly, as KVM only uses one ASID\n+\t * for L2 in hardware (per vCPU), so it must start fresh when L1 changes\n+\t * ASID12 to emulate different ASIDs correctly. Additionally, the MMU\n+\t * context is not tagged by the ASID, so the shadow NPTs cannot be\n+\t * reused across different L2 ASIDs.\n+\t *\n+\t * Note, last_asid is initialized as 0, so the first nested VM-Enter\n+\t * after setting EFER.SVME will always flush the TLB to avoid using\n+\t * stale entries.\n \t *\n-\t * - Flush TLBs for both L1 and L2 remote TLB flush\n-\t * - Honor L1's request to flush an ASID on nested VMRUN\n-\t * - Sync nested NPT MMU on VMRUN that flushes L2's ASID[*]\n-\t * - Don't crush a pending TLB flush in vmcb02 on nested VMRUN\n-\t * - Flush L1's ASID on KVM_REQ_TLB_FLUSH_GUEST\n+\t * If L1 requested a full TLB flush for all ASIDs (including its own),\n+\t * L1's own ASID is also flushed on nested VM-Exit, before running L1.\n \t *\n-\t * [*] Unlike nested EPT, SVM's ASID management can invalidate nested\n-\t * NPT guest-physical mappings on VMRUN.\n+\t * TLB_CONTROL_FLUSH_ASID and TLB_CONTROL_FLUSH_ASID_LOCAL are handled\n+\t * equally for simplicity.\n \t */\n-\tkvm_make_request(KVM_REQ_MMU_SYNC, vcpu);\n-\tkvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu);\n+\tif (new_asid || (svm-\u003enested.ctl.tlb_ctl != TLB_CONTROL_DO_NOTHING)) {\n+\t\tif (nested_npt_enabled(svm))\n+\t\t\tkvm_make_request(KVM_REQ_MMU_SYNC_ALL_ROOTS, vcpu);\n+\t\tkvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu);\n+\t}\n+\n+\t/*\n+\t * If L1 and L2 share the same ASID in hardware (when using the fallback\n+\t * ASID for both, or for SEV guests), flush it on nested transitions.\n+\t */\n+\tif (svm-\u003easid == svm-\u003enested.asid02) {\n+\t\tWARN_ON_ONCE(svm-\u003easid != fallback_asid \u0026\u0026 !is_sev_guest(vcpu));\n+\t\tkvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu);\n+\t}\n+}\n+\n+static void nested_svm_exit_tlb_flush(struct kvm_vcpu *vcpu)\n+{\n+\tstruct vcpu_svm *svm = to_svm(vcpu);\n+\n+\tkvm_hv_nested_transtion_tlb_flush(vcpu, npt_enabled);\n+\n+\t/* Flush L1's own ASID if it requested a *full* TLB flush on VMRUN */\n+\tif (svm-\u003enested.ctl.tlb_ctl == TLB_CONTROL_FLUSH_ALL_ASID)\n+\t\tkvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu);\n+\n+\tif (svm-\u003easid == svm-\u003enested.asid02)\n+\t\tkvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu);\n }\n \n /*\n@@ -860,7 +901,7 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm)\n \tstruct vmcb *vmcb01 = svm-\u003evmcb01.ptr;\n \tstruct kvm_vcpu *vcpu = \u0026svm-\u003evcpu;\n \n-\tnested_svm_transition_tlb_flush(vcpu);\n+\tnested_svm_entry_tlb_flush(vcpu);\n \n \t/* Enter Guest-Mode */\n \tenter_guest_mode(vcpu);\n@@ -936,10 +977,7 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm)\n \telse\n \t\tvmcb02-\u003econtrol.bus_lock_counter = 0;\n \n-\t/* Done at vmrun: asid. */\n-\n-\t/* Also overwritten later if necessary. */\n-\tvmcb02-\u003econtrol.tlb_ctl = TLB_CONTROL_DO_NOTHING;\n+\tvmcb02-\u003econtrol.asid = svm-\u003enested.asid02;\n \n \t/* Use vmcb01 MMU and format if guest does not use nNPT */\n \tif (nested_npt_enabled(svm)) {\n@@ -1038,6 +1076,20 @@ static void nested_svm_copy_common_state(struct vmcb *from_vmcb, struct vmcb *to\n \tto_vmcb-\u003esave.spec_ctrl = from_vmcb-\u003esave.spec_ctrl;\n }\n \n+static void nested_svm_service_local_tlb_flushes(struct kvm_vcpu *vcpu)\n+{\n+\tstruct vcpu_svm *svm = to_svm(vcpu);\n+\n+\t/*\n+\t * TLB flushes are applied to the VMCB, so apply any pending TLB flushes\n+\t * on the outgoing VMCB before switching to a new one. A TLB flush could\n+\t * purge the relevant HV TLB flush FIFO depending on guest_mode, so make\n+\t * sure the VMCB and guest_mode context is consistent.\n+\t */\n+\tWARN_ON_ONCE(is_guest_mode(vcpu) != (svm-\u003evmcb == svm-\u003enested.vmcb02.ptr));\n+\tkvm_service_local_tlb_flush_requests(vcpu);\n+}\n+\n int enter_svm_guest_mode(struct kvm_vcpu *vcpu, u64 vmcb12_gpa, bool from_vmrun)\n {\n \tstruct vcpu_svm *svm = to_svm(vcpu);\n@@ -1069,6 +1121,8 @@ int enter_svm_guest_mode(struct kvm_vcpu *vcpu, u64 vmcb12_gpa, bool from_vmrun)\n \n \tnested_svm_copy_common_state(svm-\u003evmcb01.ptr, svm-\u003enested.vmcb02.ptr);\n \n+\tnested_svm_service_local_tlb_flushes(vcpu);\n+\n \tsvm_switch_vmcb(svm, \u0026svm-\u003enested.vmcb02);\n \tnested_vmcb02_prepare_control(svm);\n \tnested_vmcb02_prepare_save(svm);\n@@ -1142,7 +1196,7 @@ int nested_svm_vmrun(struct kvm_vcpu *vcpu)\n \t\treturn ret;\n \t}\n \n-\tif (WARN_ON_ONCE(!svm-\u003enested.initialized))\n+\tif (WARN_ON_ONCE(!svm-\u003enested.vmcb02.ptr))\n \t\treturn -EINVAL;\n \n \tvmcb12_gpa = kvm_rax_read(vcpu);\n@@ -1194,6 +1248,13 @@ int nested_svm_vmrun(struct kvm_vcpu *vcpu)\n \t\tsvm-\u003evmcb-\u003econtrol.exit_info_1 = 0;\n \t\tsvm-\u003evmcb-\u003econtrol.exit_info_2 = 0;\n \n+\t\t/*\n+\t\t * Force a flush+sync on the next nested VMRUN. Otherwise,\n+\t\t * pending MMU syncs due to a changed ASID are lost, as\n+\t\t * last_asid was already updated.\n+\t\t */\n+\t\tsvm-\u003enested.last_asid = 0;\n+\n \t\tnested_svm_vmexit(svm);\n \t}\n \n@@ -1331,7 +1392,9 @@ void nested_svm_vmexit(struct vcpu_svm *svm)\n \tif (nested_svm_vmexit_update_vmcb12(vcpu))\n \t\tkvm_make_request(KVM_REQ_TRIPLE_FAULT, vcpu);\n \n-\t/* Exit Guest-Mode */\n+\tnested_svm_service_local_tlb_flushes(vcpu);\n+\n+\tsvm_switch_vmcb(svm, \u0026svm-\u003evmcb01);\n \tleave_guest_mode(vcpu);\n \tsvm_pmu_handle_nested_transition(svm);\n \n@@ -1358,8 +1421,6 @@ void nested_svm_vmexit(struct vcpu_svm *svm)\n \tif (guest_cpu_cap_has(vcpu, X86_FEATURE_ERAPS))\n \t\tvmcb01-\u003econtrol.erap_ctl |= ERAP_CONTROL_CLEAR_RAP;\n \n-\tsvm_switch_vmcb(svm, \u0026svm-\u003evmcb01);\n-\n \t/*\n \t * Rules for synchronizing int_ctl bits from vmcb02 to vmcb01:\n \t *\n@@ -1438,7 +1499,7 @@ void nested_svm_vmexit(struct vcpu_svm *svm)\n \tsvm-\u003evcpu.arch.dr7 = DR7_FIXED_1;\n \tkvm_update_dr7(\u0026svm-\u003evcpu);\n \n-\tnested_svm_transition_tlb_flush(vcpu);\n+\tnested_svm_exit_tlb_flush(vcpu);\n \n \tnested_svm_uninit_mmu_context(vcpu);\n \n@@ -1489,21 +1550,36 @@ int svm_allocate_nested(struct vcpu_svm *svm)\n {\n \tstruct page *vmcb02_page;\n \n-\tif (svm-\u003enested.initialized)\n+\tif (svm-\u003enested.vmcb02.ptr)\n \t\treturn 0;\n \n \tvmcb02_page = snp_safe_alloc_page();\n \tif (!vmcb02_page)\n \t\treturn -ENOMEM;\n-\tsvm-\u003enested.vmcb02.ptr = page_address(vmcb02_page);\n-\tsvm-\u003enested.vmcb02.pa = __sme_set(page_to_pfn(vmcb02_page) \u003c\u003c PAGE_SHIFT);\n \n \tsvm-\u003enested.msrpm = svm_vcpu_alloc_msrpm();\n \tif (!svm-\u003enested.msrpm)\n \t\tgoto err_free_vmcb02;\n \n \tsvm-\u003enested.vmcb02.cpu = -1;\n-\tsvm-\u003enested.initialized = true;\n+\tsvm-\u003enested.vmcb02.ptr = page_address(vmcb02_page);\n+\tsvm-\u003enested.vmcb02.pa = __sme_set(page_to_pfn(vmcb02_page) \u003c\u003c PAGE_SHIFT);\n+\n+\tsvm-\u003enested.asid02 = allocate_asid(\u0026svm-\u003evcpu);\n+\n+\t/*\n+\t * KVM uses a fallback ASID when out of ASIDs, and fails vCPU creation\n+\t * if there's no fallback ASID. ASID allocation must succeed here.\n+\t */\n+\tKVM_BUG_ON(!svm-\u003enested.asid02, svm-\u003evcpu.kvm);\n+\n+\t/*\n+\t * Clear last_asid to ensure that the ASID is flushed on the first\n+\t * nested VM-Enter. Otherwise, stale TLB entries from a previous life of\n+\t * the ASID (e.g. different vCPU or even different VM) could be used.\n+\t */\n+\tsvm-\u003enested.last_asid = 0;\n+\n \treturn 0;\n \n err_free_vmcb02:\n@@ -1513,12 +1589,14 @@ int svm_allocate_nested(struct vcpu_svm *svm)\n \n void svm_free_nested(struct vcpu_svm *svm)\n {\n-\tif (!svm-\u003enested.initialized)\n+\tif (!svm-\u003enested.vmcb02.ptr)\n \t\treturn;\n \n \tif (WARN_ON_ONCE(svm-\u003evmcb != svm-\u003evmcb01.ptr))\n \t\tsvm_switch_vmcb(svm, \u0026svm-\u003evmcb01);\n \n+\tfree_asid(svm-\u003enested.asid02);\n+\n \tsvm_vcpu_free_msrpm(svm-\u003enested.msrpm);\n \tsvm-\u003enested.msrpm = NULL;\n \n@@ -1533,8 +1611,6 @@ void svm_free_nested(struct vcpu_svm *svm)\n \t * When the vmcb02 is freed, this optimization becomes invalid.\n \t */\n \tsvm-\u003enested.last_vmcb12_gpa = INVALID_GPA;\n-\n-\tsvm-\u003enested.initialized = false;\n }\n \n void svm_leave_nested(struct kvm_vcpu *vcpu)\n@@ -1545,6 +1621,9 @@ void svm_leave_nested(struct kvm_vcpu *vcpu)\n \t\tvcpu-\u003earch.nested_run_pending = 0;\n \t\tsvm-\u003enested.vmcb12_gpa = INVALID_GPA;\n \n+\t\tnested_svm_service_local_tlb_flushes(vcpu);\n+\n+\t\tsvm_switch_vmcb(svm, \u0026svm-\u003evmcb01);\n \t\tleave_guest_mode(vcpu);\n \n \t\t/*\n@@ -1556,7 +1635,7 @@ void svm_leave_nested(struct kvm_vcpu *vcpu)\n \t\t */\n \t\t__svm_pmu_handle_nested_transition(svm, true);\n \n-\t\tsvm_switch_vmcb(svm, \u0026svm-\u003evmcb01);\n+\t\tnested_svm_exit_tlb_flush(vcpu);\n \n \t\tnested_svm_uninit_mmu_context(vcpu);\n \t\tvmcb_mark_all_dirty(svm-\u003evmcb);\n@@ -1853,7 +1932,7 @@ void nested_svm_update_tsc_ratio_msr(struct kvm_vcpu *vcpu)\n \tsvm_write_tsc_multiplier(vcpu);\n }\n \n-/* Inverse operation of nested_copy_vmcb_control_to_cache(). asid is copied too. */\n+/* Inverse operation of nested_copy_vmcb_control_to_cache() */\n static void nested_copy_vmcb_cache_to_control(struct vmcb_control_area *dst,\n \t\t\t\t\t struct vmcb_ctrl_area_cached *from)\n {\n@@ -2084,6 +2163,8 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,\n \tsvm_copy_vmrun_state(vcpu, \u0026svm-\u003evmcb01.ptr-\u003esave, save);\n \tnested_copy_vmcb_control_to_cache(svm, ctl);\n \n+\tnested_svm_service_local_tlb_flushes(vcpu);\n+\n \tsvm_switch_vmcb(svm, \u0026svm-\u003enested.vmcb02);\n \n \tif (use_separate_l2_pat)\ndiff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c\nindex fad72cab76cf4..646108e70e21a 100644\n--- a/arch/x86/kvm/svm/sev.c\n+++ b/arch/x86/kvm/svm/sev.c\n@@ -315,11 +315,6 @@ static int sev_asid_new(struct kvm_sev_info *sev, unsigned long vm_type)\n \treturn ret;\n }\n \n-static unsigned int sev_get_asid(struct kvm *kvm)\n-{\n-\treturn to_kvm_sev_info(kvm)-\u003easid;\n-}\n-\n static void sev_asid_free(struct kvm_sev_info *sev)\n {\n \tstruct svm_cpu_data *sd;\n@@ -2037,6 +2032,28 @@ static void sev_unlock_two_vms(struct kvm *dst_kvm, struct kvm *src_kvm)\n \tatomic_set_release(\u0026src_sev-\u003emigration_in_progress, 0);\n }\n \n+static void sev_vcpu_migrate_asid(struct vcpu_svm *dst_svm,\n+\t\t\t\t unsigned int asid)\n+{\n+\t/*\n+\t * Free the (potentially non-SEV) ASID on the destination vCPU before\n+\t * setting the new SEV ASID.\n+\t */\n+\tfree_asid(dst_svm-\u003easid);\n+\tdst_svm-\u003easid = asid;\n+\n+\t/*\n+\t * If nested is already initialized on the destination vCPU, also free\n+\t * the previous ASID (in case it was non-SEV) and update the nested ASID\n+\t * to match the new SEV ASID.\n+\t */\n+\tif (dst_svm-\u003enested.vmcb02.ptr) {\n+\t\tfree_asid(dst_svm-\u003enested.asid02);\n+\t\tdst_svm-\u003enested.asid02 = asid;\n+\t\tdst_svm-\u003enested.vmcb02.ptr-\u003econtrol.asid = asid;\n+\t}\n+}\n+\n static void sev_migrate_from(struct kvm *dst_kvm, struct kvm *src_kvm)\n {\n \tstruct kvm_sev_info *dst = to_kvm_sev_info(dst_kvm);\n@@ -2101,6 +2118,7 @@ static void sev_migrate_from(struct kvm *dst_kvm, struct kvm *src_kvm)\n \tkvm_for_each_vcpu(i, dst_vcpu, dst_kvm) {\n \t\tdst_svm = to_svm(dst_vcpu);\n \n+\t\tsev_vcpu_migrate_asid(dst_svm, dst-\u003easid);\n \t\tsev_init_vmcb(dst_svm, false);\n \n \t\tif (!dst-\u003ees_active)\n@@ -3623,9 +3641,6 @@ int pre_sev_run(struct vcpu_svm *svm, int cpu)\n \tif (!cpumask_test_cpu(cpu, to_kvm_sev_info(kvm)-\u003ehave_run_cpus))\n \t\tcpumask_set_cpu(cpu, to_kvm_sev_info(kvm)-\u003ehave_run_cpus);\n \n-\t/* Assign the asid allocated with this SEV guest */\n-\tsvm-\u003easid = asid;\n-\n \t/*\n \t * Flush guest TLB:\n \t *\n@@ -3637,7 +3652,7 @@ int pre_sev_run(struct vcpu_svm *svm, int cpu)\n \t\treturn 0;\n \n \tsd-\u003esev_vmcbs[asid] = svm-\u003evmcb;\n-\tsvm-\u003evmcb-\u003econtrol.tlb_ctl = TLB_CONTROL_FLUSH_ASID;\n+\tvmcb_set_flush_asid(svm-\u003evmcb);\n \treturn 0;\n }\n \n@@ -4889,6 +4904,10 @@ void sev_init_vmcb(struct vcpu_svm *svm, bool init_event)\n \tsvm-\u003evmcb-\u003econtrol.misc_ctl |= SVM_MISC_ENABLE_SEV;\n \tclr_exception_intercept(svm, UD_VECTOR);\n \n+\tWARN_ON_ONCE(svm-\u003easid != sev_get_asid(vcpu-\u003ekvm));\n+\tsvm-\u003evmcb-\u003econtrol.asid = svm-\u003easid;\n+\tvmcb_mark_dirty(svm-\u003evmcb, VMCB_ASID);\n+\n \t/*\n \t * Don't intercept #GP for SEV guests, e.g. for the VMware backdoor, as\n \t * KVM can't decrypt guest memory to decode the faulting instruction.\ndiff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c\nindex 136069dfcd44f..436d13f106369 100644\n--- a/arch/x86/kvm/svm/svm.c\n+++ b/arch/x86/kvm/svm/svm.c\n@@ -192,6 +192,8 @@ DEFINE_PER_CPU(struct svm_cpu_data, svm_data);\n \n static DEFINE_MUTEX(vmcb_dump_mutex);\n \n+kvm_tlb_tag_t fallback_asid;\n+\n /*\n * Only MSR_TSC_AUX is switched via the user return hook. EFER is switched via\n * the VMCB, and the SYSCALL/SYSENTER MSRs are handled by VMLOAD/VMSAVE.\n@@ -583,15 +585,6 @@ static int svm_enable_virtualization_cpu(void)\n \t\treturn r;\n \n \tsd = per_cpu_ptr(\u0026svm_data, me);\n-\t/*\n-\t * Bump the current asid_generation value to ensure any vCPU that\n-\t * previously ran on this CPU sees a stale generation and is forced\n-\t * to acquire a new ASID, preventing a latent ASID collision.\n-\t */\n-\tsd-\u003easid_generation++;\n-\tsd-\u003emax_asid = cpuid_ebx(SVM_CPUID_FUNC) - 1;\n-\tsd-\u003enext_asid = sd-\u003emax_asid + 1;\n-\tsd-\u003emin_asid = max_sev_asid + 1;\n \n \twrmsrq(MSR_VM_HSAVE_PA, sd-\u003esave_area_pa);\n \n@@ -994,6 +987,8 @@ static void svm_hardware_unsetup(void)\n \n \t__free_pages(__sme_pa_to_page(iopm_base), get_order(IOPM_SIZE));\n \tiopm_base = 0;\n+\n+\tkvm_destroy_tlb_tags();\n }\n \n static void init_seg(struct vmcb_seg *seg)\n@@ -1246,8 +1241,8 @@ static void init_vmcb(struct kvm_vcpu *vcpu, bool init_event)\n \tif (gmet_enabled)\n \t\tcontrol-\u003emisc_ctl |= SVM_MISC_ENABLE_GMET;\n \n-\tsvm-\u003ecurrent_vmcb-\u003easid_generation = 0;\n-\tsvm-\u003easid = 0;\n+\tcontrol-\u003easid = svm-\u003easid;\n+\tvmcb_set_flush_asid(vmcb);\n \n \tsvm-\u003enested.vmcb12_gpa = INVALID_GPA;\n \tsvm-\u003enested.last_vmcb12_gpa = INVALID_GPA;\n@@ -1375,6 +1370,12 @@ static int svm_vcpu_create(struct kvm_vcpu *vcpu)\n \t\tgoto error_free_avic;\n \t}\n \n+\tsvm-\u003easid = allocate_asid(vcpu);\n+\tif (!svm-\u003easid) {\n+\t\terr = -EINVAL;\n+\t\tgoto error_free_msrpm;\n+\t}\n+\n \tsvm-\u003ex2avic_msrs_intercepted = true;\n \tsvm-\u003elbr_msrs_intercepted = true;\n \n@@ -1386,6 +1387,8 @@ static int svm_vcpu_create(struct kvm_vcpu *vcpu)\n \n \treturn 0;\n \n+error_free_msrpm:\n+\tsvm_vcpu_free_msrpm(svm-\u003emsrpm);\n error_free_avic:\n \tavic_vcpu_free(vcpu);\n error_free_sev:\n@@ -1402,6 +1405,8 @@ static int svm_vcpu_create(struct kvm_vcpu *vcpu)\n static void svm_vcpu_free(struct kvm_vcpu *vcpu)\n {\n \tstruct vcpu_svm *svm = to_svm(vcpu);\n+\tstruct svm_cpu_data *sd;\n+\tint cpu;\n \n \tWARN_ON_ONCE(!list_empty(\u0026svm-\u003eir_list));\n \n@@ -1417,6 +1422,13 @@ static void svm_vcpu_free(struct kvm_vcpu *vcpu)\n \n \t__free_page(__sme_pa_to_page(svm-\u003evmcb01.pa));\n \tsvm_vcpu_free_msrpm(svm-\u003emsrpm);\n+\n+\tfree_asid(svm-\u003easid);\n+\n+\tfor_each_possible_cpu(cpu) {\n+\t\tsd = per_cpu_ptr(\u0026svm_data, cpu);\n+\t\tcmpxchg(\u0026sd-\u003efallback_asid_vcpu, vcpu, NULL);\n+\t}\n }\n \n #ifdef CONFIG_CPU_MITIGATIONS\n@@ -1942,18 +1954,6 @@ static void svm_update_exception_bitmap(struct kvm_vcpu *vcpu)\n \t}\n }\n \n-static void new_asid(struct vcpu_svm *svm, struct svm_cpu_data *sd)\n-{\n-\tif (sd-\u003enext_asid \u003e sd-\u003emax_asid) {\n-\t\t++sd-\u003easid_generation;\n-\t\tsd-\u003enext_asid = sd-\u003emin_asid;\n-\t\tsd-\u003eflush_all_asids = true;\n-\t}\n-\n-\tsvm-\u003ecurrent_vmcb-\u003easid_generation = sd-\u003easid_generation;\n-\tsvm-\u003easid = sd-\u003enext_asid++;\n-}\n-\n static void svm_set_dr6(struct kvm_vcpu *vcpu, unsigned long value)\n {\n \tstruct vmcb *vmcb = to_svm(vcpu)-\u003evmcb;\n@@ -2468,17 +2468,61 @@ static int clgi_interception(struct kvm_vcpu *vcpu)\n \n static int invlpga_interception(struct kvm_vcpu *vcpu)\n {\n+\tstruct vcpu_svm *svm = to_svm(vcpu);\n \t/* FIXME: Handle an address size prefix. */\n \tgva_t gva = kvm_rax_read(vcpu);\n \tu32 asid = kvm_ecx_read(vcpu);\n+\tint cpu;\n \n \tif (nested_svm_check_permissions(vcpu))\n \t\treturn 1;\n \n \ttrace_kvm_invlpga(to_svm(vcpu)-\u003evmcb-\u003esave.rip, asid, gva);\n \n-\t/* Let's treat INVLPGA the same as INVLPG (can be optimized!) */\n-\tkvm_mmu_invlpg(vcpu, gva);\n+\t/*\n+\t * INVLPG on a non-canonical address is a NOP according to the SDM,\n+\t * assume the same behavior from INVLPGA since the APM doesn't specify.\n+\t */\n+\tif (is_noncanonical_invlpg_address(gva, vcpu))\n+\t\treturn kvm_skip_emulated_instruction(vcpu);\n+\n+\t/*\n+\t * If L1 is flushing its own ASID, handle it like INVLPG. Do the same if\n+\t * this is an unintercepted INVLPGA from L2, as\n+\t * kvm_mmu_free_guest_mode_roots() does not handle being in guest mode.\n+\t */\n+\tif (!asid || is_guest_mode(vcpu)) {\n+\t\tkvm_mmu_invlpg(vcpu, gva);\n+\t\treturn kvm_skip_emulated_instruction(vcpu);\n+\t}\n+\n+\t/*\n+\t * Do nothing if L1 is flushing a different L2 ASID than the one KVM is\n+\t * currently tracking. KVM tracks a single L2 ASID, and performs a TLB\n+\t * flush (and MMU resync if needed) when L1 switches ASIDs anyway.\n+\t */\n+\tif (asid != svm-\u003enested.last_asid)\n+\t\treturn kvm_skip_emulated_instruction(vcpu);\n+\n+\t/*\n+\t * Use INVLPGA to flush L2's ASID in hardware if running on the same\n+\t * CPU, otherwise fallback to a full ASID flush.\n+\t */\n+\tcpu = get_cpu();\n+\tif (cpu == svm-\u003enested.vmcb02.cpu)\n+\t\tinvlpga(gva, svm-\u003enested.asid02);\n+\telse\n+\t\tvmcb_set_flush_asid(svm-\u003enested.vmcb02.ptr);\n+\tput_cpu();\n+\n+\t/*\n+\t * If NPT is disabled, free L2's shadow roots as they may have stale\n+\t * translations for the flushed address. Do not sync the mappings for\n+\t * the flushed address, as it involves walking L2 shadow page tables\n+\t * with L1's MMU context.\n+\t */\n+\tif (!npt_enabled)\n+\t\tkvm_mmu_free_guest_mode_roots(vcpu-\u003ekvm, \u0026vcpu-\u003earch.root_mmu);\n \n \treturn kvm_skip_emulated_instruction(vcpu);\n }\n@@ -3875,37 +3919,28 @@ static void svm_set_nested_run_soft_int_state(struct kvm_vcpu *vcpu)\n \n static int pre_svm_run(struct kvm_vcpu *vcpu)\n {\n-\tstruct svm_cpu_data *sd = per_cpu_ptr(\u0026svm_data, vcpu-\u003ecpu);\n+\tstruct svm_cpu_data *sd = this_cpu_ptr(\u0026svm_data);\n \tstruct vcpu_svm *svm = to_svm(vcpu);\n \n \t/*\n \t * If the previous VMRUN of the VMCB occurred on a different physical\n-\t * cpu, then mark the VMCB dirty as hardware's clean bits are per pCPU.\n-\t *\n-\t * Reset the ASID generation in both VMCBs. This will lead to assigning\n-\t * a new ASID now, and then again when switching to the other VMCB.\n-\t * However, this is needed as the ASID is shared between the VMCBs, and\n-\t * otherwise it would be possible to use an ASID allocated on one pCPU\n-\t * on another, for example:\n-\t * - vCPU migrates from pCPU A to pCPU B, allocates a new ASID.\n-\t * - vCPU migrates back to pCPU A, and then switches the VMCB.\n-\t * - The new VMCB does not detect a pCPU change and runs on pCPU A with\n-\t * the new ASID allocated on pCPU B, which is potentially used by\n-\t * another vCPU/VM.\n+\t * cpu, then mark the VMCB dirty and flush the ASID, as hardware's clean\n+\t * bits and TLB entries are per pCPU.\n \t */\n \tif (unlikely(svm-\u003ecurrent_vmcb-\u003ecpu != vcpu-\u003ecpu)) {\n \t\tvmcb_mark_all_dirty(svm-\u003evmcb);\n \t\tsvm-\u003ecurrent_vmcb-\u003ecpu = vcpu-\u003ecpu;\n-\t\tsvm-\u003evmcb01.asid_generation = 0;\n-\t\tsvm-\u003enested.vmcb02.asid_generation = 0;\n+\t\tvmcb_set_flush_asid(svm-\u003evmcb);\n }\n \n \tif (is_sev_guest(vcpu))\n \t\treturn pre_sev_run(svm, vcpu-\u003ecpu);\n \n-\t/* FIXME: handle wraparound of asid_generation */\n-\tif (svm-\u003ecurrent_vmcb-\u003easid_generation != sd-\u003easid_generation)\n-\t\tnew_asid(svm, sd);\n+\tif (unlikely(svm-\u003evmcb-\u003econtrol.asid == fallback_asid \u0026\u0026\n+\t\t READ_ONCE(sd-\u003efallback_asid_vcpu) != vcpu)) {\n+\t\tvmcb_set_flush_asid(svm-\u003evmcb);\n+\t\tWRITE_ONCE(sd-\u003efallback_asid_vcpu, vcpu);\n+\t}\n \n \treturn 0;\n }\n@@ -4317,7 +4352,7 @@ static void svm_flush_tlb_asid(struct kvm_vcpu *vcpu)\n \t * A TLB flush for the current ASID flushes both \"host\" and \"guest\" TLB\n \t * entries, and thus is a superset of Hyper-V's fine grained flushing.\n \t */\n-\tkvm_hv_vcpu_purge_flush_tlb(vcpu);\n+\tkvm_hv_purge_tlb_flush_fifo(vcpu);\n \n \t/*\n \t * Flush only the current ASID even if the TLB flush was invoked via\n@@ -4326,10 +4361,7 @@ static void svm_flush_tlb_asid(struct kvm_vcpu *vcpu)\n \t * unconditionally does a TLB flush on both nested VM-Enter and nested\n \t * VM-Exit (via kvm_mmu_reset_context()).\n \t */\n-\tif (cpu_feature_enabled(X86_FEATURE_FLUSHBYASID))\n-\t\tsvm-\u003evmcb-\u003econtrol.tlb_ctl = TLB_CONTROL_FLUSH_ASID;\n-\telse\n-\t\tsvm-\u003ecurrent_vmcb-\u003easid_generation--;\n+\tvmcb_set_flush_asid(svm-\u003evmcb);\n }\n \n static void svm_flush_tlb_current(struct kvm_vcpu *vcpu)\n@@ -4350,6 +4382,8 @@ static void svm_flush_tlb_current(struct kvm_vcpu *vcpu)\n \n static void svm_flush_tlb_all(struct kvm_vcpu *vcpu)\n {\n+\tstruct vcpu_svm *svm = to_svm(vcpu);\n+\n \t/*\n \t * When running on Hyper-V with EnlightenedNptTlb enabled, remote TLB\n \t * flushes should be routed to hv_flush_remote_tlbs() without requesting\n@@ -4360,7 +4394,11 @@ static void svm_flush_tlb_all(struct kvm_vcpu *vcpu)\n \tif (WARN_ON_ONCE(svm_hv_is_enlightened_tlb_enabled(vcpu)))\n \t\thv_flush_remote_tlbs(vcpu-\u003ekvm);\n \n-\tsvm_flush_tlb_asid(vcpu);\n+\tkvm_hv_purge_all_tlb_flush_fifos(vcpu);\n+\n+\tvmcb_set_flush_asid(svm-\u003evmcb01.ptr);\n+\tif (svm-\u003enested.vmcb02.ptr)\n+\t\tvmcb_set_flush_asid(svm-\u003enested.vmcb02.ptr);\n }\n \n static void svm_flush_tlb_guest(struct kvm_vcpu *vcpu)\n@@ -4647,13 +4685,6 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)\n \n \tsync_lapic_to_cr8(vcpu);\n \n-\tif (unlikely(svm-\u003easid != svm-\u003evmcb-\u003econtrol.asid)) {\n-\t\tsvm-\u003evmcb-\u003econtrol.asid = svm-\u003easid;\n-\t\tvmcb_mark_dirty(svm-\u003evmcb, VMCB_ASID);\n-\t}\n-\tif (this_cpu_ptr(\u0026svm_data)-\u003eflush_all_asids)\n-\t\tsvm-\u003evmcb-\u003econtrol.tlb_ctl = TLB_CONTROL_FLUSH_ALL_ASID;\n-\n \tsvm-\u003evmcb-\u003esave.cr2 = vcpu-\u003earch.cr2;\n \n \tif (guest_cpu_cap_has(vcpu, X86_FEATURE_ERAPS) \u0026\u0026\n@@ -4745,8 +4776,7 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)\n \t}\n \n \tif (!svm_is_vmrun_failure(svm-\u003evmcb-\u003econtrol.exit_code)) {\n-\t\tthis_cpu_ptr(\u0026svm_data)-\u003eflush_all_asids = false;\n-\t\tsvm-\u003evmcb-\u003econtrol.tlb_ctl = TLB_CONTROL_DO_NOTHING;\n+\t\tvmcb_clr_flush_asid(svm-\u003evmcb);\n \n \t\t/*\n \t\t * Unconditionally mask off the CLEAR_RAP bit, the AND is just\n@@ -5659,9 +5689,8 @@ static __init void svm_set_cpu_caps(void)\n \t\tkvm_cpu_cap_set(X86_FEATURE_VMCBCLEAN);\n \n \t\t/*\n-\t\t * KVM currently flushes TLBs on *every* nested SVM transition,\n-\t\t * and so for all intents and purposes KVM supports flushing by\n-\t\t * ASID, i.e. KVM is guaranteed to honor every L1 ASID flush.\n+\t\t * KVM handles all TLB_CONTROL values set by L1, even if the\n+\t\t * underlying CPU does not.\n \t\t */\n \t\tkvm_cpu_cap_set(X86_FEATURE_FLUSHBYASID);\n \n@@ -5757,6 +5786,7 @@ static __init void svm_set_cpu_caps(void)\n \n static __init int svm_hardware_setup(void)\n {\n+\tunsigned long nr_asids;\n \tvoid *iopm_va;\n \tint cpu, r;\n \n@@ -5914,6 +5944,14 @@ static __init int svm_hardware_setup(void)\n \n \tkvm_caps.inapplicable_quirks \u0026= ~KVM_X86_QUIRK_CD_NW_CLEARED;\n \n+\t/* Consumes max_sev_asid initialized by sev_hardware_setup() */\n+\tnr_asids = cpuid_ebx(SVM_CPUID_FUNC);\n+\tr = kvm_init_tlb_tags(nr_asids, max_sev_asid + 1);\n+\tif (r)\n+\t\tgoto err;\n+\n+\tfallback_asid = kvm_alloc_tlb_tag();\n+\n \tfor_each_possible_cpu(cpu) {\n \t\tr = svm_cpu_init(cpu);\n \t\tif (r)\ndiff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h\nindex 6d0242966dc87..15c4aa3b02c2d 100644\n--- a/arch/x86/kvm/svm/svm.h\n+++ b/arch/x86/kvm/svm/svm.h\n@@ -26,6 +26,7 @@\n #include \"regs.h\"\n #include \"x86.h\"\n #include \"pmu.h\"\n+#include \"mmu.h\"\n \n /*\n * Helpers to convert to/from physical addresses for pages whose address is\n@@ -145,7 +146,6 @@ struct kvm_vmcb_info {\n \tstruct vmcb *ptr;\n \tunsigned long pa;\n \tint cpu;\n-\tuint64_t asid_generation;\n };\n \n struct vmcb_save_area_cached {\n@@ -234,7 +234,7 @@ struct svm_nested_state {\n \t */\n \tstruct vmcb_save_area_cached save;\n \n-\tbool initialized;\n+\tkvm_tlb_tag_t asid02;\n \n \t/*\n \t * Indicates whether MSR bitmap for L2 needs to be rebuilt due to\n@@ -244,6 +244,8 @@ struct svm_nested_state {\n \t * on its side.\n \t */\n \tbool force_msr_bitmap_recalc;\n+\n+\tu32 last_asid;\n };\n \n struct vcpu_sev_es_state {\n@@ -285,7 +287,7 @@ struct vcpu_svm {\n \tstruct vmcb *vmcb;\n \tstruct kvm_vmcb_info vmcb01;\n \tstruct kvm_vmcb_info *current_vmcb;\n-\tu32 asid;\n+\tkvm_tlb_tag_t asid;\n \tu32 sysenter_esp_hi;\n \tu32 sysenter_eip_hi;\n \tuint64_t tsc_aux;\n@@ -372,17 +374,14 @@ struct vcpu_svm {\n };\n \n struct svm_cpu_data {\n-\tu64 asid_generation;\n-\tu32 max_asid;\n-\tu32 next_asid;\n-\tu32 min_asid;\n-\n-\tbool flush_all_asids;\n \tbool bp_spec_reduce_set;\n \n \tstruct vmcb *save_area;\n \tunsigned long save_area_pa;\n \n+\t/* Last vCPU to use fallback_asid on this CPU */\n+\tstruct kvm_vcpu *fallback_asid_vcpu;\n+\n \t/* index = sev_asid, value = vmcb pointer */\n \tstruct vmcb **sev_vmcbs;\n };\n@@ -432,6 +431,11 @@ static __always_inline bool is_sev_snp_guest(struct kvm_vcpu *vcpu)\n {\n \treturn ____sev_snp_guest(vcpu-\u003ekvm);\n }\n+\n+static __always_inline unsigned int sev_get_asid(struct kvm *kvm)\n+{\n+\treturn to_kvm_sev_info(kvm)-\u003easid;\n+}\n #else\n static __always_inline bool is_sev_guest(struct kvm_vcpu *vcpu)\n {\n@@ -446,6 +450,11 @@ static __always_inline bool is_sev_snp_guest(struct kvm_vcpu *vcpu)\n {\n \treturn false;\n }\n+\n+static __always_inline unsigned int sev_get_asid(struct kvm *kvm)\n+{\n+\treturn 0;\n+}\n #endif\n \n static inline bool ghcb_gpa_is_registered(struct vcpu_svm *svm, u64 val)\n@@ -480,6 +489,19 @@ static inline void vmcb_set_gpat(struct vmcb *vmcb, u64 data)\n \tvmcb_mark_dirty(vmcb, VMCB_NPT);\n }\n \n+static inline void vmcb_set_flush_asid(struct vmcb *vmcb)\n+{\n+\tif (cpu_feature_enabled(X86_FEATURE_FLUSHBYASID))\n+\t\tvmcb-\u003econtrol.tlb_ctl = TLB_CONTROL_FLUSH_ASID;\n+\telse\n+\t\tvmcb-\u003econtrol.tlb_ctl = TLB_CONTROL_FLUSH_ALL_ASID;\n+}\n+\n+static inline void vmcb_clr_flush_asid(struct vmcb *vmcb)\n+{\n+\tvmcb-\u003econtrol.tlb_ctl = TLB_CONTROL_DO_NOTHING;\n+}\n+\n static __always_inline struct vcpu_svm *to_svm(struct kvm_vcpu *vcpu)\n {\n \treturn container_of(vcpu, struct vcpu_svm, vcpu);\n@@ -1057,6 +1079,21 @@ static inline struct vmcb_save_area *sev_decrypt_vmsa(struct kvm_vcpu *vcpu)\n static inline void sev_free_decrypted_vmsa(struct kvm_vcpu *vcpu, struct vmcb_save_area *vmsa) {}\n #endif\n \n+extern kvm_tlb_tag_t fallback_asid;\n+\n+static inline kvm_tlb_tag_t allocate_asid(struct kvm_vcpu *vcpu)\n+{\n+\tif (is_sev_guest(vcpu))\n+\t\treturn sev_get_asid(vcpu-\u003ekvm);\n+\treturn kvm_alloc_tlb_tag() ?: fallback_asid;\n+}\n+\n+static inline void free_asid(kvm_tlb_tag_t asid)\n+{\n+\tif (asid \u003e max_sev_asid \u0026\u0026 asid != fallback_asid)\n+\t\tkvm_free_tlb_tag(asid);\n+}\n+\n /* vmenter.S */\n \n void __svm_sev_es_vcpu_run(struct vcpu_svm *svm, unsigned int flags,\ndiff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c\nindex b341af4c6057a..a1caa2bd04d7a 100644\n--- a/arch/x86/kvm/vmx/vmx.c\n+++ b/arch/x86/kvm/vmx/vmx.c\n@@ -598,9 +598,6 @@ DEFINE_PER_CPU(struct vmcs *, current_vmcs);\n */\n static DEFINE_PER_CPU(struct list_head, loaded_vmcss_on_cpu);\n \n-static DECLARE_BITMAP(vmx_vpid_bitmap, VMX_NR_VPIDS);\n-static DEFINE_SPINLOCK(vmx_vpid_lock);\n-\n struct vmcs_config vmcs_config __ro_after_init;\n struct vmx_capability vmx_capability __ro_after_init;\n \n@@ -4092,31 +4089,6 @@ static void seg_setup(int seg)\n \tvmcs_write32(sf-\u003ear_bytes, ar);\n }\n \n-int allocate_vpid(void)\n-{\n-\tint vpid;\n-\n-\tif (!enable_vpid)\n-\t\treturn 0;\n-\tspin_lock(\u0026vmx_vpid_lock);\n-\tvpid = find_first_zero_bit(vmx_vpid_bitmap, VMX_NR_VPIDS);\n-\tif (vpid \u003c VMX_NR_VPIDS)\n-\t\t__set_bit(vpid, vmx_vpid_bitmap);\n-\telse\n-\t\tvpid = 0;\n-\tspin_unlock(\u0026vmx_vpid_lock);\n-\treturn vpid;\n-}\n-\n-void free_vpid(int vpid)\n-{\n-\tif (!enable_vpid || vpid == 0)\n-\t\treturn;\n-\tspin_lock(\u0026vmx_vpid_lock);\n-\t__clear_bit(vpid, vmx_vpid_bitmap);\n-\tspin_unlock(\u0026vmx_vpid_lock);\n-}\n-\n static void vmx_msr_bitmap_l01_changed(struct vcpu_vmx *vmx)\n {\n \t/*\n@@ -8415,6 +8387,8 @@ void vmx_hardware_unsetup(void)\n \n \tif (nested)\n \t\tnested_vmx_hardware_unsetup();\n+\n+\tdestroy_vpids();\n }\n \n void vmx_vm_destroy(struct kvm *kvm)\n@@ -8640,8 +8614,6 @@ __init int vmx_hardware_setup(void)\n \tkvm_caps.has_bus_lock_exit = cpu_has_vmx_bus_lock_detection();\n \tkvm_caps.has_notify_vmexit = cpu_has_notify_vmexit();\n \n-\tset_bit(0, vmx_vpid_bitmap); /* 0 is reserved for host */\n-\n \tif (enable_ept)\n \t\tkvm_mmu_set_ept_masks(enable_ept_ad_bits);\n \telse\n@@ -8681,6 +8653,10 @@ __init int vmx_hardware_setup(void)\n \n \tvmx_set_cpu_caps();\n \n+\tr = init_vpids();\n+\tif (r)\n+\t\treturn r;\n+\n \t/*\n \t * Configure nested capabilities after core CPU capabilities so that\n \t * nested support can be conditional on base support, e.g. so that KVM\n@@ -8688,8 +8664,10 @@ __init int vmx_hardware_setup(void)\n \t */\n \tif (nested) {\n \t\tr = nested_vmx_hardware_setup(kvm_vmx_exit_handlers);\n-\t\tif (r)\n+\t\tif (r) {\n+\t\t\tdestroy_vpids();\n \t\t\treturn r;\n+\t\t}\n \t}\n \tvmx_nested_ops.enabled = nested;\n \ndiff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h\nindex 6c9dd5b2365d5..3e1ecd9148e9c 100644\n--- a/arch/x86/kvm/vmx/vmx.h\n+++ b/arch/x86/kvm/vmx/vmx.h\n@@ -182,7 +182,7 @@ struct nested_vmx {\n \tu64 pre_vmenter_ssp;\n \tu64 pre_vmenter_ssp_tbl;\n \n-\tu16 vpid02;\n+\tkvm_tlb_tag_t vpid02;\n \tu16 last_vpid;\n \n \tint tsc_autostore_slot;\n@@ -256,7 +256,7 @@ struct vcpu_vmx {\n \t\t\tu32 ar;\n \t\t} seg[8];\n \t} segment_cache;\n-\tint vpid;\n+\tkvm_tlb_tag_t vpid;\n \n \t/* Support for a guest hypervisor (nested VMX) */\n \tstruct nested_vmx nested;\n@@ -297,9 +297,29 @@ struct kvm_vmx {\n \tu64 *pid_table;\n };\n \n+static __always_inline int init_vpids(void)\n+{\n+\treturn enable_vpid ? kvm_init_tlb_tags(VMX_NR_VPIDS, 1) : 0;\n+}\n+\n+static __always_inline void destroy_vpids(void)\n+{\n+\tif (enable_vpid)\n+\t\tkvm_destroy_tlb_tags();\n+}\n+\n+static __always_inline kvm_tlb_tag_t allocate_vpid(void)\n+{\n+\treturn enable_vpid ? kvm_alloc_tlb_tag() : 0;\n+}\n+\n+static __always_inline void free_vpid(kvm_tlb_tag_t vpid)\n+{\n+\tif (enable_vpid)\n+\t\tkvm_free_tlb_tag(vpid);\n+}\n+\n void vmx_vcpu_load_vmcs(struct kvm_vcpu *vcpu, int cpu);\n-int allocate_vpid(void);\n-void free_vpid(int vpid);\n void vmx_set_constant_host_state(struct vcpu_vmx *vmx);\n void vmx_prepare_switch_to_guest(struct kvm_vcpu *vcpu);\n void vmx_set_host_fs_gs(struct vmcs_host_state *host, u16 fs_sel, u16 gs_sel,\ndiff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c\nindex 32563a91fba99..a54e9da2ed09e 100644\n--- a/arch/x86/kvm/x86.c\n+++ b/arch/x86/kvm/x86.c\n@@ -2020,8 +2020,7 @@ static void kvm_vcpu_flush_tlb_guest(struct kvm_vcpu *vcpu)\n \t\t * a forced sync of the shadow page tables. Ensure all the\n \t\t * roots are synced and the guest TLB in hardware is clean.\n \t\t */\n-\t\tkvm_mmu_sync_roots(vcpu);\n-\t\tkvm_mmu_sync_prev_roots(vcpu);\n+\t\tkvm_mmu_sync_all_roots(vcpu);\n \t}\n \n \tkvm_x86_call(flush_tlb_guest)(vcpu);\n@@ -2030,7 +2029,7 @@ static void kvm_vcpu_flush_tlb_guest(struct kvm_vcpu *vcpu)\n \t * Flushing all \"guest\" TLB is always a superset of Hyper-V's fine\n \t * grained flushing.\n \t */\n-\tkvm_hv_vcpu_purge_flush_tlb(vcpu);\n+\tkvm_hv_purge_tlb_flush_fifo(vcpu);\n }\n \n \n@@ -8167,8 +8166,14 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)\n \t\t\tif (unlikely(r))\n \t\t\t\tgoto out;\n \t\t}\n-\t\tif (kvm_check_request(KVM_REQ_MMU_SYNC, vcpu))\n+\n+\t\tif (kvm_check_request(KVM_REQ_MMU_SYNC_ALL_ROOTS, vcpu)) {\n+\t\t\tkvm_mmu_sync_all_roots(vcpu);\n+\t\t\tkvm_clear_request(KVM_REQ_MMU_SYNC, vcpu);\n+\t\t} else if (kvm_check_request(KVM_REQ_MMU_SYNC, vcpu)) {\n \t\t\tkvm_mmu_sync_roots(vcpu);\n+\t\t}\n+\n \t\tif (kvm_check_request(KVM_REQ_LOAD_MMU_PGD, vcpu))\n \t\t\tkvm_mmu_load_pgd(vcpu);\n \ndiff --git a/tools/testing/selftests/kvm/Makefile.kvm b/tools/testing/selftests/kvm/Makefile.kvm\nindex be5eb2dd85aa2..5347c56786b08 100644\n--- a/tools/testing/selftests/kvm/Makefile.kvm\n+++ b/tools/testing/selftests/kvm/Makefile.kvm\n@@ -122,6 +122,7 @@ TEST_GEN_PROGS_x86 += x86/svm_nested_shutdown_test\n TEST_GEN_PROGS_x86 += x86/svm_nested_soft_inject_test\n TEST_GEN_PROGS_x86 += x86/svm_nested_vmcb12_gpa\n TEST_GEN_PROGS_x86 += x86/svm_nested_pat_test\n+TEST_GEN_PROGS_x86 += x86/nested_tlb_flush_test\n TEST_GEN_PROGS_x86 += x86/svm_lbr_nested_state\n TEST_GEN_PROGS_x86 += x86/svm_pmu_host_guest_test\n TEST_GEN_PROGS_x86 += x86/tsc_scaling_sync\ndiff --git a/tools/testing/selftests/kvm/include/x86/processor.h b/tools/testing/selftests/kvm/include/x86/processor.h\nindex ffec2291b1e1f..62d255aee5a34 100644\n--- a/tools/testing/selftests/kvm/include/x86/processor.h\n+++ b/tools/testing/selftests/kvm/include/x86/processor.h\n@@ -1617,6 +1617,7 @@ void tdp_map(struct kvm_vm *vm, gpa_t l2_gpa, gpa_t gpa, u64 size);\n void tdp_identity_map_default_memslots(struct kvm_vm *vm);\n void tdp_identity_map_1g(struct kvm_vm *vm, u64 addr, u64 size);\n u64 *tdp_get_pte(struct kvm_vm *vm, u64 l2_gpa);\n+gpa_t tdp_copy_page_tables(struct kvm_vm *vm);\n \n /*\n * Basic CPU control in CR0\ndiff --git a/tools/testing/selftests/kvm/include/x86/svm.h b/tools/testing/selftests/kvm/include/x86/svm.h\nindex c8539166270ea..d2a6bf8503f78 100644\n--- a/tools/testing/selftests/kvm/include/x86/svm.h\n+++ b/tools/testing/selftests/kvm/include/x86/svm.h\n@@ -316,4 +316,9 @@ struct __attribute__ ((__packed__)) vmcb {\n \n #define SVM_CR0_SELECTIVE_MASK (X86_CR0_TS | X86_CR0_MP)\n \n+static inline void invlpga(unsigned long addr, u32 asid)\n+{\n+\tasm volatile(\"invlpga %0, %1\" : : \"a\"(addr), \"c\"(asid) : \"memory\");\n+}\n+\n #endif /* SELFTEST_KVM_SVM_H */\ndiff --git a/tools/testing/selftests/kvm/include/x86/vmx.h b/tools/testing/selftests/kvm/include/x86/vmx.h\nindex 8fd97434eee5c..4f2a0e3ac5ba1 100644\n--- a/tools/testing/selftests/kvm/include/x86/vmx.h\n+++ b/tools/testing/selftests/kvm/include/x86/vmx.h\n@@ -518,4 +518,25 @@ static inline bool kvm_cpu_has_vmx_virtualize_apic_accesses(void)\n void vm_enable_ept(struct kvm_vm *vm);\n void prepare_virtualize_apic_accesses(struct vmx_pages *vmx, struct kvm_vm *vm);\n \n+static inline void invvpid(unsigned long ext, u16 vpid, gva_t gva)\n+{\n+\tstruct {\n+\t\tu64 vpid : 16;\n+\t\tu64 rsvd : 48;\n+\t\tu64 gva;\n+\t} operand = { vpid, 0, gva };\n+\n+\tasm volatile(\"invvpid %0, %1\" : : \"m\"(operand), \"r\"(ext) : \"memory\");\n+}\n+\n+static inline void invept(unsigned long ext, u64 eptp)\n+{\n+\tstruct {\n+\t\tu64 eptp;\n+\t\tu64 rsvd;\n+\t} operand = { eptp, 0 };\n+\n+\tasm volatile(\"invept %0, %1\" : : \"m\"(operand), \"r\"(ext) : \"memory\");\n+}\n+\n #endif /* SELFTEST_KVM_VMX_H */\ndiff --git a/tools/testing/selftests/kvm/lib/x86/processor.c b/tools/testing/selftests/kvm/lib/x86/processor.c\nindex 8374afd08bafe..57ecf8b381b5a 100644\n--- a/tools/testing/selftests/kvm/lib/x86/processor.c\n+++ b/tools/testing/selftests/kvm/lib/x86/processor.c\n@@ -554,6 +554,42 @@ void tdp_identity_map_1g(struct kvm_vm *vm, u64 addr, u64 size)\n \t__tdp_map(vm, addr, addr, size, PG_LEVEL_1G);\n }\n \n+static gpa_t __virt_copy_page_tables(struct kvm_vm *vm, struct kvm_mmu *mmu,\n+\t\t\t\t gpa_t table_gpa, int level)\n+{\n+\tgpa_t copy_gpa = vm_alloc_page_table(vm);\n+\tu64 *copy = addr_gpa2hva(vm, copy_gpa);\n+\tgpa_t pa;\n+\tint i;\n+\n+\tmemcpy(copy, addr_gpa2hva(vm, table_gpa), vm-\u003epage_size);\n+\tif (level == PG_LEVEL_4K)\n+\t\treturn copy_gpa;\n+\n+\tfor (i = 0; i \u003c vm-\u003epage_size / sizeof(u64); i++) {\n+\t\tif (!is_present_pte(mmu, \u0026copy[i]) || is_huge_pte(mmu, \u0026copy[i]))\n+\t\t\tcontinue;\n+\n+\t\t/* Clear only the address bits, preserving C/S bits and flags */\n+\t\tpa = vm_untag_gpa(vm, PTE_GET_PA(copy[i]));\n+\t\tcopy[i] \u0026= ~pa;\n+\t\tcopy[i] |= __virt_copy_page_tables(vm, mmu, pa, level - 1);\n+\t}\n+\treturn copy_gpa;\n+}\n+\n+/*\n+ * Deep copy all levels of the TDP page tables, such that no page table pages\n+ * are shared with the original. Returns the GPA of the new root.\n+ */\n+gpa_t tdp_copy_page_tables(struct kvm_vm *vm)\n+{\n+\tstruct kvm_mmu *mmu = \u0026vm-\u003estage2_mmu;\n+\n+\tTEST_ASSERT(mmu-\u003epgd_created, \"TDP page tables not created\");\n+\treturn __virt_copy_page_tables(vm, mmu, mmu-\u003epgd, mmu-\u003epgtable_levels);\n+}\n+\n /*\n * Set Unusable Segment\n *\ndiff --git a/tools/testing/selftests/kvm/x86/nested_tlb_flush_test.c b/tools/testing/selftests/kvm/x86/nested_tlb_flush_test.c\nnew file mode 100644\nindex 0000000000000..fb32b7bf6b19f\n--- /dev/null\n+++ b/tools/testing/selftests/kvm/x86/nested_tlb_flush_test.c\n@@ -0,0 +1,526 @@\n+// SPDX-License-Identifier: GPL-2.0-only\n+/*\n+ * Copyright (C) 2026, Google LLC.\n+ *\n+ * Test TLB flushes with nested virtualization across three cases:\n+ * 1. When L1 updates L2's (shadow) page tables\n+ * 2. When L1 updates nested TDP\n+ * 3. When KVM updates mappings on the host (KVM-induced TLB flushes)\n+ */\n+#include \u003cfcntl.h\u003e\n+#include \u003cstdio.h\u003e\n+#include \u003cstdlib.h\u003e\n+#include \u003cstring.h\u003e\n+\n+#include \"test_util.h\"\n+#include \"kvm_util.h\"\n+#include \"processor.h\"\n+#include \"svm_util.h\"\n+#include \"vmx.h\"\n+\n+#define NR_ITERATIONS\t100\n+\n+#define VAL1\t\t0x11111111ULL\n+#define VAL2\t\t0x22222222ULL\n+\n+#define TEST_VADDR\t0x10000000ULL\n+#define TEST_GPA\t0x30000000ULL\n+#define TEST_NESTED_GPA\t0x40000000ULL\n+\n+#define TEST_MEMSLOT\t10\n+\n+#define SYNC_MOVE_MEMSLOT 1\n+\n+enum guest_flush_method {\n+\tTLB_FLUSH_TAG = 0,\n+\tTLB_FLUSH_VADDR,\n+\tTLB_CHANGE_TAG,\n+\tTLB_FLUSH_NONE,\n+};\n+\n+static u64 *pte_gva;\n+static gpa_t test_gpa[2];\n+static enum guest_flush_method flush_method;\n+static u32 max_asid;\n+\n+/* TDP roots for the multi-root test, root 1 is a non-aliasing copy of root 0 */\n+static gpa_t tdp_root_gpa[2];\n+/* Value L2 expects at TEST_VADDR, 0 if L2 shouldn't access TEST_VADDR */\n+static u64 l2_expected_val;\n+\n+static void svm_tlb_flush(struct svm_test_data *svm,\n+\t\t\t enum guest_flush_method method)\n+{\n+\tstruct vmcb *vmcb = svm-\u003evmcb;\n+\n+\tswitch (method) {\n+\tcase TLB_CHANGE_TAG:\n+\t\tvmcb-\u003econtrol.asid++;\n+\t\tvmcb-\u003econtrol.tlb_ctl = TLB_CONTROL_DO_NOTHING;\n+\t\tif (vmcb-\u003econtrol.asid \u003e= max_asid) {\n+\t\t\tvmcb-\u003econtrol.asid = 1;\n+\t\t\tvmcb-\u003econtrol.tlb_ctl = TLB_CONTROL_FLUSH_ALL_ASID;\n+\t\t}\n+\t\tbreak;\n+\tcase TLB_FLUSH_TAG:\n+\t\tvmcb-\u003econtrol.tlb_ctl = TLB_CONTROL_FLUSH_ASID;\n+\t\tbreak;\n+\tcase TLB_FLUSH_VADDR:\n+\t\tGUEST_ASSERT(!svm-\u003encr3_gpa);\n+\t\tvmcb-\u003econtrol.tlb_ctl = TLB_CONTROL_DO_NOTHING;\n+\t\tinvlpga(TEST_VADDR, vmcb-\u003econtrol.asid);\n+\t\tbreak;\n+\tcase TLB_FLUSH_NONE:\n+\t\tvmcb-\u003econtrol.tlb_ctl = TLB_CONTROL_DO_NOTHING;\n+\t\tbreak;\n+\t}\n+}\n+\n+static void vmx_tlb_flush(struct vmx_pages *vmx, enum guest_flush_method method)\n+{\n+\tu16 vpid = vmread(VIRTUAL_PROCESSOR_ID);\n+\n+\tswitch (method) {\n+\tcase TLB_CHANGE_TAG:\n+\t\tGUEST_ASSERT(!vmx-\u003eeptp_gpa);\n+\t\tvmwrite(VIRTUAL_PROCESSOR_ID, vpid + 1);\n+\t\tbreak;\n+\tcase TLB_FLUSH_TAG:\n+\t\tif (vmx-\u003eeptp_gpa)\n+\t\t\tinvept(1, vmread(EPT_POINTER));\n+\t\telse\n+\t\t\tinvvpid(1, vpid, 0);\n+\t\tbreak;\n+\tcase TLB_FLUSH_VADDR:\n+\t\tGUEST_ASSERT(!vmx-\u003eeptp_gpa);\n+\t\tinvvpid(0, vpid, TEST_VADDR);\n+\t\tbreak;\n+\tcase TLB_FLUSH_NONE:\n+\t\tbreak;\n+\t}\n+}\n+\n+static void prepare_l2(void *nested_state, void *l2_guest_code)\n+{\n+\tstruct vmx_pages *vmx = nested_state;\n+\n+\tif (this_cpu_has(X86_FEATURE_VMX)) {\n+\t\tu32 sec_exec_ctl;\n+\n+\t\tprepare_for_vmx_operation(vmx);\n+\t\tload_vmcs(vmx);\n+\t\tprepare_vmcs(vmx, l2_guest_code);\n+\n+\t\t/* Enable VPID */\n+\t\tsec_exec_ctl = vmread(SECONDARY_VM_EXEC_CONTROL);\n+\t\tsec_exec_ctl |= SECONDARY_EXEC_ENABLE_VPID;\n+\t\tvmwrite(SECONDARY_VM_EXEC_CONTROL, sec_exec_ctl);\n+\t\tvmwrite(VIRTUAL_PROCESSOR_ID, 1);\n+\t} else {\n+\t\tgeneric_svm_setup(nested_state, l2_guest_code);\n+\t}\n+}\n+\n+static void __run_l2(void *nested_state, bool launch,\n+\t\t enum guest_flush_method method)\n+{\n+\tstruct svm_test_data *svm;\n+\n+\tif (this_cpu_has(X86_FEATURE_VMX)) {\n+\t\tvmx_tlb_flush(nested_state, method);\n+\t\tif (launch)\n+\t\t\tvmlaunch();\n+\t\telse\n+\t\t\tvmresume();\n+\t\tGUEST_ASSERT_EQ(vmread(VM_EXIT_REASON), EXIT_REASON_VMCALL);\n+\t\tvmwrite(GUEST_RIP, vmread(GUEST_RIP) + 3); /* skip over VMCALL */\n+\t} else {\n+\t\tsvm = nested_state;\n+\t\tsvm_tlb_flush(svm, method);\n+\t\trun_guest(svm-\u003evmcb, svm-\u003evmcb_gpa);\n+\t\tGUEST_ASSERT_EQ(svm-\u003evmcb-\u003econtrol.exit_code, SVM_EXIT_VMMCALL);\n+\t\tsvm-\u003evmcb-\u003esave.rip += 3; /* skip over VMMCALL */\n+\t}\n+}\n+\n+static void run_l2(void *nested_state, bool launch)\n+{\n+\t__run_l2(nested_state, launch, flush_method);\n+}\n+\n+static void l2_guest_code(void)\n+{\n+\tu64 expected_val;\n+\n+\tfor (;;) {\n+\t\texpected_val = READ_ONCE(l2_expected_val);\n+\t\tif (expected_val)\n+\t\t\tGUEST_ASSERT_EQ(READ_ONCE(*(u64 *)TEST_VADDR), expected_val);\n+\t\tvmcall();\n+\t}\n+}\n+\n+static void l1_guest_code(void *data)\n+{\n+\tgpa_t gpa;\n+\tint i;\n+\n+\tprepare_l2(data, l2_guest_code);\n+\n+\t/*\n+\t * Alternately switch the PTE (or TDP PTE) mapping TEST_VADDR between\n+\t * two pages containing VAL1 and VAL2, flush TLBs, and verify that L2\n+\t * reads the expected value.\n+\t */\n+\tfor (i = 0; i \u003c NR_ITERATIONS; i++) {\n+\t\tgpa = test_gpa[i % 2];\n+\n+\t\t*pte_gva \u0026= ~PHYSICAL_PAGE_MASK;\n+\t\t*pte_gva |= gpa \u0026 PHYSICAL_PAGE_MASK;\n+\n+\t\tWRITE_ONCE(l2_expected_val, (i % 2 == 0) ? VAL1 : VAL2);\n+\t\trun_l2(data, i == 0);\n+\t}\n+\n+\tGUEST_DONE();\n+}\n+\n+/*\n+ * Verify that flushing L2's ASID invalidates nested TDP translations for all\n+ * TDP roots, not just the one in use at the time of the flush. L1 updates the\n+ * nested TDP PTE for TEST_VADDR in root 0, flushes L2's ASID while running L2\n+ * with root 1, then runs L2 with root 0 *without* flushing. L2 must observe\n+ * the new mapping, as the flush invalidated all translations for the ASID.\n+ *\n+ * Root 1 is a full copy of root 0, such that none of the nested TDP pages are\n+ * shared, i.e. KVM's shadow pages for root 0 are not reachable through the\n+ * shadow root for root 1.\n+ */\n+static void l1_guest_code_multi_tdp_root(void *data)\n+{\n+\tstruct svm_test_data *svm = data;\n+\tstruct vmcb *vmcb = svm-\u003evmcb;\n+\tint i;\n+\n+\tGUEST_ASSERT(this_cpu_has(X86_FEATURE_SVM));\n+\tGUEST_ASSERT_EQ(svm-\u003encr3_gpa, tdp_root_gpa[0]);\n+\n+\tprepare_l2(data, l2_guest_code);\n+\n+\tfor (i = 0; i \u003c NR_ITERATIONS; i++) {\n+\t\t*pte_gva \u0026= ~PHYSICAL_PAGE_MASK;\n+\t\t*pte_gva |= test_gpa[i % 2] \u0026 PHYSICAL_PAGE_MASK;\n+\n+\t\t/* Flush L2's ASID (per flush_method) while using root 1 */\n+\t\tvmcb-\u003econtrol.nested_cr3 = tdp_root_gpa[1];\n+\t\tWRITE_ONCE(l2_expected_val, 0);\n+\t\trun_l2(data, i == 0);\n+\n+\t\t/* Switch back to root 0 without flushing */\n+\t\tvmcb-\u003econtrol.nested_cr3 = tdp_root_gpa[0];\n+\t\tWRITE_ONCE(l2_expected_val, (i % 2 == 0) ? VAL1 : VAL2);\n+\t\t__run_l2(data, false, TLB_FLUSH_NONE);\n+\t}\n+\n+\tGUEST_DONE();\n+}\n+\n+static void prep_guest_flush_test(struct kvm_vm *vm, bool tdp_enabled,\n+\t\t\t\t bool copy_tdp_root)\n+{\n+\tgva_t page_gva[2], pte_map_gva;\n+\tgpa_t pte_gpa;\n+\tu64 *ptep;\n+\n+\t/*\n+\t * Allocate two pages in the VM and initialize them with different\n+\t * values. L1 will update the mappings to switch between these pages and\n+\t * L2 will verify that the accessed value indicates the right page.\n+\t */\n+\tpage_gva[0] = vm_alloc_page(vm);\n+\tpage_gva[1] = vm_alloc_page(vm);\n+\n+\t*(u64 *)addr_gva2hva(vm, page_gva[0]) = VAL1;\n+\t*(u64 *)addr_gva2hva(vm, page_gva[1]) = VAL2;\n+\n+\ttest_gpa[0] = addr_gva2gpa(vm, page_gva[0]);\n+\ttest_gpa[1] = addr_gva2gpa(vm, page_gva[1]);\n+\n+\tif (tdp_enabled) {\n+\t\tvirt_pg_map(vm, TEST_VADDR, TEST_NESTED_GPA);\n+\t\ttdp_map(vm, TEST_NESTED_GPA, test_gpa[0], vm-\u003epage_size);\n+\t\tptep = tdp_get_pte(vm, TEST_NESTED_GPA);\n+\t} else {\n+\t\tvirt_pg_map(vm, TEST_VADDR, test_gpa[0]);\n+\t\tptep = vm_get_pte(vm, TEST_VADDR);\n+\t}\n+\n+\t/*\n+\t * Map the PTE (or TDP PTE) for TEST_VADDR, such that L1 can read and\n+\t * update the mapping. Pass the PTE GVA to L1 to directly use it (rather\n+\t * than L1 walking the page tables.\n+\t */\n+\tpte_gpa = addr_hva2gpa(vm, ptep);\n+\tpte_map_gva = vm_unused_gva_gap(vm, vm-\u003epage_size, KVM_UTIL_MIN_VADDR);\n+\tvirt_pg_map(vm, pte_map_gva, pte_gpa \u0026 PAGE_MASK);\n+\tpte_gva = (u64 *)(pte_map_gva + (pte_gpa \u0026 ~PAGE_MASK));\n+\n+\tsync_global_to_guest(vm, pte_gva);\n+\tsync_global_to_guest(vm, test_gpa);\n+\n+\tif (!tdp_enabled)\n+\t\treturn;\n+\n+\t/*\n+\t * Must be done after all pages accessed by L2 are allocated, including\n+\t * nested state (e.g. L2's stack) and page tables for TEST_VADDR.\n+\t */\n+\ttdp_identity_map_default_memslots(vm);\n+\n+\t/* Must be done after all TDP mappings are created */\n+\tif (copy_tdp_root) {\n+\t\ttdp_root_gpa[0] = vm-\u003estage2_mmu.pgd;\n+\t\ttdp_root_gpa[1] = tdp_copy_page_tables(vm);\n+\t\tsync_global_to_guest(vm, tdp_root_gpa);\n+\t}\n+}\n+\n+static void l2_guest_code_memslot_move(void)\n+{\n+\tint i;\n+\tu64 val;\n+\n+\tfor (i = 0; i \u003c NR_ITERATIONS; i++) {\n+\t\tval = READ_ONCE(*(u64 *)TEST_VADDR);\n+\t\tGUEST_ASSERT_EQ(val, VAL1);\n+\n+\t\tvmcall();\n+\n+\t\tval = READ_ONCE(*(u64 *)TEST_VADDR);\n+\t\tGUEST_ASSERT_EQ(val, VAL2);\n+\n+\t\tGUEST_SYNC2(SYNC_MOVE_MEMSLOT, 1);\n+\n+\t\tval = READ_ONCE(*(u64 *)TEST_VADDR);\n+\t\tGUEST_ASSERT_EQ(val, VAL1);\n+\n+\t\tvmcall();\n+\t}\n+}\n+\n+static void l1_guest_code_memslot_move(void *data)\n+{\n+\tint i;\n+\tu64 val;\n+\n+\tprepare_l2(data, l2_guest_code_memslot_move);\n+\n+\tfor (i = 0; i \u003c NR_ITERATIONS; i++) {\n+\t\t/* Verify that accessing VADDR reads from page 1 in both L1 \u0026 L2 */\n+\t\tval = READ_ONCE(*(u64 *)TEST_VADDR);\n+\t\tGUEST_ASSERT_EQ(val, VAL1);\n+\n+\t\trun_l2(data, i == 0);\n+\n+\t\t/*\n+\t\t * Update VADDR to point at page 2. KVM should flush both L1 and\n+\t\t * L2's TLB translations so that further accesses go to page 2.\n+\t\t */\n+\t\tGUEST_SYNC2(SYNC_MOVE_MEMSLOT, 2);\n+\n+\t\t/*\n+\t\t * Verify that accessing VADDR reads from page 2. A stale TLB\n+\t\t * entry would lead to reading from page 1. Enter L2 and check\n+\t\t * access to page 2, to verify that a TLB flush from L1's\n+\t\t * context flushes both L1 and L2.\n+\t\t */\n+\t\tval = READ_ONCE(*(u64 *)TEST_VADDR);\n+\t\tGUEST_ASSERT_EQ(val, VAL2);\n+\n+\t\trun_l2(data, false);\n+\n+\t\t/*\n+\t\t * L2 updated VADDR to point at page 1 again. Verify access to\n+\t\t * page 1 again to verify that a TLB flush from L2's context\n+\t\t * flushes both L1 and L2 TLB translations.\n+\t\t */\n+\t\tval = READ_ONCE(*(u64 *)TEST_VADDR);\n+\t\tGUEST_ASSERT_EQ(val, VAL1);\n+\t}\n+\n+\tGUEST_DONE();\n+}\n+\n+static void prep_memslot_move_test(struct kvm_vm *vm, bool tdp_enabled)\n+{\n+\t/*\n+\t * Disable KVM_X86_QUIRK_SLOT_ZAP_ALL. By default, moving a memslot\n+\t * invalidates all MMU roots, which implicitly flushes the guest TLB\n+\t * and masks missing TLB flush bugs. Disabling it forces KVM to keep\n+\t * the same roots, relying solely on explicit TLB flushes.\n+\t */\n+\tTEST_REQUIRE(kvm_check_cap(KVM_CAP_DISABLE_QUIRKS2) \u0026 KVM_X86_QUIRK_SLOT_ZAP_ALL);\n+\tvm_enable_cap(vm, KVM_CAP_DISABLE_QUIRKS2, KVM_X86_QUIRK_SLOT_ZAP_ALL);\n+\n+\tvm_userspace_mem_region_add(vm, VM_MEM_SRC_ANONYMOUS,\n+\t\t\t\t TEST_GPA, TEST_MEMSLOT,\n+\t\t\t\t /*npages=*/2, /*flags=*/0);\n+\n+\tvirt_pg_map(vm, TEST_VADDR, TEST_GPA);\n+\n+\tif (tdp_enabled)\n+\t\ttdp_map(vm, TEST_GPA, TEST_GPA, vm-\u003epage_size);\n+\n+\t*(u64 *)addr_gpa2hva(vm, TEST_GPA) = VAL1;\n+\t*(u64 *)addr_gpa2hva(vm, TEST_GPA + vm-\u003epage_size) = VAL2;\n+\n+\t/*\n+\t * Must be done after all pages accessed by L2 are allocated, including\n+\t * nested state (e.g. L2's stack) and page tables for TEST_VADDR.\n+\t */\n+\tif (tdp_enabled)\n+\t\ttdp_identity_map_default_memslots(vm);\n+}\n+\n+static void calc_max_svm_asid(void)\n+{\n+\tconst struct kvm_cpuid_entry2 *entry;\n+\n+\tif (!kvm_cpu_has(X86_FEATURE_SVM))\n+\t\treturn;\n+\n+\tentry = get_cpuid_entry(kvm_get_supported_cpuid(), 0x8000000A, 0);\n+\tmax_asid = entry ? entry-\u003eebx : 0;\n+}\n+\n+static void run_test(void *guest_code, bool tdp_enabled)\n+{\n+\tstruct kvm_vcpu *vcpu;\n+\tgva_t nested_gva = 0;\n+\tstruct kvm_vm *vm;\n+\tstruct ucall uc;\n+\n+\tvm = vm_create_with_one_vcpu(\u0026vcpu, guest_code);\n+\tif (tdp_enabled)\n+\t\tvm_enable_tdp(vm);\n+\n+\tif (kvm_cpu_has(X86_FEATURE_VMX))\n+\t\tvcpu_alloc_vmx(vm, \u0026nested_gva);\n+\telse\n+\t\tvcpu_alloc_svm(vm, \u0026nested_gva);\n+\n+\tif (guest_code == l1_guest_code_memslot_move)\n+\t\tprep_memslot_move_test(vm, tdp_enabled);\n+\telse if (guest_code == l1_guest_code_multi_tdp_root)\n+\t\tprep_guest_flush_test(vm, tdp_enabled, true);\n+\telse\n+\t\tprep_guest_flush_test(vm, tdp_enabled, false);\n+\n+\tvcpu_args_set(vcpu, 1, nested_gva);\n+\n+\tsync_global_to_guest(vm, flush_method);\n+\tsync_global_to_guest(vm, max_asid);\n+\n+\tfor (;;) {\n+\t\tvcpu_run(vcpu);\n+\t\tTEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO);\n+\n+\t\tswitch (get_ucall(vcpu, \u0026uc)) {\n+\t\tcase UCALL_ABORT:\n+\t\t\tREPORT_GUEST_ASSERT(uc);\n+\t\t\tbreak;\n+\t\tcase UCALL_SYNC:\n+\t\t\t/*\n+\t\t\t * Move the base GPA of the memslot between TEST_GPA and\n+\t\t\t * TEST_GPA - PAGE_SIZE. This effectively moves\n+\t\t\t * TEST_VADDR to point at page 1 or page 2 in the\n+\t\t\t * memslot.\n+\t\t\t */\n+\t\t\tif (uc.args[0] == SYNC_MOVE_MEMSLOT) {\n+\t\t\t\tgpa_t new_gpa = (uc.args[1] == 1) ?\n+\t\t\t\t\tTEST_GPA : (TEST_GPA - vm-\u003epage_size);\n+\n+\t\t\t\tvm_mem_region_move(vm, TEST_MEMSLOT, new_gpa);\n+\t\t\t} else {\n+\t\t\t\tTEST_FAIL(\"Unknown sync code: %lu\", uc.args[0]);\n+\t\t\t}\n+\t\t\tbreak;\n+\t\tcase UCALL_DONE:\n+\t\t\tgoto done;\n+\t\tdefault:\n+\t\t\tTEST_FAIL(\"Unknown ucall %lu\", uc.cmd);\n+\t\t}\n+\t}\n+\n+done:\n+\tkvm_vm_free(vm);\n+}\n+\n+#define tlb_test(test_name, guest_code, tdp_setting, flush_setting)\t\t\\\n+do {\t\t\t\t\t\t\t\t\t\t\\\n+\ttdp_setting;\t\t\t\t\t\t\t\t\\\n+\tflush_setting;\t\t\t\t\t\t\t\t\\\n+\t\t\t\t\t\t\t\t\t\t\\\n+\tif (tdp_enabled \u0026\u0026 !kvm_cpu_has_tdp()) {\t\t\t\t\\\n+\t\tpr_info(\"Skipping: \" test_name \" (no TDP support)\\n\");\t\t\\\n+\t\tbreak;\t\t\t\t\t\t\t\t\\\n+\t}\t\t\t\t\t\t\t\t\t\\\n+\t\t\t\t\t\t\t\t\t\t\\\n+\t/*\t\t\t\t\t\t\t\t\t\\\n+\t * VPIDs do not tag guest-physical translations, i.e. changing or\t\\\n+\t * flushing VPIDs does not flush EPT translations, and single-context\t\\\n+\t * INVEPT only flushes the target EPTP (not all TDP roots).\t\t\\\n+\t */\t\t\t\t\t\t\t\t\t\\\n+\tif (tdp_enabled \u0026\u0026 kvm_cpu_has_ept() \u0026\u0026\t\t\t\t\t\\\n+\t (flush_method == TLB_CHANGE_TAG ||\t\t\t\t\t\\\n+\t guest_code == l1_guest_code_multi_tdp_root)) {\t\t\t\\\n+\t\tpr_info(\"Skipping: \" test_name \" (not applicable to EPTs)\\n\");\t\\\n+\t\tbreak;\t\t\t\t\t\t\t\t\\\n+\t}\t\t\t\t\t\t\t\t\t\\\n+\t\t\t\t\t\t\t\t\t\t\\\n+\tpr_info(\"Testing \" test_name \"...\\n\");\t\t\t\t\t\\\n+\trun_test(guest_code, tdp_enabled);\t\t\t\t\t\\\n+} while (0)\n+\n+int main(int argc, char *argv[])\n+{\n+\tbool tdp_enabled;\n+\n+\tTEST_REQUIRE(kvm_cpu_has(X86_FEATURE_SVM) || kvm_cpu_has(X86_FEATURE_VMX));\n+\n+\tcalc_max_svm_asid();\n+\n+\ttlb_test(\"Guest-triggered TLB flush (flush entire tag), TDP disabled\",\n+\t\t l1_guest_code, tdp_enabled = false,\n+\t\t flush_method = TLB_FLUSH_TAG);\n+\ttlb_test(\"Guest-triggered TLB flush (flush individual address), TDP disabled\",\n+\t\t l1_guest_code, tdp_enabled = false,\n+\t\t flush_method = TLB_FLUSH_VADDR);\n+\ttlb_test(\"Guest-triggered TLB flush (change tag), TDP disabled\",\n+\t\t l1_guest_code, tdp_enabled = false,\n+\t\t flush_method = TLB_CHANGE_TAG);\n+\n+\ttlb_test(\"Guest-triggered TLB flush (flush entire tag), TDP enabled\",\n+\t\t l1_guest_code, tdp_enabled = true,\n+\t\t flush_method = TLB_FLUSH_TAG);\n+\ttlb_test(\"Guest-triggered TLB flush (change tag), TDP enabled\",\n+\t\t l1_guest_code, tdp_enabled = true,\n+\t\t flush_method = TLB_CHANGE_TAG);\n+\n+\ttlb_test(\"Guest-triggered TLB flush with multiple TDP roots (flush entire tag), TDP enabled\",\n+\t\t l1_guest_code_multi_tdp_root, tdp_enabled = true,\n+\t\t flush_method = TLB_FLUSH_TAG);\n+\ttlb_test(\"Guest-triggered TLB flush with multiple TDP roots (change tag), TDP enabled\",\n+\t\t l1_guest_code_multi_tdp_root, tdp_enabled = true,\n+\t\t flush_method = TLB_CHANGE_TAG);\n+\n+\n+\ttlb_test(\"KVM-triggered TLB flush via memslot move, TDP disabled\",\n+\t\t l1_guest_code_memslot_move, tdp_enabled = false,\n+\t\t flush_method = TLB_FLUSH_NONE);\n+\ttlb_test(\"KVM-triggered TLB flush via memslot move, TDP enabled\",\n+\t\t l1_guest_code_memslot_move, tdp_enabled = true,\n+\t\t flush_method = TLB_FLUSH_NONE);\n+\n+\treturn 0;\n+}\n"
} |
| 2/1 | 2026/10/12 23:48 | agent | patch-evaluator |
3mModel:core Args: null Results: {
"EnableConfigs": [
"KVM",
"KVM_AMD",
"KVM_INTEL"
],
"FocusSymbols": [
"kvm_alloc_tlb_tag",
"kvm_free_tlb_tag",
"invlpga_interception",
"enter_svm_guest_mode",
"nested_svm_vmrun",
"nested_svm_vmexit",
"svm_allocate_nested",
"svm_free_nested",
"svm_leave_nested",
"svm_set_nested_state",
"svm_flush_tlb_all"
],
"Reasoning": "The patch series significantly reworks ASID and VPID management across KVM (x86 MMU, SVM, and VMX). It introduces a centralized TLB tag allocator in kvm/mmu/mmu.c, replaces the generation-based SVM ASID allocator with explicit tag tracking and fallback ASID handling, completely refactors nested SVM TLB flushes on guest mode transitions, reworks INVLPGA instruction interception emulation, and introduces the KVM_REQ_MMU_SYNC_ALL_ROOTS request. It also adds new invariant assertions (WARN_ON_ONCE, KVM_BUG_ON). This code is fully reachable in virtualized environments via /dev/kvm ioctls and nested virtualization execution.",
"WorthFuzzing": true
}Instruction: You are an expert Linux kernel maintainer and security engineer.
Your job is to review a provided patch series and evaluate whether it warrants fuzzing with syzkaller.
IMPORTANT: The changes have ALREADY been applied and committed as the HEAD commit in
your workspace. Do NOT rely on internal assumptions. You must actively use your code access
tools to inspect the actual source code, callers, and surrounding context.
================================================================================
1. CORE TRIAGE PHILOSOPHY
================================================================================
The goal of patch fuzzing is to discover crashes, regressions, exposed latent bugs,
and newly triggered assertions introduced by the patch series.
- REACHABILITY IS THE PRIMARY GATE:
Fuzzing can only discover bugs in code that can actually execute in standard virtualized
environments (GCE or QEMU, utilizing software-emulated devices like USB gadgets, netdev, tun/tap).
If the modified code is structurally unreachable (see Section 2), it MUST NOT be fuzzed,
regardless of whether it adds assertions or complex logic.
- DO NOT BLINDLY TRUST "NO FUNCTIONAL CHANGE" (NFCI) OR "REFACTORING" CLAIMS:
Patch authors routinely label changes as "cleanups", "refactorings", or state
"No functional change intended". Do NOT take these claims at face value.
Code refactorings that rearrange logic, introduce helper functions, or alter state management
in core subsystems frequently introduce subtle semantic shifts or uncover latent kernel bugs.
If reachable executable code is modified or refactored, it MUST be fuzzed.
- NEW OR MODIFIED ASSERTIONS IN REACHABLE CODE MUST BE FUZZED:
When a patch introduces or modifies runtime checks or assertions (e.g., WARN_ON*, VM_WARN_ON*,
BUG_ON*, lockdep_assert*) in reachable code paths, it enforces new or stricter invariants.
Even if the author believes the invariant always holds, fuzzing is essential to verify whether
an unusual sequence of operations can violate it.
================================================================================
2. WHEN TO RETURN WorthFuzzing=false (NEGATIVE CRITERIA)
================================================================================
Return WorthFuzzing=false ONLY IF all modified code falls strictly into one or more of these categories:
- Non-kernel and non-executable changes:
* Modifications to Documentation/, comments, or spelling fixes.
* User-space directories, self-tests, samples, or scripts (e.g., tools/, samples/, scripts/, usr/)
that do not affect the compiled kernel image (vmlinux) or kernel modules.
* Purely decorative logging (e.g., message strings in pr_err, printk, dev_info) or tracepoints
that do not alter control flow or data structures.
* Build system or Kconfig changes that do not alter compiled C logic.
- Structurally unreachable hardware:
* Vendor-specific PCIe switches, SmartNICs, or GPU drivers (e.g., mlxsw, pds_core, qed,
ionic, amdgpu) requiring physical ASIC/PCIe cards not emulated in standard QEMU.
- Unreachable execution paths:
* Driver teardown callbacks (.remove, .shutdown, pci_unregister_driver) executed only during
physical PCI hot-unplug or manual sysfs driver unbinding.
* Code paths exclusive to architectures other than the target architecture.
================================================================================
3. WHEN TO RETURN WorthFuzzing=true (POSITIVE CRITERIA)
================================================================================
Return WorthFuzzing=true whenever the patch touches reachable executable code, including:
- Core Subsystems:
* Any logic modifications in memory management (mm/), synchronization/locking (kernel/locking/),
BPF, scheduler, core networking, VFS, or syscall handling.
- Refactorings and Code Cleanups:
* Any restructuring of reachable data structures, helper abstractions, or algorithm flows.
- Runtime Assertions and Defensive Checks:
* Any introduction or alteration of assertions (WARN_ON*, VM_WARN_ON*, BUG_ON*, etc.) in reachable paths.
- Reachable Drivers and Protocols:
* Drivers accessible via virtual buses (virtio, USB gadget, loopback, netlink, binder, sockets, etc.).
================================================================================
4. EXTRACTING FocusSymbols (PREVENTING DILUTION)
================================================================================
When WorthFuzzing=true, you must extract specific kernel functions into FocusSymbols to guide the fuzzer:
- AVOID UBIQUITOUS LIFECYCLE HOT-PATHS:
Do NOT list generic, ubiquitous functions called by almost every program in the corpus
(including, but not limited to: general memory allocators and deallocators, page fault
and trap handlers, or core synchronization primitives; this is not an exhaustive list).
Listing ubiquitous functions causes the fuzzer to classify thousands of unrelated tests as "focused",
which severely dilutes fuzzing effort away from the actual changes.
- TARGET SPECIFIC FEATURE LOGIC AND ENTRYPOINTS:
List functions that specifically implement the logic being added or altered, or direct API entrypoints
for the subsystem feature under review.
- HANDLING STATIC INLINE FUNCTIONS IN HEADERS (.h):
Compiler-inlined static functions (such as static inlines in mm/*.h or include/linux/*.h) lack
distinct symbol addresses in vmlinux and cannot be targeted directly by symbol coverage filters.
If the changes are primarily in static inline helpers, identify non-static, feature-specific caller
functions in .c files that exercise them (avoiding ubiquitous lifecycle wrappers).
================================================================================
5. IDENTIFYING EnableConfigs
================================================================================
Identify any specific CONFIG_ options required to properly compile and reach the modified code:
- Inspect Kconfig files and #ifdef guards; do not make assumptions.
- Check "depends on" lines in Kconfig to include any non-standard parent subsystem configs needed.
- Strip any 'CONFIG_' prefix (e.g., return "NET_IPV4" instead of "CONFIG_NET_IPV4").
Prefer calling several tools at the same time to save round-trips.
Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.
Prompt: Target architecture: amd64
For your convenience, here is the diff of the changes:
commit 21e80c95a8d3388e9dc50d771315b61b19202b62
Author: syz-cluster <triage@syzkaller.com>
Date: Mon Oct 12 23:48:32 2026 +0000
syz-cluster: applied patch under review
diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index ddc61ea36ac59..60f49b3675920 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -9770,6 +9770,26 @@ the APIC ID is found in EDX for all subleaves of 0x0b and 0x1f, and in EAX
for 0x8000001e; the latter also encodes the core id and node id in bits
7:0 of EBX and ECX respectively.
+Number of ASIDs (nested SVM)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+CPUID[0x8000000A]:EBX reports the number of ASIDs supported by the processor
+on AMD CPUs. KVM's ``KVM_GET_SUPPORTED_CPUID`` hardcodes this value to 8 for
+historical reasons. This value is arbitrary and does not represent a limit
+on the supported number of ASIDs, or even a reasonable default value.
+
+KVM allows userspace to set any value (up to 0xFFFFFFFF), and emulates those
+ASIDs by multiplexing them onto hardware ASIDs. AMD CPUs have only ever
+supported 3 values: 0x40, 0x8000, and 0x1000 -- with modern CPUs advertising
+0x8000.
+
+Advertising a small number of ASIDs could lead an L1 hypervisor to perform
+unnecessary TLB flushes (e.g. older KVM would flush the entire TLB when running
+out of ASIDs). Userspace should probably set the number of ASIDs to a
+reasonably high power of 2 value to remain consistent with hardware and avoid
+any misbehavior by L1 hypervisors, ideally one of the values advertised by
+actual hardware.
+
Obsolete ioctls and capabilities
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index e2eef7057bd35..1d28bb7e92c7f 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -127,6 +127,7 @@
KVM_ARCH_REQ_FLAGS(33, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)
#define KVM_REQ_UPDATE_PROTECTED_GUEST_STATE \
KVM_ARCH_REQ_FLAGS(34, KVM_REQUEST_WAIT)
+#define KVM_REQ_MMU_SYNC_ALL_ROOTS KVM_ARCH_REQ(35)
#define INVALID_PAGE (~(hpa_t)0)
#define VALID_PAGE(x) ((x) != INVALID_PAGE)
diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c
index adafd6fa2c542..e45c01d6faeaf 100644
--- a/arch/x86/kvm/cpuid.c
+++ b/arch/x86/kvm/cpuid.c
@@ -1846,8 +1846,8 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function)
break;
}
entry->eax = 1; /* SVM revision 1 */
- entry->ebx = 8; /* Lets support 8 ASIDs in case we add proper
- ASID emulation to nested SVM */
+ entry->ebx = 8; /* Arbitrary value for historical reasons. KVM
+ practically allows and ignores all values. */
entry->ecx = 0; /* Reserved */
cpuid_entry_override(entry, CPUID_8000_000A_EDX);
break;
diff --git a/arch/x86/kvm/hyperv.h b/arch/x86/kvm/hyperv.h
index 622a6553e9ac2..b82f70e6fcea1 100644
--- a/arch/x86/kvm/hyperv.h
+++ b/arch/x86/kvm/hyperv.h
@@ -312,18 +312,34 @@ static inline struct kvm_vcpu_hv_tlb_flush_fifo *kvm_hv_get_tlb_flush_fifo(struc
return &hv_vcpu->tlb_flush_fifo[i];
}
-static inline void kvm_hv_vcpu_purge_flush_tlb(struct kvm_vcpu *vcpu)
+/* Purge pending TLB flushes in @fifo, or in all FIFOs if @fifo is NULL */
+static inline void __kvm_hv_purge_tlb_flush_fifo(struct kvm_vcpu *vcpu,
+ struct kvm_vcpu_hv_tlb_flush_fifo *fifo)
{
- struct kvm_vcpu_hv_tlb_flush_fifo *tlb_flush_fifo;
+ struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu);
+ int i;
- if (!kvm_check_request(KVM_REQ_HV_TLB_FLUSH, vcpu))
+ if (!hv_vcpu || !kvm_check_request(KVM_REQ_HV_TLB_FLUSH, vcpu))
return;
- tlb_flush_fifo = kvm_hv_get_tlb_flush_fifo(vcpu, is_guest_mode(vcpu));
- if (!tlb_flush_fifo)
- return;
+ for (i = 0; i < ARRAY_SIZE(hv_vcpu->tlb_flush_fifo); i++) {
+ if (!fifo || fifo == &hv_vcpu->tlb_flush_fifo[i])
+ kfifo_reset_out(&hv_vcpu->tlb_flush_fifo[i].entries);
+ }
+}
+
+static inline void kvm_hv_purge_tlb_flush_fifo(struct kvm_vcpu *vcpu)
+{
+ struct kvm_vcpu_hv_tlb_flush_fifo *fifo;
+
+ fifo = kvm_hv_get_tlb_flush_fifo(vcpu, is_guest_mode(vcpu));
+ if (fifo)
+ __kvm_hv_purge_tlb_flush_fifo(vcpu, fifo);
+}
- kfifo_reset_out(&tlb_flush_fifo->entries);
+static inline void kvm_hv_purge_all_tlb_flush_fifos(struct kvm_vcpu *vcpu)
+{
+ __kvm_hv_purge_tlb_flush_fifo(vcpu, NULL);
}
static inline bool guest_hv_cpuid_has_l2_tlb_flush(struct kvm_vcpu *vcpu)
@@ -396,7 +412,8 @@ static inline int kvm_hv_hypercall(struct kvm_vcpu *vcpu)
{
return HV_STATUS_ACCESS_DENIED;
}
-static inline void kvm_hv_vcpu_purge_flush_tlb(struct kvm_vcpu *vcpu) {}
+static inline void kvm_hv_purge_tlb_flush_fifo(struct kvm_vcpu *vcpu) {}
+static inline void kvm_hv_purge_all_tlb_flush_fifos(struct kvm_vcpu *vcpu) {}
static inline bool kvm_hv_synic_has_vector(struct kvm_vcpu *vcpu, int vector)
{
return false;
diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h
index 2ae7f9ed4cf86..b6ad0ff72393e 100644
--- a/arch/x86/kvm/mmu.h
+++ b/arch/x86/kvm/mmu.h
@@ -170,6 +170,13 @@ void kvm_mmu_unload(struct kvm_vcpu *vcpu);
void kvm_mmu_free_obsolete_roots(struct kvm_vcpu *vcpu);
void kvm_mmu_sync_roots(struct kvm_vcpu *vcpu);
void kvm_mmu_sync_prev_roots(struct kvm_vcpu *vcpu);
+
+static inline void kvm_mmu_sync_all_roots(struct kvm_vcpu *vcpu)
+{
+ kvm_mmu_sync_roots(vcpu);
+ kvm_mmu_sync_prev_roots(vcpu);
+}
+
void kvm_mmu_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
int bytes);
@@ -410,4 +417,12 @@ static inline bool kvm_is_gfn_alias(struct kvm *kvm, gfn_t gfn)
{
return gfn & kvm_gfn_direct_bits(kvm);
}
+
+typedef unsigned int kvm_tlb_tag_t;
+
+int kvm_init_tlb_tags(unsigned int nr_total, unsigned int nr_reserved);
+void kvm_destroy_tlb_tags(void);
+kvm_tlb_tag_t kvm_alloc_tlb_tag(void);
+void kvm_free_tlb_tag(kvm_tlb_tag_t tag);
+
#endif
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index aadd5bdc2c97a..c096c1a2b4684 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -8178,6 +8178,98 @@ void kvm_mmu_pre_destroy_vm(struct kvm *kvm)
vhost_task_stop(kvm->arch.nx_huge_page_recovery_thread);
}
+static struct {
+ spinlock_t lock;
+ unsigned long *bitmap;
+ unsigned int nr;
+ kvm_tlb_tag_t start;
+} tlb_tags;
+
+/*
+ * Initialize @nr_total TLB tags, where @nr_reserved of those tags are reserved.
+ * Reserved tags start at 0 and are contiguous. Hence, usable tags are
+ * [nr_reserved, nr_total). At least one reserved tag is required, as 0 is the
+ * host TLB tag on both VMX and AMD, and is the value returned on a failed tag
+ * allocation.
+ */
+int kvm_init_tlb_tags(unsigned int nr_total, unsigned int nr_reserved)
+{
+ /*
+ * Limit the number of TLB tags to VMX's hardcoded maximum of 0x10000
+ * to avoid wasting memory for the bitmap in the unlikely scenario the
+ * CPU supports an inordinate number of ASIDs (on AMD). If userspace
+ * wants to concurrently run tens of thousands of vCPUs, they'll likely
+ * need a solution that works for both Intel and AMD.
+ */
+ const unsigned int MAX_NR_TLB_TAGS = VMX_NR_VPIDS;
+ unsigned int nr;
+
+ if (WARN_ON_ONCE(!nr_reserved || nr_reserved > nr_total))
+ return -EINVAL;
+
+ nr = nr_total - nr_reserved;
+ if (!nr)
+ return 0;
+
+ if (nr > MAX_NR_TLB_TAGS) {
+ pr_warn_once("Number of TLB tags capped (%u instead of %u)\n",
+ MAX_NR_TLB_TAGS, nr);
+ nr = MAX_NR_TLB_TAGS;
+ }
+
+ tlb_tags.bitmap = bitmap_zalloc(nr, GFP_KERNEL);
+ if (!tlb_tags.bitmap)
+ return -ENOMEM;
+
+ tlb_tags.nr = nr;
+ tlb_tags.start = nr_reserved;
+ spin_lock_init(&tlb_tags.lock);
+ return 0;
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_init_tlb_tags);
+
+void kvm_destroy_tlb_tags(void)
+{
+ bitmap_free(tlb_tags.bitmap);
+ tlb_tags.bitmap = NULL;
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_destroy_tlb_tags);
+
+kvm_tlb_tag_t kvm_alloc_tlb_tag(void)
+{
+ unsigned int bit;
+
+ if (!tlb_tags.bitmap)
+ return 0;
+
+ guard(spinlock)(&tlb_tags.lock);
+
+ bit = find_first_zero_bit(tlb_tags.bitmap, tlb_tags.nr);
+ if (bit >= tlb_tags.nr)
+ return 0;
+
+ __set_bit(bit, tlb_tags.bitmap);
+ return tlb_tags.start + bit;
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_alloc_tlb_tag);
+
+void kvm_free_tlb_tag(kvm_tlb_tag_t tag)
+{
+ unsigned int bit;
+
+ if (!tag || WARN_ON_ONCE(tag < tlb_tags.start))
+ return;
+
+ bit = tag - tlb_tags.start;
+ if (WARN_ON_ONCE(bit >= tlb_tags.nr))
+ return;
+
+ guard(spinlock)(&tlb_tags.lock);
+
+ __clear_bit(bit, tlb_tags.bitmap);
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_free_tlb_tag);
+
#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES
static bool hugepage_test_mixed(struct kvm_memory_slot *slot, gfn_t gfn,
int level)
diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index af0ef678433d7..6fe68f2ae282d 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -556,10 +556,8 @@ void __nested_copy_vmcb_control_to_cache(struct kvm_vcpu *vcpu,
to->nested_cr3 = from->nested_cr3;
to->pause_filter_count = from->pause_filter_count;
to->pause_filter_thresh = from->pause_filter_thresh;
-
- /* Copy asid here because nested_vmcb_check_controls() will check it */
- to->asid = from->asid;
- to->clean = from->clean;
+ to->asid = from->asid;
+ to->clean = from->clean;
#ifdef CONFIG_KVM_HYPERV
/* Hyper-V extensions (Enlightened VMCB) */
@@ -688,26 +686,69 @@ static void nested_save_pending_event_to_vmcb12(struct vcpu_svm *svm,
vmcb12->control.exit_int_info = exit_int_info;
}
-static void nested_svm_transition_tlb_flush(struct kvm_vcpu *vcpu)
+static void nested_svm_entry_tlb_flush(struct kvm_vcpu *vcpu)
{
+ struct vcpu_svm *svm = to_svm(vcpu);
+ bool new_asid = false;
+
/* Handle pending Hyper-V TLB flush requests */
kvm_hv_nested_transtion_tlb_flush(vcpu, npt_enabled);
+ if (svm->nested.ctl.asid != svm->nested.last_asid) {
+ svm->nested.last_asid = svm->nested.ctl.asid;
+ new_asid = true;
+ }
+
/*
- * TODO: optimize unconditional TLB flush/MMU sync. A partial list of
- * things to fix before this can be conditional:
+ * If L1 requested a TLB flush for L2, flush L2's TLB on nested entry
+ * and sync the nested NPT MMU, as TLB_CONTROL also flushes NPT
+ * guest-physical mappings. Sync all roots, since the TLB flush is not
+ * specific to the currently loaded nCR3.
+ *
+ * Handle L1 changing L2's ASID12 similarly, as KVM only uses one ASID
+ * for L2 in hardware (per vCPU), so it must start fresh when L1 changes
+ * ASID12 to emulate different ASIDs correctly. Additionally, the MMU
+ * context is not tagged by the ASID, so the shadow NPTs cannot be
+ * reused across different L2 ASIDs.
+ *
+ * Note, last_asid is initialized as 0, so the first nested VM-Enter
+ * after setting EFER.SVME will always flush the TLB to avoid using
+ * stale entries.
*
- * - Flush TLBs for both L1 and L2 remote TLB flush
- * - Honor L1's request to flush an ASID on nested VMRUN
- * - Sync nested NPT MMU on VMRUN that flushes L2's ASID[*]
- * - Don't crush a pending TLB flush in vmcb02 on nested VMRUN
- * - Flush L1's ASID on KVM_REQ_TLB_FLUSH_GUEST
+ * If L1 requested a full TLB flush for all ASIDs (including its own),
+ * L1's own ASID is also flushed on nested VM-Exit, before running L1.
*
- * [*] Unlike nested EPT, SVM's ASID management can invalidate nested
- * NPT guest-physical mappings on VMRUN.
+ * TLB_CONTROL_FLUSH_ASID and TLB_CONTROL_FLUSH_ASID_LOCAL are handled
+ * equally for simplicity.
*/
- kvm_make_request(KVM_REQ_MMU_SYNC, vcpu);
- kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu);
+ if (new_asid || (svm->nested.ctl.tlb_ctl != TLB_CONTROL_DO_NOTHING)) {
+ if (nested_npt_enabled(svm))
+ kvm_make_request(KVM_REQ_MMU_SYNC_ALL_ROOTS, vcpu);
+ kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu);
+ }
+
+ /*
+ * If L1 and L2 share the same ASID in hardware (when using the fallback
+ * ASID for both, or for SEV guests), flush it on nested transitions.
+ */
+ if (svm->asid == svm->nested.asid02) {
+ WARN_ON_ONCE(svm->asid != fallback_asid && !is_sev_guest(vcpu));
+ kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu);
+ }
+}
+
+static void nested_svm_exit_tlb_flush(struct kvm_vcpu *vcpu)
+{
+ struct vcpu_svm *svm = to_svm(vcpu);
+
+ kvm_hv_nested_transtion_tlb_flush(vcpu, npt_enabled);
+
+ /* Flush L1's own ASID if it requested a *full* TLB flush on VMRUN */
+ if (svm->nested.ctl.tlb_ctl == TLB_CONTROL_FLUSH_ALL_ASID)
+ kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu);
+
+ if (svm->asid == svm->nested.asid02)
+ kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu);
}
/*
@@ -860,7 +901,7 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm)
struct vmcb *vmcb01 = svm->vmcb01.ptr;
struct kvm_vcpu *vcpu = &svm->vcpu;
- nested_svm_transition_tlb_flush(vcpu);
+ nested_svm_entry_tlb_flush(vcpu);
/* Enter Guest-Mode */
enter_guest_mode(vcpu);
@@ -936,10 +977,7 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm)
else
vmcb02->control.bus_lock_counter = 0;
- /* Done at vmrun: asid. */
-
- /* Also overwritten later if necessary. */
- vmcb02->control.tlb_ctl = TLB_CONTROL_DO_NOTHING;
+ vmcb02->control.asid = svm->nested.asid02;
/* Use vmcb01 MMU and format if guest does not use nNPT */
if (nested_npt_enabled(svm)) {
@@ -1038,6 +1076,20 @@ static void nested_svm_copy_common_state(struct vmcb *from_vmcb, struct vmcb *to
to_vmcb->save.spec_ctrl = from_vmcb->save.spec_ctrl;
}
+static void nested_svm_service_local_tlb_flushes(struct kvm_vcpu *vcpu)
+{
+ struct vcpu_svm *svm = to_svm(vcpu);
+
+ /*
+ * TLB flushes are applied to the VMCB, so apply any pending TLB flushes
+ * on the outgoing VMCB before switching to a new one. A TLB flush could
+ * purge the relevant HV TLB flush FIFO depending on guest_mode, so make
+ * sure the VMCB and guest_mode context is consistent.
+ */
+ WARN_ON_ONCE(is_guest_mode(vcpu) != (svm->vmcb == svm->nested.vmcb02.ptr));
+ kvm_service_local_tlb_flush_requests(vcpu);
+}
+
int enter_svm_guest_mode(struct kvm_vcpu *vcpu, u64 vmcb12_gpa, bool from_vmrun)
{
struct vcpu_svm *svm = to_svm(vcpu);
@@ -1069,6 +1121,8 @@ int enter_svm_guest_mode(struct kvm_vcpu *vcpu, u64 vmcb12_gpa, bool from_vmrun)
nested_svm_copy_common_state(svm->vmcb01.ptr, svm->nested.vmcb02.ptr);
+ nested_svm_service_local_tlb_flushes(vcpu);
+
svm_switch_vmcb(svm, &svm->nested.vmcb02);
nested_vmcb02_prepare_control(svm);
nested_vmcb02_prepare_save(svm);
@@ -1142,7 +1196,7 @@ int nested_svm_vmrun(struct kvm_vcpu *vcpu)
return ret;
}
- if (WARN_ON_ONCE(!svm->nested.initialized))
+ if (WARN_ON_ONCE(!svm->nested.vmcb02.ptr))
return -EINVAL;
vmcb12_gpa = kvm_rax_read(vcpu);
@@ -1194,6 +1248,13 @@ int nested_svm_vmrun(struct kvm_vcpu *vcpu)
svm->vmcb->control.exit_info_1 = 0;
svm->vmcb->control.exit_info_2 = 0;
+ /*
+ * Force a flush+sync on the next nested VMRUN. Otherwise,
+ * pending MMU syncs due to a changed ASID are lost, as
+ * last_asid was already updated.
+ */
+ svm->nested.last_asid = 0;
+
nested_svm_vmexit(svm);
}
@@ -1331,7 +1392,9 @@ void nested_svm_vmexit(struct vcpu_svm *svm)
if (nested_svm_vmexit_update_vmcb12(vcpu))
kvm_make_request(KVM_REQ_TRIPLE_FAULT, vcpu);
- /* Exit Guest-Mode */
+ nested_svm_service_local_tlb_flushes(vcpu);
+
+ svm_switch_vmcb(svm, &svm->vmcb01);
leave_guest_mode(vcpu);
svm_pmu_handle_nested_transition(svm);
@@ -1358,8 +1421,6 @@ void nested_svm_vmexit(struct vcpu_svm *svm)
if (guest_cpu_cap_has(vcpu, X86_FEATURE_ERAPS))
vmcb01->control.erap_ctl |= ERAP_CONTROL_CLEAR_RAP;
- svm_switch_vmcb(svm, &svm->vmcb01);
-
/*
* Rules for synchronizing int_ctl bits from vmcb02 to vmcb01:
*
@@ -1438,7 +1499,7 @@ void nested_svm_vmexit(struct vcpu_svm *svm)
svm->vcpu.arch.dr7 = DR7_FIXED_1;
kvm_update_dr7(&svm->vcpu);
- nested_svm_transition_tlb_flush(vcpu);
+ nested_svm_exit_tlb_flush(vcpu);
nested_svm_uninit_mmu_context(vcpu);
@@ -1489,21 +1550,36 @@ int svm_allocate_nested(struct vcpu_svm *svm)
{
struct page *vmcb02_page;
- if (svm->nested.initialized)
+ if (svm->nested.vmcb02.ptr)
return 0;
vmcb02_page = snp_safe_alloc_page();
if (!vmcb02_page)
return -ENOMEM;
- svm->nested.vmcb02.ptr = page_address(vmcb02_page);
- svm->nested.vmcb02.pa = __sme_set(page_to_pfn(vmcb02_page) << PAGE_SHIFT);
svm->nested.msrpm = svm_vcpu_alloc_msrpm();
if (!svm->nested.msrpm)
goto err_free_vmcb02;
svm->nested.vmcb02.cpu = -1;
- svm->nested.initialized = true;
+ svm->nested.vmcb02.ptr = page_address(vmcb02_page);
+ svm->nested.vmcb02.pa = __sme_set(page_to_pfn(vmcb02_page) << PAGE_SHIFT);
+
+ svm->nested.asid02 = allocate_asid(&svm->vcpu);
+
+ /*
+ * KVM uses a fallback ASID when out of ASIDs, and fails vCPU creation
+ * if there's no fallback ASID. ASID allocation must succeed here.
+ */
+ KVM_BUG_ON(!svm->nested.asid02, svm->vcpu.kvm);
+
+ /*
+ * Clear last_asid to ensure that the ASID is flushed on the first
+ * nested VM-Enter. Otherwise, stale TLB entries from a previous life of
+ * the ASID (e.g. different vCPU or even different VM) could be used.
+ */
+ svm->nested.last_asid = 0;
+
return 0;
err_free_vmcb02:
@@ -1513,12 +1589,14 @@ int svm_allocate_nested(struct vcpu_svm *svm)
void svm_free_nested(struct vcpu_svm *svm)
{
- if (!svm->nested.initialized)
+ if (!svm->nested.vmcb02.ptr)
return;
if (WARN_ON_ONCE(svm->vmcb != svm->vmcb01.ptr))
svm_switch_vmcb(svm, &svm->vmcb01);
+ free_asid(svm->nested.asid02);
+
svm_vcpu_free_msrpm(svm->nested.msrpm);
svm->nested.msrpm = NULL;
@@ -1533,8 +1611,6 @@ void svm_free_nested(struct vcpu_svm *svm)
* When the vmcb02 is freed, this optimization becomes invalid.
*/
svm->nested.last_vmcb12_gpa = INVALID_GPA;
-
- svm->nested.initialized = false;
}
void svm_leave_nested(struct kvm_vcpu *vcpu)
@@ -1545,6 +1621,9 @@ void svm_leave_nested(struct kvm_vcpu *vcpu)
vcpu->arch.nested_run_pending = 0;
svm->nested.vmcb12_gpa = INVALID_GPA;
+ nested_svm_service_local_tlb_flushes(vcpu);
+
+ svm_switch_vmcb(svm, &svm->vmcb01);
leave_guest_mode(vcpu);
/*
@@ -1556,7 +1635,7 @@ void svm_leave_nested(struct kvm_vcpu *vcpu)
*/
__svm_pmu_handle_nested_transition(svm, true);
- svm_switch_vmcb(svm, &svm->vmcb01);
+ nested_svm_exit_tlb_flush(vcpu);
nested_svm_uninit_mmu_context(vcpu);
vmcb_mark_all_dirty(svm->vmcb);
@@ -1853,7 +1932,7 @@ void nested_svm_update_tsc_ratio_msr(struct kvm_vcpu *vcpu)
svm_write_tsc_multiplier(vcpu);
}
-/* Inverse operation of nested_copy_vmcb_control_to_cache(). asid is copied too. */
+/* Inverse operation of nested_copy_vmcb_control_to_cache() */
static void nested_copy_vmcb_cache_to_control(struct vmcb_control_area *dst,
struct vmcb_ctrl_area_cached *from)
{
@@ -2084,6 +2163,8 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
svm_copy_vmrun_state(vcpu, &svm->vmcb01.ptr->save, save);
nested_copy_vmcb_control_to_cache(svm, ctl);
+ nested_svm_service_local_tlb_flushes(vcpu);
+
svm_switch_vmcb(svm, &svm->nested.vmcb02);
if (use_separate_l2_pat)
diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index fad72cab76cf4..646108e70e21a 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -315,11 +315,6 @@ static int sev_asid_new(struct kvm_sev_info *sev, unsigned long vm_type)
return ret;
}
-static unsigned int sev_get_asid(struct kvm *kvm)
-{
- return to_kvm_sev_info(kvm)->asid;
-}
-
static void sev_asid_free(struct kvm_sev_info *sev)
{
struct svm_cpu_data *sd;
@@ -2037,6 +2032,28 @@ static void sev_unlock_two_vms(struct kvm *dst_kvm, struct kvm *src_kvm)
atomic_set_release(&src_sev->migration_in_progress, 0);
}
+static void sev_vcpu_migrate_asid(struct vcpu_svm *dst_svm,
+ unsigned int asid)
+{
+ /*
+ * Free the (potentially non-SEV) ASID on the destination vCPU before
+ * setting the new SEV ASID.
+ */
+ free_asid(dst_svm->asid);
+ dst_svm->asid = asid;
+
+ /*
+ * If nested is already initialized on the destination vCPU, also free
+ * the previous ASID (in case it was non-SEV) and update the nested ASID
+ * to match the new SEV ASID.
+ */
+ if (dst_svm->nested.vmcb02.ptr) {
+ free_asid(dst_svm->nested.asid02);
+ dst_svm->nested.asid02 = asid;
+ dst_svm->nested.vmcb02.ptr->control.asid = asid;
+ }
+}
+
static void sev_migrate_from(struct kvm *dst_kvm, struct kvm *src_kvm)
{
struct kvm_sev_info *dst = to_kvm_sev_info(dst_kvm);
@@ -2101,6 +2118,7 @@ static void sev_migrate_from(struct kvm *dst_kvm, struct kvm *src_kvm)
kvm_for_each_vcpu(i, dst_vcpu, dst_kvm) {
dst_svm = to_svm(dst_vcpu);
+ sev_vcpu_migrate_asid(dst_svm, dst->asid);
sev_init_vmcb(dst_svm, false);
if (!dst->es_active)
@@ -3623,9 +3641,6 @@ int pre_sev_run(struct vcpu_svm *svm, int cpu)
if (!cpumask_test_cpu(cpu, to_kvm_sev_info(kvm)->have_run_cpus))
cpumask_set_cpu(cpu, to_kvm_sev_info(kvm)->have_run_cpus);
- /* Assign the asid allocated with this SEV guest */
- svm->asid = asid;
-
/*
* Flush guest TLB:
*
@@ -3637,7 +3652,7 @@ int pre_sev_run(struct vcpu_svm *svm, int cpu)
return 0;
sd->sev_vmcbs[asid] = svm->vmcb;
- svm->vmcb->control.tlb_ctl = TLB_CONTROL_FLUSH_ASID;
+ vmcb_set_flush_asid(svm->vmcb);
return 0;
}
@@ -4889,6 +4904,10 @@ void sev_init_vmcb(struct vcpu_svm *svm, bool init_event)
svm->vmcb->control.misc_ctl |= SVM_MISC_ENABLE_SEV;
clr_exception_intercept(svm, UD_VECTOR);
+ WARN_ON_ONCE(svm->asid != sev_get_asid(vcpu->kvm));
+ svm->vmcb->control.asid = svm->asid;
+ vmcb_mark_dirty(svm->vmcb, VMCB_ASID);
+
/*
* Don't intercept #GP for SEV guests, e.g. for the VMware backdoor, as
* KVM can't decrypt guest memory to decode the faulting instruction.
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 136069dfcd44f..436d13f106369 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -192,6 +192,8 @@ DEFINE_PER_CPU(struct svm_cpu_data, svm_data);
static DEFINE_MUTEX(vmcb_dump_mutex);
+kvm_tlb_tag_t fallback_asid;
+
/*
* Only MSR_TSC_AUX is switched via the user return hook. EFER is switched via
* the VMCB, and the SYSCALL/SYSENTER MSRs are handled by VMLOAD/VMSAVE.
@@ -583,15 +585,6 @@ static int svm_enable_virtualization_cpu(void)
return r;
sd = per_cpu_ptr(&svm_data, me);
- /*
- * Bump the current asid_generation value to ensure any vCPU that
- * previously ran on this CPU sees a stale generation and is forced
- * to acquire a new ASID, preventing a latent ASID collision.
- */
- sd->asid_generation++;
- sd->max_asid = cpuid_ebx(SVM_CPUID_FUNC) - 1;
- sd->next_asid = sd->max_asid + 1;
- sd->min_asid = max_sev_asid + 1;
wrmsrq(MSR_VM_HSAVE_PA, sd->save_area_pa);
@@ -994,6 +987,8 @@ static void svm_hardware_unsetup(void)
__free_pages(__sme_pa_to_page(iopm_base), get_order(IOPM_SIZE));
iopm_base = 0;
+
+ kvm_destroy_tlb_tags();
}
static void init_seg(struct vmcb_seg *seg)
@@ -1246,8 +1241,8 @@ static void init_vmcb(struct kvm_vcpu *vcpu, bool init_event)
if (gmet_enabled)
control->misc_ctl |= SVM_MISC_ENABLE_GMET;
- svm->current_vmcb->asid_generation = 0;
- svm->asid = 0;
+ control->asid = svm->asid;
+ vmcb_set_flush_asid(vmcb);
svm->nested.vmcb12_gpa = INVALID_GPA;
svm->nested.last_vmcb12_gpa = INVALID_GPA;
@@ -1375,6 +1370,12 @@ static int svm_vcpu_create(struct kvm_vcpu *vcpu)
goto error_free_avic;
}
+ svm->asid = allocate_asid(vcpu);
+ if (!svm->asid) {
+ err = -EINVAL;
+ goto error_free_msrpm;
+ }
+
svm->x2avic_msrs_intercepted = true;
svm->lbr_msrs_intercepted = true;
@@ -1386,6 +1387,8 @@ static int svm_vcpu_create(struct kvm_vcpu *vcpu)
return 0;
+error_free_msrpm:
+ svm_vcpu_free_msrpm(svm->msrpm);
error_free_avic:
avic_vcpu_free(vcpu);
error_free_sev:
@@ -1402,6 +1405,8 @@ static int svm_vcpu_create(struct kvm_vcpu *vcpu)
static void svm_vcpu_free(struct kvm_vcpu *vcpu)
{
struct vcpu_svm *svm = to_svm(vcpu);
+ struct svm_cpu_data *sd;
+ int cpu;
WARN_ON_ONCE(!list_empty(&svm->ir_list));
@@ -1417,6 +1422,13 @@ static void svm_vcpu_free(struct kvm_vcpu *vcpu)
__free_page(__sme_pa_to_page(svm->vmcb01.pa));
svm_vcpu_free_msrpm(svm->msrpm);
+
+ free_asid(svm->asid);
+
+ for_each_possible_cpu(cpu) {
+ sd = per_cpu_ptr(&svm_data, cpu);
+ cmpxchg(&sd->fallback_asid_vcpu, vcpu, NULL);
+ }
}
#ifdef CONFIG_CPU_MITIGATIONS
@@ -1942,18 +1954,6 @@ static void svm_update_exception_bitmap(struct kvm_vcpu *vcpu)
}
}
-static void new_asid(struct vcpu_svm *svm, struct svm_cpu_data *sd)
-{
- if (sd->next_asid > sd->max_asid) {
- ++sd->asid_generation;
- sd->next_asid = sd->min_asid;
- sd->flush_all_asids = true;
- }
-
- svm->current_vmcb->asid_generation = sd->asid_generation;
- svm->asid = sd->next_asid++;
-}
-
static void svm_set_dr6(struct kvm_vcpu *vcpu, unsigned long value)
{
struct vmcb *vmcb = to_svm(vcpu)->vmcb;
@@ -2468,17 +2468,61 @@ static int clgi_interception(struct kvm_vcpu *vcpu)
static int invlpga_interception(struct kvm_vcpu *vcpu)
{
+ struct vcpu_svm *svm = to_svm(vcpu);
/* FIXME: Handle an address size prefix. */
gva_t gva = kvm_rax_read(vcpu);
u32 asid = kvm_ecx_read(vcpu);
+ int cpu;
if (nested_svm_check_permissions(vcpu))
return 1;
trace_kvm_invlpga(to_svm(vcpu)->vmcb->save.rip, asid, gva);
- /* Let's treat INVLPGA the same as INVLPG (can be optimized!) */
- kvm_mmu_invlpg(vcpu, gva);
+ /*
+ * INVLPG on a non-canonical address is a NOP according to the SDM,
+ * assume the same behavior from INVLPGA since the APM doesn't specify.
+ */
+ if (is_noncanonical_invlpg_address(gva, vcpu))
+ return kvm_skip_emulated_instruction(vcpu);
+
+ /*
+ * If L1 is flushing its own ASID, handle it like INVLPG. Do the same if
+ * this is an unintercepted INVLPGA from L2, as
+ * kvm_mmu_free_guest_mode_roots() does not handle being in guest mode.
+ */
+ if (!asid || is_guest_mode(vcpu)) {
+ kvm_mmu_invlpg(vcpu, gva);
+ return kvm_skip_emulated_instruction(vcpu);
+ }
+
+ /*
+ * Do nothing if L1 is flushing a different L2 ASID than the one KVM is
+ * currently tracking. KVM tracks a single L2 ASID, and performs a TLB
+ * flush (and MMU resync if needed) when L1 switches ASIDs anyway.
+ */
+ if (asid != svm->nested.last_asid)
+ return kvm_skip_emulated_instruction(vcpu);
+
+ /*
+ * Use INVLPGA to flush L2's ASID in hardware if running on the same
+ * CPU, otherwise fallback to a full ASID flush.
+ */
+ cpu = get_cpu();
+ if (cpu == svm->nested.vmcb02.cpu)
+ invlpga(gva, svm->nested.asid02);
+ else
+ vmcb_set_flush_asid(svm->nested.vmcb02.ptr);
+ put_cpu();
+
+ /*
+ * If NPT is disabled, free L2's shadow roots as they may have stale
+ * translations for the flushed address. Do not sync the mappings for
+ * the flushed address, as it involves walking L2 shadow page tables
+ * with L1's MMU context.
+ */
+ if (!npt_enabled)
+ kvm_mmu_free_guest_mode_roots(vcpu->kvm, &vcpu->arch.root_mmu);
return kvm_skip_emulated_instruction(vcpu);
}
@@ -3875,37 +3919,28 @@ static void svm_set_nested_run_soft_int_state(struct kvm_vcpu *vcpu)
static int pre_svm_run(struct kvm_vcpu *vcpu)
{
- struct svm_cpu_data *sd = per_cpu_ptr(&svm_data, vcpu->cpu);
+ struct svm_cpu_data *sd = this_cpu_ptr(&svm_data);
struct vcpu_svm *svm = to_svm(vcpu);
/*
* If the previous VMRUN of the VMCB occurred on a different physical
- * cpu, then mark the VMCB dirty as hardware's clean bits are per pCPU.
- *
- * Reset the ASID generation in both VMCBs. This will lead to assigning
- * a new ASID now, and then again when switching to the other VMCB.
- * However, this is needed as the ASID is shared between the VMCBs, and
- * otherwise it would be possible to use an ASID allocated on one pCPU
- * on another, for example:
- * - vCPU migrates from pCPU A to pCPU B, allocates a new ASID.
- * - vCPU migrates back to pCPU A, and then switches the VMCB.
- * - The new VMCB does not detect a pCPU change and runs on pCPU A with
- * the new ASID allocated on pCPU B, which is potentially used by
- * another vCPU/VM.
+ * cpu, then mark the VMCB dirty and flush the ASID, as hardware's clean
+ * bits and TLB entries are per pCPU.
*/
if (unlikely(svm->current_vmcb->cpu != vcpu->cpu)) {
vmcb_mark_all_dirty(svm->vmcb);
svm->current_vmcb->cpu = vcpu->cpu;
- svm->vmcb01.asid_generation = 0;
- svm->nested.vmcb02.asid_generation = 0;
+ vmcb_set_flush_asid(svm->vmcb);
}
if (is_sev_guest(vcpu))
return pre_sev_run(svm, vcpu->cpu);
- /* FIXME: handle wraparound of asid_generation */
- if (svm->current_vmcb->asid_generation != sd->asid_generation)
- new_asid(svm, sd);
+ if (unlikely(svm->vmcb->control.asid == fallback_asid &&
+ READ_ONCE(sd->fallback_asid_vcpu) != vcpu)) {
+ vmcb_set_flush_asid(svm->vmcb);
+ WRITE_ONCE(sd->fallback_asid_vcpu, vcpu);
+ }
return 0;
}
@@ -4317,7 +4352,7 @@ static void svm_flush_tlb_asid(struct kvm_vcpu *vcpu)
* A TLB flush for the current ASID flushes both "host" and "guest" TLB
* entries, and thus is a superset of Hyper-V's fine grained flushing.
*/
- kvm_hv_vcpu_purge_flush_tlb(vcpu);
+ kvm_hv_purge_tlb_flush_fifo(vcpu);
/*
* Flush only the current ASID even if the TLB flush was invoked via
@@ -4326,10 +4361,7 @@ static void svm_flush_tlb_asid(struct kvm_vcpu *vcpu)
* unconditionally does a TLB flush on both nested VM-Enter and nested
* VM-Exit (via kvm_mmu_reset_context()).
*/
- if (cpu_feature_enabled(X86_FEATURE_FLUSHBYASID))
- svm->vmcb->control.tlb_ctl = TLB_CONTROL_FLUSH_ASID;
- else
- svm->current_vmcb->asid_generation--;
+ vmcb_set_flush_asid(svm->vmcb);
}
static void svm_flush_tlb_current(struct kvm_vcpu *vcpu)
@@ -4350,6 +4382,8 @@ static void svm_flush_tlb_current(struct kvm_vcpu *vcpu)
static void svm_flush_tlb_all(struct kvm_vcpu *vcpu)
{
+ struct vcpu_svm *svm = to_svm(vcpu);
+
/*
* When running on Hyper-V with EnlightenedNptTlb enabled, remote TLB
* flushes should be routed to hv_flush_remote_tlbs() without requesting
@@ -4360,7 +4394,11 @@ static void svm_flush_tlb_all(struct kvm_vcpu *vcpu)
if (WARN_ON_ONCE(svm_hv_is_enlightened_tlb_enabled(vcpu)))
hv_flush_remote_tlbs(vcpu->kvm);
- svm_flush_tlb_asid(vcpu);
+ kvm_hv_purge_all_tlb_flush_fifos(vcpu);
+
+ vmcb_set_flush_asid(svm->vmcb01.ptr);
+ if (svm->nested.vmcb02.ptr)
+ vmcb_set_flush_asid(svm->nested.vmcb02.ptr);
}
static void svm_flush_tlb_guest(struct kvm_vcpu *vcpu)
@@ -4647,13 +4685,6 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)
sync_lapic_to_cr8(vcpu);
- if (unlikely(svm->asid != svm->vmcb->control.asid)) {
- svm->vmcb->control.asid = svm->asid;
- vmcb_mark_dirty(svm->vmcb, VMCB_ASID);
- }
- if (this_cpu_ptr(&svm_data)->flush_all_asids)
- svm->vmcb->control.tlb_ctl = TLB_CONTROL_FLUSH_ALL_ASID;
-
svm->vmcb->save.cr2 = vcpu->arch.cr2;
if (guest_cpu_cap_has(vcpu, X86_FEATURE_ERAPS) &&
@@ -4745,8 +4776,7 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)
}
if (!svm_is_vmrun_failure(svm->vmcb->control.exit_code)) {
- this_cpu_ptr(&svm_data)->flush_all_asids = false;
- svm->vmcb->control.tlb_ctl = TLB_CONTROL_DO_NOTHING;
+ vmcb_clr_flush_asid(svm->vmcb);
/*
* Unconditionally mask off the CLEAR_RAP bit, the AND is just
@@ -5659,9 +5689,8 @@ static __init void svm_set_cpu_caps(void)
kvm_cpu_cap_set(X86_FEATURE_VMCBCLEAN);
/*
- * KVM currently flushes TLBs on *every* nested SVM transition,
- * and so for all intents and purposes KVM supports flushing by
- * ASID, i.e. KVM is guaranteed to honor every L1 ASID flush.
+ * KVM handles all TLB_CONTROL values set by L1, even if the
+ * underlying CPU does not.
*/
kvm_cpu_cap_set(X86_FEATURE_FLUSHBYASID);
@@ -5757,6 +5786,7 @@ static __init void svm_set_cpu_caps(void)
static __init int svm_hardware_setup(void)
{
+ unsigned long nr_asids;
void *iopm_va;
int cpu, r;
@@ -5914,6 +5944,14 @@ static __init int svm_hardware_setup(void)
kvm_caps.inapplicable_quirks &= ~KVM_X86_QUIRK_CD_NW_CLEARED;
+ /* Consumes max_sev_asid initialized by sev_hardware_setup() */
+ nr_asids = cpuid_ebx(SVM_CPUID_FUNC);
+ r = kvm_init_tlb_tags(nr_asids, max_sev_asid + 1);
+ if (r)
+ goto err;
+
+ fallback_asid = kvm_alloc_tlb_tag();
+
for_each_possible_cpu(cpu) {
r = svm_cpu_init(cpu);
if (r)
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index 6d0242966dc87..15c4aa3b02c2d 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -26,6 +26,7 @@
#include "regs.h"
#include "x86.h"
#include "pmu.h"
+#include "mmu.h"
/*
* Helpers to convert to/from physical addresses for pages whose address is
@@ -145,7 +146,6 @@ struct kvm_vmcb_info {
struct vmcb *ptr;
unsigned long pa;
int cpu;
- uint64_t asid_generation;
};
struct vmcb_save_area_cached {
@@ -234,7 +234,7 @@ struct svm_nested_state {
*/
struct vmcb_save_area_cached save;
- bool initialized;
+ kvm_tlb_tag_t asid02;
/*
* Indicates whether MSR bitmap for L2 needs to be rebuilt due to
@@ -244,6 +244,8 @@ struct svm_nested_state {
* on its side.
*/
bool force_msr_bitmap_recalc;
+
+ u32 last_asid;
};
struct vcpu_sev_es_state {
@@ -285,7 +287,7 @@ struct vcpu_svm {
struct vmcb *vmcb;
struct kvm_vmcb_info vmcb01;
struct kvm_vmcb_info *current_vmcb;
- u32 asid;
+ kvm_tlb_tag_t asid;
u32 sysenter_esp_hi;
u32 sysenter_eip_hi;
uint64_t tsc_aux;
@@ -372,17 +374,14 @@ struct vcpu_svm {
};
struct svm_cpu_data {
- u64 asid_generation;
- u32 max_asid;
- u32 next_asid;
- u32 min_asid;
-
- bool flush_all_asids;
bool bp_spec_reduce_set;
struct vmcb *save_area;
unsigned long save_area_pa;
+ /* Last vCPU to use fallback_asid on this CPU */
+ struct kvm_vcpu *fallback_asid_vcpu;
+
/* index = sev_asid, value = vmcb pointer */
struct vmcb **sev_vmcbs;
};
@@ -432,6 +431,11 @@ static __always_inline bool is_sev_snp_guest(struct kvm_vcpu *vcpu)
{
return ____sev_snp_guest(vcpu->kvm);
}
+
+static __always_inline unsigned int sev_get_asid(struct kvm *kvm)
+{
+ return to_kvm_sev_info(kvm)->asid;
+}
#else
static __always_inline bool is_sev_guest(struct kvm_vcpu *vcpu)
{
@@ -446,6 +450,11 @@ static __always_inline bool is_sev_snp_guest(struct kvm_vcpu *vcpu)
{
return false;
}
+
+static __always_inline unsigned int sev_get_asid(struct kvm *kvm)
+{
+ return 0;
+}
#endif
static inline bool ghcb_gpa_is_registered(struct vcpu_svm *svm, u64 val)
@@ -480,6 +489,19 @@ static inline void vmcb_set_gpat(struct vmcb *vmcb, u64 data)
vmcb_mark_dirty(vmcb, VMCB_NPT);
}
+static inline void vmcb_set_flush_asid(struct vmcb *vmcb)
+{
+ if (cpu_feature_enabled(X86_FEATURE_FLUSHBYASID))
+ vmcb->control.tlb_ctl = TLB_CONTROL_FLUSH_ASID;
+ else
+ vmcb->control.tlb_ctl = TLB_CONTROL_FLUSH_ALL_ASID;
+}
+
+static inline void vmcb_clr_flush_asid(struct vmcb *vmcb)
+{
+ vmcb->control.tlb_ctl = TLB_CONTROL_DO_NOTHING;
+}
+
static __always_inline struct vcpu_svm *to_svm(struct kvm_vcpu *vcpu)
{
return container_of(vcpu, struct vcpu_svm, vcpu);
@@ -1057,6 +1079,21 @@ static inline struct vmcb_save_area *sev_decrypt_vmsa(struct kvm_vcpu *vcpu)
static inline void sev_free_decrypted_vmsa(struct kvm_vcpu *vcpu, struct vmcb_save_area *vmsa) {}
#endif
+extern kvm_tlb_tag_t fallback_asid;
+
+static inline kvm_tlb_tag_t allocate_asid(struct kvm_vcpu *vcpu)
+{
+ if (is_sev_guest(vcpu))
+ return sev_get_asid(vcpu->kvm);
+ return kvm_alloc_tlb_tag() ?: fallback_asid;
+}
+
+static inline void free_asid(kvm_tlb_tag_t asid)
+{
+ if (asid > max_sev_asid && asid != fallback_asid)
+ kvm_free_tlb_tag(asid);
+}
+
/* vmenter.S */
void __svm_sev_es_vcpu_run(struct vcpu_svm *svm, unsigned int flags,
diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index b341af4c6057a..a1caa2bd04d7a 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -598,9 +598,6 @@ DEFINE_PER_CPU(struct vmcs *, current_vmcs);
*/
static DEFINE_PER_CPU(struct list_head, loaded_vmcss_on_cpu);
-static DECLARE_BITMAP(vmx_vpid_bitmap, VMX_NR_VPIDS);
-static DEFINE_SPINLOCK(vmx_vpid_lock);
-
struct vmcs_config vmcs_config __ro_after_init;
struct vmx_capability vmx_capability __ro_after_init;
@@ -4092,31 +4089,6 @@ static void seg_setup(int seg)
vmcs_write32(sf->ar_bytes, ar);
}
-int allocate_vpid(void)
-{
- int vpid;
-
- if (!enable_vpid)
- return 0;
- spin_lock(&vmx_vpid_lock);
- vpid = find_first_zero_bit(vmx_vpid_bitmap, VMX_NR_VPIDS);
- if (vpid < VMX_NR_VPIDS)
- __set_bit(vpid, vmx_vpid_bitmap);
- else
- vpid = 0;
- spin_unlock(&vmx_vpid_lock);
- return vpid;
-}
-
-void free_vpid(int vpid)
-{
- if (!enable_vpid || vpid == 0)
- return;
- spin_lock(&vmx_vpid_lock);
- __clear_bit(vpid, vmx_vpid_bitmap);
- spin_unlock(&vmx_vpid_lock);
-}
-
static void vmx_msr_bitmap_l01_changed(struct vcpu_vmx *vmx)
{
/*
@@ -8415,6 +8387,8 @@ void vmx_hardware_unsetup(void)
if (nested)
nested_vmx_hardware_unsetup();
+
+ destroy_vpids();
}
void vmx_vm_destroy(struct kvm *kvm)
@@ -8640,8 +8614,6 @@ __init int vmx_hardware_setup(void)
kvm_caps.has_bus_lock_exit = cpu_has_vmx_bus_lock_detection();
kvm_caps.has_notify_vmexit = cpu_has_notify_vmexit();
- set_bit(0, vmx_vpid_bitmap); /* 0 is reserved for host */
-
if (enable_ept)
kvm_mmu_set_ept_masks(enable_ept_ad_bits);
else
@@ -8681,6 +8653,10 @@ __init int vmx_hardware_setup(void)
vmx_set_cpu_caps();
+ r = init_vpids();
+ if (r)
+ return r;
+
/*
* Configure nested capabilities after core CPU capabilities so that
* nested support can be conditional on base support, e.g. so that KVM
@@ -8688,8 +8664,10 @@ __init int vmx_hardware_setup(void)
*/
if (nested) {
r = nested_vmx_hardware_setup(kvm_vmx_exit_handlers);
- if (r)
+ if (r) {
+ destroy_vpids();
return r;
+ }
}
vmx_nested_ops.enabled = nested;
diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h
index 6c9dd5b2365d5..3e1ecd9148e9c 100644
--- a/arch/x86/kvm/vmx/vmx.h
+++ b/arch/x86/kvm/vmx/vmx.h
@@ -182,7 +182,7 @@ struct nested_vmx {
u64 pre_vmenter_ssp;
u64 pre_vmenter_ssp_tbl;
- u16 vpid02;
+ kvm_tlb_tag_t vpid02;
u16 last_vpid;
int tsc_autostore_slot;
@@ -256,7 +256,7 @@ struct vcpu_vmx {
u32 ar;
} seg[8];
} segment_cache;
- int vpid;
+ kvm_tlb_tag_t vpid;
/* Support for a guest hypervisor (nested VMX) */
struct nested_vmx nested;
@@ -297,9 +297,29 @@ struct kvm_vmx {
u64 *pid_table;
};
+static __always_inline int init_vpids(void)
+{
+ return enable_vpid ? kvm_init_tlb_tags(VMX_NR_VPIDS, 1) : 0;
+}
+
+static __always_inline void destroy_vpids(void)
+{
+ if (enable_vpid)
+ kvm_destroy_tlb_tags();
+}
+
+static __always_inline kvm_tlb_tag_t allocate_vpid(void)
+{
+ return enable_vpid ? kvm_alloc_tlb_tag() : 0;
+}
+
+static __always_inline void free_vpid(kvm_tlb_tag_t vpid)
+{
+ if (enable_vpid)
+ kvm_free_tlb_tag(vpid);
+}
+
void vmx_vcpu_load_vmcs(struct kvm_vcpu *vcpu, int cpu);
-int allocate_vpid(void);
-void free_vpid(int vpid);
void vmx_set_constant_host_state(struct vcpu_vmx *vmx);
void vmx_prepare_switch_to_guest(struct kvm_vcpu *vcpu);
void vmx_set_host_fs_gs(struct vmcs_host_state *host, u16 fs_sel, u16 gs_sel,
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 32563a91fba99..a54e9da2ed09e 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -2020,8 +2020,7 @@ static void kvm_vcpu_flush_tlb_guest(struct kvm_vcpu *vcpu)
* a forced sync of the shadow page tables. Ensure all the
* roots are synced and the guest TLB in hardware is clean.
*/
- kvm_mmu_sync_roots(vcpu);
- kvm_mmu_sync_prev_roots(vcpu);
+ kvm_mmu_sync_all_roots(vcpu);
}
kvm_x86_call(flush_tlb_guest)(vcpu);
@@ -2030,7 +2029,7 @@ static void kvm_vcpu_flush_tlb_guest(struct kvm_vcpu *vcpu)
* Flushing all "guest" TLB is always a superset of Hyper-V's fine
* grained flushing.
*/
- kvm_hv_vcpu_purge_flush_tlb(vcpu);
+ kvm_hv_purge_tlb_flush_fifo(vcpu);
}
@@ -8167,8 +8166,14 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
if (unlikely(r))
goto out;
}
- if (kvm_check_request(KVM_REQ_MMU_SYNC, vcpu))
+
+ if (kvm_check_request(KVM_REQ_MMU_SYNC_ALL_ROOTS, vcpu)) {
+ kvm_mmu_sync_all_roots(vcpu);
+ kvm_clear_request(KVM_REQ_MMU_SYNC, vcpu);
+ } else if (kvm_check_request(KVM_REQ_MMU_SYNC, vcpu)) {
kvm_mmu_sync_roots(vcpu);
+ }
+
if (kvm_check_request(KVM_REQ_LOAD_MMU_PGD, vcpu))
kvm_mmu_load_pgd(vcpu);
diff --git a/tools/testing/selftests/kvm/Makefile.kvm b/tools/testing/selftests/kvm/Makefile.kvm
index be5eb2dd85aa2..5347c56786b08 100644
--- a/tools/testing/selftests/kvm/Makefile.kvm
+++ b/tools/testing/selftests/kvm/Makefile.kvm
@@ -122,6 +122,7 @@ TEST_GEN_PROGS_x86 += x86/svm_nested_shutdown_test
TEST_GEN_PROGS_x86 += x86/svm_nested_soft_inject_test
TEST_GEN_PROGS_x86 += x86/svm_nested_vmcb12_gpa
TEST_GEN_PROGS_x86 += x86/svm_nested_pat_test
+TEST_GEN_PROGS_x86 += x86/nested_tlb_flush_test
TEST_GEN_PROGS_x86 += x86/svm_lbr_nested_state
TEST_GEN_PROGS_x86 += x86/svm_pmu_host_guest_test
TEST_GEN_PROGS_x86 += x86/tsc_scaling_sync
diff --git a/tools/testing/selftests/kvm/include/x86/processor.h b/tools/testing/selftests/kvm/include/x86/processor.h
index ffec2291b1e1f..62d255aee5a34 100644
--- a/tools/testing/selftests/kvm/include/x86/processor.h
+++ b/tools/testing/selftests/kvm/include/x86/processor.h
@@ -1617,6 +1617,7 @@ void tdp_map(struct kvm_vm *vm, gpa_t l2_gpa, gpa_t gpa, u64 size);
void tdp_identity_map_default_memslots(struct kvm_vm *vm);
void tdp_identity_map_1g(struct kvm_vm *vm, u64 addr, u64 size);
u64 *tdp_get_pte(struct kvm_vm *vm, u64 l2_gpa);
+gpa_t tdp_copy_page_tables(struct kvm_vm *vm);
/*
* Basic CPU control in CR0
diff --git a/tools/testing/selftests/kvm/include/x86/svm.h b/tools/testing/selftests/kvm/include/x86/svm.h
index c8539166270ea..d2a6bf8503f78 100644
--- a/tools/testing/selftests/kvm/include/x86/svm.h
+++ b/tools/testing/selftests/kvm/include/x86/svm.h
@@ -316,4 +316,9 @@ struct __attribute__ ((__packed__)) vmcb {
#define SVM_CR0_SELECTIVE_MASK (X86_CR0_TS | X86_CR0_MP)
+static inline void invlpga(unsigned long addr, u32 asid)
+{
+ asm volatile("invlpga %0, %1" : : "a"(addr), "c"(asid) : "memory");
+}
+
#endif /* SELFTEST_KVM_SVM_H */
diff --git a/tools/testing/selftests/kvm/include/x86/vmx.h b/tools/testing/selftests/kvm/include/x86/vmx.h
index 8fd97434eee5c..4f2a0e3ac5ba1 100644
--- a/tools/testing/selftests/kvm/include/x86/vmx.h
+++ b/tools/testing/selftests/kvm/include/x86/vmx.h
@@ -518,4 +518,25 @@ static inline bool kvm_cpu_has_vmx_virtualize_apic_accesses(void)
void vm_enable_ept(struct kvm_vm *vm);
void prepare_virtualize_apic_accesses(struct vmx_pages *vmx, struct kvm_vm *vm);
+static inline void invvpid(unsigned long ext, u16 vpid, gva_t gva)
+{
+ struct {
+ u64 vpid : 16;
+ u64 rsvd : 48;
+ u64 gva;
+ } operand = { vpid, 0, gva };
+
+ asm volatile("invvpid %0, %1" : : "m"(operand), "r"(ext) : "memory");
+}
+
+static inline void invept(unsigned long ext, u64 eptp)
+{
+ struct {
+ u64 eptp;
+ u64 rsvd;
+ } operand = { eptp, 0 };
+
+ asm volatile("invept %0, %1" : : "m"(operand), "r"(ext) : "memory");
+}
+
#endif /* SELFTEST_KVM_VMX_H */
diff --git a/tools/testing/selftests/kvm/lib/x86/processor.c b/tools/testing/selftests/kvm/lib/x86/processor.c
index 8374afd08bafe..57ecf8b381b5a 100644
--- a/tools/testing/selftests/kvm/lib/x86/processor.c
+++ b/tools/testing/selftests/kvm/lib/x86/processor.c
@@ -554,6 +554,42 @@ void tdp_identity_map_1g(struct kvm_vm *vm, u64 addr, u64 size)
__tdp_map(vm, addr, addr, size, PG_LEVEL_1G);
}
+static gpa_t __virt_copy_page_tables(struct kvm_vm *vm, struct kvm_mmu *mmu,
+ gpa_t table_gpa, int level)
+{
+ gpa_t copy_gpa = vm_alloc_page_table(vm);
+ u64 *copy = addr_gpa2hva(vm, copy_gpa);
+ gpa_t pa;
+ int i;
+
+ memcpy(copy, addr_gpa2hva(vm, table_gpa), vm->page_size);
+ if (level == PG_LEVEL_4K)
+ return copy_gpa;
+
+ for (i = 0; i < vm->page_size / sizeof(u64); i++) {
+ if (!is_present_pte(mmu, ©[i]) || is_huge_pte(mmu, ©[i]))
+ continue;
+
+ /* Clear only the address bits, preserving C/S bits and flags */
+ pa = vm_untag_gpa(vm, PTE_GET_PA(copy[i]));
+ copy[i] &= ~pa;
+ copy[i] |= __virt_copy_page_tables(vm, mmu, pa, level - 1);
+ }
+ return copy_gpa;
+}
+
+/*
+ * Deep copy all levels of the TDP page tables, such that no page table pages
+ * are shared with the original. Returns the GPA of the new root.
+ */
+gpa_t tdp_copy_page_tables(struct kvm_vm *vm)
+{
+ struct kvm_mmu *mmu = &vm->stage2_mmu;
+
+ TEST_ASSERT(mmu->pgd_created, "TDP page tables not created");
+ return __virt_copy_page_tables(vm, mmu, mmu->pgd, mmu->pgtable_levels);
+}
+
/*
* Set Unusable Segment
*
diff --git a/tools/testing/selftests/kvm/x86/nested_tlb_flush_test.c b/tools/testing/selftests/kvm/x86/nested_tlb_flush_test.c
new file mode 100644
index 0000000000000..fb32b7bf6b19f
--- /dev/null
+++ b/tools/testing/selftests/kvm/x86/nested_tlb_flush_test.c
@@ -0,0 +1,526 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2026, Google LLC.
+ *
+ * Test TLB flushes with nested virtualization across three cases:
+ * 1. When L1 updates L2's (shadow) page tables
+ * 2. When L1 updates nested TDP
+ * 3. When KVM updates mappings on the host (KVM-induced TLB flushes)
+ */
+#include <fcntl.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#include "test_util.h"
+#include "kvm_util.h"
+#include "processor.h"
+#include "svm_util.h"
+#include "vmx.h"
+
+#define NR_ITERATIONS 100
+
+#define VAL1 0x11111111ULL
+#define VAL2 0x22222222ULL
+
+#define TEST_VADDR 0x10000000ULL
+#define TEST_GPA 0x30000000ULL
+#define TEST_NESTED_GPA 0x40000000ULL
+
+#define TEST_MEMSLOT 10
+
+#define SYNC_MOVE_MEMSLOT 1
+
+enum guest_flush_method {
+ TLB_FLUSH_TAG = 0,
+ TLB_FLUSH_VADDR,
+ TLB_CHANGE_TAG,
+ TLB_FLUSH_NONE,
+};
+
+static u64 *pte_gva;
+static gpa_t test_gpa[2];
+static enum guest_flush_method flush_method;
+static u32 max_asid;
+
+/* TDP roots for the multi-root test, root 1 is a non-aliasing copy of root 0 */
+static gpa_t tdp_root_gpa[2];
+/* Value L2 expects at TEST_VADDR, 0 if L2 shouldn't access TEST_VADDR */
+static u64 l2_expected_val;
+
+static void svm_tlb_flush(struct svm_test_data *svm,
+ enum guest_flush_method method)
+{
+ struct vmcb *vmcb = svm->vmcb;
+
+ switch (method) {
+ case TLB_CHANGE_TAG:
+ vmcb->control.asid++;
+ vmcb->control.tlb_ctl = TLB_CONTROL_DO_NOTHING;
+ if (vmcb->control.asid >= max_asid) {
+ vmcb->control.asid = 1;
+ vmcb->control.tlb_ctl = TLB_CONTROL_FLUSH_ALL_ASID;
+ }
+ break;
+ case TLB_FLUSH_TAG:
+ vmcb->control.tlb_ctl = TLB_CONTROL_FLUSH_ASID;
+ break;
+ case TLB_FLUSH_VADDR:
+ GUEST_ASSERT(!svm->ncr3_gpa);
+ vmcb->control.tlb_ctl = TLB_CONTROL_DO_NOTHING;
+ invlpga(TEST_VADDR, vmcb->control.asid);
+ break;
+ case TLB_FLUSH_NONE:
+ vmcb->control.tlb_ctl = TLB_CONTROL_DO_NOTHING;
+ break;
+ }
+}
+
+static void vmx_tlb_flush(struct vmx_pages *vmx, enum guest_flush_method method)
+{
+ u16 vpid = vmread(VIRTUAL_PROCESSOR_ID);
+
+ switch (method) {
+ case TLB_CHANGE_TAG:
+ GUEST_ASSERT(!vmx->eptp_gpa);
+ vmwrite(VIRTUAL_PROCESSOR_ID, vpid + 1);
+ break;
+ case TLB_FLUSH_TAG:
+ if (vmx->eptp_gpa)
+ invept(1, vmread(EPT_POINTER));
+ else
+ invvpid(1, vpid, 0);
+ break;
+ case TLB_FLUSH_VADDR:
+ GUEST_ASSERT(!vmx->eptp_gpa);
+ invvpid(0, vpid, TEST_VADDR);
+ break;
+ case TLB_FLUSH_NONE:
+ break;
+ }
+}
+
+static void prepare_l2(void *nested_state, void *l2_guest_code)
+{
+ struct vmx_pages *vmx = nested_state;
+
+ if (this_cpu_has(X86_FEATURE_VMX)) {
+ u32 sec_exec_ctl;
+
+ prepare_for_vmx_operation(vmx);
+ load_vmcs(vmx);
+ prepare_vmcs(vmx, l2_guest_code);
+
+ /* Enable VPID */
+ sec_exec_ctl = vmread(SECONDARY_VM_EXEC_CONTROL);
+ sec_exec_ctl |= SECONDARY_EXEC_ENABLE_VPID;
+ vmwrite(SECONDARY_VM_EXEC_CONTROL, sec_exec_ctl);
+ vmwrite(VIRTUAL_PROCESSOR_ID, 1);
+ } else {
+ generic_svm_setup(nested_state, l2_guest_code);
+ }
+}
+
+static void __run_l2(void *nested_state, bool launch,
+ enum guest_flush_method method)
+{
+ struct svm_test_data *svm;
+
+ if (this_cpu_has(X86_FEATURE_VMX)) {
+ vmx_tlb_flush(nested_state, method);
+ if (launch)
+ vmlaunch();
+ else
+ vmresume();
+ GUEST_ASSERT_EQ(vmread(VM_EXIT_REASON), EXIT_REASON_VMCALL);
+ vmwrite(GUEST_RIP, vmread(GUEST_RIP) + 3); /* skip over VMCALL */
+ } else {
+ svm = nested_state;
+ svm_tlb_flush(svm, method);
+ run_guest(svm->vmcb, svm->vmcb_gpa);
+ GUEST_ASSERT_EQ(svm->vmcb->control.exit_code, SVM_EXIT_VMMCALL);
+ svm->vmcb->save.rip += 3; /* skip over VMMCALL */
+ }
+}
+
+static void run_l2(void *nested_state, bool launch)
+{
+ __run_l2(nested_state, launch, flush_method);
+}
+
+static void l2_guest_code(void)
+{
+ u64 expected_val;
+
+ for (;;) {
+ expected_val = READ_ONCE(l2_expected_val);
+ if (expected_val)
+ GUEST_ASSERT_EQ(READ_ONCE(*(u64 *)TEST_VADDR), expected_val);
+ vmcall();
+ }
+}
+
+static void l1_guest_code(void *data)
+{
+ gpa_t gpa;
+ int i;
+
+ prepare_l2(data, l2_guest_code);
+
+ /*
+ * Alternately switch the PTE (or TDP PTE) mapping TEST_VADDR between
+ * two pages containing VAL1 and VAL2, flush TLBs, and verify that L2
+ * reads the expected value.
+ */
+ for (i = 0; i < NR_ITERATIONS; i++) {
+ gpa = test_gpa[i % 2];
+
+ *pte_gva &= ~PHYSICAL_PAGE_MASK;
+ *pte_gva |= gpa & PHYSICAL_PAGE_MASK;
+
+ WRITE_ONCE(l2_expected_val, (i % 2 == 0) ? VAL1 : VAL2);
+ run_l2(data, i == 0);
+ }
+
+ GUEST_DONE();
+}
+
+/*
+ * Verify that flushing L2's ASID invalidates nested TDP translations for all
+ * TDP roots, not just the one in use at the time of the flush. L1 updates the
+ * nested TDP PTE for TEST_VADDR in root 0, flushes L2's ASID while running L2
+ * with root 1, then runs L2 with root 0 *without* flushing. L2 must observe
+ * the new mapping, as the flush invalidated all translations for the ASID.
+ *
+ * Root 1 is a full copy of root 0, such that none of the nested TDP pages are
+ * shared, i.e. KVM's shadow pages for root 0 are not reachable through the
+ * shadow root for root 1.
+ */
+static void l1_guest_code_multi_tdp_root(void *data)
+{
+ struct svm_test_data *svm = data;
+ struct vmcb *vmcb = svm->vmcb;
+ int i;
+
+ GUEST_ASSERT(this_cpu_has(X86_FEATURE_SVM));
+ GUEST_ASSERT_EQ(svm->ncr3_gpa, tdp_root_gpa[0]);
+
+ prepare_l2(data, l2_guest_code);
+
+ for (i = 0; i < NR_ITERATIONS; i++) {
+ *pte_gva &= ~PHYSICAL_PAGE_MASK;
+ *pte_gva |= test_gpa[i % 2] & PHYSICAL_PAGE_MASK;
+
+ /* Flush L2's ASID (per flush_method) while using root 1 */
+ vmcb->control.nested_cr3 = tdp_root_gpa[1];
+ WRITE_ONCE(l2_expected_val, 0);
+ run_l2(data, i == 0);
+
+ /* Switch back to root 0 without flushing */
+ vmcb->control.nested_cr3 = tdp_root_gpa[0];
+ WRITE_ONCE(l2_expected_val, (i % 2 == 0) ? VAL1 : VAL2);
+ __run_l2(data, false, TLB_FLUSH_NONE);
+ }
+
+ GUEST_DONE();
+}
+
+static void prep_guest_flush_test(struct kvm_vm *vm, bool tdp_enabled,
+ bool copy_tdp_root)
+{
+ gva_t page_gva[2], pte_map_gva;
+ gpa_t pte_gpa;
+ u64 *ptep;
+
+ /*
+ * Allocate two pages in the VM and initialize them with different
+ * values. L1 will update the mappings to switch between these pages and
+ * L2 will verify that the accessed value indicates the right page.
+ */
+ page_gva[0] = vm_alloc_page(vm);
+ page_gva[1] = vm_alloc_page(vm);
+
+ *(u64 *)addr_gva2hva(vm, page_gva[0]) = VAL1;
+ *(u64 *)addr_gva2hva(vm, page_gva[1]) = VAL2;
+
+ test_gpa[0] = addr_gva2gpa(vm, page_gva[0]);
+ test_gpa[1] = addr_gva2gpa(vm, page_gva[1]);
+
+ if (tdp_enabled) {
+ virt_pg_map(vm, TEST_VADDR, TEST_NESTED_GPA);
+ tdp_map(vm, TEST_NESTED_GPA, test_gpa[0], vm->page_size);
+ ptep = tdp_get_pte(vm, TEST_NESTED_GPA);
+ } else {
+ virt_pg_map(vm, TEST_VADDR, test_gpa[0]);
+ ptep = vm_get_pte(vm, TEST_VADDR);
+ }
+
+ /*
+ * Map the PTE (or TDP PTE) for TEST_VADDR, such that L1 can read and
+ * update the mapping. Pass the PTE GVA to L1 to directly use it (rather
+ * than L1 walking the page tables.
+ */
+ pte_gpa = addr_hva2gpa(vm, ptep);
+ pte_map_gva = vm_unused_gva_gap(vm, vm->page_size, KVM_UTIL_MIN_VADDR);
+ virt_pg_map(vm, pte_map_gva, pte_gpa & PAGE_MASK);
+ pte_gva = (u64 *)(pte_map_gva + (pte_gpa & ~PAGE_MASK));
+
+ sync_global_to_guest(vm, pte_gva);
+ sync_global_to_guest(vm, test_gpa);
+
+ if (!tdp_enabled)
+ return;
+
+ /*
+ * Must be done after all pages accessed by L2 are allocated, including
+ * nested state (e.g. L2's stack) and page tables for TEST_VADDR.
+ */
+ tdp_identity_map_default_memslots(vm);
+
+ /* Must be done after all TDP mappings are created */
+ if (copy_tdp_root) {
+ tdp_root_gpa[0] = vm->stage2_mmu.pgd;
+ tdp_root_gpa[1] = tdp_copy_page_tables(vm);
+ sync_global_to_guest(vm, tdp_root_gpa);
+ }
+}
+
+static void l2_guest_code_memslot_move(void)
+{
+ int i;
+ u64 val;
+
+ for (i = 0; i < NR_ITERATIONS; i++) {
+ val = READ_ONCE(*(u64 *)TEST_VADDR);
+ GUEST_ASSERT_EQ(val, VAL1);
+
+ vmcall();
+
+ val = READ_ONCE(*(u64 *)TEST_VADDR);
+ GUEST_ASSERT_EQ(val, VAL2);
+
+ GUEST_SYNC2(SYNC_MOVE_MEMSLOT, 1);
+
+ val = READ_ONCE(*(u64 *)TEST_VADDR);
+ GUEST_ASSERT_EQ(val, VAL1);
+
+ vmcall();
+ }
+}
+
+static void l1_guest_code_memslot_move(void *data)
+{
+ int i;
+ u64 val;
+
+ prepare_l2(data, l2_guest_code_memslot_move);
+
+ for (i = 0; i < NR_ITERATIONS; i++) {
+ /* Verify that accessing VADDR reads from page 1 in both L1 & L2 */
+ val = READ_ONCE(*(u64 *)TEST_VADDR);
+ GUEST_ASSERT_EQ(val, VAL1);
+
+ run_l2(data, i == 0);
+
+ /*
+ * Update VADDR to point at page 2. KVM should flush both L1 and
+ * L2's TLB translations so that further accesses go to page 2.
+ */
+ GUEST_SYNC2(SYNC_MOVE_MEMSLOT, 2);
+
+ /*
+ * Verify that accessing VADDR reads from page 2. A stale TLB
+ * entry would lead to reading from page 1. Enter L2 and check
+ * access to page 2, to verify that a TLB flush from L1's
+ * context flushes both L1 and L2.
+ */
+ val = READ_ONCE(*(u64 *)TEST_VADDR);
+ GUEST_ASSERT_EQ(val, VAL2);
+
+ run_l2(data, false);
+
+ /*
+ * L2 updated VADDR to point at page 1 again. Verify access to
+ * page 1 again to verify that a TLB flush from L2's context
+ * flushes both L1 and L2 TLB translations.
+ */
+ val = READ_ONCE(*(u64 *)TEST_VADDR);
+ GUEST_ASSERT_EQ(val, VAL1);
+ }
+
+ GUEST_DONE();
+}
+
+static void prep_memslot_move_test(struct kvm_vm *vm, bool tdp_enabled)
+{
+ /*
+ * Disable KVM_X86_QUIRK_SLOT_ZAP_ALL. By default, moving a memslot
+ * invalidates all MMU roots, which implicitly flushes the guest TLB
+ * and masks missing TLB flush bugs. Disabling it forces KVM to keep
+ * the same roots, relying solely on explicit TLB flushes.
+ */
+ TEST_REQUIRE(kvm_check_cap(KVM_CAP_DISABLE_QUIRKS2) & KVM_X86_QUIRK_SLOT_ZAP_ALL);
+ vm_enable_cap(vm, KVM_CAP_DISABLE_QUIRKS2, KVM_X86_QUIRK_SLOT_ZAP_ALL);
+
+ vm_userspace_mem_region_add(vm, VM_MEM_SRC_ANONYMOUS,
+ TEST_GPA, TEST_MEMSLOT,
+ /*npages=*/2, /*flags=*/0);
+
+ virt_pg_map(vm, TEST_VADDR, TEST_GPA);
+
+ if (tdp_enabled)
+ tdp_map(vm, TEST_GPA, TEST_GPA, vm->page_size);
+
+ *(u64 *)addr_gpa2hva(vm, TEST_GPA) = VAL1;
+ *(u64 *)addr_gpa2hva(vm, TEST_GPA + vm->page_size) = VAL2;
+
+ /*
+ * Must be done after all pages accessed by L2 are allocated, including
+ * nested state (e.g. L2's stack) and page tables for TEST_VADDR.
+ */
+ if (tdp_enabled)
+ tdp_identity_map_default_memslots(vm);
+}
+
+static void calc_max_svm_asid(void)
+{
+ const struct kvm_cpuid_entry2 *entry;
+
+ if (!kvm_cpu_has(X86_FEATURE_SVM))
+ return;
+
+ entry = get_cpuid_entry(kvm_get_supported_cpuid(), 0x8000000A, 0);
+ max_asid = entry ? entry->ebx : 0;
+}
+
+static void run_test(void *guest_code, bool tdp_enabled)
+{
+ struct kvm_vcpu *vcpu;
+ gva_t nested_gva = 0;
+ struct kvm_vm *vm;
+ struct ucall uc;
+
+ vm = vm_create_with_one_vcpu(&vcpu, guest_code);
+ if (tdp_enabled)
+ vm_enable_tdp(vm);
+
+ if (kvm_cpu_has(X86_FEATURE_VMX))
+ vcpu_alloc_vmx(vm, &nested_gva);
+ else
+ vcpu_alloc_svm(vm, &nested_gva);
+
+ if (guest_code == l1_guest_code_memslot_move)
+ prep_memslot_move_test(vm, tdp_enabled);
+ else if (guest_code == l1_guest_code_multi_tdp_root)
+ prep_guest_flush_test(vm, tdp_enabled, true);
+ else
+ prep_guest_flush_test(vm, tdp_enabled, false);
+
+ vcpu_args_set(vcpu, 1, nested_gva);
+
+ sync_global_to_guest(vm, flush_method);
+ sync_global_to_guest(vm, max_asid);
+
+ for (;;) {
+ vcpu_run(vcpu);
+ TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO);
+
+ switch (get_ucall(vcpu, &uc)) {
+ case UCALL_ABORT:
+ REPORT_GUEST_ASSERT(uc);
+ break;
+ case UCALL_SYNC:
+ /*
+ * Move the base GPA of the memslot between TEST_GPA and
+ * TEST_GPA - PAGE_SIZE. This effectively moves
+ * TEST_VADDR to point at page 1 or page 2 in the
+ * memslot.
+ */
+ if (uc.args[0] == SYNC_MOVE_MEMSLOT) {
+ gpa_t new_gpa = (uc.args[1] == 1) ?
+ TEST_GPA : (TEST_GPA - vm->page_size);
+
+ vm_mem_region_move(vm, TEST_MEMSLOT, new_gpa);
+ } else {
+ TEST_FAIL("Unknown sync code: %lu", uc.args[0]);
+ }
+ break;
+ case UCALL_DONE:
+ goto done;
+ default:
+ TEST_FAIL("Unknown ucall %lu", uc.cmd);
+ }
+ }
+
+done:
+ kvm_vm_free(vm);
+}
+
+#define tlb_test(test_name, guest_code, tdp_setting, flush_setting) \
+do { \
+ tdp_setting; \
+ flush_setting; \
+ \
+ if (tdp_enabled && !kvm_cpu_has_tdp()) { \
+ pr_info("Skipping: " test_name " (no TDP support)\n"); \
+ break; \
+ } \
+ \
+ /* \
+ * VPIDs do not tag guest-physical translations, i.e. changing or \
+ * flushing VPIDs does not flush EPT translations, and single-context \
+ * INVEPT only flushes the target EPTP (not all TDP roots). \
+ */ \
+ if (tdp_enabled && kvm_cpu_has_ept() && \
+ (flush_method == TLB_CHANGE_TAG || \
+ guest_code == l1_guest_code_multi_tdp_root)) { \
+ pr_info("Skipping: " test_name " (not applicable to EPTs)\n"); \
+ break; \
+ } \
+ \
+ pr_info("Testing " test_name "...\n"); \
+ run_test(guest_code, tdp_enabled); \
+} while (0)
+
+int main(int argc, char *argv[])
+{
+ bool tdp_enabled;
+
+ TEST_REQUIRE(kvm_cpu_has(X86_FEATURE_SVM) || kvm_cpu_has(X86_FEATURE_VMX));
+
+ calc_max_svm_asid();
+
+ tlb_test("Guest-triggered TLB flush (flush entire tag), TDP disabled",
+ l1_guest_code, tdp_enabled = false,
+ flush_method = TLB_FLUSH_TAG);
+ tlb_test("Guest-triggered TLB flush (flush individual address), TDP disabled",
+ l1_guest_code, tdp_enabled = false,
+ flush_method = TLB_FLUSH_VADDR);
+ tlb_test("Guest-triggered TLB flush (change tag), TDP disabled",
+ l1_guest_code, tdp_enabled = false,
+ flush_method = TLB_CHANGE_TAG);
+
+ tlb_test("Guest-triggered TLB flush (flush entire tag), TDP enabled",
+ l1_guest_code, tdp_enabled = true,
+ flush_method = TLB_FLUSH_TAG);
+ tlb_test("Guest-triggered TLB flush (change tag), TDP enabled",
+ l1_guest_code, tdp_enabled = true,
+ flush_method = TLB_CHANGE_TAG);
+
+ tlb_test("Guest-triggered TLB flush with multiple TDP roots (flush entire tag), TDP enabled",
+ l1_guest_code_multi_tdp_root, tdp_enabled = true,
+ flush_method = TLB_FLUSH_TAG);
+ tlb_test("Guest-triggered TLB flush with multiple TDP roots (change tag), TDP enabled",
+ l1_guest_code_multi_tdp_root, tdp_enabled = true,
+ flush_method = TLB_CHANGE_TAG);
+
+
+ tlb_test("KVM-triggered TLB flush via memslot move, TDP disabled",
+ l1_guest_code_memslot_move, tdp_enabled = false,
+ flush_method = TLB_FLUSH_NONE);
+ tlb_test("KVM-triggered TLB flush via memslot move, TDP enabled",
+ l1_guest_code_memslot_move, tdp_enabled = true,
+ flush_method = TLB_FLUSH_NONE);
+
+ return 0;
+}
|
| 3/2 | 2026/10/12 23:48 | llm | patch-evaluator |
1mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 29472 output: 63 thoughts: 3358 Thoughts: **KVM Patch Review: A Deep Dive** Okay, let's break down this KVM patch for x86. My initial thought is to analyze the changes to determine fuzzing suitability. First, I see the diff targets amd64, which is expected. The changes span several key files. I notice documentation updates, which are good, but not my immediate focus. The core of the patch involves ASID and TLB management, especially in nested virtualization (SVM). I'm seeing additions like `KVM_REQ_MMU_SYNC_ALL_ROOTS`, and extensive rework of ASID allocation and flushing logic. Crucially, there's a new TLB tag allocator using bitmaps. It seems like the patch introduces a generic TLB tag allocator for both VMX and SVM. The nested SVM code is being completely revamped. We have `nested_svm_transition_tlb_flush` being replaced by `nested_svm_entry_tlb_flush` and `nested_svm_exit_tlb_flush`, and `nested_svm_service_local_tlb_flushes` is now called in multiple places. It is tracking `last_asid` and checking for `new_asid` for synchronization. The ASID management in SVM has undergone a significant overhaul. We're moving from a per-CPU generation counter to `kvm_alloc_tlb_tag()`. There are updates to `invlpga_interception` to handle non-canonical addresses, and the handling of full ASID flushes. There is also fallback ASID support in `pre_svm_run`. VPID allocation has been switched from custom bitmap to `kvm_init_tlb_tags`, `kvm_alloc_tlb_tag`, and `kvm_free_tlb_tag` in VMX. There's the new `KVM_REQ_MMU_SYNC_ALL_ROOTS` being handled in `vcpu_enter_guest`, which raises my attention as this is a core KVM entry point. The patch also updates `kvm_vcpu_flush_tlb_guest`. Importantly, there's a selftest added. This is a good sign, but doesn't preclude the need for fuzzing. My next step is determining the fuzzing reachability and focus. KVM is definitely reachable via standard syscalls like `/dev/kvm`. Syzkaller already has KVM support, including nested virtualization and ioctls. I need to confirm whether nested SVM/VMX can be fuzzed in QEMU (yes, QEMU on x86 supports nested SVM/VMX, and KVM asid and VPID allocation can be accessed by ioctls on `/dev/kvm`, so there is reachability). The key here is that both VMX and SVM code is touched, and that nested virtualization is modified. The changes in TLB tag allocation and ASID lifecycle are important to note. All of this, plus the presence of assertions, makes this patch *extremely* interesting for fuzzing. Now, let's get specific about focusing the fuzzer. I need to identify which functions are *new* or modified, and not just ubiquitous lifecycle paths. So I need to go through the changes and identify which ones actually exercise the logic being modified. From the C files, I've identified the following: - `kvm_init_tlb_tags`, `kvm_destroy_tlb_tags`, `kvm_alloc_tlb_tag`, `kvm_free_tlb_tag` in `arch/x86/kvm/mmu/mmu.c`. - `nested_svm_entry_tlb_flush`, `nested_svm_exit_tlb_flush`, `nested_svm_service_local_tlb_flushes`, `enter_svm_guest_mode`, `nested_svm_vmrun`, `nested_svm_vmexit`, `svm_allocate_nested`, `svm_free_nested`, `svm_leave_nested`, `svm_set_nested_state` in `arch/x86/kvm/svm/nested.c` - `sev_vcpu_migrate_asid`, `sev_migrate_from`, `pre_sev_run`, `sev_init_vmcb` in `arch/x86/kvm/svm/sev.c`. - `invlpga_interception`, `pre_svm_run`, `svm_flush_tlb_asid`, `svm_flush_tlb_all`, `svm_vcpu_create`, `svm_vcpu_free`, ` |
| 4/2 | 2026/10/12 23:50 | tool | grepper |
0mArgs:{
"Expression": "(nested_svm_entry_tlb_flush|invlpga_interception|kvm_alloc_tlb_tag|allocate_asid)",
"PathPrefixes": [
"arch/x86/kvm"
]
}Results: {
"Output": "arch/x86/kvm/mmu.h=424=void kvm_destroy_tlb_tags(void);\narch/x86/kvm/mmu.h:425:kvm_tlb_tag_t kvm_alloc_tlb_tag(void);\narch/x86/kvm/mmu.h-426-void kvm_free_tlb_tag(kvm_tlb_tag_t tag);\n--\narch/x86/kvm/mmu/mmu.c=8236=EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_destroy_tlb_tags);\narch/x86/kvm/mmu/mmu.c-8237-\narch/x86/kvm/mmu/mmu.c:8238:kvm_tlb_tag_t kvm_alloc_tlb_tag(void)\narch/x86/kvm/mmu/mmu.c-8239-{\n--\narch/x86/kvm/mmu/mmu.c-8253-}\narch/x86/kvm/mmu/mmu.c:8254:EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_alloc_tlb_tag);\narch/x86/kvm/mmu/mmu.c-8255-\n--\narch/x86/kvm/svm/nested.c=656=static void nested_save_pending_event_to_vmcb12(struct vcpu_svm *svm,\n--\narch/x86/kvm/svm/nested.c-688-\narch/x86/kvm/svm/nested.c:689:static void nested_svm_entry_tlb_flush(struct kvm_vcpu *vcpu)\narch/x86/kvm/svm/nested.c-690-{\n--\narch/x86/kvm/svm/nested.c=894=static void nested_vmcb02_prepare_control(struct vcpu_svm *svm)\n--\narch/x86/kvm/svm/nested.c-903-\narch/x86/kvm/svm/nested.c:904:\tnested_svm_entry_tlb_flush(vcpu);\narch/x86/kvm/svm/nested.c-905-\n--\narch/x86/kvm/svm/nested.c=1549=int svm_allocate_nested(struct vcpu_svm *svm)\n--\narch/x86/kvm/svm/nested.c-1567-\narch/x86/kvm/svm/nested.c:1568:\tsvm-\u003enested.asid02 = allocate_asid(\u0026svm-\u003evcpu);\narch/x86/kvm/svm/nested.c-1569-\n--\narch/x86/kvm/svm/svm.c=1339=static int svm_vcpu_create(struct kvm_vcpu *vcpu)\n--\narch/x86/kvm/svm/svm.c-1372-\narch/x86/kvm/svm/svm.c:1373:\tsvm-\u003easid = allocate_asid(vcpu);\narch/x86/kvm/svm/svm.c-1374-\tif (!svm-\u003easid) {\n--\narch/x86/kvm/svm/svm.c=2457=static int clgi_interception(struct kvm_vcpu *vcpu)\n--\narch/x86/kvm/svm/svm.c-2468-\narch/x86/kvm/svm/svm.c:2469:static int invlpga_interception(struct kvm_vcpu *vcpu)\narch/x86/kvm/svm/svm.c-2470-{\n--\narch/x86/kvm/svm/svm.c=3483=static int (*const svm_exit_handlers[])(struct kvm_vcpu *vcpu) = {\n--\narch/x86/kvm/svm/svm.c-3526-\t[SVM_EXIT_INVLPG]\t\t\t= invlpg_interception,\narch/x86/kvm/svm/svm.c:3527:\t[SVM_EXIT_INVLPGA]\t\t\t= invlpga_interception,\narch/x86/kvm/svm/svm.c-3528-\t[SVM_EXIT_IOIO]\t\t\t\t= io_interception,\n--\narch/x86/kvm/svm/svm.c=5787=static __init int svm_hardware_setup(void)\n--\narch/x86/kvm/svm/svm.c-5952-\narch/x86/kvm/svm/svm.c:5953:\tfallback_asid = kvm_alloc_tlb_tag();\narch/x86/kvm/svm/svm.c-5954-\n--\narch/x86/kvm/svm/svm.h=1082=extern kvm_tlb_tag_t fallback_asid;\narch/x86/kvm/svm/svm.h-1083-\narch/x86/kvm/svm/svm.h:1084:static inline kvm_tlb_tag_t allocate_asid(struct kvm_vcpu *vcpu)\narch/x86/kvm/svm/svm.h-1085-{\n--\narch/x86/kvm/svm/svm.h-1087-\t\treturn sev_get_asid(vcpu-\u003ekvm);\narch/x86/kvm/svm/svm.h:1088:\treturn kvm_alloc_tlb_tag() ?: fallback_asid;\narch/x86/kvm/svm/svm.h-1089-}\n--\narch/x86/kvm/vmx/vmx.h=311=static __always_inline kvm_tlb_tag_t allocate_vpid(void)\narch/x86/kvm/vmx/vmx.h-312-{\narch/x86/kvm/vmx/vmx.h:313:\treturn enable_vpid ? kvm_alloc_tlb_tag() : 0;\narch/x86/kvm/vmx/vmx.h-314-}\n"
} |
| 5/2 | 2026/10/12 23:50 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 35539 output: 32 thoughts: 25 Thoughts: Here's my take on the situation: **KVM Configuration Deep Dive** Okay, time to dive into the kernel configuration. I need to make absolutely certain the KVM infrastructure is correctly set up. My immediate focus is on verifying the necessary Kconfig options are enabled – specifically, `KVM`, `KVM_AMD`, and `KVM_INTEL`. It's a fundamental step; I can't proceed without ensuring the core modules are present and correctly configured to support hardware virtualization. I need to confirm the architecture-specific KVM modules are linked in with the correct flags based on the host. I know better than to assume they're all enabled by default; there's always the chance some distros or custom configurations have something disabled. I'm thinking I'll start by querying the kernel config directly, perhaps using `make menuconfig` or `make kernel_config` or something similar, to verify the state of those modules. Any deviation from what I expect could indicate a fundamental misunderstanding or, worse, an error in how I'm handling the build environment. I also expect there to be a few additional supporting flags needed. I'll need to check those as well. |
| 6/2 | 2026/10/12 23:50 | tool | grepper |
0mArgs:{
"Expression": "config KVM",
"PathPrefixes": [
"arch/x86/kvm"
]
}Results: {
"Output": "arch/x86/kvm/Kconfig=18=if VIRTUALIZATION\narch/x86/kvm/Kconfig-19-\narch/x86/kvm/Kconfig:20:config KVM_X86\narch/x86/kvm/Kconfig-21-\tdef_tristate KVM if (KVM_INTEL != n || KVM_AMD != n)\n--\narch/x86/kvm/Kconfig-51-\narch/x86/kvm/Kconfig:52:config KVM\narch/x86/kvm/Kconfig-53-\ttristate \"Kernel-based Virtual Machine (KVM) support\"\n--\narch/x86/kvm/Kconfig-68-\narch/x86/kvm/Kconfig:69:config KVM_WERROR\narch/x86/kvm/Kconfig-70-\tbool \"Compile KVM with -Werror\"\n--\narch/x86/kvm/Kconfig-82-\narch/x86/kvm/Kconfig:83:config KVM_VM_MEMORY_ATTRIBUTES\narch/x86/kvm/Kconfig-84-\tbool \"Enable per-VM PRIVATE vs. SHARED attributes (for CoCo VMs)\"\n--\narch/x86/kvm/Kconfig-94-\narch/x86/kvm/Kconfig:95:config KVM_SW_PROTECTED_VM\narch/x86/kvm/Kconfig-96-\tbool \"Enable support for KVM software-protected VMs\"\n--\narch/x86/kvm/Kconfig-106-\narch/x86/kvm/Kconfig:107:config KVM_INTEL\narch/x86/kvm/Kconfig-108-\ttristate \"KVM for Intel (and compatible) processors support\"\n--\narch/x86/kvm/Kconfig-117-\narch/x86/kvm/Kconfig:118:config KVM_INTEL_PROVE_VE\narch/x86/kvm/Kconfig-119- bool \"Check that guests do not receive #VE exceptions\"\n--\narch/x86/kvm/Kconfig=133=config X86_SGX_KVM\n--\narch/x86/kvm/Kconfig-144-\narch/x86/kvm/Kconfig:145:config KVM_INTEL_TDX\narch/x86/kvm/Kconfig-146-\tbool \"Intel Trust Domain Extensions (TDX) support\"\n--\narch/x86/kvm/Kconfig-155-\narch/x86/kvm/Kconfig:156:config KVM_AMD\narch/x86/kvm/Kconfig-157-\ttristate \"KVM for AMD processors support\"\n--\narch/x86/kvm/Kconfig-165-\narch/x86/kvm/Kconfig:166:config KVM_AMD_SEV\narch/x86/kvm/Kconfig-167-\tbool \"AMD Secure Encrypted Virtualization (SEV) support\"\n--\narch/x86/kvm/Kconfig-181-\narch/x86/kvm/Kconfig:182:config KVM_IOAPIC\narch/x86/kvm/Kconfig-183-\tbool \"I/O APIC, PIC, and PIT emulation\"\n--\narch/x86/kvm/Kconfig-191-\narch/x86/kvm/Kconfig:192:config KVM_SMM\narch/x86/kvm/Kconfig-193-\tbool \"System Management Mode emulation\"\n--\narch/x86/kvm/Kconfig-202-\narch/x86/kvm/Kconfig:203:config KVM_HYPERV\narch/x86/kvm/Kconfig-204-\tbool \"Support for Microsoft Hyper-V emulation\"\n--\narch/x86/kvm/Kconfig-216-\narch/x86/kvm/Kconfig:217:config KVM_XEN\narch/x86/kvm/Kconfig-218-\tbool \"Support for Xen hypercall interface\"\n--\narch/x86/kvm/Kconfig-225-\narch/x86/kvm/Kconfig:226:config KVM_PROVE_MMU\narch/x86/kvm/Kconfig-227-\tbool \"Prove KVM MMU correctness\"\n--\narch/x86/kvm/Kconfig-238-\narch/x86/kvm/Kconfig:239:config KVM_EXTERNAL_WRITE_TRACKING\narch/x86/kvm/Kconfig-240-\tbool\narch/x86/kvm/Kconfig-241-\narch/x86/kvm/Kconfig:242:config KVM_MAX_NR_VCPUS\narch/x86/kvm/Kconfig-243-\tint \"Maximum number of vCPUs per KVM guest\"\n"
} |
| 7/2 | 2026/10/12 23:50 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 37144 output: 37 thoughts: 24 Thoughts: Here's my take on the thought process: **Diving into the KVM x86 Build Process** Okay, so I need to understand the build process for the KVM x86 components. My usual go-to is the `Makefile`. That's where all the magic happens. Time to navigate into the `arch/x86/kvm/` directory and see what's what. I'm expecting to find the file dependencies, compiler flags, and linking steps detailed there. Hopefully, it's well-structured and follows standard conventions. I'm especially interested in how the core KVM modules are compiled and linked with the x86 architecture-specific code. Let's see if there are any interesting optimizations or specific build configurations in use for KVM. |
| 8/2 | 2026/10/12 23:50 | tool | grepper |
0mArgs:{
"Expression": "nested\\.o|mmu\\.o",
"PathPrefixes": [
"arch/x86/kvm"
]
}Results: {
"Output": "arch/x86/kvm/Makefile=8=kvm-y\t\t\t+= x86.o emulate.o irq.o lapic.o cpuid.o msrs.o pmu.o regs.o \\\narch/x86/kvm/Makefile:9:\t\t\t mtrr.o debugfs.o mmu/mmu.o mmu/page_track.o mmu/spte.o\narch/x86/kvm/Makefile-10-\narch/x86/kvm/Makefile:11:kvm-$(CONFIG_X86_64) += mmu/tdp_iter.o mmu/tdp_mmu.o\narch/x86/kvm/Makefile-12-kvm-$(CONFIG_KVM_IOAPIC) += i8259.o i8254.o ioapic.o\n--\narch/x86/kvm/Makefile=17=kvm-intel-y\t\t+= vmx/vmx.o vmx/vmenter.o vmx/pmu_intel.o vmx/vmcs12.o \\\narch/x86/kvm/Makefile:18:\t\t\t vmx/nested.o vmx/posted_intr.o vmx/main.o\narch/x86/kvm/Makefile-19-\n--\narch/x86/kvm/Makefile=22=kvm-intel-$(CONFIG_KVM_INTEL_TDX)\t+= vmx/tdx.o\narch/x86/kvm/Makefile-23-\narch/x86/kvm/Makefile:24:kvm-amd-y\t\t+= svm/svm.o svm/vmenter.o svm/pmu.o svm/nested.o svm/avic.o\narch/x86/kvm/Makefile-25-\n"
} |
| 9/2 | 2026/10/12 23:50 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 37825 output: 32 thoughts: 158 Thoughts: Okay, let's break this down. **KVM Module Dependencies and Configuration** Right, so, let's see, my mental map of the KVM kernel modules is clarifying. I'm seeing the component relationships: * `kvm.ko` is the core, and it pulls in the architecture-agnostic pieces like `x86.o` and the memory management unit, `mmu/mmu.o`. Makes sense, the fundamental virtualization machinery. * `kvm-amd.ko` is obviously AMD-specific, containing the Secure Virtual Machine (`svm`) implementation, including the nested virtualization (`svm/nested.o`) and the Secure Encrypted Virtualization (`svm/sev.o`) support. * Similarly, `kvm-intel.ko` takes care of the Intel side of things with the Virtual Machine Extensions (`vmx/vmx.o`) module. Now, let's connect this to the configuration. I'm seeing the `CONFIG_KVM`, `CONFIG_KVM_AMD`, and `CONFIG_KVM_INTEL` Kconfig options, which drive the compilation of these modules. I should also check if `CONFIG_KVM_AMD_SEV` matters. Right, that ties into the `svm/sev.c` file being compiled into `kvm-amd.ko` *conditionally*, based on that SEV config option. So, SEV is included only if I enable it in the build config. Okay, that's straightforward. |
| 10/2 | 2026/10/12 23:50 | tool | grepper |
0mArgs:{
"Expression": "sev\\.o",
"PathPrefixes": [
"arch/x86/kvm"
]
}Results: {
"Output": "arch/x86/kvm/Makefile=24=kvm-amd-y\t\t+= svm/svm.o svm/vmenter.o svm/pmu.o svm/nested.o svm/avic.o\narch/x86/kvm/Makefile-25-\narch/x86/kvm/Makefile:26:kvm-amd-$(CONFIG_KVM_AMD_SEV)\t+= svm/sev.o\narch/x86/kvm/Makefile-27-kvm-amd-$(CONFIG_KVM_HYPERV)\t+= svm/hyperv.o\n"
} |
| 11/2 | 2026/10/12 23:50 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 38495 output: 34 thoughts: 233 Thoughts: **Analyzing KVM and SEV Integration** Okay, so I'm seeing changes in `svm/sev.c`, specifically around `sev_vcpu_migrate_asid`, `pre_sev_run`, and `sev_init_vmcb`. This immediately flags my attention – it's definitely related to SEV, Secure Encrypted Virtualization. Good, I remember seeing that `KVM_AMD_SEV` config option, which is a boolean under `KVM_AMD`. But, before I go deeper down that rabbit hole, I should check something else. Hold on, let me quickly pivot. I see changes to `arch/x86/kvm/hyperv.h` with those `__kvm_hv_purge_tlb_flush_fifo`, `kvm_hv_purge_tlb_flush_fifo`, and `kvm_hv_purge_all_tlb_flush_fifos` functions. That screams Hyper-V interaction. Okay, let's see, what's the `KVM_HYPERV` Kconfig option look like? Ah, it's a boolean, depends on KVM, and offers "Support for Microsoft Hyper-V emulation." Now, the real question – is this enabled by default in a typical config or defconfig? I need to check `arch/x86/kvm/Kconfig` to see the defaults. This could be relevant to how these SEV changes are interacting with or affecting Hyper-V emulation, or vice-versa. Gotta see if there's any implicit coupling here. Gotta check the default configuration now. |
| 12/2 | 2026/10/12 23:50 | tool | grepper |
0mArgs:{
"Expression": "default y",
"PathPrefixes": [
"arch/x86/kvm/Kconfig"
]
}Results: {
"Output": "arch/x86/kvm/Kconfig=8=menuconfig VIRTUALIZATION\narch/x86/kvm/Kconfig-9-\tbool \"Virtualization\"\narch/x86/kvm/Kconfig:10:\tdefault y\narch/x86/kvm/Kconfig-11-\thelp\n--\narch/x86/kvm/Kconfig=145=config KVM_INTEL_TDX\narch/x86/kvm/Kconfig-146-\tbool \"Intel Trust Domain Extensions (TDX) support\"\narch/x86/kvm/Kconfig:147:\tdefault y\narch/x86/kvm/Kconfig-148-\tdepends on INTEL_TDX_HOST\n--\narch/x86/kvm/Kconfig=166=config KVM_AMD_SEV\narch/x86/kvm/Kconfig-167-\tbool \"AMD Secure Encrypted Virtualization (SEV) support\"\narch/x86/kvm/Kconfig:168:\tdefault y\narch/x86/kvm/Kconfig-169-\tdepends on KVM_AMD \u0026\u0026 X86_64\n--\narch/x86/kvm/Kconfig=182=config KVM_IOAPIC\narch/x86/kvm/Kconfig-183-\tbool \"I/O APIC, PIC, and PIT emulation\"\narch/x86/kvm/Kconfig:184:\tdefault y\narch/x86/kvm/Kconfig-185-\tdepends on KVM_X86\n--\narch/x86/kvm/Kconfig=192=config KVM_SMM\narch/x86/kvm/Kconfig-193-\tbool \"System Management Mode emulation\"\narch/x86/kvm/Kconfig:194:\tdefault y\narch/x86/kvm/Kconfig-195-\tdepends on KVM_X86\n--\narch/x86/kvm/Kconfig=203=config KVM_HYPERV\n--\narch/x86/kvm/Kconfig-205-\tdepends on KVM_X86\narch/x86/kvm/Kconfig:206:\tdefault y\narch/x86/kvm/Kconfig-207-\thelp\n"
} |
| 13/2 | 2026/10/12 23:50 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 39709 output: 36 thoughts: 1400 Thoughts: **KVM Configuration and TLB Tag Management Deep Dive**
Okay, let's start by clarifying the relevant KVM configurations. My initial thought process revolves around ensuring the correct modules are enabled, specifically `KVM`, `KVM_AMD`, and `KVM_INTEL`. I'm concerned about the necessary configurations to reach the core code. To get there, `KVM` (or `KVM_X86`) is essential, along with either `KVM_AMD` (for Secure Virtual Machine/SVM changes) or `KVM_INTEL` (for Virtual Machine Extensions/VMX changes).
It's natural to wonder if both `KVM_AMD` and `KVM_INTEL` can be enabled simultaneously. Fortunately, common distribution kernels and typical syzkaller x86_64 configs do exactly this: `CONFIG_KVM_INTEL=y` (or m) and `CONFIG_KVM_AMD=y` (or m) are both enabled. What about `KVM_AMD_SEV`? It defaults to 'y' if `KVM_AMD` is enabled. What about `KVM_HYPERV`? Similarly, it defaults to 'y' if `KVM_X86` is enabled.
Now, let's carefully check the FocusSymbols, so I have a full understanding of the code being changed. This means I'm going to look at functions created or modified.
In `arch/x86/kvm/mmu/mmu.c`, these functions have come into view:
* `kvm_init_tlb_tags`
* `kvm_destroy_tlb_tags`
* `kvm_alloc_tlb_tag`
* `kvm_free_tlb_tag`
A key concern is whether `kvm_alloc_tlb_tag` and `kvm_free_tlb_tag` are exported (non-static), because this means they're accessible outside the mmu.c module. And indeed, they are:
* `kvm_tlb_tag_t kvm_alloc_tlb_tag(void)` - EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_alloc_tlb_tag)
* `void kvm_free_tlb_tag(kvm_tlb_tag_t tag)` - EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_free_tlb_tag)
* `int kvm_init_tlb_tags(unsigned int nr_total, unsigned int nr_reserved)` - EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_init_tlb_tags)
* `void kvm_destroy_tlb_tags(void)` - EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_destroy_tlb_tags)
I need to emphasize that these functions are *not* general memory allocators like `kmalloc/kfree`. They're specific to allocating hardware TLB tags/VPIDs for KVM virtual CPUs. They are only called when creating/freeing vCPUs, or setting up hardware or a nested vCPU within KVM.
Now, let's delve deeper into their usage, specifically focusing on the SVM (Secure Virtual Machine) side. I'll check `allocate_asid` and `free_asid`:
In `arch/x86/kvm/svm/svm.h`, we find:
```c
static inline kvm_tlb_tag_t allocate_asid(struct kvm_vcpu *vcpu)
{
if (is_sev_guest(vcpu))
return sev_get_asid(vcpu->kvm);
return kvm_alloc_tlb_tag() ?: fallback_asid;
}
static inline void free_asid(kvm_tlb_tag_t asid)
{
if (asid > max_sev_asid && asid != fallback_asid)
kvm_free_tlb_tag(asid);
}
```
Since they're `static inline` in `svm.h`, they don't have explicit symbols in the vmlinux or kvm-amd.ko modules, which is an important observation.
My next task is to trace the callers within .c files for the SVM implementation:
* `svm_allocate_nested` (calls `allocate_asid`, checks `KVM_BUG_ON(!svm->nested.asid02)`)
* `svm_free_nested` (calls `
|
| 14/2 | 2026/10/12 23:50 | tool | grepper |
0mArgs:{
"Expression": "allocate_vpid",
"PathPrefixes": [
"arch/x86/kvm/vmx"
]
}Results: {
"Output": "arch/x86/kvm/vmx/nested.c=5421=static int enter_vmx_operation(struct kvm_vcpu *vcpu)\n--\narch/x86/kvm/vmx/nested.c-5444-\narch/x86/kvm/vmx/nested.c:5445:\tvmx-\u003enested.vpid02 = allocate_vpid();\narch/x86/kvm/vmx/nested.c-5446-\n--\narch/x86/kvm/vmx/vmx.c=7497=int vmx_vcpu_create(struct kvm_vcpu *vcpu)\n--\narch/x86/kvm/vmx/vmx.c-7509-\narch/x86/kvm/vmx/vmx.c:7510:\tvmx-\u003evpid = allocate_vpid();\narch/x86/kvm/vmx/vmx.c-7511-\n--\narch/x86/kvm/vmx/vmx.h=305=static __always_inline void destroy_vpids(void)\n--\narch/x86/kvm/vmx/vmx.h-310-\narch/x86/kvm/vmx/vmx.h:311:static __always_inline kvm_tlb_tag_t allocate_vpid(void)\narch/x86/kvm/vmx/vmx.h-312-{\n"
} |
| 15/2 | 2026/10/12 23:50 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 42560 output: 36 thoughts: 674 Thoughts: **Analyzing KVM's TLB Flushing and MMU Synchronization** Okay, let's break this down. My initial observation is that in the VMX code path, `vmx_vcpu_create` and `enter_vmx_operation` use `allocate_vpid`, and `free_vpid` is correspondingly called in `vmx_vcpu_free` and `free_nested`. That's a basic VMX lifecycle. Now, let's dive into `arch/x86/kvm/x86.c`. I need to understand the TLB flush mechanism and how the MMU is synchronized, particularly the path `kvm_vcpu_flush_tlb_guest`. It calls `kvm_mmu_sync_all_roots`, which, as a static inline, ultimately calls `kvm_mmu_sync_roots` and `kvm_mmu_sync_prev_roots`. This makes sense for a complete flush operation. Looking further, `vcpu_enter_guest` checks for `KVM_REQ_MMU_SYNC_ALL_ROOTS` and `KVM_REQ_MMU_SYNC`. When present, they trigger the relevant MMU synchronization functions. Now, where is `KVM_REQ_MMU_SYNC_ALL_ROOTS` actually requested? Aha, it's requested within `nested_svm_entry_tlb_flush`. This function looks critical. The condition for requesting `KVM_REQ_MMU_SYNC_ALL_ROOTS` depends on `new_asid` and `svm->nested.ctl.tlb_ctl`. The request is conditionally called if nested NPT is enabled. This makes sense from a nested virtualization perspective. `nested_svm_entry_tlb_flush` itself is static, so I want to understand its scope. It's called from `nested_vmcb02_prepare_control`. `nested_vmcb02_prepare_control` in turn is called from `enter_svm_guest_mode`, and that function is called by both `nested_svm_vmrun` and `svm_set_nested_state`. So, the TLB flush mechanism is directly tied to the SVM entry path within nested virtualization. This is the complete flow. |
| 16/2 | 2026/10/12 23:50 | tool | grepper |
0mArgs:{
"Expression": "enter_svm_guest_mode",
"PathPrefixes": [
"arch/x86/kvm"
]
}Results: {
"Output": "arch/x86/kvm/svm/nested.c=1079=static void nested_svm_service_local_tlb_flushes(struct kvm_vcpu *vcpu)\n--\narch/x86/kvm/svm/nested.c-1092-\narch/x86/kvm/svm/nested.c:1093:int enter_svm_guest_mode(struct kvm_vcpu *vcpu, u64 vmcb12_gpa, bool from_vmrun)\narch/x86/kvm/svm/nested.c-1094-{\n--\narch/x86/kvm/svm/nested.c=1175=int nested_svm_vmrun(struct kvm_vcpu *vcpu)\n--\narch/x86/kvm/svm/nested.c-1240-\narch/x86/kvm/svm/nested.c:1241:\tif (enter_svm_guest_mode(vcpu, vmcb12_gpa, true) ||\narch/x86/kvm/svm/nested.c-1242-\t !nested_svm_merge_msrpm(vcpu)) {\n--\narch/x86/kvm/svm/svm.c=5219=static int svm_leave_smm(struct kvm_vcpu *vcpu, const union kvm_smram *smram)\n--\narch/x86/kvm/svm/svm.c-5269-\narch/x86/kvm/svm/svm.c:5270:\tif (enter_svm_guest_mode(vcpu, smram64-\u003esvm_guest_vmcb_gpa, false) != 0)\narch/x86/kvm/svm/svm.c-5271-\t\treturn 1;\n--\narch/x86/kvm/svm/svm.h=903=int __init nested_svm_init_msrpm_merge_offsets(void);\narch/x86/kvm/svm/svm.h-904-\narch/x86/kvm/svm/svm.h:905:int enter_svm_guest_mode(struct kvm_vcpu *vcpu, u64 vmcb_gpa, bool from_vmrun);\narch/x86/kvm/svm/svm.h-906-void svm_leave_nested(struct kvm_vcpu *vcpu);\n"
} |
| 17/2 | 2026/10/12 23:50 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 44382 output: 37 thoughts: 4110 Thoughts: **Focus Symbol Selection for KVM/SVM TLB Flush and Nested SVM Implementation** Okay, so I've been diving into this KVM code, specifically the AMD Secure Virtual Machine (SVM) and its nested virtualization features. My goal is to pinpoint the most effective focus symbols for fuzzing this area. The main changes appear to revolve around TLB flushes, ASID management, and nested virtualization state transitions. I've been tracing the execution path, examining the core functions, and making some decisions based on the guidance provided, and as an expert, I'll provide an explanation of my thought process. Firstly, I identified the key functions in `svm/nested.c` like `nested_svm_vmrun`, `nested_svm_vmexit`, `svm_allocate_nested`, `svm_free_nested`, `svm_leave_nested`, and `svm_set_nested_state`. These are all non-static functions, and therefore are reasonable choices for symbols because they directly handle nested SVM entry and exit logic, VM state management, and the crucial interactions between nested virtualization levels. `enter_svm_guest_mode` also is essential since it deals with entry to guest mode. Then, the focus shifted to `svm/svm.c` and its handling of TLB flushes and interceptions. I examined `invlpga_interception` and found it handles `invlpga` instruction execution, with hardware-level `invlpga` calls. Also `pre_svm_run` is important to examine the state prior to a VMRUN and `svm_flush_tlb_all` to check flush operations. I looked at `svm_flush_tlb_asid` for potential symbolization, which may be beneficial if we want more granular control over the TLB flush operations. `svm_vcpu_create` and `svm_vcpu_free` were also marked for the same reason. The introduction of TLB tagging required digging into `mmu/mmu.c`, which includes core functions `kvm_alloc_tlb_tag`, `kvm_free_tlb_tag`, `kvm_init_tlb_tags`, and `kvm_destroy_tlb_tags`, which are critical for the new TLB tag allocation infrastructure. These are good candidates as they deal directly with the core logic. I specifically looked at `nested_svm_entry_tlb_flush` and `nested_svm_exit_tlb_flush` because they are not static functions; however, because they are inside nested.c, their callers such as `enter_svm_guest_mode`, `nested_svm_vmrun`, `nested_svm_vmexit` and `svm_leave_nested` are probably sufficient. I looked at the kernel config and it seems the following configs are necessary: - `KVM` (or `KVM_X86`) - `KVM_AMD` (for SVM and nested SVM) - `KVM_INTEL` (for VMX VPID tag allocator) |
| 18/2 | 2026/10/12 23:51 | tool | grepper |
0mArgs:{
"Expression": "config KVM_AMD",
"PathPrefixes": [
"arch/x86/kvm/Kconfig"
]
}Results: {
"Output": "arch/x86/kvm/Kconfig=145=config KVM_INTEL_TDX\n--\narch/x86/kvm/Kconfig-155-\narch/x86/kvm/Kconfig:156:config KVM_AMD\narch/x86/kvm/Kconfig-157-\ttristate \"KVM for AMD processors support\"\n--\narch/x86/kvm/Kconfig-165-\narch/x86/kvm/Kconfig:166:config KVM_AMD_SEV\narch/x86/kvm/Kconfig-167-\tbool \"AMD Secure Encrypted Virtualization (SEV) support\"\n"
} |
| 19/2 | 2026/10/12 23:51 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 49364 output: 39 thoughts: 31 Thoughts: Okay, here's my take on those lines of thought, geared towards a fellow expert: **Delving into KVM's X86 Configuration Dependencies** Alright, before I dive too deep into this particular KVM issue, I need to get a clear picture of the full dependency web. I'm going to start by going straight to the source – let's pull up the `arch/x86/kvm/Kconfig` file, specifically lines 150 through 180. That chunk should give me the precise dependencies for the relevant feature or module I'm interested in. I want to see the exact conditionals, the relationships, and what other config options are gating this. This should allow me to grasp how to properly configure the module. No sense in spinning wheels when the config itself lays out the map. Understanding these dependencies is key for a truly in-depth diagnosis of the problem at hand, or for crafting the best possible solution. I'm expecting this section to illuminate the relationships and help me cut through any potential noise, so I can keep my focus on the critical factors. |
| 20/2 | 2026/10/12 23:51 | tool | read-file |
0mArgs:{
"File": "arch/x86/kvm/Kconfig",
"FirstLine": 150,
"LineCount": 35
}Results: {
"Contents": " 150:\t\thelp\n 151:\t\t Provides support for launching Intel Trust Domain Extensions (TDX)\n 152:\t\t confidential VMs on Intel processors.\n 153:\t\n 154:\t\t If unsure, say N.\n 155:\t\n 156:\tconfig KVM_AMD\n 157:\t\ttristate \"KVM for AMD processors support\"\n 158:\t\tdepends on KVM \u0026\u0026 (CPU_SUP_AMD || CPU_SUP_HYGON)\n 159:\t\thelp\n 160:\t\t Provides support for KVM on AMD processors equipped with the AMD-V\n 161:\t\t (SVM) extensions.\n 162:\t\n 163:\t\t To compile this as a module, choose M here: the module\n 164:\t\t will be called kvm-amd.\n 165:\t\n 166:\tconfig KVM_AMD_SEV\n 167:\t\tbool \"AMD Secure Encrypted Virtualization (SEV) support\"\n 168:\t\tdefault y\n 169:\t\tdepends on KVM_AMD \u0026\u0026 X86_64\n 170:\t\tdepends on CRYPTO_DEV_SP_PSP \u0026\u0026 !(KVM_AMD=y \u0026\u0026 CRYPTO_DEV_CCP_DD=m)\n 171:\t\tselect ARCH_HAS_CC_PLATFORM\n 172:\t\tselect HAVE_KVM_ARCH_GMEM_CONVERT\n 173:\t\tselect HAVE_KVM_ARCH_GMEM_RECLAIM\n 174:\t\tselect HAVE_KVM_ARCH_GMEM_INVALIDATE\n 175:\t\tselect HAVE_KVM_ARCH_GMEM_POPULATE\n 176:\t\thelp\n 177:\t\t Provides support for launching encrypted VMs which use Secure\n 178:\t\t Encrypted Virtualization (SEV), Secure Encrypted Virtualization with\n 179:\t\t Encrypted State (SEV-ES), and Secure Encrypted Virtualization with\n 180:\t\t Secure Nested Paging (SEV-SNP) technologies on AMD processors.\n 181:\t\n 182:\tconfig KVM_IOAPIC\n 183:\t\tbool \"I/O APIC, PIC, and PIT emulation\"\n 184:\t\tdefault y\n"
} |
| 21/2 | 2026/10/12 23:51 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 50244 output: 39 thoughts: 13 Thoughts: Okay, here's my summary, tailored for an expert audience, as if it's *my* train of thought: **Checking a Specific Configuration Item** Right, let's take a quick look at `config KVM_INTEL`. It's crucial to confirm this setting. With my level of experience, I recognize its significance immediately. We're talking about hardware virtualization support specifically for Intel processors. Given the context, ensuring this is enabled is a fundamental starting point. We can't proceed without it. |
| 22/2 | 2026/10/12 23:51 | tool | read-file |
0mArgs:{
"File": "arch/x86/kvm/Kconfig",
"FirstLine": 100,
"LineCount": 25
}Results: {
"Contents": " 100:\t\t Enable support for KVM software-protected VMs. Currently, software-\n 101:\t\t protected VMs are purely a development and testing vehicle for\n 102:\t\t KVM_CREATE_GUEST_MEMFD. Attempting to run a \"real\" VM workload as a\n 103:\t\t software-protected VM will fail miserably.\n 104:\t\n 105:\t\t If unsure, say \"N\".\n 106:\t\n 107:\tconfig KVM_INTEL\n 108:\t\ttristate \"KVM for Intel (and compatible) processors support\"\n 109:\t\tdepends on KVM \u0026\u0026 IA32_FEAT_CTL\n 110:\t\tselect X86_FRED if X86_64\n 111:\t\thelp\n 112:\t\t Provides support for KVM on processors equipped with Intel's VT\n 113:\t\t extensions, a.k.a. Virtual Machine Extensions (VMX).\n 114:\t\n 115:\t\t To compile this as a module, choose M here: the module\n 116:\t\t will be called kvm-intel.\n 117:\t\n 118:\tconfig KVM_INTEL_PROVE_VE\n 119:\t bool \"Check that guests do not receive #VE exceptions\"\n 120:\t depends on KVM_INTEL \u0026\u0026 EXPERT\n 121:\t help\n 122:\t Checks that KVM's page table management code will not incorrectly\n 123:\t let guests receive a virtualization exception. Virtualization\n 124:\t exceptions will be trapped by the hypervisor rather than injected\n"
} |
| 23/2 | 2026/10/12 23:51 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 50838 output: 38 thoughts: 9 Thoughts: Okay, here's my take on those thoughts, tailored for an expert audience, and written as if they were my own: **Initial Assessment: KVM Configuration Check** Alright, time to dig in. The immediate prompt – "Let's check `config KVM`" – triggers a specific mental checklist. It's a shorthand, really, a pointer to a set of core considerations when dealing with a KVM environment. I'm already anticipating the common pitfalls and areas that need careful scrutiny. I'm probably thinking about the potential aspects of the KVM configuration, like: the kernel modules and their versions, nested virtualization state, virtual network setup (bridges, VLANs, etc.), resource allocation (CPU, memory, storage), security configurations (SELinux or AppArmor), and virtual machine definitions. There's a subtle implicit "I know what I'm looking for" built into that command; it speaks volumes. It’s not just a blind command execution, it's a diagnostic starting point. I'm ready to interpret the output in context, based on the assumed deployment scenario. The goal is to quickly ascertain the health and integrity of this particular KVM instance, and ensure everything is configured as expected, or to identify deviations that warrant further investigation. |
| 24/2 | 2026/10/12 23:51 | tool | read-file |
0mArgs:{
"File": "arch/x86/kvm/Kconfig",
"FirstLine": 45,
"LineCount": 25
}Results: {
"Contents": " 45:\t\tselect KVM_VFIO\n 46:\t\tselect HAVE_KVM_PM_NOTIFIER if PM\n 47:\t\tselect KVM_GENERIC_HARDWARE_ENABLING\n 48:\t\tselect KVM_GENERIC_PRE_FAULT_MEMORY\n 49:\t\tselect KVM_WERROR if WERROR\n 50:\t\tselect KVM_GUEST_MEMFD if X86_64\n 51:\t\n 52:\tconfig KVM\n 53:\t\ttristate \"Kernel-based Virtual Machine (KVM) support\"\n 54:\t\tdepends on X86_LOCAL_APIC\n 55:\t\thelp\n 56:\t\t Support hosting fully virtualized guest machines using hardware\n 57:\t\t virtualization extensions. You will need a fairly recent\n 58:\t\t processor equipped with virtualization extensions. You will also\n 59:\t\t need to select one or more of the processor modules below.\n 60:\t\n 61:\t\t This module provides access to the hardware capabilities through\n 62:\t\t a character device node named /dev/kvm.\n 63:\t\n 64:\t\t To compile this as a module, choose M here: the module\n 65:\t\t will be called kvm.\n 66:\t\n 67:\t\t If unsure, say N.\n 68:\t\n 69:\tconfig KVM_WERROR\n"
} |
| 25/2 | 2026/10/12 23:51 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 51509 output: 308 thoughts: 2150 Thoughts: **My Analysis of KVM Focus Symbols and Configuration** Okay, so I've been tasked with evaluating a KVM patch series for fuzzing focus and configuration. My initial thought is to make sure we're hitting all the relevant code paths and that the fuzzing effort is targeted effectively. First, let's get the configurations correct. We have `KVM`, `KVM_AMD`, and `KVM_INTEL` as primary contenders. I need to quickly determine if `KVM_HYPERV` is essential, even though it defaults to 'y' when `KVM_X86` is enabled, but it's important to include it in case it's relevant given the changes regarding `kvm_hv_purge_tlb_flush_fifo` and `kvm_hv_purge_all_tlb_flush_fifos`, and `KVM_AMD_SEV` is for SEV ASID migration. So, based on this review, I'm confident in the `EnableConfigs`: `["KVM", "KVM_AMD", "KVM_INTEL"]`. Now, let's dive into the core of the matter: the FocusSymbols. I'm going to review the potential list against the defined guidelines, which are pretty clear: avoid ubiquitous lifecycle functions, target specific feature logic, and handle static inline functions correctly. Here's my current thought process regarding each function: 1. `kvm_alloc_tlb_tag`, `kvm_free_tlb_tag`, and `kvm_init_tlb_tags`: These are all part of the newly introduced TLB tag bitmap allocator. Seems like a good target. 2. `invlpga_interception`: Non-static function, important for handling INVLPGA instructions and ASID validation. Sounds like a good target. 3. `nested_svm_vmrun`, `nested_svm_vmexit`, `enter_svm_guest_mode`, `svm_allocate_nested`, `svm_free_nested`, `svm_leave_nested`, `svm_set_nested_state`: These all appear to handle nested virtualization, ASID allocation, and TLB flushes. Definitely important. 4. `svm_flush_tlb_all`: This one was modified to purge all HV FIFOs and mark a flush for both vmcb01 and vmcb02. Seems relevant. I need to confirm if static inline functions are addressed. Upon review, `nested_svm_entry_tlb_flush` and `nested_svm_exit_tlb_flush` are indeed static within `nested.c`. Since the compiler *could* inline them, targeting the caller functions (`enter_svm_guest_mode`, `nested_svm_vmrun`, `nested_svm_vmexit`, `svm_leave_nested`) directly hits the guideline. Looking closer at `svm_vcpu_create` and `svm_vcpu_free`, while `static`, they're callbacks for the vCPU lifecycle related to `asid` allocation. Seems like a legitimate feature-specific entry point in SVM, assigned to `kvm_x86_ops.vcpu_create` and `vcpu_free`. Important to add. Lastly, `pre_svm_run` (also `static`): called from `svm_vcpu_run`. This is also a valid target. Alright, time for a final check to ensure these symbols are *not* ubiquitous. After a quick scan, none of the functions, are ubiquitous, making them ideal targets. They are all specific to KVM SVM nested virtualization, ASID tracking, and TLB tagging, and should be reachable. They should be reachable in standard virtualized environments running KVM with QEMU with nested virtualization or standard x86 KVM fuzzing. My analysis of the WorthFuzzing criterion is a resounding YES! The patch set is heavily refactoring ASID allocation/tracking in KVM SVM, VPID allocation in VMX, adds a new TLB tag allocator in the MMU, modifies nested SVM entry/exit TLB flush logic, adds runtime assertions, alters INVLPGA, and introduces `KVM_REQ_MMU_SYNC_ALL_ROOTS`. This is all executable kernel code, reachable via `/dev/kvm` ioctls and guest execution. Here's my distilled reasoning: 1. **Patch Functionality:** This patch series introduces a centralized TLB tag allocator, overhauled SVM ASID lifecycle management, refactored nested SVM TLB invalidation, changed how `invlpga_interception` works, and added runtime assertions. 2. **Reachability:** This code modifies KVM MMU, SVM, and VMX code that are reachable from users |
| 26/2 | 2026/10/12 23:51 | tool | set-results |
0mArgs:{
"EnableConfigs": [
"KVM",
"KVM_AMD",
"KVM_INTEL"
],
"FocusSymbols": [
"kvm_alloc_tlb_tag",
"kvm_free_tlb_tag",
"invlpga_interception",
"enter_svm_guest_mode",
"nested_svm_vmrun",
"nested_svm_vmexit",
"svm_allocate_nested",
"svm_free_nested",
"svm_leave_nested",
"svm_set_nested_state",
"svm_flush_tlb_all"
],
"Reasoning": "The patch series significantly reworks ASID and VPID management across KVM (x86 MMU, SVM, and VMX). It introduces a centralized TLB tag allocator in kvm/mmu/mmu.c, replaces the generation-based SVM ASID allocator with explicit tag tracking and fallback ASID handling, completely refactors nested SVM TLB flushes on guest mode transitions, reworks INVLPGA instruction interception emulation, and introduces the KVM_REQ_MMU_SYNC_ALL_ROOTS request. It also adds new invariant assertions (WARN_ON_ONCE, KVM_BUG_ON). This code is fully reachable in virtualized environments via /dev/kvm ioctls and nested virtualization execution.",
"WorthFuzzing": true
}Results: {
"EnableConfigs": [
"KVM",
"KVM_AMD",
"KVM_INTEL"
],
"FocusSymbols": [
"kvm_alloc_tlb_tag",
"kvm_free_tlb_tag",
"invlpga_interception",
"enter_svm_guest_mode",
"nested_svm_vmrun",
"nested_svm_vmexit",
"svm_allocate_nested",
"svm_free_nested",
"svm_leave_nested",
"svm_set_nested_state",
"svm_flush_tlb_all"
],
"Reasoning": "The patch series significantly reworks ASID and VPID management across KVM (x86 MMU, SVM, and VMX). It introduces a centralized TLB tag allocator in kvm/mmu/mmu.c, replaces the generation-based SVM ASID allocator with explicit tag tracking and fallback ASID handling, completely refactors nested SVM TLB flushes on guest mode transitions, reworks INVLPGA instruction interception emulation, and introduces the KVM_REQ_MMU_SYNC_ALL_ROOTS request. It also adds new invariant assertions (WARN_ON_ONCE, KVM_BUG_ON). This code is fully reachable in virtualized environments via /dev/kvm ioctls and nested virtualization execution.",
"WorthFuzzing": true
} |
| 27/1 | 2026/10/12 23:51 | action | If |
1mArgs:{
"WorthFuzzing": true
}Results: null |
| 28/2 | 2026/10/12 23:51 | agent | kmsan-evaluator |
1mModel:core Args: null Results: {
"KMSANReasoning": "The patch series refactors KVM's TLB tag (VPID/ASID) allocation and updates nested SVM TLB flushing and ASID management:\n1. It replaces the separate VMX VPID bitmap allocator with a generic TLB tag allocator (`kvm_init_tlb_tags`, `kvm_alloc_tlb_tag`, `kvm_free_tlb_tag`), using `bitmap_zalloc()` which guarantees zero-initialized allocations.\n2. It tracks nested ASIDs and flushes on SVM (`asid02`, `last_asid`), explicitly initializing `last_asid = 0` and properly allocating/freeing ASIDs. The parent `struct vcpu_svm` is allocated via `kmem_cache_zalloc()`.\n3. No new kernel-to-userspace copies or information leak vectors are introduced. The internal fields (`asid02`, `last_asid`, `fallback_asid`) are internal to KVM and not copied to userspace via `KVM_GET_NESTED_STATE` or any other ioctl.\n4. Surrounding buffer accesses, bounds checks, and data length calculations are unaffected.\nAny potential risks introduced by this patch series (such as bitmap indexing errors, concurrency/locking races, use-after-free or double-free of tags) are detected by KASAN, LOCKDEP, and built-in WARN_ON_ONCE checks. There are no uninitialized memory risks that would uniquely warrant KMSAN fuzzing.",
"NeedsKMSAN": false
}Instruction: You are an expert Linux kernel security engineer specializing in kernel memory
error detectors (KASAN and KMSAN). Your job is to review the provided patch series and
determine if the code changes justify spawning a dedicated KMSAN (KernelMemorySanitizer)
fuzzing session in addition to standard KASAN fuzzing.
CRITICAL DISTINCTION BETWEEN KASAN AND KMSAN:
- Standard KASAN kernel builds (upstream-apparmor-kasan.config) already enable
a comprehensive suite of debugging tools and sanitizers, including KASAN
(out-of-bounds accesses, use-after-free, double free, invalid free), LOCKDEP
(locking bugs and deadlocks), UB-sanitizers, and memory corruption checks.
- KMSAN (KernelMemorySanitizer) detects reads of UNINITIALIZED memory (stack, heap,
or page allocations) and kernel-to-user memory info-leaks.
Rule: THERE IS NO SENSE IN RUNNING A KMSAN SESSION IF A BUG CAN BE CAUGHT BY KASAN,
LOCKDEP, OR OTHER STANDARD BUG DETECTORS.
A dedicated KMSAN fuzzing session incurs significant resource costs. You must ONLY
set NeedsKMSAN=true if the code changes introduce or expose UNINITIALIZED MEMORY risks
that are detected ONLY by KMSAN.
Look holistically at the patch series and surrounding code. Even if no direct
uninitialized field accesses or new buffer allocations are added in the diff itself,
a patch may alter control flow, bounds checking, or data length calculations in ways
that change how the rest of the code operates on existing buffers (e.g. allowing
uninitialized stack/heap memory to be read, copied to user space, or used in control
flow). Do not hesitate to use your code access tools to inspect the surrounding code,
called functions, and callers.
Set NeedsKMSAN=true ONLY IF the patch introduces or modifies:
1. Kernel structures sent to user space (via copy_to_user, put_user, netlink skb
attributes, ioctl output arguments, socket options, or BPF buffers) where fields
or structure padding might not be fully initialized/zeroed.
2. Conditional logic or branching that depends on potentially uninitialized variables
or struct fields.
3. Allocation or initialization of complex data structures where uninitialized fields
could be read later in reachable code paths.
4. Bounds checks, lengths, or logic in a way that allows surrounding code to access
uninitialized bytes of existing buffers.
Set NeedsKMSAN=false IF:
- The code changes primarily risk out-of-bounds access, array overflows, NULL pointer
dereferences, locking deadlocks, or use-after-free bugs (these are already caught
by KASAN, LOCKDEP, or standard bug detectors).
- All stack/heap structures touched or introduced by the patch are fully zeroed
or initialized (e.g. using = {0}, memset, kzalloc) before being read or copied.
- The patch does not introduce any risk of uninitialized memory usage or info-leaks.
Use your code access tools to inspect the surrounding code if necessary, then provide
detailed KMSANReasoning contrasting KASAN vs KMSAN applicability for this patch.
Prefer calling several tools at the same time to save round-trips.
Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.
Prompt: Target architecture: amd64
For your convenience, here is the diff of the changes:
commit 21e80c95a8d3388e9dc50d771315b61b19202b62
Author: syz-cluster <triage@syzkaller.com>
Date: Mon Oct 12 23:48:32 2026 +0000
syz-cluster: applied patch under review
diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index ddc61ea36ac59..60f49b3675920 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -9770,6 +9770,26 @@ the APIC ID is found in EDX for all subleaves of 0x0b and 0x1f, and in EAX
for 0x8000001e; the latter also encodes the core id and node id in bits
7:0 of EBX and ECX respectively.
+Number of ASIDs (nested SVM)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+CPUID[0x8000000A]:EBX reports the number of ASIDs supported by the processor
+on AMD CPUs. KVM's ``KVM_GET_SUPPORTED_CPUID`` hardcodes this value to 8 for
+historical reasons. This value is arbitrary and does not represent a limit
+on the supported number of ASIDs, or even a reasonable default value.
+
+KVM allows userspace to set any value (up to 0xFFFFFFFF), and emulates those
+ASIDs by multiplexing them onto hardware ASIDs. AMD CPUs have only ever
+supported 3 values: 0x40, 0x8000, and 0x1000 -- with modern CPUs advertising
+0x8000.
+
+Advertising a small number of ASIDs could lead an L1 hypervisor to perform
+unnecessary TLB flushes (e.g. older KVM would flush the entire TLB when running
+out of ASIDs). Userspace should probably set the number of ASIDs to a
+reasonably high power of 2 value to remain consistent with hardware and avoid
+any misbehavior by L1 hypervisors, ideally one of the values advertised by
+actual hardware.
+
Obsolete ioctls and capabilities
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index e2eef7057bd35..1d28bb7e92c7f 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -127,6 +127,7 @@
KVM_ARCH_REQ_FLAGS(33, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)
#define KVM_REQ_UPDATE_PROTECTED_GUEST_STATE \
KVM_ARCH_REQ_FLAGS(34, KVM_REQUEST_WAIT)
+#define KVM_REQ_MMU_SYNC_ALL_ROOTS KVM_ARCH_REQ(35)
#define INVALID_PAGE (~(hpa_t)0)
#define VALID_PAGE(x) ((x) != INVALID_PAGE)
diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c
index adafd6fa2c542..e45c01d6faeaf 100644
--- a/arch/x86/kvm/cpuid.c
+++ b/arch/x86/kvm/cpuid.c
@@ -1846,8 +1846,8 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function)
break;
}
entry->eax = 1; /* SVM revision 1 */
- entry->ebx = 8; /* Lets support 8 ASIDs in case we add proper
- ASID emulation to nested SVM */
+ entry->ebx = 8; /* Arbitrary value for historical reasons. KVM
+ practically allows and ignores all values. */
entry->ecx = 0; /* Reserved */
cpuid_entry_override(entry, CPUID_8000_000A_EDX);
break;
diff --git a/arch/x86/kvm/hyperv.h b/arch/x86/kvm/hyperv.h
index 622a6553e9ac2..b82f70e6fcea1 100644
--- a/arch/x86/kvm/hyperv.h
+++ b/arch/x86/kvm/hyperv.h
@@ -312,18 +312,34 @@ static inline struct kvm_vcpu_hv_tlb_flush_fifo *kvm_hv_get_tlb_flush_fifo(struc
return &hv_vcpu->tlb_flush_fifo[i];
}
-static inline void kvm_hv_vcpu_purge_flush_tlb(struct kvm_vcpu *vcpu)
+/* Purge pending TLB flushes in @fifo, or in all FIFOs if @fifo is NULL */
+static inline void __kvm_hv_purge_tlb_flush_fifo(struct kvm_vcpu *vcpu,
+ struct kvm_vcpu_hv_tlb_flush_fifo *fifo)
{
- struct kvm_vcpu_hv_tlb_flush_fifo *tlb_flush_fifo;
+ struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu);
+ int i;
- if (!kvm_check_request(KVM_REQ_HV_TLB_FLUSH, vcpu))
+ if (!hv_vcpu || !kvm_check_request(KVM_REQ_HV_TLB_FLUSH, vcpu))
return;
- tlb_flush_fifo = kvm_hv_get_tlb_flush_fifo(vcpu, is_guest_mode(vcpu));
- if (!tlb_flush_fifo)
- return;
+ for (i = 0; i < ARRAY_SIZE(hv_vcpu->tlb_flush_fifo); i++) {
+ if (!fifo || fifo == &hv_vcpu->tlb_flush_fifo[i])
+ kfifo_reset_out(&hv_vcpu->tlb_flush_fifo[i].entries);
+ }
+}
+
+static inline void kvm_hv_purge_tlb_flush_fifo(struct kvm_vcpu *vcpu)
+{
+ struct kvm_vcpu_hv_tlb_flush_fifo *fifo;
+
+ fifo = kvm_hv_get_tlb_flush_fifo(vcpu, is_guest_mode(vcpu));
+ if (fifo)
+ __kvm_hv_purge_tlb_flush_fifo(vcpu, fifo);
+}
- kfifo_reset_out(&tlb_flush_fifo->entries);
+static inline void kvm_hv_purge_all_tlb_flush_fifos(struct kvm_vcpu *vcpu)
+{
+ __kvm_hv_purge_tlb_flush_fifo(vcpu, NULL);
}
static inline bool guest_hv_cpuid_has_l2_tlb_flush(struct kvm_vcpu *vcpu)
@@ -396,7 +412,8 @@ static inline int kvm_hv_hypercall(struct kvm_vcpu *vcpu)
{
return HV_STATUS_ACCESS_DENIED;
}
-static inline void kvm_hv_vcpu_purge_flush_tlb(struct kvm_vcpu *vcpu) {}
+static inline void kvm_hv_purge_tlb_flush_fifo(struct kvm_vcpu *vcpu) {}
+static inline void kvm_hv_purge_all_tlb_flush_fifos(struct kvm_vcpu *vcpu) {}
static inline bool kvm_hv_synic_has_vector(struct kvm_vcpu *vcpu, int vector)
{
return false;
diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h
index 2ae7f9ed4cf86..b6ad0ff72393e 100644
--- a/arch/x86/kvm/mmu.h
+++ b/arch/x86/kvm/mmu.h
@@ -170,6 +170,13 @@ void kvm_mmu_unload(struct kvm_vcpu *vcpu);
void kvm_mmu_free_obsolete_roots(struct kvm_vcpu *vcpu);
void kvm_mmu_sync_roots(struct kvm_vcpu *vcpu);
void kvm_mmu_sync_prev_roots(struct kvm_vcpu *vcpu);
+
+static inline void kvm_mmu_sync_all_roots(struct kvm_vcpu *vcpu)
+{
+ kvm_mmu_sync_roots(vcpu);
+ kvm_mmu_sync_prev_roots(vcpu);
+}
+
void kvm_mmu_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new,
int bytes);
@@ -410,4 +417,12 @@ static inline bool kvm_is_gfn_alias(struct kvm *kvm, gfn_t gfn)
{
return gfn & kvm_gfn_direct_bits(kvm);
}
+
+typedef unsigned int kvm_tlb_tag_t;
+
+int kvm_init_tlb_tags(unsigned int nr_total, unsigned int nr_reserved);
+void kvm_destroy_tlb_tags(void);
+kvm_tlb_tag_t kvm_alloc_tlb_tag(void);
+void kvm_free_tlb_tag(kvm_tlb_tag_t tag);
+
#endif
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index aadd5bdc2c97a..c096c1a2b4684 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -8178,6 +8178,98 @@ void kvm_mmu_pre_destroy_vm(struct kvm *kvm)
vhost_task_stop(kvm->arch.nx_huge_page_recovery_thread);
}
+static struct {
+ spinlock_t lock;
+ unsigned long *bitmap;
+ unsigned int nr;
+ kvm_tlb_tag_t start;
+} tlb_tags;
+
+/*
+ * Initialize @nr_total TLB tags, where @nr_reserved of those tags are reserved.
+ * Reserved tags start at 0 and are contiguous. Hence, usable tags are
+ * [nr_reserved, nr_total). At least one reserved tag is required, as 0 is the
+ * host TLB tag on both VMX and AMD, and is the value returned on a failed tag
+ * allocation.
+ */
+int kvm_init_tlb_tags(unsigned int nr_total, unsigned int nr_reserved)
+{
+ /*
+ * Limit the number of TLB tags to VMX's hardcoded maximum of 0x10000
+ * to avoid wasting memory for the bitmap in the unlikely scenario the
+ * CPU supports an inordinate number of ASIDs (on AMD). If userspace
+ * wants to concurrently run tens of thousands of vCPUs, they'll likely
+ * need a solution that works for both Intel and AMD.
+ */
+ const unsigned int MAX_NR_TLB_TAGS = VMX_NR_VPIDS;
+ unsigned int nr;
+
+ if (WARN_ON_ONCE(!nr_reserved || nr_reserved > nr_total))
+ return -EINVAL;
+
+ nr = nr_total - nr_reserved;
+ if (!nr)
+ return 0;
+
+ if (nr > MAX_NR_TLB_TAGS) {
+ pr_warn_once("Number of TLB tags capped (%u instead of %u)\n",
+ MAX_NR_TLB_TAGS, nr);
+ nr = MAX_NR_TLB_TAGS;
+ }
+
+ tlb_tags.bitmap = bitmap_zalloc(nr, GFP_KERNEL);
+ if (!tlb_tags.bitmap)
+ return -ENOMEM;
+
+ tlb_tags.nr = nr;
+ tlb_tags.start = nr_reserved;
+ spin_lock_init(&tlb_tags.lock);
+ return 0;
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_init_tlb_tags);
+
+void kvm_destroy_tlb_tags(void)
+{
+ bitmap_free(tlb_tags.bitmap);
+ tlb_tags.bitmap = NULL;
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_destroy_tlb_tags);
+
+kvm_tlb_tag_t kvm_alloc_tlb_tag(void)
+{
+ unsigned int bit;
+
+ if (!tlb_tags.bitmap)
+ return 0;
+
+ guard(spinlock)(&tlb_tags.lock);
+
+ bit = find_first_zero_bit(tlb_tags.bitmap, tlb_tags.nr);
+ if (bit >= tlb_tags.nr)
+ return 0;
+
+ __set_bit(bit, tlb_tags.bitmap);
+ return tlb_tags.start + bit;
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_alloc_tlb_tag);
+
+void kvm_free_tlb_tag(kvm_tlb_tag_t tag)
+{
+ unsigned int bit;
+
+ if (!tag || WARN_ON_ONCE(tag < tlb_tags.start))
+ return;
+
+ bit = tag - tlb_tags.start;
+ if (WARN_ON_ONCE(bit >= tlb_tags.nr))
+ return;
+
+ guard(spinlock)(&tlb_tags.lock);
+
+ __clear_bit(bit, tlb_tags.bitmap);
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_free_tlb_tag);
+
#ifdef CONFIG_KVM_VM_MEMORY_ATTRIBUTES
static bool hugepage_test_mixed(struct kvm_memory_slot *slot, gfn_t gfn,
int level)
diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index af0ef678433d7..6fe68f2ae282d 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -556,10 +556,8 @@ void __nested_copy_vmcb_control_to_cache(struct kvm_vcpu *vcpu,
to->nested_cr3 = from->nested_cr3;
to->pause_filter_count = from->pause_filter_count;
to->pause_filter_thresh = from->pause_filter_thresh;
-
- /* Copy asid here because nested_vmcb_check_controls() will check it */
- to->asid = from->asid;
- to->clean = from->clean;
+ to->asid = from->asid;
+ to->clean = from->clean;
#ifdef CONFIG_KVM_HYPERV
/* Hyper-V extensions (Enlightened VMCB) */
@@ -688,26 +686,69 @@ static void nested_save_pending_event_to_vmcb12(struct vcpu_svm *svm,
vmcb12->control.exit_int_info = exit_int_info;
}
-static void nested_svm_transition_tlb_flush(struct kvm_vcpu *vcpu)
+static void nested_svm_entry_tlb_flush(struct kvm_vcpu *vcpu)
{
+ struct vcpu_svm *svm = to_svm(vcpu);
+ bool new_asid = false;
+
/* Handle pending Hyper-V TLB flush requests */
kvm_hv_nested_transtion_tlb_flush(vcpu, npt_enabled);
+ if (svm->nested.ctl.asid != svm->nested.last_asid) {
+ svm->nested.last_asid = svm->nested.ctl.asid;
+ new_asid = true;
+ }
+
/*
- * TODO: optimize unconditional TLB flush/MMU sync. A partial list of
- * things to fix before this can be conditional:
+ * If L1 requested a TLB flush for L2, flush L2's TLB on nested entry
+ * and sync the nested NPT MMU, as TLB_CONTROL also flushes NPT
+ * guest-physical mappings. Sync all roots, since the TLB flush is not
+ * specific to the currently loaded nCR3.
+ *
+ * Handle L1 changing L2's ASID12 similarly, as KVM only uses one ASID
+ * for L2 in hardware (per vCPU), so it must start fresh when L1 changes
+ * ASID12 to emulate different ASIDs correctly. Additionally, the MMU
+ * context is not tagged by the ASID, so the shadow NPTs cannot be
+ * reused across different L2 ASIDs.
+ *
+ * Note, last_asid is initialized as 0, so the first nested VM-Enter
+ * after setting EFER.SVME will always flush the TLB to avoid using
+ * stale entries.
*
- * - Flush TLBs for both L1 and L2 remote TLB flush
- * - Honor L1's request to flush an ASID on nested VMRUN
- * - Sync nested NPT MMU on VMRUN that flushes L2's ASID[*]
- * - Don't crush a pending TLB flush in vmcb02 on nested VMRUN
- * - Flush L1's ASID on KVM_REQ_TLB_FLUSH_GUEST
+ * If L1 requested a full TLB flush for all ASIDs (including its own),
+ * L1's own ASID is also flushed on nested VM-Exit, before running L1.
*
- * [*] Unlike nested EPT, SVM's ASID management can invalidate nested
- * NPT guest-physical mappings on VMRUN.
+ * TLB_CONTROL_FLUSH_ASID and TLB_CONTROL_FLUSH_ASID_LOCAL are handled
+ * equally for simplicity.
*/
- kvm_make_request(KVM_REQ_MMU_SYNC, vcpu);
- kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu);
+ if (new_asid || (svm->nested.ctl.tlb_ctl != TLB_CONTROL_DO_NOTHING)) {
+ if (nested_npt_enabled(svm))
+ kvm_make_request(KVM_REQ_MMU_SYNC_ALL_ROOTS, vcpu);
+ kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu);
+ }
+
+ /*
+ * If L1 and L2 share the same ASID in hardware (when using the fallback
+ * ASID for both, or for SEV guests), flush it on nested transitions.
+ */
+ if (svm->asid == svm->nested.asid02) {
+ WARN_ON_ONCE(svm->asid != fallback_asid && !is_sev_guest(vcpu));
+ kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu);
+ }
+}
+
+static void nested_svm_exit_tlb_flush(struct kvm_vcpu *vcpu)
+{
+ struct vcpu_svm *svm = to_svm(vcpu);
+
+ kvm_hv_nested_transtion_tlb_flush(vcpu, npt_enabled);
+
+ /* Flush L1's own ASID if it requested a *full* TLB flush on VMRUN */
+ if (svm->nested.ctl.tlb_ctl == TLB_CONTROL_FLUSH_ALL_ASID)
+ kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu);
+
+ if (svm->asid == svm->nested.asid02)
+ kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu);
}
/*
@@ -860,7 +901,7 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm)
struct vmcb *vmcb01 = svm->vmcb01.ptr;
struct kvm_vcpu *vcpu = &svm->vcpu;
- nested_svm_transition_tlb_flush(vcpu);
+ nested_svm_entry_tlb_flush(vcpu);
/* Enter Guest-Mode */
enter_guest_mode(vcpu);
@@ -936,10 +977,7 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm)
else
vmcb02->control.bus_lock_counter = 0;
- /* Done at vmrun: asid. */
-
- /* Also overwritten later if necessary. */
- vmcb02->control.tlb_ctl = TLB_CONTROL_DO_NOTHING;
+ vmcb02->control.asid = svm->nested.asid02;
/* Use vmcb01 MMU and format if guest does not use nNPT */
if (nested_npt_enabled(svm)) {
@@ -1038,6 +1076,20 @@ static void nested_svm_copy_common_state(struct vmcb *from_vmcb, struct vmcb *to
to_vmcb->save.spec_ctrl = from_vmcb->save.spec_ctrl;
}
+static void nested_svm_service_local_tlb_flushes(struct kvm_vcpu *vcpu)
+{
+ struct vcpu_svm *svm = to_svm(vcpu);
+
+ /*
+ * TLB flushes are applied to the VMCB, so apply any pending TLB flushes
+ * on the outgoing VMCB before switching to a new one. A TLB flush could
+ * purge the relevant HV TLB flush FIFO depending on guest_mode, so make
+ * sure the VMCB and guest_mode context is consistent.
+ */
+ WARN_ON_ONCE(is_guest_mode(vcpu) != (svm->vmcb == svm->nested.vmcb02.ptr));
+ kvm_service_local_tlb_flush_requests(vcpu);
+}
+
int enter_svm_guest_mode(struct kvm_vcpu *vcpu, u64 vmcb12_gpa, bool from_vmrun)
{
struct vcpu_svm *svm = to_svm(vcpu);
@@ -1069,6 +1121,8 @@ int enter_svm_guest_mode(struct kvm_vcpu *vcpu, u64 vmcb12_gpa, bool from_vmrun)
nested_svm_copy_common_state(svm->vmcb01.ptr, svm->nested.vmcb02.ptr);
+ nested_svm_service_local_tlb_flushes(vcpu);
+
svm_switch_vmcb(svm, &svm->nested.vmcb02);
nested_vmcb02_prepare_control(svm);
nested_vmcb02_prepare_save(svm);
@@ -1142,7 +1196,7 @@ int nested_svm_vmrun(struct kvm_vcpu *vcpu)
return ret;
}
- if (WARN_ON_ONCE(!svm->nested.initialized))
+ if (WARN_ON_ONCE(!svm->nested.vmcb02.ptr))
return -EINVAL;
vmcb12_gpa = kvm_rax_read(vcpu);
@@ -1194,6 +1248,13 @@ int nested_svm_vmrun(struct kvm_vcpu *vcpu)
svm->vmcb->control.exit_info_1 = 0;
svm->vmcb->control.exit_info_2 = 0;
+ /*
+ * Force a flush+sync on the next nested VMRUN. Otherwise,
+ * pending MMU syncs due to a changed ASID are lost, as
+ * last_asid was already updated.
+ */
+ svm->nested.last_asid = 0;
+
nested_svm_vmexit(svm);
}
@@ -1331,7 +1392,9 @@ void nested_svm_vmexit(struct vcpu_svm *svm)
if (nested_svm_vmexit_update_vmcb12(vcpu))
kvm_make_request(KVM_REQ_TRIPLE_FAULT, vcpu);
- /* Exit Guest-Mode */
+ nested_svm_service_local_tlb_flushes(vcpu);
+
+ svm_switch_vmcb(svm, &svm->vmcb01);
leave_guest_mode(vcpu);
svm_pmu_handle_nested_transition(svm);
@@ -1358,8 +1421,6 @@ void nested_svm_vmexit(struct vcpu_svm *svm)
if (guest_cpu_cap_has(vcpu, X86_FEATURE_ERAPS))
vmcb01->control.erap_ctl |= ERAP_CONTROL_CLEAR_RAP;
- svm_switch_vmcb(svm, &svm->vmcb01);
-
/*
* Rules for synchronizing int_ctl bits from vmcb02 to vmcb01:
*
@@ -1438,7 +1499,7 @@ void nested_svm_vmexit(struct vcpu_svm *svm)
svm->vcpu.arch.dr7 = DR7_FIXED_1;
kvm_update_dr7(&svm->vcpu);
- nested_svm_transition_tlb_flush(vcpu);
+ nested_svm_exit_tlb_flush(vcpu);
nested_svm_uninit_mmu_context(vcpu);
@@ -1489,21 +1550,36 @@ int svm_allocate_nested(struct vcpu_svm *svm)
{
struct page *vmcb02_page;
- if (svm->nested.initialized)
+ if (svm->nested.vmcb02.ptr)
return 0;
vmcb02_page = snp_safe_alloc_page();
if (!vmcb02_page)
return -ENOMEM;
- svm->nested.vmcb02.ptr = page_address(vmcb02_page);
- svm->nested.vmcb02.pa = __sme_set(page_to_pfn(vmcb02_page) << PAGE_SHIFT);
svm->nested.msrpm = svm_vcpu_alloc_msrpm();
if (!svm->nested.msrpm)
goto err_free_vmcb02;
svm->nested.vmcb02.cpu = -1;
- svm->nested.initialized = true;
+ svm->nested.vmcb02.ptr = page_address(vmcb02_page);
+ svm->nested.vmcb02.pa = __sme_set(page_to_pfn(vmcb02_page) << PAGE_SHIFT);
+
+ svm->nested.asid02 = allocate_asid(&svm->vcpu);
+
+ /*
+ * KVM uses a fallback ASID when out of ASIDs, and fails vCPU creation
+ * if there's no fallback ASID. ASID allocation must succeed here.
+ */
+ KVM_BUG_ON(!svm->nested.asid02, svm->vcpu.kvm);
+
+ /*
+ * Clear last_asid to ensure that the ASID is flushed on the first
+ * nested VM-Enter. Otherwise, stale TLB entries from a previous life of
+ * the ASID (e.g. different vCPU or even different VM) could be used.
+ */
+ svm->nested.last_asid = 0;
+
return 0;
err_free_vmcb02:
@@ -1513,12 +1589,14 @@ int svm_allocate_nested(struct vcpu_svm *svm)
void svm_free_nested(struct vcpu_svm *svm)
{
- if (!svm->nested.initialized)
+ if (!svm->nested.vmcb02.ptr)
return;
if (WARN_ON_ONCE(svm->vmcb != svm->vmcb01.ptr))
svm_switch_vmcb(svm, &svm->vmcb01);
+ free_asid(svm->nested.asid02);
+
svm_vcpu_free_msrpm(svm->nested.msrpm);
svm->nested.msrpm = NULL;
@@ -1533,8 +1611,6 @@ void svm_free_nested(struct vcpu_svm *svm)
* When the vmcb02 is freed, this optimization becomes invalid.
*/
svm->nested.last_vmcb12_gpa = INVALID_GPA;
-
- svm->nested.initialized = false;
}
void svm_leave_nested(struct kvm_vcpu *vcpu)
@@ -1545,6 +1621,9 @@ void svm_leave_nested(struct kvm_vcpu *vcpu)
vcpu->arch.nested_run_pending = 0;
svm->nested.vmcb12_gpa = INVALID_GPA;
+ nested_svm_service_local_tlb_flushes(vcpu);
+
+ svm_switch_vmcb(svm, &svm->vmcb01);
leave_guest_mode(vcpu);
/*
@@ -1556,7 +1635,7 @@ void svm_leave_nested(struct kvm_vcpu *vcpu)
*/
__svm_pmu_handle_nested_transition(svm, true);
- svm_switch_vmcb(svm, &svm->vmcb01);
+ nested_svm_exit_tlb_flush(vcpu);
nested_svm_uninit_mmu_context(vcpu);
vmcb_mark_all_dirty(svm->vmcb);
@@ -1853,7 +1932,7 @@ void nested_svm_update_tsc_ratio_msr(struct kvm_vcpu *vcpu)
svm_write_tsc_multiplier(vcpu);
}
-/* Inverse operation of nested_copy_vmcb_control_to_cache(). asid is copied too. */
+/* Inverse operation of nested_copy_vmcb_control_to_cache() */
static void nested_copy_vmcb_cache_to_control(struct vmcb_control_area *dst,
struct vmcb_ctrl_area_cached *from)
{
@@ -2084,6 +2163,8 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,
svm_copy_vmrun_state(vcpu, &svm->vmcb01.ptr->save, save);
nested_copy_vmcb_control_to_cache(svm, ctl);
+ nested_svm_service_local_tlb_flushes(vcpu);
+
svm_switch_vmcb(svm, &svm->nested.vmcb02);
if (use_separate_l2_pat)
diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index fad72cab76cf4..646108e70e21a 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -315,11 +315,6 @@ static int sev_asid_new(struct kvm_sev_info *sev, unsigned long vm_type)
return ret;
}
-static unsigned int sev_get_asid(struct kvm *kvm)
-{
- return to_kvm_sev_info(kvm)->asid;
-}
-
static void sev_asid_free(struct kvm_sev_info *sev)
{
struct svm_cpu_data *sd;
@@ -2037,6 +2032,28 @@ static void sev_unlock_two_vms(struct kvm *dst_kvm, struct kvm *src_kvm)
atomic_set_release(&src_sev->migration_in_progress, 0);
}
+static void sev_vcpu_migrate_asid(struct vcpu_svm *dst_svm,
+ unsigned int asid)
+{
+ /*
+ * Free the (potentially non-SEV) ASID on the destination vCPU before
+ * setting the new SEV ASID.
+ */
+ free_asid(dst_svm->asid);
+ dst_svm->asid = asid;
+
+ /*
+ * If nested is already initialized on the destination vCPU, also free
+ * the previous ASID (in case it was non-SEV) and update the nested ASID
+ * to match the new SEV ASID.
+ */
+ if (dst_svm->nested.vmcb02.ptr) {
+ free_asid(dst_svm->nested.asid02);
+ dst_svm->nested.asid02 = asid;
+ dst_svm->nested.vmcb02.ptr->control.asid = asid;
+ }
+}
+
static void sev_migrate_from(struct kvm *dst_kvm, struct kvm *src_kvm)
{
struct kvm_sev_info *dst = to_kvm_sev_info(dst_kvm);
@@ -2101,6 +2118,7 @@ static void sev_migrate_from(struct kvm *dst_kvm, struct kvm *src_kvm)
kvm_for_each_vcpu(i, dst_vcpu, dst_kvm) {
dst_svm = to_svm(dst_vcpu);
+ sev_vcpu_migrate_asid(dst_svm, dst->asid);
sev_init_vmcb(dst_svm, false);
if (!dst->es_active)
@@ -3623,9 +3641,6 @@ int pre_sev_run(struct vcpu_svm *svm, int cpu)
if (!cpumask_test_cpu(cpu, to_kvm_sev_info(kvm)->have_run_cpus))
cpumask_set_cpu(cpu, to_kvm_sev_info(kvm)->have_run_cpus);
- /* Assign the asid allocated with this SEV guest */
- svm->asid = asid;
-
/*
* Flush guest TLB:
*
@@ -3637,7 +3652,7 @@ int pre_sev_run(struct vcpu_svm *svm, int cpu)
return 0;
sd->sev_vmcbs[asid] = svm->vmcb;
- svm->vmcb->control.tlb_ctl = TLB_CONTROL_FLUSH_ASID;
+ vmcb_set_flush_asid(svm->vmcb);
return 0;
}
@@ -4889,6 +4904,10 @@ void sev_init_vmcb(struct vcpu_svm *svm, bool init_event)
svm->vmcb->control.misc_ctl |= SVM_MISC_ENABLE_SEV;
clr_exception_intercept(svm, UD_VECTOR);
+ WARN_ON_ONCE(svm->asid != sev_get_asid(vcpu->kvm));
+ svm->vmcb->control.asid = svm->asid;
+ vmcb_mark_dirty(svm->vmcb, VMCB_ASID);
+
/*
* Don't intercept #GP for SEV guests, e.g. for the VMware backdoor, as
* KVM can't decrypt guest memory to decode the faulting instruction.
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 136069dfcd44f..436d13f106369 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -192,6 +192,8 @@ DEFINE_PER_CPU(struct svm_cpu_data, svm_data);
static DEFINE_MUTEX(vmcb_dump_mutex);
+kvm_tlb_tag_t fallback_asid;
+
/*
* Only MSR_TSC_AUX is switched via the user return hook. EFER is switched via
* the VMCB, and the SYSCALL/SYSENTER MSRs are handled by VMLOAD/VMSAVE.
@@ -583,15 +585,6 @@ static int svm_enable_virtualization_cpu(void)
return r;
sd = per_cpu_ptr(&svm_data, me);
- /*
- * Bump the current asid_generation value to ensure any vCPU that
- * previously ran on this CPU sees a stale generation and is forced
- * to acquire a new ASID, preventing a latent ASID collision.
- */
- sd->asid_generation++;
- sd->max_asid = cpuid_ebx(SVM_CPUID_FUNC) - 1;
- sd->next_asid = sd->max_asid + 1;
- sd->min_asid = max_sev_asid + 1;
wrmsrq(MSR_VM_HSAVE_PA, sd->save_area_pa);
@@ -994,6 +987,8 @@ static void svm_hardware_unsetup(void)
__free_pages(__sme_pa_to_page(iopm_base), get_order(IOPM_SIZE));
iopm_base = 0;
+
+ kvm_destroy_tlb_tags();
}
static void init_seg(struct vmcb_seg *seg)
@@ -1246,8 +1241,8 @@ static void init_vmcb(struct kvm_vcpu *vcpu, bool init_event)
if (gmet_enabled)
control->misc_ctl |= SVM_MISC_ENABLE_GMET;
- svm->current_vmcb->asid_generation = 0;
- svm->asid = 0;
+ control->asid = svm->asid;
+ vmcb_set_flush_asid(vmcb);
svm->nested.vmcb12_gpa = INVALID_GPA;
svm->nested.last_vmcb12_gpa = INVALID_GPA;
@@ -1375,6 +1370,12 @@ static int svm_vcpu_create(struct kvm_vcpu *vcpu)
goto error_free_avic;
}
+ svm->asid = allocate_asid(vcpu);
+ if (!svm->asid) {
+ err = -EINVAL;
+ goto error_free_msrpm;
+ }
+
svm->x2avic_msrs_intercepted = true;
svm->lbr_msrs_intercepted = true;
@@ -1386,6 +1387,8 @@ static int svm_vcpu_create(struct kvm_vcpu *vcpu)
return 0;
+error_free_msrpm:
+ svm_vcpu_free_msrpm(svm->msrpm);
error_free_avic:
avic_vcpu_free(vcpu);
error_free_sev:
@@ -1402,6 +1405,8 @@ static int svm_vcpu_create(struct kvm_vcpu *vcpu)
static void svm_vcpu_free(struct kvm_vcpu *vcpu)
{
struct vcpu_svm *svm = to_svm(vcpu);
+ struct svm_cpu_data *sd;
+ int cpu;
WARN_ON_ONCE(!list_empty(&svm->ir_list));
@@ -1417,6 +1422,13 @@ static void svm_vcpu_free(struct kvm_vcpu *vcpu)
__free_page(__sme_pa_to_page(svm->vmcb01.pa));
svm_vcpu_free_msrpm(svm->msrpm);
+
+ free_asid(svm->asid);
+
+ for_each_possible_cpu(cpu) {
+ sd = per_cpu_ptr(&svm_data, cpu);
+ cmpxchg(&sd->fallback_asid_vcpu, vcpu, NULL);
+ }
}
#ifdef CONFIG_CPU_MITIGATIONS
@@ -1942,18 +1954,6 @@ static void svm_update_exception_bitmap(struct kvm_vcpu *vcpu)
}
}
-static void new_asid(struct vcpu_svm *svm, struct svm_cpu_data *sd)
-{
- if (sd->next_asid > sd->max_asid) {
- ++sd->asid_generation;
- sd->next_asid = sd->min_asid;
- sd->flush_all_asids = true;
- }
-
- svm->current_vmcb->asid_generation = sd->asid_generation;
- svm->asid = sd->next_asid++;
-}
-
static void svm_set_dr6(struct kvm_vcpu *vcpu, unsigned long value)
{
struct vmcb *vmcb = to_svm(vcpu)->vmcb;
@@ -2468,17 +2468,61 @@ static int clgi_interception(struct kvm_vcpu *vcpu)
static int invlpga_interception(struct kvm_vcpu *vcpu)
{
+ struct vcpu_svm *svm = to_svm(vcpu);
/* FIXME: Handle an address size prefix. */
gva_t gva = kvm_rax_read(vcpu);
u32 asid = kvm_ecx_read(vcpu);
+ int cpu;
if (nested_svm_check_permissions(vcpu))
return 1;
trace_kvm_invlpga(to_svm(vcpu)->vmcb->save.rip, asid, gva);
- /* Let's treat INVLPGA the same as INVLPG (can be optimized!) */
- kvm_mmu_invlpg(vcpu, gva);
+ /*
+ * INVLPG on a non-canonical address is a NOP according to the SDM,
+ * assume the same behavior from INVLPGA since the APM doesn't specify.
+ */
+ if (is_noncanonical_invlpg_address(gva, vcpu))
+ return kvm_skip_emulated_instruction(vcpu);
+
+ /*
+ * If L1 is flushing its own ASID, handle it like INVLPG. Do the same if
+ * this is an unintercepted INVLPGA from L2, as
+ * kvm_mmu_free_guest_mode_roots() does not handle being in guest mode.
+ */
+ if (!asid || is_guest_mode(vcpu)) {
+ kvm_mmu_invlpg(vcpu, gva);
+ return kvm_skip_emulated_instruction(vcpu);
+ }
+
+ /*
+ * Do nothing if L1 is flushing a different L2 ASID than the one KVM is
+ * currently tracking. KVM tracks a single L2 ASID, and performs a TLB
+ * flush (and MMU resync if needed) when L1 switches ASIDs anyway.
+ */
+ if (asid != svm->nested.last_asid)
+ return kvm_skip_emulated_instruction(vcpu);
+
+ /*
+ * Use INVLPGA to flush L2's ASID in hardware if running on the same
+ * CPU, otherwise fallback to a full ASID flush.
+ */
+ cpu = get_cpu();
+ if (cpu == svm->nested.vmcb02.cpu)
+ invlpga(gva, svm->nested.asid02);
+ else
+ vmcb_set_flush_asid(svm->nested.vmcb02.ptr);
+ put_cpu();
+
+ /*
+ * If NPT is disabled, free L2's shadow roots as they may have stale
+ * translations for the flushed address. Do not sync the mappings for
+ * the flushed address, as it involves walking L2 shadow page tables
+ * with L1's MMU context.
+ */
+ if (!npt_enabled)
+ kvm_mmu_free_guest_mode_roots(vcpu->kvm, &vcpu->arch.root_mmu);
return kvm_skip_emulated_instruction(vcpu);
}
@@ -3875,37 +3919,28 @@ static void svm_set_nested_run_soft_int_state(struct kvm_vcpu *vcpu)
static int pre_svm_run(struct kvm_vcpu *vcpu)
{
- struct svm_cpu_data *sd = per_cpu_ptr(&svm_data, vcpu->cpu);
+ struct svm_cpu_data *sd = this_cpu_ptr(&svm_data);
struct vcpu_svm *svm = to_svm(vcpu);
/*
* If the previous VMRUN of the VMCB occurred on a different physical
- * cpu, then mark the VMCB dirty as hardware's clean bits are per pCPU.
- *
- * Reset the ASID generation in both VMCBs. This will lead to assigning
- * a new ASID now, and then again when switching to the other VMCB.
- * However, this is needed as the ASID is shared between the VMCBs, and
- * otherwise it would be possible to use an ASID allocated on one pCPU
- * on another, for example:
- * - vCPU migrates from pCPU A to pCPU B, allocates a new ASID.
- * - vCPU migrates back to pCPU A, and then switches the VMCB.
- * - The new VMCB does not detect a pCPU change and runs on pCPU A with
- * the new ASID allocated on pCPU B, which is potentially used by
- * another vCPU/VM.
+ * cpu, then mark the VMCB dirty and flush the ASID, as hardware's clean
+ * bits and TLB entries are per pCPU.
*/
if (unlikely(svm->current_vmcb->cpu != vcpu->cpu)) {
vmcb_mark_all_dirty(svm->vmcb);
svm->current_vmcb->cpu = vcpu->cpu;
- svm->vmcb01.asid_generation = 0;
- svm->nested.vmcb02.asid_generation = 0;
+ vmcb_set_flush_asid(svm->vmcb);
}
if (is_sev_guest(vcpu))
return pre_sev_run(svm, vcpu->cpu);
- /* FIXME: handle wraparound of asid_generation */
- if (svm->current_vmcb->asid_generation != sd->asid_generation)
- new_asid(svm, sd);
+ if (unlikely(svm->vmcb->control.asid == fallback_asid &&
+ READ_ONCE(sd->fallback_asid_vcpu) != vcpu)) {
+ vmcb_set_flush_asid(svm->vmcb);
+ WRITE_ONCE(sd->fallback_asid_vcpu, vcpu);
+ }
return 0;
}
@@ -4317,7 +4352,7 @@ static void svm_flush_tlb_asid(struct kvm_vcpu *vcpu)
* A TLB flush for the current ASID flushes both "host" and "guest" TLB
* entries, and thus is a superset of Hyper-V's fine grained flushing.
*/
- kvm_hv_vcpu_purge_flush_tlb(vcpu);
+ kvm_hv_purge_tlb_flush_fifo(vcpu);
/*
* Flush only the current ASID even if the TLB flush was invoked via
@@ -4326,10 +4361,7 @@ static void svm_flush_tlb_asid(struct kvm_vcpu *vcpu)
* unconditionally does a TLB flush on both nested VM-Enter and nested
* VM-Exit (via kvm_mmu_reset_context()).
*/
- if (cpu_feature_enabled(X86_FEATURE_FLUSHBYASID))
- svm->vmcb->control.tlb_ctl = TLB_CONTROL_FLUSH_ASID;
- else
- svm->current_vmcb->asid_generation--;
+ vmcb_set_flush_asid(svm->vmcb);
}
static void svm_flush_tlb_current(struct kvm_vcpu *vcpu)
@@ -4350,6 +4382,8 @@ static void svm_flush_tlb_current(struct kvm_vcpu *vcpu)
static void svm_flush_tlb_all(struct kvm_vcpu *vcpu)
{
+ struct vcpu_svm *svm = to_svm(vcpu);
+
/*
* When running on Hyper-V with EnlightenedNptTlb enabled, remote TLB
* flushes should be routed to hv_flush_remote_tlbs() without requesting
@@ -4360,7 +4394,11 @@ static void svm_flush_tlb_all(struct kvm_vcpu *vcpu)
if (WARN_ON_ONCE(svm_hv_is_enlightened_tlb_enabled(vcpu)))
hv_flush_remote_tlbs(vcpu->kvm);
- svm_flush_tlb_asid(vcpu);
+ kvm_hv_purge_all_tlb_flush_fifos(vcpu);
+
+ vmcb_set_flush_asid(svm->vmcb01.ptr);
+ if (svm->nested.vmcb02.ptr)
+ vmcb_set_flush_asid(svm->nested.vmcb02.ptr);
}
static void svm_flush_tlb_guest(struct kvm_vcpu *vcpu)
@@ -4647,13 +4685,6 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)
sync_lapic_to_cr8(vcpu);
- if (unlikely(svm->asid != svm->vmcb->control.asid)) {
- svm->vmcb->control.asid = svm->asid;
- vmcb_mark_dirty(svm->vmcb, VMCB_ASID);
- }
- if (this_cpu_ptr(&svm_data)->flush_all_asids)
- svm->vmcb->control.tlb_ctl = TLB_CONTROL_FLUSH_ALL_ASID;
-
svm->vmcb->save.cr2 = vcpu->arch.cr2;
if (guest_cpu_cap_has(vcpu, X86_FEATURE_ERAPS) &&
@@ -4745,8 +4776,7 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)
}
if (!svm_is_vmrun_failure(svm->vmcb->control.exit_code)) {
- this_cpu_ptr(&svm_data)->flush_all_asids = false;
- svm->vmcb->control.tlb_ctl = TLB_CONTROL_DO_NOTHING;
+ vmcb_clr_flush_asid(svm->vmcb);
/*
* Unconditionally mask off the CLEAR_RAP bit, the AND is just
@@ -5659,9 +5689,8 @@ static __init void svm_set_cpu_caps(void)
kvm_cpu_cap_set(X86_FEATURE_VMCBCLEAN);
/*
- * KVM currently flushes TLBs on *every* nested SVM transition,
- * and so for all intents and purposes KVM supports flushing by
- * ASID, i.e. KVM is guaranteed to honor every L1 ASID flush.
+ * KVM handles all TLB_CONTROL values set by L1, even if the
+ * underlying CPU does not.
*/
kvm_cpu_cap_set(X86_FEATURE_FLUSHBYASID);
@@ -5757,6 +5786,7 @@ static __init void svm_set_cpu_caps(void)
static __init int svm_hardware_setup(void)
{
+ unsigned long nr_asids;
void *iopm_va;
int cpu, r;
@@ -5914,6 +5944,14 @@ static __init int svm_hardware_setup(void)
kvm_caps.inapplicable_quirks &= ~KVM_X86_QUIRK_CD_NW_CLEARED;
+ /* Consumes max_sev_asid initialized by sev_hardware_setup() */
+ nr_asids = cpuid_ebx(SVM_CPUID_FUNC);
+ r = kvm_init_tlb_tags(nr_asids, max_sev_asid + 1);
+ if (r)
+ goto err;
+
+ fallback_asid = kvm_alloc_tlb_tag();
+
for_each_possible_cpu(cpu) {
r = svm_cpu_init(cpu);
if (r)
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index 6d0242966dc87..15c4aa3b02c2d 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -26,6 +26,7 @@
#include "regs.h"
#include "x86.h"
#include "pmu.h"
+#include "mmu.h"
/*
* Helpers to convert to/from physical addresses for pages whose address is
@@ -145,7 +146,6 @@ struct kvm_vmcb_info {
struct vmcb *ptr;
unsigned long pa;
int cpu;
- uint64_t asid_generation;
};
struct vmcb_save_area_cached {
@@ -234,7 +234,7 @@ struct svm_nested_state {
*/
struct vmcb_save_area_cached save;
- bool initialized;
+ kvm_tlb_tag_t asid02;
/*
* Indicates whether MSR bitmap for L2 needs to be rebuilt due to
@@ -244,6 +244,8 @@ struct svm_nested_state {
* on its side.
*/
bool force_msr_bitmap_recalc;
+
+ u32 last_asid;
};
struct vcpu_sev_es_state {
@@ -285,7 +287,7 @@ struct vcpu_svm {
struct vmcb *vmcb;
struct kvm_vmcb_info vmcb01;
struct kvm_vmcb_info *current_vmcb;
- u32 asid;
+ kvm_tlb_tag_t asid;
u32 sysenter_esp_hi;
u32 sysenter_eip_hi;
uint64_t tsc_aux;
@@ -372,17 +374,14 @@ struct vcpu_svm {
};
struct svm_cpu_data {
- u64 asid_generation;
- u32 max_asid;
- u32 next_asid;
- u32 min_asid;
-
- bool flush_all_asids;
bool bp_spec_reduce_set;
struct vmcb *save_area;
unsigned long save_area_pa;
+ /* Last vCPU to use fallback_asid on this CPU */
+ struct kvm_vcpu *fallback_asid_vcpu;
+
/* index = sev_asid, value = vmcb pointer */
struct vmcb **sev_vmcbs;
};
@@ -432,6 +431,11 @@ static __always_inline bool is_sev_snp_guest(struct kvm_vcpu *vcpu)
{
return ____sev_snp_guest(vcpu->kvm);
}
+
+static __always_inline unsigned int sev_get_asid(struct kvm *kvm)
+{
+ return to_kvm_sev_info(kvm)->asid;
+}
#else
static __always_inline bool is_sev_guest(struct kvm_vcpu *vcpu)
{
@@ -446,6 +450,11 @@ static __always_inline bool is_sev_snp_guest(struct kvm_vcpu *vcpu)
{
return false;
}
+
+static __always_inline unsigned int sev_get_asid(struct kvm *kvm)
+{
+ return 0;
+}
#endif
static inline bool ghcb_gpa_is_registered(struct vcpu_svm *svm, u64 val)
@@ -480,6 +489,19 @@ static inline void vmcb_set_gpat(struct vmcb *vmcb, u64 data)
vmcb_mark_dirty(vmcb, VMCB_NPT);
}
+static inline void vmcb_set_flush_asid(struct vmcb *vmcb)
+{
+ if (cpu_feature_enabled(X86_FEATURE_FLUSHBYASID))
+ vmcb->control.tlb_ctl = TLB_CONTROL_FLUSH_ASID;
+ else
+ vmcb->control.tlb_ctl = TLB_CONTROL_FLUSH_ALL_ASID;
+}
+
+static inline void vmcb_clr_flush_asid(struct vmcb *vmcb)
+{
+ vmcb->control.tlb_ctl = TLB_CONTROL_DO_NOTHING;
+}
+
static __always_inline struct vcpu_svm *to_svm(struct kvm_vcpu *vcpu)
{
return container_of(vcpu, struct vcpu_svm, vcpu);
@@ -1057,6 +1079,21 @@ static inline struct vmcb_save_area *sev_decrypt_vmsa(struct kvm_vcpu *vcpu)
static inline void sev_free_decrypted_vmsa(struct kvm_vcpu *vcpu, struct vmcb_save_area *vmsa) {}
#endif
+extern kvm_tlb_tag_t fallback_asid;
+
+static inline kvm_tlb_tag_t allocate_asid(struct kvm_vcpu *vcpu)
+{
+ if (is_sev_guest(vcpu))
+ return sev_get_asid(vcpu->kvm);
+ return kvm_alloc_tlb_tag() ?: fallback_asid;
+}
+
+static inline void free_asid(kvm_tlb_tag_t asid)
+{
+ if (asid > max_sev_asid && asid != fallback_asid)
+ kvm_free_tlb_tag(asid);
+}
+
/* vmenter.S */
void __svm_sev_es_vcpu_run(struct vcpu_svm *svm, unsigned int flags,
diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index b341af4c6057a..a1caa2bd04d7a 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -598,9 +598,6 @@ DEFINE_PER_CPU(struct vmcs *, current_vmcs);
*/
static DEFINE_PER_CPU(struct list_head, loaded_vmcss_on_cpu);
-static DECLARE_BITMAP(vmx_vpid_bitmap, VMX_NR_VPIDS);
-static DEFINE_SPINLOCK(vmx_vpid_lock);
-
struct vmcs_config vmcs_config __ro_after_init;
struct vmx_capability vmx_capability __ro_after_init;
@@ -4092,31 +4089,6 @@ static void seg_setup(int seg)
vmcs_write32(sf->ar_bytes, ar);
}
-int allocate_vpid(void)
-{
- int vpid;
-
- if (!enable_vpid)
- return 0;
- spin_lock(&vmx_vpid_lock);
- vpid = find_first_zero_bit(vmx_vpid_bitmap, VMX_NR_VPIDS);
- if (vpid < VMX_NR_VPIDS)
- __set_bit(vpid, vmx_vpid_bitmap);
- else
- vpid = 0;
- spin_unlock(&vmx_vpid_lock);
- return vpid;
-}
-
-void free_vpid(int vpid)
-{
- if (!enable_vpid || vpid == 0)
- return;
- spin_lock(&vmx_vpid_lock);
- __clear_bit(vpid, vmx_vpid_bitmap);
- spin_unlock(&vmx_vpid_lock);
-}
-
static void vmx_msr_bitmap_l01_changed(struct vcpu_vmx *vmx)
{
/*
@@ -8415,6 +8387,8 @@ void vmx_hardware_unsetup(void)
if (nested)
nested_vmx_hardware_unsetup();
+
+ destroy_vpids();
}
void vmx_vm_destroy(struct kvm *kvm)
@@ -8640,8 +8614,6 @@ __init int vmx_hardware_setup(void)
kvm_caps.has_bus_lock_exit = cpu_has_vmx_bus_lock_detection();
kvm_caps.has_notify_vmexit = cpu_has_notify_vmexit();
- set_bit(0, vmx_vpid_bitmap); /* 0 is reserved for host */
-
if (enable_ept)
kvm_mmu_set_ept_masks(enable_ept_ad_bits);
else
@@ -8681,6 +8653,10 @@ __init int vmx_hardware_setup(void)
vmx_set_cpu_caps();
+ r = init_vpids();
+ if (r)
+ return r;
+
/*
* Configure nested capabilities after core CPU capabilities so that
* nested support can be conditional on base support, e.g. so that KVM
@@ -8688,8 +8664,10 @@ __init int vmx_hardware_setup(void)
*/
if (nested) {
r = nested_vmx_hardware_setup(kvm_vmx_exit_handlers);
- if (r)
+ if (r) {
+ destroy_vpids();
return r;
+ }
}
vmx_nested_ops.enabled = nested;
diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h
index 6c9dd5b2365d5..3e1ecd9148e9c 100644
--- a/arch/x86/kvm/vmx/vmx.h
+++ b/arch/x86/kvm/vmx/vmx.h
@@ -182,7 +182,7 @@ struct nested_vmx {
u64 pre_vmenter_ssp;
u64 pre_vmenter_ssp_tbl;
- u16 vpid02;
+ kvm_tlb_tag_t vpid02;
u16 last_vpid;
int tsc_autostore_slot;
@@ -256,7 +256,7 @@ struct vcpu_vmx {
u32 ar;
} seg[8];
} segment_cache;
- int vpid;
+ kvm_tlb_tag_t vpid;
/* Support for a guest hypervisor (nested VMX) */
struct nested_vmx nested;
@@ -297,9 +297,29 @@ struct kvm_vmx {
u64 *pid_table;
};
+static __always_inline int init_vpids(void)
+{
+ return enable_vpid ? kvm_init_tlb_tags(VMX_NR_VPIDS, 1) : 0;
+}
+
+static __always_inline void destroy_vpids(void)
+{
+ if (enable_vpid)
+ kvm_destroy_tlb_tags();
+}
+
+static __always_inline kvm_tlb_tag_t allocate_vpid(void)
+{
+ return enable_vpid ? kvm_alloc_tlb_tag() : 0;
+}
+
+static __always_inline void free_vpid(kvm_tlb_tag_t vpid)
+{
+ if (enable_vpid)
+ kvm_free_tlb_tag(vpid);
+}
+
void vmx_vcpu_load_vmcs(struct kvm_vcpu *vcpu, int cpu);
-int allocate_vpid(void);
-void free_vpid(int vpid);
void vmx_set_constant_host_state(struct vcpu_vmx *vmx);
void vmx_prepare_switch_to_guest(struct kvm_vcpu *vcpu);
void vmx_set_host_fs_gs(struct vmcs_host_state *host, u16 fs_sel, u16 gs_sel,
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 32563a91fba99..a54e9da2ed09e 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -2020,8 +2020,7 @@ static void kvm_vcpu_flush_tlb_guest(struct kvm_vcpu *vcpu)
* a forced sync of the shadow page tables. Ensure all the
* roots are synced and the guest TLB in hardware is clean.
*/
- kvm_mmu_sync_roots(vcpu);
- kvm_mmu_sync_prev_roots(vcpu);
+ kvm_mmu_sync_all_roots(vcpu);
}
kvm_x86_call(flush_tlb_guest)(vcpu);
@@ -2030,7 +2029,7 @@ static void kvm_vcpu_flush_tlb_guest(struct kvm_vcpu *vcpu)
* Flushing all "guest" TLB is always a superset of Hyper-V's fine
* grained flushing.
*/
- kvm_hv_vcpu_purge_flush_tlb(vcpu);
+ kvm_hv_purge_tlb_flush_fifo(vcpu);
}
@@ -8167,8 +8166,14 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
if (unlikely(r))
goto out;
}
- if (kvm_check_request(KVM_REQ_MMU_SYNC, vcpu))
+
+ if (kvm_check_request(KVM_REQ_MMU_SYNC_ALL_ROOTS, vcpu)) {
+ kvm_mmu_sync_all_roots(vcpu);
+ kvm_clear_request(KVM_REQ_MMU_SYNC, vcpu);
+ } else if (kvm_check_request(KVM_REQ_MMU_SYNC, vcpu)) {
kvm_mmu_sync_roots(vcpu);
+ }
+
if (kvm_check_request(KVM_REQ_LOAD_MMU_PGD, vcpu))
kvm_mmu_load_pgd(vcpu);
diff --git a/tools/testing/selftests/kvm/Makefile.kvm b/tools/testing/selftests/kvm/Makefile.kvm
index be5eb2dd85aa2..5347c56786b08 100644
--- a/tools/testing/selftests/kvm/Makefile.kvm
+++ b/tools/testing/selftests/kvm/Makefile.kvm
@@ -122,6 +122,7 @@ TEST_GEN_PROGS_x86 += x86/svm_nested_shutdown_test
TEST_GEN_PROGS_x86 += x86/svm_nested_soft_inject_test
TEST_GEN_PROGS_x86 += x86/svm_nested_vmcb12_gpa
TEST_GEN_PROGS_x86 += x86/svm_nested_pat_test
+TEST_GEN_PROGS_x86 += x86/nested_tlb_flush_test
TEST_GEN_PROGS_x86 += x86/svm_lbr_nested_state
TEST_GEN_PROGS_x86 += x86/svm_pmu_host_guest_test
TEST_GEN_PROGS_x86 += x86/tsc_scaling_sync
diff --git a/tools/testing/selftests/kvm/include/x86/processor.h b/tools/testing/selftests/kvm/include/x86/processor.h
index ffec2291b1e1f..62d255aee5a34 100644
--- a/tools/testing/selftests/kvm/include/x86/processor.h
+++ b/tools/testing/selftests/kvm/include/x86/processor.h
@@ -1617,6 +1617,7 @@ void tdp_map(struct kvm_vm *vm, gpa_t l2_gpa, gpa_t gpa, u64 size);
void tdp_identity_map_default_memslots(struct kvm_vm *vm);
void tdp_identity_map_1g(struct kvm_vm *vm, u64 addr, u64 size);
u64 *tdp_get_pte(struct kvm_vm *vm, u64 l2_gpa);
+gpa_t tdp_copy_page_tables(struct kvm_vm *vm);
/*
* Basic CPU control in CR0
diff --git a/tools/testing/selftests/kvm/include/x86/svm.h b/tools/testing/selftests/kvm/include/x86/svm.h
index c8539166270ea..d2a6bf8503f78 100644
--- a/tools/testing/selftests/kvm/include/x86/svm.h
+++ b/tools/testing/selftests/kvm/include/x86/svm.h
@@ -316,4 +316,9 @@ struct __attribute__ ((__packed__)) vmcb {
#define SVM_CR0_SELECTIVE_MASK (X86_CR0_TS | X86_CR0_MP)
+static inline void invlpga(unsigned long addr, u32 asid)
+{
+ asm volatile("invlpga %0, %1" : : "a"(addr), "c"(asid) : "memory");
+}
+
#endif /* SELFTEST_KVM_SVM_H */
diff --git a/tools/testing/selftests/kvm/include/x86/vmx.h b/tools/testing/selftests/kvm/include/x86/vmx.h
index 8fd97434eee5c..4f2a0e3ac5ba1 100644
--- a/tools/testing/selftests/kvm/include/x86/vmx.h
+++ b/tools/testing/selftests/kvm/include/x86/vmx.h
@@ -518,4 +518,25 @@ static inline bool kvm_cpu_has_vmx_virtualize_apic_accesses(void)
void vm_enable_ept(struct kvm_vm *vm);
void prepare_virtualize_apic_accesses(struct vmx_pages *vmx, struct kvm_vm *vm);
+static inline void invvpid(unsigned long ext, u16 vpid, gva_t gva)
+{
+ struct {
+ u64 vpid : 16;
+ u64 rsvd : 48;
+ u64 gva;
+ } operand = { vpid, 0, gva };
+
+ asm volatile("invvpid %0, %1" : : "m"(operand), "r"(ext) : "memory");
+}
+
+static inline void invept(unsigned long ext, u64 eptp)
+{
+ struct {
+ u64 eptp;
+ u64 rsvd;
+ } operand = { eptp, 0 };
+
+ asm volatile("invept %0, %1" : : "m"(operand), "r"(ext) : "memory");
+}
+
#endif /* SELFTEST_KVM_VMX_H */
diff --git a/tools/testing/selftests/kvm/lib/x86/processor.c b/tools/testing/selftests/kvm/lib/x86/processor.c
index 8374afd08bafe..57ecf8b381b5a 100644
--- a/tools/testing/selftests/kvm/lib/x86/processor.c
+++ b/tools/testing/selftests/kvm/lib/x86/processor.c
@@ -554,6 +554,42 @@ void tdp_identity_map_1g(struct kvm_vm *vm, u64 addr, u64 size)
__tdp_map(vm, addr, addr, size, PG_LEVEL_1G);
}
+static gpa_t __virt_copy_page_tables(struct kvm_vm *vm, struct kvm_mmu *mmu,
+ gpa_t table_gpa, int level)
+{
+ gpa_t copy_gpa = vm_alloc_page_table(vm);
+ u64 *copy = addr_gpa2hva(vm, copy_gpa);
+ gpa_t pa;
+ int i;
+
+ memcpy(copy, addr_gpa2hva(vm, table_gpa), vm->page_size);
+ if (level == PG_LEVEL_4K)
+ return copy_gpa;
+
+ for (i = 0; i < vm->page_size / sizeof(u64); i++) {
+ if (!is_present_pte(mmu, ©[i]) || is_huge_pte(mmu, ©[i]))
+ continue;
+
+ /* Clear only the address bits, preserving C/S bits and flags */
+ pa = vm_untag_gpa(vm, PTE_GET_PA(copy[i]));
+ copy[i] &= ~pa;
+ copy[i] |= __virt_copy_page_tables(vm, mmu, pa, level - 1);
+ }
+ return copy_gpa;
+}
+
+/*
+ * Deep copy all levels of the TDP page tables, such that no page table pages
+ * are shared with the original. Returns the GPA of the new root.
+ */
+gpa_t tdp_copy_page_tables(struct kvm_vm *vm)
+{
+ struct kvm_mmu *mmu = &vm->stage2_mmu;
+
+ TEST_ASSERT(mmu->pgd_created, "TDP page tables not created");
+ return __virt_copy_page_tables(vm, mmu, mmu->pgd, mmu->pgtable_levels);
+}
+
/*
* Set Unusable Segment
*
diff --git a/tools/testing/selftests/kvm/x86/nested_tlb_flush_test.c b/tools/testing/selftests/kvm/x86/nested_tlb_flush_test.c
new file mode 100644
index 0000000000000..fb32b7bf6b19f
--- /dev/null
+++ b/tools/testing/selftests/kvm/x86/nested_tlb_flush_test.c
@@ -0,0 +1,526 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2026, Google LLC.
+ *
+ * Test TLB flushes with nested virtualization across three cases:
+ * 1. When L1 updates L2's (shadow) page tables
+ * 2. When L1 updates nested TDP
+ * 3. When KVM updates mappings on the host (KVM-induced TLB flushes)
+ */
+#include <fcntl.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#include "test_util.h"
+#include "kvm_util.h"
+#include "processor.h"
+#include "svm_util.h"
+#include "vmx.h"
+
+#define NR_ITERATIONS 100
+
+#define VAL1 0x11111111ULL
+#define VAL2 0x22222222ULL
+
+#define TEST_VADDR 0x10000000ULL
+#define TEST_GPA 0x30000000ULL
+#define TEST_NESTED_GPA 0x40000000ULL
+
+#define TEST_MEMSLOT 10
+
+#define SYNC_MOVE_MEMSLOT 1
+
+enum guest_flush_method {
+ TLB_FLUSH_TAG = 0,
+ TLB_FLUSH_VADDR,
+ TLB_CHANGE_TAG,
+ TLB_FLUSH_NONE,
+};
+
+static u64 *pte_gva;
+static gpa_t test_gpa[2];
+static enum guest_flush_method flush_method;
+static u32 max_asid;
+
+/* TDP roots for the multi-root test, root 1 is a non-aliasing copy of root 0 */
+static gpa_t tdp_root_gpa[2];
+/* Value L2 expects at TEST_VADDR, 0 if L2 shouldn't access TEST_VADDR */
+static u64 l2_expected_val;
+
+static void svm_tlb_flush(struct svm_test_data *svm,
+ enum guest_flush_method method)
+{
+ struct vmcb *vmcb = svm->vmcb;
+
+ switch (method) {
+ case TLB_CHANGE_TAG:
+ vmcb->control.asid++;
+ vmcb->control.tlb_ctl = TLB_CONTROL_DO_NOTHING;
+ if (vmcb->control.asid >= max_asid) {
+ vmcb->control.asid = 1;
+ vmcb->control.tlb_ctl = TLB_CONTROL_FLUSH_ALL_ASID;
+ }
+ break;
+ case TLB_FLUSH_TAG:
+ vmcb->control.tlb_ctl = TLB_CONTROL_FLUSH_ASID;
+ break;
+ case TLB_FLUSH_VADDR:
+ GUEST_ASSERT(!svm->ncr3_gpa);
+ vmcb->control.tlb_ctl = TLB_CONTROL_DO_NOTHING;
+ invlpga(TEST_VADDR, vmcb->control.asid);
+ break;
+ case TLB_FLUSH_NONE:
+ vmcb->control.tlb_ctl = TLB_CONTROL_DO_NOTHING;
+ break;
+ }
+}
+
+static void vmx_tlb_flush(struct vmx_pages *vmx, enum guest_flush_method method)
+{
+ u16 vpid = vmread(VIRTUAL_PROCESSOR_ID);
+
+ switch (method) {
+ case TLB_CHANGE_TAG:
+ GUEST_ASSERT(!vmx->eptp_gpa);
+ vmwrite(VIRTUAL_PROCESSOR_ID, vpid + 1);
+ break;
+ case TLB_FLUSH_TAG:
+ if (vmx->eptp_gpa)
+ invept(1, vmread(EPT_POINTER));
+ else
+ invvpid(1, vpid, 0);
+ break;
+ case TLB_FLUSH_VADDR:
+ GUEST_ASSERT(!vmx->eptp_gpa);
+ invvpid(0, vpid, TEST_VADDR);
+ break;
+ case TLB_FLUSH_NONE:
+ break;
+ }
+}
+
+static void prepare_l2(void *nested_state, void *l2_guest_code)
+{
+ struct vmx_pages *vmx = nested_state;
+
+ if (this_cpu_has(X86_FEATURE_VMX)) {
+ u32 sec_exec_ctl;
+
+ prepare_for_vmx_operation(vmx);
+ load_vmcs(vmx);
+ prepare_vmcs(vmx, l2_guest_code);
+
+ /* Enable VPID */
+ sec_exec_ctl = vmread(SECONDARY_VM_EXEC_CONTROL);
+ sec_exec_ctl |= SECONDARY_EXEC_ENABLE_VPID;
+ vmwrite(SECONDARY_VM_EXEC_CONTROL, sec_exec_ctl);
+ vmwrite(VIRTUAL_PROCESSOR_ID, 1);
+ } else {
+ generic_svm_setup(nested_state, l2_guest_code);
+ }
+}
+
+static void __run_l2(void *nested_state, bool launch,
+ enum guest_flush_method method)
+{
+ struct svm_test_data *svm;
+
+ if (this_cpu_has(X86_FEATURE_VMX)) {
+ vmx_tlb_flush(nested_state, method);
+ if (launch)
+ vmlaunch();
+ else
+ vmresume();
+ GUEST_ASSERT_EQ(vmread(VM_EXIT_REASON), EXIT_REASON_VMCALL);
+ vmwrite(GUEST_RIP, vmread(GUEST_RIP) + 3); /* skip over VMCALL */
+ } else {
+ svm = nested_state;
+ svm_tlb_flush(svm, method);
+ run_guest(svm->vmcb, svm->vmcb_gpa);
+ GUEST_ASSERT_EQ(svm->vmcb->control.exit_code, SVM_EXIT_VMMCALL);
+ svm->vmcb->save.rip += 3; /* skip over VMMCALL */
+ }
+}
+
+static void run_l2(void *nested_state, bool launch)
+{
+ __run_l2(nested_state, launch, flush_method);
+}
+
+static void l2_guest_code(void)
+{
+ u64 expected_val;
+
+ for (;;) {
+ expected_val = READ_ONCE(l2_expected_val);
+ if (expected_val)
+ GUEST_ASSERT_EQ(READ_ONCE(*(u64 *)TEST_VADDR), expected_val);
+ vmcall();
+ }
+}
+
+static void l1_guest_code(void *data)
+{
+ gpa_t gpa;
+ int i;
+
+ prepare_l2(data, l2_guest_code);
+
+ /*
+ * Alternately switch the PTE (or TDP PTE) mapping TEST_VADDR between
+ * two pages containing VAL1 and VAL2, flush TLBs, and verify that L2
+ * reads the expected value.
+ */
+ for (i = 0; i < NR_ITERATIONS; i++) {
+ gpa = test_gpa[i % 2];
+
+ *pte_gva &= ~PHYSICAL_PAGE_MASK;
+ *pte_gva |= gpa & PHYSICAL_PAGE_MASK;
+
+ WRITE_ONCE(l2_expected_val, (i % 2 == 0) ? VAL1 : VAL2);
+ run_l2(data, i == 0);
+ }
+
+ GUEST_DONE();
+}
+
+/*
+ * Verify that flushing L2's ASID invalidates nested TDP translations for all
+ * TDP roots, not just the one in use at the time of the flush. L1 updates the
+ * nested TDP PTE for TEST_VADDR in root 0, flushes L2's ASID while running L2
+ * with root 1, then runs L2 with root 0 *without* flushing. L2 must observe
+ * the new mapping, as the flush invalidated all translations for the ASID.
+ *
+ * Root 1 is a full copy of root 0, such that none of the nested TDP pages are
+ * shared, i.e. KVM's shadow pages for root 0 are not reachable through the
+ * shadow root for root 1.
+ */
+static void l1_guest_code_multi_tdp_root(void *data)
+{
+ struct svm_test_data *svm = data;
+ struct vmcb *vmcb = svm->vmcb;
+ int i;
+
+ GUEST_ASSERT(this_cpu_has(X86_FEATURE_SVM));
+ GUEST_ASSERT_EQ(svm->ncr3_gpa, tdp_root_gpa[0]);
+
+ prepare_l2(data, l2_guest_code);
+
+ for (i = 0; i < NR_ITERATIONS; i++) {
+ *pte_gva &= ~PHYSICAL_PAGE_MASK;
+ *pte_gva |= test_gpa[i % 2] & PHYSICAL_PAGE_MASK;
+
+ /* Flush L2's ASID (per flush_method) while using root 1 */
+ vmcb->control.nested_cr3 = tdp_root_gpa[1];
+ WRITE_ONCE(l2_expected_val, 0);
+ run_l2(data, i == 0);
+
+ /* Switch back to root 0 without flushing */
+ vmcb->control.nested_cr3 = tdp_root_gpa[0];
+ WRITE_ONCE(l2_expected_val, (i % 2 == 0) ? VAL1 : VAL2);
+ __run_l2(data, false, TLB_FLUSH_NONE);
+ }
+
+ GUEST_DONE();
+}
+
+static void prep_guest_flush_test(struct kvm_vm *vm, bool tdp_enabled,
+ bool copy_tdp_root)
+{
+ gva_t page_gva[2], pte_map_gva;
+ gpa_t pte_gpa;
+ u64 *ptep;
+
+ /*
+ * Allocate two pages in the VM and initialize them with different
+ * values. L1 will update the mappings to switch between these pages and
+ * L2 will verify that the accessed value indicates the right page.
+ */
+ page_gva[0] = vm_alloc_page(vm);
+ page_gva[1] = vm_alloc_page(vm);
+
+ *(u64 *)addr_gva2hva(vm, page_gva[0]) = VAL1;
+ *(u64 *)addr_gva2hva(vm, page_gva[1]) = VAL2;
+
+ test_gpa[0] = addr_gva2gpa(vm, page_gva[0]);
+ test_gpa[1] = addr_gva2gpa(vm, page_gva[1]);
+
+ if (tdp_enabled) {
+ virt_pg_map(vm, TEST_VADDR, TEST_NESTED_GPA);
+ tdp_map(vm, TEST_NESTED_GPA, test_gpa[0], vm->page_size);
+ ptep = tdp_get_pte(vm, TEST_NESTED_GPA);
+ } else {
+ virt_pg_map(vm, TEST_VADDR, test_gpa[0]);
+ ptep = vm_get_pte(vm, TEST_VADDR);
+ }
+
+ /*
+ * Map the PTE (or TDP PTE) for TEST_VADDR, such that L1 can read and
+ * update the mapping. Pass the PTE GVA to L1 to directly use it (rather
+ * than L1 walking the page tables.
+ */
+ pte_gpa = addr_hva2gpa(vm, ptep);
+ pte_map_gva = vm_unused_gva_gap(vm, vm->page_size, KVM_UTIL_MIN_VADDR);
+ virt_pg_map(vm, pte_map_gva, pte_gpa & PAGE_MASK);
+ pte_gva = (u64 *)(pte_map_gva + (pte_gpa & ~PAGE_MASK));
+
+ sync_global_to_guest(vm, pte_gva);
+ sync_global_to_guest(vm, test_gpa);
+
+ if (!tdp_enabled)
+ return;
+
+ /*
+ * Must be done after all pages accessed by L2 are allocated, including
+ * nested state (e.g. L2's stack) and page tables for TEST_VADDR.
+ */
+ tdp_identity_map_default_memslots(vm);
+
+ /* Must be done after all TDP mappings are created */
+ if (copy_tdp_root) {
+ tdp_root_gpa[0] = vm->stage2_mmu.pgd;
+ tdp_root_gpa[1] = tdp_copy_page_tables(vm);
+ sync_global_to_guest(vm, tdp_root_gpa);
+ }
+}
+
+static void l2_guest_code_memslot_move(void)
+{
+ int i;
+ u64 val;
+
+ for (i = 0; i < NR_ITERATIONS; i++) {
+ val = READ_ONCE(*(u64 *)TEST_VADDR);
+ GUEST_ASSERT_EQ(val, VAL1);
+
+ vmcall();
+
+ val = READ_ONCE(*(u64 *)TEST_VADDR);
+ GUEST_ASSERT_EQ(val, VAL2);
+
+ GUEST_SYNC2(SYNC_MOVE_MEMSLOT, 1);
+
+ val = READ_ONCE(*(u64 *)TEST_VADDR);
+ GUEST_ASSERT_EQ(val, VAL1);
+
+ vmcall();
+ }
+}
+
+static void l1_guest_code_memslot_move(void *data)
+{
+ int i;
+ u64 val;
+
+ prepare_l2(data, l2_guest_code_memslot_move);
+
+ for (i = 0; i < NR_ITERATIONS; i++) {
+ /* Verify that accessing VADDR reads from page 1 in both L1 & L2 */
+ val = READ_ONCE(*(u64 *)TEST_VADDR);
+ GUEST_ASSERT_EQ(val, VAL1);
+
+ run_l2(data, i == 0);
+
+ /*
+ * Update VADDR to point at page 2. KVM should flush both L1 and
+ * L2's TLB translations so that further accesses go to page 2.
+ */
+ GUEST_SYNC2(SYNC_MOVE_MEMSLOT, 2);
+
+ /*
+ * Verify that accessing VADDR reads from page 2. A stale TLB
+ * entry would lead to reading from page 1. Enter L2 and check
+ * access to page 2, to verify that a TLB flush from L1's
+ * context flushes both L1 and L2.
+ */
+ val = READ_ONCE(*(u64 *)TEST_VADDR);
+ GUEST_ASSERT_EQ(val, VAL2);
+
+ run_l2(data, false);
+
+ /*
+ * L2 updated VADDR to point at page 1 again. Verify access to
+ * page 1 again to verify that a TLB flush from L2's context
+ * flushes both L1 and L2 TLB translations.
+ */
+ val = READ_ONCE(*(u64 *)TEST_VADDR);
+ GUEST_ASSERT_EQ(val, VAL1);
+ }
+
+ GUEST_DONE();
+}
+
+static void prep_memslot_move_test(struct kvm_vm *vm, bool tdp_enabled)
+{
+ /*
+ * Disable KVM_X86_QUIRK_SLOT_ZAP_ALL. By default, moving a memslot
+ * invalidates all MMU roots, which implicitly flushes the guest TLB
+ * and masks missing TLB flush bugs. Disabling it forces KVM to keep
+ * the same roots, relying solely on explicit TLB flushes.
+ */
+ TEST_REQUIRE(kvm_check_cap(KVM_CAP_DISABLE_QUIRKS2) & KVM_X86_QUIRK_SLOT_ZAP_ALL);
+ vm_enable_cap(vm, KVM_CAP_DISABLE_QUIRKS2, KVM_X86_QUIRK_SLOT_ZAP_ALL);
+
+ vm_userspace_mem_region_add(vm, VM_MEM_SRC_ANONYMOUS,
+ TEST_GPA, TEST_MEMSLOT,
+ /*npages=*/2, /*flags=*/0);
+
+ virt_pg_map(vm, TEST_VADDR, TEST_GPA);
+
+ if (tdp_enabled)
+ tdp_map(vm, TEST_GPA, TEST_GPA, vm->page_size);
+
+ *(u64 *)addr_gpa2hva(vm, TEST_GPA) = VAL1;
+ *(u64 *)addr_gpa2hva(vm, TEST_GPA + vm->page_size) = VAL2;
+
+ /*
+ * Must be done after all pages accessed by L2 are allocated, including
+ * nested state (e.g. L2's stack) and page tables for TEST_VADDR.
+ */
+ if (tdp_enabled)
+ tdp_identity_map_default_memslots(vm);
+}
+
+static void calc_max_svm_asid(void)
+{
+ const struct kvm_cpuid_entry2 *entry;
+
+ if (!kvm_cpu_has(X86_FEATURE_SVM))
+ return;
+
+ entry = get_cpuid_entry(kvm_get_supported_cpuid(), 0x8000000A, 0);
+ max_asid = entry ? entry->ebx : 0;
+}
+
+static void run_test(void *guest_code, bool tdp_enabled)
+{
+ struct kvm_vcpu *vcpu;
+ gva_t nested_gva = 0;
+ struct kvm_vm *vm;
+ struct ucall uc;
+
+ vm = vm_create_with_one_vcpu(&vcpu, guest_code);
+ if (tdp_enabled)
+ vm_enable_tdp(vm);
+
+ if (kvm_cpu_has(X86_FEATURE_VMX))
+ vcpu_alloc_vmx(vm, &nested_gva);
+ else
+ vcpu_alloc_svm(vm, &nested_gva);
+
+ if (guest_code == l1_guest_code_memslot_move)
+ prep_memslot_move_test(vm, tdp_enabled);
+ else if (guest_code == l1_guest_code_multi_tdp_root)
+ prep_guest_flush_test(vm, tdp_enabled, true);
+ else
+ prep_guest_flush_test(vm, tdp_enabled, false);
+
+ vcpu_args_set(vcpu, 1, nested_gva);
+
+ sync_global_to_guest(vm, flush_method);
+ sync_global_to_guest(vm, max_asid);
+
+ for (;;) {
+ vcpu_run(vcpu);
+ TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO);
+
+ switch (get_ucall(vcpu, &uc)) {
+ case UCALL_ABORT:
+ REPORT_GUEST_ASSERT(uc);
+ break;
+ case UCALL_SYNC:
+ /*
+ * Move the base GPA of the memslot between TEST_GPA and
+ * TEST_GPA - PAGE_SIZE. This effectively moves
+ * TEST_VADDR to point at page 1 or page 2 in the
+ * memslot.
+ */
+ if (uc.args[0] == SYNC_MOVE_MEMSLOT) {
+ gpa_t new_gpa = (uc.args[1] == 1) ?
+ TEST_GPA : (TEST_GPA - vm->page_size);
+
+ vm_mem_region_move(vm, TEST_MEMSLOT, new_gpa);
+ } else {
+ TEST_FAIL("Unknown sync code: %lu", uc.args[0]);
+ }
+ break;
+ case UCALL_DONE:
+ goto done;
+ default:
+ TEST_FAIL("Unknown ucall %lu", uc.cmd);
+ }
+ }
+
+done:
+ kvm_vm_free(vm);
+}
+
+#define tlb_test(test_name, guest_code, tdp_setting, flush_setting) \
+do { \
+ tdp_setting; \
+ flush_setting; \
+ \
+ if (tdp_enabled && !kvm_cpu_has_tdp()) { \
+ pr_info("Skipping: " test_name " (no TDP support)\n"); \
+ break; \
+ } \
+ \
+ /* \
+ * VPIDs do not tag guest-physical translations, i.e. changing or \
+ * flushing VPIDs does not flush EPT translations, and single-context \
+ * INVEPT only flushes the target EPTP (not all TDP roots). \
+ */ \
+ if (tdp_enabled && kvm_cpu_has_ept() && \
+ (flush_method == TLB_CHANGE_TAG || \
+ guest_code == l1_guest_code_multi_tdp_root)) { \
+ pr_info("Skipping: " test_name " (not applicable to EPTs)\n"); \
+ break; \
+ } \
+ \
+ pr_info("Testing " test_name "...\n"); \
+ run_test(guest_code, tdp_enabled); \
+} while (0)
+
+int main(int argc, char *argv[])
+{
+ bool tdp_enabled;
+
+ TEST_REQUIRE(kvm_cpu_has(X86_FEATURE_SVM) || kvm_cpu_has(X86_FEATURE_VMX));
+
+ calc_max_svm_asid();
+
+ tlb_test("Guest-triggered TLB flush (flush entire tag), TDP disabled",
+ l1_guest_code, tdp_enabled = false,
+ flush_method = TLB_FLUSH_TAG);
+ tlb_test("Guest-triggered TLB flush (flush individual address), TDP disabled",
+ l1_guest_code, tdp_enabled = false,
+ flush_method = TLB_FLUSH_VADDR);
+ tlb_test("Guest-triggered TLB flush (change tag), TDP disabled",
+ l1_guest_code, tdp_enabled = false,
+ flush_method = TLB_CHANGE_TAG);
+
+ tlb_test("Guest-triggered TLB flush (flush entire tag), TDP enabled",
+ l1_guest_code, tdp_enabled = true,
+ flush_method = TLB_FLUSH_TAG);
+ tlb_test("Guest-triggered TLB flush (change tag), TDP enabled",
+ l1_guest_code, tdp_enabled = true,
+ flush_method = TLB_CHANGE_TAG);
+
+ tlb_test("Guest-triggered TLB flush with multiple TDP roots (flush entire tag), TDP enabled",
+ l1_guest_code_multi_tdp_root, tdp_enabled = true,
+ flush_method = TLB_FLUSH_TAG);
+ tlb_test("Guest-triggered TLB flush with multiple TDP roots (change tag), TDP enabled",
+ l1_guest_code_multi_tdp_root, tdp_enabled = true,
+ flush_method = TLB_CHANGE_TAG);
+
+
+ tlb_test("KVM-triggered TLB flush via memslot move, TDP disabled",
+ l1_guest_code_memslot_move, tdp_enabled = false,
+ flush_method = TLB_FLUSH_NONE);
+ tlb_test("KVM-triggered TLB flush via memslot move, TDP enabled",
+ l1_guest_code_memslot_move, tdp_enabled = true,
+ flush_method = TLB_FLUSH_NONE);
+
+ return 0;
+}
|
| 29/3 | 2026/10/12 23:51 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 28684 output: 37 thoughts: 1212 Thoughts: **KMSAN Fuzzing Evaluation for This Patch Series** Okay, let's break this down. My initial assessment is that this patch series, focusing on TLB tag allocation and ASID management in KVM/x86 (specifically VMX and SVM), warrants careful scrutiny, particularly with KMSAN. The refactoring itself, while likely improving performance and maintainability, touches some critical areas that are ripe for potential memory safety issues. First, the scope: This touches core MMU code, ASID/VPID allocation, and nested virtualization, which increases complexity and potential for subtle bugs. The fact that it's touching both VMX and SVM makes it interesting, as any shared logic could introduce cross-hypervisor issues if not handled with extreme care. Specifically, I'm noting the introduction of `kvm_alloc_tlb_tag`, the use of a bitmap for allocation (`bitmap_zalloc`), and the associated spinlock. The allocation/deallocation patterns using those new functions (`svm_vcpu_create`, `svm_vcpu_free`, and `sev_vcpu_migrate_asid` among others) are what I'm focused on. If these allocations are not handled right, that's where uninitialized memory or use-after-free conditions might arise. The `fallback_asid` is a potential point of interest. The nested SVM TLB flushing logic (`nested_svm_entry_tlb_flush`, `nested_svm_exit_tlb_flush`) is a key area. The `KVM_REQ_MMU_SYNC_ALL_ROOTS` is always a flag for potential bugs. I'm looking for where these are used, and how they may lead to userspace or privileged memory corruption. Are there race conditions during TLB flushing that could expose uninitialized memory to a guest? The migration handling (`sev_vcpu_migrate_asid`) is also a concern. ASIDs can be tricky to manage during migration. Any inconsistency could expose the host to a guest. My primary concern now is confirming that there is no data copied to userspace, or that no uninitialized memory could be exposed to userspace, either directly or indirectly through conditional reads. Given the changes to ASID management, nested state, and TLB flushes, I need to check how nested ASID values are handled and if there is any data leakage. I need to confirm if there are any `KVM_GET_NESTED_STATE`, `svm_get_nested_state`, `svm_set_nested_state` ioctls or similar that might expose internal state. Let's do a search on `nested_state` and `nested.asid` within the relevant code, and investigate. Based on the initial assessment, the risks appear significant enough to warrant a focused KMSAN fuzzing session. |
| 30/3 | 2026/10/12 23:52 | tool | grepper |
0mArgs:{
"Expression": "svm_get_nested_state",
"PathPrefixes": [
"arch/x86/kvm/"
]
}Results: {
"Output": "arch/x86/kvm/svm/nested.c=1936=static void nested_copy_vmcb_cache_to_control(struct vmcb_control_area *dst,\n--\narch/x86/kvm/svm/nested.c-1970-\narch/x86/kvm/svm/nested.c:1971:static int svm_get_nested_state(struct kvm_vcpu *vcpu,\narch/x86/kvm/svm/nested.c-1972-\t\t\t\tstruct kvm_nested_state __user *user_kvm_nested_state,\n--\narch/x86/kvm/svm/nested.c=2041=static int svm_set_nested_state(struct kvm_vcpu *vcpu,\n--\narch/x86/kvm/svm/nested.c-2204-\narch/x86/kvm/svm/nested.c:2205:static bool svm_get_nested_state_pages(struct kvm_vcpu *vcpu)\narch/x86/kvm/svm/nested.c-2206-{\n--\narch/x86/kvm/svm/nested.c=2253=struct kvm_x86_nested_ops svm_nested_ops __initdata = {\n--\narch/x86/kvm/svm/nested.c-2258-\t.triple_fault = nested_svm_triple_fault,\narch/x86/kvm/svm/nested.c:2259:\t.get_nested_state_pages = svm_get_nested_state_pages,\narch/x86/kvm/svm/nested.c:2260:\t.get_state = svm_get_nested_state,\narch/x86/kvm/svm/nested.c-2261-\t.set_state = svm_set_nested_state,\n--\narch/x86/kvm/svm/svm.c=4645=static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)\n--\narch/x86/kvm/svm/svm.c-4809-\t *\narch/x86/kvm/svm/svm.c:4810:\t * FIXME: Rework svm_get_nested_state() to not pull data from the\narch/x86/kvm/svm/svm.c-4811-\t * cache (except for maybe int_ctl).\n"
} |
| 31/3 | 2026/10/12 23:52 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 31180 output: 43 thoughts: 58 Thoughts: Here's my analysis of those code sections: **Diving Deep into Nested SVM State Management** Okay, so I'm looking at `arch/x86/kvm/svm/nested.c`, specifically around lines 1930 to 2040. My focus right now is on understanding how we're handling the nested SVM state. Specifically, I'm digging into `svm_get_nested_state` and `nested_copy_vmcb_cache_to_control`. I need to figure out how we're actually extracting and managing the VMCB caches in this nested context. It looks like `svm_get_nested_state` is probably responsible for gathering the current state of a nested VM, right? It's likely involved in retrieving critical information about the current guest execution context. I'll need to trace the data flow to fully understand how it interacts with the underlying hardware, how it handles potential nested virtualization issues, and how it reports the nested state to the main KVM framework. Then there's `nested_copy_vmcb_cache_to_control`. This function’s name is pretty descriptive. It's likely responsible for transferring the cached VMCB data into the control structures. I need to understand what this transfer represents, and its purpose. Is this caching done for performance reasons, or to handle the nested nature of SVM, or both? Are there any potential consistency problems in the cache and control copies? Are there any data synchronization primitives used to ensure data integrity during this process? My immediate goal is to trace the execution path and identify the core data structures and mechanisms involved in copying and updating the nested VM control block caches. I'm keen to spot any potential synchronization issues or inefficiencies in this area, because this kind of code is often critical to the overall performance and stability of nested virtualization. I'm really curious to see if there are any subtle optimization opportunities or potential bug risks in these routines. I'll be looking carefully at the memory access patterns and any interactions with the SVM hardware itself. |
| 32/3 | 2026/10/12 23:52 | tool | read-file |
0mArgs:{
"File": "arch/x86/kvm/svm/nested.c",
"FirstLine": 1930,
"LineCount": 80
}Results: {
"Contents": "1930:\t\t\tkvm_calc_nested_tsc_multiplier(vcpu-\u003earch.l1_tsc_scaling_ratio,\n1931:\t\t\t\t\t\t svm-\u003etsc_ratio_msr);\n1932:\t\tsvm_write_tsc_multiplier(vcpu);\n1933:\t}\n1934:\t\n1935:\t/* Inverse operation of nested_copy_vmcb_control_to_cache() */\n1936:\tstatic void nested_copy_vmcb_cache_to_control(struct vmcb_control_area *dst,\n1937:\t\t\t\t\t\t struct vmcb_ctrl_area_cached *from)\n1938:\t{\n1939:\t\tunsigned int i;\n1940:\t\n1941:\t\tmemset(dst, 0, sizeof(struct vmcb_control_area));\n1942:\t\n1943:\t\tfor (i = 0; i \u003c MAX_INTERCEPT; i++)\n1944:\t\t\tdst-\u003eintercepts[i] = from-\u003eintercepts[i];\n1945:\t\n1946:\t\tdst-\u003eiopm_base_pa = from-\u003eiopm_base_pa;\n1947:\t\tdst-\u003emsrpm_base_pa = from-\u003emsrpm_base_pa;\n1948:\t\tdst-\u003etsc_offset = from-\u003etsc_offset;\n1949:\t\tdst-\u003easid = from-\u003easid;\n1950:\t\tdst-\u003etlb_ctl = from-\u003etlb_ctl;\n1951:\t\tdst-\u003eerap_ctl = from-\u003eerap_ctl;\n1952:\t\tdst-\u003eint_ctl = from-\u003eint_ctl;\n1953:\t\tdst-\u003eint_vector = from-\u003eint_vector;\n1954:\t\tdst-\u003eint_state = from-\u003eint_state;\n1955:\t\tdst-\u003eexit_code = from-\u003eexit_code;\n1956:\t\tdst-\u003eexit_info_1 = from-\u003eexit_info_1;\n1957:\t\tdst-\u003eexit_info_2 = from-\u003eexit_info_2;\n1958:\t\tdst-\u003eexit_int_info = from-\u003eexit_int_info;\n1959:\t\tdst-\u003eexit_int_info_err = from-\u003eexit_int_info_err;\n1960:\t\tdst-\u003emisc_ctl\t\t = from-\u003emisc_ctl;\n1961:\t\tdst-\u003eevent_inj = from-\u003eevent_inj;\n1962:\t\tdst-\u003eevent_inj_err = from-\u003eevent_inj_err;\n1963:\t\tdst-\u003enext_rip = from-\u003enext_rip;\n1964:\t\tdst-\u003enested_cr3\t\t = from-\u003enested_cr3;\n1965:\t\tdst-\u003emisc_ctl2\t\t = from-\u003emisc_ctl2;\n1966:\t\tdst-\u003epause_filter_count = from-\u003epause_filter_count;\n1967:\t\tdst-\u003epause_filter_thresh = from-\u003epause_filter_thresh;\n1968:\t\t/* 'clean' and 'hv_enlightenments' are not changed by KVM */\n1969:\t}\n1970:\t\n1971:\tstatic int svm_get_nested_state(struct kvm_vcpu *vcpu,\n1972:\t\t\t\t\tstruct kvm_nested_state __user *user_kvm_nested_state,\n1973:\t\t\t\t\tu32 user_data_size)\n1974:\t{\n1975:\t\tstruct vcpu_svm *svm;\n1976:\t\tstruct vmcb_control_area *ctl;\n1977:\t\tunsigned long r;\n1978:\t\tstruct kvm_nested_state kvm_state = {\n1979:\t\t\t.flags = 0,\n1980:\t\t\t.format = KVM_STATE_NESTED_FORMAT_SVM,\n1981:\t\t\t.size = sizeof(kvm_state),\n1982:\t\t};\n1983:\t\tstruct vmcb __user *user_vmcb = (struct vmcb __user *)\n1984:\t\t\t\u0026user_kvm_nested_state-\u003edata.svm[0];\n1985:\t\n1986:\t\tif (!vcpu)\n1987:\t\t\treturn kvm_state.size + KVM_STATE_NESTED_SVM_VMCB_SIZE;\n1988:\t\n1989:\t\tsvm = to_svm(vcpu);\n1990:\t\n1991:\t\tif (user_data_size \u003c kvm_state.size)\n1992:\t\t\tgoto out;\n1993:\t\n1994:\t\t/* First fill in the header and copy it out. */\n1995:\t\tif (is_guest_mode(vcpu)) {\n1996:\t\t\tkvm_state.hdr.svm.vmcb_pa = svm-\u003enested.vmcb12_gpa;\n1997:\t\t\tkvm_state.hdr.svm.gpat = 0;\n1998:\t\t\tif (l2_has_separate_pat(vcpu))\n1999:\t\t\t\tkvm_state.hdr.svm.gpat = svm-\u003evmcb-\u003esave.g_pat;\n2000:\t\t\tkvm_state.size += KVM_STATE_NESTED_SVM_VMCB_SIZE;\n2001:\t\t\tkvm_state.flags |= KVM_STATE_NESTED_GUEST_MODE;\n2002:\t\n2003:\t\t\tif (vcpu-\u003earch.nested_run_pending)\n2004:\t\t\t\tkvm_state.flags |= KVM_STATE_NESTED_RUN_PENDING;\n2005:\t\t}\n2006:\t\n2007:\t\tif (gif_set(svm))\n2008:\t\t\tkvm_state.flags |= KVM_STATE_NESTED_GIF_SET;\n2009:\t\n"
} |
| 33/3 | 2026/10/12 23:52 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 33213 output: 43 thoughts: 17 Thoughts: Okay, here's my attempt at summarizing that thought process, tailored for an expert audience, and written in the first person: **Delving into `svm_get_nested_state`** Alright, let's get down to brass tacks. I'm focusing in on `svm_get_nested_state` now. I'm immediately curious about the next logical step within that function. My brain's already anticipating the flow, the conditional branches, any potential edge cases that might emerge. I'm mentally tracing the call stack, wondering if we're dealing with a deeply nested structure or if this is relatively shallow. Given the "nested" aspect, I'm expecting some kind of recursive interaction or at least a complex series of state transitions that this function is meant to unravel. What's it going to reveal? The specific implementation details are what I want to dive into, since understanding the data dependencies in this section is always important. |
| 34/3 | 2026/10/12 23:52 | tool | read-file |
0mArgs:{
"File": "arch/x86/kvm/svm/nested.c",
"FirstLine": 2010,
"LineCount": 40
}Results: {
"Contents": "2010:\t\tif (copy_to_user(user_kvm_nested_state, \u0026kvm_state, sizeof(kvm_state)))\n2011:\t\t\treturn -EFAULT;\n2012:\t\n2013:\t\tif (!is_guest_mode(vcpu))\n2014:\t\t\tgoto out;\n2015:\t\n2016:\t\t/*\n2017:\t\t * Copy over the full size of the VMCB rather than just the size\n2018:\t\t * of the structs.\n2019:\t\t */\n2020:\t\tif (clear_user(user_vmcb, KVM_STATE_NESTED_SVM_VMCB_SIZE))\n2021:\t\t\treturn -EFAULT;\n2022:\t\n2023:\t\tctl = kzalloc_obj(*ctl);\n2024:\t\tif (!ctl)\n2025:\t\t\treturn -ENOMEM;\n2026:\t\n2027:\t\tnested_copy_vmcb_cache_to_control(ctl, \u0026svm-\u003enested.ctl);\n2028:\t\tr = copy_to_user(\u0026user_vmcb-\u003econtrol, ctl,\n2029:\t\t\t\t sizeof(user_vmcb-\u003econtrol));\n2030:\t\tkfree(ctl);\n2031:\t\tif (r)\n2032:\t\t\treturn -EFAULT;\n2033:\t\n2034:\t\tif (copy_to_user(\u0026user_vmcb-\u003esave, \u0026svm-\u003evmcb01.ptr-\u003esave,\n2035:\t\t\t\t sizeof(user_vmcb-\u003esave)))\n2036:\t\t\treturn -EFAULT;\n2037:\tout:\n2038:\t\treturn kvm_state.size;\n2039:\t}\n2040:\t\n2041:\tstatic int svm_set_nested_state(struct kvm_vcpu *vcpu,\n2042:\t\t\t\t\tstruct kvm_nested_state __user *user_kvm_nested_state,\n2043:\t\t\t\t\tstruct kvm_nested_state *kvm_state)\n2044:\t{\n2045:\t\tstruct vcpu_svm *svm = to_svm(vcpu);\n2046:\t\tstruct vmcb __user *user_vmcb = (struct vmcb __user *)\n2047:\t\t\t\u0026user_kvm_nested_state-\u003edata.svm[0];\n2048:\t\tstruct vmcb_control_area *ctl;\n2049:\t\tstruct vmcb_save_area *save;\n"
} |
| 35/3 | 2026/10/12 23:52 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 34125 output: 57 thoughts: 330 Thoughts: **Analyzing the Recent Code Changes** Okay, let's break this down. First, I'm noting the standard initialization patterns here, especially the use of `kzalloc_obj` and `memset` for `ctl` and the `kvm_state`. Good practice, nothing out of the ordinary, and it seems this part of the code wasn't even touched by the patch. Now, the interesting part: I need to understand what's changed around the `struct svm_nested_state`. The diff in `arch/x86/kvm/svm/svm.h` immediately catches my eye. It looks like the `initialized` flag has been replaced with `asid02` (of type `kvm_tlb_tag_t`) and a new `last_asid` member of type `u32` has been added. That's a significant change; these fields are clearly related to TLB management and ASID handling, possibly at the nested virtualization level. The `force_msr_bitmap_recalc` flag gives me a good point of reference for where these changes might fit in, as it could be interacting with the MSR bitmap for L2. My next step is to locate the `struct vcpu_svm` allocation or the call to `svm_create_vcpu`. This is where the `svm_nested_state` is actually used, so I need to see how these newly added or changed fields are being utilized in context. I'll search for those identifiers within the codebase to understand the implications of this refactoring. |
| 36/3 | 2026/10/12 23:52 | tool | grepper |
0mArgs:{
"Expression": "kmem_cache_zalloc.*kvm_vcpu|svm_create_vcpu|svm_vcpu_create",
"PathPrefixes": [
"arch/x86/kvm/"
]
}Results: {
"Output": "arch/x86/kvm/svm/svm.c=1333=void svm_switch_vmcb(struct vcpu_svm *svm, struct kvm_vmcb_info *target_vmcb)\n--\narch/x86/kvm/svm/svm.c-1338-\narch/x86/kvm/svm/svm.c:1339:static int svm_vcpu_create(struct kvm_vcpu *vcpu)\narch/x86/kvm/svm/svm.c-1340-{\n--\narch/x86/kvm/svm/svm.c=5491=struct kvm_x86_ops svm_x86_ops __initdata = {\n--\narch/x86/kvm/svm/svm.c-5502-\t.vcpu_precreate = avic_vcpu_precreate,\narch/x86/kvm/svm/svm.c:5503:\t.vcpu_create = svm_vcpu_create,\narch/x86/kvm/svm/svm.c-5504-\t.vcpu_free = svm_vcpu_free,\n"
} |
| 37/3 | 2026/10/12 23:52 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 35147 output: 39 thoughts: 107 Thoughts: **Investigating Virtual CPU Allocation** Okay, so I need to understand how the virtual CPU (`vcpu`) and its Secure Virtual Machine counterpart (`vcpu_svm`) are being allocated within the kernel. My initial thought is to start with the `x86.c` file, looking at the `kvm_arch_vcpu_create` function. I seem to recall it might use either `kmem_cache_zalloc` to allocate from `x86_fpu_cache` or potentially the `kvm_vcpu_cache`. To be sure, I should just grep for `vcpu_create` within the `arch/x86/kvm/x86.c` directory. This should help me quickly identify all the relevant call sites. This is a common and efficient way to track how these resources are allocated and initialized. |
| 38/3 | 2026/10/12 23:52 | tool | grepper |
0mArgs:{
"Expression": "kvm_arch_vcpu_create",
"PathPrefixes": [
"arch/x86/kvm/"
]
}Results: {
"Output": "arch/x86/kvm/vmx/tdx.c=825=void tdx_vcpu_put(struct kvm_vcpu *vcpu)\n--\narch/x86/kvm/vmx/tdx.c-840- * 3.1 tdx_vcpu_create() transitions vCPU state to VCPU_TD_STATE_UNINITIALIZED.\narch/x86/kvm/vmx/tdx.c:841: * 3.2 vcpu_load() and vcpu_put() in kvm_arch_vcpu_create().\narch/x86/kvm/vmx/tdx.c:842: * 3.3 (conditional) if any error encountered after kvm_arch_vcpu_create()\narch/x86/kvm/vmx/tdx.c-843- * kvm_arch_vcpu_destroy() --\u003e tdx_vcpu_free().\n--\narch/x86/kvm/x86.c=9397=int kvm_arch_vcpu_precreate(struct kvm *kvm, unsigned int id)\n--\narch/x86/kvm/x86.c-9435-\narch/x86/kvm/x86.c:9436:int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)\narch/x86/kvm/x86.c-9437-{\n"
} |
| 39/3 | 2026/10/12 23:52 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 35850 output: 35 thoughts: 1959 Thoughts: **Analysis of Potential Uninitialized Memory Access in KVM/SVM Code** Alright, let's break this down. My primary concern is ensuring that the code doesn't inadvertently use uninitialized memory, potentially leading to security vulnerabilities or unexpected behavior. I started by looking at the lifecycle of the `vcpu_svm` structure and its nested structures. The allocation in `kvm_vm_ioctl_create_vcpu` using `kmem_cache_zalloc` is promising, as it guarantees zero-initialization of the entire `vcpu_svm` structure. Furthermore, the explicit setting of `svm->nested.last_asid` to 0 in `svm_allocate_nested` is a good practice, eliminating a potential source of uninitialized data. Next, the TLB tag management is sound. The `bitmap_zalloc` calls in `kvm_init_tlb_tags` ensures the bitmaps are cleared on initialization. The subsequent functions `kvm_alloc_tlb_tag`, `kvm_free_tlb_tag`, and `kvm_destroy_tlb_tags` all operate on the bitmap safely, ensuring no uninitialized bits are accessed. I then reviewed the `__kvm_hv_purge_tlb_flush_fifo` function. The use of `kfifo_reset_out` to reset the read pointer of the kfifo does not present a problem. The `invlpga_interception` function, which is critical for handling INVLPGA instructions from the guest, initially seemed fine. The registers read into `gva` and `asid` were from vcpu registers, which should be initialized. However, the use of `svm->nested.last_asid` sparked a question. Where does this structure come from? The investigation has lead me to inquire about the origins of the `svm->nested.ctl` structure. To proceed, I'll need to locate all instances where `nested.ctl` is used within the `arch/x86/kvm/svm/` directory to understand how it's initialized and updated. This will determine if there's a potential window for uninitialized memory access related to TLB control within the nested virtualization setup. |
| 40/3 | 2026/10/12 23:53 | tool | grepper |
0mArgs:{
"Expression": "nested\\.ctl",
"PathPrefixes": [
"arch/x86/kvm/svm/"
]
}Results: {
"Output": "\nFull output is too long, showing 200 out of 290 lines.\nUse more precise expression or PathPrefixes if possible.\n\nNumber of matching lines per file (4 files in total):\narch/x86/kvm/svm/hyperv.h:2\narch/x86/kvm/svm/nested.c:41\narch/x86/kvm/svm/svm.c:10\narch/x86/kvm/svm/svm.h:7\n\narch/x86/kvm/svm/hyperv.h=15=static inline void nested_svm_hv_update_vm_vp_ids(struct kvm_vcpu *vcpu)\n--\narch/x86/kvm/svm/hyperv.h-17-\tstruct vcpu_svm *svm = to_svm(vcpu);\narch/x86/kvm/svm/hyperv.h:18:\tstruct hv_vmcb_enlightenments *hve = \u0026svm-\u003enested.ctl.hv_enlightenments;\narch/x86/kvm/svm/hyperv.h-19-\tstruct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu);\n--\narch/x86/kvm/svm/hyperv.h=29=static inline bool nested_svm_l2_tlb_flush_enabled(struct kvm_vcpu *vcpu)\n--\narch/x86/kvm/svm/hyperv.h-31-\tstruct vcpu_svm *svm = to_svm(vcpu);\narch/x86/kvm/svm/hyperv.h:32:\tstruct hv_vmcb_enlightenments *hve = \u0026svm-\u003enested.ctl.hv_enlightenments;\narch/x86/kvm/svm/hyperv.h-33-\tstruct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu);\n--\narch/x86/kvm/svm/nested.c=74=static u64 nested_svm_get_tdp_pdptr(struct kvm_vcpu *vcpu, int index)\n--\narch/x86/kvm/svm/nested.c-76-\tstruct vcpu_svm *svm = to_svm(vcpu);\narch/x86/kvm/svm/nested.c:77:\tu64 cr3 = svm-\u003enested.ctl.nested_cr3;\narch/x86/kvm/svm/nested.c-78-\tu64 pdpte;\n--\narch/x86/kvm/svm/nested.c=92=static unsigned long nested_svm_get_tdp_cr3(struct kvm_vcpu *vcpu)\n--\narch/x86/kvm/svm/nested.c-95-\narch/x86/kvm/svm/nested.c:96:\treturn svm-\u003enested.ctl.nested_cr3;\narch/x86/kvm/svm/nested.c-97-}\n--\narch/x86/kvm/svm/nested.c=99=static void nested_svm_init_mmu_context(struct kvm_vcpu *vcpu)\n--\narch/x86/kvm/svm/nested.c-113-\t\t\t\tsvm-\u003evmcb01.ptr-\u003esave.efer,\narch/x86/kvm/svm/nested.c:114:\t\t\t\tsvm-\u003enested.ctl.nested_cr3,\narch/x86/kvm/svm/nested.c:115:\t\t\t\tsvm-\u003enested.ctl.misc_ctl);\narch/x86/kvm/svm/nested.c-116-\n--\narch/x86/kvm/svm/nested.c=127=static bool nested_vmcb_needs_vls_intercept(struct vcpu_svm *svm)\n--\narch/x86/kvm/svm/nested.c-134-\narch/x86/kvm/svm/nested.c:135:\tif (!(svm-\u003enested.ctl.misc_ctl2 \u0026 SVM_MISC2_ENABLE_V_VMLOAD_VMSAVE))\narch/x86/kvm/svm/nested.c-136-\t\treturn true;\n--\narch/x86/kvm/svm/nested.c=141=void nested_vmcb02_recalc_intercepts(struct vcpu_svm *svm)\narch/x86/kvm/svm/nested.c-142-{\narch/x86/kvm/svm/nested.c:143:\tstruct vmcb_ctrl_area_cached *vmcb12_ctrl = \u0026svm-\u003enested.ctl;\narch/x86/kvm/svm/nested.c-144-\tstruct vmcb *vmcb02 = svm-\u003enested.vmcb02.ptr;\n--\narch/x86/kvm/svm/nested.c-185-\t */\narch/x86/kvm/svm/nested.c:186:\tif (!vmcb12_is_intercept(\u0026svm-\u003enested.ctl, INTERCEPT_PAUSE))\narch/x86/kvm/svm/nested.c-187-\t\tvmcb_clr_intercept(\u0026vmcb02-\u003econtrol, INTERCEPT_PAUSE);\n--\narch/x86/kvm/svm/nested.c=295=static bool nested_svm_merge_msrpm(struct kvm_vcpu *vcpu)\n--\narch/x86/kvm/svm/nested.c-311-\tif (!svm-\u003enested.force_msr_bitmap_recalc) {\narch/x86/kvm/svm/nested.c:312:\t\tstruct hv_vmcb_enlightenments *hve = \u0026svm-\u003enested.ctl.hv_enlightenments;\narch/x86/kvm/svm/nested.c-313-\n--\narch/x86/kvm/svm/nested.c-315-\t\t hve-\u003ehv_enlightenments_control.msr_bitmap \u0026\u0026\narch/x86/kvm/svm/nested.c:316:\t\t (svm-\u003enested.ctl.clean \u0026 BIT(HV_VMCB_NESTED_ENLIGHTENMENTS)))\narch/x86/kvm/svm/nested.c-317-\t\t\tgoto set_msrpm_base_pa;\n--\narch/x86/kvm/svm/nested.c-320-\narch/x86/kvm/svm/nested.c:321:\tif (!(vmcb12_is_intercept(\u0026svm-\u003enested.ctl, INTERCEPT_MSR_PROT)))\narch/x86/kvm/svm/nested.c-322-\t\treturn true;\n--\narch/x86/kvm/svm/nested.c-328-\narch/x86/kvm/svm/nested.c:329:\t\tgpa = svm-\u003enested.ctl.msrpm_base_pa + (p * sizeof(l1_val));\narch/x86/kvm/svm/nested.c-330-\n--\narch/x86/kvm/svm/nested.c=487=int nested_svm_check_cached_vmcb12(struct kvm_vcpu *vcpu)\n--\narch/x86/kvm/svm/nested.c-492-\t\t\t\t l2_has_separate_pat(vcpu)) ||\narch/x86/kvm/svm/nested.c:493:\t !nested_vmcb_check_controls(vcpu, \u0026svm-\u003enested.ctl))\narch/x86/kvm/svm/nested.c-494-\t\treturn -EINVAL;\n--\narch/x86/kvm/svm/nested.c=571=void nested_copy_vmcb_control_to_cache(struct vcpu_svm *svm,\n--\narch/x86/kvm/svm/nested.c-573-{\narch/x86/kvm/svm/nested.c:574:\t__nested_copy_vmcb_control_to_cache(\u0026svm-\u003evcpu, \u0026svm-\u003enested.ctl, control);\narch/x86/kvm/svm/nested.c-575-}\n--\narch/x86/kvm/svm/nested.c=621=void nested_sync_control_from_vmcb02(struct vcpu_svm *svm)\n--\narch/x86/kvm/svm/nested.c-623-\tu32 mask;\narch/x86/kvm/svm/nested.c:624:\tsvm-\u003enested.ctl.event_inj = svm-\u003evmcb-\u003econtrol.event_inj;\narch/x86/kvm/svm/nested.c:625:\tsvm-\u003enested.ctl.event_inj_err = svm-\u003evmcb-\u003econtrol.event_inj_err;\narch/x86/kvm/svm/nested.c:626:\tsvm-\u003enested.ctl.int_state\t= svm-\u003evmcb-\u003econtrol.int_state;\narch/x86/kvm/svm/nested.c-627-\n--\narch/x86/kvm/svm/nested.c-638-\tif (svm_is_intercept(svm, INTERCEPT_VINTR) \u0026\u0026\narch/x86/kvm/svm/nested.c:639:\t !vmcb12_is_intercept(\u0026svm-\u003enested.ctl, INTERCEPT_VINTR))\narch/x86/kvm/svm/nested.c-640-\t\tmask \u0026= ~V_IRQ_MASK;\n--\narch/x86/kvm/svm/nested.c-647-\narch/x86/kvm/svm/nested.c:648:\tsvm-\u003enested.ctl.int_ctl \u0026= ~mask;\narch/x86/kvm/svm/nested.c:649:\tsvm-\u003enested.ctl.int_ctl |= svm-\u003evmcb-\u003econtrol.int_ctl \u0026 mask;\narch/x86/kvm/svm/nested.c-650-}\n--\narch/x86/kvm/svm/nested.c=689=static void nested_svm_entry_tlb_flush(struct kvm_vcpu *vcpu)\n--\narch/x86/kvm/svm/nested.c-696-\narch/x86/kvm/svm/nested.c:697:\tif (svm-\u003enested.ctl.asid != svm-\u003enested.last_asid) {\narch/x86/kvm/svm/nested.c:698:\t\tsvm-\u003enested.last_asid = svm-\u003enested.ctl.asid;\narch/x86/kvm/svm/nested.c-699-\t\tnew_asid = true;\n--\narch/x86/kvm/svm/nested.c-723-\t */\narch/x86/kvm/svm/nested.c:724:\tif (new_asid || (svm-\u003enested.ctl.tlb_ctl != TLB_CONTROL_DO_NOTHING)) {\narch/x86/kvm/svm/nested.c-725-\t\tif (nested_npt_enabled(svm))\n--\narch/x86/kvm/svm/nested.c=740=static void nested_svm_exit_tlb_flush(struct kvm_vcpu *vcpu)\n--\narch/x86/kvm/svm/nested.c-746-\t/* Flush L1's own ASID if it requested a *full* TLB flush on VMRUN */\narch/x86/kvm/svm/nested.c:747:\tif (svm-\u003enested.ctl.tlb_ctl == TLB_CONTROL_FLUSH_ALL_ASID)\narch/x86/kvm/svm/nested.c-748-\t\tkvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu);\n--\narch/x86/kvm/svm/nested.c=782=static void nested_vmcb02_prepare_save(struct vcpu_svm *svm)\narch/x86/kvm/svm/nested.c-783-{\narch/x86/kvm/svm/nested.c:784:\tstruct vmcb_ctrl_area_cached *control = \u0026svm-\u003enested.ctl;\narch/x86/kvm/svm/nested.c-785-\tstruct vmcb_save_area_cached *save = \u0026svm-\u003enested.save;\n--\narch/x86/kvm/svm/nested.c=894=static void nested_vmcb02_prepare_control(struct vcpu_svm *svm)\n--\narch/x86/kvm/svm/nested.c-898-\narch/x86/kvm/svm/nested.c:899:\tstruct vmcb_ctrl_area_cached *vmcb12_ctrl = \u0026svm-\u003enested.ctl;\narch/x86/kvm/svm/nested.c-900-\tstruct vmcb *vmcb02 = svm-\u003enested.vmcb02.ptr;\n--\narch/x86/kvm/svm/nested.c-984-\t\tvmcb02-\u003econtrol.misc_ctl \u0026= ~SVM_MISC_ENABLE_GMET;\narch/x86/kvm/svm/nested.c:985:\t\tvmcb02-\u003econtrol.misc_ctl |= (svm-\u003enested.ctl.misc_ctl \u0026 SVM_MISC_ENABLE_GMET);\narch/x86/kvm/svm/nested.c-986-\n--\narch/x86/kvm/svm/nested.c=1093=int enter_svm_guest_mode(struct kvm_vcpu *vcpu, u64 vmcb12_gpa, bool from_vmrun)\n--\narch/x86/kvm/svm/nested.c-1095-\tstruct vcpu_svm *svm = to_svm(vcpu);\narch/x86/kvm/svm/nested.c:1096:\tstruct vmcb_ctrl_area_cached *control = \u0026svm-\u003enested.ctl;\narch/x86/kvm/svm/nested.c-1097-\tstruct vmcb_save_area_cached *save = \u0026svm-\u003enested.save;\n--\narch/x86/kvm/svm/nested.c=1318=static int nested_svm_vmexit_update_vmcb12(struct kvm_vcpu *vcpu)\n--\narch/x86/kvm/svm/nested.c-1368-\narch/x86/kvm/svm/nested.c:1369:\tif (svm-\u003enested.ctl.misc_ctl2 \u0026 SVM_MISC2_ENABLE_V_LBR)\narch/x86/kvm/svm/nested.c-1370-\t\tsvm_copy_lbrs(\u0026vmcb12-\u003esave, \u0026vmcb02-\u003esave);\n--\narch/x86/kvm/svm/nested.c-1373-\tvmcb12-\u003econtrol.event_inj_err\t = 0;\narch/x86/kvm/svm/nested.c:1374:\tvmcb12-\u003econtrol.int_ctl = svm-\u003enested.ctl.int_ctl;\narch/x86/kvm/svm/nested.c-1375-\n--\narch/x86/kvm/svm/nested.c=1386=void nested_svm_vmexit(struct vcpu_svm *svm)\n--\narch/x86/kvm/svm/nested.c-1445-\narch/x86/kvm/svm/nested.c:1446:\tif (!(svm-\u003enested.ctl.misc_ctl2 \u0026 SVM_MISC2_ENABLE_V_LBR)) {\narch/x86/kvm/svm/nested.c-1447-\t\tsvm_copy_lbrs(\u0026vmcb01-\u003esave, \u0026vmcb02-\u003esave);\n--\narch/x86/kvm/svm/nested.c-1485-\narch/x86/kvm/svm/nested.c:1486:\tsvm-\u003enested.ctl.nested_cr3 = 0;\narch/x86/kvm/svm/nested.c-1487-\n--\narch/x86/kvm/svm/nested.c=1538=static void nested_svm_triple_fault(struct kvm_vcpu *vcpu)\n--\narch/x86/kvm/svm/nested.c-1541-\narch/x86/kvm/svm/nested.c:1542:\tif (!vmcb12_is_intercept(\u0026svm-\u003enested.ctl, INTERCEPT_SHUTDOWN))\narch/x86/kvm/svm/nested.c-1543-\t\treturn;\n--\narch/x86/kvm/svm/nested.c=1652=static int nested_svm_exit_handled_msr(struct vcpu_svm *svm)\narch/x86/kvm/svm/nested.c-1653-{\narch/x86/kvm/svm/nested.c:1654:\tgpa_t base = svm-\u003enested.ctl.msrpm_base_pa;\narch/x86/kvm/svm/nested.c-1655-\tint write, bit_nr;\n--\narch/x86/kvm/svm/nested.c-1658-\narch/x86/kvm/svm/nested.c:1659:\tif (!(vmcb12_is_intercept(\u0026svm-\u003enested.ctl, INTERCEPT_MSR_PROT)))\narch/x86/kvm/svm/nested.c-1660-\t\treturn NESTED_EXIT_HOST;\n--\narch/x86/kvm/svm/nested.c=1677=static int nested_svm_intercept_ioio(struct vcpu_svm *svm)\n--\narch/x86/kvm/svm/nested.c-1683-\narch/x86/kvm/svm/nested.c:1684:\tif (!(vmcb12_is_intercept(\u0026svm-\u003enested.ctl, INTERCEPT_IOIO_PROT)))\narch/x86/kvm/svm/nested.c-1685-\t\treturn NESTED_EXIT_HOST;\n--\narch/x86/kvm/svm/nested.c-1689-\t\tSVM_IOIO_SIZE_SHIFT;\narch/x86/kvm/svm/nested.c:1690:\tgpa = svm-\u003enested.ctl.iopm_base_pa + (port / 8);\narch/x86/kvm/svm/nested.c-1691-\tstart_bit = port % 8;\n--\narch/x86/kvm/svm/nested.c=1702=static int nested_svm_intercept(struct vcpu_svm *svm)\n--\narch/x86/kvm/svm/nested.c-1725-\tdefault:\narch/x86/kvm/svm/nested.c:1726:\t\tif (vmcb12_is_intercept(\u0026svm-\u003enested.ctl, exit_code))\narch/x86/kvm/svm/nested.c-1727-\t\t\tvmexit = NESTED_EXIT_DONE;\n--\narch/x86/kvm/svm/nested.c=1761=static bool nested_svm_is_exception_vmexit(struct kvm_vcpu *vcpu, u8 vector,\n--\narch/x86/kvm/svm/nested.c-1765-\narch/x86/kvm/svm/nested.c:1766:\treturn (svm-\u003enested.ctl.intercepts[INTERCEPT_EXCEPTION] \u0026 BIT(vector));\narch/x86/kvm/svm/nested.c-1767-}\n--\narch/x86/kvm/svm/nested.c=1804=static inline bool nested_exit_on_init(struct vcpu_svm *svm)\narch/x86/kvm/svm/nested.c-1805-{\narch/x86/kvm/svm/nested.c:1806:\treturn vmcb12_is_intercept(\u0026svm-\u003enested.ctl, INTERCEPT_INIT);\narch/x86/kvm/svm/nested.c-1807-}\n--\narch/x86/kvm/svm/nested.c=1971=static int svm_get_nested_state(struct kvm_vcpu *vcpu,\n--\narch/x86/kvm/svm/nested.c-2026-\narch/x86/kvm/svm/nested.c:2027:\tnested_copy_vmcb_cache_to_control(ctl, \u0026svm-\u003enested.ctl);\narch/x86/kvm/svm/nested.c-2028-\tr = copy_to_user(\u0026user_vmcb-\u003econtrol, ctl,\n--\narch/x86/kvm/svm/nested.c=2235=static gpa_t svm_translate_nested_gpa(struct kvm_vcpu *vcpu, gpa_t gpa,\n\n"
} |
| 41/3 | 2026/10/12 23:53 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 43641 output: 351 thoughts: 4672 Thoughts: **KMSAN Assessment: KVM Nested Virtualization and TLB Tagging Patch** Alright, let's break this down. This patch is fundamentally about improving and unifying the way KVM tracks and manages TLB tags, particularly for ASIDs in AMD's nested SVM and potentially for VPIDs in Intel's VMX. The key changes involve: a new TLB tag allocator using bitmaps, refactoring nested ASID management, adding a new MMU sync request, and updating related code. My initial thought was to dig into where these structures are used and if KMSAN needs to be enabled. However, on closer inspection, it's clear that this patch *doesn't* introduce the kind of uninitialized memory usage that KMSAN is designed to catch. The new TLB tag bitmap is allocated with `bitmap_zalloc`, meaning it's guaranteed to be zero-initialized. Any new data structures or fields, like the added `asid`, `asid02`, and `last_asid` fields within `vcpu_svm` and nested structures, are allocated using `kmem_cache_zalloc` and explicitly initialized to reasonable values (like `0`), ensuring no garbage values are ever read. Furthermore, I don't see any changes that expose kernel structures with uninitialized data to user space. It is essential that all data copied to userspace is correctly initialized and zeroed. The ASID and TLB state are internal and not copied out. The potential failure modes here are logic errors (e.g., stale TLB entries, incorrect ASID usage), concurrency bugs (races in the bitmap allocator, lock contention), and *potentially* out-of-bounds access if the new TLB tag arithmetic is wrong. But, those are the types of bugs that KASAN, LOCKDEP, KCSAN, and general bug checks are designed to detect. This patch does not seem to introduce any risk of uninitialized memory usage or info-leaks, KMSAN would not be helpful. In summary, given that all memory allocations are properly initialized and no kernel data is exposed to user space in a manner where it might be possible to leak uninitialized data, setting `NeedsKMSAN=false` is the correct choice here. |
| 42/3 | 2026/10/12 23:53 | tool | set-results |
0mArgs:{
"KMSANReasoning": "The patch series refactors KVM's TLB tag (VPID/ASID) allocation and updates nested SVM TLB flushing and ASID management:\n1. It replaces the separate VMX VPID bitmap allocator with a generic TLB tag allocator (`kvm_init_tlb_tags`, `kvm_alloc_tlb_tag`, `kvm_free_tlb_tag`), using `bitmap_zalloc()` which guarantees zero-initialized allocations.\n2. It tracks nested ASIDs and flushes on SVM (`asid02`, `last_asid`), explicitly initializing `last_asid = 0` and properly allocating/freeing ASIDs. The parent `struct vcpu_svm` is allocated via `kmem_cache_zalloc()`.\n3. No new kernel-to-userspace copies or information leak vectors are introduced. The internal fields (`asid02`, `last_asid`, `fallback_asid`) are internal to KVM and not copied to userspace via `KVM_GET_NESTED_STATE` or any other ioctl.\n4. Surrounding buffer accesses, bounds checks, and data length calculations are unaffected.\nAny potential risks introduced by this patch series (such as bitmap indexing errors, concurrency/locking races, use-after-free or double-free of tags) are detected by KASAN, LOCKDEP, and built-in WARN_ON_ONCE checks. There are no uninitialized memory risks that would uniquely warrant KMSAN fuzzing.",
"NeedsKMSAN": false
}Results: {
"KMSANReasoning": "The patch series refactors KVM's TLB tag (VPID/ASID) allocation and updates nested SVM TLB flushing and ASID management:\n1. It replaces the separate VMX VPID bitmap allocator with a generic TLB tag allocator (`kvm_init_tlb_tags`, `kvm_alloc_tlb_tag`, `kvm_free_tlb_tag`), using `bitmap_zalloc()` which guarantees zero-initialized allocations.\n2. It tracks nested ASIDs and flushes on SVM (`asid02`, `last_asid`), explicitly initializing `last_asid = 0` and properly allocating/freeing ASIDs. The parent `struct vcpu_svm` is allocated via `kmem_cache_zalloc()`.\n3. No new kernel-to-userspace copies or information leak vectors are introduced. The internal fields (`asid02`, `last_asid`, `fallback_asid`) are internal to KVM and not copied to userspace via `KVM_GET_NESTED_STATE` or any other ioctl.\n4. Surrounding buffer accesses, bounds checks, and data length calculations are unaffected.\nAny potential risks introduced by this patch series (such as bitmap indexing errors, concurrency/locking races, use-after-free or double-free of tags) are detected by KASAN, LOCKDEP, and built-in WARN_ON_ONCE checks. There are no uninitialized memory risks that would uniquely warrant KMSAN fuzzing.",
"NeedsKMSAN": false
} |
| Total Calls | Total Tokens | Avg Tokens | Total Duration (Seconds) | Avg Duration (Seconds) |
|---|
| Total Calls | Total Duration (Seconds) | Avg Duration (Seconds) |
|---|