On a host with an emulated vPMU (guest PMU MSR accesses trap and each guest counter is backed by a host perf_event), the per-emulated- instruction PMU accounting added by commit 9cd803d496e7 ("KVM: x86: Update vPMCs when retiring instructions") is pathologically expensive. kvm_pmu_incr_counter() requests a counter reprogram (KVM_REQ_PMU) on every emulated instruction that matches a programmed counter. The reprogram is drained on the vCPU's next VM-entry, where reprogram_counter() runs the full pmc_pause_counter() + perf_event_period() + perf_event_enable() sequence: ctx->mutex, a ctx_resched() of the PMU context, and a burst of serialized PMU-MSR writes. Because the batched reprogram is serviced on the next VM-entry regardless of which exit preceded it, ordinary exits -- notably the guest's 1kHz timer tick -- absorb the cost, inflating timer interrupts into the hundreds of microseconds and, under some workloads, escalating to a guest CSD lockup. Add emulated_counter_reprogram_tolerance, a module parameter bounding how many emulated instructions may accumulate on a counter before KVM forces a reprogram. 0 (default) preserves today's behavior of reprogramming on every emulated instruction. A larger value batches the reprograms. Batching does not lose counts: pmc_read_counter() and reprogram_counter() already fold pmc->emulated_counter into the value observed on guest counter reads, so RDPMC/RDMSR remain accurate. The only effect is that an overflow-driven PMI may be delivered up to "tolerance" instructions late -- acceptable given the overflow PMI is not cycle-accurate to begin with. The knob only affects the emulated vPMU; a mediated vPMU increments its counter directly and is unchanged. Suggested-by: David Woodhouse Suggested-by: Sean Christopherson Signed-off-by: Luka Absandze --- v1 (RFC): https://lore.kernel.org/kvm/20260720192221.72912-1-absandze@amazon.de/ Changes since v1: - Drop the new KVM_CAP_X86_DISABLE_PMU_SW_ACCOUNTING capability and its Documentation, i.e. add no new uAPI. - Instead of disabling emulated-instruction accounting outright, batch the counter reprograms behind a tolerance threshold, per David's suggestion to bound the reprogram rate rather than skip accounting [1]. This keeps guest counter reads accurate and only bounds how late an overflow PMI is delivered. - Expose the threshold as the emulated_counter_reprogram_tolerance module param (uint, 0644); 0 preserves current behavior. [1] https://lore.kernel.org/kvm/f5552576fc749b4b453514f52921332cc104c6a6.camel@infradead.org/ Results (single run, not averaged over multiple runs): Host: AMD EPYC 7R13 (Milan), emulated vPMU. Guest: 8 vCPUs under an 8-thread busy load, running a binary that toggles a PMU counter group cross-vCPU once per second while an instructions-retired counter is resident. Latency is the duration of the guest's local-APIC timer interrupt handler (__sysvec_apic_timer_interrupt), measured in-guest with ftrace function_graph over 30s (~160k ticks per pass). The module param was changed live on one running guest between passes. tolerance mean p50 p99 p99.9 max ---------- -------- ------- ------- ------- -------- 0 (default) 148.973 11.080 464.200 469.569 919.690 (us) 1000 3.194 2.950 7.251 9.831 470.100 100000 2.974 2.940 3.869 4.400 8.060 10000000 3.054 2.940 7.129 9.211 209.880 These are numbers from a single run and are meant only to show the order of magnitude, not to be precise. arch/x86/kvm/pmu.c | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/arch/x86/kvm/pmu.c b/arch/x86/kvm/pmu.c index dd1c57593f48..33a73271186a 100644 --- a/arch/x86/kvm/pmu.c +++ b/arch/x86/kvm/pmu.c @@ -39,6 +39,14 @@ bool __read_mostly enable_pmu = true; EXPORT_SYMBOL_FOR_KVM_INTERNAL(enable_pmu); module_param(enable_pmu, bool, 0444); +/* + * Number of KVM-emulated instructions that may accumulate on an emulated + * (perf-based) vPMU counter before KVM forces a counter reprogram to fold the + * emulated count into the backing perf_event + */ +static uint __read_mostly emulated_counter_reprogram_tolerance; +module_param(emulated_counter_reprogram_tolerance, uint, 0644); + /* Enable/disabled mediated PMU virtualization. */ bool __read_mostly enable_mediated_pmu; EXPORT_SYMBOL_FOR_KVM_INTERNAL(enable_mediated_pmu); @@ -1072,7 +1080,15 @@ static void kvm_pmu_incr_counter(struct kvm_pmc *pmc) */ if (!kvm_vcpu_has_mediated_pmu(vcpu)) { pmc->emulated_counter++; - kvm_pmu_request_counter_reprogram(pmc); + + /* + * Batch reprograms: only force one once the accumulated + * emulated count exceeds the tolerance. The count is still + * reflected in guest counter reads via pmc->emulated_counter; + * this only bounds how late an overflow-driven PMI arrives. + */ + if (pmc->emulated_counter > emulated_counter_reprogram_tolerance) + kvm_pmu_request_counter_reprogram(pmc); return; } base-commit: 1590cf0329716306e948a8fc29f1d3ee87d3989f -- 2.47.3