When the guest uses the APIC periodic timer, if the next period has already expired, e.g. due to the period being smaller than the delay in processing the timer, the delta will be negative. nsec_to_cycles() may then convert this delta into an absolute value larger than guest_l1_tsc, resulting in a negative tscdeadline. Since the hv timer supports a maximum bit width of cpu_preemption_timer_multi + 32, this causes the hv timer setup to fail and switch to the sw timer. Moreover, due to the commit 98c25ead5eda ("KVM: VMX: Move preemption timer <=> hrtimer dance to common x86"), if the guest is using the sw timer before blocking, it will continue to use the sw timer after being woken up, and will not switch back to the hv timer until the relevant APIC timer register is reprogrammed. Since the periodic timer does not require frequent APIC timer register programming, the guest may continue to use the software timer for an extended period. Fixes: d8f2f498d9ed ("x86/kvm: fix LAPIC timer drift when guest uses periodic mode") Signed-off-by: fuqiang wang --- arch/x86/kvm/lapic.c | 22 +++++++++++++++------- 1 file changed, 15 insertions(+), 7 deletions(-) diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c index 0ae7f913d782..fa07a303767c 100644 --- a/arch/x86/kvm/lapic.c +++ b/arch/x86/kvm/lapic.c @@ -2131,18 +2131,26 @@ static void advance_periodic_target_expiration(struct kvm_lapic *apic) ktime_t delta; /* - * Synchronize both deadlines to the same time source or - * differences in the periods (caused by differences in the - * underlying clocks or numerical approximation errors) will - * cause the two to drift apart over time as the errors - * accumulate. + * Use kernel time as the time source for both deadlines so that they + * stay synchronized. Computing each deadline independently will cause + * the two deadlines to drift apart over time as differences in the + * periods accumulate, e.g. due to differences in the underlying clocks + * or numerical approximation errors. */ apic->lapic_timer.target_expiration = ktime_add_ns(apic->lapic_timer.target_expiration, apic->lapic_timer.period); delta = ktime_sub(apic->lapic_timer.target_expiration, now); - apic->lapic_timer.tscdeadline = kvm_read_l1_tsc(apic->vcpu, tscl) + - nsec_to_cycles(apic->vcpu, delta); + + /* + * Don't adjust the tscdeadline if the next period has already expired, + * e.g. due to software overhead resulting in delays larger than the + * period. Blindly adding a negative delta could cause the deadline to + * become excessively large due to the deadline being an unsigned value. + */ + apic->lapic_timer.tscdeadline = kvm_read_l1_tsc(apic->vcpu, tscl); + if (delta > 0) + apic->lapic_timer.tscdeadline += nsec_to_cycles(apic->vcpu, delta); } static void start_sw_period(struct kvm_lapic *apic) -- 2.47.0 When a virtual machine uses the hv timer during suspend, the kvm timer does not advance. After a long period, if the VM is woken up, there will be a large gap between target_expiration and now. Since each timer expiration only advances target_expiration by one period, the timer expiration function will be repeatedly executed. Without the previous patch merged, the advanced target_expiration is less than now, which causes tscdeadline to be set to a negative value. This results in HV timer setup failure and a fallback to the SW timer. After switching to the SW timer, apic_timer_fn is repeatedly executed within a single clock interrupt handler, leading to a hardlockup: NMI watchdog: Watchdog detected hard LOCKUP on cpu 45 ... RIP: 0010:advance_periodic_target_expiration+0x4d/0x80 [kvm] ... RSP: 0018:ff4f88f5d98d8ef0 EFLAGS: 00000046 RAX: fff0103f91be678e RBX: fff0103f91be678e RCX: 00843a7d9e127bcc RDX: 0000000000000002 RSI: 0052ca4003697505 RDI: ff440d5bfbdbd500 RBP: ff440d5956f99200 R08: ff2ff2a42deb6a84 R09: 000000000002a6c0 R10: 0122d794016332b3 R11: 0000000000000000 R12: ff440db1af39cfc0 R13: ff440db1af39cfc0 R14: ffffffffc0d4a560 R15: ff440db1af39d0f8 FS: 00007f04a6ffd700(0000) GS:ff440db1af380000(0000) knlGS:000000e38a3b8000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000000d5651feff8 CR3: 000000684e038002 CR4: 0000000000773ee0 PKRU: 55555554 Call Trace: apic_timer_fn+0x31/0x50 [kvm] __hrtimer_run_queues+0x100/0x280 hrtimer_interrupt+0x100/0x210 ? ttwu_do_wakeup+0x19/0x160 smp_apic_timer_interrupt+0x6a/0x130 apic_timer_interrupt+0xf/0x20 After the previous patch is merged, the HV timer can no longer fall back to the SW timer. Additionally, while target_expiration is catching up to the current time, the VMX-preemption timer is set to 0 before each VM entry. According to Intel SDM 27.7.4 “VMX-Preemption Timer”: if the VMX-preemption timer has already expired at VM entry, a VM exit will occur before any instruction is executed. As a result, the guest cannot execute any instructions during this period, and therefore has no opportunity to reach vcpu_block() to switch to the SW timer. Thus, a hardlockup will not occur. However, it is still necessary to eliminate unnecessary multiple catch-ups. Therefore, if the advanced target_expiration is still less than now, we catch up to now in the current handling. Signed-off-by: fuqiang wang --- arch/x86/kvm/lapic.c | 22 +++++++++++++++------- 1 file changed, 15 insertions(+), 7 deletions(-) diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c index fa07a303767c..ba30de871929 100644 --- a/arch/x86/kvm/lapic.c +++ b/arch/x86/kvm/lapic.c @@ -2140,17 +2140,25 @@ static void advance_periodic_target_expiration(struct kvm_lapic *apic) apic->lapic_timer.target_expiration = ktime_add_ns(apic->lapic_timer.target_expiration, apic->lapic_timer.period); - delta = ktime_sub(apic->lapic_timer.target_expiration, now); /* - * Don't adjust the tscdeadline if the next period has already expired, - * e.g. due to software overhead resulting in delays larger than the - * period. Blindly adding a negative delta could cause the deadline to - * become excessively large due to the deadline being an unsigned value. + * When the vm is suspend, the hv timer also stops advancing. After it + * is resumed, this may result in a large delta. If the + * target_expiration only advances by one period each time, it will + * cause KVM to frequently handle timer expirations. */ + if (apic->lapic_timer.period > 0 && + ktime_before(apic->lapic_timer.target_expiration, now)) + apic->lapic_timer.target_expiration = now; + + delta = ktime_sub(apic->lapic_timer.target_expiration, now); apic->lapic_timer.tscdeadline = kvm_read_l1_tsc(apic->vcpu, tscl); - if (delta > 0) - apic->lapic_timer.tscdeadline += nsec_to_cycles(apic->vcpu, delta); + /* + * Note: delta must not be negative. Otherwise, blindly adding a + * negative delta could cause the deadline to become excessively large + * due to the deadline being an unsigned value. + */ + apic->lapic_timer.tscdeadline += nsec_to_cycles(apic->vcpu, delta); } static void start_sw_period(struct kvm_lapic *apic) -- 2.47.0