6.12-stable review patch. If anyone has any objections, please let me know. ------------------ From: EJ Campbell commit 81d99abe2bd81379abfb595e403bbbee6fbfc624 upstream. An async #PF "page not present" event arrives through the #PF vector, and its reason sits in the per-CPU apf_reason.flags until __kvm_handle_async_pf() reads and clears it. If an NMI arrives after the event is injected and before the #PF handler reads the flags, and the NMI handler itself takes a #PF, that fault reads and clears the flags belonging to the #PF it interrupted. It then finds interrupts disabled in the NMI context it interrupted, and panics: Kernel panic - not syncing: Host injected async #PF in interrupt disabled region Call Trace: panic+0x52/0x60 __kvm_handle_async_pf+0xb0/0xb0 exc_page_fault+0x71/0xb0 asm_exc_page_fault+0x27/0x30 RIP: 0010:__get_user_nocheck_8+0x6/0x20 perf_callchain_user+0x147/0x1a0 get_perf_callchain+0x12d/0x1c0 ... perf_event_nmi_handler+0x26/0x50 exc_nmi+0xe8/0x110 RIP: 0010:asm_exc_page_fault+0x0/0x30 The last RIP line shows the NMI landing on the first instruction of the #PF entry. perf with user call chains takes such a fault: the sampling NMI walks the user stack with __get_user(), which faults when a saved frame pointer is not a valid user address or the walk runs off the top of the stack. The same window exists when KVM, running as an L1 hypervisor with DELIVERY_AS_PF_VMEXIT, reads the flags itself after a #PF VM-exit. The NMI entry code saves and restores CR2 for this nesting, but not the async #PF reason. Andy Lutomirski described the race, with perf as the example, when async #PF delivery was reworked in 2020 [1]. Saving and clearing the flags across the NMI, next to CR2, was discussed there but not merged. Since that rework, Linux has stopped setting KVM_ASYNC_PF_SEND_ALWAYS, so the host delivers "page not present" only for user mode faults and a kernel-mode fault in NMI context (including kernel-mode #MC and #DB) cannot be an async page fault. In that case, return without touching the flags so that the interrupted handler finds them intact. Tested on v7.3-rc5 in a 4-vCPU, 2 GiB guest restored from a snapshot whose memory is served on demand through userfaultfd, so that reading it back takes async #PFs, while every thread samples hardware cycles with user call chains and points its frame pointer at an unmapped page. Without this patch the guest panicked as above in 3 of 3 runs. With it, the guest survived 3 of 3 runs of 60 seconds, and the host counted 148,441 to 153,334 kvm_async_pf_not_present events in the first 20 seconds of each. On 6.18.50 the results were the same: the guest panicked without the patch and survived with it. Link: https://lore.kernel.org/all/ed71d0967113a35f670a9625a058b8e6e0b2f104.1583547991.git.luto@kernel.org/ [1] Link: https://lore.kernel.org/all/20211126123145.2772-1-jiangshanlai@gmail.com/ [2] Cc: stable@vger.kernel.org # v5.9+ Signed-off-by: EJ Campbell Link: https://patch.msgid.link/CADkxfh-uQTca_6jB1JPDq1MsKffF_UKGQR339oQSJNcoV75GNQ@mail.gmail.com Signed-off-by: Paolo Bonzini Signed-off-by: Greg Kroah-Hartman --- arch/x86/kernel/kvm.c | 32 +++++++++++++++++++++++++++++--- 1 file changed, 29 insertions(+), 3 deletions(-) --- a/arch/x86/kernel/kvm.c +++ b/arch/x86/kernel/kvm.c @@ -255,11 +255,39 @@ noinstr u32 kvm_read_and_reset_apf_flags } EXPORT_SYMBOL_GPL(kvm_read_and_reset_apf_flags); +static __always_inline bool kvm_apf_reason_pending(void) +{ + return __this_cpu_read(async_pf_enabled) && + __this_cpu_read(apf_reason.flags); +} + noinstr bool __kvm_handle_async_pf(struct pt_regs *regs, u32 token) { - u32 flags = kvm_read_and_reset_apf_flags(); irqentry_state_t state; + u32 flags; + + if (!kvm_apf_reason_pending()) + return false; + + /* + * The guest does not enable KVM_ASYNC_PF_SEND_ALWAYS, so the host + * delivers async #PF only to user mode (or as a #PF VM-exit, which KVM + * reads itself). A #PF from kernel mode can still arrive between that + * delivery and the read of the reason flags, inside an NMI that + * interrupted it, for instance perf reading user memory for a call + * chain. Such a fault is not async, so leave the async PF flags + * untouched for the outer handler. Outside NMI context, on the other + * hand, async PF flags on a kernel-mode fault mean the host is broken. + */ + if (!user_mode(regs)) { + if (likely(in_nmi())) + return false; + state = irqentry_enter(regs); + instrumentation_begin(); + panic("Host injected async #PF in kernel mode\n"); + } + flags = kvm_read_and_reset_apf_flags(); if (!flags) return false; @@ -275,8 +303,6 @@ noinstr bool __kvm_handle_async_pf(struc panic("Host injected async #PF in interrupt disabled region\n"); if (flags & KVM_PV_REASON_PAGE_NOT_PRESENT) { - if (unlikely(!(user_mode(regs)))) - panic("Host injected async #PF in kernel mode\n"); /* Page is swapped out by the host. */ kvm_async_pf_task_wait_schedule(token); } else {