Configure MDCR_EL2 for a Partitioned PMU in kvm_arm_setup_mdcr_el2(): set HPME, HPMD, and HCCD (plus HLP on FEAT_PMUv3p5 so host counters retain 64-bit overflow width), program HPMN to the guest's general purpose counter count, and clear coarse-grained TPM and TPMCR traps when FEAT_FGT is available. Protect MDCR_EL2 updates with local_daif_save() so GICv3 pseudo-NMI PMU handlers toggling MDCR_EL2.HPME cannot race with kvm_arm_setup_mdcr_el2(). Update kvm_nested_setup_mdcr_el2() for the two nested execution contexts: - Nested guest context (is_nested_ctxt(vcpu), L2 at vEL1/vEL0): NV_MDCR_GUEST_INCLUDE already propagates L1's virtual TPM and TPMCR. When L1 sub-partitions its counters (vHPMN < nr_pmu_counters, so kvm_pmu_hyp_counter_mask(vcpu) is non-zero), physical MDCR_EL2.HPMN still equals nr_pmu_counters, so hardware alone cannot stop L2 from accessing L1's hypervisor counters (vHPMN..nr_pmu_counters-1) or reading nr_pmu_counters from PMCR_EL0.N. Force TPM and TPMCR so L0 emulates 0..vHPMN-1 locally and forwards hypervisor counter traps to L1. - Nested hypervisor context (else branch, L1): under VHE, both L1 vEL2 (physical EL1) and VHE host EL0 (physical EL0) run in this context without exiting to L0 across ERET and SVC. When L1's PMUSERENR_EL0 is 0, hardware traps EL0 PMU accesses directly to L1 vEL2, allowing TPM and TPMCR to stay clear for untrapped L1 vEL2 access. When L1 enables EL0 access in PMUSERENR_EL0, propagate L1's virtual TPM/TPMCR (and set TPM/TPMCR if L1 sub-partitions counters) so VHE host EL0 cannot bypass L1's traps or vHPMN boundary. Signed-off-by: Colton Lewis --- arch/arm64/kvm/debug.c | 41 +++++++++++++++++++++++++++++++++++++++-- arch/arm64/kvm/nested.c | 35 ++++++++++++++++++++++++----------- 2 files changed, 63 insertions(+), 13 deletions(-) diff --git a/arch/arm64/kvm/debug.c b/arch/arm64/kvm/debug.c index 89c9fb0cfeee2..a1cb4c076bdb4 100644 --- a/arch/arm64/kvm/debug.c +++ b/arch/arm64/kvm/debug.c @@ -11,6 +11,7 @@ #include #include +#include #include #include #include @@ -37,9 +38,10 @@ static int cpu_has_spe(u64 dfr0) */ static void kvm_arm_setup_mdcr_el2(struct kvm_vcpu *vcpu) { + unsigned long daif; u64 mdcr; - preempt_disable(); + daif = local_daif_save(); /* * This also clears MDCR_EL2_E2PB_MASK and MDCR_EL2_E2TB_MASK @@ -53,6 +55,41 @@ static void kvm_arm_setup_mdcr_el2(struct kvm_vcpu *vcpu) MDCR_EL2_TDRA | MDCR_EL2_TDOSA); + if (kvm_pmu_is_partitioned(vcpu->kvm)) { + u8 nr_host_cntr = *host_data_ptr(nr_event_counters); + u8 nr_guest_cntr = vcpu->kvm->arch.nr_pmu_counters; + + if (cpumask_test_cpu(smp_processor_id(), + vcpu->kvm->arch.supported_cpus) && + nr_guest_cntr < nr_host_cntr && + (cpus_have_final_cap(ARM64_HAS_HPMN0) || nr_guest_cntr > 0)) { + u64 hpmn = FIELD_GET(MDCR_EL2_HPMN, read_sysreg(mdcr_el2)); + bool hpme; + + if (hpmn < nr_host_cntr) + hpme = !!(read_sysreg(mdcr_el2) & MDCR_EL2_HPME); + else + hpme = !!(read_sysreg(pmcr_el0) & ARMV8_PMU_PMCR_E); + + mdcr |= (MDCR_EL2_HPMD | MDCR_EL2_HCCD); + if (is_pmuv3p5(read_pmuver())) + mdcr |= MDCR_EL2_HLP; + if (hpme) + mdcr |= MDCR_EL2_HPME; + + /* + * Take out the coarse grain traps if we are using + * fine grain traps and enforce counter access with + * HPMN. + */ + mdcr &= ~MDCR_EL2_HPMN; + mdcr |= FIELD_PREP(MDCR_EL2_HPMN, nr_guest_cntr); + + if (cpus_have_final_cap(ARM64_HAS_FGT)) + mdcr &= ~(MDCR_EL2_TPM | MDCR_EL2_TPMCR); + } + } + /* Is the VM being debugged by userspace? */ if (vcpu->guest_debug) /* Route all software debug exceptions to EL2 */ @@ -73,7 +110,7 @@ static void kvm_arm_setup_mdcr_el2(struct kvm_vcpu *vcpu) if (has_vhe()) write_sysreg(mdcr, mdcr_el2); - preempt_enable(); + local_daif_restore(daif); } void kvm_init_host_debug_data(void) diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c index b80d90ede9004..06a535c54b708 100644 --- a/arch/arm64/kvm/nested.c +++ b/arch/arm64/kvm/nested.c @@ -2241,16 +2241,29 @@ void kvm_nested_setup_mdcr_el2(struct kvm_vcpu *vcpu, u64 *mdcr) { u64 guest_mdcr = __vcpu_sys_reg(vcpu, MDCR_EL2); - if (is_nested_ctxt(vcpu)) + if (is_nested_ctxt(vcpu)) { *mdcr |= (guest_mdcr & NV_MDCR_GUEST_INCLUDE); - /* - * In yet another example where FEAT_NV2 is fscking broken, accesses - * to MDSCR_EL1 are redirected to the VNCR despite having an effect - * at EL2. Use a big hammer to apply sanity. - * - * Unless of course we have FEAT_FGT, in which case we can precisely - * trap MDSCR_EL1. - */ - else if (!cpus_have_final_cap(ARM64_HAS_FGT)) - *mdcr |= MDCR_EL2_TDA; + if (kvm_pmu_is_partitioned(vcpu->kvm) && + kvm_pmu_hyp_counter_mask(vcpu)) + *mdcr |= MDCR_EL2_TPM | MDCR_EL2_TPMCR; + } else { + /* + * In yet another example where FEAT_NV2 is fscking broken, + * accesses to MDSCR_EL1 are redirected to the VNCR despite + * having an effect at EL2. Use a big hammer to apply sanity. + * + * Unless of course we have FEAT_FGT, in which case we can + * precisely trap MDSCR_EL1. + */ + if (!cpus_have_final_cap(ARM64_HAS_FGT)) + *mdcr |= MDCR_EL2_TDA; + + if (kvm_pmu_is_partitioned(vcpu->kvm) && + (__vcpu_sys_reg(vcpu, PMUSERENR_EL0) & + ARMV8_PMU_USERENR_MASK)) { + *mdcr |= guest_mdcr & (MDCR_EL2_TPM | MDCR_EL2_TPMCR); + if (kvm_pmu_hyp_counter_mask(vcpu)) + *mdcr |= MDCR_EL2_TPM | MDCR_EL2_TPMCR; + } + } } -- 2.56.1.708.g77570a6a1d-goog