From patchwork Mon Nov 29 20:00:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645885 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8930DC433EF for ; Mon, 29 Nov 2021 22:19:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232922AbhK2WW5 (ORCPT ); Mon, 29 Nov 2021 17:22:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56692 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231944AbhK2WVS (ORCPT ); Mon, 29 Nov 2021 17:21:18 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E5373C08EADF for ; Mon, 29 Nov 2021 12:02:14 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 32CDDCE139A for ; Mon, 29 Nov 2021 20:02:13 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5E887C53FAD; Mon, 29 Nov 2021 20:02:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216131; bh=o/ak3OJlS7RwPUXCcH3UzNik1EFF5uPRaId91AaLh8s=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=t6mwydoOkFrlxefFJdGKzZ43dQcHxVRUewurzfbDmZvns0BNdphFxCrKedgNqfgNB zjvzSNoe89edOnO1+xV8C7GeFm+Jl4mkUOCMqSizftQ8TEhA5sQTRktaKVPpZpyCcZ 3OK7qi9ihOSbi4qnURfQEMQPitS39yzeoNTRSpj4TgkyHyTurAhGVzBF211yTfnzer xZtPg72NjUBhXvTbAKAkxj83evSJhrW+eUIPMe2tGYGDXvuIM5roK04PYZRySJ7J1Z mSHGM5+HyVl0a6L5hMedLcsntWTZctFYRlzZgQLbP/2aSxVM5C6JeBU7OLu8maWP2m eTFHEXOJ2rNyA== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmqn-008gvR-DM; Mon, 29 Nov 2021 20:02:09 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 01/69] KVM: arm64: Save PSTATE early on exit Date: Mon, 29 Nov 2021 20:00:42 +0000 Message-Id: <20211129200150.351436-2-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org In order to be able to use primitives such as vcpu_mode_is_32bit(), we need to synchronize the guest PSTATE. However, this is currently done deep into the bowels of the world-switch code, and we do have helpers evaluating this much earlier (__vgic_v3_perform_cpuif_access and handle_aarch32_guest, for example). Move the saving of the guest pstate into the early fixups, which cures the first issue. The second one will be addressed separately. Tested-by: Fuad Tabba Reviewed-by: Fuad Tabba Signed-off-by: Marc Zyngier Reviewed-by: Russell King (Oracle) --- arch/arm64/kvm/hyp/include/hyp/switch.h | 6 ++++++ arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h | 7 ++++++- 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h index 7a0af1d39303..d79fd101615f 100644 --- a/arch/arm64/kvm/hyp/include/hyp/switch.h +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h @@ -429,6 +429,12 @@ static inline bool kvm_hyp_handle_exit(struct kvm_vcpu *vcpu, u64 *exit_code) */ static inline bool fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code) { + /* + * Save PSTATE early so that we can evaluate the vcpu mode + * early on. + */ + vcpu->arch.ctxt.regs.pstate = read_sysreg_el2(SYS_SPSR); + if (ARM_EXCEPTION_CODE(*exit_code) != ARM_EXCEPTION_IRQ) vcpu->arch.fault.esr_el2 = read_sysreg_el2(SYS_ESR); diff --git a/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h b/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h index de7e14c862e6..7ecca8b07851 100644 --- a/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h +++ b/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h @@ -70,7 +70,12 @@ static inline void __sysreg_save_el1_state(struct kvm_cpu_context *ctxt) static inline void __sysreg_save_el2_return_state(struct kvm_cpu_context *ctxt) { ctxt->regs.pc = read_sysreg_el2(SYS_ELR); - ctxt->regs.pstate = read_sysreg_el2(SYS_SPSR); + /* + * Guest PSTATE gets saved at guest fixup time in all + * cases. We still need to handle the nVHE host side here. + */ + if (!has_vhe() && ctxt->__hyp_running_vcpu) + ctxt->regs.pstate = read_sysreg_el2(SYS_SPSR); if (cpus_have_final_cap(ARM64_HAS_RAS_EXTN)) ctxt_sys_reg(ctxt, DISR_EL1) = read_sysreg_s(SYS_VDISR_EL2); From patchwork Mon Nov 29 20:00:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645893 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 06319C433FE for ; Mon, 29 Nov 2021 22:19:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230099AbhK2WXF (ORCPT ); Mon, 29 Nov 2021 17:23:05 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56596 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232437AbhK2WVS (ORCPT ); Mon, 29 Nov 2021 17:21:18 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 22D3BC08EAE4 for ; Mon, 29 Nov 2021 12:02:15 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 6D0DBCE13DE for ; Mon, 29 Nov 2021 20:02:13 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 989C2C53FCD; Mon, 29 Nov 2021 20:02:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216131; bh=m3jPFU3mjJWLueVcfv+mJbvgBUGfUsTVQ0Ctnxxc81o=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=pVxsnTUTo9ugk+jioqIluBvc9s7QseHN6CQBngnPBKPXBSEbPLirUB4bP1o0CSGM7 Y95rjgy/qM+sDotJrmbzrZY3u2MsPAbybgVytHkobcNIa08arh5JqRojtdC17nyet2 t0oN3kW715np5GSJhaQBp3yn/Y1tnT6OZzk5kE33cN3wrWvqKZekr/v6c8Sv5I3CIQ O5nNS4FtST93h8XmPKJMwImHpcJLQUCNXDChE0mi6Yr7h/GRIHyuZWlzRILyKtHxYQ 4lwuq6a3RkEfMh96f/hSqBFr1zibxVUbKqvdYxr2jmavCwLlt8Rupd1xzI78DS/Y23 L8toxb4LLEBBg== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmqn-008gvR-Ns; Mon, 29 Nov 2021 20:02:09 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 02/69] KVM: arm64: Move pkvm's special 32bit handling into a generic infrastructure Date: Mon, 29 Nov 2021 20:00:43 +0000 Message-Id: <20211129200150.351436-3-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Protected KVM is trying to turn AArch32 exceptions into an illegal exception entry. Unfortunately, it does that in a way that is a bit abrupt, and too early for PSTATE to be available. Instead, move it to the fixup code, which is a more reasonable place for it. This will also be useful for the NV code. Reviewed-by: Fuad Tabba Signed-off-by: Marc Zyngier --- arch/arm64/kvm/hyp/include/hyp/switch.h | 8 ++++++++ arch/arm64/kvm/hyp/nvhe/switch.c | 8 +------- arch/arm64/kvm/hyp/vhe/switch.c | 4 ++++ 3 files changed, 13 insertions(+), 7 deletions(-) diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h index d79fd101615f..96c5f3fb7838 100644 --- a/arch/arm64/kvm/hyp/include/hyp/switch.h +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h @@ -403,6 +403,8 @@ typedef bool (*exit_handler_fn)(struct kvm_vcpu *, u64 *); static const exit_handler_fn *kvm_get_exit_handler_array(struct kvm_vcpu *vcpu); +static void early_exit_filter(struct kvm_vcpu *vcpu, u64 *exit_code); + /* * Allow the hypervisor to handle the exit with an exit handler if it has one. * @@ -435,6 +437,12 @@ static inline bool fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code) */ vcpu->arch.ctxt.regs.pstate = read_sysreg_el2(SYS_SPSR); + /* + * Check whether we want to repaint the state one way or + * another. + */ + early_exit_filter(vcpu, exit_code); + if (ARM_EXCEPTION_CODE(*exit_code) != ARM_EXCEPTION_IRQ) vcpu->arch.fault.esr_el2 = read_sysreg_el2(SYS_ESR); diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/switch.c index c0e3fed26d93..d13115a12434 100644 --- a/arch/arm64/kvm/hyp/nvhe/switch.c +++ b/arch/arm64/kvm/hyp/nvhe/switch.c @@ -233,7 +233,7 @@ static const exit_handler_fn *kvm_get_exit_handler_array(struct kvm_vcpu *vcpu) * Returns false if the guest ran in AArch32 when it shouldn't have, and * thus should exit to the host, or true if a the guest run loop can continue. */ -static bool handle_aarch32_guest(struct kvm_vcpu *vcpu, u64 *exit_code) +static void early_exit_filter(struct kvm_vcpu *vcpu, u64 *exit_code) { struct kvm *kvm = kern_hyp_va(vcpu->kvm); @@ -248,10 +248,7 @@ static bool handle_aarch32_guest(struct kvm_vcpu *vcpu, u64 *exit_code) vcpu->arch.target = -1; *exit_code &= BIT(ARM_EXIT_WITH_SERROR_BIT); *exit_code |= ARM_EXCEPTION_IL; - return false; } - - return true; } /* Switch to the guest for legacy non-VHE systems */ @@ -316,9 +313,6 @@ int __kvm_vcpu_run(struct kvm_vcpu *vcpu) /* Jump in the fire! */ exit_code = __guest_enter(vcpu); - if (unlikely(!handle_aarch32_guest(vcpu, &exit_code))) - break; - /* And we're baaack! */ } while (fixup_guest_exit(vcpu, &exit_code)); diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c index 5a2cb5d9bc4b..fbb26b93c347 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -112,6 +112,10 @@ static const exit_handler_fn *kvm_get_exit_handler_array(struct kvm_vcpu *vcpu) return hyp_exit_handlers; } +static void early_exit_filter(struct kvm_vcpu *vcpu, u64 *exit_code) +{ +} + /* Switch to the guest for VHE systems running in EL2 */ static int __kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu) { From patchwork Mon Nov 29 20:00:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645605 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CFDD6C433EF for ; Mon, 29 Nov 2021 20:04:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1349891AbhK2UHd (ORCPT ); Mon, 29 Nov 2021 15:07:33 -0500 Received: from sin.source.kernel.org ([145.40.73.55]:49344 "EHLO sin.source.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229538AbhK2UFc (ORCPT ); Mon, 29 Nov 2021 15:05:32 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id AF0F0CE13F9 for ; Mon, 29 Nov 2021 20:02:13 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D9AFBC53FCF; Mon, 29 Nov 2021 20:02:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216131; bh=JIlqAddKl/85lpIm3FWWCZ7fbOCEiWhQkqCIpNen/Oo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=uKxRWBZJtPq2mZZ7uVYc9BHyGwFM+Ms9WGiTI6X+o1PyFZuKEM7EOcPRpoGYrZMXI hGM0gaXx+DI+R7M4kHe4k8jamiJW3CIPx/E547crz5BcBbRXTJWsxQLUbM84V1LweL +G0y21TZBbM5qxOP3ltoq7t+kybNiduAEwQ9zlYMeA9a3ORuZWp6rhzyfoUm5vomLC cRO4e0G+lsFAGsK4gYVHlyzvBNruz4fVoJ8ZGAl6sy+1YrOW7rg4GysCKWW32CUB0i A2tA1dYznbyRPePw435Eeg/a55ldEr9Ym9N08fTUJHoQOoOtq7oVoler4/CJ2NtkUA kzhasaUXuYyeg== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmqo-008gvR-2h; Mon, 29 Nov 2021 20:02:10 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 03/69] KVM: arm64: Add minimal handling for the ARMv8.7 PMU Date: Mon, 29 Nov 2021 20:00:44 +0000 Message-Id: <20211129200150.351436-4-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org When running a KVM guest hosted on an ARMv8.7 machine, the host kernel complains that it doesn't know about the architected number of events. Fix it by adding the PMUver code corresponding to PMUv3 for ARMv8.7. Reviewed-by: Alexandru Elisei Tested-by: Alexandru Elisei Signed-off-by: Marc Zyngier Link: https://lore.kernel.org/r/20211126115533.217903-1-maz@kernel.org Reviewed-by: Russell King (Oracle) --- arch/arm64/include/asm/sysreg.h | 1 + arch/arm64/kvm/pmu-emul.c | 1 + 2 files changed, 2 insertions(+) diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h index 16b3f1a1d468..615dd6278f8b 100644 --- a/arch/arm64/include/asm/sysreg.h +++ b/arch/arm64/include/asm/sysreg.h @@ -937,6 +937,7 @@ #define ID_AA64DFR0_PMUVER_8_1 0x4 #define ID_AA64DFR0_PMUVER_8_4 0x5 #define ID_AA64DFR0_PMUVER_8_5 0x6 +#define ID_AA64DFR0_PMUVER_8_7 0x7 #define ID_AA64DFR0_PMUVER_IMP_DEF 0xf #define ID_AA64DFR0_PMSVER_8_2 0x1 diff --git a/arch/arm64/kvm/pmu-emul.c b/arch/arm64/kvm/pmu-emul.c index a5e4bbf5e68f..ca92cc5c71c6 100644 --- a/arch/arm64/kvm/pmu-emul.c +++ b/arch/arm64/kvm/pmu-emul.c @@ -28,6 +28,7 @@ static u32 kvm_pmu_event_mask(struct kvm *kvm) case ID_AA64DFR0_PMUVER_8_1: case ID_AA64DFR0_PMUVER_8_4: case ID_AA64DFR0_PMUVER_8_5: + case ID_AA64DFR0_PMUVER_8_7: return GENMASK(15, 0); default: /* Shouldn't be here, just for sanity */ WARN_ONCE(1, "Unknown PMU version %d\n", kvm->arch.pmuver); From patchwork Mon Nov 29 20:00:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645891 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 199EEC433EF for ; Mon, 29 Nov 2021 22:19:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233331AbhK2WXC (ORCPT ); Mon, 29 Nov 2021 17:23:02 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56610 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232427AbhK2WVT (ORCPT ); Mon, 29 Nov 2021 17:21:19 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 08958C08EAE7 for ; Mon, 29 Nov 2021 12:02:17 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 3D44DCE140B for ; Mon, 29 Nov 2021 20:02:14 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5DBA2C53FD1; Mon, 29 Nov 2021 20:02:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216132; bh=Df8PadvHjOAqr8B6CfNfQ511OzaDacc78f6/QUTpGHM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=FJRnyB5Na7g06iVpC+drouPpHCnoaOOzZ1lPAz3dVShLvCfHCf/T/UKNSf29W+uUQ pAnu4jkBS9Ztz2S8AqKW2mRRK2y1OjSSZKzSX3jejDB8dEUQqBCIGnw5hNs7aQZusq gb5mQJKMFH4Z6dLVm19ADFucCoMvOd0vRWt0Qzgv20yLL1qrbRvjOh8gdHPtJEh1n8 ZSNw1o3pLroa1AFJh1minHo8ztiS+fxX5iCJ5KQkxQA4H0H8GLpYryEeq+Uvj5/i15 pt3zDpWm7VbO9D1Nl7hebiPz6uURpKS1Borw1qP0MExomZiWdlZIQZEARqc3LxWBR5 4vST3E01r6nVQ== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmqo-008gvR-DG; Mon, 29 Nov 2021 20:02:10 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 04/69] KVM: arm64: Rework kvm_pgtable initialisation Date: Mon, 29 Nov 2021 20:00:45 +0000 Message-Id: <20211129200150.351436-5-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Ganapatrao reported that the kvm_pgtable->mmu pointer is more or less hardcoded to the main S2 mmu structure, while the nested code needs it to point to other instances (as we have one instance per nested context). Rework the initialisation of the kvm_pgtable structure so that this assumtion doesn't hold true anymore. This requires some minor changes to the order in which things are initialised (the mmu->arch pointer being the critical one). Reported-by: Ganapatrao Kulkarni Reviewed-by: Ganapatrao Kulkarni Signed-off-by: Marc Zyngier Reviewed-by: Russell King (Oracle) --- arch/arm64/include/asm/kvm_pgtable.h | 9 ++++----- arch/arm64/kvm/hyp/nvhe/mem_protect.c | 4 ++-- arch/arm64/kvm/hyp/pgtable.c | 6 +++--- arch/arm64/kvm/mmu.c | 4 ++-- 4 files changed, 11 insertions(+), 12 deletions(-) diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index 027783829584..8b808e70c2b2 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -270,8 +270,7 @@ u64 kvm_get_vtcr(u64 mmfr0, u64 mmfr1, u32 phys_shift); /** * __kvm_pgtable_stage2_init() - Initialise a guest stage-2 page-table. * @pgt: Uninitialised page-table structure to initialise. - * @arch: Arch-specific KVM structure representing the guest virtual - * machine. + * @mmu: S2 MMU context for this S2 translation * @mm_ops: Memory management callbacks. * @flags: Stage-2 configuration flags. * @force_pte_cb: Function that returns true if page level mappings must @@ -279,13 +278,13 @@ u64 kvm_get_vtcr(u64 mmfr0, u64 mmfr1, u32 phys_shift); * * Return: 0 on success, negative error code on failure. */ -int __kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm_arch *arch, +int __kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm_s2_mmu *mmu, struct kvm_pgtable_mm_ops *mm_ops, enum kvm_pgtable_stage2_flags flags, kvm_pgtable_force_pte_cb_t force_pte_cb); -#define kvm_pgtable_stage2_init(pgt, arch, mm_ops) \ - __kvm_pgtable_stage2_init(pgt, arch, mm_ops, 0, NULL) +#define kvm_pgtable_stage2_init(pgt, mmu, mm_ops) \ + __kvm_pgtable_stage2_init(pgt, mmu, mm_ops, 0, NULL) /** * kvm_pgtable_stage2_destroy() - Destroy an unused guest stage-2 page-table. diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvhe/mem_protect.c index c1a90dd022b8..7d7b7037dc68 100644 --- a/arch/arm64/kvm/hyp/nvhe/mem_protect.c +++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c @@ -103,19 +103,19 @@ int kvm_host_prepare_stage2(void *pgt_pool_base) prepare_host_vtcr(); hyp_spin_lock_init(&host_kvm.lock); + mmu->arch = &host_kvm.arch; ret = prepare_s2_pool(pgt_pool_base); if (ret) return ret; - ret = __kvm_pgtable_stage2_init(&host_kvm.pgt, &host_kvm.arch, + ret = __kvm_pgtable_stage2_init(&host_kvm.pgt, mmu, &host_kvm.mm_ops, KVM_HOST_S2_FLAGS, host_stage2_force_pte_cb); if (ret) return ret; mmu->pgd_phys = __hyp_pa(host_kvm.pgt.pgd); - mmu->arch = &host_kvm.arch; mmu->pgt = &host_kvm.pgt; WRITE_ONCE(mmu->vmid.vmid_gen, 0); WRITE_ONCE(mmu->vmid.vmid, 0); diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index f8ceebe4982e..8cdbc43fa651 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -1116,13 +1116,13 @@ int kvm_pgtable_stage2_flush(struct kvm_pgtable *pgt, u64 addr, u64 size) } -int __kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm_arch *arch, +int __kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm_s2_mmu *mmu, struct kvm_pgtable_mm_ops *mm_ops, enum kvm_pgtable_stage2_flags flags, kvm_pgtable_force_pte_cb_t force_pte_cb) { size_t pgd_sz; - u64 vtcr = arch->vtcr; + u64 vtcr = mmu->arch->vtcr; u32 ia_bits = VTCR_EL2_IPA(vtcr); u32 sl0 = FIELD_GET(VTCR_EL2_SL0_MASK, vtcr); u32 start_level = VTCR_EL2_TGRAN_SL0_BASE - sl0; @@ -1135,7 +1135,7 @@ int __kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm_arch *arch, pgt->ia_bits = ia_bits; pgt->start_level = start_level; pgt->mm_ops = mm_ops; - pgt->mmu = &arch->mmu; + pgt->mmu = mmu; pgt->flags = flags; pgt->force_pte_cb = force_pte_cb; diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 326cdfec74a1..9eec548fccd1 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -516,7 +516,8 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu) if (!pgt) return -ENOMEM; - err = kvm_pgtable_stage2_init(pgt, &kvm->arch, &kvm_s2_mm_ops); + mmu->arch = &kvm->arch; + err = kvm_pgtable_stage2_init(pgt, mmu, &kvm_s2_mm_ops); if (err) goto out_free_pgtable; @@ -529,7 +530,6 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu) for_each_possible_cpu(cpu) *per_cpu_ptr(mmu->last_vcpu_ran, cpu) = -1; - mmu->arch = &kvm->arch; mmu->pgt = pgt; mmu->pgd_phys = __pa(pgt->pgd); WRITE_ONCE(mmu->vmid.vmid_gen, 0); From patchwork Mon Nov 29 20:00:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645895 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 50722C433EF for ; Mon, 29 Nov 2021 22:19:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233371AbhK2WXH (ORCPT ); Mon, 29 Nov 2021 17:23:07 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56688 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230377AbhK2WVT (ORCPT ); Mon, 29 Nov 2021 17:21:19 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 52B4FC08EAE5 for ; Mon, 29 Nov 2021 12:02:16 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id B935CB815D7 for ; Mon, 29 Nov 2021 20:02:14 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A7B93C53FD2; Mon, 29 Nov 2021 20:02:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216132; bh=r1KVpSR7EgIPE8I4/GpAKDCPw+V3aHcPhm5guWg0kK0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=cLEypiGX/Kz7h6mdeCJwpUhz3LiCZn0KCrGZ3Fpyr5abbPxTc2I8Pf4KvWMXQJWCw OXxCmvjqPrli5MqoRrySZ1QypS3IXkjIs6ztJzUQ3mNES7SAPaFhRE1Z/AOxNPf41X /fgnIDRmYqDs4P5N8eow/MN7uxtwRjXLZ0wKQ6J9ZumwVzNhLOWxHiTMTpt4LnGUk0 /GRnEzmsZuZdvJgFMXLZQ9i3p8l1t4qXwt+DeZjICcLEdveIOARaro9XT35uaZ0qM2 McAWtybaxeqOaid4Qi2BfLFDMa5qV1Hq78ygJ4BAx81MaefY+cJhlcdggKTrEgOOyy VDgT3pL391XdQ== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmqo-008gvR-NI; Mon, 29 Nov 2021 20:02:10 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 05/69] KVM: arm64: Allow preservation of the S2 SW bits Date: Mon, 29 Nov 2021 20:00:46 +0000 Message-Id: <20211129200150.351436-6-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The S2 page table code has a limited use the SW bits, but we are about to need them to encode some guest Stage-2 information (its mapping size in the form of the TTL encoding). Propagate the SW bits specified by the caller, and store them into the corresponding entry. Signed-off-by: Marc Zyngier Reviewed-by: Alexandru Elisei Reviewed-by: Russell King (Oracle) --- arch/arm64/kvm/hyp/pgtable.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 8cdbc43fa651..d69e400b2de6 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -1064,9 +1064,6 @@ int kvm_pgtable_stage2_relax_perms(struct kvm_pgtable *pgt, u64 addr, u32 level; kvm_pte_t set = 0, clr = 0; - if (prot & KVM_PTE_LEAF_ATTR_HI_SW) - return -EINVAL; - if (prot & KVM_PGTABLE_PROT_R) set |= KVM_PTE_LEAF_ATTR_LO_S2_S2AP_R; @@ -1076,6 +1073,10 @@ int kvm_pgtable_stage2_relax_perms(struct kvm_pgtable *pgt, u64 addr, if (prot & KVM_PGTABLE_PROT_X) clr |= KVM_PTE_LEAF_ATTR_HI_S2_XN; + /* Always propagate the SW bits */ + clr |= FIELD_PREP(KVM_PTE_LEAF_ATTR_HI_SW, 0xf); + set |= prot & KVM_PTE_LEAF_ATTR_HI_SW; + ret = stage2_update_leaf_attrs(pgt, addr, 1, set, clr, NULL, &level); if (!ret) kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, pgt->mmu, addr, level); From patchwork Mon Nov 29 20:00:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645609 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DE32EC433F5 for ; Mon, 29 Nov 2021 20:04:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1351709AbhK2UHf (ORCPT ); Mon, 29 Nov 2021 15:07:35 -0500 Received: from sin.source.kernel.org ([145.40.73.55]:49408 "EHLO sin.source.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230121AbhK2UFe (ORCPT ); Mon, 29 Nov 2021 15:05:34 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 430BBCE1410 for ; Mon, 29 Nov 2021 20:02:15 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id CB656C53FD0; Mon, 29 Nov 2021 20:02:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216132; bh=Ed7J/sCVqzfVqnfSwWqX/Wg6aMUVnxL/QI6OOMqQYig=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=EcWzGCn0x2yR8t0aRP3ltIdtwS1Qastp0wJ3oIUUlfJLqKfcDPwU7vcEFQ+wYGrx2 8Hj2n1hWr18/BLbX/Beo8laQNlh98CivMbqIuh9ffy6+V1a2yeWHJqWWpcaK+nN9iF Fo098ZDmKpaot+GXROiXWIU8ioOKn5IbREGevI+tdQ00GJIlcCpY/1/inTe2owxfjj aPerEt6wTZr22PvngJrKpnosLV5kgRxmg+N8RAPPkT6uryiSdk3mpVB4u3tZj+0xSm wrd8LdQB972LR3iegQTTrT8zPU92gubSbdMx2NEdPjXQOnKASGBE8hG4KX/BlmZxwB 1HVIVWoW1kipw== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmqp-008gvR-1Q; Mon, 29 Nov 2021 20:02:11 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 06/69] arm64: Add ARM64_HAS_NESTED_VIRT cpufeature Date: Mon, 29 Nov 2021 20:00:47 +0000 Message-Id: <20211129200150.351436-7-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Jintack Lim Add a new ARM64_HAS_NESTED_VIRT feature to indicate that the CPU has the ARMv8.3 nested virtualization capability, together with the 'kvm-arm.mode=nested' command line option. This will be used to support nested virtualization in KVM. Signed-off-by: Jintack Lim Signed-off-by: Andre Przywara Signed-off-by: Christoffer Dall [maz: moved the command-line option to kvm-arm.mode] Signed-off-by: Marc Zyngier --- .../admin-guide/kernel-parameters.txt | 7 +++++- arch/arm64/include/asm/kvm_host.h | 5 ++++ arch/arm64/kernel/cpufeature.c | 24 +++++++++++++++++++ arch/arm64/kvm/arm.c | 5 ++++ arch/arm64/tools/cpucaps | 1 + 5 files changed, 41 insertions(+), 1 deletion(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 9725c546a0d4..008d567b042f 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -2380,9 +2380,14 @@ state is kept private from the host. Not valid if the kernel is running in EL2. + nested: VHE-based mode with support for nested + virtualization. Requires at least ARMv8.3 + hardware. + Defaults to VHE/nVHE based on hardware support. Setting mode to "protected" will disable kexec and hibernation - for the host. + for the host. "nested" is experimental and should be + used with extreme caution. kvm-arm.vgic_v3_group0_trap= [KVM,ARM] Trap guest accesses to GICv3 group-0 diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 2a5f7f38006f..20d7cb450ef3 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -58,9 +58,14 @@ enum kvm_mode { KVM_MODE_DEFAULT, KVM_MODE_PROTECTED, + KVM_MODE_NV, KVM_MODE_NONE, }; +#ifdef CONFIG_KVM enum kvm_mode kvm_get_mode(void); +#else +static inline enum kvm_mode kvm_get_mode(void) { return KVM_MODE_NONE; }; +#endif DECLARE_STATIC_KEY_FALSE(userspace_irqchip_in_use); diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 6f3e677d88f1..4d57e98cdde5 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -1761,6 +1761,20 @@ static void cpu_copy_el2regs(const struct arm64_cpu_capabilities *__unused) write_sysreg(read_sysreg(tpidr_el1), tpidr_el2); } +static bool has_nested_virt_support(const struct arm64_cpu_capabilities *cap, + int scope) +{ + if (kvm_get_mode() != KVM_MODE_NV) + return false; + + if (!has_cpuid_feature(cap, scope)) { + pr_warn("unavailable: %s\n", cap->desc); + return false; + } + + return true; +} + static void cpu_has_fwb(const struct arm64_cpu_capabilities *__unused) { u64 val = read_sysreg_s(SYS_CLIDR_EL1); @@ -1994,6 +2008,16 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .matches = runs_at_el2, .cpu_enable = cpu_copy_el2regs, }, + { + .desc = "Nested Virtualization Support", + .capability = ARM64_HAS_NESTED_VIRT, + .type = ARM64_CPUCAP_SYSTEM_FEATURE, + .matches = has_nested_virt_support, + .sys_reg = SYS_ID_AA64MMFR2_EL1, + .sign = FTR_UNSIGNED, + .field_pos = ID_AA64MMFR2_NV_SHIFT, + .min_field_value = 1, + }, { .capability = ARM64_HAS_32BIT_EL0_DO_NOT_USE, .type = ARM64_CPUCAP_SYSTEM_FEATURE, diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 2f03cbfefe67..68da54d58cd0 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -2174,6 +2174,11 @@ static int __init early_kvm_mode_cfg(char *arg) return 0; } + if (strcmp(arg, "nested") == 0 && !WARN_ON(!is_kernel_in_hyp_mode())) { + kvm_mode = KVM_MODE_NV; + return 0; + } + if (strcmp(arg, "none") == 0) { kvm_mode = KVM_MODE_NONE; return 0; diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps index 870c39537dd0..a49864b56a07 100644 --- a/arch/arm64/tools/cpucaps +++ b/arch/arm64/tools/cpucaps @@ -26,6 +26,7 @@ HAS_GENERIC_AUTH_IMP_DEF HAS_IRQ_PRIO_MASKING HAS_LDAPR HAS_LSE_ATOMICS +HAS_NESTED_VIRT HAS_NO_FPSIMD HAS_NO_HW_PREFETCH HAS_PAN From patchwork Mon Nov 29 20:00:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645607 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56F6BC433FE for ; Mon, 29 Nov 2021 20:04:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350289AbhK2UHe (ORCPT ); Mon, 29 Nov 2021 15:07:34 -0500 Received: from ams.source.kernel.org ([145.40.68.75]:39364 "EHLO ams.source.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229688AbhK2UFd (ORCPT ); Mon, 29 Nov 2021 15:05:33 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id C05CCB815DA for ; Mon, 29 Nov 2021 20:02:14 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 317C7C53FD3; Mon, 29 Nov 2021 20:02:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216133; bh=pIwGsPjTdVawd4oyenBzZnOtLgPUkCj2S2moQP6b3Ws=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=W1G3NO6rkKyTeYqwhmnnqlPbw2Cc3mmcMibzno4ZuXq1Nr+2e6nUFz8XUp/l7aXqM LGYln5VcZJ9xnXSijnRjjA/Mwz96UYaUdzRl0EmtHWI/PSFBNwwsfTeyjacu54ojOs qgwrXGA3gIJ4ZvSQGHNrz/EPoTL4Oqlt48UHZOHHWYU+cgmDc1FXzcOdKd//nPFkyN yfLI2nHP9sq+2yCkNvYKLBXdR/A0yAsrgtjyaqegVY8e9t6KEl1RjgT1Ai1YB10wL8 sDxPAT1VBxTcazsww7bCTN+YgsEvDRcubItYz6ccdMzw/hh5IDVgimi0HmvH7HLpdi K+CcsKEwkGfbw== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmqp-008gvR-Ae; Mon, 29 Nov 2021 20:02:11 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 07/69] KVM: arm64: nv: Introduce nested virtualization VCPU feature Date: Mon, 29 Nov 2021 20:00:48 +0000 Message-Id: <20211129200150.351436-8-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Christoffer Dall Introduce the feature bit and a primitive that checks if the feature is set behind a static key check based on the cpus_have_const_cap check. Checking nested_virt_in_use() on systems without nested virt enabled should have neglgible overhead. We don't yet allow userspace to actually set this feature. Signed-off-by: Christoffer Dall Signed-off-by: Marc Zyngier Reviewed-by: Ganapatrao Kulkarni Reviewed-by: Russell King (Oracle) --- arch/arm64/include/asm/kvm_nested.h | 14 ++++++++++++++ arch/arm64/include/uapi/asm/kvm.h | 1 + 2 files changed, 15 insertions(+) create mode 100644 arch/arm64/include/asm/kvm_nested.h diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h new file mode 100644 index 000000000000..1028ac65a897 --- /dev/null +++ b/arch/arm64/include/asm/kvm_nested.h @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __ARM64_KVM_NESTED_H +#define __ARM64_KVM_NESTED_H + +#include + +static inline bool nested_virt_in_use(const struct kvm_vcpu *vcpu) +{ + return (!__is_defined(__KVM_NVHE_HYPERVISOR__) && + cpus_have_final_cap(ARM64_HAS_NESTED_VIRT) && + test_bit(KVM_ARM_VCPU_HAS_EL2, vcpu->arch.features)); +} + +#endif /* __ARM64_KVM_NESTED_H */ diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h index b3edde68bc3e..395a4c039bcc 100644 --- a/arch/arm64/include/uapi/asm/kvm.h +++ b/arch/arm64/include/uapi/asm/kvm.h @@ -106,6 +106,7 @@ struct kvm_regs { #define KVM_ARM_VCPU_SVE 4 /* enable SVE for this CPU */ #define KVM_ARM_VCPU_PTRAUTH_ADDRESS 5 /* VCPU uses address authentication */ #define KVM_ARM_VCPU_PTRAUTH_GENERIC 6 /* VCPU uses generic authentication */ +#define KVM_ARM_VCPU_HAS_EL2 7 /* Support nested virtualization */ struct kvm_vcpu_init { __u32 target; From patchwork Mon Nov 29 20:00:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645897 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F66FC433EF for ; Mon, 29 Nov 2021 22:19:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229624AbhK2WXI (ORCPT ); Mon, 29 Nov 2021 17:23:08 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56592 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232590AbhK2WVT (ORCPT ); Mon, 29 Nov 2021 17:21:19 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1A89FC08EAE8 for ; Mon, 29 Nov 2021 12:02:17 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 67759CE1413 for ; Mon, 29 Nov 2021 20:02:15 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 92D9AC53FAD; Mon, 29 Nov 2021 20:02:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216133; bh=r2zyjUbW431vWjGfEyo5zmYNkop2GFGeaSvf8QOmSMo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=EjYmTQirDu3m4f5lKlxvDLTYNjGSTemd7eeSZYsjZ0VxQd478nYCG5i7aTWeoZo+D 2yG75h2tKkwJHJGxk2QHkhgGx4z4TpZozGO6DKH5a2Y6mBinhzfQ3mPaHzIG9oj4m0 JbYB6IryWwoomt5rTBEHZTX34U1jp2Su3Ld88OT1cYWbUCLlMG0c8OtP5j07OPAJNY slrFUiSftmudeph1C8l9tYJedNHM9mcJEVMC/3n5n5egLKAawi/3q696wy/vRaoaA0 vegq+DNslIiL3pKXBT7LvK6t2RA4u1wlKFb8Wc+LHjdVRe4hZL0bsav3R+Baiw7mdk iVwZHj4aMHZmQ== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmqp-008gvR-Lh; Mon, 29 Nov 2021 20:02:11 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 08/69] KVM: arm64: nv: Reset VCPU to EL2 registers if VCPU nested virt is set Date: Mon, 29 Nov 2021 20:00:49 +0000 Message-Id: <20211129200150.351436-9-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Christoffer Dall Reset the VCPU with PSTATE.M = EL2h when the nested virtualization feature is enabled on the VCPU. Signed-off-by: Christoffer Dall [maz: rework register reset not to use empty data structures] Signed-off-by: Marc Zyngier Reviewed-by: Russell King (Oracle) --- arch/arm64/kvm/reset.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/arch/arm64/kvm/reset.c b/arch/arm64/kvm/reset.c index 426bd7fbc3fd..38a7182819fb 100644 --- a/arch/arm64/kvm/reset.c +++ b/arch/arm64/kvm/reset.c @@ -27,6 +27,7 @@ #include #include #include +#include #include /* Maximum phys_shift supported for any VM on this host */ @@ -38,6 +39,9 @@ static u32 kvm_ipa_limit; #define VCPU_RESET_PSTATE_EL1 (PSR_MODE_EL1h | PSR_A_BIT | PSR_I_BIT | \ PSR_F_BIT | PSR_D_BIT) +#define VCPU_RESET_PSTATE_EL2 (PSR_MODE_EL2h | PSR_A_BIT | PSR_I_BIT | \ + PSR_F_BIT | PSR_D_BIT) + #define VCPU_RESET_PSTATE_SVC (PSR_AA32_MODE_SVC | PSR_AA32_A_BIT | \ PSR_AA32_I_BIT | PSR_AA32_F_BIT) @@ -176,8 +180,8 @@ static bool vcpu_allowed_register_width(struct kvm_vcpu *vcpu) if (!cpus_have_const_cap(ARM64_HAS_32BIT_EL1) && is32bit) return false; - /* MTE is incompatible with AArch32 */ - if (kvm_has_mte(vcpu->kvm) && is32bit) + /* MTE and NV are incompatible with AArch32 */ + if ((kvm_has_mte(vcpu->kvm) || nested_virt_in_use(vcpu)) && is32bit) return false; /* Check that the vcpus are either all 32bit or all 64bit */ @@ -255,6 +259,8 @@ int kvm_reset_vcpu(struct kvm_vcpu *vcpu) default: if (test_bit(KVM_ARM_VCPU_EL1_32BIT, vcpu->arch.features)) { pstate = VCPU_RESET_PSTATE_SVC; + } else if (nested_virt_in_use(vcpu)) { + pstate = VCPU_RESET_PSTATE_EL2; } else { pstate = VCPU_RESET_PSTATE_EL1; } From patchwork Mon Nov 29 20:00:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645889 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B4C8CC433F5 for ; Mon, 29 Nov 2021 22:19:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232459AbhK2WW7 (ORCPT ); Mon, 29 Nov 2021 17:22:59 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56698 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232440AbhK2WVT (ORCPT ); Mon, 29 Nov 2021 17:21:19 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 53E59C08EAE6 for ; Mon, 29 Nov 2021 12:02:16 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 1B13AB815D5 for ; Mon, 29 Nov 2021 20:02:15 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D1FEBC53FCD; Mon, 29 Nov 2021 20:02:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216133; bh=k9i/mm0VR0eYW8zZqETLayHba2OMuNJ6fbzdXFUWLPs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=UH/FrwMgXFSirwdBURHI7OLrZJgb6U52JaGkFoO1DGQVApEMnRUd3Q5dGcRVHxqBd JjmS6oiaRMmYHDn2B+4/6PJuZsluuCMNhr53PvPrksdlwQoLmjBg2uk5rSCtIVO3Ak RbP8NZPouaOIACpzV/EP+mlaRP1Ty8mL51P5NUSixMGy5lRw+zhN9W1IYhlF/D9Sii 6omncsNC9SJM1jmZ1441sNsdgZlWZTM0BhYwHSu6Q0yXSVkFBWelG0URphiv/TgTbE LJi4ZgPc8ejI7yDXktfCtZgivnYYdKVykiH1p5Whrw6gH8h6uSVdL56cntQtgeVWx2 CWW1mD2sNkElw== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmqq-008gvR-1a; Mon, 29 Nov 2021 20:02:12 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 09/69] KVM: arm64: nv: Allow userspace to set PSR_MODE_EL2x Date: Mon, 29 Nov 2021 20:00:50 +0000 Message-Id: <20211129200150.351436-10-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Christoffer Dall We were not allowing userspace to set a more privileged mode for the VCPU than EL1, but we should allow this when nested virtualization is enabled for the VCPU. Signed-off-by: Christoffer Dall Signed-off-by: Marc Zyngier Reviewed-by: Russell King (Oracle) --- arch/arm64/kvm/guest.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/arch/arm64/kvm/guest.c b/arch/arm64/kvm/guest.c index e116c7767730..84d97f0394cb 100644 --- a/arch/arm64/kvm/guest.c +++ b/arch/arm64/kvm/guest.c @@ -24,6 +24,7 @@ #include #include #include +#include #include #include "trace.h" @@ -259,6 +260,11 @@ static int set_core_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) if (vcpu_el1_is_32bit(vcpu)) return -EINVAL; break; + case PSR_MODE_EL2h: + case PSR_MODE_EL2t: + if (vcpu_el1_is_32bit(vcpu) || !nested_virt_in_use(vcpu)) + return -EINVAL; + break; default: err = -EINVAL; goto out; From patchwork Mon Nov 29 20:00:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645673 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B39D9C433F5 for ; Mon, 29 Nov 2021 20:09:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232336AbhK2UND (ORCPT ); Mon, 29 Nov 2021 15:13:03 -0500 Received: from ams.source.kernel.org ([145.40.68.75]:43192 "EHLO ams.source.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232386AbhK2ULC (ORCPT ); Mon, 29 Nov 2021 15:11:02 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id B4529B8162E for ; Mon, 29 Nov 2021 20:07:43 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 70352C53FC7; Mon, 29 Nov 2021 20:07:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216462; bh=0qGiU00G9uRkqZI7ck+b3DyYWFEuyaGVfcPpiN7wNaw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=R/xThz0NC89STgtDguIEp9UXDf8oa+yvLCTOfVfr5GF8qlGRL4WAZ2NEkfJrrAjVl Q4TSUIbzW3hGKq6VEiCtZb4WqS5HLvEVYdRUTxvTF+awdJr+k9dXrDHco2pNrOfISJ +Ts0TQR7go/+kPsgfdIXMaHSNeszuaYK/m7syCZmL7+lfT7h19m96L0W+nlDiWl+Ol 4RrN8cgHp5v5R2t4Ms+ew9KiyTm2jxinJ7O06gRgEXtVhlJifSM5wXXc1zH0yWPJ/Z m3W7WPHT14WtglVxnMwMjOhwxx5rMeL/fNzKgH7LQ4gcjYwrXIdNf7gV80yM9DWvBb keugPq5K0mSkw== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmqq-008gvR-Ba; Mon, 29 Nov 2021 20:02:12 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 10/69] KVM: arm64: nv: Add EL2 system registers to vcpu context Date: Mon, 29 Nov 2021 20:00:51 +0000 Message-Id: <20211129200150.351436-11-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Add the minimal set of EL2 system registers to the vcpu context. Nothing uses them just yet. Reviewed-by: Andre Przywara Signed-off-by: Marc Zyngier Reviewed-by: Russell King (Oracle) --- arch/arm64/include/asm/kvm_host.h | 33 ++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 20d7cb450ef3..4f642a2e9c34 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -221,12 +221,43 @@ enum vcpu_sysreg { TFSR_EL1, /* Tag Fault Status Register (EL1) */ TFSRE0_EL1, /* Tag Fault Status Register (EL0) */ - /* 32bit specific registers. Keep them at the end of the range */ + /* 32bit specific registers. */ DACR32_EL2, /* Domain Access Control Register */ IFSR32_EL2, /* Instruction Fault Status Register */ FPEXC32_EL2, /* Floating-Point Exception Control Register */ DBGVCR32_EL2, /* Debug Vector Catch Register */ + /* EL2 registers */ + VPIDR_EL2, /* Virtualization Processor ID Register */ + VMPIDR_EL2, /* Virtualization Multiprocessor ID Register */ + SCTLR_EL2, /* System Control Register (EL2) */ + ACTLR_EL2, /* Auxiliary Control Register (EL2) */ + HCR_EL2, /* Hypervisor Configuration Register */ + MDCR_EL2, /* Monitor Debug Configuration Register (EL2) */ + CPTR_EL2, /* Architectural Feature Trap Register (EL2) */ + HSTR_EL2, /* Hypervisor System Trap Register */ + HACR_EL2, /* Hypervisor Auxiliary Control Register */ + TTBR0_EL2, /* Translation Table Base Register 0 (EL2) */ + TTBR1_EL2, /* Translation Table Base Register 1 (EL2) */ + TCR_EL2, /* Translation Control Register (EL2) */ + VTTBR_EL2, /* Virtualization Translation Table Base Register */ + VTCR_EL2, /* Virtualization Translation Control Register */ + SPSR_EL2, /* EL2 saved program status register */ + ELR_EL2, /* EL2 exception link register */ + AFSR0_EL2, /* Auxiliary Fault Status Register 0 (EL2) */ + AFSR1_EL2, /* Auxiliary Fault Status Register 1 (EL2) */ + ESR_EL2, /* Exception Syndrome Register (EL2) */ + FAR_EL2, /* Hypervisor IPA Fault Address Register */ + HPFAR_EL2, /* Hypervisor IPA Fault Address Register */ + MAIR_EL2, /* Memory Attribute Indirection Register (EL2) */ + AMAIR_EL2, /* Auxiliary Memory Attribute Indirection Register (EL2) */ + VBAR_EL2, /* Vector Base Address Register (EL2) */ + RVBAR_EL2, /* Reset Vector Base Address Register */ + CONTEXTIDR_EL2, /* Context ID Register (EL2) */ + TPIDR_EL2, /* EL2 Software Thread ID Register */ + CNTHCTL_EL2, /* Counter-timer Hypervisor Control register */ + SP_EL2, /* EL2 Stack Pointer */ + NR_SYS_REGS /* Nothing after this line! */ }; From patchwork Mon Nov 29 20:00:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645719 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 49E3DC433FE for ; Mon, 29 Nov 2021 21:24:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230269AbhK2V2A (ORCPT ); Mon, 29 Nov 2021 16:28:00 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44632 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231126AbhK2VZu (ORCPT ); Mon, 29 Nov 2021 16:25:50 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E088CC091D38 for ; Mon, 29 Nov 2021 12:07:32 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 88D84B8159E for ; Mon, 29 Nov 2021 20:07:32 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4577DC53FD1; Mon, 29 Nov 2021 20:07:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216451; bh=1Zg7MxAD5sCmymbTc5EDtJN7PJqUKc+Pm1ZgNIxAhTo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=W4XTdAZp+1uhCCcn0OpsCMPgwIR5JUIoCu7wuBOcTv6GnWqnYOSpk7oOLwNLzsslk jHjD6r3f1eIbv40BL/aKgl6QJnHny40IfBmr/BkGMam9zH+2up8EtXhLVkZ8QJPICr teUlY1ZlnGVNKUBoL1QWSixEXgPhBlKqNy0ZSPXKmzkE9BJolLS18vwVual0GyFncW tzxTJrUYOgMEAjgsq8cQ/TFMQziy2s/80QFumABtoAbNevB79XkTR5HKVNJCEGH2bm T6arsamaHwYfWDhww4EYkt00qTPgHLnTyi7fRcHHIZuuNECZx2LE/7SHbSyFH/BcGu R8zzIRbsBpvOA== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmqq-008gvR-Oz; Mon, 29 Nov 2021 20:02:12 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 11/69] KVM: arm64: nv: Add nested virt VCPU primitives for vEL2 VCPU state Date: Mon, 29 Nov 2021 20:00:52 +0000 Message-Id: <20211129200150.351436-12-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Christoffer Dall When running a nested hypervisor we commonly have to figure out if the VCPU mode is running in the context of a guest hypervisor or guest guest, or just a normal guest. Add convenient primitives for this. Signed-off-by: Christoffer Dall Signed-off-by: Marc Zyngier Reviewed-by: Russell King (Oracle) --- arch/arm64/include/asm/kvm_emulate.h | 55 ++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h index f4871e47b2d0..f4b079945d0f 100644 --- a/arch/arm64/include/asm/kvm_emulate.h +++ b/arch/arm64/include/asm/kvm_emulate.h @@ -176,6 +176,61 @@ static __always_inline void vcpu_set_reg(struct kvm_vcpu *vcpu, u8 reg_num, vcpu_gp_regs(vcpu)->regs[reg_num] = val; } +static inline bool vcpu_mode_el2_ctxt(const struct kvm_cpu_context *ctxt) +{ + unsigned long cpsr = ctxt->regs.pstate; + + switch (cpsr & (PSR_MODE32_BIT | PSR_MODE_MASK)) { + case PSR_MODE_EL2h: + case PSR_MODE_EL2t: + return true; + default: + return false; + } +} + +static inline bool vcpu_mode_el2(const struct kvm_vcpu *vcpu) +{ + return vcpu_mode_el2_ctxt(&vcpu->arch.ctxt); +} + +static inline bool __vcpu_el2_e2h_is_set(const struct kvm_cpu_context *ctxt) +{ + return ctxt_sys_reg(ctxt, HCR_EL2) & HCR_E2H; +} + +static inline bool vcpu_el2_e2h_is_set(const struct kvm_vcpu *vcpu) +{ + return __vcpu_el2_e2h_is_set(&vcpu->arch.ctxt); +} + +static inline bool __vcpu_el2_tge_is_set(const struct kvm_cpu_context *ctxt) +{ + return ctxt_sys_reg(ctxt, HCR_EL2) & HCR_TGE; +} + +static inline bool vcpu_el2_tge_is_set(const struct kvm_vcpu *vcpu) +{ + return __vcpu_el2_tge_is_set(&vcpu->arch.ctxt); +} + +static inline bool __is_hyp_ctxt(const struct kvm_cpu_context *ctxt) +{ + /* + * We are in a hypervisor context if the vcpu mode is EL2 or + * E2H and TGE bits are set. The latter means we are in the user space + * of the VHE kernel. ARMv8.1 ARM describes this as 'InHost' + */ + return vcpu_mode_el2_ctxt(ctxt) || + (__vcpu_el2_e2h_is_set(ctxt) && __vcpu_el2_tge_is_set(ctxt)) || + WARN_ON(__vcpu_el2_tge_is_set(ctxt)); +} + +static inline bool is_hyp_ctxt(const struct kvm_vcpu *vcpu) +{ + return __is_hyp_ctxt(&vcpu->arch.ctxt); +} + /* * The layout of SPSR for an AArch32 state is different when observed from an * AArch64 SPSR_ELx or an AArch32 SPSR_*. This function generates the AArch32 From patchwork Mon Nov 29 20:00:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645677 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9D38AC43219 for ; Mon, 29 Nov 2021 20:09:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234337AbhK2UNN (ORCPT ); Mon, 29 Nov 2021 15:13:13 -0500 Received: from sin.source.kernel.org ([145.40.73.55]:50598 "EHLO sin.source.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348444AbhK2UJk (ORCPT ); Mon, 29 Nov 2021 15:09:40 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id DB63BCE13DE for ; Mon, 29 Nov 2021 20:06:20 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 14815C53FAD; Mon, 29 Nov 2021 20:06:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216379; bh=4TJi1kUBu6v9+wzbXvDlDzBwCOCb2L+5EUFbJIjb+NQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=VkSRkgJHQr9xrb4whnF2jgDS9//gz7hb68jmqK9tnQkBV5Dskcve3TZwVYfx8eCQN e7CKNBKi2W+S9CeyZAsDYpVby4VmvZNfp78NNxejjMl2K4MgOGrl22RBz/xpXyHxpO qlZm0KaWVxo7etXipoP/bRpSu7KS7NKc0Ury/iPuP/dBBE2CRNuLat30vjvgahzxFO qtGNoDvZskbvW7Ndef+0zGm+ssFYQvfT6WiGkrz/LGxxE22htXErzW1Uko/F1V4CXH +zT5JGINnmZpgqx8aALYBByhDssFK9l9bEletvFM0kTtmfD6RuOm4VGsDTpIhHrYRX +eraax23+If7w== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmqr-008gvR-3Z; Mon, 29 Nov 2021 20:02:13 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 12/69] KVM: arm64: nv: Handle HCR_EL2.NV system register traps Date: Mon, 29 Nov 2021 20:00:53 +0000 Message-Id: <20211129200150.351436-13-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Jintack Lim ARM v8.3 introduces a new bit in the HCR_EL2, which is the NV bit. When this bit is set, accessing EL2 registers in EL1 traps to EL2. In addition, executing the following instructions in EL1 will trap to EL2: tlbi, at, eret, and msr/mrs instructions to access SP_EL1. Most of the instructions that trap to EL2 with the NV bit were undef at EL1 prior to ARM v8.3. The only instruction that was not undef is eret. This patch sets up a handler for EL2 registers and SP_EL1 register accesses at EL1. The host hypervisor keeps those register values in memory, and will emulate their behavior. This patch doesn't set the NV bit yet. It will be set in a later patch once nested virtualization support is completed. Signed-off-by: Jintack Lim [maz: added SCTLR_EL2 RES0/RES1 handling] Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/sysreg.h | 41 +++++++++++- arch/arm64/kvm/sys_regs.c | 109 ++++++++++++++++++++++++++++++-- 2 files changed, 144 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h index 615dd6278f8b..c77fe5401826 100644 --- a/arch/arm64/include/asm/sysreg.h +++ b/arch/arm64/include/asm/sysreg.h @@ -531,10 +531,26 @@ #define SYS_PMCCFILTR_EL0 sys_reg(3, 3, 14, 15, 7) +#define SYS_VPIDR_EL2 sys_reg(3, 4, 0, 0, 0) +#define SYS_VMPIDR_EL2 sys_reg(3, 4, 0, 0, 5) + #define SYS_SCTLR_EL2 sys_reg(3, 4, 1, 0, 0) +#define SYS_ACTLR_EL2 sys_reg(3, 4, 1, 0, 1) +#define SYS_HCR_EL2 sys_reg(3, 4, 1, 1, 0) +#define SYS_MDCR_EL2 sys_reg(3, 4, 1, 1, 1) +#define SYS_CPTR_EL2 sys_reg(3, 4, 1, 1, 2) +#define SYS_HSTR_EL2 sys_reg(3, 4, 1, 1, 3) #define SYS_HFGRTR_EL2 sys_reg(3, 4, 1, 1, 4) #define SYS_HFGWTR_EL2 sys_reg(3, 4, 1, 1, 5) #define SYS_HFGITR_EL2 sys_reg(3, 4, 1, 1, 6) +#define SYS_HACR_EL2 sys_reg(3, 4, 1, 1, 7) + +#define SYS_TTBR0_EL2 sys_reg(3, 4, 2, 0, 0) +#define SYS_TTBR1_EL2 sys_reg(3, 4, 2, 0, 1) +#define SYS_TCR_EL2 sys_reg(3, 4, 2, 0, 2) +#define SYS_VTTBR_EL2 sys_reg(3, 4, 2, 1, 0) +#define SYS_VTCR_EL2 sys_reg(3, 4, 2, 1, 2) + #define SYS_ZCR_EL2 sys_reg(3, 4, 1, 2, 0) #define SYS_TRFCR_EL2 sys_reg(3, 4, 1, 2, 1) #define SYS_DACR32_EL2 sys_reg(3, 4, 3, 0, 0) @@ -543,14 +559,26 @@ #define SYS_HAFGRTR_EL2 sys_reg(3, 4, 3, 1, 6) #define SYS_SPSR_EL2 sys_reg(3, 4, 4, 0, 0) #define SYS_ELR_EL2 sys_reg(3, 4, 4, 0, 1) +#define SYS_SP_EL1 sys_reg(3, 4, 4, 1, 0) #define SYS_IFSR32_EL2 sys_reg(3, 4, 5, 0, 1) +#define SYS_AFSR0_EL2 sys_reg(3, 4, 5, 1, 0) +#define SYS_AFSR1_EL2 sys_reg(3, 4, 5, 1, 1) #define SYS_ESR_EL2 sys_reg(3, 4, 5, 2, 0) #define SYS_VSESR_EL2 sys_reg(3, 4, 5, 2, 3) #define SYS_FPEXC32_EL2 sys_reg(3, 4, 5, 3, 0) #define SYS_TFSR_EL2 sys_reg(3, 4, 5, 6, 0) #define SYS_FAR_EL2 sys_reg(3, 4, 6, 0, 0) -#define SYS_VDISR_EL2 sys_reg(3, 4, 12, 1, 1) +#define SYS_FAR_EL2 sys_reg(3, 4, 6, 0, 0) +#define SYS_HPFAR_EL2 sys_reg(3, 4, 6, 0, 4) + +#define SYS_MAIR_EL2 sys_reg(3, 4, 10, 2, 0) +#define SYS_AMAIR_EL2 sys_reg(3, 4, 10, 3, 0) + +#define SYS_VBAR_EL2 sys_reg(3, 4, 12, 0, 0) +#define SYS_RVBAR_EL2 sys_reg(3, 4, 12, 0, 1) +#define SYS_RMR_EL2 sys_reg(3, 4, 12, 0, 2) +#define SYS_VDISR_EL2 sys_reg(3, 4, 12, 1, 1) #define __SYS__AP0Rx_EL2(x) sys_reg(3, 4, 12, 8, x) #define SYS_ICH_AP0R0_EL2 __SYS__AP0Rx_EL2(0) #define SYS_ICH_AP0R1_EL2 __SYS__AP0Rx_EL2(1) @@ -592,15 +620,24 @@ #define SYS_ICH_LR14_EL2 __SYS__LR8_EL2(6) #define SYS_ICH_LR15_EL2 __SYS__LR8_EL2(7) +#define SYS_CONTEXTIDR_EL2 sys_reg(3, 4, 13, 0, 1) +#define SYS_TPIDR_EL2 sys_reg(3, 4, 13, 0, 2) + +#define SYS_CNTVOFF_EL2 sys_reg(3, 4, 14, 0, 3) +#define SYS_CNTHCTL_EL2 sys_reg(3, 4, 14, 1, 0) + /* VHE encodings for architectural EL0/1 system registers */ #define SYS_SCTLR_EL12 sys_reg(3, 5, 1, 0, 0) #define SYS_CPACR_EL12 sys_reg(3, 5, 1, 0, 2) #define SYS_ZCR_EL12 sys_reg(3, 5, 1, 2, 0) + #define SYS_TTBR0_EL12 sys_reg(3, 5, 2, 0, 0) #define SYS_TTBR1_EL12 sys_reg(3, 5, 2, 0, 1) #define SYS_TCR_EL12 sys_reg(3, 5, 2, 0, 2) + #define SYS_SPSR_EL12 sys_reg(3, 5, 4, 0, 0) #define SYS_ELR_EL12 sys_reg(3, 5, 4, 0, 1) + #define SYS_AFSR0_EL12 sys_reg(3, 5, 5, 1, 0) #define SYS_AFSR1_EL12 sys_reg(3, 5, 5, 1, 1) #define SYS_ESR_EL12 sys_reg(3, 5, 5, 2, 0) @@ -618,6 +655,8 @@ #define SYS_CNTV_CTL_EL02 sys_reg(3, 5, 14, 3, 1) #define SYS_CNTV_CVAL_EL02 sys_reg(3, 5, 14, 3, 2) +#define SYS_SP_EL2 sys_reg(3, 6, 4, 1, 0) + /* Common SCTLR_ELx flags. */ #define SCTLR_ELx_DSSBS (BIT(44)) #define SCTLR_ELx_ATA (BIT(43)) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index e3ec1a44f94d..a23701f29858 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -105,6 +105,46 @@ static u32 get_ccsidr(u32 csselr) return ccsidr; } +static bool access_rw(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + if (p->is_write) + vcpu_write_sys_reg(vcpu, p->regval, r->reg); + else + p->regval = vcpu_read_sys_reg(vcpu, r->reg); + + return true; +} + +static bool access_sctlr_el2(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + if (p->is_write) { + u64 val = p->regval; + + if (vcpu_el2_e2h_is_set(vcpu) && vcpu_el2_tge_is_set(vcpu)) { + val &= ~(GENMASK_ULL(63,45) | GENMASK_ULL(34, 32) | + BIT_ULL(17) | BIT_ULL(9)); + val |= SCTLR_EL1_RES1; + } else { + val &= ~(GENMASK_ULL(63,45) | BIT_ULL(42) | + GENMASK_ULL(39, 38) | GENMASK_ULL(35, 32) | + BIT_ULL(26) | BIT_ULL(24) | BIT_ULL(20) | + BIT_ULL(17) | GENMASK_ULL(15, 14) | + GENMASK(10, 7)); + val |= SCTLR_EL2_RES1; + } + + vcpu_write_sys_reg(vcpu, val, r->reg); + } else { + p->regval = vcpu_read_sys_reg(vcpu, r->reg); + } + + return true; +} + /* * See note at ARMv7 ARM B1.14.4 (TL;DR: S/W ops are not easily virtualized). */ @@ -263,6 +303,14 @@ static bool trap_raz_wi(struct kvm_vcpu *vcpu, return read_zero(vcpu, p); } +static bool trap_undef(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + kvm_inject_undefined(vcpu); + return false; +} + /* * ARMv8.1 mandates at least a trivial LORegion implementation, where all the * RW registers are RES0 (which we can implement as RAZ/WI). On an ARMv8.0 @@ -342,12 +390,9 @@ static bool trap_debug_regs(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) { - if (p->is_write) { - vcpu_write_sys_reg(vcpu, p->regval, r->reg); + access_rw(vcpu, p, r); + if (p->is_write) vcpu->arch.flags |= KVM_ARM64_DEBUG_DIRTY; - } else { - p->regval = vcpu_read_sys_reg(vcpu, r->reg); - } trace_trap_reg(__func__, r->reg, p->is_write, p->regval); @@ -1411,6 +1456,18 @@ static unsigned int mte_visibility(const struct kvm_vcpu *vcpu, .set_user = set_raz_id_reg, \ } +static bool access_sp_el1(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + if (p->is_write) + __vcpu_sys_reg(vcpu, SP_EL1) = p->regval; + else + p->regval = __vcpu_sys_reg(vcpu, SP_EL1); + + return true; +} + /* * Architected system registers. * Important: Must be sorted ascending by Op0, Op1, CRn, CRm, Op2 @@ -1825,9 +1882,51 @@ static const struct sys_reg_desc sys_reg_descs[] = { { PMU_SYS_REG(SYS_PMCCFILTR_EL0), .access = access_pmu_evtyper, .reset = reset_val, .reg = PMCCFILTR_EL0, .val = 0 }, + { SYS_DESC(SYS_VPIDR_EL2), access_rw, reset_val, VPIDR_EL2, 0 }, + { SYS_DESC(SYS_VMPIDR_EL2), access_rw, reset_val, VMPIDR_EL2, 0 }, + + { SYS_DESC(SYS_SCTLR_EL2), access_sctlr_el2, reset_val, SCTLR_EL2, SCTLR_EL2_RES1 }, + { SYS_DESC(SYS_ACTLR_EL2), access_rw, reset_val, ACTLR_EL2, 0 }, + { SYS_DESC(SYS_HCR_EL2), access_rw, reset_val, HCR_EL2, 0 }, + { SYS_DESC(SYS_MDCR_EL2), access_rw, reset_val, MDCR_EL2, 0 }, + { SYS_DESC(SYS_CPTR_EL2), access_rw, reset_val, CPTR_EL2, CPTR_EL2_DEFAULT }, + { SYS_DESC(SYS_HSTR_EL2), access_rw, reset_val, HSTR_EL2, 0 }, + { SYS_DESC(SYS_HACR_EL2), access_rw, reset_val, HACR_EL2, 0 }, + + { SYS_DESC(SYS_TTBR0_EL2), access_rw, reset_val, TTBR0_EL2, 0 }, + { SYS_DESC(SYS_TTBR1_EL2), access_rw, reset_val, TTBR1_EL2, 0 }, + { SYS_DESC(SYS_TCR_EL2), access_rw, reset_val, TCR_EL2, TCR_EL2_RES1 }, + { SYS_DESC(SYS_VTTBR_EL2), access_rw, reset_val, VTTBR_EL2, 0 }, + { SYS_DESC(SYS_VTCR_EL2), access_rw, reset_val, VTCR_EL2, 0 }, + { SYS_DESC(SYS_DACR32_EL2), NULL, reset_unknown, DACR32_EL2 }, + { SYS_DESC(SYS_SPSR_EL2), access_rw, reset_val, SPSR_EL2, 0 }, + { SYS_DESC(SYS_ELR_EL2), access_rw, reset_val, ELR_EL2, 0 }, + { SYS_DESC(SYS_SP_EL1), access_sp_el1}, + { SYS_DESC(SYS_IFSR32_EL2), NULL, reset_unknown, IFSR32_EL2 }, + { SYS_DESC(SYS_AFSR0_EL2), access_rw, reset_val, AFSR0_EL2, 0 }, + { SYS_DESC(SYS_AFSR1_EL2), access_rw, reset_val, AFSR1_EL2, 0 }, + { SYS_DESC(SYS_ESR_EL2), access_rw, reset_val, ESR_EL2, 0 }, { SYS_DESC(SYS_FPEXC32_EL2), NULL, reset_val, FPEXC32_EL2, 0x700 }, + + { SYS_DESC(SYS_FAR_EL2), access_rw, reset_val, FAR_EL2, 0 }, + { SYS_DESC(SYS_HPFAR_EL2), access_rw, reset_val, HPFAR_EL2, 0 }, + + { SYS_DESC(SYS_MAIR_EL2), access_rw, reset_val, MAIR_EL2, 0 }, + { SYS_DESC(SYS_AMAIR_EL2), access_rw, reset_val, AMAIR_EL2, 0 }, + + { SYS_DESC(SYS_VBAR_EL2), access_rw, reset_val, VBAR_EL2, 0 }, + { SYS_DESC(SYS_RVBAR_EL2), access_rw, reset_val, RVBAR_EL2, 0 }, + { SYS_DESC(SYS_RMR_EL2), trap_undef }, + + { SYS_DESC(SYS_CONTEXTIDR_EL2), access_rw, reset_val, CONTEXTIDR_EL2, 0 }, + { SYS_DESC(SYS_TPIDR_EL2), access_rw, reset_val, TPIDR_EL2, 0 }, + + { SYS_DESC(SYS_CNTVOFF_EL2), access_rw, reset_val, CNTVOFF_EL2, 0 }, + { SYS_DESC(SYS_CNTHCTL_EL2), access_rw, reset_val, CNTHCTL_EL2, 0 }, + + { SYS_DESC(SYS_SP_EL2), NULL, reset_unknown, SP_EL2 }, }; static bool trap_dbgdidr(struct kvm_vcpu *vcpu, From patchwork Mon Nov 29 20:00:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645663 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1BFF0C433EF for ; Mon, 29 Nov 2021 20:08:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233598AbhK2UMJ (ORCPT ); Mon, 29 Nov 2021 15:12:09 -0500 Received: from sin.source.kernel.org ([145.40.73.55]:50856 "EHLO sin.source.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241950AbhK2UKI (ORCPT ); Mon, 29 Nov 2021 15:10:08 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id DC91ACE1582 for ; Mon, 29 Nov 2021 20:06:49 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 57588C53FD0; Mon, 29 Nov 2021 20:06:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216408; bh=5RWTuHCcT/NzXwjHXEmEf2WlTNGpWMr+EAFCdFEO4UA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hj7A7QRdCifiqUwhQXxUiyS1K/kg+1DXLRYtYgx++mxLK+DXxN0dOu63VggZrkTet pOGcfv6YFWR5X1XPCzKRUbahZI2zChMnNbQYcBjDBFBgqAFcrPcZDJTvtpFUJu6eIp w2aHqlJ6oxxOYK3UKjEzeWLjSVGexGoPtMum0GxogOsoMVnYwH1NXL4slIX2wTqknE Blw7mpyqaMQsykhQKj9wC0VoLoG4+kBIdSPg/Pc6iUi557T2qbekySX/1Wekd4t3UX LAXQKY9Crwk/fp7BlGBLALhIVM4/TKYKYiKuVXOzL3KqCREvXh6aTW8L1Zt9PmxL1S lE/5fPyKUyKDQ== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmqr-008gvR-Et; Mon, 29 Nov 2021 20:02:13 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 13/69] KVM: arm64: nv: Reset VMPIDR_EL2 and VPIDR_EL2 to sane values Date: Mon, 29 Nov 2021 20:00:54 +0000 Message-Id: <20211129200150.351436-14-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Christoffer Dall The VMPIDR_EL2 and VPIDR_EL2 are architecturally UNKNOWN at reset, but let's be nice to a guest hypervisor behaving foolishly and reset these to something reasonable anyway. Signed-off-by: Christoffer Dall Signed-off-by: Marc Zyngier Reviewed-by: Russell King (Oracle) --- arch/arm64/kvm/sys_regs.c | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index a23701f29858..730a24468915 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -618,7 +618,7 @@ static void reset_actlr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r) vcpu_write_sys_reg(vcpu, actlr, ACTLR_EL1); } -static void reset_mpidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r) +static u64 compute_reset_mpidr(struct kvm_vcpu *vcpu) { u64 mpidr; @@ -632,7 +632,24 @@ static void reset_mpidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r) mpidr = (vcpu->vcpu_id & 0x0f) << MPIDR_LEVEL_SHIFT(0); mpidr |= ((vcpu->vcpu_id >> 4) & 0xff) << MPIDR_LEVEL_SHIFT(1); mpidr |= ((vcpu->vcpu_id >> 12) & 0xff) << MPIDR_LEVEL_SHIFT(2); - vcpu_write_sys_reg(vcpu, (1ULL << 31) | mpidr, MPIDR_EL1); + mpidr |= (1ULL << 31); + + return mpidr; +} + +static void reset_mpidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r) +{ + vcpu_write_sys_reg(vcpu, compute_reset_mpidr(vcpu), MPIDR_EL1); +} + +static void reset_vmpidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r) +{ + vcpu_write_sys_reg(vcpu, compute_reset_mpidr(vcpu), VMPIDR_EL2); +} + +static void reset_vpidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r) +{ + vcpu_write_sys_reg(vcpu, read_cpuid_id(), VPIDR_EL2); } static unsigned int pmu_visibility(const struct kvm_vcpu *vcpu, @@ -1882,8 +1899,8 @@ static const struct sys_reg_desc sys_reg_descs[] = { { PMU_SYS_REG(SYS_PMCCFILTR_EL0), .access = access_pmu_evtyper, .reset = reset_val, .reg = PMCCFILTR_EL0, .val = 0 }, - { SYS_DESC(SYS_VPIDR_EL2), access_rw, reset_val, VPIDR_EL2, 0 }, - { SYS_DESC(SYS_VMPIDR_EL2), access_rw, reset_val, VMPIDR_EL2, 0 }, + { SYS_DESC(SYS_VPIDR_EL2), access_rw, reset_vpidr, VPIDR_EL2 }, + { SYS_DESC(SYS_VMPIDR_EL2), access_rw, reset_vmpidr, VMPIDR_EL2 }, { SYS_DESC(SYS_SCTLR_EL2), access_sctlr_el2, reset_val, SCTLR_EL2, SCTLR_EL2_RES1 }, { SYS_DESC(SYS_ACTLR_EL2), access_rw, reset_val, ACTLR_EL2, 0 }, From patchwork Mon Nov 29 20:00:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645761 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B7C80C433EF for ; Mon, 29 Nov 2021 21:44:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232793AbhK2VsB (ORCPT ); Mon, 29 Nov 2021 16:48:01 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49454 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232075AbhK2Vpw (ORCPT ); Mon, 29 Nov 2021 16:45:52 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E61CFC08EB37 for ; Mon, 29 Nov 2021 12:06:25 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 0F02FCE140C for ; Mon, 29 Nov 2021 20:06:24 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1E6D5C53FCD; Mon, 29 Nov 2021 20:06:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216382; bh=rh/pviVzMLqsX5+T/GJs5xCn6Bds3SmEx6SOqcX+U8g=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=aOWai1yS8P7i4+hJNipO48jDl/A0w6aKhTf2UfqP0ZfNGwTUFMugJz3750RkzkCVq wn9+MDb0H+Znk98rWEZAS4wZcFgCiNZq5WzRclzPuE3RCF6TBBMk1FgJJeLf8969B6 DYiFh8N94mcevoZ7cwzInUfF58aq/BEl4LqR/lwJ+AeYL1Uco3hZX/Tat+9CVYz3tL aWErrtcUsyhA4O2RagzEU9dUPVM5UkJ8+f3ayk4eGqgEcFLBKY0XBU26NFhlnXIbc5 KDZarVQ1hQsG8GKK/Et5088bZsUP+jW+P/8WolUTfawY3Iou5mls/fgFUfiBg/lHzA djOIW9yoFwNTQ== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmqr-008gvR-Qx; Mon, 29 Nov 2021 20:02:13 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 14/69] KVM: arm64: nv: Support virtual EL2 exceptions Date: Mon, 29 Nov 2021 20:00:55 +0000 Message-Id: <20211129200150.351436-15-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Jintack Lim Support injecting exceptions and performing exception returns to and from virtual EL2. This must be done entirely in software except when taking an exception from vEL0 to vEL2 when the virtual HCR_EL2.{E2H,TGE} == {1,1} (a VHE guest hypervisor). Signed-off-by: Jintack Lim Signed-off-by: Christoffer Dall [maz: switch to common exception injection framework] Signed-off-by: Marc Zyngier Reviewed-by: Ganapatrao Kulkarni --- .mailmap | 1 + arch/arm64/include/asm/kvm_arm.h | 17 +++ arch/arm64/include/asm/kvm_emulate.h | 10 ++ arch/arm64/include/asm/kvm_host.h | 1 + arch/arm64/kvm/Makefile | 2 +- arch/arm64/kvm/emulate-nested.c | 176 +++++++++++++++++++++++++++ arch/arm64/kvm/hyp/exception.c | 49 ++++++-- arch/arm64/kvm/inject_fault.c | 68 +++++++++-- arch/arm64/kvm/trace_arm.h | 59 +++++++++ 9 files changed, 362 insertions(+), 21 deletions(-) create mode 100644 arch/arm64/kvm/emulate-nested.c diff --git a/.mailmap b/.mailmap index 14314e3c5d5e..491238a888cb 100644 --- a/.mailmap +++ b/.mailmap @@ -167,6 +167,7 @@ Jeff Layton Jens Axboe Jens Osterkamp Jernej Skrabec + Jiri Slaby Jiri Slaby Jiri Slaby diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h index a39fcf318c77..589a6b92d741 100644 --- a/arch/arm64/include/asm/kvm_arm.h +++ b/arch/arm64/include/asm/kvm_arm.h @@ -359,4 +359,21 @@ #define CPACR_EL1_TTA (1 << 28) #define CPACR_EL1_DEFAULT (CPACR_EL1_FPEN | CPACR_EL1_ZEN_EL1EN) +#define kvm_mode_names \ + { PSR_MODE_EL0t, "EL0t" }, \ + { PSR_MODE_EL1t, "EL1t" }, \ + { PSR_MODE_EL1h, "EL1h" }, \ + { PSR_MODE_EL2t, "EL2t" }, \ + { PSR_MODE_EL2h, "EL2h" }, \ + { PSR_MODE_EL3t, "EL3t" }, \ + { PSR_MODE_EL3h, "EL3h" }, \ + { PSR_AA32_MODE_USR, "32-bit USR" }, \ + { PSR_AA32_MODE_FIQ, "32-bit FIQ" }, \ + { PSR_AA32_MODE_IRQ, "32-bit IRQ" }, \ + { PSR_AA32_MODE_SVC, "32-bit SVC" }, \ + { PSR_AA32_MODE_ABT, "32-bit ABT" }, \ + { PSR_AA32_MODE_HYP, "32-bit HYP" }, \ + { PSR_AA32_MODE_UND, "32-bit UND" }, \ + { PSR_AA32_MODE_SYS, "32-bit SYS" } + #endif /* __ARM64_KVM_ARM_H__ */ diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h index f4b079945d0f..46c849ba281e 100644 --- a/arch/arm64/include/asm/kvm_emulate.h +++ b/arch/arm64/include/asm/kvm_emulate.h @@ -33,6 +33,12 @@ enum exception_type { except_type_serror = 0x180, }; +#define kvm_exception_type_names \ + { except_type_sync, "SYNC" }, \ + { except_type_irq, "IRQ" }, \ + { except_type_fiq, "FIQ" }, \ + { except_type_serror, "SERROR" } + bool kvm_condition_valid32(const struct kvm_vcpu *vcpu); void kvm_skip_instr32(struct kvm_vcpu *vcpu); @@ -41,6 +47,10 @@ void kvm_inject_vabt(struct kvm_vcpu *vcpu); void kvm_inject_dabt(struct kvm_vcpu *vcpu, unsigned long addr); void kvm_inject_pabt(struct kvm_vcpu *vcpu, unsigned long addr); +void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu); +int kvm_inject_nested_sync(struct kvm_vcpu *vcpu, u64 esr_el2); +int kvm_inject_nested_irq(struct kvm_vcpu *vcpu); + static __always_inline bool vcpu_el1_is_32bit(struct kvm_vcpu *vcpu) { return !(vcpu->arch.hcr_el2 & HCR_RW); diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 4f642a2e9c34..7b6fe18ee450 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -477,6 +477,7 @@ struct kvm_vcpu_arch { #define KVM_ARM64_EXCEPT_AA64_ELx_SERR (3 << 9) #define KVM_ARM64_EXCEPT_AA64_EL1 (0 << 11) #define KVM_ARM64_EXCEPT_AA64_EL2 (1 << 11) +#define KVM_ARM64_EXCEPT_AA64_EL_MASK (1 << 11) /* * Overlaps with KVM_ARM64_EXCEPT_MASK on purpose so that it can't be diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile index 989bb5dad2c8..1f602526e9a2 100644 --- a/arch/arm64/kvm/Makefile +++ b/arch/arm64/kvm/Makefile @@ -16,7 +16,7 @@ kvm-y := $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/eventfd.o \ inject_fault.o va_layout.o handle_exit.o \ guest.o debug.o reset.o sys_regs.o \ vgic-sys-reg-v3.o fpsimd.o pmu.o \ - arch_timer.o trng.o\ + arch_timer.o trng.o emulate-nested.o \ vgic/vgic.o vgic/vgic-init.o \ vgic/vgic-irqfd.o vgic/vgic-v2.o \ vgic/vgic-v3.o vgic/vgic-v4.o \ diff --git a/arch/arm64/kvm/emulate-nested.c b/arch/arm64/kvm/emulate-nested.c new file mode 100644 index 000000000000..339e8272b01e --- /dev/null +++ b/arch/arm64/kvm/emulate-nested.c @@ -0,0 +1,176 @@ +/* + * Copyright (C) 2016 - Linaro and Columbia University + * Author: Jintack Lim + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see . + */ + +#include +#include + +#include +#include + +#include "hyp/include/hyp/adjust_pc.h" + +#include "trace.h" + +void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu) +{ + u64 spsr, elr, mode; + bool direct_eret; + + /* + * Going through the whole put/load motions is a waste of time + * if this is a VHE guest hypervisor returning to its own + * userspace, or the hypervisor performing a local exception + * return. No need to save/restore registers, no need to + * switch S2 MMU. Just do the canonical ERET. + */ + spsr = vcpu_read_sys_reg(vcpu, SPSR_EL2); + mode = spsr & (PSR_MODE_MASK | PSR_MODE32_BIT); + + direct_eret = (mode == PSR_MODE_EL0t && + vcpu_el2_e2h_is_set(vcpu) && + vcpu_el2_tge_is_set(vcpu)); + direct_eret |= (mode == PSR_MODE_EL2h || mode == PSR_MODE_EL2t); + + if (direct_eret) { + *vcpu_pc(vcpu) = vcpu_read_sys_reg(vcpu, ELR_EL2); + *vcpu_cpsr(vcpu) = spsr; + trace_kvm_nested_eret(vcpu, *vcpu_pc(vcpu), spsr); + return; + } + + preempt_disable(); + kvm_arch_vcpu_put(vcpu); + + elr = __vcpu_sys_reg(vcpu, ELR_EL2); + + trace_kvm_nested_eret(vcpu, elr, spsr); + + /* + * Note that the current exception level is always the virtual EL2, + * since we set HCR_EL2.NV bit only when entering the virtual EL2. + */ + *vcpu_pc(vcpu) = elr; + *vcpu_cpsr(vcpu) = spsr; + + kvm_arch_vcpu_load(vcpu, smp_processor_id()); + preempt_enable(); +} + +static void kvm_inject_el2_exception(struct kvm_vcpu *vcpu, u64 esr_el2, + enum exception_type type) +{ + trace_kvm_inject_nested_exception(vcpu, esr_el2, type); + + switch (type) { + case except_type_sync: + vcpu->arch.flags |= KVM_ARM64_EXCEPT_AA64_ELx_SYNC; + break; + case except_type_irq: + vcpu->arch.flags |= KVM_ARM64_EXCEPT_AA64_ELx_IRQ; + break; + default: + WARN_ONCE(1, "Unsupported EL2 exception injection %d\n", type); + } + + vcpu->arch.flags |= (KVM_ARM64_EXCEPT_AA64_EL2 | + KVM_ARM64_PENDING_EXCEPTION); + + vcpu_write_sys_reg(vcpu, esr_el2, ESR_EL2); +} + +/* + * Emulate taking an exception to EL2. + * See ARM ARM J8.1.2 AArch64.TakeException() + */ +static int kvm_inject_nested(struct kvm_vcpu *vcpu, u64 esr_el2, + enum exception_type type) +{ + u64 pstate, mode; + bool direct_inject; + + if (!nested_virt_in_use(vcpu)) { + kvm_err("Unexpected call to %s for the non-nesting configuration\n", + __func__); + return -EINVAL; + } + + /* + * As for ERET, we can avoid doing too much on the injection path by + * checking that we either took the exception from a VHE host + * userspace or from vEL2. In these cases, there is no change in + * translation regime (or anything else), so let's do as little as + * possible. + */ + pstate = *vcpu_cpsr(vcpu); + mode = pstate & (PSR_MODE_MASK | PSR_MODE32_BIT); + + direct_inject = (mode == PSR_MODE_EL0t && + vcpu_el2_e2h_is_set(vcpu) && + vcpu_el2_tge_is_set(vcpu)); + direct_inject |= (mode == PSR_MODE_EL2h || mode == PSR_MODE_EL2t); + + if (direct_inject) { + kvm_inject_el2_exception(vcpu, esr_el2, type); + return 1; + } + + preempt_disable(); + kvm_arch_vcpu_put(vcpu); + + kvm_inject_el2_exception(vcpu, esr_el2, type); + + /* + * A hard requirement is that a switch between EL1 and EL2 + * contexts has to happen between a put/load, so that we can + * pick the correct timer and interrupt configuration, among + * other things. + * + * Make sure the exception actually took place before we load + * the new context. + */ + __kvm_adjust_pc(vcpu); + + kvm_arch_vcpu_load(vcpu, smp_processor_id()); + preempt_enable(); + + return 1; +} + +int kvm_inject_nested_sync(struct kvm_vcpu *vcpu, u64 esr_el2) +{ + return kvm_inject_nested(vcpu, esr_el2, except_type_sync); +} + +int kvm_inject_nested_irq(struct kvm_vcpu *vcpu) +{ + /* + * Do not inject an irq if the: + * - Current exception level is EL2, and + * - virtual HCR_EL2.TGE == 0 + * - virtual HCR_EL2.IMO == 0 + * + * See Table D1-17 "Physical interrupt target and masking when EL3 is + * not implemented and EL2 is implemented" in ARM DDI 0487C.a. + */ + + if (vcpu_mode_el2(vcpu) && !vcpu_el2_tge_is_set(vcpu) && + !(__vcpu_sys_reg(vcpu, HCR_EL2) & HCR_IMO)) + return 1; + + /* esr_el2 value doesn't matter for exits due to irqs. */ + return kvm_inject_nested(vcpu, 0, except_type_irq); +} diff --git a/arch/arm64/kvm/hyp/exception.c b/arch/arm64/kvm/hyp/exception.c index 0418399e0a20..4ef5e86efd8b 100644 --- a/arch/arm64/kvm/hyp/exception.c +++ b/arch/arm64/kvm/hyp/exception.c @@ -13,6 +13,7 @@ #include #include #include +#include #if !defined (__KVM_NVHE_HYPERVISOR__) && !defined (__KVM_VHE_HYPERVISOR__) #error Hypervisor code only! @@ -22,7 +23,9 @@ static inline u64 __vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, int reg) { u64 val; - if (__vcpu_read_sys_reg_from_cpu(reg, &val)) + if (unlikely(nested_virt_in_use(vcpu))) + return vcpu_read_sys_reg(vcpu, reg); + else if (__vcpu_read_sys_reg_from_cpu(reg, &val)) return val; return __vcpu_sys_reg(vcpu, reg); @@ -30,14 +33,24 @@ static inline u64 __vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, int reg) static inline void __vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg) { - if (__vcpu_write_sys_reg_to_cpu(val, reg)) - return; - - __vcpu_sys_reg(vcpu, reg) = val; + if (unlikely(nested_virt_in_use(vcpu))) + vcpu_write_sys_reg(vcpu, val, reg); + else if (!__vcpu_write_sys_reg_to_cpu(val, reg)) + __vcpu_sys_reg(vcpu, reg) = val; } -static void __vcpu_write_spsr(struct kvm_vcpu *vcpu, u64 val) +static void __vcpu_write_spsr(struct kvm_vcpu *vcpu, unsigned long target_mode, + u64 val) { + if (unlikely(nested_virt_in_use(vcpu))) { + if (target_mode == PSR_MODE_EL1h) + vcpu_write_sys_reg(vcpu, val, SPSR_EL1); + else + vcpu_write_sys_reg(vcpu, val, SPSR_EL2); + + return; + } + write_sysreg_el1(val, SYS_SPSR); } @@ -97,6 +110,11 @@ static void enter_exception64(struct kvm_vcpu *vcpu, unsigned long target_mode, sctlr = __vcpu_read_sys_reg(vcpu, SCTLR_EL1); __vcpu_write_sys_reg(vcpu, *vcpu_pc(vcpu), ELR_EL1); break; + case PSR_MODE_EL2h: + vbar = __vcpu_read_sys_reg(vcpu, VBAR_EL2); + sctlr = __vcpu_read_sys_reg(vcpu, SCTLR_EL2); + __vcpu_write_sys_reg(vcpu, *vcpu_pc(vcpu), ELR_EL2); + break; default: /* Don't do that */ BUG(); @@ -149,7 +167,7 @@ static void enter_exception64(struct kvm_vcpu *vcpu, unsigned long target_mode, new |= target_mode; *vcpu_cpsr(vcpu) = new; - __vcpu_write_spsr(vcpu, old); + __vcpu_write_spsr(vcpu, target_mode, old); } /* @@ -320,11 +338,22 @@ static void kvm_inject_exception(struct kvm_vcpu *vcpu) KVM_ARM64_EXCEPT_AA64_EL1): enter_exception64(vcpu, PSR_MODE_EL1h, except_type_sync); break; + + case (KVM_ARM64_EXCEPT_AA64_ELx_SYNC | + KVM_ARM64_EXCEPT_AA64_EL2): + enter_exception64(vcpu, PSR_MODE_EL2h, except_type_sync); + break; + + case (KVM_ARM64_EXCEPT_AA64_ELx_IRQ | + KVM_ARM64_EXCEPT_AA64_EL2): + enter_exception64(vcpu, PSR_MODE_EL2h, except_type_irq); + break; + default: /* - * Only EL1_SYNC makes sense so far, EL2_{SYNC,IRQ} - * will be implemented at some point. Everything - * else gets silently ignored. + * Only EL1_SYNC and EL2_{SYNC,IRQ} makes + * sense so far. Everything else gets silently + * ignored. */ break; } diff --git a/arch/arm64/kvm/inject_fault.c b/arch/arm64/kvm/inject_fault.c index b47df73e98d7..5dcf3f8b08b8 100644 --- a/arch/arm64/kvm/inject_fault.c +++ b/arch/arm64/kvm/inject_fault.c @@ -12,19 +12,58 @@ #include #include +#include #include +static void pend_sync_exception(struct kvm_vcpu *vcpu) +{ + vcpu->arch.flags |= (KVM_ARM64_EXCEPT_AA64_ELx_SYNC | + KVM_ARM64_PENDING_EXCEPTION); + + /* If not nesting, EL1 is the only possible exception target */ + if (likely(!nested_virt_in_use(vcpu))) { + vcpu->arch.flags |= KVM_ARM64_EXCEPT_AA64_EL1; + return; + } + + /* + * With NV, we need to pick between EL1 and EL2. Note that we + * never deal with a nesting exception here, hence never + * changing context, and the exception itself can be delayed + * until the next entry. + */ + switch(*vcpu_cpsr(vcpu) & PSR_MODE_MASK) { + case PSR_MODE_EL2h: + case PSR_MODE_EL2t: + vcpu->arch.flags |= KVM_ARM64_EXCEPT_AA64_EL2; + break; + case PSR_MODE_EL1h: + case PSR_MODE_EL1t: + vcpu->arch.flags |= KVM_ARM64_EXCEPT_AA64_EL1; + break; + case PSR_MODE_EL0t: + if (vcpu_el2_tge_is_set(vcpu) & HCR_TGE) + vcpu->arch.flags |= KVM_ARM64_EXCEPT_AA64_EL2; + else + vcpu->arch.flags |= KVM_ARM64_EXCEPT_AA64_EL1; + break; + default: + BUG(); + } +} + +static bool match_target_el(struct kvm_vcpu *vcpu, unsigned long target) +{ + return (vcpu->arch.flags & KVM_ARM64_EXCEPT_AA64_EL_MASK) == target; +} + static void inject_abt64(struct kvm_vcpu *vcpu, bool is_iabt, unsigned long addr) { unsigned long cpsr = *vcpu_cpsr(vcpu); bool is_aarch32 = vcpu_mode_is_32bit(vcpu); u32 esr = 0; - vcpu->arch.flags |= (KVM_ARM64_EXCEPT_AA64_EL1 | - KVM_ARM64_EXCEPT_AA64_ELx_SYNC | - KVM_ARM64_PENDING_EXCEPTION); - - vcpu_write_sys_reg(vcpu, addr, FAR_EL1); + pend_sync_exception(vcpu); /* * Build an {i,d}abort, depending on the level and the @@ -45,16 +84,22 @@ static void inject_abt64(struct kvm_vcpu *vcpu, bool is_iabt, unsigned long addr if (!is_iabt) esr |= ESR_ELx_EC_DABT_LOW << ESR_ELx_EC_SHIFT; - vcpu_write_sys_reg(vcpu, esr | ESR_ELx_FSC_EXTABT, ESR_EL1); + esr |= ESR_ELx_FSC_EXTABT; + + if (match_target_el(vcpu, KVM_ARM64_EXCEPT_AA64_EL1)) { + vcpu_write_sys_reg(vcpu, addr, FAR_EL1); + vcpu_write_sys_reg(vcpu, esr, ESR_EL1); + } else { + vcpu_write_sys_reg(vcpu, addr, FAR_EL2); + vcpu_write_sys_reg(vcpu, esr, ESR_EL2); + } } static void inject_undef64(struct kvm_vcpu *vcpu) { u32 esr = (ESR_ELx_EC_UNKNOWN << ESR_ELx_EC_SHIFT); - vcpu->arch.flags |= (KVM_ARM64_EXCEPT_AA64_EL1 | - KVM_ARM64_EXCEPT_AA64_ELx_SYNC | - KVM_ARM64_PENDING_EXCEPTION); + pend_sync_exception(vcpu); /* * Build an unknown exception, depending on the instruction @@ -63,7 +108,10 @@ static void inject_undef64(struct kvm_vcpu *vcpu) if (kvm_vcpu_trap_il_is32bit(vcpu)) esr |= ESR_ELx_IL; - vcpu_write_sys_reg(vcpu, esr, ESR_EL1); + if (match_target_el(vcpu, KVM_ARM64_EXCEPT_AA64_EL1)) + vcpu_write_sys_reg(vcpu, esr, ESR_EL1); + else + vcpu_write_sys_reg(vcpu, esr, ESR_EL2); } #define DFSR_FSC_EXTABT_LPAE 0x10 diff --git a/arch/arm64/kvm/trace_arm.h b/arch/arm64/kvm/trace_arm.h index 33e4e7dd2719..f3e46a976125 100644 --- a/arch/arm64/kvm/trace_arm.h +++ b/arch/arm64/kvm/trace_arm.h @@ -2,6 +2,7 @@ #if !defined(_TRACE_ARM_ARM64_KVM_H) || defined(TRACE_HEADER_MULTI_READ) #define _TRACE_ARM_ARM64_KVM_H +#include #include #include @@ -301,6 +302,64 @@ TRACE_EVENT(kvm_timer_emulate, __entry->timer_idx, __entry->should_fire) ); +TRACE_EVENT(kvm_nested_eret, + TP_PROTO(struct kvm_vcpu *vcpu, unsigned long elr_el2, + unsigned long spsr_el2), + TP_ARGS(vcpu, elr_el2, spsr_el2), + + TP_STRUCT__entry( + __field(struct kvm_vcpu *, vcpu) + __field(unsigned long, elr_el2) + __field(unsigned long, spsr_el2) + __field(unsigned long, target_mode) + __field(unsigned long, hcr_el2) + ), + + TP_fast_assign( + __entry->vcpu = vcpu; + __entry->elr_el2 = elr_el2; + __entry->spsr_el2 = spsr_el2; + __entry->target_mode = spsr_el2 & (PSR_MODE_MASK | PSR_MODE32_BIT); + __entry->hcr_el2 = __vcpu_sys_reg(vcpu, HCR_EL2); + ), + + TP_printk("elr_el2: 0x%lx spsr_el2: 0x%08lx (M: %s) hcr_el2: %lx", + __entry->elr_el2, __entry->spsr_el2, + __print_symbolic(__entry->target_mode, kvm_mode_names), + __entry->hcr_el2) +); + +TRACE_EVENT(kvm_inject_nested_exception, + TP_PROTO(struct kvm_vcpu *vcpu, u64 esr_el2, int type), + TP_ARGS(vcpu, esr_el2, type), + + TP_STRUCT__entry( + __field(struct kvm_vcpu *, vcpu) + __field(unsigned long, esr_el2) + __field(int, type) + __field(unsigned long, spsr_el2) + __field(unsigned long, pc) + __field(unsigned long, source_mode) + __field(unsigned long, hcr_el2) + ), + + TP_fast_assign( + __entry->vcpu = vcpu; + __entry->esr_el2 = esr_el2; + __entry->type = type; + __entry->spsr_el2 = *vcpu_cpsr(vcpu); + __entry->pc = *vcpu_pc(vcpu); + __entry->source_mode = *vcpu_cpsr(vcpu) & (PSR_MODE_MASK | PSR_MODE32_BIT); + __entry->hcr_el2 = __vcpu_sys_reg(vcpu, HCR_EL2); + ), + + TP_printk("%s: esr_el2 0x%lx elr_el2: 0x%lx spsr_el2: 0x%08lx (M: %s) hcr_el2: %lx", + __print_symbolic(__entry->type, kvm_exception_type_names), + __entry->esr_el2, __entry->pc, __entry->spsr_el2, + __print_symbolic(__entry->source_mode, kvm_mode_names), + __entry->hcr_el2) +); + #endif /* _TRACE_ARM_ARM64_KVM_H */ #undef TRACE_INCLUDE_PATH From patchwork Mon Nov 29 20:00:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645767 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3FC90C433F5 for ; Mon, 29 Nov 2021 21:46:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232266AbhK2Vto (ORCPT ); Mon, 29 Nov 2021 16:49:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49404 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229722AbhK2Vrn (ORCPT ); Mon, 29 Nov 2021 16:47:43 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 75C3AC08EB3C for ; Mon, 29 Nov 2021 12:06:37 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id E03C6CE1410 for ; Mon, 29 Nov 2021 20:06:35 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 218E1C53FCD; Mon, 29 Nov 2021 20:06:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216394; bh=rvoIj5sBGV8NY7D6FsGInJvNPEPNgZDlDa3Ec2tMPLA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=BUrEvQgIPIX/tY3L8hxXp2CmBWG0KgRQdqsUDv8MxExJYJdaDChMk1BU1+/xrpUTX TvCNHAMsWO3GLcuP3kQt/Y2RyeRac6RKB9sLkHacqcR/30KXlKfvKUMWhJikuZbfrd +pymOOdYD4pqnWAFrGMlNSYP/15WXbowSpVaOuZrImg65eiQQ/aXDJI+sMMhfOpZKH UNQrC0kvGqvU+sOaLuspN20yjeYrysDwAoPj/ybdDTr3pA5eY3LjuEW5HB4I83LTgM 2V0VIvOH3cZEpDs3B37rSZO6k59e+NlFHM/Yg4u7m7g8atMhB3L8MFCHItetV5Wdzg Zb83gaRO4RLWw== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmqs-008gvR-5A; Mon, 29 Nov 2021 20:02:14 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 15/69] KVM: arm64: nv: Inject HVC exceptions to the virtual EL2 Date: Mon, 29 Nov 2021 20:00:56 +0000 Message-Id: <20211129200150.351436-16-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Jintack Lim As we expect all PSCI calls from the L1 hypervisor to be performed using SMC when nested virtualization is enabled, it is clear that all HVC instruction from the VM (including from the virtual EL2) are supposed to handled in the virtual EL2. Forward these to EL2 as required. Signed-off-by: Jintack Lim [maz: add handling of HCR_EL2.HCD] Signed-off-by: Marc Zyngier Reviewed-by: Russell King (Oracle) Reviewed-by: Alexandru Elisei --- arch/arm64/kvm/handle_exit.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/arch/arm64/kvm/handle_exit.c b/arch/arm64/kvm/handle_exit.c index 275a27368a04..1fd1c6dfd6a0 100644 --- a/arch/arm64/kvm/handle_exit.c +++ b/arch/arm64/kvm/handle_exit.c @@ -16,6 +16,7 @@ #include #include #include +#include #include #include @@ -40,6 +41,16 @@ static int handle_hvc(struct kvm_vcpu *vcpu) kvm_vcpu_hvc_get_imm(vcpu)); vcpu->stat.hvc_exit_stat++; + /* Forward hvc instructions to the virtual EL2 if the guest has EL2. */ + if (nested_virt_in_use(vcpu)) { + if (vcpu_read_sys_reg(vcpu, HCR_EL2) & HCR_HCD) + kvm_inject_undefined(vcpu); + else + kvm_inject_nested_sync(vcpu, kvm_vcpu_get_esr(vcpu)); + + return 1; + } + ret = kvm_hvc_call_handler(vcpu); if (ret < 0) { vcpu_set_reg(vcpu, 0, ~0UL); From patchwork Mon Nov 29 20:00:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645739 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 10F15C433FE for ; Mon, 29 Nov 2021 21:42:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232398AbhK2VqA (ORCPT ); Mon, 29 Nov 2021 16:46:00 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48148 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230164AbhK2Vn5 (ORCPT ); Mon, 29 Nov 2021 16:43:57 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5BDAAC08EB2E for ; Mon, 29 Nov 2021 12:05:44 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 2596CB815E9 for ; Mon, 29 Nov 2021 20:05:43 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E841DC53FAD; Mon, 29 Nov 2021 20:05:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216341; bh=TV2sBZRwdHftC0dlVJ0yGN93HhbX2idCTJEoozqtzwY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=gpZSdb1gb3OJUOErbA20l/whhEI5HkiFIKpyaNEGqvngcOFYONXsqSbixtJTyBzk1 3N8yk6zd0wubJK8jajQAQMq9rPa9K1JOYLe5dmNYo2hYCg8OC1KdzpKzdD2G7mIGeY jTyAWzUW8RmQrwI2+HMZh60PfSyP402E3w6Vw8hrfXprCzbZHFb/6Qb05IkHcLAJxF 3f5fFvrxn1fXAfcnrbgkc3pzC8fjXEQc4imEf8CBoZnh6+9Ts+K62kNeIfP2e4HU68 PLxJX+JeZ1bMkKD62yWdMYlG6IHfzaEAlTzZRAXjYiXHHp7pFhUD2aFD4D6bZjcmG2 HSHvgS5ZINYGA== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmqs-008gvR-Ew; Mon, 29 Nov 2021 20:02:14 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 16/69] KVM: arm64: nv: Handle trapped ERET from virtual EL2 Date: Mon, 29 Nov 2021 20:00:57 +0000 Message-Id: <20211129200150.351436-17-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Christoffer Dall When a guest hypervisor running virtual EL2 in EL1 executes an ERET instruction, we will have set HCR_EL2.NV which traps ERET to EL2, so that we can emulate the exception return in software. Signed-off-by: Christoffer Dall Signed-off-by: Marc Zyngier Reviewed-by: Russell King (Oracle) Reviewed-by: Alexandru Elisei --- arch/arm64/include/asm/esr.h | 5 +++++ arch/arm64/include/asm/kvm_arm.h | 2 +- arch/arm64/kvm/handle_exit.c | 10 ++++++++++ 3 files changed, 16 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/esr.h b/arch/arm64/include/asm/esr.h index d52a0b269ee8..6835e4231119 100644 --- a/arch/arm64/include/asm/esr.h +++ b/arch/arm64/include/asm/esr.h @@ -257,6 +257,11 @@ (((e) & ESR_ELx_SYS64_ISS_OP2_MASK) >> \ ESR_ELx_SYS64_ISS_OP2_SHIFT)) +/* ISS field definitions for ERET/ERETAA/ERETAB trapping */ + +#define ESR_ELx_ERET_ISS_ERET_ERETAx 0x2 +#define ESR_ELx_ERET_ISS_ERETA_ERATAB 0x1 + /* * ISS field definitions for floating-point exception traps * (FP_EXC_32/FP_EXC_64). diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h index 589a6b92d741..0a0ee998ec5a 100644 --- a/arch/arm64/include/asm/kvm_arm.h +++ b/arch/arm64/include/asm/kvm_arm.h @@ -353,7 +353,7 @@ ECN(SP_ALIGN), ECN(FP_EXC32), ECN(FP_EXC64), ECN(SERROR), \ ECN(BREAKPT_LOW), ECN(BREAKPT_CUR), ECN(SOFTSTP_LOW), \ ECN(SOFTSTP_CUR), ECN(WATCHPT_LOW), ECN(WATCHPT_CUR), \ - ECN(BKPT32), ECN(VECTOR32), ECN(BRK64) + ECN(BKPT32), ECN(VECTOR32), ECN(BRK64), ECN(ERET) #define CPACR_EL1_FPEN (3 << 20) #define CPACR_EL1_TTA (1 << 28) diff --git a/arch/arm64/kvm/handle_exit.c b/arch/arm64/kvm/handle_exit.c index 1fd1c6dfd6a0..95ae624d6aa8 100644 --- a/arch/arm64/kvm/handle_exit.c +++ b/arch/arm64/kvm/handle_exit.c @@ -169,6 +169,15 @@ static int kvm_handle_ptrauth(struct kvm_vcpu *vcpu) return 1; } +static int kvm_handle_eret(struct kvm_vcpu *vcpu) +{ + if (kvm_vcpu_get_esr(vcpu) & ESR_ELx_ERET_ISS_ERET_ERETAx) + return kvm_handle_ptrauth(vcpu); + + kvm_emulate_nested_eret(vcpu); + return 1; +} + static exit_handle_fn arm_exit_handlers[] = { [0 ... ESR_ELx_EC_MAX] = kvm_handle_unknown_ec, [ESR_ELx_EC_WFx] = kvm_handle_wfx, @@ -183,6 +192,7 @@ static exit_handle_fn arm_exit_handlers[] = { [ESR_ELx_EC_SMC64] = handle_smc, [ESR_ELx_EC_SYS64] = kvm_handle_sys_reg, [ESR_ELx_EC_SVE] = handle_sve, + [ESR_ELx_EC_ERET] = kvm_handle_eret, [ESR_ELx_EC_IABT_LOW] = kvm_handle_guest_abort, [ESR_ELx_EC_DABT_LOW] = kvm_handle_guest_abort, [ESR_ELx_EC_SOFTSTP_LOW]= kvm_handle_guest_debug, From patchwork Mon Nov 29 20:00:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645723 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A2D32C433EF for ; Mon, 29 Nov 2021 21:24:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231151AbhK2V2B (ORCPT ); Mon, 29 Nov 2021 16:28:01 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44204 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231149AbhK2VZu (ORCPT ); Mon, 29 Nov 2021 16:25:50 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 768AFC091D34 for ; Mon, 29 Nov 2021 12:07:21 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id E343BCE1415 for ; Mon, 29 Nov 2021 20:07:19 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 39F8EC53FCF; Mon, 29 Nov 2021 20:07:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216438; bh=rMT0KBgk12w5iRVerxRsvVEXfThM4QCeR6Jk8Pujwmo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=rifcdI6l8WjhlLYjlt/9A6EMsAB9GdnMU4eiuEtIzAO1i/Od2g3mO2w5RNtVwWDmN IbgHIgjjF/nnq/AOrbHots+bw1v33BrnB6KkshFZr3oN7U4utMAn1W1fS3MiLo3tJ4 DdIps38jYXnRL60ZU4nSFdsA54J1lwTHKrbpkWUfDEWTBc0A6FysveNj3NIhwGhapf CluvdxALrXbCF49TKgQMIpNo3kIGtT6tOJ0fGGE/h7+F0nHOgeuvNkn0f+99X+384a MHBs9P7WgrW2wk2j31MhSTgDAnnqdoxarLwLnBu/w2fbhoMJYf8xyJNK3ZESGxZURK HxqdXh7NDbQRg== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmqs-008gvR-OK; Mon, 29 Nov 2021 20:02:14 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 17/69] KVM: arm64: nv: Add non-VHE-EL2->EL1 translation helpers Date: Mon, 29 Nov 2021 20:00:58 +0000 Message-Id: <20211129200150.351436-18-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Some EL2 system registers immediately affect the current execution of the system, so we need to use their respective EL1 counterparts. For this we need to define a mapping between the two. In general, this only affects non-VHE guest hypervisors, as VHE system registers are compatible with the EL1 counterparts. These helpers will get used in subsequent patches. Co-developed-by: Andre Przywara Signed-off-by: Andre Przywara Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_nested.h | 50 +++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index 1028ac65a897..67a2c0d05233 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -2,6 +2,7 @@ #ifndef __ARM64_KVM_NESTED_H #define __ARM64_KVM_NESTED_H +#include #include static inline bool nested_virt_in_use(const struct kvm_vcpu *vcpu) @@ -11,4 +12,53 @@ static inline bool nested_virt_in_use(const struct kvm_vcpu *vcpu) test_bit(KVM_ARM_VCPU_HAS_EL2, vcpu->arch.features)); } +/* Translation helpers from non-VHE EL2 to EL1 */ +static inline u64 tcr_el2_ips_to_tcr_el1_ps(u64 tcr_el2) +{ + return (u64)FIELD_GET(TCR_EL2_PS_MASK, tcr_el2) << TCR_IPS_SHIFT; +} + +static inline u64 translate_tcr_el2_to_tcr_el1(u64 tcr) +{ + return TCR_EPD1_MASK | /* disable TTBR1_EL1 */ + ((tcr & TCR_EL2_TBI) ? TCR_TBI0 : 0) | + tcr_el2_ips_to_tcr_el1_ps(tcr) | + (tcr & TCR_EL2_TG0_MASK) | + (tcr & TCR_EL2_ORGN0_MASK) | + (tcr & TCR_EL2_IRGN0_MASK) | + (tcr & TCR_EL2_T0SZ_MASK); +} + +static inline u64 translate_cptr_el2_to_cpacr_el1(u64 cptr_el2) +{ + u64 cpacr_el1 = 0; + + if (!(cptr_el2 & CPTR_EL2_TFP)) + cpacr_el1 |= CPACR_EL1_FPEN; + if (cptr_el2 & CPTR_EL2_TTA) + cpacr_el1 |= CPACR_EL1_TTA; + if (!(cptr_el2 & CPTR_EL2_TZ)) + cpacr_el1 |= CPACR_EL1_ZEN; + + return cpacr_el1; +} + +static inline u64 translate_sctlr_el2_to_sctlr_el1(u64 sctlr) +{ + /* Bit 20 is RES1 in SCTLR_EL1, but RES0 in SCTLR_EL2 */ + return sctlr | BIT(20); +} + +static inline u64 translate_ttbr0_el2_to_ttbr0_el1(u64 ttbr0) +{ + /* Force ASID to 0 (ASID 0 or RES0) */ + return ttbr0 & ~GENMASK_ULL(63, 48); +} + +static inline u64 translate_cnthctl_el2_to_cntkctl_el1(u64 cnthctl) +{ + return ((FIELD_GET(CNTHCTL_EL1PCTEN | CNTHCTL_EL1PCEN, cnthctl) << 10) | + (cnthctl & (CNTHCTL_EVNTI | CNTHCTL_EVNTDIR | CNTHCTL_EVNTEN))); +} + #endif /* __ARM64_KVM_NESTED_H */ From patchwork Mon Nov 29 20:00:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645731 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 226B7C433F5 for ; Mon, 29 Nov 2021 21:42:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229473AbhK2Vpy (ORCPT ); Mon, 29 Nov 2021 16:45:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48118 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232165AbhK2Vnw (ORCPT ); Mon, 29 Nov 2021 16:43:52 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B403FC091D14 for ; Mon, 29 Nov 2021 12:05:38 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 5660BB815E6 for ; Mon, 29 Nov 2021 20:05:37 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id ED6BFC53FAD; Mon, 29 Nov 2021 20:05:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216336; bh=vZ5+4M/yS/H0Yg3zlLXG6/fGkpBg6B4sjXc53adq3EI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ogw3ZFnAfhBapauo0udzHMlJ+bHG6Y1dZbhMrvq+kQqTPm1n20fYG8VKUCyUdFE5s m4f7/gDmx7Y91sR9FDThOBNejb10KvPQecDoqPeojDLbwrUO8PGL6xUfTcfGhdrEeD sSdAx4JsontFLbxLcp7eb6qDs1jtoGtSorwGv0S3YV+tITRyP4Jx3bJ6566gCvyOHM dxfMi5gjMA6Ap0J0s1gUivfwHVXvEp+VFV9mEgZxznz4H55bwRsmuRNEckji7qacPP 6nb/wCxYQw3d0+OPz0uclUSh3Xk4ptKQ0ZuyD8ypzHJqmIrovz5rx1sBlqh8csNFxk wBdGhKIkudw8Q== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmqt-008gvR-1X; Mon, 29 Nov 2021 20:02:15 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 18/69] KVM: arm64: nv: Handle virtual EL2 registers in vcpu_read/write_sys_reg() Date: Mon, 29 Nov 2021 20:00:59 +0000 Message-Id: <20211129200150.351436-19-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org KVM internally uses accessor functions when reading or writing the guest's system registers. This takes care of accessing either the stored copy or using the "live" EL1 system registers when the host uses VHE. With the introduction of virtual EL2 we add a bunch of EL2 system registers, which now must also be taken care of: - If the guest is running in vEL2, and we access an EL1 sysreg, we must revert to the stored version of that, and not use the CPU's copy. - If the guest is running in vEL1, and we access an EL2 sysreg, we must also use the stored version, since the CPU carries the EL1 copy. - Some EL2 system registers are supposed to affect the current execution of the system, so we need to put them into their respective EL1 counterparts. For this we need to define a mapping between the two. This is done using the newly introduced struct el2_sysreg_map. - Some EL2 system registers have a different format than their EL1 counterpart, so we need to translate them before writing them to the CPU. This is done using an (optional) translate function in the map. - There are the three special registers SP_EL2, SPSR_EL2 and ELR_EL2, which need some separate handling (SPSR_EL2 is being handled in a separate patch). All of these cases are now wrapped into the existing accessor functions, so KVM users wouldn't need to care whether they access EL2 or EL1 registers and also which state the guest is in. This handles what was formerly known as the "shadow state" dynamically, without requiring a separate copy for each vCPU EL. Co-developed-by: Andre Przywara Signed-off-by: Andre Przywara Signed-off-by: Marc Zyngier Reviewed-by: Ganapatrao Kulkarni Reviewed-by: Alexandru Elisei --- arch/arm64/kvm/sys_regs.c | 143 ++++++++++++++++++++++++++++++++++++-- 1 file changed, 139 insertions(+), 4 deletions(-) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 730a24468915..61596355e42d 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -24,6 +24,7 @@ #include #include #include +#include #include #include @@ -64,23 +65,157 @@ static bool write_to_read_only(struct kvm_vcpu *vcpu, return false; } +#define PURE_EL2_SYSREG(el2) \ + case el2: { \ + *el1r = el2; \ + return true; \ + } + +#define MAPPED_EL2_SYSREG(el2, el1, fn) \ + case el2: { \ + *xlate = fn; \ + *el1r = el1; \ + return true; \ + } + +static bool get_el2_mapping(unsigned int reg, + unsigned int *el1r, u64 (**xlate)(u64)) +{ + switch (reg) { + PURE_EL2_SYSREG( VPIDR_EL2 ); + PURE_EL2_SYSREG( VMPIDR_EL2 ); + PURE_EL2_SYSREG( ACTLR_EL2 ); + PURE_EL2_SYSREG( HCR_EL2 ); + PURE_EL2_SYSREG( MDCR_EL2 ); + PURE_EL2_SYSREG( HSTR_EL2 ); + PURE_EL2_SYSREG( HACR_EL2 ); + PURE_EL2_SYSREG( VTTBR_EL2 ); + PURE_EL2_SYSREG( VTCR_EL2 ); + PURE_EL2_SYSREG( RVBAR_EL2 ); + PURE_EL2_SYSREG( TPIDR_EL2 ); + PURE_EL2_SYSREG( HPFAR_EL2 ); + PURE_EL2_SYSREG( ELR_EL2 ); + PURE_EL2_SYSREG( SPSR_EL2 ); + MAPPED_EL2_SYSREG(SCTLR_EL2, SCTLR_EL1, + translate_sctlr_el2_to_sctlr_el1 ); + MAPPED_EL2_SYSREG(CPTR_EL2, CPACR_EL1, + translate_cptr_el2_to_cpacr_el1 ); + MAPPED_EL2_SYSREG(TTBR0_EL2, TTBR0_EL1, + translate_ttbr0_el2_to_ttbr0_el1 ); + MAPPED_EL2_SYSREG(TTBR1_EL2, TTBR1_EL1, NULL ); + MAPPED_EL2_SYSREG(TCR_EL2, TCR_EL1, + translate_tcr_el2_to_tcr_el1 ); + MAPPED_EL2_SYSREG(VBAR_EL2, VBAR_EL1, NULL ); + MAPPED_EL2_SYSREG(AFSR0_EL2, AFSR0_EL1, NULL ); + MAPPED_EL2_SYSREG(AFSR1_EL2, AFSR1_EL1, NULL ); + MAPPED_EL2_SYSREG(ESR_EL2, ESR_EL1, NULL ); + MAPPED_EL2_SYSREG(FAR_EL2, FAR_EL1, NULL ); + MAPPED_EL2_SYSREG(MAIR_EL2, MAIR_EL1, NULL ); + MAPPED_EL2_SYSREG(AMAIR_EL2, AMAIR_EL1, NULL ); + MAPPED_EL2_SYSREG(CNTHCTL_EL2, CNTKCTL_EL1, + translate_cnthctl_el2_to_cntkctl_el1 ); + default: + return false; + } +} + u64 vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, int reg) { u64 val = 0x8badf00d8badf00d; + u64 (*xlate)(u64) = NULL; + unsigned int el1r; + + if (!vcpu->arch.sysregs_loaded_on_cpu) + goto memory_read; + + if (unlikely(get_el2_mapping(reg, &el1r, &xlate))) { + if (!is_hyp_ctxt(vcpu)) + goto memory_read; + + /* + * ELR_EL2 is special cased for now. + */ + switch (reg) { + case ELR_EL2: + return read_sysreg_el1(SYS_ELR); + } + + /* + * If this register does not have an EL1 counterpart, + * then read the stored EL2 version. + */ + if (reg == el1r) + goto memory_read; + + /* + * If we have a non-VHE guest and that the sysreg + * requires translation to be used at EL1, use the + * in-memory copy instead. + */ + if (!vcpu_el2_e2h_is_set(vcpu) && xlate) + goto memory_read; + + /* Get the current version of the EL1 counterpart. */ + WARN_ON(!__vcpu_read_sys_reg_from_cpu(el1r, &val)); + return val; + } + + /* EL1 register can't be on the CPU if the guest is in vEL2. */ + if (unlikely(is_hyp_ctxt(vcpu))) + goto memory_read; - if (vcpu->arch.sysregs_loaded_on_cpu && - __vcpu_read_sys_reg_from_cpu(reg, &val)) + if (__vcpu_read_sys_reg_from_cpu(reg, &val)) return val; +memory_read: return __vcpu_sys_reg(vcpu, reg); } void vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg) { - if (vcpu->arch.sysregs_loaded_on_cpu && - __vcpu_write_sys_reg_to_cpu(val, reg)) + u64 (*xlate)(u64) = NULL; + unsigned int el1r; + + if (!vcpu->arch.sysregs_loaded_on_cpu) + goto memory_write; + + if (unlikely(get_el2_mapping(reg, &el1r, &xlate))) { + if (!is_hyp_ctxt(vcpu)) + goto memory_write; + + /* + * Always store a copy of the write to memory to avoid having + * to reverse-translate virtual EL2 system registers for a + * non-VHE guest hypervisor. + */ + __vcpu_sys_reg(vcpu, reg) = val; + + switch (reg) { + case ELR_EL2: + write_sysreg_el1(val, SYS_ELR); + return; + } + + /* No EL1 counterpart? We're done here.? */ + if (reg == el1r) + return; + + if (!vcpu_el2_e2h_is_set(vcpu) && xlate) + val = xlate(val); + + /* Redirect this to the EL1 version of the register. */ + WARN_ON(!__vcpu_write_sys_reg_to_cpu(val, el1r)); + return; + } + + /* EL1 register can't be on the CPU if the guest is in vEL2. */ + if (unlikely(is_hyp_ctxt(vcpu))) + goto memory_write; + + if (__vcpu_write_sys_reg_to_cpu(val, reg)) return; +memory_write: __vcpu_sys_reg(vcpu, reg) = val; } From patchwork Mon Nov 29 20:01:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645623 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9BD8CC433F5 for ; Mon, 29 Nov 2021 20:08:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233189AbhK2UL4 (ORCPT ); Mon, 29 Nov 2021 15:11:56 -0500 Received: from ams.source.kernel.org ([145.40.68.75]:42322 "EHLO ams.source.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235687AbhK2UJ4 (ORCPT ); Mon, 29 Nov 2021 15:09:56 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 562E2B815EB for ; Mon, 29 Nov 2021 20:06:37 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1D2EDC53FCF; Mon, 29 Nov 2021 20:06:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216396; bh=H3AaFkblo8PjcjwlndSlAD9qu3bXi9UTi+PNsCoShEk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=EadVzWt9vGMarSCDkpBlahNGeixUO+siTuCrFLHOUWmRpgKkQ9zq23Ga6cnJ6BPzx 6sY1g/L/fDaxGB/6NwCs6cn3lo9F7hRIjpYVxFdCsbEvvJpkSW3Lu2jTwuQPS+RkKz HJeI2FL2pFpdw4Qm0ReGLMHi5v2eG58XtHSb/MDWUKFfwQomivGHyNeEcVCb44POZp X4SYulSk/iPXVX0Dp8kL/PJpcooBC6LtTUN2jV0LIXLkiF3fTRp4232AMQ7k6soYne pluycdg/SekzgZDomuqmT3O4kNKgup9yTzKlJU9KXfsdelrOiHfFUgwlcryc/mBCwU lgOd2V9tUJRpQ== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmqt-008gvR-Bd; Mon, 29 Nov 2021 20:02:15 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 19/69] KVM: arm64: nv: Handle SPSR_EL2 specially Date: Mon, 29 Nov 2021 20:01:00 +0000 Message-Id: <20211129200150.351436-20-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org SPSR_EL2 needs special attention when running nested on ARMv8.3: If taking an exception while running at vEL2 (actually EL1), the HW will update the SPSR_EL1 register with the EL1 mode. We need to track this in order to make sure that accesses to the virtual view of SPSR_EL2 is correct. To do so, we place an illegal value in SPSR_EL1.M, and patch it accordingly if required when accessing it. Signed-off-by: Marc Zyngier Reviewed-by: Alexandru Elisei --- arch/arm64/include/asm/kvm_emulate.h | 37 ++++++++++++++++++++++++++++ arch/arm64/kvm/sys_regs.c | 23 +++++++++++++++-- 2 files changed, 58 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h index 46c849ba281e..2db11ff3fdad 100644 --- a/arch/arm64/include/asm/kvm_emulate.h +++ b/arch/arm64/include/asm/kvm_emulate.h @@ -241,6 +241,43 @@ static inline bool is_hyp_ctxt(const struct kvm_vcpu *vcpu) return __is_hyp_ctxt(&vcpu->arch.ctxt); } +static inline u64 __fixup_spsr_el2_write(struct kvm_cpu_context *ctxt, u64 val) +{ + if (!__vcpu_el2_e2h_is_set(ctxt)) { + /* + * Clear the .M field when writing SPSR to the CPU, so that we + * can detect when the CPU clobbered our SPSR copy during a + * local exception. + */ + val &= ~0xc; + } + + return val; +} + +static inline u64 __fixup_spsr_el2_read(const struct kvm_cpu_context *ctxt, u64 val) +{ + if (__vcpu_el2_e2h_is_set(ctxt)) + return val; + + /* + * SPSR.M == 0 means the CPU has not touched the SPSR, so the + * register has still the value we saved on the last write. + */ + if ((val & 0xc) == 0) + return ctxt_sys_reg(ctxt, SPSR_EL2); + + /* + * Otherwise there was a "local" exception on the CPU, + * which from the guest's point of view was being taken from + * EL2 to EL2, although it actually happened to be from + * EL1 to EL1. + * So we need to fix the .M field in SPSR, to make it look + * like EL2, which is what the guest would expect. + */ + return (val & ~0x0c) | CurrentEL_EL2; +} + /* * The layout of SPSR for an AArch32 state is different when observed from an * AArch64 SPSR_ELx or an AArch32 SPSR_*. This function generates the AArch32 diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 61596355e42d..875040bcfbe1 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -133,11 +133,14 @@ u64 vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, int reg) goto memory_read; /* - * ELR_EL2 is special cased for now. + * ELR_EL2 and SPSR_EL2 are special cased for now. */ switch (reg) { case ELR_EL2: return read_sysreg_el1(SYS_ELR); + case SPSR_EL2: + val = read_sysreg_el1(SYS_SPSR); + return __fixup_spsr_el2_read(&vcpu->arch.ctxt, val); } /* @@ -194,6 +197,10 @@ void vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg) case ELR_EL2: write_sysreg_el1(val, SYS_ELR); return; + case SPSR_EL2: + val = __fixup_spsr_el2_write(&vcpu->arch.ctxt, val); + write_sysreg_el1(val, SYS_SPSR); + return; } /* No EL1 counterpart? We're done here.? */ @@ -1620,6 +1627,18 @@ static bool access_sp_el1(struct kvm_vcpu *vcpu, return true; } +static bool access_spsr_el2(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + if (p->is_write) + vcpu_write_sys_reg(vcpu, p->regval, SPSR_EL2); + else + p->regval = vcpu_read_sys_reg(vcpu, SPSR_EL2); + + return true; +} + /* * Architected system registers. * Important: Must be sorted ascending by Op0, Op1, CRn, CRm, Op2 @@ -2052,7 +2071,7 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_VTCR_EL2), access_rw, reset_val, VTCR_EL2, 0 }, { SYS_DESC(SYS_DACR32_EL2), NULL, reset_unknown, DACR32_EL2 }, - { SYS_DESC(SYS_SPSR_EL2), access_rw, reset_val, SPSR_EL2, 0 }, + { SYS_DESC(SYS_SPSR_EL2), access_spsr_el2, reset_val, SPSR_EL2, 0 }, { SYS_DESC(SYS_ELR_EL2), access_rw, reset_val, ELR_EL2, 0 }, { SYS_DESC(SYS_SP_EL1), access_sp_el1}, From patchwork Mon Nov 29 20:01:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645779 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 22B9AC433F5 for ; Mon, 29 Nov 2021 21:46:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232339AbhK2Vty (ORCPT ); Mon, 29 Nov 2021 16:49:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49892 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232458AbhK2Vrw (ORCPT ); Mon, 29 Nov 2021 16:47:52 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4BCFBC091D2C for ; Mon, 29 Nov 2021 12:07:10 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id D06BCCE1410 for ; Mon, 29 Nov 2021 20:07:09 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4BA8EC53FD0; Mon, 29 Nov 2021 20:07:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216428; bh=KL8kMrCOfU3Bi8GZMSXLpGFz3bOYBmmc7KK3lpfO310=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DRJmlyqaOSLX0sOWghGpFawVs8A6Zf/5sttywwz+Lmx8SuMRCS6Xx4KpZiY40QL/N CD91mctsCItLeSZ/S5Uy997IHLx+mFI11gSzwcMnV65UY/eUNzOHuZJB7cEv3mQuVy gbtG+uzR01rbmPuMf7g/wpzDnvJHhHLt5KCFtoTk8DhvNC5brgu2v7dBPX2/VyFB4+ IP9/D1iXXy2QH1L39TCGYQ52ihigD9IVbtxBZLBb3okVFZgvBifkcEZCPguKk+gMhY gLzOqCRe8i/7ufXpNyN4QaRH2d0XxBnmG7KKyLN5kK69vsLwA8QNr/jGNTtcL67Ja+ wYRWsTW1YFbcA== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmqt-008gvR-MB; Mon, 29 Nov 2021 20:02:15 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 20/69] KVM: arm64: nv: Handle HCR_EL2.E2H specially Date: Mon, 29 Nov 2021 20:01:01 +0000 Message-Id: <20211129200150.351436-21-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org HCR_EL2.E2H is nasty, as a flip of this bit completely changes the way we deal with a lot of the state. So when the guest flips this bit (sysregs are live), do the put/load dance so that we have a consistent state. Yes, this is slow. Don't do it. Suggested-by: Alexandru Elisei Signed-off-by: Marc Zyngier --- arch/arm64/kvm/sys_regs.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 875040bcfbe1..c1408dff58fa 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -183,9 +183,24 @@ void vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg) goto memory_write; if (unlikely(get_el2_mapping(reg, &el1r, &xlate))) { + bool need_put_load; + if (!is_hyp_ctxt(vcpu)) goto memory_write; + /* + * HCR_EL2.E2H is nasty: it changes the way we interpret a + * lot of the EL2 state, so treat is as a full state + * transition. + */ + need_put_load = ((reg == HCR_EL2) && + vcpu_el2_e2h_is_set(vcpu) != !!(val & HCR_E2H)); + + if (need_put_load) { + preempt_disable(); + kvm_arch_vcpu_put(vcpu); + } + /* * Always store a copy of the write to memory to avoid having * to reverse-translate virtual EL2 system registers for a @@ -193,6 +208,11 @@ void vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg) */ __vcpu_sys_reg(vcpu, reg) = val; + if (need_put_load) { + kvm_arch_vcpu_load(vcpu, smp_processor_id()); + preempt_enable(); + } + switch (reg) { case ELR_EL2: write_sysreg_el1(val, SYS_ELR); From patchwork Mon Nov 29 20:01:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645777 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 195ECC433FE for ; Mon, 29 Nov 2021 21:46:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232328AbhK2Vtw (ORCPT ); Mon, 29 Nov 2021 16:49:52 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48992 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229748AbhK2Vru (ORCPT ); Mon, 29 Nov 2021 16:47:50 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C9396C091D25 for ; Mon, 29 Nov 2021 12:07:00 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 50D4DCE140F for ; Mon, 29 Nov 2021 20:07:00 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7E722C53FCD; Mon, 29 Nov 2021 20:06:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216418; bh=bwwaVcJv+N2+5b3SLI43azsUDNwvtFJhii6e+BfZVrQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=NTB5WfMrfDzPYiGAsFtXVySZB/zTx0MK6oT0HMR7opsoa76P5ojffQXSljwTMHG3g XwZUr96Lw9CZa4Vc1K7MP2AbjqMPEujTL2V929ZaX2vKQWocQ2q7Y7fPoxfYgGH4fP hN09aOBioqizEF5z8O0f+lCgI2fEImcAWlShs2/D99d9TG6qJMQoTXsnP49A6Cd2tq fJZ0r1HvsekdvUEF7lHIpCrmeVTzuq24Qo0Xsr2P2wDVord5X+Y2MNyISENs3sibUn UXxU11SCxqAGlBlrFPNihIDy0oGmBPoivR24YshAmYXoNnUZc6Rkfltny/VDfayb6/ UwUxOeuyvGBvQ== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmqt-008gvR-W8; Mon, 29 Nov 2021 20:02:16 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 21/69] KVM: arm64: nv: Save/Restore vEL2 sysregs Date: Mon, 29 Nov 2021 20:01:02 +0000 Message-Id: <20211129200150.351436-22-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Whenever we need to restore the guest's system registers to the CPU, we now need to take care of the EL2 system registers as well. Most of them are accessed via traps only, but some have an immediate effect and also a guest running in VHE mode would expect them to be accessible via their EL1 encoding, which we do not trap. For vEL2 we write the virtual EL2 registers with an identical format directly into their EL1 counterpart, and translate the few registers that have a different format for the same effect on the execution when running a non-VHE guest guest hypervisor. Based on an initial patch from Andre Przywara, rewritten many times since. Signed-off-by: Marc Zyngier --- arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h | 5 +- arch/arm64/kvm/hyp/nvhe/sysreg-sr.c | 2 +- arch/arm64/kvm/hyp/vhe/sysreg-sr.c | 125 ++++++++++++++++++++- 3 files changed, 127 insertions(+), 5 deletions(-) diff --git a/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h b/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h index 7ecca8b07851..283f780f5f56 100644 --- a/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h +++ b/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h @@ -92,9 +92,10 @@ static inline void __sysreg_restore_user_state(struct kvm_cpu_context *ctxt) write_sysreg(ctxt_sys_reg(ctxt, TPIDRRO_EL0), tpidrro_el0); } -static inline void __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt) +static inline void __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt, + u64 mpidr) { - write_sysreg(ctxt_sys_reg(ctxt, MPIDR_EL1), vmpidr_el2); + write_sysreg(mpidr, vmpidr_el2); write_sysreg(ctxt_sys_reg(ctxt, CSSELR_EL1), csselr_el1); if (has_vhe() || diff --git a/arch/arm64/kvm/hyp/nvhe/sysreg-sr.c b/arch/arm64/kvm/hyp/nvhe/sysreg-sr.c index 29305022bc04..dba101565de3 100644 --- a/arch/arm64/kvm/hyp/nvhe/sysreg-sr.c +++ b/arch/arm64/kvm/hyp/nvhe/sysreg-sr.c @@ -28,7 +28,7 @@ void __sysreg_save_state_nvhe(struct kvm_cpu_context *ctxt) void __sysreg_restore_state_nvhe(struct kvm_cpu_context *ctxt) { - __sysreg_restore_el1_state(ctxt); + __sysreg_restore_el1_state(ctxt, ctxt_sys_reg(ctxt, MPIDR_EL1)); __sysreg_restore_common_state(ctxt); __sysreg_restore_user_state(ctxt); __sysreg_restore_el2_return_state(ctxt); diff --git a/arch/arm64/kvm/hyp/vhe/sysreg-sr.c b/arch/arm64/kvm/hyp/vhe/sysreg-sr.c index 007a12dd4351..96eed1d5634e 100644 --- a/arch/arm64/kvm/hyp/vhe/sysreg-sr.c +++ b/arch/arm64/kvm/hyp/vhe/sysreg-sr.c @@ -13,6 +13,96 @@ #include #include #include +#include + +static void __sysreg_save_vel2_state(struct kvm_cpu_context *ctxt) +{ + /* These registers are common with EL1 */ + ctxt_sys_reg(ctxt, CSSELR_EL1) = read_sysreg(csselr_el1); + ctxt_sys_reg(ctxt, PAR_EL1) = read_sysreg(par_el1); + ctxt_sys_reg(ctxt, TPIDR_EL1) = read_sysreg(tpidr_el1); + + ctxt_sys_reg(ctxt, ESR_EL2) = read_sysreg_el1(SYS_ESR); + ctxt_sys_reg(ctxt, AFSR0_EL2) = read_sysreg_el1(SYS_AFSR0); + ctxt_sys_reg(ctxt, AFSR1_EL2) = read_sysreg_el1(SYS_AFSR1); + ctxt_sys_reg(ctxt, FAR_EL2) = read_sysreg_el1(SYS_FAR); + ctxt_sys_reg(ctxt, MAIR_EL2) = read_sysreg_el1(SYS_MAIR); + ctxt_sys_reg(ctxt, VBAR_EL2) = read_sysreg_el1(SYS_VBAR); + ctxt_sys_reg(ctxt, CONTEXTIDR_EL2) = read_sysreg_el1(SYS_CONTEXTIDR); + ctxt_sys_reg(ctxt, AMAIR_EL2) = read_sysreg_el1(SYS_AMAIR); + + /* + * In VHE mode those registers are compatible between EL1 and EL2, + * and the guest uses the _EL1 versions on the CPU naturally. + * So we save them into their _EL2 versions here. + * For nVHE mode we trap accesses to those registers, so our + * _EL2 copy in sys_regs[] is always up-to-date and we don't need + * to save anything here. + */ + if (__vcpu_el2_e2h_is_set(ctxt)) { + ctxt_sys_reg(ctxt, SCTLR_EL2) = read_sysreg_el1(SYS_SCTLR); + ctxt_sys_reg(ctxt, CPTR_EL2) = read_sysreg_el1(SYS_CPACR); + ctxt_sys_reg(ctxt, TTBR0_EL2) = read_sysreg_el1(SYS_TTBR0); + ctxt_sys_reg(ctxt, TTBR1_EL2) = read_sysreg_el1(SYS_TTBR1); + ctxt_sys_reg(ctxt, TCR_EL2) = read_sysreg_el1(SYS_TCR); + ctxt_sys_reg(ctxt, CNTHCTL_EL2) = read_sysreg_el1(SYS_CNTKCTL); + } + + ctxt_sys_reg(ctxt, SP_EL2) = read_sysreg(sp_el1); + ctxt_sys_reg(ctxt, ELR_EL2) = read_sysreg_el1(SYS_ELR); + ctxt_sys_reg(ctxt, SPSR_EL2) = __fixup_spsr_el2_read(ctxt, read_sysreg_el1(SYS_SPSR)); +} + +static void __sysreg_restore_vel2_state(struct kvm_cpu_context *ctxt) +{ + u64 val; + + /* These registers are common with EL1 */ + write_sysreg(ctxt_sys_reg(ctxt, CSSELR_EL1), csselr_el1); + write_sysreg(ctxt_sys_reg(ctxt, PAR_EL1), par_el1); + write_sysreg(ctxt_sys_reg(ctxt, TPIDR_EL1), tpidr_el1); + + write_sysreg(read_cpuid_id(), vpidr_el2); + write_sysreg(ctxt_sys_reg(ctxt, MPIDR_EL1), vmpidr_el2); + write_sysreg_el1(ctxt_sys_reg(ctxt, MAIR_EL2), SYS_MAIR); + write_sysreg_el1(ctxt_sys_reg(ctxt, VBAR_EL2), SYS_VBAR); + write_sysreg_el1(ctxt_sys_reg(ctxt, CONTEXTIDR_EL2),SYS_CONTEXTIDR); + write_sysreg_el1(ctxt_sys_reg(ctxt, AMAIR_EL2), SYS_AMAIR); + + if (__vcpu_el2_e2h_is_set(ctxt)) { + /* + * In VHE mode those registers are compatible between + * EL1 and EL2. + */ + write_sysreg_el1(ctxt_sys_reg(ctxt, SCTLR_EL2), SYS_SCTLR); + write_sysreg_el1(ctxt_sys_reg(ctxt, CPTR_EL2), SYS_CPACR); + write_sysreg_el1(ctxt_sys_reg(ctxt, TTBR0_EL2), SYS_TTBR0); + write_sysreg_el1(ctxt_sys_reg(ctxt, TTBR1_EL2), SYS_TTBR1); + write_sysreg_el1(ctxt_sys_reg(ctxt, TCR_EL2), SYS_TCR); + write_sysreg_el1(ctxt_sys_reg(ctxt, CNTHCTL_EL2), SYS_CNTKCTL); + } else { + val = translate_sctlr_el2_to_sctlr_el1(ctxt_sys_reg(ctxt, SCTLR_EL2)); + write_sysreg_el1(val, SYS_SCTLR); + val = translate_cptr_el2_to_cpacr_el1(ctxt_sys_reg(ctxt, CPTR_EL2)); + write_sysreg_el1(val, SYS_CPACR); + val = translate_ttbr0_el2_to_ttbr0_el1(ctxt_sys_reg(ctxt, TTBR0_EL2)); + write_sysreg_el1(val, SYS_TTBR0); + val = translate_tcr_el2_to_tcr_el1(ctxt_sys_reg(ctxt, TCR_EL2)); + write_sysreg_el1(val, SYS_TCR); + val = translate_cnthctl_el2_to_cntkctl_el1(ctxt_sys_reg(ctxt, CNTHCTL_EL2)); + write_sysreg_el1(val, SYS_CNTKCTL); + } + + write_sysreg_el1(ctxt_sys_reg(ctxt, ESR_EL2), SYS_ESR); + write_sysreg_el1(ctxt_sys_reg(ctxt, AFSR0_EL2), SYS_AFSR0); + write_sysreg_el1(ctxt_sys_reg(ctxt, AFSR1_EL2), SYS_AFSR1); + write_sysreg_el1(ctxt_sys_reg(ctxt, FAR_EL2), SYS_FAR); + write_sysreg(ctxt_sys_reg(ctxt, SP_EL2), sp_el1); + write_sysreg_el1(ctxt_sys_reg(ctxt, ELR_EL2), SYS_ELR); + + val = __fixup_spsr_el2_write(ctxt, ctxt_sys_reg(ctxt, SPSR_EL2)); + write_sysreg_el1(val, SYS_SPSR); +} /* * VHE: Host and guest must save mdscr_el1 and sp_el0 (and the PC and @@ -65,6 +155,7 @@ void kvm_vcpu_load_sysregs_vhe(struct kvm_vcpu *vcpu) { struct kvm_cpu_context *guest_ctxt = &vcpu->arch.ctxt; struct kvm_cpu_context *host_ctxt; + u64 mpidr; host_ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt; __sysreg_save_user_state(host_ctxt); @@ -77,7 +168,29 @@ void kvm_vcpu_load_sysregs_vhe(struct kvm_vcpu *vcpu) */ __sysreg32_restore_state(vcpu); __sysreg_restore_user_state(guest_ctxt); - __sysreg_restore_el1_state(guest_ctxt); + + if (unlikely(__is_hyp_ctxt(guest_ctxt))) { + __sysreg_restore_vel2_state(guest_ctxt); + } else { + if (nested_virt_in_use(vcpu)) { + /* + * Only set VPIDR_EL2 for nested VMs, as this is the + * only time it changes. We'll restore the MIDR_EL1 + * view on put. + */ + write_sysreg(ctxt_sys_reg(guest_ctxt, VPIDR_EL2), vpidr_el2); + + /* + * As we're restoring a nested guest, set the value + * provided by the guest hypervisor. + */ + mpidr = ctxt_sys_reg(guest_ctxt, VMPIDR_EL2); + } else { + mpidr = ctxt_sys_reg(guest_ctxt, MPIDR_EL1); + } + + __sysreg_restore_el1_state(guest_ctxt, mpidr); + } vcpu->arch.sysregs_loaded_on_cpu = true; @@ -103,12 +216,20 @@ void kvm_vcpu_put_sysregs_vhe(struct kvm_vcpu *vcpu) host_ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt; deactivate_traps_vhe_put(vcpu); - __sysreg_save_el1_state(guest_ctxt); + if (unlikely(__is_hyp_ctxt(guest_ctxt))) + __sysreg_save_vel2_state(guest_ctxt); + else + __sysreg_save_el1_state(guest_ctxt); + __sysreg_save_user_state(guest_ctxt); __sysreg32_save_state(vcpu); /* Restore host user state */ __sysreg_restore_user_state(host_ctxt); + /* If leaving a nesting guest, restore MPIDR_EL1 default view */ + if (nested_virt_in_use(vcpu)) + write_sysreg(read_cpuid_id(), vpidr_el2); + vcpu->arch.sysregs_loaded_on_cpu = false; } From patchwork Mon Nov 29 20:01:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645625 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9DD9FC433EF for ; Mon, 29 Nov 2021 20:08:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233130AbhK2ULw (ORCPT ); Mon, 29 Nov 2021 15:11:52 -0500 Received: from sin.source.kernel.org ([145.40.73.55]:50696 "EHLO sin.source.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230220AbhK2UJs (ORCPT ); Mon, 29 Nov 2021 15:09:48 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 09FC1CE13F9 for ; Mon, 29 Nov 2021 20:06:29 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 576C4C53FAD; Mon, 29 Nov 2021 20:06:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216387; bh=r0zuLK3lw1N9AgFVnRFi6s6vtDbl4C1MSE/bfxIwS0E=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=gn/YQhouLH6ioc0jJYvye40Yq8u7cd/dCcA5akegBakhVDo69D0S/i5F11tB71wta 4TdYNnquMEbQgQlYnrdH2Y8J9WV2qJQ2IIShfju4T/2ztzG44h6okIOgmeHmHhKw4H VKqJOvQeaS9Dyuoot0EFOIhb6+dVywmcUOkuKQ3+e223A+Kpbt23768pCTeHSTxcgH N61cefzLhB7gZNe5kqsrcDi6j8SBWNT6C9R5zRlxBGBvdo/kV4U5VVoK3yjYz8SkHs 2DXdsMmWKYjlPIhgdHoUeNZwF3tL2HzDegCjEGcKxbqsMQQpjlCdqyrAc4Ur3JLhQs rFIqae8O7eVbw== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmqu-008gvR-A5; Mon, 29 Nov 2021 20:02:16 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 22/69] KVM: arm64: nv: Emulate PSTATE.M for a guest hypervisor Date: Mon, 29 Nov 2021 20:01:03 +0000 Message-Id: <20211129200150.351436-23-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Christoffer Dall We can no longer blindly copy the VCPU's PSTATE into SPSR_EL2 and return to the guest and vice versa when taking an exception to the hypervisor, because we emulate virtual EL2 in EL1 and therefore have to translate the mode field from EL2 to EL1 and vice versa. This requires keeping track of the state we enter the guest, for which we transiently use a dedicated flag. Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_host.h | 1 + arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h | 19 ++++++++++++++++- arch/arm64/kvm/hyp/vhe/switch.c | 24 ++++++++++++++++++++++ 3 files changed, 43 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 7b6fe18ee450..6a7b13edc5cb 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -456,6 +456,7 @@ struct kvm_vcpu_arch { #define KVM_ARM64_EXCEPT_MASK (7 << 9) /* Target EL/MODE */ #define KVM_ARM64_DEBUG_STATE_SAVE_SPE (1 << 12) /* Save SPE context if active */ #define KVM_ARM64_DEBUG_STATE_SAVE_TRBE (1 << 13) /* Save TRBE context if active */ +#define KVM_ARM64_IN_HYP_CONTEXT (1 << 14) /* Guest running in HYP context */ #define KVM_GUESTDBG_VALID_MASK (KVM_GUESTDBG_ENABLE | \ KVM_GUESTDBG_USE_SW_BP | \ diff --git a/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h b/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h index 283f780f5f56..e3689c6ce4cc 100644 --- a/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h +++ b/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h @@ -157,9 +157,26 @@ static inline void __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt, write_sysreg_el1(ctxt_sys_reg(ctxt, SPSR_EL1), SYS_SPSR); } +/* Read the VCPU state's PSTATE, but translate (v)EL2 to EL1. */ +static inline u64 to_hw_pstate(const struct kvm_cpu_context *ctxt) +{ + u64 mode = ctxt->regs.pstate & (PSR_MODE_MASK | PSR_MODE32_BIT); + + switch (mode) { + case PSR_MODE_EL2t: + mode = PSR_MODE_EL1t; + break; + case PSR_MODE_EL2h: + mode = PSR_MODE_EL1h; + break; + } + + return (ctxt->regs.pstate & ~(PSR_MODE_MASK | PSR_MODE32_BIT)) | mode; +} + static inline void __sysreg_restore_el2_return_state(struct kvm_cpu_context *ctxt) { - u64 pstate = ctxt->regs.pstate; + u64 pstate = to_hw_pstate(ctxt); u64 mode = pstate & PSR_AA32_MODE_MASK; /* diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c index fbb26b93c347..9d5572ceb001 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -114,6 +114,25 @@ static const exit_handler_fn *kvm_get_exit_handler_array(struct kvm_vcpu *vcpu) static void early_exit_filter(struct kvm_vcpu *vcpu, u64 *exit_code) { + /* + * If we were in HYP context on entry, adjust the PSTATE view + * so that the usual helpers work correctly. + */ + if (unlikely(vcpu->arch.flags & KVM_ARM64_IN_HYP_CONTEXT)) { + u64 mode = *vcpu_cpsr(vcpu) & (PSR_MODE_MASK | PSR_MODE32_BIT); + + switch (mode) { + case PSR_MODE_EL1t: + mode = PSR_MODE_EL2t; + break; + case PSR_MODE_EL1h: + mode = PSR_MODE_EL2h; + break; + } + + *vcpu_cpsr(vcpu) &= ~(PSR_MODE_MASK | PSR_MODE32_BIT); + *vcpu_cpsr(vcpu) |= mode; + } } /* Switch to the guest for VHE systems running in EL2 */ @@ -148,6 +167,11 @@ static int __kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu) sysreg_restore_guest_state_vhe(guest_ctxt); __debug_switch_to_guest(vcpu); + if (is_hyp_ctxt(vcpu)) + vcpu->arch.flags |= KVM_ARM64_IN_HYP_CONTEXT; + else + vcpu->arch.flags &= ~KVM_ARM64_IN_HYP_CONTEXT; + do { /* Jump in the fire! */ exit_code = __guest_enter(vcpu); From patchwork Mon Nov 29 20:01:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645735 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 17E47C433FE for ; Mon, 29 Nov 2021 21:42:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232354AbhK2Vp5 (ORCPT ); Mon, 29 Nov 2021 16:45:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48120 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232158AbhK2Vnw (ORCPT ); Mon, 29 Nov 2021 16:43:52 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 56397C091D11 for ; Mon, 29 Nov 2021 12:05:36 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 1E0EAB8162E for ; Mon, 29 Nov 2021 20:05:35 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C8DB2C53FC7; Mon, 29 Nov 2021 20:05:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216333; bh=yFzHvZHmLWVQ5Alroj3R7LTAUDXudcyHYsHt6+CZyUo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=lCP49rAxfLBrtts+Cylj0RyWq0HatgNVC5Hh12QS8tk3y3uRDhwVThjGtBst0ZKYF 1Vsilg9n0EQmZHsK10/OVXgfVGYUL3QqdTm6uwt1R1orBinPqacJAHZPpCAXyK1y6f Exn3iIRBrXUxDqxliXqLUr77wTZSEGGzzIhZpRoFyUJxW1LOOB3Z9dQ9d7DseyDlnq RIVltBKjxaQt4EGdym83x5Z9pxwIXmqokF93yM3/lv/azPzh2nmxe2a7Vou7R0NJ4f uwnp/hIye0t1Ys2HSF3x43hSf3ZtNjNENwoY7EdT4oSaSngvHJGh5WvEfLqJTJfT8y Q+d7dnArbdBBA== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmqu-008gvR-Lr; Mon, 29 Nov 2021 20:02:16 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 23/69] KVM: arm64: nv: Trap EL1 VM register accesses in virtual EL2 Date: Mon, 29 Nov 2021 20:01:04 +0000 Message-Id: <20211129200150.351436-24-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Christoffer Dall When running in virtual EL2 mode, we actually run the hardware in EL1 and therefore have to use the EL1 registers to ensure correct operation. By setting the HCR.TVM and HCR.TVRM we ensure that the virtual EL2 mode doesn't shoot itself in the foot when setting up what it believes to be a different mode's system register state (for example when preparing to switch to a VM). We can leverage the existing sysregs infrastructure to support trapped accesses to these registers. Signed-off-by: Christoffer Dall Signed-off-by: Marc Zyngier --- arch/arm64/kvm/hyp/include/hyp/switch.h | 4 +--- arch/arm64/kvm/hyp/nvhe/switch.c | 2 +- arch/arm64/kvm/hyp/vhe/switch.c | 7 ++++++- arch/arm64/kvm/sys_regs.c | 19 ++++++++++++++++--- 4 files changed, 24 insertions(+), 8 deletions(-) diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h index 96c5f3fb7838..0987abd4d57d 100644 --- a/arch/arm64/kvm/hyp/include/hyp/switch.h +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h @@ -111,10 +111,8 @@ static inline void __deactivate_traps_common(struct kvm_vcpu *vcpu) write_sysreg(0, pmuserenr_el0); } -static inline void ___activate_traps(struct kvm_vcpu *vcpu) +static inline void ___activate_traps(struct kvm_vcpu *vcpu, u64 hcr) { - u64 hcr = vcpu->arch.hcr_el2; - if (cpus_have_final_cap(ARM64_WORKAROUND_CAVIUM_TX2_219_TVM)) hcr |= HCR_TVM; diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/switch.c index d13115a12434..b005eabcaddc 100644 --- a/arch/arm64/kvm/hyp/nvhe/switch.c +++ b/arch/arm64/kvm/hyp/nvhe/switch.c @@ -39,7 +39,7 @@ static void __activate_traps(struct kvm_vcpu *vcpu) { u64 val; - ___activate_traps(vcpu); + ___activate_traps(vcpu, vcpu->arch.hcr_el2); __activate_traps_common(vcpu); val = vcpu->arch.cptr_el2; diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c index 9d5572ceb001..87a68496073a 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -33,9 +33,14 @@ DEFINE_PER_CPU(unsigned long, kvm_hyp_vector); static void __activate_traps(struct kvm_vcpu *vcpu) { + u64 hcr = vcpu->arch.hcr_el2; u64 val; - ___activate_traps(vcpu); + /* Trap VM sysreg accesses if an EL2 guest is not using VHE. */ + if (vcpu_mode_el2(vcpu) && !vcpu_el2_e2h_is_set(vcpu)) + hcr |= HCR_TVM | HCR_TRVM; + + ___activate_traps(vcpu, hcr); val = read_sysreg(cpacr_el1); val |= CPACR_EL1_TTA; diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index c1408dff58fa..19f7e2260462 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -350,8 +350,15 @@ static void get_access_mask(const struct sys_reg_desc *r, u64 *mask, u64 *shift) /* * Generic accessor for VM registers. Only called as long as HCR_TVM - * is set. If the guest enables the MMU, we stop trapping the VM - * sys_regs and leave it in complete control of the caches. + * is set. + * + * This is set in two cases: either (1) we're running at vEL2, or (2) + * we're running at EL1 and the guest has its MMU off. + * + * (1) TVM/TRVM is set, as we need to virtualise some of the VM + * registers for the guest hypervisor + * (2) Once the guest enables the MMU, we stop trapping the VM sys_regs + * and leave it in complete control of the caches. */ static bool access_vm_reg(struct kvm_vcpu *vcpu, struct sys_reg_params *p, @@ -360,7 +367,13 @@ static bool access_vm_reg(struct kvm_vcpu *vcpu, bool was_enabled = vcpu_has_cache_enabled(vcpu); u64 val, mask, shift; - BUG_ON(!p->is_write); + /* We don't expect TRVM on the host */ + BUG_ON(!vcpu_mode_el2(vcpu) && !p->is_write); + + if (!p->is_write) { + p->regval = vcpu_read_sys_reg(vcpu, r->reg); + return true; + } get_access_mask(r, &mask, &shift); From patchwork Mon Nov 29 20:01:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645737 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3E231C433F5 for ; Mon, 29 Nov 2021 21:42:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230291AbhK2Vp7 (ORCPT ); Mon, 29 Nov 2021 16:45:59 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48590 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230287AbhK2Vn5 (ORCPT ); Mon, 29 Nov 2021 16:43:57 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B8472C08EB2F for ; Mon, 29 Nov 2021 12:05:47 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 11299CE13D8 for ; Mon, 29 Nov 2021 20:05:46 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 43B18C53FAD; Mon, 29 Nov 2021 20:05:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216344; bh=ofFIfw0lWsxOgQT8syN+9sJTRkuGAun/IGzUODLKan0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Zhimbe8kGKANj8iqD7hkUhs+E8KBftqpLay+20dLKxmRWfmiGPUtPu/OPv691wTEo f3Tm/SY4vWVHoMj3ZM/EulotTIT5QPlfRZv4meAdz3MgUiM7QarIZp4EGzM/BQ9hyN vF2I7ozBoc7PQANobNY32LyHfYhlr+YtDdszVtw9Lfd9JK61MO1mwpCjRHk8Vx6T1n LID1ESW5Rf4cUbngybaEJ+XsAEwJlziy8yRzsrOqwYe9MnNY5M660PlKRcV4Otl9d8 jmnD0KgG/x3dg9LJGK5MqV+sLGRcN0keCLf6v9vbhBjP+Gbm6CARkXCAMCc9CbkkFI HUUeAi6DjNEWA== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmqu-008gvR-VZ; Mon, 29 Nov 2021 20:02:17 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 24/69] KVM: arm64: nv: Trap SPSR_EL1, ELR_EL1 and VBAR_EL1 from virtual EL2 Date: Mon, 29 Nov 2021 20:01:05 +0000 Message-Id: <20211129200150.351436-25-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Jintack Lim For the same reason we trap virtual memory register accesses at virtual EL2, we need to trap SPSR_EL1, ELR_EL1 and VBAR_EL1 accesses. ARM v8.3 introduces the HCR_EL2.NV1 bit to be able to trap on those register accesses in EL1. Do not set this bit until the whole nesting support is completed. Signed-off-by: Jintack Lim Signed-off-by: Marc Zyngier --- arch/arm64/kvm/sys_regs.c | 29 ++++++++++++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 19f7e2260462..5960178b7d7c 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -1660,6 +1660,30 @@ static bool access_sp_el1(struct kvm_vcpu *vcpu, return true; } +static bool access_elr(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + if (p->is_write) + vcpu_write_sys_reg(vcpu, p->regval, ELR_EL1); + else + p->regval = vcpu_read_sys_reg(vcpu, ELR_EL1); + + return true; +} + +static bool access_spsr(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + if (p->is_write) + __vcpu_sys_reg(vcpu, SPSR_EL1) = p->regval; + else + p->regval = __vcpu_sys_reg(vcpu, SPSR_EL1); + + return true; +} + static bool access_spsr_el2(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) @@ -1822,6 +1846,9 @@ static const struct sys_reg_desc sys_reg_descs[] = { PTRAUTH_KEY(APDB), PTRAUTH_KEY(APGA), + { SYS_DESC(SYS_SPSR_EL1), access_spsr}, + { SYS_DESC(SYS_ELR_EL1), access_elr}, + { SYS_DESC(SYS_AFSR0_EL1), access_vm_reg, reset_unknown, AFSR0_EL1 }, { SYS_DESC(SYS_AFSR1_EL1), access_vm_reg, reset_unknown, AFSR1_EL1 }, { SYS_DESC(SYS_ESR_EL1), access_vm_reg, reset_unknown, ESR_EL1 }, @@ -1869,7 +1896,7 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_LORC_EL1), trap_loregion }, { SYS_DESC(SYS_LORID_EL1), trap_loregion }, - { SYS_DESC(SYS_VBAR_EL1), NULL, reset_val, VBAR_EL1, 0 }, + { SYS_DESC(SYS_VBAR_EL1), access_rw, reset_val, VBAR_EL1, 0 }, { SYS_DESC(SYS_DISR_EL1), NULL, reset_val, DISR_EL1, 0 }, { SYS_DESC(SYS_ICC_IAR0_EL1), write_to_read_only }, From patchwork Mon Nov 29 20:01:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645613 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70972C4332F for ; Mon, 29 Nov 2021 20:07:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232335AbhK2ULA (ORCPT ); Mon, 29 Nov 2021 15:11:00 -0500 Received: from ams.source.kernel.org ([145.40.68.75]:41736 "EHLO ams.source.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1379639AbhK2UI7 (ORCPT ); Mon, 29 Nov 2021 15:08:59 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 5B593B815DA for ; Mon, 29 Nov 2021 20:05:40 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 13972C53FAD; Mon, 29 Nov 2021 20:05:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216339; bh=ZEGb6k7HLw8peHKuI5UTm57bULuTvzQmr9EREQlX0W0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=U6ZrIbvP73EsBkq5/lU7R66KHKlqNoTJRwGXX5HlXsFyoKW3USol5HuaoSHq97oM8 QaPaMXJj50sg6T0PT1PzDtVleazI5OF5NTyNfSaM1w/3LuDf0Tf9WIQ5Pwwko4wGYH N2LKi+DVVfEoYDQlv4pSpj2HtB4VxdGAvWL6oQbS8VWa1dgpCiYQGmcmts+mVWJVhF d7ApzJMTXsbYQWLMPJQUTs7P2s7LFwlbQTR13T5Q48Ir5KUfw27Zd2ByU1wMsA1RUc Nf9aGRfMHseLyNRfpZo7cCplQP/lDSAKXg70s0pkX7M5AtWmrVXmyjVFuXeItzKevQ pMSz9LqFIRdeA== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmqv-008gvR-Ap; Mon, 29 Nov 2021 20:02:17 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 25/69] KVM: arm64: nv: Trap CPACR_EL1 access in virtual EL2 Date: Mon, 29 Nov 2021 20:01:06 +0000 Message-Id: <20211129200150.351436-26-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Jintack Lim For the same reason we trap virtual memory register accesses in virtual EL2, we trap CPACR_EL1 access too; We allow the virtual EL2 mode to access EL1 system register state instead of the virtual EL2 one. Signed-off-by: Jintack Lim Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_arm.h | 2 +- arch/arm64/kvm/hyp/vhe/switch.c | 3 +++ arch/arm64/kvm/sys_regs.c | 2 +- 3 files changed, 5 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h index 0a0ee998ec5a..2eccf883e8fe 100644 --- a/arch/arm64/include/asm/kvm_arm.h +++ b/arch/arm64/include/asm/kvm_arm.h @@ -276,7 +276,7 @@ #define CPTR_EL2_TFP_SHIFT 10 /* Hyp Coprocessor Trap Register */ -#define CPTR_EL2_TCPAC (1 << 31) +#define CPTR_EL2_TCPAC (1U << 31) #define CPTR_EL2_TAM (1 << 30) #define CPTR_EL2_TTA (1 << 20) #define CPTR_EL2_TFP (1 << CPTR_EL2_TFP_SHIFT) diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c index 87a68496073a..57f43e607819 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -65,6 +65,9 @@ static void __activate_traps(struct kvm_vcpu *vcpu) __activate_traps_fpsimd32(vcpu); } + if (vcpu_mode_el2(vcpu) && !vcpu_el2_e2h_is_set(vcpu)) + val |= CPTR_EL2_TCPAC; + write_sysreg(val, cpacr_el1); write_sysreg(__this_cpu_read(kvm_hyp_vector), vbar_el1); diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 5960178b7d7c..6490b0e3dcaf 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -1829,7 +1829,7 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_SCTLR_EL1), access_vm_reg, reset_val, SCTLR_EL1, 0x00C50078 }, { SYS_DESC(SYS_ACTLR_EL1), access_actlr, reset_actlr, ACTLR_EL1 }, - { SYS_DESC(SYS_CPACR_EL1), NULL, reset_val, CPACR_EL1, 0 }, + { SYS_DESC(SYS_CPACR_EL1), access_rw, reset_val, CPACR_EL1, 0 }, MTE_REG(RGSR_EL1), MTE_REG(GCR_EL1), From patchwork Mon Nov 29 20:01:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645615 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CEFDCC433EF for ; Mon, 29 Nov 2021 20:07:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232597AbhK2ULO (ORCPT ); Mon, 29 Nov 2021 15:11:14 -0500 Received: from sin.source.kernel.org ([145.40.73.55]:50374 "EHLO sin.source.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238888AbhK2UJN (ORCPT ); Mon, 29 Nov 2021 15:09:13 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 9629BCE1409 for ; Mon, 29 Nov 2021 20:05:54 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C6C12C53FCF; Mon, 29 Nov 2021 20:05:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216352; bh=iTAjg+2Hz6NY3nku7Tg2LsgdREFDIXDRK8hNw6p9WrY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Jv2CpZdt/f6zWR+0mCulX33+j5p6SuWMGFU6zIPsrAwq2Oz0RoumGD+XxViNAFdga qU+/KcLchDn5KyTHDMteqm1c7HEESaEXl8eJXITVcFfp7e1mtRnMsWi8Fiux/FOJC1 fPn0zq54Iy32zWOl68tT/PnBpJwCRyY+YghPdppuQ0PB/3gCPDoCxiWrBOb8GkejEz WClNbKS+xz+nuOzY148carnVY4Frvhu1vAgGP9PTYKVTigFX2yQ2ODxxV3bDlVJ/mw l+kfqEB551n4d7NbdRav7Tt3PNmohevyquJoYGFiDYupKrf/0RhcCoz40OKwMo528b Vd11y6ei3sGqg== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmqv-008gvR-ML; Mon, 29 Nov 2021 20:02:17 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 26/69] KVM: arm64: nv: Handle PSCI call via smc from the guest Date: Mon, 29 Nov 2021 20:01:07 +0000 Message-Id: <20211129200150.351436-27-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Jintack Lim VMs used to execute hvc #0 for the psci call if EL3 is not implemented. However, when we come to provide the virtual EL2 mode to the VM, the host OS inside the VM calls kvm_call_hyp() which is also hvc #0. So, it's hard to differentiate between them from the host hypervisor's point of view. So, let the VM execute smc instruction for the psci call. On ARMv8.3, even if EL3 is not implemented, a smc instruction executed at non-secure EL1 is trapped to EL2 if HCR_EL2.TSC==1, rather than being treated as UNDEFINED. So, the host hypervisor can handle this psci call without any confusion. Signed-off-by: Jintack Lim Signed-off-by: Marc Zyngier --- arch/arm64/kvm/handle_exit.c | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-) diff --git a/arch/arm64/kvm/handle_exit.c b/arch/arm64/kvm/handle_exit.c index 95ae624d6aa8..72b981004b1d 100644 --- a/arch/arm64/kvm/handle_exit.c +++ b/arch/arm64/kvm/handle_exit.c @@ -62,6 +62,8 @@ static int handle_hvc(struct kvm_vcpu *vcpu) static int handle_smc(struct kvm_vcpu *vcpu) { + int ret; + /* * "If an SMC instruction executed at Non-secure EL1 is * trapped to EL2 because HCR_EL2.TSC is 1, the exception is a @@ -69,10 +71,28 @@ static int handle_smc(struct kvm_vcpu *vcpu) * * We need to advance the PC after the trap, as it would * otherwise return to the same address... + * + * If imm is non-zero, it's not defined, so just skip it. + */ + if (kvm_vcpu_hvc_get_imm(vcpu)) { + vcpu_set_reg(vcpu, 0, ~0UL); + kvm_incr_pc(vcpu); + return 1; + } + + /* + * If imm is zero, it's a psci call. + * Note that on ARMv8.3, even if EL3 is not implemented, SMC executed + * at Non-secure EL1 is trapped to EL2 if HCR_EL2.TSC==1, rather than + * being treated as UNDEFINED. */ - vcpu_set_reg(vcpu, 0, ~0UL); + ret = kvm_hvc_call_handler(vcpu); + if (ret < 0) + vcpu_set_reg(vcpu, 0, ~0UL); + kvm_incr_pc(vcpu); - return 1; + + return ret; } /* From patchwork Mon Nov 29 20:01:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645759 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 79431C433EF for ; Mon, 29 Nov 2021 21:44:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232347AbhK2Vr7 (ORCPT ); Mon, 29 Nov 2021 16:47:59 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48570 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232066AbhK2Vpw (ORCPT ); Mon, 29 Nov 2021 16:45:52 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A300FC091D1E for ; Mon, 29 Nov 2021 12:06:10 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 2B24DCE140F for ; Mon, 29 Nov 2021 20:06:10 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7561FC53FC7; Mon, 29 Nov 2021 20:06:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216368; bh=W8ACxOJzCP75ucwzH7CsTdmnHubYK+R9v0dcd/GPhSw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=AAAxQfvfuSSin+Lr0LGBi64vEdPLhrbILnn3s83UQJYS7eLY5JDQqF/WE17sb7eSJ sGS2aibyDgT/GE+9dG39pho5/wP4kxwa8gRpp5icJXZqvBMQc8QaY3gBNY2WGLQXAH Imza7A5oGFuF02rdj4oi0B6bgwWtMeb9dArxn4KPPSA8Q49SUjB+xfSSEtBlcoNS9D CwKQ7dayJwa03wOuO1AzIQ0YdMu4TDzdahV0QgX0E2T1BWv9lQpGqsKPWJ5xLH/Qu1 4Ny6V/iFEsYHzNC2Ls+noAnchOiVaKPo8xe6jM64Z1N0rSM3GjcDUJ31Uqy7IRB/7K 8xGl1o3ZctQow== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmqw-008gvR-1D; Mon, 29 Nov 2021 20:02:18 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 27/69] KVM: arm64: nv: Respect virtual HCR_EL2.TWX setting Date: Mon, 29 Nov 2021 20:01:08 +0000 Message-Id: <20211129200150.351436-28-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Jintack Lim Forward exceptions due to WFI or WFE instructions to the virtual EL2 if they are not coming from the virtual EL2 and virtual HCR_EL2.TWX is set. Signed-off-by: Jintack Lim Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_nested.h | 2 ++ arch/arm64/kvm/Makefile | 2 +- arch/arm64/kvm/handle_exit.c | 11 +++++++- arch/arm64/kvm/nested.c | 40 +++++++++++++++++++++++++++++ 4 files changed, 53 insertions(+), 2 deletions(-) create mode 100644 arch/arm64/kvm/nested.c diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index 67a2c0d05233..4c2ac9650a3e 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -61,4 +61,6 @@ static inline u64 translate_cnthctl_el2_to_cntkctl_el1(u64 cnthctl) (cnthctl & (CNTHCTL_EVNTI | CNTHCTL_EVNTDIR | CNTHCTL_EVNTEN))); } +int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe); + #endif /* __ARM64_KVM_NESTED_H */ diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile index 1f602526e9a2..923ee13728b4 100644 --- a/arch/arm64/kvm/Makefile +++ b/arch/arm64/kvm/Makefile @@ -16,7 +16,7 @@ kvm-y := $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/eventfd.o \ inject_fault.o va_layout.o handle_exit.o \ guest.o debug.o reset.o sys_regs.o \ vgic-sys-reg-v3.o fpsimd.o pmu.o \ - arch_timer.o trng.o emulate-nested.o \ + arch_timer.o trng.o emulate-nested.o nested.o \ vgic/vgic.o vgic/vgic-init.o \ vgic/vgic-irqfd.o vgic/vgic-v2.o \ vgic/vgic-v3.o vgic/vgic-v4.o \ diff --git a/arch/arm64/kvm/handle_exit.c b/arch/arm64/kvm/handle_exit.c index 72b981004b1d..68a9579aa13e 100644 --- a/arch/arm64/kvm/handle_exit.c +++ b/arch/arm64/kvm/handle_exit.c @@ -119,7 +119,16 @@ static int handle_no_fpsimd(struct kvm_vcpu *vcpu) */ static int kvm_handle_wfx(struct kvm_vcpu *vcpu) { - if (kvm_vcpu_get_esr(vcpu) & ESR_ELx_WFx_ISS_WFE) { + bool is_wfe = !!(kvm_vcpu_get_esr(vcpu) & ESR_ELx_WFx_ISS_WFE); + + if (nested_virt_in_use(vcpu)) { + int ret = handle_wfx_nested(vcpu, is_wfe); + + if (ret != -EINVAL) + return ret; + } + + if (is_wfe) { trace_kvm_wfx_arm64(*vcpu_pc(vcpu), true); vcpu->stat.wfe_exit_stat++; kvm_vcpu_on_spin(vcpu, vcpu_mode_priv(vcpu)); diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c new file mode 100644 index 000000000000..42a96c8d2adc --- /dev/null +++ b/arch/arm64/kvm/nested.c @@ -0,0 +1,40 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2017 - Columbia University and Linaro Ltd. + * Author: Jintack Lim + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see . + */ + +#include +#include + +#include + +/* + * Inject wfx to the virtual EL2 if this is not from the virtual EL2 and + * the virtual HCR_EL2.TWX is set. Otherwise, let the host hypervisor + * handle this. + */ +int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe) +{ + u64 hcr_el2 = __vcpu_sys_reg(vcpu, HCR_EL2); + + if (vcpu_mode_el2(vcpu)) + return -EINVAL; + + if ((is_wfe && (hcr_el2 & HCR_TWE)) || (!is_wfe && (hcr_el2 & HCR_TWI))) + return kvm_inject_nested_sync(vcpu, kvm_vcpu_get_esr(vcpu)); + + return -EINVAL; +} From patchwork Mon Nov 29 20:01:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645727 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EE2C2C433EF for ; Mon, 29 Nov 2021 21:38:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231268AbhK2Vlx (ORCPT ); Mon, 29 Nov 2021 16:41:53 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48124 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230286AbhK2Vjv (ORCPT ); Mon, 29 Nov 2021 16:39:51 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0DAF6C091D3B for ; Mon, 29 Nov 2021 12:07:39 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 5DFE9CE1682 for ; Mon, 29 Nov 2021 20:07:37 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id AC375C53FAD; Mon, 29 Nov 2021 20:07:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216455; bh=DG5uIX2bquWNN63M+COe51RmuG6Tckf1/Wal1EhHeWQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=PgN2Z/OT8rVmyE94YzzntX/KUyOTHCZnRlRXDduWjQrOkZKgVv6yfHHBGolbg7Nqx 6bDu9wn/R9jJdYNFgupIxbU0za3fZrON2Ho/DvmeQQo8RNQcBl59h19dBQU1nkha7U NfGGcz77oIMxRDzLBUVM2qk5ydaEfvYKDbpzcfA7+8mEVbz/6SLhxo68cIQfSZy/fq AfSRGOI+FnYDqL2MxD7qVlXskidM4rvG2jn0PsgffeUxH3zvZHFhqCcOFCJAbJiH4i 0Tfk07PnuF6ndJjeES3rqAPN8kc0OT2Wi6Nzr4vncjjAbPTAfCrq/g8gJlP8uglu9L 1majtVC9ZYfKw== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmqw-008gvR-CX; Mon, 29 Nov 2021 20:02:18 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 28/69] KVM: arm64: nv: Respect virtual CPTR_EL2.{TFP,FPEN} settings Date: Mon, 29 Nov 2021 20:01:09 +0000 Message-Id: <20211129200150.351436-29-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Jintack Lim Forward traps due to FP/ASIMD register accesses to the virtual EL2 if virtual CPTR_EL2.TFP is set (with HCR_EL2.E2H == 0) or CPTR_EL2.FPEN is configure to do so (with HCR_EL2.E2h == 1). Signed-off-by: Jintack Lim Signed-off-by: Christoffer Dall [maz: account for HCR_EL2.E2H when testing for TFP/FPEN, with all the hard work actually being done by Chase Conklin] Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_emulate.h | 26 +++++++++++++++++++++++++ arch/arm64/kvm/handle_exit.c | 16 +++++++++++---- arch/arm64/kvm/hyp/include/hyp/switch.h | 8 ++++++-- 3 files changed, 44 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h index 2db11ff3fdad..105fb0d4a366 100644 --- a/arch/arm64/include/asm/kvm_emulate.h +++ b/arch/arm64/include/asm/kvm_emulate.h @@ -11,12 +11,14 @@ #ifndef __ARM64_KVM_EMULATE_H__ #define __ARM64_KVM_EMULATE_H__ +#include #include #include #include #include #include +#include #include #include #include @@ -324,6 +326,30 @@ static inline bool vcpu_mode_priv(const struct kvm_vcpu *vcpu) return mode != PSR_MODE_EL0t; } +static inline bool guest_hyp_fpsimd_traps_enabled(const struct kvm_vcpu *vcpu) +{ + u64 val; + + if (!nested_virt_in_use(vcpu)) + return false; + + val = vcpu_read_sys_reg(vcpu, CPTR_EL2); + + if (!vcpu_el2_e2h_is_set(vcpu)) + return (val & CPTR_EL2_TFP); + + switch (FIELD_GET(CPACR_EL1_FPEN, val)) { + case 0b00: + case 0b10: + return true; + case 0b01: + return vcpu_el2_tge_is_set(vcpu) && !vcpu_mode_el2(vcpu); + case 0b11: + default: /* GCC is dumb */ + return false; + } +} + static __always_inline u32 kvm_vcpu_get_esr(const struct kvm_vcpu *vcpu) { return vcpu->arch.fault.esr_el2; diff --git a/arch/arm64/kvm/handle_exit.c b/arch/arm64/kvm/handle_exit.c index 68a9579aa13e..7721c7c36137 100644 --- a/arch/arm64/kvm/handle_exit.c +++ b/arch/arm64/kvm/handle_exit.c @@ -96,11 +96,19 @@ static int handle_smc(struct kvm_vcpu *vcpu) } /* - * Guest access to FP/ASIMD registers are routed to this handler only - * when the system doesn't support FP/ASIMD. + * This handles the cases where the system does not support FP/ASIMD or when + * we are running nested virtualization and the guest hypervisor is trapping + * FP/ASIMD accesses by its guest guest. + * + * All other handling of guest vs. host FP/ASIMD register state is handled in + * fixup_guest_exit(). */ -static int handle_no_fpsimd(struct kvm_vcpu *vcpu) +static int kvm_handle_fpasimd(struct kvm_vcpu *vcpu) { + if (guest_hyp_fpsimd_traps_enabled(vcpu)) + return kvm_inject_nested_sync(vcpu, kvm_vcpu_get_esr(vcpu)); + + /* This is the case when the system doesn't support FP/ASIMD. */ kvm_inject_undefined(vcpu); return 1; } @@ -229,7 +237,7 @@ static exit_handle_fn arm_exit_handlers[] = { [ESR_ELx_EC_BREAKPT_LOW]= kvm_handle_guest_debug, [ESR_ELx_EC_BKPT32] = kvm_handle_guest_debug, [ESR_ELx_EC_BRK64] = kvm_handle_guest_debug, - [ESR_ELx_EC_FP_ASIMD] = handle_no_fpsimd, + [ESR_ELx_EC_FP_ASIMD] = kvm_handle_fpasimd, [ESR_ELx_EC_PAC] = kvm_handle_ptrauth, }; diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h index 0987abd4d57d..110b347f887a 100644 --- a/arch/arm64/kvm/hyp/include/hyp/switch.h +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h @@ -184,8 +184,12 @@ static bool kvm_hyp_handle_fpsimd(struct kvm_vcpu *vcpu, u64 *exit_code) esr_ec = kvm_vcpu_trap_get_class(vcpu); - /* Don't handle SVE traps for non-SVE vcpus here: */ - if (!sve_guest && esr_ec != ESR_ELx_EC_FP_ASIMD) + /* + * Don't handle SVE traps for non-SVE vcpus here. This + * includes NV guests for the time being. + */ + if (!sve_guest && (esr_ec != ESR_ELx_EC_FP_ASIMD || + guest_hyp_fpsimd_traps_enabled(vcpu))) return false; /* Valid trap. Switch the context: */ From patchwork Mon Nov 29 20:01:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645749 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66D4AC433EF for ; Mon, 29 Nov 2021 21:44:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232664AbhK2Vry (ORCPT ); Mon, 29 Nov 2021 16:47:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49452 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232068AbhK2Vpw (ORCPT ); Mon, 29 Nov 2021 16:45:52 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CE8A3C091D1F for ; Mon, 29 Nov 2021 12:06:16 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 2519BCE140D for ; Mon, 29 Nov 2021 20:06:15 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 573BCC53FCD; Mon, 29 Nov 2021 20:06:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216373; bh=F5fLWHzkHEzNeL+P76dTE8PtKUva3K+S3mn8YzOplZA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=VY38iPV5cjOzyzkjl2e9QEqagKov23ls9BJBhQc7PyAza4TIaeDHyKgVtsUoWjJuR /ZtvMtIYu7GXwyKzScHRNjPvM+h2D1AMzETuu8xQWXqFKoOhd/q7facj6ifPTW+idL JJkuR1Oel23JX9z9g2WAnV8n7uRLstfSVbrNfiXJh30V2+HDOi9QnvKj7Roz82zrfl I+uEzm96jbbb2ACxSKeFD3t7c29OxP+OOfjEVn+cz0uhEpCMyNb6duIGSAeQ7C0QwM QCHSsD+HKmh+MA2QV8avIA1SpaQBrGhKYPDrli70FBG6lYEEM60ZzsRYLPfFW5NibQ Jftdj4WhUPbGQ== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmqw-008gvR-Mx; Mon, 29 Nov 2021 20:02:18 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 29/69] KVM: arm64: nv: Respect the virtual HCR_EL2.NV bit setting Date: Mon, 29 Nov 2021 20:01:10 +0000 Message-Id: <20211129200150.351436-30-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Jintack Lim Forward traps due to HCR_EL2.NV bit to the virtual EL2 if they are not coming from the virtual EL2 and the virtual HCR_EL2.NV bit is set. In addition to EL2 register accesses, setting NV bit will also make EL12 register accesses trap to EL2. To emulate this for the virtual EL2, forword traps due to EL12 register accessses to the virtual EL2 if the virtual HCR_EL2.NV bit is set. This is for recursive nested virtualization. Signed-off-by: Jintack Lim [Moved code to emulate-nested.c] Signed-off-by: Christoffer Dall Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_arm.h | 1 + arch/arm64/include/asm/kvm_nested.h | 2 ++ arch/arm64/kvm/emulate-nested.c | 27 +++++++++++++++++++++++++++ arch/arm64/kvm/handle_exit.c | 7 +++++++ arch/arm64/kvm/sys_regs.c | 24 ++++++++++++++++++++++++ 5 files changed, 61 insertions(+) diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h index 2eccf883e8fe..9759bc893a51 100644 --- a/arch/arm64/include/asm/kvm_arm.h +++ b/arch/arm64/include/asm/kvm_arm.h @@ -20,6 +20,7 @@ #define HCR_AMVOFFEN (UL(1) << 51) #define HCR_FIEN (UL(1) << 47) #define HCR_FWB (UL(1) << 46) +#define HCR_NV (UL(1) << 42) #define HCR_API (UL(1) << 41) #define HCR_APK (UL(1) << 40) #define HCR_TEA (UL(1) << 37) diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index 4c2ac9650a3e..26cba7b4d743 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -62,5 +62,7 @@ static inline u64 translate_cnthctl_el2_to_cntkctl_el1(u64 cnthctl) } int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe); +extern bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit); +extern bool forward_nv_traps(struct kvm_vcpu *vcpu); #endif /* __ARM64_KVM_NESTED_H */ diff --git a/arch/arm64/kvm/emulate-nested.c b/arch/arm64/kvm/emulate-nested.c index 339e8272b01e..8c7f2fe24bc6 100644 --- a/arch/arm64/kvm/emulate-nested.c +++ b/arch/arm64/kvm/emulate-nested.c @@ -25,11 +25,38 @@ #include "trace.h" +bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit) +{ + bool control_bit_set; + + if (!nested_virt_in_use(vcpu)) + return false; + + control_bit_set = __vcpu_sys_reg(vcpu, HCR_EL2) & control_bit; + if (!vcpu_mode_el2(vcpu) && control_bit_set) { + kvm_inject_nested_sync(vcpu, kvm_vcpu_get_esr(vcpu)); + return true; + } + return false; +} + +bool forward_nv_traps(struct kvm_vcpu *vcpu) +{ + return forward_traps(vcpu, HCR_NV); +} + void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu) { u64 spsr, elr, mode; bool direct_eret; + /* + * Forward this trap to the virtual EL2 if the virtual + * HCR_EL2.NV bit is set and this is coming from !EL2. + */ + if (forward_nv_traps(vcpu)) + return; + /* * Going through the whole put/load motions is a waste of time * if this is a VHE guest hypervisor returning to its own diff --git a/arch/arm64/kvm/handle_exit.c b/arch/arm64/kvm/handle_exit.c index 7721c7c36137..6ff709c124d0 100644 --- a/arch/arm64/kvm/handle_exit.c +++ b/arch/arm64/kvm/handle_exit.c @@ -64,6 +64,13 @@ static int handle_smc(struct kvm_vcpu *vcpu) { int ret; + /* + * Forward this trapped smc instruction to the virtual EL2 if + * the guest has asked for it. + */ + if (forward_traps(vcpu, HCR_TSC)) + return 1; + /* * "If an SMC instruction executed at Non-secure EL1 is * trapped to EL2 because HCR_EL2.TSC is 1, the exception is a diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 6490b0e3dcaf..3468b8df8661 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -267,10 +267,19 @@ static u32 get_ccsidr(u32 csselr) return ccsidr; } +static bool el12_reg(struct sys_reg_params *p) +{ + /* All *_EL12 registers have Op1=5. */ + return (p->Op1 == 5); +} + static bool access_rw(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) { + if (el12_reg(p) && forward_nv_traps(vcpu)) + return false; + if (p->is_write) vcpu_write_sys_reg(vcpu, p->regval, r->reg); else @@ -283,6 +292,9 @@ static bool access_sctlr_el2(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) { + if (el12_reg(p) && forward_nv_traps(vcpu)) + return false; + if (p->is_write) { u64 val = p->regval; @@ -367,6 +379,9 @@ static bool access_vm_reg(struct kvm_vcpu *vcpu, bool was_enabled = vcpu_has_cache_enabled(vcpu); u64 val, mask, shift; + if (el12_reg(p) && forward_nv_traps(vcpu)) + return false; + /* We don't expect TRVM on the host */ BUG_ON(!vcpu_mode_el2(vcpu) && !p->is_write); @@ -1664,6 +1679,9 @@ static bool access_elr(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) { + if (el12_reg(p) && forward_nv_traps(vcpu)) + return false; + if (p->is_write) vcpu_write_sys_reg(vcpu, p->regval, ELR_EL1); else @@ -1676,6 +1694,9 @@ static bool access_spsr(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) { + if (el12_reg(p) && forward_nv_traps(vcpu)) + return false; + if (p->is_write) __vcpu_sys_reg(vcpu, SPSR_EL1) = p->regval; else @@ -1688,6 +1709,9 @@ static bool access_spsr_el2(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) { + if (el12_reg(p) && forward_nv_traps(vcpu)) + return false; + if (p->is_write) vcpu_write_sys_reg(vcpu, p->regval, SPSR_EL2); else From patchwork Mon Nov 29 20:01:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645905 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE9F2C433EF for ; Mon, 29 Nov 2021 22:20:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229764AbhK2WXf (ORCPT ); Mon, 29 Nov 2021 17:23:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56546 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233217AbhK2WVd (ORCPT ); Mon, 29 Nov 2021 17:21:33 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D4C0BC091D3D for ; Mon, 29 Nov 2021 12:07:53 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 52DABCE140F for ; Mon, 29 Nov 2021 20:07:52 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 83AFCC53FCD; Mon, 29 Nov 2021 20:07:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216470; bh=iJui46ZcjZoyFxy28ez7102NEjA5q3RQEoAi8jCgTJI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=KDvMQKBk+sYYiGsmoh2H2oe2+SEOMSHIZ5dLPnqUbcNfRdpqOaICLXukmri4U2IhN wZ0In22rJkTmx6PQipCEfphs8PZYD/Nh+ppxXQmBViJAkWfdvCNXJx0mJeqMKoP1Zp Rpfg1cJ4CV/QGjt0Ftkl86k6iXeEFxCNFe9rw3ZGx/5mk4dclHat0bhqMnVL2H9BNF wQ6pfgrZFv5bBDk5siZ6Cai4pxT5UU7xAE0nOoq7ZnwTtHXx2nZlk4BpU0zk9sykKt qSLCBbucVN3vNYdhdnkW+jnagE4MHbSGBFWcz57TL6Ygvwc/kr6u+isWvp8nPP10fO iCjeZtg53pt5g== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmqx-008gvR-08; Mon, 29 Nov 2021 20:02:19 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 30/69] KVM: arm64: nv: Respect virtual HCR_EL2.TVM and TRVM settings Date: Mon, 29 Nov 2021 20:01:11 +0000 Message-Id: <20211129200150.351436-31-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Jintack Lim Forward the EL1 virtual memory register traps to the virtual EL2 if they are not coming from the virtual EL2 and the virtual HCR_EL2.TVM or TRVM bit is set. This is for recursive nested virtualization. Signed-off-by: Jintack Lim Signed-off-by: Marc Zyngier --- arch/arm64/kvm/sys_regs.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 3468b8df8661..e96877fc3b2a 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -382,6 +382,13 @@ static bool access_vm_reg(struct kvm_vcpu *vcpu, if (el12_reg(p) && forward_nv_traps(vcpu)) return false; + if (!el12_reg(p)) { + u64 bit = p->is_write ? HCR_TVM : HCR_TRVM; + + if (forward_traps(vcpu, bit)) + return false; + } + /* We don't expect TRVM on the host */ BUG_ON(!vcpu_mode_el2(vcpu) && !p->is_write); From patchwork Mon Nov 29 20:01:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645671 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 15313C433EF for ; Mon, 29 Nov 2021 20:09:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234207AbhK2UNC (ORCPT ); Mon, 29 Nov 2021 15:13:02 -0500 Received: from sin.source.kernel.org ([145.40.73.55]:51378 "EHLO sin.source.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232336AbhK2ULB (ORCPT ); Mon, 29 Nov 2021 15:11:01 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 14402CE1685 for ; Mon, 29 Nov 2021 20:07:42 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5DABEC53FAD; Mon, 29 Nov 2021 20:07:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216460; bh=1V+tSJ47CQKMJjjBnFGrwulS3BEp8dzYBPSx5F6d1xg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Ivh9zUktPPU5JLepP58QrEPSQHNPFkllJpR6WUNviJEwg/vwcV15zNcby65cM9Tqx P5UQ8ODxxU0vGPm2voN00Wj9rPPRQOy0Lq55AZ+ORo0rkWhXEfBNLCqkmP+iSeQ4nM t3F+uOI9QiXCxEzEACpOHF2HErFj+1dH+85sm6TmTzi/CAg1UG5yM2dl4h286fnsoo 6SpGptL/FPrJJdDM9AVoHpNxjKipzpppkr9jDOpSSqdeQvIYuvz45Ohke7JxwUkWB/ OiBEwKv9A6sQoz8Du7smT1KwlcUNFCy6xhVMLAZOS2IyNTz1WuAvAn0T5TKa32El7v lfsuySoyD225g== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmqx-008gvR-9J; Mon, 29 Nov 2021 20:02:19 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 31/69] KVM: arm64: nv: Respect the virtual HCR_EL2.NV1 bit setting Date: Mon, 29 Nov 2021 20:01:12 +0000 Message-Id: <20211129200150.351436-32-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Jintack Lim Forward ELR_EL1, SPSR_EL1 and VBAR_EL1 traps to the virtual EL2 if the virtual HCR_EL2.NV bit is set. This is for recursive nested virtualization. Signed-off-by: Jintack Lim Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_arm.h | 1 + arch/arm64/kvm/sys_regs.c | 28 +++++++++++++++++++++++++++- 2 files changed, 28 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h index 9759bc893a51..68af5509e4b0 100644 --- a/arch/arm64/include/asm/kvm_arm.h +++ b/arch/arm64/include/asm/kvm_arm.h @@ -20,6 +20,7 @@ #define HCR_AMVOFFEN (UL(1) << 51) #define HCR_FIEN (UL(1) << 47) #define HCR_FWB (UL(1) << 46) +#define HCR_NV1 (UL(1) << 43) #define HCR_NV (UL(1) << 42) #define HCR_API (UL(1) << 41) #define HCR_APK (UL(1) << 40) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index e96877fc3b2a..511e06b6f603 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -288,6 +288,22 @@ static bool access_rw(struct kvm_vcpu *vcpu, return true; } +/* This function is to support the recursive nested virtualization */ +static bool forward_nv1_traps(struct kvm_vcpu *vcpu, struct sys_reg_params *p) +{ + return forward_traps(vcpu, HCR_NV1); +} + +static bool access_vbar_el1(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + if (forward_nv1_traps(vcpu, p)) + return false; + + return access_rw(vcpu, p, r); +} + static bool access_sctlr_el2(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) @@ -1682,6 +1698,7 @@ static bool access_sp_el1(struct kvm_vcpu *vcpu, return true; } + static bool access_elr(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) @@ -1689,6 +1706,9 @@ static bool access_elr(struct kvm_vcpu *vcpu, if (el12_reg(p) && forward_nv_traps(vcpu)) return false; + if (!el12_reg(p) && forward_nv1_traps(vcpu, p)) + return false; + if (p->is_write) vcpu_write_sys_reg(vcpu, p->regval, ELR_EL1); else @@ -1704,6 +1724,9 @@ static bool access_spsr(struct kvm_vcpu *vcpu, if (el12_reg(p) && forward_nv_traps(vcpu)) return false; + if (!el12_reg(p) && forward_nv1_traps(vcpu, p)) + return false; + if (p->is_write) __vcpu_sys_reg(vcpu, SPSR_EL1) = p->regval; else @@ -1719,6 +1742,9 @@ static bool access_spsr_el2(struct kvm_vcpu *vcpu, if (el12_reg(p) && forward_nv_traps(vcpu)) return false; + if (!el12_reg(p) && forward_nv1_traps(vcpu, p)) + return false; + if (p->is_write) vcpu_write_sys_reg(vcpu, p->regval, SPSR_EL2); else @@ -1927,7 +1953,7 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_LORC_EL1), trap_loregion }, { SYS_DESC(SYS_LORID_EL1), trap_loregion }, - { SYS_DESC(SYS_VBAR_EL1), access_rw, reset_val, VBAR_EL1, 0 }, + { SYS_DESC(SYS_VBAR_EL1), access_vbar_el1, reset_val, VBAR_EL1, 0 }, { SYS_DESC(SYS_DISR_EL1), NULL, reset_val, DISR_EL1, 0 }, { SYS_DESC(SYS_ICC_IAR0_EL1), write_to_read_only }, From patchwork Mon Nov 29 20:01:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645725 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C98B7C4332F for ; Mon, 29 Nov 2021 21:38:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231269AbhK2Vly (ORCPT ); Mon, 29 Nov 2021 16:41:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47262 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230287AbhK2Vjw (ORCPT ); Mon, 29 Nov 2021 16:39:52 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4FA08C091D35 for ; Mon, 29 Nov 2021 12:07:29 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id BC69DCE167D for ; Mon, 29 Nov 2021 20:07:27 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 11CECC53FC7; Mon, 29 Nov 2021 20:07:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216446; bh=++qy0lG1eKiHd1cXmJ9EfdurXim5AWnrzLZyRWTE+Tc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=SvtHfOncWUdrGjxvIKIH0RoktARrxLEAVhK6MEpXR9lh9qMefdUbwFKHZVRaBuBvh oMIW6w6AiW5grG8RnaJWq47YsZdIUZJf0XrXY/X3Wru4HzQ6qwgx2Cytg26VPj1F1p gAvAdHYVoyx5hetH/3x+nZGaUoVvLHuY4Hc7B1L2+Fv2CP9P8RyHniseSbj2fII7jO 2UnS33Ji6JZtKXUOb9d/JJdPHbRgo23QSrfhcd0xM8wqjcEbMl573lHlPvJrgljhWi MycyYF3s2ZnI7qU4yyJGcHBlNXkp6YlKVu53vfokDWDHa1aK0r2WJFcjX8HmnKI2/5 7dy1hy3ZjMVvA== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmqx-008gvR-Kt; Mon, 29 Nov 2021 20:02:19 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 32/69] KVM: arm64: nv: Emulate EL12 register accesses from the virtual EL2 Date: Mon, 29 Nov 2021 20:01:13 +0000 Message-Id: <20211129200150.351436-33-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Jintack Lim With HCR_EL2.NV bit set, accesses to EL12 registers in the virtual EL2 trap to EL2. Handle those traps just like we do for EL1 registers. One exception is CNTKCTL_EL12. We don't trap on CNTKCTL_EL1 for non-VHE virtual EL2 because we don't have to. However, accessing CNTKCTL_EL12 will trap since it's one of the EL12 registers controlled by HCR_EL2.NV bit. Therefore, add a handler for it and don't treat it as a non-trap-registers when preparing a shadow context. Signed-off-by: Jintack Lim Signed-off-by: Marc Zyngier --- arch/arm64/kvm/sys_regs.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 511e06b6f603..692cade54caf 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -2214,6 +2214,23 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_CNTVOFF_EL2), access_rw, reset_val, CNTVOFF_EL2, 0 }, { SYS_DESC(SYS_CNTHCTL_EL2), access_rw, reset_val, CNTHCTL_EL2, 0 }, + { SYS_DESC(SYS_SCTLR_EL12), access_vm_reg, reset_val, SCTLR_EL1, 0x00C50078 }, + { SYS_DESC(SYS_CPACR_EL12), access_rw, reset_val, CPACR_EL1, 0 }, + { SYS_DESC(SYS_TTBR0_EL12), access_vm_reg, reset_unknown, TTBR0_EL1 }, + { SYS_DESC(SYS_TTBR1_EL12), access_vm_reg, reset_unknown, TTBR1_EL1 }, + { SYS_DESC(SYS_TCR_EL12), access_vm_reg, reset_val, TCR_EL1, 0 }, + { SYS_DESC(SYS_SPSR_EL12), access_spsr}, + { SYS_DESC(SYS_ELR_EL12), access_elr}, + { SYS_DESC(SYS_AFSR0_EL12), access_vm_reg, reset_unknown, AFSR0_EL1 }, + { SYS_DESC(SYS_AFSR1_EL12), access_vm_reg, reset_unknown, AFSR1_EL1 }, + { SYS_DESC(SYS_ESR_EL12), access_vm_reg, reset_unknown, ESR_EL1 }, + { SYS_DESC(SYS_FAR_EL12), access_vm_reg, reset_unknown, FAR_EL1 }, + { SYS_DESC(SYS_MAIR_EL12), access_vm_reg, reset_unknown, MAIR_EL1 }, + { SYS_DESC(SYS_AMAIR_EL12), access_vm_reg, reset_amair_el1, AMAIR_EL1 }, + { SYS_DESC(SYS_VBAR_EL12), access_rw, reset_val, VBAR_EL1, 0 }, + { SYS_DESC(SYS_CONTEXTIDR_EL12), access_vm_reg, reset_val, CONTEXTIDR_EL1, 0 }, + { SYS_DESC(SYS_CNTKCTL_EL12), access_rw, reset_val, CNTKCTL_EL1, 0 }, + { SYS_DESC(SYS_SP_EL2), NULL, reset_unknown, SP_EL2 }, }; From patchwork Mon Nov 29 20:01:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645717 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C3934C433EF for ; Mon, 29 Nov 2021 21:24:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231408AbhK2V1x (ORCPT ); Mon, 29 Nov 2021 16:27:53 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44636 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231160AbhK2VZu (ORCPT ); Mon, 29 Nov 2021 16:25:50 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 344C3C091D3E for ; Mon, 29 Nov 2021 12:07:57 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id B46C7CE1409 for ; Mon, 29 Nov 2021 20:07:56 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 13300C53FCF; Mon, 29 Nov 2021 20:07:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216475; bh=Ph49KKBZ+O22t+hKyvWjXj2PVYxm0X5Rb2BvJLz8sHs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=cJuSURiTc+KIGijIrvc2xeAEkXgjVpGvvvbLINxP40eqOBVXlF4Ch6hMy5WwDVVYn yzlspQBDDmIzIl15pgLxwo/DBLYn0MBl8yY4OTHnSLqg3yfaTXBFiF2iVtBN7YnTyU L6a5/6JDAzAXfPL24UDyZOPVnFaozbIUiF3DsT+GKlMJfZQ6aC4uzF/2o+wrPtL/G9 jWUP7dL9r2KyfrcNBIQZj0tD+LePoo0K752ktNfrSMyH7Z8/dIaj/LHl8FJQ6I/Nf2 dyxe6T+GhqX2yp6azjXKqbKfzbg4r4IQd2qskCHjQzHVy426kHTMazUwPD889pIFeK GoRxbBZrnSKIw== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmqx-008gvR-U7; Mon, 29 Nov 2021 20:02:19 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 33/69] KVM: arm64: nv: Forward debug traps to the nested guest Date: Mon, 29 Nov 2021 20:01:14 +0000 Message-Id: <20211129200150.351436-34-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org On handling a debug trap, check whether we need to forward it to the guest before handling it. Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_nested.h | 2 ++ arch/arm64/kvm/emulate-nested.c | 9 +++++++-- arch/arm64/kvm/sys_regs.c | 3 +++ 3 files changed, 12 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index 26cba7b4d743..07c15f51cf86 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -62,6 +62,8 @@ static inline u64 translate_cnthctl_el2_to_cntkctl_el1(u64 cnthctl) } int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe); +extern bool __forward_traps(struct kvm_vcpu *vcpu, unsigned int reg, + u64 control_bit); extern bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit); extern bool forward_nv_traps(struct kvm_vcpu *vcpu); diff --git a/arch/arm64/kvm/emulate-nested.c b/arch/arm64/kvm/emulate-nested.c index 8c7f2fe24bc6..5fbf8978acf2 100644 --- a/arch/arm64/kvm/emulate-nested.c +++ b/arch/arm64/kvm/emulate-nested.c @@ -25,14 +25,14 @@ #include "trace.h" -bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit) +bool __forward_traps(struct kvm_vcpu *vcpu, unsigned int reg, u64 control_bit) { bool control_bit_set; if (!nested_virt_in_use(vcpu)) return false; - control_bit_set = __vcpu_sys_reg(vcpu, HCR_EL2) & control_bit; + control_bit_set = __vcpu_sys_reg(vcpu, reg) & control_bit; if (!vcpu_mode_el2(vcpu) && control_bit_set) { kvm_inject_nested_sync(vcpu, kvm_vcpu_get_esr(vcpu)); return true; @@ -40,6 +40,11 @@ bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit) return false; } +bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit) +{ + return __forward_traps(vcpu, HCR_EL2, control_bit); +} + bool forward_nv_traps(struct kvm_vcpu *vcpu) { return forward_traps(vcpu, HCR_NV); diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 692cade54caf..9deedd5a058f 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -603,6 +603,9 @@ static bool trap_debug_regs(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) { + if (__forward_traps(vcpu, MDCR_EL2, MDCR_EL2_TDA | MDCR_EL2_TDE)) + return false; + access_rw(vcpu, p, r); if (p->is_write) vcpu->arch.flags |= KVM_ARM64_DEBUG_DIRTY; From patchwork Mon Nov 29 20:01:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645669 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4DB75C433EF for ; Mon, 29 Nov 2021 20:09:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232114AbhK2UM7 (ORCPT ); Mon, 29 Nov 2021 15:12:59 -0500 Received: from sin.source.kernel.org ([145.40.73.55]:51366 "EHLO sin.source.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232115AbhK2UK6 (ORCPT ); Mon, 29 Nov 2021 15:10:58 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 5C423CE13D8 for ; Mon, 29 Nov 2021 20:07:39 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C94E3C53FCD; Mon, 29 Nov 2021 20:07:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216457; bh=9drlAwh2leh8BhdKijYQ2wL0kV3gJrPM63BYQ9ujVxU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=tq1+FEw6be9oWD6aDpE0DK7ixkrprlJLK8LULkipLNCJfD2d26EbBIfkJ9x3ScjN9 gE6MqTDHIn23HZDUmvzzETQrRZkZgRxdCUFMKVdEzhK3k+5WI3S8bRyBnrGqihpoSt kj6UcKkCInlqR/Slo7+UqfPHd6F/zusNVoNZU9QhK4eQr9M8WynJh/15ifQndJ4otn zBsevBRyo7zVSc8ya+JGrWeMOgKD1K3vmWxT2oJk/RAE+Bq5DznfLdpM90VDNChq3i 0yeNi2eoI4iFBFBwh9w8ROYKeimC/JrdZaCniA6F1zHOC0KCCd8QZ8vcpJeWaJxPVG mG5JmyTL3dzhw== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmqy-008gvR-8N; Mon, 29 Nov 2021 20:02:20 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 34/69] KVM: arm64: nv: Configure HCR_EL2 for nested virtualization Date: Mon, 29 Nov 2021 20:01:15 +0000 Message-Id: <20211129200150.351436-35-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Jintack Lim We enable nested virtualization by setting the HCR NV and NV1 bit. When the virtual E2H bit is set, we can support EL2 register accesses via EL1 registers from the virtual EL2 by doing trap-and-emulate. A better alternative, however, is to allow the virtual EL2 to access EL2 register states without trap. This can be easily achieved by not traping EL1 registers since those registers already have EL2 register states. Signed-off-by: Jintack Lim Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_arm.h | 1 + arch/arm64/kvm/hyp/vhe/switch.c | 38 +++++++++++++++++++++++++++++--- 2 files changed, 36 insertions(+), 3 deletions(-) diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h index 68af5509e4b0..b8a0d410035b 100644 --- a/arch/arm64/include/asm/kvm_arm.h +++ b/arch/arm64/include/asm/kvm_arm.h @@ -87,6 +87,7 @@ HCR_BSU_IS | HCR_FB | HCR_TACR | \ HCR_AMO | HCR_SWIO | HCR_TIDCP | HCR_RW | HCR_TLOR | \ HCR_FMO | HCR_IMO | HCR_PTW ) +#define HCR_GUEST_NV_FILTER_FLAGS (HCR_ATA | HCR_API | HCR_APK | HCR_RW) #define HCR_VIRT_EXCP_MASK (HCR_VSE | HCR_VI | HCR_VF) #define HCR_HOST_NVHE_FLAGS (HCR_RW | HCR_API | HCR_APK | HCR_ATA) #define HCR_HOST_NVHE_PROTECTED_FLAGS (HCR_HOST_NVHE_FLAGS | HCR_TSC) diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c index 57f43e607819..da80c969e623 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -36,9 +36,41 @@ static void __activate_traps(struct kvm_vcpu *vcpu) u64 hcr = vcpu->arch.hcr_el2; u64 val; - /* Trap VM sysreg accesses if an EL2 guest is not using VHE. */ - if (vcpu_mode_el2(vcpu) && !vcpu_el2_e2h_is_set(vcpu)) - hcr |= HCR_TVM | HCR_TRVM; + if (is_hyp_ctxt(vcpu)) { + hcr |= HCR_NV; + + if (!vcpu_el2_e2h_is_set(vcpu)) { + /* + * For a guest hypervisor on v8.0, trap and emulate + * the EL1 virtual memory control register accesses. + */ + hcr |= HCR_TVM | HCR_TRVM | HCR_NV1; + } else { + /* + * For a guest hypervisor on v8.1 (VHE), allow to + * access the EL1 virtual memory control registers + * natively. These accesses are to access EL2 register + * states. + * Note that we still need to respect the virtual + * HCR_EL2 state. + */ + u64 vhcr_el2 = __vcpu_sys_reg(vcpu, HCR_EL2); + + vhcr_el2 &= ~HCR_GUEST_NV_FILTER_FLAGS; + + /* + * We already set TVM to handle set/way cache maint + * ops traps, this somewhat collides with the nested + * virt trapping for nVHE. So turn this off for now + * here, in the hope that VHE guests won't ever do this. + * TODO: find out whether it's worth to support both + * cases at the same time. + */ + hcr &= ~HCR_TVM; + + hcr |= vhcr_el2 & (HCR_TVM | HCR_TRVM); + } + } ___activate_traps(vcpu, hcr); From patchwork Mon Nov 29 20:01:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645621 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 71E58C433F5 for ; Mon, 29 Nov 2021 20:08:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233144AbhK2ULy (ORCPT ); Mon, 29 Nov 2021 15:11:54 -0500 Received: from ams.source.kernel.org ([145.40.68.75]:42266 "EHLO ams.source.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230243AbhK2UJv (ORCPT ); Mon, 29 Nov 2021 15:09:51 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 1CB93B815E9 for ; Mon, 29 Nov 2021 20:06:33 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id CE28BC53FCD; Mon, 29 Nov 2021 20:06:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216391; bh=HOZdzEjZoeKGETrJPFOERpPygAZITG+WMe69qICsnxY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Ava+keXKw6IIMzpQUO+tL1qONJ2H21WVoLC4+orWHICkvY+0kswSXgLyDPzkNeN6t yTeJqq3pCyICyTur1hzTP/Bk8TmcCC0wUuUKW6DahMdPzgQaFWifPDt0a4ft/+k5VO k77rKHZBJH8NPMVtp0paNX3W73Su5YoO4SMnQTPOJ3N1KEsJ9GYZXWBvuM3zZtFN4X 65fnN+EoXxVPgXyKzMqrgce87NqIw7r1B8hEI9dwb9zA5Tu+8EnR75ztwJmdnTziAp VINw0fzo/NkineRWz9ZprgZmgCUdVj84Tcbd5g+TDEa6/soWD8csP1n+UMc56wQQ3f 530ctogYlo39A== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmqy-008gvR-I8; Mon, 29 Nov 2021 20:02:20 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 35/69] KVM: arm64: nv: Only toggle cache for virtual EL2 when SCTLR_EL2 changes Date: Mon, 29 Nov 2021 20:01:16 +0000 Message-Id: <20211129200150.351436-36-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Christoffer Dall So far we were flushing almost the entire universe whenever a VM would load/unload the SCTLR_EL1 and the two versions of that register had different MMU enabled settings. This turned out to be so slow that it prevented forward progress for a nested VM, because a scheduler timer tick interrupt would always be pending when we reached the nested VM. To avoid this problem, we consider the SCTLR_EL2 when evaluating if caches are on or off when entering virtual EL2 (because this is the value that we end up shadowing onto the hardware EL1 register). Signed-off-by: Christoffer Dall Signed-off-by: Jintack Lim Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_mmu.h | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index 02d378887743..c018c7b40761 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -115,6 +115,7 @@ alternative_cb_end #include #include #include +#include void kvm_update_va_mask(struct alt_instr *alt, __le32 *origptr, __le32 *updptr, int nr_inst); @@ -185,7 +186,10 @@ struct kvm; static inline bool vcpu_has_cache_enabled(struct kvm_vcpu *vcpu) { - return (vcpu_read_sys_reg(vcpu, SCTLR_EL1) & 0b101) == 0b101; + if (vcpu_mode_el2(vcpu)) + return (__vcpu_sys_reg(vcpu, SCTLR_EL2) & 0b101) == 0b101; + else + return (vcpu_read_sys_reg(vcpu, SCTLR_EL1) & 0b101) == 0b101; } static inline void __clean_dcache_guest_page(void *va, size_t size) From patchwork Mon Nov 29 20:01:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645781 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 81035C433EF for ; Mon, 29 Nov 2021 21:46:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232573AbhK2Vtx (ORCPT ); Mon, 29 Nov 2021 16:49:53 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49452 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232339AbhK2Vrw (ORCPT ); Mon, 29 Nov 2021 16:47:52 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 86BA3C091D2B for ; Mon, 29 Nov 2021 12:07:08 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 12814CE13D8 for ; Mon, 29 Nov 2021 20:07:08 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3CA76C53FD1; Mon, 29 Nov 2021 20:07:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216426; bh=cJeHssu6W6tbH2sGWKdERubXg5G7pf2q9AVmfzXNOeU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=iH5Ls/9sv96MP0xvUNz8SkNdcxHqbG7BxO4FafXsNJxPmSTbocopJ82UiwN3F5eaN 4SZVH6nomQkVcIJ+y2XCUxjxzm999MLme+wbfZ/py5bvreC+m+SR2V0SpFm5K8zrlN GAAtqkUXPU+VTlXq28wmfK5Ll5GDD7Hh0tVfXX0pcCi9fSacKbFBSi3OC9OLMOHVWI nD4Gr41GypD+6Z289lrQGYhDx8nuK5Bl5koqfL+j2vZ+kySHhAaBdM437KtAXupuBs mrl+kEX3CIL5hIgGljkEhB60taK6geIYxbo7vl4OcmH7ZlJzp5aQbJSAXXL/9v0rWC Hu1Y4forKNFPg== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmqy-008gvR-Qx; Mon, 29 Nov 2021 20:02:20 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 36/69] KVM: arm64: nv: Filter out unsupported features from ID regs Date: Mon, 29 Nov 2021 20:01:17 +0000 Message-Id: <20211129200150.351436-37-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org As there is a number of features that we either can't support, or don't want to support right away with NV, let's add some basic filtering so that we don't advertize silly things to the EL2 guest. Whilst we are at it, avertize ARMv8.4-TTL as well as ARMv8.5-GTG. Signed-off-by: Marc Zyngier Reviewed-by: Ganapatrao Kulkarni --- arch/arm64/include/asm/kvm_nested.h | 6 ++ arch/arm64/kvm/nested.c | 152 ++++++++++++++++++++++++++++ arch/arm64/kvm/sys_regs.c | 4 +- arch/arm64/kvm/sys_regs.h | 2 + 4 files changed, 163 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index 07c15f51cf86..026ddaad972c 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -67,4 +67,10 @@ extern bool __forward_traps(struct kvm_vcpu *vcpu, unsigned int reg, extern bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit); extern bool forward_nv_traps(struct kvm_vcpu *vcpu); +struct sys_reg_params; +struct sys_reg_desc; + +void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p, + const struct sys_reg_desc *r); + #endif /* __ARM64_KVM_NESTED_H */ diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c index 42a96c8d2adc..19b674983e13 100644 --- a/arch/arm64/kvm/nested.c +++ b/arch/arm64/kvm/nested.c @@ -20,6 +20,10 @@ #include #include +#include +#include + +#include "sys_regs.h" /* * Inject wfx to the virtual EL2 if this is not from the virtual EL2 and @@ -38,3 +42,151 @@ int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe) return -EINVAL; } + +/* + * Our emulated CPU doesn't support all the possible features. For the + * sake of simplicity (and probably mental sanity), wipe out a number + * of feature bits we don't intend to support for the time being. + * This list should get updated as new features get added to the NV + * support, and new extension to the architecture. + */ +void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + u32 id = sys_reg((u32)r->Op0, (u32)r->Op1, + (u32)r->CRn, (u32)r->CRm, (u32)r->Op2); + u64 val, tmp; + + if (!nested_virt_in_use(v)) + return; + + val = p->regval; + + switch (id) { + case SYS_ID_AA64ISAR0_EL1: + /* Support everything but O.S. and Range TLBIs */ + val &= ~(FEATURE(ID_AA64ISAR0_TLB) | + GENMASK_ULL(27, 24) | + GENMASK_ULL(3, 0)); + break; + + case SYS_ID_AA64ISAR1_EL1: + /* Support everything but PtrAuth and Spec Invalidation */ + val &= ~(GENMASK_ULL(63, 56) | + FEATURE(ID_AA64ISAR1_SPECRES) | + FEATURE(ID_AA64ISAR1_GPI) | + FEATURE(ID_AA64ISAR1_GPA) | + FEATURE(ID_AA64ISAR1_API) | + FEATURE(ID_AA64ISAR1_APA)); + break; + + case SYS_ID_AA64PFR0_EL1: + /* No AMU, MPAM, S-EL2, RAS or SVE */ + val &= ~(GENMASK_ULL(55, 52) | + FEATURE(ID_AA64PFR0_AMU) | + FEATURE(ID_AA64PFR0_MPAM) | + FEATURE(ID_AA64PFR0_SEL2) | + FEATURE(ID_AA64PFR0_RAS) | + FEATURE(ID_AA64PFR0_SVE) | + FEATURE(ID_AA64PFR0_EL3) | + FEATURE(ID_AA64PFR0_EL2)); + /* 64bit EL2/EL3 only */ + val |= FIELD_PREP(FEATURE(ID_AA64PFR0_EL2), 0b0001); + val |= FIELD_PREP(FEATURE(ID_AA64PFR0_EL3), 0b0001); + break; + + case SYS_ID_AA64PFR1_EL1: + /* Only support SSBS */ + val &= FEATURE(ID_AA64PFR1_SSBS); + break; + + case SYS_ID_AA64MMFR0_EL1: + /* Hide ECV, FGT, ExS, Secure Memory */ + val &= ~(GENMASK_ULL(63, 43) | + FEATURE(ID_AA64MMFR0_TGRAN4_2) | + FEATURE(ID_AA64MMFR0_TGRAN16_2) | + FEATURE(ID_AA64MMFR0_TGRAN64_2) | + FEATURE(ID_AA64MMFR0_SNSMEM)); + + /* Disallow unsupported S2 page sizes */ + switch (PAGE_SIZE) { + case SZ_64K: + val |= FIELD_PREP(FEATURE(ID_AA64MMFR0_TGRAN16_2), 0b0001); + fallthrough; + case SZ_16K: + val |= FIELD_PREP(FEATURE(ID_AA64MMFR0_TGRAN4_2), 0b0001); + fallthrough; + case SZ_4K: + /* Support everything */ + break; + } + /* Advertize supported S2 page sizes */ + switch (PAGE_SIZE) { + case SZ_4K: + val |= FIELD_PREP(FEATURE(ID_AA64MMFR0_TGRAN4_2), 0b0010); + fallthrough; + case SZ_16K: + val |= FIELD_PREP(FEATURE(ID_AA64MMFR0_TGRAN16_2), 0b0010); + fallthrough; + case SZ_64K: + val |= FIELD_PREP(FEATURE(ID_AA64MMFR0_TGRAN64_2), 0b0010); + break; + } + /* Cap PARange to 40bits */ + tmp = FIELD_GET(FEATURE(ID_AA64MMFR0_PARANGE), val); + if (tmp > 0b0010) { + val &= ~FEATURE(ID_AA64MMFR0_PARANGE); + val |= FIELD_PREP(FEATURE(ID_AA64MMFR0_PARANGE), 0b0010); + } + break; + + case SYS_ID_AA64MMFR1_EL1: + val &= (FEATURE(ID_AA64MMFR1_PAN) | + FEATURE(ID_AA64MMFR1_LOR) | + FEATURE(ID_AA64MMFR1_HPD) | + FEATURE(ID_AA64MMFR1_VHE) | + FEATURE(ID_AA64MMFR1_VMIDBITS)); + break; + + case SYS_ID_AA64MMFR2_EL1: + val &= ~(FEATURE(ID_AA64MMFR2_EVT) | + FEATURE(ID_AA64MMFR2_BBM) | + FEATURE(ID_AA64MMFR2_TTL) | + GENMASK_ULL(47, 44) | + FEATURE(ID_AA64MMFR2_ST) | + FEATURE(ID_AA64MMFR2_CCIDX) | + FEATURE(ID_AA64MMFR2_LVA)); + + /* Force TTL support */ + val |= FIELD_PREP(FEATURE(ID_AA64MMFR2_TTL), 0b0001); + break; + + case SYS_ID_AA64DFR0_EL1: + /* Only limited support for PMU, Debug, BPs and WPs */ + val &= (FEATURE(ID_AA64DFR0_PMSVER) | + FEATURE(ID_AA64DFR0_WRPS) | + FEATURE(ID_AA64DFR0_BRPS) | + FEATURE(ID_AA64DFR0_DEBUGVER)); + + /* Cap PMU to ARMv8.1 */ + tmp = FIELD_GET(FEATURE(ID_AA64DFR0_PMUVER), val); + if (tmp > 0b0100) { + val &= ~FEATURE(ID_AA64DFR0_PMUVER); + val |= FIELD_PREP(FEATURE(ID_AA64DFR0_PMUVER), 0b0100); + } + /* Cap Debug to ARMv8.1 */ + tmp = FIELD_GET(FEATURE(ID_AA64DFR0_DEBUGVER), val); + if (tmp > 0b0111) { + val &= ~FEATURE(ID_AA64DFR0_DEBUGVER); + val |= FIELD_PREP(FEATURE(ID_AA64DFR0_DEBUGVER), 0b0111); + } + break; + + default: + /* Unknown register, just wipe it clean */ + val = 0; + break; + } + + p->regval = val; +} diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 9deedd5a058f..19b33ccb61b8 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -1431,8 +1431,10 @@ static bool access_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r) { bool raz = sysreg_visible_as_raz(vcpu, r); + bool ret = __access_id_reg(vcpu, p, r, raz); - return __access_id_reg(vcpu, p, r, raz); + access_nested_id_reg(vcpu, p, r); + return ret; } static bool access_raz_id_reg(struct kvm_vcpu *vcpu, diff --git a/arch/arm64/kvm/sys_regs.h b/arch/arm64/kvm/sys_regs.h index cc0cc95a0280..d260c26b1834 100644 --- a/arch/arm64/kvm/sys_regs.h +++ b/arch/arm64/kvm/sys_regs.h @@ -201,4 +201,6 @@ const struct sys_reg_desc *find_reg_by_id(u64 id, CRn(sys_reg_CRn(reg)), CRm(sys_reg_CRm(reg)), \ Op2(sys_reg_Op2(reg)) +#define FEATURE(x) (GENMASK_ULL(x##_SHIFT + 3, x##_SHIFT)) + #endif /* __ARM64_KVM_SYS_REGS_LOCAL_H__ */ From patchwork Mon Nov 29 20:01:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645729 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52118C433F5 for ; Mon, 29 Nov 2021 21:42:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230129AbhK2Vps (ORCPT ); Mon, 29 Nov 2021 16:45:48 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48534 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229947AbhK2Vno (ORCPT ); Mon, 29 Nov 2021 16:43:44 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 43E70C08EB29 for ; Mon, 29 Nov 2021 12:05:32 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 0DD4DB815E0 for ; Mon, 29 Nov 2021 20:05:31 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B8EE3C53FAD; Mon, 29 Nov 2021 20:05:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216329; bh=usQKK1QDwEOQfu0+rZQRJzPmnngXGsisGciL9kGVuek=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=YVknn53E9Gy4gXhztyKswrGYO9rs/YUAhJspXakKIKEXmswoxyOlP5DGBTGrBCVcb sjfGNcIsDJT52U4idpWUu0gxa1wZHC53hJXKpmOvD7MC8dCIHsog4EYC1EF5h94U9n WpRZ2M6xb8TahhIosEztIiNxbxdDLJYdN9F9zncXeKSmOo7eyk18QlROdDK8uhnCIW btPHiFe+Yw+S+MqZYS98VTu24aAmegvzp4DkYMiHehWe0T8WMo/Bo9Ycmckq9mX9Vc wnMNfpUDzng7ElY/SmFwfkxVQrTbBIGnvScptk/FyIhRQwUuHUaE74sV7Ih+0yERCL wER6oA+TBZkzQ== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmqz-008gvR-3p; Mon, 29 Nov 2021 20:02:21 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 37/69] KVM: arm64: nv: Hide RAS from nested guests Date: Mon, 29 Nov 2021 20:01:18 +0000 Message-Id: <20211129200150.351436-38-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org We don't want to expose complicated features to guests until we have a good grasp on the basic CPU emulation. So let's pretend that RAS, doesn't exist in a nested guest. We already hide the feature bits, let's now make sure VDISR_EL1 will UNDEF. Signed-off-by: Marc Zyngier --- arch/arm64/kvm/sys_regs.c | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 19b33ccb61b8..3922ecc00869 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -2212,6 +2212,7 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_VBAR_EL2), access_rw, reset_val, VBAR_EL2, 0 }, { SYS_DESC(SYS_RVBAR_EL2), access_rw, reset_val, RVBAR_EL2, 0 }, { SYS_DESC(SYS_RMR_EL2), trap_undef }, + { SYS_DESC(SYS_VDISR_EL2), trap_undef }, { SYS_DESC(SYS_CONTEXTIDR_EL2), access_rw, reset_val, CONTEXTIDR_EL2, 0 }, { SYS_DESC(SYS_TPIDR_EL2), access_rw, reset_val, TPIDR_EL2, 0 }, From patchwork Mon Nov 29 20:01:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645617 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CFE6AC4332F for ; Mon, 29 Nov 2021 20:08:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232731AbhK2ULU (ORCPT ); Mon, 29 Nov 2021 15:11:20 -0500 Received: from ams.source.kernel.org ([145.40.68.75]:41984 "EHLO ams.source.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1379878AbhK2UJS (ORCPT ); Mon, 29 Nov 2021 15:09:18 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id C590CB81632 for ; Mon, 29 Nov 2021 20:05:59 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 35BDEC53FCF; Mon, 29 Nov 2021 20:05:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216358; bh=kF1ZILcXVrn3bd3UFJOtJ+0jVXpzh/pOYlqmtkgNu9k=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=b6vFfZjlY4x8ixasujWNjpSGRqR52lKqGqzNlMXIkToQZx2QXWdBR83/NtgvTqLfT 12iAAEDuOLuH+rL8vGBfJlTSVuaLOtHUG1hm4Y9LPiCZRzvfVGJD2YD/G0jtz63lDn hBCx7QmeBq7xRPBbBIWjW/iP+NeFT96WfL+uN8omDSaIYIcF7UPgJY5fMtfWYhm/CM 0JkIJ97iQgaxiqraW23+Yu+OtmseC/G3bbAoicSsHq2tUd018tKm65B5+KEGJl4Lz5 fdVeprxHzEkldCpr73CTt2zpKGbSRMH1bFh3GCEWV/L3Hu0j4p6jJcVVs81K8hrnUY Vf+7/AV5JEKsA== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmqz-008gvR-HR; Mon, 29 Nov 2021 20:02:21 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 38/69] KVM: arm64: nv: Support multiple nested Stage-2 mmu structures Date: Mon, 29 Nov 2021 20:01:19 +0000 Message-Id: <20211129200150.351436-39-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Add Stage-2 mmu data structures for virtual EL2 and for nested guests. We don't yet populate shadow Stage-2 page tables, but we now have a framework for getting to a shadow Stage-2 pgd. We allocate twice the number of vcpus as Stage-2 mmu structures because that's sufficient for each vcpu running two translation regimes without having to flush the Stage-2 page tables. Co-developed-by: Christoffer Dall Signed-off-by: Christoffer Dall Signed-off-by: Marc Zyngier Reviewed-by: Ganapatrao Kulkarni --- arch/arm64/include/asm/kvm_host.h | 29 +++++ arch/arm64/include/asm/kvm_mmu.h | 9 ++ arch/arm64/include/asm/kvm_nested.h | 7 + arch/arm64/kvm/arm.c | 16 ++- arch/arm64/kvm/mmu.c | 29 +++-- arch/arm64/kvm/nested.c | 195 ++++++++++++++++++++++++++++ 6 files changed, 275 insertions(+), 10 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 6a7b13edc5cb..00c3366129b8 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -102,14 +102,43 @@ struct kvm_s2_mmu { int __percpu *last_vcpu_ran; struct kvm_arch *arch; + + /* + * For a shadow stage-2 MMU, the virtual vttbr programmed by the guest + * hypervisor. Unused for kvm_arch->mmu. Set to 1 when the structure + * contains no valid information. + */ + u64 vttbr; + + /* true when this represents a nested context where virtual HCR_EL2.VM == 1 */ + bool nested_stage2_enabled; + + /* + * 0: Nobody is currently using this, check vttbr for validity + * >0: Somebody is actively using this. + */ + atomic_t refcnt; }; +static inline bool kvm_s2_mmu_valid(struct kvm_s2_mmu *mmu) +{ + return !(mmu->vttbr & 1); +} + struct kvm_arch_memory_slot { }; struct kvm_arch { struct kvm_s2_mmu mmu; + /* + * Stage 2 paging stage for VMs with nested virtual using a virtual + * VMID. + */ + struct kvm_s2_mmu *nested_mmus; + size_t nested_mmus_size; + int nested_mmus_next; + /* VTCR_EL2 value for this VM */ u64 vtcr; diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index c018c7b40761..7250594e3d68 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -116,6 +116,7 @@ alternative_cb_end #include #include #include +#include void kvm_update_va_mask(struct alt_instr *alt, __le32 *origptr, __le32 *updptr, int nr_inst); @@ -159,6 +160,7 @@ int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size, void **haddr); void free_hyp_pgds(void); +void kvm_unmap_stage2_range(struct kvm_s2_mmu *mmu, phys_addr_t start, u64 size); void stage2_unmap_vm(struct kvm *kvm); int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu); void kvm_free_stage2_pgd(struct kvm_s2_mmu *mmu); @@ -294,5 +296,12 @@ static inline struct kvm *kvm_s2_mmu_to_kvm(struct kvm_s2_mmu *mmu) { return container_of(mmu->arch, struct kvm, arch); } + +static inline u64 get_vmid(u64 vttbr) +{ + return (vttbr & VTTBR_VMID_MASK(kvm_get_vmid_bits())) >> + VTTBR_VMID_SHIFT; +} + #endif /* __ASSEMBLY__ */ #endif /* __ARM64_KVM_MMU_H__ */ diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index 026ddaad972c..473ecd1d60d0 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -61,6 +61,13 @@ static inline u64 translate_cnthctl_el2_to_cntkctl_el1(u64 cnthctl) (cnthctl & (CNTHCTL_EVNTI | CNTHCTL_EVNTDIR | CNTHCTL_EVNTEN))); } +extern void kvm_init_nested(struct kvm *kvm); +extern int kvm_vcpu_init_nested(struct kvm_vcpu *vcpu); +extern void kvm_init_nested_s2_mmu(struct kvm_s2_mmu *mmu); +extern struct kvm_s2_mmu *lookup_s2_mmu(struct kvm *kvm, u64 vttbr, u64 hcr); +extern void kvm_vcpu_load_hw_mmu(struct kvm_vcpu *vcpu); +extern void kvm_vcpu_put_hw_mmu(struct kvm_vcpu *vcpu); + int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe); extern bool __forward_traps(struct kvm_vcpu *vcpu, unsigned int reg, u64 control_bit); diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 68da54d58cd0..1dbf63319b99 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -37,6 +37,7 @@ #include #include #include +#include #include #include @@ -146,6 +147,8 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type) if (ret) return ret; + kvm_init_nested(kvm); + ret = create_hyp_mappings(kvm, kvm + 1, PAGE_HYP); if (ret) goto out_free_stage2_pgd; @@ -389,6 +392,9 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu) struct kvm_s2_mmu *mmu; int *last_ran; + if (nested_virt_in_use(vcpu)) + kvm_vcpu_load_hw_mmu(vcpu); + mmu = vcpu->arch.hw_mmu; last_ran = this_cpu_ptr(mmu->last_vcpu_ran); @@ -437,6 +443,9 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu) kvm_vgic_put(vcpu); kvm_vcpu_pmu_restore_host(vcpu); + if (nested_virt_in_use(vcpu)) + kvm_vcpu_put_hw_mmu(vcpu); + vcpu->cpu = -1; } @@ -1088,8 +1097,13 @@ static int kvm_vcpu_set_target(struct kvm_vcpu *vcpu, vcpu->arch.target = phys_target; + /* Prepare for nested if required */ + ret = kvm_vcpu_init_nested(vcpu); + /* Now we know what it is, we can reset it. */ - ret = kvm_reset_vcpu(vcpu); + if (!ret) + ret = kvm_reset_vcpu(vcpu); + if (ret) { vcpu->arch.target = -1; bitmap_zero(vcpu->arch.features, KVM_VCPU_MAX_FEATURES); diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 9eec548fccd1..ab1653b8e601 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -162,7 +162,7 @@ static void invalidate_icache_guest_page(void *va, size_t size) * does. */ /** - * unmap_stage2_range -- Clear stage2 page table entries to unmap a range + * __unmap_stage2_range -- Clear stage2 page table entries to unmap a range * @mmu: The KVM stage-2 MMU pointer * @start: The intermediate physical base address of the range to unmap * @size: The size of the area to unmap @@ -185,7 +185,7 @@ static void __unmap_stage2_range(struct kvm_s2_mmu *mmu, phys_addr_t start, u64 may_block)); } -static void unmap_stage2_range(struct kvm_s2_mmu *mmu, phys_addr_t start, u64 size) +void kvm_unmap_stage2_range(struct kvm_s2_mmu *mmu, phys_addr_t start, u64 size) { __unmap_stage2_range(mmu, start, size, true); } @@ -507,7 +507,20 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu) int cpu, err; struct kvm_pgtable *pgt; + /* + * If we already have our page tables in place, and that the + * MMU context is the canonical one, we have a bug somewhere, + * as this is only supposed to ever happen once per VM. + * + * Otherwise, we're building nested page tables, and that's + * probably because userspace called KVM_ARM_VCPU_INIT more + * than once on the same vcpu. Since that's actually legal, + * don't kick a fuss and leave gracefully. + */ if (mmu->pgt != NULL) { + if (&kvm->arch.mmu != mmu) + return 0; + kvm_err("kvm_arch already initialized?\n"); return -EINVAL; } @@ -533,6 +546,9 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu) mmu->pgt = pgt; mmu->pgd_phys = __pa(pgt->pgd); WRITE_ONCE(mmu->vmid.vmid_gen, 0); + + kvm_init_nested_s2_mmu(mmu); + return 0; out_destroy_pgtable: @@ -578,7 +594,7 @@ static void stage2_unmap_memslot(struct kvm *kvm, if (!(vma->vm_flags & VM_PFNMAP)) { gpa_t gpa = addr + (vm_start - memslot->userspace_addr); - unmap_stage2_range(&kvm->arch.mmu, gpa, vm_end - vm_start); + kvm_unmap_stage2_range(&kvm->arch.mmu, gpa, vm_end - vm_start); } hva = vm_end; } while (hva < reg_end); @@ -1556,11 +1572,6 @@ void kvm_arch_memslots_updated(struct kvm *kvm, u64 gen) { } -void kvm_arch_flush_shadow_all(struct kvm *kvm) -{ - kvm_free_stage2_pgd(&kvm->arch.mmu); -} - void kvm_arch_flush_shadow_memslot(struct kvm *kvm, struct kvm_memory_slot *slot) { @@ -1568,7 +1579,7 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm, phys_addr_t size = slot->npages << PAGE_SHIFT; spin_lock(&kvm->mmu_lock); - unmap_stage2_range(&kvm->arch.mmu, gpa, size); + kvm_unmap_stage2_range(&kvm->arch.mmu, gpa, size); spin_unlock(&kvm->mmu_lock); } diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c index 19b674983e13..b034a2343374 100644 --- a/arch/arm64/kvm/nested.c +++ b/arch/arm64/kvm/nested.c @@ -19,12 +19,189 @@ #include #include +#include #include +#include #include #include #include "sys_regs.h" +void kvm_init_nested(struct kvm *kvm) +{ + kvm->arch.nested_mmus = NULL; + kvm->arch.nested_mmus_size = 0; +} + +int kvm_vcpu_init_nested(struct kvm_vcpu *vcpu) +{ + struct kvm *kvm = vcpu->kvm; + struct kvm_s2_mmu *tmp; + int num_mmus; + int ret = -ENOMEM; + + if (!test_bit(KVM_ARM_VCPU_HAS_EL2, vcpu->arch.features)) + return 0; + + if (!cpus_have_final_cap(ARM64_HAS_NESTED_VIRT)) + return -EINVAL; + + mutex_lock(&kvm->lock); + + /* + * Let's treat memory allocation failures as benign: If we fail to + * allocate anything, return an error and keep the allocated array + * alive. Userspace may try to recover by intializing the vcpu + * again, and there is no reason to affect the whole VM for this. + */ + num_mmus = atomic_read(&kvm->online_vcpus) * 2; + tmp = krealloc(kvm->arch.nested_mmus, + num_mmus * sizeof(*kvm->arch.nested_mmus), + GFP_KERNEL_ACCOUNT | __GFP_ZERO); + if (tmp) { + /* + * If we went through a realocation, adjust the MMU + * back-pointers in the previously initialised + * pg_table structures. + */ + if (kvm->arch.nested_mmus != tmp) { + int i; + + for (i = 0; i < num_mmus - 2; i++) + tmp[i].pgt->mmu = &tmp[i]; + } + + if (kvm_init_stage2_mmu(kvm, &tmp[num_mmus - 1]) || + kvm_init_stage2_mmu(kvm, &tmp[num_mmus - 2])) { + kvm_free_stage2_pgd(&tmp[num_mmus - 1]); + kvm_free_stage2_pgd(&tmp[num_mmus - 2]); + } else { + kvm->arch.nested_mmus_size = num_mmus; + ret = 0; + } + + kvm->arch.nested_mmus = tmp; + } + + mutex_unlock(&kvm->lock); + return ret; +} + +/* Must be called with kvm->lock held */ +struct kvm_s2_mmu *lookup_s2_mmu(struct kvm *kvm, u64 vttbr, u64 hcr) +{ + bool nested_stage2_enabled = hcr & HCR_VM; + int i; + + /* Don't consider the CnP bit for the vttbr match */ + vttbr = vttbr & ~VTTBR_CNP_BIT; + + /* + * Two possibilities when looking up a S2 MMU context: + * + * - either S2 is enabled in the guest, and we need a context that + * is S2-enabled and matches the full VTTBR (VMID+BADDR), which + * makes it safe from a TLB conflict perspective (a broken guest + * won't be able to generate them), + * + * - or S2 is disabled, and we need a context that is S2-disabled + * and matches the VMID only, as all TLBs are tagged by VMID even + * if S2 translation is enabled. + */ + for (i = 0; i < kvm->arch.nested_mmus_size; i++) { + struct kvm_s2_mmu *mmu = &kvm->arch.nested_mmus[i]; + + if (!kvm_s2_mmu_valid(mmu)) + continue; + + if (nested_stage2_enabled && + mmu->nested_stage2_enabled && + vttbr == mmu->vttbr) + return mmu; + + if (!nested_stage2_enabled && + !mmu->nested_stage2_enabled && + get_vmid(vttbr) == get_vmid(mmu->vttbr)) + return mmu; + } + return NULL; +} + +static struct kvm_s2_mmu *get_s2_mmu_nested(struct kvm_vcpu *vcpu) +{ + struct kvm *kvm = vcpu->kvm; + u64 vttbr = vcpu_read_sys_reg(vcpu, VTTBR_EL2); + u64 hcr= vcpu_read_sys_reg(vcpu, HCR_EL2); + struct kvm_s2_mmu *s2_mmu; + int i; + + s2_mmu = lookup_s2_mmu(kvm, vttbr, hcr); + if (s2_mmu) + goto out; + + /* + * Make sure we don't always search from the same point, or we + * will always reuse a potentially active context, leaving + * free contexts unused. + */ + for (i = kvm->arch.nested_mmus_next; + i < (kvm->arch.nested_mmus_size + kvm->arch.nested_mmus_next); + i++) { + s2_mmu = &kvm->arch.nested_mmus[i % kvm->arch.nested_mmus_size]; + + if (atomic_read(&s2_mmu->refcnt) == 0) + break; + } + BUG_ON(atomic_read(&s2_mmu->refcnt)); /* We have struct MMUs to spare */ + + /* Set the scene for the next search */ + kvm->arch.nested_mmus_next = (i + 1) % kvm->arch.nested_mmus_size; + + if (kvm_s2_mmu_valid(s2_mmu)) { + /* Clear the old state */ + kvm_unmap_stage2_range(s2_mmu, 0, kvm_phys_size(kvm)); + if (s2_mmu->vmid.vmid_gen) + kvm_call_hyp(__kvm_tlb_flush_vmid, s2_mmu); + } + + /* + * The virtual VMID (modulo CnP) will be used as a key when matching + * an existing kvm_s2_mmu. + */ + s2_mmu->vttbr = vttbr & ~VTTBR_CNP_BIT; + s2_mmu->nested_stage2_enabled = hcr & HCR_VM; + +out: + atomic_inc(&s2_mmu->refcnt); + return s2_mmu; +} + +void kvm_init_nested_s2_mmu(struct kvm_s2_mmu *mmu) +{ + mmu->vttbr = 1; + mmu->nested_stage2_enabled = false; + atomic_set(&mmu->refcnt, 0); +} + +void kvm_vcpu_load_hw_mmu(struct kvm_vcpu *vcpu) +{ + if (is_hyp_ctxt(vcpu)) { + vcpu->arch.hw_mmu = &vcpu->kvm->arch.mmu; + } else { + spin_lock(&vcpu->kvm->mmu_lock); + vcpu->arch.hw_mmu = get_s2_mmu_nested(vcpu); + spin_unlock(&vcpu->kvm->mmu_lock); + } +} + +void kvm_vcpu_put_hw_mmu(struct kvm_vcpu *vcpu) +{ + if (vcpu->arch.hw_mmu != &vcpu->kvm->arch.mmu) { + atomic_dec(&vcpu->arch.hw_mmu->refcnt); + vcpu->arch.hw_mmu = NULL; + } +} + /* * Inject wfx to the virtual EL2 if this is not from the virtual EL2 and * the virtual HCR_EL2.TWX is set. Otherwise, let the host hypervisor @@ -43,6 +220,24 @@ int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe) return -EINVAL; } +void kvm_arch_flush_shadow_all(struct kvm *kvm) +{ + int i; + + for (i = 0; i < kvm->arch.nested_mmus_size; i++) { + struct kvm_s2_mmu *mmu = &kvm->arch.nested_mmus[i]; + + WARN_ON(atomic_read(&mmu->refcnt)); + + if (!atomic_read(&mmu->refcnt)) + kvm_free_stage2_pgd(mmu); + } + kfree(kvm->arch.nested_mmus); + kvm->arch.nested_mmus = NULL; + kvm->arch.nested_mmus_size = 0; + kvm_free_stage2_pgd(&kvm->arch.mmu); +} + /* * Our emulated CPU doesn't support all the possible features. For the * sake of simplicity (and probably mental sanity), wipe out a number From patchwork Mon Nov 29 20:01:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645755 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 55668C433EF for ; Mon, 29 Nov 2021 21:44:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232095AbhK2Vr5 (ORCPT ); Mon, 29 Nov 2021 16:47:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49446 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232048AbhK2Vpw (ORCPT ); Mon, 29 Nov 2021 16:45:52 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 86541C091D1C for ; Mon, 29 Nov 2021 12:06:06 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id D8AA4CE140C for ; Mon, 29 Nov 2021 20:06:05 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2D4CEC53FAD; Mon, 29 Nov 2021 20:06:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216364; bh=w8pDdRJcrwVDDb0ewTMpJ1QlCAiVNFEIVQTNnE1IWTc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=EqiG/jKUsdCS5u1CWRDR45FMRsdpYaKxSC3SuLbEhARISuUXRIHK441T/lUVkEDch 7a5pg6R+ZXPzQ3eQ/AL/OmM8YeRk+OkC2n8HAsljIco9DcBzC+v+kEIKAiLxDcBLhR ApD0psLeteODfYwvD+W4EI+Vr8en2MJ1TVk8lhtcjw22P3dr3N/JOU/w4igPeUXKNB FK8TcWSp8WQWzARlzwXVZawUA9yXqZoOeJVJjmdyl4qdyqgfjSPrYdyhQaw2hJm91Y jtH1jRdDHrf906kU02/ax83HJsxW73SMxfL7nUhLomDk6At6970EiYNCaLsc2ml4HV DX77+bloMiZ9Q== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmqz-008gvR-T5; Mon, 29 Nov 2021 20:02:22 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 39/69] KVM: arm64: nv: Implement nested Stage-2 page table walk logic Date: Mon, 29 Nov 2021 20:01:20 +0000 Message-Id: <20211129200150.351436-40-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Christoffer Dall Based on the pseudo-code in the ARM ARM, implement a stage 2 software page table walker. Signed-off-by: Christoffer Dall Signed-off-by: Jintack Lim [maz: heavily reworked for future ARMv8.4-TTL support] Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/esr.h | 1 + arch/arm64/include/asm/kvm_arm.h | 2 + arch/arm64/include/asm/kvm_nested.h | 13 ++ arch/arm64/kvm/nested.c | 267 ++++++++++++++++++++++++++++ 4 files changed, 283 insertions(+) diff --git a/arch/arm64/include/asm/esr.h b/arch/arm64/include/asm/esr.h index 6835e4231119..206bfb701791 100644 --- a/arch/arm64/include/asm/esr.h +++ b/arch/arm64/include/asm/esr.h @@ -130,6 +130,7 @@ #define ESR_ELx_CM (UL(1) << ESR_ELx_CM_SHIFT) /* ISS field definitions for exceptions taken in to Hyp */ +#define ESR_ELx_FSC_ADDRSZ (0x00) #define ESR_ELx_CV (UL(1) << 24) #define ESR_ELx_COND_SHIFT (20) #define ESR_ELx_COND_MASK (UL(0xF) << ESR_ELx_COND_SHIFT) diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h index b8a0d410035b..12f4bc8a805b 100644 --- a/arch/arm64/include/asm/kvm_arm.h +++ b/arch/arm64/include/asm/kvm_arm.h @@ -272,6 +272,8 @@ #define VTTBR_VMID_SHIFT (UL(48)) #define VTTBR_VMID_MASK(size) (_AT(u64, (1 << size) - 1) << VTTBR_VMID_SHIFT) +#define SCTLR_EE (UL(1) << 25) + /* Hyp System Trap Register */ #define HSTR_EL2_T(x) (1 << x) diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index 473ecd1d60d0..b784d7891851 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -68,6 +68,19 @@ extern struct kvm_s2_mmu *lookup_s2_mmu(struct kvm *kvm, u64 vttbr, u64 hcr); extern void kvm_vcpu_load_hw_mmu(struct kvm_vcpu *vcpu); extern void kvm_vcpu_put_hw_mmu(struct kvm_vcpu *vcpu); +struct kvm_s2_trans { + phys_addr_t output; + unsigned long block_size; + bool writable; + bool readable; + int level; + u32 esr; + u64 upper_attr; +}; + +extern int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa, + struct kvm_s2_trans *result); + int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe); extern bool __forward_traps(struct kvm_vcpu *vcpu, unsigned int reg, u64 control_bit); diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c index b034a2343374..01c5ee5f3545 100644 --- a/arch/arm64/kvm/nested.c +++ b/arch/arm64/kvm/nested.c @@ -87,6 +87,273 @@ int kvm_vcpu_init_nested(struct kvm_vcpu *vcpu) return ret; } +struct s2_walk_info { + int (*read_desc)(phys_addr_t pa, u64 *desc, void *data); + void *data; + u64 baddr; + unsigned int max_pa_bits; + unsigned int pgshift; + unsigned int pgsize; + unsigned int ps; + unsigned int sl; + unsigned int t0sz; + bool be; + bool el1_aarch32; +}; + +static unsigned int ps_to_output_size(unsigned int ps) +{ + switch (ps) { + case 0: return 32; + case 1: return 36; + case 2: return 40; + case 3: return 42; + case 4: return 44; + case 5: + default: + return 48; + } +} + +static u32 compute_fsc(int level, u32 fsc) +{ + return fsc | (level & 0x3); +} + +static int check_base_s2_limits(struct s2_walk_info *wi, + int level, int input_size, int stride) +{ + int start_size; + + /* Check translation limits */ + switch (wi->pgsize) { + case SZ_64K: + if (level == 0 || (level == 1 && wi->max_pa_bits <= 42)) + return -EFAULT; + break; + case SZ_16K: + if (level == 0 || (level == 1 && wi->max_pa_bits <= 40)) + return -EFAULT; + break; + case SZ_4K: + if (level < 0 || (level == 0 && wi->max_pa_bits <= 42)) + return -EFAULT; + break; + } + + /* Check input size limits */ + if (input_size > wi->max_pa_bits && + (!wi->el1_aarch32 || input_size > 40)) + return -EFAULT; + + /* Check number of entries in starting level table */ + start_size = input_size - ((3 - level) * stride + wi->pgshift); + if (start_size < 1 || start_size > stride + 4) + return -EFAULT; + + return 0; +} + +/* Check if output is within boundaries */ +static int check_output_size(struct s2_walk_info *wi, phys_addr_t output) +{ + unsigned int output_size = ps_to_output_size(wi->ps); + + if (output_size > wi->max_pa_bits) + output_size = wi->max_pa_bits; + + if (output_size != 48 && (output & GENMASK_ULL(47, output_size))) + return -1; + + return 0; +} + +/* + * This is essentially a C-version of the pseudo code from the ARM ARM + * AArch64.TranslationTableWalk function. I strongly recommend looking at + * that pseudocode in trying to understand this. + * + * Must be called with the kvm->srcu read lock held + */ +static int walk_nested_s2_pgd(phys_addr_t ipa, + struct s2_walk_info *wi, struct kvm_s2_trans *out) +{ + int first_block_level, level, stride, input_size, base_lower_bound; + phys_addr_t base_addr; + unsigned int addr_top, addr_bottom; + u64 desc; /* page table entry */ + int ret; + phys_addr_t paddr; + + switch (wi->pgsize) { + case SZ_64K: + case SZ_16K: + level = 3 - wi->sl; + first_block_level = 2; + break; + case SZ_4K: + level = 2 - wi->sl; + first_block_level = 1; + break; + default: + /* GCC is braindead */ + unreachable(); + } + + stride = wi->pgshift - 3; + input_size = 64 - wi->t0sz; + if (input_size > 48 || input_size < 25) + return -EFAULT; + + ret = check_base_s2_limits(wi, level, input_size, stride); + if (WARN_ON(ret)) + return ret; + + base_lower_bound = 3 + input_size - ((3 - level) * stride + + wi->pgshift); + base_addr = wi->baddr & GENMASK_ULL(47, base_lower_bound); + + if (check_output_size(wi, base_addr)) { + out->esr = compute_fsc(level, ESR_ELx_FSC_ADDRSZ); + return 1; + } + + addr_top = input_size - 1; + + while (1) { + phys_addr_t index; + + addr_bottom = (3 - level) * stride + wi->pgshift; + index = (ipa & GENMASK_ULL(addr_top, addr_bottom)) + >> (addr_bottom - 3); + + paddr = base_addr | index; + ret = wi->read_desc(paddr, &desc, wi->data); + if (ret < 0) + return ret; + + /* + * Handle reversedescriptors if endianness differs between the + * host and the guest hypervisor. + */ + if (wi->be) + desc = be64_to_cpu(desc); + else + desc = le64_to_cpu(desc); + + /* Check for valid descriptor at this point */ + if (!(desc & 1) || ((desc & 3) == 1 && level == 3)) { + out->esr = compute_fsc(level, ESR_ELx_FSC_FAULT); + out->upper_attr = desc; + return 1; + } + + /* We're at the final level or block translation level */ + if ((desc & 3) == 1 || level == 3) + break; + + if (check_output_size(wi, desc)) { + out->esr = compute_fsc(level, ESR_ELx_FSC_ADDRSZ); + out->upper_attr = desc; + return 1; + } + + base_addr = desc & GENMASK_ULL(47, wi->pgshift); + + level += 1; + addr_top = addr_bottom - 1; + } + + if (level < first_block_level) { + out->esr = compute_fsc(level, ESR_ELx_FSC_FAULT); + out->upper_attr = desc; + return 1; + } + + /* + * We don't use the contiguous bit in the stage-2 ptes, so skip check + * for misprogramming of the contiguous bit. + */ + + if (check_output_size(wi, desc)) { + out->esr = compute_fsc(level, ESR_ELx_FSC_ADDRSZ); + out->upper_attr = desc; + return 1; + } + + if (!(desc & BIT(10))) { + out->esr = compute_fsc(level, ESR_ELx_FSC_ACCESS); + out->upper_attr = desc; + return 1; + } + + /* Calculate and return the result */ + paddr = (desc & GENMASK_ULL(47, addr_bottom)) | + (ipa & GENMASK_ULL(addr_bottom - 1, 0)); + out->output = paddr; + out->block_size = 1UL << ((3 - level) * stride + wi->pgshift); + out->readable = desc & (0b01 << 6); + out->writable = desc & (0b10 << 6); + out->level = level; + out->upper_attr = desc & GENMASK_ULL(63, 52); + return 0; +} + +static int read_guest_s2_desc(phys_addr_t pa, u64 *desc, void *data) +{ + struct kvm_vcpu *vcpu = data; + + return kvm_read_guest(vcpu->kvm, pa, desc, sizeof(*desc)); +} + +static void vtcr_to_walk_info(u64 vtcr, struct s2_walk_info *wi) +{ + wi->t0sz = vtcr & TCR_EL2_T0SZ_MASK; + + switch (vtcr & VTCR_EL2_TG0_MASK) { + case VTCR_EL2_TG0_4K: + wi->pgshift = 12; break; + case VTCR_EL2_TG0_16K: + wi->pgshift = 14; break; + case VTCR_EL2_TG0_64K: + default: + wi->pgshift = 16; break; + } + + wi->pgsize = 1UL << wi->pgshift; + wi->ps = (vtcr & VTCR_EL2_PS_MASK) >> VTCR_EL2_PS_SHIFT; + wi->sl = (vtcr & VTCR_EL2_SL0_MASK) >> VTCR_EL2_SL0_SHIFT; + wi->max_pa_bits = VTCR_EL2_IPA(vtcr); +} + +int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa, + struct kvm_s2_trans *result) +{ + u64 vtcr = vcpu_read_sys_reg(vcpu, VTCR_EL2); + struct s2_walk_info wi; + int ret; + + result->esr = 0; + + if (!nested_virt_in_use(vcpu)) + return 0; + + wi.read_desc = read_guest_s2_desc; + wi.data = vcpu; + wi.baddr = vcpu_read_sys_reg(vcpu, VTTBR_EL2); + + vtcr_to_walk_info(vtcr, &wi); + + wi.be = vcpu_read_sys_reg(vcpu, SCTLR_EL2) & SCTLR_EE; + wi.el1_aarch32 = vcpu_mode_is_32bit(vcpu); + + ret = walk_nested_s2_pgd(gipa, &wi, result); + if (ret) + result->esr |= (kvm_vcpu_get_esr(vcpu) & ~ESR_ELx_FSC); + + return ret; +} + /* Must be called with kvm->lock held */ struct kvm_s2_mmu *lookup_s2_mmu(struct kvm *kvm, u64 vttbr, u64 hcr) { From patchwork Mon Nov 29 20:01:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645899 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A848EC433F5 for ; Mon, 29 Nov 2021 22:20:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233058AbhK2WX2 (ORCPT ); Mon, 29 Nov 2021 17:23:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56568 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233140AbhK2WVa (ORCPT ); Mon, 29 Nov 2021 17:21:30 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3D8A2C08EB3F for ; Mon, 29 Nov 2021 12:06:55 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id B3E2AB815FE for ; Mon, 29 Nov 2021 20:06:54 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 54438C53FAD; Mon, 29 Nov 2021 20:06:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216413; bh=TdVLgnojYlVMKndPn3E1EWu7ajgwz9M5a0Jdi+5Uhro=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ehRNwG8EPSQfVYvzi48WUD68iKqa4AZJEAa3lV7VnC4+ssTT2lpgagcdLdaN9Qr0i Wt+FID92aRVEdPzUE2CQmrkU0y4CZc7fs0IaZ5UCtRasg6wCauYb/UtzxMYzLvgAnY SMO8+M3NA3UHZspAK2dEPNqJMRm/BS179j79q6EoC0EuAHc3gMX9azczP69frfnIRm wH+fQkFPB+F7JmwQCMjGkGfPUWVymKz0Soxo/hCG1L1JXXHd3h50jL8t3YOk3fFV1+ M/Zygp5QtVmp/DZR093cfp3XoOB6wxMstpwp0AiL85QDVzP/SwecRB+LBCzE/nQkZL rT/7EbwWdhUFg== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmr0-008gvR-8D; Mon, 29 Nov 2021 20:02:22 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 40/69] KVM: arm64: nv: Handle shadow stage 2 page faults Date: Mon, 29 Nov 2021 20:01:21 +0000 Message-Id: <20211129200150.351436-41-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org If we are faulting on a shadow stage 2 translation, we first walk the guest hypervisor's stage 2 page table to see if it has a mapping. If not, we inject a stage 2 page fault to the virtual EL2. Otherwise, we create a mapping in the shadow stage 2 page table. Note that we have to deal with two IPAs when we got a shadow stage 2 page fault. One is the address we faulted on, and is in the L2 guest phys space. The other is from the guest stage-2 page table walk, and is in the L1 guest phys space. To differentiate them, we rename variables so that fault_ipa is used for the former and ipa is used for the latter. Co-developed-by: Christoffer Dall Co-developed-by: Jintack Lim Signed-off-by: Christoffer Dall Signed-off-by: Jintack Lim [maz: rewrote this multiple times...] Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_emulate.h | 6 ++ arch/arm64/include/asm/kvm_nested.h | 18 +++++ arch/arm64/kvm/mmu.c | 102 +++++++++++++++++++++++---- arch/arm64/kvm/nested.c | 48 +++++++++++++ 4 files changed, 159 insertions(+), 15 deletions(-) diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h index 105fb0d4a366..94d8814b7c8b 100644 --- a/arch/arm64/include/asm/kvm_emulate.h +++ b/arch/arm64/include/asm/kvm_emulate.h @@ -602,4 +602,10 @@ static inline bool vcpu_has_feature(struct kvm_vcpu *vcpu, int feature) return test_bit(feature, vcpu->arch.features); } +static inline bool kvm_is_shadow_s2_fault(struct kvm_vcpu *vcpu) +{ + return (vcpu->arch.hw_mmu != &vcpu->kvm->arch.mmu && + vcpu->arch.hw_mmu->nested_stage2_enabled); +} + #endif /* __ARM64_KVM_EMULATE_H__ */ diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index b784d7891851..4f93a5dab183 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -78,9 +78,27 @@ struct kvm_s2_trans { u64 upper_attr; }; +static inline phys_addr_t kvm_s2_trans_output(struct kvm_s2_trans *trans) +{ + return trans->output; +} + +static inline unsigned long kvm_s2_trans_size(struct kvm_s2_trans *trans) +{ + return trans->block_size; +} + +static inline u32 kvm_s2_trans_esr(struct kvm_s2_trans *trans) +{ + return trans->esr; +} + extern int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa, struct kvm_s2_trans *result); +extern int kvm_s2_handle_perm_fault(struct kvm_vcpu *vcpu, + struct kvm_s2_trans *trans); +extern int kvm_inject_s2_fault(struct kvm_vcpu *vcpu, u64 esr_el2); int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe); extern bool __forward_traps(struct kvm_vcpu *vcpu, unsigned int reg, u64 control_bit); diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index ab1653b8e601..fc3613d3f503 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -845,7 +845,7 @@ static bool fault_supports_stage2_huge_mapping(struct kvm_memory_slot *memslot, static unsigned long transparent_hugepage_adjust(struct kvm *kvm, struct kvm_memory_slot *memslot, unsigned long hva, kvm_pfn_t *pfnp, - phys_addr_t *ipap) + phys_addr_t *ipap, phys_addr_t *fault_ipap) { kvm_pfn_t pfn = *pfnp; @@ -874,6 +874,7 @@ transparent_hugepage_adjust(struct kvm *kvm, struct kvm_memory_slot *memslot, * to PG_head and switch the pfn from a tail page to the head * page accordingly. */ + *fault_ipap &= PMD_MASK; *ipap &= PMD_MASK; kvm_release_pfn_clean(pfn); pfn &= ~(PTRS_PER_PMD - 1); @@ -956,15 +957,17 @@ static int sanitise_mte_tags(struct kvm *kvm, kvm_pfn_t pfn, } static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, - struct kvm_memory_slot *memslot, unsigned long hva, - unsigned long fault_status) + struct kvm_s2_trans *nested, + struct kvm_memory_slot *memslot, + unsigned long hva, unsigned long fault_status) { int ret = 0; - bool write_fault, writable, force_pte = false; + bool write_fault, writable; bool exec_fault; bool device = false; bool shared; unsigned long mmu_seq; + phys_addr_t ipa = fault_ipa; struct kvm *kvm = vcpu->kvm; struct kvm_mmu_memory_cache *memcache = &vcpu->arch.mmu_page_cache; struct vm_area_struct *vma; @@ -976,6 +979,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, unsigned long vma_pagesize, fault_granule; enum kvm_pgtable_prot prot = KVM_PGTABLE_PROT_R; struct kvm_pgtable *pgt; + unsigned long max_map_size = PUD_SIZE; fault_granule = 1UL << ARM64_HW_PGTABLE_LEVEL_SHIFT(fault_level); write_fault = kvm_is_write_fault(vcpu); @@ -1004,7 +1008,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, * memslots. */ if (logging_active) { - force_pte = true; + max_map_size = vma_pagesize = PAGE_SIZE; vma_shift = PAGE_SHIFT; } else { vma_shift = get_vma_page_shift(vma, hva); @@ -1028,7 +1032,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, fallthrough; case CONT_PTE_SHIFT: vma_shift = PAGE_SHIFT; - force_pte = true; + max_map_size = PAGE_SIZE; fallthrough; case PAGE_SHIFT: break; @@ -1037,10 +1041,25 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, } vma_pagesize = 1UL << vma_shift; + + if (kvm_is_shadow_s2_fault(vcpu)) { + ipa = kvm_s2_trans_output(nested); + + /* + * If we're about to create a shadow stage 2 entry, then we + * can only create a block mapping if the guest stage 2 page + * table uses at least as big a mapping. + */ + max_map_size = min(kvm_s2_trans_size(nested), max_map_size); + } + + vma_pagesize = min(vma_pagesize, max_map_size); + if (vma_pagesize == PMD_SIZE || vma_pagesize == PUD_SIZE) fault_ipa &= ~(vma_pagesize - 1); - gfn = fault_ipa >> PAGE_SHIFT; + gfn = ipa >> PAGE_SHIFT; + mmap_read_unlock(current->mm); /* @@ -1113,12 +1132,14 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, * If we are not forced to use page mapping, check if we are * backed by a THP and thus use block mapping if possible. */ - if (vma_pagesize == PAGE_SIZE && !(force_pte || device)) { + if (vma_pagesize == PAGE_SIZE && + !(max_map_size == PAGE_SIZE || device)) { if (fault_status == FSC_PERM && fault_granule > PAGE_SIZE) vma_pagesize = fault_granule; else vma_pagesize = transparent_hugepage_adjust(kvm, memslot, hva, &pfn, + &ipa, &fault_ipa); } @@ -1202,8 +1223,10 @@ static void handle_access_fault(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa) int kvm_handle_guest_abort(struct kvm_vcpu *vcpu) { unsigned long fault_status; - phys_addr_t fault_ipa; + phys_addr_t fault_ipa; /* The address we faulted on */ + phys_addr_t ipa; /* Always the IPA in the L1 guest phys space */ struct kvm_memory_slot *memslot; + struct kvm_s2_trans nested_trans; unsigned long hva; bool is_iabt, write_fault, writable; gfn_t gfn; @@ -1211,7 +1234,7 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu) fault_status = kvm_vcpu_trap_get_fault_type(vcpu); - fault_ipa = kvm_vcpu_get_fault_ipa(vcpu); + ipa = fault_ipa = kvm_vcpu_get_fault_ipa(vcpu); is_iabt = kvm_vcpu_trap_is_iabt(vcpu); /* Synchronous External Abort? */ @@ -1232,6 +1255,12 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu) /* Check the stage-2 fault is trans. fault or write fault */ if (fault_status != FSC_FAULT && fault_status != FSC_PERM && fault_status != FSC_ACCESS) { + /* + * We must never see an address size fault on shadow stage 2 + * page table walk, because we would have injected an addr + * size fault when we walked the nested s2 page and not + * create the shadow entry. + */ kvm_err("Unsupported FSC: EC=%#x xFSC=%#lx ESR_EL2=%#lx\n", kvm_vcpu_trap_get_class(vcpu), (unsigned long)kvm_vcpu_trap_get_fault(vcpu), @@ -1241,7 +1270,49 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu) idx = srcu_read_lock(&vcpu->kvm->srcu); - gfn = fault_ipa >> PAGE_SHIFT; + /* + * We may have faulted on a shadow stage 2 page table if we are + * running a nested guest. In this case, we have to resolve the L2 + * IPA to the L1 IPA first, before knowing what kind of memory should + * back the L1 IPA. + * + * If the shadow stage 2 page table walk faults, then we simply inject + * this to the guest and carry on. + */ + if (kvm_is_shadow_s2_fault(vcpu)) { + u32 esr; + + ret = kvm_walk_nested_s2(vcpu, fault_ipa, &nested_trans); + esr = kvm_s2_trans_esr(&nested_trans); + if (esr) + kvm_inject_s2_fault(vcpu, esr); + if (ret) + goto out_unlock; + + ret = kvm_s2_handle_perm_fault(vcpu, &nested_trans); + esr = kvm_s2_trans_esr(&nested_trans); + if (esr) + kvm_inject_s2_fault(vcpu, esr); + if (ret) + goto out_unlock; + + ipa = kvm_s2_trans_output(&nested_trans); + } else { + nested_trans = (struct kvm_s2_trans) { + /* + * Default to RWX so that we don't filter + * anything while evaluating the permissions. + */ + .writable = true, + .readable = true, + .upper_attr = 0, + .output = ipa, + .level = kvm_vcpu_trap_get_fault_level(vcpu), + .esr = kvm_vcpu_get_esr(vcpu), + }; + } + + gfn = ipa >> PAGE_SHIFT; memslot = gfn_to_memslot(vcpu->kvm, gfn); hva = gfn_to_hva_memslot_prot(memslot, gfn, &writable); write_fault = kvm_is_write_fault(vcpu); @@ -1285,13 +1356,13 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu) * faulting VA. This is always 12 bits, irrespective * of the page size. */ - fault_ipa |= kvm_vcpu_get_hfar(vcpu) & ((1 << 12) - 1); - ret = io_mem_abort(vcpu, fault_ipa); + ipa |= kvm_vcpu_get_hfar(vcpu) & ((1 << 12) - 1); + ret = io_mem_abort(vcpu, ipa); goto out_unlock; } /* Userspace should not be able to register out-of-bounds IPAs */ - VM_BUG_ON(fault_ipa >= kvm_phys_size(vcpu->kvm)); + VM_BUG_ON(ipa >= kvm_phys_size(vcpu->kvm)); if (fault_status == FSC_ACCESS) { handle_access_fault(vcpu, fault_ipa); @@ -1299,7 +1370,8 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu) goto out_unlock; } - ret = user_mem_abort(vcpu, fault_ipa, memslot, hva, fault_status); + ret = user_mem_abort(vcpu, fault_ipa, &nested_trans, + memslot, hva, fault_status); if (ret == 0) ret = 1; out: diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c index 01c5ee5f3545..c34e96955e04 100644 --- a/arch/arm64/kvm/nested.c +++ b/arch/arm64/kvm/nested.c @@ -120,6 +120,15 @@ static u32 compute_fsc(int level, u32 fsc) return fsc | (level & 0x3); } +static int esr_s2_fault(struct kvm_vcpu *vcpu, int level, u32 fsc) +{ + u32 esr; + + esr = kvm_vcpu_get_esr(vcpu) & ~ESR_ELx_FSC; + esr |= compute_fsc(level, fsc); + return esr; +} + static int check_base_s2_limits(struct s2_walk_info *wi, int level, int input_size, int stride) { @@ -469,6 +478,45 @@ void kvm_vcpu_put_hw_mmu(struct kvm_vcpu *vcpu) } } +/* + * Returns non-zero if permission fault is handled by injecting it to the next + * level hypervisor. + */ +int kvm_s2_handle_perm_fault(struct kvm_vcpu *vcpu, struct kvm_s2_trans *trans) +{ + unsigned long fault_status = kvm_vcpu_trap_get_fault_type(vcpu); + bool forward_fault = false; + + trans->esr = 0; + + if (fault_status != FSC_PERM) + return 0; + + if (kvm_vcpu_trap_is_iabt(vcpu)) { + forward_fault = (trans->upper_attr & BIT(54)); + } else { + bool write_fault = kvm_is_write_fault(vcpu); + + forward_fault = ((write_fault && !trans->writable) || + (!write_fault && !trans->readable)); + } + + if (forward_fault) { + trans->esr = esr_s2_fault(vcpu, trans->level, ESR_ELx_FSC_PERM); + return 1; + } + + return 0; +} + +int kvm_inject_s2_fault(struct kvm_vcpu *vcpu, u64 esr_el2) +{ + vcpu_write_sys_reg(vcpu, vcpu->arch.fault.far_el2, FAR_EL2); + vcpu_write_sys_reg(vcpu, vcpu->arch.fault.hpfar_el2, HPFAR_EL2); + + return kvm_inject_nested_sync(vcpu, esr_el2); +} + /* * Inject wfx to the virtual EL2 if this is not from the virtual EL2 and * the virtual HCR_EL2.TWX is set. Otherwise, let the host hypervisor From patchwork Mon Nov 29 20:01:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645611 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 38330C433EF for ; Mon, 29 Nov 2021 20:07:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231905AbhK2UKu (ORCPT ); Mon, 29 Nov 2021 15:10:50 -0500 Received: from sin.source.kernel.org ([145.40.73.55]:50198 "EHLO sin.source.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348333AbhK2UIs (ORCPT ); Mon, 29 Nov 2021 15:08:48 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 5A8D9CE13DF for ; Mon, 29 Nov 2021 20:05:29 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 83125C53FCD; Mon, 29 Nov 2021 20:05:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216327; bh=2ThkwV02KLfELC/usIrKEsYVmRRm/yHVr4F9Cx+Dngw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=G0uH7D1z0AFrAjWMCNMBiB7zMbl+T0kAkO07VKHKPSVBx5qmHNHAga3ggCW3vBCr1 YuFMMLXcJatV32Y+2Xxx8INGb3vHG91VUDPBQpf+Gq/pAlaYFuyRQ1nI6ex5H/wQj2 8mRm6LpeXQ22opHOZ31qz9CCf4uYF/eGf5c+yv83SdYQwLhAZmlI7Z4RFA/IADoc05 0NvkaK/XmLg/DI2C7OLwDB5ZPgT+iYkz910fmk4xuxrn5NuAIVYWKiwxRssVOuUOye VIWHX6XCpNQEkESiD6wHrpmEjDa/Lfh/damAI3goQUkbSagJqCE8m5OGJGdINV7b23 yIad/gVFy8KyA== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmr0-008gvR-J1; Mon, 29 Nov 2021 20:02:22 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 41/69] KVM: arm64: nv: Restrict S2 RD/WR permissions to match the guest's Date: Mon, 29 Nov 2021 20:01:22 +0000 Message-Id: <20211129200150.351436-42-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org When mapping a page in a shadow stage-2, special care must be taken not to be more permissive than the guest is (writable or readable page when the guest hasn't set that permission). Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_nested.h | 15 +++++++++++++++ arch/arm64/kvm/mmu.c | 14 +++++++++++++- arch/arm64/kvm/nested.c | 2 +- 3 files changed, 29 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index 4f93a5dab183..3f3d8e10bd99 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -93,6 +93,21 @@ static inline u32 kvm_s2_trans_esr(struct kvm_s2_trans *trans) return trans->esr; } +static inline bool kvm_s2_trans_readable(struct kvm_s2_trans *trans) +{ + return trans->readable; +} + +static inline bool kvm_s2_trans_writable(struct kvm_s2_trans *trans) +{ + return trans->writable; +} + +static inline bool kvm_s2_trans_executable(struct kvm_s2_trans *trans) +{ + return !(trans->upper_attr & BIT(54)); +} + extern int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa, struct kvm_s2_trans *result); diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index fc3613d3f503..d63f60f95abc 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -1123,6 +1123,17 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, if (exec_fault && device) return -ENOEXEC; + /* + * Potentially reduce shadow S2 permissions to match the guest's own + * S2. For exec faults, we'd only reach this point if the guest + * actually allowed it (see kvm_s2_handle_perm_fault). + */ + if (kvm_is_shadow_s2_fault(vcpu)) { + writable &= kvm_s2_trans_writable(nested); + if (!kvm_s2_trans_readable(nested)) + prot &= ~KVM_PGTABLE_PROT_R; + } + spin_lock(&kvm->mmu_lock); pgt = vcpu->arch.hw_mmu->pgt; if (mmu_notifier_retry(kvm, mmu_seq)) @@ -1161,7 +1172,8 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, if (device) prot |= KVM_PGTABLE_PROT_DEVICE; - else if (cpus_have_const_cap(ARM64_HAS_CACHE_DIC)) + else if (cpus_have_const_cap(ARM64_HAS_CACHE_DIC) && + kvm_s2_trans_executable(nested)) prot |= KVM_PGTABLE_PROT_X; /* diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c index c34e96955e04..fb748f2b9e87 100644 --- a/arch/arm64/kvm/nested.c +++ b/arch/arm64/kvm/nested.c @@ -493,7 +493,7 @@ int kvm_s2_handle_perm_fault(struct kvm_vcpu *vcpu, struct kvm_s2_trans *trans) return 0; if (kvm_vcpu_trap_is_iabt(vcpu)) { - forward_fault = (trans->upper_attr & BIT(54)); + forward_fault = !kvm_s2_trans_executable(trans); } else { bool write_fault = kvm_is_write_fault(vcpu); From patchwork Mon Nov 29 20:01:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645783 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DF665C43219 for ; Mon, 29 Nov 2021 21:46:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232591AbhK2Vtz (ORCPT ); Mon, 29 Nov 2021 16:49:55 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49448 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231949AbhK2Vrw (ORCPT ); Mon, 29 Nov 2021 16:47:52 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 30EF3C08EB48 for ; Mon, 29 Nov 2021 12:07:04 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id C7463B8159E for ; Mon, 29 Nov 2021 20:07:02 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5C8D6C53FC7; Mon, 29 Nov 2021 20:07:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216421; bh=FGszoEsrzsIxwm4QSb55LvccHTocLIQ9JXX1nNcUxE4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=mSfmGy0sMPGIB5fNuLUc6YPgfg87FxUcMQ0+c4ijVhQrSEPM/gr4mdpfoJl0jh82Y GCvjYArYijH9LXX6eDeOcMQze72me/vpMy9vDktaD+C60l45sGbY1fWSzSmgqxQNSo WoCChxkJWZ8UjU7cFscPvLN9mv67G2XpvaPzYKptMGBgEgr3o+MGiWI5G4fteVpG8R Fp8TQwh/iIrXZ8wnxeYOJHIZ9QdfnyVNDLIHIbEmRHO+wV/LBfpcrmxAr9fF7Il3Ci ycdn9oMGHpm29p3IC55IWJMI1MntmFQLQ8bkjew06y56Q5mqqNYnMoN6NW/rOfDoUm /iyUfQlmG8HUw== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmr0-008gvR-UJ; Mon, 29 Nov 2021 20:02:23 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 42/69] KVM: arm64: nv: Unmap/flush shadow stage 2 page tables Date: Mon, 29 Nov 2021 20:01:23 +0000 Message-Id: <20211129200150.351436-43-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Christoffer Dall Unmap/flush shadow stage 2 page tables for the nested VMs as well as the stage 2 page table for the guest hypervisor. Note: A bunch of the code in mmu.c relating to MMU notifiers is currently dealt with in an extremely abrupt way, for example by clearing out an entire shadow stage-2 table. This will be handled in a more efficient way using the reverse mapping feature in a later version of the patch series. Signed-off-by: Christoffer Dall Signed-off-by: Jintack Lim Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_mmu.h | 3 +++ arch/arm64/include/asm/kvm_nested.h | 3 +++ arch/arm64/kvm/mmu.c | 31 +++++++++++++++++++---- arch/arm64/kvm/nested.c | 39 +++++++++++++++++++++++++++++ 4 files changed, 71 insertions(+), 5 deletions(-) diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index 7250594e3d68..63b2eb5f4bcb 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -158,6 +158,8 @@ int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size, void __iomem **haddr); int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size, void **haddr); +void kvm_stage2_flush_range(struct kvm_s2_mmu *mmu, + phys_addr_t addr, phys_addr_t end); void free_hyp_pgds(void); void kvm_unmap_stage2_range(struct kvm_s2_mmu *mmu, phys_addr_t start, u64 size); @@ -166,6 +168,7 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu); void kvm_free_stage2_pgd(struct kvm_s2_mmu *mmu); int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa, phys_addr_t pa, unsigned long size, bool writable); +void kvm_stage2_wp_range(struct kvm_s2_mmu *mmu, phys_addr_t addr, phys_addr_t end); int kvm_handle_guest_abort(struct kvm_vcpu *vcpu); diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index 3f3d8e10bd99..2987806850f0 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -114,6 +114,9 @@ extern int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa, extern int kvm_s2_handle_perm_fault(struct kvm_vcpu *vcpu, struct kvm_s2_trans *trans); extern int kvm_inject_s2_fault(struct kvm_vcpu *vcpu, u64 esr_el2); +extern void kvm_nested_s2_wp(struct kvm *kvm); +extern void kvm_nested_s2_clear(struct kvm *kvm); +extern void kvm_nested_s2_flush(struct kvm *kvm); int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe); extern bool __forward_traps(struct kvm_vcpu *vcpu, unsigned int reg, u64 control_bit); diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index d63f60f95abc..a9e743f39bfd 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -190,13 +190,20 @@ void kvm_unmap_stage2_range(struct kvm_s2_mmu *mmu, phys_addr_t start, u64 size) __unmap_stage2_range(mmu, start, size, true); } +void kvm_stage2_flush_range(struct kvm_s2_mmu *mmu, + phys_addr_t addr, phys_addr_t end) +{ + stage2_apply_range_resched(kvm_s2_mmu_to_kvm(mmu), addr, end, kvm_pgtable_stage2_flush); +} + static void stage2_flush_memslot(struct kvm *kvm, struct kvm_memory_slot *memslot) { phys_addr_t addr = memslot->base_gfn << PAGE_SHIFT; phys_addr_t end = addr + PAGE_SIZE * memslot->npages; + struct kvm_s2_mmu *mmu = &kvm->arch.mmu; - stage2_apply_range_resched(kvm, addr, end, kvm_pgtable_stage2_flush); + kvm_stage2_flush_range(mmu, addr, end); } /** @@ -219,6 +226,8 @@ static void stage2_flush_vm(struct kvm *kvm) kvm_for_each_memslot(memslot, slots) stage2_flush_memslot(kvm, memslot); + kvm_nested_s2_flush(kvm); + spin_unlock(&kvm->mmu_lock); srcu_read_unlock(&kvm->srcu, idx); } @@ -621,6 +630,8 @@ void stage2_unmap_vm(struct kvm *kvm) kvm_for_each_memslot(memslot, slots) stage2_unmap_memslot(kvm, memslot); + kvm_nested_s2_clear(kvm); + spin_unlock(&kvm->mmu_lock); mmap_read_unlock(current->mm); srcu_read_unlock(&kvm->srcu, idx); @@ -690,12 +701,12 @@ int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa, } /** - * stage2_wp_range() - write protect stage2 memory region range + * kvm_stage2_wp_range() - write protect stage2 memory region range * @mmu: The KVM stage-2 MMU pointer * @addr: Start address of range * @end: End address of range */ -static void stage2_wp_range(struct kvm_s2_mmu *mmu, phys_addr_t addr, phys_addr_t end) +void kvm_stage2_wp_range(struct kvm_s2_mmu *mmu, phys_addr_t addr, phys_addr_t end) { struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu); stage2_apply_range_resched(kvm, addr, end, kvm_pgtable_stage2_wrprotect); @@ -727,7 +738,8 @@ static void kvm_mmu_wp_memory_region(struct kvm *kvm, int slot) end = (memslot->base_gfn + memslot->npages) << PAGE_SHIFT; spin_lock(&kvm->mmu_lock); - stage2_wp_range(&kvm->arch.mmu, start, end); + kvm_stage2_wp_range(&kvm->arch.mmu, start, end); + kvm_nested_s2_wp(kvm); spin_unlock(&kvm->mmu_lock); kvm_flush_remote_tlbs(kvm); } @@ -751,7 +763,7 @@ static void kvm_mmu_write_protect_pt_masked(struct kvm *kvm, phys_addr_t start = (base_gfn + __ffs(mask)) << PAGE_SHIFT; phys_addr_t end = (base_gfn + __fls(mask) + 1) << PAGE_SHIFT; - stage2_wp_range(&kvm->arch.mmu, start, end); + kvm_stage2_wp_range(&kvm->arch.mmu, start, end); } /* @@ -766,6 +778,7 @@ void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm, gfn_t gfn_offset, unsigned long mask) { kvm_mmu_write_protect_pt_masked(kvm, slot, gfn_offset, mask); + kvm_nested_s2_wp(kvm); } static void kvm_send_hwpoison_signal(unsigned long address, short lsb) @@ -1405,6 +1418,7 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range) (range->end - range->start) << PAGE_SHIFT, range->may_block); + kvm_nested_s2_clear(kvm); return false; } @@ -1436,6 +1450,7 @@ bool kvm_set_spte_gfn(struct kvm *kvm, struct kvm_gfn_range *range) PAGE_SIZE, __pfn_to_phys(pfn), KVM_PGTABLE_PROT_R, NULL); + kvm_nested_s2_clear(kvm); return false; } @@ -1454,6 +1469,11 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) range->start << PAGE_SHIFT); pte = __pte(kpte); return pte_valid(pte) && pte_young(pte); + + /* + * TODO: Handle nested_mmu structures here using the reverse mapping in + * a later version of patch series. + */ } bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) @@ -1664,6 +1684,7 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm, spin_lock(&kvm->mmu_lock); kvm_unmap_stage2_range(&kvm->arch.mmu, gpa, size); + kvm_nested_s2_clear(kvm); spin_unlock(&kvm->mmu_lock); } diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c index fb748f2b9e87..b7e10f6178b9 100644 --- a/arch/arm64/kvm/nested.c +++ b/arch/arm64/kvm/nested.c @@ -517,6 +517,45 @@ int kvm_inject_s2_fault(struct kvm_vcpu *vcpu, u64 esr_el2) return kvm_inject_nested_sync(vcpu, esr_el2); } +/* expects kvm->mmu_lock to be held */ +void kvm_nested_s2_wp(struct kvm *kvm) +{ + int i; + + for (i = 0; i < kvm->arch.nested_mmus_size; i++) { + struct kvm_s2_mmu *mmu = &kvm->arch.nested_mmus[i]; + + if (kvm_s2_mmu_valid(mmu)) + kvm_stage2_wp_range(mmu, 0, kvm_phys_size(kvm)); + } +} + +/* expects kvm->mmu_lock to be held */ +void kvm_nested_s2_clear(struct kvm *kvm) +{ + int i; + + for (i = 0; i < kvm->arch.nested_mmus_size; i++) { + struct kvm_s2_mmu *mmu = &kvm->arch.nested_mmus[i]; + + if (kvm_s2_mmu_valid(mmu)) + kvm_unmap_stage2_range(mmu, 0, kvm_phys_size(kvm)); + } +} + +/* expects kvm->mmu_lock to be held */ +void kvm_nested_s2_flush(struct kvm *kvm) +{ + int i; + + for (i = 0; i < kvm->arch.nested_mmus_size; i++) { + struct kvm_s2_mmu *mmu = &kvm->arch.nested_mmus[i]; + + if (kvm_s2_mmu_valid(mmu)) + kvm_stage2_flush_range(mmu, 0, kvm_phys_size(kvm)); + } +} + /* * Inject wfx to the virtual EL2 if this is not from the virtual EL2 and * the virtual HCR_EL2.TWX is set. Otherwise, let the host hypervisor From patchwork Mon Nov 29 20:01:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645785 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4DF5CC433FE for ; Mon, 29 Nov 2021 21:46:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232604AbhK2Vtz (ORCPT ); Mon, 29 Nov 2021 16:49:55 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49446 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232335AbhK2Vrw (ORCPT ); Mon, 29 Nov 2021 16:47:52 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AA83BC091D2A for ; Mon, 29 Nov 2021 12:07:06 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 26AD6CE1415 for ; Mon, 29 Nov 2021 20:07:05 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 966E2C53FCD; Mon, 29 Nov 2021 20:07:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216423; bh=8Q0UoKsQ1/LEa+qH82lBUBclRYVG/wZJY3gepjlHcJ4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ho/ZRfPVLvxO15GsT0DOBx3SZPHVNE0BpXvmGZGKuILKZ7dwWlw3I9ptBk+wN6vss XPp6o3CpuXNbQZ3VAaWPWuOprWFQ0lvctAwZTV79ul4eVZbFmslSZK2PNLZvxYBAyK wR4Cvm/xhGbAkkDFGnbX9J5CnD3E464ByC61DraioaLpdBM7T0LkrKqCAcrBEiHsqg HDFIBj1lDDAtG4xMAy0ZnciK1lJNZUf11Bjhs9HbwoNWr1uVMAnEZXTciET1ubojkB egb1P3qun4F3Po6ez+5rPSKuEqTXFxHEBFD6f5iMGeTLMJtPjKJE1EylsDPSkwA8rV 6RvsaaqlRvmHA== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmr2-008gvR-1t; Mon, 29 Nov 2021 20:02:24 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 43/69] KVM: arm64: nv: Introduce sys_reg_desc.forward_trap Date: Mon, 29 Nov 2021 20:01:24 +0000 Message-Id: <20211129200150.351436-44-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Jintack Lim This introduces a function prototype to determine if we need to forward system instruction traps to the virtual EL2. The implementation of forward_trap functions for each system instruction will be added in later patches. Signed-off-by: Jintack Lim Signed-off-by: Marc Zyngier --- arch/arm64/kvm/sys_regs.c | 8 ++++++++ arch/arm64/kvm/sys_regs.h | 6 ++++++ 2 files changed, 14 insertions(+) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 3922ecc00869..77f07f960d84 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -2598,6 +2598,14 @@ static void perform_access(struct kvm_vcpu *vcpu, */ BUG_ON(!r->access); + /* + * Forward this trap to the virtual EL2 if the guest hypervisor has + * configured to trap the current instruction. + */ + if (nested_virt_in_use(vcpu) && r->forward_trap + && unlikely(r->forward_trap(vcpu))) + return; + /* Skip instruction if instructed so */ if (likely(r->access(vcpu, params, r))) kvm_incr_pc(vcpu); diff --git a/arch/arm64/kvm/sys_regs.h b/arch/arm64/kvm/sys_regs.h index d260c26b1834..267b2f37444c 100644 --- a/arch/arm64/kvm/sys_regs.h +++ b/arch/arm64/kvm/sys_regs.h @@ -72,6 +72,12 @@ struct sys_reg_desc { int (*set_user)(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd, const struct kvm_one_reg *reg, void __user *uaddr); + /* + * Forward the trap to the virtual EL2 if the guest hypervisor has + * configured to trap the current instruction. + */ + bool (*forward_trap)(struct kvm_vcpu *vcpu); + /* Return mask of REG_* runtime visibility overrides */ unsigned int (*visibility)(const struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd); From patchwork Mon Nov 29 20:01:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645675 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7ACC5C433EF for ; Mon, 29 Nov 2021 20:09:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234254AbhK2UNG (ORCPT ); Mon, 29 Nov 2021 15:13:06 -0500 Received: from ams.source.kernel.org ([145.40.68.75]:43218 "EHLO ams.source.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232424AbhK2ULE (ORCPT ); Mon, 29 Nov 2021 15:11:04 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id ECAACB81636 for ; Mon, 29 Nov 2021 20:07:45 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id AAF2BC53FD1; Mon, 29 Nov 2021 20:07:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216464; bh=EYeh0Lg5aZN74mfocLytDlC5zUmfjAuH0WowocTwWvc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=uj6n4AqFMtrk71GZ8maLMN3czt9ScQKmmEei66FkAqWF3eXhNOG2O7VzTW5AMnDb0 MClBwmZIA5dUR+TeVIW+Ik2fqyY8h5XCXOS9chCKklklxKMs6lKipsfxxNRHzwErph kdrLi6Nopgs2TqN7dSGD5zZo+V6QHSMoov85gfRy91ECTSg5sLYQns3bTizRly+l7D DLbKyW1jzyVvl4NpX3x2cx3gqruVaZHJGsyeGt8tSKK/dJ1wYtqsP46n5Z1KBt6nPb faM7pmZ+hMgF64s1b9K9FHZNz/YJGuikifKIw/PPV0/OapGw4F77304u7wy7TodtsJ eytUUfRltWWtA== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmr2-008gvR-Bj; Mon, 29 Nov 2021 20:02:24 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 44/69] KVM: arm64: nv: Set a handler for the system instruction traps Date: Mon, 29 Nov 2021 20:01:25 +0000 Message-Id: <20211129200150.351436-45-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Jintack Lim When HCR.NV bit is set, execution of the EL2 translation regime address aranslation instructions and TLB maintenance instructions are trapped to EL2. In addition, execution of the EL1 translation regime address aranslation instructions and TLB maintenance instructions that are only accessible from EL2 and above are trapped to EL2. In these cases, ESR_EL2.EC will be set to 0x18. Rework the system instruction emulation framework to handle potentially all system instruction traps other than MSR/MRS instructions. Those system instructions would be AT and TLBI instructions controlled by HCR_EL2.NV, AT, and TTLB bits. Signed-off-by: Jintack Lim [maz: squashed two patches together, redispatched various bits around] Signed-off-by: Marc Zyngier Reviewed-by: Ganapatrao Kulkarni --- arch/arm64/include/asm/kvm_host.h | 4 +-- arch/arm64/kvm/handle_exit.c | 2 +- arch/arm64/kvm/sys_regs.c | 48 +++++++++++++++++++++++++------ 3 files changed, 42 insertions(+), 12 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 00c3366129b8..35f3d7939484 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -411,7 +411,7 @@ struct kvm_vcpu_arch { /* * Guest registers we preserve during guest debugging. * - * These shadow registers are updated by the kvm_handle_sys_reg + * These shadow registers are updated by the kvm_handle_sys * trap handler if the guest accesses or updates them while we * are using guest debug. */ @@ -729,7 +729,7 @@ int kvm_handle_cp14_32(struct kvm_vcpu *vcpu); int kvm_handle_cp14_64(struct kvm_vcpu *vcpu); int kvm_handle_cp15_32(struct kvm_vcpu *vcpu); int kvm_handle_cp15_64(struct kvm_vcpu *vcpu); -int kvm_handle_sys_reg(struct kvm_vcpu *vcpu); +int kvm_handle_sys(struct kvm_vcpu *vcpu); void kvm_reset_sys_regs(struct kvm_vcpu *vcpu); diff --git a/arch/arm64/kvm/handle_exit.c b/arch/arm64/kvm/handle_exit.c index 6ff709c124d0..8b3b758e674e 100644 --- a/arch/arm64/kvm/handle_exit.c +++ b/arch/arm64/kvm/handle_exit.c @@ -234,7 +234,7 @@ static exit_handle_fn arm_exit_handlers[] = { [ESR_ELx_EC_SMC32] = handle_smc, [ESR_ELx_EC_HVC64] = handle_hvc, [ESR_ELx_EC_SMC64] = handle_smc, - [ESR_ELx_EC_SYS64] = kvm_handle_sys_reg, + [ESR_ELx_EC_SYS64] = kvm_handle_sys, [ESR_ELx_EC_SVE] = handle_sve, [ESR_ELx_EC_ERET] = kvm_handle_eret, [ESR_ELx_EC_IABT_LOW] = kvm_handle_guest_abort, diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 77f07f960d84..222f8c6f1d7e 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -1770,10 +1770,6 @@ static bool access_spsr_el2(struct kvm_vcpu *vcpu, * more demanding guest... */ static const struct sys_reg_desc sys_reg_descs[] = { - { SYS_DESC(SYS_DC_ISW), access_dcsw }, - { SYS_DESC(SYS_DC_CSW), access_dcsw }, - { SYS_DESC(SYS_DC_CISW), access_dcsw }, - DBG_BCR_BVR_WCR_WVR_EL1(0), DBG_BCR_BVR_WCR_WVR_EL1(1), { SYS_DESC(SYS_MDCCINT_EL1), trap_debug_regs, reset_val, MDCCINT_EL1, 0 }, @@ -2240,6 +2236,14 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_SP_EL2), NULL, reset_unknown, SP_EL2 }, }; +#define SYS_INSN_TO_DESC(insn, access_fn, forward_fn) \ + { SYS_DESC((insn)), (access_fn), NULL, 0, 0, NULL, NULL, (forward_fn) } +static struct sys_reg_desc sys_insn_descs[] = { + { SYS_DESC(SYS_DC_ISW), access_dcsw }, + { SYS_DESC(SYS_DC_CSW), access_dcsw }, + { SYS_DESC(SYS_DC_CISW), access_dcsw }, +}; + static bool trap_dbgdidr(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) @@ -2794,6 +2798,24 @@ static int emulate_sys_reg(struct kvm_vcpu *vcpu, return 1; } +static int emulate_sys_instr(struct kvm_vcpu *vcpu, struct sys_reg_params *p) +{ + const struct sys_reg_desc *r; + + /* Search from the system instruction table. */ + r = find_reg(p, sys_insn_descs, ARRAY_SIZE(sys_insn_descs)); + + if (likely(r)) { + perform_access(vcpu, p, r); + } else { + kvm_err("Unsupported guest sys instruction at: %lx\n", + *vcpu_pc(vcpu)); + print_sys_reg_instr(p); + kvm_inject_undefined(vcpu); + } + return 1; +} + /** * kvm_reset_sys_regs - sets system registers to reset value * @vcpu: The VCPU pointer @@ -2811,10 +2833,11 @@ void kvm_reset_sys_regs(struct kvm_vcpu *vcpu) } /** - * kvm_handle_sys_reg -- handles a mrs/msr trap on a guest sys_reg access + * kvm_handle_sys-- handles a system instruction or mrs/msr instruction trap + on a guest execution * @vcpu: The VCPU pointer */ -int kvm_handle_sys_reg(struct kvm_vcpu *vcpu) +int kvm_handle_sys(struct kvm_vcpu *vcpu) { struct sys_reg_params params; unsigned long esr = kvm_vcpu_get_esr(vcpu); @@ -2826,10 +2849,16 @@ int kvm_handle_sys_reg(struct kvm_vcpu *vcpu) params = esr_sys64_to_params(esr); params.regval = vcpu_get_reg(vcpu, Rt); - ret = emulate_sys_reg(vcpu, ¶ms); + if (params.Op0 == 1) { + /* System instructions */ + ret = emulate_sys_instr(vcpu, ¶ms); + } else { + /* MRS/MSR instructions */ + ret = emulate_sys_reg(vcpu, ¶ms); + if (!params.is_write) + vcpu_set_reg(vcpu, Rt, params.regval); + } - if (!params.is_write) - vcpu_set_reg(vcpu, Rt, params.regval); return ret; } @@ -3245,6 +3274,7 @@ void kvm_sys_reg_table_init(void) BUG_ON(check_sysreg_table(cp15_regs, ARRAY_SIZE(cp15_regs), true)); BUG_ON(check_sysreg_table(cp15_64_regs, ARRAY_SIZE(cp15_64_regs), true)); BUG_ON(check_sysreg_table(invariant_sys_regs, ARRAY_SIZE(invariant_sys_regs), false)); + BUG_ON(check_sysreg_table(sys_insn_descs, ARRAY_SIZE(sys_insn_descs), false)); /* We abuse the reset function to overwrite the table itself. */ for (i = 0; i < ARRAY_SIZE(invariant_sys_regs); i++) From patchwork Mon Nov 29 20:01:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645775 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1CEDFC433F5 for ; Mon, 29 Nov 2021 21:46:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232494AbhK2Vtu (ORCPT ); Mon, 29 Nov 2021 16:49:50 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48974 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231933AbhK2Vrq (ORCPT ); Mon, 29 Nov 2021 16:47:46 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F2B66C091D21 for ; Mon, 29 Nov 2021 12:06:48 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 9C1FDB815EC for ; Mon, 29 Nov 2021 20:06:47 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 25942C53FCF; Mon, 29 Nov 2021 20:06:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216406; bh=v8PNXU8eqe6gtDxTIrJTfRk0WR0eAamzcjambcxVTW0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=oBkvJaedPzSo1gv2aKXraLhvZBV+0E4xSwSDPhtbixhlFW9uZXm+WyFLET2EOFqg3 jy6Zds39aMOq0s1x7GKrvawExrJe+O56DJJ2T6nGBmx4xkPCZLXVoxztcoV6WtXkFM 05DQl8lJP9ktmHy4Sq7Uzfg5g3HJSWpg2pHGZ7SDmBDE0n6pdoD/U/yH6u7+vbCh2O Rz4v+959Y+3UkJe9p+ZUdw77G/gGHp8F39KKaU0Vtv9cAYs0c8HPDOJ2ukDGiLJbcM CeKrTVHx29PlD8ULCAB966oxg6F2f/A1jASYAs8S9/LmyJkyW6SW62xHtk3/+/kKzx 2pAPkz32Mh3iA== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmr2-008gvR-LX; Mon, 29 Nov 2021 20:02:24 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 45/69] KVM: arm64: nv: Trap and emulate AT instructions from virtual EL2 Date: Mon, 29 Nov 2021 20:01:26 +0000 Message-Id: <20211129200150.351436-46-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Jintack Lim When supporting nested virtualization a guest hypervisor executing AT instructions must be trapped and emulated by the host hypervisor, because untrapped AT instructions operating on S1E1 will use the wrong translation regieme (the one used to emulate virtual EL2 in EL1 instead of virtual EL1) and AT instructions operating on S12 will not work from EL1. This patch does several things. 1. List and define all AT system instructions to emulate and document the emulation design. 2. Implement AT instruction handling logic in EL2. This will be used to emulate AT instructions executed in the virtual EL2. AT instruction emulation works by loading the proper processor context, which depends on the trapped instruction and the virtual HCR_EL2, to the EL1 virtual memory control registers and executing AT instructions. Note that ctxt->hw_sys_regs is expected to have the proper processor context before calling the handling function(__kvm_at_insn) implemented in this patch. 4. Emulate AT S1E[01] instructions by issuing the same instructions in EL2. We set the physical EL1 registers, NV and NV1 bits as described in the AT instruction emulation overview. 5. Emulate AT A12E[01] instructions in two steps: First, do the stage-1 translation by reusing the existing AT emulation functions. Second, do the stage-2 translation by walking the guest hypervisor's stage-2 page table in software. Record the translation result to PAR_EL1. 6. Emulate AT S1E2 instructions by issuing the corresponding S1E1 instructions in EL2. We set the physical EL1 registers and the HCR_EL2 register as described in the AT instruction emulation overview. 7. Forward system instruction traps to the virtual EL2 if the corresponding virtual AT bit is set in the virtual HCR_EL2. [ Much logic above has been reworked by Marc Zyngier ] Signed-off-by: Jintack Lim Signed-off-by: Marc Zyngier Signed-off-by: Christoffer Dall --- arch/arm64/include/asm/kvm_arm.h | 2 + arch/arm64/include/asm/kvm_asm.h | 2 + arch/arm64/include/asm/sysreg.h | 17 +++ arch/arm64/kvm/Makefile | 2 +- arch/arm64/kvm/at.c | 231 +++++++++++++++++++++++++++++++ arch/arm64/kvm/hyp/vhe/switch.c | 13 +- arch/arm64/kvm/sys_regs.c | 201 ++++++++++++++++++++++++++- 7 files changed, 463 insertions(+), 5 deletions(-) create mode 100644 arch/arm64/kvm/at.c diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h index 12f4bc8a805b..b603466803d2 100644 --- a/arch/arm64/include/asm/kvm_arm.h +++ b/arch/arm64/include/asm/kvm_arm.h @@ -20,6 +20,7 @@ #define HCR_AMVOFFEN (UL(1) << 51) #define HCR_FIEN (UL(1) << 47) #define HCR_FWB (UL(1) << 46) +#define HCR_AT (UL(1) << 44) #define HCR_NV1 (UL(1) << 43) #define HCR_NV (UL(1) << 42) #define HCR_API (UL(1) << 41) @@ -118,6 +119,7 @@ #define VTCR_EL2_TG0_16K TCR_TG0_16K #define VTCR_EL2_TG0_64K TCR_TG0_64K #define VTCR_EL2_SH0_MASK TCR_SH0_MASK +#define VTCR_EL2_SH0_SHIFT TCR_SH0_SHIFT #define VTCR_EL2_SH0_INNER TCR_SH0_INNER #define VTCR_EL2_ORGN0_MASK TCR_ORGN0_MASK #define VTCR_EL2_ORGN0_WBWA TCR_ORGN0_WBWA diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h index 50d5e4de244c..abea7e164400 100644 --- a/arch/arm64/include/asm/kvm_asm.h +++ b/arch/arm64/include/asm/kvm_asm.h @@ -207,6 +207,8 @@ extern void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa, extern void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu); extern void __kvm_timer_set_cntvoff(u64 cntvoff); +extern void __kvm_at_s1e01(struct kvm_vcpu *vcpu, u32 op, u64 vaddr); +extern void __kvm_at_s1e2(struct kvm_vcpu *vcpu, u32 op, u64 vaddr); extern int __kvm_vcpu_run(struct kvm_vcpu *vcpu); diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h index c77fe5401826..b673adeb190f 100644 --- a/arch/arm64/include/asm/sysreg.h +++ b/arch/arm64/include/asm/sysreg.h @@ -657,6 +657,23 @@ #define SYS_SP_EL2 sys_reg(3, 6, 4, 1, 0) +/* AT instructions */ +#define AT_Op0 1 +#define AT_CRn 7 + +#define OP_AT_S1E1R sys_insn(AT_Op0, 0, AT_CRn, 8, 0) +#define OP_AT_S1E1W sys_insn(AT_Op0, 0, AT_CRn, 8, 1) +#define OP_AT_S1E0R sys_insn(AT_Op0, 0, AT_CRn, 8, 2) +#define OP_AT_S1E0W sys_insn(AT_Op0, 0, AT_CRn, 8, 3) +#define OP_AT_S1E1RP sys_insn(AT_Op0, 0, AT_CRn, 9, 0) +#define OP_AT_S1E1WP sys_insn(AT_Op0, 0, AT_CRn, 9, 1) +#define OP_AT_S1E2R sys_insn(AT_Op0, 4, AT_CRn, 8, 0) +#define OP_AT_S1E2W sys_insn(AT_Op0, 4, AT_CRn, 8, 1) +#define OP_AT_S12E1R sys_insn(AT_Op0, 4, AT_CRn, 8, 4) +#define OP_AT_S12E1W sys_insn(AT_Op0, 4, AT_CRn, 8, 5) +#define OP_AT_S12E0R sys_insn(AT_Op0, 4, AT_CRn, 8, 6) +#define OP_AT_S12E0W sys_insn(AT_Op0, 4, AT_CRn, 8, 7) + /* Common SCTLR_ELx flags. */ #define SCTLR_ELx_DSSBS (BIT(44)) #define SCTLR_ELx_ATA (BIT(43)) diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile index 923ee13728b4..8e19da1a07e1 100644 --- a/arch/arm64/kvm/Makefile +++ b/arch/arm64/kvm/Makefile @@ -16,7 +16,7 @@ kvm-y := $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/eventfd.o \ inject_fault.o va_layout.o handle_exit.o \ guest.o debug.o reset.o sys_regs.o \ vgic-sys-reg-v3.o fpsimd.o pmu.o \ - arch_timer.o trng.o emulate-nested.o nested.o \ + arch_timer.o trng.o emulate-nested.o nested.o at.o \ vgic/vgic.o vgic/vgic-init.o \ vgic/vgic-irqfd.o vgic/vgic-v2.o \ vgic/vgic-v3.o vgic/vgic-v4.o \ diff --git a/arch/arm64/kvm/at.c b/arch/arm64/kvm/at.c new file mode 100644 index 000000000000..c345ef98ca1e --- /dev/null +++ b/arch/arm64/kvm/at.c @@ -0,0 +1,231 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2017 - Linaro Ltd + * Author: Jintack Lim + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see . + */ + +#include +#include + +struct mmu_config { + u64 ttbr0; + u64 ttbr1; + u64 tcr; + u64 sctlr; + u64 vttbr; + u64 vtcr; + u64 hcr; +}; + +static void __mmu_config_save(struct mmu_config *config) +{ + config->ttbr0 = read_sysreg_el1(SYS_TTBR0); + config->ttbr1 = read_sysreg_el1(SYS_TTBR1); + config->tcr = read_sysreg_el1(SYS_TCR); + config->sctlr = read_sysreg_el1(SYS_SCTLR); + config->vttbr = read_sysreg(vttbr_el2); + config->vtcr = read_sysreg(vtcr_el2); + config->hcr = read_sysreg(hcr_el2); +} + +static void __mmu_config_restore(struct mmu_config *config) +{ + write_sysreg_el1(config->ttbr0, SYS_TTBR0); + write_sysreg_el1(config->ttbr1, SYS_TTBR1); + write_sysreg_el1(config->tcr, SYS_TCR); + write_sysreg_el1(config->sctlr, SYS_SCTLR); + write_sysreg(config->vttbr, vttbr_el2); + write_sysreg(config->vtcr, vtcr_el2); + write_sysreg(config->hcr, hcr_el2); + + isb(); +} + +void __kvm_at_s1e01(struct kvm_vcpu *vcpu, u32 op, u64 vaddr) +{ + struct kvm_cpu_context *ctxt = &vcpu->arch.ctxt; + struct mmu_config config; + struct kvm_s2_mmu *mmu; + + spin_lock(&vcpu->kvm->mmu_lock); + + /* + * If HCR_EL2.{E2H,TGE} == {1,1}, the MMU context is already + * the right one (as we trapped from vEL2). + */ + if (vcpu_el2_e2h_is_set(vcpu) && vcpu_el2_tge_is_set(vcpu)) + goto skip_mmu_switch; + + /* + * FIXME: Obtaining the S2 MMU for a guest guest is horribly + * racy, and we may not find it (evicted by another vcpu, for + * example). + */ + mmu = lookup_s2_mmu(vcpu->kvm, + vcpu_read_sys_reg(vcpu, VTTBR_EL2), + vcpu_read_sys_reg(vcpu, HCR_EL2)); + + if (WARN_ON(!mmu)) + goto out; + + /* We've trapped, so everything is live on the CPU. */ + __mmu_config_save(&config); + + write_sysreg_el1(ctxt_sys_reg(ctxt, TTBR0_EL1), SYS_TTBR0); + write_sysreg_el1(ctxt_sys_reg(ctxt, TTBR1_EL1), SYS_TTBR1); + write_sysreg_el1(ctxt_sys_reg(ctxt, TCR_EL1), SYS_TCR); + write_sysreg_el1(ctxt_sys_reg(ctxt, SCTLR_EL1), SYS_SCTLR); + write_sysreg(kvm_get_vttbr(mmu), vttbr_el2); + /* + * REVISIT: do we need anything from the guest's VTCR_EL2? If + * looks like keeping the hosts configuration is the right + * thing to do at this stage (and we could avoid save/restore + * it. Keep the host's version for now. + */ + write_sysreg((config.hcr & ~HCR_TGE) | HCR_VM, hcr_el2); + + isb(); + +skip_mmu_switch: + + switch (op) { + case OP_AT_S1E1R: + case OP_AT_S1E1RP: + asm volatile("at s1e1r, %0" : : "r" (vaddr)); + break; + case OP_AT_S1E1W: + case OP_AT_S1E1WP: + asm volatile("at s1e1w, %0" : : "r" (vaddr)); + break; + case OP_AT_S1E0R: + asm volatile("at s1e0r, %0" : : "r" (vaddr)); + break; + case OP_AT_S1E0W: + asm volatile("at s1e0w, %0" : : "r" (vaddr)); + break; + default: + WARN_ON_ONCE(1); + break; + } + + isb(); + + ctxt_sys_reg(ctxt, PAR_EL1) = read_sysreg(par_el1); + + /* + * Failed? let's leave the building now. + * + * FIXME: how about a failed translation because the shadow S2 + * wasn't populated? We may need to perform a SW PTW, + * populating our shadow S2 and retry the instruction. + */ + if (ctxt_sys_reg(ctxt, PAR_EL1) & 1) + goto nopan; + + /* No PAN? No problem. */ + if (!(*vcpu_cpsr(vcpu) & PSR_PAN_BIT)) + goto nopan; + + /* + * For PAN-involved AT operations, perform the same + * translation, using EL0 this time. + */ + switch (op) { + case OP_AT_S1E1RP: + asm volatile("at s1e0r, %0" : : "r" (vaddr)); + break; + case OP_AT_S1E1WP: + asm volatile("at s1e0w, %0" : : "r" (vaddr)); + break; + default: + goto nopan; + } + + /* + * If the EL0 translation has succeeded, we need to pretend + * the AT operation has failed, as the PAN setting forbids + * such a translation. + * + * FIXME: we hardcode a Level-3 permission fault. We really + * should return the real fault level. + */ + if (!(read_sysreg(par_el1) & 1)) + ctxt_sys_reg(ctxt, PAR_EL1) = 0x1f; + +nopan: + if (!(vcpu_el2_e2h_is_set(vcpu) && vcpu_el2_tge_is_set(vcpu))) + __mmu_config_restore(&config); + +out: + spin_unlock(&vcpu->kvm->mmu_lock); +} + +void __kvm_at_s1e2(struct kvm_vcpu *vcpu, u32 op, u64 vaddr) +{ + struct kvm_cpu_context *ctxt = &vcpu->arch.ctxt; + struct mmu_config config; + struct kvm_s2_mmu *mmu; + u64 val; + + spin_lock(&vcpu->kvm->mmu_lock); + + mmu = &vcpu->kvm->arch.mmu; + + /* We've trapped, so everything is live on the CPU. */ + __mmu_config_save(&config); + + if (vcpu_el2_e2h_is_set(vcpu)) { + write_sysreg_el1(ctxt_sys_reg(ctxt, TTBR0_EL2), SYS_TTBR0); + write_sysreg_el1(ctxt_sys_reg(ctxt, TTBR1_EL2), SYS_TTBR1); + write_sysreg_el1(ctxt_sys_reg(ctxt, TCR_EL2), SYS_TCR); + write_sysreg_el1(ctxt_sys_reg(ctxt, SCTLR_EL2), SYS_SCTLR); + + val = config.hcr; + } else { + write_sysreg_el1(ctxt_sys_reg(ctxt, TTBR0_EL2), SYS_TTBR0); + val = translate_tcr_el2_to_tcr_el1(ctxt_sys_reg(ctxt, TCR_EL2)); + write_sysreg_el1(val, SYS_TCR); + val = translate_sctlr_el2_to_sctlr_el1(ctxt_sys_reg(ctxt, SCTLR_EL2)); + write_sysreg_el1(val, SYS_SCTLR); + + val = config.hcr | HCR_NV | HCR_NV1; + } + + write_sysreg(kvm_get_vttbr(mmu), vttbr_el2); + /* FIXME: write S2 MMU VTCR_EL2? */ + write_sysreg((val & ~HCR_TGE) | HCR_VM, hcr_el2); + + isb(); + + switch (op) { + case OP_AT_S1E2R: + asm volatile("at s1e1r, %0" : : "r" (vaddr)); + break; + case OP_AT_S1E2W: + asm volatile("at s1e1w, %0" : : "r" (vaddr)); + break; + default: + WARN_ON_ONCE(1); + break; + } + + isb(); + + /* FIXME: handle failed translation due to shadow S2 */ + ctxt_sys_reg(ctxt, PAR_EL1) = read_sysreg(par_el1); + + __mmu_config_restore(&config); + spin_unlock(&vcpu->kvm->mmu_lock); +} diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c index da80c969e623..657a74bab73c 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -42,9 +42,10 @@ static void __activate_traps(struct kvm_vcpu *vcpu) if (!vcpu_el2_e2h_is_set(vcpu)) { /* * For a guest hypervisor on v8.0, trap and emulate - * the EL1 virtual memory control register accesses. + * the EL1 virtual memory control register accesses + * as well as the AT S1 operations. */ - hcr |= HCR_TVM | HCR_TRVM | HCR_NV1; + hcr |= HCR_TVM | HCR_TRVM | HCR_AT | HCR_NV1; } else { /* * For a guest hypervisor on v8.1 (VHE), allow to @@ -69,6 +70,14 @@ static void __activate_traps(struct kvm_vcpu *vcpu) hcr &= ~HCR_TVM; hcr |= vhcr_el2 & (HCR_TVM | HCR_TRVM); + + /* + * If we're using the EL1 translation regime + * (TGE clear), then ensure that AT S1 ops are + * trapped too. + */ + if (!vcpu_el2_tge_is_set(vcpu)) + hcr |= HCR_AT; } } diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 222f8c6f1d7e..aa4d4e187c9f 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -1703,6 +1703,10 @@ static bool access_sp_el1(struct kvm_vcpu *vcpu, return true; } +static bool forward_at_traps(struct kvm_vcpu *vcpu) +{ + return forward_traps(vcpu, HCR_AT); +} static bool access_elr(struct kvm_vcpu *vcpu, struct sys_reg_params *p, @@ -2236,12 +2240,205 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_SP_EL2), NULL, reset_unknown, SP_EL2 }, }; -#define SYS_INSN_TO_DESC(insn, access_fn, forward_fn) \ - { SYS_DESC((insn)), (access_fn), NULL, 0, 0, NULL, NULL, (forward_fn) } +static bool handle_s1e01(struct kvm_vcpu *vcpu, struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + int sys_encoding = sys_insn(p->Op0, p->Op1, p->CRn, p->CRm, p->Op2); + + __kvm_at_s1e01(vcpu, sys_encoding, p->regval); + + return true; +} + +static bool handle_s1e2(struct kvm_vcpu *vcpu, struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + int sys_encoding = sys_insn(p->Op0, p->Op1, p->CRn, p->CRm, p->Op2); + + __kvm_at_s1e2(vcpu, sys_encoding, p->regval); + + return true; +} + +static u64 setup_par_aborted(u32 esr) +{ + u64 par = 0; + + /* S [9]: fault in the stage 2 translation */ + par |= (1 << 9); + /* FST [6:1]: Fault status code */ + par |= (esr << 1); + /* F [0]: translation is aborted */ + par |= 1; + + return par; +} + +static u64 setup_par_completed(struct kvm_vcpu *vcpu, struct kvm_s2_trans *out) +{ + u64 par, vtcr_sh0; + + /* F [0]: Translation is completed successfully */ + par = 0; + /* ATTR [63:56] */ + par |= out->upper_attr; + /* PA [47:12] */ + par |= out->output & GENMASK_ULL(11, 0); + /* RES1 [11] */ + par |= (1UL << 11); + /* SH [8:7]: Shareability attribute */ + vtcr_sh0 = vcpu_read_sys_reg(vcpu, VTCR_EL2) & VTCR_EL2_SH0_MASK; + par |= (vtcr_sh0 >> VTCR_EL2_SH0_SHIFT) << 7; + + return par; +} + +static bool handle_s12(struct kvm_vcpu *vcpu, struct sys_reg_params *p, + const struct sys_reg_desc *r, bool write) +{ + u64 par, va; + u32 esr; + phys_addr_t ipa; + struct kvm_s2_trans out; + int ret; + + /* Do the stage-1 translation */ + handle_s1e01(vcpu, p, r); + par = vcpu_read_sys_reg(vcpu, PAR_EL1); + if (par & 1) { + /* The stage-1 translation aborted */ + return true; + } + + /* Do the stage-2 translation */ + va = p->regval; + ipa = (par & GENMASK_ULL(47, 12)) | (va & GENMASK_ULL(11, 0)); + out.esr = 0; + ret = kvm_walk_nested_s2(vcpu, ipa, &out); + if (ret < 0) + return false; + + /* Check if the stage-2 PTW is aborted */ + if (out.esr) { + esr = out.esr; + goto s2_trans_abort; + } + + /* Check the access permission */ + if ((!write && !out.readable) || (write && !out.writable)) { + esr = ESR_ELx_FSC_PERM; + esr |= out.level & 0x3; + goto s2_trans_abort; + } + + vcpu_write_sys_reg(vcpu, setup_par_completed(vcpu, &out), PAR_EL1); + return true; + +s2_trans_abort: + vcpu_write_sys_reg(vcpu, setup_par_aborted(esr), PAR_EL1); + return true; +} + +static bool handle_s12r(struct kvm_vcpu *vcpu, struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + return handle_s12(vcpu, p, r, false); +} + +static bool handle_s12w(struct kvm_vcpu *vcpu, struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + return handle_s12(vcpu, p, r, true); +} + +/* + * AT instruction emulation + * + * We emulate AT instructions executed in the virtual EL2. + * Basic strategy for the stage-1 translation emulation is to load proper + * context, which depends on the trapped instruction and the virtual HCR_EL2, + * to the EL1 virtual memory control registers and execute S1E[01] instructions + * in EL2. See below for more detail. + * + * For the stage-2 translation, which is necessary for S12E[01] emulation, + * we walk the guest hypervisor's stage-2 page table in software. + * + * The stage-1 translation emulations can be divided into two groups depending + * on the translation regime. + * + * 1. EL2 AT instructions: S1E2x + * +-----------------------------------------------------------------------+ + * | | Setting for the emulation | + * | Virtual HCR_EL2.E2H on trap |-----------------------------------------+ + * | | Phys EL1 regs | Phys NV, NV1 | Phys TGE | + * |-----------------------------------------------------------------------| + * | 0 | vEL2 | (1, 1) | 0 | + * | 1 | vEL2 | (0, 0) | 0 | + * +-----------------------------------------------------------------------+ + * + * We emulate the EL2 AT instructions by loading virtual EL2 context + * to the EL1 virtual memory control registers and executing corresponding + * EL1 AT instructions. + * + * We set physical NV and NV1 bits to use EL2 page table format for non-VHE + * guest hypervisor (i.e. HCR_EL2.E2H == 0). As a VHE guest hypervisor uses the + * EL1 page table format, we don't set those bits. + * + * We should clear physical TGE bit not to use the EL2 translation regime when + * the host uses the VHE feature. + * + * + * 2. EL0/EL1 AT instructions: S1E[01]x, S12E1x + * +----------------------------------------------------------------------+ + * | Virtual HCR_EL2 on trap | Setting for the emulation | + * |----------------------------------------------------------------------+ + * | (vE2H, vTGE) | (vNV, vNV1) | Phys EL1 regs | Phys NV, NV1 | Phys TGE | + * |----------------------------------------------------------------------| + * | (0, 0)* | (0, 0) | vEL1 | (0, 0) | 0 | + * | (0, 0) | (1, 1) | vEL1 | (1, 1) | 0 | + * | (1, 1) | (0, 0) | vEL2 | (0, 0) | 0 | + * | (1, 1) | (1, 1) | vEL2 | (1, 1) | 0 | + * +----------------------------------------------------------------------+ + * + * *For (0, 0) in the 'Virtual HCR_EL2 on trap' column, it actually means + * (1, 1). Keep them (0, 0) just for the readability. + * + * We set physical EL1 virtual memory control registers depending on + * (vE2H, vTGE) pair. When the pair is (0, 0) where AT instructions are + * supposed to use EL0/EL1 translation regime, we load the EL1 registers with + * the virtual EL1 registers (i.e. EL1 registers from the guest hypervisor's + * point of view). When the pair is (1, 1), however, AT instructions are defined + * to apply EL2 translation regime. To emulate this behavior, we load the EL1 + * registers with the virtual EL2 context. (i.e the shadow registers) + * + * We respect the virtual NV and NV1 bit for the emulation. When those bits are + * set, it means that a guest hypervisor would like to use EL2 page table format + * for the EL1 translation regime. We emulate this by setting the physical + * NV and NV1 bits. + */ + +#define SYS_INSN_TO_DESC(insn, access_fn, forward_fn) \ + { SYS_DESC(OP_##insn), (access_fn), NULL, 0, 0, \ + NULL, NULL, (forward_fn) } static struct sys_reg_desc sys_insn_descs[] = { { SYS_DESC(SYS_DC_ISW), access_dcsw }, + + SYS_INSN_TO_DESC(AT_S1E1R, handle_s1e01, forward_at_traps), + SYS_INSN_TO_DESC(AT_S1E1W, handle_s1e01, forward_at_traps), + SYS_INSN_TO_DESC(AT_S1E0R, handle_s1e01, forward_at_traps), + SYS_INSN_TO_DESC(AT_S1E0W, handle_s1e01, forward_at_traps), + SYS_INSN_TO_DESC(AT_S1E1RP, handle_s1e01, forward_at_traps), + SYS_INSN_TO_DESC(AT_S1E1WP, handle_s1e01, forward_at_traps), + { SYS_DESC(SYS_DC_CSW), access_dcsw }, { SYS_DESC(SYS_DC_CISW), access_dcsw }, + + SYS_INSN_TO_DESC(AT_S1E2R, handle_s1e2, forward_nv_traps), + SYS_INSN_TO_DESC(AT_S1E2W, handle_s1e2, forward_nv_traps), + SYS_INSN_TO_DESC(AT_S12E1R, handle_s12r, forward_nv_traps), + SYS_INSN_TO_DESC(AT_S12E1W, handle_s12w, forward_nv_traps), + SYS_INSN_TO_DESC(AT_S12E0R, handle_s12r, forward_nv_traps), + SYS_INSN_TO_DESC(AT_S12E0W, handle_s12w, forward_nv_traps), }; static bool trap_dbgdidr(struct kvm_vcpu *vcpu, From patchwork Mon Nov 29 20:01:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645763 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 660D8C433F5 for ; Mon, 29 Nov 2021 21:44:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232401AbhK2VsC (ORCPT ); Mon, 29 Nov 2021 16:48:02 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49472 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232342AbhK2Vp5 (ORCPT ); Mon, 29 Nov 2021 16:45:57 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 742DFC08EB3B for ; Mon, 29 Nov 2021 12:06:32 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 18EC4B815E6 for ; Mon, 29 Nov 2021 20:06:31 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8A118C53FAD; Mon, 29 Nov 2021 20:06:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216389; bh=erWNsUmKJB0tY9YbZdwV7bYc6xRayoPj0aSus1CdCtQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Z2DAhStbr8R2+yMgqZ9GJOWbZkxVxmmC/PRu8vToktR9Tc51iLuF3tfCvEzyTFhQk sbuyNLezvkE4ZPhCYAoCv7zX9gAFHlPlfuI6rQ8PwKE0uy0tjJHl6+8zubhAWlj0hh 1r6A5Kgu6DmRBfsftJkeZqIaXsS7Uk0FDnEG+w8vC0aVxMCGeo8bFb0NTN/SIlyF95 dc3EkIiIpKGU6ztrFmM7U0gDdGM9XoTsECrfvsAOxE7AsVUBDMeJPFFgfUpp/11Ltu Hj8DgzKXgi6DFLTuiAMUVHU3uZsgpkpCc1yKSTnksQqdpjOp+lLeHhgmwIv5OMGD5F RSRfhxyL6+bVQ== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmr3-008gvR-3L; Mon, 29 Nov 2021 20:02:25 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 46/69] KVM: arm64: nv: Trap and emulate TLBI instructions from virtual EL2 Date: Mon, 29 Nov 2021 20:01:27 +0000 Message-Id: <20211129200150.351436-47-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org When supporting nested virtualization a guest hypervisor executing TLBI instructions must be trapped and emulated by the host hypervisor, because the guest hypervisor can only affect physical TLB entries relating to its own execution environment (virtual EL2 in EL1) but not to the nested guests as required by the semantics of the instructions and TLBI instructions might also result in updates (invalidations) to shadow page tables. This patch does several things. 1. List and define all TLBI system instructions to emulate. 2. Emulate TLBI ALLE2(IS) instruction executed in the virtual EL2. Since we emulate the virtual EL2 in the EL1, we invalidate EL1&0 regime stage 1 TLB entries with setting vttbr_el2 having the VMID of the virtual EL2. 3. Emulate TLBI VAE2* instruction executed in the virtual EL2. Based on the same principle as TLBI ALLE2 instruction, we can simply emulate those instructions by executing corresponding VAE1* instructions with the virtual EL2's VMID assigned by the host hypervisor. Note that we are able to emulate TLBI ALLE2IS precisely by only invalidating stage 1 TLB entries via TLBI VMALL1IS instruction, but to make it simeple, we reuse the existing function, __kvm_tlb_flush_vmid(), which invalidates both of stage 1 and 2 TLB entries. 4. TLBI ALLE1(IS) instruction invalidates all EL1&0 regime stage 1 and 2 TLB entries (on all PEs in the same Inner Shareable domain). To emulate these instructions, we first need to clear all the mappings in the shadow page tables since executing those instructions implies the change of mappings in the stage 2 page tables maintained by the guest hypervisor. We then need to invalidate all EL1&0 regime stage 1 and 2 TLB entries of all VMIDs, which are assigned by the host hypervisor, for this VM. 5. Based on the same principle as TLBI ALLE1(IS) emulation, we clear the mappings in the shadow stage-2 page tables and invalidate TLB entries. But this time we do it only for the current VMID from the guest hypervisor's perspective, not for all VMIDs. 6. Based on the same principle as TLBI ALLE1(IS) and TLBI VMALLS12E1(IS) emulation, we clear the mappings in the shadow stage-2 page tables and invalidate TLB entries. We do it only for one mapping for the current VMID from the guest hypervisor's view. 7. Forward system instruction traps to the virtual EL2 if a corresponding bit in the virtual HCR_EL2 is set. 8. Even though a guest hypervisor can execute TLBI instructions that are accesible at EL1 without trap, it's wrong; All those TLBI instructions work based on current VMID, and when running a guest hypervisor current VMID is the one for itself, not the one from the virtual vttbr_el2. So letting a guest hypervisor execute those TLBI instructions results in invalidating its own TLB entries and leaving invalid TLB entries unhandled. Therefore we trap and emulate those TLBI instructions. The emulation is simple; we find a shadow VMID mapped to the virtual vttbr_el2, set it in the physical vttbr_el2, then execute the same instruction in EL2. We don't set HCR_EL2.TTLB bit yet. [ Changes performed by Marc Zynger: The TLBI handling code more or less directly execute the same instruction that has been trapped (with an EL2->EL1 conversion in the case of an EL2 TLBI), but that's unfortunately not enough: - TLBIs must be upgraded to the Inner Shareable domain to account for vcpu migration, just like we already have with HCR_EL2.FB. - The DSB instruction that synchronises these must thus be on the Inner Shareable domain as well. - Prior to executing the TLBI, we need another DSB ISHST to make sure that the update to the page tables is now visible. Ordering of system instructions fixed - The current TLB invalidation code is pretty buggy, as it assume a page mapping. On the contrary, it is likely that TLB invalidation will cover more than a single page, and the size should be decided by the guests configuration (and not the host's). Since we don't cache the guest mapping sizes in the shadow PT yet, let's assume the worse case (a block mapping) and invalidate that. Take this opportunity to fix the decoding of the parameter (it isn't a straight IPA). - In general, we always emulate local TBL invalidations as being as upgraded to the Inner Shareable domain so that we can easily deal with vcpu migration. This is consistent with the fact that we set HCR_EL2.FB when running non-nested VMs. So let's emulate TLBI ALLE2 as ALLE2IS. ] [ Changes performed by Christoffer Dall: Sometimes when we are invalidating the TLB for a certain S2 MMU context, this context can also have EL2 context associated with it and we have to invalidate this too. ] Signed-off-by: Jintack Lim Signed-off-by: Christoffer Dall Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_asm.h | 2 + arch/arm64/include/asm/sysreg.h | 36 ++++++ arch/arm64/kvm/hyp/vhe/switch.c | 8 +- arch/arm64/kvm/hyp/vhe/tlb.c | 81 ++++++++++++ arch/arm64/kvm/mmu.c | 19 ++- arch/arm64/kvm/sys_regs.c | 212 +++++++++++++++++++++++++++++++ 6 files changed, 353 insertions(+), 5 deletions(-) diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h index abea7e164400..7f56f1b14445 100644 --- a/arch/arm64/include/asm/kvm_asm.h +++ b/arch/arm64/include/asm/kvm_asm.h @@ -205,6 +205,8 @@ extern void __kvm_flush_cpu_context(struct kvm_s2_mmu *mmu); extern void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa, int level); extern void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu); +extern void __kvm_tlb_vae2is(struct kvm_s2_mmu *mmu, u64 va, u64 sys_encoding); +extern void __kvm_tlb_el1_instr(struct kvm_s2_mmu *mmu, u64 val, u64 sys_encoding); extern void __kvm_timer_set_cntvoff(u64 cntvoff); extern void __kvm_at_s1e01(struct kvm_vcpu *vcpu, u32 op, u64 vaddr); diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h index b673adeb190f..199333b02ece 100644 --- a/arch/arm64/include/asm/sysreg.h +++ b/arch/arm64/include/asm/sysreg.h @@ -674,6 +674,42 @@ #define OP_AT_S12E0R sys_insn(AT_Op0, 4, AT_CRn, 8, 6) #define OP_AT_S12E0W sys_insn(AT_Op0, 4, AT_CRn, 8, 7) +/* TLBI instructions */ +#define TLBI_Op0 1 +#define TLBI_Op1_EL1 0 /* Accessible from EL1 or higher */ +#define TLBI_Op1_EL2 4 /* Accessible from EL2 or higher */ +#define TLBI_CRn 8 +#define tlbi_insn_el1(CRm, Op2) sys_insn(TLBI_Op0, TLBI_Op1_EL1, TLBI_CRn, (CRm), (Op2)) +#define tlbi_insn_el2(CRm, Op2) sys_insn(TLBI_Op0, TLBI_Op1_EL2, TLBI_CRn, (CRm), (Op2)) + +#define OP_TLBI_VMALLE1IS tlbi_insn_el1(3, 0) +#define OP_TLBI_VAE1IS tlbi_insn_el1(3, 1) +#define OP_TLBI_ASIDE1IS tlbi_insn_el1(3, 2) +#define OP_TLBI_VAAE1IS tlbi_insn_el1(3, 3) +#define OP_TLBI_VALE1IS tlbi_insn_el1(3, 5) +#define OP_TLBI_VAALE1IS tlbi_insn_el1(3, 7) +#define OP_TLBI_VMALLE1 tlbi_insn_el1(7, 0) +#define OP_TLBI_VAE1 tlbi_insn_el1(7, 1) +#define OP_TLBI_ASIDE1 tlbi_insn_el1(7, 2) +#define OP_TLBI_VAAE1 tlbi_insn_el1(7, 3) +#define OP_TLBI_VALE1 tlbi_insn_el1(7, 5) +#define OP_TLBI_VAALE1 tlbi_insn_el1(7, 7) + +#define OP_TLBI_IPAS2E1IS tlbi_insn_el2(0, 1) +#define OP_TLBI_IPAS2LE1IS tlbi_insn_el2(0, 5) +#define OP_TLBI_ALLE2IS tlbi_insn_el2(3, 0) +#define OP_TLBI_VAE2IS tlbi_insn_el2(3, 1) +#define OP_TLBI_ALLE1IS tlbi_insn_el2(3, 4) +#define OP_TLBI_VALE2IS tlbi_insn_el2(3, 5) +#define OP_TLBI_VMALLS12E1IS tlbi_insn_el2(3, 6) +#define OP_TLBI_IPAS2E1 tlbi_insn_el2(4, 1) +#define OP_TLBI_IPAS2LE1 tlbi_insn_el2(4, 5) +#define OP_TLBI_ALLE2 tlbi_insn_el2(7, 0) +#define OP_TLBI_VAE2 tlbi_insn_el2(7, 1) +#define OP_TLBI_ALLE1 tlbi_insn_el2(7, 4) +#define OP_TLBI_VALE2 tlbi_insn_el2(7, 5) +#define OP_TLBI_VMALLS12E1 tlbi_insn_el2(7, 6) + /* Common SCTLR_ELx flags. */ #define SCTLR_ELx_DSSBS (BIT(44)) #define SCTLR_ELx_ATA (BIT(43)) diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c index 657a74bab73c..71a4914761d0 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -45,7 +45,7 @@ static void __activate_traps(struct kvm_vcpu *vcpu) * the EL1 virtual memory control register accesses * as well as the AT S1 operations. */ - hcr |= HCR_TVM | HCR_TRVM | HCR_AT | HCR_NV1; + hcr |= HCR_TVM | HCR_TRVM | HCR_AT | HCR_TTLB | HCR_NV1; } else { /* * For a guest hypervisor on v8.1 (VHE), allow to @@ -73,11 +73,11 @@ static void __activate_traps(struct kvm_vcpu *vcpu) /* * If we're using the EL1 translation regime - * (TGE clear), then ensure that AT S1 ops are - * trapped too. + * (TGE clear), then ensure that AT S1 and + * TLBI E1 ops are trapped too. */ if (!vcpu_el2_tge_is_set(vcpu)) - hcr |= HCR_AT; + hcr |= HCR_AT | HCR_TTLB; } } diff --git a/arch/arm64/kvm/hyp/vhe/tlb.c b/arch/arm64/kvm/hyp/vhe/tlb.c index 24cef9b87f9e..c4389db4cc22 100644 --- a/arch/arm64/kvm/hyp/vhe/tlb.c +++ b/arch/arm64/kvm/hyp/vhe/tlb.c @@ -161,3 +161,84 @@ void __kvm_flush_vm_context(void) dsb(ish); } + +void __kvm_tlb_vae2is(struct kvm_s2_mmu *mmu, u64 va, u64 sys_encoding) +{ + struct tlb_inv_context cxt; + + dsb(ishst); + + /* Switch to requested VMID */ + __tlb_switch_to_guest(mmu, &cxt); + + /* + * Execute the EL1 version of TLBI VAE2* instruction, forcing + * an upgrade to the Inner Shareable domain in order to + * perform the invalidation on all CPUs. + */ + switch (sys_encoding) { + case OP_TLBI_VAE2: + case OP_TLBI_VAE2IS: + __tlbi(vae1is, va); + break; + case OP_TLBI_VALE2: + case OP_TLBI_VALE2IS: + __tlbi(vale1is, va); + break; + default: + break; + } + dsb(ish); + isb(); + + __tlb_switch_to_host(&cxt); +} + +void __kvm_tlb_el1_instr(struct kvm_s2_mmu *mmu, u64 val, u64 sys_encoding) +{ + struct tlb_inv_context cxt; + + dsb(ishst); + + /* Switch to requested VMID */ + __tlb_switch_to_guest(mmu, &cxt); + + /* + * Execute the same instruction as the guest hypervisor did, + * expanding the scope of local TLB invalidations to the Inner + * Shareable domain so that it takes place on all CPUs. This + * is equivalent to having HCR_EL2.FB set. + */ + switch (sys_encoding) { + case OP_TLBI_VMALLE1: + case OP_TLBI_VMALLE1IS: + __tlbi(vmalle1is); + break; + case OP_TLBI_VAE1: + case OP_TLBI_VAE1IS: + __tlbi(vae1is, val); + break; + case OP_TLBI_ASIDE1: + case OP_TLBI_ASIDE1IS: + __tlbi(aside1is, val); + break; + case OP_TLBI_VAAE1: + case OP_TLBI_VAAE1IS: + __tlbi(vaae1is, val); + break; + case OP_TLBI_VALE1: + case OP_TLBI_VALE1IS: + __tlbi(vale1is, val); + break; + case OP_TLBI_VAALE1: + case OP_TLBI_VAALE1IS: + __tlbi(vaale1is, val); + break; + default: + break; + } + dsb(ish); + isb(); + + __tlb_switch_to_host(&cxt); +} diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index a9e743f39bfd..7e640cae732d 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -80,8 +80,25 @@ static bool memslot_is_logging(struct kvm_memory_slot *memslot) */ void kvm_flush_remote_tlbs(struct kvm *kvm) { + struct kvm_s2_mmu *mmu = &kvm->arch.mmu; + ++kvm->stat.generic.remote_tlb_flush_requests; - kvm_call_hyp(__kvm_tlb_flush_vmid, &kvm->arch.mmu); + + if (mmu == &kvm->arch.mmu) { + /* + * For a normal (i.e. non-nested) guest, flush entries for the + * given VMID * + */ + kvm_call_hyp(__kvm_tlb_flush_vmid, mmu); + } else { + /* + * When supporting nested virtualization, we can have multiple + * VMIDs in play for each VCPU in the VM, so it's really not + * worth it to try to quiesce the system and flush all the + * VMIDs that may be in use, instead just nuke the whole thing. + */ + kvm_call_hyp(__kvm_flush_vm_context); + } } static bool kvm_is_device_pfn(unsigned long pfn) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index aa4d4e187c9f..5bd1ea835cf0 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -1708,6 +1708,11 @@ static bool forward_at_traps(struct kvm_vcpu *vcpu) return forward_traps(vcpu, HCR_AT); } +static bool forward_ttlb_traps(struct kvm_vcpu *vcpu) +{ + return forward_traps(vcpu, HCR_TTLB); +} + static bool access_elr(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) @@ -2351,6 +2356,185 @@ static bool handle_s12w(struct kvm_vcpu *vcpu, struct sys_reg_params *p, return handle_s12(vcpu, p, r, true); } +static bool handle_alle2is(struct kvm_vcpu *vcpu, struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + /* + * To emulate invalidating all EL2 regime stage 1 TLB entries for all + * PEs, executing TLBI VMALLE1IS is enough. But reuse the existing + * interface for the simplicity; invalidating stage 2 entries doesn't + * affect the correctness. + */ + __kvm_tlb_flush_vmid(&vcpu->kvm->arch.mmu); + return true; +} + +static bool handle_vae2is(struct kvm_vcpu *vcpu, struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + int sys_encoding = sys_insn(p->Op0, p->Op1, p->CRn, p->CRm, p->Op2); + + /* + * Based on the same principle as TLBI ALLE2 instruction + * emulation, we emulate TLBI VAE2* instructions by executing + * corresponding TLBI VAE1* instructions with the virtual + * EL2's VMID assigned by the host hypervisor. + */ + __kvm_tlb_vae2is(&vcpu->kvm->arch.mmu, p->regval, sys_encoding); + return true; +} + +static bool handle_alle1is(struct kvm_vcpu *vcpu, struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + struct kvm_s2_mmu *mmu = &vcpu->kvm->arch.mmu; + spin_lock(&vcpu->kvm->mmu_lock); + + /* + * Clear all mappings in the shadow page tables and invalidate the stage + * 1 and 2 TLB entries via kvm_tlb_flush_vmid_ipa(). + */ + kvm_nested_s2_clear(vcpu->kvm); + + if (mmu->vmid.vmid_gen) { + /* + * Invalidate the stage 1 and 2 TLB entries for the host OS + * in a VM only if there is one. + */ + __kvm_tlb_flush_vmid(mmu); + } + + spin_unlock(&vcpu->kvm->mmu_lock); + + return true; +} + +static bool handle_vmalls12e1is(struct kvm_vcpu *vcpu, struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + u64 vttbr = vcpu_read_sys_reg(vcpu, VTTBR_EL2); + struct kvm_s2_mmu *mmu; + + spin_lock(&vcpu->kvm->mmu_lock); + + mmu = lookup_s2_mmu(vcpu->kvm, vttbr, HCR_VM); + if (mmu) + kvm_unmap_stage2_range(mmu, 0, kvm_phys_size(vcpu->kvm)); + + mmu = lookup_s2_mmu(vcpu->kvm, vttbr, 0); + if (mmu) + kvm_unmap_stage2_range(mmu, 0, kvm_phys_size(vcpu->kvm)); + + spin_unlock(&vcpu->kvm->mmu_lock); + + return true; +} + +static bool handle_ipas2e1is(struct kvm_vcpu *vcpu, struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + u64 vttbr = vcpu_read_sys_reg(vcpu, VTTBR_EL2); + u64 vtcr = vcpu_read_sys_reg(vcpu, VTCR_EL2); + struct kvm_s2_mmu *mmu; + u64 base_addr; + int max_size; + + /* + * We drop a number of things from the supplied value: + * + * - NS bit: we're non-secure only. + * + * - TTL field: We already have the granule size from the + * VTCR_EL2.TG0 field, and the level is only relevant to the + * guest's S2PT. + * + * - IPA[51:48]: We don't support 52bit IPA just yet... + * + * And of course, adjust the IPA to be on an actual address. + */ + base_addr = (p->regval & GENMASK_ULL(35, 0)) << 12; + + /* Compute the maximum extent of the invalidation */ + switch ((vtcr & VTCR_EL2_TG0_MASK)) { + case VTCR_EL2_TG0_4K: + max_size = SZ_1G; + break; + case VTCR_EL2_TG0_16K: + max_size = SZ_32M; + break; + case VTCR_EL2_TG0_64K: + /* + * No, we do not support 52bit IPA in nested yet. Once + * we do, this should be 4TB. + */ + /* FIXME: remove the 52bit PA support from the IDregs */ + max_size = SZ_512M; + break; + default: + BUG(); + } + + spin_lock(&vcpu->kvm->mmu_lock); + + mmu = lookup_s2_mmu(vcpu->kvm, vttbr, HCR_VM); + if (mmu) + kvm_unmap_stage2_range(mmu, base_addr, max_size); + + mmu = lookup_s2_mmu(vcpu->kvm, vttbr, 0); + if (mmu) + kvm_unmap_stage2_range(mmu, base_addr, max_size); + + spin_unlock(&vcpu->kvm->mmu_lock); + + return true; +} + +static bool handle_tlbi_el1(struct kvm_vcpu *vcpu, struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + u32 sys_encoding = sys_insn(p->Op0, p->Op1, p->CRn, p->CRm, p->Op2); + + /* + * If we're here, this is because we've trapped on a EL1 TLBI + * instruction that affects the EL1 translation regime while + * we're running in a context that doesn't allow us to let the + * HW do its thing (aka vEL2): + * + * - HCR_EL2.E2H == 0 : a non-VHE guest + * - HCR_EL2.{E2H,TGE} == { 1, 0 } : a VHE guest in guest mode + * + * We don't expect these helpers to ever be called when running + * in a vEL1 context. + */ + + WARN_ON(!vcpu_mode_el2(vcpu)); + + mutex_lock(&vcpu->kvm->lock); + + if ((__vcpu_sys_reg(vcpu, HCR_EL2) & (HCR_E2H | HCR_TGE)) != (HCR_E2H | HCR_TGE)) { + u64 virtual_vttbr = vcpu_read_sys_reg(vcpu, VTTBR_EL2); + struct kvm_s2_mmu *mmu; + + mmu = lookup_s2_mmu(vcpu->kvm, virtual_vttbr, HCR_VM); + if (mmu) + __kvm_tlb_el1_instr(mmu, p->regval, sys_encoding); + + mmu = lookup_s2_mmu(vcpu->kvm, virtual_vttbr, 0); + if (mmu) + __kvm_tlb_el1_instr(mmu, p->regval, sys_encoding); + } else { + /* + * ARMv8.4-NV allows the guest to change TGE behind + * our back, so we always trap EL1 TLBIs from vEL2... + */ + __kvm_tlb_el1_instr(&vcpu->kvm->arch.mmu, p->regval, sys_encoding); + } + + mutex_unlock(&vcpu->kvm->lock); + + return true; +} + /* * AT instruction emulation * @@ -2433,12 +2617,40 @@ static struct sys_reg_desc sys_insn_descs[] = { { SYS_DESC(SYS_DC_CSW), access_dcsw }, { SYS_DESC(SYS_DC_CISW), access_dcsw }, + SYS_INSN_TO_DESC(TLBI_VMALLE1IS, handle_tlbi_el1, forward_ttlb_traps), + SYS_INSN_TO_DESC(TLBI_VAE1IS, handle_tlbi_el1, forward_ttlb_traps), + SYS_INSN_TO_DESC(TLBI_ASIDE1IS, handle_tlbi_el1, forward_ttlb_traps), + SYS_INSN_TO_DESC(TLBI_VAAE1IS, handle_tlbi_el1, forward_ttlb_traps), + SYS_INSN_TO_DESC(TLBI_VALE1IS, handle_tlbi_el1, forward_ttlb_traps), + SYS_INSN_TO_DESC(TLBI_VAALE1IS, handle_tlbi_el1, forward_ttlb_traps), + SYS_INSN_TO_DESC(TLBI_VMALLE1, handle_tlbi_el1, forward_ttlb_traps), + SYS_INSN_TO_DESC(TLBI_VAE1, handle_tlbi_el1, forward_ttlb_traps), + SYS_INSN_TO_DESC(TLBI_ASIDE1, handle_tlbi_el1, forward_ttlb_traps), + SYS_INSN_TO_DESC(TLBI_VAAE1, handle_tlbi_el1, forward_ttlb_traps), + SYS_INSN_TO_DESC(TLBI_VALE1, handle_tlbi_el1, forward_ttlb_traps), + SYS_INSN_TO_DESC(TLBI_VAALE1, handle_tlbi_el1, forward_ttlb_traps), + SYS_INSN_TO_DESC(AT_S1E2R, handle_s1e2, forward_nv_traps), SYS_INSN_TO_DESC(AT_S1E2W, handle_s1e2, forward_nv_traps), SYS_INSN_TO_DESC(AT_S12E1R, handle_s12r, forward_nv_traps), SYS_INSN_TO_DESC(AT_S12E1W, handle_s12w, forward_nv_traps), SYS_INSN_TO_DESC(AT_S12E0R, handle_s12r, forward_nv_traps), SYS_INSN_TO_DESC(AT_S12E0W, handle_s12w, forward_nv_traps), + + SYS_INSN_TO_DESC(TLBI_IPAS2E1IS, handle_ipas2e1is, forward_nv_traps), + SYS_INSN_TO_DESC(TLBI_IPAS2LE1IS, handle_ipas2e1is, forward_nv_traps), + SYS_INSN_TO_DESC(TLBI_ALLE2IS, handle_alle2is, forward_nv_traps), + SYS_INSN_TO_DESC(TLBI_VAE2IS, handle_vae2is, forward_nv_traps), + SYS_INSN_TO_DESC(TLBI_ALLE1IS, handle_alle1is, forward_nv_traps), + SYS_INSN_TO_DESC(TLBI_VALE2IS, handle_vae2is, forward_nv_traps), + SYS_INSN_TO_DESC(TLBI_VMALLS12E1IS, handle_vmalls12e1is, forward_nv_traps), + SYS_INSN_TO_DESC(TLBI_IPAS2E1, handle_ipas2e1is, forward_nv_traps), + SYS_INSN_TO_DESC(TLBI_IPAS2LE1, handle_ipas2e1is, forward_nv_traps), + SYS_INSN_TO_DESC(TLBI_ALLE2, handle_alle2is, forward_nv_traps), + SYS_INSN_TO_DESC(TLBI_VAE2, handle_vae2is, forward_nv_traps), + SYS_INSN_TO_DESC(TLBI_ALLE1, handle_alle1is, forward_nv_traps), + SYS_INSN_TO_DESC(TLBI_VALE2, handle_vae2is, forward_nv_traps), + SYS_INSN_TO_DESC(TLBI_VMALLS12E1, handle_vmalls12e1is, forward_nv_traps), }; static bool trap_dbgdidr(struct kvm_vcpu *vcpu, From patchwork Mon Nov 29 20:01:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645741 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 93E2CC433EF for ; Mon, 29 Nov 2021 21:44:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231894AbhK2Vro (ORCPT ); Mon, 29 Nov 2021 16:47:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48528 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231424AbhK2Vpn (ORCPT ); Mon, 29 Nov 2021 16:45:43 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A5945C08EB30 for ; Mon, 29 Nov 2021 12:05:50 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id F1F84CE13DE for ; Mon, 29 Nov 2021 20:05:48 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2DC68C53FC7; Mon, 29 Nov 2021 20:05:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216347; bh=THxfTratzuqnEYOjcyY3aQOXMptJwOp65u+vqN5et1c=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=g9s2zOs6sQmgspPbLkSCbYEv4Bw+wREmA1o4IPHSDOSEbi318fK9wOhWBMsdCkMjz sWHM4mKPgn0kGGf9/aRL+ONrMvLL6rEshGFxNuarekB2oEg46XBNXomJrWDTH+tOzs p/TjjuMUKw2IQTJ/4gj3AU2yOw0Q3IWfl8Mtveg9IaJ2DBdGMluVfFTsh8D4oALUYv To9W1c4S9JFfelUokt+0AJdKdeckP0PdduG5FnA7X8OXKuhZWYnEQRK7tmhpPjs66c klfkbPDbebpMq0SoyfpSUY11bxzzRRKZe4589jRNolIXurrxwkuOwqGPRrg+4CYA53 ZkJJsTXZEDTuQ== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmr4-008gvR-BT; Mon, 29 Nov 2021 20:02:27 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 47/69] KVM: arm64: nv: Fold guest's HCR_EL2 configuration into the host's Date: Mon, 29 Nov 2021 20:01:28 +0000 Message-Id: <20211129200150.351436-48-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org When entering a L2 guest (nested virt enabled, but not in hypervisor context), we need to honor the traps the L1 guest has asked enabled. For now, just OR the guest's HCR_EL2 into the host's. We may have to do some filtering in the future though. Signed-off-by: Marc Zyngier --- arch/arm64/kvm/hyp/vhe/switch.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c index 71a4914761d0..ef4488db6dc1 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -79,6 +79,11 @@ static void __activate_traps(struct kvm_vcpu *vcpu) if (!vcpu_el2_tge_is_set(vcpu)) hcr |= HCR_AT | HCR_TTLB; } + } else if (nested_virt_in_use(vcpu)) { + u64 vhcr_el2 = __vcpu_sys_reg(vcpu, HCR_EL2); + + vhcr_el2 &= ~HCR_GUEST_NV_FILTER_FLAGS; + hcr |= vhcr_el2; } ___activate_traps(vcpu, hcr); From patchwork Mon Nov 29 20:01:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645747 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7DF53C433FE for ; Mon, 29 Nov 2021 21:44:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232640AbhK2Vrx (ORCPT ); Mon, 29 Nov 2021 16:47:53 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48568 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231981AbhK2Vpw (ORCPT ); Mon, 29 Nov 2021 16:45:52 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7BEB1C091D20 for ; Mon, 29 Nov 2021 12:06:18 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 1A04DB815E0 for ; Mon, 29 Nov 2021 20:06:17 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7931BC53FCD; Mon, 29 Nov 2021 20:06:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216375; bh=WVQ0zLLOYtU2kESH+F7N92d+PDL0AMboi+5Gu3LJGuM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=UQYxG3VC+Jcmo/wYW6iZ/s2RCdhPifG8Mt/CAjVBgEv4PSuESftdRXrUZVdQspzv2 pqnppsE7vvBB8FuMHaMXBOskcBCqzrpQvzC3yEi5mVWlHv3p6KdhWrmdHk5IqubnWh rrqb1OWcG4pbk7mqJ3tE/h1zeS+KfjOQ2J9WD0OxSAgv36MbB2jZaM3SJsYegbdP72 B7FC9l6jfAnDvFycirJPjwkZb8aMO+7VqIvYhOEk6zc8ACvllx8LRTliEi+cLCla0i 6YMNpfRF8zzruBr/9M+jMGOL3n8HRKIKCOT4j3nPww1qp7BXlzhL9l0g+jxkf86t+h eOgQ+ES0lioqA== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmr6-008gvR-L3; Mon, 29 Nov 2021 20:02:29 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 48/69] KVM: arm64: nv: arch_timer: Support hyp timer emulation Date: Mon, 29 Nov 2021 20:01:29 +0000 Message-Id: <20211129200150.351436-49-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Christoffer Dall Emulating EL2 also means emulating the EL2 timers. To do so, we expand our timer framework to deal with at most 4 timers. At any given time, two timers are using the HW timers, and the two others are purely emulated. The role of deciding which is which at any given time is left to a mapping function which is called every time we need to make such a decision. Signed-off-by: Christoffer Dall [maz: added CNTVOFF support, general reworking for v4.8] Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_host.h | 4 + arch/arm64/kvm/arch_timer.c | 165 ++++++++++++++++++++++++++++-- arch/arm64/kvm/sys_regs.c | 7 +- arch/arm64/kvm/trace_arm.h | 6 +- arch/arm64/kvm/vgic/vgic.c | 15 +++ include/kvm/arm_arch_timer.h | 8 +- include/kvm/arm_vgic.h | 1 + 7 files changed, 194 insertions(+), 12 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 35f3d7939484..96edd6348bd1 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -286,6 +286,10 @@ enum vcpu_sysreg { TPIDR_EL2, /* EL2 Software Thread ID Register */ CNTHCTL_EL2, /* Counter-timer Hypervisor Control register */ SP_EL2, /* EL2 Stack Pointer */ + CNTHP_CTL_EL2, + CNTHP_CVAL_EL2, + CNTHV_CTL_EL2, + CNTHV_CVAL_EL2, NR_SYS_REGS /* Nothing after this line! */ }; diff --git a/arch/arm64/kvm/arch_timer.c b/arch/arm64/kvm/arch_timer.c index 3df67c127489..3e80c3bdd8ce 100644 --- a/arch/arm64/kvm/arch_timer.c +++ b/arch/arm64/kvm/arch_timer.c @@ -16,6 +16,7 @@ #include #include #include +#include #include #include @@ -40,6 +41,16 @@ static const struct kvm_irq_level default_vtimer_irq = { .level = 1, }; +static const struct kvm_irq_level default_hptimer_irq = { + .irq = 26, + .level = 1, +}; + +static const struct kvm_irq_level default_hvtimer_irq = { + .irq = 28, + .level = 1, +}; + static bool kvm_timer_irq_can_fire(struct arch_timer_context *timer_ctx); static void kvm_timer_update_irq(struct kvm_vcpu *vcpu, bool new_level, struct arch_timer_context *timer_ctx); @@ -51,6 +62,11 @@ static void kvm_arm_timer_write(struct kvm_vcpu *vcpu, static u64 kvm_arm_timer_read(struct kvm_vcpu *vcpu, struct arch_timer_context *timer, enum kvm_arch_timer_regs treg); +static bool kvm_arch_timer_get_input_level(int vintid); + +static struct irq_ops arch_timer_irq_ops = { + .get_input_level = kvm_arch_timer_get_input_level, +}; u32 timer_get_ctl(struct arch_timer_context *ctxt) { @@ -61,6 +77,10 @@ u32 timer_get_ctl(struct arch_timer_context *ctxt) return __vcpu_sys_reg(vcpu, CNTV_CTL_EL0); case TIMER_PTIMER: return __vcpu_sys_reg(vcpu, CNTP_CTL_EL0); + case TIMER_HVTIMER: + return __vcpu_sys_reg(vcpu, CNTHV_CTL_EL2); + case TIMER_HPTIMER: + return __vcpu_sys_reg(vcpu, CNTHP_CTL_EL2); default: WARN_ON(1); return 0; @@ -76,6 +96,10 @@ u64 timer_get_cval(struct arch_timer_context *ctxt) return __vcpu_sys_reg(vcpu, CNTV_CVAL_EL0); case TIMER_PTIMER: return __vcpu_sys_reg(vcpu, CNTP_CVAL_EL0); + case TIMER_HVTIMER: + return __vcpu_sys_reg(vcpu, CNTHV_CVAL_EL2); + case TIMER_HPTIMER: + return __vcpu_sys_reg(vcpu, CNTHP_CVAL_EL2); default: WARN_ON(1); return 0; @@ -105,6 +129,12 @@ static void timer_set_ctl(struct arch_timer_context *ctxt, u32 ctl) case TIMER_PTIMER: __vcpu_sys_reg(vcpu, CNTP_CTL_EL0) = ctl; break; + case TIMER_HVTIMER: + __vcpu_sys_reg(vcpu, CNTHV_CTL_EL2) = ctl; + break; + case TIMER_HPTIMER: + __vcpu_sys_reg(vcpu, CNTHP_CTL_EL2) = ctl; + break; default: WARN_ON(1); } @@ -121,6 +151,12 @@ static void timer_set_cval(struct arch_timer_context *ctxt, u64 cval) case TIMER_PTIMER: __vcpu_sys_reg(vcpu, CNTP_CVAL_EL0) = cval; break; + case TIMER_HVTIMER: + __vcpu_sys_reg(vcpu, CNTHV_CVAL_EL2) = cval; + break; + case TIMER_HPTIMER: + __vcpu_sys_reg(vcpu, CNTHP_CVAL_EL2) = cval; + break; default: WARN_ON(1); } @@ -146,13 +182,27 @@ u64 kvm_phys_timer_read(void) static void get_timer_map(struct kvm_vcpu *vcpu, struct timer_map *map) { - if (has_vhe()) { + if (nested_virt_in_use(vcpu)) { + if (is_hyp_ctxt(vcpu)) { + map->direct_vtimer = vcpu_hvtimer(vcpu); + map->direct_ptimer = vcpu_hptimer(vcpu); + map->emul_vtimer = vcpu_vtimer(vcpu); + map->emul_ptimer = vcpu_ptimer(vcpu); + } else { + map->direct_vtimer = vcpu_vtimer(vcpu); + map->direct_ptimer = vcpu_ptimer(vcpu); + map->emul_vtimer = vcpu_hvtimer(vcpu); + map->emul_ptimer = vcpu_hptimer(vcpu); + } + } else if (has_vhe()) { map->direct_vtimer = vcpu_vtimer(vcpu); map->direct_ptimer = vcpu_ptimer(vcpu); + map->emul_vtimer = NULL; map->emul_ptimer = NULL; } else { map->direct_vtimer = vcpu_vtimer(vcpu); map->direct_ptimer = NULL; + map->emul_vtimer = NULL; map->emul_ptimer = vcpu_ptimer(vcpu); } @@ -325,9 +375,11 @@ static bool kvm_timer_should_fire(struct arch_timer_context *timer_ctx) switch (index) { case TIMER_VTIMER: + case TIMER_HVTIMER: cnt_ctl = read_sysreg_el0(SYS_CNTV_CTL); break; case TIMER_PTIMER: + case TIMER_HPTIMER: cnt_ctl = read_sysreg_el0(SYS_CNTP_CTL); break; case NR_KVM_TIMERS: @@ -358,6 +410,7 @@ bool kvm_timer_is_pending(struct kvm_vcpu *vcpu) return kvm_timer_should_fire(map.direct_vtimer) || kvm_timer_should_fire(map.direct_ptimer) || + kvm_timer_should_fire(map.emul_vtimer) || kvm_timer_should_fire(map.emul_ptimer); } @@ -438,6 +491,7 @@ static void timer_save_state(struct arch_timer_context *ctx) switch (index) { case TIMER_VTIMER: + case TIMER_HVTIMER: timer_set_ctl(ctx, read_sysreg_el0(SYS_CNTV_CTL)); timer_set_cval(ctx, read_sysreg_el0(SYS_CNTV_CVAL)); @@ -447,6 +501,7 @@ static void timer_save_state(struct arch_timer_context *ctx) break; case TIMER_PTIMER: + case TIMER_HPTIMER: timer_set_ctl(ctx, read_sysreg_el0(SYS_CNTP_CTL)); timer_set_cval(ctx, read_sysreg_el0(SYS_CNTP_CVAL)); @@ -484,6 +539,7 @@ static void kvm_timer_blocking(struct kvm_vcpu *vcpu) */ if (!kvm_timer_irq_can_fire(map.direct_vtimer) && !kvm_timer_irq_can_fire(map.direct_ptimer) && + !kvm_timer_irq_can_fire(map.emul_vtimer) && !kvm_timer_irq_can_fire(map.emul_ptimer)) return; @@ -517,11 +573,13 @@ static void timer_restore_state(struct arch_timer_context *ctx) switch (index) { case TIMER_VTIMER: + case TIMER_HVTIMER: write_sysreg_el0(timer_get_cval(ctx), SYS_CNTV_CVAL); isb(); write_sysreg_el0(timer_get_ctl(ctx), SYS_CNTV_CTL); break; case TIMER_PTIMER: + case TIMER_HPTIMER: write_sysreg_el0(timer_get_cval(ctx), SYS_CNTP_CVAL); isb(); write_sysreg_el0(timer_get_ctl(ctx), SYS_CNTP_CTL); @@ -598,6 +656,42 @@ static void kvm_timer_vcpu_load_nogic(struct kvm_vcpu *vcpu) enable_percpu_irq(host_vtimer_irq, host_vtimer_irq_flags); } +static void kvm_timer_vcpu_load_nested_switch(struct kvm_vcpu *vcpu, + struct timer_map *map) +{ + int hw, ret; + + if (!irqchip_in_kernel(vcpu->kvm)) + return; + + /* + * We only ever unmap the vtimer irq on a VHE system that runs nested + * virtualization, in which case we have both a valid emul_vtimer, + * emul_ptimer, direct_vtimer, and direct_ptimer. + * + * Since this is called from kvm_timer_vcpu_load(), a change between + * vEL2 and vEL1/0 will have just happened, and the timer_map will + * represent this, and therefore we switch the emul/direct mappings + * below. + */ + hw = kvm_vgic_get_map(vcpu, map->direct_vtimer->irq.irq); + if (hw < 0) { + kvm_vgic_unmap_phys_irq(vcpu, map->emul_vtimer->irq.irq); + kvm_vgic_unmap_phys_irq(vcpu, map->emul_ptimer->irq.irq); + + ret = kvm_vgic_map_phys_irq(vcpu, + map->direct_vtimer->host_timer_irq, + map->direct_vtimer->irq.irq, + &arch_timer_irq_ops); + WARN_ON_ONCE(ret); + ret = kvm_vgic_map_phys_irq(vcpu, + map->direct_ptimer->host_timer_irq, + map->direct_ptimer->irq.irq, + &arch_timer_irq_ops); + WARN_ON_ONCE(ret); + } +} + void kvm_timer_vcpu_load(struct kvm_vcpu *vcpu) { struct arch_timer_cpu *timer = vcpu_timer(vcpu); @@ -609,6 +703,9 @@ void kvm_timer_vcpu_load(struct kvm_vcpu *vcpu) get_timer_map(vcpu, &map); if (static_branch_likely(&has_gic_active_state)) { + if (nested_virt_in_use(vcpu)) + kvm_timer_vcpu_load_nested_switch(vcpu, &map); + kvm_timer_vcpu_load_gic(map.direct_vtimer); if (map.direct_ptimer) kvm_timer_vcpu_load_gic(map.direct_ptimer); @@ -624,6 +721,8 @@ void kvm_timer_vcpu_load(struct kvm_vcpu *vcpu) if (map.direct_ptimer) timer_restore_state(map.direct_ptimer); + if (map.emul_vtimer) + timer_emulate(map.emul_vtimer); if (map.emul_ptimer) timer_emulate(map.emul_ptimer); } @@ -669,6 +768,8 @@ void kvm_timer_vcpu_put(struct kvm_vcpu *vcpu) * In any case, we re-schedule the hrtimer for the physical timer when * coming back to the VCPU thread in kvm_timer_vcpu_load(). */ + if (map.emul_vtimer) + soft_timer_cancel(&map.emul_vtimer->hrtimer); if (map.emul_ptimer) soft_timer_cancel(&map.emul_ptimer->hrtimer); @@ -729,10 +830,14 @@ int kvm_timer_vcpu_reset(struct kvm_vcpu *vcpu) */ timer_set_ctl(vcpu_vtimer(vcpu), 0); timer_set_ctl(vcpu_ptimer(vcpu), 0); + timer_set_ctl(vcpu_hvtimer(vcpu), 0); + timer_set_ctl(vcpu_hptimer(vcpu), 0); if (timer->enabled) { kvm_timer_update_irq(vcpu, false, vcpu_vtimer(vcpu)); kvm_timer_update_irq(vcpu, false, vcpu_ptimer(vcpu)); + kvm_timer_update_irq(vcpu, false, vcpu_hvtimer(vcpu)); + kvm_timer_update_irq(vcpu, false, vcpu_hptimer(vcpu)); if (irqchip_in_kernel(vcpu->kvm)) { kvm_vgic_reset_mapped_irq(vcpu, map.direct_vtimer->irq.irq); @@ -741,6 +846,8 @@ int kvm_timer_vcpu_reset(struct kvm_vcpu *vcpu) } } + if (map.emul_vtimer) + soft_timer_cancel(&map.emul_vtimer->hrtimer); if (map.emul_ptimer) soft_timer_cancel(&map.emul_ptimer->hrtimer); @@ -771,30 +878,47 @@ void kvm_timer_vcpu_init(struct kvm_vcpu *vcpu) struct arch_timer_cpu *timer = vcpu_timer(vcpu); struct arch_timer_context *vtimer = vcpu_vtimer(vcpu); struct arch_timer_context *ptimer = vcpu_ptimer(vcpu); + struct arch_timer_context *hvtimer = vcpu_hvtimer(vcpu); + struct arch_timer_context *hptimer = vcpu_hptimer(vcpu); vtimer->vcpu = vcpu; ptimer->vcpu = vcpu; + hvtimer->vcpu = vcpu; + hptimer->vcpu = vcpu; /* Synchronize cntvoff across all vtimers of a VM. */ update_vtimer_cntvoff(vcpu, kvm_phys_timer_read()); timer_set_offset(ptimer, 0); + timer_set_offset(hvtimer, 0); + timer_set_offset(hptimer, 0); hrtimer_init(&timer->bg_timer, CLOCK_MONOTONIC, HRTIMER_MODE_ABS_HARD); timer->bg_timer.function = kvm_bg_timer_expire; hrtimer_init(&vtimer->hrtimer, CLOCK_MONOTONIC, HRTIMER_MODE_ABS_HARD); hrtimer_init(&ptimer->hrtimer, CLOCK_MONOTONIC, HRTIMER_MODE_ABS_HARD); + hrtimer_init(&hvtimer->hrtimer, CLOCK_MONOTONIC, HRTIMER_MODE_ABS_HARD); + hrtimer_init(&hptimer->hrtimer, CLOCK_MONOTONIC, HRTIMER_MODE_ABS_HARD); + vtimer->hrtimer.function = kvm_hrtimer_expire; ptimer->hrtimer.function = kvm_hrtimer_expire; + hvtimer->hrtimer.function = kvm_hrtimer_expire; + hptimer->hrtimer.function = kvm_hrtimer_expire; vtimer->irq.irq = default_vtimer_irq.irq; ptimer->irq.irq = default_ptimer_irq.irq; + hvtimer->irq.irq = default_hvtimer_irq.irq; + hptimer->irq.irq = default_hptimer_irq.irq; vtimer->host_timer_irq = host_vtimer_irq; ptimer->host_timer_irq = host_ptimer_irq; + hvtimer->host_timer_irq = host_vtimer_irq; + hptimer->host_timer_irq = host_ptimer_irq; vtimer->host_timer_irq_flags = host_vtimer_irq_flags; ptimer->host_timer_irq_flags = host_ptimer_irq_flags; + hvtimer->host_timer_irq_flags = host_vtimer_irq_flags; + hptimer->host_timer_irq_flags = host_ptimer_irq_flags; } static void kvm_timer_init_interrupt(void *info) @@ -901,6 +1025,10 @@ static u64 kvm_arm_timer_read(struct kvm_vcpu *vcpu, val = kvm_phys_timer_read() - timer_get_offset(timer); break; + case TIMER_REG_VOFF: + val = timer_get_offset(timer); + break; + default: BUG(); } @@ -943,6 +1071,10 @@ static void kvm_arm_timer_write(struct kvm_vcpu *vcpu, timer_set_cval(timer, val); break; + case TIMER_REG_VOFF: + timer_set_offset(timer, val); + break; + default: BUG(); } @@ -1041,10 +1173,6 @@ static const struct irq_domain_ops timer_domain_ops = { .free = timer_irq_domain_free, }; -static struct irq_ops arch_timer_irq_ops = { - .get_input_level = kvm_arch_timer_get_input_level, -}; - static void kvm_irq_fixup_flags(unsigned int virq, u32 *flags) { *flags = irq_get_trigger_type(virq); @@ -1189,7 +1317,7 @@ void kvm_timer_vcpu_terminate(struct kvm_vcpu *vcpu) static bool timer_irqs_are_valid(struct kvm_vcpu *vcpu) { - int vtimer_irq, ptimer_irq; + int vtimer_irq, ptimer_irq, hvtimer_irq, hptimer_irq; int i, ret; vtimer_irq = vcpu_vtimer(vcpu)->irq.irq; @@ -1202,16 +1330,28 @@ static bool timer_irqs_are_valid(struct kvm_vcpu *vcpu) if (ret) return false; + hvtimer_irq = vcpu_hvtimer(vcpu)->irq.irq; + ret = kvm_vgic_set_owner(vcpu, hvtimer_irq, vcpu_hvtimer(vcpu)); + if (ret) + return false; + + hptimer_irq = vcpu_hptimer(vcpu)->irq.irq; + ret = kvm_vgic_set_owner(vcpu, hptimer_irq, vcpu_hptimer(vcpu)); + if (ret) + return false; + kvm_for_each_vcpu(i, vcpu, vcpu->kvm) { if (vcpu_vtimer(vcpu)->irq.irq != vtimer_irq || - vcpu_ptimer(vcpu)->irq.irq != ptimer_irq) + vcpu_ptimer(vcpu)->irq.irq != ptimer_irq || + vcpu_hvtimer(vcpu)->irq.irq != hvtimer_irq || + vcpu_hptimer(vcpu)->irq.irq != hptimer_irq) return false; } return true; } -bool kvm_arch_timer_get_input_level(int vintid) +static bool kvm_arch_timer_get_input_level(int vintid) { struct kvm_vcpu *vcpu = kvm_get_running_vcpu(); struct arch_timer_context *timer; @@ -1220,6 +1360,10 @@ bool kvm_arch_timer_get_input_level(int vintid) timer = vcpu_vtimer(vcpu); else if (vintid == vcpu_ptimer(vcpu)->irq.irq) timer = vcpu_ptimer(vcpu); + else if (vintid == vcpu_hvtimer(vcpu)->irq.irq) + timer = vcpu_hvtimer(vcpu); + else if (vintid == vcpu_hptimer(vcpu)->irq.irq) + timer = vcpu_hptimer(vcpu); else BUG(); @@ -1302,6 +1446,7 @@ static void set_timer_irqs(struct kvm *kvm, int vtimer_irq, int ptimer_irq) kvm_for_each_vcpu(i, vcpu, kvm) { vcpu_vtimer(vcpu)->irq.irq = vtimer_irq; vcpu_ptimer(vcpu)->irq.irq = ptimer_irq; + /* TODO: Add support for hv/hp timers */ } } @@ -1312,6 +1457,8 @@ int kvm_arm_timer_set_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr) struct arch_timer_context *ptimer = vcpu_ptimer(vcpu); int irq; + /* TODO: Add support for hv/hp timers */ + if (!irqchip_in_kernel(vcpu->kvm)) return -EINVAL; @@ -1344,6 +1491,8 @@ int kvm_arm_timer_get_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr) struct arch_timer_context *timer; int irq; + /* TODO: Add support for hv/hp timers */ + switch (attr->attr) { case KVM_ARM_VCPU_TIMER_IRQ_VTIMER: timer = vcpu_vtimer(vcpu); diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 5bd1ea835cf0..e6c9a01670c9 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -1325,6 +1325,11 @@ static bool access_arch_timer(struct kvm_vcpu *vcpu, tmr = TIMER_PTIMER; treg = TIMER_REG_CVAL; break; + case SYS_CNTVOFF_EL2: + tmr = TIMER_VTIMER; + treg = TIMER_REG_VOFF; + break; + default: BUG(); } @@ -2222,7 +2227,7 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_CONTEXTIDR_EL2), access_rw, reset_val, CONTEXTIDR_EL2, 0 }, { SYS_DESC(SYS_TPIDR_EL2), access_rw, reset_val, TPIDR_EL2, 0 }, - { SYS_DESC(SYS_CNTVOFF_EL2), access_rw, reset_val, CNTVOFF_EL2, 0 }, + { SYS_DESC(SYS_CNTVOFF_EL2), access_arch_timer }, { SYS_DESC(SYS_CNTHCTL_EL2), access_rw, reset_val, CNTHCTL_EL2, 0 }, { SYS_DESC(SYS_SCTLR_EL12), access_vm_reg, reset_val, SCTLR_EL1, 0x00C50078 }, diff --git a/arch/arm64/kvm/trace_arm.h b/arch/arm64/kvm/trace_arm.h index f3e46a976125..6ce5c025218d 100644 --- a/arch/arm64/kvm/trace_arm.h +++ b/arch/arm64/kvm/trace_arm.h @@ -206,6 +206,7 @@ TRACE_EVENT(kvm_get_timer_map, __field( unsigned long, vcpu_id ) __field( int, direct_vtimer ) __field( int, direct_ptimer ) + __field( int, emul_vtimer ) __field( int, emul_ptimer ) ), @@ -214,14 +215,17 @@ TRACE_EVENT(kvm_get_timer_map, __entry->direct_vtimer = arch_timer_ctx_index(map->direct_vtimer); __entry->direct_ptimer = (map->direct_ptimer) ? arch_timer_ctx_index(map->direct_ptimer) : -1; + __entry->emul_vtimer = + (map->emul_vtimer) ? arch_timer_ctx_index(map->emul_vtimer) : -1; __entry->emul_ptimer = (map->emul_ptimer) ? arch_timer_ctx_index(map->emul_ptimer) : -1; ), - TP_printk("VCPU: %ld, dv: %d, dp: %d, ep: %d", + TP_printk("VCPU: %ld, dv: %d, dp: %d, ev: %d, ep: %d", __entry->vcpu_id, __entry->direct_vtimer, __entry->direct_ptimer, + __entry->emul_vtimer, __entry->emul_ptimer) ); diff --git a/arch/arm64/kvm/vgic/vgic.c b/arch/arm64/kvm/vgic/vgic.c index 5dad4996cfb2..21d088e8597d 100644 --- a/arch/arm64/kvm/vgic/vgic.c +++ b/arch/arm64/kvm/vgic/vgic.c @@ -573,6 +573,21 @@ int kvm_vgic_unmap_phys_irq(struct kvm_vcpu *vcpu, unsigned int vintid) return 0; } +int kvm_vgic_get_map(struct kvm_vcpu *vcpu, unsigned int vintid) +{ + struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, vcpu, vintid); + unsigned long flags; + int ret = -1; + + raw_spin_lock_irqsave(&irq->irq_lock, flags); + if (irq->hw) + ret = irq->hwintid; + raw_spin_unlock_irqrestore(&irq->irq_lock, flags); + + vgic_put_irq(vcpu->kvm, irq); + return ret; +} + /** * kvm_vgic_set_owner - Set the owner of an interrupt for a VM * diff --git a/include/kvm/arm_arch_timer.h b/include/kvm/arm_arch_timer.h index 51c19381108c..0a76dac8cb6a 100644 --- a/include/kvm/arm_arch_timer.h +++ b/include/kvm/arm_arch_timer.h @@ -13,6 +13,8 @@ enum kvm_arch_timers { TIMER_PTIMER, TIMER_VTIMER, + TIMER_HVTIMER, + TIMER_HPTIMER, NR_KVM_TIMERS }; @@ -21,6 +23,7 @@ enum kvm_arch_timer_regs { TIMER_REG_CVAL, TIMER_REG_TVAL, TIMER_REG_CTL, + TIMER_REG_VOFF, }; struct arch_timer_context { @@ -47,6 +50,7 @@ struct arch_timer_context { struct timer_map { struct arch_timer_context *direct_vtimer; struct arch_timer_context *direct_ptimer; + struct arch_timer_context *emul_vtimer; struct arch_timer_context *emul_ptimer; }; @@ -85,12 +89,12 @@ void kvm_timer_vcpu_put(struct kvm_vcpu *vcpu); void kvm_timer_init_vhe(void); -bool kvm_arch_timer_get_input_level(int vintid); - #define vcpu_timer(v) (&(v)->arch.timer_cpu) #define vcpu_get_timer(v,t) (&vcpu_timer(v)->timers[(t)]) #define vcpu_vtimer(v) (&(v)->arch.timer_cpu.timers[TIMER_VTIMER]) #define vcpu_ptimer(v) (&(v)->arch.timer_cpu.timers[TIMER_PTIMER]) +#define vcpu_hvtimer(v) (&(v)->arch.timer_cpu.timers[TIMER_HVTIMER]) +#define vcpu_hptimer(v) (&(v)->arch.timer_cpu.timers[TIMER_HPTIMER]) #define arch_timer_ctx_index(ctx) ((ctx) - vcpu_timer((ctx)->vcpu)->timers) diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h index e602d848fc1a..510122b68d8a 100644 --- a/include/kvm/arm_vgic.h +++ b/include/kvm/arm_vgic.h @@ -373,6 +373,7 @@ int kvm_vgic_inject_irq(struct kvm *kvm, int cpuid, unsigned int intid, int kvm_vgic_map_phys_irq(struct kvm_vcpu *vcpu, unsigned int host_irq, u32 vintid, struct irq_ops *ops); int kvm_vgic_unmap_phys_irq(struct kvm_vcpu *vcpu, unsigned int vintid); +int kvm_vgic_get_map(struct kvm_vcpu *vcpu, unsigned int vintid); bool kvm_vgic_map_is_active(struct kvm_vcpu *vcpu, unsigned int vintid); int kvm_vgic_vcpu_pending_irq(struct kvm_vcpu *vcpu); From patchwork Mon Nov 29 20:01:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645903 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C1830C433EF for ; Mon, 29 Nov 2021 22:20:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232303AbhK2WXa (ORCPT ); Mon, 29 Nov 2021 17:23:30 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56732 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233188AbhK2WVc (ORCPT ); Mon, 29 Nov 2021 17:21:32 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D5FCFC08EB4D for ; Mon, 29 Nov 2021 12:07:50 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 9E64BB815DA for ; Mon, 29 Nov 2021 20:07:49 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5C1F3C53FAD; Mon, 29 Nov 2021 20:07:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216468; bh=UU8kyF9hlHMIZSfl8J6xiEEU78BrTsnLzvIg32swVzY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=no7E06Jq5bDJD19btz4Yqa2+jGaAmcgUlA130HqtVivtDMns+VSxjFokHJzu47D+B hi3Ls5x9HSebNT3vwd3PihXwlT2RGP7ZZR6vJRCzr30vXDe9ulcKQclCTfw4LZ2pE6 M73Y8e89SUUfBkxnoMwJd9adgKidRWJwguNfl0zqj33l5iLrRYUp4Lo4bTHROZClGx bEx90oDP6pXXGQelf2J9VsfJDMdFEKQbrYE3Ffnxk/jHg1CBJBccuIcJ0RatM/JDup twmIK7TjX46gU+xGz54OxPzlE+61vjDQ19vU1uDuHhL/KOCHRoqAE2u+dzxtF+SiPE cqOVpFmGqTpSw== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmr7-008gvR-Q7; Mon, 29 Nov 2021 20:02:31 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 49/69] KVM: arm64: nv: Add handling of EL2-specific timer registers Date: Mon, 29 Nov 2021 20:01:30 +0000 Message-Id: <20211129200150.351436-50-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Add the required handling for EL2 and EL02 registers, as well as EL1 registers used in the E2H context. Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/sysreg.h | 6 +++ arch/arm64/kvm/sys_regs.c | 87 +++++++++++++++++++++++++++++++++ 2 files changed, 93 insertions(+) diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h index 199333b02ece..71e6a0410e7c 100644 --- a/arch/arm64/include/asm/sysreg.h +++ b/arch/arm64/include/asm/sysreg.h @@ -625,6 +625,12 @@ #define SYS_CNTVOFF_EL2 sys_reg(3, 4, 14, 0, 3) #define SYS_CNTHCTL_EL2 sys_reg(3, 4, 14, 1, 0) +#define SYS_CNTHP_TVAL_EL2 sys_reg(3, 4, 14, 2, 0) +#define SYS_CNTHP_CTL_EL2 sys_reg(3, 4, 14, 2, 1) +#define SYS_CNTHP_CVAL_EL2 sys_reg(3, 4, 14, 2, 2) +#define SYS_CNTHV_TVAL_EL2 sys_reg(3, 4, 14, 3, 0) +#define SYS_CNTHV_CTL_EL2 sys_reg(3, 4, 14, 3, 1) +#define SYS_CNTHV_CVAL_EL2 sys_reg(3, 4, 14, 3, 2) /* VHE encodings for architectural EL0/1 system registers */ #define SYS_SCTLR_EL12 sys_reg(3, 5, 1, 0, 0) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index e6c9a01670c9..4415b03aa278 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -1311,20 +1311,92 @@ static bool access_arch_timer(struct kvm_vcpu *vcpu, switch (reg) { case SYS_CNTP_TVAL_EL0: + if (vcpu_mode_el2(vcpu) && vcpu_el2_e2h_is_set(vcpu)) + tmr = TIMER_HPTIMER; + else + tmr = TIMER_PTIMER; + treg = TIMER_REG_TVAL; + break; + case SYS_AARCH32_CNTP_TVAL: + case SYS_CNTP_TVAL_EL02: tmr = TIMER_PTIMER; treg = TIMER_REG_TVAL; break; + + case SYS_CNTV_TVAL_EL02: + tmr = TIMER_VTIMER; + treg = TIMER_REG_TVAL; + break; + + case SYS_CNTHP_TVAL_EL2: + tmr = TIMER_HPTIMER; + treg = TIMER_REG_TVAL; + break; + + case SYS_CNTHV_TVAL_EL2: + tmr = TIMER_HVTIMER; + treg = TIMER_REG_TVAL; + break; + case SYS_CNTP_CTL_EL0: + if (vcpu_mode_el2(vcpu) && vcpu_el2_e2h_is_set(vcpu)) + tmr = TIMER_HPTIMER; + else + tmr = TIMER_PTIMER; + treg = TIMER_REG_CTL; + break; + case SYS_AARCH32_CNTP_CTL: + case SYS_CNTP_CTL_EL02: tmr = TIMER_PTIMER; treg = TIMER_REG_CTL; break; + + case SYS_CNTV_CTL_EL02: + tmr = TIMER_VTIMER; + treg = TIMER_REG_CTL; + break; + + case SYS_CNTHP_CTL_EL2: + tmr = TIMER_HPTIMER; + treg = TIMER_REG_CTL; + break; + + case SYS_CNTHV_CTL_EL2: + tmr = TIMER_HVTIMER; + treg = TIMER_REG_CTL; + break; + case SYS_CNTP_CVAL_EL0: + if (vcpu_mode_el2(vcpu) && vcpu_el2_e2h_is_set(vcpu)) + tmr = TIMER_HPTIMER; + else + tmr = TIMER_PTIMER; + treg = TIMER_REG_CVAL; + break; + case SYS_AARCH32_CNTP_CVAL: + case SYS_CNTP_CVAL_EL02: tmr = TIMER_PTIMER; treg = TIMER_REG_CVAL; break; + + case SYS_CNTV_CVAL_EL02: + tmr = TIMER_VTIMER; + treg = TIMER_REG_CVAL; + break; + + case SYS_CNTHP_CVAL_EL2: + tmr = TIMER_HPTIMER; + treg = TIMER_REG_CVAL; + break; + + case SYS_CNTHV_CVAL_EL2: + tmr = TIMER_HVTIMER; + treg = TIMER_REG_CVAL; + break; + case SYS_CNTVOFF_EL2: tmr = TIMER_VTIMER; treg = TIMER_REG_VOFF; @@ -2230,6 +2302,13 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_CNTVOFF_EL2), access_arch_timer }, { SYS_DESC(SYS_CNTHCTL_EL2), access_rw, reset_val, CNTHCTL_EL2, 0 }, + { SYS_DESC(SYS_CNTHP_TVAL_EL2), access_arch_timer }, + { SYS_DESC(SYS_CNTHP_CTL_EL2), access_arch_timer }, + { SYS_DESC(SYS_CNTHP_CVAL_EL2), access_arch_timer }, + { SYS_DESC(SYS_CNTHV_TVAL_EL2), access_arch_timer }, + { SYS_DESC(SYS_CNTHV_CTL_EL2), access_arch_timer }, + { SYS_DESC(SYS_CNTHV_CVAL_EL2), access_arch_timer }, + { SYS_DESC(SYS_SCTLR_EL12), access_vm_reg, reset_val, SCTLR_EL1, 0x00C50078 }, { SYS_DESC(SYS_CPACR_EL12), access_rw, reset_val, CPACR_EL1, 0 }, { SYS_DESC(SYS_TTBR0_EL12), access_vm_reg, reset_unknown, TTBR0_EL1 }, @@ -2247,6 +2326,14 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_CONTEXTIDR_EL12), access_vm_reg, reset_val, CONTEXTIDR_EL1, 0 }, { SYS_DESC(SYS_CNTKCTL_EL12), access_rw, reset_val, CNTKCTL_EL1, 0 }, + { SYS_DESC(SYS_CNTP_TVAL_EL02), access_arch_timer }, + { SYS_DESC(SYS_CNTP_CTL_EL02), access_arch_timer }, + { SYS_DESC(SYS_CNTP_CVAL_EL02), access_arch_timer }, + + { SYS_DESC(SYS_CNTV_TVAL_EL02), access_arch_timer }, + { SYS_DESC(SYS_CNTV_CTL_EL02), access_arch_timer }, + { SYS_DESC(SYS_CNTV_CVAL_EL02), access_arch_timer }, + { SYS_DESC(SYS_SP_EL2), NULL, reset_unknown, SP_EL2 }, }; From patchwork Mon Nov 29 20:01:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645715 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 723AFC433EF for ; Mon, 29 Nov 2021 21:24:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230018AbhK2V1s (ORCPT ); Mon, 29 Nov 2021 16:27:48 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45068 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230311AbhK2VZs (ORCPT ); Mon, 29 Nov 2021 16:25:48 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 12045C08EC8D for ; Mon, 29 Nov 2021 12:07:55 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 8E448CE140F for ; Mon, 29 Nov 2021 20:07:54 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C19B3C53FCD; Mon, 29 Nov 2021 20:07:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216472; bh=0ZrEv8b2Hv1ieoyvth57uqiT5TKdSAV9jYm5QmTCATA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=FmHjcbQKxlNggiDIPP1qU4HA8D+B0bA245z1JOifpB4c+qScY12exHd/HxhAKe5r6 j56161/CRZ8OCyauOVFn3DmGFl0QUt/jLO3gGUdOCJN+c0F0S326lSw1Q3Wz0Auq8p hQuaitNhb4B3vMjlp3mdHMgkYWGc4DVcFVqExpJAnonoZBhgdXrXmOkaQcT9tdhObg fFqK8gmejLcDF6DxHV0SNPb9a7shS/fgpINpkaTggu7R4TmTfO1Yu2V6ljM2soQAfg NetSF5pJiXH2i39fu2oLpsQWsYDllTMn0YX8+ofKkozKY4/RRIi84P/Fp53XAXRD/a NVJl7ejAaUcgA== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmr9-008gvR-OD; Mon, 29 Nov 2021 20:02:32 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 50/69] KVM: arm64: nv: Load timer before the GIC Date: Mon, 29 Nov 2021 20:01:31 +0000 Message-Id: <20211129200150.351436-51-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org In order for vgic_v3_load_nested to be able to observe which timer interrupts have the HW bit set for the current context, the timers must have been loaded in the new mode and the right timer mapped to their corresponding HW IRQs. At the moment, we load the GIC first, meaning that timer interrupts injected to an L2 guest will never have the HW bit set (we see the old configuration). Swapping the two loads solves this particular problem. Signed-off-by: Marc Zyngier --- arch/arm64/kvm/arm.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 1dbf63319b99..b04616bf1839 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -414,8 +414,8 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu) vcpu->cpu = cpu; - kvm_vgic_load(vcpu); kvm_timer_vcpu_load(vcpu); + kvm_vgic_load(vcpu); if (has_vhe()) kvm_vcpu_load_sysregs_vhe(vcpu); kvm_arch_vcpu_load_fp(vcpu); From patchwork Mon Nov 29 20:01:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645721 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CA4ECC433F5 for ; Mon, 29 Nov 2021 21:24:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229868AbhK2V17 (ORCPT ); Mon, 29 Nov 2021 16:27:59 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44202 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231151AbhK2VZu (ORCPT ); Mon, 29 Nov 2021 16:25:50 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DF00CC091D36 for ; Mon, 29 Nov 2021 12:07:31 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 83BA3B8159E for ; Mon, 29 Nov 2021 20:07:30 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id F1F4BC53FAD; Mon, 29 Nov 2021 20:07:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216449; bh=8cdoFre0JzsPC1BQ92v3Ji0LSjERCSUVm5O2TBaSd3w=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=UfDK3tLWOXa/qip59SWFfwuCJMXPMeHoUojQlrmhzIamJ7o5alxrCqyFDymq6uFCB 7W/rEXXIu9Qn4aSf6mUcUx5x3kQlNe1xC18Dn0VCx9xRvqDA3ecoH4GO6wUHTXPtxq AvbLGLpRABGz/MgDAXd4Xi1/tBZJqghM6LFKl0OdEoJJ9eNrUCKkjgvkdj+b83LNuN 6XlVwnEGEAnY4lmAPEpjF81M47qqQZWM5wZQ6OQXNqyAYGr9xdNlxOg4jw6TnExYTt ifJxaRptvIBFAKCMvW4z2Oits6XFuN9J1a5xRtUmhQwK/peLHIpcEf0D/F+ww3Uea+ hiTi3VLjx1gYA== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmrA-008gvR-Q8; Mon, 29 Nov 2021 20:02:32 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 51/69] KVM: arm64: nv: Nested GICv3 Support Date: Mon, 29 Nov 2021 20:01:32 +0000 Message-Id: <20211129200150.351436-52-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Jintack Lim When entering a nested VM, we set up the hypervisor control interface based on what the guest hypervisor has set. Especially, we investigate each list register written by the guest hypervisor whether HW bit is set. If so, we translate hw irq number from the guest's point of view to the real hardware irq number if there is a mapping. Signed-off-by: Jintack Lim [Redesigned execution flow around vcpu load/put] Signed-off-by: Christoffer Dall [Rewritten to support GICv3 instead of GICv2] Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_emulate.h | 8 +- arch/arm64/include/asm/kvm_host.h | 11 +- arch/arm64/include/asm/kvm_nested.h | 1 + arch/arm64/kvm/Makefile | 2 +- arch/arm64/kvm/arm.c | 13 +- arch/arm64/kvm/nested.c | 16 +++ arch/arm64/kvm/sys_regs.c | 179 ++++++++++++++++++++++++++- arch/arm64/kvm/vgic/vgic-v3-nested.c | 178 ++++++++++++++++++++++++++ arch/arm64/kvm/vgic/vgic-v3.c | 26 ++++ arch/arm64/kvm/vgic/vgic.c | 27 ++++ include/kvm/arm_vgic.h | 18 +++ 11 files changed, 469 insertions(+), 10 deletions(-) create mode 100644 arch/arm64/kvm/vgic/vgic-v3-nested.c diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h index 94d8814b7c8b..1664430be698 100644 --- a/arch/arm64/include/asm/kvm_emulate.h +++ b/arch/arm64/include/asm/kvm_emulate.h @@ -505,7 +505,13 @@ static inline bool kvm_is_write_fault(struct kvm_vcpu *vcpu) static inline unsigned long kvm_vcpu_get_mpidr_aff(struct kvm_vcpu *vcpu) { - return vcpu_read_sys_reg(vcpu, MPIDR_EL1) & MPIDR_HWID_BITMASK; + /* + * Use the in-memory view for MPIDR_EL1. It can't be changed by the + * guest, and is also accessed from the context of *another* vcpu, + * so anything using some other state (such as the NV state that is + * used by vcpu_read_sys_reg) will eventually go wrong. + */ + return __vcpu_sys_reg(vcpu, MPIDR_EL1) & MPIDR_HWID_BITMASK; } static inline void kvm_vcpu_set_be(struct kvm_vcpu *vcpu) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 96edd6348bd1..323215873388 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -42,11 +42,12 @@ #define KVM_REQ_SLEEP \ KVM_ARCH_REQ_FLAGS(0, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP) -#define KVM_REQ_IRQ_PENDING KVM_ARCH_REQ(1) -#define KVM_REQ_VCPU_RESET KVM_ARCH_REQ(2) -#define KVM_REQ_RECORD_STEAL KVM_ARCH_REQ(3) -#define KVM_REQ_RELOAD_GICv4 KVM_ARCH_REQ(4) -#define KVM_REQ_RELOAD_PMU KVM_ARCH_REQ(5) +#define KVM_REQ_IRQ_PENDING KVM_ARCH_REQ(1) +#define KVM_REQ_VCPU_RESET KVM_ARCH_REQ(2) +#define KVM_REQ_RECORD_STEAL KVM_ARCH_REQ(3) +#define KVM_REQ_RELOAD_GICv4 KVM_ARCH_REQ(4) +#define KVM_REQ_RELOAD_PMU KVM_ARCH_REQ(5) +#define KVM_REQ_GUEST_HYP_IRQ_PENDING KVM_ARCH_REQ(6) #define KVM_DIRTY_LOG_MANUAL_CAPS (KVM_DIRTY_LOG_MANUAL_PROTECT_ENABLE | \ KVM_DIRTY_LOG_INITIALLY_SET) diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index 2987806850f0..5fa3c634c8e1 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -67,6 +67,7 @@ extern void kvm_init_nested_s2_mmu(struct kvm_s2_mmu *mmu); extern struct kvm_s2_mmu *lookup_s2_mmu(struct kvm *kvm, u64 vttbr, u64 hcr); extern void kvm_vcpu_load_hw_mmu(struct kvm_vcpu *vcpu); extern void kvm_vcpu_put_hw_mmu(struct kvm_vcpu *vcpu); +extern void check_nested_vcpu_requests(struct kvm_vcpu *vcpu); struct kvm_s2_trans { phys_addr_t output; diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile index 8e19da1a07e1..399b19d3a64a 100644 --- a/arch/arm64/kvm/Makefile +++ b/arch/arm64/kvm/Makefile @@ -22,6 +22,6 @@ kvm-y := $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/eventfd.o \ vgic/vgic-v3.o vgic/vgic-v4.o \ vgic/vgic-mmio.o vgic/vgic-mmio-v2.o \ vgic/vgic-mmio-v3.o vgic/vgic-kvm-device.o \ - vgic/vgic-its.o vgic/vgic-debug.o + vgic/vgic-its.o vgic/vgic-debug.o vgic/vgic-v3-nested.o kvm-$(CONFIG_HW_PERF_EVENTS) += pmu-emul.o diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index b04616bf1839..0bb46bce5ffb 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -715,6 +715,8 @@ static void check_vcpu_requests(struct kvm_vcpu *vcpu) if (kvm_check_request(KVM_REQ_RELOAD_PMU, vcpu)) kvm_pmu_handle_pmcr(vcpu, __vcpu_sys_reg(vcpu, PMCR_EL0)); + + check_nested_vcpu_requests(vcpu); } } @@ -813,9 +815,16 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) if (!ret) ret = 1; - update_vmid(&vcpu->arch.hw_mmu->vmid); - + /* + * A nested exeption triggered by a vcpu request (such + * as an interrupt injected in a guest hypervisor) can + * change the currently used VMID (by switching to a + * different translation regime. It is thus necesary + * to update the VMID *after* all requests have been + * processed. + */ check_vcpu_requests(vcpu); + update_vmid(&vcpu->arch.hw_mmu->vmid); /* * Preparing the interrupts to be injected also diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c index b7e10f6178b9..198169648c3c 100644 --- a/arch/arm64/kvm/nested.c +++ b/arch/arm64/kvm/nested.c @@ -592,6 +592,22 @@ void kvm_arch_flush_shadow_all(struct kvm *kvm) kvm_free_stage2_pgd(&kvm->arch.mmu); } +bool vgic_state_is_nested(struct kvm_vcpu *vcpu) +{ + bool imo = __vcpu_sys_reg(vcpu, HCR_EL2) & HCR_IMO; + bool fmo = __vcpu_sys_reg(vcpu, HCR_EL2) & HCR_FMO; + + WARN_ONCE(imo != fmo, "Separate virtual IRQ/FIQ settings not supported\n"); + + return nested_virt_in_use(vcpu) && imo && fmo && !is_hyp_ctxt(vcpu); +} + +void check_nested_vcpu_requests(struct kvm_vcpu *vcpu) +{ + if (kvm_check_request(KVM_REQ_GUEST_HYP_IRQ_PENDING, vcpu)) + kvm_inject_nested_irq(vcpu); +} + /* * Our emulated CPU doesn't support all the possible features. For the * sake of simplicity (and probably mental sanity), wipe out a number diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 4415b03aa278..55f3e94c24f1 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -16,6 +16,8 @@ #include #include +#include + #include #include #include @@ -444,6 +446,19 @@ static bool access_actlr(struct kvm_vcpu *vcpu, return true; } +/* + * The architecture says that non-secure write accesses to this register from + * EL1 are trapped to EL2, if either: + * - HCR_EL2.FMO==1, or + * - HCR_EL2.IMO==1 + */ +static bool sgi_traps_to_vel2(struct kvm_vcpu *vcpu) +{ + return (nested_virt_in_use(vcpu) && + !vcpu_mode_el2(vcpu) && + !!(vcpu_read_sys_reg(vcpu, HCR_EL2) & (HCR_IMO | HCR_FMO))); +} + /* * Trap handler for the GICv3 SGI generation system register. * Forward the request to the VGIC emulation. @@ -459,6 +474,11 @@ static bool access_gic_sgi(struct kvm_vcpu *vcpu, if (!p->is_write) return read_from_write_only(vcpu, p, r); + if (sgi_traps_to_vel2(vcpu)) { + kvm_inject_nested_sync(vcpu, kvm_vcpu_get_esr(vcpu)); + return false; + } + /* * In a system where GICD_CTLR.DS=1, a ICC_SGI0R_EL1 access generates * Group0 SGIs only, while ICC_SGI1R_EL1 can generate either group, @@ -502,7 +522,13 @@ static bool access_gic_sre(struct kvm_vcpu *vcpu, if (p->is_write) return ignore_write(vcpu, p); - p->regval = vcpu->arch.vgic_cpu.vgic_v3.vgic_sre; + if (p->Op1 == 4) { /* ICC_SRE_EL2 */ + p->regval = (ICC_SRE_EL2_ENABLE | ICC_SRE_EL2_SRE | + ICC_SRE_EL1_DIB | ICC_SRE_EL1_DFB); + } else { /* ICC_SRE_EL1 */ + p->regval = vcpu->arch.vgic_cpu.vgic_v3.vgic_sre; + } + return true; } @@ -1844,6 +1870,122 @@ static bool access_spsr_el2(struct kvm_vcpu *vcpu, return true; } +static bool access_gic_apr(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.nested_vgic_v3; + u32 index, *base; + + index = r->Op2; + if (r->CRm == 8) + base = cpu_if->vgic_ap0r; + else + base = cpu_if->vgic_ap1r; + + if (p->is_write) + base[index] = p->regval; + else + p->regval = base[index]; + + return true; +} + +static bool access_gic_hcr(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.nested_vgic_v3; + + if (p->is_write) + cpu_if->vgic_hcr = p->regval; + else + p->regval = cpu_if->vgic_hcr; + + return true; +} + +static bool access_gic_vtr(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + if (p->is_write) + return write_to_read_only(vcpu, p, r); + + p->regval = kvm_vgic_global_state.ich_vtr_el2; + + return true; +} + +static bool access_gic_misr(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + if (p->is_write) + return write_to_read_only(vcpu, p, r); + + p->regval = vgic_v3_get_misr(vcpu); + + return true; +} + +static bool access_gic_eisr(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + if (p->is_write) + return write_to_read_only(vcpu, p, r); + + p->regval = vgic_v3_get_eisr(vcpu); + + return true; +} + +static bool access_gic_elrsr(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + if (p->is_write) + return write_to_read_only(vcpu, p, r); + + p->regval = vgic_v3_get_elrsr(vcpu); + + return true; +} + +static bool access_gic_vmcr(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.nested_vgic_v3; + + if (p->is_write) + cpu_if->vgic_vmcr = p->regval; + else + p->regval = cpu_if->vgic_vmcr; + + return true; +} + +static bool access_gic_lr(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.nested_vgic_v3; + u32 index; + + index = p->Op2; + if (p->CRm == 13) + index += 8; + + if (p->is_write) + cpu_if->vgic_lr[index] = p->regval; + else + p->regval = cpu_if->vgic_lr[index]; + + return true; +} + /* * Architected system registers. * Important: Must be sorted ascending by Op0, Op1, CRn, CRm, Op2 @@ -2296,6 +2438,41 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_RMR_EL2), trap_undef }, { SYS_DESC(SYS_VDISR_EL2), trap_undef }, + { SYS_DESC(SYS_ICH_AP0R0_EL2), access_gic_apr }, + { SYS_DESC(SYS_ICH_AP0R1_EL2), access_gic_apr }, + { SYS_DESC(SYS_ICH_AP0R2_EL2), access_gic_apr }, + { SYS_DESC(SYS_ICH_AP0R3_EL2), access_gic_apr }, + { SYS_DESC(SYS_ICH_AP1R0_EL2), access_gic_apr }, + { SYS_DESC(SYS_ICH_AP1R1_EL2), access_gic_apr }, + { SYS_DESC(SYS_ICH_AP1R2_EL2), access_gic_apr }, + { SYS_DESC(SYS_ICH_AP1R3_EL2), access_gic_apr }, + + { SYS_DESC(SYS_ICC_SRE_EL2), access_gic_sre }, + + { SYS_DESC(SYS_ICH_HCR_EL2), access_gic_hcr }, + { SYS_DESC(SYS_ICH_VTR_EL2), access_gic_vtr }, + { SYS_DESC(SYS_ICH_MISR_EL2), access_gic_misr }, + { SYS_DESC(SYS_ICH_EISR_EL2), access_gic_eisr }, + { SYS_DESC(SYS_ICH_ELRSR_EL2), access_gic_elrsr }, + { SYS_DESC(SYS_ICH_VMCR_EL2), access_gic_vmcr }, + + { SYS_DESC(SYS_ICH_LR0_EL2), access_gic_lr }, + { SYS_DESC(SYS_ICH_LR1_EL2), access_gic_lr }, + { SYS_DESC(SYS_ICH_LR2_EL2), access_gic_lr }, + { SYS_DESC(SYS_ICH_LR3_EL2), access_gic_lr }, + { SYS_DESC(SYS_ICH_LR4_EL2), access_gic_lr }, + { SYS_DESC(SYS_ICH_LR5_EL2), access_gic_lr }, + { SYS_DESC(SYS_ICH_LR6_EL2), access_gic_lr }, + { SYS_DESC(SYS_ICH_LR7_EL2), access_gic_lr }, + { SYS_DESC(SYS_ICH_LR8_EL2), access_gic_lr }, + { SYS_DESC(SYS_ICH_LR9_EL2), access_gic_lr }, + { SYS_DESC(SYS_ICH_LR10_EL2), access_gic_lr }, + { SYS_DESC(SYS_ICH_LR11_EL2), access_gic_lr }, + { SYS_DESC(SYS_ICH_LR12_EL2), access_gic_lr }, + { SYS_DESC(SYS_ICH_LR13_EL2), access_gic_lr }, + { SYS_DESC(SYS_ICH_LR14_EL2), access_gic_lr }, + { SYS_DESC(SYS_ICH_LR15_EL2), access_gic_lr }, + { SYS_DESC(SYS_CONTEXTIDR_EL2), access_rw, reset_val, CONTEXTIDR_EL2, 0 }, { SYS_DESC(SYS_TPIDR_EL2), access_rw, reset_val, TPIDR_EL2, 0 }, diff --git a/arch/arm64/kvm/vgic/vgic-v3-nested.c b/arch/arm64/kvm/vgic/vgic-v3-nested.c new file mode 100644 index 000000000000..bfe0ccde514f --- /dev/null +++ b/arch/arm64/kvm/vgic/vgic-v3-nested.c @@ -0,0 +1,178 @@ +#include +#include +#include +#include +#include +#include + +#include + +#include +#include +#include + +#include "vgic.h" + +static inline struct vgic_v3_cpu_if *vcpu_nested_if(struct kvm_vcpu *vcpu) +{ + return &vcpu->arch.vgic_cpu.nested_vgic_v3; +} + +static inline struct vgic_v3_cpu_if *vcpu_shadow_if(struct kvm_vcpu *vcpu) +{ + return &vcpu->arch.vgic_cpu.shadow_vgic_v3; +} + +static inline bool lr_triggers_eoi(u64 lr) +{ + return !(lr & (ICH_LR_STATE | ICH_LR_HW)) && (lr & ICH_LR_EOI); +} + +u16 vgic_v3_get_eisr(struct kvm_vcpu *vcpu) +{ + struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); + u16 reg = 0; + int i; + + for (i = 0; i < kvm_vgic_global_state.nr_lr; i++) { + if (lr_triggers_eoi(cpu_if->vgic_lr[i])) + reg |= BIT(i); + } + + return reg; +} + +u16 vgic_v3_get_elrsr(struct kvm_vcpu *vcpu) +{ + struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); + u16 reg = 0; + int i; + + for (i = 0; i < kvm_vgic_global_state.nr_lr; i++) { + if (!(cpu_if->vgic_lr[i] & ICH_LR_STATE)) + reg |= BIT(i); + } + + return reg; +} + +u64 vgic_v3_get_misr(struct kvm_vcpu *vcpu) +{ + struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); + int nr_lr = kvm_vgic_global_state.nr_lr; + u64 reg = 0; + + if (vgic_v3_get_eisr(vcpu)) + reg |= ICH_MISR_EOI; + + if (cpu_if->vgic_hcr & ICH_HCR_UIE) { + int used_lrs; + + used_lrs = nr_lr - hweight16(vgic_v3_get_elrsr(vcpu)); + if (used_lrs <= 1) + reg |= ICH_MISR_U; + } + + /* TODO: Support remaining bits in this register */ + return reg; +} + +/* + * For LRs which have HW bit set such as timer interrupts, we modify them to + * have the host hardware interrupt number instead of the virtual one programmed + * by the guest hypervisor. + */ +static void vgic_v3_create_shadow_lr(struct kvm_vcpu *vcpu) +{ + struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); + struct vgic_v3_cpu_if *s_cpu_if = vcpu_shadow_if(vcpu); + struct vgic_irq *irq; + int i, used_lrs = 0; + + for (i = 0; i < kvm_vgic_global_state.nr_lr; i++) { + u64 lr = cpu_if->vgic_lr[i]; + int l1_irq; + + if (!(lr & ICH_LR_HW)) + goto next; + + /* We have the HW bit set */ + l1_irq = (lr & ICH_LR_PHYS_ID_MASK) >> ICH_LR_PHYS_ID_SHIFT; + irq = vgic_get_irq(vcpu->kvm, vcpu, l1_irq); + + if (!irq || !irq->hw) { + /* There was no real mapping, so nuke the HW bit */ + lr &= ~ICH_LR_HW; + if (irq) + vgic_put_irq(vcpu->kvm, irq); + goto next; + } + + /* Translate the virtual mapping to the real one */ + lr &= ~ICH_LR_EOI; /* Why? */ + lr &= ~ICH_LR_PHYS_ID_MASK; + lr |= (u64)irq->hwintid << ICH_LR_PHYS_ID_SHIFT; + vgic_put_irq(vcpu->kvm, irq); + +next: + s_cpu_if->vgic_lr[i] = lr; + used_lrs = i + 1; + } + + s_cpu_if->used_lrs = used_lrs; +} + +/* + * Change the shadow HWIRQ field back to the virtual value before copying over + * the entire shadow struct to the nested state. + */ +static void vgic_v3_fixup_shadow_lr_state(struct kvm_vcpu *vcpu) +{ + struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); + struct vgic_v3_cpu_if *s_cpu_if = vcpu_shadow_if(vcpu); + int lr; + + for (lr = 0; lr < kvm_vgic_global_state.nr_lr; lr++) { + s_cpu_if->vgic_lr[lr] &= ~ICH_LR_PHYS_ID_MASK; + s_cpu_if->vgic_lr[lr] |= cpu_if->vgic_lr[lr] & ICH_LR_PHYS_ID_MASK; + } +} + +void vgic_v3_load_nested(struct kvm_vcpu *vcpu) +{ + struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu; + + vgic_cpu->shadow_vgic_v3 = vgic_cpu->nested_vgic_v3; + vgic_v3_create_shadow_lr(vcpu); + __vgic_v3_restore_state(vcpu_shadow_if(vcpu)); +} + +void vgic_v3_put_nested(struct kvm_vcpu *vcpu) +{ + struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu; + + __vgic_v3_save_state(vcpu_shadow_if(vcpu)); + + /* + * Translate the shadow state HW fields back to the virtual ones + * before copying the shadow struct back to the nested one. + */ + vgic_v3_fixup_shadow_lr_state(vcpu); + vgic_cpu->nested_vgic_v3 = vgic_cpu->shadow_vgic_v3; +} + +void vgic_v3_handle_nested_maint_irq(struct kvm_vcpu *vcpu) +{ + struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); + + /* + * If we exit a nested VM with a pending maintenance interrupt from the + * GIC, then we need to forward this to the guest hypervisor so that it + * can re-sync the appropriate LRs and sample level triggered interrupts + * again. + */ + if (vgic_state_is_nested(vcpu) && + (cpu_if->vgic_hcr & ICH_HCR_EN) && + vgic_v3_get_misr(vcpu)) + kvm_inject_nested_irq(vcpu); +} diff --git a/arch/arm64/kvm/vgic/vgic-v3.c b/arch/arm64/kvm/vgic/vgic-v3.c index 04f62c4b07fb..ae4d6489373d 100644 --- a/arch/arm64/kvm/vgic/vgic-v3.c +++ b/arch/arm64/kvm/vgic/vgic-v3.c @@ -8,6 +8,7 @@ #include #include #include +#include #include #include "vgic.h" @@ -277,6 +278,12 @@ void vgic_v3_enable(struct kvm_vcpu *vcpu) vgic_v3->vgic_sre = (ICC_SRE_EL1_DIB | ICC_SRE_EL1_DFB | ICC_SRE_EL1_SRE); + /* + * If nesting is allowed, force GICv3 onto the nested + * guests as well. + */ + if (nested_virt_in_use(vcpu)) + vcpu->arch.vgic_cpu.nested_vgic_v3.vgic_sre = vgic_v3->vgic_sre; vcpu->arch.vgic_cpu.pendbaser = INITIAL_PENDBASER_VALUE; } else { vgic_v3->vgic_sre = 0; @@ -707,6 +714,13 @@ void vgic_v3_load(struct kvm_vcpu *vcpu) { struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v3; + /* + * vgic_v3_load_nested only affects the LRs in the shadow + * state, so it is fine to pass the nested state around. + */ + if (vgic_state_is_nested(vcpu)) + cpu_if = &vcpu->arch.vgic_cpu.nested_vgic_v3; + /* * If dealing with a GICv2 emulation on GICv3, VMCR_EL2.VFIQen * is dependent on ICC_SRE_EL1.SRE, and we have to perform the @@ -720,6 +734,9 @@ void vgic_v3_load(struct kvm_vcpu *vcpu) if (has_vhe()) __vgic_v3_activate_traps(cpu_if); + if (vgic_state_is_nested(vcpu)) + vgic_v3_load_nested(vcpu); + WARN_ON(vgic_v4_load(vcpu)); } @@ -727,6 +744,9 @@ void vgic_v3_vmcr_sync(struct kvm_vcpu *vcpu) { struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v3; + if (vgic_state_is_nested(vcpu)) + cpu_if = &vcpu->arch.vgic_cpu.shadow_vgic_v3; + if (likely(cpu_if->vgic_sre)) cpu_if->vgic_vmcr = kvm_call_hyp_ret(__vgic_v3_read_vmcr); } @@ -739,8 +759,14 @@ void vgic_v3_put(struct kvm_vcpu *vcpu) vgic_v3_vmcr_sync(vcpu); + if (vgic_state_is_nested(vcpu)) + cpu_if = &vcpu->arch.vgic_cpu.shadow_vgic_v3; + kvm_call_hyp(__vgic_v3_save_aprs, cpu_if); if (has_vhe()) __vgic_v3_deactivate_traps(cpu_if); + + if (vgic_state_is_nested(vcpu)) + vgic_v3_put_nested(vcpu); } diff --git a/arch/arm64/kvm/vgic/vgic.c b/arch/arm64/kvm/vgic/vgic.c index 21d088e8597d..6503aa208c9f 100644 --- a/arch/arm64/kvm/vgic/vgic.c +++ b/arch/arm64/kvm/vgic/vgic.c @@ -876,6 +876,10 @@ void kvm_vgic_sync_hwstate(struct kvm_vcpu *vcpu) { int used_lrs; + /* If nesting, this is a load/put affair, not flush/sync. */ + if (vgic_state_is_nested(vcpu)) + return; + /* An empty ap_list_head implies used_lrs == 0 */ if (list_empty(&vcpu->arch.vgic_cpu.ap_list_head)) return; @@ -920,6 +924,29 @@ void kvm_vgic_flush_hwstate(struct kvm_vcpu *vcpu) !vgic_supports_direct_msis(vcpu->kvm)) return; + /* + * If in a nested state, we must return early. Two possibilities: + * + * - If we have any pending IRQ for the guest and the guest + * expects IRQs to be handled in its virtual EL2 mode (the + * virtual IMO bit is set) and it is not already running in + * virtual EL2 mode, then we have to emulate an IRQ + * exception to virtual EL2. + * + * We do that by placing a request to ourselves which will + * abort the entry procedure and inject the exception at the + * beginning of the run loop. + * + * - Otherwise, do exactly *NOTHING*. The guest state is + * already loaded, and we can carry on with running it. + */ + if (vgic_state_is_nested(vcpu)) { + if (kvm_vgic_vcpu_pending_irq(vcpu)) + kvm_make_request(KVM_REQ_GUEST_HYP_IRQ_PENDING, vcpu); + + return; + } + DEBUG_SPINLOCK_BUG_ON(!irqs_disabled()); if (!list_empty(&vcpu->arch.vgic_cpu.ap_list_head)) { diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h index 510122b68d8a..a0db49e6c781 100644 --- a/include/kvm/arm_vgic.h +++ b/include/kvm/arm_vgic.h @@ -325,6 +325,15 @@ struct vgic_cpu { struct vgic_irq private_irqs[VGIC_NR_PRIVATE_IRQS]; + /* CPU vif control registers for the virtual GICH interface */ + struct vgic_v3_cpu_if nested_vgic_v3; + + /* + * The shadow vif control register loaded to the hardware when + * running a nested L2 guest with the virtual IMO/FMO bit set. + */ + struct vgic_v3_cpu_if shadow_vgic_v3; + raw_spinlock_t ap_list_lock; /* Protects the ap_list */ /* @@ -382,6 +391,13 @@ void kvm_vgic_load(struct kvm_vcpu *vcpu); void kvm_vgic_put(struct kvm_vcpu *vcpu); void kvm_vgic_vmcr_sync(struct kvm_vcpu *vcpu); +void vgic_v3_load_nested(struct kvm_vcpu *vcpu); +void vgic_v3_put_nested(struct kvm_vcpu *vcpu); +void vgic_v3_handle_nested_maint_irq(struct kvm_vcpu *vcpu); +u16 vgic_v3_get_eisr(struct kvm_vcpu *vcpu); +u16 vgic_v3_get_elrsr(struct kvm_vcpu *vcpu); +u64 vgic_v3_get_misr(struct kvm_vcpu *vcpu); + #define irqchip_in_kernel(k) (!!((k)->arch.vgic.in_kernel)) #define vgic_initialized(k) ((k)->arch.vgic.initialized) #define vgic_ready(k) ((k)->arch.vgic.ready) @@ -426,4 +442,6 @@ int vgic_v4_load(struct kvm_vcpu *vcpu); void vgic_v4_commit(struct kvm_vcpu *vcpu); int vgic_v4_put(struct kvm_vcpu *vcpu, bool need_db); +bool vgic_state_is_nested(struct kvm_vcpu *vcpu); + #endif /* __KVM_ARM_VGIC_H */ From patchwork Mon Nov 29 20:01:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645733 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6F4C9C433EF for ; Mon, 29 Nov 2021 21:42:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232108AbhK2Vp4 (ORCPT ); Mon, 29 Nov 2021 16:45:56 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48126 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232134AbhK2Vnw (ORCPT ); Mon, 29 Nov 2021 16:43:52 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9EDE5C091D10 for ; Mon, 29 Nov 2021 12:05:35 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 990D7CE140D for ; Mon, 29 Nov 2021 20:05:33 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C86F7C53FD1; Mon, 29 Nov 2021 20:05:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216331; bh=tTxcULXoo3QMlH9nFuTANsqJJDvpHaZXohlcPR2wbWA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=q3YpLz8+uJ97q1qgocMm/qBME7Or4leBztDeHl8LLfcYLQwYdiEJ/8K9Vb83elpLS mFO249KYbRxRrYifgBfI5Nvv7LmfU4phIFVF7Ye9Ag8Sj/b6phUu9/IStJqnZ5y4Um sDvsEQ4gkFhh9GyNvalpFl0MQGdhpbBCCIq1n4KZZlsJereKlXEeG5+KIKkKnR8g9/ zF7RF0iJRzkkLLgNxnmJDgmJdAat2KUflypGZqxwkZU4OiIlsXHQ2ot9PSDIBAYgb0 xSesnCXlh9OHCvSAnmx+9V/TOlFItkKjBsuDlc0as9QyqpcJpLkfE2ObRRZ8tm79GS DPic66p0ICqXA== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmrB-008gvR-64; Mon, 29 Nov 2021 20:02:33 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 52/69] KVM: arm64: nv: Don't load the GICv4 context on entering a nested guest Date: Mon, 29 Nov 2021 20:01:33 +0000 Message-Id: <20211129200150.351436-53-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org When entering a nested guest (vgic_state_is_nested() == true), special care must be taken *not* to make the vPE resident, as these are interrupts targetting the L1 guest, and not any nested guest. By not making the vPE resident, we guarantee that the delivery of an vLPI will result in a doorbell, forcing an exit from the nested guest and a switch to the L1 guest to handle the interrupt. Signed-off-by: Marc Zyngier --- arch/arm64/kvm/vgic/vgic-v3.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/arch/arm64/kvm/vgic/vgic-v3.c b/arch/arm64/kvm/vgic/vgic-v3.c index ae4d6489373d..7b575df62fa9 100644 --- a/arch/arm64/kvm/vgic/vgic-v3.c +++ b/arch/arm64/kvm/vgic/vgic-v3.c @@ -736,8 +736,8 @@ void vgic_v3_load(struct kvm_vcpu *vcpu) if (vgic_state_is_nested(vcpu)) vgic_v3_load_nested(vcpu); - - WARN_ON(vgic_v4_load(vcpu)); + else + WARN_ON(vgic_v4_load(vcpu)); } void vgic_v3_vmcr_sync(struct kvm_vcpu *vcpu) @@ -755,6 +755,12 @@ void vgic_v3_put(struct kvm_vcpu *vcpu) { struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v3; + /* + * vgic_v4_put will do nothing if we were not resident. This + * covers both the cases where we've blocked (we already have + * done a vgic_v4_put) and when running a nested guest (the + * vPE was never resident in order to generate a doorbell). + */ WARN_ON(vgic_v4_put(vcpu, false)); vgic_v3_vmcr_sync(vcpu); From patchwork Mon Nov 29 20:01:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645757 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C50DEC433F5 for ; Mon, 29 Nov 2021 21:44:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232770AbhK2Vr6 (ORCPT ); Mon, 29 Nov 2021 16:47:58 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49450 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232072AbhK2Vpw (ORCPT ); Mon, 29 Nov 2021 16:45:52 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 180BAC08EB38 for ; Mon, 29 Nov 2021 12:06:27 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id D466EB815EB for ; Mon, 29 Nov 2021 20:06:25 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 96F66C53FC7; Mon, 29 Nov 2021 20:06:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216384; bh=hQjYY7dIibxjv8EfQGlPyiMMiLP5soaWxDM3zMZFfu0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=l0bJ9mYp6fYzJKROrE4sSorG38/awt0U0m6AbwzlhTWJtCUU2Rfk5FChhFUdjbXk7 avdT1DPfCz7WUEvDfxesmId9WC14pE786gCVfiYJAIFYPBO4zRvyAgWV8s00rk6ZsI DIpT9EYHSpoWCg55W9kSVmZAj5qUgQSoZuuJ9tXkQHNtc5Kgdg/XfL82V4JpsV598z omT0GzedG9Woibkdebcz3WNGbpXX2RZjyceGhOKk9/m8/i8ka/+cNu4mln5JqyunAc EY8FcdY2qOCrzs7GLZDmRT78CCXDjVSuStcZvJyu7RsFNL9r7PpTsUPom2GfBZioeg zTDOAcxzL3W+g== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmrB-008gvR-FG; Mon, 29 Nov 2021 20:02:33 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 53/69] KVM: arm64: nv: vgic: Emulate the HW bit in software Date: Mon, 29 Nov 2021 20:01:34 +0000 Message-Id: <20211129200150.351436-54-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Christoffer Dall Should the guest hypervisor use the HW bit in the LRs, we need to emulate the deactivation from the L2 guest into the L1 distributor emulation, which is handled by L0. It's all good fun. Signed-off-by: Christoffer Dall Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_hyp.h | 2 ++ arch/arm64/kvm/hyp/vgic-v3-sr.c | 2 +- arch/arm64/kvm/vgic/vgic-v3-nested.c | 32 ++++++++++++++++++++++++++++ arch/arm64/kvm/vgic/vgic.c | 6 ++++-- include/kvm/arm_vgic.h | 1 + 5 files changed, 40 insertions(+), 3 deletions(-) diff --git a/arch/arm64/include/asm/kvm_hyp.h b/arch/arm64/include/asm/kvm_hyp.h index 5afd14ab15b9..0d1bd78c2179 100644 --- a/arch/arm64/include/asm/kvm_hyp.h +++ b/arch/arm64/include/asm/kvm_hyp.h @@ -57,6 +57,8 @@ DECLARE_PER_CPU(struct kvm_nvhe_init_params, kvm_init_params); int __vgic_v2_perform_cpuif_access(struct kvm_vcpu *vcpu); +u64 __gic_v3_get_lr(unsigned int lr); + void __vgic_v3_save_state(struct vgic_v3_cpu_if *cpu_if); void __vgic_v3_restore_state(struct vgic_v3_cpu_if *cpu_if); void __vgic_v3_activate_traps(struct vgic_v3_cpu_if *cpu_if); diff --git a/arch/arm64/kvm/hyp/vgic-v3-sr.c b/arch/arm64/kvm/hyp/vgic-v3-sr.c index 20db2f281cf2..0601d73de11b 100644 --- a/arch/arm64/kvm/hyp/vgic-v3-sr.c +++ b/arch/arm64/kvm/hyp/vgic-v3-sr.c @@ -18,7 +18,7 @@ #define vtr_to_nr_pre_bits(v) ((((u32)(v) >> 26) & 7) + 1) #define vtr_to_nr_apr_regs(v) (1 << (vtr_to_nr_pre_bits(v) - 5)) -static u64 __gic_v3_get_lr(unsigned int lr) +u64 __gic_v3_get_lr(unsigned int lr) { switch (lr & 0xf) { case 0: diff --git a/arch/arm64/kvm/vgic/vgic-v3-nested.c b/arch/arm64/kvm/vgic/vgic-v3-nested.c index bfe0ccde514f..3ee3438b5e22 100644 --- a/arch/arm64/kvm/vgic/vgic-v3-nested.c +++ b/arch/arm64/kvm/vgic/vgic-v3-nested.c @@ -138,6 +138,38 @@ static void vgic_v3_fixup_shadow_lr_state(struct kvm_vcpu *vcpu) } } +void vgic_v3_sync_nested(struct kvm_vcpu *vcpu) +{ + struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); + struct vgic_v3_cpu_if *s_cpu_if = vcpu_shadow_if(vcpu); + struct vgic_irq *irq; + int i; + + for (i = 0; i < s_cpu_if->used_lrs; i++) { + u64 lr = cpu_if->vgic_lr[i]; + int l1_irq; + + if (!(lr & ICH_LR_HW) || !(lr & ICH_LR_STATE)) + continue; + + /* + * If we had a HW lr programmed by the guest hypervisor, we + * need to emulate the HW effect between the guest hypervisor + * and the nested guest. + */ + l1_irq = (lr & ICH_LR_PHYS_ID_MASK) >> ICH_LR_PHYS_ID_SHIFT; + irq = vgic_get_irq(vcpu->kvm, vcpu, l1_irq); + if (!irq) + continue; /* oh well, the guest hyp is broken */ + + lr = __gic_v3_get_lr(i); + if (!(lr & ICH_LR_STATE)) + irq->active = false; + + vgic_put_irq(vcpu->kvm, irq); + } +} + void vgic_v3_load_nested(struct kvm_vcpu *vcpu) { struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu; diff --git a/arch/arm64/kvm/vgic/vgic.c b/arch/arm64/kvm/vgic/vgic.c index 6503aa208c9f..afe415b5f8a0 100644 --- a/arch/arm64/kvm/vgic/vgic.c +++ b/arch/arm64/kvm/vgic/vgic.c @@ -876,9 +876,11 @@ void kvm_vgic_sync_hwstate(struct kvm_vcpu *vcpu) { int used_lrs; - /* If nesting, this is a load/put affair, not flush/sync. */ - if (vgic_state_is_nested(vcpu)) + /* If nesting, emulate the HW effect from L0 to L1 */ + if (vgic_state_is_nested(vcpu)) { + vgic_v3_sync_nested(vcpu); return; + } /* An empty ap_list_head implies used_lrs == 0 */ if (list_empty(&vcpu->arch.vgic_cpu.ap_list_head)) diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h index a0db49e6c781..a3fc08988a5b 100644 --- a/include/kvm/arm_vgic.h +++ b/include/kvm/arm_vgic.h @@ -391,6 +391,7 @@ void kvm_vgic_load(struct kvm_vcpu *vcpu); void kvm_vgic_put(struct kvm_vcpu *vcpu); void kvm_vgic_vmcr_sync(struct kvm_vcpu *vcpu); +void vgic_v3_sync_nested(struct kvm_vcpu *vcpu); void vgic_v3_load_nested(struct kvm_vcpu *vcpu); void vgic_v3_put_nested(struct kvm_vcpu *vcpu); void vgic_v3_handle_nested_maint_irq(struct kvm_vcpu *vcpu); From patchwork Mon Nov 29 20:01:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645773 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8E5CFC433EF for ; Mon, 29 Nov 2021 21:46:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230155AbhK2Vtw (ORCPT ); Mon, 29 Nov 2021 16:49:52 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49430 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232326AbhK2Vrt (ORCPT ); Mon, 29 Nov 2021 16:47:49 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 96C42C091D24 for ; Mon, 29 Nov 2021 12:06:59 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id E712CCE140F for ; Mon, 29 Nov 2021 20:06:57 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 617D4C53FCD; Mon, 29 Nov 2021 20:06:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216416; bh=dOY51q/eJxOeWtmXrkwWBV+MbhitSkeMTqRT3uYhWnU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Z281vntSJeVIgGOy8fRirAEx00N1YtmMLE5jOD8c1t9Ty/hA4sgtywYGrIRK+N3sy Q+Tn4cM4HqzdK+1tIKZAm9MV3VUBe8c36YHdbGZLEuinEWH95yHm+upgnHNNdznopj taaSFi9zT7f6VqpLu4slaiIwiPyvSm7eFW1vv0cIa337+Ugdmvm6/L1MBfbHwE3U2L WZCIXaoRlqqwYIjfUUynCHXVGNkWQhoQmAhc5zQW5/kuJjzf6kRMbICaUgMhSksoE8 j0QE6RvKUtBFp08Oq2M+Hor0GDACzeOWwBHKdsQnkyb0i3CkO5vWhy60iS60tmmFhX VZ7t2YjVcXrvA== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmrB-008gvR-Q6; Mon, 29 Nov 2021 20:02:33 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 54/69] KVM: arm64: nv: vgic: Allow userland to set VGIC maintenance IRQ Date: Mon, 29 Nov 2021 20:01:35 +0000 Message-Id: <20211129200150.351436-55-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Andre Przywara The VGIC maintenance IRQ signals various conditions about the LRs, when the GIC's virtualization extension is used. So far we didn't need it, but nested virtualization needs to know about this interrupt, so add a userland interface to setup the IRQ number. The architecture mandates that it must be a PPI, on top of that this code only exports a per-device option, so the PPI is the same on all VCPUs. Signed-off-by: Andre Przywara [added some bits of documentation] Signed-off-by: Marc Zyngier --- .../virt/kvm/devices/arm-vgic-v3.rst | 12 +++++++++- arch/arm64/include/uapi/asm/kvm.h | 1 + arch/arm64/kvm/vgic/vgic-kvm-device.c | 22 +++++++++++++++++++ include/kvm/arm_vgic.h | 3 +++ tools/arch/arm/include/uapi/asm/kvm.h | 1 + 5 files changed, 38 insertions(+), 1 deletion(-) diff --git a/Documentation/virt/kvm/devices/arm-vgic-v3.rst b/Documentation/virt/kvm/devices/arm-vgic-v3.rst index 51e5e5762571..1901e651cc00 100644 --- a/Documentation/virt/kvm/devices/arm-vgic-v3.rst +++ b/Documentation/virt/kvm/devices/arm-vgic-v3.rst @@ -284,8 +284,18 @@ Groups: | Aff3 | Aff2 | Aff1 | Aff0 | Errors: - ======= ============================================= -EINVAL vINTID is not multiple of 32 or info field is not VGIC_LEVEL_INFO_LINE_LEVEL ======= ============================================= + + KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ + Attributes: + + The attr field of kvm_device_attr encodes the following values: + + bits: | 31 .... 5 | 4 .... 0 | + values: | RES0 | vINTID | + + The vINTID specifies which interrupt is generated when the vGIC + must generate a maintenance interrupt. This must be a PPI. diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h index 395a4c039bcc..b2811b90fa13 100644 --- a/arch/arm64/include/uapi/asm/kvm.h +++ b/arch/arm64/include/uapi/asm/kvm.h @@ -346,6 +346,7 @@ struct kvm_arm_copy_mte_tags { #define KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS 6 #define KVM_DEV_ARM_VGIC_GRP_LEVEL_INFO 7 #define KVM_DEV_ARM_VGIC_GRP_ITS_REGS 8 +#define KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ 9 #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT 10 #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_MASK \ (0x3fffffULL << KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT) diff --git a/arch/arm64/kvm/vgic/vgic-kvm-device.c b/arch/arm64/kvm/vgic/vgic-kvm-device.c index 0d000d2fe8d2..81b16c52d1bb 100644 --- a/arch/arm64/kvm/vgic/vgic-kvm-device.c +++ b/arch/arm64/kvm/vgic/vgic-kvm-device.c @@ -251,6 +251,12 @@ static int vgic_get_common_attr(struct kvm_device *dev, VGIC_NR_PRIVATE_IRQS, uaddr); break; } + case KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ: { + u32 __user *uaddr = (u32 __user *)(long)attr->addr; + + r = put_user(dev->kvm->arch.vgic.maint_irq, uaddr); + break; + } } return r; @@ -637,6 +643,21 @@ static int vgic_v3_set_attr(struct kvm_device *dev, reg = tmp32; return vgic_v3_attr_regs_access(dev, attr, ®, true); } + case KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ: { + u32 __user *uaddr = (u32 __user *)(long)attr->addr; + u32 val; + + if (get_user(val, uaddr)) + return -EFAULT; + + /* Must be a PPI. */ + if ((val >= VGIC_NR_PRIVATE_IRQS) || (val < VGIC_NR_SGIS)) + return -EINVAL; + + dev->kvm->arch.vgic.maint_irq = val; + + return 0; + } case KVM_DEV_ARM_VGIC_GRP_CTRL: { int ret; @@ -722,6 +743,7 @@ static int vgic_v3_has_attr(struct kvm_device *dev, case KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS: return vgic_v3_has_attr_regs(dev, attr); case KVM_DEV_ARM_VGIC_GRP_NR_IRQS: + case KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ: return 0; case KVM_DEV_ARM_VGIC_GRP_LEVEL_INFO: { if (((attr->attr & KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_MASK) >> diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h index a3fc08988a5b..3ffc89c86144 100644 --- a/include/kvm/arm_vgic.h +++ b/include/kvm/arm_vgic.h @@ -238,6 +238,9 @@ struct vgic_dist { int nr_spis; + /* The GIC maintenance IRQ for nested hypervisors. */ + u32 maint_irq; + /* base addresses in guest physical address space: */ gpa_t vgic_dist_base; /* distributor */ union { diff --git a/tools/arch/arm/include/uapi/asm/kvm.h b/tools/arch/arm/include/uapi/asm/kvm.h index 03cd7c19a683..d5dd96902817 100644 --- a/tools/arch/arm/include/uapi/asm/kvm.h +++ b/tools/arch/arm/include/uapi/asm/kvm.h @@ -246,6 +246,7 @@ struct kvm_vcpu_events { #define KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS 6 #define KVM_DEV_ARM_VGIC_GRP_LEVEL_INFO 7 #define KVM_DEV_ARM_VGIC_GRP_ITS_REGS 8 +#define KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ 9 #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT 10 #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_MASK \ (0x3fffffULL << KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT) From patchwork Mon Nov 29 20:01:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645753 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4F7C6C4332F for ; Mon, 29 Nov 2021 21:44:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232711AbhK2Vr4 (ORCPT ); Mon, 29 Nov 2021 16:47:56 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49444 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232057AbhK2Vpw (ORCPT ); Mon, 29 Nov 2021 16:45:52 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6D744C08EB35 for ; Mon, 29 Nov 2021 12:06:04 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id B954ECE1407 for ; Mon, 29 Nov 2021 20:06:02 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id ECA29C53FAD; Mon, 29 Nov 2021 20:06:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216361; bh=UTV2ggFVZ9pv6QLOOaB0bMKGFmSJDLhx2Wp4d28z1Mc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=mwoPIrVDsqT1fAfQ9uRq5Zj6d+idcRpgMwWHcsx9AYoNWU8MgSv78OpMG4x8W1jWK o32kmYo710IKGgbWJMVPjzojycyQ6kUonoligWtHsVIrfDJgI6vDwGT4q4eWdVSpUK arzHhp6jblMtYs/BiKvL0q7UMv1f0OFd9U6BSP0vLCp8lOc1B3uWrgZW5X/oZePdhk va2iuxKkukp7szAXrdDwzvkB8MAuGBHCrRH/MjpVM6hdIK7fIl/qLtZ0ivoUh0kJAx 0cGvK42NsuYtXvVx8+bJ0z5m3GXbmJXibb5oeD6SNKRmJvijpn++3QyUnxbpFlnHGS QPfBGUbX3k5kQ== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmrC-008gvR-3E; Mon, 29 Nov 2021 20:02:35 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 55/69] KVM: arm64: nv: Implement maintenance interrupt forwarding Date: Mon, 29 Nov 2021 20:01:36 +0000 Message-Id: <20211129200150.351436-56-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org When we take a maintenance interrupt, we need to decide whether it is generated on an action from the guest, or if it is something that needs to be forwarded to the guest hypervisor. Signed-off-by: Marc Zyngier --- arch/arm64/kvm/vgic/vgic-init.c | 30 ++++++++++++++++++++++++++++ arch/arm64/kvm/vgic/vgic-v3-nested.c | 25 +++++++++++++++++++---- 2 files changed, 51 insertions(+), 4 deletions(-) diff --git a/arch/arm64/kvm/vgic/vgic-init.c b/arch/arm64/kvm/vgic/vgic-init.c index 0a06d0648970..1d49912af42d 100644 --- a/arch/arm64/kvm/vgic/vgic-init.c +++ b/arch/arm64/kvm/vgic/vgic-init.c @@ -6,10 +6,12 @@ #include #include #include +#include #include #include #include #include +#include #include "vgic.h" /* @@ -221,6 +223,16 @@ int kvm_vgic_vcpu_init(struct kvm_vcpu *vcpu) if (!irqchip_in_kernel(vcpu->kvm)) return 0; + if (nested_virt_in_use(vcpu)) { + /* FIXME: remove this hack */ + if (vcpu->kvm->arch.vgic.maint_irq == 0) + vcpu->kvm->arch.vgic.maint_irq = kvm_vgic_global_state.maint_irq; + ret = kvm_vgic_set_owner(vcpu, vcpu->kvm->arch.vgic.maint_irq, + vcpu); + if (ret) + return ret; + } + /* * If we are creating a VCPU with a GICv3 we must also register the * KVM io device for the redistributor that belongs to this VCPU. @@ -473,12 +485,23 @@ static int vgic_init_cpu_dying(unsigned int cpu) static irqreturn_t vgic_maintenance_handler(int irq, void *data) { + struct kvm_vcpu *vcpu = *(struct kvm_vcpu **)data; + /* * We cannot rely on the vgic maintenance interrupt to be * delivered synchronously. This means we can only use it to * exit the VM, and we perform the handling of EOIed * interrupts on the exit path (see vgic_fold_lr_state). */ + + /* If not nested, deactivate */ + if (!vcpu || !vgic_state_is_nested(vcpu)) { + irq_set_irqchip_state(irq, IRQCHIP_STATE_ACTIVE, false); + return IRQ_HANDLED; + } + + /* Assume nested from now */ + vgic_v3_handle_nested_maint_irq(vcpu); return IRQ_HANDLED; } @@ -577,6 +600,13 @@ int kvm_vgic_hyp_init(void) return ret; } + ret = irq_set_vcpu_affinity(kvm_vgic_global_state.maint_irq, + kvm_get_running_vcpus()); + if (ret) { + kvm_err("Error setting vcpu affinity\n"); + goto out_free_irq; + } + ret = cpuhp_setup_state(CPUHP_AP_KVM_ARM_VGIC_INIT_STARTING, "kvm/arm/vgic:starting", vgic_init_cpu_starting, vgic_init_cpu_dying); diff --git a/arch/arm64/kvm/vgic/vgic-v3-nested.c b/arch/arm64/kvm/vgic/vgic-v3-nested.c index 3ee3438b5e22..4ba426e2324d 100644 --- a/arch/arm64/kvm/vgic/vgic-v3-nested.c +++ b/arch/arm64/kvm/vgic/vgic-v3-nested.c @@ -173,10 +173,20 @@ void vgic_v3_sync_nested(struct kvm_vcpu *vcpu) void vgic_v3_load_nested(struct kvm_vcpu *vcpu) { struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu; + struct vgic_irq *irq; + unsigned long flags; vgic_cpu->shadow_vgic_v3 = vgic_cpu->nested_vgic_v3; vgic_v3_create_shadow_lr(vcpu); __vgic_v3_restore_state(vcpu_shadow_if(vcpu)); + + irq = vgic_get_irq(vcpu->kvm, vcpu, vcpu->kvm->arch.vgic.maint_irq); + raw_spin_lock_irqsave(&irq->irq_lock, flags); + if (irq->line_level || irq->active) + irq_set_irqchip_state(kvm_vgic_global_state.maint_irq, + IRQCHIP_STATE_ACTIVE, true); + raw_spin_unlock_irqrestore(&irq->irq_lock, flags); + vgic_put_irq(vcpu->kvm, irq); } void vgic_v3_put_nested(struct kvm_vcpu *vcpu) @@ -191,11 +201,14 @@ void vgic_v3_put_nested(struct kvm_vcpu *vcpu) */ vgic_v3_fixup_shadow_lr_state(vcpu); vgic_cpu->nested_vgic_v3 = vgic_cpu->shadow_vgic_v3; + irq_set_irqchip_state(kvm_vgic_global_state.maint_irq, + IRQCHIP_STATE_ACTIVE, false); } void vgic_v3_handle_nested_maint_irq(struct kvm_vcpu *vcpu) { struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); + bool state; /* * If we exit a nested VM with a pending maintenance interrupt from the @@ -203,8 +216,12 @@ void vgic_v3_handle_nested_maint_irq(struct kvm_vcpu *vcpu) * can re-sync the appropriate LRs and sample level triggered interrupts * again. */ - if (vgic_state_is_nested(vcpu) && - (cpu_if->vgic_hcr & ICH_HCR_EN) && - vgic_v3_get_misr(vcpu)) - kvm_inject_nested_irq(vcpu); + if (!vgic_state_is_nested(vcpu)) + return; + + state = cpu_if->vgic_hcr & ICH_HCR_EN; + state &= vgic_v3_get_misr(vcpu); + + kvm_vgic_inject_irq(vcpu->kvm, vcpu->vcpu_id, + vcpu->kvm->arch.vgic.maint_irq, state, vcpu); } From patchwork Mon Nov 29 20:01:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645789 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99A21C433EF for ; Mon, 29 Nov 2021 21:46:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232141AbhK2Vt6 (ORCPT ); Mon, 29 Nov 2021 16:49:58 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49910 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232111AbhK2Vr5 (ORCPT ); Mon, 29 Nov 2021 16:47:57 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 71D51C091D2F for ; Mon, 29 Nov 2021 12:07:14 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id C5A78CE140F for ; Mon, 29 Nov 2021 20:07:12 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 250B4C53FC7; Mon, 29 Nov 2021 20:07:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216431; bh=YRAEqvmApOPt3HyGrB4klDsv8OSyIsYUJiXxEthpNUM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=L3ct5Z8TQjnonyp4pxIOjkzfYbb7yl8+AekAsdpVb6G2nmBRjawQiFlyrgXFUGhTw RYXmCjMpqHyEU4pRahXRIXTLoar/nSEdtg809HDhfADxbXl7odo6LRFUwyUPIo3IeC iZNtUAk+6deWyz/r0gof3d+pEY9HTDboN6RqYJLZgnwmRtikF1Az5fDxW1Ezjt8o+q FuamczylYMg2g2330GuOkjLopqdhrRIevSJRjRVvJahGR4LAN/ZniE32qGxEohZ8e+ ixLOaAZ4+SFEuxrf+vUbQGp8FdnfSMqExU0BW7y619QtdGZkTGKbY9NuLf3jGEVE8h 4PQOhAoxJuzJw== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmrE-008gvR-AI; Mon, 29 Nov 2021 20:02:37 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 56/69] KVM: arm64: nv: Add nested GICv3 tracepoints Date: Mon, 29 Nov 2021 20:01:37 +0000 Message-Id: <20211129200150.351436-57-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Christoffer Dall Adding tracepoints to be able to peek into the shadow LRs used when running a guest guest. Signed-off-by: Christoffer Dall Signed-off-by: Marc Zyngier --- arch/arm64/kvm/vgic/vgic-nested-trace.h | 137 ++++++++++++++++++++++++ arch/arm64/kvm/vgic/vgic-v3-nested.c | 13 ++- 2 files changed, 149 insertions(+), 1 deletion(-) create mode 100644 arch/arm64/kvm/vgic/vgic-nested-trace.h diff --git a/arch/arm64/kvm/vgic/vgic-nested-trace.h b/arch/arm64/kvm/vgic/vgic-nested-trace.h new file mode 100644 index 000000000000..f1a074c791a6 --- /dev/null +++ b/arch/arm64/kvm/vgic/vgic-nested-trace.h @@ -0,0 +1,137 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#if !defined(_TRACE_VGIC_NESTED_H) || defined(TRACE_HEADER_MULTI_READ) +#define _TRACE_VGIC_NESTED_H + +#include + +#undef TRACE_SYSTEM +#define TRACE_SYSTEM kvm + +#define SLR_ENTRY_VALS(x) \ + " ", \ + !!(__entry->lrs[x] & ICH_LR_HW), \ + !!(__entry->lrs[x] & ICH_LR_PENDING_BIT), \ + !!(__entry->lrs[x] & ICH_LR_ACTIVE_BIT), \ + __entry->lrs[x] & ICH_LR_VIRTUAL_ID_MASK, \ + (__entry->lrs[x] & ICH_LR_PHYS_ID_MASK) >> ICH_LR_PHYS_ID_SHIFT, \ + (__entry->orig_lrs[x] & ICH_LR_PHYS_ID_MASK) >> ICH_LR_PHYS_ID_SHIFT + +TRACE_EVENT(vgic_create_shadow_lrs, + TP_PROTO(struct kvm_vcpu *vcpu, int nr_lr, u64 *lrs, u64 *orig_lrs), + TP_ARGS(vcpu, nr_lr, lrs, orig_lrs), + + TP_STRUCT__entry( + __field( int, nr_lr ) + __array( u64, lrs, 16 ) + __array( u64, orig_lrs, 16 ) + ), + + TP_fast_assign( + __entry->nr_lr = nr_lr; + memcpy(__entry->lrs, lrs, 16 * sizeof(u64)); + memcpy(__entry->orig_lrs, orig_lrs, 16 * sizeof(u64)); + ), + + TP_printk("nr_lr: %d\n" + "%50sLR[ 0]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" + "%50sLR[ 1]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" + "%50sLR[ 2]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" + "%50sLR[ 3]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" + "%50sLR[ 4]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" + "%50sLR[ 5]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" + "%50sLR[ 6]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" + "%50sLR[ 7]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" + "%50sLR[ 8]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" + "%50sLR[ 9]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" + "%50sLR[10]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" + "%50sLR[11]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" + "%50sLR[12]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" + "%50sLR[13]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" + "%50sLR[14]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" + "%50sLR[15]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)", + __entry->nr_lr, + SLR_ENTRY_VALS(0), SLR_ENTRY_VALS(1), SLR_ENTRY_VALS(2), + SLR_ENTRY_VALS(3), SLR_ENTRY_VALS(4), SLR_ENTRY_VALS(5), + SLR_ENTRY_VALS(6), SLR_ENTRY_VALS(7), SLR_ENTRY_VALS(8), + SLR_ENTRY_VALS(9), SLR_ENTRY_VALS(10), SLR_ENTRY_VALS(11), + SLR_ENTRY_VALS(12), SLR_ENTRY_VALS(13), SLR_ENTRY_VALS(14), + SLR_ENTRY_VALS(15)) +); + +#define LR_ENTRY_VALS(x) \ + " ", \ + !!(__entry->lrs[x] & ICH_LR_HW), \ + !!(__entry->lrs[x] & ICH_LR_PENDING_BIT), \ + !!(__entry->lrs[x] & ICH_LR_ACTIVE_BIT), \ + __entry->lrs[x] & ICH_LR_VIRTUAL_ID_MASK, \ + (__entry->lrs[x] & ICH_LR_PHYS_ID_MASK) >> ICH_LR_PHYS_ID_SHIFT + +TRACE_EVENT(vgic_put_nested, + TP_PROTO(struct kvm_vcpu *vcpu, int nr_lr, u64 *lrs), + TP_ARGS(vcpu, nr_lr, lrs), + + TP_STRUCT__entry( + __field( int, nr_lr ) + __array( u64, lrs, 16 ) + ), + + TP_fast_assign( + __entry->nr_lr = nr_lr; + memcpy(__entry->lrs, lrs, 16 * sizeof(u64)); + ), + + TP_printk("nr_lr: %d\n" + "%50sLR[ 0]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" + "%50sLR[ 1]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" + "%50sLR[ 2]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" + "%50sLR[ 3]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" + "%50sLR[ 4]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" + "%50sLR[ 5]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" + "%50sLR[ 6]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" + "%50sLR[ 7]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" + "%50sLR[ 8]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" + "%50sLR[ 9]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" + "%50sLR[10]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" + "%50sLR[11]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" + "%50sLR[12]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" + "%50sLR[13]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" + "%50sLR[14]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" + "%50sLR[15]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu", + __entry->nr_lr, + LR_ENTRY_VALS(0), LR_ENTRY_VALS(1), LR_ENTRY_VALS(2), + LR_ENTRY_VALS(3), LR_ENTRY_VALS(4), LR_ENTRY_VALS(5), + LR_ENTRY_VALS(6), LR_ENTRY_VALS(7), LR_ENTRY_VALS(8), + LR_ENTRY_VALS(9), LR_ENTRY_VALS(10), LR_ENTRY_VALS(11), + LR_ENTRY_VALS(12), LR_ENTRY_VALS(13), LR_ENTRY_VALS(14), + LR_ENTRY_VALS(15)) +); + +TRACE_EVENT(vgic_nested_hw_emulate, + TP_PROTO(int lr, u64 lr_val, u32 l1_intid), + TP_ARGS(lr, lr_val, l1_intid), + + TP_STRUCT__entry( + __field( int, lr ) + __field( u64, lr_val ) + __field( u32, l1_intid ) + ), + + TP_fast_assign( + __entry->lr = lr; + __entry->lr_val = lr_val; + __entry->l1_intid = l1_intid; + ), + + TP_printk("lr: %d LR %llx L1 INTID: %u\n", + __entry->lr, __entry->lr_val, __entry->l1_intid) +); + +#endif /* _TRACE_VGIC_NESTED_H */ + +#undef TRACE_INCLUDE_PATH +#define TRACE_INCLUDE_PATH vgic/ +#undef TRACE_INCLUDE_FILE +#define TRACE_INCLUDE_FILE vgic-nested-trace + +/* This part must be outside protection */ +#include diff --git a/arch/arm64/kvm/vgic/vgic-v3-nested.c b/arch/arm64/kvm/vgic/vgic-v3-nested.c index 4ba426e2324d..94b1edb67011 100644 --- a/arch/arm64/kvm/vgic/vgic-v3-nested.c +++ b/arch/arm64/kvm/vgic/vgic-v3-nested.c @@ -13,6 +13,9 @@ #include "vgic.h" +#define CREATE_TRACE_POINTS +#include "vgic-nested-trace.h" + static inline struct vgic_v3_cpu_if *vcpu_nested_if(struct kvm_vcpu *vcpu) { return &vcpu->arch.vgic_cpu.nested_vgic_v3; @@ -119,6 +122,9 @@ static void vgic_v3_create_shadow_lr(struct kvm_vcpu *vcpu) used_lrs = i + 1; } + trace_vgic_create_shadow_lrs(vcpu, kvm_vgic_global_state.nr_lr, + s_cpu_if->vgic_lr, cpu_if->vgic_lr); + s_cpu_if->used_lrs = used_lrs; } @@ -163,8 +169,10 @@ void vgic_v3_sync_nested(struct kvm_vcpu *vcpu) continue; /* oh well, the guest hyp is broken */ lr = __gic_v3_get_lr(i); - if (!(lr & ICH_LR_STATE)) + if (!(lr & ICH_LR_STATE)) { + trace_vgic_nested_hw_emulate(i, lr, l1_irq); irq->active = false; + } vgic_put_irq(vcpu->kvm, irq); } @@ -195,6 +203,9 @@ void vgic_v3_put_nested(struct kvm_vcpu *vcpu) __vgic_v3_save_state(vcpu_shadow_if(vcpu)); + trace_vgic_put_nested(vcpu, kvm_vgic_global_state.nr_lr, + vcpu_shadow_if(vcpu)->vgic_lr); + /* * Translate the shadow state HW fields back to the virtual ones * before copying the shadow struct back to the nested one. From patchwork Mon Nov 29 20:01:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645743 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 00FF8C433F5 for ; Mon, 29 Nov 2021 21:44:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232310AbhK2Vrs (ORCPT ); Mon, 29 Nov 2021 16:47:48 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48534 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232341AbhK2Vpq (ORCPT ); Mon, 29 Nov 2021 16:45:46 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 89B38C08EB32 for ; Mon, 29 Nov 2021 12:05:51 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 0E8D2CE13DE for ; Mon, 29 Nov 2021 20:05:51 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 401A3C53FCF; Mon, 29 Nov 2021 20:05:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216349; bh=7k18Z+x3oCj8uRXPCQOA9rwRPaRgEpzPws/JgwdeSx0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Xe/TL8h/UsLgyA3rkGLGcvBkSsrDe6dXeDSLzrEYhwhLUmMW4Uc918FMRMj/Ogb71 idghjVuj6pjBpZW5kfQudV3ZhF+OYL7EHaTz0eR1oxMQ53wrCkM8UsVqbwHV/c0KLb QcgT+7WxhNdspyUq37JRUQj7epU/WWZEarlbD0KNZbC7EGshIQlxxXsiSC8ncEa4fZ 6/qDi6uZ+LCr/FM8tnS0j/M6skvqXEryCUWcV9vpp1kVbMqv9mna0Xm4rZKhjUBmNm cLQLymX2QuWIYdZAGjf6NgvsMrK7J+CGxO/EabYFzvXwlN9okNAZMB2hiATUzknWAP 3rf+SKGQwpoKw== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmrF-008gvR-G4; Mon, 29 Nov 2021 20:02:37 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 57/69] KVM: arm64: nv: Allow userspace to request KVM_ARM_VCPU_NESTED_VIRT Date: Mon, 29 Nov 2021 20:01:38 +0000 Message-Id: <20211129200150.351436-58-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Since we're (almost) feature complete, let's allow userspace to request KVM_ARM_VCPU_NESTED_VIRT by bumping the KVM_VCPU_MAX_FEATURES up. We also now advertise the feature to userspace with a new capability. It's going to be great... Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_host.h | 2 +- arch/arm64/kvm/arm.c | 3 +++ include/uapi/linux/kvm.h | 1 + 3 files changed, 5 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 323215873388..911dd25dba8e 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -38,7 +38,7 @@ #define KVM_MAX_VCPUS VGIC_V3_MAX_CPUS -#define KVM_VCPU_MAX_FEATURES 7 +#define KVM_VCPU_MAX_FEATURES 8 #define KVM_REQ_SLEEP \ KVM_ARCH_REQ_FLAGS(0, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP) diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 0bb46bce5ffb..dc1fda709cf5 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -257,6 +257,9 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) case KVM_CAP_ARM_EL1_32BIT: r = cpus_have_const_cap(ARM64_HAS_32BIT_EL1); break; + case KVM_CAP_ARM_EL2: + r = cpus_have_const_cap(ARM64_HAS_NESTED_VIRT); + break; case KVM_CAP_GUEST_DEBUG_HW_BPS: r = get_num_brps(); break; diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h index 1daa45268de2..65d1862ab093 100644 --- a/include/uapi/linux/kvm.h +++ b/include/uapi/linux/kvm.h @@ -1131,6 +1131,7 @@ struct kvm_ppc_resize_hpt { #define KVM_CAP_EXIT_ON_EMULATION_FAILURE 204 #define KVM_CAP_ARM_MTE 205 #define KVM_CAP_VM_MOVE_ENC_CONTEXT_FROM 206 +#define KVM_CAP_ARM_EL2 207 #ifdef KVM_CAP_IRQ_ROUTING From patchwork Mon Nov 29 20:01:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645667 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C1CFEC433FE for ; Mon, 29 Nov 2021 20:09:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231776AbhK2UMo (ORCPT ); Mon, 29 Nov 2021 15:12:44 -0500 Received: from ams.source.kernel.org ([145.40.68.75]:43010 "EHLO ams.source.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231811AbhK2UKn (ORCPT ); Mon, 29 Nov 2021 15:10:43 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id C4A49B815FE for ; Mon, 29 Nov 2021 20:07:24 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7E7E0C53FCF; Mon, 29 Nov 2021 20:07:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216443; bh=cSLyIVV3jyyOi65ucV97mpaNj+BNKs5gCIo0BmVWpyg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hb+2Jv7Bkg8+uKEWqwdveMtpGIrCoF43d9keDNo7xsBlSB8tVyr6qBv14xTcetbBs 1M8UmCkA1fldyfkckM/ERmM29z711PPhQ4Qdaxm6XRIh2fBBpOjneEEZzjyfMj1lXr K3Tf2OMAluER5ifnhfT4k0WjnqSU15wT/3CMMi3qlzZJLkGPtWa4tmRakLut4V+Qed sXfkEOGaHEdbu5kJIIHfAkXragIV4zPjP/xcxGc+ksD2EFAHcSkq2AXuOtTXxF4C8o rVUiUl8Bo3/N/8Tz1ERI6qGj4hXpSWnOuoUaRfKeWpR13lVdAUjc5f5WxPZEdlHW0l qvoxXLfjkMH1g== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmrF-008gvR-Qk; Mon, 29 Nov 2021 20:02:37 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 58/69] KVM: arm64: nv: Add handling of ARMv8.4-TTL TLB invalidation Date: Mon, 29 Nov 2021 20:01:39 +0000 Message-Id: <20211129200150.351436-59-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Support guest-provided information information to find out about the range of required invalidation. Signed-off-by: Marc Zyngier Reviewed-by: Ganapatrao Kulkarni --- arch/arm64/include/asm/kvm_nested.h | 1 + arch/arm64/kvm/nested.c | 57 +++++++++++++++++++++ arch/arm64/kvm/sys_regs.c | 78 ++++++++++++++++++----------- 3 files changed, 108 insertions(+), 28 deletions(-) diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index 5fa3c634c8e1..7c47ad655e2e 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -123,6 +123,7 @@ extern bool __forward_traps(struct kvm_vcpu *vcpu, unsigned int reg, u64 control_bit); extern bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit); extern bool forward_nv_traps(struct kvm_vcpu *vcpu); +unsigned int ttl_to_size(u8 ttl); struct sys_reg_params; struct sys_reg_desc; diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c index 198169648c3c..6f738b5f57dd 100644 --- a/arch/arm64/kvm/nested.c +++ b/arch/arm64/kvm/nested.c @@ -363,6 +363,63 @@ int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa, return ret; } + +unsigned int ttl_to_size(u8 ttl) +{ + int level = ttl & 3; + int gran = (ttl >> 2) & 3; + unsigned int max_size = 0; + + switch (gran) { + case TLBI_TTL_TG_4K: + switch (level) { + case 0: + break; + case 1: + max_size = SZ_1G; + break; + case 2: + max_size = SZ_2M; + break; + case 3: + max_size = SZ_4K; + break; + } + break; + case TLBI_TTL_TG_16K: + switch (level) { + case 0: + case 1: + break; + case 2: + max_size = SZ_32M; + break; + case 3: + max_size = SZ_16K; + break; + } + break; + case TLBI_TTL_TG_64K: + switch (level) { + case 0: + case 1: + /* No 52bit IPA support */ + break; + case 2: + max_size = SZ_512M; + break; + case 3: + max_size = SZ_64K; + break; + } + break; + default: /* No size information */ + break; + } + + return max_size; +} + /* Must be called with kvm->lock held */ struct kvm_s2_mmu *lookup_s2_mmu(struct kvm *kvm, u64 vttbr, u64 hcr) { diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 55f3e94c24f1..e0f088de2cad 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -2699,59 +2699,81 @@ static bool handle_vmalls12e1is(struct kvm_vcpu *vcpu, struct sys_reg_params *p, return true; } +static unsigned long compute_tlb_inval_range(struct kvm_vcpu *vcpu, + struct kvm_s2_mmu *mmu, + u64 val) +{ + unsigned long max_size; + u8 ttl = 0; + + if (cpus_have_const_cap(ARM64_HAS_ARMv8_4_TTL)) { + ttl = FIELD_GET(GENMASK_ULL(47, 44), val); + } + + max_size = ttl_to_size(ttl); + + if (!max_size) { + u64 vtcr = vcpu_read_sys_reg(vcpu, VTCR_EL2); + + /* Compute the maximum extent of the invalidation */ + switch ((vtcr & VTCR_EL2_TG0_MASK)) { + case VTCR_EL2_TG0_4K: + max_size = SZ_1G; + break; + case VTCR_EL2_TG0_16K: + max_size = SZ_32M; + break; + case VTCR_EL2_TG0_64K: + /* + * No, we do not support 52bit IPA in nested yet. Once + * we do, this should be 4TB. + */ + /* FIXME: remove the 52bit PA support from the IDregs */ + max_size = SZ_512M; + break; + default: + BUG(); + } + } + + WARN_ON(!max_size); + return max_size; +} + static bool handle_ipas2e1is(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) { u64 vttbr = vcpu_read_sys_reg(vcpu, VTTBR_EL2); - u64 vtcr = vcpu_read_sys_reg(vcpu, VTCR_EL2); struct kvm_s2_mmu *mmu; u64 base_addr; - int max_size; + unsigned long max_size; /* * We drop a number of things from the supplied value: * * - NS bit: we're non-secure only. * - * - TTL field: We already have the granule size from the - * VTCR_EL2.TG0 field, and the level is only relevant to the - * guest's S2PT. - * * - IPA[51:48]: We don't support 52bit IPA just yet... * * And of course, adjust the IPA to be on an actual address. */ base_addr = (p->regval & GENMASK_ULL(35, 0)) << 12; - /* Compute the maximum extent of the invalidation */ - switch ((vtcr & VTCR_EL2_TG0_MASK)) { - case VTCR_EL2_TG0_4K: - max_size = SZ_1G; - break; - case VTCR_EL2_TG0_16K: - max_size = SZ_32M; - break; - case VTCR_EL2_TG0_64K: - /* - * No, we do not support 52bit IPA in nested yet. Once - * we do, this should be 4TB. - */ - /* FIXME: remove the 52bit PA support from the IDregs */ - max_size = SZ_512M; - break; - default: - BUG(); - } - spin_lock(&vcpu->kvm->mmu_lock); mmu = lookup_s2_mmu(vcpu->kvm, vttbr, HCR_VM); - if (mmu) + if (mmu) { + max_size = compute_tlb_inval_range(vcpu, mmu, p->regval); + base_addr &= ~(max_size - 1); kvm_unmap_stage2_range(mmu, base_addr, max_size); + } mmu = lookup_s2_mmu(vcpu->kvm, vttbr, 0); - if (mmu) + if (mmu) { + max_size = compute_tlb_inval_range(vcpu, mmu, p->regval); + base_addr &= ~(max_size - 1); kvm_unmap_stage2_range(mmu, base_addr, max_size); + } spin_unlock(&vcpu->kvm->mmu_lock); From patchwork Mon Nov 29 20:01:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645661 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E93C5C433EF for ; Mon, 29 Nov 2021 20:08:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233530AbhK2UMG (ORCPT ); Mon, 29 Nov 2021 15:12:06 -0500 Received: from sin.source.kernel.org ([145.40.73.55]:50820 "EHLO sin.source.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350289AbhK2UKD (ORCPT ); Mon, 29 Nov 2021 15:10:03 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 859C7CE140D for ; Mon, 29 Nov 2021 20:06:44 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id EFE1CC53FC7; Mon, 29 Nov 2021 20:06:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216403; bh=2DVtb19jSGYirnF4XWDZJn/U5l1nFubDr9q7zTYWHSY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=cerOmPmEILbqEqnBA7DlZ9rwFZQjeem2zx4n6Sv5fB8wBoDm/f6DA+ZswyD0vaqgd tOOyhVwYxkTFWOsuq+PqDqAP2+F4wnfdo0umT4ug4rO7pGMT+l2BsdaSc1z/WsXafj VtGxM0TdbRpNYmPq8n+OunrO42/x7B9CAyDrjorHzPairGLw7G+KrHhI5vuGTm0CFt hlTq1XkMityhGFEXFUi+AjSUvgmMpKKbSIqVPci6cZSUEVDR1SE29A+S8pHpby0Xxh /X/JQey+rMMwkL5IImh3yonWK0FRboKUxmn1Wa2KWqtTsjFsFeWt9aav0xRfEQKFNt rWEKT6mSIz+yg== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmrG-008gvR-4a; Mon, 29 Nov 2021 20:02:38 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 59/69] KVM: arm64: nv: Invalidate TLBs based on shadow S2 TTL-like information Date: Mon, 29 Nov 2021 20:01:40 +0000 Message-Id: <20211129200150.351436-60-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org In order to be able to make S2 TLB invalidations more performant on NV, let's use a scheme derived from the ARMv8.4 TTL extension. If bits [56:55] in the descriptor are non-zero, they indicate a level which can be used as an invalidation range. Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_nested.h | 4 ++ arch/arm64/kvm/nested.c | 98 +++++++++++++++++++++++++++++ arch/arm64/kvm/sys_regs.c | 13 ++-- 3 files changed, 110 insertions(+), 5 deletions(-) diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index 7c47ad655e2e..34499c496ae6 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -123,6 +123,8 @@ extern bool __forward_traps(struct kvm_vcpu *vcpu, unsigned int reg, u64 control_bit); extern bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit); extern bool forward_nv_traps(struct kvm_vcpu *vcpu); +u8 get_guest_mapping_ttl(struct kvm_vcpu *vcpu, struct kvm_s2_mmu *mmu, + u64 addr); unsigned int ttl_to_size(u8 ttl); struct sys_reg_params; @@ -131,4 +133,6 @@ struct sys_reg_desc; void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p, const struct sys_reg_desc *r); +#define KVM_NV_GUEST_MAP_SZ GENMASK_ULL(56, 55) + #endif /* __ARM64_KVM_NESTED_H */ diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c index 6f738b5f57dd..e11bdee15df2 100644 --- a/arch/arm64/kvm/nested.c +++ b/arch/arm64/kvm/nested.c @@ -16,6 +16,7 @@ * along with this program. If not, see . */ +#include #include #include @@ -363,6 +364,29 @@ int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa, return ret; } +static int read_host_s2_desc(phys_addr_t pa, u64 *desc, void *data) +{ + u64 *va = phys_to_virt(pa); + + *desc = *va; + + return 0; +} + +static int kvm_walk_shadow_s2(struct kvm_s2_mmu *mmu, phys_addr_t gipa, + struct kvm_s2_trans *result) +{ + struct s2_walk_info wi = { }; + + wi.read_desc = read_host_s2_desc; + wi.baddr = mmu->pgd_phys; + + vtcr_to_walk_info(mmu->arch->vtcr, &wi); + + wi.be = IS_ENABLED(CONFIG_CPU_BIG_ENDIAN); + + return walk_nested_s2_pgd(gipa, &wi, result); +} unsigned int ttl_to_size(u8 ttl) { @@ -420,6 +444,80 @@ unsigned int ttl_to_size(u8 ttl) return max_size; } +/* + * Compute the equivalent of the TTL field by parsing the shadow PT. + * The granule size is extracted from VTCR_EL2.TG0 while the level is + * retrieved from first entry carrying the level as a tag. + */ +u8 get_guest_mapping_ttl(struct kvm_vcpu *vcpu, struct kvm_s2_mmu *mmu, + u64 addr) +{ + u64 tmp, sz = 0, vtcr = vcpu_read_sys_reg(vcpu, VTCR_EL2); + struct kvm_s2_trans out; + u8 ttl, level; + + switch (vtcr & VTCR_EL2_TG0_MASK) { + case VTCR_EL2_TG0_4K: + ttl = (1 << 2); + break; + case VTCR_EL2_TG0_16K: + ttl = (2 << 2); + break; + case VTCR_EL2_TG0_64K: + ttl = (3 << 2); + break; + default: + BUG(); + } + + tmp = addr; + +again: + /* Iteratively compute the block sizes for a particular granule size */ + switch (vtcr & VTCR_EL2_TG0_MASK) { + case VTCR_EL2_TG0_4K: + if (sz < SZ_4K) sz = SZ_4K; + else if (sz < SZ_2M) sz = SZ_2M; + else if (sz < SZ_1G) sz = SZ_1G; + else sz = 0; + break; + case VTCR_EL2_TG0_16K: + if (sz < SZ_16K) sz = SZ_16K; + else if (sz < SZ_32M) sz = SZ_32M; + else sz = 0; + break; + case VTCR_EL2_TG0_64K: + if (sz < SZ_64K) sz = SZ_64K; + else if (sz < SZ_512M) sz = SZ_512M; + else sz = 0; + break; + default: + BUG(); + } + + if (sz == 0) + return 0; + + tmp &= ~(sz - 1); + out = (struct kvm_s2_trans) { }; + kvm_walk_shadow_s2(mmu, tmp, &out); + level = FIELD_GET(KVM_NV_GUEST_MAP_SZ, out.upper_attr); + if (!level) + goto again; + + ttl |= level; + + /* + * We now have found some level information in the shadow S2. Check + * that the resulting range is actually including the original IPA. + */ + sz = ttl_to_size(ttl); + if (addr < (tmp + sz)) + return ttl; + + return 0; +} + /* Must be called with kvm->lock held */ struct kvm_s2_mmu *lookup_s2_mmu(struct kvm *kvm, u64 vttbr, u64 hcr) { diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index e0f088de2cad..7400a76f6261 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -2704,10 +2704,13 @@ static unsigned long compute_tlb_inval_range(struct kvm_vcpu *vcpu, u64 val) { unsigned long max_size; - u8 ttl = 0; + u8 ttl; - if (cpus_have_const_cap(ARM64_HAS_ARMv8_4_TTL)) { - ttl = FIELD_GET(GENMASK_ULL(47, 44), val); + ttl = FIELD_GET(GENMASK_ULL(47, 44), val); + + if (!(cpus_have_const_cap(ARM64_HAS_ARMv8_4_TTL) && ttl)) { + u64 addr = (val & GENMASK_ULL(35, 0)) << 12; + ttl = get_guest_mapping_ttl(vcpu, mmu, addr); } max_size = ttl_to_size(ttl); @@ -2748,6 +2751,8 @@ static bool handle_ipas2e1is(struct kvm_vcpu *vcpu, struct sys_reg_params *p, u64 base_addr; unsigned long max_size; + spin_lock(&vcpu->kvm->mmu_lock); + /* * We drop a number of things from the supplied value: * @@ -2759,8 +2764,6 @@ static bool handle_ipas2e1is(struct kvm_vcpu *vcpu, struct sys_reg_params *p, */ base_addr = (p->regval & GENMASK_ULL(35, 0)) << 12; - spin_lock(&vcpu->kvm->mmu_lock); - mmu = lookup_s2_mmu(vcpu->kvm, vttbr, HCR_VM); if (mmu) { max_size = compute_tlb_inval_range(vcpu, mmu, p->regval); From patchwork Mon Nov 29 20:01:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645619 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90788C433F5 for ; Mon, 29 Nov 2021 20:08:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233077AbhK2ULp (ORCPT ); Mon, 29 Nov 2021 15:11:45 -0500 Received: from sin.source.kernel.org ([145.40.73.55]:50518 "EHLO sin.source.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344222AbhK2UJc (ORCPT ); Mon, 29 Nov 2021 15:09:32 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id F2CF8CE1407 for ; Mon, 29 Nov 2021 20:06:12 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 33100C53FC7; Mon, 29 Nov 2021 20:06:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216371; bh=6hZQyOZ9NUfBRMqXUMWM6jZMoyFklw6GsxNcnhbH+Z8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Mm0IUw36bb3TIF8aBdIUZer4551CmkWvpdRbCjaDiTmUjkDfFzpLJip21Rd5jVtI2 PyKb3MBnubetP86b1tOKYMfCgB8Q8Y3VMsQpI48oRsqT6VUBgI7ui4ZZLseqoIqtbt V1RK98kVw7tDjau+YzvsA2bMpgcnRJEY5XUSd236n1D5rXb95VvBVBI4Xw1+W9Izfv 3dITW+u+OSLv32sxWDuH5J11tn/3pdY43Eyx29uFZyLWZtjFDcdA6Cyc+e69EyeFTY w6XHAfG+hg14b4yhL7XKi/+RcEgtQkxuSuXAFIrMiX6CKFNSIf7N6hixiACbELQYOU i+8YFtKFkWz1g== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmrG-008gvR-Ed; Mon, 29 Nov 2021 20:02:38 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 60/69] KVM: arm64: nv: Tag shadow S2 entries with nested level Date: Mon, 29 Nov 2021 20:01:41 +0000 Message-Id: <20211129200150.351436-61-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Populate bits [56:55] of the leaf entry with the level provided by the guest's S2 translation. Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_nested.h | 8 ++++++++ arch/arm64/kvm/mmu.c | 6 ++++++ 2 files changed, 14 insertions(+) diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index 34499c496ae6..76fbf5de1f2c 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -5,6 +5,8 @@ #include #include +#include + static inline bool nested_virt_in_use(const struct kvm_vcpu *vcpu) { return (!__is_defined(__KVM_NVHE_HYPERVISOR__) && @@ -135,4 +137,10 @@ void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p, #define KVM_NV_GUEST_MAP_SZ GENMASK_ULL(56, 55) +static inline u64 kvm_encode_nested_level(struct kvm_s2_trans *trans) +{ + return FIELD_PREP(KVM_PGTABLE_PROT_SW1 | KVM_PGTABLE_PROT_SW0, + trans->level); +} + #endif /* __ARM64_KVM_NESTED_H */ diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 7e640cae732d..388a6d25bf34 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -1157,11 +1157,17 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, * Potentially reduce shadow S2 permissions to match the guest's own * S2. For exec faults, we'd only reach this point if the guest * actually allowed it (see kvm_s2_handle_perm_fault). + * + * Also encode the level of the nested translation in the SW bits of + * the PTE/PMD/PUD. This will be retrived on TLB invalidation from + * the guest. */ if (kvm_is_shadow_s2_fault(vcpu)) { writable &= kvm_s2_trans_writable(nested); if (!kvm_s2_trans_readable(nested)) prot &= ~KVM_PGTABLE_PROT_R; + + prot |= kvm_encode_nested_level(nested); } spin_lock(&kvm->mmu_lock); From patchwork Mon Nov 29 20:01:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645901 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE832C433EF for ; Mon, 29 Nov 2021 22:20:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233092AbhK2WX3 (ORCPT ); Mon, 29 Nov 2021 17:23:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56560 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233171AbhK2WVb (ORCPT ); Mon, 29 Nov 2021 17:21:31 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1CE84C091D39 for ; Mon, 29 Nov 2021 12:07:35 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id B8020B8159E for ; Mon, 29 Nov 2021 20:07:34 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7052BC53FD1; Mon, 29 Nov 2021 20:07:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216453; bh=DODeF9o8Sl2vkui25obN2Ct4mXtkcYS1+9H7aC1HxUc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=MXc+IN92EBu4f/S2rydJWBg9Q6A8A2MrodhVfaR1fVuv8Xv35eMEFlzO//ilAlYUm Q/MFd0HQtn6ItWWopzC2vqN35M4n4EjJ4PfZtF25Nsgc/AK+Rxe1Y5RepewpKMEidi bPpa2jO9zvR6Gt9h5uGBKgNJ44rpxBKILR8oMPmpHldnCDkUH96tbrtDQHmKLL/AwN smP3TCVx+p5YEnfso7Uo+8M725+dsi5Tq08DWswJKUbO8xVFqKQHhKrEhIXGdDPF01 EUbw6Wyi/RdBA2fUVZeKHwaZkTn2ulGkEFRf4HYGUD/LlluJupdUVq1uM1bUvH9tFD +NjviMnjrYD9w== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmrG-008gvR-Nj; Mon, 29 Nov 2021 20:02:38 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 61/69] KVM: arm64: nv: Add include containing the VNCR_EL2 offsets Date: Mon, 29 Nov 2021 20:01:42 +0000 Message-Id: <20211129200150.351436-62-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org VNCR_EL2 points to a page containing a number of system registers accessed by a guest hypervisor when ARMv8.4-NV is enabled. Let's document the offsets in that page, as we are going to use this layout. Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/vncr_mapping.h | 74 +++++++++++++++++++++++++++ 1 file changed, 74 insertions(+) create mode 100644 arch/arm64/include/asm/vncr_mapping.h diff --git a/arch/arm64/include/asm/vncr_mapping.h b/arch/arm64/include/asm/vncr_mapping.h new file mode 100644 index 000000000000..c0a2acd5045c --- /dev/null +++ b/arch/arm64/include/asm/vncr_mapping.h @@ -0,0 +1,74 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * System register offsets in the VNCR page + * All offsets are *byte* displacements! + */ + +#ifndef __ARM64_VNCR_MAPPING_H__ +#define __ARM64_VNCR_MAPPING_H__ + +#define VNCR_VTTBR_EL2 0x020 +#define VNCR_VTCR_EL2 0x040 +#define VNCR_VMPIDR_EL2 0x050 +#define VNCR_CNTVOFF_EL2 0x060 +#define VNCR_HCR_EL2 0x078 +#define VNCR_HSTR_EL2 0x080 +#define VNCR_VPIDR_EL2 0x088 +#define VNCR_TPIDR_EL2 0x090 +#define VNCR_VNCR_EL2 0x0B0 +#define VNCR_CPACR_EL1 0x100 +#define VNCR_CONTEXTIDR_EL1 0x108 +#define VNCR_SCTLR_EL1 0x110 +#define VNCR_ACTLR_EL1 0x118 +#define VNCR_TCR_EL1 0x120 +#define VNCR_AFSR0_EL1 0x128 +#define VNCR_AFSR1_EL1 0x130 +#define VNCR_ESR_EL1 0x138 +#define VNCR_MAIR_EL1 0x140 +#define VNCR_AMAIR_EL1 0x148 +#define VNCR_MDSCR_EL1 0x158 +#define VNCR_SPSR_EL1 0x160 +#define VNCR_CNTV_CVAL_EL0 0x168 +#define VNCR_CNTV_CTL_EL0 0x170 +#define VNCR_CNTP_CVAL_EL0 0x178 +#define VNCR_CNTP_CTL_EL0 0x180 +#define VNCR_SCXTNUM_EL1 0x188 +#define VNCR_TFSR_EL1 0x190 +#define VNCR_ZCR_EL1 0x1E0 +#define VNCR_TTBR0_EL1 0x200 +#define VNCR_TTBR1_EL1 0x210 +#define VNCR_FAR_EL1 0x220 +#define VNCR_ELR_EL1 0x230 +#define VNCR_SP_EL1 0x240 +#define VNCR_VBAR_EL1 0x250 +#define VNCR_ICH_LR0_EL2 0x400 +// VNCR_ICH_LRN_EL2(n) VNCR_ICH_LR0_EL2+8*((n) & 7) +#define VNCR_ICH_AP0R0_EL2 0x480 +// VNCR_ICH_AP0RN_EL2(n) VNCR_ICH_AP0R0_EL2+8*((n) & 3) +#define VNCR_ICH_AP1R0_EL2 0x4A0 +// VNCR_ICH_AP1RN_EL2(n) VNCR_ICH_AP1R0_EL2+8*((n) & 3) +#define VNCR_ICH_HCR_EL2 0x4C0 +#define VNCR_ICH_VMCR_EL2 0x4C8 +#define VNCR_VDISR_EL2 0x500 +#define VNCR_PMBLIMITR_EL1 0x800 +#define VNCR_PMBPTR_EL1 0x810 +#define VNCR_PMBSR_EL1 0x820 +#define VNCR_PMSCR_EL1 0x828 +#define VNCR_PMSEVFR_EL1 0x830 +#define VNCR_PMSICR_EL1 0x838 +#define VNCR_PMSIRR_EL1 0x840 +#define VNCR_PMSLATFR_EL1 0x848 +#define VNCR_TRFCR_EL1 0x880 +#define VNCR_MPAM1_EL1 0x900 +#define VNCR_MPAMHCR_EL2 0x930 +#define VNCR_MPAMVPMV_EL2 0x938 +#define VNCR_MPAMVPM0_EL2 0x940 +#define VNCR_MPAMVPM1_EL2 0x948 +#define VNCR_MPAMVPM2_EL2 0x950 +#define VNCR_MPAMVPM3_EL2 0x958 +#define VNCR_MPAMVPM4_EL2 0x960 +#define VNCR_MPAMVPM5_EL2 0x968 +#define VNCR_MPAMVPM6_EL2 0x970 +#define VNCR_MPAMVPM7_EL2 0x978 + +#endif /* __ARM64_VNCR_MAPPING_H__ */ From patchwork Mon Nov 29 20:01:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645745 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C966CC433F5 for ; Mon, 29 Nov 2021 21:44:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232614AbhK2Vrx (ORCPT ); Mon, 29 Nov 2021 16:47:53 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48572 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232041AbhK2Vpw (ORCPT ); Mon, 29 Nov 2021 16:45:52 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2CB18C091D1B for ; Mon, 29 Nov 2021 12:06:00 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id BC4A6CE1407 for ; Mon, 29 Nov 2021 20:05:57 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D235EC53FAD; Mon, 29 Nov 2021 20:05:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216356; bh=7bSxbiUOSJvunqPGUc5ktmwT6CKTn+ndFlA8Du+FKDY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=T3/9l9KxsDQr4L51ohPtTm4vlkGvwbDwK5+xcyGRch1mKwS3ULgtGwWk6pQTACPdg JJebK8cqbAFVx7xTAtZF9ntKG5DlkMgb8PhvKmU2s4EcYGqyL8KRcd7pXfuDiC5ySX 9vMnvPQR9QzFyKyx9ULw2jR8uGMTyHD8QLgTkZiy6htS8q+YqoZ97FBDvRUe6lTQiF sAFULg5JmH4eP3fhOJ+Q55tmNgYQVlhj5wWV6DE8yI5XzaQseTop2rtlX4KXQOSaCT YMJetIAKGfhGAflFxjS1ruqnTQ+2PWE7EQKu4deeXgxLU7eKKtnX179TqxU/QnfIVn AtAtEwP41z2zQ== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmrH-008gvR-2H; Mon, 29 Nov 2021 20:02:39 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 62/69] KVM: arm64: nv: Map VNCR-capable registers to a separate page Date: Mon, 29 Nov 2021 20:01:43 +0000 Message-Id: <20211129200150.351436-63-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org With ARMv8.4-NV, registers that can be directly accessed in memory by the guest have to live at architected offsets in a special page. Let's annotate the sysreg enum to reflect the offset at which they are in this page, whith a little twist: If running on HW that doesn't have the ARMv8.4-NV feature, or even a VM that doesn't use NV, we store all the system registers in the usual sys_regs array. The only difference with the pre-8.4 situation is that VNCR-capable registers are at a "similar" offset as in the VNCR page (we can compute the actual offset at compile time), and that the sys_regs array is both bigger and sparse. Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_host.h | 99 ++++++++++++++++++++----------- 1 file changed, 64 insertions(+), 35 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 911dd25dba8e..327dd7439c06 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -27,6 +27,7 @@ #include #include #include +#include #define __KVM_HAVE_ARCH_INTC_INITIALIZED @@ -181,31 +182,32 @@ struct kvm_vcpu_fault_info { u64 disr_el1; /* Deferred [SError] Status Register */ }; +/* + * VNCR() just places the VNCR_capable registers in the enum after + * __VNCR_START__, and the value (after correction) to be an 8-byte offset + * from the VNCR base. As we don't require the enum to be otherwise ordered, + * we need the terrible hack below to ensure that we correctly size the + * sys_regs array, no matter what. + * + * The __MAX__ macro has been lifted from Sean Eron Anderson's wonderful + * treasure trove of bit hacks: + * https://graphics.stanford.edu/~seander/bithacks.html#IntegerMinOrMax + */ +#define __MAX__(x,y) ((x) ^ (((x) ^ (y)) & -((x) < (y)))) +#define VNCR(r) \ + __before_##r, \ + r = __VNCR_START__ + ((VNCR_ ## r) / 8), \ + __after_##r = __MAX__(__before_##r - 1, r) + enum vcpu_sysreg { __INVALID_SYSREG__, /* 0 is reserved as an invalid value */ MPIDR_EL1, /* MultiProcessor Affinity Register */ CSSELR_EL1, /* Cache Size Selection Register */ - SCTLR_EL1, /* System Control Register */ - ACTLR_EL1, /* Auxiliary Control Register */ - CPACR_EL1, /* Coprocessor Access Control */ - ZCR_EL1, /* SVE Control */ - TTBR0_EL1, /* Translation Table Base Register 0 */ - TTBR1_EL1, /* Translation Table Base Register 1 */ - TCR_EL1, /* Translation Control Register */ - ESR_EL1, /* Exception Syndrome Register */ - AFSR0_EL1, /* Auxiliary Fault Status Register 0 */ - AFSR1_EL1, /* Auxiliary Fault Status Register 1 */ - FAR_EL1, /* Fault Address Register */ - MAIR_EL1, /* Memory Attribute Indirection Register */ - VBAR_EL1, /* Vector Base Address Register */ - CONTEXTIDR_EL1, /* Context ID Register */ TPIDR_EL0, /* Thread ID, User R/W */ TPIDRRO_EL0, /* Thread ID, User R/O */ TPIDR_EL1, /* Thread ID, Privileged */ - AMAIR_EL1, /* Aux Memory Attribute Indirection Register */ CNTKCTL_EL1, /* Timer Control Register (EL1) */ PAR_EL1, /* Physical Address Register */ - MDSCR_EL1, /* Monitor Debug System Control Register */ MDCCINT_EL1, /* Monitor Debug Comms Channel Interrupt Enable Reg */ DISR_EL1, /* Deferred Interrupt Status Register */ @@ -235,20 +237,9 @@ enum vcpu_sysreg { APGAKEYLO_EL1, APGAKEYHI_EL1, - ELR_EL1, - SP_EL1, - SPSR_EL1, - - CNTVOFF_EL2, - CNTV_CVAL_EL0, - CNTV_CTL_EL0, - CNTP_CVAL_EL0, - CNTP_CTL_EL0, - /* Memory Tagging Extension registers */ RGSR_EL1, /* Random Allocation Tag Seed Register */ GCR_EL1, /* Tag Control Register */ - TFSR_EL1, /* Tag Fault Status Register (EL1) */ TFSRE0_EL1, /* Tag Fault Status Register (EL0) */ /* 32bit specific registers. */ @@ -258,20 +249,14 @@ enum vcpu_sysreg { DBGVCR32_EL2, /* Debug Vector Catch Register */ /* EL2 registers */ - VPIDR_EL2, /* Virtualization Processor ID Register */ - VMPIDR_EL2, /* Virtualization Multiprocessor ID Register */ SCTLR_EL2, /* System Control Register (EL2) */ ACTLR_EL2, /* Auxiliary Control Register (EL2) */ - HCR_EL2, /* Hypervisor Configuration Register */ MDCR_EL2, /* Monitor Debug Configuration Register (EL2) */ CPTR_EL2, /* Architectural Feature Trap Register (EL2) */ - HSTR_EL2, /* Hypervisor System Trap Register */ HACR_EL2, /* Hypervisor Auxiliary Control Register */ TTBR0_EL2, /* Translation Table Base Register 0 (EL2) */ TTBR1_EL2, /* Translation Table Base Register 1 (EL2) */ TCR_EL2, /* Translation Control Register (EL2) */ - VTTBR_EL2, /* Virtualization Translation Table Base Register */ - VTCR_EL2, /* Virtualization Translation Control Register */ SPSR_EL2, /* EL2 saved program status register */ ELR_EL2, /* EL2 exception link register */ AFSR0_EL2, /* Auxiliary Fault Status Register 0 (EL2) */ @@ -284,7 +269,6 @@ enum vcpu_sysreg { VBAR_EL2, /* Vector Base Address Register (EL2) */ RVBAR_EL2, /* Reset Vector Base Address Register */ CONTEXTIDR_EL2, /* Context ID Register (EL2) */ - TPIDR_EL2, /* EL2 Software Thread ID Register */ CNTHCTL_EL2, /* Counter-timer Hypervisor Control register */ SP_EL2, /* EL2 Stack Pointer */ CNTHP_CTL_EL2, @@ -292,6 +276,42 @@ enum vcpu_sysreg { CNTHV_CTL_EL2, CNTHV_CVAL_EL2, + __VNCR_START__, /* Any VNCR-capable reg goes after this point */ + + VNCR(SCTLR_EL1),/* System Control Register */ + VNCR(ACTLR_EL1),/* Auxiliary Control Register */ + VNCR(CPACR_EL1),/* Coprocessor Access Control */ + VNCR(ZCR_EL1), /* SVE Control */ + VNCR(TTBR0_EL1),/* Translation Table Base Register 0 */ + VNCR(TTBR1_EL1),/* Translation Table Base Register 1 */ + VNCR(TCR_EL1), /* Translation Control Register */ + VNCR(ESR_EL1), /* Exception Syndrome Register */ + VNCR(AFSR0_EL1),/* Auxiliary Fault Status Register 0 */ + VNCR(AFSR1_EL1),/* Auxiliary Fault Status Register 1 */ + VNCR(FAR_EL1), /* Fault Address Register */ + VNCR(MAIR_EL1), /* Memory Attribute Indirection Register */ + VNCR(VBAR_EL1), /* Vector Base Address Register */ + VNCR(CONTEXTIDR_EL1), /* Context ID Register */ + VNCR(AMAIR_EL1),/* Aux Memory Attribute Indirection Register */ + VNCR(MDSCR_EL1),/* Monitor Debug System Control Register */ + VNCR(ELR_EL1), + VNCR(SP_EL1), + VNCR(SPSR_EL1), + VNCR(TFSR_EL1), /* Tag Fault Status Register (EL1) */ + VNCR(VPIDR_EL2),/* Virtualization Processor ID Register */ + VNCR(VMPIDR_EL2),/* Virtualization Multiprocessor ID Register */ + VNCR(HCR_EL2), /* Hypervisor Configuration Register */ + VNCR(HSTR_EL2), /* Hypervisor System Trap Register */ + VNCR(VTTBR_EL2),/* Virtualization Translation Table Base Register */ + VNCR(VTCR_EL2), /* Virtualization Translation Control Register */ + VNCR(TPIDR_EL2),/* EL2 Software Thread ID Register */ + + VNCR(CNTVOFF_EL2), + VNCR(CNTV_CVAL_EL0), + VNCR(CNTV_CTL_EL0), + VNCR(CNTP_CVAL_EL0), + VNCR(CNTP_CTL_EL0), + NR_SYS_REGS /* Nothing after this line! */ }; @@ -308,6 +328,9 @@ struct kvm_cpu_context { u64 sys_regs[NR_SYS_REGS]; struct kvm_vcpu *__hyp_running_vcpu; + + /* This pointer has to be 4kB aligned. */ + u64 *vncr_array; }; struct kvm_pmu_events { @@ -541,7 +564,13 @@ struct kvm_vcpu_arch { * for system registers that are never context switched, but only * emulated. */ -#define __ctxt_sys_reg(c,r) (&(c)->sys_regs[(r)]) +static inline u64 *__ctxt_sys_reg(const struct kvm_cpu_context *ctxt, int r) +{ + if (unlikely(r >= __VNCR_START__ && ctxt->vncr_array)) + return &ctxt->vncr_array[r - __VNCR_START__]; + + return (u64 *)&ctxt->sys_regs[r]; +} #define ctxt_sys_reg(c,r) (*__ctxt_sys_reg(c,r)) From patchwork Mon Nov 29 20:01:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645769 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BCBFDC433EF for ; Mon, 29 Nov 2021 21:46:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232245AbhK2Vtr (ORCPT ); Mon, 29 Nov 2021 16:49:47 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48978 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231929AbhK2Vrq (ORCPT ); Mon, 29 Nov 2021 16:47:46 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5BD7DC08EB3E for ; Mon, 29 Nov 2021 12:06:42 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id D20EAB815E0 for ; Mon, 29 Nov 2021 20:06:41 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 72788C53FCF; Mon, 29 Nov 2021 20:06:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216400; bh=fT7T/fXWnOa1rLuwoJTWenFMrKmq+HeFBMxexUgJabc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=mRLgufpjrK1H1dcX0qi2yBboJRBmzFjFRcYTpoVfScsrE6qVY1OeXy3/DCJxEcveq DsjZJ4rO8d3skcJCTeZ8AxQ+kiWfFe29dP6v0fXLy79vWmXODPm4RWlsZ1cUvBY8SY NvVIx1X+SwN0/bg5s5UUFRn6xdytKluuLga+JALv4vp42ZgsxNVNhXKNwBNNi0URVZ M348ghcnPZ108JuyBB1Frea8tpnNTavFD6uQi1dWbqxuKcA1xfUYy+2zlp7l7ahKCT /dzqiPyjeOOxBaaEkY5t8dr9+8hMmRwk7PlvpQcseykFF0ym9Y5uAWPdxxJ8EEACgZ wYbe/wKx/hbwA== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmrH-008gvR-CI; Mon, 29 Nov 2021 20:02:39 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 63/69] KVM: arm64: nv: Move nested vgic state into the sysreg file Date: Mon, 29 Nov 2021 20:01:44 +0000 Message-Id: <20211129200150.351436-64-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The vgic nested state needs to be accessible from the VNCR page, and thus needs to be part of the normal sysreg file. Let's move it there. Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_host.h | 9 +++ arch/arm64/kvm/sys_regs.c | 53 +++++++++++------ arch/arm64/kvm/vgic/vgic-v3-nested.c | 88 ++++++++++++++-------------- arch/arm64/kvm/vgic/vgic-v3.c | 17 ++++-- arch/arm64/kvm/vgic/vgic.h | 10 ++++ include/kvm/arm_vgic.h | 7 --- 6 files changed, 110 insertions(+), 74 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 327dd7439c06..c21551551ddf 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -312,6 +312,15 @@ enum vcpu_sysreg { VNCR(CNTP_CVAL_EL0), VNCR(CNTP_CTL_EL0), + VNCR(ICH_LR0_EL2), + ICH_LR15_EL2 = ICH_LR0_EL2 + 15, + VNCR(ICH_AP0R0_EL2), + ICH_AP0R3_EL2 = ICH_AP0R0_EL2 + 3, + VNCR(ICH_AP1R0_EL2), + ICH_AP1R3_EL2 = ICH_AP1R0_EL2 + 3, + VNCR(ICH_HCR_EL2), + VNCR(ICH_VMCR_EL2), + NR_SYS_REGS /* Nothing after this line! */ }; diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 7400a76f6261..e8ab052be122 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -1874,17 +1874,17 @@ static bool access_gic_apr(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) { - struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.nested_vgic_v3; - u32 index, *base; + u64 *base; + u8 index; index = r->Op2; if (r->CRm == 8) - base = cpu_if->vgic_ap0r; + base = __ctxt_sys_reg(&vcpu->arch.ctxt, ICH_AP0R0_EL2); else - base = cpu_if->vgic_ap1r; + base = __ctxt_sys_reg(&vcpu->arch.ctxt, ICH_AP1R0_EL2); if (p->is_write) - base[index] = p->regval; + base[index] = lower_32_bits(p->regval); else p->regval = base[index]; @@ -1895,12 +1895,10 @@ static bool access_gic_hcr(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) { - struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.nested_vgic_v3; - if (p->is_write) - cpu_if->vgic_hcr = p->regval; + __vcpu_sys_reg(vcpu, ICH_HCR_EL2) = lower_32_bits(p->regval); else - p->regval = cpu_if->vgic_hcr; + p->regval = __vcpu_sys_reg(vcpu, ICH_HCR_EL2); return true; } @@ -1957,12 +1955,19 @@ static bool access_gic_vmcr(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) { - struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.nested_vgic_v3; - if (p->is_write) - cpu_if->vgic_vmcr = p->regval; + __vcpu_sys_reg(vcpu, ICH_VMCR_EL2) = (p->regval & + (ICH_VMCR_ENG0_MASK | + ICH_VMCR_ENG1_MASK | + ICH_VMCR_PMR_MASK | + ICH_VMCR_BPR0_MASK | + ICH_VMCR_BPR1_MASK | + ICH_VMCR_EOIM_MASK | + ICH_VMCR_CBPR_MASK | + ICH_VMCR_FIQ_EN_MASK | + ICH_VMCR_ACK_CTL_MASK)); else - p->regval = cpu_if->vgic_vmcr; + p->regval = __vcpu_sys_reg(vcpu, ICH_VMCR_EL2); return true; } @@ -1971,17 +1976,29 @@ static bool access_gic_lr(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) { - struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.nested_vgic_v3; u32 index; + u64 *base; + base = __ctxt_sys_reg(&vcpu->arch.ctxt, ICH_LR0_EL2); index = p->Op2; if (p->CRm == 13) index += 8; - if (p->is_write) - cpu_if->vgic_lr[index] = p->regval; - else - p->regval = cpu_if->vgic_lr[index]; + if (p->is_write) { + u64 mask = (ICH_LR_VIRTUAL_ID_MASK | + ICH_LR_GROUP | + ICH_LR_HW | + ICH_LR_STATE); + + if (p->regval & ICH_LR_HW) + mask |= ICH_LR_PHYS_ID_MASK; + else + mask |= ICH_LR_EOI; + + base[index] = p->regval & mask; + } else { + p->regval = base[index]; + } return true; } diff --git a/arch/arm64/kvm/vgic/vgic-v3-nested.c b/arch/arm64/kvm/vgic/vgic-v3-nested.c index 94b1edb67011..51f7a521e829 100644 --- a/arch/arm64/kvm/vgic/vgic-v3-nested.c +++ b/arch/arm64/kvm/vgic/vgic-v3-nested.c @@ -16,11 +16,6 @@ #define CREATE_TRACE_POINTS #include "vgic-nested-trace.h" -static inline struct vgic_v3_cpu_if *vcpu_nested_if(struct kvm_vcpu *vcpu) -{ - return &vcpu->arch.vgic_cpu.nested_vgic_v3; -} - static inline struct vgic_v3_cpu_if *vcpu_shadow_if(struct kvm_vcpu *vcpu) { return &vcpu->arch.vgic_cpu.shadow_vgic_v3; @@ -33,12 +28,11 @@ static inline bool lr_triggers_eoi(u64 lr) u16 vgic_v3_get_eisr(struct kvm_vcpu *vcpu) { - struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); u16 reg = 0; int i; for (i = 0; i < kvm_vgic_global_state.nr_lr; i++) { - if (lr_triggers_eoi(cpu_if->vgic_lr[i])) + if (lr_triggers_eoi(__vcpu_sys_reg(vcpu, ICH_LRN(i)))) reg |= BIT(i); } @@ -47,12 +41,11 @@ u16 vgic_v3_get_eisr(struct kvm_vcpu *vcpu) u16 vgic_v3_get_elrsr(struct kvm_vcpu *vcpu) { - struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); u16 reg = 0; int i; for (i = 0; i < kvm_vgic_global_state.nr_lr; i++) { - if (!(cpu_if->vgic_lr[i] & ICH_LR_STATE)) + if (!(__vcpu_sys_reg(vcpu, ICH_LRN(i)) & ICH_LR_STATE)) reg |= BIT(i); } @@ -61,14 +54,13 @@ u16 vgic_v3_get_elrsr(struct kvm_vcpu *vcpu) u64 vgic_v3_get_misr(struct kvm_vcpu *vcpu) { - struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); int nr_lr = kvm_vgic_global_state.nr_lr; u64 reg = 0; if (vgic_v3_get_eisr(vcpu)) reg |= ICH_MISR_EOI; - if (cpu_if->vgic_hcr & ICH_HCR_UIE) { + if (__vcpu_sys_reg(vcpu, ICH_HCR_EL2) & ICH_HCR_UIE) { int used_lrs; used_lrs = nr_lr - hweight16(vgic_v3_get_elrsr(vcpu)); @@ -87,13 +79,12 @@ u64 vgic_v3_get_misr(struct kvm_vcpu *vcpu) */ static void vgic_v3_create_shadow_lr(struct kvm_vcpu *vcpu) { - struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); struct vgic_v3_cpu_if *s_cpu_if = vcpu_shadow_if(vcpu); struct vgic_irq *irq; int i, used_lrs = 0; for (i = 0; i < kvm_vgic_global_state.nr_lr; i++) { - u64 lr = cpu_if->vgic_lr[i]; + u64 lr = __vcpu_sys_reg(vcpu, ICH_LRN(i)); int l1_irq; if (!(lr & ICH_LR_HW)) @@ -123,36 +114,20 @@ static void vgic_v3_create_shadow_lr(struct kvm_vcpu *vcpu) } trace_vgic_create_shadow_lrs(vcpu, kvm_vgic_global_state.nr_lr, - s_cpu_if->vgic_lr, cpu_if->vgic_lr); + s_cpu_if->vgic_lr, + __ctxt_sys_reg(&vcpu->arch.ctxt, ICH_LR0_EL2)); s_cpu_if->used_lrs = used_lrs; } -/* - * Change the shadow HWIRQ field back to the virtual value before copying over - * the entire shadow struct to the nested state. - */ -static void vgic_v3_fixup_shadow_lr_state(struct kvm_vcpu *vcpu) -{ - struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); - struct vgic_v3_cpu_if *s_cpu_if = vcpu_shadow_if(vcpu); - int lr; - - for (lr = 0; lr < kvm_vgic_global_state.nr_lr; lr++) { - s_cpu_if->vgic_lr[lr] &= ~ICH_LR_PHYS_ID_MASK; - s_cpu_if->vgic_lr[lr] |= cpu_if->vgic_lr[lr] & ICH_LR_PHYS_ID_MASK; - } -} - void vgic_v3_sync_nested(struct kvm_vcpu *vcpu) { - struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); struct vgic_v3_cpu_if *s_cpu_if = vcpu_shadow_if(vcpu); struct vgic_irq *irq; int i; for (i = 0; i < s_cpu_if->used_lrs; i++) { - u64 lr = cpu_if->vgic_lr[i]; + u64 lr = __vcpu_sys_reg(vcpu, ICH_LRN(i)); int l1_irq; if (!(lr & ICH_LR_HW) || !(lr & ICH_LR_STATE)) @@ -178,14 +153,27 @@ void vgic_v3_sync_nested(struct kvm_vcpu *vcpu) } } +void vgic_v3_create_shadow_state(struct kvm_vcpu *vcpu) +{ + struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.shadow_vgic_v3; + int i; + + cpu_if->vgic_hcr = __vcpu_sys_reg(vcpu, ICH_HCR_EL2); + cpu_if->vgic_vmcr = __vcpu_sys_reg(vcpu, ICH_VMCR_EL2); + + for (i = 0; i < 4; i++) { + cpu_if->vgic_ap0r[i] = __vcpu_sys_reg(vcpu, ICH_AP0RN(i)); + cpu_if->vgic_ap1r[i] = __vcpu_sys_reg(vcpu, ICH_AP1RN(i)); + } + + vgic_v3_create_shadow_lr(vcpu); +} + void vgic_v3_load_nested(struct kvm_vcpu *vcpu) { - struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu; struct vgic_irq *irq; unsigned long flags; - vgic_cpu->shadow_vgic_v3 = vgic_cpu->nested_vgic_v3; - vgic_v3_create_shadow_lr(vcpu); __vgic_v3_restore_state(vcpu_shadow_if(vcpu)); irq = vgic_get_irq(vcpu->kvm, vcpu, vcpu->kvm->arch.vgic.maint_irq); @@ -199,26 +187,40 @@ void vgic_v3_load_nested(struct kvm_vcpu *vcpu) void vgic_v3_put_nested(struct kvm_vcpu *vcpu) { - struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu; + struct vgic_v3_cpu_if *s_cpu_if = vcpu_shadow_if(vcpu); + int i; - __vgic_v3_save_state(vcpu_shadow_if(vcpu)); + __vgic_v3_save_state(s_cpu_if); - trace_vgic_put_nested(vcpu, kvm_vgic_global_state.nr_lr, - vcpu_shadow_if(vcpu)->vgic_lr); + trace_vgic_put_nested(vcpu, kvm_vgic_global_state.nr_lr, s_cpu_if->vgic_lr); /* * Translate the shadow state HW fields back to the virtual ones * before copying the shadow struct back to the nested one. */ - vgic_v3_fixup_shadow_lr_state(vcpu); - vgic_cpu->nested_vgic_v3 = vgic_cpu->shadow_vgic_v3; + __vcpu_sys_reg(vcpu, ICH_HCR_EL2) = s_cpu_if->vgic_hcr; + __vcpu_sys_reg(vcpu, ICH_VMCR_EL2) = s_cpu_if->vgic_vmcr; + + for (i = 0; i < 4; i++) { + __vcpu_sys_reg(vcpu, ICH_AP0RN(i)) = s_cpu_if->vgic_ap0r[i]; + __vcpu_sys_reg(vcpu, ICH_AP1RN(i)) = s_cpu_if->vgic_ap1r[i]; + } + + for (i = 0; i < kvm_vgic_global_state.nr_lr; i++) { + u64 val = __vcpu_sys_reg(vcpu, ICH_LRN(i)); + + val &= ~ICH_LR_STATE; + val |= s_cpu_if->vgic_lr[i] & ICH_LR_STATE; + + __vcpu_sys_reg(vcpu, ICH_LRN(i)) = val; + } + irq_set_irqchip_state(kvm_vgic_global_state.maint_irq, IRQCHIP_STATE_ACTIVE, false); } void vgic_v3_handle_nested_maint_irq(struct kvm_vcpu *vcpu) { - struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); bool state; /* @@ -230,7 +232,7 @@ void vgic_v3_handle_nested_maint_irq(struct kvm_vcpu *vcpu) if (!vgic_state_is_nested(vcpu)) return; - state = cpu_if->vgic_hcr & ICH_HCR_EN; + state = __vcpu_sys_reg(vcpu, ICH_HCR_EL2) & ICH_HCR_EN; state &= vgic_v3_get_misr(vcpu); kvm_vgic_inject_irq(vcpu->kvm, vcpu->vcpu_id, diff --git a/arch/arm64/kvm/vgic/vgic-v3.c b/arch/arm64/kvm/vgic/vgic-v3.c index 7b575df62fa9..abbbc0f1cb1a 100644 --- a/arch/arm64/kvm/vgic/vgic-v3.c +++ b/arch/arm64/kvm/vgic/vgic-v3.c @@ -280,10 +280,11 @@ void vgic_v3_enable(struct kvm_vcpu *vcpu) ICC_SRE_EL1_SRE); /* * If nesting is allowed, force GICv3 onto the nested - * guests as well. + * guests as well by setting the shadow state to the + * same value. */ if (nested_virt_in_use(vcpu)) - vcpu->arch.vgic_cpu.nested_vgic_v3.vgic_sre = vgic_v3->vgic_sre; + vcpu->arch.vgic_cpu.shadow_vgic_v3.vgic_sre = vgic_v3->vgic_sre; vcpu->arch.vgic_cpu.pendbaser = INITIAL_PENDBASER_VALUE; } else { vgic_v3->vgic_sre = 0; @@ -715,11 +716,15 @@ void vgic_v3_load(struct kvm_vcpu *vcpu) struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v3; /* - * vgic_v3_load_nested only affects the LRs in the shadow - * state, so it is fine to pass the nested state around. + * If the vgic is in nested state, populate the shadow state + * from the guest's nested state. As vgic_v3_load_nested() + * will only load LRs, let's deal with the rest of the state + * here as if it was a non-nested state. Cunning. */ - if (vgic_state_is_nested(vcpu)) - cpu_if = &vcpu->arch.vgic_cpu.nested_vgic_v3; + if (vgic_state_is_nested(vcpu)) { + vgic_v3_create_shadow_state(vcpu); + cpu_if = &vcpu->arch.vgic_cpu.shadow_vgic_v3; + } /* * If dealing with a GICv2 emulation on GICv3, VMCR_EL2.VFIQen diff --git a/arch/arm64/kvm/vgic/vgic.h b/arch/arm64/kvm/vgic/vgic.h index 3fd6c86a7ef3..ffdfe7bd9aea 100644 --- a/arch/arm64/kvm/vgic/vgic.h +++ b/arch/arm64/kvm/vgic/vgic.h @@ -323,4 +323,14 @@ void vgic_v4_teardown(struct kvm *kvm); void vgic_v4_configure_vsgis(struct kvm *kvm); void vgic_v4_get_vlpi_state(struct vgic_irq *irq, bool *val); +void vgic_v3_sync_nested(struct kvm_vcpu *vcpu); +void vgic_v3_create_shadow_state(struct kvm_vcpu *vcpu); +void vgic_v3_load_nested(struct kvm_vcpu *vcpu); +void vgic_v3_put_nested(struct kvm_vcpu *vcpu); +void vgic_v3_handle_nested_maint_irq(struct kvm_vcpu *vcpu); + +#define ICH_LRN(n) (ICH_LR0_EL2 + (n)) +#define ICH_AP0RN(n) (ICH_AP0R0_EL2 + (n)) +#define ICH_AP1RN(n) (ICH_AP1R0_EL2 + (n)) + #endif diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h index 3ffc89c86144..88bb525ed658 100644 --- a/include/kvm/arm_vgic.h +++ b/include/kvm/arm_vgic.h @@ -328,9 +328,6 @@ struct vgic_cpu { struct vgic_irq private_irqs[VGIC_NR_PRIVATE_IRQS]; - /* CPU vif control registers for the virtual GICH interface */ - struct vgic_v3_cpu_if nested_vgic_v3; - /* * The shadow vif control register loaded to the hardware when * running a nested L2 guest with the virtual IMO/FMO bit set. @@ -394,10 +391,6 @@ void kvm_vgic_load(struct kvm_vcpu *vcpu); void kvm_vgic_put(struct kvm_vcpu *vcpu); void kvm_vgic_vmcr_sync(struct kvm_vcpu *vcpu); -void vgic_v3_sync_nested(struct kvm_vcpu *vcpu); -void vgic_v3_load_nested(struct kvm_vcpu *vcpu); -void vgic_v3_put_nested(struct kvm_vcpu *vcpu); -void vgic_v3_handle_nested_maint_irq(struct kvm_vcpu *vcpu); u16 vgic_v3_get_eisr(struct kvm_vcpu *vcpu); u16 vgic_v3_get_elrsr(struct kvm_vcpu *vcpu); u64 vgic_v3_get_misr(struct kvm_vcpu *vcpu); From patchwork Mon Nov 29 20:01:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645787 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D555BC433F5 for ; Mon, 29 Nov 2021 21:46:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233020AbhK2Vt6 (ORCPT ); Mon, 29 Nov 2021 16:49:58 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49472 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232048AbhK2Vr5 (ORCPT ); Mon, 29 Nov 2021 16:47:57 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 959CCC091D30 for ; Mon, 29 Nov 2021 12:07:15 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 17211CE16B6 for ; Mon, 29 Nov 2021 20:07:15 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4A752C53FAD; Mon, 29 Nov 2021 20:07:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216433; bh=/ysEoteWJ0uRsBiYXo1IVEx3ohrHhoTwm2G//NhhS58=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ty2ekLnc5BmWo/XKBRnEWYm4thmOIvF4XspmAgo3G82FIjFS98yN6F2OahJ61iugX /0Zwjk8J7BQWuwHYCCqNizsbEcwFS/cPsX4ICYdAlVeKhfOr8VyLOlTm6cTZoUEgX4 LgQ24Mb+D/9KZovUTbRoYu1slIVfQxEWvXEDRRHyD9CGOVBgki6rTwANhiZZNT3gwg W0ktOApek5VN/NJ09QIi1bLsJKYUN9vXUEpxpoACVFt2oIcmugzVensNXMN4pok4JM ZRGEo79pV6XzYKT20w34JBT8nPX9GpH583u8jtdbjKLoIJiLw3o5bDg0m6ZRxSqZK9 nOQ2RVwxlfL4Q== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmrH-008gvR-NR; Mon, 29 Nov 2021 20:02:39 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 64/69] KVM: arm64: Add ARMv8.4 Enhanced Nested Virt cpufeature Date: Mon, 29 Nov 2021 20:01:45 +0000 Message-Id: <20211129200150.351436-65-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Add the detection code for the ARMv8.4-NV feature. Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_nested.h | 6 ++++++ arch/arm64/kernel/cpufeature.c | 10 ++++++++++ arch/arm64/tools/cpucaps | 1 + 3 files changed, 17 insertions(+) diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index 76fbf5de1f2c..7fb7ae97ed88 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -14,6 +14,12 @@ static inline bool nested_virt_in_use(const struct kvm_vcpu *vcpu) test_bit(KVM_ARM_VCPU_HAS_EL2, vcpu->arch.features)); } +static inline bool enhanced_nested_virt_in_use(const struct kvm_vcpu *vcpu) +{ + return cpus_have_final_cap(ARM64_HAS_ENHANCED_NESTED_VIRT) && + nested_virt_in_use(vcpu); +} + /* Translation helpers from non-VHE EL2 to EL1 */ static inline u64 tcr_el2_ips_to_tcr_el1_ps(u64 tcr_el2) { diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 4d57e98cdde5..0c83f4b38ed8 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -2018,6 +2018,16 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .field_pos = ID_AA64MMFR2_NV_SHIFT, .min_field_value = 1, }, + { + .desc = "Enhanced Nested Virtualization Support", + .capability = ARM64_HAS_ENHANCED_NESTED_VIRT, + .type = ARM64_CPUCAP_SYSTEM_FEATURE, + .matches = has_nested_virt_support, + .sys_reg = SYS_ID_AA64MMFR2_EL1, + .sign = FTR_UNSIGNED, + .field_pos = ID_AA64MMFR2_NV_SHIFT, + .min_field_value = 2, + }, { .capability = ARM64_HAS_32BIT_EL0_DO_NOT_USE, .type = ARM64_CPUCAP_SYSTEM_FEATURE, diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps index a49864b56a07..3f6f567d2811 100644 --- a/arch/arm64/tools/cpucaps +++ b/arch/arm64/tools/cpucaps @@ -19,6 +19,7 @@ HAS_DCPODP HAS_DCPOP HAS_E0PD HAS_ECV +HAS_ENHANCED_NESTED_VIRT HAS_EPAN HAS_GENERIC_AUTH HAS_GENERIC_AUTH_ARCH From patchwork Mon Nov 29 20:01:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645791 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56454C433F5 for ; Mon, 29 Nov 2021 21:48:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231694AbhK2Vvo (ORCPT ); Mon, 29 Nov 2021 16:51:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49404 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231749AbhK2Vtn (ORCPT ); Mon, 29 Nov 2021 16:49:43 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 76DA1C091D33 for ; Mon, 29 Nov 2021 12:07:17 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 03F5DCE1415 for ; Mon, 29 Nov 2021 20:07:17 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 74A25C53FAD; Mon, 29 Nov 2021 20:07:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216435; bh=73WEl2HJUcNvT9k7Ef97DgLn0GX0d41PDzy3ICstomo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ACVv2xOGlyYai2+rLvYimdDuBF7DCOsW3O/vWj3NCEWeKfgeciNo8hXMdUIrIm19Z r+8LRRLKMC9hkza8gErRokgllQ/cl22Raga/zhEqsCdTVFug4nunjzklplm8Z6cplT kqz4/UVpL+M1yZwiWwcBAvXBTdf6uUgGUfcqRUCKhXX58kSWbIeBSqyNi933ZeQt4S kbhc58vyabkcDLgYOZQYuNzJoLtYmC0U6DZvSUk7XiKu7FU9HcEHGXBJD112h3UaFI TuM12uE3juqt64+s9c3ikulgbGPtBZwLCGcOi7Zx6iOvlKYfrHfoGxTNbjHInvPCBI DdzL2jrUGfZRQ== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmrI-008gvR-03; Mon, 29 Nov 2021 20:02:40 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 65/69] KVM: arm64: nv: Sync nested timer state with ARMv8.4 Date: Mon, 29 Nov 2021 20:01:46 +0000 Message-Id: <20211129200150.351436-66-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Christoffer Dall Emulating the ARMv8.4-NV timers is a bit odd, as the timers can be reconfigured behind our back without the hypervisor even noticing. In the VHE case, that's an actual regression in the architecture... Signed-off-by: Christoffer Dall Signed-off-by: Marc Zyngier --- arch/arm64/kvm/arch_timer.c | 37 ++++++++++++++++++++++++++++++++++++ arch/arm64/kvm/arm.c | 3 +++ include/kvm/arm_arch_timer.h | 1 + 3 files changed, 41 insertions(+) diff --git a/arch/arm64/kvm/arch_timer.c b/arch/arm64/kvm/arch_timer.c index 3e80c3bdd8ce..35fbf0e83b38 100644 --- a/arch/arm64/kvm/arch_timer.c +++ b/arch/arm64/kvm/arch_timer.c @@ -786,6 +786,43 @@ void kvm_timer_vcpu_put(struct kvm_vcpu *vcpu) set_cntvoff(0); } +void kvm_timer_sync_nested(struct kvm_vcpu *vcpu) +{ + if (!is_hyp_ctxt(vcpu)) + return; + + /* + * Guest hypervisors using ARMv8.4 enhanced nested virt support have + * their EL1 timer register accesses redirected to the VNCR page. + */ + if (!vcpu_el2_e2h_is_set(vcpu)) { + /* + * For a non-VHE guest hypervisor, we update the hardware + * timer registers with the latest value written by the guest + * to the VNCR page and let the hardware take care of the + * rest. + */ + write_sysreg_el0(__vcpu_sys_reg(vcpu, CNTV_CTL_EL0), SYS_CNTV_CTL); + write_sysreg_el0(__vcpu_sys_reg(vcpu, CNTV_CVAL_EL0), SYS_CNTV_CVAL); + write_sysreg_el0(__vcpu_sys_reg(vcpu, CNTP_CTL_EL0), SYS_CNTP_CTL); + write_sysreg_el0(__vcpu_sys_reg(vcpu, CNTP_CVAL_EL0), SYS_CNTP_CVAL); + } else { + /* + * For a VHE guest hypervisor, the emulated state (which + * is stored in the VNCR page) could have been updated behind + * our back, and we must reset the emulation of the timers. + */ + + struct timer_map map; + get_timer_map(vcpu, &map); + + soft_timer_cancel(&map.emul_vtimer->hrtimer); + soft_timer_cancel(&map.emul_ptimer->hrtimer); + timer_emulate(map.emul_vtimer); + timer_emulate(map.emul_ptimer); + } +} + /* * With a userspace irqchip we have to check if the guest de-asserted the * timer and if so, unmask the timer irq signal on the host interrupt diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index dc1fda709cf5..f20bdf1deb72 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -902,6 +902,9 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) if (static_branch_unlikely(&userspace_irqchip_in_use)) kvm_timer_sync_user(vcpu); + if (enhanced_nested_virt_in_use(vcpu)) + kvm_timer_sync_nested(vcpu); + kvm_arch_vcpu_ctxsync_fp(vcpu); /* diff --git a/include/kvm/arm_arch_timer.h b/include/kvm/arm_arch_timer.h index 0a76dac8cb6a..89b08e5b456e 100644 --- a/include/kvm/arm_arch_timer.h +++ b/include/kvm/arm_arch_timer.h @@ -68,6 +68,7 @@ int kvm_timer_hyp_init(bool); int kvm_timer_enable(struct kvm_vcpu *vcpu); int kvm_timer_vcpu_reset(struct kvm_vcpu *vcpu); void kvm_timer_vcpu_init(struct kvm_vcpu *vcpu); +void kvm_timer_sync_nested(struct kvm_vcpu *vcpu); void kvm_timer_sync_user(struct kvm_vcpu *vcpu); bool kvm_timer_should_notify_user(struct kvm_vcpu *vcpu); void kvm_timer_update_run(struct kvm_vcpu *vcpu); From patchwork Mon Nov 29 20:01:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645765 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D79D6C433EF for ; Mon, 29 Nov 2021 21:46:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231879AbhK2Vto (ORCPT ); Mon, 29 Nov 2021 16:49:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49848 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231890AbhK2Vrn (ORCPT ); Mon, 29 Nov 2021 16:47:43 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BFDC1C08EB3D for ; Mon, 29 Nov 2021 12:06:40 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 874D1B815E0 for ; Mon, 29 Nov 2021 20:06:39 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 479A0C53FD0; Mon, 29 Nov 2021 20:06:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216398; bh=RUzBdFFRKwomJbEXUVng8IsQUXhzc/leqrUw6Lmm0PE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=rHkGPDG649+5QGiYEEA7f2YHI1eidd1UWQLt3GVjRoA3Iv7tDw2cATK07JzyvrQP/ lZhcGxS2J0xFcJU4fODGwrDET2kZOTl/QLVkH7AUR/j3zSp6ExEYIARdUaeTbd6SFW ZUYzvX956lNA7rgwsl5dQ1qgORB7/w31/kN07xVROe7A8QVUV8zJVumv9O/sfF1D9e qAaEvt8f50mSr2BD3N+OjPg+reVv+KvoWY+TW0igUOgeePh9yxtGv7opG3WmplXrvl p181BrHWx4Q6oYeM8vAXu3STP8Yx7zC39MdvJmcW7bYdr7jIp9c6TDZVQSlM5xSTNU Ise2JMsFWvQkQ== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmrI-008gvR-Bq; Mon, 29 Nov 2021 20:02:40 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 66/69] KVM: arm64: nv: Allocate VNCR page when required Date: Mon, 29 Nov 2021 20:01:47 +0000 Message-Id: <20211129200150.351436-67-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org If running a NV guest on an ARMv8.4-NV capable system, let's allocate an additional page that will be used by the hypervisor to fulfill system register accesses. Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_host.h | 3 ++- arch/arm64/kvm/nested.c | 10 ++++++++++ arch/arm64/kvm/reset.c | 1 + 3 files changed, 13 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index c21551551ddf..595516450637 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -575,7 +575,8 @@ struct kvm_vcpu_arch { */ static inline u64 *__ctxt_sys_reg(const struct kvm_cpu_context *ctxt, int r) { - if (unlikely(r >= __VNCR_START__ && ctxt->vncr_array)) + if (unlikely(cpus_have_final_cap(ARM64_HAS_ENHANCED_NESTED_VIRT) && + r >= __VNCR_START__ && ctxt->vncr_array)) return &ctxt->vncr_array[r - __VNCR_START__]; return (u64 *)&ctxt->sys_regs[r]; diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c index e11bdee15df2..7c9dd1edf011 100644 --- a/arch/arm64/kvm/nested.c +++ b/arch/arm64/kvm/nested.c @@ -47,6 +47,14 @@ int kvm_vcpu_init_nested(struct kvm_vcpu *vcpu) if (!cpus_have_final_cap(ARM64_HAS_NESTED_VIRT)) return -EINVAL; + if (cpus_have_final_cap(ARM64_HAS_ENHANCED_NESTED_VIRT)) { + if (!vcpu->arch.ctxt.vncr_array) + vcpu->arch.ctxt.vncr_array = (u64 *)__get_free_page(GFP_KERNEL | __GFP_ZERO); + + if (!vcpu->arch.ctxt.vncr_array) + return -ENOMEM; + } + mutex_lock(&kvm->lock); /* @@ -76,6 +84,8 @@ int kvm_vcpu_init_nested(struct kvm_vcpu *vcpu) kvm_init_stage2_mmu(kvm, &tmp[num_mmus - 2])) { kvm_free_stage2_pgd(&tmp[num_mmus - 1]); kvm_free_stage2_pgd(&tmp[num_mmus - 2]); + free_page((unsigned long)vcpu->arch.ctxt.vncr_array); + vcpu->arch.ctxt.vncr_array = NULL; } else { kvm->arch.nested_mmus_size = num_mmus; ret = 0; diff --git a/arch/arm64/kvm/reset.c b/arch/arm64/kvm/reset.c index 38a7182819fb..9f580241b0bd 100644 --- a/arch/arm64/kvm/reset.c +++ b/arch/arm64/kvm/reset.c @@ -146,6 +146,7 @@ bool kvm_arm_vcpu_is_finalized(struct kvm_vcpu *vcpu) void kvm_arm_vcpu_destroy(struct kvm_vcpu *vcpu) { kfree(vcpu->arch.sve_state); + free_page((unsigned long)vcpu->arch.ctxt.vncr_array); } static void kvm_vcpu_reset_sve(struct kvm_vcpu *vcpu) From patchwork Mon Nov 29 20:01:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645751 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63B03C433F5 for ; Mon, 29 Nov 2021 21:44:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232706AbhK2Vrz (ORCPT ); Mon, 29 Nov 2021 16:47:55 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49448 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232065AbhK2Vpw (ORCPT ); Mon, 29 Nov 2021 16:45:52 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 87947C091D1D for ; Mon, 29 Nov 2021 12:06:08 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 11134CE140C for ; Mon, 29 Nov 2021 20:06:08 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 667B5C53FC7; Mon, 29 Nov 2021 20:06:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216366; bh=aXOmQCoo+3BavrCGq7viBjKwLUX5oZCvjH/YTIoVHFc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ZkYMQdIGxDYCyy5+mmC80aKxvFcXXV125SCOG8IFKgyo34KHmrzb5/KSdyZXQdyI3 +PfVnkIlSfCOFBI/32NUyLZZgPCc0j62ujrWFv1q5Hrnz/JLpyxz10Bivr1Nedximl Q1hEIQyUdDWjVcYmJxnDSat+Z3h+nILWOLDaiaHOlHmF+hRfCXgVlpWIdCAzC/vdqw IUwZgjlhr7HySNT1EVMuAKC3q5lD3hJO/d1paPIBaNvHX+dBBuG7UL/KmIL2gUUNXN clBlz3oH60CQxXOujIU9/FvWiiXqiOHT/rdtr1RQvAX7Io+igapwYx4ifd9vXPbAxS epVzE7vExqYgA== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmrI-008gvR-NK; Mon, 29 Nov 2021 20:02:40 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 67/69] KVM: arm64: nv: Enable ARMv8.4-NV support Date: Mon, 29 Nov 2021 20:01:48 +0000 Message-Id: <20211129200150.351436-68-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org As all the VNCR-capable system registers are nicely separated from the rest of the crowd, let's set HCR_EL2.NV2 on and let the ball rolling. Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_arm.h | 1 + arch/arm64/include/asm/kvm_emulate.h | 23 +++++++++++++---------- arch/arm64/include/asm/sysreg.h | 1 + arch/arm64/kvm/hyp/vhe/switch.c | 14 +++++++++++++- 4 files changed, 28 insertions(+), 11 deletions(-) diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h index b603466803d2..18c35446249f 100644 --- a/arch/arm64/include/asm/kvm_arm.h +++ b/arch/arm64/include/asm/kvm_arm.h @@ -20,6 +20,7 @@ #define HCR_AMVOFFEN (UL(1) << 51) #define HCR_FIEN (UL(1) << 47) #define HCR_FWB (UL(1) << 46) +#define HCR_NV2 (UL(1) << 45) #define HCR_AT (UL(1) << 44) #define HCR_NV1 (UL(1) << 43) #define HCR_NV (UL(1) << 42) diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h index 1664430be698..f282997e4a4c 100644 --- a/arch/arm64/include/asm/kvm_emulate.h +++ b/arch/arm64/include/asm/kvm_emulate.h @@ -245,21 +245,24 @@ static inline bool is_hyp_ctxt(const struct kvm_vcpu *vcpu) static inline u64 __fixup_spsr_el2_write(struct kvm_cpu_context *ctxt, u64 val) { - if (!__vcpu_el2_e2h_is_set(ctxt)) { - /* - * Clear the .M field when writing SPSR to the CPU, so that we - * can detect when the CPU clobbered our SPSR copy during a - * local exception. - */ - val &= ~0xc; - } + struct kvm_vcpu *vcpu = container_of(ctxt, struct kvm_vcpu, arch.ctxt); + + if (enhanced_nested_virt_in_use(vcpu) || __vcpu_el2_e2h_is_set(ctxt)) + return val; - return val; + /* + * Clear the .M field when writing SPSR to the CPU, so that we + * can detect when the CPU clobbered our SPSR copy during a + * local exception. + */ + return val &= ~0xc; } static inline u64 __fixup_spsr_el2_read(const struct kvm_cpu_context *ctxt, u64 val) { - if (__vcpu_el2_e2h_is_set(ctxt)) + struct kvm_vcpu *vcpu = container_of(ctxt, struct kvm_vcpu, arch.ctxt); + + if (enhanced_nested_virt_in_use(vcpu) || __vcpu_el2_e2h_is_set(ctxt)) return val; /* diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h index 71e6a0410e7c..5de90138d0a4 100644 --- a/arch/arm64/include/asm/sysreg.h +++ b/arch/arm64/include/asm/sysreg.h @@ -550,6 +550,7 @@ #define SYS_TCR_EL2 sys_reg(3, 4, 2, 0, 2) #define SYS_VTTBR_EL2 sys_reg(3, 4, 2, 1, 0) #define SYS_VTCR_EL2 sys_reg(3, 4, 2, 1, 2) +#define SYS_VNCR_EL2 sys_reg(3, 4, 2, 2, 0) #define SYS_ZCR_EL2 sys_reg(3, 4, 1, 2, 0) #define SYS_TRFCR_EL2 sys_reg(3, 4, 1, 2, 1) diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c index ef4488db6dc1..5cadda79089a 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -45,7 +45,13 @@ static void __activate_traps(struct kvm_vcpu *vcpu) * the EL1 virtual memory control register accesses * as well as the AT S1 operations. */ - hcr |= HCR_TVM | HCR_TRVM | HCR_AT | HCR_TTLB | HCR_NV1; + if (enhanced_nested_virt_in_use(vcpu)) { + hcr &= ~HCR_TVM; + } else { + hcr |= HCR_TVM | HCR_TRVM | HCR_TTLB; + } + + hcr |= HCR_AT | HCR_NV1; } else { /* * For a guest hypervisor on v8.1 (VHE), allow to @@ -79,6 +85,12 @@ static void __activate_traps(struct kvm_vcpu *vcpu) if (!vcpu_el2_tge_is_set(vcpu)) hcr |= HCR_AT | HCR_TTLB; } + + if (enhanced_nested_virt_in_use(vcpu)) { + hcr |= HCR_AT | HCR_TTLB | HCR_NV2; + write_sysreg_s(vcpu->arch.ctxt.vncr_array, + SYS_VNCR_EL2); + } } else if (nested_virt_in_use(vcpu)) { u64 vhcr_el2 = __vcpu_sys_reg(vcpu, HCR_EL2); From patchwork Mon Nov 29 20:01:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645665 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9A1CAC433F5 for ; Mon, 29 Nov 2021 20:09:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231637AbhK2UMl (ORCPT ); Mon, 29 Nov 2021 15:12:41 -0500 Received: from ams.source.kernel.org ([145.40.68.75]:42994 "EHLO ams.source.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229935AbhK2UKl (ORCPT ); Mon, 29 Nov 2021 15:10:41 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id A5C22B815D4 for ; Mon, 29 Nov 2021 20:07:22 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 56279C53FD1; Mon, 29 Nov 2021 20:07:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216441; bh=KgGc5JJnyi+xoK5NH4X6bBiIwukmrEnb3nTjfxdMvYs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=P5/kxhwLYUpTNjFQPl07X15iWM7Puie4W0qXKlsI0nCQIwSm2vNPGG4JXlAPmMbqL leElLrzDLpKJj8clKWKzpBWKLGhjGWMouOK+t71bKIB0+J/Gsh4L0zGOV1MqRvqQiM LwW9Pb116GhM8Bpr4ql6tf3zVXsrYjzgrN627OJkt8uVSyZUKkdKGEQYTaVFUiKSWP NCRnEGHfx61c4ztDdCErsrtapGmKDg10bkTL95O3NTQkb08beSWiDGHmdCITxl2b9W VGzB+gaYA7Xfe54zOrcE0RAUYCAtlDAy+Ip9hqBhki6Fb1LkJNXfvjEFVnH8f+yBkx l/iPWDAWbCROw== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmrJ-008gvR-0I; Mon, 29 Nov 2021 20:02:41 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 68/69] KVM: arm64: nv: Fast-track 'InHost' exception returns Date: Mon, 29 Nov 2021 20:01:49 +0000 Message-Id: <20211129200150.351436-69-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org A significant part of the ARMv8.3-NV extension is to trap ERET instructions so that the hypervisor gets a chance to switch from a vEL2 L1 guest to an EL1 L2 guest. But this also has the unfortunate consequence of trapping ERET in unsuspecting circumstances, such as staying at vEL2 (interrupt handling while being in the guest hypervisor), or returning to host userspace in the case of a VHE guest. Although we already make some effort to handle these ERET quicker by not doing the put/load dance, it is still way too far down the line for it to be efficient enough. For these cases, it would ideal to ERET directly, no question asked. Of course, we can't do that. But the next best thing is to do it as early as possible, in fixup_guest_exit(), much as we would handle FPSIMD exceptions. Signed-off-by: Marc Zyngier --- arch/arm64/kvm/emulate-nested.c | 26 ++----------------- arch/arm64/kvm/hyp/vhe/switch.c | 46 +++++++++++++++++++++++++++++++++ 2 files changed, 48 insertions(+), 24 deletions(-) diff --git a/arch/arm64/kvm/emulate-nested.c b/arch/arm64/kvm/emulate-nested.c index 5fbf8978acf2..97e26a4539a9 100644 --- a/arch/arm64/kvm/emulate-nested.c +++ b/arch/arm64/kvm/emulate-nested.c @@ -52,8 +52,7 @@ bool forward_nv_traps(struct kvm_vcpu *vcpu) void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu) { - u64 spsr, elr, mode; - bool direct_eret; + u64 spsr, elr; /* * Forward this trap to the virtual EL2 if the virtual @@ -62,31 +61,10 @@ void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu) if (forward_nv_traps(vcpu)) return; - /* - * Going through the whole put/load motions is a waste of time - * if this is a VHE guest hypervisor returning to its own - * userspace, or the hypervisor performing a local exception - * return. No need to save/restore registers, no need to - * switch S2 MMU. Just do the canonical ERET. - */ - spsr = vcpu_read_sys_reg(vcpu, SPSR_EL2); - mode = spsr & (PSR_MODE_MASK | PSR_MODE32_BIT); - - direct_eret = (mode == PSR_MODE_EL0t && - vcpu_el2_e2h_is_set(vcpu) && - vcpu_el2_tge_is_set(vcpu)); - direct_eret |= (mode == PSR_MODE_EL2h || mode == PSR_MODE_EL2t); - - if (direct_eret) { - *vcpu_pc(vcpu) = vcpu_read_sys_reg(vcpu, ELR_EL2); - *vcpu_cpsr(vcpu) = spsr; - trace_kvm_nested_eret(vcpu, *vcpu_pc(vcpu), spsr); - return; - } - preempt_disable(); kvm_arch_vcpu_put(vcpu); + spsr = __vcpu_sys_reg(vcpu, SPSR_EL2); elr = __vcpu_sys_reg(vcpu, ELR_EL2); trace_kvm_nested_eret(vcpu, elr, spsr); diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c index 5cadda79089a..6cbe6a89dbdb 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -162,6 +162,51 @@ void deactivate_traps_vhe_put(struct kvm_vcpu *vcpu) __deactivate_traps_common(vcpu); } +static bool kvm_hyp_handle_eret(struct kvm_vcpu *vcpu, u64 *exit_code) +{ + struct kvm_cpu_context *ctxt = &vcpu->arch.ctxt; + u64 spsr, mode; + + /* + * Going through the whole put/load motions is a waste of time + * if this is a VHE guest hypervisor returning to its own + * userspace, or the hypervisor performing a local exception + * return. No need to save/restore registers, no need to + * switch S2 MMU. Just do the canonical ERET. + * + * Unless the trap has to be forwarded further down the line, + * of course... + */ + if (__vcpu_sys_reg(vcpu, HCR_EL2) & HCR_NV) + return false; + + spsr = read_sysreg_el1(SYS_SPSR); + spsr = __fixup_spsr_el2_read(ctxt, spsr); + mode = spsr & (PSR_MODE_MASK | PSR_MODE32_BIT); + + switch (mode) { + case PSR_MODE_EL0t: + if (!(vcpu_el2_e2h_is_set(vcpu) && vcpu_el2_tge_is_set(vcpu))) + return false; + break; + case PSR_MODE_EL2t: + mode = PSR_MODE_EL1t; + break; + case PSR_MODE_EL2h: + mode = PSR_MODE_EL1h; + break; + default: + return false; + } + + spsr = (spsr & ~(PSR_MODE_MASK | PSR_MODE32_BIT)) | mode; + + write_sysreg_el2(spsr, SYS_SPSR); + write_sysreg_el2(read_sysreg_el1(SYS_ELR), SYS_ELR); + + return true; +} + static const exit_handler_fn hyp_exit_handlers[] = { [0 ... ESR_ELx_EC_MAX] = NULL, [ESR_ELx_EC_CP15_32] = kvm_hyp_handle_cp15_32, @@ -171,6 +216,7 @@ static const exit_handler_fn hyp_exit_handlers[] = { [ESR_ELx_EC_IABT_LOW] = kvm_hyp_handle_iabt_low, [ESR_ELx_EC_DABT_LOW] = kvm_hyp_handle_dabt_low, [ESR_ELx_EC_PAC] = kvm_hyp_handle_ptrauth, + [ESR_ELx_EC_ERET] = kvm_hyp_handle_eret, }; static const exit_handler_fn *kvm_get_exit_handler_array(struct kvm_vcpu *vcpu) From patchwork Mon Nov 29 20:01:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12645771 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D1DC2C433FE for ; Mon, 29 Nov 2021 21:46:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232471AbhK2Vtr (ORCPT ); Mon, 29 Nov 2021 16:49:47 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49860 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232291AbhK2Vrq (ORCPT ); Mon, 29 Nov 2021 16:47:46 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C2C49C091D22 for ; Mon, 29 Nov 2021 12:06:53 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 8AD7DB815FE for ; Mon, 29 Nov 2021 20:06:52 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3BCC1C53FD1; Mon, 29 Nov 2021 20:06:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638216411; bh=revNVqtiu+mMGd8sw9pvXnutQlvPit8tuWd9vnCjHW8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=QCxa71LfezMgLgmBMXiE/8tW8ZyOqpuOFAImsYj7oQ8aSeIBg7Llz8SYtp1kJ3LGo 8NNmmMGliwSDDedPeZpy7xklUJkK6vOhXNH0C94SVrQf4YmxKAA6kQ7KHTKLDdZmOO +fJzWJqJVLWoVWJPd0Rm3mQDLV4Ipfdei1eY0pbytTIoxofriRyDpi0+To4bElFr2M pnoXW4qxd9UTlBp3y8vtSHMyY9LEf7ZDTuYo5ubF7QZJuQ7L0aCW2Z+cydNtB1/Lxl KgpGkAwWr7XW8nZJrtI04rAWMfgyriWgFi936ly09eCNdfsu8sqKi2E1gOJ1EwzUJL VUZFUh7hNekgg== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mrmrJ-008gvR-CC; Mon, 29 Nov 2021 20:02:41 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , Ganapatrao Kulkarni , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v5 69/69] KVM: arm64: nv: Fast-track EL1 TLBIs for VHE guests Date: Mon, 29 Nov 2021 20:01:50 +0000 Message-Id: <20211129200150.351436-70-maz@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211129200150.351436-1-maz@kernel.org> References: <20211129200150.351436-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, gankulkarni@os.amperecomputing.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Due to the way ARMv8.4-NV suppresses traps when accessing EL2 system registers, we can't track when the guest changes its HCR_EL2.TGE setting. This means we always trap EL1 TLBIs, even if they don't affect any guest. This obviously has a huge impact on performance, as we handle TLBI traps as a normal exit, and a normal VHE host issues thousands of TLBIs when booting (and quite a few when running userspace). A cheap way to reduce the overhead is to handle the limited case of {E2H,TGE}=={1,1} as a guest fixup, as we already have the right mmu configuration in place. Just execute the decoded instruction right away and return to the guest. Signed-off-by: Marc Zyngier --- arch/arm64/kvm/hyp/vhe/switch.c | 43 ++++++++++++++++++++++++++++++++- arch/arm64/kvm/hyp/vhe/tlb.c | 6 +++-- arch/arm64/kvm/sys_regs.c | 25 ++++++------------- 3 files changed, 54 insertions(+), 20 deletions(-) diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c index 6cbe6a89dbdb..fa2fd7e911b2 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -162,6 +162,47 @@ void deactivate_traps_vhe_put(struct kvm_vcpu *vcpu) __deactivate_traps_common(vcpu); } +static bool kvm_hyp_handle_tlbi_el1(struct kvm_vcpu *vcpu, u64 *exit_code) +{ + u32 instr; + u64 val; + + /* + * Ideally, we would never trap on EL1 TLB invalidations when the + * guest's HCR_EL2.{E2H,TGE} == {1,1}. But "thanks" to ARMv8.4, we + * don't trap writes to HCR_EL2, meaning that we can't track + * changes to the virtual TGE bit. So we leave HCR_EL2.TTLB set on + * the host. Oopsie... + * + * In order to speed-up EL1 TLBIs from the vEL2 guest when TGE is + * set, try and handle these invalidation as quickly as possible, + * without fully exiting (unless this needs forwarding). + */ + if (!enhanced_nested_virt_in_use(vcpu) || + !vcpu_mode_el2(vcpu) || + (__vcpu_sys_reg(vcpu, HCR_EL2) & (HCR_E2H | HCR_TGE)) != (HCR_E2H | HCR_TGE)) + return false; + + instr = esr_sys64_to_sysreg(kvm_vcpu_get_esr(vcpu)); + if (sys_reg_Op0(instr) != TLBI_Op0 || + sys_reg_Op1(instr) != TLBI_Op1_EL1) + return false; + + val = vcpu_get_reg(vcpu, kvm_vcpu_sys_get_rt(vcpu)); + __kvm_tlb_el1_instr(NULL, val, instr); + __kvm_skip_instr(vcpu); + + return true; +} + +static bool kvm_hyp_handle_sysreg_vhe(struct kvm_vcpu *vcpu, u64 *exit_code) +{ + if (kvm_hyp_handle_tlbi_el1(vcpu, exit_code)) + return true; + + return kvm_hyp_handle_sysreg(vcpu, exit_code); +} + static bool kvm_hyp_handle_eret(struct kvm_vcpu *vcpu, u64 *exit_code) { struct kvm_cpu_context *ctxt = &vcpu->arch.ctxt; @@ -210,7 +251,7 @@ static bool kvm_hyp_handle_eret(struct kvm_vcpu *vcpu, u64 *exit_code) static const exit_handler_fn hyp_exit_handlers[] = { [0 ... ESR_ELx_EC_MAX] = NULL, [ESR_ELx_EC_CP15_32] = kvm_hyp_handle_cp15_32, - [ESR_ELx_EC_SYS64] = kvm_hyp_handle_sysreg, + [ESR_ELx_EC_SYS64] = kvm_hyp_handle_sysreg_vhe, [ESR_ELx_EC_SVE] = kvm_hyp_handle_fpsimd, [ESR_ELx_EC_FP_ASIMD] = kvm_hyp_handle_fpsimd, [ESR_ELx_EC_IABT_LOW] = kvm_hyp_handle_iabt_low, diff --git a/arch/arm64/kvm/hyp/vhe/tlb.c b/arch/arm64/kvm/hyp/vhe/tlb.c index c4389db4cc22..beb162468c0b 100644 --- a/arch/arm64/kvm/hyp/vhe/tlb.c +++ b/arch/arm64/kvm/hyp/vhe/tlb.c @@ -201,7 +201,8 @@ void __kvm_tlb_el1_instr(struct kvm_s2_mmu *mmu, u64 val, u64 sys_encoding) dsb(ishst); /* Switch to requested VMID */ - __tlb_switch_to_guest(mmu, &cxt); + if (mmu) + __tlb_switch_to_guest(mmu, &cxt); /* * Execute the same instruction as the guest hypervisor did, @@ -240,5 +241,6 @@ void __kvm_tlb_el1_instr(struct kvm_s2_mmu *mmu, u64 val, u64 sys_encoding) dsb(ish); isb(); - __tlb_switch_to_host(&cxt); + if (mmu) + __tlb_switch_to_host(&cxt); } diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index e8ab052be122..acfd3c72faf6 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -2804,6 +2804,8 @@ static bool handle_tlbi_el1(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) { u32 sys_encoding = sys_insn(p->Op0, p->Op1, p->CRn, p->CRm, p->Op2); + u64 virtual_vttbr = vcpu_read_sys_reg(vcpu, VTTBR_EL2); + struct kvm_s2_mmu *mmu; /* * If we're here, this is because we've trapped on a EL1 TLBI @@ -2822,24 +2824,13 @@ static bool handle_tlbi_el1(struct kvm_vcpu *vcpu, struct sys_reg_params *p, mutex_lock(&vcpu->kvm->lock); - if ((__vcpu_sys_reg(vcpu, HCR_EL2) & (HCR_E2H | HCR_TGE)) != (HCR_E2H | HCR_TGE)) { - u64 virtual_vttbr = vcpu_read_sys_reg(vcpu, VTTBR_EL2); - struct kvm_s2_mmu *mmu; - - mmu = lookup_s2_mmu(vcpu->kvm, virtual_vttbr, HCR_VM); - if (mmu) - __kvm_tlb_el1_instr(mmu, p->regval, sys_encoding); + mmu = lookup_s2_mmu(vcpu->kvm, virtual_vttbr, HCR_VM); + if (mmu) + __kvm_tlb_el1_instr(mmu, p->regval, sys_encoding); - mmu = lookup_s2_mmu(vcpu->kvm, virtual_vttbr, 0); - if (mmu) - __kvm_tlb_el1_instr(mmu, p->regval, sys_encoding); - } else { - /* - * ARMv8.4-NV allows the guest to change TGE behind - * our back, so we always trap EL1 TLBIs from vEL2... - */ - __kvm_tlb_el1_instr(&vcpu->kvm->arch.mmu, p->regval, sys_encoding); - } + mmu = lookup_s2_mmu(vcpu->kvm, virtual_vttbr, 0); + if (mmu) + __kvm_tlb_el1_instr(mmu, p->regval, sys_encoding); mutex_unlock(&vcpu->kvm->lock);