From patchwork Mon May 10 16:59:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12249187 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 06A23C433ED for ; Mon, 10 May 2021 17:58:01 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 77079611BD for ; Mon, 10 May 2021 17:58:00 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 77079611BD Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=zQIhlWMHsvjmYz42cEIhrzO7X/WKayPqcCFaGiKoThQ=; b=GkWzRqwKpq1w/N8L0TZn7bik4 znNEmSc1AhXfkep3rDRVZ+G2GL/2EsGhgcNVilR5XiNGfZFABEFRS4HfnBO3K3kpWBpZOZvuIPffj l2MZ2sw+Adw9I4vZKENEigf8DpLyvG4xo/ncrnDCmJCGKDHy7LXh90iM+/vmbWtb6zQfl52hElSti AsqCM90Hl/LBbQkZ1YdUFekTRImjnW6yBmXAHqAdhwUbXNSIuyG3V5NUj4+UcA6WGQsgvWFx1I/MH /t5am7gHgMYfVXeBIWwqAEdwzWWjOvGp/cEcNY+86DQLgr/buU9vjcgDVHn8oJRcxyXg0erMLbvYc RAFmHgWvw==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lgA8b-00FJ1s-AZ; Mon, 10 May 2021 17:56:13 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lg9iW-00F9Ry-Fo for linux-arm-kernel@desiato.infradead.org; Mon, 10 May 2021 17:29:17 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=8sy+Gu6f9RAF0WUyR+BhhbAAAxB2yWa/0qGWAiZdzbQ=; b=FbtTK/AHclAQ6GosQpSj1TyBwf OUfmAS6H6YFRNi0cRhgdDVun6FrPvfg+3Igpny2qk4cvXRTejXnt6dDGrQLAjduwS4opuneCz40rn 2kAkEFxmZXRv3MIT9yO5QpMuyhjoqZ8GHGF/jPLiXO1d1AkyE0S90BPFQbavG4z5Kq8bELoZsCSEv jmUH8kBxQyDqVmuBT0sfV0Fod4SuBBSe3VYj4X5iGHPtImHpawP36Zr3lL1rVIQLx2n5KDRt4qPVG iVAQGwQEI7dYinvjS6opR93m/Fajhg6UAhwre+MaTlcFcMiXUaOKdi2QCoh2UQpAo+inwkmSayWuD KDPtSsfg==; Received: from mail.kernel.org ([198.145.29.99]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lg9iT-008yp9-OP for linux-arm-kernel@lists.infradead.org; Mon, 10 May 2021 17:29:15 +0000 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 7BBA46147F; Mon, 10 May 2021 17:29:13 +0000 (UTC) Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1lg9Gs-000Uqg-59; Mon, 10 May 2021 18:00:42 +0100 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v4 65/66] KVM: arm64: nv: Fast-track 'InHost' exception returns Date: Mon, 10 May 2021 17:59:19 +0100 Message-Id: <20210510165920.1913477-66-maz@kernel.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210510165920.1913477-1-maz@kernel.org> References: <20210510165920.1913477-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210510_102913_858529_51868F8C X-CRM114-Status: GOOD ( 23.16 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org A significant part of the ARMv8.3-NV extension is to trap ERET instructions so that the hypervisor gets a chance to switch from a vEL2 L1 guest to an EL1 L2 guest. But this also has the unfortunate consequence of trapping ERET in unsuspecting circumstances, such as staying at vEL2 (interrupt handling while being in the guest hypervisor), or returning to host userspace in the case of a VHE guest. Although we already make some effort to handle these ERET quicker by not doing the put/load dance, it is still way too far down the line for it to be efficient enough. For these cases, it would ideal to ERET directly, no question asked. Of course, we can't do that. But the next best thing is to do it as early as possible, in fixup_guest_exit(), much as we would handle FPSIMD exceptions. Signed-off-by: Marc Zyngier --- arch/arm64/kvm/emulate-nested.c | 26 ++-------------- arch/arm64/kvm/hyp/vhe/switch.c | 53 ++++++++++++++++++++++++++++++++- 2 files changed, 54 insertions(+), 25 deletions(-) diff --git a/arch/arm64/kvm/emulate-nested.c b/arch/arm64/kvm/emulate-nested.c index df4661515183..43197ff4a806 100644 --- a/arch/arm64/kvm/emulate-nested.c +++ b/arch/arm64/kvm/emulate-nested.c @@ -52,8 +52,7 @@ bool forward_nv_traps(struct kvm_vcpu *vcpu) void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu) { - u64 spsr, elr, mode; - bool direct_eret; + u64 spsr, elr; /* * Forward this trap to the virtual EL2 if the virtual @@ -62,31 +61,10 @@ void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu) if (forward_nv_traps(vcpu)) return; - /* - * Going through the whole put/load motions is a waste of time - * if this is a VHE guest hypervisor returning to its own - * userspace, or the hypervisor performing a local exception - * return. No need to save/restore registers, no need to - * switch S2 MMU. Just do the canonical ERET. - */ - spsr = vcpu_read_sys_reg(vcpu, SPSR_EL2); - mode = spsr & (PSR_MODE_MASK | PSR_MODE32_BIT); - - direct_eret = (mode == PSR_MODE_EL0t && - vcpu_el2_e2h_is_set(vcpu) && - vcpu_el2_tge_is_set(vcpu)); - direct_eret |= (mode == PSR_MODE_EL2h || mode == PSR_MODE_EL2t); - - if (direct_eret) { - *vcpu_pc(vcpu) = vcpu_read_sys_reg(vcpu, ELR_EL2); - *vcpu_cpsr(vcpu) = spsr; - trace_kvm_nested_eret(vcpu, *vcpu_pc(vcpu), spsr); - return; - } - preempt_disable(); kvm_arch_vcpu_put(vcpu); + spsr = __vcpu_sys_reg(vcpu, SPSR_EL2); elr = __vcpu_sys_reg(vcpu, ELR_EL2); trace_kvm_nested_eret(vcpu, elr, spsr); diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c index bd4ae1296de4..b3e7d996b54d 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -166,6 +166,56 @@ void deactivate_traps_vhe_put(void) __deactivate_traps_common(); } +static bool __hyp_handle_eret(struct kvm_vcpu *vcpu) +{ + struct kvm_cpu_context *ctxt = &vcpu->arch.ctxt; + u64 spsr, mode; + + /* + * Going through the whole put/load motions is a waste of time + * if this is a VHE guest hypervisor returning to its own + * userspace, or the hypervisor performing a local exception + * return. No need to save/restore registers, no need to + * switch S2 MMU. Just do the canonical ERET. Unless the trap + * has to be forwarded further down the line, of course... + */ + if (kvm_vcpu_trap_get_class(vcpu) != ESR_ELx_EC_ERET) + return false; + + /* + * Let the trap forwarding be handled by the normal exception + * handling code. + */ + if (__vcpu_sys_reg(vcpu, HCR_EL2) & HCR_NV) + return false; + + spsr = read_sysreg_el1(SYS_SPSR); + spsr = __fixup_spsr_el2_read(ctxt, spsr); + mode = spsr & (PSR_MODE_MASK | PSR_MODE32_BIT); + + switch (mode) { + case PSR_MODE_EL0t: + if (!(vcpu_el2_e2h_is_set(vcpu) && vcpu_el2_tge_is_set(vcpu))) + return false; + break; + case PSR_MODE_EL2t: + mode = PSR_MODE_EL1t; + break; + case PSR_MODE_EL2h: + mode = PSR_MODE_EL1h; + break; + default: + return false; + } + + spsr = (spsr & ~(PSR_MODE_MASK | PSR_MODE32_BIT)) | mode; + + write_sysreg_el2(spsr, SYS_SPSR); + write_sysreg_el2(read_sysreg_el1(SYS_ELR), SYS_ELR); + + return true; +} + static bool fixup_guest_exit_vhe(struct kvm_vcpu *vcpu, u64 *exit_code, bool hyp_ctxt) { @@ -207,7 +257,8 @@ static bool fixup_guest_exit_vhe(struct kvm_vcpu *vcpu, u64 *exit_code, fixup_guest_exit_prologue(vcpu, exit_code); if (*exit_code == ARM_EXCEPTION_TRAP) { - /* more to come here */ + if (__hyp_handle_eret(vcpu)) + return true; } return fixup_guest_exit(vcpu, exit_code);