From patchwork Mon May 10 16:59:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 12249141 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1F558C433ED for ; Mon, 10 May 2021 17:48:35 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 81C016112F for ; Mon, 10 May 2021 17:48:34 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 81C016112F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=FTjrtrYUz66wNwlU32hmA20+2YWYtvvowIuMi5qZyO4=; b=gSd80rrsGYuT2kVYg46At0+B2 mbGK//7Y3SE6s5selhP4HmJ2A+eSmqfrx5yIPE7zxosNp/9qXxxajkqT7rCr8jm6liHAbid/Inoie UZVOgleBswqAcXyFLCQSbVIAD/9Lo8R/FKvHOi9btplHFIzrBwF2bxfg8WEhgnB9MJLoxQeYnzZiO RVuvwxrhWwAOpul0XUmrfEdM1LHqxrr0gmx7Xw3LRgeojSZTRYHw0cmyQZTmqPZemdfagbJDH7gP7 7VlsNbp63BviXVFPDwaBIPxamyjruDt60pU16tk1p9pxN/G5LZF9J26HRIORS6nZjal37C6AA0W8/ /9Qs+2mwQ==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lg9ze-00FFsr-5f; Mon, 10 May 2021 17:46:58 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lg9hw-00F9BB-7U for linux-arm-kernel@desiato.infradead.org; Mon, 10 May 2021 17:28:40 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=pL1CAI2OxIMSGeB5GuiHe+RbhWpAij3Llsg9PFJnLYM=; b=28YW8p0lz4lpmJKax6coolC98U gENjshQQqR5zjynnN8gx/x5r3bFaBoqVQFNxiNBD+U2C185JbWGrpRmJ94jo7NGtVI0yp9urmUf2s jB09OKin8CNFjbyBr9nPpuGZ/pUgs+8/mXt6m2dhkf3n2f0hUPWI3GVmQ9N955TVy1rGNbBRVzhHA aJKm5K+6qzs8jS2EkuDowsBGgCMoHGBdAAndv6ex4/G+FY0Dybn3lZ3Mekf6GxuiBxeKZaJyrNZfa mqWxJZAMXxxnHzCUTPTtx5loCPkdik0iCMXFNZHlR0hxaCqLgF0pTIC4LZYYk39TapzV56geQezor XmBAKajQ==; Received: from mail.kernel.org ([198.145.29.99]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lg9ht-008yiP-9N for linux-arm-kernel@lists.infradead.org; Mon, 10 May 2021 17:28:39 +0000 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id D839761627; Mon, 10 May 2021 17:28:36 +0000 (UTC) Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1lg9Gp-000Uqg-C8; Mon, 10 May 2021 18:00:39 +0100 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Jintack Lim , Haibo Xu , James Morse , Suzuki K Poulose , Alexandru Elisei , kernel-team@android.com Subject: [PATCH v4 61/66] KVM: arm64: nv: Synchronize PSTATE early on exit Date: Mon, 10 May 2021 17:59:15 +0100 Message-Id: <20210510165920.1913477-62-maz@kernel.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210510165920.1913477-1-maz@kernel.org> References: <20210510165920.1913477-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, jintack@cs.columbia.edu, haibo.xu@linaro.org, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210510_102837_398641_DDCE43C9 X-CRM114-Status: GOOD ( 24.57 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The NV code relies on predicates such as is_hyp_ctxt() being reliable. In turn, is_hyp_ctxt() relies on things like PSTATE and the virtual HCR_EL2 being accurate. But with ARMv8.4-NV removing trapping for a large part of the EL2 system registers (among which HCR_EL2), we can't use such trapping to synchronize the rest of the state. Let's look at the following sequence for a VHE guest: (1) enter guest in host EL0 (2) guest traps to guest vEL2 (no hypervisor intervention) (3) guest clears virtual HCR_EL2.TGE (no trap either) (4) host interrupt fires, exit (5) is_hyp_ctxt() now says "guest" (PSTATE.M==EL1 and TGE==0) It is obvious that such behaviour would be rather unfortunate, and lead to interesting, difficult to catch bugs specially if preemption kicks in (yes, I wasted a whole week chasing this one). In order to preserve the invariant that a guest entered in host context must exit in the same context, we must make sure that is_hyp_ctxt() works correctly. Since we can always observe the guest value of HCR_EL2.{E2H,TGE} in the VNCR_EL2 page, we solely need to synchronize PSTATE as early as possible. This basically amounts to moving from_hw_pstate() as close as possible to the guest exit point, and fixup_guest_exit() seems as good a place as any. Signed-off-by: Marc Zyngier --- arch/arm64/kvm/hyp/include/hyp/switch.h | 16 ++++-- arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h | 26 ++-------- arch/arm64/kvm/hyp/nvhe/switch.c | 8 ++- arch/arm64/kvm/hyp/vhe/switch.c | 57 +++++++++++++++++++++- 4 files changed, 78 insertions(+), 29 deletions(-) diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h index 0790eb2b7545..6011f32fdb32 100644 --- a/arch/arm64/kvm/hyp/include/hyp/switch.h +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h @@ -408,11 +408,11 @@ static inline bool __hyp_handle_ptrauth(struct kvm_vcpu *vcpu) } /* - * Return true when we were able to fixup the guest exit and should return to - * the guest, false when we should restore the host state and return to the - * main run loop. + * Prologue for the guest fixup, populating ESR_EL2 and fixing up PC + * if required. */ -static inline bool fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code) +static inline void fixup_guest_exit_prologue(struct kvm_vcpu *vcpu, + u64 *exit_code) { if (ARM_EXCEPTION_CODE(*exit_code) != ARM_EXCEPTION_IRQ) vcpu->arch.fault.esr_el2 = read_sysreg_el2(SYS_ESR); @@ -431,7 +431,15 @@ static inline bool fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code) if (esr_ec == ESR_ELx_EC_HVC32 || esr_ec == ESR_ELx_EC_HVC64) write_sysreg_el2(read_sysreg_el2(SYS_ELR) - 4, SYS_ELR); } +} +/* + * Return true when we were able to fixup the guest exit and should return to + * the guest, false when we should restore the host state and return to the + * main run loop. + */ +static inline bool fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code) +{ /* * We're using the raw exception code in order to only process * the trap if no SError is pending. We will come back to the diff --git a/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h b/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h index 92715fa01e88..1931c8667d52 100644 --- a/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h +++ b/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h @@ -51,32 +51,12 @@ static inline void __sysreg_save_el1_state(struct kvm_cpu_context *ctxt) ctxt_sys_reg(ctxt, SPSR_EL1) = read_sysreg_el1(SYS_SPSR); } -static inline u64 from_hw_pstate(const struct kvm_cpu_context *ctxt) -{ - u64 reg = read_sysreg_el2(SYS_SPSR); - - if (__is_hyp_ctxt(ctxt)) { - u64 mode = reg & (PSR_MODE_MASK | PSR_MODE32_BIT); - - switch (mode) { - case PSR_MODE_EL1t: - mode = PSR_MODE_EL2t; - break; - case PSR_MODE_EL1h: - mode = PSR_MODE_EL2h; - break; - } - - return (reg & ~(PSR_MODE_MASK | PSR_MODE32_BIT)) | mode; - } - - return reg; -} - static inline void __sysreg_save_el2_return_state(struct kvm_cpu_context *ctxt) { + /* On VHE, PSTATE is saved in fixup_guest_exit_vhe() */ + if (!has_vhe()) + ctxt->regs.pstate = read_sysreg_el2(SYS_SPSR); ctxt->regs.pc = read_sysreg_el2(SYS_ELR); - ctxt->regs.pstate = from_hw_pstate(ctxt); if (cpus_have_final_cap(ARM64_HAS_RAS_EXTN)) ctxt_sys_reg(ctxt, DISR_EL1) = read_sysreg_s(SYS_VDISR_EL2); diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/switch.c index 2b0f8675fe3b..b9b9c8e0a9f2 100644 --- a/arch/arm64/kvm/hyp/nvhe/switch.c +++ b/arch/arm64/kvm/hyp/nvhe/switch.c @@ -166,6 +166,12 @@ static void __pmu_switch_to_host(struct kvm_cpu_context *host_ctxt) write_sysreg(pmu->events_host, pmcntenset_el0); } +static bool fixup_guest_exit_nvhe(struct kvm_vcpu *vcpu, u64 *exit_code) +{ + fixup_guest_exit_prologue(vcpu, exit_code); + return fixup_guest_exit(vcpu, exit_code); +} + /* Switch to the guest for legacy non-VHE systems */ int __kvm_vcpu_run(struct kvm_vcpu *vcpu) { @@ -227,7 +233,7 @@ int __kvm_vcpu_run(struct kvm_vcpu *vcpu) exit_code = __guest_enter(vcpu); /* And we're baaack! */ - } while (fixup_guest_exit(vcpu, &exit_code)); + } while (fixup_guest_exit_nvhe(vcpu, &exit_code)); __sysreg_save_state_nvhe(guest_ctxt); __sysreg32_save_state(vcpu); diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c index 2725dc62ab09..b665a3cc288e 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -154,12 +154,60 @@ void deactivate_traps_vhe_put(void) __deactivate_traps_common(); } +static bool fixup_guest_exit_vhe(struct kvm_vcpu *vcpu, u64 *exit_code, + bool hyp_ctxt) +{ + u64 pstate = read_sysreg_el2(SYS_SPSR); + + /* + * Sync pstate back as early as possible, so that is_hyp_ctxt() + * reflects the exact context. It is otherwise possible to get + * confused with a VHE guest and ARMv8.4-NV, such as: + * + * (1) enter guest in host EL0 + * (2) guest traps to guest vEL2 (no hypervisor intervention) + * (3) guest clears virtual HCR_EL2.TGE (no trap either) + * (4) host interrupt fires, exit + * (5) is_hyp_ctxt() now says "guest" (pstate.M==EL1 and TGE==0) + * + * If host preemption occurs, vcpu_load/put() will be very confused. + * + * Consider this as the prologue before the fixup prologue... + */ + + if (unlikely(hyp_ctxt)) { + u64 mode = pstate & PSR_MODE_MASK; + + switch (mode) { + case PSR_MODE_EL1t: + mode = PSR_MODE_EL2t; + break; + case PSR_MODE_EL1h: + mode = PSR_MODE_EL2h; + break; + } + + pstate = (pstate & ~PSR_MODE_MASK) | mode; + } + + *vcpu_cpsr(vcpu) = pstate; + + fixup_guest_exit_prologue(vcpu, exit_code); + + if (*exit_code == ARM_EXCEPTION_TRAP) { + /* more to come here */ + } + + return fixup_guest_exit(vcpu, exit_code); +} + /* Switch to the guest for VHE systems running in EL2 */ static int __kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu) { struct kvm_cpu_context *host_ctxt; struct kvm_cpu_context *guest_ctxt; u64 exit_code; + bool hyp_ctxt; host_ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt; host_ctxt->__hyp_running_vcpu = vcpu; @@ -186,12 +234,19 @@ static int __kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu) sysreg_restore_guest_state_vhe(guest_ctxt); __debug_switch_to_guest(vcpu); + /* + * Being in HYP context or not is an invariant here. If we enter in + * a given context, we exit in the same context. We can thus only + * sample the context once. + */ + WRITE_ONCE(hyp_ctxt, is_hyp_ctxt(vcpu)); + do { /* Jump in the fire! */ exit_code = __guest_enter(vcpu); /* And we're baaack! */ - } while (fixup_guest_exit(vcpu, &exit_code)); + } while (fixup_guest_exit_vhe(vcpu, &exit_code, READ_ONCE(hyp_ctxt))); sysreg_save_guest_state_vhe(guest_ctxt);