From patchwork Fri Sep 28 13:39:17 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dave Martin X-Patchwork-Id: 10619971 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 23EA415A6 for ; Fri, 28 Sep 2018 13:48:10 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 131902ACB5 for ; Fri, 28 Sep 2018 13:48:10 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 04F8F2A19B; Fri, 28 Sep 2018 13:48:10 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 3C48B2A19B for ; Fri, 28 Sep 2018 13:48:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=Uc35050D16ILApWjjdtAPvfM45QSCgjjK/gejSFp75o=; b=OWFePxQX+fylnr3nFvX+Mjwv/1 2eurw70Sn4KeNIdzOZDrKftsO/7RCdv6vaneeKRulYSvMSYjezEVxqz/3VaL9dSM013Fk5QgWRv16 4lxi64idBfqn2xAHWoPt0Mk9I2J0MdgurPK8K7ppA1KI/y+kngzkwRDbEw3iW17ta9NgScPDvyjx3 d34cK2EMYciYnoa44yh2kSz778YxjHK3VPgnDqr2mXAoG2TO2qnXR0SExfpW2zW94d8dDIjhslVu1 xyWvZlSlW5sLZnYMKvfmJ2/P5w2pL0k3ZUs82tOMkeavAPVsntJBC/R5K7XD18mYfQ/EYRIwovFrd M8Ad+hng==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1g5t7j-0003j6-L8; Fri, 28 Sep 2018 13:48:03 +0000 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70] helo=foss.arm.com) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1g5t0r-0007vh-D4 for linux-arm-kernel@lists.infradead.org; Fri, 28 Sep 2018 13:42:57 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id EDBFE1D31; Fri, 28 Sep 2018 06:40:40 -0700 (PDT) Received: from e103592.Emea.Arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 419423F5B7; Fri, 28 Sep 2018 06:40:39 -0700 (PDT) From: Dave Martin To: kvmarm@lists.cs.columbia.edu Subject: [RFC PATCH v2 13/23] KVM: arm64/sve: Context switch the SVE registers Date: Fri, 28 Sep 2018 14:39:17 +0100 Message-Id: <1538141967-15375-14-git-send-email-Dave.Martin@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1538141967-15375-1-git-send-email-Dave.Martin@arm.com> References: <1538141967-15375-1-git-send-email-Dave.Martin@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20180928_064057_524229_F9DF0DAB X-CRM114-Status: GOOD ( 23.29 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Okamoto Takayuki , Christoffer Dall , Ard Biesheuvel , Marc Zyngier , Catalin Marinas , Will Deacon , =?utf-8?q?Alex_Benn=C3=A9e?= , linux-arm-kernel@lists.infradead.org MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP In order to give each vcpu its own view of the SVE registers, this patch adds context storage via a new sve_state pointer in struct vcpu_arch. An additional member sve_max_vl is also added for each vcpu, to determine the maximum vector length visible to the guest and thus the value to be configured in ZCR_EL2.LEN while the is active. This also determines the layout and size of the storage in sve_state, which is read and written by the same backend functions that are used for context-switching the SVE state for host tasks. On SVE-enabled vcpus, SVE access traps are now handled by switching in the vcpu's SVE context and disabling the trap before returning to the guest. On other vcpus, the trap is not handled and an exit back to the host occurs, where the handle_sve() fallback path reflects an undefined instruction exception back to the guest, consistently with the behaviour of non-SVE-capable hardware (as was done unconditionally prior to this patch). No SVE handling is added on non-VHE-only paths, since VHE is an architectural and Kconfig prerequisite of SVE. Signed-off-by: Dave Martin Reviewed-by: Alex Bennée --- Changes since RFCv1: * Add a if_sve () helper macro to efficiently skip or optimise out SVE conditional support code for the SVE-unsupported case. This reduces the verbose boilerplate at the affected sites. * In the style of sve_pffr(), a vcpu_sve_pffr() helper is added to provide the FFR anchor pointer for sve_load_state() in the hyp switch code. This help avoid some open-coded pointer mungeing which is not very readable. * The condition for calling __hyp_switch_fpsimd() is abstracted for better readability. --- arch/arm64/include/asm/kvm_host.h | 6 ++++ arch/arm64/kvm/fpsimd.c | 5 +-- arch/arm64/kvm/hyp/switch.c | 71 ++++++++++++++++++++++++++++++--------- 3 files changed, 65 insertions(+), 17 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 76cbb95e..8e9cd43 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -210,6 +210,8 @@ typedef struct kvm_cpu_context kvm_cpu_context_t; struct kvm_vcpu_arch { struct kvm_cpu_context ctxt; + void *sve_state; + unsigned int sve_max_vl; /* HYP configuration */ u64 hcr_el2; @@ -302,6 +304,10 @@ struct kvm_vcpu_arch { bool sysregs_loaded_on_cpu; }; +/* Pointer to the vcpu's SVE FFR for sve_{save,load}_state() */ +#define vcpu_sve_pffr(vcpu) ((void *)((char *)((vcpu)->arch.sve_state) + \ + sve_ffr_offset((vcpu)->arch.sve_max_vl))) + /* vcpu_arch flags field values: */ #define KVM_ARM64_DEBUG_DIRTY (1 << 0) #define KVM_ARM64_FP_ENABLED (1 << 1) /* guest FP regs loaded */ diff --git a/arch/arm64/kvm/fpsimd.c b/arch/arm64/kvm/fpsimd.c index 29e5585..3474388 100644 --- a/arch/arm64/kvm/fpsimd.c +++ b/arch/arm64/kvm/fpsimd.c @@ -86,10 +86,11 @@ void kvm_arch_vcpu_ctxsync_fp(struct kvm_vcpu *vcpu) if (vcpu->arch.flags & KVM_ARM64_FP_ENABLED) { fpsimd_bind_state_to_cpu(&vcpu->arch.ctxt.gp_regs.fp_regs, - NULL, sve_max_vl); + vcpu->arch.sve_state, + vcpu->arch.sve_max_vl); clear_thread_flag(TIF_FOREIGN_FPSTATE); - clear_thread_flag(TIF_SVE); + update_thread_flag(TIF_SVE, vcpu_has_sve(vcpu)); } } diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c index 085ed06..9941349 100644 --- a/arch/arm64/kvm/hyp/switch.c +++ b/arch/arm64/kvm/hyp/switch.c @@ -98,7 +98,10 @@ static void activate_traps_vhe(struct kvm_vcpu *vcpu) val = read_sysreg(cpacr_el1); val |= CPACR_EL1_TTA; val &= ~CPACR_EL1_ZEN; - if (!update_fp_enabled(vcpu)) { + if (update_fp_enabled(vcpu)) { + if (vcpu_has_sve(vcpu)) + val |= CPACR_EL1_ZEN; + } else { val &= ~CPACR_EL1_FPEN; __activate_traps_fpsimd32(vcpu); } @@ -332,16 +335,29 @@ static bool __hyp_text __skip_instr(struct kvm_vcpu *vcpu) } } -static bool __hyp_text __hyp_switch_fpsimd(struct kvm_vcpu *vcpu) +/* + * if () with a gating check for SVE support to minimise branch + * mispredictions in non-SVE systems. + * (system_supports_sve() is resolved at build time or via a static key.) + */ +#define if_sve(cond) if (system_supports_sve() && (cond)) + +static bool __hyp_text __hyp_switch_fpsimd(struct kvm_vcpu *vcpu, + bool guest_has_sve) { struct user_fpsimd_state *host_fpsimd = vcpu->arch.host_fpsimd_state; - if (has_vhe()) - write_sysreg(read_sysreg(cpacr_el1) | CPACR_EL1_FPEN, - cpacr_el1); - else + if (has_vhe()) { + u64 reg = read_sysreg(cpacr_el1) | CPACR_EL1_FPEN; + + if_sve (guest_has_sve) + reg |= CPACR_EL1_ZEN; + + write_sysreg(reg, cpacr_el1); + } else { write_sysreg(read_sysreg(cptr_el2) & ~(u64)CPTR_EL2_TFP, cptr_el2); + } isb(); @@ -350,8 +366,7 @@ static bool __hyp_text __hyp_switch_fpsimd(struct kvm_vcpu *vcpu) * In the SVE case, VHE is assumed: it is enforced by * Kconfig and kvm_arch_init(). */ - if (system_supports_sve() && - (vcpu->arch.flags & KVM_ARM64_HOST_SVE_IN_USE)) { + if_sve (vcpu->arch.flags & KVM_ARM64_HOST_SVE_IN_USE) { struct thread_struct *thread = container_of( host_fpsimd, struct thread_struct, uw.fpsimd_state); @@ -364,11 +379,14 @@ static bool __hyp_text __hyp_switch_fpsimd(struct kvm_vcpu *vcpu) vcpu->arch.flags &= ~KVM_ARM64_FP_HOST; } - __fpsimd_restore_state(&vcpu->arch.ctxt.gp_regs.fp_regs); - - if (system_supports_sve() && - vcpu->arch.flags & KVM_ARM64_GUEST_HAS_SVE) + if_sve (guest_has_sve) { + sve_load_state(vcpu_sve_pffr(vcpu), + &vcpu->arch.ctxt.gp_regs.fp_regs.fpsr, + sve_vq_from_vl(vcpu->arch.sve_max_vl) - 1); write_sysreg_s(vcpu->arch.ctxt.sys_regs[ZCR_EL1], SYS_ZCR_EL12); + } else { + __fpsimd_restore_state(&vcpu->arch.ctxt.gp_regs.fp_regs); + } /* Skip restoring fpexc32 for AArch64 guests */ if (!(read_sysreg(hcr_el2) & HCR_RW)) @@ -380,6 +398,26 @@ static bool __hyp_text __hyp_switch_fpsimd(struct kvm_vcpu *vcpu) return true; } +static inline bool __hyp_text __hyp_trap_is_fpsimd(struct kvm_vcpu *vcpu, + bool guest_has_sve) +{ + + u8 trap_class; + + if (!system_supports_fpsimd()) + return false; + + trap_class = kvm_vcpu_trap_get_class(vcpu); + + if (trap_class == ESR_ELx_EC_FP_ASIMD) + return true; + + if_sve (guest_has_sve && trap_class == ESR_ELx_EC_SVE) + return true; + + return false; +} + /* * Return true when we were able to fixup the guest exit and should return to * the guest, false when we should restore the host state and return to the @@ -387,6 +425,8 @@ static bool __hyp_text __hyp_switch_fpsimd(struct kvm_vcpu *vcpu) */ static bool __hyp_text fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code) { + bool guest_has_sve; + if (ARM_EXCEPTION_CODE(*exit_code) != ARM_EXCEPTION_IRQ) vcpu->arch.fault.esr_el2 = read_sysreg_el2(esr); @@ -404,10 +444,11 @@ static bool __hyp_text fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code) * and restore the guest context lazily. * If FP/SIMD is not implemented, handle the trap and inject an * undefined instruction exception to the guest. + * Similarly for trapped SVE accesses. */ - if (system_supports_fpsimd() && - kvm_vcpu_trap_get_class(vcpu) == ESR_ELx_EC_FP_ASIMD) - return __hyp_switch_fpsimd(vcpu); + guest_has_sve = vcpu_has_sve(vcpu); + if (__hyp_trap_is_fpsimd(vcpu, guest_has_sve)) + return __hyp_switch_fpsimd(vcpu, guest_has_sve); if (!__populate_fault_info(vcpu)) return true;