From patchwork Fri Mar 21 00:16:01 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14024772 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 68675C28B30 for ; Fri, 21 Mar 2025 01:49:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Cc:To:In-Reply-To:References :Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=McDxZasD/M7IQ6Jn7sHIcHjRPaMImkKi5TvB+FHfbZk=; b=2R4N8aTb6dseIUPtfziqX26J9y PUVBFNo1/RqiK8f4q/US9bKH4IMVtUpAjd6w2giZC9FKOXnpc025OSLw+FdpPmIm2DuCCik29LUa8 gwANQ6y7AF5XvSnVoLosYt9IoBbyauLP4Wjm1CIgXGd93RCyHPfyd5Q5Q23bVG7W/hLwWXoATCNdn iUmyDXMCwvlGLGzoI4XB4Tr3H8BxAciW44xwHqLBKdwB6OKTqtpGhf/iqgw4k0/x6UJS8Lb4KNWw/ xgdE9UafpYUiqd7t/huktrOl+jHT6nf+5u3qu86EOkSRu7RxIB/muIMEc+lZVOU7tiwoiOkNmfacB N5bN9xXA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tvRVi-0000000Deph-14gD; Fri, 21 Mar 2025 01:49:22 +0000 Received: from sea.source.kernel.org ([172.234.252.31]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tvQ5Y-0000000DXSs-1jJv for linux-arm-kernel@lists.infradead.org; Fri, 21 Mar 2025 00:18:17 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sea.source.kernel.org (Postfix) with ESMTP id 6323144EFD; Fri, 21 Mar 2025 00:18:15 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 463FBC4CEEC; Fri, 21 Mar 2025 00:18:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1742516295; bh=IkZsuY+kAaVDLrs8HWKvRFz6/h6ZoxMySJFddJZVVGc=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=HhiNkYVWAeRC2RvW4BcuNKDTM0/B1aYTF44nGEDLsHSDrUrA7ZG+hVEcZr0lS/MjA 4Iposz+TZ0CJDhh0QdEkaN00tWs+59OLc7gm39AkX4mi0RIuE6ip6g737Ka4uR6Bxc LWgJizA6e77UTwRXIAQ7z+NjcgkpUczT5thvI83r9zIkT3x1SNKxBUkIN2MhPeAIZQ bYqJYBvsqZ3uoRS/EiZL1xSpjuhz/ZhJUF1CrKIXrRld+I4LAsZRh58bjXElPsMnov 51hIC6rvsx1bnMbWOe+rD9X7+in6nXTdIBX5qWAfw7umppC+nSv8jEvouZtK7FFhYY XhE9A12SgFlRg== From: Mark Brown Date: Fri, 21 Mar 2025 00:16:01 +0000 Subject: [PATCH 6.6 1/8] KVM: arm64: Calculate cptr_el2 traps on activating traps MIME-Version: 1.0 Message-Id: <20250321-stable-sve-6-6-v1-1-0b3a6a14ea53@kernel.org> References: <20250321-stable-sve-6-6-v1-0-0b3a6a14ea53@kernel.org> In-Reply-To: <20250321-stable-sve-6-6-v1-0-0b3a6a14ea53@kernel.org> To: Greg Kroah-Hartman , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Catalin Marinas , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, stable@vger.kernel.org, Mark Brown , Fuad Tabba , James Clark X-Mailer: b4 0.15-dev-1b0d6 X-Developer-Signature: v=1; a=openpgp-sha256; l=7504; i=broonie@kernel.org; h=from:subject:message-id; bh=JSYgABijzaBEeRBZ792dyLFvcMocImtFap9K59dqIIQ=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBn3LA6vxysY9lgminsnLplSF2R0v15vrkadze3SGzB yuP0ohiJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCZ9ywOgAKCRAk1otyXVSH0J7FCA CFlv5e6TtLoZ19s1IAM8KyYVpP+nV5rks+ZA/9P/T6jpVPAA6DFB32VLwPiny2NiYudGUVM6Rb7fYS JByJxHEHvTU0Mkwgji6LHDUo/QpKF0I8QkON8T1XuhJ8X0mo/HGpieVUlkPQrWcnrR3tOASVui7USq R3jzdDKrolhqYygtPhgOPexLccf92NQJKjKgZxraShlV8HBFxscw6qvzSj2q1M0oAtdZcGJ2g0YbXR EyZnVwxIJd2hkionDQlrlYslt3ffevg8frgUpvqn+/t1ud7V4En51+3/MAMlVGvgOXaIj4cgbZnnZN pqiJS5XzEF9xGda/OlXekt3DPMuTAa X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250320_171816_501533_4B9CD743 X-CRM114-Status: GOOD ( 18.47 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Fuad Tabba [ Upstream commit 2fd5b4b0e7b440602455b79977bfa64dea101e6c ] Similar to VHE, calculate the value of cptr_el2 from scratch on activate traps. This removes the need to store cptr_el2 in every vcpu structure. Moreover, some traps, such as whether the guest owns the fp registers, need to be set on every vcpu run. Reported-by: James Clark Fixes: 5294afdbf45a ("KVM: arm64: Exclude FP ownership from kvm_vcpu_arch") Signed-off-by: Fuad Tabba Link: https://lore.kernel.org/r/20241216105057.579031-13-tabba@google.com Signed-off-by: Marc Zyngier Signed-off-by: Mark Brown --- arch/arm64/include/asm/kvm_host.h | 1 - arch/arm64/kvm/arm.c | 1 - arch/arm64/kvm/hyp/nvhe/hyp-main.c | 2 -- arch/arm64/kvm/hyp/nvhe/pkvm.c | 27 -------------------- arch/arm64/kvm/hyp/nvhe/switch.c | 52 +++++++++++++++++++++++--------------- 5 files changed, 32 insertions(+), 51 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index b84ed3ad91a9eba3a30a6081371f2ec98963de06..6a165ec5d3b74ece3e98e7bf45f3ea94cc30e6ec 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -486,7 +486,6 @@ struct kvm_vcpu_arch { /* Values of trap registers for the guest. */ u64 hcr_el2; u64 mdcr_el2; - u64 cptr_el2; /* Values of trap registers for the host before guest entry. */ u64 mdcr_el2_host; diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index ffdc2c4d07ee83dbeb78b76d93f700f53af35be6..9818cde948ca9c7028220322aea4ff3706542c93 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -1309,7 +1309,6 @@ static int kvm_arch_vcpu_ioctl_vcpu_init(struct kvm_vcpu *vcpu, } vcpu_reset_hcr(vcpu); - vcpu->arch.cptr_el2 = kvm_get_reset_cptr_el2(vcpu); /* * Handle the "start in power-off" case. diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c index 2385fd03ed87c6c0450a1c2dbe830e95ce235ecc..67cc07283e642ab07e1c98c5745e2c6a2dd4f36e 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c +++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c @@ -36,7 +36,6 @@ static void flush_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu) hyp_vcpu->vcpu.arch.hcr_el2 = host_vcpu->arch.hcr_el2; hyp_vcpu->vcpu.arch.mdcr_el2 = host_vcpu->arch.mdcr_el2; - hyp_vcpu->vcpu.arch.cptr_el2 = host_vcpu->arch.cptr_el2; hyp_vcpu->vcpu.arch.iflags = host_vcpu->arch.iflags; hyp_vcpu->vcpu.arch.fp_state = host_vcpu->arch.fp_state; @@ -59,7 +58,6 @@ static void sync_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu) host_vcpu->arch.ctxt = hyp_vcpu->vcpu.arch.ctxt; host_vcpu->arch.hcr_el2 = hyp_vcpu->vcpu.arch.hcr_el2; - host_vcpu->arch.cptr_el2 = hyp_vcpu->vcpu.arch.cptr_el2; host_vcpu->arch.fault = hyp_vcpu->vcpu.arch.fault; diff --git a/arch/arm64/kvm/hyp/nvhe/pkvm.c b/arch/arm64/kvm/hyp/nvhe/pkvm.c index 8033ef353a5da406dba355ab73854dfa39e93c27..9e7612343ad66d3dc13b8ee858a4e61e917901c9 100644 --- a/arch/arm64/kvm/hyp/nvhe/pkvm.c +++ b/arch/arm64/kvm/hyp/nvhe/pkvm.c @@ -26,8 +26,6 @@ static void pvm_init_traps_aa64pfr0(struct kvm_vcpu *vcpu) const u64 feature_ids = pvm_read_id_reg(vcpu, SYS_ID_AA64PFR0_EL1); u64 hcr_set = HCR_RW; u64 hcr_clear = 0; - u64 cptr_set = 0; - u64 cptr_clear = 0; /* Protected KVM does not support AArch32 guests. */ BUILD_BUG_ON(FIELD_GET(ARM64_FEATURE_MASK(ID_AA64PFR0_EL1_EL0), @@ -57,21 +55,10 @@ static void pvm_init_traps_aa64pfr0(struct kvm_vcpu *vcpu) /* Trap AMU */ if (!FIELD_GET(ARM64_FEATURE_MASK(ID_AA64PFR0_EL1_AMU), feature_ids)) { hcr_clear |= HCR_AMVOFFEN; - cptr_set |= CPTR_EL2_TAM; - } - - /* Trap SVE */ - if (!FIELD_GET(ARM64_FEATURE_MASK(ID_AA64PFR0_EL1_SVE), feature_ids)) { - if (has_hvhe()) - cptr_clear |= CPACR_EL1_ZEN_EL0EN | CPACR_EL1_ZEN_EL1EN; - else - cptr_set |= CPTR_EL2_TZ; } vcpu->arch.hcr_el2 |= hcr_set; vcpu->arch.hcr_el2 &= ~hcr_clear; - vcpu->arch.cptr_el2 |= cptr_set; - vcpu->arch.cptr_el2 &= ~cptr_clear; } /* @@ -101,7 +88,6 @@ static void pvm_init_traps_aa64dfr0(struct kvm_vcpu *vcpu) const u64 feature_ids = pvm_read_id_reg(vcpu, SYS_ID_AA64DFR0_EL1); u64 mdcr_set = 0; u64 mdcr_clear = 0; - u64 cptr_set = 0; /* Trap/constrain PMU */ if (!FIELD_GET(ARM64_FEATURE_MASK(ID_AA64DFR0_EL1_PMUVer), feature_ids)) { @@ -128,17 +114,8 @@ static void pvm_init_traps_aa64dfr0(struct kvm_vcpu *vcpu) if (!FIELD_GET(ARM64_FEATURE_MASK(ID_AA64DFR0_EL1_TraceFilt), feature_ids)) mdcr_set |= MDCR_EL2_TTRF; - /* Trap Trace */ - if (!FIELD_GET(ARM64_FEATURE_MASK(ID_AA64DFR0_EL1_TraceVer), feature_ids)) { - if (has_hvhe()) - cptr_set |= CPACR_EL1_TTA; - else - cptr_set |= CPTR_EL2_TTA; - } - vcpu->arch.mdcr_el2 |= mdcr_set; vcpu->arch.mdcr_el2 &= ~mdcr_clear; - vcpu->arch.cptr_el2 |= cptr_set; } /* @@ -189,10 +166,6 @@ static void pvm_init_trap_regs(struct kvm_vcpu *vcpu) /* Clear res0 and set res1 bits to trap potential new features. */ vcpu->arch.hcr_el2 &= ~(HCR_RES0); vcpu->arch.mdcr_el2 &= ~(MDCR_EL2_RES0); - if (!has_hvhe()) { - vcpu->arch.cptr_el2 |= CPTR_NVHE_EL2_RES1; - vcpu->arch.cptr_el2 &= ~(CPTR_NVHE_EL2_RES0); - } } /* diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/switch.c index c353a06ee7e6d624b41997021379b7b4cf77453d..1026be1964d9664b8b6de19e048aba91ee6463b0 100644 --- a/arch/arm64/kvm/hyp/nvhe/switch.c +++ b/arch/arm64/kvm/hyp/nvhe/switch.c @@ -36,34 +36,46 @@ DEFINE_PER_CPU(unsigned long, kvm_hyp_vector); extern void kvm_nvhe_prepare_backtrace(unsigned long fp, unsigned long pc); -static void __activate_traps(struct kvm_vcpu *vcpu) +static void __activate_cptr_traps(struct kvm_vcpu *vcpu) { - u64 val; + u64 val = CPTR_EL2_TAM; /* Same bit irrespective of E2H */ - ___activate_traps(vcpu); - __activate_traps_common(vcpu); + if (has_hvhe()) { + val |= CPACR_ELx_TTA; - val = vcpu->arch.cptr_el2; - val |= CPTR_EL2_TAM; /* Same bit irrespective of E2H */ - val |= has_hvhe() ? CPACR_EL1_TTA : CPTR_EL2_TTA; - if (cpus_have_final_cap(ARM64_SME)) { - if (has_hvhe()) - val &= ~(CPACR_EL1_SMEN_EL1EN | CPACR_EL1_SMEN_EL0EN); - else - val |= CPTR_EL2_TSM; - } + if (guest_owns_fp_regs(vcpu)) { + val |= CPACR_ELx_FPEN; + if (vcpu_has_sve(vcpu)) + val |= CPACR_ELx_ZEN; + } + } else { + val |= CPTR_EL2_TTA | CPTR_NVHE_EL2_RES1; - if (!guest_owns_fp_regs(vcpu)) { - if (has_hvhe()) - val &= ~(CPACR_EL1_FPEN_EL0EN | CPACR_EL1_FPEN_EL1EN | - CPACR_EL1_ZEN_EL0EN | CPACR_EL1_ZEN_EL1EN); - else - val |= CPTR_EL2_TFP | CPTR_EL2_TZ; + /* + * Always trap SME since it's not supported in KVM. + * TSM is RES1 if SME isn't implemented. + */ + val |= CPTR_EL2_TSM; - __activate_traps_fpsimd32(vcpu); + if (!vcpu_has_sve(vcpu) || !guest_owns_fp_regs(vcpu)) + val |= CPTR_EL2_TZ; + + if (!guest_owns_fp_regs(vcpu)) + val |= CPTR_EL2_TFP; } + if (!guest_owns_fp_regs(vcpu)) + __activate_traps_fpsimd32(vcpu); + kvm_write_cptr_el2(val); +} + +static void __activate_traps(struct kvm_vcpu *vcpu) +{ + ___activate_traps(vcpu); + __activate_traps_common(vcpu); + __activate_cptr_traps(vcpu); + write_sysreg(__this_cpu_read(kvm_hyp_vector), vbar_el2); if (cpus_have_final_cap(ARM64_WORKAROUND_SPECULATIVE_AT)) { From patchwork Fri Mar 21 00:16:02 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14024736 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E3EDCC28B30 for ; Fri, 21 Mar 2025 00:47:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Cc:To:In-Reply-To:References :Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=/keNk94Jj0RLJp9bKdnJNiC36sWIq4WGOXGleXCKUZM=; b=O7UtsT+8UuYm8B7MQ4r/fNPtnw vwpVjuy4D1itb2us2TVZARrOj5lYGukT7f8U/jJG7oPV33yI6QksoRMBcOX3wKehzQsE9ie/hqgCH 3why1YgQp1tisGphp59LviBvozrZxXtxbouhUpE7Ru0cCwrmRfRbhwqXxvtB5oqWGa74Eq7dEAFfs LUnUwa7RXfYKNizOjw4kaNwfZ1MofEXakyEJ7jknIE0j1GegIUBW1GU2mdgEf2WbAhP3saYoew4Va +Ut+quyrcaN673IchbywfxXXNhrQIsp85aPL48c5miAxLsfh1xp6fZ3pErvB3pUbzndnwmL0PNiao 6Rop3znQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tvQXN-0000000DaYl-2JJ3; Fri, 21 Mar 2025 00:47:01 +0000 Received: from nyc.source.kernel.org ([147.75.193.91]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tvQ5e-0000000DXTi-381f for linux-arm-kernel@lists.infradead.org; Fri, 21 Mar 2025 00:18:27 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id B66A0A49777; Fri, 21 Mar 2025 00:12:52 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5B2EEC4CEE3; Fri, 21 Mar 2025 00:18:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1742516301; bh=UbWww6ZZCpLEum2jlpLA8W/p3yJo1aXcx0b1YN9WsH4=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=FLYDaHn8wRwuyMSE+yljDayCuGrvW3NaEp11czrX0vW63Qgb5ZKLx86eKcOLQLsEz 1hOHRdh85DhjeTOHf8W9HIf6H6JA9EuRIPvSdLd8I1jTJFR8qpzZI+vmGOqiS/LqSm 0S0WrikJtKvJ8y8INVZbdC3mM8c7vNwCwEDaN0YdhM0C0yI2u9YumbxOL6R7J7T8a/ lpleJapMvrc8rielFmoxGbXupNyFBM3EHHygykwi3/EWtR6Kk2f/38cHl236E+EQc/ eqrJ02WSp/ucjF/Iiqd8FxV9Effh51q7Q39QDOEQxDnCKBQSRmyS8S2ptIPe7o/oK5 xIcMhsXqaKxvQ== From: Mark Brown Date: Fri, 21 Mar 2025 00:16:02 +0000 Subject: [PATCH 6.6 2/8] KVM: arm64: Unconditionally save+flush host FPSIMD/SVE/SME state MIME-Version: 1.0 Message-Id: <20250321-stable-sve-6-6-v1-2-0b3a6a14ea53@kernel.org> References: <20250321-stable-sve-6-6-v1-0-0b3a6a14ea53@kernel.org> In-Reply-To: <20250321-stable-sve-6-6-v1-0-0b3a6a14ea53@kernel.org> To: Greg Kroah-Hartman , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Catalin Marinas , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, stable@vger.kernel.org, Mark Brown , Mark Rutland , Eric Auger , Wilco Dijkstra , Eric Auger , Florian Weimer , Fuad Tabba , Jeremy Linton , Paolo Bonzini X-Mailer: b4 0.15-dev-1b0d6 X-Developer-Signature: v=1; a=openpgp-sha256; l=6160; i=broonie@kernel.org; h=from:subject:message-id; bh=1GT1GHGJkE6mMiHosvFrZPe9zuk67Q6/Q/eJEMkzMPE=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBn3LA7XVSV2UC+a4fX1hxzBSPcO8gNLi0mspm/i10/ dd3hnKuJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCZ9ywOwAKCRAk1otyXVSH0LxMB/ 9MlW0SFYEJskV9nXzTGC/Iw897O08ZbXf+p9zefH3dMbq0jbne0GYrU5AOToosoCh9lqhaDQXV1vNp Pmopr9kh2FYMEHTcSfgRLY76KqPeizJryb5lf9B/zMtb1xh72T0T+Rzqn7EoeRFGW9rVfjKfYru3v0 tGZkGyfCofbw5dPblJ5Pqr/yMGxNDA1CNmLzI3mSJM0LLcSKBjEecLNbmMRAV+VREPR/BnsVgxp3/d PhNk2a90Ec8APYJHfQEEt3q+e0Q2HH4PT1jZJu+jYabpbrI1aZ5VkG913X0s+VQZkoIk1d3hkasgoL Mvl+B6zdmtBXF8HXNKJ3e97dUq23Jx X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250320_171822_926912_B1C24F94 X-CRM114-Status: GOOD ( 27.05 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Mark Rutland [ Upstream commit fbc7e61195e23f744814e78524b73b59faa54ab4 ] There are several problems with the way hyp code lazily saves the host's FPSIMD/SVE state, including: * Host SVE being discarded unexpectedly due to inconsistent configuration of TIF_SVE and CPACR_ELx.ZEN. This has been seen to result in QEMU crashes where SVE is used by memmove(), as reported by Eric Auger: https://issues.redhat.com/browse/RHEL-68997 * Host SVE state is discarded *after* modification by ptrace, which was an unintentional ptrace ABI change introduced with lazy discarding of SVE state. * The host FPMR value can be discarded when running a non-protected VM, where FPMR support is not exposed to a VM, and that VM uses FPSIMD/SVE. In these cases the hyp code does not save the host's FPMR before unbinding the host's FPSIMD/SVE/SME state, leaving a stale value in memory. Avoid these by eagerly saving and "flushing" the host's FPSIMD/SVE/SME state when loading a vCPU such that KVM does not need to save any of the host's FPSIMD/SVE/SME state. For clarity, fpsimd_kvm_prepare() is removed and the necessary call to fpsimd_save_and_flush_cpu_state() is placed in kvm_arch_vcpu_load_fp(). As 'fpsimd_state' and 'fpmr_ptr' should not be used, they are set to NULL; all uses of these will be removed in subsequent patches. Historical problems go back at least as far as v5.17, e.g. erroneous assumptions about TIF_SVE being clear in commit: 8383741ab2e773a9 ("KVM: arm64: Get rid of host SVE tracking/saving") ... and so this eager save+flush probably needs to be backported to ALL stable trees. Fixes: 93ae6b01bafee8fa ("KVM: arm64: Discard any SVE state when entering KVM guests") Fixes: 8c845e2731041f0f ("arm64/sve: Leave SVE enabled on syscall if we don't context switch") Fixes: ef3be86021c3bdf3 ("KVM: arm64: Add save/restore support for FPMR") Reported-by: Eric Auger Reported-by: Wilco Dijkstra Reviewed-by: Mark Brown Tested-by: Mark Brown Tested-by: Eric Auger Acked-by: Will Deacon Cc: Catalin Marinas Cc: Florian Weimer Cc: Fuad Tabba Cc: Jeremy Linton Cc: Marc Zyngier Cc: Oliver Upton Cc: Paolo Bonzini Signed-off-by: Mark Rutland Reviewed-by: Oliver Upton Link: https://lore.kernel.org/r/20250210195226.1215254-2-mark.rutland@arm.com Signed-off-by: Marc Zyngier [ Mark: Handle vcpu/host flag conflict ] Signed-off-by: Mark Rutland Signed-off-by: Mark Brown --- arch/arm64/kernel/fpsimd.c | 25 ------------------------- arch/arm64/kvm/fpsimd.c | 31 ++++++++----------------------- 2 files changed, 8 insertions(+), 48 deletions(-) diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c index 0137d987631e089c935560eccbc678a258580232..bd4f6c6ee0f318e1f60d40755c0f4915edb415b8 100644 --- a/arch/arm64/kernel/fpsimd.c +++ b/arch/arm64/kernel/fpsimd.c @@ -1707,31 +1707,6 @@ void fpsimd_signal_preserve_current_state(void) sve_to_fpsimd(current); } -/* - * Called by KVM when entering the guest. - */ -void fpsimd_kvm_prepare(void) -{ - if (!system_supports_sve()) - return; - - /* - * KVM does not save host SVE state since we can only enter - * the guest from a syscall so the ABI means that only the - * non-saved SVE state needs to be saved. If we have left - * SVE enabled for performance reasons then update the task - * state to be FPSIMD only. - */ - get_cpu_fpsimd_context(); - - if (test_and_clear_thread_flag(TIF_SVE)) { - sve_to_fpsimd(current); - current->thread.fp_type = FP_STATE_FPSIMD; - } - - put_cpu_fpsimd_context(); -} - /* * Associate current's FPSIMD context with this cpu * The caller must have ownership of the cpu FPSIMD context before calling diff --git a/arch/arm64/kvm/fpsimd.c b/arch/arm64/kvm/fpsimd.c index 8c1d0d4853df48abf4d089bbde153bcee8d0e6d0..8b55de502c8c220e15e3a6b782d5012b9349b612 100644 --- a/arch/arm64/kvm/fpsimd.c +++ b/arch/arm64/kvm/fpsimd.c @@ -79,14 +79,16 @@ void kvm_arch_vcpu_load_fp(struct kvm_vcpu *vcpu) if (!system_supports_fpsimd()) return; - fpsimd_kvm_prepare(); - /* - * We will check TIF_FOREIGN_FPSTATE just before entering the - * guest in kvm_arch_vcpu_ctxflush_fp() and override this to - * FP_STATE_FREE if the flag set. + * Ensure that any host FPSIMD/SVE/SME state is saved and unbound such + * that the host kernel is responsible for restoring this state upon + * return to userspace, and the hyp code doesn't need to save anything. + * + * When the host may use SME, fpsimd_save_and_flush_cpu_state() ensures + * that PSTATE.{SM,ZA} == {0,0}. */ - vcpu->arch.fp_state = FP_STATE_HOST_OWNED; + fpsimd_save_and_flush_cpu_state(); + vcpu->arch.fp_state = FP_STATE_FREE; vcpu_clear_flag(vcpu, HOST_SVE_ENABLED); if (read_sysreg(cpacr_el1) & CPACR_EL1_ZEN_EL0EN) @@ -96,23 +98,6 @@ void kvm_arch_vcpu_load_fp(struct kvm_vcpu *vcpu) vcpu_clear_flag(vcpu, HOST_SME_ENABLED); if (read_sysreg(cpacr_el1) & CPACR_EL1_SMEN_EL0EN) vcpu_set_flag(vcpu, HOST_SME_ENABLED); - - /* - * If PSTATE.SM is enabled then save any pending FP - * state and disable PSTATE.SM. If we leave PSTATE.SM - * enabled and the guest does not enable SME via - * CPACR_EL1.SMEN then operations that should be valid - * may generate SME traps from EL1 to EL1 which we - * can't intercept and which would confuse the guest. - * - * Do the same for PSTATE.ZA in the case where there - * is state in the registers which has not already - * been saved, this is very unlikely to happen. - */ - if (read_sysreg_s(SYS_SVCR) & (SVCR_SM_MASK | SVCR_ZA_MASK)) { - vcpu->arch.fp_state = FP_STATE_FREE; - fpsimd_save_and_flush_cpu_state(); - } } } From patchwork Fri Mar 21 00:16:03 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14024737 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DEFF8C28B30 for ; Fri, 21 Mar 2025 00:48:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Cc:To:In-Reply-To:References :Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=+RlBFQKcUm851xKaMijOjPf8mHiI9ukDgAYZnStDiIk=; b=S2u7/i5SuR0v91jnjNu7G3g6ys w9MUkSjNj1S7RDk6ClDHBS+aTMiOXK/upJ7qNKQ84yqFMruuWaOxjsoy9CXYf7M/eIH0ZSEpl63sF FmyoZCVfDarL9K4GS+10KnH//QcXP6iohgnc4CMpoiZSlnbJ7IyeZvMY2OX50HQUSxWFvtqoQNilg WwyhbRX0l29g9rqWnnE3uyI+hFG/cDEzt86qtsmPS/9zGQDzu2PzhuCZHaLULacJeo0pHp0Ju2MCh 4yZzMa/U5OMi99xLpuRdrepk48a8Zc+/QHxjopzvgt+7QRsj+PcqQIVbqFii6qjFxJiRN4EfBG6w8 zlTgORiA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tvQZ3-0000000Dahe-1AiR; Fri, 21 Mar 2025 00:48:45 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tvQ5i-0000000DXUO-01rH for linux-arm-kernel@lists.infradead.org; Fri, 21 Mar 2025 00:18:29 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 6E8DB5C6827; Fri, 21 Mar 2025 00:16:08 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6BE19C4CEDD; Fri, 21 Mar 2025 00:18:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1742516305; bh=ehJMCPrMwHOlFomdHlkXQQW8qPj0SLCNghFqe+fTFOM=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=uhok4P0S6bgzI3Z7aFQKSSP/QCgeqUlIoUmipEsI4u6KJ2HPCO3jjFiuZcXamCWG9 P3bjVGzgsz8uWUslAoCpwoK+rVvPeox3BoE1niMr10DmrUjWyt4hAFYjhQD76k4k+a QAtcb25F+gNpDsglUV1/hVjwAmr2YNkb0OVAfdQCJBtRy2uywPzuNsGPWuZR4FG+Ts Nfs1moG/y1hz79wbnkK9hQNToVmLnkw4W8K9ZbCoQ6gpOO4PQCaRu8FSxUfSeRrRmq WG13BzkxQrtouG3M+fY7goSEHoXdTb3JVc9E9kjaNZSCN7pVdTHZhQNVfoY4a82rwF B4gsMLORcLTBA== From: Mark Brown Date: Fri, 21 Mar 2025 00:16:03 +0000 Subject: [PATCH 6.6 3/8] KVM: arm64: Remove host FPSIMD saving for non-protected KVM MIME-Version: 1.0 Message-Id: <20250321-stable-sve-6-6-v1-3-0b3a6a14ea53@kernel.org> References: <20250321-stable-sve-6-6-v1-0-0b3a6a14ea53@kernel.org> In-Reply-To: <20250321-stable-sve-6-6-v1-0-0b3a6a14ea53@kernel.org> To: Greg Kroah-Hartman , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Catalin Marinas , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, stable@vger.kernel.org, Mark Brown , Mark Rutland , Fuad Tabba X-Mailer: b4 0.15-dev-1b0d6 X-Developer-Signature: v=1; a=openpgp-sha256; l=3882; i=broonie@kernel.org; h=from:subject:message-id; bh=RFWrMPCwzs0NH7GB08LQFJjXWfWPvpZMwKDg1phGkog=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBn3LA8ZtpxXliTO/VwpxtBvZVJylNzVfVYCbmDkzaj Xd/IK9aJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCZ9ywPAAKCRAk1otyXVSH0MbzB/ wP53p1K4tkyva6gQKK/UQ2Am19w+vVgl1NQvUVQqQLG9B3EKdjC22dwXpA7xe8hotvZZdSHx9pjV7r tLOBh6K7tR4/HMm+D0XHhkhwaW7aa1kMUG/qomMwllR2qknV+1n2J6rnUZ+ZVb37Fk6MEXfAqw0ent j60z/STYlcC0O7caND8km2ZkzA1eQHCgfAO5JFacPKMAOqmEUGkU/wMx7zZlYmYMk7rJA0pMtV9LN1 ulJgMiNYrDFJ33yFuEer869qv7lQfrsDRKOE6Hv7n8kldwUgodnXc+pCqr9ba084WVPN7sUO0cF+ti 7KluJRVG4ZWjQWnlHBo+dXSO+hfwdK X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250320_171826_139541_0259BC00 X-CRM114-Status: GOOD ( 17.48 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Mark Rutland [ Upstream commit 8eca7f6d5100b6997df4f532090bc3f7e0203bef ] Now that the host eagerly saves its own FPSIMD/SVE/SME state, non-protected KVM never needs to save the host FPSIMD/SVE/SME state, and the code to do this is never used. Protected KVM still needs to save/restore the host FPSIMD/SVE state to avoid leaking guest state to the host (and to avoid revealing to the host whether the guest used FPSIMD/SVE/SME), and that code needs to be retained. Remove the unused code and data structures. To avoid the need for a stub copy of kvm_hyp_save_fpsimd_host() in the VHE hyp code, the nVHE/hVHE version is moved into the shared switch header, where it is only invoked when KVM is in protected mode. Signed-off-by: Mark Rutland Reviewed-by: Mark Brown Tested-by: Mark Brown Acked-by: Will Deacon Cc: Catalin Marinas Cc: Fuad Tabba Cc: Marc Zyngier Cc: Oliver Upton Reviewed-by: Oliver Upton Link: https://lore.kernel.org/r/20250210195226.1215254-3-mark.rutland@arm.com Signed-off-by: Marc Zyngier Signed-off-by: Mark Brown --- arch/arm64/include/asm/kvm_host.h | 1 - arch/arm64/kvm/fpsimd.c | 2 -- arch/arm64/kvm/hyp/include/hyp/switch.h | 4 ---- arch/arm64/kvm/hyp/nvhe/hyp-main.c | 1 - 4 files changed, 8 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 6a165ec5d3b74ece3e98e7bf45f3ea94cc30e6ec..3891963d42e00c8f999886dc5d7322bbacbc6c7f 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -535,7 +535,6 @@ struct kvm_vcpu_arch { struct kvm_guest_debug_arch vcpu_debug_state; struct kvm_guest_debug_arch external_debug_state; - struct user_fpsimd_state *host_fpsimd_state; /* hyp VA */ struct task_struct *parent_task; struct { diff --git a/arch/arm64/kvm/fpsimd.c b/arch/arm64/kvm/fpsimd.c index 8b55de502c8c220e15e3a6b782d5012b9349b612..7c36d2a7aa3196056f76acfe8f9c41763ed67d9d 100644 --- a/arch/arm64/kvm/fpsimd.c +++ b/arch/arm64/kvm/fpsimd.c @@ -49,8 +49,6 @@ int kvm_arch_vcpu_run_map_fp(struct kvm_vcpu *vcpu) if (ret) return ret; - vcpu->arch.host_fpsimd_state = kern_hyp_va(fpsimd); - /* * We need to keep current's task_struct pinned until its data has been * unshared with the hypervisor to make sure it is not re-used by the diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h index 9cfe6bd1dbe459cb3588bccd94359369a546947e..1cdc8d161e7dd3aeeb6b03170e239b0c1ace7790 100644 --- a/arch/arm64/kvm/hyp/include/hyp/switch.h +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h @@ -321,10 +321,6 @@ static bool kvm_hyp_handle_fpsimd(struct kvm_vcpu *vcpu, u64 *exit_code) } isb(); - /* Write out the host state if it's in the registers */ - if (vcpu->arch.fp_state == FP_STATE_HOST_OWNED) - __fpsimd_save_state(vcpu->arch.host_fpsimd_state); - /* Restore the guest state */ if (sve_guest) __hyp_sve_restore_guest(vcpu); diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c index 67cc07283e642ab07e1c98c5745e2c6a2dd4f36e..8390173f7f5b38e01c04ae6563853df11a1c3df2 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c +++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c @@ -41,7 +41,6 @@ static void flush_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu) hyp_vcpu->vcpu.arch.fp_state = host_vcpu->arch.fp_state; hyp_vcpu->vcpu.arch.debug_ptr = kern_hyp_va(host_vcpu->arch.debug_ptr); - hyp_vcpu->vcpu.arch.host_fpsimd_state = host_vcpu->arch.host_fpsimd_state; hyp_vcpu->vcpu.arch.vsesr_el2 = host_vcpu->arch.vsesr_el2; From patchwork Fri Mar 21 00:16:04 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14024738 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 27866C28B30 for ; Fri, 21 Mar 2025 00:50:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Cc:To:In-Reply-To:References :Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=0e+mUjDAWf3s0YKL6KQvH3NwIpYY0unoMYXfb9E1b+g=; b=XKs2au3Cx4trO6Pw8l4l3L9Gqi 9eMMhDCsiOwGibCyVID7DAfGcs6Z3jQdBH5v8byP0n8Ts0MKHuvuytnwORQ745r54xU69f63t2+Km rCCMjncUkV4sg3iuJP6Rkwd6eguFlDkO4K01EDXzL0upFw3gH42MCR/JRbMBILf9JQvd8j1TGka+p Zm7eiV/I/NZnWB+wjKweBHt0S3ufwYfxp5R8DFw7hhhEUagQR4wpcX4ppkYgPgsjI9cOH19OGHm85 ox6AB5hGR6trJ4KG0lJBa6X9a/ziZ0jTGcpzrLkmth5BHe7tUEtQwped7vW9yVnXYBtX9mf9G4AOF omsB/J4w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tvQai-0000000DapP-433Y; Fri, 21 Mar 2025 00:50:28 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tvQ5k-0000000DXV5-3dyW for linux-arm-kernel@lists.infradead.org; Fri, 21 Mar 2025 00:18:30 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 4AEA25C6820; Fri, 21 Mar 2025 00:16:11 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 85AE3C4CEDD; Fri, 21 Mar 2025 00:18:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1742516308; bh=CFBkZ3D3qqKor5Kj9cL5tM3zTXoy9Y0nRxvtgtnhaUQ=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=pIVATjvSPxV7eaF1uh1SB3g2F3VirR4rPbKKUZnzb+gD8uIISrkYKbRDHG6ujaOv0 Coqp+rN5bUMGgY4EEAdIoXSt3qkvcZ82q3TLLdsmlG8DgJhNrGs7D53PwxiAtO8Ss1 e0sU24+C3MRPDkz7YZo1m6A9x6rZpmIBPj8G+UGacbTSCKnaAhjJgMVrCW9xl1Te6F Gv1UHwVhr6Tp2Q8KxQeZ++thAAHOb1LumHpZc7sCQLnGisiw8be1WmdWJbYG6mbzgh PtX7nNY2KZAIy5L5VMwIwSjxhPUg+QsenQp2Gr8n8HkwOueDkDxQEfZl6B46HVr+F2 E9kb27/LopShA== From: Mark Brown Date: Fri, 21 Mar 2025 00:16:04 +0000 Subject: [PATCH 6.6 4/8] KVM: arm64: Remove VHE host restore of CPACR_EL1.ZEN MIME-Version: 1.0 Message-Id: <20250321-stable-sve-6-6-v1-4-0b3a6a14ea53@kernel.org> References: <20250321-stable-sve-6-6-v1-0-0b3a6a14ea53@kernel.org> In-Reply-To: <20250321-stable-sve-6-6-v1-0-0b3a6a14ea53@kernel.org> To: Greg Kroah-Hartman , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Catalin Marinas , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, stable@vger.kernel.org, Mark Brown , Mark Rutland , Fuad Tabba X-Mailer: b4 0.15-dev-1b0d6 X-Developer-Signature: v=1; a=openpgp-sha256; l=3631; i=broonie@kernel.org; h=from:subject:message-id; bh=bEhjaWWeo3MZO4hv+4A0+U6DdeP5Zvjor/41KIDmUy4=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBn3LA9h7iGC6z3d780UO9FNjZjJlZmARitK1wAo5KK CyUCMcmJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCZ9ywPQAKCRAk1otyXVSH0CueCA CCr8ZNxeoYBGooYidPdc4468iwsk8Q3DIHQHwgFNO/Wa9FS57xkTpMSZFsBJEhksiGasBeDTcQMOQV YfDj4w8ImQclIvmXq/WPHewaesljGxLjGaHGCapiX5XpVeXJTuHvG5IP/2/dK2WKPcUTsoxZ0f7wm7 awdhgw+iZA/ntG3LSKmP7LP83HyzQ1CE6xyBu/r/NVW0zgv5tr22bRdd2qKKivuK0SPsdUKS8RUjDl mW4suwnCmYV56lizBnBYD0+AX5hYR/KuDL8KDBFUUYOnjCAgohQIBS7Tzz762mKkOTVUrqfqygl8VZ xr8T/jywDbQDXm7I7bOj3HMe0KnvVN X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250320_171829_021067_A3594C93 X-CRM114-Status: GOOD ( 17.35 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Mark Rutland [ Upstream commit 459f059be702056d91537b99a129994aa6ccdd35 ] When KVM is in VHE mode, the host kernel tries to save and restore the configuration of CPACR_EL1.ZEN (i.e. CPTR_EL2.ZEN when HCR_EL2.E2H=1) across kvm_arch_vcpu_load_fp() and kvm_arch_vcpu_put_fp(), since the configuration may be clobbered by hyp when running a vCPU. This logic is currently redundant. The VHE hyp code unconditionally configures CPTR_EL2.ZEN to 0b01 when returning to the host, permitting host kernel usage of SVE. Now that the host eagerly saves and unbinds its own FPSIMD/SVE/SME state, there's no need to save/restore the state of the EL0 SVE trap. The kernel can safely save/restore state without trapping, as described above, and will restore userspace state (including trap controls) before returning to userspace. Remove the redundant logic. Signed-off-by: Mark Rutland Reviewed-by: Mark Brown Tested-by: Mark Brown Acked-by: Will Deacon Cc: Catalin Marinas Cc: Fuad Tabba Cc: Marc Zyngier Cc: Oliver Upton Reviewed-by: Oliver Upton Link: https://lore.kernel.org/r/20250210195226.1215254-4-mark.rutland@arm.com Signed-off-by: Marc Zyngier [Rework for refactoring of where the flags are stored -- broonie] Signed-off-by: Mark Brown --- arch/arm64/include/asm/kvm_host.h | 2 -- arch/arm64/kvm/fpsimd.c | 16 ---------------- 2 files changed, 18 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 3891963d42e00c8f999886dc5d7322bbacbc6c7f..9c1056d6f6859522dcee4b8c4e4804a7abac93d5 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -717,8 +717,6 @@ struct kvm_vcpu_arch { /* vcpu running in HYP context */ #define VCPU_HYP_CONTEXT __vcpu_single_flag(iflags, BIT(7)) -/* SVE enabled for host EL0 */ -#define HOST_SVE_ENABLED __vcpu_single_flag(sflags, BIT(0)) /* SME enabled for EL0 */ #define HOST_SME_ENABLED __vcpu_single_flag(sflags, BIT(1)) /* Physical CPU not in supported_cpus */ diff --git a/arch/arm64/kvm/fpsimd.c b/arch/arm64/kvm/fpsimd.c index 7c36d2a7aa3196056f76acfe8f9c41763ed67d9d..f1fe7abbcb83393e7c3402405bb7b1fdbf6024ff 100644 --- a/arch/arm64/kvm/fpsimd.c +++ b/arch/arm64/kvm/fpsimd.c @@ -88,10 +88,6 @@ void kvm_arch_vcpu_load_fp(struct kvm_vcpu *vcpu) fpsimd_save_and_flush_cpu_state(); vcpu->arch.fp_state = FP_STATE_FREE; - vcpu_clear_flag(vcpu, HOST_SVE_ENABLED); - if (read_sysreg(cpacr_el1) & CPACR_EL1_ZEN_EL0EN) - vcpu_set_flag(vcpu, HOST_SVE_ENABLED); - if (system_supports_sme()) { vcpu_clear_flag(vcpu, HOST_SME_ENABLED); if (read_sysreg(cpacr_el1) & CPACR_EL1_SMEN_EL0EN) @@ -189,18 +185,6 @@ void kvm_arch_vcpu_put_fp(struct kvm_vcpu *vcpu) } fpsimd_save_and_flush_cpu_state(); - } else if (has_vhe() && system_supports_sve()) { - /* - * The FPSIMD/SVE state in the CPU has not been touched, and we - * have SVE (and VHE): CPACR_EL1 (alias CPTR_EL2) has been - * reset by kvm_reset_cptr_el2() in the Hyp code, disabling SVE - * for EL0. To avoid spurious traps, restore the trap state - * seen by kvm_arch_vcpu_load_fp(): - */ - if (vcpu_get_flag(vcpu, HOST_SVE_ENABLED)) - sysreg_clear_set(CPACR_EL1, 0, CPACR_EL1_ZEN_EL0EN); - else - sysreg_clear_set(CPACR_EL1, CPACR_EL1_ZEN_EL0EN, 0); } local_irq_restore(flags); From patchwork Fri Mar 21 00:16:05 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14024739 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C1DB7C28B30 for ; Fri, 21 Mar 2025 00:52:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Cc:To:In-Reply-To:References :Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=gU4hIFZh9SFhhoeYu01WX0gsA4lkw5uhrBAuwj/pkTQ=; b=HeA8GbN3xhpQ3/9mAdCViDA3No HstuN61isbHMWDVZ4blv1+WcDPD7yhBiXkbsU40dxxZGCNK3eQQ7VC1S+Er41hYNu/fXrpEbTETiQ iExnpxbO3CUNIynq6KY5Q5DS6zkI2uEaXhSbqAASRhDWJ4rX0UjcWe00a/CMXeyPOUp0l1hvLJ0gx Q+ftWrMStiVG8/s/RGNpzIgL9Tia7f/s+bnwT1EXSbYWgagmNCSN1kcOnMqvosx42xB47tJlMQ5ux R9chuU2A0YMf3DO+AAzwTo6x04qrtnSXUJCw8hHuvWuShwucmMHEJfxwT78CdBsVUQNjHkY74jEDB vqc77DCQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tvQcP-0000000DawK-2bFb; Fri, 21 Mar 2025 00:52:13 +0000 Received: from tor.source.kernel.org ([172.105.4.254]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tvQ5o-0000000DXVi-00OY for linux-arm-kernel@lists.infradead.org; Fri, 21 Mar 2025 00:18:32 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by tor.source.kernel.org (Postfix) with ESMTP id 52F7A615BB; Fri, 21 Mar 2025 00:18:29 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 607DBC4CEEA; Fri, 21 Mar 2025 00:18:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1742516310; bh=ZqZu/OIKLFWaTlZe8i8EThKet2GXAnjLMbBwEC506ms=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=BFJ/BWWtUsZOaVEr/IxKaarSBg4XbMl8SlTMKBYvTx2PBaCZfGKeG+gxJgpbN+rDU ZsJAUce7wJ1LLDfUlX6hBsaoUVuVGLKEGLoLLjsZecEIF+4TWCkHolEO9Fe6QtAV7c 7JarArWBWvJFPG3KXeBlkIaiUW8cefUiwodowrf7jOzYd25zNXbE2OH2bXa7G/Q1gh gdsmYsjuT8FJ5RB5A2TDm3AC8Fd4Pmie5+72ZKVwmoaMQXHJ/wxpfnX7bfZgRO6Pb4 ke20UF0F2v0BqHq/iBXU7WbeVTvP2I3TJgFmRqpe35xxk23wRiBsadCjtHltkvI/cy ca6YYGLXckBXQ== From: Mark Brown Date: Fri, 21 Mar 2025 00:16:05 +0000 Subject: [PATCH 6.6 5/8] KVM: arm64: Remove VHE host restore of CPACR_EL1.SMEN MIME-Version: 1.0 Message-Id: <20250321-stable-sve-6-6-v1-5-0b3a6a14ea53@kernel.org> References: <20250321-stable-sve-6-6-v1-0-0b3a6a14ea53@kernel.org> In-Reply-To: <20250321-stable-sve-6-6-v1-0-0b3a6a14ea53@kernel.org> To: Greg Kroah-Hartman , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Catalin Marinas , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, stable@vger.kernel.org, Mark Brown , Mark Rutland , Fuad Tabba X-Mailer: b4 0.15-dev-1b0d6 X-Developer-Signature: v=1; a=openpgp-sha256; l=4610; i=broonie@kernel.org; h=from:subject:message-id; bh=KPoLYPWBSzZcnvIISNKfbhEVULIfJGkJjRCo5GPoRtE=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBn3LA9pLOeOCWXk9Bbg48Gc9v306SXIZRAlw1DjtCr WLkRbLyJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCZ9ywPQAKCRAk1otyXVSH0KgXB/ 9ZLaJiuCyWeNYOKY8BZOaoZ6dmEqQ0N45Y5Xzgtr/vTR8M44uNGSe2zyltCQQBuwo5psuyWo76ouL6 1kW7Onpbr+xcgreb0/VaNkPKnDorCSYBYpQQ/8om2dXaWJUkirAqiyLO97exbHaEvtxjOtkf+OyxWA 26Nf2+zg/RDq+v46eh2SjmaE+0rV1DQjugqFW0FoPR5jzM1GpVT2QVUzPQYoCH1GLRgIsP519oRMck 2t96jxJwPB43jw18VD23fGh/Svvav5xmTVcZocGRwdxINY4v9rsxsF0hw0dCF1QYmIaZYTR8C2n9OA D8k23BSRQx7ezSkYx8z5aafCr/LAKy X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Mark Rutland [ Upstream commit 407a99c4654e8ea65393f412c421a55cac539f5b ] When KVM is in VHE mode, the host kernel tries to save and restore the configuration of CPACR_EL1.SMEN (i.e. CPTR_EL2.SMEN when HCR_EL2.E2H=1) across kvm_arch_vcpu_load_fp() and kvm_arch_vcpu_put_fp(), since the configuration may be clobbered by hyp when running a vCPU. This logic has historically been broken, and is currently redundant. This logic was originally introduced in commit: 861262ab86270206 ("KVM: arm64: Handle SME host state when running guests") At the time, the VHE hyp code would reset CPTR_EL2.SMEN to 0b00 when returning to the host, trapping host access to SME state. Unfortunately, this was unsafe as the host could take a softirq before calling kvm_arch_vcpu_put_fp(), and if a softirq handler were to use kernel mode NEON the resulting attempt to save the live FPSIMD/SVE/SME state would result in a fatal trap. That issue was limited to VHE mode. For nVHE/hVHE modes, KVM always saved/restored the host kernel's CPACR_EL1 value, and configured CPTR_EL2.TSM to 0b0, ensuring that host usage of SME would not be trapped. The issue above was incidentally fixed by commit: 375110ab51dec5dc ("KVM: arm64: Fix resetting SME trap values on reset for (h)VHE") That commit changed the VHE hyp code to configure CPTR_EL2.SMEN to 0b01 when returning to the host, permitting host kernel usage of SME, avoiding the issue described above. At the time, this was not identified as a fix for commit 861262ab86270206. Now that the host eagerly saves and unbinds its own FPSIMD/SVE/SME state, there's no need to save/restore the state of the EL0 SME trap. The kernel can safely save/restore state without trapping, as described above, and will restore userspace state (including trap controls) before returning to userspace. Remove the redundant logic. Signed-off-by: Mark Rutland Reviewed-by: Mark Brown Tested-by: Mark Brown Acked-by: Will Deacon Cc: Catalin Marinas Cc: Fuad Tabba Cc: Marc Zyngier Cc: Oliver Upton Reviewed-by: Oliver Upton Link: https://lore.kernel.org/r/20250210195226.1215254-5-mark.rutland@arm.com Signed-off-by: Marc Zyngier [Update for rework of flags storage -- broonie] Signed-off-by: Mark Brown --- arch/arm64/include/asm/kvm_host.h | 2 -- arch/arm64/kvm/fpsimd.c | 23 ----------------------- 2 files changed, 25 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 9c1056d6f6859522dcee4b8c4e4804a7abac93d5..27c53b603d693d1cfe8d81f94a59d3afb665be8e 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -717,8 +717,6 @@ struct kvm_vcpu_arch { /* vcpu running in HYP context */ #define VCPU_HYP_CONTEXT __vcpu_single_flag(iflags, BIT(7)) -/* SME enabled for EL0 */ -#define HOST_SME_ENABLED __vcpu_single_flag(sflags, BIT(1)) /* Physical CPU not in supported_cpus */ #define ON_UNSUPPORTED_CPU __vcpu_single_flag(sflags, BIT(2)) /* WFIT instruction trapped */ diff --git a/arch/arm64/kvm/fpsimd.c b/arch/arm64/kvm/fpsimd.c index f1fe7abbcb83393e7c3402405bb7b1fdbf6024ff..0492a1fc718130934f8bada836fe42bea6db7abb 100644 --- a/arch/arm64/kvm/fpsimd.c +++ b/arch/arm64/kvm/fpsimd.c @@ -87,12 +87,6 @@ void kvm_arch_vcpu_load_fp(struct kvm_vcpu *vcpu) */ fpsimd_save_and_flush_cpu_state(); vcpu->arch.fp_state = FP_STATE_FREE; - - if (system_supports_sme()) { - vcpu_clear_flag(vcpu, HOST_SME_ENABLED); - if (read_sysreg(cpacr_el1) & CPACR_EL1_SMEN_EL0EN) - vcpu_set_flag(vcpu, HOST_SME_ENABLED); - } } /* @@ -157,23 +151,6 @@ void kvm_arch_vcpu_put_fp(struct kvm_vcpu *vcpu) local_irq_save(flags); - /* - * If we have VHE then the Hyp code will reset CPACR_EL1 to - * the default value and we need to reenable SME. - */ - if (has_vhe() && system_supports_sme()) { - /* Also restore EL0 state seen on entry */ - if (vcpu_get_flag(vcpu, HOST_SME_ENABLED)) - sysreg_clear_set(CPACR_EL1, 0, - CPACR_EL1_SMEN_EL0EN | - CPACR_EL1_SMEN_EL1EN); - else - sysreg_clear_set(CPACR_EL1, - CPACR_EL1_SMEN_EL0EN, - CPACR_EL1_SMEN_EL1EN); - isb(); - } - if (vcpu->arch.fp_state == FP_STATE_GUEST_OWNED) { if (vcpu_has_sve(vcpu)) { __vcpu_sys_reg(vcpu, ZCR_EL1) = read_sysreg_el1(SYS_ZCR); From patchwork Fri Mar 21 00:16:06 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14024740 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 29EACC28B30 for ; Fri, 21 Mar 2025 00:54:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Cc:To:In-Reply-To:References :Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=Qu3K3aoOMHxYbvOE7HsEPJHl6nOHLZP+0pULLgRS/yo=; b=2p+ewuXQpT5xjMss68C75BMeb/ /4AEl6w0myiQlUVk6lWqXlqycikhk3D9pCk25riXKrHjL2YyPSOv1DoKaQZFZ1aSUBIwb0zCCeO2n pSYStw9A/PgXC03mvwogwwkr07BwzQNIS94w+3kqcUkZrCTkiggsHDxc6GTlatNxAJGrtDsgMitZk iVp9vkEkSNvwHPF1d0JENWY33abl1Vx2GiuxBVZQYirz0zpUelEYcmbPdMLalXYIkfvrN5v1ct6VM ddaNs7D0vyjsv0+bvc1IbLKwbYoCjNRuIW6nvtf5A1ZFxaSIB2qmfC6+pznlPUhly6jkaBQeL5pse ANaJVDbA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tvQe4-0000000Db1s-1Tjk; Fri, 21 Mar 2025 00:53:56 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tvQ5q-0000000DXWT-24BI for linux-arm-kernel@lists.infradead.org; Fri, 21 Mar 2025 00:18:35 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id DC9B35C6827; Fri, 21 Mar 2025 00:16:16 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3F1BFC4CEDD; Fri, 21 Mar 2025 00:18:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1742516313; bh=8frXYbVIRlCO2TsbNo9h7TxjGLDAzQkQrF5jGasR4mo=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=P2Td6R6iGz/1KCNvOnZPvud988ysi8a6noGYICTJelKqNZeQ/kT3OH+h+uSW8OCo9 hG5XiFZ66dmSHFqn/eC9viw1sIlaWkRjeTK9+4vVFMKkRqvVqrEJhLzsUIFDB+HaC5 SvfcdiecSTCrxr94ILGICCN+tuD4w9DK14lmRBO2PZaLh6zXFir8McWMw+rbQVBER/ VvDrYoDrtpmYiZX/zRP4IlHZ/UhuP2RHc9LLohkgY8El+J/JTExDfcTtt0OqQKgu8F WyuOKKXRETPKpE+0C37oJjHaCz/+wBTl9L/FlAzpEn6CVs1bUjwZlZKaMdc/kOr75m 913DJfgw+QI3A== From: Mark Brown Date: Fri, 21 Mar 2025 00:16:06 +0000 Subject: [PATCH 6.6 6/8] KVM: arm64: Refactor exit handlers MIME-Version: 1.0 Message-Id: <20250321-stable-sve-6-6-v1-6-0b3a6a14ea53@kernel.org> References: <20250321-stable-sve-6-6-v1-0-0b3a6a14ea53@kernel.org> In-Reply-To: <20250321-stable-sve-6-6-v1-0-0b3a6a14ea53@kernel.org> To: Greg Kroah-Hartman , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Catalin Marinas , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, stable@vger.kernel.org, Mark Brown , Mark Rutland , Fuad Tabba X-Mailer: b4 0.15-dev-1b0d6 X-Developer-Signature: v=1; a=openpgp-sha256; l=7959; i=broonie@kernel.org; h=from:subject:message-id; bh=H/XezYaBHJc6oSGLeAcx2HLjyrzh+PPZu7PTkcdqfRY=; b=owGbwMvMwMWocq27KDak/QLjabUkhvQ7G+xEmTJTD9Q+MVXYyMMuIrznrsVDGc7cZL1Noc/9X0UZ FFh2MhqzMDByMciKKbKsfZaxKj1cYuv8R/NfwQxiZQKZwsDFKQAT2aHC/j9wV16G2e+wxmuOiqrbM9 c8CGx4svmgdarV5Se7P+27svYMB1/HPzdzFf38X2bP74Y3vX5k9Nmo2eaSVmV1RFTJw78VwcctGx8E bXd8r7bW8YmjjE2cOZ/edcGTne/frYz8qcGlntDVKNtokmySvHEhU7NWTnH3JysLhjLp6eI1X5uOSa 1ePK/NJWbCqxCtkNUSvaclk56VrjVi/KQRFmt4Omw5A/vvmt08e2tsDTWbDL7u8kq9u3yDcpZ9xMv9 67Nu2xjJCk36rNssGdJ+XNTrU8ucTtNte/u/X9/Yd18yX/frNgNb36N+u827ePLFRbeZz55wXt88on 9Nc89JzU+TDhyTrt1WbxvMuNQHAA== X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250320_171834_655597_4E2D3A82 X-CRM114-Status: GOOD ( 23.85 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Mark Rutland [ Upstream commit 9b66195063c5a145843547b1d692bd189be85287 ] The hyp exit handling logic is largely shared between VHE and nVHE/hVHE, with common logic in arch/arm64/kvm/hyp/include/hyp/switch.h. The code in the header depends on function definitions provided by arch/arm64/kvm/hyp/vhe/switch.c and arch/arm64/kvm/hyp/nvhe/switch.c when they include the header. This is an unusual header dependency, and prevents the use of arch/arm64/kvm/hyp/include/hyp/switch.h in other files as this would result in compiler warnings regarding missing definitions, e.g. | In file included from arch/arm64/kvm/hyp/nvhe/hyp-main.c:8: | ./arch/arm64/kvm/hyp/include/hyp/switch.h:733:31: warning: 'kvm_get_exit_handler_array' used but never defined | 733 | static const exit_handler_fn *kvm_get_exit_handler_array(struct kvm_vcpu *vcpu); | | ^~~~~~~~~~~~~~~~~~~~~~~~~~ | ./arch/arm64/kvm/hyp/include/hyp/switch.h:735:13: warning: 'early_exit_filter' used but never defined | 735 | static void early_exit_filter(struct kvm_vcpu *vcpu, u64 *exit_code); | | ^~~~~~~~~~~~~~~~~ Refactor the logic such that the header doesn't depend on anything from the C files. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland Reviewed-by: Mark Brown Tested-by: Mark Brown Acked-by: Will Deacon Cc: Catalin Marinas Cc: Fuad Tabba Cc: Marc Zyngier Cc: Oliver Upton Reviewed-by: Oliver Upton Link: https://lore.kernel.org/r/20250210195226.1215254-7-mark.rutland@arm.com Signed-off-by: Marc Zyngier Signed-off-by: Mark Brown --- arch/arm64/kvm/hyp/include/hyp/switch.h | 30 ++++++------------------------ arch/arm64/kvm/hyp/nvhe/switch.c | 27 +++++++++++++++------------ arch/arm64/kvm/hyp/vhe/switch.c | 9 ++++----- 3 files changed, 25 insertions(+), 41 deletions(-) diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h index 1cdc8d161e7dd3aeeb6b03170e239b0c1ace7790..db9570affe09d36f2fe4641789a24111303be5a2 100644 --- a/arch/arm64/kvm/hyp/include/hyp/switch.h +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h @@ -592,23 +592,16 @@ static bool kvm_hyp_handle_dabt_low(struct kvm_vcpu *vcpu, u64 *exit_code) typedef bool (*exit_handler_fn)(struct kvm_vcpu *, u64 *); -static const exit_handler_fn *kvm_get_exit_handler_array(struct kvm_vcpu *vcpu); - -static void early_exit_filter(struct kvm_vcpu *vcpu, u64 *exit_code); - /* * Allow the hypervisor to handle the exit with an exit handler if it has one. * * Returns true if the hypervisor handled the exit, and control should go back * to the guest, or false if it hasn't. */ -static inline bool kvm_hyp_handle_exit(struct kvm_vcpu *vcpu, u64 *exit_code) +static inline bool kvm_hyp_handle_exit(struct kvm_vcpu *vcpu, u64 *exit_code, + const exit_handler_fn *handlers) { - const exit_handler_fn *handlers = kvm_get_exit_handler_array(vcpu); - exit_handler_fn fn; - - fn = handlers[kvm_vcpu_trap_get_class(vcpu)]; - + exit_handler_fn fn = handlers[kvm_vcpu_trap_get_class(vcpu)]; if (fn) return fn(vcpu, exit_code); @@ -638,20 +631,9 @@ static inline void synchronize_vcpu_pstate(struct kvm_vcpu *vcpu, u64 *exit_code * the guest, false when we should restore the host state and return to the * main run loop. */ -static inline bool fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code) +static inline bool __fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code, + const exit_handler_fn *handlers) { - /* - * Save PSTATE early so that we can evaluate the vcpu mode - * early on. - */ - synchronize_vcpu_pstate(vcpu, exit_code); - - /* - * Check whether we want to repaint the state one way or - * another. - */ - early_exit_filter(vcpu, exit_code); - if (ARM_EXCEPTION_CODE(*exit_code) != ARM_EXCEPTION_IRQ) vcpu->arch.fault.esr_el2 = read_sysreg_el2(SYS_ESR); @@ -681,7 +663,7 @@ static inline bool fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code) goto exit; /* Check if there's an exit handler and allow it to handle the exit. */ - if (kvm_hyp_handle_exit(vcpu, exit_code)) + if (kvm_hyp_handle_exit(vcpu, exit_code, handlers)) goto guest; exit: /* Return to the host kernel and handle the exit */ diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/switch.c index 1026be1964d9664b8b6de19e048aba91ee6463b0..aa827e2a2b1e5b2c67304221706f406b42ceafb2 100644 --- a/arch/arm64/kvm/hyp/nvhe/switch.c +++ b/arch/arm64/kvm/hyp/nvhe/switch.c @@ -225,21 +225,22 @@ static const exit_handler_fn *kvm_get_exit_handler_array(struct kvm_vcpu *vcpu) return hyp_exit_handlers; } -/* - * Some guests (e.g., protected VMs) are not be allowed to run in AArch32. - * The ARMv8 architecture does not give the hypervisor a mechanism to prevent a - * guest from dropping to AArch32 EL0 if implemented by the CPU. If the - * hypervisor spots a guest in such a state ensure it is handled, and don't - * trust the host to spot or fix it. The check below is based on the one in - * kvm_arch_vcpu_ioctl_run(). - * - * Returns false if the guest ran in AArch32 when it shouldn't have, and - * thus should exit to the host, or true if a the guest run loop can continue. - */ -static void early_exit_filter(struct kvm_vcpu *vcpu, u64 *exit_code) +static inline bool fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code) { + const exit_handler_fn *handlers = kvm_get_exit_handler_array(vcpu); struct kvm *kvm = kern_hyp_va(vcpu->kvm); + synchronize_vcpu_pstate(vcpu, exit_code); + + /* + * Some guests (e.g., protected VMs) are not be allowed to run in + * AArch32. The ARMv8 architecture does not give the hypervisor a + * mechanism to prevent a guest from dropping to AArch32 EL0 if + * implemented by the CPU. If the hypervisor spots a guest in such a + * state ensure it is handled, and don't trust the host to spot or fix + * it. The check below is based on the one in + * kvm_arch_vcpu_ioctl_run(). + */ if (kvm_vm_is_protected(kvm) && vcpu_mode_is_32bit(vcpu)) { /* * As we have caught the guest red-handed, decide that it isn't @@ -252,6 +253,8 @@ static void early_exit_filter(struct kvm_vcpu *vcpu, u64 *exit_code) *exit_code &= BIT(ARM_EXIT_WITH_SERROR_BIT); *exit_code |= ARM_EXCEPTION_IL; } + + return __fixup_guest_exit(vcpu, exit_code, handlers); } /* Switch to the guest for legacy non-VHE systems */ diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c index 448b17080d3617cbf5d2fceec7e7fa62866760d4..d0c7725879a7b38cea53f1dfb111933e0ad89845 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -172,13 +172,10 @@ static const exit_handler_fn hyp_exit_handlers[] = { [ESR_ELx_EC_PAC] = kvm_hyp_handle_ptrauth, }; -static const exit_handler_fn *kvm_get_exit_handler_array(struct kvm_vcpu *vcpu) +static inline bool fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code) { - return hyp_exit_handlers; -} + synchronize_vcpu_pstate(vcpu, exit_code); -static void early_exit_filter(struct kvm_vcpu *vcpu, u64 *exit_code) -{ /* * If we were in HYP context on entry, adjust the PSTATE view * so that the usual helpers work correctly. @@ -198,6 +195,8 @@ static void early_exit_filter(struct kvm_vcpu *vcpu, u64 *exit_code) *vcpu_cpsr(vcpu) &= ~(PSR_MODE_MASK | PSR_MODE32_BIT); *vcpu_cpsr(vcpu) |= mode; } + + return __fixup_guest_exit(vcpu, exit_code, hyp_exit_handlers); } /* Switch to the guest for VHE systems running in EL2 */ From patchwork Fri Mar 21 00:16:07 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14024741 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3EA34C28B30 for ; Fri, 21 Mar 2025 00:55:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Cc:To:In-Reply-To:References :Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=RzhGsPmMkoyyLSBcn1SYYHCdC3LsTsOD6qaVBDV9/48=; b=TlFNxdhpmsLBGpBNhDWbp5rIOL rK0cN0GYVV73Z1Dn3MaDBp9JBIavtheOih1GggBspFGTwQK2pOuBMknySfYA0MWGTahlBYXke0sAk sqLsFQpcODBp86lIlG+lllq2DnmG9MdB3KhLq+YKdydYI7NBOQ+tFLUk7gVXAnW7FQJuRYAApF7h+ E3y6ZEEs/RfbTFxrfgAm14YUTDcfrd+FAM3kEDGj5Y3rROp1c2oDZzvefrVzsQkjxnUlZIiUjJ27j PGVa08YyhFaw8QBfx8xtpV+8k45tBz4/6SJiE1CKCGJR3rKqneU/zwBzxZuL+7VdFKPantEccRZR0 +x/M+ifg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tvQfk-0000000Db94-0DM9; Fri, 21 Mar 2025 00:55:40 +0000 Received: from sea.source.kernel.org ([172.234.252.31]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tvQ5s-0000000DXWu-48I0 for linux-arm-kernel@lists.infradead.org; Fri, 21 Mar 2025 00:18:39 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sea.source.kernel.org (Postfix) with ESMTP id 362FD44CC7; Fri, 21 Mar 2025 00:18:36 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1D0D4C4CEDD; Fri, 21 Mar 2025 00:18:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1742516316; bh=jkTFzDWDYNdcP+4j918lQr5dcKScYFxg9md8oDLcIV0=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=tcFiHLMSwlFR1YPntJxIBqMvBs3M5VwWJG/9gCAuYhQ+VgQ1XqjBKHz4c1CdQxJSa cXatYzPJpRGmhxL7m+l4F2xRiKeOYIRvQXMuxdB9PxqYpY/29iRDuEC29rff9zgu5E myAkofQlPQhCCoFL3rP0Yue41Gz0l8fasVLte7PFWoM5z36S3La+GddrGVrIsFECAv ZTYDcRnF/XmWs1LE/3f8JhJZ5Xt9dwwk/9fex0/eGUh/x3MYfeX9zYilmePGaYxap9 r2KUA/plBe5ASzeoiWSAj7RQ77mbwP0SgdqeWplmurxkvdBXVnh8yLCFnw4P7JPcMC qoxxEyej7X/bw== From: Mark Brown Date: Fri, 21 Mar 2025 00:16:07 +0000 Subject: [PATCH 6.6 7/8] KVM: arm64: Mark some header functions as inline MIME-Version: 1.0 Message-Id: <20250321-stable-sve-6-6-v1-7-0b3a6a14ea53@kernel.org> References: <20250321-stable-sve-6-6-v1-0-0b3a6a14ea53@kernel.org> In-Reply-To: <20250321-stable-sve-6-6-v1-0-0b3a6a14ea53@kernel.org> To: Greg Kroah-Hartman , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Catalin Marinas , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, stable@vger.kernel.org, Mark Brown , Mark Rutland , Fuad Tabba X-Mailer: b4 0.15-dev-1b0d6 X-Developer-Signature: v=1; a=openpgp-sha256; l=5173; i=broonie@kernel.org; h=from:subject:message-id; bh=SZD6jQv12jzrN1l/A+Uen0+19apJEMW5/7iJNyxb5VM=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBn3LA/eDwoCjG4uKo35H5aQB6mTfciqgmzrNXa55fF u9ILdJOJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCZ9ywPwAKCRAk1otyXVSH0KUvB/ 93p92Nec24IXz9UTXJLF+M8/HjbShk81FcPrxb9mg17aqfRKD56u1CFdiKqv2AoP53SuSCe/OOzn5z OPDdc8OQG6XzhjKtXFVu/+uPmRTO8LQrsQfP/VnZPltxIl7GIsqW5eF/HBmN3WaD5WBd9XdU1Zr6zI yPu2vvp19mtcl48k/eKMF7LIq44LClLB0H0UtNLY6W+fLXPTKXr2iq5OpEZhZ8uAxLJEc43SFi5BNv uxGg4TVOfUadyWkXUSjLEFg6pKQS4XSqXgo9OCr2hBLIlVgQCOSIeiDK6d7GnqqLAVLr/NcTiPDVSo Jwb5CeU/7IBlboYhcdmHC/rO8AOEIz X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250320_171837_072137_C52A48F5 X-CRM114-Status: GOOD ( 13.20 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Mark Rutland [ Upstream commit f9dd00de1e53a47763dfad601635d18542c3836d ] The shared hyp switch header has a number of static functions which might not be used by all files that include the header, and when unused they will provoke compiler warnings, e.g. | In file included from arch/arm64/kvm/hyp/nvhe/hyp-main.c:8: | ./arch/arm64/kvm/hyp/include/hyp/switch.h:703:13: warning: 'kvm_hyp_handle_dabt_low' defined but not used [-Wunused-function] | 703 | static bool kvm_hyp_handle_dabt_low(struct kvm_vcpu *vcpu, u64 *exit_code) | | ^~~~~~~~~~~~~~~~~~~~~~~ | ./arch/arm64/kvm/hyp/include/hyp/switch.h:682:13: warning: 'kvm_hyp_handle_cp15_32' defined but not used [-Wunused-function] | 682 | static bool kvm_hyp_handle_cp15_32(struct kvm_vcpu *vcpu, u64 *exit_code) | | ^~~~~~~~~~~~~~~~~~~~~~ | ./arch/arm64/kvm/hyp/include/hyp/switch.h:662:13: warning: 'kvm_hyp_handle_sysreg' defined but not used [-Wunused-function] | 662 | static bool kvm_hyp_handle_sysreg(struct kvm_vcpu *vcpu, u64 *exit_code) | | ^~~~~~~~~~~~~~~~~~~~~ | ./arch/arm64/kvm/hyp/include/hyp/switch.h:458:13: warning: 'kvm_hyp_handle_fpsimd' defined but not used [-Wunused-function] | 458 | static bool kvm_hyp_handle_fpsimd(struct kvm_vcpu *vcpu, u64 *exit_code) | | ^~~~~~~~~~~~~~~~~~~~~ | ./arch/arm64/kvm/hyp/include/hyp/switch.h:329:13: warning: 'kvm_hyp_handle_mops' defined but not used [-Wunused-function] | 329 | static bool kvm_hyp_handle_mops(struct kvm_vcpu *vcpu, u64 *exit_code) | | ^~~~~~~~~~~~~~~~~~~ Mark these functions as 'inline' to suppress this warning. This shouldn't result in any functional change. At the same time, avoid the use of __alias() in the header and alias kvm_hyp_handle_iabt_low() and kvm_hyp_handle_watchpt_low() to kvm_hyp_handle_memory_fault() using CPP, matching the style in the rest of the kernel. For consistency, kvm_hyp_handle_memory_fault() is also marked as 'inline'. Signed-off-by: Mark Rutland Reviewed-by: Mark Brown Tested-by: Mark Brown Acked-by: Will Deacon Cc: Catalin Marinas Cc: Fuad Tabba Cc: Marc Zyngier Cc: Oliver Upton Reviewed-by: Oliver Upton Link: https://lore.kernel.org/r/20250210195226.1215254-8-mark.rutland@arm.com Signed-off-by: Marc Zyngier Signed-off-by: Mark Brown --- arch/arm64/kvm/hyp/include/hyp/switch.h | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h index db9570affe09d36f2fe4641789a24111303be5a2..77a9f68ca4904a2b39ca59b7ee7cf54b9fe11f66 100644 --- a/arch/arm64/kvm/hyp/include/hyp/switch.h +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h @@ -279,7 +279,7 @@ static inline void __hyp_sve_restore_guest(struct kvm_vcpu *vcpu) * If FP/SIMD is not implemented, handle the trap and inject an undefined * instruction exception to the guest. Similarly for trapped SVE accesses. */ -static bool kvm_hyp_handle_fpsimd(struct kvm_vcpu *vcpu, u64 *exit_code) +static inline bool kvm_hyp_handle_fpsimd(struct kvm_vcpu *vcpu, u64 *exit_code) { bool sve_guest; u8 esr_ec; @@ -518,7 +518,7 @@ static bool handle_ampere1_tcr(struct kvm_vcpu *vcpu) return true; } -static bool kvm_hyp_handle_sysreg(struct kvm_vcpu *vcpu, u64 *exit_code) +static inline bool kvm_hyp_handle_sysreg(struct kvm_vcpu *vcpu, u64 *exit_code) { if (cpus_have_final_cap(ARM64_WORKAROUND_CAVIUM_TX2_219_TVM) && handle_tx2_tvm(vcpu)) @@ -541,7 +541,7 @@ static bool kvm_hyp_handle_sysreg(struct kvm_vcpu *vcpu, u64 *exit_code) return false; } -static bool kvm_hyp_handle_cp15_32(struct kvm_vcpu *vcpu, u64 *exit_code) +static inline bool kvm_hyp_handle_cp15_32(struct kvm_vcpu *vcpu, u64 *exit_code) { if (static_branch_unlikely(&vgic_v3_cpuif_trap) && __vgic_v3_perform_cpuif_access(vcpu) == 1) @@ -550,19 +550,18 @@ static bool kvm_hyp_handle_cp15_32(struct kvm_vcpu *vcpu, u64 *exit_code) return false; } -static bool kvm_hyp_handle_memory_fault(struct kvm_vcpu *vcpu, u64 *exit_code) +static inline bool kvm_hyp_handle_memory_fault(struct kvm_vcpu *vcpu, + u64 *exit_code) { if (!__populate_fault_info(vcpu)) return true; return false; } -static bool kvm_hyp_handle_iabt_low(struct kvm_vcpu *vcpu, u64 *exit_code) - __alias(kvm_hyp_handle_memory_fault); -static bool kvm_hyp_handle_watchpt_low(struct kvm_vcpu *vcpu, u64 *exit_code) - __alias(kvm_hyp_handle_memory_fault); +#define kvm_hyp_handle_iabt_low kvm_hyp_handle_memory_fault +#define kvm_hyp_handle_watchpt_low kvm_hyp_handle_memory_fault -static bool kvm_hyp_handle_dabt_low(struct kvm_vcpu *vcpu, u64 *exit_code) +static inline bool kvm_hyp_handle_dabt_low(struct kvm_vcpu *vcpu, u64 *exit_code) { if (kvm_hyp_handle_memory_fault(vcpu, exit_code)) return true; From patchwork Fri Mar 21 00:16:08 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14024743 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5AF27C28B30 for ; Fri, 21 Mar 2025 00:57:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Cc:To:In-Reply-To:References :Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=pHJlXVeaQbbaoFzj3i/OBzkc7qFYz9JDpig/eJDoCT8=; b=2xtPyXaII8lCJylxG5xdcS0vud J9f1TMoL38vj/AS5KuXRqjFud8svUQVq3BBvYmmFG6pXQZ0Mv8y22oK7SEaONlPZxpoJV4sT/CxLM 9wlZCW4BuEFzD6TJKXmQRByuOFD9uKytuxOKWqH54HgKToODDGcKemytimzE7cK1PiiyUloWzoc95 XhAU9eEaa/lvcP6BT6+sMP53Fn5o0pyx61IedUEEmsoM/DUNPtgDhvET6pHtXLp+A4hdX7OqZysla 5OstXtliMIlDOse9BPhY/AtVOvvOpMWlDEZkWoqq2pdiWCaY7t9rhqp4kCJXai2nvdJ6u4wT0apGl WHXZ6LAA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tvQhP-0000000DbFe-35A0; Fri, 21 Mar 2025 00:57:23 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tvQ5w-0000000DXXO-1roI for linux-arm-kernel@lists.infradead.org; Fri, 21 Mar 2025 00:18:44 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id C1D875C6824; Fri, 21 Mar 2025 00:16:22 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id EB4E6C4CEE3; Fri, 21 Mar 2025 00:18:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1742516319; bh=PnRpM4598mZWrSme/MU6XEBMxzI66FdoajFyiek5haI=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=IJSrEVl7Eadm949SG6lFUOtQc71zNGKHSMMv0xLNwkmJyLVOD+8vcXd+QOSr8zWUV YRhumlNxdbjay7uCwCfM0AcryYdvruP259ZlaIi+NaNYGNcEEM8MJHMr2gagxulNRE m3N5Fg7PQAgZzoECgnWM1P5Q2seOVZmFfnZpIsGxZrBSa/Ag1z1rZJ3abNTv3aLv2E wYLTA88nY2EVEMWigKAw8cMIZnKGnFdpfpQR1gcQIYh4+mz/W8vIQ+JpcYRq8x3z9y Hloc0J80vcUo+a4yK0bF5XD3HPjLiLBx287nbU6qlhFQDO80lDRZxtXcNKRguXKm+d joR/X19xwdnxw== From: Mark Brown Date: Fri, 21 Mar 2025 00:16:08 +0000 Subject: [PATCH 6.6 8/8] KVM: arm64: Eagerly switch ZCR_EL{1,2} MIME-Version: 1.0 Message-Id: <20250321-stable-sve-6-6-v1-8-0b3a6a14ea53@kernel.org> References: <20250321-stable-sve-6-6-v1-0-0b3a6a14ea53@kernel.org> In-Reply-To: <20250321-stable-sve-6-6-v1-0-0b3a6a14ea53@kernel.org> To: Greg Kroah-Hartman , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Catalin Marinas , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, stable@vger.kernel.org, Mark Brown , Mark Rutland , Fuad Tabba X-Mailer: b4 0.15-dev-1b0d6 X-Developer-Signature: v=1; a=openpgp-sha256; l=13672; i=broonie@kernel.org; h=from:subject:message-id; bh=VtC0IuN84WFW7O11FEgk7OJFCF+Z2vAa4/lai70a8gw=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBn3LBA/PgCMZ06kuRd90l7zeGmb9dCKRudEu6HZl/F oor/8HKJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCZ9ywQAAKCRAk1otyXVSH0JAFB/ 9jkilsimYJVTOzLt7DE8tseKfozUG0s6vQDFT252oEhZ1pxWNwucPeNY/DhrdWYfCGswm6NI5reN3S DVvmMFOsmbBQUxPH6M/UZTeCpArkOVG4kpph24elpftXrKgnXgpdabYnDFlLRHmGBKWi1fVWehSdQx 2mn5KglebLGV12KMgtm9exK0A0nVWKOgTqwqXfNkm0oA/nOQcZlHKwTkglPgITsm/U4Q9YhnfUWdcz sJx1PvatrQY2AAdY6I25vWV9YuyK4l8C9gJ4+amDQNLr9G0o9pEzaHp+w0ouY16zVSgK5qgxMvIilu 0SzcI9fqIhrl18nAiGjTI3Tz4XoJxE X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250320_171840_589471_733C47C5 X-CRM114-Status: GOOD ( 32.15 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Mark Rutland [ Upstream commit 59419f10045bc955d2229819c7cf7a8b0b9c5b59 ] In non-protected KVM modes, while the guest FPSIMD/SVE/SME state is live on the CPU, the host's active SVE VL may differ from the guest's maximum SVE VL: * For VHE hosts, when a VM uses NV, ZCR_EL2 contains a value constrained by the guest hypervisor, which may be less than or equal to that guest's maximum VL. Note: in this case the value of ZCR_EL1 is immaterial due to E2H. * For nVHE/hVHE hosts, ZCR_EL1 contains a value written by the guest, which may be less than or greater than the guest's maximum VL. Note: in this case hyp code traps host SVE usage and lazily restores ZCR_EL2 to the host's maximum VL, which may be greater than the guest's maximum VL. This can be the case between exiting a guest and kvm_arch_vcpu_put_fp(). If a softirq is taken during this period and the softirq handler tries to use kernel-mode NEON, then the kernel will fail to save the guest's FPSIMD/SVE state, and will pend a SIGKILL for the current thread. This happens because kvm_arch_vcpu_ctxsync_fp() binds the guest's live FPSIMD/SVE state with the guest's maximum SVE VL, and fpsimd_save_user_state() verifies that the live SVE VL is as expected before attempting to save the register state: | if (WARN_ON(sve_get_vl() != vl)) { | force_signal_inject(SIGKILL, SI_KERNEL, 0, 0); | return; | } Fix this and make this a bit easier to reason about by always eagerly switching ZCR_EL{1,2} at hyp during guest<->host transitions. With this happening, there's no need to trap host SVE usage, and the nVHE/nVHE __deactivate_cptr_traps() logic can be simplified to enable host access to all present FPSIMD/SVE/SME features. In protected nVHE/hVHE modes, the host's state is always saved/restored by hyp, and the guest's state is saved prior to exit to the host, so from the host's PoV the guest never has live FPSIMD/SVE/SME state, and the host's ZCR_EL1 is never clobbered by hyp. Fixes: 8c8010d69c132273 ("KVM: arm64: Save/restore SVE state for nVHE") Fixes: 2e3cf82063a00ea0 ("KVM: arm64: nv: Ensure correct VL is loaded before saving SVE state") Signed-off-by: Mark Rutland Reviewed-by: Mark Brown Tested-by: Mark Brown Cc: Catalin Marinas Cc: Fuad Tabba Cc: Marc Zyngier Cc: Oliver Upton Cc: Will Deacon Reviewed-by: Oliver Upton Link: https://lore.kernel.org/r/20250210195226.1215254-9-mark.rutland@arm.com Signed-off-by: Marc Zyngier [ v6.6 lacks pKVM saving of host SVE state, pull in discovery of maximum host VL separately -- broonie ] Signed-off-by: Mark Brown --- arch/arm64/include/asm/kvm_host.h | 1 + arch/arm64/include/asm/kvm_hyp.h | 1 + arch/arm64/kvm/fpsimd.c | 19 ++++++------ arch/arm64/kvm/hyp/entry.S | 5 +++ arch/arm64/kvm/hyp/include/hyp/switch.h | 55 +++++++++++++++++++++++++++++++++ arch/arm64/kvm/hyp/nvhe/hyp-main.c | 12 ++----- arch/arm64/kvm/hyp/nvhe/pkvm.c | 2 ++ arch/arm64/kvm/hyp/nvhe/switch.c | 33 +++++++++++++++++--- arch/arm64/kvm/hyp/vhe/switch.c | 4 +++ arch/arm64/kvm/reset.c | 3 ++ 10 files changed, 113 insertions(+), 22 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 27c53b603d693d1cfe8d81f94a59d3afb665be8e..df73e452d2cc0e5fd815d398a932763a5aa7607a 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -73,6 +73,7 @@ static inline enum kvm_mode kvm_get_mode(void) { return KVM_MODE_NONE; }; #endif extern unsigned int __ro_after_init kvm_sve_max_vl; +extern unsigned int __ro_after_init kvm_host_sve_max_vl; int __init kvm_arm_init_sve(void); u32 __attribute_const__ kvm_target_cpu(void); diff --git a/arch/arm64/include/asm/kvm_hyp.h b/arch/arm64/include/asm/kvm_hyp.h index 66efd67ea7e8d6dd98dc6088a6004fdb646df2a3..51cd106cd840a006264d9a316f2dc6332e8327ad 100644 --- a/arch/arm64/include/asm/kvm_hyp.h +++ b/arch/arm64/include/asm/kvm_hyp.h @@ -145,5 +145,6 @@ extern u64 kvm_nvhe_sym(id_aa64smfr0_el1_sys_val); extern unsigned long kvm_nvhe_sym(__icache_flags); extern unsigned int kvm_nvhe_sym(kvm_arm_vmid_bits); +extern unsigned int kvm_nvhe_sym(kvm_host_sve_max_vl); #endif /* __ARM64_KVM_HYP_H__ */ diff --git a/arch/arm64/kvm/fpsimd.c b/arch/arm64/kvm/fpsimd.c index 0492a1fc718130934f8bada836fe42bea6db7abb..e57db49ea468e41e411fd896321c3b1b87a098ec 100644 --- a/arch/arm64/kvm/fpsimd.c +++ b/arch/arm64/kvm/fpsimd.c @@ -152,15 +152,16 @@ void kvm_arch_vcpu_put_fp(struct kvm_vcpu *vcpu) local_irq_save(flags); if (vcpu->arch.fp_state == FP_STATE_GUEST_OWNED) { - if (vcpu_has_sve(vcpu)) { - __vcpu_sys_reg(vcpu, ZCR_EL1) = read_sysreg_el1(SYS_ZCR); - - /* Restore the VL that was saved when bound to the CPU */ - if (!has_vhe()) - sve_cond_update_zcr_vq(vcpu_sve_max_vq(vcpu) - 1, - SYS_ZCR_EL1); - } - + /* + * Flush (save and invalidate) the fpsimd/sve state so that if + * the host tries to use fpsimd/sve, it's not using stale data + * from the guest. + * + * Flushing the state sets the TIF_FOREIGN_FPSTATE bit for the + * context unconditionally, in both nVHE and VHE. This allows + * the kernel to restore the fpsimd/sve state, including ZCR_EL1 + * when needed. + */ fpsimd_save_and_flush_cpu_state(); } diff --git a/arch/arm64/kvm/hyp/entry.S b/arch/arm64/kvm/hyp/entry.S index f3aa7738b477d6a83e41db42cc161fc6f9aa6291..f02d5701fc21c685122aa5ba92309f7f4def3ae3 100644 --- a/arch/arm64/kvm/hyp/entry.S +++ b/arch/arm64/kvm/hyp/entry.S @@ -44,6 +44,11 @@ alternative_if ARM64_HAS_RAS_EXTN alternative_else_nop_endif mrs x1, isr_el1 cbz x1, 1f + + // Ensure that __guest_enter() always provides a context + // synchronization event so that callers don't need ISBs for anything + // that would usually be synchonized by the ERET. + isb mov x0, #ARM_EXCEPTION_IRQ ret diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h index 77a9f68ca4904a2b39ca59b7ee7cf54b9fe11f66..526085401f66387137a0d78d1d520af8b2306e43 100644 --- a/arch/arm64/kvm/hyp/include/hyp/switch.h +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h @@ -273,6 +273,61 @@ static inline void __hyp_sve_restore_guest(struct kvm_vcpu *vcpu) write_sysreg_el1(__vcpu_sys_reg(vcpu, ZCR_EL1), SYS_ZCR); } +static inline void fpsimd_lazy_switch_to_guest(struct kvm_vcpu *vcpu) +{ + u64 zcr_el1, zcr_el2; + + if (!guest_owns_fp_regs(vcpu)) + return; + + if (vcpu_has_sve(vcpu)) { + zcr_el2 = vcpu_sve_max_vq(vcpu) - 1; + + write_sysreg_el2(zcr_el2, SYS_ZCR); + + zcr_el1 = __vcpu_sys_reg(vcpu, ZCR_EL1); + write_sysreg_el1(zcr_el1, SYS_ZCR); + } +} + +static inline void fpsimd_lazy_switch_to_host(struct kvm_vcpu *vcpu) +{ + u64 zcr_el1, zcr_el2; + + if (!guest_owns_fp_regs(vcpu)) + return; + + /* + * When the guest owns the FP regs, we know that guest+hyp traps for + * any FPSIMD/SVE/SME features exposed to the guest have been disabled + * by either fpsimd_lazy_switch_to_guest() or kvm_hyp_handle_fpsimd() + * prior to __guest_entry(). As __guest_entry() guarantees a context + * synchronization event, we don't need an ISB here to avoid taking + * traps for anything that was exposed to the guest. + */ + if (vcpu_has_sve(vcpu)) { + zcr_el1 = read_sysreg_el1(SYS_ZCR); + __vcpu_sys_reg(vcpu, ZCR_EL1) = zcr_el1; + + /* + * The guest's state is always saved using the guest's max VL. + * Ensure that the host has the guest's max VL active such that + * the host can save the guest's state lazily, but don't + * artificially restrict the host to the guest's max VL. + */ + if (has_vhe()) { + zcr_el2 = vcpu_sve_max_vq(vcpu) - 1; + write_sysreg_el2(zcr_el2, SYS_ZCR); + } else { + zcr_el2 = sve_vq_from_vl(kvm_host_sve_max_vl) - 1; + write_sysreg_el2(zcr_el2, SYS_ZCR); + + zcr_el1 = vcpu_sve_max_vq(vcpu) - 1; + write_sysreg_el1(zcr_el1, SYS_ZCR); + } + } +} + /* * We trap the first access to the FP/SIMD to save the host context and * restore the guest context lazily. diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c index 8390173f7f5b38e01c04ae6563853df11a1c3df2..350d1775a5ce880ac23bb7adddbed644e45d0e40 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c +++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c @@ -5,6 +5,7 @@ */ #include +#include #include #include @@ -95,7 +96,9 @@ static void handle___kvm_vcpu_run(struct kvm_cpu_context *host_ctxt) pkvm_put_hyp_vcpu(hyp_vcpu); } else { /* The host is fully trusted, run its vCPU directly. */ + fpsimd_lazy_switch_to_guest(host_vcpu); ret = __kvm_vcpu_run(host_vcpu); + fpsimd_lazy_switch_to_host(host_vcpu); } out: @@ -416,15 +419,6 @@ void handle_trap(struct kvm_cpu_context *host_ctxt) case ESR_ELx_EC_SMC64: handle_host_smc(host_ctxt); break; - case ESR_ELx_EC_SVE: - if (has_hvhe()) - sysreg_clear_set(cpacr_el1, 0, (CPACR_EL1_ZEN_EL1EN | - CPACR_EL1_ZEN_EL0EN)); - else - sysreg_clear_set(cptr_el2, CPTR_EL2_TZ, 0); - isb(); - sve_cond_update_zcr_vq(ZCR_ELx_LEN_MASK, SYS_ZCR_EL2); - break; case ESR_ELx_EC_IABT_LOW: case ESR_ELx_EC_DABT_LOW: handle_host_mem_abort(host_ctxt); diff --git a/arch/arm64/kvm/hyp/nvhe/pkvm.c b/arch/arm64/kvm/hyp/nvhe/pkvm.c index 9e7612343ad66d3dc13b8ee858a4e61e917901c9..03acc8343c5d1b23ce0ea1fa2fcabaf2a93902c5 100644 --- a/arch/arm64/kvm/hyp/nvhe/pkvm.c +++ b/arch/arm64/kvm/hyp/nvhe/pkvm.c @@ -18,6 +18,8 @@ unsigned long __icache_flags; /* Used by kvm_get_vttbr(). */ unsigned int kvm_arm_vmid_bits; +unsigned int kvm_host_sve_max_vl; + /* * Set trap register values based on features in ID_AA64PFR0. */ diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/switch.c index aa827e2a2b1e5b2c67304221706f406b42ceafb2..27ad6933bec003d722d3d66519b6225cd207cad7 100644 --- a/arch/arm64/kvm/hyp/nvhe/switch.c +++ b/arch/arm64/kvm/hyp/nvhe/switch.c @@ -40,6 +40,9 @@ static void __activate_cptr_traps(struct kvm_vcpu *vcpu) { u64 val = CPTR_EL2_TAM; /* Same bit irrespective of E2H */ + if (!guest_owns_fp_regs(vcpu)) + __activate_traps_fpsimd32(vcpu); + if (has_hvhe()) { val |= CPACR_ELx_TTA; @@ -48,6 +51,8 @@ static void __activate_cptr_traps(struct kvm_vcpu *vcpu) if (vcpu_has_sve(vcpu)) val |= CPACR_ELx_ZEN; } + + write_sysreg(val, cpacr_el1); } else { val |= CPTR_EL2_TTA | CPTR_NVHE_EL2_RES1; @@ -62,12 +67,32 @@ static void __activate_cptr_traps(struct kvm_vcpu *vcpu) if (!guest_owns_fp_regs(vcpu)) val |= CPTR_EL2_TFP; + + write_sysreg(val, cptr_el2); } +} - if (!guest_owns_fp_regs(vcpu)) - __activate_traps_fpsimd32(vcpu); +static void __deactivate_cptr_traps(struct kvm_vcpu *vcpu) +{ + if (has_hvhe()) { + u64 val = CPACR_ELx_FPEN; + + if (cpus_have_final_cap(ARM64_SVE)) + val |= CPACR_ELx_ZEN; + if (cpus_have_final_cap(ARM64_SME)) + val |= CPACR_ELx_SMEN; + + write_sysreg(val, cpacr_el1); + } else { + u64 val = CPTR_NVHE_EL2_RES1; + + if (!cpus_have_final_cap(ARM64_SVE)) + val |= CPTR_EL2_TZ; + if (!cpus_have_final_cap(ARM64_SME)) + val |= CPTR_EL2_TSM; - kvm_write_cptr_el2(val); + write_sysreg(val, cptr_el2); + } } static void __activate_traps(struct kvm_vcpu *vcpu) @@ -120,7 +145,7 @@ static void __deactivate_traps(struct kvm_vcpu *vcpu) write_sysreg(this_cpu_ptr(&kvm_init_params)->hcr_el2, hcr_el2); - kvm_reset_cptr_el2(vcpu); + __deactivate_cptr_traps(vcpu); write_sysreg(__kvm_hyp_host_vector, vbar_el2); } diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c index d0c7725879a7b38cea53f1dfb111933e0ad89845..31304da0d617313244caff582853965a7a206f72 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -212,6 +212,8 @@ static int __kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu) sysreg_save_host_state_vhe(host_ctxt); + fpsimd_lazy_switch_to_guest(vcpu); + /* * ARM erratum 1165522 requires us to configure both stage 1 and * stage 2 translation for the guest context before we clear @@ -247,6 +249,8 @@ static int __kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu) __deactivate_traps(vcpu); + fpsimd_lazy_switch_to_host(vcpu); + sysreg_restore_host_state_vhe(host_ctxt); if (vcpu->arch.fp_state == FP_STATE_GUEST_OWNED) diff --git a/arch/arm64/kvm/reset.c b/arch/arm64/kvm/reset.c index 7a65a35ee4ac4165b4147c724d28b59f8150a0d2..43a53a403f5102c81bba0d928c0f78331f800269 100644 --- a/arch/arm64/kvm/reset.c +++ b/arch/arm64/kvm/reset.c @@ -46,11 +46,14 @@ static u32 __ro_after_init kvm_ipa_limit; PSR_AA32_I_BIT | PSR_AA32_F_BIT) unsigned int __ro_after_init kvm_sve_max_vl; +unsigned int __ro_after_init kvm_host_sve_max_vl; int __init kvm_arm_init_sve(void) { if (system_supports_sve()) { kvm_sve_max_vl = sve_max_virtualisable_vl(); + kvm_host_sve_max_vl = sve_max_vl(); + kvm_nvhe_sym(kvm_host_sve_max_vl) = kvm_host_sve_max_vl; /* * The get_sve_reg()/set_sve_reg() ioctl interface will need