From patchwork Tue Apr 8 18:09:56 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14043532 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 927E6C369A1 for ; Tue, 8 Apr 2025 18:26:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Cc:To:In-Reply-To:References :Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=j7YQrHygwgkPoJf/MfgSb7sVcYw33GA+KxOxJM2BSjo=; b=zgpboTJx/MIgE334tH/qB54LbL aklJD/6ZJtHRNvIouv0YkC6UTAVDVvjR5AqaOR5tMlSURHqJTU7HjP++LHHaeH1feUITFvTiHQBh2 T/co/ie/cuxIWPB5m6nnSxAvZYPnZgnjv30pzWEBc950EQXtg/wJ4axniLmg39YCYfls1uRLwLZx4 KyT4tCyuHMbeUUKBmeZD7gZ3wuP+52a6cUjRqff4WF0y8+pFYVx/eN6v4V4Wbdo1lALSGa6OtoV+9 SZFFaXNmtgb7+q2jBcXdaA3XJLR5fW14/jB1HFePddr8A4is5GnUuvJHyEVfU87SavZrzQXp2k7Da JTPHoHbQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u2De0-000000056Te-3IuI; Tue, 08 Apr 2025 18:25:56 +0000 Received: from sea.source.kernel.org ([172.234.252.31]) by bombadil.infradead.org with esmtps (Exim 4.98.2 #2 (Red Hat Linux)) id 1u2DaW-000000055r6-3NI7 for linux-arm-kernel@lists.infradead.org; Tue, 08 Apr 2025 18:22:21 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sea.source.kernel.org (Postfix) with ESMTP id 8DE7C44B0D; Tue, 8 Apr 2025 18:22:19 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A86E8C4CEEC; Tue, 8 Apr 2025 18:22:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744136540; bh=m/dwRDdQP4Th7ONfZTRuvj+AHHMjMEezB28FzQTPJTg=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=cleALQBSbh8B/Zhg9RszuxECxGyu7NNBZ/1SMIsvKNIAwnBCf19AdEUwVUlClXO4N 5z0zxe9KbIpBeGMlHNiJwmNDpv8syOFUu0wEfzPSruDlZyTx6PW5oRmFIh/QHBctmO JUrXu++9hOJoxPrtd297kB4nJ5QllVn79RcZHImwO1yIlPt9NV3I8ihiJjXIVDbcWd n3wrYsEzPMdMzAKJzoUBtcRHhAN9UhK4IddE0qHshd6EuFDEFADPaULxuBuhv5+WI0 m+2vEI/So2YVEnFKssD4ZUdyVeEnWoJDxLlY0V+cp91HGt+i06CY2q+wS7+92qrPOX IXO0G8Lr1tFtA== From: Mark Brown Date: Tue, 08 Apr 2025 19:09:56 +0100 Subject: [PATCH 5.15 v3 01/11] KVM: arm64: Get rid of host SVE tracking/saving MIME-Version: 1.0 Message-Id: <20250408-stable-sve-5-15-v3-1-ca9a6b850f55@kernel.org> References: <20250408-stable-sve-5-15-v3-0-ca9a6b850f55@kernel.org> In-Reply-To: <20250408-stable-sve-5-15-v3-0-ca9a6b850f55@kernel.org> To: Greg Kroah-Hartman , Marc Zyngier , James Morse , Suzuki K Poulose , Catalin Marinas , Will Deacon , Oleg Nesterov , Oliver Upton Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-kernel@vger.kernel.org, stable@vger.kernel.org, Mark Brown X-Mailer: b4 0.15-dev-c25d1 X-Developer-Signature: v=1; a=openpgp-sha256; l=6329; i=broonie@kernel.org; h=from:subject:message-id; bh=O3k+gOgTUGiM5t6hYHLkd+5jlCwRM5xVq/AQhFHUOBY=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBn9WlM3QOnn7g6n/M/+7XLgjdx0mDw5QJUdy6HYhQ8 9+RXF/iJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCZ/VpTAAKCRAk1otyXVSH0MIHB/ 9j4L5NDGCIuTCQOnPjBwi+ZqVWmTsxPIaDxgKk/GWQK4VlaUY8MiBsYMOEaJbeqn0CKpgj6sHeuOHL Cr5rRKxOqiFvkpCr6Gsbj161kFgpuqJHdJxOhF/eXCtWmehvRHPxNupiboxtGDuo/xl1iljy9NTGpm oI21i6QG4i1Q0U5N6gEaZP+84MOISavam5wjq+iWOOGwmZj6mVLxgkk/8nL2/lISXXGAihXvJYwBQF f9T9YmMAL1tD4Kz2+nP2qz9wPtAWa+FokhT9FAKhkj4eyLqAsvOnOZhUvF4FjRRnI2pbhI2w11cWbI iT+fHu+RIgJ732RhJAHr6vGvdSzE0T X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250408_112220_895193_8B29A964 X-CRM114-Status: GOOD ( 23.04 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Marc Zyngier [ Upstream commit 8383741ab2e773a992f1f0f8acdca5e7a4687c49 ] The SVE host tracking in KVM is pretty involved. It relies on a set of flags tracking the ownership of the SVE register, as well as that of the EL0 access. It is also pretty scary: __hyp_sve_save_host() computes a thread_struct pointer and obtains a sve_state which gets directly accessed without further ado, even on nVHE. How can this even work? The answer to that is that it doesn't, and that this is mostly dead code. Closer examination shows that on executing a syscall, userspace loses its SVE state entirely. This is part of the ABI. Another thing to notice is that although the kernel provides helpers such as kernel_neon_begin()/end(), they only deal with the FP/NEON state, and not SVE. Given that you can only execute a guest as the result of a syscall, and that the kernel cannot use SVE by itself, it becomes pretty obvious that there is never any host SVE state to save, and that this code is only there to increase confusion. Get rid of the TIF_SVE tracking and host save infrastructure altogether. Reviewed-by: Mark Brown Signed-off-by: Marc Zyngier Signed-off-by: Mark Brown --- arch/arm64/include/asm/kvm_host.h | 1 - arch/arm64/kvm/fpsimd.c | 20 +++++--------------- arch/arm64/kvm/hyp/include/hyp/switch.h | 27 +++------------------------ 3 files changed, 8 insertions(+), 40 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 91038fa2e5e0..4e84d75ae91c 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -410,7 +410,6 @@ struct kvm_vcpu_arch { #define KVM_ARM64_DEBUG_DIRTY (1 << 0) #define KVM_ARM64_FP_ENABLED (1 << 1) /* guest FP regs loaded */ #define KVM_ARM64_FP_HOST (1 << 2) /* host FP regs loaded */ -#define KVM_ARM64_HOST_SVE_IN_USE (1 << 3) /* backup for host TIF_SVE */ #define KVM_ARM64_HOST_SVE_ENABLED (1 << 4) /* SVE enabled for EL0 */ #define KVM_ARM64_GUEST_HAS_SVE (1 << 5) /* SVE exposed to guest */ #define KVM_ARM64_VCPU_SVE_FINALIZED (1 << 6) /* SVE config completed */ diff --git a/arch/arm64/kvm/fpsimd.c b/arch/arm64/kvm/fpsimd.c index 5621020b28de..2d15e1d6e214 100644 --- a/arch/arm64/kvm/fpsimd.c +++ b/arch/arm64/kvm/fpsimd.c @@ -66,22 +66,15 @@ int kvm_arch_vcpu_run_map_fp(struct kvm_vcpu *vcpu) * * Here, we just set the correct metadata to indicate that the FPSIMD * state in the cpu regs (if any) belongs to current on the host. - * - * TIF_SVE is backed up here, since it may get clobbered with guest state. - * This flag is restored by kvm_arch_vcpu_put_fp(vcpu). */ void kvm_arch_vcpu_load_fp(struct kvm_vcpu *vcpu) { BUG_ON(!current->mm); + BUG_ON(test_thread_flag(TIF_SVE)); - vcpu->arch.flags &= ~(KVM_ARM64_FP_ENABLED | - KVM_ARM64_HOST_SVE_IN_USE | - KVM_ARM64_HOST_SVE_ENABLED); + vcpu->arch.flags &= ~KVM_ARM64_FP_ENABLED; vcpu->arch.flags |= KVM_ARM64_FP_HOST; - if (test_thread_flag(TIF_SVE)) - vcpu->arch.flags |= KVM_ARM64_HOST_SVE_IN_USE; - if (read_sysreg(cpacr_el1) & CPACR_EL1_ZEN_EL0EN) vcpu->arch.flags |= KVM_ARM64_HOST_SVE_ENABLED; } @@ -115,13 +108,11 @@ void kvm_arch_vcpu_ctxsync_fp(struct kvm_vcpu *vcpu) void kvm_arch_vcpu_put_fp(struct kvm_vcpu *vcpu) { unsigned long flags; - bool host_has_sve = system_supports_sve(); - bool guest_has_sve = vcpu_has_sve(vcpu); local_irq_save(flags); if (vcpu->arch.flags & KVM_ARM64_FP_ENABLED) { - if (guest_has_sve) { + if (vcpu_has_sve(vcpu)) { __vcpu_sys_reg(vcpu, ZCR_EL1) = read_sysreg_el1(SYS_ZCR); /* Restore the VL that was saved when bound to the CPU */ @@ -131,7 +122,7 @@ void kvm_arch_vcpu_put_fp(struct kvm_vcpu *vcpu) } fpsimd_save_and_flush_cpu_state(); - } else if (has_vhe() && host_has_sve) { + } else if (has_vhe() && system_supports_sve()) { /* * The FPSIMD/SVE state in the CPU has not been touched, and we * have SVE (and VHE): CPACR_EL1 (alias CPTR_EL2) has been @@ -145,8 +136,7 @@ void kvm_arch_vcpu_put_fp(struct kvm_vcpu *vcpu) sysreg_clear_set(CPACR_EL1, CPACR_EL1_ZEN_EL0EN, 0); } - update_thread_flag(TIF_SVE, - vcpu->arch.flags & KVM_ARM64_HOST_SVE_IN_USE); + update_thread_flag(TIF_SVE, 0); local_irq_restore(flags); } diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h index ecd41844eda0..269ec3587210 100644 --- a/arch/arm64/kvm/hyp/include/hyp/switch.h +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h @@ -207,16 +207,6 @@ static inline bool __populate_fault_info(struct kvm_vcpu *vcpu) return __get_fault_info(esr, &vcpu->arch.fault); } -static inline void __hyp_sve_save_host(struct kvm_vcpu *vcpu) -{ - struct thread_struct *thread; - - thread = container_of(vcpu->arch.host_fpsimd_state, struct thread_struct, - uw.fpsimd_state); - - __sve_save_state(sve_pffr(thread), &vcpu->arch.host_fpsimd_state->fpsr); -} - static inline void __hyp_sve_restore_guest(struct kvm_vcpu *vcpu) { sve_cond_update_zcr_vq(vcpu_sve_max_vq(vcpu) - 1, SYS_ZCR_EL2); @@ -228,21 +218,14 @@ static inline void __hyp_sve_restore_guest(struct kvm_vcpu *vcpu) /* Check for an FPSIMD/SVE trap and handle as appropriate */ static inline bool __hyp_handle_fpsimd(struct kvm_vcpu *vcpu) { - bool sve_guest, sve_host; + bool sve_guest; u8 esr_ec; u64 reg; if (!system_supports_fpsimd()) return false; - if (system_supports_sve()) { - sve_guest = vcpu_has_sve(vcpu); - sve_host = vcpu->arch.flags & KVM_ARM64_HOST_SVE_IN_USE; - } else { - sve_guest = false; - sve_host = false; - } - + sve_guest = vcpu_has_sve(vcpu); esr_ec = kvm_vcpu_trap_get_class(vcpu); if (esr_ec != ESR_ELx_EC_FP_ASIMD && esr_ec != ESR_ELx_EC_SVE) @@ -269,11 +252,7 @@ static inline bool __hyp_handle_fpsimd(struct kvm_vcpu *vcpu) isb(); if (vcpu->arch.flags & KVM_ARM64_FP_HOST) { - if (sve_host) - __hyp_sve_save_host(vcpu); - else - __fpsimd_save_state(vcpu->arch.host_fpsimd_state); - + __fpsimd_save_state(vcpu->arch.host_fpsimd_state); vcpu->arch.flags &= ~KVM_ARM64_FP_HOST; } From patchwork Tue Apr 8 18:09:57 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14043533 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C723CC369A1 for ; Tue, 8 Apr 2025 18:27:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Cc:To:In-Reply-To:References :Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=GNOyp6GigbpFbkWc21TYYEkbgjhm7GRBbYgFaFWOY4A=; b=qHjwAyO4fEvg99o9ri+6HGEVWD 9OThRemOat33i11LCLX6+0ASeKVtmMy1DkHZ0Ia5pWYqY2+qLrSSsJBT7daqaAfPVjdkuR6TKdZkX LZhivsuTOk6/bhp3l7Pczztit6klLPU0WO1Nkx4QOhc0hpIfOV2RMrstZsPJQ+jC1v+mrXgGhtRZY GhB3mEHx75ZcOMFP9T2tD+vR3rRaWezlmxcdIireP82GjVkbeT/k6ShFDHRLMXUwoW61/1hotnwmY r5+0WAg6oyYlarLQy4c2rtREe+ikL14Nwv23CYBO5jdCjZC8gGHYOtSdFMeXcTP4fkJ3/e4z6CC4A FVPGrlFw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u2Dfk-000000056jz-1n6b; Tue, 08 Apr 2025 18:27:44 +0000 Received: from nyc.source.kernel.org ([147.75.193.91]) by bombadil.infradead.org with esmtps (Exim 4.98.2 #2 (Red Hat Linux)) id 1u2Daa-000000055rj-1gdl for linux-arm-kernel@lists.infradead.org; Tue, 08 Apr 2025 18:22:25 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id BDF39A48D1E; Tue, 8 Apr 2025 18:16:54 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8B3DCC4CEE5; Tue, 8 Apr 2025 18:22:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744136543; bh=DHaryj/zCLrbSQFZpiYi3QKZgzGIpgsLSnX9179Q5K8=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=D2Ppd5i4W+A1WmJFUyd6uuPQjtTKicLwsVf6YDcZAmC0qV9YADvJS0dMIvEEMk0y3 ee/K+LWSRm9iZefWkk8E41WYxUGP35gfjJG1sB0WLEmwLv1USasIyy1AsfKoTyI/VD ebC0DVs5R9OrDKB/1o6pzg9lOrtYN8iIO+vBtqcPNYj3rZpKx2aYY6igrF5burginR 4etQQDuyBSAKGMo1vlLGfp+lhi0t+AwySNsnxpcXsVvJMXzBDtBEfEOTPzxPB70GJ3 llpDSQZTS5h8eVaV8DW3tTr9dr1O/f1o5+46sZaj7fx6fLYma0G6lBCgNKcm++Lvd9 ttRNNwZzvTtSQ== From: Mark Brown Date: Tue, 08 Apr 2025 19:09:57 +0100 Subject: [PATCH 5.15 v3 02/11] KVM: arm64: Always start with clearing SVE flag on load MIME-Version: 1.0 Message-Id: <20250408-stable-sve-5-15-v3-2-ca9a6b850f55@kernel.org> References: <20250408-stable-sve-5-15-v3-0-ca9a6b850f55@kernel.org> In-Reply-To: <20250408-stable-sve-5-15-v3-0-ca9a6b850f55@kernel.org> To: Greg Kroah-Hartman , Marc Zyngier , James Morse , Suzuki K Poulose , Catalin Marinas , Will Deacon , Oleg Nesterov , Oliver Upton Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-kernel@vger.kernel.org, stable@vger.kernel.org, Mark Brown X-Mailer: b4 0.15-dev-c25d1 X-Developer-Signature: v=1; a=openpgp-sha256; l=1451; i=broonie@kernel.org; h=from:subject:message-id; bh=06yjEiaT5GfiKOx7oXipPxuoURJltRxjW1MKje5sij8=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBn9WlNXXxeqvd2CLOCotpZJXbf0hekyj7Jzjju625g ZKOXEd+JATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCZ/VpTQAKCRAk1otyXVSH0GqCB/ 9TF13fM9RkX5iTdLahTVsnI3XOcb3c0bGqgpF8O+KkzOMaAI66sl6Nz6qF74i4If+djyJKwLGCWQ1y +euvFe60hGhhzlnbmJ7SMhI7M+hi1hjadiWuTCRq7uHK3zof4ibjJl3Iq6I/Ip8N9Njc5J+m9iHN89 90cQRwvyugmwyTcWgd4vNSixakvZM8Egmedt6+rlf6IwRw2EGvbCro8sfT618cHmw4N1jcU00TQ3b3 z9HM4hsi9a6l8HQ5w6Pulh5oOBpFarhX7r+TcG50k24Fc3sFEtn2CrPM9dVxEUmsi4lZlKnF8Gj/NH Hwwvql2qocwQt9XzF+6B55eXicY3ay X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250408_112224_572125_7E17178F X-CRM114-Status: GOOD ( 12.12 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Marc Zyngier [ Upstream commit d52d165d67c5aa26c8c89909003c94a66492d23d ] On each vcpu load, we set the KVM_ARM64_HOST_SVE_ENABLED flag if SVE is enabled for EL0 on the host. This is used to restore the correct state on vpcu put. However, it appears that nothing ever clears this flag. Once set, it will stick until the vcpu is destroyed, which has the potential to spuriously enable SVE for userspace. We probably never saw the issue because no VMM uses SVE, but that's still pretty bad. Unconditionally clearing the flag on vcpu load addresses the issue. Fixes: 8383741ab2e7 ("KVM: arm64: Get rid of host SVE tracking/saving") Signed-off-by: Marc Zyngier Cc: stable@vger.kernel.org Reviewed-by: Mark Brown Link: https://lore.kernel.org/r/20220528113829.1043361-2-maz@kernel.org Signed-off-by: Mark Brown --- arch/arm64/kvm/fpsimd.c | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/arm64/kvm/fpsimd.c b/arch/arm64/kvm/fpsimd.c index 2d15e1d6e214..24734bfcfaa0 100644 --- a/arch/arm64/kvm/fpsimd.c +++ b/arch/arm64/kvm/fpsimd.c @@ -75,6 +75,7 @@ void kvm_arch_vcpu_load_fp(struct kvm_vcpu *vcpu) vcpu->arch.flags &= ~KVM_ARM64_FP_ENABLED; vcpu->arch.flags |= KVM_ARM64_FP_HOST; + vcpu->arch.flags &= ~KVM_ARM64_HOST_SVE_ENABLED; if (read_sysreg(cpacr_el1) & CPACR_EL1_ZEN_EL0EN) vcpu->arch.flags |= KVM_ARM64_HOST_SVE_ENABLED; } From patchwork Tue Apr 8 18:09:58 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14043535 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4D87AC36010 for ; Tue, 8 Apr 2025 18:29:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Cc:To:In-Reply-To:References :Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=2TlbJsVg7ADUXbeqN4HViqbS/v7ZZAnjfGTYi4tNQpg=; b=iTyPOva4NvSnON65/+UfQtRP44 aa9DhNgiqO6RpPPNCqoP+rqDpC55hKp6sUeX5tkvoB16xXbPQgySwkjNTpkuJvvhcby2rIb11RKje 0dRmU0z1/ShLOnRg+y7Kb35wwnliUOfZydTl1HC/s1/bwRkWY5RrWFZnHL2cCYEeGgyFgzYvmJX3l 7UcwwyIn5eY2PBdJ/VxUhkZOEEdtR/FWUMTvWEdFZ3SV+ccAlihp6FPwmoJiQ1d3HXFpJ6GV86Ebp ed8WlFp2w+hrFVVvDHnJfpgVtDFFGqEzm/FlXS/sKb8UkO1SL12fz888+ND16naFC8a8q66xZaHQj /SVkDiew==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u2DhV-000000057IC-3jbw; Tue, 08 Apr 2025 18:29:33 +0000 Received: from nyc.source.kernel.org ([2604:1380:45d1:ec00::3]) by bombadil.infradead.org with esmtps (Exim 4.98.2 #2 (Red Hat Linux)) id 1u2Dad-000000055sF-0QsS for linux-arm-kernel@lists.infradead.org; Tue, 08 Apr 2025 18:22:28 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id C8FFAA48FDE; Tue, 8 Apr 2025 18:16:57 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 88A7CC4CEE8; Tue, 8 Apr 2025 18:22:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744136546; bh=ROS5jr/XnGZ06dtqqkVH5FTegaK3TD2euUktm9dfK+A=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=u4vKl6rvmfLdqWK4Kul+5+fDibO2yJ5yWaJFerR5uj8Kwks3J+wPO2DeDJgasR4cs ls8MaNp8Ezu/I27KvuyKHIDmv9ucOgh0tYWU6FroXMZY8VJneGO6tvMXElp4X9cHPL UyJ7sqxzMC2IPlEfnmbZD/KsEwNgve1ICU9l+AUg3iH6PqYUaj+r5fKov7uPb22Fgj j30dXXdkGUnBEsXBqQpgp9mgEOJfDpqfCqensolvXKIFXZtOJto16LITogSgwqmbFU oOaDomzF+9fw5c4ik5e14pmgZfq3CCsgrTrIss4+vQ7k4++Qdt7t6aSMrCO43QU7Wm HJ29r5B5BebpA== From: Mark Brown Date: Tue, 08 Apr 2025 19:09:58 +0100 Subject: [PATCH 5.15 v3 03/11] KVM: arm64: Discard any SVE state when entering KVM guests MIME-Version: 1.0 Message-Id: <20250408-stable-sve-5-15-v3-3-ca9a6b850f55@kernel.org> References: <20250408-stable-sve-5-15-v3-0-ca9a6b850f55@kernel.org> In-Reply-To: <20250408-stable-sve-5-15-v3-0-ca9a6b850f55@kernel.org> To: Greg Kroah-Hartman , Marc Zyngier , James Morse , Suzuki K Poulose , Catalin Marinas , Will Deacon , Oleg Nesterov , Oliver Upton Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-kernel@vger.kernel.org, stable@vger.kernel.org, Mark Brown , Mark Rutland X-Mailer: b4 0.15-dev-c25d1 X-Developer-Signature: v=1; a=openpgp-sha256; l=3562; i=broonie@kernel.org; h=from:subject:message-id; bh=ROS5jr/XnGZ06dtqqkVH5FTegaK3TD2euUktm9dfK+A=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBn9WlN98cQ/PSAdnjHKGF2x/DeH/D/JoC7PCjo5IQz vy8RWU+JATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCZ/VpTQAKCRAk1otyXVSH0DySB/ 9phxVh+yQ1r2/VJPq78DO5XvJd9IID/lMbU4lEYeOwXylwsQWtN43or0QjkjMPjnX2pyBGCaE1T+AD Neht04TMqS+Fdrg8Y8jvuEtFY0KBT2S9a4ioZdpnaVeCXhvtPGXsCajBGJtwiLvJt8oSbrvt0gAVyk MG4krlmzmmn7+GffR1EAg+OTuoy/6w1atZldRfrh/mkWDIAHLLiu6tz4nrTD47CKo6s4IJrJCkrLmY I7X8/7muNEluxJXGX/X7CWtUEYbZ1c5TrpFc8J1tVVKSqZilOVWguyQ1a5zmLP3BMOFipST4qrd54f PKIGN/lgN76BgMMPzN5B6tnZcR3Ys8 X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250408_112227_273780_4D9228AF X-CRM114-Status: GOOD ( 17.95 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org [ Upstream commit 93ae6b01bafee8fa385aa25ee7ebdb40057f6abe ] Since 8383741ab2e773a99 (KVM: arm64: Get rid of host SVE tracking/saving) KVM has not tracked the host SVE state, relying on the fact that we currently disable SVE whenever we perform a syscall. This may not be true in future since performance optimisation may result in us keeping SVE enabled in order to avoid needing to take access traps to reenable it. Handle this by clearing TIF_SVE and converting the stored task state to FPSIMD format when preparing to run the guest. This is done with a new call fpsimd_kvm_prepare() to keep the direct state manipulation functions internal to fpsimd.c. Signed-off-by: Mark Brown Reviewed-by: Catalin Marinas Reviewed-by: Marc Zyngier Link: https://lore.kernel.org/r/20221115094640.112848-2-broonie@kernel.org Signed-off-by: Will Deacon [ Mark: trivial backport to v6.1 ] Signed-off-by: Mark Rutland Signed-off-by: Mark Brown --- arch/arm64/include/asm/fpsimd.h | 1 + arch/arm64/kernel/fpsimd.c | 23 +++++++++++++++++++++++ arch/arm64/kvm/fpsimd.c | 4 +++- 3 files changed, 27 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/fpsimd.h b/arch/arm64/include/asm/fpsimd.h index 9a62884183e5..f7faf0f4507c 100644 --- a/arch/arm64/include/asm/fpsimd.h +++ b/arch/arm64/include/asm/fpsimd.h @@ -44,6 +44,7 @@ extern void fpsimd_signal_preserve_current_state(void); extern void fpsimd_preserve_current_state(void); extern void fpsimd_restore_current_state(void); extern void fpsimd_update_current_state(struct user_fpsimd_state const *state); +extern void fpsimd_kvm_prepare(void); extern void fpsimd_bind_state_to_cpu(struct user_fpsimd_state *state, void *sve_state, unsigned int sve_vl); diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c index e22571e57ae1..57e89361edcb 100644 --- a/arch/arm64/kernel/fpsimd.c +++ b/arch/arm64/kernel/fpsimd.c @@ -1108,6 +1108,29 @@ void fpsimd_signal_preserve_current_state(void) sve_to_fpsimd(current); } +/* + * Called by KVM when entering the guest. + */ +void fpsimd_kvm_prepare(void) +{ + if (!system_supports_sve()) + return; + + /* + * KVM does not save host SVE state since we can only enter + * the guest from a syscall so the ABI means that only the + * non-saved SVE state needs to be saved. If we have left + * SVE enabled for performance reasons then update the task + * state to be FPSIMD only. + */ + get_cpu_fpsimd_context(); + + if (test_and_clear_thread_flag(TIF_SVE)) + sve_to_fpsimd(current); + + put_cpu_fpsimd_context(); +} + /* * Associate current's FPSIMD context with this cpu * The caller must have ownership of the cpu FPSIMD context before calling diff --git a/arch/arm64/kvm/fpsimd.c b/arch/arm64/kvm/fpsimd.c index 24734bfcfaa0..16e29f03dcbf 100644 --- a/arch/arm64/kvm/fpsimd.c +++ b/arch/arm64/kvm/fpsimd.c @@ -70,12 +70,14 @@ int kvm_arch_vcpu_run_map_fp(struct kvm_vcpu *vcpu) void kvm_arch_vcpu_load_fp(struct kvm_vcpu *vcpu) { BUG_ON(!current->mm); - BUG_ON(test_thread_flag(TIF_SVE)); vcpu->arch.flags &= ~KVM_ARM64_FP_ENABLED; vcpu->arch.flags |= KVM_ARM64_FP_HOST; + fpsimd_kvm_prepare(); + vcpu->arch.flags &= ~KVM_ARM64_HOST_SVE_ENABLED; + if (read_sysreg(cpacr_el1) & CPACR_EL1_ZEN_EL0EN) vcpu->arch.flags |= KVM_ARM64_HOST_SVE_ENABLED; } From patchwork Tue Apr 8 18:09:59 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14043709 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9F6D7C369A1 for ; Tue, 8 Apr 2025 20:57:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Cc:To:In-Reply-To:References :Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=EG4x35VJzRGNDQ7iXYuznBNfHkYfTViDF/a24bOjEI0=; b=CgXuC6WSUAjqKedDYnbRdgf11F 7bRqpLnLe/U9pdg8i0vmJhhkyyf5NnI6RE94/sX7eSMY+WZAazCY+otwntH4wlEUCKN9kFTmbjD6v vWX3i13Vp5xKtJnFrdd9g4ncK+62ArPZE66fLLvoc936t5wzWsJAp886xmG77VShk8KL0GIeT+O0j bgT8ouib1KjCuUVfqVMAk+AxdWLyzqlmwK1f0MjL/iDmmofJi1epSO9esB+qtq5VZdUGu9n1Mgilh Q5cxhXt1v+E2lOU8AkPD/ihGWp5w6M9maQmiEJXZ45zorctR9zQ++Cg9ffLIS1UU+sjvh2gBMAVWS pQbMiKpA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u2G0z-00000005O8a-2CLn; Tue, 08 Apr 2025 20:57:49 +0000 Received: from sea.source.kernel.org ([172.234.252.31]) by bombadil.infradead.org with esmtps (Exim 4.98.2 #2 (Red Hat Linux)) id 1u2Daf-000000055sz-2VWq for linux-arm-kernel@lists.infradead.org; Tue, 08 Apr 2025 18:22:31 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sea.source.kernel.org (Postfix) with ESMTP id 928294A1B8; Tue, 8 Apr 2025 18:22:28 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 92E87C4CEE8; Tue, 8 Apr 2025 18:22:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744136549; bh=S2ZI/vNeQK0jD3CUdjsvBZSRNxhtI9CvKwBhMfYkT/U=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=Bkn4lpViaF8ECmsLv6VEuufq+Vje1V2hJaoxeifVo4Gr9zlWPvsG8qLvGJEZZeKtW 9a9pkNm6XUkfQeoiABCgOR3WJiZ+/jbbGW5K6DUkgHLZxtb9QT7huHBrk66K0ZUoJ5 fb2/jE7AoECs+4060Fy178Rfom9T2PylXLVt7B6xQ5F5LUXZgofsPVoJM8RtW1riti Jki01zH+gCWR0gO/NwS/6ZvWIsRR2fawgqezAzwlZ0vPFRLiKkaj8WEJ3pUL1twFUt lYSTA13QQnVuS+IpK5dLXXDQIsQzwvMzQRylNYJJ1nqEMbLi5qz7ON1o+DOP/iZiBK zHOjPzPTuPdNw== From: Mark Brown Date: Tue, 08 Apr 2025 19:09:59 +0100 Subject: [PATCH 5.15 v3 04/11] arm64/fpsimd: Track the saved FPSIMD state type separately to TIF_SVE MIME-Version: 1.0 Message-Id: <20250408-stable-sve-5-15-v3-4-ca9a6b850f55@kernel.org> References: <20250408-stable-sve-5-15-v3-0-ca9a6b850f55@kernel.org> In-Reply-To: <20250408-stable-sve-5-15-v3-0-ca9a6b850f55@kernel.org> To: Greg Kroah-Hartman , Marc Zyngier , James Morse , Suzuki K Poulose , Catalin Marinas , Will Deacon , Oleg Nesterov , Oliver Upton Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-kernel@vger.kernel.org, stable@vger.kernel.org, Mark Brown , Mark Rutland X-Mailer: b4 0.15-dev-c25d1 X-Developer-Signature: v=1; a=openpgp-sha256; l=12553; i=broonie@kernel.org; h=from:subject:message-id; bh=S2ZI/vNeQK0jD3CUdjsvBZSRNxhtI9CvKwBhMfYkT/U=; b=owGbwMvMwMWocq27KDak/QLjabUkhvSvmX68a4/lfTWNOK2z6MIR9VYJ1UoX2y+fPNsv6irP49Xa K+PZyWjMwsDIxSArpsiy9lnGqvRwia3zH81/BTOIlQlkCgMXpwBM5NFL9v+RCn0LD75/qiz1OZVf/c rstyuKYnd7KO7PP+95JWHX/cBLChofww3eng3y+1fVGcaQvHyLX7Ji34VMnVaDm2zsCipiEjoPfHiy g9at9+w3XMwSejq4bumqxHv6Wz02el2P+yFhsG/ahK6fkY2N5xlm2XeeCPbVL7O3FGEy8EowWV7Q+I 4zp7v0Zy9DKtf99/L9sfzdb+TiFhr4exjY7c6xsb57/+GlGTkmD47UMloIXrLK2L+fhUk5/M7+zQ76 W26rK1V96wjQM77lVL+i1Cayvm6TxC63Vc/NhbYe5W39knYi+aatovMpLj6mvOBu+Q0dem5L1FYHxf gZ3VeI6Tbew1/s5JK0K/qD8jMA X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250408_112229_684179_84FCF438 X-CRM114-Status: GOOD ( 29.39 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org [ Upstream commit baa8515281b30861cff3da7db70662d2a25c6440 ] When we save the state for the floating point registers this can be done in the form visible through either the FPSIMD V registers or the SVE Z and P registers. At present we track which format is currently used based on TIF_SVE and the SME streaming mode state but particularly in the SVE case this limits our options for optimising things, especially around syscalls. Introduce a new enum which we place together with saved floating point state in both thread_struct and the KVM guest state which explicitly states which format is active and keep it up to date when we change it. At present we do not use this state except to verify that it has the expected value when loading the state, future patches will introduce functional changes. Signed-off-by: Mark Brown Reviewed-by: Catalin Marinas Reviewed-by: Marc Zyngier Link: https://lore.kernel.org/r/20221115094640.112848-3-broonie@kernel.org Signed-off-by: Will Deacon [ Mark: fix conflicts due to earlier backports ] Signed-off-by: Mark Rutland Signed-off-by: Mark Brown --- arch/arm64/include/asm/fpsimd.h | 3 +- arch/arm64/include/asm/kvm_host.h | 12 ++++++++ arch/arm64/include/asm/processor.h | 6 ++++ arch/arm64/kernel/fpsimd.c | 58 ++++++++++++++++++++++++++++---------- arch/arm64/kernel/process.c | 3 ++ arch/arm64/kernel/ptrace.c | 3 ++ arch/arm64/kernel/signal.c | 3 ++ arch/arm64/kvm/fpsimd.c | 3 +- 8 files changed, 74 insertions(+), 17 deletions(-) diff --git a/arch/arm64/include/asm/fpsimd.h b/arch/arm64/include/asm/fpsimd.h index f7faf0f4507c..9912bfd020be 100644 --- a/arch/arm64/include/asm/fpsimd.h +++ b/arch/arm64/include/asm/fpsimd.h @@ -47,7 +47,8 @@ extern void fpsimd_update_current_state(struct user_fpsimd_state const *state); extern void fpsimd_kvm_prepare(void); extern void fpsimd_bind_state_to_cpu(struct user_fpsimd_state *state, - void *sve_state, unsigned int sve_vl); + void *sve_state, unsigned int sve_vl, + enum fp_type *type); extern void fpsimd_flush_task_state(struct task_struct *target); extern void fpsimd_save_and_flush_cpu_state(void); diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 4e84d75ae91c..ecae31b0dab3 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -280,7 +280,19 @@ struct vcpu_reset_state { struct kvm_vcpu_arch { struct kvm_cpu_context ctxt; + + /* + * Guest floating point state + * + * The architecture has two main floating point extensions, + * the original FPSIMD and SVE. These have overlapping + * register views, with the FPSIMD V registers occupying the + * low 128 bits of the SVE Z registers. When the core + * floating point code saves the register state of a task it + * records which view it saved in fp_type. + */ void *sve_state; + enum fp_type fp_type; unsigned int sve_max_vl; /* Stage 2 paging state used by the hardware on next switch */ diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h index 7364530de0a7..d5c11a994291 100644 --- a/arch/arm64/include/asm/processor.h +++ b/arch/arm64/include/asm/processor.h @@ -115,6 +115,11 @@ struct debug_info { #endif }; +enum fp_type { + FP_STATE_FPSIMD, + FP_STATE_SVE, +}; + struct cpu_context { unsigned long x19; unsigned long x20; @@ -145,6 +150,7 @@ struct thread_struct { struct user_fpsimd_state fpsimd_state; } uw; + enum fp_type fp_type; /* registers FPSIMD or SVE? */ unsigned int fpsimd_cpu; void *sve_state; /* SVE registers, if any */ unsigned int sve_vl; /* SVE vector length */ diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c index 57e89361edcb..4e702ff0d196 100644 --- a/arch/arm64/kernel/fpsimd.c +++ b/arch/arm64/kernel/fpsimd.c @@ -117,6 +117,7 @@ struct fpsimd_last_state_struct { struct user_fpsimd_state *st; void *sve_state; unsigned int sve_vl; + enum fp_type *fp_type; }; static DEFINE_PER_CPU(struct fpsimd_last_state_struct, fpsimd_last_state); @@ -243,14 +244,6 @@ static void sve_free(struct task_struct *task) * The task can execute SVE instructions while in userspace without * trapping to the kernel. * - * When stored, Z0-Z31 (incorporating Vn in bits[127:0] or the - * corresponding Zn), P0-P15 and FFR are encoded in in - * task->thread.sve_state, formatted appropriately for vector - * length task->thread.sve_vl. - * - * task->thread.sve_state must point to a valid buffer at least - * sve_state_size(task) bytes in size. - * * During any syscall, the kernel may optionally clear TIF_SVE and * discard the vector state except for the FPSIMD subset. * @@ -260,7 +253,15 @@ static void sve_free(struct task_struct *task) * do_sve_acc() to be called, which does some preparation and then * sets TIF_SVE. * - * When stored, FPSIMD registers V0-V31 are encoded in + * During any syscall, the kernel may optionally clear TIF_SVE and + * discard the vector state except for the FPSIMD subset. + * + * The data will be stored in one of two formats: + * + * * FPSIMD only - FP_STATE_FPSIMD: + * + * When the FPSIMD only state stored task->thread.fp_type is set to + * FP_STATE_FPSIMD, the FPSIMD registers V0-V31 are encoded in * task->thread.uw.fpsimd_state; bits [max : 128] for each of Z0-Z31 are * logically zero but not stored anywhere; P0-P15 and FFR are not * stored and have unspecified values from userspace's point of @@ -270,6 +271,19 @@ static void sve_free(struct task_struct *task) * task->thread.sve_state does not need to be non-NULL, valid or any * particular size: it must not be dereferenced. * + * * SVE state - FP_STATE_SVE: + * + * When the full SVE state is stored task->thread.fp_type is set to + * FP_STATE_SVE and Z0-Z31 (incorporating Vn in bits[127:0] or the + * corresponding Zn), P0-P15 and FFR are encoded in in + * task->thread.sve_state, formatted appropriately for vector + * length task->thread.sve_vl or, if SVCR.SM is set, + * task->thread.sme_vl. The storage for the vector registers in + * task->thread.uw.fpsimd_state should be ignored. + * + * task->thread.sve_state must point to a valid buffer at least + * sve_state_size(task) bytes in size. + * * * FPSR and FPCR are always stored in task->thread.uw.fpsimd_state * irrespective of whether TIF_SVE is clear or set, since these are * not vector length dependent. @@ -287,12 +301,15 @@ static void task_fpsimd_load(void) WARN_ON(!system_supports_fpsimd()); WARN_ON(!have_cpu_fpsimd_context()); - if (IS_ENABLED(CONFIG_ARM64_SVE) && test_thread_flag(TIF_SVE)) + if (IS_ENABLED(CONFIG_ARM64_SVE) && test_thread_flag(TIF_SVE)) { + WARN_ON_ONCE(current->thread.fp_type != FP_STATE_SVE); sve_load_state(sve_pffr(¤t->thread), ¤t->thread.uw.fpsimd_state.fpsr, sve_vq_from_vl(current->thread.sve_vl) - 1); - else + } else { + WARN_ON_ONCE(current->thread.fp_type != FP_STATE_FPSIMD); fpsimd_load_state(¤t->thread.uw.fpsimd_state); + } } /* @@ -324,8 +341,11 @@ static void fpsimd_save(void) sve_save_state((char *)last->sve_state + sve_ffr_offset(last->sve_vl), &last->st->fpsr); - } else + *last->fp_type = FP_STATE_SVE; + } else { fpsimd_save_state(last->st); + *last->fp_type = FP_STATE_FPSIMD; + } } } @@ -624,8 +644,10 @@ int sve_set_vector_length(struct task_struct *task, } fpsimd_flush_task_state(task); - if (test_and_clear_tsk_thread_flag(task, TIF_SVE)) + if (test_and_clear_tsk_thread_flag(task, TIF_SVE)) { sve_to_fpsimd(task); + task->thread.fp_type = FP_STATE_FPSIMD; + } if (task == current) put_cpu_fpsimd_context(); @@ -1079,6 +1101,8 @@ void fpsimd_flush_thread(void) current->thread.sve_vl_onexec = 0; } + current->thread.fp_type = FP_STATE_FPSIMD; + put_cpu_fpsimd_context(); } @@ -1125,8 +1149,10 @@ void fpsimd_kvm_prepare(void) */ get_cpu_fpsimd_context(); - if (test_and_clear_thread_flag(TIF_SVE)) + if (test_and_clear_thread_flag(TIF_SVE)) { sve_to_fpsimd(current); + current->thread.fp_type = FP_STATE_FPSIMD; + } put_cpu_fpsimd_context(); } @@ -1145,6 +1171,7 @@ static void fpsimd_bind_task_to_cpu(void) last->st = ¤t->thread.uw.fpsimd_state; last->sve_state = current->thread.sve_state; last->sve_vl = current->thread.sve_vl; + last->fp_type = ¤t->thread.fp_type; current->thread.fpsimd_cpu = smp_processor_id(); if (system_supports_sve()) { @@ -1159,7 +1186,7 @@ static void fpsimd_bind_task_to_cpu(void) } void fpsimd_bind_state_to_cpu(struct user_fpsimd_state *st, void *sve_state, - unsigned int sve_vl) + unsigned int sve_vl, enum fp_type *type) { struct fpsimd_last_state_struct *last = this_cpu_ptr(&fpsimd_last_state); @@ -1170,6 +1197,7 @@ void fpsimd_bind_state_to_cpu(struct user_fpsimd_state *st, void *sve_state, last->st = st; last->sve_state = sve_state; last->sve_vl = sve_vl; + last->fp_type = type; } /* diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c index 839edc4ed7a4..a01f2288ee9a 100644 --- a/arch/arm64/kernel/process.c +++ b/arch/arm64/kernel/process.c @@ -307,6 +307,9 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) dst->thread.sve_state = NULL; clear_tsk_thread_flag(dst, TIF_SVE); + + dst->thread.fp_type = FP_STATE_FPSIMD; + /* clear any pending asynchronous tag fault raised by the parent */ clear_tsk_thread_flag(dst, TIF_MTE_ASYNC_FAULT); diff --git a/arch/arm64/kernel/ptrace.c b/arch/arm64/kernel/ptrace.c index 92de9212b7b5..4ef9e688508c 100644 --- a/arch/arm64/kernel/ptrace.c +++ b/arch/arm64/kernel/ptrace.c @@ -829,6 +829,7 @@ static int sve_set(struct task_struct *target, ret = __fpr_set(target, regset, pos, count, kbuf, ubuf, SVE_PT_FPSIMD_OFFSET); clear_tsk_thread_flag(target, TIF_SVE); + target->thread.fp_type = FP_STATE_FPSIMD; goto out; } @@ -848,6 +849,7 @@ static int sve_set(struct task_struct *target, if (!target->thread.sve_state) { ret = -ENOMEM; clear_tsk_thread_flag(target, TIF_SVE); + target->thread.fp_type = FP_STATE_FPSIMD; goto out; } @@ -858,6 +860,7 @@ static int sve_set(struct task_struct *target, */ fpsimd_sync_to_sve(target); set_tsk_thread_flag(target, TIF_SVE); + target->thread.fp_type = FP_STATE_SVE; BUILD_BUG_ON(SVE_PT_SVE_OFFSET != sizeof(header)); start = SVE_PT_SVE_OFFSET; diff --git a/arch/arm64/kernel/signal.c b/arch/arm64/kernel/signal.c index b3e1beccf458..768de05b616b 100644 --- a/arch/arm64/kernel/signal.c +++ b/arch/arm64/kernel/signal.c @@ -207,6 +207,7 @@ static int restore_fpsimd_context(struct fpsimd_context __user *ctx) __get_user_error(fpsimd.fpcr, &ctx->fpcr, err); clear_thread_flag(TIF_SVE); + current->thread.fp_type = FP_STATE_FPSIMD; /* load the hardware registers from the fpsimd_state structure */ if (!err) @@ -271,6 +272,7 @@ static int restore_sve_fpsimd_context(struct user_ctxs *user) if (sve.head.size <= sizeof(*user->sve)) { clear_thread_flag(TIF_SVE); + current->thread.fp_type = FP_STATE_FPSIMD; goto fpsimd_only; } @@ -303,6 +305,7 @@ static int restore_sve_fpsimd_context(struct user_ctxs *user) return -EFAULT; set_thread_flag(TIF_SVE); + current->thread.fp_type = FP_STATE_SVE; fpsimd_only: /* copy the FP and status/control registers */ diff --git a/arch/arm64/kvm/fpsimd.c b/arch/arm64/kvm/fpsimd.c index 16e29f03dcbf..f1aaad9e14bc 100644 --- a/arch/arm64/kvm/fpsimd.c +++ b/arch/arm64/kvm/fpsimd.c @@ -95,7 +95,8 @@ void kvm_arch_vcpu_ctxsync_fp(struct kvm_vcpu *vcpu) if (vcpu->arch.flags & KVM_ARM64_FP_ENABLED) { fpsimd_bind_state_to_cpu(&vcpu->arch.ctxt.fp_regs, vcpu->arch.sve_state, - vcpu->arch.sve_max_vl); + vcpu->arch.sve_max_vl, + &vcpu->arch.fp_type); clear_thread_flag(TIF_FOREIGN_FPSTATE); update_thread_flag(TIF_SVE, vcpu_has_sve(vcpu)); From patchwork Tue Apr 8 18:10:00 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14043554 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 94087C369A1 for ; Tue, 8 Apr 2025 18:33:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Cc:To:In-Reply-To:References :Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=WBk7j3k/f4GdJZMcjeeZIIR6kWDG+6or3YZicsD0Sn4=; b=FSd4O+QiRl+Zd/nEnCnL84eZOF 1VybbJ5QEY46dChs16okVRNIm3BW+XtzcuzlJ1hIPMavBwlhpXi1/I13jTH+IkeoM5xoDUvXQDY/b 3GsGt7xVg+jzrWHb/uikYRIAzsv8Q4X2VVPYnisRph7DN1YWWPM9u+jJfUo7bLE/iLq4JamMCMr/u Alnww9/9y0mHGcSgKx6e71olMxuYbB9PfH4Prb0T/WU+94N/NyGq/Qp/YdWIMw85oaadS2QDj1buQ 0TlnRgQM2/ZNdyjagkSuG4zoApUjenG7g+f9FFGsNyGNjl3qBcGvxyjaRNRQCxk8hQc7Ac05gX076 1LbjZPeg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u2Dkz-000000057gm-3T09; Tue, 08 Apr 2025 18:33:09 +0000 Received: from sea.source.kernel.org ([2600:3c0a:e001:78e:0:1991:8:25]) by bombadil.infradead.org with esmtps (Exim 4.98.2 #2 (Red Hat Linux)) id 1u2Dai-000000055tx-3H4L for linux-arm-kernel@lists.infradead.org; Tue, 08 Apr 2025 18:22:34 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sea.source.kernel.org (Postfix) with ESMTP id A2B554A1B3; Tue, 8 Apr 2025 18:22:31 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9F179C4CEE5; Tue, 8 Apr 2025 18:22:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744136552; bh=tZlWo93wGbEj8iC3H71eb10+hrSLcFB1nwvaPgD4lfo=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=vNzR5YL5P0z1bamMipf0BYlgJAzt1U2E/5qgXbeEpiwbSYL73UkaNKMiTdqeQgNhG aUwXelCawHbRmNcwBvE9xYNWaWqe3x2cu+vrL99spJ0QIob5xVa6xVVd518X87+m9k xI9kMV/s1T/c3/ZoooHd5PJxJyvgkF37OL0VbSoCNDNhHdfPYmLSIoFrjvspQmqs6L DdYMQMSFfmzjPHruVoRYVR9Zss80HcUcrxFnYQdcBBxj6d+rhMlDHAsP5hGv5M+f9O O9c1tmSVWo0gf51HqBYoqO6Q/1gALlHRAxrEysJHvxJhpv9S6Lzh8LX1Bezdk+X6DC +c5yOJuGuRd8g== From: Mark Brown Date: Tue, 08 Apr 2025 19:10:00 +0100 Subject: [PATCH 5.15 v3 05/11] arm64/fpsimd: Have KVM explicitly say which FP registers to save MIME-Version: 1.0 Message-Id: <20250408-stable-sve-5-15-v3-5-ca9a6b850f55@kernel.org> References: <20250408-stable-sve-5-15-v3-0-ca9a6b850f55@kernel.org> In-Reply-To: <20250408-stable-sve-5-15-v3-0-ca9a6b850f55@kernel.org> To: Greg Kroah-Hartman , Marc Zyngier , James Morse , Suzuki K Poulose , Catalin Marinas , Will Deacon , Oleg Nesterov , Oliver Upton Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-kernel@vger.kernel.org, stable@vger.kernel.org, Mark Brown , Mark Rutland X-Mailer: b4 0.15-dev-c25d1 X-Developer-Signature: v=1; a=openpgp-sha256; l=8729; i=broonie@kernel.org; h=from:subject:message-id; bh=tZlWo93wGbEj8iC3H71eb10+hrSLcFB1nwvaPgD4lfo=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBn9WlPtpFa1cQjm4fP2KNKn8f+hZHWbeyQslbRyIdT wAzyGJeJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCZ/VpTwAKCRAk1otyXVSH0BsaB/ 0WMH/sw2XY/drWFoW3Lrk2GbED0ARVYWPEqYBSIcH7wpRHExlpGANP/X44C0PihwFavlMObgsu0J0P AD/+Mhjg6ZGg2u1ubiH4sEn2zgcL828xwfFfV0/7YClzhsAFbXyzaOEE+VAf43VO3nVe+IwMBfT4KU Ge3i0xoO5qjU5uuptYi9whbU7OZ6569Jlcpf5fjpPJqnANRP+CjRRvG6AdMKAnteLD6qiVaR3OPBJH YgmM+tbyGxB/TSfVCUciGKzH3Gaj3+Nt4dIWzpW/DpYksGMFyNcjVNonyMOvAxQectWxz05iFeMbF6 PLBslF3o2az3jnS3WY/yUNMB+TP2FI X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250408_112232_869298_9C8C7A17 X-CRM114-Status: GOOD ( 31.00 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org [ Upstream commit deeb8f9a80fdae5a62525656d65c7070c28bd3a4 ] In order to avoid needlessly saving and restoring the guest registers KVM relies on the host FPSMID code to save the guest registers when we context switch away from the guest. This is done by binding the KVM guest state to the CPU on top of the task state that was originally there, then carefully managing the TIF_SVE flag for the task to cause the host to save the full SVE state when needed regardless of the needs of the host task. This works well enough but isn't terribly direct about what is going on and makes it much more complicated to try to optimise what we're doing with the SVE register state. Let's instead have KVM pass in the register state it wants saving when it binds to the CPU. We introduce a new FP_STATE_CURRENT for use during normal task binding to indicate that we should base our decisions on the current task. This should not be used when actually saving. Ideally we might want to use a separate enum for the type to save but this enum and the enum values would then need to be named which has problems with clarity and ambiguity. In order to ease any future debugging that might be required this patch does not actually update any of the decision making about what to save, it merely starts tracking the new information and warns if the requested state is not what we would otherwise have decided to save. Signed-off-by: Mark Brown Reviewed-by: Catalin Marinas Reviewed-by: Marc Zyngier Link: https://lore.kernel.org/r/20221115094640.112848-4-broonie@kernel.org Signed-off-by: Will Deacon [ Mark: trivial backport ] Signed-off-by: Mark Rutland Signed-off-by: Mark Brown --- arch/arm64/include/asm/fpsimd.h | 2 +- arch/arm64/include/asm/processor.h | 1 + arch/arm64/kernel/fpsimd.c | 79 +++++++++++++++++++++++++++----------- arch/arm64/kvm/fpsimd.c | 13 ++++++- 4 files changed, 70 insertions(+), 25 deletions(-) diff --git a/arch/arm64/include/asm/fpsimd.h b/arch/arm64/include/asm/fpsimd.h index 9912bfd020be..7a407c3767b6 100644 --- a/arch/arm64/include/asm/fpsimd.h +++ b/arch/arm64/include/asm/fpsimd.h @@ -48,7 +48,7 @@ extern void fpsimd_kvm_prepare(void); extern void fpsimd_bind_state_to_cpu(struct user_fpsimd_state *state, void *sve_state, unsigned int sve_vl, - enum fp_type *type); + enum fp_type *type, enum fp_type to_save); extern void fpsimd_flush_task_state(struct task_struct *target); extern void fpsimd_save_and_flush_cpu_state(void); diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h index d5c11a994291..1da032444dac 100644 --- a/arch/arm64/include/asm/processor.h +++ b/arch/arm64/include/asm/processor.h @@ -116,6 +116,7 @@ struct debug_info { }; enum fp_type { + FP_STATE_CURRENT, /* Save based on current task state. */ FP_STATE_FPSIMD, FP_STATE_SVE, }; diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c index 4e702ff0d196..105b8aa0c038 100644 --- a/arch/arm64/kernel/fpsimd.c +++ b/arch/arm64/kernel/fpsimd.c @@ -118,6 +118,7 @@ struct fpsimd_last_state_struct { void *sve_state; unsigned int sve_vl; enum fp_type *fp_type; + enum fp_type to_save; }; static DEFINE_PER_CPU(struct fpsimd_last_state_struct, fpsimd_last_state); @@ -269,7 +270,8 @@ static void sve_free(struct task_struct *task) * but userspace is discouraged from relying on this. * * task->thread.sve_state does not need to be non-NULL, valid or any - * particular size: it must not be dereferenced. + * particular size: it must not be dereferenced and any data stored + * there should be considered stale and not referenced. * * * SVE state - FP_STATE_SVE: * @@ -282,7 +284,9 @@ static void sve_free(struct task_struct *task) * task->thread.uw.fpsimd_state should be ignored. * * task->thread.sve_state must point to a valid buffer at least - * sve_state_size(task) bytes in size. + * sve_state_size(task) bytes in size. The data stored in + * task->thread.uw.fpsimd_state.vregs should be considered stale + * and not referenced. * * * FPSR and FPCR are always stored in task->thread.uw.fpsimd_state * irrespective of whether TIF_SVE is clear or set, since these are @@ -321,32 +325,57 @@ static void fpsimd_save(void) struct fpsimd_last_state_struct const *last = this_cpu_ptr(&fpsimd_last_state); /* set by fpsimd_bind_task_to_cpu() or fpsimd_bind_state_to_cpu() */ + bool save_sve_regs = false; + unsigned long vl; WARN_ON(!system_supports_fpsimd()); WARN_ON(!have_cpu_fpsimd_context()); - if (!test_thread_flag(TIF_FOREIGN_FPSTATE)) { - if (IS_ENABLED(CONFIG_ARM64_SVE) && - test_thread_flag(TIF_SVE)) { - if (WARN_ON(sve_get_vl() != last->sve_vl)) { - /* - * Can't save the user regs, so current would - * re-enter user with corrupt state. - * There's no way to recover, so kill it: - */ - force_signal_inject(SIGKILL, SI_KERNEL, 0, 0); - return; - } - - sve_save_state((char *)last->sve_state + - sve_ffr_offset(last->sve_vl), - &last->st->fpsr); - *last->fp_type = FP_STATE_SVE; - } else { - fpsimd_save_state(last->st); - *last->fp_type = FP_STATE_FPSIMD; + if (test_thread_flag(TIF_FOREIGN_FPSTATE)) + return; + + if (IS_ENABLED(CONFIG_ARM64_SVE) && + test_thread_flag(TIF_SVE)) { + if (WARN_ON(sve_get_vl() != last->sve_vl)) { + /* + * Can't save the user regs, so current would + * re-enter user with corrupt state. + * There's no way to recover, so kill it: + */ + force_signal_inject(SIGKILL, SI_KERNEL, 0, 0); + return; } } + + if (test_thread_flag(TIF_SVE)) { + save_sve_regs = true; + vl = last->sve_vl; + } + + /* + * Validate that an explicitly specified state to save is + * consistent with the task state. + */ + switch (last->to_save) { + case FP_STATE_CURRENT: + break; + case FP_STATE_FPSIMD: + WARN_ON_ONCE(save_sve_regs); + break; + case FP_STATE_SVE: + WARN_ON_ONCE(!save_sve_regs); + break; + } + + if (IS_ENABLED(CONFIG_ARM64_SVE) && save_sve_regs) { + sve_save_state((char *)last->sve_state + + sve_ffr_offset(last->sve_vl), + &last->st->fpsr); + *last->fp_type = FP_STATE_SVE; + } else { + fpsimd_save_state(last->st); + *last->fp_type = FP_STATE_FPSIMD; + } } /* @@ -987,6 +1016,7 @@ void do_sve_acc(unsigned long esr, struct pt_regs *regs) } else { fpsimd_to_sve(current); fpsimd_flush_task_state(current); + current->thread.fp_type = FP_STATE_SVE; } put_cpu_fpsimd_context(); @@ -1172,6 +1202,7 @@ static void fpsimd_bind_task_to_cpu(void) last->sve_state = current->thread.sve_state; last->sve_vl = current->thread.sve_vl; last->fp_type = ¤t->thread.fp_type; + last->to_save = FP_STATE_CURRENT; current->thread.fpsimd_cpu = smp_processor_id(); if (system_supports_sve()) { @@ -1186,7 +1217,8 @@ static void fpsimd_bind_task_to_cpu(void) } void fpsimd_bind_state_to_cpu(struct user_fpsimd_state *st, void *sve_state, - unsigned int sve_vl, enum fp_type *type) + unsigned int sve_vl, enum fp_type *type, + enum fp_type to_save) { struct fpsimd_last_state_struct *last = this_cpu_ptr(&fpsimd_last_state); @@ -1198,6 +1230,7 @@ void fpsimd_bind_state_to_cpu(struct user_fpsimd_state *st, void *sve_state, last->sve_state = sve_state; last->sve_vl = sve_vl; last->fp_type = type; + last->to_save = to_save; } /* diff --git a/arch/arm64/kvm/fpsimd.c b/arch/arm64/kvm/fpsimd.c index f1aaad9e14bc..54a31c97eb7a 100644 --- a/arch/arm64/kvm/fpsimd.c +++ b/arch/arm64/kvm/fpsimd.c @@ -90,13 +90,24 @@ void kvm_arch_vcpu_load_fp(struct kvm_vcpu *vcpu) */ void kvm_arch_vcpu_ctxsync_fp(struct kvm_vcpu *vcpu) { + enum fp_type fp_type; + WARN_ON_ONCE(!irqs_disabled()); if (vcpu->arch.flags & KVM_ARM64_FP_ENABLED) { + if (vcpu_has_sve(vcpu)) + fp_type = FP_STATE_SVE; + else + fp_type = FP_STATE_FPSIMD; + + /* + * Currently we do not support SME guests so SVCR is + * always 0 and we just need a variable to point to. + */ fpsimd_bind_state_to_cpu(&vcpu->arch.ctxt.fp_regs, vcpu->arch.sve_state, vcpu->arch.sve_max_vl, - &vcpu->arch.fp_type); + &vcpu->arch.fp_type, fp_type); clear_thread_flag(TIF_FOREIGN_FPSTATE); update_thread_flag(TIF_SVE, vcpu_has_sve(vcpu)); From patchwork Tue Apr 8 18:10:01 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14043555 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B55CCC369A1 for ; Tue, 8 Apr 2025 18:35:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Cc:To:In-Reply-To:References :Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=/qmIEMi6XOGv3HGJoHRNphCYSx6Cktjdm3HBMwNssGY=; b=cQL/2PGrDtc/8Hc/8WhDMEL6fn QBjvePlw0X/H2CjN9y7mGfSz8nB27ToXccKJd9Wz10yKX1Adl9+I2YQpZkQL5Atr1ZIbWPX7uz54s qzRAIjbXScI94tcmUXyXCEgpGPd6hNV8J+O8XNZ3LjgDiYaaBisjRE65T+0ilEe6tXok7xPZekCSe mwxqYeXDjr7eqkehaCYsX/KsJnuyPc5FudYldyJ5vP7FvDKxtY4Kmuz2J3+WUPzo2Rx6EMkFh/k1G hmO+je7AfZDHn4A4+x7eoUYGy6pHuGxgwJpagQ19rwtIVq9zq3aijKe5TvNlHLiVRKHbWhAGisysU e9XYAkiQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u2Dmj-000000057zS-1zYM; Tue, 08 Apr 2025 18:34:57 +0000 Received: from tor.source.kernel.org ([172.105.4.254]) by bombadil.infradead.org with esmtps (Exim 4.98.2 #2 (Red Hat Linux)) id 1u2Dam-000000055ue-1Tqy for linux-arm-kernel@lists.infradead.org; Tue, 08 Apr 2025 18:22:36 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by tor.source.kernel.org (Postfix) with ESMTP id 9797B615EA; Tue, 8 Apr 2025 18:22:26 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A99B4C4CEEA; Tue, 8 Apr 2025 18:22:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744136555; bh=nlVU4m2cGOe9nm23h8UEok/akX92Rm8UP7rBstIa2ww=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=lojfsMf9gw+LIQg1phMfxXgBHDBF2ucn+4t3sagjbTpKcMnPVNjPSw+Lc300sZHsW zjhpVpeeapGkLk7ggYeFrgyqyQTYc6CYXh5n0kCyUklQD5MriWnv+fnjWaa3Du2Cgc IK4x9O2YJW1RFIatz5HoFgYHu+ycacJKTDMJsPTtmcGlGiC9P1QSFQR5Fu20nP+3ba /YFN0DcKUXRObQGqyZXUBeISs9WIutO6qzWuvYFn0NfR0vS3HbT6I2hnRZfzHrKUr3 cGcinZPxFDqVOmYifT4PdVmfp/weEWL7W/KwWB9ib445D7WtWa5FscIyV6aGAU/Cou +in+dAIsrTaaw== From: Mark Brown Date: Tue, 08 Apr 2025 19:10:01 +0100 Subject: [PATCH 5.15 v3 06/11] arm64/fpsimd: Stop using TIF_SVE to manage register saving in KVM MIME-Version: 1.0 Message-Id: <20250408-stable-sve-5-15-v3-6-ca9a6b850f55@kernel.org> References: <20250408-stable-sve-5-15-v3-0-ca9a6b850f55@kernel.org> In-Reply-To: <20250408-stable-sve-5-15-v3-0-ca9a6b850f55@kernel.org> To: Greg Kroah-Hartman , Marc Zyngier , James Morse , Suzuki K Poulose , Catalin Marinas , Will Deacon , Oleg Nesterov , Oliver Upton Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-kernel@vger.kernel.org, stable@vger.kernel.org, Mark Brown , Mark Rutland X-Mailer: b4 0.15-dev-c25d1 X-Developer-Signature: v=1; a=openpgp-sha256; l=3851; i=broonie@kernel.org; h=from:subject:message-id; bh=nlVU4m2cGOe9nm23h8UEok/akX92Rm8UP7rBstIa2ww=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBn9WlQYZfUEY3DdpCSENxoDAtQWgzrglcU6/I6BlVp K4PHXHyJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCZ/VpUAAKCRAk1otyXVSH0LSAB/ 9K3BdacM2anNa6iWUcXNhccjOobP2/QLCPWUhsg+KF4RU/gMuW8NXQAnRJilGvRnKiF3METwZxmdf9 uuKYTNLscADqv/S0ibEtU6sy3iYKvIMYICf6LAo/dJsoHicOzOWGvPa3defVNBIvrokKs30bAURcpp hHfaxP0LTg/nZrtcSko8ZDu5RdheUQajb1fdAMj8RHR8EGwVdD5Io1sO7ireHJiDAzX8yjWyZDloUO pZC556mhAO7h9B7PxoETGQscpsbo0le8SaXtSSf3a9pHlU3yoHllNQGL7NU2nMeBcZJg0F1mRw9bEm XEcOm2Fs+QAzhFsopbU7RdNXPLJz9V X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org [ Upstream commit 62021cc36add7b2c015b837f7893f2fb4b8c2586 ] Now that we are explicitly telling the host FP code which register state it needs to save we can remove the manipulation of TIF_SVE from the KVM code, simplifying it and allowing us to optimise our handling of normal tasks. Remove the manipulation of TIF_SVE from KVM and instead rely on to_save to ensure we save the correct data for it. There should be no functional or performance impact from this change. Signed-off-by: Mark Brown Reviewed-by: Catalin Marinas Reviewed-by: Marc Zyngier Link: https://lore.kernel.org/r/20221115094640.112848-5-broonie@kernel.org Signed-off-by: Will Deacon [ Mark: trivial backport ] Signed-off-by: Mark Rutland Signed-off-by: Mark Brown --- arch/arm64/kernel/fpsimd.c | 40 ++++++++++++++++------------------------ arch/arm64/kvm/fpsimd.c | 3 --- 2 files changed, 16 insertions(+), 27 deletions(-) diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c index 105b8aa0c038..e8f10daaa0d7 100644 --- a/arch/arm64/kernel/fpsimd.c +++ b/arch/arm64/kernel/fpsimd.c @@ -318,7 +318,13 @@ static void task_fpsimd_load(void) /* * Ensure FPSIMD/SVE storage in memory for the loaded context is up to - * date with respect to the CPU registers. + * date with respect to the CPU registers. Note carefully that the + * current context is the context last bound to the CPU stored in + * last, if KVM is involved this may be the guest VM context rather + * than the host thread for the VM pointed to by current. This means + * that we must always reference the state storage via last rather + * than via current, if we are saving KVM state then it will have + * ensured that the type of registers to save is set in last->to_save. */ static void fpsimd_save(void) { @@ -334,9 +340,15 @@ static void fpsimd_save(void) if (test_thread_flag(TIF_FOREIGN_FPSTATE)) return; - if (IS_ENABLED(CONFIG_ARM64_SVE) && - test_thread_flag(TIF_SVE)) { - if (WARN_ON(sve_get_vl() != last->sve_vl)) { + if ((last->to_save == FP_STATE_CURRENT && test_thread_flag(TIF_SVE)) || + last->to_save == FP_STATE_SVE) { + save_sve_regs = true; + vl = last->sve_vl; + } + + if (IS_ENABLED(CONFIG_ARM64_SVE) && save_sve_regs) { + /* Get the configured VL from RDVL, will account for SM */ + if (WARN_ON(sve_get_vl() != vl)) { /* * Can't save the user regs, so current would * re-enter user with corrupt state. @@ -347,26 +359,6 @@ static void fpsimd_save(void) } } - if (test_thread_flag(TIF_SVE)) { - save_sve_regs = true; - vl = last->sve_vl; - } - - /* - * Validate that an explicitly specified state to save is - * consistent with the task state. - */ - switch (last->to_save) { - case FP_STATE_CURRENT: - break; - case FP_STATE_FPSIMD: - WARN_ON_ONCE(save_sve_regs); - break; - case FP_STATE_SVE: - WARN_ON_ONCE(!save_sve_regs); - break; - } - if (IS_ENABLED(CONFIG_ARM64_SVE) && save_sve_regs) { sve_save_state((char *)last->sve_state + sve_ffr_offset(last->sve_vl), diff --git a/arch/arm64/kvm/fpsimd.c b/arch/arm64/kvm/fpsimd.c index 54a31c97eb7a..2e0f44f4c470 100644 --- a/arch/arm64/kvm/fpsimd.c +++ b/arch/arm64/kvm/fpsimd.c @@ -110,7 +110,6 @@ void kvm_arch_vcpu_ctxsync_fp(struct kvm_vcpu *vcpu) &vcpu->arch.fp_type, fp_type); clear_thread_flag(TIF_FOREIGN_FPSTATE); - update_thread_flag(TIF_SVE, vcpu_has_sve(vcpu)); } } @@ -151,7 +150,5 @@ void kvm_arch_vcpu_put_fp(struct kvm_vcpu *vcpu) sysreg_clear_set(CPACR_EL1, CPACR_EL1_ZEN_EL0EN, 0); } - update_thread_flag(TIF_SVE, 0); - local_irq_restore(flags); } From patchwork Tue Apr 8 18:10:02 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14043556 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0AD1CC36010 for ; Tue, 8 Apr 2025 18:36:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Cc:To:In-Reply-To:References :Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=f1HjBXlklLcdesQqC3h2XM3HL2gCYP/Vt59GYna5i34=; b=4A351SOkYbmgMe7YF5AnsQZ3hF A9vuI99mo+4kNOn4XUQihtegEzG3tYPNISD961yb1IAXUNkCrvA2nwSE6OxoOCmhNNEG11m0CogA9 POFU9FdGRRA2echcLhagCfYyVG3s5F9g5BxpPR7+Ae3ZMbDFCyjKGXaLRnO+Fylyvci6svK7kbsiT KiDDFIcQq+AjyVXcQdASJMcfnIqSPWHR6egqzMxA8zEJ8YjeuIp16xMbjal5LMip6uc5OK+7K9YI9 VkWw1iuuOLJsdqYEVjcQOzsFqMQFUy9/QrDMjjcahWnPAmPUj22G2cFVsDNnhdRiuMKeVRAWSeS39 TdOnu6LQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u2DoT-000000058Bu-0i81; Tue, 08 Apr 2025 18:36:45 +0000 Received: from sea.source.kernel.org ([2600:3c0a:e001:78e:0:1991:8:25]) by bombadil.infradead.org with esmtps (Exim 4.98.2 #2 (Red Hat Linux)) id 1u2Dap-000000055vw-498c for linux-arm-kernel@lists.infradead.org; Tue, 08 Apr 2025 18:22:41 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sea.source.kernel.org (Postfix) with ESMTP id 080FB4A1B5; Tue, 8 Apr 2025 18:22:39 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id BA857C4CEE5; Tue, 8 Apr 2025 18:22:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744136559; bh=snqshUSLnJbd+v3fb51JH5LHVcFkGZDOPske10wHJPE=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=AR5OGwjNDfXPbXYhciE1n44y2H+OKESTbpp9vEWRCdndPEAd1tImFap3UUrqgf5XT 6WTOWuzeb7nOCp72RhMi9VxNVxFkHBRZFei9UMIwFWTZb2v5xV38XQroJZrnnrN1ui ccipwFN+Z/+L3fkAr1tnG7ECrLl/i4ZKyEsSOHBMqexkqEONGKiGt7/RBeN7A2F1BX KE6sBaT3uIOrodXt52uP2gMxETYORqmB+1SPX2nDOddx2+9v/unuFZ75Je2gnGyk9k Hfg92g2lzHQn5xzwoadSbWEj00VOF+d5EMfs/dJmJDs92Gx+gP+0EDeqcZvDU2usDI 74E5VVl0odNLw== From: Mark Brown Date: Tue, 08 Apr 2025 19:10:02 +0100 Subject: [PATCH 5.15 v3 07/11] KVM: arm64: Unconditionally save+flush host FPSIMD/SVE/SME state MIME-Version: 1.0 Message-Id: <20250408-stable-sve-5-15-v3-7-ca9a6b850f55@kernel.org> References: <20250408-stable-sve-5-15-v3-0-ca9a6b850f55@kernel.org> In-Reply-To: <20250408-stable-sve-5-15-v3-0-ca9a6b850f55@kernel.org> To: Greg Kroah-Hartman , Marc Zyngier , James Morse , Suzuki K Poulose , Catalin Marinas , Will Deacon , Oleg Nesterov , Oliver Upton Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-kernel@vger.kernel.org, stable@vger.kernel.org, Mark Brown , Mark Rutland , Eric Auger , Wilco Dijkstra , Eric Auger , Florian Weimer , Fuad Tabba , Jeremy Linton , Paolo Bonzini X-Mailer: b4 0.15-dev-c25d1 X-Developer-Signature: v=1; a=openpgp-sha256; l=5038; i=broonie@kernel.org; h=from:subject:message-id; bh=Ejt+NbeaO37nvme363QMF97yG2cqvt7ZXstCZKzmrH4=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBn9WlR2WLA4NWswrks6bn91iHqIEp4z8jsy9k/Hjos mHjeagOJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCZ/VpUQAKCRAk1otyXVSH0OeRB/ 45wYrknZM36rCncP0naHOHCCEcEZz0B+KrmexGOnSojHpkqCw32TSOhY7HHXe+h5vI6iISVuKrhcvE FNqHEGVq88E/We9/OjTE8pLewXNGSjsDPgwV53+c+BZ0UHak9IubnKHaxqYek410krR156DVMG83PQ RJKp2RDR72vNgGBGAEN7BM+CHDrQV2bEG2aM7hacsmm9uG77lBafbWJ1c9CLF9I5o0cMSx7XDImhcQ AUlWUYhVlgliKVOAswZgL2+AzqtTiHM7G+NExg/VVb/0cxJYP6+yU3cSus357MedfdRZs6l0d7Bflj xtyB5q5ZsSykXBvm2ITCvMfSGRQo6N X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250408_112240_077941_CAC15241 X-CRM114-Status: GOOD ( 23.38 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Mark Rutland [ Upstream commit fbc7e61195e23f744814e78524b73b59faa54ab4 ] There are several problems with the way hyp code lazily saves the host's FPSIMD/SVE state, including: * Host SVE being discarded unexpectedly due to inconsistent configuration of TIF_SVE and CPACR_ELx.ZEN. This has been seen to result in QEMU crashes where SVE is used by memmove(), as reported by Eric Auger: https://issues.redhat.com/browse/RHEL-68997 * Host SVE state is discarded *after* modification by ptrace, which was an unintentional ptrace ABI change introduced with lazy discarding of SVE state. * The host FPMR value can be discarded when running a non-protected VM, where FPMR support is not exposed to a VM, and that VM uses FPSIMD/SVE. In these cases the hyp code does not save the host's FPMR before unbinding the host's FPSIMD/SVE/SME state, leaving a stale value in memory. Avoid these by eagerly saving and "flushing" the host's FPSIMD/SVE/SME state when loading a vCPU such that KVM does not need to save any of the host's FPSIMD/SVE/SME state. For clarity, fpsimd_kvm_prepare() is removed and the necessary call to fpsimd_save_and_flush_cpu_state() is placed in kvm_arch_vcpu_load_fp(). As 'fpsimd_state' and 'fpmr_ptr' should not be used, they are set to NULL; all uses of these will be removed in subsequent patches. Historical problems go back at least as far as v5.17, e.g. erroneous assumptions about TIF_SVE being clear in commit: 8383741ab2e773a9 ("KVM: arm64: Get rid of host SVE tracking/saving") ... and so this eager save+flush probably needs to be backported to ALL stable trees. Fixes: 93ae6b01bafee8fa ("KVM: arm64: Discard any SVE state when entering KVM guests") Fixes: 8c845e2731041f0f ("arm64/sve: Leave SVE enabled on syscall if we don't context switch") Fixes: ef3be86021c3bdf3 ("KVM: arm64: Add save/restore support for FPMR") Reported-by: Eric Auger Reported-by: Wilco Dijkstra Reviewed-by: Mark Brown Tested-by: Mark Brown Tested-by: Eric Auger Acked-by: Will Deacon Cc: Catalin Marinas Cc: Florian Weimer Cc: Fuad Tabba Cc: Jeremy Linton Cc: Marc Zyngier Cc: Oliver Upton Cc: Paolo Bonzini Signed-off-by: Mark Rutland Reviewed-by: Oliver Upton Link: https://lore.kernel.org/r/20250210195226.1215254-2-mark.rutland@arm.com Signed-off-by: Marc Zyngier [ Mark: Handle vcpu/host flag conflict, remove host_data_ptr() ] Signed-off-by: Mark Rutland Signed-off-by: Mark Brown --- arch/arm64/kernel/fpsimd.c | 25 ------------------------- arch/arm64/kvm/fpsimd.c | 13 ++++++++++--- 2 files changed, 10 insertions(+), 28 deletions(-) diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c index e8f10daaa0d7..4be9d9fd4fb7 100644 --- a/arch/arm64/kernel/fpsimd.c +++ b/arch/arm64/kernel/fpsimd.c @@ -1154,31 +1154,6 @@ void fpsimd_signal_preserve_current_state(void) sve_to_fpsimd(current); } -/* - * Called by KVM when entering the guest. - */ -void fpsimd_kvm_prepare(void) -{ - if (!system_supports_sve()) - return; - - /* - * KVM does not save host SVE state since we can only enter - * the guest from a syscall so the ABI means that only the - * non-saved SVE state needs to be saved. If we have left - * SVE enabled for performance reasons then update the task - * state to be FPSIMD only. - */ - get_cpu_fpsimd_context(); - - if (test_and_clear_thread_flag(TIF_SVE)) { - sve_to_fpsimd(current); - current->thread.fp_type = FP_STATE_FPSIMD; - } - - put_cpu_fpsimd_context(); -} - /* * Associate current's FPSIMD context with this cpu * The caller must have ownership of the cpu FPSIMD context before calling diff --git a/arch/arm64/kvm/fpsimd.c b/arch/arm64/kvm/fpsimd.c index 2e0f44f4c470..2afa2521bce1 100644 --- a/arch/arm64/kvm/fpsimd.c +++ b/arch/arm64/kvm/fpsimd.c @@ -74,9 +74,16 @@ void kvm_arch_vcpu_load_fp(struct kvm_vcpu *vcpu) vcpu->arch.flags &= ~KVM_ARM64_FP_ENABLED; vcpu->arch.flags |= KVM_ARM64_FP_HOST; - fpsimd_kvm_prepare(); - - vcpu->arch.flags &= ~KVM_ARM64_HOST_SVE_ENABLED; + /* + * Ensure that any host FPSIMD/SVE/SME state is saved and unbound such + * that the host kernel is responsible for restoring this state upon + * return to userspace, and the hyp code doesn't need to save anything. + * + * When the host may use SME, fpsimd_save_and_flush_cpu_state() ensures + * that PSTATE.{SM,ZA} == {0,0}. + */ + fpsimd_save_and_flush_cpu_state(); + vcpu->arch.flags &= ~KVM_ARM64_FP_HOST; if (read_sysreg(cpacr_el1) & CPACR_EL1_ZEN_EL0EN) vcpu->arch.flags |= KVM_ARM64_HOST_SVE_ENABLED; From patchwork Tue Apr 8 18:10:03 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14043708 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0C341C36010 for ; Tue, 8 Apr 2025 20:58:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Cc:To:In-Reply-To:References :Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=Qfc2Pi3v+Bb2gOAjnZ8Rj2WDjilYIvZCAJa0GQFdo30=; b=I+MLuvr1whMAvxglKu1aCYeA33 nH5TGZfDlrDkby8kqEpirzj7yf+ZVvCIFKFOe6ZaHg8fZQG2scTpvCubiDZim9o9ByD0jR70rxo7V G2cOqnEKHIojRg5OgpMqlypzvCflF5z70yQonu0XdmarspoevvwqCc1cYtaoQGNU7+4NZP4uVqG1z ocBu0ECz4baDTZYIPS+AYVLsciOGQMmmdOTY+Q2jbfoqvnWgOY2hy5EsDLI0zGC9a7ELxzLNwhijG Y8c8Cqycb1C3STTAAEZ2ki1k+uMAEQwqRAHpTFzI3OXk9tNjwJsEnyew8yRh2cJrS8mTS4DQo7wVs r9d55ONQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u2G10-00000005O8q-0CIC; Tue, 08 Apr 2025 20:57:50 +0000 Received: from nyc.source.kernel.org ([147.75.193.91]) by bombadil.infradead.org with esmtps (Exim 4.98.2 #2 (Red Hat Linux)) id 1u2Dat-000000055wQ-3TTh for linux-arm-kernel@lists.infradead.org; Tue, 08 Apr 2025 18:22:45 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id 7AEEBA48FE7; Tue, 8 Apr 2025 18:17:14 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0E306C4CEE5; Tue, 8 Apr 2025 18:22:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744136562; bh=e9qvF2o7uVNtfQASnyy4CdQbnEmB3uPMMDe207tDMbc=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=FQDJhzAfb7nvmMDonzRGZJgcy6Hot1cEy7VNFNxuAqjrYmWQfOQ/YXgmeQFwCyXya C2BIPGW6heAsTP2wRczA/JlbvT3d/oNXfevVyCytTS6AHm4u9DOIYBVZbJn+rZ8Y1b /GOQtcRzpfA3nDGV00J+WNySrrBsIpXhXGWlrgudmvuiHKA+xGeBPPRohP+KYDmyHZ UzxFehpSsMSwgkGUvmvz1xApLOPs8zpkHX3V6bLNoaTOH2nfc1p+gX/9mfTcBCSyX3 aqQcPwupK2Q1xqwrLpPb4ZXP4ZlTIiCxbGj6zO0fIl8Pevrpv3cT1Ei6IrGcJG5bf5 Y1aVEKElsJN/w== From: Mark Brown Date: Tue, 08 Apr 2025 19:10:03 +0100 Subject: [PATCH 5.15 v3 08/11] KVM: arm64: Remove host FPSIMD saving for non-protected KVM MIME-Version: 1.0 Message-Id: <20250408-stable-sve-5-15-v3-8-ca9a6b850f55@kernel.org> References: <20250408-stable-sve-5-15-v3-0-ca9a6b850f55@kernel.org> In-Reply-To: <20250408-stable-sve-5-15-v3-0-ca9a6b850f55@kernel.org> To: Greg Kroah-Hartman , Marc Zyngier , James Morse , Suzuki K Poulose , Catalin Marinas , Will Deacon , Oleg Nesterov , Oliver Upton Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-kernel@vger.kernel.org, stable@vger.kernel.org, Mark Brown , Mark Rutland , Fuad Tabba X-Mailer: b4 0.15-dev-c25d1 X-Developer-Signature: v=1; a=openpgp-sha256; l=2943; i=broonie@kernel.org; h=from:subject:message-id; bh=IYSm9LOHiakFHli8QpJhz4RyCAaedNNXODj93c0ElHY=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBn9WlR+WNqKkh3Yl7jjk3LaM97k5ATftDWIUcZmY0+ QpFoGXqJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCZ/VpUQAKCRAk1otyXVSH0Lm+B/ 9XV/VfIVsrLFZ3h9eV9VQQQ5JX1xethAlNJ5u1v2QLLnuurnmNtvzXNIImCzIoJkxm9DlWYyzZvujq F26k5SWKIP0aF0uFjop96gXh/LTheFP0+8D744GZNHd2EQK7g6p7MZogkl9Z9wq6tAzdfbnmgijDkQ R3ZiltSus/XaiY2GkIryApFcCMsaO0M/fFShD6NJDWDi4pUn1CNaiZX9aVqURPmretvym1yy8gFCNJ 8TLiS/n3UvECm0dmUw+O2de6yk0+fUiSVNNbKQYKZ7LrSi+XfEBr8Mz5+qQBr0jmGLd68Q81yrPJ+f sSADzc8K5JZqllM3epXaqhQF2D4dsG X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250408_112243_999557_6B8B66B0 X-CRM114-Status: GOOD ( 15.43 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Mark Rutland [ Upstream commit 8eca7f6d5100b6997df4f532090bc3f7e0203bef ] Now that the host eagerly saves its own FPSIMD/SVE/SME state, non-protected KVM never needs to save the host FPSIMD/SVE/SME state, and the code to do this is never used. Protected KVM still needs to save/restore the host FPSIMD/SVE state to avoid leaking guest state to the host (and to avoid revealing to the host whether the guest used FPSIMD/SVE/SME), and that code needs to be retained. Remove the unused code and data structures. To avoid the need for a stub copy of kvm_hyp_save_fpsimd_host() in the VHE hyp code, the nVHE/hVHE version is moved into the shared switch header, where it is only invoked when KVM is in protected mode. Signed-off-by: Mark Rutland Reviewed-by: Mark Brown Tested-by: Mark Brown Acked-by: Will Deacon Cc: Catalin Marinas Cc: Fuad Tabba Cc: Marc Zyngier Cc: Oliver Upton Reviewed-by: Oliver Upton Link: https://lore.kernel.org/r/20250210195226.1215254-3-mark.rutland@arm.com Signed-off-by: Marc Zyngier Signed-off-by: Mark Brown --- arch/arm64/include/asm/kvm_host.h | 1 - arch/arm64/kvm/fpsimd.c | 1 - arch/arm64/kvm/hyp/include/hyp/switch.h | 6 +----- 3 files changed, 1 insertion(+), 7 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index ecae31b0dab3..06e8d4645ecd 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -333,7 +333,6 @@ struct kvm_vcpu_arch { struct kvm_guest_debug_arch external_debug_state; struct thread_info *host_thread_info; /* hyp VA */ - struct user_fpsimd_state *host_fpsimd_state; /* hyp VA */ struct { /* {Break,watch}point registers */ diff --git a/arch/arm64/kvm/fpsimd.c b/arch/arm64/kvm/fpsimd.c index 2afa2521bce1..1ef9d6cb91ee 100644 --- a/arch/arm64/kvm/fpsimd.c +++ b/arch/arm64/kvm/fpsimd.c @@ -55,7 +55,6 @@ int kvm_arch_vcpu_run_map_fp(struct kvm_vcpu *vcpu) } vcpu->arch.host_thread_info = kern_hyp_va(ti); - vcpu->arch.host_fpsimd_state = kern_hyp_va(fpsimd); error: return ret; } diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h index 269ec3587210..cc102e46b0e2 100644 --- a/arch/arm64/kvm/hyp/include/hyp/switch.h +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h @@ -251,11 +251,7 @@ static inline bool __hyp_handle_fpsimd(struct kvm_vcpu *vcpu) } isb(); - if (vcpu->arch.flags & KVM_ARM64_FP_HOST) { - __fpsimd_save_state(vcpu->arch.host_fpsimd_state); - vcpu->arch.flags &= ~KVM_ARM64_FP_HOST; - } - + /* Restore the guest state */ if (sve_guest) __hyp_sve_restore_guest(vcpu); else From patchwork Tue Apr 8 18:10:04 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14043557 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E5DA0C36010 for ; Tue, 8 Apr 2025 18:40:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Cc:To:In-Reply-To:References :Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=mC0NXmwp3hUFyxgYzqhl2oziOdKHiyicw+7VAZXZeac=; b=PsAxBjPt7M4PZZoBOAWjgGGqC9 zQhJIcA7poBIdTn4QLZnmO+s8cDFwWRSMP2ox49FQ+qz7Sf+XltQcRp9T5ndHa0uCxE0+jWao7hGE pVZUj4f0fUpaOUzVvAVlufxipoyNVWT0WBiILvSagVlNG0XJH4VWLoZmJeJhzMsFSAbXYdx4qQCPq STKLM9UpMbaTbbwsKjHc25QFAuEKSoSnoPgvm9Eeyfr1wIK3kZRzHOOEpDr1E8B7pfefwSHhpl8Fb 4BerwlYFuig2OfQ1Wy1FLespqRTO/bcjWDJNkYSsf9evr93ljMJCET2Bq2h31OClnVyaLK4knHrhh hnTaKgvw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u2Drw-000000058X5-0WUi; Tue, 08 Apr 2025 18:40:20 +0000 Received: from nyc.source.kernel.org ([147.75.193.91]) by bombadil.infradead.org with esmtps (Exim 4.98.2 #2 (Red Hat Linux)) id 1u2Dax-000000055xB-0GBT for linux-arm-kernel@lists.infradead.org; Tue, 08 Apr 2025 18:22:48 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id B7D71A48FE6; Tue, 8 Apr 2025 18:17:17 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 447B5C4CEE9; Tue, 8 Apr 2025 18:22:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744136566; bh=94B73exkvxZIp7PjZglkCZBY40cUsa7OpOScJsLUoog=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=PqvFcV7MB/OV15mYLd+gHbJSSGEFznA0DxuB2PBVbE7W4085ugfWvdYgDdqylxt0D 5Qf8xVh99HLh+fKnd2NjKZhzt+EWmXLHwyigsSDpr15dXYkK61Z1L0wpkS+63S3UIX VDmmarfGl2Rew/V8277bDGLIJhb8WcQTFMbR+lKVFoJt6GJyUgSPufX2MdteB94LdM k8/Xr+4PPugCU8ewDKtPgqt4dfjQXSu+ahYrKUlAz5YxyztryMEG92aO2m+3jBS/0p VJOjQSlPKG/bRt/8FgZkC/Ecp1b5NjOKiVIhaVKd8jIPWIY2F2jf/Q4A3RXFhGe/91 sDMg42jnTHkDg== From: Mark Brown Date: Tue, 08 Apr 2025 19:10:04 +0100 Subject: [PATCH 5.15 v3 09/11] KVM: arm64: Remove VHE host restore of CPACR_EL1.ZEN MIME-Version: 1.0 Message-Id: <20250408-stable-sve-5-15-v3-9-ca9a6b850f55@kernel.org> References: <20250408-stable-sve-5-15-v3-0-ca9a6b850f55@kernel.org> In-Reply-To: <20250408-stable-sve-5-15-v3-0-ca9a6b850f55@kernel.org> To: Greg Kroah-Hartman , Marc Zyngier , James Morse , Suzuki K Poulose , Catalin Marinas , Will Deacon , Oleg Nesterov , Oliver Upton Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-kernel@vger.kernel.org, stable@vger.kernel.org, Mark Brown , Mark Rutland , Fuad Tabba X-Mailer: b4 0.15-dev-c25d1 X-Developer-Signature: v=1; a=openpgp-sha256; l=3514; i=broonie@kernel.org; h=from:subject:message-id; bh=z0T8Mw2Gv92zuKeVWJq8mDeqr184OynN7N95pkdbt3E=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBn9WlSA5GbnGtX3RsCslIUJ3vjbvkc54JHy7pM4Ogz TkYpdP2JATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCZ/VpUgAKCRAk1otyXVSH0IPdB/ 9qgLuVL0XNOu2uxbW9SdLR/JkjAJ1tmBtaOhOGfqkAa2zW4epsXR3iVIJeSj2NMMTJJ4mC5NMC4zk4 HBymsKA7vqRIQQ/RCtlTpNj2B6hpg+ZvYPgZr//O0LXVN1G7iI7KtBRocC3zDiYwLPe+qrsosUbTjD LcfEnXdqe/J6yRBkmo1GdiC1K1OEG8YNK6CmQmLTG7PKwv/3tf8VqL97zm+qKb5FGxXwD5qsoow7WU qXzvZOdEnMYa7hJF2oUd6ZWsJoQvzrZYHQ7nm7XoOzfDAx7oO5QQcw3/4kxOvHwNzK3Ypzw3liOMYQ 5qZaSqkTIyH7q5lHnF3Q1Jh5h/8J2o X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250408_112247_240977_6943A7E5 X-CRM114-Status: GOOD ( 16.13 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Mark Rutland [ Upstream commit 459f059be702056d91537b99a129994aa6ccdd35 ] When KVM is in VHE mode, the host kernel tries to save and restore the configuration of CPACR_EL1.ZEN (i.e. CPTR_EL2.ZEN when HCR_EL2.E2H=1) across kvm_arch_vcpu_load_fp() and kvm_arch_vcpu_put_fp(), since the configuration may be clobbered by hyp when running a vCPU. This logic is currently redundant. The VHE hyp code unconditionally configures CPTR_EL2.ZEN to 0b01 when returning to the host, permitting host kernel usage of SVE. Now that the host eagerly saves and unbinds its own FPSIMD/SVE/SME state, there's no need to save/restore the state of the EL0 SVE trap. The kernel can safely save/restore state without trapping, as described above, and will restore userspace state (including trap controls) before returning to userspace. Remove the redundant logic. Signed-off-by: Mark Rutland Reviewed-by: Mark Brown Tested-by: Mark Brown Acked-by: Will Deacon Cc: Catalin Marinas Cc: Fuad Tabba Cc: Marc Zyngier Cc: Oliver Upton Reviewed-by: Oliver Upton Link: https://lore.kernel.org/r/20250210195226.1215254-4-mark.rutland@arm.com Signed-off-by: Marc Zyngier [Rework for refactoring of where the flags are stored -- broonie] Signed-off-by: Mark Brown --- arch/arm64/include/asm/kvm_host.h | 1 - arch/arm64/kvm/fpsimd.c | 15 --------------- 2 files changed, 16 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 06e8d4645ecd..3d4e2396a2d7 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -421,7 +421,6 @@ struct kvm_vcpu_arch { #define KVM_ARM64_DEBUG_DIRTY (1 << 0) #define KVM_ARM64_FP_ENABLED (1 << 1) /* guest FP regs loaded */ #define KVM_ARM64_FP_HOST (1 << 2) /* host FP regs loaded */ -#define KVM_ARM64_HOST_SVE_ENABLED (1 << 4) /* SVE enabled for EL0 */ #define KVM_ARM64_GUEST_HAS_SVE (1 << 5) /* SVE exposed to guest */ #define KVM_ARM64_VCPU_SVE_FINALIZED (1 << 6) /* SVE config completed */ #define KVM_ARM64_GUEST_HAS_PTRAUTH (1 << 7) /* PTRAUTH exposed to guest */ diff --git a/arch/arm64/kvm/fpsimd.c b/arch/arm64/kvm/fpsimd.c index 1ef9d6cb91ee..1360ddd4137b 100644 --- a/arch/arm64/kvm/fpsimd.c +++ b/arch/arm64/kvm/fpsimd.c @@ -83,9 +83,6 @@ void kvm_arch_vcpu_load_fp(struct kvm_vcpu *vcpu) */ fpsimd_save_and_flush_cpu_state(); vcpu->arch.flags &= ~KVM_ARM64_FP_HOST; - - if (read_sysreg(cpacr_el1) & CPACR_EL1_ZEN_EL0EN) - vcpu->arch.flags |= KVM_ARM64_HOST_SVE_ENABLED; } /* @@ -142,18 +139,6 @@ void kvm_arch_vcpu_put_fp(struct kvm_vcpu *vcpu) } fpsimd_save_and_flush_cpu_state(); - } else if (has_vhe() && system_supports_sve()) { - /* - * The FPSIMD/SVE state in the CPU has not been touched, and we - * have SVE (and VHE): CPACR_EL1 (alias CPTR_EL2) has been - * reset to CPACR_EL1_DEFAULT by the Hyp code, disabling SVE - * for EL0. To avoid spurious traps, restore the trap state - * seen by kvm_arch_vcpu_load_fp(): - */ - if (vcpu->arch.flags & KVM_ARM64_HOST_SVE_ENABLED) - sysreg_clear_set(CPACR_EL1, 0, CPACR_EL1_ZEN_EL0EN); - else - sysreg_clear_set(CPACR_EL1, CPACR_EL1_ZEN_EL0EN, 0); } local_irq_restore(flags); From patchwork Tue Apr 8 18:10:05 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14043710 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7BF36C369A5 for ; Tue, 8 Apr 2025 20:58:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Cc:To:In-Reply-To:References :Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=bOtXHvr43L8fUw/Kk5YiMVxP+bSauiBwcbiOsJq5p8E=; b=PNAbZknHLnnTCIuKtMBoGAq5qy bTSZMCA93TU9WcxXHb7iEpNiOUvJ6oAiq692hNTxVUd4f61wA8ckP3Y2hGZyuEKYmKdMtoAVyjIAT tQHh68gaXxz3d9mX9qNdJkz74k4F+Pt/Ms4Bm3Q3qSe2iY5Ewv3DvX9oVfoRpx3S9OqWxs8F3V7Pi i0vksmX+hmK6NaBmT07ZExXqPH9d2L27wgjKL7/j0olZvxJZaL0VPjI5sRvS6mFQYnsXa4hrF7Bn8 dvBni8EWCDrRz1lmvWFqP2v7L1oxx9GB/tir5El+EZmUNtFYBeix4XXHEz+lMV9FOqtgX7tSI28Ej YLBWeLaw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u2G10-00000005O9W-2i5T; Tue, 08 Apr 2025 20:57:50 +0000 Received: from nyc.source.kernel.org ([2604:1380:45d1:ec00::3]) by bombadil.infradead.org with esmtps (Exim 4.98.2 #2 (Red Hat Linux)) id 1u2Db0-000000055xo-1x8s for linux-arm-kernel@lists.infradead.org; Tue, 08 Apr 2025 18:22:51 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id F3946A49003; Tue, 8 Apr 2025 18:17:20 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 84094C4CEE5; Tue, 8 Apr 2025 18:22:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744136569; bh=5LpLtUug8+WAmrpDqxb97Aqky7PzaY6BPA6LoGHVYvM=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=K6w53RL/njlqY0wn/F5lqIGLhGFCBLu13G38E/bwQu+W2R7K7YYSpxZfF8ZieMfdb PqL/XdU5IBL0yTtCKXUoQciGkVW4JaVBhGzgLqzTblC21T1/yGI6M67GpvUXDfa/ZE LffQgRKQu1ZJAVWbM2zW90miAXXSrJDSkT+d29qtusrhnceuJBcxORCR8GxZawyOb1 Idx6WVErfpIS4fC6NMwnS/QtUZM/EHhrKLVckQbPnENfgULSFPF6OPjtKhIS8z+xDQ N3VntJo66pco1NfIDjw3X5O9UabQcpopNLNYLEUE5jtvJhb+7Pkxh25gDbIDwr1HW5 GPy1r5YtLUGig== From: Mark Brown Date: Tue, 08 Apr 2025 19:10:05 +0100 Subject: [PATCH 5.15 v3 10/11] KVM: arm64: Calculate cptr_el2 traps on activating traps MIME-Version: 1.0 Message-Id: <20250408-stable-sve-5-15-v3-10-ca9a6b850f55@kernel.org> References: <20250408-stable-sve-5-15-v3-0-ca9a6b850f55@kernel.org> In-Reply-To: <20250408-stable-sve-5-15-v3-0-ca9a6b850f55@kernel.org> To: Greg Kroah-Hartman , Marc Zyngier , James Morse , Suzuki K Poulose , Catalin Marinas , Will Deacon , Oleg Nesterov , Oliver Upton Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-kernel@vger.kernel.org, stable@vger.kernel.org, Mark Brown , Fuad Tabba , James Clark X-Mailer: b4 0.15-dev-c25d1 X-Developer-Signature: v=1; a=openpgp-sha256; l=3292; i=broonie@kernel.org; h=from:subject:message-id; bh=dzuSkTHzORViq39mf97JEZt+XWlNRsFaxOQ4IWqXP1g=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBn9WlT6cQHEa2BVDcqOeuHIhwSftALpooykroaE+iL 3g7I1rGJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCZ/VpUwAKCRAk1otyXVSH0GOpB/ 4i21OpuHSiUOWXc7d6C0MTkEAWUC4+yh60s2WyUhntsumPjgap/cghV7sbRHlsJhzExNgOHavbtFI7 arjJjClACQlEUkiV9Q5lr1XsVycETnRLrqpg7af2NM8cYUT+G7P/2tQjLbDiwhgp8HL6sneawkzYXY Xg8wM7kE+Jmh2UsGOsWFwatuKXZKHbhYKMizMLh3bM1oKCQwCh+ShvL3aVl+pxJ3XMoQE5zdZ0O8Is PA140HkoYIDhe3huIdjXtxDNKA3VVo+eTiAsFLOmcZCgPk8MPOOg3u5qCKP2gSPz452LZKMqUVNIdR TqED8/Pw1ydAF07vUx2sHNHSqfnKPC X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250408_112250_636506_3E77D299 X-CRM114-Status: GOOD ( 15.86 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Fuad Tabba [ Upstream commit 2fd5b4b0e7b440602455b79977bfa64dea101e6c ] Similar to VHE, calculate the value of cptr_el2 from scratch on activate traps. This removes the need to store cptr_el2 in every vcpu structure. Moreover, some traps, such as whether the guest owns the fp registers, need to be set on every vcpu run. Reported-by: James Clark Fixes: 5294afdbf45a ("KVM: arm64: Exclude FP ownership from kvm_vcpu_arch") Signed-off-by: Fuad Tabba Link: https://lore.kernel.org/r/20241216105057.579031-13-tabba@google.com Signed-off-by: Marc Zyngier Signed-off-by: Mark Brown --- arch/arm64/include/asm/kvm_host.h | 1 - arch/arm64/kvm/arm.c | 1 - arch/arm64/kvm/hyp/nvhe/switch.c | 35 ++++++++++++++++++++++++++--------- 3 files changed, 26 insertions(+), 11 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 3d4e2396a2d7..2e0952134e2e 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -301,7 +301,6 @@ struct kvm_vcpu_arch { /* Values of trap registers for the guest. */ u64 hcr_el2; u64 mdcr_el2; - u64 cptr_el2; /* Values of trap registers for the host before guest entry. */ u64 mdcr_el2_host; diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 9ded5443de48..5ca8782edb96 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -1138,7 +1138,6 @@ static int kvm_arch_vcpu_ioctl_vcpu_init(struct kvm_vcpu *vcpu, } vcpu_reset_hcr(vcpu); - vcpu->arch.cptr_el2 = CPTR_EL2_DEFAULT; /* * Handle the "start in power-off" case. diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/switch.c index 4db5409f40c4..c0885197f2a5 100644 --- a/arch/arm64/kvm/hyp/nvhe/switch.c +++ b/arch/arm64/kvm/hyp/nvhe/switch.c @@ -34,21 +34,38 @@ DEFINE_PER_CPU(struct kvm_host_data, kvm_host_data); DEFINE_PER_CPU(struct kvm_cpu_context, kvm_hyp_ctxt); DEFINE_PER_CPU(unsigned long, kvm_hyp_vector); -static void __activate_traps(struct kvm_vcpu *vcpu) +static bool guest_owns_fp_regs(struct kvm_vcpu *vcpu) { - u64 val; + return vcpu->arch.flags & KVM_ARM64_FP_ENABLED; +} - ___activate_traps(vcpu); - __activate_traps_common(vcpu); +static void __activate_cptr_traps(struct kvm_vcpu *vcpu) +{ + u64 val = CPTR_EL2_TAM; /* Same bit irrespective of E2H */ - val = vcpu->arch.cptr_el2; - val |= CPTR_EL2_TTA | CPTR_EL2_TAM; - if (!update_fp_enabled(vcpu)) { - val |= CPTR_EL2_TFP | CPTR_EL2_TZ; - __activate_traps_fpsimd32(vcpu); + /* !hVHE case upstream */ + if (1) { + val |= CPTR_EL2_TTA | CPTR_NVHE_EL2_RES1; + + if (!vcpu_has_sve(vcpu) || !guest_owns_fp_regs(vcpu)) + val |= CPTR_EL2_TZ; + + if (!guest_owns_fp_regs(vcpu)) + val |= CPTR_EL2_TFP; } + if (!guest_owns_fp_regs(vcpu)) + __activate_traps_fpsimd32(vcpu); + write_sysreg(val, cptr_el2); +} + +static void __activate_traps(struct kvm_vcpu *vcpu) +{ + ___activate_traps(vcpu); + __activate_traps_common(vcpu); + __activate_cptr_traps(vcpu); + write_sysreg(__this_cpu_read(kvm_hyp_vector), vbar_el2); if (cpus_have_final_cap(ARM64_WORKAROUND_SPECULATIVE_AT)) { From patchwork Tue Apr 8 18:10:06 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14043560 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 38F6AC369A1 for ; Tue, 8 Apr 2025 18:44:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Cc:To:In-Reply-To:References :Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=MV50P6hRI3KzUliHqBLYdOCZItbY15amgHPFy4fSIrA=; b=Lt+Tulb/xzvEbmVE0G0GEW75ge gbr5D10UIFKKMKor+EkInia7FUGzrd1xAF1ICEL0oR+kcLQw1fLERLDHe7M0Lscb5gX5rFCT4ympY CAijvyVkemnAE7Nmr6K1nG1rE4F1i/uVJ8G37+oOeupcg1f5YgseoIFgWSK7fPScyXsnzcJ4uYaUX 86NV75mFt1fextq/2F/BA+GQ/AaXXEP3+rGh4lylvPVXPILeFV726DtNCwGtSrSKQXB9IXpxNdHmx 02K7SOTEHl91mFNmSk+eT78B4MvDJj7/H/n1EWFKbxdb4JThsQDjpbvKCj3PjYN+oAun0HjCc9oYa xxTUr1mg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1u2DvP-0000000594m-3r8X; Tue, 08 Apr 2025 18:43:55 +0000 Received: from nyc.source.kernel.org ([147.75.193.91]) by bombadil.infradead.org with esmtps (Exim 4.98.2 #2 (Red Hat Linux)) id 1u2Db3-000000055yX-2nIo for linux-arm-kernel@lists.infradead.org; Tue, 08 Apr 2025 18:22:55 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id 3B0D5A48FEA; Tue, 8 Apr 2025 18:17:24 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id BD090C4CEEA; Tue, 8 Apr 2025 18:22:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744136572; bh=ZKoFePMNCshD50hjfLmDWFxIxGgrZ+11urqHkfqB0ao=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=ZCXtcD3G3j7+yFsCuOS+1YSIUnuZ0nOyBKBF4D85HA4+I7qggig6kqxhrtONYdwib iSq0iXY/TtVxjmDMtpIpuBCliYwryNbGJIhmeS1j1pytFH2RQns56slF8LfIk2B36j xjoSw9tsIDftPb/fjwOxTCfkYBXSVB7AeZaB/SKKsLUabm0fEaufuQOnE9hw/YOn/v NyOWVe/Px4kK1FmV2byxJngwDz3DJrhkpc0/hgaxlTdh37Qkkp/ncSi7Soa9TnqEF3 7D/KZWUft7sZjujSVvVz9fmyfDjQj5p8j7zX/5mKf9GRMScLQOP88In+5ptMj/2fgZ H7YVmCuLBwANg== From: Mark Brown Date: Tue, 08 Apr 2025 19:10:06 +0100 Subject: [PATCH 5.15 v3 11/11] KVM: arm64: Eagerly switch ZCR_EL{1,2} MIME-Version: 1.0 Message-Id: <20250408-stable-sve-5-15-v3-11-ca9a6b850f55@kernel.org> References: <20250408-stable-sve-5-15-v3-0-ca9a6b850f55@kernel.org> In-Reply-To: <20250408-stable-sve-5-15-v3-0-ca9a6b850f55@kernel.org> To: Greg Kroah-Hartman , Marc Zyngier , James Morse , Suzuki K Poulose , Catalin Marinas , Will Deacon , Oleg Nesterov , Oliver Upton Cc: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-kernel@vger.kernel.org, stable@vger.kernel.org, Mark Brown , Mark Rutland , Fuad Tabba X-Mailer: b4 0.15-dev-c25d1 X-Developer-Signature: v=1; a=openpgp-sha256; l=12846; i=broonie@kernel.org; h=from:subject:message-id; bh=6f4SDNyGZNLjxVE6DsFNZSv+dF8z3VxQEDFwAdXgt+k=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBn9WlU3O1zh0SLnOtRX3lUXawZU1keWZGORZ6IEWWG RdMizqaJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCZ/VpVAAKCRAk1otyXVSH0MBXB/ 9Pr2PVDFCO5zr29dgVZ/Ly5HFO/ZZhlLXU/a6/JEKqziAg7VeOm0tukoosG+Te1HUw8LrgWiOHnPAL 5QietVair/am1og5J2T0PxNiflL6DjLsKFChZK/HpNZPHqgduCZWjeiUpARp2JGoSCBI7/S3QxpyCa NS5RhT6TFZWcOWsgf88HwQgoR1E5U/uFcobPdL225b2QjRdQFMHm+FiNBM5RomgFOvakhFcqAHAhH+ ASu65ADufKsEO+R1ujRLsqlB6pa3rNUUk5ENFnoOCwFIIz3BTCLmXt7glo+7i2gCy6+9APd6dzGc5x mpN4FwdK10CgIdezGZKij2mkIL3le1 X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250408_112253_855600_63DA865F X-CRM114-Status: GOOD ( 32.41 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Mark Rutland [ Upstream commit 59419f10045bc955d2229819c7cf7a8b0b9c5b59 ] In non-protected KVM modes, while the guest FPSIMD/SVE/SME state is live on the CPU, the host's active SVE VL may differ from the guest's maximum SVE VL: * For VHE hosts, when a VM uses NV, ZCR_EL2 contains a value constrained by the guest hypervisor, which may be less than or equal to that guest's maximum VL. Note: in this case the value of ZCR_EL1 is immaterial due to E2H. * For nVHE/hVHE hosts, ZCR_EL1 contains a value written by the guest, which may be less than or greater than the guest's maximum VL. Note: in this case hyp code traps host SVE usage and lazily restores ZCR_EL2 to the host's maximum VL, which may be greater than the guest's maximum VL. This can be the case between exiting a guest and kvm_arch_vcpu_put_fp(). If a softirq is taken during this period and the softirq handler tries to use kernel-mode NEON, then the kernel will fail to save the guest's FPSIMD/SVE state, and will pend a SIGKILL for the current thread. This happens because kvm_arch_vcpu_ctxsync_fp() binds the guest's live FPSIMD/SVE state with the guest's maximum SVE VL, and fpsimd_save_user_state() verifies that the live SVE VL is as expected before attempting to save the register state: | if (WARN_ON(sve_get_vl() != vl)) { | force_signal_inject(SIGKILL, SI_KERNEL, 0, 0); | return; | } Fix this and make this a bit easier to reason about by always eagerly switching ZCR_EL{1,2} at hyp during guest<->host transitions. With this happening, there's no need to trap host SVE usage, and the nVHE/nVHE __deactivate_cptr_traps() logic can be simplified to enable host access to all present FPSIMD/SVE/SME features. In protected nVHE/hVHE modes, the host's state is always saved/restored by hyp, and the guest's state is saved prior to exit to the host, so from the host's PoV the guest never has live FPSIMD/SVE/SME state, and the host's ZCR_EL1 is never clobbered by hyp. Fixes: 8c8010d69c132273 ("KVM: arm64: Save/restore SVE state for nVHE") Fixes: 2e3cf82063a00ea0 ("KVM: arm64: nv: Ensure correct VL is loaded before saving SVE state") Signed-off-by: Mark Rutland Reviewed-by: Mark Brown Tested-by: Mark Brown Cc: Catalin Marinas Cc: Fuad Tabba Cc: Marc Zyngier Cc: Oliver Upton Cc: Will Deacon Reviewed-by: Oliver Upton Link: https://lore.kernel.org/r/20250210195226.1215254-9-mark.rutland@arm.com Signed-off-by: Marc Zyngier [ v6.6 lacks pKVM saving of host SVE state, pull in discovery of maximum host VL separately -- broonie ] Signed-off-by: Mark Brown --- arch/arm64/include/asm/kvm_host.h | 1 + arch/arm64/include/asm/kvm_hyp.h | 7 +++++ arch/arm64/kvm/fpsimd.c | 19 ++++++------ arch/arm64/kvm/hyp/entry.S | 5 +++ arch/arm64/kvm/hyp/include/hyp/switch.h | 55 +++++++++++++++++++++++++++++++++ arch/arm64/kvm/hyp/nvhe/hyp-main.c | 9 +++--- arch/arm64/kvm/hyp/nvhe/switch.c | 31 ++++++++++--------- arch/arm64/kvm/hyp/vhe/switch.c | 4 +++ arch/arm64/kvm/reset.c | 3 ++ 9 files changed, 106 insertions(+), 28 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 2e0952134e2e..6d7b6b5d076d 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -64,6 +64,7 @@ enum kvm_mode kvm_get_mode(void); DECLARE_STATIC_KEY_FALSE(userspace_irqchip_in_use); extern unsigned int kvm_sve_max_vl; +extern unsigned int kvm_host_sve_max_vl; int kvm_arm_init_sve(void); u32 __attribute_const__ kvm_target_cpu(void); diff --git a/arch/arm64/include/asm/kvm_hyp.h b/arch/arm64/include/asm/kvm_hyp.h index 657d0c94cf82..308df86f9a4b 100644 --- a/arch/arm64/include/asm/kvm_hyp.h +++ b/arch/arm64/include/asm/kvm_hyp.h @@ -117,5 +117,12 @@ void __noreturn __host_enter(struct kvm_cpu_context *host_ctxt); extern u64 kvm_nvhe_sym(id_aa64mmfr0_el1_sys_val); extern u64 kvm_nvhe_sym(id_aa64mmfr1_el1_sys_val); +extern unsigned int kvm_nvhe_sym(kvm_host_sve_max_vl); + +static inline bool guest_owns_fp_regs(struct kvm_vcpu *vcpu) +{ + return vcpu->arch.flags & KVM_ARM64_FP_ENABLED; +} + #endif /* __ARM64_KVM_HYP_H__ */ diff --git a/arch/arm64/kvm/fpsimd.c b/arch/arm64/kvm/fpsimd.c index 1360ddd4137b..cfda503c8b3f 100644 --- a/arch/arm64/kvm/fpsimd.c +++ b/arch/arm64/kvm/fpsimd.c @@ -129,15 +129,16 @@ void kvm_arch_vcpu_put_fp(struct kvm_vcpu *vcpu) local_irq_save(flags); if (vcpu->arch.flags & KVM_ARM64_FP_ENABLED) { - if (vcpu_has_sve(vcpu)) { - __vcpu_sys_reg(vcpu, ZCR_EL1) = read_sysreg_el1(SYS_ZCR); - - /* Restore the VL that was saved when bound to the CPU */ - if (!has_vhe()) - sve_cond_update_zcr_vq(vcpu_sve_max_vq(vcpu) - 1, - SYS_ZCR_EL1); - } - + /* + * Flush (save and invalidate) the fpsimd/sve state so that if + * the host tries to use fpsimd/sve, it's not using stale data + * from the guest. + * + * Flushing the state sets the TIF_FOREIGN_FPSTATE bit for the + * context unconditionally, in both nVHE and VHE. This allows + * the kernel to restore the fpsimd/sve state, including ZCR_EL1 + * when needed. + */ fpsimd_save_and_flush_cpu_state(); } diff --git a/arch/arm64/kvm/hyp/entry.S b/arch/arm64/kvm/hyp/entry.S index 435346ea1504..d8c94c45cb2f 100644 --- a/arch/arm64/kvm/hyp/entry.S +++ b/arch/arm64/kvm/hyp/entry.S @@ -44,6 +44,11 @@ alternative_if ARM64_HAS_RAS_EXTN alternative_else_nop_endif mrs x1, isr_el1 cbz x1, 1f + + // Ensure that __guest_enter() always provides a context + // synchronization event so that callers don't need ISBs for anything + // that would usually be synchonized by the ERET. + isb mov x0, #ARM_EXCEPTION_IRQ ret diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h index cc102e46b0e2..797544662a95 100644 --- a/arch/arm64/kvm/hyp/include/hyp/switch.h +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h @@ -215,6 +215,61 @@ static inline void __hyp_sve_restore_guest(struct kvm_vcpu *vcpu) write_sysreg_el1(__vcpu_sys_reg(vcpu, ZCR_EL1), SYS_ZCR); } +static inline void fpsimd_lazy_switch_to_guest(struct kvm_vcpu *vcpu) +{ + u64 zcr_el1, zcr_el2; + + if (!guest_owns_fp_regs(vcpu)) + return; + + if (vcpu_has_sve(vcpu)) { + zcr_el2 = vcpu_sve_max_vq(vcpu) - 1; + + write_sysreg_el2(zcr_el2, SYS_ZCR); + + zcr_el1 = __vcpu_sys_reg(vcpu, ZCR_EL1); + write_sysreg_el1(zcr_el1, SYS_ZCR); + } +} + +static inline void fpsimd_lazy_switch_to_host(struct kvm_vcpu *vcpu) +{ + u64 zcr_el1, zcr_el2; + + if (!guest_owns_fp_regs(vcpu)) + return; + + /* + * When the guest owns the FP regs, we know that guest+hyp traps for + * any FPSIMD/SVE/SME features exposed to the guest have been disabled + * by either fpsimd_lazy_switch_to_guest() or kvm_hyp_handle_fpsimd() + * prior to __guest_entry(). As __guest_entry() guarantees a context + * synchronization event, we don't need an ISB here to avoid taking + * traps for anything that was exposed to the guest. + */ + if (vcpu_has_sve(vcpu)) { + zcr_el1 = read_sysreg_el1(SYS_ZCR); + __vcpu_sys_reg(vcpu, ZCR_EL1) = zcr_el1; + + /* + * The guest's state is always saved using the guest's max VL. + * Ensure that the host has the guest's max VL active such that + * the host can save the guest's state lazily, but don't + * artificially restrict the host to the guest's max VL. + */ + if (has_vhe()) { + zcr_el2 = vcpu_sve_max_vq(vcpu) - 1; + write_sysreg_el2(zcr_el2, SYS_ZCR); + } else { + zcr_el2 = sve_vq_from_vl(kvm_host_sve_max_vl) - 1; + write_sysreg_el2(zcr_el2, SYS_ZCR); + + zcr_el1 = vcpu_sve_max_vq(vcpu) - 1; + write_sysreg_el1(zcr_el1, SYS_ZCR); + } + } +} + /* Check for an FPSIMD/SVE trap and handle as appropriate */ static inline bool __hyp_handle_fpsimd(struct kvm_vcpu *vcpu) { diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c index 2da6aa8da868..a446883d5b9a 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c +++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c @@ -19,13 +19,17 @@ DEFINE_PER_CPU(struct kvm_nvhe_init_params, kvm_init_params); +unsigned int kvm_host_sve_max_vl; + void __kvm_hyp_host_forward_smc(struct kvm_cpu_context *host_ctxt); static void handle___kvm_vcpu_run(struct kvm_cpu_context *host_ctxt) { DECLARE_REG(struct kvm_vcpu *, vcpu, host_ctxt, 1); + fpsimd_lazy_switch_to_guest(kern_hyp_va(vcpu)); cpu_reg(host_ctxt, 1) = __kvm_vcpu_run(kern_hyp_va(vcpu)); + fpsimd_lazy_switch_to_host(kern_hyp_va(vcpu)); } static void handle___kvm_adjust_pc(struct kvm_cpu_context *host_ctxt) @@ -237,11 +241,6 @@ void handle_trap(struct kvm_cpu_context *host_ctxt) case ESR_ELx_EC_SMC64: handle_host_smc(host_ctxt); break; - case ESR_ELx_EC_SVE: - sysreg_clear_set(cptr_el2, CPTR_EL2_TZ, 0); - isb(); - sve_cond_update_zcr_vq(ZCR_ELx_LEN_MASK, SYS_ZCR_EL2); - break; case ESR_ELx_EC_IABT_LOW: case ESR_ELx_EC_DABT_LOW: handle_host_mem_abort(host_ctxt); diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/switch.c index c0885197f2a5..fff7491d8351 100644 --- a/arch/arm64/kvm/hyp/nvhe/switch.c +++ b/arch/arm64/kvm/hyp/nvhe/switch.c @@ -34,15 +34,13 @@ DEFINE_PER_CPU(struct kvm_host_data, kvm_host_data); DEFINE_PER_CPU(struct kvm_cpu_context, kvm_hyp_ctxt); DEFINE_PER_CPU(unsigned long, kvm_hyp_vector); -static bool guest_owns_fp_regs(struct kvm_vcpu *vcpu) -{ - return vcpu->arch.flags & KVM_ARM64_FP_ENABLED; -} - static void __activate_cptr_traps(struct kvm_vcpu *vcpu) { u64 val = CPTR_EL2_TAM; /* Same bit irrespective of E2H */ + if (!guest_owns_fp_regs(vcpu)) + __activate_traps_fpsimd32(vcpu); + /* !hVHE case upstream */ if (1) { val |= CPTR_EL2_TTA | CPTR_NVHE_EL2_RES1; @@ -52,12 +50,22 @@ static void __activate_cptr_traps(struct kvm_vcpu *vcpu) if (!guest_owns_fp_regs(vcpu)) val |= CPTR_EL2_TFP; + + write_sysreg(val, cptr_el2); } +} - if (!guest_owns_fp_regs(vcpu)) - __activate_traps_fpsimd32(vcpu); +static void __deactivate_cptr_traps(struct kvm_vcpu *vcpu) +{ + /* !hVHE case upstream */ + if (1) { + u64 val = CPTR_NVHE_EL2_RES1; - write_sysreg(val, cptr_el2); + if (!cpus_have_final_cap(ARM64_SVE)) + val |= CPTR_EL2_TZ; + + write_sysreg(val, cptr_el2); + } } static void __activate_traps(struct kvm_vcpu *vcpu) @@ -86,7 +94,6 @@ static void __activate_traps(struct kvm_vcpu *vcpu) static void __deactivate_traps(struct kvm_vcpu *vcpu) { extern char __kvm_hyp_host_vector[]; - u64 cptr; ___deactivate_traps(vcpu); @@ -111,11 +118,7 @@ static void __deactivate_traps(struct kvm_vcpu *vcpu) write_sysreg(this_cpu_ptr(&kvm_init_params)->hcr_el2, hcr_el2); - cptr = CPTR_EL2_DEFAULT; - if (vcpu_has_sve(vcpu) && (vcpu->arch.flags & KVM_ARM64_FP_ENABLED)) - cptr |= CPTR_EL2_TZ; - - write_sysreg(cptr, cptr_el2); + __deactivate_cptr_traps(vcpu); write_sysreg(__kvm_hyp_host_vector, vbar_el2); } diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c index 813e6e2178c1..d8a8628a9d70 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -114,6 +114,8 @@ static int __kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu) sysreg_save_host_state_vhe(host_ctxt); + fpsimd_lazy_switch_to_guest(vcpu); + /* * ARM erratum 1165522 requires us to configure both stage 1 and * stage 2 translation for the guest context before we clear @@ -144,6 +146,8 @@ static int __kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu) __deactivate_traps(vcpu); + fpsimd_lazy_switch_to_host(vcpu); + sysreg_restore_host_state_vhe(host_ctxt); if (vcpu->arch.flags & KVM_ARM64_FP_ENABLED) diff --git a/arch/arm64/kvm/reset.c b/arch/arm64/kvm/reset.c index 5ce36b0a3343..deb205638279 100644 --- a/arch/arm64/kvm/reset.c +++ b/arch/arm64/kvm/reset.c @@ -42,11 +42,14 @@ static u32 kvm_ipa_limit; PSR_AA32_I_BIT | PSR_AA32_F_BIT) unsigned int kvm_sve_max_vl; +unsigned int kvm_host_sve_max_vl; int kvm_arm_init_sve(void) { if (system_supports_sve()) { kvm_sve_max_vl = sve_max_virtualisable_vl; + kvm_host_sve_max_vl = sve_max_vl; + kvm_nvhe_sym(kvm_host_sve_max_vl) = kvm_host_sve_max_vl; /* * The get_sve_reg()/set_sve_reg() ioctl interface will need