[53/89] KVM: arm64: Lazy host FP save/restore

Message ID	20220519134204.5379-54-will@kernel.org (mailing list archive)
State	New, archived
Headers	show Return-Path: <linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org> From: Will Deacon <will@kernel.org> To: kvmarm@lists.cs.columbia.edu Cc: Will Deacon <will@kernel.org>, Ard Biesheuvel <ardb@kernel.org>, Sean Christopherson <seanjc@google.com>, Alexandru Elisei <alexandru.elisei@arm.com>, Andy Lutomirski <luto@amacapital.net>, Catalin Marinas <catalin.marinas@arm.com>, James Morse <james.morse@arm.com>, Chao Peng <chao.p.peng@linux.intel.com>, Quentin Perret <qperret@google.com>, Suzuki K Poulose <suzuki.poulose@arm.com>, Michael Roth <michael.roth@amd.com>, Mark Rutland <mark.rutland@arm.com>, Fuad Tabba <tabba@google.com>, Oliver Upton <oupton@google.com>, Marc Zyngier <maz@kernel.org>, kernel-team@android.com, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Subject: [PATCH 53/89] KVM: arm64: Lazy host FP save/restore Date: Thu, 19 May 2022 14:41:28 +0100 Message-Id: <20220519134204.5379-54-will@kernel.org> In-Reply-To: <20220519134204.5379-1-will@kernel.org> References: <20220519134204.5379-1-will@kernel.org> MIME-Version: 1.0 Precedence: list Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" <linux-arm-kernel-bounces@lists.infradead.org> Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org
Series	KVM: arm64: Base support for the pKVM hypervisor at EL2 \| expand [00/89] KVM: arm64: Base support for the pKVM hypervisor at EL2 [01/89] KVM: arm64: Handle all ID registers trapped for a protected VM [02/89] KVM: arm64: Remove redundant hyp_assert_lock_held() assertions [03/89] KVM: arm64: Return error from kvm_arch_init_vm() on allocation failure [04/89] KVM: arm64: Ignore 'kvm-arm.mode=protected' when using VHE [05/89] KVM: arm64: Extend comment in has_vhe() [06/89] KVM: arm64: Drop stale comment [07/89] KVM: arm64: Move hyp refcount manipulation helpers [08/89] KVM: arm64: Back hyp_vmemmap for all of memory [09/89] KVM: arm64: Unify identifiers used to distinguish host and hypervisor [10/89] KVM: arm64: Implement do_donate() helper for donating memory [11/89] KVM: arm64: Prevent the donation of no-map pages [12/89] KVM: arm64: Add helpers to pin memory shared with hyp [13/89] KVM: arm64: Include asm/kvm_mmu.h in nvhe/mem_protect.h [14/89] KVM: arm64: Add hyp_spinlock_t static initializer [15/89] KVM: arm64: Introduce shadow VM state at EL2 [16/89] KVM: arm64: Instantiate VM shadow data from EL1 [17/89] KVM: arm64: Make hyp stage-1 refcnt correct on the whole range [18/89] KVM: arm64: Factor out private range VA allocation [19/89] KVM: arm64: Add pcpu fixmap infrastructure at EL2 [20/89] KVM: arm64: Provide I-cache invalidation by VA at EL2 [21/89] KVM: arm64: Allow non-coallescable pages in a hyp_pool [22/89] KVM: arm64: Add generic hyp_memcache helpers [23/89] KVM: arm64: Instantiate guest stage-2 page-tables at EL2 [24/89] KVM: arm64: Return guest memory from EL2 via dedicated teardown memcache [25/89] KVM: arm64: Add flags to struct hyp_page [26/89] KVM: arm64: Provide a hypercall for the host to reclaim guest memory [27/89] KVM: arm64: Extend memory sharing to allow host-to-guest transitions [28/89] KVM: arm64: Consolidate stage-2 init in one function [29/89] KVM: arm64: Check for PTE validity when checking for executable/cacheable [30/89] KVM: arm64: Do not allow memslot changes after first VM run under pKVM [31/89] KVM: arm64: Disallow dirty logging and RO memslots with pKVM [32/89] KVM: arm64: Use the shadow vCPU structure in handle___kvm_vcpu_run() [33/89] KVM: arm64: Handle guest stage-2 page-tables entirely at EL2 [34/89] KVM: arm64: Don't access kvm_arm_hyp_percpu_base at EL1 [35/89] KVM: arm64: Unmap kvm_arm_hyp_percpu_base from the host [36/89] KVM: arm64: Maintain a copy of 'kvm_arm_vmid_bits' at EL2 [37/89] KVM: arm64: Explicitly map kvm_vgic_global_state at EL2 [38/89] KVM: arm64: Don't map host sections in pkvm [39/89] KVM: arm64: Extend memory donation to allow host-to-guest transitions [40/89] KVM: arm64: Split up nvhe/fixed_config.h [41/89] KVM: arm64: Make vcpu_{read, write}_sys_reg available to HYP code [42/89] KVM: arm64: Simplify vgic-v3 hypercalls [43/89] KVM: arm64: Add the {flush, sync}_vgic_state() primitives [44/89] KVM: arm64: Introduce predicates to check for protected state [45/89] KVM: arm64: Add the {flush, sync}_timer_state() primitives [46/89] KVM: arm64: Introduce the pkvm_vcpu_{load, put} hypercalls [47/89] KVM: arm64: Add current vcpu and shadow_state lookup primitive [48/89] KVM: arm64: Skip __kvm_adjust_pc() for protected vcpus [49/89] KVM: arm64: Add hyp per_cpu variable to track current physical cpu number [50/89] KVM: arm64: Ensure that TLBs and I-cache are private to each vcpu [51/89] KVM: arm64: Introduce per-EC entry/exit handlers [52/89] KVM: arm64: Introduce lazy-ish state sync for non-protected VMs [53/89] KVM: arm64: Lazy host FP save/restore [54/89] KVM: arm64: Reduce host/shadow vcpu state copying [55/89] KVM: arm64: Do not pass the vcpu to __pkvm_host_map_guest() [56/89] KVM: arm64: Check directly whether the vcpu is protected [57/89] KVM: arm64: Trap debug break and watch from guest [58/89] KVM: arm64: Restrict protected VM capabilities [59/89] KVM: arm64: Do not support MTE for protected VMs [60/89] KVM: arm64: Refactor reset_mpidr to extract its computation [61/89] KVM: arm64: Reset sysregs for protected VMs [62/89] KVM: arm64: Move pkvm_vcpu_init_traps to shadow vcpu init [63/89] KVM: arm64: Fix initializing traps in protected mode [64/89] KVM: arm64: Advertise GICv3 sysreg interface to protected guests [65/89] KVM: arm64: Force injection of a data abort on NISV MMIO exit [66/89] KVM: arm64: Donate memory to protected guests [67/89] KVM: arm64: Add EL2 entry/exit handlers for pKVM guests [68/89] KVM: arm64: Move vgic state between host and shadow vcpu structures [69/89] KVM: arm64: Do not update virtual timer state for protected VMs [70/89] KVM: arm64: Refactor kvm_vcpu_enable_ptrauth() for hyp use [71/89] KVM: arm64: Initialize shadow vm state at hyp [72/89] KVM: arm64: Track the SVE state in the shadow vcpu [73/89] KVM: arm64: Add HVC handling for protected guests at EL2 [74/89] KVM: arm64: Move pstate reset values to kvm_arm.h [75/89] KVM: arm64: Move some kvm_psci functions to a shared header [76/89] KVM: arm64: Factor out vcpu_reset code for core registers and PSCI [77/89] KVM: arm64: Handle PSCI for protected VMs in EL2 [78/89] KVM: arm64: Don't expose TLBI hypercalls after de-privilege [79/89] KVM: arm64: Add is_pkvm_initialized() helper [80/89] KVM: arm64: Refactor enter_exception64() [81/89] KVM: arm64: Inject SIGSEGV on illegal accesses [82/89] KVM: arm64: Support TLB invalidation in guest context [83/89] KVM: arm64: Avoid BBM when changing only s/w bits in Stage-2 PTE [84/89] KVM: arm64: Extend memory sharing to allow guest-to-host transitions [85/89] KVM: arm64: Document the KVM/arm64-specific calls in hypercalls.rst [86/89] KVM: arm64: Reformat/beautify PTP hypercall documentation [87/89] KVM: arm64: Expose memory sharing hypercalls to protected guests [88/89] KVM: arm64: Introduce KVM_VM_TYPE_ARM_PROTECTED machine type for PVMs [89/89] Documentation: KVM: Add some documentation for Protected KVM on arm64

Message ID

20220519134204.5379-54-will@kernel.org (mailing list archive)

State

New, archived

Headers

From: Will Deacon <will@kernel.org>
To: kvmarm@lists.cs.columbia.edu
Cc: Will Deacon <will@kernel.org>, Ard Biesheuvel <ardb@kernel.org>,
 Sean Christopherson <seanjc@google.com>,
 Alexandru Elisei <alexandru.elisei@arm.com>,
 Andy Lutomirski <luto@amacapital.net>,
 Catalin Marinas <catalin.marinas@arm.com>,
 James Morse <james.morse@arm.com>, Chao Peng <chao.p.peng@linux.intel.com>,
 Quentin Perret <qperret@google.com>,
 Suzuki K Poulose <suzuki.poulose@arm.com>,
 Michael Roth <michael.roth@amd.com>, Mark Rutland <mark.rutland@arm.com>,
 Fuad Tabba <tabba@google.com>, Oliver Upton <oupton@google.com>,
 Marc Zyngier <maz@kernel.org>, kernel-team@android.com,
 kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org
Subject: [PATCH 53/89] KVM: arm64: Lazy host FP save/restore
Date: Thu, 19 May 2022 14:41:28 +0100
Message-Id: <20220519134204.5379-54-will@kernel.org>
In-Reply-To: <20220519134204.5379-1-will@kernel.org>
References: <20220519134204.5379-1-will@kernel.org>
MIME-Version: 1.0
Precedence: list
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Sender: "linux-arm-kernel" <linux-arm-kernel-bounces@lists.infradead.org>
Errors-To: 
 linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org

Series

KVM: arm64: Base support for the pKVM hypervisor at EL2 | expand

Commit Message

Will Deacon May 19, 2022, 1:41 p.m. UTC

From: Marc Zyngier <maz@kernel.org>

Implement lazy save/restore of the host FPSIMD register state at EL2.
This allows us to save/restore guest FPSIMD registers without involving
the host and means that we can avoid having to repopulate the shadow
register state on every flush.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 arch/arm64/kvm/hyp/nvhe/hyp-main.c | 57 ++++++++++++++++++++++++++----
 1 file changed, 51 insertions(+), 6 deletions(-)

diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c
index 2a12d6f710ef..228736a9ab40 100644
--- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c
+++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c
@@ -20,6 +20,14 @@ 
 
 #include <linux/irqchip/arm-gic-v3.h>
 
+/*
+ * Host FPSIMD state. Written to when the guest accesses its own FPSIMD state,
+ * and read when the guest state is live and we need to switch back to the host.
+ *
+ * Only valid when the KVM_ARM64_FP_ENABLED flag is set in the shadow structure.
+ */
+static DEFINE_PER_CPU(struct user_fpsimd_state, loaded_host_fpsimd_state);
+
 DEFINE_PER_CPU(struct kvm_nvhe_init_params, kvm_init_params);
 
 void __kvm_hyp_host_forward_smc(struct kvm_cpu_context *host_ctxt);
@@ -195,10 +203,8 @@  static void flush_shadow_state(struct kvm_shadow_vcpu_state *shadow_state)
 
 	shadow_vcpu->arch.hcr_el2	= host_vcpu->arch.hcr_el2;
 	shadow_vcpu->arch.mdcr_el2	= host_vcpu->arch.mdcr_el2;
-	shadow_vcpu->arch.cptr_el2	= host_vcpu->arch.cptr_el2;
 
 	shadow_vcpu->arch.debug_ptr	= kern_hyp_va(host_vcpu->arch.debug_ptr);
-	shadow_vcpu->arch.host_fpsimd_state = host_vcpu->arch.host_fpsimd_state;
 
 	shadow_vcpu->arch.vsesr_el2	= host_vcpu->arch.vsesr_el2;
 
@@ -235,7 +241,6 @@  static void sync_shadow_state(struct kvm_shadow_vcpu_state *shadow_state,
 	host_vcpu->arch.ctxt		= shadow_vcpu->arch.ctxt;
 
 	host_vcpu->arch.hcr_el2		= shadow_vcpu->arch.hcr_el2;
-	host_vcpu->arch.cptr_el2	= shadow_vcpu->arch.cptr_el2;
 
 	sync_vgic_state(host_vcpu, shadow_vcpu);
 	sync_timer_state(shadow_state);
@@ -262,6 +267,27 @@  static void sync_shadow_state(struct kvm_shadow_vcpu_state *shadow_state,
 	shadow_state->exit_code = exit_reason;
 }
 
+static void fpsimd_host_restore(void)
+{
+	sysreg_clear_set(cptr_el2, CPTR_EL2_TZ | CPTR_EL2_TFP, 0);
+	isb();
+
+	if (unlikely(is_protected_kvm_enabled())) {
+		struct kvm_shadow_vcpu_state *shadow_state = pkvm_loaded_shadow_vcpu_state();
+		struct kvm_vcpu *shadow_vcpu = &shadow_state->shadow_vcpu;
+		struct user_fpsimd_state *host_fpsimd_state = this_cpu_ptr(&loaded_host_fpsimd_state);
+
+		__fpsimd_save_state(&shadow_vcpu->arch.ctxt.fp_regs);
+		__fpsimd_restore_state(host_fpsimd_state);
+
+		shadow_vcpu->arch.flags &= ~KVM_ARM64_FP_ENABLED;
+		shadow_vcpu->arch.flags |= KVM_ARM64_FP_HOST;
+	}
+
+	if (system_supports_sve())
+		sve_cond_update_zcr_vq(ZCR_ELx_LEN_MASK, SYS_ZCR_EL2);
+}
+
 static void handle___pkvm_vcpu_load(struct kvm_cpu_context *host_ctxt)
 {
 	DECLARE_REG(unsigned int, shadow_handle, host_ctxt, 1);
@@ -291,6 +317,9 @@  static void handle___pkvm_vcpu_load(struct kvm_cpu_context *host_ctxt)
 		*last_ran = shadow_vcpu->vcpu_id;
 	}
 
+	shadow_vcpu->arch.host_fpsimd_state = this_cpu_ptr(&loaded_host_fpsimd_state);
+	shadow_vcpu->arch.flags |= KVM_ARM64_FP_HOST;
+
 	if (shadow_state_is_protected(shadow_state)) {
 		/* Propagate WFx trapping flags, trap ptrauth */
 		shadow_vcpu->arch.hcr_el2 &= ~(HCR_TWE | HCR_TWI |
@@ -310,6 +339,10 @@  static void handle___pkvm_vcpu_put(struct kvm_cpu_context *host_ctxt)
 
 	if (shadow_state) {
 		struct kvm_vcpu *host_vcpu = shadow_state->host_vcpu;
+		struct kvm_vcpu *shadow_vcpu = &shadow_state->shadow_vcpu;
+
+		if (shadow_vcpu->arch.flags & KVM_ARM64_FP_ENABLED)
+			fpsimd_host_restore();
 
 		if (!shadow_state_is_protected(shadow_state) &&
 			!(READ_ONCE(host_vcpu->arch.flags) & KVM_ARM64_PKVM_STATE_DIRTY))
@@ -377,6 +410,19 @@  static void handle___kvm_vcpu_run(struct kvm_cpu_context *host_ctxt)
 		ret = __kvm_vcpu_run(&shadow_state->shadow_vcpu);
 
 		sync_shadow_state(shadow_state, ret);
+
+		if (shadow_state->shadow_vcpu.arch.flags & KVM_ARM64_FP_ENABLED) {
+			/*
+			 * The guest has used the FP, trap all accesses
+			 * from the host (both FP and SVE).
+			 */
+			u64 reg = CPTR_EL2_TFP;
+
+			if (system_supports_sve())
+				reg |= CPTR_EL2_TZ;
+
+			sysreg_clear_set(cptr_el2, 0, reg);
+		}
 	} else {
 		ret = __kvm_vcpu_run(vcpu);
 	}
@@ -707,10 +753,9 @@  void handle_trap(struct kvm_cpu_context *host_ctxt)
 	case ESR_ELx_EC_SMC64:
 		handle_host_smc(host_ctxt);
 		break;
+	case ESR_ELx_EC_FP_ASIMD:
 	case ESR_ELx_EC_SVE:
-		sysreg_clear_set(cptr_el2, CPTR_EL2_TZ, 0);
-		isb();
-		sve_cond_update_zcr_vq(ZCR_ELx_LEN_MASK, SYS_ZCR_EL2);
+		fpsimd_host_restore();
 		break;
 	case ESR_ELx_EC_IABT_LOW:
 	case ESR_ELx_EC_DABT_LOW:

[53/89] KVM: arm64: Lazy host FP save/restore

Commit Message

Patch