From patchwork Thu Mar 13 21:55:34 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 14015953 Received: from out-183.mta0.migadu.com (out-183.mta0.migadu.com [91.218.175.183]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6A6081F5619 for ; Thu, 13 Mar 2025 21:55:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.183 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741902959; cv=none; b=NaoDTUFL975cI0rikgQHbDv5Pbxzd/XwErzwjrRXVbPL5phI/DS4n9kFJgy1aRXZv84yPwmf4B49Lgn9mYOZOfWNGC9e5vMQ3M9ciJWPk7j9TbmR5CH2//xb61lGgO3IMEVHh/2gpVCcuh+VFmnxG40dGzH14+kcCXV/Afx5PwE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741902959; c=relaxed/simple; bh=F/iemi/KPT8UHZKiz10uv69/7pXoVs7eoZb8uQGWEwQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Nr4pvaxjBTLVj31RqL3HeJr3DlMjX9LPgYQMINAeGYROPZ7K7MhIQKVs6CEAiZYyb1J7zA9NCCeKq6mrX9bD1o8bVYcePesg4ikxAVccnT1cCI39fSCG0hrRzQ8V9ZWKbbiGHhmVtn5qyoAhJ+rs4/Jz9gBSAaqoRERLHaPN5iE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=H4aYF0q4; arc=none smtp.client-ip=91.218.175.183 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="H4aYF0q4" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1741902955; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yHphHcR5C73j9ilQpWdD5cjN+ERsqX+EVOJ7Jk9lwns=; b=H4aYF0q44/+vsV4bGpUORloh7M9dB1FBZ9iZOrRaUL4ME/+jrNeW3fKCIEqhXxF+VeRBVG Ln6Pkp8EG/1/Sm9M3INpZwQM0W/OOOASqOlR3i91Hom3dOpNS3hjvUf1kBec0fhW/mwf4c s9GODSXoSANjdrOiLuyiMQKLXM37HfA= From: Yosry Ahmed To: Sean Christopherson Cc: Paolo Bonzini , Maxim Levitsky , x86@kernel.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed Subject: [PATCH 1/7] KVM: VMX: Generalize VPID allocation to be vendor-neutral Date: Thu, 13 Mar 2025 21:55:34 +0000 Message-ID: <20250313215540.4171762-2-yosry.ahmed@linux.dev> In-Reply-To: <20250313215540.4171762-1-yosry.ahmed@linux.dev> References: <20250313215540.4171762-1-yosry.ahmed@linux.dev> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Generalize the VMX VPID allocation code and make move it to common code in preparation for sharing with SVM. Create a generic struct kvm_tlb_tags, representing a factory for VPIDs (or ASIDs later), and use one for VPIDs. Most of the functionality remains the same, with the following differences: - The enable_vpid checks are moved to the callers for allocate_vpid() and free_vpid(), as they are specific to VMX. - The bitmap allocation is now dynamic (which will be required for SVM), so it is initialized and cleaned up in vmx_hardware_{setup/unsetup}(). - The range of valid TLB tags is expressed in terms of min/max instead of the number of tags to support SVM use cases. Signed-off-by: Yosry Ahmed --- arch/x86/kvm/vmx/nested.c | 4 +-- arch/x86/kvm/vmx/vmx.c | 38 +++++-------------------- arch/x86/kvm/vmx/vmx.h | 4 +-- arch/x86/kvm/x86.c | 58 +++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/x86.h | 13 +++++++++ 5 files changed, 82 insertions(+), 35 deletions(-) diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c index d06e50d9c0e79..b017bd2eb2382 100644 --- a/arch/x86/kvm/vmx/nested.c +++ b/arch/x86/kvm/vmx/nested.c @@ -343,7 +343,7 @@ static void free_nested(struct kvm_vcpu *vcpu) vmx->nested.vmxon = false; vmx->nested.smm.vmxon = false; vmx->nested.vmxon_ptr = INVALID_GPA; - free_vpid(vmx->nested.vpid02); + kvm_tlb_tags_free(&vmx_vpids, vmx->nested.vpid02); vmx->nested.posted_intr_nv = -1; vmx->nested.current_vmptr = INVALID_GPA; if (enable_shadow_vmcs) { @@ -5333,7 +5333,7 @@ static int enter_vmx_operation(struct kvm_vcpu *vcpu) HRTIMER_MODE_ABS_PINNED); vmx->nested.preemption_timer.function = vmx_preemption_timer_fn; - vmx->nested.vpid02 = allocate_vpid(); + vmx->nested.vpid02 = enable_vpid ? kvm_tlb_tags_alloc(&vmx_vpids) : 0; vmx->nested.vmcs02_initialized = false; vmx->nested.vmxon = true; diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index b70ed72c1783d..f7ce75842fa26 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -496,8 +496,7 @@ DEFINE_PER_CPU(struct vmcs *, current_vmcs); */ static DEFINE_PER_CPU(struct list_head, loaded_vmcss_on_cpu); -static DECLARE_BITMAP(vmx_vpid_bitmap, VMX_NR_VPIDS); -static DEFINE_SPINLOCK(vmx_vpid_lock); +struct kvm_tlb_tags vmx_vpids; struct vmcs_config vmcs_config __ro_after_init; struct vmx_capability vmx_capability __ro_after_init; @@ -3972,31 +3971,6 @@ static void seg_setup(int seg) vmcs_write32(sf->ar_bytes, ar); } -int allocate_vpid(void) -{ - int vpid; - - if (!enable_vpid) - return 0; - spin_lock(&vmx_vpid_lock); - vpid = find_first_zero_bit(vmx_vpid_bitmap, VMX_NR_VPIDS); - if (vpid < VMX_NR_VPIDS) - __set_bit(vpid, vmx_vpid_bitmap); - else - vpid = 0; - spin_unlock(&vmx_vpid_lock); - return vpid; -} - -void free_vpid(int vpid) -{ - if (!enable_vpid || vpid == 0) - return; - spin_lock(&vmx_vpid_lock); - __clear_bit(vpid, vmx_vpid_bitmap); - spin_unlock(&vmx_vpid_lock); -} - static void vmx_msr_bitmap_l01_changed(struct vcpu_vmx *vmx) { /* @@ -7559,7 +7533,7 @@ void vmx_vcpu_free(struct kvm_vcpu *vcpu) if (enable_pml) vmx_destroy_pml_buffer(vmx); - free_vpid(vmx->vpid); + kvm_tlb_tags_free(&vmx_vpids, vmx->vpid); nested_vmx_free_vcpu(vcpu); free_loaded_vmcs(vmx->loaded_vmcs); free_page((unsigned long)vmx->ve_info); @@ -7578,7 +7552,7 @@ int vmx_vcpu_create(struct kvm_vcpu *vcpu) err = -ENOMEM; - vmx->vpid = allocate_vpid(); + vmx->vpid = enable_vpid ? kvm_tlb_tags_alloc(&vmx_vpids) : 0; /* * If PML is turned on, failure on enabling PML just results in failure @@ -7681,7 +7655,7 @@ int vmx_vcpu_create(struct kvm_vcpu *vcpu) free_pml: vmx_destroy_pml_buffer(vmx); free_vpid: - free_vpid(vmx->vpid); + kvm_tlb_tags_free(&vmx_vpids, vmx->vpid); return err; } @@ -8373,6 +8347,7 @@ void vmx_hardware_unsetup(void) nested_vmx_hardware_unsetup(); free_kvm_area(); + kvm_tlb_tags_destroy(&vmx_vpids); } void vmx_vm_destroy(struct kvm *kvm) @@ -8591,7 +8566,8 @@ __init int vmx_hardware_setup(void) kvm_caps.has_bus_lock_exit = cpu_has_vmx_bus_lock_detection(); kvm_caps.has_notify_vmexit = cpu_has_notify_vmexit(); - set_bit(0, vmx_vpid_bitmap); /* 0 is reserved for host */ + /* VPID 0 is reserved for host, so min=1 */ + kvm_tlb_tags_init(&vmx_vpids, 1, VMX_NR_VPIDS - 1); if (enable_ept) kvm_mmu_set_ept_masks(enable_ept_ad_bits, diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h index 951e44dc9d0ea..9bece3ea63eaa 100644 --- a/arch/x86/kvm/vmx/vmx.h +++ b/arch/x86/kvm/vmx/vmx.h @@ -376,10 +376,10 @@ struct kvm_vmx { u64 *pid_table; }; +extern struct kvm_tlb_tags vmx_vpids; + void vmx_vcpu_load_vmcs(struct kvm_vcpu *vcpu, int cpu, struct loaded_vmcs *buddy); -int allocate_vpid(void); -void free_vpid(int vpid); void vmx_set_constant_host_state(struct vcpu_vmx *vmx); void vmx_prepare_switch_to_guest(struct kvm_vcpu *vcpu); void vmx_set_host_fs_gs(struct vmcs_host_state *host, u16 fs_sel, u16 gs_sel, diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 69c20a68a3f01..182f18ebc62f3 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -13992,6 +13992,64 @@ int kvm_sev_es_string_io(struct kvm_vcpu *vcpu, unsigned int size, } EXPORT_SYMBOL_GPL(kvm_sev_es_string_io); +int kvm_tlb_tags_init(struct kvm_tlb_tags *tlb_tags, unsigned int min, + unsigned int max) +{ + /* + * 0 is assumed to be the host's TLB tag and is returned on failed + * allocations. + */ + if (WARN_ON_ONCE(min == 0)) + return -1; + + /* + * Allocate enough bits to index the bitmap directly by the tag, + * potentially wasting a bit of memory. + */ + tlb_tags->bitmap = bitmap_zalloc(max + 1, GFP_KERNEL); + if (!tlb_tags->bitmap) + return -1; + + tlb_tags->min = min; + tlb_tags->max = max; + spin_lock_init(&tlb_tags->lock); + return 0; +} +EXPORT_SYMBOL_GPL(kvm_tlb_tags_init); + +void kvm_tlb_tags_destroy(struct kvm_tlb_tags *tlb_tags) +{ + bitmap_free(tlb_tags->bitmap); +} +EXPORT_SYMBOL_GPL(kvm_tlb_tags_destroy); + +unsigned int kvm_tlb_tags_alloc(struct kvm_tlb_tags *tlb_tags) +{ + unsigned int tag; + + spin_lock(&tlb_tags->lock); + tag = find_next_zero_bit(tlb_tags->bitmap, tlb_tags->max + 1, + tlb_tags->min); + if (tag <= tlb_tags->max) + __set_bit(tag, tlb_tags->bitmap); + else + tag = 0; + spin_unlock(&tlb_tags->lock); + return tag; +} +EXPORT_SYMBOL_GPL(kvm_tlb_tags_alloc); + +void kvm_tlb_tags_free(struct kvm_tlb_tags *tlb_tags, unsigned int tag) +{ + if (tag < tlb_tags->min || WARN_ON_ONCE(tag > tlb_tags->max)) + return; + + spin_lock(&tlb_tags->lock); + __clear_bit(tag, tlb_tags->bitmap); + spin_unlock(&tlb_tags->lock); +} +EXPORT_SYMBOL_GPL(kvm_tlb_tags_free); + EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_entry); EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_exit); EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_fast_mmio); diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h index 9dc32a4090761..9f84e933d189b 100644 --- a/arch/x86/kvm/x86.h +++ b/arch/x86/kvm/x86.h @@ -652,4 +652,17 @@ int ____kvm_emulate_hypercall(struct kvm_vcpu *vcpu, unsigned long nr, int kvm_emulate_hypercall(struct kvm_vcpu *vcpu); +struct kvm_tlb_tags { + spinlock_t lock; + unsigned long *bitmap; + unsigned int min; + unsigned int max; +}; + +int kvm_tlb_tags_init(struct kvm_tlb_tags *tlb_tags, unsigned int min, + unsigned int max); +void kvm_tlb_tags_destroy(struct kvm_tlb_tags *tlb_tags); +unsigned int kvm_tlb_tags_alloc(struct kvm_tlb_tags *tlb_tags); +void kvm_tlb_tags_free(struct kvm_tlb_tags *tlb_tags, unsigned int tag); + #endif From patchwork Thu Mar 13 21:55:35 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 14015954 Received: from out-185.mta0.migadu.com (out-185.mta0.migadu.com [91.218.175.185]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4D49E1F5853; Thu, 13 Mar 2025 21:55:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.185 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741902961; cv=none; b=DTD+7H7Sm/GzGoNuTksSg+NrX3cOY1s5DCdlkNAhkGQhd9L0Yln+AIa8ogp76rH4ZQ7N+/c9DfFanS7+nQusbsYgIaHqPaz+bxo22V6JdSqGU1Uj/6JVbNElmK/XLcTuHhXMyMdajelLsl3NWpCSAlON8iL4uDxL3u6ku8BJ76g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741902961; c=relaxed/simple; bh=No3TmjuSSj6PGyLv12E7LcxxDwrTbVsdm4RZvrt8UCM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=q6EMbbxtYkr0YIA6QRbMEuaUkurelGEDL+9sFTlLn/qSpC29dt+pCmNo57YfN83k7T0glRne/Qy1yovt4FJ7ziHZTLkwtvhr1EGO/X372FHjPb47ZUCA59hSDX+XlzOEV28drM8y96XU9gM84UqR1rWsKXKuGg+rJiETXuoeYAU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=rrP1j9X1; arc=none smtp.client-ip=91.218.175.185 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="rrP1j9X1" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1741902957; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=U5Ik2sNVuHfm2wiyHLmIQg8nbZ4c4zHfdyoYXTXv6FE=; b=rrP1j9X1+TyHB0LGEqiDrIgc9mrrffgIiXfBGQcQuWqkQ+XjMEcDfLrrj8KcDVdmi8ykUI abZkjmsqk+CGx7ias9U3xezJ9An1elplkUNP7/jn3z0AdvYMynJLQNp/9V2DybWTXiBLye 3cTS6sQF1QgAq/JqBbPMjGfNb122Rsg= From: Yosry Ahmed To: Sean Christopherson Cc: Paolo Bonzini , Maxim Levitsky , x86@kernel.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed Subject: [PATCH 2/7] KVM: SVM: Use cached local variable in init_vmcb() Date: Thu, 13 Mar 2025 21:55:35 +0000 Message-ID: <20250313215540.4171762-3-yosry.ahmed@linux.dev> In-Reply-To: <20250313215540.4171762-1-yosry.ahmed@linux.dev> References: <20250313215540.4171762-1-yosry.ahmed@linux.dev> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT svm->vmcb->control is already cached in the 'control' local variable, so use that. Signed-off-by: Yosry Ahmed --- arch/x86/kvm/svm/svm.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index 8abeab91d329d..28a6d2c0f250f 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -1367,12 +1367,12 @@ static void init_vmcb(struct kvm_vcpu *vcpu) avic_init_vmcb(svm, vmcb); if (vnmi) - svm->vmcb->control.int_ctl |= V_NMI_ENABLE_MASK; + control->int_ctl |= V_NMI_ENABLE_MASK; if (vgif) { svm_clr_intercept(svm, INTERCEPT_STGI); svm_clr_intercept(svm, INTERCEPT_CLGI); - svm->vmcb->control.int_ctl |= V_GIF_ENABLE_MASK; + control->int_ctl |= V_GIF_ENABLE_MASK; } if (sev_guest(vcpu->kvm)) From patchwork Thu Mar 13 21:55:36 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 14015955 Received: from out-186.mta0.migadu.com (out-186.mta0.migadu.com [91.218.175.186]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CFB1F1F758F for ; Thu, 13 Mar 2025 21:56:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.186 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741902962; cv=none; b=kydpl5Npd9d61Vvoo8Gtmo9tQjhn6ukuJ2zNfp9OoqarBGcqK5z+S8Ly3Hd4/oZpwlrWuaw1PuIRho85KW25CxY0jgPibHXgQzJ3+/C3YToNbtsTFaTCEBua2WL3aElcGCiZNsvNaWrQOe1xodysiXwc1f/9zj5ZZ7cAj3c3pl0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741902962; c=relaxed/simple; bh=ruNjn6oY85pkiEVb/IVF7lfpeCNMLMhOSDdgsEu/JqE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=SwJKTMPQA63LOoJlLfSSqpXsO/PXa+Xq5mnseabtP4slLQymN9enwHTcRFQulwLTWAjX2nEPtMa5pzxJJ4SCSbQxd2m9KHRO+5UW80xUDTtTfBUEdd5Ss2MfSSgwel0n1R/jDDXctceytpgUBKDHGmWjHjiFm3301XZF3Z/ZWEo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=jEz08PGp; arc=none smtp.client-ip=91.218.175.186 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="jEz08PGp" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1741902959; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ISe8UFhYqt/XjX+dSaNqs8ihA9A8NUS4NuxdyQBmcGk=; b=jEz08PGpQ3m38FQK7n04+AT+KP6BbAucMht7gxSkgH/xEU7OrkH15dOc0F+ithY1mftfI2 Y5qLrBeC6Yh6cjSFH2fO+h1A14blFGYv8YTIc/DkcBIXn1O46lPlbjLNQYRYGQy0RM4HgS jGeb+mtiyB6LyVsvsuuMMJ8wEU5M0eg= From: Yosry Ahmed To: Sean Christopherson Cc: Paolo Bonzini , Maxim Levitsky , x86@kernel.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed Subject: [PATCH 3/7] KVM: SVM: Add helpers to set/clear ASID flush Date: Thu, 13 Mar 2025 21:55:36 +0000 Message-ID: <20250313215540.4171762-4-yosry.ahmed@linux.dev> In-Reply-To: <20250313215540.4171762-1-yosry.ahmed@linux.dev> References: <20250313215540.4171762-1-yosry.ahmed@linux.dev> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Incoming changes will add more code paths that set tlb_ctl to TLB_CONTROL_FLUSH_ASID, and will eliminate the use of TLB_CONTROL_FLUSH_ALL_ASID except as fallback when FLUSHBYASID is not available. Introduce set/clear helpers to set tlb_ctl to TLB_CONTROL_FLUSH_ASID or TLB_CONTROL_DO_NOTHING. Opportunistically move the TLB_CONTROL_* definitions to arch/x86/kvm/svm/svm.h as they are not used outside of arch/x86/kvm/svm/. Signed-off-by: Yosry Ahmed --- arch/x86/include/asm/svm.h | 5 ----- arch/x86/kvm/svm/nested.c | 2 +- arch/x86/kvm/svm/sev.c | 2 +- arch/x86/kvm/svm/svm.c | 4 ++-- arch/x86/kvm/svm/svm.h | 15 +++++++++++++++ 5 files changed, 19 insertions(+), 9 deletions(-) diff --git a/arch/x86/include/asm/svm.h b/arch/x86/include/asm/svm.h index 9b7fa99ae9513..a97da63562eb3 100644 --- a/arch/x86/include/asm/svm.h +++ b/arch/x86/include/asm/svm.h @@ -171,11 +171,6 @@ struct __attribute__ ((__packed__)) vmcb_control_area { }; -#define TLB_CONTROL_DO_NOTHING 0 -#define TLB_CONTROL_FLUSH_ALL_ASID 1 -#define TLB_CONTROL_FLUSH_ASID 3 -#define TLB_CONTROL_FLUSH_ASID_LOCAL 7 - #define V_TPR_MASK 0x0f #define V_IRQ_SHIFT 8 diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c index 834b67672d50f..3bff948bc5752 100644 --- a/arch/x86/kvm/svm/nested.c +++ b/arch/x86/kvm/svm/nested.c @@ -681,7 +681,7 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm, /* Done at vmrun: asid. */ /* Also overwritten later if necessary. */ - vmcb02->control.tlb_ctl = TLB_CONTROL_DO_NOTHING; + svm_vmcb_clear_flush_asid(vmcb02); /* nested_cr3. */ if (nested_npt_enabled(svm)) diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c index 0bc708ee27887..b393674733969 100644 --- a/arch/x86/kvm/svm/sev.c +++ b/arch/x86/kvm/svm/sev.c @@ -3479,7 +3479,7 @@ int pre_sev_run(struct vcpu_svm *svm, int cpu) return 0; sd->sev_vmcbs[asid] = svm->vmcb; - svm->vmcb->control.tlb_ctl = TLB_CONTROL_FLUSH_ASID; + svm_vmcb_set_flush_asid(svm->vmcb); vmcb_mark_dirty(svm->vmcb, VMCB_ASID); return 0; } diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index 28a6d2c0f250f..8c90686a33f44 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -4006,7 +4006,7 @@ static void svm_flush_tlb_asid(struct kvm_vcpu *vcpu) * VM-Exit (via kvm_mmu_reset_context()). */ if (static_cpu_has(X86_FEATURE_FLUSHBYASID)) - svm->vmcb->control.tlb_ctl = TLB_CONTROL_FLUSH_ASID; + svm_vmcb_set_flush_asid(svm->vmcb); else svm->current_vmcb->asid_generation--; } @@ -4373,7 +4373,7 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu, svm->nested.nested_run_pending = 0; } - svm->vmcb->control.tlb_ctl = TLB_CONTROL_DO_NOTHING; + svm_vmcb_clear_flush_asid(svm->vmcb); vmcb_mark_all_clean(svm->vmcb); /* if exit due to PF check for async PF */ diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h index d4490eaed55dd..9fd5b249b9c19 100644 --- a/arch/x86/kvm/svm/svm.h +++ b/arch/x86/kvm/svm/svm.h @@ -638,6 +638,21 @@ void svm_set_x2apic_msr_interception(struct vcpu_svm *svm, bool disable); void svm_complete_interrupt_delivery(struct kvm_vcpu *vcpu, int delivery_mode, int trig_mode, int vec); +#define TLB_CONTROL_DO_NOTHING 0 +#define TLB_CONTROL_FLUSH_ALL_ASID 1 +#define TLB_CONTROL_FLUSH_ASID 3 +#define TLB_CONTROL_FLUSH_ASID_LOCAL 7 + +static inline void svm_vmcb_set_flush_asid(struct vmcb *vmcb) +{ + vmcb->control.tlb_ctl = TLB_CONTROL_FLUSH_ASID; +} + +static inline void svm_vmcb_clear_flush_asid(struct vmcb *vmcb) +{ + vmcb->control.tlb_ctl = TLB_CONTROL_DO_NOTHING; +} + /* nested.c */ #define NESTED_EXIT_HOST 0 /* Exit handled on host level */ From patchwork Thu Mar 13 21:55:37 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 14015956 Received: from out-170.mta0.migadu.com (out-170.mta0.migadu.com [91.218.175.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BB1E91F8728 for ; Thu, 13 Mar 2025 21:56:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741902964; cv=none; b=kmzSCcHZYw/yxSHs3CJbGTYqa9il8+ojuJvG6wwUzSBZzQ+yF7uvzL1fkWFwyZ77RECE4mlb0ZtFLWa3dWx/5Kb4i6qytKDHi7SiYBhXHjVkMOu0HnJng6TetK/KwRgxlsGtHxHcRln4RYd8b1Thb0DkUUg2XDEoxKBkuaoR5Zg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741902964; c=relaxed/simple; bh=hurQQCC+/qgM2ttjbpMEz4DnkjDjk25e6iRgNLhdD30=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=etdSwH0So1E8H3xLPSGCznLkgMFvPjlqmQ1okrnKSG9LdvIhadtYnr2XLyQ1oOGwnVhSmF20B9y5doFwjhoceLe77dSMaL1M4Hr8faFlXwh6p5ywfP3dHQZU67V+3P1RUALOamccgokWxe6YHLcwai+pPkL4gsaIQ15IEcpiHNI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=FEjMtT3g; arc=none smtp.client-ip=91.218.175.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="FEjMtT3g" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1741902961; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+bUxD6QRzJrIqWxcl0ag34BYxXxitzR3TXEhcC1owgE=; b=FEjMtT3gYSKmRfUz+g9jxHgt0JRf9SEPGgBg7xgoW+dNlDT5t4R1vdrt5no6wOylDjXHke +BoK1+o1TIPyPZUX0D6mffw6M2nbfMlPN1o4KeL9UoYzgf8afC/gqr8OmfHIYOVEgm6g8q 6ebn/RcZEWGFQezjDUb8oi7S9tkKriM= From: Yosry Ahmed To: Sean Christopherson Cc: Paolo Bonzini , Maxim Levitsky , x86@kernel.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed Subject: [PATCH 4/7] KVM: SVM: Flush everything if FLUSHBYASID is not available Date: Thu, 13 Mar 2025 21:55:37 +0000 Message-ID: <20250313215540.4171762-5-yosry.ahmed@linux.dev> In-Reply-To: <20250313215540.4171762-1-yosry.ahmed@linux.dev> References: <20250313215540.4171762-1-yosry.ahmed@linux.dev> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Currently, if FLUSHBYASID is not available when performing a TLB flush, the fallback is decrementing the ASID generation to trigger allocating a new ASID. In preparation for using a static ASID per VM, just fallback to flushing everything if FLUSHBYASID is not available. This is probably worse from a performance perspective, but FLUSHBYASID has been around for ~15 years and it's not worth carrying the complexity. The fallback logic is moved within svm_vmcb_set_flush_asid(), as more callers will be added and will need the fallback as well. Suggested-by: Sean Christopherson Signed-off-by: Yosry Ahmed --- arch/x86/kvm/svm/svm.c | 5 +---- arch/x86/kvm/svm/svm.h | 5 ++++- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index 8c90686a33f44..e5064fbefb822 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -4005,10 +4005,7 @@ static void svm_flush_tlb_asid(struct kvm_vcpu *vcpu) * unconditionally does a TLB flush on both nested VM-Enter and nested * VM-Exit (via kvm_mmu_reset_context()). */ - if (static_cpu_has(X86_FEATURE_FLUSHBYASID)) - svm_vmcb_set_flush_asid(svm->vmcb); - else - svm->current_vmcb->asid_generation--; + svm_vmcb_set_flush_asid(svm->vmcb); } static void svm_flush_tlb_current(struct kvm_vcpu *vcpu) diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h index 9fd5b249b9c19..0f6426809e1b9 100644 --- a/arch/x86/kvm/svm/svm.h +++ b/arch/x86/kvm/svm/svm.h @@ -645,7 +645,10 @@ void svm_complete_interrupt_delivery(struct kvm_vcpu *vcpu, int delivery_mode, static inline void svm_vmcb_set_flush_asid(struct vmcb *vmcb) { - vmcb->control.tlb_ctl = TLB_CONTROL_FLUSH_ASID; + if (static_cpu_has(X86_FEATURE_FLUSHBYASID)) + vmcb->control.tlb_ctl = TLB_CONTROL_FLUSH_ASID; + else + vmcb->control.tlb_ctl = TLB_CONTROL_FLUSH_ALL_ASID; } static inline void svm_vmcb_clear_flush_asid(struct vmcb *vmcb) From patchwork Thu Mar 13 21:55:38 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 14015957 Received: from out-186.mta0.migadu.com (out-186.mta0.migadu.com [91.218.175.186]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5C91E1F758F for ; Thu, 13 Mar 2025 21:56:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.186 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741902968; cv=none; b=ld79IN2cOnZxaZ2rU59rR6biCbuRnM730oBrbXDb1ugzmDgW4Jrb42LyTn6++wJopKLIqvr6IeBgS4AJ8qK11ya56yP+UMVbGb9PWN7tPvGQxhrzEDIsGgUWF2rzIRSqkl4KMOQ9UwLBIg7S3nPYAP4vqhdW2ZMsUCtYiyHXSuc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741902968; c=relaxed/simple; bh=8wECG4JQ2fba//svmJ31tdiX1CS6jJ9gPveOAmkxM84=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=EdnUc4DFGQqd0M/eBlkhhRQ4UDPj8qIIj8nkiMzXCbnUbwWLEs1JvwOOHFprwwvwCzfljYGKS1o463l/XpPRjS5MyO2LDAEoZngyLpQ3IwP+soWjg4/VhhbnIlcvMje+qF/XhhU/bkXoeVttnLjvOznm2vjs2Pa5h251NmazZsQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=aLhl171H; arc=none smtp.client-ip=91.218.175.186 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="aLhl171H" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1741902962; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SGGkSGip9KY3u6EkXyE1FS/Ys6OM8b0hhAlvTYZWAS0=; b=aLhl171HN7X/zTuTz2xwy45+nCXqc3zC3/b4OU/z1teoXnbh3I6bdtY/VhwwR0D5Rzaw98 wk4GqJJlrb1/9YxDtI5IagU1MpHJ6QdSYhHAHdoZAe9gx+oSSjg/kozTFqiaa8a/N5s5LC KS7oR0mfIlcS0GMv4Sqs1ZwIKVfxKxQ= From: Yosry Ahmed To: Sean Christopherson Cc: Paolo Bonzini , Maxim Levitsky , x86@kernel.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed Subject: [PATCH 5/7] KVM: SVM: Flush the ASID when running on a new CPU Date: Thu, 13 Mar 2025 21:55:38 +0000 Message-ID: <20250313215540.4171762-6-yosry.ahmed@linux.dev> In-Reply-To: <20250313215540.4171762-1-yosry.ahmed@linux.dev> References: <20250313215540.4171762-1-yosry.ahmed@linux.dev> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Currently, when a vCPU is migrated to a new physical CPU, the ASID generation is reset to trigger allocating a new ASID. In preparation for using a static ASID per VM, just flush the ASID in this case (falling back to flushing everything if FLUSBYASID is not available). Suggested-by: Sean Christopherson Signed-off-by: Yosry Ahmed --- arch/x86/kvm/svm/svm.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index e5064fbefb822..b8a3fc81fc9c8 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -3626,12 +3626,12 @@ static int pre_svm_run(struct kvm_vcpu *vcpu) struct vcpu_svm *svm = to_svm(vcpu); /* - * If the previous vmrun of the vmcb occurred on a different physical - * cpu, then mark the vmcb dirty and assign a new asid. Hardware's - * vmcb clean bits are per logical CPU, as are KVM's asid assignments. + * If the previous VMRUN of the VMCB occurred on a different physical + * CPU, then mark the VMCB dirty and flush the ASID. Hardware's + * VMCB clean bits are per logical CPU, as are KVM's ASID assignments. */ if (unlikely(svm->current_vmcb->cpu != vcpu->cpu)) { - svm->current_vmcb->asid_generation = 0; + svm_vmcb_set_flush_asid(svm->vmcb); vmcb_mark_all_dirty(svm->vmcb); svm->current_vmcb->cpu = vcpu->cpu; } From patchwork Thu Mar 13 21:55:39 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 14015958 Received: from out-180.mta0.migadu.com (out-180.mta0.migadu.com [91.218.175.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D5E151FA165 for ; Thu, 13 Mar 2025 21:56:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741902969; cv=none; b=d2j2L+qLqeHFDj+BntX4T//8z9RdtEFiOSY4LmXhSkSBpb+HhSbb/1SFMCYuhL0V3bOpQk7qRVwdC5QyR+RhN3Kaz8eqeUA75s34EBck4MrSujR4PPpXdeovPjNHEUMKgMcC85oVEeb6fuUiMTlupW9W/pDq1j0BmNz6oq31Xy0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741902969; c=relaxed/simple; bh=KZF6AG6C1NKjQm0Gj4rRLGWi5sL6P3Jw6TBX+kZmt9s=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=tIWmsV6KKm3G7oEO7mFMf1yI4CRtiHZtTg8FAFTHpGfNHtTvhB2i1ONy10f+TeZKi//ytXntbiFZld2txgxyFi34yJt5zFjdnma00kQFefjijMDKk7ob3AdoSwLCyuXhoZKj7IyPMqRZpIPas/GH8dyBGo3xmgrRuu13yhyDtcc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=N/iYxiqQ; arc=none smtp.client-ip=91.218.175.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="N/iYxiqQ" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1741902964; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SA+kqYv+of311iNb51ZDSMltMHcQFYpEsM71pONh+2A=; b=N/iYxiqQCVTsYE2qWVkMvlqeBDTErmMT/ZAZJvrRH1ioaHgxDxlmsc1wm4ynYAlr+JUEHF HFuZ/VbVFxZ27nnZoDdPdpj3NVRzPui4AJG97LNlv8epoDHClW+8w6R4ilB/KjKlGEprcX aXwUGmOBeQcZLDkePQzaWIfbmWWlWxg= From: Yosry Ahmed To: Sean Christopherson Cc: Paolo Bonzini , Maxim Levitsky , x86@kernel.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed Subject: [PATCH 6/7] KVM: SVM: Use a single ASID per VM Date: Thu, 13 Mar 2025 21:55:39 +0000 Message-ID: <20250313215540.4171762-7-yosry.ahmed@linux.dev> In-Reply-To: <20250313215540.4171762-1-yosry.ahmed@linux.dev> References: <20250313215540.4171762-1-yosry.ahmed@linux.dev> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT The ASID generation and dynamic ASID allocation logic is now only used by initialization the generation to 0 to trigger a new ASID allocation per-vCPU on the first VMRUN, so the ASID is more-or-less static per-vCPU. Moreover, having a unique ASID per-vCPU is not required. ASIDs are local to each physical CPU, and the ASID is flushed when a vCPU is migrated to a new physical CPU anyway. SEV VMs have been using a single ASID per VM already for other reasons. Use a static ASID per VM and drop the dynamic ASID allocation logic. The ASID is allocated during vCPU reset (SEV allocates its own ASID), and the ASID is always flushed on first use in case it was used by another VM previously. On VMRUN, WARN if the ASID in the VMCB does not match the VM's ASID, and update it accordingly. Also, flush the ASID on every VMRUN if the VM failed to allocate a unique ASID. This would probably wreck performance if it happens, but it should be an edge case as most AMD CPUs have over 32k ASIDs. Suggested-by: Sean Christopherson Signed-off-by: Yosry Ahmed --- arch/x86/kvm/svm/nested.c | 2 ++ arch/x86/kvm/svm/sev.c | 7 +++-- arch/x86/kvm/svm/svm.c | 60 +++++++++++++++++++++------------------ arch/x86/kvm/svm/svm.h | 8 +----- 4 files changed, 40 insertions(+), 37 deletions(-) diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c index 3bff948bc5752..d6a07644c8734 100644 --- a/arch/x86/kvm/svm/nested.c +++ b/arch/x86/kvm/svm/nested.c @@ -643,6 +643,7 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm, struct kvm_vcpu *vcpu = &svm->vcpu; struct vmcb *vmcb01 = svm->vmcb01.ptr; struct vmcb *vmcb02 = svm->nested.vmcb02.ptr; + struct kvm_svm *kvm_svm = to_kvm_svm(vcpu->kvm); u32 pause_count12; u32 pause_thresh12; @@ -677,6 +678,7 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm, vmcb02->control.nested_ctl = vmcb01->control.nested_ctl; vmcb02->control.iopm_base_pa = vmcb01->control.iopm_base_pa; vmcb02->control.msrpm_base_pa = vmcb01->control.msrpm_base_pa; + vmcb02->control.asid = kvm_svm->asid; /* Done at vmrun: asid. */ diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c index b393674733969..1ee04d6b9356b 100644 --- a/arch/x86/kvm/svm/sev.c +++ b/arch/x86/kvm/svm/sev.c @@ -3465,8 +3465,10 @@ int pre_sev_run(struct vcpu_svm *svm, int cpu) if (sev_es_guest(kvm) && !VALID_PAGE(svm->vmcb->control.vmsa_pa)) return -EINVAL; - /* Assign the asid allocated with this SEV guest */ - svm->asid = asid; + if (WARN_ON_ONCE(svm->vmcb->control.asid != asid)) { + svm->vmcb->control.asid = asid; + vmcb_mark_dirty(svm->vmcb, VMCB_ASID); + } /* * Flush guest TLB: @@ -4509,6 +4511,7 @@ static void sev_es_init_vmcb(struct vcpu_svm *svm) void sev_init_vmcb(struct vcpu_svm *svm) { svm->vmcb->control.nested_ctl |= SVM_NESTED_CTL_SEV_ENABLE; + svm->vmcb->control.asid = sev_get_asid(svm->vcpu.kvm); clr_exception_intercept(svm, UD_VECTOR); /* diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index b8a3fc81fc9c8..c5e2733fb856d 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -249,6 +249,8 @@ static unsigned long iopm_base; DEFINE_PER_CPU(struct svm_cpu_data, svm_data); +static struct kvm_tlb_tags svm_asids; + /* * Only MSR_TSC_AUX is switched via the user return hook. EFER is switched via * the VMCB, and the SYSCALL/SYSENTER MSRs are handled by VMLOAD/VMSAVE. @@ -621,10 +623,6 @@ static int svm_enable_virtualization_cpu(void) return -EBUSY; sd = per_cpu_ptr(&svm_data, me); - sd->asid_generation = 1; - sd->max_asid = cpuid_ebx(SVM_CPUID_FUNC) - 1; - sd->next_asid = sd->max_asid + 1; - sd->min_asid = max_sev_asid + 1; wrmsrl(MSR_EFER, efer | EFER_SVME); @@ -1126,6 +1124,7 @@ static void svm_hardware_unsetup(void) __free_pages(__sme_pa_to_page(iopm_base), get_order(IOPM_SIZE)); iopm_base = 0; + kvm_tlb_tags_destroy(&svm_asids); } static void init_seg(struct vmcb_seg *seg) @@ -1234,6 +1233,7 @@ static inline void init_vmcb_after_set_cpuid(struct kvm_vcpu *vcpu) static void init_vmcb(struct kvm_vcpu *vcpu) { + struct kvm_svm *kvm_svm = to_kvm_svm(vcpu->kvm); struct vcpu_svm *svm = to_svm(vcpu); struct vmcb *vmcb = svm->vmcb01.ptr; struct vmcb_control_area *control = &vmcb->control; @@ -1339,8 +1339,6 @@ static void init_vmcb(struct kvm_vcpu *vcpu) save->g_pat = vcpu->arch.pat; save->cr3 = 0; } - svm->current_vmcb->asid_generation = 0; - svm->asid = 0; svm->nested.vmcb12_gpa = INVALID_GPA; svm->nested.last_vmcb12_gpa = INVALID_GPA; @@ -1375,8 +1373,14 @@ static void init_vmcb(struct kvm_vcpu *vcpu) control->int_ctl |= V_GIF_ENABLE_MASK; } - if (sev_guest(vcpu->kvm)) + /* sev_init_vmcb() will assign its own ASID */ + if (sev_guest(vcpu->kvm)) { sev_init_vmcb(svm); + WARN_ON_ONCE(!control->asid); + } else { + control->asid = kvm_svm->asid; + svm_vmcb_set_flush_asid(svm->vmcb); + } svm_hv_init_vmcb(vmcb); init_vmcb_after_set_cpuid(vcpu); @@ -1982,19 +1986,6 @@ static void svm_update_exception_bitmap(struct kvm_vcpu *vcpu) } } -static void new_asid(struct vcpu_svm *svm, struct svm_cpu_data *sd) -{ - if (sd->next_asid > sd->max_asid) { - ++sd->asid_generation; - sd->next_asid = sd->min_asid; - svm->vmcb->control.tlb_ctl = TLB_CONTROL_FLUSH_ALL_ASID; - vmcb_mark_dirty(svm->vmcb, VMCB_ASID); - } - - svm->current_vmcb->asid_generation = sd->asid_generation; - svm->asid = sd->next_asid++; -} - static void svm_set_dr6(struct kvm_vcpu *vcpu, unsigned long value) { struct vmcb *vmcb = to_svm(vcpu)->vmcb; @@ -3622,7 +3613,7 @@ static int svm_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath) static int pre_svm_run(struct kvm_vcpu *vcpu) { - struct svm_cpu_data *sd = per_cpu_ptr(&svm_data, vcpu->cpu); + struct kvm_svm *kvm_svm = to_kvm_svm(vcpu->kvm); struct vcpu_svm *svm = to_svm(vcpu); /* @@ -3639,9 +3630,15 @@ static int pre_svm_run(struct kvm_vcpu *vcpu) if (sev_guest(vcpu->kvm)) return pre_sev_run(svm, vcpu->cpu); - /* FIXME: handle wraparound of asid_generation */ - if (svm->current_vmcb->asid_generation != sd->asid_generation) - new_asid(svm, sd); + /* Flush the ASID on every VMRUN if kvm_svm->asid allocation failed */ + if (unlikely(!kvm_svm->asid)) + svm_vmcb_set_flush_asid(svm->vmcb); + + if (WARN_ON_ONCE(svm->vmcb->control.asid != kvm_svm->asid)) { + svm_vmcb_set_flush_asid(svm->vmcb); + svm->vmcb->control.asid = kvm_svm->asid; + vmcb_mark_dirty(svm->vmcb, VMCB_ASID); + } return 0; } @@ -4289,10 +4286,6 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu, sync_lapic_to_cr8(vcpu); - if (unlikely(svm->asid != svm->vmcb->control.asid)) { - svm->vmcb->control.asid = svm->asid; - vmcb_mark_dirty(svm->vmcb, VMCB_ASID); - } svm->vmcb->save.cr2 = vcpu->arch.cr2; svm_hv_update_vp_id(svm->vmcb, vcpu); @@ -5024,12 +5017,16 @@ static void svm_vcpu_deliver_sipi_vector(struct kvm_vcpu *vcpu, u8 vector) static void svm_vm_destroy(struct kvm *kvm) { + struct kvm_svm *kvm_svm = to_kvm_svm(kvm); + avic_vm_destroy(kvm); sev_vm_destroy(kvm); + kvm_tlb_tags_free(&svm_asids, kvm_svm->asid); } static int svm_vm_init(struct kvm *kvm) { + struct kvm_svm *kvm_svm = to_kvm_svm(kvm); int type = kvm->arch.vm_type; if (type != KVM_X86_DEFAULT_VM && @@ -5051,6 +5048,7 @@ static int svm_vm_init(struct kvm *kvm) return ret; } + kvm_svm->asid = kvm_tlb_tags_alloc(&svm_asids); return 0; } @@ -5332,6 +5330,7 @@ static __init int svm_hardware_setup(void) void *iopm_va; int r; unsigned int order = get_order(IOPM_SIZE); + unsigned int min_asid, max_asid; /* * NX is required for shadow paging and for NPT if the NX huge pages @@ -5424,6 +5423,11 @@ static __init int svm_hardware_setup(void) */ sev_hardware_setup(); + /* Consumes max_sev_asid initialized by sev_hardware_setup() */ + min_asid = max_sev_asid + 1; + max_asid = cpuid_ebx(SVM_CPUID_FUNC) - 1; + kvm_tlb_tags_init(&svm_asids, min_asid, max_asid); + svm_hv_hardware_setup(); for_each_possible_cpu(cpu) { diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h index 0f6426809e1b9..4c6664ba4048d 100644 --- a/arch/x86/kvm/svm/svm.h +++ b/arch/x86/kvm/svm/svm.h @@ -119,6 +119,7 @@ struct kvm_svm { /* Struct members for AVIC */ u32 avic_vm_id; + unsigned int asid; struct page *avic_logical_id_table_page; struct page *avic_physical_id_table_page; struct hlist_node hnode; @@ -132,7 +133,6 @@ struct kvm_vmcb_info { struct vmcb *ptr; unsigned long pa; int cpu; - uint64_t asid_generation; }; struct vmcb_save_area_cached { @@ -247,7 +247,6 @@ struct vcpu_svm { struct vmcb *vmcb; struct kvm_vmcb_info vmcb01; struct kvm_vmcb_info *current_vmcb; - u32 asid; u32 sysenter_esp_hi; u32 sysenter_eip_hi; uint64_t tsc_aux; @@ -330,11 +329,6 @@ struct vcpu_svm { }; struct svm_cpu_data { - u64 asid_generation; - u32 max_asid; - u32 next_asid; - u32 min_asid; - struct vmcb *save_area; unsigned long save_area_pa; From patchwork Thu Mar 13 21:55:40 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 14015959 Received: from out-179.mta0.migadu.com (out-179.mta0.migadu.com [91.218.175.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C23901FA854 for ; Thu, 13 Mar 2025 21:56:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741902971; cv=none; b=ns9gtuHo7iQQQVs7S3FMXqP3rboxMIZWSnwmzD6JSRV9tHtZkEVxeH8qOx8P88G5Cla3TkGkGcppdOwTckluPxUOQChWqyhoWyB2/njtltC0J3zwn3A3kozQipTohY8aOMhDrqXWYGpBoAtYVFXX6EIxtfiPsp+a4f/xyTgfh7s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741902971; c=relaxed/simple; bh=0t+zX0HpA8zT88d4wD6BeZXcNvONGsx3iB3nDx8sBHU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=MiC3cfG3/dYFuSnpElTUb79veVARkgTlSxWrWsHsyESBH5VHds2PEtyUHUk3fA23vYDJhHKJwobc5Q83ssqjevVSnAioiLvj/q864m/F9BBOknDkVqAZAzHxgyk82bt0a5QbFOoZofcnXp9kXf2PVQeWi4rFPJkEbVPoerc+02A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=lICVW+tK; arc=none smtp.client-ip=91.218.175.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="lICVW+tK" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1741902966; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=02QFPfOHKNlwJVYXcIvUHxvnbWRxOfELTFB2ejfTyvY=; b=lICVW+tKNQLbo3DdJ0O57TAfgeNd/mBKvfCyhNmUiNES939D0kwmknmM43I4U+lf/tMKmN 00FzrxMSQeYM1O4syIWifBj1t4Bi3Db0C9Qzfm5TcUacoXiKRuWK/Rt3i5eUX3lU+38070 1Awydq35vC+xqGC/YiX/Ur5aM4wtzpg= From: Yosry Ahmed To: Sean Christopherson Cc: Paolo Bonzini , Maxim Levitsky , x86@kernel.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed Subject: [PATCH 7/7] KVM: SVM: Share more code between pre_sev_run() and pre_svm_run() Date: Thu, 13 Mar 2025 21:55:40 +0000 Message-ID: <20250313215540.4171762-8-yosry.ahmed@linux.dev> In-Reply-To: <20250313215540.4171762-1-yosry.ahmed@linux.dev> References: <20250313215540.4171762-1-yosry.ahmed@linux.dev> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT pre_svm_run() and pre_sev_run() now do some redundant work, and the control flow is not super clear. Specifically: - Both functions check if the ASID in the VMCB is the expected one. - Both functions check if the vCPU moved to a different physical CPU. - Both functions issue an ASID TLB flush if needed. Pass the ASID and whether or not SEV requires a TLB flush from pre_sev_run() to pre_svm_run(), and use the logic there instead. pre_sev_run() now only performs SEV-specific checks. Note that pre_sev_run() used svm->vcpu.arch.last_vmentry_cpu to check if the vCPU moved to a different physical CPU, while pre_svm_run uses svm->current_vmcb->cpu. The former tracks the CPU per vCPU, while the latter tracks it per VMCB. For SEV, they both should be equivalent since there is a single VMCB per-vCPU (nested is not supported). Signed-off-by: Yosry Ahmed --- arch/x86/kvm/svm/sev.c | 27 ++++++++++----------------- arch/x86/kvm/svm/svm.c | 10 ++++++---- arch/x86/kvm/svm/svm.h | 2 +- 3 files changed, 17 insertions(+), 22 deletions(-) diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c index 1ee04d6b9356b..607139757f8ff 100644 --- a/arch/x86/kvm/svm/sev.c +++ b/arch/x86/kvm/svm/sev.c @@ -3451,11 +3451,11 @@ void sev_es_unmap_ghcb(struct vcpu_svm *svm) svm->sev_es.ghcb = NULL; } -int pre_sev_run(struct vcpu_svm *svm, int cpu) +int pre_sev_run(struct vcpu_svm *svm, unsigned int *asid, bool *need_flush) { + int cpu = svm->vcpu.cpu; struct svm_cpu_data *sd = per_cpu_ptr(&svm_data, cpu); struct kvm *kvm = svm->vcpu.kvm; - unsigned int asid = sev_get_asid(kvm); /* * Reject KVM_RUN if userspace attempts to run the vCPU with an invalid @@ -3465,24 +3465,17 @@ int pre_sev_run(struct vcpu_svm *svm, int cpu) if (sev_es_guest(kvm) && !VALID_PAGE(svm->vmcb->control.vmsa_pa)) return -EINVAL; - if (WARN_ON_ONCE(svm->vmcb->control.asid != asid)) { - svm->vmcb->control.asid = asid; - vmcb_mark_dirty(svm->vmcb, VMCB_ASID); - } - /* - * Flush guest TLB: - * - * 1) when different VMCB for the same ASID is to be run on the same host CPU. - * 2) or this VMCB was executed on different host CPU in previous VMRUNs. + * Flush the guest TLB when a difference VMCB for the same ASID is to be + * run on the same host CPU. The caller will also flush the TLB if the + * VMCB was executed on a different host CPU in previous VMRUNs. */ - if (sd->sev_vmcbs[asid] == svm->vmcb && - svm->vcpu.arch.last_vmentry_cpu == cpu) - return 0; + *asid = sev_get_asid(kvm); + if (sd->sev_vmcbs[*asid] != svm->vmcb) { + sd->sev_vmcbs[*asid] = svm->vmcb; + *need_flush = true; + } - sd->sev_vmcbs[asid] = svm->vmcb; - svm_vmcb_set_flush_asid(svm->vmcb); - vmcb_mark_dirty(svm->vmcb, VMCB_ASID); return 0; } diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index c5e2733fb856d..6b338d84e7b93 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -3615,21 +3615,23 @@ static int pre_svm_run(struct kvm_vcpu *vcpu) { struct kvm_svm *kvm_svm = to_kvm_svm(vcpu->kvm); struct vcpu_svm *svm = to_svm(vcpu); + unsigned int asid = kvm_svm->asid; + bool sev_need_flush = false; + + if (sev_guest(vcpu->kvm) && pre_sev_run(svm, &asid, &sev_need_flush)) + return -1; /* * If the previous VMRUN of the VMCB occurred on a different physical * CPU, then mark the VMCB dirty and flush the ASID. Hardware's * VMCB clean bits are per logical CPU, as are KVM's ASID assignments. */ - if (unlikely(svm->current_vmcb->cpu != vcpu->cpu)) { + if (unlikely(sev_need_flush || svm->current_vmcb->cpu != vcpu->cpu)) { svm_vmcb_set_flush_asid(svm->vmcb); vmcb_mark_all_dirty(svm->vmcb); svm->current_vmcb->cpu = vcpu->cpu; } - if (sev_guest(vcpu->kvm)) - return pre_sev_run(svm, vcpu->cpu); - /* Flush the ASID on every VMRUN if kvm_svm->asid allocation failed */ if (unlikely(!kvm_svm->asid)) svm_vmcb_set_flush_asid(svm->vmcb); diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h index 4c6664ba4048d..f25e99c79d07d 100644 --- a/arch/x86/kvm/svm/svm.h +++ b/arch/x86/kvm/svm/svm.h @@ -754,7 +754,7 @@ void avic_refresh_virtual_apic_mode(struct kvm_vcpu *vcpu); /* sev.c */ -int pre_sev_run(struct vcpu_svm *svm, int cpu); +int pre_sev_run(struct vcpu_svm *svm, unsigned int *asid, bool *need_flush); void sev_init_vmcb(struct vcpu_svm *svm); void sev_vcpu_after_set_cpuid(struct vcpu_svm *svm); int sev_es_string_io(struct vcpu_svm *svm, int size, unsigned int port, int in);