From patchwork Wed Feb 5 18:23:50 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13961701 Received: from out-179.mta0.migadu.com (out-179.mta0.migadu.com [91.218.175.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CC9C9193436 for ; Wed, 5 Feb 2025 18:24:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738779863; cv=none; b=jd4pZDy+a9yTKJCWPPyAq1ggSgbCuCYulvi37yWWbxL3a/oCZDUuT1XYNWREVdVWCMBf8s8Y8I4sJg8ZILbWgMp+C2O+DnRM//77zqovLroYsiGjQAB5Mt5/pRhC2DImhTObL6jkbi7fzATtxgRnq8nXJRctj9MWKczuqmzSvlw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738779863; c=relaxed/simple; bh=1rJz09m5d3GNK9R8RloegGPmL0GypyAT2cL3qxMRJ7E=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=UOo7jZpT0H7XVsDnxjS+yUfYgEw/lhteJYUI0QUJ3o/1Gpg0HJoiPJJoxFuJzyKSAntihP40C1ehGCXFjD1nxhQ9dAk2k9r6z1uNsH4Qiq63tbG8VP9dOSFi9NWmiRCMZz50q2U3NfcBoHlgtFk7oLg9OxZ+vH2sEnF+cBBsV6g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=ktlYzEGk; arc=none smtp.client-ip=91.218.175.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="ktlYzEGk" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1738779858; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=6BJ3Tmh6YFKL8BG0vfP0eplTWJclNBXEO176k3JDmdA=; b=ktlYzEGkyGOtoo6Qo9pwF0NojdVFk1dWuEK/8+Vi/qOpTqNI+RuNmMvuW4lUM88aeSJmtV /bSe1EIM7zyZPgQJeGRE1Pe/3biGH3+ub3oFSUZgH9Zxky+3DEcS9KEPB+sZO6Fqz+xujl LpZHs9tqf2CNGZ3gmX9kHUvfFWK3iug= From: Yosry Ahmed To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH 01/13] KVM: nSVM: Track the ASID per-VMCB Date: Wed, 5 Feb 2025 18:23:50 +0000 Message-ID: <20250205182402.2147495-2-yosry.ahmed@linux.dev> In-Reply-To: <20250205182402.2147495-1-yosry.ahmed@linux.dev> References: <20250205182402.2147495-1-yosry.ahmed@linux.dev> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT The ASID is currently tracked per-vCPU, because the same ASID is used by L1 and L2. That ASID is flushed on every transition between L1 and L2. Track the ASID separately for each VMCB (similar to the asid_generation), giving L2 a separate ASID. This is in preparation for doing fine-grained TLB flushes on nested transitions instead of unconditional full flushes. The ASIDs are still not fully maintained (e.g. a remote flush will only flush the current ASID), so keep the TLB flush on every transition until this is sorted out. L1's ASID will be flushed on KVM_REQ_TLB_FLUSH_GUEST if it is the active context, so remove the TODO in nested_svm_transition_tlb_flush() about it. Signed-off-by: Yosry Ahmed --- arch/x86/kvm/svm/nested.c | 1 - arch/x86/kvm/svm/sev.c | 2 +- arch/x86/kvm/svm/svm.c | 12 +++++++----- arch/x86/kvm/svm/svm.h | 2 +- 4 files changed, 9 insertions(+), 8 deletions(-) diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c index 04c375bf1ac2a..bbe4f3ac9f250 100644 --- a/arch/x86/kvm/svm/nested.c +++ b/arch/x86/kvm/svm/nested.c @@ -495,7 +495,6 @@ static void nested_svm_transition_tlb_flush(struct kvm_vcpu *vcpu) * - Honor L1's request to flush an ASID on nested VMRUN * - Sync nested NPT MMU on VMRUN that flushes L2's ASID[*] * - Don't crush a pending TLB flush in vmcb02 on nested VMRUN - * - Flush L1's ASID on KVM_REQ_TLB_FLUSH_GUEST * * [*] Unlike nested EPT, SVM's ASID management can invalidate nested * NPT guest-physical mappings on VMRUN. diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c index 799f8494b599c..b0adfd0537d00 100644 --- a/arch/x86/kvm/svm/sev.c +++ b/arch/x86/kvm/svm/sev.c @@ -3468,7 +3468,7 @@ void pre_sev_run(struct vcpu_svm *svm, int cpu) unsigned int asid = sev_get_asid(svm->vcpu.kvm); /* Assign the asid allocated with this SEV guest */ - svm->asid = asid; + svm->current_vmcb->asid = asid; /* * Flush guest TLB: diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index 7640a84e554a6..08340ae57777b 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -1335,8 +1335,10 @@ static void init_vmcb(struct kvm_vcpu *vcpu) save->g_pat = vcpu->arch.pat; save->cr3 = 0; } - svm->current_vmcb->asid_generation = 0; - svm->asid = 0; + svm->vmcb01.asid_generation = 0; + svm->vmcb01.asid = 0; + svm->nested.vmcb02.asid_generation = 0; + svm->nested.vmcb02.asid = 0; svm->nested.vmcb12_gpa = INVALID_GPA; svm->nested.last_vmcb12_gpa = INVALID_GPA; @@ -1988,7 +1990,7 @@ static void new_asid(struct vcpu_svm *svm, struct svm_cpu_data *sd) } svm->current_vmcb->asid_generation = sd->asid_generation; - svm->asid = sd->next_asid++; + svm->current_vmcb->asid = sd->next_asid++; } static void svm_set_dr6(struct vcpu_svm *svm, unsigned long value) @@ -4235,8 +4237,8 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu, sync_lapic_to_cr8(vcpu); - if (unlikely(svm->asid != svm->vmcb->control.asid)) { - svm->vmcb->control.asid = svm->asid; + if (unlikely(svm->current_vmcb->asid != svm->vmcb->control.asid)) { + svm->vmcb->control.asid = svm->current_vmcb->asid; vmcb_mark_dirty(svm->vmcb, VMCB_ASID); } svm->vmcb->save.cr2 = vcpu->arch.cr2; diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h index 9d7cdb8fbf872..ebbb0b1a64676 100644 --- a/arch/x86/kvm/svm/svm.h +++ b/arch/x86/kvm/svm/svm.h @@ -133,6 +133,7 @@ struct kvm_vmcb_info { unsigned long pa; int cpu; uint64_t asid_generation; + u32 asid; }; struct vmcb_save_area_cached { @@ -247,7 +248,6 @@ struct vcpu_svm { struct vmcb *vmcb; struct kvm_vmcb_info vmcb01; struct kvm_vmcb_info *current_vmcb; - u32 asid; u32 sysenter_esp_hi; u32 sysenter_eip_hi; uint64_t tsc_aux; From patchwork Wed Feb 5 18:23:51 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13961703 Received: from out-184.mta0.migadu.com (out-184.mta0.migadu.com [91.218.175.184]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 43AEB1FCF5B for ; Wed, 5 Feb 2025 18:24:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.184 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738779866; cv=none; b=bmBvADyD6eaerQ3rUJfHA4xZjtgwyJ4iUhxozJ8l2etIb9vCb91mSuvu6GFIiocnws/S2BCIdRJnnY7tJPDoO6pH/+NX7F4AKNeryGSBiPRfmn4aTjMw+S195JEjwq6A3m8PiMPy7GC2W8Ck8jhqplHNOxpVLKMLmv3y0mJKw8U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738779866; c=relaxed/simple; bh=R5/nFuCIKeEoK+vhrKdLcHFTynpfcJCswTg526aSFEQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=rPZ/RSU2eY2/FKJBEKYgaEPjJoh8Gr0PWOCSaUvW3p4lWpeCsFlXWI4KPLNi0ILGUrsVht6HvT/jtutRxQvce4ELhJqR3E8/QtJ4/JmHb3SJigwLJyO+qDvFFjCl5Z8njlN+SafrxRn29zGzncVM9phxHbMrMYaueeDKwNQJMYs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=IDCcqaR/; arc=none smtp.client-ip=91.218.175.184 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="IDCcqaR/" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1738779860; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QuU9TX0aBbgsPkUOX8eud2LZOZtDW9gXXMR6We2Iqqk=; b=IDCcqaR/8eyZ1PioXmRiXNValHTSJufnBPh6l5uHNkCtl8HlsxgtqD8UOvGvKSQk8vvziQ iA2rn4XBucndA/NEdMP3Z7pCVHzsGp2HFCdr2FF03J9MFvHeRFkeKS/QQt85GgF7O59Joi Lo4BaDXpSLefu+FdGOZqGPpomtwUulo= From: Yosry Ahmed To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH 02/13] KVM: nSVM: Rework svm_flush_tlb_asid() to operate on a given VMCB Date: Wed, 5 Feb 2025 18:23:51 +0000 Message-ID: <20250205182402.2147495-3-yosry.ahmed@linux.dev> In-Reply-To: <20250205182402.2147495-1-yosry.ahmed@linux.dev> References: <20250205182402.2147495-1-yosry.ahmed@linux.dev> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT svm_flush_tlb_asid() currently operates on the current VMCB. In preparation for properly tracking TLB flushes for L1 and L2 ASIDs, refactor it to work on a given VMCB. All existing callers pass the current VMCB. Create a svm_flush_tlb_guest() wrapper to use as the flush_tlb_guest() callback. kvm_hv_vcpu_purge_flush_tlb() is only called when the current VMCB is passed to maintain current behavior. No functional change intended. Signed-off-by: Yosry Ahmed --- arch/x86/kvm/svm/svm.c | 25 ++++++++++++++++++------- 1 file changed, 18 insertions(+), 7 deletions(-) diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index 08340ae57777b..2108b48ba4959 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -3954,7 +3954,7 @@ static void svm_enable_nmi_window(struct kvm_vcpu *vcpu) svm->vmcb->save.rflags |= (X86_EFLAGS_TF | X86_EFLAGS_RF); } -static void svm_flush_tlb_asid(struct kvm_vcpu *vcpu) +static void svm_flush_tlb_asid(struct kvm_vcpu *vcpu, struct kvm_vmcb_info *vmcb) { struct vcpu_svm *svm = to_svm(vcpu); @@ -3963,7 +3963,8 @@ static void svm_flush_tlb_asid(struct kvm_vcpu *vcpu) * A TLB flush for the current ASID flushes both "host" and "guest" TLB * entries, and thus is a superset of Hyper-V's fine grained flushing. */ - kvm_hv_vcpu_purge_flush_tlb(vcpu); + if (vmcb == svm->current_vmcb) + kvm_hv_vcpu_purge_flush_tlb(vcpu); /* * Flush only the current ASID even if the TLB flush was invoked via @@ -3973,14 +3974,15 @@ static void svm_flush_tlb_asid(struct kvm_vcpu *vcpu) * VM-Exit (via kvm_mmu_reset_context()). */ if (static_cpu_has(X86_FEATURE_FLUSHBYASID)) - svm->vmcb->control.tlb_ctl = TLB_CONTROL_FLUSH_ASID; + vmcb->ptr->control.tlb_ctl = TLB_CONTROL_FLUSH_ASID; else - svm->current_vmcb->asid_generation--; + vmcb->asid_generation--; } static void svm_flush_tlb_current(struct kvm_vcpu *vcpu) { hpa_t root_tdp = vcpu->arch.mmu->root.hpa; + struct vcpu_svm *svm = to_svm(vcpu); /* * When running on Hyper-V with EnlightenedNptTlb enabled, explicitly @@ -3991,11 +3993,13 @@ static void svm_flush_tlb_current(struct kvm_vcpu *vcpu) if (svm_hv_is_enlightened_tlb_enabled(vcpu) && VALID_PAGE(root_tdp)) hyperv_flush_guest_mapping(root_tdp); - svm_flush_tlb_asid(vcpu); + svm_flush_tlb_asid(vcpu, svm->current_vmcb); } static void svm_flush_tlb_all(struct kvm_vcpu *vcpu) { + struct vcpu_svm *svm = to_svm(vcpu); + /* * When running on Hyper-V with EnlightenedNptTlb enabled, remote TLB * flushes should be routed to hv_flush_remote_tlbs() without requesting @@ -4006,7 +4010,7 @@ static void svm_flush_tlb_all(struct kvm_vcpu *vcpu) if (WARN_ON_ONCE(svm_hv_is_enlightened_tlb_enabled(vcpu))) hv_flush_remote_tlbs(vcpu->kvm); - svm_flush_tlb_asid(vcpu); + svm_flush_tlb_asid(vcpu, svm->current_vmcb); } static void svm_flush_tlb_gva(struct kvm_vcpu *vcpu, gva_t gva) @@ -4016,6 +4020,13 @@ static void svm_flush_tlb_gva(struct kvm_vcpu *vcpu, gva_t gva) invlpga(gva, svm->vmcb->control.asid); } +static void svm_flush_tlb_guest(struct kvm_vcpu *vcpu) +{ + struct vcpu_svm *svm = to_svm(vcpu); + + svm_flush_tlb_asid(vcpu, svm->current_vmcb); +} + static inline void sync_cr8_to_lapic(struct kvm_vcpu *vcpu) { struct vcpu_svm *svm = to_svm(vcpu); @@ -5055,7 +5066,7 @@ static struct kvm_x86_ops svm_x86_ops __initdata = { .flush_tlb_all = svm_flush_tlb_all, .flush_tlb_current = svm_flush_tlb_current, .flush_tlb_gva = svm_flush_tlb_gva, - .flush_tlb_guest = svm_flush_tlb_asid, + .flush_tlb_guest = svm_flush_tlb_guest, .vcpu_pre_run = svm_vcpu_pre_run, .vcpu_run = svm_vcpu_run, From patchwork Wed Feb 5 18:23:52 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13961704 Received: from out-184.mta0.migadu.com (out-184.mta0.migadu.com [91.218.175.184]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 251651FF1B9 for ; Wed, 5 Feb 2025 18:24:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.184 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738779871; cv=none; b=ubf0I4A9HZkviwbixdFkWFxGF2ROBnNGfuyXKJuSOTWAtsLlnBJt9n4Ufmg/gtL6w6lwF2/NMGO6XobOjHb+0AVTICD+L6TLbSIqyf7zZtL9/FC5FlnX5hXQcxUH6cdon0T3yd7cd2y6Mhcigo7O1ldiZHTKa90egJ+U7TOotHw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738779871; c=relaxed/simple; bh=y0ETfi4XxlyM14bktngNlCHmHhFeNkK4dgV/ueMQpq4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=H9NhsWeoxfQ2zF6z486Ssdkrr8haRt+7GyoBVmSzrxtCtGHXjt3ph7ViexushcleUQnoSgFfv2cmKpd7tVFm+YLyNMPJWeDKy2xO7/doMQWnaprbIve8ZxbgmYzK5ufLnzAJE0vCGYRTZvKloPdugB0dQGs+q8IBPFXMY9QEysY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=Y7NDq911; arc=none smtp.client-ip=91.218.175.184 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="Y7NDq911" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1738779861; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3IxGgLbg8MJVF0QNKewRjo2LHkpbHKqZR8+vO+mj01Q=; b=Y7NDq911WyabGgas+JmWUKo3HC5yqM4hDYmWnVZulr7ECW0ZQ+5keifiDDp6CANFYKF+Qo N9fhl2GIhBmW7nf8imgUqsqp5hZ4DC0W/mnmHo9wc97BiZMZdA9sm41ZWl1tK+3of7Nwii C46vGj5RTtQrLvYTzRM96qqoqy46B3g= From: Yosry Ahmed To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH 03/13] KVM: nSVM: Split nested_svm_transition_tlb_flush() into entry/exit fns Date: Wed, 5 Feb 2025 18:23:52 +0000 Message-ID: <20250205182402.2147495-4-yosry.ahmed@linux.dev> In-Reply-To: <20250205182402.2147495-1-yosry.ahmed@linux.dev> References: <20250205182402.2147495-1-yosry.ahmed@linux.dev> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT The handling for the entry and exit TLB flushes will diverge significantly in the following changes. Instead of adding an 'is_vmenter' argument like nested_vmx_transition_tlb_flush(), just split the function into two variants for 'entry' and 'exit'. No functional change intended. Signed-off-by: Yosry Ahmed --- arch/x86/kvm/svm/nested.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c index bbe4f3ac9f250..2eba36af44f22 100644 --- a/arch/x86/kvm/svm/nested.c +++ b/arch/x86/kvm/svm/nested.c @@ -482,7 +482,7 @@ static void nested_save_pending_event_to_vmcb12(struct vcpu_svm *svm, vmcb12->control.exit_int_info = exit_int_info; } -static void nested_svm_transition_tlb_flush(struct kvm_vcpu *vcpu) +static void nested_svm_entry_tlb_flush(struct kvm_vcpu *vcpu) { /* Handle pending Hyper-V TLB flush requests */ kvm_hv_nested_transtion_tlb_flush(vcpu, npt_enabled); @@ -503,6 +503,16 @@ static void nested_svm_transition_tlb_flush(struct kvm_vcpu *vcpu) kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu); } +static void nested_svm_exit_tlb_flush(struct kvm_vcpu *vcpu) +{ + /* Handle pending Hyper-V TLB flush requests */ + kvm_hv_nested_transtion_tlb_flush(vcpu, npt_enabled); + + /* See nested_svm_entry_tlb_flush() */ + kvm_make_request(KVM_REQ_MMU_SYNC, vcpu); + kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu); +} + /* * Load guest's/host's cr3 on nested vmentry or vmexit. @nested_npt is true * if we are emulating VM-Entry into a guest with NPT enabled. @@ -645,7 +655,7 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm, u32 pause_count12; u32 pause_thresh12; - nested_svm_transition_tlb_flush(vcpu); + nested_svm_entry_tlb_flush(vcpu); /* Enter Guest-Mode */ enter_guest_mode(vcpu); @@ -1131,7 +1141,7 @@ int nested_svm_vmexit(struct vcpu_svm *svm) kvm_vcpu_unmap(vcpu, &map); - nested_svm_transition_tlb_flush(vcpu); + nested_svm_exit_tlb_flush(vcpu); nested_svm_uninit_mmu_context(vcpu); From patchwork Wed Feb 5 18:23:53 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13961705 Received: from out-189.mta0.migadu.com (out-189.mta0.migadu.com [91.218.175.189]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 64F221FF1C4 for ; Wed, 5 Feb 2025 18:24:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.189 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738779872; cv=none; b=Vm7HLHrbbEEwrMD5OGpLbVaKN57rrEGt16rvtom2STV4iywhIuDcEsQy8ZziE0o4gdd0Yfqjc0tY4sFobRRDlAJhOnyPSlY3BCUldmCv0mHjczLOkRUuKvtoB5V1FI6VM4SEWQ1Wtxr4gTu7HHQ5f4BnklOFSQKNOTRSQheERIc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738779872; c=relaxed/simple; bh=3FhFsdcBPRFfzAegygTDKvXo3Zl/PnarMy0+zg+4/oM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=QwrnWkkx1wfEdsuEvtEdUC3SmrSJtrR/5Ca/IezzKaV9leMOUdomXm41BJXAB2jm3OMDN37HoAgiheeZ3rD2WjSve7DjwJaOtP1XdibjqoOQ5VZBh148jK1IXrh1hfGquE4t1BgkvkurvtlfLv8msF3L9KGSpPw8IzaMChYW+Zw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=b+UQ3vNG; arc=none smtp.client-ip=91.218.175.189 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="b+UQ3vNG" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1738779868; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wU7GujvGutRhBR40lU9N1rdEDOfXC2UW00gPytDGFSk=; b=b+UQ3vNGNV4dldBEeKe49bFGKNESkCSTJwHcgIGGMBVfokJ/GXjeAGUb+3t4UdiN1JYDf5 k8og9YxmKQvU1g3WLfTEpVVvDPdPDG42l9ymxiNu8Q7oUpEDb2WZ/TKgMcLNNg7pcn5IFC zd07cpg+2DSn7+qhaS13N5moCKIWoVE= From: Yosry Ahmed To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH 04/13] KVM: SVM: Introduce helpers for updating TLB_CONTROL Date: Wed, 5 Feb 2025 18:23:53 +0000 Message-ID: <20250205182402.2147495-5-yosry.ahmed@linux.dev> In-Reply-To: <20250205182402.2147495-1-yosry.ahmed@linux.dev> References: <20250205182402.2147495-1-yosry.ahmed@linux.dev> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Introduce helpers for updating TLB_CONTROL in the VMCB instead of directly setting it. Two helpers are introduced: - svm_add_tlb_ctl_flush(): Combines a new TLB_CONTROL value with the existing one. - svm_clear_tlb_ctl_flush(): Clears the TLB_CONTROL field. The goal is to prevent overwriting a TLB_CONTROL value with something that results in less TLB entries being flushed. This does not currently happen as KVM only sets TLB_CONTROL_FLUSH_ASID when servicing a flush request, and TLB_CONTROL_FLUSH_ALL_ASID when allocating a new ASID. The latter always happens after the former so no unsafe overwrite happens. However, future changes may result in subtle bugs where the TLB_CONTROL field is incorrectly overwritten. The new helpers prevent that. A separate helper is used for clearing the TLB flush because it is semantically different. In this case, KVM knowingly ignores the existing value of TLB_CONTROL. Also, although svm_add_tlb_ctl_flush() would just work for TLB_CONTROL_DO_NOTHING, the logic becomes inconsistent (use the biggest hammer unless no hammer at all is requested). Opportunistically move the TLB_CONTROL_* definitions to arch/x86/kvm/svm/svm.h as they are not used outside of arch/x86/kvm/svm/. No functional change intended. Signed-off-by: Yosry Ahmed --- arch/x86/include/asm/svm.h | 6 ------ arch/x86/kvm/svm/nested.c | 2 +- arch/x86/kvm/svm/sev.c | 2 +- arch/x86/kvm/svm/svm.c | 6 +++--- arch/x86/kvm/svm/svm.h | 29 +++++++++++++++++++++++++++++ 5 files changed, 34 insertions(+), 11 deletions(-) diff --git a/arch/x86/include/asm/svm.h b/arch/x86/include/asm/svm.h index 2b59b9951c90e..e6bccf8f90982 100644 --- a/arch/x86/include/asm/svm.h +++ b/arch/x86/include/asm/svm.h @@ -169,12 +169,6 @@ struct __attribute__ ((__packed__)) vmcb_control_area { }; }; - -#define TLB_CONTROL_DO_NOTHING 0 -#define TLB_CONTROL_FLUSH_ALL_ASID 1 -#define TLB_CONTROL_FLUSH_ASID 3 -#define TLB_CONTROL_FLUSH_ASID_LOCAL 7 - #define V_TPR_MASK 0x0f #define V_IRQ_SHIFT 8 diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c index 2eba36af44f22..0e9b0592c1f83 100644 --- a/arch/x86/kvm/svm/nested.c +++ b/arch/x86/kvm/svm/nested.c @@ -690,7 +690,7 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm, /* Done at vmrun: asid. */ /* Also overwritten later if necessary. */ - vmcb02->control.tlb_ctl = TLB_CONTROL_DO_NOTHING; + svm_clear_tlb_ctl_flush(vmcb02); /* nested_cr3. */ if (nested_npt_enabled(svm)) diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c index b0adfd0537d00..3af296d6c04f6 100644 --- a/arch/x86/kvm/svm/sev.c +++ b/arch/x86/kvm/svm/sev.c @@ -3481,7 +3481,7 @@ void pre_sev_run(struct vcpu_svm *svm, int cpu) return; sd->sev_vmcbs[asid] = svm->vmcb; - svm->vmcb->control.tlb_ctl = TLB_CONTROL_FLUSH_ASID; + svm_add_tlb_ctl_flush(svm->vmcb, TLB_CONTROL_FLUSH_ASID); vmcb_mark_dirty(svm->vmcb, VMCB_ASID); } diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index 2108b48ba4959..a2d601cd4c283 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -1985,7 +1985,7 @@ static void new_asid(struct vcpu_svm *svm, struct svm_cpu_data *sd) if (sd->next_asid > sd->max_asid) { ++sd->asid_generation; sd->next_asid = sd->min_asid; - svm->vmcb->control.tlb_ctl = TLB_CONTROL_FLUSH_ALL_ASID; + svm_add_tlb_ctl_flush(svm->vmcb, TLB_CONTROL_FLUSH_ALL_ASID); vmcb_mark_dirty(svm->vmcb, VMCB_ASID); } @@ -3974,7 +3974,7 @@ static void svm_flush_tlb_asid(struct kvm_vcpu *vcpu, struct kvm_vmcb_info *vmcb * VM-Exit (via kvm_mmu_reset_context()). */ if (static_cpu_has(X86_FEATURE_FLUSHBYASID)) - vmcb->ptr->control.tlb_ctl = TLB_CONTROL_FLUSH_ASID; + svm_add_tlb_ctl_flush(vmcb->ptr, TLB_CONTROL_FLUSH_ASID); else vmcb->asid_generation--; } @@ -4317,7 +4317,7 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu, svm->nested.nested_run_pending = 0; } - svm->vmcb->control.tlb_ctl = TLB_CONTROL_DO_NOTHING; + svm_clear_tlb_ctl_flush(svm->vmcb); vmcb_mark_all_clean(svm->vmcb); /* if exit due to PF check for async PF */ diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h index ebbb0b1a64676..6a73d6ed1e428 100644 --- a/arch/x86/kvm/svm/svm.h +++ b/arch/x86/kvm/svm/svm.h @@ -611,6 +611,35 @@ void svm_set_x2apic_msr_interception(struct vcpu_svm *svm, bool disable); void svm_complete_interrupt_delivery(struct kvm_vcpu *vcpu, int delivery_mode, int trig_mode, int vec); +#define TLB_CONTROL_DO_NOTHING 0 +#define TLB_CONTROL_FLUSH_ALL_ASID 1 +#define TLB_CONTROL_FLUSH_ASID 3 +#define TLB_CONTROL_FLUSH_ASID_LOCAL 7 + +/* + * Clearing TLB flushes is done separately because combining + * TLB_CONTROL_DO_NOTHING with others is counter-intuitive. + */ +static inline void svm_add_tlb_ctl_flush(struct vmcb *vmcb, u8 tlb_ctl) +{ + if (WARN_ON_ONCE(tlb_ctl == TLB_CONTROL_DO_NOTHING)) + return; + + /* + * Apply the least targeted (most inclusive) TLB flush. Apart from + * TLB_CONTROL_DO_NOTHING, lower values of tlb_ctl are less targeted. + */ + if (vmcb->control.tlb_ctl == TLB_CONTROL_DO_NOTHING) + vmcb->control.tlb_ctl = tlb_ctl; + else + vmcb->control.tlb_ctl = min(vmcb->control.tlb_ctl, tlb_ctl); +} + +static inline void svm_clear_tlb_ctl_flush(struct vmcb *vmcb) +{ + vmcb->control.tlb_ctl = TLB_CONTROL_DO_NOTHING; +} + /* nested.c */ #define NESTED_EXIT_HOST 0 /* Exit handled on host level */ From patchwork Wed Feb 5 18:23:54 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13961706 Received: from out-171.mta0.migadu.com (out-171.mta0.migadu.com [91.218.175.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DD96D1FF7A0 for ; Wed, 5 Feb 2025 18:24:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738779873; cv=none; b=hZzu/dOM46Ar/SpJUM207yXBi3GXppXsD6DnRFwseAgI0E1+cHNpFWXxVH80jvYC3Q8S3cAh2eRgYyXCvfmgK0DXTSZrZ5Ii2c6Cl7p5AKyQYP6kJzS0nXaO+m6i4Pm4EeltCkMFw2sh9kJpYIoaQ6H/hiuz1se4JGeodT4ey6E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738779873; c=relaxed/simple; bh=LgRIp5yvhsB+a0uyfuYo1zyr/44Jkdw2aSFiug2omYc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Cl8ydbR9Y8Z0lmfTHqBDsB83Rjp1nHKq+zWc2W3M9lfKLUgLQBTUOvTPCi9X4AC0+FhWRBvTFGDDDtPKzM4Hp6PWRdNjvSfKSuRElECtWUeNPmDNgW6ttaRs0+5egftv1XAQuQAEhBAi+Sye/9ZFO7QSJJOEDdEkZDxI6VXs3KM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=YjpfcRyc; arc=none smtp.client-ip=91.218.175.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="YjpfcRyc" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1738779869; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MxlUxHUE2R3T/zv/8G0eMFUCZCbOt6xBlqOK6ZjmqgA=; b=YjpfcRycWUspV0lkPxuIl/4NFg7Ylb6RUjgPO6qWR5e8rRtl20N3IgIiSSO0Or/yNYe37W Nmv39v1XjIiMdSjCXX+snGRZkkkHNdBuR1W8RuIHa0QhmXQOT7yWspQjMt4Cizi4gbbOmx Q5aZe4aRApbDtQatmSaaVySVjyJhYIk= From: Yosry Ahmed To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH 05/13] KVM: x86/mmu: rename __kvm_mmu_invalidate_addr() Date: Wed, 5 Feb 2025 18:23:54 +0000 Message-ID: <20250205182402.2147495-6-yosry.ahmed@linux.dev> In-Reply-To: <20250205182402.2147495-1-yosry.ahmed@linux.dev> References: <20250205182402.2147495-1-yosry.ahmed@linux.dev> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT In preparation for creating another helper for kvm_mmu_invalidate_addr(), rename __kvm_mmu_invalidate_addr() to kvm_mmu_invalidate_addr_in_root(). No functional change intended. Signed-off-by: Yosry Ahmed --- arch/x86/kvm/mmu/mmu.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 3ff55a18347da..b93f560a2c0e8 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -6120,8 +6120,9 @@ void kvm_mmu_print_sptes(struct kvm_vcpu *vcpu, gpa_t gpa, const char *msg) } EXPORT_SYMBOL_GPL(kvm_mmu_print_sptes); -static void __kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, - u64 addr, hpa_t root_hpa) +static void kvm_mmu_invalidate_addr_in_root(struct kvm_vcpu *vcpu, + struct kvm_mmu *mmu, + u64 addr, hpa_t root_hpa) { struct kvm_shadow_walk_iterator iterator; @@ -6177,11 +6178,11 @@ void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, return; if (roots & KVM_MMU_ROOT_CURRENT) - __kvm_mmu_invalidate_addr(vcpu, mmu, addr, mmu->root.hpa); + kvm_mmu_invalidate_addr_in_root(vcpu, mmu, addr, mmu->root.hpa); for (i = 0; i < KVM_MMU_NUM_PREV_ROOTS; i++) { if (roots & KVM_MMU_ROOT_PREVIOUS(i)) - __kvm_mmu_invalidate_addr(vcpu, mmu, addr, mmu->prev_roots[i].hpa); + kvm_mmu_invalidate_addr_in_root(vcpu, mmu, addr, mmu->prev_roots[i].hpa); } } EXPORT_SYMBOL_GPL(kvm_mmu_invalidate_addr); From patchwork Wed Feb 5 18:23:55 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13961707 Received: from out-175.mta0.migadu.com (out-175.mta0.migadu.com [91.218.175.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5D10120F09B for ; Wed, 5 Feb 2025 18:24:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738779877; cv=none; b=g0vHgjMqspDpoPLSz09pI3z13/JXuhHWC2rP6lL/1kTVYVSPFBWD1PXS+PHzzBUFWYnhf8NF7Wrzss2gTiUEZMPUgQeqc5gukXQXRyo6yJmwGFz5zfXdnhz6XHw6Aiiq0O/ZUMlmTcP13dKNpQFbqIzyPQ9xrGuwF5YfARtKfts= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738779877; c=relaxed/simple; bh=PAfQqhxa85USuXKn06RgSkuInnQbNs5bBN77CzeXAjM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=F5ySfPLyTqDhsQK1nn0RvaGIAcKXWsA/WzVk8AOfiNo56bDE8owthU9W7E6ZvDP35UsTgqohjpviTSZNTwNFJsFFdnJ7JmB0zcuMHwXKKCL9OsHfx9OmStBbwKLf4oziIys66GMRSR/iu7/OXrpteM4DeHpVPr4Ju+wzsNvZzJU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=nHGUqtZs; arc=none smtp.client-ip=91.218.175.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="nHGUqtZs" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1738779871; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qPn1PgFe4AwJI+C+l1VhnOCa43xf5YTOK+x82W3HwlY=; b=nHGUqtZs5lpSAOQvux06QCutFWkYJbBVT/d/RmPjTMgNf8S0mKoTxa28XpxKduG/KUk+tb YgAn8BV22tJRwK8nx6OuQXYA6Wu6lovMEIKIqZHenG5MM7caw6lOTBFV8R3b78sNmyPgOc ippMZ3LllFaxYl7a+WyCBa9TvPZ/F6Q= From: Yosry Ahmed To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH 06/13] KVM: x86/mmu: Allow skipping the gva flush in kvm_mmu_invalidate_addr() Date: Wed, 5 Feb 2025 18:23:55 +0000 Message-ID: <20250205182402.2147495-7-yosry.ahmed@linux.dev> In-Reply-To: <20250205182402.2147495-1-yosry.ahmed@linux.dev> References: <20250205182402.2147495-1-yosry.ahmed@linux.dev> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Refactor a helper out of kvm_mmu_invalidate_addr() that allows skipping the gva flush. This will be used when an invalidation is needed but the GVA TLB translations that require invalidation are not of the current context (e.g. when emulating INVLPGA for L1 to flush L2's translations). No functional change intended. Signed-off-by: Yosry Ahmed --- arch/x86/kvm/mmu/mmu.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index b93f560a2c0e8..ac133abc9c173 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -6158,15 +6158,15 @@ static void kvm_mmu_invalidate_addr_in_root(struct kvm_vcpu *vcpu, write_unlock(&vcpu->kvm->mmu_lock); } -void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, - u64 addr, unsigned long roots) +static void __kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, + u64 addr, unsigned long roots, bool gva_flush) { int i; WARN_ON_ONCE(roots & ~KVM_MMU_ROOTS_ALL); /* It's actually a GPA for vcpu->arch.guest_mmu. */ - if (mmu != &vcpu->arch.guest_mmu) { + if (gva_flush && mmu != &vcpu->arch.guest_mmu) { /* INVLPG on a non-canonical address is a NOP according to the SDM. */ if (is_noncanonical_invlpg_address(addr, vcpu)) return; @@ -6185,6 +6185,12 @@ void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, kvm_mmu_invalidate_addr_in_root(vcpu, mmu, addr, mmu->prev_roots[i].hpa); } } + +void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, + u64 addr, unsigned long roots) +{ + __kvm_mmu_invalidate_addr(vcpu, mmu, addr, roots, true); +} EXPORT_SYMBOL_GPL(kvm_mmu_invalidate_addr); void kvm_mmu_invlpg(struct kvm_vcpu *vcpu, gva_t gva) From patchwork Wed Feb 5 18:23:56 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13961708 Received: from out-181.mta0.migadu.com (out-181.mta0.migadu.com [91.218.175.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9BFB72116E1 for ; Wed, 5 Feb 2025 18:24:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738779877; cv=none; b=fdQZjHvTKuSzf0otR0eEzqlhNnmR3vyDxL/HYPyTpQXT/ky/3fZV0YyJz6ixRs7OOpeEvXqMNtX6XqHwwItP1QoRyQNL1ftpDONT8J2iR0lClYCs7PLBa/Bf0nmdQpDYzxFvZoF0mc2ezYPk6+siZYmeGiypIzr5CkdM6S60iZg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738779877; c=relaxed/simple; bh=zF9HTCv7mSgTL3YrGXtFVrB6+Y1fi2Pvd+FTWNRpeR0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=i22b3hJsGU83Ioyl1qYj2Z8fO+cIq20AFzYoCbTy32YDnOSPbDGv5pR3KlWbhJwMpHeLdPukW0FFZIggxQP/khioDqRGURUA5ulc9qhjOBI04MADPdO2PQ7yssPdNt7F5meWtrunhgstjo6/AbO6x+2o63WHgKNiY8N42VKlifs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=E04eNAhV; arc=none smtp.client-ip=91.218.175.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="E04eNAhV" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1738779873; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jz1utprcDIKkMo3Ywq+21FXOydI1N6MlW8m2PYuMMdM=; b=E04eNAhVibFslE0+RudNcvJg/LpVZFO+QtTaETUQgDnx+WlN2D3XHlAAxZtxIaZam+l+EB eDaNPIYUE+wnmz0G7wnSWJDXIdrJl5RvZ5Xgtk4Ktf02/laGUDP7tWmE0tGB9YNtr7CMIF VgxmH0hRH3v6ZcRwE+4jlr1uAViKs4U= From: Yosry Ahmed To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH 07/13] KVM: nSVM: Handle INVLPGA interception correctly Date: Wed, 5 Feb 2025 18:23:56 +0000 Message-ID: <20250205182402.2147495-8-yosry.ahmed@linux.dev> In-Reply-To: <20250205182402.2147495-1-yosry.ahmed@linux.dev> References: <20250205182402.2147495-1-yosry.ahmed@linux.dev> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Currently, INVPLGA interception handles it like INVLPG, which flushes L1's TLB translations for the address. It was implemented in this way because L1 and L2 shared an ASID. Now, L1 and L2 have separate ASIDs. It is still harmless to flush L1's translations, but it's only correct because all translations are flushed on nested transitions anyway. In preparation for stopping unconditional flushes on nested transitions, handle INVPLGA interception properly. If L1 specified zero as the ASID, this is equivalent to INVLPG, so handle it as such. Otherwise, use INVPLGA to flush the translations of the appropriate ASID tracked by KVM, if any. Sync the shadow MMU as well, as L1 invalidated L2's mappings. Opportunistically update svm_flush_tlb_gva() to use svm->current_vmcb->asid instead of svm->vmcb->control.asid for consistency. The two should always be in sync except when KVM allocates a new ASID in pre_svm_run(), and they are shortly brought back in sync in svm_vcpu_run(). However, if future changes add more code paths where KVM allocates a new ASID, flushing the potentially old ASID in svm->vmcb->control.asid would be unnecessary overhead (although probably not much different from flushing the newly allocated ASID). Signed-off-by: Yosry Ahmed --- arch/x86/include/asm/kvm_host.h | 2 ++ arch/x86/kvm/mmu/mmu.c | 5 +++-- arch/x86/kvm/svm/svm.c | 40 ++++++++++++++++++++++++++++++--- 3 files changed, 42 insertions(+), 5 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 5193c3dfbce15..1e147bb2e560f 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -2213,6 +2213,8 @@ int kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 error_code, void *insn, int insn_len); void kvm_mmu_print_sptes(struct kvm_vcpu *vcpu, gpa_t gpa, const char *msg); void kvm_mmu_invlpg(struct kvm_vcpu *vcpu, gva_t gva); +void __kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, + u64 addr, unsigned long roots, bool gva_flush); void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, u64 addr, unsigned long roots); void kvm_mmu_invpcid_gva(struct kvm_vcpu *vcpu, gva_t gva, unsigned long pcid); diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index ac133abc9c173..f5e0d2c8f4bbe 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -6158,8 +6158,8 @@ static void kvm_mmu_invalidate_addr_in_root(struct kvm_vcpu *vcpu, write_unlock(&vcpu->kvm->mmu_lock); } -static void __kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, - u64 addr, unsigned long roots, bool gva_flush) +void __kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, + u64 addr, unsigned long roots, bool gva_flush) { int i; @@ -6185,6 +6185,7 @@ static void __kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu kvm_mmu_invalidate_addr_in_root(vcpu, mmu, addr, mmu->prev_roots[i].hpa); } } +EXPORT_SYMBOL_GPL(__kvm_mmu_invalidate_addr); void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, u64 addr, unsigned long roots) diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index a2d601cd4c283..9e29f87d3bd93 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -2483,6 +2483,7 @@ static int clgi_interception(struct kvm_vcpu *vcpu) static int invlpga_interception(struct kvm_vcpu *vcpu) { + struct vcpu_svm *svm = to_svm(vcpu); gva_t gva = kvm_rax_read(vcpu); u32 asid = kvm_rcx_read(vcpu); @@ -2492,8 +2493,41 @@ static int invlpga_interception(struct kvm_vcpu *vcpu) trace_kvm_invlpga(to_svm(vcpu)->vmcb->save.rip, asid, gva); - /* Let's treat INVLPGA the same as INVLPG (can be optimized!) */ - kvm_mmu_invlpg(vcpu, gva); + /* + * APM is silent about using INVLPGA to flush the host ASID (i.e. 0). + * Do the logical thing and handle it like INVLPG. + */ + if (asid == 0) { + kvm_mmu_invlpg(vcpu, gva); + return kvm_skip_emulated_instruction(vcpu); + } + + /* + * Check if L1 specified the L2 ASID we are currently tracking. If it + * isn't, do nothing as we have to handle the TLB flush when switching + * to the new ASID anyway. APM mentions that INVLPGA is typically only + * meaningful with shadow paging, so also do nothing if L1 is using + * nested NPT. + */ + if (!nested_npt_enabled(svm) && asid == svm->nested.last_asid) + invlpga(gva, svm->nested.vmcb02.asid); + + /* + * If NPT is disabled, sync the shadow page tables as L1 is invalidating + * mappings for L2. Sync all roots as ASIDs are not tracked in the MMU + * role. + * + * As we are not flushing the current context, skip the gva flush from + * __kvm_mmu_invalidate_addr(), it would flush the wrong ASID anyway. + * The correct TLB flush was done above (if needed). + * + * This always operates on root_mmu because L1 and L2 share an MMU when + * NPT is disabled. This can be optimized by invalidating guest roots + * only. + */ + if (!npt_enabled) + __kvm_mmu_invalidate_addr(vcpu, &vcpu->arch.root_mmu, gva, + KVM_MMU_ROOTS_ALL, false); return kvm_skip_emulated_instruction(vcpu); } @@ -4017,7 +4051,7 @@ static void svm_flush_tlb_gva(struct kvm_vcpu *vcpu, gva_t gva) { struct vcpu_svm *svm = to_svm(vcpu); - invlpga(gva, svm->vmcb->control.asid); + invlpga(gva, svm->current_vmcb->asid); } static void svm_flush_tlb_guest(struct kvm_vcpu *vcpu) From patchwork Wed Feb 5 18:23:57 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13961709 Received: from out-177.mta0.migadu.com (out-177.mta0.migadu.com [91.218.175.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0FA281FE455 for ; Wed, 5 Feb 2025 18:24:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738779878; cv=none; b=rtRNaMV/DpqZX5GbH34t+n7MD/dbVir6Pb5iW31eGY4iq3a4yeNnuayizxPZwiyeDpqOYG29LHT6xmA0uAwLHLHX+mEv0SRnTXptFqoR99rOk/Les8zYtvI/hPASnNG1haA/m1GyEbUJJmsVQLFTIlvxlTR7ucoJ7Kow6u1sKYg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738779878; c=relaxed/simple; bh=h4MbKQQTqAjzHmrKLlep4WU41WTJGnth1WFuNMwjcy4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=YdO75jJXxo4Y0ugU/nWLdqSFapEb+FK5DVKSnqBdf5ykl6OjEytVyOj28YgKcy18nHwoxRuzTvMsRFalVOgI551YFESl3IKMXt5BjQTquBaz4zZQhzCDfXJ4jDFIZ2pmwWngIz4tSH/m/V1N1IPXKI5/JJOnqHhvwclWeph7DDs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=fXu3/Omx; arc=none smtp.client-ip=91.218.175.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="fXu3/Omx" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1738779875; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7l1Q8RiV52bQukEA9QV4TzD72aVSvrPFZT5jtx/k5qI=; b=fXu3/Omxtd7C5jp6v+NCR/mH9LGxXvrUbCwc1kExAyT3rx7OhT/2agJPeeIl/3WnZy2SJd NMWKovwBx/kc5ysPRB7XKpks9Ny2V9ebpzuu2VArEV2PNJ5K/o/1plqYE4t/Yya3L/s5tA iAkX+uRXLxwQCvpOfm/pTvvhbYjLQZw= From: Yosry Ahmed To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH 08/13] KVM: nSVM: Flush both L1 and L2 ASIDs on KVM_REQ_TLB_FLUSH Date: Wed, 5 Feb 2025 18:23:57 +0000 Message-ID: <20250205182402.2147495-9-yosry.ahmed@linux.dev> In-Reply-To: <20250205182402.2147495-1-yosry.ahmed@linux.dev> References: <20250205182402.2147495-1-yosry.ahmed@linux.dev> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT KVM_REQ_TLB_FLUSH is used to flush all TLB entries for all contexts (e.g. in kvm_flush_remote_tlbs()). Flush both L1 and L2 ASIDs in svm_flush_tlb_all() to handle it appropriately. This is currently not required as nested transitions do unconditional TLB flushes, but this is a step toward eliminating that. Signed-off-by: Yosry Ahmed --- arch/x86/kvm/svm/nested.c | 1 - arch/x86/kvm/svm/svm.c | 4 +++- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c index 0e9b0592c1f83..0735177b95a1d 100644 --- a/arch/x86/kvm/svm/nested.c +++ b/arch/x86/kvm/svm/nested.c @@ -491,7 +491,6 @@ static void nested_svm_entry_tlb_flush(struct kvm_vcpu *vcpu) * TODO: optimize unconditional TLB flush/MMU sync. A partial list of * things to fix before this can be conditional: * - * - Flush TLBs for both L1 and L2 remote TLB flush * - Honor L1's request to flush an ASID on nested VMRUN * - Sync nested NPT MMU on VMRUN that flushes L2's ASID[*] * - Don't crush a pending TLB flush in vmcb02 on nested VMRUN diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index 9e29f87d3bd93..8342c7eadbba8 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -4044,7 +4044,9 @@ static void svm_flush_tlb_all(struct kvm_vcpu *vcpu) if (WARN_ON_ONCE(svm_hv_is_enlightened_tlb_enabled(vcpu))) hv_flush_remote_tlbs(vcpu->kvm); - svm_flush_tlb_asid(vcpu, svm->current_vmcb); + svm_flush_tlb_asid(vcpu, &svm->vmcb01); + if (svm->nested.initialized) + svm_flush_tlb_asid(vcpu, &svm->nested.vmcb02); } static void svm_flush_tlb_gva(struct kvm_vcpu *vcpu, gva_t gva) From patchwork Wed Feb 5 18:23:58 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13961710 Received: from out-177.mta0.migadu.com (out-177.mta0.migadu.com [91.218.175.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7F143214807 for ; Wed, 5 Feb 2025 18:24:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738779883; cv=none; b=pSVJMA5sSGvD1fXL6x7hM1lMGbCYYdOfCUMrF4E1jweHXQxB1qjdvP/7vqCSOClRFqRDK19U1ufh5mfp3/r4uHpzy1KSjUWaDsFnD2ZO9+F0VeXexhzFxBTIw1hzHeJlotPC49gRLnUR5XsT7LaXyaw5QWhFDo6mx09DzKvQT2Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738779883; c=relaxed/simple; bh=3TYqX4BFP2U6TMLgJgMdg5nWWzFi+RBdInLSw9YpA9s=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ZWYbJPIA0zfkx46jwyAP5YTG8ISqeOlBpIA+WeDf1fU8VcmcSUcoBputqE4cxJa25kNBOKWLQjwnge+Xvn6RsB/DhElwSc5glhPOdFvctCeBLAOM2YB/XoFiJga/w4355OfYSxs3C8KfphODvPgQWrW8BMGVEO1IKnB/8fQMUow= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=bSgJm9po; arc=none smtp.client-ip=91.218.175.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="bSgJm9po" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1738779876; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=k/M4ph76ynZ139V+HDMXT5ZDD/lLnaQbCHrkiu67TFc=; b=bSgJm9poGibDaSCn0OBneCA2AEMaZVtnqWrQoFq4Z+g7BAysGff0CIi5ysd4f0KXXd4fGi ciwhn8mtaJBAQi9yZoIlL48FnYfyb9ZhbZ9cWMRHoOKd442IXeAjvpms9fko27W9eswvky nr4qZ7QrMu3uYow7JwIBkuKVmTTEPps= From: Yosry Ahmed To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH 09/13] KVM: nSVM: Handle nested TLB flush requests through TLB_CONTROL Date: Wed, 5 Feb 2025 18:23:58 +0000 Message-ID: <20250205182402.2147495-10-yosry.ahmed@linux.dev> In-Reply-To: <20250205182402.2147495-1-yosry.ahmed@linux.dev> References: <20250205182402.2147495-1-yosry.ahmed@linux.dev> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Handle L1's requests to flush L2's TLB through the TLB_CONTROL field of VMCB12. This is currently redundant because a full flush is executed on every nested transition, but is a step towards removing that. TLB_CONTROL_FLUSH_ALL_ASID flushes all ASIDs from L1's perspective, including its own, so do a guest TLB flush on both transitions. Never propagate TLB_CONTROL_FLUSH_ALL_ASID from the guest to the actual VMCB, as this gives the guest the power to flush the entire physical TLB (including translations for the host and other VMs). For other cases, the TLB flush is only done when entering L2. The nested NPT MMU is also sync'd because TLB_CONTROL also flushes NPT guest-physical mappings. All TLB_CONTROL values can be handled by KVM regardless of FLUSHBYASID support on the underlying CPU, so keep advertising FLUSHBYASID to the guest unconditionally. Signed-off-by: Yosry Ahmed --- arch/x86/kvm/svm/nested.c | 42 ++++++++++++++++++++++++++++++++------- arch/x86/kvm/svm/svm.c | 6 +++--- 2 files changed, 38 insertions(+), 10 deletions(-) diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c index 0735177b95a1d..e2c59eb2907e8 100644 --- a/arch/x86/kvm/svm/nested.c +++ b/arch/x86/kvm/svm/nested.c @@ -484,19 +484,36 @@ static void nested_save_pending_event_to_vmcb12(struct vcpu_svm *svm, static void nested_svm_entry_tlb_flush(struct kvm_vcpu *vcpu) { + struct vcpu_svm *svm = to_svm(vcpu); + /* Handle pending Hyper-V TLB flush requests */ kvm_hv_nested_transtion_tlb_flush(vcpu, npt_enabled); + /* + * If L1 requested a TLB flush for L2, flush L2's TLB on nested entry + * and sync the nested NPT MMU, as TLB_CONTROL also flushes NPT + * guest-physical mappings. We technically only need to flush guest_mode + * page tables. + * + * KVM_REQ_TLB_FLUSH_GUEST will flush L2's ASID even if the underlying + * CPU does not support FLUSHBYASID (by assigning a new ASID), so we + * can handle all TLB_CONTROL values from L1 regardless. + * + * Note that TLB_CONTROL_FLUSH_ASID_LOCAL is handled exactly like + * TLB_CONTROL_FLUSH_ASID. We can technically flush less TLB entries, + * but this would require significantly more complexity. + */ + if (svm->nested.ctl.tlb_ctl != TLB_CONTROL_DO_NOTHING) { + if (nested_npt_enabled(svm)) + kvm_make_request(KVM_REQ_MMU_SYNC, vcpu); + kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu); + } + /* * TODO: optimize unconditional TLB flush/MMU sync. A partial list of * things to fix before this can be conditional: * - * - Honor L1's request to flush an ASID on nested VMRUN - * - Sync nested NPT MMU on VMRUN that flushes L2's ASID[*] * - Don't crush a pending TLB flush in vmcb02 on nested VMRUN - * - * [*] Unlike nested EPT, SVM's ASID management can invalidate nested - * NPT guest-physical mappings on VMRUN. */ kvm_make_request(KVM_REQ_MMU_SYNC, vcpu); kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu); @@ -504,9 +521,18 @@ static void nested_svm_entry_tlb_flush(struct kvm_vcpu *vcpu) static void nested_svm_exit_tlb_flush(struct kvm_vcpu *vcpu) { + struct vcpu_svm *svm = to_svm(vcpu); + /* Handle pending Hyper-V TLB flush requests */ kvm_hv_nested_transtion_tlb_flush(vcpu, npt_enabled); + /* + * If L1 had requested a full TLB flush when entering L2, also flush its + * TLB entries when exiting back to L1. + */ + if (svm->nested.ctl.tlb_ctl == TLB_CONTROL_FLUSH_ALL_ASID) + kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu); + /* See nested_svm_entry_tlb_flush() */ kvm_make_request(KVM_REQ_MMU_SYNC, vcpu); kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu); @@ -825,7 +851,8 @@ int enter_svm_guest_mode(struct kvm_vcpu *vcpu, u64 vmcb12_gpa, nested_svm_copy_common_state(svm->vmcb01.ptr, svm->nested.vmcb02.ptr); svm_switch_vmcb(svm, &svm->nested.vmcb02); - nested_vmcb02_prepare_control(svm, vmcb12->save.rip, vmcb12->save.cs.base); + nested_vmcb02_prepare_control(svm, vmcb12->save.rip, + vmcb12->save.cs.base); nested_vmcb02_prepare_save(svm, vmcb12); ret = nested_svm_load_cr3(&svm->vcpu, svm->nested.save.cr3, @@ -1764,7 +1791,8 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu, nested_copy_vmcb_control_to_cache(svm, ctl); svm_switch_vmcb(svm, &svm->nested.vmcb02); - nested_vmcb02_prepare_control(svm, svm->vmcb->save.rip, svm->vmcb->save.cs.base); + nested_vmcb02_prepare_control(svm, svm->vmcb->save.rip, + svm->vmcb->save.cs.base); /* * While the nested guest CR3 is already checked and set by diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index 8342c7eadbba8..5e7b1c9bfa605 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -5242,9 +5242,9 @@ static __init void svm_set_cpu_caps(void) kvm_cpu_cap_set(X86_FEATURE_VMCBCLEAN); /* - * KVM currently flushes TLBs on *every* nested SVM transition, - * and so for all intents and purposes KVM supports flushing by - * ASID, i.e. KVM is guaranteed to honor every L1 ASID flush. + * KVM currently handles all TLB_CONTROL values set by L1, even + * if the underlying CPU does not. See + * nested_svm_transition_tlb_flush(). */ kvm_cpu_cap_set(X86_FEATURE_FLUSHBYASID); From patchwork Wed Feb 5 18:23:59 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13961711 Received: from out-171.mta0.migadu.com (out-171.mta0.migadu.com [91.218.175.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D44BA214A90 for ; Wed, 5 Feb 2025 18:24:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738779889; cv=none; b=XV56ZX2axYoLa2iXbpsUGQffvhhRm6CGeG4olJuP80dIdjylmU/Q88Vfiaivii2SedAmkOv4+yz7jPl0vhXnXEGzYBjL4X6hUEhJalxDlm+4dcrMpEI038JUqaIM9RMfqvpkVTYH6w7JZ4FKbIaeGyd+r5LEgDqDhkZbB0gQkVk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738779889; c=relaxed/simple; bh=J80J7QWwNwqKJI9OkYLRwkYf70xFczDYSSkcpDuLb08=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=iUoouVolccYAjXMKtUfCfEDKyZsBtDmAyge5cULtWK1xCAw1K3a3P/w/8CwCGEtlk+OYvaS3qUkSRpJgQ2ng5CHy8XHYuWX4Vx+17R7WCR2kO8vRBWQMGEQgqZfV0MKmZwbfcrDWU0Xots3L8P/NB2094+MCIf037WFChtfRoL4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=rFixJRd4; arc=none smtp.client-ip=91.218.175.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="rFixJRd4" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1738779881; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rrSQ3jUA061xY0J/9WOEI/tvDvAK9ExwlT3CVuaXaqw=; b=rFixJRd4jqemOUEt6Oub7sb4hNpmJV/0S9sRHsSEfePv6qJ537nFtEQB29iWNNGtz90xAj 4gmeuc2iGq4CGW8FIQqoDcNS+Wd/yG8cjpPywroYkeqcbAR+kl/p3TpQ6NbskxPQnwfNWz jHoFrfP4qYSC4XFCJ+gcYRDs2q+IaAU= From: Yosry Ahmed To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH 10/13] KVM: nSVM: Flush the TLB if L1 changes L2's ASID Date: Wed, 5 Feb 2025 18:23:59 +0000 Message-ID: <20250205182402.2147495-11-yosry.ahmed@linux.dev> In-Reply-To: <20250205182402.2147495-1-yosry.ahmed@linux.dev> References: <20250205182402.2147495-1-yosry.ahmed@linux.dev> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT KVM tracks a single ASID for L2 guests. L1 could change the ASID it has assigned L2 due to switching to a different L2 guest or simply to avoid flushing L2's existing ASID. Flush L2's TLB when this happens to avoid reusing TLB entries from the old ASID (from L1's perspective). Remove the comment in __nested_copy_vmcb_control_to_cache() about the cached ASID usage, as this changes makes it stale by adding another usage. This is heavily inspired by nVMX's handling of last_vpid. Signed-off-by: Yosry Ahmed --- arch/x86/kvm/svm/nested.c | 5 ++++- arch/x86/kvm/svm/svm.h | 2 ++ 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c index e2c59eb2907e8..12bb391884299 100644 --- a/arch/x86/kvm/svm/nested.c +++ b/arch/x86/kvm/svm/nested.c @@ -368,7 +368,6 @@ void __nested_copy_vmcb_control_to_cache(struct kvm_vcpu *vcpu, to->pause_filter_count = from->pause_filter_count; to->pause_filter_thresh = from->pause_filter_thresh; - /* Copy asid here because nested_vmcb_check_controls will check it. */ to->asid = from->asid; to->msrpm_base_pa &= ~0x0fffULL; to->iopm_base_pa &= ~0x0fffULL; @@ -509,6 +508,10 @@ static void nested_svm_entry_tlb_flush(struct kvm_vcpu *vcpu) kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu); } + if (svm->nested.ctl.asid != svm->nested.last_asid) { + svm->nested.last_asid = svm->nested.ctl.asid; + kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu); + } /* * TODO: optimize unconditional TLB flush/MMU sync. A partial list of * things to fix before this can be conditional: diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h index 6a73d6ed1e428..f2352135b99d3 100644 --- a/arch/x86/kvm/svm/svm.h +++ b/arch/x86/kvm/svm/svm.h @@ -211,6 +211,8 @@ struct svm_nested_state { * on its side. */ bool force_msr_bitmap_recalc; + + u32 last_asid; }; struct vcpu_sev_es_state { From patchwork Wed Feb 5 18:24:00 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13961712 Received: from out-173.mta0.migadu.com (out-173.mta0.migadu.com [91.218.175.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 87B9A1FECDC for ; Wed, 5 Feb 2025 18:24:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738779896; cv=none; b=KwGtC7RXnn1PpRIUS/6QamgMyXQe+uBgPTM/Pdmq38VlsshAtMX+EEiVWM20enzCkW5UinD8kQRpNjVHV5ZmUuVcxp0zDAokblQbQV52Vc1McoX6j1W3YMn+jXCvp+hj5jt0H8hMPzBS+iKosNT0JK9GI/pe/YMIoVqVu6iPkYc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738779896; c=relaxed/simple; bh=G8vnaOWQJ5Oj78kftPCmluJb4+vY84aUlbOf6mIvYHw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=EmJYszxk90KXez2IRcU18p66i2N8eJA1mwDr4aXxfKxOgX/uvZCLNiE3FaG8mEhlqqLNe2ROGK244ZYhP8hp81Rmp1ak3xfNcrT7oQWVAyyuR8V3oIkJQSQIUfXGW5ZDkWtRiMrd9LAsn2yFKoBnB4U3eHjKOXPHlxN7bnKo6qA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=CqAP90UN; arc=none smtp.client-ip=91.218.175.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="CqAP90UN" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1738779887; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=x+aZxHkGDKGQVqlD/AOZPROKXjugIKgS0ANHq/kESss=; b=CqAP90UNeXqSyZD8+VjrVbhHejxIjyT3xxMfgHcrStUdktFl3WI/fg/Js7ETTtmvHefEeT WKCqFdv/FnvOXGuLZXH/RqPSgbGk7CB6AcCii/ohGGW4mj3O0gbLqpzznksn6lXtteewaf 4CSQipKW0M9pftq3YQR7h8lP8PICE6M= From: Yosry Ahmed To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH 11/13] KVM: nSVM: Do not reset TLB_CONTROL in VMCB02 on nested entry Date: Wed, 5 Feb 2025 18:24:00 +0000 Message-ID: <20250205182402.2147495-12-yosry.ahmed@linux.dev> In-Reply-To: <20250205182402.2147495-1-yosry.ahmed@linux.dev> References: <20250205182402.2147495-1-yosry.ahmed@linux.dev> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT TLB_CONTROL is reset to TLB_CONTROL_DO_NOTHING on nested transitions to L2. This is unnecessary because it should always be TLB_CONTROL_DO_NOTHING at this point. The flow for setting TLB_CONTROL is as follows: 1. In vcpu_enter_guest(), servicing a TLB flush request may set it to TLB_CONTROL_FLUSH_ASID in svm_flush_tlb_asid(). 2. In svm_vcpu_run() -> pre_svm_run(), it may get upgraded to TLB_CONTROL_FLUSH_ALL_ASID when assigning a new ASID. 3. In svm_cpu_run(), it gets reset to TLB_CONTROL_DO_NOTHING after the guest is run. Hence, TLB_CONTROL is reset after each run and there is no need to do it again on every nested transition to L2. There is a TODO in nested_svm_transition_tlb_flush() about this reset crushing pending TLB flushes. Remove it, as the reset is not really crushing anything as explained above. Signed-off-by: Yosry Ahmed --- arch/x86/kvm/svm/nested.c | 12 ++---------- 1 file changed, 2 insertions(+), 10 deletions(-) diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c index 12bb391884299..8e40ff21f7353 100644 --- a/arch/x86/kvm/svm/nested.c +++ b/arch/x86/kvm/svm/nested.c @@ -512,12 +512,7 @@ static void nested_svm_entry_tlb_flush(struct kvm_vcpu *vcpu) svm->nested.last_asid = svm->nested.ctl.asid; kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu); } - /* - * TODO: optimize unconditional TLB flush/MMU sync. A partial list of - * things to fix before this can be conditional: - * - * - Don't crush a pending TLB flush in vmcb02 on nested VMRUN - */ + /* TODO: optimize unconditional TLB flush/MMU sync */ kvm_make_request(KVM_REQ_MMU_SYNC, vcpu); kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu); } @@ -536,7 +531,7 @@ static void nested_svm_exit_tlb_flush(struct kvm_vcpu *vcpu) if (svm->nested.ctl.tlb_ctl == TLB_CONTROL_FLUSH_ALL_ASID) kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu); - /* See nested_svm_entry_tlb_flush() */ + /* TODO: optimize unconditional TLB flush/MMU sync */ kvm_make_request(KVM_REQ_MMU_SYNC, vcpu); kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu); } @@ -717,9 +712,6 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm, /* Done at vmrun: asid. */ - /* Also overwritten later if necessary. */ - svm_clear_tlb_ctl_flush(vmcb02); - /* nested_cr3. */ if (nested_npt_enabled(svm)) nested_svm_init_mmu_context(vcpu); From patchwork Wed Feb 5 18:24:01 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13961713 Received: from out-182.mta0.migadu.com (out-182.mta0.migadu.com [91.218.175.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C40A8215061 for ; Wed, 5 Feb 2025 18:24:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738779897; cv=none; b=T2uhaLBNIIbwHlqHN9IhdtNBLCsD9qCp79CkmFoKQnOyD/FNghz/K+FuRFY9dG/BeRrry1Z+SlSN92eLmMOQqxSsrwT+E3r3t5Eh1o77dewng3xuwOY2SXZbyxGLOTXqjqVGHwqakW0v+vhchmyq0xuPZyo3YGfi3RQPaiSLRYI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738779897; c=relaxed/simple; bh=tlOgXnaqKp9NUL+LXoFpiVzobJNqv5f08XPkh3OR01g=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Ms92UnNbx6zMnMWS7gdULZ+abDYRBCBu+rHg5EOdAYDweZsyR5yyihbtNYe2pwIjFAKoQZVPzYmGo6TMNEgqIdBLAizwuIQlzy0Re2mh4jqoFGVOlaIM7zSW6OSC5egEI1HNViSFcSgJ3neXNIyx/NSXFPrP4O8McnlTUwRDG60= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=UD97/NXJ; arc=none smtp.client-ip=91.218.175.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="UD97/NXJ" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1738779894; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wkp6NpfUIvQgIoWnkXuz59xSTMhu75OTgY4oeK1GYRA=; b=UD97/NXJGvSJt2fE2zbn0hCtPHezCKTJ92bbca+FHrrqM+TyfBFw8AP6tv7HUdN6DtoJCB jjwZtx5gVVsk2OK6Yc8+JU8y89pPU6tGQydwcGUwo8EWGqxE9jHeYbl4Z0NgQ1izG8EO0w MRd7rm+/lDrrZ6onq+T/CVlGV3SsE/s= From: Yosry Ahmed To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH 12/13] KVM: nSVM: Service local TLB flushes before nested transitions Date: Wed, 5 Feb 2025 18:24:01 +0000 Message-ID: <20250205182402.2147495-13-yosry.ahmed@linux.dev> In-Reply-To: <20250205182402.2147495-1-yosry.ahmed@linux.dev> References: <20250205182402.2147495-1-yosry.ahmed@linux.dev> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT KVM does not track TLB flush requests for L1 vs. L2. Hence, service local flush that target the current context before switching to a new one. Since ASIDs are tracked per-VMCB, service the flushes before every VMCB switch. This is conceptually similar to how nVMX calls kvm_service_local_tlb_flush_requests() in nested_vmx_enter_non_root_mode() and nested_vmx_vmexit(), with the following differences: 1. nVMX tracks the current VPID based on is_guest_mode(), so local TLB flushes are serviced before enter_guest_mode() and leave_guest_mode(). On the other hand, nSVM tracks the current ASID based on the current VMCB, so the TLB flushes are serviced before an VMCB switch. 2. nVMX only enters and leaves guest mode in nested_vmx_enter_non_root_mode() and nested_vmx_vmexit(). Other paths like vmx_set_nested_state() and vmx_leave_nested() call into these two functions. On the other hand, nSVM open codes the switch in functions like svm_set_nested_state() and svm_leave_nested(), so servicing the flush in svm_switch_svm() is probably most reliable. Signed-off-by: Yosry Ahmed --- arch/x86/kvm/svm/svm.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index 5e7b1c9bfa605..6daa7efa9262b 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -1421,6 +1421,12 @@ static void svm_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) void svm_switch_vmcb(struct vcpu_svm *svm, struct kvm_vmcb_info *target_vmcb) { + /* + * ASIDs are tracked per-VMCB. Perform any pending TLB flushes for the + * current VMCB before switching to a new one. + */ + kvm_service_local_tlb_flush_requests(&svm->vcpu); + svm->current_vmcb = target_vmcb; svm->vmcb = target_vmcb->ptr; } From patchwork Wed Feb 5 18:24:02 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13961714 Received: from out-187.mta0.migadu.com (out-187.mta0.migadu.com [91.218.175.187]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 753111FF1AA for ; Wed, 5 Feb 2025 18:25:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.187 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738779903; cv=none; b=pxint9CMXKVGZstbNxzGQ+rPJubS6R4FiDvhz9rA7+fQ8xbJqQUNtgVWuMicAP0I5AqG74+mTbAsvZEO+qWssdGt782Ue2k4lt63/za7fQB5CS7b2+rkPuHoH8creyEGBEcGRGgc0st/yLK8V/qO68Od0js0TQHSThk/hUl23gE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738779903; c=relaxed/simple; bh=GLqD8wX6QjQPstH23m8VLQ2kgBKnCTTz2dgc/Mx61XI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=uod3CoGIY8qWt5QpF18OmS1/UhKZON2ZGmkszeZIp+x32tZQb5kiLf9UHUapu0DM8jVWhV/JdUUpABT0tmnx4dH2SHNR2K9AWDBv4EtsMUzoHS28PJPeEBDKX705Qhjt2VNFnDvlf10bpZHy/8UM2bx/QuluRfySM7rTtMsJRxk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=Z3bcQSIp; arc=none smtp.client-ip=91.218.175.187 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="Z3bcQSIp" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1738779895; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rbKihAM6jCwO3XM7OJjQoAQ3a9+Wagzk7/DSH7vKMpY=; b=Z3bcQSIpN96MbgruJpGdJKmlI6571SNyo4X4lRZC/X6mHDOcKxBJyQa0EtkzO2WwUc3f90 DbjomJnbR1TNFqRlu3a47u2u+B0i+mAKtpVxz0HOJaicZu4F6HFYY5AubNymNK5Cv/5nLY 8kzvWm8U8B/bAEXLPS9kFllXyioXxxk= From: Yosry Ahmed To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH 13/13] KVM: nSVM: Stop bombing the TLB on nested transitions Date: Wed, 5 Feb 2025 18:24:02 +0000 Message-ID: <20250205182402.2147495-14-yosry.ahmed@linux.dev> In-Reply-To: <20250205182402.2147495-1-yosry.ahmed@linux.dev> References: <20250205182402.2147495-1-yosry.ahmed@linux.dev> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Now that nested TLB flushes are properly tracked with a well-maintained separate ASID for L2 and proper handling of L1's TLB flush requests, drop the unconditional flushes and syncs on nested transitions. On a Milan machine, an L1 and L2 guests were booted, both with a single vCPU, and pinned to a single physical CPU to maximize TLB collisions. In this setup, the cpuid_rate microbenchmark [1] showed the following changes with this patch: +--------+--------+-------------------+----------------------+ | L0 | L1 | cpuid_rate (base) | cpuid_rate (patched) | +========+========+===================+======================+ | NPT | NPT | 256621 | 301113 (+17.3%) | | NPT | Shadow | 180017 | 203347 (+12.96%) | | Shadow | Shadow | 177006 | 189150 (+6.86%) | +--------+--------+-------------------+----------------------+ [1]https://lore.kernel.org/kvm/20231109180646.2963718-1-khorenko@virtuozzo.com/ Signed-off-by: Yosry Ahmed --- arch/x86/kvm/svm/nested.c | 7 ------- 1 file changed, 7 deletions(-) diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c index 8e40ff21f7353..45a187d4c23d1 100644 --- a/arch/x86/kvm/svm/nested.c +++ b/arch/x86/kvm/svm/nested.c @@ -512,9 +512,6 @@ static void nested_svm_entry_tlb_flush(struct kvm_vcpu *vcpu) svm->nested.last_asid = svm->nested.ctl.asid; kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu); } - /* TODO: optimize unconditional TLB flush/MMU sync */ - kvm_make_request(KVM_REQ_MMU_SYNC, vcpu); - kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu); } static void nested_svm_exit_tlb_flush(struct kvm_vcpu *vcpu) @@ -530,10 +527,6 @@ static void nested_svm_exit_tlb_flush(struct kvm_vcpu *vcpu) */ if (svm->nested.ctl.tlb_ctl == TLB_CONTROL_FLUSH_ALL_ASID) kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu); - - /* TODO: optimize unconditional TLB flush/MMU sync */ - kvm_make_request(KVM_REQ_MMU_SYNC, vcpu); - kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu); } /*