From patchwork Mon Dec 13 11:25:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Bonzini X-Patchwork-Id: 12673771 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B903FC433F5 for ; Mon, 13 Dec 2021 11:25:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235027AbhLMLZY (ORCPT ); Mon, 13 Dec 2021 06:25:24 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:23410 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235465AbhLMLZW (ORCPT ); Mon, 13 Dec 2021 06:25:22 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1639394722; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MjRcoCyY5PH9ZdTSsY1tkuIprpWWOIhRjLNdMeQeEGo=; b=VUsPKAuKL9L+fLkwg/NkqtICukKbeiAy03O/SXAfidk/SZSnwHjCndeYVq/XoOqMrdHEAd xvoSEB0X9OaGgIPnkX4KxdRzKQcO/WHWlXTrilzBxw/4yl/YMTAJXlggM4mfvtTbx3OXe7 cV5kdrNYIa+UdV4oLCGNHFZd0sBFiG4= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-135-ajcxGUn4OqKa1oqVWHpm_Q-1; Mon, 13 Dec 2021 06:25:18 -0500 X-MC-Unique: ajcxGUn4OqKa1oqVWHpm_Q-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id CC451801962; Mon, 13 Dec 2021 11:25:16 +0000 (UTC) Received: from virtlab701.virt.lab.eng.bos.redhat.com (virtlab701.virt.lab.eng.bos.redhat.com [10.19.152.228]) by smtp.corp.redhat.com (Postfix) with ESMTP id 01AAE5E272; Mon, 13 Dec 2021 11:25:15 +0000 (UTC) From: Paolo Bonzini To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: seanjc@google.com, ignat@cloudflare.com, bgardon@google.com, dmatlack@google.com, stevensd@chromium.org, kernel-team@cloudflare.com, stable@vger.kernel.org Subject: [PATCH 1/2] KVM: x86: allow kvm_tdp_mmu_zap_invalidated_roots with write-locked mmu_lock Date: Mon, 13 Dec 2021 06:25:13 -0500 Message-Id: <20211213112514.78552-2-pbonzini@redhat.com> In-Reply-To: <20211213112514.78552-1-pbonzini@redhat.com> References: <20211213112514.78552-1-pbonzini@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Zapping within a write-side critical section is more efficient, so it is desirable if we know that no vCPU is running (such as within the .release MMU notifier callback). Prepare for reusing kvm_tdp_mmu_zap_invalidated_roots in such scenarios. Fixes: b7cccd397f31 ("KVM: x86/mmu: Fast invalidation for TDP MMU") Cc: stable@vger.kernel.org Signed-off-by: Paolo Bonzini --- arch/x86/kvm/mmu/mmu.c | 2 +- arch/x86/kvm/mmu/tdp_mmu.c | 17 +++++++++-------- arch/x86/kvm/mmu/tdp_mmu.h | 2 +- 3 files changed, 11 insertions(+), 10 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 4a3bcdd3cfe7..6fe4ab8fc0ca 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -5695,7 +5695,7 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm) if (is_tdp_mmu_enabled(kvm)) { read_lock(&kvm->mmu_lock); - kvm_tdp_mmu_zap_invalidated_roots(kvm); + kvm_tdp_mmu_zap_invalidated_roots(kvm, true); read_unlock(&kvm->mmu_lock); } } diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 1db8496259ad..f2dd5c97bbc2 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -821,14 +821,18 @@ static struct kvm_mmu_page *next_invalidated_root(struct kvm *kvm, * only has to do a trivial amount of work. Since the roots are invalid, * no new SPTEs should be created under them. */ -void kvm_tdp_mmu_zap_invalidated_roots(struct kvm *kvm) +void kvm_tdp_mmu_zap_invalidated_roots(struct kvm *kvm, bool shared) { struct kvm_mmu_page *next_root; struct kvm_mmu_page *root; bool flush = false; - lockdep_assert_held_read(&kvm->mmu_lock); + kvm_lockdep_assert_mmu_lock_held(kvm, shared); + /* + * rcu_read_lock is only needed for shared == true, but we + * always take it for simplicity. + */ rcu_read_lock(); root = next_invalidated_root(kvm, NULL); @@ -838,13 +842,10 @@ void kvm_tdp_mmu_zap_invalidated_roots(struct kvm *kvm) rcu_read_unlock(); - flush = zap_gfn_range(kvm, root, 0, -1ull, true, flush, true); + flush = zap_gfn_range(kvm, root, 0, -1ull, true, flush, shared); - /* - * Put the reference acquired in - * kvm_tdp_mmu_invalidate_roots - */ - kvm_tdp_mmu_put_root(kvm, root, true); + /* Put the reference acquired in kvm_tdp_mmu_invalidate_roots. */ + kvm_tdp_mmu_put_root(kvm, root, shared); root = next_root; diff --git a/arch/x86/kvm/mmu/tdp_mmu.h b/arch/x86/kvm/mmu/tdp_mmu.h index 3899004a5d91..24809f4ed090 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.h +++ b/arch/x86/kvm/mmu/tdp_mmu.h @@ -46,7 +46,7 @@ static inline bool kvm_tdp_mmu_zap_sp(struct kvm *kvm, struct kvm_mmu_page *sp) void kvm_tdp_mmu_zap_all(struct kvm *kvm); void kvm_tdp_mmu_invalidate_all_roots(struct kvm *kvm); -void kvm_tdp_mmu_zap_invalidated_roots(struct kvm *kvm); +void kvm_tdp_mmu_zap_invalidated_roots(struct kvm *kvm, bool shared); int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault); From patchwork Mon Dec 13 11:25:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paolo Bonzini X-Patchwork-Id: 12673773 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1A53AC433EF for ; Mon, 13 Dec 2021 11:25:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235593AbhLMLZb (ORCPT ); Mon, 13 Dec 2021 06:25:31 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:24162 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235473AbhLMLZW (ORCPT ); Mon, 13 Dec 2021 06:25:22 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1639394722; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NgW2HZbqk/vaW0uvmcY9DBIAYMKjrYwVqr2SxXfM1k4=; b=Ps0QhxUSq3/AlzN1kjdEiJx7M2yQ+CgACzvR/hWkxWYcX0dwATrMZpn9M86XGrReyivKZf ILu0/quxRraXhLmgVfzo8o+qZnCCAFKtTNm7Z1nI13LECp1su5RGiImNQoPdnM8ti4zAZt U68jSZw//or4zGVJ984aL6/4G1Q3Eac= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-207-2hzbBmnZM3uE2eCdvOx4TA-1; Mon, 13 Dec 2021 06:25:19 -0500 X-MC-Unique: 2hzbBmnZM3uE2eCdvOx4TA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 95334193F568; Mon, 13 Dec 2021 11:25:17 +0000 (UTC) Received: from virtlab701.virt.lab.eng.bos.redhat.com (virtlab701.virt.lab.eng.bos.redhat.com [10.19.152.228]) by smtp.corp.redhat.com (Postfix) with ESMTP id DA6EF5ED25; Mon, 13 Dec 2021 11:25:16 +0000 (UTC) From: Paolo Bonzini To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: seanjc@google.com, ignat@cloudflare.com, bgardon@google.com, dmatlack@google.com, stevensd@chromium.org, kernel-team@cloudflare.com, stable@vger.kernel.org Subject: [PATCH 2/2] KVM: x86: zap invalid roots in kvm_tdp_mmu_zap_all Date: Mon, 13 Dec 2021 06:25:14 -0500 Message-Id: <20211213112514.78552-3-pbonzini@redhat.com> In-Reply-To: <20211213112514.78552-1-pbonzini@redhat.com> References: <20211213112514.78552-1-pbonzini@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org kvm_tdp_mmu_zap_all is intended to visit all roots and zap their page tables, which flushes the accessed and dirty bits out to the Linux "struct page"s. Missing some of the roots has catastrophic effects, because kvm_tdp_mmu_zap_all is called when the MMU notifier is being removed and any PTEs left behind might become dangling by the time kvm-arch_destroy_vm tears down the roots for good. Unfortunately that is exactly what kvm_tdp_mmu_zap_all is doing: it visits all roots via for_each_tdp_mmu_root_yield_safe, which in turn uses kvm_tdp_mmu_get_root to skip invalid roots. If the current root is invalid at the time of kvm_tdp_mmu_zap_all, its page tables will remain in place but will later be zapped during kvm_arch_destroy_vm. To fix this, ensure that kvm_tdp_mmu_zap_all goes over all roots, including the invalid ones. The easiest way to do so is for kvm_tdp_mmu_zap_all to do the same as kvm_mmu_zap_all_fast: invalidate all roots, and then zap the invalid roots. However, there is no need to go through tdp_mmu_zap_spte_atomic because there are no running vCPUs. Fixes: b7cccd397f31 ("KVM: x86/mmu: Fast invalidation for TDP MMU") Cc: stable@vger.kernel.org Reported-by: Ignat Korchagin Signed-off-by: Paolo Bonzini --- arch/x86/kvm/mmu/tdp_mmu.c | 25 +++++++++++++------------ 1 file changed, 13 insertions(+), 12 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index f2dd5c97bbc2..ce3fafb6c9a7 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -779,18 +779,6 @@ bool __kvm_tdp_mmu_zap_gfn_range(struct kvm *kvm, int as_id, gfn_t start, return flush; } -void kvm_tdp_mmu_zap_all(struct kvm *kvm) -{ - bool flush = false; - int i; - - for (i = 0; i < KVM_ADDRESS_SPACE_NUM; i++) - flush = kvm_tdp_mmu_zap_gfn_range(kvm, i, 0, -1ull, flush); - - if (flush) - kvm_flush_remote_tlbs(kvm); -} - static struct kvm_mmu_page *next_invalidated_root(struct kvm *kvm, struct kvm_mmu_page *prev_root) { @@ -888,6 +876,19 @@ void kvm_tdp_mmu_invalidate_all_roots(struct kvm *kvm) root->role.invalid = true; } +void kvm_tdp_mmu_zap_all(struct kvm *kvm) +{ + /* + * We need to zap all roots, including already-invalid ones. The + * easiest way is to ensure there's only invalid roots which then, + * for efficiency, we zap while mmu_lock is taken exclusively. + * Since the MMU notifier is being torn down, contention on the + * mmu_lock is not an issue. + */ + kvm_tdp_mmu_invalidate_all_roots(kvm); + kvm_tdp_mmu_zap_invalidated_roots(kvm, false); +} + /* * Installs a last-level SPTE to handle a TDP page fault. * (NPT/EPT violation/misconfiguration)