From patchwork Wed Dec 21 22:24:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ben Gardon X-Patchwork-Id: 13079190 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E8035C10F1B for ; Wed, 21 Dec 2022 22:24:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234937AbiLUWY1 (ORCPT ); Wed, 21 Dec 2022 17:24:27 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36130 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234858AbiLUWYY (ORCPT ); Wed, 21 Dec 2022 17:24:24 -0500 Received: from mail-pf1-x44a.google.com (mail-pf1-x44a.google.com [IPv6:2607:f8b0:4864:20::44a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 06D402715F for ; Wed, 21 Dec 2022 14:24:24 -0800 (PST) Received: by mail-pf1-x44a.google.com with SMTP id z13-20020aa79f8d000000b00576b614b7d2so9181083pfr.14 for ; Wed, 21 Dec 2022 14:24:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=aKlgYMYIrlGLbMYP963Rt32Noqo2pCtoxK8D34uyWJ4=; b=PWqhNCgkLNsL4LTq6n3P2N/wL2IC5fNtKlHcljZPjv1KMJIXNnu+jM/wiOaUfvSQzQ sqxmogZSTOlLja/osRpS7Ow8EfzSUYnreoaLWkkFdbixfiBmjccx0+KswydFi9OPd7G+ ufafOm2aNUYdbkAWCv2FGT5m/chD2Utam3lq9+XJO5/Q1b+hNSmrMeQga388z7ynT7v8 NX7DKNfpSeQMnVeRYugzoXNHWHSvxsw+I6ma6X+E6cBtDFf2MAyzy1YNcTGk8tmCzLNd ney/iJw+QhWYPcoct/qe9KDj8ngJbHLXsqU23YGjejcPJ/Eqxv/IM84tu17ag52e0e8x 5slQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=aKlgYMYIrlGLbMYP963Rt32Noqo2pCtoxK8D34uyWJ4=; b=Vc/LBMPTGvxZ+dMNqQ5LbsTt6i8ZjBuLJHqfbFvVu18xWlcJDvYTAMZg+HzOLaWLZ8 ffupc6u2bmzUrwlaHah0XFxaduKniPw8UCLdFWnjyCYillAhCDjfSXxLc+gtBrcTIbkA QpbyeuF02nJnGNwES+I1do8qePlab38YB5/b1gOWGt2aXiqXrwELd2ClBEOprILYPKsi kL8LrGCCKmlCB8Gn9GwtuSr0My3lXEChktWuHqv6dabj2un/8Ot0wPxkw/liKnp1wk9O SU58DYXlQgVwdCAWjoJC6jwfjRJhv9gUEyJmVKF+qyVWDipw6Iskb6C6h/lt/1ZIDELT KfTQ== X-Gm-Message-State: AFqh2kqDH0DPwzhckIS9wIQY/I4PiQ/1Rymk8LJCmE8zvCGHtza1zruP VrcfSQccQyd4pLh9gWtuMg0EP9BJ4z/K X-Google-Smtp-Source: AMrXdXtezmpm9/qcQiNZToPZHhHL5FScFsYx6ESyDed8IH5dss9kLF5PXnxx3ct4s9IE+lK/su3Wnrko61my X-Received: from sweer.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:e45]) (user=bgardon job=sendgmr) by 2002:a17:90a:6845:b0:219:2bc2:f71e with SMTP id e5-20020a17090a684500b002192bc2f71emr274766pjm.142.1671661463529; Wed, 21 Dec 2022 14:24:23 -0800 (PST) Date: Wed, 21 Dec 2022 22:24:05 +0000 In-Reply-To: <20221221222418.3307832-1-bgardon@google.com> Mime-Version: 1.0 References: <20221221222418.3307832-1-bgardon@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20221221222418.3307832-2-bgardon@google.com> Subject: [RFC 01/14] KVM: x86/MMU: Add shadow_mmu.(c|h) From: Ben Gardon To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: Paolo Bonzini , Peter Xu , Sean Christopherson , David Matlack , Vipin Sharma , Nagareddy Reddy , Ben Gardon Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org As a first step to splitting the Shadow MMU out of KVM MMU common code, add separate files for it with some of the boilerplate and includes the Shadow MMU will need. No functional change intended. Signed-off-by: Ben Gardon --- arch/x86/kvm/Makefile | 2 +- arch/x86/kvm/mmu/mmu.c | 1 + arch/x86/kvm/mmu/shadow_mmu.c | 21 +++++++++++++++++++++ arch/x86/kvm/mmu/shadow_mmu.h | 8 ++++++++ 4 files changed, 31 insertions(+), 1 deletion(-) create mode 100644 arch/x86/kvm/mmu/shadow_mmu.c create mode 100644 arch/x86/kvm/mmu/shadow_mmu.h diff --git a/arch/x86/kvm/Makefile b/arch/x86/kvm/Makefile index 80e3fe184d17..d6e94660b006 100644 --- a/arch/x86/kvm/Makefile +++ b/arch/x86/kvm/Makefile @@ -12,7 +12,7 @@ include $(srctree)/virt/kvm/Makefile.kvm kvm-y += x86.o emulate.o i8259.o irq.o lapic.o \ i8254.o ioapic.o irq_comm.o cpuid.o pmu.o mtrr.o \ hyperv.o debugfs.o mmu/mmu.o mmu/page_track.o \ - mmu/spte.o + mmu/spte.o mmu/shadow_mmu.o ifdef CONFIG_HYPERV kvm-y += kvm_onhyperv.o diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 4736d7849c60..07b99a7ce830 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -20,6 +20,7 @@ #include "mmu.h" #include "mmu_internal.h" #include "tdp_mmu.h" +#include "shadow_mmu.h" #include "x86.h" #include "kvm_cache_regs.h" #include "smm.h" diff --git a/arch/x86/kvm/mmu/shadow_mmu.c b/arch/x86/kvm/mmu/shadow_mmu.c new file mode 100644 index 000000000000..7bce5ec52b2e --- /dev/null +++ b/arch/x86/kvm/mmu/shadow_mmu.c @@ -0,0 +1,21 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * KVM Shadow MMU + * + * This file implements the Shadow MMU: the KVM MMU implementation which has + * developed organically from hardware which did not have second level paging, + * and so used "shadow paging" to virtualize guest memory. The Shadow MMU is + * an alternative to the TDP MMU which only supports hardware with Two + * Dimentional Paging. (e.g. EPT on Intel or NPT on AMD CPUs.) Note that the + * Shadow MMU also supports TDP, it's just less scalable. The Shadow and TDP + * MMUs can cooperate to support nested virtualization on hardware with TDP. + */ +#include "mmu.h" +#include "mmu_internal.h" +#include "mmutrace.h" +#include "shadow_mmu.h" +#include "spte.h" + +#include +#include +#include diff --git a/arch/x86/kvm/mmu/shadow_mmu.h b/arch/x86/kvm/mmu/shadow_mmu.h new file mode 100644 index 000000000000..719b10f6c403 --- /dev/null +++ b/arch/x86/kvm/mmu/shadow_mmu.h @@ -0,0 +1,8 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef __KVM_X86_MMU_SHADOW_MMU_H +#define __KVM_X86_MMU_SHADOW_MMU_H + +#include + +#endif /* __KVM_X86_MMU_SHADOW_MMU_H */ From patchwork Wed Dec 21 22:24:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ben Gardon X-Patchwork-Id: 13079191 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 37260C4332F for ; Wed, 21 Dec 2022 22:24:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234942AbiLUWY3 (ORCPT ); Wed, 21 Dec 2022 17:24:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36148 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229601AbiLUWY0 (ORCPT ); Wed, 21 Dec 2022 17:24:26 -0500 Received: from mail-pl1-x64a.google.com (mail-pl1-x64a.google.com [IPv6:2607:f8b0:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DAE102715A for ; Wed, 21 Dec 2022 14:24:25 -0800 (PST) Received: by mail-pl1-x64a.google.com with SMTP id l7-20020a170902f68700b00192495b1f10so153147plg.2 for ; Wed, 21 Dec 2022 14:24:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=mr/upkq9nUstxPAXFOYz+TIWTjnNhJ7646Ga8zDx7MA=; b=jcxCfXrS0rhqjdrrTg9RFmHommcZLL9JJIOJmSEjlqGB6hsqaBvlotnqSYyzuM2o1r oyxgX7LusyGPkW4t09A2Ps3Tbd/T0zkw/iHV6uV3gwWeBDbz3xZpksS8uIEZQReN8k4D gTZ/kgRejuP4J7gYltEmSRz4GYSehjtuE/FLqamWE/P7u15/c9ZToC0iogQQIP8Pejds ii5k/zaJDE/8yX30EqZr7Fdi6ezF9uKNfAO4Wkpzv0osK8ojhKjyjCCtLfLzO/HKX78u wOc1P5laTqWlvK39uvp0tnnasQLlOXiwegfhLRQh58XXgN51EcR94i5Ib2o2lx+A3Bi6 sIuw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=mr/upkq9nUstxPAXFOYz+TIWTjnNhJ7646Ga8zDx7MA=; b=UmmHhhsYnuM1beYVIJfYBNsGIC9ycCQTEyRvYYlhfytcvXfK4HYP8Kx+ZrZ8DiaG3R MWMK+oTKlKC/Nbi0933h5Fp1W6l7Ilca/u8OXN6Kkvy7U9VBIjSbRgXrwG8IpA7HGYoq gfhOp0mpBVd5dhsIwOmtv4Ot0kodF/zVSOWGzYrefktQARSTDtR//hWyTQUEi2VGcNrW Vw+Ps7gnRavD2RzEC2cONfUS6BOVFFHYuIb1v/ocveKqjwMhSym72K3NcCg6AokSfzOw PadOPEKpqI6lFZhZ0stjARHZ+kQ3pqadAOzTPSSMBIvyFWFDAf2Rhz6RxoOWtITvM+nF 4XxA== X-Gm-Message-State: AFqh2krOU6xDrPMzKAeioFVL4GuRjl+XWtzKpa1XVelaq8Qx8Vyg1Jwe Jl5J+M5rn0N7iX9SKrcYLgsFLAkV0GyG X-Google-Smtp-Source: AMrXdXtD9skHtEgX98zo+vsV8XbpbOjpDWDpMzej9vuGx7SmxRfWxOXIU8CMO9TrBWF83dJViVPY5tL4GZ10 X-Received: from sweer.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:e45]) (user=bgardon job=sendgmr) by 2002:a62:2781:0:b0:57a:6e2a:c236 with SMTP id n123-20020a622781000000b0057a6e2ac236mr218098pfn.82.1671661465306; Wed, 21 Dec 2022 14:24:25 -0800 (PST) Date: Wed, 21 Dec 2022 22:24:06 +0000 In-Reply-To: <20221221222418.3307832-1-bgardon@google.com> Mime-Version: 1.0 References: <20221221222418.3307832-1-bgardon@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20221221222418.3307832-3-bgardon@google.com> Subject: [RFC 02/14] KVM: x86/MMU: Expose functions for the Shadow MMU From: Ben Gardon To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: Paolo Bonzini , Peter Xu , Sean Christopherson , David Matlack , Vipin Sharma , Nagareddy Reddy , Ben Gardon Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Expose various common MMU functions which the Shadow MMU will need via mmu_internal.h. This just slightly reduces the work needed to move the shadow MMU code out of mmu.c, which will already be a massive change. No functional change intended. Signed-off-by: Ben Gardon --- arch/x86/kvm/mmu/mmu.c | 41 ++++++++++++++------------------- arch/x86/kvm/mmu/mmu_internal.h | 24 +++++++++++++++++++ 2 files changed, 41 insertions(+), 24 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 07b99a7ce830..729a2799d4d7 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -156,9 +156,9 @@ struct kvm_shadow_walk_iterator { ({ spte = mmu_spte_get_lockless(_walker.sptep); 1; }); \ __shadow_walk_next(&(_walker), spte)) -static struct kmem_cache *pte_list_desc_cache; +struct kmem_cache *pte_list_desc_cache; struct kmem_cache *mmu_page_header_cache; -static struct percpu_counter kvm_total_used_mmu_pages; +struct percpu_counter kvm_total_used_mmu_pages; static void mmu_spte_set(u64 *sptep, u64 spte); @@ -234,11 +234,6 @@ static struct kvm_mmu_role_regs vcpu_to_role_regs(struct kvm_vcpu *vcpu) return regs; } -static inline bool kvm_available_flush_tlb_with_range(void) -{ - return kvm_x86_ops.tlb_remote_flush_with_range; -} - static void kvm_flush_remote_tlbs_with_range(struct kvm *kvm, struct kvm_tlb_range *range) { @@ -262,8 +257,8 @@ void kvm_flush_remote_tlbs_with_address(struct kvm *kvm, kvm_flush_remote_tlbs_with_range(kvm, &range); } -static void mark_mmio_spte(struct kvm_vcpu *vcpu, u64 *sptep, u64 gfn, - unsigned int access) +void mark_mmio_spte(struct kvm_vcpu *vcpu, u64 *sptep, u64 gfn, + unsigned int access) { u64 spte = make_mmio_spte(vcpu, gfn, access); @@ -610,7 +605,7 @@ static bool mmu_spte_age(u64 *sptep) return true; } -static void walk_shadow_page_lockless_begin(struct kvm_vcpu *vcpu) +void walk_shadow_page_lockless_begin(struct kvm_vcpu *vcpu) { if (is_tdp_mmu(vcpu->arch.mmu)) { kvm_tdp_mmu_walk_lockless_begin(); @@ -629,7 +624,7 @@ static void walk_shadow_page_lockless_begin(struct kvm_vcpu *vcpu) } } -static void walk_shadow_page_lockless_end(struct kvm_vcpu *vcpu) +void walk_shadow_page_lockless_end(struct kvm_vcpu *vcpu) { if (is_tdp_mmu(vcpu->arch.mmu)) { kvm_tdp_mmu_walk_lockless_end(); @@ -822,8 +817,8 @@ void track_possible_nx_huge_page(struct kvm *kvm, struct kvm_mmu_page *sp) &kvm->arch.possible_nx_huge_pages); } -static void account_nx_huge_page(struct kvm *kvm, struct kvm_mmu_page *sp, - bool nx_huge_page_possible) +void account_nx_huge_page(struct kvm *kvm, struct kvm_mmu_page *sp, + bool nx_huge_page_possible) { sp->nx_huge_page_disallowed = true; @@ -857,16 +852,15 @@ void untrack_possible_nx_huge_page(struct kvm *kvm, struct kvm_mmu_page *sp) list_del_init(&sp->possible_nx_huge_page_link); } -static void unaccount_nx_huge_page(struct kvm *kvm, struct kvm_mmu_page *sp) +void unaccount_nx_huge_page(struct kvm *kvm, struct kvm_mmu_page *sp) { sp->nx_huge_page_disallowed = false; untrack_possible_nx_huge_page(kvm, sp); } -static struct kvm_memory_slot * -gfn_to_memslot_dirty_bitmap(struct kvm_vcpu *vcpu, gfn_t gfn, - bool no_dirty_log) +struct kvm_memory_slot *gfn_to_memslot_dirty_bitmap(struct kvm_vcpu *vcpu, + gfn_t gfn, bool no_dirty_log) { struct kvm_memory_slot *slot; @@ -1403,7 +1397,7 @@ bool kvm_mmu_slot_gfn_write_protect(struct kvm *kvm, return write_protected; } -static bool kvm_vcpu_write_protect_gfn(struct kvm_vcpu *vcpu, u64 gfn) +bool kvm_vcpu_write_protect_gfn(struct kvm_vcpu *vcpu, u64 gfn) { struct kvm_memory_slot *slot; @@ -1902,9 +1896,8 @@ static int kvm_sync_page(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, return ret; } -static bool kvm_mmu_remote_flush_or_zap(struct kvm *kvm, - struct list_head *invalid_list, - bool remote_flush) +bool kvm_mmu_remote_flush_or_zap(struct kvm *kvm, struct list_head *invalid_list, + bool remote_flush) { if (!remote_flush && list_empty(invalid_list)) return false; @@ -1916,7 +1909,7 @@ static bool kvm_mmu_remote_flush_or_zap(struct kvm *kvm, return true; } -static bool is_obsolete_sp(struct kvm *kvm, struct kvm_mmu_page *sp) +bool is_obsolete_sp(struct kvm *kvm, struct kvm_mmu_page *sp) { if (sp->role.invalid) return true; @@ -6148,7 +6141,7 @@ static inline bool need_topup(struct kvm_mmu_memory_cache *cache, int min) return kvm_mmu_memory_cache_nr_free_objects(cache) < min; } -static bool need_topup_split_caches_or_resched(struct kvm *kvm) +bool need_topup_split_caches_or_resched(struct kvm *kvm) { if (need_resched() || rwlock_needbreak(&kvm->mmu_lock)) return true; @@ -6163,7 +6156,7 @@ static bool need_topup_split_caches_or_resched(struct kvm *kvm) need_topup(&kvm->arch.split_shadow_page_cache, 1); } -static int topup_split_caches(struct kvm *kvm) +int topup_split_caches(struct kvm *kvm) { /* * Allocating rmap list entries when splitting huge pages for nested diff --git a/arch/x86/kvm/mmu/mmu_internal.h b/arch/x86/kvm/mmu/mmu_internal.h index dbaf6755c5a7..856e2e0a8420 100644 --- a/arch/x86/kvm/mmu/mmu_internal.h +++ b/arch/x86/kvm/mmu/mmu_internal.h @@ -131,7 +131,9 @@ struct kvm_mmu_page { #endif }; +extern struct kmem_cache *pte_list_desc_cache; extern struct kmem_cache *mmu_page_header_cache; +extern struct percpu_counter kvm_total_used_mmu_pages; static inline int kvm_mmu_role_as_id(union kvm_mmu_page_role role) { @@ -317,6 +319,28 @@ void disallowed_hugepage_adjust(struct kvm_page_fault *fault, u64 spte, int cur_ void *mmu_memory_cache_alloc(struct kvm_mmu_memory_cache *mc); void track_possible_nx_huge_page(struct kvm *kvm, struct kvm_mmu_page *sp); +void account_nx_huge_page(struct kvm *kvm, struct kvm_mmu_page *sp, + bool nx_huge_page_possible); void untrack_possible_nx_huge_page(struct kvm *kvm, struct kvm_mmu_page *sp); +void unaccount_nx_huge_page(struct kvm *kvm, struct kvm_mmu_page *sp); +static inline bool kvm_available_flush_tlb_with_range(void) +{ + return kvm_x86_ops.tlb_remote_flush_with_range; +} + +void mark_mmio_spte(struct kvm_vcpu *vcpu, u64 *sptep, u64 gfn, + unsigned int access); +struct kvm_memory_slot *gfn_to_memslot_dirty_bitmap(struct kvm_vcpu *vcpu, + gfn_t gfn, bool no_dirty_log); +bool kvm_vcpu_write_protect_gfn(struct kvm_vcpu *vcpu, u64 gfn); +bool kvm_mmu_remote_flush_or_zap(struct kvm *kvm, struct list_head *invalid_list, + bool remote_flush); +bool is_obsolete_sp(struct kvm *kvm, struct kvm_mmu_page *sp); + +void walk_shadow_page_lockless_begin(struct kvm_vcpu *vcpu); +void walk_shadow_page_lockless_end(struct kvm_vcpu *vcpu); + +bool need_topup_split_caches_or_resched(struct kvm *kvm); +int topup_split_caches(struct kvm *kvm); #endif /* __KVM_X86_MMU_INTERNAL_H */ From patchwork Wed Dec 21 22:24:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ben Gardon X-Patchwork-Id: 13079192 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21021C4332F for ; Wed, 21 Dec 2022 22:24:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234964AbiLUWYe (ORCPT ); Wed, 21 Dec 2022 17:24:34 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36228 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234948AbiLUWYb (ORCPT ); Wed, 21 Dec 2022 17:24:31 -0500 Received: from mail-pl1-x649.google.com (mail-pl1-x649.google.com [IPv6:2607:f8b0:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 578BB2715E for ; Wed, 21 Dec 2022 14:24:29 -0800 (PST) Received: by mail-pl1-x649.google.com with SMTP id jc4-20020a17090325c400b00189ceee4049so152620plb.3 for ; Wed, 21 Dec 2022 14:24:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=yXovaXGmQ2hj+xhfXzlYPf6jy3g3M6Z97i9sJYKtL7o=; b=XUqeRrYpmjbz24aOUCRdZhU/hasfAHWkBLZ4lTvglulOiCrqV4g30TxCuKN75BwGCU 4LXBLsZDthjsLf+2AAZudjXxcPvh/idm+4kfCCahPLRDtw4awMTHPJRM+3GLuCE2z2xW UXWY9noDLDSwD7MOFEHp549vimmJUdsZlZLlkat6N1yZkyca8ji4la9bIC+7aRA7MNjJ YtbFM6nqgK4y9kxkqqzX2AYDFPzZeTYjsNQEJ2emwoQuCWJEmRyoXNEcGGrMTsjlBGmI JZjMUvnzAPwUHXLd7BkZn8APBej2IkOxylMhK0+Z0QbMqCvFV2hi5iU8e6JCqA3ryxd/ vIwg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=yXovaXGmQ2hj+xhfXzlYPf6jy3g3M6Z97i9sJYKtL7o=; b=S25hcHMGVzrIpcrgRM9zjR54QMNcTx5nKhcmd79tw2NElUsThtBgtBvRTUnEcUMMbP +wAQnOBn7SSXqXgVtioGl0l+BD8SfVPhhf/4qCAZ9ZwzYA4w6OTUoHWmAKijUgfUVvs1 1gU2FSCTnVjhrhsaIsEyL4JkBry0REPTePzUmjfcvCt3OjMXtDigUs93VqcaoFhIrFNQ C7k/eCeb80mu++/ySRuxgd2/u1DNeoTTOKifJDhsB6OIr5zFpEMBGe40zbUey+JPKjiJ xJHRCnR7gv8ss1vNcseYt108LpqlvQ9MVZS3vFerUUIn1kRlPNQ6G8BJyzieko9dm0+e 8MOQ== X-Gm-Message-State: AFqh2kpSLN9s3FrHy7ouaceU6bla8r7T4jRnM6H2jkThLqDJ2PH3AdLJ SJxnTdZeMNI053yEWUff9ghnWOQlG6bD X-Google-Smtp-Source: AMrXdXv8RdH6TwKdO23ZhAUnv4pAyWPuCeF7bltNA6yJNFTlLDH8JfMjr5eV9VokhVrwkrGyuruwOWr5Fb3F X-Received: from sweer.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:e45]) (user=bgardon job=sendgmr) by 2002:a17:90a:9913:b0:225:aa06:8fcb with SMTP id b19-20020a17090a991300b00225aa068fcbmr167806pjp.78.1671661468931; Wed, 21 Dec 2022 14:24:28 -0800 (PST) Date: Wed, 21 Dec 2022 22:24:08 +0000 In-Reply-To: <20221221222418.3307832-1-bgardon@google.com> Mime-Version: 1.0 References: <20221221222418.3307832-1-bgardon@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20221221222418.3307832-5-bgardon@google.com> Subject: [RFC 04/14] KVM: x86/MMU: Expose functions for paging_tmpl.h From: Ben Gardon To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: Paolo Bonzini , Peter Xu , Sean Christopherson , David Matlack , Vipin Sharma , Nagareddy Reddy , Ben Gardon Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org In preparation for moving paging_tmpl.h to shadow_mmu.c, expose various functions it needs through mmu_internal.h. This includes modifying the BUILD_MMU_ROLE_ACCESSOR macro so that it does not automatically include the static label, since some but not all of the accessors are needed by paging_tmpl.h. No functional change intended. Signed-off-by: Ben Gardon --- arch/x86/kvm/mmu/mmu.c | 32 ++++++++++++++++---------------- arch/x86/kvm/mmu/mmu_internal.h | 16 ++++++++++++++++ 2 files changed, 32 insertions(+), 16 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index bf14e181eb12..a17e8a79e4df 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -153,18 +153,18 @@ BUILD_MMU_ROLE_REGS_ACCESSOR(efer, lma, EFER_LMA); * and the vCPU may be incorrect/irrelevant. */ #define BUILD_MMU_ROLE_ACCESSOR(base_or_ext, reg, name) \ -static inline bool __maybe_unused is_##reg##_##name(struct kvm_mmu *mmu) \ +inline bool __maybe_unused is_##reg##_##name(struct kvm_mmu *mmu) \ { \ return !!(mmu->cpu_role. base_or_ext . reg##_##name); \ } BUILD_MMU_ROLE_ACCESSOR(base, cr0, wp); -BUILD_MMU_ROLE_ACCESSOR(ext, cr4, pse); +static BUILD_MMU_ROLE_ACCESSOR(ext, cr4, pse); BUILD_MMU_ROLE_ACCESSOR(ext, cr4, smep); -BUILD_MMU_ROLE_ACCESSOR(ext, cr4, smap); -BUILD_MMU_ROLE_ACCESSOR(ext, cr4, pke); -BUILD_MMU_ROLE_ACCESSOR(ext, cr4, la57); +static BUILD_MMU_ROLE_ACCESSOR(ext, cr4, smap); +static BUILD_MMU_ROLE_ACCESSOR(ext, cr4, pke); +static BUILD_MMU_ROLE_ACCESSOR(ext, cr4, la57); BUILD_MMU_ROLE_ACCESSOR(base, efer, nx); -BUILD_MMU_ROLE_ACCESSOR(ext, efer, lma); +static BUILD_MMU_ROLE_ACCESSOR(ext, efer, lma); static inline bool is_cr0_pg(struct kvm_mmu *mmu) { @@ -210,7 +210,7 @@ void kvm_flush_remote_tlbs_with_address(struct kvm *kvm, kvm_flush_remote_tlbs_with_range(kvm, &range); } -static gfn_t get_mmio_spte_gfn(u64 spte) +gfn_t get_mmio_spte_gfn(u64 spte) { u64 gpa = spte & shadow_nonpresent_or_rsvd_lower_gfn_mask; @@ -240,7 +240,7 @@ static bool check_mmio_spte(struct kvm_vcpu *vcpu, u64 spte) return likely(kvm_gen == spte_gen); } -static int is_cpuid_PSE36(void) +int is_cpuid_PSE36(void) { return 1; } @@ -279,7 +279,7 @@ void walk_shadow_page_lockless_end(struct kvm_vcpu *vcpu) } } -static int mmu_topup_memory_caches(struct kvm_vcpu *vcpu, bool maybe_indirect) +int mmu_topup_memory_caches(struct kvm_vcpu *vcpu, bool maybe_indirect) { int r; @@ -818,8 +818,8 @@ static int kvm_handle_error_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn) return -EFAULT; } -static int handle_abnormal_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault, - unsigned int access) +int handle_abnormal_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault, + unsigned int access) { /* The pfn is invalid, report the error! */ if (unlikely(is_error_pfn(fault->pfn))) @@ -1275,8 +1275,8 @@ static int handle_mmio_page_fault(struct kvm_vcpu *vcpu, u64 addr, bool direct) return RET_PF_RETRY; } -static bool page_fault_handle_page_track(struct kvm_vcpu *vcpu, - struct kvm_page_fault *fault) +bool page_fault_handle_page_track(struct kvm_vcpu *vcpu, + struct kvm_page_fault *fault) { if (unlikely(fault->rsvd)) return false; @@ -1338,7 +1338,7 @@ void kvm_arch_async_page_ready(struct kvm_vcpu *vcpu, struct kvm_async_pf *work) kvm_mmu_do_page_fault(vcpu, work->cr2_or_gpa, 0, true); } -static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) +int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) { struct kvm_memory_slot *slot = fault->slot; bool async; @@ -1403,8 +1403,8 @@ static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) * Returns true if the page fault is stale and needs to be retried, i.e. if the * root was invalidated by a memslot update or a relevant mmu_notifier fired. */ -static bool is_page_fault_stale(struct kvm_vcpu *vcpu, - struct kvm_page_fault *fault, int mmu_seq) +bool is_page_fault_stale(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault, + int mmu_seq) { struct kvm_mmu_page *sp = to_shadow_page(vcpu->arch.mmu->root.hpa); diff --git a/arch/x86/kvm/mmu/mmu_internal.h b/arch/x86/kvm/mmu/mmu_internal.h index 74a99b67f09e..957376fcb333 100644 --- a/arch/x86/kvm/mmu/mmu_internal.h +++ b/arch/x86/kvm/mmu/mmu_internal.h @@ -341,6 +341,22 @@ bool is_obsolete_sp(struct kvm *kvm, struct kvm_mmu_page *sp); void walk_shadow_page_lockless_begin(struct kvm_vcpu *vcpu); void walk_shadow_page_lockless_end(struct kvm_vcpu *vcpu); +int mmu_topup_memory_caches(struct kvm_vcpu *vcpu, bool maybe_indirect); bool need_topup_split_caches_or_resched(struct kvm *kvm); int topup_split_caches(struct kvm *kvm); + +bool is_page_fault_stale(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault, + int mmu_seq); +bool page_fault_handle_page_track(struct kvm_vcpu *vcpu, + struct kvm_page_fault *fault); +int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault); +int handle_abnormal_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault, + unsigned int access); + +gfn_t get_mmio_spte_gfn(u64 spte); + +bool is_efer_nx(struct kvm_mmu *mmu); +bool is_cr4_smep(struct kvm_mmu *mmu); +bool is_cr0_wp(struct kvm_mmu *mmu); +int is_cpuid_PSE36(void); #endif /* __KVM_X86_MMU_INTERNAL_H */ From patchwork Wed Dec 21 22:24:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ben Gardon X-Patchwork-Id: 13079193 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 087D9C4332F for ; Wed, 21 Dec 2022 22:25:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234967AbiLUWZD (ORCPT ); Wed, 21 Dec 2022 17:25:03 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36322 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234960AbiLUWYd (ORCPT ); Wed, 21 Dec 2022 17:24:33 -0500 Received: from mail-pf1-x44a.google.com (mail-pf1-x44a.google.com [IPv6:2607:f8b0:4864:20::44a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 65342275C6 for ; Wed, 21 Dec 2022 14:24:31 -0800 (PST) Received: by mail-pf1-x44a.google.com with SMTP id u3-20020a056a00124300b0056d4ab0c7cbso1849pfi.7 for ; Wed, 21 Dec 2022 14:24:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=8kZrCNGBjg1WS7ODt19uiL6/LMaIleV0jy9dXg5qXsU=; b=gm4wVDeMjNDC25vIMLZJDuIuxtNrgMkW/U8DzY9/JjzdiZzvFqg5Z9nrrcuVrdI+Cr mlp9FAy3A+XDVeEZyHG8Xc52Dea5StINuVBgplWUrUUWZ69CNhfzLT4HzTN4LN1TEk0D IpYExoj/5fAFNK6+d1bgopQ28Zgn7SVezYfuMlI3pItzNuDAxxrhAWNTk67M+IKLP4c2 n+qybJS7g4bTiF/kDELV1VGdQmRTOxAQSwONIrFtaMW76yMhZ3d0Kp39BOtqavNeur3r Auhjsa2ghpUls+QQDvUrWngcD+RIh8YLSET0PWut8cdrrGwgVsH0f/PqQl4TB9PEBsIV uuCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=8kZrCNGBjg1WS7ODt19uiL6/LMaIleV0jy9dXg5qXsU=; b=my9+ZGlkppHM0/5jmu7u7EMNUaNYzHpnM0ERdQeAX6T930N4xSTA7YVi9TxwbYTOzl KorRwPJtSyDAqUpVEU/L0G9m8Ec5IwoynggJ10dbkjkID9LVnYHeMwxe+jKizYqZYPch +L/PwR6mDDZ+uZwFTrP8PsKCAVbgAR/KvmSvwapznVl9UZPDCq49b9MMJ3mU5cWKWYPS wcjPvkMMwiL1Ksb7CFiVUj5RLLYVVEgDY1wZX4Z8z1hlRMYv+pyVRFH7sAIyg2ZUyYAc GEQYFH+ZBt9CKrFXDMGIQ2V5gOtIQIQ1xua4hGtihMXkTXkZ3Ek1S/vAc9JjXv2ByaII 5snA== X-Gm-Message-State: AFqh2kqFf4q3ib7r9YksdTTu0h2xnprd8vTn9sEWKhj+PF401odaqIVT SkIC/m6n+a5V0HJ92xquLnbgCOY4pk5u X-Google-Smtp-Source: AMrXdXv30EHXL+5DK+Y+8Ijs4U9jsgv9W+gn+nLjMgEdO1gBmY8O8wHR5Scv0SFd17HJN2nRWWz7fI2pi91H X-Received: from sweer.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:e45]) (user=bgardon job=sendgmr) by 2002:a17:902:d510:b0:189:ced9:c9e7 with SMTP id b16-20020a170902d51000b00189ced9c9e7mr251984plg.108.1671661470555; Wed, 21 Dec 2022 14:24:30 -0800 (PST) Date: Wed, 21 Dec 2022 22:24:09 +0000 In-Reply-To: <20221221222418.3307832-1-bgardon@google.com> Mime-Version: 1.0 References: <20221221222418.3307832-1-bgardon@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20221221222418.3307832-6-bgardon@google.com> Subject: [RFC 05/14] KVM: x86/MMU: Move paging_tmpl.h includes to shadow_mmu.c From: Ben Gardon To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: Paolo Bonzini , Peter Xu , Sean Christopherson , David Matlack , Vipin Sharma , Nagareddy Reddy , Ben Gardon Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Move the integration point for paging_tmpl.h to shadow_mmu.c since paging_tmpl.h is ostensibly part of the Shadow MMU. This requires modifying some of the definitions to be non-static and then exporting the pre-processed function names through shadow_mmu.h since they are needed for mmu context callbacks in mmu.c. This will facilitate cleanups in following commits because many of the functions being exposed by shadow_mmu.h are only needed by paging_tmpl.h. Those functions will no longer need to be exported. sync_mmio_spte() is only used by paging_tmpl.h, so move it along with the includes. No functional change intended. Signed-off-by: Ben Gardon --- arch/x86/kvm/mmu/mmu.c | 29 ----------------------------- arch/x86/kvm/mmu/paging_tmpl.h | 11 +++++------ arch/x86/kvm/mmu/shadow_mmu.c | 30 ++++++++++++++++++++++++++++++ arch/x86/kvm/mmu/shadow_mmu.h | 25 ++++++++++++++++++++++++- 4 files changed, 59 insertions(+), 36 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index a17e8a79e4df..dd97e346c786 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1699,35 +1699,6 @@ static unsigned long get_cr3(struct kvm_vcpu *vcpu) return kvm_read_cr3(vcpu); } -static bool sync_mmio_spte(struct kvm_vcpu *vcpu, u64 *sptep, gfn_t gfn, - unsigned int access) -{ - if (unlikely(is_mmio_spte(*sptep))) { - if (gfn != get_mmio_spte_gfn(*sptep)) { - mmu_spte_clear_no_track(sptep); - return true; - } - - mark_mmio_spte(vcpu, sptep, gfn, access); - return true; - } - - return false; -} - -#define PTTYPE_EPT 18 /* arbitrary */ -#define PTTYPE PTTYPE_EPT -#include "paging_tmpl.h" -#undef PTTYPE - -#define PTTYPE 64 -#include "paging_tmpl.h" -#undef PTTYPE - -#define PTTYPE 32 -#include "paging_tmpl.h" -#undef PTTYPE - static void __reset_rsvds_bits_mask(struct rsvd_bits_validate *rsvd_check, u64 pa_bits_rsvd, int level, bool nx, bool gbpages, diff --git a/arch/x86/kvm/mmu/paging_tmpl.h b/arch/x86/kvm/mmu/paging_tmpl.h index 0f6455072055..2e3b2aca64ad 100644 --- a/arch/x86/kvm/mmu/paging_tmpl.h +++ b/arch/x86/kvm/mmu/paging_tmpl.h @@ -787,7 +787,7 @@ FNAME(is_self_change_mapping)(struct kvm_vcpu *vcpu, * Returns: 1 if we need to emulate the instruction, 0 otherwise, or * a negative value on error. */ -static int FNAME(page_fault)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) +int FNAME(page_fault)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) { struct guest_walker walker; int r; @@ -897,7 +897,7 @@ static gpa_t FNAME(get_level1_sp_gpa)(struct kvm_mmu_page *sp) return gfn_to_gpa(sp->gfn) + offset * sizeof(pt_element_t); } -static void FNAME(invlpg)(struct kvm_vcpu *vcpu, gva_t gva, hpa_t root_hpa) +void FNAME(invlpg)(struct kvm_vcpu *vcpu, gva_t gva, hpa_t root_hpa) { struct kvm_shadow_walk_iterator iterator; struct kvm_mmu_page *sp; @@ -957,9 +957,8 @@ static void FNAME(invlpg)(struct kvm_vcpu *vcpu, gva_t gva, hpa_t root_hpa) } /* Note, @addr is a GPA when gva_to_gpa() translates an L2 GPA to an L1 GPA. */ -static gpa_t FNAME(gva_to_gpa)(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, - gpa_t addr, u64 access, - struct x86_exception *exception) +gpa_t FNAME(gva_to_gpa)(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, gpa_t addr, + u64 access, struct x86_exception *exception) { struct guest_walker walker; gpa_t gpa = INVALID_GPA; @@ -992,7 +991,7 @@ static gpa_t FNAME(gva_to_gpa)(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, * 0: the sp is synced and no tlb flushing is required * > 0: the sp is synced and tlb flushing is required */ -static int FNAME(sync_page)(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp) +int FNAME(sync_page)(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp) { union kvm_mmu_page_role root_role = vcpu->arch.mmu->root_role; int i; diff --git a/arch/x86/kvm/mmu/shadow_mmu.c b/arch/x86/kvm/mmu/shadow_mmu.c index 05d8f5be559d..86b5fb75d50a 100644 --- a/arch/x86/kvm/mmu/shadow_mmu.c +++ b/arch/x86/kvm/mmu/shadow_mmu.c @@ -10,6 +10,7 @@ * Shadow MMU also supports TDP, it's just less scalable. The Shadow and TDP * MMUs can cooperate to support nested virtualization on hardware with TDP. */ +#include "ioapic.h" #include "mmu.h" #include "mmu_internal.h" #include "mmutrace.h" @@ -2798,6 +2799,35 @@ void shadow_page_table_clear_flood(struct kvm_vcpu *vcpu, gva_t addr) walk_shadow_page_lockless_end(vcpu); } +static bool sync_mmio_spte(struct kvm_vcpu *vcpu, u64 *sptep, gfn_t gfn, + unsigned int access) +{ + if (unlikely(is_mmio_spte(*sptep))) { + if (gfn != get_mmio_spte_gfn(*sptep)) { + mmu_spte_clear_no_track(sptep); + return true; + } + + mark_mmio_spte(vcpu, sptep, gfn, access); + return true; + } + + return false; +} + +#define PTTYPE_EPT 18 /* arbitrary */ +#define PTTYPE PTTYPE_EPT +#include "paging_tmpl.h" +#undef PTTYPE + +#define PTTYPE 64 +#include "paging_tmpl.h" +#undef PTTYPE + +#define PTTYPE 32 +#include "paging_tmpl.h" +#undef PTTYPE + static bool is_obsolete_root(struct kvm *kvm, hpa_t root_hpa) { struct kvm_mmu_page *sp; diff --git a/arch/x86/kvm/mmu/shadow_mmu.h b/arch/x86/kvm/mmu/shadow_mmu.h index 83876047c1f5..00d2f9abecf0 100644 --- a/arch/x86/kvm/mmu/shadow_mmu.h +++ b/arch/x86/kvm/mmu/shadow_mmu.h @@ -73,7 +73,6 @@ bool kvm_test_age_rmap(struct kvm *kvm, struct kvm_rmap_head *rmap_head, int level, pte_t unused); void drop_parent_pte(struct kvm_mmu_page *sp, u64 *parent_pte); -int nonpaging_sync_page(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp); int mmu_sync_children(struct kvm_vcpu *vcpu, struct kvm_mmu_page *parent, bool can_yield); void __clear_sp_write_flooding_count(struct kvm_mmu_page *sp); @@ -150,4 +149,28 @@ void kvm_rmap_zap_collapsible_sptes(struct kvm *kvm, const struct kvm_memory_slot *slot); unsigned long mmu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc); + +/* Exports from paging_tmpl.h */ +gpa_t paging32_gva_to_gpa(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, + gpa_t vaddr, u64 access, + struct x86_exception *exception); +gpa_t paging64_gva_to_gpa(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, + gpa_t vaddr, u64 access, + struct x86_exception *exception); +gpa_t ept_gva_to_gpa(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, gpa_t vaddr, + u64 access, struct x86_exception *exception); + +int paging32_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault); +int paging64_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault); +int ept_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault); + +int paging32_sync_page(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp); +int paging64_sync_page(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp); +int ept_sync_page(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp); +/* Defined in shadow_mmu.c. */ +int nonpaging_sync_page(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp); + +void paging32_invlpg(struct kvm_vcpu *vcpu, gva_t gva, hpa_t root); +void paging64_invlpg(struct kvm_vcpu *vcpu, gva_t gva, hpa_t root); +void ept_invlpg(struct kvm_vcpu *vcpu, gva_t gva, hpa_t root); #endif /* __KVM_X86_MMU_SHADOW_MMU_H */ From patchwork Wed Dec 21 22:24:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ben Gardon X-Patchwork-Id: 13079194 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 97103C4332F for ; Wed, 21 Dec 2022 22:25:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235029AbiLUWZK (ORCPT ); Wed, 21 Dec 2022 17:25:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36282 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234975AbiLUWYi (ORCPT ); Wed, 21 Dec 2022 17:24:38 -0500 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3D99327174 for ; Wed, 21 Dec 2022 14:24:33 -0800 (PST) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-460ab8a327eso2986857b3.23 for ; Wed, 21 Dec 2022 14:24:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=PpjDc5kQAW+k4ZQYHkXtWYHcVTdhQtxJVF/UKPc9J1o=; b=iOK91BqR2al5dx6yxJYNdJ0KjbgTezozD9d7Mfm19BVc0Kn0bIzeV8O1a2Amyvc21A k4zIUHIPW0rEnb7NHeK7QnH/T9jDadvayvwB+uwAIHMBE+jIW0LKKiX43+hFzNddGEDL PsoSH7FSpEWOZODYGU8x75XtalOTNUa5zPk+Dw5g9eAW8EGY1XV1zBxEfiZC/iqpNyik T1BVGUnWV6RWCRPu8ua0k42ZlNt7NDeOb9uT5nVz4LTvl55RESt8bk00TJGWB9wl+4Xk 8Q+0BypI3uvPrwdnt6Xpgenc3i+yQX0gL4bvsoyke4sHKc9qjqTPrazgLRNuAaA9Jrhb se0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=PpjDc5kQAW+k4ZQYHkXtWYHcVTdhQtxJVF/UKPc9J1o=; b=l3MITiakzivud1UilmWiKBsOKE9og6kZQI9Tauri95L+xa1kzmxhvxuEQ3niI+Sb7h +8BCA8nRVVLN/RWNiy0Bwd1BPN7FCWdEAuZSZYHXDWjjwXCE1jLxooC0hgjnp4KPfWTP RCCKSSIp+4RGj4hFH310ftvtiWhaJohZCTWglSxobIQwpChbQgX5ZY0yvW+sX7QHbDlr N+lXayqcDYvi3voY26tVxKjW44yz9bU1VjLD/Du7wyoD+hhRlBVd6uEyehTTZTK6hDGh ZXPzcwc8sQhw7KCQq4n0bs3kCIx3ylDLzErF3BlZVKw7vFoPsJKqVBtgACn0vfjWba5u r8WQ== X-Gm-Message-State: AFqh2koIIPd/2o1UDWQxG/t4fx/MhQHPGjF5R+gX0eumFK4iOJo91ZF3 ZYYEBIWA1kGP1J8of0iLEucdlYPk7jrr X-Google-Smtp-Source: AMrXdXu88AaC3b5v889t8A+RxymtaH7gcPHvKNpTnvt/AXsdpsQsEau7v6rQsJqLyekXwlH5RmEX9VxdYxs2 X-Received: from sweer.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:e45]) (user=bgardon job=sendgmr) by 2002:a05:6902:513:b0:6fa:5a47:c208 with SMTP id x19-20020a056902051300b006fa5a47c208mr376658ybs.472.1671661472442; Wed, 21 Dec 2022 14:24:32 -0800 (PST) Date: Wed, 21 Dec 2022 22:24:10 +0000 In-Reply-To: <20221221222418.3307832-1-bgardon@google.com> Mime-Version: 1.0 References: <20221221222418.3307832-1-bgardon@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20221221222418.3307832-7-bgardon@google.com> Subject: [RFC 06/14] KVM: x86/MMU: Clean up Shadow MMU exports From: Ben Gardon To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: Paolo Bonzini , Peter Xu , Sean Christopherson , David Matlack , Vipin Sharma , Nagareddy Reddy , Ben Gardon Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Now that paging_tmpl.h is included from shadow_mmu.c, there's no need to export many of the functions currrently in shadow_mmu.h, so remove those exports and mark the functions static. This cleans up the interface of the Shadow MMU, and will allow the implementation to keep the details of rmap_heads internal. No functional change intended. Signed-off-by: Ben Gardon --- arch/x86/kvm/mmu/shadow_mmu.c | 78 +++++++++++++++++++++-------------- arch/x86/kvm/mmu/shadow_mmu.h | 51 +---------------------- 2 files changed, 48 insertions(+), 81 deletions(-) diff --git a/arch/x86/kvm/mmu/shadow_mmu.c b/arch/x86/kvm/mmu/shadow_mmu.c index 86b5fb75d50a..090b4788f7de 100644 --- a/arch/x86/kvm/mmu/shadow_mmu.c +++ b/arch/x86/kvm/mmu/shadow_mmu.c @@ -21,6 +21,20 @@ #include #include +struct kvm_shadow_walk_iterator { + u64 addr; + hpa_t shadow_addr; + u64 *sptep; + int level; + unsigned index; +}; + +#define for_each_shadow_entry_using_root(_vcpu, _root, _addr, _walker) \ + for (shadow_walk_init_using_root(&(_walker), (_vcpu), \ + (_root), (_addr)); \ + shadow_walk_okay(&(_walker)); \ + shadow_walk_next(&(_walker))) + #define for_each_shadow_entry(_vcpu, _addr, _walker) \ for (shadow_walk_init(&(_walker), _vcpu, _addr); \ shadow_walk_okay(&(_walker)); \ @@ -227,7 +241,7 @@ static u64 mmu_spte_update_no_track(u64 *sptep, u64 new_spte) * * Returns true if the TLB needs to be flushed */ -bool mmu_spte_update(u64 *sptep, u64 new_spte) +static bool mmu_spte_update(u64 *sptep, u64 new_spte) { bool flush = false; u64 old_spte = mmu_spte_update_no_track(sptep, new_spte); @@ -311,7 +325,7 @@ static u64 mmu_spte_clear_track_bits(struct kvm *kvm, u64 *sptep) * Directly clear spte without caring the state bits of sptep, * it is used to set the upper level spte. */ -void mmu_spte_clear_no_track(u64 *sptep) +static void mmu_spte_clear_no_track(u64 *sptep) { __update_clear_spte_fast(sptep, 0ull); } @@ -354,7 +368,7 @@ static void mmu_free_pte_list_desc(struct pte_list_desc *pte_list_desc) static bool sp_has_gptes(struct kvm_mmu_page *sp); -gfn_t kvm_mmu_page_get_gfn(struct kvm_mmu_page *sp, int index) +static gfn_t kvm_mmu_page_get_gfn(struct kvm_mmu_page *sp, int index) { if (sp->role.passthrough) return sp->gfn; @@ -410,8 +424,8 @@ static void kvm_mmu_page_set_translation(struct kvm_mmu_page *sp, int index, sp->gfn, kvm_mmu_page_get_gfn(sp, index), gfn); } -void kvm_mmu_page_set_access(struct kvm_mmu_page *sp, int index, - unsigned int access) +static void kvm_mmu_page_set_access(struct kvm_mmu_page *sp, int index, + unsigned int access) { gfn_t gfn = kvm_mmu_page_get_gfn(sp, index); @@ -627,7 +641,7 @@ struct kvm_rmap_head *gfn_to_rmap(gfn_t gfn, int level, return &slot->arch.rmap[level - PG_LEVEL_4K][idx]; } -bool rmap_can_add(struct kvm_vcpu *vcpu) +static bool rmap_can_add(struct kvm_vcpu *vcpu) { struct kvm_mmu_memory_cache *mc; @@ -735,7 +749,7 @@ static u64 *rmap_get_next(struct rmap_iterator *iter) for (_spte_ = rmap_get_first(_rmap_head_, _iter_); \ _spte_; _spte_ = rmap_get_next(_iter_)) -void drop_spte(struct kvm *kvm, u64 *sptep) +static void drop_spte(struct kvm *kvm, u64 *sptep) { u64 old_spte = mmu_spte_clear_track_bits(kvm, sptep); @@ -1112,7 +1126,7 @@ static void mmu_page_remove_parent_pte(struct kvm_mmu_page *sp, pte_list_remove(parent_pte, &sp->parent_ptes); } -void drop_parent_pte(struct kvm_mmu_page *sp, u64 *parent_pte) +static void drop_parent_pte(struct kvm_mmu_page *sp, u64 *parent_pte) { mmu_page_remove_parent_pte(sp, parent_pte); mmu_spte_clear_no_track(parent_pte); @@ -1342,8 +1356,8 @@ static void mmu_pages_clear_parents(struct mmu_page_path *parents) } while (!sp->unsync_children); } -int mmu_sync_children(struct kvm_vcpu *vcpu, struct kvm_mmu_page *parent, - bool can_yield) +static int mmu_sync_children(struct kvm_vcpu *vcpu, struct kvm_mmu_page *parent, + bool can_yield) { int i; struct kvm_mmu_page *sp; @@ -1389,7 +1403,7 @@ void __clear_sp_write_flooding_count(struct kvm_mmu_page *sp) atomic_set(&sp->write_flooding_count, 0); } -void clear_sp_write_flooding_count(u64 *spte) +static void clear_sp_write_flooding_count(u64 *spte) { __clear_sp_write_flooding_count(sptep_to_sp(spte)); } @@ -1602,9 +1616,9 @@ static union kvm_mmu_page_role kvm_mmu_child_role(u64 *sptep, bool direct, return role; } -struct kvm_mmu_page *kvm_mmu_get_child_sp(struct kvm_vcpu *vcpu, u64 *sptep, - gfn_t gfn, bool direct, - unsigned int access) +static struct kvm_mmu_page *kvm_mmu_get_child_sp(struct kvm_vcpu *vcpu, + u64 *sptep, gfn_t gfn, + bool direct, unsigned int access) { union kvm_mmu_page_role role; @@ -1615,8 +1629,9 @@ struct kvm_mmu_page *kvm_mmu_get_child_sp(struct kvm_vcpu *vcpu, u64 *sptep, return kvm_mmu_get_shadow_page(vcpu, gfn, role); } -void shadow_walk_init_using_root(struct kvm_shadow_walk_iterator *iterator, - struct kvm_vcpu *vcpu, hpa_t root, u64 addr) +static void shadow_walk_init_using_root(struct kvm_shadow_walk_iterator *iterator, + struct kvm_vcpu *vcpu, hpa_t root, + u64 addr) { iterator->addr = addr; iterator->shadow_addr = root; @@ -1643,14 +1658,14 @@ void shadow_walk_init_using_root(struct kvm_shadow_walk_iterator *iterator, } } -void shadow_walk_init(struct kvm_shadow_walk_iterator *iterator, - struct kvm_vcpu *vcpu, u64 addr) +static void shadow_walk_init(struct kvm_shadow_walk_iterator *iterator, + struct kvm_vcpu *vcpu, u64 addr) { shadow_walk_init_using_root(iterator, vcpu, vcpu->arch.mmu->root.hpa, addr); } -bool shadow_walk_okay(struct kvm_shadow_walk_iterator *iterator) +static bool shadow_walk_okay(struct kvm_shadow_walk_iterator *iterator) { if (iterator->level < PG_LEVEL_4K) return false; @@ -1672,7 +1687,7 @@ static void __shadow_walk_next(struct kvm_shadow_walk_iterator *iterator, --iterator->level; } -void shadow_walk_next(struct kvm_shadow_walk_iterator *iterator) +static void shadow_walk_next(struct kvm_shadow_walk_iterator *iterator) { __shadow_walk_next(iterator, *iterator->sptep); } @@ -1703,13 +1718,14 @@ static void __link_shadow_page(struct kvm *kvm, mark_unsync(sptep); } -void link_shadow_page(struct kvm_vcpu *vcpu, u64 *sptep, struct kvm_mmu_page *sp) +static void link_shadow_page(struct kvm_vcpu *vcpu, u64 *sptep, + struct kvm_mmu_page *sp) { __link_shadow_page(vcpu->kvm, &vcpu->arch.mmu_pte_list_desc_cache, sptep, sp, true); } -void validate_direct_spte(struct kvm_vcpu *vcpu, u64 *sptep, - unsigned direct_access) +static void validate_direct_spte(struct kvm_vcpu *vcpu, u64 *sptep, + unsigned direct_access) { if (is_shadow_present_pte(*sptep) && !is_large_pte(*sptep)) { struct kvm_mmu_page *child; @@ -1731,8 +1747,8 @@ void validate_direct_spte(struct kvm_vcpu *vcpu, u64 *sptep, } /* Returns the number of zapped non-leaf child shadow pages. */ -int mmu_page_zap_pte(struct kvm *kvm, struct kvm_mmu_page *sp, u64 *spte, - struct list_head *invalid_list) +static int mmu_page_zap_pte(struct kvm *kvm, struct kvm_mmu_page *sp, u64 *spte, + struct list_head *invalid_list) { u64 pte; struct kvm_mmu_page *child; @@ -2144,9 +2160,9 @@ int mmu_try_to_unsync_pages(struct kvm *kvm, const struct kvm_memory_slot *slot, return 0; } -int mmu_set_spte(struct kvm_vcpu *vcpu, struct kvm_memory_slot *slot, - u64 *sptep, unsigned int pte_access, gfn_t gfn, - kvm_pfn_t pfn, struct kvm_page_fault *fault) +static int mmu_set_spte(struct kvm_vcpu *vcpu, struct kvm_memory_slot *slot, + u64 *sptep, unsigned int pte_access, gfn_t gfn, + kvm_pfn_t pfn, struct kvm_page_fault *fault) { struct kvm_mmu_page *sp = sptep_to_sp(sptep); int level = sp->role.level; @@ -2251,8 +2267,8 @@ static int direct_pte_prefetch_many(struct kvm_vcpu *vcpu, return 0; } -void __direct_pte_prefetch(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, - u64 *sptep) +static void __direct_pte_prefetch(struct kvm_vcpu *vcpu, + struct kvm_mmu_page *sp, u64 *sptep) { u64 *spte, *start = NULL; int i; @@ -2788,7 +2804,7 @@ int get_walk(struct kvm_vcpu *vcpu, u64 addr, u64 *sptes, int *root_level) return leaf; } -void shadow_page_table_clear_flood(struct kvm_vcpu *vcpu, gva_t addr) +static void shadow_page_table_clear_flood(struct kvm_vcpu *vcpu, gva_t addr) { struct kvm_shadow_walk_iterator iterator; u64 spte; diff --git a/arch/x86/kvm/mmu/shadow_mmu.h b/arch/x86/kvm/mmu/shadow_mmu.h index 00d2f9abecf0..20c65a0ea52c 100644 --- a/arch/x86/kvm/mmu/shadow_mmu.h +++ b/arch/x86/kvm/mmu/shadow_mmu.h @@ -23,32 +23,11 @@ struct pte_list_desc { u64 *sptes[PTE_LIST_EXT]; }; +/* Only exported for debugfs.c. */ unsigned int pte_list_count(struct kvm_rmap_head *rmap_head); -struct kvm_shadow_walk_iterator { - u64 addr; - hpa_t shadow_addr; - u64 *sptep; - int level; - unsigned index; -}; - -#define for_each_shadow_entry_using_root(_vcpu, _root, _addr, _walker) \ - for (shadow_walk_init_using_root(&(_walker), (_vcpu), \ - (_root), (_addr)); \ - shadow_walk_okay(&(_walker)); \ - shadow_walk_next(&(_walker))) - -bool mmu_spte_update(u64 *sptep, u64 new_spte); -void mmu_spte_clear_no_track(u64 *sptep); -gfn_t kvm_mmu_page_get_gfn(struct kvm_mmu_page *sp, int index); -void kvm_mmu_page_set_access(struct kvm_mmu_page *sp, int index, - unsigned int access); - struct kvm_rmap_head *gfn_to_rmap(gfn_t gfn, int level, const struct kvm_memory_slot *slot); -bool rmap_can_add(struct kvm_vcpu *vcpu); -void drop_spte(struct kvm *kvm, u64 *sptep); bool rmap_write_protect(struct kvm_rmap_head *rmap_head, bool pt_protect); bool __rmap_clear_dirty(struct kvm *kvm, struct kvm_rmap_head *rmap_head, const struct kvm_memory_slot *slot); @@ -72,30 +51,8 @@ bool kvm_test_age_rmap(struct kvm *kvm, struct kvm_rmap_head *rmap_head, struct kvm_memory_slot *slot, gfn_t gfn, int level, pte_t unused); -void drop_parent_pte(struct kvm_mmu_page *sp, u64 *parent_pte); -int mmu_sync_children(struct kvm_vcpu *vcpu, struct kvm_mmu_page *parent, - bool can_yield); void __clear_sp_write_flooding_count(struct kvm_mmu_page *sp); -void clear_sp_write_flooding_count(u64 *spte); - -struct kvm_mmu_page *kvm_mmu_get_child_sp(struct kvm_vcpu *vcpu, u64 *sptep, - gfn_t gfn, bool direct, - unsigned int access); - -void shadow_walk_init_using_root(struct kvm_shadow_walk_iterator *iterator, - struct kvm_vcpu *vcpu, hpa_t root, u64 addr); -void shadow_walk_init(struct kvm_shadow_walk_iterator *iterator, - struct kvm_vcpu *vcpu, u64 addr); -bool shadow_walk_okay(struct kvm_shadow_walk_iterator *iterator); -void shadow_walk_next(struct kvm_shadow_walk_iterator *iterator); - -void link_shadow_page(struct kvm_vcpu *vcpu, u64 *sptep, struct kvm_mmu_page *sp); - -void validate_direct_spte(struct kvm_vcpu *vcpu, u64 *sptep, - unsigned direct_access); -int mmu_page_zap_pte(struct kvm *kvm, struct kvm_mmu_page *sp, u64 *spte, - struct list_head *invalid_list); bool __kvm_mmu_prepare_zap_page(struct kvm *kvm, struct kvm_mmu_page *sp, struct list_head *invalid_list, int *nr_zapped); @@ -107,11 +64,6 @@ int make_mmu_pages_available(struct kvm_vcpu *vcpu); int kvm_mmu_unprotect_page_virt(struct kvm_vcpu *vcpu, gva_t gva); -int mmu_set_spte(struct kvm_vcpu *vcpu, struct kvm_memory_slot *slot, - u64 *sptep, unsigned int pte_access, gfn_t gfn, - kvm_pfn_t pfn, struct kvm_page_fault *fault); -void __direct_pte_prefetch(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, - u64 *sptep); int __direct_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault); u64 *fast_pf_get_last_sptep(struct kvm_vcpu *vcpu, gpa_t gpa, u64 *spte); @@ -121,7 +73,6 @@ int mmu_alloc_special_roots(struct kvm_vcpu *vcpu); int get_walk(struct kvm_vcpu *vcpu, u64 addr, u64 *sptes, int *root_level); -void shadow_page_table_clear_flood(struct kvm_vcpu *vcpu, gva_t addr); void kvm_mmu_pte_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new, int bytes, struct kvm_page_track_notifier_node *node); From patchwork Wed Dec 21 22:24:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ben Gardon X-Patchwork-Id: 13079195 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B8CBCC4332F for ; Wed, 21 Dec 2022 22:25:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235044AbiLUWZM (ORCPT ); Wed, 21 Dec 2022 17:25:12 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36608 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234982AbiLUWYm (ORCPT ); Wed, 21 Dec 2022 17:24:42 -0500 Received: from mail-pg1-x54a.google.com (mail-pg1-x54a.google.com [IPv6:2607:f8b0:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A5508275D6 for ; Wed, 21 Dec 2022 14:24:34 -0800 (PST) Received: by mail-pg1-x54a.google.com with SMTP id f132-20020a636a8a000000b00473d0b600ebso99341pgc.14 for ; Wed, 21 Dec 2022 14:24:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=6Y1uFW2wjlbXMFly7tpXboV7VZ3E/dcx8Q2Gfsts8Tk=; b=j6BY0U68byYLgySr44DngXyEm134/1hFKLFShEVTVJ/IuhmuHUhPj2IFs5+alWpjes FnmgU/a+mWQ9lDW3CaNzgmNQp5HsZoHpHcyV/b1il72r6Xzaj9qXk5NXsYY0j+JXwNjk FZejHNVrbE5LJsqGqrP4ZPA/zp5q0qfmZqF/ezW2x9qxFl5GNHdMhzaOZVSwWHoYKYqH 08xYqohGPLH5zNwDVBBmvSSBKQHJMlkqT5i+Wtxx9V3Dvv+kdODLKmik7SKFpsmBhFQW cfjubSaMmD3KdtMWQs+eq26B7yPez7mGSsQDiVaiRL+/oXIuPIBq7yXEg4mMa1UUUuq9 /2Vg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6Y1uFW2wjlbXMFly7tpXboV7VZ3E/dcx8Q2Gfsts8Tk=; b=BQudgkn5h3umfvef9xFlTotCqNiWV4VvWBSebwd1U5BIhpKA8jLpmhEtKwMFvGHQSS HrejaNe78gllFg4BWxG40S83C5VgG+qcM0AgoL+DZ0tkNPhHT0zbSqZYLfrC6qBKJ1eZ ONmdatrUlA7aBBrUxAbEoyWE/ysCteBsqq9MdXRd+gtMnwOrWNTDoK1bFn4jWGYDLcGC t/gb2O4KwWYR7X04NlNEBrDxWy3S872g9tM+tQlFwsVFa7b3xlx9a0rt7JL2Tp1ThxRb y+oKL9l+eW38k+OFWOu4eqJ7oz8MzAFqra5/wKfSfUrG3NqA6wnBO/8rIZeACv7Ht3os HMxA== X-Gm-Message-State: AFqh2kplDVod0zdOCPBUWSKSk/TJDchXrKJp0FgFbY+Zkrr8s12JyX1W J6COLdVR6twykSqMslJDHsyABJSvlBfz X-Google-Smtp-Source: AMrXdXvazExc/WwGrAE0aK1dWeGFS+8c0Y9JCSXflaOtoGh5jJfoK8tAFLTB0ITsB+joVbHqUcbCzmpTgbnp X-Received: from sweer.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:e45]) (user=bgardon job=sendgmr) by 2002:a17:90a:8183:b0:219:9874:c7d3 with SMTP id e3-20020a17090a818300b002199874c7d3mr320888pjn.221.1671661474119; Wed, 21 Dec 2022 14:24:34 -0800 (PST) Date: Wed, 21 Dec 2022 22:24:11 +0000 In-Reply-To: <20221221222418.3307832-1-bgardon@google.com> Mime-Version: 1.0 References: <20221221222418.3307832-1-bgardon@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20221221222418.3307832-8-bgardon@google.com> Subject: [RFC 07/14] KVM: x86/MMU: Cleanup shrinker interface with Shadow MMU From: Ben Gardon To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: Paolo Bonzini , Peter Xu , Sean Christopherson , David Matlack , Vipin Sharma , Nagareddy Reddy , Ben Gardon Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The MMU shrinker currently only operates on the Shadow MMU, but having the entire implemenatation in shadow_mmu.c is awkward since much of the function isn't Shadow MMU specific. There has also been talk of changing the target of the shrinker to the MMU caches rather than already allocated page tables. As a result, it makes sense to move some of the implementation back to mmu.c. No functional change intended. Signed-off-by: Ben Gardon --- arch/x86/kvm/mmu/mmu.c | 43 ++++++++++++++++++++++++ arch/x86/kvm/mmu/shadow_mmu.c | 62 ++++++++--------------------------- arch/x86/kvm/mmu/shadow_mmu.h | 3 +- 3 files changed, 58 insertions(+), 50 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index dd97e346c786..4c45a5b63356 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -3147,6 +3147,49 @@ static unsigned long mmu_shrink_count(struct shrinker *shrink, return percpu_counter_read_positive(&kvm_total_used_mmu_pages); } +unsigned long mmu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) +{ + struct kvm *kvm; + int nr_to_scan = sc->nr_to_scan; + unsigned long freed = 0; + + mutex_lock(&kvm_lock); + + list_for_each_entry(kvm, &vm_list, vm_list) { + /* + * Never scan more than sc->nr_to_scan VM instances. + * Will not hit this condition practically since we do not try + * to shrink more than one VM and it is very unlikely to see + * !n_used_mmu_pages so many times. + */ + if (!nr_to_scan--) + break; + + /* + * n_used_mmu_pages is accessed without holding kvm->mmu_lock + * here. We may skip a VM instance errorneosly, but we do not + * want to shrink a VM that only started to populate its MMU + * anyway. + */ + if (!kvm->arch.n_used_mmu_pages && + !kvm_shadow_mmu_has_zapped_obsolete_pages(kvm)) + continue; + + freed = kvm_shadow_mmu_shrink_scan(kvm, sc->nr_to_scan); + + /* + * unfair on small ones + * per-vm shrinkers cry out + * sadness comes quickly + */ + list_move_tail(&kvm->vm_list, &vm_list); + break; + } + + mutex_unlock(&kvm_lock); + return freed; +} + static struct shrinker mmu_shrinker = { .count_objects = mmu_shrink_count, .scan_objects = mmu_shrink_scan, diff --git a/arch/x86/kvm/mmu/shadow_mmu.c b/arch/x86/kvm/mmu/shadow_mmu.c index 090b4788f7de..1259c4a3b140 100644 --- a/arch/x86/kvm/mmu/shadow_mmu.c +++ b/arch/x86/kvm/mmu/shadow_mmu.c @@ -3147,7 +3147,7 @@ void kvm_zap_obsolete_pages(struct kvm *kvm) kvm_mmu_commit_zap_page(kvm, &kvm->arch.zapped_obsolete_pages); } -static bool kvm_has_zapped_obsolete_pages(struct kvm *kvm) +bool kvm_shadow_mmu_has_zapped_obsolete_pages(struct kvm *kvm) { return unlikely(!list_empty_careful(&kvm->arch.zapped_obsolete_pages)); } @@ -3416,60 +3416,24 @@ void kvm_rmap_zap_collapsible_sptes(struct kvm *kvm, kvm_arch_flush_remote_tlbs_memslot(kvm, slot); } -unsigned long mmu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) +unsigned long kvm_shadow_mmu_shrink_scan(struct kvm *kvm, int pages_to_free) { - struct kvm *kvm; - int nr_to_scan = sc->nr_to_scan; unsigned long freed = 0; + int idx; - mutex_lock(&kvm_lock); - - list_for_each_entry(kvm, &vm_list, vm_list) { - int idx; - LIST_HEAD(invalid_list); - - /* - * Never scan more than sc->nr_to_scan VM instances. - * Will not hit this condition practically since we do not try - * to shrink more than one VM and it is very unlikely to see - * !n_used_mmu_pages so many times. - */ - if (!nr_to_scan--) - break; - /* - * n_used_mmu_pages is accessed without holding kvm->mmu_lock - * here. We may skip a VM instance errorneosly, but we do not - * want to shrink a VM that only started to populate its MMU - * anyway. - */ - if (!kvm->arch.n_used_mmu_pages && - !kvm_has_zapped_obsolete_pages(kvm)) - continue; - - idx = srcu_read_lock(&kvm->srcu); - write_lock(&kvm->mmu_lock); - - if (kvm_has_zapped_obsolete_pages(kvm)) { - kvm_mmu_commit_zap_page(kvm, - &kvm->arch.zapped_obsolete_pages); - goto unlock; - } + idx = srcu_read_lock(&kvm->srcu); + write_lock(&kvm->mmu_lock); - freed = kvm_mmu_zap_oldest_mmu_pages(kvm, sc->nr_to_scan); + if (kvm_shadow_mmu_has_zapped_obsolete_pages(kvm)) { + kvm_mmu_commit_zap_page(kvm, &kvm->arch.zapped_obsolete_pages); + goto out; + } -unlock: - write_unlock(&kvm->mmu_lock); - srcu_read_unlock(&kvm->srcu, idx); + freed = kvm_mmu_zap_oldest_mmu_pages(kvm, pages_to_free); - /* - * unfair on small ones - * per-vm shrinkers cry out - * sadness comes quickly - */ - list_move_tail(&kvm->vm_list, &vm_list); - break; - } +out: + write_unlock(&kvm->mmu_lock); + srcu_read_unlock(&kvm->srcu, idx); - mutex_unlock(&kvm_lock); return freed; } diff --git a/arch/x86/kvm/mmu/shadow_mmu.h b/arch/x86/kvm/mmu/shadow_mmu.h index 20c65a0ea52c..9952aa1e86cf 100644 --- a/arch/x86/kvm/mmu/shadow_mmu.h +++ b/arch/x86/kvm/mmu/shadow_mmu.h @@ -99,7 +99,8 @@ void kvm_shadow_mmu_try_split_huge_pages(struct kvm *kvm, void kvm_rmap_zap_collapsible_sptes(struct kvm *kvm, const struct kvm_memory_slot *slot); -unsigned long mmu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc); +bool kvm_shadow_mmu_has_zapped_obsolete_pages(struct kvm *kvm); +unsigned long kvm_shadow_mmu_shrink_scan(struct kvm *kvm, int pages_to_free); /* Exports from paging_tmpl.h */ gpa_t paging32_gva_to_gpa(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, From patchwork Wed Dec 21 22:24:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ben Gardon X-Patchwork-Id: 13079196 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A8074C10F1B for ; Wed, 21 Dec 2022 22:25:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235054AbiLUWZQ (ORCPT ); Wed, 21 Dec 2022 17:25:16 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37072 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235002AbiLUWZC (ORCPT ); Wed, 21 Dec 2022 17:25:02 -0500 Received: from mail-pl1-x649.google.com (mail-pl1-x649.google.com [IPv6:2607:f8b0:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1F953275E6 for ; Wed, 21 Dec 2022 14:24:36 -0800 (PST) Received: by mail-pl1-x649.google.com with SMTP id o5-20020a170903210500b0018f9edbba6eso134842ple.11 for ; Wed, 21 Dec 2022 14:24:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=BM/X6lf5Q5u/NqA/5lapzPbr/TRyR56BZ+pcmqdxJ08=; b=qb64BZX2skXQRlPzR/gphfOKSPRkDwvBMkn7oaOZpTZ2gp5EHreY2v4lXCy1a9ufgK xN2ODq1pNGyJ8WN+gbl5Ci9n4tmFG1VHrj2V2fhMwYoUFMMUHFOvA6vV2L6tsnpg8ygl tXUsBzQ9SDD3ITbzPXAV49ShZhPEEvUx778mW6XLnI+FkqcU0BR+o0nw9YXw9yC0wa9h UlbCyVeUoOUdGp0IYDzYl8rnePgr5SU26yYQ1K8wRgygvowtWShEeKO5b1Xbv74eRFD4 DZsd2gZiGHEBZb6KIU0q90BsAN3EJ8DQb4+qAU+ctS2eav0Bkp6We3Q3zzxklyeENq31 Hjxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=BM/X6lf5Q5u/NqA/5lapzPbr/TRyR56BZ+pcmqdxJ08=; b=wD7duxyY6jh3xS5rzGa+Xe76hihrjW5AY+fAnyjM3JiZtkFJWndsZflsSjQ0Xq4kWC wtefMHGQGa4g9xs3Nxi5ciCwnf0QURA3PgrZ5tnOH3MWuWLWlOPWEcFQKu6HFEjjEFW6 0PtFrhHuMscpNCKqvMZUrXV4y0MRUAP2qSdci6njL+DrFLG015HNqmx5zJthR4f5l4u1 Hdx4oqylth4tN0L+FdY8gRM8vEAmuqFoJs5AcZIULPiOW7yBLo0PWZzHh4/YAt+RkPrM w28NK8SC8kFj3nM0EG2kWvii45c7RXEnkXgo3vYOrmTdVGY4lNtYB+8CXq4maehuEw5u HPrQ== X-Gm-Message-State: AFqh2kpdvEpjPIr0Fm6+LwCDkzOLXIyTygeDlwErM5OLzjkJJxRQ2xku x/qjdCcRHi47nrzeFoDTiSNJnsBMCtwG X-Google-Smtp-Source: AMrXdXuIYxb8wwaa0x4dHvr8LQ0LAxW67fag3ohNfE+PlNGCJ7DafcYsm6xdERiSUGJuPa4Bc9sSe1NrmAmI X-Received: from sweer.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:e45]) (user=bgardon job=sendgmr) by 2002:a05:6a00:1d8f:b0:578:7184:b495 with SMTP id z15-20020a056a001d8f00b005787184b495mr270797pfw.16.1671661475626; Wed, 21 Dec 2022 14:24:35 -0800 (PST) Date: Wed, 21 Dec 2022 22:24:12 +0000 In-Reply-To: <20221221222418.3307832-1-bgardon@google.com> Mime-Version: 1.0 References: <20221221222418.3307832-1-bgardon@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20221221222418.3307832-9-bgardon@google.com> Subject: [RFC 08/14] KVM: x86/MMU: Clean up naming of exported Shadow MMU functions From: Ben Gardon To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: Paolo Bonzini , Peter Xu , Sean Christopherson , David Matlack , Vipin Sharma , Nagareddy Reddy , Ben Gardon Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Change the naming scheme on several functions exported from the shadow MMU to match the naming scheme used by the TDP MMU: kvm_shadow_mmu_. More cleanups will follow to convert the remaining functions to a similar naming scheme, but for now, start with the trivial renames. No functional change intended. Signed-off-by: Ben Gardon --- arch/x86/kvm/mmu/mmu.c | 19 ++++++++++--------- arch/x86/kvm/mmu/paging_tmpl.h | 2 +- arch/x86/kvm/mmu/shadow_mmu.c | 19 ++++++++++--------- arch/x86/kvm/mmu/shadow_mmu.h | 17 +++++++++-------- 4 files changed, 30 insertions(+), 27 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 4c45a5b63356..bacb519ba7b4 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1129,7 +1129,7 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *vcpu) int r; write_lock(&vcpu->kvm->mmu_lock); - r = make_mmu_pages_available(vcpu); + r = kvm_shadow_mmu_make_pages_available(vcpu); if (r < 0) goto out_unlock; @@ -1204,7 +1204,7 @@ static bool get_mmio_spte(struct kvm_vcpu *vcpu, u64 addr, u64 *sptep) if (is_tdp_mmu(vcpu->arch.mmu)) leaf = kvm_tdp_mmu_get_walk(vcpu, addr, sptes, &root); else - leaf = get_walk(vcpu, addr, sptes, &root); + leaf = kvm_shadow_mmu_get_walk(vcpu, addr, sptes, &root); walk_shadow_page_lockless_end(vcpu); @@ -1469,14 +1469,14 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault if (is_page_fault_stale(vcpu, fault, mmu_seq)) goto out_unlock; - r = make_mmu_pages_available(vcpu); + r = kvm_shadow_mmu_make_pages_available(vcpu); if (r) goto out_unlock; if (is_tdp_mmu_fault) r = kvm_tdp_mmu_map(vcpu, fault); else - r = __direct_map(vcpu, fault); + r = kvm_shadow_mmu_direct_map(vcpu, fault); out_unlock: if (is_tdp_mmu_fault) @@ -1514,7 +1514,7 @@ int kvm_handle_page_fault(struct kvm_vcpu *vcpu, u64 error_code, trace_kvm_page_fault(vcpu, fault_address, error_code); if (kvm_event_needs_reinjection(vcpu)) - kvm_mmu_unprotect_page_virt(vcpu, fault_address); + kvm_shadow_mmu_unprotect_page_virt(vcpu, fault_address); r = kvm_mmu_page_fault(vcpu, fault_address, error_code, insn, insn_len); } else if (flags & KVM_PV_REASON_PAGE_NOT_PRESENT) { @@ -2791,7 +2791,8 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm) * In order to ensure all vCPUs drop their soon-to-be invalid roots, * invalidating TDP MMU roots must be done while holding mmu_lock for * write and in the same critical section as making the reload request, - * e.g. before kvm_zap_obsolete_pages() could drop mmu_lock and yield. + * e.g. before kvm_shadow_mmu_zap_obsolete_pages() could drop mmu_lock + * and yield. */ if (is_tdp_mmu_enabled(kvm)) kvm_tdp_mmu_invalidate_all_roots(kvm); @@ -2806,7 +2807,7 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm) */ kvm_make_all_cpus_request(kvm, KVM_REQ_MMU_FREE_OBSOLETE_ROOTS); - kvm_zap_obsolete_pages(kvm); + kvm_shadow_mmu_zap_obsolete_pages(kvm); write_unlock(&kvm->mmu_lock); @@ -2892,7 +2893,7 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end) kvm_mmu_invalidate_begin(kvm, gfn_start, gfn_end); - flush = kvm_rmap_zap_gfn_range(kvm, gfn_start, gfn_end); + flush = kvm_shadow_mmu_zap_gfn_range(kvm, gfn_start, gfn_end); if (is_tdp_mmu_enabled(kvm)) { for (i = 0; i < KVM_ADDRESS_SPACE_NUM; i++) @@ -3036,7 +3037,7 @@ void kvm_mmu_zap_collapsible_sptes(struct kvm *kvm, { if (kvm_memslots_have_rmaps(kvm)) { write_lock(&kvm->mmu_lock); - kvm_rmap_zap_collapsible_sptes(kvm, slot); + kvm_shadow_mmu_zap_collapsible_sptes(kvm, slot); write_unlock(&kvm->mmu_lock); } diff --git a/arch/x86/kvm/mmu/paging_tmpl.h b/arch/x86/kvm/mmu/paging_tmpl.h index 2e3b2aca64ad..85c0b186cb0a 100644 --- a/arch/x86/kvm/mmu/paging_tmpl.h +++ b/arch/x86/kvm/mmu/paging_tmpl.h @@ -874,7 +874,7 @@ int FNAME(page_fault)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) if (is_page_fault_stale(vcpu, fault, mmu_seq)) goto out_unlock; - r = make_mmu_pages_available(vcpu); + r = kvm_shadow_mmu_make_pages_available(vcpu); if (r) goto out_unlock; r = FNAME(fetch)(vcpu, fault, &walker); diff --git a/arch/x86/kvm/mmu/shadow_mmu.c b/arch/x86/kvm/mmu/shadow_mmu.c index 1259c4a3b140..e36b4d9c67f2 100644 --- a/arch/x86/kvm/mmu/shadow_mmu.c +++ b/arch/x86/kvm/mmu/shadow_mmu.c @@ -1965,7 +1965,7 @@ static inline unsigned long kvm_mmu_available_pages(struct kvm *kvm) return 0; } -int make_mmu_pages_available(struct kvm_vcpu *vcpu) +int kvm_shadow_mmu_make_pages_available(struct kvm_vcpu *vcpu) { unsigned long avail = kvm_mmu_available_pages(vcpu->kvm); @@ -2029,7 +2029,7 @@ int kvm_mmu_unprotect_page(struct kvm *kvm, gfn_t gfn) return r; } -int kvm_mmu_unprotect_page_virt(struct kvm_vcpu *vcpu, gva_t gva) +int kvm_shadow_mmu_unprotect_page_virt(struct kvm_vcpu *vcpu, gva_t gva) { gpa_t gpa; int r; @@ -2319,7 +2319,7 @@ static void direct_pte_prefetch(struct kvm_vcpu *vcpu, u64 *sptep) __direct_pte_prefetch(vcpu, sp, sptep); } -int __direct_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) +int kvm_shadow_mmu_direct_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) { struct kvm_shadow_walk_iterator it; struct kvm_mmu_page *sp; @@ -2537,7 +2537,7 @@ int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) return r; write_lock(&vcpu->kvm->mmu_lock); - r = make_mmu_pages_available(vcpu); + r = kvm_shadow_mmu_make_pages_available(vcpu); if (r < 0) goto out_unlock; @@ -2785,7 +2785,8 @@ void kvm_mmu_sync_prev_roots(struct kvm_vcpu *vcpu) * * Must be called between walk_shadow_page_lockless_{begin,end}. */ -int get_walk(struct kvm_vcpu *vcpu, u64 addr, u64 *sptes, int *root_level) +int kvm_shadow_mmu_get_walk(struct kvm_vcpu *vcpu, u64 addr, u64 *sptes, + int *root_level) { struct kvm_shadow_walk_iterator iterator; int leaf = -1; @@ -3091,7 +3092,7 @@ __always_inline bool slot_handle_level_4k(struct kvm *kvm, } #define BATCH_ZAP_PAGES 10 -void kvm_zap_obsolete_pages(struct kvm *kvm) +void kvm_shadow_mmu_zap_obsolete_pages(struct kvm *kvm) { struct kvm_mmu_page *sp, *node; int nr_zapped, batch = 0; @@ -3152,7 +3153,7 @@ bool kvm_shadow_mmu_has_zapped_obsolete_pages(struct kvm *kvm) return unlikely(!list_empty_careful(&kvm->arch.zapped_obsolete_pages)); } -bool kvm_rmap_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end) +bool kvm_shadow_mmu_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end) { const struct kvm_memory_slot *memslot; struct kvm_memslots *slots; @@ -3404,8 +3405,8 @@ static bool kvm_mmu_zap_collapsible_spte(struct kvm *kvm, return need_tlb_flush; } -void kvm_rmap_zap_collapsible_sptes(struct kvm *kvm, - const struct kvm_memory_slot *slot) +void kvm_shadow_mmu_zap_collapsible_sptes(struct kvm *kvm, + const struct kvm_memory_slot *slot) { /* * Note, use KVM_MAX_HUGEPAGE_LEVEL - 1 since there's no need to zap diff --git a/arch/x86/kvm/mmu/shadow_mmu.h b/arch/x86/kvm/mmu/shadow_mmu.h index 9952aa1e86cf..148cc3593d2b 100644 --- a/arch/x86/kvm/mmu/shadow_mmu.h +++ b/arch/x86/kvm/mmu/shadow_mmu.h @@ -60,18 +60,19 @@ bool kvm_mmu_prepare_zap_page(struct kvm *kvm, struct kvm_mmu_page *sp, struct list_head *invalid_list); void kvm_mmu_commit_zap_page(struct kvm *kvm, struct list_head *invalid_list); -int make_mmu_pages_available(struct kvm_vcpu *vcpu); +int kvm_shadow_mmu_make_pages_available(struct kvm_vcpu *vcpu); -int kvm_mmu_unprotect_page_virt(struct kvm_vcpu *vcpu, gva_t gva); +int kvm_shadow_mmu_unprotect_page_virt(struct kvm_vcpu *vcpu, gva_t gva); -int __direct_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault); +int kvm_shadow_mmu_direct_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault); u64 *fast_pf_get_last_sptep(struct kvm_vcpu *vcpu, gpa_t gpa, u64 *spte); hpa_t mmu_alloc_root(struct kvm_vcpu *vcpu, gfn_t gfn, int quadrant, u8 level); int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu); int mmu_alloc_special_roots(struct kvm_vcpu *vcpu); -int get_walk(struct kvm_vcpu *vcpu, u64 addr, u64 *sptes, int *root_level); +int kvm_shadow_mmu_get_walk(struct kvm_vcpu *vcpu, u64 addr, u64 *sptes, + int *root_level); void kvm_mmu_pte_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new, int bytes, struct kvm_page_track_notifier_node *node); @@ -86,8 +87,8 @@ bool slot_handle_level(struct kvm *kvm, const struct kvm_memory_slot *memslot, bool slot_handle_level_4k(struct kvm *kvm, const struct kvm_memory_slot *memslot, slot_level_handler fn, bool flush_on_yield); -void kvm_zap_obsolete_pages(struct kvm *kvm); -bool kvm_rmap_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end); +void kvm_shadow_mmu_zap_obsolete_pages(struct kvm *kvm); +bool kvm_shadow_mmu_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end); bool slot_rmap_write_protect(struct kvm *kvm, struct kvm_rmap_head *rmap_head, const struct kvm_memory_slot *slot); @@ -96,8 +97,8 @@ void kvm_shadow_mmu_try_split_huge_pages(struct kvm *kvm, const struct kvm_memory_slot *slot, gfn_t start, gfn_t end, int target_level); -void kvm_rmap_zap_collapsible_sptes(struct kvm *kvm, - const struct kvm_memory_slot *slot); +void kvm_shadow_mmu_zap_collapsible_sptes(struct kvm *kvm, + const struct kvm_memory_slot *slot); bool kvm_shadow_mmu_has_zapped_obsolete_pages(struct kvm *kvm); unsigned long kvm_shadow_mmu_shrink_scan(struct kvm *kvm, int pages_to_free); From patchwork Wed Dec 21 22:24:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ben Gardon X-Patchwork-Id: 13079197 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D6949C4167B for ; Wed, 21 Dec 2022 22:25:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234970AbiLUWZS (ORCPT ); Wed, 21 Dec 2022 17:25:18 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37082 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235004AbiLUWZC (ORCPT ); Wed, 21 Dec 2022 17:25:02 -0500 Received: from mail-pl1-x64a.google.com (mail-pl1-x64a.google.com [IPv6:2607:f8b0:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7E23A27178 for ; Wed, 21 Dec 2022 14:24:37 -0800 (PST) Received: by mail-pl1-x64a.google.com with SMTP id l7-20020a170902f68700b00192495b1f10so153401plg.2 for ; Wed, 21 Dec 2022 14:24:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=BoetL47dMtSYpyhZbJzKl1EqSh6TREkVR0ldKUR5DTE=; b=B2Il06BXZ5r75uO6Eh6VbInYjOxKdKJBrLZcwxld+Hj9NTKOaFJ4tEln9Ebd43HD3V XGsxjryczXGgpPcb9lEHqJHq6iSZOJDDZYCR9sJdtAcozqn8ZhJQBFnBtQAGVb8fs88i tS4KjdO/CHK6ygrF/kxg+BGf1GBeSdrqZz1Bs1DCwuZ4ALwelH4arRjImcrNksCQH0PJ fre2RUTACKiTUUq7uDfxqdnR50CEEj1LXW0xeHx730J4o2IACZ+mv8pTROJWGX0w8D/f digzIkmHy7fIFib/DTm+CSnIbYO09B6k8GNnVK//AhuZ3XugZ2+LvEEfUXboUxjCh8cR TbTg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=BoetL47dMtSYpyhZbJzKl1EqSh6TREkVR0ldKUR5DTE=; b=XTtoeWDBKVqWl2OFHZI6VySvNRRU2Fh4M0mYDQYXjEElmOvn3hickuNQH0dxSIwgfd QgMK1JIYiZ78kKCYNCNiirpqHAhXNH50iBsdQ20AVqwGZ3GAoaTWEXDJcvVMbuJolROg htpok9niYcHc5oSuKIhMd4Q1e4H9pGTMkTSEbgnebg4zE4fiL+M7JDIJiISzhCZwUi2G RSjACZ6PSyIvuQ2uoVRu8EFnm8i1WVOmv8u4r9XrdFLJUWEyoG8Jn9Rrc6S27Sk/EtGh ibkpfFVL2S8VJQJDm59nyepTHIp0HmlbXYXbaSEP4WvaStzdtcFq+Ct9M09++6/rV21q ZGIg== X-Gm-Message-State: AFqh2krV8kn1E1ZBhLNvgDJ1lgpRx9TwsUlN4y6qIvfoUOn4P/yH4277 OEFTkTSdRuJmHu6npA7T7DslMUmk0fxB X-Google-Smtp-Source: AMrXdXvcPw/batAzM2ubwgjdBVty+O9txrpne8GQWWnHTUKRra0t6kSK2wryQb6O1CIF9FnsBgLoCAGFiI5U X-Received: from sweer.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:e45]) (user=bgardon job=sendgmr) by 2002:a63:9359:0:b0:478:bc15:bf28 with SMTP id w25-20020a639359000000b00478bc15bf28mr148390pgm.542.1671661477243; Wed, 21 Dec 2022 14:24:37 -0800 (PST) Date: Wed, 21 Dec 2022 22:24:13 +0000 In-Reply-To: <20221221222418.3307832-1-bgardon@google.com> Mime-Version: 1.0 References: <20221221222418.3307832-1-bgardon@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20221221222418.3307832-10-bgardon@google.com> Subject: [RFC 09/14] KVM: x86/MMU: Only make pages available on Shadow MMU fault From: Ben Gardon To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: Paolo Bonzini , Peter Xu , Sean Christopherson , David Matlack , Vipin Sharma , Nagareddy Reddy , Ben Gardon Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Now that the Shadow MMU has been factored out of mmu.c and the naming sheme has been cleaned up, it's clear that there's an unnecessary operation in direct_page_fault(). Since the MMU page quota is only applied to the Shadow MMU, there's no point to calling kvm_shadow_mmu_make_pages_available on a fault where the TDP MMU is going to handle installing new TDP PTEs. Signed-off-by: Ben Gardon --- arch/x86/kvm/mmu/mmu.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index bacb519ba7b4..568b36de9eeb 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1469,14 +1469,14 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault if (is_page_fault_stale(vcpu, fault, mmu_seq)) goto out_unlock; - r = kvm_shadow_mmu_make_pages_available(vcpu); - if (r) - goto out_unlock; - if (is_tdp_mmu_fault) r = kvm_tdp_mmu_map(vcpu, fault); - else + else { + r = kvm_shadow_mmu_make_pages_available(vcpu); + if (r) + goto out_unlock; r = kvm_shadow_mmu_direct_map(vcpu, fault); + } out_unlock: if (is_tdp_mmu_fault) From patchwork Wed Dec 21 22:24:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ben Gardon X-Patchwork-Id: 13079198 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C222C4332F for ; Wed, 21 Dec 2022 22:25:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235041AbiLUWZn (ORCPT ); Wed, 21 Dec 2022 17:25:43 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37256 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234988AbiLUWZL (ORCPT ); Wed, 21 Dec 2022 17:25:11 -0500 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B89B5275E0 for ; Wed, 21 Dec 2022 14:24:39 -0800 (PST) Received: by mail-pj1-x1049.google.com with SMTP id r17-20020a17090aa09100b0021903e75f14so7486088pjp.9 for ; Wed, 21 Dec 2022 14:24:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=6Jaams5HOGHGqoL3nWqw9y/u+ytv5a24WDfV/4t3+EM=; b=LDwwXjzscFBywLfN9uix9hm9llnv4amTnGjRnjZ8F1YoWJ+Sx8vfVGDC4kpHeC9EgO WEfcsu1baIScabXlw607E3i2788fHMyO1BW42G+5VcwQgeSz+v+pFd1u/ZvGxxcZewMy glCt5NEDvFxgZLf2WpUwBjGnB+61qLYoZe+yZIM+MdVv4lqAAJJSJ8No2FOJDwGC+2eo SShcXyTKuRFrZ6bHGPgKLETpTxyVlmx46vjGkcTyWPMWi+A6svFpJxcraGBV8jVYAheJ I/ZflFDXmfnhxn6uK6MeCWxfG0eYMFUoT0giUJZCSfsJHkNM5Axy8YtX7OUSRSnNuQ77 hUKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6Jaams5HOGHGqoL3nWqw9y/u+ytv5a24WDfV/4t3+EM=; b=Gl1cp3Snfi9lipwlTM26e3egkHiLrTjDRYdkWttcLAawMnezGPlgyOih8TDFRFNSn2 CcQHypI1PPZpoUIYBWHA0SwuuMSvTUG5W3JvEGFyKGq6CBoFgarn1tVihavvnzYbFhAn 3LDGQkTd0fIIkEW8eKTmPPguIp/c+sbXsXaG88Wm8dCOFfZ4qxjCp4+OWuenDMGdQQlI NFS1jsSSNslSXNBHf7IQcblGCsGSscTvk0rdVhlOUYvGa8G5wiDiBcY9hUCf7Z3ki53K ulsYFr8WfKLqlLzusJxkqGvIpjkGjia5hSGkj000THBDbFKoH2nttaZ2B8OV7t8v45ol LJ9g== X-Gm-Message-State: AFqh2kq8dWAnuBPYrptyoS6EHWehwEwWYhdwGXhwcjc71s3NQ3Tv0b9t cG3Ktn2zD7Y4dbSwgsyeTo7SUfYwoeIo X-Google-Smtp-Source: AMrXdXv3nPPSKzmL5MK8HgqmkhV+zzWahoEysQ0nKWMO9k5C3xkAafusuOpERZuBT6vSmNlvtCPFkPwx5seR X-Received: from sweer.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:e45]) (user=bgardon job=sendgmr) by 2002:a62:2f82:0:b0:576:670e:5de2 with SMTP id v124-20020a622f82000000b00576670e5de2mr274183pfv.70.1671661479091; Wed, 21 Dec 2022 14:24:39 -0800 (PST) Date: Wed, 21 Dec 2022 22:24:14 +0000 In-Reply-To: <20221221222418.3307832-1-bgardon@google.com> Mime-Version: 1.0 References: <20221221222418.3307832-1-bgardon@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20221221222418.3307832-11-bgardon@google.com> Subject: [RFC 10/14] KVM: x86/MMU: Fix naming on prepare / commit zap page functions From: Ben Gardon To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: Paolo Bonzini , Peter Xu , Sean Christopherson , David Matlack , Vipin Sharma , Nagareddy Reddy , Ben Gardon Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Since the various prepare / commit zap page functions are part of the Shadow MMU and used all over both shadow_mmu.c and mmu.c, add _shadow_ to the function names to match the rest of the Shadow MMU interface. Since there are so many uses of these functions, this rename gets its own commit. No functional change intended. Signed-off-by: Ben Gardon --- arch/x86/kvm/mmu/mmu.c | 21 +++++++-------- arch/x86/kvm/mmu/shadow_mmu.c | 48 ++++++++++++++++++----------------- arch/x86/kvm/mmu/shadow_mmu.h | 13 +++++----- 3 files changed, 43 insertions(+), 39 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 568b36de9eeb..160dd143a814 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -270,8 +270,9 @@ void walk_shadow_page_lockless_end(struct kvm_vcpu *vcpu) kvm_tdp_mmu_walk_lockless_end(); } else { /* - * Make sure the write to vcpu->mode is not reordered in front of - * reads to sptes. If it does, kvm_mmu_commit_zap_page() can see us + * Make sure the write to vcpu->mode is not reordered in front + * of reads to sptes. If it does, + * kvm_shadow_mmu_commit_zap_page() can see us * OUTSIDE_GUEST_MODE and proceed to free the shadow page table. */ smp_store_release(&vcpu->mode, OUTSIDE_GUEST_MODE); @@ -608,7 +609,7 @@ bool kvm_mmu_remote_flush_or_zap(struct kvm *kvm, struct list_head *invalid_list return false; if (!list_empty(invalid_list)) - kvm_mmu_commit_zap_page(kvm, invalid_list); + kvm_shadow_mmu_commit_zap_page(kvm, invalid_list); else kvm_flush_remote_tlbs(kvm); return true; @@ -1062,7 +1063,7 @@ static void mmu_free_root_page(struct kvm *kvm, hpa_t *root_hpa, if (is_tdp_mmu_page(sp)) kvm_tdp_mmu_put_root(kvm, sp, false); else if (!--sp->root_count && sp->role.invalid) - kvm_mmu_prepare_zap_page(kvm, sp, invalid_list); + kvm_shadow_mmu_prepare_zap_page(kvm, sp, invalid_list); *root_hpa = INVALID_PAGE; } @@ -1115,7 +1116,7 @@ void kvm_mmu_free_roots(struct kvm *kvm, struct kvm_mmu *mmu, mmu->root.pgd = 0; } - kvm_mmu_commit_zap_page(kvm, &invalid_list); + kvm_shadow_mmu_commit_zap_page(kvm, &invalid_list); write_unlock(&kvm->mmu_lock); } EXPORT_SYMBOL_GPL(kvm_mmu_free_roots); @@ -1417,8 +1418,8 @@ bool is_page_fault_stale(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault, * there is a pending request to free obsolete roots. The request is * only a hint that the current root _may_ be obsolete and needs to be * reloaded, e.g. if the guest frees a PGD that KVM is tracking as a - * previous root, then __kvm_mmu_prepare_zap_page() signals all vCPUs - * to reload even if no vCPU is actively using the root. + * previous root, then __kvm_shadow_mmu_prepare_zap_page() signals all + * vCPUs to reload even if no vCPU is actively using the root. */ if (!sp && kvm_test_request(KVM_REQ_MMU_FREE_OBSOLETE_ROOTS, vcpu)) return true; @@ -3103,13 +3104,13 @@ void kvm_mmu_zap_all(struct kvm *kvm) list_for_each_entry_safe(sp, node, &kvm->arch.active_mmu_pages, link) { if (WARN_ON(sp->role.invalid)) continue; - if (__kvm_mmu_prepare_zap_page(kvm, sp, &invalid_list, &ign)) + if (__kvm_shadow_mmu_prepare_zap_page(kvm, sp, &invalid_list, &ign)) goto restart; if (cond_resched_rwlock_write(&kvm->mmu_lock)) goto restart; } - kvm_mmu_commit_zap_page(kvm, &invalid_list); + kvm_shadow_mmu_commit_zap_page(kvm, &invalid_list); if (is_tdp_mmu_enabled(kvm)) kvm_tdp_mmu_zap_all(kvm); @@ -3452,7 +3453,7 @@ static void kvm_recover_nx_huge_pages(struct kvm *kvm) else if (is_tdp_mmu_page(sp)) flush |= kvm_tdp_mmu_zap_sp(kvm, sp); else - kvm_mmu_prepare_zap_page(kvm, sp, &invalid_list); + kvm_shadow_mmu_prepare_zap_page(kvm, sp, &invalid_list); WARN_ON_ONCE(sp->nx_huge_page_disallowed); if (need_resched() || rwlock_needbreak(&kvm->mmu_lock)) { diff --git a/arch/x86/kvm/mmu/shadow_mmu.c b/arch/x86/kvm/mmu/shadow_mmu.c index e36b4d9c67f2..2d1a4026cf00 100644 --- a/arch/x86/kvm/mmu/shadow_mmu.c +++ b/arch/x86/kvm/mmu/shadow_mmu.c @@ -1280,7 +1280,7 @@ static int kvm_sync_page(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, int ret = vcpu->arch.mmu->sync_page(vcpu, sp); if (ret < 0) - kvm_mmu_prepare_zap_page(vcpu->kvm, sp, invalid_list); + kvm_shadow_mmu_prepare_zap_page(vcpu->kvm, sp, invalid_list); return ret; } @@ -1442,8 +1442,8 @@ static struct kvm_mmu_page *kvm_mmu_find_shadow_page(struct kvm *kvm, * upper-level page will be write-protected. */ if (role.level > PG_LEVEL_4K && sp->unsync) - kvm_mmu_prepare_zap_page(kvm, sp, - &invalid_list); + kvm_shadow_mmu_prepare_zap_page(kvm, sp, + &invalid_list); continue; } @@ -1485,7 +1485,7 @@ static struct kvm_mmu_page *kvm_mmu_find_shadow_page(struct kvm *kvm, ++kvm->stat.mmu_cache_miss; out: - kvm_mmu_commit_zap_page(kvm, &invalid_list); + kvm_shadow_mmu_commit_zap_page(kvm, &invalid_list); if (collisions > kvm->stat.max_mmu_page_hash_collisions) kvm->stat.max_mmu_page_hash_collisions = collisions; @@ -1768,8 +1768,8 @@ static int mmu_page_zap_pte(struct kvm *kvm, struct kvm_mmu_page *sp, u64 *spte, */ if (tdp_enabled && invalid_list && child->role.guest_mode && !child->parent_ptes.val) - return kvm_mmu_prepare_zap_page(kvm, child, - invalid_list); + return kvm_shadow_mmu_prepare_zap_page(kvm, + child, invalid_list); } } else if (is_mmio_spte(pte)) { mmu_spte_clear_no_track(spte); @@ -1814,7 +1814,7 @@ static int mmu_zap_unsync_children(struct kvm *kvm, struct kvm_mmu_page *sp; for_each_sp(pages, sp, parents, i) { - kvm_mmu_prepare_zap_page(kvm, sp, invalid_list); + kvm_shadow_mmu_prepare_zap_page(kvm, sp, invalid_list); mmu_pages_clear_parents(&parents); zapped++; } @@ -1823,9 +1823,9 @@ static int mmu_zap_unsync_children(struct kvm *kvm, return zapped; } -bool __kvm_mmu_prepare_zap_page(struct kvm *kvm, struct kvm_mmu_page *sp, - struct list_head *invalid_list, - int *nr_zapped) +bool __kvm_shadow_mmu_prepare_zap_page(struct kvm *kvm, struct kvm_mmu_page *sp, + struct list_head *invalid_list, + int *nr_zapped) { bool list_unstable, zapped_root = false; @@ -1886,16 +1886,17 @@ bool __kvm_mmu_prepare_zap_page(struct kvm *kvm, struct kvm_mmu_page *sp, return list_unstable; } -bool kvm_mmu_prepare_zap_page(struct kvm *kvm, struct kvm_mmu_page *sp, - struct list_head *invalid_list) +bool kvm_shadow_mmu_prepare_zap_page(struct kvm *kvm, struct kvm_mmu_page *sp, + struct list_head *invalid_list) { int nr_zapped; - __kvm_mmu_prepare_zap_page(kvm, sp, invalid_list, &nr_zapped); + __kvm_shadow_mmu_prepare_zap_page(kvm, sp, invalid_list, &nr_zapped); return nr_zapped; } -void kvm_mmu_commit_zap_page(struct kvm *kvm, struct list_head *invalid_list) +void kvm_shadow_mmu_commit_zap_page(struct kvm *kvm, + struct list_head *invalid_list) { struct kvm_mmu_page *sp, *nsp; @@ -1940,8 +1941,8 @@ static unsigned long kvm_mmu_zap_oldest_mmu_pages(struct kvm *kvm, if (sp->root_count) continue; - unstable = __kvm_mmu_prepare_zap_page(kvm, sp, &invalid_list, - &nr_zapped); + unstable = __kvm_shadow_mmu_prepare_zap_page(kvm, sp, + &invalid_list, &nr_zapped); total_zapped += nr_zapped; if (total_zapped >= nr_to_zap) break; @@ -1950,7 +1951,7 @@ static unsigned long kvm_mmu_zap_oldest_mmu_pages(struct kvm *kvm, goto restart; } - kvm_mmu_commit_zap_page(kvm, &invalid_list); + kvm_shadow_mmu_commit_zap_page(kvm, &invalid_list); kvm->stat.mmu_recycled += total_zapped; return total_zapped; @@ -2021,9 +2022,9 @@ int kvm_mmu_unprotect_page(struct kvm *kvm, gfn_t gfn) pgprintk("%s: gfn %llx role %x\n", __func__, gfn, sp->role.word); r = 1; - kvm_mmu_prepare_zap_page(kvm, sp, &invalid_list); + kvm_shadow_mmu_prepare_zap_page(kvm, sp, &invalid_list); } - kvm_mmu_commit_zap_page(kvm, &invalid_list); + kvm_shadow_mmu_commit_zap_page(kvm, &invalid_list); write_unlock(&kvm->mmu_lock); return r; @@ -3020,7 +3021,8 @@ void kvm_mmu_pte_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new, for_each_gfn_valid_sp_with_gptes(vcpu->kvm, sp, gfn) { if (detect_write_misaligned(sp, gpa, bytes) || detect_write_flooding(sp)) { - kvm_mmu_prepare_zap_page(vcpu->kvm, sp, &invalid_list); + kvm_shadow_mmu_prepare_zap_page(vcpu->kvm, sp, + &invalid_list); ++vcpu->kvm->stat.mmu_flooded; continue; } @@ -3128,7 +3130,7 @@ void kvm_shadow_mmu_zap_obsolete_pages(struct kvm *kvm) goto restart; } - unstable = __kvm_mmu_prepare_zap_page(kvm, sp, + unstable = __kvm_shadow_mmu_prepare_zap_page(kvm, sp, &kvm->arch.zapped_obsolete_pages, &nr_zapped); batch += nr_zapped; @@ -3145,7 +3147,7 @@ void kvm_shadow_mmu_zap_obsolete_pages(struct kvm *kvm) * kvm_mmu_load()), and the reload in the caller ensure no vCPUs are * running with an obsolete MMU. */ - kvm_mmu_commit_zap_page(kvm, &kvm->arch.zapped_obsolete_pages); + kvm_shadow_mmu_commit_zap_page(kvm, &kvm->arch.zapped_obsolete_pages); } bool kvm_shadow_mmu_has_zapped_obsolete_pages(struct kvm *kvm) @@ -3426,7 +3428,7 @@ unsigned long kvm_shadow_mmu_shrink_scan(struct kvm *kvm, int pages_to_free) write_lock(&kvm->mmu_lock); if (kvm_shadow_mmu_has_zapped_obsolete_pages(kvm)) { - kvm_mmu_commit_zap_page(kvm, &kvm->arch.zapped_obsolete_pages); + kvm_shadow_mmu_commit_zap_page(kvm, &kvm->arch.zapped_obsolete_pages); goto out; } diff --git a/arch/x86/kvm/mmu/shadow_mmu.h b/arch/x86/kvm/mmu/shadow_mmu.h index 148cc3593d2b..af201d34d0b2 100644 --- a/arch/x86/kvm/mmu/shadow_mmu.h +++ b/arch/x86/kvm/mmu/shadow_mmu.h @@ -53,12 +53,13 @@ bool kvm_test_age_rmap(struct kvm *kvm, struct kvm_rmap_head *rmap_head, void __clear_sp_write_flooding_count(struct kvm_mmu_page *sp); -bool __kvm_mmu_prepare_zap_page(struct kvm *kvm, struct kvm_mmu_page *sp, - struct list_head *invalid_list, - int *nr_zapped); -bool kvm_mmu_prepare_zap_page(struct kvm *kvm, struct kvm_mmu_page *sp, - struct list_head *invalid_list); -void kvm_mmu_commit_zap_page(struct kvm *kvm, struct list_head *invalid_list); +bool __kvm_shadow_mmu_prepare_zap_page(struct kvm *kvm, struct kvm_mmu_page *sp, + struct list_head *invalid_list, + int *nr_zapped); +bool kvm_shadow_mmu_prepare_zap_page(struct kvm *kvm, struct kvm_mmu_page *sp, + struct list_head *invalid_list); +void kvm_shadow_mmu_commit_zap_page(struct kvm *kvm, + struct list_head *invalid_list); int kvm_shadow_mmu_make_pages_available(struct kvm_vcpu *vcpu); From patchwork Wed Dec 21 22:24:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ben Gardon X-Patchwork-Id: 13079199 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4971DC10F1B for ; Wed, 21 Dec 2022 22:25:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235113AbiLUWZo (ORCPT ); Wed, 21 Dec 2022 17:25:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37076 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235042AbiLUWZM (ORCPT ); Wed, 21 Dec 2022 17:25:12 -0500 Received: from mail-pg1-x54a.google.com (mail-pg1-x54a.google.com [IPv6:2607:f8b0:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B76D82790D for ; Wed, 21 Dec 2022 14:24:41 -0800 (PST) Received: by mail-pg1-x54a.google.com with SMTP id g1-20020a636b01000000b00479222d9875so102332pgc.12 for ; Wed, 21 Dec 2022 14:24:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=G8yCm8/A9muZfDD3Vt2EwogqpDhIyDMvg2eScRp1Hns=; b=RZtEbegJX7+lnjjBMFgZ9yF+y8dpXdZXUnYPKvKz5AnEC3U28/Sy5KCutLHCHP9qii KFytERkZFCHgqDPy7+veHu2TMRasyzT+t0xfCh1UPiqN16FymNAzzxa9XHMJQqDLLKKp 3/i9EgNeEXg34KUtXzZGsmODzlMYPaFeGTQ552wdnUk2RqqvGPcmMEDbt0xbfEhgtDHY s9ncMZW7Lx8E9FfxdqlSAEhyEFNFZhS+Swfg9V0tQRaF2yuhmHLZFhYEJNCMf3OQH8tp LU9mFt24OB62q0EKzcPAS4NYVShoOWZexvsM428AQaFopIGAMH8lOwOOoXMkVOrXXsV8 9SYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=G8yCm8/A9muZfDD3Vt2EwogqpDhIyDMvg2eScRp1Hns=; b=LQy/YPq+7+1rAUiV/iBN6qFihJ3LnvdKLH3IcaFyh7ue721xGXpJyITyAZNHWqlvS0 UyFfKYm/XmJfetso+NRjAUd0QL3fpEYAzZ4roj/SH9skYVtmZuzeXVCow7g/JPFNdwxC 9ltBWImsgF6arOWpxTelJ/jpNPOKwML2J38t+6TMEHlQ4R58ISW38EUQRJOMXzTpbfRm +y4Lv1K7T+cqHVYLxWJdGl5CgQlW4qvDRpTfDoOvCwukIZYYVMOV14JykvfWxTXav0/4 aCsPe66lQX6YwFuhYt5AhYJ2c29Y2WpumCgl6FXhL7L/5EAaxp8JEwXRJYMFe6pSwCrq XQgw== X-Gm-Message-State: AFqh2kpJyXE3iGZKGzVaY17xDQ+hq9ce5UNc+sM3uHYjrGUbDzWv7FZQ SAMeR5QuvASGFncr8Wu5aAUb1qdpU6x2 X-Google-Smtp-Source: AMrXdXs312qIWY+dZtObZ6u1B9Ff0KtgXEvz31qgLoC/IptoeHSVZDxiHLVxBuFUD5n2iAI2eXbWE4zug7Py X-Received: from sweer.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:e45]) (user=bgardon job=sendgmr) by 2002:aa7:9f4c:0:b0:56d:8d19:f331 with SMTP id h12-20020aa79f4c000000b0056d8d19f331mr259462pfr.7.1671661480589; Wed, 21 Dec 2022 14:24:40 -0800 (PST) Date: Wed, 21 Dec 2022 22:24:15 +0000 In-Reply-To: <20221221222418.3307832-1-bgardon@google.com> Mime-Version: 1.0 References: <20221221222418.3307832-1-bgardon@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20221221222418.3307832-12-bgardon@google.com> Subject: [RFC 11/14] KVM: x86/MMU: Factor Shadow MMU wrprot / clear dirty ops out of mmu.c From: Ben Gardon To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: Paolo Bonzini , Peter Xu , Sean Christopherson , David Matlack , Vipin Sharma , Nagareddy Reddy , Ben Gardon Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org There are several functions in mmu.c which bifrucate to the Shadow and/or TDP MMU implementations. In most of these, the Shadow MMU implementation is open-coded. Wrap these instances in a nice function which just needs kvm and slot arguments or similar. This matches the TDP MMU interface and will allow for some nice cleanups in a following commit. No functional change intended. Signed-off-by: Ben Gardon --- arch/x86/kvm/mmu/mmu.c | 52 ++++++---------------------- arch/x86/kvm/mmu/shadow_mmu.c | 64 +++++++++++++++++++++++++++++++++++ arch/x86/kvm/mmu/shadow_mmu.h | 15 ++++++++ 3 files changed, 90 insertions(+), 41 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 160dd143a814..ce2a6dd38c67 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -417,23 +417,13 @@ static void kvm_mmu_write_protect_pt_masked(struct kvm *kvm, struct kvm_memory_slot *slot, gfn_t gfn_offset, unsigned long mask) { - struct kvm_rmap_head *rmap_head; - if (is_tdp_mmu_enabled(kvm)) kvm_tdp_mmu_clear_dirty_pt_masked(kvm, slot, slot->base_gfn + gfn_offset, mask, true); - if (!kvm_memslots_have_rmaps(kvm)) - return; - - while (mask) { - rmap_head = gfn_to_rmap(slot->base_gfn + gfn_offset + __ffs(mask), - PG_LEVEL_4K, slot); - rmap_write_protect(rmap_head, false); - - /* clear the first set bit */ - mask &= mask - 1; - } + if (kvm_memslots_have_rmaps(kvm)) + kvm_shadow_mmu_write_protect_pt_masked(kvm, slot, gfn_offset, + mask); } /** @@ -450,23 +440,13 @@ static void kvm_mmu_clear_dirty_pt_masked(struct kvm *kvm, struct kvm_memory_slot *slot, gfn_t gfn_offset, unsigned long mask) { - struct kvm_rmap_head *rmap_head; - if (is_tdp_mmu_enabled(kvm)) kvm_tdp_mmu_clear_dirty_pt_masked(kvm, slot, slot->base_gfn + gfn_offset, mask, false); - if (!kvm_memslots_have_rmaps(kvm)) - return; - - while (mask) { - rmap_head = gfn_to_rmap(slot->base_gfn + gfn_offset + __ffs(mask), - PG_LEVEL_4K, slot); - __rmap_clear_dirty(kvm, rmap_head, slot); - - /* clear the first set bit */ - mask &= mask - 1; - } + if (kvm_memslots_have_rmaps(kvm)) + kvm_shadow_mmu_clear_dirty_pt_masked(kvm, slot, gfn_offset, + mask); } /** @@ -524,16 +504,11 @@ bool kvm_mmu_slot_gfn_write_protect(struct kvm *kvm, struct kvm_memory_slot *slot, u64 gfn, int min_level) { - struct kvm_rmap_head *rmap_head; - int i; bool write_protected = false; - if (kvm_memslots_have_rmaps(kvm)) { - for (i = min_level; i <= KVM_MAX_HUGEPAGE_LEVEL; ++i) { - rmap_head = gfn_to_rmap(gfn, i, slot); - write_protected |= rmap_write_protect(rmap_head, true); - } - } + if (kvm_memslots_have_rmaps(kvm)) + write_protected |= + kvm_shadow_mmu_write_protect_gfn(kvm, slot, gfn, min_level); if (is_tdp_mmu_enabled(kvm)) write_protected |= @@ -2917,8 +2892,7 @@ void kvm_mmu_slot_remove_write_access(struct kvm *kvm, { if (kvm_memslots_have_rmaps(kvm)) { write_lock(&kvm->mmu_lock); - slot_handle_level(kvm, memslot, slot_rmap_write_protect, - start_level, KVM_MAX_HUGEPAGE_LEVEL, false); + kvm_shadow_mmu_wrprot_slot(kvm, memslot, start_level); write_unlock(&kvm->mmu_lock); } @@ -3069,11 +3043,7 @@ void kvm_mmu_slot_leaf_clear_dirty(struct kvm *kvm, { if (kvm_memslots_have_rmaps(kvm)) { write_lock(&kvm->mmu_lock); - /* - * Clear dirty bits only on 4k SPTEs since the legacy MMU only - * support dirty logging at a 4k granularity. - */ - slot_handle_level_4k(kvm, memslot, __rmap_clear_dirty, false); + kvm_shadow_mmu_clear_dirty_slot(kvm, memslot); write_unlock(&kvm->mmu_lock); } diff --git a/arch/x86/kvm/mmu/shadow_mmu.c b/arch/x86/kvm/mmu/shadow_mmu.c index 2d1a4026cf00..80b8c78daaeb 100644 --- a/arch/x86/kvm/mmu/shadow_mmu.c +++ b/arch/x86/kvm/mmu/shadow_mmu.c @@ -3440,3 +3440,67 @@ unsigned long kvm_shadow_mmu_shrink_scan(struct kvm *kvm, int pages_to_free) return freed; } + +void kvm_shadow_mmu_write_protect_pt_masked(struct kvm *kvm, + struct kvm_memory_slot *slot, + gfn_t gfn_offset, unsigned long mask) +{ + struct kvm_rmap_head *rmap_head; + + while (mask) { + rmap_head = gfn_to_rmap(slot->base_gfn + gfn_offset + __ffs(mask), + PG_LEVEL_4K, slot); + rmap_write_protect(rmap_head, false); + + /* clear the first set bit */ + mask &= mask - 1; + } +} + +void kvm_shadow_mmu_clear_dirty_pt_masked(struct kvm *kvm, + struct kvm_memory_slot *slot, + gfn_t gfn_offset, unsigned long mask) +{ + struct kvm_rmap_head *rmap_head; + + while (mask) { + rmap_head = gfn_to_rmap(slot->base_gfn + gfn_offset + __ffs(mask), + PG_LEVEL_4K, slot); + __rmap_clear_dirty(kvm, rmap_head, slot); + + /* clear the first set bit */ + mask &= mask - 1; + } +} + +bool kvm_shadow_mmu_write_protect_gfn(struct kvm *kvm, + struct kvm_memory_slot *slot, + u64 gfn, int min_level) +{ + struct kvm_rmap_head *rmap_head; + int i; + bool write_protected = false; + + if (kvm_memslots_have_rmaps(kvm)) { + for (i = min_level; i <= KVM_MAX_HUGEPAGE_LEVEL; ++i) { + rmap_head = gfn_to_rmap(gfn, i, slot); + write_protected |= rmap_write_protect(rmap_head, true); + } + } + + return write_protected; +} + +void kvm_shadow_mmu_clear_dirty_slot(struct kvm *kvm, + const struct kvm_memory_slot *memslot) +{ + slot_handle_level_4k(kvm, memslot, __rmap_clear_dirty, false); +} + +void kvm_shadow_mmu_wrprot_slot(struct kvm *kvm, + const struct kvm_memory_slot *memslot, + int start_level) +{ + slot_handle_level(kvm, memslot, slot_rmap_write_protect, + start_level, KVM_MAX_HUGEPAGE_LEVEL, false); +} diff --git a/arch/x86/kvm/mmu/shadow_mmu.h b/arch/x86/kvm/mmu/shadow_mmu.h index af201d34d0b2..c322eeaa0688 100644 --- a/arch/x86/kvm/mmu/shadow_mmu.h +++ b/arch/x86/kvm/mmu/shadow_mmu.h @@ -104,6 +104,21 @@ void kvm_shadow_mmu_zap_collapsible_sptes(struct kvm *kvm, bool kvm_shadow_mmu_has_zapped_obsolete_pages(struct kvm *kvm); unsigned long kvm_shadow_mmu_shrink_scan(struct kvm *kvm, int pages_to_free); +void kvm_shadow_mmu_write_protect_pt_masked(struct kvm *kvm, + struct kvm_memory_slot *slot, + gfn_t gfn_offset, unsigned long mask); +void kvm_shadow_mmu_clear_dirty_pt_masked(struct kvm *kvm, + struct kvm_memory_slot *slot, + gfn_t gfn_offset, unsigned long mask); +bool kvm_shadow_mmu_write_protect_gfn(struct kvm *kvm, + struct kvm_memory_slot *slot, + u64 gfn, int min_level); +void kvm_shadow_mmu_clear_dirty_slot(struct kvm *kvm, + const struct kvm_memory_slot *memslot); +void kvm_shadow_mmu_wrprot_slot(struct kvm *kvm, + const struct kvm_memory_slot *memslot, + int start_level); + /* Exports from paging_tmpl.h */ gpa_t paging32_gva_to_gpa(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, gpa_t vaddr, u64 access, From patchwork Wed Dec 21 22:24:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ben Gardon X-Patchwork-Id: 13079200 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BD944C4332F for ; Wed, 21 Dec 2022 22:25:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235123AbiLUWZq (ORCPT ); Wed, 21 Dec 2022 17:25:46 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37326 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234982AbiLUWZN (ORCPT ); Wed, 21 Dec 2022 17:25:13 -0500 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DE8D2275FA for ; Wed, 21 Dec 2022 14:24:42 -0800 (PST) Received: by mail-pj1-x1049.google.com with SMTP id r17-20020a17090aa09100b0021903e75f14so7486121pjp.9 for ; Wed, 21 Dec 2022 14:24:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ZvHSZ7wyO6HpWC7GgWotvlWeVl4F+Q0dZdGlQyXVA6g=; b=SZmxxj3D7ZU/s/vea1hbQX3dVt8uH8sCEHNP+G6nLRcEHVdJ8hshrQVvHHLhSFn40n gJwzzUv13Wwj4TPXk4zLMaNE7Il3+japKHq9hAOS3Uzam7+8ANCSXsQO5uZjmDyrib8v n7BsILS36PW4iMO4uWVAOs6+sx5t4+1pg/u9I3KpAsMyLw8mHxIVg/Wt2pfxeyHDqcxc HGT2NGqNDuzehTa0QijHcR97mXOs1J+H+2mJ46JHFhPk6XZfnisD+R37bMDh2M9WiZq3 MSr8s3Qhiy5xIfqLB/smHZVEetk3udAPOdilIivf5+njgF/jOhq3oLoDwhHLecPyYFNr afyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ZvHSZ7wyO6HpWC7GgWotvlWeVl4F+Q0dZdGlQyXVA6g=; b=UHuzIOCfPt+GjHDk6nhYABx2eUPd0sJLejYevBs9Itk27/dc3rbA/8qFjTmrznCEsr DAJfhGP5fQLYvHYOk8/SizYOWGlGa+jkkTNk0QEvcH/wZUnk4UwtNzTfl27NHyZZagnt oJCerCo2+mOZoX1lKniB6Ndau36KNNexQVRpNGia8JBSrAgbixWYaAzsbUyFl0161lOl KW8t+uiPZRPmyAtJd4g2n2KpYVogl60EkT7DnABA0fvSsMXdVax8KMYPf2lE+UN8gFij NgFv3oXg+vH+GqvXMATq8KSAAGzT0Kkr13JH04J6yIK+7YcFc9boZVeRe1X5h32GMJyx kwEg== X-Gm-Message-State: AFqh2ko0YLxRolLBdEFQc+dfvPHxy+4fDnUDIZtexwJ7lYRDQeLyRQpV a8Ol1cM7QM/x8WaxBa1xkzwFXNsiE0gk X-Google-Smtp-Source: AMrXdXsBIYopM+oXFd1TuMnwioq69ljz0C7ryUPd48AdQSG9d1WWHfGQSnTqu93hPAaLpy1hYVRjWli9HEic X-Received: from sweer.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:e45]) (user=bgardon job=sendgmr) by 2002:a05:6a00:3404:b0:577:4103:8da with SMTP id cn4-20020a056a00340400b00577410308damr245364pfb.1.1671661482452; Wed, 21 Dec 2022 14:24:42 -0800 (PST) Date: Wed, 21 Dec 2022 22:24:16 +0000 In-Reply-To: <20221221222418.3307832-1-bgardon@google.com> Mime-Version: 1.0 References: <20221221222418.3307832-1-bgardon@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20221221222418.3307832-13-bgardon@google.com> Subject: [RFC 12/14] KVM: x86/MMU: Remove unneeded exports from shadow_mmu.c From: Ben Gardon To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: Paolo Bonzini , Peter Xu , Sean Christopherson , David Matlack , Vipin Sharma , Nagareddy Reddy , Ben Gardon Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Now that the various dirty logging / wrprot function implementations are in shadow_mmu.c, do another round of cleanups to remove functions which no longer need to be exposed and can be marked static. No functional change intended. Signed-off-by: Ben Gardon --- arch/x86/kvm/mmu/shadow_mmu.c | 30 +++++++++++++++++------------- arch/x86/kvm/mmu/shadow_mmu.h | 18 ------------------ 2 files changed, 17 insertions(+), 31 deletions(-) diff --git a/arch/x86/kvm/mmu/shadow_mmu.c b/arch/x86/kvm/mmu/shadow_mmu.c index 80b8c78daaeb..77472eb9b06a 100644 --- a/arch/x86/kvm/mmu/shadow_mmu.c +++ b/arch/x86/kvm/mmu/shadow_mmu.c @@ -632,8 +632,8 @@ unsigned int pte_list_count(struct kvm_rmap_head *rmap_head) return count; } -struct kvm_rmap_head *gfn_to_rmap(gfn_t gfn, int level, - const struct kvm_memory_slot *slot) +static struct kvm_rmap_head *gfn_to_rmap(gfn_t gfn, int level, + const struct kvm_memory_slot *slot) { unsigned long idx; @@ -801,7 +801,7 @@ static bool spte_write_protect(u64 *sptep, bool pt_protect) return mmu_spte_update(sptep, spte); } -bool rmap_write_protect(struct kvm_rmap_head *rmap_head, bool pt_protect) +static bool rmap_write_protect(struct kvm_rmap_head *rmap_head, bool pt_protect) { u64 *sptep; struct rmap_iterator iter; @@ -840,8 +840,8 @@ static bool spte_wrprot_for_clear_dirty(u64 *sptep) * - W bit on ad-disabled SPTEs. * Returns true iff any D or W bits were cleared. */ -bool __rmap_clear_dirty(struct kvm *kvm, struct kvm_rmap_head *rmap_head, - const struct kvm_memory_slot *slot) +static bool __rmap_clear_dirty(struct kvm *kvm, struct kvm_rmap_head *rmap_head, + const struct kvm_memory_slot *slot) { u64 *sptep; struct rmap_iterator iter; @@ -3045,6 +3045,11 @@ void kvm_mmu_pte_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new, write_unlock(&vcpu->kvm->mmu_lock); } +/* The return value indicates if tlb flush on all vcpus is needed. */ +typedef bool (*slot_level_handler) (struct kvm *kvm, + struct kvm_rmap_head *rmap_head, + const struct kvm_memory_slot *slot); + /* The caller should hold mmu-lock before calling this function. */ static __always_inline bool slot_handle_level_range(struct kvm *kvm, const struct kvm_memory_slot *memslot, @@ -3073,10 +3078,10 @@ slot_handle_level_range(struct kvm *kvm, const struct kvm_memory_slot *memslot, return flush; } -__always_inline bool slot_handle_level(struct kvm *kvm, - const struct kvm_memory_slot *memslot, - slot_level_handler fn, int start_level, - int end_level, bool flush_on_yield) +static __always_inline bool +slot_handle_level(struct kvm *kvm, const struct kvm_memory_slot *memslot, + slot_level_handler fn, int start_level, int end_level, + bool flush_on_yield) { return slot_handle_level_range(kvm, memslot, fn, start_level, end_level, memslot->base_gfn, @@ -3084,10 +3089,9 @@ __always_inline bool slot_handle_level(struct kvm *kvm, flush_on_yield, false); } -__always_inline bool slot_handle_level_4k(struct kvm *kvm, - const struct kvm_memory_slot *memslot, - slot_level_handler fn, - bool flush_on_yield) +static __always_inline bool +slot_handle_level_4k(struct kvm *kvm, const struct kvm_memory_slot *memslot, + slot_level_handler fn, bool flush_on_yield) { return slot_handle_level(kvm, memslot, fn, PG_LEVEL_4K, PG_LEVEL_4K, flush_on_yield); diff --git a/arch/x86/kvm/mmu/shadow_mmu.h b/arch/x86/kvm/mmu/shadow_mmu.h index c322eeaa0688..397fb463ef54 100644 --- a/arch/x86/kvm/mmu/shadow_mmu.h +++ b/arch/x86/kvm/mmu/shadow_mmu.h @@ -26,11 +26,6 @@ struct pte_list_desc { /* Only exported for debugfs.c. */ unsigned int pte_list_count(struct kvm_rmap_head *rmap_head); -struct kvm_rmap_head *gfn_to_rmap(gfn_t gfn, int level, - const struct kvm_memory_slot *slot); -bool rmap_write_protect(struct kvm_rmap_head *rmap_head, bool pt_protect); -bool __rmap_clear_dirty(struct kvm *kvm, struct kvm_rmap_head *rmap_head, - const struct kvm_memory_slot *slot); bool kvm_zap_rmap(struct kvm *kvm, struct kvm_rmap_head *rmap_head, struct kvm_memory_slot *slot, gfn_t gfn, int level, pte_t unused); @@ -78,22 +73,9 @@ int kvm_shadow_mmu_get_walk(struct kvm_vcpu *vcpu, u64 addr, u64 *sptes, void kvm_mmu_pte_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new, int bytes, struct kvm_page_track_notifier_node *node); -/* The return value indicates if tlb flush on all vcpus is needed. */ -typedef bool (*slot_level_handler) (struct kvm *kvm, - struct kvm_rmap_head *rmap_head, - const struct kvm_memory_slot *slot); -bool slot_handle_level(struct kvm *kvm, const struct kvm_memory_slot *memslot, - slot_level_handler fn, int start_level, int end_level, - bool flush_on_yield); -bool slot_handle_level_4k(struct kvm *kvm, const struct kvm_memory_slot *memslot, - slot_level_handler fn, bool flush_on_yield); - void kvm_shadow_mmu_zap_obsolete_pages(struct kvm *kvm); bool kvm_shadow_mmu_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end); -bool slot_rmap_write_protect(struct kvm *kvm, struct kvm_rmap_head *rmap_head, - const struct kvm_memory_slot *slot); - void kvm_shadow_mmu_try_split_huge_pages(struct kvm *kvm, const struct kvm_memory_slot *slot, gfn_t start, gfn_t end, From patchwork Wed Dec 21 22:24:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ben Gardon X-Patchwork-Id: 13079201 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 15988C4332F for ; Wed, 21 Dec 2022 22:26:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235134AbiLUW0D (ORCPT ); Wed, 21 Dec 2022 17:26:03 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37548 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235011AbiLUWZX (ORCPT ); Wed, 21 Dec 2022 17:25:23 -0500 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AEEAD27935 for ; Wed, 21 Dec 2022 14:24:44 -0800 (PST) Received: by mail-pj1-x1049.google.com with SMTP id il11-20020a17090b164b00b00219a4366109so1980280pjb.0 for ; Wed, 21 Dec 2022 14:24:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=lUvSzm6zhuFwhwDfBHcpPhD05Pwase/3rciSEa3uh/0=; b=Vfbr638K23Tkpy4chmhkz5LuwrNuGxo4NEmcxSAginABFnm1kw7ZenYH4gp9QgSO11 VQEQb+QJWIIWf6UTTkicMqeyvNOQ352ckTefjpJ6hIeef8iMLigA+xLhQvl76CZDldhG /vfNHHh2mcYSByw2dshKNTQmiFQt9db7lIIZKsspjCSPfGPOckOaqu0Re/ZpfkAgeV5V 4s01ixfykyOpDlcceEuPlsaGCSvIfjrIOGdaPFeqj8oct91A/9XvjCMreyP4kDEFUQJM d+Txg1y1hD7UEgILJf9ZMrBjmrH+CglkTPz4qTdePAikPG7sByHhuN1IRTByAXxE7Gvv F1Eg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=lUvSzm6zhuFwhwDfBHcpPhD05Pwase/3rciSEa3uh/0=; b=6a3fyxByLWC1zKH89xy6jo+0ob8k5GvuajApu5TlxZo5B0BqEBGH/WWMTP5yJI8o9U VKDiW2Ieo5yLRsmB+ylQ9x5S+HWBgRHXHmrFtO7W1ijV5LCExzbfXx/nh+crtT48stRm OTBJCGNEaOZwGmT6BsXuSoXWj0hJ3p0ocgm7jAy2plASs3Qy5xr1I1kq/OJWCx04QJPQ a5viwowrW3Jx8Zw6E6tlJcdvu4O5QkI2OPnIwQhZiCB3HS86ZCaveyJdTI8zM7ZE71JA 3+l8ABI7irIzVIrqo8BZXtpmiyp9y7x9lxZO3Asbtuow4VH3pHmBgkKGKM3FjLeR4yxx k6EQ== X-Gm-Message-State: AFqh2kqbePr+04U4voW0QXhHqXYuzmVjSoGotmzJkl4v46/47IJGJIIB jfdZukIQNfYbNmh6d9wC5HGbejQDs8NE X-Google-Smtp-Source: AMrXdXtJshUqhw+9mehxMoK1nRrMONQlvaH9Vi3VdQOpEWUV7f5Sq0x8urLAg0f8htgiC5XpMFkfjkBaJ071 X-Received: from sweer.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:e45]) (user=bgardon job=sendgmr) by 2002:a05:6a00:1d81:b0:576:ba28:29a8 with SMTP id z1-20020a056a001d8100b00576ba2829a8mr204908pfw.47.1671661484083; Wed, 21 Dec 2022 14:24:44 -0800 (PST) Date: Wed, 21 Dec 2022 22:24:17 +0000 In-Reply-To: <20221221222418.3307832-1-bgardon@google.com> Mime-Version: 1.0 References: <20221221222418.3307832-1-bgardon@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20221221222418.3307832-14-bgardon@google.com> Subject: [RFC 13/14] KVM: x86/MMU: Wrap uses of kvm_handle_gfn_range in mmu.c From: Ben Gardon To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: Paolo Bonzini , Peter Xu , Sean Christopherson , David Matlack , Vipin Sharma , Nagareddy Reddy , Ben Gardon Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org handle_gfn_range + callback is not a bad interface, but it requires exporting the whole callback scheme to mmu.c. Simplify the interface with some basic wrapper functions, making the callback scheme internal to shadow_mmu.c. No functional change intended. Signed-off-by: Ben Gardon --- arch/x86/kvm/mmu/mmu.c | 8 +++--- arch/x86/kvm/mmu/shadow_mmu.c | 54 +++++++++++++++++++++++++---------- arch/x86/kvm/mmu/shadow_mmu.h | 25 ++++------------ 3 files changed, 48 insertions(+), 39 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index ce2a6dd38c67..ceb3146016d0 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -530,7 +530,7 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range) bool flush = false; if (kvm_memslots_have_rmaps(kvm)) - flush = kvm_handle_gfn_range(kvm, range, kvm_zap_rmap); + flush = kvm_shadow_mmu_unmap_gfn_range(kvm, range); if (is_tdp_mmu_enabled(kvm)) flush = kvm_tdp_mmu_unmap_gfn_range(kvm, range, flush); @@ -543,7 +543,7 @@ bool kvm_set_spte_gfn(struct kvm *kvm, struct kvm_gfn_range *range) bool flush = false; if (kvm_memslots_have_rmaps(kvm)) - flush = kvm_handle_gfn_range(kvm, range, kvm_set_pte_rmap); + flush = kvm_shadow_mmu_set_spte_gfn(kvm, range); if (is_tdp_mmu_enabled(kvm)) flush |= kvm_tdp_mmu_set_spte_gfn(kvm, range); @@ -556,7 +556,7 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) bool young = false; if (kvm_memslots_have_rmaps(kvm)) - young = kvm_handle_gfn_range(kvm, range, kvm_age_rmap); + young = kvm_shadow_mmu_age_gfn_range(kvm, range); if (is_tdp_mmu_enabled(kvm)) young |= kvm_tdp_mmu_age_gfn_range(kvm, range); @@ -569,7 +569,7 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) bool young = false; if (kvm_memslots_have_rmaps(kvm)) - young = kvm_handle_gfn_range(kvm, range, kvm_test_age_rmap); + young = kvm_shadow_mmu_test_age_gfn(kvm, range); if (is_tdp_mmu_enabled(kvm)) young |= kvm_tdp_mmu_test_age_gfn(kvm, range); diff --git a/arch/x86/kvm/mmu/shadow_mmu.c b/arch/x86/kvm/mmu/shadow_mmu.c index 77472eb9b06a..1c6ff6fe3d2c 100644 --- a/arch/x86/kvm/mmu/shadow_mmu.c +++ b/arch/x86/kvm/mmu/shadow_mmu.c @@ -862,16 +862,16 @@ static bool __kvm_zap_rmap(struct kvm *kvm, struct kvm_rmap_head *rmap_head, return kvm_zap_all_rmap_sptes(kvm, rmap_head); } -bool kvm_zap_rmap(struct kvm *kvm, struct kvm_rmap_head *rmap_head, - struct kvm_memory_slot *slot, gfn_t gfn, int level, - pte_t unused) +static bool kvm_zap_rmap(struct kvm *kvm, struct kvm_rmap_head *rmap_head, + struct kvm_memory_slot *slot, gfn_t gfn, int level, + pte_t unused) { return __kvm_zap_rmap(kvm, rmap_head, slot); } -bool kvm_set_pte_rmap(struct kvm *kvm, struct kvm_rmap_head *rmap_head, - struct kvm_memory_slot *slot, gfn_t gfn, int level, - pte_t pte) +static bool kvm_set_pte_rmap(struct kvm *kvm, struct kvm_rmap_head *rmap_head, + struct kvm_memory_slot *slot, gfn_t gfn, int level, + pte_t pte) { u64 *sptep; struct rmap_iterator iter; @@ -978,9 +978,13 @@ static void slot_rmap_walk_next(struct slot_rmap_walk_iterator *iterator) slot_rmap_walk_okay(_iter_); \ slot_rmap_walk_next(_iter_)) -__always_inline bool kvm_handle_gfn_range(struct kvm *kvm, - struct kvm_gfn_range *range, - rmap_handler_t handler) +typedef bool (*rmap_handler_t)(struct kvm *kvm, struct kvm_rmap_head *rmap_head, + struct kvm_memory_slot *slot, gfn_t gfn, + int level, pte_t pte); + +static __always_inline bool +kvm_handle_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range, + rmap_handler_t handler) { struct slot_rmap_walk_iterator iterator; bool ret = false; @@ -993,9 +997,9 @@ __always_inline bool kvm_handle_gfn_range(struct kvm *kvm, return ret; } -bool kvm_age_rmap(struct kvm *kvm, struct kvm_rmap_head *rmap_head, - struct kvm_memory_slot *slot, gfn_t gfn, int level, - pte_t unused) +static bool kvm_age_rmap(struct kvm *kvm, struct kvm_rmap_head *rmap_head, + struct kvm_memory_slot *slot, gfn_t gfn, int level, + pte_t unused) { u64 *sptep; struct rmap_iterator iter; @@ -1007,9 +1011,9 @@ bool kvm_age_rmap(struct kvm *kvm, struct kvm_rmap_head *rmap_head, return young; } -bool kvm_test_age_rmap(struct kvm *kvm, struct kvm_rmap_head *rmap_head, - struct kvm_memory_slot *slot, gfn_t gfn, - int level, pte_t unused) +static bool kvm_test_age_rmap(struct kvm *kvm, struct kvm_rmap_head *rmap_head, + struct kvm_memory_slot *slot, gfn_t gfn, + int level, pte_t unused) { u64 *sptep; struct rmap_iterator iter; @@ -3508,3 +3512,23 @@ void kvm_shadow_mmu_wrprot_slot(struct kvm *kvm, slot_handle_level(kvm, memslot, slot_rmap_write_protect, start_level, KVM_MAX_HUGEPAGE_LEVEL, false); } + +bool kvm_shadow_mmu_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range) +{ + return kvm_handle_gfn_range(kvm, range, kvm_zap_rmap); +} + +bool kvm_shadow_mmu_set_spte_gfn(struct kvm *kvm, struct kvm_gfn_range *range) +{ + return kvm_handle_gfn_range(kvm, range, kvm_set_pte_rmap); +} + +bool kvm_shadow_mmu_age_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range) +{ + return kvm_handle_gfn_range(kvm, range, kvm_age_rmap); +} + +bool kvm_shadow_mmu_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) +{ + return kvm_handle_gfn_range(kvm, range, kvm_test_age_rmap); +} diff --git a/arch/x86/kvm/mmu/shadow_mmu.h b/arch/x86/kvm/mmu/shadow_mmu.h index 397fb463ef54..2ded3d674cb0 100644 --- a/arch/x86/kvm/mmu/shadow_mmu.h +++ b/arch/x86/kvm/mmu/shadow_mmu.h @@ -26,26 +26,6 @@ struct pte_list_desc { /* Only exported for debugfs.c. */ unsigned int pte_list_count(struct kvm_rmap_head *rmap_head); -bool kvm_zap_rmap(struct kvm *kvm, struct kvm_rmap_head *rmap_head, - struct kvm_memory_slot *slot, gfn_t gfn, int level, - pte_t unused); -bool kvm_set_pte_rmap(struct kvm *kvm, struct kvm_rmap_head *rmap_head, - struct kvm_memory_slot *slot, gfn_t gfn, int level, - pte_t pte); - -typedef bool (*rmap_handler_t)(struct kvm *kvm, struct kvm_rmap_head *rmap_head, - struct kvm_memory_slot *slot, gfn_t gfn, - int level, pte_t pte); -bool kvm_handle_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range, - rmap_handler_t handler); - -bool kvm_age_rmap(struct kvm *kvm, struct kvm_rmap_head *rmap_head, - struct kvm_memory_slot *slot, gfn_t gfn, int level, - pte_t unused); -bool kvm_test_age_rmap(struct kvm *kvm, struct kvm_rmap_head *rmap_head, - struct kvm_memory_slot *slot, gfn_t gfn, - int level, pte_t unused); - void __clear_sp_write_flooding_count(struct kvm_mmu_page *sp); bool __kvm_shadow_mmu_prepare_zap_page(struct kvm *kvm, struct kvm_mmu_page *sp, @@ -101,6 +81,11 @@ void kvm_shadow_mmu_wrprot_slot(struct kvm *kvm, const struct kvm_memory_slot *memslot, int start_level); +bool kvm_shadow_mmu_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range); +bool kvm_shadow_mmu_set_spte_gfn(struct kvm *kvm, struct kvm_gfn_range *range); +bool kvm_shadow_mmu_age_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range); +bool kvm_shadow_mmu_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range); + /* Exports from paging_tmpl.h */ gpa_t paging32_gva_to_gpa(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, gpa_t vaddr, u64 access, From patchwork Wed Dec 21 22:24:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ben Gardon X-Patchwork-Id: 13079202 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B2DC8C4332F for ; Wed, 21 Dec 2022 22:26:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235144AbiLUW0J (ORCPT ); Wed, 21 Dec 2022 17:26:09 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37884 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235090AbiLUWZj (ORCPT ); Wed, 21 Dec 2022 17:25:39 -0500 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 51E2F27B0E for ; Wed, 21 Dec 2022 14:24:46 -0800 (PST) Received: by mail-pj1-x1049.google.com with SMTP id o11-20020a17090a9f8b00b00225b041ba39so254115pjp.7 for ; Wed, 21 Dec 2022 14:24:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=4TdMmKeh9xdWHOqsXuzcwDMeXjRCekoXNuBTvbORk4Y=; b=KtV0pVIMfC+oSFEy+6xnaTAWg/otjtJQ3S9SFSJiaqziSxkFyLCWYd++OxLcdI3jLy 4C3EcV/Ykpd+HnLaclxh7dAhhVAMH+ziDlp47PaFQ91e+QjdlZFuVFPdGMEr8SQDDGbx 4/Qpcj76ezTxKGP/PoSZw5IxJNR4zFVMJw+5YccygGPHkQL/5G9lr6mY+qVDAXiRMPvO ySvjpodRibFvIKqH+VUZxLjHX8bpBBDQMz2OR1ENE/Bah4ZAlIy3N86pUGBWVap6Wjn7 WB8Yrqmpcq1Rq2w5AxKxOvBTJUTDeI9MXknD17Fi+uhVGb3ypwV5HytB2WWzpjdGu1z2 BZTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=4TdMmKeh9xdWHOqsXuzcwDMeXjRCekoXNuBTvbORk4Y=; b=zxLFy8mW38uxBnTWwoZ/WExLcn4LPkcSyYmPrwE7pHN3k65ugtCs4a4mYq0NSl5UsG 7jKh087zQJIbR/veiuGYUcwaVECbccz6OzNA99VviS7uIosRy9xIrYUoQZWsevQHn5MR Ji1Kj9VSktASCB9yMbRHRG5JEB7UQC6DQhrOr7HOIKgjFhcGmEmDYVfI/lM1FVyEgNv+ GXrOBCQ5Er64MM63mnulOrAK8k7yKSGospnIYzhJUeT0V10liS/EUTQ8jM4eFMrJiVqD t4ITi1yoyoELWeiKq02/4cz5ZZiL2RF5t72QAfw5liCwg+6XnfOJcOBiAp12ReVaqaEf Ybbg== X-Gm-Message-State: AFqh2kr+qW3JyZ/fhddhSyb+7Cp2esupmKi6xMbfNwghNTov7CSvMUb+ 6pXE6QOJBxzp27qxgQrW/SQCsFnV2KfV X-Google-Smtp-Source: AMrXdXvNqVXX6zU/V4HwnaQF1OmpHJ80B/IXAmbw1M/S+cN3+VYASEke/ukeHnemFxM/uBOEi3zro3UdIF9T X-Received: from sweer.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:e45]) (user=bgardon job=sendgmr) by 2002:a17:902:ab4e:b0:189:5c50:ce5 with SMTP id ij14-20020a170902ab4e00b001895c500ce5mr204757plb.14.1671661485557; Wed, 21 Dec 2022 14:24:45 -0800 (PST) Date: Wed, 21 Dec 2022 22:24:18 +0000 In-Reply-To: <20221221222418.3307832-1-bgardon@google.com> Mime-Version: 1.0 References: <20221221222418.3307832-1-bgardon@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20221221222418.3307832-15-bgardon@google.com> Subject: [RFC 14/14] KVM: x86/MMU: Add kvm_shadow_mmu_ to the last few functions in shadow_mmu.h From: Ben Gardon To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: Paolo Bonzini , Peter Xu , Sean Christopherson , David Matlack , Vipin Sharma , Nagareddy Reddy , Ben Gardon Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Fix up the names of the last few Shadow MMU functions in shadow_mmu.h. This gives a clean and obvious interface between the shared x86 MMU code and the Shadow MMU. There are still a few functions exported from paging_tmpl.h that are left as-is, but changing those will need to be done separately, if at all. No functional change intended. Signed-off-by: Ben Gardon --- arch/x86/kvm/mmu/mmu.c | 23 ++++++++++-------- arch/x86/kvm/mmu/shadow_mmu.c | 44 +++++++++++++++++++---------------- arch/x86/kvm/mmu/shadow_mmu.h | 16 +++++++------ 3 files changed, 46 insertions(+), 37 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index ceb3146016d0..8f3b96af470d 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -921,9 +921,11 @@ static int fast_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) u64 new_spte; if (is_tdp_mmu(vcpu->arch.mmu)) - sptep = kvm_tdp_mmu_fast_pf_get_last_sptep(vcpu, fault->addr, &spte); + sptep = kvm_tdp_mmu_fast_pf_get_last_sptep(vcpu, + fault->addr, &spte); else - sptep = fast_pf_get_last_sptep(vcpu, fault->addr, &spte); + sptep = kvm_shadow_mmu_fast_pf_get_last_sptep(vcpu, + fault->addr, &spte); if (!is_shadow_present_pte(spte)) break; @@ -1113,7 +1115,7 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *vcpu) root = kvm_tdp_mmu_get_vcpu_root_hpa(vcpu); mmu->root.hpa = root; } else if (shadow_root_level >= PT64_ROOT_4LEVEL) { - root = mmu_alloc_root(vcpu, 0, 0, shadow_root_level); + root = kvm_shadow_mmu_alloc_root(vcpu, 0, 0, shadow_root_level); mmu->root.hpa = root; } else if (shadow_root_level == PT32E_ROOT_LEVEL) { if (WARN_ON_ONCE(!mmu->pae_root)) { @@ -1124,8 +1126,8 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *vcpu) for (i = 0; i < 4; ++i) { WARN_ON_ONCE(IS_VALID_PAE_ROOT(mmu->pae_root[i])); - root = mmu_alloc_root(vcpu, i << (30 - PAGE_SHIFT), 0, - PT32_ROOT_LEVEL); + root = kvm_shadow_mmu_alloc_root(vcpu, + i << (30 - PAGE_SHIFT), 0, PT32_ROOT_LEVEL); mmu->pae_root[i] = root | PT_PRESENT_MASK | shadow_me_value; } @@ -1665,7 +1667,7 @@ void kvm_mmu_new_pgd(struct kvm_vcpu *vcpu, gpa_t new_pgd) * count. Otherwise, clear the write flooding count. */ if (!new_role.direct) - __clear_sp_write_flooding_count( + kvm_shadow_mmu_clear_sp_write_flooding_count( to_shadow_page(vcpu->arch.mmu->root.hpa)); } EXPORT_SYMBOL_GPL(kvm_mmu_new_pgd); @@ -2447,13 +2449,13 @@ int kvm_mmu_load(struct kvm_vcpu *vcpu) r = mmu_topup_memory_caches(vcpu, !vcpu->arch.mmu->root_role.direct); if (r) goto out; - r = mmu_alloc_special_roots(vcpu); + r = kvm_shadow_mmu_alloc_special_roots(vcpu); if (r) goto out; if (vcpu->arch.mmu->root_role.direct) r = mmu_alloc_direct_roots(vcpu); else - r = mmu_alloc_shadow_roots(vcpu); + r = kvm_shadow_mmu_alloc_shadow_roots(vcpu); if (r) goto out; @@ -2679,7 +2681,8 @@ static int __kvm_mmu_create(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu) * generally doesn't use PAE paging and can skip allocating the PDP * table. The main exception, handled here, is SVM's 32-bit NPT. The * other exception is for shadowing L1's 32-bit or PAE NPT on 64-bit - * KVM; that horror is handled on-demand by mmu_alloc_special_roots(). + * KVM; that horror is handled on-demand by + * kvm_shadow_mmu_alloc_special_roots(). */ if (tdp_enabled && kvm_mmu_get_tdp_level(vcpu) > PT32E_ROOT_LEVEL) return 0; @@ -2820,7 +2823,7 @@ int kvm_mmu_init_vm(struct kvm *kvm) if (r < 0) return r; - node->track_write = kvm_mmu_pte_write; + node->track_write = kvm_shadow_mmu_pte_write; node->track_flush_slot = kvm_mmu_invalidate_zap_pages_in_memslot; kvm_page_track_register_notifier(kvm, node); diff --git a/arch/x86/kvm/mmu/shadow_mmu.c b/arch/x86/kvm/mmu/shadow_mmu.c index 1c6ff6fe3d2c..6f3e201af670 100644 --- a/arch/x86/kvm/mmu/shadow_mmu.c +++ b/arch/x86/kvm/mmu/shadow_mmu.c @@ -1402,14 +1402,14 @@ static int mmu_sync_children(struct kvm_vcpu *vcpu, struct kvm_mmu_page *parent, return 0; } -void __clear_sp_write_flooding_count(struct kvm_mmu_page *sp) +void kvm_shadow_mmu_clear_sp_write_flooding_count(struct kvm_mmu_page *sp) { atomic_set(&sp->write_flooding_count, 0); } static void clear_sp_write_flooding_count(u64 *spte) { - __clear_sp_write_flooding_count(sptep_to_sp(spte)); + kvm_shadow_mmu_clear_sp_write_flooding_count(sptep_to_sp(spte)); } /* @@ -1480,7 +1480,7 @@ static struct kvm_mmu_page *kvm_mmu_find_shadow_page(struct kvm *kvm, kvm_flush_remote_tlbs(kvm); } - __clear_sp_write_flooding_count(sp); + kvm_shadow_mmu_clear_sp_write_flooding_count(sp); goto out; } @@ -1605,12 +1605,13 @@ static union kvm_mmu_page_role kvm_mmu_child_role(u64 *sptep, bool direct, * Concretely, a 4-byte PDE consumes bits 31:22, while an 8-byte PDE * consumes bits 29:21. To consume bits 31:30, KVM's uses 4 shadow * PDPTEs; those 4 PAE page directories are pre-allocated and their - * quadrant is assigned in mmu_alloc_root(). A 4-byte PTE consumes - * bits 21:12, while an 8-byte PTE consumes bits 20:12. To consume - * bit 21 in the PTE (the child here), KVM propagates that bit to the - * quadrant, i.e. sets quadrant to '0' or '1'. The parent 8-byte PDE - * covers bit 21 (see above), thus the quadrant is calculated from the - * _least_ significant bit of the PDE index. + * quadrant is assigned in kvm_shadow_mmu_alloc_root(). + * A 4-byte PTE consumes bits 21:12, while an 8-byte PTE consumes + * bits 20:12. To consume bit 21 in the PTE (the child here), KVM + * propagates that bit to the quadrant, i.e. sets quadrant to + * '0' or '1'. The parent 8-byte PDE covers bit 21 (see above), thus + * the quadrant is calculated from the _least_ significant bit of the + * PDE index. */ if (role.has_4_byte_gpte) { WARN_ON_ONCE(role.level != PG_LEVEL_4K); @@ -2377,7 +2378,8 @@ int kvm_shadow_mmu_direct_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *faul * - Must be called between walk_shadow_page_lockless_{begin,end}. * - The returned sptep must not be used after walk_shadow_page_lockless_end. */ -u64 *fast_pf_get_last_sptep(struct kvm_vcpu *vcpu, gpa_t gpa, u64 *spte) +u64 *kvm_shadow_mmu_fast_pf_get_last_sptep(struct kvm_vcpu *vcpu, gpa_t gpa, + u64 *spte) { struct kvm_shadow_walk_iterator iterator; u64 old_spte; @@ -2430,7 +2432,8 @@ static int mmu_check_root(struct kvm_vcpu *vcpu, gfn_t root_gfn) return ret; } -hpa_t mmu_alloc_root(struct kvm_vcpu *vcpu, gfn_t gfn, int quadrant, u8 level) +hpa_t kvm_shadow_mmu_alloc_root(struct kvm_vcpu *vcpu, gfn_t gfn, int quadrant, + u8 level) { union kvm_mmu_page_role role = vcpu->arch.mmu->root_role; struct kvm_mmu_page *sp; @@ -2447,7 +2450,7 @@ hpa_t mmu_alloc_root(struct kvm_vcpu *vcpu, gfn_t gfn, int quadrant, u8 level) return __pa(sp->spt); } -static int mmu_first_shadow_root_alloc(struct kvm *kvm) +static int kvm_shadow_mmu_first_shadow_root_alloc(struct kvm *kvm) { struct kvm_memslots *slots; struct kvm_memory_slot *slot; @@ -2508,7 +2511,7 @@ static int mmu_first_shadow_root_alloc(struct kvm *kvm) return r; } -int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) +int kvm_shadow_mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) { struct kvm_mmu *mmu = vcpu->arch.mmu; u64 pdptrs[4], pm_mask; @@ -2537,7 +2540,7 @@ int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) } } - r = mmu_first_shadow_root_alloc(vcpu->kvm); + r = kvm_shadow_mmu_first_shadow_root_alloc(vcpu->kvm); if (r) return r; @@ -2551,8 +2554,8 @@ int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) * write-protect the guests page table root. */ if (mmu->cpu_role.base.level >= PT64_ROOT_4LEVEL) { - root = mmu_alloc_root(vcpu, root_gfn, 0, - mmu->root_role.level); + root = kvm_shadow_mmu_alloc_root(vcpu, root_gfn, 0, + mmu->root_role.level); mmu->root.hpa = root; goto set_root_pgd; } @@ -2605,7 +2608,8 @@ int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) */ quadrant = (mmu->cpu_role.base.level == PT32_ROOT_LEVEL) ? i : 0; - root = mmu_alloc_root(vcpu, root_gfn, quadrant, PT32_ROOT_LEVEL); + root = kvm_shadow_mmu_alloc_root(vcpu, root_gfn, quadrant, + PT32_ROOT_LEVEL); mmu->pae_root[i] = root | pm_mask; } @@ -2624,7 +2628,7 @@ int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) return r; } -int mmu_alloc_special_roots(struct kvm_vcpu *vcpu) +int kvm_shadow_mmu_alloc_special_roots(struct kvm_vcpu *vcpu) { struct kvm_mmu *mmu = vcpu->arch.mmu; bool need_pml5 = mmu->root_role.level > PT64_ROOT_4LEVEL; @@ -2997,8 +3001,8 @@ static u64 *get_written_sptes(struct kvm_mmu_page *sp, gpa_t gpa, int *nspte) return spte; } -void kvm_mmu_pte_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new, - int bytes, struct kvm_page_track_notifier_node *node) +void kvm_shadow_mmu_pte_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new, + int bytes, struct kvm_page_track_notifier_node *node) { gfn_t gfn = gpa >> PAGE_SHIFT; struct kvm_mmu_page *sp; diff --git a/arch/x86/kvm/mmu/shadow_mmu.h b/arch/x86/kvm/mmu/shadow_mmu.h index 2ded3d674cb0..a3e6daa36236 100644 --- a/arch/x86/kvm/mmu/shadow_mmu.h +++ b/arch/x86/kvm/mmu/shadow_mmu.h @@ -26,7 +26,7 @@ struct pte_list_desc { /* Only exported for debugfs.c. */ unsigned int pte_list_count(struct kvm_rmap_head *rmap_head); -void __clear_sp_write_flooding_count(struct kvm_mmu_page *sp); +void kvm_shadow_mmu_clear_sp_write_flooding_count(struct kvm_mmu_page *sp); bool __kvm_shadow_mmu_prepare_zap_page(struct kvm *kvm, struct kvm_mmu_page *sp, struct list_head *invalid_list, @@ -41,17 +41,19 @@ int kvm_shadow_mmu_make_pages_available(struct kvm_vcpu *vcpu); int kvm_shadow_mmu_unprotect_page_virt(struct kvm_vcpu *vcpu, gva_t gva); int kvm_shadow_mmu_direct_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault); -u64 *fast_pf_get_last_sptep(struct kvm_vcpu *vcpu, gpa_t gpa, u64 *spte); +u64 *kvm_shadow_mmu_fast_pf_get_last_sptep(struct kvm_vcpu *vcpu, gpa_t gpa, + u64 *spte); -hpa_t mmu_alloc_root(struct kvm_vcpu *vcpu, gfn_t gfn, int quadrant, u8 level); -int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu); -int mmu_alloc_special_roots(struct kvm_vcpu *vcpu); +hpa_t kvm_shadow_mmu_alloc_root(struct kvm_vcpu *vcpu, gfn_t gfn, int quadrant, + u8 level); +int kvm_shadow_mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu); +int kvm_shadow_mmu_alloc_special_roots(struct kvm_vcpu *vcpu); int kvm_shadow_mmu_get_walk(struct kvm_vcpu *vcpu, u64 addr, u64 *sptes, int *root_level); -void kvm_mmu_pte_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new, - int bytes, struct kvm_page_track_notifier_node *node); +void kvm_shadow_mmu_pte_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new, + int bytes, struct kvm_page_track_notifier_node *node); void kvm_shadow_mmu_zap_obsolete_pages(struct kvm *kvm); bool kvm_shadow_mmu_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end);