From patchwork Mon Apr 1 23:29:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13613118 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B60A8CD1288 for ; Mon, 1 Apr 2024 23:30:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=Bi4V3H23aI59Udt30bTZgX73HxDkKeTJOm1EifvuQfw=; b=VcKzNMoyGOWDj21acQzM6C0A7T sKEic+3OlqAbpku7DUqCZF89UFBjmFnL69kD34+LhZeL2xvcswxh4+5Ep+d7z5OOdkl2Szj926Uh7 PtGicjKx3JPDy4frSqver+BBbjl8OoCGGk3dK8hhS5K9YVVTrxRNEzLbKBHfgGZt8n5IebwJGdLyE 74CVnm9ADxzZVDcwiRfOYcheCT61QzYGGPQl917UKLE4WSktP7n9+gcCRd2Tr3v+x52tyhKPe8p5u QJ7R32CS4AX9hWD6HPz8WvMBz7D+diB/2pmqysFUFi4hUQ/UofRos9AK+M808bUfpTUMoCvut05NF OSw2Nq5w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rrR6R-000000097PZ-3Y9G; Mon, 01 Apr 2024 23:30:12 +0000 Received: from mail-ua1-x94a.google.com ([2607:f8b0:4864:20::94a]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rrR6I-000000097Gl-0qqD for linux-arm-kernel@lists.infradead.org; Mon, 01 Apr 2024 23:30:04 +0000 Received: by mail-ua1-x94a.google.com with SMTP id a1e0cc1a2514c-7e05948d6e0so1710271241.2 for ; Mon, 01 Apr 2024 16:30:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1712014201; x=1712619001; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=PXGb1c3s19z+8U24R5nbxlFRwqU1gq1c8Sy/WKQX1Zo=; b=O45NTnQ0OcqcXOVH5JVjrjbQw7gWg5KWU59aTvsZd9mj/ExrWJbzzoRCbdZVtxt3ZX C9dZgstJBySt2NOeTK0899vpqxO5949TedvzicxafQcSqpLioA+bKCYqp4/5ScJRxODb bpZXmhPy6JxWSbgT8n6/j52VoXda6MgWu+s5cnVUcqhf8JnTCOfJvbc+EQ/ywoVOT3te vAPLwN4lVElkoBdnWwswnjzzU1Lqfv4wlTogYC6QnXrFbFwR9RXCglch/lFf/XDYEnce Jz8wjFkszUGsl+Xv07XlPWX2HXmnaBOuRkTLf+Wc7QgHUFvbsUHoAjvay8HebEnAorUo kuhg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712014201; x=1712619001; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=PXGb1c3s19z+8U24R5nbxlFRwqU1gq1c8Sy/WKQX1Zo=; b=jrvyEgyV2bZTej00FiUho+0O0Axb7MvpGvO9Cn7enIgUP1obAaZl0PEsPmtXKDhjW1 fEUk7HAnLSRMOiOODrNUwAocRhCyaEG4+1THgkEFL9r7UncdlnFLJ4sNCNr9ZHwLpPfm AaCgVlJV4ulSIJpuwbYKANbPKKNzDt4hXli8cFrSMxH4gtUfkUbdLCG2SpibrnSU+xlC EJ9mb2DV9TSari6BUlgbnzfxjc0zo3vvuEI5/UsSQA11d5x3EcfJvMhTaWijaEUkXpuR Jy1nEqVpFRtkX7AhCyec0TiKWyz4TidqfFEtju75exZozb2LRJ1zPZpzs+Cd+WuFvT/s QrnA== X-Forwarded-Encrypted: i=1; AJvYcCU3Zu0UMMv87e3wvmiVSJznB4KHk2f6/jiJuRweoIh08zcxFISbjGuxRKR6sSYr4kWOb/X/hhLPPyyfQ4HRIYI52HfB0fgLAPSp/j3P8Os5B6aGeDY= X-Gm-Message-State: AOJu0YwLZu435LbO8QsDdpDmx40Jy3OKs6U/sH1xfUb8ZZGSvg7XWJr8 TzR1Bs69RBtGDSAT3nOh4II+JI8mamj6XGD0NMalvbX6HFgzS8UeBR6kzBrhJPYcIfGjJsiyDWn RM5mF4UHPh6dzFXkMUw== X-Google-Smtp-Source: AGHT+IHiAeh9QdQ8JIxLLHU+30Xn5sQK+tQdEgkzewt+NNIBZBtyidDAUXciX6U3q2UCp3x2qiBYdkguFZKdqu9L X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:ab0:3792:0:b0:7e3:297b:9df1 with SMTP id d18-20020ab03792000000b007e3297b9df1mr67902uav.1.1712014200934; Mon, 01 Apr 2024 16:30:00 -0700 (PDT) Date: Mon, 1 Apr 2024 23:29:44 +0000 In-Reply-To: <20240401232946.1837665-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240401232946.1837665-1-jthoughton@google.com> X-Mailer: git-send-email 2.44.0.478.gd926399ef9-goog Message-ID: <20240401232946.1837665-6-jthoughton@google.com> Subject: [PATCH v3 5/7] KVM: x86: Participate in bitmap-based PTE aging From: James Houghton To: Andrew Morton , Paolo Bonzini Cc: Yu Zhao , David Matlack , Marc Zyngier , Oliver Upton , Sean Christopherson , Jonathan Corbet , James Morse , Suzuki K Poulose , Zenghui Yu , Catalin Marinas , Will Deacon , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers , Shaoqin Huang , Gavin Shan , Ricardo Koller , Raghavendra Rao Ananta , Ryan Roberts , David Rientjes , Axel Rasmussen , linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, James Houghton X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240401_163002_716094_F193191A X-CRM114-Status: GOOD ( 18.80 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Only handle the TDP MMU case for now. In other cases, if a bitmap was not provided, fallback to the slowpath that takes mmu_lock, or, if a bitmap was provided, inform the caller that the bitmap is unreliable. Suggested-by: Yu Zhao Signed-off-by: James Houghton --- arch/x86/include/asm/kvm_host.h | 14 ++++++++++++++ arch/x86/kvm/mmu/mmu.c | 16 ++++++++++++++-- arch/x86/kvm/mmu/tdp_mmu.c | 10 +++++++++- 3 files changed, 37 insertions(+), 3 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 3b58e2306621..c30918d0887e 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -2324,4 +2324,18 @@ int memslot_rmap_alloc(struct kvm_memory_slot *slot, unsigned long npages); */ #define KVM_EXIT_HYPERCALL_MBZ GENMASK_ULL(31, 1) +#define kvm_arch_prepare_bitmap_age kvm_arch_prepare_bitmap_age +static inline bool kvm_arch_prepare_bitmap_age(struct mmu_notifier *mn) +{ + /* + * Indicate that we support bitmap-based aging when using the TDP MMU + * and the accessed bit is available in the TDP page tables. + * + * We have no other preparatory work to do here, so we do not need to + * redefine kvm_arch_finish_bitmap_age(). + */ + return IS_ENABLED(CONFIG_X86_64) && tdp_mmu_enabled + && shadow_accessed_mask; +} + #endif /* _ASM_X86_KVM_HOST_H */ diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 992e651540e8..fae1a75750bb 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1674,8 +1674,14 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { bool young = false; - if (kvm_memslots_have_rmaps(kvm)) + if (kvm_memslots_have_rmaps(kvm)) { + if (range->lockless) { + kvm_age_set_unreliable(range); + return false; + } + young = kvm_handle_gfn_range(kvm, range, kvm_age_rmap); + } if (tdp_mmu_enabled) young |= kvm_tdp_mmu_age_gfn_range(kvm, range); @@ -1687,8 +1693,14 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { bool young = false; - if (kvm_memslots_have_rmaps(kvm)) + if (kvm_memslots_have_rmaps(kvm)) { + if (range->lockless) { + kvm_age_set_unreliable(range); + return false; + } + young = kvm_handle_gfn_range(kvm, range, kvm_test_age_rmap); + } if (tdp_mmu_enabled) young |= kvm_tdp_mmu_test_age_gfn(kvm, range); diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index d078157e62aa..edea01bc145f 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -1217,6 +1217,9 @@ static bool age_gfn_range(struct kvm *kvm, struct tdp_iter *iter, if (!is_accessed_spte(iter->old_spte)) return false; + if (!kvm_gfn_should_age(range, iter->gfn)) + return false; + if (spte_ad_enabled(iter->old_spte)) { iter->old_spte = tdp_mmu_clear_spte_bits(iter->sptep, iter->old_spte, @@ -1250,7 +1253,12 @@ bool kvm_tdp_mmu_age_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range) static bool test_age_gfn(struct kvm *kvm, struct tdp_iter *iter, struct kvm_gfn_range *range) { - return is_accessed_spte(iter->old_spte); + bool young = is_accessed_spte(iter->old_spte); + + if (young) + kvm_gfn_record_young(range, iter->gfn); + + return young; } bool kvm_tdp_mmu_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)