From patchwork Fri May 26 23:44:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13257439 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C3C4C77B7A for ; Fri, 26 May 2023 23:44:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 89B446B0071; Fri, 26 May 2023 19:44:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 84CB6900003; Fri, 26 May 2023 19:44:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6EC806B0074; Fri, 26 May 2023 19:44:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 605B86B0071 for ; Fri, 26 May 2023 19:44:45 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 33B7C160FC0 for ; Fri, 26 May 2023 23:44:45 +0000 (UTC) X-FDA: 80834038530.13.930F4C8 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf13.hostedemail.com (Postfix) with ESMTP id 68D9A2000D for ; Fri, 26 May 2023 23:44:42 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=hvb8rxkE; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf13.hostedemail.com: domain of 3aURxZAYKCOAaWbJCQIQQING.EQONKPWZ-OOMXCEM.QTI@flex--yuzhao.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3aURxZAYKCOAaWbJCQIQQING.EQONKPWZ-OOMXCEM.QTI@flex--yuzhao.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1685144682; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kGsSI0lPP8ERxd6p+DSa9eEPYGyuTkZRYkJGde6ThZo=; b=vuBwDxvwGsHoJYNSDZQgJ2teMjD5JFK1d55BeO9rv7RJrQYwSwQ1ZN5avb2jZGZooJCKRe AzlzbMnyD7/Uebp/AaVM++/9a6eAXNK1NSgC/BZ94iZhtyAKSCcPsrX1TdtFr9mIpB5XYz +MUR6nOGXZHpxNvU0sRJXh5hSHUi3iw= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=hvb8rxkE; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf13.hostedemail.com: domain of 3aURxZAYKCOAaWbJCQIQQING.EQONKPWZ-OOMXCEM.QTI@flex--yuzhao.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3aURxZAYKCOAaWbJCQIQQING.EQONKPWZ-OOMXCEM.QTI@flex--yuzhao.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1685144682; a=rsa-sha256; cv=none; b=D3q3wsw5oF3hBnb/YXdCXYsOkSA0R+AjyYHwYL/Uq1zEY+x9Z2p0ikZs7W8UCC5Mty+Oyk jHk0XDpl13Js0bt0ff8fsjcI7iFbEMV/YkKtlTt3/mOYMNhSp9B2TWBYiCeISLElLKUPX1 pQmHFN4CU+HVPOuj2Arh0DA6CEk/ns4= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-ba87bd29e9dso1727263276.3 for ; Fri, 26 May 2023 16:44:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1685144681; x=1687736681; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=kGsSI0lPP8ERxd6p+DSa9eEPYGyuTkZRYkJGde6ThZo=; b=hvb8rxkERCbIh1bZc0q4NRTJ+rJW/WU+U0AR8hQmLag3Ue+qSWO6cakjY3a2+aFQIE x+dhqQS7EQpNz5Tcg/879WfdtsT2g9m9IEV/ar3WsH6hpivs+AgK+n3cV9xivqmDDPOm KKmHGpOcNrwi3l5Yw+fAvneY5R+j+Mh2hjIBHhqkVzBeZTwwqArwApBbzG0OmYJUYGc6 DRb2e1iDRkrr5VQNcGKChaSGvbQt6NfDiW4utZo1zEzUWIhsS0vIBmPt/jCICfYU7AGj 2EXBMubac0epYeTWkMmnp5dg4awgt7i7k/gPemJKnVVjNCrCZqeAJZJODgbRp4LsgzpY rjvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685144681; x=1687736681; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=kGsSI0lPP8ERxd6p+DSa9eEPYGyuTkZRYkJGde6ThZo=; b=gKhBgjX0nuK1sirKyOWpU1firK3+sQ5KVUfOoFH9Xqka1MRs0yERAtM+Kuz1T40Qbg 2Q0dM+jkhEwYR0Wg11LbMUVBZJyIHPOb4nf6xhfBztajlBQiUyQUJevzUvGXE2zY2Vc7 X0vIzY0dxIf+Lx88iKY6RHXCHoqBnUSw2KQUMc7auxwMKbQzzVVVazB59pQN+edihpok jTowh0JOuWABp7ePTBYz3jxOXRJKiuVf0ZNSs2f92o/WbGMXnw0dAygHi1JEGxdpQHhk lMxc17wm+jEgZwXDTG7gg73q/sNpQ1XCySybuUVpyWiOfDBHf/QJ9sMBRC/9OMa70Hzg CqsA== X-Gm-Message-State: AC+VfDwDjPq+Jk2dFsIIAEHESoxjsSg+stg5kZ0WkNNKId/RJatsjYDI XKqNAsa0w5bXFRczAfQuYeykBUJt/yE= X-Google-Smtp-Source: ACHHUZ6fi8C05HO0+KMDQENWGjZ4c7vAIwn8WK8PM0ILwAQbwBWVWIEplGkrZ1gydvcndVbPZA/WyjMMkr8= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:910f:8a15:592b:2087]) (user=yuzhao job=sendgmr) by 2002:a05:6902:50d:b0:ba8:3e2d:58f8 with SMTP id x13-20020a056902050d00b00ba83e2d58f8mr1858255ybs.5.1685144681575; Fri, 26 May 2023 16:44:41 -0700 (PDT) Date: Fri, 26 May 2023 17:44:26 -0600 In-Reply-To: <20230526234435.662652-1-yuzhao@google.com> Message-Id: <20230526234435.662652-2-yuzhao@google.com> Mime-Version: 1.0 References: <20230526234435.662652-1-yuzhao@google.com> X-Mailer: git-send-email 2.41.0.rc0.172.g3f132b7071-goog Subject: [PATCH mm-unstable v2 01/10] mm/kvm: add mmu_notifier_ops->test_clear_young() From: Yu Zhao To: Andrew Morton , Paolo Bonzini Cc: Alistair Popple , Anup Patel , Ben Gardon , Borislav Petkov , Catalin Marinas , Chao Peng , Christophe Leroy , Dave Hansen , Fabiano Rosas , Gaosheng Cui , Gavin Shan , "H. Peter Anvin" , Ingo Molnar , James Morse , "Jason A. Donenfeld" , Jason Gunthorpe , Jonathan Corbet , Marc Zyngier , Masami Hiramatsu , Michael Ellerman , Michael Larabel , Mike Rapoport , Nicholas Piggin , Oliver Upton , Paul Mackerras , Peter Xu , Sean Christopherson , Steven Rostedt , Suzuki K Poulose , Thomas Gleixner , Thomas Huth , Will Deacon , Zenghui Yu , kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-trace-kernel@vger.kernel.org, x86@kernel.org, linux-mm@google.com, Yu Zhao X-Rspamd-Queue-Id: 68D9A2000D X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: scsxoe36ptq1exd9w6ojyibpkcyr9paj X-HE-Tag: 1685144682-930241 X-HE-Meta: U2FsdGVkX1+/wgFXi4sgSRjL1pdGVunzuu+B17IbFlK6v/MCMpMeZGT5a9yaHahexkpHjHohITq9GWajI18Dqaj6Wd9GtmNsyDDdJhcKwSdWgmxlvoMONT9ha8anT7vlnuPbvbZ03OJVmNzEZiaaiV2dZiGk+WQIWcL7hfl6DM36JzcQrw7tjqdTz6f0jMnsZzoZegd2ql3SMlGA08Hnl8LyTRXbwEKpVi/+WRGnpPcLi9iQu+UGesijUTSKI52GWFd6pHe2AWDTYDmSyFF3QfXPot+XPI02KknTl6kExkOmsQ1WGGYDcLncVsxXajxhvWuWu+b+ocWa4qcz8OhHaWxPYXw2KKZDnGm8v+/Rs4OvkSN7ymLMfSsUZz8hMKkjlDpGOtU5oTGqbrQR5C3k01OXLmgRfi2BvkJb45b6/cDoGxArw6UWeTiFlCpEDzimBXgz+Gr8OGKe5C0rM0YK2PoQ/UTiyoj7hsUmUcp2zVnsgmoepLWXbD/w5SzbfCya1u1AKQ1TqHXriRmCrewKtRAQa9j3HCqaJyL+q30XiLlxj3jrfx/SWwAMEVD2qvClGPT13FfWBB3OpqBSV4o+85k38fOcr9Gp0+VDZmGx81vUvilrRkbd9e8OPjtlZlwAdVfKzlLwY4THumeyanTCx7vodOeNSKnrYD5i+jAMOkulvJ383+0zYupPHDWQIEWhJ4Q2+1u46x2+q621OsicbFG3FaoJO413f5vK4zDI8YCliCztypEiP/wUzJTEo0EGkvf5//EpxJM1lZ1/tMN3WUM0VCYAwujtacO8vkGl0R58Iq5Axhh5Uk6rHDWmWWFxQEFqXAeQY71d+lrEXI86QJq75LPbc26jwTnKcx0aWtuXgGoHUAqvgKs9ualfogNlpnH3jEd9xCuaF0w3DKRrdzakaeNPQiTWiY5LpkNMVMk2++DX3kZwbjMmJmQ2EFOix7+FyuxWZ16xL6CqnHs EAZOnvkg DcnhHs74o4vFGERclI2C9+aExZphszfZC7DrKFjd/ANYx02MVr3fCawl3pAACLwhEEtCoAn30w4WiyYDdGjq71WRmyXm1ZLSOchZHLMKUZX36hx0cCb+2A7qEg6RpPSES46kWdZ5udxpYEIr+KK6VqvY/HoC+64Ov9Y9fQImCYgviApVpJZLY5na2HwsVWHWllE6HOR8/zjtDSYg7H/c0gWOlEJ5YBEb88nJ0Q0C1oG2nIxnkT4QuJ5+WW0RSvUmqZVbVNtfdVdr24i3UaSr+2xliMUEtu6eWdisIz4W5C0LdRTmNAVREPvVsiDedNdUrycUs/0gM1fGXG9kPRwdte8a7O10DXrNGiicVyBMHcTSvlK/pmBB8hM6MnrpEOxdRYna/QWzP1SSRLXRS7VJej77imbqmzok8PvqQrrrdjlhUFVj2wec5LccPeJ3odKmTbReTa0XHrGrOVLoHRDjCwZYPNjY2iy5h1BYtsnmctT+CKFn1jYVkRRGuepfmDMa6njV138llGS/B8qpVB54NWKnHVv+e9Y3KUOeHdv8fPo3dRI1mpf9R3PN89nNes72AYoLsApeTnRaGdC7A8AxmP/lfr7TDrNp8SxO5Pqx+JWw2cX6tQEqdlgKqJJK5vUCgZwFnOfoAdHD0JeHvmMy8u81Aew== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add mmu_notifier_ops->test_clear_young() to supersede test_young() and clear_young(). test_clear_young() has a fast path, which if supported, allows its callers to safely clear the accessed bit without taking kvm->mmu_lock. The fast path requires arch-specific code that generally relies on RCU and CAS: the former protects KVM page tables from being freed while the latter clears the accessed bit atomically against both the hardware and other software page table walkers. If the fast path is unsupported, test_clear_young() falls back to the existing slow path where kvm->mmu_lock is then taken. test_clear_young() can also operate on a range of KVM PTEs individually according to a bitmap, if the caller provides it. Signed-off-by: Yu Zhao --- include/linux/kvm_host.h | 22 +++++++++++ include/linux/mmu_notifier.h | 52 ++++++++++++++++++++++++ mm/mmu_notifier.c | 24 ++++++++++++ virt/kvm/kvm_main.c | 76 +++++++++++++++++++++++++++++++++++- 4 files changed, 173 insertions(+), 1 deletion(-) diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 0e571e973bc2..374262545f96 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -258,6 +258,7 @@ int kvm_async_pf_wakeup_all(struct kvm_vcpu *vcpu); #ifdef KVM_ARCH_WANT_MMU_NOTIFIER struct kvm_gfn_range { struct kvm_memory_slot *slot; + void *args; gfn_t start; gfn_t end; pte_t pte; @@ -267,6 +268,27 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range); bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range); bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range); bool kvm_set_spte_gfn(struct kvm *kvm, struct kvm_gfn_range *range); +bool kvm_should_clear_young(struct kvm_gfn_range *range, gfn_t gfn); +bool kvm_arch_test_clear_young(struct kvm *kvm, struct kvm_gfn_range *range); +#endif + +/* + * Architectures that implement kvm_arch_test_clear_young() should override + * kvm_arch_has_test_clear_young(). + * + * kvm_arch_has_test_clear_young() is allowed to return false positive, i.e., it + * can return true if kvm_arch_test_clear_young() is supported but disabled due + * to some runtime constraint. In this case, kvm_arch_test_clear_young() should + * return true; otherwise, it should return false. + * + * For each young KVM PTE, kvm_arch_test_clear_young() should call + * kvm_should_clear_young() to decide whether to clear the accessed bit. + */ +#ifndef kvm_arch_has_test_clear_young +static inline bool kvm_arch_has_test_clear_young(void) +{ + return false; +} #endif enum { diff --git a/include/linux/mmu_notifier.h b/include/linux/mmu_notifier.h index 64a3e051c3c4..dfdbb370682d 100644 --- a/include/linux/mmu_notifier.h +++ b/include/linux/mmu_notifier.h @@ -60,6 +60,8 @@ enum mmu_notifier_event { }; #define MMU_NOTIFIER_RANGE_BLOCKABLE (1 << 0) +#define MMU_NOTIFIER_RANGE_LOCKLESS (1 << 1) +#define MMU_NOTIFIER_RANGE_YOUNG (1 << 2) struct mmu_notifier_ops { /* @@ -122,6 +124,10 @@ struct mmu_notifier_ops { struct mm_struct *mm, unsigned long address); + int (*test_clear_young)(struct mmu_notifier *mn, struct mm_struct *mm, + unsigned long start, unsigned long end, + bool clear, unsigned long *bitmap); + /* * change_pte is called in cases that pte mapping to page is changed: * for example, when ksm remaps pte to point to a new shared page. @@ -392,6 +398,9 @@ extern int __mmu_notifier_clear_young(struct mm_struct *mm, unsigned long end); extern int __mmu_notifier_test_young(struct mm_struct *mm, unsigned long address); +extern int __mmu_notifier_test_clear_young(struct mm_struct *mm, + unsigned long start, unsigned long end, + bool clear, unsigned long *bitmap); extern void __mmu_notifier_change_pte(struct mm_struct *mm, unsigned long address, pte_t pte); extern int __mmu_notifier_invalidate_range_start(struct mmu_notifier_range *r); @@ -440,6 +449,35 @@ static inline int mmu_notifier_test_young(struct mm_struct *mm, return 0; } +/* + * mmu_notifier_test_clear_young() returns nonzero if any of the KVM PTEs within + * a given range was young. Specifically, it returns MMU_NOTIFIER_RANGE_LOCKLESS + * if the fast path was successful, MMU_NOTIFIER_RANGE_YOUNG otherwise. + * + * The last parameter to the function is a bitmap and only the fast path + * supports it: if it is NULL, the function falls back to the slow path if the + * fast path was unsuccessful; otherwise, the function bails out. + * + * The bitmap has the following specifications: + * 1. The number of bits should be at least (end-start)/PAGE_SIZE. + * 2. The offset of each bit should be relative to the end, i.e., the offset + * corresponding to addr should be (end-addr)/PAGE_SIZE-1. This is convenient + * for batching while forward looping. + * + * When testing, this function sets the corresponding bit in the bitmap for each + * young KVM PTE. When clearing, this function clears the accessed bit for each + * young KVM PTE whose corresponding bit in the bitmap is set. + */ +static inline int mmu_notifier_test_clear_young(struct mm_struct *mm, + unsigned long start, unsigned long end, + bool clear, unsigned long *bitmap) +{ + if (mm_has_notifiers(mm)) + return __mmu_notifier_test_clear_young(mm, start, end, clear, bitmap); + + return 0; +} + static inline void mmu_notifier_change_pte(struct mm_struct *mm, unsigned long address, pte_t pte) { @@ -684,12 +722,26 @@ static inline int mmu_notifier_clear_flush_young(struct mm_struct *mm, return 0; } +static inline int mmu_notifier_clear_young(struct mm_struct *mm, + unsigned long start, + unsigned long end) +{ + return 0; +} + static inline int mmu_notifier_test_young(struct mm_struct *mm, unsigned long address) { return 0; } +static inline int mmu_notifier_test_clear_young(struct mm_struct *mm, + unsigned long start, unsigned long end, + bool clear, unsigned long *bitmap) +{ + return 0; +} + static inline void mmu_notifier_change_pte(struct mm_struct *mm, unsigned long address, pte_t pte) { diff --git a/mm/mmu_notifier.c b/mm/mmu_notifier.c index 50c0dde1354f..7e6aba4bddcb 100644 --- a/mm/mmu_notifier.c +++ b/mm/mmu_notifier.c @@ -424,6 +424,30 @@ int __mmu_notifier_test_young(struct mm_struct *mm, return young; } +int __mmu_notifier_test_clear_young(struct mm_struct *mm, + unsigned long start, unsigned long end, + bool clear, unsigned long *bitmap) +{ + int idx; + struct mmu_notifier *mn; + int young = 0; + + idx = srcu_read_lock(&srcu); + + hlist_for_each_entry_srcu(mn, &mm->notifier_subscriptions->list, hlist, + srcu_read_lock_held(&srcu)) { + if (mn->ops->test_clear_young) + young |= mn->ops->test_clear_young(mn, mm, start, end, clear, bitmap); + + if (young && !clear) + break; + } + + srcu_read_unlock(&srcu, idx); + + return young; +} + void __mmu_notifier_change_pte(struct mm_struct *mm, unsigned long address, pte_t pte) { diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 51e4882d0873..31ee58754b19 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -541,6 +541,7 @@ typedef void (*on_lock_fn_t)(struct kvm *kvm, unsigned long start, typedef void (*on_unlock_fn_t)(struct kvm *kvm); struct kvm_hva_range { + void *args; unsigned long start; unsigned long end; pte_t pte; @@ -549,6 +550,7 @@ struct kvm_hva_range { on_unlock_fn_t on_unlock; bool flush_on_ret; bool may_block; + bool lockless; }; /* @@ -602,6 +604,8 @@ static __always_inline int __kvm_handle_hva_range(struct kvm *kvm, hva_end = min(range->end, slot->userspace_addr + (slot->npages << PAGE_SHIFT)); + gfn_range.args = range->args; + /* * To optimize for the likely case where the address * range is covered by zero or one memslots, don't @@ -619,7 +623,7 @@ static __always_inline int __kvm_handle_hva_range(struct kvm *kvm, gfn_range.end = hva_to_gfn_memslot(hva_end + PAGE_SIZE - 1, slot); gfn_range.slot = slot; - if (!locked) { + if (!range->lockless && !locked) { locked = true; KVM_MMU_LOCK(kvm); if (!IS_KVM_NULL_FN(range->on_lock)) @@ -628,6 +632,9 @@ static __always_inline int __kvm_handle_hva_range(struct kvm *kvm, break; } ret |= range->handler(kvm, &gfn_range); + + if (range->lockless && ret) + break; } } @@ -880,6 +887,72 @@ static int kvm_mmu_notifier_test_young(struct mmu_notifier *mn, kvm_test_age_gfn); } +struct test_clear_young_args { + unsigned long *bitmap; + unsigned long end; + bool clear; + bool young; +}; + +bool kvm_should_clear_young(struct kvm_gfn_range *range, gfn_t gfn) +{ + struct test_clear_young_args *args = range->args; + + VM_WARN_ON_ONCE(gfn < range->start || gfn >= range->end); + + args->young = true; + + if (args->bitmap) { + int offset = hva_to_gfn_memslot(args->end - 1, range->slot) - gfn; + + if (args->clear) + return test_bit(offset, args->bitmap); + + __set_bit(offset, args->bitmap); + } + + return args->clear; +} + +static int kvm_mmu_notifier_test_clear_young(struct mmu_notifier *mn, struct mm_struct *mm, + unsigned long start, unsigned long end, + bool clear, unsigned long *bitmap) +{ + struct kvm *kvm = mmu_notifier_to_kvm(mn); + struct kvm_hva_range range = { + .start = start, + .end = end, + .on_lock = (void *)kvm_null_fn, + .on_unlock = (void *)kvm_null_fn, + }; + + trace_kvm_age_hva(start, end); + + if (kvm_arch_has_test_clear_young()) { + struct test_clear_young_args args = { + .bitmap = bitmap, + .end = end, + .clear = clear, + }; + + range.args = &args; + range.lockless = true; + range.handler = kvm_arch_test_clear_young; + + if (!__kvm_handle_hva_range(kvm, &range)) + return args.young ? MMU_NOTIFIER_RANGE_LOCKLESS : 0; + } + + if (bitmap) + return 0; + + range.args = NULL; + range.lockless = false; + range.handler = clear ? kvm_age_gfn : kvm_test_age_gfn; + + return __kvm_handle_hva_range(kvm, &range) ? MMU_NOTIFIER_RANGE_YOUNG : 0; +} + static void kvm_mmu_notifier_release(struct mmu_notifier *mn, struct mm_struct *mm) { @@ -898,6 +971,7 @@ static const struct mmu_notifier_ops kvm_mmu_notifier_ops = { .clear_flush_young = kvm_mmu_notifier_clear_flush_young, .clear_young = kvm_mmu_notifier_clear_young, .test_young = kvm_mmu_notifier_test_young, + .test_clear_young = kvm_mmu_notifier_test_clear_young, .change_pte = kvm_mmu_notifier_change_pte, .release = kvm_mmu_notifier_release, }; From patchwork Fri May 26 23:44:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13257440 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B582EC7EE2C for ; Fri, 26 May 2023 23:44:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8C389900003; Fri, 26 May 2023 19:44:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8239C280001; Fri, 26 May 2023 19:44:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 69DD3900004; Fri, 26 May 2023 19:44:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 569BF900003 for ; Fri, 26 May 2023 19:44:46 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 24F8F40FBD for ; Fri, 26 May 2023 23:44:46 +0000 (UTC) X-FDA: 80834038572.06.E821FBE Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf14.hostedemail.com (Postfix) with ESMTP id 4F2F710000B for ; Fri, 26 May 2023 23:44:44 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=HaeURiP8; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf14.hostedemail.com: domain of 3a0RxZAYKCOIcYdLESKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--yuzhao.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3a0RxZAYKCOIcYdLESKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--yuzhao.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1685144684; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=74b92Wk/fESh4Us3jjAt9ER/dw3pjjcYNf6aM3EASmw=; b=Epe3oOD1HOfFasKpFMWMz7K0+Q1Nm9GTiYYEnGRQWfMD0vEog6ruhOJr4I7yqFZHYaHRgf Z58DQMB1EPSE/FRWJX/5ds81b//ZmlHi+mjysTXHVF5b/7oEkTtjoAA7CL5ns/pCCQcp15 TV+W2aBJUDkC1JLKrmR5GkGnD409b7o= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=HaeURiP8; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf14.hostedemail.com: domain of 3a0RxZAYKCOIcYdLESKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--yuzhao.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3a0RxZAYKCOIcYdLESKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--yuzhao.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1685144684; a=rsa-sha256; cv=none; b=ftQ/Z1o2SxHIfGTDdoTCRPNgz3I7CHEQZZV3yVH4YLtJ4CCKdpvso7mXQrIOl0bZyq9Fws EqrNBDbbZCTUQTehkd4yHHLV4+iv+yf5vWL5TdeAMhrxnLVQEBHJFf/HY18wCU6pBGwbxw TFe+kBeN5wPXviO1wYx0XvukM5snJi0= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-565d1b86a64so352957b3.3 for ; Fri, 26 May 2023 16:44:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1685144683; x=1687736683; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=74b92Wk/fESh4Us3jjAt9ER/dw3pjjcYNf6aM3EASmw=; b=HaeURiP8Vv+S5DfSxjNudO8jlQJ3LkpTBadHgzP5bqFZBs2b3foI2By3BUQ7HvtvPw W7QXj6uSkaA/QUNh4WGZeI/+nH65GkUhIUrM4lyZvIxfASIS2yU+XXbU6dLJN4ghefvO HpvDwxabNs2VhwI1FmYErw8clASpd/3GLJz6qPYetihMIY0W6vexsuS/yVGVZpRWw3il an2SbsIcj+SL6k3rb4VmBY4XZ/nf3i0WPF6L2efKh0GQc6ZIGr6GiKv97NKXrU9K5c92 qSgR2Y5nDu/cn+wzryf1LV/5CRQgKNqjczW+qgHgoPm+B6opAwXo+9yxImX5iYvtFZ8h Yz0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685144683; x=1687736683; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=74b92Wk/fESh4Us3jjAt9ER/dw3pjjcYNf6aM3EASmw=; b=Gw/OcgjcKPVyHW3HjO4rz2FkW9F5Xza1R4rqbLlgmKrZyy9WSR0IHRjwfjuOw/wiBm GXtbXm6e6HYit2Xhd6fxmcemyA5YWOx8j1OxlFaeoesSlI8Yc3TAzEiir2zD5Tn+X7O6 T5mqujl4rXbxFRfIpuAaQ3lIv8mKgmoTdJTJULrd4zaGdbI1TvaWJ8VzpUK7pco7JPES +MyxVDCNXRih0Xmh/cMj66nmh0fUWlbzcayN4KjkskCA4QvT5Ofk+gAt/bCht1ohNqvi 02mddZ0TTp2WM3nLjC3FJgxgukK1ktMFmNBGSa07Lir1i5KDBvvZmvJUxUp4HymmqRLR Oz6w== X-Gm-Message-State: AC+VfDxUnldYvOW/ylkSj5HLZ4eXzG/OH5BjPYhkrD7MNOghkZX4oJC2 osBdr1R03oQ6B9h2ID2MdEZsHc0mZfM= X-Google-Smtp-Source: ACHHUZ77t8VJ8lKvCLJIppCq4GXisPABEajtkBmVe9W7YGxSZPGguPwTv3qLn5OBBx4lDI9spcGEUr4VdcA= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:910f:8a15:592b:2087]) (user=yuzhao job=sendgmr) by 2002:a81:e608:0:b0:561:c4ef:1def with SMTP id u8-20020a81e608000000b00561c4ef1defmr2027484ywl.0.1685144683190; Fri, 26 May 2023 16:44:43 -0700 (PDT) Date: Fri, 26 May 2023 17:44:27 -0600 In-Reply-To: <20230526234435.662652-1-yuzhao@google.com> Message-Id: <20230526234435.662652-3-yuzhao@google.com> Mime-Version: 1.0 References: <20230526234435.662652-1-yuzhao@google.com> X-Mailer: git-send-email 2.41.0.rc0.172.g3f132b7071-goog Subject: [PATCH mm-unstable v2 02/10] mm/kvm: use mmu_notifier_ops->test_clear_young() From: Yu Zhao To: Andrew Morton , Paolo Bonzini Cc: Alistair Popple , Anup Patel , Ben Gardon , Borislav Petkov , Catalin Marinas , Chao Peng , Christophe Leroy , Dave Hansen , Fabiano Rosas , Gaosheng Cui , Gavin Shan , "H. Peter Anvin" , Ingo Molnar , James Morse , "Jason A. Donenfeld" , Jason Gunthorpe , Jonathan Corbet , Marc Zyngier , Masami Hiramatsu , Michael Ellerman , Michael Larabel , Mike Rapoport , Nicholas Piggin , Oliver Upton , Paul Mackerras , Peter Xu , Sean Christopherson , Steven Rostedt , Suzuki K Poulose , Thomas Gleixner , Thomas Huth , Will Deacon , Zenghui Yu , kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-trace-kernel@vger.kernel.org, x86@kernel.org, linux-mm@google.com, Yu Zhao X-Rspamd-Queue-Id: 4F2F710000B X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: ggmzopx9hyecmkbrhkethei9sgarna8t X-HE-Tag: 1685144684-830906 X-HE-Meta: U2FsdGVkX1+mEQ3Fsskp/VW0KwdDjEKRySZ/snTVjMWwdS17uWAFj7t5/UaocLBlr+Kfa5F4PaHNcwz8hxq0a9Ce+bgdalshkocwX9sMOBWkxxzVzX+rhwVl//VoPcbFyRk0oluPtdrHy+JY1ll8Lc+LtvjPi3b/4H3bltQcSg6zxuuWjvWbv04oG4t54I/0unareSjFWw0KL2dLUQHbDZsOWswXRT1ZxqOY9PLBPrsCSFczwFY0uH17DwLeoFxDOLVckNfJ2OntAUHJjdjn+8dlNobFEK6Jpz11uBDHW17SJWXjgZ2KhPMLFfyfLKaCpBcjKQzURDPe7/eiaysHV4H9r/qkyIM1eimT8KnQS9ofuVjaGMHL++MZ5K/1vfxC0h6Sc5dP4Bi1QYVjBlCsJA0H8ZMMaocU50nd3jZCHKi/zNsUB6UQT1UFAsCu/4Zc9uLo3ozxVssSNAbit5qWtVWtt6MLCavSKekd9f3yLzv/lwUYqVCpJYETIHqW0y7UMMMSFTQVafoL22lfnmCyRpb04zgha9TzxOJG9ZInw4V9JOlx/sP7Vq8ArYg1idJ/P9MVczqD1GVm6H0gltXG3fmBRETb3eLvbiSZqvMkd44bBudET5drjYW2v+y4SkrO7a6egw7fx0X1Z6fgiY+K3ur3MJeohRLDUYK8gPsKAALZMQu9TWE+/2nA2HQyAMVQ4wkPh1wcQ5FcZT2yDU9xkhZVrtn5kMAn61xxdqXpPcDxvcfi75bwSyIAgk/uL7LrOEFXzraJyCGt1POSEjHz5lAWivGX/Hm1zA7HoDxXCCoVw12xnkAJ9ndKrqCGT7+rGyRg44wxA+0vKxJIfEjgsV3VYeWyH2L+d9Wf90g8GRxRE+RFqw6lJwALCU9BGsuc7eFDebMIZh2dNM/G+136ymMJnDvCSpE8/X9Ui/At4Mn3+q0uo2+ROLCZgx3HhgGNT87PAeSJSACJGV7YIGH B5khJO9F V/dfdZHtWweiQKz96A1mhLzc9rStRFNdjTlkmwc1gXtI6r03Wd/wi4e2+yPxc11HtOrjIN9KwnmqUNkvVV8J8gexIT9KqKKUh7rZzLIEjuhEACvOmiJSu/7ggnpK5lBsk1heo4Krtn5/ENWjCEpdBainuDWSjUT62G82nEBqJfPlsfBU+rhlGsYU1WT68FfKuKNjpy3R3tK+QXh6DlOE2+NNGx7GRJ3aTdftysqgLLT/U2iv/OuhflU6hr94O9ssb7glemNIPQ5F7mGnDamA83yu9T8PN5t/376LItjo+e0r9FluKRrAdsTZPRXxlvRpJ9ip5gompVI8ut6CxNevFIVORpYp+1adb85PqoyfuywfOg2j2/mueNvZ57DvCXP0oVCV5pQMn3CmBiTEiP/JPlKi56n50lP1aYIa6E4dzA3ucY9hFHdI/TbfNuLclMY08ytT6vwAMmm69LsAiAKVpgYFy+mTopxUUSL5c3Rg/zbU9Hq57qKZAeSIKQ4XPQaDN78JgGtH1Rgd4JvVfMZn55dCoKfofZ2K2zveGbkP5ij1WDX7OBNXNpVvMm7FNUmag9vT4K/Ids1fpADieuxQwra7B5IGvODM5uPk15tcTciRV4Ex0350plop3hhpD+sH/QKIlsdMhHAqTyKz2kAaXHtOK/A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Replace test_young() and clear_young() with test_clear_young(). Signed-off-by: Yu Zhao --- include/linux/mmu_notifier.h | 29 ++----------------- include/trace/events/kvm.h | 15 ---------- mm/mmu_notifier.c | 42 ---------------------------- virt/kvm/kvm_main.c | 54 ------------------------------------ 4 files changed, 2 insertions(+), 138 deletions(-) diff --git a/include/linux/mmu_notifier.h b/include/linux/mmu_notifier.h index dfdbb370682d..c8f35fc08703 100644 --- a/include/linux/mmu_notifier.h +++ b/include/linux/mmu_notifier.h @@ -104,26 +104,6 @@ struct mmu_notifier_ops { unsigned long start, unsigned long end); - /* - * clear_young is a lightweight version of clear_flush_young. Like the - * latter, it is supposed to test-and-clear the young/accessed bitflag - * in the secondary pte, but it may omit flushing the secondary tlb. - */ - int (*clear_young)(struct mmu_notifier *subscription, - struct mm_struct *mm, - unsigned long start, - unsigned long end); - - /* - * test_young is called to check the young/accessed bitflag in - * the secondary pte. This is used to know if the page is - * frequently used without actually clearing the flag or tearing - * down the secondary mapping on the page. - */ - int (*test_young)(struct mmu_notifier *subscription, - struct mm_struct *mm, - unsigned long address); - int (*test_clear_young)(struct mmu_notifier *mn, struct mm_struct *mm, unsigned long start, unsigned long end, bool clear, unsigned long *bitmap); @@ -393,11 +373,6 @@ extern void __mmu_notifier_release(struct mm_struct *mm); extern int __mmu_notifier_clear_flush_young(struct mm_struct *mm, unsigned long start, unsigned long end); -extern int __mmu_notifier_clear_young(struct mm_struct *mm, - unsigned long start, - unsigned long end); -extern int __mmu_notifier_test_young(struct mm_struct *mm, - unsigned long address); extern int __mmu_notifier_test_clear_young(struct mm_struct *mm, unsigned long start, unsigned long end, bool clear, unsigned long *bitmap); @@ -437,7 +412,7 @@ static inline int mmu_notifier_clear_young(struct mm_struct *mm, unsigned long end) { if (mm_has_notifiers(mm)) - return __mmu_notifier_clear_young(mm, start, end); + return __mmu_notifier_test_clear_young(mm, start, end, true, NULL); return 0; } @@ -445,7 +420,7 @@ static inline int mmu_notifier_test_young(struct mm_struct *mm, unsigned long address) { if (mm_has_notifiers(mm)) - return __mmu_notifier_test_young(mm, address); + return __mmu_notifier_test_clear_young(mm, address, address + 1, false, NULL); return 0; } diff --git a/include/trace/events/kvm.h b/include/trace/events/kvm.h index 3bd31ea23fee..46c347e56e60 100644 --- a/include/trace/events/kvm.h +++ b/include/trace/events/kvm.h @@ -489,21 +489,6 @@ TRACE_EVENT(kvm_age_hva, __entry->start, __entry->end) ); -TRACE_EVENT(kvm_test_age_hva, - TP_PROTO(unsigned long hva), - TP_ARGS(hva), - - TP_STRUCT__entry( - __field( unsigned long, hva ) - ), - - TP_fast_assign( - __entry->hva = hva; - ), - - TP_printk("mmu notifier test age hva: %#016lx", __entry->hva) -); - #endif /* _TRACE_KVM_MAIN_H */ /* This part must be outside protection */ diff --git a/mm/mmu_notifier.c b/mm/mmu_notifier.c index 7e6aba4bddcb..c7e9747c9920 100644 --- a/mm/mmu_notifier.c +++ b/mm/mmu_notifier.c @@ -382,48 +382,6 @@ int __mmu_notifier_clear_flush_young(struct mm_struct *mm, return young; } -int __mmu_notifier_clear_young(struct mm_struct *mm, - unsigned long start, - unsigned long end) -{ - struct mmu_notifier *subscription; - int young = 0, id; - - id = srcu_read_lock(&srcu); - hlist_for_each_entry_rcu(subscription, - &mm->notifier_subscriptions->list, hlist, - srcu_read_lock_held(&srcu)) { - if (subscription->ops->clear_young) - young |= subscription->ops->clear_young(subscription, - mm, start, end); - } - srcu_read_unlock(&srcu, id); - - return young; -} - -int __mmu_notifier_test_young(struct mm_struct *mm, - unsigned long address) -{ - struct mmu_notifier *subscription; - int young = 0, id; - - id = srcu_read_lock(&srcu); - hlist_for_each_entry_rcu(subscription, - &mm->notifier_subscriptions->list, hlist, - srcu_read_lock_held(&srcu)) { - if (subscription->ops->test_young) { - young = subscription->ops->test_young(subscription, mm, - address); - if (young) - break; - } - } - srcu_read_unlock(&srcu, id); - - return young; -} - int __mmu_notifier_test_clear_young(struct mm_struct *mm, unsigned long start, unsigned long end, bool clear, unsigned long *bitmap) diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 31ee58754b19..977baaf1b248 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -674,25 +674,6 @@ static __always_inline int kvm_handle_hva_range(struct mmu_notifier *mn, return __kvm_handle_hva_range(kvm, &range); } -static __always_inline int kvm_handle_hva_range_no_flush(struct mmu_notifier *mn, - unsigned long start, - unsigned long end, - hva_handler_t handler) -{ - struct kvm *kvm = mmu_notifier_to_kvm(mn); - const struct kvm_hva_range range = { - .start = start, - .end = end, - .pte = __pte(0), - .handler = handler, - .on_lock = (void *)kvm_null_fn, - .on_unlock = (void *)kvm_null_fn, - .flush_on_ret = false, - .may_block = false, - }; - - return __kvm_handle_hva_range(kvm, &range); -} static void kvm_mmu_notifier_change_pte(struct mmu_notifier *mn, struct mm_struct *mm, unsigned long address, @@ -854,39 +835,6 @@ static int kvm_mmu_notifier_clear_flush_young(struct mmu_notifier *mn, return kvm_handle_hva_range(mn, start, end, __pte(0), kvm_age_gfn); } -static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn, - struct mm_struct *mm, - unsigned long start, - unsigned long end) -{ - trace_kvm_age_hva(start, end); - - /* - * Even though we do not flush TLB, this will still adversely - * affect performance on pre-Haswell Intel EPT, where there is - * no EPT Access Bit to clear so that we have to tear down EPT - * tables instead. If we find this unacceptable, we can always - * add a parameter to kvm_age_hva so that it effectively doesn't - * do anything on clear_young. - * - * Also note that currently we never issue secondary TLB flushes - * from clear_young, leaving this job up to the regular system - * cadence. If we find this inaccurate, we might come up with a - * more sophisticated heuristic later. - */ - return kvm_handle_hva_range_no_flush(mn, start, end, kvm_age_gfn); -} - -static int kvm_mmu_notifier_test_young(struct mmu_notifier *mn, - struct mm_struct *mm, - unsigned long address) -{ - trace_kvm_test_age_hva(address); - - return kvm_handle_hva_range_no_flush(mn, address, address + 1, - kvm_test_age_gfn); -} - struct test_clear_young_args { unsigned long *bitmap; unsigned long end; @@ -969,8 +917,6 @@ static const struct mmu_notifier_ops kvm_mmu_notifier_ops = { .invalidate_range_start = kvm_mmu_notifier_invalidate_range_start, .invalidate_range_end = kvm_mmu_notifier_invalidate_range_end, .clear_flush_young = kvm_mmu_notifier_clear_flush_young, - .clear_young = kvm_mmu_notifier_clear_young, - .test_young = kvm_mmu_notifier_test_young, .test_clear_young = kvm_mmu_notifier_test_clear_young, .change_pte = kvm_mmu_notifier_change_pte, .release = kvm_mmu_notifier_release, From patchwork Fri May 26 23:44:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13257441 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 155A3C77B7C for ; Fri, 26 May 2023 23:44:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C9288280002; Fri, 26 May 2023 19:44:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C4680280001; Fri, 26 May 2023 19:44:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B0A7D280002; Fri, 26 May 2023 19:44:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id A3E64280001 for ; Fri, 26 May 2023 19:44:47 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 66FE3C0E2B for ; Fri, 26 May 2023 23:44:47 +0000 (UTC) X-FDA: 80834038614.21.37ADEAA Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf18.hostedemail.com (Postfix) with ESMTP id 9C1341C0003 for ; Fri, 26 May 2023 23:44:45 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=lFfC+K1p; spf=pass (imf18.hostedemail.com: domain of 3bERxZAYKCOMdZeMFTLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--yuzhao.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3bERxZAYKCOMdZeMFTLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1685144685; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=W5mqYLgSgL41U5MVABcpsNGg2VTYVhdnWBmYAGgA32M=; b=V2LBbKAYcwZBMfSbWU/QrDPlSH853EJdyON4w30y3Gqh9NEtfAeQwpGvCYSC7NG4aptcs+ RenFWWUHuBm4L0rqqN19m1l9Jd1pQt5TfIPKcCspzhzT86s97x9ZNmIchjJmUrs6g8ruNS xf6uP/xlnO5ToT+EPNSl/WE23Ot8CEo= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1685144685; a=rsa-sha256; cv=none; b=H++zYbPytM/BjsG43Qc2H2MJHbDyF+yKpuxEhi4YzvwJm6JXPtp+dJA6y7xO+39Yhov3s4 lDmnuJzxvbK1Bz6iyfM9SC7UN9w3ah+E6QRcsb5qfrinpgV1gIr6Yj9LMZGC5Lgw43xxA/ 9f+zzoiENL7hbla5Zc+dtA8TTQBw31k= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=lFfC+K1p; spf=pass (imf18.hostedemail.com: domain of 3bERxZAYKCOMdZeMFTLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--yuzhao.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3bERxZAYKCOMdZeMFTLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-561f201a646so31029717b3.1 for ; Fri, 26 May 2023 16:44:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1685144684; x=1687736684; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=W5mqYLgSgL41U5MVABcpsNGg2VTYVhdnWBmYAGgA32M=; b=lFfC+K1prBwz+QWauf5t3Z5nGOG+XH3WsUV6BbAJbu8yEh9dwSmQ7iKcNTjxofo1xQ drv4Lmvn48X7bRRKv2GPWasyFIShdI23SCQ+h/TYUmoXc/hOnlnscK719N8Jpf3m7n6U mEaD6Epy7t6qJzvv/9f9Jp/JqQmnMe81faOyaQAqHTEXDufnGK7wY+Ey5ZFKaBi7dzJd 6O8PEoHW6NsTzElKeYmJHsoCaRTvtaP/AkDf6+cRfXVKyOf9lMpoYxPIHrUwcDj+nl2+ 7q+JpYkRKelWoBaQmJuf7/axxLP6kaz1KnqZQAEmA4XWZSzVgynpinko26WGtZXkTjpp oXNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685144684; x=1687736684; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=W5mqYLgSgL41U5MVABcpsNGg2VTYVhdnWBmYAGgA32M=; b=HagqZUNRNJJ7KoLZfDHmELcc6xj30ARoYK6+sGtRyif72dmBHLC/wcfB9hGBPuxCM/ pJfgKuGV35NSKub70VbvjEVsrJk/0nCYdU0L1E/FVmZIre8ag92lUEVs+8e5RNAOhT/m jwjB7ZlGH7/wuxiCICczcKyJzS5oIHiL4nnkRKfK8VQ3JiSnHOJ1yraIGIKZBck6+XZX w6/N8MPk2eypWN7JumCMFNPXqC+7fvCrCPH0XxkUaZgZ3RgIkx3KgS0hydLd1a5/om45 N98pv/E2dPGnzbwQ2X2oSl3cvAyJ76r5RQ1LK7L7CdXImYYp50/n8EoM2U7zzwJISmps rkcg== X-Gm-Message-State: AC+VfDyVrO/PdBTxoTF9V8VX5/HFT3IP+mHIAX213kcKJy1ikFX3pnzI RZinscQ7XVCHDj377CuVsUvzOkB3U6g= X-Google-Smtp-Source: ACHHUZ5aYhy2MBmaWD1q0FxpVfKGE+Nrskt2Z7iv2nNf760O+qZ11RhzmW2rToUdRrv6PXs76j2V3Kvga7o= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:910f:8a15:592b:2087]) (user=yuzhao job=sendgmr) by 2002:a81:4421:0:b0:565:9f59:664f with SMTP id r33-20020a814421000000b005659f59664fmr2006806ywa.6.1685144684605; Fri, 26 May 2023 16:44:44 -0700 (PDT) Date: Fri, 26 May 2023 17:44:28 -0600 In-Reply-To: <20230526234435.662652-1-yuzhao@google.com> Message-Id: <20230526234435.662652-4-yuzhao@google.com> Mime-Version: 1.0 References: <20230526234435.662652-1-yuzhao@google.com> X-Mailer: git-send-email 2.41.0.rc0.172.g3f132b7071-goog Subject: [PATCH mm-unstable v2 03/10] kvm/arm64: export stage2_try_set_pte() and macros From: Yu Zhao To: Andrew Morton , Paolo Bonzini Cc: Alistair Popple , Anup Patel , Ben Gardon , Borislav Petkov , Catalin Marinas , Chao Peng , Christophe Leroy , Dave Hansen , Fabiano Rosas , Gaosheng Cui , Gavin Shan , "H. Peter Anvin" , Ingo Molnar , James Morse , "Jason A. Donenfeld" , Jason Gunthorpe , Jonathan Corbet , Marc Zyngier , Masami Hiramatsu , Michael Ellerman , Michael Larabel , Mike Rapoport , Nicholas Piggin , Oliver Upton , Paul Mackerras , Peter Xu , Sean Christopherson , Steven Rostedt , Suzuki K Poulose , Thomas Gleixner , Thomas Huth , Will Deacon , Zenghui Yu , kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-trace-kernel@vger.kernel.org, x86@kernel.org, linux-mm@google.com, Yu Zhao X-Rspamd-Queue-Id: 9C1341C0003 X-Rspam-User: X-Stat-Signature: orhsopoxpbbjfwpedig8wh6q7tfgy9ex X-Rspamd-Server: rspam03 X-HE-Tag: 1685144685-193300 X-HE-Meta: U2FsdGVkX19gPqbQ9WUVz3Bv/gCrF23owNNj+VDMewbBMwhcH93mZuD7lM5ezGChShlg+av4GSkoMKlawlkcv4YvRmaS/7nfCl9uAvE13atMFst7hbcOLkp86X6YGt9aDZDlctStyJXhn3Yt8Kl3adwlkIalU5tvHZzQjFJ85BSApssqm2lyA6DDehyQQPJ7nookPCqBcmSHjP+mkuO/xaLkD1tXsA7R7vjhBZ+bk+EJoxVsv0rjeyYGwZ0EF9eH9DKTpPMmQG8pJoL9hhQcUdTby9us/rHp0PBwB1Cc3bEONNGBPdlY2OI1tTc6u99q52sb2X8as2vGJFDl5vHtXhGaIFxcqzEmzgZvOw/cg7tK36j4R06of+9QO5OU8jN4EGdPUsNpBRCawTdxmLUMaKZzL1bRywSHNKEsgXjn8rQH4bLkSwRUOQ9G8Hg+OX9PdMBxqmseXszgpLFa784ExXx2cGChe9Q5+O3P3O8oFHDZ68o0mjXcNN5bVVCzZ5SeTrEnkTonWwJ0lWqdJXhTPPPUcu5AgmOtHoillF2nvB9OVoBKC9hlXu1osk+c7eaPCCy5cqAlV4g9cez+t0cWOn62qUBaBsR8daB1XyT91C4N9apJ/xSY13vR0xgez1GRuZVa9I8lCkNB/pGDtFw+0DLIIVhPdB8eKJFjKOJUshThcsdeGx9GUrxuq/bw+croT8zUy6Ieh2hKDduyuNdbuoejwspwsd3rlin9AGaKyRZh+9vspPwkanJrQZoWQzFlA65smZvJSGfQ01b4ZqC4Bs+m7bKAtB/KsAMWJd5DSgesri3uAa6JHPfJrLEYlG3vB3D3fNntCyrr87+Ix3qMOTd33vUQFCIp6KgeVawin/NeTQfmp5ePaYVUTj90TlM6a/g0NeHzL7uG64rYkVCPTYZVj3jxQ7RAm65QlcqkOUYB9AgNStlRPBZpocn3hhmU2Upr8h4KzlrHefODM67 QHxmeTqN uCVrATd1Ru9vXN9m4ZQyrdnV5KayrYWsVPhkBVRrv4YQ4v72ycMboaphEX53sRmXfJ7gTrhUlGVBJ2fZKZo7R61iGQAKP+K3OrcY8oeuG1zdD0u6BJyl9W3fvRBJqkmmpM1IYUmXd6esmxrShb/nRzek7X2etBl21ek8XrzIWBaqIPWobV9FsUa1d3ltT28FI0ClKVMxgsOK1jM/8OPDDdhKNfqhMB7CtVukbm1LeV6DQGCb6x+wh4dvkJ1NC3Acvr1YjhwgIkAR2WKbQRnaRRKhud7tXlN7o9M9BPVPkPcGIOVBepNYzChqM4y79DmstMHYT2n2lJyEHkCDuB7anMwAUISha3Ub/XXy3r5imenyvHw1CDtIE8PAHFfRilieopaNOkhDeNAq0KAtKJrM/LB2U/jSCI7fgTNx01BVOHGkhFObonbdeRh0IHbTHe7TnvGwiLl5Wh8O2+QBPS9qBz4L1eTqhHfBLLGj4tHqZeWm3f+d8cx+Wbc3JnAYKCd4RYtXL2QASn0Bc08g2uQ7hlxDEfewG6FHOgxz7IvAIUjkTAziQ9LKrQhhG8qDtYvuwTzZ8f46wXg7nDjmgQCzkW4Csbh7fiSL2I1pf/oRoJDNg3NEeFZtgC059yHvULxVUNJJj+VJwd2dWQyV2VfUw0YJYp05VehVQqATE X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: stage2_try_set_pte() and KVM_PTE_LEAF_ATTR_LO_S2_AF are needed to implement kvm_arch_test_clear_young(). Signed-off-by: Yu Zhao --- arch/arm64/include/asm/kvm_pgtable.h | 53 ++++++++++++++++++++++++++++ arch/arm64/kvm/hyp/pgtable.c | 53 ---------------------------- 2 files changed, 53 insertions(+), 53 deletions(-) diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index dc3c072e862f..ff520598b62c 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -44,6 +44,49 @@ typedef u64 kvm_pte_t; #define KVM_PHYS_INVALID (-1ULL) +#define KVM_PTE_TYPE BIT(1) +#define KVM_PTE_TYPE_BLOCK 0 +#define KVM_PTE_TYPE_PAGE 1 +#define KVM_PTE_TYPE_TABLE 1 + +#define KVM_PTE_LEAF_ATTR_LO GENMASK(11, 2) + +#define KVM_PTE_LEAF_ATTR_LO_S1_ATTRIDX GENMASK(4, 2) +#define KVM_PTE_LEAF_ATTR_LO_S1_AP GENMASK(7, 6) +#define KVM_PTE_LEAF_ATTR_LO_S1_AP_RO 3 +#define KVM_PTE_LEAF_ATTR_LO_S1_AP_RW 1 +#define KVM_PTE_LEAF_ATTR_LO_S1_SH GENMASK(9, 8) +#define KVM_PTE_LEAF_ATTR_LO_S1_SH_IS 3 +#define KVM_PTE_LEAF_ATTR_LO_S1_AF BIT(10) + +#define KVM_PTE_LEAF_ATTR_LO_S2_MEMATTR GENMASK(5, 2) +#define KVM_PTE_LEAF_ATTR_LO_S2_S2AP_R BIT(6) +#define KVM_PTE_LEAF_ATTR_LO_S2_S2AP_W BIT(7) +#define KVM_PTE_LEAF_ATTR_LO_S2_SH GENMASK(9, 8) +#define KVM_PTE_LEAF_ATTR_LO_S2_SH_IS 3 +#define KVM_PTE_LEAF_ATTR_LO_S2_AF BIT(10) + +#define KVM_PTE_LEAF_ATTR_HI GENMASK(63, 51) + +#define KVM_PTE_LEAF_ATTR_HI_SW GENMASK(58, 55) + +#define KVM_PTE_LEAF_ATTR_HI_S1_XN BIT(54) + +#define KVM_PTE_LEAF_ATTR_HI_S2_XN BIT(54) + +#define KVM_PTE_LEAF_ATTR_S2_PERMS (KVM_PTE_LEAF_ATTR_LO_S2_S2AP_R | \ + KVM_PTE_LEAF_ATTR_LO_S2_S2AP_W | \ + KVM_PTE_LEAF_ATTR_HI_S2_XN) + +#define KVM_INVALID_PTE_OWNER_MASK GENMASK(9, 2) +#define KVM_MAX_OWNER_ID 1 + +/* + * Used to indicate a pte for which a 'break-before-make' sequence is in + * progress. + */ +#define KVM_INVALID_PTE_LOCKED BIT(10) + static inline bool kvm_pte_valid(kvm_pte_t pte) { return pte & KVM_PTE_VALID; @@ -224,6 +267,16 @@ static inline bool kvm_pgtable_walk_shared(const struct kvm_pgtable_visit_ctx *c return ctx->flags & KVM_PGTABLE_WALK_SHARED; } +static inline bool stage2_try_set_pte(const struct kvm_pgtable_visit_ctx *ctx, kvm_pte_t new) +{ + if (!kvm_pgtable_walk_shared(ctx)) { + WRITE_ONCE(*ctx->ptep, new); + return true; + } + + return cmpxchg(ctx->ptep, ctx->old, new) == ctx->old; +} + /** * struct kvm_pgtable_walker - Hook into a page-table walk. * @cb: Callback function to invoke during the walk. diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 5282cb9ca4cf..24678ccba76a 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -12,49 +12,6 @@ #include -#define KVM_PTE_TYPE BIT(1) -#define KVM_PTE_TYPE_BLOCK 0 -#define KVM_PTE_TYPE_PAGE 1 -#define KVM_PTE_TYPE_TABLE 1 - -#define KVM_PTE_LEAF_ATTR_LO GENMASK(11, 2) - -#define KVM_PTE_LEAF_ATTR_LO_S1_ATTRIDX GENMASK(4, 2) -#define KVM_PTE_LEAF_ATTR_LO_S1_AP GENMASK(7, 6) -#define KVM_PTE_LEAF_ATTR_LO_S1_AP_RO 3 -#define KVM_PTE_LEAF_ATTR_LO_S1_AP_RW 1 -#define KVM_PTE_LEAF_ATTR_LO_S1_SH GENMASK(9, 8) -#define KVM_PTE_LEAF_ATTR_LO_S1_SH_IS 3 -#define KVM_PTE_LEAF_ATTR_LO_S1_AF BIT(10) - -#define KVM_PTE_LEAF_ATTR_LO_S2_MEMATTR GENMASK(5, 2) -#define KVM_PTE_LEAF_ATTR_LO_S2_S2AP_R BIT(6) -#define KVM_PTE_LEAF_ATTR_LO_S2_S2AP_W BIT(7) -#define KVM_PTE_LEAF_ATTR_LO_S2_SH GENMASK(9, 8) -#define KVM_PTE_LEAF_ATTR_LO_S2_SH_IS 3 -#define KVM_PTE_LEAF_ATTR_LO_S2_AF BIT(10) - -#define KVM_PTE_LEAF_ATTR_HI GENMASK(63, 51) - -#define KVM_PTE_LEAF_ATTR_HI_SW GENMASK(58, 55) - -#define KVM_PTE_LEAF_ATTR_HI_S1_XN BIT(54) - -#define KVM_PTE_LEAF_ATTR_HI_S2_XN BIT(54) - -#define KVM_PTE_LEAF_ATTR_S2_PERMS (KVM_PTE_LEAF_ATTR_LO_S2_S2AP_R | \ - KVM_PTE_LEAF_ATTR_LO_S2_S2AP_W | \ - KVM_PTE_LEAF_ATTR_HI_S2_XN) - -#define KVM_INVALID_PTE_OWNER_MASK GENMASK(9, 2) -#define KVM_MAX_OWNER_ID 1 - -/* - * Used to indicate a pte for which a 'break-before-make' sequence is in - * progress. - */ -#define KVM_INVALID_PTE_LOCKED BIT(10) - struct kvm_pgtable_walk_data { struct kvm_pgtable_walker *walker; @@ -702,16 +659,6 @@ static bool stage2_pte_is_locked(kvm_pte_t pte) return !kvm_pte_valid(pte) && (pte & KVM_INVALID_PTE_LOCKED); } -static bool stage2_try_set_pte(const struct kvm_pgtable_visit_ctx *ctx, kvm_pte_t new) -{ - if (!kvm_pgtable_walk_shared(ctx)) { - WRITE_ONCE(*ctx->ptep, new); - return true; - } - - return cmpxchg(ctx->ptep, ctx->old, new) == ctx->old; -} - /** * stage2_try_break_pte() - Invalidates a pte according to the * 'break-before-make' requirements of the From patchwork Fri May 26 23:44:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13257481 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C2E9CC77B7C for ; Fri, 26 May 2023 23:50:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5B301900003; Fri, 26 May 2023 19:50:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 561636B0072; Fri, 26 May 2023 19:50:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 429D6900003; Fri, 26 May 2023 19:50:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 2FAE76B0071 for ; Fri, 26 May 2023 19:50:47 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id D9E5C160FC5 for ; Fri, 26 May 2023 23:50:46 +0000 (UTC) X-FDA: 80834053692.15.E727C00 Received: from mail-vk1-f202.google.com (mail-vk1-f202.google.com [209.85.221.202]) by imf15.hostedemail.com (Postfix) with ESMTP id E8620A000E for ; Fri, 26 May 2023 23:50:44 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=tM0X4+0+; spf=pass (imf15.hostedemail.com: domain of 3bkRxZAYKCOUfbgOHVNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--yuzhao.bounces.google.com designates 209.85.221.202 as permitted sender) smtp.mailfrom=3bkRxZAYKCOUfbgOHVNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1685145044; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9t4P2GvyPz5aecPxpWswvlll2je3r362Snycxj82bkU=; b=kxF7Kg+IU8pTKAumyznfCwUjioGCq1rjuHmi3VvgHNcbwkoqCl3yJBcY3CJvU4FT343pfj 3r5Dayt6PnQpaSRG5SpBgVf5vRck0MkT+nCaoV87bc1dfogvhCKBVw9iX/qq2tlXQ0rHv6 5F1sq0HD0Ax06IRfTK2IQBUIZz7OU2k= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1685145045; a=rsa-sha256; cv=none; b=ZdJ/gc7NEYZrx9D9wvpACT6/+YAm9NwxwzHsM10PbU1ynFu0UbbWaQ1586OzdPxhHkJr2n V6+OZ/pWNkW/s2z5qnNY12ur+K6/Hi9fiENhqENHbbRZow3/K/wdLV/hcYVUdIawft/eGN iwpsv1u507eZWIz1c4d+BUAeEmhcMIE= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=tM0X4+0+; spf=pass (imf15.hostedemail.com: domain of 3bkRxZAYKCOUfbgOHVNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--yuzhao.bounces.google.com designates 209.85.221.202 as permitted sender) smtp.mailfrom=3bkRxZAYKCOUfbgOHVNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-vk1-f202.google.com with SMTP id 71dfb90a1353d-456f19ee641so1163307e0c.0 for ; Fri, 26 May 2023 16:50:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1685145044; x=1687737044; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=9t4P2GvyPz5aecPxpWswvlll2je3r362Snycxj82bkU=; b=tM0X4+0+KNJG57eNQDBDXkefF1mnzRqNrsk5Lpmj4vnB41bmjx5s/E/aQiCko2nR2H dc3/aDoEiGxOM/755MExbY4j+S9GCT2IB7fJzmZDNsjcj/a/lwsOBIRWjUXkEOGKXfxZ qHMEHhgAFNNtWFUavJ2kDc95CHOzOWbPGWKVA2lnykxkVX95BXBAGQoxzGvIKIa+vXLL 9fniYEWvL3B7m0gVvrngoh/GiWh4KRDGcuy1MNvBDsUl+EY6G/HAstXJhUsQ/wP/gzvx OmJGbc+fOFofTDeZFlpMX6+spIxmCmgjkvRld21l9RPsFXfr0Knu6VwHGYHYd0biT+0I UiAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685145044; x=1687737044; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=9t4P2GvyPz5aecPxpWswvlll2je3r362Snycxj82bkU=; b=PK6hYNttwNGaRYpV9vqKbphQ1ELDtCwF3sjSigeG8tj+2YCmuvoVc2zBDOtg7NRpQ2 2qbb7rOl8yd3iG5MNhVT2ma8knIwIeKeLeSC9H8DpedtOjm9xRXMZwss6zt6ltrdfgkU ZjJzmXe75jHe5VXPmJRSihxm3ILp8Mmr0nqLlE5/JW1cydDQqaM5vWgcLY+4qgz4Ez2h NjXgqTpo+kOwiLlWeujhGv4MjFRHMxzoAcssbR4BC2Q/mlwt/LstTyy4bi+Ntk06BGI5 oH14jN5eJaD3vfeUmEcq9AvEssD11AzNHgXQix1q4ZuCv0Dd+FyGVS2pSn6N+2O9z4jz CQWQ== X-Gm-Message-State: AC+VfDw++XYNU+1YnCVPr3ODBOar2zBPEAn1yRPFmgKmXxjZpRyVcS7I acjq1kgkxzxwwnk1I+Empa9c80xpDRE= X-Google-Smtp-Source: ACHHUZ671mv9b1WNngTqma0ZpDsFyu7jCdBVYj6cC/Q6bUpKz7y4SPqrDazly1Z4TuFOLliyXqQloqgktcM= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:910f:8a15:592b:2087]) (user=yuzhao job=sendgmr) by 2002:a81:b627:0:b0:559:d859:d749 with SMTP id u39-20020a81b627000000b00559d859d749mr437593ywh.5.1685144686225; Fri, 26 May 2023 16:44:46 -0700 (PDT) Date: Fri, 26 May 2023 17:44:29 -0600 In-Reply-To: <20230526234435.662652-1-yuzhao@google.com> Message-Id: <20230526234435.662652-5-yuzhao@google.com> Mime-Version: 1.0 References: <20230526234435.662652-1-yuzhao@google.com> X-Mailer: git-send-email 2.41.0.rc0.172.g3f132b7071-goog Subject: [PATCH mm-unstable v2 04/10] kvm/arm64: make stage2 page tables RCU safe From: Yu Zhao To: Andrew Morton , Paolo Bonzini Cc: Alistair Popple , Anup Patel , Ben Gardon , Borislav Petkov , Catalin Marinas , Chao Peng , Christophe Leroy , Dave Hansen , Fabiano Rosas , Gaosheng Cui , Gavin Shan , "H. Peter Anvin" , Ingo Molnar , James Morse , "Jason A. Donenfeld" , Jason Gunthorpe , Jonathan Corbet , Marc Zyngier , Masami Hiramatsu , Michael Ellerman , Michael Larabel , Mike Rapoport , Nicholas Piggin , Oliver Upton , Paul Mackerras , Peter Xu , Sean Christopherson , Steven Rostedt , Suzuki K Poulose , Thomas Gleixner , Thomas Huth , Will Deacon , Zenghui Yu , kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-trace-kernel@vger.kernel.org, x86@kernel.org, linux-mm@google.com, Yu Zhao X-Stat-Signature: wrjydh43aas6a9a6sdscuozcp75wyy6e X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: E8620A000E X-Rspam-User: X-HE-Tag: 1685145044-680445 X-HE-Meta: U2FsdGVkX19s8jf0E8xNt7QqMeT/63QrcSWlBCzrb6BXpErlOxv/7uC77UQ6oQrSELa8EXeG8+jUqprgk4RkxiQFb1SoGrVtuB99PYCF+h19DjeL89Q3Y56yoxcmo2SgJPo6rBiIHYyY7frFhz9DdIRch6GTYqcxe0QRv7/Q/GUfLc1rO0Y93AzztLAbrBlybx2wKIDydhYBZ+mvSjaH7NT+Cm9nFM3JWgrmTyX2auF0VzhQlcDuyuYjFCcP+KkGPtzkZjKMAeA4yTIp4J8uaVRXKJ49wY50RP0kkVCFLoJ0/RNB6lBWOKlBstGUWjtQIpvbvzvDLWYRywQ8rv6IZEmnuWq12RTK/5C7kxnBA6ha8Zg997lqSP3sdmkbergGh/VCGFWE5ks8TQnBI4Tr6gYqIPWHmO8IgwmmJbxnKs7nuHWTGjNz/SZoa10lKOEjAMGzbKK468Ux9d1uJ8NZ60APXacyJB+aUsXHz6QMEtJHNpI7U0Zxw+D2hN2gThSG9X6s623OJthCuYdhBVANtC+ZSBTfr3Hq2IY5OAfp3QmWC1vpTW6Qjq4V0qtVi/SNhKMhsLYlXg7RQh2FHedslCx6l9x44PEI1XOgrDfisauLz7r0Qx9odQAnToKgqKtoC/gpSAIyrseXiirnc179V/j7UmXXp0108r8BYikrocdmgW5o3lo39MrvVJ6iKXwIPVYAMql7ptIVt54cnMV6wm2K3f7g1CZzH7/5HlnJ/2fdx0jZ9HkRyRgWVsJoHdSTU0oeQDp/caHFpaVFX2LITMg2eurXLNKuPU2MmCEdNn7W3/vbUrj8C5kY0hBmB09ymGeHHn7j8sGZ44RsQcJsMqlplIZtTHpsYcqp8Mr1KYVf//Vdbj1TejG+Yam1EcGqFOCrNCSz1pR/qCDg/g0oiFGa8IHsr0wFSQ6Bk/K5pWfY3j1yHBMLRUe7fk9CXUuoQyRFQIHa+64fpKNkD2O GB7qrMOd +EJSmGKsEmz8AlxCRgWbA4TO+Q4cbHOlwzLWUn1t8Oi1cTR78EZhZoRU6M0Lsiuqw3tzH5tkfN2ivpjre4pDRzkzu3m6ESxA/lIefYCXO1kDO3G72hQ2XonNeR860nQYTvWcIMMPZ/IK4WGGffXhlfnn1KRwtzZ9yGlovmat1lwIO5bwSZ2yNfOvMsPKIc7H/n+U0HpsfEz4jTovJm2qj6dL7lbxgAV0JoZtj1z1TiiWN8Qp1WO6xY/LicoBL9IV8pFvd//eklzYWIAPL3MqWYpiTJEMJ4T5ARkwbxFTFz8z9DYskYHfL+9BEdpAkkeweLPjALkIalC9guTMNCVe30Z97IIXSSzuuUirG90kVnwI3ag5Kn8/4nENluVO3mXH7uHAn8H/iDU1b/jNZwS28AD3aN61VacQ8VYvewni1OMu78GLMpJWP2NLfDEAuBhAGGg2rlpCFYCy/0rXR31FYToL0rYF7NFdDvm5CV6WCZhcQQsIhbQaDwlqTyMETebIwqBVuxO0DHiUTzvYpvkQLiXxG9GEQDTnDEnfKObctemv2wtUf1ce5ktDhnYNDEQarCFHvoqirI0YdXId7Vv6NeV5UrggU0M0clgH0Ovy4GDwZzd0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Stage2 page tables are currently not RCU safe against unmapping or VM destruction. The previous mmu_notifier_ops members rely on kvm->mmu_lock to synchronize with those operations. However, the new mmu_notifier_ops member test_clear_young() provides a fast path that does not take kvm->mmu_lock. To implement kvm_arch_test_clear_young() for that path, unmapped page tables need to be freed by RCU and kvm_free_stage2_pgd() needs to be after mmu_notifier_unregister(). Remapping, specifically stage2_free_removed_table(), is already RCU safe. Signed-off-by: Yu Zhao --- arch/arm64/include/asm/kvm_pgtable.h | 2 ++ arch/arm64/kvm/arm.c | 1 + arch/arm64/kvm/hyp/pgtable.c | 8 ++++++-- arch/arm64/kvm/mmu.c | 17 ++++++++++++++++- 4 files changed, 25 insertions(+), 3 deletions(-) diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index ff520598b62c..5cab52e3a35f 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -153,6 +153,7 @@ static inline bool kvm_level_supports_block_mapping(u32 level) * @put_page: Decrement the refcount on a page. When the * refcount reaches 0 the page is automatically * freed. + * @put_page_rcu: RCU variant of the above. * @page_count: Return the refcount of a page. * @phys_to_virt: Convert a physical address into a virtual * address mapped in the current context. @@ -170,6 +171,7 @@ struct kvm_pgtable_mm_ops { void (*free_removed_table)(void *addr, u32 level); void (*get_page)(void *addr); void (*put_page)(void *addr); + void (*put_page_rcu)(void *addr); int (*page_count)(void *addr); void* (*phys_to_virt)(phys_addr_t phys); phys_addr_t (*virt_to_phys)(void *addr); diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 14391826241c..ee93271035d9 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -191,6 +191,7 @@ vm_fault_t kvm_arch_vcpu_fault(struct kvm_vcpu *vcpu, struct vm_fault *vmf) */ void kvm_arch_destroy_vm(struct kvm *kvm) { + kvm_free_stage2_pgd(&kvm->arch.mmu); bitmap_free(kvm->arch.pmu_filter); free_cpumask_var(kvm->arch.supported_cpus); diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 24678ccba76a..dbace4c6a841 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -988,8 +988,12 @@ static int stage2_unmap_walker(const struct kvm_pgtable_visit_ctx *ctx, mm_ops->dcache_clean_inval_poc(kvm_pte_follow(ctx->old, mm_ops), kvm_granule_size(ctx->level)); - if (childp) - mm_ops->put_page(childp); + if (childp) { + if (mm_ops->put_page_rcu) + mm_ops->put_page_rcu(childp); + else + mm_ops->put_page(childp); + } return 0; } diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 3b9d4d24c361..c3b3e2afe26f 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -172,6 +172,21 @@ static int kvm_host_page_count(void *addr) return page_count(virt_to_page(addr)); } +static void kvm_s2_rcu_put_page(struct rcu_head *head) +{ + put_page(container_of(head, struct page, rcu_head)); +} + +static void kvm_s2_put_page_rcu(void *addr) +{ + struct page *page = virt_to_page(addr); + + if (kvm_host_page_count(addr) == 1) + kvm_account_pgtable_pages(addr, -1); + + call_rcu(&page->rcu_head, kvm_s2_rcu_put_page); +} + static phys_addr_t kvm_host_pa(void *addr) { return __pa(addr); @@ -704,6 +719,7 @@ static struct kvm_pgtable_mm_ops kvm_s2_mm_ops = { .free_removed_table = stage2_free_removed_table, .get_page = kvm_host_get_page, .put_page = kvm_s2_put_page, + .put_page_rcu = kvm_s2_put_page_rcu, .page_count = kvm_host_page_count, .phys_to_virt = kvm_host_va, .virt_to_phys = kvm_host_pa, @@ -1877,7 +1893,6 @@ void kvm_arch_memslots_updated(struct kvm *kvm, u64 gen) void kvm_arch_flush_shadow_all(struct kvm *kvm) { - kvm_free_stage2_pgd(&kvm->arch.mmu); } void kvm_arch_flush_shadow_memslot(struct kvm *kvm, From patchwork Fri May 26 23:44:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13257442 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A6A64C77B7A for ; Fri, 26 May 2023 23:44:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 233A2280003; Fri, 26 May 2023 19:44:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1E32D280001; Fri, 26 May 2023 19:44:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 05C36280003; Fri, 26 May 2023 19:44:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id EBE48280001 for ; Fri, 26 May 2023 19:44:50 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id C1126AED8F for ; Fri, 26 May 2023 23:44:50 +0000 (UTC) X-FDA: 80834038740.09.9392CCC Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf17.hostedemail.com (Postfix) with ESMTP id 7369640014 for ; Fri, 26 May 2023 23:44:48 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=A+5tnxly; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of 3b0RxZAYKCOYgchPIWOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--yuzhao.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3b0RxZAYKCOYgchPIWOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--yuzhao.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1685144688; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=gdtrmeQ7r8fSzyqqoeow0Oco6w18CVCYcKcBQXKvLk0=; b=NPmAOQobc7iY+qicdMu2Nal46l0MoZWenUtxvWC3I8u/vTvaRvcxbIrkp5p2loihZEMat0 yUKTFdTWtOBzcbZSvwayrFQzqG/i7gTXlFvcY0jyFiiMhFk/CxzWyNWhRi6GKavcVCx8Dr dlFse8XK2/n6/NJp5NM+Y/lV1D6bizg= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=A+5tnxly; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of 3b0RxZAYKCOYgchPIWOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--yuzhao.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3b0RxZAYKCOYgchPIWOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--yuzhao.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1685144688; a=rsa-sha256; cv=none; b=CSO5jqQttPE1dU5fzbEfsZG+UWaWQp3bMytoLslzFgeZr1Shbroa4Ranqh+c+p6fUvkVSu Q/DRSGojXyAHxW2xSYIbHYkUW0A4mxXpS2PwddxPLmS7jxyy2zXDbJ6fBHDHJCP3UtOih0 aVFHumHhe6dXbEW9IFNMYlFI+mbGiEk= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-babb78a3daaso1692967276.1 for ; Fri, 26 May 2023 16:44:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1685144687; x=1687736687; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=gdtrmeQ7r8fSzyqqoeow0Oco6w18CVCYcKcBQXKvLk0=; b=A+5tnxly4jAziM7PCgC2JSUrvZIuWUL9UuTqWd7F/s/V7U9ZvBSUtbCMn4PDjXhEkh xPkXA8ijRXU8kfBAAJh45kvENT/cQ+0BqqVDKv4CzNn3FeibDZ4esB2AS5+kj2xWW1A/ 5f+rTWaRryOWr0eLpQP68cwlR3USUfp7Ely4h3ulPbhNc/8a6O9EbhRXY0Hc8yu56mx8 LQMCwDasuefCJVI+rIDEYvr3/19bdInHAEUvVq6phC2Jc21wzdoWQlzM7WDhIq/i18eK BEwyErOKi0xkhmnwWH0N/n1UzNpwg5kRKa2pc24aeVs5XXh0p7GOcbD7gzDkKgQaeHXr C7IQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685144687; x=1687736687; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=gdtrmeQ7r8fSzyqqoeow0Oco6w18CVCYcKcBQXKvLk0=; b=g8/JufTnF2NTynxa84YKtQwp5eluAh9OXZOMtk0HqEqqq3rt7Aku96vY4DK5vK7SyX O0XO/6yZgNK0LrlG+BdHCl5zmXEqvZMWk5Xew6tJNe0L5jzuu24zhgLiVRHXPD1uTISM 9gITp7W6tc/MU4YBdAZ2vSANfZwXhmMcu+jIGY901m4KhKuzYQlZvkPQp9nDF8ZZGvPA q+sGWeaZU2FU4Qdppdq4hhlyHBmdM9F4HBu7GEIz5aO54BUCH4f6W3klnvbE3xmofLzT 1r9T64vopMPDr0CoVDyrpeAVvEOPq7Zf4ggXxmwEHHsHBfTX8/0OFhoLbV9G+wY4zpV0 a+2g== X-Gm-Message-State: AC+VfDySL9sIGUS6OAtGwMuM4qcOh00E7lIvbYh4KuKolq+iTVV/uXg5 sAgbgMFBx6yQ0PNw74BIxNzyfL1UemI= X-Google-Smtp-Source: ACHHUZ75roT4ZouTjM2lH4DZFyTBmS3QLSjPTFGxgUaEGtzpG1LtCtXvcTQXcsV4/+3guLPQ6FLc+WkYaOc= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:910f:8a15:592b:2087]) (user=yuzhao job=sendgmr) by 2002:a25:b28d:0:b0:bad:155a:1004 with SMTP id k13-20020a25b28d000000b00bad155a1004mr1830575ybj.2.1685144687512; Fri, 26 May 2023 16:44:47 -0700 (PDT) Date: Fri, 26 May 2023 17:44:30 -0600 In-Reply-To: <20230526234435.662652-1-yuzhao@google.com> Message-Id: <20230526234435.662652-6-yuzhao@google.com> Mime-Version: 1.0 References: <20230526234435.662652-1-yuzhao@google.com> X-Mailer: git-send-email 2.41.0.rc0.172.g3f132b7071-goog Subject: [PATCH mm-unstable v2 05/10] kvm/arm64: add kvm_arch_test_clear_young() From: Yu Zhao To: Andrew Morton , Paolo Bonzini Cc: Alistair Popple , Anup Patel , Ben Gardon , Borislav Petkov , Catalin Marinas , Chao Peng , Christophe Leroy , Dave Hansen , Fabiano Rosas , Gaosheng Cui , Gavin Shan , "H. Peter Anvin" , Ingo Molnar , James Morse , "Jason A. Donenfeld" , Jason Gunthorpe , Jonathan Corbet , Marc Zyngier , Masami Hiramatsu , Michael Ellerman , Michael Larabel , Mike Rapoport , Nicholas Piggin , Oliver Upton , Paul Mackerras , Peter Xu , Sean Christopherson , Steven Rostedt , Suzuki K Poulose , Thomas Gleixner , Thomas Huth , Will Deacon , Zenghui Yu , kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-trace-kernel@vger.kernel.org, x86@kernel.org, linux-mm@google.com, Yu Zhao X-Rspam-User: X-Stat-Signature: miudf7xkzcik9j8g78t1w1nhsf8zcdkq X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 7369640014 X-HE-Tag: 1685144688-465266 X-HE-Meta: U2FsdGVkX1/4yW44XV5X+AfA7SSAlqq8quhpfnZ7tRZGWf0PAL/RMDWvq1B6i2jvstzTM3TbynPNb2KMuQsawX3+jqagfR7RXcz66wgD6aK4+SDb2jpYVB8CDM68rkDBAarA473IFZo7rJlPZQoHL3gyrjm0asIqmGlD8SlLgpjySPxb8nBTESB9BWv8ijpV3Dyj7QkehkOo82tTcSn1dekrpBR606Yl/sJ6Kw/9ilRJgmF9OZnjp3uh8RtZM2WKLGxOrTyJ/BLOkYshF1mkscMz2/Nfx9O2tvA0hs/fBWXB9b885p0BWjG6IgX+lbpFP4v3cN20PPlwtc/fTxqFdbWUJnwe2JwRzGmpzEDK2FAl6Eeg6Ra08h9Yzv18yV/z+JcMVtpgjokmpT47XLPjMBdqOqvuEwcOoa8h3tYcRjZrvhoOvKqmpc0f7DQplHyLBidxFumRjGFVq0CkmMPh/NRfpIkXytlKDyOeqJa7YC5v7sDKyT0KtDzFRKfW7ywYemrCSRw1vTQRK8hoILmLbD1MufNUOMDeRMa9axH99c5LCmuLFmoNca35ed/T5XaDUPidJDMd09ClDr2tc8jAs4pVmiOgHbJmOSzFDEHWi3UdSCncIIETva2DXH2sTIhHJ8r4uuaPwIBmetL0KZbBSBIJb191Q9btAUBDxe3TbWLq755gxHdlKq81xjHV5mhUBUB3njZu0KFpaz35GFURgYLewasLVT2g3AhovHq+zzSuCD5xVGFubZxmiXkPMoTBPwTV34wkBCwMQlXY/ukwoKtDD0SNiOJ5+EBwM0muIC2veSEmavMRiwlCKJqktOLGLUs04IMw8naV44ueOPqoltiltZHONeJpAq4IvSGgiE2XAxiKWXh7oX3PBOfUI1Szqcgbw0Ii4y5Y0YNuW/bnUTHPEPT1aZBTT+cP1jSKTxGe06IMR/WKPRR2ct+A3PgiyE69lugxw5LwlIn0j66 tGiX2T6y gKxdp2uSJ6mF7S1ul0IijZQwuPoUXtBL2fFRUki3suuD8sN45CD7OJymvFr/06M4OT48d9WLAFLzA2ipX18/9ZykrDDvUXYjloZgysb0QpIieySXYjX7hRv8Ak0Q5e31upxAUBwG/ydg/YE0Nv5GSCDGMRypU3TI6JSlxXoi/7T+B9BJl0lXNgybfYYHp5GdUny1mPanVzz1zEhCGIZ9wBmljPxIDbKoVm20v0OvKl+xXLEe5KvTZcq1uWgjFl0X5SPsCtfnSiZHiPzuF+hkyUC8m4ar5PTsjtEE0oPM/aswUDh5CKMEzWZTB7T6UcXtFMX6m7uxNgzzVHpDhqjOnbxqNupTyes9p8qfulA6OlUw5QB8ONKE2AQfdByhLWjnpnjMLtN9XZ1IfxZELMXTIPWZCVoE3TeeElU6EgwfYdmi6gk6zA4UFQu/Bxcwa+K4jnM2Hfcul3eRN6axH4n/mXk3vZEO3RAjveb1OigDshU7UpdrFTBX9jd6jjq+d9QfQIJo99nKs25dIEd71ge/5vPh2AXiZsQqxE1n72tiOEkEWDVHWd2VR6reN55OkbqWMRdK2PT7ZNAl0OagoCJJyKntTdPiKrQQAN3oFZ4b6GBKo7V9R8I1Dx1AYOumLykeW5GJFZo2WVzO5cpZrbWP6ObccBg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Implement kvm_arch_test_clear_young() to support the fast path in mmu_notifier_ops->test_clear_young(). It focuses on a simple case, i.e., hardware sets the accessed bit in KVM PTEs and VMs are not protected, where it can rely on RCU and cmpxchg to safely clear the accessed bit without taking kvm->mmu_lock. Complex cases fall back to the existing slow path where kvm->mmu_lock is then taken. Signed-off-by: Yu Zhao --- arch/arm64/include/asm/kvm_host.h | 6 ++++++ arch/arm64/kvm/mmu.c | 36 +++++++++++++++++++++++++++++++ 2 files changed, 42 insertions(+) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 7e7e19ef6993..da32b0890716 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -1113,4 +1113,10 @@ static inline void kvm_hyp_reserve(void) { } void kvm_arm_vcpu_power_off(struct kvm_vcpu *vcpu); bool kvm_arm_vcpu_stopped(struct kvm_vcpu *vcpu); +#define kvm_arch_has_test_clear_young kvm_arch_has_test_clear_young +static inline bool kvm_arch_has_test_clear_young(void) +{ + return cpu_has_hw_af() && !is_protected_kvm_enabled(); +} + #endif /* __ARM64_KVM_HOST_H__ */ diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index c3b3e2afe26f..26a8d955b49c 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -1678,6 +1678,42 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) range->start << PAGE_SHIFT); } +static int stage2_test_clear_young(const struct kvm_pgtable_visit_ctx *ctx, + enum kvm_pgtable_walk_flags flags) +{ + kvm_pte_t new = ctx->old & ~KVM_PTE_LEAF_ATTR_LO_S2_AF; + + VM_WARN_ON_ONCE(!page_count(virt_to_page(ctx->ptep))); + + if (!kvm_pte_valid(new)) + return 0; + + if (new == ctx->old) + return 0; + + if (kvm_should_clear_young(ctx->arg, ctx->addr / PAGE_SIZE)) + stage2_try_set_pte(ctx, new); + + return 0; +} + +bool kvm_arch_test_clear_young(struct kvm *kvm, struct kvm_gfn_range *range) +{ + u64 start = range->start * PAGE_SIZE; + u64 end = range->end * PAGE_SIZE; + struct kvm_pgtable_walker walker = { + .cb = stage2_test_clear_young, + .arg = range, + .flags = KVM_PGTABLE_WALK_LEAF | KVM_PGTABLE_WALK_SHARED, + }; + + BUILD_BUG_ON(is_hyp_code()); + + kvm_pgtable_walk(kvm->arch.mmu.pgt, start, end - start, &walker); + + return false; +} + phys_addr_t kvm_mmu_get_httbr(void) { return __pa(hyp_pgtable->pgd); From patchwork Fri May 26 23:44:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13257443 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2FD3CC7EE32 for ; Fri, 26 May 2023 23:44:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0F976280004; Fri, 26 May 2023 19:44:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0AD43280001; Fri, 26 May 2023 19:44:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E66BA280004; Fri, 26 May 2023 19:44:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D4C0B280001 for ; Fri, 26 May 2023 19:44:51 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id B6EBA160FC5 for ; Fri, 26 May 2023 23:44:51 +0000 (UTC) X-FDA: 80834038782.24.BF7E2F0 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf23.hostedemail.com (Postfix) with ESMTP id EC15114000E for ; Fri, 26 May 2023 23:44:49 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=Y62V38pt; spf=pass (imf23.hostedemail.com: domain of 3cURxZAYKCOgiejRKYQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--yuzhao.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3cURxZAYKCOgiejRKYQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1685144690; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Bguo3hUZaLYxse9v+ik+MX3ufqUdHR10jXNpIhPD3cE=; b=Rk/x/EQnCQgW77M+9mgqLGtllhQrakp5WuXpwAbk2weufeJ9nZN+Tpgx+GNs4E9lgxq1JR aCJrprgQcqF085Y8bbnlPgauSvZjcWuabnUsQZ3wUYM8mC1N7WirCTPJNsDbZ/eRo2Kpv4 6+U6NHXa5hbIQk6DVtHe+/meVjA2Ik0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1685144690; a=rsa-sha256; cv=none; b=fUHl8GqxtNvFRx1T1JgXcN8spQsnM8Y8kCdD3+wSTIYYGYVcW0pPyIqFDDN6RUmKsvQXkz sz6G7HIGYv1GbSplg5c4whG6EPm3qX1obiYyhfxWLMJydSgpKc2V8KJJws86rTBOKY0ecD 3tw159IqLi7mTyjJqmn4JUPputmtaiw= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=Y62V38pt; spf=pass (imf23.hostedemail.com: domain of 3cURxZAYKCOgiejRKYQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--yuzhao.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3cURxZAYKCOgiejRKYQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-ba2b9ecfadaso2800723276.2 for ; Fri, 26 May 2023 16:44:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1685144689; x=1687736689; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Bguo3hUZaLYxse9v+ik+MX3ufqUdHR10jXNpIhPD3cE=; b=Y62V38pt3Kf7P9M2YivEC5661SUHDQjGPJeLNpi65nB4y+p3TsqoRR0z/9AOIr1Kjf gl3yg0EBv9UJcyGGhkoV+eP5nxHHEckaVjNM+WV0ph8y6N8NKNrOoxejQzf6xVLF0GNF CMPzW7iyFKrk0I+YLOBH1dB2cwpR8z8XT9p+1tlyni+H85cDXJGB/ZGkaiDpbVC82/o/ GKwW1qcFloQI+vCY/C+X0I8xbKAWwMlWN0KpQsh6osmLARaX2g1tNlgMEbkEwKadQaPu 8s3poXNR7qoLyRgjp4108zRCziAP6oNak8GIUn6lV7gRCswVoQjlt0PKr+oTkaiyWyY0 Ciug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685144689; x=1687736689; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Bguo3hUZaLYxse9v+ik+MX3ufqUdHR10jXNpIhPD3cE=; b=H31dRLORFYTONfmSI10zKEamFgZ/Hg1vtAnWIiIJZG2YXgL8hwUUvPw+Ocm2RoGrTy b3K5s4m+tNlrumCkisygv1WzvU7WmO5bCFJRxPaakzm01AePPLKxeX9CLMy6fppROlCb o6wc2zuoa7NqqQytnuGfjMgDTaVv7YCHUmqFehSjd45CWuUkwo6PcA/oI5QoBKft/HGt ekVRco3bmkuTUIKrXiwfVDtnmgjtwrsJm6nN5uKqT1XWWO4cuvUd4qNMNLUC+n4/jW8C /UUig1txtUKlIxHBSSZHyW2ME74DTZ3IQVVm7zQN7Jd1bilUGTAoppwwVHKXIKLQkUxS WpQw== X-Gm-Message-State: AC+VfDyl+aGxp6cpsYwLPfFDImb2KbYqxfKw51WTM9Lfek2dO9rEqJJM ou8xtQ/xL3uwbiZ0S0dUZ8VuuSGKHvE= X-Google-Smtp-Source: ACHHUZ72Yke0umLVJsAYBjLDCmB2C1ZbHNslgBIJrMfaEGzX80HaCkEwKhsoBR2khvT0ady9wDj4p4x/v3M= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:910f:8a15:592b:2087]) (user=yuzhao job=sendgmr) by 2002:a25:3cb:0:b0:ba8:337a:d8a3 with SMTP id 194-20020a2503cb000000b00ba8337ad8a3mr1807757ybd.11.1685144689141; Fri, 26 May 2023 16:44:49 -0700 (PDT) Date: Fri, 26 May 2023 17:44:31 -0600 In-Reply-To: <20230526234435.662652-1-yuzhao@google.com> Message-Id: <20230526234435.662652-7-yuzhao@google.com> Mime-Version: 1.0 References: <20230526234435.662652-1-yuzhao@google.com> X-Mailer: git-send-email 2.41.0.rc0.172.g3f132b7071-goog Subject: [PATCH mm-unstable v2 06/10] kvm/powerpc: make radix page tables RCU safe From: Yu Zhao To: Andrew Morton , Paolo Bonzini Cc: Alistair Popple , Anup Patel , Ben Gardon , Borislav Petkov , Catalin Marinas , Chao Peng , Christophe Leroy , Dave Hansen , Fabiano Rosas , Gaosheng Cui , Gavin Shan , "H. Peter Anvin" , Ingo Molnar , James Morse , "Jason A. Donenfeld" , Jason Gunthorpe , Jonathan Corbet , Marc Zyngier , Masami Hiramatsu , Michael Ellerman , Michael Larabel , Mike Rapoport , Nicholas Piggin , Oliver Upton , Paul Mackerras , Peter Xu , Sean Christopherson , Steven Rostedt , Suzuki K Poulose , Thomas Gleixner , Thomas Huth , Will Deacon , Zenghui Yu , kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-trace-kernel@vger.kernel.org, x86@kernel.org, linux-mm@google.com, Yu Zhao X-Rspamd-Queue-Id: EC15114000E X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 7ap9hm7mp34j5bcq5cjqyxg38rcy69eq X-HE-Tag: 1685144689-120046 X-HE-Meta: U2FsdGVkX1+iYDX2St+BYOtGdcoEA+99VXlwRt7rOd3rFW7thVBKgdspwmdjUDUNYqVEkyM+paOJYmHTsf+zEjap5i2K5BUn9nqgMwVvUQwqXHsECrYqnUiZlAxbbranDsC+YCopujd/mu5oWksv5DwbiQeTXYI+BWaXWEWryKObXD6Q8EJ1ube6BD4dcwyMWBx/wdO/PCb1yN0T5acDtw4OR3KsK29ITzTaHt2CoV8bqgiqnPR2+6LbMIr7VGZQ48KRF0+nCdGPoAH/en2kFYBggME5TlzWJUcrKG3TM8X/bjPoRAff2lfmWzKbbusPfg0dcMjgUkw/yBRHx7rwQdsZ4qwaZrrH2mRD1AC3Xgor+pQFg7entVZw401JzZGMf0E0sTINkoqlD8h7nOOlnTx14GvYfEfxuJlTYets2uPlCSz2TGLLhUsIq919wkT/+w6fA51+QRXlwiOZgTh4gQySxfsheKUwJZh+tGNTMd3xEce7boPJaeVvKDlql0m/jODNXaDcqv10beQUfhcfuJGYYYgWvykQoEWkOU5J4lhvL7gSYBVq74YdwKKmW6cF4AoHRXdCXnHLwCdxEzqXZFGXSaiS3ZvT7lonVb05yzd/hkH6a4gvV8AuwxXaOWToXPOBhZp4kwduwvodGQXRIHDW0aCBjPs2JtjuRQZZ1R+xB1utZV1yBLQqeB+9dwcb1N+kqprYngUDASV5JrVhOGFTRxSbKEmKSbzZiHiIwaDlJFk5SOFl7lBlm37Nz0JJvq1Nry/9KE4vRizQzVqEUwW/Q+YS5NVzXKvhUppkwH7Rhg6xzXrKFyWjgmP3ehUdoXD/vbFDxlueCH8Wd3So3FZPspJYAi+uhICpj7KjKHzNHt38Q8qJz8CGgTG9i9DyUO6tTPpE/Ve4YsQnWpNsho+wVkkcroi6UHnXHnuvgH7fyhNmTiqN8LZsy8hoF0FhgzymyG5zMy1X1OCyt9T GA+m/Yem z/iQqFXb/khHSyE98n/WI+18QmfpPo/uYT+D/RwFyyN+OscGrWCiy3kcEg577vgQlohGruLkljYM+c+IGhao4/XxHlgSmgTsOcwFpYFnZRZkuB3UyB1h6hT0u7hmhhs6Y4OmpasKyy6Abo16GgP5/aiXWjOmH2KvCqWoYILmld2hKEoRgpeldZs8xBdP6Rxwxm9QKdPaX6qXhBEP5EgQ/u66lPNraAoQUZ4vaAhgkCsL2BrcN7u1TfqZiEdhGWxvtvEqepi375scRt/wb549T12HBQB9YCMpPH5RRNJsRTz5nb1j0qhawoUMikyrTxO0fmsxYT2CA8XWa9l+zPA+aUgTOQQ0HvgfY34k5DCeJBpReyy4J/Irj5qTNFvGg9MewMFNObqbNdTy2NN+xEewSTGjbl+/Z6L7kwyDTXhA22z+jS0Q6Bu4iSLSKKgi3IN2UU35rKr1uTWOcHRVjfVFZt5R99v1XJOfxhNLdyzwkjFsBEkpbOP6hmSI7nnGpEHld5KxyfHtzTJb6RwiREGcWOJ9Z6gaqnA/5VlPu1ga8xbyRjpL3aAySB5D6x0IrQxgCMaUqTgJhZkLFymI5lBnDMpTR8sJYTyLGtQJZ7bst7FVfY/I= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: KVM page tables are currently not RCU safe against remapping, i.e., kvmppc_unmap_free_pmd_entry_table() et al. The previous mmu_notifier_ops members rely on kvm->mmu_lock to synchronize with that operation. However, the new mmu_notifier_ops member test_clear_young() provides a fast path that does not take kvm->mmu_lock. To implement kvm_arch_test_clear_young() for that path, orphan page tables need to be freed by RCU. Unmapping, specifically kvm_unmap_radix(), does not free page tables, hence not a concern. Signed-off-by: Yu Zhao --- arch/powerpc/kvm/book3s_64_mmu_radix.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/arch/powerpc/kvm/book3s_64_mmu_radix.c b/arch/powerpc/kvm/book3s_64_mmu_radix.c index 461307b89c3a..3b65b3b11041 100644 --- a/arch/powerpc/kvm/book3s_64_mmu_radix.c +++ b/arch/powerpc/kvm/book3s_64_mmu_radix.c @@ -1469,13 +1469,15 @@ int kvmppc_radix_init(void) { unsigned long size = sizeof(void *) << RADIX_PTE_INDEX_SIZE; - kvm_pte_cache = kmem_cache_create("kvm-pte", size, size, 0, pte_ctor); + kvm_pte_cache = kmem_cache_create("kvm-pte", size, size, + SLAB_TYPESAFE_BY_RCU, pte_ctor); if (!kvm_pte_cache) return -ENOMEM; size = sizeof(void *) << RADIX_PMD_INDEX_SIZE; - kvm_pmd_cache = kmem_cache_create("kvm-pmd", size, size, 0, pmd_ctor); + kvm_pmd_cache = kmem_cache_create("kvm-pmd", size, size, + SLAB_TYPESAFE_BY_RCU, pmd_ctor); if (!kvm_pmd_cache) { kmem_cache_destroy(kvm_pte_cache); return -ENOMEM; From patchwork Fri May 26 23:44:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13257444 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 93A9AC7EE37 for ; Fri, 26 May 2023 23:44:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D2E5D280005; Fri, 26 May 2023 19:44:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CB7D3280001; Fri, 26 May 2023 19:44:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B587E280005; Fri, 26 May 2023 19:44:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id A90AE280001 for ; Fri, 26 May 2023 19:44:53 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 83F641C7EF5 for ; Fri, 26 May 2023 23:44:53 +0000 (UTC) X-FDA: 80834038866.11.EB0B97A Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf27.hostedemail.com (Postfix) with ESMTP id C6C2340002 for ; Fri, 26 May 2023 23:44:51 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=k8BERP9A; spf=pass (imf27.hostedemail.com: domain of 3ckRxZAYKCOkjfkSLZRZZRWP.NZXWTYfi-XXVgLNV.ZcR@flex--yuzhao.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3ckRxZAYKCOkjfkSLZRZZRWP.NZXWTYfi-XXVgLNV.ZcR@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1685144691; a=rsa-sha256; cv=none; b=FKhb5+QFgF8b0WBUbkouYW4DTsEiQLpOUgFSFeAvTXYfOmHA9EKO9o3hPj1ljcFFf0uKJw oUCZknVXR47bwEuL5uC1QaIBdpBjMFX3tEtqGB9tT8j8cxikxx23kCobpeVH3hmt/JMxVE qyXEeGeNwQlpRI2XrDaeftMPJmUjmzk= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=k8BERP9A; spf=pass (imf27.hostedemail.com: domain of 3ckRxZAYKCOkjfkSLZRZZRWP.NZXWTYfi-XXVgLNV.ZcR@flex--yuzhao.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3ckRxZAYKCOkjfkSLZRZZRWP.NZXWTYfi-XXVgLNV.ZcR@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1685144691; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SAnqMQmXLaJfeZI+VeXqIwvB0doaLL+gp8U5/HUId5I=; b=4BLJuD+qnpvdMqkXGRkRz56J4DW6G7YwZgDwQP0w0M4rA5WrKbhFsNaHxEP5zwE2qa1KDE Iy+wpt1VajCDH7ECXKlKwVTRGVCljVv6c8An5IoYA36IjGptqYiMiC/ShejdhFounUXtz0 OeIovAn5a+xf5lY8XP7+2izSZq9ZiO4= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-bacd408046cso2802351276.3 for ; Fri, 26 May 2023 16:44:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1685144691; x=1687736691; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=SAnqMQmXLaJfeZI+VeXqIwvB0doaLL+gp8U5/HUId5I=; b=k8BERP9A+6Uiw+Uaig20mMncTvjKAsRCKq+HC63zo5f14BiSqxPWFPVFrxvd4dUkfQ bKRY2Xn6Sj5cpDeqFqXaOaTxVBSBg6TnHG9z0eetRbywYU3ie/H8/CJOp1+Ez2wWWiIG CY1cPbCWNOAs/19D6iZC6kEHwAF0m7RMLVET0mpluU6zBeVTNhcDLR1hhBJpslaBZ/Nm 0fQXpAKGqnmpqrKCUVaNiYpgSsJF+6ODE5plgt1dvul/aWCITvYKB/5mkPp/iio/HINg 59adq2fXr8dYaO8RPmc4VSfW2nQVaqhTR9xX4O6C0F0j/4E6MdCu9N0KSqKFRTup9DJ6 82/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685144691; x=1687736691; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=SAnqMQmXLaJfeZI+VeXqIwvB0doaLL+gp8U5/HUId5I=; b=Q+g3fMYyEouYSdH8eKs0a2BiAhHg5UeihnL/xVmYDDly7pUZv4C7/q43r5vQ8N0ADa Rm6pwbTO5MZVb9JQiqbYH6sSXl/NhMUupQSkAVeAbKlBardPC1/HCe8vftpXAW9PXtLv DAlfODM2qA0gcrE7Ohaj26VzALA09vZadfk+7lw+k2FOiLGVnIFLPZSUC4U0Q7Dj7hLk ZW3Deiuf6f7ZPr+Qa6vigD+tEVBl9Yi04JjB172jQsay2xUijmJA9deidD80DwFwIua/ hHZ4YaAl8n0MF3U7Fj/mjCXZI1h0TRby23YlFRl9tnRwfBT1rrZqyGDZQqQmSRjZfwJ/ y7yg== X-Gm-Message-State: AC+VfDw3ApYOQZvToug9IX5mGSkT7WY4wN9Go9HwN+hdo1XYccFHfgYH Ksl1dl2pnwlHRX3Hh3JxKLBknNMRDeQ= X-Google-Smtp-Source: ACHHUZ56f5xRS5UFm0fGZF6KKL2sEC5mGRlgoj/Qp341qhbzjJy6m3cfCKx+HZdIQ3EgPejfKdLHbTHVxEI= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:910f:8a15:592b:2087]) (user=yuzhao job=sendgmr) by 2002:a25:8211:0:b0:ba6:e7ee:bb99 with SMTP id q17-20020a258211000000b00ba6e7eebb99mr1850590ybk.12.1685144690927; Fri, 26 May 2023 16:44:50 -0700 (PDT) Date: Fri, 26 May 2023 17:44:32 -0600 In-Reply-To: <20230526234435.662652-1-yuzhao@google.com> Message-Id: <20230526234435.662652-8-yuzhao@google.com> Mime-Version: 1.0 References: <20230526234435.662652-1-yuzhao@google.com> X-Mailer: git-send-email 2.41.0.rc0.172.g3f132b7071-goog Subject: [PATCH mm-unstable v2 07/10] kvm/powerpc: add kvm_arch_test_clear_young() From: Yu Zhao To: Andrew Morton , Paolo Bonzini Cc: Alistair Popple , Anup Patel , Ben Gardon , Borislav Petkov , Catalin Marinas , Chao Peng , Christophe Leroy , Dave Hansen , Fabiano Rosas , Gaosheng Cui , Gavin Shan , "H. Peter Anvin" , Ingo Molnar , James Morse , "Jason A. Donenfeld" , Jason Gunthorpe , Jonathan Corbet , Marc Zyngier , Masami Hiramatsu , Michael Ellerman , Michael Larabel , Mike Rapoport , Nicholas Piggin , Oliver Upton , Paul Mackerras , Peter Xu , Sean Christopherson , Steven Rostedt , Suzuki K Poulose , Thomas Gleixner , Thomas Huth , Will Deacon , Zenghui Yu , kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-trace-kernel@vger.kernel.org, x86@kernel.org, linux-mm@google.com, Yu Zhao X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: C6C2340002 X-Stat-Signature: cdxxaoyoxm76dwie77o4ts8x6bi48i9a X-Rspam-User: X-HE-Tag: 1685144691-364560 X-HE-Meta: U2FsdGVkX18GzbxuDbWe7srPA+KMXKzipcfHatE+b2HnpVs8UZ5HShNc15tMjFjqrRQjwY3Q4W9amEVNwVJZkKxsmC7SSbyhlfFhTTGedzIWRVR56nf0QRQpram0DlN4iCAE7kU5/CkrVVhLmMrOJ1hxH8NJ9/sHu1ytuj7nmj0AXR4ieAxSVMyI9yRvc3t417dfly+glMyS0uqauQbCmtNGin1bEyo0uUIgeeD9Y9K25494mTjxEItx5uJvENYOZsXabV8TYML0ugaRne1Gf1KNYVXDRQwit/6yMPZrhC/VufZprY/S2XXivQgXLwBLDrcRzsD4zFV2w3xNMenk8NlwlKjUyk0L1PZBq7MoSGAfxe3QUZiCna2cHX5JVMvmMl0uoWvnuyTJWP3i0nhJFFgyBEXxanPeTkgCTAvM0UovsyKX6CxV9bQURPWLaxvpV0qTkInpoK/OuTXacwRhUpB4A5yDLYzFaSSCersOldzZjuunQ2qa+zaf4r2jBXIiNGD2UPlhhD4k3BVnnb6nxDKYTQAca9v9iP9e9UrjeYfvFe3VVkZDZY8tYqQCW1RCdoKhdapj+4RQ5MS5jNEel5UXWuVaCaVae37xJbauPlVe4t4wclnw2tyWPAEAmrGYUgJ8pxekE7JbX/esBL653lnDfagLvKoHwuNV1dDa7grvECvFnSfH46wVSfPQmGoiM5lKkG1/fmq7nmmJkT+1jBQrZlbMJWYJdEiX3QgJmncg+6kop1kadc+6diiDiJAe+vNmWUnDSgS+k+rLRokfW/AxDSaQ3IEv0JLNlfPJUNSi3Hg5Ndhf5zL3+bh0p1cqcKpGPq5wiWjJoLdtSBHsfJ16Bvy9esq4XQgQ6UmiCN/80QgfprzuV4lUTDidXi2sPi3g6yEBAbyUWxyw2a21ijVCiEQ1xr+pkzrkXQeChGiIjLWVPmuCqm9njScaHLb8o/dJ6roNKTrq8n9m95k x72ki8P6 Cmm8FybulAYe5RutEvBXsfeFSZqNsBUZ8bdi3vKO1W0DMXV4L2laFL/VPEjMQrx/r6z83K66Cz0sQ5Qca4eTtXyzyrgdaaIFa8TdsaJNe3qkqq9S9NKJUY9J1jEMf1xaSF1X/MZ0USwrjkDAyWiIYnXiQssh7o9BdAKjq4u9IlWA4yic19jTmjRIY4AaiAQblJx0rXFY3DbEfKwBtPjldX7SpJ3WXDSU3HurpcKyLIga8+5mDtz448yg8Yu9tcb+Hsi96zL1BDsuF345j0ARyO1TZiOJF5uOhrPGLADLZqShmJoyIYx6vn/He1EdWSJNyUaS9jWXZP9j0dmRxqG//gjOVdpdBr+TN1G7kc11LcajEhepvAYeho1Hh9bfuHEy5ogpalSgFcmB/QO3St0IBk0LPij0S9bXHAYhtDEknHDn4aoKQAMYgrFERWGTnB+rG5Zn5iY3yh/Hqz5f99jL7C6mlt9WItpAeHjS0jxR1xN7irenQiBAwvKCK4uztfTOIZsWSXY0hqoy6TBXi32HybUZyNUB2ATtYKHovnRRREm96iJnt4P6VPsyOR8Dnh87jUMK8zUn/oYuLwqT4A00zsdyFhKp6m7tNwLOTtFTGytwZg90JmftwTJ5ODpn7RfUboR5XDNqL6UQRxIKipuZM86z4XA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Implement kvm_arch_test_clear_young() to support the fast path in mmu_notifier_ops->test_clear_young(). It focuses on a simple case, i.e., radix MMU sets the accessed bit in KVM PTEs and VMs are not nested, where it can rely on RCU and pte_xchg() to safely clear the accessed bit without taking kvm->mmu_lock. Complex cases fall back to the existing slow path where kvm->mmu_lock is then taken. Signed-off-by: Yu Zhao --- arch/powerpc/include/asm/kvm_host.h | 8 ++++ arch/powerpc/include/asm/kvm_ppc.h | 1 + arch/powerpc/kvm/book3s.c | 6 +++ arch/powerpc/kvm/book3s.h | 1 + arch/powerpc/kvm/book3s_64_mmu_radix.c | 59 ++++++++++++++++++++++++++ arch/powerpc/kvm/book3s_hv.c | 5 +++ 6 files changed, 80 insertions(+) diff --git a/arch/powerpc/include/asm/kvm_host.h b/arch/powerpc/include/asm/kvm_host.h index 14ee0dece853..75c260ea8a9e 100644 --- a/arch/powerpc/include/asm/kvm_host.h +++ b/arch/powerpc/include/asm/kvm_host.h @@ -883,4 +883,12 @@ static inline void kvm_arch_sched_in(struct kvm_vcpu *vcpu, int cpu) {} static inline void kvm_arch_vcpu_blocking(struct kvm_vcpu *vcpu) {} static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {} +#define kvm_arch_has_test_clear_young kvm_arch_has_test_clear_young +static inline bool kvm_arch_has_test_clear_young(void) +{ + return IS_ENABLED(CONFIG_KVM_BOOK3S_HV_POSSIBLE) && + cpu_has_feature(CPU_FTR_HVMODE) && cpu_has_feature(CPU_FTR_ARCH_300) && + radix_enabled(); +} + #endif /* __POWERPC_KVM_HOST_H__ */ diff --git a/arch/powerpc/include/asm/kvm_ppc.h b/arch/powerpc/include/asm/kvm_ppc.h index 79a9c0bb8bba..ff1af6a7b44f 100644 --- a/arch/powerpc/include/asm/kvm_ppc.h +++ b/arch/powerpc/include/asm/kvm_ppc.h @@ -287,6 +287,7 @@ struct kvmppc_ops { bool (*unmap_gfn_range)(struct kvm *kvm, struct kvm_gfn_range *range); bool (*age_gfn)(struct kvm *kvm, struct kvm_gfn_range *range); bool (*test_age_gfn)(struct kvm *kvm, struct kvm_gfn_range *range); + bool (*test_clear_young)(struct kvm *kvm, struct kvm_gfn_range *range); bool (*set_spte_gfn)(struct kvm *kvm, struct kvm_gfn_range *range); void (*free_memslot)(struct kvm_memory_slot *slot); int (*init_vm)(struct kvm *kvm); diff --git a/arch/powerpc/kvm/book3s.c b/arch/powerpc/kvm/book3s.c index 686d8d9eda3e..37bf40b0c4ff 100644 --- a/arch/powerpc/kvm/book3s.c +++ b/arch/powerpc/kvm/book3s.c @@ -899,6 +899,12 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) return kvm->arch.kvm_ops->test_age_gfn(kvm, range); } +bool kvm_arch_test_clear_young(struct kvm *kvm, struct kvm_gfn_range *range) +{ + return !kvm->arch.kvm_ops->test_clear_young || + kvm->arch.kvm_ops->test_clear_young(kvm, range); +} + bool kvm_set_spte_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { return kvm->arch.kvm_ops->set_spte_gfn(kvm, range); diff --git a/arch/powerpc/kvm/book3s.h b/arch/powerpc/kvm/book3s.h index 58391b4b32ed..fa2659e21ccc 100644 --- a/arch/powerpc/kvm/book3s.h +++ b/arch/powerpc/kvm/book3s.h @@ -12,6 +12,7 @@ extern void kvmppc_core_flush_memslot_hv(struct kvm *kvm, extern bool kvm_unmap_gfn_range_hv(struct kvm *kvm, struct kvm_gfn_range *range); extern bool kvm_age_gfn_hv(struct kvm *kvm, struct kvm_gfn_range *range); extern bool kvm_test_age_gfn_hv(struct kvm *kvm, struct kvm_gfn_range *range); +extern bool kvm_test_clear_young_hv(struct kvm *kvm, struct kvm_gfn_range *range); extern bool kvm_set_spte_gfn_hv(struct kvm *kvm, struct kvm_gfn_range *range); extern int kvmppc_mmu_init_pr(struct kvm_vcpu *vcpu); diff --git a/arch/powerpc/kvm/book3s_64_mmu_radix.c b/arch/powerpc/kvm/book3s_64_mmu_radix.c index 3b65b3b11041..0a392e9a100a 100644 --- a/arch/powerpc/kvm/book3s_64_mmu_radix.c +++ b/arch/powerpc/kvm/book3s_64_mmu_radix.c @@ -1088,6 +1088,65 @@ bool kvm_test_age_radix(struct kvm *kvm, struct kvm_memory_slot *memslot, return ref; } +bool kvm_test_clear_young_hv(struct kvm *kvm, struct kvm_gfn_range *range) +{ + bool err; + gfn_t gfn = range->start; + + rcu_read_lock(); + + err = !kvm_is_radix(kvm); + if (err) + goto unlock; + + /* + * Case 1: This function kvmppc_switch_mmu_to_hpt() + * + * rcu_read_lock() + * Test kvm_is_radix() kvm->arch.radix = 0 + * Use kvm->arch.pgtable synchronize_rcu() + * rcu_read_unlock() + * kvmppc_free_radix() + * + * + * Case 2: This function kvmppc_switch_mmu_to_radix() + * + * kvmppc_init_vm_radix() + * smp_wmb() + * Test kvm_is_radix() kvm->arch.radix = 1 + * smp_rmb() + * Use kvm->arch.pgtable + */ + smp_rmb(); + + while (gfn < range->end) { + pte_t *ptep; + pte_t old, new; + unsigned int shift; + + ptep = find_kvm_secondary_pte_unlocked(kvm, gfn * PAGE_SIZE, &shift); + if (!ptep) + goto next; + + VM_WARN_ON_ONCE(!page_count(virt_to_page(ptep))); + + old = READ_ONCE(*ptep); + if (!pte_present(old) || !pte_young(old)) + goto next; + + new = pte_mkold(old); + + if (kvm_should_clear_young(range, gfn)) + pte_xchg(ptep, old, new); +next: + gfn += shift ? BIT(shift - PAGE_SHIFT) : 1; + } +unlock: + rcu_read_unlock(); + + return err; +} + /* Returns the number of PAGE_SIZE pages that are dirty */ static int kvm_radix_test_clear_dirty(struct kvm *kvm, struct kvm_memory_slot *memslot, int pagenum) diff --git a/arch/powerpc/kvm/book3s_hv.c b/arch/powerpc/kvm/book3s_hv.c index 130bafdb1430..20a81ec9fde8 100644 --- a/arch/powerpc/kvm/book3s_hv.c +++ b/arch/powerpc/kvm/book3s_hv.c @@ -5262,6 +5262,8 @@ int kvmppc_switch_mmu_to_hpt(struct kvm *kvm) spin_lock(&kvm->mmu_lock); kvm->arch.radix = 0; spin_unlock(&kvm->mmu_lock); + /* see the comments in kvm_test_clear_young_hv() */ + synchronize_rcu(); kvmppc_free_radix(kvm); lpcr = LPCR_VPM1; @@ -5286,6 +5288,8 @@ int kvmppc_switch_mmu_to_radix(struct kvm *kvm) if (err) return err; kvmppc_rmap_reset(kvm); + /* see the comments in kvm_test_clear_young_hv() */ + smp_wmb(); /* Mutual exclusion with kvm_unmap_gfn_range etc. */ spin_lock(&kvm->mmu_lock); kvm->arch.radix = 1; @@ -6185,6 +6189,7 @@ static struct kvmppc_ops kvm_ops_hv = { .unmap_gfn_range = kvm_unmap_gfn_range_hv, .age_gfn = kvm_age_gfn_hv, .test_age_gfn = kvm_test_age_gfn_hv, + .test_clear_young = kvm_test_clear_young_hv, .set_spte_gfn = kvm_set_spte_gfn_hv, .free_memslot = kvmppc_core_free_memslot_hv, .init_vm = kvmppc_core_init_vm_hv, From patchwork Fri May 26 23:44:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13257445 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 49E86C83005 for ; Fri, 26 May 2023 23:44:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F2E5F280006; Fri, 26 May 2023 19:44:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EB3CA280001; Fri, 26 May 2023 19:44:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D547F280006; Fri, 26 May 2023 19:44:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id C4E29280001 for ; Fri, 26 May 2023 19:44:55 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 8A44012030A for ; Fri, 26 May 2023 23:44:55 +0000 (UTC) X-FDA: 80834038950.27.5226FB2 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf22.hostedemail.com (Postfix) with ESMTP id C8EB9C0006 for ; Fri, 26 May 2023 23:44:53 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="Ne/YRQ/s"; spf=pass (imf22.hostedemail.com: domain of 3dERxZAYKCOslhmUNbTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--yuzhao.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3dERxZAYKCOslhmUNbTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1685144693; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7JLLUnbPXAaoH9i/N14KC8FRF4yfK0mZvbZkcTBOq5Y=; b=oHv4RBY9+cz67bWDu3JLTE6mPB2ZW6V4vudB2QxHP8n+IgsL3Osaut6bypsdwZqqvrq3rj PzDIxVUZMGFIMzKK2aXACQA1fBvZE641gsdts194B8dEjGzaqFdEssmQNNjOJs/me78oXc wGhiAuahxSEEEDeoUtyC9t1/WTACsTk= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="Ne/YRQ/s"; spf=pass (imf22.hostedemail.com: domain of 3dERxZAYKCOslhmUNbTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--yuzhao.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3dERxZAYKCOslhmUNbTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1685144693; a=rsa-sha256; cv=none; b=1YYlVPQQAZfSKM60lTEDGP0j47v/YIF9vx/JdSuJTxbcmtgWAUnJ8Kw0hbWOEVfAUTbCGO UogATFSEfn1HArBoYEVU+dxwtdsKVNQNFgD6eeWf8aGy9k7diRG8s/2gYRDtErnACmbPS1 yl+Jut87GbiElgfVNJX37ZzekpB01eo= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-ba8cf175f5bso5006623276.0 for ; Fri, 26 May 2023 16:44:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1685144693; x=1687736693; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=7JLLUnbPXAaoH9i/N14KC8FRF4yfK0mZvbZkcTBOq5Y=; b=Ne/YRQ/svpJFGXhXpgHmwxc0D9cLMxIoz8hghy2YEExILLlaUzGTTLTu8b32WLIGJt Pl2W5ENMW5Sa2Kwrgh0yVLYa4j/IXKl/wxvEFqusCYfl9i2LCglCwg4qFOwnu/SiZlAZ epK5zgrczZWcQF0/2oLgZ8laj6CsPXbNpnBrvrF3XhoGhjELY5RWTIQgLbNJgOlO3uxt 4d+XiEr2KggZNk0D2a/3M0F1IRHplA5m+1NXrAN0tP918Y2nokL0n9zRLbfOWAMyaJ2Q beEb9W4X5NKj/DpGHg7ZQVC9QnQZ7wDRATscsNx8htwusD9k+VmLxpoV1j1SSPGezxSx 58Rg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685144693; x=1687736693; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=7JLLUnbPXAaoH9i/N14KC8FRF4yfK0mZvbZkcTBOq5Y=; b=W0TUccfdljpkuke+jUh2tr2yXMSbj6AScokeH2cax7sK+J+edjJSSvScJ2+cr/awWA GAti20dIaK6+FhyKPbmUaSIn1Z1gvb+hCIAIMe9+hKSDCyPw+/ki3KfV3Iovy6QdzQV6 47cAlDaKwsyMJgaIEtpee6EzdHIa7g95RghHLnrnMnxnAyA/t7s/sC6twYWRGjmahINm 4j+IozW119Ky0jUUW85JL1BVmEFzwiHVvU20/g7jrsZAwOH3nfopGYopXJ5rL/VNIBZ7 y60maNTrCItJCuT92R5+nYykbnaj0Q9C/l7JE0aoMlWEAd0r4M8KmrWDO0oiBoni2Rzd s9Tg== X-Gm-Message-State: AC+VfDx/YTH2pWuvI+SghHcLI3l3AmeIjqmFzx5krgxwrNNR7Kz1iMb/ bZHDCWvPIJcLk2Wpp5nB8c/QerYGzSI= X-Google-Smtp-Source: ACHHUZ5CZlpEmT2VkUcPhwDn9jTP7EScM/+97DNWp/Z/n1uD6F2Bv8HNZfloYPpxuZT6MB0BvBWbDvtVv+8= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:910f:8a15:592b:2087]) (user=yuzhao job=sendgmr) by 2002:a5b:9c6:0:b0:ba8:381b:f764 with SMTP id y6-20020a5b09c6000000b00ba8381bf764mr354063ybq.3.1685144692904; Fri, 26 May 2023 16:44:52 -0700 (PDT) Date: Fri, 26 May 2023 17:44:33 -0600 In-Reply-To: <20230526234435.662652-1-yuzhao@google.com> Message-Id: <20230526234435.662652-9-yuzhao@google.com> Mime-Version: 1.0 References: <20230526234435.662652-1-yuzhao@google.com> X-Mailer: git-send-email 2.41.0.rc0.172.g3f132b7071-goog Subject: [PATCH mm-unstable v2 08/10] kvm/x86: move tdp_mmu_enabled and shadow_accessed_mask From: Yu Zhao To: Andrew Morton , Paolo Bonzini Cc: Alistair Popple , Anup Patel , Ben Gardon , Borislav Petkov , Catalin Marinas , Chao Peng , Christophe Leroy , Dave Hansen , Fabiano Rosas , Gaosheng Cui , Gavin Shan , "H. Peter Anvin" , Ingo Molnar , James Morse , "Jason A. Donenfeld" , Jason Gunthorpe , Jonathan Corbet , Marc Zyngier , Masami Hiramatsu , Michael Ellerman , Michael Larabel , Mike Rapoport , Nicholas Piggin , Oliver Upton , Paul Mackerras , Peter Xu , Sean Christopherson , Steven Rostedt , Suzuki K Poulose , Thomas Gleixner , Thomas Huth , Will Deacon , Zenghui Yu , kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-trace-kernel@vger.kernel.org, x86@kernel.org, linux-mm@google.com, Yu Zhao X-Rspamd-Queue-Id: C8EB9C0006 X-Rspam-User: X-Stat-Signature: e643kxugz45kpec8z7un5hzsfnht1nky X-Rspamd-Server: rspam01 X-HE-Tag: 1685144693-372945 X-HE-Meta: U2FsdGVkX19BZZjnLEQH1cVIRBuanOClxP3+AjekSpERrLSmZJ4i1er98h6tLsHr2RBQhqv7n/qf4y7+Xv/qBqVCO85jcf0h3dbpBbu/qFf/GPexzF3BXG4r9oLGV2z9NxuNHKZglgvIgrjugkT/BrOx++r6wI79doDTTRyJ8fyj7ffNpOs1sxNln4KeQoD0AvvOV+uf+Zenag/zwdI8WFg354XOP7a4fylFiRVNS1IAYS0omIy1Lr47WqgkURwRyTvSmWX8S9LC9f8XwLgNcmekZIe4Og1NGs4ISkap1WOzMt6Xciw+DgJa1cYnfjW7kCsjLvZMepA7IAlUi1RR+KNd/gnWC7heXNOnajlltrQnwAPZOWZeXMWn3mJ6AD639G8SuMyxTE6F9hMfFjJffIavCyhA087lkASEfDTnow9ypCRr9dGJUl1h5obEme2JWfK/MPEopRsk6KBqz38K7vNbmWW6fL5yDr1PG3EFX4BPyGnyWT12f17UyXiwnZ9Fo3oOaF/V6Bxqjwpf1rEVfypvaLFcrfjJDjdwq1tsqHUZMnmYZNA60KdZ+EJmXpjtAQqXCp/p5QO/HuyWa7L8bRLpD5uRb8RSa7f0x8AMjWkx6pJ+QWqqGADzJPLTsqJratUsnMjKTZFSSphn51d34M664spItm/jrWJ/C7lNmtinNGqAUqworeqOkTjWD7L5L6nZHRBKM2hCC7Vnz4XNiPBlt255SRBUU+34ruOeBFYMzQeb+3poS6nxijFLDzzC2hghT+TPCId31cmwFbVOaqiTeF6x8ZA0IQP+qzE3fNvQ2RfnlV/XQ14kiZoqmf5JhB59mIhQ7vTul6Lu0a9hYYkla7ek2oHpkvlYEnOwL0/MBXpNwFbUYoGtyGwz3oVcbTijF9NPJF3quBhJ2GlcR3Tx+lIEtuliXo44kM8js0ClmQF+PvBIwkOM3dCqmRuAWGNik1IzHptEGFHkgkn umD7lsZh mfWW+5n8IfJMLqe08vt0hc39y4jQRh+UK3G485iThpL8I+nW82U2DXs4Y7votMovOnf5PfPMVpHFl6jpm6JqQtH6b2wRo/eaTtjy4YbPJAWgEwVYzffnPcDIZ5+/36WoUmrmkAUlGODdRpeKaGWIiXc1ixehNtdftLGTIBPFY/aQCMf/PLXm9+T4H641I+4808wNw3CIxPkF2Q5OqAeGzRQJLwsY1YjdFkzslX75E++AFauwTm5PxkRxxRhvgJ7lFgsn2EQVlmYV6MZ/N+tS14j2h8KbMVHpcLANiDGT1lLWvPHhJrbsPuYGXXzHOWTE7zsq7r+aOTSQiY5zLEMX5OKAYS7GPqyxDmopeTOdPPN9853JrrE2ZphrLmUdQZ4n6v+N2aOAs73t0+NKUeyVcHyjYtVv8hFPpuYb07wYWlecVgs2ueZM0Z/XOqnQW1BdlVHKWaMvzJERdiisGuOY/Dnqpo61/pM8ksqVxRX/nIZSbOd9vk24oHHfiZQc6sWzI5JiFNrCFIL0Jubqk20bkGR+47BuZyBr/FIqrIS8ngDj4UL8DceXCWcFkaPsbXZ/sEYzEM15AtkNo2MjIzVgr2pUNsYUv26kxXxocFhKNxo/h3pDC/ECqZL/PKX7CgB/ekt3cW/uZ+pf2Z65AWYKwnq/BoA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: tdp_mmu_enabled and shadow_accessed_mask are needed to implement kvm_arch_has_test_clear_young(). Signed-off-by: Yu Zhao --- arch/x86/include/asm/kvm_host.h | 6 ++++++ arch/x86/kvm/mmu.h | 6 ------ arch/x86/kvm/mmu/spte.h | 1 - 3 files changed, 6 insertions(+), 7 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index fb9d1f2d6136..753c67072c47 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1772,6 +1772,7 @@ struct kvm_arch_async_pf { extern u32 __read_mostly kvm_nr_uret_msrs; extern u64 __read_mostly host_efer; +extern u64 __read_mostly shadow_accessed_mask; extern bool __read_mostly allow_smaller_maxphyaddr; extern bool __read_mostly enable_apicv; extern struct kvm_x86_ops kvm_x86_ops; @@ -1855,6 +1856,11 @@ void kvm_fire_mask_notifiers(struct kvm *kvm, unsigned irqchip, unsigned pin, bool mask); extern bool tdp_enabled; +#ifdef CONFIG_X86_64 +extern bool tdp_mmu_enabled; +#else +#define tdp_mmu_enabled false +#endif u64 vcpu_tsc_khz(struct kvm_vcpu *vcpu); diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h index 92d5a1924fc1..84aedb2671ef 100644 --- a/arch/x86/kvm/mmu.h +++ b/arch/x86/kvm/mmu.h @@ -253,12 +253,6 @@ static inline bool kvm_shadow_root_allocated(struct kvm *kvm) return smp_load_acquire(&kvm->arch.shadow_root_allocated); } -#ifdef CONFIG_X86_64 -extern bool tdp_mmu_enabled; -#else -#define tdp_mmu_enabled false -#endif - static inline bool kvm_memslots_have_rmaps(struct kvm *kvm) { return !tdp_mmu_enabled || kvm_shadow_root_allocated(kvm); diff --git a/arch/x86/kvm/mmu/spte.h b/arch/x86/kvm/mmu/spte.h index 1279db2eab44..a82c4fa1c47b 100644 --- a/arch/x86/kvm/mmu/spte.h +++ b/arch/x86/kvm/mmu/spte.h @@ -153,7 +153,6 @@ extern u64 __read_mostly shadow_mmu_writable_mask; extern u64 __read_mostly shadow_nx_mask; extern u64 __read_mostly shadow_x_mask; /* mutual exclusive with nx_mask */ extern u64 __read_mostly shadow_user_mask; -extern u64 __read_mostly shadow_accessed_mask; extern u64 __read_mostly shadow_dirty_mask; extern u64 __read_mostly shadow_mmio_value; extern u64 __read_mostly shadow_mmio_mask; From patchwork Fri May 26 23:44:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13257446 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DACB2C77B7C for ; Fri, 26 May 2023 23:44:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4F29D280007; Fri, 26 May 2023 19:44:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 47A01280001; Fri, 26 May 2023 19:44:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 31B44280007; Fri, 26 May 2023 19:44:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 1ECF8280001 for ; Fri, 26 May 2023 19:44:58 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id EBADF80ED3 for ; Fri, 26 May 2023 23:44:57 +0000 (UTC) X-FDA: 80834039034.15.428470A Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf26.hostedemail.com (Postfix) with ESMTP id 1F036140015 for ; Fri, 26 May 2023 23:44:55 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=yrD5cypj; spf=pass (imf26.hostedemail.com: domain of 3d0RxZAYKCO4okpXQeWeeWbU.SecbYdkn-ccalQSa.ehW@flex--yuzhao.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3d0RxZAYKCO4okpXQeWeeWbU.SecbYdkn-ccalQSa.ehW@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1685144696; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=M3AF7fBmc1xnJAMOiZdfCsfXZHiS6oPk41ReBM4zbdo=; b=cgUS1gbK+cfXOfJayyYXaz7eXXDqmzUxRhu0WMEPTfVqwQikGU3jQgBdZK5c63vxkP4tBO +CYRK/LCZ9SL8L4hBy+NbnM1VPZibAADTQKgrI7Le8KtlCQfLGsGzOqTJVTamblnE9A6vl X+iXJ5IbgdMn3BE+BhuZju4W+11Kjj0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1685144696; a=rsa-sha256; cv=none; b=SYWmc0edeldSQ3scI1GSFCl9VZSZMmOV7vx3+iKmhHSIRG3TKblpWEHbXj1AhKgsU0JoTO sOtbSjQWYQktfSaM23G0jC/DL2+TxrAMHkiwk1RdJ1eoc/f6uQYdRoGNeZvxyfUkmx08XD vGnPTJFTh7gmXUJTeSEYAgIzxtq3PJE= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=yrD5cypj; spf=pass (imf26.hostedemail.com: domain of 3d0RxZAYKCO4okpXQeWeeWbU.SecbYdkn-ccalQSa.ehW@flex--yuzhao.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3d0RxZAYKCO4okpXQeWeeWbU.SecbYdkn-ccalQSa.ehW@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-bad0f5d6a7dso2621781276.0 for ; Fri, 26 May 2023 16:44:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1685144695; x=1687736695; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=M3AF7fBmc1xnJAMOiZdfCsfXZHiS6oPk41ReBM4zbdo=; b=yrD5cypjwj76oGySeMKoL+EObdL+VlHBR0ZE4dlUQxoYbTEWiiVpSKdOJmkcaaiF/Z 4zilNg4E7S1qiA0OC/fMHUKvaq4lLe3h3XrcfBRajVZEzuTWsMjYGvjMBJ/wRyLbmfd/ x0p59R3lRKLAmUF0+sTBQhD8rRciT1hrCi2rbco+ObNSFPCgz7YI6gzDi295X90OlgMM zrJNip6i/8aKO3FHb/HgogYFOfPOgx60+DOedoHxymQQDAPdmrus+K28IPt1M7E0g0cI asbBmGj30I6PgAgE5VquUWTvHUX2/5VYaFLlsw0xBC7qUsoM7lDeTwedn+JU6nSVVIPC w2tA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685144695; x=1687736695; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=M3AF7fBmc1xnJAMOiZdfCsfXZHiS6oPk41ReBM4zbdo=; b=ZyigEojnWAnyX0whqY+mYMzdgkaBB8znBriLON0THAw4OfyrwBaX185M0WIuOkpiqQ 0x01oz4w2ZQ4rULYJaCOsaydbWjj/8KDN0ZG4kv53gMzcz6kr+LtcGCekmS+UHfGx2t4 DyT4qXhiH3TLcHDsav7Ipz0K1phLNGBU57ooOk8AErpXlXUik7VOTkd+bKMotDFGt7wK oNo/Km+Lbll7O+t99e/Vx8Eg2hnmOWbTPu9BgptvmFsOS1O3qYmQDmPTHsKSmybTImoP XWh1Xy9YBlYuGdZJb318PCZUmWMaZfSzCMXW17D7qoIxgoWk5c/FLJEZyjjPAhWJwy8M Egog== X-Gm-Message-State: AC+VfDyF18SGEV9gKBU500nZyNyU6uAL7HBIkpk4jzRZtAw/W8AKUKvx AhU2+2e+7whJQvcF/DNThtoDywci7Dc= X-Google-Smtp-Source: ACHHUZ4QgTvrucO0Q6McE6sU/ZYyF1Ujw9/eAceyjgq3u2aouYR16g/56Sc2t/zSHX7sxuPwIu+K0d0WjYg= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:910f:8a15:592b:2087]) (user=yuzhao job=sendgmr) by 2002:a05:6902:1341:b0:bac:6bb:2549 with SMTP id g1-20020a056902134100b00bac06bb2549mr1840571ybu.7.1685144695170; Fri, 26 May 2023 16:44:55 -0700 (PDT) Date: Fri, 26 May 2023 17:44:34 -0600 In-Reply-To: <20230526234435.662652-1-yuzhao@google.com> Message-Id: <20230526234435.662652-10-yuzhao@google.com> Mime-Version: 1.0 References: <20230526234435.662652-1-yuzhao@google.com> X-Mailer: git-send-email 2.41.0.rc0.172.g3f132b7071-goog Subject: [PATCH mm-unstable v2 09/10] kvm/x86: add kvm_arch_test_clear_young() From: Yu Zhao To: Andrew Morton , Paolo Bonzini Cc: Alistair Popple , Anup Patel , Ben Gardon , Borislav Petkov , Catalin Marinas , Chao Peng , Christophe Leroy , Dave Hansen , Fabiano Rosas , Gaosheng Cui , Gavin Shan , "H. Peter Anvin" , Ingo Molnar , James Morse , "Jason A. Donenfeld" , Jason Gunthorpe , Jonathan Corbet , Marc Zyngier , Masami Hiramatsu , Michael Ellerman , Michael Larabel , Mike Rapoport , Nicholas Piggin , Oliver Upton , Paul Mackerras , Peter Xu , Sean Christopherson , Steven Rostedt , Suzuki K Poulose , Thomas Gleixner , Thomas Huth , Will Deacon , Zenghui Yu , kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-trace-kernel@vger.kernel.org, x86@kernel.org, linux-mm@google.com, Yu Zhao X-Rspamd-Queue-Id: 1F036140015 X-Rspam-User: X-Stat-Signature: qgcc6ipecptgtotwudxhk1j55dkr6cpi X-Rspamd-Server: rspam03 X-HE-Tag: 1685144695-580725 X-HE-Meta: U2FsdGVkX18iNr9TsD8L4rw2Nkoemy2GZX2ET9azkj+B3HN2uudG6pIiKTq9IILIXmQTldmMWtA7pEykDpmr/HmvGPubNTalqq/da53TGJacTb389RTg9ckdGK1Nyg1Vn6+iUPVFNloE6S2LaPtnWXH+pCoxZU2v3LUVsaGMdNA6VyAdO3uEW17kbJ8xEI4mYCS0qFSzOpfUrtfo+yxELg8pbIH95thocL0ZFUC2TN9Zp0eGJs4GSBnScb7V5IX8kBoY+v03KsWNjI+UjiG/ZGQLpNdVVJ+CZt2TTP169vZKBxV3McpCffTlk88blJrhc1U8/NwNOh26yVQ9yYkTeA5JIoNXCNi9lDZdWP0t61WuGxAoFy4xlVF6Jmtw3Kaf2a1gAjHhhST8zZLQPW6KVFZgUywbpwYax9+8DYFcuEj1Wv2IePisOhkLNG0k7ex59lG2e2LUp9SdL/8cfYCZfGvFtB2zTRnKh1YMHmxFG5Rvj9EXDQ9UbhX3hrqlbH+rHpsjAtYalyMFYnnbV4UvNos7M6xtE7AfWhQX5HN9tntV3vMsc56979nHgOL5OPzDiMlWha9ryvIXduWwRRIlTBsQMT3PtmEkMxSoN8TDfoikCmLV8/YzOmshVMJci1diRQRrE29UQDEK2P+1SeT8duOHG9IS+VDz/VsaC7AZPKIKi1WNa2tGXQZdaUmmO/QOG0z0bPy2yB8+kypeMJXDStCOPBpD/ISkrUrSiLOSaeRtaT+P6ws1AXQHZgrL22u6QpC6YXr+58esrVE3T9XbyJHbJeSk7nDWfu4GNLwfnGXuPK+40ZLFwNLxbjcY1opMatgcUzKY7QkYBJT39at6ZZrDmKAQ0Nw8VBJttSRl88xv83T5XH6p8Q31Ku9lArpfBa8V9dzQGSyzv5px7nopD/gx+lOw1JlqH2aBU62mk1ZcG9BiRfYfOdPexBuzRMIGsi0biL7aw1uXqCJKaa4 y/k+EOlN XUiOpZY8ISzjP2zJ5kVoDj8Fc9VPxRGeGk3h4j/x7FS67pXdknCMhtAxlDnt03Cw/bjhtd5MzoOwOJH7g+fW7QRMnXX4+XocO33hXWvV13+yxILnaRs/UrzLUEZ60Ssofd673yPcmqzEhlqObzqyDdnvQ1IPOJvv4mLY2S/1VGK6zImxPriWXi5p4oiqyGWzKB13/aqs/+yVSI67+8I/8AGFNAPTyo3ADj36YKGpd5SefSIrAtFtu21HarEfX7Fp3for4msTsAyJLvyQjEEQzq7apV8sNRepFjdOgrmsFXDkQTlzZveyb8dEPqtyRWO3Lu/LZ53n9L+XHgaqpp+S6mtJtIIQ6SU4qSmjqBZ5lTt3qkdifhush39ij0qXNONlGA4F+XgsKJvtNg7QTOhz3qGbGjhAbuqr8ss2Xt5+PyF7t7ZIMqO36XBfoj2ckUI1rk5DCl7TJaTaLzyEGARESwnzXErKsDxFEzQEWKX+3JmkZ6qujSCeiyFVJHKt+PPNssPg19C0XdlQELD+kPEEzPCdVY+EDZhvsyVwI76YIsNIYQUloaDi9G7FSXcIZsiJqbz4zOH2vbYeTec//+ipeEPiY4MJWAnTcTzvAWOAPoM+gCQXfHTWTxxHZyWeJJCDOXmz+w/Hv2WKKAN0rJmOzz1AGiQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Implement kvm_arch_test_clear_young() to support the fast path in mmu_notifier_ops->test_clear_young(). It focuses on a simple case, i.e., TDP MMU sets the accessed bit in KVM PTEs and VMs are not nested, where it can rely on RCU and clear_bit() to safely clear the accessed bit without taking kvm->mmu_lock. Complex cases fall back to the existing slow path where kvm->mmu_lock is then taken. Signed-off-by: Yu Zhao --- arch/x86/include/asm/kvm_host.h | 7 +++++++ arch/x86/kvm/mmu/tdp_mmu.c | 34 +++++++++++++++++++++++++++++++++ 2 files changed, 41 insertions(+) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 753c67072c47..d6dfdebe3d94 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -2223,4 +2223,11 @@ int memslot_rmap_alloc(struct kvm_memory_slot *slot, unsigned long npages); */ #define KVM_EXIT_HYPERCALL_MBZ GENMASK_ULL(31, 1) +#define kvm_arch_has_test_clear_young kvm_arch_has_test_clear_young +static inline bool kvm_arch_has_test_clear_young(void) +{ + return IS_ENABLED(CONFIG_X86_64) && + (!IS_REACHABLE(CONFIG_KVM) || (tdp_mmu_enabled && shadow_accessed_mask)); +} + #endif /* _ASM_X86_KVM_HOST_H */ diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 08340219c35a..6875a819e007 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -1232,6 +1232,40 @@ bool kvm_tdp_mmu_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) return kvm_tdp_mmu_handle_gfn(kvm, range, test_age_gfn); } +bool kvm_arch_test_clear_young(struct kvm *kvm, struct kvm_gfn_range *range) +{ + struct kvm_mmu_page *root; + int offset = ffs(shadow_accessed_mask) - 1; + + if (kvm_shadow_root_allocated(kvm)) + return true; + + rcu_read_lock(); + + list_for_each_entry_rcu(root, &kvm->arch.tdp_mmu_roots, link) { + struct tdp_iter iter; + + if (kvm_mmu_page_as_id(root) != range->slot->as_id) + continue; + + tdp_root_for_each_leaf_pte(iter, root, range->start, range->end) { + u64 *sptep = rcu_dereference(iter.sptep); + + VM_WARN_ON_ONCE(!page_count(virt_to_page(sptep))); + + if (!(iter.old_spte & shadow_accessed_mask)) + continue; + + if (kvm_should_clear_young(range, iter.gfn)) + clear_bit(offset, (unsigned long *)sptep); + } + } + + rcu_read_unlock(); + + return false; +} + static bool set_spte_gfn(struct kvm *kvm, struct tdp_iter *iter, struct kvm_gfn_range *range) { From patchwork Fri May 26 23:44:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13257447 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4470AC7EE33 for ; Fri, 26 May 2023 23:45:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 62D4D280008; Fri, 26 May 2023 19:45:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 58E10280001; Fri, 26 May 2023 19:45:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 42E96280008; Fri, 26 May 2023 19:45:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 32698280001 for ; Fri, 26 May 2023 19:45:00 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 097A41C7EF5 for ; Fri, 26 May 2023 23:45:00 +0000 (UTC) X-FDA: 80834039160.04.D62225B Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf26.hostedemail.com (Postfix) with ESMTP id 3494D140013 for ; Fri, 26 May 2023 23:44:58 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="Mb/TwMrv"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf26.hostedemail.com: domain of 3eURxZAYKCPAqmrZSgYggYdW.Ugedafmp-eecnSUc.gjY@flex--yuzhao.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3eURxZAYKCPAqmrZSgYggYdW.Ugedafmp-eecnSUc.gjY@flex--yuzhao.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1685144698; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/RAbaZ7nXlJx1O5X4p6tDmC38Mqs0IR+WFkcAEID/ew=; b=LO9WqiHBwiUqZfU8PPr6i7LZBwTlb0jr1Bq9pErphpiHgSMAuwvAWkxnT0bZTaDk5skeD7 IpVNNv2/gxa0mVgM51/u6nktYfbBem6mAvV6ypyNkcPcTo6KoYdyC4ABmpFuYyMRGQ/uVU 4ZQhiKKCO5REMrVqzyGoJv/gUiur8k8= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="Mb/TwMrv"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf26.hostedemail.com: domain of 3eURxZAYKCPAqmrZSgYggYdW.Ugedafmp-eecnSUc.gjY@flex--yuzhao.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3eURxZAYKCPAqmrZSgYggYdW.Ugedafmp-eecnSUc.gjY@flex--yuzhao.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1685144698; a=rsa-sha256; cv=none; b=gmosSy3IhCbdx1jcwUCVPGEFtLe++9ACvIhvW/5UfVCfpgS51zQ+zaK7S4G0eUVfuFxAwY rB6ng+4muZoWfhCAqAjS2umaxD7szCPGyp/ZfRbpx2ZNKr21qqn88cNS/NMcFoCg+rQbUV 6fgWAKbE0qtx3ZtTuLP5msBj1qGZkOM= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-56552a72cfbso30180937b3.3 for ; Fri, 26 May 2023 16:44:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1685144697; x=1687736697; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=/RAbaZ7nXlJx1O5X4p6tDmC38Mqs0IR+WFkcAEID/ew=; b=Mb/TwMrvjKVwx825EpuGBZelUYT9y1q5MsHWLZEtcnYl8b+S/FmG3PWbpa3sjl3UbU EMKVg4Z0PKkBIeb7T3roSS0cZqaFmagaFcdMijquvBPxR+8cgeB8lhD+sxDLIAW92iZI 96AhG517DtCX6M6wF+UEXQXmnvh+bt6+TMXBvO0EA6fZL0y2SwCLgjF2MAZnONkhWo1g K/7v7mGu5qj4MRpb1kQzn2rxfgDVCTzMnRuNoX97x4mDqSBErwlSjFlQrZ/jaGBbYaIg V4TtJk4J+bx9E13IjCR46uiaNAT2c4P7QhpLAYBCKMArI7L7xnoEXLbituEB7FbRo2Qm 2ytw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685144697; x=1687736697; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/RAbaZ7nXlJx1O5X4p6tDmC38Mqs0IR+WFkcAEID/ew=; b=j9jEh4G+euHKRuhdAfnyfsQzlLegbzM2jEA0MzmWDFOEdCbGR48BJXh2f8qkQEXHXi XzuUzim3WU6hQmTBD7Ov8DzrtSH0Jy6v8si+JmRV5NGQ7l2ejwpl/3wki/DjHjAsPmaw TdzyrQc9LgT9oeXcnnqBzjTf2zYF1vC764yVwfRrh7zHS4gQJY3JkBRbkAUDS07r1mxr sMUi7YFPbZkCH2ZzVUomAYs5MkuzslpJOfFrPe1Sn/ePNQvv58kkC86rIH7bZyIu3k9A H+Qu6mrsnZD99he7dzNIRQIs/PVqjfqDhugQ6MJ3Bq4LvhyhyfS47K+OTznEkGjE0rn3 e8EQ== X-Gm-Message-State: AC+VfDxbXUCQ3Us4YhiJS7aqedz3Ij/46/AoPANNodKMlzX9/AJLMSXb oCqvXpnx6B/wAWFd5oqybGJOY//mUJs= X-Google-Smtp-Source: ACHHUZ5acKzCjiqXNcqUiyX95r/zGZ1H0IqWVk5xr51BxFwYKtaOrUj82KqB/feXRTG2Ot17Gvh9/gGxmKY= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:910f:8a15:592b:2087]) (user=yuzhao job=sendgmr) by 2002:a81:c542:0:b0:561:1d3b:af3f with SMTP id o2-20020a81c542000000b005611d3baf3fmr2060709ywj.8.1685144697389; Fri, 26 May 2023 16:44:57 -0700 (PDT) Date: Fri, 26 May 2023 17:44:35 -0600 In-Reply-To: <20230526234435.662652-1-yuzhao@google.com> Message-Id: <20230526234435.662652-11-yuzhao@google.com> Mime-Version: 1.0 References: <20230526234435.662652-1-yuzhao@google.com> X-Mailer: git-send-email 2.41.0.rc0.172.g3f132b7071-goog Subject: [PATCH mm-unstable v2 10/10] mm: multi-gen LRU: use mmu_notifier_test_clear_young() From: Yu Zhao To: Andrew Morton , Paolo Bonzini Cc: Alistair Popple , Anup Patel , Ben Gardon , Borislav Petkov , Catalin Marinas , Chao Peng , Christophe Leroy , Dave Hansen , Fabiano Rosas , Gaosheng Cui , Gavin Shan , "H. Peter Anvin" , Ingo Molnar , James Morse , "Jason A. Donenfeld" , Jason Gunthorpe , Jonathan Corbet , Marc Zyngier , Masami Hiramatsu , Michael Ellerman , Michael Larabel , Mike Rapoport , Nicholas Piggin , Oliver Upton , Paul Mackerras , Peter Xu , Sean Christopherson , Steven Rostedt , Suzuki K Poulose , Thomas Gleixner , Thomas Huth , Will Deacon , Zenghui Yu , kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-trace-kernel@vger.kernel.org, x86@kernel.org, linux-mm@google.com, Yu Zhao X-Rspamd-Queue-Id: 3494D140013 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: zrzz1usbima3pe93sp5fwxnekts1ko5x X-HE-Tag: 1685144698-996771 X-HE-Meta: U2FsdGVkX1+5KP+wpR3/8l7Q/TYQWi2lQ98r97IFladVFmejEEUS1ycUvEG1A5pPM+yAFoEHQsf2BvZ4qJNP52m/VZNWwnO0nbewTEW9J5+ARIyfNWWxGfpz6BOSnEJxqDwXtHLHM4N6hiYPsn98WEz2cNY6ts6RMs72WVroMl3Ef41h0wpuKsgpsHBlCYFtYqX6W+/ZwEg0CEKLebPBkOfzx/H8H4G8sySE7+ZbTKS2EZZ4BHfQ17H9crDDEoR0iIU2c4r0D9Cj2IJg7ACKDT4eXRYzfLYDkKNds+y45NcpzdmtOXaQfnETFfSjNY9NUmNqU4poQqil9WN2PLPlZIFnP14hwwTKEjN8rjC3wBNydSFtgzP/qFxVX8sL4LwuGR1pEoKbcrebpCvHjTxRW8EI74AGdIzO4UNC+NBKzF6krkQHx+CXjgfQkGWNuKNJRrpH0p2wKDtyR0o5icZ2V141JLsGGHTDi9YE9MYmp1iC5Kr0+d4wD/1WqAp0MxMKpQSRSz55s9VDnP4B8KIbKGZBO9eImM30OO2OARUsU5gTreUWbiID79xmPWfxqqOGEBwtOdhforD9tsrrdpfKELcUGIG6S4m2UvnPmbPXRm9hxRgMQoOhszteBL0yXUphI1qD2ms8+WRFGGOOlfz7gCmC+czxdX5h5t+4WSPDU6lyOk7Bu/c97Z7HBmY4seXl+7lN9Og+9Q9S8r0VPi//J+x+a9cK/1w/UP0rwx7aWcPhm6pxT5fAhlaNWTkwlTnvzkfW3Cna1OdwCZuDc6yUF7kNQOtrpUT5pmVvIHcRe6HXRZBv2ihUr+H9wEvSNzKfE2MBRoHExbV/10ZW5ATX/gSDePGxvUktkWfT5BLDY+BnkJWckaTODOrHMWKdVEmXsuGjQw68+gctEKaorski5/0TrWw/xJDnRHtyhtG9aeGiaXwthqyzBL8/eR6EGKSNmpkr3mtlIKmlDLfcfIj RUF1X3VY V68Dzd7oNbA5zmfU6lVcDEKLqxnkJxYq7PfWrmgXT5IqPkFshVddvetrQpewTCYONZr/ZQKHaJGKpEIfwddOEQokvrUIfA/jxXmWw3eTQFFi7+s4BpstJGC6SNAxg+PEqDEM6uDZ3PXFoRLxbbmU3shbAUSSfzB6bhpcPl4hDaItwpnuYlDEH18P+236Evvho+/DGIqIvZ53BKOCMvJB+Kb8zQ8oJ+oYIzGyvcKejEMHPxx+kEgOZ///aHqEQU7sXjimvreLwxJOHbwR+QKPNzLwqxLX1Z/Nr5JfX4pSQZtWXcj61rWv1xZh9ksiuVlSWqnBmYfeSEWuzI2eJ9mWCv0l4cR7T8C8gcJJg5B5d3E/lHoCqCO7mVILsULm4xGHJLHBLM5R6NVlmDPj6hmZzXD7R31gZ5RlLFIWBfOUWPy1pxM4GC0iMtQXQo7Y2nfETMMR6pSbKZycgWJdO0lbaghbCOy93PZBvhAme+Ht6iyH3eQ58mEJpPerKPXkg45fFgGYQyXe5Zg0b2L6A6KAqtkqlNPifGQXtUbSXhJjjiqeBbdIkgD8DWloHxeDz5Unf2j7N/TtJzI1L2Jh5wUPe47B4dXtqw8Msl6lSP5Ji4WzNDrXuuQzbyI7eJmg/M7ZTv/MiyGYu0II1rW/UDtYmxJf3oIKZFxdAJLS6Nweh1wWU+w0YT/gsPSzJ/TuCGeDEzPQ8IVhLD9kqG9Nn+YizWIh3a79LfjjtXCpZ73ZlpoveuIY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use mmu_notifier_test_clear_young() to handle KVM PTEs in batches when the fast path is supported. This reduces the contention on kvm->mmu_lock when the host is under heavy memory pressure. An existing selftest can quickly demonstrate the effectiveness of this patch. On a generic workstation equipped with 128 CPUs and 256GB DRAM: $ sudo max_guest_memory_test -c 64 -m 250 -s 250 MGLRU run2 ------------------ Before [1] ~64s After ~51s kswapd (MGLRU before) 100.00% balance_pgdat 100.00% shrink_node 100.00% shrink_one 99.99% try_to_shrink_lruvec 99.71% evict_folios 97.29% shrink_folio_list ==>> 13.05% folio_referenced 12.83% rmap_walk_file 12.31% folio_referenced_one 7.90% __mmu_notifier_clear_young 7.72% kvm_mmu_notifier_clear_young 7.34% _raw_write_lock kswapd (MGLRU after) 100.00% balance_pgdat 100.00% shrink_node 100.00% shrink_one 99.99% try_to_shrink_lruvec 99.59% evict_folios 80.37% shrink_folio_list ==>> 3.74% folio_referenced 3.59% rmap_walk_file 3.19% folio_referenced_one 2.53% lru_gen_look_around 1.06% __mmu_notifier_test_clear_young [1] "mm: rmap: Don't flush TLB after checking PTE young for page reference" was included so that the comparison is apples to apples. https://lore.kernel.org/r/20220706112041.3831-1-21cnbao@gmail.com/ Signed-off-by: Yu Zhao --- Documentation/admin-guide/mm/multigen_lru.rst | 6 +- include/linux/mmzone.h | 6 +- mm/rmap.c | 8 +- mm/vmscan.c | 139 ++++++++++++++++-- 4 files changed, 138 insertions(+), 21 deletions(-) diff --git a/Documentation/admin-guide/mm/multigen_lru.rst b/Documentation/admin-guide/mm/multigen_lru.rst index 33e068830497..0ae2a6d4d94c 100644 --- a/Documentation/admin-guide/mm/multigen_lru.rst +++ b/Documentation/admin-guide/mm/multigen_lru.rst @@ -48,6 +48,10 @@ Values Components verified on x86 varieties other than Intel and AMD. If it is disabled, the multi-gen LRU will suffer a negligible performance degradation. +0x0008 Clearing the accessed bit in KVM page table entries in large + batches, when KVM MMU sets it (e.g., on x86_64). This can + improve the performance of guests when the host is under memory + pressure. [yYnN] Apply to all the components above. ====== =============================================================== @@ -56,7 +60,7 @@ E.g., echo y >/sys/kernel/mm/lru_gen/enabled cat /sys/kernel/mm/lru_gen/enabled - 0x0007 + 0x000f echo 5 >/sys/kernel/mm/lru_gen/enabled cat /sys/kernel/mm/lru_gen/enabled 0x0005 diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 5a7ada0413da..1b148f39fabc 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -369,6 +369,7 @@ enum { LRU_GEN_CORE, LRU_GEN_MM_WALK, LRU_GEN_NONLEAF_YOUNG, + LRU_GEN_KVM_MMU_WALK, NR_LRU_GEN_CAPS }; @@ -471,7 +472,7 @@ struct lru_gen_mm_walk { }; void lru_gen_init_lruvec(struct lruvec *lruvec); -void lru_gen_look_around(struct page_vma_mapped_walk *pvmw); +bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw); #ifdef CONFIG_MEMCG @@ -559,8 +560,9 @@ static inline void lru_gen_init_lruvec(struct lruvec *lruvec) { } -static inline void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) +static inline bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw) { + return false; } #ifdef CONFIG_MEMCG diff --git a/mm/rmap.c b/mm/rmap.c index ae127f60a4fb..3a2c4e6a0887 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -825,12 +825,10 @@ static bool folio_referenced_one(struct folio *folio, return false; /* To break the loop */ } - if (pvmw.pte) { - if (lru_gen_enabled() && pte_young(*pvmw.pte)) { - lru_gen_look_around(&pvmw); + if (lru_gen_enabled() && pvmw.pte) { + if (lru_gen_look_around(&pvmw)) referenced++; - } - + } else if (pvmw.pte) { if (ptep_clear_flush_young_notify(vma, address, pvmw.pte)) referenced++; diff --git a/mm/vmscan.c b/mm/vmscan.c index ef687f9be13c..3f734588b600 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -58,6 +58,7 @@ #include #include #include +#include #include #include @@ -3244,6 +3245,20 @@ static bool should_clear_pmd_young(void) return arch_has_hw_nonleaf_pmd_young() && get_cap(LRU_GEN_NONLEAF_YOUNG); } +#if IS_ENABLED(CONFIG_KVM) +#include + +static bool should_walk_kvm_mmu(void) +{ + return kvm_arch_has_test_clear_young() && get_cap(LRU_GEN_KVM_MMU_WALK); +} +#else +static bool should_walk_kvm_mmu(void) +{ + return false; +} +#endif + /****************************************************************************** * shorthand helpers ******************************************************************************/ @@ -3982,6 +3997,55 @@ static struct folio *get_pfn_folio(unsigned long pfn, struct mem_cgroup *memcg, return folio; } +static bool test_spte_young(struct mm_struct *mm, unsigned long addr, unsigned long end, + unsigned long *bitmap, unsigned long *last) +{ + if (!should_walk_kvm_mmu()) + return false; + + if (*last > addr) + goto done; + + *last = end - addr > MIN_LRU_BATCH * PAGE_SIZE ? + addr + MIN_LRU_BATCH * PAGE_SIZE - 1 : end - 1; + bitmap_zero(bitmap, MIN_LRU_BATCH); + + mmu_notifier_test_clear_young(mm, addr, *last + 1, false, bitmap); +done: + return test_bit((*last - addr) / PAGE_SIZE, bitmap); +} + +static void clear_spte_young(struct mm_struct *mm, unsigned long addr, + unsigned long *bitmap, unsigned long *last) +{ + int i; + unsigned long start, end = *last + 1; + + if (addr + PAGE_SIZE != end) + return; + + i = find_last_bit(bitmap, MIN_LRU_BATCH); + if (i == MIN_LRU_BATCH) + return; + + start = end - (i + 1) * PAGE_SIZE; + + i = find_first_bit(bitmap, MIN_LRU_BATCH); + + end -= i * PAGE_SIZE; + + mmu_notifier_test_clear_young(mm, start, end, true, bitmap); +} + +static void skip_spte_young(struct mm_struct *mm, unsigned long addr, + unsigned long *bitmap, unsigned long *last) +{ + if (*last > addr) + __clear_bit((*last - addr) / PAGE_SIZE, bitmap); + + clear_spte_young(mm, addr, bitmap, last); +} + static bool suitable_to_scan(int total, int young) { int n = clamp_t(int, cache_line_size() / sizeof(pte_t), 2, 8); @@ -3997,6 +4061,8 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, pte_t *pte; spinlock_t *ptl; unsigned long addr; + DECLARE_BITMAP(bitmap, MIN_LRU_BATCH); + unsigned long last = 0; int total = 0; int young = 0; struct lru_gen_mm_walk *walk = args->private; @@ -4015,6 +4081,7 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, pte = pte_offset_map(pmd, start & PMD_MASK); restart: for (i = pte_index(start), addr = start; addr != end; i++, addr += PAGE_SIZE) { + bool ret; unsigned long pfn; struct folio *folio; @@ -4022,20 +4089,27 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, walk->mm_stats[MM_LEAF_TOTAL]++; pfn = get_pte_pfn(pte[i], args->vma, addr); - if (pfn == -1) + if (pfn == -1) { + skip_spte_young(args->vma->vm_mm, addr, bitmap, &last); continue; + } - if (!pte_young(pte[i])) { + ret = test_spte_young(args->vma->vm_mm, addr, end, bitmap, &last); + if (!ret && !pte_young(pte[i])) { + skip_spte_young(args->vma->vm_mm, addr, bitmap, &last); walk->mm_stats[MM_LEAF_OLD]++; continue; } folio = get_pfn_folio(pfn, memcg, pgdat, walk->can_swap); - if (!folio) + if (!folio) { + skip_spte_young(args->vma->vm_mm, addr, bitmap, &last); continue; + } - if (!ptep_test_and_clear_young(args->vma, addr, pte + i)) - VM_WARN_ON_ONCE(true); + clear_spte_young(args->vma->vm_mm, addr, bitmap, &last); + if (pte_young(pte[i])) + ptep_test_and_clear_young(args->vma, addr, pte + i); young++; walk->mm_stats[MM_LEAF_YOUNG]++; @@ -4629,6 +4703,23 @@ static void lru_gen_age_node(struct pglist_data *pgdat, struct scan_control *sc) * rmap/PT walk feedback ******************************************************************************/ +static bool should_look_around(struct vm_area_struct *vma, unsigned long addr, + pte_t *pte, int *young) +{ + int ret = mmu_notifier_clear_young(vma->vm_mm, addr, addr + PAGE_SIZE); + + if (pte_young(*pte)) { + ptep_test_and_clear_young(vma, addr, pte); + *young = true; + return true; + } + + if (ret) + *young = true; + + return ret & MMU_NOTIFIER_RANGE_LOCKLESS; +} + /* * This function exploits spatial locality when shrink_folio_list() walks the * rmap. It scans the adjacent PTEs of a young PTE and promotes hot pages. If @@ -4636,12 +4727,14 @@ static void lru_gen_age_node(struct pglist_data *pgdat, struct scan_control *sc) * the PTE table to the Bloom filter. This forms a feedback loop between the * eviction and the aging. */ -void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) +bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw) { int i; unsigned long start; unsigned long end; struct lru_gen_mm_walk *walk; + DECLARE_BITMAP(bitmap, MIN_LRU_BATCH); + unsigned long last = 0; int young = 0; pte_t *pte = pvmw->pte; unsigned long addr = pvmw->address; @@ -4655,8 +4748,11 @@ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) lockdep_assert_held(pvmw->ptl); VM_WARN_ON_ONCE_FOLIO(folio_test_lru(folio), folio); + if (!should_look_around(pvmw->vma, addr, pte, &young)) + return young; + if (spin_is_contended(pvmw->ptl)) - return; + return young; /* avoid taking the LRU lock under the PTL when possible */ walk = current->reclaim_state ? current->reclaim_state->mm_walk : NULL; @@ -4664,6 +4760,9 @@ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) start = max(addr & PMD_MASK, pvmw->vma->vm_start); end = min(addr | ~PMD_MASK, pvmw->vma->vm_end - 1) + 1; + if (end - start == PAGE_SIZE) + return young; + if (end - start > MIN_LRU_BATCH * PAGE_SIZE) { if (addr - start < MIN_LRU_BATCH * PAGE_SIZE / 2) end = start + MIN_LRU_BATCH * PAGE_SIZE; @@ -4677,28 +4776,37 @@ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) /* folio_update_gen() requires stable folio_memcg() */ if (!mem_cgroup_trylock_pages(memcg)) - return; + return young; arch_enter_lazy_mmu_mode(); pte -= (addr - start) / PAGE_SIZE; for (i = 0, addr = start; addr != end; i++, addr += PAGE_SIZE) { + bool ret; unsigned long pfn; pfn = get_pte_pfn(pte[i], pvmw->vma, addr); - if (pfn == -1) + if (pfn == -1) { + skip_spte_young(pvmw->vma->vm_mm, addr, bitmap, &last); continue; + } - if (!pte_young(pte[i])) + ret = test_spte_young(pvmw->vma->vm_mm, addr, end, bitmap, &last); + if (!ret && !pte_young(pte[i])) { + skip_spte_young(pvmw->vma->vm_mm, addr, bitmap, &last); continue; + } folio = get_pfn_folio(pfn, memcg, pgdat, !walk || walk->can_swap); - if (!folio) + if (!folio) { + skip_spte_young(pvmw->vma->vm_mm, addr, bitmap, &last); continue; + } - if (!ptep_test_and_clear_young(pvmw->vma, addr, pte + i)) - VM_WARN_ON_ONCE(true); + clear_spte_young(pvmw->vma->vm_mm, addr, bitmap, &last); + if (pte_young(pte[i])) + ptep_test_and_clear_young(pvmw->vma, addr, pte + i); young++; @@ -4728,6 +4836,8 @@ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) /* feedback from rmap walkers to page table walkers */ if (suitable_to_scan(i, young)) update_bloom_filter(lruvec, max_seq, pvmw->pmd); + + return young; } /****************************************************************************** @@ -5745,6 +5855,9 @@ static ssize_t enabled_show(struct kobject *kobj, struct kobj_attribute *attr, c if (should_clear_pmd_young()) caps |= BIT(LRU_GEN_NONLEAF_YOUNG); + if (should_walk_kvm_mmu()) + caps |= BIT(LRU_GEN_KVM_MMU_WALK); + return sysfs_emit(buf, "0x%04x\n", caps); }