From patchwork Mon Apr 1 23:29:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13613090 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70472CD1288 for ; Mon, 1 Apr 2024 23:30:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6ABFE6B0092; Mon, 1 Apr 2024 19:30:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 683886B0093; Mon, 1 Apr 2024 19:30:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4FC2C6B0095; Mon, 1 Apr 2024 19:30:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 34CC96B0092 for ; Mon, 1 Apr 2024 19:30:00 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 0107F403E9 for ; Mon, 1 Apr 2024 23:29:59 +0000 (UTC) X-FDA: 81962558160.25.B6001D8 Received: from mail-vk1-f202.google.com (mail-vk1-f202.google.com [209.85.221.202]) by imf14.hostedemail.com (Postfix) with ESMTP id 5CAF310000F for ; Mon, 1 Apr 2024 23:29:58 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=oIV+2i66; spf=pass (imf14.hostedemail.com: domain of 3dUMLZgoKCGAHRFMSEFRMLEMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--jthoughton.bounces.google.com designates 209.85.221.202 as permitted sender) smtp.mailfrom=3dUMLZgoKCGAHRFMSEFRMLEMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712014198; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XjV19huKRvwcv3bjf5GpogmLoEGdyfdviROgtmpM2BI=; b=8panYJlSyTmBE6He4UNolYp8+LXgrHmxd6h17h4tOLtwowfhXZm4uDm4T46JL0gRGl3w0I uX3PaxUnjXVMZYAa/R+cqsOj2MmdpeLaFbEmVgC0U9KHE9bkVmD3Tk9EUpgg/K6HoodHo0 SNCpbDwz7FQbx+4E4uJsHuVuKUsi9Fk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712014198; a=rsa-sha256; cv=none; b=IlYRpnPwyjuwcwRNNL3gNdSoAYv+3zsNL55SinfQPyqfpQzt1M/e/vv6r5n7MipAeCEKBA WMF9oNCBH3eNA+SblKjRYjN1dc5Mf54wvx4ysbFao4c3Zz3qWQG5gLWgCLsa9GrLaWfzBX 8y0phIeBr4uiHAYtvehmVXbD5yU29LM= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=oIV+2i66; spf=pass (imf14.hostedemail.com: domain of 3dUMLZgoKCGAHRFMSEFRMLEMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--jthoughton.bounces.google.com designates 209.85.221.202 as permitted sender) smtp.mailfrom=3dUMLZgoKCGAHRFMSEFRMLEMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-vk1-f202.google.com with SMTP id 71dfb90a1353d-4d8a322b233so1782827e0c.2 for ; Mon, 01 Apr 2024 16:29:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1712014197; x=1712618997; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=XjV19huKRvwcv3bjf5GpogmLoEGdyfdviROgtmpM2BI=; b=oIV+2i66RPf9s1bjNqFu3PNQb6zEliTY13Ct+pPI5O3NmlUuXKKMpAZSTm3b08fR0Q l/V0nl7MZ7wDdb81s8Kk9DRVk8FDkYJhS04a5RGRpS/MVAmfN7sU/9SZsdBRX2u5bFc8 +dGGKYxKeIoVRkC3UmLgth8h8Kj+gL/1Fl1jCRNw2qnIl4c115hYLrNFCEgAiI8mZ49z HMvk5/jPbKtdOX7u5QLTDcKqH1qD5a1CQKPt7RHjdS+pfRcsmmqHLhsxnBJE5aydOlo6 kkjipZ6DrDrzYIAaedAoTYtD6WpXoyFiNNvIMycU1VoQ50/UUPmujg9GahqRGWZOW+97 cuww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712014197; x=1712618997; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=XjV19huKRvwcv3bjf5GpogmLoEGdyfdviROgtmpM2BI=; b=QPa9/mlIgQtthwtJ4zdq64uAk/Vy1t8Cm+u+gpk0R1x1k6agHfq3bLlIZ00PrHH7DX 3PBZccRQ+s7hlzXgHTnNxcQHgcuvduc+G6+bd/VUDD0IFDleTk1shnm3H0VduD8QJjVu fBd7BytI2B4/SusJTEgC7uuPTZrM6yHzXV8s6/P2398jZxRNOs2MWA2x1lcOTXEYtEYo 6ctlManaqJg4ibPzZx9NEcSd8ZmOeftXisH4qjMQgp+Q4otBvkqPCsJyEHnr8/jBKVTd RBX3rtVJLzPDDr7pPvIrrxGWx3xbJc4mTgrvjmx6nr47IXecvNzJ/DeAkVANarqsA5c+ uZhg== X-Forwarded-Encrypted: i=1; AJvYcCWJEalqqC2cRJJne5PixYY6GkRFzxkpPLgWUsi3GzWidQIjRiw87tGveVuBHLuKgN9I7z/wwicf6+RYzKm4Pu1Q9X0= X-Gm-Message-State: AOJu0YzdsXOmxKqiaFBERcIEb5nBRYFtp/2WrwxJ/7nwM3tT1DHgJPPy vU+eG209jWN+WBtLT0CqMlEVln4y9pYPZnDoxtKpGU3wKdehppNZS7DganrH4J9UWF2iohQro51 GSsPl20Xhv1FMFQaKpg== X-Google-Smtp-Source: AGHT+IFUopEMKakeSmmDQQkpzKxlNJ65mvpQqylZZQK+yo+BzjzMVCQHtrhU8U2Q57twwQQQfTybPo3O3BawFKgE X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:ac5:c281:0:b0:4d8:73c1:9ec7 with SMTP id h1-20020ac5c281000000b004d873c19ec7mr84232vkk.0.1712014197363; Mon, 01 Apr 2024 16:29:57 -0700 (PDT) Date: Mon, 1 Apr 2024 23:29:40 +0000 In-Reply-To: <20240401232946.1837665-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240401232946.1837665-1-jthoughton@google.com> X-Mailer: git-send-email 2.44.0.478.gd926399ef9-goog Message-ID: <20240401232946.1837665-2-jthoughton@google.com> Subject: [PATCH v3 1/7] mm: Add a bitmap into mmu_notifier_{clear,test}_young From: James Houghton To: Andrew Morton , Paolo Bonzini Cc: Yu Zhao , David Matlack , Marc Zyngier , Oliver Upton , Sean Christopherson , Jonathan Corbet , James Morse , Suzuki K Poulose , Zenghui Yu , Catalin Marinas , Will Deacon , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers , Shaoqin Huang , Gavin Shan , Ricardo Koller , Raghavendra Rao Ananta , Ryan Roberts , David Rientjes , Axel Rasmussen , linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, James Houghton X-Rspamd-Queue-Id: 5CAF310000F X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: wtera7n5ys4uo1wwynzuiwoamsss661x X-HE-Tag: 1712014198-370401 X-HE-Meta: U2FsdGVkX19a8K43CI/ZTXtxm+IpLTHV+Bngyxu2rHk6MKhuZztmUf3QZFCyJfQia3PrUGOPLqtRa4JBD3HWd4Nz1sqv5zPW8RefqpKY0eCVLVTpW514V2grooVwuzWWu1p6hLoQPEMhsErNq4ygNfW9Qm76W+9xmI/duwxRS0Lyl+HKUQI+0n0KvV3XqmkyIKLroMk06gUFJ3nQAf3Rqk8uC0rp0FFFpwa708Wifs3zb40NaBpN2k1K2+AYlC9sxj4UpM0/W7b+eXgR64HJjcLABMAeMHep2NDZZy/utInGEfkYGB056v+gPE7PS1x95GrhzTPoPfAHDStnAKTVxQKPRgkqhpOrRfilTQRwxXggVVxjp2PSjUmaWRT4man3hu7B/RgfqDLlfugIjMZ1wrxkso79NHlyG9AkvlrWfKHdAbOTAcpYMd3hmAPBb+EvMgQR0n0vsKsIIobf1frj3iydyoSBR4GjPzFeowGvsAneFRkXPEEOqkLGu/Y3M2ByA1RtuiFQZX06i3wGH6jicK6FNipOFpPX4lD7AXn4dBNn43sGAKose8oY/dPt70gYrnKdaUtQvnlpybSKH5TWMe9h2Y9SKT5ihvU8JipGgYG6tPn71az063SR+1wwGMDM1YzYXs55wvOm0UlFqqjMWblm/YlPGdO8GBOTjTUjlysbNjCJ4JqTycjXZK0y/oxMd8HZF5TOesEbS9BEETDFexEobcxSMfj1pnbtqytJxHIBVN55k2iPh0E4pjhqcs6H6Mf43PZhjq6EeGk4+gVX9+soLou8ScQkhCeQ4gYFeBXhmxP6QJMve8Ehd9rlzWuy3N72HPQOQBNYMSpiUYdDOiWfM0fUN9FdZEMzU5tmY7Us0RnOL62t29jnxDlTGH5uwf/cRkPhT91fh5/xIprdiQlFEHqd/O0t6vFRcuryQ1uZhOeWrkblTdD9VrE1ByNCOg+bAOyFU0NjVaOkwz0 lOJoraI0 21/YtUVTDPJ5qQ+nc4M6xmUMqoWn5XgaZbUUGawOxFNNIRrBAOsDlGA+ytDpNYrR0ORxXPO9Xi7+RBYhj0IVj4UwJFYSuBMVpuK2CQcnNwhc6weMtN5o95NGGfa/jbmws2Hf+xOnTXhzmY3qw/SYzFCCPQGtIHpBLo4YPi7pNqiGzHpVwh5SHig81pe73PnnFAzfFIFMS6qsWsoYN3CRftWrDdahF/Hr9N0Hi3S1CN0LGigtEOMT/SSdltUFWNIpFC9wpveDmLkP4VBBJFG8vNXUmJusjc86OrBWXBWfqUFh6AWMpBKIS2xhCsSrv2SINrpY1MCPgnstO6nQVYtZv1VpFqA6gEDiFsFQ/tBGMAcbcf+KCmuY44yQUhwzTCqaflYoIMgmX87eAveWzUHLFaXfItMyrShDy7cZHx8do2NfteSM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The bitmap is provided for secondary MMUs to use if they support it. For test_young(), after it returns, the bitmap represents the pages that were young in the interval [start, end). For clear_young, it represents the pages that we wish the secondary MMU to clear the accessed/young bit for. If a bitmap is not provided, the mmu_notifier_{test,clear}_young() API should be unchanged except that if young PTEs are found and the architecture supports passing in a bitmap, instead of returning 1, MMU_NOTIFIER_YOUNG_FAST is returned. This allows MGLRU's look-around logic to work faster, resulting in a 4% improvement in real workloads[1]. Also introduce MMU_NOTIFIER_YOUNG_FAST to indicate to main mm that doing look-around is likely to be beneficial. If the secondary MMU doesn't support the bitmap, it must return an int that contains MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE. [1]: https://lore.kernel.org/all/20230609005935.42390-1-yuzhao@google.com/ Suggested-by: Yu Zhao Signed-off-by: James Houghton --- include/linux/mmu_notifier.h | 93 +++++++++++++++++++++++++++++++++--- include/trace/events/kvm.h | 13 +++-- mm/mmu_notifier.c | 20 +++++--- virt/kvm/kvm_main.c | 19 ++++++-- 4 files changed, 123 insertions(+), 22 deletions(-) diff --git a/include/linux/mmu_notifier.h b/include/linux/mmu_notifier.h index f349e08a9dfe..daaa9db625d3 100644 --- a/include/linux/mmu_notifier.h +++ b/include/linux/mmu_notifier.h @@ -61,6 +61,10 @@ enum mmu_notifier_event { #define MMU_NOTIFIER_RANGE_BLOCKABLE (1 << 0) +#define MMU_NOTIFIER_YOUNG (1 << 0) +#define MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE (1 << 1) +#define MMU_NOTIFIER_YOUNG_FAST (1 << 2) + struct mmu_notifier_ops { /* * Called either by mmu_notifier_unregister or when the mm is @@ -106,21 +110,36 @@ struct mmu_notifier_ops { * clear_young is a lightweight version of clear_flush_young. Like the * latter, it is supposed to test-and-clear the young/accessed bitflag * in the secondary pte, but it may omit flushing the secondary tlb. + * + * If @bitmap is given but is not supported, return + * MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE. + * + * If the walk is done "quickly" and there were young PTEs, + * MMU_NOTIFIER_YOUNG_FAST is returned. */ int (*clear_young)(struct mmu_notifier *subscription, struct mm_struct *mm, unsigned long start, - unsigned long end); + unsigned long end, + unsigned long *bitmap); /* * test_young is called to check the young/accessed bitflag in * the secondary pte. This is used to know if the page is * frequently used without actually clearing the flag or tearing * down the secondary mapping on the page. + * + * If @bitmap is given but is not supported, return + * MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE. + * + * If the walk is done "quickly" and there were young PTEs, + * MMU_NOTIFIER_YOUNG_FAST is returned. */ int (*test_young)(struct mmu_notifier *subscription, struct mm_struct *mm, - unsigned long address); + unsigned long start, + unsigned long end, + unsigned long *bitmap); /* * change_pte is called in cases that pte mapping to page is changed: @@ -388,10 +407,11 @@ extern int __mmu_notifier_clear_flush_young(struct mm_struct *mm, unsigned long start, unsigned long end); extern int __mmu_notifier_clear_young(struct mm_struct *mm, - unsigned long start, - unsigned long end); + unsigned long start, unsigned long end, + unsigned long *bitmap); extern int __mmu_notifier_test_young(struct mm_struct *mm, - unsigned long address); + unsigned long start, unsigned long end, + unsigned long *bitmap); extern void __mmu_notifier_change_pte(struct mm_struct *mm, unsigned long address, pte_t pte); extern int __mmu_notifier_invalidate_range_start(struct mmu_notifier_range *r); @@ -427,7 +447,25 @@ static inline int mmu_notifier_clear_young(struct mm_struct *mm, unsigned long end) { if (mm_has_notifiers(mm)) - return __mmu_notifier_clear_young(mm, start, end); + return __mmu_notifier_clear_young(mm, start, end, NULL); + return 0; +} + +/* + * When @bitmap is not provided, clear the young bits in the secondary + * MMUs for all of the pages in the interval [start, end). + * + * If any subscribed secondary MMU does not support @bitmap, this function + * will return an integer containing MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE. + * Some work may have been done in the secondary MMU. + */ +static inline int mmu_notifier_clear_young_bitmap(struct mm_struct *mm, + unsigned long start, + unsigned long end, + unsigned long *bitmap) +{ + if (mm_has_notifiers(mm)) + return __mmu_notifier_clear_young(mm, start, end, bitmap); return 0; } @@ -435,7 +473,25 @@ static inline int mmu_notifier_test_young(struct mm_struct *mm, unsigned long address) { if (mm_has_notifiers(mm)) - return __mmu_notifier_test_young(mm, address); + return __mmu_notifier_test_young(mm, address, address + 1, + NULL); + return 0; +} + +/* + * When @bitmap is not provided, test the young bits in the secondary + * MMUs for all of the pages in the interval [start, end). + * + * If any subscribed secondary MMU does not support @bitmap, this function + * will return an integer containing MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE. + */ +static inline int mmu_notifier_test_young_bitmap(struct mm_struct *mm, + unsigned long start, + unsigned long end, + unsigned long *bitmap) +{ + if (mm_has_notifiers(mm)) + return __mmu_notifier_test_young(mm, start, end, bitmap); return 0; } @@ -644,12 +700,35 @@ static inline int mmu_notifier_clear_flush_young(struct mm_struct *mm, return 0; } +static inline int mmu_notifier_clear_young(struct mm_struct *mm, + unsigned long start, + unsigned long end) +{ + return 0; +} + +static inline int mmu_notifier_clear_young_bitmap(struct mm_struct *mm, + unsigned long start, + unsigned long end, + unsigned long *bitmap) +{ + return 0; +} + static inline int mmu_notifier_test_young(struct mm_struct *mm, unsigned long address) { return 0; } +static inline int mmu_notifier_test_young_bitmap(struct mm_struct *mm, + unsigned long start, + unsigned long end, + unsigned long *bitmap) +{ + return 0; +} + static inline void mmu_notifier_change_pte(struct mm_struct *mm, unsigned long address, pte_t pte) { diff --git a/include/trace/events/kvm.h b/include/trace/events/kvm.h index 011fba6b5552..e4ace8cfdbba 100644 --- a/include/trace/events/kvm.h +++ b/include/trace/events/kvm.h @@ -490,18 +490,21 @@ TRACE_EVENT(kvm_age_hva, ); TRACE_EVENT(kvm_test_age_hva, - TP_PROTO(unsigned long hva), - TP_ARGS(hva), + TP_PROTO(unsigned long start, unsigned long end), + TP_ARGS(start, end), TP_STRUCT__entry( - __field( unsigned long, hva ) + __field( unsigned long, start ) + __field( unsigned long, end ) ), TP_fast_assign( - __entry->hva = hva; + __entry->start = start; + __entry->end = end; ), - TP_printk("mmu notifier test age hva: %#016lx", __entry->hva) + TP_printk("mmu notifier test age hva: %#016lx -- %#016lx", + __entry->start, __entry->end) ); #endif /* _TRACE_KVM_MAIN_H */ diff --git a/mm/mmu_notifier.c b/mm/mmu_notifier.c index ec3b068cbbe6..e70c6222944c 100644 --- a/mm/mmu_notifier.c +++ b/mm/mmu_notifier.c @@ -384,7 +384,8 @@ int __mmu_notifier_clear_flush_young(struct mm_struct *mm, int __mmu_notifier_clear_young(struct mm_struct *mm, unsigned long start, - unsigned long end) + unsigned long end, + unsigned long *bitmap) { struct mmu_notifier *subscription; int young = 0, id; @@ -395,7 +396,8 @@ int __mmu_notifier_clear_young(struct mm_struct *mm, srcu_read_lock_held(&srcu)) { if (subscription->ops->clear_young) young |= subscription->ops->clear_young(subscription, - mm, start, end); + mm, start, end, + bitmap); } srcu_read_unlock(&srcu, id); @@ -403,7 +405,8 @@ int __mmu_notifier_clear_young(struct mm_struct *mm, } int __mmu_notifier_test_young(struct mm_struct *mm, - unsigned long address) + unsigned long start, unsigned long end, + unsigned long *bitmap) { struct mmu_notifier *subscription; int young = 0, id; @@ -413,9 +416,14 @@ int __mmu_notifier_test_young(struct mm_struct *mm, &mm->notifier_subscriptions->list, hlist, srcu_read_lock_held(&srcu)) { if (subscription->ops->test_young) { - young = subscription->ops->test_young(subscription, mm, - address); - if (young) + young |= subscription->ops->test_young(subscription, mm, + start, end, + bitmap); + if (young && !bitmap) + /* + * We're not using a bitmap, so there is no + * need to check any more secondary MMUs. + */ break; } } diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index fb49c2a60200..ca4b1ef9dfc2 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -917,10 +917,15 @@ static int kvm_mmu_notifier_clear_flush_young(struct mmu_notifier *mn, static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn, struct mm_struct *mm, unsigned long start, - unsigned long end) + unsigned long end, + unsigned long *bitmap) { trace_kvm_age_hva(start, end); + /* We don't support bitmaps. Don't test or clear anything. */ + if (bitmap) + return MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE; + /* * Even though we do not flush TLB, this will still adversely * affect performance on pre-Haswell Intel EPT, where there is @@ -939,11 +944,17 @@ static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn, static int kvm_mmu_notifier_test_young(struct mmu_notifier *mn, struct mm_struct *mm, - unsigned long address) + unsigned long start, + unsigned long end, + unsigned long *bitmap) { - trace_kvm_test_age_hva(address); + trace_kvm_test_age_hva(start, end); + + /* We don't support bitmaps. Don't test or clear anything. */ + if (bitmap) + return MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE; - return kvm_handle_hva_range_no_flush(mn, address, address + 1, + return kvm_handle_hva_range_no_flush(mn, start, end, kvm_test_age_gfn); } From patchwork Mon Apr 1 23:29:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13613091 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89334CD1292 for ; Mon, 1 Apr 2024 23:30:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 573D06B0093; Mon, 1 Apr 2024 19:30:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 54B8A6B0095; Mon, 1 Apr 2024 19:30:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3C4596B0098; Mon, 1 Apr 2024 19:30:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 21ECD6B0093 for ; Mon, 1 Apr 2024 19:30:01 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id D264F1206E3 for ; Mon, 1 Apr 2024 23:30:00 +0000 (UTC) X-FDA: 81962558160.15.3FA6B21 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf23.hostedemail.com (Postfix) with ESMTP id 0CD0E14001A for ; Mon, 1 Apr 2024 23:29:58 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ZkqoVvtj; spf=pass (imf23.hostedemail.com: domain of 3dkMLZgoKCGEISGNTFGSNMFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--jthoughton.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3dkMLZgoKCGEISGNTFGSNMFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712014199; a=rsa-sha256; cv=none; b=lDtiimyQme94wZoY+BpD7+mA7rdM/v3sgQETteTBUf9B6j5+rPxfQ+T0qTvggHIUYT6NPU Px0LhRx4I3moAP+fGsvONI8Hj2WJJsFp9PvYWfW3pRSE3AFZaQz1ql/2ga2gJy0oMuHjv9 QDMjQ/+famlIePHMDolAViqMeTCS400= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ZkqoVvtj; spf=pass (imf23.hostedemail.com: domain of 3dkMLZgoKCGEISGNTFGSNMFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--jthoughton.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3dkMLZgoKCGEISGNTFGSNMFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712014199; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lJfaFUFI3gJojofhvI2fMvWTfkDCiiHnhDdEzunFXno=; b=VzBipoXzwC2u4B5WQMuCQGcZ8JiTmnMq4ts3wmdARx7+6BkffFCmKKixVrigjgJib8mcxf yBK3z9gMx++T3oRdct0A1O8ezuoe7WaDmyrd3oAQZkPlBHqnJnEQ17xoKwqr0oQMY+lTlv Md6AngHmu3syznIR94QVOBy9QJZH50o= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-614bc5c53a1so27295307b3.1 for ; Mon, 01 Apr 2024 16:29:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1712014198; x=1712618998; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=lJfaFUFI3gJojofhvI2fMvWTfkDCiiHnhDdEzunFXno=; b=ZkqoVvtj+XhZUkSO4Ezd0KNXtuwZQ5uC/Oy/bc/0M7XT/H2k9o59JWlfb2y48fFgC+ 04vscDBixNWZbfP+0vxP2h/p48r13NZhwTKVeyha6vcFLaXymYMzorScyeHzSssfU24U WBmr9HJVkVTYGdjE1da1bfpJwuZgpxUUUYRKMfmfUrOaewwtlV/6X+HYKiXChKQyUexP 6WYQzdhyd0pj0ReIH8FE8mIUvBeXSDdzJW9xSwQT2qCnkI3CcxfX32csTtWmJrtyilLo pLUYWol9L1is9xCqi2XoqjpE/8HJ+I9Vyd7Tn+wnFYD5lzlBzMm/CwSYQ/KQ8BJSgB40 r4Qg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712014198; x=1712618998; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=lJfaFUFI3gJojofhvI2fMvWTfkDCiiHnhDdEzunFXno=; b=XfnjYlO7u1kMd2wZhgyBstUJFD28U2ry9qtjZtnnOsnQeukRb8e/S3NsiKyWtqhVUH mnowoZbHJW86xHUNig5dEXp12hQVcCYsdDAFoQmp9jZlK++WoOECAseMRT13KTorjI/t kLrBGri9zU/3xUmp04YHDLNPvj6Jn1pLk7sJyk0poTQsolvdeZ06KFKymZZbHDjB/m2p P/MV6uCyDticpkDlJ4QKJjsMQyqSR0W2/Zj04uOMO/xj5S9P2kny/iQDrPtJEPTJpW0w PwS3P9aJd/RZqVUkLgvIMExcfPlIZ3U6WmNp2Vua2NOGc+75RUqTlPxegRFeskh1d+cJ pd0w== X-Forwarded-Encrypted: i=1; AJvYcCWWAlTyFkFt64Y42x0QNSjI6evy2cSKN0E1VSyZxB5QfgQJhxYfCz0ggpD92tFHcMG27JHfCM2VPqgODjLZheI/TVI= X-Gm-Message-State: AOJu0Yx0pGhCHNBx5pwhQV2enyWJMk4d5lWONZLNU5gjqo0SpdNAoCBF 8DAQLN7mdVwSZ/Pxt2NPb1Ff9XqJfYUMbfJ5SF1YOO/Mg07OX2i3xWt2ioloJOJONeqmuhNwMYk RFS2rE011bfmnHzoYYQ== X-Google-Smtp-Source: AGHT+IEzS+9pOebrRYAPHZcYp9+yLQ7vZUDEjcM1NuDRlvVAs7z0PGB+6rnD36HZsSFqff67vdpG7JIIt1VZPePZ X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a05:6902:1006:b0:dcc:94b7:a7a3 with SMTP id w6-20020a056902100600b00dcc94b7a7a3mr767796ybt.12.1712014198123; Mon, 01 Apr 2024 16:29:58 -0700 (PDT) Date: Mon, 1 Apr 2024 23:29:41 +0000 In-Reply-To: <20240401232946.1837665-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240401232946.1837665-1-jthoughton@google.com> X-Mailer: git-send-email 2.44.0.478.gd926399ef9-goog Message-ID: <20240401232946.1837665-3-jthoughton@google.com> Subject: [PATCH v3 2/7] KVM: Move MMU notifier function declarations From: James Houghton To: Andrew Morton , Paolo Bonzini Cc: Yu Zhao , David Matlack , Marc Zyngier , Oliver Upton , Sean Christopherson , Jonathan Corbet , James Morse , Suzuki K Poulose , Zenghui Yu , Catalin Marinas , Will Deacon , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers , Shaoqin Huang , Gavin Shan , Ricardo Koller , Raghavendra Rao Ananta , Ryan Roberts , David Rientjes , Axel Rasmussen , linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, James Houghton X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 0CD0E14001A X-Stat-Signature: 5nonjbautpq5joegjzc6gnxy11n16o3e X-Rspam-User: X-HE-Tag: 1712014198-348167 X-HE-Meta: U2FsdGVkX18v/RY3NTUW971brUiFf7SUz63olyF/i3PbJ7kwYFhNpxoj6QaiOzXRHllF5WJWVYEzACUAatsU6J1TBTFmONUP3PmJsBjNF6590xgB6ShgDu/1pFiuv1gIsbzhLPVoeQEvyFoO4hbXIYaTiY4aZd8faMHjIa2RySEXwulQ7A8hyHFYveULnQMgjFBPnnJkgV4uYL5Tp8kJaJ718nx0pKQUFs3TaHl1Ss9Z8LFR7Z5iE3zA/aztApy8UPLRBkDBpihaBnxbYyF5N9J31fBhbq+GlqZDBjowPl+17MnpCUByI1kdN+mLESYfYRaLFF1J+6/WM1rJFqO3jrBNCdYSDlJgZDuVv0pB+Jrdy45JY/OYVDziUxMYK4zjt878j6gikZOh5aSUvkXmhbOnwZ8rwW57/ZX8EAI3JAKrv2uon9RAxZ1ctiWnd1ldtffjMJv8NuHHGDjJY7lxVCt0JH1hpMaAizMnAXdzq2BWUE+yeDUFz29Nmu7Y82B3QB/8d9J8pW54ludYFUr/Mit3Zwmm8+hMGTASNRuW2aeGg4W+a8qQjJmlK9WAIqvsNtQbxNrhZSFfCzYlPmCuPjld8eYy9LKd9DKe11PSu50XRmH0oEdTbSFXHFTgUK/7ZZNVfsv3+18QE5e24y8G9pXLraol5alF/kKVqqu1lJzthqfZ7XnrW3GsDRusYwyibQv1N51VXtwcvKH62rtmPFhl1hywPyT3qO28zQ74KS98pPDlHzNWntaCBUpgSeqMPWhcA2q28bCFm9bF/n8o5UEBdMEXJ2I0rbvnNv+Z3apKKmr48T0seTIjAgA0RUTbtIlnBEJIGCDgHv+ai7Gcpfmnt2DnagV3E5p98+6/bG9d3Ddx7ZbHvf5tXjo9yJRJNQnZSQLoTiMu38JmGNzrLcnhCXxGcqeC6tAaFf1WMORTdY/g3UE1VyI2JlDmqLpg1IxJP8h3zWjhnqkKtJN QsBHpL5C NI2QcloFIGEh2Ho1tdoAeAB3VQlO4esjRQzSlFpYz0oJE1ezMBgjnNA+8GUAcgEXOPtjjOl/RJ4fCecsTPc8dwOfrBqmMd98htwPWw9ZXF56z4ZuSi3m7N4j83jqdOZ//xXN+EqqkDGMTxlfbRyPbxIDEaayI1kqxux8OsFkGNlY0teumY2VmshXqfBeuZITAo7FhqjvaEF/2Bndz872pDRjej47h/U+5l0Zr347GBHrT/aNxywLGZ9Re0i7xW4STNGcaW49Dm4ILetHgDGd8x9yf7gmEhVugZnIat87x6f6wkbB6fLCxRDK+cer3CXfldcKGxkeYAu3o7ryJnGCzEvNLRpZmG958YOuhEy0E6om25ib/6Ddjg9UCsSND6kLeyFVj30WPVz4/J1k/b5eZ/QTQVnkP9H0kqE1G+RagV0RIsNbm5rMYmW4LUQV8Mq2V0fgP3w0sJ3XQtp/aXIFe7pSSDoSgdfX4fF8YqFhUM4azOjyOeZ/zUSxkzHMBX/KOWKciA//n0JJqs0PVRXz7Glktt61G07h+EOYjSM3GKx6EmGbzuss9QZhXSiN8Vd5w36MQXeo4nVg0uvWczKPiMqgieb6FxTBIev2b7nWt7M9gX5yOnyKBaoFdLZt+3JMAs3TAbxMvfSWMBDA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: To allow new MMU-notifier-related functions to use gfn_to_hva_memslot(), move some declarations around. Also move mmu_notifier_to_kvm() for wider use later. Signed-off-by: James Houghton --- include/linux/kvm_host.h | 41 +++++++++++++++++++++------------------- virt/kvm/kvm_main.c | 5 ----- 2 files changed, 22 insertions(+), 24 deletions(-) diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 48f31dcd318a..1800d03a06a9 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -257,25 +257,6 @@ bool kvm_setup_async_pf(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, int kvm_async_pf_wakeup_all(struct kvm_vcpu *vcpu); #endif -#ifdef CONFIG_KVM_GENERIC_MMU_NOTIFIER -union kvm_mmu_notifier_arg { - pte_t pte; - unsigned long attributes; -}; - -struct kvm_gfn_range { - struct kvm_memory_slot *slot; - gfn_t start; - gfn_t end; - union kvm_mmu_notifier_arg arg; - bool may_block; -}; -bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range); -bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range); -bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range); -bool kvm_set_spte_gfn(struct kvm *kvm, struct kvm_gfn_range *range); -#endif - enum { OUTSIDE_GUEST_MODE, IN_GUEST_MODE, @@ -2012,6 +1993,11 @@ extern const struct kvm_stats_header kvm_vcpu_stats_header; extern const struct _kvm_stats_desc kvm_vcpu_stats_desc[]; #ifdef CONFIG_KVM_GENERIC_MMU_NOTIFIER +static inline struct kvm *mmu_notifier_to_kvm(struct mmu_notifier *mn) +{ + return container_of(mn, struct kvm, mmu_notifier); +} + static inline int mmu_invalidate_retry(struct kvm *kvm, unsigned long mmu_seq) { if (unlikely(kvm->mmu_invalidate_in_progress)) @@ -2089,6 +2075,23 @@ static inline bool mmu_invalidate_retry_gfn_unsafe(struct kvm *kvm, return READ_ONCE(kvm->mmu_invalidate_seq) != mmu_seq; } + +union kvm_mmu_notifier_arg { + pte_t pte; + unsigned long attributes; +}; + +struct kvm_gfn_range { + struct kvm_memory_slot *slot; + gfn_t start; + gfn_t end; + union kvm_mmu_notifier_arg arg; + bool may_block; +}; +bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range); +bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range); +bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range); +bool kvm_set_spte_gfn(struct kvm *kvm, struct kvm_gfn_range *range); #endif #ifdef CONFIG_HAVE_KVM_IRQ_ROUTING diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index ca4b1ef9dfc2..d0545d88c802 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -534,11 +534,6 @@ void kvm_destroy_vcpus(struct kvm *kvm) EXPORT_SYMBOL_GPL(kvm_destroy_vcpus); #ifdef CONFIG_KVM_GENERIC_MMU_NOTIFIER -static inline struct kvm *mmu_notifier_to_kvm(struct mmu_notifier *mn) -{ - return container_of(mn, struct kvm, mmu_notifier); -} - typedef bool (*gfn_handler_t)(struct kvm *kvm, struct kvm_gfn_range *range); typedef void (*on_lock_fn_t)(struct kvm *kvm); From patchwork Mon Apr 1 23:29:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13613092 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CBD45CD128A for ; Mon, 1 Apr 2024 23:30:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 43FE16B0095; Mon, 1 Apr 2024 19:30:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3C40A6B0099; Mon, 1 Apr 2024 19:30:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 107726B009A; Mon, 1 Apr 2024 19:30:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id E43E66B0095 for ; Mon, 1 Apr 2024 19:30:01 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id A77A240168 for ; Mon, 1 Apr 2024 23:30:01 +0000 (UTC) X-FDA: 81962558202.25.59652EB Received: from mail-ua1-f73.google.com (mail-ua1-f73.google.com [209.85.222.73]) by imf16.hostedemail.com (Postfix) with ESMTP id DDAC618000C for ; Mon, 1 Apr 2024 23:29:59 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=1O0Be61P; spf=pass (imf16.hostedemail.com: domain of 3dkMLZgoKCGEISGNTFGSNMFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--jthoughton.bounces.google.com designates 209.85.222.73 as permitted sender) smtp.mailfrom=3dkMLZgoKCGEISGNTFGSNMFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712014199; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PTb42yX5D5jCiyjo92iMcOe2nJRX5BH9KgDDQYg5TSM=; b=w7pCGSbP89bcUEKyUl2nDW3LZYDcZpnaG5eYMiI6Qnq9SlATPyEv9/MC1jjFo+f8d41dRX hsxjcpVclU5NH64PbioP+sT/gZZ6z5NIHe2SblJvksyD15ZvZebW/NBfi/a3yev4dcq4/W dVmCC0S3aVmLpvwL+KI3rJoMNRmJqcM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712014199; a=rsa-sha256; cv=none; b=0OL2NRg9g8KGD4dTGqU5yUPBoIBy8H4P/5XeSbWp8Jkyk0sOVFRI14ffRKHlVjX9HPMqmK TAoYsx/Xevhwk9gMg2mc/4PtL9qNz8Z/2liGXPXV4FylfoNrqf/ya8C0zGTgXRHK/3TedF qf/qJ2ih6Jm9o3w20GYPeYMLFyGpeRs= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=1O0Be61P; spf=pass (imf16.hostedemail.com: domain of 3dkMLZgoKCGEISGNTFGSNMFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--jthoughton.bounces.google.com designates 209.85.222.73 as permitted sender) smtp.mailfrom=3dkMLZgoKCGEISGNTFGSNMFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-ua1-f73.google.com with SMTP id a1e0cc1a2514c-7e374d439daso645812241.1 for ; Mon, 01 Apr 2024 16:29:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1712014199; x=1712618999; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=PTb42yX5D5jCiyjo92iMcOe2nJRX5BH9KgDDQYg5TSM=; b=1O0Be61PLthPfu5zpKQ8tpRbPKg7KH0zbn1WOWEyNwiIH40uyHzEG1ybXSlkZtmnkW 8F1ErUJdSPWVSap+5XvJ5BuqXMIk5lKRS2MFQblwrPMOHp8VsSje5u5FzfYKaV17dsWy bk72M0zc3KJzFHdDnxLwdxZVV4EzL4tvE3PvHMZHnvB/BVoD8fVUDdShX99dmfiErHkI Fh12xGzZZ8DpF5SGBXGdLzjw5dwf9KyybjQ13M8HUwsc0o4spXMimDDosLzvJHT35Mm/ RNteNRQduKdWuC7hm4k/3gJ7e4H9SyjdSEfy+5vQTllQxuhaUbTZSYvBdr/ZRykUigXc BKYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712014199; x=1712618999; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=PTb42yX5D5jCiyjo92iMcOe2nJRX5BH9KgDDQYg5TSM=; b=c9AdJrfABY2wGi6s68J6lf0txYrxT3ExY8EVZr3a/eX96URyX9yPBdYE69GMpvDV1O bEWBEmRyvnjBKGtWTDnxvRbQkTNmJ1h6GeeP3oWT82WgCAJi4xWyvURuqxJNolix/NyX uovPqmfgnyJDsVdbfNherWXv8anrlGOE7KMZ3V9tAIH7XJi4LhLxlfjYIovVZFW2lCWE hFsYrJfONBrzggEbET8c2bfvUaA5UcZ78ZUJXxHDphamECDMuc3bFbBF8ozj1lbC5xtF SyWAbOHfAFU5FdXUHmy5ZfvlV0n+YXqzpwht/kM/SYSiGxlGas0JoZ9xFqGmOazth/sK +pZg== X-Forwarded-Encrypted: i=1; AJvYcCVn/o/Q0c4ZADnNwPZlUKxjDRsEGJS7Ay7PepUFqrwJ/B5NeURcjovz+PdHF5E8HE40CiIZEi54R1WSe0n0YhmyR/8= X-Gm-Message-State: AOJu0Yx4G7IZfaCBkzxa5HjiSiGG3h54YIU0n7P9faQQ5T39lK/7sqRk Z5CdiNNahJdiRg7cEeX5bhvtaQBtJuo0Xfd3XgJj7Yi1P92Hiz/NAjTZnibdZPDU/kkpca2aocJ NdBnT3MyFJyKqgzPADA== X-Google-Smtp-Source: AGHT+IGmJlmVbaC/lfrLJ5ZQ93VOyi/dOsu4Z25niMdu8RAczUBaed8myhkzYvDdfTTf9baVVIW2FHfRFR9HE1rY X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a05:6102:2d0d:b0:470:5ca5:3e95 with SMTP id ih13-20020a0561022d0d00b004705ca53e95mr1009542vsb.2.1712014198891; Mon, 01 Apr 2024 16:29:58 -0700 (PDT) Date: Mon, 1 Apr 2024 23:29:42 +0000 In-Reply-To: <20240401232946.1837665-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240401232946.1837665-1-jthoughton@google.com> X-Mailer: git-send-email 2.44.0.478.gd926399ef9-goog Message-ID: <20240401232946.1837665-4-jthoughton@google.com> Subject: [PATCH v3 3/7] KVM: Add basic bitmap support into kvm_mmu_notifier_test/clear_young From: James Houghton To: Andrew Morton , Paolo Bonzini Cc: Yu Zhao , David Matlack , Marc Zyngier , Oliver Upton , Sean Christopherson , Jonathan Corbet , James Morse , Suzuki K Poulose , Zenghui Yu , Catalin Marinas , Will Deacon , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers , Shaoqin Huang , Gavin Shan , Ricardo Koller , Raghavendra Rao Ananta , Ryan Roberts , David Rientjes , Axel Rasmussen , linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, James Houghton X-Stat-Signature: p5imjcoin9mkumooikoxwxdg8y1paeg7 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: DDAC618000C X-Rspam-User: X-HE-Tag: 1712014199-315345 X-HE-Meta: U2FsdGVkX1+3uILlmS5BPE6Gwc4p4EyJSooxLx2HGPGQet6N6jbXHZN3VxoVuPT78RYYHLZ1uLtH6qo/dUYXXAVsVY5gYECFmWp2ejBSVKY64vJNB+j9N9CSnlqiksy/PLv83l97i/1PS38QcuRsazQjzqjEdAqbEsLya303/VtZUgAuajX6eqUfJ+Jp2RPK5HUVQxDCqBODXI44evfwj9W+taKe8zIucw4+8kNTd2GUSftzDS5q5LdSRGuUz/a04tEkblJBILJ9fTlNFgxEQABhx/NMY4R0lKGsI2OHgUt8wFmfJsSgVZomaMggUw5IdhEd0SZUWbgnqRTrQI9+FlTcfRmSKJEa/JXV6OcToMC2u072xm7tqmvHxRHSVf3OnBoi4CbhpM9VCiRLD2+Oiodae1PRST7tbkb+4Z8ZW9j7MahvEkf63vjbsHm3UgJifvIoZmdypLNwe/UeVTNa6uVodtCB9hL3ubO4czK/ywi3QmN+j7IbvK+4xGRvKax8hP0hHO5VNoyyXxK+iWIfaIVIOcLfiZR6FlhUEzgLr241fi7C+oxOTHl6twfemfyW8a2I9tGo45ipqVR6k03mTspdtxbQ13Opax3V4CAECo/Km/AjeaMmq+71FiKx6E/hqQ6rI3PrBvog7zs6vkkaso8X0TOfOEc1Ew2mDRwos1BxUEBcveZPXQ+Z+npXJuoSK/B8+WKc8o2f0NF4emyjB0oulUPb8lT5V2n7bvU1R1fFt+w/L5qxcAFlCYjB00ReXDw1B/kUcOQNLwm2XYee3+9uxFy9ATxpUjo/5yRJqHvZXz2AVJNbinIpcn3pa9a9fdgkA2axRGYchnC4HpVEpztXQBBA32OQr1aDbpt7nvNARa+JuuIRZ0V6db5NJ8X4wLKhV3gsVrRtMyi7F84y0hUkAkkfPiVxcqodMdBpHeVZbev0MePCH6a9aNgXcGOraKSr9L/KuHEVswUWFLv CML/2m4H JsIYDhIo/teqSPyZ6J2JR9ejObHm94yBASUKCJailsapAivpyFSLAMUdAofQGuam7IA038vp8JQs5w7cULuxN7UOVlcNVMrEUzSPMahL3FDpmkUVCodkDjmSeJAMYRzVXNzlr1VaeD/UBizDfPOmFJCxiEotEnd49AEnFN8WUQPmm4N5hQvmb+O5fbOFQvld+UAVZltNgPB6V6X7iTJQRk1op23m129gkIJ6NI9sqvmCJEKwpNZXq/YIa7RkUuBSSt/fDOWbKTwLWQO4XnxP8eNpr+jpXWR/IF2FDSvWVmBfJWxs1uE4+loy1z7duc4Xp/3eUtFkRbiOB4nHhRr3wB8pL2P6JVK/FmVe/PUNVohTd5VujoPzZx+3iApDFntPvVyVshvvRRG32PHMWRLN8LjeMpPMowCqZ9eTCJDaZzoaxCw38xbTyNkoGpFZmPhCtQv17xwNICNZZnExcFROidJez5gokFjwmP6agD9xmLJsRSEWikV6/9nzqSXk401ZfFhEP6cJEMf4XoGi7MbQHb+tPnueoMa++E8CB46/R3tK/jRb/jxkbD3CHYqQyngkytf9XI7MY970/0HRfiNrxEY2mB0y8hqJKmq2lYyXu3fmdZXqspmmZOw5QNW+n6URerzuiJ+1JUJOTewlXaSYSGb5LqsF3bN5+7Ym+7ahdc2UXsRgMQVoMdSrOFA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add kvm_arch_prepare_bitmap_age() for architectures to indiciate that they support bitmap-based aging in kvm_mmu_notifier_test_clear_young() and that they do not need KVM to grab the MMU lock for writing. This function allows architectures to do other locking or other preparatory work that it needs. If an architecture does not implement kvm_arch_prepare_bitmap_age() or is unable to do bitmap-based aging at runtime (and marks the bitmap as unreliable): 1. If a bitmap was provided, we inform the caller that the bitmap is unreliable (MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE). 2. If a bitmap was not provided, fall back to the old logic. Also add logic for architectures to easily use the provided bitmap if they are able. The expectation is that the architecture's implementation of kvm_gfn_test_age() will use kvm_gfn_record_young(), and kvm_gfn_age() will use kvm_gfn_should_age(). Suggested-by: Yu Zhao Signed-off-by: James Houghton --- include/linux/kvm_host.h | 60 ++++++++++++++++++++++++++ virt/kvm/kvm_main.c | 92 +++++++++++++++++++++++++++++----------- 2 files changed, 127 insertions(+), 25 deletions(-) diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 1800d03a06a9..5862fd7b5f9b 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -1992,6 +1992,26 @@ extern const struct _kvm_stats_desc kvm_vm_stats_desc[]; extern const struct kvm_stats_header kvm_vcpu_stats_header; extern const struct _kvm_stats_desc kvm_vcpu_stats_desc[]; +/* + * Architectures that support using bitmaps for kvm_age_gfn() and + * kvm_test_age_gfn should return true for kvm_arch_prepare_bitmap_age() + * and do any work they need to prepare. The subsequent walk will not + * automatically grab the KVM MMU lock, so some architectures may opt + * to grab it. + * + * If true is returned, a subsequent call to kvm_arch_finish_bitmap_age() is + * guaranteed. + */ +#ifndef kvm_arch_prepare_bitmap_age +static inline bool kvm_arch_prepare_bitmap_age(struct mmu_notifier *mn) +{ + return false; +} +#endif +#ifndef kvm_arch_finish_bitmap_age +static inline void kvm_arch_finish_bitmap_age(struct mmu_notifier *mn) {} +#endif + #ifdef CONFIG_KVM_GENERIC_MMU_NOTIFIER static inline struct kvm *mmu_notifier_to_kvm(struct mmu_notifier *mn) { @@ -2076,9 +2096,16 @@ static inline bool mmu_invalidate_retry_gfn_unsafe(struct kvm *kvm, return READ_ONCE(kvm->mmu_invalidate_seq) != mmu_seq; } +struct test_clear_young_metadata { + unsigned long *bitmap; + unsigned long bitmap_offset_end; + unsigned long end; + bool unreliable; +}; union kvm_mmu_notifier_arg { pte_t pte; unsigned long attributes; + struct test_clear_young_metadata *metadata; }; struct kvm_gfn_range { @@ -2087,11 +2114,44 @@ struct kvm_gfn_range { gfn_t end; union kvm_mmu_notifier_arg arg; bool may_block; + bool lockless; }; bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range); bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range); bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range); bool kvm_set_spte_gfn(struct kvm *kvm, struct kvm_gfn_range *range); + +static inline void kvm_age_set_unreliable(struct kvm_gfn_range *range) +{ + struct test_clear_young_metadata *args = range->arg.metadata; + + args->unreliable = true; +} +static inline unsigned long kvm_young_bitmap_offset(struct kvm_gfn_range *range, + gfn_t gfn) +{ + struct test_clear_young_metadata *args = range->arg.metadata; + + return hva_to_gfn_memslot(args->end - 1, range->slot) - gfn; +} +static inline void kvm_gfn_record_young(struct kvm_gfn_range *range, gfn_t gfn) +{ + struct test_clear_young_metadata *args = range->arg.metadata; + + WARN_ON_ONCE(gfn < range->start || gfn >= range->end); + if (args->bitmap) + __set_bit(kvm_young_bitmap_offset(range, gfn), args->bitmap); +} +static inline bool kvm_gfn_should_age(struct kvm_gfn_range *range, gfn_t gfn) +{ + struct test_clear_young_metadata *args = range->arg.metadata; + + WARN_ON_ONCE(gfn < range->start || gfn >= range->end); + if (args->bitmap) + return test_bit(kvm_young_bitmap_offset(range, gfn), + args->bitmap); + return true; +} #endif #ifdef CONFIG_HAVE_KVM_IRQ_ROUTING diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index d0545d88c802..7d80321e2ece 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -550,6 +550,7 @@ struct kvm_mmu_notifier_range { on_lock_fn_t on_lock; bool flush_on_ret; bool may_block; + bool lockless; }; /* @@ -598,6 +599,8 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm, struct kvm_memslots *slots; int i, idx; + BUILD_BUG_ON(sizeof(gfn_range.arg) != sizeof(gfn_range.arg.pte)); + if (WARN_ON_ONCE(range->end <= range->start)) return r; @@ -637,15 +640,18 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm, gfn_range.start = hva_to_gfn_memslot(hva_start, slot); gfn_range.end = hva_to_gfn_memslot(hva_end + PAGE_SIZE - 1, slot); gfn_range.slot = slot; + gfn_range.lockless = range->lockless; if (!r.found_memslot) { r.found_memslot = true; - KVM_MMU_LOCK(kvm); - if (!IS_KVM_NULL_FN(range->on_lock)) - range->on_lock(kvm); - - if (IS_KVM_NULL_FN(range->handler)) - break; + if (!range->lockless) { + KVM_MMU_LOCK(kvm); + if (!IS_KVM_NULL_FN(range->on_lock)) + range->on_lock(kvm); + + if (IS_KVM_NULL_FN(range->handler)) + break; + } } r.ret |= range->handler(kvm, &gfn_range); } @@ -654,7 +660,7 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm, if (range->flush_on_ret && r.ret) kvm_flush_remote_tlbs(kvm); - if (r.found_memslot) + if (r.found_memslot && !range->lockless) KVM_MMU_UNLOCK(kvm); srcu_read_unlock(&kvm->srcu, idx); @@ -682,19 +688,24 @@ static __always_inline int kvm_handle_hva_range(struct mmu_notifier *mn, return __kvm_handle_hva_range(kvm, &range).ret; } -static __always_inline int kvm_handle_hva_range_no_flush(struct mmu_notifier *mn, - unsigned long start, - unsigned long end, - gfn_handler_t handler) +static __always_inline int kvm_handle_hva_range_no_flush( + struct mmu_notifier *mn, + unsigned long start, + unsigned long end, + gfn_handler_t handler, + union kvm_mmu_notifier_arg arg, + bool lockless) { struct kvm *kvm = mmu_notifier_to_kvm(mn); const struct kvm_mmu_notifier_range range = { .start = start, .end = end, .handler = handler, + .arg = arg, .on_lock = (void *)kvm_null_fn, .flush_on_ret = false, .may_block = false, + .lockless = lockless, }; return __kvm_handle_hva_range(kvm, &range).ret; @@ -909,15 +920,36 @@ static int kvm_mmu_notifier_clear_flush_young(struct mmu_notifier *mn, kvm_age_gfn); } -static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn, - struct mm_struct *mm, - unsigned long start, - unsigned long end, - unsigned long *bitmap) +static int kvm_mmu_notifier_test_clear_young(struct mmu_notifier *mn, + struct mm_struct *mm, + unsigned long start, + unsigned long end, + unsigned long *bitmap, + bool clear) { - trace_kvm_age_hva(start, end); + if (kvm_arch_prepare_bitmap_age(mn)) { + struct test_clear_young_metadata args = { + .bitmap = bitmap, + .end = end, + .unreliable = false, + }; + union kvm_mmu_notifier_arg arg = { + .metadata = &args + }; + bool young; + + young = kvm_handle_hva_range_no_flush( + mn, start, end, + clear ? kvm_age_gfn : kvm_test_age_gfn, + arg, true); + + kvm_arch_finish_bitmap_age(mn); - /* We don't support bitmaps. Don't test or clear anything. */ + if (!args.unreliable) + return young ? MMU_NOTIFIER_YOUNG_FAST : 0; + } + + /* A bitmap was passed but the architecture doesn't support bitmaps */ if (bitmap) return MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE; @@ -934,7 +966,21 @@ static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn, * cadence. If we find this inaccurate, we might come up with a * more sophisticated heuristic later. */ - return kvm_handle_hva_range_no_flush(mn, start, end, kvm_age_gfn); + return kvm_handle_hva_range_no_flush( + mn, start, end, clear ? kvm_age_gfn : kvm_test_age_gfn, + KVM_MMU_NOTIFIER_NO_ARG, false); +} + +static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn, + struct mm_struct *mm, + unsigned long start, + unsigned long end, + unsigned long *bitmap) +{ + trace_kvm_age_hva(start, end); + + return kvm_mmu_notifier_test_clear_young(mn, mm, start, end, bitmap, + true); } static int kvm_mmu_notifier_test_young(struct mmu_notifier *mn, @@ -945,12 +991,8 @@ static int kvm_mmu_notifier_test_young(struct mmu_notifier *mn, { trace_kvm_test_age_hva(start, end); - /* We don't support bitmaps. Don't test or clear anything. */ - if (bitmap) - return MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE; - - return kvm_handle_hva_range_no_flush(mn, start, end, - kvm_test_age_gfn); + return kvm_mmu_notifier_test_clear_young(mn, mm, start, end, bitmap, + false); } static void kvm_mmu_notifier_release(struct mmu_notifier *mn, From patchwork Mon Apr 1 23:29:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13613093 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A561ACD1288 for ; Mon, 1 Apr 2024 23:30:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D3CD86B0099; Mon, 1 Apr 2024 19:30:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CC5A06B009A; Mon, 1 Apr 2024 19:30:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B1A0D6B009B; Mon, 1 Apr 2024 19:30:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 8DB536B0099 for ; Mon, 1 Apr 2024 19:30:02 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 5F27C40168 for ; Mon, 1 Apr 2024 23:30:02 +0000 (UTC) X-FDA: 81962558244.23.6C18572 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf12.hostedemail.com (Postfix) with ESMTP id 98DB140005 for ; Mon, 1 Apr 2024 23:30:00 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=LW88t150; spf=pass (imf12.hostedemail.com: domain of 3d0MLZgoKCGIJTHOUGHTONGOOGLE.COMLINUX-MMKVACK.ORG@flex--jthoughton.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3d0MLZgoKCGIJTHOUGHTONGOOGLE.COMLINUX-MMKVACK.ORG@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712014200; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nylYGqt19cj6oDr0RFoeiQ+kw/0yLdSX5tVYYbKq8QU=; b=tn8ti6xlbF0hzmb5CI3z1o4mgGEMr6wByThLdgLM70xbHrUz/ryWPB7jqhFhul/THcm5uG wNMJ4hWfof9hTZMKMDzNypU1NuUOS9O76JbuLlZ3i6YZhs3MwwCcFdmG0iYfrwd6pwnUwT 4kTU+nLu+pmohBNrr9bYi50EK+UNwtA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712014200; a=rsa-sha256; cv=none; b=aF5I5RgrNE1YFtC3OlVJ5IqA/zIT6VzNPap8DRA9o0atg5L2HrAyjEBxpC3fAFbRIa1fIO rcBC4gEpZucCEAJzrDDvEgYHmcNs9VW5MxNgZYblor1aIrcDa1EDm1Zam1rT4heUuIWAlK Yyenj7SNkmHiwtpzbqXoaSo1jz5MtFw= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=LW88t150; spf=pass (imf12.hostedemail.com: domain of 3d0MLZgoKCGIJTHOUGHTONGOOGLE.COMLINUX-MMKVACK.ORG@flex--jthoughton.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3d0MLZgoKCGIJTHOUGHTONGOOGLE.COMLINUX-MMKVACK.ORG@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dcc05887ee9so5314179276.1 for ; Mon, 01 Apr 2024 16:30:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1712014199; x=1712618999; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=nylYGqt19cj6oDr0RFoeiQ+kw/0yLdSX5tVYYbKq8QU=; b=LW88t1503tBI1C28ZHfg6LMe4iwcBX8DrnZcb69fncKvkiFpn0Ji1EF8DUOwB0C7qX riFNKd1jJL52kCeqylMWcQc4IzBGLbmoel+oVl84NmWZPorV8ILhH7HhlqBiPDkxTdWF tAKc+ul95kclD01AVBi4hdN4/WvrJtsC2Bp9IJBjnpfNKNFRUfoYNYH4LbhtuxHC4QA6 /9wEYkl+6DJKcHkyVE4npDSXYPH0zXnlVc1BFqf6zODNiKba8pMRSr/yZlWPMU8CK1pk Xxi/xrorS0fZswYMj61puqJXpYCc5imz1331wSSPyPJA5WidUGOpmzEgYUoLeSTAEUQh +NZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712014199; x=1712618999; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=nylYGqt19cj6oDr0RFoeiQ+kw/0yLdSX5tVYYbKq8QU=; b=d2o4b9ZAc9gt84fTFh0fBp9qs1E8e63fTAsDwglhUjCatk555CMg3zFIEhEARRIqdB cHb16iwtCJZSUSytDjUFrpTWd2IQBHLb2xMTb5gBa79aTX7s1bMa8bcyd2+5a8MqiysS 0VLXGoDu3ST2wkdIL1kQIHRCpoJuG4IoMcfXg93BzTSVS4dRiHyxmagU34AYJWO9T7qp wKQajm0Hht/WdO5bKbGA0/0X/O5Y/jZh1cGKmgO3WyUz8xPwbTU/kcAeGWnNjkUDITJd NyJvLcqON1S0AGmQz3Ptb6Nzn4IbvIllqybXKb/TmgcDsYksmEtqXxqkxBhtAyHnhVks QsIw== X-Forwarded-Encrypted: i=1; AJvYcCUR/7PuIbh5sBUd8ZVEbhLYo3tejnI2iLl9iDpXsK+AxPfTcHL8vii2/4eXPTP6bDH4xQr9+dUEjl/aDOH1Y4RN4nU= X-Gm-Message-State: AOJu0YwwuwC2bS20zYpFhnV5YSEf4A3ygv1f+bcHJ/MqCEml3HVCfRDS KebYnrrFEVFtiCrfjfjYKTfniLT3QjdxmrofdvfpaSmfTXbef76N/BbBil3KjfLiTboUWempaGJ E3bGqM9IH/hRl0o8GWQ== X-Google-Smtp-Source: AGHT+IH11gFjd5KLSe+p+dgHN5WTYlvUkDoQzbaal5VNFeM0iKdCiBmTQIhFAj+2vV8eggKrfvh7CUqHaFUquRNc X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a05:6902:2182:b0:dcd:2f3e:4d18 with SMTP id dl2-20020a056902218200b00dcd2f3e4d18mr765892ybb.12.1712014199648; Mon, 01 Apr 2024 16:29:59 -0700 (PDT) Date: Mon, 1 Apr 2024 23:29:43 +0000 In-Reply-To: <20240401232946.1837665-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240401232946.1837665-1-jthoughton@google.com> X-Mailer: git-send-email 2.44.0.478.gd926399ef9-goog Message-ID: <20240401232946.1837665-5-jthoughton@google.com> Subject: [PATCH v3 4/7] KVM: x86: Move tdp_mmu_enabled and shadow_accessed_mask From: James Houghton To: Andrew Morton , Paolo Bonzini Cc: Yu Zhao , David Matlack , Marc Zyngier , Oliver Upton , Sean Christopherson , Jonathan Corbet , James Morse , Suzuki K Poulose , Zenghui Yu , Catalin Marinas , Will Deacon , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers , Shaoqin Huang , Gavin Shan , Ricardo Koller , Raghavendra Rao Ananta , Ryan Roberts , David Rientjes , Axel Rasmussen , linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, James Houghton X-Rspamd-Queue-Id: 98DB140005 X-Rspam-User: X-Stat-Signature: fcrn1p5jw56b34umj5b53pprznrftpbi X-Rspamd-Server: rspam03 X-HE-Tag: 1712014200-736411 X-HE-Meta: U2FsdGVkX1+pm0Nxr2O05BCDuSJ8htxTWmdj2uE08jcLaj7qVC2xfFiU75k/y+TfwuM/dpszKfmzhOZc8WGiyuln7Dr+EBDev8q6rrOCnnCSBaUDX2gpFBQnUrWxe5yXohycuK9XZD8N9Kc8Q0We+FBcbcYHkIQ2E7e97GQlI26n+4I5Oc2Mh8r/lIAn3kdLK+wF8pnD6WHtz7DYlVwflCpd7yV0jWUrQUjeVsHQGJ2SB4IMjwEFXiTl5RatJQZYftw9FIO/B2WYzeNOx8heUYcy64h5pPY8iPS5i2Ag/RAohsERD1n1VnQf0/fF83D7ywC7b+/a1mApiDX9mRbLVcT17WlwaVTkWGhyYEdN9wTS19u5ASCXow2p9VjE6kR8CiNRu4SyNugXUhnTKa+lpJN9PfCFvg4ObsTLjolP0HYcLK5eOLP3rnrD4mrXwy8Wqw9nBjy5cEABzMyWrGIWpotKWsM+SiKW9tPSFz+KK+DnaolrarCWLrLIsmjC9JOCoQbBD6eRAbOQkvCOP9w6K66ebPV2CFmIlz3aNUQIjZobixfJqbSeyuW2v1dEuO8zhoYFm087pHaqIkKV16+/D20x6FbFgKqpPxziWten6ELYMRg/WhciJxQcfVehkceeA5YaV6yS1a+aMbk0F/RQXgtmMof3Va5C9Z+ReaX/W17sxQkr7U0iYlQWYe4WVpNozFV0B7hwjvL8VirDz7MLPTJAFr8m49bxTtnF0JpOx2Afbuv3sxA3P3haV8cQJQmR3oiqr0bW1uhkDw/OXQlRkx3e7OZi/p7+nloTShIYOz4gkoe53gz5POKzDUqOQ4oVEBslr0gUT1ihbHwpo/+7gBYidv8wyj2oLoHqspW8FaIarDqlIU0dIaWIT/31RiRXbyfG6OTBIXFIIqa7FH3SwjsJXkafB4OB1DNoviBfrLp/lSKTCP2d0AFAx3IQHa10pKgcj3fIg0pAC0ou45v 5yie6mvx lDwcjm2OfbjxsKolmMOguu1K54RAdY3gIxU4MLlLiJTWQ7/HwDMsxBWa/LKzjrJJOu4I7PtDsDHkF7Uwwaee0NLD6bkiVhGIe1VOtdhdKjPYB7p3gEAn80ePUh1RMLSvlMJgkR2jB4ysDZ92yBuKH/QiKDnS5ibznF1IEVfQ2HPHPCNJkKlObK8LJo+C736gGm3uuQXTQ1hpgerUi2uPxAqR/4O7hBc5z7t+cYHP0u9d0ZQPqMOyII+G/dDR536LjaRqtPcg+JrbptzMztl8dni7bRTicB1HZNwKydkoErNSIkgVY4kGkhUTlYJcEcmArdc0Vbj1vVK2gTTO2RMFKCDuNK8t900WcRdHvFQiJhfWit4F2Znhf125/roYgGGBgiQOzNdAnSzUU0i7UskU96YVj8r6AOh7wjDVyzk5yELxmbND+alCRP5bNRdoqB3x0rKGV0bev0I5OzXTjh1KhqGfESNYdjUwyb2zVgEWCfv3hwuxDb1k4A2w47X99/1304gBQmwNGiw0hZY2l+F4vENBa+jdvtsnb+odrDRr4gtPc6yTzILiR5Qs0dIiNtEZBnLqp7kgdo/V0cuZjcRrmBM3oXeNQsynVjdrbCpLhBW7blFJxTBwX3TWjKnZK/S0IDybRexFg1zGSSdYm3sc7NXIQHmsMaHrJ3bXFgtmyjIlXx18XxpbTJeOa+w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Yu Zhao tdp_mmu_enabled and shadow_accessed_mask are needed to implement kvm_arch_prepare_bitmap_age(). Signed-off-by: Yu Zhao Signed-off-by: James Houghton --- arch/x86/include/asm/kvm_host.h | 6 ++++++ arch/x86/kvm/mmu.h | 6 ------ arch/x86/kvm/mmu/spte.h | 1 - 3 files changed, 6 insertions(+), 7 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 16e07a2eee19..3b58e2306621 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1847,6 +1847,7 @@ struct kvm_arch_async_pf { extern u32 __read_mostly kvm_nr_uret_msrs; extern u64 __read_mostly host_efer; +extern u64 __read_mostly shadow_accessed_mask; extern bool __read_mostly allow_smaller_maxphyaddr; extern bool __read_mostly enable_apicv; extern struct kvm_x86_ops kvm_x86_ops; @@ -1952,6 +1953,11 @@ void kvm_fire_mask_notifiers(struct kvm *kvm, unsigned irqchip, unsigned pin, bool mask); extern bool tdp_enabled; +#ifdef CONFIG_X86_64 +extern bool tdp_mmu_enabled; +#else +#define tdp_mmu_enabled false +#endif u64 vcpu_tsc_khz(struct kvm_vcpu *vcpu); diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h index 60f21bb4c27b..8ae279035900 100644 --- a/arch/x86/kvm/mmu.h +++ b/arch/x86/kvm/mmu.h @@ -270,12 +270,6 @@ static inline bool kvm_shadow_root_allocated(struct kvm *kvm) return smp_load_acquire(&kvm->arch.shadow_root_allocated); } -#ifdef CONFIG_X86_64 -extern bool tdp_mmu_enabled; -#else -#define tdp_mmu_enabled false -#endif - static inline bool kvm_memslots_have_rmaps(struct kvm *kvm) { return !tdp_mmu_enabled || kvm_shadow_root_allocated(kvm); diff --git a/arch/x86/kvm/mmu/spte.h b/arch/x86/kvm/mmu/spte.h index a129951c9a88..f791fe045c7d 100644 --- a/arch/x86/kvm/mmu/spte.h +++ b/arch/x86/kvm/mmu/spte.h @@ -154,7 +154,6 @@ extern u64 __read_mostly shadow_mmu_writable_mask; extern u64 __read_mostly shadow_nx_mask; extern u64 __read_mostly shadow_x_mask; /* mutual exclusive with nx_mask */ extern u64 __read_mostly shadow_user_mask; -extern u64 __read_mostly shadow_accessed_mask; extern u64 __read_mostly shadow_dirty_mask; extern u64 __read_mostly shadow_mmio_value; extern u64 __read_mostly shadow_mmio_mask; From patchwork Mon Apr 1 23:29:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13613094 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76A99CD1288 for ; Mon, 1 Apr 2024 23:30:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 12FE86B009A; Mon, 1 Apr 2024 19:30:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 090546B009B; Mon, 1 Apr 2024 19:30:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E4B3E6B009C; Mon, 1 Apr 2024 19:30:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id C4E1D6B009A for ; Mon, 1 Apr 2024 19:30:03 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 909751A01C3 for ; Mon, 1 Apr 2024 23:30:03 +0000 (UTC) X-FDA: 81962558286.30.1D4426B Received: from mail-ua1-f74.google.com (mail-ua1-f74.google.com [209.85.222.74]) by imf04.hostedemail.com (Postfix) with ESMTP id CBBC440002 for ; Mon, 1 Apr 2024 23:30:01 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="Ms/iRFzr"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf04.hostedemail.com: domain of 3eEMLZgoKCGMKUIPVHIUPOHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--jthoughton.bounces.google.com designates 209.85.222.74 as permitted sender) smtp.mailfrom=3eEMLZgoKCGMKUIPVHIUPOHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--jthoughton.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712014201; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PXGb1c3s19z+8U24R5nbxlFRwqU1gq1c8Sy/WKQX1Zo=; b=PuuFLRErzlwF/IUg1ddMPVb0ueaCVfV6di174ggyvzrqQ8TmeaQQHtx3UYKwLfUPex2Ouu SzMVtrkxucqlFWrNHqEcT3CPgzgR0H3qF7UbJRkosrpNqdCprGvgs/cWX5pZODwCoyrCUW hVaXx7kpwAZ3O2Sa+WW+t2NA9/OyQr0= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="Ms/iRFzr"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf04.hostedemail.com: domain of 3eEMLZgoKCGMKUIPVHIUPOHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--jthoughton.bounces.google.com designates 209.85.222.74 as permitted sender) smtp.mailfrom=3eEMLZgoKCGMKUIPVHIUPOHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--jthoughton.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712014201; a=rsa-sha256; cv=none; b=JrRyRLKks7R66yDbkgxuVi641EvLC2KLK5QZ6UwEZ+exRgXDY4Z+wg3wWlyfAb+hYsJJqE jJh0A3ICfH632pT7K6ej1x2nHwLboSNWH7cixUx5RE2GSmgWpd5WDrT0xNf+NeIGwGOKEl 4Z561dvzMZ+MabTKtArRdOnCvSlQmr4= Received: by mail-ua1-f74.google.com with SMTP id a1e0cc1a2514c-7e03a706222so2050493241.3 for ; Mon, 01 Apr 2024 16:30:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1712014201; x=1712619001; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=PXGb1c3s19z+8U24R5nbxlFRwqU1gq1c8Sy/WKQX1Zo=; b=Ms/iRFzrsqWVvFeDD99Z243gyY0mUoqfTdYXoypDXFxHXSk2VgfGv0sB4JrAVs2R8x n730B4Qu7PQbcS60kSuaaQN2s7lwa/s60H5xGEaa09F/ZSWEEqI58x/CRtj0yWC/JQBk zRytFRVEPzKw1EQe3zn+grsl0aNPb4lh5MpPCxPq/MO+6zru5Zxclqe79FW+sxQOm2Fe Mz5dUoAnNSBWqG42Unc76iSpbSL0ZHL9EabjbC7I3X5sY/s4E6aBLn/AF59Usf8rIJF5 IBOcR1iWMx42+CQWfW0sv29MnpkUlsB5YtSyeWgFODCAfHYcYmkWRIdfobUoqVST5gLa GxyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712014201; x=1712619001; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=PXGb1c3s19z+8U24R5nbxlFRwqU1gq1c8Sy/WKQX1Zo=; b=l+zJAcxrwBY2TUIKH6RtlLOZf8iIwAyTQPfHTfvNsGcEXspe+xFG76G0NdDL0160W8 Epext39/89zK6OxaseI2vMR8fLgJLkCbTcEx/A20pw13OvdE3pR+YGp/qak45/WQ0tDa MWkUAPGZxeZuq9GCK1he5dXVQLQKF5a9VSc8OPNGEIv9vd1kk7wsIjpCJPqS8etVYi7r DmBdheqTPt1WULWE7iIAe8cSxjoOV2BrT7XuQCZS9TeJPECq7IRFokKnotwdwcDxnjFP CUpc1BD7QHiYANXANW2YAJEweIXbq3FQ2hBFFbnqa9Un4phYECvSv5d7DoNdQKqyt1G1 sH3Q== X-Forwarded-Encrypted: i=1; AJvYcCVkUQFb5uAq1B1u77UkfLPfjxdvi3j3pPZo8LQSJ5UfQB2NHLAXHi7Cp2f70LcxCBlfz4DNBygOE6czcxjBek7RSPc= X-Gm-Message-State: AOJu0YyXaNTnbcObRydy6rhIOksFSO6uZ7Np6RZ9a2kt43mhi+1ecLa4 wzPptE8COOH/UccrzFNCC7TmBECDg1fFJtPSLr80b53yFCBafZqr/pKs+THgCZHC09BYYzW2uwH EKlT6kRtORiNHej26Xw== X-Google-Smtp-Source: AGHT+IHiAeh9QdQ8JIxLLHU+30Xn5sQK+tQdEgkzewt+NNIBZBtyidDAUXciX6U3q2UCp3x2qiBYdkguFZKdqu9L X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:ab0:3792:0:b0:7e3:297b:9df1 with SMTP id d18-20020ab03792000000b007e3297b9df1mr67902uav.1.1712014200934; Mon, 01 Apr 2024 16:30:00 -0700 (PDT) Date: Mon, 1 Apr 2024 23:29:44 +0000 In-Reply-To: <20240401232946.1837665-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240401232946.1837665-1-jthoughton@google.com> X-Mailer: git-send-email 2.44.0.478.gd926399ef9-goog Message-ID: <20240401232946.1837665-6-jthoughton@google.com> Subject: [PATCH v3 5/7] KVM: x86: Participate in bitmap-based PTE aging From: James Houghton To: Andrew Morton , Paolo Bonzini Cc: Yu Zhao , David Matlack , Marc Zyngier , Oliver Upton , Sean Christopherson , Jonathan Corbet , James Morse , Suzuki K Poulose , Zenghui Yu , Catalin Marinas , Will Deacon , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers , Shaoqin Huang , Gavin Shan , Ricardo Koller , Raghavendra Rao Ananta , Ryan Roberts , David Rientjes , Axel Rasmussen , linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, James Houghton X-Rspamd-Queue-Id: CBBC440002 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: n9iu9punrfaz7816hruw1e4ak5b98uex X-HE-Tag: 1712014201-402313 X-HE-Meta: U2FsdGVkX19Q/Ova7zkBIOpod2gLDR7sqKo7oH1V27bBzkDsquJ80qroTuDeGqzdofe+ucX3v+nOWUaIbgxTCE8sv8jOgBxXMiuWiOi8vPsGa7kGP5w/2Anu7D/M6JSPuFoE+nWEyy4T+KPOBDqRrrCMCWYrN9pW/r9n4nq4X3i+xD9KI2QeV2v5/7jpy5zQfgdw8yEXl6Gdn2j6UuuUKk1MiwN9p0w3ODtqnHpbZiRXysmUd0xjUkRFee4mcEnCagMLgEFFUNTXQ1mt5+R0Hdd1SZ5Y1A5Badorlek2bG5vQyJpDKqes1WwDag+n4IYV92RxSskm1uuB+aAAqy6fB9kcwpH/lkF9TaFz/HUgMsVMOROg5y4rbIRGo5h8sL/+txuXIgB3z0SD4+DUH1iSjCCJzvPlfe9GsD3jzogWWiOuZd42+Kk1VkyzDxxWAp9nhWz5Tdry/gWYDu2GYuMbMgCrhm6fVxJP65kDiQTHDgkVZoa5d0Vh3GHp2w9R44td57Zt0qNR7HUoH7NKvS2b2NQZgBEP1TR7GLYseSRHlvNy+RchzZvTdcsMZZaWVfyiZDws0m5u9KvJrqRO/t46kRl54ki+LfF5TQwAmxSeaVZBrdmiEOZ1MLAPQrmbdPGnCXIXCSp1Op+Xtl7RrUsgfivd4Ob+4APXRCoh1rtgNzwkyZ+q4e/R8hX9Sb79AKC7PrrqgMzhdue6pruC3j1QvtD4uKDKVm4juYrv/n5OilbEjbI/fe2vfU0ZTy1xZotbezpB42AnnhCTBi+6p0QjP6EeSd0ofo15GGaYtmom1GzwVwc2gVIe+EGYWFAgtGIo9RFyMY9rCP99JuP45T0iFEXELb3TyVHUZ2+ZAy68Y05+4+yvIpcMr9c75SDUM16cAeEdatuucCMjz5zq22BZqPitMqA4nTkTLD+52TGdReP5bKP6hZdlZKrW7JId+0bka3fdErjwQ497hMij+Y Pf9ixXLb ASdK5IMawXOWyZXMLhlFFrRMSCfUoxWMheEQSMhQ+z9i3krjYJXfCoI9ReuXUlLivpBa4/u0umrYJwryuN1C4qe6o99dzOOmKhWXRZvvmJTlcj6Gj3uqF9+FHR0xnTW+BfwSBMqvEqIQHFrvbUmONSxoKowsm/ZVu71N3UV+H7DQO9AwrUySs36LnCCcJw/HHP3+mWGX2HzweB4lB3kv5Ibf8UObEy/q0VzJvEdPOySxkT5UUXuHDQCsmW1BoIEW719Ga/EUVv2q0lWKlFAVTOZEOBZ/a2lTw9lNAzdRnJOMlkYzhYNJaKYHRHNCdlMkqFEr5rwtqXiuN2OYb0oj/UQOUPiYWvOoQZu6LxJ90vsKvt0qsl0IezUyChcxIuPVN1OfVc4P50EmvdjKMyfMOKB67tPg3VMMrY8BdUxhUWv4l3e+XGfUKg9xWn/mbUAyf3zMe+QI/8TvvhfChMLyhHoKhvvwgyxA99S5aoFyi0OSARrzYFjE32/c/hH4hbcPiQkWjMKY1hWpwb0ruG8VBowfHN+BWuTTUvNngXr4DIHdNo9hX3uDW9StxwpCX+JUYpnM9clJyjUKB/Gj/HXf1Io2j0Cxy8LWrWqGPX/sTuiNQjIS9iiWE9etCE1ddO1Tvwt2soXLXZw1GDBs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Only handle the TDP MMU case for now. In other cases, if a bitmap was not provided, fallback to the slowpath that takes mmu_lock, or, if a bitmap was provided, inform the caller that the bitmap is unreliable. Suggested-by: Yu Zhao Signed-off-by: James Houghton --- arch/x86/include/asm/kvm_host.h | 14 ++++++++++++++ arch/x86/kvm/mmu/mmu.c | 16 ++++++++++++++-- arch/x86/kvm/mmu/tdp_mmu.c | 10 +++++++++- 3 files changed, 37 insertions(+), 3 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 3b58e2306621..c30918d0887e 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -2324,4 +2324,18 @@ int memslot_rmap_alloc(struct kvm_memory_slot *slot, unsigned long npages); */ #define KVM_EXIT_HYPERCALL_MBZ GENMASK_ULL(31, 1) +#define kvm_arch_prepare_bitmap_age kvm_arch_prepare_bitmap_age +static inline bool kvm_arch_prepare_bitmap_age(struct mmu_notifier *mn) +{ + /* + * Indicate that we support bitmap-based aging when using the TDP MMU + * and the accessed bit is available in the TDP page tables. + * + * We have no other preparatory work to do here, so we do not need to + * redefine kvm_arch_finish_bitmap_age(). + */ + return IS_ENABLED(CONFIG_X86_64) && tdp_mmu_enabled + && shadow_accessed_mask; +} + #endif /* _ASM_X86_KVM_HOST_H */ diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 992e651540e8..fae1a75750bb 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1674,8 +1674,14 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { bool young = false; - if (kvm_memslots_have_rmaps(kvm)) + if (kvm_memslots_have_rmaps(kvm)) { + if (range->lockless) { + kvm_age_set_unreliable(range); + return false; + } + young = kvm_handle_gfn_range(kvm, range, kvm_age_rmap); + } if (tdp_mmu_enabled) young |= kvm_tdp_mmu_age_gfn_range(kvm, range); @@ -1687,8 +1693,14 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { bool young = false; - if (kvm_memslots_have_rmaps(kvm)) + if (kvm_memslots_have_rmaps(kvm)) { + if (range->lockless) { + kvm_age_set_unreliable(range); + return false; + } + young = kvm_handle_gfn_range(kvm, range, kvm_test_age_rmap); + } if (tdp_mmu_enabled) young |= kvm_tdp_mmu_test_age_gfn(kvm, range); diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index d078157e62aa..edea01bc145f 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -1217,6 +1217,9 @@ static bool age_gfn_range(struct kvm *kvm, struct tdp_iter *iter, if (!is_accessed_spte(iter->old_spte)) return false; + if (!kvm_gfn_should_age(range, iter->gfn)) + return false; + if (spte_ad_enabled(iter->old_spte)) { iter->old_spte = tdp_mmu_clear_spte_bits(iter->sptep, iter->old_spte, @@ -1250,7 +1253,12 @@ bool kvm_tdp_mmu_age_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range) static bool test_age_gfn(struct kvm *kvm, struct tdp_iter *iter, struct kvm_gfn_range *range) { - return is_accessed_spte(iter->old_spte); + bool young = is_accessed_spte(iter->old_spte); + + if (young) + kvm_gfn_record_young(range, iter->gfn); + + return young; } bool kvm_tdp_mmu_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) From patchwork Mon Apr 1 23:29:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13613095 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1CEC4CD128A for ; Mon, 1 Apr 2024 23:30:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 099656B009B; Mon, 1 Apr 2024 19:30:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EE8A26B009C; Mon, 1 Apr 2024 19:30:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D16E46B009D; Mon, 1 Apr 2024 19:30:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id B1E176B009B for ; Mon, 1 Apr 2024 19:30:04 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 7424180579 for ; Mon, 1 Apr 2024 23:30:04 +0000 (UTC) X-FDA: 81962558328.08.18A44A8 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf03.hostedemail.com (Postfix) with ESMTP id AE4FC2000E for ; Mon, 1 Apr 2024 23:30:02 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="RjQ/DflK"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf03.hostedemail.com: domain of 3eUMLZgoKCGQLVJQWIJVQPIQQING.EQONKPWZ-OOMXCEM.QTI@flex--jthoughton.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3eUMLZgoKCGQLVJQWIJVQPIQQING.EQONKPWZ-OOMXCEM.QTI@flex--jthoughton.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712014202; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=CifQvULm5VcvU50LwCct+JTf2gCEq9urL1yip38iv3E=; b=w938jMF/OfAeIj7TZ5hM9s6OhsV0lEFMjCN2nH6ALjLU3PT+YW1z0dxXC5VSlnrN3Un2c0 pR5KytUelYk3Ouge1b2m83LrIv1QjsveOBm4Pbmj5KUVfId9go5fKq+X9fNDRvnSrFsrFu cFPtrm5NWk2pGnrjJa8k6QPy+ffAB/E= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="RjQ/DflK"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf03.hostedemail.com: domain of 3eUMLZgoKCGQLVJQWIJVQPIQQING.EQONKPWZ-OOMXCEM.QTI@flex--jthoughton.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3eUMLZgoKCGQLVJQWIJVQPIQQING.EQONKPWZ-OOMXCEM.QTI@flex--jthoughton.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712014202; a=rsa-sha256; cv=none; b=wH3FmuKGQv0KsUqVazJXX0UR61vRLOKhDTPwFl2ZtcS498gRBC6anJ9bR4oMrV7jZ/WtdN Yt/7zwC37huvl8c6KGYKgAyp7anjZnhy0J3PE1huB89icQVSaEF/9agWW6CN9P8eDVGo3/ Ng19D9RBtFk4uHaH9pSh+33XqYGSCxw= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-60cbba6f571so83605397b3.1 for ; Mon, 01 Apr 2024 16:30:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1712014202; x=1712619002; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=CifQvULm5VcvU50LwCct+JTf2gCEq9urL1yip38iv3E=; b=RjQ/DflKtlx8GhMXSGiREMtiBoBrT8lAJhOR/NhNIma7MdJGgr9W+/dh3cCKEVQCy5 y+S6aGxAFBmHRGxEhkrspS51h96uOq8jvV5OxKbJwMOdnOJqKdbW2r6eLcWSYvMisvLa Z8KHGh3lG3w0lIG9j/VJHDnZs/k1o2d9OOjL43pRUPR9RxDgEN40o7IbXVZaXlfmUOUj /C+L93e2Ax6jPpOpcXSKAuMvtoqARNgkZK9gF+2D+b6hsn3HavhOSZRTnyxuGjoYmRnD 1vv9O93+hLIjFXjPsSdj1FeBjj2zG/UDqGz4Eqe3cKPVDUVc1i45Bd6MtXROTUxdY0vB 2nvw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712014202; x=1712619002; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=CifQvULm5VcvU50LwCct+JTf2gCEq9urL1yip38iv3E=; b=dZnD2amQoE7iV1FRGKxzeVRtB5od7N/FoPPYzsqUBBM0kPumT76N1fe0c9DjZNqFY9 Otn2xezYm5N8+d79CjvQsnsoCX1Ik6PiIEFs6h6D+DaTgr6GLTwAZHLZQWEN3tTFZOVc dLN2bWHESq6+FlJHaPbH24WiyztdHuHtUmFElBVwCiZsGLk6tQdTuYvWfUgYiR/BF1oF k1tJcklDQz9JdRF85ltMifn4KILwa1t/B7CXFviMO0a/+4lG++F6K5oyX1rVXuciaxgY tPS0vi+CUKzBYWuyM+cumbOij9Ie9VS3McvTAXfshzD1t0722tbv0EKrHpHNqPk6DYll CR4Q== X-Forwarded-Encrypted: i=1; AJvYcCX5UCB7D8xnyhjboFuaN5ah7Se9SpiCYVLsvT2y5gNNnncvpcvqWZy4OvqelrsjsTk8u32hEA/7SE52YbfuWuEK3vQ= X-Gm-Message-State: AOJu0Yyrb3nTyZrMlgYZXMXnXeJTkrlmxzodTwMwGhsBP7nDsCz95zSX zxxxnZ+9TRpCpKYeJ0VYx8dLjPN9kFGvoHQCuPiYGH5kIPZVh9NUAF3eue8vkDDBsoYAylZhiOJ BLg2wDUQIscxrQCA6kg== X-Google-Smtp-Source: AGHT+IGQAk6u/Sk8y21e9rySBKwdkEDK33GoLWkr+tAsmcolw1O4jbXBJxOpzuSZWzDHw9bmaDUMZf63oZp0WmR7 X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a05:690c:650e:b0:615:165b:8dde with SMTP id hw14-20020a05690c650e00b00615165b8ddemr268650ywb.10.1712014201787; Mon, 01 Apr 2024 16:30:01 -0700 (PDT) Date: Mon, 1 Apr 2024 23:29:45 +0000 In-Reply-To: <20240401232946.1837665-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240401232946.1837665-1-jthoughton@google.com> X-Mailer: git-send-email 2.44.0.478.gd926399ef9-goog Message-ID: <20240401232946.1837665-7-jthoughton@google.com> Subject: [PATCH v3 6/7] KVM: arm64: Participate in bitmap-based PTE aging From: James Houghton To: Andrew Morton , Paolo Bonzini Cc: Yu Zhao , David Matlack , Marc Zyngier , Oliver Upton , Sean Christopherson , Jonathan Corbet , James Morse , Suzuki K Poulose , Zenghui Yu , Catalin Marinas , Will Deacon , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers , Shaoqin Huang , Gavin Shan , Ricardo Koller , Raghavendra Rao Ananta , Ryan Roberts , David Rientjes , Axel Rasmussen , linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, James Houghton X-Rspamd-Queue-Id: AE4FC2000E X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: qbmnff7h6umjz6w4mhpc5xrdnwyobwcb X-HE-Tag: 1712014202-405218 X-HE-Meta: U2FsdGVkX19LLbtJ3brrigVCCRdPiYTe51MVfpYzbrwn19gGVpeoBXQu2HmdzcBj2E/my66eP6bSPlkBIahZ1o50jK2B6ndqVxJPf+JikzKl8p5uSNwIHU2es8fYu1kiU+Kqar3G4WSXihGC6zt9U1rC3MR3KtFHQkYTQHxUlqlEn2uW2SCpa68ADb7CuwGplGWFT4wv+gXn06yXPZ7lyLg/2NB8B7+ltRIZDl+Z2iOjMje61XC7xkSCCfAT6YFXBvnzZyyictWLs8egiNeJQ6+H2it3YHWNUfgc5V2/xWONS/6Yj+W2rnuftjVI783NPwyfgUY+dvkXL65j52iGkm5t2Mi86gvXjnNejxCbbDmFZTvclTnhu9IWhC5PYQswD2NvSux/KK0WbhkdKAAc85R29Qrvz9cnSCSSTo/Y3KnqFq7vetyZRklMv2UypOtpgIioXgxph6tyy0j3vj7zGqY1SJxMVKsxs4SPltsuAE0KEn2GQ7JN9ulY7p1NZyTB5zk6CC2KKmDMp/VTjPDquj6yrLSfbocbdqo3bylfKcRqYNMQAkHtsyQks7g2jU2UyZGKwjkbtpj6CxCxNo5ylD2MhmiBMTpX/2PJxm1zIV0gBea467tiSmZ86uw4w55fPW+v5MYkmnez8QFH2TlCuxX2utS1+82QIdW0x0aI5PQLGoszrY4Um7+2N5QMf/54aeyZ+4q8oF3RJpQ3LNmuhqkKnNoc+J+bVni4UoYAPXAfQT/77D1R7RGnCzvZifStlVsaz42z7dxlkW9pZP/QDikc/Cz+wi83/qxV37m6o3VTDDL7RGohcQMzzoAQkUNlUvbCT/T3pJyIUfBja9uWHJ6kQZjem8qdkuYqNUZDmCDjJ5xo6pSQUvtWmRgMAy7iJoSKTuuvAYmcv2J5k/9rj8gcGWjK5PS8Z7HybMt6RYz7ZpDAMmKmbfBEEOaU444OUBH0SckaPXkKbd7U8A3 1CVriGGY xj35qzrP3bjgn3ETTL1hLzaBZ7RM6YLheCXqBviz0ZJv9Xz1ypNrje0DiTfmtVePNqijzcRsgi65CfUGC/5ZQagnhQoq8vXxQRd47VrABw6zHQEw6VuD4WNAvAqh78vCvGvw5OlYTUDQLuRoHQE/jeO8tNNbTK0IsvUsO2IHFUEi7TlqO2/LhPeFAVrRk+qu6suTBhA+joVHDAa3GC7X3zS/EMQ1nQDMAlbehq9fS/vVG78HRIfoBXA/4nC1NxLd4Rs1jrhzCw6Wx3Mx4ImXRtcnKG2RwIjh7MlGhtPLNnb5551DvNcWsW2IIw7LTE4wk7znQQNsu2bW313BMlLYlsdAZtMKvPv4mIiWPBn7fo4SNKmpxlK15qzdmCSd3KWtuQbzIFTnpuLS4dSCFHfXI1nDvKVDdOryp7Og9PzInHNU58O5ICVZNw2bDpckFMKB+1Zam4lw1c2lICoRVNCD3CYF2J6x24EDTbQbYAJQ/KPJ6/ZuYDxnaS0x+T2UEnSpDJWtw9oP5udam0fsLdTI3SO/MM1WTNQNoLQsl5/oTtFC4yUa7rHMfh/7cfOwkNyKFmu9+oOcauKJjujPr25+TS/ku4ZJf9DCE4ucz1ZmxJzYjmFnK8QTgrPuTPO0n+xuCi7n4QUGvuBEzeB4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Participate in bitmap-based aging while grabbing the KVM MMU lock for reading. Ideally we wouldn't need to grab this lock at all, but that would require a more intrustive and risky change. Also pass KVM_PGTABLE_WALK_SHARED, as this software walker is safe to run in parallel with other walkers. It is safe only to grab the KVM MMU lock for reading as the kvm_pgtable is destroyed while holding the lock for writing, and freeing of the page table pages is either done while holding the MMU lock for writing or after an RCU grace period. When mkold == false, record the young pages in the passed-in bitmap. When mkold == true, only age the pages that need aging according to the passed-in bitmap. Suggested-by: Yu Zhao Signed-off-by: James Houghton --- arch/arm64/include/asm/kvm_host.h | 5 +++++ arch/arm64/include/asm/kvm_pgtable.h | 4 +++- arch/arm64/kvm/hyp/pgtable.c | 21 ++++++++++++++------- arch/arm64/kvm/mmu.c | 23 +++++++++++++++++++++-- 4 files changed, 43 insertions(+), 10 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 9e8a496fb284..e503553cb356 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -1331,4 +1331,9 @@ bool kvm_arm_vcpu_stopped(struct kvm_vcpu *vcpu); (get_idreg_field((kvm), id, fld) >= expand_field_sign(id, fld, min) && \ get_idreg_field((kvm), id, fld) <= expand_field_sign(id, fld, max)) +#define kvm_arch_prepare_bitmap_age kvm_arch_prepare_bitmap_age +bool kvm_arch_prepare_bitmap_age(struct mmu_notifier *mn); +#define kvm_arch_finish_bitmap_age kvm_arch_finish_bitmap_age +void kvm_arch_finish_bitmap_age(struct mmu_notifier *mn); + #endif /* __ARM64_KVM_HOST_H__ */ diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index 19278dfe7978..1976b4e26188 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -644,6 +644,7 @@ kvm_pte_t kvm_pgtable_stage2_mkyoung(struct kvm_pgtable *pgt, u64 addr); * @addr: Intermediate physical address to identify the page-table entry. * @size: Size of the address range to visit. * @mkold: True if the access flag should be cleared. + * @range: The kvm_gfn_range that is being used for the memslot walker. * * The offset of @addr within a page is ignored. * @@ -657,7 +658,8 @@ kvm_pte_t kvm_pgtable_stage2_mkyoung(struct kvm_pgtable *pgt, u64 addr); * Return: True if any of the visited PTEs had the access flag set. */ bool kvm_pgtable_stage2_test_clear_young(struct kvm_pgtable *pgt, u64 addr, - u64 size, bool mkold); + u64 size, bool mkold, + struct kvm_gfn_range *range); /** * kvm_pgtable_stage2_relax_perms() - Relax the permissions enforced by a diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 3fae5830f8d2..e881d3595aca 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -1281,6 +1281,7 @@ kvm_pte_t kvm_pgtable_stage2_mkyoung(struct kvm_pgtable *pgt, u64 addr) } struct stage2_age_data { + struct kvm_gfn_range *range; bool mkold; bool young; }; @@ -1290,20 +1291,24 @@ static int stage2_age_walker(const struct kvm_pgtable_visit_ctx *ctx, { kvm_pte_t new = ctx->old & ~KVM_PTE_LEAF_ATTR_LO_S2_AF; struct stage2_age_data *data = ctx->arg; + gfn_t gfn = ctx->addr / PAGE_SIZE; if (!kvm_pte_valid(ctx->old) || new == ctx->old) return 0; data->young = true; + /* - * stage2_age_walker() is always called while holding the MMU lock for - * write, so this will always succeed. Nonetheless, this deliberately - * follows the race detection pattern of the other stage-2 walkers in - * case the locking mechanics of the MMU notifiers is ever changed. + * stage2_age_walker() may not be holding the MMU lock for write, so + * follow the race detection pattern of the other stage-2 walkers. */ - if (data->mkold && !stage2_try_set_pte(ctx, new)) - return -EAGAIN; + if (data->mkold) { + if (kvm_gfn_should_age(data->range, gfn) && + !stage2_try_set_pte(ctx, new)) + return -EAGAIN; + } else + kvm_gfn_record_young(data->range, gfn); /* * "But where's the TLBI?!", you scream. @@ -1315,10 +1320,12 @@ static int stage2_age_walker(const struct kvm_pgtable_visit_ctx *ctx, } bool kvm_pgtable_stage2_test_clear_young(struct kvm_pgtable *pgt, u64 addr, - u64 size, bool mkold) + u64 size, bool mkold, + struct kvm_gfn_range *range) { struct stage2_age_data data = { .mkold = mkold, + .range = range, }; struct kvm_pgtable_walker walker = { .cb = stage2_age_walker, diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 18680771cdb0..104cc23e9bb3 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -1802,6 +1802,25 @@ bool kvm_set_spte_gfn(struct kvm *kvm, struct kvm_gfn_range *range) return false; } +bool kvm_arch_prepare_bitmap_age(struct mmu_notifier *mn) +{ + struct kvm *kvm = mmu_notifier_to_kvm(mn); + + /* + * We need to hold the MMU lock for reading to prevent page tables + * from being freed underneath us. + */ + read_lock(&kvm->mmu_lock); + return true; +} + +void kvm_arch_finish_bitmap_age(struct mmu_notifier *mn) +{ + struct kvm *kvm = mmu_notifier_to_kvm(mn); + + read_unlock(&kvm->mmu_lock); +} + bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { u64 size = (range->end - range->start) << PAGE_SHIFT; @@ -1811,7 +1830,7 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) return kvm_pgtable_stage2_test_clear_young(kvm->arch.mmu.pgt, range->start << PAGE_SHIFT, - size, true); + size, true, range); } bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) @@ -1823,7 +1842,7 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) return kvm_pgtable_stage2_test_clear_young(kvm->arch.mmu.pgt, range->start << PAGE_SHIFT, - size, false); + size, false, range); } phys_addr_t kvm_mmu_get_httbr(void) From patchwork Mon Apr 1 23:29:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13613096 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BF3BBCD128A for ; Mon, 1 Apr 2024 23:30:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CCB436B009C; Mon, 1 Apr 2024 19:30:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C78176B009E; Mon, 1 Apr 2024 19:30:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ACB4B6B009F; Mon, 1 Apr 2024 19:30:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 87AAD6B009C for ; Mon, 1 Apr 2024 19:30:05 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 55E24A07B9 for ; Mon, 1 Apr 2024 23:30:05 +0000 (UTC) X-FDA: 81962558370.12.47452E0 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf07.hostedemail.com (Postfix) with ESMTP id 76D374000C for ; Mon, 1 Apr 2024 23:30:03 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ugUFBl1V; spf=pass (imf07.hostedemail.com: domain of 3ekMLZgoKCGUMWKRXJKWRQJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--jthoughton.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3ekMLZgoKCGUMWKRXJKWRQJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712014203; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WpAS/uks+xeMlZlwJrS/cYcvCVm3xLsD9iJKs+t5VOw=; b=74TfjjnKT2SkCkrnQxCOrDaVd8dy4WJvCUi3xJ25v4asQnU8OJtYJOcTKOZ8cYa1sxhRy/ 6bHFdXucDVTYgLmBzP5eCSEE1C1ohESP2aNLSZAwAdcaR301NIqVCE1BdPHYuz2qAFG6lk KFUMdkk5bnpyOU/UJWLz57p8xMVSj7U= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712014203; a=rsa-sha256; cv=none; b=ftl2mKBJK/A0fBiyJ5cR7W9G/Ze2h3QA8PDV1Q7sf2lbArn8tW+S0LvlhN958pC1Dq2Ciq 4QSCpLZlb8vVOcQsTfRolpVujQ9cVW89qAZI2+nc7k8gj6S288cU7g4wW7Kn9nNFGqxuKd IYwIUqbjXoB42cKoaZGUtwQDWYGtddI= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ugUFBl1V; spf=pass (imf07.hostedemail.com: domain of 3ekMLZgoKCGUMWKRXJKWRQJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--jthoughton.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3ekMLZgoKCGUMWKRXJKWRQJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dd8e82dd47eso6071486276.2 for ; Mon, 01 Apr 2024 16:30:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1712014202; x=1712619002; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=WpAS/uks+xeMlZlwJrS/cYcvCVm3xLsD9iJKs+t5VOw=; b=ugUFBl1VhkHTGZfOcgfbkpOrjqYnG6jYiwx3jTpTyqAp5WeBBC17OWpII2z6ZO3X+9 OZ66bqkKvT6UGb3XbN0ggC/bJ2b7t50deuuxxaCrblh2oEIAwftq17mYWFi6htbJG2hZ PyVa9xi/az7RXMP8N+8rdv/aGgSa+eBJJhDnnHzOF6z44C0P27KUwdomt1+CyOzpdfyn xBMF8gUV8wwECN9py6rI/au5e4YyipQ4E+K6Vi0L9Lq06vWCk5Givu44kQk8RpN1U+QL VJmXtW7AzlYf9ZrkjcPVoB7MD5d2KNhyTUeivGefoeIpmBT9JOWvEOMVQ65BBgLG3cvR qBog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712014202; x=1712619002; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=WpAS/uks+xeMlZlwJrS/cYcvCVm3xLsD9iJKs+t5VOw=; b=QwUH4EBz/agbBp1i6AZoqn3FVrNY+PDXi++vi/l3Tdd7yahOAPVUhONcB1aacmsHIu 1Pt/+mzOqij+alP4gEdJ5F1xJIn8QpYsnxC9DZ0mtkbPuvYTtkuxDRuVeXHxbVUzDQue 8hYy6L/9B5nkoTHW6ClhRMsGceViqV9iP4kJ5pYVAG537c36ZGQerloAxVgOWlPOJZuO qMYFbGW1dcOHPtOngfn4JVdTIiiqjZHLVvyqP3Kv9Bw2+WV5P71DgpUPXBNbj0SjYuIT y0GmeN3oufbUK3taK0QVGCnKm5RUUiV8+i6u0O9oaZlwoebKmx+dP7PVWUcZ79kuwTI3 6xYw== X-Forwarded-Encrypted: i=1; AJvYcCU9rFAR7t9BGLlZUTQ/xL+ubR3ju99QmlUyz58zEvJwu0sSlwRl/2ft5Wf1JASvr0SJJGZeR9qtjMHyLNHC8haOs4g= X-Gm-Message-State: AOJu0Ywyyoedr0SFfX0xFDpjhe52EWtIm0qjVtLJ84eUkmxdFDxZmph4 clvFj2AqkcW+tNN12rixKzGns3LAA2ILaAxGCXOV+G1QjhvMEj5H+uxxcjFnXinabw1wXtSjAzr l6ICAroQXtDwcGSD/og== X-Google-Smtp-Source: AGHT+IFDD8pqYO4xCu07bbvePXFmo0Pl8SwWr49Tzn/Rs0hc66IZhVLSZQfRP6lopGz4i0uv+3xgxtfkPILQecVq X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a05:6902:2202:b0:dcd:59a5:7545 with SMTP id dm2-20020a056902220200b00dcd59a57545mr764461ybb.10.1712014202562; Mon, 01 Apr 2024 16:30:02 -0700 (PDT) Date: Mon, 1 Apr 2024 23:29:46 +0000 In-Reply-To: <20240401232946.1837665-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240401232946.1837665-1-jthoughton@google.com> X-Mailer: git-send-email 2.44.0.478.gd926399ef9-goog Message-ID: <20240401232946.1837665-8-jthoughton@google.com> Subject: [PATCH v3 7/7] mm: multi-gen LRU: use mmu_notifier_test_clear_young() From: James Houghton To: Andrew Morton , Paolo Bonzini Cc: Yu Zhao , David Matlack , Marc Zyngier , Oliver Upton , Sean Christopherson , Jonathan Corbet , James Morse , Suzuki K Poulose , Zenghui Yu , Catalin Marinas , Will Deacon , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers , Shaoqin Huang , Gavin Shan , Ricardo Koller , Raghavendra Rao Ananta , Ryan Roberts , David Rientjes , Axel Rasmussen , linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, James Houghton X-Rspamd-Queue-Id: 76D374000C X-Rspam-User: X-Stat-Signature: 5joc4swr439acwefiyxsffpyc94isj1u X-Rspamd-Server: rspam03 X-HE-Tag: 1712014203-204129 X-HE-Meta: U2FsdGVkX1/urf+zyP46cZGrlvhKc1Qcl8kU7M+uUTAuw23fLqcJd7wWw4jdxdnUuW8+PR2CyAD2gqPlfjM6iO63DqvBGOOOsSAMnX3VaApm3fYZj88faJvaeVjn32bMkqKCYsICNvK3De9KHZjynRuYJD1rtVc38o91wbsHFHP42iQUHkzUrra1QdDvJXnSi/XSmMHsbH9KpWz86xaIVZX0MAvY9GQV6RlY6L5LziZ+2sDRQ/KxkKs4mNGRgptycNZ/vx6KgGelJktucyA3oXu6Hw5eabpG49RqCYLRI+qxtECo9cKLi2cqWubWhTOB9ujiqToDhDsCMpuO46ODw+9XEc4NK9qv8UB7KJt5fLGENHYzaVRwnnxsctb2vnlIzmygj4v4h9y0hdsdU0o2CEAtFf6rFrG26zJVnlPPed2CSQmfzOD2Lu9AgX3ET8rGagw67i+AmMWyMYrNVdqKvcAzdyaIeGd9+h7hbXr4oa3VvfiueQWVuNQFiOyxXdHPenJI+vm+E88or9Tl1+J6CIBtItCOEgQwLINQhrz4JUx+BnRON66IQss10f61j6IniuCrg68W+snI2uOViwxVFBcuGHiO81nqRcMiqe8CDtsZfQ2iAaRBZyPZj8uUt1/3M/8S77eZP7QE+S9n/Cvka2588YiLp8cDXzoaLE/NRVE94UAe7MtOXQYtQSO7Yma+5t4FP6jUpSe2AdjjQMlUT1plnyCOabxM3QFXlWr9zT0VZBKgR8LzgAa+dB+soAro81LNOdLd3XrUO9YSQ9+DQ5X2j4I4eerq97a/QW7SbNO5aOJFNJFOTIYYUDV2CebN0pGzTcaBISDD6x4B6xRlmQYWWEw0xnUG0WJ1IYWIXm28OxoWIdwZpv5heqYpYAq8+n7IfUhMhhBSuoC0IDAppnbBgJMMoiLtxtBU2kD0quSYmhJYpf5jzdPSZ0xQIQ2KRsQ0vbWEwQNsrYs01km pSZWZ/vc cUZlX7kXYbOTfgyoimCcyiLQrG3/7PNPMjKRxxqwbIX81LF6yXYt1h3dqIMkMiD3bE6cj7ZKu6lVMgmtQZxMjWu6LKmNtu+i0341+I8MA9oiG9FegV4KKBmbnbYHOPbtBye64vD3zvieZ5ihNw8T2rx9nxpf819hPRL426ma0tf/7HK571w+bJ/NQlfwWrbDLkxeX2x5ElbWq6i28qGIO9HKV19BsalvubkO7E0oM5aP7iIG3BXegz1w82S3JLwkJicXzEoLwltO+AezaCZgLz6Or+uafh3XvwIuvbmnKbo03hTK+qOc4dSYt74/XVPMogUwNxT9xeWja0Mk13gwOANfDTOMWpLCj3L3U7efZ+C3kfbb2op/qBbTpH02iQElipxcUryzST+/fAxcZUxo05hF26b8jOpdgoRoNKvTMpCfcExowoBeXxKgkwOz72rtUL0+dnGF20WdI5w+U5pMSsd/lbo8Abuds+6Ja4CwOghQaSZIDrG1r3jWQH7Ih/J1tiWj3MZhbGoj445tPLl7VKjd7gUXFkQe8ZKWuhl/sH6gfv78dOBVMPvKzrxe3SaeBuLxHe7kica9HEGVJHP6OgznDTA0XTl2pFRBLBssYHpqTi27ojS2BR4vZQKU548EjMLpGKoiyjdguxxVTjr1p2Yv/t+FWFwUWFshLUrTCiUK5800E5URSR6lQlt8rFbCiWi081SxiaRITDElS36qOcybn65QSchfbdJi3f4wBBw0lAgkcxp9037saouxIGrfvTA/O85I98LviPfo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Yu Zhao Use mmu_notifier_{test,clear}_young_bitmap() to handle KVM PTEs in batches when the fast path is supported. This reduces the contention on kvm->mmu_lock when the host is under heavy memory pressure. An existing selftest can quickly demonstrate the effectiveness of this patch. On a generic workstation equipped with 128 CPUs and 256GB DRAM: $ sudo max_guest_memory_test -c 64 -m 250 -s 250 MGLRU run2 ------------------ Before [1] ~64s After ~51s kswapd (MGLRU before) 100.00% balance_pgdat 100.00% shrink_node 100.00% shrink_one 99.99% try_to_shrink_lruvec 99.71% evict_folios 97.29% shrink_folio_list ==>> 13.05% folio_referenced 12.83% rmap_walk_file 12.31% folio_referenced_one 7.90% __mmu_notifier_clear_young 7.72% kvm_mmu_notifier_clear_young 7.34% _raw_write_lock kswapd (MGLRU after) 100.00% balance_pgdat 100.00% shrink_node 100.00% shrink_one 99.99% try_to_shrink_lruvec 99.59% evict_folios 80.37% shrink_folio_list ==>> 3.74% folio_referenced 3.59% rmap_walk_file 3.19% folio_referenced_one 2.53% lru_gen_look_around 1.06% __mmu_notifier_test_clear_young [1] "mm: rmap: Don't flush TLB after checking PTE young for page reference" was included so that the comparison is apples to apples. https://lore.kernel.org/r/20220706112041.3831-1-21cnbao@gmail.com/ Signed-off-by: Yu Zhao Signed-off-by: James Houghton --- Documentation/admin-guide/mm/multigen_lru.rst | 6 +- include/linux/mmzone.h | 6 +- mm/rmap.c | 9 +- mm/vmscan.c | 183 ++++++++++++++---- 4 files changed, 159 insertions(+), 45 deletions(-) diff --git a/Documentation/admin-guide/mm/multigen_lru.rst b/Documentation/admin-guide/mm/multigen_lru.rst index 33e068830497..0ae2a6d4d94c 100644 --- a/Documentation/admin-guide/mm/multigen_lru.rst +++ b/Documentation/admin-guide/mm/multigen_lru.rst @@ -48,6 +48,10 @@ Values Components verified on x86 varieties other than Intel and AMD. If it is disabled, the multi-gen LRU will suffer a negligible performance degradation. +0x0008 Clearing the accessed bit in KVM page table entries in large + batches, when KVM MMU sets it (e.g., on x86_64). This can + improve the performance of guests when the host is under memory + pressure. [yYnN] Apply to all the components above. ====== =============================================================== @@ -56,7 +60,7 @@ E.g., echo y >/sys/kernel/mm/lru_gen/enabled cat /sys/kernel/mm/lru_gen/enabled - 0x0007 + 0x000f echo 5 >/sys/kernel/mm/lru_gen/enabled cat /sys/kernel/mm/lru_gen/enabled 0x0005 diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index c11b7cde81ef..a98de5106990 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -397,6 +397,7 @@ enum { LRU_GEN_CORE, LRU_GEN_MM_WALK, LRU_GEN_NONLEAF_YOUNG, + LRU_GEN_KVM_MMU_WALK, NR_LRU_GEN_CAPS }; @@ -554,7 +555,7 @@ struct lru_gen_memcg { void lru_gen_init_pgdat(struct pglist_data *pgdat); void lru_gen_init_lruvec(struct lruvec *lruvec); -void lru_gen_look_around(struct page_vma_mapped_walk *pvmw); +bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw); void lru_gen_init_memcg(struct mem_cgroup *memcg); void lru_gen_exit_memcg(struct mem_cgroup *memcg); @@ -573,8 +574,9 @@ static inline void lru_gen_init_lruvec(struct lruvec *lruvec) { } -static inline void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) +static inline bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw) { + return false; } static inline void lru_gen_init_memcg(struct mem_cgroup *memcg) diff --git a/mm/rmap.c b/mm/rmap.c index 56b313aa2ebf..41e9fc25684e 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -871,13 +871,10 @@ static bool folio_referenced_one(struct folio *folio, continue; } - if (pvmw.pte) { - if (lru_gen_enabled() && - pte_young(ptep_get(pvmw.pte))) { - lru_gen_look_around(&pvmw); + if (lru_gen_enabled() && pvmw.pte) { + if (lru_gen_look_around(&pvmw)) referenced++; - } - + } else if (pvmw.pte) { if (ptep_clear_flush_young_notify(vma, address, pvmw.pte)) referenced++; diff --git a/mm/vmscan.c b/mm/vmscan.c index 293120fe54f3..fd65f3466dfc 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -56,6 +56,7 @@ #include #include #include +#include #include #include @@ -2596,6 +2597,11 @@ static bool should_clear_pmd_young(void) return arch_has_hw_nonleaf_pmd_young() && get_cap(LRU_GEN_NONLEAF_YOUNG); } +static bool should_walk_kvm_mmu(void) +{ + return get_cap(LRU_GEN_KVM_MMU_WALK); +} + /****************************************************************************** * shorthand helpers ******************************************************************************/ @@ -3293,7 +3299,8 @@ static bool get_next_vma(unsigned long mask, unsigned long size, struct mm_walk return false; } -static unsigned long get_pte_pfn(pte_t pte, struct vm_area_struct *vma, unsigned long addr) +static unsigned long get_pte_pfn(pte_t pte, struct vm_area_struct *vma, unsigned long addr, + struct pglist_data *pgdat) { unsigned long pfn = pte_pfn(pte); @@ -3308,10 +3315,15 @@ static unsigned long get_pte_pfn(pte_t pte, struct vm_area_struct *vma, unsigned if (WARN_ON_ONCE(!pfn_valid(pfn))) return -1; + /* try to avoid unnecessary memory loads */ + if (pfn < pgdat->node_start_pfn || pfn >= pgdat_end_pfn(pgdat)) + return -1; + return pfn; } -static unsigned long get_pmd_pfn(pmd_t pmd, struct vm_area_struct *vma, unsigned long addr) +static unsigned long get_pmd_pfn(pmd_t pmd, struct vm_area_struct *vma, unsigned long addr, + struct pglist_data *pgdat) { unsigned long pfn = pmd_pfn(pmd); @@ -3326,6 +3338,10 @@ static unsigned long get_pmd_pfn(pmd_t pmd, struct vm_area_struct *vma, unsigned if (WARN_ON_ONCE(!pfn_valid(pfn))) return -1; + /* try to avoid unnecessary memory loads */ + if (pfn < pgdat->node_start_pfn || pfn >= pgdat_end_pfn(pgdat)) + return -1; + return pfn; } @@ -3334,10 +3350,6 @@ static struct folio *get_pfn_folio(unsigned long pfn, struct mem_cgroup *memcg, { struct folio *folio; - /* try to avoid unnecessary memory loads */ - if (pfn < pgdat->node_start_pfn || pfn >= pgdat_end_pfn(pgdat)) - return NULL; - folio = pfn_folio(pfn); if (folio_nid(folio) != pgdat->node_id) return NULL; @@ -3352,6 +3364,52 @@ static struct folio *get_pfn_folio(unsigned long pfn, struct mem_cgroup *memcg, return folio; } +static bool test_spte_young(struct mm_struct *mm, unsigned long addr, unsigned long end, + unsigned long *bitmap, unsigned long *last) +{ + if (*last > addr) + goto done; + + *last = end - addr > MIN_LRU_BATCH * PAGE_SIZE ? + addr + MIN_LRU_BATCH * PAGE_SIZE - 1 : end - 1; + bitmap_zero(bitmap, MIN_LRU_BATCH); + + mmu_notifier_test_young_bitmap(mm, addr, *last + 1, bitmap); +done: + return test_bit((*last - addr) / PAGE_SIZE, bitmap); +} + +static void clear_spte_young(struct mm_struct *mm, unsigned long addr, + unsigned long *bitmap, unsigned long *last) +{ + int i; + unsigned long start, end = *last + 1; + + if (addr + PAGE_SIZE != end) + return; + + i = find_last_bit(bitmap, MIN_LRU_BATCH); + if (i == MIN_LRU_BATCH) + return; + + start = end - (i + 1) * PAGE_SIZE; + + i = find_first_bit(bitmap, MIN_LRU_BATCH); + + end -= i * PAGE_SIZE; + + mmu_notifier_clear_young_bitmap(mm, start, end, bitmap); +} + +static void skip_spte_young(struct mm_struct *mm, unsigned long addr, + unsigned long *bitmap, unsigned long *last) +{ + if (*last > addr) + __clear_bit((*last - addr) / PAGE_SIZE, bitmap); + + clear_spte_young(mm, addr, bitmap, last); +} + static bool suitable_to_scan(int total, int young) { int n = clamp_t(int, cache_line_size() / sizeof(pte_t), 2, 8); @@ -3367,6 +3425,8 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, pte_t *pte; spinlock_t *ptl; unsigned long addr; + DECLARE_BITMAP(bitmap, MIN_LRU_BATCH); + unsigned long last = 0; int total = 0; int young = 0; struct lru_gen_mm_walk *walk = args->private; @@ -3386,6 +3446,7 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, arch_enter_lazy_mmu_mode(); restart: for (i = pte_index(start), addr = start; addr != end; i++, addr += PAGE_SIZE) { + bool ret; unsigned long pfn; struct folio *folio; pte_t ptent = ptep_get(pte + i); @@ -3393,21 +3454,28 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, total++; walk->mm_stats[MM_LEAF_TOTAL]++; - pfn = get_pte_pfn(ptent, args->vma, addr); - if (pfn == -1) + pfn = get_pte_pfn(ptent, args->vma, addr, pgdat); + if (pfn == -1) { + skip_spte_young(args->vma->vm_mm, addr, bitmap, &last); continue; + } - if (!pte_young(ptent)) { + ret = test_spte_young(args->vma->vm_mm, addr, end, bitmap, &last); + if (!ret && !pte_young(ptent)) { + skip_spte_young(args->vma->vm_mm, addr, bitmap, &last); walk->mm_stats[MM_LEAF_OLD]++; continue; } folio = get_pfn_folio(pfn, memcg, pgdat, walk->can_swap); - if (!folio) + if (!folio) { + skip_spte_young(args->vma->vm_mm, addr, bitmap, &last); continue; + } - if (!ptep_test_and_clear_young(args->vma, addr, pte + i)) - VM_WARN_ON_ONCE(true); + clear_spte_young(args->vma->vm_mm, addr, bitmap, &last); + if (pte_young(ptent)) + ptep_test_and_clear_young(args->vma, addr, pte + i); young++; walk->mm_stats[MM_LEAF_YOUNG]++; @@ -3473,22 +3541,24 @@ static void walk_pmd_range_locked(pud_t *pud, unsigned long addr, struct vm_area /* don't round down the first address */ addr = i ? (*first & PMD_MASK) + i * PMD_SIZE : *first; - pfn = get_pmd_pfn(pmd[i], vma, addr); - if (pfn == -1) - goto next; - - if (!pmd_trans_huge(pmd[i])) { - if (should_clear_pmd_young()) + if (pmd_present(pmd[i]) && !pmd_trans_huge(pmd[i])) { + if (should_clear_pmd_young() && !mm_has_notifiers(args->mm)) pmdp_test_and_clear_young(vma, addr, pmd + i); goto next; } + pfn = get_pmd_pfn(pmd[i], vma, addr, pgdat); + if (pfn == -1) + goto next; + folio = get_pfn_folio(pfn, memcg, pgdat, walk->can_swap); if (!folio) goto next; - if (!pmdp_test_and_clear_young(vma, addr, pmd + i)) + if (!pmdp_clear_young_notify(vma, addr, pmd + i)) { + walk->mm_stats[MM_LEAF_OLD]++; goto next; + } walk->mm_stats[MM_LEAF_YOUNG]++; @@ -3545,19 +3615,18 @@ static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end, } if (pmd_trans_huge(val)) { - unsigned long pfn = pmd_pfn(val); struct pglist_data *pgdat = lruvec_pgdat(walk->lruvec); + unsigned long pfn = get_pmd_pfn(val, vma, addr, pgdat); walk->mm_stats[MM_LEAF_TOTAL]++; - if (!pmd_young(val)) { - walk->mm_stats[MM_LEAF_OLD]++; + if (pfn == -1) continue; - } - /* try to avoid unnecessary memory loads */ - if (pfn < pgdat->node_start_pfn || pfn >= pgdat_end_pfn(pgdat)) + if (!pmd_young(val) && !mm_has_notifiers(args->mm)) { + walk->mm_stats[MM_LEAF_OLD]++; continue; + } walk_pmd_range_locked(pud, addr, vma, args, bitmap, &first); continue; @@ -3565,7 +3634,7 @@ static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end, walk->mm_stats[MM_NONLEAF_TOTAL]++; - if (should_clear_pmd_young()) { + if (should_clear_pmd_young() && !mm_has_notifiers(args->mm)) { if (!pmd_young(val)) continue; @@ -3646,6 +3715,9 @@ static void walk_mm(struct mm_struct *mm, struct lru_gen_mm_walk *walk) struct lruvec *lruvec = walk->lruvec; struct mem_cgroup *memcg = lruvec_memcg(lruvec); + if (!should_walk_kvm_mmu() && mm_has_notifiers(mm)) + return; + walk->next_addr = FIRST_USER_ADDRESS; do { @@ -4011,6 +4083,23 @@ static void lru_gen_age_node(struct pglist_data *pgdat, struct scan_control *sc) * rmap/PT walk feedback ******************************************************************************/ +static bool should_look_around(struct vm_area_struct *vma, unsigned long addr, + pte_t *pte, int *young) +{ + int ret = mmu_notifier_clear_young(vma->vm_mm, addr, addr + PAGE_SIZE); + + if (pte_young(ptep_get(pte))) { + ptep_test_and_clear_young(vma, addr, pte); + *young = true; + return true; + } + + if (ret) + *young = true; + + return ret & MMU_NOTIFIER_YOUNG_FAST; +} + /* * This function exploits spatial locality when shrink_folio_list() walks the * rmap. It scans the adjacent PTEs of a young PTE and promotes hot pages. If @@ -4018,12 +4107,14 @@ static void lru_gen_age_node(struct pglist_data *pgdat, struct scan_control *sc) * the PTE table to the Bloom filter. This forms a feedback loop between the * eviction and the aging. */ -void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) +bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw) { int i; unsigned long start; unsigned long end; struct lru_gen_mm_walk *walk; + DECLARE_BITMAP(bitmap, MIN_LRU_BATCH); + unsigned long last = 0; int young = 0; pte_t *pte = pvmw->pte; unsigned long addr = pvmw->address; @@ -4040,12 +4131,15 @@ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) lockdep_assert_held(pvmw->ptl); VM_WARN_ON_ONCE_FOLIO(folio_test_lru(folio), folio); + if (!should_look_around(pvmw->vma, addr, pte, &young)) + return young; + if (spin_is_contended(pvmw->ptl)) - return; + return young; /* exclude special VMAs containing anon pages from COW */ if (vma->vm_flags & VM_SPECIAL) - return; + return young; /* avoid taking the LRU lock under the PTL when possible */ walk = current->reclaim_state ? current->reclaim_state->mm_walk : NULL; @@ -4053,6 +4147,9 @@ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) start = max(addr & PMD_MASK, vma->vm_start); end = min(addr | ~PMD_MASK, vma->vm_end - 1) + 1; + if (end - start == PAGE_SIZE) + return young; + if (end - start > MIN_LRU_BATCH * PAGE_SIZE) { if (addr - start < MIN_LRU_BATCH * PAGE_SIZE / 2) end = start + MIN_LRU_BATCH * PAGE_SIZE; @@ -4066,29 +4163,38 @@ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) /* folio_update_gen() requires stable folio_memcg() */ if (!mem_cgroup_trylock_pages(memcg)) - return; + return young; arch_enter_lazy_mmu_mode(); pte -= (addr - start) / PAGE_SIZE; for (i = 0, addr = start; addr != end; i++, addr += PAGE_SIZE) { + bool ret; unsigned long pfn; pte_t ptent = ptep_get(pte + i); - pfn = get_pte_pfn(ptent, vma, addr); - if (pfn == -1) + pfn = get_pte_pfn(ptent, vma, addr, pgdat); + if (pfn == -1) { + skip_spte_young(vma->vm_mm, addr, bitmap, &last); continue; + } - if (!pte_young(ptent)) + ret = test_spte_young(pvmw->vma->vm_mm, addr, end, bitmap, &last); + if (!ret && !pte_young(ptent)) { + skip_spte_young(pvmw->vma->vm_mm, addr, bitmap, &last); continue; + } folio = get_pfn_folio(pfn, memcg, pgdat, can_swap); - if (!folio) + if (!folio) { + skip_spte_young(vma->vm_mm, addr, bitmap, &last); continue; + } - if (!ptep_test_and_clear_young(vma, addr, pte + i)) - VM_WARN_ON_ONCE(true); + clear_spte_young(vma->vm_mm, addr, bitmap, &last); + if (pte_young(ptent)) + ptep_test_and_clear_young(vma, addr, pte + i); young++; @@ -4118,6 +4224,8 @@ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) /* feedback from rmap walkers to page table walkers */ if (mm_state && suitable_to_scan(i, young)) update_bloom_filter(mm_state, max_seq, pvmw->pmd); + + return young; } /****************************************************************************** @@ -5154,6 +5262,9 @@ static ssize_t enabled_show(struct kobject *kobj, struct kobj_attribute *attr, c if (should_clear_pmd_young()) caps |= BIT(LRU_GEN_NONLEAF_YOUNG); + if (should_walk_kvm_mmu()) + caps |= BIT(LRU_GEN_KVM_MMU_WALK); + return sysfs_emit(buf, "0x%04x\n", caps); }