From patchwork Mon Apr 1 23:29:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13613090 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70472CD1288 for ; Mon, 1 Apr 2024 23:30:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6ABFE6B0092; Mon, 1 Apr 2024 19:30:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 683886B0093; Mon, 1 Apr 2024 19:30:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4FC2C6B0095; Mon, 1 Apr 2024 19:30:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 34CC96B0092 for ; Mon, 1 Apr 2024 19:30:00 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 0107F403E9 for ; Mon, 1 Apr 2024 23:29:59 +0000 (UTC) X-FDA: 81962558160.25.B6001D8 Received: from mail-vk1-f202.google.com (mail-vk1-f202.google.com [209.85.221.202]) by imf14.hostedemail.com (Postfix) with ESMTP id 5CAF310000F for ; Mon, 1 Apr 2024 23:29:58 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=oIV+2i66; spf=pass (imf14.hostedemail.com: domain of 3dUMLZgoKCGAHRFMSEFRMLEMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--jthoughton.bounces.google.com designates 209.85.221.202 as permitted sender) smtp.mailfrom=3dUMLZgoKCGAHRFMSEFRMLEMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712014198; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XjV19huKRvwcv3bjf5GpogmLoEGdyfdviROgtmpM2BI=; b=8panYJlSyTmBE6He4UNolYp8+LXgrHmxd6h17h4tOLtwowfhXZm4uDm4T46JL0gRGl3w0I uX3PaxUnjXVMZYAa/R+cqsOj2MmdpeLaFbEmVgC0U9KHE9bkVmD3Tk9EUpgg/K6HoodHo0 SNCpbDwz7FQbx+4E4uJsHuVuKUsi9Fk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712014198; a=rsa-sha256; cv=none; b=IlYRpnPwyjuwcwRNNL3gNdSoAYv+3zsNL55SinfQPyqfpQzt1M/e/vv6r5n7MipAeCEKBA WMF9oNCBH3eNA+SblKjRYjN1dc5Mf54wvx4ysbFao4c3Zz3qWQG5gLWgCLsa9GrLaWfzBX 8y0phIeBr4uiHAYtvehmVXbD5yU29LM= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=oIV+2i66; spf=pass (imf14.hostedemail.com: domain of 3dUMLZgoKCGAHRFMSEFRMLEMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--jthoughton.bounces.google.com designates 209.85.221.202 as permitted sender) smtp.mailfrom=3dUMLZgoKCGAHRFMSEFRMLEMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-vk1-f202.google.com with SMTP id 71dfb90a1353d-4d8a322b233so1782827e0c.2 for ; Mon, 01 Apr 2024 16:29:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1712014197; x=1712618997; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=XjV19huKRvwcv3bjf5GpogmLoEGdyfdviROgtmpM2BI=; b=oIV+2i66RPf9s1bjNqFu3PNQb6zEliTY13Ct+pPI5O3NmlUuXKKMpAZSTm3b08fR0Q l/V0nl7MZ7wDdb81s8Kk9DRVk8FDkYJhS04a5RGRpS/MVAmfN7sU/9SZsdBRX2u5bFc8 +dGGKYxKeIoVRkC3UmLgth8h8Kj+gL/1Fl1jCRNw2qnIl4c115hYLrNFCEgAiI8mZ49z HMvk5/jPbKtdOX7u5QLTDcKqH1qD5a1CQKPt7RHjdS+pfRcsmmqHLhsxnBJE5aydOlo6 kkjipZ6DrDrzYIAaedAoTYtD6WpXoyFiNNvIMycU1VoQ50/UUPmujg9GahqRGWZOW+97 cuww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712014197; x=1712618997; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=XjV19huKRvwcv3bjf5GpogmLoEGdyfdviROgtmpM2BI=; b=QPa9/mlIgQtthwtJ4zdq64uAk/Vy1t8Cm+u+gpk0R1x1k6agHfq3bLlIZ00PrHH7DX 3PBZccRQ+s7hlzXgHTnNxcQHgcuvduc+G6+bd/VUDD0IFDleTk1shnm3H0VduD8QJjVu fBd7BytI2B4/SusJTEgC7uuPTZrM6yHzXV8s6/P2398jZxRNOs2MWA2x1lcOTXEYtEYo 6ctlManaqJg4ibPzZx9NEcSd8ZmOeftXisH4qjMQgp+Q4otBvkqPCsJyEHnr8/jBKVTd RBX3rtVJLzPDDr7pPvIrrxGWx3xbJc4mTgrvjmx6nr47IXecvNzJ/DeAkVANarqsA5c+ uZhg== X-Forwarded-Encrypted: i=1; AJvYcCWJEalqqC2cRJJne5PixYY6GkRFzxkpPLgWUsi3GzWidQIjRiw87tGveVuBHLuKgN9I7z/wwicf6+RYzKm4Pu1Q9X0= X-Gm-Message-State: AOJu0YzdsXOmxKqiaFBERcIEb5nBRYFtp/2WrwxJ/7nwM3tT1DHgJPPy vU+eG209jWN+WBtLT0CqMlEVln4y9pYPZnDoxtKpGU3wKdehppNZS7DganrH4J9UWF2iohQro51 GSsPl20Xhv1FMFQaKpg== X-Google-Smtp-Source: AGHT+IFUopEMKakeSmmDQQkpzKxlNJ65mvpQqylZZQK+yo+BzjzMVCQHtrhU8U2Q57twwQQQfTybPo3O3BawFKgE X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:ac5:c281:0:b0:4d8:73c1:9ec7 with SMTP id h1-20020ac5c281000000b004d873c19ec7mr84232vkk.0.1712014197363; Mon, 01 Apr 2024 16:29:57 -0700 (PDT) Date: Mon, 1 Apr 2024 23:29:40 +0000 In-Reply-To: <20240401232946.1837665-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240401232946.1837665-1-jthoughton@google.com> X-Mailer: git-send-email 2.44.0.478.gd926399ef9-goog Message-ID: <20240401232946.1837665-2-jthoughton@google.com> Subject: [PATCH v3 1/7] mm: Add a bitmap into mmu_notifier_{clear,test}_young From: James Houghton To: Andrew Morton , Paolo Bonzini Cc: Yu Zhao , David Matlack , Marc Zyngier , Oliver Upton , Sean Christopherson , Jonathan Corbet , James Morse , Suzuki K Poulose , Zenghui Yu , Catalin Marinas , Will Deacon , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers , Shaoqin Huang , Gavin Shan , Ricardo Koller , Raghavendra Rao Ananta , Ryan Roberts , David Rientjes , Axel Rasmussen , linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, James Houghton X-Rspamd-Queue-Id: 5CAF310000F X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: wtera7n5ys4uo1wwynzuiwoamsss661x X-HE-Tag: 1712014198-370401 X-HE-Meta: U2FsdGVkX19a8K43CI/ZTXtxm+IpLTHV+Bngyxu2rHk6MKhuZztmUf3QZFCyJfQia3PrUGOPLqtRa4JBD3HWd4Nz1sqv5zPW8RefqpKY0eCVLVTpW514V2grooVwuzWWu1p6hLoQPEMhsErNq4ygNfW9Qm76W+9xmI/duwxRS0Lyl+HKUQI+0n0KvV3XqmkyIKLroMk06gUFJ3nQAf3Rqk8uC0rp0FFFpwa708Wifs3zb40NaBpN2k1K2+AYlC9sxj4UpM0/W7b+eXgR64HJjcLABMAeMHep2NDZZy/utInGEfkYGB056v+gPE7PS1x95GrhzTPoPfAHDStnAKTVxQKPRgkqhpOrRfilTQRwxXggVVxjp2PSjUmaWRT4man3hu7B/RgfqDLlfugIjMZ1wrxkso79NHlyG9AkvlrWfKHdAbOTAcpYMd3hmAPBb+EvMgQR0n0vsKsIIobf1frj3iydyoSBR4GjPzFeowGvsAneFRkXPEEOqkLGu/Y3M2ByA1RtuiFQZX06i3wGH6jicK6FNipOFpPX4lD7AXn4dBNn43sGAKose8oY/dPt70gYrnKdaUtQvnlpybSKH5TWMe9h2Y9SKT5ihvU8JipGgYG6tPn71az063SR+1wwGMDM1YzYXs55wvOm0UlFqqjMWblm/YlPGdO8GBOTjTUjlysbNjCJ4JqTycjXZK0y/oxMd8HZF5TOesEbS9BEETDFexEobcxSMfj1pnbtqytJxHIBVN55k2iPh0E4pjhqcs6H6Mf43PZhjq6EeGk4+gVX9+soLou8ScQkhCeQ4gYFeBXhmxP6QJMve8Ehd9rlzWuy3N72HPQOQBNYMSpiUYdDOiWfM0fUN9FdZEMzU5tmY7Us0RnOL62t29jnxDlTGH5uwf/cRkPhT91fh5/xIprdiQlFEHqd/O0t6vFRcuryQ1uZhOeWrkblTdD9VrE1ByNCOg+bAOyFU0NjVaOkwz0 lOJoraI0 21/YtUVTDPJ5qQ+nc4M6xmUMqoWn5XgaZbUUGawOxFNNIRrBAOsDlGA+ytDpNYrR0ORxXPO9Xi7+RBYhj0IVj4UwJFYSuBMVpuK2CQcnNwhc6weMtN5o95NGGfa/jbmws2Hf+xOnTXhzmY3qw/SYzFCCPQGtIHpBLo4YPi7pNqiGzHpVwh5SHig81pe73PnnFAzfFIFMS6qsWsoYN3CRftWrDdahF/Hr9N0Hi3S1CN0LGigtEOMT/SSdltUFWNIpFC9wpveDmLkP4VBBJFG8vNXUmJusjc86OrBWXBWfqUFh6AWMpBKIS2xhCsSrv2SINrpY1MCPgnstO6nQVYtZv1VpFqA6gEDiFsFQ/tBGMAcbcf+KCmuY44yQUhwzTCqaflYoIMgmX87eAveWzUHLFaXfItMyrShDy7cZHx8do2NfteSM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The bitmap is provided for secondary MMUs to use if they support it. For test_young(), after it returns, the bitmap represents the pages that were young in the interval [start, end). For clear_young, it represents the pages that we wish the secondary MMU to clear the accessed/young bit for. If a bitmap is not provided, the mmu_notifier_{test,clear}_young() API should be unchanged except that if young PTEs are found and the architecture supports passing in a bitmap, instead of returning 1, MMU_NOTIFIER_YOUNG_FAST is returned. This allows MGLRU's look-around logic to work faster, resulting in a 4% improvement in real workloads[1]. Also introduce MMU_NOTIFIER_YOUNG_FAST to indicate to main mm that doing look-around is likely to be beneficial. If the secondary MMU doesn't support the bitmap, it must return an int that contains MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE. [1]: https://lore.kernel.org/all/20230609005935.42390-1-yuzhao@google.com/ Suggested-by: Yu Zhao Signed-off-by: James Houghton --- include/linux/mmu_notifier.h | 93 +++++++++++++++++++++++++++++++++--- include/trace/events/kvm.h | 13 +++-- mm/mmu_notifier.c | 20 +++++--- virt/kvm/kvm_main.c | 19 ++++++-- 4 files changed, 123 insertions(+), 22 deletions(-) diff --git a/include/linux/mmu_notifier.h b/include/linux/mmu_notifier.h index f349e08a9dfe..daaa9db625d3 100644 --- a/include/linux/mmu_notifier.h +++ b/include/linux/mmu_notifier.h @@ -61,6 +61,10 @@ enum mmu_notifier_event { #define MMU_NOTIFIER_RANGE_BLOCKABLE (1 << 0) +#define MMU_NOTIFIER_YOUNG (1 << 0) +#define MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE (1 << 1) +#define MMU_NOTIFIER_YOUNG_FAST (1 << 2) + struct mmu_notifier_ops { /* * Called either by mmu_notifier_unregister or when the mm is @@ -106,21 +110,36 @@ struct mmu_notifier_ops { * clear_young is a lightweight version of clear_flush_young. Like the * latter, it is supposed to test-and-clear the young/accessed bitflag * in the secondary pte, but it may omit flushing the secondary tlb. + * + * If @bitmap is given but is not supported, return + * MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE. + * + * If the walk is done "quickly" and there were young PTEs, + * MMU_NOTIFIER_YOUNG_FAST is returned. */ int (*clear_young)(struct mmu_notifier *subscription, struct mm_struct *mm, unsigned long start, - unsigned long end); + unsigned long end, + unsigned long *bitmap); /* * test_young is called to check the young/accessed bitflag in * the secondary pte. This is used to know if the page is * frequently used without actually clearing the flag or tearing * down the secondary mapping on the page. + * + * If @bitmap is given but is not supported, return + * MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE. + * + * If the walk is done "quickly" and there were young PTEs, + * MMU_NOTIFIER_YOUNG_FAST is returned. */ int (*test_young)(struct mmu_notifier *subscription, struct mm_struct *mm, - unsigned long address); + unsigned long start, + unsigned long end, + unsigned long *bitmap); /* * change_pte is called in cases that pte mapping to page is changed: @@ -388,10 +407,11 @@ extern int __mmu_notifier_clear_flush_young(struct mm_struct *mm, unsigned long start, unsigned long end); extern int __mmu_notifier_clear_young(struct mm_struct *mm, - unsigned long start, - unsigned long end); + unsigned long start, unsigned long end, + unsigned long *bitmap); extern int __mmu_notifier_test_young(struct mm_struct *mm, - unsigned long address); + unsigned long start, unsigned long end, + unsigned long *bitmap); extern void __mmu_notifier_change_pte(struct mm_struct *mm, unsigned long address, pte_t pte); extern int __mmu_notifier_invalidate_range_start(struct mmu_notifier_range *r); @@ -427,7 +447,25 @@ static inline int mmu_notifier_clear_young(struct mm_struct *mm, unsigned long end) { if (mm_has_notifiers(mm)) - return __mmu_notifier_clear_young(mm, start, end); + return __mmu_notifier_clear_young(mm, start, end, NULL); + return 0; +} + +/* + * When @bitmap is not provided, clear the young bits in the secondary + * MMUs for all of the pages in the interval [start, end). + * + * If any subscribed secondary MMU does not support @bitmap, this function + * will return an integer containing MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE. + * Some work may have been done in the secondary MMU. + */ +static inline int mmu_notifier_clear_young_bitmap(struct mm_struct *mm, + unsigned long start, + unsigned long end, + unsigned long *bitmap) +{ + if (mm_has_notifiers(mm)) + return __mmu_notifier_clear_young(mm, start, end, bitmap); return 0; } @@ -435,7 +473,25 @@ static inline int mmu_notifier_test_young(struct mm_struct *mm, unsigned long address) { if (mm_has_notifiers(mm)) - return __mmu_notifier_test_young(mm, address); + return __mmu_notifier_test_young(mm, address, address + 1, + NULL); + return 0; +} + +/* + * When @bitmap is not provided, test the young bits in the secondary + * MMUs for all of the pages in the interval [start, end). + * + * If any subscribed secondary MMU does not support @bitmap, this function + * will return an integer containing MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE. + */ +static inline int mmu_notifier_test_young_bitmap(struct mm_struct *mm, + unsigned long start, + unsigned long end, + unsigned long *bitmap) +{ + if (mm_has_notifiers(mm)) + return __mmu_notifier_test_young(mm, start, end, bitmap); return 0; } @@ -644,12 +700,35 @@ static inline int mmu_notifier_clear_flush_young(struct mm_struct *mm, return 0; } +static inline int mmu_notifier_clear_young(struct mm_struct *mm, + unsigned long start, + unsigned long end) +{ + return 0; +} + +static inline int mmu_notifier_clear_young_bitmap(struct mm_struct *mm, + unsigned long start, + unsigned long end, + unsigned long *bitmap) +{ + return 0; +} + static inline int mmu_notifier_test_young(struct mm_struct *mm, unsigned long address) { return 0; } +static inline int mmu_notifier_test_young_bitmap(struct mm_struct *mm, + unsigned long start, + unsigned long end, + unsigned long *bitmap) +{ + return 0; +} + static inline void mmu_notifier_change_pte(struct mm_struct *mm, unsigned long address, pte_t pte) { diff --git a/include/trace/events/kvm.h b/include/trace/events/kvm.h index 011fba6b5552..e4ace8cfdbba 100644 --- a/include/trace/events/kvm.h +++ b/include/trace/events/kvm.h @@ -490,18 +490,21 @@ TRACE_EVENT(kvm_age_hva, ); TRACE_EVENT(kvm_test_age_hva, - TP_PROTO(unsigned long hva), - TP_ARGS(hva), + TP_PROTO(unsigned long start, unsigned long end), + TP_ARGS(start, end), TP_STRUCT__entry( - __field( unsigned long, hva ) + __field( unsigned long, start ) + __field( unsigned long, end ) ), TP_fast_assign( - __entry->hva = hva; + __entry->start = start; + __entry->end = end; ), - TP_printk("mmu notifier test age hva: %#016lx", __entry->hva) + TP_printk("mmu notifier test age hva: %#016lx -- %#016lx", + __entry->start, __entry->end) ); #endif /* _TRACE_KVM_MAIN_H */ diff --git a/mm/mmu_notifier.c b/mm/mmu_notifier.c index ec3b068cbbe6..e70c6222944c 100644 --- a/mm/mmu_notifier.c +++ b/mm/mmu_notifier.c @@ -384,7 +384,8 @@ int __mmu_notifier_clear_flush_young(struct mm_struct *mm, int __mmu_notifier_clear_young(struct mm_struct *mm, unsigned long start, - unsigned long end) + unsigned long end, + unsigned long *bitmap) { struct mmu_notifier *subscription; int young = 0, id; @@ -395,7 +396,8 @@ int __mmu_notifier_clear_young(struct mm_struct *mm, srcu_read_lock_held(&srcu)) { if (subscription->ops->clear_young) young |= subscription->ops->clear_young(subscription, - mm, start, end); + mm, start, end, + bitmap); } srcu_read_unlock(&srcu, id); @@ -403,7 +405,8 @@ int __mmu_notifier_clear_young(struct mm_struct *mm, } int __mmu_notifier_test_young(struct mm_struct *mm, - unsigned long address) + unsigned long start, unsigned long end, + unsigned long *bitmap) { struct mmu_notifier *subscription; int young = 0, id; @@ -413,9 +416,14 @@ int __mmu_notifier_test_young(struct mm_struct *mm, &mm->notifier_subscriptions->list, hlist, srcu_read_lock_held(&srcu)) { if (subscription->ops->test_young) { - young = subscription->ops->test_young(subscription, mm, - address); - if (young) + young |= subscription->ops->test_young(subscription, mm, + start, end, + bitmap); + if (young && !bitmap) + /* + * We're not using a bitmap, so there is no + * need to check any more secondary MMUs. + */ break; } } diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index fb49c2a60200..ca4b1ef9dfc2 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -917,10 +917,15 @@ static int kvm_mmu_notifier_clear_flush_young(struct mmu_notifier *mn, static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn, struct mm_struct *mm, unsigned long start, - unsigned long end) + unsigned long end, + unsigned long *bitmap) { trace_kvm_age_hva(start, end); + /* We don't support bitmaps. Don't test or clear anything. */ + if (bitmap) + return MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE; + /* * Even though we do not flush TLB, this will still adversely * affect performance on pre-Haswell Intel EPT, where there is @@ -939,11 +944,17 @@ static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn, static int kvm_mmu_notifier_test_young(struct mmu_notifier *mn, struct mm_struct *mm, - unsigned long address) + unsigned long start, + unsigned long end, + unsigned long *bitmap) { - trace_kvm_test_age_hva(address); + trace_kvm_test_age_hva(start, end); + + /* We don't support bitmaps. Don't test or clear anything. */ + if (bitmap) + return MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE; - return kvm_handle_hva_range_no_flush(mn, address, address + 1, + return kvm_handle_hva_range_no_flush(mn, start, end, kvm_test_age_gfn); }