From patchwork Wed Jul 24 01:10:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13740501 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E5D69C3DA64 for ; Wed, 24 Jul 2024 01:11:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 46EE76B008A; Tue, 23 Jul 2024 21:11:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 41B7E6B008C; Tue, 23 Jul 2024 21:11:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 110F56B0092; Tue, 23 Jul 2024 21:11:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id DAC356B008A for ; Tue, 23 Jul 2024 21:11:20 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 82C5EC057C for ; Wed, 24 Jul 2024 01:11:20 +0000 (UTC) X-FDA: 82372867920.06.360CD04 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf22.hostedemail.com (Postfix) with ESMTP id C08F4C0007 for ; Wed, 24 Jul 2024 01:11:18 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=QYzRCHBv; spf=pass (imf22.hostedemail.com: domain of 3tVSgZgoKCBwBL9GM89LGF8GG8D6.4GEDAFMP-EECN24C.GJ8@flex--jthoughton.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3tVSgZgoKCBwBL9GM89LGF8GG8D6.4GEDAFMP-EECN24C.GJ8@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1721783441; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MLmhrACopMN8CDbTxEqiwIPa4YrGM6ZGpV4/JEvInYA=; b=LrZ2Ea5LngaZQ6+lIyd25mZIDfw32PXDX4pUyKcPSMR1V8WNmxNUesRpSuCq5e8GRB3eXD DUbvyu+UCzPjhSATeIKKYR3NBvSr4ZxGiVpJFK4Kp7Ep4bc3aO8R0rXeTrynYZaZeG0Tvn TjDExfsGLsrreWKicZnvpjotvKL2JoM= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=QYzRCHBv; spf=pass (imf22.hostedemail.com: domain of 3tVSgZgoKCBwBL9GM89LGF8GG8D6.4GEDAFMP-EECN24C.GJ8@flex--jthoughton.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3tVSgZgoKCBwBL9GM89LGF8GG8D6.4GEDAFMP-EECN24C.GJ8@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1721783441; a=rsa-sha256; cv=none; b=hh3BEw6uaLBtfalB283ZBAPs4XulcCm6RRkQ14yTBfxC/gdZspJM4HWUGXH8YSauBe805Y ctdi3N/DhmqNLyFTX1dkQMnB+vRpmFCFQwMSSYJRJ8HRNwb1GyofUR0FZdifRSFEgUTF2A BJ/ZPiJ9x8bkJkHZjhmpYLHbtj4XAzU= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-e08723e9d7cso7458371276.0 for ; Tue, 23 Jul 2024 18:11:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1721783478; x=1722388278; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=MLmhrACopMN8CDbTxEqiwIPa4YrGM6ZGpV4/JEvInYA=; b=QYzRCHBvrkCiCYa8dt3hBzRvo+OJkWG/JFj4RLOMPAXwg5E4VI2Di7mzw9ow6inx37 t3mnq6afbV1tjKJIZxG/tc8UVL7NR1b0WKixVWrSpGfScZNgZ6U8AWqXVMwJyLVnKHr2 bQX+AyxLB8udMiUcQ3h2gX2q25H86DyE4qJnIGKButvkYTcELATrTDw3zZMcz3Kse4Lx d81n3FdtC8PVnsNdgPENbAtDcNeOx1cv4Ag9hQSGOBq+eBtx5nUlnBE9hLhVKk5aJ+99 UNYk9mP0Tn1jG0SmMgMghqlZMvla7jQRsvClmqPOapfb6Ji7JS09lbJWIpft/EQ27uXk zPVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721783478; x=1722388278; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=MLmhrACopMN8CDbTxEqiwIPa4YrGM6ZGpV4/JEvInYA=; b=qYk6+70X4c72CBCmIG9vnU85VZN1bvJYeUDgAfsMfp+IXF8ByDxn78w+ZcRpXnmcDA GLmdvNbTWYA91JBttYnm3AEoWd/xwv7xnO68+QjZuOlSWrGB3yyPsj6I6khVs+j9wYdw lizo8l8nw93teGKjV0pxeKobr/OOU785nTS4htg2W1bII6G7HlMuRvyhc4/um6IyNur5 JEaLS9HxNvJZuT2HsNRUApkBwEApxnOziDpq0erEvN55Szn6XzMYaZ4k2PpEFvsmDtV1 98q3GbwwMp3U9KL5GsdyviS2Rxf6Dcq1rvKrO5AS+82MV1ol1mO9P7NgG2WhuTlCBOrQ vCoA== X-Forwarded-Encrypted: i=1; AJvYcCWq4tJLqvSb64H7f+0kUAd0DTBY2956e4FPrgbXqPAzFBDN3ss0j5gYLYiPHmKlzOd9MZ7QTtqPbTtM6vKA1t+pBOI= X-Gm-Message-State: AOJu0YybSG99geoRXxale6bvMijh1YzxFwP9cmbbMGNsW7J4NNbVTP4C GtQxLrjvOwtBvB/YZT8uZFX+IyB3EHLIHEMtsD+vaHVAe7Nqh0KN0GEkaKnSa9u30rlKsGjhuJc Hsfo5wiVyVCWmBdRWiQ== X-Google-Smtp-Source: AGHT+IGmLYw0C2YUyk3sIv0ePZsCocY4rFKls0LnHS1LhedGuIns0QZPtcJEKudMyoCE03fu3hfho3QJFpB7QIkQ X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a05:6902:100f:b0:e05:fb86:1909 with SMTP id 3f1490d57ef6-e0b0982aef8mr23085276.6.1721783477681; Tue, 23 Jul 2024 18:11:17 -0700 (PDT) Date: Wed, 24 Jul 2024 01:10:30 +0000 In-Reply-To: <20240724011037.3671523-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240724011037.3671523-1-jthoughton@google.com> X-Mailer: git-send-email 2.46.0.rc1.232.g9752f9e123-goog Message-ID: <20240724011037.3671523-6-jthoughton@google.com> Subject: [PATCH v6 05/11] mm: Add fast_only bool to test_young and clear_young MMU notifiers From: James Houghton To: Andrew Morton , Paolo Bonzini Cc: Ankit Agrawal , Axel Rasmussen , Catalin Marinas , David Matlack , David Rientjes , James Houghton , James Morse , Jason Gunthorpe , Jonathan Corbet , Marc Zyngier , Oliver Upton , Raghavendra Rao Ananta , Ryan Roberts , Sean Christopherson , Shaoqin Huang , Suzuki K Poulose , Wei Xu , Will Deacon , Yu Zhao , Zenghui Yu , kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org X-Stat-Signature: rnts9hd6bo4wqj6zmdrpi1piwy8dzonb X-Rspam-User: X-Rspamd-Queue-Id: C08F4C0007 X-Rspamd-Server: rspam02 X-HE-Tag: 1721783478-344211 X-HE-Meta: U2FsdGVkX1+j+Z396AiEP8rvWfYKPYseITtZmJjIMe8I4TJu7WYHwRESH1qI/OHAwBqfkW6uhkBQWS39+VpOlFamtzmncappbgK4AUQQVJ7v1NriHVJ/8fdFygxBk9tz1Rm4YyKh5RkOyeCugldIp6LF5u7c0uDmw+sRFgnuUrdclgOn3UDNOJu1AxmxoOSDZmbpS4SLrTgxe6v+kPCIgNtUDi/vtJS+gFFE0Y+kpGLxMYvzCiI+82jmnwyZrnMEkQvjjMCH6bicMl/UveeW/pHtda6uzByVwxdcz9nqEBA1wIqQKt0ws7XyM2GiIgr6wOaua/9RTs9RfUvtON49gdLGwIdmPxBqzqii4XzwbUFZQQQOvSKfvTs0ra5tRYtSv/jc9d+SHHU1rzwjcqGBlfqEVPsqSJoOWZbLDNM60rnQwricz9D3pTHPSP66Wa7PCYr+J9aEuO7zMbsR5xf94zEeQ1X2kzBYxLbi8gebPA8BMPBt/+qgDuKOUUAN7e5e7ExPuMWZW9B48A8RKHLarXJ4ND7emIYjLICGRk/jr+ewxbbGag25v+V1Epbx7+sbCY/EFrHqi4eGyDsLWVRpxMLz00oVi4f93Jv4RjQDKVdG64IAER6QY/rtCwux1dUhw4f23VaR0G6m6WWINfMevq58IAnj1Xu0gfjyXSADcX68+jp0OGf56q5l+Pobg/iDdTehWwWGk308ub4CfmsIVtv9ueR1fAf0bo0nqh0KcUw62OFcm1urUNgbHwc9Xu2bPLJ0FJYMOJCn4WsRb8p0ZPwxU1hsOL3NuMDmgKT37NS4ajeaiR6u77ZJlyhwCCYEOTEYK3JMYimQSt0OKY6ornMzspkc3YAQjR8YoUbjyIa6rSHVaKmupszFc/gPCwYMIUI3xSUAUOaxr9Avq2T5S/3vsusmiXcLvnB42ar+exN0DVof9dLbLsi4DXwxm0lx89Or29z3FaFvFXYs0h6 +TPLoT47 6xLQWbohVgvJB0G8/6aJ6CP5PYeoQefh2h+WZtF6gpkllARetz2YSHvipuOVQZar66Q3cvM6CZjzDzSS1laV4uaD8NUrMCnNoyLpkE3RGbE4j4++pLqH+KktKElX9HE/eOHTQRBZDxvN1pemFgUkQU/aa8y9Y++eWSNKis3rMtyvmSCtciES+8Yq8sBv8kNM6QIUCoSCkUnk1TUw1G0gjPZBjBWtL2P6SgyrWyd+qtQobWL+0e3zJW2ZHrshBSdEQY0+XNpCbvlSupntHSmyRHb5K/EuJ/mZm7YDqZ+x3esWuJxhVYFo3x4thnrd1LSLlwiHzgeQ3EChk6m5syOuIXQL6MP+jQzwiF68LRqxhtlnuzcKo5VB1RATgAjyRwOmRdJnpAVYeE5OZMdgVzx/OaMrPK81MgDlZHYpNbCPBSJVa4mxgtJu5M+DBqOTT310uog7Vxx5+WMWXIuBHMOqDVoD0Rqj+VMW/8vTgg9/sVVIIRyHfFfTWczOrfxVe3PS8C2N7SjOYsBw7D109Kq9rOVIPbRkYcMnli5FszTURA99PyBJ5547t3/2yvqeba4NU6YlU018idDl/IibFY7Gf7ucglxrDFZTfqUi5 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: For implementers, the fast_only bool indicates that the age information needs to be harvested such that we do not slow down other MMU operations, and ideally that we are not ourselves slowed down by other MMU operations. Usually this means that the implementation should be lockless. Also add mmu_notifier_test_young_fast_only() and mmu_notifier_clear_young_fast_only() helpers to set fast_only for these notifiers. Signed-off-by: James Houghton --- include/linux/mmu_notifier.h | 46 +++++++++++++++++++++++++++++++----- include/trace/events/kvm.h | 19 +++++++++------ mm/mmu_notifier.c | 12 ++++++---- virt/kvm/kvm_main.c | 12 ++++++---- 4 files changed, 67 insertions(+), 22 deletions(-) diff --git a/include/linux/mmu_notifier.h b/include/linux/mmu_notifier.h index e2dd57ca368b..45c5995ebd84 100644 --- a/include/linux/mmu_notifier.h +++ b/include/linux/mmu_notifier.h @@ -110,7 +110,8 @@ struct mmu_notifier_ops { int (*clear_young)(struct mmu_notifier *subscription, struct mm_struct *mm, unsigned long start, - unsigned long end); + unsigned long end, + bool fast_only); /* * test_young is called to check the young/accessed bitflag in @@ -120,7 +121,8 @@ struct mmu_notifier_ops { */ int (*test_young)(struct mmu_notifier *subscription, struct mm_struct *mm, - unsigned long address); + unsigned long address, + bool fast_only); /* * invalidate_range_start() and invalidate_range_end() must be @@ -380,9 +382,11 @@ extern int __mmu_notifier_clear_flush_young(struct mm_struct *mm, unsigned long end); extern int __mmu_notifier_clear_young(struct mm_struct *mm, unsigned long start, - unsigned long end); + unsigned long end, + bool fast_only); extern int __mmu_notifier_test_young(struct mm_struct *mm, - unsigned long address); + unsigned long address, + bool fast_only); extern int __mmu_notifier_invalidate_range_start(struct mmu_notifier_range *r); extern void __mmu_notifier_invalidate_range_end(struct mmu_notifier_range *r); extern void __mmu_notifier_arch_invalidate_secondary_tlbs(struct mm_struct *mm, @@ -416,7 +420,16 @@ static inline int mmu_notifier_clear_young(struct mm_struct *mm, unsigned long end) { if (mm_has_notifiers(mm)) - return __mmu_notifier_clear_young(mm, start, end); + return __mmu_notifier_clear_young(mm, start, end, false); + return 0; +} + +static inline int mmu_notifier_clear_young_fast_only(struct mm_struct *mm, + unsigned long start, + unsigned long end) +{ + if (mm_has_notifiers(mm)) + return __mmu_notifier_clear_young(mm, start, end, true); return 0; } @@ -424,7 +437,15 @@ static inline int mmu_notifier_test_young(struct mm_struct *mm, unsigned long address) { if (mm_has_notifiers(mm)) - return __mmu_notifier_test_young(mm, address); + return __mmu_notifier_test_young(mm, address, false); + return 0; +} + +static inline int mmu_notifier_test_young_fast_only(struct mm_struct *mm, + unsigned long address) +{ + if (mm_has_notifiers(mm)) + return __mmu_notifier_test_young(mm, address, true); return 0; } @@ -613,12 +634,25 @@ static inline int mmu_notifier_clear_young(struct mm_struct *mm, return 0; } +static inline int mmu_notifier_clear_young_fast_only(struct mm_struct *mm, + unsigned long start, + unsigned long end) +{ + return 0; +} + static inline int mmu_notifier_test_young(struct mm_struct *mm, unsigned long address) { return 0; } +static inline int mmu_notifier_test_young_fast_only(struct mm_struct *mm, + unsigned long address) +{ + return 0; +} + static inline void mmu_notifier_invalidate_range_start(struct mmu_notifier_range *range) { diff --git a/include/trace/events/kvm.h b/include/trace/events/kvm.h index 74e40d5d4af4..6d9485cf3e51 100644 --- a/include/trace/events/kvm.h +++ b/include/trace/events/kvm.h @@ -457,36 +457,41 @@ TRACE_EVENT(kvm_unmap_hva_range, ); TRACE_EVENT(kvm_age_hva, - TP_PROTO(unsigned long start, unsigned long end), - TP_ARGS(start, end), + TP_PROTO(unsigned long start, unsigned long end, bool fast_only), + TP_ARGS(start, end, fast_only), TP_STRUCT__entry( __field( unsigned long, start ) __field( unsigned long, end ) + __field( bool, fast_only ) ), TP_fast_assign( __entry->start = start; __entry->end = end; + __entry->fast_only = fast_only; ), - TP_printk("mmu notifier age hva: %#016lx -- %#016lx", - __entry->start, __entry->end) + TP_printk("mmu notifier age hva: %#016lx -- %#016lx fast_only: %d", + __entry->start, __entry->end, __entry->fast_only) ); TRACE_EVENT(kvm_test_age_hva, - TP_PROTO(unsigned long hva), - TP_ARGS(hva), + TP_PROTO(unsigned long hva, bool fast_only), + TP_ARGS(hva, fast_only), TP_STRUCT__entry( __field( unsigned long, hva ) + __field( bool, fast_only ) ), TP_fast_assign( __entry->hva = hva; + __entry->fast_only = fast_only; ), - TP_printk("mmu notifier test age hva: %#016lx", __entry->hva) + TP_printk("mmu notifier test age hva: %#016lx fast_only: %d", + __entry->hva, __entry->fast_only) ); #endif /* _TRACE_KVM_MAIN_H */ diff --git a/mm/mmu_notifier.c b/mm/mmu_notifier.c index 8982e6139d07..f9a0ca6ffe65 100644 --- a/mm/mmu_notifier.c +++ b/mm/mmu_notifier.c @@ -384,7 +384,8 @@ int __mmu_notifier_clear_flush_young(struct mm_struct *mm, int __mmu_notifier_clear_young(struct mm_struct *mm, unsigned long start, - unsigned long end) + unsigned long end, + bool fast_only) { struct mmu_notifier *subscription; int young = 0, id; @@ -395,7 +396,8 @@ int __mmu_notifier_clear_young(struct mm_struct *mm, srcu_read_lock_held(&srcu)) { if (subscription->ops->clear_young) young |= subscription->ops->clear_young(subscription, - mm, start, end); + mm, start, end, + fast_only); } srcu_read_unlock(&srcu, id); @@ -403,7 +405,8 @@ int __mmu_notifier_clear_young(struct mm_struct *mm, } int __mmu_notifier_test_young(struct mm_struct *mm, - unsigned long address) + unsigned long address, + bool fast_only) { struct mmu_notifier *subscription; int young = 0, id; @@ -414,7 +417,8 @@ int __mmu_notifier_test_young(struct mm_struct *mm, srcu_read_lock_held(&srcu)) { if (subscription->ops->test_young) { young = subscription->ops->test_young(subscription, mm, - address); + address, + fast_only); if (young) break; } diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 33f8997a5c29..959b6d5d8ce4 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -874,7 +874,7 @@ static int kvm_mmu_notifier_clear_flush_young(struct mmu_notifier *mn, unsigned long start, unsigned long end) { - trace_kvm_age_hva(start, end); + trace_kvm_age_hva(start, end, false); return kvm_handle_hva_range(mn, start, end, kvm_age_gfn); } @@ -882,9 +882,10 @@ static int kvm_mmu_notifier_clear_flush_young(struct mmu_notifier *mn, static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn, struct mm_struct *mm, unsigned long start, - unsigned long end) + unsigned long end, + bool fast_only) { - trace_kvm_age_hva(start, end); + trace_kvm_age_hva(start, end, fast_only); /* * Even though we do not flush TLB, this will still adversely @@ -904,9 +905,10 @@ static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn, static int kvm_mmu_notifier_test_young(struct mmu_notifier *mn, struct mm_struct *mm, - unsigned long address) + unsigned long address, + bool fast_only) { - trace_kvm_test_age_hva(address); + trace_kvm_test_age_hva(address, fast_only); return kvm_handle_hva_range_no_flush(mn, address, address + 1, kvm_test_age_gfn);