From patchwork Mon Apr 1 23:29:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13613092 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CBD45CD128A for ; Mon, 1 Apr 2024 23:30:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 43FE16B0095; Mon, 1 Apr 2024 19:30:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3C40A6B0099; Mon, 1 Apr 2024 19:30:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 107726B009A; Mon, 1 Apr 2024 19:30:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id E43E66B0095 for ; Mon, 1 Apr 2024 19:30:01 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id A77A240168 for ; Mon, 1 Apr 2024 23:30:01 +0000 (UTC) X-FDA: 81962558202.25.59652EB Received: from mail-ua1-f73.google.com (mail-ua1-f73.google.com [209.85.222.73]) by imf16.hostedemail.com (Postfix) with ESMTP id DDAC618000C for ; Mon, 1 Apr 2024 23:29:59 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=1O0Be61P; spf=pass (imf16.hostedemail.com: domain of 3dkMLZgoKCGEISGNTFGSNMFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--jthoughton.bounces.google.com designates 209.85.222.73 as permitted sender) smtp.mailfrom=3dkMLZgoKCGEISGNTFGSNMFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712014199; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PTb42yX5D5jCiyjo92iMcOe2nJRX5BH9KgDDQYg5TSM=; b=w7pCGSbP89bcUEKyUl2nDW3LZYDcZpnaG5eYMiI6Qnq9SlATPyEv9/MC1jjFo+f8d41dRX hsxjcpVclU5NH64PbioP+sT/gZZ6z5NIHe2SblJvksyD15ZvZebW/NBfi/a3yev4dcq4/W dVmCC0S3aVmLpvwL+KI3rJoMNRmJqcM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712014199; a=rsa-sha256; cv=none; b=0OL2NRg9g8KGD4dTGqU5yUPBoIBy8H4P/5XeSbWp8Jkyk0sOVFRI14ffRKHlVjX9HPMqmK TAoYsx/Xevhwk9gMg2mc/4PtL9qNz8Z/2liGXPXV4FylfoNrqf/ya8C0zGTgXRHK/3TedF qf/qJ2ih6Jm9o3w20GYPeYMLFyGpeRs= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=1O0Be61P; spf=pass (imf16.hostedemail.com: domain of 3dkMLZgoKCGEISGNTFGSNMFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--jthoughton.bounces.google.com designates 209.85.222.73 as permitted sender) smtp.mailfrom=3dkMLZgoKCGEISGNTFGSNMFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-ua1-f73.google.com with SMTP id a1e0cc1a2514c-7e374d439daso645812241.1 for ; Mon, 01 Apr 2024 16:29:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1712014199; x=1712618999; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=PTb42yX5D5jCiyjo92iMcOe2nJRX5BH9KgDDQYg5TSM=; b=1O0Be61PLthPfu5zpKQ8tpRbPKg7KH0zbn1WOWEyNwiIH40uyHzEG1ybXSlkZtmnkW 8F1ErUJdSPWVSap+5XvJ5BuqXMIk5lKRS2MFQblwrPMOHp8VsSje5u5FzfYKaV17dsWy bk72M0zc3KJzFHdDnxLwdxZVV4EzL4tvE3PvHMZHnvB/BVoD8fVUDdShX99dmfiErHkI Fh12xGzZZ8DpF5SGBXGdLzjw5dwf9KyybjQ13M8HUwsc0o4spXMimDDosLzvJHT35Mm/ RNteNRQduKdWuC7hm4k/3gJ7e4H9SyjdSEfy+5vQTllQxuhaUbTZSYvBdr/ZRykUigXc BKYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712014199; x=1712618999; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=PTb42yX5D5jCiyjo92iMcOe2nJRX5BH9KgDDQYg5TSM=; b=c9AdJrfABY2wGi6s68J6lf0txYrxT3ExY8EVZr3a/eX96URyX9yPBdYE69GMpvDV1O bEWBEmRyvnjBKGtWTDnxvRbQkTNmJ1h6GeeP3oWT82WgCAJi4xWyvURuqxJNolix/NyX uovPqmfgnyJDsVdbfNherWXv8anrlGOE7KMZ3V9tAIH7XJi4LhLxlfjYIovVZFW2lCWE hFsYrJfONBrzggEbET8c2bfvUaA5UcZ78ZUJXxHDphamECDMuc3bFbBF8ozj1lbC5xtF SyWAbOHfAFU5FdXUHmy5ZfvlV0n+YXqzpwht/kM/SYSiGxlGas0JoZ9xFqGmOazth/sK +pZg== X-Forwarded-Encrypted: i=1; AJvYcCVn/o/Q0c4ZADnNwPZlUKxjDRsEGJS7Ay7PepUFqrwJ/B5NeURcjovz+PdHF5E8HE40CiIZEi54R1WSe0n0YhmyR/8= X-Gm-Message-State: AOJu0Yx4G7IZfaCBkzxa5HjiSiGG3h54YIU0n7P9faQQ5T39lK/7sqRk Z5CdiNNahJdiRg7cEeX5bhvtaQBtJuo0Xfd3XgJj7Yi1P92Hiz/NAjTZnibdZPDU/kkpca2aocJ NdBnT3MyFJyKqgzPADA== X-Google-Smtp-Source: AGHT+IGmJlmVbaC/lfrLJ5ZQ93VOyi/dOsu4Z25niMdu8RAczUBaed8myhkzYvDdfTTf9baVVIW2FHfRFR9HE1rY X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a05:6102:2d0d:b0:470:5ca5:3e95 with SMTP id ih13-20020a0561022d0d00b004705ca53e95mr1009542vsb.2.1712014198891; Mon, 01 Apr 2024 16:29:58 -0700 (PDT) Date: Mon, 1 Apr 2024 23:29:42 +0000 In-Reply-To: <20240401232946.1837665-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240401232946.1837665-1-jthoughton@google.com> X-Mailer: git-send-email 2.44.0.478.gd926399ef9-goog Message-ID: <20240401232946.1837665-4-jthoughton@google.com> Subject: [PATCH v3 3/7] KVM: Add basic bitmap support into kvm_mmu_notifier_test/clear_young From: James Houghton To: Andrew Morton , Paolo Bonzini Cc: Yu Zhao , David Matlack , Marc Zyngier , Oliver Upton , Sean Christopherson , Jonathan Corbet , James Morse , Suzuki K Poulose , Zenghui Yu , Catalin Marinas , Will Deacon , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers , Shaoqin Huang , Gavin Shan , Ricardo Koller , Raghavendra Rao Ananta , Ryan Roberts , David Rientjes , Axel Rasmussen , linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, James Houghton X-Stat-Signature: p5imjcoin9mkumooikoxwxdg8y1paeg7 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: DDAC618000C X-Rspam-User: X-HE-Tag: 1712014199-315345 X-HE-Meta: U2FsdGVkX1+3uILlmS5BPE6Gwc4p4EyJSooxLx2HGPGQet6N6jbXHZN3VxoVuPT78RYYHLZ1uLtH6qo/dUYXXAVsVY5gYECFmWp2ejBSVKY64vJNB+j9N9CSnlqiksy/PLv83l97i/1PS38QcuRsazQjzqjEdAqbEsLya303/VtZUgAuajX6eqUfJ+Jp2RPK5HUVQxDCqBODXI44evfwj9W+taKe8zIucw4+8kNTd2GUSftzDS5q5LdSRGuUz/a04tEkblJBILJ9fTlNFgxEQABhx/NMY4R0lKGsI2OHgUt8wFmfJsSgVZomaMggUw5IdhEd0SZUWbgnqRTrQI9+FlTcfRmSKJEa/JXV6OcToMC2u072xm7tqmvHxRHSVf3OnBoi4CbhpM9VCiRLD2+Oiodae1PRST7tbkb+4Z8ZW9j7MahvEkf63vjbsHm3UgJifvIoZmdypLNwe/UeVTNa6uVodtCB9hL3ubO4czK/ywi3QmN+j7IbvK+4xGRvKax8hP0hHO5VNoyyXxK+iWIfaIVIOcLfiZR6FlhUEzgLr241fi7C+oxOTHl6twfemfyW8a2I9tGo45ipqVR6k03mTspdtxbQ13Opax3V4CAECo/Km/AjeaMmq+71FiKx6E/hqQ6rI3PrBvog7zs6vkkaso8X0TOfOEc1Ew2mDRwos1BxUEBcveZPXQ+Z+npXJuoSK/B8+WKc8o2f0NF4emyjB0oulUPb8lT5V2n7bvU1R1fFt+w/L5qxcAFlCYjB00ReXDw1B/kUcOQNLwm2XYee3+9uxFy9ATxpUjo/5yRJqHvZXz2AVJNbinIpcn3pa9a9fdgkA2axRGYchnC4HpVEpztXQBBA32OQr1aDbpt7nvNARa+JuuIRZ0V6db5NJ8X4wLKhV3gsVrRtMyi7F84y0hUkAkkfPiVxcqodMdBpHeVZbev0MePCH6a9aNgXcGOraKSr9L/KuHEVswUWFLv CML/2m4H JsIYDhIo/teqSPyZ6J2JR9ejObHm94yBASUKCJailsapAivpyFSLAMUdAofQGuam7IA038vp8JQs5w7cULuxN7UOVlcNVMrEUzSPMahL3FDpmkUVCodkDjmSeJAMYRzVXNzlr1VaeD/UBizDfPOmFJCxiEotEnd49AEnFN8WUQPmm4N5hQvmb+O5fbOFQvld+UAVZltNgPB6V6X7iTJQRk1op23m129gkIJ6NI9sqvmCJEKwpNZXq/YIa7RkUuBSSt/fDOWbKTwLWQO4XnxP8eNpr+jpXWR/IF2FDSvWVmBfJWxs1uE4+loy1z7duc4Xp/3eUtFkRbiOB4nHhRr3wB8pL2P6JVK/FmVe/PUNVohTd5VujoPzZx+3iApDFntPvVyVshvvRRG32PHMWRLN8LjeMpPMowCqZ9eTCJDaZzoaxCw38xbTyNkoGpFZmPhCtQv17xwNICNZZnExcFROidJez5gokFjwmP6agD9xmLJsRSEWikV6/9nzqSXk401ZfFhEP6cJEMf4XoGi7MbQHb+tPnueoMa++E8CB46/R3tK/jRb/jxkbD3CHYqQyngkytf9XI7MY970/0HRfiNrxEY2mB0y8hqJKmq2lYyXu3fmdZXqspmmZOw5QNW+n6URerzuiJ+1JUJOTewlXaSYSGb5LqsF3bN5+7Ym+7ahdc2UXsRgMQVoMdSrOFA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add kvm_arch_prepare_bitmap_age() for architectures to indiciate that they support bitmap-based aging in kvm_mmu_notifier_test_clear_young() and that they do not need KVM to grab the MMU lock for writing. This function allows architectures to do other locking or other preparatory work that it needs. If an architecture does not implement kvm_arch_prepare_bitmap_age() or is unable to do bitmap-based aging at runtime (and marks the bitmap as unreliable): 1. If a bitmap was provided, we inform the caller that the bitmap is unreliable (MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE). 2. If a bitmap was not provided, fall back to the old logic. Also add logic for architectures to easily use the provided bitmap if they are able. The expectation is that the architecture's implementation of kvm_gfn_test_age() will use kvm_gfn_record_young(), and kvm_gfn_age() will use kvm_gfn_should_age(). Suggested-by: Yu Zhao Signed-off-by: James Houghton --- include/linux/kvm_host.h | 60 ++++++++++++++++++++++++++ virt/kvm/kvm_main.c | 92 +++++++++++++++++++++++++++++----------- 2 files changed, 127 insertions(+), 25 deletions(-) diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 1800d03a06a9..5862fd7b5f9b 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -1992,6 +1992,26 @@ extern const struct _kvm_stats_desc kvm_vm_stats_desc[]; extern const struct kvm_stats_header kvm_vcpu_stats_header; extern const struct _kvm_stats_desc kvm_vcpu_stats_desc[]; +/* + * Architectures that support using bitmaps for kvm_age_gfn() and + * kvm_test_age_gfn should return true for kvm_arch_prepare_bitmap_age() + * and do any work they need to prepare. The subsequent walk will not + * automatically grab the KVM MMU lock, so some architectures may opt + * to grab it. + * + * If true is returned, a subsequent call to kvm_arch_finish_bitmap_age() is + * guaranteed. + */ +#ifndef kvm_arch_prepare_bitmap_age +static inline bool kvm_arch_prepare_bitmap_age(struct mmu_notifier *mn) +{ + return false; +} +#endif +#ifndef kvm_arch_finish_bitmap_age +static inline void kvm_arch_finish_bitmap_age(struct mmu_notifier *mn) {} +#endif + #ifdef CONFIG_KVM_GENERIC_MMU_NOTIFIER static inline struct kvm *mmu_notifier_to_kvm(struct mmu_notifier *mn) { @@ -2076,9 +2096,16 @@ static inline bool mmu_invalidate_retry_gfn_unsafe(struct kvm *kvm, return READ_ONCE(kvm->mmu_invalidate_seq) != mmu_seq; } +struct test_clear_young_metadata { + unsigned long *bitmap; + unsigned long bitmap_offset_end; + unsigned long end; + bool unreliable; +}; union kvm_mmu_notifier_arg { pte_t pte; unsigned long attributes; + struct test_clear_young_metadata *metadata; }; struct kvm_gfn_range { @@ -2087,11 +2114,44 @@ struct kvm_gfn_range { gfn_t end; union kvm_mmu_notifier_arg arg; bool may_block; + bool lockless; }; bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range); bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range); bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range); bool kvm_set_spte_gfn(struct kvm *kvm, struct kvm_gfn_range *range); + +static inline void kvm_age_set_unreliable(struct kvm_gfn_range *range) +{ + struct test_clear_young_metadata *args = range->arg.metadata; + + args->unreliable = true; +} +static inline unsigned long kvm_young_bitmap_offset(struct kvm_gfn_range *range, + gfn_t gfn) +{ + struct test_clear_young_metadata *args = range->arg.metadata; + + return hva_to_gfn_memslot(args->end - 1, range->slot) - gfn; +} +static inline void kvm_gfn_record_young(struct kvm_gfn_range *range, gfn_t gfn) +{ + struct test_clear_young_metadata *args = range->arg.metadata; + + WARN_ON_ONCE(gfn < range->start || gfn >= range->end); + if (args->bitmap) + __set_bit(kvm_young_bitmap_offset(range, gfn), args->bitmap); +} +static inline bool kvm_gfn_should_age(struct kvm_gfn_range *range, gfn_t gfn) +{ + struct test_clear_young_metadata *args = range->arg.metadata; + + WARN_ON_ONCE(gfn < range->start || gfn >= range->end); + if (args->bitmap) + return test_bit(kvm_young_bitmap_offset(range, gfn), + args->bitmap); + return true; +} #endif #ifdef CONFIG_HAVE_KVM_IRQ_ROUTING diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index d0545d88c802..7d80321e2ece 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -550,6 +550,7 @@ struct kvm_mmu_notifier_range { on_lock_fn_t on_lock; bool flush_on_ret; bool may_block; + bool lockless; }; /* @@ -598,6 +599,8 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm, struct kvm_memslots *slots; int i, idx; + BUILD_BUG_ON(sizeof(gfn_range.arg) != sizeof(gfn_range.arg.pte)); + if (WARN_ON_ONCE(range->end <= range->start)) return r; @@ -637,15 +640,18 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm, gfn_range.start = hva_to_gfn_memslot(hva_start, slot); gfn_range.end = hva_to_gfn_memslot(hva_end + PAGE_SIZE - 1, slot); gfn_range.slot = slot; + gfn_range.lockless = range->lockless; if (!r.found_memslot) { r.found_memslot = true; - KVM_MMU_LOCK(kvm); - if (!IS_KVM_NULL_FN(range->on_lock)) - range->on_lock(kvm); - - if (IS_KVM_NULL_FN(range->handler)) - break; + if (!range->lockless) { + KVM_MMU_LOCK(kvm); + if (!IS_KVM_NULL_FN(range->on_lock)) + range->on_lock(kvm); + + if (IS_KVM_NULL_FN(range->handler)) + break; + } } r.ret |= range->handler(kvm, &gfn_range); } @@ -654,7 +660,7 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm, if (range->flush_on_ret && r.ret) kvm_flush_remote_tlbs(kvm); - if (r.found_memslot) + if (r.found_memslot && !range->lockless) KVM_MMU_UNLOCK(kvm); srcu_read_unlock(&kvm->srcu, idx); @@ -682,19 +688,24 @@ static __always_inline int kvm_handle_hva_range(struct mmu_notifier *mn, return __kvm_handle_hva_range(kvm, &range).ret; } -static __always_inline int kvm_handle_hva_range_no_flush(struct mmu_notifier *mn, - unsigned long start, - unsigned long end, - gfn_handler_t handler) +static __always_inline int kvm_handle_hva_range_no_flush( + struct mmu_notifier *mn, + unsigned long start, + unsigned long end, + gfn_handler_t handler, + union kvm_mmu_notifier_arg arg, + bool lockless) { struct kvm *kvm = mmu_notifier_to_kvm(mn); const struct kvm_mmu_notifier_range range = { .start = start, .end = end, .handler = handler, + .arg = arg, .on_lock = (void *)kvm_null_fn, .flush_on_ret = false, .may_block = false, + .lockless = lockless, }; return __kvm_handle_hva_range(kvm, &range).ret; @@ -909,15 +920,36 @@ static int kvm_mmu_notifier_clear_flush_young(struct mmu_notifier *mn, kvm_age_gfn); } -static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn, - struct mm_struct *mm, - unsigned long start, - unsigned long end, - unsigned long *bitmap) +static int kvm_mmu_notifier_test_clear_young(struct mmu_notifier *mn, + struct mm_struct *mm, + unsigned long start, + unsigned long end, + unsigned long *bitmap, + bool clear) { - trace_kvm_age_hva(start, end); + if (kvm_arch_prepare_bitmap_age(mn)) { + struct test_clear_young_metadata args = { + .bitmap = bitmap, + .end = end, + .unreliable = false, + }; + union kvm_mmu_notifier_arg arg = { + .metadata = &args + }; + bool young; + + young = kvm_handle_hva_range_no_flush( + mn, start, end, + clear ? kvm_age_gfn : kvm_test_age_gfn, + arg, true); + + kvm_arch_finish_bitmap_age(mn); - /* We don't support bitmaps. Don't test or clear anything. */ + if (!args.unreliable) + return young ? MMU_NOTIFIER_YOUNG_FAST : 0; + } + + /* A bitmap was passed but the architecture doesn't support bitmaps */ if (bitmap) return MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE; @@ -934,7 +966,21 @@ static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn, * cadence. If we find this inaccurate, we might come up with a * more sophisticated heuristic later. */ - return kvm_handle_hva_range_no_flush(mn, start, end, kvm_age_gfn); + return kvm_handle_hva_range_no_flush( + mn, start, end, clear ? kvm_age_gfn : kvm_test_age_gfn, + KVM_MMU_NOTIFIER_NO_ARG, false); +} + +static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn, + struct mm_struct *mm, + unsigned long start, + unsigned long end, + unsigned long *bitmap) +{ + trace_kvm_age_hva(start, end); + + return kvm_mmu_notifier_test_clear_young(mn, mm, start, end, bitmap, + true); } static int kvm_mmu_notifier_test_young(struct mmu_notifier *mn, @@ -945,12 +991,8 @@ static int kvm_mmu_notifier_test_young(struct mmu_notifier *mn, { trace_kvm_test_age_hva(start, end); - /* We don't support bitmaps. Don't test or clear anything. */ - if (bitmap) - return MMU_NOTIFIER_YOUNG_BITMAP_UNRELIABLE; - - return kvm_handle_hva_range_no_flush(mn, start, end, - kvm_test_age_gfn); + return kvm_mmu_notifier_test_clear_young(mn, mm, start, end, bitmap, + false); } static void kvm_mmu_notifier_release(struct mmu_notifier *mn,