From patchwork Mon Apr 1 23:29:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13613095 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1CEC4CD128A for ; Mon, 1 Apr 2024 23:30:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 099656B009B; Mon, 1 Apr 2024 19:30:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EE8A26B009C; Mon, 1 Apr 2024 19:30:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D16E46B009D; Mon, 1 Apr 2024 19:30:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id B1E176B009B for ; Mon, 1 Apr 2024 19:30:04 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 7424180579 for ; Mon, 1 Apr 2024 23:30:04 +0000 (UTC) X-FDA: 81962558328.08.18A44A8 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf03.hostedemail.com (Postfix) with ESMTP id AE4FC2000E for ; Mon, 1 Apr 2024 23:30:02 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="RjQ/DflK"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf03.hostedemail.com: domain of 3eUMLZgoKCGQLVJQWIJVQPIQQING.EQONKPWZ-OOMXCEM.QTI@flex--jthoughton.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3eUMLZgoKCGQLVJQWIJVQPIQQING.EQONKPWZ-OOMXCEM.QTI@flex--jthoughton.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712014202; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=CifQvULm5VcvU50LwCct+JTf2gCEq9urL1yip38iv3E=; b=w938jMF/OfAeIj7TZ5hM9s6OhsV0lEFMjCN2nH6ALjLU3PT+YW1z0dxXC5VSlnrN3Un2c0 pR5KytUelYk3Ouge1b2m83LrIv1QjsveOBm4Pbmj5KUVfId9go5fKq+X9fNDRvnSrFsrFu cFPtrm5NWk2pGnrjJa8k6QPy+ffAB/E= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="RjQ/DflK"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf03.hostedemail.com: domain of 3eUMLZgoKCGQLVJQWIJVQPIQQING.EQONKPWZ-OOMXCEM.QTI@flex--jthoughton.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3eUMLZgoKCGQLVJQWIJVQPIQQING.EQONKPWZ-OOMXCEM.QTI@flex--jthoughton.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712014202; a=rsa-sha256; cv=none; b=wH3FmuKGQv0KsUqVazJXX0UR61vRLOKhDTPwFl2ZtcS498gRBC6anJ9bR4oMrV7jZ/WtdN Yt/7zwC37huvl8c6KGYKgAyp7anjZnhy0J3PE1huB89icQVSaEF/9agWW6CN9P8eDVGo3/ Ng19D9RBtFk4uHaH9pSh+33XqYGSCxw= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-60cbba6f571so83605397b3.1 for ; Mon, 01 Apr 2024 16:30:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1712014202; x=1712619002; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=CifQvULm5VcvU50LwCct+JTf2gCEq9urL1yip38iv3E=; b=RjQ/DflKtlx8GhMXSGiREMtiBoBrT8lAJhOR/NhNIma7MdJGgr9W+/dh3cCKEVQCy5 y+S6aGxAFBmHRGxEhkrspS51h96uOq8jvV5OxKbJwMOdnOJqKdbW2r6eLcWSYvMisvLa Z8KHGh3lG3w0lIG9j/VJHDnZs/k1o2d9OOjL43pRUPR9RxDgEN40o7IbXVZaXlfmUOUj /C+L93e2Ax6jPpOpcXSKAuMvtoqARNgkZK9gF+2D+b6hsn3HavhOSZRTnyxuGjoYmRnD 1vv9O93+hLIjFXjPsSdj1FeBjj2zG/UDqGz4Eqe3cKPVDUVc1i45Bd6MtXROTUxdY0vB 2nvw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712014202; x=1712619002; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=CifQvULm5VcvU50LwCct+JTf2gCEq9urL1yip38iv3E=; b=dZnD2amQoE7iV1FRGKxzeVRtB5od7N/FoPPYzsqUBBM0kPumT76N1fe0c9DjZNqFY9 Otn2xezYm5N8+d79CjvQsnsoCX1Ik6PiIEFs6h6D+DaTgr6GLTwAZHLZQWEN3tTFZOVc dLN2bWHESq6+FlJHaPbH24WiyztdHuHtUmFElBVwCiZsGLk6tQdTuYvWfUgYiR/BF1oF k1tJcklDQz9JdRF85ltMifn4KILwa1t/B7CXFviMO0a/+4lG++F6K5oyX1rVXuciaxgY tPS0vi+CUKzBYWuyM+cumbOij9Ie9VS3McvTAXfshzD1t0722tbv0EKrHpHNqPk6DYll CR4Q== X-Forwarded-Encrypted: i=1; AJvYcCX5UCB7D8xnyhjboFuaN5ah7Se9SpiCYVLsvT2y5gNNnncvpcvqWZy4OvqelrsjsTk8u32hEA/7SE52YbfuWuEK3vQ= X-Gm-Message-State: AOJu0Yyrb3nTyZrMlgYZXMXnXeJTkrlmxzodTwMwGhsBP7nDsCz95zSX zxxxnZ+9TRpCpKYeJ0VYx8dLjPN9kFGvoHQCuPiYGH5kIPZVh9NUAF3eue8vkDDBsoYAylZhiOJ BLg2wDUQIscxrQCA6kg== X-Google-Smtp-Source: AGHT+IGQAk6u/Sk8y21e9rySBKwdkEDK33GoLWkr+tAsmcolw1O4jbXBJxOpzuSZWzDHw9bmaDUMZf63oZp0WmR7 X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a05:690c:650e:b0:615:165b:8dde with SMTP id hw14-20020a05690c650e00b00615165b8ddemr268650ywb.10.1712014201787; Mon, 01 Apr 2024 16:30:01 -0700 (PDT) Date: Mon, 1 Apr 2024 23:29:45 +0000 In-Reply-To: <20240401232946.1837665-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240401232946.1837665-1-jthoughton@google.com> X-Mailer: git-send-email 2.44.0.478.gd926399ef9-goog Message-ID: <20240401232946.1837665-7-jthoughton@google.com> Subject: [PATCH v3 6/7] KVM: arm64: Participate in bitmap-based PTE aging From: James Houghton To: Andrew Morton , Paolo Bonzini Cc: Yu Zhao , David Matlack , Marc Zyngier , Oliver Upton , Sean Christopherson , Jonathan Corbet , James Morse , Suzuki K Poulose , Zenghui Yu , Catalin Marinas , Will Deacon , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers , Shaoqin Huang , Gavin Shan , Ricardo Koller , Raghavendra Rao Ananta , Ryan Roberts , David Rientjes , Axel Rasmussen , linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, James Houghton X-Rspamd-Queue-Id: AE4FC2000E X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: qbmnff7h6umjz6w4mhpc5xrdnwyobwcb X-HE-Tag: 1712014202-405218 X-HE-Meta: U2FsdGVkX19LLbtJ3brrigVCCRdPiYTe51MVfpYzbrwn19gGVpeoBXQu2HmdzcBj2E/my66eP6bSPlkBIahZ1o50jK2B6ndqVxJPf+JikzKl8p5uSNwIHU2es8fYu1kiU+Kqar3G4WSXihGC6zt9U1rC3MR3KtFHQkYTQHxUlqlEn2uW2SCpa68ADb7CuwGplGWFT4wv+gXn06yXPZ7lyLg/2NB8B7+ltRIZDl+Z2iOjMje61XC7xkSCCfAT6YFXBvnzZyyictWLs8egiNeJQ6+H2it3YHWNUfgc5V2/xWONS/6Yj+W2rnuftjVI783NPwyfgUY+dvkXL65j52iGkm5t2Mi86gvXjnNejxCbbDmFZTvclTnhu9IWhC5PYQswD2NvSux/KK0WbhkdKAAc85R29Qrvz9cnSCSSTo/Y3KnqFq7vetyZRklMv2UypOtpgIioXgxph6tyy0j3vj7zGqY1SJxMVKsxs4SPltsuAE0KEn2GQ7JN9ulY7p1NZyTB5zk6CC2KKmDMp/VTjPDquj6yrLSfbocbdqo3bylfKcRqYNMQAkHtsyQks7g2jU2UyZGKwjkbtpj6CxCxNo5ylD2MhmiBMTpX/2PJxm1zIV0gBea467tiSmZ86uw4w55fPW+v5MYkmnez8QFH2TlCuxX2utS1+82QIdW0x0aI5PQLGoszrY4Um7+2N5QMf/54aeyZ+4q8oF3RJpQ3LNmuhqkKnNoc+J+bVni4UoYAPXAfQT/77D1R7RGnCzvZifStlVsaz42z7dxlkW9pZP/QDikc/Cz+wi83/qxV37m6o3VTDDL7RGohcQMzzoAQkUNlUvbCT/T3pJyIUfBja9uWHJ6kQZjem8qdkuYqNUZDmCDjJ5xo6pSQUvtWmRgMAy7iJoSKTuuvAYmcv2J5k/9rj8gcGWjK5PS8Z7HybMt6RYz7ZpDAMmKmbfBEEOaU444OUBH0SckaPXkKbd7U8A3 1CVriGGY xj35qzrP3bjgn3ETTL1hLzaBZ7RM6YLheCXqBviz0ZJv9Xz1ypNrje0DiTfmtVePNqijzcRsgi65CfUGC/5ZQagnhQoq8vXxQRd47VrABw6zHQEw6VuD4WNAvAqh78vCvGvw5OlYTUDQLuRoHQE/jeO8tNNbTK0IsvUsO2IHFUEi7TlqO2/LhPeFAVrRk+qu6suTBhA+joVHDAa3GC7X3zS/EMQ1nQDMAlbehq9fS/vVG78HRIfoBXA/4nC1NxLd4Rs1jrhzCw6Wx3Mx4ImXRtcnKG2RwIjh7MlGhtPLNnb5551DvNcWsW2IIw7LTE4wk7znQQNsu2bW313BMlLYlsdAZtMKvPv4mIiWPBn7fo4SNKmpxlK15qzdmCSd3KWtuQbzIFTnpuLS4dSCFHfXI1nDvKVDdOryp7Og9PzInHNU58O5ICVZNw2bDpckFMKB+1Zam4lw1c2lICoRVNCD3CYF2J6x24EDTbQbYAJQ/KPJ6/ZuYDxnaS0x+T2UEnSpDJWtw9oP5udam0fsLdTI3SO/MM1WTNQNoLQsl5/oTtFC4yUa7rHMfh/7cfOwkNyKFmu9+oOcauKJjujPr25+TS/ku4ZJf9DCE4ucz1ZmxJzYjmFnK8QTgrPuTPO0n+xuCi7n4QUGvuBEzeB4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Participate in bitmap-based aging while grabbing the KVM MMU lock for reading. Ideally we wouldn't need to grab this lock at all, but that would require a more intrustive and risky change. Also pass KVM_PGTABLE_WALK_SHARED, as this software walker is safe to run in parallel with other walkers. It is safe only to grab the KVM MMU lock for reading as the kvm_pgtable is destroyed while holding the lock for writing, and freeing of the page table pages is either done while holding the MMU lock for writing or after an RCU grace period. When mkold == false, record the young pages in the passed-in bitmap. When mkold == true, only age the pages that need aging according to the passed-in bitmap. Suggested-by: Yu Zhao Signed-off-by: James Houghton --- arch/arm64/include/asm/kvm_host.h | 5 +++++ arch/arm64/include/asm/kvm_pgtable.h | 4 +++- arch/arm64/kvm/hyp/pgtable.c | 21 ++++++++++++++------- arch/arm64/kvm/mmu.c | 23 +++++++++++++++++++++-- 4 files changed, 43 insertions(+), 10 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 9e8a496fb284..e503553cb356 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -1331,4 +1331,9 @@ bool kvm_arm_vcpu_stopped(struct kvm_vcpu *vcpu); (get_idreg_field((kvm), id, fld) >= expand_field_sign(id, fld, min) && \ get_idreg_field((kvm), id, fld) <= expand_field_sign(id, fld, max)) +#define kvm_arch_prepare_bitmap_age kvm_arch_prepare_bitmap_age +bool kvm_arch_prepare_bitmap_age(struct mmu_notifier *mn); +#define kvm_arch_finish_bitmap_age kvm_arch_finish_bitmap_age +void kvm_arch_finish_bitmap_age(struct mmu_notifier *mn); + #endif /* __ARM64_KVM_HOST_H__ */ diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/kvm_pgtable.h index 19278dfe7978..1976b4e26188 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -644,6 +644,7 @@ kvm_pte_t kvm_pgtable_stage2_mkyoung(struct kvm_pgtable *pgt, u64 addr); * @addr: Intermediate physical address to identify the page-table entry. * @size: Size of the address range to visit. * @mkold: True if the access flag should be cleared. + * @range: The kvm_gfn_range that is being used for the memslot walker. * * The offset of @addr within a page is ignored. * @@ -657,7 +658,8 @@ kvm_pte_t kvm_pgtable_stage2_mkyoung(struct kvm_pgtable *pgt, u64 addr); * Return: True if any of the visited PTEs had the access flag set. */ bool kvm_pgtable_stage2_test_clear_young(struct kvm_pgtable *pgt, u64 addr, - u64 size, bool mkold); + u64 size, bool mkold, + struct kvm_gfn_range *range); /** * kvm_pgtable_stage2_relax_perms() - Relax the permissions enforced by a diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 3fae5830f8d2..e881d3595aca 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -1281,6 +1281,7 @@ kvm_pte_t kvm_pgtable_stage2_mkyoung(struct kvm_pgtable *pgt, u64 addr) } struct stage2_age_data { + struct kvm_gfn_range *range; bool mkold; bool young; }; @@ -1290,20 +1291,24 @@ static int stage2_age_walker(const struct kvm_pgtable_visit_ctx *ctx, { kvm_pte_t new = ctx->old & ~KVM_PTE_LEAF_ATTR_LO_S2_AF; struct stage2_age_data *data = ctx->arg; + gfn_t gfn = ctx->addr / PAGE_SIZE; if (!kvm_pte_valid(ctx->old) || new == ctx->old) return 0; data->young = true; + /* - * stage2_age_walker() is always called while holding the MMU lock for - * write, so this will always succeed. Nonetheless, this deliberately - * follows the race detection pattern of the other stage-2 walkers in - * case the locking mechanics of the MMU notifiers is ever changed. + * stage2_age_walker() may not be holding the MMU lock for write, so + * follow the race detection pattern of the other stage-2 walkers. */ - if (data->mkold && !stage2_try_set_pte(ctx, new)) - return -EAGAIN; + if (data->mkold) { + if (kvm_gfn_should_age(data->range, gfn) && + !stage2_try_set_pte(ctx, new)) + return -EAGAIN; + } else + kvm_gfn_record_young(data->range, gfn); /* * "But where's the TLBI?!", you scream. @@ -1315,10 +1320,12 @@ static int stage2_age_walker(const struct kvm_pgtable_visit_ctx *ctx, } bool kvm_pgtable_stage2_test_clear_young(struct kvm_pgtable *pgt, u64 addr, - u64 size, bool mkold) + u64 size, bool mkold, + struct kvm_gfn_range *range) { struct stage2_age_data data = { .mkold = mkold, + .range = range, }; struct kvm_pgtable_walker walker = { .cb = stage2_age_walker, diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 18680771cdb0..104cc23e9bb3 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -1802,6 +1802,25 @@ bool kvm_set_spte_gfn(struct kvm *kvm, struct kvm_gfn_range *range) return false; } +bool kvm_arch_prepare_bitmap_age(struct mmu_notifier *mn) +{ + struct kvm *kvm = mmu_notifier_to_kvm(mn); + + /* + * We need to hold the MMU lock for reading to prevent page tables + * from being freed underneath us. + */ + read_lock(&kvm->mmu_lock); + return true; +} + +void kvm_arch_finish_bitmap_age(struct mmu_notifier *mn) +{ + struct kvm *kvm = mmu_notifier_to_kvm(mn); + + read_unlock(&kvm->mmu_lock); +} + bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { u64 size = (range->end - range->start) << PAGE_SHIFT; @@ -1811,7 +1830,7 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) return kvm_pgtable_stage2_test_clear_young(kvm->arch.mmu.pgt, range->start << PAGE_SHIFT, - size, true); + size, true, range); } bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) @@ -1823,7 +1842,7 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) return kvm_pgtable_stage2_test_clear_young(kvm->arch.mmu.pgt, range->start << PAGE_SHIFT, - size, false); + size, false, range); } phys_addr_t kvm_mmu_get_httbr(void)