From patchwork Thu Jan 25 16:42:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13531349 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5B38DC47422 for ; Thu, 25 Jan 2024 16:45:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DF24B6B0071; Thu, 25 Jan 2024 11:45:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D7AB76B0081; Thu, 25 Jan 2024 11:45:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BF6416B0098; Thu, 25 Jan 2024 11:45:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id A7D756B0071 for ; Thu, 25 Jan 2024 11:45:54 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 5BE6F1404E9 for ; Thu, 25 Jan 2024 16:45:54 +0000 (UTC) X-FDA: 81718410228.11.6CB7555 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf15.hostedemail.com (Postfix) with ESMTP id 8FB6AA000F for ; Thu, 25 Jan 2024 16:45:52 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf15.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706201152; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KHvUGMYd0nZcyyUZw5YWOWKPzifS3EXkLNN8Z9a6idc=; b=rHkJn0rHFLbI8+hxl/ZAwszi66h/TRoU3kQHpjbxxc+yls/zu2zNq+yC/0kA+Homoup9Vq EXmDvVqZHb1Ujy6cLeupU83B3AUZdbVP69ODb7xoA9B+dlQEHAcnkogVmuDzVZ7+VA+5E/ swA9U+sSY7G2bwjcPfPBuVQkn11TVxI= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf15.hostedemail.com: domain of alexandru.elisei@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=alexandru.elisei@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706201152; a=rsa-sha256; cv=none; b=r2SVfXbi5YzJ9KQJVEemcjhZGAvkdgKIm1XJsj3DZRVd6cqC5dsIZaQ5aMbOSRdXS2mG3a abtV8wS/nnig5ZXMxJ8enbTOB0ofX3P3Jb0M4WNu/7DEVI0V8XDhl8JL/kxvXOiS+CyOXe x5arnGDcGG5fefbGHukeGs25xxMRy9Q= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 4F38316F8; Thu, 25 Jan 2024 08:46:36 -0800 (PST) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 5D8F93F5A1; Thu, 25 Jan 2024 08:45:46 -0800 (PST) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC v3 32/35] KVM: arm64: mte: Reserve tag storage for virtual machines with MTE Date: Thu, 25 Jan 2024 16:42:53 +0000 Message-Id: <20240125164256.4147-33-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240125164256.4147-1-alexandru.elisei@arm.com> References: <20240125164256.4147-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 8FB6AA000F X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 1a84gskmkyhouhormr6z1mhre1pnqk6j X-HE-Tag: 1706201152-723360 X-HE-Meta: U2FsdGVkX19e/aX6oxpjh3Bov+XsT1MtgqkRGfTTVwWcTEtdAdaKK4D/dCq2kWsiBdDVkjsxgBmIgAUnLb8dkSHCvO4u97q6kvnky/SEQbV6jNgvsdykqm9nLOGBP20YzD1emTrNVrmu6bt7m+MToHHWb3Mg/jdUQ7jiX5a//twqQEwJ/VzZsBu6QgCldXvRqEfiDmF7anZvjKps/DmKUrhG3Qpj7tda14s1Fs2TWmeJclL8yd1Pkrru5LcXayse9Tbudj7/dBIAMrLIshJxKbSebP2FX5vCAG2Oy557VuEFjog8yc2nzs0M/VBXR9KKm/7l3ewQZXxDBSKxJ+xwxwX7UVeUsTNamP+hw5NUhFokQGfaTNWTIXJsn8/BOJ7V4RD9iaBMLeArgc+4XyTTpwYzW/9uC+awp0gNEk9Qb/szOJGcWqYBFGbv0BDvcg3PK9GfTespz5matuh+WN9ACBd7GAaAav2DaGl5b2Tz6shaDHuGRSY4RspMZRCsF8ioePY5g5oPLjQCMUkI0e4jsjsP2MyDuZnfrH6z9iFn7pLRQjgJKWmF4brWmMdfpl4GVuqm5qsTAlf9dwGnePF0bXkj6Sn7jQxh+VfWvsn2mQY9pOk4/QMnqwdmrXYvg1mbcoJVixyd5cBOFrwxNtd4X7wGUOfdJhNGi7k8z19jVoLjvrDCJrWrNYziaxYWRw7AshJxnrJL+3ipos3LQ0RRhL73gCFDnllhd0FfK+epPkSDYmmyocAYQ60oLQUky6jq82VCTkl3L+JP/iAAKq33QfXTJfCQ/wMvm0Nbh+w7W+DoflW2eYk/UEXHJRbR3XxcZ+sGhjJc13GZ2UHekcgZLktgTdgJnhKf9jb0VDKkKruaCVV5uctpQmFvQh50DCXOoQ0IXcIcyBS1J/cQS05oYYX+kJK1JLihX1d9h3zSpB/5E3JtAzHzy47ECQPR6APd/Ry69NALTondiph5cBn 3qkmm5Sg /rN2C/+yQ3M7DJquUo8QRFxuBmrI73MTTS3VCebtURw01Io6r4iTdqqULPYJsrEgC2nufKZTnkyPEqV7EedTkwnRKOkcAHJB5vdEslUwvckDh6MvXiGIfCrp3Lc4jQn0J5emKXWa3rVua0uNCQTxVlCAd+HrIv1uv9Go+By2FFAL/kvSZ/VqqiORxsfCbg+PLCq2u X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: KVM allows MTE enabled VMs to be created when the backing VMA does not have MTE enabled. As a result, pages allocated for the virtual machine's memory won't have tag storage reserved. Try to reserve tag storage the first time the page is accessed by the guest. This is similar to how pages mapped without tag storage in an MTE VMA are handled. Signed-off-by: Alexandru Elisei --- Changes since rfc v2: * New patch. arch/arm64/include/asm/mte_tag_storage.h | 10 ++++++ arch/arm64/include/asm/pgtable.h | 7 +++- arch/arm64/kvm/mmu.c | 43 ++++++++++++++++++++++++ arch/arm64/mm/fault.c | 2 +- 4 files changed, 60 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/mte_tag_storage.h b/arch/arm64/include/asm/mte_tag_storage.h index 40590a8c3748..32940ef7bcdf 100644 --- a/arch/arm64/include/asm/mte_tag_storage.h +++ b/arch/arm64/include/asm/mte_tag_storage.h @@ -34,6 +34,8 @@ void free_tag_storage(struct page *page, int order); bool page_tag_storage_reserved(struct page *page); bool page_is_tag_storage(struct page *page); +int replace_folio_with_tagged(struct folio *folio); + vm_fault_t handle_folio_missing_tag_storage(struct folio *folio, struct vm_fault *vmf, bool *map_pte); vm_fault_t mte_try_transfer_swap_tags(swp_entry_t entry, struct page *page); @@ -67,6 +69,14 @@ static inline bool page_tag_storage_reserved(struct page *page) { return true; } +static inline bool page_is_tag_storage(struct page *page) +{ + return false; +} +static inline int replace_folio_with_tagged(struct folio *folio) +{ + return -EINVAL; +} #endif /* CONFIG_ARM64_MTE_TAG_STORAGE */ #endif /* !__ASSEMBLY__ */ diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index d0473538c926..7f89606ad617 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1108,7 +1108,12 @@ static inline void arch_swap_restore(swp_entry_t entry, struct folio *folio) #define __HAVE_ARCH_FREE_PAGES_PREPARE static inline void arch_free_pages_prepare(struct page *page, int order) { - if (tag_storage_enabled() && page_mte_tagged(page)) + /* + * KVM can free a page after tag storage has been reserved and before is + * marked as tagged, hence use page_tag_storage_reserved() instead of + * page_mte_tagged() to check for tag storage. + */ + if (tag_storage_enabled() && page_tag_storage_reserved(page)) free_tag_storage(page, order); } diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index b7517c4a19c4..986a9544228d 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -1361,6 +1361,8 @@ static void sanitise_mte_tags(struct kvm *kvm, kvm_pfn_t pfn, if (!kvm_has_mte(kvm)) return; + WARN_ON_ONCE(tag_storage_enabled() && !page_tag_storage_reserved(pfn_to_page(pfn))); + for (i = 0; i < nr_pages; i++, page++) { if (try_page_mte_tagging(page)) { mte_clear_page_tags(page_address(page)); @@ -1374,6 +1376,39 @@ static bool kvm_vma_mte_allowed(struct vm_area_struct *vma) return vma->vm_flags & VM_MTE_ALLOWED; } +/* + * Called with an elevated reference on the pfn. If successful, the reference + * count is not changed. If it returns an error, the elevated reference is + * dropped. + */ +static int kvm_mte_reserve_tag_storage(kvm_pfn_t pfn) +{ + struct folio *folio; + int ret; + + folio = page_folio(pfn_to_page(pfn)); + + if (page_tag_storage_reserved(folio_page(folio, 0))) + return 0; + + if (page_is_tag_storage(folio_page(folio, 0))) + goto migrate; + + ret = reserve_tag_storage(folio_page(folio, 0), folio_order(folio), + GFP_HIGHUSER_MOVABLE); + if (!ret) + return 0; + +migrate: + replace_folio_with_tagged(folio); + /* + * If migration succeeds, the fault needs to be replayed because 'pfn' + * has been unmapped. If migration fails, KVM will try to reserve tag + * storage again by replaying the fault. + */ + return -EAGAIN; +} + static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, struct kvm_memory_slot *memslot, unsigned long hva, bool fault_is_perm) @@ -1488,6 +1523,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, pfn = __gfn_to_pfn_memslot(memslot, gfn, false, false, NULL, write_fault, &writable, NULL); + if (pfn == KVM_PFN_ERR_HWPOISON) { kvm_send_hwpoison_signal(hva, vma_shift); return 0; @@ -1518,6 +1554,13 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, if (exec_fault && device) return -ENOEXEC; + if (tag_storage_enabled() && !fault_is_perm && !device && + kvm_has_mte(kvm) && mte_allowed) { + ret = kvm_mte_reserve_tag_storage(pfn); + if (ret) + return ret == -EAGAIN ? 0 : ret; + } + read_lock(&kvm->mmu_lock); pgt = vcpu->arch.hw_mmu->pgt; if (mmu_invalidate_retry(kvm, mmu_seq)) diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 01450ab91a87..5c12232bdf0b 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -974,7 +974,7 @@ void tag_clear_highpage(struct page *page) * Called with an elevated reference on the folio. * Returns with the elevated reference dropped. */ -static int replace_folio_with_tagged(struct folio *folio) +int replace_folio_with_tagged(struct folio *folio) { struct migration_target_control mtc = { .nid = NUMA_NO_NODE,