From patchwork Mon Mar 25 18:51:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 13602764 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E639FC54E58 for ; Mon, 25 Mar 2024 18:52:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Badp/H7pIzrEP2Yb/w4CeIdS/fRL3JgRTaZmqkjinBo=; b=RFQZdIOCysgy+p 4LBOwAYKSPSbQPrhNNLtPbUnqvIFFCEw9n+00phbhfSmhX4pN+FCslAja2FhkmH1KxHMVfDUebbif 06PY9FhFWA+w9urfRASK8a7p3ZeszYIWOtyZFlweTPgg6zko7VwXPTqTTAfPm+0zcf5b2JEy1vsXm aS5Z2k8juJ/nwlMek+LgkfpzK7AvcDvTJG1CSoAhynKa99bloeIqw1FjdxkKbuwhC2oZDDW3EeQlA zi9UfCOkDOpPBUg0bvOCLcv3QXqx3nRM0D4I2CGsJjgSRICUrbeG++ZaKx8CUaNwdmol6rPk4EGpy bv9yVi6MCjP6IAgjqFnA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1ropQi-00000001RuG-3Ypd; Mon, 25 Mar 2024 18:52:20 +0000 Received: from sin.source.kernel.org ([145.40.73.55]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1ropQa-00000001RqQ-1fo7 for linux-arm-kernel@lists.infradead.org; Mon, 25 Mar 2024 18:52:14 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id 1138ACE1C89; Mon, 25 Mar 2024 18:52:10 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 99C6BC43390; Mon, 25 Mar 2024 18:52:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1711392729; bh=pp/mGSQctVF+cnNA7rY+yLh0nt0v3ISjCZuBkz0b7/c=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Kfp1r3RNey85n3/rXGeFdsweLpXnsDwBT8P+kuoPDamv87azNwUfyPm8oT54Gu6ud NRr3V11e2WCQERYxb9vJ4fie+wIN95H+Gid589cRG3qclGvejQ82l0Q86YczpakUJV jCUfCsjL2gXTlbY0Qv2k5xfiwzvlBrJY72VQQuqrxeQ15Hbr/EiPfxGoY5D+ToLSej A6DptsQSaDy51gxGhwbmjJeayilvAG8/UYTM3bE+4PJDn2A4BOucf6k0w4Z5JT+3Zg QLPaSDjiAeQWDygI3e/yPrkw7PhER2nhKccFmYuYxu+lAFBQo1Gbzppa5XNU5UlfJJ limZC/M0lwlOA== From: Will Deacon To: kvmarm@lists.linux.dev Cc: linux-arm-kernel@lists.infradead.org, Will Deacon , Catalin Marinas , Gavin Shan , Marc Zyngier , Mostafa Saleh , Oliver Upton , Quentin Perret , Raghavendra Rao Ananta , Ryan Roberts , Shaoqin Huang , Suzuki K Poulose , Zenghui Yu Subject: [PATCH 1/3] KVM: arm64: Don't defer TLB invalidation when zapping table entries Date: Mon, 25 Mar 2024 18:51:56 +0000 Message-Id: <20240325185158.8565-2-will@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20240325185158.8565-1-will@kernel.org> References: <20240325185158.8565-1-will@kernel.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240325_115212_695838_28983D7A X-CRM114-Status: GOOD ( 13.40 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Commit 7657ea920c54 ("KVM: arm64: Use TLBI range-based instructions for unmap") introduced deferred TLB invalidation for the stage-2 page-table so that range-based invalidation can be used for the accumulated addresses. This works fine if the structure of the page-tables remains unchanged, but if entire tables are zapped and subsequently freed then we transiently leave the hardware page-table walker with a reference to freed memory thanks to the translation walk caches. For example, stage2_unmap_walker() will free page-table pages: if (childp) mm_ops->put_page(childp); and issue the TLB invalidation later in kvm_pgtable_stage2_unmap(): if (stage2_unmap_defer_tlb_flush(pgt)) /* Perform the deferred TLB invalidations */ kvm_tlb_flush_vmid_range(pgt->mmu, addr, size); For now, take the conservative approach and invalidate the TLB eagerly when we clear a table entry. Cc: Raghavendra Rao Ananta Cc: Shaoqin Huang Cc: Marc Zyngier Cc: Oliver Upton Signed-off-by: Will Deacon --- arch/arm64/kvm/hyp/pgtable.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 3fae5830f8d2..de0b667ba296 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -896,9 +896,11 @@ static void stage2_unmap_put_pte(const struct kvm_pgtable_visit_ctx *ctx, if (kvm_pte_valid(ctx->old)) { kvm_clear_pte(ctx->ptep); - if (!stage2_unmap_defer_tlb_flush(pgt)) + if (!stage2_unmap_defer_tlb_flush(pgt) || + kvm_pte_table(ctx->old, ctx->level)) { kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, ctx->addr, ctx->level); + } } mm_ops->put_page(ctx->ptep); From patchwork Mon Mar 25 18:51:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 13602763 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DE645C54E64 for ; Mon, 25 Mar 2024 18:52:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=guw71TxnbIcXcQL2p0cguWE4AhRVUl2m10pxaZtjFd8=; b=YP1mQ/0iFICl0m 11xH96zFhsLsvCMiAAkJre4lcySwUQsqKYVi9PfU+eS8wqCSNPT6BYGNNeWgW7D9VB6Q4B2O4t3Pn Trw+66REJy7YvOh4yywRuHi+H8f0MiEI4gQTvJuhdeJW5K4cHsoz1QPmTRgxij0d7Li8hklyZoxDt JcpwzjHC6fLQMDiABOtghFqTeXqBlltOiTxXWTfc69qopgqwfVoraiDe7Jf4avJaeNJRcMbth8Clt 0HvUnUH1ILdr/b5kCCPgInS0gxhQosl4HtzSy6/ZpAogOdJwBjCe3jCBRfqLcfeJwb39Q53qFFlH0 +9lygSkwPGPUuXRkZl1Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1ropQj-00000001Ruc-3OlK; Mon, 25 Mar 2024 18:52:22 +0000 Received: from sin.source.kernel.org ([2604:1380:40e1:4800::1]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1ropQc-00000001Rra-3dnl for linux-arm-kernel@lists.infradead.org; Mon, 25 Mar 2024 18:52:16 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id 24562CE1AB9; Mon, 25 Mar 2024 18:52:13 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C67E3C433C7; Mon, 25 Mar 2024 18:52:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1711392732; bh=bXN/HevD7CPAdyg/mpq/H+7HLHjel9WBtmGPHdrhUQY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=iqkXUcEqtXtdLyi6K+9/BQsjj4e5kgjUopHt/DtwiYzGrlNzSLd6ydKyOAsF3XNQh 28vLSImXPQUJihOMxFLRUXi3anhq6aG0oB18h2tVn8K/vH/+OMh+jYAyh1WtUWQoGo vm72biZGh5ILWtD4RfILoKyMmNnZlv6JG7XpLHC0J4d0/CT2psTgt/7G+W64Ob1obc W1LG9WUIQmN5EVr+Am9B6ozT8scdoSie+tZ9QnPPDlN1h+U80pGTypfvUgqHAJw2P1 Q3UGJIgW2I9fJ4uvfcKo698NSDS835GfFBLVn6VJr9d58mnxc0ehYcmP1UXxJ+APu1 8Cp+DiMItY3Gg== From: Will Deacon To: kvmarm@lists.linux.dev Cc: linux-arm-kernel@lists.infradead.org, Will Deacon , Catalin Marinas , Gavin Shan , Marc Zyngier , Mostafa Saleh , Oliver Upton , Quentin Perret , Raghavendra Rao Ananta , Ryan Roberts , Shaoqin Huang , Suzuki K Poulose , Zenghui Yu Subject: [PATCH 2/3] KVM: arm64: Don't pass a TLBI level hint when zapping table entries Date: Mon, 25 Mar 2024 18:51:57 +0000 Message-Id: <20240325185158.8565-3-will@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20240325185158.8565-1-will@kernel.org> References: <20240325185158.8565-1-will@kernel.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240325_115215_200859_B06C7527 X-CRM114-Status: GOOD ( 10.69 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The TLBI level hints are for leaf entries only, so take care not to pass them incorrectly after clearing a table entry. Cc: Gavin Shan Cc: Marc Zyngier Cc: Quentin Perret Fixes: 82bb02445de5 ("KVM: arm64: Implement kvm_pgtable_hyp_unmap() at EL2") Fixes: 6d9d2115c480 ("KVM: arm64: Add support for stage-2 map()/unmap() in generic page-table") Signed-off-by: Will Deacon --- arch/arm64/kvm/hyp/pgtable.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index de0b667ba296..a40dafc43bb6 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -528,7 +528,7 @@ static int hyp_unmap_walker(const struct kvm_pgtable_visit_ctx *ctx, kvm_clear_pte(ctx->ptep); dsb(ishst); - __tlbi_level(vae2is, __TLBI_VADDR(ctx->addr, 0), ctx->level); + __tlbi_level(vae2is, __TLBI_VADDR(ctx->addr, 0), TLBI_TTL_UNKNOWN); } else { if (ctx->end - ctx->addr < granule) return -EINVAL; @@ -896,10 +896,12 @@ static void stage2_unmap_put_pte(const struct kvm_pgtable_visit_ctx *ctx, if (kvm_pte_valid(ctx->old)) { kvm_clear_pte(ctx->ptep); - if (!stage2_unmap_defer_tlb_flush(pgt) || - kvm_pte_table(ctx->old, ctx->level)) { - kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, - ctx->addr, ctx->level); + if (kvm_pte_table(ctx->old, ctx->level)) { + kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, ctx->addr, + TLBI_TTL_UNKNOWN); + } else if (!stage2_unmap_defer_tlb_flush(pgt)) { + kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, ctx->addr, + ctx->level); } } From patchwork Mon Mar 25 18:51:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 13602765 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A385BCD11DC for ; Mon, 25 Mar 2024 18:52:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Wpjgn+OrxT3r/PoYz/OkjOpQe9r32q74zbsgN/8XaTA=; b=JYW3hwK0Ly1o9Q clQSa6aQLUbsNLj0Udbh84e6opvayDkE0ZdpnGwFIpqA6qoxIZW8y9k6m5LjQafXU0diA95cpNaoP bK/5uysYxlD0hCL6oqygZbJp4uv5gwUwjz39HilU1tVHj8llYP9XwusQ20Pm/MHtZwQ8uzJmT4Ic9 PhXCGNfx8SLYaMRelqr3uZitBx3VYvHB/egoDPgrjVDtEWJKHZbv7Tq6w6ujqoAW1CJDlxBAzfA32 d1LOB67dfvltRAHub+C9+kq0wPOVDK4xqKHPUlQRD465jba3xu8EUJ+AfqI3lw7m5AjpjmEIBsCsd x3w4P5D8jAnICyt+m75g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1ropQl-00000001Rvb-3c40; Mon, 25 Mar 2024 18:52:23 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1ropQe-00000001Rss-2mzY for linux-arm-kernel@lists.infradead.org; Mon, 25 Mar 2024 18:52:18 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id CDD1361155; Mon, 25 Mar 2024 18:52:15 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id DAA95C433F1; Mon, 25 Mar 2024 18:52:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1711392735; bh=5VC9JxVG/MnzPe32qgOZJvC9+h/6mt0fn+04mmR6EA0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=WRR+3X41/frNKQpo6GkyBimMxAofL3xe8b6d8Gkd1GzgbE1ZcKJKHWtZhrPwAPm1D vlk6iVLTeRzZPStF6jiWZOk7aX7o1mmLuclHZjWII5F4fImPb54vSiWGTv4a8OsML1 iemPkxJJ3cvhehYXVsOA8C1yEZSTp8/YP4JjlsiPYxO8HZWeFAYf9x1ewoqmCT056I DkzsJhdysXzYTu+6XRsU+qlXGiIqmergAfW2m8av7xTg7qj65tS02BtV7BoiiBX357 t2vXubitUsJGEhqPlXTN4oC6E/buIg+NrBPKxMTgrHE1yJiQmruuLnvlRm9qtufck/ dCMFKx+OKpLhQ== From: Will Deacon To: kvmarm@lists.linux.dev Cc: linux-arm-kernel@lists.infradead.org, Will Deacon , Catalin Marinas , Gavin Shan , Marc Zyngier , Mostafa Saleh , Oliver Upton , Quentin Perret , Raghavendra Rao Ananta , Ryan Roberts , Shaoqin Huang , Suzuki K Poulose , Zenghui Yu Subject: [PATCH 3/3] KVM: arm64: Use TLBI_TTL_UNKNOWN in __kvm_tlb_flush_vmid_range() Date: Mon, 25 Mar 2024 18:51:58 +0000 Message-Id: <20240325185158.8565-4-will@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20240325185158.8565-1-will@kernel.org> References: <20240325185158.8565-1-will@kernel.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240325_115217_007841_2D5D7828 X-CRM114-Status: GOOD ( 11.83 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Commit c910f2b65518 ("arm64/mm: Update tlb invalidation routines for FEAT_LPA2") updated the __tlbi_level() macro to take the target level as an argument, with TLBI_TTL_UNKNOWN (rather than 0) indicating that the caller cannot provide level information. Unfortunately, the two implementations of __kvm_tlb_flush_vmid_range() were not updated and so now ask for an level 0 invalidation if FEAT_LPA2 is implemented. Fix the problem by passing TLBI_TTL_UNKNOWN instead of 0 as the level argument to __flush_s2_tlb_range_op() in __kvm_tlb_flush_vmid_range(). Cc: Ryan Roberts Cc: Catalin Marinas Cc: Oliver Upton Cc: Marc Zyngier Fixes: c910f2b65518 ("arm64/mm: Update tlb invalidation routines for FEAT_LPA2") Signed-off-by: Will Deacon Reviewed-by: Ryan Roberts --- arch/arm64/kvm/hyp/nvhe/tlb.c | 3 ++- arch/arm64/kvm/hyp/vhe/tlb.c | 3 ++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/arch/arm64/kvm/hyp/nvhe/tlb.c b/arch/arm64/kvm/hyp/nvhe/tlb.c index a60fb13e2192..2fc68da4036d 100644 --- a/arch/arm64/kvm/hyp/nvhe/tlb.c +++ b/arch/arm64/kvm/hyp/nvhe/tlb.c @@ -154,7 +154,8 @@ void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu, /* Switch to requested VMID */ __tlb_switch_to_guest(mmu, &cxt, false); - __flush_s2_tlb_range_op(ipas2e1is, start, pages, stride, 0); + __flush_s2_tlb_range_op(ipas2e1is, start, pages, stride, + TLBI_TTL_UNKNOWN); dsb(ish); __tlbi(vmalle1is); diff --git a/arch/arm64/kvm/hyp/vhe/tlb.c b/arch/arm64/kvm/hyp/vhe/tlb.c index b32e2940df7d..1a60b95381e8 100644 --- a/arch/arm64/kvm/hyp/vhe/tlb.c +++ b/arch/arm64/kvm/hyp/vhe/tlb.c @@ -171,7 +171,8 @@ void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu, /* Switch to requested VMID */ __tlb_switch_to_guest(mmu, &cxt); - __flush_s2_tlb_range_op(ipas2e1is, start, pages, stride, 0); + __flush_s2_tlb_range_op(ipas2e1is, start, pages, stride, + TLBI_TTL_UNKNOWN); dsb(ish); __tlbi(vmalle1is);