From patchwork Fri May 19 00:52:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra Rao Ananta X-Patchwork-Id: 13247570 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7FAE5C7EE29 for ; Fri, 19 May 2023 00:53:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=/i61NqOmuRCc6t6ZuT2LcJJN9dpaRErvIZNQilBePpA=; b=FtiCHIzILW3uj2bOBJSvDXXWWz 5JDYmLHxSIeMGk/QwhFHUkKR6iEa9pPhPZ/gwlGY+I9c+Zzo80VrrFxx57ZsMEVh2M/B8dt+356SO miHDo4FbuifSLx0VmG8z282MbaW29RHQbQOJ3xhTVkiu1v9NtVBfN4UdyCDbaaUV7zJx/NmnP0M78 bQX474W1917tBSn2+B/JjOSpR0WSex3pO47nDkGm/7Q+LHKrjoI8kaDtHP1T7TNvgROZ6Tuc2BWuA 2GcQGSQ85lqUoQyi8k5027Lq29KC3ER7cKtVu/wqeOOnqAHrpyN8m1XAUC+vizPpxPLgoFGtygjit +1CWV85Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1pzoMK-00EbMJ-0s; Fri, 19 May 2023 00:52:40 +0000 Received: from mail-io1-xd49.google.com ([2607:f8b0:4864:20::d49]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pzoMH-00EbL5-2c for linux-arm-kernel@lists.infradead.org; Fri, 19 May 2023 00:52:39 +0000 Received: by mail-io1-xd49.google.com with SMTP id ca18e2360f4ac-76c7107b84bso208598139f.3 for ; Thu, 18 May 2023 17:52:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1684457556; x=1687049556; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=1BCTijk0qbakQetFWj3/kUngjgg7qU/t2LH2AytbwyA=; b=eE5te6zoFAK2wnSpxBZQ8/GvCVM/YU5q8XLGMd7hL2YzwpknB+5DitnUAkizbYbh1n IY8Bkrk8FDzrSE2EN9Tahdj4W/aVn5jI+L/enmtskNyQOEGmxcjB3GKrd5yq0n3O8xvF Dl+tkUsHET+v3dVcKShodU8sBuCVZR5cl4QhZ2Ow8/gr5Jxb7fafGSMf/DV80tJ1zxlL lsHpt4BVrtsw5TOlgwRbkkp1PGQhssx6eiqt+Iy+8S4Su3kHt5p/AcyZK1nXWVxsKqhL tJPjhJ21dseJrn4qx41zMbONsEifZviXGrzV1yfD3veI7kV0zyT79kgeTSev08o1M9lD +FDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684457556; x=1687049556; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=1BCTijk0qbakQetFWj3/kUngjgg7qU/t2LH2AytbwyA=; b=PEx6PGvprUq0qIhQcc4At+7hB4fO+0i6TCdc83UjHfacyOkNRNaUBI0ChGGzePDuYp 6FW7dBOpTRFGta3CgUgon9HSeNuSddz+rQoZx3u9tfc2IaJ0j8V7iAIApcM7ScuxF7vw tRcSbiYyYNfFohuKiwz5jblfEBDDwJYJanaScW9vN8sJjwscb6yAeqEmrxca90QvmAsZ f/tV4AbuBxEmk8Y20uidBGx2ZKcBDhkQg8scT7ljqzAcrT0ZguzDZ2CKcpzL0DaqxqEa YY/Ew5l/AVm74YHQUp10VbfBbgef7maIw2rU5Hm4dfzAhAf8fr9JQC1MwmooFuLIXb5V MWsw== X-Gm-Message-State: AC+VfDw5JhHULFznTMomHYwfsH8uU3H8sR7aNbmcnUT/ZV9VUtqCfQJO KSQWF49yPMX2Ir5hVgHgM6LA70CkB5xV X-Google-Smtp-Source: ACHHUZ581E6RpVMDY5203uPsCEyRhBUbBRGNjHjBc8BerIRAb1q/rKHdq3qeP58gIFJtWoY+BNvHyKVGxYvL X-Received: from rananta-linux.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:22b5]) (user=rananta job=sendgmr) by 2002:a02:95c3:0:b0:41a:9030:981 with SMTP id b61-20020a0295c3000000b0041a90300981mr4875jai.0.1684457556756; Thu, 18 May 2023 17:52:36 -0700 (PDT) Date: Fri, 19 May 2023 00:52:26 +0000 In-Reply-To: <20230519005231.3027912-1-rananta@google.com> Mime-Version: 1.0 References: <20230519005231.3027912-1-rananta@google.com> X-Mailer: git-send-email 2.40.1.698.g37aff9b760-goog Message-ID: <20230519005231.3027912-2-rananta@google.com> Subject: [PATCH v4 1/6] arm64: tlb: Refactor the core flush algorithm of __flush_tlb_range From: Raghavendra Rao Ananta To: Oliver Upton , Marc Zyngier , James Morse , Suzuki K Poulose Cc: Ricardo Koller , Paolo Bonzini , Jing Zhang , Colton Lewis , Raghavendra Rao Anata , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Catalin Marinas X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230518_175237_849461_DF76031B X-CRM114-Status: GOOD ( 18.34 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently, the core TLB flush functionality of __flush_tlb_range() hardcodes vae1is (and variants) for the flush operation. In the upcoming patches, the KVM code reuses this core algorithm with ipas2e1is for range based TLB invalidations based on the IPA. Hence, extract the core flush functionality of __flush_tlb_range() into its own macro that accepts an 'op' argument to pass any TLBI operation, such that other callers (KVM) can benefit. No functional changes intended. Signed-off-by: Raghavendra Rao Ananta Reviewed-by: Catalin Marinas --- arch/arm64/include/asm/tlbflush.h | 108 +++++++++++++++--------------- 1 file changed, 55 insertions(+), 53 deletions(-) diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h index 412a3b9a3c25d..4775378b6da1b 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -278,14 +278,61 @@ static inline void flush_tlb_page(struct vm_area_struct *vma, */ #define MAX_TLBI_OPS PTRS_PER_PTE +/* When the CPU does not support TLB range operations, flush the TLB + * entries one by one at the granularity of 'stride'. If the TLB + * range ops are supported, then: + * + * 1. If 'pages' is odd, flush the first page through non-range + * operations; + * + * 2. For remaining pages: the minimum range granularity is decided + * by 'scale', so multiple range TLBI operations may be required. + * Start from scale = 0, flush the corresponding number of pages + * ((num+1)*2^(5*scale+1) starting from 'addr'), then increase it + * until no pages left. + * + * Note that certain ranges can be represented by either num = 31 and + * scale or num = 0 and scale + 1. The loop below favours the latter + * since num is limited to 30 by the __TLBI_RANGE_NUM() macro. + */ +#define __flush_tlb_range_op(op, start, pages, stride, \ + asid, tlb_level, tlbi_user) do { \ + int num = 0; \ + int scale = 0; \ + unsigned long addr; \ + \ + while (pages > 0) { \ + if (!system_supports_tlb_range() || \ + pages % 2 == 1) { \ + addr = __TLBI_VADDR(start, asid); \ + __tlbi_level(op, addr, tlb_level); \ + if (tlbi_user) \ + __tlbi_user_level(op, addr, tlb_level); \ + start += stride; \ + pages -= stride >> PAGE_SHIFT; \ + continue; \ + } \ + \ + num = __TLBI_RANGE_NUM(pages, scale); \ + if (num >= 0) { \ + addr = __TLBI_VADDR_RANGE(start, asid, scale, \ + num, tlb_level); \ + __tlbi(r##op, addr); \ + if (tlbi_user) \ + __tlbi_user(r##op, addr); \ + start += __TLBI_RANGE_PAGES(num, scale) << PAGE_SHIFT; \ + pages -= __TLBI_RANGE_PAGES(num, scale); \ + } \ + scale++; \ + } \ +} while (0) + static inline void __flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end, unsigned long stride, bool last_level, int tlb_level) { - int num = 0; - int scale = 0; - unsigned long asid, addr, pages; + unsigned long asid, pages; start = round_down(start, stride); end = round_up(end, stride); @@ -307,56 +354,11 @@ static inline void __flush_tlb_range(struct vm_area_struct *vma, dsb(ishst); asid = ASID(vma->vm_mm); - /* - * When the CPU does not support TLB range operations, flush the TLB - * entries one by one at the granularity of 'stride'. If the TLB - * range ops are supported, then: - * - * 1. If 'pages' is odd, flush the first page through non-range - * operations; - * - * 2. For remaining pages: the minimum range granularity is decided - * by 'scale', so multiple range TLBI operations may be required. - * Start from scale = 0, flush the corresponding number of pages - * ((num+1)*2^(5*scale+1) starting from 'addr'), then increase it - * until no pages left. - * - * Note that certain ranges can be represented by either num = 31 and - * scale or num = 0 and scale + 1. The loop below favours the latter - * since num is limited to 30 by the __TLBI_RANGE_NUM() macro. - */ - while (pages > 0) { - if (!system_supports_tlb_range() || - pages % 2 == 1) { - addr = __TLBI_VADDR(start, asid); - if (last_level) { - __tlbi_level(vale1is, addr, tlb_level); - __tlbi_user_level(vale1is, addr, tlb_level); - } else { - __tlbi_level(vae1is, addr, tlb_level); - __tlbi_user_level(vae1is, addr, tlb_level); - } - start += stride; - pages -= stride >> PAGE_SHIFT; - continue; - } - - num = __TLBI_RANGE_NUM(pages, scale); - if (num >= 0) { - addr = __TLBI_VADDR_RANGE(start, asid, scale, - num, tlb_level); - if (last_level) { - __tlbi(rvale1is, addr); - __tlbi_user(rvale1is, addr); - } else { - __tlbi(rvae1is, addr); - __tlbi_user(rvae1is, addr); - } - start += __TLBI_RANGE_PAGES(num, scale) << PAGE_SHIFT; - pages -= __TLBI_RANGE_PAGES(num, scale); - } - scale++; - } + if (last_level) + __flush_tlb_range_op(vale1is, start, pages, stride, asid, tlb_level, true); + else + __flush_tlb_range_op(vae1is, start, pages, stride, asid, tlb_level, true); + dsb(ish); } From patchwork Fri May 19 00:52:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra Rao Ananta X-Patchwork-Id: 13247567 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AA198C77B73 for ; Fri, 19 May 2023 00:53:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=IQHDMjXp5IcGDUTCq5b2lDt0+SjGpoYrwKQhtbCx1xg=; b=roM1Ap8V470Jdh9iqSplD51Td6 UTKj1bR8GQnoby5IFtE2q88S4MNp2gNAXr2+xVDNPS36LC8mVZfM3XDtvPhRPWhh8vCPE8FqcLe/A RcjdljTMa+NA+HU9mR/JTLZBRNWHHegUUNhGDDqYn8cqLi5LrkjjPpvopV0FPBn1wKfJhBRZ2QUtR U+dLGhR4jzhoXeoAvirnOqvgI9lM8SxAlQux4/FqGSBCSK6cJ1mHQbsNJv4accJbUq0HQsgiO/aQn uhxtjdcsgF99qhEms/9eosfrILoF/014gg5xZ4nAHbV6sqC5lWq6E/F4UQwnqx93zI3BjLO2MC4aL 6pY3C2zA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1pzoMR-00EbQd-0Z; Fri, 19 May 2023 00:52:47 +0000 Received: from mail-io1-xd49.google.com ([2607:f8b0:4864:20::d49]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pzoMI-00EbLb-2N for linux-arm-kernel@lists.infradead.org; Fri, 19 May 2023 00:52:40 +0000 Received: by mail-io1-xd49.google.com with SMTP id ca18e2360f4ac-76fe42b7f7dso421188539f.0 for ; Thu, 18 May 2023 17:52:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1684457557; x=1687049557; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=6Jo/RCBNZEVvtCkzM3auZwKFc9mCMN65uMrYmj8Cg6c=; b=kBespD8hLjQXTQczEpM+vtlVlvFRNvCvTwFkfOOMbDJNZGnbpkolporHMFfB8gqI+r 4Swj3TIK/pJudwZ4nwJn0CA9k9ZC4Ni7zhcNjYzPweBpisRgVJ5jEq6jsD1AiZlwV7de 0dQLS3rMgDzJGHCtnWKs3cO3VBGqa1/sOj8vMeZsBfDu2/CjK7AH5mweDyyi6XIBj7sr 8Z3OmXV4+eNON2F9Gu3pHJO2lT3w4v+vDaojHuDwiiZoIAFtnC3uxbWj3QCtyPMVM036 bp9nXnILknAgqio//mWIB3wZvVMy4jX0mqescCGh9GQZd77HNzTxW1BXEyPie8DgwGQN caRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684457557; x=1687049557; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6Jo/RCBNZEVvtCkzM3auZwKFc9mCMN65uMrYmj8Cg6c=; b=kr70dWKXgtLk9gYiRTFsq/+oiwGj49f6iRTUv+eJEe4psX2E0yFec/C7m9s05Nko3w 6eOhKbvzOl6b/dzPjc2pdK9VEPpuERddveFadlCwBljqxeWiRuRAjxlFQVlB6003B4U6 /qtY6fb2bboey6Emvl0n2UUT8izIcbzkytLp5g1HMz2++pxt3P+mk4wJ9LoZfbxi/1zy 905IpN6qQRNYmhiriXRdTedFyeOoUU02haH/t3r21k5VpdfgkxRrmrBLebYnCKJgF50S ByxQQQf3sVJpja4VyDq7Mq49XEbfVhWYzTlb14GW7ShFzb1WV7w6P+eSIVFnIb7/7De6 z1SA== X-Gm-Message-State: AC+VfDx54Qn8TNZdn10fnsD3L4d1L+7VRgp/rjcStPiTLerkSl8upYrz OKUd1Rc27DtXfbY6vyKoDlSqbTszDKL4 X-Google-Smtp-Source: ACHHUZ6jvrPcWHxA7WBN9olQ1vxAdHrwW+CSix/U34owWIIgzWv9ikGQKqOo589qtSQqIbhWXYTMGpi013xy X-Received: from rananta-linux.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:22b5]) (user=rananta job=sendgmr) by 2002:a5e:8e4b:0:b0:76c:6fd8:282 with SMTP id r11-20020a5e8e4b000000b0076c6fd80282mr87940ioo.2.1684457557713; Thu, 18 May 2023 17:52:37 -0700 (PDT) Date: Fri, 19 May 2023 00:52:27 +0000 In-Reply-To: <20230519005231.3027912-1-rananta@google.com> Mime-Version: 1.0 References: <20230519005231.3027912-1-rananta@google.com> X-Mailer: git-send-email 2.40.1.698.g37aff9b760-goog Message-ID: <20230519005231.3027912-3-rananta@google.com> Subject: [PATCH v4 2/6] KVM: arm64: Implement __kvm_tlb_flush_vmid_range() From: Raghavendra Rao Ananta To: Oliver Upton , Marc Zyngier , James Morse , Suzuki K Poulose Cc: Ricardo Koller , Paolo Bonzini , Jing Zhang , Colton Lewis , Raghavendra Rao Anata , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230518_175238_775142_2A250A87 X-CRM114-Status: GOOD ( 15.04 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Define __kvm_tlb_flush_vmid_range() (for VHE and nVHE) to flush a range of stage-2 page-tables using IPA in one go. If the system supports FEAT_TLBIRANGE, the following patches would conviniently replace global TLBI such as vmalls12e1is in the map, unmap, and dirty-logging paths with ripas2e1is instead. Signed-off-by: Raghavendra Rao Ananta --- arch/arm64/include/asm/kvm_asm.h | 3 +++ arch/arm64/kvm/hyp/nvhe/hyp-main.c | 11 +++++++++ arch/arm64/kvm/hyp/nvhe/tlb.c | 39 ++++++++++++++++++++++++++++++ arch/arm64/kvm/hyp/vhe/tlb.c | 35 +++++++++++++++++++++++++++ 4 files changed, 88 insertions(+) diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h index 43c3bc0f9544d..33352d9399e32 100644 --- a/arch/arm64/include/asm/kvm_asm.h +++ b/arch/arm64/include/asm/kvm_asm.h @@ -79,6 +79,7 @@ enum __kvm_host_smccc_func { __KVM_HOST_SMCCC_FUNC___pkvm_init_vm, __KVM_HOST_SMCCC_FUNC___pkvm_init_vcpu, __KVM_HOST_SMCCC_FUNC___pkvm_teardown_vm, + __KVM_HOST_SMCCC_FUNC___kvm_tlb_flush_vmid_range, }; #define DECLARE_KVM_VHE_SYM(sym) extern char sym[] @@ -225,6 +226,8 @@ extern void __kvm_flush_vm_context(void); extern void __kvm_flush_cpu_context(struct kvm_s2_mmu *mmu); extern void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa, int level); +extern void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu, + phys_addr_t start, phys_addr_t end); extern void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu); extern void __kvm_timer_set_cntvoff(u64 cntvoff); diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c index 728e01d4536b0..81d30737dc7c9 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c +++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c @@ -125,6 +125,16 @@ static void handle___kvm_tlb_flush_vmid_ipa(struct kvm_cpu_context *host_ctxt) __kvm_tlb_flush_vmid_ipa(kern_hyp_va(mmu), ipa, level); } +static void +handle___kvm_tlb_flush_vmid_range(struct kvm_cpu_context *host_ctxt) +{ + DECLARE_REG(struct kvm_s2_mmu *, mmu, host_ctxt, 1); + DECLARE_REG(phys_addr_t, start, host_ctxt, 2); + DECLARE_REG(phys_addr_t, end, host_ctxt, 3); + + __kvm_tlb_flush_vmid_range(kern_hyp_va(mmu), start, end); +} + static void handle___kvm_tlb_flush_vmid(struct kvm_cpu_context *host_ctxt) { DECLARE_REG(struct kvm_s2_mmu *, mmu, host_ctxt, 1); @@ -315,6 +325,7 @@ static const hcall_t host_hcall[] = { HANDLE_FUNC(__kvm_vcpu_run), HANDLE_FUNC(__kvm_flush_vm_context), HANDLE_FUNC(__kvm_tlb_flush_vmid_ipa), + HANDLE_FUNC(__kvm_tlb_flush_vmid_range), HANDLE_FUNC(__kvm_tlb_flush_vmid), HANDLE_FUNC(__kvm_flush_cpu_context), HANDLE_FUNC(__kvm_timer_set_cntvoff), diff --git a/arch/arm64/kvm/hyp/nvhe/tlb.c b/arch/arm64/kvm/hyp/nvhe/tlb.c index 978179133f4b9..d4ea549c4b5c4 100644 --- a/arch/arm64/kvm/hyp/nvhe/tlb.c +++ b/arch/arm64/kvm/hyp/nvhe/tlb.c @@ -130,6 +130,45 @@ void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, __tlb_switch_to_host(&cxt); } +void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu, + phys_addr_t start, phys_addr_t end) +{ + struct tlb_inv_context cxt; + unsigned long pages, stride; + + /* + * Since the range of addresses may not be mapped at + * the same level, assume the worst case as PAGE_SIZE + */ + stride = PAGE_SIZE; + start = round_down(start, stride); + end = round_up(end, stride); + pages = (end - start) >> PAGE_SHIFT; + + if (!system_supports_tlb_range() || pages >= MAX_TLBI_RANGE_PAGES) { + __kvm_tlb_flush_vmid(mmu); + return; + } + + dsb(ishst); + + /* Switch to requested VMID */ + __tlb_switch_to_guest(mmu, &cxt); + + __flush_tlb_range_op(ipas2e1is, start, pages, stride, 0, 0, false); + + dsb(ish); + __tlbi(vmalle1is); + dsb(ish); + isb(); + + /* See the comment below in __kvm_tlb_flush_vmid_ipa() */ + if (icache_is_vpipt()) + icache_inval_all_pou(); + + __tlb_switch_to_host(&cxt); +} + void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu) { struct tlb_inv_context cxt; diff --git a/arch/arm64/kvm/hyp/vhe/tlb.c b/arch/arm64/kvm/hyp/vhe/tlb.c index 24cef9b87f9e9..f34d6dd9e4674 100644 --- a/arch/arm64/kvm/hyp/vhe/tlb.c +++ b/arch/arm64/kvm/hyp/vhe/tlb.c @@ -111,6 +111,41 @@ void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, __tlb_switch_to_host(&cxt); } +void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu, + phys_addr_t start, phys_addr_t end) +{ + struct tlb_inv_context cxt; + unsigned long pages, stride; + + /* + * Since the range of addresses may not be mapped at + * the same level, assume the worst case as PAGE_SIZE + */ + stride = PAGE_SIZE; + start = round_down(start, stride); + end = round_up(end, stride); + pages = (end - start) >> PAGE_SHIFT; + + if (!system_supports_tlb_range() || pages >= MAX_TLBI_RANGE_PAGES) { + __kvm_tlb_flush_vmid(mmu); + return; + } + + dsb(ishst); + + /* Switch to requested VMID */ + __tlb_switch_to_guest(mmu, &cxt); + + __flush_tlb_range_op(ipas2e1is, start, pages, stride, 0, 0, false); + + dsb(ish); + __tlbi(vmalle1is); + dsb(ish); + isb(); + + __tlb_switch_to_host(&cxt); +} + void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu) { struct tlb_inv_context cxt; From patchwork Fri May 19 00:52:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra Rao Ananta X-Patchwork-Id: 13247569 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A5E90C7EE2E for ; Fri, 19 May 2023 00:53:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=9lYkkekw31VSWdJIIKvTIHgXVLVmY4ZhWBACKFDE/eA=; b=TcLhXjJ4zu2vtU/EahoCFP/DKr JyW5NwvtvHeizhlIvDfyttar25L+Cabqx7xUecE/PSPhTIA6odGdSS8qM/4MwDzrwB9AMO8yz92bH bALr1owWo1a16aB349f+UCdw7kym4GiKilC3x/xqN1IpUwLMV3+Hc7l8rXacPMQXL1T/fwUPKV7ye 3PlmsXk/kMF5L0Du3Nf6CxTaghnwiJjdoboKCmYkMMz/iTUhqcoe3YDLZGbe4OYYbGWIUosbvQqXd gvxGjvBfqjj/RC7Kw4fH6CIH3JLsGClrsAuJ9RDjp1KrYWNaOSIRe+OEWWl+tQg+y5dB1Vvyx1b/N 4Z6mTd4A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1pzoMR-00EbR3-2F; Fri, 19 May 2023 00:52:47 +0000 Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pzoMK-00EbM2-2p for linux-arm-kernel@lists.infradead.org; Fri, 19 May 2023 00:52:42 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-ba8337a5861so684242276.0 for ; Thu, 18 May 2023 17:52:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1684457559; x=1687049559; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=eI3+o1vQveSU3bkuomrpolxp+t8dV7nu1enlALoGo84=; b=a993HPTO+g13j6obSzomXT8KISL/8mFSfYyDK/WG+ODvYVrzRo67FTVKRPNz4OtgR8 wbVvJI1UMWLNfzwvZ6Pc+IJB58PSbJkfDiM7qjnlf7+dulbrrXb1G2a556mStetHAB6X JhHn1bNc4gU6kdv2mB7svIf9qBYfY7+waIVlwadN4raN8L/2tiFCRgVB+TPiKZLB7r/M 5U85nJno9Reys1JZsaXHeJhYDz/wgCGwk4E2MFHocZEBRMlHKx4oBrY4LsUH0GALzyEz 5Yv9sRc5Gbx9MggBYHqKoqG75yo3yYxMGrKKLnLWjFd6xpGajIigzjT6XfGU/JHAF/8O qoJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684457559; x=1687049559; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=eI3+o1vQveSU3bkuomrpolxp+t8dV7nu1enlALoGo84=; b=Hhy1P/kKeIfGXQqSZP1TmIM/wuWc4aitRcWjBTC+VIHbekVI92D794K+syrgLK4CSq gS8Eb+RhsMpGn0lw6VeWRrRCcpuK1XBEF7Fa8C6VGMimEYfVX2BfexIFYaYn7O5UEA14 ZTrvwBpEw18+/gKaDLmxgy1FShCd1FhhzGu9qPWqrFuiik+zOmOh2na7VLrCaUHlfU+z qrZMacPYbnwEFRKRjX8O0P2924RftbtHavInRFuDxoq7taqmRFTEek74JzT77oHMci2n vUhixKrak8rYdf4QAn0Mizj54WBI3tGF/LIruj1V5q6exitv1IkXzb6aARmIa5L3lKkz TYHw== X-Gm-Message-State: AC+VfDxDrlPcuEpfX9saz9j71agIctZP8xJryg5ZA8ywiLCPiovhIDx1 cHxToAjAlOrW7jSFUCkNI2Yp2GhMuGxI X-Google-Smtp-Source: ACHHUZ7MGOicfXqOEIRYhYQir8c9VFNkG60dMuEYJ6FiAi22aFdneLwrpZOUgMMNGFimpKhr++I/T6Q/ojCK X-Received: from rananta-linux.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:22b5]) (user=rananta job=sendgmr) by 2002:a05:6902:1343:b0:ba8:4d1c:dd04 with SMTP id g3-20020a056902134300b00ba84d1cdd04mr94689ybu.1.1684457558794; Thu, 18 May 2023 17:52:38 -0700 (PDT) Date: Fri, 19 May 2023 00:52:28 +0000 In-Reply-To: <20230519005231.3027912-1-rananta@google.com> Mime-Version: 1.0 References: <20230519005231.3027912-1-rananta@google.com> X-Mailer: git-send-email 2.40.1.698.g37aff9b760-goog Message-ID: <20230519005231.3027912-4-rananta@google.com> Subject: [PATCH v4 3/6] KVM: arm64: Implement kvm_arch_flush_remote_tlbs_range() From: Raghavendra Rao Ananta To: Oliver Upton , Marc Zyngier , James Morse , Suzuki K Poulose Cc: Ricardo Koller , Paolo Bonzini , Jing Zhang , Colton Lewis , Raghavendra Rao Anata , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230518_175240_918290_8751DBA7 X-CRM114-Status: GOOD ( 10.63 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Implement kvm_arch_flush_remote_tlbs_range() for arm64 to invalidate the given range in the TLB. Signed-off-by: Raghavendra Rao Ananta --- arch/arm64/include/asm/kvm_host.h | 3 +++ arch/arm64/kvm/hyp/nvhe/tlb.c | 4 +--- arch/arm64/kvm/mmu.c | 11 +++++++++++ 3 files changed, 15 insertions(+), 3 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 81ab41b84f436..343fb530eea9c 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -1081,6 +1081,9 @@ struct kvm *kvm_arch_alloc_vm(void); #define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS int kvm_arch_flush_remote_tlbs(struct kvm *kvm); +#define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS_RANGE +int kvm_arch_flush_remote_tlbs_range(struct kvm *kvm, gfn_t start_gfn, u64 pages); + static inline bool kvm_vm_is_protected(struct kvm *kvm) { return false; diff --git a/arch/arm64/kvm/hyp/nvhe/tlb.c b/arch/arm64/kvm/hyp/nvhe/tlb.c index d4ea549c4b5c4..d2c7c1bc6d441 100644 --- a/arch/arm64/kvm/hyp/nvhe/tlb.c +++ b/arch/arm64/kvm/hyp/nvhe/tlb.c @@ -150,10 +150,8 @@ void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu, return; } - dsb(ishst); - /* Switch to requested VMID */ - __tlb_switch_to_guest(mmu, &cxt); + __tlb_switch_to_guest(mmu, &cxt, false); __flush_tlb_range_op(ipas2e1is, start, pages, stride, 0, 0, false); diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index d0a0d3dca9316..e3673b4c10292 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -92,6 +92,17 @@ int kvm_arch_flush_remote_tlbs(struct kvm *kvm) return 0; } +int kvm_arch_flush_remote_tlbs_range(struct kvm *kvm, gfn_t start_gfn, u64 pages) +{ + phys_addr_t start, end; + + start = start_gfn << PAGE_SHIFT; + end = (start_gfn + pages) << PAGE_SHIFT; + + kvm_call_hyp(__kvm_tlb_flush_vmid_range, &kvm->arch.mmu, start, end); + return 0; +} + static bool kvm_is_device_pfn(unsigned long pfn) { return !pfn_is_map_memory(pfn); From patchwork Fri May 19 00:52:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra Rao Ananta X-Patchwork-Id: 13247566 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E675EC77B7D for ; Fri, 19 May 2023 00:53:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=F0OuMlRDlXPxy6/2h6QRpVWrvSZl2BnQPWvM0/zZWy8=; b=sZssT+rSksKcXLk9NYjXqJ84Qb e0V3H8ya0bK4JMsS65wkK4hmIt3aubS4WF01VCAkNJRchSiqROKpGwa31nwlzGbYP0MLI2GXWCRqE /bo3h5Tf/f5aPwG4dt3cFJhAIMOZb9N2hmDBcuKqbg3BTfPypmGS8WSlHOvbyQnSGAe1q+ioGYPn/ jvN2ETtk+PjzuBNZ0zmYW08g28S0X/8Jw732OHOIYFvXcyiMQq8ijk6Szlvi50rNxWfd6GKC1lmnR k3FbdWJP3bOYFCzdQVufrAda6FJoR3uwJaC4d857W/Ha8NcOwqL7PhQuSxb2ITd8y6yCXRv1tnPq/ SUslIdgg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1pzoMS-00EbRP-0e; Fri, 19 May 2023 00:52:48 +0000 Received: from mail-il1-x14a.google.com ([2607:f8b0:4864:20::14a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pzoML-00EbMI-1t for linux-arm-kernel@lists.infradead.org; Fri, 19 May 2023 00:52:42 +0000 Received: by mail-il1-x14a.google.com with SMTP id e9e14a558f8ab-3383a678225so38179565ab.3 for ; Thu, 18 May 2023 17:52:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1684457559; x=1687049559; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Ux8TuKlucx7+2ANrW96xfpSzC0srXjlkGkFfov8Qh8A=; b=nfSM+9p6k+4DETvto+y0GWvj0T9Pn8cMCjrch3oOD/Wt9bRMqYA9UGLB5sbBIFm3bE ErGeQr5UdDquGK62ZHTcCSsHQAXPCCLZVOkbzbU6VHi8oy1BBkXlDTx1rUSKGHyXEkcw CRrFCWWvlBNXVCva7myDBYAhGGf/U7gd9GAipG1KEYCz1LLOkJLWaSE8T5nm1cEdVlR8 5tP2tqRGprEouckvewWTYmoYRDgGUs1jXpJueLe9UdjlIwEd5LWNV3rcOeywTjR8qcTr 7fzQ54BFSSQK1VZGjDitJwXDp65Cwumn+1ym/giNrZSEg0XMB2EwauEgdfRfOpuh9nka y5xA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684457559; x=1687049559; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Ux8TuKlucx7+2ANrW96xfpSzC0srXjlkGkFfov8Qh8A=; b=Hs2fo8iAVzIuhvykW19mJCg8drJqpnhKobHNDk18UNVvQk2/mAPT3jF2GUqQBv/IqZ 475N3nIVHUAijK/bNTO/LAMNB+azuc8r8Qx423LhgixdKiWFbFXZDouYJ9yS0L39pWMe fYFrMLz49GhBzzXgoDdpuZChSlE6brDbhCvISzDcfNFdxzF9+l4wpZdN44JGSUT8alGv 2oJRNOF6w7Anl8POqaWtxDqGpkdhmidag/E6SKdqhGXlFdRD4q2qFwixqoYr+2jFZrG1 l76hKbJdNsF24JV9k+tHx4Zuq+fPcsnO+6w1nLT0xPj4Uz8eWC+K2n/HcJMt7Y+Cf1oY qtyA== X-Gm-Message-State: AC+VfDw+t3gpgjOVFr8u93MmhmtIDPKkxoLucqQJHfqvHGCSlSXx6NWd lHgO2lZ2WPJtTYn1I5gDC9bNKw5tKTxl X-Google-Smtp-Source: ACHHUZ4mrHRBUPTPC63HrBmyYMpoPQXoQJAtvWnxdjoUIHpL+YcJ3GzassZ0JGPJascAshL3vG7BYy3W5E7i X-Received: from rananta-linux.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:22b5]) (user=rananta job=sendgmr) by 2002:a05:6e02:786:b0:335:fee3:37c1 with SMTP id q6-20020a056e02078600b00335fee337c1mr28210ils.3.1684457559778; Thu, 18 May 2023 17:52:39 -0700 (PDT) Date: Fri, 19 May 2023 00:52:29 +0000 In-Reply-To: <20230519005231.3027912-1-rananta@google.com> Mime-Version: 1.0 References: <20230519005231.3027912-1-rananta@google.com> X-Mailer: git-send-email 2.40.1.698.g37aff9b760-goog Message-ID: <20230519005231.3027912-5-rananta@google.com> Subject: [PATCH v4 4/6] KVM: arm64: Flush only the memslot after write-protect From: Raghavendra Rao Ananta To: Oliver Upton , Marc Zyngier , James Morse , Suzuki K Poulose Cc: Ricardo Koller , Paolo Bonzini , Jing Zhang , Colton Lewis , Raghavendra Rao Anata , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230518_175241_624716_A3871E4C X-CRM114-Status: GOOD ( 11.27 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org After write-protecting the region, currently KVM invalidates the entire TLB entries using kvm_flush_remote_tlbs(). Instead, scope the invalidation only to the targeted memslot. If supported, the architecture would use the range-based TLBI instructions to flush the memslot or else fallback to flushing all of the TLBs. Signed-off-by: Raghavendra Rao Ananta --- arch/arm64/kvm/mmu.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index e3673b4c10292..2ea6eb4ea763e 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -996,7 +996,7 @@ static void kvm_mmu_wp_memory_region(struct kvm *kvm, int slot) write_lock(&kvm->mmu_lock); stage2_wp_range(&kvm->arch.mmu, start, end); write_unlock(&kvm->mmu_lock); - kvm_flush_remote_tlbs(kvm); + kvm_flush_remote_tlbs_memslot(kvm, memslot); } /** From patchwork Fri May 19 00:52:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra Rao Ananta X-Patchwork-Id: 13247571 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 37115C77B73 for ; Fri, 19 May 2023 00:53:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=UVz0btcj+8jYo6EwfyMsznlvLXf77KT7Pf1/JvcwOwk=; b=zDsOekMynOt/fnKRF8NbP8atj6 Pqx6sxRK6hiZGNZA+sKixMlZDDarYs3RLfbfQNymnMsmhvj+dOtO5vAbYFDU3WIcgsosd9qjmdZ9i u4o4lNjWkaH+vNw6QzZ8DEKsBFKH8P/NJ7QbWVRMMEBteF7TiLnbSoUoq5Gd60ShXZ7qQQfTtkgZf n/WSm8xFNauDey/0xzVxBQJKTbMV08eshOYM2DLPci5ChRyf43bvV5DhQ+8juV3EdAPGKUjPMwWm0 L+rDPQuXk31AKLrzcKkKyi81NWWT3e04oQvlYwSbFjkG9zVWMytWGbDtv224p+7MXB/pdxZt7Wa0m wqcPiN+A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1pzoMS-00EbRn-2I; Fri, 19 May 2023 00:52:48 +0000 Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pzoMM-00EbMv-21 for linux-arm-kernel@lists.infradead.org; Fri, 19 May 2023 00:52:43 +0000 Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-ba8c3186735so1376421276.3 for ; Thu, 18 May 2023 17:52:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1684457561; x=1687049561; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=e6DsOoIF+x/A7ZYSTdRHf2cD4dqAihxaeRbkm/Wf8QE=; b=5jParhefjh0HI3DzorDBwf9sb8RHRycHwmfDukeUSFn84zbi/9wRDfQaKoBx20Z2pS jgAJQ4ttTjCt4O4bkCtGffdz6NOopVPRPNoSjGutVL/MA59xkj/2i64IMZ4xsfb5JZvo kQknAlNduF12wnSsQxHbve4iQrVfw5KgPyNjRsYycjJCm6HjknBcGXCHSIYIS5n8fG/P xg/hoe/5+I5sWP+59TvQr8H21v4IS7jO4EfQnXFGK7qjua8eIK0OYqkQwb0pnsQqec5q 4KabcCQrLkUiQ1gkV268Wti7RamOnI6hXcNQ2d3KXg8+bwW6KOn9g17yKdwKwZ3kjqrY RESA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684457561; x=1687049561; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=e6DsOoIF+x/A7ZYSTdRHf2cD4dqAihxaeRbkm/Wf8QE=; b=aF+39Jaq9jEc7J5eNhRAq6Rm3QZATJb4jmsjWHNRgsMYsk3VJj53MHuFVnZuo/Ug8H PLxBuxCFFpG6SYH1L+Q8SpiCRLE+ubVvZYqLDUzgPR3xFWKpVxsi/6SRZgyhaUgjRQa8 lbevttPZpglot+YK5tNmTuEKptjLxx6QXW/3EOlZgMFyEuwXGTL10qVeeDHHwzFHQJpX uOCetbiHrfstteNzckxboL39kr3k7lvy/zuYoZQ3JomgmnqJhw2UdPcc1PC+rteSwEbs JbdJVPeSTfyE5Q8YvgDHBpQBc14ciixbN8R0xcd8BsWuSGZTXcyZJvyy+OEiASMdOk8l gbOw== X-Gm-Message-State: AC+VfDzI1I78ETE6g6sr6++ZqDaWjfoUISXmi+oj0XXrvY5LHlgGoxr/ sjOCgUi+BmZuEB9FjwceHjZllkfdDNIm X-Google-Smtp-Source: ACHHUZ4Lgs2jxcq+llYDlHCAR5udROfxgqvoQbhs56Qn9JKtSHbYquhrdkyDxuf1xI7e7oDzhrf6uwFOdXpl X-Received: from rananta-linux.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:22b5]) (user=rananta job=sendgmr) by 2002:a25:b906:0:b0:ba8:9096:df50 with SMTP id x6-20020a25b906000000b00ba89096df50mr26549ybj.9.1684457560955; Thu, 18 May 2023 17:52:40 -0700 (PDT) Date: Fri, 19 May 2023 00:52:30 +0000 In-Reply-To: <20230519005231.3027912-1-rananta@google.com> Mime-Version: 1.0 References: <20230519005231.3027912-1-rananta@google.com> X-Mailer: git-send-email 2.40.1.698.g37aff9b760-goog Message-ID: <20230519005231.3027912-6-rananta@google.com> Subject: [PATCH v4 5/6] KVM: arm64: Invalidate the table entries upon a range From: Raghavendra Rao Ananta To: Oliver Upton , Marc Zyngier , James Morse , Suzuki K Poulose Cc: Ricardo Koller , Paolo Bonzini , Jing Zhang , Colton Lewis , Raghavendra Rao Anata , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230518_175242_661082_409336F8 X-CRM114-Status: GOOD ( 11.92 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently, during the operations such as a hugepage collapse, KVM would flush the entire VM's context using 'vmalls12e1is' TLBI operation. Specifically, if the VM is faulting on many hugepages (say after dirty-logging), it creates a performance penalty for the guest whose pages have already been faulted earlier as they would have to refill their TLBs again. Instead, call __kvm_tlb_flush_vmid_range() for table entries. If the system supports it, only the required range will be flushed. Else, it'll fallback to the previous mechanism. Signed-off-by: Raghavendra Rao Ananta --- arch/arm64/kvm/hyp/pgtable.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 3d61bd3e591d2..b8f0dbd12f773 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -745,10 +745,13 @@ static bool stage2_try_break_pte(const struct kvm_pgtable_visit_ctx *ctx, * Perform the appropriate TLB invalidation based on the evicted pte * value (if any). */ - if (kvm_pte_table(ctx->old, ctx->level)) - kvm_call_hyp(__kvm_tlb_flush_vmid, mmu); - else if (kvm_pte_valid(ctx->old)) + if (kvm_pte_table(ctx->old, ctx->level)) { + u64 end = ctx->addr + kvm_granule_size(ctx->level); + + kvm_call_hyp(__kvm_tlb_flush_vmid_range, mmu, ctx->addr, end); + } else if (kvm_pte_valid(ctx->old)) { kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, ctx->addr, ctx->level); + } if (stage2_pte_is_counted(ctx->old)) mm_ops->put_page(ctx->ptep); From patchwork Fri May 19 00:52:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra Rao Ananta X-Patchwork-Id: 13247568 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1B35DC7EE2C for ; Fri, 19 May 2023 00:53:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=WA/NIqd1HjIopurgCldcVc0tLzNYl7IfjDvzKUUyxQ4=; b=GU8kPLMI5P19H1YQRU2y7zT0Vq ViwmFZFngDJCv9NGDEPn2mSwh8Ll+90pRyH9EHN0+9A9r62EK8dGUKpJcSE/3WjumrO/b/lcmgj/8 ioiYn+1l1GlwFJxGmw5s4KNmpTrvgR0kF+gh+bloR/2sXdvN/Dl0e0F+8CuzpHdrsBzlcYyoMInHX tz7MFwSEfFDbbp+rvI49p9clEMA/ViJ55oVubjiizWF4/J4lUNN8JegZkTYRIZm8uPaZ8EWoDu34w QLn3Ey3i6nP/L9rqhYZBfmvO6u5rYH9qX5SBy/51S9iFI3p313k7TT/Jm1GKSBih9A/4l9dQM5Gzw 5JC7GC2Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1pzoMT-00EbSL-0w; Fri, 19 May 2023 00:52:49 +0000 Received: from mail-io1-xd49.google.com ([2607:f8b0:4864:20::d49]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pzoMM-00EbNa-35 for linux-arm-kernel@lists.infradead.org; Fri, 19 May 2023 00:52:44 +0000 Received: by mail-io1-xd49.google.com with SMTP id ca18e2360f4ac-7603d830533so26392439f.1 for ; Thu, 18 May 2023 17:52:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1684457562; x=1687049562; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=rGsmPoc/T4Ss3kfVhJrmlF6LuTv5aUCcul/iCKRGGfw=; b=xW05sinAfw8C44WTP/9vyg4AsrI69Imj2ZmwYQLJZF8kEHkqon5WhJrYk6DNXK7Gw6 ORpvYUQiCd6GWx/VkoJW+RFSdfhvlRh0N9mjXQWCD6GhGofxw22goCURFVI2zQtlWjI3 q40Px7tzzEXTuwbuR/ZiN0k40rk4wWlizlUEyM3BENmLsLioAlGtY649VgAWbXD8mNoT jqpj50br7Y7C26Wwzs7ZYrY5XUZtlZuQUEmP94MjpBLWtfJKC0Hj0gvsmYFqFZ6EU0/5 BTWcttwAOtqDV1LAtO7UFkbUjq6YDBOiaJ3QlnQPpYOq6lbHz3ZzDuYKfXJ/7nDGH2YH /vSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684457562; x=1687049562; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=rGsmPoc/T4Ss3kfVhJrmlF6LuTv5aUCcul/iCKRGGfw=; b=S4iaa6Vfw9tXoiSlZ27ZIpirynVXcvkhJB4Z5PH0PfCMvGkvYpygOoWAzuJAHlKUDA hlHcGpqAiWGlFkxNWytieyoZfGUuI1Ea7HSub0RIlQmEnqu5e3jkUZ0oBUk6IuW6PhMQ Hq0p5WJhhe7UcoyIpFqGtBgpZvTVdmlnwNbLZxrtJx1z4tXEGIrMXEyir02eWrkr6sV7 gYSNoM80dQ24+wXZ86rDNg/ryDFIF0cPxAcncplmvrTUx6kjvPi6+Ea4nQfJ59ehQw7N 00ckSFQShFDJUI+BKcUdPkzdV+7KFCc54SBmKTcs50n2BdbhyTYPOJYMsrrENY6Wc7kg kksw== X-Gm-Message-State: AC+VfDxnE9d/r+vLHEwK24gGfpBXZCA0ICy9K4snfjINxoJALaYQXlB5 zv4OtPD2Mqx9B6cgZp3XH8Vbv+IlKw1g X-Google-Smtp-Source: ACHHUZ6TXO388gzfHLwp1jHRgCji0izRBrbXcyR7uAZvijYpyuzGsSLXba63Usu2McZd5g1WZCldx8ML+Rhn X-Received: from rananta-linux.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:22b5]) (user=rananta job=sendgmr) by 2002:a05:6602:2ac4:b0:763:b184:fe92 with SMTP id m4-20020a0566022ac400b00763b184fe92mr5060981iov.0.1684457562041; Thu, 18 May 2023 17:52:42 -0700 (PDT) Date: Fri, 19 May 2023 00:52:31 +0000 In-Reply-To: <20230519005231.3027912-1-rananta@google.com> Mime-Version: 1.0 References: <20230519005231.3027912-1-rananta@google.com> X-Mailer: git-send-email 2.40.1.698.g37aff9b760-goog Message-ID: <20230519005231.3027912-7-rananta@google.com> Subject: [PATCH v4 6/6] KVM: arm64: Use TLBI range-based intructions for unmap From: Raghavendra Rao Ananta To: Oliver Upton , Marc Zyngier , James Morse , Suzuki K Poulose Cc: Ricardo Koller , Paolo Bonzini , Jing Zhang , Colton Lewis , Raghavendra Rao Anata , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230518_175243_002197_1B12735F X-CRM114-Status: GOOD ( 16.44 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The current implementation of the stage-2 unmap walker traverses the given range and, as a part of break-before-make, performs TLB invalidations with a DSB for every PTE. A multitude of this combination could cause a performance bottleneck. Hence, if the system supports FEAT_TLBIRANGE, defer the TLB invalidations until the entire walk is finished, and then use range-based instructions to invalidate the TLBs in one go. Condition this upon S2FWB in order to avoid walking the page-table again to perform the CMOs after issuing the TLBI. Rename stage2_put_pte() to stage2_unmap_put_pte() as the function now serves the stage-2 unmap walker specifically, rather than acting generic. Signed-off-by: Raghavendra Rao Ananta --- arch/arm64/kvm/hyp/pgtable.c | 35 ++++++++++++++++++++++++++++++----- 1 file changed, 30 insertions(+), 5 deletions(-) diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index b8f0dbd12f773..5832ee3418fb0 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -771,16 +771,34 @@ static void stage2_make_pte(const struct kvm_pgtable_visit_ctx *ctx, kvm_pte_t n smp_store_release(ctx->ptep, new); } -static void stage2_put_pte(const struct kvm_pgtable_visit_ctx *ctx, struct kvm_s2_mmu *mmu, - struct kvm_pgtable_mm_ops *mm_ops) +static bool stage2_unmap_defer_tlb_flush(struct kvm_pgtable *pgt) { + /* + * If FEAT_TLBIRANGE is implemented, defer the individial PTE + * TLB invalidations until the entire walk is finished, and + * then use the range-based TLBI instructions to do the + * invalidations. Condition this upon S2FWB in order to avoid + * a page-table walk again to perform the CMOs after TLBI. + */ + return system_supports_tlb_range() && stage2_has_fwb(pgt); +} + +static void stage2_unmap_put_pte(const struct kvm_pgtable_visit_ctx *ctx, + struct kvm_s2_mmu *mmu, + struct kvm_pgtable_mm_ops *mm_ops) +{ + struct kvm_pgtable *pgt = ctx->arg; + /* * Clear the existing PTE, and perform break-before-make with * TLB maintenance if it was valid. */ if (kvm_pte_valid(ctx->old)) { kvm_clear_pte(ctx->ptep); - kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, ctx->addr, ctx->level); + + if (!stage2_unmap_defer_tlb_flush(pgt)) + kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, + ctx->addr, ctx->level); } mm_ops->put_page(ctx->ptep); @@ -1015,7 +1033,7 @@ static int stage2_unmap_walker(const struct kvm_pgtable_visit_ctx *ctx, * block entry and rely on the remaining portions being faulted * back lazily. */ - stage2_put_pte(ctx, mmu, mm_ops); + stage2_unmap_put_pte(ctx, mmu, mm_ops); if (need_flush && mm_ops->dcache_clean_inval_poc) mm_ops->dcache_clean_inval_poc(kvm_pte_follow(ctx->old, mm_ops), @@ -1029,13 +1047,20 @@ static int stage2_unmap_walker(const struct kvm_pgtable_visit_ctx *ctx, int kvm_pgtable_stage2_unmap(struct kvm_pgtable *pgt, u64 addr, u64 size) { + int ret; struct kvm_pgtable_walker walker = { .cb = stage2_unmap_walker, .arg = pgt, .flags = KVM_PGTABLE_WALK_LEAF | KVM_PGTABLE_WALK_TABLE_POST, }; - return kvm_pgtable_walk(pgt, addr, size, &walker); + ret = kvm_pgtable_walk(pgt, addr, size, &walker); + if (stage2_unmap_defer_tlb_flush(pgt)) + /* Perform the deferred TLB invalidations */ + kvm_call_hyp(__kvm_tlb_flush_vmid_range, pgt->mmu, + addr, addr + size); + + return ret; } struct stage2_attr_data {