From patchwork Fri May 19 00:52:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra Rao Ananta X-Patchwork-Id: 13247570 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7FAE5C7EE29 for ; Fri, 19 May 2023 00:53:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=/i61NqOmuRCc6t6ZuT2LcJJN9dpaRErvIZNQilBePpA=; b=FtiCHIzILW3uj2bOBJSvDXXWWz 5JDYmLHxSIeMGk/QwhFHUkKR6iEa9pPhPZ/gwlGY+I9c+Zzo80VrrFxx57ZsMEVh2M/B8dt+356SO miHDo4FbuifSLx0VmG8z282MbaW29RHQbQOJ3xhTVkiu1v9NtVBfN4UdyCDbaaUV7zJx/NmnP0M78 bQX474W1917tBSn2+B/JjOSpR0WSex3pO47nDkGm/7Q+LHKrjoI8kaDtHP1T7TNvgROZ6Tuc2BWuA 2GcQGSQ85lqUoQyi8k5027Lq29KC3ER7cKtVu/wqeOOnqAHrpyN8m1XAUC+vizPpxPLgoFGtygjit +1CWV85Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1pzoMK-00EbMJ-0s; Fri, 19 May 2023 00:52:40 +0000 Received: from mail-io1-xd49.google.com ([2607:f8b0:4864:20::d49]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pzoMH-00EbL5-2c for linux-arm-kernel@lists.infradead.org; Fri, 19 May 2023 00:52:39 +0000 Received: by mail-io1-xd49.google.com with SMTP id ca18e2360f4ac-76c7107b84bso208598139f.3 for ; Thu, 18 May 2023 17:52:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1684457556; x=1687049556; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=1BCTijk0qbakQetFWj3/kUngjgg7qU/t2LH2AytbwyA=; b=eE5te6zoFAK2wnSpxBZQ8/GvCVM/YU5q8XLGMd7hL2YzwpknB+5DitnUAkizbYbh1n IY8Bkrk8FDzrSE2EN9Tahdj4W/aVn5jI+L/enmtskNyQOEGmxcjB3GKrd5yq0n3O8xvF Dl+tkUsHET+v3dVcKShodU8sBuCVZR5cl4QhZ2Ow8/gr5Jxb7fafGSMf/DV80tJ1zxlL lsHpt4BVrtsw5TOlgwRbkkp1PGQhssx6eiqt+Iy+8S4Su3kHt5p/AcyZK1nXWVxsKqhL tJPjhJ21dseJrn4qx41zMbONsEifZviXGrzV1yfD3veI7kV0zyT79kgeTSev08o1M9lD +FDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684457556; x=1687049556; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=1BCTijk0qbakQetFWj3/kUngjgg7qU/t2LH2AytbwyA=; b=PEx6PGvprUq0qIhQcc4At+7hB4fO+0i6TCdc83UjHfacyOkNRNaUBI0ChGGzePDuYp 6FW7dBOpTRFGta3CgUgon9HSeNuSddz+rQoZx3u9tfc2IaJ0j8V7iAIApcM7ScuxF7vw tRcSbiYyYNfFohuKiwz5jblfEBDDwJYJanaScW9vN8sJjwscb6yAeqEmrxca90QvmAsZ f/tV4AbuBxEmk8Y20uidBGx2ZKcBDhkQg8scT7ljqzAcrT0ZguzDZ2CKcpzL0DaqxqEa YY/Ew5l/AVm74YHQUp10VbfBbgef7maIw2rU5Hm4dfzAhAf8fr9JQC1MwmooFuLIXb5V MWsw== X-Gm-Message-State: AC+VfDw5JhHULFznTMomHYwfsH8uU3H8sR7aNbmcnUT/ZV9VUtqCfQJO KSQWF49yPMX2Ir5hVgHgM6LA70CkB5xV X-Google-Smtp-Source: ACHHUZ581E6RpVMDY5203uPsCEyRhBUbBRGNjHjBc8BerIRAb1q/rKHdq3qeP58gIFJtWoY+BNvHyKVGxYvL X-Received: from rananta-linux.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:22b5]) (user=rananta job=sendgmr) by 2002:a02:95c3:0:b0:41a:9030:981 with SMTP id b61-20020a0295c3000000b0041a90300981mr4875jai.0.1684457556756; Thu, 18 May 2023 17:52:36 -0700 (PDT) Date: Fri, 19 May 2023 00:52:26 +0000 In-Reply-To: <20230519005231.3027912-1-rananta@google.com> Mime-Version: 1.0 References: <20230519005231.3027912-1-rananta@google.com> X-Mailer: git-send-email 2.40.1.698.g37aff9b760-goog Message-ID: <20230519005231.3027912-2-rananta@google.com> Subject: [PATCH v4 1/6] arm64: tlb: Refactor the core flush algorithm of __flush_tlb_range From: Raghavendra Rao Ananta To: Oliver Upton , Marc Zyngier , James Morse , Suzuki K Poulose Cc: Ricardo Koller , Paolo Bonzini , Jing Zhang , Colton Lewis , Raghavendra Rao Anata , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Catalin Marinas X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230518_175237_849461_DF76031B X-CRM114-Status: GOOD ( 18.34 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently, the core TLB flush functionality of __flush_tlb_range() hardcodes vae1is (and variants) for the flush operation. In the upcoming patches, the KVM code reuses this core algorithm with ipas2e1is for range based TLB invalidations based on the IPA. Hence, extract the core flush functionality of __flush_tlb_range() into its own macro that accepts an 'op' argument to pass any TLBI operation, such that other callers (KVM) can benefit. No functional changes intended. Signed-off-by: Raghavendra Rao Ananta Reviewed-by: Catalin Marinas --- arch/arm64/include/asm/tlbflush.h | 108 +++++++++++++++--------------- 1 file changed, 55 insertions(+), 53 deletions(-) diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h index 412a3b9a3c25d..4775378b6da1b 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -278,14 +278,61 @@ static inline void flush_tlb_page(struct vm_area_struct *vma, */ #define MAX_TLBI_OPS PTRS_PER_PTE +/* When the CPU does not support TLB range operations, flush the TLB + * entries one by one at the granularity of 'stride'. If the TLB + * range ops are supported, then: + * + * 1. If 'pages' is odd, flush the first page through non-range + * operations; + * + * 2. For remaining pages: the minimum range granularity is decided + * by 'scale', so multiple range TLBI operations may be required. + * Start from scale = 0, flush the corresponding number of pages + * ((num+1)*2^(5*scale+1) starting from 'addr'), then increase it + * until no pages left. + * + * Note that certain ranges can be represented by either num = 31 and + * scale or num = 0 and scale + 1. The loop below favours the latter + * since num is limited to 30 by the __TLBI_RANGE_NUM() macro. + */ +#define __flush_tlb_range_op(op, start, pages, stride, \ + asid, tlb_level, tlbi_user) do { \ + int num = 0; \ + int scale = 0; \ + unsigned long addr; \ + \ + while (pages > 0) { \ + if (!system_supports_tlb_range() || \ + pages % 2 == 1) { \ + addr = __TLBI_VADDR(start, asid); \ + __tlbi_level(op, addr, tlb_level); \ + if (tlbi_user) \ + __tlbi_user_level(op, addr, tlb_level); \ + start += stride; \ + pages -= stride >> PAGE_SHIFT; \ + continue; \ + } \ + \ + num = __TLBI_RANGE_NUM(pages, scale); \ + if (num >= 0) { \ + addr = __TLBI_VADDR_RANGE(start, asid, scale, \ + num, tlb_level); \ + __tlbi(r##op, addr); \ + if (tlbi_user) \ + __tlbi_user(r##op, addr); \ + start += __TLBI_RANGE_PAGES(num, scale) << PAGE_SHIFT; \ + pages -= __TLBI_RANGE_PAGES(num, scale); \ + } \ + scale++; \ + } \ +} while (0) + static inline void __flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end, unsigned long stride, bool last_level, int tlb_level) { - int num = 0; - int scale = 0; - unsigned long asid, addr, pages; + unsigned long asid, pages; start = round_down(start, stride); end = round_up(end, stride); @@ -307,56 +354,11 @@ static inline void __flush_tlb_range(struct vm_area_struct *vma, dsb(ishst); asid = ASID(vma->vm_mm); - /* - * When the CPU does not support TLB range operations, flush the TLB - * entries one by one at the granularity of 'stride'. If the TLB - * range ops are supported, then: - * - * 1. If 'pages' is odd, flush the first page through non-range - * operations; - * - * 2. For remaining pages: the minimum range granularity is decided - * by 'scale', so multiple range TLBI operations may be required. - * Start from scale = 0, flush the corresponding number of pages - * ((num+1)*2^(5*scale+1) starting from 'addr'), then increase it - * until no pages left. - * - * Note that certain ranges can be represented by either num = 31 and - * scale or num = 0 and scale + 1. The loop below favours the latter - * since num is limited to 30 by the __TLBI_RANGE_NUM() macro. - */ - while (pages > 0) { - if (!system_supports_tlb_range() || - pages % 2 == 1) { - addr = __TLBI_VADDR(start, asid); - if (last_level) { - __tlbi_level(vale1is, addr, tlb_level); - __tlbi_user_level(vale1is, addr, tlb_level); - } else { - __tlbi_level(vae1is, addr, tlb_level); - __tlbi_user_level(vae1is, addr, tlb_level); - } - start += stride; - pages -= stride >> PAGE_SHIFT; - continue; - } - - num = __TLBI_RANGE_NUM(pages, scale); - if (num >= 0) { - addr = __TLBI_VADDR_RANGE(start, asid, scale, - num, tlb_level); - if (last_level) { - __tlbi(rvale1is, addr); - __tlbi_user(rvale1is, addr); - } else { - __tlbi(rvae1is, addr); - __tlbi_user(rvae1is, addr); - } - start += __TLBI_RANGE_PAGES(num, scale) << PAGE_SHIFT; - pages -= __TLBI_RANGE_PAGES(num, scale); - } - scale++; - } + if (last_level) + __flush_tlb_range_op(vale1is, start, pages, stride, asid, tlb_level, true); + else + __flush_tlb_range_op(vae1is, start, pages, stride, asid, tlb_level, true); + dsb(ish); }