From patchwork Tue Jun 6 19:28:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra Rao Ananta X-Patchwork-Id: 13269673 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 51854C83003 for ; Tue, 6 Jun 2023 19:29:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=Yzu5pwjhTUQbI/bVMQCq4iqSj0imd7Y5SE3pqdUcbSE=; b=ULiEaHq3Dhk0ebRxkPGQ2efay/ jBs21mcnLsWSaNiBS9hhfKo7OGP8Iau47I+TynNRenZX9FjtPGPANT64VINdoBWX3uL4Mw+lQXDvK bnWO4ul8pfxusLWmwOlcoEp9IE3Tef5wFNvXQAtv7eJ+CgunHJLzYNVl5IgqXusSQBhfDi3draI5v RATmkFCGutzn2Dkht0eWjqdyRB3mP4xhq7JPtAdq1qw23ghGENbiITcViismE1g3wKsBc6slnxaLC swjz5coNU0maNT2Zuq8YG6Byn7sV3fUpVtT6KTrFq/s5g2MOsoL0etxPSCjwGnK2t1kFAl8Bk0BaM jZGnVZvQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1q6cMh-002xe7-1F; Tue, 06 Jun 2023 19:29:11 +0000 Received: from mail-io1-xd49.google.com ([2607:f8b0:4864:20::d49]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1q6cMa-002xYD-2a for linux-arm-kernel@lists.infradead.org; Tue, 06 Jun 2023 19:29:07 +0000 Received: by mail-io1-xd49.google.com with SMTP id ca18e2360f4ac-7776b76cc59so562908339f.2 for ; Tue, 06 Jun 2023 12:29:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686079743; x=1688671743; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=WFJK44bg6mTFZNh04T55o8cqW/XBy3bKKtAhRnNGyNM=; b=d70o+mmg/FlsnhdyTBCBIyehP5Om6Pi4yYAx8ru+yyhjlJj4dQB5ICb64pXkWgEIks zKDhwL4Vu2j6CIbfJTHkJeB6o4e6AE+P95KDOo1cPZtFK41yeFXHDkV479VAHm1k95Da EfdPWWpizZJo4KoZQvbTPMHWFMWOfhDMYax306cTGzHURmuxfF3y3EuxDxZ4M6zzm4yE ACvMInyOE9fEjfp3nb/+yUAmMe85XXR/7ims7FSGsxkN/9pZwB60OoHuGbWNXFiUq5gq r/gsGyEq4rnUhc1sYYx1YmTufkR+WCSQN8FBaCZ3SeFPQqhBnquezv46nGxcZw4V6dOc rczg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686079743; x=1688671743; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=WFJK44bg6mTFZNh04T55o8cqW/XBy3bKKtAhRnNGyNM=; b=DSf+oBaLuC6j4Weh7QS16l9s5VGd/un6nYm5qpE0hWpV6Izb/RDdygEqh1Ldg7bzqe 2DGpKd1NzzF2HpuszkJxk17kZKAHHJPJVaw1ySZRp+37qYEd9nU8aTSMoTXEsxKJwOsQ i3z1+3j5JhPOnF3fpmVi6QOiGx6ZC5Sr3mPN17UToHyEj+2/yPFuKAf2bMF7/nfm3/DW V3epnob9dPH+CHkkwdEaB+iRzMhlg9MhT0oyLmj2O5n9jlrsnpllPywie3jMVk2INqE1 nfGC9LeUVFlG8I/uF2VGpeywJwGOf8Yw0kSS2UO+TYtZpN44JduOnaMTeQvJT3cogymo OVJg== X-Gm-Message-State: AC+VfDxscFi0OKB3pL2S+GeVWedmGOnT3+9L275hNfDM6s1uHxHg3rHc vNOmHvn9tsypfIdoSSjG/7OEiOo86pbC X-Google-Smtp-Source: ACHHUZ62Hug8P99peZzbJUWEuxmHtWhbRFuSd2AYuFgkpN4/A/feJCg3inpG9Gm+vzkRn3a+VIu54mcG95EX X-Received: from rananta-linux.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:22b5]) (user=rananta job=sendgmr) by 2002:a6b:6102:0:b0:774:8f36:bb8e with SMTP id v2-20020a6b6102000000b007748f36bb8emr1571026iob.2.1686079743400; Tue, 06 Jun 2023 12:29:03 -0700 (PDT) Date: Tue, 6 Jun 2023 19:28:52 +0000 In-Reply-To: <20230606192858.3600174-1-rananta@google.com> Mime-Version: 1.0 References: <20230606192858.3600174-1-rananta@google.com> X-Mailer: git-send-email 2.41.0.rc0.172.g3f132b7071-goog Message-ID: <20230606192858.3600174-2-rananta@google.com> Subject: [PATCH v5 1/7] arm64: tlb: Refactor the core flush algorithm of __flush_tlb_range From: Raghavendra Rao Ananta To: Oliver Upton , Marc Zyngier , James Morse , Suzuki K Poulose Cc: Paolo Bonzini , Jing Zhang , Colton Lewis , Raghavendra Rao Anata , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Catalin Marinas X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230606_122904_843752_CA919147 X-CRM114-Status: GOOD ( 18.30 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently, the core TLB flush functionality of __flush_tlb_range() hardcodes vae1is (and variants) for the flush operation. In the upcoming patches, the KVM code reuses this core algorithm with ipas2e1is for range based TLB invalidations based on the IPA. Hence, extract the core flush functionality of __flush_tlb_range() into its own macro that accepts an 'op' argument to pass any TLBI operation, such that other callers (KVM) can benefit. No functional changes intended. Signed-off-by: Raghavendra Rao Ananta Reviewed-by: Catalin Marinas --- arch/arm64/include/asm/tlbflush.h | 108 +++++++++++++++--------------- 1 file changed, 55 insertions(+), 53 deletions(-) diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h index 412a3b9a3c25d..4775378b6da1b 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -278,14 +278,61 @@ static inline void flush_tlb_page(struct vm_area_struct *vma, */ #define MAX_TLBI_OPS PTRS_PER_PTE +/* When the CPU does not support TLB range operations, flush the TLB + * entries one by one at the granularity of 'stride'. If the TLB + * range ops are supported, then: + * + * 1. If 'pages' is odd, flush the first page through non-range + * operations; + * + * 2. For remaining pages: the minimum range granularity is decided + * by 'scale', so multiple range TLBI operations may be required. + * Start from scale = 0, flush the corresponding number of pages + * ((num+1)*2^(5*scale+1) starting from 'addr'), then increase it + * until no pages left. + * + * Note that certain ranges can be represented by either num = 31 and + * scale or num = 0 and scale + 1. The loop below favours the latter + * since num is limited to 30 by the __TLBI_RANGE_NUM() macro. + */ +#define __flush_tlb_range_op(op, start, pages, stride, \ + asid, tlb_level, tlbi_user) do { \ + int num = 0; \ + int scale = 0; \ + unsigned long addr; \ + \ + while (pages > 0) { \ + if (!system_supports_tlb_range() || \ + pages % 2 == 1) { \ + addr = __TLBI_VADDR(start, asid); \ + __tlbi_level(op, addr, tlb_level); \ + if (tlbi_user) \ + __tlbi_user_level(op, addr, tlb_level); \ + start += stride; \ + pages -= stride >> PAGE_SHIFT; \ + continue; \ + } \ + \ + num = __TLBI_RANGE_NUM(pages, scale); \ + if (num >= 0) { \ + addr = __TLBI_VADDR_RANGE(start, asid, scale, \ + num, tlb_level); \ + __tlbi(r##op, addr); \ + if (tlbi_user) \ + __tlbi_user(r##op, addr); \ + start += __TLBI_RANGE_PAGES(num, scale) << PAGE_SHIFT; \ + pages -= __TLBI_RANGE_PAGES(num, scale); \ + } \ + scale++; \ + } \ +} while (0) + static inline void __flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end, unsigned long stride, bool last_level, int tlb_level) { - int num = 0; - int scale = 0; - unsigned long asid, addr, pages; + unsigned long asid, pages; start = round_down(start, stride); end = round_up(end, stride); @@ -307,56 +354,11 @@ static inline void __flush_tlb_range(struct vm_area_struct *vma, dsb(ishst); asid = ASID(vma->vm_mm); - /* - * When the CPU does not support TLB range operations, flush the TLB - * entries one by one at the granularity of 'stride'. If the TLB - * range ops are supported, then: - * - * 1. If 'pages' is odd, flush the first page through non-range - * operations; - * - * 2. For remaining pages: the minimum range granularity is decided - * by 'scale', so multiple range TLBI operations may be required. - * Start from scale = 0, flush the corresponding number of pages - * ((num+1)*2^(5*scale+1) starting from 'addr'), then increase it - * until no pages left. - * - * Note that certain ranges can be represented by either num = 31 and - * scale or num = 0 and scale + 1. The loop below favours the latter - * since num is limited to 30 by the __TLBI_RANGE_NUM() macro. - */ - while (pages > 0) { - if (!system_supports_tlb_range() || - pages % 2 == 1) { - addr = __TLBI_VADDR(start, asid); - if (last_level) { - __tlbi_level(vale1is, addr, tlb_level); - __tlbi_user_level(vale1is, addr, tlb_level); - } else { - __tlbi_level(vae1is, addr, tlb_level); - __tlbi_user_level(vae1is, addr, tlb_level); - } - start += stride; - pages -= stride >> PAGE_SHIFT; - continue; - } - - num = __TLBI_RANGE_NUM(pages, scale); - if (num >= 0) { - addr = __TLBI_VADDR_RANGE(start, asid, scale, - num, tlb_level); - if (last_level) { - __tlbi(rvale1is, addr); - __tlbi_user(rvale1is, addr); - } else { - __tlbi(rvae1is, addr); - __tlbi_user(rvae1is, addr); - } - start += __TLBI_RANGE_PAGES(num, scale) << PAGE_SHIFT; - pages -= __TLBI_RANGE_PAGES(num, scale); - } - scale++; - } + if (last_level) + __flush_tlb_range_op(vale1is, start, pages, stride, asid, tlb_level, true); + else + __flush_tlb_range_op(vae1is, start, pages, stride, asid, tlb_level, true); + dsb(ish); }