From patchwork Sat Oct 28 23:11:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13439620 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 935F9C4332F for ; Sat, 28 Oct 2023 23:13:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=tOzV3e+Yrc67QKkHCwd/tbsfpYx3pnBGH+6LlJpFHU0=; b=sGZScO9a1aDV8z XLYs/q3IHB1Ln0jyq4x0TAhgDAT77Ialk0QULmF0PzWgDHuHMYiXgXzm5fbd9y2/6dxdYo/RlfjwA +U9rhiv5p8d/bHgBHWSxXkw2jubmyYnBVNoIvrJoU/w+fyotLS5HSDa2sjp0nKOKbd5ScRz5HfbIF ddTJDgD1p1y5FVSWw9GALH1t3fmNhEV7w4UBa/SiJE3AMZUTH+4BYlTVN6YR34z3BINPg2u9AYGK7 aJxJp0BBmsD7Zofif8i+dJ2Z+UxgYHWqPL5YyhhyduFiUcDQFqygtrmun9mIOTM4k+Ay8EIBFWWpO MSsdIWHUqUI/6QmoOuLw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qwsV1-000vSu-23; Sat, 28 Oct 2023 23:13:47 +0000 Received: from mail-pl1-x62b.google.com ([2607:f8b0:4864:20::62b]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qwsUy-000vRh-0v for linux-riscv@lists.infradead.org; Sat, 28 Oct 2023 23:13:45 +0000 Received: by mail-pl1-x62b.google.com with SMTP id d9443c01a7336-1c9d407bb15so29567485ad.0 for ; Sat, 28 Oct 2023 16:13:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1698534823; x=1699139623; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=2EsAJ3DvynnbB6jUfsinRvKaNQwCusxxXujfrC8RtXI=; b=Xq6bv516PfZx9bhhMnWKAHFaAWIddjlhmEw/CJRcvYmZG6Kq9YHZNf5bRnLlCm9PjN upIGCj71xt+ORV76iABIDDbxDZ9X0q+H/6Uluhi6LSuNtsjbS8fZoArolFepdb03Fc9u ghr6X6GaXjpDa1Sa1W1MzuH4U/GEEPF417wrhl1hbPZlZIvDAqVwDmCNlkZWcwQQkH/N 04VFcHjUNW3GMh9FObeGsS8p5xBk3to54Y5tTuMRL8jjIROuabZetV0blqZJpW1c+flD Zq7PnP+/eeXK+XFI6SXmApj48NKzh5qwcKX/XLXvVwv0i+kOMsPrpttYPH66/5lLfbdB 3xhQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698534823; x=1699139623; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2EsAJ3DvynnbB6jUfsinRvKaNQwCusxxXujfrC8RtXI=; b=cmGX/TX4Y4BIgFppDTklyTt6dMoIQOdzLt3WJuv6foiplNol14ss9xJbOylaBx330a tl7ICaGP+ttDjZvxIOP9PDqRHhvh3WsCI/bJrTe4FjrNSf7qPr+QpnJ9dqyenOKky/fd TocvGWr6WYu1rUmZxgG+pkA/i1XVCM6JSxY/1x47gzXHf0cxdhRoQ/3rSLi5SJa+TpzX XLETL3KWDBXv25kPHU6OsI+ALfm/qCoGsEd071WD1276lzxXHRCxEy2kgAfATQ7PG+Yq Gv34dPZiDsZTrhDX6UrD4yF8Ji9p7ZnumdoK41RYI0XnMncibxup0YVOVoPm5B4tKFNY aSaQ== X-Gm-Message-State: AOJu0YwLQWihsxxPGlinenrqQM3vCF7qsPdfbaDTQvY3jIJ2VMD4E7kJ dWdmj8fgkFkrryHvGDPmELiDWQ== X-Google-Smtp-Source: AGHT+IFw/0gO2jty39BfEl53e0TAm8jThBxaHB3Pi6HPPc0H5aL70GOaasoPy4Ex/mKBNftI1RCeRA== X-Received: by 2002:a17:902:cf41:b0:1ca:8252:a91 with SMTP id e1-20020a170902cf4100b001ca82520a91mr7454451plg.41.1698534823149; Sat, 28 Oct 2023 16:13:43 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id u17-20020a17090341d100b001b8622c1ad2sm3679345ple.130.2023.10.28.16.13.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Oct 2023 16:13:42 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , Alexandre Ghiti , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Andrew Jones , Lad Prabhakar , Samuel Holland Subject: [PATCH v2 01/11] riscv: Improve tlb_flush() Date: Sat, 28 Oct 2023 16:11:59 -0700 Message-ID: <20231028231339.3116618-2-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231028231339.3116618-1-samuel.holland@sifive.com> References: <20231028231339.3116618-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231028_161344_323512_EAB52FA9 X-CRM114-Status: GOOD ( 11.99 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Alexandre Ghiti For now, tlb_flush() simply calls flush_tlb_mm() which results in a flush of the whole TLB. So let's use mmu_gather fields to provide a more fine-grained flush of the TLB. Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones Tested-by: Lad Prabhakar # On RZ/Five SMARC Signed-off-by: Samuel Holland --- Changes in v2: - Rebase on Alexandre's "riscv: tlb flush improvements" series v5 arch/riscv/include/asm/tlb.h | 8 +++++++- arch/riscv/include/asm/tlbflush.h | 3 +++ arch/riscv/mm/tlbflush.c | 7 +++++++ 3 files changed, 17 insertions(+), 1 deletion(-) diff --git a/arch/riscv/include/asm/tlb.h b/arch/riscv/include/asm/tlb.h index 120bcf2ed8a8..1eb5682b2af6 100644 --- a/arch/riscv/include/asm/tlb.h +++ b/arch/riscv/include/asm/tlb.h @@ -15,7 +15,13 @@ static void tlb_flush(struct mmu_gather *tlb); static inline void tlb_flush(struct mmu_gather *tlb) { - flush_tlb_mm(tlb->mm); +#ifdef CONFIG_MMU + if (tlb->fullmm || tlb->need_flush_all) + flush_tlb_mm(tlb->mm); + else + flush_tlb_mm_range(tlb->mm, tlb->start, tlb->end, + tlb_get_unmap_size(tlb)); +#endif } #endif /* _ASM_RISCV_TLB_H */ diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index a09196f8de68..f5c4fb0ae642 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -32,6 +32,8 @@ static inline void local_flush_tlb_page(unsigned long addr) #if defined(CONFIG_SMP) && defined(CONFIG_MMU) void flush_tlb_all(void); void flush_tlb_mm(struct mm_struct *mm); +void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, + unsigned long end, unsigned int page_size); void flush_tlb_page(struct vm_area_struct *vma, unsigned long addr); void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end); @@ -52,6 +54,7 @@ static inline void flush_tlb_range(struct vm_area_struct *vma, } #define flush_tlb_mm(mm) flush_tlb_all() +#define flush_tlb_mm_range(mm, start, end, page_size) flush_tlb_all() #endif /* !CONFIG_SMP || !CONFIG_MMU */ /* Flush a range of kernel pages */ diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 77be59aadc73..fa03289853d8 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -132,6 +132,13 @@ void flush_tlb_mm(struct mm_struct *mm) __flush_tlb_range(mm, 0, -1, PAGE_SIZE); } +void flush_tlb_mm_range(struct mm_struct *mm, + unsigned long start, unsigned long end, + unsigned int page_size) +{ + __flush_tlb_range(mm, start, end - start, page_size); +} + void flush_tlb_page(struct vm_area_struct *vma, unsigned long addr) { __flush_tlb_range(vma->vm_mm, addr, PAGE_SIZE, PAGE_SIZE); From patchwork Sat Oct 28 23:12:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13439622 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 67493C00141 for ; Sat, 28 Oct 2023 23:14:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=UTdTCNVfuJqxnTixd26Ly5vZCfKD+1FZCa1EDUWVXmw=; b=1e5XsP0+w1wgys 14OQO5hTs8MHEk32jEzlIGsKfw7WxBwv2K0kQA2oL5LUL2+W5HQAasyzea13D8BQQL2jbJLvbi2KH Zbu2bgVThLEN8pJo2hCcUG+1Sd1o88LqfvUXcaU6hkHjULVnbumgTWJkJK0YHWrJ8cniYJmjCWQUu qbipKa39KBIpUzIBZ6JSVVOD7Yuy3AtyCZ0/hpfosOFu3edWKwu5BKFQm0hZVYP9SZ4mbd4IZOigI VcrOih/sJgu0o3RUTor69GwbqjZLfQ1Xmu04ngSOjmUa88B6y7LSTW+m5ZraRcLwDyGsk8aOdR2kc bt/KQT2b2URxIGd5jkPQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qwsV5-000vVS-1S; Sat, 28 Oct 2023 23:13:51 +0000 Received: from mail-pf1-x42e.google.com ([2607:f8b0:4864:20::42e]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qwsUz-000vS8-2W for linux-riscv@lists.infradead.org; Sat, 28 Oct 2023 23:13:47 +0000 Received: by mail-pf1-x42e.google.com with SMTP id d2e1a72fcca58-6b709048d8eso2874933b3a.2 for ; Sat, 28 Oct 2023 16:13:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1698534824; x=1699139624; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=h4NRNhpDELMiOf33wYuzLfhwPwCzaIpnJue9EEzIQdY=; b=bQEkK+8Aox+2ymqbxmZ+yoGMQ644R/uIL5aBiBqyzecTPCD7/4gPy0Rbln0AkhYp6A Qrs/niVH2ViRpj5CdVhhj5YYqTbE786K35b41CUjLlbjUWaoLvvQAKyhT7FJu8F+1AYE e5UlYHLEVktx/bHveEcz+bhzv8wfbne2CpuvMwo1d8XfbEGRVW4SocfJEESZCKJTvUbh nq80u5h399fRl+MPZovwZFjbcJ3t0ueaGj+X2DfO60ntzdVfHW55MEryU68kUf5fqdxV 5G0ZrIT5axY7s+xwjxKZw8jA6mJXoGdycUd/MJlIo50AkoFJHGMj442jMGx28rs1nN57 Hsog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698534824; x=1699139624; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=h4NRNhpDELMiOf33wYuzLfhwPwCzaIpnJue9EEzIQdY=; b=R/F/YMY/AM8LMyF3GDDSVvwvaYJYjb3bTHbHRLCioDtXFXVUD6zsxfVd8qkdT9ScM4 Tjuk+GhYNCmdzk6n3+N64i9jD3/thXxCV9SBD4kKznm91WfOBurdCm0EIVnRtYwDRB6S 7c65O/CjVPS6ZGEAyL8J5v3WX0jsFY7DcFdCljA1SAVLeZnonlAMi3UwyLHjg3Oe6Fqq YlSTkTGTcQqJW+ScxhbVq8QhwM8DKMCaYFwwP4CY2+EEVQnAJkKuoL7MwTNnKGShJTU6 S8DboGkhw3f8/0rsBX9jlw68zMNCrgtzFon+9H6takHeAKxtdiS5TndZf8S935yCv1GM /AEA== X-Gm-Message-State: AOJu0Yysde/yV/ZYrIGElA8d1FPhakBuuCjb2AvCFPyJIKI9V0BPUxo3 t7LFIMr8oK5XlB6MRsQ1I8IsjxPsgMZu8MXYq5o= X-Google-Smtp-Source: AGHT+IHFtwqkvMLL8vTXw7lsMTaEWiNW/nYYuBPFSSXqkCgtp1/aoWCoS2LcxRJlFYSU2eWn0N2URQ== X-Received: by 2002:a05:6a21:3b45:b0:16b:a5fb:eee5 with SMTP id zy5-20020a056a213b4500b0016ba5fbeee5mr5984191pzb.28.1698534824609; Sat, 28 Oct 2023 16:13:44 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id u17-20020a17090341d100b001b8622c1ad2sm3679345ple.130.2023.10.28.16.13.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Oct 2023 16:13:43 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , Alexandre Ghiti , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Samuel Holland Subject: [PATCH v2 02/11] riscv: Improve flush_tlb_range() for hugetlb pages Date: Sat, 28 Oct 2023 16:12:00 -0700 Message-ID: <20231028231339.3116618-3-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231028231339.3116618-1-samuel.holland@sifive.com> References: <20231028231339.3116618-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231028_161345_819312_F29B36C1 X-CRM114-Status: GOOD ( 14.54 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Alexandre Ghiti flush_tlb_range() uses a fixed stride of PAGE_SIZE and in its current form, when a hugetlb mapping needs to be flushed, flush_tlb_range() flushes the whole tlb: so set a stride of the size of the hugetlb mapping in order to only flush the hugetlb mapping. However, if the hugepage is a NAPOT region, all PTEs that constitute this mapping must be invalidated, so the stride size must actually be the size of the PTE. Note that THPs are directly handled by flush_pmd_tlb_range(). Signed-off-by: Alexandre Ghiti [Samuel: Removed CONFIG_RISCV_ISA_SVNAPOT check] Signed-off-by: Samuel Holland --- Changes in v2: - Rebase on Alexandre's "riscv: tlb flush improvements" series v5 arch/riscv/mm/tlbflush.c | 29 ++++++++++++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-) diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index fa03289853d8..b6d712a82306 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -3,6 +3,7 @@ #include #include #include +#include #include #include @@ -147,7 +148,33 @@ void flush_tlb_page(struct vm_area_struct *vma, unsigned long addr) void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end) { - __flush_tlb_range(vma->vm_mm, start, end - start, PAGE_SIZE); + unsigned long stride_size; + + if (!is_vm_hugetlb_page(vma)) { + stride_size = PAGE_SIZE; + } else { + stride_size = huge_page_size(hstate_vma(vma)); + + /* + * As stated in the privileged specification, every PTE in a + * NAPOT region must be invalidated, so reset the stride in that + * case. + */ + if (has_svnapot()) { + if (stride_size >= PGDIR_SIZE) + stride_size = PGDIR_SIZE; + else if (stride_size >= P4D_SIZE) + stride_size = P4D_SIZE; + else if (stride_size >= PUD_SIZE) + stride_size = PUD_SIZE; + else if (stride_size >= PMD_SIZE) + stride_size = PMD_SIZE; + else + stride_size = PAGE_SIZE; + } + } + + __flush_tlb_range(vma->vm_mm, start, end - start, stride_size); } #ifdef CONFIG_TRANSPARENT_HUGEPAGE void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, From patchwork Sat Oct 28 23:12:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13439624 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6E4D1C001B5 for ; Sat, 28 Oct 2023 23:14:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=L4tN0TN3ID15AVAC7LlQmkCIlBCKKFoH9WygVxgM1Kk=; b=Dgp3QoNZ3T0Yzl qJVcC7ZiwK41auzyKhZ4WV2FGTezVgOFQuCgiSPMHbKSvmS6WOXihUKSC/YyuG6VBgp+hVQiN0uwx PIVxyIWo5hgGADkCLsPrCxzBA01Ti+TBKCmmzC2O1d0asHfuWKnYRkjRSIqpmNnwFLWEZNfQ16AiE 9do6LHW/nxPDxzrZveB3XJ3ghkfF6ORMdCQDAsadU7mv+7Luy2yzDHYqTqI7zWQYPjymCK9jl3Y63 X8wmiAE7bKei+cA4VgBf4TxkDgIFEDPAO86BeMmLqSTLKyMNHe2Pa4mr/MliLS9FhiJa6fexD5z/f UOQhUIROUbyqxBWgGKjw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qwsV6-000vVm-0d; Sat, 28 Oct 2023 23:13:52 +0000 Received: from mail-pl1-x62e.google.com ([2607:f8b0:4864:20::62e]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qwsV0-000vSX-37 for linux-riscv@lists.infradead.org; Sat, 28 Oct 2023 23:13:48 +0000 Received: by mail-pl1-x62e.google.com with SMTP id d9443c01a7336-1cc1e1e74beso17386075ad.1 for ; Sat, 28 Oct 2023 16:13:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1698534826; x=1699139626; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=KwX4pskItPHUM4maFDdS5O3Er1CtrhdQQ55njchpgeM=; b=U1vnSzAigMqB1xWWHECE7gV6jUF2OCklA/OFdTzKNGkvBGKbiT4H5yuTqNE35j4dJV sEBLNA8C7ZvBI2+iXeJ4v1skOKHOxjj3fzmU9gREc7GQr94h9uqO8LYuyfmTq+/asO6S uBIB3g/GjiSLcANTdsn36jX0v4vEW9gWJDq8lZN7Crsp3T8mp3OvEfuARC34IMLuVgPx /X9SB20qrtFn+Ty1ie8wh71IbLI8Y3cYtVI/KE1qBsBilnS/ro9klBnVkV3zTDIbrL4b 3nxjFjK/KverzEAdhlcBd+AxS0f8bs2nhH8kYy1u46lVmrhpHKwC/ERFw9S3qFLPfa/4 gryA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698534826; x=1699139626; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KwX4pskItPHUM4maFDdS5O3Er1CtrhdQQ55njchpgeM=; b=UrIPS8zwYrRj5ihzoljRBwjJWIeaLJv5F5iHU/TZglZMuskUPHinNT5e3CgQw3t/eg QqY7v/FXXoAzLK4nWllNqkdbyOAeqWJiet5nlL/OOLElb1LQWyiudi0Ai0G2WpeVlQNC POE07Q4D4YaQkUgLexBSv6UE+HF4MdM3/HXAy+AdNH3y8w7Pp25iipOizMD0QXXAPznf htfz3UGTX+D/dp+39+Ivconq/A5kf2eXKivt6bsMN9MP4dJsSK8cA3mkAdWO2CCBHWpr baxRTTn9uyg3x7SvR1PTayW8pas+77VS/YpYdUD9mUW02Tp9vdzS8LJOcnB3SKuWkkQo bXpg== X-Gm-Message-State: AOJu0YxLtA9IvFYk6fh9dFslvRCOKLaMOYwi4t13a1QENN7VMjq8q6Ac Z0SR3IlLNs7MpPjC8x/FvmRRFs2J3CApsWrRnpw= X-Google-Smtp-Source: AGHT+IEP1RLLbzkV6SDyhDMWypu6mUWTgp+BfL6QnGjrDAvgGu3DpZZT/BgKJhnh9HCZ+gZpZ5q05A== X-Received: by 2002:a17:902:8c85:b0:1c9:c0fa:dfb7 with SMTP id t5-20020a1709028c8500b001c9c0fadfb7mr6367498plo.57.1698534825930; Sat, 28 Oct 2023 16:13:45 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id u17-20020a17090341d100b001b8622c1ad2sm3679345ple.130.2023.10.28.16.13.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Oct 2023 16:13:45 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , Alexandre Ghiti , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Mayuresh Chitale , Andrew Jones , Lad Prabhakar , Samuel Holland Subject: [PATCH v2 03/11] riscv: Make __flush_tlb_range() loop over pte instead of flushing the whole tlb Date: Sat, 28 Oct 2023 16:12:01 -0700 Message-ID: <20231028231339.3116618-4-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231028231339.3116618-1-samuel.holland@sifive.com> References: <20231028231339.3116618-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231028_161347_019263_65DA4560 X-CRM114-Status: GOOD ( 22.02 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Alexandre Ghiti Currently, when the range to flush covers more than one page (a 4K page or a hugepage), __flush_tlb_range() flushes the whole tlb. Flushing the whole tlb comes with a greater cost than flushing a single entry so we should flush single entries up to a certain threshold so that: threshold * cost of flushing a single entry < cost of flushing the whole tlb. Co-developed-by: Mayuresh Chitale Signed-off-by: Mayuresh Chitale Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones Tested-by: Lad Prabhakar # On RZ/Five SMARC [Samuel: Fixed type of nr_ptes_in_range] Signed-off-by: Samuel Holland --- Changes in v2: - Rebase on Alexandre's "riscv: tlb flush improvements" series v5 arch/riscv/include/asm/sbi.h | 3 - arch/riscv/include/asm/tlbflush.h | 3 + arch/riscv/kernel/sbi.c | 32 +++------ arch/riscv/mm/tlbflush.c | 115 +++++++++++++++--------------- 4 files changed, 72 insertions(+), 81 deletions(-) diff --git a/arch/riscv/include/asm/sbi.h b/arch/riscv/include/asm/sbi.h index 5b4a1bf5f439..b79d0228144f 100644 --- a/arch/riscv/include/asm/sbi.h +++ b/arch/riscv/include/asm/sbi.h @@ -273,9 +273,6 @@ void sbi_set_timer(uint64_t stime_value); void sbi_shutdown(void); void sbi_send_ipi(unsigned int cpu); int sbi_remote_fence_i(const struct cpumask *cpu_mask); -int sbi_remote_sfence_vma(const struct cpumask *cpu_mask, - unsigned long start, - unsigned long size); int sbi_remote_sfence_vma_asid(const struct cpumask *cpu_mask, unsigned long start, diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index f5c4fb0ae642..170a49c531c6 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -11,6 +11,9 @@ #include #include +#define FLUSH_TLB_MAX_SIZE ((unsigned long)-1) +#define FLUSH_TLB_NO_ASID ((unsigned long)-1) + #ifdef CONFIG_MMU extern unsigned long asid_mask; diff --git a/arch/riscv/kernel/sbi.c b/arch/riscv/kernel/sbi.c index c672c8ba9a2a..5a62ed1da453 100644 --- a/arch/riscv/kernel/sbi.c +++ b/arch/riscv/kernel/sbi.c @@ -11,6 +11,7 @@ #include #include #include +#include /* default SBI version is 0.1 */ unsigned long sbi_spec_version __ro_after_init = SBI_SPEC_VERSION_DEFAULT; @@ -376,32 +377,15 @@ int sbi_remote_fence_i(const struct cpumask *cpu_mask) } EXPORT_SYMBOL(sbi_remote_fence_i); -/** - * sbi_remote_sfence_vma() - Execute SFENCE.VMA instructions on given remote - * harts for the specified virtual address range. - * @cpu_mask: A cpu mask containing all the target harts. - * @start: Start of the virtual address - * @size: Total size of the virtual address range. - * - * Return: 0 on success, appropriate linux error code otherwise. - */ -int sbi_remote_sfence_vma(const struct cpumask *cpu_mask, - unsigned long start, - unsigned long size) -{ - return __sbi_rfence(SBI_EXT_RFENCE_REMOTE_SFENCE_VMA, - cpu_mask, start, size, 0, 0); -} -EXPORT_SYMBOL(sbi_remote_sfence_vma); - /** * sbi_remote_sfence_vma_asid() - Execute SFENCE.VMA instructions on given - * remote harts for a virtual address range belonging to a specific ASID. + * remote harts for a virtual address range belonging to a specific ASID or not. * * @cpu_mask: A cpu mask containing all the target harts. * @start: Start of the virtual address * @size: Total size of the virtual address range. - * @asid: The value of address space identifier (ASID). + * @asid: The value of address space identifier (ASID), or FLUSH_TLB_NO_ASID + * for flushing all address spaces. * * Return: 0 on success, appropriate linux error code otherwise. */ @@ -410,8 +394,12 @@ int sbi_remote_sfence_vma_asid(const struct cpumask *cpu_mask, unsigned long size, unsigned long asid) { - return __sbi_rfence(SBI_EXT_RFENCE_REMOTE_SFENCE_VMA_ASID, - cpu_mask, start, size, asid, 0); + if (asid == FLUSH_TLB_NO_ASID) + return __sbi_rfence(SBI_EXT_RFENCE_REMOTE_SFENCE_VMA, + cpu_mask, start, size, 0, 0); + else + return __sbi_rfence(SBI_EXT_RFENCE_REMOTE_SFENCE_VMA_ASID, + cpu_mask, start, size, asid, 0); } EXPORT_SYMBOL(sbi_remote_sfence_vma_asid); diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index b6d712a82306..e46fefc70927 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -9,28 +9,50 @@ static inline void local_flush_tlb_all_asid(unsigned long asid) { - __asm__ __volatile__ ("sfence.vma x0, %0" - : - : "r" (asid) - : "memory"); + if (asid != FLUSH_TLB_NO_ASID) + __asm__ __volatile__ ("sfence.vma x0, %0" + : + : "r" (asid) + : "memory"); + else + local_flush_tlb_all(); } static inline void local_flush_tlb_page_asid(unsigned long addr, unsigned long asid) { - __asm__ __volatile__ ("sfence.vma %0, %1" - : - : "r" (addr), "r" (asid) - : "memory"); + if (asid != FLUSH_TLB_NO_ASID) + __asm__ __volatile__ ("sfence.vma %0, %1" + : + : "r" (addr), "r" (asid) + : "memory"); + else + local_flush_tlb_page(addr); } -static inline void local_flush_tlb_range(unsigned long start, - unsigned long size, unsigned long stride) +/* + * Flush entire TLB if number of entries to be flushed is greater + * than the threshold below. + */ +static unsigned long tlb_flush_all_threshold __read_mostly = 64; + +static void local_flush_tlb_range_threshold_asid(unsigned long start, + unsigned long size, + unsigned long stride, + unsigned long asid) { - if (size <= stride) - local_flush_tlb_page(start); - else - local_flush_tlb_all(); + unsigned long nr_ptes_in_range = DIV_ROUND_UP(size, stride); + int i; + + if (nr_ptes_in_range > tlb_flush_all_threshold) { + local_flush_tlb_all_asid(asid); + return; + } + + for (i = 0; i < nr_ptes_in_range; ++i) { + local_flush_tlb_page_asid(start, asid); + start += stride; + } } static inline void local_flush_tlb_range_asid(unsigned long start, @@ -38,8 +60,10 @@ static inline void local_flush_tlb_range_asid(unsigned long start, { if (size <= stride) local_flush_tlb_page_asid(start, asid); - else + else if (size == FLUSH_TLB_MAX_SIZE) local_flush_tlb_all_asid(asid); + else + local_flush_tlb_range_threshold_asid(start, size, stride, asid); } static void __ipi_flush_tlb_all(void *info) @@ -52,7 +76,7 @@ void flush_tlb_all(void) if (riscv_use_ipi_for_rfence()) on_each_cpu(__ipi_flush_tlb_all, NULL, 1); else - sbi_remote_sfence_vma(NULL, 0, -1); + sbi_remote_sfence_vma_asid(NULL, 0, FLUSH_TLB_MAX_SIZE, FLUSH_TLB_NO_ASID); } struct flush_tlb_range_data { @@ -69,18 +93,12 @@ static void __ipi_flush_tlb_range_asid(void *info) local_flush_tlb_range_asid(d->start, d->size, d->stride, d->asid); } -static void __ipi_flush_tlb_range(void *info) -{ - struct flush_tlb_range_data *d = info; - - local_flush_tlb_range(d->start, d->size, d->stride); -} - static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, unsigned long size, unsigned long stride) { struct flush_tlb_range_data ftd; struct cpumask *cmask = mm_cpumask(mm); + unsigned long asid = FLUSH_TLB_NO_ASID; unsigned int cpuid; bool broadcast; @@ -90,39 +108,24 @@ static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, cpuid = get_cpu(); /* check if the tlbflush needs to be sent to other CPUs */ broadcast = cpumask_any_but(cmask, cpuid) < nr_cpu_ids; - if (static_branch_unlikely(&use_asid_allocator)) { - unsigned long asid = atomic_long_read(&mm->context.id) & asid_mask; - - if (broadcast) { - if (riscv_use_ipi_for_rfence()) { - ftd.asid = asid; - ftd.start = start; - ftd.size = size; - ftd.stride = stride; - on_each_cpu_mask(cmask, - __ipi_flush_tlb_range_asid, - &ftd, 1); - } else - sbi_remote_sfence_vma_asid(cmask, - start, size, asid); - } else { - local_flush_tlb_range_asid(start, size, stride, asid); - } + + if (static_branch_unlikely(&use_asid_allocator)) + asid = atomic_long_read(&mm->context.id) & asid_mask; + + if (broadcast) { + if (riscv_use_ipi_for_rfence()) { + ftd.asid = asid; + ftd.start = start; + ftd.size = size; + ftd.stride = stride; + on_each_cpu_mask(cmask, + __ipi_flush_tlb_range_asid, + &ftd, 1); + } else + sbi_remote_sfence_vma_asid(cmask, + start, size, asid); } else { - if (broadcast) { - if (riscv_use_ipi_for_rfence()) { - ftd.asid = 0; - ftd.start = start; - ftd.size = size; - ftd.stride = stride; - on_each_cpu_mask(cmask, - __ipi_flush_tlb_range, - &ftd, 1); - } else - sbi_remote_sfence_vma(cmask, start, size); - } else { - local_flush_tlb_range(start, size, stride); - } + local_flush_tlb_range_asid(start, size, stride, asid); } put_cpu(); @@ -130,7 +133,7 @@ static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, void flush_tlb_mm(struct mm_struct *mm) { - __flush_tlb_range(mm, 0, -1, PAGE_SIZE); + __flush_tlb_range(mm, 0, FLUSH_TLB_MAX_SIZE, PAGE_SIZE); } void flush_tlb_mm_range(struct mm_struct *mm, From patchwork Sat Oct 28 23:12:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13439623 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 758B1C001DB for ; Sat, 28 Oct 2023 23:14:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=plDiYhilybIB3i0SmyMCHW6uc3uIlMYAfeg8QEzFg2k=; b=gqPl9Vm/ACVx0k g+MV5w/QxXRE0xzj/m9RR+DUW0RlfCC/Fg8M+A56lFUqE9gC8gpiqWK72efvGf4B570cfgs50qNnH XREPyMZmM7qFIIKq1cS2xh15KBYyuWkT3A/WxwiJK5xbAXMtGBxkfiuelJ+j6p1NPjqod+HI1+9Ds o4eYMUFovvG4NYjzc09VTzlDz/Rb/CPaLhE7guVdYOHRKmrwocX6O7yUrbzL7kip4HEjxG2Tg4HCO 82yCTIgWDZacup+cLxL0D71T4nyxNig2KDnQmR7uRZSuGUfHAfzdBuV24408TK29n0Rt6d9gax8tW j+scr5pY5Ciz7xs5GqcA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qwsVB-000vYe-1m; Sat, 28 Oct 2023 23:13:57 +0000 Received: from mail-oo1-xc2c.google.com ([2607:f8b0:4864:20::c2c]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qwsV3-000vSy-03 for linux-riscv@lists.infradead.org; Sat, 28 Oct 2023 23:13:50 +0000 Received: by mail-oo1-xc2c.google.com with SMTP id 006d021491bc7-581edcde26cso2063469eaf.1 for ; Sat, 28 Oct 2023 16:13:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1698534827; x=1699139627; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=56YUmquglDtdI2AgsFp8Dfqen0eRNcOUaE781b7rud8=; b=hWbAgnUioEl2nWEFFhlShO/jIRbFOvoL3x5uxCwXGiDR+p/5RvT7TKTEpMDRGTSPzb EkP4AxhcDU7GTX7XUT99gBmDCkDeq8voqg5lx7oHw1mg9V/iHLG1ODu4AEDcNSJSGJD6 5aDCbkFRjQHl6Gtniw3zECuK0RYK9mIlHtCIXb51aypWnsfUzGH+VXPS2Koh3eR2m4q8 EV9iWowye0w9nf6fjJ0enph+mnGA14W/NVMJK2NlGXbkNvKUY8RLqwvHCGtkX9+dhuK3 Yh85Cw5RCzK1gOgjy2MV8k5iyywzeDNTvVO5utNo8YXj2k6rRfAxZ6Zy4LV9BjIXLTGg G7rQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698534827; x=1699139627; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=56YUmquglDtdI2AgsFp8Dfqen0eRNcOUaE781b7rud8=; b=KuK1MNl/1ZUgVeO1jUqLxVuKlKZxcMb/MzMhKn/N2+LCEcR40vNKBZzt7kOPIF9GQ0 fOpt9Ecq1mrHjxnJAq9wnAhfYv6VT6ItksayGTnOZlvCb73rA+pK6JvKLI6QuZbUgk+v ns5MA6017JWHt9xensg51qmF9MDvAEciz6wPjzQCr62rX7YWBaCJ6y7EErWjWonyJAbX KNRfMqMgGdaBfSj1OGUgbX+HlSszqy6GuKHm0RwF9kTKxxzWwL+nWdl9VT5N72tslovM hGULlecpEo43HcC/KsNjOWRk8SK00AicnQnkaAH3nFmPTAx7822Z5gJ9yzwrqRw362aA LGTg== X-Gm-Message-State: AOJu0YyjZGOKvfz5bTwMilS4xbwXbqL0piOK5V+W3ebeezc8XpoaJ4K+ AvY48DPfX5QZ0blsrDnBa5Mbzw== X-Google-Smtp-Source: AGHT+IGq5Mcxzf3tN07mJTIgv1Wxotaz2Uaine2ZZiRfV+MdLhpoDUKwMV8rRW6x4wnAP4zHnD4SGA== X-Received: by 2002:a05:6358:903:b0:168:dc03:5b90 with SMTP id r3-20020a056358090300b00168dc035b90mr9086199rwi.23.1698534827455; Sat, 28 Oct 2023 16:13:47 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id u17-20020a17090341d100b001b8622c1ad2sm3679345ple.130.2023.10.28.16.13.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Oct 2023 16:13:46 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , Alexandre Ghiti , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Andrew Jones , Lad Prabhakar , Samuel Holland Subject: [PATCH v2 04/11] riscv: Improve flush_tlb_kernel_range() Date: Sat, 28 Oct 2023 16:12:02 -0700 Message-ID: <20231028231339.3116618-5-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231028231339.3116618-1-samuel.holland@sifive.com> References: <20231028231339.3116618-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231028_161349_055965_CDDAD197 X-CRM114-Status: GOOD ( 17.12 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Alexandre Ghiti This function used to simply flush the whole tlb of all harts, be more subtile and try to only flush the range. The problem is that we can only use PAGE_SIZE as stride since we don't know the size of the underlying mapping and then this function will be improved only if the size of the region to flush is < threshold * PAGE_SIZE. Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones Tested-by: Lad Prabhakar # On RZ/Five SMARC [Samuel: Use cpu_online_mask and merge if statements] Signed-off-by: Samuel Holland --- Changes in v2: - Rebase on Alexandre's "riscv: tlb flush improvements" series v5 arch/riscv/include/asm/tlbflush.h | 11 +++++----- arch/riscv/mm/tlbflush.c | 34 ++++++++++++++++++++++--------- 2 files changed, 30 insertions(+), 15 deletions(-) diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index 170a49c531c6..8f3418c5f172 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -40,6 +40,7 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, void flush_tlb_page(struct vm_area_struct *vma, unsigned long addr); void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end); +void flush_tlb_kernel_range(unsigned long start, unsigned long end); #ifdef CONFIG_TRANSPARENT_HUGEPAGE #define __HAVE_ARCH_FLUSH_PMD_TLB_RANGE void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, @@ -56,15 +57,15 @@ static inline void flush_tlb_range(struct vm_area_struct *vma, local_flush_tlb_all(); } -#define flush_tlb_mm(mm) flush_tlb_all() -#define flush_tlb_mm_range(mm, start, end, page_size) flush_tlb_all() -#endif /* !CONFIG_SMP || !CONFIG_MMU */ - /* Flush a range of kernel pages */ static inline void flush_tlb_kernel_range(unsigned long start, unsigned long end) { - flush_tlb_all(); + local_flush_tlb_all(); } +#define flush_tlb_mm(mm) flush_tlb_all() +#define flush_tlb_mm_range(mm, start, end, page_size) flush_tlb_all() +#endif /* !CONFIG_SMP || !CONFIG_MMU */ + #endif /* _ASM_RISCV_TLBFLUSH_H */ diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index e46fefc70927..e6659d7368b3 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -97,20 +97,27 @@ static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, unsigned long size, unsigned long stride) { struct flush_tlb_range_data ftd; - struct cpumask *cmask = mm_cpumask(mm); + const struct cpumask *cmask; unsigned long asid = FLUSH_TLB_NO_ASID; - unsigned int cpuid; bool broadcast; - if (cpumask_empty(cmask)) - return; + if (mm) { + unsigned int cpuid; + + cmask = mm_cpumask(mm); + if (cpumask_empty(cmask)) + return; - cpuid = get_cpu(); - /* check if the tlbflush needs to be sent to other CPUs */ - broadcast = cpumask_any_but(cmask, cpuid) < nr_cpu_ids; + cpuid = get_cpu(); + /* check if the tlbflush needs to be sent to other CPUs */ + broadcast = cpumask_any_but(cmask, cpuid) < nr_cpu_ids; - if (static_branch_unlikely(&use_asid_allocator)) - asid = atomic_long_read(&mm->context.id) & asid_mask; + if (static_branch_unlikely(&use_asid_allocator)) + asid = atomic_long_read(&mm->context.id) & asid_mask; + } else { + cmask = cpu_online_mask; + broadcast = true; + } if (broadcast) { if (riscv_use_ipi_for_rfence()) { @@ -128,7 +135,8 @@ static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, local_flush_tlb_range_asid(start, size, stride, asid); } - put_cpu(); + if (mm) + put_cpu(); } void flush_tlb_mm(struct mm_struct *mm) @@ -179,6 +187,12 @@ void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, __flush_tlb_range(vma->vm_mm, start, end - start, stride_size); } + +void flush_tlb_kernel_range(unsigned long start, unsigned long end) +{ + __flush_tlb_range(NULL, start, end - start, PAGE_SIZE); +} + #ifdef CONFIG_TRANSPARENT_HUGEPAGE void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end) From patchwork Sat Oct 28 23:12:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13439625 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 825FBC001B2 for ; Sat, 28 Oct 2023 23:14:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=e2rd1gZXId6htHBPaudLwIQXbx2S1eD5QVtUdyy7LPo=; b=r7/05BDIIexUJq TrYhq/hjtxXxALua9Jy7aF2nJodLvBlmkdnSK3OVmq+Sjc4YwFjK6NxBUyqT/9Jfr8qSH9douNsqh bHa01UvJ/eBt8XYUVgCexDK0Zt+K7RnbdUFUycROLX0UX+QN3nEBWV5sA07KfL/vp0mRtbhuU3fPi ZMzxoGnQBzvE/I8WS9wcdzQ7cPfdrP/NTzFi0e98TehL0sJB172QI+Rpl17Pfwe59Ezga22ORqelL nJQqQz1ls6kYsWXod6zXrQUb1eCJ8ye54OHxvMCrNLI3A9a3id1FzCyTmi2W++7vkuwoXQOByTBDI 6EUA/UQdDW/hZ9wHb7aA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qwsVC-000vZH-1B; Sat, 28 Oct 2023 23:13:58 +0000 Received: from mail-pl1-x62b.google.com ([2607:f8b0:4864:20::62b]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qwsV3-000vTJ-0r for linux-riscv@lists.infradead.org; Sat, 28 Oct 2023 23:13:50 +0000 Received: by mail-pl1-x62b.google.com with SMTP id d9443c01a7336-1cc20df5187so12487395ad.0 for ; Sat, 28 Oct 2023 16:13:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1698534829; x=1699139629; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7iD5kVeuyzksz9qFDj4nk3gJEMMhm0SiL+jEgfo9gAo=; b=g3A+5bVkMqxwFOBTngd9Uh7BPn4awyGxnrFN+vzOlm9z9yr3KYjbHLFv7AdNF2d+0G hpEcEQRZl2vy3GbgC0XvhBUIul2v0h1KUu5KpjrqGB3HuHRsJ3rHrTbZeHeLoiD1h/pp ggtxzjHyzxPZ+AUAo9Q2HwuYI5/OMCts+SABuL2sR/o3tA4ZiHa0xucYV5/Ku1R3ZtWc nDbNoDXS5kiztBs67zoCpRS07V+h8fFiDXLz+MBLt12KNBGUamYOBLPlIFVgFsel4M+0 LEXRAv/vJcRJKaSpSi+/AnXunvajVpgiSBnbUyNjvaNjKqJtVfG7/mvj1SIT8h7x+PHy 14uw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698534829; x=1699139629; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7iD5kVeuyzksz9qFDj4nk3gJEMMhm0SiL+jEgfo9gAo=; b=ggVCCHsyzOFvMsz5e1KHrxT3pUKpsj4EzyPDuiznthYRq0PPYDUMLet/0LboAG76nX Vc/+kCyKvhGfU0n+ql34Lzo8g3m+Wth03aEKS4jIAxseOlCsSzdHQjHAxsGfCvT63ILq +HqIGGQt2w/DPllG70FygCZXtvLap9wcOfH+Y6eoHJS7LDAiZI558ozhr9csMmfV5Mco eTb8mJJ0F08FDMQD1kRLXhXwFGIiM5jc2ag7NPdSAQfoBJb1hi/yttM6PwbaibyPMwqx Q8LIT4XuggLILJfjFFqpwm0+H9/Em9v3lIIRHD+4rJDpFfqEwBLgSZCu/wBPEJD6+OcA 9F6g== X-Gm-Message-State: AOJu0Ywnnjzso1hfkQt4BPHkr7+ONvE2uew00YFzjNepzG1HGeFVvkmk v1c2Pc9qWBQZKtGuttSWhyZe2g== X-Google-Smtp-Source: AGHT+IGZfjM6oWIducpDDQ7hM02X3ekD2Mrbs3+cVpCJdtVQrXLcSNbo2i119+LD0HVTSm0uB+XkgQ== X-Received: by 2002:a17:903:11d1:b0:1cc:f41:8f82 with SMTP id q17-20020a17090311d100b001cc0f418f82mr8777913plh.16.1698534828754; Sat, 28 Oct 2023 16:13:48 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id u17-20020a17090341d100b001b8622c1ad2sm3679345ple.130.2023.10.28.16.13.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Oct 2023 16:13:48 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , Alexandre Ghiti , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Samuel Holland Subject: [PATCH v2 05/11] riscv: mm: Combine the SMP and UP TLB flush code Date: Sat, 28 Oct 2023 16:12:03 -0700 Message-ID: <20231028231339.3116618-6-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231028231339.3116618-1-samuel.holland@sifive.com> References: <20231028231339.3116618-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231028_161349_319524_CD71C271 X-CRM114-Status: GOOD ( 13.75 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org In SMP configurations, all TLB flushing narrower than flush_tlb_all() goes through __flush_tlb_range(). Do the same in UP configurations. This allows UP configurations to take advantage of recent improvements to the code in tlbflush.c, such as support for huge pages and flushing multiple-page ranges. Signed-off-by: Samuel Holland --- Changes in v2: - Move the SMP/UP merge earlier in the series to avoid build issues - Make a copy of __flush_tlb_range() instead of adding ifdefs inside - local_flush_tlb_all() is the only function used on !MMU (smpboot.c) arch/riscv/include/asm/tlbflush.h | 33 +++++++------------------------ arch/riscv/mm/Makefile | 5 +---- arch/riscv/mm/tlbflush.c | 13 ++++++++++++ 3 files changed, 21 insertions(+), 30 deletions(-) diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index 8f3418c5f172..317a1811aa51 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -27,13 +27,12 @@ static inline void local_flush_tlb_page(unsigned long addr) { ALT_FLUSH_TLB_PAGE(__asm__ __volatile__ ("sfence.vma %0" : : "r" (addr) : "memory")); } -#else /* CONFIG_MMU */ -#define local_flush_tlb_all() do { } while (0) -#define local_flush_tlb_page(addr) do { } while (0) -#endif /* CONFIG_MMU */ -#if defined(CONFIG_SMP) && defined(CONFIG_MMU) +#ifdef CONFIG_SMP void flush_tlb_all(void); +#else +#define flush_tlb_all() local_flush_tlb_all() +#endif void flush_tlb_mm(struct mm_struct *mm); void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, unsigned long end, unsigned int page_size); @@ -46,26 +45,8 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end); void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end); #endif -#else /* CONFIG_SMP && CONFIG_MMU */ - -#define flush_tlb_all() local_flush_tlb_all() -#define flush_tlb_page(vma, addr) local_flush_tlb_page(addr) - -static inline void flush_tlb_range(struct vm_area_struct *vma, - unsigned long start, unsigned long end) -{ - local_flush_tlb_all(); -} - -/* Flush a range of kernel pages */ -static inline void flush_tlb_kernel_range(unsigned long start, - unsigned long end) -{ - local_flush_tlb_all(); -} - -#define flush_tlb_mm(mm) flush_tlb_all() -#define flush_tlb_mm_range(mm, start, end, page_size) flush_tlb_all() -#endif /* !CONFIG_SMP || !CONFIG_MMU */ +#else /* CONFIG_MMU */ +#define local_flush_tlb_all() do { } while (0) +#endif /* CONFIG_MMU */ #endif /* _ASM_RISCV_TLBFLUSH_H */ diff --git a/arch/riscv/mm/Makefile b/arch/riscv/mm/Makefile index 9c454f90fd3d..64f901674e35 100644 --- a/arch/riscv/mm/Makefile +++ b/arch/riscv/mm/Makefile @@ -13,15 +13,12 @@ endif KCOV_INSTRUMENT_init.o := n obj-y += init.o -obj-$(CONFIG_MMU) += extable.o fault.o pageattr.o +obj-$(CONFIG_MMU) += extable.o fault.o pageattr.o tlbflush.o obj-y += cacheflush.o obj-y += context.o obj-y += pgtable.o obj-y += pmem.o -ifeq ($(CONFIG_MMU),y) -obj-$(CONFIG_SMP) += tlbflush.o -endif obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o obj-$(CONFIG_PTDUMP_CORE) += ptdump.o obj-$(CONFIG_KASAN) += kasan_init.o diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index e6659d7368b3..22d7ed5abf8e 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -66,6 +66,7 @@ static inline void local_flush_tlb_range_asid(unsigned long start, local_flush_tlb_range_threshold_asid(start, size, stride, asid); } +#ifdef CONFIG_SMP static void __ipi_flush_tlb_all(void *info) { local_flush_tlb_all(); @@ -138,6 +139,18 @@ static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, if (mm) put_cpu(); } +#else +static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, + unsigned long size, unsigned long stride) +{ + unsigned long asid = FLUSH_TLB_NO_ASID; + + if (mm && static_branch_unlikely(&use_asid_allocator)) + asid = atomic_long_read(&mm->context.id) & asid_mask; + + local_flush_tlb_range_asid(start, size, stride, asid); +} +#endif void flush_tlb_mm(struct mm_struct *mm) { From patchwork Sat Oct 28 23:12:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13439629 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B1D77C0018C for ; Sat, 28 Oct 2023 23:14:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Ke9uUiY/eXBNOC3zyRMy0bsdoypBLwQJezndPcpqsFE=; b=yyYFE9HFuS0hKp kjmH+wFzFmlYImZzxlbF1OqV/yaMt02ji6NNzq63r4IW10fYY6dm7CKwOt7Bd73BKpt/wCOnYYMu0 7tvF4K5fDJWtwzI9WRGNA6LOkmkAaXUgF48UjTT70bjuixkH8ANnl6nzWGDnDk9sev6zKptGjjJ7Q rmnj1VcNtkPMG7T59BE85AbOP29lzAY2jvqxN1DdGSdVAiNuj+WhbDOufBiXzq8pYGi8ajfVKe5Rk mLVQqsWHDf+VhEj803zlNtrI7k7JeOQ8HSCH72nbAipzSXtbrvmagC2P/WeGG4WVJJN5tcRF9rFyU 5KtOgR50IW9CG/N+bMqw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qwsVD-000va2-0q; Sat, 28 Oct 2023 23:13:59 +0000 Received: from mail-pj1-x1035.google.com ([2607:f8b0:4864:20::1035]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qwsV5-000vUH-0Q for linux-riscv@lists.infradead.org; Sat, 28 Oct 2023 23:13:54 +0000 Received: by mail-pj1-x1035.google.com with SMTP id 98e67ed59e1d1-2800bdf888dso1290874a91.1 for ; Sat, 28 Oct 2023 16:13:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1698534830; x=1699139630; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=UP8DHxv66mJzZKknfJMFXXbwrunx+Hl9Ii6vLteV5ng=; b=M69abNZ5UueVM84bI2TGc5h2ZfS1NC/rnvRzcJMI2f1QbXqX8Esp5YNMoCUo5SpZ00 iwALbgBW7mvfrcKq6QadP/Xf/ezTl8oZ6sE+VedzLraNB8tIVVrUnRrwr31JrbuEdAOM jcI8cZX/Au41L2qF8Kqq91A/4JQU/TG8jUQmtNlrmoVIQCvcrxO0X5jW8w26o6BCTfPt yCwo2mejRKTfMivWqKJn2aR3u6LFKtscbkq+MF45d60/WR/8i9HK5cdhFSxzBhuO9fzV zg1UmuxPa8guinOl5rVtYcwr90L191g/q+GmxK3U3IG/6HjH2JzVZkyjlkiV+9djiKlN Vz4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698534830; x=1699139630; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UP8DHxv66mJzZKknfJMFXXbwrunx+Hl9Ii6vLteV5ng=; b=Ps54MVKOBHTeOyBcGH8AtIbylZAjzTWNHdOq1HgarR3gUV1vdwAZ7VdnuelhQFobw/ oFWREtrOBYeJS2zH1qsy4TVyCSqBlFt7wx0jKGa2oegwGuIfRHCz+xN/Lvt32MOPvfp3 X3EYlgSIzipMnGWjL9PxMZuFqv+idIQYzw3dmYSUYmI0936dpneVEynWYjDgpK35SntS WkaW/oDA5KB8fkxYRXt0vqHDQCsA3mjR7V2CItdTe5IlCQ2JR4OkBX/0jBGdy8UaQXkN H4Sd4HYmpZahZDSn3WocTLJN8SGouURpEJ4uo1cxY1XcJqFrOSKuFfdhCgH5n1aIezj4 shlQ== X-Gm-Message-State: AOJu0YzYZyH9/ZPjoFcLwkyP8TWGsHzGY5/x3T6YTdHrD2Q5v9/5fOKb 5pTCbvWfxRkRoIr9L/z/+xJlpw== X-Google-Smtp-Source: AGHT+IGnkbRXw2gehdqQUYovjjCjlvZO0f8kuIzIEbkVVgufqh5Fzfqz26Y3uN2f14MpzFpb+9CFvQ== X-Received: by 2002:a17:90a:1506:b0:280:2a3f:9938 with SMTP id l6-20020a17090a150600b002802a3f9938mr2800689pja.12.1698534830039; Sat, 28 Oct 2023 16:13:50 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id u17-20020a17090341d100b001b8622c1ad2sm3679345ple.130.2023.10.28.16.13.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Oct 2023 16:13:49 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , Alexandre Ghiti , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Samuel Holland Subject: [PATCH v2 06/11] riscv: Apply SiFive CIP-1200 workaround to single-ASID sfence.vma Date: Sat, 28 Oct 2023 16:12:04 -0700 Message-ID: <20231028231339.3116618-7-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231028231339.3116618-1-samuel.holland@sifive.com> References: <20231028231339.3116618-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231028_161351_170270_1780A016 X-CRM114-Status: GOOD ( 14.05 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org commit 3f1e782998cd ("riscv: add ASID-based tlbflushing methods") added calls to the sfence.vma instruction with rs2 != x0. These single-ASID instruction variants are also affected by SiFive errata CIP-1200. Until now, the errata workaround was not needed for the single-ASID sfence.vma variants, because they were only used when the ASID allocator was enabled, and the affected SiFive platforms do not support multiple ASIDs. However, we are going to start using those sfence.vma variants regardless of ASID support, so now we need alternatives covering them. Signed-off-by: Samuel Holland --- Changes in v2: - Rebase on Alexandre's "riscv: tlb flush improvements" series v5 arch/riscv/include/asm/errata_list.h | 12 +++++++++++- arch/riscv/include/asm/tlbflush.h | 19 ++++++++++++++++++- arch/riscv/mm/tlbflush.c | 23 ----------------------- 3 files changed, 29 insertions(+), 25 deletions(-) diff --git a/arch/riscv/include/asm/errata_list.h b/arch/riscv/include/asm/errata_list.h index b55b434f0059..d3f3c237adad 100644 --- a/arch/riscv/include/asm/errata_list.h +++ b/arch/riscv/include/asm/errata_list.h @@ -44,11 +44,21 @@ ALTERNATIVE(__stringify(RISCV_PTR do_page_fault), \ CONFIG_ERRATA_SIFIVE_CIP_453) #else /* !__ASSEMBLY__ */ -#define ALT_FLUSH_TLB_PAGE(x) \ +#define ALT_SFENCE_VMA_ASID(asid) \ +asm(ALTERNATIVE("sfence.vma x0, %0", "sfence.vma", SIFIVE_VENDOR_ID, \ + ERRATA_SIFIVE_CIP_1200, CONFIG_ERRATA_SIFIVE_CIP_1200) \ + : : "r" (asid) : "memory") + +#define ALT_SFENCE_VMA_ADDR(addr) \ asm(ALTERNATIVE("sfence.vma %0", "sfence.vma", SIFIVE_VENDOR_ID, \ ERRATA_SIFIVE_CIP_1200, CONFIG_ERRATA_SIFIVE_CIP_1200) \ : : "r" (addr) : "memory") +#define ALT_SFENCE_VMA_ADDR_ASID(addr, asid) \ +asm(ALTERNATIVE("sfence.vma %0, %1", "sfence.vma", SIFIVE_VENDOR_ID, \ + ERRATA_SIFIVE_CIP_1200, CONFIG_ERRATA_SIFIVE_CIP_1200) \ + : : "r" (addr), "r" (asid) : "memory") + /* * _val is marked as "will be overwritten", so need to set it to 0 * in the default case. diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index 317a1811aa51..e529a643be17 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -22,10 +22,27 @@ static inline void local_flush_tlb_all(void) __asm__ __volatile__ ("sfence.vma" : : : "memory"); } +static inline void local_flush_tlb_all_asid(unsigned long asid) +{ + if (asid != FLUSH_TLB_NO_ASID) + ALT_SFENCE_VMA_ASID(asid); + else + local_flush_tlb_all(); +} + /* Flush one page from local TLB */ static inline void local_flush_tlb_page(unsigned long addr) { - ALT_FLUSH_TLB_PAGE(__asm__ __volatile__ ("sfence.vma %0" : : "r" (addr) : "memory")); + ALT_SFENCE_VMA_ADDR(addr); +} + +static inline void local_flush_tlb_page_asid(unsigned long addr, + unsigned long asid) +{ + if (asid != FLUSH_TLB_NO_ASID) + ALT_SFENCE_VMA_ADDR_ASID(addr, asid); + else + local_flush_tlb_page(addr); } #ifdef CONFIG_SMP diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 22d7ed5abf8e..0feccb8932d2 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -7,29 +7,6 @@ #include #include -static inline void local_flush_tlb_all_asid(unsigned long asid) -{ - if (asid != FLUSH_TLB_NO_ASID) - __asm__ __volatile__ ("sfence.vma x0, %0" - : - : "r" (asid) - : "memory"); - else - local_flush_tlb_all(); -} - -static inline void local_flush_tlb_page_asid(unsigned long addr, - unsigned long asid) -{ - if (asid != FLUSH_TLB_NO_ASID) - __asm__ __volatile__ ("sfence.vma %0, %1" - : - : "r" (addr), "r" (asid) - : "memory"); - else - local_flush_tlb_page(addr); -} - /* * Flush entire TLB if number of entries to be flushed is greater * than the threshold below. From patchwork Sat Oct 28 23:12:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13439628 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7F8F4C0018A for ; Sat, 28 Oct 2023 23:14:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=5XbaQwqFVbgzOHWqoTGF/RnM9IITJi/bPg/ei6CudIo=; b=iPSF5GGzq8Y/1v IzAjwdRlWpnN9UWFPxCyMr9oXpfhg1DjhXmRk1tgOoz0IZ8JzrF97xCtnR1CHC6ZmK7Rqs1qTSUiP reGQ0md1HTUWPiyfg2BCsmQw2KZdmi6gaRwwl+o+gu4MgMUMey+kUq8Kd9x9fbiexaLjQPcbFsgEF 1KF5btxrDV7Y3PBw65Bx1VB/WL+mGEE4mAEqLsTj0vhkxvlbbtJ1zxyJ4fcLomIDcd2vkOSJXZmuB jhN3BruhMhxv2aaXWMaI1EFDmbRdkm4JoxpZ4TxPBqm/+/FP2CW1FWhFCaIQWtoX4EORdBZuDUPIg SriVwGK0nNgFaoGRAnBg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qwsVE-000vaT-0T; Sat, 28 Oct 2023 23:14:00 +0000 Received: from mail-pf1-x430.google.com ([2607:f8b0:4864:20::430]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qwsV6-000vVW-1Y for linux-riscv@lists.infradead.org; Sat, 28 Oct 2023 23:13:54 +0000 Received: by mail-pf1-x430.google.com with SMTP id d2e1a72fcca58-6b1d1099a84so3256429b3a.1 for ; Sat, 28 Oct 2023 16:13:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1698534831; x=1699139631; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=uq0QdLYJN1f+TxJhAAAn9k5Zt34COvQfV/ogec/O0wM=; b=kCMp54Ga45P0QNBclINqmUNESsqz8NIfWZRnvRcHy26WNkaF/E41KUXwqbjYiy+GJn +gfC2+Z4YfG5c9QyHEf3khsfeKojt1kos+chpR8pL54WWSeFGKdIKXrBYKorr9qm235K +B8qSv41FA7kxv0+iOazuqm4mALw2nxM2TFMIFRDAwHnHudzLPEdd0xfObu/R6mB25Jt JPHo3eQNZVHZiqudpsWeCba7TUv7xgCzp8UBWrvFLdNY2v3B0fChpfZISZkurWDIz8CX rn9tzjcK9FJRB5o/nneFSwVvYB3NRh3OhldFOrlAmXLtgF/xDngoBaKe/JxpzwHn7dBR Idfg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698534831; x=1699139631; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=uq0QdLYJN1f+TxJhAAAn9k5Zt34COvQfV/ogec/O0wM=; b=a0bHgKX28+s4bf+eZg7bx/AeZDuX0tkkmva3aKm9HzFrq12oEu909N45pmw8xbApMT aEnArZY+uzCxyu1yR4SXB2glK9sYE/4p6yYsScVAwqjv52mdYa5IIP3Xf9rgdqrVdcv0 xxykCn0LS3Z05ThVN1tQoKyyG6+eytxekuEdT58jaVRWzWUet+axjQQLlo4Nbx88lRz4 7tp63bl4ndif5m/Bv4V9//sBEJoLh9JZKf8Uu4fcIrNjwnXb7nYLOclUv+HHQPgUDsXK wv2ThRq8KufQm1aaJfJtcJjYtDHAQ6tuDLeqGO8/irKxQNu3C5znijBgopb98NDrZ82p h10A== X-Gm-Message-State: AOJu0YwoPhIeZBDfujpJpfF+eb/DEPfPi8/7vxMvOpkwDCiH/pKfLI7r jCNg7gFjhxndAFWLIFj/qlXmBg== X-Google-Smtp-Source: AGHT+IGqJfNl+X6UXX6MsfY+UkH1b50Ad4rlkkQrptVRUTaYY/ZhDSmvjBpzXH88ANRSqVScovaqNA== X-Received: by 2002:a17:902:6b8c:b0:1c9:bf14:4f94 with SMTP id p12-20020a1709026b8c00b001c9bf144f94mr6530418plk.44.1698534831309; Sat, 28 Oct 2023 16:13:51 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id u17-20020a17090341d100b001b8622c1ad2sm3679345ple.130.2023.10.28.16.13.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Oct 2023 16:13:50 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , Alexandre Ghiti , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Samuel Holland Subject: [PATCH v2 07/11] riscv: mm: Introduce cntx2asid/cntx2version helper macros Date: Sat, 28 Oct 2023 16:12:05 -0700 Message-ID: <20231028231339.3116618-8-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231028231339.3116618-1-samuel.holland@sifive.com> References: <20231028231339.3116618-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231028_161352_551189_D6FDFB20 X-CRM114-Status: GOOD ( 15.36 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org When using the ASID allocator, the MM context ID contains two values: the ASID in the lower bits, and the allocator version number in the remaining bits. Use macros to make this separation more obvious. Signed-off-by: Samuel Holland --- (no changes since v1) arch/riscv/include/asm/mmu.h | 3 +++ arch/riscv/mm/context.c | 12 ++++++------ arch/riscv/mm/tlbflush.c | 4 ++-- 3 files changed, 11 insertions(+), 8 deletions(-) diff --git a/arch/riscv/include/asm/mmu.h b/arch/riscv/include/asm/mmu.h index 355504b37f8e..a550fbf770be 100644 --- a/arch/riscv/include/asm/mmu.h +++ b/arch/riscv/include/asm/mmu.h @@ -26,6 +26,9 @@ typedef struct { #endif } mm_context_t; +#define cntx2asid(cntx) ((cntx) & asid_mask) +#define cntx2version(cntx) ((cntx) & ~asid_mask) + void __init create_pgd_mapping(pgd_t *pgdp, uintptr_t va, phys_addr_t pa, phys_addr_t sz, pgprot_t prot); #endif /* __ASSEMBLY__ */ diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index 217fd4de6134..43d005f63253 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -81,7 +81,7 @@ static void __flush_context(void) if (cntx == 0) cntx = per_cpu(reserved_context, i); - __set_bit(cntx & asid_mask, context_asid_map); + __set_bit(cntx2asid(cntx), context_asid_map); per_cpu(reserved_context, i) = cntx; } @@ -102,7 +102,7 @@ static unsigned long __new_context(struct mm_struct *mm) lockdep_assert_held(&context_lock); if (cntx != 0) { - unsigned long newcntx = ver | (cntx & asid_mask); + unsigned long newcntx = ver | cntx2asid(cntx); /* * If our current CONTEXT was active during a rollover, we @@ -115,7 +115,7 @@ static unsigned long __new_context(struct mm_struct *mm) * We had a valid CONTEXT in a previous life, so try to * re-use it if possible. */ - if (!__test_and_set_bit(cntx & asid_mask, context_asid_map)) + if (!__test_and_set_bit(cntx2asid(cntx), context_asid_map)) return newcntx; } @@ -168,7 +168,7 @@ static void set_mm_asid(struct mm_struct *mm, unsigned int cpu) */ old_active_cntx = atomic_long_read(&per_cpu(active_context, cpu)); if (old_active_cntx && - ((cntx & ~asid_mask) == atomic_long_read(¤t_version)) && + (cntx2version(cntx) == atomic_long_read(¤t_version)) && atomic_long_cmpxchg_relaxed(&per_cpu(active_context, cpu), old_active_cntx, cntx)) goto switch_mm_fast; @@ -177,7 +177,7 @@ static void set_mm_asid(struct mm_struct *mm, unsigned int cpu) /* Check that our ASID belongs to the current_version. */ cntx = atomic_long_read(&mm->context.id); - if ((cntx & ~asid_mask) != atomic_long_read(¤t_version)) { + if (cntx2version(cntx) != atomic_long_read(¤t_version)) { cntx = __new_context(mm); atomic_long_set(&mm->context.id, cntx); } @@ -191,7 +191,7 @@ static void set_mm_asid(struct mm_struct *mm, unsigned int cpu) switch_mm_fast: csr_write(CSR_SATP, virt_to_pfn(mm->pgd) | - ((cntx & asid_mask) << SATP_ASID_SHIFT) | + (cntx2asid(cntx) << SATP_ASID_SHIFT) | satp_mode); if (need_flush_tlb) diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 0feccb8932d2..1cfac683bda4 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -91,7 +91,7 @@ static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, broadcast = cpumask_any_but(cmask, cpuid) < nr_cpu_ids; if (static_branch_unlikely(&use_asid_allocator)) - asid = atomic_long_read(&mm->context.id) & asid_mask; + asid = cntx2asid(atomic_long_read(&mm->context.id)); } else { cmask = cpu_online_mask; broadcast = true; @@ -123,7 +123,7 @@ static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, unsigned long asid = FLUSH_TLB_NO_ASID; if (mm && static_branch_unlikely(&use_asid_allocator)) - asid = atomic_long_read(&mm->context.id) & asid_mask; + asid = cntx2asid(atomic_long_read(&mm->context.id)); local_flush_tlb_range_asid(start, size, stride, asid); } From patchwork Sat Oct 28 23:12:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13439627 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A13FEC00141 for ; Sat, 28 Oct 2023 23:14:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=AJrjgH1YuiSR3F8HtSCWFvLnLkpd69H6BapeNyndp2Y=; b=iLTTP/RjYcBqD+ 4TaDE5+szWRXcwZGAbPkkbqLyY/6zsqXkAGNgc2eQfaOZM1qIuwdvi2rN6SQtRh6fBYXDgSbA55y4 Y0sHqSBcVQZ47V4N9x8NjMA5awiceJqy2X92f+oVBP9AKNN/RrkjlXY/9WBjBE0jsHA9TJjC6pA96 39oCCY4Jfd3IyYEtZai/dQLgPWjP7+ucBnF3Eg9MwrwQEbSJJs86fV70q0S6wartfxIwYUX1L/hm3 Vyr+zM6gqI/Afxk5vYu4MzPDbe687Qaa15joc/jz7BtYgx0uEf3VS3RQTrLmwXaAV6Ovd1zFsx39P JSW7PQPo9IfI5blK9iyA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qwsVE-000vb6-33; Sat, 28 Oct 2023 23:14:00 +0000 Received: from mail-ot1-x32f.google.com ([2607:f8b0:4864:20::32f]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qwsV8-000vWg-20 for linux-riscv@lists.infradead.org; Sat, 28 Oct 2023 23:13:56 +0000 Received: by mail-ot1-x32f.google.com with SMTP id 46e09a7af769-6ce2eaf7c2bso2247886a34.0 for ; Sat, 28 Oct 2023 16:13:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1698534833; x=1699139633; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=nfPkHia9ku1mIe8MIuwK1ts/4i7vxeqzbyNHZF4YvJg=; b=ADQhDClpmfmPY8rmsgvIxYRNZYK5xa6FgVkOd/Z4WdjS9DQOUd2K17VjS7Al3Py5zo q1TOuPzfxMXZmJtMVRSdMOkyTKBaH/24xgMAWIehYVma6pWUXiu+B+G6jj8YgtqGrDNp nUfVQZ6kPepSsb6b1+UqP1EuVnHJExVDbt90JqKT4WntsmPRtabPyr4CUpJA+aNBlbuo XTSC2lnahGI9HZomJ7H7rH4Rpv8pouq2xUQ+PzUBps/V5g3BrfpG126dr4iMD3EA5JJC e3L1j7CMfJAnP3Ccydp1xyNZYwzXkE66olxB/SlLChP/Mx3PKNApyBBlQ/C0c9YA2Z5E cSHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698534833; x=1699139633; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nfPkHia9ku1mIe8MIuwK1ts/4i7vxeqzbyNHZF4YvJg=; b=TSYwVT8LEbnBONvHc6TPhEwxiTGygTPPqpo01c0j7Yqz/VdcuXEUsO60/IEiJEOz7B 8l/uqWY2evYYAXSV13AE2BcqR8cOUT+m6xa/+UHLLE9yI5CIKAt6i5ncJa6iII9huuvV mWY2knNnPxu3uQDcFKLYrrqI1+fU/gcBxNNkM2uRz3ZVnhQkW0rjir+dCZJoRkG5U0qY HFCaS6XTQF0VvX12St4hKXmQXNaPATNDh2xNsRDNRAoGa3UtThuvSos0zh03nsHB0mya faZSoBhOom9tV84cy4qmpCFBQ/tfdnUzI1KrtLyitc0NBiCyUvkFXQLACd3m4g8FPu1E IweQ== X-Gm-Message-State: AOJu0YxVKVdqoiF56QiZmaOwWhDxuaN9cUdWIylnZWPTBGsfIGx/lFxR stmpIa658i4lgLYENS3CIxUIUQ== X-Google-Smtp-Source: AGHT+IERITWcnpZawWSuuFpc/cjj/Gs86LMrNqv2CdnNpOWafKnnt4wsBVVnxZL60abLMgaX5a0ZCA== X-Received: by 2002:a05:6830:1e30:b0:6bd:c7c3:aac2 with SMTP id t16-20020a0568301e3000b006bdc7c3aac2mr6567117otr.18.1698534832734; Sat, 28 Oct 2023 16:13:52 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id u17-20020a17090341d100b001b8622c1ad2sm3679345ple.130.2023.10.28.16.13.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Oct 2023 16:13:52 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , Alexandre Ghiti , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Samuel Holland Subject: [PATCH v2 08/11] riscv: mm: Use a fixed layout for the MM context ID Date: Sat, 28 Oct 2023 16:12:06 -0700 Message-ID: <20231028231339.3116618-9-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231028231339.3116618-1-samuel.holland@sifive.com> References: <20231028231339.3116618-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231028_161354_672955_C2F779BA X-CRM114-Status: GOOD ( 12.08 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Currently, the size of the ASID field in the MM context ID dynamically depends on the number of hardware-supported ASID bits. This requires reading a global variable to extract either field from the context ID. Instead, allocate the maximum possible number of bits to the ASID field, so the layout of the context ID is known at compile-time. Signed-off-by: Samuel Holland --- (no changes since v1) arch/riscv/include/asm/mmu.h | 4 ++-- arch/riscv/include/asm/tlbflush.h | 2 -- arch/riscv/mm/context.c | 6 ++---- 3 files changed, 4 insertions(+), 8 deletions(-) diff --git a/arch/riscv/include/asm/mmu.h b/arch/riscv/include/asm/mmu.h index a550fbf770be..dc0273f7905f 100644 --- a/arch/riscv/include/asm/mmu.h +++ b/arch/riscv/include/asm/mmu.h @@ -26,8 +26,8 @@ typedef struct { #endif } mm_context_t; -#define cntx2asid(cntx) ((cntx) & asid_mask) -#define cntx2version(cntx) ((cntx) & ~asid_mask) +#define cntx2asid(cntx) ((cntx) & SATP_ASID_MASK) +#define cntx2version(cntx) ((cntx) & ~SATP_ASID_MASK) void __init create_pgd_mapping(pgd_t *pgdp, uintptr_t va, phys_addr_t pa, phys_addr_t sz, pgprot_t prot); diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index e529a643be17..62d780037169 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -15,8 +15,6 @@ #define FLUSH_TLB_NO_ASID ((unsigned long)-1) #ifdef CONFIG_MMU -extern unsigned long asid_mask; - static inline void local_flush_tlb_all(void) { __asm__ __volatile__ ("sfence.vma" : : : "memory"); diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index 43d005f63253..b5170ac1b742 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -22,7 +22,6 @@ DEFINE_STATIC_KEY_FALSE(use_asid_allocator); static unsigned long asid_bits; static unsigned long num_asids; -unsigned long asid_mask; static atomic_long_t current_version; @@ -128,7 +127,7 @@ static unsigned long __new_context(struct mm_struct *mm) goto set_asid; /* We're out of ASIDs, so increment current_version */ - ver = atomic_long_add_return_relaxed(num_asids, ¤t_version); + ver = atomic_long_add_return_relaxed(BIT(SATP_ASID_BITS), ¤t_version); /* Flush everything */ __flush_context(); @@ -247,7 +246,6 @@ static int __init asids_init(void) /* Pre-compute ASID details */ if (asid_bits) { num_asids = 1 << asid_bits; - asid_mask = num_asids - 1; } /* @@ -255,7 +253,7 @@ static int __init asids_init(void) * at-least twice more than CPUs */ if (num_asids > (2 * num_possible_cpus())) { - atomic_long_set(¤t_version, num_asids); + atomic_long_set(¤t_version, BIT(SATP_ASID_BITS)); context_asid_map = bitmap_zalloc(num_asids, GFP_KERNEL); if (!context_asid_map) From patchwork Sat Oct 28 23:12:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13439626 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B1DECC3DA79 for ; Sat, 28 Oct 2023 23:14:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=n4FUUJWMeQ50MhgT9WfVWFZlC3MyJ9tf2sNwhOkJThc=; b=OS8X/sC3ZAqCuu XSn6ZWXWFxr7Rl2BmL5fcBikoaGx7XnW5KeHKqV/ABCvxk0my6fBSRmkFLF9A4RLFy0SbmHRcaxud PCXAdsHPqdUDHwRz//+I5JnL5FOokaDpU6Q1ljDA3b0cU8xZMcjo4s9A8jsBkeX+Y8o9bQoEA4hZA eSYKaRDQ6C9c3j1jyT2CaAVxZfyYzH87GfQwRWJjPlyRFNHioC91TAXFhrFKwmyDD0nD0azJKwpoD fsk73VhTgX/UT73cvA9uKHMZovSqAiNg/wOAorwhdVylYqhn+IHLq2TYNDKmEnFJIkjj8vIyF6Kr/ yvpoVOG6QWIJWx5Pb29Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qwsVF-000vbk-28; Sat, 28 Oct 2023 23:14:01 +0000 Received: from mail-pl1-x630.google.com ([2607:f8b0:4864:20::630]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qwsV9-000vXC-0d for linux-riscv@lists.infradead.org; Sat, 28 Oct 2023 23:13:56 +0000 Received: by mail-pl1-x630.google.com with SMTP id d9443c01a7336-1c9b95943beso29440595ad.1 for ; Sat, 28 Oct 2023 16:13:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1698534834; x=1699139634; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=O8uQuAx5HbeGrqX4Ip/qQY7nv6lrxw4qacCJFbpKbko=; b=LWxVbkUIg8y4LRIwzqeIoUAxIoQIi5kPTMQAOYQzNNtkWDLItXKlPehjOpgBlBdahr Hm6cojJOjG51A5iuTqR74G9REIiNlHP8RMap47QcMVIp3f2bs2IH/OnePnlxLoKu74OM EzCD41UCfSh3hLxnREghTJ9XrUB+OqIhskf+xIvwU0ngRXq5ioDw+n1DCyJjL2L3aZis JOO5uX9MWfoejvppKnSq40O1TiX9xdp/WL9pF2dSrpwfn4RMHwf5tjsImWhvli657Tdz yXPDWff0y2dnREzZ3keVJXbR0YbQbs6JrM1sSrjZ2Jbzxmc+4LnliJCjTglakIC9rsoa s6Kw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698534834; x=1699139634; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=O8uQuAx5HbeGrqX4Ip/qQY7nv6lrxw4qacCJFbpKbko=; b=D4GvIFwRDUE1qzk7JN2o6cFV2qYoH3TxSa6bBfFPe1dCNSKTKrqZNVKVOlj8sItpy1 xdgeM8yBpCvegPOho3bwOA0FHFZ1CRoYQbk1hxiIW98CjMM/q6+VCpIK2KQ2Z2lvl9/V STBiAJvuH9aXTiH072ibmU7BVweaHyoTyLYbK3g2Lo0n0PzDqV4k5z9cOr4zQdkF1pw2 wi7zhWMDAQBan5DA/iM4HKlbhZORa4yiMCER5Yi1rwuqL7aj7vC8qBHn65r1u3T9kGHR rYvDm88gggwwPW0uQ34PE+frzNH/ZPj2j7xJYJGDQXyDOGnp0skdYGlMmBRDLnXtXH8L pM4w== X-Gm-Message-State: AOJu0YzSCS9FUCZQSTjSjstj4YoHaVYmXl9a+aqDxkv8rHcWRTNHpMhV B8fOB8Wb/nb4c2vL5iCUqo54nxdnBgafGp4qJj0= X-Google-Smtp-Source: AGHT+IEXxDKHTizsqyBXjr5oqt/mWPSp5qFgTjueov73uPEcJh4MjB79gaQaqVj5Le1ZHH1qO9h3pA== X-Received: by 2002:a17:902:c751:b0:1cc:29ef:df7d with SMTP id q17-20020a170902c75100b001cc29efdf7dmr3951181plq.65.1698534834011; Sat, 28 Oct 2023 16:13:54 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id u17-20020a17090341d100b001b8622c1ad2sm3679345ple.130.2023.10.28.16.13.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Oct 2023 16:13:53 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , Alexandre Ghiti , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Samuel Holland Subject: [PATCH v2 09/11] riscv: mm: Make asid_bits a local variable Date: Sat, 28 Oct 2023 16:12:07 -0700 Message-ID: <20231028231339.3116618-10-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231028231339.3116618-1-samuel.holland@sifive.com> References: <20231028231339.3116618-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231028_161355_233208_B72BCAC1 X-CRM114-Status: UNSURE ( 8.62 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org This variable is only used inside asids_init(). Signed-off-by: Samuel Holland --- (no changes since v1) arch/riscv/mm/context.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index b5170ac1b742..43a8bc2d5af4 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -20,7 +20,6 @@ DEFINE_STATIC_KEY_FALSE(use_asid_allocator); -static unsigned long asid_bits; static unsigned long num_asids; static atomic_long_t current_version; @@ -226,7 +225,7 @@ static inline void set_mm(struct mm_struct *prev, static int __init asids_init(void) { - unsigned long old; + unsigned long asid_bits, old; /* Figure-out number of ASID bits in HW */ old = csr_read(CSR_SATP); From patchwork Sat Oct 28 23:12:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13439630 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 23F55C4167B for ; Sat, 28 Oct 2023 23:14:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Li/VAFYbscj6f9+EV6EJIvnlY67sDrtoWicRX6372nQ=; b=X4Bo/SomXEONMh VP17Z03U4wZ4AmzN+Fc+TZfoaKVhSbMUipTvNeQvc5FdaqFrR6Zs0w3DeEbH+8dUCVWym4MRVnwkt TVwuN4m379z0/uddDLdQ5RUvgeBE4xqUesbGqk4YoygPHU/poMBvOKVE9DqvqD85KKXZa8YtPcCm5 Iw5BwisQSyUAzyyqpG7qyYYVcIXUPBW5Ro3eLG877ikb6ciMtkGmxXgHrSPH0L8+eoUylJkSOi+LM +Yj36vpH12KTq/1vjKNYp8eq+8EkcRDCx+a3WYkqApbobbNor+Z6dTCy8fcsgFl5OEqHfFv8wkS79 1TCNaVl+Gvw/2GQEKoxQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qwsVP-000vjh-2N; Sat, 28 Oct 2023 23:14:11 +0000 Received: from mail-oo1-xc32.google.com ([2607:f8b0:4864:20::c32]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qwsVA-000vY2-2z for linux-riscv@lists.infradead.org; Sat, 28 Oct 2023 23:13:58 +0000 Received: by mail-oo1-xc32.google.com with SMTP id 006d021491bc7-5849fc56c62so2063451eaf.3 for ; Sat, 28 Oct 2023 16:13:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1698534835; x=1699139635; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MROdnZxFJAn7brM6ye/jIKPJcvVCbRqlgsjUSxtP0PY=; b=cN5KGH3zyV/wkUFHz6vq7alGBC1izKw+RcEdHAX0laRx/fJ4lHSseYB4q5vvavUR/X Gba+yMSZhpd5j/Z+MCXx0iS7s4lAWGSAQMsoUT3kBA9bVb1867SIaWboKNwxbcSjkDvn ysZRnXhKuGWgO86kZ2UAry8taZ8PTdd9JD7mxlivIX5M+i76NVR9rdNCbg4pU1T9GGUq Za+dYUa25u8OLoVDYl4wZ6hz6B3lVYJw38pC+szfMCzbeX2+Vz8jQprBOeeFWxIB+24z jd1neXTtjdmzg8/9fEH0+Kd8sqQk57DgM9Uaq91Zl4zKzYAUImPmsVNrrI9MlvCw8hdu AMXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698534835; x=1699139635; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MROdnZxFJAn7brM6ye/jIKPJcvVCbRqlgsjUSxtP0PY=; b=OZY/o2m87noVEtGM7xH9Ih2U81YXbMhbuRgdhfKsWUT18meVFSm8TV9RZ97x0bsPAV p3oVx3rl6QpcGM7qhOmDc0WAuy7IIR+7IJgzfROTM10PspGbz+TaQ1rrWfWOcllqtFyY hpwpChIsDbCxNvIV345BM89zFm2TvFf36RT3Jv3/AssQCYMMMzxfRi870MD66qkZwXXN JQ1BPQRcO3mDnw93MNW6uKpuVSlqb0WAWR5X5febefymqne0t1KgW13nJsz075ycm6/H PcZ3I4agJjpsWZNWJ2nXirTYFOQjhq/J4EL2eln5pK0gv967bAEjSjzvVZhCZ47Y0hya OePQ== X-Gm-Message-State: AOJu0YyzjQ8vMvseN39SPeUJyIgCGEN9JkB0gBeKAjOzMw9YO1ba+Ifr 2rd2hhEw1zrkH7v5ces+pN6HwA== X-Google-Smtp-Source: AGHT+IEpjr2TCXm5am55hbUBmsnjcw94CHvXcnlavbn7wyDrZYS6wclRzO1RWsfOFMjAji60OjxQDg== X-Received: by 2002:a05:6358:5e08:b0:168:d6cd:7b2e with SMTP id q8-20020a0563585e0800b00168d6cd7b2emr6151237rwn.29.1698534835540; Sat, 28 Oct 2023 16:13:55 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id u17-20020a17090341d100b001b8622c1ad2sm3679345ple.130.2023.10.28.16.13.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Oct 2023 16:13:54 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , Alexandre Ghiti , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Samuel Holland Subject: [PATCH v2 10/11] riscv: mm: Preserve global TLB entries when switching contexts Date: Sat, 28 Oct 2023 16:12:08 -0700 Message-ID: <20231028231339.3116618-11-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231028231339.3116618-1-samuel.holland@sifive.com> References: <20231028231339.3116618-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231028_161356_963992_9570B58E X-CRM114-Status: GOOD ( 14.38 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org If the CPU does not support multiple ASIDs, all MM contexts use ASID 0. In this case, it is still beneficial to flush the TLB by ASID, as the single-ASID variant of the sfence.vma instruction preserves TLB entries for global (kernel) pages. This optimization is recommended by the RISC-V privileged specification: If the implementation does not provide ASIDs, or software chooses to always use ASID 0, then after every satp write, software should execute SFENCE.VMA with rs1=x0. In the common case that no global translations have been modified, rs2 should be set to a register other than x0 but which contains the value zero, so that global translations are not flushed. It is not possible to apply this optimization when using the ASID allocator, because that code must flush the TLB for all ASIDs at once when incrementing the version number. Signed-off-by: Samuel Holland --- (no changes since v1) arch/riscv/mm/context.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index 43a8bc2d5af4..3ca9b653df7d 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -200,7 +200,7 @@ static void set_mm_noasid(struct mm_struct *mm) { /* Switch the page table and blindly nuke entire local TLB */ csr_write(CSR_SATP, virt_to_pfn(mm->pgd) | satp_mode); - local_flush_tlb_all(); + local_flush_tlb_all_asid(0); } static inline void set_mm(struct mm_struct *prev, From patchwork Sat Oct 28 23:12:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13439631 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 78402C4167D for ; Sat, 28 Oct 2023 23:14:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Fk647J3C9xovr+MO8HH24TmMsBdXSGzUlCAMCGH1VaY=; b=OuVxvq1CWXfSDG 6S/nNZ/rj3wXE5cH0Ho9U0vTozYRNNgU+gSjU832e1D/uXzhHfdDIbhoFxtgCeHsfNkX0IwRShFhW 0nZLndeEZRwaA3BHkzqtwdhtqZ+9twXqeYPu5ig23aTEJxtAHpFBhTKGjyQjL+IE0tlVtNdCwxKZ0 56doIBcNti1BsL8eUQGw1VyROBNuuQj9eAQyeYFE0P+sdcf9GNaKXG+UnpcgYC8a2aHWS/Ja4arss /Pm61PWeTSZX47RXsYxxFUShycl0Iy/i3XVHXk0ATxXv970/jxguRvM8foMI9SONi024Kuqr6OVrp rQdf9sSfgPRL5P70Ki0g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qwsVQ-000vkH-1k; Sat, 28 Oct 2023 23:14:12 +0000 Received: from mail-pl1-x632.google.com ([2607:f8b0:4864:20::632]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qwsVC-000vYd-03 for linux-riscv@lists.infradead.org; Sat, 28 Oct 2023 23:13:59 +0000 Received: by mail-pl1-x632.google.com with SMTP id d9443c01a7336-1cc329ce84cso6296765ad.2 for ; Sat, 28 Oct 2023 16:13:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1698534837; x=1699139637; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RYq2JBnvfDkZMtFq+gFMcak9lyRAM53Oqx2LdQZI3vI=; b=Lqjk+0rB3Y+3RxX58KWbLyCBtFZcvwffMxxQaMzl1Wgt8udLA57DUQqhArnphsdZn2 K25W6sq11+bKunr+8kdqw13y+yhWzLT8zqaZm/mGdi+WPutn1tizOUAQxjM4+yya0vXY HnDEjeQYYQ4JT/5Eiv3gghwp5bL4jCN3ZYMkt2IOYfrfUGu1nNFRm+Zusjz9r6mDHRDB SdGiiZI31IOx0/0PwgP6OriLfOQjiKL0YZaetSyEl/flARow2eJpQT80CxkqfM/4wD8l 8nbsHCWsrF4tSdXperJWlu8jDVGMsKosV/FxTj05xPWOPbTZUrFoTv6WNJzXnFCAzVlM d4Ow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698534837; x=1699139637; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RYq2JBnvfDkZMtFq+gFMcak9lyRAM53Oqx2LdQZI3vI=; b=o+VLJ6tyBUNtcjilnF4amqXiMcsX+zaMAWf+u0kzBuJC9SKljtkjLC+fJSp93wdYuF h4pY8UOu9y2Kel9Uxjwx2DuwfDzmNKBga7QtUz0Rc1LzErFdHgdUumRw6QPekXsro+6P Q/P/F8zHwpA3VwUigG2G42495v5a8PIeTfZ49cEUmz9ufMcgfBKLfp1taV8obCBIOFZv vKuX0+0LWPNvqUklLEo+Bxco3rhJEEJ+jeNLZK5yI+o7pjx1eDKjXBTAHj0GI4/wgQ4y cCisuatbHBkWT8Zdc/OO4mvWmGnbTaDnMRS76idLqRIw1FJdLXHu0jVbNWHCVAtoIcbM 5YCA== X-Gm-Message-State: AOJu0YyGZk9FIUIjKCAONO128bstWPj/PPYk5MLlGCc7D8KKnjvxxRMA WOksOBrXTDzjFmVZp950z59M/w== X-Google-Smtp-Source: AGHT+IHR5gJkl+2hmALdgeC6Mspb2OIhkdCFo9rAzDT7R89li8y2sxi8zbkov4fwyI0EL6o8dg+K/w== X-Received: by 2002:a17:902:f542:b0:1c5:a7b7:291c with SMTP id h2-20020a170902f54200b001c5a7b7291cmr8203243plf.12.1698534836728; Sat, 28 Oct 2023 16:13:56 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id u17-20020a17090341d100b001b8622c1ad2sm3679345ple.130.2023.10.28.16.13.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Oct 2023 16:13:56 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , Alexandre Ghiti , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Samuel Holland Subject: [PATCH v2 11/11] riscv: mm: Always use ASID to flush MM contexts Date: Sat, 28 Oct 2023 16:12:09 -0700 Message-ID: <20231028231339.3116618-12-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231028231339.3116618-1-samuel.holland@sifive.com> References: <20231028231339.3116618-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231028_161358_087697_82B9EBC5 X-CRM114-Status: GOOD ( 12.34 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Even if multiple ASIDs are not supported, using the single-ASID variant of the sfence.vma instruction preserves TLB entries for global (kernel) pages. So it is always most efficient to use the single-ASID code path. Signed-off-by: Samuel Holland --- Changes in v2: - Update both copies of __flush_tlb_range() arch/riscv/include/asm/mmu_context.h | 2 -- arch/riscv/mm/context.c | 3 +-- arch/riscv/mm/tlbflush.c | 5 ++--- 3 files changed, 3 insertions(+), 7 deletions(-) diff --git a/arch/riscv/include/asm/mmu_context.h b/arch/riscv/include/asm/mmu_context.h index 7030837adc1a..b0659413a080 100644 --- a/arch/riscv/include/asm/mmu_context.h +++ b/arch/riscv/include/asm/mmu_context.h @@ -33,8 +33,6 @@ static inline int init_new_context(struct task_struct *tsk, return 0; } -DECLARE_STATIC_KEY_FALSE(use_asid_allocator); - #include #endif /* _ASM_RISCV_MMU_CONTEXT_H */ diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index 3ca9b653df7d..20057085ab8a 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -18,8 +18,7 @@ #ifdef CONFIG_MMU -DEFINE_STATIC_KEY_FALSE(use_asid_allocator); - +static DEFINE_STATIC_KEY_FALSE(use_asid_allocator); static unsigned long num_asids; static atomic_long_t current_version; diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 1cfac683bda4..9d06a3e9d330 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -90,8 +90,7 @@ static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, /* check if the tlbflush needs to be sent to other CPUs */ broadcast = cpumask_any_but(cmask, cpuid) < nr_cpu_ids; - if (static_branch_unlikely(&use_asid_allocator)) - asid = cntx2asid(atomic_long_read(&mm->context.id)); + asid = cntx2asid(atomic_long_read(&mm->context.id)); } else { cmask = cpu_online_mask; broadcast = true; @@ -122,7 +121,7 @@ static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, { unsigned long asid = FLUSH_TLB_NO_ASID; - if (mm && static_branch_unlikely(&use_asid_allocator)) + if (mm) asid = cntx2asid(atomic_long_read(&mm->context.id)); local_flush_tlb_range_asid(start, size, stride, asid);