From patchwork Wed Mar 27 04:49:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13605565 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3D0CDC47DD9 for ; Wed, 27 Mar 2024 04:51:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=tu5VeRxAcBXL+A7zvKZWAWq/LOErTxqYVBjqZCPPAy0=; b=iaBapHhMX0jL6w 0slL16C475econOp9PvPOIQoOkw4cjQFYY8svifYlXn15Gwqs6QesB0ZWfpbugC1EZWd1ToIJBFkD 7ZGP9egafNwQxkjf2IcXWL+VoPr3Amy8wahgMB4VDXXhH1yQey8SZbJvZ4pl4dmnsOlm2WXG6rtDV FkDaDy2bvP1PzJzCB3SODT5KBHNSdGuRNRWhqh4HHvHLZkOlPAggZEsRmV8Oun1QD22QTkwsfasUQ kftoil84/TTByTeFLAUVMg1vlf/W7d9GXwPK40zH8gb5WolpPX6yADRfMxg6CgRKigDWGxqcffQyD m7/orb5g5vZZKPNeSocw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpLFR-00000007S5Y-1Wfw; Wed, 27 Mar 2024 04:50:49 +0000 Received: from mail-pf1-x436.google.com ([2607:f8b0:4864:20::436]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpLFM-00000007S22-0tPW for linux-riscv@lists.infradead.org; Wed, 27 Mar 2024 04:50:45 +0000 Received: by mail-pf1-x436.google.com with SMTP id d2e1a72fcca58-6ea8ee55812so3848592b3a.0 for ; Tue, 26 Mar 2024 21:50:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1711515039; x=1712119839; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=shQKeKcQeNJUU/JQ5KWmpXdfmdrhkhe16Q2MMRo9oOg=; b=M+2Ek6SUbsjhve0zlKtSKCsQFEFr4YxFie/LxO9WwQVwaTHHo46xt34FR7SGELq37Z QUS50fjtiid197YvyaoEyWWPlEdgWftMNPx64lWNf8nu6PaoqhwIjpYY76LQoW/RvAvU 9nqwNURqHReY71WFVO5k7XnITYIZvL0gXhRmDz19ZXc83CRF6lzFh1LPGzZhiYgRlcnV TU52izChvtHVXF9EHNdWYOU9h/xJO2ythCFjH/WVtGk1dhoES3yoTJZvKVkKJdRvXljG YtBXAjeTYCmlLkLbbhAzg4x46BIsI2Ad11ia1fTbk7HNRD94I21LsuIbjeZVUvSw9Rgj VkpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711515039; x=1712119839; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=shQKeKcQeNJUU/JQ5KWmpXdfmdrhkhe16Q2MMRo9oOg=; b=bzr4e06vZ1IoTXnrOTbBcEf31jfthPOeMHG21bCxT0qA2QsRLkaUSKWKEuX5fDLPG3 4URQrFlAbchA7DGPG9YYYJ8z7CWMJrf1Nlqfrrj6ZIm9ZJqskAMyRiXRPrmL6vL3e7ho DgDbImwOV2CnnDyTmDYmwn2uEom+LpwYRmj31xgQMX6Oe6XNnNExGe5Cb007zehxccYN erFcSqIibF+0hz+zVcwsSHyc6xa48PmO7cenmsLg7R+uZUlGO/lXVMVZJFu2Qj8kS4RX twRNtr+cuy8V1Xwao78VghvMncrYRte1seaFk3ubgWRPAh0/jJ4ZN9TiBrzyT6f0HMyo Aceg== X-Forwarded-Encrypted: i=1; AJvYcCXUAQR5XTny7hiCIEVjj5JvJJyyHBOHDs6dl4HhjOWyFd3LTYmkXh7LILJC8v6Z3jAZsjwJgRmL3JD7DExbYM4h35T7llsL1xIqJOyABEeW X-Gm-Message-State: AOJu0YyrnhfKTSvuDQ7RB/ytfnnQA/rRMB4FZg320W5xA7GRsQiFXTqt WcemMM8iTtP7Noo7dXALBDbATPUHA42brUx5sM99AnGGFbQawk/wTdO7AKHGwA4= X-Google-Smtp-Source: AGHT+IHpHyQ5wkbbDg1/9QOIdZT7U8XEUGKwcMibXVBUL9WTi2GIa5kveh+VvtS/wwBFRvp4M1UHpQ== X-Received: by 2002:a05:6a21:164e:b0:1a3:4979:f25e with SMTP id no14-20020a056a21164e00b001a34979f25emr128528pzb.59.1711515039607; Tue, 26 Mar 2024 21:50:39 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id e31-20020a63501f000000b005e4666261besm8351500pgb.50.2024.03.26.21.50.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Mar 2024 21:50:38 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Jisheng Zhang , Yunhui Cui , Samuel Holland Subject: [PATCH v6 01/13] riscv: Flush the instruction cache during SMP bringup Date: Tue, 26 Mar 2024 21:49:42 -0700 Message-ID: <20240327045035.368512-2-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.1 In-Reply-To: <20240327045035.368512-1-samuel.holland@sifive.com> References: <20240327045035.368512-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240326_215044_308267_CEAE24B7 X-CRM114-Status: GOOD ( 10.73 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Instruction cache flush IPIs are sent only to CPUs in cpu_online_mask, so they will not target a CPU until it calls set_cpu_online() earlier in smp_callin(). As a result, if instruction memory is modified between the CPU coming out of reset and that point, then its instruction cache may contain stale data. Therefore, the instruction cache must be flushed after the set_cpu_online() synchronization point. Fixes: 08f051eda33b ("RISC-V: Flush I$ when making a dirty page executable") Reviewed-by: Alexandre Ghiti Signed-off-by: Samuel Holland --- (no changes since v4) Changes in v4: - New patch for v4 arch/riscv/kernel/smpboot.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/arch/riscv/kernel/smpboot.c b/arch/riscv/kernel/smpboot.c index d41090fc3203..4b3c50da48ba 100644 --- a/arch/riscv/kernel/smpboot.c +++ b/arch/riscv/kernel/smpboot.c @@ -26,7 +26,7 @@ #include #include -#include +#include #include #include #include @@ -234,9 +234,10 @@ asmlinkage __visible void smp_callin(void) riscv_user_isa_enable(); /* - * Remote TLB flushes are ignored while the CPU is offline, so emit - * a local TLB flush right now just in case. + * Remote cache and TLB flushes are ignored while the CPU is offline, + * so flush them both right now just in case. */ + local_flush_icache_all(); local_flush_tlb_all(); complete(&cpu_running); /* From patchwork Wed Mar 27 04:49:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13605561 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A47ABCD11DD for ; Wed, 27 Mar 2024 04:51:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=teofedqYUCla7P2gcfoEo6QiTmk4qJi5ImScnEK9wDs=; b=vko0Ibfiyn42dN HxY0gHyED131lVPdgLYdzE3Ctc/e02SaAzx/p9mQa42SisGQ1lOnGQR+DFtT38Ae0l8OdTQssn6Cs J1OlP4dJOTFTOmMmHDqTUy1uVcYkhcWrQWYfajjg2FVfFOP5iWrT1641Cr+q73Ft0AiNzPGHufYup 0UiIDtC82bfTCf9G/iBpyUfiHfi5dr6VGX1qMgxK3hGtEI1rzM4cQjEsnAPlVTQTgtBQQlGl/EuHY YV1py6W/mZ6DaQdKwgFEhHLmeyIjWNnlxabF7KtmIjvaaVQjUJ4UIBUkPnKrlYWGpEGAOvFeZT6G4 LLFmdW95Gef54WFK1xrQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpLFT-00000007S6e-3Jh9; Wed, 27 Mar 2024 04:50:51 +0000 Received: from mail-ot1-x329.google.com ([2607:f8b0:4864:20::329]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpLFN-00000007S2E-0aDS for linux-riscv@lists.infradead.org; Wed, 27 Mar 2024 04:50:46 +0000 Received: by mail-ot1-x329.google.com with SMTP id 46e09a7af769-6e6e0867be8so978219a34.2 for ; Tue, 26 Mar 2024 21:50:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1711515043; x=1712119843; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Tmp/TOSalTk1E9Om53lqcGFTwB3eFzfKle7hmp+7bao=; b=GXhuVno0S5cz4b8UFrMKyu13MukdM1yZg68RHqRIiQ+6Rw2yFHtMAsjgp7SQHPXlhp seYrlCfd2xgAF2VIw0wXfyV1vgxD8LSNPeONI03ZvVuwlIIn27r7o6eaXNpzeEfaV2/4 ZqGvdfqxAEd1mHLCibgzdfp202zRdQXjwUEtulhLgwRyxHjE39TMNdYBBsqiY5Nk5YaZ JNkhSrUsfa0EJ8HbTcGnCzEvKXoCNlFj5bjMCnfShyx4GzEtIpJ+XyJNAXw3SU3qSwna eMpQLEtXF0Ss40RwW7ZQViGRP3KLOA9C8cuFJsksI7nYkcXthaRlFaiWZFbFSRm5gumN 36zA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711515043; x=1712119843; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Tmp/TOSalTk1E9Om53lqcGFTwB3eFzfKle7hmp+7bao=; b=lsA5JxlAfs3EZSdDvH37qm4JA7Ft62sdUhNdiSV0nhVJb6d/ZGKMhRrQe/xQcj1trf +P240uWZA6iFZYEo158J8CvRN7V3LwtKz7hYl8f7tC6TWOo0ih17ft+vz9FrOoxTsq2+ bi7LmxeZj96W2EBs9aNp3+PCkmq4SZKzw0SnfpZs3TwwkqVOYZrZ6zqCroH7X8jGwGcL 1//BgHGTbBLuQo96H6IJNPWilHNdDLRUZpwt+hjlJm85CXyEU2xbRbwPQwoU1dSqiW80 PfPdXKpRm7D1xmgeXqw5/Ho7OomPMlHEn7q5zFkSxphaxeCayFYaZ2dalBr1aPQkfTqu TPHw== X-Forwarded-Encrypted: i=1; AJvYcCU1Fe/ZlaIZPY22aX2aAQyPMEmRW1W4VcLjiB36J8PS2YbFZi/maVgrbQ7Xw/DnBUhDayapFaETIYP8wYaXRHTRnsw6K+yObxN5+qxaOlco X-Gm-Message-State: AOJu0YyEG1eclZ8irkGXL7YJkKeJFzjkw4urWXFZL7XByUi79KJaRa5r /kxjv5T9i41YAJLUvv03chxLIz5HO6Z3QarLxhW5EKdhR1kq6wcr9T9Bj/CrvS4= X-Google-Smtp-Source: AGHT+IFl3z8IwkcLePPXROsXhoQYUv/uXEkbIlJCNYH90sGzXXkCB+mcXoO2kI0LDgwvd8XvzM/1fA== X-Received: by 2002:a05:6830:1bd5:b0:6e6:99da:8944 with SMTP id v21-20020a0568301bd500b006e699da8944mr3176474ota.26.1711515043004; Tue, 26 Mar 2024 21:50:43 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id e31-20020a63501f000000b005e4666261besm8351500pgb.50.2024.03.26.21.50.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Mar 2024 21:50:42 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Jisheng Zhang , Yunhui Cui , Samuel Holland Subject: [PATCH v6 02/13] riscv: Factor out page table TLB synchronization Date: Tue, 26 Mar 2024 21:49:43 -0700 Message-ID: <20240327045035.368512-3-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.1 In-Reply-To: <20240327045035.368512-1-samuel.holland@sifive.com> References: <20240327045035.368512-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240326_215045_214784_7D668169 X-CRM114-Status: GOOD ( 11.14 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org The logic is the same for all page table levels. See commit 69be3fb111e7 ("riscv: enable MMU_GATHER_RCU_TABLE_FREE for SMP && MMU"). Signed-off-by: Samuel Holland Reviewed-by: Alexandre Ghiti --- Changes in v6: - Move riscv_tlb_remove_ptdesc() definition to fix 32-bit build Changes in v5: - New patch for v5 arch/riscv/include/asm/pgalloc.h | 31 +++++++++++++------------------ 1 file changed, 13 insertions(+), 18 deletions(-) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index deaf971253a2..b34587da8882 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -15,6 +15,14 @@ #define __HAVE_ARCH_PUD_FREE #include +static inline void riscv_tlb_remove_ptdesc(struct mmu_gather *tlb, void *pt) +{ + if (riscv_use_ipi_for_rfence()) + tlb_remove_page_ptdesc(tlb, pt); + else + tlb_remove_ptdesc(tlb, pt); +} + static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, pte_t *pte) { @@ -102,10 +110,7 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, struct ptdesc *ptdesc = virt_to_ptdesc(pud); pagetable_pud_dtor(ptdesc); - if (riscv_use_ipi_for_rfence()) - tlb_remove_page_ptdesc(tlb, ptdesc); - else - tlb_remove_ptdesc(tlb, ptdesc); + riscv_tlb_remove_ptdesc(tlb, ptdesc); } } @@ -139,12 +144,8 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, unsigned long addr) { - if (pgtable_l5_enabled) { - if (riscv_use_ipi_for_rfence()) - tlb_remove_page_ptdesc(tlb, virt_to_ptdesc(p4d)); - else - tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); - } + if (pgtable_l5_enabled) + riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); } #endif /* __PAGETABLE_PMD_FOLDED */ @@ -176,10 +177,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, struct ptdesc *ptdesc = virt_to_ptdesc(pmd); pagetable_pmd_dtor(ptdesc); - if (riscv_use_ipi_for_rfence()) - tlb_remove_page_ptdesc(tlb, ptdesc); - else - tlb_remove_ptdesc(tlb, ptdesc); + riscv_tlb_remove_ptdesc(tlb, ptdesc); } #endif /* __PAGETABLE_PMD_FOLDED */ @@ -190,10 +188,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, struct ptdesc *ptdesc = page_ptdesc(pte); pagetable_pte_dtor(ptdesc); - if (riscv_use_ipi_for_rfence()) - tlb_remove_page_ptdesc(tlb, ptdesc); - else - tlb_remove_ptdesc(tlb, ptdesc); + riscv_tlb_remove_ptdesc(tlb, ptdesc); } #endif /* CONFIG_MMU */ From patchwork Wed Mar 27 04:49:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13605560 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C7970CD11DD for ; Wed, 27 Mar 2024 04:50:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=bsOGRGD/P2fHfI7kaVkM6JGrQD2Nal1QOMYtriDq/qc=; b=iwDkl/8zpJMq+w joeeJFXI5UzrYERsRPHIQsDVm67aYudFmItIFrALjLA0AtvpKXxmSLTP7x2Hh+xn/Q5ItT2hMVweH B9oJIagEq9INMXVqJpG17F4kIAhwT2fu/9SJteogZ3m3VU0QFBvTw8Ji/Afn91zw0ygMIG06LLElB 4NHRuuOmpMNrxF1JO2vPYYEDiGj2b8NgjYwb7TbYvo7EaCQrpDAPleMMVdIuyJGumQstJTP5/r5Jn EtWq8lPoI4uh4ZX3XgCAh0YID5nLpmCw/Pn6It3cgRSbr6xnwgEBxO1vlg6CObwAPsccgGC+lhK/X EkrAOXQOAuyH/o38jbYA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpLFU-00000007S7S-2VoL; Wed, 27 Mar 2024 04:50:52 +0000 Received: from mail-yw1-x112f.google.com ([2607:f8b0:4864:20::112f]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpLFP-00000007S3h-46No for linux-riscv@lists.infradead.org; Wed, 27 Mar 2024 04:50:49 +0000 Received: by mail-yw1-x112f.google.com with SMTP id 00721157ae682-609f060cbafso76555237b3.0 for ; Tue, 26 Mar 2024 21:50:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1711515046; x=1712119846; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=G6C+MSH7tzUf74TudLETMF5DBzv2rpp8bzfwEKTodUs=; b=dMKLKmmJO6Vq610XFs/fagWLOGdYuNX+BW9lkUe1kAf9oLRvOqJgsWO2LhAkSqog7m 9n4112HtB5fHnheksVfMCWQETYgPEIDIGfRdHKTWdg9oG5UdVU6cQl0DtL+mKlxaEhzh 59CjUYYTqKWReD/hmfc36BcKb6Ley03xn2SGqbfoCMXCdmxx3sQ4Vokb0suS4M3MPpkT m4tFoR4/1ZUtSulVaybdP5ZGq1/v97JCU/Q0dO9PcAfftBl8mxDyM3I9l7SN57VOSuV+ mJCx8VowsvXbMaxhjGI7CbtMArOcYhNqrjhf4QgdqqsWrrput8NrjemDBh7hX5hIcioc Io/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711515046; x=1712119846; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=G6C+MSH7tzUf74TudLETMF5DBzv2rpp8bzfwEKTodUs=; b=JDdeG1i+IHPFzQHtND13Jm9Dmgl8ORxKYoHluZp6Cqtz1OWtz66Mr1Di9TayjWBzno uaIFA7ApzhVkhJMDnXL7eUsE+qZ2v9pQkEWo/CLsxm4mcRYvGKEZuE/F6L3lb22Uy4Na 09V3SECOHAxYnL687FGBuc5i/HxeyGhQRX/O3OCFPio8Y2D5MBGYAZVpir+tsP6ud0q6 aWman475vyOyquaCRzcTlzf7KQR8dOWakfDGYHsi7Unqon33pwknB9v5ICNORcw4ZPs5 +6viGDs/dknjZIKlZmEInRJhw6Jj0lsWOvfZbEeWLLoOK1T7aaTrAt5lA5/Vu+gEcSU3 iKsg== X-Forwarded-Encrypted: i=1; AJvYcCWT3Af3d+2DVMvuNz15zArqFhgKc9NpdownR+ZDzgQjSDyqv6094kndG+OO28D0kTrAG5YCcsWb4dYnVerdgTOq6Q1TdDGden/X3Swcu+0e X-Gm-Message-State: AOJu0YwNU3UXpYJ0OhXNHRFILK0HajLMOfNZS/kLrHpP2Qeygy3ze67b Kohk/zkino0SkdQz/YEGcKX9lUcSPLjLSVIMRm7fBtDbuuSqKyw5ItC29X2QHF4= X-Google-Smtp-Source: AGHT+IH+jSoiPAnTh406w99fZLlqeG+exqQi31XclYwr8IOVXBgedOOFyirc6G8eGQn9SYpg/+yAIQ== X-Received: by 2002:a81:8084:0:b0:611:9be5:4e17 with SMTP id q126-20020a818084000000b006119be54e17mr18863ywf.51.1711515044780; Tue, 26 Mar 2024 21:50:44 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id e31-20020a63501f000000b005e4666261besm8351500pgb.50.2024.03.26.21.50.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Mar 2024 21:50:43 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Jisheng Zhang , Yunhui Cui , Samuel Holland , Anup Patel Subject: [PATCH v6 03/13] riscv: Use IPIs for remote cache/TLB flushes by default Date: Tue, 26 Mar 2024 21:49:44 -0700 Message-ID: <20240327045035.368512-4-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.1 In-Reply-To: <20240327045035.368512-1-samuel.holland@sifive.com> References: <20240327045035.368512-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240326_215048_065456_FF23F569 X-CRM114-Status: GOOD ( 23.44 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org An IPI backend is always required in an SMP configuration, but an SBI implementation is not. For example, SBI will be unavailable when the kernel runs in M mode. For this reason, consider IPI delivery of cache and TLB flushes to be the base case, and any other implementation (such as the SBI remote fence extension) to be an optimization. Generally, if IPIs can be delivered without firmware assistance, they are assumed to be faster than SBI calls due to the SBI context switch overhead. However, when SBI is used as the IPI backend, then the context switch cost must be paid anyway, and performing the cache/TLB flush directly in the SBI implementation is more efficient than injecting an interrupt to S-mode. This is the only existing scenario where riscv_ipi_set_virq_range() is called with use_for_rfence set to false. sbi_ipi_init() already checks riscv_ipi_have_virq_range(), so it only calls riscv_ipi_set_virq_range() when no other IPI device is available. This allows moving the static key and dropping the use_for_rfence parameter. This decouples the static key from the irqchip driver probe order. Furthermore, the static branch only makes sense when CONFIG_RISCV_SBI is enabled. Optherwise, IPIs must be used. Add a fallback definition of riscv_use_sbi_for_rfence() which handles this case and removes the need to check CONFIG_RISCV_SBI elsewhere, such as in cacheflush.c. Reviewed-by: Anup Patel Signed-off-by: Samuel Holland Reviewed-by: Alexandre Ghiti --- Changes in v6: - Clarify the commit message for patch 3 based on ML discussion Changes in v5: - Also switch to riscv_use_sbi_for_rfence() in asm/pgalloc.h Changes in v4: - New patch for v4 arch/riscv/include/asm/pgalloc.h | 7 ++++--- arch/riscv/include/asm/sbi.h | 4 ++++ arch/riscv/include/asm/smp.h | 15 ++------------- arch/riscv/kernel/sbi-ipi.c | 11 ++++++++++- arch/riscv/kernel/smp.c | 11 +---------- arch/riscv/mm/cacheflush.c | 5 ++--- arch/riscv/mm/tlbflush.c | 31 ++++++++++++++----------------- drivers/clocksource/timer-clint.c | 2 +- 8 files changed, 38 insertions(+), 48 deletions(-) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index b34587da8882..f52264304f77 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -8,6 +8,7 @@ #define _ASM_RISCV_PGALLOC_H #include +#include #include #ifdef CONFIG_MMU @@ -17,10 +18,10 @@ static inline void riscv_tlb_remove_ptdesc(struct mmu_gather *tlb, void *pt) { - if (riscv_use_ipi_for_rfence()) - tlb_remove_page_ptdesc(tlb, pt); - else + if (riscv_use_sbi_for_rfence()) tlb_remove_ptdesc(tlb, pt); + else + tlb_remove_page_ptdesc(tlb, pt); } static inline void pmd_populate_kernel(struct mm_struct *mm, diff --git a/arch/riscv/include/asm/sbi.h b/arch/riscv/include/asm/sbi.h index 6e68f8dff76b..ea84392ca9d7 100644 --- a/arch/riscv/include/asm/sbi.h +++ b/arch/riscv/include/asm/sbi.h @@ -375,8 +375,12 @@ unsigned long riscv_cached_marchid(unsigned int cpu_id); unsigned long riscv_cached_mimpid(unsigned int cpu_id); #if IS_ENABLED(CONFIG_SMP) && IS_ENABLED(CONFIG_RISCV_SBI) +DECLARE_STATIC_KEY_FALSE(riscv_sbi_for_rfence); +#define riscv_use_sbi_for_rfence() \ + static_branch_unlikely(&riscv_sbi_for_rfence) void sbi_ipi_init(void); #else +static inline bool riscv_use_sbi_for_rfence(void) { return false; } static inline void sbi_ipi_init(void) { } #endif diff --git a/arch/riscv/include/asm/smp.h b/arch/riscv/include/asm/smp.h index 0d555847cde6..7ac80e9f2288 100644 --- a/arch/riscv/include/asm/smp.h +++ b/arch/riscv/include/asm/smp.h @@ -49,12 +49,7 @@ void riscv_ipi_disable(void); bool riscv_ipi_have_virq_range(void); /* Set the IPI interrupt numbers for arch (called by irqchip drivers) */ -void riscv_ipi_set_virq_range(int virq, int nr, bool use_for_rfence); - -/* Check if we can use IPIs for remote FENCEs */ -DECLARE_STATIC_KEY_FALSE(riscv_ipi_for_rfence); -#define riscv_use_ipi_for_rfence() \ - static_branch_unlikely(&riscv_ipi_for_rfence) +void riscv_ipi_set_virq_range(int virq, int nr); /* Check other CPUs stop or not */ bool smp_crash_stop_failed(void); @@ -104,16 +99,10 @@ static inline bool riscv_ipi_have_virq_range(void) return false; } -static inline void riscv_ipi_set_virq_range(int virq, int nr, - bool use_for_rfence) +static inline void riscv_ipi_set_virq_range(int virq, int nr) { } -static inline bool riscv_use_ipi_for_rfence(void) -{ - return false; -} - #endif /* CONFIG_SMP */ #if defined(CONFIG_HOTPLUG_CPU) && (CONFIG_SMP) diff --git a/arch/riscv/kernel/sbi-ipi.c b/arch/riscv/kernel/sbi-ipi.c index a4559695ce62..1026e22955cc 100644 --- a/arch/riscv/kernel/sbi-ipi.c +++ b/arch/riscv/kernel/sbi-ipi.c @@ -13,6 +13,9 @@ #include #include +DEFINE_STATIC_KEY_FALSE(riscv_sbi_for_rfence); +EXPORT_SYMBOL_GPL(riscv_sbi_for_rfence); + static int sbi_ipi_virq; static void sbi_ipi_handle(struct irq_desc *desc) @@ -72,6 +75,12 @@ void __init sbi_ipi_init(void) "irqchip/sbi-ipi:starting", sbi_ipi_starting_cpu, NULL); - riscv_ipi_set_virq_range(virq, BITS_PER_BYTE, false); + riscv_ipi_set_virq_range(virq, BITS_PER_BYTE); pr_info("providing IPIs using SBI IPI extension\n"); + + /* + * Use the SBI remote fence extension to avoid + * the extra context switch needed to handle IPIs. + */ + static_branch_enable(&riscv_sbi_for_rfence); } diff --git a/arch/riscv/kernel/smp.c b/arch/riscv/kernel/smp.c index 45dd4035416e..8e6eb64459af 100644 --- a/arch/riscv/kernel/smp.c +++ b/arch/riscv/kernel/smp.c @@ -171,10 +171,7 @@ bool riscv_ipi_have_virq_range(void) return (ipi_virq_base) ? true : false; } -DEFINE_STATIC_KEY_FALSE(riscv_ipi_for_rfence); -EXPORT_SYMBOL_GPL(riscv_ipi_for_rfence); - -void riscv_ipi_set_virq_range(int virq, int nr, bool use_for_rfence) +void riscv_ipi_set_virq_range(int virq, int nr) { int i, err; @@ -197,12 +194,6 @@ void riscv_ipi_set_virq_range(int virq, int nr, bool use_for_rfence) /* Enabled IPIs for boot CPU immediately */ riscv_ipi_enable(); - - /* Update RFENCE static key */ - if (use_for_rfence) - static_branch_enable(&riscv_ipi_for_rfence); - else - static_branch_disable(&riscv_ipi_for_rfence); } static const char * const ipi_names[] = { diff --git a/arch/riscv/mm/cacheflush.c b/arch/riscv/mm/cacheflush.c index bc61ee5975e4..d76fc73e594b 100644 --- a/arch/riscv/mm/cacheflush.c +++ b/arch/riscv/mm/cacheflush.c @@ -21,7 +21,7 @@ void flush_icache_all(void) { local_flush_icache_all(); - if (IS_ENABLED(CONFIG_RISCV_SBI) && !riscv_use_ipi_for_rfence()) + if (riscv_use_sbi_for_rfence()) sbi_remote_fence_i(NULL); else on_each_cpu(ipi_remote_fence_i, NULL, 1); @@ -69,8 +69,7 @@ void flush_icache_mm(struct mm_struct *mm, bool local) * with flush_icache_deferred(). */ smp_mb(); - } else if (IS_ENABLED(CONFIG_RISCV_SBI) && - !riscv_use_ipi_for_rfence()) { + } else if (riscv_use_sbi_for_rfence()) { sbi_remote_fence_i(&others); } else { on_each_cpu_mask(&others, ipi_remote_fence_i, NULL, 1); diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 893566e004b7..0435605b07d0 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -79,10 +79,10 @@ static void __ipi_flush_tlb_all(void *info) void flush_tlb_all(void) { - if (riscv_use_ipi_for_rfence()) - on_each_cpu(__ipi_flush_tlb_all, NULL, 1); - else + if (riscv_use_sbi_for_rfence()) sbi_remote_sfence_vma_asid(NULL, 0, FLUSH_TLB_MAX_SIZE, FLUSH_TLB_NO_ASID); + else + on_each_cpu(__ipi_flush_tlb_all, NULL, 1); } struct flush_tlb_range_data { @@ -103,7 +103,6 @@ static void __flush_tlb_range(struct cpumask *cmask, unsigned long asid, unsigned long start, unsigned long size, unsigned long stride) { - struct flush_tlb_range_data ftd; bool broadcast; if (cpumask_empty(cmask)) @@ -119,20 +118,18 @@ static void __flush_tlb_range(struct cpumask *cmask, unsigned long asid, broadcast = true; } - if (broadcast) { - if (riscv_use_ipi_for_rfence()) { - ftd.asid = asid; - ftd.start = start; - ftd.size = size; - ftd.stride = stride; - on_each_cpu_mask(cmask, - __ipi_flush_tlb_range_asid, - &ftd, 1); - } else - sbi_remote_sfence_vma_asid(cmask, - start, size, asid); - } else { + if (!broadcast) { local_flush_tlb_range_asid(start, size, stride, asid); + } else if (riscv_use_sbi_for_rfence()) { + sbi_remote_sfence_vma_asid(cmask, start, size, asid); + } else { + struct flush_tlb_range_data ftd; + + ftd.asid = asid; + ftd.start = start; + ftd.size = size; + ftd.stride = stride; + on_each_cpu_mask(cmask, __ipi_flush_tlb_range_asid, &ftd, 1); } if (cmask != cpu_online_mask) diff --git a/drivers/clocksource/timer-clint.c b/drivers/clocksource/timer-clint.c index 09fd292eb83d..0bdd9d7ec545 100644 --- a/drivers/clocksource/timer-clint.c +++ b/drivers/clocksource/timer-clint.c @@ -251,7 +251,7 @@ static int __init clint_timer_init_dt(struct device_node *np) } irq_set_chained_handler(clint_ipi_irq, clint_ipi_interrupt); - riscv_ipi_set_virq_range(rc, BITS_PER_BYTE, true); + riscv_ipi_set_virq_range(rc, BITS_PER_BYTE); clint_clear_ipi(); #endif From patchwork Wed Mar 27 04:49:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13605559 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id ED4C2CD1283 for ; Wed, 27 Mar 2024 04:50:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=lrriymgxX93etsYDBS+L/gV7QG7wWtpcBJZx/3qje7s=; b=IVaORGZWv5NLZW mche0ddDcAfrvqxYAQ0S8QFYrc5n71hoAIzNBrz5DZvWTJjJCm3dzsZgRVRfJbyQ4TQ3lOJpqHRCQ Nl++SFycdiP2ybmu3itq/a13vhxdLTw1WUNXh/jSt7NHtu8g6Wuc2kmRGi1fNd5yila8GJVMAuh4s amFYBGRzsojFDG9AhSYpt4g8hK+ULbn8xGwiUXnBfUD2z5a9bquvxAtelMmw173pVUuQUGfw3i6Dr eoiJ7GquLtmELUom6AWd0tnMG+8jqIIEVXQfvXEBDECv6Oeijo43W79VWBewUgzE2Ap5agGLNsVpR xKg5n9ynYdPbLCpNlMJQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpLFV-00000007S8o-3Tl3; Wed, 27 Mar 2024 04:50:53 +0000 Received: from mail-pf1-x42d.google.com ([2607:f8b0:4864:20::42d]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpLFQ-00000007S4D-1UaZ for linux-riscv@lists.infradead.org; Wed, 27 Mar 2024 04:50:50 +0000 Received: by mail-pf1-x42d.google.com with SMTP id d2e1a72fcca58-6e6b22af648so376546b3a.0 for ; Tue, 26 Mar 2024 21:50:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1711515046; x=1712119846; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=2fwmOl0CxtBfQxlaR9NQyd7IGBnsfsK+MLDyKM1aSvU=; b=hIfjkzLEGm7IZMN6P9MJpU8OXCtGNej4QShWAwxp6xHB2yC8yLGn7dRXPqdYx0BfRM 8grd/fz+qEny4vMbb3Kp1EDyrog/d5MFC+GIvMinLMGBx8SySk2h4hBnrd0qAuroBEr4 l+HSsnlufn5gMvjsI4YspYzlOv/BnV03e032m8fBfBn52PIGMkoV62wCulXOSG12QYAS c2TaoGJ97YImlVcsNNapUFNoawdGN9gyWtyWlOf/D4j1kTFunzCG+rtzQ9tsCRc1tfQ/ 6235rTzSwtOh/mLT+pICIDNgpJ8TdSZeR/33u7TRgml6yhWoFP4gsStmAMlgB1evxVS2 4KpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711515046; x=1712119846; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2fwmOl0CxtBfQxlaR9NQyd7IGBnsfsK+MLDyKM1aSvU=; b=eMxt5yL20+P2h+c2cO6osFM9MiqdNzO/Z1L7hmGjrl86pdtIOcAOXN5H0uIK6i/t/H 4xjkSvBgiGg7DX/8PWQR/beKUOhxoYoY1LaJPlgldU8KtEYg9WLLO+akZ1WIfGuc89zk Jkba1k4UjTwd6z7p4rGqzPxsaLwN3ogo0oTRprtriuD30ZT4jd0ONR3hDBU4OAnt1z5A p9ggPzH8mQ+FKkX048TICfxr5IVhRAeNDP20KNZRFEiCWFsfiD4rE0uDVFcOQymuGJtS wdoRNZTFd9iP3triNLGXBf+CSB+HHY+u5XHHUHGaGvRXxBgLgOIs5s6TpeGbUN4LPogm jrSQ== X-Forwarded-Encrypted: i=1; AJvYcCUgWrnxjJBUSV4anVuXkurv6YofwMvac+wVY2sVnVgXHhiS83fokDLhJRkPOdxXxJncqqdbaxfuKN7s/h+cHG6DGCsW40CGUviE4Bw59hds X-Gm-Message-State: AOJu0YzbQ4/LUgwMvp1wJtob8R8ErbXwn4PLVRATvo0Uj7x66cumFpVc G6EfTzMXdI8OEZ88jEymVLDB/Clzs1EdUMzfhZsLe8qTKhCI4SJ8m8yLcMGbZro= X-Google-Smtp-Source: AGHT+IGHAmpkwghveoQCxqBYVJp1RVR15SSZ2ypXvD8LXUp94lm0FkTd5l29fFm2kWdFZwCld1auyw== X-Received: by 2002:a05:6a20:3d11:b0:1a3:6a11:4898 with SMTP id y17-20020a056a203d1100b001a36a114898mr4924839pzi.3.1711515046438; Tue, 26 Mar 2024 21:50:46 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id e31-20020a63501f000000b005e4666261besm8351500pgb.50.2024.03.26.21.50.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Mar 2024 21:50:45 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Jisheng Zhang , Yunhui Cui , Samuel Holland Subject: [PATCH v6 04/13] riscv: mm: Broadcast kernel TLB flushes only when needed Date: Tue, 26 Mar 2024 21:49:45 -0700 Message-ID: <20240327045035.368512-5-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.1 In-Reply-To: <20240327045035.368512-1-samuel.holland@sifive.com> References: <20240327045035.368512-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240326_215048_487146_C4265DF8 X-CRM114-Status: GOOD ( 13.12 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org __flush_tlb_range() avoids broadcasting TLB flushes when an mm context is only active on the local CPU. Apply this same optimization to TLB flushes of kernel memory when only one CPU is online. This check can be constant-folded when SMP is disabled. Reviewed-by: Alexandre Ghiti Signed-off-by: Samuel Holland --- (no changes since v4) Changes in v4: - New patch for v4 arch/riscv/mm/tlbflush.c | 18 +++++------------- 1 file changed, 5 insertions(+), 13 deletions(-) diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 0435605b07d0..da821315d43e 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -103,22 +103,15 @@ static void __flush_tlb_range(struct cpumask *cmask, unsigned long asid, unsigned long start, unsigned long size, unsigned long stride) { - bool broadcast; + unsigned int cpu; if (cpumask_empty(cmask)) return; - if (cmask != cpu_online_mask) { - unsigned int cpuid; + cpu = get_cpu(); - cpuid = get_cpu(); - /* check if the tlbflush needs to be sent to other CPUs */ - broadcast = cpumask_any_but(cmask, cpuid) < nr_cpu_ids; - } else { - broadcast = true; - } - - if (!broadcast) { + /* Check if the TLB flush needs to be sent to other CPUs. */ + if (cpumask_any_but(cmask, cpu) >= nr_cpu_ids) { local_flush_tlb_range_asid(start, size, stride, asid); } else if (riscv_use_sbi_for_rfence()) { sbi_remote_sfence_vma_asid(cmask, start, size, asid); @@ -132,8 +125,7 @@ static void __flush_tlb_range(struct cpumask *cmask, unsigned long asid, on_each_cpu_mask(cmask, __ipi_flush_tlb_range_asid, &ftd, 1); } - if (cmask != cpu_online_mask) - put_cpu(); + put_cpu(); } static inline unsigned long get_mm_asid(struct mm_struct *mm) From patchwork Wed Mar 27 04:49:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13605562 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CD6FBC47DD9 for ; Wed, 27 Mar 2024 04:51:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=o4K1VPcUq+vePdYuWL6kFIUKDK0NpmYWn/D+b/i9/e4=; b=ach9XJVIMTplXY IMBQrBZ7UGgOsyDJRDasQ2W3RNmcMSucT3TTGeAd08uBzLXY3xFn5vhKG9Nf/wguJro6KhDyf47BX oHN7afQe5XgAOlFHvOpluGartqVaGKWUUwBK7SnW+sySns9HESo235X37P7wDv0UVB1u52P8UIaj2 Pw6HAJ1yd5znkO0ofkV3JshfrhCAS7N+GztYRpxXo9PBw2QTldF7Nc289PUPDka0lYPE2Bh3Gz5l9 ulKI70xhBtJIhTS7gNddCNti/lTp7VNZd3gkRapFOWthvy6fwZbyRFi+Gzl46wEZUih/oJq0Ep0tM 2gh2SlV4U4ALv9l56a8g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpLFg-00000007SHP-13DV; Wed, 27 Mar 2024 04:51:04 +0000 Received: from mail-ot1-x330.google.com ([2607:f8b0:4864:20::330]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpLFS-00000007S5P-2gfT for linux-riscv@lists.infradead.org; Wed, 27 Mar 2024 04:50:52 +0000 Received: by mail-ot1-x330.google.com with SMTP id 46e09a7af769-6e67b5d6dd8so3499274a34.2 for ; Tue, 26 Mar 2024 21:50:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1711515048; x=1712119848; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8r90DmKfTzo12+b9oHF/xBbsXVRSBOTytxxA308aMe8=; b=GxO3Xank/kMjp5XH4evAv3gt3R6MOoCT78H3wxxp6lZK4K8166EPpNdFTCH3usQ45L lvylOgut4TgM08rFs3GhM8mnnXJum9ghNkutz6cH1TrTXgVBLyDydDv37Id2K5sKkwfu 1VNfRwM3bzKgDD3rXfe8ufAsDNXaKO2WUohopKNYTlJv40uy1V0Otjk4SiMJlg3YWGpx tCLQ46w2fy7b6BNHeJAMyeEL6ju8brwaXAgSMVo7UFrfbRUB1Rcm51kkg7YN+Eninc8f 8a/3nRihwYLgzS3VO3WF6ZvMeHDoyhGzeXm0ePxtxm8fji64eqDefQFvH/vsyCP+OdLS yGbQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711515048; x=1712119848; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8r90DmKfTzo12+b9oHF/xBbsXVRSBOTytxxA308aMe8=; b=l+ddh1XruM+GUeKHlD866zrRGgoliSskZ9S4RavM+OScA3I47G7s0p/BnHB9ysIYTZ dwigtLm72Kuw3629kYZ92SLzqI5Gcb9zq98dxu98Scg8b4ddUeJCIzPdOWWm4LqfEo5o iGtas9mvMeivyrdPYtl3TuNB+vX7p/+Y6mUmsAzsMykmJrAZ6qsLiJJXkNcjbpb8mnSK 1Mx9lG8u3ow0tXgYD/78O4Entyn2JX/bQc93WsjrzaR0Wu6XL40IB8KHI7m6ik2MiacO gmDHD5yuDxoLsbAx1kPWWjPhmhHUif7g737/TK0D4ssr3JT1393w0zIvW3bZC6RgWL4J S3Lw== X-Forwarded-Encrypted: i=1; AJvYcCXg2MEFluTXXXf/x80p/zgMM2yK/e4al/tEnFjg/mG6zYP5l4v518CS3rSNvBqE0JtsOLhc0fbQ3GW8h1aoZZSH0tmh87WQPQJAKM+ubIyk X-Gm-Message-State: AOJu0YxcQaupzyuPrXRt1N3zQMF8AzksTu0I9kZ4vXT21ursS03xe2Ep v4/JIZNHIxTeh6hKKi/n4tCLpxS7NVwsVltdeycVo5QOtnlsmpzIdNBIb19J1Tg= X-Google-Smtp-Source: AGHT+IE/gtW/a2EQvN+qhLpDTcQ1EZrF+9VNRUlZn60iPj1iarim6P+gGyjDapgmLmO07m/XYQ3rgg== X-Received: by 2002:a05:6830:1e72:b0:6e6:7db1:b444 with SMTP id m18-20020a0568301e7200b006e67db1b444mr154078otr.36.1711515048291; Tue, 26 Mar 2024 21:50:48 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id e31-20020a63501f000000b005e4666261besm8351500pgb.50.2024.03.26.21.50.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Mar 2024 21:50:47 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Jisheng Zhang , Yunhui Cui , Samuel Holland Subject: [PATCH v6 05/13] riscv: Only send remote fences when some other CPU is online Date: Tue, 26 Mar 2024 21:49:46 -0700 Message-ID: <20240327045035.368512-6-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.1 In-Reply-To: <20240327045035.368512-1-samuel.holland@sifive.com> References: <20240327045035.368512-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240326_215050_737926_17540266 X-CRM114-Status: GOOD ( 11.95 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org If no other CPU is online, a local cache or TLB flush is sufficient. These checks can be constant-folded when SMP is disabled. Signed-off-by: Samuel Holland Reviewed-by: Alexandre Ghiti --- (no changes since v4) Changes in v4: - New patch for v4 arch/riscv/mm/cacheflush.c | 4 +++- arch/riscv/mm/tlbflush.c | 4 +++- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/arch/riscv/mm/cacheflush.c b/arch/riscv/mm/cacheflush.c index d76fc73e594b..f5be1fec8191 100644 --- a/arch/riscv/mm/cacheflush.c +++ b/arch/riscv/mm/cacheflush.c @@ -21,7 +21,9 @@ void flush_icache_all(void) { local_flush_icache_all(); - if (riscv_use_sbi_for_rfence()) + if (num_online_cpus() < 2) + return; + else if (riscv_use_sbi_for_rfence()) sbi_remote_fence_i(NULL); else on_each_cpu(ipi_remote_fence_i, NULL, 1); diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index da821315d43e..0901aa47b58f 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -79,7 +79,9 @@ static void __ipi_flush_tlb_all(void *info) void flush_tlb_all(void) { - if (riscv_use_sbi_for_rfence()) + if (num_online_cpus() < 2) + local_flush_tlb_all(); + else if (riscv_use_sbi_for_rfence()) sbi_remote_sfence_vma_asid(NULL, 0, FLUSH_TLB_MAX_SIZE, FLUSH_TLB_NO_ASID); else on_each_cpu(__ipi_flush_tlb_all, NULL, 1); From patchwork Wed Mar 27 04:49:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13605563 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 68B0DC54E67 for ; Wed, 27 Mar 2024 04:51:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=dBu1KsUZbe3eQmmdlYEm4Ar7jXufBsKuJOkGmOmQgsU=; b=ZhROBW37JWpS/d Wz2AzmmrNfkBtN+9apLp+b7DPczFF0/OMRBfNvKhAekanOAbdqHGNfNcUE+fL3lbIRImfcbug7cch N/adlQfQksNT6FwD82nFLuhdT6+uJtWRcdy3fc+gae2WdTdjpG17Pak53rLZfvyiQVwYfaQxIZoF5 /tBta39m55daRkInQYPGqWu4UBU+qQRRF+Z4iv87nZLqMqejGGHBe+a46+BJgobZy5TSEw4gduP7o mXQ1LQNv8XDKy7Irf4u7b9e1nhozN4i532q1irr15GruG2S36H1KhMi15a5Q5MP7nvftMteshICEz CXGXUq38PyYo/OqwE8CA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpLFi-00000007SJA-0nz6; Wed, 27 Mar 2024 04:51:06 +0000 Received: from mail-pf1-x435.google.com ([2607:f8b0:4864:20::435]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpLFU-00000007S6Z-3ha5 for linux-riscv@lists.infradead.org; Wed, 27 Mar 2024 04:50:55 +0000 Received: by mail-pf1-x435.google.com with SMTP id d2e1a72fcca58-6e6afb754fcso5566932b3a.3 for ; Tue, 26 Mar 2024 21:50:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1711515051; x=1712119851; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7nL2m+hLarkGQvecSEd0KAq0QtYwqUItfZBnVAx3yeI=; b=ZLpALuIxe5/f+yfMJKUokUy81U52X+w89SeKSg/APeaUHqOaSggljlTWOY/R1aDoHf obvEqcFA2BSexxHIaPrp1sHW9tfDU0EMXEwsqRDuJ/0eQc6g+zxchGmB3GUQqKA4xO2S Vyfwe4IWFfRQvp1O29boOf4g3cn6A1Xynn6g0qrQZUTfOfD6QAqflSQU3+ZVBYr3RzYz IFGT6p3QfZRk1RAuHC/2g/530Jj5HmwyvjLiuC7H6jTPbyFNmT58WrnpSATf0T1vTAJk J6kGkJFzpa1btsK73X0RsDYZ5E4eUAXQxTzGbv9noUynGEyTcTffiyDY4Fvd2LRDT+TA Q6oA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711515051; x=1712119851; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7nL2m+hLarkGQvecSEd0KAq0QtYwqUItfZBnVAx3yeI=; b=ttl5Ulm9RpENV8R6aJzRcqtHhwegxNn4eM/mexKemLcf3jonaxqAiNUMyvq+c6BclU K3pNOLQ5EtacLqyy1I0acX/RPMAaTuSZE8w0UXLdl7yGKV2AeY8cts48QfNk+lr8lr1y gtvmv002/3YszJYz+yY7tACDKljR9WO+3smmx6UMJyMpIBNnMoO7IscANPhY/QuMay8j pCe6vKiQRK1yjq8htwYlMxi2nc6DywiUxA9Q3u4TU090hUesUjyC14E57n5pF6t/35mk LaejOEgSePHNsd7O/Pmohw4nIVPuob/7dkw8o/bg3QmqQj1dB7ZGcu8QPe6tc1upleUp VHVg== X-Forwarded-Encrypted: i=1; AJvYcCVRWcB5s0NHQeb/ewf8bTjbzXEYCicuY8Cri4Db8xl8zjSoobweGbK31mAkOkY7J1v/fbWZPkv1fQd3ovHJaCcbM147xq7n/KuCLIMTuLSD X-Gm-Message-State: AOJu0YxgnsGKpmOJPZONCeySjAzN0gE92a3LsTjc9W+lnOGUuHDGGnds uk1rg3Zz7flmCd4kySXg+7AmNWKUv8XdoTYtqH3FXHmUJnAR+uXWJlKPbSP6+X8= X-Google-Smtp-Source: AGHT+IGyz4xXUqNq5aXCdK4lWIBgBxDZJLK6nH/Xm7w+Te1ce++GEK9gbqC/aCbf24OJ2Eh82pVCjA== X-Received: by 2002:a05:6a00:13a2:b0:6ea:8e89:7eca with SMTP id t34-20020a056a0013a200b006ea8e897ecamr1940335pfg.32.1711515050263; Tue, 26 Mar 2024 21:50:50 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id e31-20020a63501f000000b005e4666261besm8351500pgb.50.2024.03.26.21.50.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Mar 2024 21:50:49 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Jisheng Zhang , Yunhui Cui , Samuel Holland Subject: [PATCH v6 06/13] riscv: mm: Combine the SMP and UP TLB flush code Date: Tue, 26 Mar 2024 21:49:47 -0700 Message-ID: <20240327045035.368512-7-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.1 In-Reply-To: <20240327045035.368512-1-samuel.holland@sifive.com> References: <20240327045035.368512-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240326_215052_990709_92DFA4BB X-CRM114-Status: GOOD ( 13.63 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org In SMP configurations, all TLB flushing narrower than flush_tlb_all() goes through __flush_tlb_range(). Do the same in UP configurations. This allows UP configurations to take advantage of recent improvements to the code in tlbflush.c, such as support for huge pages and flushing multiple-page ranges. Reviewed-by: Alexandre Ghiti Signed-off-by: Samuel Holland Reviewed-by: Yunhui Cui --- (no changes since v4) Changes in v4: - Merge the two copies of __flush_tlb_range() and rely on the compiler to optimize out the broadcast path (both clang and gcc do this) - Merge the two copies of flush_tlb_all() and rely on constant folding Changes in v2: - Move the SMP/UP merge earlier in the series to avoid build issues - Make a copy of __flush_tlb_range() instead of adding ifdefs inside - local_flush_tlb_all() is the only function used on !MMU (smpboot.c) arch/riscv/Kconfig | 2 +- arch/riscv/include/asm/tlbflush.h | 31 +++---------------------------- arch/riscv/mm/Makefile | 5 +---- 3 files changed, 5 insertions(+), 33 deletions(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index be09c8836d56..442532819a44 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -60,7 +60,7 @@ config RISCV select ARCH_USE_MEMTEST select ARCH_USE_QUEUED_RWLOCKS select ARCH_USES_CFI_TRAPS if CFI_CLANG - select ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH if SMP && MMU + select ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH if MMU select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT if MMU select ARCH_WANT_FRAME_POINTERS select ARCH_WANT_GENERAL_HUGETLB if !RISCV_ISA_SVNAPOT diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index 4112cc8d1d69..4f86424b1ba5 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -27,12 +27,7 @@ static inline void local_flush_tlb_page(unsigned long addr) { ALT_FLUSH_TLB_PAGE(__asm__ __volatile__ ("sfence.vma %0" : : "r" (addr) : "memory")); } -#else /* CONFIG_MMU */ -#define local_flush_tlb_all() do { } while (0) -#define local_flush_tlb_page(addr) do { } while (0) -#endif /* CONFIG_MMU */ -#if defined(CONFIG_SMP) && defined(CONFIG_MMU) void flush_tlb_all(void); void flush_tlb_mm(struct mm_struct *mm); void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, @@ -54,28 +49,8 @@ void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *batch, unsigned long uaddr); void arch_flush_tlb_batched_pending(struct mm_struct *mm); void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch); - -#else /* CONFIG_SMP && CONFIG_MMU */ - -#define flush_tlb_all() local_flush_tlb_all() -#define flush_tlb_page(vma, addr) local_flush_tlb_page(addr) - -static inline void flush_tlb_range(struct vm_area_struct *vma, - unsigned long start, unsigned long end) -{ - local_flush_tlb_all(); -} - -/* Flush a range of kernel pages */ -static inline void flush_tlb_kernel_range(unsigned long start, - unsigned long end) -{ - local_flush_tlb_all(); -} - -#define flush_tlb_mm(mm) flush_tlb_all() -#define flush_tlb_mm_range(mm, start, end, page_size) flush_tlb_all() -#define local_flush_tlb_kernel_range(start, end) flush_tlb_all() -#endif /* !CONFIG_SMP || !CONFIG_MMU */ +#else /* CONFIG_MMU */ +#define local_flush_tlb_all() do { } while (0) +#endif /* CONFIG_MMU */ #endif /* _ASM_RISCV_TLBFLUSH_H */ diff --git a/arch/riscv/mm/Makefile b/arch/riscv/mm/Makefile index 2c869f8026a8..cbe4d775ef56 100644 --- a/arch/riscv/mm/Makefile +++ b/arch/riscv/mm/Makefile @@ -13,14 +13,11 @@ endif KCOV_INSTRUMENT_init.o := n obj-y += init.o -obj-$(CONFIG_MMU) += extable.o fault.o pageattr.o pgtable.o +obj-$(CONFIG_MMU) += extable.o fault.o pageattr.o pgtable.o tlbflush.o obj-y += cacheflush.o obj-y += context.o obj-y += pmem.o -ifeq ($(CONFIG_MMU),y) -obj-$(CONFIG_SMP) += tlbflush.o -endif obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o obj-$(CONFIG_PTDUMP_CORE) += ptdump.o obj-$(CONFIG_KASAN) += kasan_init.o From patchwork Wed Mar 27 04:49:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13605564 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EF244CD1283 for ; Wed, 27 Mar 2024 04:51:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=MGA437xq9kEX9q8EoGE5UD1KG700DcLA7ZNenOan194=; b=s0FCISyU9ITUPW myJvluiH89ja7rDpH+CBtZPVSQqmlovTmgpfDML+UdeAmFNcBDpk7F0TBlHyZQVZh1jrUYsXVoaED g2T2hyODFQmJeZYHh3SFmS9St9giy5whc9IPZ5bBM2YN/bqEY5wMF46I5XqRb5eUdg6Jha5Ij8xp5 uSnTOit8MVhpYWauWlP5lSvEJJEj0tfEpVw6aW7lOzH5CGFylihs8e4KJ1EU+6jo6TcinHuVW0p+M G3Jz0xtGlcazhTlY8m+yRUriHhn06K+HjjKdlLrBnOhUQ8zKs35Vk+kvKG9NUJG5eMFvLd3hZQPnH MOWP9daV9ILiLxvumn9A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpLFj-00000007SKR-3UrB; Wed, 27 Mar 2024 04:51:07 +0000 Received: from mail-ot1-x334.google.com ([2607:f8b0:4864:20::334]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpLFW-00000007S7Q-0cM0 for linux-riscv@lists.infradead.org; Wed, 27 Mar 2024 04:50:57 +0000 Received: by mail-ot1-x334.google.com with SMTP id 46e09a7af769-6e6a1d24770so3576025a34.0 for ; Tue, 26 Mar 2024 21:50:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1711515052; x=1712119852; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kHzKH2DTKf6Vc5BlA0YNBJTz56l+llAyGgYGMd65KTs=; b=DNEPdltqFT19c2YPEYswVK6s5rTb446YAv82B4yjd5SxAi4S5Yv/i1F1E+yUrhhm2V g7H5+CIaIpzo5qZjsTXQdQ1EsHiaTCZnFB6cOO0GSiqOrXAiawNdHNrG+c1aNcw86Mv3 opFixqcuDMKgHjNDjiFKQtV0ROr/3ZRIsS+cXF7NCnmEPZQHe+WEAuLfPTAVdr+grTSD ulavVZAT5Ljplk1vZfXq6xX/YE9kj4syFdgd+3qhpNKt3ED+zyRkxY1XFE+zbdVbhQHw dIsntfrzuOqEV2y7UJ2pYQ21Sg9AKDSmKZAAGLOD5ZaAVUS1Y79kK7je2vx222mT8fgR gHfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711515052; x=1712119852; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kHzKH2DTKf6Vc5BlA0YNBJTz56l+llAyGgYGMd65KTs=; b=fivzGXeiDseAEYI4HL8GSgkQ8jFvYtv0HR7UZGCq/28O1fa5Jr2UPdDI2Y6aHvf2rN IpUjNP5RErjMoLf1voqMnq0MXikRzqRiZXT+cehtlSkZvsf4jc7ASSkN17tyEFyVXhRF BJKFr1/YqnFusxM9MqdjjXk0IFBMy0IaGYUBwfbEDIq+h2gVZtF9UznWJJGkcEZ4yjuA ydBKXZVQ7Nw1FIWU298FRz53k/AYD58xJiQRPUVxZ2J0XxHiosvRQxQaGldautvTzW9D rQ4aRI36MuKMIwgIwbSLEfQ33MZtrR1aCKQI79rD0VnU2KuQNMrrfRzmWlX13f1wjdJ6 rpzQ== X-Forwarded-Encrypted: i=1; AJvYcCV8ZDzxO8mEyHi/fSbqSHJrRvLhm+vkCEJDgL8feZ3U0jbsjmy828T26vt55iU6utjR5JbY/VIsYpcQaUC3Lg8euDeSu2lH1z7MjBw7jZ6P X-Gm-Message-State: AOJu0Yxj9GBNesmS96haauPXLoe6g8X03hfCblhMVFD3F520TdbA0b0N 5jVp2CtXxh0OCc2ABY0Tj78XnYysTWYYg71T4V/OybIJfIc7sIGewt6R98NO3l0= X-Google-Smtp-Source: AGHT+IFNZ9hbtEq1CmgyYTIu4vUTxuEiR33JD4yc+xyNKXepJuE48avCWWfj7WZgftYdmFVpAMAO4A== X-Received: by 2002:a9d:6e0c:0:b0:6e6:b2ed:f12a with SMTP id e12-20020a9d6e0c000000b006e6b2edf12amr1764968otr.30.1711515052040; Tue, 26 Mar 2024 21:50:52 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id e31-20020a63501f000000b005e4666261besm8351500pgb.50.2024.03.26.21.50.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Mar 2024 21:50:51 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Jisheng Zhang , Yunhui Cui , Samuel Holland Subject: [PATCH v6 07/13] riscv: Apply SiFive CIP-1200 workaround to single-ASID sfence.vma Date: Tue, 26 Mar 2024 21:49:48 -0700 Message-ID: <20240327045035.368512-8-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.1 In-Reply-To: <20240327045035.368512-1-samuel.holland@sifive.com> References: <20240327045035.368512-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240326_215054_565706_D26FF5AB X-CRM114-Status: GOOD ( 14.14 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org commit 3f1e782998cd ("riscv: add ASID-based tlbflushing methods") added calls to the sfence.vma instruction with rs2 != x0. These single-ASID instruction variants are also affected by SiFive errata CIP-1200. Until now, the errata workaround was not needed for the single-ASID sfence.vma variants, because they were only used when the ASID allocator was enabled, and the affected SiFive platforms do not support multiple ASIDs. However, we are going to start using those sfence.vma variants regardless of ASID support, so now we need alternatives covering them. Signed-off-by: Samuel Holland --- (no changes since v2) Changes in v2: - Rebase on Alexandre's "riscv: tlb flush improvements" series v5 arch/riscv/include/asm/errata_list.h | 12 +++++++++++- arch/riscv/include/asm/tlbflush.h | 19 ++++++++++++++++++- arch/riscv/mm/tlbflush.c | 23 ----------------------- 3 files changed, 29 insertions(+), 25 deletions(-) diff --git a/arch/riscv/include/asm/errata_list.h b/arch/riscv/include/asm/errata_list.h index 1f2dbfb8a8bf..35ce26899960 100644 --- a/arch/riscv/include/asm/errata_list.h +++ b/arch/riscv/include/asm/errata_list.h @@ -43,11 +43,21 @@ ALTERNATIVE(__stringify(RISCV_PTR do_page_fault), \ CONFIG_ERRATA_SIFIVE_CIP_453) #else /* !__ASSEMBLY__ */ -#define ALT_FLUSH_TLB_PAGE(x) \ +#define ALT_SFENCE_VMA_ASID(asid) \ +asm(ALTERNATIVE("sfence.vma x0, %0", "sfence.vma", SIFIVE_VENDOR_ID, \ + ERRATA_SIFIVE_CIP_1200, CONFIG_ERRATA_SIFIVE_CIP_1200) \ + : : "r" (asid) : "memory") + +#define ALT_SFENCE_VMA_ADDR(addr) \ asm(ALTERNATIVE("sfence.vma %0", "sfence.vma", SIFIVE_VENDOR_ID, \ ERRATA_SIFIVE_CIP_1200, CONFIG_ERRATA_SIFIVE_CIP_1200) \ : : "r" (addr) : "memory") +#define ALT_SFENCE_VMA_ADDR_ASID(addr, asid) \ +asm(ALTERNATIVE("sfence.vma %0, %1", "sfence.vma", SIFIVE_VENDOR_ID, \ + ERRATA_SIFIVE_CIP_1200, CONFIG_ERRATA_SIFIVE_CIP_1200) \ + : : "r" (addr), "r" (asid) : "memory") + /* * _val is marked as "will be overwritten", so need to set it to 0 * in the default case. diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index 4f86424b1ba5..463b615d7728 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -22,10 +22,27 @@ static inline void local_flush_tlb_all(void) __asm__ __volatile__ ("sfence.vma" : : : "memory"); } +static inline void local_flush_tlb_all_asid(unsigned long asid) +{ + if (asid != FLUSH_TLB_NO_ASID) + ALT_SFENCE_VMA_ASID(asid); + else + local_flush_tlb_all(); +} + /* Flush one page from local TLB */ static inline void local_flush_tlb_page(unsigned long addr) { - ALT_FLUSH_TLB_PAGE(__asm__ __volatile__ ("sfence.vma %0" : : "r" (addr) : "memory")); + ALT_SFENCE_VMA_ADDR(addr); +} + +static inline void local_flush_tlb_page_asid(unsigned long addr, + unsigned long asid) +{ + if (asid != FLUSH_TLB_NO_ASID) + ALT_SFENCE_VMA_ADDR_ASID(addr, asid); + else + local_flush_tlb_page(addr); } void flush_tlb_all(void); diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 0901aa47b58f..ad7bdcfcc219 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -7,29 +7,6 @@ #include #include -static inline void local_flush_tlb_all_asid(unsigned long asid) -{ - if (asid != FLUSH_TLB_NO_ASID) - __asm__ __volatile__ ("sfence.vma x0, %0" - : - : "r" (asid) - : "memory"); - else - local_flush_tlb_all(); -} - -static inline void local_flush_tlb_page_asid(unsigned long addr, - unsigned long asid) -{ - if (asid != FLUSH_TLB_NO_ASID) - __asm__ __volatile__ ("sfence.vma %0, %1" - : - : "r" (addr), "r" (asid) - : "memory"); - else - local_flush_tlb_page(addr); -} - /* * Flush entire TLB if number of entries to be flushed is greater * than the threshold below. From patchwork Wed Mar 27 04:49:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13605566 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AD97AC47DD9 for ; Wed, 27 Mar 2024 04:51:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=ndj9Pa4swxTbZX/sMsA9Yk9IZTwZOwfKqB5IycSgCks=; b=wexvIbtrNFYmAp T5w6i+o93ZPrJu7EkgqQ80TeBU+ZVjU8wa2HJXVgViDLGvt7g9UwtEahDF/4kH32OrLeD4tClku/+ gah2FWTeLYBsNOfthQ43j+y9n+PeMyOi3T6gDle35NE3htolFE1b9QLn2tY1Gb/fOlKimI94U4nxv +iz6cgHdYjrlUV4bRdIdyIh70fKb8NpWb3/9bdKWuQ6ywbxzslbqDBO5XvIC9l0mqlhMAuinQ+yb+ jrfrBeeFv5+Gd2Tcf7DgRPveS6wP6kALB9C3bRWiowphUBN/jLts/BfkWQbNqfHxHD/+YG3ARxmp9 eBpuKfxxSDdjEEhH9cRA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpLFl-00000007SLf-23I4; Wed, 27 Mar 2024 04:51:10 +0000 Received: from mail-ot1-x329.google.com ([2607:f8b0:4864:20::329]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpLFW-00000007S9K-3bBF for linux-riscv@lists.infradead.org; Wed, 27 Mar 2024 04:50:57 +0000 Received: by mail-ot1-x329.google.com with SMTP id 46e09a7af769-6e695470280so3200749a34.3 for ; Tue, 26 Mar 2024 21:50:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1711515054; x=1712119854; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4DMxbhiunBS0lKLURYVBJG3nyObzwUO8ZrXza9ckh2c=; b=DuZYa9wn6B7OAYl4LZlptRs5sgKJEINP+E2oLua/pB6Sy8wmkvsmD4KFJN5Grp46JI IB2qCFnI6buX2/x94+wAXWaHNV1cCIF1dDZoRpcqwDk8OV3hKK3+JlYxWH5F9rQoYV6t GFjyXogA/Wi+SYP1ZCJyYtioV1TgZQWjt8u8+lbEkv9EdIUY0OIO9/wJlksKNKLnffTA CkGMezpG6QuZdhIynsaXSNvrHLdufkBXeZD1twz1oldattNntiGCOkO4iDcm2uy4CYAA apxyVQHaPSK22R00rUlbiCaIvIOG8Fk6/dR4AvBrbqjZkMjx0dgM6WEdEFHIZJqCAsfW d9NA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711515054; x=1712119854; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4DMxbhiunBS0lKLURYVBJG3nyObzwUO8ZrXza9ckh2c=; b=i4UXgCZ7+5QznEq0Oj953AqZUyHBdZ4tPSUDD7P8fAWva7o7WXg+9+vuwp8bZxJVG7 cTwcyrJS+HrZkt/QVjmBrpJOsLEd4jN9ka4qyThyiE5zGQn66+6Vq7e+YnyL/Q156DUe WZ5iOId0PmTXLPcsgUWDFiPlq4goezgI4JGiPeMYNkt9zSl+tPy2AHOhwm/gTDHNV+yP 7/NBqPYPIsse9UY2XKIgNY1Z7mPjKKixnmQn0yayzBfQKgKgKtu8vezSv9D0kh0uA522 8Sw82ACKOUV6paj2Q1NvZ82IEAPIROf03foubK/hfyS2H1mmiSFhUKoEXNU7Lz408NIC hCRw== X-Forwarded-Encrypted: i=1; AJvYcCXXAh9ayhq2+mEDujywqXhqnh6femLy96oP4mTwXuZFvxGGcbiD3txb2d84wi/JMO1NLaCBrnxfxn/ar5xUi4rCd8BMitFFpyHvN4U4WEMS X-Gm-Message-State: AOJu0YzwdjNEZkHv8oD4KhkqS+SLEiB7bGMq2w8JMxiSvfKYV3lhwmlL VvcVYSn9EB6sQ6761kunqi+HUKIicRGWMBK4bCyhDpy6RbGfTsI7xfPX6bjClWI= X-Google-Smtp-Source: AGHT+IEKVJUxYPHGmGQ7jZ/KC0+y46rPbmkW2DQ4LqVeHQ3jmUBdRhZe5PVjdoDThc5iPwQM0jPRUg== X-Received: by 2002:a9d:61d8:0:b0:6e6:9af5:752b with SMTP id h24-20020a9d61d8000000b006e69af5752bmr1730488otk.0.1711515053696; Tue, 26 Mar 2024 21:50:53 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id e31-20020a63501f000000b005e4666261besm8351500pgb.50.2024.03.26.21.50.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Mar 2024 21:50:52 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Jisheng Zhang , Yunhui Cui , Samuel Holland Subject: [PATCH v6 08/13] riscv: Avoid TLB flush loops when affected by SiFive CIP-1200 Date: Tue, 26 Mar 2024 21:49:49 -0700 Message-ID: <20240327045035.368512-9-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.1 In-Reply-To: <20240327045035.368512-1-samuel.holland@sifive.com> References: <20240327045035.368512-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240326_215055_265218_B7CAA052 X-CRM114-Status: GOOD ( 16.05 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Implementations affected by SiFive errata CIP-1200 have a bug which forces the kernel to always use the global variant of the sfence.vma instruction. When affected by this errata, do not attempt to flush a range of addresses; each iteration of the loop would actually flush the whole TLB instead. Instead, minimize the overall number of sfence.vma instructions. Signed-off-by: Samuel Holland Reviewed-by: Yunhui Cui --- Changes in v6: - Clarify the commit message for patch 8 based on ML discussion Changes in v4: - Only set tlb_flush_all_threshold when CONFIG_MMU=y. Changes in v3: - New patch for v3 arch/riscv/errata/sifive/errata.c | 5 +++++ arch/riscv/include/asm/tlbflush.h | 2 ++ arch/riscv/mm/tlbflush.c | 2 +- 3 files changed, 8 insertions(+), 1 deletion(-) diff --git a/arch/riscv/errata/sifive/errata.c b/arch/riscv/errata/sifive/errata.c index 3d9a32d791f7..716cfedad3a2 100644 --- a/arch/riscv/errata/sifive/errata.c +++ b/arch/riscv/errata/sifive/errata.c @@ -42,6 +42,11 @@ static bool errata_cip_1200_check_func(unsigned long arch_id, unsigned long imp return false; if ((impid & 0xffffff) > 0x200630 || impid == 0x1200626) return false; + +#ifdef CONFIG_MMU + tlb_flush_all_threshold = 0; +#endif + return true; } diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index 463b615d7728..8e329721375b 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -66,6 +66,8 @@ void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *batch, unsigned long uaddr); void arch_flush_tlb_batched_pending(struct mm_struct *mm); void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch); + +extern unsigned long tlb_flush_all_threshold; #else /* CONFIG_MMU */ #define local_flush_tlb_all() do { } while (0) #endif /* CONFIG_MMU */ diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index ad7bdcfcc219..18af7b5053af 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -11,7 +11,7 @@ * Flush entire TLB if number of entries to be flushed is greater * than the threshold below. */ -static unsigned long tlb_flush_all_threshold __read_mostly = 64; +unsigned long tlb_flush_all_threshold __read_mostly = 64; static void local_flush_tlb_range_threshold_asid(unsigned long start, unsigned long size, From patchwork Wed Mar 27 04:49:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13605567 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4EA8FC47DD9 for ; Wed, 27 Mar 2024 04:51:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=qfhbttmgTnNPQNXrbqm4JXGtDLTolvJcRITkJmJfhEM=; b=u89/zK7IBUaR6x Q3QcK391N6DmhVidBSlfGhwXFP4yZGLw36tnUOpRbZorR/PGXeh7h2P+PJopDmLOAWSxuO+OeJLbf pVj5nywhy8ys/dn8J8yV4pEWtmzocmr9DrC0YESOcTE/SFKcGkurfunbErYYwWvGXtr/dBkyJCn/x d/vvyqhlvzbPuYHNuObvYw9m/KzTxkyAOMkyxA9VK1rlIB+84BHU3Rw73NdZUaV83urIeOdnm/ah0 bI5uRMCkfJsVW5ST4jzyuHy5wAOI7HnhiavDzlyvhLB+aGNkqeV6D+y5+mBKXvFZpyNosqOt3TuHj IUNMQT4nwHgJXfFdywng==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpLFp-00000007SQG-3pYm; Wed, 27 Mar 2024 04:51:13 +0000 Received: from mail-oo1-xc31.google.com ([2607:f8b0:4864:20::c31]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpLFa-00000007SBE-04z8 for linux-riscv@lists.infradead.org; Wed, 27 Mar 2024 04:51:00 +0000 Received: by mail-oo1-xc31.google.com with SMTP id 006d021491bc7-5a58009fe88so511982eaf.0 for ; Tue, 26 Mar 2024 21:50:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1711515056; x=1712119856; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=2OOp6ZEyIwMU5OnG6xTEWcirae/pY2nPqfU9WqSousc=; b=HkwhKZLzRbsy2sQOljdB8PmViYtNG3McbyPyc8S6h2x36ovWfIQZmclBh6AHbJ1vR7 L1DLXtpSZczJ194DtUycNzGTtwy3NWf5gS9AQrjvDmmKz81HX8Pk0cDC8GcPMzsuYub3 ZsJk6cvPg3VTohlYozx2cJCU2Q/kG7O0Go/uo04nSEhuHwv9hPEqGXclIFdRC412wMe1 9JPmUNfeq0xwgHeRxzkyt3X35gxR16KZEK9lxWpGc9hQlIQKav3fM66cbLl11thpBAre JPIFtcTL8erFFUDdQ2Ow+p9K0p1wwp3zTL7SUVWoFHar+TMjh0u/VxVn7YCk1uQ+IKUu 6BQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711515056; x=1712119856; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2OOp6ZEyIwMU5OnG6xTEWcirae/pY2nPqfU9WqSousc=; b=XdaUj009/rlQ93HjWDO/Ua3e5PSg4xj5crb1tHjlBJlyOZtJUJJycIOdA+PS6HxhlZ vYm1T+WemwfHLQiwJyNH1NZ4C/1UqHvcLdQ5QB0BABQ5FkvbNs+ru2dJXbyTyTEb+hNh J+KR2HLehcppjjpLU9fkC74kgnNyjBWLr3LlCwrFfq9Fzijz/BTq7OsiUqNx32oXIEKO lCD4QZZCpITAXHrQr6ihTJGi32wy+Ocwz4XxEWf9dTufr4el04DPtIrlOLhjjjoeWhL7 SZJT7DYJ1KZx/qXOMpVtfRi5ikSZIWXgtFQIXpXgAaEIISgFu714yFE4DVFlgpKc6gdn aPmg== X-Forwarded-Encrypted: i=1; AJvYcCW6EceSwc3N4j/NkvNFbdgYB57xnzWhVa3+Uc4o1ZL+bMcTj7GFPFZ58A/4O2p3wF3uP6AD2TiK2Gd4oVLpWZKUdYZBWyhvajt7cIpkZvwX X-Gm-Message-State: AOJu0YyKWoSM9rI6484TQNfnzOZf2s4bsxpn4rjBRL9emIzWiEnhfMT6 uOGIfOxtGrg5ENF2nLKWgycOpt8Xe6mbhY/YxOW4KMo2xzpYnsh6wnuhtPIUipY= X-Google-Smtp-Source: AGHT+IH2eQoyDSJi6zqBjW5QAslrBjHf9N1FFSb9F+mmhA++Fk/xLCVtnw/qNQbwZOYB8ND1BkvynQ== X-Received: by 2002:a05:6358:7584:b0:17c:1e6c:14c0 with SMTP id x4-20020a056358758400b0017c1e6c14c0mr311341rwf.16.1711515055737; Tue, 26 Mar 2024 21:50:55 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id e31-20020a63501f000000b005e4666261besm8351500pgb.50.2024.03.26.21.50.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Mar 2024 21:50:54 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Jisheng Zhang , Yunhui Cui , Samuel Holland Subject: [PATCH v6 09/13] riscv: mm: Introduce cntx2asid/cntx2version helper macros Date: Tue, 26 Mar 2024 21:49:50 -0700 Message-ID: <20240327045035.368512-10-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.1 In-Reply-To: <20240327045035.368512-1-samuel.holland@sifive.com> References: <20240327045035.368512-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240326_215058_297581_F71C7450 X-CRM114-Status: GOOD ( 14.28 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org When using the ASID allocator, the MM context ID contains two values: the ASID in the lower bits, and the allocator version number in the remaining bits. Use macros to make this separation more obvious. Reviewed-by: Alexandre Ghiti Signed-off-by: Samuel Holland --- (no changes since v1) arch/riscv/include/asm/mmu.h | 3 +++ arch/riscv/mm/context.c | 12 ++++++------ arch/riscv/mm/tlbflush.c | 2 +- 3 files changed, 10 insertions(+), 7 deletions(-) diff --git a/arch/riscv/include/asm/mmu.h b/arch/riscv/include/asm/mmu.h index 355504b37f8e..a550fbf770be 100644 --- a/arch/riscv/include/asm/mmu.h +++ b/arch/riscv/include/asm/mmu.h @@ -26,6 +26,9 @@ typedef struct { #endif } mm_context_t; +#define cntx2asid(cntx) ((cntx) & asid_mask) +#define cntx2version(cntx) ((cntx) & ~asid_mask) + void __init create_pgd_mapping(pgd_t *pgdp, uintptr_t va, phys_addr_t pa, phys_addr_t sz, pgprot_t prot); #endif /* __ASSEMBLY__ */ diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index ba8eb3944687..b562b3c44487 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -81,7 +81,7 @@ static void __flush_context(void) if (cntx == 0) cntx = per_cpu(reserved_context, i); - __set_bit(cntx & asid_mask, context_asid_map); + __set_bit(cntx2asid(cntx), context_asid_map); per_cpu(reserved_context, i) = cntx; } @@ -102,7 +102,7 @@ static unsigned long __new_context(struct mm_struct *mm) lockdep_assert_held(&context_lock); if (cntx != 0) { - unsigned long newcntx = ver | (cntx & asid_mask); + unsigned long newcntx = ver | cntx2asid(cntx); /* * If our current CONTEXT was active during a rollover, we @@ -115,7 +115,7 @@ static unsigned long __new_context(struct mm_struct *mm) * We had a valid CONTEXT in a previous life, so try to * re-use it if possible. */ - if (!__test_and_set_bit(cntx & asid_mask, context_asid_map)) + if (!__test_and_set_bit(cntx2asid(cntx), context_asid_map)) return newcntx; } @@ -168,7 +168,7 @@ static void set_mm_asid(struct mm_struct *mm, unsigned int cpu) */ old_active_cntx = atomic_long_read(&per_cpu(active_context, cpu)); if (old_active_cntx && - ((cntx & ~asid_mask) == atomic_long_read(¤t_version)) && + (cntx2version(cntx) == atomic_long_read(¤t_version)) && atomic_long_cmpxchg_relaxed(&per_cpu(active_context, cpu), old_active_cntx, cntx)) goto switch_mm_fast; @@ -177,7 +177,7 @@ static void set_mm_asid(struct mm_struct *mm, unsigned int cpu) /* Check that our ASID belongs to the current_version. */ cntx = atomic_long_read(&mm->context.id); - if ((cntx & ~asid_mask) != atomic_long_read(¤t_version)) { + if (cntx2version(cntx) != atomic_long_read(¤t_version)) { cntx = __new_context(mm); atomic_long_set(&mm->context.id, cntx); } @@ -191,7 +191,7 @@ static void set_mm_asid(struct mm_struct *mm, unsigned int cpu) switch_mm_fast: csr_write(CSR_SATP, virt_to_pfn(mm->pgd) | - ((cntx & asid_mask) << SATP_ASID_SHIFT) | + (cntx2asid(cntx) << SATP_ASID_SHIFT) | satp_mode); if (need_flush_tlb) diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 18af7b5053af..35266dd9a9a2 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -110,7 +110,7 @@ static void __flush_tlb_range(struct cpumask *cmask, unsigned long asid, static inline unsigned long get_mm_asid(struct mm_struct *mm) { return static_branch_unlikely(&use_asid_allocator) ? - atomic_long_read(&mm->context.id) & asid_mask : FLUSH_TLB_NO_ASID; + cntx2asid(atomic_long_read(&mm->context.id)) : FLUSH_TLB_NO_ASID; } void flush_tlb_mm(struct mm_struct *mm) From patchwork Wed Mar 27 04:49:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13605568 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E117FC47DD9 for ; Wed, 27 Mar 2024 04:51:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=qcvpCEED0It/EbtQ04L50Y4KCybYxBmXEmNm2oR6zQM=; b=MrK2E5XlaPTbq2 +HWq87RmbIEBHB6+uPhrWG3z2UOpGjTlSlZ+oNbvVKAmhfOxjWWD8NLM2dohAmMxL2N+xnraNT6rc XQDfyE8cqofPJFoxsqQPCg/HbADM7dSJXw45nC0Oqx6CM3p+g2rABmR0ToK7QQPY2vpLHVpCmB6lX 6CYVPsVBPx7knYaN8oc952LAbfuaOBam2USA/dbvjv5Bx2xA9YNhONhmngz2tqR1ynb6s+WLE4+F4 1Hq4IHeuPG95McJyKfTLqTtJ01IZwxIHvj5g2P1YBrI/tO74lzJE5PfzH7BcF64S9m2oetSMroNGO Co+tHXkkcEBmwLUsbIFw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpLFu-00000007SU6-1aV4; Wed, 27 Mar 2024 04:51:18 +0000 Received: from mail-oo1-xc2a.google.com ([2607:f8b0:4864:20::c2a]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpLFb-00000007SCZ-44DA for linux-riscv@lists.infradead.org; Wed, 27 Mar 2024 04:51:03 +0000 Received: by mail-oo1-xc2a.google.com with SMTP id 006d021491bc7-5a4817189c8so3716064eaf.1 for ; Tue, 26 Mar 2024 21:50:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1711515057; x=1712119857; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=A+Ju3+5NsQ7zqLi78mVdBh82QtKzTZZzrfFOA+rFDFs=; b=b+NGbJGayM7JfvvbQ5CZzTposj+4Z3Ftg8bnqzEk1jW8sRKB7wYbf6K0g6CaGiWDNQ cLQUyeoJKG15i1cu5gZxP0waak/MD1wUxSv2UtRzMMEZkDWVn57UO3C+mHnCgA0nb+ah Q6mHElMaV+l/WSEOgiLFMuiGPJJOmlCev44N/zsQvH45Ju3CBLjNuE1I3sBs0nAt2S+s kVErFSe3kOHJ3dExtkTH9BCPN+oFpbnPtLYPndR81mbTxJls10132Fs84crlJia8BCzH ldC2tsOCdUq7At1zZ/JJw2a4BW+ORvOkxqZ2Y0Z5rdOX5taO3mt0+JdFXvnD9ojJzr+h NndA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711515057; x=1712119857; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=A+Ju3+5NsQ7zqLi78mVdBh82QtKzTZZzrfFOA+rFDFs=; b=NviCz3JyAFQn093bMye4buk+OsN+0J+42BHrJD4ub2fenlm4VGhT+PMIodTf7GnUxf hRgbgLaAFqS67vYAGQ6TW6TX0GDNzmO9h4loBogC6IoQukIrM448CgLQYmI4vPQKFPAx WV9pEygq+8zzqbAkomR8RsM34GuFuEi1eBzc5KJTeN43vg5UQJWgc1R8o8nxZ7JBgX2u /J4cfQiLvAErQp+FU4+AcpfFw86L4ry0fi7y+Lm2BV1RLs9jhujOvS9bJCgTAaRKGu6r 0O5Wz4VLKvK4BZyu8e4RS4n+6n2B+gbhdKJ8d00rKB60smi9k+l+/qb0U7XIox5/pBAG zBAA== X-Forwarded-Encrypted: i=1; AJvYcCXFLIrrP0Hof7ZbM7JtPHyal88kuvqWgQfT5j/VXWtq7PzM/FJzIU5qi+tRz434Ea1GSKp0D6tFTVUr7jtcyl6+I43CmdGGBGusjtZQtpkc X-Gm-Message-State: AOJu0YzoQ7i8StdLBS+7plpzVGn+t/3DZ8wjCcDo0vzsNRafe+V+alOq 1wwvT1OW5VcUb/rcvBQ9tmpBT4A1rNkPlbemQrSKTnPQ6bmPli3mrTXJwTmtvb4= X-Google-Smtp-Source: AGHT+IE4clbeICIX6LaCSNcjSNKRNzkyOLoTskHKcKR+t8ek8fOhgjRruwacqNmtd1HR0FedGGJk+g== X-Received: by 2002:a05:6358:6397:b0:17e:c8d9:328 with SMTP id k23-20020a056358639700b0017ec8d90328mr5244771rwh.11.1711515057489; Tue, 26 Mar 2024 21:50:57 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id e31-20020a63501f000000b005e4666261besm8351500pgb.50.2024.03.26.21.50.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Mar 2024 21:50:56 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Jisheng Zhang , Yunhui Cui , Samuel Holland Subject: [PATCH v6 10/13] riscv: mm: Use a fixed layout for the MM context ID Date: Tue, 26 Mar 2024 21:49:51 -0700 Message-ID: <20240327045035.368512-11-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.1 In-Reply-To: <20240327045035.368512-1-samuel.holland@sifive.com> References: <20240327045035.368512-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240326_215100_394179_B83FB282 X-CRM114-Status: GOOD ( 12.60 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Currently, the size of the ASID field in the MM context ID dynamically depends on the number of hardware-supported ASID bits. This requires reading a global variable to extract either field from the context ID. Instead, allocate the maximum possible number of bits to the ASID field, so the layout of the context ID is known at compile-time. Reviewed-by: Alexandre Ghiti Signed-off-by: Samuel Holland --- (no changes since v1) arch/riscv/include/asm/mmu.h | 4 ++-- arch/riscv/include/asm/tlbflush.h | 2 -- arch/riscv/mm/context.c | 6 ++---- 3 files changed, 4 insertions(+), 8 deletions(-) diff --git a/arch/riscv/include/asm/mmu.h b/arch/riscv/include/asm/mmu.h index a550fbf770be..dc0273f7905f 100644 --- a/arch/riscv/include/asm/mmu.h +++ b/arch/riscv/include/asm/mmu.h @@ -26,8 +26,8 @@ typedef struct { #endif } mm_context_t; -#define cntx2asid(cntx) ((cntx) & asid_mask) -#define cntx2version(cntx) ((cntx) & ~asid_mask) +#define cntx2asid(cntx) ((cntx) & SATP_ASID_MASK) +#define cntx2version(cntx) ((cntx) & ~SATP_ASID_MASK) void __init create_pgd_mapping(pgd_t *pgdp, uintptr_t va, phys_addr_t pa, phys_addr_t sz, pgprot_t prot); diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index 8e329721375b..72e559934952 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -15,8 +15,6 @@ #define FLUSH_TLB_NO_ASID ((unsigned long)-1) #ifdef CONFIG_MMU -extern unsigned long asid_mask; - static inline void local_flush_tlb_all(void) { __asm__ __volatile__ ("sfence.vma" : : : "memory"); diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index b562b3c44487..5315af06cd4d 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -22,7 +22,6 @@ DEFINE_STATIC_KEY_FALSE(use_asid_allocator); static unsigned long asid_bits; static unsigned long num_asids; -unsigned long asid_mask; static atomic_long_t current_version; @@ -128,7 +127,7 @@ static unsigned long __new_context(struct mm_struct *mm) goto set_asid; /* We're out of ASIDs, so increment current_version */ - ver = atomic_long_add_return_relaxed(num_asids, ¤t_version); + ver = atomic_long_add_return_relaxed(BIT(SATP_ASID_BITS), ¤t_version); /* Flush everything */ __flush_context(); @@ -247,7 +246,6 @@ static int __init asids_init(void) /* Pre-compute ASID details */ if (asid_bits) { num_asids = 1 << asid_bits; - asid_mask = num_asids - 1; } /* @@ -255,7 +253,7 @@ static int __init asids_init(void) * at-least twice more than CPUs */ if (num_asids > (2 * num_possible_cpus())) { - atomic_long_set(¤t_version, num_asids); + atomic_long_set(¤t_version, BIT(SATP_ASID_BITS)); context_asid_map = bitmap_zalloc(num_asids, GFP_KERNEL); if (!context_asid_map) From patchwork Wed Mar 27 04:49:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13605569 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 44DEFC47DD9 for ; Wed, 27 Mar 2024 04:51:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=GPO4FAsYN7bEGiElsjkzPhFKQfJ4daSR4nAocKvg4wk=; b=UmTMpq6CWtrJhl OJQxTTr+vqZuIfHwVBtjlYVbKSPqfXJ9nkvvnCYCkAhfRRKIdhqBsospilYbmxXojkj60cirUgtng /1KJaSYOjep0Hvdjp6ITMqlmZzvOOsqntt83XwuRRNKm6wtWwqt9C6QhpRbQawRLKmlVw4cfzptQn +G8cwVX3Fj+c+ux1lVG7cbQaxa8bkP61sbXnGQHhol0YlNu3PAy3xHc9myZUYyhG6l78jtSalmy7U s6KTIhJhZ6DVa7+6IIY+12fRAik7UNRhDR3/r5vKTclA2JNDDlbZTTg0l+MkGfEf0eS4XPo++/a7d cpUEZAYQ1lYArR0RfI0A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpLFz-00000007SYy-1Sa2; Wed, 27 Mar 2024 04:51:23 +0000 Received: from mail-oo1-xc2f.google.com ([2607:f8b0:4864:20::c2f]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpLFd-00000007SDG-1UVq for linux-riscv@lists.infradead.org; Wed, 27 Mar 2024 04:51:04 +0000 Received: by mail-oo1-xc2f.google.com with SMTP id 006d021491bc7-5a4f608432bso3402266eaf.2 for ; Tue, 26 Mar 2024 21:50:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1711515059; x=1712119859; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4wFLc5gcnBs285NRnicKVmmQVV53EnA/2rANcUM2uCA=; b=QxiTk4/dC9QT48dwc4qNB1noxiAJKPYKeouA95n9OFJgLEjRBBRpPbLaLVum/LKIiZ tzWtvvnA+1kNaPW/xI52EZxxrD/Tp3aNndSdEZZzyuRkGRALVjtFejLYS5CRRwjCEkP+ W7/cvBbuXG8pp2prRPY97CIsciGsaxmIWPbiIVS2HldBKow51SAc+h3nzGA/sxgLdZSY u5snN5e0QfC9HfSvoPs1Te16ZziBLD1O+RO6YhlwV48+wlueTR7h4pv8SqhvCdo6E0UW MJWvpl0pX/JKR7ZE6cdSw17PID4wG2zepP6LkLK5++XA5W48qWPVRkXs2uiyQz7skmXY 4W6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711515059; x=1712119859; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4wFLc5gcnBs285NRnicKVmmQVV53EnA/2rANcUM2uCA=; b=WR4/i3h7/piV8QdHZ3SdR/QuuQokDKPac9u0eW2ko5aaMDODqLK8PfCmnWiLwNsZBi k12VaV9Z4Es0Y18m9WKr2v60HjpGCS+BEgciMlFTpGFd6asd1cxPrYqjSU7bW8oe2h3a tvwTy3JlmyOSQsmpH233QsguKQ7LuNOWxD+NcN7xKHO4tc7eBU+Kj+AYuzf5FxKtje4v bBlpYlTInbzc8JWn5Nb6IsGi9njQVnVyRcen6bfgAoDvUvAPtpYV1/Kq0xuKp2WrjVtC awpLgO/OK7MbkNLCFjBVfakLTESQNyCjmqRlsFkULRvcRRJvbXtnn6fED1j4uMXPyj8h 299w== X-Forwarded-Encrypted: i=1; AJvYcCWiGxm2ATqA9JSW/YOCHarqHmtrhbHWAKCpXFQJOkx5p1ksS/mMzkLQojXdk3frxVfZMwWs6moGxBA2cyfvdFiGIQQ1HrIQagdL8HEoezIX X-Gm-Message-State: AOJu0YxdLmQobesFe0zP/ubeAggW7z014QtoLVPlXLDmjNVAqwmcq8bm 8BmGGfTtELq2EdEcqAAlBZpnM/K3sLoTyeXikf3r72CtDwA3cDX69VfUFYOXNVU= X-Google-Smtp-Source: AGHT+IFmC6K+A8yipt+L5BT3VYhAZMjFopo36khv4UzVVH+BHr/EbytiKBlOS5fSbYkXOITYtqGpvA== X-Received: by 2002:a05:6358:320c:b0:17e:bb52:927f with SMTP id a12-20020a056358320c00b0017ebb52927fmr1922247rwe.1.1711515059146; Tue, 26 Mar 2024 21:50:59 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id e31-20020a63501f000000b005e4666261besm8351500pgb.50.2024.03.26.21.50.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Mar 2024 21:50:58 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Jisheng Zhang , Yunhui Cui , Samuel Holland Subject: [PATCH v6 11/13] riscv: mm: Make asid_bits a local variable Date: Tue, 26 Mar 2024 21:49:52 -0700 Message-ID: <20240327045035.368512-12-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.1 In-Reply-To: <20240327045035.368512-1-samuel.holland@sifive.com> References: <20240327045035.368512-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240326_215101_585531_29561FED X-CRM114-Status: UNSURE ( 9.10 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org This variable is only used inside asids_init(). Reviewed-by: Alexandre Ghiti Signed-off-by: Samuel Holland --- (no changes since v1) arch/riscv/mm/context.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index 5315af06cd4d..0bf6d0070a14 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -20,7 +20,6 @@ DEFINE_STATIC_KEY_FALSE(use_asid_allocator); -static unsigned long asid_bits; static unsigned long num_asids; static atomic_long_t current_version; @@ -226,7 +225,7 @@ static inline void set_mm(struct mm_struct *prev, static int __init asids_init(void) { - unsigned long old; + unsigned long asid_bits, old; /* Figure-out number of ASID bits in HW */ old = csr_read(CSR_SATP); From patchwork Wed Mar 27 04:49:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13605603 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E915CC54E67 for ; Wed, 27 Mar 2024 06:03:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=rLB3xJwyDt+V4H9HcdYD7X17kR2W/K5QMJXAG7DV0No=; b=E4Gw5TThaFwt4J +NwoYEUneqoaoZcIKT5R5NGzTtJWMIPFFmV1px8Lfil9i0bGxQ5l5jDzE+CnCJQUM0TqxnIH9yy3R wUWivdEne5aQa0iT2WW+L58Ocru68Mx6KuTqUKVnz7Y2N77RpZ8DKTEJma9KVRUqnEwiKuEdIFIi9 OcD/epFMUlX1c0Nt2LAuC8wRSvJqaGd2vgaslC5hGHybmYDpXm9ZsDJpgAyiJagKhv2sYvUO5Qy/g Y/gfHEV7yhUAQCz8iQnDXC/tXcNiz7gAGseeGnb+iuYq33mVuiVAESWEopN6RA5OU3fjLivsQQYWX gocLOBemqBI7d0YgJbdA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpMNQ-00000007bkW-2640; Wed, 27 Mar 2024 06:03:08 +0000 Received: from mail-pf1-x429.google.com ([2607:f8b0:4864:20::429]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpLFg-00000007SFR-0BBt for linux-riscv@lists.infradead.org; Wed, 27 Mar 2024 04:51:06 +0000 Received: by mail-pf1-x429.google.com with SMTP id d2e1a72fcca58-6ea9a605ca7so381506b3a.0 for ; Tue, 26 Mar 2024 21:51:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1711515061; x=1712119861; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=hwpkv2OuO7YHUplKH98oFVfOa+CFbclxXNu0ysmduyU=; b=PV09Steyhc52RLNpCRmB+LTuOKqR0PsYqJyEW0YmkZ3z8J3Y9Aghwc6+zpbUrEICq0 eSyc6DFGjweenKY2HTKxoxwDa+Xq/Jf54ux4V30sWJ+u7ZVkP9nQaJaGXkk5sxw0xosN 8eisEFLnU+mmUBmRzXW2SF0f+XU6V1PQcFucdgQWpBKRcwN9zdmpxVgtfSD/Zskcz3OB 4IDwLk+4/Ph3y8cefxs3KEIgDDOcfFopoegs0K6WziiCe0sp//EII4Yvzwy+l7ZB+rRm vab2JEQfI6FNHtXnC/cdIlKK0Z9I31C2pvRLH58etAz8cCp7O3YnQuf7eXrBNYiA+/ly VQ+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711515061; x=1712119861; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hwpkv2OuO7YHUplKH98oFVfOa+CFbclxXNu0ysmduyU=; b=cuVNFaj+4EP8FCdq28Z5lLjYPFZC0yNqPFThWoMI+mRzfl3q1LezBG0yOKHI1LmLy/ 3KPG/iQr4f2nWlmJhusZuUS50NNC7EZfYcMwlxqLDTyzoZFEd7hqEXzBDB3QmGkcyQUn RcjV0dsFsTgPHGtj2vJCp3HJq1+rA0D2PqFs6Pu4m8BSHqDKssy2AJ/nWyHVD9eOzkCu 1esTe1OxyK+WWA0c47GN2+cChwwNY7/4E6TL/UF7APpOcHfDXv0LOSjZ4sYyi433aOlu LhjtwJNEJax6I+xqXDxUDZ58sCKki9w42n0vVfNi3GkXhyHaHoWBPMY7FSEQj4JX/LOY DM5w== X-Forwarded-Encrypted: i=1; AJvYcCV2XWAeDvcbvekxcCWCAnaqxqq2oOJUIOuzwzaNXDjEDK7YdmpH9jKp2yosB8U3X60qsHD/OaOkD4GwAl/AUTiD495d50+nHOTtFazjuNll X-Gm-Message-State: AOJu0Yw2epBioYqqEFPFgQhHaTqMA/dMjCgt1RoUtzN4vJBPUluJLYgT fuY9h5Is++QVIx11fQD/YBP18V3tqD2YmM0j+wTCEWJQqaNibyHns1QHD5y8MHY= X-Google-Smtp-Source: AGHT+IFIatp35KNpgDUDmJycAxTP3nvYaAVXQURqJcQtx3/AD7qiy3AeB8dChiHf0CcNcubYBhb1CQ== X-Received: by 2002:a05:6a20:7345:b0:1a1:67c0:c751 with SMTP id v5-20020a056a20734500b001a167c0c751mr5157753pzc.17.1711515061281; Tue, 26 Mar 2024 21:51:01 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id e31-20020a63501f000000b005e4666261besm8351500pgb.50.2024.03.26.21.50.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Mar 2024 21:51:00 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Jisheng Zhang , Yunhui Cui , Samuel Holland Subject: [PATCH v6 12/13] riscv: mm: Preserve global TLB entries when switching contexts Date: Tue, 26 Mar 2024 21:49:53 -0700 Message-ID: <20240327045035.368512-13-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.1 In-Reply-To: <20240327045035.368512-1-samuel.holland@sifive.com> References: <20240327045035.368512-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240326_215104_986643_296CEE5E X-CRM114-Status: GOOD ( 14.57 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org If the CPU does not support multiple ASIDs, all MM contexts use ASID 0. In this case, it is still beneficial to flush the TLB by ASID, as the single-ASID variant of the sfence.vma instruction preserves TLB entries for global (kernel) pages. This optimization is recommended by the RISC-V privileged specification: If the implementation does not provide ASIDs, or software chooses to always use ASID 0, then after every satp write, software should execute SFENCE.VMA with rs1=x0. In the common case that no global translations have been modified, rs2 should be set to a register other than x0 but which contains the value zero, so that global translations are not flushed. It is not possible to apply this optimization when using the ASID allocator, because that code must flush the TLB for all ASIDs at once when incrementing the version number. Reviewed-by: Alexandre Ghiti Signed-off-by: Samuel Holland --- (no changes since v1) arch/riscv/mm/context.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index 0bf6d0070a14..60cb0b82240e 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -200,7 +200,7 @@ static void set_mm_noasid(struct mm_struct *mm) { /* Switch the page table and blindly nuke entire local TLB */ csr_write(CSR_SATP, virt_to_pfn(mm->pgd) | satp_mode); - local_flush_tlb_all(); + local_flush_tlb_all_asid(0); } static inline void set_mm(struct mm_struct *prev, From patchwork Wed Mar 27 04:49:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13605602 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8457DC47DD9 for ; Wed, 27 Mar 2024 06:03:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=r/0aBd0jCMuqrVKN3Fmy1lljujzzw/vR+Mmo9prsuqQ=; b=Rhh7QVU5O0bBXk gPtq/QmF8V7Mfe3TgiPOAgGrRVcvyv/mzHHRsK1IIfkxt56iNO5zX+9RzyQ7FNZLxXOyiMT00os4X uiVNm3L0x+mWL/ZIbu6Y6mkfgCM7x6Xduwc1bY6GFbgqJTtqW5+AwujiwRHfhmJrUagphxJYIBwrL 7dLAiz8Wm7X7Ju1d5Yvzyskna9ggbT2J6SXIgG6cYAwCs5SEBx7oD/FS6FLvHl07n6Y3D4WFlmlJh 1MoEyBFLzuQtSxVW07T965lRBor4LSJyOVa1+MY1c8/4MgZWaTLq1ecPNIZ1Tpz66QBrQvWhjvSC9 N0sJeNeb/ZtrYDfiG1gw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpMNR-00000007bkk-0F0E; Wed, 27 Mar 2024 06:03:09 +0000 Received: from mail-oo1-xc32.google.com ([2607:f8b0:4864:20::c32]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpLFm-00000007SHO-099h for linux-riscv@lists.infradead.org; Wed, 27 Mar 2024 04:51:12 +0000 Received: by mail-oo1-xc32.google.com with SMTP id 006d021491bc7-5a58009fe88so512009eaf.0 for ; Tue, 26 Mar 2024 21:51:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1711515063; x=1712119863; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DStjBcAn1SHrL/HpDMugRz0dC8r4hdt1hrlyujsEs2s=; b=Mjv7BPCmksj3UCvBBEhJNei7Kpbmx+VJbhCb6ot4XO13Awm23VHkEXQ3kA6CnL5jNW zM8rXs6jUCNxR8ulwPfFxzvtC5zBzXLdE1/FODhmoaOCObHPTVRUPQ+j9ybghzEeruqb LSRbuyTyA8nHe6/h+7WZpEFkAtpBF3yyjUHxDbkI5f0n0Wt7dD+1iQqf+zDORkfgMKEN GGinxgXAeSUYBwmw2R9CAKIpvGNtIsolv22wZLVgPnRBb4dFEfARhhnNXbJbMcRXa9xx HNujjExkDlWH8OJBJsTBtu8KeickdgTjp0csLM+SF48MehInz+SC2k1MMqk+HrQkiYce 0SWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711515063; x=1712119863; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DStjBcAn1SHrL/HpDMugRz0dC8r4hdt1hrlyujsEs2s=; b=JVcf8WVS8iEysPvBB5WtDGQlg7DCVdUAHUZulF3dZ4FMR97w/bNVtmjoFhbqo5bKeK vxK3xFykg+YRUsiS5rOuZzX6KEHVLOg7fvMMQMbLP/bFEmp4thMqsSRRYTGHFy6RblWF 85bJEB0uXFTv+jFQY28pSYVonFdVGGn6WSB+73DED7B3yhs+3umyGq7LBpZR/rnVzTq2 /drfDGBUU6y+a6XPBhruIAhclUTo328TPHeleqEg4lqw9bCZSRTzDtvymqqiw1FXLVtX +Gni2gMl2He2LCFY04R1ExUZcMcHr3OPJ4v1ufVJwq3vW/Kx9uZWFyxqRh5mQ4pMqDzQ 0dkQ== X-Forwarded-Encrypted: i=1; AJvYcCXXejvL5DbfO8WsXlWWsxRrby7fIZ5UmfnrGAGukYrE78icYbH0uCXw1XINZ7ojbcKvIFnZPep9OqhQAQaU6jCWN8Z3ZjzghlsXLsGw98/U X-Gm-Message-State: AOJu0Yycclp8Uf9qEKQmxz2cAbD4iBLMqe6MjwhU0p6vm1Nj0MO2Hl9p pFPA35Kzg8aMZjFHOXwT5WkWfGEqjIIrSTEIeo+zg7B9JQeU+kfpbm1sfSxUfVQ= X-Google-Smtp-Source: AGHT+IElUFan01DMTcKq7mpLuQEXv/M6ZjDhhp1pW7v7cPBwp502ELftTLKlJV+xiDuinHxlNWw0zA== X-Received: by 2002:a05:6358:520d:b0:17f:1d34:6253 with SMTP id b13-20020a056358520d00b0017f1d346253mr268628rwa.15.1711515063614; Tue, 26 Mar 2024 21:51:03 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id e31-20020a63501f000000b005e4666261besm8351500pgb.50.2024.03.26.21.51.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Mar 2024 21:51:02 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Jisheng Zhang , Yunhui Cui , Samuel Holland Subject: [PATCH v6 13/13] riscv: mm: Always use an ASID to flush mm contexts Date: Tue, 26 Mar 2024 21:49:54 -0700 Message-ID: <20240327045035.368512-14-samuel.holland@sifive.com> X-Mailer: git-send-email 2.43.1 In-Reply-To: <20240327045035.368512-1-samuel.holland@sifive.com> References: <20240327045035.368512-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240326_215110_672752_218BC99B X-CRM114-Status: GOOD ( 11.35 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Even if multiple ASIDs are not supported, using the single-ASID variant of the sfence.vma instruction preserves TLB entries for global (kernel) pages. So it is always more efficient to use the single-ASID code path. Reviewed-by: Alexandre Ghiti Signed-off-by: Samuel Holland --- (no changes since v5) Changes in v5: - Leave use_asid_allocator declared in asm/mmu_context.h Changes in v4: - There is now only one copy of __flush_tlb_range() Changes in v2: - Update both copies of __flush_tlb_range() arch/riscv/mm/tlbflush.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 35266dd9a9a2..44e7ed4e194f 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -109,8 +109,7 @@ static void __flush_tlb_range(struct cpumask *cmask, unsigned long asid, static inline unsigned long get_mm_asid(struct mm_struct *mm) { - return static_branch_unlikely(&use_asid_allocator) ? - cntx2asid(atomic_long_read(&mm->context.id)) : FLUSH_TLB_NO_ASID; + return cntx2asid(atomic_long_read(&mm->context.id)); } void flush_tlb_mm(struct mm_struct *mm)