From patchwork Sun Oct 1 10:34:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jisheng Zhang X-Patchwork-Id: 13405360 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EA258E748ED for ; Sun, 1 Oct 2023 10:46:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=/CmIeSc75OtY3Rj/vuDHL1mR8LrsaDI1eFhtZHTc1BM=; b=4yUXqmE7R9F/AB PoeVRAkwLVkgZ4AFAPudCDLiLmyLyy+Z/tgxkXaaHT3rgG3/kCGLOTv3BlByEBXPUnbd8PeEIVWqf xNRw95c4EQHI4xCVR3UkqOVR6rzetbGRTJz+Iy+7ytfAFpayTuPYVganyjxKTmJv3eVvj+MG8AYxs wNWxzx5PDp03/XZQZpl9RVj3d1QBQCGdf3vPQ7dOC+pR9U62z+C/xG38j/T5N2T6EYjjy00aH6XiY oQtMPxalysyWugKGEC/a+JsBeeSaXjVum0+dnFUrEEr9jnz6U3cdObA8jqyChSGzZ7yM79lSrcrEC z0zI1UgxnCUpYTcDlHMg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qmtyH-00Ay4D-1X; Sun, 01 Oct 2023 10:46:45 +0000 Received: from sin.source.kernel.org ([145.40.73.55]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qmtyD-00Ay2n-2K for linux-riscv@lists.infradead.org; Sun, 01 Oct 2023 10:46:43 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id D6A0ACE0ADD; Sun, 1 Oct 2023 10:46:39 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A5046C433CA; Sun, 1 Oct 2023 10:46:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1696157199; bh=VjY251ue30j0sLCI7mKUqPHPf0ahphTmXpRrxHQRsJw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=B4J53C/15sVQl3KD8Z4lPUr+2IAotBBbvcrc6Pt9muHulH7PMqMoE+SFV2COh2/Yb lWwwHbsHYWgLoOOr3Y1h9OrVIz5awnNeIjhomNVSI/51jGmUejJLP4sRYF8HHyWuZQ c767BYja53ZR7gy/wfHt2PiNLh83QVK48msSKgbj02qVzdyoGg5UHYX3CFB0F7KLEY QSvx3pvoH0up5ec1mlU4mM4preuE3VHldfGj4tVlnba1rJr8KiRS/aGGj6JLvDDYGW I6YzrZTPT6Vs8IsPA/F6bRIRBtyup+5vju2BbanUHBHGUbud6jvOa8FBivpthiqHC4 H3G2LKZdMvVag== From: Jisheng Zhang To: Paul Walmsley , Palmer Dabbelt , Albert Ou Cc: Guo Ren , linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, Emil Renner Berthing Subject: [PATCH v2 1/2] riscv: errata: thead: use riscv_nonstd_cache_ops for CMO Date: Sun, 1 Oct 2023 18:34:32 +0800 Message-Id: <20231001103433.3187-2-jszhang@kernel.org> X-Mailer: git-send-email 2.40.0 In-Reply-To: <20231001103433.3187-1-jszhang@kernel.org> References: <20231001103433.3187-1-jszhang@kernel.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231001_034642_106303_3BEDCC01 X-CRM114-Status: GOOD ( 14.73 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Previously, we use alternative mechanism to dynamically patch the CMO operations for THEAD C906/C910 during boot for performance reason. But as pointed out by Arnd, "there is already a significant cost in accessing the invalidated cache lines afterwards, which is likely going to be much higher than the cost of an indirect branch". And indeed, there's no performance difference with GMAC and EMMC per my test on Sipeed Lichee Pi 4A board. Use riscv_nonstd_cache_ops for THEAD C906/C910 CMO to simplify the alternative code, and to acchieve Arnd's goal -- "I think moving the THEAD ops at the same level as all nonstandard operations makes sense, but I'd still leave CMO as an explicit fast path that avoids the indirect branch. This seems like the right thing to do both for readability and for platforms on which the indirect branch has a noticeable overhead." Signed-off-by: Jisheng Zhang Tested-by: Emil Renner Berthing Reviewed-by: Conor Dooley --- arch/riscv/Kconfig.errata | 1 + arch/riscv/errata/thead/errata.c | 75 +++++++++++++++++++++++++++- arch/riscv/include/asm/errata_list.h | 50 +++---------------- 3 files changed, 80 insertions(+), 46 deletions(-) diff --git a/arch/riscv/Kconfig.errata b/arch/riscv/Kconfig.errata index 566bcefeab50..d7972914f9de 100644 --- a/arch/riscv/Kconfig.errata +++ b/arch/riscv/Kconfig.errata @@ -78,6 +78,7 @@ config ERRATA_THEAD_CMO bool "Apply T-Head cache management errata" depends on ERRATA_THEAD && MMU select RISCV_DMA_NONCOHERENT + select RISCV_NONSTANDARD_CACHE_OPS default y help This will apply the cache management errata to handle the diff --git a/arch/riscv/errata/thead/errata.c b/arch/riscv/errata/thead/errata.c index 0554ed4bf087..3fefeb1b456e 100644 --- a/arch/riscv/errata/thead/errata.c +++ b/arch/riscv/errata/thead/errata.c @@ -12,8 +12,10 @@ #include #include #include +#include #include #include +#include #include #include @@ -33,6 +35,75 @@ static bool errata_probe_pbmt(unsigned int stage, return false; } +/* + * dcache.ipa rs1 (invalidate, physical address) + * | 31 - 25 | 24 - 20 | 19 - 15 | 14 - 12 | 11 - 7 | 6 - 0 | + * 0000001 01010 rs1 000 00000 0001011 + * dache.iva rs1 (invalida, virtual address) + * 0000001 00110 rs1 000 00000 0001011 + * + * dcache.cpa rs1 (clean, physical address) + * | 31 - 25 | 24 - 20 | 19 - 15 | 14 - 12 | 11 - 7 | 6 - 0 | + * 0000001 01001 rs1 000 00000 0001011 + * dcache.cva rs1 (clean, virtual address) + * 0000001 00101 rs1 000 00000 0001011 + * + * dcache.cipa rs1 (clean then invalidate, physical address) + * | 31 - 25 | 24 - 20 | 19 - 15 | 14 - 12 | 11 - 7 | 6 - 0 | + * 0000001 01011 rs1 000 00000 0001011 + * dcache.civa rs1 (... virtual address) + * 0000001 00111 rs1 000 00000 0001011 + * + * sync.s (make sure all cache operations finished) + * | 31 - 25 | 24 - 20 | 19 - 15 | 14 - 12 | 11 - 7 | 6 - 0 | + * 0000000 11001 00000 000 00000 0001011 + */ +#define THEAD_inval_A0 ".long 0x0265000b" +#define THEAD_clean_A0 ".long 0x0255000b" +#define THEAD_flush_A0 ".long 0x0275000b" +#define THEAD_SYNC_S ".long 0x0190000b" + +#define THEAD_CMO_OP(_op, _start, _size, _cachesize) \ +asm volatile("mv a0, %1\n\t" \ + "j 2f\n\t" \ + "3:\n\t" \ + THEAD_##_op##_A0 "\n\t" \ + "add a0, a0, %0\n\t" \ + "2:\n\t" \ + "bltu a0, %2, 3b\n\t" \ + THEAD_SYNC_S \ + : : "r"(_cachesize), \ + "r"((unsigned long)(_start) & ~((_cachesize) - 1UL)), \ + "r"((unsigned long)(_start) + (_size)) \ + : "a0") + +static void thead_errata_cache_inv(phys_addr_t paddr, size_t size) +{ + void *vaddr = phys_to_virt(paddr); + + THEAD_CMO_OP(inval, vaddr, size, riscv_cbom_block_size); +} + +static void thead_errata_cache_wback(phys_addr_t paddr, size_t size) +{ + void *vaddr = phys_to_virt(paddr); + + THEAD_CMO_OP(clean, vaddr, size, riscv_cbom_block_size); +} + +static void thead_errata_cache_wback_inv(phys_addr_t paddr, size_t size) +{ + void *vaddr = phys_to_virt(paddr); + + THEAD_CMO_OP(flush, vaddr, size, riscv_cbom_block_size); +} + +static const struct riscv_nonstd_cache_ops thead_errata_cmo_ops = { + .wback = &thead_errata_cache_wback, + .inv = &thead_errata_cache_inv, + .wback_inv = &thead_errata_cache_wback_inv, +}; + static bool errata_probe_cmo(unsigned int stage, unsigned long arch_id, unsigned long impid) { @@ -48,6 +119,7 @@ static bool errata_probe_cmo(unsigned int stage, if (stage == RISCV_ALTERNATIVES_BOOT) { riscv_cbom_block_size = L1_CACHE_BYTES; riscv_noncoherent_supported(); + riscv_noncoherent_register_cache_ops(&thead_errata_cmo_ops); } return true; @@ -77,8 +149,7 @@ static u32 thead_errata_probe(unsigned int stage, if (errata_probe_pbmt(stage, archid, impid)) cpu_req_errata |= BIT(ERRATA_THEAD_PBMT); - if (errata_probe_cmo(stage, archid, impid)) - cpu_req_errata |= BIT(ERRATA_THEAD_CMO); + errata_probe_cmo(stage, archid, impid); if (errata_probe_pmu(stage, archid, impid)) cpu_req_errata |= BIT(ERRATA_THEAD_PMU); diff --git a/arch/riscv/include/asm/errata_list.h b/arch/riscv/include/asm/errata_list.h index b55b434f0059..ea33288f8a25 100644 --- a/arch/riscv/include/asm/errata_list.h +++ b/arch/riscv/include/asm/errata_list.h @@ -24,9 +24,8 @@ #ifdef CONFIG_ERRATA_THEAD #define ERRATA_THEAD_PBMT 0 -#define ERRATA_THEAD_CMO 1 -#define ERRATA_THEAD_PMU 2 -#define ERRATA_THEAD_NUMBER 3 +#define ERRATA_THEAD_PMU 1 +#define ERRATA_THEAD_NUMBER 2 #endif #ifdef __ASSEMBLY__ @@ -94,54 +93,17 @@ asm volatile(ALTERNATIVE( \ #define ALT_THEAD_PMA(_val) #endif -/* - * dcache.ipa rs1 (invalidate, physical address) - * | 31 - 25 | 24 - 20 | 19 - 15 | 14 - 12 | 11 - 7 | 6 - 0 | - * 0000001 01010 rs1 000 00000 0001011 - * dache.iva rs1 (invalida, virtual address) - * 0000001 00110 rs1 000 00000 0001011 - * - * dcache.cpa rs1 (clean, physical address) - * | 31 - 25 | 24 - 20 | 19 - 15 | 14 - 12 | 11 - 7 | 6 - 0 | - * 0000001 01001 rs1 000 00000 0001011 - * dcache.cva rs1 (clean, virtual address) - * 0000001 00101 rs1 000 00000 0001011 - * - * dcache.cipa rs1 (clean then invalidate, physical address) - * | 31 - 25 | 24 - 20 | 19 - 15 | 14 - 12 | 11 - 7 | 6 - 0 | - * 0000001 01011 rs1 000 00000 0001011 - * dcache.civa rs1 (... virtual address) - * 0000001 00111 rs1 000 00000 0001011 - * - * sync.s (make sure all cache operations finished) - * | 31 - 25 | 24 - 20 | 19 - 15 | 14 - 12 | 11 - 7 | 6 - 0 | - * 0000000 11001 00000 000 00000 0001011 - */ -#define THEAD_inval_A0 ".long 0x0265000b" -#define THEAD_clean_A0 ".long 0x0255000b" -#define THEAD_flush_A0 ".long 0x0275000b" -#define THEAD_SYNC_S ".long 0x0190000b" - #define ALT_CMO_OP(_op, _start, _size, _cachesize) \ -asm volatile(ALTERNATIVE_2( \ - __nops(6), \ +asm volatile(ALTERNATIVE( \ + __nops(5), \ "mv a0, %1\n\t" \ "j 2f\n\t" \ "3:\n\t" \ CBO_##_op(a0) \ "add a0, a0, %0\n\t" \ "2:\n\t" \ - "bltu a0, %2, 3b\n\t" \ - "nop", 0, RISCV_ISA_EXT_ZICBOM, CONFIG_RISCV_ISA_ZICBOM, \ - "mv a0, %1\n\t" \ - "j 2f\n\t" \ - "3:\n\t" \ - THEAD_##_op##_A0 "\n\t" \ - "add a0, a0, %0\n\t" \ - "2:\n\t" \ - "bltu a0, %2, 3b\n\t" \ - THEAD_SYNC_S, THEAD_VENDOR_ID, \ - ERRATA_THEAD_CMO, CONFIG_ERRATA_THEAD_CMO) \ + "bltu a0, %2, 3b\n\t", \ + 0, RISCV_ISA_EXT_ZICBOM, CONFIG_RISCV_ISA_ZICBOM) \ : : "r"(_cachesize), \ "r"((unsigned long)(_start) & ~((_cachesize) - 1UL)), \ "r"((unsigned long)(_start) + (_size)) \ From patchwork Sun Oct 1 10:34:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jisheng Zhang X-Patchwork-Id: 13405361 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 16ED8E748E9 for ; Sun, 1 Oct 2023 10:46:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=L6vXunRpdm4ugUupIegIm5r+KSK18CTc+qibxqWABjs=; b=Tnjx0wcZB8m9mr uznpUC3JDPmuE/QPn4FNRd/NCJdnfCz6RF24Abpliue/5jOZhasuNIAb9gfJP9kzJi9G1gfNbaVrv 4cOqy2/5zjUd7EqVysnVf4aVUEfQ54a703Gohf6HGqK30TW6bUfWYbJOxnTi5eHaykQtcCSD8A1s5 UAgHNDB+ticdQU0DrpvlGDGvs98F1ySBK9LS99dJidUgtoAR+SxFA1BOIMM3ZwwW428EOS5sh4OH0 CFbxRu1VSAGQKwVBHQyyDzldxgN10QU9rcw7uEB3uc0dtpG8Sfe9T4rkcjRiwI5PydbL/TsWKSplj pRitPQP1AdVgIHqpXhsA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qmtyM-00Ay5R-0a; Sun, 01 Oct 2023 10:46:50 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qmtyJ-00Ay3M-0O for linux-riscv@lists.infradead.org; Sun, 01 Oct 2023 10:46:49 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by ams.source.kernel.org (Postfix) with ESMTP id 40DF1B808C0; Sun, 1 Oct 2023 10:46:42 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 87CD2C433C7; Sun, 1 Oct 2023 10:46:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1696157200; bh=qzR32gMwqLa17IpLtlTXacrXoBXks+o3rMWn8BtgPT4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=oWMkJN6TASGuqSj1ckYx9rlqW8PyzufTainBHUKNuDudf/9J6MqGNc/nnhFRYY6VB sFiPvIg7hp8ixx5AsnFiX9vCoXlQwWcQjD8eegcnSBAEH7+YL1zii4F+YHCdTMQn7m Pgnrzh/SWyow94vQq6hDjKOl4whgjOBgtr8kvAVGnrEyIWHHQMNwUTbOfGPlSUMTBn OASAXWm8fjDenqygce35YaWFaQkunJ7OR7iBX0HjRwLLjC0b+CIg00cgnup9iqrgbA jeOTryhmi2osIN/R6hjNGE+wMdUQSPWzi64R5altT2reUbJCzugsy3j7fPefdQiAZK n0pYF0hK2bE+g== From: Jisheng Zhang To: Paul Walmsley , Palmer Dabbelt , Albert Ou Cc: Guo Ren , linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 2/2] riscv: errata: thead: use pa based instructions for CMO Date: Sun, 1 Oct 2023 18:34:33 +0800 Message-Id: <20231001103433.3187-3-jszhang@kernel.org> X-Mailer: git-send-email 2.40.0 In-Reply-To: <20231001103433.3187-1-jszhang@kernel.org> References: <20231001103433.3187-1-jszhang@kernel.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231001_034647_306246_8DC88480 X-CRM114-Status: UNSURE ( 7.77 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org T-HEAD CPUs such as C906/C910/C920 support phy address based CMO, use them so that we don't need to convert to virt address. Signed-off-by: Jisheng Zhang --- arch/riscv/errata/thead/errata.c | 18 ++++++------------ 1 file changed, 6 insertions(+), 12 deletions(-) diff --git a/arch/riscv/errata/thead/errata.c b/arch/riscv/errata/thead/errata.c index 3fefeb1b456e..632557f36b19 100644 --- a/arch/riscv/errata/thead/errata.c +++ b/arch/riscv/errata/thead/errata.c @@ -58,9 +58,9 @@ static bool errata_probe_pbmt(unsigned int stage, * | 31 - 25 | 24 - 20 | 19 - 15 | 14 - 12 | 11 - 7 | 6 - 0 | * 0000000 11001 00000 000 00000 0001011 */ -#define THEAD_inval_A0 ".long 0x0265000b" -#define THEAD_clean_A0 ".long 0x0255000b" -#define THEAD_flush_A0 ".long 0x0275000b" +#define THEAD_inval_A0 ".long 0x02a5000b" +#define THEAD_clean_A0 ".long 0x0295000b" +#define THEAD_flush_A0 ".long 0x02b5000b" #define THEAD_SYNC_S ".long 0x0190000b" #define THEAD_CMO_OP(_op, _start, _size, _cachesize) \ @@ -79,23 +79,17 @@ asm volatile("mv a0, %1\n\t" \ static void thead_errata_cache_inv(phys_addr_t paddr, size_t size) { - void *vaddr = phys_to_virt(paddr); - - THEAD_CMO_OP(inval, vaddr, size, riscv_cbom_block_size); + THEAD_CMO_OP(inval, paddr, size, riscv_cbom_block_size); } static void thead_errata_cache_wback(phys_addr_t paddr, size_t size) { - void *vaddr = phys_to_virt(paddr); - - THEAD_CMO_OP(clean, vaddr, size, riscv_cbom_block_size); + THEAD_CMO_OP(clean, paddr, size, riscv_cbom_block_size); } static void thead_errata_cache_wback_inv(phys_addr_t paddr, size_t size) { - void *vaddr = phys_to_virt(paddr); - - THEAD_CMO_OP(flush, vaddr, size, riscv_cbom_block_size); + THEAD_CMO_OP(flush, paddr, size, riscv_cbom_block_size); } static const struct riscv_nonstd_cache_ops thead_errata_cmo_ops = {