From patchwork Thu Apr 18 11:49:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xu Lu X-Patchwork-Id: 13634568 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 72E29C4345F for ; Thu, 18 Apr 2024 11:50:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=I5Gv8Ks59zLnP4qNj9mKyKcLYfoGxn1SHJrcwQ8BBaY=; b=UmZA9m+1WfxFmo JlVqhXegr3giILcRFY5V+WiOIjU6XAMkVTPwZxaL1UHe5F7NcOOloDrxyH0s8N+jdylYdCURGuMNp 5j3Bp12/W40Bv6Mri5cjs4teUwWo9270elyimrXhRpoCu8I4dV10VIFgA1SWMW0WUZfEelcHOiuBX jN6LrNdvX7vN1l7vja2XnEKAien4QSujHU/u57knNxTMu040yCrzCAnnEaf3QCpOXaTapdNmq50C9 VmMLmwFX61Ne7+NfJ/z1OzhJOjD7HpG2bvhTUGGv5dwbP+MFfvYOSAsPyKFZW8RYO7cZuHK6Kiv/o Gpn9rDniFFV79kx6wnlA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rxQHH-000000023gK-0dH9; Thu, 18 Apr 2024 11:50:07 +0000 Received: from mail-pg1-x530.google.com ([2607:f8b0:4864:20::530]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rxQHD-000000023cc-2ztH for linux-riscv@lists.infradead.org; Thu, 18 Apr 2024 11:50:06 +0000 Received: by mail-pg1-x530.google.com with SMTP id 41be03b00d2f7-5bdbe2de25fso521043a12.3 for ; Thu, 18 Apr 2024 04:49:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1713440998; x=1714045798; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=GJdeD5iiUhWW9ibJfrIuQV6O5oUuL7RJYhdQuUEPhzw=; b=f9W0AL6t4wBe6M4jv0JTNjIY94CwBNKP9a0Y84N6S/Spvi5sRw2/VfD6uqNOviBGIL 8O37cFUBTSWz9oFTLaM1UDhDQWwxAOI3D6eaie/4HKUrrixAxXxXJxKpZ7omm6VhCevu PV3Q5+JZm9vba1vXw//lrc/A32XDqrDbcFeiYF4dDKlvlUvyfcuQ8aJ41l2mmNpVtZsN YF9pFB8hFeuYKbcqWI4I8rsdRHvXJR5OIkD1bMuha413+XevGjY5p11noM9SAd1O3lxo ePJLtPXa3GUUZU/foxSJAXY725j6tUPaMVqsmKewQZmBAY8Ne4jbCB4qfKUZ6yG3jUHD eg0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713440998; x=1714045798; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GJdeD5iiUhWW9ibJfrIuQV6O5oUuL7RJYhdQuUEPhzw=; b=TvDpr8i/ESaontvwYPM++JhZuCto48X6/kroUjrM/PiQvmRsa5GveAPFxzGSZsm9SC Y2E9i8A8sKnSdN9WQ444aYsCzBptFzO3EpFcDV6KQw5ZNOjrgFLIw4Q0/6ws9UHbYj7G lqBUTzwhj+85TOjgHFbGm556NjfsLN/YvGo9PzFxk70B7Iv/DGjU44voJV4fCMaC4dey 970X7/CApeB9hsAyqvRiZs6mhxXAj1Yh/ebmtvUaaWhTIt92mvknD5ZHBPaabx1tdf/y SBL/nB/SI9H54ykqDqBrbk3MJvTjPI10pANbqpYqJcBdgKRv9SCnE3v4fmzqfHZnvVQ1 lNkQ== X-Gm-Message-State: AOJu0YxhkLR7Gb854W1SoOTnUyvJfOiegFKR7Bsjp+5Sgu4MUukvt+Vl gIJNXaKzXtj3/HUQk1YZfpJwyncJRQLfjU2SrlzRqJzXBo7JBgbpjeNTVB9S/Ec= X-Google-Smtp-Source: AGHT+IEhQRyarVDWCbICPcIgg9trabajpb3eBKbcprMDbL7Q13Be7F8tBjQiW5L97nkQIRBDFgy6Pg== X-Received: by 2002:a17:90a:e2c8:b0:2a0:7758:31ac with SMTP id fr8-20020a17090ae2c800b002a0775831acmr2355963pjb.25.1713440997972; Thu, 18 Apr 2024 04:49:57 -0700 (PDT) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.55]) by smtp.gmail.com with ESMTPSA id y7-20020a17090aa40700b002abdb19f499sm1032293pjp.26.2024.04.18.04.49.54 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 18 Apr 2024 04:49:57 -0700 (PDT) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, andy.chiu@sifive.com, guoren@kernel.org Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, lihangjing@bytedance.com, dengliang.1214@bytedance.com, xieyongji@bytedance.com, chaiwen.cc@bytedance.com, Xu Lu Subject: [RFC 1/2] riscv: process: Introduce idle thread using Zawrs extension Date: Thu, 18 Apr 2024 19:49:41 +0800 Message-Id: <20240418114942.52770-2-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20240418114942.52770-1-luxu.kernel@bytedance.com> References: <20240418114942.52770-1-luxu.kernel@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240418_045003_831647_5AA022B2 X-CRM114-Status: GOOD ( 17.44 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org The Zawrs extension introduces a new instruction WRS.NTO, which will register a reservation set and causes the hart to temporarily stall execution in a low-power state until a store occurs to the reservation set or an interrupt is observed. This commit implements new version of idle thread for RISC-V via Zawrs extension. Signed-off-by: Xu Lu Reviewed-by: Hangjing Li Reviewed-by: Liang Deng Reviewed-by: Wen Chai --- arch/riscv/Kconfig | 24 +++++++++++++++++ arch/riscv/include/asm/cpuidle.h | 11 +------- arch/riscv/include/asm/hwcap.h | 1 + arch/riscv/include/asm/processor.h | 17 +++++++++++++ arch/riscv/kernel/cpu.c | 5 ++++ arch/riscv/kernel/cpufeature.c | 1 + arch/riscv/kernel/process.c | 41 +++++++++++++++++++++++++++++- 7 files changed, 89 insertions(+), 11 deletions(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index be09c8836d56..a0d344e9803f 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -19,6 +19,7 @@ config RISCV select ARCH_ENABLE_SPLIT_PMD_PTLOCK if PGTABLE_LEVELS > 2 select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE select ARCH_HAS_BINFMT_FLAT + select ARCH_HAS_CPU_FINALIZE_INIT select ARCH_HAS_CURRENT_STACK_POINTER select ARCH_HAS_DEBUG_VIRTUAL if MMU select ARCH_HAS_DEBUG_VM_PGTABLE @@ -525,6 +526,20 @@ config RISCV_ISA_SVPBMT If you don't know what to do here, say Y. +config RISCV_ISA_ZAWRS + bool "Zawrs extension support for wait-on-reservation-set instructions" + depends on RISCV_ALTERNATIVE + default y + help + Adds support to dynamically detect the presence of the Zawrs + extension and enable its usage. + + The Zawrs extension defines a pair of instructions to be used + in polling loops that allows a core to enter a low-power state + and wait on a store to a memory location. + + If you don't know what to do here, say Y. + config TOOLCHAIN_HAS_V bool default y @@ -1075,6 +1090,15 @@ endmenu # "Power management options" menu "CPU Power Management" +config RISCV_ZAWRS_IDLE + bool "Idle thread using ZAWRS extensions" + depends on RISCV_ISA_ZAWRS + default y + help + Adds support to implement idle thread using ZAWRS extension. + + If you don't know what to do here, say Y. + source "drivers/cpuidle/Kconfig" source "drivers/cpufreq/Kconfig" diff --git a/arch/riscv/include/asm/cpuidle.h b/arch/riscv/include/asm/cpuidle.h index 71fdc607d4bc..94c9ecb46571 100644 --- a/arch/riscv/include/asm/cpuidle.h +++ b/arch/riscv/include/asm/cpuidle.h @@ -10,15 +10,6 @@ #include #include -static inline void cpu_do_idle(void) -{ - /* - * Add mb() here to ensure that all - * IO/MEM accesses are completed prior - * to entering WFI. - */ - mb(); - wait_for_interrupt(); -} +void cpu_do_idle(void); #endif diff --git a/arch/riscv/include/asm/hwcap.h b/arch/riscv/include/asm/hwcap.h index e17d0078a651..5b358c3cf212 100644 --- a/arch/riscv/include/asm/hwcap.h +++ b/arch/riscv/include/asm/hwcap.h @@ -81,6 +81,7 @@ #define RISCV_ISA_EXT_ZTSO 72 #define RISCV_ISA_EXT_ZACAS 73 #define RISCV_ISA_EXT_XANDESPMU 74 +#define RISCV_ISA_EXT_ZAWRS 75 #define RISCV_ISA_EXT_XLINUXENVCFG 127 diff --git a/arch/riscv/include/asm/processor.h b/arch/riscv/include/asm/processor.h index 0faf5f161f1e..1143367de8c6 100644 --- a/arch/riscv/include/asm/processor.h +++ b/arch/riscv/include/asm/processor.h @@ -157,6 +157,21 @@ static inline void wait_for_interrupt(void) __asm__ __volatile__ ("wfi"); } +static inline void wrs_nto(unsigned long *addr) +{ + int val; + + __asm__ __volatile__( +#ifdef CONFIG_64BIT + "lr.d %[p], %[v] \n\t" +#else + "lr.w %[p], %[v] \n\t" +#endif + ".long 0x00d00073 \n\t" + : [p] "=&r" (val), [v] "+A" (*addr) + : : "memory"); +} + extern phys_addr_t dma32_phys_limit; struct device_node; @@ -183,6 +198,8 @@ extern int set_unalign_ctl(struct task_struct *tsk, unsigned int val); #define GET_UNALIGN_CTL(tsk, addr) get_unalign_ctl((tsk), (addr)) #define SET_UNALIGN_CTL(tsk, val) set_unalign_ctl((tsk), (val)) +extern void select_idle_routine(void); + #endif /* __ASSEMBLY__ */ #endif /* _ASM_RISCV_PROCESSOR_H */ diff --git a/arch/riscv/kernel/cpu.c b/arch/riscv/kernel/cpu.c index d11d6320fb0d..69cebd41f5f3 100644 --- a/arch/riscv/kernel/cpu.c +++ b/arch/riscv/kernel/cpu.c @@ -22,6 +22,11 @@ bool arch_match_cpu_phys_id(int cpu, u64 phys_id) return phys_id == cpuid_to_hartid_map(cpu); } +void __init arch_cpu_finalize_init(void) +{ + select_idle_routine(); +} + /* * Returns the hart ID of the given device tree node, or -ENODEV if the node * isn't an enabled and valid RISC-V hart node. diff --git a/arch/riscv/kernel/cpufeature.c b/arch/riscv/kernel/cpufeature.c index 3ed2359eae35..c080e6ca54ba 100644 --- a/arch/riscv/kernel/cpufeature.c +++ b/arch/riscv/kernel/cpufeature.c @@ -305,6 +305,7 @@ const struct riscv_isa_ext_data riscv_isa_ext[] = { __RISCV_ISA_EXT_DATA(svnapot, RISCV_ISA_EXT_SVNAPOT), __RISCV_ISA_EXT_DATA(svpbmt, RISCV_ISA_EXT_SVPBMT), __RISCV_ISA_EXT_DATA(xandespmu, RISCV_ISA_EXT_XANDESPMU), + __RISCV_ISA_EXT_DATA(zawrs, RISCV_ISA_EXT_ZAWRS), }; const size_t riscv_isa_ext_count = ARRAY_SIZE(riscv_isa_ext); diff --git a/arch/riscv/kernel/process.c b/arch/riscv/kernel/process.c index 92922dbd5b5c..9f0f7b888bc1 100644 --- a/arch/riscv/kernel/process.c +++ b/arch/riscv/kernel/process.c @@ -15,6 +15,7 @@ #include #include #include +#include #include #include @@ -37,11 +38,49 @@ EXPORT_SYMBOL(__stack_chk_guard); extern asmlinkage void ret_from_fork(void); -void arch_cpu_idle(void) +static __cpuidle void default_idle(void) +{ + /* + * Add mb() here to ensure that all + * IO/MEM accesses are completed prior + * to entering WFI. + */ + mb(); + wait_for_interrupt(); +} + +static __cpuidle void wrs_idle(void) +{ + /* + * Add mb() here to ensure that all + * IO/MEM accesses are completed prior + * to entering WRS.NTO. + */ + mb(); + wrs_nto(¤t_thread_info()->flags); +} + +DEFINE_STATIC_CALL_NULL(riscv_idle, default_idle); + +void __cpuidle cpu_do_idle(void) +{ + static_call(riscv_idle)(); +} + +void __cpuidle arch_cpu_idle(void) { cpu_do_idle(); } +void __init select_idle_routine(void) +{ + if (IS_ENABLED(CONFIG_RISCV_ZAWRS_IDLE) && + riscv_has_extension_likely(RISCV_ISA_EXT_ZAWRS)) + static_call_update(riscv_idle, wrs_idle); + else + static_call_update(riscv_idle, default_idle); +} + int set_unalign_ctl(struct task_struct *tsk, unsigned int val) { if (!unaligned_ctl_available()) From patchwork Thu Apr 18 11:49:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xu Lu X-Patchwork-Id: 13634569 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 18BCAC05052 for ; Thu, 18 Apr 2024 11:50:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=1x4X3uxEU82DrCbtP10iBjswK7874Eq1BvZ8k2sEsJU=; b=IAiA1kGteLzO1I kEXejXhD/gewXhsn22cJjIPw/Uj6eMtJzUeGSrpFCQxAHxNGKQC8qWGpW6dCIokSGpdsqJj8akKZZ o4Md/VMC8Rgj/eoBshwCbqXVBwFk5dsMhvweAcVJyxkFBIm8aAVOON6EZnJxzEZuJI77rs1tMqC2I GXTVtzCu43skH0kmODkRNKRxku/n+wQtTO11aZbY7nb9Q1cfvXyfTp2LV6fJp4k9bA+Pb+DsNs49Z FWLY3mwrC0x6LP7Yx0/MGDZZRzRo7OGUI/0xaxMv8nEu/5lhbxmnxr1ZbB8XIQsXjJ0E0BHrm7RI8 5FVqpgfMLVb0kbHmNwIQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rxQHL-000000023i4-0uGI; Thu, 18 Apr 2024 11:50:11 +0000 Received: from mail-pg1-x531.google.com ([2607:f8b0:4864:20::531]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rxQHE-000000023e1-3gb8 for linux-riscv@lists.infradead.org; Thu, 18 Apr 2024 11:50:06 +0000 Received: by mail-pg1-x531.google.com with SMTP id 41be03b00d2f7-5f415fd71f8so552538a12.3 for ; Thu, 18 Apr 2024 04:50:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1713441002; x=1714045802; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=zm558O2QVDPk1eRrVljSEfZNVakmWiw+kO/s+2oVqPM=; b=Dz40QhprgOWI0GMC5EEPOHciie6OTZ8n+ZufyTBltKh/0uQO+BylarLJqJOryCUNMY ZnFYKBlUeqWhxupLUkSEn2jUVaFihbWI6QSfolUQHX7sQw70zBQj1OLjXl3ErFCENHmq j0F6rhrwuIdmduFVoqrRuCEfLwBQjLITiiu+j/2R5KaP/+k1C+j/kVMqwJvIPdzoBcKo F87Cjths5JUwic1A6wIQJXHfJx25GAb9WXYooxEgJSE/StmIYwWW4taY10nMhdVqOp43 OQoACKWq0Hf9RVAljzbMLIKGwghzUuw9OVNOw8jNn41Cx89S2erx0j5mdBSAOj53gcrt STAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713441002; x=1714045802; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=zm558O2QVDPk1eRrVljSEfZNVakmWiw+kO/s+2oVqPM=; b=KL5Pj70bCKn1sFIXwhmAr82C+YCOsJNe+d+re6BklF/uL3hy+693yM5w+YF4NwEbIt dAdrRgCt/dgrDaa9hscdWozcSNxjfjD6RBDNk3vifvvPlwb5JvtLd0qSDAP+0zi8DNb6 3NdWhyGR+/MHyR20RINWoN9Salm4beuPNn+fW6SP3WoKVpaF/bqy9b/CmsnrWuE/mbsa i3/aHi8ThBSV0FJZE/+b6mheZw5BnKvslIV/Nc4CmB5d1yNyAqhjGiHn3fwpeBFfjXha zgtZlKnZEnV45FmTRt+XeDSTR/iLBDyvISFPaWSJfdTtIcE9E0jPUaaKWXYO3Edyv/3q JNVQ== X-Gm-Message-State: AOJu0YyTFRqQSh4NIr8UBbHcHW4Otibvr5kgm+h5d6vcD7Ca//yQN3ee 0eebm8REAFq3fSiL1pT/FnvMRjOi2VF0iPYOi7gBfTNgP4zR0JfTtooxXK6dYIM= X-Google-Smtp-Source: AGHT+IG08hqpF8uwiPrN1zWB27zGM+lpl666QMpL8Ui14+w3YozU4FioSKQ64+ALHNkfMcQdGOYAQA== X-Received: by 2002:a17:90a:fd82:b0:2ab:9c0d:da69 with SMTP id cx2-20020a17090afd8200b002ab9c0dda69mr1994655pjb.49.1713441002289; Thu, 18 Apr 2024 04:50:02 -0700 (PDT) Received: from J9GPGXL7NT.bytedance.net ([61.213.176.55]) by smtp.gmail.com with ESMTPSA id y7-20020a17090aa40700b002abdb19f499sm1032293pjp.26.2024.04.18.04.49.58 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 18 Apr 2024 04:50:02 -0700 (PDT) From: Xu Lu To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, andy.chiu@sifive.com, guoren@kernel.org Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, lihangjing@bytedance.com, dengliang.1214@bytedance.com, xieyongji@bytedance.com, chaiwen.cc@bytedance.com, Xu Lu Subject: [RFC 2/2] riscv: Use Zawrs to accelerate IPI to idle cpu Date: Thu, 18 Apr 2024 19:49:42 +0800 Message-Id: <20240418114942.52770-3-luxu.kernel@bytedance.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20240418114942.52770-1-luxu.kernel@bytedance.com> References: <20240418114942.52770-1-luxu.kernel@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240418_045004_947756_5D5EDB9D X-CRM114-Status: GOOD ( 14.30 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org When sending IPI to a cpu which has entered idle state using Zawrs extension, there is no need to send a physical software interrupt. Instead, we can write the IPI information to the address reserved by target cpu, which will wake it from WRS.NTO. Then the target cpu can handle the IPI directly without falling into traditional interrupt handling routine. Signed-off-by: Xu Lu --- arch/riscv/include/asm/processor.h | 14 +++++++ arch/riscv/include/asm/smp.h | 14 +++++++ arch/riscv/kernel/process.c | 65 +++++++++++++++++++++++++++++- arch/riscv/kernel/smp.c | 39 ++++++++++++------ 4 files changed, 118 insertions(+), 14 deletions(-) diff --git a/arch/riscv/include/asm/processor.h b/arch/riscv/include/asm/processor.h index 1143367de8c6..76091cf2e8be 100644 --- a/arch/riscv/include/asm/processor.h +++ b/arch/riscv/include/asm/processor.h @@ -172,6 +172,20 @@ static inline void wrs_nto(unsigned long *addr) : : "memory"); } +static inline void wrs_nto_if(int *addr, int val) +{ + int prev; + + __asm__ __volatile__( + "lr.w %[p], %[a] \n\t" + "bne %[p], %[v], 1f \n\t" + ".long 0x00d00073 \n\t" + "1: \n\t" + : [p] "=&r" (prev), [a] "+A" (*addr) + : [v] "r" (val) + : "memory"); +} + extern phys_addr_t dma32_phys_limit; struct device_node; diff --git a/arch/riscv/include/asm/smp.h b/arch/riscv/include/asm/smp.h index 0d555847cde6..2f27fd743092 100644 --- a/arch/riscv/include/asm/smp.h +++ b/arch/riscv/include/asm/smp.h @@ -19,6 +19,20 @@ extern unsigned long boot_cpu_hartid; #include +enum ipi_message_type { + IPI_RESCHEDULE, + IPI_CALL_FUNC, + IPI_CPU_STOP, + IPI_CPU_CRASH_STOP, + IPI_IRQ_WORK, + IPI_TIMER, + IPI_MAX +}; + +int ipi_virq_base_get(void); + +irqreturn_t handle_IPI(int irq, void *data); + /* * Mapping between linux logical cpu index and hartid. */ diff --git a/arch/riscv/kernel/process.c b/arch/riscv/kernel/process.c index 9f0f7b888bc1..7d6bf780d334 100644 --- a/arch/riscv/kernel/process.c +++ b/arch/riscv/kernel/process.c @@ -16,6 +16,7 @@ #include #include #include +#include #include #include @@ -27,6 +28,7 @@ #include #include #include +#include register unsigned long gp_in_global __asm__("gp"); @@ -38,6 +40,8 @@ EXPORT_SYMBOL(__stack_chk_guard); extern asmlinkage void ret_from_fork(void); +DEFINE_PER_CPU(atomic_t, idle_ipi_mask); + static __cpuidle void default_idle(void) { /* @@ -49,6 +53,16 @@ static __cpuidle void default_idle(void) wait_for_interrupt(); } +static __cpuidle void default_idle_enter(void) +{ + /* Do nothing */ +} + +static __cpuidle void default_idle_exit(void) +{ + /* Do nothing */ +} + static __cpuidle void wrs_idle(void) { /* @@ -57,10 +71,42 @@ static __cpuidle void wrs_idle(void) * to entering WRS.NTO. */ mb(); +#ifdef CONFIG_SMP + wrs_nto_if(&this_cpu_ptr(&idle_ipi_mask)->counter, BIT(IPI_MAX)); +#else wrs_nto(¤t_thread_info()->flags); +#endif +} + +static __cpuidle void wrs_idle_enter(void) +{ +#ifdef CONFIG_SMP + atomic_set(this_cpu_ptr(&idle_ipi_mask), BIT(IPI_MAX)); +#endif +} + +static __cpuidle void wrs_idle_exit(void) +{ +#ifdef CONFIG_SMP + int pending; + unsigned long flags; + enum ipi_message_type ipi; + + local_irq_save(flags); + pending = atomic_xchg_relaxed(this_cpu_ptr(&idle_ipi_mask), 0); + for (ipi = IPI_RESCHEDULE; ipi < IPI_MAX; ipi++) + if (pending & BIT(ipi)) { + irq_enter(); + handle_IPI(ipi_virq_base_get() + ipi, NULL); + irq_exit(); + } + local_irq_restore(flags); +#endif } DEFINE_STATIC_CALL_NULL(riscv_idle, default_idle); +DEFINE_STATIC_CALL_NULL(riscv_idle_enter, default_idle_enter); +DEFINE_STATIC_CALL_NULL(riscv_idle_exit, default_idle_exit); void __cpuidle cpu_do_idle(void) { @@ -72,13 +118,28 @@ void __cpuidle arch_cpu_idle(void) cpu_do_idle(); } +void __cpuidle arch_cpu_idle_enter(void) +{ + static_call(riscv_idle_enter)(); +} + +void __cpuidle arch_cpu_idle_exit(void) +{ + static_call(riscv_idle_exit)(); +} + void __init select_idle_routine(void) { if (IS_ENABLED(CONFIG_RISCV_ZAWRS_IDLE) && - riscv_has_extension_likely(RISCV_ISA_EXT_ZAWRS)) + riscv_has_extension_likely(RISCV_ISA_EXT_ZAWRS)) { static_call_update(riscv_idle, wrs_idle); - else + static_call_update(riscv_idle_enter, wrs_idle_enter); + static_call_update(riscv_idle_exit, wrs_idle_exit); + } else { static_call_update(riscv_idle, default_idle); + static_call_update(riscv_idle_enter, default_idle_enter); + static_call_update(riscv_idle_exit, default_idle_exit); + } } int set_unalign_ctl(struct task_struct *tsk, unsigned int val) diff --git a/arch/riscv/kernel/smp.c b/arch/riscv/kernel/smp.c index 45dd4035416e..b5416ee41967 100644 --- a/arch/riscv/kernel/smp.c +++ b/arch/riscv/kernel/smp.c @@ -26,16 +26,6 @@ #include #include -enum ipi_message_type { - IPI_RESCHEDULE, - IPI_CALL_FUNC, - IPI_CPU_STOP, - IPI_CPU_CRASH_STOP, - IPI_IRQ_WORK, - IPI_TIMER, - IPI_MAX -}; - unsigned long __cpuid_to_hartid_map[NR_CPUS] __ro_after_init = { [0 ... NR_CPUS-1] = INVALID_HARTID }; @@ -94,14 +84,34 @@ static inline void ipi_cpu_crash_stop(unsigned int cpu, struct pt_regs *regs) } #endif +#if defined(CONFIG_RISCV_ZAWRS_IDLE) && defined(CONFIG_SMP) +DECLARE_PER_CPU(atomic_t, idle_ipi_mask); +#endif + static void send_ipi_mask(const struct cpumask *mask, enum ipi_message_type op) { +#if defined(CONFIG_RISCV_ZAWRS_IDLE) && defined(CONFIG_SMP) + int cpu, val; + + for_each_cpu(cpu, mask) { + val = atomic_fetch_or_relaxed(BIT(op), per_cpu_ptr(&idle_ipi_mask, cpu)); + if (likely(!(val & BIT(IPI_MAX)))) + __ipi_send_mask(ipi_desc[op], cpumask_of(cpu)); + } +#else __ipi_send_mask(ipi_desc[op], mask); +#endif } static void send_ipi_single(int cpu, enum ipi_message_type op) { - __ipi_send_mask(ipi_desc[op], cpumask_of(cpu)); +#if defined(CONFIG_RISCV_ZAWRS_IDLE) && defined(CONFIG_SMP) + int val; + + val = atomic_fetch_or_relaxed(BIT(op), per_cpu_ptr(&idle_ipi_mask, cpu)); + if (likely(!(val & BIT(IPI_MAX)))) +#endif + __ipi_send_mask(ipi_desc[op], cpumask_of(cpu)); } #ifdef CONFIG_IRQ_WORK @@ -111,7 +121,7 @@ void arch_irq_work_raise(void) } #endif -static irqreturn_t handle_IPI(int irq, void *data) +irqreturn_t handle_IPI(int irq, void *data) { int ipi = irq - ipi_virq_base; @@ -332,3 +342,8 @@ void arch_smp_send_reschedule(int cpu) send_ipi_single(cpu, IPI_RESCHEDULE); } EXPORT_SYMBOL_GPL(arch_smp_send_reschedule); + +int ipi_virq_base_get(void) +{ + return ipi_virq_base; +}