From patchwork Sun Sep 10 08:28:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guo Ren X-Patchwork-Id: 13378468 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A2A43EEB57E for ; Sun, 10 Sep 2023 08:30:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346508AbjIJIaU (ORCPT ); Sun, 10 Sep 2023 04:30:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34052 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245572AbjIJIaT (ORCPT ); Sun, 10 Sep 2023 04:30:19 -0400 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 88059197; Sun, 10 Sep 2023 01:30:15 -0700 (PDT) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B04EEC433CA; Sun, 10 Sep 2023 08:30:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1694334615; bh=Z6iRHs/3ezKlGvEDJaadEcfbR2OWUTdx821B0UQrJ0E=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=vGqJX8UUqlRFp6uTjHgw4CF9dl5VBj6KpBXS8EDXe+7FJT/GTT2mnftqKCOhqQluj 9HYFg/Gv/98xNDbPf71SKzJ5yUHjwqiekijV3Gl/Z/tVRYsyH2b/SwtriQURW0dS1v EmJ7PvcvmXEQXJybTg3UovrPUEe6ZYhERrZYnT3oyB/NMb1eeaWVqA+VwsFDQW5XHO ITyuMfJ44qXNu1OtoEIXy/xpXWStsGJBI3GQt1k4g/D9BEYsUuMb8MwhCfnLY66Mv7 /6j1kvDg6QSnxgBi5Oj3hGq9Qha8C37PEiUneW+kjCPaFzFX3UfBRY7So76gSQEdbe nAz4x+JZXTNUg== From: guoren@kernel.org To: paul.walmsley@sifive.com, anup@brainfault.org, peterz@infradead.org, mingo@redhat.com, will@kernel.org, palmer@rivosinc.com, longman@redhat.com, boqun.feng@gmail.com, tglx@linutronix.de, paulmck@kernel.org, rostedt@goodmis.org, rdunlap@infradead.org, catalin.marinas@arm.com, conor.dooley@microchip.com, xiaoguang.xing@sophgo.com, bjorn@rivosinc.com, alexghiti@rivosinc.com, keescook@chromium.org, greentime.hu@sifive.com, ajones@ventanamicro.com, jszhang@kernel.org, wefu@redhat.com, wuwei2016@iscas.ac.cn, leobras@redhat.com Cc: linux-arch@vger.kernel.org, linux-riscv@lists.infradead.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-csky@vger.kernel.org, Guo Ren , Guo Ren Subject: [PATCH V11 01/17] asm-generic: ticket-lock: Reuse arch_spinlock_t of qspinlock Date: Sun, 10 Sep 2023 04:28:55 -0400 Message-Id: <20230910082911.3378782-2-guoren@kernel.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20230910082911.3378782-1-guoren@kernel.org> References: <20230910082911.3378782-1-guoren@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Guo Ren The arch_spinlock_t of qspinlock has contained the atomic_t val, which satisfies the ticket-lock requirement. Thus, unify the arch_spinlock_t into qspinlock_types.h. This is the preparation for the next combo spinlock. Signed-off-by: Guo Ren Signed-off-by: Guo Ren Reviewed-by: Leonardo Bras --- include/asm-generic/spinlock.h | 14 +++++++------- include/asm-generic/spinlock_types.h | 12 ++---------- 2 files changed, 9 insertions(+), 17 deletions(-) diff --git a/include/asm-generic/spinlock.h b/include/asm-generic/spinlock.h index 90803a826ba0..4773334ee638 100644 --- a/include/asm-generic/spinlock.h +++ b/include/asm-generic/spinlock.h @@ -32,7 +32,7 @@ static __always_inline void arch_spin_lock(arch_spinlock_t *lock) { - u32 val = atomic_fetch_add(1<<16, lock); + u32 val = atomic_fetch_add(1<<16, &lock->val); u16 ticket = val >> 16; if (ticket == (u16)val) @@ -46,31 +46,31 @@ static __always_inline void arch_spin_lock(arch_spinlock_t *lock) * have no outstanding writes due to the atomic_fetch_add() the extra * orderings are free. */ - atomic_cond_read_acquire(lock, ticket == (u16)VAL); + atomic_cond_read_acquire(&lock->val, ticket == (u16)VAL); smp_mb(); } static __always_inline bool arch_spin_trylock(arch_spinlock_t *lock) { - u32 old = atomic_read(lock); + u32 old = atomic_read(&lock->val); if ((old >> 16) != (old & 0xffff)) return false; - return atomic_try_cmpxchg(lock, &old, old + (1<<16)); /* SC, for RCsc */ + return atomic_try_cmpxchg(&lock->val, &old, old + (1<<16)); /* SC, for RCsc */ } static __always_inline void arch_spin_unlock(arch_spinlock_t *lock) { u16 *ptr = (u16 *)lock + IS_ENABLED(CONFIG_CPU_BIG_ENDIAN); - u32 val = atomic_read(lock); + u32 val = atomic_read(&lock->val); smp_store_release(ptr, (u16)val + 1); } static __always_inline int arch_spin_value_unlocked(arch_spinlock_t lock) { - u32 val = lock.counter; + u32 val = lock.val.counter; return ((val >> 16) == (val & 0xffff)); } @@ -84,7 +84,7 @@ static __always_inline int arch_spin_is_locked(arch_spinlock_t *lock) static __always_inline int arch_spin_is_contended(arch_spinlock_t *lock) { - u32 val = atomic_read(lock); + u32 val = atomic_read(&lock->val); return (s16)((val >> 16) - (val & 0xffff)) > 1; } diff --git a/include/asm-generic/spinlock_types.h b/include/asm-generic/spinlock_types.h index 8962bb730945..f534aa5de394 100644 --- a/include/asm-generic/spinlock_types.h +++ b/include/asm-generic/spinlock_types.h @@ -3,15 +3,7 @@ #ifndef __ASM_GENERIC_SPINLOCK_TYPES_H #define __ASM_GENERIC_SPINLOCK_TYPES_H -#include -typedef atomic_t arch_spinlock_t; - -/* - * qrwlock_types depends on arch_spinlock_t, so we must typedef that before the - * include. - */ -#include - -#define __ARCH_SPIN_LOCK_UNLOCKED ATOMIC_INIT(0) +#include +#include #endif /* __ASM_GENERIC_SPINLOCK_TYPES_H */ From patchwork Sun Sep 10 08:28:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guo Ren X-Patchwork-Id: 13378469 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4FA2EEE14D8 for ; Sun, 10 Sep 2023 08:30:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346520AbjIJIa1 (ORCPT ); Sun, 10 Sep 2023 04:30:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58264 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245572AbjIJIa0 (ORCPT ); Sun, 10 Sep 2023 04:30:26 -0400 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7B07CCD3; Sun, 10 Sep 2023 01:30:21 -0700 (PDT) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9F584C433CD; Sun, 10 Sep 2023 08:30:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1694334621; bh=sgIzdUVfUlz4ZBKJCMetJmqx5CuZ4IfTQ/ixE/F16i0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=XjJDS3USkDOTkcPlZq+znKZH3NQ0VRK3AWYR9Ugd8hezhMyxilRJ8EIzYaya16lUI PvbLGS4XfQupAXc8h/m1312mCC8RVyj/upQjMLYyFp44l/aC6G4KHgsROTuQzE+ROk 4pQecwN/6T9Xyq+c8BFSWrow9/OEKEEkUTVseA7ad32O5gUeBZUSgwUxLgpi/VbhX2 EUN7EYGSp6XnD9GtNk/e2+XtT5u0pQCqa0FJe6lqqwPW5KK0UR/QR7B8snKnHpUvEi Mmha8QjZ8BK9NiFrE0Sx17Y/0K5pCjmZ67Ap1E2F3DqjvGftTCS/I2t8arimBTE5CI uLoRDkCiPrHqg== From: guoren@kernel.org To: paul.walmsley@sifive.com, anup@brainfault.org, peterz@infradead.org, mingo@redhat.com, will@kernel.org, palmer@rivosinc.com, longman@redhat.com, boqun.feng@gmail.com, tglx@linutronix.de, paulmck@kernel.org, rostedt@goodmis.org, rdunlap@infradead.org, catalin.marinas@arm.com, conor.dooley@microchip.com, xiaoguang.xing@sophgo.com, bjorn@rivosinc.com, alexghiti@rivosinc.com, keescook@chromium.org, greentime.hu@sifive.com, ajones@ventanamicro.com, jszhang@kernel.org, wefu@redhat.com, wuwei2016@iscas.ac.cn, leobras@redhat.com Cc: linux-arch@vger.kernel.org, linux-riscv@lists.infradead.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-csky@vger.kernel.org, Guo Ren , Guo Ren Subject: [PATCH V11 02/17] asm-generic: ticket-lock: Move into ticket_spinlock.h Date: Sun, 10 Sep 2023 04:28:56 -0400 Message-Id: <20230910082911.3378782-3-guoren@kernel.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20230910082911.3378782-1-guoren@kernel.org> References: <20230910082911.3378782-1-guoren@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Guo Ren Move ticket-lock definition into an independent file. This is the preparation for the next combo spinlock of riscv. Signed-off-by: Guo Ren Signed-off-by: Guo Ren Reviewed-by: Leonardo Bras --- include/asm-generic/spinlock.h | 87 +--------------------- include/asm-generic/ticket_spinlock.h | 103 ++++++++++++++++++++++++++ 2 files changed, 104 insertions(+), 86 deletions(-) create mode 100644 include/asm-generic/ticket_spinlock.h diff --git a/include/asm-generic/spinlock.h b/include/asm-generic/spinlock.h index 4773334ee638..970590baf61b 100644 --- a/include/asm-generic/spinlock.h +++ b/include/asm-generic/spinlock.h @@ -1,94 +1,9 @@ /* SPDX-License-Identifier: GPL-2.0 */ -/* - * 'Generic' ticket-lock implementation. - * - * It relies on atomic_fetch_add() having well defined forward progress - * guarantees under contention. If your architecture cannot provide this, stick - * to a test-and-set lock. - * - * It also relies on atomic_fetch_add() being safe vs smp_store_release() on a - * sub-word of the value. This is generally true for anything LL/SC although - * you'd be hard pressed to find anything useful in architecture specifications - * about this. If your architecture cannot do this you might be better off with - * a test-and-set. - * - * It further assumes atomic_*_release() + atomic_*_acquire() is RCpc and hence - * uses atomic_fetch_add() which is RCsc to create an RCsc hot path, along with - * a full fence after the spin to upgrade the otherwise-RCpc - * atomic_cond_read_acquire(). - * - * The implementation uses smp_cond_load_acquire() to spin, so if the - * architecture has WFE like instructions to sleep instead of poll for word - * modifications be sure to implement that (see ARM64 for example). - * - */ - #ifndef __ASM_GENERIC_SPINLOCK_H #define __ASM_GENERIC_SPINLOCK_H -#include -#include - -static __always_inline void arch_spin_lock(arch_spinlock_t *lock) -{ - u32 val = atomic_fetch_add(1<<16, &lock->val); - u16 ticket = val >> 16; - - if (ticket == (u16)val) - return; - - /* - * atomic_cond_read_acquire() is RCpc, but rather than defining a - * custom cond_read_rcsc() here we just emit a full fence. We only - * need the prior reads before subsequent writes ordering from - * smb_mb(), but as atomic_cond_read_acquire() just emits reads and we - * have no outstanding writes due to the atomic_fetch_add() the extra - * orderings are free. - */ - atomic_cond_read_acquire(&lock->val, ticket == (u16)VAL); - smp_mb(); -} - -static __always_inline bool arch_spin_trylock(arch_spinlock_t *lock) -{ - u32 old = atomic_read(&lock->val); - - if ((old >> 16) != (old & 0xffff)) - return false; - - return atomic_try_cmpxchg(&lock->val, &old, old + (1<<16)); /* SC, for RCsc */ -} - -static __always_inline void arch_spin_unlock(arch_spinlock_t *lock) -{ - u16 *ptr = (u16 *)lock + IS_ENABLED(CONFIG_CPU_BIG_ENDIAN); - u32 val = atomic_read(&lock->val); - - smp_store_release(ptr, (u16)val + 1); -} - -static __always_inline int arch_spin_value_unlocked(arch_spinlock_t lock) -{ - u32 val = lock.val.counter; - - return ((val >> 16) == (val & 0xffff)); -} - -static __always_inline int arch_spin_is_locked(arch_spinlock_t *lock) -{ - arch_spinlock_t val = READ_ONCE(*lock); - - return !arch_spin_value_unlocked(val); -} - -static __always_inline int arch_spin_is_contended(arch_spinlock_t *lock) -{ - u32 val = atomic_read(&lock->val); - - return (s16)((val >> 16) - (val & 0xffff)) > 1; -} - +#include #include #endif /* __ASM_GENERIC_SPINLOCK_H */ diff --git a/include/asm-generic/ticket_spinlock.h b/include/asm-generic/ticket_spinlock.h new file mode 100644 index 000000000000..cfcff22b37b3 --- /dev/null +++ b/include/asm-generic/ticket_spinlock.h @@ -0,0 +1,103 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +/* + * 'Generic' ticket-lock implementation. + * + * It relies on atomic_fetch_add() having well defined forward progress + * guarantees under contention. If your architecture cannot provide this, stick + * to a test-and-set lock. + * + * It also relies on atomic_fetch_add() being safe vs smp_store_release() on a + * sub-word of the value. This is generally true for anything LL/SC although + * you'd be hard pressed to find anything useful in architecture specifications + * about this. If your architecture cannot do this you might be better off with + * a test-and-set. + * + * It further assumes atomic_*_release() + atomic_*_acquire() is RCpc and hence + * uses atomic_fetch_add() which is RCsc to create an RCsc hot path, along with + * a full fence after the spin to upgrade the otherwise-RCpc + * atomic_cond_read_acquire(). + * + * The implementation uses smp_cond_load_acquire() to spin, so if the + * architecture has WFE like instructions to sleep instead of poll for word + * modifications be sure to implement that (see ARM64 for example). + * + */ + +#ifndef __ASM_GENERIC_TICKET_SPINLOCK_H +#define __ASM_GENERIC_TICKET_SPINLOCK_H + +#include +#include + +static __always_inline void ticket_spin_lock(arch_spinlock_t *lock) +{ + u32 val = atomic_fetch_add(1<<16, &lock->val); + u16 ticket = val >> 16; + + if (ticket == (u16)val) + return; + + /* + * atomic_cond_read_acquire() is RCpc, but rather than defining a + * custom cond_read_rcsc() here we just emit a full fence. We only + * need the prior reads before subsequent writes ordering from + * smb_mb(), but as atomic_cond_read_acquire() just emits reads and we + * have no outstanding writes due to the atomic_fetch_add() the extra + * orderings are free. + */ + atomic_cond_read_acquire(&lock->val, ticket == (u16)VAL); + smp_mb(); +} + +static __always_inline bool ticket_spin_trylock(arch_spinlock_t *lock) +{ + u32 old = atomic_read(&lock->val); + + if ((old >> 16) != (old & 0xffff)) + return false; + + return atomic_try_cmpxchg(&lock->val, &old, old + (1<<16)); /* SC, for RCsc */ +} + +static __always_inline void ticket_spin_unlock(arch_spinlock_t *lock) +{ + u16 *ptr = (u16 *)lock + IS_ENABLED(CONFIG_CPU_BIG_ENDIAN); + u32 val = atomic_read(&lock->val); + + smp_store_release(ptr, (u16)val + 1); +} + +static __always_inline int ticket_spin_value_unlocked(arch_spinlock_t lock) +{ + u32 val = lock.val.counter; + + return ((val >> 16) == (val & 0xffff)); +} + +static __always_inline int ticket_spin_is_locked(arch_spinlock_t *lock) +{ + arch_spinlock_t val = READ_ONCE(*lock); + + return !ticket_spin_value_unlocked(val); +} + +static __always_inline int ticket_spin_is_contended(arch_spinlock_t *lock) +{ + u32 val = atomic_read(&lock->val); + + return (s16)((val >> 16) - (val & 0xffff)) > 1; +} + +/* + * Remapping spinlock architecture specific functions to the corresponding + * ticket spinlock functions. + */ +#define arch_spin_is_locked(l) ticket_spin_is_locked(l) +#define arch_spin_is_contended(l) ticket_spin_is_contended(l) +#define arch_spin_value_unlocked(l) ticket_spin_value_unlocked(l) +#define arch_spin_lock(l) ticket_spin_lock(l) +#define arch_spin_trylock(l) ticket_spin_trylock(l) +#define arch_spin_unlock(l) ticket_spin_unlock(l) + +#endif /* __ASM_GENERIC_TICKET_SPINLOCK_H */ From patchwork Sun Sep 10 08:28:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guo Ren X-Patchwork-Id: 13378470 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 74E43EEB57E for ; Sun, 10 Sep 2023 08:30:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346531AbjIJIah (ORCPT ); Sun, 10 Sep 2023 04:30:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50948 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346506AbjIJIaf (ORCPT ); Sun, 10 Sep 2023 04:30:35 -0400 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 67F4ECC0; Sun, 10 Sep 2023 01:30:27 -0700 (PDT) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8F0A2C433CC; Sun, 10 Sep 2023 08:30:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1694334627; bh=JHxtCSFAXXj+aAh2d1UzrcmVnbP7XJ2TAFbFny2j3pM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=gkC6EgOJCLxEIUZxkbIuR1YvLQRbsEdcIriRqReFcbydI85IYQob0oUNpPRgyojoN Ngd6SVyuYUJtvWMoqh6PbN8d13dOhFeQ6J2Gzu/aCowb49eOF7LBbcoUn40G3KqnDS Pdtvoj+iD080UL1k8d9qE19wDPh1uSP0OwUxMwi2n7mncnsNvPCJCMOyMx9hCYGlzd XwOnTEXflDp0+8Vfk/PHZSusn8wvbkhZFtrOifwwgZWYXuqhfdfpCog0ptjNWSqU6D Ig3SYK1z5xii6D7nfdzIEgQbg+tJf3vrftC1cnvT4pqDpP2WsggkJXCzcJPS5Hw+wr QhrAAJDCXpvvg== From: guoren@kernel.org To: paul.walmsley@sifive.com, anup@brainfault.org, peterz@infradead.org, mingo@redhat.com, will@kernel.org, palmer@rivosinc.com, longman@redhat.com, boqun.feng@gmail.com, tglx@linutronix.de, paulmck@kernel.org, rostedt@goodmis.org, rdunlap@infradead.org, catalin.marinas@arm.com, conor.dooley@microchip.com, xiaoguang.xing@sophgo.com, bjorn@rivosinc.com, alexghiti@rivosinc.com, keescook@chromium.org, greentime.hu@sifive.com, ajones@ventanamicro.com, jszhang@kernel.org, wefu@redhat.com, wuwei2016@iscas.ac.cn, leobras@redhat.com Cc: linux-arch@vger.kernel.org, linux-riscv@lists.infradead.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-csky@vger.kernel.org, Guo Ren , Guo Ren Subject: [PATCH V11 03/17] riscv: Use Zicbop in arch_xchg when available Date: Sun, 10 Sep 2023 04:28:57 -0400 Message-Id: <20230910082911.3378782-4-guoren@kernel.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20230910082911.3378782-1-guoren@kernel.org> References: <20230910082911.3378782-1-guoren@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Guo Ren Cache-block prefetch instructions are HINTs to the hardware to indicate that software intends to perform a particular type of memory access in the near future. Enable ARCH_HAS_PREFETCHW and improve the arch_xchg for qspinlock xchg_tail. Signed-off-by: Guo Ren Signed-off-by: Guo Ren --- arch/riscv/Kconfig | 15 +++++++++++++++ arch/riscv/include/asm/cmpxchg.h | 4 +++- arch/riscv/include/asm/hwcap.h | 1 + arch/riscv/include/asm/insn-def.h | 5 +++++ arch/riscv/include/asm/processor.h | 13 +++++++++++++ arch/riscv/kernel/cpufeature.c | 1 + 6 files changed, 38 insertions(+), 1 deletion(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index e9ae6fa232c3..2c346fe169c1 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -617,6 +617,21 @@ config RISCV_ISA_ZICBOZ If you don't know what to do here, say Y. +config RISCV_ISA_ZICBOP + bool "Zicbop extension support for cache block prefetch" + depends on MMU + depends on RISCV_ALTERNATIVE + default y + help + Adds support to dynamically detect the presence of the ZICBOP + extension (Cache Block Prefetch Operations) and enable its + usage. + + The Zicbop extension can be used to prefetch cache block for + read/write/instruction fetch. + + If you don't know what to do here, say Y. + config TOOLCHAIN_HAS_ZIHINTPAUSE bool default y diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index 702725727671..56eff7a9d2d2 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -11,6 +11,7 @@ #include #include +#include #define __arch_xchg_masked(prepend, append, r, p, n) \ ({ \ @@ -25,6 +26,7 @@ \ __asm__ __volatile__ ( \ prepend \ + PREFETCHW_ASM(%5) \ "0: lr.w %0, %2\n" \ " and %1, %0, %z4\n" \ " or %1, %1, %z3\n" \ @@ -32,7 +34,7 @@ " bnez %1, 0b\n" \ append \ : "=&r" (__retx), "=&r" (__rc), "+A" (*(__ptr32b)) \ - : "rJ" (__newx), "rJ" (~__mask) \ + : "rJ" (__newx), "rJ" (~__mask), "rJ" (__ptr32b) \ : "memory"); \ \ r = (__typeof__(*(p)))((__retx & __mask) >> __s); \ diff --git a/arch/riscv/include/asm/hwcap.h b/arch/riscv/include/asm/hwcap.h index b7b58258f6c7..78b7b8b53778 100644 --- a/arch/riscv/include/asm/hwcap.h +++ b/arch/riscv/include/asm/hwcap.h @@ -58,6 +58,7 @@ #define RISCV_ISA_EXT_ZICSR 40 #define RISCV_ISA_EXT_ZIFENCEI 41 #define RISCV_ISA_EXT_ZIHPM 42 +#define RISCV_ISA_EXT_ZICBOP 43 #define RISCV_ISA_EXT_MAX 64 diff --git a/arch/riscv/include/asm/insn-def.h b/arch/riscv/include/asm/insn-def.h index 6960beb75f32..dc590d331894 100644 --- a/arch/riscv/include/asm/insn-def.h +++ b/arch/riscv/include/asm/insn-def.h @@ -134,6 +134,7 @@ #define RV_OPCODE_MISC_MEM RV_OPCODE(15) #define RV_OPCODE_SYSTEM RV_OPCODE(115) +#define RV_OPCODE_PREFETCH RV_OPCODE(19) #define HFENCE_VVMA(vaddr, asid) \ INSN_R(OPCODE_SYSTEM, FUNC3(0), FUNC7(17), \ @@ -196,4 +197,8 @@ INSN_I(OPCODE_MISC_MEM, FUNC3(2), __RD(0), \ RS1(base), SIMM12(4)) +#define CBO_prefetchw(base) \ + INSN_R(OPCODE_PREFETCH, FUNC3(6), FUNC7(0), \ + RD(x0), RS1(base), RS2(x0)) + #endif /* __ASM_INSN_DEF_H */ diff --git a/arch/riscv/include/asm/processor.h b/arch/riscv/include/asm/processor.h index de9da852f78d..7ad3a24212e8 100644 --- a/arch/riscv/include/asm/processor.h +++ b/arch/riscv/include/asm/processor.h @@ -12,6 +12,8 @@ #include #include +#include +#include #ifdef CONFIG_64BIT #define DEFAULT_MAP_WINDOW (UL(1) << (MMAP_VA_BITS - 1)) @@ -103,6 +105,17 @@ static inline void arch_thread_struct_whitelist(unsigned long *offset, #define KSTK_EIP(tsk) (ulong)(task_pt_regs(tsk)->epc) #define KSTK_ESP(tsk) (ulong)(task_pt_regs(tsk)->sp) +#define ARCH_HAS_PREFETCHW +#define PREFETCHW_ASM(base) ALTERNATIVE(__nops(1), \ + CBO_prefetchw(base), \ + 0, \ + RISCV_ISA_EXT_ZICBOP, \ + CONFIG_RISCV_ISA_ZICBOP) +static inline void prefetchw(const void *ptr) +{ + asm volatile(PREFETCHW_ASM(%0) + : : "r" (ptr) : "memory"); +} /* Do necessary setup to start up a newly executed thread. */ extern void start_thread(struct pt_regs *regs, diff --git a/arch/riscv/kernel/cpufeature.c b/arch/riscv/kernel/cpufeature.c index ef7b4fd9e876..e0b897db0b97 100644 --- a/arch/riscv/kernel/cpufeature.c +++ b/arch/riscv/kernel/cpufeature.c @@ -159,6 +159,7 @@ const struct riscv_isa_ext_data riscv_isa_ext[] = { __RISCV_ISA_EXT_DATA(h, RISCV_ISA_EXT_h), __RISCV_ISA_EXT_DATA(zicbom, RISCV_ISA_EXT_ZICBOM), __RISCV_ISA_EXT_DATA(zicboz, RISCV_ISA_EXT_ZICBOZ), + __RISCV_ISA_EXT_DATA(zicbop, RISCV_ISA_EXT_ZICBOP), __RISCV_ISA_EXT_DATA(zicntr, RISCV_ISA_EXT_ZICNTR), __RISCV_ISA_EXT_DATA(zicsr, RISCV_ISA_EXT_ZICSR), __RISCV_ISA_EXT_DATA(zifencei, RISCV_ISA_EXT_ZIFENCEI), From patchwork Sun Sep 10 08:28:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guo Ren X-Patchwork-Id: 13378471 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63D0AEEB57E for ; Sun, 10 Sep 2023 08:30:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346532AbjIJIan (ORCPT ); Sun, 10 Sep 2023 04:30:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40974 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231841AbjIJIam (ORCPT ); Sun, 10 Sep 2023 04:30:42 -0400 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4AB5718F; Sun, 10 Sep 2023 01:30:33 -0700 (PDT) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 79BD6C433C7; Sun, 10 Sep 2023 08:30:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1694334633; bh=RIvCrY8aBdKMNBfBShTR8o+UM7FnOQ4QZsRxMNLpjdw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=KjCnXhMG6KClfl718naMf98+jPxX1V1N4S6PUpmhSsX9BjtbUPsH5h4JgW4UIvFl5 gJxh4UC7UMdy5IkSGSnGA5IYbGLCzbKvkQF+/mACEoVG9tFS95mZ8t/PSBeqmckfLT 5VLp7eBOhVtNCI5mLqSH1O/MY8PzUChOhu9Y792vLHaPgIMY2pJARxF50VH7M46PAt 1aEFPzbYLaJDVXFZCwqidYQ+quX4JjQ9QjO9QLknsAGmTnDweASiVmnIoNQjI9yf++ LXOLFwOKc8/XuL5qoy95gnzseByq1vbYUJ9Nj2dXCMj7Np+2SdZ1SBREM6Frqrg1iw y8KMhHo2aCNtA== From: guoren@kernel.org To: paul.walmsley@sifive.com, anup@brainfault.org, peterz@infradead.org, mingo@redhat.com, will@kernel.org, palmer@rivosinc.com, longman@redhat.com, boqun.feng@gmail.com, tglx@linutronix.de, paulmck@kernel.org, rostedt@goodmis.org, rdunlap@infradead.org, catalin.marinas@arm.com, conor.dooley@microchip.com, xiaoguang.xing@sophgo.com, bjorn@rivosinc.com, alexghiti@rivosinc.com, keescook@chromium.org, greentime.hu@sifive.com, ajones@ventanamicro.com, jszhang@kernel.org, wefu@redhat.com, wuwei2016@iscas.ac.cn, leobras@redhat.com Cc: linux-arch@vger.kernel.org, linux-riscv@lists.infradead.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-csky@vger.kernel.org, Guo Ren , Guo Ren Subject: [PATCH V11 04/17] locking/qspinlock: Improve xchg_tail for number of cpus >= 16k Date: Sun, 10 Sep 2023 04:28:58 -0400 Message-Id: <20230910082911.3378782-5-guoren@kernel.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20230910082911.3378782-1-guoren@kernel.org> References: <20230910082911.3378782-1-guoren@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Guo Ren The target of xchg_tail is to write the tail to the lock value, so adding prefetchw could help the next cmpxchg step, which may decrease the cmpxchg retry loops of xchg_tail. Some processors may utilize this feature to give a forward guarantee, e.g., RISC-V XuanTie processors would block the snoop channel & irq for several cycles when prefetch.w instruction (from Zicbop extension) retired, which guarantees the next cmpxchg succeeds. Signed-off-by: Guo Ren Signed-off-by: Guo Ren Reviewed-by: Leonardo Bras --- kernel/locking/qspinlock.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/kernel/locking/qspinlock.c b/kernel/locking/qspinlock.c index d3f99060b60f..96b54e2ade86 100644 --- a/kernel/locking/qspinlock.c +++ b/kernel/locking/qspinlock.c @@ -223,7 +223,10 @@ static __always_inline void clear_pending_set_locked(struct qspinlock *lock) */ static __always_inline u32 xchg_tail(struct qspinlock *lock, u32 tail) { - u32 old, new, val = atomic_read(&lock->val); + u32 old, new, val; + + prefetchw(&lock->val); + val = atomic_read(&lock->val); for (;;) { new = (val & _Q_LOCKED_PENDING_MASK) | tail; From patchwork Sun Sep 10 08:28:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guo Ren X-Patchwork-Id: 13378472 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F347EEB57E for ; Sun, 10 Sep 2023 08:30:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346526AbjIJIaw (ORCPT ); Sun, 10 Sep 2023 04:30:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41202 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231841AbjIJIav (ORCPT ); Sun, 10 Sep 2023 04:30:51 -0400 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 459B1CF9; Sun, 10 Sep 2023 01:30:39 -0700 (PDT) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 63325C433CD; Sun, 10 Sep 2023 08:30:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1694334638; bh=8ceHF5hJ2VUWSA5tS63XBWWnM2Tfev0qABmdpbRodUU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ds4oWT64qxmqLF1VUOM7mLA94WTxN6eaBisAjMu1Sv8P4X/ibeZwPHwPNnjy+hNNx V0L4xmLYMz7RG73NFPrHwJQGj+u6kWHOHX4yjTTy0WkQIRcawABZdCIxNyuhKVw7us O+VqQP+z3hWWkCcN7WjKCaLN5UP6K7X94jv3+mlIHO+J0A1UjjyFjkRNdqvEJLiS+m dyPSE60MXgwNEcg5Zav20AW00YiL/8mSvrPLUo1/YI+XwO3Dk4asSqBpuaSKa2yRMG DFUVKcER1a2gidEt1CPeDMfUeiAztMfMZY3PoJNtv2HhbtYQ/a0zLeU81pu5kxIa5l VkEwlBYYsMZYg== From: guoren@kernel.org To: paul.walmsley@sifive.com, anup@brainfault.org, peterz@infradead.org, mingo@redhat.com, will@kernel.org, palmer@rivosinc.com, longman@redhat.com, boqun.feng@gmail.com, tglx@linutronix.de, paulmck@kernel.org, rostedt@goodmis.org, rdunlap@infradead.org, catalin.marinas@arm.com, conor.dooley@microchip.com, xiaoguang.xing@sophgo.com, bjorn@rivosinc.com, alexghiti@rivosinc.com, keescook@chromium.org, greentime.hu@sifive.com, ajones@ventanamicro.com, jszhang@kernel.org, wefu@redhat.com, wuwei2016@iscas.ac.cn, leobras@redhat.com Cc: linux-arch@vger.kernel.org, linux-riscv@lists.infradead.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-csky@vger.kernel.org, Guo Ren , Guo Ren Subject: [PATCH V11 05/17] riscv: qspinlock: Add basic queued_spinlock support Date: Sun, 10 Sep 2023 04:28:59 -0400 Message-Id: <20230910082911.3378782-6-guoren@kernel.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20230910082911.3378782-1-guoren@kernel.org> References: <20230910082911.3378782-1-guoren@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Guo Ren The requirements of qspinlock have been documented by commit: a8ad07e5240c ("asm-generic: qspinlock: Indicate the use of mixed-size atomics"). Although RISC-V ISA gives out a weaker forward guarantee LR/SC, which doesn't satisfy the requirements of qspinlock above, it won't prevent some riscv vendors from implementing a strong fwd guarantee LR/SC in microarchitecture to match xchg_tail requirement. T-HEAD C9xx processor is the one. We've tested the patch on SOPHGO sg2042 & th1520 and passed the stress test on Fedora & Ubuntu & OpenEuler ... Here is the performance comparison between qspinlock and ticket_lock on sg2042 (64 cores): sysbench test=threads threads=32 yields=100 lock=8 (+13.8%): queued_spinlock 0.5109/0.00 ticket_spinlock 0.5814/0.00 perf futex/hash (+6.7%): queued_spinlock 1444393 operations/sec (+- 0.09%) ticket_spinlock 1353215 operations/sec (+- 0.15%) perf futex/wake-parallel (+8.6%): queued_spinlock (waking 1/64 threads) in 0.0253 ms (+-2.90%) ticket_spinlock (waking 1/64 threads) in 0.0275 ms (+-3.12%) perf futex/requeue (+4.2%): queued_spinlock Requeued 64 of 64 threads in 0.0785 ms (+-0.55%) ticket_spinlock Requeued 64 of 64 threads in 0.0818 ms (+-4.12%) System Benchmarks (+6.4%) queued_spinlock: System Benchmarks Index Values BASELINE RESULT INDEX Dhrystone 2 using register variables 116700.0 628613745.4 53865.8 Double-Precision Whetstone 55.0 182422.8 33167.8 Execl Throughput 43.0 13116.6 3050.4 File Copy 1024 bufsize 2000 maxblocks 3960.0 7762306.2 19601.8 File Copy 256 bufsize 500 maxblocks 1655.0 3417556.8 20649.9 File Copy 4096 bufsize 8000 maxblocks 5800.0 7427995.7 12806.9 Pipe Throughput 12440.0 23058600.5 18535.9 Pipe-based Context Switching 4000.0 2835617.7 7089.0 Process Creation 126.0 12537.3 995.0 Shell Scripts (1 concurrent) 42.4 57057.4 13456.9 Shell Scripts (8 concurrent) 6.0 7367.1 12278.5 System Call Overhead 15000.0 33308301.3 22205.5 ======== System Benchmarks Index Score 12426.1 ticket_spinlock: System Benchmarks Index Values BASELINE RESULT INDEX Dhrystone 2 using register variables 116700.0 626541701.9 53688.2 Double-Precision Whetstone 55.0 181921.0 33076.5 Execl Throughput 43.0 12625.1 2936.1 File Copy 1024 bufsize 2000 maxblocks 3960.0 6553792.9 16550.0 File Copy 256 bufsize 500 maxblocks 1655.0 3189231.6 19270.3 File Copy 4096 bufsize 8000 maxblocks 5800.0 7221277.0 12450.5 Pipe Throughput 12440.0 20594018.7 16554.7 Pipe-based Context Switching 4000.0 2571117.7 6427.8 Process Creation 126.0 10798.4 857.0 Shell Scripts (1 concurrent) 42.4 57227.5 13497.1 Shell Scripts (8 concurrent) 6.0 7329.2 12215.3 System Call Overhead 15000.0 30766778.4 20511.2 ======== System Benchmarks Index Score 11670.7 The qspinlock has a significant improvement on SOPHGO SG2042 64 cores platform than the ticket_lock. Signed-off-by: Guo Ren Signed-off-by: Guo Ren Reviewed-by: Leonardo Bras --- arch/riscv/Kconfig | 16 ++++++++++++++++ arch/riscv/include/asm/Kbuild | 3 ++- arch/riscv/include/asm/spinlock.h | 17 +++++++++++++++++ 3 files changed, 35 insertions(+), 1 deletion(-) create mode 100644 arch/riscv/include/asm/spinlock.h diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 2c346fe169c1..7f39bfc75744 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -471,6 +471,22 @@ config NODES_SHIFT Specify the maximum number of NUMA Nodes available on the target system. Increases memory reserved to accommodate various tables. +choice + prompt "RISC-V spinlock type" + default RISCV_TICKET_SPINLOCKS + +config RISCV_TICKET_SPINLOCKS + bool "Using ticket spinlock" + +config RISCV_QUEUED_SPINLOCKS + bool "Using queued spinlock" + depends on SMP && MMU + select ARCH_USE_QUEUED_SPINLOCKS + help + Make sure your micro arch LL/SC has a strong forward progress guarantee. + Otherwise, stay at ticket-lock. +endchoice + config RISCV_ALTERNATIVE bool depends on !XIP_KERNEL diff --git a/arch/riscv/include/asm/Kbuild b/arch/riscv/include/asm/Kbuild index 504f8b7e72d4..a0dc85e4a754 100644 --- a/arch/riscv/include/asm/Kbuild +++ b/arch/riscv/include/asm/Kbuild @@ -2,10 +2,11 @@ generic-y += early_ioremap.h generic-y += flat.h generic-y += kvm_para.h +generic-y += mcs_spinlock.h generic-y += parport.h -generic-y += spinlock.h generic-y += spinlock_types.h generic-y += qrwlock.h generic-y += qrwlock_types.h +generic-y += qspinlock.h generic-y += user.h generic-y += vmlinux.lds.h diff --git a/arch/riscv/include/asm/spinlock.h b/arch/riscv/include/asm/spinlock.h new file mode 100644 index 000000000000..c644a92d4548 --- /dev/null +++ b/arch/riscv/include/asm/spinlock.h @@ -0,0 +1,17 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef __ASM_RISCV_SPINLOCK_H +#define __ASM_RISCV_SPINLOCK_H + +#ifdef CONFIG_QUEUED_SPINLOCKS +#define _Q_PENDING_LOOPS (1 << 9) +#endif + +#ifdef CONFIG_QUEUED_SPINLOCKS +#include +#include +#else +#include +#endif + +#endif /* __ASM_RISCV_SPINLOCK_H */ From patchwork Sun Sep 10 08:29:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guo Ren X-Patchwork-Id: 13378473 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40853EE7FF4 for ; Sun, 10 Sep 2023 08:31:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346537AbjIJIbF (ORCPT ); Sun, 10 Sep 2023 04:31:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41528 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240157AbjIJIbE (ORCPT ); Sun, 10 Sep 2023 04:31:04 -0400 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2AA19CC9; Sun, 10 Sep 2023 01:30:45 -0700 (PDT) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 53691C433CA; Sun, 10 Sep 2023 08:30:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1694334644; bh=beXd/IVecz7TWfZ1o1vVfgkat9gjWh6TojSXwShn+Fs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=lw8mWyH1Wj5Bp+aLA3Jt3V19kTg+0ySS5A03IqS68yHWCUwq+AgSThFxwuHwCXuvN tzhiTtdDq2b+MvQ9ip6np9IQLnQmSxIupkB4VpLps+8LaO+cksqfyxqWC4a7hksP+1 ZaJ5eskfyJh1ledR/St6b5kIAXDXXS2bwNJ1YwlMprljDlaOstKACijaXwBFxe4BzN nhlRa+aX2As7LN0ruPacVqwGDlEE5HGMbEJ/+BXZb7pPM79VAS6vPwlJ1vnEWEu+aF 0kH+Xa8q0icePxqjOSdXOXh8bbfBRL0kj0UaAMSIN2og10VuFNSrDNdzTGwwF9ZMps TEwdp0UqEx5Gw== From: guoren@kernel.org To: paul.walmsley@sifive.com, anup@brainfault.org, peterz@infradead.org, mingo@redhat.com, will@kernel.org, palmer@rivosinc.com, longman@redhat.com, boqun.feng@gmail.com, tglx@linutronix.de, paulmck@kernel.org, rostedt@goodmis.org, rdunlap@infradead.org, catalin.marinas@arm.com, conor.dooley@microchip.com, xiaoguang.xing@sophgo.com, bjorn@rivosinc.com, alexghiti@rivosinc.com, keescook@chromium.org, greentime.hu@sifive.com, ajones@ventanamicro.com, jszhang@kernel.org, wefu@redhat.com, wuwei2016@iscas.ac.cn, leobras@redhat.com Cc: linux-arch@vger.kernel.org, linux-riscv@lists.infradead.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-csky@vger.kernel.org, Guo Ren , Guo Ren Subject: [PATCH V11 06/17] riscv: qspinlock: Introduce combo spinlock Date: Sun, 10 Sep 2023 04:29:00 -0400 Message-Id: <20230910082911.3378782-7-guoren@kernel.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20230910082911.3378782-1-guoren@kernel.org> References: <20230910082911.3378782-1-guoren@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Guo Ren Combo spinlock could support queued and ticket in one Linux Image and select them during boot time via errata mechanism. Here is the func size (Bytes) comparison table below: TYPE : COMBO | TICKET | QUEUED arch_spin_lock : 106 | 60 | 50 arch_spin_unlock : 54 | 36 | 26 arch_spin_trylock : 110 | 72 | 54 arch_spin_is_locked : 48 | 34 | 20 arch_spin_is_contended : 56 | 40 | 24 rch_spin_value_unlocked : 48 | 34 | 24 One example of disassemble combo arch_spin_unlock: 0xffffffff8000409c <+14>: nop # detour slot 0xffffffff800040a0 <+18>: fence rw,w # queued spinlock start 0xffffffff800040a4 <+22>: sb zero,0(a4) # queued spinlock end 0xffffffff800040a8 <+26>: ld s0,8(sp) 0xffffffff800040aa <+28>: addi sp,sp,16 0xffffffff800040ac <+30>: ret 0xffffffff800040ae <+32>: lw a5,0(a4) # ticket spinlock start 0xffffffff800040b0 <+34>: sext.w a5,a5 0xffffffff800040b2 <+36>: fence rw,w 0xffffffff800040b6 <+40>: addiw a5,a5,1 0xffffffff800040b8 <+42>: slli a5,a5,0x30 0xffffffff800040ba <+44>: srli a5,a5,0x30 0xffffffff800040bc <+46>: sh a5,0(a4) # ticket spinlock end 0xffffffff800040c0 <+50>: ld s0,8(sp) 0xffffffff800040c2 <+52>: addi sp,sp,16 0xffffffff800040c4 <+54>: ret The qspinlock is smaller and faster than ticket-lock when all are in fast-path, and combo spinlock could provide a compatible Linux Image for different micro-arch design (weak/strict fwd guarantee LR/SC) processors. Signed-off-by: Guo Ren Signed-off-by: Guo Ren --- arch/riscv/Kconfig | 9 +++- arch/riscv/include/asm/spinlock.h | 78 ++++++++++++++++++++++++++++++- arch/riscv/kernel/setup.c | 14 ++++++ 3 files changed, 98 insertions(+), 3 deletions(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 7f39bfc75744..4bcff2860f48 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -473,7 +473,7 @@ config NODES_SHIFT choice prompt "RISC-V spinlock type" - default RISCV_TICKET_SPINLOCKS + default RISCV_COMBO_SPINLOCKS config RISCV_TICKET_SPINLOCKS bool "Using ticket spinlock" @@ -485,6 +485,13 @@ config RISCV_QUEUED_SPINLOCKS help Make sure your micro arch LL/SC has a strong forward progress guarantee. Otherwise, stay at ticket-lock. + +config RISCV_COMBO_SPINLOCKS + bool "Using combo spinlock" + depends on SMP && MMU + select ARCH_USE_QUEUED_SPINLOCKS + help + Select queued spinlock or ticket-lock via errata. endchoice config RISCV_ALTERNATIVE diff --git a/arch/riscv/include/asm/spinlock.h b/arch/riscv/include/asm/spinlock.h index c644a92d4548..8ea0fee80652 100644 --- a/arch/riscv/include/asm/spinlock.h +++ b/arch/riscv/include/asm/spinlock.h @@ -7,11 +7,85 @@ #define _Q_PENDING_LOOPS (1 << 9) #endif +#ifdef CONFIG_RISCV_COMBO_SPINLOCKS +#include + +#undef arch_spin_is_locked +#undef arch_spin_is_contended +#undef arch_spin_value_unlocked +#undef arch_spin_lock +#undef arch_spin_trylock +#undef arch_spin_unlock + +#include +#include + +#undef arch_spin_is_locked +#undef arch_spin_is_contended +#undef arch_spin_value_unlocked +#undef arch_spin_lock +#undef arch_spin_trylock +#undef arch_spin_unlock + +DECLARE_STATIC_KEY_TRUE(combo_qspinlock_key); + +static __always_inline void arch_spin_lock(arch_spinlock_t *lock) +{ + if (static_branch_likely(&combo_qspinlock_key)) + queued_spin_lock(lock); + else + ticket_spin_lock(lock); +} + +static __always_inline bool arch_spin_trylock(arch_spinlock_t *lock) +{ + if (static_branch_likely(&combo_qspinlock_key)) + return queued_spin_trylock(lock); + else + return ticket_spin_trylock(lock); +} + +static __always_inline void arch_spin_unlock(arch_spinlock_t *lock) +{ + if (static_branch_likely(&combo_qspinlock_key)) + queued_spin_unlock(lock); + else + ticket_spin_unlock(lock); +} + +static __always_inline int arch_spin_value_unlocked(arch_spinlock_t lock) +{ + if (static_branch_likely(&combo_qspinlock_key)) + return queued_spin_value_unlocked(lock); + else + return ticket_spin_value_unlocked(lock); +} + +static __always_inline int arch_spin_is_locked(arch_spinlock_t *lock) +{ + if (static_branch_likely(&combo_qspinlock_key)) + return queued_spin_is_locked(lock); + else + return ticket_spin_is_locked(lock); +} + +static __always_inline int arch_spin_is_contended(arch_spinlock_t *lock) +{ + if (static_branch_likely(&combo_qspinlock_key)) + return queued_spin_is_contended(lock); + else + return ticket_spin_is_contended(lock); +} +#else /* CONFIG_RISCV_COMBO_SPINLOCKS */ + #ifdef CONFIG_QUEUED_SPINLOCKS #include -#include #else -#include +#include #endif +#endif /* CONFIG_RISCV_COMBO_SPINLOCKS */ + +#include + #endif /* __ASM_RISCV_SPINLOCK_H */ diff --git a/arch/riscv/kernel/setup.c b/arch/riscv/kernel/setup.c index 32c2e1eb71bd..a447cf360a18 100644 --- a/arch/riscv/kernel/setup.c +++ b/arch/riscv/kernel/setup.c @@ -269,6 +269,18 @@ static void __init parse_dtb(void) #endif } +#ifdef CONFIG_RISCV_COMBO_SPINLOCKS +DEFINE_STATIC_KEY_TRUE(combo_qspinlock_key); +EXPORT_SYMBOL(combo_qspinlock_key); +#endif + +static void __init riscv_spinlock_init(void) +{ +#ifdef CONFIG_RISCV_COMBO_SPINLOCKS + static_branch_disable(&combo_qspinlock_key); +#endif +} + extern void __init init_rt_signal_env(void); void __init setup_arch(char **cmdline_p) @@ -317,6 +329,8 @@ void __init setup_arch(char **cmdline_p) riscv_isa_extension_available(NULL, ZICBOM)) riscv_noncoherent_supported(); riscv_set_dma_cache_alignment(); + + riscv_spinlock_init(); } static int __init topology_init(void) From patchwork Sun Sep 10 08:29:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guo Ren X-Patchwork-Id: 13378474 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 00F0BEE14D8 for ; Sun, 10 Sep 2023 08:31:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345295AbjIJIbQ (ORCPT ); Sun, 10 Sep 2023 04:31:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40024 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346538AbjIJIbO (ORCPT ); Sun, 10 Sep 2023 04:31:14 -0400 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 377FF10D4; Sun, 10 Sep 2023 01:30:51 -0700 (PDT) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 40C15C433D9; Sun, 10 Sep 2023 08:30:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1694334650; bh=SoZvKFZXtbiskPhol9IbrrRdWVVYxE0tEnebIhQLYjY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=cWrcngtzC02RvUnRT/XTeVpudmXjw3I1qeGhGggw3Iq9PSZxVJwmBblHZvCiFPaQc 1tKRdtRI/VPiltgsxVs7id5SPtt+CGH6TLvl7trrg/AD0PmC+/gyiabYMAJuYYpaND BupmgNGxEAL3Lc/D+TS7Gxeacs2qGBGMSVfeIBlobhUCMbhKCmrLofJoAR27zMnAHD MgUMuDrZY7p1ouq9HWZmI1LZR1gbRvDph7p5rjaYpNWRv/hYXfTYZmE/0kcKcvJ96H qpbnT7fIx9lqxnOaeZ4XiPRWmmSK2nAc1C9FBmnpFD0BBm/Fk52hHbKi1xdE/YZIyg Wl35o60jfaamQ== From: guoren@kernel.org To: paul.walmsley@sifive.com, anup@brainfault.org, peterz@infradead.org, mingo@redhat.com, will@kernel.org, palmer@rivosinc.com, longman@redhat.com, boqun.feng@gmail.com, tglx@linutronix.de, paulmck@kernel.org, rostedt@goodmis.org, rdunlap@infradead.org, catalin.marinas@arm.com, conor.dooley@microchip.com, xiaoguang.xing@sophgo.com, bjorn@rivosinc.com, alexghiti@rivosinc.com, keescook@chromium.org, greentime.hu@sifive.com, ajones@ventanamicro.com, jszhang@kernel.org, wefu@redhat.com, wuwei2016@iscas.ac.cn, leobras@redhat.com Cc: linux-arch@vger.kernel.org, linux-riscv@lists.infradead.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-csky@vger.kernel.org, Guo Ren , Guo Ren Subject: [PATCH V11 07/17] riscv: qspinlock: Introduce qspinlock param for command line Date: Sun, 10 Sep 2023 04:29:01 -0400 Message-Id: <20230910082911.3378782-8-guoren@kernel.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20230910082911.3378782-1-guoren@kernel.org> References: <20230910082911.3378782-1-guoren@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Guo Ren Allow cmdline to force the kernel to use queued_spinlock when CONFIG_RISCV_COMBO_SPINLOCKS=y. Signed-off-by: Guo Ren Signed-off-by: Guo Ren --- Documentation/admin-guide/kernel-parameters.txt | 2 ++ arch/riscv/kernel/setup.c | 16 +++++++++++++++- 2 files changed, 17 insertions(+), 1 deletion(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 7dfb540c4f6c..61cacb8dfd0e 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -4693,6 +4693,8 @@ [KNL] Number of legacy pty's. Overwrites compiled-in default number. + qspinlock [RISCV] Force to use qspinlock or auto-detect spinlock. + qspinlock.numa_spinlock_threshold_ns= [NUMA, PV_OPS] Set the time threshold in nanoseconds for the number of intra-node lock hand-offs before the diff --git a/arch/riscv/kernel/setup.c b/arch/riscv/kernel/setup.c index a447cf360a18..0f084f037651 100644 --- a/arch/riscv/kernel/setup.c +++ b/arch/riscv/kernel/setup.c @@ -270,6 +270,15 @@ static void __init parse_dtb(void) } #ifdef CONFIG_RISCV_COMBO_SPINLOCKS +bool enable_qspinlock_key = false; +static int __init queued_spinlock_setup(char *p) +{ + enable_qspinlock_key = true; + + return 0; +} +early_param("qspinlock", queued_spinlock_setup); + DEFINE_STATIC_KEY_TRUE(combo_qspinlock_key); EXPORT_SYMBOL(combo_qspinlock_key); #endif @@ -277,7 +286,12 @@ EXPORT_SYMBOL(combo_qspinlock_key); static void __init riscv_spinlock_init(void) { #ifdef CONFIG_RISCV_COMBO_SPINLOCKS - static_branch_disable(&combo_qspinlock_key); + if (!enable_qspinlock_key) { + static_branch_disable(&combo_qspinlock_key); + pr_info("Ticket spinlock: enabled\n"); + } else { + pr_info("Queued spinlock: enabled\n"); + } #endif } From patchwork Sun Sep 10 08:29:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guo Ren X-Patchwork-Id: 13378475 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C63D8EEB57E for ; Sun, 10 Sep 2023 08:31:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346538AbjIJIbY (ORCPT ); Sun, 10 Sep 2023 04:31:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59726 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232989AbjIJIbX (ORCPT ); Sun, 10 Sep 2023 04:31:23 -0400 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 21BA110FB; Sun, 10 Sep 2023 01:30:57 -0700 (PDT) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 29D41C433C8; Sun, 10 Sep 2023 08:30:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1694334656; bh=b/96e8ctDdPWQM+BjwYA3uNNS7hx8TS7HxYP5gGaMAk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=fm0Let9OCBGIoNmVO+tWV2aF3bPlay5iI8Eg7PyWvfi7GFcR5h1ddebm76ADuzdV7 zlubzUhzZnN7ilI1SHCW7LnbefedHejQCfbyv+YDibAe25M1WWhJ9HM9Ygu0MKgpjg PKSfW55K9A1jzU6xGL6jPhfzwTc9IUt0MaV2II6TtCd7IS6usoMlN/ZAWvixkzjOtq rNv5W9HtexkqJSf7hSzpCXV6D7KKxWjve2+8hp9N0bAmWZ4eg3LOdV9hAVxEpYjXJq ERqn1H+ZIRbtVR34pV7D++ZQWT9jRN0vpbpKgjQ8Vr9eBPWd1mHOtDfm7kL9MJO2mb D2Tzp6TLhUQkg== From: guoren@kernel.org To: paul.walmsley@sifive.com, anup@brainfault.org, peterz@infradead.org, mingo@redhat.com, will@kernel.org, palmer@rivosinc.com, longman@redhat.com, boqun.feng@gmail.com, tglx@linutronix.de, paulmck@kernel.org, rostedt@goodmis.org, rdunlap@infradead.org, catalin.marinas@arm.com, conor.dooley@microchip.com, xiaoguang.xing@sophgo.com, bjorn@rivosinc.com, alexghiti@rivosinc.com, keescook@chromium.org, greentime.hu@sifive.com, ajones@ventanamicro.com, jszhang@kernel.org, wefu@redhat.com, wuwei2016@iscas.ac.cn, leobras@redhat.com Cc: linux-arch@vger.kernel.org, linux-riscv@lists.infradead.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-csky@vger.kernel.org, Guo Ren , Guo Ren Subject: [PATCH V11 08/17] riscv: qspinlock: Add virt_spin_lock() support for KVM guest Date: Sun, 10 Sep 2023 04:29:02 -0400 Message-Id: <20230910082911.3378782-9-guoren@kernel.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20230910082911.3378782-1-guoren@kernel.org> References: <20230910082911.3378782-1-guoren@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Guo Ren Add a static key controlling whether virt_spin_lock() should be called or not. When running on bare metal set the new key to false. The KVM guests fall back to a Test-and-Set spinlock, because fair locks have horrible lock 'holder' preemption issues. The virt_spin_lock_key would shortcut for the queued_spin_lock_slowpath() function that allow virt_spin_lock to hijack it. Signed-off-by: Guo Ren Signed-off-by: Guo Ren --- .../admin-guide/kernel-parameters.txt | 4 +++ arch/riscv/include/asm/sbi.h | 8 +++++ arch/riscv/include/asm/spinlock.h | 22 ++++++++++++++ arch/riscv/kernel/sbi.c | 2 +- arch/riscv/kernel/setup.c | 30 ++++++++++++++++++- 5 files changed, 64 insertions(+), 2 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 61cacb8dfd0e..f75bedc50e00 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -3927,6 +3927,10 @@ no_uaccess_flush [PPC] Don't flush the L1-D cache after accessing user data. + no_virt_spin [RISC-V] Disable virt_spin_lock in KVM guest to use + native_queued_spinlock when the nopvspin option is enabled. + This would help vcpu=pcpu scenarios. + novmcoredd [KNL,KDUMP] Disable device dump. Device dump allows drivers to append dump data to vmcore so you can collect driver diff --git a/arch/riscv/include/asm/sbi.h b/arch/riscv/include/asm/sbi.h index 501e06e52078..e0233b3d7a5f 100644 --- a/arch/riscv/include/asm/sbi.h +++ b/arch/riscv/include/asm/sbi.h @@ -50,6 +50,13 @@ enum sbi_ext_base_fid { SBI_EXT_BASE_GET_MIMPID, }; +enum sbi_ext_base_impl_id { + SBI_EXT_BASE_IMPL_ID_BBL = 0, + SBI_EXT_BASE_IMPL_ID_OPENSBI, + SBI_EXT_BASE_IMPL_ID_XVISOR, + SBI_EXT_BASE_IMPL_ID_KVM, +}; + enum sbi_ext_time_fid { SBI_EXT_TIME_SET_TIMER = 0, }; @@ -269,6 +276,7 @@ int sbi_console_getchar(void); long sbi_get_mvendorid(void); long sbi_get_marchid(void); long sbi_get_mimpid(void); +long sbi_get_firmware_id(void); void sbi_set_timer(uint64_t stime_value); void sbi_shutdown(void); void sbi_send_ipi(unsigned int cpu); diff --git a/arch/riscv/include/asm/spinlock.h b/arch/riscv/include/asm/spinlock.h index 8ea0fee80652..6b38d6616f14 100644 --- a/arch/riscv/include/asm/spinlock.h +++ b/arch/riscv/include/asm/spinlock.h @@ -4,6 +4,28 @@ #define __ASM_RISCV_SPINLOCK_H #ifdef CONFIG_QUEUED_SPINLOCKS +/* + * The KVM guests fall back to a Test-and-Set spinlock, because fair locks + * have horrible lock 'holder' preemption issues. The virt_spin_lock_key + * would shortcut for the queued_spin_lock_slowpath() function that allow + * virt_spin_lock to hijack it. + */ +DECLARE_STATIC_KEY_TRUE(virt_spin_lock_key); + +#define virt_spin_lock virt_spin_lock +static inline bool virt_spin_lock(struct qspinlock *lock) +{ + if (!static_branch_likely(&virt_spin_lock_key)) + return false; + + do { + while (atomic_read(&lock->val) != 0) + cpu_relax(); + } while (atomic_cmpxchg(&lock->val, 0, _Q_LOCKED_VAL) != 0); + + return true; +} + #define _Q_PENDING_LOOPS (1 << 9) #endif diff --git a/arch/riscv/kernel/sbi.c b/arch/riscv/kernel/sbi.c index 88eea3a99ee0..cdd45edc8db4 100644 --- a/arch/riscv/kernel/sbi.c +++ b/arch/riscv/kernel/sbi.c @@ -555,7 +555,7 @@ static inline long sbi_get_spec_version(void) return __sbi_base_ecall(SBI_EXT_BASE_GET_SPEC_VERSION); } -static inline long sbi_get_firmware_id(void) +long sbi_get_firmware_id(void) { return __sbi_base_ecall(SBI_EXT_BASE_GET_IMP_ID); } diff --git a/arch/riscv/kernel/setup.c b/arch/riscv/kernel/setup.c index 0f084f037651..c57d15b05160 100644 --- a/arch/riscv/kernel/setup.c +++ b/arch/riscv/kernel/setup.c @@ -26,6 +26,7 @@ #include #include #include +#include #include #include #include @@ -283,16 +284,43 @@ DEFINE_STATIC_KEY_TRUE(combo_qspinlock_key); EXPORT_SYMBOL(combo_qspinlock_key); #endif +#ifdef CONFIG_QUEUED_SPINLOCKS +static bool no_virt_spin_key = false; +DEFINE_STATIC_KEY_TRUE(virt_spin_lock_key); + +static int __init no_virt_spin_setup(char *p) +{ + no_virt_spin_key = true; + + return 0; +} +early_param("no_virt_spin", no_virt_spin_setup); + +static void __init virt_spin_lock_init(void) +{ + if (sbi_get_firmware_id() != SBI_EXT_BASE_IMPL_ID_KVM || + no_virt_spin_key) + static_branch_disable(&virt_spin_lock_key); + else + pr_info("Enable virt_spin_lock\n"); +} +#endif + static void __init riscv_spinlock_init(void) { #ifdef CONFIG_RISCV_COMBO_SPINLOCKS - if (!enable_qspinlock_key) { + if (!enable_qspinlock_key && + (sbi_get_firmware_id() != SBI_EXT_BASE_IMPL_ID_KVM)) { static_branch_disable(&combo_qspinlock_key); pr_info("Ticket spinlock: enabled\n"); } else { pr_info("Queued spinlock: enabled\n"); } #endif + +#ifdef CONFIG_QUEUED_SPINLOCKS + virt_spin_lock_init(); +#endif } extern void __init init_rt_signal_env(void); From patchwork Sun Sep 10 08:29:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guo Ren X-Patchwork-Id: 13378476 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F3316EEB57E for ; Sun, 10 Sep 2023 08:31:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346541AbjIJIbk (ORCPT ); Sun, 10 Sep 2023 04:31:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52198 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232989AbjIJIbj (ORCPT ); Sun, 10 Sep 2023 04:31:39 -0400 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4C66219B4; Sun, 10 Sep 2023 01:31:03 -0700 (PDT) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1AA79C43395; Sun, 10 Sep 2023 08:30:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1694334662; bh=3murlTvGUwbnyKtHCIZhxZPJhHjJerN3MgRLx0eVQ7E=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=RSYpp5ajmtaC1ah1CnQFUyhv+4DzJG5aRRFple1r1te7W7gYezMHBYFQm96quOKmJ Kt9xrYZkiICPPMf9RCONHwdEe74PP0agYfojznBtsRenXfwJP/BjM45e86IGakjq5s qCQVTr4ZgFzMzBSrNJuXnFoqrcmHV3u6DKwF10hZ7RO0HIHjbfgSkMcR7mPsBmVJSe gZYRKdNdsKpW3ZZxXCVJe1/j1ZnGIqWmhOc8KN65rLumWFNOr1WGxpIEhYT4Csn4qm pS/0vY+vjU2WoBXUZ+DqdzBOCvehB/RFYvqI7xSvSZ0eDQAQqyc5SVb6n2vFzN35+A kqdARpTOsJNCQ== From: guoren@kernel.org To: paul.walmsley@sifive.com, anup@brainfault.org, peterz@infradead.org, mingo@redhat.com, will@kernel.org, palmer@rivosinc.com, longman@redhat.com, boqun.feng@gmail.com, tglx@linutronix.de, paulmck@kernel.org, rostedt@goodmis.org, rdunlap@infradead.org, catalin.marinas@arm.com, conor.dooley@microchip.com, xiaoguang.xing@sophgo.com, bjorn@rivosinc.com, alexghiti@rivosinc.com, keescook@chromium.org, greentime.hu@sifive.com, ajones@ventanamicro.com, jszhang@kernel.org, wefu@redhat.com, wuwei2016@iscas.ac.cn, leobras@redhat.com Cc: linux-arch@vger.kernel.org, linux-riscv@lists.infradead.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-csky@vger.kernel.org, Guo Ren , Guo Ren Subject: [PATCH V11 09/17] riscv: qspinlock: errata: Add ERRATA_THEAD_WRITE_ONCE fixup Date: Sun, 10 Sep 2023 04:29:03 -0400 Message-Id: <20230910082911.3378782-10-guoren@kernel.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20230910082911.3378782-1-guoren@kernel.org> References: <20230910082911.3378782-1-guoren@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Guo Ren The early version of T-Head C9xx cores has a store merge buffer delay problem. The store merge buffer could improve the store queue performance by merging multi-store requests, but when there are not continued store requests, the prior single store request would be waiting in the store queue for a long time. That would cause significant problems for communication between multi-cores. This problem was found on sg2042 & th1520 platforms with the qspinlock lock torture test. So appending a fence w.o could immediately flush the store merge buffer and let other cores see the write result. This will apply the WRITE_ONCE errata to handle the non-standard behavior via appending a fence w.o instruction for WRITE_ONCE(). Signed-off-by: Guo Ren Signed-off-by: Guo Ren Reviewed-by: Leonardo Bras --- arch/riscv/Kconfig.errata | 19 +++++++++++++++++++ arch/riscv/errata/thead/errata.c | 20 ++++++++++++++++++++ arch/riscv/include/asm/errata_list.h | 13 ------------- arch/riscv/include/asm/rwonce.h | 24 ++++++++++++++++++++++++ arch/riscv/include/asm/vendorid_list.h | 14 ++++++++++++++ include/asm-generic/rwonce.h | 2 ++ 6 files changed, 79 insertions(+), 13 deletions(-) create mode 100644 arch/riscv/include/asm/rwonce.h diff --git a/arch/riscv/Kconfig.errata b/arch/riscv/Kconfig.errata index 1aa85a427ff3..c919cc3f1a3a 100644 --- a/arch/riscv/Kconfig.errata +++ b/arch/riscv/Kconfig.errata @@ -77,4 +77,23 @@ config ERRATA_THEAD_PMU If you don't know what to do here, say "Y". +config ERRATA_THEAD_WRITE_ONCE + bool "Apply T-Head WRITE_ONCE errata" + depends on ERRATA_THEAD + default y + help + The early version of T-Head C9xx cores has a store merge buffer + delay problem. The store merge buffer could improve the store queue + performance by merging multi-store requests, but when there are no + continued store requests, the prior single store request would be + waiting in the store queue for a long time. That would cause + significant problems for communication between multi-cores. Appending + a fence w.o could immediately flush the store merge buffer and let + other cores see the write result. + + This will apply the WRITE_ONCE errata to handle the non-standard + behavior via appending a fence w.o instruction for WRITE_ONCE(). + + If you don't know what to do here, say "Y". + endmenu # "CPU errata selection" diff --git a/arch/riscv/errata/thead/errata.c b/arch/riscv/errata/thead/errata.c index be84b14f0118..751eb5a7f614 100644 --- a/arch/riscv/errata/thead/errata.c +++ b/arch/riscv/errata/thead/errata.c @@ -69,6 +69,23 @@ static bool errata_probe_pmu(unsigned int stage, return true; } +static bool errata_probe_write_once(unsigned int stage, + unsigned long arch_id, unsigned long impid) +{ + if (!IS_ENABLED(CONFIG_ERRATA_THEAD_WRITE_ONCE)) + return false; + + /* target-c9xx cores report arch_id and impid as 0 */ + if (arch_id != 0 || impid != 0) + return false; + + if (stage == RISCV_ALTERNATIVES_BOOT || + stage == RISCV_ALTERNATIVES_MODULE) + return true; + + return false; +} + static u32 thead_errata_probe(unsigned int stage, unsigned long archid, unsigned long impid) { @@ -83,6 +100,9 @@ static u32 thead_errata_probe(unsigned int stage, if (errata_probe_pmu(stage, archid, impid)) cpu_req_errata |= BIT(ERRATA_THEAD_PMU); + if (errata_probe_write_once(stage, archid, impid)) + cpu_req_errata |= BIT(ERRATA_THEAD_WRITE_ONCE); + return cpu_req_errata; } diff --git a/arch/riscv/include/asm/errata_list.h b/arch/riscv/include/asm/errata_list.h index 712cab7adffe..fbb2b8d39321 100644 --- a/arch/riscv/include/asm/errata_list.h +++ b/arch/riscv/include/asm/errata_list.h @@ -11,19 +11,6 @@ #include #include -#ifdef CONFIG_ERRATA_SIFIVE -#define ERRATA_SIFIVE_CIP_453 0 -#define ERRATA_SIFIVE_CIP_1200 1 -#define ERRATA_SIFIVE_NUMBER 2 -#endif - -#ifdef CONFIG_ERRATA_THEAD -#define ERRATA_THEAD_PBMT 0 -#define ERRATA_THEAD_CMO 1 -#define ERRATA_THEAD_PMU 2 -#define ERRATA_THEAD_NUMBER 3 -#endif - #ifdef __ASSEMBLY__ #define ALT_INSN_FAULT(x) \ diff --git a/arch/riscv/include/asm/rwonce.h b/arch/riscv/include/asm/rwonce.h new file mode 100644 index 000000000000..be0b8864969d --- /dev/null +++ b/arch/riscv/include/asm/rwonce.h @@ -0,0 +1,24 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef __ASM_RWONCE_H +#define __ASM_RWONCE_H + +#include +#include +#include + +#define __WRITE_ONCE(x, val) \ +do { \ + *(volatile typeof(x) *)&(x) = (val); \ + asm volatile(ALTERNATIVE( \ + __nops(1), \ + "fence w, o\n\t", \ + THEAD_VENDOR_ID, \ + ERRATA_THEAD_WRITE_ONCE, \ + CONFIG_ERRATA_THEAD_WRITE_ONCE) \ + : : : "memory"); \ +} while (0) + +#include + +#endif /* __ASM_RWONCE_H */ diff --git a/arch/riscv/include/asm/vendorid_list.h b/arch/riscv/include/asm/vendorid_list.h index cb89af3f0704..73078cfe4029 100644 --- a/arch/riscv/include/asm/vendorid_list.h +++ b/arch/riscv/include/asm/vendorid_list.h @@ -8,4 +8,18 @@ #define SIFIVE_VENDOR_ID 0x489 #define THEAD_VENDOR_ID 0x5b7 +#ifdef CONFIG_ERRATA_SIFIVE +#define ERRATA_SIFIVE_CIP_453 0 +#define ERRATA_SIFIVE_CIP_1200 1 +#define ERRATA_SIFIVE_NUMBER 2 +#endif + +#ifdef CONFIG_ERRATA_THEAD +#define ERRATA_THEAD_PBMT 0 +#define ERRATA_THEAD_CMO 1 +#define ERRATA_THEAD_PMU 2 +#define ERRATA_THEAD_WRITE_ONCE 3 +#define ERRATA_THEAD_NUMBER 4 +#endif + #endif diff --git a/include/asm-generic/rwonce.h b/include/asm-generic/rwonce.h index 8d0a6280e982..fb07fe8c6e45 100644 --- a/include/asm-generic/rwonce.h +++ b/include/asm-generic/rwonce.h @@ -50,10 +50,12 @@ __READ_ONCE(x); \ }) +#ifndef __WRITE_ONCE #define __WRITE_ONCE(x, val) \ do { \ *(volatile typeof(x) *)&(x) = (val); \ } while (0) +#endif #define WRITE_ONCE(x, val) \ do { \ From patchwork Sun Sep 10 08:29:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guo Ren X-Patchwork-Id: 13378477 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2AB52EEB57E for ; Sun, 10 Sep 2023 08:31:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346545AbjIJIbr (ORCPT ); Sun, 10 Sep 2023 04:31:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33872 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232989AbjIJIbq (ORCPT ); Sun, 10 Sep 2023 04:31:46 -0400 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 002F2E7E; Sun, 10 Sep 2023 01:31:11 -0700 (PDT) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 44B89C433AD; Sun, 10 Sep 2023 08:31:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1694334668; bh=rVLNDURpOo6qyHiuDOTv1S9pmkvmttipsDX3ZB78S3Q=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=J4vv6ZK4m8+ZAYwWtLVLh0MGfPo2+DIhTEV60AiAjhHORFtDmWtIkS+BlESCUmkEo mXfGVL2xe+CW+Cw/i1vYBvg149LAmnA/Sm/7SYj7BinnCDfFR8xqmrKiSB7IEuZHoG L0d04TaCfcKTbiCn3rGUimhpuT7v+Y4/NG9+vXxA0CRX5Tyk2mOHA2hMjOojDrAkVp +k1ByKdKCbaA0SV5+tafSpELg24ZkI1yV2ErBGv7QZQhSNtRNOIMtfx1Xiuj0tQgze 0o/Utrj89jmKhS4VOu/9A/1NwYpSKxiuWJKAlyzGG05mhN+Y7XKQrKXBsGUZ1r0dsS uA04kMd6m4fdA== From: guoren@kernel.org To: paul.walmsley@sifive.com, anup@brainfault.org, peterz@infradead.org, mingo@redhat.com, will@kernel.org, palmer@rivosinc.com, longman@redhat.com, boqun.feng@gmail.com, tglx@linutronix.de, paulmck@kernel.org, rostedt@goodmis.org, rdunlap@infradead.org, catalin.marinas@arm.com, conor.dooley@microchip.com, xiaoguang.xing@sophgo.com, bjorn@rivosinc.com, alexghiti@rivosinc.com, keescook@chromium.org, greentime.hu@sifive.com, ajones@ventanamicro.com, jszhang@kernel.org, wefu@redhat.com, wuwei2016@iscas.ac.cn, leobras@redhat.com Cc: linux-arch@vger.kernel.org, linux-riscv@lists.infradead.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-csky@vger.kernel.org, Guo Ren , Guo Ren Subject: [PATCH V11 10/17] riscv: qspinlock: errata: Enable qspinlock for T-HEAD processors Date: Sun, 10 Sep 2023 04:29:04 -0400 Message-Id: <20230910082911.3378782-11-guoren@kernel.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20230910082911.3378782-1-guoren@kernel.org> References: <20230910082911.3378782-1-guoren@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Guo Ren According to qspinlock requirements, RISC-V gives out a weak LR/SC forward progress guarantee which does not satisfy qspinlock. But many vendors could produce stronger forward guarantee LR/SC to ensure the xchg_tail could be finished in time on any kind of hart. T-HEAD is the vendor which implements strong forward guarantee LR/SC instruction pairs, so enable qspinlock for T-HEAD with errata init help. Signed-off-by: Guo Ren Signed-off-by: Guo Ren Reviewed-by: Leonardo Bras --- arch/riscv/errata/thead/errata.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/arch/riscv/errata/thead/errata.c b/arch/riscv/errata/thead/errata.c index 751eb5a7f614..0df6a67302c0 100644 --- a/arch/riscv/errata/thead/errata.c +++ b/arch/riscv/errata/thead/errata.c @@ -86,6 +86,13 @@ static bool errata_probe_write_once(unsigned int stage, return false; } +extern bool enable_qspinlock_key; +static void errata_probe_qspinlock(unsigned int stage) +{ + if (stage == RISCV_ALTERNATIVES_BOOT) + enable_qspinlock_key = true; +} + static u32 thead_errata_probe(unsigned int stage, unsigned long archid, unsigned long impid) { @@ -103,6 +110,8 @@ static u32 thead_errata_probe(unsigned int stage, if (errata_probe_write_once(stage, archid, impid)) cpu_req_errata |= BIT(ERRATA_THEAD_WRITE_ONCE); + errata_probe_qspinlock(stage); + return cpu_req_errata; } From patchwork Sun Sep 10 08:29:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guo Ren X-Patchwork-Id: 13378478 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4E7C0EEB57E for ; Sun, 10 Sep 2023 08:31:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346550AbjIJIb4 (ORCPT ); Sun, 10 Sep 2023 04:31:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38322 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232989AbjIJIb4 (ORCPT ); Sun, 10 Sep 2023 04:31:56 -0400 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8890210F0; Sun, 10 Sep 2023 01:31:16 -0700 (PDT) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2FBB0C433C9; Sun, 10 Sep 2023 08:31:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1694334674; bh=CGbBQmlOMiU7KGhDaeYsvrtFbv9g7V9LknoZMK3XHU0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=lCThd1tl2tykwK3eWiOPcnl54BI9mVFr+fDdyPCbtkpojBfLGMNDVVulF+O6+y3Pi a/UIQXm6F1pkP4NTkDdtFmtqHJEsWvz5qlSRaCi2ob3CwxdJ0TPGEv5BwpjtiBYtS/ jXoIh671CI4UCjavdWqPK7z0oStaNzKhs2ySCztbPt71JH0S3floPylQ/ivZig0W33 Rdvp+sXwCArfgZs//eEHJXowarZ1IiymCnXkKcPRHZNHK7i8s41ifWjY75cn6/TtSQ D+wgz/sZ9onk7/9mZcLYezW6yZUq7AhbIk6O289NricZFW75KXAEkEL37X8H+2t8iF SsPZ6CdUG0yJw== From: guoren@kernel.org To: paul.walmsley@sifive.com, anup@brainfault.org, peterz@infradead.org, mingo@redhat.com, will@kernel.org, palmer@rivosinc.com, longman@redhat.com, boqun.feng@gmail.com, tglx@linutronix.de, paulmck@kernel.org, rostedt@goodmis.org, rdunlap@infradead.org, catalin.marinas@arm.com, conor.dooley@microchip.com, xiaoguang.xing@sophgo.com, bjorn@rivosinc.com, alexghiti@rivosinc.com, keescook@chromium.org, greentime.hu@sifive.com, ajones@ventanamicro.com, jszhang@kernel.org, wefu@redhat.com, wuwei2016@iscas.ac.cn, leobras@redhat.com Cc: linux-arch@vger.kernel.org, linux-riscv@lists.infradead.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-csky@vger.kernel.org, Guo Ren , Guo Ren Subject: [PATCH V11 11/17] RISC-V: paravirt: pvqspinlock: Add paravirt qspinlock skeleton Date: Sun, 10 Sep 2023 04:29:05 -0400 Message-Id: <20230910082911.3378782-12-guoren@kernel.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20230910082911.3378782-1-guoren@kernel.org> References: <20230910082911.3378782-1-guoren@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Guo Ren Using static_call to switch between: native_queued_spin_lock_slowpath() __pv_queued_spin_lock_slowpath() native_queued_spin_unlock() __pv_queued_spin_unlock() Finish the pv_wait implementation, but pv_kick needs the SBI definition of the next patches. Signed-off-by: Guo Ren Signed-off-by: Guo Ren Reviewed-by: Leonardo Bras --- arch/riscv/include/asm/Kbuild | 1 - arch/riscv/include/asm/qspinlock.h | 35 +++++++++++++ arch/riscv/include/asm/qspinlock_paravirt.h | 29 +++++++++++ arch/riscv/include/asm/spinlock.h | 2 +- arch/riscv/kernel/qspinlock_paravirt.c | 57 +++++++++++++++++++++ arch/riscv/kernel/setup.c | 4 ++ 6 files changed, 126 insertions(+), 2 deletions(-) create mode 100644 arch/riscv/include/asm/qspinlock.h create mode 100644 arch/riscv/include/asm/qspinlock_paravirt.h create mode 100644 arch/riscv/kernel/qspinlock_paravirt.c diff --git a/arch/riscv/include/asm/Kbuild b/arch/riscv/include/asm/Kbuild index a0dc85e4a754..b89cb3b73c13 100644 --- a/arch/riscv/include/asm/Kbuild +++ b/arch/riscv/include/asm/Kbuild @@ -7,6 +7,5 @@ generic-y += parport.h generic-y += spinlock_types.h generic-y += qrwlock.h generic-y += qrwlock_types.h -generic-y += qspinlock.h generic-y += user.h generic-y += vmlinux.lds.h diff --git a/arch/riscv/include/asm/qspinlock.h b/arch/riscv/include/asm/qspinlock.h new file mode 100644 index 000000000000..7d4f416c908c --- /dev/null +++ b/arch/riscv/include/asm/qspinlock.h @@ -0,0 +1,35 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (c), 2023 Alibaba Cloud + * Authors: + * Guo Ren + */ + +#ifndef _ASM_RISCV_QSPINLOCK_H +#define _ASM_RISCV_QSPINLOCK_H + +#ifdef CONFIG_PARAVIRT_SPINLOCKS +#include + +/* How long a lock should spin before we consider blocking */ +#define SPIN_THRESHOLD (1 << 15) + +void native_queued_spin_lock_slowpath(struct qspinlock *lock, u32 val); +void __pv_init_lock_hash(void); +void __pv_queued_spin_lock_slowpath(struct qspinlock *lock, u32 val); + +static inline void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) +{ + static_call(pv_queued_spin_lock_slowpath)(lock, val); +} + +#define queued_spin_unlock queued_spin_unlock +static inline void queued_spin_unlock(struct qspinlock *lock) +{ + static_call(pv_queued_spin_unlock)(lock); +} +#endif /* CONFIG_PARAVIRT_SPINLOCKS */ + +#include + +#endif /* _ASM_RISCV_QSPINLOCK_H */ diff --git a/arch/riscv/include/asm/qspinlock_paravirt.h b/arch/riscv/include/asm/qspinlock_paravirt.h new file mode 100644 index 000000000000..9681e851f69d --- /dev/null +++ b/arch/riscv/include/asm/qspinlock_paravirt.h @@ -0,0 +1,29 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (c), 2023 Alibaba Cloud + * Authors: + * Guo Ren + */ + +#ifndef _ASM_RISCV_QSPINLOCK_PARAVIRT_H +#define _ASM_RISCV_QSPINLOCK_PARAVIRT_H + +void pv_wait(u8 *ptr, u8 val); +void pv_kick(int cpu); + +void dummy_queued_spin_lock_slowpath(struct qspinlock *lock, u32 val); +void dummy_queued_spin_unlock(struct qspinlock *lock); + +DECLARE_STATIC_CALL(pv_queued_spin_lock_slowpath, dummy_queued_spin_lock_slowpath); +DECLARE_STATIC_CALL(pv_queued_spin_unlock, dummy_queued_spin_unlock); + +void __init pv_qspinlock_init(void); + +static inline bool pv_is_native_spin_unlock(void) +{ + return false; +} + +void __pv_queued_spin_unlock(struct qspinlock *lock); + +#endif /* _ASM_RISCV_QSPINLOCK_PARAVIRT_H */ diff --git a/arch/riscv/include/asm/spinlock.h b/arch/riscv/include/asm/spinlock.h index 6b38d6616f14..ed4253f491fe 100644 --- a/arch/riscv/include/asm/spinlock.h +++ b/arch/riscv/include/asm/spinlock.h @@ -39,7 +39,7 @@ static inline bool virt_spin_lock(struct qspinlock *lock) #undef arch_spin_trylock #undef arch_spin_unlock -#include +#include #include #undef arch_spin_is_locked diff --git a/arch/riscv/kernel/qspinlock_paravirt.c b/arch/riscv/kernel/qspinlock_paravirt.c new file mode 100644 index 000000000000..85ff5a3ec234 --- /dev/null +++ b/arch/riscv/kernel/qspinlock_paravirt.c @@ -0,0 +1,57 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (c), 2023 Alibaba Cloud + * Authors: + * Guo Ren + */ + +#include +#include +#include + +void pv_kick(int cpu) +{ + return; +} + +void pv_wait(u8 *ptr, u8 val) +{ + unsigned long flags; + + if (in_nmi()) + return; + + local_irq_save(flags); + if (READ_ONCE(*ptr) != val) + goto out; + + /* wait_for_interrupt(); */ +out: + local_irq_restore(flags); +} + +static void native_queued_spin_unlock(struct qspinlock *lock) +{ + smp_store_release(&lock->locked, 0); +} + +DEFINE_STATIC_CALL(pv_queued_spin_lock_slowpath, native_queued_spin_lock_slowpath); +EXPORT_STATIC_CALL(pv_queued_spin_lock_slowpath); + +DEFINE_STATIC_CALL(pv_queued_spin_unlock, native_queued_spin_unlock); +EXPORT_STATIC_CALL(pv_queued_spin_unlock); + +void __init pv_qspinlock_init(void) +{ + if (num_possible_cpus() == 1) + return; + + if(sbi_get_firmware_id() != SBI_EXT_BASE_IMPL_ID_KVM) + return; + + pr_info("PV qspinlocks enabled\n"); + __pv_init_lock_hash(); + + static_call_update(pv_queued_spin_lock_slowpath, __pv_queued_spin_lock_slowpath); + static_call_update(pv_queued_spin_unlock, __pv_queued_spin_unlock); +} diff --git a/arch/riscv/kernel/setup.c b/arch/riscv/kernel/setup.c index c57d15b05160..88690751f2ee 100644 --- a/arch/riscv/kernel/setup.c +++ b/arch/riscv/kernel/setup.c @@ -321,6 +321,10 @@ static void __init riscv_spinlock_init(void) #ifdef CONFIG_QUEUED_SPINLOCKS virt_spin_lock_init(); #endif + +#ifdef CONFIG_PARAVIRT_SPINLOCKS + pv_qspinlock_init(); +#endif } extern void __init init_rt_signal_env(void); From patchwork Sun Sep 10 08:29:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guo Ren X-Patchwork-Id: 13378479 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 556C7EE7FF4 for ; Sun, 10 Sep 2023 08:32:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346551AbjIJIcE (ORCPT ); Sun, 10 Sep 2023 04:32:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35202 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244293AbjIJIcC (ORCPT ); Sun, 10 Sep 2023 04:32:02 -0400 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D7014E71; Sun, 10 Sep 2023 01:31:21 -0700 (PDT) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1B287C433B7; Sun, 10 Sep 2023 08:31:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1694334680; bh=PlD8W8FqmIw1B4tEQBaEhpv4wvhKKufMUHFNMnWNSRc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=gKuELwcQ4brAGVU9V37RxmhBlSvMSLPD/dRoPArB009I0S8ge7Qzu9JIIdscgSOKB 1uFBmbhBm8QCK8D6TxOAps/6OEtvE5z60y8GDzpz8AQ0OuYM9/xUgRzT/0KmV1feG8 UHja5oVgpTrr1CdaOFkK/1OFi22hCvQliP9mlonlc6B6bl8XfdQaJZUx49apuzOZZd S7zQG6geh4EIutDJNDjS7LJPyP51UXHZSwHkPraqT5rqzCbeNf1Rcm8gDIUDelwV6H 7n16XaUkkhmp/UM5wwDJ86XKRzJvJkS1y+CoZMZMuWmvRtRhTx/rnJ6RWDDcCpu+8t 78WDq+gK1D9lg== From: guoren@kernel.org To: paul.walmsley@sifive.com, anup@brainfault.org, peterz@infradead.org, mingo@redhat.com, will@kernel.org, palmer@rivosinc.com, longman@redhat.com, boqun.feng@gmail.com, tglx@linutronix.de, paulmck@kernel.org, rostedt@goodmis.org, rdunlap@infradead.org, catalin.marinas@arm.com, conor.dooley@microchip.com, xiaoguang.xing@sophgo.com, bjorn@rivosinc.com, alexghiti@rivosinc.com, keescook@chromium.org, greentime.hu@sifive.com, ajones@ventanamicro.com, jszhang@kernel.org, wefu@redhat.com, wuwei2016@iscas.ac.cn, leobras@redhat.com Cc: linux-arch@vger.kernel.org, linux-riscv@lists.infradead.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-csky@vger.kernel.org, Guo Ren , Guo Ren Subject: [PATCH V11 12/17] RISC-V: paravirt: pvqspinlock: Add nopvspin kernel parameter Date: Sun, 10 Sep 2023 04:29:06 -0400 Message-Id: <20230910082911.3378782-13-guoren@kernel.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20230910082911.3378782-1-guoren@kernel.org> References: <20230910082911.3378782-1-guoren@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Guo Ren Disables the qspinlock slow path using PV optimizations which allow the hypervisor to 'idle' the guest on lock contention. Signed-off-by: Guo Ren Signed-off-by: Guo Ren Reviewed-by: Leonardo Bras --- Documentation/admin-guide/kernel-parameters.txt | 2 +- arch/riscv/kernel/qspinlock_paravirt.c | 13 +++++++++++++ 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index f75bedc50e00..e74aed631573 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -3857,7 +3857,7 @@ as generic guest with no PV drivers. Currently support XEN HVM, KVM, HYPER_V and VMWARE guest. - nopvspin [X86,XEN,KVM] + nopvspin [X86,XEN,KVM,RISC-V] Disables the qspinlock slow path using PV optimizations which allow the hypervisor to 'idle' the guest on lock contention. diff --git a/arch/riscv/kernel/qspinlock_paravirt.c b/arch/riscv/kernel/qspinlock_paravirt.c index 85ff5a3ec234..a0ad4657f437 100644 --- a/arch/riscv/kernel/qspinlock_paravirt.c +++ b/arch/riscv/kernel/qspinlock_paravirt.c @@ -41,8 +41,21 @@ EXPORT_STATIC_CALL(pv_queued_spin_lock_slowpath); DEFINE_STATIC_CALL(pv_queued_spin_unlock, native_queued_spin_unlock); EXPORT_STATIC_CALL(pv_queued_spin_unlock); +static bool nopvspin; +static __init int parse_nopvspin(char *arg) +{ + nopvspin = true; + return 0; +} +early_param("nopvspin", parse_nopvspin); + void __init pv_qspinlock_init(void) { + if (nopvspin) { + pr_info("PV qspinlocks disabled\n"); + return; + } + if (num_possible_cpus() == 1) return; From patchwork Sun Sep 10 08:29:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guo Ren X-Patchwork-Id: 13378480 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7746EEB57E for ; Sun, 10 Sep 2023 08:32:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344946AbjIJIcI (ORCPT ); Sun, 10 Sep 2023 04:32:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35344 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235671AbjIJIcI (ORCPT ); Sun, 10 Sep 2023 04:32:08 -0400 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 863B8170C; Sun, 10 Sep 2023 01:31:28 -0700 (PDT) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 06695C433C7; Sun, 10 Sep 2023 08:31:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1694334686; bh=XRClpokZATGms/5hSmQVWqg3PgTsbj60ag4pgT82d1g=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DAOn5xYy1snXrOmFmNehUeHzj1Z0ze8me0Irn4+sU5MWVFFo5IVtlgSxXBj+N5x66 SThfOeVincosujHlIPmZwaPH7xmI30VIyYyLOLJE3Bl/vsoZAaPr1jyY3JN/Ne08Fh a2k8iW8/2r/OtbySdghJbCUCJTkyPRaE1dVLz9HACSYHii/unurbLmqg9m5QaHjxLA EqJD/2+Sl/GAUpU9Ueyi2qO6xrtIM5p2E/EELjjJFh4Hd8QRVtQXv9J2o0kNjFnfAk G0voEZGJh1qlLocmGgE7uF+4hz00VcMN4Cpu/u15U7gqDQYozJOJU9lJUB45CjGWhU 4c3Ju+DiYrSGw== From: guoren@kernel.org To: paul.walmsley@sifive.com, anup@brainfault.org, peterz@infradead.org, mingo@redhat.com, will@kernel.org, palmer@rivosinc.com, longman@redhat.com, boqun.feng@gmail.com, tglx@linutronix.de, paulmck@kernel.org, rostedt@goodmis.org, rdunlap@infradead.org, catalin.marinas@arm.com, conor.dooley@microchip.com, xiaoguang.xing@sophgo.com, bjorn@rivosinc.com, alexghiti@rivosinc.com, keescook@chromium.org, greentime.hu@sifive.com, ajones@ventanamicro.com, jszhang@kernel.org, wefu@redhat.com, wuwei2016@iscas.ac.cn, leobras@redhat.com Cc: linux-arch@vger.kernel.org, linux-riscv@lists.infradead.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-csky@vger.kernel.org, Guo Ren , Guo Ren Subject: [PATCH V11 13/17] RISC-V: paravirt: pvqspinlock: Add SBI implementation Date: Sun, 10 Sep 2023 04:29:07 -0400 Message-Id: <20230910082911.3378782-14-guoren@kernel.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20230910082911.3378782-1-guoren@kernel.org> References: <20230910082911.3378782-1-guoren@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Guo Ren Implement pv_kick with SBI implementation, and add SBI_EXT_PVLOCK extension detection. Signed-off-by: Guo Ren Signed-off-by: Guo Ren Reviewed-by: Leonardo Bras --- arch/riscv/include/asm/sbi.h | 6 ++++++ arch/riscv/kernel/qspinlock_paravirt.c | 7 ++++++- 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/arch/riscv/include/asm/sbi.h b/arch/riscv/include/asm/sbi.h index e0233b3d7a5f..3533f8d4f3e2 100644 --- a/arch/riscv/include/asm/sbi.h +++ b/arch/riscv/include/asm/sbi.h @@ -30,6 +30,7 @@ enum sbi_ext_id { SBI_EXT_HSM = 0x48534D, SBI_EXT_SRST = 0x53525354, SBI_EXT_PMU = 0x504D55, + SBI_EXT_PVLOCK = 0xAB0401, /* Experimentals extensions must lie within this range */ SBI_EXT_EXPERIMENTAL_START = 0x08000000, @@ -243,6 +244,11 @@ enum sbi_pmu_ctr_type { /* Flags defined for counter stop function */ #define SBI_PMU_STOP_FLAG_RESET (1 << 0) +/* SBI PVLOCK (kick cpu out of wfi) */ +enum sbi_ext_pvlock_fid { + SBI_EXT_PVLOCK_KICK_CPU = 0, +}; + #define SBI_SPEC_VERSION_DEFAULT 0x1 #define SBI_SPEC_VERSION_MAJOR_SHIFT 24 #define SBI_SPEC_VERSION_MAJOR_MASK 0x7f diff --git a/arch/riscv/kernel/qspinlock_paravirt.c b/arch/riscv/kernel/qspinlock_paravirt.c index a0ad4657f437..571626f350be 100644 --- a/arch/riscv/kernel/qspinlock_paravirt.c +++ b/arch/riscv/kernel/qspinlock_paravirt.c @@ -11,6 +11,8 @@ void pv_kick(int cpu) { + sbi_ecall(SBI_EXT_PVLOCK, SBI_EXT_PVLOCK_KICK_CPU, + cpuid_to_hartid_map(cpu), 0, 0, 0, 0, 0); return; } @@ -25,7 +27,7 @@ void pv_wait(u8 *ptr, u8 val) if (READ_ONCE(*ptr) != val) goto out; - /* wait_for_interrupt(); */ + wait_for_interrupt(); out: local_irq_restore(flags); } @@ -62,6 +64,9 @@ void __init pv_qspinlock_init(void) if(sbi_get_firmware_id() != SBI_EXT_BASE_IMPL_ID_KVM) return; + if (!sbi_probe_extension(SBI_EXT_PVLOCK)) + return; + pr_info("PV qspinlocks enabled\n"); __pv_init_lock_hash(); From patchwork Sun Sep 10 08:29:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guo Ren X-Patchwork-Id: 13378481 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1940FEE57DF for ; Sun, 10 Sep 2023 08:32:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346554AbjIJIcP (ORCPT ); Sun, 10 Sep 2023 04:32:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35390 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235671AbjIJIcO (ORCPT ); Sun, 10 Sep 2023 04:32:14 -0400 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 073B01728; Sun, 10 Sep 2023 01:31:34 -0700 (PDT) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E823CC433C8; Sun, 10 Sep 2023 08:31:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1694334692; bh=tGHFuxIP3SbPPf/Epb/Xnkxu5WfdeddjXtXxQeK/obc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=YUKo/luMDwQ8ZTHKumNn75SdYE5FDukIjOgss5a5AbXFbueaYWrxKwu9Qkyj8QpzH 6ZTe9MLwdEREPEFTO+BHHd0E1QiUV/q/RCTlcE04SMv7lx+zDLCynlX3vKHZt2z7va E/11VNyj8a4aIGWv24FBkPQordBCQuvBtQNul7krmNBWS3/gMvved/HvortUZXOw1B HHCp8xJQaao+4BHm+WuJofLhiQSVvwQG3eqADFFZTcafRLFP/1waYYPh0PlZUpFy+J VzQ8ykVD7bRApcaRbqsfoR8egd0/EE0OxCXQL4dsYaYUykxPJHUg9F7Cy83ImCsGQ2 qLELasFU6ESIg== From: guoren@kernel.org To: paul.walmsley@sifive.com, anup@brainfault.org, peterz@infradead.org, mingo@redhat.com, will@kernel.org, palmer@rivosinc.com, longman@redhat.com, boqun.feng@gmail.com, tglx@linutronix.de, paulmck@kernel.org, rostedt@goodmis.org, rdunlap@infradead.org, catalin.marinas@arm.com, conor.dooley@microchip.com, xiaoguang.xing@sophgo.com, bjorn@rivosinc.com, alexghiti@rivosinc.com, keescook@chromium.org, greentime.hu@sifive.com, ajones@ventanamicro.com, jszhang@kernel.org, wefu@redhat.com, wuwei2016@iscas.ac.cn, leobras@redhat.com Cc: linux-arch@vger.kernel.org, linux-riscv@lists.infradead.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-csky@vger.kernel.org, Guo Ren , Guo Ren Subject: [PATCH V11 14/17] RISC-V: paravirt: pvqspinlock: Add kconfig entry Date: Sun, 10 Sep 2023 04:29:08 -0400 Message-Id: <20230910082911.3378782-15-guoren@kernel.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20230910082911.3378782-1-guoren@kernel.org> References: <20230910082911.3378782-1-guoren@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Guo Ren Add kconfig entry for paravirt_spinlock, an unfair qspinlock virtualization-friendly backend, by halting the virtual CPU rather than spinning. Signed-off-by: Guo Ren Signed-off-by: Guo Ren Reviewed-by: Leonardo Bras --- arch/riscv/Kconfig | 12 ++++++++++++ arch/riscv/kernel/Makefile | 1 + 2 files changed, 13 insertions(+) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 4bcff2860f48..ec0da24ed6fb 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -813,6 +813,18 @@ config RELOCATABLE If unsure, say N. +config PARAVIRT_SPINLOCKS + bool "Paravirtualization layer for spinlocks" + depends on QUEUED_SPINLOCKS + default y + help + Paravirtualized spinlocks allow a unfair qspinlock to replace the + test-set kvm-guest virt spinlock implementation with something + virtualization-friendly, for example, halt the virtual CPU rather + than spinning. + + If you are unsure how to answer this question, answer Y. + endmenu # "Kernel features" menu "Boot options" diff --git a/arch/riscv/kernel/Makefile b/arch/riscv/kernel/Makefile index 671ad85f28f2..114b29234c46 100644 --- a/arch/riscv/kernel/Makefile +++ b/arch/riscv/kernel/Makefile @@ -103,3 +103,4 @@ obj-$(CONFIG_ARCH_RV64ILP32) += compat_signal.o obj-$(CONFIG_64BIT) += pi/ obj-$(CONFIG_ACPI) += acpi.o +obj-$(CONFIG_PARAVIRT_SPINLOCKS) += qspinlock_paravirt.o From patchwork Sun Sep 10 08:29:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guo Ren X-Patchwork-Id: 13378482 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8BDD3EE14D8 for ; Sun, 10 Sep 2023 08:32:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346556AbjIJIcR (ORCPT ); Sun, 10 Sep 2023 04:32:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58030 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344604AbjIJIcQ (ORCPT ); Sun, 10 Sep 2023 04:32:16 -0400 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EE2BD1997; Sun, 10 Sep 2023 01:31:39 -0700 (PDT) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D082BC43391; Sun, 10 Sep 2023 08:31:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1694334698; bh=qrr1aiptYWkzV6mJ+JveBdzevGp6E6VWqi93W6ZxXiE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ockEB3FWo6VNlEOVi+53v6nY/NVlJOFru658yPlAxx+CNOPYb9KAT026JAnM6ie1m 5y2LgD5kyFuLiAaY2dx5CZO10n7khioAb3Af0robFRff/5EO/R9JAyI2ff4Zsv4QoO aKNEhC+Ko27a5S6UxJ6RbPX4/fbqmQyiBSxaQ9FnzHVa0BGVSt0rMlpwlAACTgiSyX FRlhTkWXNKPjQMA2gEV4gyCDxL0R/1S5qHVH+zS7JsIAOpfPRpoYLn/GbjL3mopBXN Nqc7EOcOMnTIwZdqBAdfHwD96FYAjvCpFnFsnBOzHYRpud9o04q725m0IxTcy3fTRi V+fkfXXqucIZA== From: guoren@kernel.org To: paul.walmsley@sifive.com, anup@brainfault.org, peterz@infradead.org, mingo@redhat.com, will@kernel.org, palmer@rivosinc.com, longman@redhat.com, boqun.feng@gmail.com, tglx@linutronix.de, paulmck@kernel.org, rostedt@goodmis.org, rdunlap@infradead.org, catalin.marinas@arm.com, conor.dooley@microchip.com, xiaoguang.xing@sophgo.com, bjorn@rivosinc.com, alexghiti@rivosinc.com, keescook@chromium.org, greentime.hu@sifive.com, ajones@ventanamicro.com, jszhang@kernel.org, wefu@redhat.com, wuwei2016@iscas.ac.cn, leobras@redhat.com Cc: linux-arch@vger.kernel.org, linux-riscv@lists.infradead.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-csky@vger.kernel.org, Guo Ren , Guo Ren Subject: [PATCH V11 15/17] RISC-V: paravirt: pvqspinlock: Add trace point for pv_kick/wait Date: Sun, 10 Sep 2023 04:29:09 -0400 Message-Id: <20230910082911.3378782-16-guoren@kernel.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20230910082911.3378782-1-guoren@kernel.org> References: <20230910082911.3378782-1-guoren@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Guo Ren Add trace point for pv_kick/wait, here is the output: entries-in-buffer/entries-written: 33927/33927 #P:12 _-----=> irqs-off/BH-disabled / _----=> need-resched | / _---=> hardirq/softirq || / _--=> preempt-depth ||| / _-=> migrate-disable |||| / delay TASK-PID CPU# ||||| TIMESTAMP FUNCTION | | | ||||| | | sh-100 [001] d..2. 28.312294: pv_wait: cpu 1 out of wfi -0 [000] d.h4. 28.322030: pv_kick: cpu 0 kick target cpu 1 sh-100 [001] d..2. 30.982631: pv_wait: cpu 1 out of wfi -0 [000] d.h4. 30.993289: pv_kick: cpu 0 kick target cpu 1 sh-100 [002] d..2. 44.987573: pv_wait: cpu 2 out of wfi -0 [000] d.h4. 44.989000: pv_kick: cpu 0 kick target cpu 2 -0 [003] d.s3. 51.593978: pv_kick: cpu 3 kick target cpu 4 rcu_sched-15 [004] d..2. 51.595192: pv_wait: cpu 4 out of wfi lock_torture_wr-115 [004] ...2. 52.656482: pv_kick: cpu 4 kick target cpu 2 lock_torture_wr-113 [002] d..2. 52.659146: pv_wait: cpu 2 out of wfi lock_torture_wr-114 [008] d..2. 52.659507: pv_wait: cpu 8 out of wfi lock_torture_wr-114 [008] d..2. 52.663503: pv_wait: cpu 8 out of wfi lock_torture_wr-113 [002] ...2. 52.666128: pv_kick: cpu 2 kick target cpu 8 lock_torture_wr-114 [008] d..2. 52.667261: pv_wait: cpu 8 out of wfi lock_torture_wr-114 [009] .n.2. 53.141515: pv_kick: cpu 9 kick target cpu 11 lock_torture_wr-113 [002] d..2. 53.143339: pv_wait: cpu 2 out of wfi lock_torture_wr-116 [007] d..2. 53.143412: pv_wait: cpu 7 out of wfi lock_torture_wr-118 [000] d..2. 53.143457: pv_wait: cpu 0 out of wfi lock_torture_wr-115 [008] d..2. 53.143481: pv_wait: cpu 8 out of wfi lock_torture_wr-117 [011] d..2. 53.143522: pv_wait: cpu 11 out of wfi lock_torture_wr-117 [011] ...2. 53.143987: pv_kick: cpu 11 kick target cpu 8 lock_torture_wr-115 [008] ...2. 53.144269: pv_kick: cpu 8 kick target cpu 7 Signed-off-by: Guo Ren Signed-off-by: Guo Ren Reviewed-by: Leonardo Bras --- arch/riscv/kernel/qspinlock_paravirt.c | 8 +++ .../kernel/trace_events_filter_paravirt.h | 60 +++++++++++++++++++ 2 files changed, 68 insertions(+) create mode 100644 arch/riscv/kernel/trace_events_filter_paravirt.h diff --git a/arch/riscv/kernel/qspinlock_paravirt.c b/arch/riscv/kernel/qspinlock_paravirt.c index 571626f350be..5d298e989b99 100644 --- a/arch/riscv/kernel/qspinlock_paravirt.c +++ b/arch/riscv/kernel/qspinlock_paravirt.c @@ -9,10 +9,16 @@ #include #include +#define CREATE_TRACE_POINTS +#include "trace_events_filter_paravirt.h" + void pv_kick(int cpu) { sbi_ecall(SBI_EXT_PVLOCK, SBI_EXT_PVLOCK_KICK_CPU, cpuid_to_hartid_map(cpu), 0, 0, 0, 0, 0); + + trace_pv_kick(smp_processor_id(), cpu); + return; } @@ -28,6 +34,8 @@ void pv_wait(u8 *ptr, u8 val) goto out; wait_for_interrupt(); + + trace_pv_wait(smp_processor_id()); out: local_irq_restore(flags); } diff --git a/arch/riscv/kernel/trace_events_filter_paravirt.h b/arch/riscv/kernel/trace_events_filter_paravirt.h new file mode 100644 index 000000000000..9ff5aa451b12 --- /dev/null +++ b/arch/riscv/kernel/trace_events_filter_paravirt.h @@ -0,0 +1,60 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c), 2023 Alibaba Cloud + * Authors: + * Guo Ren + */ +#undef TRACE_SYSTEM +#define TRACE_SYSTEM paravirt + +#if !defined(_TRACE_PARAVIRT_H) || defined(TRACE_HEADER_MULTI_READ) +#define _TRACE_PARAVIRT_H + +#include + +TRACE_EVENT(pv_kick, + TP_PROTO(int cpu, int target), + TP_ARGS(cpu, target), + + TP_STRUCT__entry( + __field(int, cpu) + __field(int, target) + ), + + TP_fast_assign( + __entry->cpu = cpu; + __entry->target = target; + ), + + TP_printk("cpu %d kick target cpu %d", + __entry->cpu, + __entry->target + ) +); + +TRACE_EVENT(pv_wait, + TP_PROTO(int cpu), + TP_ARGS(cpu), + + TP_STRUCT__entry( + __field(int, cpu) + ), + + TP_fast_assign( + __entry->cpu = cpu; + ), + + TP_printk("cpu %d out of wfi", + __entry->cpu + ) +); + +#endif /* _TRACE_PARAVIRT_H || TRACE_HEADER_MULTI_READ */ + +#undef TRACE_INCLUDE_PATH +#undef TRACE_INCLUDE_FILE +#define TRACE_INCLUDE_PATH ../../../arch/riscv/kernel/ +#define TRACE_INCLUDE_FILE trace_events_filter_paravirt + +/* This part must be outside protection */ +#include From patchwork Sun Sep 10 08:29:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guo Ren X-Patchwork-Id: 13378483 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E4F46EE7FF4 for ; Sun, 10 Sep 2023 08:32:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346559AbjIJIcX (ORCPT ); Sun, 10 Sep 2023 04:32:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58844 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346562AbjIJIcW (ORCPT ); Sun, 10 Sep 2023 04:32:22 -0400 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5DBE7CCC; Sun, 10 Sep 2023 01:31:44 -0700 (PDT) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B9E9EC433CC; Sun, 10 Sep 2023 08:31:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1694334704; bh=tpG0qs5PAINQkIjiDJ8exNbKv+PcxlsE0HzNne/Tprg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=sN4CiTuLj+fKGOxNmVD/dq9nzK5ZoHMKqLYmlFAv7css2f2JamMteH2zIgF0kI2h9 LtrkfsHrJv2ttQ2tR7fZgNZfKHPdaXkuEYLDf0xHxwsKxf/Ij18aiXvrD/G6+4W8LL UESq0HFfC1VSX89J7WeTrY7V2KWW51N7KqM/cUXNQq0lM0EVqpmx4L5J6JmVeUz18F dnJcT4QKqMl4BXyq5wA0t2eafC2mMNWIS4jvnv0GlKG25iQcJ9QcQOWZ8N8QADPTj9 FKnOqJu2xzpgjWHoT+W2IjoPTh2/w7nWS26LAEl94FA8EYuKG9AxTcxJTygsgWJckW nnTsHXsZDjPBA== From: guoren@kernel.org To: paul.walmsley@sifive.com, anup@brainfault.org, peterz@infradead.org, mingo@redhat.com, will@kernel.org, palmer@rivosinc.com, longman@redhat.com, boqun.feng@gmail.com, tglx@linutronix.de, paulmck@kernel.org, rostedt@goodmis.org, rdunlap@infradead.org, catalin.marinas@arm.com, conor.dooley@microchip.com, xiaoguang.xing@sophgo.com, bjorn@rivosinc.com, alexghiti@rivosinc.com, keescook@chromium.org, greentime.hu@sifive.com, ajones@ventanamicro.com, jszhang@kernel.org, wefu@redhat.com, wuwei2016@iscas.ac.cn, leobras@redhat.com Cc: linux-arch@vger.kernel.org, linux-riscv@lists.infradead.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-csky@vger.kernel.org, Guo Ren , Guo Ren Subject: [PATCH V11 16/17] RISC-V: paravirt: pvqspinlock: KVM: Add paravirt qspinlock skeleton Date: Sun, 10 Sep 2023 04:29:10 -0400 Message-Id: <20230910082911.3378782-17-guoren@kernel.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20230910082911.3378782-1-guoren@kernel.org> References: <20230910082911.3378782-1-guoren@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Guo Ren Add the files functions needed to support the SBI PVLOCK (paravirt qspinlock kick_cpu) extension. This is a preparation for the next core implementation of kick_cpu. Signed-off-by: Guo Ren Signed-off-by: Guo Ren --- arch/riscv/include/asm/kvm_vcpu_sbi.h | 1 + arch/riscv/include/uapi/asm/kvm.h | 1 + arch/riscv/kvm/Makefile | 1 + arch/riscv/kvm/vcpu_sbi.c | 4 +++ arch/riscv/kvm/vcpu_sbi_pvlock.c | 38 +++++++++++++++++++++++++++ 5 files changed, 45 insertions(+) create mode 100644 arch/riscv/kvm/vcpu_sbi_pvlock.c diff --git a/arch/riscv/include/asm/kvm_vcpu_sbi.h b/arch/riscv/include/asm/kvm_vcpu_sbi.h index cdcf0ff07be7..7b4d60b54d7e 100644 --- a/arch/riscv/include/asm/kvm_vcpu_sbi.h +++ b/arch/riscv/include/asm/kvm_vcpu_sbi.h @@ -71,6 +71,7 @@ extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_srst; extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_hsm; extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_experimental; extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_vendor; +extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_pvlock; #ifdef CONFIG_RISCV_PMU_SBI extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_pmu; diff --git a/arch/riscv/include/uapi/asm/kvm.h b/arch/riscv/include/uapi/asm/kvm.h index 992c5e407104..d005c229f2da 100644 --- a/arch/riscv/include/uapi/asm/kvm.h +++ b/arch/riscv/include/uapi/asm/kvm.h @@ -148,6 +148,7 @@ enum KVM_RISCV_SBI_EXT_ID { KVM_RISCV_SBI_EXT_PMU, KVM_RISCV_SBI_EXT_EXPERIMENTAL, KVM_RISCV_SBI_EXT_VENDOR, + KVM_RISCV_SBI_EXT_PVLOCK, KVM_RISCV_SBI_EXT_MAX, }; diff --git a/arch/riscv/kvm/Makefile b/arch/riscv/kvm/Makefile index 4c2067fc59fc..6112750a3a0c 100644 --- a/arch/riscv/kvm/Makefile +++ b/arch/riscv/kvm/Makefile @@ -26,6 +26,7 @@ kvm-$(CONFIG_RISCV_SBI_V01) += vcpu_sbi_v01.o kvm-y += vcpu_sbi_base.o kvm-y += vcpu_sbi_replace.o kvm-y += vcpu_sbi_hsm.o +kvm-y += vcpu_sbi_pvlock.o kvm-y += vcpu_timer.o kvm-$(CONFIG_RISCV_PMU_SBI) += vcpu_pmu.o vcpu_sbi_pmu.o kvm-y += aia.o diff --git a/arch/riscv/kvm/vcpu_sbi.c b/arch/riscv/kvm/vcpu_sbi.c index 9cd97091c723..c03c3d489b2b 100644 --- a/arch/riscv/kvm/vcpu_sbi.c +++ b/arch/riscv/kvm/vcpu_sbi.c @@ -74,6 +74,10 @@ static const struct kvm_riscv_sbi_extension_entry sbi_ext[] = { .ext_idx = KVM_RISCV_SBI_EXT_VENDOR, .ext_ptr = &vcpu_sbi_ext_vendor, }, + { + .ext_idx = KVM_RISCV_SBI_EXT_PVLOCK, + .ext_ptr = &vcpu_sbi_ext_pvlock, + }, }; void kvm_riscv_vcpu_sbi_forward(struct kvm_vcpu *vcpu, struct kvm_run *run) diff --git a/arch/riscv/kvm/vcpu_sbi_pvlock.c b/arch/riscv/kvm/vcpu_sbi_pvlock.c new file mode 100644 index 000000000000..544a456c5041 --- /dev/null +++ b/arch/riscv/kvm/vcpu_sbi_pvlock.c @@ -0,0 +1,38 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (c), 2023 Alibaba Cloud + * + * Authors: + * Guo Ren + */ + +#include +#include +#include +#include +#include + +static int kvm_sbi_ext_pvlock_handler(struct kvm_vcpu *vcpu, struct kvm_run *run, + struct kvm_vcpu_sbi_return *retdata) +{ + int ret = 0; + struct kvm_cpu_context *cp = &vcpu->arch.guest_context; + unsigned long funcid = cp->a6; + + switch (funcid) { + case SBI_EXT_PVLOCK_KICK_CPU: + break; + default: + ret = SBI_ERR_NOT_SUPPORTED; + } + + retdata->err_val = ret; + + return 0; +} + +const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_pvlock = { + .extid_start = SBI_EXT_PVLOCK, + .extid_end = SBI_EXT_PVLOCK, + .handler = kvm_sbi_ext_pvlock_handler, +}; From patchwork Sun Sep 10 08:29:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guo Ren X-Patchwork-Id: 13378484 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4EF13EEB57E for ; Sun, 10 Sep 2023 08:32:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244902AbjIJIcg (ORCPT ); Sun, 10 Sep 2023 04:32:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48932 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241570AbjIJIcf (ORCPT ); Sun, 10 Sep 2023 04:32:35 -0400 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0191210F6; Sun, 10 Sep 2023 01:32:04 -0700 (PDT) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A4096C43397; Sun, 10 Sep 2023 08:31:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1694334710; bh=ZxSSeBb/S16STGNIjLgLnRd4gdTK7GWBFlxeOcsXSwE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=r+LRgGMW/xx8261kzbec/7ectryiunjI7NOKoSUJZMQ5GO9xW2FzxB+MpQTjPJwrP GqZmGTDK/Cn5Uldm9Mx54IZVN/eDq1LT+Hrd6ayDASK2ZXJSqpekvxQPcGwsuaX5J2 U41UBqPSUW/oUH3AuoETyRMVkb9twscE6eYAz1hJSWU9FodNY8BV2gv7nBtWI0BMYk gDPHGGttiPk/+RxqXz0RzsFAlZOZxTaiTRLevDEXWEaO/LtMjz+hekQH0J1QF/dJuq CVBIe+GUx3doKxK4ICkFhf1vXR716rIvyf8B97hkgFqJ6R+xJQkXq0oOtqrgScC3qw ALUrMl2iBo61w== From: guoren@kernel.org To: paul.walmsley@sifive.com, anup@brainfault.org, peterz@infradead.org, mingo@redhat.com, will@kernel.org, palmer@rivosinc.com, longman@redhat.com, boqun.feng@gmail.com, tglx@linutronix.de, paulmck@kernel.org, rostedt@goodmis.org, rdunlap@infradead.org, catalin.marinas@arm.com, conor.dooley@microchip.com, xiaoguang.xing@sophgo.com, bjorn@rivosinc.com, alexghiti@rivosinc.com, keescook@chromium.org, greentime.hu@sifive.com, ajones@ventanamicro.com, jszhang@kernel.org, wefu@redhat.com, wuwei2016@iscas.ac.cn, leobras@redhat.com Cc: linux-arch@vger.kernel.org, linux-riscv@lists.infradead.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-csky@vger.kernel.org, Guo Ren , Guo Ren Subject: [PATCH V11 17/17] RISC-V: paravirt: pvqspinlock: KVM: Implement kvm_sbi_ext_pvlock_kick_cpu() Date: Sun, 10 Sep 2023 04:29:11 -0400 Message-Id: <20230910082911.3378782-18-guoren@kernel.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20230910082911.3378782-1-guoren@kernel.org> References: <20230910082911.3378782-1-guoren@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Guo Ren We only need to call the kvm_vcpu_kick() and bring target_vcpu from the halt state. No irq raised, no other request, just a pure vcpu_kick. Signed-off-by: Guo Ren Signed-off-by: Guo Ren Reviewed-by: Leonardo Bras --- arch/riscv/kvm/vcpu_sbi_pvlock.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/arch/riscv/kvm/vcpu_sbi_pvlock.c b/arch/riscv/kvm/vcpu_sbi_pvlock.c index 544a456c5041..914fc58aedfe 100644 --- a/arch/riscv/kvm/vcpu_sbi_pvlock.c +++ b/arch/riscv/kvm/vcpu_sbi_pvlock.c @@ -12,6 +12,24 @@ #include #include +static int kvm_sbi_ext_pvlock_kick_cpu(struct kvm_vcpu *vcpu) +{ + struct kvm_cpu_context *cp = &vcpu->arch.guest_context; + struct kvm *kvm = vcpu->kvm; + struct kvm_vcpu *target; + + target = kvm_get_vcpu_by_id(kvm, cp->a0); + if (!target) + return SBI_ERR_INVALID_PARAM; + + kvm_vcpu_kick(target); + + if (READ_ONCE(target->ready)) + kvm_vcpu_yield_to(target); + + return SBI_SUCCESS; +} + static int kvm_sbi_ext_pvlock_handler(struct kvm_vcpu *vcpu, struct kvm_run *run, struct kvm_vcpu_sbi_return *retdata) { @@ -21,6 +39,7 @@ static int kvm_sbi_ext_pvlock_handler(struct kvm_vcpu *vcpu, struct kvm_run *run switch (funcid) { case SBI_EXT_PVLOCK_KICK_CPU: + ret = kvm_sbi_ext_pvlock_kick_cpu(vcpu); break; default: ret = SBI_ERR_NOT_SUPPORTED;