From patchwork Wed Aug 2 16:46:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guo Ren X-Patchwork-Id: 13338526 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 80550C41513 for ; Wed, 2 Aug 2023 16:48:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231594AbjHBQsA (ORCPT ); Wed, 2 Aug 2023 12:48:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47088 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231319AbjHBQr6 (ORCPT ); Wed, 2 Aug 2023 12:47:58 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A9FF2213D; Wed, 2 Aug 2023 09:47:56 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 3EED061A3C; Wed, 2 Aug 2023 16:47:56 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 20BA5C433C8; Wed, 2 Aug 2023 16:47:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1690994875; bh=Z6iRHs/3ezKlGvEDJaadEcfbR2OWUTdx821B0UQrJ0E=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GkWdAEGd9TseNN0qp96cN4WkCQAhxA2DityKQmyfUHsB1s59LTPNugNKd6yQM6/Fv 6o8J42OZII+vTxFwUNfQYbiMAx9yjIzH5JSPYp++Oko4HVzKoSt1D6jHcIKwR1vDHh 600fBd/GGC7pcq/VEl5TYQ9Q2SpEWk9bSReER9eMX6qXwGdB8vZgeaHXY4urxEMDDI CPn/CKmkQWH+x/hiveGi0sZn6FIY6HPMWwZ1UOkUllnixCL9VWr6EjSPR4zYcQmPxC g408xaMyWGrv7PkXvX/970RVacdcg6ORRCijRYCoZEPduxmFxae2zPTYZE9dM/x/vb zjbIFPW02AsNA== From: guoren@kernel.org To: paul.walmsley@sifive.com, anup@brainfault.org, peterz@infradead.org, mingo@redhat.com, will@kernel.org, palmer@rivosinc.com, longman@redhat.com, boqun.feng@gmail.com, tglx@linutronix.de, paulmck@kernel.org, rostedt@goodmis.org, rdunlap@infradead.org, catalin.marinas@arm.com, conor.dooley@microchip.com, xiaoguang.xing@sophgo.com, bjorn@rivosinc.com, alexghiti@rivosinc.com, keescook@chromium.org, greentime.hu@sifive.com, ajones@ventanamicro.com, jszhang@kernel.org, wefu@redhat.com, wuwei2016@iscas.ac.cn Cc: linux-arch@vger.kernel.org, linux-riscv@lists.infradead.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-csky@vger.kernel.org, Guo Ren , Guo Ren Subject: [PATCH V10 01/19] asm-generic: ticket-lock: Reuse arch_spinlock_t of qspinlock Date: Wed, 2 Aug 2023 12:46:43 -0400 Message-Id: <20230802164701.192791-2-guoren@kernel.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20230802164701.192791-1-guoren@kernel.org> References: <20230802164701.192791-1-guoren@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Guo Ren The arch_spinlock_t of qspinlock has contained the atomic_t val, which satisfies the ticket-lock requirement. Thus, unify the arch_spinlock_t into qspinlock_types.h. This is the preparation for the next combo spinlock. Signed-off-by: Guo Ren Signed-off-by: Guo Ren --- include/asm-generic/spinlock.h | 14 +++++++------- include/asm-generic/spinlock_types.h | 12 ++---------- 2 files changed, 9 insertions(+), 17 deletions(-) diff --git a/include/asm-generic/spinlock.h b/include/asm-generic/spinlock.h index 90803a826ba0..4773334ee638 100644 --- a/include/asm-generic/spinlock.h +++ b/include/asm-generic/spinlock.h @@ -32,7 +32,7 @@ static __always_inline void arch_spin_lock(arch_spinlock_t *lock) { - u32 val = atomic_fetch_add(1<<16, lock); + u32 val = atomic_fetch_add(1<<16, &lock->val); u16 ticket = val >> 16; if (ticket == (u16)val) @@ -46,31 +46,31 @@ static __always_inline void arch_spin_lock(arch_spinlock_t *lock) * have no outstanding writes due to the atomic_fetch_add() the extra * orderings are free. */ - atomic_cond_read_acquire(lock, ticket == (u16)VAL); + atomic_cond_read_acquire(&lock->val, ticket == (u16)VAL); smp_mb(); } static __always_inline bool arch_spin_trylock(arch_spinlock_t *lock) { - u32 old = atomic_read(lock); + u32 old = atomic_read(&lock->val); if ((old >> 16) != (old & 0xffff)) return false; - return atomic_try_cmpxchg(lock, &old, old + (1<<16)); /* SC, for RCsc */ + return atomic_try_cmpxchg(&lock->val, &old, old + (1<<16)); /* SC, for RCsc */ } static __always_inline void arch_spin_unlock(arch_spinlock_t *lock) { u16 *ptr = (u16 *)lock + IS_ENABLED(CONFIG_CPU_BIG_ENDIAN); - u32 val = atomic_read(lock); + u32 val = atomic_read(&lock->val); smp_store_release(ptr, (u16)val + 1); } static __always_inline int arch_spin_value_unlocked(arch_spinlock_t lock) { - u32 val = lock.counter; + u32 val = lock.val.counter; return ((val >> 16) == (val & 0xffff)); } @@ -84,7 +84,7 @@ static __always_inline int arch_spin_is_locked(arch_spinlock_t *lock) static __always_inline int arch_spin_is_contended(arch_spinlock_t *lock) { - u32 val = atomic_read(lock); + u32 val = atomic_read(&lock->val); return (s16)((val >> 16) - (val & 0xffff)) > 1; } diff --git a/include/asm-generic/spinlock_types.h b/include/asm-generic/spinlock_types.h index 8962bb730945..f534aa5de394 100644 --- a/include/asm-generic/spinlock_types.h +++ b/include/asm-generic/spinlock_types.h @@ -3,15 +3,7 @@ #ifndef __ASM_GENERIC_SPINLOCK_TYPES_H #define __ASM_GENERIC_SPINLOCK_TYPES_H -#include -typedef atomic_t arch_spinlock_t; - -/* - * qrwlock_types depends on arch_spinlock_t, so we must typedef that before the - * include. - */ -#include - -#define __ARCH_SPIN_LOCK_UNLOCKED ATOMIC_INIT(0) +#include +#include #endif /* __ASM_GENERIC_SPINLOCK_TYPES_H */ From patchwork Wed Aug 2 16:46:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guo Ren X-Patchwork-Id: 13338527 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 95FE5C001E0 for ; Wed, 2 Aug 2023 16:48:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231442AbjHBQsS (ORCPT ); Wed, 2 Aug 2023 12:48:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47390 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231637AbjHBQsQ (ORCPT ); Wed, 2 Aug 2023 12:48:16 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 73F312706; Wed, 2 Aug 2023 09:48:13 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 1037861A0D; Wed, 2 Aug 2023 16:48:13 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B32D9C433C8; Wed, 2 Aug 2023 16:47:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1690994892; bh=sgIzdUVfUlz4ZBKJCMetJmqx5CuZ4IfTQ/ixE/F16i0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Wmu46JkGCDo6zaZLidoc9nt6bFZsvfYzwQYZie47Sodhkvu2MBtrSZn0nnQsDTShh xoRfMUwHEPsQkRiRZDm5yFSYGVC4KiAwfH8m9r3dBxxc8wIJD1cpKGUfYBp3rd09W3 QTRqfelv4Upu2MoQr8tIicZ1lvQ5BOXKquIvkL8uRju5ld0GyBHpd7o1J+k+vN6/IJ 7RUDNTiykWSNxwrb2eIJHUz3nfqgjYbXMUKoyQSUg128dZEyh51KjJ6CoJLXJaHuiO 9m7qoB8Yu0HFq2W7dWO6xhupFhznj9m0FCb2JLlQ78g/sS7d3SRjQ79g9PNgZK5JdV IW+6etgrHhAlg== From: guoren@kernel.org To: paul.walmsley@sifive.com, anup@brainfault.org, peterz@infradead.org, mingo@redhat.com, will@kernel.org, palmer@rivosinc.com, longman@redhat.com, boqun.feng@gmail.com, tglx@linutronix.de, paulmck@kernel.org, rostedt@goodmis.org, rdunlap@infradead.org, catalin.marinas@arm.com, conor.dooley@microchip.com, xiaoguang.xing@sophgo.com, bjorn@rivosinc.com, alexghiti@rivosinc.com, keescook@chromium.org, greentime.hu@sifive.com, ajones@ventanamicro.com, jszhang@kernel.org, wefu@redhat.com, wuwei2016@iscas.ac.cn Cc: linux-arch@vger.kernel.org, linux-riscv@lists.infradead.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-csky@vger.kernel.org, Guo Ren , Guo Ren Subject: [PATCH V10 02/19] asm-generic: ticket-lock: Move into ticket_spinlock.h Date: Wed, 2 Aug 2023 12:46:44 -0400 Message-Id: <20230802164701.192791-3-guoren@kernel.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20230802164701.192791-1-guoren@kernel.org> References: <20230802164701.192791-1-guoren@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Guo Ren Move ticket-lock definition into an independent file. This is the preparation for the next combo spinlock of riscv. Signed-off-by: Guo Ren Signed-off-by: Guo Ren --- include/asm-generic/spinlock.h | 87 +--------------------- include/asm-generic/ticket_spinlock.h | 103 ++++++++++++++++++++++++++ 2 files changed, 104 insertions(+), 86 deletions(-) create mode 100644 include/asm-generic/ticket_spinlock.h diff --git a/include/asm-generic/spinlock.h b/include/asm-generic/spinlock.h index 4773334ee638..970590baf61b 100644 --- a/include/asm-generic/spinlock.h +++ b/include/asm-generic/spinlock.h @@ -1,94 +1,9 @@ /* SPDX-License-Identifier: GPL-2.0 */ -/* - * 'Generic' ticket-lock implementation. - * - * It relies on atomic_fetch_add() having well defined forward progress - * guarantees under contention. If your architecture cannot provide this, stick - * to a test-and-set lock. - * - * It also relies on atomic_fetch_add() being safe vs smp_store_release() on a - * sub-word of the value. This is generally true for anything LL/SC although - * you'd be hard pressed to find anything useful in architecture specifications - * about this. If your architecture cannot do this you might be better off with - * a test-and-set. - * - * It further assumes atomic_*_release() + atomic_*_acquire() is RCpc and hence - * uses atomic_fetch_add() which is RCsc to create an RCsc hot path, along with - * a full fence after the spin to upgrade the otherwise-RCpc - * atomic_cond_read_acquire(). - * - * The implementation uses smp_cond_load_acquire() to spin, so if the - * architecture has WFE like instructions to sleep instead of poll for word - * modifications be sure to implement that (see ARM64 for example). - * - */ - #ifndef __ASM_GENERIC_SPINLOCK_H #define __ASM_GENERIC_SPINLOCK_H -#include -#include - -static __always_inline void arch_spin_lock(arch_spinlock_t *lock) -{ - u32 val = atomic_fetch_add(1<<16, &lock->val); - u16 ticket = val >> 16; - - if (ticket == (u16)val) - return; - - /* - * atomic_cond_read_acquire() is RCpc, but rather than defining a - * custom cond_read_rcsc() here we just emit a full fence. We only - * need the prior reads before subsequent writes ordering from - * smb_mb(), but as atomic_cond_read_acquire() just emits reads and we - * have no outstanding writes due to the atomic_fetch_add() the extra - * orderings are free. - */ - atomic_cond_read_acquire(&lock->val, ticket == (u16)VAL); - smp_mb(); -} - -static __always_inline bool arch_spin_trylock(arch_spinlock_t *lock) -{ - u32 old = atomic_read(&lock->val); - - if ((old >> 16) != (old & 0xffff)) - return false; - - return atomic_try_cmpxchg(&lock->val, &old, old + (1<<16)); /* SC, for RCsc */ -} - -static __always_inline void arch_spin_unlock(arch_spinlock_t *lock) -{ - u16 *ptr = (u16 *)lock + IS_ENABLED(CONFIG_CPU_BIG_ENDIAN); - u32 val = atomic_read(&lock->val); - - smp_store_release(ptr, (u16)val + 1); -} - -static __always_inline int arch_spin_value_unlocked(arch_spinlock_t lock) -{ - u32 val = lock.val.counter; - - return ((val >> 16) == (val & 0xffff)); -} - -static __always_inline int arch_spin_is_locked(arch_spinlock_t *lock) -{ - arch_spinlock_t val = READ_ONCE(*lock); - - return !arch_spin_value_unlocked(val); -} - -static __always_inline int arch_spin_is_contended(arch_spinlock_t *lock) -{ - u32 val = atomic_read(&lock->val); - - return (s16)((val >> 16) - (val & 0xffff)) > 1; -} - +#include #include #endif /* __ASM_GENERIC_SPINLOCK_H */ diff --git a/include/asm-generic/ticket_spinlock.h b/include/asm-generic/ticket_spinlock.h new file mode 100644 index 000000000000..cfcff22b37b3 --- /dev/null +++ b/include/asm-generic/ticket_spinlock.h @@ -0,0 +1,103 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +/* + * 'Generic' ticket-lock implementation. + * + * It relies on atomic_fetch_add() having well defined forward progress + * guarantees under contention. If your architecture cannot provide this, stick + * to a test-and-set lock. + * + * It also relies on atomic_fetch_add() being safe vs smp_store_release() on a + * sub-word of the value. This is generally true for anything LL/SC although + * you'd be hard pressed to find anything useful in architecture specifications + * about this. If your architecture cannot do this you might be better off with + * a test-and-set. + * + * It further assumes atomic_*_release() + atomic_*_acquire() is RCpc and hence + * uses atomic_fetch_add() which is RCsc to create an RCsc hot path, along with + * a full fence after the spin to upgrade the otherwise-RCpc + * atomic_cond_read_acquire(). + * + * The implementation uses smp_cond_load_acquire() to spin, so if the + * architecture has WFE like instructions to sleep instead of poll for word + * modifications be sure to implement that (see ARM64 for example). + * + */ + +#ifndef __ASM_GENERIC_TICKET_SPINLOCK_H +#define __ASM_GENERIC_TICKET_SPINLOCK_H + +#include +#include + +static __always_inline void ticket_spin_lock(arch_spinlock_t *lock) +{ + u32 val = atomic_fetch_add(1<<16, &lock->val); + u16 ticket = val >> 16; + + if (ticket == (u16)val) + return; + + /* + * atomic_cond_read_acquire() is RCpc, but rather than defining a + * custom cond_read_rcsc() here we just emit a full fence. We only + * need the prior reads before subsequent writes ordering from + * smb_mb(), but as atomic_cond_read_acquire() just emits reads and we + * have no outstanding writes due to the atomic_fetch_add() the extra + * orderings are free. + */ + atomic_cond_read_acquire(&lock->val, ticket == (u16)VAL); + smp_mb(); +} + +static __always_inline bool ticket_spin_trylock(arch_spinlock_t *lock) +{ + u32 old = atomic_read(&lock->val); + + if ((old >> 16) != (old & 0xffff)) + return false; + + return atomic_try_cmpxchg(&lock->val, &old, old + (1<<16)); /* SC, for RCsc */ +} + +static __always_inline void ticket_spin_unlock(arch_spinlock_t *lock) +{ + u16 *ptr = (u16 *)lock + IS_ENABLED(CONFIG_CPU_BIG_ENDIAN); + u32 val = atomic_read(&lock->val); + + smp_store_release(ptr, (u16)val + 1); +} + +static __always_inline int ticket_spin_value_unlocked(arch_spinlock_t lock) +{ + u32 val = lock.val.counter; + + return ((val >> 16) == (val & 0xffff)); +} + +static __always_inline int ticket_spin_is_locked(arch_spinlock_t *lock) +{ + arch_spinlock_t val = READ_ONCE(*lock); + + return !ticket_spin_value_unlocked(val); +} + +static __always_inline int ticket_spin_is_contended(arch_spinlock_t *lock) +{ + u32 val = atomic_read(&lock->val); + + return (s16)((val >> 16) - (val & 0xffff)) > 1; +} + +/* + * Remapping spinlock architecture specific functions to the corresponding + * ticket spinlock functions. + */ +#define arch_spin_is_locked(l) ticket_spin_is_locked(l) +#define arch_spin_is_contended(l) ticket_spin_is_contended(l) +#define arch_spin_value_unlocked(l) ticket_spin_value_unlocked(l) +#define arch_spin_lock(l) ticket_spin_lock(l) +#define arch_spin_trylock(l) ticket_spin_trylock(l) +#define arch_spin_unlock(l) ticket_spin_unlock(l) + +#endif /* __ASM_GENERIC_TICKET_SPINLOCK_H */ From patchwork Wed Aug 2 16:46:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guo Ren X-Patchwork-Id: 13338528 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 441D6C04FE0 for ; Wed, 2 Aug 2023 16:49:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231656AbjHBQs5 (ORCPT ); Wed, 2 Aug 2023 12:48:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47394 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232272AbjHBQsb (ORCPT ); Wed, 2 Aug 2023 12:48:31 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 027EBE4D; Wed, 2 Aug 2023 09:48:30 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 9356D61A3C; Wed, 2 Aug 2023 16:48:29 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8B243C433CA; Wed, 2 Aug 2023 16:48:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1690994909; bh=Au/Lyz8DXodsj2BximBqrlWp9CS+kuzK/WJVuS2wC2I=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=als1lRCOJxdh/U1/AvTkFei2EbgRZxakwU0D9uVRfRIomZL/fq8CEWW3moTZe54Es WzmElOVBSjsv3lXViFfoKxolbN2RO8feD/ampz9MdfxWLuKS2nn4UqFjg9F0UmpgmD mhXOUQxD3MgpD9WqvU+ob/LHeeQtO2IieOhOsdeQ6N2dzJEZ1GYF0iiZeMafPn3TDM ra0UTrgk8/Of75IqyQUCzWAtCiDyAxFsFEEzA3sWx6Ft3AvbJfsfcibhgPb+AsQu22 sptAKkxGEn90iHZUf6HJkucTb2r5aJi3d8cIAsPueG123ytbFSthz+8F7+fIYyJ76l qMWL+iex6NlJw== From: guoren@kernel.org To: paul.walmsley@sifive.com, anup@brainfault.org, peterz@infradead.org, mingo@redhat.com, will@kernel.org, palmer@rivosinc.com, longman@redhat.com, boqun.feng@gmail.com, tglx@linutronix.de, paulmck@kernel.org, rostedt@goodmis.org, rdunlap@infradead.org, catalin.marinas@arm.com, conor.dooley@microchip.com, xiaoguang.xing@sophgo.com, bjorn@rivosinc.com, alexghiti@rivosinc.com, keescook@chromium.org, greentime.hu@sifive.com, ajones@ventanamicro.com, jszhang@kernel.org, wefu@redhat.com, wuwei2016@iscas.ac.cn Cc: linux-arch@vger.kernel.org, linux-riscv@lists.infradead.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-csky@vger.kernel.org, Guo Ren , Guo Ren Subject: [PATCH V10 03/19] riscv: qspinlock: errata: Add ERRATA_THEAD_WRITE_ONCE fixup Date: Wed, 2 Aug 2023 12:46:45 -0400 Message-Id: <20230802164701.192791-4-guoren@kernel.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20230802164701.192791-1-guoren@kernel.org> References: <20230802164701.192791-1-guoren@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Guo Ren The early version of T-Head C9xx cores has a store merge buffer delay problem. The store merge buffer could improve the store queue performance by merging multi-store requests, but when there are not continued store requests, the prior single store request would be waiting in the store queue for a long time. That would cause significant problems for communication between multi-cores. This problem was found on sg2042 & th1520 platforms with the qspinlock lock torture test. So appending a fence w.o could immediately flush the store merge buffer and let other cores see the write result. This will apply the WRITE_ONCE errata to handle the non-standard behavior via appending a fence w.o instruction for WRITE_ONCE(). Signed-off-by: Guo Ren Signed-off-by: Guo Ren --- arch/riscv/Kconfig.errata | 19 +++++++++++++++++++ arch/riscv/errata/thead/errata.c | 20 ++++++++++++++++++++ arch/riscv/include/asm/errata_list.h | 13 ------------- arch/riscv/include/asm/rwonce.h | 24 ++++++++++++++++++++++++ arch/riscv/include/asm/vendorid_list.h | 14 ++++++++++++++ include/asm-generic/rwonce.h | 2 ++ 6 files changed, 79 insertions(+), 13 deletions(-) create mode 100644 arch/riscv/include/asm/rwonce.h diff --git a/arch/riscv/Kconfig.errata b/arch/riscv/Kconfig.errata index 0c8f4652cd82..4745a5c57e7c 100644 --- a/arch/riscv/Kconfig.errata +++ b/arch/riscv/Kconfig.errata @@ -77,4 +77,23 @@ config ERRATA_THEAD_PMU If you don't know what to do here, say "Y". +config ERRATA_THEAD_WRITE_ONCE + bool "Apply T-Head WRITE_ONCE errata" + depends on ERRATA_THEAD + default y + help + The early version of T-Head C9xx cores has a store merge buffer + delay problem. The store merge buffer could improve the store queue + performance by merging multi-store requests, but when there are no + continued store requests, the prior single store request would be + waiting in the store queue for a long time. That would cause + significant problems for communication between multi-cores. Appending + a fence w.o could immediately flush the store merge buffer and let + other cores see the write result. + + This will apply the WRITE_ONCE errata to handle the non-standard + behavior via appending a fence w.o instruction for WRITE_ONCE(). + + If you don't know what to do here, say "Y". + endmenu # "CPU errata selection" diff --git a/arch/riscv/errata/thead/errata.c b/arch/riscv/errata/thead/errata.c index be84b14f0118..881729746d2e 100644 --- a/arch/riscv/errata/thead/errata.c +++ b/arch/riscv/errata/thead/errata.c @@ -69,6 +69,23 @@ static bool errata_probe_pmu(unsigned int stage, return true; } +static bool errata_probe_write_once(unsigned int stage, + unsigned long arch_id, unsigned long impid) +{ + if (!IS_ENABLED(CONFIG_ERRATA_THEAD_WRITE_ONCE)) + return false; + + /* target-c9xx cores report arch_id and impid as 0 */ + if (arch_id != 0 || impid != 0) + return false; + + if (stage == RISCV_ALTERNATIVES_EARLY_BOOT || + stage == RISCV_ALTERNATIVES_MODULE) + return true; + + return false; +} + static u32 thead_errata_probe(unsigned int stage, unsigned long archid, unsigned long impid) { @@ -83,6 +100,9 @@ static u32 thead_errata_probe(unsigned int stage, if (errata_probe_pmu(stage, archid, impid)) cpu_req_errata |= BIT(ERRATA_THEAD_PMU); + if (errata_probe_write_once(stage, archid, impid)) + cpu_req_errata |= BIT(ERRATA_THEAD_WRITE_ONCE); + return cpu_req_errata; } diff --git a/arch/riscv/include/asm/errata_list.h b/arch/riscv/include/asm/errata_list.h index 712cab7adffe..fbb2b8d39321 100644 --- a/arch/riscv/include/asm/errata_list.h +++ b/arch/riscv/include/asm/errata_list.h @@ -11,19 +11,6 @@ #include #include -#ifdef CONFIG_ERRATA_SIFIVE -#define ERRATA_SIFIVE_CIP_453 0 -#define ERRATA_SIFIVE_CIP_1200 1 -#define ERRATA_SIFIVE_NUMBER 2 -#endif - -#ifdef CONFIG_ERRATA_THEAD -#define ERRATA_THEAD_PBMT 0 -#define ERRATA_THEAD_CMO 1 -#define ERRATA_THEAD_PMU 2 -#define ERRATA_THEAD_NUMBER 3 -#endif - #ifdef __ASSEMBLY__ #define ALT_INSN_FAULT(x) \ diff --git a/arch/riscv/include/asm/rwonce.h b/arch/riscv/include/asm/rwonce.h new file mode 100644 index 000000000000..be0b8864969d --- /dev/null +++ b/arch/riscv/include/asm/rwonce.h @@ -0,0 +1,24 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef __ASM_RWONCE_H +#define __ASM_RWONCE_H + +#include +#include +#include + +#define __WRITE_ONCE(x, val) \ +do { \ + *(volatile typeof(x) *)&(x) = (val); \ + asm volatile(ALTERNATIVE( \ + __nops(1), \ + "fence w, o\n\t", \ + THEAD_VENDOR_ID, \ + ERRATA_THEAD_WRITE_ONCE, \ + CONFIG_ERRATA_THEAD_WRITE_ONCE) \ + : : : "memory"); \ +} while (0) + +#include + +#endif /* __ASM_RWONCE_H */ diff --git a/arch/riscv/include/asm/vendorid_list.h b/arch/riscv/include/asm/vendorid_list.h index cb89af3f0704..73078cfe4029 100644 --- a/arch/riscv/include/asm/vendorid_list.h +++ b/arch/riscv/include/asm/vendorid_list.h @@ -8,4 +8,18 @@ #define SIFIVE_VENDOR_ID 0x489 #define THEAD_VENDOR_ID 0x5b7 +#ifdef CONFIG_ERRATA_SIFIVE +#define ERRATA_SIFIVE_CIP_453 0 +#define ERRATA_SIFIVE_CIP_1200 1 +#define ERRATA_SIFIVE_NUMBER 2 +#endif + +#ifdef CONFIG_ERRATA_THEAD +#define ERRATA_THEAD_PBMT 0 +#define ERRATA_THEAD_CMO 1 +#define ERRATA_THEAD_PMU 2 +#define ERRATA_THEAD_WRITE_ONCE 3 +#define ERRATA_THEAD_NUMBER 4 +#endif + #endif diff --git a/include/asm-generic/rwonce.h b/include/asm-generic/rwonce.h index 8d0a6280e982..fb07fe8c6e45 100644 --- a/include/asm-generic/rwonce.h +++ b/include/asm-generic/rwonce.h @@ -50,10 +50,12 @@ __READ_ONCE(x); \ }) +#ifndef __WRITE_ONCE #define __WRITE_ONCE(x, val) \ do { \ *(volatile typeof(x) *)&(x) = (val); \ } while (0) +#endif #define WRITE_ONCE(x, val) \ do { \ From patchwork Wed Aug 2 16:46:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guo Ren X-Patchwork-Id: 13338529 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3A66C41513 for ; Wed, 2 Aug 2023 16:49:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231669AbjHBQtZ (ORCPT ); Wed, 2 Aug 2023 12:49:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48072 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232271AbjHBQtF (ORCPT ); Wed, 2 Aug 2023 12:49:05 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 07C912D78; Wed, 2 Aug 2023 09:48:46 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 9A51C61A3C; Wed, 2 Aug 2023 16:48:45 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 07D8BC433CA; Wed, 2 Aug 2023 16:48:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1690994925; bh=XpJMmQpizgl+1q4faLgZyV4Sr5SWoDQzo87jc2k0hEU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Rgxujrsdn7f+bcZFJbQ7fNjqz5y2CrBgF0JiPl0DEbMPu20EiePiHkH95Lmot58DW LviM3KysginZdajdwkNuvDoxVNN9RrLMOjPSuZ2jRVFlm0Qny2n4KMliWznJBLtDdM PysXqLOrB6cwwXJ7jKWDbYVbH0jnEiJgiskSOUOEj3Oeoj/ZstrC+OQYehdAcghRJK G7L0DezO+mrmdTlz+SbrU0xEuUAXSYjh3wFuhnVhocrtahkE+ip2vFt+KKjTcY8dze FrXIL/40ohIWf2ObIM0DmN2LJ5i1atvvrNMdz0WAJ+g3S86+2/6kXwSMt708WfOkjC iMkIyJYOTzcAA== From: guoren@kernel.org To: paul.walmsley@sifive.com, anup@brainfault.org, peterz@infradead.org, mingo@redhat.com, will@kernel.org, palmer@rivosinc.com, longman@redhat.com, boqun.feng@gmail.com, tglx@linutronix.de, paulmck@kernel.org, rostedt@goodmis.org, rdunlap@infradead.org, catalin.marinas@arm.com, conor.dooley@microchip.com, xiaoguang.xing@sophgo.com, bjorn@rivosinc.com, alexghiti@rivosinc.com, keescook@chromium.org, greentime.hu@sifive.com, ajones@ventanamicro.com, jszhang@kernel.org, wefu@redhat.com, wuwei2016@iscas.ac.cn Cc: linux-arch@vger.kernel.org, linux-riscv@lists.infradead.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-csky@vger.kernel.org, Guo Ren , Guo Ren Subject: [PATCH V10 04/19] riscv: qspinlock: Add basic queued_spinlock support Date: Wed, 2 Aug 2023 12:46:46 -0400 Message-Id: <20230802164701.192791-5-guoren@kernel.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20230802164701.192791-1-guoren@kernel.org> References: <20230802164701.192791-1-guoren@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Guo Ren The requirements of qspinlock have been documented by commit: a8ad07e5240c ("asm-generic: qspinlock: Indicate the use of mixed-size atomics"). Although RISC-V ISA gives out a weaker forward guarantee LR/SC, which doesn't satisfy the requirements of qspinlock above, it won't prevent some riscv vendors from implementing a strong fwd guarantee LR/SC in microarchitecture to match xchg_tail requirement. T-HEAD C9xx processor is the one. We've tested the patch on SOPHGO sg2042 & th1520 and passed the stress test on Fedora & Ubuntu & OpenEuler ... Here is the performance comparison between qspinlock and ticket_lock on sg2042 (64 cores): sysbench test=threads threads=32 yields=100 lock=8 (+13.8%): queued_spinlock 0.5109/0.00 ticket_spinlock 0.5814/0.00 perf futex/hash (+6.7%): queued_spinlock 1444393 operations/sec (+- 0.09%) ticket_spinlock 1353215 operations/sec (+- 0.15%) perf futex/wake-parallel (+8.6%): queued_spinlock (waking 1/64 threads) in 0.0253 ms (+-2.90%) ticket_spinlock (waking 1/64 threads) in 0.0275 ms (+-3.12%) perf futex/requeue (+4.2%): queued_spinlock Requeued 64 of 64 threads in 0.0785 ms (+-0.55%) ticket_spinlock Requeued 64 of 64 threads in 0.0818 ms (+-4.12%) System Benchmarks (+6.4%) queued_spinlock: System Benchmarks Index Values BASELINE RESULT INDEX Dhrystone 2 using register variables 116700.0 628613745.4 53865.8 Double-Precision Whetstone 55.0 182422.8 33167.8 Execl Throughput 43.0 13116.6 3050.4 File Copy 1024 bufsize 2000 maxblocks 3960.0 7762306.2 19601.8 File Copy 256 bufsize 500 maxblocks 1655.0 3417556.8 20649.9 File Copy 4096 bufsize 8000 maxblocks 5800.0 7427995.7 12806.9 Pipe Throughput 12440.0 23058600.5 18535.9 Pipe-based Context Switching 4000.0 2835617.7 7089.0 Process Creation 126.0 12537.3 995.0 Shell Scripts (1 concurrent) 42.4 57057.4 13456.9 Shell Scripts (8 concurrent) 6.0 7367.1 12278.5 System Call Overhead 15000.0 33308301.3 22205.5 ======== System Benchmarks Index Score 12426.1 ticket_spinlock: System Benchmarks Index Values BASELINE RESULT INDEX Dhrystone 2 using register variables 116700.0 626541701.9 53688.2 Double-Precision Whetstone 55.0 181921.0 33076.5 Execl Throughput 43.0 12625.1 2936.1 File Copy 1024 bufsize 2000 maxblocks 3960.0 6553792.9 16550.0 File Copy 256 bufsize 500 maxblocks 1655.0 3189231.6 19270.3 File Copy 4096 bufsize 8000 maxblocks 5800.0 7221277.0 12450.5 Pipe Throughput 12440.0 20594018.7 16554.7 Pipe-based Context Switching 4000.0 2571117.7 6427.8 Process Creation 126.0 10798.4 857.0 Shell Scripts (1 concurrent) 42.4 57227.5 13497.1 Shell Scripts (8 concurrent) 6.0 7329.2 12215.3 System Call Overhead 15000.0 30766778.4 20511.2 ======== System Benchmarks Index Score 11670.7 The qspinlock has a significant improvement on SOPHGO SG2042 64 cores platform than the ticket_lock. Signed-off-by: Guo Ren Signed-off-by: Guo Ren --- arch/riscv/Kconfig | 16 ++++++++++++++++ arch/riscv/include/asm/Kbuild | 3 ++- arch/riscv/include/asm/cmpxchg.h | 24 ++++++++++++++++++++++++ arch/riscv/include/asm/spinlock.h | 17 +++++++++++++++++ 4 files changed, 59 insertions(+), 1 deletion(-) create mode 100644 arch/riscv/include/asm/spinlock.h diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 93ff677d2be5..e89a3bea3dc1 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -438,6 +438,22 @@ config NODES_SHIFT Specify the maximum number of NUMA Nodes available on the target system. Increases memory reserved to accommodate various tables. +choice + prompt "RISC-V spinlock type" + default RISCV_TICKET_SPINLOCKS + +config RISCV_TICKET_SPINLOCKS + bool "Using ticket spinlock" + +config RISCV_QUEUED_SPINLOCKS + bool "Using queued spinlock" + depends on SMP && MMU + select ARCH_USE_QUEUED_SPINLOCKS + help + Make sure your micro arch LL/SC has a strong forward progress guarantee. + Otherwise, stay at ticket-lock. +endchoice + config RISCV_ALTERNATIVE bool depends on !XIP_KERNEL diff --git a/arch/riscv/include/asm/Kbuild b/arch/riscv/include/asm/Kbuild index 504f8b7e72d4..a0dc85e4a754 100644 --- a/arch/riscv/include/asm/Kbuild +++ b/arch/riscv/include/asm/Kbuild @@ -2,10 +2,11 @@ generic-y += early_ioremap.h generic-y += flat.h generic-y += kvm_para.h +generic-y += mcs_spinlock.h generic-y += parport.h -generic-y += spinlock.h generic-y += spinlock_types.h generic-y += qrwlock.h generic-y += qrwlock_types.h +generic-y += qspinlock.h generic-y += user.h generic-y += vmlinux.lds.h diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index 2f4726d3cfcc..d12231d752a4 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -11,12 +11,36 @@ #include #include +static inline ulong __xchg16_relaxed(ulong new, void *ptr) +{ + ulong ret, tmp; + ulong shif = ((ulong)ptr & 2) ? 16 : 0; + ulong mask = 0xffff << shif; + ulong *__ptr = (ulong *)((ulong)ptr & ~2); + + __asm__ __volatile__ ( + "0: lr.w %0, %2\n" + " and %1, %0, %z3\n" + " or %1, %1, %z4\n" + " sc.w %1, %1, %2\n" + " bnez %1, 0b\n" + : "=&r" (ret), "=&r" (tmp), "+A" (*__ptr) + : "rJ" (~mask), "rJ" (new << shif) + : "memory"); + + return (ulong)((ret & mask) >> shif); +} + #define __xchg_relaxed(ptr, new, size) \ ({ \ __typeof__(ptr) __ptr = (ptr); \ __typeof__(new) __new = (new); \ __typeof__(*(ptr)) __ret; \ switch (size) { \ + case 2: \ + __ret = (__typeof__(*(ptr))) \ + __xchg16_relaxed((ulong)__new, __ptr); \ + break; \ case 4: \ __asm__ __volatile__ ( \ " amoswap.w %0, %2, %1\n" \ diff --git a/arch/riscv/include/asm/spinlock.h b/arch/riscv/include/asm/spinlock.h new file mode 100644 index 000000000000..c644a92d4548 --- /dev/null +++ b/arch/riscv/include/asm/spinlock.h @@ -0,0 +1,17 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef __ASM_RISCV_SPINLOCK_H +#define __ASM_RISCV_SPINLOCK_H + +#ifdef CONFIG_QUEUED_SPINLOCKS +#define _Q_PENDING_LOOPS (1 << 9) +#endif + +#ifdef CONFIG_QUEUED_SPINLOCKS +#include +#include +#else +#include +#endif + +#endif /* __ASM_RISCV_SPINLOCK_H */ From patchwork Wed Aug 2 16:46:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guo Ren X-Patchwork-Id: 13338531 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 73029C07E8A for ; Wed, 2 Aug 2023 16:49:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232040AbjHBQt0 (ORCPT ); Wed, 2 Aug 2023 12:49:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48098 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232583AbjHBQtH (ORCPT ); Wed, 2 Aug 2023 12:49:07 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3C436173A; Wed, 2 Aug 2023 09:49:02 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 74D47619D7; Wed, 2 Aug 2023 16:49:01 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 07D0EC433C8; Wed, 2 Aug 2023 16:48:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1690994940; bh=aihpQWSsOm2PcncftaveeMsJLoT6t46RobNm6fedvI0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=gbOQExxxuCtl7NH7eEJ6hZqSndwRoPGdr6xkLNldxD4i6d4f0tfBTN5SijbtBjNvx fdi8poo5hOLsGtXmkhJcW1h9mQKVPkjqiGLD7kBl5pDcvzZgngOHIN7AMts+s5i3Qj fdBVjn6mxmEqIbXZUouDJ4xRfdc78I8Z3VWhBuxUmvUAX/2fkI46rYY93zJKRTAnfZ 8qSJXsZETG38yrWe6xpLlaUj5vllcbH/4ubvCaSE6xiccTKV+nHZLKQAqK1nkFTZ7J igQd0L4wWlGyOVq+znHVp7JrBhdfRkz+eb4jb9ntBJcSScgyImGIJd8jyTdlAD3Kq6 BC8aj1j1UYJXw== From: guoren@kernel.org To: paul.walmsley@sifive.com, anup@brainfault.org, peterz@infradead.org, mingo@redhat.com, will@kernel.org, palmer@rivosinc.com, longman@redhat.com, boqun.feng@gmail.com, tglx@linutronix.de, paulmck@kernel.org, rostedt@goodmis.org, rdunlap@infradead.org, catalin.marinas@arm.com, conor.dooley@microchip.com, xiaoguang.xing@sophgo.com, bjorn@rivosinc.com, alexghiti@rivosinc.com, keescook@chromium.org, greentime.hu@sifive.com, ajones@ventanamicro.com, jszhang@kernel.org, wefu@redhat.com, wuwei2016@iscas.ac.cn Cc: linux-arch@vger.kernel.org, linux-riscv@lists.infradead.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-csky@vger.kernel.org, Guo Ren , Guo Ren Subject: [PATCH V10 05/19] riscv: qspinlock: Introduce combo spinlock Date: Wed, 2 Aug 2023 12:46:47 -0400 Message-Id: <20230802164701.192791-6-guoren@kernel.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20230802164701.192791-1-guoren@kernel.org> References: <20230802164701.192791-1-guoren@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Guo Ren Combo spinlock could support queued and ticket in one Linux Image and select them during boot time via errata mechanism. Here is the func size (Bytes) comparison table below: TYPE : COMBO | TICKET | QUEUED arch_spin_lock : 106 | 60 | 50 arch_spin_unlock : 54 | 36 | 26 arch_spin_trylock : 110 | 72 | 54 arch_spin_is_locked : 48 | 34 | 20 arch_spin_is_contended : 56 | 40 | 24 rch_spin_value_unlocked : 48 | 34 | 24 One example of disassemble combo arch_spin_unlock: 0xffffffff8000409c <+14>: nop # detour slot 0xffffffff800040a0 <+18>: fence rw,w # queued spinlock start 0xffffffff800040a4 <+22>: sb zero,0(a4) # queued spinlock end 0xffffffff800040a8 <+26>: ld s0,8(sp) 0xffffffff800040aa <+28>: addi sp,sp,16 0xffffffff800040ac <+30>: ret 0xffffffff800040ae <+32>: lw a5,0(a4) # ticket spinlock start 0xffffffff800040b0 <+34>: sext.w a5,a5 0xffffffff800040b2 <+36>: fence rw,w 0xffffffff800040b6 <+40>: addiw a5,a5,1 0xffffffff800040b8 <+42>: slli a5,a5,0x30 0xffffffff800040ba <+44>: srli a5,a5,0x30 0xffffffff800040bc <+46>: sh a5,0(a4) # ticket spinlock end 0xffffffff800040c0 <+50>: ld s0,8(sp) 0xffffffff800040c2 <+52>: addi sp,sp,16 0xffffffff800040c4 <+54>: ret The qspinlock is smaller and faster than ticket-lock when all are in fast-path, and combo spinlock could provide a compatible Linux Image for different micro-arch design (weak/strict fwd guarantee) processors. Signed-off-by: Guo Ren Signed-off-by: Guo Ren --- arch/riscv/Kconfig | 9 +++- arch/riscv/include/asm/hwcap.h | 1 + arch/riscv/include/asm/spinlock.h | 87 ++++++++++++++++++++++++++++++- arch/riscv/kernel/cpufeature.c | 10 ++++ 4 files changed, 104 insertions(+), 3 deletions(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index e89a3bea3dc1..119e774a3dcf 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -440,7 +440,7 @@ config NODES_SHIFT choice prompt "RISC-V spinlock type" - default RISCV_TICKET_SPINLOCKS + default RISCV_COMBO_SPINLOCKS config RISCV_TICKET_SPINLOCKS bool "Using ticket spinlock" @@ -452,6 +452,13 @@ config RISCV_QUEUED_SPINLOCKS help Make sure your micro arch LL/SC has a strong forward progress guarantee. Otherwise, stay at ticket-lock. + +config RISCV_COMBO_SPINLOCKS + bool "Using combo spinlock" + depends on SMP && MMU + select ARCH_USE_QUEUED_SPINLOCKS + help + Select queued spinlock or ticket-lock via errata. endchoice config RISCV_ALTERNATIVE diff --git a/arch/riscv/include/asm/hwcap.h b/arch/riscv/include/asm/hwcap.h index f041bfa7f6a0..08ae75a694c2 100644 --- a/arch/riscv/include/asm/hwcap.h +++ b/arch/riscv/include/asm/hwcap.h @@ -54,6 +54,7 @@ #define RISCV_ISA_EXT_ZIFENCEI 41 #define RISCV_ISA_EXT_ZIHPM 42 +#define RISCV_ISA_EXT_XTICKETLOCK 63 #define RISCV_ISA_EXT_MAX 64 #define RISCV_ISA_EXT_NAME_LEN_MAX 32 diff --git a/arch/riscv/include/asm/spinlock.h b/arch/riscv/include/asm/spinlock.h index c644a92d4548..9eb3ad31e564 100644 --- a/arch/riscv/include/asm/spinlock.h +++ b/arch/riscv/include/asm/spinlock.h @@ -7,11 +7,94 @@ #define _Q_PENDING_LOOPS (1 << 9) #endif +#ifdef CONFIG_RISCV_COMBO_SPINLOCKS +#include + +#undef arch_spin_is_locked +#undef arch_spin_is_contended +#undef arch_spin_value_unlocked +#undef arch_spin_lock +#undef arch_spin_trylock +#undef arch_spin_unlock + +#include +#include + +#undef arch_spin_is_locked +#undef arch_spin_is_contended +#undef arch_spin_value_unlocked +#undef arch_spin_lock +#undef arch_spin_trylock +#undef arch_spin_unlock + +#define COMBO_DETOUR \ + asm_volatile_goto(ALTERNATIVE( \ + "nop", \ + "j %l[ticket_spin_lock]", \ + 0, \ + RISCV_ISA_EXT_XTICKETLOCK, \ + CONFIG_RISCV_COMBO_SPINLOCKS) \ + : : : : ticket_spin_lock); + +static __always_inline void arch_spin_lock(arch_spinlock_t *lock) +{ + COMBO_DETOUR + queued_spin_lock(lock); + return; +ticket_spin_lock: + ticket_spin_lock(lock); +} + +static __always_inline bool arch_spin_trylock(arch_spinlock_t *lock) +{ + COMBO_DETOUR + return queued_spin_trylock(lock); +ticket_spin_lock: + return ticket_spin_trylock(lock); +} + +static __always_inline void arch_spin_unlock(arch_spinlock_t *lock) +{ + COMBO_DETOUR + queued_spin_unlock(lock); + return; +ticket_spin_lock: + ticket_spin_unlock(lock); +} + +static __always_inline int arch_spin_value_unlocked(arch_spinlock_t lock) +{ + COMBO_DETOUR + return queued_spin_value_unlocked(lock); +ticket_spin_lock: + return ticket_spin_value_unlocked(lock); +} + +static __always_inline int arch_spin_is_locked(arch_spinlock_t *lock) +{ + COMBO_DETOUR + return queued_spin_is_locked(lock); +ticket_spin_lock: + return ticket_spin_is_locked(lock); +} + +static __always_inline int arch_spin_is_contended(arch_spinlock_t *lock) +{ + COMBO_DETOUR + return queued_spin_is_contended(lock); +ticket_spin_lock: + return ticket_spin_is_contended(lock); +} +#else /* CONFIG_RISCV_COMBO_SPINLOCKS */ + #ifdef CONFIG_QUEUED_SPINLOCKS #include -#include #else -#include +#include #endif +#endif /* CONFIG_RISCV_COMBO_SPINLOCKS */ + +#include + #endif /* __ASM_RISCV_SPINLOCK_H */ diff --git a/arch/riscv/kernel/cpufeature.c b/arch/riscv/kernel/cpufeature.c index bdcf460ea53d..e65b0e54152d 100644 --- a/arch/riscv/kernel/cpufeature.c +++ b/arch/riscv/kernel/cpufeature.c @@ -324,6 +324,16 @@ void __init riscv_fill_hwcap(void) set_bit(RISCV_ISA_EXT_ZICSR, isainfo->isa); set_bit(RISCV_ISA_EXT_ZIFENCEI, isainfo->isa); +#ifdef CONFIG_RISCV_COMBO_SPINLOCKS + /* + * The RISC-V Linux used queued spinlock at first; then, we used ticket_lock + * as default or queued spinlock by choice. Because ticket_lock would dirty + * spinlock value, the only way is to change from queued_spinlock to + * ticket_spinlock, but can not be vice. + */ + set_bit(RISCV_ISA_EXT_XTICKETLOCK, isainfo->isa); +#endif + /* * These ones were as they were part of the base ISA when the * port & dt-bindings were upstreamed, and so can be set From patchwork Wed Aug 2 16:46:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guo Ren X-Patchwork-Id: 13338530 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 81DF9C0729B for ; Wed, 2 Aug 2023 16:49:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232151AbjHBQt2 (ORCPT ); Wed, 2 Aug 2023 12:49:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48234 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232950AbjHBQtT (ORCPT ); Wed, 2 Aug 2023 12:49:19 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D979C1706; Wed, 2 Aug 2023 09:49:17 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 6EAE161A17; Wed, 2 Aug 2023 16:49:17 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E672FC433C8; Wed, 2 Aug 2023 16:49:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1690994956; bh=zXadGqM9fefJG2o17N1iiWReM7OXWmGr0IUdQMbdNag=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=sDcnqDZRSZQ9uG7QJ0aVY5DisKCP515Eyzx7mvxzsL6P5Ul/fPNNvSb32twmSd8it qieRnrGYw7HOj5JqJemsIUZpitoct9zldeVgAPk7BGpFg0GqYqJrv50b/Z50HtqDUp XfHXw9O4jBEhAnsasVvlR0HBZpfDl0V9JUrXc6qihxYjkweOOXC8REcSnqv4d0waYU Ty7CSFAQ8X4j0hWebV+aGG0czYf6lnmYjfkJgXTCQOf2LMxmrFOHxHh5eICKrAlpgx 5lH/Q2Jf841KXPYPQlDJkI+zB0CjAaKJphP5IvKA5WhjLlqlnxheOJQxLDirACCoiI aGGEYb4YF7f3w== From: guoren@kernel.org To: paul.walmsley@sifive.com, anup@brainfault.org, peterz@infradead.org, mingo@redhat.com, will@kernel.org, palmer@rivosinc.com, longman@redhat.com, boqun.feng@gmail.com, tglx@linutronix.de, paulmck@kernel.org, rostedt@goodmis.org, rdunlap@infradead.org, catalin.marinas@arm.com, conor.dooley@microchip.com, xiaoguang.xing@sophgo.com, bjorn@rivosinc.com, alexghiti@rivosinc.com, keescook@chromium.org, greentime.hu@sifive.com, ajones@ventanamicro.com, jszhang@kernel.org, wefu@redhat.com, wuwei2016@iscas.ac.cn Cc: linux-arch@vger.kernel.org, linux-riscv@lists.infradead.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-csky@vger.kernel.org, Guo Ren , Guo Ren Subject: [PATCH V10 06/19] riscv: qspinlock: Allow force qspinlock from the command line Date: Wed, 2 Aug 2023 12:46:48 -0400 Message-Id: <20230802164701.192791-7-guoren@kernel.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20230802164701.192791-1-guoren@kernel.org> References: <20230802164701.192791-1-guoren@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Guo Ren Allow cmdline to force the kernel to use queued_spinlock when CONFIG_RISCV_COMBO_SPINLOCKS=y. Signed-off-by: Guo Ren Signed-off-by: Guo Ren --- Documentation/admin-guide/kernel-parameters.txt | 3 +++ arch/riscv/include/asm/cpufeature.h | 2 ++ arch/riscv/kernel/cpufeature.c | 15 ++++++++++++++- 3 files changed, 19 insertions(+), 1 deletion(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index be40bfbf4380..de6b7ee752cd 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -4666,6 +4666,9 @@ quiet [KNL] Disable most log messages + qspinlock [RISCV] Forces kernel to use queued_spinlock when + CONFIG_RISCV_COMBO_SPINLOCKS=y. + r128= [HW,DRM] radix_hcall_invalidate=on [PPC/PSERIES] diff --git a/arch/riscv/include/asm/cpufeature.h b/arch/riscv/include/asm/cpufeature.h index 23fed53b8815..2bf0343661da 100644 --- a/arch/riscv/include/asm/cpufeature.h +++ b/arch/riscv/include/asm/cpufeature.h @@ -30,4 +30,6 @@ DECLARE_PER_CPU(long, misaligned_access_speed); /* Per-cpu ISA extensions. */ extern struct riscv_isainfo hart_isa[NR_CPUS]; +extern bool force_qspinlock; + #endif diff --git a/arch/riscv/kernel/cpufeature.c b/arch/riscv/kernel/cpufeature.c index e65b0e54152d..f8dbbe1bbd34 100644 --- a/arch/riscv/kernel/cpufeature.c +++ b/arch/riscv/kernel/cpufeature.c @@ -99,6 +99,17 @@ static bool riscv_isa_extension_check(int id) return true; } +#ifdef CONFIG_QUEUED_SPINLOCKS +bool force_qspinlock = false; +static int __init force_queued_spinlock(char *p) +{ + force_qspinlock = true; + pr_info("Force kernel to use queued_spinlock"); + return 0; +} +early_param("qspinlock", force_queued_spinlock); +#endif + void __init riscv_fill_hwcap(void) { struct device_node *node; @@ -331,7 +342,9 @@ void __init riscv_fill_hwcap(void) * spinlock value, the only way is to change from queued_spinlock to * ticket_spinlock, but can not be vice. */ - set_bit(RISCV_ISA_EXT_XTICKETLOCK, isainfo->isa); + if (!force_qspinlock) { + set_bit(RISCV_ISA_EXT_XTICKETLOCK, isainfo->isa); + } #endif /* From patchwork Wed Aug 2 16:46:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guo Ren X-Patchwork-Id: 13338532 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51EC6C04FE0 for ; Wed, 2 Aug 2023 16:50:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232228AbjHBQt5 (ORCPT ); Wed, 2 Aug 2023 12:49:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48826 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232949AbjHBQto (ORCPT ); Wed, 2 Aug 2023 12:49:44 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EB8D230E0; Wed, 2 Aug 2023 09:49:34 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id D92C261A18; Wed, 2 Aug 2023 16:49:33 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id DE991C433C7; Wed, 2 Aug 2023 16:49:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1690994973; bh=qTocF+5HifQQ2/cevfHJiy/Dt4gxsiBcGO8u/t7qAjw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HzKhu1rw4zVY83o3aI0QRxuEoefnVtHfGIm54LDyeZfq6g0LxLrXs7I4vNqtzFs6+ 2q8sf2QEoId6+sKw73J7aEIT5YLQLPVa/vzfO+h6M85J04AkY6O7Ww9717CT31VAfz pL+nmf8pE/EDqMdwrvEl4pC2GTEn7nWlvDpt2xBBAnqn4xFaeqsL1+2Z7fTA2UiWmV h8wzWKSgpZM7N5ip55dDhWMilqgGJn4HLsfjl3vf/97Wq88D754dkMU9ojregLnLNz uxMU25MAhuAxaBFytRB/nZn8ozCqw4yybE4TlWd0BStHvq5hgLlpZYJwCbuxOjwIs7 NuiGKj3a2TZrA== From: guoren@kernel.org To: paul.walmsley@sifive.com, anup@brainfault.org, peterz@infradead.org, mingo@redhat.com, will@kernel.org, palmer@rivosinc.com, longman@redhat.com, boqun.feng@gmail.com, tglx@linutronix.de, paulmck@kernel.org, rostedt@goodmis.org, rdunlap@infradead.org, catalin.marinas@arm.com, conor.dooley@microchip.com, xiaoguang.xing@sophgo.com, bjorn@rivosinc.com, alexghiti@rivosinc.com, keescook@chromium.org, greentime.hu@sifive.com, ajones@ventanamicro.com, jszhang@kernel.org, wefu@redhat.com, wuwei2016@iscas.ac.cn Cc: linux-arch@vger.kernel.org, linux-riscv@lists.infradead.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-csky@vger.kernel.org, Guo Ren , Guo Ren Subject: [PATCH V10 07/19] riscv: qspinlock: errata: Introduce ERRATA_THEAD_QSPINLOCK Date: Wed, 2 Aug 2023 12:46:49 -0400 Message-Id: <20230802164701.192791-8-guoren@kernel.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20230802164701.192791-1-guoren@kernel.org> References: <20230802164701.192791-1-guoren@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Guo Ren According to qspinlock requirements, RISC-V gives out a weak LR/SC forward progress guarantee which does not satisfy qspinlock. But many vendors could produce stronger forward guarantee LR/SC to ensure the xchg_tail could be finished in time on any kind of hart. T-HEAD is the vendor which implements strong forward guarantee LR/SC instruction pairs, so enable qspinlock for T-HEAD with errata help. T-HEAD early version of processors has the merge buffer delay problem, so we need ERRATA_WRITEONCE to support qspinlock. Signed-off-by: Guo Ren Signed-off-by: Guo Ren --- arch/riscv/Kconfig.errata | 13 +++++++++++++ arch/riscv/errata/thead/errata.c | 24 ++++++++++++++++++++++++ arch/riscv/include/asm/errata_list.h | 20 ++++++++++++++++++++ arch/riscv/include/asm/vendorid_list.h | 3 ++- arch/riscv/kernel/cpufeature.c | 3 ++- 5 files changed, 61 insertions(+), 2 deletions(-) diff --git a/arch/riscv/Kconfig.errata b/arch/riscv/Kconfig.errata index 4745a5c57e7c..eb43677b13cc 100644 --- a/arch/riscv/Kconfig.errata +++ b/arch/riscv/Kconfig.errata @@ -96,4 +96,17 @@ config ERRATA_THEAD_WRITE_ONCE If you don't know what to do here, say "Y". +config ERRATA_THEAD_QSPINLOCK + bool "Apply T-Head queued spinlock errata" + depends on ERRATA_THEAD + default y + help + The T-HEAD C9xx processors implement strong fwd guarantee LR/SC to + match the xchg_tail requirement of qspinlock. + + This will apply the QSPINLOCK errata to handle the non-standard + behavior via using qspinlock instead of ticket_lock. + + If you don't know what to do here, say "Y". + endmenu # "CPU errata selection" diff --git a/arch/riscv/errata/thead/errata.c b/arch/riscv/errata/thead/errata.c index 881729746d2e..d560dc45c0e7 100644 --- a/arch/riscv/errata/thead/errata.c +++ b/arch/riscv/errata/thead/errata.c @@ -86,6 +86,27 @@ static bool errata_probe_write_once(unsigned int stage, return false; } +static bool errata_probe_qspinlock(unsigned int stage, + unsigned long arch_id, unsigned long impid) +{ + if (!IS_ENABLED(CONFIG_ERRATA_THEAD_QSPINLOCK)) + return false; + + /* + * The queued_spinlock torture would get in livelock without + * ERRATA_THEAD_WRITE_ONCE fixup for the early versions of T-HEAD + * processors. + */ + if (arch_id == 0 && impid == 0 && + !IS_ENABLED(CONFIG_ERRATA_THEAD_WRITE_ONCE)) + return false; + + if (stage == RISCV_ALTERNATIVES_EARLY_BOOT) + return true; + + return false; +} + static u32 thead_errata_probe(unsigned int stage, unsigned long archid, unsigned long impid) { @@ -103,6 +124,9 @@ static u32 thead_errata_probe(unsigned int stage, if (errata_probe_write_once(stage, archid, impid)) cpu_req_errata |= BIT(ERRATA_THEAD_WRITE_ONCE); + if (errata_probe_qspinlock(stage, archid, impid)) + cpu_req_errata |= BIT(ERRATA_THEAD_QSPINLOCK); + return cpu_req_errata; } diff --git a/arch/riscv/include/asm/errata_list.h b/arch/riscv/include/asm/errata_list.h index fbb2b8d39321..a696d18d1b0d 100644 --- a/arch/riscv/include/asm/errata_list.h +++ b/arch/riscv/include/asm/errata_list.h @@ -141,6 +141,26 @@ asm volatile(ALTERNATIVE( \ : "=r" (__ovl) : \ : "memory") +static __always_inline bool +riscv_has_errata_thead_qspinlock(void) +{ + if (IS_ENABLED(CONFIG_RISCV_ALTERNATIVE)) { + asm_volatile_goto( + ALTERNATIVE( + "j %l[l_no]", "nop", + THEAD_VENDOR_ID, + ERRATA_THEAD_QSPINLOCK, + CONFIG_ERRATA_THEAD_QSPINLOCK) + : : : : l_no); + } else { + goto l_no; + } + + return true; +l_no: + return false; +} + #endif /* __ASSEMBLY__ */ #endif diff --git a/arch/riscv/include/asm/vendorid_list.h b/arch/riscv/include/asm/vendorid_list.h index 73078cfe4029..1f1d03877f5f 100644 --- a/arch/riscv/include/asm/vendorid_list.h +++ b/arch/riscv/include/asm/vendorid_list.h @@ -19,7 +19,8 @@ #define ERRATA_THEAD_CMO 1 #define ERRATA_THEAD_PMU 2 #define ERRATA_THEAD_WRITE_ONCE 3 -#define ERRATA_THEAD_NUMBER 4 +#define ERRATA_THEAD_QSPINLOCK 4 +#define ERRATA_THEAD_NUMBER 5 #endif #endif diff --git a/arch/riscv/kernel/cpufeature.c b/arch/riscv/kernel/cpufeature.c index f8dbbe1bbd34..d9694fe40a9a 100644 --- a/arch/riscv/kernel/cpufeature.c +++ b/arch/riscv/kernel/cpufeature.c @@ -342,7 +342,8 @@ void __init riscv_fill_hwcap(void) * spinlock value, the only way is to change from queued_spinlock to * ticket_spinlock, but can not be vice. */ - if (!force_qspinlock) { + if (!force_qspinlock && + !riscv_has_errata_thead_qspinlock()) { set_bit(RISCV_ISA_EXT_XTICKETLOCK, isainfo->isa); } #endif From patchwork Wed Aug 2 16:46:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guo Ren X-Patchwork-Id: 13338533 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3B7EAC04FE2 for ; Wed, 2 Aug 2023 16:50:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232272AbjHBQua (ORCPT ); Wed, 2 Aug 2023 12:50:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49138 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232714AbjHBQuI (ORCPT ); Wed, 2 Aug 2023 12:50:08 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E1F732D72; Wed, 2 Aug 2023 09:49:50 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 7E84A61904; Wed, 2 Aug 2023 16:49:50 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5AD4FC433C7; Wed, 2 Aug 2023 16:49:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1690994989; bh=go9pWbNchcwuDXo4BS51ZWl4krGce3v8bmUXX/P16DY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ofclQu8EDW9/sQM3zWUZMd80r7/7jjuHKIkp5Xzt/QQSpSCh0MFzNl3O0AkzPYFbA CU7pEG3Y0/TkvfISPDhwLJObQpG8OKd0+HnLknFYXFXsHWzYY7pXQuE3Fe1rYYh+Cq LOK0Bja1ObZMnukpwha/6gTzBcjXQTY5c3LcEBwxpARTkW1XuXkSPxZsC0pel8JJMF LXH0MOaABlGD8CRWGilKgijG32MJSknHiH2r+OxsDc3lURtiVKPUuphTRApY4Whiek epVJKAMIEXHV3dGVCeKI8LdBM+CaGH561XWJeLRx5hudDOxwnMkz9Sy2RK+Pj4Qtd1 c+rDQ+OUloG6A== From: guoren@kernel.org To: paul.walmsley@sifive.com, anup@brainfault.org, peterz@infradead.org, mingo@redhat.com, will@kernel.org, palmer@rivosinc.com, longman@redhat.com, boqun.feng@gmail.com, tglx@linutronix.de, paulmck@kernel.org, rostedt@goodmis.org, rdunlap@infradead.org, catalin.marinas@arm.com, conor.dooley@microchip.com, xiaoguang.xing@sophgo.com, bjorn@rivosinc.com, alexghiti@rivosinc.com, keescook@chromium.org, greentime.hu@sifive.com, ajones@ventanamicro.com, jszhang@kernel.org, wefu@redhat.com, wuwei2016@iscas.ac.cn Cc: linux-arch@vger.kernel.org, linux-riscv@lists.infradead.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-csky@vger.kernel.org, Guo Ren , Guo Ren Subject: [PATCH V10 08/19] riscv: qspinlock: Use new static key for controlling call of virt_spin_lock() Date: Wed, 2 Aug 2023 12:46:50 -0400 Message-Id: <20230802164701.192791-9-guoren@kernel.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20230802164701.192791-1-guoren@kernel.org> References: <20230802164701.192791-1-guoren@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Guo Ren Add a static key controlling whether virt_spin_lock() should be called or not. When running on bare metal set the new key to false. The KVM guests fall back to a Test-and-Set spinlock, because fair locks have horrible lock 'holder' preemption issues. The virt_spin_lock_key would shortcut for the queued_spin_lock_slowpath() function that allow virt_spin_lock to hijack it. Signed-off-by: Guo Ren Signed-off-by: Guo Ren --- arch/riscv/Kconfig | 1 + arch/riscv/include/asm/sbi.h | 8 ++++++++ arch/riscv/include/asm/spinlock.h | 22 ++++++++++++++++++++++ arch/riscv/kernel/cpufeature.c | 4 +++- arch/riscv/kernel/sbi.c | 2 +- arch/riscv/kernel/setup.c | 19 +++++++++++++++++++ 6 files changed, 54 insertions(+), 2 deletions(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 119e774a3dcf..42ae45c42b4d 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -20,6 +20,7 @@ config RISCV select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE select ARCH_HAS_BINFMT_FLAT select ARCH_HAS_CURRENT_STACK_POINTER + select ARCH_HAS_CPU_FINALIZE_INIT select ARCH_HAS_DEBUG_VIRTUAL if MMU select ARCH_HAS_DEBUG_VM_PGTABLE select ARCH_HAS_DEBUG_WX diff --git a/arch/riscv/include/asm/sbi.h b/arch/riscv/include/asm/sbi.h index e1523c8624cc..b7ced34b79a3 100644 --- a/arch/riscv/include/asm/sbi.h +++ b/arch/riscv/include/asm/sbi.h @@ -51,6 +51,13 @@ enum sbi_ext_base_fid { SBI_EXT_BASE_GET_MIMPID, }; +enum sbi_ext_base_impl_id { + SBI_EXT_BASE_IMPL_ID_BBL = 0, + SBI_EXT_BASE_IMPL_ID_OPENSBI, + SBI_EXT_BASE_IMPL_ID_XVISOR, + SBI_EXT_BASE_IMPL_ID_KVM, +}; + enum sbi_ext_time_fid { SBI_EXT_TIME_SET_TIMER = 0, }; @@ -286,6 +293,7 @@ int sbi_console_getchar(void); long sbi_get_mvendorid(void); long sbi_get_marchid(void); long sbi_get_mimpid(void); +long sbi_get_firmware_id(void); void sbi_set_timer(uint64_t stime_value); void sbi_shutdown(void); void sbi_send_ipi(unsigned int cpu); diff --git a/arch/riscv/include/asm/spinlock.h b/arch/riscv/include/asm/spinlock.h index 9eb3ad31e564..13f3e14500c0 100644 --- a/arch/riscv/include/asm/spinlock.h +++ b/arch/riscv/include/asm/spinlock.h @@ -4,6 +4,28 @@ #define __ASM_RISCV_SPINLOCK_H #ifdef CONFIG_QUEUED_SPINLOCKS +/* + * The KVM guests fall back to a Test-and-Set spinlock, because fair locks + * have horrible lock 'holder' preemption issues. The virt_spin_lock_key + * would shortcut for the queued_spin_lock_slowpath() function that allow + * virt_spin_lock to hijack it. + */ +DECLARE_STATIC_KEY_TRUE(virt_spin_lock_key); + +#define virt_spin_lock virt_spin_lock +static inline bool virt_spin_lock(struct qspinlock *lock) +{ + if (!static_branch_likely(&virt_spin_lock_key)) + return false; + + do { + while (atomic_read(&lock->val) != 0) + cpu_relax(); + } while (atomic_cmpxchg(&lock->val, 0, _Q_LOCKED_VAL) != 0); + + return true; +} + #define _Q_PENDING_LOOPS (1 << 9) #endif diff --git a/arch/riscv/kernel/cpufeature.c b/arch/riscv/kernel/cpufeature.c index d9694fe40a9a..26826aa590e9 100644 --- a/arch/riscv/kernel/cpufeature.c +++ b/arch/riscv/kernel/cpufeature.c @@ -21,6 +21,7 @@ #include #include #include +#include #include #define NUM_ALPHA_EXTS ('z' - 'a' + 1) @@ -343,7 +344,8 @@ void __init riscv_fill_hwcap(void) * ticket_spinlock, but can not be vice. */ if (!force_qspinlock && - !riscv_has_errata_thead_qspinlock()) { + !riscv_has_errata_thead_qspinlock() && + (sbi_get_firmware_id() != SBI_EXT_BASE_IMPL_ID_KVM)) { set_bit(RISCV_ISA_EXT_XTICKETLOCK, isainfo->isa); } #endif diff --git a/arch/riscv/kernel/sbi.c b/arch/riscv/kernel/sbi.c index c672c8ba9a2a..398b768a02e6 100644 --- a/arch/riscv/kernel/sbi.c +++ b/arch/riscv/kernel/sbi.c @@ -555,7 +555,7 @@ static inline long sbi_get_spec_version(void) return __sbi_base_ecall(SBI_EXT_BASE_GET_SPEC_VERSION); } -static inline long sbi_get_firmware_id(void) +long sbi_get_firmware_id(void) { return __sbi_base_ecall(SBI_EXT_BASE_GET_IMP_ID); } diff --git a/arch/riscv/kernel/setup.c b/arch/riscv/kernel/setup.c index 971fe776e2f8..def89fd8ea55 100644 --- a/arch/riscv/kernel/setup.c +++ b/arch/riscv/kernel/setup.c @@ -26,6 +26,7 @@ #include #include #include +#include #include #include #include @@ -264,6 +265,19 @@ static void __init parse_dtb(void) #endif } +#ifdef CONFIG_QUEUED_SPINLOCKS +DEFINE_STATIC_KEY_TRUE(virt_spin_lock_key); + +static void __init virt_spin_lock_init(void) +{ + if (sbi_get_firmware_id() != SBI_EXT_BASE_IMPL_ID_KVM || + force_qspinlock) + static_branch_disable(&virt_spin_lock_key); +} +#else +static void __init virt_spin_lock_init(void) {} +#endif + extern void __init init_rt_signal_env(void); void __init setup_arch(char **cmdline_p) @@ -313,6 +327,11 @@ void __init setup_arch(char **cmdline_p) riscv_noncoherent_supported(); } +void __init arch_cpu_finalize_init(void) +{ + virt_spin_lock_init(); +} + static int __init topology_init(void) { int i, ret; From patchwork Wed Aug 2 16:46:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guo Ren X-Patchwork-Id: 13338534 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 31F9BC04FDF for ; Wed, 2 Aug 2023 16:51:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233196AbjHBQvD (ORCPT ); Wed, 2 Aug 2023 12:51:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50086 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232603AbjHBQu4 (ORCPT ); Wed, 2 Aug 2023 12:50:56 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C39AE2D61; Wed, 2 Aug 2023 09:50:07 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id B204F6194E; Wed, 2 Aug 2023 16:50:06 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id EB4E6C433C7; Wed, 2 Aug 2023 16:49:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1690995006; bh=Rjshp45LQArowQ9hv/4+Aw89M657nw9ZwvAThzS+oLI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Ul+JWWftqBfrSiIMM8uwIhvGf1Rd8Ct2roXkDyPWcrDpfT26ipur7QUGcgxWigjHl 6Js1tarsK154ZTfax8O3aqCZ9YQrRB1M1wT5D2ndbUEriWKgYHG+IFuHzq9ZsaFysj l0fhDVZsGKgxwAxMlkaC2bFeATrZ7zUAEXnpbE41OMgL8vPwDyMb8HPv3hfQ6d5ObD 1cR7XpXxYNVOeYYdzdmHNk2zTtOuRQq8c458BUnQ79m2Er1mxQhZ19rlbtp9aKAVSH iZ42oA761ZIxJhuUAVrZtunPv1ymTfGoYfo546c3CZtcTZChJ2kVRKgsvv+PyhD9L6 IDWH+ww3fTdqA== From: guoren@kernel.org To: paul.walmsley@sifive.com, anup@brainfault.org, peterz@infradead.org, mingo@redhat.com, will@kernel.org, palmer@rivosinc.com, longman@redhat.com, boqun.feng@gmail.com, tglx@linutronix.de, paulmck@kernel.org, rostedt@goodmis.org, rdunlap@infradead.org, catalin.marinas@arm.com, conor.dooley@microchip.com, xiaoguang.xing@sophgo.com, bjorn@rivosinc.com, alexghiti@rivosinc.com, keescook@chromium.org, greentime.hu@sifive.com, ajones@ventanamicro.com, jszhang@kernel.org, wefu@redhat.com, wuwei2016@iscas.ac.cn Cc: linux-arch@vger.kernel.org, linux-riscv@lists.infradead.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-csky@vger.kernel.org, Guo Ren , Guo Ren Subject: [PATCH V10 09/19] RISC-V: paravirt: pvqspinlock: Add paravirt qspinlock skeleton Date: Wed, 2 Aug 2023 12:46:51 -0400 Message-Id: <20230802164701.192791-10-guoren@kernel.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20230802164701.192791-1-guoren@kernel.org> References: <20230802164701.192791-1-guoren@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Guo Ren Using static_call to switch between: native_queued_spin_lock_slowpath() __pv_queued_spin_lock_slowpath() native_queued_spin_unlock() __pv_queued_spin_unlock() Finish the pv_wait implementation, but pv_kick needs the SBI definition of the next patches. Signed-off-by: Guo Ren Signed-off-by: Guo Ren --- arch/riscv/include/asm/Kbuild | 1 - arch/riscv/include/asm/paravirt.h | 20 +++++++++ arch/riscv/include/asm/qspinlock.h | 29 ++++++++++++ arch/riscv/include/asm/qspinlock_paravirt.h | 7 +++ arch/riscv/include/asm/spinlock.h | 2 +- arch/riscv/kernel/paravirt.c | 50 +++++++++++++++++++++ arch/riscv/kernel/setup.c | 3 ++ 7 files changed, 110 insertions(+), 2 deletions(-) create mode 100644 arch/riscv/include/asm/qspinlock.h create mode 100644 arch/riscv/include/asm/qspinlock_paravirt.h diff --git a/arch/riscv/include/asm/Kbuild b/arch/riscv/include/asm/Kbuild index a0dc85e4a754..b89cb3b73c13 100644 --- a/arch/riscv/include/asm/Kbuild +++ b/arch/riscv/include/asm/Kbuild @@ -7,6 +7,5 @@ generic-y += parport.h generic-y += spinlock_types.h generic-y += qrwlock.h generic-y += qrwlock_types.h -generic-y += qspinlock.h generic-y += user.h generic-y += vmlinux.lds.h diff --git a/arch/riscv/include/asm/paravirt.h b/arch/riscv/include/asm/paravirt.h index 10ba3d6bae4f..ed7eebbedae8 100644 --- a/arch/riscv/include/asm/paravirt.h +++ b/arch/riscv/include/asm/paravirt.h @@ -26,4 +26,24 @@ int __init pv_time_init(void); #endif // CONFIG_PARAVIRT +#ifdef CONFIG_PARAVIRT_SPINLOCKS + +void pv_wait(u8 *ptr, u8 val); +void pv_kick(int cpu); + +void dummy_queued_spin_lock_slowpath(struct qspinlock *lock, u32 val); +void dummy_queued_spin_unlock(struct qspinlock *lock); + +DECLARE_STATIC_CALL(pv_queued_spin_lock_slowpath, dummy_queued_spin_lock_slowpath); +DECLARE_STATIC_CALL(pv_queued_spin_unlock, dummy_queued_spin_unlock); + +void __init pv_qspinlock_init(void); + +static inline bool pv_is_native_spin_unlock(void) +{ + return false; +} + +#endif /* CONFIG_PARAVIRT_SPINLOCKS */ + #endif diff --git a/arch/riscv/include/asm/qspinlock.h b/arch/riscv/include/asm/qspinlock.h new file mode 100644 index 000000000000..003e9560a0d1 --- /dev/null +++ b/arch/riscv/include/asm/qspinlock.h @@ -0,0 +1,29 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_RISCV_QSPINLOCK_H +#define _ASM_RISCV_QSPINLOCK_H + +#ifdef CONFIG_PARAVIRT_SPINLOCKS +#include + +/* How long a lock should spin before we consider blocking */ +#define SPIN_THRESHOLD (1 << 15) + +void native_queued_spin_lock_slowpath(struct qspinlock *lock, u32 val); +void __pv_init_lock_hash(void); +void __pv_queued_spin_lock_slowpath(struct qspinlock *lock, u32 val); + +static inline void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) +{ + static_call(pv_queued_spin_lock_slowpath)(lock, val); +} + +#define queued_spin_unlock queued_spin_unlock +static inline void queued_spin_unlock(struct qspinlock *lock) +{ + static_call(pv_queued_spin_unlock)(lock); +} +#endif /* CONFIG_PARAVIRT_SPINLOCKS */ + +#include + +#endif /* _ASM_RISCV_QSPINLOCK_H */ diff --git a/arch/riscv/include/asm/qspinlock_paravirt.h b/arch/riscv/include/asm/qspinlock_paravirt.h new file mode 100644 index 000000000000..ff52b41d8288 --- /dev/null +++ b/arch/riscv/include/asm/qspinlock_paravirt.h @@ -0,0 +1,7 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_RISCV_QSPINLOCK_PARAVIRT_H +#define _ASM_RISCV_QSPINLOCK_PARAVIRT_H + +void __pv_queued_spin_unlock(struct qspinlock *lock); + +#endif /* _ASM_RISCV_QSPINLOCK_PARAVIRT_H */ diff --git a/arch/riscv/include/asm/spinlock.h b/arch/riscv/include/asm/spinlock.h index 13f3e14500c0..a8ba39e5f8dd 100644 --- a/arch/riscv/include/asm/spinlock.h +++ b/arch/riscv/include/asm/spinlock.h @@ -39,7 +39,7 @@ static inline bool virt_spin_lock(struct qspinlock *lock) #undef arch_spin_trylock #undef arch_spin_unlock -#include +#include #include #undef arch_spin_is_locked diff --git a/arch/riscv/kernel/paravirt.c b/arch/riscv/kernel/paravirt.c index 35816fc10470..1bacb2cf3872 100644 --- a/arch/riscv/kernel/paravirt.c +++ b/arch/riscv/kernel/paravirt.c @@ -130,3 +130,53 @@ int __init pv_time_init(void) return 0; } + +#ifdef CONFIG_PARAVIRT_SPINLOCKS +#include + +void pv_kick(int cpu) +{ + return; +} + +void pv_wait(u8 *ptr, u8 val) +{ + unsigned long flags; + + if (in_nmi()) + return; + + local_irq_save(flags); + if (READ_ONCE(*ptr) != val) + goto out; + + /* wait_for_interrupt(); */ +out: + local_irq_restore(flags); +} + +static void native_queued_spin_unlock(struct qspinlock *lock) +{ + smp_store_release(&lock->locked, 0); +} + +DEFINE_STATIC_CALL(pv_queued_spin_lock_slowpath, native_queued_spin_lock_slowpath); +DEFINE_STATIC_CALL(pv_queued_spin_unlock, native_queued_spin_unlock); +EXPORT_SYMBOL(__SCK__pv_queued_spin_lock_slowpath); +EXPORT_SYMBOL(__SCK__pv_queued_spin_unlock); + +void __init pv_qspinlock_init(void) +{ + if (num_possible_cpus() == 1) + return; + + if(sbi_get_firmware_id() != SBI_EXT_BASE_IMPL_ID_KVM) + return; + + pr_info("PV qspinlocks enabled\n"); + __pv_init_lock_hash(); + + static_call_update(pv_queued_spin_lock_slowpath, __pv_queued_spin_lock_slowpath); + static_call_update(pv_queued_spin_unlock, __pv_queued_spin_unlock); +} +#endif diff --git a/arch/riscv/kernel/setup.c b/arch/riscv/kernel/setup.c index def89fd8ea55..40f5b9402562 100644 --- a/arch/riscv/kernel/setup.c +++ b/arch/riscv/kernel/setup.c @@ -329,6 +329,9 @@ void __init setup_arch(char **cmdline_p) void __init arch_cpu_finalize_init(void) { +#ifdef CONFIG_PARAVIRT_SPINLOCKS + pv_qspinlock_init(); +#endif virt_spin_lock_init(); } From patchwork Wed Aug 2 16:46:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guo Ren X-Patchwork-Id: 13338535 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39771C04FDF for ; Wed, 2 Aug 2023 16:51:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232483AbjHBQvT (ORCPT ); Wed, 2 Aug 2023 12:51:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49418 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232933AbjHBQvC (ORCPT ); Wed, 2 Aug 2023 12:51:02 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A9EBF3A9C; Wed, 2 Aug 2023 09:50:26 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 128D561A3E; Wed, 2 Aug 2023 16:50:24 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2738EC433C7; Wed, 2 Aug 2023 16:50:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1690995023; bh=bat3FwrPV4hDxOZrOP6fq0ZdEXX2+aD2vo+X+inPItQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=RfQbWF0ryBMZj9lS9rACiEr4RyMgk2VRALrNTpfZyd/PIu53tAKeFgwib7bdW4zgW Br5hCTMFt+wAqOYAVBGRrleMQdJ1zwsuT2VBnLiuK65c5QMuNiUZfjuHs+VKnQG7Ad bmqjurJ/ybOrPtp7GjhcCMlki05JpshSkd0YoUSwoRA23Y/o1JHgiw1pDl2fCzwQiT 0PcPKqBmXV9B9h5QM1X+B3PseKNH00sUvtl0jHkC8SLg+Lvifp2+eFNqPzEZf4YKaN 6a0pLmlHG3Dq5HNId3aCD3yZvcXzWunuLXXrsu26ScaqEgy1Zyl2lomyhFjEeebdZ6 h2gds1o4mNRxA== From: guoren@kernel.org To: paul.walmsley@sifive.com, anup@brainfault.org, peterz@infradead.org, mingo@redhat.com, will@kernel.org, palmer@rivosinc.com, longman@redhat.com, boqun.feng@gmail.com, tglx@linutronix.de, paulmck@kernel.org, rostedt@goodmis.org, rdunlap@infradead.org, catalin.marinas@arm.com, conor.dooley@microchip.com, xiaoguang.xing@sophgo.com, bjorn@rivosinc.com, alexghiti@rivosinc.com, keescook@chromium.org, greentime.hu@sifive.com, ajones@ventanamicro.com, jszhang@kernel.org, wefu@redhat.com, wuwei2016@iscas.ac.cn Cc: linux-arch@vger.kernel.org, linux-riscv@lists.infradead.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-csky@vger.kernel.org, Guo Ren , Guo Ren Subject: [PATCH V10 10/19] RISC-V: paravirt: pvqspinlock: KVM: Add paravirt qspinlock skeleton Date: Wed, 2 Aug 2023 12:46:52 -0400 Message-Id: <20230802164701.192791-11-guoren@kernel.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20230802164701.192791-1-guoren@kernel.org> References: <20230802164701.192791-1-guoren@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Guo Ren Add the files functions needed to support the SBI PVLOCK (paravirt qspinlock kick_cpu) extension. This is a preparation for the next core implementation of kick_cpu. Signed-off-by: Guo Ren Signed-off-by: Guo Ren --- arch/riscv/include/asm/kvm_vcpu_sbi.h | 1 + arch/riscv/include/uapi/asm/kvm.h | 1 + arch/riscv/kvm/Makefile | 1 + arch/riscv/kvm/vcpu_sbi.c | 4 +++ arch/riscv/kvm/vcpu_sbi_pvlock.c | 38 +++++++++++++++++++++++++++ 5 files changed, 45 insertions(+) create mode 100644 arch/riscv/kvm/vcpu_sbi_pvlock.c diff --git a/arch/riscv/include/asm/kvm_vcpu_sbi.h b/arch/riscv/include/asm/kvm_vcpu_sbi.h index cdcf0ff07be7..7b4d60b54d7e 100644 --- a/arch/riscv/include/asm/kvm_vcpu_sbi.h +++ b/arch/riscv/include/asm/kvm_vcpu_sbi.h @@ -71,6 +71,7 @@ extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_srst; extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_hsm; extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_experimental; extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_vendor; +extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_pvlock; #ifdef CONFIG_RISCV_PMU_SBI extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_pmu; diff --git a/arch/riscv/include/uapi/asm/kvm.h b/arch/riscv/include/uapi/asm/kvm.h index 930fdc4101cd..e2100f994854 100644 --- a/arch/riscv/include/uapi/asm/kvm.h +++ b/arch/riscv/include/uapi/asm/kvm.h @@ -141,6 +141,7 @@ enum KVM_RISCV_SBI_EXT_ID { KVM_RISCV_SBI_EXT_PMU, KVM_RISCV_SBI_EXT_EXPERIMENTAL, KVM_RISCV_SBI_EXT_VENDOR, + KVM_RISCV_SBI_EXT_PVLOCK, KVM_RISCV_SBI_EXT_MAX, }; diff --git a/arch/riscv/kvm/Makefile b/arch/riscv/kvm/Makefile index fee0671e2dc1..c704da7b0a42 100644 --- a/arch/riscv/kvm/Makefile +++ b/arch/riscv/kvm/Makefile @@ -25,6 +25,7 @@ kvm-$(CONFIG_RISCV_SBI_V01) += vcpu_sbi_v01.o kvm-y += vcpu_sbi_base.o kvm-y += vcpu_sbi_replace.o kvm-y += vcpu_sbi_hsm.o +kvm-y += vcpu_sbi_pvlock.o kvm-y += vcpu_timer.o kvm-$(CONFIG_RISCV_PMU_SBI) += vcpu_pmu.o vcpu_sbi_pmu.o kvm-y += aia.o diff --git a/arch/riscv/kvm/vcpu_sbi.c b/arch/riscv/kvm/vcpu_sbi.c index 7b46e04fb667..ea225d48edb2 100644 --- a/arch/riscv/kvm/vcpu_sbi.c +++ b/arch/riscv/kvm/vcpu_sbi.c @@ -74,6 +74,10 @@ static const struct kvm_riscv_sbi_extension_entry sbi_ext[] = { .ext_idx = KVM_RISCV_SBI_EXT_VENDOR, .ext_ptr = &vcpu_sbi_ext_vendor, }, + { + .ext_idx = KVM_RISCV_SBI_EXT_PVLOCK, + .ext_ptr = &vcpu_sbi_ext_pvlock, + }, }; void kvm_riscv_vcpu_sbi_forward(struct kvm_vcpu *vcpu, struct kvm_run *run) diff --git a/arch/riscv/kvm/vcpu_sbi_pvlock.c b/arch/riscv/kvm/vcpu_sbi_pvlock.c new file mode 100644 index 000000000000..544a456c5041 --- /dev/null +++ b/arch/riscv/kvm/vcpu_sbi_pvlock.c @@ -0,0 +1,38 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (c), 2023 Alibaba Cloud + * + * Authors: + * Guo Ren + */ + +#include +#include +#include +#include +#include + +static int kvm_sbi_ext_pvlock_handler(struct kvm_vcpu *vcpu, struct kvm_run *run, + struct kvm_vcpu_sbi_return *retdata) +{ + int ret = 0; + struct kvm_cpu_context *cp = &vcpu->arch.guest_context; + unsigned long funcid = cp->a6; + + switch (funcid) { + case SBI_EXT_PVLOCK_KICK_CPU: + break; + default: + ret = SBI_ERR_NOT_SUPPORTED; + } + + retdata->err_val = ret; + + return 0; +} + +const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_pvlock = { + .extid_start = SBI_EXT_PVLOCK, + .extid_end = SBI_EXT_PVLOCK, + .handler = kvm_sbi_ext_pvlock_handler, +}; From patchwork Wed Aug 2 16:46:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guo Ren X-Patchwork-Id: 13338536 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 004C8C04A6A for ; Wed, 2 Aug 2023 16:51:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231867AbjHBQvd (ORCPT ); Wed, 2 Aug 2023 12:51:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49562 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231452AbjHBQvM (ORCPT ); Wed, 2 Aug 2023 12:51:12 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 71C283C2F; Wed, 2 Aug 2023 09:50:42 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 24AB46194E; Wed, 2 Aug 2023 16:50:40 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8C33EC433C7; Wed, 2 Aug 2023 16:50:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1690995039; bh=ZxSSeBb/S16STGNIjLgLnRd4gdTK7GWBFlxeOcsXSwE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Re6hHhbVV1BPv0jkxSRI9lbIULPSLrArCk6YQsxBcILuiJJmeoU/B3uTrAvZW8Hsd NESYQvpaFa3FeTrzeq5alcZ+8qf3hCjwa1dYlRqVehWgABTIjXr0SOjxmkO5qRpSNq 1qIXDIaUuS+1gRLf0ruSHeazVgAk3gjiR9cBr17jFfhwbqsItlZSZChiEXW9Aeudck ng8PI4CEgU7l6nTNJRMMj0DoG12Gwuhgv8z9TUe+XHbdIBKHy1e11wN96y5M4+BC+R XiHmg97d52zUNBHlHhFRH+ttLsm02XcUzSw/7ie2wcPZp3vbS+OXOV8lwqhgE8Ytdi cNN+zltkvBpng== From: guoren@kernel.org To: paul.walmsley@sifive.com, anup@brainfault.org, peterz@infradead.org, mingo@redhat.com, will@kernel.org, palmer@rivosinc.com, longman@redhat.com, boqun.feng@gmail.com, tglx@linutronix.de, paulmck@kernel.org, rostedt@goodmis.org, rdunlap@infradead.org, catalin.marinas@arm.com, conor.dooley@microchip.com, xiaoguang.xing@sophgo.com, bjorn@rivosinc.com, alexghiti@rivosinc.com, keescook@chromium.org, greentime.hu@sifive.com, ajones@ventanamicro.com, jszhang@kernel.org, wefu@redhat.com, wuwei2016@iscas.ac.cn Cc: linux-arch@vger.kernel.org, linux-riscv@lists.infradead.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-csky@vger.kernel.org, Guo Ren , Guo Ren Subject: [PATCH V10 11/19] RISC-V: paravirt: pvqspinlock: KVM: Implement kvm_sbi_ext_pvlock_kick_cpu() Date: Wed, 2 Aug 2023 12:46:53 -0400 Message-Id: <20230802164701.192791-12-guoren@kernel.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20230802164701.192791-1-guoren@kernel.org> References: <20230802164701.192791-1-guoren@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Guo Ren We only need to call the kvm_vcpu_kick() and bring target_vcpu from the halt state. No irq raised, no other request, just a pure vcpu_kick. Signed-off-by: Guo Ren Signed-off-by: Guo Ren --- arch/riscv/kvm/vcpu_sbi_pvlock.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/arch/riscv/kvm/vcpu_sbi_pvlock.c b/arch/riscv/kvm/vcpu_sbi_pvlock.c index 544a456c5041..914fc58aedfe 100644 --- a/arch/riscv/kvm/vcpu_sbi_pvlock.c +++ b/arch/riscv/kvm/vcpu_sbi_pvlock.c @@ -12,6 +12,24 @@ #include #include +static int kvm_sbi_ext_pvlock_kick_cpu(struct kvm_vcpu *vcpu) +{ + struct kvm_cpu_context *cp = &vcpu->arch.guest_context; + struct kvm *kvm = vcpu->kvm; + struct kvm_vcpu *target; + + target = kvm_get_vcpu_by_id(kvm, cp->a0); + if (!target) + return SBI_ERR_INVALID_PARAM; + + kvm_vcpu_kick(target); + + if (READ_ONCE(target->ready)) + kvm_vcpu_yield_to(target); + + return SBI_SUCCESS; +} + static int kvm_sbi_ext_pvlock_handler(struct kvm_vcpu *vcpu, struct kvm_run *run, struct kvm_vcpu_sbi_return *retdata) { @@ -21,6 +39,7 @@ static int kvm_sbi_ext_pvlock_handler(struct kvm_vcpu *vcpu, struct kvm_run *run switch (funcid) { case SBI_EXT_PVLOCK_KICK_CPU: + ret = kvm_sbi_ext_pvlock_kick_cpu(vcpu); break; default: ret = SBI_ERR_NOT_SUPPORTED; From patchwork Wed Aug 2 16:46:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guo Ren X-Patchwork-Id: 13338537 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 30B53C04FDF for ; Wed, 2 Aug 2023 16:51:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229987AbjHBQvw (ORCPT ); Wed, 2 Aug 2023 12:51:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49728 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233215AbjHBQvZ (ORCPT ); Wed, 2 Aug 2023 12:51:25 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8762830ED; Wed, 2 Aug 2023 09:50:56 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id E8EAE61A43; Wed, 2 Aug 2023 16:50:55 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8C843C433C8; Wed, 2 Aug 2023 16:50:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1690995055; bh=Ex6OYX3JGNnUHd85CnRKWsm8lmD+bGAwdtplvxxj8PY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=um7BNA7itHG6dHP1/TJ2qBnWLjq8LpHkLPRfh1qtGRbNzY7ZAXLFTcYLA+tJ6mLLP jKEjkHopqqvN+oHeq15IcuNa1QsUbkSzrWjKAT649fqQW0PuY/YQ88ATN23zqp/W/7 ThJSk6zBrbelucVEcgJ2wvGt3ZI1/ZpuTm/yvaQP8PajaVwGcVRMnHlM+LlFwPd9XO uQwO0jbLuHeszx11KXEt6scpahkYLC+c19wzJR8sVOlcF/PbRo/yUHnKdSgpxrwXmH 3zvGNqg+OtbF1LKbaLWUWXqGWaXGQEWygJ9D7So5hpuqI7YGXBR91UK6+tb74VvBC9 Hz1m3QjgRrJ+Q== From: guoren@kernel.org To: paul.walmsley@sifive.com, anup@brainfault.org, peterz@infradead.org, mingo@redhat.com, will@kernel.org, palmer@rivosinc.com, longman@redhat.com, boqun.feng@gmail.com, tglx@linutronix.de, paulmck@kernel.org, rostedt@goodmis.org, rdunlap@infradead.org, catalin.marinas@arm.com, conor.dooley@microchip.com, xiaoguang.xing@sophgo.com, bjorn@rivosinc.com, alexghiti@rivosinc.com, keescook@chromium.org, greentime.hu@sifive.com, ajones@ventanamicro.com, jszhang@kernel.org, wefu@redhat.com, wuwei2016@iscas.ac.cn Cc: linux-arch@vger.kernel.org, linux-riscv@lists.infradead.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-csky@vger.kernel.org, Guo Ren , Guo Ren Subject: [PATCH V10 12/19] RISC-V: paravirt: pvqspinlock: Add nopvspin kernel parameter Date: Wed, 2 Aug 2023 12:46:54 -0400 Message-Id: <20230802164701.192791-13-guoren@kernel.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20230802164701.192791-1-guoren@kernel.org> References: <20230802164701.192791-1-guoren@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Guo Ren Disables the qspinlock slow path using PV optimizations which allow the hypervisor to 'idle' the guest on lock contention. Signed-off-by: Guo Ren Signed-off-by: Guo Ren --- Documentation/admin-guide/kernel-parameters.txt | 2 +- arch/riscv/kernel/paravirt.c | 13 +++++++++++++ 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index de6b7ee752cd..1a8878f6bfbd 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -3820,7 +3820,7 @@ as generic guest with no PV drivers. Currently support XEN HVM, KVM, HYPER_V and VMWARE guest. - nopvspin [X86,XEN,KVM] + nopvspin [X86,XEN,KVM,RISC-V] Disables the qspinlock slow path using PV optimizations which allow the hypervisor to 'idle' the guest on lock contention. diff --git a/arch/riscv/kernel/paravirt.c b/arch/riscv/kernel/paravirt.c index 1bacb2cf3872..b55c3d3c0c17 100644 --- a/arch/riscv/kernel/paravirt.c +++ b/arch/riscv/kernel/paravirt.c @@ -165,8 +165,21 @@ DEFINE_STATIC_CALL(pv_queued_spin_unlock, native_queued_spin_unlock); EXPORT_SYMBOL(__SCK__pv_queued_spin_lock_slowpath); EXPORT_SYMBOL(__SCK__pv_queued_spin_unlock); +static bool nopvspin; +static __init int parse_nopvspin(char *arg) +{ + nopvspin = true; + return 0; +} +early_param("nopvspin", parse_nopvspin); + void __init pv_qspinlock_init(void) { + if (nopvspin) { + pr_info("PV qspinlocks disabled\n"); + return; + } + if (num_possible_cpus() == 1) return; From patchwork Wed Aug 2 16:46:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guo Ren X-Patchwork-Id: 13338560 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4259BC04E69 for ; Wed, 2 Aug 2023 16:52:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231209AbjHBQwK (ORCPT ); Wed, 2 Aug 2023 12:52:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49476 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232949AbjHBQvn (ORCPT ); Wed, 2 Aug 2023 12:51:43 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DDD1E2D78; Wed, 2 Aug 2023 09:51:11 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 9B0ED61908; Wed, 2 Aug 2023 16:51:11 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 58DEEC433C7; Wed, 2 Aug 2023 16:50:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1690995071; bh=dxzBeiuyveM1CqbBbR/OUAZCLNjbox3EtC7Uc1CLsFA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=AZHg3j2gpQXq8yxOAM4GpN9PE3ejI3b4vk7YLvJyuGpx3rdDINpsU3MdseCcbmuc9 QFjIz2nQUHLERtIuyMiIZFOPxJOyltQOjBIm+uKoZLr4GBcGwf970LhFkTKS/5LJ5w gGQAYkOgE5T7IYT+3XRPczF8PjEUTCKS1/6GOkUqXykTfoalRJoXjx1U/TgNUacsoi K1Uq7YIwnG7lsoul6l8J6xnsefYP2GBpMemmAjzPW5GB5pQNIlZ4eo87p2NHZ/u+hI 7nSLWqNDPTXWQUuyUJyNuLQdRx/QbRy4+Kcq273fYGc7AFHKfrHnPLdZM4nvRSiG44 yEIdPpvksg4kw== From: guoren@kernel.org To: paul.walmsley@sifive.com, anup@brainfault.org, peterz@infradead.org, mingo@redhat.com, will@kernel.org, palmer@rivosinc.com, longman@redhat.com, boqun.feng@gmail.com, tglx@linutronix.de, paulmck@kernel.org, rostedt@goodmis.org, rdunlap@infradead.org, catalin.marinas@arm.com, conor.dooley@microchip.com, xiaoguang.xing@sophgo.com, bjorn@rivosinc.com, alexghiti@rivosinc.com, keescook@chromium.org, greentime.hu@sifive.com, ajones@ventanamicro.com, jszhang@kernel.org, wefu@redhat.com, wuwei2016@iscas.ac.cn Cc: linux-arch@vger.kernel.org, linux-riscv@lists.infradead.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-csky@vger.kernel.org, Guo Ren , Guo Ren Subject: [PATCH V10 13/19] RISC-V: paravirt: pvqspinlock: Remove unnecessary definitions of cmpxchg & xchg Date: Wed, 2 Aug 2023 12:46:55 -0400 Message-Id: <20230802164701.192791-14-guoren@kernel.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20230802164701.192791-1-guoren@kernel.org> References: <20230802164701.192791-1-guoren@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Guo Ren The custom xchg/cmpxchg_release macro definitions have no difference from the common code from the binary view. The xchg32/64 macro definitions have been abandoned in Linux. Thus, remove all of them. This is a preparation for the next cmpxchg_small & xchg8 patches. Signed-off-by: Guo Ren Signed-off-by: Guo Ren --- arch/riscv/include/asm/cmpxchg.h | 93 -------------------------------- 1 file changed, 93 deletions(-) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index d12231d752a4..3ab37215ed86 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -103,41 +103,6 @@ static inline ulong __xchg16_relaxed(ulong new, void *ptr) _x_, sizeof(*(ptr))); \ }) -#define __xchg_release(ptr, new, size) \ -({ \ - __typeof__(ptr) __ptr = (ptr); \ - __typeof__(new) __new = (new); \ - __typeof__(*(ptr)) __ret; \ - switch (size) { \ - case 4: \ - __asm__ __volatile__ ( \ - RISCV_RELEASE_BARRIER \ - " amoswap.w %0, %2, %1\n" \ - : "=r" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ - break; \ - case 8: \ - __asm__ __volatile__ ( \ - RISCV_RELEASE_BARRIER \ - " amoswap.d %0, %2, %1\n" \ - : "=r" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ - break; \ - default: \ - BUILD_BUG(); \ - } \ - __ret; \ -}) - -#define arch_xchg_release(ptr, x) \ -({ \ - __typeof__(*(ptr)) _x_ = (x); \ - (__typeof__(*(ptr))) __xchg_release((ptr), \ - _x_, sizeof(*(ptr))); \ -}) - #define __arch_xchg(ptr, new, size) \ ({ \ __typeof__(ptr) __ptr = (ptr); \ @@ -170,18 +135,6 @@ static inline ulong __xchg16_relaxed(ulong new, void *ptr) (__typeof__(*(ptr))) __arch_xchg((ptr), _x_, sizeof(*(ptr))); \ }) -#define xchg32(ptr, x) \ -({ \ - BUILD_BUG_ON(sizeof(*(ptr)) != 4); \ - arch_xchg((ptr), (x)); \ -}) - -#define xchg64(ptr, x) \ -({ \ - BUILD_BUG_ON(sizeof(*(ptr)) != 8); \ - arch_xchg((ptr), (x)); \ -}) - /* * Atomic compare and exchange. Compare OLD with MEM, if identical, * store NEW in MEM. Return the initial value in MEM. Success is @@ -277,52 +230,6 @@ static inline ulong __xchg16_relaxed(ulong new, void *ptr) _o_, _n_, sizeof(*(ptr))); \ }) -#define __cmpxchg_release(ptr, old, new, size) \ -({ \ - __typeof__(ptr) __ptr = (ptr); \ - __typeof__(*(ptr)) __old = (old); \ - __typeof__(*(ptr)) __new = (new); \ - __typeof__(*(ptr)) __ret; \ - register unsigned int __rc; \ - switch (size) { \ - case 4: \ - __asm__ __volatile__ ( \ - RISCV_RELEASE_BARRIER \ - "0: lr.w %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc.w %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - "1:\n" \ - : "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \ - : "rJ" ((long)__old), "rJ" (__new) \ - : "memory"); \ - break; \ - case 8: \ - __asm__ __volatile__ ( \ - RISCV_RELEASE_BARRIER \ - "0: lr.d %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc.d %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - "1:\n" \ - : "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \ - : "rJ" (__old), "rJ" (__new) \ - : "memory"); \ - break; \ - default: \ - BUILD_BUG(); \ - } \ - __ret; \ -}) - -#define arch_cmpxchg_release(ptr, o, n) \ -({ \ - __typeof__(*(ptr)) _o_ = (o); \ - __typeof__(*(ptr)) _n_ = (n); \ - (__typeof__(*(ptr))) __cmpxchg_release((ptr), \ - _o_, _n_, sizeof(*(ptr))); \ -}) - #define __cmpxchg(ptr, old, new, size) \ ({ \ __typeof__(ptr) __ptr = (ptr); \ From patchwork Wed Aug 2 16:46:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guo Ren X-Patchwork-Id: 13338561 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E015DC001E0 for ; Wed, 2 Aug 2023 16:52:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229795AbjHBQwo (ORCPT ); Wed, 2 Aug 2023 12:52:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50172 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233051AbjHBQwQ (ORCPT ); Wed, 2 Aug 2023 12:52:16 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B81E33C34; Wed, 2 Aug 2023 09:51:39 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 5BD1C61A3E; Wed, 2 Aug 2023 16:51:27 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 13B42C433C8; Wed, 2 Aug 2023 16:51:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1690995086; bh=1/GsGHwmBv8uH+dMkqUv9IEb/BPlKAA9d7hlqgNtS/w=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=WjEKhwxTykGowm9ZIjdHGLeI/AhTS0fwaekb++vJl1EBhZj0uTbvwoTA6Nva8ivX/ VavcZY4KiNAy1NF8+diCDFt3/LnkTYMsCc5vgegL3MI9LGArnb96JGQtIZm9D6hCHV iy1r+Fg6640k5qONYfPBeFyx25cFaoF799+WquNgLGDLeus6Vzl0wEgr35NwYJzQTl xWLjI9wCXkHllFQXic4zh6NB3qBQ746warrABJTL9JMCj64UsNKKC8uJX2otK6yblb JxNUFElHUiNTPpBqwJCePYvh83fH5f/4baqwXva0qpsFD/AJy3kqRerGvW5Brm0E2Y M3uIfGIt6q+QA== From: guoren@kernel.org To: paul.walmsley@sifive.com, anup@brainfault.org, peterz@infradead.org, mingo@redhat.com, will@kernel.org, palmer@rivosinc.com, longman@redhat.com, boqun.feng@gmail.com, tglx@linutronix.de, paulmck@kernel.org, rostedt@goodmis.org, rdunlap@infradead.org, catalin.marinas@arm.com, conor.dooley@microchip.com, xiaoguang.xing@sophgo.com, bjorn@rivosinc.com, alexghiti@rivosinc.com, keescook@chromium.org, greentime.hu@sifive.com, ajones@ventanamicro.com, jszhang@kernel.org, wefu@redhat.com, wuwei2016@iscas.ac.cn Cc: linux-arch@vger.kernel.org, linux-riscv@lists.infradead.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-csky@vger.kernel.org, Guo Ren , Guo Ren Subject: [PATCH V10 14/19] RISC-V: paravirt: pvqspinlock: Add xchg8 & cmpxchg_small support Date: Wed, 2 Aug 2023 12:46:56 -0400 Message-Id: <20230802164701.192791-15-guoren@kernel.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20230802164701.192791-1-guoren@kernel.org> References: <20230802164701.192791-1-guoren@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Guo Ren The pvqspinlock needs additional sub-word atomic operations. Here is the list: - xchg8 (RCsc) - cmpxchg8/16_relaxed - cmpxchg8/16_release (Rcpc) - cmpxchg8_acquire (RCpc) - cmpxchg8 (RCsc) Although paravirt qspinlock doesn't have the native_qspinlock fairness, giving a strong forward progress guarantee to these atomic semantics could prevent unnecessary tries, which would cause cache line bouncing. Signed-off-by: Guo Ren Signed-off-by: Guo Ren --- arch/riscv/include/asm/cmpxchg.h | 177 +++++++++++++++++++++++++++++++ 1 file changed, 177 insertions(+) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index 3ab37215ed86..2fd797c04e7a 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -103,12 +103,37 @@ static inline ulong __xchg16_relaxed(ulong new, void *ptr) _x_, sizeof(*(ptr))); \ }) +static inline ulong __xchg8(ulong new, void *ptr) +{ + ulong ret, tmp; + ulong shif = ((ulong)ptr & 3) * 8; + ulong mask = 0xff << shif; + ulong *__ptr = (ulong *)((ulong)ptr & ~3); + + __asm__ __volatile__ ( + "0: lr.w %0, %2\n" + " and %1, %0, %z3\n" + " or %1, %1, %z4\n" + " sc.w.rl %1, %1, %2\n" + " bnez %1, 0b\n" + "fence w, rw\n" + : "=&r" (ret), "=&r" (tmp), "+A" (*__ptr) + : "rJ" (~mask), "rJ" (new << shif) + : "memory"); + + return (ulong)((ret & mask) >> shif); +} + #define __arch_xchg(ptr, new, size) \ ({ \ __typeof__(ptr) __ptr = (ptr); \ __typeof__(new) __new = (new); \ __typeof__(*(ptr)) __ret; \ switch (size) { \ + case 1: \ + __ret = (__typeof__(*(ptr))) \ + __xchg8((ulong)__new, __ptr); \ + break; \ case 4: \ __asm__ __volatile__ ( \ " amoswap.w.aqrl %0, %2, %1\n" \ @@ -140,6 +165,51 @@ static inline ulong __xchg16_relaxed(ulong new, void *ptr) * store NEW in MEM. Return the initial value in MEM. Success is * indicated by comparing RETURN with OLD. */ +static inline ulong __cmpxchg_small_relaxed(void *ptr, ulong old, + ulong new, ulong size) +{ + ulong shift; + ulong ret, mask, temp; + volatile ulong *ptr32; + + /* Mask inputs to the correct size. */ + mask = GENMASK((size * BITS_PER_BYTE) - 1, 0); + old &= mask; + new &= mask; + + /* + * Calculate a shift & mask that correspond to the value we wish to + * compare & exchange within the naturally aligned 4 byte integer + * that includes it. + */ + shift = (ulong)ptr & 0x3; + shift *= BITS_PER_BYTE; + old <<= shift; + new <<= shift; + mask <<= shift; + + /* + * Calculate a pointer to the naturally aligned 4 byte integer that + * includes our byte of interest, and load its value. + */ + ptr32 = (volatile ulong *)((ulong)ptr & ~0x3); + + __asm__ __volatile__ ( + "0: lr.w %0, %2\n" + " and %1, %0, %z3\n" + " bne %1, %z5, 1f\n" + " and %1, %0, %z4\n" + " or %1, %1, %z6\n" + " sc.w %1, %1, %2\n" + " bnez %1, 0b\n" + "1:\n" + : "=&r" (ret), "=&r" (temp), "+A" (*ptr32) + : "rJ" (mask), "rJ" (~mask), "rJ" (old), "rJ" (new) + : "memory"); + + return (ret & mask) >> shift; +} + #define __cmpxchg_relaxed(ptr, old, new, size) \ ({ \ __typeof__(ptr) __ptr = (ptr); \ @@ -148,6 +218,11 @@ static inline ulong __xchg16_relaxed(ulong new, void *ptr) __typeof__(*(ptr)) __ret; \ register unsigned int __rc; \ switch (size) { \ + case 1: \ + __ret = (__typeof__(*(ptr))) \ + __cmpxchg_small_relaxed(__ptr, (ulong)__old, \ + (ulong)__new, (ulong)size); \ + break; \ case 4: \ __asm__ __volatile__ ( \ "0: lr.w %0, %2\n" \ @@ -184,6 +259,52 @@ static inline ulong __xchg16_relaxed(ulong new, void *ptr) _o_, _n_, sizeof(*(ptr))); \ }) +static inline ulong __cmpxchg_small_acquire(void *ptr, ulong old, + ulong new, ulong size) +{ + ulong shift; + ulong ret, mask, temp; + volatile ulong *ptr32; + + /* Mask inputs to the correct size. */ + mask = GENMASK((size * BITS_PER_BYTE) - 1, 0); + old &= mask; + new &= mask; + + /* + * Calculate a shift & mask that correspond to the value we wish to + * compare & exchange within the naturally aligned 4 byte integer + * that includes it. + */ + shift = (ulong)ptr & 0x3; + shift *= BITS_PER_BYTE; + old <<= shift; + new <<= shift; + mask <<= shift; + + /* + * Calculate a pointer to the naturally aligned 4 byte integer that + * includes our byte of interest, and load its value. + */ + ptr32 = (volatile ulong *)((ulong)ptr & ~0x3); + + __asm__ __volatile__ ( + "0: lr.w %0, %2\n" + " and %1, %0, %z3\n" + " bne %1, %z5, 1f\n" + " and %1, %0, %z4\n" + " or %1, %1, %z6\n" + " sc.w %1, %1, %2\n" + " bnez %1, 0b\n" + RISCV_ACQUIRE_BARRIER + "1:\n" + : "=&r" (ret), "=&r" (temp), "+A" (*ptr32) + : "rJ" (mask), "rJ" (~mask), "rJ" (old), "rJ" (new) + : "memory"); + + return (ret & mask) >> shift; +} + #define __cmpxchg_acquire(ptr, old, new, size) \ ({ \ __typeof__(ptr) __ptr = (ptr); \ @@ -192,6 +313,12 @@ static inline ulong __xchg16_relaxed(ulong new, void *ptr) __typeof__(*(ptr)) __ret; \ register unsigned int __rc; \ switch (size) { \ + case 1: \ + case 2: \ + __ret = (__typeof__(*(ptr))) \ + __cmpxchg_small_acquire(__ptr, (ulong)__old, \ + (ulong)__new, (ulong)size); \ + break; \ case 4: \ __asm__ __volatile__ ( \ "0: lr.w %0, %2\n" \ @@ -230,6 +357,51 @@ static inline ulong __xchg16_relaxed(ulong new, void *ptr) _o_, _n_, sizeof(*(ptr))); \ }) +static inline ulong __cmpxchg_small(void *ptr, ulong old, + ulong new, ulong size) +{ + ulong shift; + ulong ret, mask, temp; + volatile ulong *ptr32; + + /* Mask inputs to the correct size. */ + mask = GENMASK((size * BITS_PER_BYTE) - 1, 0); + old &= mask; + new &= mask; + + /* + * Calculate a shift & mask that correspond to the value we wish to + * compare & exchange within the naturally aligned 4 byte integer + * that includes it. + */ + shift = (ulong)ptr & 0x3; + shift *= BITS_PER_BYTE; + old <<= shift; + new <<= shift; + mask <<= shift; + + /* + * Calculate a pointer to the naturally aligned 4 byte integer that + * includes our byte of interest, and load its value. + */ + ptr32 = (volatile ulong *)((ulong)ptr & ~0x3); + + __asm__ __volatile__ ( + "0: lr.w %0, %2\n" + " and %1, %0, %z3\n" + " bne %1, %z5, 1f\n" + " and %1, %0, %z4\n" + " or %1, %1, %z6\n" + " sc.w.rl %1, %1, %2\n" + " bnez %1, 0b\n" + " fence w, rw\n" + "1:\n" + : "=&r" (ret), "=&r" (temp), "+A" (*ptr32) + : "rJ" (mask), "rJ" (~mask), "rJ" (old), "rJ" (new) + : "memory"); + + return (ret & mask) >> shift; +} #define __cmpxchg(ptr, old, new, size) \ ({ \ __typeof__(ptr) __ptr = (ptr); \ @@ -238,6 +410,11 @@ static inline ulong __xchg16_relaxed(ulong new, void *ptr) __typeof__(*(ptr)) __ret; \ register unsigned int __rc; \ switch (size) { \ + case 1: \ + __ret = (__typeof__(*(ptr))) \ + __cmpxchg_small(__ptr, (ulong)__old, \ + (ulong)__new, (ulong)size); \ + break; \ case 4: \ __asm__ __volatile__ ( \ "0: lr.w %0, %2\n" \ From patchwork Wed Aug 2 16:46:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guo Ren X-Patchwork-Id: 13338563 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 16C78C0729B for ; Wed, 2 Aug 2023 16:53:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231874AbjHBQxP (ORCPT ); Wed, 2 Aug 2023 12:53:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50282 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232164AbjHBQwg (ORCPT ); Wed, 2 Aug 2023 12:52:36 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 033AE30E9; Wed, 2 Aug 2023 09:51:55 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 9F80261908; Wed, 2 Aug 2023 16:51:43 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B8079C433C8; Wed, 2 Aug 2023 16:51:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1690995103; bh=JHp6Qjv2j2L5yGDwaLMLoclwEyHeCzWtLnvdae52dY8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=XGxIQYG3X/Qxf61mdxd0nkNtxlOKIqUxTzKy2Iy5j9R40n6pMtZuxEMoTnmWzPFal 6erBZnxtXplixtxjAI14gHKWh7bWpa+VYQnSylkL8twtIZeBjO9M0mQN8lqqbhSkUm rM51X8dPdeolxLSAOeSQTFOT1WC9gSfUIiKTue36Dswah4389rsBzD7Mp67PcCVRZq nLiD0M2LIXm0Xib4XjMwxXVBCgkRVCNvq/y/XCkHgQA3b7flv1XlZgfv1nzyUnrkM3 W6T5DBCZCxBxhktENBH7MTh3tvUN99FApLkFhO6yITo5mBMZMsZIuBi7bEMHDk3ijR miiinner3tOnw== From: guoren@kernel.org To: paul.walmsley@sifive.com, anup@brainfault.org, peterz@infradead.org, mingo@redhat.com, will@kernel.org, palmer@rivosinc.com, longman@redhat.com, boqun.feng@gmail.com, tglx@linutronix.de, paulmck@kernel.org, rostedt@goodmis.org, rdunlap@infradead.org, catalin.marinas@arm.com, conor.dooley@microchip.com, xiaoguang.xing@sophgo.com, bjorn@rivosinc.com, alexghiti@rivosinc.com, keescook@chromium.org, greentime.hu@sifive.com, ajones@ventanamicro.com, jszhang@kernel.org, wefu@redhat.com, wuwei2016@iscas.ac.cn Cc: linux-arch@vger.kernel.org, linux-riscv@lists.infradead.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-csky@vger.kernel.org, Guo Ren , Guo Ren Subject: [PATCH V10 15/19] RISC-V: paravirt: pvqspinlock: Add SBI implementation Date: Wed, 2 Aug 2023 12:46:57 -0400 Message-Id: <20230802164701.192791-16-guoren@kernel.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20230802164701.192791-1-guoren@kernel.org> References: <20230802164701.192791-1-guoren@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Guo Ren Implement pv_kick with SBI implementation, and add SBI_EXT_PVLOCK extension detection. Signed-off-by: Guo Ren Signed-off-by: Guo Ren --- arch/riscv/include/asm/sbi.h | 6 ++++++ arch/riscv/kernel/paravirt.c | 7 ++++++- 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/arch/riscv/include/asm/sbi.h b/arch/riscv/include/asm/sbi.h index b7ced34b79a3..26b4ec039f32 100644 --- a/arch/riscv/include/asm/sbi.h +++ b/arch/riscv/include/asm/sbi.h @@ -31,6 +31,7 @@ enum sbi_ext_id { SBI_EXT_SRST = 0x53525354, SBI_EXT_PMU = 0x504D55, SBI_EXT_STA = 0x535441, + SBI_EXT_PVLOCK = 0xAB0401, /* Experimentals extensions must lie within this range */ SBI_EXT_EXPERIMENTAL_START = 0x08000000, @@ -244,6 +245,11 @@ enum sbi_pmu_ctr_type { /* Flags defined for counter stop function */ #define SBI_PMU_STOP_FLAG_RESET (1 << 0) +/* SBI PVLOCK (kick cpu out of wfi) */ +enum sbi_ext_pvlock_fid { + SBI_EXT_PVLOCK_KICK_CPU = 0, +}; + /* SBI STA (steal-time accounting) extension */ enum sbi_ext_sta_fid { SBI_EXT_STA_STEAL_TIME_SET_SHMEM = 0, diff --git a/arch/riscv/kernel/paravirt.c b/arch/riscv/kernel/paravirt.c index b55c3d3c0c17..564d64f11e4f 100644 --- a/arch/riscv/kernel/paravirt.c +++ b/arch/riscv/kernel/paravirt.c @@ -136,6 +136,8 @@ int __init pv_time_init(void) void pv_kick(int cpu) { + sbi_ecall(SBI_EXT_PVLOCK, SBI_EXT_PVLOCK_KICK_CPU, + cpuid_to_hartid_map(cpu), 0, 0, 0, 0, 0); return; } @@ -150,7 +152,7 @@ void pv_wait(u8 *ptr, u8 val) if (READ_ONCE(*ptr) != val) goto out; - /* wait_for_interrupt(); */ + wait_for_interrupt(); out: local_irq_restore(flags); } @@ -186,6 +188,9 @@ void __init pv_qspinlock_init(void) if(sbi_get_firmware_id() != SBI_EXT_BASE_IMPL_ID_KVM) return; + if (!sbi_probe_extension(SBI_EXT_PVLOCK)) + return; + pr_info("PV qspinlocks enabled\n"); __pv_init_lock_hash(); From patchwork Wed Aug 2 16:46:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guo Ren X-Patchwork-Id: 13338562 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C4DFFC04FE0 for ; Wed, 2 Aug 2023 16:53:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230307AbjHBQxO (ORCPT ); Wed, 2 Aug 2023 12:53:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50322 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233451AbjHBQwj (ORCPT ); Wed, 2 Aug 2023 12:52:39 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A2A1A3594; Wed, 2 Aug 2023 09:52:00 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 665F061A0D; Wed, 2 Aug 2023 16:51:59 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 17FD1C433C8; Wed, 2 Aug 2023 16:51:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1690995118; bh=//LTBV3xtswilX9AAt8SaQJo9J60NYzf9Q4p8qd3nq0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ec6jPCAuR3FS/NUe+ahVBDwwzJtZT0QQx4PS1ef6BZOb0kZD/ocsGIn873rcg1aCT MWEqlqSuKzbbkoYLVzOfNKiPREf+rHk7IrgySkp7nO5mnusalgpof9DWuqunetYYtm FaYi6LjVpusePPnVQMcBZPyO+I+2iyFf9vezvM+xOOsEFDvGSnVbvVxQNTA+WHssLO VJFkpJfObVNT7vGwiRAg2HTPJG3CUOEhrh+yQDjMaKZMvsSV1dujW2vwC814Y+dRvJ s9dCGuyyt9dVGl4bHJIrQXCVb8nVp8JDSzeLvom7te0x87nFPR7bmFcbiyikd54Nn+ G8435ns6CyMDw== From: guoren@kernel.org To: paul.walmsley@sifive.com, anup@brainfault.org, peterz@infradead.org, mingo@redhat.com, will@kernel.org, palmer@rivosinc.com, longman@redhat.com, boqun.feng@gmail.com, tglx@linutronix.de, paulmck@kernel.org, rostedt@goodmis.org, rdunlap@infradead.org, catalin.marinas@arm.com, conor.dooley@microchip.com, xiaoguang.xing@sophgo.com, bjorn@rivosinc.com, alexghiti@rivosinc.com, keescook@chromium.org, greentime.hu@sifive.com, ajones@ventanamicro.com, jszhang@kernel.org, wefu@redhat.com, wuwei2016@iscas.ac.cn Cc: linux-arch@vger.kernel.org, linux-riscv@lists.infradead.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-csky@vger.kernel.org, Guo Ren , Guo Ren Subject: [PATCH V10 16/19] RISC-V: paravirt: pvqspinlock: Add kconfig entry Date: Wed, 2 Aug 2023 12:46:58 -0400 Message-Id: <20230802164701.192791-17-guoren@kernel.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20230802164701.192791-1-guoren@kernel.org> References: <20230802164701.192791-1-guoren@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Guo Ren Add kconfig entry for paravirt_spinlock, an unfair qspinlock virtualization-friendly backend, by halting the virtual CPU rather than spinning. Signed-off-by: Guo Ren Signed-off-by: Guo Ren --- arch/riscv/Kconfig | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 42ae45c42b4d..13f345b54581 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -770,6 +770,7 @@ config RELOCATABLE config PARAVIRT bool "Enable paravirtualization code" depends on RISCV_SBI + select PARAVIRT_SPINLOCKS default y help This changes the kernel so it can modify itself when it is run @@ -788,6 +789,17 @@ config PARAVIRT_TIME_ACCOUNTING If in doubt, say N here. +config PARAVIRT_SPINLOCKS + bool "Paravirtualization layer for spinlocks" + depends on PARAVIRT && SMP + help + Paravirtualized spinlocks allow a unfair qspinlock to replace the + test-set kvm-guest virt spinlock implementation with something + virtualization-friendly, for example, halt the virtual CPU rather + than spinning. + + If you are unsure how to answer this question, answer Y. + endmenu # "Kernel features" menu "Boot options" From patchwork Wed Aug 2 16:46:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guo Ren X-Patchwork-Id: 13338564 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9D6DBC001E0 for ; Wed, 2 Aug 2023 16:53:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231960AbjHBQxR (ORCPT ); Wed, 2 Aug 2023 12:53:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49478 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233883AbjHBQwy (ORCPT ); Wed, 2 Aug 2023 12:52:54 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7D6E830DD; Wed, 2 Aug 2023 09:52:16 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 0229D615C2; Wed, 2 Aug 2023 16:52:16 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5C253C433C8; Wed, 2 Aug 2023 16:51:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1690995135; bh=hP8Bz7RWW4+moS/X2zthnCXyEaOfAgK0ZYvCm4g+kP4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=eJlajARTiQw0wmVH+8BmDyVwH/J6ZNHqhetqesuZrKodpf/s0TzQgKvDPVtSsU046 flSuT+qJyT9JBueeF390ynX7srNGmdjsZwCWsc02/FhDqCHVHTK2PwIFuTCNQRkZiw 0x9hXbKPu4Y4RRHsQATOFk4yd91Ucmb24v0LOiCbGwGaC5WFB/t4faWrvL5CxoO/6T ohqw0moDbQn0NuDLMmX/2bisP9DszzXcl5ls31EakEUd5oROo6MhlXwrfI1vdlA9d1 xjNil637eLOAwM4Lw13tawI1IjccWixfpEh/zJNhxsXrVOadWm+EN/jt2Ebra34x9u 9WUkOsu5POSoQ== From: guoren@kernel.org To: paul.walmsley@sifive.com, anup@brainfault.org, peterz@infradead.org, mingo@redhat.com, will@kernel.org, palmer@rivosinc.com, longman@redhat.com, boqun.feng@gmail.com, tglx@linutronix.de, paulmck@kernel.org, rostedt@goodmis.org, rdunlap@infradead.org, catalin.marinas@arm.com, conor.dooley@microchip.com, xiaoguang.xing@sophgo.com, bjorn@rivosinc.com, alexghiti@rivosinc.com, keescook@chromium.org, greentime.hu@sifive.com, ajones@ventanamicro.com, jszhang@kernel.org, wefu@redhat.com, wuwei2016@iscas.ac.cn Cc: linux-arch@vger.kernel.org, linux-riscv@lists.infradead.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-csky@vger.kernel.org, Guo Ren , Guo Ren Subject: [PATCH V10 17/19] RISC-V: paravirt: pvqspinlock: Add trace point for pv_kick/wait Date: Wed, 2 Aug 2023 12:46:59 -0400 Message-Id: <20230802164701.192791-18-guoren@kernel.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20230802164701.192791-1-guoren@kernel.org> References: <20230802164701.192791-1-guoren@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Guo Ren Add trace point for pv_kick/wait, here is the output: entries-in-buffer/entries-written: 33927/33927 #P:12 _-----=> irqs-off/BH-disabled / _----=> need-resched | / _---=> hardirq/softirq || / _--=> preempt-depth ||| / _-=> migrate-disable |||| / delay TASK-PID CPU# ||||| TIMESTAMP FUNCTION | | | ||||| | | sh-100 [001] d..2. 28.312294: pv_wait: cpu 1 out of wfi -0 [000] d.h4. 28.322030: pv_kick: cpu 0 kick target cpu 1 sh-100 [001] d..2. 30.982631: pv_wait: cpu 1 out of wfi -0 [000] d.h4. 30.993289: pv_kick: cpu 0 kick target cpu 1 sh-100 [002] d..2. 44.987573: pv_wait: cpu 2 out of wfi -0 [000] d.h4. 44.989000: pv_kick: cpu 0 kick target cpu 2 -0 [003] d.s3. 51.593978: pv_kick: cpu 3 kick target cpu 4 rcu_sched-15 [004] d..2. 51.595192: pv_wait: cpu 4 out of wfi lock_torture_wr-115 [004] ...2. 52.656482: pv_kick: cpu 4 kick target cpu 2 lock_torture_wr-113 [002] d..2. 52.659146: pv_wait: cpu 2 out of wfi lock_torture_wr-114 [008] d..2. 52.659507: pv_wait: cpu 8 out of wfi lock_torture_wr-114 [008] d..2. 52.663503: pv_wait: cpu 8 out of wfi lock_torture_wr-113 [002] ...2. 52.666128: pv_kick: cpu 2 kick target cpu 8 lock_torture_wr-114 [008] d..2. 52.667261: pv_wait: cpu 8 out of wfi lock_torture_wr-114 [009] .n.2. 53.141515: pv_kick: cpu 9 kick target cpu 11 lock_torture_wr-113 [002] d..2. 53.143339: pv_wait: cpu 2 out of wfi lock_torture_wr-116 [007] d..2. 53.143412: pv_wait: cpu 7 out of wfi lock_torture_wr-118 [000] d..2. 53.143457: pv_wait: cpu 0 out of wfi lock_torture_wr-115 [008] d..2. 53.143481: pv_wait: cpu 8 out of wfi lock_torture_wr-117 [011] d..2. 53.143522: pv_wait: cpu 11 out of wfi lock_torture_wr-117 [011] ...2. 53.143987: pv_kick: cpu 11 kick target cpu 8 lock_torture_wr-115 [008] ...2. 53.144269: pv_kick: cpu 8 kick target cpu 7 Signed-off-by: Guo Ren Signed-off-by: Guo Ren --- arch/riscv/kernel/paravirt.c | 8 +++ .../kernel/trace_events_filter_paravirt.h | 60 +++++++++++++++++++ 2 files changed, 68 insertions(+) create mode 100644 arch/riscv/kernel/trace_events_filter_paravirt.h diff --git a/arch/riscv/kernel/paravirt.c b/arch/riscv/kernel/paravirt.c index 564d64f11e4f..cc80e968ab13 100644 --- a/arch/riscv/kernel/paravirt.c +++ b/arch/riscv/kernel/paravirt.c @@ -134,10 +134,16 @@ int __init pv_time_init(void) #ifdef CONFIG_PARAVIRT_SPINLOCKS #include +#define CREATE_TRACE_POINTS +#include "trace_events_filter_paravirt.h" + void pv_kick(int cpu) { sbi_ecall(SBI_EXT_PVLOCK, SBI_EXT_PVLOCK_KICK_CPU, cpuid_to_hartid_map(cpu), 0, 0, 0, 0, 0); + + trace_pv_kick(smp_processor_id(), cpu); + return; } @@ -153,6 +159,8 @@ void pv_wait(u8 *ptr, u8 val) goto out; wait_for_interrupt(); + + trace_pv_wait(smp_processor_id()); out: local_irq_restore(flags); } diff --git a/arch/riscv/kernel/trace_events_filter_paravirt.h b/arch/riscv/kernel/trace_events_filter_paravirt.h new file mode 100644 index 000000000000..9ff5aa451b12 --- /dev/null +++ b/arch/riscv/kernel/trace_events_filter_paravirt.h @@ -0,0 +1,60 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c), 2023 Alibaba Cloud + * Authors: + * Guo Ren + */ +#undef TRACE_SYSTEM +#define TRACE_SYSTEM paravirt + +#if !defined(_TRACE_PARAVIRT_H) || defined(TRACE_HEADER_MULTI_READ) +#define _TRACE_PARAVIRT_H + +#include + +TRACE_EVENT(pv_kick, + TP_PROTO(int cpu, int target), + TP_ARGS(cpu, target), + + TP_STRUCT__entry( + __field(int, cpu) + __field(int, target) + ), + + TP_fast_assign( + __entry->cpu = cpu; + __entry->target = target; + ), + + TP_printk("cpu %d kick target cpu %d", + __entry->cpu, + __entry->target + ) +); + +TRACE_EVENT(pv_wait, + TP_PROTO(int cpu), + TP_ARGS(cpu), + + TP_STRUCT__entry( + __field(int, cpu) + ), + + TP_fast_assign( + __entry->cpu = cpu; + ), + + TP_printk("cpu %d out of wfi", + __entry->cpu + ) +); + +#endif /* _TRACE_PARAVIRT_H || TRACE_HEADER_MULTI_READ */ + +#undef TRACE_INCLUDE_PATH +#undef TRACE_INCLUDE_FILE +#define TRACE_INCLUDE_PATH ../../../arch/riscv/kernel/ +#define TRACE_INCLUDE_FILE trace_events_filter_paravirt + +/* This part must be outside protection */ +#include From patchwork Wed Aug 2 16:47:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guo Ren X-Patchwork-Id: 13338565 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BA47FC04E69 for ; Wed, 2 Aug 2023 16:53:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233215AbjHBQxc (ORCPT ); Wed, 2 Aug 2023 12:53:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50068 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234360AbjHBQxH (ORCPT ); Wed, 2 Aug 2023 12:53:07 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 464004224; Wed, 2 Aug 2023 09:52:32 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 847BE61924; Wed, 2 Aug 2023 16:52:31 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 66F4CC433C9; Wed, 2 Aug 2023 16:52:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1690995150; bh=I970M18cE0sneLmm5R/BlQhLI3AT81WZcHrpx7H84QU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=YYb/WMImTOTxkBja+DTFICqCdtWSOLtqAZ5e1F5s3EX0MaIYrFheo3mNiGqu1aaVC S6ilJfphcbY8Jphf2Q/QQar969Sfbq4aa/GKGRVw5XhOgm4OhP8/t5uVLhsZbYCdMt nrsC+aJmUnfMOFlO9tVcOAiqRXH2mFVgyBkVJGx27GUqib/X36BaB8BlvNPST0hC4w Cjy0dC8DVwqni8YakWrHZ7gpQsv3q0/mCzodTqg45S0c5vqJBCqgo1NiH6o1nHeO6e VzACfbqiOtwTGDMd4WwSyNp1MqFES+QdQ9MIUMZ8k96yQwGIRMklwWPOQUhdzRDdWR 5eh6ORHu6V/UQ== From: guoren@kernel.org To: paul.walmsley@sifive.com, anup@brainfault.org, peterz@infradead.org, mingo@redhat.com, will@kernel.org, palmer@rivosinc.com, longman@redhat.com, boqun.feng@gmail.com, tglx@linutronix.de, paulmck@kernel.org, rostedt@goodmis.org, rdunlap@infradead.org, catalin.marinas@arm.com, conor.dooley@microchip.com, xiaoguang.xing@sophgo.com, bjorn@rivosinc.com, alexghiti@rivosinc.com, keescook@chromium.org, greentime.hu@sifive.com, ajones@ventanamicro.com, jszhang@kernel.org, wefu@redhat.com, wuwei2016@iscas.ac.cn Cc: linux-arch@vger.kernel.org, linux-riscv@lists.infradead.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-csky@vger.kernel.org, Guo Ren , Guo Ren Subject: [PATCH V10 18/19] locking/qspinlock: Move pv_ops into x86 directory Date: Wed, 2 Aug 2023 12:47:00 -0400 Message-Id: <20230802164701.192791-19-guoren@kernel.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20230802164701.192791-1-guoren@kernel.org> References: <20230802164701.192791-1-guoren@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Guo Ren The pv_ops belongs to x86 custom infrastructure and cleans up the cna_configure_spin_lock_slowpath() with standard code. This is preparation for riscv support CNA qspoinlock. Signed-off-by: Guo Ren Signed-off-by: Guo Ren --- arch/x86/include/asm/qspinlock.h | 3 ++- arch/x86/kernel/alternative.c | 6 +++++- kernel/locking/qspinlock_cna.h | 14 ++++++-------- 3 files changed, 13 insertions(+), 10 deletions(-) diff --git a/arch/x86/include/asm/qspinlock.h b/arch/x86/include/asm/qspinlock.h index f48a2a250e57..100adad70bf5 100644 --- a/arch/x86/include/asm/qspinlock.h +++ b/arch/x86/include/asm/qspinlock.h @@ -28,7 +28,8 @@ static __always_inline u32 queued_fetch_set_pending_acquire(struct qspinlock *lo } #ifdef CONFIG_NUMA_AWARE_SPINLOCKS -extern void cna_configure_spin_lock_slowpath(void); +extern bool cna_configure_spin_lock_slowpath(void); +extern void __cna_queued_spin_lock_slowpath(struct qspinlock *lock, u32 val); #endif #ifdef CONFIG_PARAVIRT_SPINLOCKS diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index c36df5aa3ab1..68b7392016c3 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -1538,7 +1538,11 @@ void __init alternative_instructions(void) paravirt_set_cap(); #if defined(CONFIG_NUMA_AWARE_SPINLOCKS) - cna_configure_spin_lock_slowpath(); + if (pv_ops.lock.queued_spin_lock_slowpath == native_queued_spin_lock_slowpath) { + if (cna_configure_spin_lock_slowpath()) + pv_ops.lock.queued_spin_lock_slowpath = + __cna_queued_spin_lock_slowpath; + } #endif /* diff --git a/kernel/locking/qspinlock_cna.h b/kernel/locking/qspinlock_cna.h index 17d56c739e57..5e297dc687d9 100644 --- a/kernel/locking/qspinlock_cna.h +++ b/kernel/locking/qspinlock_cna.h @@ -406,20 +406,18 @@ void __cna_queued_spin_lock_slowpath(struct qspinlock *lock, u32 val); * multiple NUMA nodes in native environment, unless the user has * overridden this default behavior by setting the numa_spinlock flag. */ -void __init cna_configure_spin_lock_slowpath(void) +bool __init cna_configure_spin_lock_slowpath(void) { if (numa_spinlock_flag < 0) - return; + return false; - if (numa_spinlock_flag == 0 && (nr_node_ids < 2 || - pv_ops.lock.queued_spin_lock_slowpath != - native_queued_spin_lock_slowpath)) - return; + if (numa_spinlock_flag == 0 && nr_node_ids < 2) + return false; cna_init_nodes(); - pv_ops.lock.queued_spin_lock_slowpath = __cna_queued_spin_lock_slowpath; - pr_info("Enabling CNA spinlock\n"); + + return true; } From patchwork Wed Aug 2 16:47:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guo Ren X-Patchwork-Id: 13338566 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F35F2C04A6A for ; Wed, 2 Aug 2023 16:53:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232503AbjHBQxy (ORCPT ); Wed, 2 Aug 2023 12:53:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50148 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232258AbjHBQx0 (ORCPT ); Wed, 2 Aug 2023 12:53:26 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 575FE3C34; Wed, 2 Aug 2023 09:52:48 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 45269619CB; Wed, 2 Aug 2023 16:52:47 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id F1DBBC433C7; Wed, 2 Aug 2023 16:52:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1690995166; bh=qLA1GcmlLqlS/G9GD0l2jo6tFZm2YFKYDFEvmq8fWH0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=JbRaMX5uUEpDntEj6CsADqaoo1o4xqrMsuZKpnZpuW5guztpTHECFifz6plIF2Cu1 pAwRR3jKsuDGKUrKsJxrpPlwr62BM3MwxqtvFJHl6JAgVCy/PTe8vVbCP/v/thApXy SJEhn0XoAeXpAmovhqc4Gqxz6e/FvjMsSg1F9G3/e8C3Tz+3zFH67PmNy9Hs2iz1AX HuV0tdgWXi+sqV+Pf53/SuZXZdsE+0uuZbRCaIzwmjSCDMV9miPlEDcM3GfjH8Cj6v MoA6RM5MjLl4wlZDm2DxW7Y6GONu+AlThJhHsvT6rBGHa0YSTPiYOmT7lthvsH9tu0 oa46fefL6iKdg== From: guoren@kernel.org To: paul.walmsley@sifive.com, anup@brainfault.org, peterz@infradead.org, mingo@redhat.com, will@kernel.org, palmer@rivosinc.com, longman@redhat.com, boqun.feng@gmail.com, tglx@linutronix.de, paulmck@kernel.org, rostedt@goodmis.org, rdunlap@infradead.org, catalin.marinas@arm.com, conor.dooley@microchip.com, xiaoguang.xing@sophgo.com, bjorn@rivosinc.com, alexghiti@rivosinc.com, keescook@chromium.org, greentime.hu@sifive.com, ajones@ventanamicro.com, jszhang@kernel.org, wefu@redhat.com, wuwei2016@iscas.ac.cn Cc: linux-arch@vger.kernel.org, linux-riscv@lists.infradead.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-csky@vger.kernel.org, Guo Ren , Guo Ren Subject: [PATCH V10 19/19] locking/qspinlock: riscv: Add Compact NUMA-aware lock support Date: Wed, 2 Aug 2023 12:47:01 -0400 Message-Id: <20230802164701.192791-20-guoren@kernel.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20230802164701.192791-1-guoren@kernel.org> References: <20230802164701.192791-1-guoren@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Guo Ren Connect riscv to Compact NUMA-aware lock (CNA), which uses PRARAVIRT_SPINLOCKS static_call hooks. See numa_spinlock= of Documentation/admin-guide/kernel-parameters.txt for trying. Signed-off-by: Guo Ren Signed-off-by: Guo Ren --- arch/riscv/Kconfig | 18 ++++++++++++++++++ arch/riscv/include/asm/qspinlock.h | 5 +++++ arch/riscv/kernel/paravirt.c | 12 +++++++++++- 3 files changed, 34 insertions(+), 1 deletion(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 13f345b54581..ff483ccd26b9 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -800,6 +800,24 @@ config PARAVIRT_SPINLOCKS If you are unsure how to answer this question, answer Y. +config NUMA_AWARE_SPINLOCKS + bool "Numa-aware spinlocks" + depends on NUMA + depends on QUEUED_SPINLOCKS + depends on 64BIT + # For now, we depend on PARAVIRT_SPINLOCKS to make the patching work. + depends on PARAVIRT_SPINLOCKS + default y + help + Introduce NUMA (Non Uniform Memory Access) awareness into + the slow path of spinlocks. + + In this variant of qspinlock, the kernel will try to keep the lock + on the same node, thus reducing the number of remote cache misses, + while trading some of the short term fairness for better performance. + + Say N if you want absolute first come first serve fairness. + endmenu # "Kernel features" menu "Boot options" diff --git a/arch/riscv/include/asm/qspinlock.h b/arch/riscv/include/asm/qspinlock.h index 003e9560a0d1..e6f2a0621af0 100644 --- a/arch/riscv/include/asm/qspinlock.h +++ b/arch/riscv/include/asm/qspinlock.h @@ -12,6 +12,11 @@ void native_queued_spin_lock_slowpath(struct qspinlock *lock, u32 val); void __pv_init_lock_hash(void); void __pv_queued_spin_lock_slowpath(struct qspinlock *lock, u32 val); +#ifdef CONFIG_NUMA_AWARE_SPINLOCKS +bool cna_configure_spin_lock_slowpath(void); +void __cna_queued_spin_lock_slowpath(struct qspinlock *lock, u32 val); +#endif + static inline void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val) { static_call(pv_queued_spin_lock_slowpath)(lock, val); diff --git a/arch/riscv/kernel/paravirt.c b/arch/riscv/kernel/paravirt.c index cc80e968ab13..9466f693a98c 100644 --- a/arch/riscv/kernel/paravirt.c +++ b/arch/riscv/kernel/paravirt.c @@ -193,8 +193,10 @@ void __init pv_qspinlock_init(void) if (num_possible_cpus() == 1) return; - if(sbi_get_firmware_id() != SBI_EXT_BASE_IMPL_ID_KVM) + if(sbi_get_firmware_id() != SBI_EXT_BASE_IMPL_ID_KVM) { + goto cna_qspinlock; return; + } if (!sbi_probe_extension(SBI_EXT_PVLOCK)) return; @@ -204,5 +206,13 @@ void __init pv_qspinlock_init(void) static_call_update(pv_queued_spin_lock_slowpath, __pv_queued_spin_lock_slowpath); static_call_update(pv_queued_spin_unlock, __pv_queued_spin_unlock); + return; + +cna_qspinlock: +#ifdef CONFIG_NUMA_AWARE_SPINLOCKS + if (cna_configure_spin_lock_slowpath()) + static_call_update(pv_queued_spin_lock_slowpath, + __cna_queued_spin_lock_slowpath); +#endif } #endif