From patchwork Tue Nov 24 13:43:53 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guo Ren X-Patchwork-Id: 11929097 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 515D4C2D0E4 for ; Tue, 24 Nov 2020 13:52:03 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id BCCBA20888 for ; Tue, 24 Nov 2020 13:52:02 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="v/iMDl//"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="DlyXjC0w" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BCCBA20888 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:MIME-Version:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:In-Reply-To:References:List-Owner; bh=OXAFQl1I0nnAECWonqzjRFomBFF5F4P0HqmPKYIcXfU=; b=v/iMDl//tJBbU8/AFcYZzPL7SO BaZ87yV/NUZuOxIXj8SXAnftoJMgWlP+fGYtcBedhlG9blJ0LaPiyVw63cgf8fk8HWHVRQmOpwiA7 ZsLhTyHEleIGt5RKz7aKGOYuWCTQh2i0f5jri0KI6kCp9vd3KFUlk+bqwBL2EsNhX4l3eimqdI2if HSZbtfKuhoWm3qBVFgX59cVT57wN4KynS7I9W1wXrj3cgHGHy415cVTGSU//ByppdQKK+wnHeDjeA mk27am66upWxcTj3rpS+0KhgNSWB658QGPiOWXvxcc/q8nH+Qqq4bbPh+Hz0WI0+cYx9+cF5kNqjM ocCb+zeA==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1khYjZ-0006yB-Rl; Tue, 24 Nov 2020 13:51:53 +0000 Received: from mail.kernel.org ([198.145.29.99]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1khYcs-00036E-Ta for linux-riscv@lists.infradead.org; Tue, 24 Nov 2020 13:45:02 +0000 Received: from localhost.localdomain (unknown [42.120.72.69]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id D1B542076E; Tue, 24 Nov 2020 13:44:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1606225496; bh=V0Z2l62pZVOyWz2QxKXkhgR4d3BoO1ZllG7B/R8NhHc=; h=From:To:Cc:Subject:Date:From; b=DlyXjC0wBFsfrgvblLPR6rkcE6sa6zPCCs0HE+InMN9hdlOGzX1DU5KKA2uwQesy5 yFUkOT01rrOcUgisAlUh6oBMC6SsNEi02doaReIzRB9cN15GVTF/PdFgSEa2E3TfHI vK5pGMyhjDXhfeE7UywkSbB7W85ZqT5NvapllYPU= From: guoren@kernel.org To: peterz@infradead.org, arnd@arndb.de, palmerdabbelt@google.com, paul.walmsley@sifive.com, anup@brainfault.org Subject: [PATCH 1/5] riscv: Coding convention for xchg Date: Tue, 24 Nov 2020 13:43:53 +0000 Message-Id: <1606225437-22948-1-git-send-email-guoren@kernel.org> X-Mailer: git-send-email 2.7.4 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201124_084459_459337_78135906 X-CRM114-Status: GOOD ( 12.51 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Guo Ren , linux-kernel@vger.kernel.org, linux-csky@vger.kernel.org, Michael Clark , guoren@kernel.org, linux-riscv@lists.infradead.org MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Guo Ren This is prepare for QUEUED_SPINLOCKS which need xchg support short type value. - Remove unused codes (xchg32, xchg64, cmpxchg32 ...) - Combine xchg_relaxed, xchg_acquire, xchg_release into one asm - Make atomic.aq/rl with seperated fence acquire & release Signed-off-by: Guo Ren Cc: Peter Zijlstra Cc: Michael Clark --- arch/riscv/include/asm/atomic.h | 31 ++++----- arch/riscv/include/asm/cmpxchg.h | 135 +++------------------------------------ arch/riscv/include/asm/fence.h | 6 ++ 3 files changed, 30 insertions(+), 142 deletions(-) diff --git a/arch/riscv/include/asm/atomic.h b/arch/riscv/include/asm/atomic.h index 400a8c8..632fbe4 100644 --- a/arch/riscv/include/asm/atomic.h +++ b/arch/riscv/include/asm/atomic.h @@ -19,11 +19,8 @@ #include #include -#define __atomic_acquire_fence() \ - __asm__ __volatile__(RISCV_ACQUIRE_BARRIER "" ::: "memory") - -#define __atomic_release_fence() \ - __asm__ __volatile__(RISCV_RELEASE_BARRIER "" ::: "memory"); +#define __atomic_acquire_fence() __acquire_fence() +#define __atomic_release_fence() __release_fence() static __always_inline int atomic_read(const atomic_t *v) { @@ -242,58 +239,58 @@ static __always_inline s64 atomic64_fetch_add_unless(atomic64_t *v, s64 a, s64 u * atomic_{cmp,}xchg is required to have exactly the same ordering semantics as * {cmp,}xchg and the operations that return, so they need a full barrier. */ -#define ATOMIC_OP(c_t, prefix, size) \ +#define ATOMIC_OP(c_t, prefix) \ static __always_inline \ c_t atomic##prefix##_xchg_relaxed(atomic##prefix##_t *v, c_t n) \ { \ - return __xchg_relaxed(&(v->counter), n, size); \ + return xchg_relaxed(&(v->counter), n); \ } \ static __always_inline \ c_t atomic##prefix##_xchg_acquire(atomic##prefix##_t *v, c_t n) \ { \ - return __xchg_acquire(&(v->counter), n, size); \ + return xchg_acquire(&(v->counter), n); \ } \ static __always_inline \ c_t atomic##prefix##_xchg_release(atomic##prefix##_t *v, c_t n) \ { \ - return __xchg_release(&(v->counter), n, size); \ + return xchg_release(&(v->counter), n); \ } \ static __always_inline \ c_t atomic##prefix##_xchg(atomic##prefix##_t *v, c_t n) \ { \ - return __xchg(&(v->counter), n, size); \ + return xchg(&(v->counter), n); \ } \ static __always_inline \ c_t atomic##prefix##_cmpxchg_relaxed(atomic##prefix##_t *v, \ c_t o, c_t n) \ { \ - return __cmpxchg_relaxed(&(v->counter), o, n, size); \ + return cmpxchg_relaxed(&(v->counter), o, n); \ } \ static __always_inline \ c_t atomic##prefix##_cmpxchg_acquire(atomic##prefix##_t *v, \ c_t o, c_t n) \ { \ - return __cmpxchg_acquire(&(v->counter), o, n, size); \ + return cmpxchg_acquire(&(v->counter), o, n); \ } \ static __always_inline \ c_t atomic##prefix##_cmpxchg_release(atomic##prefix##_t *v, \ c_t o, c_t n) \ { \ - return __cmpxchg_release(&(v->counter), o, n, size); \ + return cmpxchg_release(&(v->counter), o, n); \ } \ static __always_inline \ c_t atomic##prefix##_cmpxchg(atomic##prefix##_t *v, c_t o, c_t n) \ { \ - return __cmpxchg(&(v->counter), o, n, size); \ + return cmpxchg(&(v->counter), o, n); \ } #ifdef CONFIG_GENERIC_ATOMIC64 #define ATOMIC_OPS() \ - ATOMIC_OP(int, , 4) + ATOMIC_OP(int, ) #else #define ATOMIC_OPS() \ - ATOMIC_OP(int, , 4) \ - ATOMIC_OP(s64, 64, 8) + ATOMIC_OP(int, ) \ + ATOMIC_OP(s64, 64) #endif ATOMIC_OPS() diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index 262e5bb..5609185 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -44,118 +44,31 @@ _x_, sizeof(*(ptr))); \ }) -#define __xchg_acquire(ptr, new, size) \ -({ \ - __typeof__(ptr) __ptr = (ptr); \ - __typeof__(new) __new = (new); \ - __typeof__(*(ptr)) __ret; \ - switch (size) { \ - case 4: \ - __asm__ __volatile__ ( \ - " amoswap.w %0, %2, %1\n" \ - RISCV_ACQUIRE_BARRIER \ - : "=r" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ - break; \ - case 8: \ - __asm__ __volatile__ ( \ - " amoswap.d %0, %2, %1\n" \ - RISCV_ACQUIRE_BARRIER \ - : "=r" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ - break; \ - default: \ - BUILD_BUG(); \ - } \ - __ret; \ -}) - #define xchg_acquire(ptr, x) \ ({ \ - __typeof__(*(ptr)) _x_ = (x); \ - (__typeof__(*(ptr))) __xchg_acquire((ptr), \ - _x_, sizeof(*(ptr))); \ -}) - -#define __xchg_release(ptr, new, size) \ -({ \ - __typeof__(ptr) __ptr = (ptr); \ - __typeof__(new) __new = (new); \ __typeof__(*(ptr)) __ret; \ - switch (size) { \ - case 4: \ - __asm__ __volatile__ ( \ - RISCV_RELEASE_BARRIER \ - " amoswap.w %0, %2, %1\n" \ - : "=r" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ - break; \ - case 8: \ - __asm__ __volatile__ ( \ - RISCV_RELEASE_BARRIER \ - " amoswap.d %0, %2, %1\n" \ - : "=r" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ - break; \ - default: \ - BUILD_BUG(); \ - } \ + __typeof__(*(ptr)) _x_ = (x); \ + __ret = __xchg_relaxed((ptr), _x_, sizeof(*(ptr))); \ + __acquire_fence(); \ __ret; \ }) #define xchg_release(ptr, x) \ ({ \ __typeof__(*(ptr)) _x_ = (x); \ - (__typeof__(*(ptr))) __xchg_release((ptr), \ + __release_fence(); \ + (__typeof__(*(ptr))) __xchg_relaxed((ptr), \ _x_, sizeof(*(ptr))); \ }) -#define __xchg(ptr, new, size) \ -({ \ - __typeof__(ptr) __ptr = (ptr); \ - __typeof__(new) __new = (new); \ - __typeof__(*(ptr)) __ret; \ - switch (size) { \ - case 4: \ - __asm__ __volatile__ ( \ - " amoswap.w.aqrl %0, %2, %1\n" \ - : "=r" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ - break; \ - case 8: \ - __asm__ __volatile__ ( \ - " amoswap.d.aqrl %0, %2, %1\n" \ - : "=r" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ - break; \ - default: \ - BUILD_BUG(); \ - } \ - __ret; \ -}) - #define xchg(ptr, x) \ ({ \ + __typeof__(*(ptr)) __ret; \ __typeof__(*(ptr)) _x_ = (x); \ - (__typeof__(*(ptr))) __xchg((ptr), _x_, sizeof(*(ptr))); \ -}) - -#define xchg32(ptr, x) \ -({ \ - BUILD_BUG_ON(sizeof(*(ptr)) != 4); \ - xchg((ptr), (x)); \ -}) - -#define xchg64(ptr, x) \ -({ \ - BUILD_BUG_ON(sizeof(*(ptr)) != 8); \ - xchg((ptr), (x)); \ + __smp_mb(); \ + __ret = __xchg_relaxed((ptr), _x_, sizeof(*(ptr))); \ + __smp_mb(); \ + __ret; \ }) /* @@ -344,32 +257,4 @@ (__typeof__(*(ptr))) __cmpxchg((ptr), \ _o_, _n_, sizeof(*(ptr))); \ }) - -#define cmpxchg_local(ptr, o, n) \ - (__cmpxchg_relaxed((ptr), (o), (n), sizeof(*(ptr)))) - -#define cmpxchg32(ptr, o, n) \ -({ \ - BUILD_BUG_ON(sizeof(*(ptr)) != 4); \ - cmpxchg((ptr), (o), (n)); \ -}) - -#define cmpxchg32_local(ptr, o, n) \ -({ \ - BUILD_BUG_ON(sizeof(*(ptr)) != 4); \ - cmpxchg_relaxed((ptr), (o), (n)) \ -}) - -#define cmpxchg64(ptr, o, n) \ -({ \ - BUILD_BUG_ON(sizeof(*(ptr)) != 8); \ - cmpxchg((ptr), (o), (n)); \ -}) - -#define cmpxchg64_local(ptr, o, n) \ -({ \ - BUILD_BUG_ON(sizeof(*(ptr)) != 8); \ - cmpxchg_relaxed((ptr), (o), (n)); \ -}) - #endif /* _ASM_RISCV_CMPXCHG_H */ diff --git a/arch/riscv/include/asm/fence.h b/arch/riscv/include/asm/fence.h index 2b443a3..3832601 100644 --- a/arch/riscv/include/asm/fence.h +++ b/arch/riscv/include/asm/fence.h @@ -9,4 +9,10 @@ #define RISCV_RELEASE_BARRIER #endif +#define __acquire_fence() \ + __asm__ __volatile__(RISCV_ACQUIRE_BARRIER "" ::: "memory") + +#define __release_fence() \ + __asm__ __volatile__(RISCV_RELEASE_BARRIER "" ::: "memory") + #endif /* _ASM_RISCV_FENCE_H */ From patchwork Tue Nov 24 13:43:54 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guo Ren X-Patchwork-Id: 11929099 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 17153C6379D for ; Tue, 24 Nov 2020 13:52:05 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9ABFA20872 for ; Tue, 24 Nov 2020 13:52:04 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="OKOUmHah"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="1iUcbfV4" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9ABFA20872 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:MIME-Version:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:References:In-Reply-To:Message-Id:Date:Subject:To: From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=xeIsTSkRS1eL4sPnt1gqGNckKzLiwsEkVdoFdyzjMRg=; b=OKOUmHahbPdTSrgLnwgTFTQWCw ZWjCQhvkai/F36+C+I3/87I7XX/L0pO2QnPu5crDUfZ8Apn/Sa0QZvZmAqBfOT6L1fawx7mTRbj0P sBiwUdaRQ4bHXDyPb8Zb8djYh2y/aQo+iKG/+IIeZvVMIrRw8gARyItcKP/lzW3NpvxRakUBU/JGO 2WHJ/UA8nrO/xVKG7TogsJDF7bRH1qwRBAnezDqXXbOUzi9FtDBCfADzXEziPXVjmCwCSsb2U7A// kJnP7ajvXlo+FBjzlXZhwyx+eP5gIYMoEjlCaK1P8WAUWR7LvTeDwr2ZgxRwt/nR88UD5j2GnrAdJ fo5HoZ/w==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1khYjc-0006zd-A9; Tue, 24 Nov 2020 13:51:56 +0000 Received: from mail.kernel.org ([198.145.29.99]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1khYd0-0003Ae-TQ for linux-riscv@lists.infradead.org; Tue, 24 Nov 2020 13:45:15 +0000 Received: from localhost.localdomain (unknown [42.120.72.69]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id DA03F2067D; Tue, 24 Nov 2020 13:44:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1606225504; bh=LemYfSoLF+U2R4PPrq4UpblySoUjDJiNIGRdRoINvRw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=1iUcbfV4nZXIE8Drim85JLvqc6s6LMIsxUfvsT5qPY1LoRlgO9CRAs8CmVWmn8InS 7EUFR/G7VhEwesMMOO+MrKEqPBUOewAGRqkGcF7NmztTACwH7ZNz8a2wfLC6zvxLqb UohwFFAI1DOd7uIOFHs4aLMVlRYKd6VUXBczNNNs= From: guoren@kernel.org To: peterz@infradead.org, arnd@arndb.de, palmerdabbelt@google.com, paul.walmsley@sifive.com, anup@brainfault.org Subject: [PATCH 2/5] riscv: Add QUEUED_SPINLOCKS & QUEUED_RWLOCKS supported Date: Tue, 24 Nov 2020 13:43:54 +0000 Message-Id: <1606225437-22948-2-git-send-email-guoren@kernel.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1606225437-22948-1-git-send-email-guoren@kernel.org> References: <1606225437-22948-1-git-send-email-guoren@kernel.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201124_084507_216615_AE6598C2 X-CRM114-Status: GOOD ( 17.63 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Guo Ren , linux-kernel@vger.kernel.org, linux-csky@vger.kernel.org, Michael Clark , guoren@kernel.org, linux-riscv@lists.infradead.org MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Guo Ren riscv only support lr.wd/s(c).w(d) with word(double word) size & align access. There are not lr.h/sc.h instructions. But qspinlock.c need xchg with short type variable: xchg_tail -> xchg_releaxed(&lock->tail, ... typedef struct qspinlock { union { atomic_t val; /* * By using the whole 2nd least significant byte for the * pending bit, we can allow better optimization of the lock * acquisition for the pending bit holder. */ struct { u8 locked; u8 pending; }; struct { u16 locked_pending; u16 tail; /* half word*/ }; }; } arch_spinlock_t; So we add short emulation in xchg with word length and it only solve qspinlock's requirement. Michael have sent qspinlock before, ref to Link below. Link: https://lore.kernel.org/linux-riscv/20190211043829.30096-1-michaeljclark@mac.com/ Signed-off-by: Guo Ren Cc: Peter Zijlstra Cc: Michael Clark --- arch/riscv/Kconfig | 2 + arch/riscv/include/asm/Kbuild | 3 + arch/riscv/include/asm/cmpxchg.h | 36 +++++++++ arch/riscv/include/asm/spinlock.h | 126 +------------------------------- arch/riscv/include/asm/spinlock_types.h | 15 +--- 5 files changed, 46 insertions(+), 136 deletions(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 2d61c4c..d757ba4 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -36,6 +36,8 @@ config RISCV select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT if MMU select ARCH_WANT_FRAME_POINTERS select ARCH_WANT_HUGE_PMD_SHARE if 64BIT + select ARCH_USE_QUEUED_RWLOCKS + select ARCH_USE_QUEUED_SPINLOCKS select CLONE_BACKWARDS select CLINT_TIMER if !MMU select COMMON_CLK diff --git a/arch/riscv/include/asm/Kbuild b/arch/riscv/include/asm/Kbuild index 59dd7be..6f5f438 100644 --- a/arch/riscv/include/asm/Kbuild +++ b/arch/riscv/include/asm/Kbuild @@ -6,3 +6,6 @@ generic-y += kvm_para.h generic-y += local64.h generic-y += user.h generic-y += vmlinux.lds.h +generic-y += mcs_spinlock.h +generic-y += qrwlock.h +generic-y += qspinlock.h diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index 5609185..e178700 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -16,7 +16,43 @@ __typeof__(ptr) __ptr = (ptr); \ __typeof__(new) __new = (new); \ __typeof__(*(ptr)) __ret; \ + register unsigned long __rc, tmp, align, addr; \ switch (size) { \ + case 2: \ + align = ((unsigned long) __ptr & 0x3); \ + addr = ((unsigned long) __ptr & ~0x3); \ + if (align) { \ + __asm__ __volatile__ ( \ + "0: lr.w %0, 0(%z4)\n" \ + " move %1, %0\n" \ + " slli %1, %1, 16\n" \ + " srli %1, %1, 16\n" \ + " move %2, %z3\n" \ + " slli %2, %2, 16\n" \ + " or %1, %2, %1\n" \ + " sc.w %2, %1, 0(%z4)\n" \ + " bnez %2, 0b\n" \ + " srli %0, %0, 16\n" \ + : "=&r" (__ret), "=&r" (tmp), "=&r" (__rc) \ + : "rJ" (__new), "rJ"(addr) \ + : "memory"); \ + } else { \ + __asm__ __volatile__ ( \ + "0: lr.w %0, (%z4)\n" \ + " move %1, %0\n" \ + " srli %1, %1, 16\n" \ + " slli %1, %1, 16\n" \ + " move %2, %z3\n" \ + " or %1, %2, %1\n" \ + " sc.w %2, %1, 0(%z4)\n" \ + " bnez %2, 0b\n" \ + " slli %0, %0, 16\n" \ + " srli %0, %0, 16\n" \ + : "=&r" (__ret), "=&r" (tmp), "=&r" (__rc) \ + : "rJ" (__new), "rJ"(addr) \ + : "memory"); \ + } \ + break; \ case 4: \ __asm__ __volatile__ ( \ " amoswap.w %0, %2, %1\n" \ diff --git a/arch/riscv/include/asm/spinlock.h b/arch/riscv/include/asm/spinlock.h index f4f7fa1..b8deb3a 100644 --- a/arch/riscv/include/asm/spinlock.h +++ b/arch/riscv/include/asm/spinlock.h @@ -7,129 +7,7 @@ #ifndef _ASM_RISCV_SPINLOCK_H #define _ASM_RISCV_SPINLOCK_H -#include -#include -#include - -/* - * Simple spin lock operations. These provide no fairness guarantees. - */ - -/* FIXME: Replace this with a ticket lock, like MIPS. */ - -#define arch_spin_is_locked(x) (READ_ONCE((x)->lock) != 0) - -static inline void arch_spin_unlock(arch_spinlock_t *lock) -{ - smp_store_release(&lock->lock, 0); -} - -static inline int arch_spin_trylock(arch_spinlock_t *lock) -{ - int tmp = 1, busy; - - __asm__ __volatile__ ( - " amoswap.w %0, %2, %1\n" - RISCV_ACQUIRE_BARRIER - : "=r" (busy), "+A" (lock->lock) - : "r" (tmp) - : "memory"); - - return !busy; -} - -static inline void arch_spin_lock(arch_spinlock_t *lock) -{ - while (1) { - if (arch_spin_is_locked(lock)) - continue; - - if (arch_spin_trylock(lock)) - break; - } -} - -/***********************************************************/ - -static inline void arch_read_lock(arch_rwlock_t *lock) -{ - int tmp; - - __asm__ __volatile__( - "1: lr.w %1, %0\n" - " bltz %1, 1b\n" - " addi %1, %1, 1\n" - " sc.w %1, %1, %0\n" - " bnez %1, 1b\n" - RISCV_ACQUIRE_BARRIER - : "+A" (lock->lock), "=&r" (tmp) - :: "memory"); -} - -static inline void arch_write_lock(arch_rwlock_t *lock) -{ - int tmp; - - __asm__ __volatile__( - "1: lr.w %1, %0\n" - " bnez %1, 1b\n" - " li %1, -1\n" - " sc.w %1, %1, %0\n" - " bnez %1, 1b\n" - RISCV_ACQUIRE_BARRIER - : "+A" (lock->lock), "=&r" (tmp) - :: "memory"); -} - -static inline int arch_read_trylock(arch_rwlock_t *lock) -{ - int busy; - - __asm__ __volatile__( - "1: lr.w %1, %0\n" - " bltz %1, 1f\n" - " addi %1, %1, 1\n" - " sc.w %1, %1, %0\n" - " bnez %1, 1b\n" - RISCV_ACQUIRE_BARRIER - "1:\n" - : "+A" (lock->lock), "=&r" (busy) - :: "memory"); - - return !busy; -} - -static inline int arch_write_trylock(arch_rwlock_t *lock) -{ - int busy; - - __asm__ __volatile__( - "1: lr.w %1, %0\n" - " bnez %1, 1f\n" - " li %1, -1\n" - " sc.w %1, %1, %0\n" - " bnez %1, 1b\n" - RISCV_ACQUIRE_BARRIER - "1:\n" - : "+A" (lock->lock), "=&r" (busy) - :: "memory"); - - return !busy; -} - -static inline void arch_read_unlock(arch_rwlock_t *lock) -{ - __asm__ __volatile__( - RISCV_RELEASE_BARRIER - " amoadd.w x0, %1, %0\n" - : "+A" (lock->lock) - : "r" (-1) - : "memory"); -} - -static inline void arch_write_unlock(arch_rwlock_t *lock) -{ - smp_store_release(&lock->lock, 0); -} +#include +#include #endif /* _ASM_RISCV_SPINLOCK_H */ diff --git a/arch/riscv/include/asm/spinlock_types.h b/arch/riscv/include/asm/spinlock_types.h index f398e76..d033a97 100644 --- a/arch/riscv/include/asm/spinlock_types.h +++ b/arch/riscv/include/asm/spinlock_types.h @@ -6,20 +6,11 @@ #ifndef _ASM_RISCV_SPINLOCK_TYPES_H #define _ASM_RISCV_SPINLOCK_TYPES_H -#ifndef __LINUX_SPINLOCK_TYPES_H +#if !defined(__LINUX_SPINLOCK_TYPES_H) && !defined(_ASM_RISCV_SPINLOCK_H) # error "please don't include this file directly" #endif -typedef struct { - volatile unsigned int lock; -} arch_spinlock_t; - -#define __ARCH_SPIN_LOCK_UNLOCKED { 0 } - -typedef struct { - volatile unsigned int lock; -} arch_rwlock_t; - -#define __ARCH_RW_LOCK_UNLOCKED { 0 } +#include +#include #endif /* _ASM_RISCV_SPINLOCK_TYPES_H */ From patchwork Tue Nov 24 13:43:55 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guo Ren X-Patchwork-Id: 11929101 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-14.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,UNWANTED_LANGUAGE_BODY,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 93F08C56201 for ; Tue, 24 Nov 2020 13:52:07 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E50F920872 for ; Tue, 24 Nov 2020 13:52:06 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="DFf2IlgL"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="Fr62qrVm" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E50F920872 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:MIME-Version:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:References:In-Reply-To:Message-Id:Date:Subject:To: From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=+gyiiNQ0hp900r3vDwd3o3zeqShRt5vFdEUvL5AzKmI=; b=DFf2IlgLBeJNVg3gJ2OLk1rl5X YAHv2245OJULhTlW4JXI9YgaCx+0uzzKJcGcDPj41o8F7TCtHhFP2c9+7BrBXOlcKJGFCuHiFz/3g 8hZJ/VxlXbW4g9eDLqQQbH1SNLCnaR8Ya2/QQLNjRcD6LHnyioxpMZjsjaGwNTaqOYhUS9kjkbctN 209exCm7sJ7B4WxIq37/R2gl3saQKxpvOy6OCqEWYgjpNWXbXrnmMj33007u46Nf66+Z/iYquP+Ki vwzrfcJPb3Op7QKH9O5a1TI53VHiGVaGxIGWX2Hhytft4PgQ4aABVDtr1+KgcFMEkokBPQOABOZQ+ opnolvFA==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1khYje-000711-Ts; Tue, 24 Nov 2020 13:51:59 +0000 Received: from mail.kernel.org ([198.145.29.99]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1khYd8-0003EE-6L for linux-riscv@lists.infradead.org; Tue, 24 Nov 2020 13:45:19 +0000 Received: from localhost.localdomain (unknown [42.120.72.69]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id F3A502083E; Tue, 24 Nov 2020 13:45:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1606225511; bh=DtG+IMRw20y2hmtpwA24L+hLLydVfP+wjdec7CN4NLs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Fr62qrVmPcWWFBSTHa8Z+mPLD0Gpy6BOGwYb001UPRubUQ7CJ4svbYhk1yesBSxg6 fK6YbIZbLserD+kmm5GIVgzcn9ZS5RYwB8AKxfzqyA4btPWwu/LUrLRsvn0D6V7Slv k2w92oIx/T2QQRFoJAcNITvBpDOSJ7ROzmnX6xEQ= From: guoren@kernel.org To: peterz@infradead.org, arnd@arndb.de, palmerdabbelt@google.com, paul.walmsley@sifive.com, anup@brainfault.org Subject: [PATCH 3/5] csky: Remove simple spinlock implementation Date: Tue, 24 Nov 2020 13:43:55 +0000 Message-Id: <1606225437-22948-3-git-send-email-guoren@kernel.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1606225437-22948-1-git-send-email-guoren@kernel.org> References: <1606225437-22948-1-git-send-email-guoren@kernel.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201124_084514_567811_4D5E8C3F X-CRM114-Status: GOOD ( 15.23 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-riscv@lists.infradead.org, Guo Ren , guoren@kernel.org, linux-kernel@vger.kernel.org, linux-csky@vger.kernel.org MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Guo Ren There are two implementation of spinlock in csky: - simple one (NR_CPU = 1,2) - tick's one (NR_CPU = 3,4,5...) We needn't the baby codes of simple one now. Link: https://lore.kernel.org/linux-csky/20200807081253.GD2674@hirez.programming.kicks-ass.net/#t Signed-off-by: Guo Ren Cc: Peter Zijlstra --- arch/csky/Kconfig | 2 +- arch/csky/include/asm/spinlock.h | 164 --------------------------------- arch/csky/include/asm/spinlock_types.h | 10 -- 3 files changed, 1 insertion(+), 175 deletions(-) diff --git a/arch/csky/Kconfig b/arch/csky/Kconfig index 268fad5..14ee229 100644 --- a/arch/csky/Kconfig +++ b/arch/csky/Kconfig @@ -7,7 +7,7 @@ config CSKY select ARCH_HAS_SYNC_DMA_FOR_CPU select ARCH_HAS_SYNC_DMA_FOR_DEVICE select ARCH_USE_BUILTIN_BSWAP - select ARCH_USE_QUEUED_RWLOCKS if NR_CPUS>2 + select ARCH_USE_QUEUED_RWLOCKS select ARCH_WANT_FRAME_POINTERS if !CPU_CK610 select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT select COMMON_CLK diff --git a/arch/csky/include/asm/spinlock.h b/arch/csky/include/asm/spinlock.h index 7cf3f2b..9feb0fd 100644 --- a/arch/csky/include/asm/spinlock.h +++ b/arch/csky/include/asm/spinlock.h @@ -6,8 +6,6 @@ #include #include -#ifdef CONFIG_QUEUED_RWLOCKS - /* * Ticket-based spin-locking. */ @@ -91,166 +89,4 @@ static inline int arch_spin_is_contended(arch_spinlock_t *lock) /* See include/linux/spinlock.h */ #define smp_mb__after_spinlock() smp_mb() -#else /* CONFIG_QUEUED_RWLOCKS */ - -/* - * Test-and-set spin-locking. - */ -static inline void arch_spin_lock(arch_spinlock_t *lock) -{ - u32 *p = &lock->lock; - u32 tmp; - - asm volatile ( - "1: ldex.w %0, (%1) \n" - " bnez %0, 1b \n" - " movi %0, 1 \n" - " stex.w %0, (%1) \n" - " bez %0, 1b \n" - : "=&r" (tmp) - : "r"(p) - : "cc"); - smp_mb(); -} - -static inline void arch_spin_unlock(arch_spinlock_t *lock) -{ - smp_mb(); - WRITE_ONCE(lock->lock, 0); -} - -static inline int arch_spin_trylock(arch_spinlock_t *lock) -{ - u32 *p = &lock->lock; - u32 tmp; - - asm volatile ( - "1: ldex.w %0, (%1) \n" - " bnez %0, 2f \n" - " movi %0, 1 \n" - " stex.w %0, (%1) \n" - " bez %0, 1b \n" - " movi %0, 0 \n" - "2: \n" - : "=&r" (tmp) - : "r"(p) - : "cc"); - - if (!tmp) - smp_mb(); - - return !tmp; -} - -#define arch_spin_is_locked(x) (READ_ONCE((x)->lock) != 0) - -/* - * read lock/unlock/trylock - */ -static inline void arch_read_lock(arch_rwlock_t *lock) -{ - u32 *p = &lock->lock; - u32 tmp; - - asm volatile ( - "1: ldex.w %0, (%1) \n" - " blz %0, 1b \n" - " addi %0, 1 \n" - " stex.w %0, (%1) \n" - " bez %0, 1b \n" - : "=&r" (tmp) - : "r"(p) - : "cc"); - smp_mb(); -} - -static inline void arch_read_unlock(arch_rwlock_t *lock) -{ - u32 *p = &lock->lock; - u32 tmp; - - smp_mb(); - asm volatile ( - "1: ldex.w %0, (%1) \n" - " subi %0, 1 \n" - " stex.w %0, (%1) \n" - " bez %0, 1b \n" - : "=&r" (tmp) - : "r"(p) - : "cc"); -} - -static inline int arch_read_trylock(arch_rwlock_t *lock) -{ - u32 *p = &lock->lock; - u32 tmp; - - asm volatile ( - "1: ldex.w %0, (%1) \n" - " blz %0, 2f \n" - " addi %0, 1 \n" - " stex.w %0, (%1) \n" - " bez %0, 1b \n" - " movi %0, 0 \n" - "2: \n" - : "=&r" (tmp) - : "r"(p) - : "cc"); - - if (!tmp) - smp_mb(); - - return !tmp; -} - -/* - * write lock/unlock/trylock - */ -static inline void arch_write_lock(arch_rwlock_t *lock) -{ - u32 *p = &lock->lock; - u32 tmp; - - asm volatile ( - "1: ldex.w %0, (%1) \n" - " bnez %0, 1b \n" - " subi %0, 1 \n" - " stex.w %0, (%1) \n" - " bez %0, 1b \n" - : "=&r" (tmp) - : "r"(p) - : "cc"); - smp_mb(); -} - -static inline void arch_write_unlock(arch_rwlock_t *lock) -{ - smp_mb(); - WRITE_ONCE(lock->lock, 0); -} - -static inline int arch_write_trylock(arch_rwlock_t *lock) -{ - u32 *p = &lock->lock; - u32 tmp; - - asm volatile ( - "1: ldex.w %0, (%1) \n" - " bnez %0, 2f \n" - " subi %0, 1 \n" - " stex.w %0, (%1) \n" - " bez %0, 1b \n" - " movi %0, 0 \n" - "2: \n" - : "=&r" (tmp) - : "r"(p) - : "cc"); - - if (!tmp) - smp_mb(); - - return !tmp; -} - -#endif /* CONFIG_QUEUED_RWLOCKS */ #endif /* __ASM_CSKY_SPINLOCK_H */ diff --git a/arch/csky/include/asm/spinlock_types.h b/arch/csky/include/asm/spinlock_types.h index 88b8243..8ff0f6f 100644 --- a/arch/csky/include/asm/spinlock_types.h +++ b/arch/csky/include/asm/spinlock_types.h @@ -22,16 +22,6 @@ typedef struct { #define __ARCH_SPIN_LOCK_UNLOCKED { { 0 } } -#ifdef CONFIG_QUEUED_RWLOCKS #include -#else /* CONFIG_NR_CPUS > 2 */ - -typedef struct { - u32 lock; -} arch_rwlock_t; - -#define __ARCH_RW_LOCK_UNLOCKED { 0 } - -#endif /* CONFIG_QUEUED_RWLOCKS */ #endif /* __ASM_CSKY_SPINLOCK_TYPES_H */ From patchwork Tue Nov 24 13:43:56 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guo Ren X-Patchwork-Id: 11929103 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D04B2C56201 for ; Tue, 24 Nov 2020 13:52:18 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4B30520872 for ; Tue, 24 Nov 2020 13:52:18 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="lOYUgrlP"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="GeSbrNGV" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4B30520872 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:MIME-Version:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:References:In-Reply-To:Message-Id:Date:Subject:To: From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=WCOYnYkkzOUNSpRE1syOjcew4z1mN9zb4hyprwt/DYE=; b=lOYUgrlPA9cUDMANZd49Jpznl+ UgIyEwBRSNBkvIM9yl2HfII8GBxFU4zsumeR6u9zzSkbnpGL+mziVJho35K5cFpHezUjqwLvngJ2y ejtYiWK7mW2qkWDujaH3yiyCpOynAtTDPodwR68rhTiLcO62HehApdr3b+3O+ZP+8GixXaOgehKLr 4tlFOctd2TH+94rO9V7LBIapZvRaz+Gq/inU5KnOc+IYLsd+Yrdu2oJB5CfyOski76uq4hesRbs2y /n59NlsBOdk1vP3yDvVAN/qdPU6DQ5CbZ3mGw9wpBmra/DcqVJMIZVNEO/st859ei2KonJRQ84WIb sl2fUWaQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1khYjo-00077e-GG; Tue, 24 Nov 2020 13:52:08 +0000 Received: from mail.kernel.org ([198.145.29.99]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1khYdD-0003Hk-I1 for linux-riscv@lists.infradead.org; Tue, 24 Nov 2020 13:45:27 +0000 Received: from localhost.localdomain (unknown [42.120.72.69]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 3A1E020888; Tue, 24 Nov 2020 13:45:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1606225517; bh=RUSvMWORfq539a0TAZ04lnu9G2BdKaaWBYoxWuQSlSo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GeSbrNGVsY6eoOquzuMJjQ0JMFFagXr4dv+XnASZt9l+xgtyHZnHEwecU61ivi5qd 8RVRhlJ95JHo3MEkMlZJ1UI1Po7OZzzaLH3pX8w9xWI6zJ3RmmvxAPEJfVpGeIzJtp zgIXFXqz9FcTKA7H4CKyqNUF8EiTj5sulbdgU35g= From: guoren@kernel.org To: peterz@infradead.org, arnd@arndb.de, palmerdabbelt@google.com, paul.walmsley@sifive.com, anup@brainfault.org Subject: [PATCH 4/5] csky: Add QUEUED_SPINLOCKS supported Date: Tue, 24 Nov 2020 13:43:56 +0000 Message-Id: <1606225437-22948-4-git-send-email-guoren@kernel.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1606225437-22948-1-git-send-email-guoren@kernel.org> References: <1606225437-22948-1-git-send-email-guoren@kernel.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201124_084520_104265_4EB959BA X-CRM114-Status: GOOD ( 18.84 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Guo Ren , "Paul E . McKenney" , linux-kernel@vger.kernel.org, linux-csky@vger.kernel.org, guoren@kernel.org, linux-riscv@lists.infradead.org MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Guo Ren Abiv2 only support ldex.w/stex.w with word(double word) size & align access. There are not short type instructions. But qspinlock.c need xchg with short type variable: xchg_tail -> xchg_releaxed(&lock->tail, ... typedef struct qspinlock { union { atomic_t val; /* * By using the whole 2nd least significant byte for the * pending bit, we can allow better optimization of the lock * acquisition for the pending bit holder. */ struct { u8 locked; u8 pending; }; struct { u16 locked_pending; u16 tail; /* half word*/ }; }; } arch_spinlock_t; So we add short emulation in xchg with word length and it only solve qspinlock's requirement. Signed-off-by: Guo Ren Cc: Peter Zijlstra Cc: Paul E. McKenney Cc: Arnd Bergmann --- arch/csky/Kconfig | 1 + arch/csky/include/asm/Kbuild | 2 + arch/csky/include/asm/cmpxchg.h | 43 ++++++++++++++++-- arch/csky/include/asm/spinlock.h | 82 +--------------------------------- arch/csky/include/asm/spinlock_types.h | 18 +------- 5 files changed, 46 insertions(+), 100 deletions(-) diff --git a/arch/csky/Kconfig b/arch/csky/Kconfig index 14ee229..ac02b17 100644 --- a/arch/csky/Kconfig +++ b/arch/csky/Kconfig @@ -8,6 +8,7 @@ config CSKY select ARCH_HAS_SYNC_DMA_FOR_DEVICE select ARCH_USE_BUILTIN_BSWAP select ARCH_USE_QUEUED_RWLOCKS + select ARCH_USE_QUEUED_SPINLOCKS select ARCH_WANT_FRAME_POINTERS if !CPU_CK610 select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT select COMMON_CLK diff --git a/arch/csky/include/asm/Kbuild b/arch/csky/include/asm/Kbuild index 64876e59..f814d46 100644 --- a/arch/csky/include/asm/Kbuild +++ b/arch/csky/include/asm/Kbuild @@ -3,7 +3,9 @@ generic-y += asm-offsets.h generic-y += gpio.h generic-y += kvm_para.h generic-y += local64.h +generic-y += mcs_spinlock.h generic-y += qrwlock.h +generic-y += qspinlock.h generic-y += seccomp.h generic-y += user.h generic-y += vmlinux.lds.h diff --git a/arch/csky/include/asm/cmpxchg.h b/arch/csky/include/asm/cmpxchg.h index 8922453..ca03e90 100644 --- a/arch/csky/include/asm/cmpxchg.h +++ b/arch/csky/include/asm/cmpxchg.h @@ -12,9 +12,46 @@ extern void __bad_xchg(void); ({ \ __typeof__(ptr) __ptr = (ptr); \ __typeof__(new) __new = (new); \ - __typeof__(*(ptr)) __ret; \ - unsigned long tmp; \ + __typeof__(*(ptr)) __ret = 0; \ + unsigned long tmp, tmp2, align, addr; \ switch (size) { \ + case 2: \ + align = ((unsigned long) __ptr & 0x3); \ + addr = ((unsigned long) __ptr & ~0x3); \ + smp_mb(); \ + if (align) { \ + asm volatile ( \ + "1: ldex.w %0, (%4) \n" \ + " mov %1, %0 \n" \ + " lsli %1, 16 \n" \ + " lsri %1, 16 \n" \ + " mov %2, %3 \n" \ + " lsli %2, 16 \n" \ + " or %1, %2 \n" \ + " stex.w %1, (%4) \n" \ + " bez %1, 1b \n" \ + " lsri %0, 16 \n" \ + : "=&r" (__ret), "=&r" (tmp), \ + "=&r" (tmp2) \ + : "r" (__new), "r"(addr) \ + :); \ + } else { \ + asm volatile ( \ + "1: ldex.w %0, (%3) \n" \ + " mov %1, %0 \n" \ + " lsri %1, 16 \n" \ + " lsli %1, 16 \n" \ + " or %1, %2 \n" \ + " stex.w %1, (%3) \n" \ + " bez %1, 1b \n" \ + " lsli %0, 16 \n" \ + " lsri %0, 16 \n" \ + : "=&r" (__ret), "=&r" (tmp) \ + : "r" (__new), "r"(addr) \ + :); \ + } \ + smp_mb(); \ + break; \ case 4: \ smp_mb(); \ asm volatile ( \ @@ -41,7 +78,7 @@ extern void __bad_xchg(void); __typeof__(new) __new = (new); \ __typeof__(new) __tmp; \ __typeof__(old) __old = (old); \ - __typeof__(*(ptr)) __ret; \ + __typeof__(*(ptr)) __ret = 0; \ switch (size) { \ case 4: \ smp_mb(); \ diff --git a/arch/csky/include/asm/spinlock.h b/arch/csky/include/asm/spinlock.h index 9feb0fd..6d21bdb 100644 --- a/arch/csky/include/asm/spinlock.h +++ b/arch/csky/include/asm/spinlock.h @@ -3,88 +3,8 @@ #ifndef __ASM_CSKY_SPINLOCK_H #define __ASM_CSKY_SPINLOCK_H -#include -#include - -/* - * Ticket-based spin-locking. - */ -static inline void arch_spin_lock(arch_spinlock_t *lock) -{ - arch_spinlock_t lockval; - u32 ticket_next = 1 << TICKET_NEXT; - u32 *p = &lock->lock; - u32 tmp; - - asm volatile ( - "1: ldex.w %0, (%2) \n" - " mov %1, %0 \n" - " add %0, %3 \n" - " stex.w %0, (%2) \n" - " bez %0, 1b \n" - : "=&r" (tmp), "=&r" (lockval) - : "r"(p), "r"(ticket_next) - : "cc"); - - while (lockval.tickets.next != lockval.tickets.owner) - lockval.tickets.owner = READ_ONCE(lock->tickets.owner); - - smp_mb(); -} - -static inline int arch_spin_trylock(arch_spinlock_t *lock) -{ - u32 tmp, contended, res; - u32 ticket_next = 1 << TICKET_NEXT; - u32 *p = &lock->lock; - - do { - asm volatile ( - " ldex.w %0, (%3) \n" - " movi %2, 1 \n" - " rotli %1, %0, 16 \n" - " cmpne %1, %0 \n" - " bt 1f \n" - " movi %2, 0 \n" - " add %0, %0, %4 \n" - " stex.w %0, (%3) \n" - "1: \n" - : "=&r" (res), "=&r" (tmp), "=&r" (contended) - : "r"(p), "r"(ticket_next) - : "cc"); - } while (!res); - - if (!contended) - smp_mb(); - - return !contended; -} - -static inline void arch_spin_unlock(arch_spinlock_t *lock) -{ - smp_mb(); - WRITE_ONCE(lock->tickets.owner, lock->tickets.owner + 1); -} - -static inline int arch_spin_value_unlocked(arch_spinlock_t lock) -{ - return lock.tickets.owner == lock.tickets.next; -} - -static inline int arch_spin_is_locked(arch_spinlock_t *lock) -{ - return !arch_spin_value_unlocked(READ_ONCE(*lock)); -} - -static inline int arch_spin_is_contended(arch_spinlock_t *lock) -{ - struct __raw_tickets tickets = READ_ONCE(lock->tickets); - - return (tickets.next - tickets.owner) > 1; -} -#define arch_spin_is_contended arch_spin_is_contended - #include +#include /* See include/linux/spinlock.h */ #define smp_mb__after_spinlock() smp_mb() diff --git a/arch/csky/include/asm/spinlock_types.h b/arch/csky/include/asm/spinlock_types.h index 8ff0f6f..82f5fd5 100644 --- a/arch/csky/include/asm/spinlock_types.h +++ b/arch/csky/include/asm/spinlock_types.h @@ -3,25 +3,11 @@ #ifndef __ASM_CSKY_SPINLOCK_TYPES_H #define __ASM_CSKY_SPINLOCK_TYPES_H -#ifndef __LINUX_SPINLOCK_TYPES_H +#if !defined(__LINUX_SPINLOCK_TYPES_H) && !defined(__ASM_CSKY_SPINLOCK_H) # error "please don't include this file directly" #endif -#define TICKET_NEXT 16 - -typedef struct { - union { - u32 lock; - struct __raw_tickets { - /* little endian */ - u16 owner; - u16 next; - } tickets; - }; -} arch_spinlock_t; - -#define __ARCH_SPIN_LOCK_UNLOCKED { { 0 } } - +#include #include #endif /* __ASM_CSKY_SPINLOCK_TYPES_H */ From patchwork Tue Nov 24 13:43:57 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guo Ren X-Patchwork-Id: 11929105 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CAE4DC56201 for ; Tue, 24 Nov 2020 13:52:23 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3ABD220872 for ; Tue, 24 Nov 2020 13:52:23 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="z5vw/g4p"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="OWeVcruD" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3ABD220872 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:MIME-Version:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:References:In-Reply-To:Message-Id:Date:Subject:To: From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=BqGbOG2iEBzr2hojX890kEas9UgUSuzFhOCfXIbFtt8=; b=z5vw/g4p5xzqoNRmQa+T0GSkrn /71zykC2TGn+ZYorzV6n+wicSSc6C9HDBSe234plH/SME4k8uraHh1ptIJCwXOUVqaoiTzk1OOCTv aPRAgoNQcH1vje9ycHS3ftLLUie+VrHzp0s9/+2HVgYZl8trT+urtG5RDa01dVuCd+tQcCdDzB2dN x1AnR+JbSnUJ2KJDKaBwJ7cwhbSNM8yNXdk+4W4oT06PGL0hhEq2mo1bXYb6SeEBSDnbqIdHaDnTl peKYysHGvLNdDkJeWjeksi6zEg+Imtx9WlxZepLWqbHQTZ39xBkNPfO6FfCwB8E1Lr1IWnhgy0TBv MZpHbPEQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1khYjs-00079Y-Fh; Tue, 24 Nov 2020 13:52:12 +0000 Received: from mail.kernel.org ([198.145.29.99]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1khYdM-0003Mf-Ia for linux-riscv@lists.infradead.org; Tue, 24 Nov 2020 13:45:35 +0000 Received: from localhost.localdomain (unknown [42.120.72.69]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 8C06C2087D; Tue, 24 Nov 2020 13:45:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1606225526; bh=levK7wUVTU/CP5ZuAopBIMCjvkEhOFNbIvK0MW6TTKo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=OWeVcruDOlUtIfI0n7ZzMU2zPgvQdLwAOfe98mOnNadTtB3A8/V+xCqiryPqyrw92 kxi5lbehXyS1u7FIu9nPfJi6IoNbfFNIj82JJFRbhVxB1BTxIl9A7VVw7Cv2ydJtub f3UcNt6X9jLcrcqcNtAVm3sllMbhaDFiKNsbk1wE= From: guoren@kernel.org To: peterz@infradead.org, arnd@arndb.de, palmerdabbelt@google.com, paul.walmsley@sifive.com, anup@brainfault.org Subject: [PATCH 5/5] csky: Optimize atomic operations with correct barrier usage Date: Tue, 24 Nov 2020 13:43:57 +0000 Message-Id: <1606225437-22948-5-git-send-email-guoren@kernel.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1606225437-22948-1-git-send-email-guoren@kernel.org> References: <1606225437-22948-1-git-send-email-guoren@kernel.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201124_084529_171099_122CE4D9 X-CRM114-Status: GOOD ( 17.45 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Guo Ren , "Paul E . McKenney" , linux-kernel@vger.kernel.org, linux-csky@vger.kernel.org, guoren@kernel.org, linux-riscv@lists.infradead.org MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Guo Ren The implementation of csky atomic operations in the past was very rough. Frankly speaking, the implementation is wrong and limits hardware performance. Optimize the performance of atomic, spinlock, cmpxchg more fine-grained by increasing acquire/release memory barriers. Here are the details of the modification: - Add acquire/release barrier for cmpxchg.h. - Remove custom atomic.h implementations. Signed-off-by: Guo Ren Cc: Peter Zijlstra Cc: Paul E. McKenney Cc: Arnd Bergmann --- arch/csky/include/asm/atomic.h | 203 +--------------------------------------- arch/csky/include/asm/barrier.h | 64 ++++++++++--- arch/csky/include/asm/cmpxchg.h | 85 ++++++++++++++--- 3 files changed, 128 insertions(+), 224 deletions(-) diff --git a/arch/csky/include/asm/atomic.h b/arch/csky/include/asm/atomic.h index e369d73..c699c41 100644 --- a/arch/csky/include/asm/atomic.h +++ b/arch/csky/include/asm/atomic.h @@ -3,209 +3,10 @@ #ifndef __ASM_CSKY_ATOMIC_H #define __ASM_CSKY_ATOMIC_H -#include -#include #include -#ifdef CONFIG_CPU_HAS_LDSTEX - -#define __atomic_add_unless __atomic_add_unless -static inline int __atomic_add_unless(atomic_t *v, int a, int u) -{ - unsigned long tmp, ret; - - smp_mb(); - - asm volatile ( - "1: ldex.w %0, (%3) \n" - " mov %1, %0 \n" - " cmpne %0, %4 \n" - " bf 2f \n" - " add %0, %2 \n" - " stex.w %0, (%3) \n" - " bez %0, 1b \n" - "2: \n" - : "=&r" (tmp), "=&r" (ret) - : "r" (a), "r"(&v->counter), "r"(u) - : "memory"); - - if (ret != u) - smp_mb(); - - return ret; -} - -#define ATOMIC_OP(op, c_op) \ -static inline void atomic_##op(int i, atomic_t *v) \ -{ \ - unsigned long tmp; \ - \ - asm volatile ( \ - "1: ldex.w %0, (%2) \n" \ - " " #op " %0, %1 \n" \ - " stex.w %0, (%2) \n" \ - " bez %0, 1b \n" \ - : "=&r" (tmp) \ - : "r" (i), "r"(&v->counter) \ - : "memory"); \ -} - -#define ATOMIC_OP_RETURN(op, c_op) \ -static inline int atomic_##op##_return(int i, atomic_t *v) \ -{ \ - unsigned long tmp, ret; \ - \ - smp_mb(); \ - asm volatile ( \ - "1: ldex.w %0, (%3) \n" \ - " " #op " %0, %2 \n" \ - " mov %1, %0 \n" \ - " stex.w %0, (%3) \n" \ - " bez %0, 1b \n" \ - : "=&r" (tmp), "=&r" (ret) \ - : "r" (i), "r"(&v->counter) \ - : "memory"); \ - smp_mb(); \ - \ - return ret; \ -} - -#define ATOMIC_FETCH_OP(op, c_op) \ -static inline int atomic_fetch_##op(int i, atomic_t *v) \ -{ \ - unsigned long tmp, ret; \ - \ - smp_mb(); \ - asm volatile ( \ - "1: ldex.w %0, (%3) \n" \ - " mov %1, %0 \n" \ - " " #op " %0, %2 \n" \ - " stex.w %0, (%3) \n" \ - " bez %0, 1b \n" \ - : "=&r" (tmp), "=&r" (ret) \ - : "r" (i), "r"(&v->counter) \ - : "memory"); \ - smp_mb(); \ - \ - return ret; \ -} - -#else /* CONFIG_CPU_HAS_LDSTEX */ - -#include - -#define __atomic_add_unless __atomic_add_unless -static inline int __atomic_add_unless(atomic_t *v, int a, int u) -{ - unsigned long tmp, ret, flags; - - raw_local_irq_save(flags); - - asm volatile ( - " ldw %0, (%3) \n" - " mov %1, %0 \n" - " cmpne %0, %4 \n" - " bf 2f \n" - " add %0, %2 \n" - " stw %0, (%3) \n" - "2: \n" - : "=&r" (tmp), "=&r" (ret) - : "r" (a), "r"(&v->counter), "r"(u) - : "memory"); - - raw_local_irq_restore(flags); - - return ret; -} - -#define ATOMIC_OP(op, c_op) \ -static inline void atomic_##op(int i, atomic_t *v) \ -{ \ - unsigned long tmp, flags; \ - \ - raw_local_irq_save(flags); \ - \ - asm volatile ( \ - " ldw %0, (%2) \n" \ - " " #op " %0, %1 \n" \ - " stw %0, (%2) \n" \ - : "=&r" (tmp) \ - : "r" (i), "r"(&v->counter) \ - : "memory"); \ - \ - raw_local_irq_restore(flags); \ -} - -#define ATOMIC_OP_RETURN(op, c_op) \ -static inline int atomic_##op##_return(int i, atomic_t *v) \ -{ \ - unsigned long tmp, ret, flags; \ - \ - raw_local_irq_save(flags); \ - \ - asm volatile ( \ - " ldw %0, (%3) \n" \ - " " #op " %0, %2 \n" \ - " stw %0, (%3) \n" \ - " mov %1, %0 \n" \ - : "=&r" (tmp), "=&r" (ret) \ - : "r" (i), "r"(&v->counter) \ - : "memory"); \ - \ - raw_local_irq_restore(flags); \ - \ - return ret; \ -} - -#define ATOMIC_FETCH_OP(op, c_op) \ -static inline int atomic_fetch_##op(int i, atomic_t *v) \ -{ \ - unsigned long tmp, ret, flags; \ - \ - raw_local_irq_save(flags); \ - \ - asm volatile ( \ - " ldw %0, (%3) \n" \ - " mov %1, %0 \n" \ - " " #op " %0, %2 \n" \ - " stw %0, (%3) \n" \ - : "=&r" (tmp), "=&r" (ret) \ - : "r" (i), "r"(&v->counter) \ - : "memory"); \ - \ - raw_local_irq_restore(flags); \ - \ - return ret; \ -} - -#endif /* CONFIG_CPU_HAS_LDSTEX */ - -#define atomic_add_return atomic_add_return -ATOMIC_OP_RETURN(add, +) -#define atomic_sub_return atomic_sub_return -ATOMIC_OP_RETURN(sub, -) - -#define atomic_fetch_add atomic_fetch_add -ATOMIC_FETCH_OP(add, +) -#define atomic_fetch_sub atomic_fetch_sub -ATOMIC_FETCH_OP(sub, -) -#define atomic_fetch_and atomic_fetch_and -ATOMIC_FETCH_OP(and, &) -#define atomic_fetch_or atomic_fetch_or -ATOMIC_FETCH_OP(or, |) -#define atomic_fetch_xor atomic_fetch_xor -ATOMIC_FETCH_OP(xor, ^) - -#define atomic_and atomic_and -ATOMIC_OP(and, &) -#define atomic_or atomic_or -ATOMIC_OP(or, |) -#define atomic_xor atomic_xor -ATOMIC_OP(xor, ^) - -#undef ATOMIC_FETCH_OP -#undef ATOMIC_OP_RETURN -#undef ATOMIC_OP +#define __atomic_acquire_fence() __smp_acquire_fence() +#define __atomic_release_fence() __smp_release_fence() #include diff --git a/arch/csky/include/asm/barrier.h b/arch/csky/include/asm/barrier.h index a430e7f..6f8269b 100644 --- a/arch/csky/include/asm/barrier.h +++ b/arch/csky/include/asm/barrier.h @@ -16,24 +16,64 @@ * sync.i: inherit from sync, but also flush cpu pipeline * sync.is: the same with sync.i + sync.s * - * bar.brwarw: ordering barrier for all load/store instructions before it - * bar.brwarws: ordering barrier for all load/store instructions before it - * and shareable to other cores - * bar.brar: ordering barrier for all load instructions before it - * bar.brars: ordering barrier for all load instructions before it - * and shareable to other cores - * bar.bwaw: ordering barrier for all store instructions before it - * bar.bwaws: ordering barrier for all store instructions before it - * and shareable to other cores + * + * bar.brwarws: ordering barrier for all load/store instructions + * before/after it and share to other harts + * + * |31|30 26|25 21|20 16|15 10|9 5|4 0| + * 1 10000 s0000 00000 100001 00001 0 bw br aw ar + * + * b: before + * a: after + * r: read + * w: write + * s: share to other harts + * + * Here are all combinations: + * + * bar.brws + * bar.brs + * bar.bws + * bar.arws + * bar.ars + * bar.aws + * bar.brwarws + * bar.brarws + * bar.bwarws + * bar.brwars + * bar.brwaws + * bar.brars + * bar.bwaws */ #ifdef CONFIG_CPU_HAS_CACHEV2 #define mb() asm volatile ("sync.s\n":::"memory") #ifdef CONFIG_SMP -#define __smp_mb() asm volatile ("bar.brwarws\n":::"memory") -#define __smp_rmb() asm volatile ("bar.brars\n":::"memory") -#define __smp_wmb() asm volatile ("bar.bwaws\n":::"memory") + +#define __bar_brws() asm volatile (".long 0x842cc200\n":::"memory") +#define __bar_brs() asm volatile (".long 0x8424c200\n":::"memory") +#define __bar_bws() asm volatile (".long 0x8428c200\n":::"memory") +#define __bar_arws() asm volatile (".long 0x8423c200\n":::"memory") +#define __bar_ars() asm volatile (".long 0x8421c200\n":::"memory") +#define __bar_aws() asm volatile (".long 0x8422c200\n":::"memory") +#define __bar_brwarws() asm volatile (".long 0x842fc200\n":::"memory") +#define __bar_brarws() asm volatile (".long 0x8427c200\n":::"memory") +#define __bar_bwarws() asm volatile (".long 0x842bc200\n":::"memory") +#define __bar_brwars() asm volatile (".long 0x842dc200\n":::"memory") +#define __bar_brwaws() asm volatile (".long 0x842ec200\n":::"memory") +#define __bar_brars() asm volatile (".long 0x8425c200\n":::"memory") +#define __bar_brars() asm volatile (".long 0x8425c200\n":::"memory") +#define __bar_bwaws() asm volatile (".long 0x842ac200\n":::"memory") + +#define __smp_mb() __bar_brwarws() +#define __smp_rmb() __bar_brars() +#define __smp_wmb() __bar_bwaws() + +#define ACQUIRE_FENCE ".long 0x8427c200\n" +#define __smp_acquire_fence() __bar_brarws() +#define __smp_release_fence() __bar_brwaws() + #endif /* CONFIG_SMP */ #define sync_is() asm volatile ("sync.is\n":::"memory") diff --git a/arch/csky/include/asm/cmpxchg.h b/arch/csky/include/asm/cmpxchg.h index ca03e90..3030608 100644 --- a/arch/csky/include/asm/cmpxchg.h +++ b/arch/csky/include/asm/cmpxchg.h @@ -8,7 +8,7 @@ extern void __bad_xchg(void); -#define __xchg(new, ptr, size) \ +#define __xchg_relaxed(new, ptr, size) \ ({ \ __typeof__(ptr) __ptr = (ptr); \ __typeof__(new) __new = (new); \ @@ -18,7 +18,6 @@ extern void __bad_xchg(void); case 2: \ align = ((unsigned long) __ptr & 0x3); \ addr = ((unsigned long) __ptr & ~0x3); \ - smp_mb(); \ if (align) { \ asm volatile ( \ "1: ldex.w %0, (%4) \n" \ @@ -50,10 +49,8 @@ extern void __bad_xchg(void); : "r" (__new), "r"(addr) \ :); \ } \ - smp_mb(); \ break; \ case 4: \ - smp_mb(); \ asm volatile ( \ "1: ldex.w %0, (%3) \n" \ " mov %1, %2 \n" \ @@ -62,7 +59,6 @@ extern void __bad_xchg(void); : "=&r" (__ret), "=&r" (tmp) \ : "r" (__new), "r"(__ptr) \ :); \ - smp_mb(); \ break; \ default: \ __bad_xchg(); \ @@ -70,9 +66,32 @@ extern void __bad_xchg(void); __ret; \ }) -#define xchg(ptr, x) (__xchg((x), (ptr), sizeof(*(ptr)))) +#define xchg_relaxed(ptr, x) \ +({ \ + __xchg_relaxed((x), (ptr), sizeof(*(ptr))); \ +}) + +#define xchg_acquire(ptr, x) \ +({ \ + __typeof__(*(ptr)) __ret; \ + __ret = xchg_relaxed(ptr, x); \ + __smp_acquire_fence(); \ + __ret; \ +}) + +#define xchg_release(ptr, x) \ +({ \ + __smp_release_fence(); \ + xchg_relaxed(ptr, x); \ +}) + +#define xchg(ptr, x) \ +({ \ + __smp_release_fence(); \ + xchg_acquire(ptr, x); \ +}) -#define __cmpxchg(ptr, old, new, size) \ +#define __cmpxchg_relaxed(ptr, old, new, size) \ ({ \ __typeof__(ptr) __ptr = (ptr); \ __typeof__(new) __new = (new); \ @@ -81,7 +100,6 @@ extern void __bad_xchg(void); __typeof__(*(ptr)) __ret = 0; \ switch (size) { \ case 4: \ - smp_mb(); \ asm volatile ( \ "1: ldex.w %0, (%3) \n" \ " cmpne %0, %4 \n" \ @@ -93,7 +111,6 @@ extern void __bad_xchg(void); : "=&r" (__ret), "=&r" (__tmp) \ : "r" (__new), "r"(__ptr), "r"(__old) \ :); \ - smp_mb(); \ break; \ default: \ __bad_xchg(); \ @@ -101,8 +118,54 @@ extern void __bad_xchg(void); __ret; \ }) -#define cmpxchg(ptr, o, n) \ - (__cmpxchg((ptr), (o), (n), sizeof(*(ptr)))) +#define cmpxchg_relaxed(ptr, o, n) \ + (__cmpxchg_relaxed((ptr), (o), (n), sizeof(*(ptr)))) + +#define cmpxchg_release(ptr, o, n) \ +({ \ + __smp_release_fence(); \ + cmpxchg_relaxed(ptr, o, n); \ +}) + +#define __cmpxchg_acquire(ptr, old, new, size) \ +({ \ + __typeof__(ptr) __ptr = (ptr); \ + __typeof__(new) __new = (new); \ + __typeof__(new) __tmp; \ + __typeof__(old) __old = (old); \ + __typeof__(*(ptr)) __ret = 0; \ + switch (size) { \ + case 4: \ + asm volatile ( \ + "1: ldex.w %0, (%3) \n" \ + " cmpne %0, %4 \n" \ + " bt 2f \n" \ + " mov %1, %2 \n" \ + " stex.w %1, (%3) \n" \ + " bez %1, 1b \n" \ + ACQUIRE_FENCE \ + "2: \n" \ + : "=&r" (__ret), "=&r" (__tmp) \ + : "r" (__new), "r"(__ptr), "r"(__old) \ + :); \ + break; \ + default: \ + __bad_xchg(); \ + } \ + __ret; \ +}) + +#define cmpxchg_acquire(ptr, o, n) \ + (__cmpxchg_acquire((ptr), (o), (n), sizeof(*(ptr)))) + +#define cmpxchg(ptr, o, n) \ +({ \ + __typeof__(*(ptr)) __ret; \ + __smp_release_fence(); \ + __ret = cmpxchg_acquire(ptr, o, n); \ + __ret; \ +}) + #else #include #endif