From patchwork Sat May 5 10:00:55 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ingo Molnar X-Patchwork-Id: 10382091 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id E571560159 for ; Sat, 5 May 2018 10:01:19 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CEA9F29548 for ; Sat, 5 May 2018 10:01:19 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C0FEB29566; Sat, 5 May 2018 10:01:19 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: * X-Spam-Status: No, score=1.1 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, FSL_HELO_FAKE, MAILING_LIST_MULTI autolearn=no version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 2DC9329548 for ; Sat, 5 May 2018 10:01:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=ZYEujMQBU88ta7AK1XWOWesLPgFSRSK7bC8BZ3C2aBw=; b=dxAlaxh2R9tQ8t HxJyQLRzTMoObb9JKCBhHTh/zTkxNphSwkzn0ypjPYLoDWu8XcXhVrXzL2HgJowb1700pP/G9jWOp vrtEMn2WDcvKba5gFcO4Cc05eICEEiNDP5akxIyNAKWEmocjYowfDbnwPSD8n83G/OnTJGqMwcJGX 4EskoN66FRBymVezMFd0AIv6UO6oWjR3BD+sZFXshqb1tqVkKxdkaiNzaiGIVjNWvVbTQ8gVL3P50 KNmDVJx2mfB5bsLxVdoU+dqlY2n+AZnVb6O9y0yegjr5f9Fcn/iTnQkVNLZPj30l4HqNtVOWZWLQA 8HJQLpOZtK257KxkUrFw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1fEu0A-00047M-Nc; Sat, 05 May 2018 10:01:14 +0000 Received: from mail-wm0-x244.google.com ([2a00:1450:400c:c09::244]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1fEu06-00045p-Fk for linux-arm-kernel@lists.infradead.org; Sat, 05 May 2018 10:01:12 +0000 Received: by mail-wm0-x244.google.com with SMTP id f6so7314242wmc.4 for ; Sat, 05 May 2018 03:00:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=a1tmzfTfB7Uzo4CM64mhrtXfokj8RIDzUXa2eFp0im4=; b=TDON3l/7RMlBgpZLw4bvm0p4zTEZvItFiXD/qalLTRW9r6mA4rKl0tqisQobT+URbX emihvs8syWLfi+afgkeoJfjjQlEHYSSrSOKcHPRlknBy3IibDRpSrmEm9QHObKtNx5SP ocCfwHaCQAEGL+Oq/dzI8W1rr8Fn3Ij4Hj2Gj8+SB7S41Po8i55nhjlHYKrduVdczMhB l88vZZ+kyuJtYfJJ5tUmv2ro8SACEtNHfvnoXC40iO5lHq0ZYf6eo/5JLzkK24OzlHL7 ieX9vzqCsspsq+HjVnMhM94cMrrElEXg3x7DQE8HcHW5sTkCDqqUHiWqv/gnu/3ZMVE6 tqfA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=a1tmzfTfB7Uzo4CM64mhrtXfokj8RIDzUXa2eFp0im4=; b=sGK64KMP7kiTDpaOZTfxhzx1G2uhFqTAekbS65wektVswHHGCFInZOWp3u/0775Apx XvD6oZlzm0Rc/FEVYV6PU7yhm3NR2y0C998jGORnWnHG0ydzebZt55Bn5BdFkZ/RpKj7 T1DtMqYJTFClQOSoQPzszThd6ZrmWGsLIKoAoJkHzaaRvbkH9ceKB8PrWLK9VGxAOVU3 f7q38PW+z6emyaqVJnEnRmABK1FGV0qyyi/OX4iDf0QRYc7DOrzRL3yBKqXPaZXyBuhH a9ry3ckvB8K3kmzddPNDnZOfwalV1O/UFc7HxOXWhYn8HlF6vqSuKk2rtynEZFNVd+dy 3TWw== X-Gm-Message-State: ALQs6tAamYhxwMvk59xk65mokHlb7ZT2OC+ZOE1W4v2W+h4l603lAvjQ P4WW4zTKjSNf2NxoKrvNP3w= X-Google-Smtp-Source: AB8JxZoa3Orx5C0cnufswZIcWsRmgNkfjhJgcKOqrNMYcjm0+66UPs+2gJ4Cnwococvsvfu9GOvdEw== X-Received: by 10.28.144.75 with SMTP id s72mr18818251wmd.85.1525514458061; Sat, 05 May 2018 03:00:58 -0700 (PDT) Received: from gmail.com (2E8B0CD5.catv.pool.telekom.hu. [46.139.12.213]) by smtp.gmail.com with ESMTPSA id o53-v6sm23008949wrc.96.2018.05.05.03.00.56 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 05 May 2018 03:00:57 -0700 (PDT) Date: Sat, 5 May 2018 12:00:55 +0200 From: Ingo Molnar To: Peter Zijlstra , Benjamin Herrenschmidt , Paul Mackerras , Michael Ellerman Subject: [RFC PATCH] locking/atomics/powerpc: Introduce optimized cmpxchg_release() family of APIs for PowerPC Message-ID: <20180505100055.yc4upauxo5etq5ud@gmail.com> References: <20180504173937.25300-1-mark.rutland@arm.com> <20180504173937.25300-2-mark.rutland@arm.com> <20180504180105.GS12217@hirez.programming.kicks-ass.net> <20180504180909.dnhfflibjwywnm4l@lakrids.cambridge.arm.com> <20180505081100.nsyrqrpzq2vd27bk@gmail.com> <20180505084721.GA32344@noisy.programming.kicks-ass.net> <20180505090403.p2ywuen42rnlwizq@gmail.com> <20180505093829.xfylnedwd5nonhae@gmail.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20180505093829.xfylnedwd5nonhae@gmail.com> User-Agent: NeoMutt/20170609 (1.8.3) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20180505_030110_554292_FE076DF3 X-CRM114-Status: GOOD ( 20.84 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , catalin.marinas@arm.com, boqun.feng@gmail.com, will.deacon@arm.com, linux-kernel@vger.kernel.org, dvyukov@google.com, aryabinin@virtuozzo.com, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP * Ingo Molnar wrote: > > So there's no loss in arch flexibility. > > BTW., PowerPC for example is already in such a situation, it does not define > atomic_cmpxchg_release(), only the other APIs: > > #define atomic_cmpxchg(v, o, n) (cmpxchg(&((v)->counter), (o), (n))) > #define atomic_cmpxchg_relaxed(v, o, n) \ > cmpxchg_relaxed(&((v)->counter), (o), (n)) > #define atomic_cmpxchg_acquire(v, o, n) \ > cmpxchg_acquire(&((v)->counter), (o), (n)) > > Was it really the intention on the PowerPC side that the generic code falls back > to cmpxchg(), i.e.: > > # define atomic_cmpxchg_release(...) __atomic_op_release(atomic_cmpxchg, __VA_ARGS__) > > Which after macro expansion becomes: > > smp_mb__before_atomic(); > atomic_cmpxchg_relaxed(v, o, n); > > smp_mb__before_atomic() on PowerPC falls back to the generic __smp_mb(), which > falls back to mb(), which on PowerPC is the 'sync' instruction. > > Isn't this a inefficiency bug? > > While I'm pretty clueless about PowerPC low level cmpxchg atomics, they appear to > have the following basic structure: > > full cmpxchg(): > > PPC_ATOMIC_ENTRY_BARRIER # sync > ldarx + stdcx > PPC_ATOMIC_EXIT_BARRIER # sync > > cmpxchg_relaxed(): > > ldarx + stdcx > > cmpxchg_acquire(): > > ldarx + stdcx > PPC_ACQUIRE_BARRIER # lwsync > > The logical extension for cmpxchg_release() would be: > > cmpxchg_release(): > > PPC_RELEASE_BARRIER # lwsync > ldarx + stdcx > > But instead we silently get the generic fallback, which does: > > smp_mb__before_atomic(); > atomic_cmpxchg_relaxed(v, o, n); > > Which maps to: > > sync > ldarx + stdcx > > Note that it uses a full barrier instead of lwsync (does that stand for > 'lightweight sync'?). > > Even if it turns out we need the full barrier, with the overly finegrained > structure of the atomics this detail is totally undocumented and non-obvious. The patch below fills in those bits and implements the optimized cmpxchg_release() family of APIs. The end effect should be that cmpxchg_release() will now use 'lwsync' instead of 'sync' on PowerPC, for the following APIs: cmpxchg_release() cmpxchg64_release() atomic_cmpxchg_release() atomic64_cmpxchg_release() I based this choice of the release barrier on an existing bitops low level PowerPC method: DEFINE_BITOP(clear_bits_unlock, andc, PPC_RELEASE_BARRIER) This clearly suggests that PPC_RELEASE_BARRIER is in active use and 'lwsync' is the 'release barrier' instruction, if I interpreted that right. But I know very little about PowerPC so this might be spectacularly wrong. It's totally untested as well. I also pretty sick today so my mental capabilities are significantly reduced ... So not signed off and such. Thanks, Ingo --- arch/powerpc/include/asm/atomic.h | 4 ++ arch/powerpc/include/asm/cmpxchg.h | 81 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 85 insertions(+) diff --git a/arch/powerpc/include/asm/atomic.h b/arch/powerpc/include/asm/atomic.h index 682b3e6a1e21..f7a6f29acb12 100644 --- a/arch/powerpc/include/asm/atomic.h +++ b/arch/powerpc/include/asm/atomic.h @@ -213,6 +213,8 @@ static __inline__ int atomic_dec_return_relaxed(atomic_t *v) cmpxchg_relaxed(&((v)->counter), (o), (n)) #define atomic_cmpxchg_acquire(v, o, n) \ cmpxchg_acquire(&((v)->counter), (o), (n)) +#define atomic_cmpxchg_release(v, o, n) \ + cmpxchg_release(&((v)->counter), (o), (n)) #define atomic_xchg(v, new) (xchg(&((v)->counter), new)) #define atomic_xchg_relaxed(v, new) xchg_relaxed(&((v)->counter), (new)) @@ -519,6 +521,8 @@ static __inline__ long atomic64_dec_if_positive(atomic64_t *v) cmpxchg_relaxed(&((v)->counter), (o), (n)) #define atomic64_cmpxchg_acquire(v, o, n) \ cmpxchg_acquire(&((v)->counter), (o), (n)) +#define atomic64_cmpxchg_release(v, o, n) \ + cmpxchg_release(&((v)->counter), (o), (n)) #define atomic64_xchg(v, new) (xchg(&((v)->counter), new)) #define atomic64_xchg_relaxed(v, new) xchg_relaxed(&((v)->counter), (new)) diff --git a/arch/powerpc/include/asm/cmpxchg.h b/arch/powerpc/include/asm/cmpxchg.h index 9b001f1f6b32..6e46310b1833 100644 --- a/arch/powerpc/include/asm/cmpxchg.h +++ b/arch/powerpc/include/asm/cmpxchg.h @@ -213,10 +213,12 @@ __xchg_relaxed(void *ptr, unsigned long x, unsigned int size) CMPXCHG_GEN(u8, , PPC_ATOMIC_ENTRY_BARRIER, PPC_ATOMIC_EXIT_BARRIER, "memory"); CMPXCHG_GEN(u8, _local, , , "memory"); CMPXCHG_GEN(u8, _acquire, , PPC_ACQUIRE_BARRIER, "memory"); +CMPXCHG_GEN(u8, _release, PPC_RELEASE_BARRIER, , "memory"); CMPXCHG_GEN(u8, _relaxed, , , "cc"); CMPXCHG_GEN(u16, , PPC_ATOMIC_ENTRY_BARRIER, PPC_ATOMIC_EXIT_BARRIER, "memory"); CMPXCHG_GEN(u16, _local, , , "memory"); CMPXCHG_GEN(u16, _acquire, , PPC_ACQUIRE_BARRIER, "memory"); +CMPXCHG_GEN(u16, _release, PPC_RELEASE_BARRIER, , "memory"); CMPXCHG_GEN(u16, _relaxed, , , "cc"); static __always_inline unsigned long @@ -314,6 +316,29 @@ __cmpxchg_u32_acquire(u32 *p, unsigned long old, unsigned long new) return prev; } +static __always_inline unsigned long +__cmpxchg_u32_release(u32 *p, unsigned long old, unsigned long new) +{ + unsigned long prev; + + __asm__ __volatile__ ( + PPC_RELEASE_BARRIER +"1: lwarx %0,0,%2 # __cmpxchg_u32_release\n" +" cmpw 0,%0,%3\n" +" bne- 2f\n" + PPC405_ERR77(0, %2) +" stwcx. %4,0,%2\n" +" bne- 1b\n" + "\n" +"2:" + : "=&r" (prev), "+m" (*p) + : "r" (p), "r" (old), "r" (new) + : "cc", "memory"); + + return prev; +} + + #ifdef CONFIG_PPC64 static __always_inline unsigned long __cmpxchg_u64(volatile unsigned long *p, unsigned long old, unsigned long new) @@ -397,6 +422,27 @@ __cmpxchg_u64_acquire(u64 *p, unsigned long old, unsigned long new) return prev; } + +static __always_inline unsigned long +__cmpxchg_u64_release(u64 *p, unsigned long old, unsigned long new) +{ + unsigned long prev; + + __asm__ __volatile__ ( + PPC_RELEASE_BARRIER +"1: ldarx %0,0,%2 # __cmpxchg_u64_release\n" +" cmpd 0,%0,%3\n" +" bne- 2f\n" +" stdcx. %4,0,%2\n" +" bne- 1b\n" + "\n" +"2:" + : "=&r" (prev), "+m" (*p) + : "r" (p), "r" (old), "r" (new) + : "cc", "memory"); + + return prev; +} #endif static __always_inline unsigned long @@ -478,6 +524,27 @@ __cmpxchg_acquire(void *ptr, unsigned long old, unsigned long new, BUILD_BUG_ON_MSG(1, "Unsupported size for __cmpxchg_acquire"); return old; } + +static __always_inline unsigned long +__cmpxchg_release(void *ptr, unsigned long old, unsigned long new, + unsigned int size) +{ + switch (size) { + case 1: + return __cmpxchg_u8_release(ptr, old, new); + case 2: + return __cmpxchg_u16_release(ptr, old, new); + case 4: + return __cmpxchg_u32_release(ptr, old, new); +#ifdef CONFIG_PPC64 + case 8: + return __cmpxchg_u64_release(ptr, old, new); +#endif + } + BUILD_BUG_ON_MSG(1, "Unsupported size for __cmpxchg_release"); + return old; +} + #define cmpxchg(ptr, o, n) \ ({ \ __typeof__(*(ptr)) _o_ = (o); \ @@ -512,6 +579,15 @@ __cmpxchg_acquire(void *ptr, unsigned long old, unsigned long new, (unsigned long)_o_, (unsigned long)_n_, \ sizeof(*(ptr))); \ }) + +#define cmpxchg_release(ptr, o, n) \ +({ \ + __typeof__(*(ptr)) _o_ = (o); \ + __typeof__(*(ptr)) _n_ = (n); \ + (__typeof__(*(ptr))) __cmpxchg_release((ptr), \ + (unsigned long)_o_, (unsigned long)_n_, \ + sizeof(*(ptr))); \ +}) #ifdef CONFIG_PPC64 #define cmpxchg64(ptr, o, n) \ ({ \ @@ -533,6 +609,11 @@ __cmpxchg_acquire(void *ptr, unsigned long old, unsigned long new, BUILD_BUG_ON(sizeof(*(ptr)) != 8); \ cmpxchg_acquire((ptr), (o), (n)); \ }) +#define cmpxchg64_release(ptr, o, n) \ +({ \ + BUILD_BUG_ON(sizeof(*(ptr)) != 8); \ + cmpxchg_release((ptr), (o), (n)); \ +}) #else #include #define cmpxchg64_local(ptr, o, n) __cmpxchg64_local_generic((ptr), (o), (n))