From patchwork Wed Apr 5 14:17:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uros Bizjak X-Patchwork-Id: 13201836 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E118FC761A6 for ; Wed, 5 Apr 2023 14:18:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238494AbjDEOSd (ORCPT ); Wed, 5 Apr 2023 10:18:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42850 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238402AbjDEOSZ (ORCPT ); Wed, 5 Apr 2023 10:18:25 -0400 Received: from mail-ed1-x530.google.com (mail-ed1-x530.google.com [IPv6:2a00:1450:4864:20::530]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 67F4B658B; Wed, 5 Apr 2023 07:17:51 -0700 (PDT) Received: by mail-ed1-x530.google.com with SMTP id eg48so141372311edb.13; Wed, 05 Apr 2023 07:17:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680704268; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=IR5uya/6k59Z6vjpJ5eLjtKZNYMu0vN6LL4bpPspKvA=; b=hzv4HYQ5T2nwEJbzGw8dJN/I+F4Ay7RqJgdT+/nU+XX7ZoGg8P8l9L8teClCdkpmvP 3VP7uCz0Z+YqdgWXL+MH2tFmXHoR+WPJu2bf+Fr9qb/QvhkGpEoSrJ0NhG0eAx/0QiTJ UQgtBAkuIqKtB9v5p/5vYSrI08EmluGXK/WD03V/0fdbsdMfx/eWMRDY+C4T7hDoaYaj NOE2x6GoeZ01oYXffPbqWgJlIPw1nNp4nyMKkgNQHZrTuLm7YYwS07BBvNmSqjpH876L /ISIl/kFQdHhrGtYe0xrraw5d1lTLHXAuQ1Znq44pBtvm8+Xiwqr3RT/FspKMHRWIraX TSSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680704268; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=IR5uya/6k59Z6vjpJ5eLjtKZNYMu0vN6LL4bpPspKvA=; b=eEA4SHokIN/xju+j22I2sq5+C0O3OqfR8wkjk6a+Vjzf3jJUa0jXFaZf+TsjXwhXeQ 4FM/e4/P9dsW1TA3IzzN5cgoonm7foAJqrxqxOTnpxF41Ns2GGK8C/77Mm/dxCul0ySX bBO0dRKf3o/bNDJVtRoheozB7U0eL1TXiHo9N3q8ALRLr8z/PxL0N5tSt6fEEwm92hBh VAyAXJKgYW1QWZPwBL1EMvI94fQZQY5iykA00A3TfJhn/+MRGErW5ls5VlMvUoaLMCIK GcMc4zoalesCunWHXwaujH6yeGxSqx7Waaqp+NG2UzTfEmjYaOH4mxYfO1jA+XK+A72n uEMQ== X-Gm-Message-State: AAQBX9et+pmj+LhA3k8rBehF6XZ5a//TVYrbW1L5VTAf9yELvIk5dIUm inaE5LJfPd2VckeDdSKjrGPl451bId2cE2gl X-Google-Smtp-Source: AKy350apZB614sloiIbQFvOjRXtcEBijYMBgr256Z113Bx8z9JeSHopzv9p9oPRNRJ0o5MWKc6tu/g== X-Received: by 2002:a17:906:ece1:b0:93d:ae74:fa9e with SMTP id qt1-20020a170906ece100b0093dae74fa9emr3077789ejb.7.1680704268119; Wed, 05 Apr 2023 07:17:48 -0700 (PDT) Received: from localhost.localdomain ([46.248.82.114]) by smtp.gmail.com with ESMTPSA id g6-20020a170906348600b009334219656dsm7381246ejb.56.2023.04.05.07.17.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Apr 2023 07:17:47 -0700 (PDT) From: Uros Bizjak To: linux-alpha@vger.kernel.org, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-perf-users@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Uros Bizjak , Will Deacon , Peter Zijlstra , Boqun Feng , Mark Rutland Subject: [PATCH v2 1/5] locking/atomic: Add generic try_cmpxchg{,64}_local support Date: Wed, 5 Apr 2023 16:17:06 +0200 Message-Id: <20230405141710.3551-2-ubizjak@gmail.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230405141710.3551-1-ubizjak@gmail.com> References: <20230405141710.3551-1-ubizjak@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-mips@vger.kernel.org Add generic support for try_cmpxchg{,64}_local and their falbacks. These provides the generic try_cmpxchg_local family of functions from the arch_ prefixed version, also adding explicit instrumentation. Cc: Will Deacon Cc: Peter Zijlstra Cc: Boqun Feng Cc: Mark Rutland Signed-off-by: Uros Bizjak Acked-by: Mark Rutland --- include/linux/atomic/atomic-arch-fallback.h | 24 ++++++++++++++++++++- include/linux/atomic/atomic-instrumented.h | 20 ++++++++++++++++- scripts/atomic/gen-atomic-fallback.sh | 4 ++++ scripts/atomic/gen-atomic-instrumented.sh | 2 +- 4 files changed, 47 insertions(+), 3 deletions(-) diff --git a/include/linux/atomic/atomic-arch-fallback.h b/include/linux/atomic/atomic-arch-fallback.h index 77bc5522e61c..36c92851cdee 100644 --- a/include/linux/atomic/atomic-arch-fallback.h +++ b/include/linux/atomic/atomic-arch-fallback.h @@ -217,6 +217,28 @@ #endif /* arch_try_cmpxchg64_relaxed */ +#ifndef arch_try_cmpxchg_local +#define arch_try_cmpxchg_local(_ptr, _oldp, _new) \ +({ \ + typeof(*(_ptr)) *___op = (_oldp), ___o = *___op, ___r; \ + ___r = arch_cmpxchg_local((_ptr), ___o, (_new)); \ + if (unlikely(___r != ___o)) \ + *___op = ___r; \ + likely(___r == ___o); \ +}) +#endif /* arch_try_cmpxchg_local */ + +#ifndef arch_try_cmpxchg64_local +#define arch_try_cmpxchg64_local(_ptr, _oldp, _new) \ +({ \ + typeof(*(_ptr)) *___op = (_oldp), ___o = *___op, ___r; \ + ___r = arch_cmpxchg64_local((_ptr), ___o, (_new)); \ + if (unlikely(___r != ___o)) \ + *___op = ___r; \ + likely(___r == ___o); \ +}) +#endif /* arch_try_cmpxchg64_local */ + #ifndef arch_atomic_read_acquire static __always_inline int arch_atomic_read_acquire(const atomic_t *v) @@ -2456,4 +2478,4 @@ arch_atomic64_dec_if_positive(atomic64_t *v) #endif #endif /* _LINUX_ATOMIC_FALLBACK_H */ -// b5e87bdd5ede61470c29f7a7e4de781af3770f09 +// 1f49bd4895a4b7a5383906649027205c52ec80ab diff --git a/include/linux/atomic/atomic-instrumented.h b/include/linux/atomic/atomic-instrumented.h index 7a139ec030b0..14a9212cc987 100644 --- a/include/linux/atomic/atomic-instrumented.h +++ b/include/linux/atomic/atomic-instrumented.h @@ -2066,6 +2066,24 @@ atomic_long_dec_if_positive(atomic_long_t *v) arch_sync_cmpxchg(__ai_ptr, __VA_ARGS__); \ }) +#define try_cmpxchg_local(ptr, oldp, ...) \ +({ \ + typeof(ptr) __ai_ptr = (ptr); \ + typeof(oldp) __ai_oldp = (oldp); \ + instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ + instrument_atomic_write(__ai_oldp, sizeof(*__ai_oldp)); \ + arch_try_cmpxchg_local(__ai_ptr, __ai_oldp, __VA_ARGS__); \ +}) + +#define try_cmpxchg64_local(ptr, oldp, ...) \ +({ \ + typeof(ptr) __ai_ptr = (ptr); \ + typeof(oldp) __ai_oldp = (oldp); \ + instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ + instrument_atomic_write(__ai_oldp, sizeof(*__ai_oldp)); \ + arch_try_cmpxchg64_local(__ai_ptr, __ai_oldp, __VA_ARGS__); \ +}) + #define cmpxchg_double(ptr, ...) \ ({ \ typeof(ptr) __ai_ptr = (ptr); \ @@ -2083,4 +2101,4 @@ atomic_long_dec_if_positive(atomic_long_t *v) }) #endif /* _LINUX_ATOMIC_INSTRUMENTED_H */ -// 764f741eb77a7ad565dc8d99ce2837d5542e8aee +// 456e206c7e4e681126c482e4edcc6f46921ac731 diff --git a/scripts/atomic/gen-atomic-fallback.sh b/scripts/atomic/gen-atomic-fallback.sh index 3a07695e3c89..6e853f0dad8d 100755 --- a/scripts/atomic/gen-atomic-fallback.sh +++ b/scripts/atomic/gen-atomic-fallback.sh @@ -225,6 +225,10 @@ for cmpxchg in "cmpxchg" "cmpxchg64"; do gen_try_cmpxchg_fallbacks "${cmpxchg}" done +for cmpxchg in "cmpxchg_local" "cmpxchg64_local"; do + gen_try_cmpxchg_fallback "${cmpxchg}" "" +done + grep '^[a-z]' "$1" | while read name meta args; do gen_proto "${meta}" "${name}" "atomic" "int" ${args} done diff --git a/scripts/atomic/gen-atomic-instrumented.sh b/scripts/atomic/gen-atomic-instrumented.sh index 77c06526a574..c8165e9431bf 100755 --- a/scripts/atomic/gen-atomic-instrumented.sh +++ b/scripts/atomic/gen-atomic-instrumented.sh @@ -173,7 +173,7 @@ for xchg in "xchg" "cmpxchg" "cmpxchg64" "try_cmpxchg" "try_cmpxchg64"; do done done -for xchg in "cmpxchg_local" "cmpxchg64_local" "sync_cmpxchg"; do +for xchg in "cmpxchg_local" "cmpxchg64_local" "sync_cmpxchg" "try_cmpxchg_local" "try_cmpxchg64_local" ; do gen_xchg "${xchg}" "" "" printf "\n" done From patchwork Wed Apr 5 14:17:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uros Bizjak X-Patchwork-Id: 13201837 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 31397C77B60 for ; Wed, 5 Apr 2023 14:18:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238514AbjDEOSf (ORCPT ); Wed, 5 Apr 2023 10:18:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42818 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238451AbjDEOSa (ORCPT ); Wed, 5 Apr 2023 10:18:30 -0400 Received: from mail-ed1-x52e.google.com (mail-ed1-x52e.google.com [IPv6:2a00:1450:4864:20::52e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B1151C1; Wed, 5 Apr 2023 07:17:53 -0700 (PDT) Received: by mail-ed1-x52e.google.com with SMTP id b20so141489775edd.1; Wed, 05 Apr 2023 07:17:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680704270; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5Aezn9KwOiaMHT8QBIvXqxw3/7pRoTbwoU4+L6kaZfo=; b=h6B0D1guAAz/+Fh3o2MFRuBp2njF7Fye/Xxf2r9nX8V7xNi91MOio5aDjUQBXijRT0 75EwXqWNYR/1gi60t0DEKttu9WLaqB3/MY8gXnTwiu4Oz4MkVmFrs2JDhb+L8fnPcptQ 7nrJoQYBcxE0Ipea11+gFcYnJH92f8uuVoDqyuTltBWba1oiCcF6uWTEYtwCT5xdTLlI qcI67O3MKKtm2Ll2/izFTSnFMbusRse/8HmyoHl63V/KzowbwYrV1X/+8pt5J0I287/X g/ngG/l9UiDfge5nIclREoyJnIxf0VVxBnBlDK8hGh94SIMNrCXES8Wpb/gmo+IR/6pP 21wA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680704270; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5Aezn9KwOiaMHT8QBIvXqxw3/7pRoTbwoU4+L6kaZfo=; b=r/8UG7SMi8mmqluMZiaS8+A3FcRxef+e0l7LrTLHji1vBGlMpGTOcJ+6wvg6BG6io3 Xwr9WAMi2irDL3rApOJmIFUbvrg/MEUHwln4r2snAySlkpEIMJMkKL/lJOIgZD7ONoE/ sjjFj/dC9aLAcIu9eWoKYzWjhZrKryKiGfgaM3nCYzpGAA2Tc7eYrXeuFojwxSOU2fzl EejMbB2IKjbPBU3iRifDiPNW02rY0Z7PGZQzmPumDjDrjkzOJ6Q5cZiNPEFeH258ps7o 9MJZtiG8UugzZHruTc3RLDxEjNJWitx7LgS5YJgf6R+58K/OSITKO2yj3CoJPEODBYIO WNbQ== X-Gm-Message-State: AAQBX9f8jXM1pm/VoKWQOvA3OY0Ai4ABtvQ9kb2N4XZM5mPUY93epFvL iMYyVyD+6uopp5rRjY1XXZISPO496/J9Mclt X-Google-Smtp-Source: AKy350Z/3xgh/xSykpZuUX3JQoJOOenna0pCFqtY30j4WZoVD6fIcb1JmDg2U9Ji+C4VhgW2S0ytyw== X-Received: by 2002:a17:906:d977:b0:8b0:f277:5cde with SMTP id rp23-20020a170906d97700b008b0f2775cdemr3133825ejb.32.1680704270128; Wed, 05 Apr 2023 07:17:50 -0700 (PDT) Received: from localhost.localdomain ([46.248.82.114]) by smtp.gmail.com with ESMTPSA id g6-20020a170906348600b009334219656dsm7381246ejb.56.2023.04.05.07.17.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Apr 2023 07:17:49 -0700 (PDT) From: Uros Bizjak To: linux-alpha@vger.kernel.org, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-perf-users@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Uros Bizjak , Arnd Bergmann Subject: [PATCH v2 2/5] locking/generic: Wire up local{,64}_try_cmpxchg Date: Wed, 5 Apr 2023 16:17:07 +0200 Message-Id: <20230405141710.3551-3-ubizjak@gmail.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230405141710.3551-1-ubizjak@gmail.com> References: <20230405141710.3551-1-ubizjak@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-mips@vger.kernel.org Implement generic support for local{,64}_try_cmpxchg. Redirect to the atomic_ family of functions when the target does not provide its own local.h definitions. For 64-bit targets, implement local64_try_cmpxchg and local64_cmpxchg using typed C wrappers that call local_ family of functions and provide additional checking of their input arguments. Cc: Arnd Bergmann Signed-off-by: Uros Bizjak Acked-by: Mark Rutland --- include/asm-generic/local.h | 1 + include/asm-generic/local64.h | 12 +++++++++++- 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/include/asm-generic/local.h b/include/asm-generic/local.h index fca7f1d84818..7f97018df66f 100644 --- a/include/asm-generic/local.h +++ b/include/asm-generic/local.h @@ -42,6 +42,7 @@ typedef struct #define local_inc_return(l) atomic_long_inc_return(&(l)->a) #define local_cmpxchg(l, o, n) atomic_long_cmpxchg((&(l)->a), (o), (n)) +#define local_try_cmpxchg(l, po, n) atomic_long_try_cmpxchg((&(l)->a), (po), (n)) #define local_xchg(l, n) atomic_long_xchg((&(l)->a), (n)) #define local_add_unless(l, _a, u) atomic_long_add_unless((&(l)->a), (_a), (u)) #define local_inc_not_zero(l) atomic_long_inc_not_zero(&(l)->a) diff --git a/include/asm-generic/local64.h b/include/asm-generic/local64.h index 765be0b7d883..14963a7a6253 100644 --- a/include/asm-generic/local64.h +++ b/include/asm-generic/local64.h @@ -42,7 +42,16 @@ typedef struct { #define local64_sub_return(i, l) local_sub_return((i), (&(l)->a)) #define local64_inc_return(l) local_inc_return(&(l)->a) -#define local64_cmpxchg(l, o, n) local_cmpxchg((&(l)->a), (o), (n)) +static inline s64 local64_cmpxchg(local64_t *l, s64 old, s64 new) +{ + return local_cmpxchg(&l->a, old, new); +} + +static inline bool local64_try_cmpxchg(local64_t *l, s64 *old, s64 new) +{ + return local_try_cmpxchg(&l->a, (long *)old, new); +} + #define local64_xchg(l, n) local_xchg((&(l)->a), (n)) #define local64_add_unless(l, _a, u) local_add_unless((&(l)->a), (_a), (u)) #define local64_inc_not_zero(l) local_inc_not_zero(&(l)->a) @@ -81,6 +90,7 @@ typedef struct { #define local64_inc_return(l) atomic64_inc_return(&(l)->a) #define local64_cmpxchg(l, o, n) atomic64_cmpxchg((&(l)->a), (o), (n)) +#define local64_try_cmpxchg(l, po, n) atomic64_try_cmpxchg((&(l)->a), (po), (n)) #define local64_xchg(l, n) atomic64_xchg((&(l)->a), (n)) #define local64_add_unless(l, _a, u) atomic64_add_unless((&(l)->a), (_a), (u)) #define local64_inc_not_zero(l) atomic64_inc_not_zero(&(l)->a) From patchwork Wed Apr 5 14:17:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uros Bizjak X-Patchwork-Id: 13201838 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 65DA9C76188 for ; Wed, 5 Apr 2023 14:18:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238535AbjDEOSy (ORCPT ); Wed, 5 Apr 2023 10:18:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42850 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238476AbjDEOSc (ORCPT ); Wed, 5 Apr 2023 10:18:32 -0400 Received: from mail-ed1-x52f.google.com (mail-ed1-x52f.google.com [IPv6:2a00:1450:4864:20::52f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 992365FCC; Wed, 5 Apr 2023 07:17:55 -0700 (PDT) Received: by mail-ed1-x52f.google.com with SMTP id eh3so141437858edb.11; Wed, 05 Apr 2023 07:17:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680704272; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jH8CK6/Zc/t5YGDy3q47KHMB123gE9O3L+8qZD7dC+A=; b=XZjaJuc9oq0eEJuqYQaQCSxyJEBrGWscwxPcLCz1SGmMY8nwBoCrKPTVkKuDR6NHM0 nd8qEhQmNfk/r4MHTfpwl3GDx8xszhuQDekAlI7AuV5wPPTHtKtudXEJvnYoUIEF8a5w BCncuKCXN80M4D+9c/uTSAQxM1AULldg3sOFYIYNysgVeYWsg+Xf3uXNSO3204XksTVt VY7rOKvL0GsKooMWqYkHs8pZAlDl7U2kRNPRjRx+Zj41LQ7/h5JXvJnY1puZvDO1QMoW 76SsXCfs7PmO+YxBWiArT3kI1VrIE5rfXzPTV1J+iRMhJsk2mSVTfV9ok5E9WbeCZyIn ETvg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680704272; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jH8CK6/Zc/t5YGDy3q47KHMB123gE9O3L+8qZD7dC+A=; b=0A264gp2fJZQ38cz9f6l/6ypN69/+DMEmLTdu74v9y/4Zrkne8gB0+MXZ40mHNneh/ HQtFrDqACEhN5PBULKBAEudA1IAC81HFhp0kq85Z9zGjXHU/Xuv5ZoF+iQ8kEc7I2Lf/ GU29ogkA4auVzBylAfMyDpKYbZIf/8kXpgECsZhGUexisWVX6tm/mYdDPywQb/NuQ3Uq mp30H3SLu05OzH8vNFTN+U6Vj0OR6ROiLESyVblWVDUuJYxe0mYHqhNAnGaaQtHuGtiO h4QH8WounMADGb9HOYIB8q662PLXJV61ji7Om8fGHpAKlRoXfbcXHraWWAurhKRM4Tmv ipsg== X-Gm-Message-State: AAQBX9d4VYwaDT5zw9AGI72aslyb04ROLmIUlAvLv1ov91vUAYS0Ckec y0WMZni+7UxcTHu046k4aEiXjHLinjJfCzwW X-Google-Smtp-Source: AKy350adAYW+TEWiUmLpAezMNrhF5/NZq/p/j5R7S+Q7+ZgtSrt1TPPbRawQQVHNg0S3+/m7HJyxGA== X-Received: by 2002:a17:907:20bb:b0:930:8714:6739 with SMTP id pw27-20020a17090720bb00b0093087146739mr3007454ejb.30.1680704272405; Wed, 05 Apr 2023 07:17:52 -0700 (PDT) Received: from localhost.localdomain ([46.248.82.114]) by smtp.gmail.com with ESMTPSA id g6-20020a170906348600b009334219656dsm7381246ejb.56.2023.04.05.07.17.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Apr 2023 07:17:52 -0700 (PDT) From: Uros Bizjak To: linux-alpha@vger.kernel.org, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-perf-users@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Uros Bizjak , Richard Henderson , Ivan Kokshaysky , Matt Turner , Huacai Chen , WANG Xuerui , Jiaxun Yang , Jun Yi , Thomas Bogendoerfer , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Subject: [PATCH v2 3/5] locking/arch: Wire up local_try_cmpxchg Date: Wed, 5 Apr 2023 16:17:08 +0200 Message-Id: <20230405141710.3551-4-ubizjak@gmail.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230405141710.3551-1-ubizjak@gmail.com> References: <20230405141710.3551-1-ubizjak@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-mips@vger.kernel.org Implement target specific support for local_try_cmpxchg and local_cmpxchg using typed C wrappers that call their _local counterpart and provide additional checking of their input arguments. Cc: Richard Henderson Cc: Ivan Kokshaysky Cc: Matt Turner Cc: Huacai Chen Cc: WANG Xuerui Cc: Jiaxun Yang Cc: Jun Yi Cc: Thomas Bogendoerfer Cc: Michael Ellerman Cc: Nicholas Piggin Cc: Christophe Leroy Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: "H. Peter Anvin" Signed-off-by: Uros Bizjak --- arch/alpha/include/asm/local.h | 12 ++++++++++-- arch/loongarch/include/asm/local.h | 13 +++++++++++-- arch/mips/include/asm/local.h | 13 +++++++++++-- arch/powerpc/include/asm/local.h | 11 +++++++++++ arch/x86/include/asm/local.h | 13 +++++++++++-- 5 files changed, 54 insertions(+), 8 deletions(-) diff --git a/arch/alpha/include/asm/local.h b/arch/alpha/include/asm/local.h index fab26a1c93d5..0fcaad642cc3 100644 --- a/arch/alpha/include/asm/local.h +++ b/arch/alpha/include/asm/local.h @@ -52,8 +52,16 @@ static __inline__ long local_sub_return(long i, local_t * l) return result; } -#define local_cmpxchg(l, o, n) \ - (cmpxchg_local(&((l)->a.counter), (o), (n))) +static __inline__ long local_cmpxchg(local_t *l, long old, long new) +{ + return cmpxchg_local(&l->a.counter, old, new); +} + +static __inline__ bool local_try_cmpxchg(local_t *l, long *old, long new) +{ + return try_cmpxchg_local(&l->a.counter, (s64 *)old, new); +} + #define local_xchg(l, n) (xchg_local(&((l)->a.counter), (n))) /** diff --git a/arch/loongarch/include/asm/local.h b/arch/loongarch/include/asm/local.h index 65fbbae9fc4d..83e995b30e47 100644 --- a/arch/loongarch/include/asm/local.h +++ b/arch/loongarch/include/asm/local.h @@ -56,8 +56,17 @@ static inline long local_sub_return(long i, local_t *l) return result; } -#define local_cmpxchg(l, o, n) \ - ((long)cmpxchg_local(&((l)->a.counter), (o), (n))) +static inline long local_cmpxchg(local_t *l, long old, long new) +{ + return cmpxchg_local(&l->a.counter, old, new); +} + +static inline bool local_try_cmpxchg(local_t *l, long *old, long new) +{ + typeof(l->a.counter) *__old = (typeof(l->a.counter) *) old; + return try_cmpxchg_local(&l->a.counter, __old, new); +} + #define local_xchg(l, n) (atomic_long_xchg((&(l)->a), (n))) /** diff --git a/arch/mips/include/asm/local.h b/arch/mips/include/asm/local.h index 08366b1fd273..5daf6fe8e3e9 100644 --- a/arch/mips/include/asm/local.h +++ b/arch/mips/include/asm/local.h @@ -94,8 +94,17 @@ static __inline__ long local_sub_return(long i, local_t * l) return result; } -#define local_cmpxchg(l, o, n) \ - ((long)cmpxchg_local(&((l)->a.counter), (o), (n))) +static __inline__ long local_cmpxchg(local_t *l, long old, long new) +{ + return cmpxchg_local(&l->a.counter, old, new); +} + +static __inline__ bool local_try_cmpxchg(local_t *l, long *old, long new) +{ + typeof(l->a.counter) *__old = (typeof(l->a.counter) *) old; + return try_cmpxchg_local(&l->a.counter, __old, new); +} + #define local_xchg(l, n) (atomic_long_xchg((&(l)->a), (n))) /** diff --git a/arch/powerpc/include/asm/local.h b/arch/powerpc/include/asm/local.h index bc4bd19b7fc2..45492fb5bf22 100644 --- a/arch/powerpc/include/asm/local.h +++ b/arch/powerpc/include/asm/local.h @@ -90,6 +90,17 @@ static __inline__ long local_cmpxchg(local_t *l, long o, long n) return t; } +static __inline__ bool local_try_cmpxchg(local_t *l, long *po, long n) +{ + long o = *po, r; + + r = local_cmpxchg(l, o, n); + if (unlikely(r != o)) + *po = r; + + return likely(r == o); +} + static __inline__ long local_xchg(local_t *l, long n) { long t; diff --git a/arch/x86/include/asm/local.h b/arch/x86/include/asm/local.h index 349a47acaa4a..56d4ef604b91 100644 --- a/arch/x86/include/asm/local.h +++ b/arch/x86/include/asm/local.h @@ -120,8 +120,17 @@ static inline long local_sub_return(long i, local_t *l) #define local_inc_return(l) (local_add_return(1, l)) #define local_dec_return(l) (local_sub_return(1, l)) -#define local_cmpxchg(l, o, n) \ - (cmpxchg_local(&((l)->a.counter), (o), (n))) +static inline long local_cmpxchg(local_t *l, long old, long new) +{ + return cmpxchg_local(&l->a.counter, old, new); +} + +static inline bool local_try_cmpxchg(local_t *l, long *old, long new) +{ + typeof(l->a.counter) *__old = (typeof(l->a.counter) *) old; + return try_cmpxchg_local(&l->a.counter, __old, new); +} + /* Always has a lock prefix */ #define local_xchg(l, n) (xchg(&((l)->a.counter), (n))) From patchwork Wed Apr 5 14:17:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uros Bizjak X-Patchwork-Id: 13201839 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 31397C77B60 for ; Wed, 5 Apr 2023 14:19:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238546AbjDEOTL (ORCPT ); Wed, 5 Apr 2023 10:19:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42898 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238488AbjDEOSd (ORCPT ); Wed, 5 Apr 2023 10:18:33 -0400 Received: from mail-ed1-x535.google.com (mail-ed1-x535.google.com [IPv6:2a00:1450:4864:20::535]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 08DEE30EB; Wed, 5 Apr 2023 07:17:58 -0700 (PDT) Received: by mail-ed1-x535.google.com with SMTP id y4so141541211edo.2; Wed, 05 Apr 2023 07:17:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680704274; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=fOC1m1IyvFx7hqlIu85ltj9Okdj8PWnyI3fah+Gac6o=; b=dw4QAoMpVGLEOeYQ9Yo6hrg7uuDz1mqcfEuifQdcdnb1mCRfnp4USqcIZ/tj+yoqqd NuZo8Fl4HHUAk4my79oi/1gAt96ZaHSFf0NyBNIA2ujrdK0vV+BtAz9cZUh1HP/21L01 ZfOBKFtABC40OHga/xcc6LZLy5bp81QTL5BA33wmnqyDDroTbAF7IVTNrZhDCZYY5uav cpNPwfW8+ndEboG7KmpMryAu2BP5Ht2BhpnVlW+EgEVP3Hv8o6yChfq1iPaWIKJDgzD/ 9Y/9bUrxWUeU4eBT8lu+LsRKzoAiRRbIgYAetGEXNMYLkTxdyCEUikhfLAP1CktMT2oa eHBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680704274; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=fOC1m1IyvFx7hqlIu85ltj9Okdj8PWnyI3fah+Gac6o=; b=fskONODB1V2ldmXITXJT7s4WB7h4edhjrUT+MGQFpF0nH9IS/QkXNe5utcGDXzwRp+ w/zKKNyUEpVQ5EUupAmjs6CBx0TYYf4YF1ny0RWr42PyP+Tj5JXlKNn1nyh8isg7nlsP m2V5fX94uITi8CRYliQ0dOehfPPcV12tejLwIW878oMwViKVd7FignWve93uDqjQGFhC 4negMbRNtZg8rBYCBHEY8nvgxWFZkY3IHMF2wpdhN8t7p/Pl+AV+YK74hw0/FF233dFB 97JqOnhHJMp6ifCFLKTBOM4XWJ5+d6l1P3i7B0r0U5gmqDqqos95PmTtXzDB2RCdoGJU lyTw== X-Gm-Message-State: AAQBX9c/6ONW1L39/eVXhms6kiJNrer/BlNP0/rPShlpDjJE6ejYgQbm jcF2ePQN5VozB+5gyCrxw8E3ASVtIbnZGhXp X-Google-Smtp-Source: AKy350bz5YnNlQahgd6VXOTs26Uhu2FUvH9WVqbCQND0Gop8QRbR/cXdIQACyHoPPOUtbLiBje74GQ== X-Received: by 2002:a17:906:4a8b:b0:944:43e:7983 with SMTP id x11-20020a1709064a8b00b00944043e7983mr3280252eju.67.1680704274221; Wed, 05 Apr 2023 07:17:54 -0700 (PDT) Received: from localhost.localdomain ([46.248.82.114]) by smtp.gmail.com with ESMTPSA id g6-20020a170906348600b009334219656dsm7381246ejb.56.2023.04.05.07.17.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Apr 2023 07:17:53 -0700 (PDT) From: Uros Bizjak To: linux-alpha@vger.kernel.org, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-perf-users@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Uros Bizjak , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Subject: [PATCH v2 4/5] locking/x86: Define arch_try_cmpxchg_local Date: Wed, 5 Apr 2023 16:17:09 +0200 Message-Id: <20230405141710.3551-5-ubizjak@gmail.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230405141710.3551-1-ubizjak@gmail.com> References: <20230405141710.3551-1-ubizjak@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-mips@vger.kernel.org Define target specific arch_try_cmpxchg_local. This definition overrides the generic arch_try_cmpxchg_local fallback definition and enables target-specific implementation of try_cmpxchg_local. Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: "H. Peter Anvin" Signed-off-by: Uros Bizjak --- arch/x86/include/asm/cmpxchg.h | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/arch/x86/include/asm/cmpxchg.h b/arch/x86/include/asm/cmpxchg.h index 94fbe6ae7431..540573f515b7 100644 --- a/arch/x86/include/asm/cmpxchg.h +++ b/arch/x86/include/asm/cmpxchg.h @@ -221,9 +221,15 @@ extern void __add_wrong_size(void) #define __try_cmpxchg(ptr, pold, new, size) \ __raw_try_cmpxchg((ptr), (pold), (new), (size), LOCK_PREFIX) +#define __try_cmpxchg_local(ptr, pold, new, size) \ + __raw_try_cmpxchg((ptr), (pold), (new), (size), "") + #define arch_try_cmpxchg(ptr, pold, new) \ __try_cmpxchg((ptr), (pold), (new), sizeof(*(ptr))) +#define arch_try_cmpxchg_local(ptr, pold, new) \ + __try_cmpxchg_local((ptr), (pold), (new), sizeof(*(ptr))) + /* * xadd() adds "inc" to "*ptr" and atomically returns the previous * value of "*ptr". From patchwork Wed Apr 5 14:17:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uros Bizjak X-Patchwork-Id: 13201840 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0503C76188 for ; Wed, 5 Apr 2023 14:19:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238570AbjDEOTT (ORCPT ); Wed, 5 Apr 2023 10:19:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42826 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238420AbjDEOSe (ORCPT ); Wed, 5 Apr 2023 10:18:34 -0400 Received: from mail-ed1-x52e.google.com (mail-ed1-x52e.google.com [IPv6:2a00:1450:4864:20::52e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DA25A30D4; Wed, 5 Apr 2023 07:18:03 -0700 (PDT) Received: by mail-ed1-x52e.google.com with SMTP id ek18so141537725edb.6; Wed, 05 Apr 2023 07:18:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680704276; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=tX2Wz3EA82OevW9OfamI7/Kvuo0JRd1kfWWZWf7tBPI=; b=dcJpf62STM2t7wBeMFWRHA65D4e8ki42FaDdkuuO3agKqVH6m85Cy4lQ8Jc6WqjxoF fnnlug9El9w16kuSLGTsPbq74oNaQ+P8JDxMAcNUZHQhE8u+5jySMEVYMmtE4egR+MrD Yo63AeJfrWzh2LWw4CUsO/Jz8F4pDt2KwqewSmosxEslt2xhDsBYXNkotYBZ0u2SUb3b bY44Ptodk1vZVbJl1WFw1p9lD7pDptPbaK3ykNtKWYarwScxbwyxAFGsy3vIZ++4+u08 7sJWtj5rpBofs9g4DRnm1uwhZ6aYf1PTSfC6VMfZ4+8IxgDtdbMLLd8hp6MXeYVptMAQ HEzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680704276; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tX2Wz3EA82OevW9OfamI7/Kvuo0JRd1kfWWZWf7tBPI=; b=8Fv0C2oIfqzNYT5EW+Tebedn6a1Ldu9NM/vBrhRq7CyNBY4FmBYj1uhTF7Su/c3M3U +bWh7mJFhOE1/scfDX5mvoD2CZ82P908e80ZyKPVSrvtDNI0hj6ymAgzKJtKhUOsfWM7 l4bIzBj8JybWExmvd3VaHc4+N3Q2mB6iGELdrq6QJtXWqQbpwDMytWmQIJE6PKpcS3QU FH5U+nl3f7O99ZIKoNUQ1OKdMilAhOAthnIAGECaiYW3qwT1bHmpzmRRPNfY4QfG19iO UL42ezOQuqlON9jKxv5jXBDSzRDbK5/JHopJXBGB/ozSxj3r7tiUU6mbWVcLO6yATzST F9tg== X-Gm-Message-State: AAQBX9fc0h4azbPJSz0jkkBg6V292q7RU6RF1kS7yKIg9+57Iup0iB2W bmLvBV9umLIo0SrmmU0i+mQ7yrs+Fs0vyMyw X-Google-Smtp-Source: AKy350ZbYZNSgst5kJdLcblY6mgsM4CglBWBhr5I5xeGEgTa/9eOQOFET04v+j0TrQ94wu4NswT8dQ== X-Received: by 2002:a17:906:53c3:b0:947:791b:fdcb with SMTP id p3-20020a17090653c300b00947791bfdcbmr2772881ejo.21.1680704276052; Wed, 05 Apr 2023 07:17:56 -0700 (PDT) Received: from localhost.localdomain ([46.248.82.114]) by smtp.gmail.com with ESMTPSA id g6-20020a170906348600b009334219656dsm7381246ejb.56.2023.04.05.07.17.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Apr 2023 07:17:55 -0700 (PDT) From: Uros Bizjak To: linux-alpha@vger.kernel.org, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-arch@vger.kernel.org, linux-perf-users@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Uros Bizjak Subject: [PATCH v2 5/5] events: Illustrate the transition to local{,64}_try_cmpxchg Date: Wed, 5 Apr 2023 16:17:10 +0200 Message-Id: <20230405141710.3551-6-ubizjak@gmail.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230405141710.3551-1-ubizjak@gmail.com> References: <20230405141710.3551-1-ubizjak@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-mips@vger.kernel.org This patch illustrates the transition to local{,64}_try_cmpxchg. It is not intended to be merged as-is. Signed-off-by: Uros Bizjak --- arch/x86/events/core.c | 9 ++++----- kernel/events/ring_buffer.c | 5 +++-- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/arch/x86/events/core.c b/arch/x86/events/core.c index d096b04bf80e..d9310e9363f1 100644 --- a/arch/x86/events/core.c +++ b/arch/x86/events/core.c @@ -129,13 +129,12 @@ u64 x86_perf_event_update(struct perf_event *event) * exchange a new raw count - then add that new-prev delta * count to the generic event atomically: */ -again: prev_raw_count = local64_read(&hwc->prev_count); - rdpmcl(hwc->event_base_rdpmc, new_raw_count); - if (local64_cmpxchg(&hwc->prev_count, prev_raw_count, - new_raw_count) != prev_raw_count) - goto again; + do { + rdpmcl(hwc->event_base_rdpmc, new_raw_count); + } while (!local64_try_cmpxchg(&hwc->prev_count, &prev_raw_count, + new_raw_count)); /* * Now we have the new raw value and have updated the prev diff --git a/kernel/events/ring_buffer.c b/kernel/events/ring_buffer.c index 273a0fe7910a..111ab85ee97d 100644 --- a/kernel/events/ring_buffer.c +++ b/kernel/events/ring_buffer.c @@ -191,9 +191,10 @@ __perf_output_begin(struct perf_output_handle *handle, perf_output_get_handle(handle); + offset = local_read(&rb->head); do { tail = READ_ONCE(rb->user_page->data_tail); - offset = head = local_read(&rb->head); + head = offset; if (!rb->overwrite) { if (unlikely(!ring_buffer_has_space(head, tail, perf_data_size(rb), @@ -217,7 +218,7 @@ __perf_output_begin(struct perf_output_handle *handle, head += size; else head -= size; - } while (local_cmpxchg(&rb->head, offset, head) != offset); + } while (!local_try_cmpxchg(&rb->head, &offset, head)); if (backward) { offset = head;