From patchwork Wed Aug 9 02:13:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leonardo Bras X-Patchwork-Id: 13347335 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4C01CC001DB for ; Wed, 9 Aug 2023 02:14:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Avx4GgzS5eYJhbU16Xi4I4BzjD0lLLUmJWZ2WyALGjs=; b=36c1iVkHS+ThII enIQ3o3LPIh1uRkvi2at8evULMeI2z0pffcCP+NCX1aRigB7S0YS4jlMeBzDshy9oDyMkBsMhuVsh iR9S+HStYi3pC5F1KW9JVLsq/kFUD4BYcFku8oAxROMwOGVQZ1XH/Du6F2Jq/9T2GMJRc90CzgdXW JPoIWsY5X6z4xtCLORJF4RzR40bLCyrFKI0Ns0KSSxdfyHtQqudajr2CCSdVtTa/8ukSC0rzxrK6/ xrrZLaykum5KyVEtJDLsO9QNDbUF8EDsp913BatxOnbdUieysiPjZ9VQwRee1/ygYxBEbJG06s7O8 M790ISOJvqRAdf+lwR2Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qTYi5-003sVQ-2y; Wed, 09 Aug 2023 02:14:05 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qTYi2-003sUn-2d for linux-riscv@lists.infradead.org; Wed, 09 Aug 2023 02:14:04 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1691547241; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cG8axZDwBuWJQ+k2FsV81xz+WKLWi2QNSuhJ6eUeaqw=; b=EQzQH4IB2Jt1UtQ5iFTsJSpJjidMzpeiw/3PrT5dTf1mawdtXd8rN9yhhZ/fv0Hkp/hDZU 1Fv2PIn7yTZEgcBqT3PAMe9NTh273EbIRggn/FfkmvJl70GAcMuhBYfv7pxDCc34ehVHt/ WmuC6Bv+Xr5KkNy3GX6B8h3N1Et2Gf0= Received: from mail-oa1-f71.google.com (mail-oa1-f71.google.com [209.85.160.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-145-mG_5LcyqNSqsl8ZeOWfeNA-1; Tue, 08 Aug 2023 22:13:58 -0400 X-MC-Unique: mG_5LcyqNSqsl8ZeOWfeNA-1 Received: by mail-oa1-f71.google.com with SMTP id 586e51a60fabf-1befca4fdfaso10794453fac.2 for ; Tue, 08 Aug 2023 19:13:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691547238; x=1692152038; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cG8axZDwBuWJQ+k2FsV81xz+WKLWi2QNSuhJ6eUeaqw=; b=TX90+HC7cVaThLh89g3MRAvbT4xDYyIHHQPEPHyj8XwbM0jbah6S02qCEkY9CDdgl4 ohkLxN2aNYyuvdEdhSB0zyeL19RwFgRZwgrPy6P4TB8OxIfr9caXXVXfjY8CyAmTHQbe +BwVcXKCk04eyeB3l6Tbl3WNPq6GM8w2oBt1XKR8lSn3U6h28Z0kmMgRE+AkQIt4sA8G hIKC/INSpL/I5fJODVWAPKfhLzX+BtC7QG5CwwmrcS/MiRHlTwSPjlu1m/AzLo34pfSv fiSxIzlObcQ1SxrAV8ZELFb7s2wsqi9PIvUY34B7dZQwL8/7naUygqFLDQ7OXK8jdzC9 UHVg== X-Gm-Message-State: AOJu0YzeFtYrsy2ZIGOuA3jaX+5phLnCUMq4Ulv+j/pFKce3CcAke7Ml KzFbJ8/72TY2ZPR5cw/s5ySgAClYKPGfAsSX+g2FHiiKZ/0CIHJMdUerShu5CEcnIWQile6Ac7d vXakSVYjmh5iLWYHNhfqNgNzAEKIZ X-Received: by 2002:a05:6870:c084:b0:1bf:4f5e:55e2 with SMTP id c4-20020a056870c08400b001bf4f5e55e2mr1543833oad.53.1691547237742; Tue, 08 Aug 2023 19:13:57 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHukU42SPYGuHy2oEtAblgnpaB5UkpskCwb9snxi5LMKs5NyCZyz7VTISypO0xyDpTH5uoIvA== X-Received: by 2002:a05:6870:c084:b0:1bf:4f5e:55e2 with SMTP id c4-20020a056870c08400b001bf4f5e55e2mr1543823oad.53.1691547237498; Tue, 08 Aug 2023 19:13:57 -0700 (PDT) Received: from localhost.localdomain ([2804:431:c7ec:e667:6b7d:ed55:c363:a088]) by smtp.gmail.com with ESMTPSA id v10-20020a4a8c4a000000b0054fba751207sm6475518ooj.47.2023.08.08.19.13.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Aug 2023 19:13:57 -0700 (PDT) From: Leonardo Bras To: Will Deacon , Peter Zijlstra , Boqun Feng , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Leonardo Bras , Andrea Parri , Ingo Molnar , Geert Uytterhoeven , Andrzej Hajda , Palmer Dabbelt , Guo Ren Cc: linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org Subject: [RFC PATCH v4 1/5] riscv/cmpxchg: Deduplicate xchg() asm functions Date: Tue, 8 Aug 2023 23:13:06 -0300 Message-ID: <20230809021311.1390578-3-leobras@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230809021311.1390578-2-leobras@redhat.com> References: <20230809021311.1390578-2-leobras@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230808_191402_928065_03CE1741 X-CRM114-Status: GOOD ( 12.08 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org In this header every xchg define (_relaxed, _acquire, _release, vanilla) contain it's own asm file, both for 4-byte variables an 8-byte variables, on a total of 8 versions of mostly the same asm. This is usually bad, as it means any change may be done in up to 8 different places. Unify those versions by creating a new define with enough parameters to generate any version of the previous 8. Then unify the result under a more general define, and simplify arch_xchg* generation. (This did not cause any change in generated asm) Signed-off-by: Leonardo Bras Reviewed-by: Andrea Parri --- arch/riscv/include/asm/cmpxchg.h | 136 +++++-------------------------- 1 file changed, 22 insertions(+), 114 deletions(-) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index 2f4726d3cfcc..ec4ea4f3f908 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -11,60 +11,30 @@ #include #include -#define __xchg_relaxed(ptr, new, size) \ +#define __arch_xchg(sfx, prepend, append, r, p, n) \ ({ \ - __typeof__(ptr) __ptr = (ptr); \ - __typeof__(new) __new = (new); \ - __typeof__(*(ptr)) __ret; \ - switch (size) { \ - case 4: \ - __asm__ __volatile__ ( \ - " amoswap.w %0, %2, %1\n" \ - : "=r" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ - break; \ - case 8: \ - __asm__ __volatile__ ( \ - " amoswap.d %0, %2, %1\n" \ - : "=r" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ - break; \ - default: \ - BUILD_BUG(); \ - } \ - __ret; \ -}) - -#define arch_xchg_relaxed(ptr, x) \ -({ \ - __typeof__(*(ptr)) _x_ = (x); \ - (__typeof__(*(ptr))) __xchg_relaxed((ptr), \ - _x_, sizeof(*(ptr))); \ + __asm__ __volatile__ ( \ + prepend \ + " amoswap" sfx " %0, %2, %1\n" \ + append \ + : "=r" (r), "+A" (*(p)) \ + : "r" (n) \ + : "memory"); \ }) -#define __xchg_acquire(ptr, new, size) \ +#define _arch_xchg(ptr, new, sfx, prepend, append) \ ({ \ - __typeof__(ptr) __ptr = (ptr); \ - __typeof__(new) __new = (new); \ + __typeof__(*(ptr)) __new = (new); \ __typeof__(*(ptr)) __ret; \ - switch (size) { \ + __typeof__(ptr) __ptr = (ptr); \ + switch (sizeof(*__ptr)) { \ case 4: \ - __asm__ __volatile__ ( \ - " amoswap.w %0, %2, %1\n" \ - RISCV_ACQUIRE_BARRIER \ - : "=r" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ + __arch_xchg(".w" sfx, prepend, append, \ + __ret, __ptr, __new); \ break; \ case 8: \ - __asm__ __volatile__ ( \ - " amoswap.d %0, %2, %1\n" \ - RISCV_ACQUIRE_BARRIER \ - : "=r" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ + __arch_xchg(".d" sfx, prepend, append, \ + __ret, __ptr, __new); \ break; \ default: \ BUILD_BUG(); \ @@ -72,79 +42,17 @@ __ret; \ }) -#define arch_xchg_acquire(ptr, x) \ -({ \ - __typeof__(*(ptr)) _x_ = (x); \ - (__typeof__(*(ptr))) __xchg_acquire((ptr), \ - _x_, sizeof(*(ptr))); \ -}) +#define arch_xchg_relaxed(ptr, x) \ + _arch_xchg(ptr, x, "", "", "") -#define __xchg_release(ptr, new, size) \ -({ \ - __typeof__(ptr) __ptr = (ptr); \ - __typeof__(new) __new = (new); \ - __typeof__(*(ptr)) __ret; \ - switch (size) { \ - case 4: \ - __asm__ __volatile__ ( \ - RISCV_RELEASE_BARRIER \ - " amoswap.w %0, %2, %1\n" \ - : "=r" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ - break; \ - case 8: \ - __asm__ __volatile__ ( \ - RISCV_RELEASE_BARRIER \ - " amoswap.d %0, %2, %1\n" \ - : "=r" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ - break; \ - default: \ - BUILD_BUG(); \ - } \ - __ret; \ -}) +#define arch_xchg_acquire(ptr, x) \ + _arch_xchg(ptr, x, "", "", RISCV_ACQUIRE_BARRIER) #define arch_xchg_release(ptr, x) \ -({ \ - __typeof__(*(ptr)) _x_ = (x); \ - (__typeof__(*(ptr))) __xchg_release((ptr), \ - _x_, sizeof(*(ptr))); \ -}) - -#define __arch_xchg(ptr, new, size) \ -({ \ - __typeof__(ptr) __ptr = (ptr); \ - __typeof__(new) __new = (new); \ - __typeof__(*(ptr)) __ret; \ - switch (size) { \ - case 4: \ - __asm__ __volatile__ ( \ - " amoswap.w.aqrl %0, %2, %1\n" \ - : "=r" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ - break; \ - case 8: \ - __asm__ __volatile__ ( \ - " amoswap.d.aqrl %0, %2, %1\n" \ - : "=r" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ - break; \ - default: \ - BUILD_BUG(); \ - } \ - __ret; \ -}) + _arch_xchg(ptr, x, "", RISCV_RELEASE_BARRIER, "") #define arch_xchg(ptr, x) \ -({ \ - __typeof__(*(ptr)) _x_ = (x); \ - (__typeof__(*(ptr))) __arch_xchg((ptr), _x_, sizeof(*(ptr))); \ -}) + _arch_xchg(ptr, x, ".aqrl", "", "") #define xchg32(ptr, x) \ ({ \ From patchwork Wed Aug 9 02:13:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leonardo Bras X-Patchwork-Id: 13347337 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9FB38C04A94 for ; Wed, 9 Aug 2023 02:14:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=GhbQNzoGMzCtmYEY/k+AVnlFkcaxvo5JicgInhaH/vo=; b=UfXidhBKHLd0kK ZF/v66gTuMB1Kj2rjH/1mMFxiP7ftau/11uinjlmbAjPbP78o9rEsC8DG3YemQIR27yIG5N3GgPHJ NKR5SrA7+m/+IkpqzmyzolHBPNiI5ppjU1E6lqm5fDVlTG9k6lHza74al2uAL680EZwXJEZhnwte1 YAngjehiUSuuyxZ8ya9LPBh21GoAbCI9xSHWT4uoO+0klLENVDV4000UpOYZX0UT7CwVEqNUu8jVi ACHH1JCL7p/nQrx1Kk/TwysFqa7T5SLyOfHX/ZTsVuF8Z/kEnBeGZRsz9bZll/CMLq5bCDdkhvdNV vOsijgB0eoOCFHbKvQTw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qTYi9-003sWV-1y; Wed, 09 Aug 2023 02:14:09 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qTYi6-003sVP-0C for linux-riscv@lists.infradead.org; Wed, 09 Aug 2023 02:14:08 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1691547245; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=68droSgtD1yUeWujN9KsDS4jg7ssA0NCkb8PZ8Tb+SA=; b=ifLKCBrTkbde3XxZnKcm/m6DiScKFCzokIMTbsLRtriUBZpguxbTkbzH34ri7Hw1VujHTV VcUfjmqJdVyy9HzBav87WQkkloP9yJg6T0na94CPFzXXxKRi2ofblkkNugzS0LV08pBxQg ppYl0F/d4ax8R3r0UcZDdhCY5FYeG9E= Received: from mail-oo1-f71.google.com (mail-oo1-f71.google.com [209.85.161.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-471-pV8jn7XNNvmeHRZY8H--eg-1; Tue, 08 Aug 2023 22:14:03 -0400 X-MC-Unique: pV8jn7XNNvmeHRZY8H--eg-1 Received: by mail-oo1-f71.google.com with SMTP id 006d021491bc7-56c7404fafdso10497304eaf.1 for ; Tue, 08 Aug 2023 19:14:03 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691547243; x=1692152043; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=68droSgtD1yUeWujN9KsDS4jg7ssA0NCkb8PZ8Tb+SA=; b=Eq7wbOldM9iM98P2TqShaC8hoN/4eka0LidWXRPlNfDFRwSXkMHZH/T0CYN1g7Jcvs vNB9km/agp/8OHV9zhpQ3dji7q6yMz7BGQcculjekwPFo5UcDXCub7CTkyfZS7lUqNxv H19C5/cgU7we62ZPpKHHTyrqmlTUu72fM0/29M0vaBOW4riKsqf54oa9U0bj4p+0UyMu tWYUu9o4ZE6aQV7fZY4ivWY5qFc5DTqCvqDxxB+UZtyBestknvDlcoVvoS8C0U+DR+Fx Gcwn5lW1OsGnwcB/YBUqFQU7vBfUU4Ljo2Ly7uh9Kw2ynucjHHftAeLTu4AaCG1chqMg 5Jgg== X-Gm-Message-State: AOJu0Yzm99pOKR/zGoKJvOfMAkSPBE1yqQ4GQNxSG1tyXHLlxVEKHiY3 8dIi6ffV/EhBmWQxfMu/d95IKc23NlkIQpzkHnNOzg+y6c22nxwyG91OtOua7/289sqi6D8Jh9f 6j0566q4IjZkhtrTWjhfBS4B6MYfl X-Received: by 2002:a4a:3005:0:b0:56c:aa7a:eaeb with SMTP id q5-20020a4a3005000000b0056caa7aeaebmr1376295oof.4.1691547243033; Tue, 08 Aug 2023 19:14:03 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFRBi7ZrL7X8EpzKEKv574j/yN24o/CUG+jxNQTD3De3Bdux9OGGevu77kaKC1CyIqeEg4tkw== X-Received: by 2002:a4a:3005:0:b0:56c:aa7a:eaeb with SMTP id q5-20020a4a3005000000b0056caa7aeaebmr1376278oof.4.1691547242715; Tue, 08 Aug 2023 19:14:02 -0700 (PDT) Received: from localhost.localdomain ([2804:431:c7ec:e667:6b7d:ed55:c363:a088]) by smtp.gmail.com with ESMTPSA id v10-20020a4a8c4a000000b0054fba751207sm6475518ooj.47.2023.08.08.19.13.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Aug 2023 19:14:02 -0700 (PDT) From: Leonardo Bras To: Will Deacon , Peter Zijlstra , Boqun Feng , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Leonardo Bras , Andrea Parri , Ingo Molnar , Geert Uytterhoeven , Andrzej Hajda , Palmer Dabbelt , Guo Ren Cc: linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org Subject: [RFC PATCH v4 2/5] riscv/cmpxchg: Deduplicate cmpxchg() asm and macros Date: Tue, 8 Aug 2023 23:13:07 -0300 Message-ID: <20230809021311.1390578-4-leobras@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230809021311.1390578-2-leobras@redhat.com> References: <20230809021311.1390578-2-leobras@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230808_191406_175597_C7F9F2AB X-CRM114-Status: GOOD ( 12.40 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org In this header every cmpxchg define (_relaxed, _acquire, _release, vanilla) contain it's own asm file, both for 4-byte variables an 8-byte variables, on a total of 8 versions of mostly the same asm. This is usually bad, as it means any change may be done in up to 8 different places. Unify those versions by creating a new define with enough parameters to generate any version of the previous 8. Then unify the result under a more general define, and simplify arch_cmpxchg* generation (This did not cause any change in generated asm) Signed-off-by: Leonardo Bras Reviewed-by: Andrea Parri --- arch/riscv/include/asm/cmpxchg.h | 187 +++++-------------------------- 1 file changed, 29 insertions(+), 158 deletions(-) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index ec4ea4f3f908..5a07646fae65 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -71,127 +71,40 @@ * store NEW in MEM. Return the initial value in MEM. Success is * indicated by comparing RETURN with OLD. */ -#define __cmpxchg_relaxed(ptr, old, new, size) \ -({ \ - __typeof__(ptr) __ptr = (ptr); \ - __typeof__(*(ptr)) __old = (old); \ - __typeof__(*(ptr)) __new = (new); \ - __typeof__(*(ptr)) __ret; \ - register unsigned int __rc; \ - switch (size) { \ - case 4: \ - __asm__ __volatile__ ( \ - "0: lr.w %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc.w %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - "1:\n" \ - : "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \ - : "rJ" ((long)__old), "rJ" (__new) \ - : "memory"); \ - break; \ - case 8: \ - __asm__ __volatile__ ( \ - "0: lr.d %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc.d %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - "1:\n" \ - : "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \ - : "rJ" (__old), "rJ" (__new) \ - : "memory"); \ - break; \ - default: \ - BUILD_BUG(); \ - } \ - __ret; \ -}) -#define arch_cmpxchg_relaxed(ptr, o, n) \ -({ \ - __typeof__(*(ptr)) _o_ = (o); \ - __typeof__(*(ptr)) _n_ = (n); \ - (__typeof__(*(ptr))) __cmpxchg_relaxed((ptr), \ - _o_, _n_, sizeof(*(ptr))); \ -}) -#define __cmpxchg_acquire(ptr, old, new, size) \ +#define __arch_cmpxchg(lr_sfx, sc_sfx, prepend, append, r, p, co, o, n) \ ({ \ - __typeof__(ptr) __ptr = (ptr); \ - __typeof__(*(ptr)) __old = (old); \ - __typeof__(*(ptr)) __new = (new); \ - __typeof__(*(ptr)) __ret; \ register unsigned int __rc; \ - switch (size) { \ - case 4: \ - __asm__ __volatile__ ( \ - "0: lr.w %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc.w %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - RISCV_ACQUIRE_BARRIER \ - "1:\n" \ - : "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \ - : "rJ" ((long)__old), "rJ" (__new) \ - : "memory"); \ - break; \ - case 8: \ - __asm__ __volatile__ ( \ - "0: lr.d %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc.d %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - RISCV_ACQUIRE_BARRIER \ - "1:\n" \ - : "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \ - : "rJ" (__old), "rJ" (__new) \ - : "memory"); \ - break; \ - default: \ - BUILD_BUG(); \ - } \ - __ret; \ -}) - -#define arch_cmpxchg_acquire(ptr, o, n) \ -({ \ - __typeof__(*(ptr)) _o_ = (o); \ - __typeof__(*(ptr)) _n_ = (n); \ - (__typeof__(*(ptr))) __cmpxchg_acquire((ptr), \ - _o_, _n_, sizeof(*(ptr))); \ + \ + __asm__ __volatile__ ( \ + prepend \ + "0: lr" lr_sfx " %0, %2\n" \ + " bne %0, %z3, 1f\n" \ + " sc" sc_sfx " %1, %z4, %2\n" \ + " bnez %1, 0b\n" \ + append \ + "1:\n" \ + : "=&r" (r), "=&r" (__rc), "+A" (*(p)) \ + : "rJ" (co o), "rJ" (n) \ + : "memory"); \ }) -#define __cmpxchg_release(ptr, old, new, size) \ +#define _arch_cmpxchg(ptr, old, new, sc_sfx, prepend, append) \ ({ \ __typeof__(ptr) __ptr = (ptr); \ __typeof__(*(ptr)) __old = (old); \ __typeof__(*(ptr)) __new = (new); \ __typeof__(*(ptr)) __ret; \ - register unsigned int __rc; \ - switch (size) { \ + \ + switch (sizeof(*__ptr)) { \ case 4: \ - __asm__ __volatile__ ( \ - RISCV_RELEASE_BARRIER \ - "0: lr.w %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc.w %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - "1:\n" \ - : "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \ - : "rJ" ((long)__old), "rJ" (__new) \ - : "memory"); \ + __arch_cmpxchg(".w", ".w" sc_sfx, prepend, append, \ + __ret, __ptr, (long), __old, __new); \ break; \ case 8: \ - __asm__ __volatile__ ( \ - RISCV_RELEASE_BARRIER \ - "0: lr.d %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc.d %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - "1:\n" \ - : "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \ - : "rJ" (__old), "rJ" (__new) \ - : "memory"); \ + __arch_cmpxchg(".d", ".d" sc_sfx, prepend, append, \ + __ret, __ptr, /**/, __old, __new); \ break; \ default: \ BUILD_BUG(); \ @@ -199,62 +112,20 @@ __ret; \ }) -#define arch_cmpxchg_release(ptr, o, n) \ -({ \ - __typeof__(*(ptr)) _o_ = (o); \ - __typeof__(*(ptr)) _n_ = (n); \ - (__typeof__(*(ptr))) __cmpxchg_release((ptr), \ - _o_, _n_, sizeof(*(ptr))); \ -}) +#define arch_cmpxchg_relaxed(ptr, o, n) \ + _arch_cmpxchg((ptr), (o), (n), "", "", "") -#define __cmpxchg(ptr, old, new, size) \ -({ \ - __typeof__(ptr) __ptr = (ptr); \ - __typeof__(*(ptr)) __old = (old); \ - __typeof__(*(ptr)) __new = (new); \ - __typeof__(*(ptr)) __ret; \ - register unsigned int __rc; \ - switch (size) { \ - case 4: \ - __asm__ __volatile__ ( \ - "0: lr.w %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc.w.rl %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - " fence rw, rw\n" \ - "1:\n" \ - : "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \ - : "rJ" ((long)__old), "rJ" (__new) \ - : "memory"); \ - break; \ - case 8: \ - __asm__ __volatile__ ( \ - "0: lr.d %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc.d.rl %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - " fence rw, rw\n" \ - "1:\n" \ - : "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \ - : "rJ" (__old), "rJ" (__new) \ - : "memory"); \ - break; \ - default: \ - BUILD_BUG(); \ - } \ - __ret; \ -}) +#define arch_cmpxchg_acquire(ptr, o, n) \ + _arch_cmpxchg((ptr), (o), (n), "", "", RISCV_ACQUIRE_BARRIER) + +#define arch_cmpxchg_release(ptr, o, n) \ + _arch_cmpxchg((ptr), (o), (n), "", RISCV_RELEASE_BARRIER, "") #define arch_cmpxchg(ptr, o, n) \ -({ \ - __typeof__(*(ptr)) _o_ = (o); \ - __typeof__(*(ptr)) _n_ = (n); \ - (__typeof__(*(ptr))) __cmpxchg((ptr), \ - _o_, _n_, sizeof(*(ptr))); \ -}) + _arch_cmpxchg((ptr), (o), (n), ".rl", "", " fence rw, rw\n") #define arch_cmpxchg_local(ptr, o, n) \ - (__cmpxchg_relaxed((ptr), (o), (n), sizeof(*(ptr)))) + arch_cmpxchg_relaxed((ptr), (o), (n)) #define arch_cmpxchg64(ptr, o, n) \ ({ \ From patchwork Wed Aug 9 02:13:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leonardo Bras X-Patchwork-Id: 13347336 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BE33EC001DB for ; Wed, 9 Aug 2023 02:14:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=VfsN9GKoSqwjoxuQU66Q7/OyJSWiB1T4NgFHlghaRXc=; b=XjmjXid/x1IJ3S DUaAecEZSYwYx6Hm89b92d8oyt9QXecF7huO+Mj2vNS6EZE7IpM3QOJbCtU8EyKNW88jlTvBkZb7w AuadRA5oRHhteH4EzyDb6qt/sTGKHDX5NPmf/HNmML4c/My79dl2dn+ft6loPnJo03a50PWlIZeFF yNrZQoteW6ERInaI2P/fYj8W81wDgftV+S5v46exjj/7n1d/r9iEYj+T1YCfmwwbcrdhZyyfX+McZ Hkq9qYXly+HYq1QdtJsc4Io7BHj+5m9ySVTnV9YjOgI5NxX73bUgbFqa7chmYguNSXaU4BZbvKFI7 TDkxA045AGSodHQsZ6HQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qTYiE-003sY4-0v; Wed, 09 Aug 2023 02:14:14 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qTYiB-003sX3-20 for linux-riscv@lists.infradead.org; Wed, 09 Aug 2023 02:14:13 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1691547250; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UzQb2VuQCXwMdF/Rvu48wxhx8ntZSb1KliAOBLDDWgc=; b=bJ0Fny7Dcn/T6HCRMIVt86N/gB6OfFsS5LJiMeIJ8k8smooMC+0tbdibzWNOe8V/LVJsDX FhKOvIgPecKvRxdvmzZK4Wpcs3vly9f9w/0dCE4tOuH+QmdQE38QQBfwpR7AcS84iti469 8g7Xaa5hwE9HQp8ZEeK6KL7cJ3nOJAY= Received: from mail-oi1-f197.google.com (mail-oi1-f197.google.com [209.85.167.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-534-lLjIWAmhPKWSBXkXBNUpKg-1; Tue, 08 Aug 2023 22:14:09 -0400 X-MC-Unique: lLjIWAmhPKWSBXkXBNUpKg-1 Received: by mail-oi1-f197.google.com with SMTP id 5614622812f47-3a483c86b74so12417046b6e.0 for ; Tue, 08 Aug 2023 19:14:09 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691547249; x=1692152049; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UzQb2VuQCXwMdF/Rvu48wxhx8ntZSb1KliAOBLDDWgc=; b=N2vxPj3zLPJYrSmJGtWCcvet0BDiV1OjoEgq4epH2333Avimdy7kAkZrjAp8+1GWEZ MTOzdODUv/lO60lnYSbhARdx7bPLTH8pTzGCrFTS6+QlVXxwxtVB36WGuVk5sMdcRPli +DKjfV7yU5cKDNJzPstzDjph7fw8bj1COBk96W3x+HEFU6N0RgyiC7CAyYAZyiUOnSbC fZLuv/fVU1bxHBFuwro16I/NQjrWGc+mkcGQ2Ur8s4PcRktcEDqTdY09BJQjw/mRBCOx mXukR4XHHU3rAgmQtH1KN+nB8ePSH6ArMTQUD4J1co03wvso+EXgauQ+H52zihY1yphf gdyg== X-Gm-Message-State: AOJu0YxBu3DB2kY335Cix8kcImQ1YVpDeFh3jbk7aQnGVUHDyU7J/ylb WlO/uXx24WheddWnD9oUQCZxkISPLiGrFMDkGS6IuCJ1LpxYJ4KVtFxYD/tgufenwGY0UeawW8y 1snB11e758NVn46DlCxZJuIlBQ4eo X-Received: by 2002:a05:6870:c082:b0:1bf:74cc:c815 with SMTP id c2-20020a056870c08200b001bf74ccc815mr1491707oad.19.1691547248948; Tue, 08 Aug 2023 19:14:08 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGzmTXNkLqFmS1q1/FR/HeyFRU6fl+VZuoTT+xk0Lf/Zu30HJE0YhZ23Lp17N5hlJIuru2Olg== X-Received: by 2002:a05:6870:c082:b0:1bf:74cc:c815 with SMTP id c2-20020a056870c08200b001bf74ccc815mr1491689oad.19.1691547248647; Tue, 08 Aug 2023 19:14:08 -0700 (PDT) Received: from localhost.localdomain ([2804:431:c7ec:e667:6b7d:ed55:c363:a088]) by smtp.gmail.com with ESMTPSA id v10-20020a4a8c4a000000b0054fba751207sm6475518ooj.47.2023.08.08.19.14.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Aug 2023 19:14:08 -0700 (PDT) From: Leonardo Bras To: Will Deacon , Peter Zijlstra , Boqun Feng , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Leonardo Bras , Andrea Parri , Ingo Molnar , Geert Uytterhoeven , Andrzej Hajda , Palmer Dabbelt , Guo Ren Cc: linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org Subject: [RFC PATCH v4 3/5] riscv/atomic.h : Deduplicate arch_atomic.* Date: Tue, 8 Aug 2023 23:13:08 -0300 Message-ID: <20230809021311.1390578-5-leobras@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230809021311.1390578-2-leobras@redhat.com> References: <20230809021311.1390578-2-leobras@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230808_191411_731957_DEB68960 X-CRM114-Status: GOOD ( 11.81 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Some functions use mostly the same asm for 32-bit and 64-bit versions. Make a macro that is generic enough and avoid code duplication. (This did not cause any change in generated asm) Signed-off-by: Leonardo Bras Reviewed-by: Guo Ren Reviewed-by: Andrea Parri --- arch/riscv/include/asm/atomic.h | 164 +++++++++++++++----------------- 1 file changed, 76 insertions(+), 88 deletions(-) diff --git a/arch/riscv/include/asm/atomic.h b/arch/riscv/include/asm/atomic.h index f5dfef6c2153..80cca7ac16fd 100644 --- a/arch/riscv/include/asm/atomic.h +++ b/arch/riscv/include/asm/atomic.h @@ -196,22 +196,28 @@ ATOMIC_OPS(xor, xor, i) #undef ATOMIC_FETCH_OP #undef ATOMIC_OP_RETURN +#define _arch_atomic_fetch_add_unless(_prev, _rc, counter, _a, _u, sfx) \ +({ \ + __asm__ __volatile__ ( \ + "0: lr." sfx " %[p], %[c]\n" \ + " beq %[p], %[u], 1f\n" \ + " add %[rc], %[p], %[a]\n" \ + " sc." sfx ".rl %[rc], %[rc], %[c]\n" \ + " bnez %[rc], 0b\n" \ + " fence rw, rw\n" \ + "1:\n" \ + : [p]"=&r" (_prev), [rc]"=&r" (_rc), [c]"+A" (counter) \ + : [a]"r" (_a), [u]"r" (_u) \ + : "memory"); \ +}) + /* This is required to provide a full barrier on success. */ static __always_inline int arch_atomic_fetch_add_unless(atomic_t *v, int a, int u) { int prev, rc; - __asm__ __volatile__ ( - "0: lr.w %[p], %[c]\n" - " beq %[p], %[u], 1f\n" - " add %[rc], %[p], %[a]\n" - " sc.w.rl %[rc], %[rc], %[c]\n" - " bnez %[rc], 0b\n" - " fence rw, rw\n" - "1:\n" - : [p]"=&r" (prev), [rc]"=&r" (rc), [c]"+A" (v->counter) - : [a]"r" (a), [u]"r" (u) - : "memory"); + _arch_atomic_fetch_add_unless(prev, rc, v->counter, a, u, "w"); + return prev; } #define arch_atomic_fetch_add_unless arch_atomic_fetch_add_unless @@ -222,77 +228,86 @@ static __always_inline s64 arch_atomic64_fetch_add_unless(atomic64_t *v, s64 a, s64 prev; long rc; - __asm__ __volatile__ ( - "0: lr.d %[p], %[c]\n" - " beq %[p], %[u], 1f\n" - " add %[rc], %[p], %[a]\n" - " sc.d.rl %[rc], %[rc], %[c]\n" - " bnez %[rc], 0b\n" - " fence rw, rw\n" - "1:\n" - : [p]"=&r" (prev), [rc]"=&r" (rc), [c]"+A" (v->counter) - : [a]"r" (a), [u]"r" (u) - : "memory"); + _arch_atomic_fetch_add_unless(prev, rc, v->counter, a, u, "d"); + return prev; } #define arch_atomic64_fetch_add_unless arch_atomic64_fetch_add_unless #endif +#define _arch_atomic_inc_unless_negative(_prev, _rc, counter, sfx) \ +({ \ + __asm__ __volatile__ ( \ + "0: lr." sfx " %[p], %[c]\n" \ + " bltz %[p], 1f\n" \ + " addi %[rc], %[p], 1\n" \ + " sc." sfx ".rl %[rc], %[rc], %[c]\n" \ + " bnez %[rc], 0b\n" \ + " fence rw, rw\n" \ + "1:\n" \ + : [p]"=&r" (_prev), [rc]"=&r" (_rc), [c]"+A" (counter) \ + : \ + : "memory"); \ +}) + static __always_inline bool arch_atomic_inc_unless_negative(atomic_t *v) { int prev, rc; - __asm__ __volatile__ ( - "0: lr.w %[p], %[c]\n" - " bltz %[p], 1f\n" - " addi %[rc], %[p], 1\n" - " sc.w.rl %[rc], %[rc], %[c]\n" - " bnez %[rc], 0b\n" - " fence rw, rw\n" - "1:\n" - : [p]"=&r" (prev), [rc]"=&r" (rc), [c]"+A" (v->counter) - : - : "memory"); + _arch_atomic_inc_unless_negative(prev, rc, v->counter, "w"); + return !(prev < 0); } #define arch_atomic_inc_unless_negative arch_atomic_inc_unless_negative +#define _arch_atomic_dec_unless_positive(_prev, _rc, counter, sfx) \ +({ \ + __asm__ __volatile__ ( \ + "0: lr." sfx " %[p], %[c]\n" \ + " bgtz %[p], 1f\n" \ + " addi %[rc], %[p], -1\n" \ + " sc." sfx ".rl %[rc], %[rc], %[c]\n" \ + " bnez %[rc], 0b\n" \ + " fence rw, rw\n" \ + "1:\n" \ + : [p]"=&r" (_prev), [rc]"=&r" (_rc), [c]"+A" (counter) \ + : \ + : "memory"); \ +}) + static __always_inline bool arch_atomic_dec_unless_positive(atomic_t *v) { int prev, rc; - __asm__ __volatile__ ( - "0: lr.w %[p], %[c]\n" - " bgtz %[p], 1f\n" - " addi %[rc], %[p], -1\n" - " sc.w.rl %[rc], %[rc], %[c]\n" - " bnez %[rc], 0b\n" - " fence rw, rw\n" - "1:\n" - : [p]"=&r" (prev), [rc]"=&r" (rc), [c]"+A" (v->counter) - : - : "memory"); + _arch_atomic_dec_unless_positive(prev, rc, v->counter, "w"); + return !(prev > 0); } #define arch_atomic_dec_unless_positive arch_atomic_dec_unless_positive +#define _arch_atomic_dec_if_positive(_prev, _rc, counter, sfx) \ +({ \ + __asm__ __volatile__ ( \ + "0: lr." sfx " %[p], %[c]\n" \ + " addi %[rc], %[p], -1\n" \ + " bltz %[rc], 1f\n" \ + " sc." sfx ".rl %[rc], %[rc], %[c]\n" \ + " bnez %[rc], 0b\n" \ + " fence rw, rw\n" \ + "1:\n" \ + : [p]"=&r" (_prev), [rc]"=&r" (_rc), [c]"+A" (counter) \ + : \ + : "memory"); \ +}) + static __always_inline int arch_atomic_dec_if_positive(atomic_t *v) { int prev, rc; - __asm__ __volatile__ ( - "0: lr.w %[p], %[c]\n" - " addi %[rc], %[p], -1\n" - " bltz %[rc], 1f\n" - " sc.w.rl %[rc], %[rc], %[c]\n" - " bnez %[rc], 0b\n" - " fence rw, rw\n" - "1:\n" - : [p]"=&r" (prev), [rc]"=&r" (rc), [c]"+A" (v->counter) - : - : "memory"); + _arch_atomic_dec_if_positive(prev, rc, v->counter, "w"); + return prev - 1; } @@ -304,17 +319,8 @@ static __always_inline bool arch_atomic64_inc_unless_negative(atomic64_t *v) s64 prev; long rc; - __asm__ __volatile__ ( - "0: lr.d %[p], %[c]\n" - " bltz %[p], 1f\n" - " addi %[rc], %[p], 1\n" - " sc.d.rl %[rc], %[rc], %[c]\n" - " bnez %[rc], 0b\n" - " fence rw, rw\n" - "1:\n" - : [p]"=&r" (prev), [rc]"=&r" (rc), [c]"+A" (v->counter) - : - : "memory"); + _arch_atomic_inc_unless_negative(prev, rc, v->counter, "d"); + return !(prev < 0); } @@ -325,17 +331,8 @@ static __always_inline bool arch_atomic64_dec_unless_positive(atomic64_t *v) s64 prev; long rc; - __asm__ __volatile__ ( - "0: lr.d %[p], %[c]\n" - " bgtz %[p], 1f\n" - " addi %[rc], %[p], -1\n" - " sc.d.rl %[rc], %[rc], %[c]\n" - " bnez %[rc], 0b\n" - " fence rw, rw\n" - "1:\n" - : [p]"=&r" (prev), [rc]"=&r" (rc), [c]"+A" (v->counter) - : - : "memory"); + _arch_atomic_dec_unless_positive(prev, rc, v->counter, "d"); + return !(prev > 0); } @@ -346,17 +343,8 @@ static __always_inline s64 arch_atomic64_dec_if_positive(atomic64_t *v) s64 prev; long rc; - __asm__ __volatile__ ( - "0: lr.d %[p], %[c]\n" - " addi %[rc], %[p], -1\n" - " bltz %[rc], 1f\n" - " sc.d.rl %[rc], %[rc], %[c]\n" - " bnez %[rc], 0b\n" - " fence rw, rw\n" - "1:\n" - : [p]"=&r" (prev), [rc]"=&r" (rc), [c]"+A" (v->counter) - : - : "memory"); + _arch_atomic_dec_if_positive(prev, rc, v->counter, "d"); + return prev - 1; } From patchwork Wed Aug 9 02:13:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leonardo Bras X-Patchwork-Id: 13347338 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E261DC001DB for ; Wed, 9 Aug 2023 02:14:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=gkKN9RT97s9MJhVSaEeAysz5zHUn4zcf9tWT4osiPF4=; b=Cxib59HGrZpYzY TdWDW8N0QhIA6jOyIb1AE8YkV0YXS+daenXxXh5j8NpNf4gcowowQCpsyLsE25NvG3clqAoaVV5en McTryfrrBGPWUuNjcTg5NGVR8UaDrKepiYKOiYyMXB/cp/76msI3ccURO7TCb89wd2WBMT3ObbA09 wV/F/E1oYQztO+/yQNI3nSFNvHNrdZ88jVbSCG8guZ0npu3bJ854WRH14Evie8lDV54ju0tf4WGaH 9AVE15Pon+9anyKdWa7OwIvRSa5N8hWNyi/uZxNeUfaDVSlg3l5a4heCn0moQn4enoaafBd0cfV5x fiG5bstPLymLT8D10ZCg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qTYiL-003saZ-02; Wed, 09 Aug 2023 02:14:21 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qTYiH-003sZ1-2C for linux-riscv@lists.infradead.org; Wed, 09 Aug 2023 02:14:19 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1691547256; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yR44ogH6UDfncEstCVw9mSlc1d7ANRRQAVF648wbdjc=; b=R9/8+LqhTELqJ9Bk6dbLJnn7xJpTL8T9LdL6Tqv90N4eU5zs6bxOBoxWLTGHVLu36gbX2k l9WkV/BIDDiwO+oGQyKJpC09fWhM5nbsc51Heqcp/04hBgzIefvEgoZ1ePilbfJW6rWYGs sx/qEXPTPjOi+HXoUANBWp3ENLroAiA= Received: from mail-oa1-f70.google.com (mail-oa1-f70.google.com [209.85.160.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-584-4ZKGajLtMdeOjwtgdLZXBA-1; Tue, 08 Aug 2023 22:14:15 -0400 X-MC-Unique: 4ZKGajLtMdeOjwtgdLZXBA-1 Received: by mail-oa1-f70.google.com with SMTP id 586e51a60fabf-1bf8e8f19b1so8850673fac.2 for ; Tue, 08 Aug 2023 19:14:15 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691547254; x=1692152054; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=yR44ogH6UDfncEstCVw9mSlc1d7ANRRQAVF648wbdjc=; b=htqF/oSWcZHjSGED+g8F8+ZKSdp16+XW3DI2HN3hWIkB/y8XWAp662XqnM8AuayhW4 Wwiqs6ZkQixbyIonZJ4oPnaqJ0Lo6e0tq2eCdtyHL09VI7H6AsPTnKqNTUAQMrTRgxm0 +ZD3mZhb2jUjWKie7qtG8KXg6TLrVRIylZLCQyt/4j344CISKgxKoUYj4NMHcIl/Ahae oaehGCbJP52pKDSv1P0RFuwh7KrXpN0RStFjBu5ahXBkW9c5FH+Sod2/x/rxd9jzMMtn p3JgctFFXL0KL5JbRE9Qs8/3vO2Zx5N4IazGK6NH/F4rTpuaSjneHxsYRE1AI6Ih6Myl uEzw== X-Gm-Message-State: AOJu0YxNgjySU4Aop4SbOuNi2EmW00r1YARTYwdN1c89y8EibUmGcPF4 5UrQ3BbSFHntA8OqOz/nn24dJnc4CTm7X2IhUHohFxBrM/7WbWQ1wtEiMjqV/W7TIlwW2NEyINc OKU4BK4hETU+txCkD/LIYmYuNaF97gyWP6d79 X-Received: by 2002:a05:6870:40d1:b0:1be:d3a1:fd9 with SMTP id l17-20020a05687040d100b001bed3a10fd9mr1815544oal.9.1691547254606; Tue, 08 Aug 2023 19:14:14 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHviWJ7FwqHHQS/IFWOJZUUEtSZUVOo6M1kD/UsKsoQJL4BRummKJuVjUiXc3Pder8uF/0zRw== X-Received: by 2002:a05:6870:40d1:b0:1be:d3a1:fd9 with SMTP id l17-20020a05687040d100b001bed3a10fd9mr1815505oal.9.1691547254170; Tue, 08 Aug 2023 19:14:14 -0700 (PDT) Received: from localhost.localdomain ([2804:431:c7ec:e667:6b7d:ed55:c363:a088]) by smtp.gmail.com with ESMTPSA id v10-20020a4a8c4a000000b0054fba751207sm6475518ooj.47.2023.08.08.19.14.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Aug 2023 19:14:13 -0700 (PDT) From: Leonardo Bras To: Will Deacon , Peter Zijlstra , Boqun Feng , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Leonardo Bras , Andrea Parri , Ingo Molnar , Geert Uytterhoeven , Andrzej Hajda , Palmer Dabbelt , Guo Ren Cc: linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org Subject: [RFC PATCH v4 4/5] riscv/cmpxchg: Implement cmpxchg for variables of size 1 and 2 Date: Tue, 8 Aug 2023 23:13:09 -0300 Message-ID: <20230809021311.1390578-6-leobras@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230809021311.1390578-2-leobras@redhat.com> References: <20230809021311.1390578-2-leobras@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230808_191417_787516_C74F8F4C X-CRM114-Status: GOOD ( 12.93 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org cmpxchg for variables of size 1-byte and 2-bytes is not yet available for riscv, even though its present in other architectures such as arm64 and x86. This could lead to not being able to implement some locking mechanisms or requiring some rework to make it work properly. Implement 1-byte and 2-bytes cmpxchg in order to achieve parity with other architectures. Signed-off-by: Leonardo Bras --- arch/riscv/include/asm/cmpxchg.h | 34 ++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index 5a07646fae65..cfada8a7cfd2 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -72,6 +72,35 @@ * indicated by comparing RETURN with OLD. */ +#define __arch_cmpxchg_masked(sc_sfx, prepend, append, r, p, o, n) \ +({ \ + u32 *__ptr32b = (u32 *)((ulong)(p) & ~0x3); \ + ulong __s = ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \ + ulong __mask = GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \ + << __s; \ + ulong __newx = (ulong)(n) << __s; \ + ulong __oldx = (ulong)(o) << __s; \ + ulong __retx; \ + ulong __rc; \ + \ + __asm__ __volatile__ ( \ + prepend \ + "0: lr.w %0, %2\n" \ + " and %1, %0, %z5\n" \ + " bne %1, %z3, 1f\n" \ + " and %1, %0, %z6\n" \ + " or %1, %1, %z4\n" \ + " sc.w" sc_sfx " %1, %1, %2\n" \ + " bnez %1, 0b\n" \ + append \ + "1:\n" \ + : "=&r" (__retx), "=&r" (__rc), "+A" (*(__ptr32b)) \ + : "rJ" ((long)__oldx), "rJ" (__newx), \ + "rJ" (__mask), "rJ" (~__mask) \ + : "memory"); \ + \ + r = (__typeof__(*(p)))((__retx & __mask) >> __s); \ +}) #define __arch_cmpxchg(lr_sfx, sc_sfx, prepend, append, r, p, co, o, n) \ ({ \ @@ -98,6 +127,11 @@ __typeof__(*(ptr)) __ret; \ \ switch (sizeof(*__ptr)) { \ + case 1: \ + case 2: \ + __arch_cmpxchg_masked(sc_sfx, prepend, append, \ + __ret, __ptr, __old, __new); \ + break; \ case 4: \ __arch_cmpxchg(".w", ".w" sc_sfx, prepend, append, \ __ret, __ptr, (long), __old, __new); \ From patchwork Wed Aug 9 02:13:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leonardo Bras X-Patchwork-Id: 13347339 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 68713C001DE for ; Wed, 9 Aug 2023 02:14:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=GanTHlYhYX5QFinaUMguOws2e83OuzgtZDeg+rWUwaw=; b=Fogeu70uJQM7Aj +bpnF3GOGLArRDv+VhQMBuUxo4icdTwm5+lzvi5PPM8sTINIyji/WfDmM3jhP8lCMUy9yi89EYHAH m/2lWSfmXOrH6Bb42TfvDO8+ifhJhc6gFeMEUvJbwRTtgMfgheG4nKmu4ZSOEovJnNN/DyYkKdRiD DViQ7XJG9kriqZi8IWDJY6ma+BCJL2f4U5ZDDfZ71+eRQnjyG2/1YwO+lcc0Y3tKno2bc1Pv+zExx +DkNip63Su2X2NLjrG7IVc+apKm/1roPrUItwa4u9gfZwR0j8nDfKJSh1yKsCQu/aOmsPxzelKiS6 fS9FxnMNNRZJF8XFpGRw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qTYiQ-003sce-2M; Wed, 09 Aug 2023 02:14:26 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qTYiN-003sbA-27 for linux-riscv@lists.infradead.org; Wed, 09 Aug 2023 02:14:24 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1691547262; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QSwkRG8bcSleBdtSsjCnyBmp/8Y+sHuiXOKQQOlQtH4=; b=XhgRnaysDIyKjwWEioRIbel9E32OKc+1kLhtv04wA4gOOJFIprsafp51CaTjvxv7QUXOJG 3ApUdtMEkvoT7LCLuY9mQml5VLptN0BpZSjWnSe1hcc5U0gSTy2i0Uypasl8EBQeeLznY7 4HaL+nIXT0KrRBFE1NUVd9nUKd0Tqfs= Received: from mail-oa1-f72.google.com (mail-oa1-f72.google.com [209.85.160.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-112-lqCKYS4ZNguUfwb2PYyNJA-1; Tue, 08 Aug 2023 22:14:21 -0400 X-MC-Unique: lqCKYS4ZNguUfwb2PYyNJA-1 Received: by mail-oa1-f72.google.com with SMTP id 586e51a60fabf-1bf2e81ce63so587077fac.1 for ; Tue, 08 Aug 2023 19:14:21 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691547261; x=1692152061; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QSwkRG8bcSleBdtSsjCnyBmp/8Y+sHuiXOKQQOlQtH4=; b=IxKJgAzG7Also46rjI1QxCPufFRzctB0z+un1F7rxN4Mf/9H5V5zxf6igYht2CVJ8v RPPJLvnCkXqiiCP/1ILCcVeoU6PsPo1gGEQqUwbZENJKpkFopdZA7bPf9USxxRqJ/yl9 ZYxh2cZP+vEMMIZpfe9tuyg/yWPuiNwvSJwdFrwtDpxvWibGoNaSofdH26u5RC674fsC a1hqMuKvXFKlHnGgNjWvrCOn9tdc1p1X/vSCqfd3Cv0ITxHCI4Rg4jwjd0GfS4NSznUs REVUArPnBE01mwaXb3JSLUFDpe38qEFdbZp6Z6e582C8MWTCnTfaf1yY1loO9CIaop3z MoGw== X-Gm-Message-State: AOJu0YyivA34ppdm0yVE9lncZOJTaI/plRC8vbqz0nbycq/mbBYmJEhX +Qa6fsU+fk1rJag/W8ZP1kdEVKrxuiXeKEgXK0sFznPNc1ENvvP9fQ6kYIQAPOJCHwoyhFtNP6e //oy0UEsdz+3u0IEf3WTh+r9Te+oe X-Received: by 2002:a05:6870:b6a9:b0:1bf:1c49:749c with SMTP id cy41-20020a056870b6a900b001bf1c49749cmr8730775oab.13.1691547260769; Tue, 08 Aug 2023 19:14:20 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEHlHClRw7hdQrrpejz4vuDyEoE85EOUKHc1btv2W4ZMe793U+R5Uo/+moePgCrObCJ/o3ANw== X-Received: by 2002:a05:6870:b6a9:b0:1bf:1c49:749c with SMTP id cy41-20020a056870b6a900b001bf1c49749cmr8730768oab.13.1691547260529; Tue, 08 Aug 2023 19:14:20 -0700 (PDT) Received: from localhost.localdomain ([2804:431:c7ec:e667:6b7d:ed55:c363:a088]) by smtp.gmail.com with ESMTPSA id v10-20020a4a8c4a000000b0054fba751207sm6475518ooj.47.2023.08.08.19.14.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Aug 2023 19:14:20 -0700 (PDT) From: Leonardo Bras To: Will Deacon , Peter Zijlstra , Boqun Feng , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Leonardo Bras , Andrea Parri , Ingo Molnar , Geert Uytterhoeven , Andrzej Hajda , Palmer Dabbelt , Guo Ren Cc: linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org Subject: [RFC PATCH v4 5/5] riscv/cmpxchg: Implement xchg for variables of size 1 and 2 Date: Tue, 8 Aug 2023 23:13:10 -0300 Message-ID: <20230809021311.1390578-7-leobras@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230809021311.1390578-2-leobras@redhat.com> References: <20230809021311.1390578-2-leobras@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230808_191423_791253_CF7DD52C X-CRM114-Status: GOOD ( 12.44 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org xchg for variables of size 1-byte and 2-bytes is not yet available for riscv, even though its present in other architectures such as arm64 and x86. This could lead to not being able to implement some locking mechanisms or requiring some rework to make it work properly. Implement 1-byte and 2-bytes xchg in order to achieve parity with other architectures. Signed-off-by: Leonardo Bras --- arch/riscv/include/asm/cmpxchg.h | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index cfada8a7cfd2..1640308f3c4d 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -11,6 +11,31 @@ #include #include +#define __arch_xchg_masked(prepend, append, r, p, n) \ +({ \ + u32 *__ptr32b = (u32 *)((ulong)(p) & ~0x3); \ + ulong __s = ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \ + ulong __mask = GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \ + << __s; \ + ulong __newx = (ulong)(n) << __s; \ + ulong __retx; \ + ulong __rc; \ + \ + __asm__ __volatile__ ( \ + prepend \ + "0: lr.w %0, %2\n" \ + " and %1, %0, %z4\n" \ + " or %1, %1, %z3\n" \ + " sc.w %1, %1, %2\n" \ + " bnez %1, 0b\n" \ + append \ + : "=&r" (__retx), "=&r" (__rc), "+A" (*(__ptr32b)) \ + : "rJ" (__newx), "rJ" (~__mask) \ + : "memory"); \ + \ + r = (__typeof__(*(p)))((__retx & __mask) >> __s); \ +}) + #define __arch_xchg(sfx, prepend, append, r, p, n) \ ({ \ __asm__ __volatile__ ( \ @@ -27,7 +52,13 @@ __typeof__(*(ptr)) __new = (new); \ __typeof__(*(ptr)) __ret; \ __typeof__(ptr) __ptr = (ptr); \ + \ switch (sizeof(*__ptr)) { \ + case 1: \ + case 2: \ + __arch_xchg_masked(prepend, append, \ + __ret, __ptr, __new); \ + break; \ case 4: \ __arch_xchg(".w" sfx, prepend, append, \ __ret, __ptr, __new); \