From patchwork Thu Aug 10 04:03:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leonardo Bras X-Patchwork-Id: 13348750 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9B762C001B0 for ; Thu, 10 Aug 2023 04:04:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=3VTRUUiuGzNDxOqgXZUryASae5qY6GeF0U6/L2fYYxY=; b=uXQT1dUyHNznrV G19EjsYWG4mRRkeQhaDDiT8IgTpCcO1VNKZpLB9AjcEtHRzB5rJwLQZAjIbDbK6XXGSIT+/1tfqaT gxE+ZwqSWKyQcRLcbXEJ8MRa9/LESrOznjfNuoBJ7wwwmZ5nHCaW86959QQYEA90u837doL3zo+KJ D+Qx5eRJCQuROC/Q1EQt106GBoCJ/eClf8ISyqz7l1C5pfVnA3T6mvRTOHXSLfwVt/fdUKvCyRimB FL3lWj//yzuPKHX3MtMqUA0CEceNXXZdOj/n8brIYL2UhZlGWY0i/WfCME0qrlB+QIJeW10zwMSUG C7CwS3EnJkSUWujibDwg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qTwuK-006Sg5-0N; Thu, 10 Aug 2023 04:04:20 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qTwuF-006SfG-24 for linux-riscv@lists.infradead.org; Thu, 10 Aug 2023 04:04:18 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1691640254; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OScrv20Vk5PZ28CoHB13qKk09OrWPyCgb0XsKA59wTM=; b=H9pLl2+aC04Oa4LpLfvcHEejhskioqLowl5XLCzlDm0Rte/x9bF5Oli3w0nhSmfNVASOgv u7apzq5CelRvkbn2loPgOynVP42QAgNRznU4uD20+wARiTL6U/97qFiEk4pLYTmvGOsmDA LelWuAiC0h/1FMJEHu/0MrGNZAzwrzs= Received: from mail-oi1-f197.google.com (mail-oi1-f197.google.com [209.85.167.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-488-hkIqhx9VOlG2McVAiMJ_Yg-1; Thu, 10 Aug 2023 00:04:13 -0400 X-MC-Unique: hkIqhx9VOlG2McVAiMJ_Yg-1 Received: by mail-oi1-f197.google.com with SMTP id 5614622812f47-3a5ab2d2b3bso570550b6e.0 for ; Wed, 09 Aug 2023 21:04:13 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691640252; x=1692245052; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OScrv20Vk5PZ28CoHB13qKk09OrWPyCgb0XsKA59wTM=; b=G3Bm3A2gqXG2TusQ+X9yempb6UTGYvGqhe8qSXMHmsNETLIWh83IXgq4zlRgFWeQtE SLLkTXa8wdev33hCTgS8nlp4kUDzXFFBfC5jcF/LTETUdu3Giv+tpzo4gRy5O4iyVZIp vJy86DZmDGoMFMGVDrv2hibS6vtyIOkLUSpE5y+0jQKV7FWSbGUiHNKauHqfX5UwH2iJ yDq5vDVZuUC5gVaTE+W5Sd9tl4qCQa4g8ratHlpwk/I7yQ+E/BSmxoMWpfdwLQvd5JCM QPH/NN2L/s4ZOWvrDuLw/VRIDvqhUESQp4wwiOmEcHZKNT8zSNNR3HHmib2Iema7V6y1 /2sw== X-Gm-Message-State: AOJu0YwfzhEAQ1LZUYRSFK1mTeAd41Uw41dQ5ezX+VgfgSyoYyhF3S6a Nz3inzg4I+7w+SeTcaQIVw9PUBOyM3LcEDMnLKTMFO8VimAZ39WsRS+89alfC3xpKvYEv9preSK VvNKpJ2Lc/rmUrhRtzdhEGCn4uEe7 X-Received: by 2002:a05:6808:16a8:b0:3a7:65ae:9ccf with SMTP id bb40-20020a05680816a800b003a765ae9ccfmr1751531oib.22.1691640252580; Wed, 09 Aug 2023 21:04:12 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGXuC0j5A0JXiqF9jKMAp5qoF1YnSFxPZWZ79s3qRiKxxZaWZdWWqxSD/vcnxGSyUwfd9n7/g== X-Received: by 2002:a05:6808:16a8:b0:3a7:65ae:9ccf with SMTP id bb40-20020a05680816a800b003a765ae9ccfmr1751518oib.22.1691640252341; Wed, 09 Aug 2023 21:04:12 -0700 (PDT) Received: from localhost.localdomain ([2804:431:c7ec:e667:6b7d:ed55:c363:a088]) by smtp.gmail.com with ESMTPSA id b12-20020aca674c000000b003a7b5193909sm310087oiy.19.2023.08.09.21.04.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Aug 2023 21:04:11 -0700 (PDT) From: Leonardo Bras To: Will Deacon , Peter Zijlstra , Boqun Feng , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Leonardo Bras , Andrea Parri , Andrzej Hajda , Arnd Bergmann , Palmer Dabbelt , Guo Ren Cc: linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org Subject: [RFC PATCH v5 1/5] riscv/cmpxchg: Deduplicate xchg() asm functions Date: Thu, 10 Aug 2023 01:03:43 -0300 Message-ID: <20230810040349.92279-3-leobras@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230810040349.92279-2-leobras@redhat.com> References: <20230810040349.92279-2-leobras@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230809_210415_760283_10D1676E X-CRM114-Status: GOOD ( 12.06 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org In this header every xchg define (_relaxed, _acquire, _release, vanilla) contain it's own asm file, both for 4-byte variables an 8-byte variables, on a total of 8 versions of mostly the same asm. This is usually bad, as it means any change may be done in up to 8 different places. Unify those versions by creating a new define with enough parameters to generate any version of the previous 8. Then unify the result under a more general define, and simplify arch_xchg* generation. (This did not cause any change in generated asm) Signed-off-by: Leonardo Bras Reviewed-by: Andrea Parri --- arch/riscv/include/asm/cmpxchg.h | 138 ++++++------------------------- 1 file changed, 23 insertions(+), 115 deletions(-) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index 2f4726d3cfcc..48478a8eecee 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -11,140 +11,48 @@ #include #include -#define __xchg_relaxed(ptr, new, size) \ +#define __arch_xchg(sfx, prepend, append, r, p, n) \ ({ \ - __typeof__(ptr) __ptr = (ptr); \ - __typeof__(new) __new = (new); \ - __typeof__(*(ptr)) __ret; \ - switch (size) { \ - case 4: \ - __asm__ __volatile__ ( \ - " amoswap.w %0, %2, %1\n" \ - : "=r" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ - break; \ - case 8: \ - __asm__ __volatile__ ( \ - " amoswap.d %0, %2, %1\n" \ - : "=r" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ - break; \ - default: \ - BUILD_BUG(); \ - } \ - __ret; \ -}) - -#define arch_xchg_relaxed(ptr, x) \ -({ \ - __typeof__(*(ptr)) _x_ = (x); \ - (__typeof__(*(ptr))) __xchg_relaxed((ptr), \ - _x_, sizeof(*(ptr))); \ + __asm__ __volatile__ ( \ + prepend \ + " amoswap" sfx " %0, %2, %1\n" \ + append \ + : "=r" (r), "+A" (*(p)) \ + : "r" (n) \ + : "memory"); \ }) -#define __xchg_acquire(ptr, new, size) \ +#define _arch_xchg(ptr, new, sfx, prepend, append) \ ({ \ __typeof__(ptr) __ptr = (ptr); \ - __typeof__(new) __new = (new); \ - __typeof__(*(ptr)) __ret; \ - switch (size) { \ + __typeof__(*(__ptr)) __new = (new); \ + __typeof__(*(__ptr)) __ret; \ + switch (sizeof(*__ptr)) { \ case 4: \ - __asm__ __volatile__ ( \ - " amoswap.w %0, %2, %1\n" \ - RISCV_ACQUIRE_BARRIER \ - : "=r" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ + __arch_xchg(".w" sfx, prepend, append, \ + __ret, __ptr, __new); \ break; \ case 8: \ - __asm__ __volatile__ ( \ - " amoswap.d %0, %2, %1\n" \ - RISCV_ACQUIRE_BARRIER \ - : "=r" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ + __arch_xchg(".d" sfx, prepend, append, \ + __ret, __ptr, __new); \ break; \ default: \ BUILD_BUG(); \ } \ - __ret; \ + (__typeof__(*(__ptr)))__ret; \ }) -#define arch_xchg_acquire(ptr, x) \ -({ \ - __typeof__(*(ptr)) _x_ = (x); \ - (__typeof__(*(ptr))) __xchg_acquire((ptr), \ - _x_, sizeof(*(ptr))); \ -}) +#define arch_xchg_relaxed(ptr, x) \ + _arch_xchg(ptr, x, "", "", "") -#define __xchg_release(ptr, new, size) \ -({ \ - __typeof__(ptr) __ptr = (ptr); \ - __typeof__(new) __new = (new); \ - __typeof__(*(ptr)) __ret; \ - switch (size) { \ - case 4: \ - __asm__ __volatile__ ( \ - RISCV_RELEASE_BARRIER \ - " amoswap.w %0, %2, %1\n" \ - : "=r" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ - break; \ - case 8: \ - __asm__ __volatile__ ( \ - RISCV_RELEASE_BARRIER \ - " amoswap.d %0, %2, %1\n" \ - : "=r" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ - break; \ - default: \ - BUILD_BUG(); \ - } \ - __ret; \ -}) +#define arch_xchg_acquire(ptr, x) \ + _arch_xchg(ptr, x, "", "", RISCV_ACQUIRE_BARRIER) #define arch_xchg_release(ptr, x) \ -({ \ - __typeof__(*(ptr)) _x_ = (x); \ - (__typeof__(*(ptr))) __xchg_release((ptr), \ - _x_, sizeof(*(ptr))); \ -}) - -#define __arch_xchg(ptr, new, size) \ -({ \ - __typeof__(ptr) __ptr = (ptr); \ - __typeof__(new) __new = (new); \ - __typeof__(*(ptr)) __ret; \ - switch (size) { \ - case 4: \ - __asm__ __volatile__ ( \ - " amoswap.w.aqrl %0, %2, %1\n" \ - : "=r" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ - break; \ - case 8: \ - __asm__ __volatile__ ( \ - " amoswap.d.aqrl %0, %2, %1\n" \ - : "=r" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ - break; \ - default: \ - BUILD_BUG(); \ - } \ - __ret; \ -}) + _arch_xchg(ptr, x, "", RISCV_RELEASE_BARRIER, "") #define arch_xchg(ptr, x) \ -({ \ - __typeof__(*(ptr)) _x_ = (x); \ - (__typeof__(*(ptr))) __arch_xchg((ptr), _x_, sizeof(*(ptr))); \ -}) + _arch_xchg(ptr, x, ".aqrl", "", "") #define xchg32(ptr, x) \ ({ \ From patchwork Thu Aug 10 04:03:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leonardo Bras X-Patchwork-Id: 13348751 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7C6A4C001B0 for ; Thu, 10 Aug 2023 04:04:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=6lN/lD4S+SlnNGYa8QysUojSwKe6obyfwcUPqrKFYIc=; b=kMy7atMQIfkrki tCLgwQFhZcCxDB34REbverieHhjBGribSrREBrmG/xE47Su+0zezry6YLFD7kDHt1QbIYsPRN1XwD 2Kkt6EDuxnHgHCUQ+pvDYsMmXO4QQ43PfkdHk7du/2NB6XKXuYYrIfSlX22dNYSu3K0fTb0gshx0Q jkKKCEuPLb51MwVG90gZ5wEMBU0xWFvq25mtR93cEv+LYvYKEE1/fCH19PoU7yhvbmvCm4J6V9ism 8QXjX56W7aAE2uG0xNdjlog6TwNk7GTTvMkjTBQA2JU1nNjl+66zpX11RAXwCH/jA8bz7Jnpd8Lwy 8b5OrjW2uM7yecFEUUPQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qTwuP-006ShL-2j; Thu, 10 Aug 2023 04:04:25 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qTwuM-006SgZ-20 for linux-riscv@lists.infradead.org; Thu, 10 Aug 2023 04:04:24 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1691640261; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WcBubYngvVAQlkkTX8jFLD9nX24FtmcCz/1wvQAzhyI=; b=Z1OXDmHMYXcKUZf1n1lI7G2iyeR6vmf1XAot3n3llJPAgyMhqk10OEQG6cuhlLu3toub/1 NsEzPVc6W+Rrw2YilA3AmLsfhVySfH33742GC7VUfw66Mm7pUdLQKzBpz3Ns2ffxzOVD13 n4GZUtDFz9yZm4+dG5CZjvfn3n49r58= Received: from mail-oo1-f70.google.com (mail-oo1-f70.google.com [209.85.161.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-49-XaZotwj-MrCKIXfwYQcSFA-1; Thu, 10 Aug 2023 00:04:18 -0400 X-MC-Unique: XaZotwj-MrCKIXfwYQcSFA-1 Received: by mail-oo1-f70.google.com with SMTP id 006d021491bc7-56ca23f8b8bso1529795eaf.0 for ; Wed, 09 Aug 2023 21:04:18 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691640258; x=1692245058; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WcBubYngvVAQlkkTX8jFLD9nX24FtmcCz/1wvQAzhyI=; b=YmyTcOU8+SZQ1wJnh88VRJBirIyaHOagACpZ1vQdmuzbIjX3amgnKGksqVVOs44CgQ 46m/OVX/2DuTDavAlayuxbLkxdOQ6TSOgGkY0Re6SK0eHac/iOuAPPPQkmIFr8lIhYrh Ul4nXfdo1XYQtqbElGWV6Wt6/bcW71CcglbNBbxdQyAimuT4Mf9YKDcAkM5efIe2hRcf +LhJL3k/bFU9SRjRdAf5HSNaEauCp7Zflpl8KfdMqriy72v+RNcvdEUNmVpS6JNBxcuy a8XzwV1BvbgfnbluIhNKWDi24OKCH5Wls2IvUnlrnDFDqtJgE06DzvE9BWtEPouDHrcQ zEKQ== X-Gm-Message-State: AOJu0YyO8mfYX5iZ+lWdGLboSSRp83quxmAre0vKpW49tZxG2DWRPyi7 z3NhYFGyTuJ7tLLf7TRQpL+SPfgRkeINTxBVZB6DsEtDox0GVKaHQQkK5p8lHNcehAAx1mXSkZG xFLe6NlVVezZZhCJCTFcIUPLi7rnr X-Received: by 2002:a05:6808:1514:b0:3a2:5d31:b299 with SMTP id u20-20020a056808151400b003a25d31b299mr594894oiw.28.1691640257840; Wed, 09 Aug 2023 21:04:17 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHyCNfWl+g7sjPkEBxDgZwisNNLc1LhGyzcSUUej8NIuwUa4eFN8ZtD/RMr0iEkJ8Ba0KVwgw== X-Received: by 2002:a05:6808:1514:b0:3a2:5d31:b299 with SMTP id u20-20020a056808151400b003a25d31b299mr594877oiw.28.1691640257554; Wed, 09 Aug 2023 21:04:17 -0700 (PDT) Received: from localhost.localdomain ([2804:431:c7ec:e667:6b7d:ed55:c363:a088]) by smtp.gmail.com with ESMTPSA id b12-20020aca674c000000b003a7b5193909sm310087oiy.19.2023.08.09.21.04.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Aug 2023 21:04:17 -0700 (PDT) From: Leonardo Bras To: Will Deacon , Peter Zijlstra , Boqun Feng , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Leonardo Bras , Andrea Parri , Andrzej Hajda , Arnd Bergmann , Palmer Dabbelt , Guo Ren Cc: linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org Subject: [RFC PATCH v5 2/5] riscv/cmpxchg: Deduplicate cmpxchg() asm and macros Date: Thu, 10 Aug 2023 01:03:44 -0300 Message-ID: <20230810040349.92279-4-leobras@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230810040349.92279-2-leobras@redhat.com> References: <20230810040349.92279-2-leobras@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230809_210422_733666_A2F4A99C X-CRM114-Status: GOOD ( 12.30 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org In this header every cmpxchg define (_relaxed, _acquire, _release, vanilla) contain it's own asm file, both for 4-byte variables an 8-byte variables, on a total of 8 versions of mostly the same asm. This is usually bad, as it means any change may be done in up to 8 different places. Unify those versions by creating a new define with enough parameters to generate any version of the previous 8. Then unify the result under a more general define, and simplify arch_cmpxchg* generation (This did not cause any change in generated asm) Signed-off-by: Leonardo Bras Reviewed-by: Andrea Parri --- arch/riscv/include/asm/cmpxchg.h | 195 ++++++------------------------- 1 file changed, 33 insertions(+), 162 deletions(-) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index 48478a8eecee..e3e0ac7ba061 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -71,190 +71,61 @@ * store NEW in MEM. Return the initial value in MEM. Success is * indicated by comparing RETURN with OLD. */ -#define __cmpxchg_relaxed(ptr, old, new, size) \ -({ \ - __typeof__(ptr) __ptr = (ptr); \ - __typeof__(*(ptr)) __old = (old); \ - __typeof__(*(ptr)) __new = (new); \ - __typeof__(*(ptr)) __ret; \ - register unsigned int __rc; \ - switch (size) { \ - case 4: \ - __asm__ __volatile__ ( \ - "0: lr.w %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc.w %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - "1:\n" \ - : "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \ - : "rJ" ((long)__old), "rJ" (__new) \ - : "memory"); \ - break; \ - case 8: \ - __asm__ __volatile__ ( \ - "0: lr.d %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc.d %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - "1:\n" \ - : "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \ - : "rJ" (__old), "rJ" (__new) \ - : "memory"); \ - break; \ - default: \ - BUILD_BUG(); \ - } \ - __ret; \ -}) -#define arch_cmpxchg_relaxed(ptr, o, n) \ -({ \ - __typeof__(*(ptr)) _o_ = (o); \ - __typeof__(*(ptr)) _n_ = (n); \ - (__typeof__(*(ptr))) __cmpxchg_relaxed((ptr), \ - _o_, _n_, sizeof(*(ptr))); \ -}) -#define __cmpxchg_acquire(ptr, old, new, size) \ +#define __arch_cmpxchg(lr_sfx, sc_sfx, prepend, append, r, p, co, o, n) \ ({ \ - __typeof__(ptr) __ptr = (ptr); \ - __typeof__(*(ptr)) __old = (old); \ - __typeof__(*(ptr)) __new = (new); \ - __typeof__(*(ptr)) __ret; \ register unsigned int __rc; \ - switch (size) { \ - case 4: \ - __asm__ __volatile__ ( \ - "0: lr.w %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc.w %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - RISCV_ACQUIRE_BARRIER \ - "1:\n" \ - : "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \ - : "rJ" ((long)__old), "rJ" (__new) \ - : "memory"); \ - break; \ - case 8: \ - __asm__ __volatile__ ( \ - "0: lr.d %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc.d %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - RISCV_ACQUIRE_BARRIER \ - "1:\n" \ - : "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \ - : "rJ" (__old), "rJ" (__new) \ - : "memory"); \ - break; \ - default: \ - BUILD_BUG(); \ - } \ - __ret; \ -}) - -#define arch_cmpxchg_acquire(ptr, o, n) \ -({ \ - __typeof__(*(ptr)) _o_ = (o); \ - __typeof__(*(ptr)) _n_ = (n); \ - (__typeof__(*(ptr))) __cmpxchg_acquire((ptr), \ - _o_, _n_, sizeof(*(ptr))); \ + \ + __asm__ __volatile__ ( \ + prepend \ + "0: lr" lr_sfx " %0, %2\n" \ + " bne %0, %z3, 1f\n" \ + " sc" sc_sfx " %1, %z4, %2\n" \ + " bnez %1, 0b\n" \ + append \ + "1:\n" \ + : "=&r" (r), "=&r" (__rc), "+A" (*(p)) \ + : "rJ" (co o), "rJ" (n) \ + : "memory"); \ }) -#define __cmpxchg_release(ptr, old, new, size) \ +#define _arch_cmpxchg(ptr, old, new, sc_sfx, prepend, append) \ ({ \ __typeof__(ptr) __ptr = (ptr); \ - __typeof__(*(ptr)) __old = (old); \ - __typeof__(*(ptr)) __new = (new); \ - __typeof__(*(ptr)) __ret; \ - register unsigned int __rc; \ - switch (size) { \ + __typeof__(*(__ptr)) __old = (old); \ + __typeof__(*(__ptr)) __new = (new); \ + __typeof__(*(__ptr)) __ret; \ + \ + switch (sizeof(*__ptr)) { \ case 4: \ - __asm__ __volatile__ ( \ - RISCV_RELEASE_BARRIER \ - "0: lr.w %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc.w %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - "1:\n" \ - : "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \ - : "rJ" ((long)__old), "rJ" (__new) \ - : "memory"); \ + __arch_cmpxchg(".w", ".w" sc_sfx, prepend, append, \ + __ret, __ptr, (long), __old, __new); \ break; \ case 8: \ - __asm__ __volatile__ ( \ - RISCV_RELEASE_BARRIER \ - "0: lr.d %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc.d %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - "1:\n" \ - : "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \ - : "rJ" (__old), "rJ" (__new) \ - : "memory"); \ + __arch_cmpxchg(".d", ".d" sc_sfx, prepend, append, \ + __ret, __ptr, /**/, __old, __new); \ break; \ default: \ BUILD_BUG(); \ } \ - __ret; \ + (__typeof__(*(__ptr)))__ret; \ }) -#define arch_cmpxchg_release(ptr, o, n) \ -({ \ - __typeof__(*(ptr)) _o_ = (o); \ - __typeof__(*(ptr)) _n_ = (n); \ - (__typeof__(*(ptr))) __cmpxchg_release((ptr), \ - _o_, _n_, sizeof(*(ptr))); \ -}) +#define arch_cmpxchg_relaxed(ptr, o, n) \ + _arch_cmpxchg((ptr), (o), (n), "", "", "") -#define __cmpxchg(ptr, old, new, size) \ -({ \ - __typeof__(ptr) __ptr = (ptr); \ - __typeof__(*(ptr)) __old = (old); \ - __typeof__(*(ptr)) __new = (new); \ - __typeof__(*(ptr)) __ret; \ - register unsigned int __rc; \ - switch (size) { \ - case 4: \ - __asm__ __volatile__ ( \ - "0: lr.w %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc.w.rl %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - " fence rw, rw\n" \ - "1:\n" \ - : "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \ - : "rJ" ((long)__old), "rJ" (__new) \ - : "memory"); \ - break; \ - case 8: \ - __asm__ __volatile__ ( \ - "0: lr.d %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc.d.rl %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - " fence rw, rw\n" \ - "1:\n" \ - : "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \ - : "rJ" (__old), "rJ" (__new) \ - : "memory"); \ - break; \ - default: \ - BUILD_BUG(); \ - } \ - __ret; \ -}) +#define arch_cmpxchg_acquire(ptr, o, n) \ + _arch_cmpxchg((ptr), (o), (n), "", "", RISCV_ACQUIRE_BARRIER) + +#define arch_cmpxchg_release(ptr, o, n) \ + _arch_cmpxchg((ptr), (o), (n), "", RISCV_RELEASE_BARRIER, "") #define arch_cmpxchg(ptr, o, n) \ -({ \ - __typeof__(*(ptr)) _o_ = (o); \ - __typeof__(*(ptr)) _n_ = (n); \ - (__typeof__(*(ptr))) __cmpxchg((ptr), \ - _o_, _n_, sizeof(*(ptr))); \ -}) + _arch_cmpxchg((ptr), (o), (n), ".rl", "", " fence rw, rw\n") #define arch_cmpxchg_local(ptr, o, n) \ - (__cmpxchg_relaxed((ptr), (o), (n), sizeof(*(ptr)))) + arch_cmpxchg_relaxed((ptr), (o), (n)) #define arch_cmpxchg64(ptr, o, n) \ ({ \ From patchwork Thu Aug 10 04:03:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leonardo Bras X-Patchwork-Id: 13348753 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 42CA9C001B0 for ; Thu, 10 Aug 2023 04:04:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=VfsN9GKoSqwjoxuQU66Q7/OyJSWiB1T4NgFHlghaRXc=; b=lTqKc5ZE/iVk8v tbRgOfbhm43mByKtINz4FxqpmvdoiCbTxoYRY5Ap25CTn3cj3lb/8QQ40M0/nweuAwUf/d6mF1eGh uy4T/xIG2Per3zIrzCHhP6v/FJ9d95bl5o14Nbw77upH2peZURPBK491P5PYtjVMQJQ/enxBeoku6 9neVmwgZuDl9iRsP/tAXUGuYn05mDR2lTs9BmH453b99R6E01RFVaQmK/as9j2yVltMQCtyXooOrW VBxiELzNDqLSLLmB0gse1K6ZVetrVwmExyHTO4/Az7bnuTwZ+vBUm1pAPvu3/7O1BJrRIx1DXzkoW BajwaccUAhOFXjG/Vssw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qTwuU-006Sit-2H; Thu, 10 Aug 2023 04:04:30 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qTwuR-006Shf-1d for linux-riscv@lists.infradead.org; Thu, 10 Aug 2023 04:04:29 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1691640266; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UzQb2VuQCXwMdF/Rvu48wxhx8ntZSb1KliAOBLDDWgc=; b=Yl0jKg/rNhmszA7A+Bh4uoKpbcFDU/mHY3ZkVDr2kDlYyPIL5Kb5rWrm2FRyBguWK7u1+y 6D75oE2e1Cp8/ziaDrZl9RYFnUhjR0ypUuYSJ+rQV+jG73LniJpOTDWADhP8++bBSnTyeH SA/nWumDkAm1OfJN+DmKJlBjSTSxXQA= Received: from mail-oi1-f200.google.com (mail-oi1-f200.google.com [209.85.167.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-290-71isEnHNPYG-RiFsjDp2yw-1; Thu, 10 Aug 2023 00:04:24 -0400 X-MC-Unique: 71isEnHNPYG-RiFsjDp2yw-1 Received: by mail-oi1-f200.google.com with SMTP id 5614622812f47-3a1ec37ccbbso633857b6e.0 for ; Wed, 09 Aug 2023 21:04:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691640263; x=1692245063; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UzQb2VuQCXwMdF/Rvu48wxhx8ntZSb1KliAOBLDDWgc=; b=FazEVBPUM3Rdg3ing8xdVOyNbhJ/21mPdsjxj2HnTGaK5hwliRjrA4shMT3PR4+gUF rbGey7rShqR6KH40lskpGrxfq2Cnx3js/Bd0m8gjCW9tkwolsTOeLRERSD+QejFAuPFA r204tRtBGaeDA1fz4GkHJOxZbTId23omIzBQMv96KQ3eRWrcvD7MSBvU0q1dUUyzsIBN sqD0Ks3yPjfiBHuRDl5UstNgQrSlrpOroLR1sVRo5uLHHe4nLxl1xZTPGPXX10iSRSJ1 oCJdrkiKBWN6BWaqX325anBo0JjZwKtwMMc0P2afdDRVlW9b5TAwUDqG3QJ3qrM7JAQP 9wbQ== X-Gm-Message-State: AOJu0YwvhmXvB9LHMhxX8tBBP3OWPhpEF9V5c+yu5S6/YZRXatsttvNm r7+FBbL+KqDMYazTvQdw8fTHDVxUeUshzxL8U8Ecm3FMG0iFwBH1hbym9M6Hem+dg8bs4tYoZY8 1skuyhoJybkjizWUIQ7xvdb82EQ5t X-Received: by 2002:a05:6808:998:b0:3a7:540f:ca71 with SMTP id a24-20020a056808099800b003a7540fca71mr1397776oic.53.1691640262937; Wed, 09 Aug 2023 21:04:22 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEoKJIRWaoDwasWuxkhBvZiGi90hdZ5+o2Vt5j+J0Py97VRwdk+5mjOtZ8GjfL51BjhINmyUg== X-Received: by 2002:a05:6808:998:b0:3a7:540f:ca71 with SMTP id a24-20020a056808099800b003a7540fca71mr1397752oic.53.1691640262690; Wed, 09 Aug 2023 21:04:22 -0700 (PDT) Received: from localhost.localdomain ([2804:431:c7ec:e667:6b7d:ed55:c363:a088]) by smtp.gmail.com with ESMTPSA id b12-20020aca674c000000b003a7b5193909sm310087oiy.19.2023.08.09.21.04.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Aug 2023 21:04:22 -0700 (PDT) From: Leonardo Bras To: Will Deacon , Peter Zijlstra , Boqun Feng , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Leonardo Bras , Andrea Parri , Andrzej Hajda , Arnd Bergmann , Palmer Dabbelt , Guo Ren Cc: linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org Subject: [RFC PATCH v5 3/5] riscv/atomic.h : Deduplicate arch_atomic.* Date: Thu, 10 Aug 2023 01:03:45 -0300 Message-ID: <20230810040349.92279-5-leobras@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230810040349.92279-2-leobras@redhat.com> References: <20230810040349.92279-2-leobras@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230809_210427_616768_440358E3 X-CRM114-Status: GOOD ( 11.74 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Some functions use mostly the same asm for 32-bit and 64-bit versions. Make a macro that is generic enough and avoid code duplication. (This did not cause any change in generated asm) Signed-off-by: Leonardo Bras Reviewed-by: Guo Ren Reviewed-by: Andrea Parri --- arch/riscv/include/asm/atomic.h | 164 +++++++++++++++----------------- 1 file changed, 76 insertions(+), 88 deletions(-) diff --git a/arch/riscv/include/asm/atomic.h b/arch/riscv/include/asm/atomic.h index f5dfef6c2153..80cca7ac16fd 100644 --- a/arch/riscv/include/asm/atomic.h +++ b/arch/riscv/include/asm/atomic.h @@ -196,22 +196,28 @@ ATOMIC_OPS(xor, xor, i) #undef ATOMIC_FETCH_OP #undef ATOMIC_OP_RETURN +#define _arch_atomic_fetch_add_unless(_prev, _rc, counter, _a, _u, sfx) \ +({ \ + __asm__ __volatile__ ( \ + "0: lr." sfx " %[p], %[c]\n" \ + " beq %[p], %[u], 1f\n" \ + " add %[rc], %[p], %[a]\n" \ + " sc." sfx ".rl %[rc], %[rc], %[c]\n" \ + " bnez %[rc], 0b\n" \ + " fence rw, rw\n" \ + "1:\n" \ + : [p]"=&r" (_prev), [rc]"=&r" (_rc), [c]"+A" (counter) \ + : [a]"r" (_a), [u]"r" (_u) \ + : "memory"); \ +}) + /* This is required to provide a full barrier on success. */ static __always_inline int arch_atomic_fetch_add_unless(atomic_t *v, int a, int u) { int prev, rc; - __asm__ __volatile__ ( - "0: lr.w %[p], %[c]\n" - " beq %[p], %[u], 1f\n" - " add %[rc], %[p], %[a]\n" - " sc.w.rl %[rc], %[rc], %[c]\n" - " bnez %[rc], 0b\n" - " fence rw, rw\n" - "1:\n" - : [p]"=&r" (prev), [rc]"=&r" (rc), [c]"+A" (v->counter) - : [a]"r" (a), [u]"r" (u) - : "memory"); + _arch_atomic_fetch_add_unless(prev, rc, v->counter, a, u, "w"); + return prev; } #define arch_atomic_fetch_add_unless arch_atomic_fetch_add_unless @@ -222,77 +228,86 @@ static __always_inline s64 arch_atomic64_fetch_add_unless(atomic64_t *v, s64 a, s64 prev; long rc; - __asm__ __volatile__ ( - "0: lr.d %[p], %[c]\n" - " beq %[p], %[u], 1f\n" - " add %[rc], %[p], %[a]\n" - " sc.d.rl %[rc], %[rc], %[c]\n" - " bnez %[rc], 0b\n" - " fence rw, rw\n" - "1:\n" - : [p]"=&r" (prev), [rc]"=&r" (rc), [c]"+A" (v->counter) - : [a]"r" (a), [u]"r" (u) - : "memory"); + _arch_atomic_fetch_add_unless(prev, rc, v->counter, a, u, "d"); + return prev; } #define arch_atomic64_fetch_add_unless arch_atomic64_fetch_add_unless #endif +#define _arch_atomic_inc_unless_negative(_prev, _rc, counter, sfx) \ +({ \ + __asm__ __volatile__ ( \ + "0: lr." sfx " %[p], %[c]\n" \ + " bltz %[p], 1f\n" \ + " addi %[rc], %[p], 1\n" \ + " sc." sfx ".rl %[rc], %[rc], %[c]\n" \ + " bnez %[rc], 0b\n" \ + " fence rw, rw\n" \ + "1:\n" \ + : [p]"=&r" (_prev), [rc]"=&r" (_rc), [c]"+A" (counter) \ + : \ + : "memory"); \ +}) + static __always_inline bool arch_atomic_inc_unless_negative(atomic_t *v) { int prev, rc; - __asm__ __volatile__ ( - "0: lr.w %[p], %[c]\n" - " bltz %[p], 1f\n" - " addi %[rc], %[p], 1\n" - " sc.w.rl %[rc], %[rc], %[c]\n" - " bnez %[rc], 0b\n" - " fence rw, rw\n" - "1:\n" - : [p]"=&r" (prev), [rc]"=&r" (rc), [c]"+A" (v->counter) - : - : "memory"); + _arch_atomic_inc_unless_negative(prev, rc, v->counter, "w"); + return !(prev < 0); } #define arch_atomic_inc_unless_negative arch_atomic_inc_unless_negative +#define _arch_atomic_dec_unless_positive(_prev, _rc, counter, sfx) \ +({ \ + __asm__ __volatile__ ( \ + "0: lr." sfx " %[p], %[c]\n" \ + " bgtz %[p], 1f\n" \ + " addi %[rc], %[p], -1\n" \ + " sc." sfx ".rl %[rc], %[rc], %[c]\n" \ + " bnez %[rc], 0b\n" \ + " fence rw, rw\n" \ + "1:\n" \ + : [p]"=&r" (_prev), [rc]"=&r" (_rc), [c]"+A" (counter) \ + : \ + : "memory"); \ +}) + static __always_inline bool arch_atomic_dec_unless_positive(atomic_t *v) { int prev, rc; - __asm__ __volatile__ ( - "0: lr.w %[p], %[c]\n" - " bgtz %[p], 1f\n" - " addi %[rc], %[p], -1\n" - " sc.w.rl %[rc], %[rc], %[c]\n" - " bnez %[rc], 0b\n" - " fence rw, rw\n" - "1:\n" - : [p]"=&r" (prev), [rc]"=&r" (rc), [c]"+A" (v->counter) - : - : "memory"); + _arch_atomic_dec_unless_positive(prev, rc, v->counter, "w"); + return !(prev > 0); } #define arch_atomic_dec_unless_positive arch_atomic_dec_unless_positive +#define _arch_atomic_dec_if_positive(_prev, _rc, counter, sfx) \ +({ \ + __asm__ __volatile__ ( \ + "0: lr." sfx " %[p], %[c]\n" \ + " addi %[rc], %[p], -1\n" \ + " bltz %[rc], 1f\n" \ + " sc." sfx ".rl %[rc], %[rc], %[c]\n" \ + " bnez %[rc], 0b\n" \ + " fence rw, rw\n" \ + "1:\n" \ + : [p]"=&r" (_prev), [rc]"=&r" (_rc), [c]"+A" (counter) \ + : \ + : "memory"); \ +}) + static __always_inline int arch_atomic_dec_if_positive(atomic_t *v) { int prev, rc; - __asm__ __volatile__ ( - "0: lr.w %[p], %[c]\n" - " addi %[rc], %[p], -1\n" - " bltz %[rc], 1f\n" - " sc.w.rl %[rc], %[rc], %[c]\n" - " bnez %[rc], 0b\n" - " fence rw, rw\n" - "1:\n" - : [p]"=&r" (prev), [rc]"=&r" (rc), [c]"+A" (v->counter) - : - : "memory"); + _arch_atomic_dec_if_positive(prev, rc, v->counter, "w"); + return prev - 1; } @@ -304,17 +319,8 @@ static __always_inline bool arch_atomic64_inc_unless_negative(atomic64_t *v) s64 prev; long rc; - __asm__ __volatile__ ( - "0: lr.d %[p], %[c]\n" - " bltz %[p], 1f\n" - " addi %[rc], %[p], 1\n" - " sc.d.rl %[rc], %[rc], %[c]\n" - " bnez %[rc], 0b\n" - " fence rw, rw\n" - "1:\n" - : [p]"=&r" (prev), [rc]"=&r" (rc), [c]"+A" (v->counter) - : - : "memory"); + _arch_atomic_inc_unless_negative(prev, rc, v->counter, "d"); + return !(prev < 0); } @@ -325,17 +331,8 @@ static __always_inline bool arch_atomic64_dec_unless_positive(atomic64_t *v) s64 prev; long rc; - __asm__ __volatile__ ( - "0: lr.d %[p], %[c]\n" - " bgtz %[p], 1f\n" - " addi %[rc], %[p], -1\n" - " sc.d.rl %[rc], %[rc], %[c]\n" - " bnez %[rc], 0b\n" - " fence rw, rw\n" - "1:\n" - : [p]"=&r" (prev), [rc]"=&r" (rc), [c]"+A" (v->counter) - : - : "memory"); + _arch_atomic_dec_unless_positive(prev, rc, v->counter, "d"); + return !(prev > 0); } @@ -346,17 +343,8 @@ static __always_inline s64 arch_atomic64_dec_if_positive(atomic64_t *v) s64 prev; long rc; - __asm__ __volatile__ ( - "0: lr.d %[p], %[c]\n" - " addi %[rc], %[p], -1\n" - " bltz %[rc], 1f\n" - " sc.d.rl %[rc], %[rc], %[c]\n" - " bnez %[rc], 0b\n" - " fence rw, rw\n" - "1:\n" - : [p]"=&r" (prev), [rc]"=&r" (rc), [c]"+A" (v->counter) - : - : "memory"); + _arch_atomic_dec_if_positive(prev, rc, v->counter, "d"); + return prev - 1; } From patchwork Thu Aug 10 04:03:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leonardo Bras X-Patchwork-Id: 13348754 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id F0A32C001B0 for ; Thu, 10 Aug 2023 04:04:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=+8LXlnLMJ+i11qJbvDt8Lqbc4WCq9mmT0tJXinXTdE4=; b=znjIJYD4QsdOnj 2Db21o51XXIT3rGlkMbLCMWCXTX1Z6VBZS3EcF3cXmH25U62+ul/idZgkbvjj82FZouNlg/gTd3np LKkI/kQxZFdp1QxWBLha0BckpcrzelwNchdRqeNZX00c5RuCZ8VE1MDYhVcszoK+Dw1KthHXSimvO JpFe1y9LbE30FIIyGyT6az3ryXUevHhfkmy5bDkwIQKeh3ev/Gk54Zw8Kw/cmcWkxy/tD0jnlm1C+ jCHGPaIRHnmY9RqarjUl3pbG2qPTWNlVRWtkS7mBWOEB1H/Zt8UgeZYCPfOhXQvTA5jugDgde3uza dw5O5+7fj5SlkgShOW9g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qTwuY-006Skc-1J; Thu, 10 Aug 2023 04:04:34 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qTwuV-006Siv-0f for linux-riscv@lists.infradead.org; Thu, 10 Aug 2023 04:04:32 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1691640270; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VLwtkHrR4Hnv0SG2htyTx6O/VUdZ3YGDzB9YQqYciKw=; b=ZJh+gD58YFKorM+mjD1OZHbfXfE2+sxf5mdm30T6EVunMdzZYqnI7ks7hp8dm0xRK55iPj O6cocUNU5aOQWSHC0FXZYDtd32oX7HOhxv4Qap+8gRFalvL1z/VTq4YOIQQOKucgE4ex+/ d0DKpIxgIrZvClrgQzWEHrmAF+/BMlA= Received: from mail-oi1-f197.google.com (mail-oi1-f197.google.com [209.85.167.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-497-e8LgNQtnOuqESZVNJTAffA-1; Thu, 10 Aug 2023 00:04:29 -0400 X-MC-Unique: e8LgNQtnOuqESZVNJTAffA-1 Received: by mail-oi1-f197.google.com with SMTP id 5614622812f47-3a7697e580fso597221b6e.1 for ; Wed, 09 Aug 2023 21:04:28 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691640268; x=1692245068; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VLwtkHrR4Hnv0SG2htyTx6O/VUdZ3YGDzB9YQqYciKw=; b=G7ertk+Rp/PtALQTALXaipxi/Uqg54TsetDjIxIc2SZwf47LEqm7GJVg2Z4TG4aDXN +41iSRZ6BiZbB52KxZlpoFS5JbrlKOtls4vnnfheRYL+630RVnP2/zWb9t8NQVvYDeUg b1HyJJd6p/EcpdwvyrjQepM1ueHmLsBHRw0iKs6V21n3zSyPkadG778dl87iV0NBHYpE mQsZfFwkNk1tFVtbgYw+i2+iw+zxGD+tzyytQEqzf6xXBRBcAu51QOAe0imDajAFdVp+ zI+tCuADMiYKOP30k31QauhY+z26sZduA0cejaZsFHmDjZ9oYuIWvUdtMfm7V2fJuWXe 3BcA== X-Gm-Message-State: AOJu0YzmgXRISYQruo7MdYgiqVVBVWyXsIXNib6Q4c4G4gklALnTzPN6 hXhgBHgC4XfvEJoATzP2V3uYtbxokWXy4wADpi9B4W5vQ6ut5sTZ+WvYZtD4qyxI9knChapn02w BEOEmf6ilnv8jUuypPwotvGaNgkv5 X-Received: by 2002:a05:6808:10cc:b0:3a3:d7de:5cfa with SMTP id s12-20020a05680810cc00b003a3d7de5cfamr1819120ois.11.1691640268299; Wed, 09 Aug 2023 21:04:28 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEAdIAbs7Buum1URpEJTh5T57eb0r1XDUCa+aKbZcc1fbEPGJr7vKxHbONmynKMK+8Fsxl9cg== X-Received: by 2002:a05:6808:10cc:b0:3a3:d7de:5cfa with SMTP id s12-20020a05680810cc00b003a3d7de5cfamr1819092ois.11.1691640268064; Wed, 09 Aug 2023 21:04:28 -0700 (PDT) Received: from localhost.localdomain ([2804:431:c7ec:e667:6b7d:ed55:c363:a088]) by smtp.gmail.com with ESMTPSA id b12-20020aca674c000000b003a7b5193909sm310087oiy.19.2023.08.09.21.04.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Aug 2023 21:04:27 -0700 (PDT) From: Leonardo Bras To: Will Deacon , Peter Zijlstra , Boqun Feng , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Leonardo Bras , Andrea Parri , Andrzej Hajda , Arnd Bergmann , Palmer Dabbelt , Guo Ren Cc: linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org Subject: [RFC PATCH v5 4/5] riscv/cmpxchg: Implement cmpxchg for variables of size 1 and 2 Date: Thu, 10 Aug 2023 01:03:46 -0300 Message-ID: <20230810040349.92279-6-leobras@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230810040349.92279-2-leobras@redhat.com> References: <20230810040349.92279-2-leobras@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230809_210431_314659_A0F9A14C X-CRM114-Status: GOOD ( 12.63 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org cmpxchg for variables of size 1-byte and 2-bytes is not yet available for riscv, even though its present in other architectures such as arm64 and x86. This could lead to not being able to implement some locking mechanisms or requiring some rework to make it work properly. Implement 1-byte and 2-bytes cmpxchg in order to achieve parity with other architectures. Signed-off-by: Leonardo Bras --- arch/riscv/include/asm/cmpxchg.h | 34 ++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index e3e0ac7ba061..ac9d0eeb74e6 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -72,6 +72,35 @@ * indicated by comparing RETURN with OLD. */ +#define __arch_cmpxchg_masked(sc_sfx, prepend, append, r, p, o, n) \ +({ \ + u32 *__ptr32b = (u32 *)((ulong)(p) & ~0x3); \ + ulong __s = ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \ + ulong __mask = GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \ + << __s; \ + ulong __newx = (ulong)(n) << __s; \ + ulong __oldx = (ulong)(o) << __s; \ + ulong __retx; \ + ulong __rc; \ + \ + __asm__ __volatile__ ( \ + prepend \ + "0: lr.w %0, %2\n" \ + " and %1, %0, %z5\n" \ + " bne %1, %z3, 1f\n" \ + " and %1, %0, %z6\n" \ + " or %1, %1, %z4\n" \ + " sc.w" sc_sfx " %1, %1, %2\n" \ + " bnez %1, 0b\n" \ + append \ + "1:\n" \ + : "=&r" (__retx), "=&r" (__rc), "+A" (*(__ptr32b)) \ + : "rJ" ((long)__oldx), "rJ" (__newx), \ + "rJ" (__mask), "rJ" (~__mask) \ + : "memory"); \ + \ + r = (__typeof__(*(p)))((__retx & __mask) >> __s); \ +}) #define __arch_cmpxchg(lr_sfx, sc_sfx, prepend, append, r, p, co, o, n) \ ({ \ @@ -98,6 +127,11 @@ __typeof__(*(__ptr)) __ret; \ \ switch (sizeof(*__ptr)) { \ + case 1: \ + case 2: \ + __arch_cmpxchg_masked(sc_sfx, prepend, append, \ + __ret, __ptr, __old, __new); \ + break; \ case 4: \ __arch_cmpxchg(".w", ".w" sc_sfx, prepend, append, \ __ret, __ptr, (long), __old, __new); \ From patchwork Thu Aug 10 04:03:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leonardo Bras X-Patchwork-Id: 13348755 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 53320C001B0 for ; Thu, 10 Aug 2023 04:04:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=izOomclOKv3b1AtOCt/0ghbMIzYcsmGSl6h9IX34lCI=; b=EmbNYxHkf2UJG6 n7BbyiPn+HQGxxOwW9XNIH/RhG+OvBQEaFFs8XjC5EJttGqz3Vom2z1+U/rdWf7lxe0nrMa5ah0Uh Zv/NxNTaZc6qCjCxBhPUGvgakVh24sa8Ps6eUSD+hqajup9Q9Aec1+LVsAa+wEZVxa/EgLOAXyFPf EaWBq2XyiydjC+bUGLJnTi79BE2oDWQCQK8O8TS1J4eG6JlhfN4F8m8myph8CjFu6fcPsSsGUrv39 ilMQKUh+UH6nn64pFBxhOIv9b72Z1qYMuI0tNaszdAIuTGfGIxXyGHWnrXWYFFUmXg44ncnzCcIKA QxkqMyDOGseiHq5XFJxg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qTwuf-006SnH-0I; Thu, 10 Aug 2023 04:04:41 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qTwuc-006SmF-1F for linux-riscv@lists.infradead.org; Thu, 10 Aug 2023 04:04:39 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1691640277; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=S9RDXJ5aWrysjU+rhwlqeMwMOsK56lpvgf+6LxPUCTg=; b=Nsy90srPitxzmQowEvTx4za6vEVJ3udyGRjbKCRQTraRRsyART3JMTHtZh2xxf2kJFVOvf adiT76RH2uudJhoZ3oTMasKB47bxDRLoPVS8Zgmp+M2O9UaT9ji0nlRsGuV+Ru1sR2S6nH Vgt7L3RDE9P/MvUts60C1LhYKQwCBnU= Received: from mail-oi1-f198.google.com (mail-oi1-f198.google.com [209.85.167.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-92-R_pjAyJZNYmlTVgBtr4S6g-1; Thu, 10 Aug 2023 00:04:34 -0400 X-MC-Unique: R_pjAyJZNYmlTVgBtr4S6g-1 Received: by mail-oi1-f198.google.com with SMTP id 5614622812f47-3a7292a2ad5so606782b6e.0 for ; Wed, 09 Aug 2023 21:04:34 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691640273; x=1692245073; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=S9RDXJ5aWrysjU+rhwlqeMwMOsK56lpvgf+6LxPUCTg=; b=iVQhxNB52dYNv7FnvJIgc9RRgBk8VKIdg27JXw+S384HHVgt3wb1b02S0SZlj88jJh DYs4N11DZLKX/lQRZITmb4yhNoe6aCCg/aOzrQEUtHTlPEgwNcZ2Kdv5tcEhekFblj8I Lob80nmKZnmoOQ//gqF9xCWNQ2UvBOrHgYewyqZpSe4mOceiWNufbxYRmJb1nqg9sHlG Opq5I/pVJnJrlQHJaFkNEAG/FrS/T/jnyBt5Vyi6wSUFLFjDpAx6q2opFkytN9If3LMH kf957PVvu/i9t0HBVJusYbUztnMmUZT6r1ivbGZAmo04d+tAnZ3n+VGzs4MCxRMqUPOU MyRg== X-Gm-Message-State: AOJu0YzdzBKwoRvs3YHtYEUVdG6QDKN10yOkwRmWmmaRy4A7uk9b55kF UA7D5CFBfQwzFO6M+avcWeOnBH1jSHlMZOLQ+Fb7aoTFYE9WaOfREh0l7lU8TXHBvPQGi9PJFYK BP5Lck+gjMD0RWSxdRwXhicdoayct X-Received: by 2002:a05:6808:2124:b0:3a7:82e8:8fd1 with SMTP id r36-20020a056808212400b003a782e88fd1mr1804200oiw.20.1691640273708; Wed, 09 Aug 2023 21:04:33 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEJGYyTweAhht6teeaheQlpzQ7zjq+oFXSk4DIrLLMDF2ExtlYA68o0L4A2hjKnDp8s+BMmiQ== X-Received: by 2002:a05:6808:2124:b0:3a7:82e8:8fd1 with SMTP id r36-20020a056808212400b003a782e88fd1mr1804180oiw.20.1691640273540; Wed, 09 Aug 2023 21:04:33 -0700 (PDT) Received: from localhost.localdomain ([2804:431:c7ec:e667:6b7d:ed55:c363:a088]) by smtp.gmail.com with ESMTPSA id b12-20020aca674c000000b003a7b5193909sm310087oiy.19.2023.08.09.21.04.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Aug 2023 21:04:33 -0700 (PDT) From: Leonardo Bras To: Will Deacon , Peter Zijlstra , Boqun Feng , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Leonardo Bras , Andrea Parri , Andrzej Hajda , Arnd Bergmann , Palmer Dabbelt , Guo Ren Cc: linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org Subject: [RFC PATCH v5 5/5] riscv/cmpxchg: Implement xchg for variables of size 1 and 2 Date: Thu, 10 Aug 2023 01:03:47 -0300 Message-ID: <20230810040349.92279-7-leobras@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230810040349.92279-2-leobras@redhat.com> References: <20230810040349.92279-2-leobras@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230809_210438_499118_C3E059CB X-CRM114-Status: GOOD ( 12.27 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org xchg for variables of size 1-byte and 2-bytes is not yet available for riscv, even though its present in other architectures such as arm64 and x86. This could lead to not being able to implement some locking mechanisms or requiring some rework to make it work properly. Implement 1-byte and 2-bytes xchg in order to achieve parity with other architectures. Signed-off-by: Leonardo Bras --- arch/riscv/include/asm/cmpxchg.h | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index ac9d0eeb74e6..26cea2395aae 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -11,6 +11,31 @@ #include #include +#define __arch_xchg_masked(prepend, append, r, p, n) \ +({ \ + u32 *__ptr32b = (u32 *)((ulong)(p) & ~0x3); \ + ulong __s = ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \ + ulong __mask = GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \ + << __s; \ + ulong __newx = (ulong)(n) << __s; \ + ulong __retx; \ + ulong __rc; \ + \ + __asm__ __volatile__ ( \ + prepend \ + "0: lr.w %0, %2\n" \ + " and %1, %0, %z4\n" \ + " or %1, %1, %z3\n" \ + " sc.w %1, %1, %2\n" \ + " bnez %1, 0b\n" \ + append \ + : "=&r" (__retx), "=&r" (__rc), "+A" (*(__ptr32b)) \ + : "rJ" (__newx), "rJ" (~__mask) \ + : "memory"); \ + \ + r = (__typeof__(*(p)))((__retx & __mask) >> __s); \ +}) + #define __arch_xchg(sfx, prepend, append, r, p, n) \ ({ \ __asm__ __volatile__ ( \ @@ -27,7 +52,13 @@ __typeof__(ptr) __ptr = (ptr); \ __typeof__(*(__ptr)) __new = (new); \ __typeof__(*(__ptr)) __ret; \ + \ switch (sizeof(*__ptr)) { \ + case 1: \ + case 2: \ + __arch_xchg_masked(prepend, append, \ + __ret, __ptr, __new); \ + break; \ case 4: \ __arch_xchg(".w" sfx, prepend, append, \ __ret, __ptr, __new); \