From patchwork Fri Aug 4 08:48:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leonardo Bras X-Patchwork-Id: 13341524 X-Patchwork-Delegate: palmer@dabbelt.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1E997C001DF for ; Fri, 4 Aug 2023 08:49:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Avx4GgzS5eYJhbU16Xi4I4BzjD0lLLUmJWZ2WyALGjs=; b=i5P3Lv9Y9BexsE zyvOmqfR8mau2TGSCILRz4GBWNobdRceGXnaSC0ArBu66b1mdQvGAxQrMtBvE92fJYUi6FFRNJlxD 9D2THEVpKcGvgWRrZ34lLnQbyD8GOEMSKypmNZtKI7CDSVSjE8HrCwsxPh5MxHmBSHzxmzViy8BNx vN0tIlcIkL6X1h23+nZso7DCvOnBQxX50mm8c54JHIRUn5Ci4UM2rubyl8rRvZkQDa7EDFAazPbmS aUh/vthf1brjibAr+uOdzGBSk1qYOae6kkOAFVlOUW15gLqGFN3VZ6JP6CWQesRdQQJCBZFYyIWwd 8ko7cyPguVKYUpjtlgXw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qRqV3-00Bvds-0G; Fri, 04 Aug 2023 08:49:33 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qRqUz-00Bvbs-2T for linux-riscv@lists.infradead.org; Fri, 04 Aug 2023 08:49:31 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1691138969; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cG8axZDwBuWJQ+k2FsV81xz+WKLWi2QNSuhJ6eUeaqw=; b=gY3wSVFp5BM8h1wOD6jkzMlGKtIicFMB0iQgaKrQVIBwbKa8ankz0l2eZb0DpINw+bjGFM /HBBjBF9ln9C4UoSTw62CYpXKj+ua84Tssz0BFDQgsXIeIY2BmTWeUlge/WoUCogvZpCTM IpaYZkFvk//nw2+f3ah34yas9VinnYw= Received: from mail-ot1-f69.google.com (mail-ot1-f69.google.com [209.85.210.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-43--v-s0DD6Psaj18BKIGjPrw-1; Fri, 04 Aug 2023 04:49:25 -0400 X-MC-Unique: -v-s0DD6Psaj18BKIGjPrw-1 Received: by mail-ot1-f69.google.com with SMTP id 46e09a7af769-6b9d8b76be5so3106583a34.1 for ; Fri, 04 Aug 2023 01:49:25 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691138965; x=1691743765; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cG8axZDwBuWJQ+k2FsV81xz+WKLWi2QNSuhJ6eUeaqw=; b=aOTA9nmR8wLDTNiAyY4nlcJxpNcT60GzSM0S3TC/atcI4phaWpKZF85LveX0kdLepu ixPonUJTcy1O6620a0lVZBTzrKa55yCIxxdcMGCkL3RnrbWC8laQ4486k/rEdTUHcvnF na4HH5ZpH6xyFN3BvcgjSUj3zOsmTdFIGMsN5QrDKf4K+QyUBc/SxOYfsxEiyNNCPOlS 5cQfqC78hm9BICHhBAU+xAfAs88V2puVx43qSyfTYTmE4xQMTg1OCHfFjX5aAwZIJ0T9 LLs6g9+ftl9O65u9UUme23QwoCvNZ+7uR5LDUTAhW3zGQtLG5IrEMoA9uvvV6+VVAofp QZPg== X-Gm-Message-State: AOJu0YycmhwfJl+XFZIAwzIR5ieW75LFg8KWG5eD1OObRDqQ1QvBApT9 GUhaKIKU8a20UDlGlUs3ekdJrp1WeEoB/m4To0JLAPyWKrXwzr+r3R933w9mi6GgL7zjhlV2AHs TY4fa/uu5xL72sppe0bcbwbeQoHUk X-Received: by 2002:a05:6870:5b98:b0:1be:c8e2:3ec3 with SMTP id em24-20020a0568705b9800b001bec8e23ec3mr1046105oab.14.1691138964800; Fri, 04 Aug 2023 01:49:24 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGfXG4yRYPIwmiiQVWTf4MQ+ktY1Xgv8O9HBUKLqIvtNOkPHWsNOPcnRegEr6JIF1zXzs2y/A== X-Received: by 2002:a05:6870:5b98:b0:1be:c8e2:3ec3 with SMTP id em24-20020a0568705b9800b001bec8e23ec3mr1046089oab.14.1691138964579; Fri, 04 Aug 2023 01:49:24 -0700 (PDT) Received: from localhost.localdomain ([2804:1b3:a801:d380:694f:4f52:764c:4b7f]) by smtp.gmail.com with ESMTPSA id f8-20020a4ab008000000b0055516447257sm685679oon.29.2023.08.04.01.49.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 04 Aug 2023 01:49:24 -0700 (PDT) From: Leonardo Bras To: Will Deacon , Peter Zijlstra , Boqun Feng , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Leonardo Bras , Andrea Parri , Geert Uytterhoeven , Andrzej Hajda , Palmer Dabbelt , Guo Ren Cc: linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org Subject: [RFC PATCH v3 1/5] riscv/cmpxchg: Deduplicate xchg() asm functions Date: Fri, 4 Aug 2023 05:48:54 -0300 Message-ID: <20230804084900.1135660-3-leobras@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230804084900.1135660-2-leobras@redhat.com> References: <20230804084900.1135660-2-leobras@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230804_014929_870986_8D5FC000 X-CRM114-Status: GOOD ( 11.68 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org In this header every xchg define (_relaxed, _acquire, _release, vanilla) contain it's own asm file, both for 4-byte variables an 8-byte variables, on a total of 8 versions of mostly the same asm. This is usually bad, as it means any change may be done in up to 8 different places. Unify those versions by creating a new define with enough parameters to generate any version of the previous 8. Then unify the result under a more general define, and simplify arch_xchg* generation. (This did not cause any change in generated asm) Signed-off-by: Leonardo Bras Reviewed-by: Andrea Parri --- arch/riscv/include/asm/cmpxchg.h | 136 +++++-------------------------- 1 file changed, 22 insertions(+), 114 deletions(-) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index 2f4726d3cfcc..ec4ea4f3f908 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -11,60 +11,30 @@ #include #include -#define __xchg_relaxed(ptr, new, size) \ +#define __arch_xchg(sfx, prepend, append, r, p, n) \ ({ \ - __typeof__(ptr) __ptr = (ptr); \ - __typeof__(new) __new = (new); \ - __typeof__(*(ptr)) __ret; \ - switch (size) { \ - case 4: \ - __asm__ __volatile__ ( \ - " amoswap.w %0, %2, %1\n" \ - : "=r" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ - break; \ - case 8: \ - __asm__ __volatile__ ( \ - " amoswap.d %0, %2, %1\n" \ - : "=r" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ - break; \ - default: \ - BUILD_BUG(); \ - } \ - __ret; \ -}) - -#define arch_xchg_relaxed(ptr, x) \ -({ \ - __typeof__(*(ptr)) _x_ = (x); \ - (__typeof__(*(ptr))) __xchg_relaxed((ptr), \ - _x_, sizeof(*(ptr))); \ + __asm__ __volatile__ ( \ + prepend \ + " amoswap" sfx " %0, %2, %1\n" \ + append \ + : "=r" (r), "+A" (*(p)) \ + : "r" (n) \ + : "memory"); \ }) -#define __xchg_acquire(ptr, new, size) \ +#define _arch_xchg(ptr, new, sfx, prepend, append) \ ({ \ - __typeof__(ptr) __ptr = (ptr); \ - __typeof__(new) __new = (new); \ + __typeof__(*(ptr)) __new = (new); \ __typeof__(*(ptr)) __ret; \ - switch (size) { \ + __typeof__(ptr) __ptr = (ptr); \ + switch (sizeof(*__ptr)) { \ case 4: \ - __asm__ __volatile__ ( \ - " amoswap.w %0, %2, %1\n" \ - RISCV_ACQUIRE_BARRIER \ - : "=r" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ + __arch_xchg(".w" sfx, prepend, append, \ + __ret, __ptr, __new); \ break; \ case 8: \ - __asm__ __volatile__ ( \ - " amoswap.d %0, %2, %1\n" \ - RISCV_ACQUIRE_BARRIER \ - : "=r" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ + __arch_xchg(".d" sfx, prepend, append, \ + __ret, __ptr, __new); \ break; \ default: \ BUILD_BUG(); \ @@ -72,79 +42,17 @@ __ret; \ }) -#define arch_xchg_acquire(ptr, x) \ -({ \ - __typeof__(*(ptr)) _x_ = (x); \ - (__typeof__(*(ptr))) __xchg_acquire((ptr), \ - _x_, sizeof(*(ptr))); \ -}) +#define arch_xchg_relaxed(ptr, x) \ + _arch_xchg(ptr, x, "", "", "") -#define __xchg_release(ptr, new, size) \ -({ \ - __typeof__(ptr) __ptr = (ptr); \ - __typeof__(new) __new = (new); \ - __typeof__(*(ptr)) __ret; \ - switch (size) { \ - case 4: \ - __asm__ __volatile__ ( \ - RISCV_RELEASE_BARRIER \ - " amoswap.w %0, %2, %1\n" \ - : "=r" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ - break; \ - case 8: \ - __asm__ __volatile__ ( \ - RISCV_RELEASE_BARRIER \ - " amoswap.d %0, %2, %1\n" \ - : "=r" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ - break; \ - default: \ - BUILD_BUG(); \ - } \ - __ret; \ -}) +#define arch_xchg_acquire(ptr, x) \ + _arch_xchg(ptr, x, "", "", RISCV_ACQUIRE_BARRIER) #define arch_xchg_release(ptr, x) \ -({ \ - __typeof__(*(ptr)) _x_ = (x); \ - (__typeof__(*(ptr))) __xchg_release((ptr), \ - _x_, sizeof(*(ptr))); \ -}) - -#define __arch_xchg(ptr, new, size) \ -({ \ - __typeof__(ptr) __ptr = (ptr); \ - __typeof__(new) __new = (new); \ - __typeof__(*(ptr)) __ret; \ - switch (size) { \ - case 4: \ - __asm__ __volatile__ ( \ - " amoswap.w.aqrl %0, %2, %1\n" \ - : "=r" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ - break; \ - case 8: \ - __asm__ __volatile__ ( \ - " amoswap.d.aqrl %0, %2, %1\n" \ - : "=r" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ - break; \ - default: \ - BUILD_BUG(); \ - } \ - __ret; \ -}) + _arch_xchg(ptr, x, "", RISCV_RELEASE_BARRIER, "") #define arch_xchg(ptr, x) \ -({ \ - __typeof__(*(ptr)) _x_ = (x); \ - (__typeof__(*(ptr))) __arch_xchg((ptr), _x_, sizeof(*(ptr))); \ -}) + _arch_xchg(ptr, x, ".aqrl", "", "") #define xchg32(ptr, x) \ ({ \ From patchwork Fri Aug 4 08:48:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leonardo Bras X-Patchwork-Id: 13341525 X-Patchwork-Delegate: palmer@dabbelt.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8F24CC001DB for ; Fri, 4 Aug 2023 08:49:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=GhbQNzoGMzCtmYEY/k+AVnlFkcaxvo5JicgInhaH/vo=; b=qLqMmTf1oM3zct diBWRQs3YKdhxtKPlhauS3rsKgNqPJ3MxjAgKwV4XZlnabaGQQcapPIi2kjliH/j7MuwrVW80AjaW /nbOCzdBtiALQ+dAqySjd0fb+fFFjvjoDTh7DJDYYVgGEAU8bmf0st0//sCxZ6RkSTt3ntnyk0P2q hR+p490LAgQs1URuI6lLOxKZo3Wc0Eb4Et5IQuxOWG1ExdErBhAhkNbF9333TZ5A34gA/ADzHyAy2 trRpWM44vQZmw3HqAnIoIIYeR/lWDNQSk5sewvMKJSebkdqKJTvJ+ctF6s0ZYhJLd1ejHGXMBOwO4 wdxpD9VH/DMuTmXY7AWA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qRqV6-00Bvfs-2T; Fri, 04 Aug 2023 08:49:36 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qRqV2-00BvdL-0U for linux-riscv@lists.infradead.org; Fri, 04 Aug 2023 08:49:34 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1691138971; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=68droSgtD1yUeWujN9KsDS4jg7ssA0NCkb8PZ8Tb+SA=; b=eTYHYXcn3PSNT8Ro9GpD1WFgnxKj1ugxrvXHZN/Zk6QP/e7MpQpP34zWY2hS/WL0xjS380 TO3ozJvK+DxusgA+gkoYjWY70JW4gXROyJKPU9dJDXgeMqM1yfuvK6YICWbi5vGKWWWo6h ykWF/alVAfLIbHZJPfcyLAT0+xxy9K0= Received: from mail-oo1-f72.google.com (mail-oo1-f72.google.com [209.85.161.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-65-GGLZQb3rPvOXZJVDntxkNg-1; Fri, 04 Aug 2023 04:49:30 -0400 X-MC-Unique: GGLZQb3rPvOXZJVDntxkNg-1 Received: by mail-oo1-f72.google.com with SMTP id 006d021491bc7-56c8f0995faso2551725eaf.0 for ; Fri, 04 Aug 2023 01:49:29 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691138969; x=1691743769; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=68droSgtD1yUeWujN9KsDS4jg7ssA0NCkb8PZ8Tb+SA=; b=F4zfrM9R90mw/6f9932uDvX2xVu2qz9eGgH1xSXnlk4zNmhOfP0LVBkCHWqH63/Upr 1QT7usGT7umRVeRHGNz1Xz/3MUPH13h9I9oWD/FT0LxkwaOosZcjpoFcXn9NwpjnYB5B KtLDHS5EVDJaTgqH2hlfvUrns1T5SU8mJwI53x0im5207+m1L6b0s4tXeiRmfK9dIyai bQtCkBQwNhgp3629RyW3u53MxpEIqKIc+RA2sFC52OUxuWenBpwLYGPlSHhqXagYymi+ 5UbaRMSIMJELXq3+qI6BVbDfQDeYhkLTRC5DhJHpukx+9LYNWRBVRnzkVLT3wEZi24nl WcHg== X-Gm-Message-State: AOJu0Yx5Mb5xQggIUruvKxRy0UEMotZjLh6oi/hu1H/i/EYZBZDKAiBa iixkr+U1pxPVT0EcKUE+EibAYVUmkx/wXSXVBC5jhzaedq5kjuD5TGEeUR1bfgUegAswYC5/aO4 Y0FvyDaxRnvSbcaAE7gYtGS6ZfkELGw1vZKDW X-Received: by 2002:a4a:3018:0:b0:56c:7428:4a35 with SMTP id q24-20020a4a3018000000b0056c74284a35mr1050180oof.7.1691138969020; Fri, 04 Aug 2023 01:49:29 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFjkpXXm0/Ur8WNB8kbC0Thn+LObgXQCNBatmDg00BqWmkNhBFSyxWSvjCulJQefjTpcM1mRQ== X-Received: by 2002:a4a:3018:0:b0:56c:7428:4a35 with SMTP id q24-20020a4a3018000000b0056c74284a35mr1050170oof.7.1691138968735; Fri, 04 Aug 2023 01:49:28 -0700 (PDT) Received: from localhost.localdomain ([2804:1b3:a801:d380:694f:4f52:764c:4b7f]) by smtp.gmail.com with ESMTPSA id f8-20020a4ab008000000b0055516447257sm685679oon.29.2023.08.04.01.49.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 04 Aug 2023 01:49:28 -0700 (PDT) From: Leonardo Bras To: Will Deacon , Peter Zijlstra , Boqun Feng , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Leonardo Bras , Andrea Parri , Geert Uytterhoeven , Andrzej Hajda , Palmer Dabbelt , Guo Ren Cc: linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org Subject: [RFC PATCH v3 2/5] riscv/cmpxchg: Deduplicate cmpxchg() asm and macros Date: Fri, 4 Aug 2023 05:48:55 -0300 Message-ID: <20230804084900.1135660-4-leobras@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230804084900.1135660-2-leobras@redhat.com> References: <20230804084900.1135660-2-leobras@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230804_014932_272601_C5BFC5A9 X-CRM114-Status: GOOD ( 11.90 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org In this header every cmpxchg define (_relaxed, _acquire, _release, vanilla) contain it's own asm file, both for 4-byte variables an 8-byte variables, on a total of 8 versions of mostly the same asm. This is usually bad, as it means any change may be done in up to 8 different places. Unify those versions by creating a new define with enough parameters to generate any version of the previous 8. Then unify the result under a more general define, and simplify arch_cmpxchg* generation (This did not cause any change in generated asm) Signed-off-by: Leonardo Bras Reviewed-by: Andrea Parri --- arch/riscv/include/asm/cmpxchg.h | 187 +++++-------------------------- 1 file changed, 29 insertions(+), 158 deletions(-) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index ec4ea4f3f908..5a07646fae65 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -71,127 +71,40 @@ * store NEW in MEM. Return the initial value in MEM. Success is * indicated by comparing RETURN with OLD. */ -#define __cmpxchg_relaxed(ptr, old, new, size) \ -({ \ - __typeof__(ptr) __ptr = (ptr); \ - __typeof__(*(ptr)) __old = (old); \ - __typeof__(*(ptr)) __new = (new); \ - __typeof__(*(ptr)) __ret; \ - register unsigned int __rc; \ - switch (size) { \ - case 4: \ - __asm__ __volatile__ ( \ - "0: lr.w %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc.w %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - "1:\n" \ - : "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \ - : "rJ" ((long)__old), "rJ" (__new) \ - : "memory"); \ - break; \ - case 8: \ - __asm__ __volatile__ ( \ - "0: lr.d %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc.d %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - "1:\n" \ - : "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \ - : "rJ" (__old), "rJ" (__new) \ - : "memory"); \ - break; \ - default: \ - BUILD_BUG(); \ - } \ - __ret; \ -}) -#define arch_cmpxchg_relaxed(ptr, o, n) \ -({ \ - __typeof__(*(ptr)) _o_ = (o); \ - __typeof__(*(ptr)) _n_ = (n); \ - (__typeof__(*(ptr))) __cmpxchg_relaxed((ptr), \ - _o_, _n_, sizeof(*(ptr))); \ -}) -#define __cmpxchg_acquire(ptr, old, new, size) \ +#define __arch_cmpxchg(lr_sfx, sc_sfx, prepend, append, r, p, co, o, n) \ ({ \ - __typeof__(ptr) __ptr = (ptr); \ - __typeof__(*(ptr)) __old = (old); \ - __typeof__(*(ptr)) __new = (new); \ - __typeof__(*(ptr)) __ret; \ register unsigned int __rc; \ - switch (size) { \ - case 4: \ - __asm__ __volatile__ ( \ - "0: lr.w %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc.w %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - RISCV_ACQUIRE_BARRIER \ - "1:\n" \ - : "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \ - : "rJ" ((long)__old), "rJ" (__new) \ - : "memory"); \ - break; \ - case 8: \ - __asm__ __volatile__ ( \ - "0: lr.d %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc.d %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - RISCV_ACQUIRE_BARRIER \ - "1:\n" \ - : "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \ - : "rJ" (__old), "rJ" (__new) \ - : "memory"); \ - break; \ - default: \ - BUILD_BUG(); \ - } \ - __ret; \ -}) - -#define arch_cmpxchg_acquire(ptr, o, n) \ -({ \ - __typeof__(*(ptr)) _o_ = (o); \ - __typeof__(*(ptr)) _n_ = (n); \ - (__typeof__(*(ptr))) __cmpxchg_acquire((ptr), \ - _o_, _n_, sizeof(*(ptr))); \ + \ + __asm__ __volatile__ ( \ + prepend \ + "0: lr" lr_sfx " %0, %2\n" \ + " bne %0, %z3, 1f\n" \ + " sc" sc_sfx " %1, %z4, %2\n" \ + " bnez %1, 0b\n" \ + append \ + "1:\n" \ + : "=&r" (r), "=&r" (__rc), "+A" (*(p)) \ + : "rJ" (co o), "rJ" (n) \ + : "memory"); \ }) -#define __cmpxchg_release(ptr, old, new, size) \ +#define _arch_cmpxchg(ptr, old, new, sc_sfx, prepend, append) \ ({ \ __typeof__(ptr) __ptr = (ptr); \ __typeof__(*(ptr)) __old = (old); \ __typeof__(*(ptr)) __new = (new); \ __typeof__(*(ptr)) __ret; \ - register unsigned int __rc; \ - switch (size) { \ + \ + switch (sizeof(*__ptr)) { \ case 4: \ - __asm__ __volatile__ ( \ - RISCV_RELEASE_BARRIER \ - "0: lr.w %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc.w %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - "1:\n" \ - : "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \ - : "rJ" ((long)__old), "rJ" (__new) \ - : "memory"); \ + __arch_cmpxchg(".w", ".w" sc_sfx, prepend, append, \ + __ret, __ptr, (long), __old, __new); \ break; \ case 8: \ - __asm__ __volatile__ ( \ - RISCV_RELEASE_BARRIER \ - "0: lr.d %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc.d %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - "1:\n" \ - : "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \ - : "rJ" (__old), "rJ" (__new) \ - : "memory"); \ + __arch_cmpxchg(".d", ".d" sc_sfx, prepend, append, \ + __ret, __ptr, /**/, __old, __new); \ break; \ default: \ BUILD_BUG(); \ @@ -199,62 +112,20 @@ __ret; \ }) -#define arch_cmpxchg_release(ptr, o, n) \ -({ \ - __typeof__(*(ptr)) _o_ = (o); \ - __typeof__(*(ptr)) _n_ = (n); \ - (__typeof__(*(ptr))) __cmpxchg_release((ptr), \ - _o_, _n_, sizeof(*(ptr))); \ -}) +#define arch_cmpxchg_relaxed(ptr, o, n) \ + _arch_cmpxchg((ptr), (o), (n), "", "", "") -#define __cmpxchg(ptr, old, new, size) \ -({ \ - __typeof__(ptr) __ptr = (ptr); \ - __typeof__(*(ptr)) __old = (old); \ - __typeof__(*(ptr)) __new = (new); \ - __typeof__(*(ptr)) __ret; \ - register unsigned int __rc; \ - switch (size) { \ - case 4: \ - __asm__ __volatile__ ( \ - "0: lr.w %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc.w.rl %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - " fence rw, rw\n" \ - "1:\n" \ - : "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \ - : "rJ" ((long)__old), "rJ" (__new) \ - : "memory"); \ - break; \ - case 8: \ - __asm__ __volatile__ ( \ - "0: lr.d %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc.d.rl %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - " fence rw, rw\n" \ - "1:\n" \ - : "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \ - : "rJ" (__old), "rJ" (__new) \ - : "memory"); \ - break; \ - default: \ - BUILD_BUG(); \ - } \ - __ret; \ -}) +#define arch_cmpxchg_acquire(ptr, o, n) \ + _arch_cmpxchg((ptr), (o), (n), "", "", RISCV_ACQUIRE_BARRIER) + +#define arch_cmpxchg_release(ptr, o, n) \ + _arch_cmpxchg((ptr), (o), (n), "", RISCV_RELEASE_BARRIER, "") #define arch_cmpxchg(ptr, o, n) \ -({ \ - __typeof__(*(ptr)) _o_ = (o); \ - __typeof__(*(ptr)) _n_ = (n); \ - (__typeof__(*(ptr))) __cmpxchg((ptr), \ - _o_, _n_, sizeof(*(ptr))); \ -}) + _arch_cmpxchg((ptr), (o), (n), ".rl", "", " fence rw, rw\n") #define arch_cmpxchg_local(ptr, o, n) \ - (__cmpxchg_relaxed((ptr), (o), (n), sizeof(*(ptr)))) + arch_cmpxchg_relaxed((ptr), (o), (n)) #define arch_cmpxchg64(ptr, o, n) \ ({ \ From patchwork Fri Aug 4 08:48:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leonardo Bras X-Patchwork-Id: 13341526 X-Patchwork-Delegate: palmer@dabbelt.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C9B07C04A94 for ; Fri, 4 Aug 2023 08:49:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=VfsN9GKoSqwjoxuQU66Q7/OyJSWiB1T4NgFHlghaRXc=; b=ERtKzP94suljuo Djypjq37EPBGIAnYzDuSgGNkSe0RzbCoQev40VoaKzyWBLmVYj3kIu8WY2HGYW/JLjhfYh5J4oVJ9 62Dh1Pdn7KleXbr3ulZ/7jfw2XiZmXNzGVqe66U2juVfU9LdZ+L2PbyzvyaikCB6YFpqTNP1C1Jg3 1N2lh+YZPnhZkS1nkJzmzuzmyf/4JXubWEPXgtEeh8fIq4VkufPVuM7E5h5cB6hoXfXjDIx/QmZWs wocVAxXDGQeKKfbKY3kD84UqQyGvFQaZHaGyvkuauyEeEaDLn5yXMQtyqlS8vTEtqr+t6pFpLPbgz tBchk0znnUV75qVg+zGQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qRqV9-00Bvgb-1M; Fri, 04 Aug 2023 08:49:39 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qRqV6-00Bvf7-1G for linux-riscv@lists.infradead.org; Fri, 04 Aug 2023 08:49:38 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1691138975; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UzQb2VuQCXwMdF/Rvu48wxhx8ntZSb1KliAOBLDDWgc=; b=AVWXNl5MYBvMcpi688XwtP7ZMgLhRG10S9H9dY5NqzzeD3tAuHju0mGWBjMx6L2m6MLQAV 94M6dGjbj3VQRxeKUwO/Dxx0811w8o8Vb2RHvCn/uUVRvZDHc4PZxojiSe7GJeE+lZyYDs oIM2gPNvflYb9g1qZ2aykUS0BTpjUZY= Received: from mail-oa1-f71.google.com (mail-oa1-f71.google.com [209.85.160.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-688-yZG3vT0jMSq1U8Citw9ABw-1; Fri, 04 Aug 2023 04:49:34 -0400 X-MC-Unique: yZG3vT0jMSq1U8Citw9ABw-1 Received: by mail-oa1-f71.google.com with SMTP id 586e51a60fabf-1bb91fb58f2so2640150fac.2 for ; Fri, 04 Aug 2023 01:49:34 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691138973; x=1691743773; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UzQb2VuQCXwMdF/Rvu48wxhx8ntZSb1KliAOBLDDWgc=; b=XTF3g1jxbqRnNJaZEDMMj9I6/VEUwRzyM7lAYa8Z3UL2oY/QJyDp4QzHwkIV1JQOxi JzlyaTNqadW1Di8c4kR3SaqVYnPV5egodJh2lrQLrXlyOEu6h9Sd2gETJckhEO4rKq+f NKYC/l6wxnbvYp8pOjdlibHS5WGlx//tThhmT9nJyy/6+Lxw0UEl7DXuW0yZ++YZczFO cqQbLihGPzwNLxNQIJ7oc1Yf5Tv/TPvb5ZfxN6bCykOyPNFme3eIBbqmT3BNatFVpO2a jz1g0/h+sYdLxoaCKIsg8IHG5CTuLHig5epQ1uMJ3zGulEHMkRu67cO2epUAAKENlRd5 1bdw== X-Gm-Message-State: AOJu0YwjTMIbDYFPtD9AH/KagkOsHeLso1bj0EAhq8N00mYtgGOzyCxe mTLrT1ndG6oUScl0jNbdpYcuDO6Q8AlEUF+PUHG6vkz+D734jiJqFNP5NKOKKhHl224zYbSDcwC A17BEbz4qMbjzERaPwOougaUuWgvC X-Received: by 2002:a05:6870:9693:b0:1bf:42b9:907f with SMTP id o19-20020a056870969300b001bf42b9907fmr1292585oaq.20.1691138973310; Fri, 04 Aug 2023 01:49:33 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHo7vyNoP4vF9peGfFA/17nq0Yzj8bINsHDzCtyGDvVS4NPtZ3F+JVXZP0nVXXJYhxCYez1Dw== X-Received: by 2002:a05:6870:9693:b0:1bf:42b9:907f with SMTP id o19-20020a056870969300b001bf42b9907fmr1292569oaq.20.1691138973053; Fri, 04 Aug 2023 01:49:33 -0700 (PDT) Received: from localhost.localdomain ([2804:1b3:a801:d380:694f:4f52:764c:4b7f]) by smtp.gmail.com with ESMTPSA id f8-20020a4ab008000000b0055516447257sm685679oon.29.2023.08.04.01.49.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 04 Aug 2023 01:49:32 -0700 (PDT) From: Leonardo Bras To: Will Deacon , Peter Zijlstra , Boqun Feng , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Leonardo Bras , Andrea Parri , Geert Uytterhoeven , Andrzej Hajda , Palmer Dabbelt , Guo Ren Cc: linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org Subject: [RFC PATCH v3 3/5] riscv/atomic.h : Deduplicate arch_atomic.* Date: Fri, 4 Aug 2023 05:48:56 -0300 Message-ID: <20230804084900.1135660-5-leobras@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230804084900.1135660-2-leobras@redhat.com> References: <20230804084900.1135660-2-leobras@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230804_014936_499068_27A00815 X-CRM114-Status: GOOD ( 11.50 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Some functions use mostly the same asm for 32-bit and 64-bit versions. Make a macro that is generic enough and avoid code duplication. (This did not cause any change in generated asm) Signed-off-by: Leonardo Bras Reviewed-by: Guo Ren Reviewed-by: Andrea Parri --- arch/riscv/include/asm/atomic.h | 164 +++++++++++++++----------------- 1 file changed, 76 insertions(+), 88 deletions(-) diff --git a/arch/riscv/include/asm/atomic.h b/arch/riscv/include/asm/atomic.h index f5dfef6c2153..80cca7ac16fd 100644 --- a/arch/riscv/include/asm/atomic.h +++ b/arch/riscv/include/asm/atomic.h @@ -196,22 +196,28 @@ ATOMIC_OPS(xor, xor, i) #undef ATOMIC_FETCH_OP #undef ATOMIC_OP_RETURN +#define _arch_atomic_fetch_add_unless(_prev, _rc, counter, _a, _u, sfx) \ +({ \ + __asm__ __volatile__ ( \ + "0: lr." sfx " %[p], %[c]\n" \ + " beq %[p], %[u], 1f\n" \ + " add %[rc], %[p], %[a]\n" \ + " sc." sfx ".rl %[rc], %[rc], %[c]\n" \ + " bnez %[rc], 0b\n" \ + " fence rw, rw\n" \ + "1:\n" \ + : [p]"=&r" (_prev), [rc]"=&r" (_rc), [c]"+A" (counter) \ + : [a]"r" (_a), [u]"r" (_u) \ + : "memory"); \ +}) + /* This is required to provide a full barrier on success. */ static __always_inline int arch_atomic_fetch_add_unless(atomic_t *v, int a, int u) { int prev, rc; - __asm__ __volatile__ ( - "0: lr.w %[p], %[c]\n" - " beq %[p], %[u], 1f\n" - " add %[rc], %[p], %[a]\n" - " sc.w.rl %[rc], %[rc], %[c]\n" - " bnez %[rc], 0b\n" - " fence rw, rw\n" - "1:\n" - : [p]"=&r" (prev), [rc]"=&r" (rc), [c]"+A" (v->counter) - : [a]"r" (a), [u]"r" (u) - : "memory"); + _arch_atomic_fetch_add_unless(prev, rc, v->counter, a, u, "w"); + return prev; } #define arch_atomic_fetch_add_unless arch_atomic_fetch_add_unless @@ -222,77 +228,86 @@ static __always_inline s64 arch_atomic64_fetch_add_unless(atomic64_t *v, s64 a, s64 prev; long rc; - __asm__ __volatile__ ( - "0: lr.d %[p], %[c]\n" - " beq %[p], %[u], 1f\n" - " add %[rc], %[p], %[a]\n" - " sc.d.rl %[rc], %[rc], %[c]\n" - " bnez %[rc], 0b\n" - " fence rw, rw\n" - "1:\n" - : [p]"=&r" (prev), [rc]"=&r" (rc), [c]"+A" (v->counter) - : [a]"r" (a), [u]"r" (u) - : "memory"); + _arch_atomic_fetch_add_unless(prev, rc, v->counter, a, u, "d"); + return prev; } #define arch_atomic64_fetch_add_unless arch_atomic64_fetch_add_unless #endif +#define _arch_atomic_inc_unless_negative(_prev, _rc, counter, sfx) \ +({ \ + __asm__ __volatile__ ( \ + "0: lr." sfx " %[p], %[c]\n" \ + " bltz %[p], 1f\n" \ + " addi %[rc], %[p], 1\n" \ + " sc." sfx ".rl %[rc], %[rc], %[c]\n" \ + " bnez %[rc], 0b\n" \ + " fence rw, rw\n" \ + "1:\n" \ + : [p]"=&r" (_prev), [rc]"=&r" (_rc), [c]"+A" (counter) \ + : \ + : "memory"); \ +}) + static __always_inline bool arch_atomic_inc_unless_negative(atomic_t *v) { int prev, rc; - __asm__ __volatile__ ( - "0: lr.w %[p], %[c]\n" - " bltz %[p], 1f\n" - " addi %[rc], %[p], 1\n" - " sc.w.rl %[rc], %[rc], %[c]\n" - " bnez %[rc], 0b\n" - " fence rw, rw\n" - "1:\n" - : [p]"=&r" (prev), [rc]"=&r" (rc), [c]"+A" (v->counter) - : - : "memory"); + _arch_atomic_inc_unless_negative(prev, rc, v->counter, "w"); + return !(prev < 0); } #define arch_atomic_inc_unless_negative arch_atomic_inc_unless_negative +#define _arch_atomic_dec_unless_positive(_prev, _rc, counter, sfx) \ +({ \ + __asm__ __volatile__ ( \ + "0: lr." sfx " %[p], %[c]\n" \ + " bgtz %[p], 1f\n" \ + " addi %[rc], %[p], -1\n" \ + " sc." sfx ".rl %[rc], %[rc], %[c]\n" \ + " bnez %[rc], 0b\n" \ + " fence rw, rw\n" \ + "1:\n" \ + : [p]"=&r" (_prev), [rc]"=&r" (_rc), [c]"+A" (counter) \ + : \ + : "memory"); \ +}) + static __always_inline bool arch_atomic_dec_unless_positive(atomic_t *v) { int prev, rc; - __asm__ __volatile__ ( - "0: lr.w %[p], %[c]\n" - " bgtz %[p], 1f\n" - " addi %[rc], %[p], -1\n" - " sc.w.rl %[rc], %[rc], %[c]\n" - " bnez %[rc], 0b\n" - " fence rw, rw\n" - "1:\n" - : [p]"=&r" (prev), [rc]"=&r" (rc), [c]"+A" (v->counter) - : - : "memory"); + _arch_atomic_dec_unless_positive(prev, rc, v->counter, "w"); + return !(prev > 0); } #define arch_atomic_dec_unless_positive arch_atomic_dec_unless_positive +#define _arch_atomic_dec_if_positive(_prev, _rc, counter, sfx) \ +({ \ + __asm__ __volatile__ ( \ + "0: lr." sfx " %[p], %[c]\n" \ + " addi %[rc], %[p], -1\n" \ + " bltz %[rc], 1f\n" \ + " sc." sfx ".rl %[rc], %[rc], %[c]\n" \ + " bnez %[rc], 0b\n" \ + " fence rw, rw\n" \ + "1:\n" \ + : [p]"=&r" (_prev), [rc]"=&r" (_rc), [c]"+A" (counter) \ + : \ + : "memory"); \ +}) + static __always_inline int arch_atomic_dec_if_positive(atomic_t *v) { int prev, rc; - __asm__ __volatile__ ( - "0: lr.w %[p], %[c]\n" - " addi %[rc], %[p], -1\n" - " bltz %[rc], 1f\n" - " sc.w.rl %[rc], %[rc], %[c]\n" - " bnez %[rc], 0b\n" - " fence rw, rw\n" - "1:\n" - : [p]"=&r" (prev), [rc]"=&r" (rc), [c]"+A" (v->counter) - : - : "memory"); + _arch_atomic_dec_if_positive(prev, rc, v->counter, "w"); + return prev - 1; } @@ -304,17 +319,8 @@ static __always_inline bool arch_atomic64_inc_unless_negative(atomic64_t *v) s64 prev; long rc; - __asm__ __volatile__ ( - "0: lr.d %[p], %[c]\n" - " bltz %[p], 1f\n" - " addi %[rc], %[p], 1\n" - " sc.d.rl %[rc], %[rc], %[c]\n" - " bnez %[rc], 0b\n" - " fence rw, rw\n" - "1:\n" - : [p]"=&r" (prev), [rc]"=&r" (rc), [c]"+A" (v->counter) - : - : "memory"); + _arch_atomic_inc_unless_negative(prev, rc, v->counter, "d"); + return !(prev < 0); } @@ -325,17 +331,8 @@ static __always_inline bool arch_atomic64_dec_unless_positive(atomic64_t *v) s64 prev; long rc; - __asm__ __volatile__ ( - "0: lr.d %[p], %[c]\n" - " bgtz %[p], 1f\n" - " addi %[rc], %[p], -1\n" - " sc.d.rl %[rc], %[rc], %[c]\n" - " bnez %[rc], 0b\n" - " fence rw, rw\n" - "1:\n" - : [p]"=&r" (prev), [rc]"=&r" (rc), [c]"+A" (v->counter) - : - : "memory"); + _arch_atomic_dec_unless_positive(prev, rc, v->counter, "d"); + return !(prev > 0); } @@ -346,17 +343,8 @@ static __always_inline s64 arch_atomic64_dec_if_positive(atomic64_t *v) s64 prev; long rc; - __asm__ __volatile__ ( - "0: lr.d %[p], %[c]\n" - " addi %[rc], %[p], -1\n" - " bltz %[rc], 1f\n" - " sc.d.rl %[rc], %[rc], %[c]\n" - " bnez %[rc], 0b\n" - " fence rw, rw\n" - "1:\n" - : [p]"=&r" (prev), [rc]"=&r" (rc), [c]"+A" (v->counter) - : - : "memory"); + _arch_atomic_dec_if_positive(prev, rc, v->counter, "d"); + return prev - 1; } From patchwork Fri Aug 4 08:48:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leonardo Bras X-Patchwork-Id: 13341527 X-Patchwork-Delegate: palmer@dabbelt.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 24D12C001DF for ; Fri, 4 Aug 2023 08:49:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=5f0NRcIvO9cFDtYq2+tpB5S3DTN+BSbNErJRm8GV/bQ=; b=U0gOfQfV56x68k frIdbqJNhpdwlCf9rMQo6SflQcNHJUYT64CqU22fKUZ6jm0lFHfzvBjkFQMTvkjGmFDVhBBGmsqt9 4LwEDvgsARDRhWZpzUPDxPSnvSDp76wlEGZBfS5++AysGX7i/zRJvrHImq8sAVPAMKbjohl2KCYRl 7+ChUcz4HFnEYFumz9srKsep068rkNptotBPPZMPjN5lvvCLoSiKRMXhpn+OODXF+pYizSctUeBbK BRAqN5hFbRbBbSze90mdov5/GRxv6UyMrBNGGwtdMK0wgc4G0R6KRlArec2RckfbfnLrBaRRZnBEw iQrzrvludnVC4MefwbRw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qRqVE-00Bviz-0M; Fri, 04 Aug 2023 08:49:44 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qRqVA-00Bvh0-1u for linux-riscv@lists.infradead.org; Fri, 04 Aug 2023 08:49:41 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1691138979; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7qr15U85Ji6JaTSGdRUgCJIrXJMY4cTkIs1Ye7aZqqk=; b=EwvGQcVx1c2et5+L+Nok74GpYdM+zTLoPdwO8zL9Dmd20SLnP/Snd91HgMBizIrPorSm0i bsX8NiSJ5/LYU4nLzjej+EKGHhADiGajXIw7H5f1JCyvZcGvBMD9E8ws9l1E4wsNmV6w5e 11fRP4TAuHSJR1uIJ9j1GDnO+SfroCk= Received: from mail-oo1-f71.google.com (mail-oo1-f71.google.com [209.85.161.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-148-fn7ZCuLLO_-BdRYt2790OQ-1; Fri, 04 Aug 2023 04:49:38 -0400 X-MC-Unique: fn7ZCuLLO_-BdRYt2790OQ-1 Received: by mail-oo1-f71.google.com with SMTP id 006d021491bc7-56d2636c132so2523331eaf.3 for ; Fri, 04 Aug 2023 01:49:38 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691138977; x=1691743777; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7qr15U85Ji6JaTSGdRUgCJIrXJMY4cTkIs1Ye7aZqqk=; b=CoKpvCEYKH7yMHgjyd4Y6AJr+jQH1c+YpGRUgOxC1D9CebHd5nD1tiO9ET8ixKCGzU NjmY2hwxzpwJGMU/WsCmSZFWZbxtm340ri+gqNgaYjTw/qJD33Yhiy4/bVQtyWtxWObm AM2nvbq66KCPJ/bvswSs5cGN2+mIklB/Ezyz2Uh9q6rKh0y0VG055sW9dlG/YpMtoFLD JodX4WczpFOH0k5wIkH4qTe69lZwibf4vXuXiAzrVIlFc85LBy9A9DFrc69SpPsK4irx 7WJVnEGbA9ohnOEoca23SemonSXWRw8F3+25MgzEEYW6Hso3b9pZLV3YZ5Cj2OYpCM/q 8S7w== X-Gm-Message-State: AOJu0YzkQhVa4ZV0LJyeI0rcIUyptdX5XhrM+UUrHuLATZMgwCIj9rNa YBhD9qiZFaL1hdrjZroWZQaytvn70Mh32793yQ1JhF7xwyBIsb1sSBtQ9+m5RJSGyMhuF7q0EzS FI0KoeO6N/ijODQkxSNyR/kKS1y1D X-Received: by 2002:a4a:d2dc:0:b0:56c:43cd:7267 with SMTP id j28-20020a4ad2dc000000b0056c43cd7267mr1155026oos.8.1691138977547; Fri, 04 Aug 2023 01:49:37 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFGpSmCpCsEW2JBmJiIqK4JHv1sEJQk4u7ARhdPhe1GUzhItUEcAzOrA7h+bfUyOp/7X0uLgw== X-Received: by 2002:a4a:d2dc:0:b0:56c:43cd:7267 with SMTP id j28-20020a4ad2dc000000b0056c43cd7267mr1155011oos.8.1691138977273; Fri, 04 Aug 2023 01:49:37 -0700 (PDT) Received: from localhost.localdomain ([2804:1b3:a801:d380:694f:4f52:764c:4b7f]) by smtp.gmail.com with ESMTPSA id f8-20020a4ab008000000b0055516447257sm685679oon.29.2023.08.04.01.49.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 04 Aug 2023 01:49:36 -0700 (PDT) From: Leonardo Bras To: Will Deacon , Peter Zijlstra , Boqun Feng , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Leonardo Bras , Andrea Parri , Geert Uytterhoeven , Andrzej Hajda , Palmer Dabbelt , Guo Ren Cc: linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org Subject: [RFC PATCH v3 4/5] riscv/cmpxchg: Implement cmpxchg for variables of size 1 and 2 Date: Fri, 4 Aug 2023 05:48:57 -0300 Message-ID: <20230804084900.1135660-6-leobras@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230804084900.1135660-2-leobras@redhat.com> References: <20230804084900.1135660-2-leobras@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230804_014940_701396_CAB7E9BF X-CRM114-Status: GOOD ( 12.67 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org cmpxchg for variables of size 1-byte and 2-bytes is not yet available for riscv, even though its present in other architectures such as arm64 and x86. This could lead to not being able to implement some locking mechanisms or requiring some rework to make it work properly. Implement 1-byte and 2-bytes cmpxchg in order to achieve parity with other architectures. Signed-off-by: Leonardo Bras --- arch/riscv/include/asm/cmpxchg.h | 35 ++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index 5a07646fae65..dfb433ac544f 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -72,6 +72,36 @@ * indicated by comparing RETURN with OLD. */ +#define __arch_cmpxchg_mask(sc_sfx, prepend, append, r, p, o, n) \ +({ \ + /* Depends on 2-byte variables being 2-byte aligned */ \ + ulong __s = ((ulong)(p) & 0x3) * BITS_PER_BYTE; \ + ulong __mask = GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \ + << __s; \ + ulong __newx = (ulong)(n) << __s; \ + ulong __oldx = (ulong)(o) << __s; \ + ulong __retx; \ + register unsigned int __rc; \ + \ + __asm__ __volatile__ ( \ + prepend \ + "0: lr.w %0, %2\n" \ + " and %0, %0, %z5\n" \ + " bne %0, %z3, 1f\n" \ + " and %1, %0, %z6\n" \ + " or %1, %1, %z4\n" \ + " sc.w" sc_sfx " %1, %1, %2\n" \ + " bnez %1, 0b\n" \ + append \ + "1:\n" \ + : "=&r" (__retx), "=&r" (__rc), "+A" (*(p)) \ + : "rJ" ((long)__oldx), "rJ" (__newx), \ + "rJ" (__mask), "rJ" (~__mask) \ + : "memory"); \ + \ + r = (__typeof__(*(p)))(__retx >> __s); \ +}) + #define __arch_cmpxchg(lr_sfx, sc_sfx, prepend, append, r, p, co, o, n) \ ({ \ @@ -98,6 +128,11 @@ __typeof__(*(ptr)) __ret; \ \ switch (sizeof(*__ptr)) { \ + case 1: \ + case 2: \ + __arch_cmpxchg_mask(sc_sfx, prepend, append, \ + __ret, __ptr, __old, __new); \ + break; \ case 4: \ __arch_cmpxchg(".w", ".w" sc_sfx, prepend, append, \ __ret, __ptr, (long), __old, __new); \ From patchwork Fri Aug 4 08:48:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leonardo Bras X-Patchwork-Id: 13341528 X-Patchwork-Delegate: palmer@dabbelt.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1F52CC001DB for ; Fri, 4 Aug 2023 08:49:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=i9XMeLI2ykkzrhRIvvi5SJ6/VvA3L/+7v/PL3O0YTi4=; b=U/h/AFjTFoc1vQ V5dOVba89UMk6phiZ80LOUcvyYHo83z3dkT6eZES5MaLQ1Soi6WnlERU5R+qhnjaRPFkLeaD+8ojo QVHLCdbPbsfFw0OJ6NCqluhSG9PMiRzTtLZvI5FlCYvSLwqdqEkS9hEpw6YhQZU2d7HFcckEcNbN3 DonqGUJk2pq6EIuhxT5z6MvuLoPTcN9BAhi4aEXCJn8XNEiDHl4Dsr7e3RH/0CL2U57+Xfp81kc6b 2gcMWsTe/3m1KND+KEmNldYiqoXHYCnOc3fFaDRATxnlCW/ohkW9D9YaMrtIOG+dAjdFqNnBwyXUc h1/+hsrF3QF+HjsBRLMA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qRqVG-00BvkX-2J; Fri, 04 Aug 2023 08:49:46 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qRqVE-00Bvit-1d for linux-riscv@lists.infradead.org; Fri, 04 Aug 2023 08:49:45 +0000 Received: from mail-oo1-f69.google.com (mail-oo1-f69.google.com [209.85.161.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-221-FA4I6SfHPDmg4IvlqGQW0A-1; Fri, 04 Aug 2023 04:49:42 -0400 X-MC-Unique: FA4I6SfHPDmg4IvlqGQW0A-1 Received: by mail-oo1-f69.google.com with SMTP id 006d021491bc7-56c85f719efso2605003eaf.0 for ; Fri, 04 Aug 2023 01:49:42 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691138981; x=1691743781; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3amwd/lpP8rvZ9IhpcoisHZeefZgWLzf3BsTgVcPLgw=; b=hji2QMJkOV7laHoh44Q8desPlTbcm5G9vUjO8GnH0YtleHuDEW52fhT48lUUeEuhT/ aDV5mVeSJi6VnF80kzg3ByDxtGzAnZYearK+jWCX8e+3goEEX4Wh17ACFINPsL2mPtbG VXd1IoIRVScdG3cy4GJEZRifHg+koNqSjrs4+nqw3ZPEmXYtp3alDuXiTpu4XxQ6AZtF CFRTq3xyE90A8erefX+oQJ0MURc5UuyVvWQDdBocH1uonsFt2hcvRhnzGXZlzkvzaCIL 6DQNBfi4XKlz0a3FUvD8s+YVgL/KODrplstAoMD3UU7OiBc0DBXP7VVPDR41QdXo2x7h dIdQ== X-Gm-Message-State: AOJu0YwyYBEjdhtrBXg3rN4DV/3O/IRQBX5yx367uSnKt11fuXof1czd SJ109QVfC5cumkCAhllCsZW7rO+9q4qY6O3Hj3Hxtcxy0ZFYowfgsff353B0AqA3DrIX2Phn3PF 1hIePo2V/nhpS6OFP+KlrTbd8Poza X-Received: by 2002:a4a:6519:0:b0:56c:e856:8b2c with SMTP id y25-20020a4a6519000000b0056ce8568b2cmr1003961ooc.9.1691138981681; Fri, 04 Aug 2023 01:49:41 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFvsWBOYEPFQASeYA1dLIyHiNIkhde3nNiq2XUol3devppgxjgdj9GmPoeRlh/oNYHjlwpP4w== X-Received: by 2002:a4a:6519:0:b0:56c:e856:8b2c with SMTP id y25-20020a4a6519000000b0056ce8568b2cmr1003954ooc.9.1691138981435; Fri, 04 Aug 2023 01:49:41 -0700 (PDT) Received: from localhost.localdomain ([2804:1b3:a801:d380:694f:4f52:764c:4b7f]) by smtp.gmail.com with ESMTPSA id f8-20020a4ab008000000b0055516447257sm685679oon.29.2023.08.04.01.49.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 04 Aug 2023 01:49:41 -0700 (PDT) From: Leonardo Bras To: Will Deacon , Peter Zijlstra , Boqun Feng , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Leonardo Bras , Andrea Parri , Geert Uytterhoeven , Andrzej Hajda , Palmer Dabbelt , Guo Ren Cc: linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org Subject: [RFC PATCH v3 5/5] riscv/cmpxchg: Implement xchg for variables of size 1 and 2 Date: Fri, 4 Aug 2023 05:48:58 -0300 Message-ID: <20230804084900.1135660-7-leobras@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230804084900.1135660-2-leobras@redhat.com> References: <20230804084900.1135660-2-leobras@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230804_014944_701101_ACED8425 X-CRM114-Status: GOOD ( 12.11 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org xchg for variables of size 1-byte and 2-bytes is not yet available for riscv, even though its present in other architectures such as arm64 and x86. This could lead to not being able to implement some locking mechanisms or requiring some rework to make it work properly. Implement 1-byte and 2-bytes xchg in order to achieve parity with other architectures. Signed-off-by: Leonardo Bras --- arch/riscv/include/asm/cmpxchg.h | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index dfb433ac544f..943f094375c7 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -11,6 +11,31 @@ #include #include +#define __arch_xchg_mask(prepend, append, r, p, n) \ +({ \ + /* Depends on 2-byte variables being 2-byte aligned */ \ + ulong __s = ((ulong)(p) & 0x3) * BITS_PER_BYTE; \ + ulong __mask = GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \ + << __s; \ + ulong __newx = (ulong)(n) << __s; \ + ulong __retx; \ + register unsigned int __rc; \ + \ + __asm__ __volatile__ ( \ + prepend \ + "0: lr.w %0, %2\n" \ + " and %1, %0, %z4\n" \ + " or %1, %1, %z3\n" \ + " sc.w %1, %1, %2\n" \ + " bnez %1, 0b\n" \ + append \ + : "=&r" (__retx), "=&r" (__rc), "+A" (*(p)) \ + : "rJ" (__newx), "rJ" (~__mask) \ + : "memory"); \ + \ + r = (__typeof__(*(p)))((__retx & __mask) >> __s); \ +}) + #define __arch_xchg(sfx, prepend, append, r, p, n) \ ({ \ __asm__ __volatile__ ( \ @@ -27,7 +52,13 @@ __typeof__(*(ptr)) __new = (new); \ __typeof__(*(ptr)) __ret; \ __typeof__(ptr) __ptr = (ptr); \ + \ switch (sizeof(*__ptr)) { \ + case 1: \ + case 2: \ + __arch_xchg_mask(prepend, append, \ + __ret, __ptr, __new); \ + break; \ case 4: \ __arch_xchg(".w" sfx, prepend, append, \ __ret, __ptr, __new); \