From patchwork Wed Jan 3 16:31:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leonardo Bras X-Patchwork-Id: 13510239 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7CB0DC3DA6E for ; Wed, 3 Jan 2024 16:32:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Rv8aH+0fIR20X2LmMQA2kHf1D1LWBNNdKO0RrIOlSP8=; b=Z7rmd3J5Nnn5Xk GKjjInOSkQwhXfdjqdQ6Xkn37+xty2kHpctkF8BfMTaFDsQnf+0EETEo4VR5cHqDDu5xpsVH7R27q 4/oTLaoYZ/l1OEMrF/aXUjD6wJAUUX/oLBTK1DYhaCTIN0LHeyEVoX+/viAUE0r2HHvg0lgmayeXx DDzk209pL5jAG1KynvJha6VMY/p/bhnySDKsvRGl7dVyJOeiXZJg51OGBIiJ1p1vsahZ3CM9iRrk3 R3YY0BFBr8+dv2cmB7LteqATUZqDY7YAxY2zCKMqs6izGw9uE9VtY4ENK4tOAE+Ypcq7JrR7b+z4U Sl+ywZufqxkCCDjLy+JA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rL4Ad-00BOxT-0F; Wed, 03 Jan 2024 16:32:43 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rL4Aa-00BOwP-2L for linux-riscv@lists.infradead.org; Wed, 03 Jan 2024 16:32:42 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1704299559; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gfK6+G/fxpQtUcPr0UJaWEhuvxeP6ij0m8scDlFf7oE=; b=EHSARiElo1V4B9VfLT/LsAkmKfrlaG5CctB+qaAJgrRQs6UzGH6EtlgepaXwTMqFkapxEk /wxGTXCuevPlklEt2GmVjnV0GqvfEC+1axaMYEOXr3qL6MXqTAGhIn7/r37FB040qoYA53 R64ySriYFuOlAPdX9XvB+yyQAcRQ1kY= Received: from mail-pl1-f197.google.com (mail-pl1-f197.google.com [209.85.214.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-549-N1etmD_xM-q8r4im84pyVw-1; Wed, 03 Jan 2024 11:32:38 -0500 X-MC-Unique: N1etmD_xM-q8r4im84pyVw-1 Received: by mail-pl1-f197.google.com with SMTP id d9443c01a7336-1d4b8ad631dso10696975ad.0 for ; Wed, 03 Jan 2024 08:32:38 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704299558; x=1704904358; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gfK6+G/fxpQtUcPr0UJaWEhuvxeP6ij0m8scDlFf7oE=; b=lU0S3UVcsxmcFN+noBKxwy0w7mUblB9hKGmLsNr0Q3ttN3tGgqK6Xy6ihvAxEMPn5t Yam5/tFW5HXvwP4BecJqqEyab+ewS360zllecPdMvN8Tfh5VhHtFr8pYZZItnNslP5P6 OsPG+Hfo9eLXbdLd+xgj/vfsbn8BoCojgp+s9kteMIU3m5pyD0Yw3xeWnF4ag+U9K9tk RzyjtgHRCwFXEmk6A0S/Rrd3v56+y7/QjS/Tg9+oXSxdLROqYb6mKVb/pM4T26mPZhAZ 2beJZd35nF5QMazHQ5evC2sC07Z+wdo9z1sLVCICWK0+YXXC0VQfaeCgm3QDPaZbeUcJ DlZA== X-Gm-Message-State: AOJu0YxEbc3mW7EEIp+XzL8HpVNbZjFrH0W/C/AFDZGRcI9WhSTBJKcv l+fLIcwKhCpcLVV1MbLU8JxWz8S9YqXuzcBPqVdTh5jMz4+EHuXKHY8ACTNF5ePwqvBHkrU0qrV Upkys613xlHMc/RO7qg6lACFHLbunZqtgJwLX X-Received: by 2002:a17:902:f7c9:b0:1d4:79b5:b26 with SMTP id h9-20020a170902f7c900b001d479b50b26mr6275119plw.78.1704299557738; Wed, 03 Jan 2024 08:32:37 -0800 (PST) X-Google-Smtp-Source: AGHT+IFTQgC8TXArNxt0zw+/Ybt1FA1l3Pis1O2gQ6NIBPnJnz+swVe6KE7mMi1vmxykVrwzFBJixQ== X-Received: by 2002:a17:902:f7c9:b0:1d4:79b5:b26 with SMTP id h9-20020a170902f7c900b001d479b50b26mr6275100plw.78.1704299557400; Wed, 03 Jan 2024 08:32:37 -0800 (PST) Received: from localhost.localdomain ([2804:431:c7ec:911:6911:ca60:846:eb46]) by smtp.gmail.com with ESMTPSA id e12-20020a170902b78c00b001cfca7b8ee7sm23930425pls.99.2024.01.03.08.32.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Jan 2024 08:32:36 -0800 (PST) From: Leonardo Bras To: Will Deacon , Peter Zijlstra , Boqun Feng , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Leonardo Bras , Guo Ren , Andrea Parri , Geert Uytterhoeven , Ingo Molnar , Andrzej Hajda Cc: linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org Subject: [PATCH v1 1/5] riscv/cmpxchg: Deduplicate xchg() asm functions Date: Wed, 3 Jan 2024 13:31:59 -0300 Message-ID: <20240103163203.72768-3-leobras@redhat.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240103163203.72768-2-leobras@redhat.com> References: <20240103163203.72768-2-leobras@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240103_083240_836658_F48DCE3F X-CRM114-Status: GOOD ( 12.16 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org In this header every xchg define (_relaxed, _acquire, _release, vanilla) contain it's own asm file, both for 4-byte variables an 8-byte variables, on a total of 8 versions of mostly the same asm. This is usually bad, as it means any change may be done in up to 8 different places. Unify those versions by creating a new define with enough parameters to generate any version of the previous 8. Then unify the result under a more general define, and simplify arch_xchg* generation. (This did not cause any change in generated asm) Signed-off-by: Leonardo Bras Reviewed-by: Guo Ren Reviewed-by: Andrea Parri Tested-by: Guo Ren --- arch/riscv/include/asm/cmpxchg.h | 138 ++++++------------------------- 1 file changed, 23 insertions(+), 115 deletions(-) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index 2f4726d3cfcc2..48478a8eecee7 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -11,140 +11,48 @@ #include #include -#define __xchg_relaxed(ptr, new, size) \ +#define __arch_xchg(sfx, prepend, append, r, p, n) \ ({ \ - __typeof__(ptr) __ptr = (ptr); \ - __typeof__(new) __new = (new); \ - __typeof__(*(ptr)) __ret; \ - switch (size) { \ - case 4: \ - __asm__ __volatile__ ( \ - " amoswap.w %0, %2, %1\n" \ - : "=r" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ - break; \ - case 8: \ - __asm__ __volatile__ ( \ - " amoswap.d %0, %2, %1\n" \ - : "=r" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ - break; \ - default: \ - BUILD_BUG(); \ - } \ - __ret; \ -}) - -#define arch_xchg_relaxed(ptr, x) \ -({ \ - __typeof__(*(ptr)) _x_ = (x); \ - (__typeof__(*(ptr))) __xchg_relaxed((ptr), \ - _x_, sizeof(*(ptr))); \ + __asm__ __volatile__ ( \ + prepend \ + " amoswap" sfx " %0, %2, %1\n" \ + append \ + : "=r" (r), "+A" (*(p)) \ + : "r" (n) \ + : "memory"); \ }) -#define __xchg_acquire(ptr, new, size) \ +#define _arch_xchg(ptr, new, sfx, prepend, append) \ ({ \ __typeof__(ptr) __ptr = (ptr); \ - __typeof__(new) __new = (new); \ - __typeof__(*(ptr)) __ret; \ - switch (size) { \ + __typeof__(*(__ptr)) __new = (new); \ + __typeof__(*(__ptr)) __ret; \ + switch (sizeof(*__ptr)) { \ case 4: \ - __asm__ __volatile__ ( \ - " amoswap.w %0, %2, %1\n" \ - RISCV_ACQUIRE_BARRIER \ - : "=r" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ + __arch_xchg(".w" sfx, prepend, append, \ + __ret, __ptr, __new); \ break; \ case 8: \ - __asm__ __volatile__ ( \ - " amoswap.d %0, %2, %1\n" \ - RISCV_ACQUIRE_BARRIER \ - : "=r" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ + __arch_xchg(".d" sfx, prepend, append, \ + __ret, __ptr, __new); \ break; \ default: \ BUILD_BUG(); \ } \ - __ret; \ + (__typeof__(*(__ptr)))__ret; \ }) -#define arch_xchg_acquire(ptr, x) \ -({ \ - __typeof__(*(ptr)) _x_ = (x); \ - (__typeof__(*(ptr))) __xchg_acquire((ptr), \ - _x_, sizeof(*(ptr))); \ -}) +#define arch_xchg_relaxed(ptr, x) \ + _arch_xchg(ptr, x, "", "", "") -#define __xchg_release(ptr, new, size) \ -({ \ - __typeof__(ptr) __ptr = (ptr); \ - __typeof__(new) __new = (new); \ - __typeof__(*(ptr)) __ret; \ - switch (size) { \ - case 4: \ - __asm__ __volatile__ ( \ - RISCV_RELEASE_BARRIER \ - " amoswap.w %0, %2, %1\n" \ - : "=r" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ - break; \ - case 8: \ - __asm__ __volatile__ ( \ - RISCV_RELEASE_BARRIER \ - " amoswap.d %0, %2, %1\n" \ - : "=r" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ - break; \ - default: \ - BUILD_BUG(); \ - } \ - __ret; \ -}) +#define arch_xchg_acquire(ptr, x) \ + _arch_xchg(ptr, x, "", "", RISCV_ACQUIRE_BARRIER) #define arch_xchg_release(ptr, x) \ -({ \ - __typeof__(*(ptr)) _x_ = (x); \ - (__typeof__(*(ptr))) __xchg_release((ptr), \ - _x_, sizeof(*(ptr))); \ -}) - -#define __arch_xchg(ptr, new, size) \ -({ \ - __typeof__(ptr) __ptr = (ptr); \ - __typeof__(new) __new = (new); \ - __typeof__(*(ptr)) __ret; \ - switch (size) { \ - case 4: \ - __asm__ __volatile__ ( \ - " amoswap.w.aqrl %0, %2, %1\n" \ - : "=r" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ - break; \ - case 8: \ - __asm__ __volatile__ ( \ - " amoswap.d.aqrl %0, %2, %1\n" \ - : "=r" (__ret), "+A" (*__ptr) \ - : "r" (__new) \ - : "memory"); \ - break; \ - default: \ - BUILD_BUG(); \ - } \ - __ret; \ -}) + _arch_xchg(ptr, x, "", RISCV_RELEASE_BARRIER, "") #define arch_xchg(ptr, x) \ -({ \ - __typeof__(*(ptr)) _x_ = (x); \ - (__typeof__(*(ptr))) __arch_xchg((ptr), _x_, sizeof(*(ptr))); \ -}) + _arch_xchg(ptr, x, ".aqrl", "", "") #define xchg32(ptr, x) \ ({ \ From patchwork Wed Jan 3 16:32:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leonardo Bras X-Patchwork-Id: 13510240 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 94229C3DA6E for ; Wed, 3 Jan 2024 16:33:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=jBf07YUN4IV4Ue0G0+n/YVnTTS0AHR4gwxHpICmu+BQ=; b=bwwQvdlicWRgic UB1HIbol0+sL2iNIQ7FXzx0q4wvXhIuC0kWEqVA+Vh0SKmSsZPh+cziFm7bqFzwrfXn/SjqNn+yHv eL2poh5I2SzxFyIBObrTHzSQdqcSmHje/yxzjLb7x3FDUSskY6f8NVZVoKo8P6yOkkG5qcMI49H6p 9fpuK1dAzVTvK7RbKeNXMkXoF5XSqzoIbwuGlHpFCfVPrV9e51nzsYekf+DDN8Sju9NTlFvS2DcFg sMHkivwQlvSAHQ2QhsP8xjaID819OjvmmtW5RMtuddXM1b8FOOdnN/ItTjB9Oh4pUF90zxRglHkWv 4PuE8+34TC5w293TGJQg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rL4An-00BP1r-0F; Wed, 03 Jan 2024 16:32:53 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rL4Ai-00BOzM-32 for linux-riscv@lists.infradead.org; Wed, 03 Jan 2024 16:32:51 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1704299568; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=n/sNmacMSvCkIR374aY0S9uvqtL96KdIcIOWwUR3yYE=; b=H++A9G2nJ6FhbVHODiukHWIm+hASZzhZ97zvRZp05LpZOWBVH5dLySbC+F7iBPhvl6oAku 1W9vbCskaZOsU7WL5VwvoZLSeG1l86mlMxryArsA+2RMAO7VZNkGj+fIPkjOUg7PuYkBc1 vtbjoYpXSozNn+OyGIDLhq5Emrx/iMA= Received: from mail-pg1-f199.google.com (mail-pg1-f199.google.com [209.85.215.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-118-xRY1EAFYMbeHZDDY4ChFeg-1; Wed, 03 Jan 2024 11:32:44 -0500 X-MC-Unique: xRY1EAFYMbeHZDDY4ChFeg-1 Received: by mail-pg1-f199.google.com with SMTP id 41be03b00d2f7-5c670f70a37so10293509a12.2 for ; Wed, 03 Jan 2024 08:32:44 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704299563; x=1704904363; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=n/sNmacMSvCkIR374aY0S9uvqtL96KdIcIOWwUR3yYE=; b=KxiIFYIJyaUYXFSQUUBYITzKNwVPWqL0g6n9yGSH0ZR9upIxg2yn25u9VSCShNNWHv GGfZSFYKRcfVjJl1Dmrt3BwHXVZ6G4gY10nroWi4urWutUaRzPxfeWJqWMNSZmxVfXrZ w2c2ijmriBvkeM+aUq6U0r+LSBu9WLulgJnjxegfstARfaurPugYYGiXNFHjIeAHhEq3 r9I8TcBnZbmSGMjaqfYLMzzrw8H1BWDa367XMFmgym8Arx7e056DbqLkyTBUk749+Pcr NaRnjPrZe1RE58woB0E+NDoOfb2f3C0mb1VdPaoh+AvPMF2WPEYhb6F4Ez3sxNO1Z4TS yZGg== X-Gm-Message-State: AOJu0YwErgQE+N/XvfFOXAmA1oBNog9udZDprRAtdHgTZ6rR9rbRMNv8 /rc8j82UED7zsrzvPJcYlPzsf/3eT7WWAV7v9Mgi3BD+uJvFIMKaUmnr8Nn53S0mmaG5a2Brh3H cMQ+Nn6YQ8NmZKKlGDdJdeubRu8D84AGt101t X-Received: by 2002:a17:902:64c2:b0:1d4:87c8:40f9 with SMTP id y2-20020a17090264c200b001d487c840f9mr10868108pli.84.1704299563370; Wed, 03 Jan 2024 08:32:43 -0800 (PST) X-Google-Smtp-Source: AGHT+IEk9KLzHdKrTHSS/KtZZppx+0qxHeyY2aaqvlMhT293khOuYzfSNdLjxjc3JpSL3I6Ppc3weQ== X-Received: by 2002:a17:902:64c2:b0:1d4:87c8:40f9 with SMTP id y2-20020a17090264c200b001d487c840f9mr10868097pli.84.1704299563019; Wed, 03 Jan 2024 08:32:43 -0800 (PST) Received: from localhost.localdomain ([2804:431:c7ec:911:6911:ca60:846:eb46]) by smtp.gmail.com with ESMTPSA id e12-20020a170902b78c00b001cfca7b8ee7sm23930425pls.99.2024.01.03.08.32.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Jan 2024 08:32:42 -0800 (PST) From: Leonardo Bras To: Will Deacon , Peter Zijlstra , Boqun Feng , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Leonardo Bras , Guo Ren , Andrea Parri , Geert Uytterhoeven , Ingo Molnar , Andrzej Hajda Cc: linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org Subject: [PATCH v1 2/5] riscv/cmpxchg: Deduplicate cmpxchg() asm and macros Date: Wed, 3 Jan 2024 13:32:00 -0300 Message-ID: <20240103163203.72768-4-leobras@redhat.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240103163203.72768-2-leobras@redhat.com> References: <20240103163203.72768-2-leobras@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240103_083249_068186_B0B0DC6C X-CRM114-Status: GOOD ( 12.46 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org In this header every cmpxchg define (_relaxed, _acquire, _release, vanilla) contain it's own asm file, both for 4-byte variables an 8-byte variables, on a total of 8 versions of mostly the same asm. This is usually bad, as it means any change may be done in up to 8 different places. Unify those versions by creating a new define with enough parameters to generate any version of the previous 8. Then unify the result under a more general define, and simplify arch_cmpxchg* generation (This did not cause any change in generated asm) Signed-off-by: Leonardo Bras Reviewed-by: Guo Ren Reviewed-by: Andrea Parri Tested-by: Guo Ren --- arch/riscv/include/asm/cmpxchg.h | 195 ++++++------------------------- 1 file changed, 33 insertions(+), 162 deletions(-) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index 48478a8eecee7..e3e0ac7ba061b 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -71,190 +71,61 @@ * store NEW in MEM. Return the initial value in MEM. Success is * indicated by comparing RETURN with OLD. */ -#define __cmpxchg_relaxed(ptr, old, new, size) \ -({ \ - __typeof__(ptr) __ptr = (ptr); \ - __typeof__(*(ptr)) __old = (old); \ - __typeof__(*(ptr)) __new = (new); \ - __typeof__(*(ptr)) __ret; \ - register unsigned int __rc; \ - switch (size) { \ - case 4: \ - __asm__ __volatile__ ( \ - "0: lr.w %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc.w %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - "1:\n" \ - : "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \ - : "rJ" ((long)__old), "rJ" (__new) \ - : "memory"); \ - break; \ - case 8: \ - __asm__ __volatile__ ( \ - "0: lr.d %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc.d %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - "1:\n" \ - : "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \ - : "rJ" (__old), "rJ" (__new) \ - : "memory"); \ - break; \ - default: \ - BUILD_BUG(); \ - } \ - __ret; \ -}) -#define arch_cmpxchg_relaxed(ptr, o, n) \ -({ \ - __typeof__(*(ptr)) _o_ = (o); \ - __typeof__(*(ptr)) _n_ = (n); \ - (__typeof__(*(ptr))) __cmpxchg_relaxed((ptr), \ - _o_, _n_, sizeof(*(ptr))); \ -}) -#define __cmpxchg_acquire(ptr, old, new, size) \ +#define __arch_cmpxchg(lr_sfx, sc_sfx, prepend, append, r, p, co, o, n) \ ({ \ - __typeof__(ptr) __ptr = (ptr); \ - __typeof__(*(ptr)) __old = (old); \ - __typeof__(*(ptr)) __new = (new); \ - __typeof__(*(ptr)) __ret; \ register unsigned int __rc; \ - switch (size) { \ - case 4: \ - __asm__ __volatile__ ( \ - "0: lr.w %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc.w %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - RISCV_ACQUIRE_BARRIER \ - "1:\n" \ - : "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \ - : "rJ" ((long)__old), "rJ" (__new) \ - : "memory"); \ - break; \ - case 8: \ - __asm__ __volatile__ ( \ - "0: lr.d %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc.d %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - RISCV_ACQUIRE_BARRIER \ - "1:\n" \ - : "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \ - : "rJ" (__old), "rJ" (__new) \ - : "memory"); \ - break; \ - default: \ - BUILD_BUG(); \ - } \ - __ret; \ -}) - -#define arch_cmpxchg_acquire(ptr, o, n) \ -({ \ - __typeof__(*(ptr)) _o_ = (o); \ - __typeof__(*(ptr)) _n_ = (n); \ - (__typeof__(*(ptr))) __cmpxchg_acquire((ptr), \ - _o_, _n_, sizeof(*(ptr))); \ + \ + __asm__ __volatile__ ( \ + prepend \ + "0: lr" lr_sfx " %0, %2\n" \ + " bne %0, %z3, 1f\n" \ + " sc" sc_sfx " %1, %z4, %2\n" \ + " bnez %1, 0b\n" \ + append \ + "1:\n" \ + : "=&r" (r), "=&r" (__rc), "+A" (*(p)) \ + : "rJ" (co o), "rJ" (n) \ + : "memory"); \ }) -#define __cmpxchg_release(ptr, old, new, size) \ +#define _arch_cmpxchg(ptr, old, new, sc_sfx, prepend, append) \ ({ \ __typeof__(ptr) __ptr = (ptr); \ - __typeof__(*(ptr)) __old = (old); \ - __typeof__(*(ptr)) __new = (new); \ - __typeof__(*(ptr)) __ret; \ - register unsigned int __rc; \ - switch (size) { \ + __typeof__(*(__ptr)) __old = (old); \ + __typeof__(*(__ptr)) __new = (new); \ + __typeof__(*(__ptr)) __ret; \ + \ + switch (sizeof(*__ptr)) { \ case 4: \ - __asm__ __volatile__ ( \ - RISCV_RELEASE_BARRIER \ - "0: lr.w %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc.w %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - "1:\n" \ - : "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \ - : "rJ" ((long)__old), "rJ" (__new) \ - : "memory"); \ + __arch_cmpxchg(".w", ".w" sc_sfx, prepend, append, \ + __ret, __ptr, (long), __old, __new); \ break; \ case 8: \ - __asm__ __volatile__ ( \ - RISCV_RELEASE_BARRIER \ - "0: lr.d %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc.d %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - "1:\n" \ - : "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \ - : "rJ" (__old), "rJ" (__new) \ - : "memory"); \ + __arch_cmpxchg(".d", ".d" sc_sfx, prepend, append, \ + __ret, __ptr, /**/, __old, __new); \ break; \ default: \ BUILD_BUG(); \ } \ - __ret; \ + (__typeof__(*(__ptr)))__ret; \ }) -#define arch_cmpxchg_release(ptr, o, n) \ -({ \ - __typeof__(*(ptr)) _o_ = (o); \ - __typeof__(*(ptr)) _n_ = (n); \ - (__typeof__(*(ptr))) __cmpxchg_release((ptr), \ - _o_, _n_, sizeof(*(ptr))); \ -}) +#define arch_cmpxchg_relaxed(ptr, o, n) \ + _arch_cmpxchg((ptr), (o), (n), "", "", "") -#define __cmpxchg(ptr, old, new, size) \ -({ \ - __typeof__(ptr) __ptr = (ptr); \ - __typeof__(*(ptr)) __old = (old); \ - __typeof__(*(ptr)) __new = (new); \ - __typeof__(*(ptr)) __ret; \ - register unsigned int __rc; \ - switch (size) { \ - case 4: \ - __asm__ __volatile__ ( \ - "0: lr.w %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc.w.rl %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - " fence rw, rw\n" \ - "1:\n" \ - : "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \ - : "rJ" ((long)__old), "rJ" (__new) \ - : "memory"); \ - break; \ - case 8: \ - __asm__ __volatile__ ( \ - "0: lr.d %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc.d.rl %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - " fence rw, rw\n" \ - "1:\n" \ - : "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \ - : "rJ" (__old), "rJ" (__new) \ - : "memory"); \ - break; \ - default: \ - BUILD_BUG(); \ - } \ - __ret; \ -}) +#define arch_cmpxchg_acquire(ptr, o, n) \ + _arch_cmpxchg((ptr), (o), (n), "", "", RISCV_ACQUIRE_BARRIER) + +#define arch_cmpxchg_release(ptr, o, n) \ + _arch_cmpxchg((ptr), (o), (n), "", RISCV_RELEASE_BARRIER, "") #define arch_cmpxchg(ptr, o, n) \ -({ \ - __typeof__(*(ptr)) _o_ = (o); \ - __typeof__(*(ptr)) _n_ = (n); \ - (__typeof__(*(ptr))) __cmpxchg((ptr), \ - _o_, _n_, sizeof(*(ptr))); \ -}) + _arch_cmpxchg((ptr), (o), (n), ".rl", "", " fence rw, rw\n") #define arch_cmpxchg_local(ptr, o, n) \ - (__cmpxchg_relaxed((ptr), (o), (n), sizeof(*(ptr)))) + arch_cmpxchg_relaxed((ptr), (o), (n)) #define arch_cmpxchg64(ptr, o, n) \ ({ \ From patchwork Wed Jan 3 16:32:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leonardo Bras X-Patchwork-Id: 13510241 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0B087C3DA6E for ; Wed, 3 Jan 2024 16:33:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=YCz8sVh82GnJYjB9IjtounbLeBocubPVfhAqmnZdCFY=; b=m7OVqjpQQ5lL2c nT+R7P7rLTHdMMX1NCrjgm0BfOaAf915ehWJCz8Bknm1E5dPpyB2nl58eoiEjJLMjAxSxHlEdj4CI tXT24wOE3iD+V1ZlJWhv1+eVNdhRqZmuErQ4wplng6KJd+xigD2TZ2bKVf2RAKnalj3k1e+8qFfXz O2AzXhyTYqMbfFWOvFnBlhR7MoJzz3SeNMP09/8f4lHxrjtiv7VJ+HqYEtzU4DDOOB2LYNXWrZsKx ITFp0mllwh3jDuBLTN3lbGQqFDCsLdKx9xXfpEqZVqNSmQdCObIDemH0ZaXmWtPjCGfmoKUjvMUa4 QihYYkQedPPQPmiXAv3A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rL4Ap-00BP33-2v; Wed, 03 Jan 2024 16:32:55 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rL4Am-00BP0v-0S for linux-riscv@lists.infradead.org; Wed, 03 Jan 2024 16:32:53 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1704299571; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DNxCEyFjygCY2k/IIyJiEePHcERbzG2CmUQITkDIUjg=; b=iVlCcCTfx907+T7ILMQ5ZLdBNP8eG3CvITj73P+yKRCbn/kaF5bKnc8sR5lUclXxGyJhaX xT5jJgI7R6KRVOcaZSaXkCTIYyFUSdjgAUfTIHSchwCSD87rwcU3APWCREPHM4wEzJKYz7 cHwASRZZwicUhPA2EmLD2Bf5NiRCP3U= Received: from mail-pl1-f200.google.com (mail-pl1-f200.google.com [209.85.214.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-622-A2XG3XHwM7iMXGgxxqVqrA-1; Wed, 03 Jan 2024 11:32:49 -0500 X-MC-Unique: A2XG3XHwM7iMXGgxxqVqrA-1 Received: by mail-pl1-f200.google.com with SMTP id d9443c01a7336-1d3f8ba12faso4805385ad.0 for ; Wed, 03 Jan 2024 08:32:49 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704299568; x=1704904368; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DNxCEyFjygCY2k/IIyJiEePHcERbzG2CmUQITkDIUjg=; b=bsUsYEY8MN0lrZdPVugL/en5isLWsIFVuE0NCFi1yMrVvPUlqelKlWyVabzQEyULpL CyFoXJCtYNcFAHiQIC1xDBnt9BLEWy//1Yu3zwvlmRkFI42+UBRX9ktDcJPmfj9qWiUN mefMfitNmegpboPV039/BXGWAZkv9TDMTHczafSt/Y+ujGssX9Daz28M1CO3Nrd+2PDo Spg5L9b4hPruDGwOG15QTJ4yMTCRGwm3auICiICVJf6QRB3fUdSyHM1RmU9beL+0SHde Pc798/MSTR4nMgsCGAnxjPK4FlCfRRNhppBOyDOuxIupKujzfQrWYM4+Llsivt/vod4t D4vQ== X-Gm-Message-State: AOJu0YyBCoAeBQhHHRaf6kAwIP4n6cN/+bxFQXklz0kY7LnulC/bkwHh o98etmKAoMl86kSxSWz14W0ZT465EBTj0orSbMZINnfBGrcqeY4Dl4Di28HeAqMIjprndU311Xm MZwXD44EaL9KTJVijT+SkWGRTS5z6V57H/SvM X-Received: by 2002:a17:903:cb:b0:1d4:94f7:b349 with SMTP id x11-20020a17090300cb00b001d494f7b349mr1393329plc.54.1704299568591; Wed, 03 Jan 2024 08:32:48 -0800 (PST) X-Google-Smtp-Source: AGHT+IGRh2QWtQD29WPJYeQ0K2F3ztPLRvH+XRo73kfZaTuSm2TpoOzI5zg6Q9eJt87gGsc7c2/5DA== X-Received: by 2002:a17:903:cb:b0:1d4:94f7:b349 with SMTP id x11-20020a17090300cb00b001d494f7b349mr1393323plc.54.1704299568269; Wed, 03 Jan 2024 08:32:48 -0800 (PST) Received: from localhost.localdomain ([2804:431:c7ec:911:6911:ca60:846:eb46]) by smtp.gmail.com with ESMTPSA id e12-20020a170902b78c00b001cfca7b8ee7sm23930425pls.99.2024.01.03.08.32.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Jan 2024 08:32:47 -0800 (PST) From: Leonardo Bras To: Will Deacon , Peter Zijlstra , Boqun Feng , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Leonardo Bras , Guo Ren , Andrea Parri , Geert Uytterhoeven , Ingo Molnar , Andrzej Hajda Cc: linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org Subject: [PATCH v1 3/5] riscv/atomic.h : Deduplicate arch_atomic.* Date: Wed, 3 Jan 2024 13:32:01 -0300 Message-ID: <20240103163203.72768-5-leobras@redhat.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240103163203.72768-2-leobras@redhat.com> References: <20240103163203.72768-2-leobras@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240103_083252_254335_DDD7D7F0 X-CRM114-Status: GOOD ( 11.74 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Some functions use mostly the same asm for 32-bit and 64-bit versions. Make a macro that is generic enough and avoid code duplication. (This did not cause any change in generated asm) Signed-off-by: Leonardo Bras Reviewed-by: Guo Ren Reviewed-by: Andrea Parri Tested-by: Guo Ren --- arch/riscv/include/asm/atomic.h | 164 +++++++++++++++----------------- 1 file changed, 76 insertions(+), 88 deletions(-) diff --git a/arch/riscv/include/asm/atomic.h b/arch/riscv/include/asm/atomic.h index f5dfef6c2153f..80cca7ac16fd3 100644 --- a/arch/riscv/include/asm/atomic.h +++ b/arch/riscv/include/asm/atomic.h @@ -196,22 +196,28 @@ ATOMIC_OPS(xor, xor, i) #undef ATOMIC_FETCH_OP #undef ATOMIC_OP_RETURN +#define _arch_atomic_fetch_add_unless(_prev, _rc, counter, _a, _u, sfx) \ +({ \ + __asm__ __volatile__ ( \ + "0: lr." sfx " %[p], %[c]\n" \ + " beq %[p], %[u], 1f\n" \ + " add %[rc], %[p], %[a]\n" \ + " sc." sfx ".rl %[rc], %[rc], %[c]\n" \ + " bnez %[rc], 0b\n" \ + " fence rw, rw\n" \ + "1:\n" \ + : [p]"=&r" (_prev), [rc]"=&r" (_rc), [c]"+A" (counter) \ + : [a]"r" (_a), [u]"r" (_u) \ + : "memory"); \ +}) + /* This is required to provide a full barrier on success. */ static __always_inline int arch_atomic_fetch_add_unless(atomic_t *v, int a, int u) { int prev, rc; - __asm__ __volatile__ ( - "0: lr.w %[p], %[c]\n" - " beq %[p], %[u], 1f\n" - " add %[rc], %[p], %[a]\n" - " sc.w.rl %[rc], %[rc], %[c]\n" - " bnez %[rc], 0b\n" - " fence rw, rw\n" - "1:\n" - : [p]"=&r" (prev), [rc]"=&r" (rc), [c]"+A" (v->counter) - : [a]"r" (a), [u]"r" (u) - : "memory"); + _arch_atomic_fetch_add_unless(prev, rc, v->counter, a, u, "w"); + return prev; } #define arch_atomic_fetch_add_unless arch_atomic_fetch_add_unless @@ -222,77 +228,86 @@ static __always_inline s64 arch_atomic64_fetch_add_unless(atomic64_t *v, s64 a, s64 prev; long rc; - __asm__ __volatile__ ( - "0: lr.d %[p], %[c]\n" - " beq %[p], %[u], 1f\n" - " add %[rc], %[p], %[a]\n" - " sc.d.rl %[rc], %[rc], %[c]\n" - " bnez %[rc], 0b\n" - " fence rw, rw\n" - "1:\n" - : [p]"=&r" (prev), [rc]"=&r" (rc), [c]"+A" (v->counter) - : [a]"r" (a), [u]"r" (u) - : "memory"); + _arch_atomic_fetch_add_unless(prev, rc, v->counter, a, u, "d"); + return prev; } #define arch_atomic64_fetch_add_unless arch_atomic64_fetch_add_unless #endif +#define _arch_atomic_inc_unless_negative(_prev, _rc, counter, sfx) \ +({ \ + __asm__ __volatile__ ( \ + "0: lr." sfx " %[p], %[c]\n" \ + " bltz %[p], 1f\n" \ + " addi %[rc], %[p], 1\n" \ + " sc." sfx ".rl %[rc], %[rc], %[c]\n" \ + " bnez %[rc], 0b\n" \ + " fence rw, rw\n" \ + "1:\n" \ + : [p]"=&r" (_prev), [rc]"=&r" (_rc), [c]"+A" (counter) \ + : \ + : "memory"); \ +}) + static __always_inline bool arch_atomic_inc_unless_negative(atomic_t *v) { int prev, rc; - __asm__ __volatile__ ( - "0: lr.w %[p], %[c]\n" - " bltz %[p], 1f\n" - " addi %[rc], %[p], 1\n" - " sc.w.rl %[rc], %[rc], %[c]\n" - " bnez %[rc], 0b\n" - " fence rw, rw\n" - "1:\n" - : [p]"=&r" (prev), [rc]"=&r" (rc), [c]"+A" (v->counter) - : - : "memory"); + _arch_atomic_inc_unless_negative(prev, rc, v->counter, "w"); + return !(prev < 0); } #define arch_atomic_inc_unless_negative arch_atomic_inc_unless_negative +#define _arch_atomic_dec_unless_positive(_prev, _rc, counter, sfx) \ +({ \ + __asm__ __volatile__ ( \ + "0: lr." sfx " %[p], %[c]\n" \ + " bgtz %[p], 1f\n" \ + " addi %[rc], %[p], -1\n" \ + " sc." sfx ".rl %[rc], %[rc], %[c]\n" \ + " bnez %[rc], 0b\n" \ + " fence rw, rw\n" \ + "1:\n" \ + : [p]"=&r" (_prev), [rc]"=&r" (_rc), [c]"+A" (counter) \ + : \ + : "memory"); \ +}) + static __always_inline bool arch_atomic_dec_unless_positive(atomic_t *v) { int prev, rc; - __asm__ __volatile__ ( - "0: lr.w %[p], %[c]\n" - " bgtz %[p], 1f\n" - " addi %[rc], %[p], -1\n" - " sc.w.rl %[rc], %[rc], %[c]\n" - " bnez %[rc], 0b\n" - " fence rw, rw\n" - "1:\n" - : [p]"=&r" (prev), [rc]"=&r" (rc), [c]"+A" (v->counter) - : - : "memory"); + _arch_atomic_dec_unless_positive(prev, rc, v->counter, "w"); + return !(prev > 0); } #define arch_atomic_dec_unless_positive arch_atomic_dec_unless_positive +#define _arch_atomic_dec_if_positive(_prev, _rc, counter, sfx) \ +({ \ + __asm__ __volatile__ ( \ + "0: lr." sfx " %[p], %[c]\n" \ + " addi %[rc], %[p], -1\n" \ + " bltz %[rc], 1f\n" \ + " sc." sfx ".rl %[rc], %[rc], %[c]\n" \ + " bnez %[rc], 0b\n" \ + " fence rw, rw\n" \ + "1:\n" \ + : [p]"=&r" (_prev), [rc]"=&r" (_rc), [c]"+A" (counter) \ + : \ + : "memory"); \ +}) + static __always_inline int arch_atomic_dec_if_positive(atomic_t *v) { int prev, rc; - __asm__ __volatile__ ( - "0: lr.w %[p], %[c]\n" - " addi %[rc], %[p], -1\n" - " bltz %[rc], 1f\n" - " sc.w.rl %[rc], %[rc], %[c]\n" - " bnez %[rc], 0b\n" - " fence rw, rw\n" - "1:\n" - : [p]"=&r" (prev), [rc]"=&r" (rc), [c]"+A" (v->counter) - : - : "memory"); + _arch_atomic_dec_if_positive(prev, rc, v->counter, "w"); + return prev - 1; } @@ -304,17 +319,8 @@ static __always_inline bool arch_atomic64_inc_unless_negative(atomic64_t *v) s64 prev; long rc; - __asm__ __volatile__ ( - "0: lr.d %[p], %[c]\n" - " bltz %[p], 1f\n" - " addi %[rc], %[p], 1\n" - " sc.d.rl %[rc], %[rc], %[c]\n" - " bnez %[rc], 0b\n" - " fence rw, rw\n" - "1:\n" - : [p]"=&r" (prev), [rc]"=&r" (rc), [c]"+A" (v->counter) - : - : "memory"); + _arch_atomic_inc_unless_negative(prev, rc, v->counter, "d"); + return !(prev < 0); } @@ -325,17 +331,8 @@ static __always_inline bool arch_atomic64_dec_unless_positive(atomic64_t *v) s64 prev; long rc; - __asm__ __volatile__ ( - "0: lr.d %[p], %[c]\n" - " bgtz %[p], 1f\n" - " addi %[rc], %[p], -1\n" - " sc.d.rl %[rc], %[rc], %[c]\n" - " bnez %[rc], 0b\n" - " fence rw, rw\n" - "1:\n" - : [p]"=&r" (prev), [rc]"=&r" (rc), [c]"+A" (v->counter) - : - : "memory"); + _arch_atomic_dec_unless_positive(prev, rc, v->counter, "d"); + return !(prev > 0); } @@ -346,17 +343,8 @@ static __always_inline s64 arch_atomic64_dec_if_positive(atomic64_t *v) s64 prev; long rc; - __asm__ __volatile__ ( - "0: lr.d %[p], %[c]\n" - " addi %[rc], %[p], -1\n" - " bltz %[rc], 1f\n" - " sc.d.rl %[rc], %[rc], %[c]\n" - " bnez %[rc], 0b\n" - " fence rw, rw\n" - "1:\n" - : [p]"=&r" (prev), [rc]"=&r" (rc), [c]"+A" (v->counter) - : - : "memory"); + _arch_atomic_dec_if_positive(prev, rc, v->counter, "d"); + return prev - 1; } From patchwork Wed Jan 3 16:32:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leonardo Bras X-Patchwork-Id: 13510242 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BD03CC47077 for ; Wed, 3 Jan 2024 16:33:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=sGYGqyPxu7fbmQUHYmySBfiUawr01wrikgw6rwZ0mdw=; b=lKSJohW4Ycmprr E3qjkhB6emPJhTopL51NzWqvE/ZpNh1LwXxBcPFKnEbpK8eX1Qg8kNQ0wjw8ZnmhS/mRJfbrvnOXK T33Zy4ezSEVxJiz7OwhwV/x9H7znG4oQ6KklNFVd563p8042nXV5kLLQpSPdfuAflmxGQtnDe8yOb APcVUVXChlBjxEgkXD6FT2Z9TKUbLpWCMQISdCJ7YG6CwnSeKiDcpikxe4n8LKnQ6eh6Sd9DTAfXG ZNiYwPAP3yWnE7YKXJVg87WaDWEsR9GgE0bBjT/Gfmud69H2qYjtpY0119ouo/ywdJtyYL19wdSYn oLEahiR+vpYxXYuy8xMA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rL4At-00BP4h-26; Wed, 03 Jan 2024 16:32:59 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rL4Ar-00BP3O-0c for linux-riscv@lists.infradead.org; Wed, 03 Jan 2024 16:32:58 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1704299576; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Zrl9wQMm72/9gkP6lQrsUXatEy2G8nXg2opkxwrV5kI=; b=KbQGIeIGD4JKjW6veg1igxK5wtP7Ypfo0QBKnFDqzvXaJ70sWci/mLdIpEKTiKlSr4nolP IVU2rk8o3Dp6PRvSSP9mFw9BjvwypMnyDg2g392IWZS1Hccn3sDxMKwypYwDBGZDnYm56v 3NcOa78PMTohtfA/Mb5hXnSn7vV6o5c= Received: from mail-pl1-f200.google.com (mail-pl1-f200.google.com [209.85.214.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-427-IVe_o-aNPZuw0Ptl36qlBw-1; Wed, 03 Jan 2024 11:32:54 -0500 X-MC-Unique: IVe_o-aNPZuw0Ptl36qlBw-1 Received: by mail-pl1-f200.google.com with SMTP id d9443c01a7336-1d4931d651aso50666485ad.2 for ; Wed, 03 Jan 2024 08:32:54 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704299574; x=1704904374; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Zrl9wQMm72/9gkP6lQrsUXatEy2G8nXg2opkxwrV5kI=; b=eQaDj/njY+/8fpETS7rG2N/JWjulYS56edCrt3P9+v3OhX0WcVaGRUMgaqs2M//4NG aRIvSq50YjC3n7Hds9XiYZVI/6ge73QUQDr7PR9CInclNOol7f93MN92g/hPLOOijDw1 qhYF0NRUjGy8cS/ceOzXake7kO6UeFwFUdBXoU/uHm9c13NJdsGaouMDHSZ4WK8hMcvl pxPyhFkEXXWPhNIL6hQOJslHtzbyeLs370gVKdNuWT3a0FU7+R26/baL/WbJG2Is2h+j plSbgNCiaSE4tpMjs4IpBfebJuKdwpVcUW2X83uXFDiSe1sXfwxeeHttQqPUv7lZiImE YfPg== X-Gm-Message-State: AOJu0YxZmwCMdC0G15XE+Ja9VtWNi8jcCZ1cTJlPDBpl3oXMyDTcUcSn OSde16HIKcrW8UHfVfaCJnd8qLnppBfEuWU1V5hrDuSUeOoMz0l0o9/WhSYaXR7w2+PfCgrjsF8 DITQXDUDGZSeVyoDGe7zjyNpN9Tn0tWSEvUS9 X-Received: by 2002:a17:902:b213:b0:1d4:3dfd:7e31 with SMTP id t19-20020a170902b21300b001d43dfd7e31mr20225794plr.139.1704299573992; Wed, 03 Jan 2024 08:32:53 -0800 (PST) X-Google-Smtp-Source: AGHT+IFuxxr3jzBnWJLb5k5Ijiau2Uo52Uln5XHUYl8Xl2mWgyNYjwzpHX8ACzhU950ngT5irx/v3w== X-Received: by 2002:a17:902:b213:b0:1d4:3dfd:7e31 with SMTP id t19-20020a170902b21300b001d43dfd7e31mr20225778plr.139.1704299573733; Wed, 03 Jan 2024 08:32:53 -0800 (PST) Received: from localhost.localdomain ([2804:431:c7ec:911:6911:ca60:846:eb46]) by smtp.gmail.com with ESMTPSA id e12-20020a170902b78c00b001cfca7b8ee7sm23930425pls.99.2024.01.03.08.32.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Jan 2024 08:32:53 -0800 (PST) From: Leonardo Bras To: Will Deacon , Peter Zijlstra , Boqun Feng , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Leonardo Bras , Guo Ren , Andrea Parri , Geert Uytterhoeven , Ingo Molnar , Andrzej Hajda Cc: linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org Subject: [PATCH v1 4/5] riscv/cmpxchg: Implement cmpxchg for variables of size 1 and 2 Date: Wed, 3 Jan 2024 13:32:02 -0300 Message-ID: <20240103163203.72768-6-leobras@redhat.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240103163203.72768-2-leobras@redhat.com> References: <20240103163203.72768-2-leobras@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240103_083257_299498_BAABD170 X-CRM114-Status: GOOD ( 12.59 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org cmpxchg for variables of size 1-byte and 2-bytes is not yet available for riscv, even though its present in other architectures such as arm64 and x86. This could lead to not being able to implement some locking mechanisms or requiring some rework to make it work properly. Implement 1-byte and 2-bytes cmpxchg in order to achieve parity with other architectures. Signed-off-by: Leonardo Bras Tested-by: Guo Ren --- arch/riscv/include/asm/cmpxchg.h | 34 ++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index e3e0ac7ba061b..ac9d0eeb74e67 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -72,6 +72,35 @@ * indicated by comparing RETURN with OLD. */ +#define __arch_cmpxchg_masked(sc_sfx, prepend, append, r, p, o, n) \ +({ \ + u32 *__ptr32b = (u32 *)((ulong)(p) & ~0x3); \ + ulong __s = ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \ + ulong __mask = GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \ + << __s; \ + ulong __newx = (ulong)(n) << __s; \ + ulong __oldx = (ulong)(o) << __s; \ + ulong __retx; \ + ulong __rc; \ + \ + __asm__ __volatile__ ( \ + prepend \ + "0: lr.w %0, %2\n" \ + " and %1, %0, %z5\n" \ + " bne %1, %z3, 1f\n" \ + " and %1, %0, %z6\n" \ + " or %1, %1, %z4\n" \ + " sc.w" sc_sfx " %1, %1, %2\n" \ + " bnez %1, 0b\n" \ + append \ + "1:\n" \ + : "=&r" (__retx), "=&r" (__rc), "+A" (*(__ptr32b)) \ + : "rJ" ((long)__oldx), "rJ" (__newx), \ + "rJ" (__mask), "rJ" (~__mask) \ + : "memory"); \ + \ + r = (__typeof__(*(p)))((__retx & __mask) >> __s); \ +}) #define __arch_cmpxchg(lr_sfx, sc_sfx, prepend, append, r, p, co, o, n) \ ({ \ @@ -98,6 +127,11 @@ __typeof__(*(__ptr)) __ret; \ \ switch (sizeof(*__ptr)) { \ + case 1: \ + case 2: \ + __arch_cmpxchg_masked(sc_sfx, prepend, append, \ + __ret, __ptr, __old, __new); \ + break; \ case 4: \ __arch_cmpxchg(".w", ".w" sc_sfx, prepend, append, \ __ret, __ptr, (long), __old, __new); \ From patchwork Wed Jan 3 16:32:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leonardo Bras X-Patchwork-Id: 13510243 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 22488C3DA6E for ; Wed, 3 Jan 2024 16:33:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Xu3HhtWXyvYM5XT19iajBWX4H11bqUU1YrUGQH7pKqM=; b=qK/nVGCW9pn1Ft v0mHQ5upDsaNwDCvNpv5D8F7ctui7SS8N3hB6KG1lYYJ5is94xhICcapOXzKBP8bR0UzDk6qFvDjc Rf5kj1SU+TOocboAWnZRO2Kkd8Ir+V1rF+eDq59rCF/YyDBRZ3YOj62sL/qUbta7OnOsq4w7T3IYe 5iuea1ITss3Z438278ZMWvFUfFJdHYGf2ux8N47Cj1gVsLf0+mfMeKBsF45kaqX9uNYPOYkufi5OS Jzl6L6IsQ1IEgT1dEGBU9nWmDFJ6+axrfcNVfubXAYkduWYD1rffXEywaZAC9k8MpuRYmt4fryKUl 8ho45xNDgLRZrzuOP96g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rL4B0-00BP9P-1v; Wed, 03 Jan 2024 16:33:06 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rL4Ax-00BP6w-25 for linux-riscv@lists.infradead.org; Wed, 03 Jan 2024 16:33:05 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1704299582; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Re2lmE+u9xJehZpa61KU1PY4uKcHNA9CavGcQorFlps=; b=PQKjEzCROz8Bxv1R1xGPl80/psMrbcKblczZe7yADc5xww7KftdGu67ZRS9ORO99mKEElS VclewbApl3QUppUTaLsdlUsRSEqxcIRdfKGSyvUbZWy7RmbaV+KasKFHoa5NzeS2RBltsT Oi5KM78iCb4CeYHlUi+HwxBZ4fnAhLE= Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-13-Toqjmkm6NlqEJb9yR3TvtQ-1; Wed, 03 Jan 2024 11:33:00 -0500 X-MC-Unique: Toqjmkm6NlqEJb9yR3TvtQ-1 Received: by mail-pl1-f199.google.com with SMTP id d9443c01a7336-1d455b34723so79204255ad.0 for ; Wed, 03 Jan 2024 08:33:00 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704299579; x=1704904379; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Re2lmE+u9xJehZpa61KU1PY4uKcHNA9CavGcQorFlps=; b=he1CZdDwEDLqQgkb0hco6X/d983ytV1byZFy1N9LG7CmwsljPkBx2p9X7/Xuae0sDV 0GPNSSMbF0lmjHiVuwgxY8hvkH+Dc2D3P+29YbH8mG+tNEuXW15WjPLLu/7t0Sewn0vC t5NBBAnOHfBkGbm+W4EFst0CRrnBtkjYJADAuOvJ5jV/FGdyc0K/JpB340uiVmpIUZ23 W917rjQOq2CjYpGYEtWT5PZZVDtkshndJ7mb5AOnvGXf/JZUQYsdH0AYeNsXJNRnizfI bPjNNJGYQKlkUqlIj98tweLcbwSwuVZKdhXRsf0Jua1yGBGCoJyLAv/tixpwyCwBLxoC ZlFw== X-Gm-Message-State: AOJu0YxSoxKXlD3PpePPj9OC3LL4x0hxxsxX+wQlIwd3c2Fw8OXngFtj Qt2KN4WL707KfESlaONpofo7AtloSawm2DkbTVpagoVkd98yqpjtIQgs1sVybYOr0byVLRw5YSm AbajjaFTCl8IfjgDOVhRrwX7+Wl5CntUiaKjv X-Received: by 2002:a17:902:b94c:b0:1d0:6ffe:a11 with SMTP id h12-20020a170902b94c00b001d06ffe0a11mr22000202pls.111.1704299579620; Wed, 03 Jan 2024 08:32:59 -0800 (PST) X-Google-Smtp-Source: AGHT+IFssUiBpWP8x7axGoljRgCI2CzrmRE+huZgi6wtwmSBGSU76WiGWvRbbX36kTz3/VUagQIV+g== X-Received: by 2002:a17:902:b94c:b0:1d0:6ffe:a11 with SMTP id h12-20020a170902b94c00b001d06ffe0a11mr22000194pls.111.1704299579375; Wed, 03 Jan 2024 08:32:59 -0800 (PST) Received: from localhost.localdomain ([2804:431:c7ec:911:6911:ca60:846:eb46]) by smtp.gmail.com with ESMTPSA id e12-20020a170902b78c00b001cfca7b8ee7sm23930425pls.99.2024.01.03.08.32.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Jan 2024 08:32:58 -0800 (PST) From: Leonardo Bras To: Will Deacon , Peter Zijlstra , Boqun Feng , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Leonardo Bras , Guo Ren , Andrea Parri , Geert Uytterhoeven , Ingo Molnar , Andrzej Hajda Cc: linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org Subject: [PATCH v1 5/5] riscv/cmpxchg: Implement xchg for variables of size 1 and 2 Date: Wed, 3 Jan 2024 13:32:03 -0300 Message-ID: <20240103163203.72768-7-leobras@redhat.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240103163203.72768-2-leobras@redhat.com> References: <20240103163203.72768-2-leobras@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240103_083303_831096_B736BB82 X-CRM114-Status: GOOD ( 12.53 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org xchg for variables of size 1-byte and 2-bytes is not yet available for riscv, even though its present in other architectures such as arm64 and x86. This could lead to not being able to implement some locking mechanisms or requiring some rework to make it work properly. Implement 1-byte and 2-bytes xchg in order to achieve parity with other architectures. Signed-off-by: Leonardo Bras Tested-by: Guo Ren --- arch/riscv/include/asm/cmpxchg.h | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index ac9d0eeb74e67..26cea2395aae8 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -11,6 +11,31 @@ #include #include +#define __arch_xchg_masked(prepend, append, r, p, n) \ +({ \ + u32 *__ptr32b = (u32 *)((ulong)(p) & ~0x3); \ + ulong __s = ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \ + ulong __mask = GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \ + << __s; \ + ulong __newx = (ulong)(n) << __s; \ + ulong __retx; \ + ulong __rc; \ + \ + __asm__ __volatile__ ( \ + prepend \ + "0: lr.w %0, %2\n" \ + " and %1, %0, %z4\n" \ + " or %1, %1, %z3\n" \ + " sc.w %1, %1, %2\n" \ + " bnez %1, 0b\n" \ + append \ + : "=&r" (__retx), "=&r" (__rc), "+A" (*(__ptr32b)) \ + : "rJ" (__newx), "rJ" (~__mask) \ + : "memory"); \ + \ + r = (__typeof__(*(p)))((__retx & __mask) >> __s); \ +}) + #define __arch_xchg(sfx, prepend, append, r, p, n) \ ({ \ __asm__ __volatile__ ( \ @@ -27,7 +52,13 @@ __typeof__(ptr) __ptr = (ptr); \ __typeof__(*(__ptr)) __new = (new); \ __typeof__(*(__ptr)) __ret; \ + \ switch (sizeof(*__ptr)) { \ + case 1: \ + case 2: \ + __arch_xchg_masked(prepend, append, \ + __ret, __ptr, __new); \ + break; \ case 4: \ __arch_xchg(".w" sfx, prepend, append, \ __ret, __ptr, __new); \