From patchwork Wed Jun 26 13:03:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13712898 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B4E7DC27C4F for ; Wed, 26 Jun 2024 13:05:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=M8LY7goC1COuKrz5TRr7ERv67T5rGsq+z8cV7YoEPRo=; b=hRqGPCsSewPd2w 4h7N3Nf/GfQJO+JQXDgYOqNpvks/0vOUtc4FzmSqOGb08WzE9+3uR8QYkJRiyYvAqIvtoNPOodMi/ 6Lf7LnGKcRVVcWO9L1tUjl2hrUbYz28bevBJbSqEwyM7IER0RuQyvJ5UAgBF+P5IACbt2lk2Eh7Ai 8avIf2zFzjQboaJvdQBCIEt7IFJLVxQswELPgsqhIb+mIKMWbtcAz0hTiomcRj9Gqb8p+Z3efobuj PN46pbw7NJ8TGt6tiwthfhSm96t4pda+4XfI/3HM6TN/vIEM7CO2hqg0UQip8be9SRxIq6FuKRvBC m2Llxu+gV4lHQmJG1O2w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sMSKa-00000006qcO-0LmJ; Wed, 26 Jun 2024 13:05:00 +0000 Received: from mail-lj1-x22f.google.com ([2a00:1450:4864:20::22f]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sMSKV-00000006qaV-03n4 for linux-riscv@lists.infradead.org; Wed, 26 Jun 2024 13:04:57 +0000 Received: by mail-lj1-x22f.google.com with SMTP id 38308e7fff4ca-2ec61eeed8eso38253821fa.0 for ; Wed, 26 Jun 2024 06:04:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1719407093; x=1720011893; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WhXCNVmR8P99l/LeuRhjy5KBZNQ4/EvtQCsDBcea+Tc=; b=e/1Vlx6rDHU1uKr3PDNUgAdLqkV5dtLkTfQ7AXc6XDAa/BIEPRrxGGG5yaE6J01x0F Pjck6l8FgxcRy4MocjAGBx+udaipGnK2xYGc7gecYxWtDLoTD3DneWxKfCPvJkNljC2b lS0GsCOZA2WuBd68D7WhA47Vmeq33j/EC1KmIyyXuYuqGaVv5aPq375DkhjA4q4tv54u zAjUwIeQd451fj3Z34JX/f2usEF+uBsGguMjGn2khrrv+UM+xaXae2ph5BMrlhbD11uB 8Aq5RWTm71kyLIIWFENsffe/OLtQWAghDvcvxmrQKwcDcNoNDw9d5CzjNjQpWx0U4A8P qqMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719407093; x=1720011893; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WhXCNVmR8P99l/LeuRhjy5KBZNQ4/EvtQCsDBcea+Tc=; b=RzBQVUexpBDMCZa44MSdigy7+f6ZMx96mOX0RaGvk5Qua+oD0kSPXZeyqPE1Jx+34H 5W27mFgnsbiJCsRixDkolp08UUVb+/jUVZAA91qDWhJxiobUPVnt7QrSq5cpcaShJQ3X BxASFxJP2Sn5M1Q5JJA/STYIEKrFCXpqQKAT8wGIPlvDOxrseErdaBIE1meEpoWuLLtS IGM8LrshvwKZmZrMYR2+0pCahCKEXOI9ivUw/J9NvFcXWk69Jm31gh+PHjxBHu3COSZK IhJ7Lu6Br6Uw8RHGNEXP3uP1hae/47kcK7QkjeEWudihvMQvD0csW/jN5Wo7TtOqRL/2 cihQ== X-Forwarded-Encrypted: i=1; AJvYcCW5EzQwWjoGAZcibb718V24OHXeSH5vKl7YF9WV6eB6by/fmVq0v0gvloul23r/HgyojEheyv2Q9Ul6J6BH3f+HRUYq6c3x7/mukTusaXcL X-Gm-Message-State: AOJu0YzkjcXRsR6dsX3t6eN8PWe8C0JtWPTckoNmALBp0FvSU9n0/6tB 4toWbXyeNJSun8HrVwmNg1sILyg9dSz3UxWwXN9tlFW3vGAt4QPUCfNztlGulHU= X-Google-Smtp-Source: AGHT+IGXVzg0cT42nAZtoqYsiJZ5s31h9gkqhz8eo+Ns1Bzdpb6n1iQQeyrD5Jr8Y6qR9urf80as6w== X-Received: by 2002:ac2:596e:0:b0:52c:db7b:b463 with SMTP id 2adb3069b0e04-52ce186459cmr6385178e87.61.1719407093047; Wed, 26 Jun 2024 06:04:53 -0700 (PDT) Received: from localhost.localdomain (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-424c842468dsm25643865e9.36.2024.06.26.06.04.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Jun 2024 06:04:52 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH v2 01/10] riscv: Implement cmpxchg32/64() using Zacas Date: Wed, 26 Jun 2024 15:03:38 +0200 Message-Id: <20240626130347.520750-2-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240626130347.520750-1-alexghiti@rivosinc.com> References: <20240626130347.520750-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240626_060455_068511_D76E86F1 X-CRM114-Status: GOOD ( 13.27 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org This adds runtime support for Zacas in cmpxchg operations. Signed-off-by: Alexandre Ghiti --- arch/riscv/Kconfig | 17 +++++++++++++++++ arch/riscv/Makefile | 3 +++ arch/riscv/include/asm/cmpxchg.h | 27 ++++++++++++++++++++++++--- 3 files changed, 44 insertions(+), 3 deletions(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 05ccba8ca33a..1caaedec88c7 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -596,6 +596,23 @@ config RISCV_ISA_V_PREEMPTIVE preemption. Enabling this config will result in higher memory consumption due to the allocation of per-task's kernel Vector context. +config TOOLCHAIN_HAS_ZACAS + bool + default y + depends on !64BIT || $(cc-option,-mabi=lp64 -march=rv64ima_zacas) + depends on !32BIT || $(cc-option,-mabi=ilp32 -march=rv32ima_zacas) + depends on AS_HAS_OPTION_ARCH + +config RISCV_ISA_ZACAS + bool "Zacas extension support for atomic CAS" + depends on TOOLCHAIN_HAS_ZACAS + default y + help + Enable the use of the Zacas ISA-extension to implement kernel atomic + cmpxchg operations when it is detected at boot. + + If you don't know what to do here, say Y. + config TOOLCHAIN_HAS_ZBB bool default y diff --git a/arch/riscv/Makefile b/arch/riscv/Makefile index 06de9d365088..9fd13d7a9cc6 100644 --- a/arch/riscv/Makefile +++ b/arch/riscv/Makefile @@ -85,6 +85,9 @@ endif # Check if the toolchain supports Zihintpause extension riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZIHINTPAUSE) := $(riscv-march-y)_zihintpause +# Check if the toolchain supports Zacas +riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZACAS) := $(riscv-march-y)_zacas + # Remove F,D,V from isa string for all. Keep extensions between "fd" and "v" by # matching non-v and non-multi-letter extensions out with the filter ([^v_]*) KBUILD_CFLAGS += -march=$(shell echo $(riscv-march-y) | sed -E 's/(rv32ima|rv64ima)fd([^v_]*)v?/\1\2/') diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index 808b4c78462e..a58a2141c6d3 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -9,6 +9,7 @@ #include #include +#include #define __arch_xchg_masked(sc_sfx, prepend, append, r, p, n) \ ({ \ @@ -134,21 +135,41 @@ r = (__typeof__(*(p)))((__retx & __mask) >> __s); \ }) -#define __arch_cmpxchg(lr_sfx, sc_sfx, prepend, append, r, p, co, o, n) \ +#define __arch_cmpxchg(lr_sfx, sc_cas_sfx, prepend, append, r, p, co, o, n) \ ({ \ + __label__ zacas, end; \ register unsigned int __rc; \ \ + if (IS_ENABLED(CONFIG_RISCV_ISA_ZACAS)) { \ + asm goto(ALTERNATIVE("nop", "j %[zacas]", 0, \ + RISCV_ISA_EXT_ZACAS, 1) \ + : : : : zacas); \ + } \ + \ __asm__ __volatile__ ( \ prepend \ "0: lr" lr_sfx " %0, %2\n" \ " bne %0, %z3, 1f\n" \ - " sc" sc_sfx " %1, %z4, %2\n" \ + " sc" sc_cas_sfx " %1, %z4, %2\n" \ " bnez %1, 0b\n" \ append \ "1:\n" \ : "=&r" (r), "=&r" (__rc), "+A" (*(p)) \ : "rJ" (co o), "rJ" (n) \ : "memory"); \ + goto end; \ + \ +zacas: \ + if (IS_ENABLED(CONFIG_RISCV_ISA_ZACAS)) { \ + __asm__ __volatile__ ( \ + prepend \ + " amocas" sc_cas_sfx " %0, %z2, %1\n" \ + append \ + : "+&r" (r), "+A" (*(p)) \ + : "rJ" (n) \ + : "memory"); \ + } \ +end:; \ }) #define _arch_cmpxchg(ptr, old, new, sc_sfx, prepend, append) \ @@ -156,7 +177,7 @@ __typeof__(ptr) __ptr = (ptr); \ __typeof__(*(__ptr)) __old = (old); \ __typeof__(*(__ptr)) __new = (new); \ - __typeof__(*(__ptr)) __ret; \ + __typeof__(*(__ptr)) __ret = (old); \ \ switch (sizeof(*__ptr)) { \ case 1: \ From patchwork Wed Jun 26 13:03:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13712899 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 68CB4C30653 for ; Wed, 26 Jun 2024 13:06:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Zz5CieM7fK8R3oPlwg+PHpbclLpuk3VKA01Omdj5cKM=; b=00dBIWBbGUuTCw pMj+cBis37Vp+qu4qkF/B26nVoQ6fnqrWBKBn4Szt5gslnpOPLwEpQUgavxyIeEEFMPqKvr1pglWw 0rn2gbJuQjSLxzzqPZNDruklFofCrZfptd0s7c2/QMazG5SNGktdZlg03iIJ6lXfXFptlNh2Cd09+ KrihUrhSfLUAjw2LUVi3A3GvugHa95hulxIS48Ud/Se8GzGY6Gon96a1aIZsuwtgKtBfBQq1WuaWM bDkqMs8sB7h54LToffVI9ua4gXbuaXCWlCfop+O0XDuQjE6nIATYAmdUmYZ3rf1v2FpU0R2AbBBP8 WaxtqcLI6K3xqdbcrn6w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sMSM4-00000006rO0-2h41; Wed, 26 Jun 2024 13:06:32 +0000 Received: from mail-lj1-x22d.google.com ([2a00:1450:4864:20::22d]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sMSLU-00000006r02-0B4k for linux-riscv@lists.infradead.org; Wed, 26 Jun 2024 13:06:02 +0000 Received: by mail-lj1-x22d.google.com with SMTP id 38308e7fff4ca-2ec3f875e68so74375051fa.0 for ; Wed, 26 Jun 2024 06:05:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1719407154; x=1720011954; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=T6LJZCR61gmm4dvekAa0Q1TIoLF/ScrG5TtOv9gDZ/w=; b=mmGkNjD4LPUVKhgWPrRLayP+q/jzFvrHawELWe9xnJMaEB6zHv5VPzQb5NxUvUyEpO XP49VYLhk9DVwL0oQ+TEU/vdghGp8ywoYXQAd78TR1Qk+bOeUfqQ5AuVp7gZULCHp66k FPXBpEDEhssU0sC6PB5NZT4B/AxHGCmsw9CFfBo4vE7N5UrAio9i67ALAZrC+qrvCO+B FLFAAIQ8ijAeEZVoKxsZivdJtUdM155P5rbm8DFEwNCZQ/+jbWDv09R901AfeLOk3l0e upJw3KWlevtSBJlWhb+vLy8VsXc1K9WoruAWLTO8DPc3grnBsNAj3YbRx9B1yMeG9km+ dgbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719407154; x=1720011954; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=T6LJZCR61gmm4dvekAa0Q1TIoLF/ScrG5TtOv9gDZ/w=; b=IJBSdlSU7gKIePS9fmhmoxUWhwN861HxKi87mYEkzWqZySxzUenTMHXara3FR+MU9L JALUxU+3ObzLlHjFW9XZOhW1t346pV/2uIG6SJDNP2Jkx1bx51Fc+ybmyaS4txOiQOvJ oslg66swERZoLgLtCFrZUIgI1GmBSMuDbwNFpILTW6EXqJeS04hW+Rybbi6f4c4601aa GVNcudYWLt8Kq4Yb+QEVWyo5+DlWt+8qFewfj5opWVhKo/VoUoBsmVg7Iza+qkw/QJLx +MleIX15DxJxu86X/5ybg7cCnS9KYAuA7opxDC2Eplb1CKaUDX3xiVjo7Jpionb+gAe3 wtKA== X-Forwarded-Encrypted: i=1; AJvYcCV5egj4y4KzrFBo3CXvczAb9ReUdRka4rXmS9y3z7ZBtRoBKvpe3nzv/+e3+3DIHN7PLLIUfV1Z+IkdSD+6ZPwVl9NnQsWk5bFfmiQQATBP X-Gm-Message-State: AOJu0YxvAEmdVBLkzTRsOQXl901iGG10/nIJorSMMdCGZ6SHfx9bNOsz m0GFgwzi5AUQSoTEKNxUulNiEd1TPD8YesW3MMR2bCfb74lswqhy0lOmG/4uhlU= X-Google-Smtp-Source: AGHT+IHX+jzGQgnJriZOGQ/oAJSCUkFR6oZJI+0k3ahW32MnlEUF+2I4ESnYZpAqS9088kQ4KTVI2w== X-Received: by 2002:ac2:5f4e:0:b0:52c:e030:144e with SMTP id 2adb3069b0e04-52ce185ecd4mr6228931e87.47.1719407154179; Wed, 26 Jun 2024 06:05:54 -0700 (PDT) Received: from localhost.localdomain (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-36638d9bd0fsm15831149f8f.52.2024.06.26.06.05.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Jun 2024 06:05:53 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH v2 02/10] dt-bindings: riscv: Add Zabha ISA extension description Date: Wed, 26 Jun 2024 15:03:39 +0200 Message-Id: <20240626130347.520750-3-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240626130347.520750-1-alexghiti@rivosinc.com> References: <20240626130347.520750-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240626_060600_370361_94F83252 X-CRM114-Status: UNSURE ( 9.57 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Add description for the Zabha ISA extension which was ratified in April 2024. Signed-off-by: Alexandre Ghiti --- Documentation/devicetree/bindings/riscv/extensions.yaml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/Documentation/devicetree/bindings/riscv/extensions.yaml b/Documentation/devicetree/bindings/riscv/extensions.yaml index 468c646247aa..e6436260bdeb 100644 --- a/Documentation/devicetree/bindings/riscv/extensions.yaml +++ b/Documentation/devicetree/bindings/riscv/extensions.yaml @@ -171,6 +171,12 @@ properties: memory types as ratified in the 20191213 version of the privileged ISA specification. + - const: zabha + description: | + The Zabha extension for Byte and Halfword Atomic Memory Operations + as ratified at commit 49f49c842ff9 ("Update to Rafified state") of + riscv-zabha. + - const: zacas description: | The Zacas extension for Atomic Compare-and-Swap (CAS) instructions From patchwork Wed Jun 26 13:03:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13712912 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 87A26C30659 for ; Wed, 26 Jun 2024 13:08:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=hFFj7mXgAXKlMXQPxX7PagnBg8KDRV4St7YY60jbbBE=; b=QH4QrJqsK14Ap+ lZYgZpihgic0jY47onb+FE79zd1fm8+SK9KMVf0a+dWsqRf6sb6Ykr7ik8C+7yFm2fWcQrlqhjfln hQLXZNDqn1pdeZzjCoRXoDauHKOPQP/VIHSF+koGST+slULimieGCsTuRHO+ZLiMRj/b3tMfl7num mhIA2XaKiOGBw6eZho1MtAtJ33CM3XEY2IayKZ1TwYiij2EKgX1aQzx0ppq12zlTOJo2w0kcxMPJt AVKOWf9Pmf8vvRsv2OsgjXF750YInvjCJ3mliM5XY/diEXJcrlHI/FF5IXsh8dAUzP7I/DPIFfViS 0gsZAwVB33F5K5msa2oQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sMSO9-00000006sYU-14Qc; Wed, 26 Jun 2024 13:08:41 +0000 Received: from mail-lj1-x232.google.com ([2a00:1450:4864:20::232]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sMSMT-00000006rcx-2eh5 for linux-riscv@lists.infradead.org; Wed, 26 Jun 2024 13:07:00 +0000 Received: by mail-lj1-x232.google.com with SMTP id 38308e7fff4ca-2ec408c6d94so75541231fa.3 for ; Wed, 26 Jun 2024 06:06:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1719407215; x=1720012015; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=tr0+ChkIZxO95Fw/MskGbHkedxBPmtMPKffgsaSfjcQ=; b=PrVrn+t9+YnQXDlWKpxyAHQSa/6a1CM4eoF8UZaqRuSGF8IgsRUSDrWv+Z6ZU1oP+/ Rr/kFbJHy+6ymPTgHS9gGWOzj/ZpddLXfRhLn8D7NbJlNs7Ju4UaKH6hYj5kkvmi1+tH UZajJ1OJ9U6ipzQh3O0SznKOU2F2CletsEd48r8EsLQ7eFZOrMLSJSTAGIDbwTZtcHWa PUlBHHfIQTO7JXcQA2QPIikmiPRuwC8XFj9Z1jX0p3vHcSyjLcM6ZHILgdKGBgEjEMfn hm4QAWtwz5DKLKOspVoX8dhPZ1Yq1ZTjsLC8XRXzoe5JOPtVQ1/icKvkTjUfgpgd4EiP DlUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719407215; x=1720012015; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tr0+ChkIZxO95Fw/MskGbHkedxBPmtMPKffgsaSfjcQ=; b=Lpr9DuLPoi6gHTCCEDKqoQDtJZpEnUYnswuSGZAMTcEWgJD/E5r595EbIOS1+eejmP /IQPtqNj0RdexxR41ovwgbuwea0jl0T1cITAFqhH6dRMAsNIpBRRycuMSt6de1c0OJAj VhCzDLPVj3+tf+d+0kl6XqsffHoNYHLMCwlAQz2t5CuqyQWTxf1L5IwCdw/GXOCPOxOm AD2WXdRNAWDc9MgYGuntPSTrrkA2/0DSzW33bosJYjGBHj7MG74huXBmP7ZAMFbQt05p Oi63iZf0UXkCkqEjD4oV29YEuYZ/+R7WUXn229aI01CaMFvrV+bxwI9oUHPCoJQwOfIl 9iIw== X-Forwarded-Encrypted: i=1; AJvYcCU8SshbJ7ufhSTC+jaZP8RuJ8WVFLIf8ttcGHh+hDI+EZjZLphPyFuDyBS4SIEQVBG52JQjMYcf6dN7NxsymedaDddq8xBzJmAhXDjisIoA X-Gm-Message-State: AOJu0YybnIm9C+54mcl2jw/09H6IW7tBFypR01r6OjwXb9VHCLopNiIx 8oU9UZxL7GhwTKq4OGh/JikKDR1GIqj8VsieFLbG8Z2Gahl9Sm3F0bMoIUTkfJE= X-Google-Smtp-Source: AGHT+IFc4UBrVIS7Rn9fDb0+V6vAa1vZLUOTYZ8/Eg+bWg3KUQ8/IYiEJJBiokMuSCcEt25man3iiw== X-Received: by 2002:a2e:9b4d:0:b0:2ec:543f:6013 with SMTP id 38308e7fff4ca-2ec5931d86amr78335911fa.13.1719407215370; Wed, 26 Jun 2024 06:06:55 -0700 (PDT) Received: from localhost.localdomain (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-424c846a7e7sm25696035e9.45.2024.06.26.06.06.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Jun 2024 06:06:55 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH v2 03/10] riscv: Implement cmpxchg8/16() using Zabha Date: Wed, 26 Jun 2024 15:03:40 +0200 Message-Id: <20240626130347.520750-4-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240626130347.520750-1-alexghiti@rivosinc.com> References: <20240626130347.520750-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240626_060657_952889_86A5292E X-CRM114-Status: GOOD ( 13.06 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org This adds runtime support for Zabha in cmpxchg8/16() operations. Signed-off-by: Alexandre Ghiti --- arch/riscv/Kconfig | 17 ++++++++++++++++ arch/riscv/Makefile | 3 +++ arch/riscv/include/asm/cmpxchg.h | 34 ++++++++++++++++++++++++++++++-- 3 files changed, 52 insertions(+), 2 deletions(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 1caaedec88c7..d3b0f92f92da 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -596,6 +596,23 @@ config RISCV_ISA_V_PREEMPTIVE preemption. Enabling this config will result in higher memory consumption due to the allocation of per-task's kernel Vector context. +config TOOLCHAIN_HAS_ZABHA + bool + default y + depends on !64BIT || $(cc-option,-mabi=lp64 -march=rv64ima_zabha) + depends on !32BIT || $(cc-option,-mabi=ilp32 -march=rv32ima_zabha) + depends on AS_HAS_OPTION_ARCH + +config RISCV_ISA_ZABHA + bool "Zabha extension support for atomic byte/halfword operations" + depends on TOOLCHAIN_HAS_ZABHA + default y + help + Enable the use of the Zabha ISA-extension to implement kernel + byte/halfword atomic memory operations when it is detected at boot. + + If you don't know what to do here, say Y. + config TOOLCHAIN_HAS_ZACAS bool default y diff --git a/arch/riscv/Makefile b/arch/riscv/Makefile index 9fd13d7a9cc6..78dcaaeebf4e 100644 --- a/arch/riscv/Makefile +++ b/arch/riscv/Makefile @@ -88,6 +88,9 @@ riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZIHINTPAUSE) := $(riscv-march-y)_zihintpause # Check if the toolchain supports Zacas riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZACAS) := $(riscv-march-y)_zacas +# Check if the toolchain supports Zabha +riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZABHA) := $(riscv-march-y)_zabha + # Remove F,D,V from isa string for all. Keep extensions between "fd" and "v" by # matching non-v and non-multi-letter extensions out with the filter ([^v_]*) KBUILD_CFLAGS += -march=$(shell echo $(riscv-march-y) | sed -E 's/(rv32ima|rv64ima)fd([^v_]*)v?/\1\2/') diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index a58a2141c6d3..b9a3fdcec919 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -105,8 +105,20 @@ * indicated by comparing RETURN with OLD. */ -#define __arch_cmpxchg_masked(sc_sfx, prepend, append, r, p, o, n) \ +#define __arch_cmpxchg_masked(sc_sfx, cas_sfx, prepend, append, r, p, o, n) \ ({ \ + __label__ no_zacas, zabha, end; \ + \ + if (IS_ENABLED(CONFIG_RISCV_ISA_ZABHA)) { \ + asm goto(ALTERNATIVE("j %[no_zacas]", "nop", 0, \ + RISCV_ISA_EXT_ZACAS, 1) \ + : : : : no_zacas); \ + asm goto(ALTERNATIVE("nop", "j %[zabha]", 0, \ + RISCV_ISA_EXT_ZABHA, 1) \ + : : : : zabha); \ + } \ + \ +no_zacas:; \ u32 *__ptr32b = (u32 *)((ulong)(p) & ~0x3); \ ulong __s = ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \ ulong __mask = GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \ @@ -133,6 +145,19 @@ : "memory"); \ \ r = (__typeof__(*(p)))((__retx & __mask) >> __s); \ + goto end; \ + \ +zabha: \ + if (IS_ENABLED(CONFIG_RISCV_ISA_ZABHA)) { \ + __asm__ __volatile__ ( \ + prepend \ + " amocas" cas_sfx " %0, %z2, %1\n" \ + append \ + : "+&r" (r), "+A" (*(p)) \ + : "rJ" (n) \ + : "memory"); \ + } \ +end:; \ }) #define __arch_cmpxchg(lr_sfx, sc_cas_sfx, prepend, append, r, p, co, o, n) \ @@ -181,8 +206,13 @@ end:; \ \ switch (sizeof(*__ptr)) { \ case 1: \ + __arch_cmpxchg_masked(sc_sfx, ".b" sc_sfx, \ + prepend, append, \ + __ret, __ptr, __old, __new); \ + break; \ case 2: \ - __arch_cmpxchg_masked(sc_sfx, prepend, append, \ + __arch_cmpxchg_masked(sc_sfx, ".h" sc_sfx, \ + prepend, append, \ __ret, __ptr, __old, __new); \ break; \ case 4: \ From patchwork Wed Jun 26 13:03:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13712913 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7AC49C27C4F for ; Wed, 26 Jun 2024 13:08:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=PAfz3aVxtqKfh7jjAE6HNaks7WIFZXxndzG3g1ZKraE=; b=H0PjCNSEig3sQ9 wzziZKGU1fvu118QZAPUW5pYsDLy1RMvmq5OU5orJ5aN3ko8B/WI/GYKUorFQi7c5eJsT/9I22rDJ dmORvPPkIDFwOpKD0xBosfM8IXUzkMMAoPLJRNO33WqZXQQM3Mar7pf7qgajDsMvG9JoJk337APP5 sr9kiauLHbEpHD/EjQKdmQmG6/IWZDvm/ucAP3OoY3aDvIiTyJNTLUQ/5evox+0B789K3wN2Y+bEb ZQ8UIx9RpPMgmzgUvqnQCKDBNuuckwEv3feaGWG00+zgVKUPo4chnEsQF1DwdKu9nioWIbL1yQPaW 5Q3m6wY3DdA1v/hGKS0Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sMSOB-00000006sa7-2uHK; Wed, 26 Jun 2024 13:08:43 +0000 Received: from mail-lj1-x229.google.com ([2a00:1450:4864:20::229]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sMSNS-00000006s9u-3La0 for linux-riscv@lists.infradead.org; Wed, 26 Jun 2024 13:08:00 +0000 Received: by mail-lj1-x229.google.com with SMTP id 38308e7fff4ca-2ebe0a81dc8so88816481fa.2 for ; Wed, 26 Jun 2024 06:07:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1719407276; x=1720012076; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6Y2lg3OgYzhfS3tcxdHdgy6tUi+qEwW8C3VWq7VXFkI=; b=Q3n/gBc6igU5gyBMlZCPjcTkUAyWe4Yx/oCgo13ia1y58alAdlYJjtjmLYHRq8L8dJ 9P2uIwjdIY2fNfGz05PHxrbNPob2MhvoCKD8+iKI3P+eiHAgfpQLRCf0Yw3UkHu8tCZR fyZZoOnXyKe9fXCqYAuDEKm6mkTxK1Db+z9/7k2nd0LunLBDZlbZKx8UFb28IfuRP5fG +kTEF3G29k/0sMiV80Yco6wkXGeMZRi+3J3nq6GeIvUlKIoXgYk1d3kUSNqTrYPMakcl eOyxeyduftpiB/PIkWVk0Xru654j88tGkg3K70KhEMxn7XSzrDbCj698RtpqZvja+dK1 fNpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719407276; x=1720012076; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6Y2lg3OgYzhfS3tcxdHdgy6tUi+qEwW8C3VWq7VXFkI=; b=tw4qp/mE6xi0JuKDEvLsYm7hUjXaNdghGWMEL44X2Q5zip795qEHd3iImpPrOC1cKP aS/dsXhbpIjJ/vIuXFVMDjTNJd/H96tysB21FsalmK7egOQJLNQN3lLNPAP3vU8GlqTw WvhWOe6SyBODmrhBRnB99fVDq2o8+FwPxvFq9ayuwnkBqRvzeO7DayY8C1GS9CYovTGs QXtdCS7qJqKc5e+F/+KXHW/WBKwXLtv9PBTeYoC/XDCT5Bf14KGytqm0oUtr/4DvZERt i/UIWbPMK9kdRfQPuU3IRexge5WQyCxuxZ1SjhGo+I+P6cX2CW0qq71iFTFj1vf5X0vm me6g== X-Forwarded-Encrypted: i=1; AJvYcCXrROdVOEcUGhneLyxplBX0tfj7WHvZDz2EynhTWSzq9GVVucegpAwvg24X+hdRlESroYHhmSsslouCy2UFQyJHOeOkvjogqpQRV4tQJzxG X-Gm-Message-State: AOJu0Yw2zz6u86hU54gUl1o3Y4r/rcJ7x7pJr3EdYwfQiomxV9KiBBjg uJwishDM1B+G/j8BWWJvFMEeBYDwOydsHKgGXdU2kZBvqHWOvJDTqiJAbl+xQSE= X-Google-Smtp-Source: AGHT+IFibXerpu/bTQ8y3LbVZ+LSuPoGA9fjYKjk5jwlDwlIOb8M9rkEFeIslajkZZ7g7g2gAUfMJA== X-Received: by 2002:a2e:720c:0:b0:2ec:4e79:b416 with SMTP id 38308e7fff4ca-2ec5b2fc299mr74660131fa.6.1719407276560; Wed, 26 Jun 2024 06:07:56 -0700 (PDT) Received: from localhost.localdomain (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-424c838bef4sm24669765e9.46.2024.06.26.06.07.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Jun 2024 06:07:56 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti , Andrea Parri Subject: [PATCH v2 04/10] riscv: Improve amocas.X use in cmpxchg() Date: Wed, 26 Jun 2024 15:03:41 +0200 Message-Id: <20240626130347.520750-5-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240626130347.520750-1-alexghiti@rivosinc.com> References: <20240626130347.520750-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240626_060758_867735_9E8E2027 X-CRM114-Status: UNSURE ( 9.84 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org cmpxchg() uses amocas.X instructions from Zacas and Zabha but still uses the LR/SC acquire/release semantics which require barriers. Let's improve that by using proper amocas acquire/release semantics in order to avoid any of those barriers. Suggested-by: Andrea Parri Signed-off-by: Alexandre Ghiti --- arch/riscv/include/asm/cmpxchg.h | 60 ++++++++++++++++++-------------- 1 file changed, 33 insertions(+), 27 deletions(-) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index b9a3fdcec919..3c65b00a0d36 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -105,7 +105,9 @@ * indicated by comparing RETURN with OLD. */ -#define __arch_cmpxchg_masked(sc_sfx, cas_sfx, prepend, append, r, p, o, n) \ +#define __arch_cmpxchg_masked(sc_sfx, cas_sfx, \ + sc_prepend, sc_append, \ + r, p, o, n) \ ({ \ __label__ no_zacas, zabha, end; \ \ @@ -129,7 +131,7 @@ no_zacas:; \ ulong __rc; \ \ __asm__ __volatile__ ( \ - prepend \ + sc_prepend \ "0: lr.w %0, %2\n" \ " and %1, %0, %z5\n" \ " bne %1, %z3, 1f\n" \ @@ -137,7 +139,7 @@ no_zacas:; \ " or %1, %1, %z4\n" \ " sc.w" sc_sfx " %1, %1, %2\n" \ " bnez %1, 0b\n" \ - append \ + sc_append \ "1:\n" \ : "=&r" (__retx), "=&r" (__rc), "+A" (*(__ptr32b)) \ : "rJ" ((long)__oldx), "rJ" (__newx), \ @@ -150,9 +152,7 @@ no_zacas:; \ zabha: \ if (IS_ENABLED(CONFIG_RISCV_ISA_ZABHA)) { \ __asm__ __volatile__ ( \ - prepend \ " amocas" cas_sfx " %0, %z2, %1\n" \ - append \ : "+&r" (r), "+A" (*(p)) \ : "rJ" (n) \ : "memory"); \ @@ -160,7 +160,9 @@ zabha: \ end:; \ }) -#define __arch_cmpxchg(lr_sfx, sc_cas_sfx, prepend, append, r, p, co, o, n) \ +#define __arch_cmpxchg(lr_sfx, sc_sfx, cas_sfx, \ + sc_prepend, sc_append, \ + r, p, co, o, n) \ ({ \ __label__ zacas, end; \ register unsigned int __rc; \ @@ -172,12 +174,12 @@ end:; \ } \ \ __asm__ __volatile__ ( \ - prepend \ + sc_prepend \ "0: lr" lr_sfx " %0, %2\n" \ " bne %0, %z3, 1f\n" \ - " sc" sc_cas_sfx " %1, %z4, %2\n" \ + " sc" sc_sfx " %1, %z4, %2\n" \ " bnez %1, 0b\n" \ - append \ + sc_append \ "1:\n" \ : "=&r" (r), "=&r" (__rc), "+A" (*(p)) \ : "rJ" (co o), "rJ" (n) \ @@ -187,9 +189,7 @@ end:; \ zacas: \ if (IS_ENABLED(CONFIG_RISCV_ISA_ZACAS)) { \ __asm__ __volatile__ ( \ - prepend \ - " amocas" sc_cas_sfx " %0, %z2, %1\n" \ - append \ + " amocas" cas_sfx " %0, %z2, %1\n" \ : "+&r" (r), "+A" (*(p)) \ : "rJ" (n) \ : "memory"); \ @@ -197,7 +197,8 @@ zacas: \ end:; \ }) -#define _arch_cmpxchg(ptr, old, new, sc_sfx, prepend, append) \ +#define _arch_cmpxchg(ptr, old, new, sc_sfx, cas_sfx, \ + sc_prepend, sc_append) \ ({ \ __typeof__(ptr) __ptr = (ptr); \ __typeof__(*(__ptr)) __old = (old); \ @@ -206,22 +207,24 @@ end:; \ \ switch (sizeof(*__ptr)) { \ case 1: \ - __arch_cmpxchg_masked(sc_sfx, ".b" sc_sfx, \ - prepend, append, \ - __ret, __ptr, __old, __new); \ + __arch_cmpxchg_masked(sc_sfx, ".b" cas_sfx, \ + sc_prepend, sc_append, \ + __ret, __ptr, __old, __new); \ break; \ case 2: \ - __arch_cmpxchg_masked(sc_sfx, ".h" sc_sfx, \ - prepend, append, \ - __ret, __ptr, __old, __new); \ + __arch_cmpxchg_masked(sc_sfx, ".h" cas_sfx, \ + sc_prepend, sc_append, \ + __ret, __ptr, __old, __new); \ break; \ case 4: \ - __arch_cmpxchg(".w", ".w" sc_sfx, prepend, append, \ - __ret, __ptr, (long), __old, __new); \ + __arch_cmpxchg(".w", ".w" sc_sfx, ".w" cas_sfx, \ + sc_prepend, sc_append, \ + __ret, __ptr, (long), __old, __new); \ break; \ case 8: \ - __arch_cmpxchg(".d", ".d" sc_sfx, prepend, append, \ - __ret, __ptr, /**/, __old, __new); \ + __arch_cmpxchg(".d", ".d" sc_sfx, ".d" cas_sfx, \ + sc_prepend, sc_append, \ + __ret, __ptr, /**/, __old, __new); \ break; \ default: \ BUILD_BUG(); \ @@ -230,16 +233,19 @@ end:; \ }) #define arch_cmpxchg_relaxed(ptr, o, n) \ - _arch_cmpxchg((ptr), (o), (n), "", "", "") + _arch_cmpxchg((ptr), (o), (n), "", "", "", "") #define arch_cmpxchg_acquire(ptr, o, n) \ - _arch_cmpxchg((ptr), (o), (n), "", "", RISCV_ACQUIRE_BARRIER) + _arch_cmpxchg((ptr), (o), (n), "", ".aq", \ + "", RISCV_ACQUIRE_BARRIER) #define arch_cmpxchg_release(ptr, o, n) \ - _arch_cmpxchg((ptr), (o), (n), "", RISCV_RELEASE_BARRIER, "") + _arch_cmpxchg((ptr), (o), (n), "", ".rl", \ + RISCV_RELEASE_BARRIER, "") #define arch_cmpxchg(ptr, o, n) \ - _arch_cmpxchg((ptr), (o), (n), ".rl", "", " fence rw, rw\n") + _arch_cmpxchg((ptr), (o), (n), ".rl", ".aqrl", \ + "", RISCV_FULL_BARRIER) #define arch_cmpxchg_local(ptr, o, n) \ arch_cmpxchg_relaxed((ptr), (o), (n)) From patchwork Wed Jun 26 13:03:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13712914 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A6899C27C4F for ; Wed, 26 Jun 2024 13:09:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=0Ngcgbrwbihv+GVAmJF88iCyZVBxECAY15YUW3Je8Ug=; b=LXRz7NXnnZKZN7 o6ZzJyPVPMcRv9lqrqtOtOe4m/x/UOSAKl0z/TPIFUdl/uH2YPmEmkNIp4SpguO0Aa+yWkq6zDSvC 2/QRDYD7Mb0QGfMfkgKvgaCwpEQF5r55OTCmETOV1fhatbAWGUqorOlyg4n8XdyvINr4qIm6sPq02 ZES5B+A0iSj9/gHrksWEoylZRfZj1uvKjN71ak4YiMG1fhAP4Nay4PCBxJUOrfbxkPmWERZSIDqy7 wwERevNmZuLa4fDpg9DJTf6OVdpCYwErDr+Cq9vw6hi+9aUReQczt+8TpZnjCvSbSXtU8rSnWLcsu fRWxJXUngH2h45yRrFHA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sMSOo-00000006sxI-32xw; Wed, 26 Jun 2024 13:09:24 +0000 Received: from mail-lj1-x230.google.com ([2a00:1450:4864:20::230]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sMSOR-00000006skU-3kba for linux-riscv@lists.infradead.org; Wed, 26 Jun 2024 13:09:02 +0000 Received: by mail-lj1-x230.google.com with SMTP id 38308e7fff4ca-2ec50a5e230so50126081fa.0 for ; Wed, 26 Jun 2024 06:08:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1719407338; x=1720012138; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DAwsmjkuECiz/mPkaZn9ympydsm9zMlHdoAoEYhhRo8=; b=vFzOWgQRLVhB7VP7Oo0CkvMVdw5ZGeQ0WL7P7jLB5tXTnmr0ZyEhCEp9Y+18cY8Vf1 7Yyk7eqEqC+toaDKKuZmrCYYJx869qnlInxjoIfDN8FuKoERRyCRhlGGBp/ash94fYd6 BU0KdiobHbdpVpYnN2Z9jK6pZtn4X9JMeuQnOHswaGFXPOy0KUIFN655Vc6fPaPguqqF GpxzmINTiLS8Urtnbr1i4xVVZ61a5Fr/Pwj8jCNmKAAdv5x7TftvEAqzfMuiAF7edql2 V6Rpqkn3+MEkJrBmflG3sLpESPRYxHioJRhjb3zaSAbDuAe9hRSznXegVPolodv70M0X 4bCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719407338; x=1720012138; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DAwsmjkuECiz/mPkaZn9ympydsm9zMlHdoAoEYhhRo8=; b=BVHkFkoBdAB7RjaFZpLajV4apJSHxwObemCJNEO+kjWMChWuFHTl48RV4Tk+rH8py5 toNTPk55N5LbE75iWSdHVUUk6FCj6JuAztXK1V8PEViyXBv8QQmSY7xhjxq/v9v2t/6+ I7fGFkRydcl9xqw6s/KLMJBHXAdcPVBzHDK1ItZ7mbg5QXSP5i+89iXIUM+Dh/9LsNkt Va6aeDGRaj6tDhsVoISGKCcaaeH8nC+DgYxhrQbXYz8xm656RG7VU6TepLEh1bXq9Hhv DztLgxLFN8SxHl/gW5zWpNGCdUqrLadqdpr1yfHAXoDYdAhV2Hnuwz432LqWaCTPvS6a MZBA== X-Forwarded-Encrypted: i=1; AJvYcCXOFmITLM1/TEunqpgdS18w7BalvP9fdk+6mtY/qwnKL8w/waem+96t16b0jBBfEGzyU1Nylzxq39SED7dRZO904o0qHUqdgMFe2utxzJEe X-Gm-Message-State: AOJu0Ywpk2ymT5EmpolEbFSSNPsdbwzRkrARSywiH+ZL0Xj1QILiQW08 PO/g7cwjK5Oz+8nnHcTxfOnBJN8mXWWIT96Wcrxm5qp02WrIjl1Y3a+5yJVEiWE= X-Google-Smtp-Source: AGHT+IE6IYtOzpJIWWby5oBtItvoKm+O8WuyMECCoA3dQcwC4BqVw/peghmNdO01PLv674LqbGf4lw== X-Received: by 2002:a05:6512:6ce:b0:52d:b1bb:370 with SMTP id 2adb3069b0e04-52db1bb0490mr1964462e87.23.1719407337732; Wed, 26 Jun 2024 06:08:57 -0700 (PDT) Received: from localhost.localdomain (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-424c8266bd5sm25627705e9.26.2024.06.26.06.08.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Jun 2024 06:08:57 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH v2 05/10] riscv: Implement arch_cmpxchg128() using Zacas Date: Wed, 26 Jun 2024 15:03:42 +0200 Message-Id: <20240626130347.520750-6-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240626130347.520750-1-alexghiti@rivosinc.com> References: <20240626130347.520750-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240626_060900_294635_54433A6D X-CRM114-Status: GOOD ( 12.72 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Now that Zacas is supported in the kernel, let's use the double word atomic version of amocas to improve the SLUB allocator. Note that we have to select fixed registers, otherwise gcc fails to pick even registers and then produces a reserved encoding which fails to assemble. Signed-off-by: Alexandre Ghiti --- arch/riscv/Kconfig | 1 + arch/riscv/include/asm/cmpxchg.h | 39 ++++++++++++++++++++++++++++++++ 2 files changed, 40 insertions(+) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index d3b0f92f92da..0bbaec0444d0 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -104,6 +104,7 @@ config RISCV select GENERIC_VDSO_TIME_NS if HAVE_GENERIC_VDSO select HARDIRQS_SW_RESEND select HAS_IOPORT if MMU + select HAVE_ALIGNED_STRUCT_PAGE select HAVE_ARCH_AUDITSYSCALL select HAVE_ARCH_HUGE_VMALLOC if HAVE_ARCH_HUGE_VMAP select HAVE_ARCH_HUGE_VMAP if MMU && 64BIT diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index 3c65b00a0d36..da42f32ea53d 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -280,4 +280,43 @@ end:; \ arch_cmpxchg_release((ptr), (o), (n)); \ }) +#ifdef CONFIG_RISCV_ISA_ZACAS + +#define system_has_cmpxchg128() \ + riscv_has_extension_unlikely(RISCV_ISA_EXT_ZACAS) + +union __u128_halves { + u128 full; + struct { + u64 low, high; + }; +}; + +#define __arch_cmpxchg128(p, o, n, cas_sfx) \ +({ \ + __typeof__(*(p)) __o = (o); \ + union __u128_halves __hn = { .full = (n) }; \ + union __u128_halves __ho = { .full = (__o) }; \ + register unsigned long x6 asm ("x6") = __hn.low; \ + register unsigned long x7 asm ("x7") = __hn.high; \ + register unsigned long x28 asm ("x28") = __ho.low; \ + register unsigned long x29 asm ("x29") = __ho.high; \ + \ + __asm__ __volatile__ ( \ + " amocas.q" cas_sfx " %0, %z3, %2" \ + : "+&r" (x28), "+&r" (x29), "+A" (*(p)) \ + : "rJ" (x6), "rJ" (x7) \ + : "memory"); \ + \ + ((u128)x29 << 64) | x28; \ +}) + +#define arch_cmpxchg128(ptr, o, n) \ + __arch_cmpxchg128((ptr), (o), (n), ".aqrl") + +#define arch_cmpxchg128_local(ptr, o, n) \ + __arch_cmpxchg128((ptr), (o), (n), "") + +#endif /* CONFIG_RISCV_ISA_ZACAS */ + #endif /* _ASM_RISCV_CMPXCHG_H */ From patchwork Wed Jun 26 13:03:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13712915 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 898FEC27C4F for ; Wed, 26 Jun 2024 13:10:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=SMAXP8hyeYzicbev34/xXIrxkH/esWdbEfOkE5rmvKA=; b=u+4w9G1y03Hxoo Lrx65rVLhSR37GnGVYTWCFaZxHGDXEm0R1nWKdMf4rhP727lWIyFbQF8VEzLK/NCrUbCpAmA8ik8/ WPEwOGllrif9Mki3zaBQUpGLebDIDnzrS2Z7e7GfcQwoRGycq1+UuFDtk6biESHjC+oxHoQZTszie eKiMz5EbNEtTxJQRXnLiQ/pOlzt9gpUo0OFvvu4EdUkr55lNDrcEqydZxmEPKdOr0//gbZA4oTMb9 aDDrmc2Z6rU0UPnTipX1j+kMwmSnpopDNAcEipCDYSfZhshWYe/g7SqQWBf8YfTtdIJgfvjY9mPct Udeu75InWxb/ARX2WooA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sMSPf-00000006tHR-3wEB; Wed, 26 Jun 2024 13:10:16 +0000 Received: from mail-wm1-x332.google.com ([2a00:1450:4864:20::332]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sMSPQ-00000006tBi-3NDs for linux-riscv@lists.infradead.org; Wed, 26 Jun 2024 13:10:02 +0000 Received: by mail-wm1-x332.google.com with SMTP id 5b1f17b1804b1-42561715d41so1611675e9.0 for ; Wed, 26 Jun 2024 06:10:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1719407399; x=1720012199; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QWrbp7YnSuSg4cTyCRLd07oshRok5ClJY3Zre9Kj0ww=; b=KCBzNdLJwrPZDLWKM6xMj9wE5nWZiySKkpKpcMrbEIDJr0pTvnkt+qbfnDQqavZSNY 7dH6Bz3lVpJEVTSvu6BhtNgIQxwUJquKdCzeprXLybI5VkzJIn0P1g2hhPLkBRg5fvw+ /Vi9R9vI5xEnb4k4eh/dHlRW2X7dLXTei2OjM4EJWPCzAqcdTFJwDdc1SbyL7on1RTV8 TN1Z8mtYdxlYq35O3lI1zraGrFf1pu9YpxxFqs0oiuym9k04LKeXkXFvAWGr7dgKCOXG 5kGv/0RQfADjnEeO1R7scdVCZUC6ephKOxsv/S4pGzei1Gu1akeDdyZR46lh8lV8oAuX F4fA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719407399; x=1720012199; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QWrbp7YnSuSg4cTyCRLd07oshRok5ClJY3Zre9Kj0ww=; b=GqX7FJ3bfAA63cx6XGkzVo+uMafDiZugOWgNPt/87LnRYn0xE0cG3SCyDU5pEOwANE +QJ114GH34yUnsOW62B5uU0YC84kOA1+WECzzfBsv/dSaqFtP/OrJa0meqAtdsEDA8HN az9bMmGh9BdFnf1xZsfJiMc+e4imFhwD+/iUnswUOvw8rD4AdGnCKcv56VsKg9Un3cUL rdCRFCdG9UWlHBu5h2WXeoKVFFZrpsoHAIZVfbxxjY8KhI4ofuQDIuviEn0TpG+R/F6z xPBrUmJf5t7yrGLb/s/vxnHxjMQF03atkMNZnsPkU7/uZh5KEcnHVNYAB75rt6akKFVd QSow== X-Forwarded-Encrypted: i=1; AJvYcCXXScutj0U9BDyiCSDMQJiggtVMhNXn5m182CQ4yK8FccQmfp568MewsUzAYh9pt60vsqsrwQsRoAg5oemaAEPuP9R8RMz0+0H6V8JCF81c X-Gm-Message-State: AOJu0Yx+6PkGRHv+8Mib2fidclB+dod3Pon/abRKZNabhlxzdnwfCe5U Fc9x5uffI9zE63c4qI+uDSExacYnF/K51e3Xt1tDzmk967NrMZkMCJKs3bIKcIc= X-Google-Smtp-Source: AGHT+IF1O9RqB95BpuZCH8OUjPkY7BW4xVu+M+7CZDEbvz5gp1RkR9X5IwXEN5r4vgMt231tIFpnRA== X-Received: by 2002:a05:6000:1c4:b0:367:3404:1c06 with SMTP id ffacd0b85a97d-36734041cecmr473355f8f.20.1719407398822; Wed, 26 Jun 2024 06:09:58 -0700 (PDT) Received: from localhost.localdomain (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3663ada0bdesm15712197f8f.113.2024.06.26.06.09.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Jun 2024 06:09:58 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH v2 06/10] riscv: Implement xchg8/16() using Zabha Date: Wed, 26 Jun 2024 15:03:43 +0200 Message-Id: <20240626130347.520750-7-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240626130347.520750-1-alexghiti@rivosinc.com> References: <20240626130347.520750-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240626_061000_923104_CC4DAA11 X-CRM114-Status: GOOD ( 12.06 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org This adds runtime support for Zabha in xchg8/16() operations. Signed-off-by: Alexandre Ghiti --- arch/riscv/include/asm/cmpxchg.h | 33 +++++++++++++++++++++++++++++--- arch/riscv/include/asm/hwcap.h | 1 + arch/riscv/kernel/cpufeature.c | 1 + 3 files changed, 32 insertions(+), 3 deletions(-) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index da42f32ea53d..eb35e2d30a97 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -11,8 +11,17 @@ #include #include -#define __arch_xchg_masked(sc_sfx, prepend, append, r, p, n) \ +#define __arch_xchg_masked(sc_sfx, swap_sfx, prepend, sc_append, \ + swap_append, r, p, n) \ ({ \ + __label__ zabha, end; \ + \ + if (IS_ENABLED(CONFIG_RISCV_ISA_ZABHA)) { \ + asm goto(ALTERNATIVE("nop", "j %[zabha]", 0, \ + RISCV_ISA_EXT_ZABHA, 1) \ + : : : : zabha); \ + } \ + \ u32 *__ptr32b = (u32 *)((ulong)(p) & ~0x3); \ ulong __s = ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \ ulong __mask = GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \ @@ -28,12 +37,25 @@ " or %1, %1, %z3\n" \ " sc.w" sc_sfx " %1, %1, %2\n" \ " bnez %1, 0b\n" \ - append \ + sc_append \ : "=&r" (__retx), "=&r" (__rc), "+A" (*(__ptr32b)) \ : "rJ" (__newx), "rJ" (~__mask) \ : "memory"); \ \ r = (__typeof__(*(p)))((__retx & __mask) >> __s); \ + goto end; \ + \ +zabha: \ + if (IS_ENABLED(CONFIG_RISCV_ISA_ZABHA)) { \ + __asm__ __volatile__ ( \ + prepend \ + " amoswap" swap_sfx " %0, %z2, %1\n" \ + swap_append \ + : "=&r" (r), "+A" (*(p)) \ + : "rJ" (n) \ + : "memory"); \ + } \ +end:; \ }) #define __arch_xchg(sfx, prepend, append, r, p, n) \ @@ -56,8 +78,13 @@ \ switch (sizeof(*__ptr)) { \ case 1: \ + __arch_xchg_masked(sc_sfx, ".b" swap_sfx, \ + prepend, sc_append, swap_append, \ + __ret, __ptr, __new); \ + break; \ case 2: \ - __arch_xchg_masked(sc_sfx, prepend, sc_append, \ + __arch_xchg_masked(sc_sfx, ".h" swap_sfx, \ + prepend, sc_append, swap_append, \ __ret, __ptr, __new); \ break; \ case 4: \ diff --git a/arch/riscv/include/asm/hwcap.h b/arch/riscv/include/asm/hwcap.h index e17d0078a651..f71ddd2ca163 100644 --- a/arch/riscv/include/asm/hwcap.h +++ b/arch/riscv/include/asm/hwcap.h @@ -81,6 +81,7 @@ #define RISCV_ISA_EXT_ZTSO 72 #define RISCV_ISA_EXT_ZACAS 73 #define RISCV_ISA_EXT_XANDESPMU 74 +#define RISCV_ISA_EXT_ZABHA 75 #define RISCV_ISA_EXT_XLINUXENVCFG 127 diff --git a/arch/riscv/kernel/cpufeature.c b/arch/riscv/kernel/cpufeature.c index 5ef48cb20ee1..c125d82c894b 100644 --- a/arch/riscv/kernel/cpufeature.c +++ b/arch/riscv/kernel/cpufeature.c @@ -257,6 +257,7 @@ const struct riscv_isa_ext_data riscv_isa_ext[] = { __RISCV_ISA_EXT_DATA(zihintpause, RISCV_ISA_EXT_ZIHINTPAUSE), __RISCV_ISA_EXT_DATA(zihpm, RISCV_ISA_EXT_ZIHPM), __RISCV_ISA_EXT_DATA(zacas, RISCV_ISA_EXT_ZACAS), + __RISCV_ISA_EXT_DATA(zabha, RISCV_ISA_EXT_ZABHA), __RISCV_ISA_EXT_DATA(zfa, RISCV_ISA_EXT_ZFA), __RISCV_ISA_EXT_DATA(zfh, RISCV_ISA_EXT_ZFH), __RISCV_ISA_EXT_DATA(zfhmin, RISCV_ISA_EXT_ZFHMIN), From patchwork Wed Jun 26 13:03:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13712916 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EC173C27C4F for ; Wed, 26 Jun 2024 13:11:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=xs4G8KbKzKlJbj7y9NGgCjChOv787yNFIEviursqOh8=; b=uwFpyFDt+82TKv dHXGuc0mL7WSQGQamlG/w1XkF7GfjHxgkcylGi6/oODvAS3zAIBaoJ8GXBpVwwCJI6rCzmQTPIpJ+ YxALQ9PqQRlUXwC/grpvg1ln7X0UV0onnq9Ak3nxAHwzWcFYBng+NAa25CLzoOPfIetlLWggAUv+P 7NRuUW7P+Oii+eXUgW7QCc5otCX3K6Dbn/zt7rfc4YHMXf7wlUeVM6gAbdbYfxZR9ycdQ/3uBFA1/ DMpiqRWbmDiUmkqKEE66SsUxDUDwkCNnNsxvJ3IKLKax6WRmwEnoEno9cAo+4wiqN8/tiylC85eRC 4EjmTsvE/pYrA/0yQs3w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sMSQa-00000006tYN-0BB0; Wed, 26 Jun 2024 13:11:12 +0000 Received: from mail-lf1-x132.google.com ([2a00:1450:4864:20::132]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sMSQQ-00000006tUW-0WBw for linux-riscv@lists.infradead.org; Wed, 26 Jun 2024 13:11:03 +0000 Received: by mail-lf1-x132.google.com with SMTP id 2adb3069b0e04-52caebc6137so6355531e87.0 for ; Wed, 26 Jun 2024 06:11:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1719407460; x=1720012260; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=p6cx0+kPy/mr4m8koR629ifIafAeU6/dHlLYYLSwNIk=; b=yjhZ6SLQuff+D61fZNaJ5telqSrK2vjZ9W9frNZ0k2J6HPfdDvlAWlRUH/g2SOXdwN kb0Q9iT+wGYlWnCEYXmsTsUcorxI84jSWjvbo3BhUQZjEovFdmvnOdiyiTxsnPiHyZ6s yPJCqyBSZ5eopnc3syIg+V8b5zYt4Le2IkrhzjMpxupLj9jyu6ii9dukkgJlT5hUJr7b rGc8hgMSlzl4YmDsBNzAXcvlmmcNCXGN4z5Ocx/4+eP0XkTOYQb/btIE9lY6Z3fnsU+9 JQKJwTTz2YM1gPxBuFI0T7nohyfx/q6UM0KQemcLRFgbrOqtJSoFzgmaTAhEsChGiRz3 RPRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719407460; x=1720012260; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=p6cx0+kPy/mr4m8koR629ifIafAeU6/dHlLYYLSwNIk=; b=hQv6VKw9LYoTnQCD4nrTxix3AcGxJ5NogHC4dle9xpx1pbJMpwqiozwT2AwKzMNWmw GQASsUiS+OtWmdC4tC7VB4KZp4NKEWXV+ul7ncgZvbG6CUJAZrN2X52ICQPVP0q2xUFi zIt/Crc3mdDJWq5N6LhXYLkyRnEqn0iFv6YOHpOLCLDqM0qMA7Uxxm36wuNFYndW2B2V Q06kgKp4wk/NO6KymvuNIC9CDhsQKUC9CxcM1L/gnt8k1B818kgj1UCM3YHDAbzapbAs XyW9Tv6f1VqbAVYgJMzaHkpQCCX5jPlO68zXNFyA91831SNmNieDq4LJA+ETODeuorPH oBlg== X-Forwarded-Encrypted: i=1; AJvYcCXny7BofA9UaCUUfEvf1SXczTA+3T9mfRc4jtbqwdu9nsE6OxY54jc4YaM82wANR5iNQWQ/z8jx2NZKLdWpAQB60dRQLsi906IUigxvtob1 X-Gm-Message-State: AOJu0YzAgzxcJdqfwmj6NkjuXtEWO0bL8XLFSTcSSJQWRqJW/MWzS3By 2+oAuQj1RsjBBDLKD7Wl1YdQ8jTMaStAeBXNmG+5VV+8BpYus/cBZfTTiX4tf1Y= X-Google-Smtp-Source: AGHT+IEtNRZjWCdEzOhHydNYZbdtsSBOyhgctkoqWdBmAHXNA7jMkTIIYGJ6ebJAtMiVgYJ66wbTgg== X-Received: by 2002:a05:6512:1cd:b0:52c:dc57:868b with SMTP id 2adb3069b0e04-52ce18324ebmr7017333e87.13.1719407460052; Wed, 26 Jun 2024 06:11:00 -0700 (PDT) Received: from localhost.localdomain (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-424c824eef1sm24890125e9.14.2024.06.26.06.10.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Jun 2024 06:10:59 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti , Andrea Parri Subject: [PATCH v2 07/10] riscv: Improve amoswap.X use in xchg() Date: Wed, 26 Jun 2024 15:03:44 +0200 Message-Id: <20240626130347.520750-8-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240626130347.520750-1-alexghiti@rivosinc.com> References: <20240626130347.520750-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240626_061102_433809_156106B3 X-CRM114-Status: UNSURE ( 9.94 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org xchg() uses amoswap.X instructions from Zabha but still uses the LR/SC acquire/release semantics which require barriers. Let's improve that by using proper amoswap acquire/release semantics in order to avoid any of those barriers. Suggested-by: Andrea Parri Signed-off-by: Alexandre Ghiti --- arch/riscv/include/asm/cmpxchg.h | 35 +++++++++++++------------------- 1 file changed, 14 insertions(+), 21 deletions(-) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index eb35e2d30a97..0e57d5fbf227 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -11,8 +11,8 @@ #include #include -#define __arch_xchg_masked(sc_sfx, swap_sfx, prepend, sc_append, \ - swap_append, r, p, n) \ +#define __arch_xchg_masked(sc_sfx, swap_sfx, sc_prepend, sc_append, \ + r, p, n) \ ({ \ __label__ zabha, end; \ \ @@ -31,7 +31,7 @@ ulong __rc; \ \ __asm__ __volatile__ ( \ - prepend \ + sc_prepend \ "0: lr.w %0, %2\n" \ " and %1, %0, %z4\n" \ " or %1, %1, %z3\n" \ @@ -48,9 +48,7 @@ zabha: \ if (IS_ENABLED(CONFIG_RISCV_ISA_ZABHA)) { \ __asm__ __volatile__ ( \ - prepend \ " amoswap" swap_sfx " %0, %z2, %1\n" \ - swap_append \ : "=&r" (r), "+A" (*(p)) \ : "rJ" (n) \ : "memory"); \ @@ -58,19 +56,17 @@ zabha: \ end:; \ }) -#define __arch_xchg(sfx, prepend, append, r, p, n) \ +#define __arch_xchg(sfx, r, p, n) \ ({ \ __asm__ __volatile__ ( \ - prepend \ " amoswap" sfx " %0, %2, %1\n" \ - append \ : "=r" (r), "+A" (*(p)) \ : "r" (n) \ : "memory"); \ }) -#define _arch_xchg(ptr, new, sc_sfx, swap_sfx, prepend, \ - sc_append, swap_append) \ +#define _arch_xchg(ptr, new, sc_sfx, swap_sfx, \ + sc_prepend, sc_append) \ ({ \ __typeof__(ptr) __ptr = (ptr); \ __typeof__(*(__ptr)) __new = (new); \ @@ -79,21 +75,19 @@ end:; \ switch (sizeof(*__ptr)) { \ case 1: \ __arch_xchg_masked(sc_sfx, ".b" swap_sfx, \ - prepend, sc_append, swap_append, \ + sc_prepend, sc_append, \ __ret, __ptr, __new); \ break; \ case 2: \ __arch_xchg_masked(sc_sfx, ".h" swap_sfx, \ - prepend, sc_append, swap_append, \ + sc_prepend, sc_append, \ __ret, __ptr, __new); \ break; \ case 4: \ - __arch_xchg(".w" swap_sfx, prepend, swap_append, \ - __ret, __ptr, __new); \ + __arch_xchg(".w" swap_sfx, __ret, __ptr, __new); \ break; \ case 8: \ - __arch_xchg(".d" swap_sfx, prepend, swap_append, \ - __ret, __ptr, __new); \ + __arch_xchg(".d" swap_sfx, __ret, __ptr, __new); \ break; \ default: \ BUILD_BUG(); \ @@ -102,17 +96,16 @@ end:; \ }) #define arch_xchg_relaxed(ptr, x) \ - _arch_xchg(ptr, x, "", "", "", "", "") + _arch_xchg(ptr, x, "", "", "", "") #define arch_xchg_acquire(ptr, x) \ - _arch_xchg(ptr, x, "", "", "", \ - RISCV_ACQUIRE_BARRIER, RISCV_ACQUIRE_BARRIER) + _arch_xchg(ptr, x, "", ".aq", "", RISCV_ACQUIRE_BARRIER) #define arch_xchg_release(ptr, x) \ - _arch_xchg(ptr, x, "", "", RISCV_RELEASE_BARRIER, "", "") + _arch_xchg(ptr, x, "", ".rl", RISCV_RELEASE_BARRIER, "") #define arch_xchg(ptr, x) \ - _arch_xchg(ptr, x, ".rl", ".aqrl", "", RISCV_FULL_BARRIER, "") + _arch_xchg(ptr, x, ".rl", ".aqrl", "", RISCV_FULL_BARRIER) #define xchg32(ptr, x) \ ({ \ From patchwork Wed Jun 26 13:03:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13712917 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E3F92C27C4F for ; Wed, 26 Jun 2024 13:12:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=81l1vnndeH6RGQ7RI3IttTQT/Ue2c84FixhX6Y1lqPk=; b=rq9Z++FXMNB2+T JZhKYv5vnhJjyMdYimQiKC2ql+SFxrdQXqzoGk7gT8WCFOJt16u/Ji6oOh/kKPUAeeLXRO0s3392H JVorTBWBEZ0smYzp87Z4wxx5OkybsPGDnUmPuYDwt/7Okz1Orzex7aK7jqTeIc/ZPHODysHgW0x7T cUeWSV99vXUNOKBfE6e4ulaYhc6/CKVSVpMELMA4mVeFOoGl+EOAjzINEqMY7uKTKy6SrIIFpuCZH +XQbTh4TmiBvHHmxT1GO+e+gEJUMAmBqHm1tnI+NMO/PkgwgO6rhRiTDs8vw5Hcoxc3Q2udJTqh00 1YlbZA2ph2ymMp7eHDdQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sMSRR-00000006tqV-2aiu; Wed, 26 Jun 2024 13:12:05 +0000 Received: from mail-lj1-x232.google.com ([2a00:1450:4864:20::232]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sMSRP-00000006tot-0f6F for linux-riscv@lists.infradead.org; Wed, 26 Jun 2024 13:12:04 +0000 Received: by mail-lj1-x232.google.com with SMTP id 38308e7fff4ca-2ec52fbb50cso44701711fa.2 for ; Wed, 26 Jun 2024 06:12:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1719407521; x=1720012321; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=heAbIXv1oMx4SoDoWG1mJ8ALiEwmwn+mI3czPDKQjFY=; b=pExovZ6Itqdbox0oA++ejT20zLW5hQcpOQoBvl7IuR8L0ZVBn7nssWdFzwwYf9sUyL c0Nvq5/I5ZE7AmkWD/EE0bMi1qm7KbGfEm+075YY3T1oYWNwgozdarDbHvWIzRsqXfCL hmW14Di2hsd1a1M6sjeG8z+uD/S5Jphok0GfZyJDyKH+17mQP1VRAVJPCaf+DjbYluRj wTdR8JNupcVlqmep68UJrxWPYE5tEaaLTnqzSshKO8Yza7eyNRLW65v1vC9BoEkauqql y/3XowJZXLmo8q2BUZAoqw4Ur8OypyKarauruu/dovnmSMcPaaGyoPi+Ls7ABAAOQIr7 jOAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719407521; x=1720012321; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=heAbIXv1oMx4SoDoWG1mJ8ALiEwmwn+mI3czPDKQjFY=; b=t+ZmaBZC8GuBVg5cEkfDef5POGL/hyQFpCMGXgvbkcyNKSCw4w5vw4r8Uh/zu2O6M0 O3ErVEXaIRO72GVSeUSYFD26uqReQSM7qFJPzS/M8ssUY1Sk70AJOqhKn1/4m7zVQ0wk HA8dwchE9c+z/qPqPZXpr6ujjfPuHyUn4r0A6xrpYAXCE0m8Iaq6z5HtLxQ9vtKl8yjj 9q4pbXzWVvWTDUzdrCrAi3/bvmF/Mhwrh5R+2+0/UvWA1CNwIdJ8ObRHqMBnXBKNMyoQ W9Wo8lxV8tGpawPmHKGcxG0vIUgRptW5mF5UG32Ws/anHCFbwS4AmKOKnvg8okRyXbfV 0REQ== X-Forwarded-Encrypted: i=1; AJvYcCU0IrZdhU51R26kxXA8+HuKcVUh+sFN4LVkxgsej8vMuPGA0yUJJJ/sJzBs2tc65QDrAQDQzxFp2wHaDCnnIT4HTakfMWJ8iZnlUuLYfIuD X-Gm-Message-State: AOJu0Yx/Hc4M5/emte+QQ9Sl/zQvZOs2MoFCIJxC+EyCLS9rhN1pxrcm nMQoGswdIk1LnrYCLIVekmcIYbmyEnuPT9FYh/ahqTqxQW4Hf+isgryRLfn2db0= X-Google-Smtp-Source: AGHT+IGnfuN9yldtMSaVACEfklvq5QMnwW2TuArhVHZwcMBHnbeoMlc6OQU0v0zbHCs2zie7YJcaew== X-Received: by 2002:a2e:890d:0:b0:2eb:e258:717f with SMTP id 38308e7fff4ca-2ec5b2f0400mr62412471fa.42.1719407521185; Wed, 26 Jun 2024 06:12:01 -0700 (PDT) Received: from localhost.localdomain (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-424c84248afsm25143335e9.31.2024.06.26.06.12.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Jun 2024 06:12:00 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Guo Ren Subject: [PATCH v2 08/10] asm-generic: ticket-lock: Reuse arch_spinlock_t of qspinlock Date: Wed, 26 Jun 2024 15:03:45 +0200 Message-Id: <20240626130347.520750-9-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240626130347.520750-1-alexghiti@rivosinc.com> References: <20240626130347.520750-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240626_061203_255486_316338FF X-CRM114-Status: GOOD ( 13.16 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Guo Ren The arch_spinlock_t of qspinlock has contained the atomic_t val, which satisfies the ticket-lock requirement. Thus, unify the arch_spinlock_t into qspinlock_types.h. This is the preparation for the next combo spinlock. Reviewed-by: Leonardo Bras Suggested-by: Arnd Bergmann Link: https://lore.kernel.org/linux-riscv/CAK8P3a2rnz9mQqhN6-e0CGUUv9rntRELFdxt_weiD7FxH7fkfQ@mail.gmail.com/ Signed-off-by: Guo Ren Signed-off-by: Guo Ren --- include/asm-generic/spinlock.h | 14 +++++++------- include/asm-generic/spinlock_types.h | 12 ++---------- 2 files changed, 9 insertions(+), 17 deletions(-) diff --git a/include/asm-generic/spinlock.h b/include/asm-generic/spinlock.h index 90803a826ba0..4773334ee638 100644 --- a/include/asm-generic/spinlock.h +++ b/include/asm-generic/spinlock.h @@ -32,7 +32,7 @@ static __always_inline void arch_spin_lock(arch_spinlock_t *lock) { - u32 val = atomic_fetch_add(1<<16, lock); + u32 val = atomic_fetch_add(1<<16, &lock->val); u16 ticket = val >> 16; if (ticket == (u16)val) @@ -46,31 +46,31 @@ static __always_inline void arch_spin_lock(arch_spinlock_t *lock) * have no outstanding writes due to the atomic_fetch_add() the extra * orderings are free. */ - atomic_cond_read_acquire(lock, ticket == (u16)VAL); + atomic_cond_read_acquire(&lock->val, ticket == (u16)VAL); smp_mb(); } static __always_inline bool arch_spin_trylock(arch_spinlock_t *lock) { - u32 old = atomic_read(lock); + u32 old = atomic_read(&lock->val); if ((old >> 16) != (old & 0xffff)) return false; - return atomic_try_cmpxchg(lock, &old, old + (1<<16)); /* SC, for RCsc */ + return atomic_try_cmpxchg(&lock->val, &old, old + (1<<16)); /* SC, for RCsc */ } static __always_inline void arch_spin_unlock(arch_spinlock_t *lock) { u16 *ptr = (u16 *)lock + IS_ENABLED(CONFIG_CPU_BIG_ENDIAN); - u32 val = atomic_read(lock); + u32 val = atomic_read(&lock->val); smp_store_release(ptr, (u16)val + 1); } static __always_inline int arch_spin_value_unlocked(arch_spinlock_t lock) { - u32 val = lock.counter; + u32 val = lock.val.counter; return ((val >> 16) == (val & 0xffff)); } @@ -84,7 +84,7 @@ static __always_inline int arch_spin_is_locked(arch_spinlock_t *lock) static __always_inline int arch_spin_is_contended(arch_spinlock_t *lock) { - u32 val = atomic_read(lock); + u32 val = atomic_read(&lock->val); return (s16)((val >> 16) - (val & 0xffff)) > 1; } diff --git a/include/asm-generic/spinlock_types.h b/include/asm-generic/spinlock_types.h index 8962bb730945..f534aa5de394 100644 --- a/include/asm-generic/spinlock_types.h +++ b/include/asm-generic/spinlock_types.h @@ -3,15 +3,7 @@ #ifndef __ASM_GENERIC_SPINLOCK_TYPES_H #define __ASM_GENERIC_SPINLOCK_TYPES_H -#include -typedef atomic_t arch_spinlock_t; - -/* - * qrwlock_types depends on arch_spinlock_t, so we must typedef that before the - * include. - */ -#include - -#define __ARCH_SPIN_LOCK_UNLOCKED ATOMIC_INIT(0) +#include +#include #endif /* __ASM_GENERIC_SPINLOCK_TYPES_H */ From patchwork Wed Jun 26 13:03:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13712918 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 42567C27C4F for ; Wed, 26 Jun 2024 13:13:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Ch1fU7HxtxblHEnw2eK8E63qnmnxpbMDLgaMVzTElew=; b=R5tKKdv6E/5St9 F+EsDxr4xtJNNDO43+8HGnA7BoGpgZXppiC9TS4ZapYbr1e6b3xGoA0kmj7HnRctmZJmrJqmzGHHT Xy6DKR8Ak7bc4ByK9MgsXdtLvVcI7q3wgS6oVncdRg7LDInIFAVOpIAjoYms6xM36QfMzSlE4mM51 zUJQSJKW7LsZoWpEfzilnhfU9LQyF64SllTsng6jOQI+vh1BmNZWDhGySzOin+Xv4eIBCAY+biExK 1dZ6qtQDXwRyjgK8id5gbW7xK662tX6KMyVYMVxJhWwKwN3AwyzdkCqZspWS8+lkSFq0f/m5J9VGJ 8vrER4KvQbOvcgctvaHw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sMSSR-00000006u95-3unZ; Wed, 26 Jun 2024 13:13:09 +0000 Received: from mail-lf1-x12c.google.com ([2a00:1450:4864:20::12c]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sMSSO-00000006u7P-3m54 for linux-riscv@lists.infradead.org; Wed, 26 Jun 2024 13:13:06 +0000 Received: by mail-lf1-x12c.google.com with SMTP id 2adb3069b0e04-52ce655ff5bso3091078e87.3 for ; Wed, 26 Jun 2024 06:13:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1719407582; x=1720012382; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=JpMFH6VmSNUk91jIQ+0kjGx11FiPCz2ltKgHshZWUPg=; b=vfJgt6Z0anA8Igc4IpR79w8YnZycmVhsZs5ENlkIvWyuDPG32fAHbvwUa90z1LDOB6 pvJKagGAeuNdYdP+TsA/64DuX0QRfnhq7rLGBkKZ7h7yRCquDYVrcKj9R+kCKLDhNyW5 BwQ651gnVbPE+91eG3oUClwwYjQZUNrXE0vNj5JAGXstBZwZtndljToaXtBWcACZdXgu Ee1gNaBZaS41O2bw2HMSqzhlEMO1f3mTmwMaQgIvw1KhpU2cfZzsf7/xSJ1vQ7fOlu7c FhIW1GFf7zb5sS+Jd3nNhb6FJR79OoqwA0X2AALur9IkeK+R77fqyFdrl4a12sLvhdM2 i/aw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719407582; x=1720012382; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JpMFH6VmSNUk91jIQ+0kjGx11FiPCz2ltKgHshZWUPg=; b=qNFz2vYPw5EkdLVdOB2Gl4gaa7cyl1lGtMNbXV/+jTZVjNW40Ka8+HendDeEpa4rzP pQHpPkFlrk9gLmvNNS1OXHfS4deX+vVR9jNOD4WaG9M6RAIS5zyA7ZvJa4muth4y6XfZ A62/Cllc/9fqU5rxE7e5OC2DkiUi0RU20/gxUZ/z4W2YOjJWqJV6ALdGzMSd2qPMCRLX 5tFwh4Kx55WmeIghv4PRqq9MU+iONFqH59kDpkM21HYUYv0Q2bffPPY27cxTSZLtONUy 6XHyBmj8lk82Mg/+K1R4dGCdY838KGZ6GoKnHXsXCJIaqyCcHoFQqZWI8LPQbeUCLrN8 Bj2A== X-Forwarded-Encrypted: i=1; AJvYcCV5iQKuYGoz+ZTX1VvbTB0cLlgbVYGKYm/HiOvC+Qc4jUvz3bPhMQVFfhfQ00wuF89KNKkElcGuNvopq1b1krNCh9li/h/nACjykURdFaHE X-Gm-Message-State: AOJu0YxOj8AOKXI9Fxwx4CiRuonSiAhyyFva5ozC4h5DvxNKDaQGC5gE fjjWqtljNH7noRjoVZfabLXB+gbVyuWSsozZDqJJ/wGSsPZ1yhJE4jtCw+n1Xfo= X-Google-Smtp-Source: AGHT+IFJ9B27bdcSX0Wfqr/hDP4pnXMetZ6wPAy775G8aSk8a7BY3E+tTCNDaFLKsaqnc6by2wVkZg== X-Received: by 2002:ac2:5984:0:b0:52c:caaf:6905 with SMTP id 2adb3069b0e04-52ce0646921mr7674671e87.58.1719407582367; Wed, 26 Jun 2024 06:13:02 -0700 (PDT) Received: from localhost.localdomain (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3663a2f6977sm15857663f8f.80.2024.06.26.06.13.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Jun 2024 06:13:02 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Guo Ren Subject: [PATCH v2 09/10] asm-generic: ticket-lock: Add separate ticket-lock.h Date: Wed, 26 Jun 2024 15:03:46 +0200 Message-Id: <20240626130347.520750-10-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240626130347.520750-1-alexghiti@rivosinc.com> References: <20240626130347.520750-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240626_061304_980825_8B81F117 X-CRM114-Status: GOOD ( 21.95 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Guo Ren Add a separate ticket-lock.h to include multiple spinlock versions and select one at compile time or runtime. Reviewed-by: Leonardo Bras Suggested-by: Arnd Bergmann Link: https://lore.kernel.org/linux-riscv/CAK8P3a2rnz9mQqhN6-e0CGUUv9rntRELFdxt_weiD7FxH7fkfQ@mail.gmail.com/ Signed-off-by: Guo Ren Signed-off-by: Guo Ren --- include/asm-generic/spinlock.h | 87 +--------------------- include/asm-generic/ticket_spinlock.h | 103 ++++++++++++++++++++++++++ 2 files changed, 104 insertions(+), 86 deletions(-) create mode 100644 include/asm-generic/ticket_spinlock.h diff --git a/include/asm-generic/spinlock.h b/include/asm-generic/spinlock.h index 4773334ee638..970590baf61b 100644 --- a/include/asm-generic/spinlock.h +++ b/include/asm-generic/spinlock.h @@ -1,94 +1,9 @@ /* SPDX-License-Identifier: GPL-2.0 */ -/* - * 'Generic' ticket-lock implementation. - * - * It relies on atomic_fetch_add() having well defined forward progress - * guarantees under contention. If your architecture cannot provide this, stick - * to a test-and-set lock. - * - * It also relies on atomic_fetch_add() being safe vs smp_store_release() on a - * sub-word of the value. This is generally true for anything LL/SC although - * you'd be hard pressed to find anything useful in architecture specifications - * about this. If your architecture cannot do this you might be better off with - * a test-and-set. - * - * It further assumes atomic_*_release() + atomic_*_acquire() is RCpc and hence - * uses atomic_fetch_add() which is RCsc to create an RCsc hot path, along with - * a full fence after the spin to upgrade the otherwise-RCpc - * atomic_cond_read_acquire(). - * - * The implementation uses smp_cond_load_acquire() to spin, so if the - * architecture has WFE like instructions to sleep instead of poll for word - * modifications be sure to implement that (see ARM64 for example). - * - */ - #ifndef __ASM_GENERIC_SPINLOCK_H #define __ASM_GENERIC_SPINLOCK_H -#include -#include - -static __always_inline void arch_spin_lock(arch_spinlock_t *lock) -{ - u32 val = atomic_fetch_add(1<<16, &lock->val); - u16 ticket = val >> 16; - - if (ticket == (u16)val) - return; - - /* - * atomic_cond_read_acquire() is RCpc, but rather than defining a - * custom cond_read_rcsc() here we just emit a full fence. We only - * need the prior reads before subsequent writes ordering from - * smb_mb(), but as atomic_cond_read_acquire() just emits reads and we - * have no outstanding writes due to the atomic_fetch_add() the extra - * orderings are free. - */ - atomic_cond_read_acquire(&lock->val, ticket == (u16)VAL); - smp_mb(); -} - -static __always_inline bool arch_spin_trylock(arch_spinlock_t *lock) -{ - u32 old = atomic_read(&lock->val); - - if ((old >> 16) != (old & 0xffff)) - return false; - - return atomic_try_cmpxchg(&lock->val, &old, old + (1<<16)); /* SC, for RCsc */ -} - -static __always_inline void arch_spin_unlock(arch_spinlock_t *lock) -{ - u16 *ptr = (u16 *)lock + IS_ENABLED(CONFIG_CPU_BIG_ENDIAN); - u32 val = atomic_read(&lock->val); - - smp_store_release(ptr, (u16)val + 1); -} - -static __always_inline int arch_spin_value_unlocked(arch_spinlock_t lock) -{ - u32 val = lock.val.counter; - - return ((val >> 16) == (val & 0xffff)); -} - -static __always_inline int arch_spin_is_locked(arch_spinlock_t *lock) -{ - arch_spinlock_t val = READ_ONCE(*lock); - - return !arch_spin_value_unlocked(val); -} - -static __always_inline int arch_spin_is_contended(arch_spinlock_t *lock) -{ - u32 val = atomic_read(&lock->val); - - return (s16)((val >> 16) - (val & 0xffff)) > 1; -} - +#include #include #endif /* __ASM_GENERIC_SPINLOCK_H */ diff --git a/include/asm-generic/ticket_spinlock.h b/include/asm-generic/ticket_spinlock.h new file mode 100644 index 000000000000..cfcff22b37b3 --- /dev/null +++ b/include/asm-generic/ticket_spinlock.h @@ -0,0 +1,103 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +/* + * 'Generic' ticket-lock implementation. + * + * It relies on atomic_fetch_add() having well defined forward progress + * guarantees under contention. If your architecture cannot provide this, stick + * to a test-and-set lock. + * + * It also relies on atomic_fetch_add() being safe vs smp_store_release() on a + * sub-word of the value. This is generally true for anything LL/SC although + * you'd be hard pressed to find anything useful in architecture specifications + * about this. If your architecture cannot do this you might be better off with + * a test-and-set. + * + * It further assumes atomic_*_release() + atomic_*_acquire() is RCpc and hence + * uses atomic_fetch_add() which is RCsc to create an RCsc hot path, along with + * a full fence after the spin to upgrade the otherwise-RCpc + * atomic_cond_read_acquire(). + * + * The implementation uses smp_cond_load_acquire() to spin, so if the + * architecture has WFE like instructions to sleep instead of poll for word + * modifications be sure to implement that (see ARM64 for example). + * + */ + +#ifndef __ASM_GENERIC_TICKET_SPINLOCK_H +#define __ASM_GENERIC_TICKET_SPINLOCK_H + +#include +#include + +static __always_inline void ticket_spin_lock(arch_spinlock_t *lock) +{ + u32 val = atomic_fetch_add(1<<16, &lock->val); + u16 ticket = val >> 16; + + if (ticket == (u16)val) + return; + + /* + * atomic_cond_read_acquire() is RCpc, but rather than defining a + * custom cond_read_rcsc() here we just emit a full fence. We only + * need the prior reads before subsequent writes ordering from + * smb_mb(), but as atomic_cond_read_acquire() just emits reads and we + * have no outstanding writes due to the atomic_fetch_add() the extra + * orderings are free. + */ + atomic_cond_read_acquire(&lock->val, ticket == (u16)VAL); + smp_mb(); +} + +static __always_inline bool ticket_spin_trylock(arch_spinlock_t *lock) +{ + u32 old = atomic_read(&lock->val); + + if ((old >> 16) != (old & 0xffff)) + return false; + + return atomic_try_cmpxchg(&lock->val, &old, old + (1<<16)); /* SC, for RCsc */ +} + +static __always_inline void ticket_spin_unlock(arch_spinlock_t *lock) +{ + u16 *ptr = (u16 *)lock + IS_ENABLED(CONFIG_CPU_BIG_ENDIAN); + u32 val = atomic_read(&lock->val); + + smp_store_release(ptr, (u16)val + 1); +} + +static __always_inline int ticket_spin_value_unlocked(arch_spinlock_t lock) +{ + u32 val = lock.val.counter; + + return ((val >> 16) == (val & 0xffff)); +} + +static __always_inline int ticket_spin_is_locked(arch_spinlock_t *lock) +{ + arch_spinlock_t val = READ_ONCE(*lock); + + return !ticket_spin_value_unlocked(val); +} + +static __always_inline int ticket_spin_is_contended(arch_spinlock_t *lock) +{ + u32 val = atomic_read(&lock->val); + + return (s16)((val >> 16) - (val & 0xffff)) > 1; +} + +/* + * Remapping spinlock architecture specific functions to the corresponding + * ticket spinlock functions. + */ +#define arch_spin_is_locked(l) ticket_spin_is_locked(l) +#define arch_spin_is_contended(l) ticket_spin_is_contended(l) +#define arch_spin_value_unlocked(l) ticket_spin_value_unlocked(l) +#define arch_spin_lock(l) ticket_spin_lock(l) +#define arch_spin_trylock(l) ticket_spin_trylock(l) +#define arch_spin_unlock(l) ticket_spin_unlock(l) + +#endif /* __ASM_GENERIC_TICKET_SPINLOCK_H */ From patchwork Wed Jun 26 13:03:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13712919 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1994CC27C4F for ; Wed, 26 Jun 2024 13:14:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=fY7R4DX1U79JlbwPNwi96m5MHeWPxKNChKj85yKfNVM=; b=QwfnNjeMm54/nL g6joRI3uklepeZtFTyR/Jj69SFL9zyihkpXNfj2OuboOdG3j6IhobiMOpiBVuVg6p0Pb+15AJDLbb y3xgzuO4tSc5w+OQ5Ufm57dYozdcU/QUpeQ1vfUbXe8xkqcd8LO+JmzRHUVttSxA9iFOZ+fP9RVW4 RQUSBGzsQ0kKGP9J2is1P/lNvwbBeh2/aYL4nWXlPce9Ru1QNNL0jX/85FE+exD+8UtkSla8DZCgs peSUAE4RDJXDIo7GRV84S/6Gj7Vzi5W2drgOlFH0qSTPnNCRmjcvxPQLVodexoUn/y0SED3R2uIAu 8PSWC/49q8uXgdg8cWVQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sMSTQ-00000006uV3-2zpM; Wed, 26 Jun 2024 13:14:08 +0000 Received: from mail-wm1-x32f.google.com ([2a00:1450:4864:20::32f]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sMSTN-00000006uTO-1ObT for linux-riscv@lists.infradead.org; Wed, 26 Jun 2024 13:14:07 +0000 Received: by mail-wm1-x32f.google.com with SMTP id 5b1f17b1804b1-42561715d41so1646755e9.0 for ; Wed, 26 Jun 2024 06:14:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1719407644; x=1720012444; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+w80jGsiG4VGpCQ7hPsjWSfG61Mxqook1j9XdvlA9Tw=; b=uhcJzCN/jFzAkSTIw0GSNeNhELm9L/JV8wtDIqQXgrSLL7SKH4oUyVr47/TPVL7fI0 MTb5chfGsNMYEw2Z2NbvnFKHUeKUBj4KHnneNX1O+U8W+Utfq8ENbnzZVuwA4lXdQlmu o8HKhLWTGDBQKvWIhdu1EbOUhvus0FsUS+8TzXFLYyLzc79AMlrLe7Zy4EIiFc/mPLL7 SN5s/OUA7W+0JkDKqM4SpzCy+eG3LmC2W5dp1CuJ3OdGjo7PZ7N9l7QAvo8hkz/ZyXDr zbTrEwZtXfd8e7BC/xq7qJDPI/B7xXZWJ/BcVydhVFXQDKbGv79cK7T/9BFIxomgxaed FKVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719407644; x=1720012444; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+w80jGsiG4VGpCQ7hPsjWSfG61Mxqook1j9XdvlA9Tw=; b=HpdPr7iwV6DIxddG39A7G/rlYmwQo76xKz+3V3hprPXOQySEOcUENfPtbMtKwW9qBf HEYlX/6ZMgz/N5aHeG0rRGWXAaQ8cNV3i8AebGjeeRRTQwl3R9NKjgkQpsMOW5X6BBH3 6hh+HAg9AFhyCLKN14di9oTujfEuh1fzS4Y4qUCHpyFiIhGsgBllfV2Xoi1YJjY4moML 58p6TOkyM99RuHti0VS3D++TfkrhFwIFD2lG3Jqr/ODLZTilRw+xZBaOG6hUtBH4d583 2jtLejkd+lJ79oWsaya8t735LShu7cgYeCjaUBNXczfliD855L5u6VZ3z1tCD7byd8KT hWUA== X-Forwarded-Encrypted: i=1; AJvYcCUXthmpLjutxKL0KB10gepI1X81NWCip22qs63RfDcj5BUJBiGCISksKD8fY1Zb7kX09iE4HQz28e2ITxI7v02u0yedlq1K5Cd4FBdwSVtW X-Gm-Message-State: AOJu0Yy4rwQFlIXdSJYfeWWQfQ9mo/EtlBZ4P8ZX0r7sUYquBBwYnvTo JxLY2pje01ILGSSwXdgiDVKdgO+IPJevEmmZL1chRfwkhl8y73wOTU0S/0k+uss= X-Google-Smtp-Source: AGHT+IFllg3IYaHR+yNuwabvI2Esa87l1QM3SHNcpI6URxgMur8IwZXQjkTo3wIWV+d+Qukem0mG5Q== X-Received: by 2002:a05:600c:1d29:b0:424:86aa:b7e7 with SMTP id 5b1f17b1804b1-424893f767amr108722745e9.9.1719407643561; Wed, 26 Jun 2024 06:14:03 -0700 (PDT) Received: from localhost.localdomain (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-424c825d8a4sm25617795e9.19.2024.06.26.06.14.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Jun 2024 06:14:03 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH v2 10/10] riscv: Add qspinlock support based on Zabha extension Date: Wed, 26 Jun 2024 15:03:47 +0200 Message-Id: <20240626130347.520750-11-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240626130347.520750-1-alexghiti@rivosinc.com> References: <20240626130347.520750-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240626_061405_403874_564894BD X-CRM114-Status: GOOD ( 22.11 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org In order to produce a generic kernel, a user can select CONFIG_COMBO_SPINLOCKS which will fallback at runtime to the ticket spinlock implementation if Zabha is not present. Note that we can't use alternatives here because the discovery of extensions is done too late and we need to start with the qspinlock implementation because the ticket spinlock implementation would pollute the spinlock value, so let's use static keys. This is largely based on Guo's work and Leonardo reviews at [1]. Link: https://lore.kernel.org/linux-riscv/20231225125847.2778638-1-guoren@kernel.org/ [1] Signed-off-by: Alexandre Ghiti --- .../locking/queued-spinlocks/arch-support.txt | 2 +- arch/riscv/Kconfig | 10 +++++ arch/riscv/include/asm/Kbuild | 4 +- arch/riscv/include/asm/spinlock.h | 39 +++++++++++++++++++ arch/riscv/kernel/setup.c | 21 ++++++++++ include/asm-generic/qspinlock.h | 2 + include/asm-generic/ticket_spinlock.h | 2 + 7 files changed, 78 insertions(+), 2 deletions(-) create mode 100644 arch/riscv/include/asm/spinlock.h diff --git a/Documentation/features/locking/queued-spinlocks/arch-support.txt b/Documentation/features/locking/queued-spinlocks/arch-support.txt index 22f2990392ff..cf26042480e2 100644 --- a/Documentation/features/locking/queued-spinlocks/arch-support.txt +++ b/Documentation/features/locking/queued-spinlocks/arch-support.txt @@ -20,7 +20,7 @@ | openrisc: | ok | | parisc: | TODO | | powerpc: | ok | - | riscv: | TODO | + | riscv: | ok | | s390: | TODO | | sh: | TODO | | sparc: | ok | diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 0bbaec0444d0..c2ba673e58ca 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -72,6 +72,7 @@ config RISCV select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP select ARCH_WANTS_NO_INSTR select ARCH_WANTS_THP_SWAP if HAVE_ARCH_TRANSPARENT_HUGEPAGE + select ARCH_WEAK_RELEASE_ACQUIRE if ARCH_USE_QUEUED_SPINLOCKS select BINFMT_FLAT_NO_DATA_START_OFFSET if !MMU select BUILDTIME_TABLE_SORT if MMU select CLINT_TIMER if RISCV_M_MODE @@ -482,6 +483,15 @@ config NODES_SHIFT Specify the maximum number of NUMA Nodes available on the target system. Increases memory reserved to accommodate various tables. +config RISCV_COMBO_SPINLOCKS + bool "Using combo spinlock" + depends on SMP && MMU && TOOLCHAIN_HAS_ZABHA + select ARCH_USE_QUEUED_SPINLOCKS + default y + help + Embed both queued spinlock and ticket lock so that the spinlock + implementation can be chosen at runtime. + config RISCV_ALTERNATIVE bool depends on !XIP_KERNEL diff --git a/arch/riscv/include/asm/Kbuild b/arch/riscv/include/asm/Kbuild index 504f8b7e72d4..ad72f2bd4cc9 100644 --- a/arch/riscv/include/asm/Kbuild +++ b/arch/riscv/include/asm/Kbuild @@ -2,10 +2,12 @@ generic-y += early_ioremap.h generic-y += flat.h generic-y += kvm_para.h +generic-y += mcs_spinlock.h generic-y += parport.h -generic-y += spinlock.h generic-y += spinlock_types.h +generic-y += ticket_spinlock.h generic-y += qrwlock.h generic-y += qrwlock_types.h +generic-y += qspinlock.h generic-y += user.h generic-y += vmlinux.lds.h diff --git a/arch/riscv/include/asm/spinlock.h b/arch/riscv/include/asm/spinlock.h new file mode 100644 index 000000000000..4856d50006f2 --- /dev/null +++ b/arch/riscv/include/asm/spinlock.h @@ -0,0 +1,39 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef __ASM_RISCV_SPINLOCK_H +#define __ASM_RISCV_SPINLOCK_H + +#ifdef CONFIG_RISCV_COMBO_SPINLOCKS +#define _Q_PENDING_LOOPS (1 << 9) + +#define __no_arch_spinlock_redefine +#include +#include +#include + +DECLARE_STATIC_KEY_TRUE(qspinlock_key); + +#define SPINLOCK_BASE_DECLARE(op, type, type_lock) \ +static __always_inline type arch_spin_##op(type_lock lock) \ +{ \ + if (static_branch_unlikely(&qspinlock_key)) \ + return queued_spin_##op(lock); \ + return ticket_spin_##op(lock); \ +} + +SPINLOCK_BASE_DECLARE(lock, void, arch_spinlock_t *) +SPINLOCK_BASE_DECLARE(unlock, void, arch_spinlock_t *) +SPINLOCK_BASE_DECLARE(is_locked, int, arch_spinlock_t *) +SPINLOCK_BASE_DECLARE(is_contended, int, arch_spinlock_t *) +SPINLOCK_BASE_DECLARE(trylock, bool, arch_spinlock_t *) +SPINLOCK_BASE_DECLARE(value_unlocked, int, arch_spinlock_t) + +#else + +#include + +#endif + +#include + +#endif /* __ASM_RISCV_SPINLOCK_H */ diff --git a/arch/riscv/kernel/setup.c b/arch/riscv/kernel/setup.c index 4f73c0ae44b2..929bccd4c9e5 100644 --- a/arch/riscv/kernel/setup.c +++ b/arch/riscv/kernel/setup.c @@ -244,6 +244,26 @@ static void __init parse_dtb(void) #endif } +DEFINE_STATIC_KEY_TRUE(qspinlock_key); +EXPORT_SYMBOL(qspinlock_key); + +static void __init riscv_spinlock_init(void) +{ + asm goto(ALTERNATIVE("j %[no_zacas]", "nop", 0, RISCV_ISA_EXT_ZACAS, 1) + : : : : no_zacas); + asm goto(ALTERNATIVE("nop", "j %[qspinlock]", 0, RISCV_ISA_EXT_ZABHA, 1) + : : : : qspinlock); + +no_zacas: + static_branch_disable(&qspinlock_key); + pr_info("Ticket spinlock: enabled\n"); + + return; + +qspinlock: + pr_info("Queued spinlock: enabled\n"); +} + extern void __init init_rt_signal_env(void); void __init setup_arch(char **cmdline_p) @@ -295,6 +315,7 @@ void __init setup_arch(char **cmdline_p) riscv_set_dma_cache_alignment(); riscv_user_isa_enable(); + riscv_spinlock_init(); } bool arch_cpu_is_hotpluggable(int cpu) diff --git a/include/asm-generic/qspinlock.h b/include/asm-generic/qspinlock.h index 0655aa5b57b2..bf47cca2c375 100644 --- a/include/asm-generic/qspinlock.h +++ b/include/asm-generic/qspinlock.h @@ -136,6 +136,7 @@ static __always_inline bool virt_spin_lock(struct qspinlock *lock) } #endif +#ifndef __no_arch_spinlock_redefine /* * Remapping spinlock architecture specific functions to the corresponding * queued spinlock functions. @@ -146,5 +147,6 @@ static __always_inline bool virt_spin_lock(struct qspinlock *lock) #define arch_spin_lock(l) queued_spin_lock(l) #define arch_spin_trylock(l) queued_spin_trylock(l) #define arch_spin_unlock(l) queued_spin_unlock(l) +#endif #endif /* __ASM_GENERIC_QSPINLOCK_H */ diff --git a/include/asm-generic/ticket_spinlock.h b/include/asm-generic/ticket_spinlock.h index cfcff22b37b3..325779970d8a 100644 --- a/include/asm-generic/ticket_spinlock.h +++ b/include/asm-generic/ticket_spinlock.h @@ -89,6 +89,7 @@ static __always_inline int ticket_spin_is_contended(arch_spinlock_t *lock) return (s16)((val >> 16) - (val & 0xffff)) > 1; } +#ifndef __no_arch_spinlock_redefine /* * Remapping spinlock architecture specific functions to the corresponding * ticket spinlock functions. @@ -99,5 +100,6 @@ static __always_inline int ticket_spin_is_contended(arch_spinlock_t *lock) #define arch_spin_lock(l) ticket_spin_lock(l) #define arch_spin_trylock(l) ticket_spin_trylock(l) #define arch_spin_unlock(l) ticket_spin_unlock(l) +#endif #endif /* __ASM_GENERIC_TICKET_SPINLOCK_H */