From patchwork Wed Jul 17 06:19:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13735134 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E7F31C3DA42 for ; Wed, 17 Jul 2024 06:21:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=yEnTsYUISvRTjIGWDLSSYp8Nhh1CbrO9Hzh6z4zVE+g=; b=yklTvWFy03Y/vQ jpTiBr/X1whiuR+qJOE0T/u3tS2D4j2Dq7RDZaBh54vxDHPVQ4rlDTbhavdWJqURfw6XSB+rR7uwP gWKpeNZkYCCGRYSgVy9+vhd3QADyK97VWWMxSDTBcfxg11ybXL32x5ZtRudT72v11MJe5tXIpyaUh nzVzMhYmpdBBpUl2PW/UnnEfvTq6+X2ZaebecK3FHKMHtDGbaOk0CQsEaLxsD4LyqacwY+CI58tKd 3TORzSQK81yqxuYYydArvJQFZkveV+jE6gk4WJuHBsFvBYBv8Na1vHKsX7or4CO0bPvzYORJNg09i v4IYlpZZaE1ZtR+qvWeA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sTy2E-0000000Cnns-3NE1; Wed, 17 Jul 2024 06:21:06 +0000 Received: from mail-wm1-x334.google.com ([2a00:1450:4864:20::334]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sTy2A-0000000CnmV-3IyP for linux-riscv@lists.infradead.org; Wed, 17 Jul 2024 06:21:05 +0000 Received: by mail-wm1-x334.google.com with SMTP id 5b1f17b1804b1-4277a5ed48bso47640385e9.2 for ; Tue, 16 Jul 2024 23:21:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1721197261; x=1721802061; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ObJKiPOVe5UThPd0qvfEFz5Dv3ABd72erfyOiynPtd0=; b=FbwPaMFQxklAkyxdzRoD1+XbT7su+CwnBjFLNNJtWIjZT99wpGNXve6SCipCf9MJxu heD5BD/o/MZb7/5oSaCPbsoXt4jWBG2eI4QiKtZE2k+iNwuDJ00HUYOXsCOzgsOK/8IP JLfsHBc4Dh8TyQtRch2ksmLU7zmFkitDMutHim1xawN+zOfsAXUrOuoNOGnmrULaMj99 eAhI+vYt6+LiA0scFGZy8oOcVQU0oABt9jAVDr2LpokbwMo5jl0yQfsNsB26T/tHXuza TwilgYgBNAe0PMqwXssRLDUL6ZtvNqwAFEr7ZCTqFVQrCGnO1T8qFBpnCSaScFb7/UGR rBhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721197261; x=1721802061; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ObJKiPOVe5UThPd0qvfEFz5Dv3ABd72erfyOiynPtd0=; b=suKePcVqCUzPHN8zoe1xMUDaWUBDfx8LhYAma/Z/VrWTxaQUzkX695+Tmo40NDwvJx Tk78ID5HmnoXA+CxVxL1G9g0xy4Ee87ZH4nlAwN3bMyyFh7dhFdsGrC8/+pcgIf8Ru9a OG4A0XsS5ScsHf5AEPlSUNZIqP19M/kju8m3K3nvo3DqwBlmVL/LX5JCROJ8JjawGnvi UTVRIQp8l2fbCX6+bFh3FYCRPLTaR0KJ5QeX2Qfus8UdoApWXaSKAldlYsxaTw0YQjmX HJ3SIdvu7lrJhb4lTljv5sGtETBLGfXkYaVc0OUzoDtKbYqqzGDeXspVpLkepXFX2Z0z QooA== X-Forwarded-Encrypted: i=1; AJvYcCWzzRBYmKLRBde26M2ZgL57jJv8w7iwbko/uJREtvYLWV1O4riZ79CH+yRoyB4rc+kYD2u2pO/IFTIhdILBzO1UB7d7oL3zgXGkmtTIHE8e X-Gm-Message-State: AOJu0Yyfc6I04yzEI4ySDI93VWg8mBGMN3oPsOFNy5hc0W9y53MUQzph mMnaeliUbyt46QEqPSx870bMtQpBtNtCN9usXtZPS3oUGyeGa0YVcRIrRm5VDGs= X-Google-Smtp-Source: AGHT+IEiqDUUVGHtWHIaIo4kIBhavsk6wpSjfOY4FyjkQmBFXrWWxjPRklD8XBy60kYG9vsl83pQBA== X-Received: by 2002:a5d:6a8c:0:b0:368:37ac:3f95 with SMTP id ffacd0b85a97d-36837ac4476mr188365f8f.31.1721197261060; Tue, 16 Jul 2024 23:21:01 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3680dab3cd1sm10867433f8f.22.2024.07.16.23.21.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Jul 2024 23:21:00 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH v3 01/11] riscv: Implement cmpxchg32/64() using Zacas Date: Wed, 17 Jul 2024 08:19:47 +0200 Message-Id: <20240717061957.140712-2-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240717061957.140712-1-alexghiti@rivosinc.com> References: <20240717061957.140712-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240716_232102_838877_F7F2814C X-CRM114-Status: GOOD ( 12.49 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org This adds runtime support for Zacas in cmpxchg operations. Signed-off-by: Alexandre Ghiti --- arch/riscv/Kconfig | 17 +++++++++++++++++ arch/riscv/Makefile | 3 +++ arch/riscv/include/asm/cmpxchg.h | 26 +++++++++++++++++++++++--- 3 files changed, 43 insertions(+), 3 deletions(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 05ccba8ca33a..1caaedec88c7 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -596,6 +596,23 @@ config RISCV_ISA_V_PREEMPTIVE preemption. Enabling this config will result in higher memory consumption due to the allocation of per-task's kernel Vector context. +config TOOLCHAIN_HAS_ZACAS + bool + default y + depends on !64BIT || $(cc-option,-mabi=lp64 -march=rv64ima_zacas) + depends on !32BIT || $(cc-option,-mabi=ilp32 -march=rv32ima_zacas) + depends on AS_HAS_OPTION_ARCH + +config RISCV_ISA_ZACAS + bool "Zacas extension support for atomic CAS" + depends on TOOLCHAIN_HAS_ZACAS + default y + help + Enable the use of the Zacas ISA-extension to implement kernel atomic + cmpxchg operations when it is detected at boot. + + If you don't know what to do here, say Y. + config TOOLCHAIN_HAS_ZBB bool default y diff --git a/arch/riscv/Makefile b/arch/riscv/Makefile index 06de9d365088..9fd13d7a9cc6 100644 --- a/arch/riscv/Makefile +++ b/arch/riscv/Makefile @@ -85,6 +85,9 @@ endif # Check if the toolchain supports Zihintpause extension riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZIHINTPAUSE) := $(riscv-march-y)_zihintpause +# Check if the toolchain supports Zacas +riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZACAS) := $(riscv-march-y)_zacas + # Remove F,D,V from isa string for all. Keep extensions between "fd" and "v" by # matching non-v and non-multi-letter extensions out with the filter ([^v_]*) KBUILD_CFLAGS += -march=$(shell echo $(riscv-march-y) | sed -E 's/(rv32ima|rv64ima)fd([^v_]*)v?/\1\2/') diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index 808b4c78462e..5d38153e2f13 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -9,6 +9,7 @@ #include #include +#include #define __arch_xchg_masked(sc_sfx, prepend, append, r, p, n) \ ({ \ @@ -134,21 +135,40 @@ r = (__typeof__(*(p)))((__retx & __mask) >> __s); \ }) -#define __arch_cmpxchg(lr_sfx, sc_sfx, prepend, append, r, p, co, o, n) \ +#define __arch_cmpxchg(lr_sfx, sc_cas_sfx, prepend, append, r, p, co, o, n) \ ({ \ + __label__ no_zacas, end; \ register unsigned int __rc; \ \ + if (IS_ENABLED(CONFIG_RISCV_ISA_ZACAS)) { \ + asm goto(ALTERNATIVE("j %[no_zacas]", "nop", 0, \ + RISCV_ISA_EXT_ZACAS, 1) \ + : : : : no_zacas); \ + \ + __asm__ __volatile__ ( \ + prepend \ + " amocas" sc_cas_sfx " %0, %z2, %1\n" \ + append \ + : "+&r" (r), "+A" (*(p)) \ + : "rJ" (n) \ + : "memory"); \ + goto end; \ + } \ + \ +no_zacas: \ __asm__ __volatile__ ( \ prepend \ "0: lr" lr_sfx " %0, %2\n" \ " bne %0, %z3, 1f\n" \ - " sc" sc_sfx " %1, %z4, %2\n" \ + " sc" sc_cas_sfx " %1, %z4, %2\n" \ " bnez %1, 0b\n" \ append \ "1:\n" \ : "=&r" (r), "=&r" (__rc), "+A" (*(p)) \ : "rJ" (co o), "rJ" (n) \ : "memory"); \ + \ +end:; \ }) #define _arch_cmpxchg(ptr, old, new, sc_sfx, prepend, append) \ @@ -156,7 +176,7 @@ __typeof__(ptr) __ptr = (ptr); \ __typeof__(*(__ptr)) __old = (old); \ __typeof__(*(__ptr)) __new = (new); \ - __typeof__(*(__ptr)) __ret; \ + __typeof__(*(__ptr)) __ret = (old); \ \ switch (sizeof(*__ptr)) { \ case 1: \ From patchwork Wed Jul 17 06:19:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13735135 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1376BC3DA4B for ; Wed, 17 Jul 2024 06:22:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Zz5CieM7fK8R3oPlwg+PHpbclLpuk3VKA01Omdj5cKM=; b=W7yOWiZLQ5+jWU UGu50qadaQYiYH1PGYuqsfDBPcvKLulpu8h5Ql6HskZ6eKEudu004xMbIrsfuIWBSMmIqJOgbLHHc ROGHkIKmd3R5w1jLN14JpQHsJMk2yT8HaDAcZXzVMqd4+yq5nVNJxNbCbGYQoQOiG0iQdfs3d7vWp 65mz81nnHteS4uGwynZmpveIRC4mHavZBmZDuTrQ1lgZnZS8b3ioXsSoWaWW3E8jQPXVUcdVGq3h5 a8zFbXHe5XfSIm6zcUrmIDDQXkjSm67TQ57pPSTnItJj1c41xNZGu2d8Gai6HUqTjgCRmUlBaHAAv /HMMkdN+V85opAilKdzg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sTy3D-0000000Co5h-0BDw; Wed, 17 Jul 2024 06:22:07 +0000 Received: from mail-wm1-x32e.google.com ([2a00:1450:4864:20::32e]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sTy3A-0000000Co4z-0rPk for linux-riscv@lists.infradead.org; Wed, 17 Jul 2024 06:22:05 +0000 Received: by mail-wm1-x32e.google.com with SMTP id 5b1f17b1804b1-426717a2d12so2369535e9.0 for ; Tue, 16 Jul 2024 23:22:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1721197322; x=1721802122; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=T6LJZCR61gmm4dvekAa0Q1TIoLF/ScrG5TtOv9gDZ/w=; b=uCQHzQfxk97VC6LNjl5UD+QF+dhyEZtisfHryiLlLXYoj2KM/WzwmfhHPS7MhpiM2a AVfXvUoVN33qG721UiBoAWd/5LXt0m7PCAUFnODkQ/VnbTAq4Skr0PMRbAgyvknSBTz8 fp7PiC7lPQgJCvXYujrXFnfu+eWPzaEMjYwbrlRgPTxiltnWW0a9UPFxZ+XZpSrURe/o 15tDScLIyZwpyG2clAy562gdPjVb020BzcmWIYobZ1rYrBLPwjJjM36VwEvXujf/TJqI 0GinXnrWA5ZUfsr/4FMMg7yPuMyvvt4EKk5VGswjBGXojidNhEYdXq6TuNIim/Dqh+VX lryA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721197322; x=1721802122; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=T6LJZCR61gmm4dvekAa0Q1TIoLF/ScrG5TtOv9gDZ/w=; b=PikXX9ph3OWS+QERxgfUXn5ew69UAYexbYwlijyxWnij2FhAJ5Togxoc/ta0ej6CoZ CoiwVHlliFloD2UBmRT4PtYWJujnowIj5UZqFfHV5f2nJwjm/V7BDKkDy88qbXEtfJD0 RodQi0DEDYo12WRynAWg5HUgTdisHUuc5v7qwaOVkRBfIIiQcOtnaorktrbt1fXHEutA MoyeRPROzc1Grq80+6wIyLQN1DruK1795TpXKJJp0VQudAgCK7V7xnquABPz+DSVoPrK IlW8ygick07HhPjMWWyf6TPzBpjkkc2RWwuqZ2k+Rzvs6xpw3zTTaXp+SpAPSa9IaOBS dseA== X-Forwarded-Encrypted: i=1; AJvYcCUrMji693+gklutPD9SQ3hKimPXg2bU/ASKdM9OUKBZIOgMWSWppw3mlc6kWOSTZmSu1D/URI1kiH3dDt7cEoS/2nBRxlnckE7qbbViCL3E X-Gm-Message-State: AOJu0Yzoq9FfLlmxiQpgxA/8A8alTmEYuK8F0iXUQqcyDt/6Cvma0ibO SXWUezi0FsQNyLe8wqfRqFvY/44Hb1WHwT8bc9Xk+wXZ6fAiTcSl8CsB4H4myME= X-Google-Smtp-Source: AGHT+IFU9J5ZIoNE9rhylPtawNo/bVXV/Y54qi/t2Pf5eJSgSGKg9qXAA/fSZkpR14Z9Af1ks6C3xQ== X-Received: by 2002:a05:600c:1d28:b0:426:6981:1bd with SMTP id 5b1f17b1804b1-427c28fa6a5mr5014165e9.5.1721197322313; Tue, 16 Jul 2024 23:22:02 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4279f23a000sm194474825e9.5.2024.07.16.23.22.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Jul 2024 23:22:01 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH v3 02/11] dt-bindings: riscv: Add Zabha ISA extension description Date: Wed, 17 Jul 2024 08:19:48 +0200 Message-Id: <20240717061957.140712-3-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240717061957.140712-1-alexghiti@rivosinc.com> References: <20240717061957.140712-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240716_232204_249657_A42DEB1A X-CRM114-Status: UNSURE ( 9.07 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Add description for the Zabha ISA extension which was ratified in April 2024. Signed-off-by: Alexandre Ghiti Reviewed-by: Guo Ren --- Documentation/devicetree/bindings/riscv/extensions.yaml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/Documentation/devicetree/bindings/riscv/extensions.yaml b/Documentation/devicetree/bindings/riscv/extensions.yaml index 468c646247aa..e6436260bdeb 100644 --- a/Documentation/devicetree/bindings/riscv/extensions.yaml +++ b/Documentation/devicetree/bindings/riscv/extensions.yaml @@ -171,6 +171,12 @@ properties: memory types as ratified in the 20191213 version of the privileged ISA specification. + - const: zabha + description: | + The Zabha extension for Byte and Halfword Atomic Memory Operations + as ratified at commit 49f49c842ff9 ("Update to Rafified state") of + riscv-zabha. + - const: zacas description: | The Zacas extension for Atomic Compare-and-Swap (CAS) instructions From patchwork Wed Jul 17 06:19:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13735136 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 11CE4C3DA60 for ; Wed, 17 Jul 2024 06:23:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=3wu+emxUvN/CLXUOaxwKsNRvNWNpi6A5jAApcRgUjc4=; b=EMRUnQMmA6lg3i E78zoLBMShh0CMq+1QIyho+Vl8i6lABgSncgekrvCTV8z7pDpFX4273ufzeyjn1uvEH8BrF7pdH7R 0QpeXkeucCu5zUWOkvIVYyZlPXaKmQmFMDnXQkF7Pj6kJ7esBERs1v7ZmaMtC/NbFnJ4eplqRSj24 cUOYm978QgTMhPkCCw6dkzoQQq9sV9Z4p7bm8TIWvygqTDgqKtk9m02vo1Jykiv0/NCuJvvpNzGxH PbBMblGkEPzjjeuEQ8sNLcJYqsiNvLjoZfVCtLVrV+qNvb73KAMvOrbhRyDXNrKj7LcaovLjdIlsA mgYy78JI5Z9preJ6Nrng==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sTy4C-0000000CoP8-0Ywb; Wed, 17 Jul 2024 06:23:08 +0000 Received: from mail-wm1-x32e.google.com ([2a00:1450:4864:20::32e]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sTy49-0000000CoNR-140R for linux-riscv@lists.infradead.org; Wed, 17 Jul 2024 06:23:06 +0000 Received: by mail-wm1-x32e.google.com with SMTP id 5b1f17b1804b1-426526d30aaso45401695e9.0 for ; Tue, 16 Jul 2024 23:23:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1721197384; x=1721802184; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0QjpPjoLwI+58peQzByots6FWlw3e8eiNriVwaJAk3o=; b=mOrGhL/uPYUD7YNwrPqX2Ze5sbZH/+JdOM1YsWzEmMPm6cCXwrOa56v0GE1TxQ9hGf bDxeGT9Ji00O7py//MWMBRI8ys3sBXPOAoHWT1IUz/5CYm2mt1S4SUaETsmBkoYiBkoJ ftST/soXmSPZdyZJKmy0ZgwUEQn2WP2pd3s9kc2FLmRTPsijTDeV74KysrsEJoalbZmS cPnf45FY3/S8bVwgvEF3oJm26aoigF6uqUcQmV6852x+H5lp2Hb+HLMKfZFnzZpPomzs CyVHYMJTUha0QrSR26U650QZayUeSMKTn5az+an8r0iAsy0ICURzobc/UjjCf2RA+fBj 1E/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721197384; x=1721802184; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0QjpPjoLwI+58peQzByots6FWlw3e8eiNriVwaJAk3o=; b=rg0SVlaCBCMOwvq6kuAgwOenkoFtxQS2g5Q0MM/uhj0Ph9f8JWn7VzF3tGdoH8PCNp G/OXbAIMqaR9J7CADDD8QrcfEV2m2hDJ/aAuwKT3umHZknjmeBsMhqNcnr4U/OMsyv5E IPNGoQinrF3ii4K1K3v83xJQfp9HTk8rjB3U1mHU2XVz/+MDdrMW2wxDnTdKw3mQHw9U /K+aSgRYRyjW3/kC5K4PrpkM280DDvMP3VfEKaNodSPKCbvOMWM7+z4l2oo6TqVhe0x6 q6RljI9QpZd/j5mHFs1fxiopj3ix+cTjr3LQ9TF3UqrGZDziLFd3tIKlSIYeMTPofYsL n8cA== X-Forwarded-Encrypted: i=1; AJvYcCWfiN80HSpV6jNcsQ6cyPT57e1ULhlIYnhOLjisAge72j/s3l7plF4gBmAeIvf4rZ4MO6P56ZFEn9Iw42ZsxpXKFXsMoDK/+ENdgmCgTqt+ X-Gm-Message-State: AOJu0Yy8OMsrGwKkUDnYLe28odIPsaMm2Ca6wPGDr0+woM1QEPHQbWGr GwcQQkzB5WYiNmfrKICdqamX8r3M3FKCB8iuscCsf6mlz9QWV3iMg6fRX2wCYQk= X-Google-Smtp-Source: AGHT+IFPLb6pDwn4ZT2PDYb1qbrSJD7FUpoL0IeRpIU13kaHx3oluVLPGx9Y3b3221ag/XCVCbja3g== X-Received: by 2002:a05:600c:3592:b0:426:5dca:50a with SMTP id 5b1f17b1804b1-427c2ce6539mr4895335e9.21.1721197383614; Tue, 16 Jul 2024 23:23:03 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3680db03e41sm10773043f8f.98.2024.07.16.23.23.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Jul 2024 23:23:03 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH v3 03/11] riscv: Implement cmpxchg8/16() using Zabha Date: Wed, 17 Jul 2024 08:19:49 +0200 Message-Id: <20240717061957.140712-4-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240717061957.140712-1-alexghiti@rivosinc.com> References: <20240717061957.140712-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240716_232305_321187_9C4AAE21 X-CRM114-Status: GOOD ( 15.49 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org This adds runtime support for Zabha in cmpxchg8/16() operations. Note that in the absence of Zacas support in the toolchain, CAS instructions from Zabha won't be used. Signed-off-by: Alexandre Ghiti --- arch/riscv/Kconfig | 17 ++++++++++++++++ arch/riscv/Makefile | 3 +++ arch/riscv/include/asm/cmpxchg.h | 33 ++++++++++++++++++++++++++++++-- arch/riscv/include/asm/hwcap.h | 1 + arch/riscv/kernel/cpufeature.c | 1 + 5 files changed, 53 insertions(+), 2 deletions(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 1caaedec88c7..d3b0f92f92da 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -596,6 +596,23 @@ config RISCV_ISA_V_PREEMPTIVE preemption. Enabling this config will result in higher memory consumption due to the allocation of per-task's kernel Vector context. +config TOOLCHAIN_HAS_ZABHA + bool + default y + depends on !64BIT || $(cc-option,-mabi=lp64 -march=rv64ima_zabha) + depends on !32BIT || $(cc-option,-mabi=ilp32 -march=rv32ima_zabha) + depends on AS_HAS_OPTION_ARCH + +config RISCV_ISA_ZABHA + bool "Zabha extension support for atomic byte/halfword operations" + depends on TOOLCHAIN_HAS_ZABHA + default y + help + Enable the use of the Zabha ISA-extension to implement kernel + byte/halfword atomic memory operations when it is detected at boot. + + If you don't know what to do here, say Y. + config TOOLCHAIN_HAS_ZACAS bool default y diff --git a/arch/riscv/Makefile b/arch/riscv/Makefile index 9fd13d7a9cc6..78dcaaeebf4e 100644 --- a/arch/riscv/Makefile +++ b/arch/riscv/Makefile @@ -88,6 +88,9 @@ riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZIHINTPAUSE) := $(riscv-march-y)_zihintpause # Check if the toolchain supports Zacas riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZACAS) := $(riscv-march-y)_zacas +# Check if the toolchain supports Zabha +riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZABHA) := $(riscv-march-y)_zabha + # Remove F,D,V from isa string for all. Keep extensions between "fd" and "v" by # matching non-v and non-multi-letter extensions out with the filter ([^v_]*) KBUILD_CFLAGS += -march=$(shell echo $(riscv-march-y) | sed -E 's/(rv32ima|rv64ima)fd([^v_]*)v?/\1\2/') diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index 5d38153e2f13..c86722a101d0 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -105,8 +105,30 @@ * indicated by comparing RETURN with OLD. */ -#define __arch_cmpxchg_masked(sc_sfx, prepend, append, r, p, o, n) \ +#define __arch_cmpxchg_masked(sc_sfx, cas_sfx, prepend, append, r, p, o, n) \ ({ \ + __label__ no_zabha_zacas, end; \ + \ + if (IS_ENABLED(CONFIG_RISCV_ISA_ZABHA) && \ + IS_ENABLED(CONFIG_RISCV_ISA_ZACAS)) { \ + asm goto(ALTERNATIVE("j %[no_zabha_zacas]", "nop", 0, \ + RISCV_ISA_EXT_ZABHA, 1) \ + : : : : no_zabha_zacas); \ + asm goto(ALTERNATIVE("j %[no_zabha_zacas]", "nop", 0, \ + RISCV_ISA_EXT_ZACAS, 1) \ + : : : : no_zabha_zacas); \ + \ + __asm__ __volatile__ ( \ + prepend \ + " amocas" cas_sfx " %0, %z2, %1\n" \ + append \ + : "+&r" (r), "+A" (*(p)) \ + : "rJ" (n) \ + : "memory"); \ + goto end; \ + } \ + \ +no_zabha_zacas:; \ u32 *__ptr32b = (u32 *)((ulong)(p) & ~0x3); \ ulong __s = ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \ ulong __mask = GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \ @@ -133,6 +155,8 @@ : "memory"); \ \ r = (__typeof__(*(p)))((__retx & __mask) >> __s); \ + \ +end:; \ }) #define __arch_cmpxchg(lr_sfx, sc_cas_sfx, prepend, append, r, p, co, o, n) \ @@ -180,8 +204,13 @@ end:; \ \ switch (sizeof(*__ptr)) { \ case 1: \ + __arch_cmpxchg_masked(sc_sfx, ".b" sc_sfx, \ + prepend, append, \ + __ret, __ptr, __old, __new); \ + break; \ case 2: \ - __arch_cmpxchg_masked(sc_sfx, prepend, append, \ + __arch_cmpxchg_masked(sc_sfx, ".h" sc_sfx, \ + prepend, append, \ __ret, __ptr, __old, __new); \ break; \ case 4: \ diff --git a/arch/riscv/include/asm/hwcap.h b/arch/riscv/include/asm/hwcap.h index e17d0078a651..f71ddd2ca163 100644 --- a/arch/riscv/include/asm/hwcap.h +++ b/arch/riscv/include/asm/hwcap.h @@ -81,6 +81,7 @@ #define RISCV_ISA_EXT_ZTSO 72 #define RISCV_ISA_EXT_ZACAS 73 #define RISCV_ISA_EXT_XANDESPMU 74 +#define RISCV_ISA_EXT_ZABHA 75 #define RISCV_ISA_EXT_XLINUXENVCFG 127 diff --git a/arch/riscv/kernel/cpufeature.c b/arch/riscv/kernel/cpufeature.c index 5ef48cb20ee1..c125d82c894b 100644 --- a/arch/riscv/kernel/cpufeature.c +++ b/arch/riscv/kernel/cpufeature.c @@ -257,6 +257,7 @@ const struct riscv_isa_ext_data riscv_isa_ext[] = { __RISCV_ISA_EXT_DATA(zihintpause, RISCV_ISA_EXT_ZIHINTPAUSE), __RISCV_ISA_EXT_DATA(zihpm, RISCV_ISA_EXT_ZIHPM), __RISCV_ISA_EXT_DATA(zacas, RISCV_ISA_EXT_ZACAS), + __RISCV_ISA_EXT_DATA(zabha, RISCV_ISA_EXT_ZABHA), __RISCV_ISA_EXT_DATA(zfa, RISCV_ISA_EXT_ZFA), __RISCV_ISA_EXT_DATA(zfh, RISCV_ISA_EXT_ZFH), __RISCV_ISA_EXT_DATA(zfhmin, RISCV_ISA_EXT_ZFHMIN), From patchwork Wed Jul 17 06:19:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13735137 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C5AFEC3DA4B for ; Wed, 17 Jul 2024 06:24:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=wdGHoWbOdbpAipZ6GduZJ2FzK/ZPUL5J8i0IboyNMkQ=; b=h9lBT6Uk3cPiTE 91EwacOlh3+Eb/FxJxbp9hbHiOQYQFeJ77vP19AvUcQtOPfp8ECmySxwsPbvzx6XFIKy6DXW7J1GZ GOhVS4bNKPpOrpmtwOJriPkmfBfaS7n7CkgJSD7AqReLvya/YTB78byjmkN4We6iiB1FURwUELfYG JNzboS5bLXPyG8+2+I/co4KLT0vbw7LWFQxPut6SsFV9hkMGGxpGV4xGPI3CGL8gijtb3OzfFO0dU BawqamOF+gQ0yaoyitB6ObBVrTYXY77H+tahlWzdaHJBgzwAkRp/5PTpoP7kbJTt+zZR1Dg4k66fL Ou3b7b1AxfckRWjEXWzQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sTy5B-0000000Cok4-0fYP; Wed, 17 Jul 2024 06:24:09 +0000 Received: from mail-wm1-x332.google.com ([2a00:1450:4864:20::332]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sTy58-0000000Coj9-1CEg for linux-riscv@lists.infradead.org; Wed, 17 Jul 2024 06:24:07 +0000 Received: by mail-wm1-x332.google.com with SMTP id 5b1f17b1804b1-4266f535e82so43766675e9.1 for ; Tue, 16 Jul 2024 23:24:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1721197445; x=1721802245; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Ds1f3nvde/2kobU8ZNZg62T2Qi5mJIAm9dc1xPbfhas=; b=aDHZn1wxHmG4ZZ1lr6mk0I/dNuDuRdAbDSJHHn0K+fnw6tMwlEzEx8HbQrvQWRrcVN jc+AtkHOknXc6HqOzKGtpdlp9rLqq1jYLnRoMYOWIugny6E0NIjUEM0wdnIIHbHViGSj sDPNBYYY39dqOwn+8hlcV/Kaa8gbi4z3rJf7J1XKh3gFmGx4/4eyyint4/V+zcZ9tO/5 totgjxbSae7D88AQKSg04A+tcuzwam+MKFS4t/on79QAsSjjGPEf8Wva/wdaXDykliZf tHqrS/I/kwneZDrQ61N/fWGJ+o4XmF05OfYaLx+lVIZ/t4FLF46rzHlDYT1oJ8Df96Al eANQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721197445; x=1721802245; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Ds1f3nvde/2kobU8ZNZg62T2Qi5mJIAm9dc1xPbfhas=; b=JU/PciT/I2HoXXcbnawPai1Nkm5zCAPa8XKNMlzvYimQTpR4it16Ai/Rd0+g9x5GRT +JY2A703P6QLJ2+nSpcGXFQE9HjkPmd6u65fdEt8v1R5fkcMFJFjB4GuN61R2t2UL3dD KcbT00ML4+Ivb8M1DjVE5GPmPM1Db724p/RkrstqsZ8o+60K0zqQ9dlCaKy26C52M6/r sboCXvwxSOK7iT4tekecjoyCIy1MEmBcscFX0QmpJ205pcRRUbQBsZdqbRSccBLu+VDh wCTqLas4X1ZOgEFwgsInnzzQN5MBk9kNwqQV2IretjwpNfRIqsp9icdbrXsgwK9RxZwv PmYA== X-Forwarded-Encrypted: i=1; AJvYcCW3jDopXNpFM/CWtEETIZgwQueKmUfAaL1715Imb5yiWgNj1Bu0hXkAsbT/DS/4CYDjVUnrhgRde1TA9xcsa1TU9oiLC+y2bVk+wp3UPWQb X-Gm-Message-State: AOJu0YxD1abY9m6Vu5Ra4f1FZiTlnZJqR12SrLCgZqSgrzjO3k0Kw99e QtN5bmSi2AAamOwY27+/6XpCVxCPcrR7f/O6oRbkOjHz9iecIoolcyz1jsZJuEM= X-Google-Smtp-Source: AGHT+IEIXpdS+STkowwy/Wm5vUbJoSbRCAM24AYCj9d6c8yOJneivGu6HLD2duOFWquiyMpaznkmeg== X-Received: by 2002:a5d:6b8d:0:b0:368:75:26e5 with SMTP id ffacd0b85a97d-368316003e7mr683121f8f.1.1721197444910; Tue, 16 Jul 2024 23:24:04 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3680dafba03sm10826606f8f.76.2024.07.16.23.24.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Jul 2024 23:24:04 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti , Andrea Parri Subject: [PATCH v3 04/11] riscv: Improve zacas fully-ordered cmpxchg() Date: Wed, 17 Jul 2024 08:19:50 +0200 Message-Id: <20240717061957.140712-5-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240717061957.140712-1-alexghiti@rivosinc.com> References: <20240717061957.140712-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240716_232406_368204_BE223C07 X-CRM114-Status: UNSURE ( 9.34 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org The current fully-ordered cmpxchgXX() implementation results in: amocas.X.rl a5,a4,(s1) fence rw,rw This provides enough sync but we can actually use the following better mapping instead: amocas.X.aqrl a5,a4,(s1) Suggested-by: Andrea Parri Signed-off-by: Alexandre Ghiti --- arch/riscv/include/asm/cmpxchg.h | 71 ++++++++++++++++++++------------ 1 file changed, 44 insertions(+), 27 deletions(-) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index c86722a101d0..97b24da38897 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -105,7 +105,10 @@ * indicated by comparing RETURN with OLD. */ -#define __arch_cmpxchg_masked(sc_sfx, cas_sfx, prepend, append, r, p, o, n) \ +#define __arch_cmpxchg_masked(sc_sfx, cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append, \ + r, p, o, n) \ ({ \ __label__ no_zabha_zacas, end; \ \ @@ -119,9 +122,9 @@ : : : : no_zabha_zacas); \ \ __asm__ __volatile__ ( \ - prepend \ + cas_prepend \ " amocas" cas_sfx " %0, %z2, %1\n" \ - append \ + cas_append \ : "+&r" (r), "+A" (*(p)) \ : "rJ" (n) \ : "memory"); \ @@ -139,7 +142,7 @@ no_zabha_zacas:; \ ulong __rc; \ \ __asm__ __volatile__ ( \ - prepend \ + sc_prepend \ "0: lr.w %0, %2\n" \ " and %1, %0, %z5\n" \ " bne %1, %z3, 1f\n" \ @@ -147,7 +150,7 @@ no_zabha_zacas:; \ " or %1, %1, %z4\n" \ " sc.w" sc_sfx " %1, %1, %2\n" \ " bnez %1, 0b\n" \ - append \ + sc_append \ "1:\n" \ : "=&r" (__retx), "=&r" (__rc), "+A" (*(__ptr32b)) \ : "rJ" ((long)__oldx), "rJ" (__newx), \ @@ -159,7 +162,10 @@ no_zabha_zacas:; \ end:; \ }) -#define __arch_cmpxchg(lr_sfx, sc_cas_sfx, prepend, append, r, p, co, o, n) \ +#define __arch_cmpxchg(lr_sfx, sc_sfx, cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append, \ + r, p, co, o, n) \ ({ \ __label__ no_zacas, end; \ register unsigned int __rc; \ @@ -170,9 +176,9 @@ end:; \ : : : : no_zacas); \ \ __asm__ __volatile__ ( \ - prepend \ - " amocas" sc_cas_sfx " %0, %z2, %1\n" \ - append \ + cas_prepend \ + " amocas" cas_sfx " %0, %z2, %1\n" \ + cas_append \ : "+&r" (r), "+A" (*(p)) \ : "rJ" (n) \ : "memory"); \ @@ -181,12 +187,12 @@ end:; \ \ no_zacas: \ __asm__ __volatile__ ( \ - prepend \ + sc_prepend \ "0: lr" lr_sfx " %0, %2\n" \ " bne %0, %z3, 1f\n" \ - " sc" sc_cas_sfx " %1, %z4, %2\n" \ + " sc" sc_sfx " %1, %z4, %2\n" \ " bnez %1, 0b\n" \ - append \ + sc_append \ "1:\n" \ : "=&r" (r), "=&r" (__rc), "+A" (*(p)) \ : "rJ" (co o), "rJ" (n) \ @@ -195,7 +201,9 @@ no_zacas: \ end:; \ }) -#define _arch_cmpxchg(ptr, old, new, sc_sfx, prepend, append) \ +#define _arch_cmpxchg(ptr, old, new, sc_sfx, cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append) \ ({ \ __typeof__(ptr) __ptr = (ptr); \ __typeof__(*(__ptr)) __old = (old); \ @@ -204,22 +212,28 @@ end:; \ \ switch (sizeof(*__ptr)) { \ case 1: \ - __arch_cmpxchg_masked(sc_sfx, ".b" sc_sfx, \ - prepend, append, \ - __ret, __ptr, __old, __new); \ + __arch_cmpxchg_masked(sc_sfx, ".b" cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append, \ + __ret, __ptr, __old, __new); \ break; \ case 2: \ - __arch_cmpxchg_masked(sc_sfx, ".h" sc_sfx, \ - prepend, append, \ - __ret, __ptr, __old, __new); \ + __arch_cmpxchg_masked(sc_sfx, ".h" cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append, \ + __ret, __ptr, __old, __new); \ break; \ case 4: \ - __arch_cmpxchg(".w", ".w" sc_sfx, prepend, append, \ - __ret, __ptr, (long), __old, __new); \ + __arch_cmpxchg(".w", ".w" sc_sfx, ".w" cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append, \ + __ret, __ptr, (long), __old, __new); \ break; \ case 8: \ - __arch_cmpxchg(".d", ".d" sc_sfx, prepend, append, \ - __ret, __ptr, /**/, __old, __new); \ + __arch_cmpxchg(".d", ".d" sc_sfx, ".d" cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append, \ + __ret, __ptr, /**/, __old, __new); \ break; \ default: \ BUILD_BUG(); \ @@ -228,16 +242,19 @@ end:; \ }) #define arch_cmpxchg_relaxed(ptr, o, n) \ - _arch_cmpxchg((ptr), (o), (n), "", "", "") + _arch_cmpxchg((ptr), (o), (n), "", "", "", "", "", "") #define arch_cmpxchg_acquire(ptr, o, n) \ - _arch_cmpxchg((ptr), (o), (n), "", "", RISCV_ACQUIRE_BARRIER) + _arch_cmpxchg((ptr), (o), (n), "", "", \ + "", RISCV_ACQUIRE_BARRIER, "", RISCV_ACQUIRE_BARRIER) #define arch_cmpxchg_release(ptr, o, n) \ - _arch_cmpxchg((ptr), (o), (n), "", RISCV_RELEASE_BARRIER, "") + _arch_cmpxchg((ptr), (o), (n), "", "", \ + RISCV_RELEASE_BARRIER, "", RISCV_RELEASE_BARRIER, "") #define arch_cmpxchg(ptr, o, n) \ - _arch_cmpxchg((ptr), (o), (n), ".rl", "", " fence rw, rw\n") + _arch_cmpxchg((ptr), (o), (n), ".rl", ".aqrl", \ + "", RISCV_FULL_BARRIER, "", "") #define arch_cmpxchg_local(ptr, o, n) \ arch_cmpxchg_relaxed((ptr), (o), (n)) From patchwork Wed Jul 17 06:19:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13735138 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7011EC3DA42 for ; Wed, 17 Jul 2024 06:25:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Ku4pHck8jJLouOvh9r8oZFkUY177cKaoXv5snPHdt8M=; b=bArMkzK8cT6/d6 VS0kyeUFKsHOs0TibvETcFsv9NJkhBCpnvQVMrs3bpH/3sjw07k2n50VSZqQvw/CEKt5cNa+D+Fww hMTjdOqI6gUr+qE2V8owLAIFWvIjWSw8YeNkPX0c1bkaDrEEcAF1Ae75OVt4DEcs52i3NCb9HpuZG ZDQu9j+2OoNxv2y70ViJZ3Bhvfghv5jUIbDkCTZ4Qo8PhchLG/hiYdYuLVIzGzvK/47HSrTwpcl0L G32ZhyPIXKkf2J0No1r0b3iVCGxBQ0LJ2d4zJ1dUcPO2Fm+MgPl/chcOAuKmbpQhCHfEv5hZ0pDBx HQGbylXkjABlFhWMtTAA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sTy6A-0000000Cp2d-1Hvy; Wed, 17 Jul 2024 06:25:10 +0000 Received: from mail-wm1-x329.google.com ([2a00:1450:4864:20::329]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sTy68-0000000Cp1Q-1E1D for linux-riscv@lists.infradead.org; Wed, 17 Jul 2024 06:25:09 +0000 Received: by mail-wm1-x329.google.com with SMTP id 5b1f17b1804b1-4266f535e82so43773035e9.1 for ; Tue, 16 Jul 2024 23:25:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1721197506; x=1721802306; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xQbGd0qu6ssbesu7Zjwn0myIi2A+oWSgEr0KZDNP1pQ=; b=oSYZUTuomlhUGJt9tE4CFLYtM5Y2+7vOdd3nodfBnbLXGD5RPZIQMZyTF9nzGpLeDN r7zncMrLfXap9cpp4RK4M7ZrD3zDOZiBqPDdHpH40igz3ruAfoUiQ73Uk2DMtDRuD/K/ h++/Lih/hynyQGwY2NjxiRHwmZMP4YAZpnH7fkLLO3BRIIReW/HoP/I+JQBZ6VhQVS+M OyrgCM+kQfRenpAao9DSD6Tgj9WZ4CWPxGjn29yzhvt8CCJzCfxXWzIr02k+VTc3sC7Y FegiISgRH1oVLjyd7Y66A8dEGvn/NGUrfhHcWaXtaDYR16GZ1Khl+2UZdJgUqYwN3kVJ MpaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721197506; x=1721802306; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xQbGd0qu6ssbesu7Zjwn0myIi2A+oWSgEr0KZDNP1pQ=; b=jiDuDA1JG4saGb75wrrvuaS55v6mclzmgFVb+1kH+nBh3pTvlevd9mKzfb3WJN0sIw I6gvqPwxBNeDaOi3q8MRnyMDbAPeLpjdgE8O79lonQKKljm/uLKYneYm8ERwPrTaNkNt 1uiNZkJNKSkNBrXRPcEk3s2Q+eyQBbCRK2HV8uFVidD5mH3MejcvWnwx+b9qjqCPIIjR SMnaV8JrVsOLXTLP/VxngOI4YjwUV/JSeyFkfIJ0K0T3UfQVfN6Rk0MMRxEVD4u9BFGv f5Se73iNX/tzPocPbGmUpFdAEaT8onwGsYgD1CFHSQbkSiKoo63xHot3fi7+6zhKyI7V 7UsQ== X-Forwarded-Encrypted: i=1; AJvYcCUgIOSTBGAqfQduVsr96RDJZ568cJsLah4KaZaH8YuHJQA88LIHQxmgkKrSoix3na1DRBLz4jSDDlrNQAoCQCuFTgqnEGqDL/aGnSRNwycJ X-Gm-Message-State: AOJu0YzGiByLRBCPscFGjJVtjFFOyCA+9ioVv4F/cWKn6vaFmxbaTwP5 FQScqAZ4EoqLVLNeiMmYnyt8Uz3psEB94gHJgW3wpYrhDF2yAeejZ4fOdZSHA0Q= X-Google-Smtp-Source: AGHT+IEFDv6V5tFWhauDPHJiZCgq6FMlgRvbm1hC+8pd4MPLkSbI6qpbIzPF0VaZZ69mfVtEYemITg== X-Received: by 2002:a05:600c:4eca:b0:426:549c:294c with SMTP id 5b1f17b1804b1-427c2d06e43mr5639015e9.35.1721197506505; Tue, 16 Jul 2024 23:25:06 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-427a5ef4617sm155918965e9.41.2024.07.16.23.25.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Jul 2024 23:25:06 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH v3 05/11] riscv: Implement arch_cmpxchg128() using Zacas Date: Wed, 17 Jul 2024 08:19:51 +0200 Message-Id: <20240717061957.140712-6-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240717061957.140712-1-alexghiti@rivosinc.com> References: <20240717061957.140712-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240716_232508_338373_B0728429 X-CRM114-Status: GOOD ( 12.52 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Now that Zacas is supported in the kernel, let's use the double word atomic version of amocas to improve the SLUB allocator. Note that we have to select fixed registers, otherwise gcc fails to pick even registers and then produces a reserved encoding which fails to assemble. Signed-off-by: Alexandre Ghiti --- arch/riscv/Kconfig | 1 + arch/riscv/include/asm/cmpxchg.h | 39 ++++++++++++++++++++++++++++++++ 2 files changed, 40 insertions(+) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index d3b0f92f92da..0bbaec0444d0 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -104,6 +104,7 @@ config RISCV select GENERIC_VDSO_TIME_NS if HAVE_GENERIC_VDSO select HARDIRQS_SW_RESEND select HAS_IOPORT if MMU + select HAVE_ALIGNED_STRUCT_PAGE select HAVE_ARCH_AUDITSYSCALL select HAVE_ARCH_HUGE_VMALLOC if HAVE_ARCH_HUGE_VMAP select HAVE_ARCH_HUGE_VMAP if MMU && 64BIT diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index 97b24da38897..608d98522557 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -289,4 +289,43 @@ end:; \ arch_cmpxchg_release((ptr), (o), (n)); \ }) +#ifdef CONFIG_RISCV_ISA_ZACAS + +#define system_has_cmpxchg128() \ + riscv_has_extension_unlikely(RISCV_ISA_EXT_ZACAS) + +union __u128_halves { + u128 full; + struct { + u64 low, high; + }; +}; + +#define __arch_cmpxchg128(p, o, n, cas_sfx) \ +({ \ + __typeof__(*(p)) __o = (o); \ + union __u128_halves __hn = { .full = (n) }; \ + union __u128_halves __ho = { .full = (__o) }; \ + register unsigned long x6 asm ("x6") = __hn.low; \ + register unsigned long x7 asm ("x7") = __hn.high; \ + register unsigned long x28 asm ("x28") = __ho.low; \ + register unsigned long x29 asm ("x29") = __ho.high; \ + \ + __asm__ __volatile__ ( \ + " amocas.q" cas_sfx " %0, %z3, %2" \ + : "+&r" (x28), "+&r" (x29), "+A" (*(p)) \ + : "rJ" (x6), "rJ" (x7) \ + : "memory"); \ + \ + ((u128)x29 << 64) | x28; \ +}) + +#define arch_cmpxchg128(ptr, o, n) \ + __arch_cmpxchg128((ptr), (o), (n), ".aqrl") + +#define arch_cmpxchg128_local(ptr, o, n) \ + __arch_cmpxchg128((ptr), (o), (n), "") + +#endif /* CONFIG_RISCV_ISA_ZACAS */ + #endif /* _ASM_RISCV_CMPXCHG_H */ From patchwork Wed Jul 17 06:19:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13735139 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 385EFC3DA42 for ; Wed, 17 Jul 2024 06:26:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=KT5iGaOwIjI6o8DIxzUsSahby8RT+P5FQjcB7YEiZ/w=; b=H1UKApr+PcLSYw u2VOwf4Kq3h+uBZqYFNDOIdXKzR4rCBxF1GFrRuBKkud6DyOTvbC0Pz35fk/GNjOPFDDR1f15mjEs AxjMZaDYjRc5gEE4Yt+DxvmUP6tsRZoZSpjR/jLBW5X6JnEWfRJ2Y+xiuJhdsgAqto7ord8PeFlN2 PK9Npiqu1GAtHs4lHi0GEvVlbTADo+q2Jp6RhayRUponscNp1rkxIj9S8fGFiHdDz9HD+nLw9NpFK AmwmHOFyLaeSxLy8CH1/lDVSp0rTAGnXyBOtMprB6l81SSJtfh0Wf+yxtF0CjFYBn/v+rCj19lOxA S05Nq36QVwD62srKrr8A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sTy79-0000000CpR4-2BA0; Wed, 17 Jul 2024 06:26:11 +0000 Received: from mail-wr1-x430.google.com ([2a00:1450:4864:20::430]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sTy77-0000000CpP8-0lwb for linux-riscv@lists.infradead.org; Wed, 17 Jul 2024 06:26:10 +0000 Received: by mail-wr1-x430.google.com with SMTP id ffacd0b85a97d-367ab76d5e1so2118698f8f.3 for ; Tue, 16 Jul 2024 23:26:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1721197568; x=1721802368; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3zMQJJ1+QlmFm418V6AGBGbWQY3rNoVY0h/UyfCjPDE=; b=3fP/BCAEzy724JQEqdY8dqI1f/C9d2eCRu/o78G+ogjBwfZTTwms13v+tGG21PFYby N0c0vgL8RZh4GLxJX1ggSAsmkBrryeHmAhE6zIfwkA25wqY41vjYjjT7Ti7U168tHP+T B6o4F8eWomBun1dhDl4wQDeyVcL7UB812f5i1hTRuPr93WUjIyMZeYCrdU0J0sbqeCGz 2eyujcZ4XiFbWCSTpKMZMbii6Y1wXvSqDvLkLUTsbYn7t1D73bCqMJY63dGJHNPp4GKj lYWvASuhmJCZdoANbtSZOYe3eQWU/gWsZfbXjVovK+Gcked8675mXvoxrt/7YHAdug4a idiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721197568; x=1721802368; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3zMQJJ1+QlmFm418V6AGBGbWQY3rNoVY0h/UyfCjPDE=; b=gF26ESbkCQaEW+h5YzQyVd2464noctQzOdkKaqEkJpmCa57lwSQSVUG4cRSsmyu0n6 fzJOzCspRZ/u5njDAS6EKWAhYqsGcbk8fJiF1cN+YN3BGYgYyukU94FlgnMudm4V1n55 RnYN7Heig3HEQkgF2wZGivXWiaFp1jVrcyY9OmzM9ShJEwWFpPzct4oEzPYM92T55Y0k iIPpZcq4TN5eymTM5aA+Cas+5CyHXepO/i4H940TJ+Fyusd72Rf2Q7J8DbDz0yikNROO gMbyQJ7i0Z+RLtSLSD95o+M0/NC0OfWZNMKPIdcwbskZwWy4GvzBQReB/vOAfMV1/T1G f+eQ== X-Forwarded-Encrypted: i=1; AJvYcCUrvRJRlWGWDEuVvKemp7qe1QU6bILx7nzXQdxea+2vrYru6d3p674hrI/YrjSyJK3GSqtTsDqPxiQNji1jykF34lJCNMJqDsO361pNpX1x X-Gm-Message-State: AOJu0YwdjZynv5OeP5YKHgkfgNxEZDPF+O41biDIs62aHD7QpIzCzEKz ZQ9qI10Q0hxz5M3RnStH9Qwy22aSvYrHSwSRMw5VoTjy84qx1/zWXnTMB2BKSK8= X-Google-Smtp-Source: AGHT+IE6uNjEd3rgzo2cnMZIenDe0Lf33upNb/sgH/fJlHpS9fKERnXLFFmQs0acmED+vAQAC04L2Q== X-Received: by 2002:adf:e681:0:b0:368:3079:4277 with SMTP id ffacd0b85a97d-3683165b50amr545623f8f.28.1721197567667; Tue, 16 Jul 2024 23:26:07 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-368111d3e7dsm10358022f8f.87.2024.07.16.23.26.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Jul 2024 23:26:07 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH v3 06/11] riscv: Implement xchg8/16() using Zabha Date: Wed, 17 Jul 2024 08:19:52 +0200 Message-Id: <20240717061957.140712-7-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240717061957.140712-1-alexghiti@rivosinc.com> References: <20240717061957.140712-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240716_232609_380854_5A15AD91 X-CRM114-Status: UNSURE ( 9.69 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org This adds runtime support for Zabha in xchg8/16() operations. Signed-off-by: Alexandre Ghiti --- arch/riscv/include/asm/cmpxchg.h | 32 +++++++++++++++++++++++++++++--- 1 file changed, 29 insertions(+), 3 deletions(-) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index 608d98522557..091e6612ddb3 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -11,8 +11,27 @@ #include #include -#define __arch_xchg_masked(sc_sfx, prepend, append, r, p, n) \ +#define __arch_xchg_masked(sc_sfx, swap_sfx, prepend, sc_append, \ + swap_append, r, p, n) \ ({ \ + __label__ no_zabha, end; \ + \ + if (IS_ENABLED(CONFIG_RISCV_ISA_ZABHA)) { \ + asm goto(ALTERNATIVE("j %[no_zabha]", "nop", 0, \ + RISCV_ISA_EXT_ZABHA, 1) \ + : : : : no_zabha); \ + \ + __asm__ __volatile__ ( \ + prepend \ + " amoswap" swap_sfx " %0, %z2, %1\n" \ + swap_append \ + : "=&r" (r), "+A" (*(p)) \ + : "rJ" (n) \ + : "memory"); \ + goto end; \ + } \ + \ +no_zabha:; \ u32 *__ptr32b = (u32 *)((ulong)(p) & ~0x3); \ ulong __s = ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \ ulong __mask = GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \ @@ -28,12 +47,14 @@ " or %1, %1, %z3\n" \ " sc.w" sc_sfx " %1, %1, %2\n" \ " bnez %1, 0b\n" \ - append \ + sc_append \ : "=&r" (__retx), "=&r" (__rc), "+A" (*(__ptr32b)) \ : "rJ" (__newx), "rJ" (~__mask) \ : "memory"); \ \ r = (__typeof__(*(p)))((__retx & __mask) >> __s); \ + \ +end:; \ }) #define __arch_xchg(sfx, prepend, append, r, p, n) \ @@ -56,8 +77,13 @@ \ switch (sizeof(*__ptr)) { \ case 1: \ + __arch_xchg_masked(sc_sfx, ".b" swap_sfx, \ + prepend, sc_append, swap_append, \ + __ret, __ptr, __new); \ + break; \ case 2: \ - __arch_xchg_masked(sc_sfx, prepend, sc_append, \ + __arch_xchg_masked(sc_sfx, ".h" swap_sfx, \ + prepend, sc_append, swap_append, \ __ret, __ptr, __new); \ break; \ case 4: \ From patchwork Wed Jul 17 06:19:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13735140 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AE16DC3DA4B for ; Wed, 17 Jul 2024 06:27:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=81l1vnndeH6RGQ7RI3IttTQT/Ue2c84FixhX6Y1lqPk=; b=mFAjHggOuiwBxs VQuFum0bt7It3oAddMV/ICo5kme4EiRyNd28ecmeS6N/HE3hHiBGC4FNQV4+ZbqpwRNA3GWxfVGSs qqpc4z7mpX392og1oeZyTZ0cwTWitIq6AydDsGwcwru1Qnu2sZQtkw1sQinwWP0lgNwe1ORNT0LX2 qB4JOxf/lO+1k0Uv6NHZEstYj+xlESf9HC6Ei9Or9c8QrQffYp6CZDrpqtT0QfjLDf1oPkO/Wa+2a EDTQwOHbOR4MGnZOOGW3T+vUYRggEuj0fv51DVxTEJcBjQyUgWCmw2TwYLasHu+70pwRCNFsW7Vjm JdwFgKHtYljQyf3uFs6A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sTy89-0000000Cpk0-3V8p; Wed, 17 Jul 2024 06:27:13 +0000 Received: from mail-wr1-x42a.google.com ([2a00:1450:4864:20::42a]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sTy86-0000000Cpix-3FXx for linux-riscv@lists.infradead.org; Wed, 17 Jul 2024 06:27:12 +0000 Received: by mail-wr1-x42a.google.com with SMTP id ffacd0b85a97d-367b0cc6c65so3921223f8f.3 for ; Tue, 16 Jul 2024 23:27:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1721197629; x=1721802429; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=heAbIXv1oMx4SoDoWG1mJ8ALiEwmwn+mI3czPDKQjFY=; b=WEdCOJ2lvPBK5w00zUXzqMeHbMUQSWKOy1Mt37P1J/ODiFo7w3mSvIWieqxMH/gonN m4FJxNu+QwiDZ/H598/EJFPsN1uBIj0MHQq3tzd8ROlN/ZJYKhDBTgMvaFSk1HYTvsxK 4x3klxhwR/BejDnIM8UVnnmDOm68wuB4J1/XSHsg0a51/+nqryDzNA45r07Csg+oYFrr al3CbBMd3gBI4eGJrY3kFb9hHi5dQWBQmWTvGDq8GleGfSGsjuFBhy94BRy9lDqC2nlr CLxxNJw2iV9aT8ZGGimPuRBDLRUbfLXQs8y+rS8Tp1mSuiM3gJRb1P0DWBGniQUhWNdw 9G4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721197629; x=1721802429; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=heAbIXv1oMx4SoDoWG1mJ8ALiEwmwn+mI3czPDKQjFY=; b=xI1lYC60g+NrQSPZ4dc1jsfkGDVxNpFW9Gyk8wDxcbMqmjddA+9v2t5xWC4jcargLT mEMkUzIJT5tRBLQkiuAhbrndJO1/0v69eBdaq6B6/deW5Og9NBcJWwgZ5zsclOxPH7ff ZV6mUvNoI5+2D4jMG9/f1FNwoytHC5TulsFbnmu40y45L/mxgym//KxHh1jMdju8keeJ SmIynUPO93343Q+w3Htu2PGtl4R2W4J8sPPSWQFN4XC0Y7N6sGYXseaL3qMRNUQkE4Mg V+l5jODFiPdmQuVj6YlAAzSEDKivCULVD45KFpLkQ8SS1tzLSY5MheQIWnRPpqtSdExy T9Ng== X-Forwarded-Encrypted: i=1; AJvYcCVeHc0sAVp1MyAdmvTD3uF4HdtnzyMpBl0L4sBp2I75qnHl58zw47wXiaC1IUFc3QXSbuUTnXi9xv50eJADl+uFlGhuJBu5LfOj8kw9QlmG X-Gm-Message-State: AOJu0Ywifn9fv6WQ/FJlZA7m6gK6YL4dg/qQZUnQ1A/KsLNqur9yqNxB Dh950zX7uughWGUcAZXDfwJij3CJnCObjBQzOd+r0psssp1RtZlXN05Ed9qves4= X-Google-Smtp-Source: AGHT+IEcx5vbXw9AngQayUYduH0av6OpvecXDO/fzCki4vNwqd2UbQND86b7G8WFV0E/r2hUOY9S4Q== X-Received: by 2002:adf:f6c8:0:b0:367:bb20:b3e1 with SMTP id ffacd0b85a97d-3683171f8bemr430134f8f.51.1721197628922; Tue, 16 Jul 2024 23:27:08 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-427a5e8e2ecsm160904995e9.21.2024.07.16.23.27.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Jul 2024 23:27:08 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Guo Ren Subject: [PATCH v3 07/11] asm-generic: ticket-lock: Reuse arch_spinlock_t of qspinlock Date: Wed, 17 Jul 2024 08:19:53 +0200 Message-Id: <20240717061957.140712-8-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240717061957.140712-1-alexghiti@rivosinc.com> References: <20240717061957.140712-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240716_232710_955046_449178CB X-CRM114-Status: GOOD ( 12.86 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Guo Ren The arch_spinlock_t of qspinlock has contained the atomic_t val, which satisfies the ticket-lock requirement. Thus, unify the arch_spinlock_t into qspinlock_types.h. This is the preparation for the next combo spinlock. Reviewed-by: Leonardo Bras Suggested-by: Arnd Bergmann Link: https://lore.kernel.org/linux-riscv/CAK8P3a2rnz9mQqhN6-e0CGUUv9rntRELFdxt_weiD7FxH7fkfQ@mail.gmail.com/ Signed-off-by: Guo Ren Signed-off-by: Guo Ren --- include/asm-generic/spinlock.h | 14 +++++++------- include/asm-generic/spinlock_types.h | 12 ++---------- 2 files changed, 9 insertions(+), 17 deletions(-) diff --git a/include/asm-generic/spinlock.h b/include/asm-generic/spinlock.h index 90803a826ba0..4773334ee638 100644 --- a/include/asm-generic/spinlock.h +++ b/include/asm-generic/spinlock.h @@ -32,7 +32,7 @@ static __always_inline void arch_spin_lock(arch_spinlock_t *lock) { - u32 val = atomic_fetch_add(1<<16, lock); + u32 val = atomic_fetch_add(1<<16, &lock->val); u16 ticket = val >> 16; if (ticket == (u16)val) @@ -46,31 +46,31 @@ static __always_inline void arch_spin_lock(arch_spinlock_t *lock) * have no outstanding writes due to the atomic_fetch_add() the extra * orderings are free. */ - atomic_cond_read_acquire(lock, ticket == (u16)VAL); + atomic_cond_read_acquire(&lock->val, ticket == (u16)VAL); smp_mb(); } static __always_inline bool arch_spin_trylock(arch_spinlock_t *lock) { - u32 old = atomic_read(lock); + u32 old = atomic_read(&lock->val); if ((old >> 16) != (old & 0xffff)) return false; - return atomic_try_cmpxchg(lock, &old, old + (1<<16)); /* SC, for RCsc */ + return atomic_try_cmpxchg(&lock->val, &old, old + (1<<16)); /* SC, for RCsc */ } static __always_inline void arch_spin_unlock(arch_spinlock_t *lock) { u16 *ptr = (u16 *)lock + IS_ENABLED(CONFIG_CPU_BIG_ENDIAN); - u32 val = atomic_read(lock); + u32 val = atomic_read(&lock->val); smp_store_release(ptr, (u16)val + 1); } static __always_inline int arch_spin_value_unlocked(arch_spinlock_t lock) { - u32 val = lock.counter; + u32 val = lock.val.counter; return ((val >> 16) == (val & 0xffff)); } @@ -84,7 +84,7 @@ static __always_inline int arch_spin_is_locked(arch_spinlock_t *lock) static __always_inline int arch_spin_is_contended(arch_spinlock_t *lock) { - u32 val = atomic_read(lock); + u32 val = atomic_read(&lock->val); return (s16)((val >> 16) - (val & 0xffff)) > 1; } diff --git a/include/asm-generic/spinlock_types.h b/include/asm-generic/spinlock_types.h index 8962bb730945..f534aa5de394 100644 --- a/include/asm-generic/spinlock_types.h +++ b/include/asm-generic/spinlock_types.h @@ -3,15 +3,7 @@ #ifndef __ASM_GENERIC_SPINLOCK_TYPES_H #define __ASM_GENERIC_SPINLOCK_TYPES_H -#include -typedef atomic_t arch_spinlock_t; - -/* - * qrwlock_types depends on arch_spinlock_t, so we must typedef that before the - * include. - */ -#include - -#define __ARCH_SPIN_LOCK_UNLOCKED ATOMIC_INIT(0) +#include +#include #endif /* __ASM_GENERIC_SPINLOCK_TYPES_H */ From patchwork Wed Jul 17 06:19:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13735141 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 78F02C3DA42 for ; Wed, 17 Jul 2024 06:28:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Ch1fU7HxtxblHEnw2eK8E63qnmnxpbMDLgaMVzTElew=; b=fFtLIJrLdNIr0R 74wBBD8gEC8Zo+Lc5Nh8+Q+vPLAXFRJjNNUyiREBwo1S+DTI4sBa/Phdg+l+7jo5zSBg7IyXrzJlf XetvkiGFjOcuYuNKBEG2Oim0WSDpEzxGz98+VjOxW+Ojx26v3rRq1Qpk9ruY3DsAKcbQf9L2LAECn cA894uS9xt/OfXchgH2n8kqrCw1HyppLCttfhSUZFBb0CilbSULtsB7z2FSwIzJ2ODMt7QgO+WErl u87RPc8M6gcqbCUHkOmvCxKa7qzVZE8HNSrY6BShi1eBd4G/msudByJOc8rAa2UqGBRZVaoulYa7s FqGJ2ieRh0ta7k/3DUfA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sTy99-0000000Cpzh-0OX3; Wed, 17 Jul 2024 06:28:15 +0000 Received: from mail-wr1-x42a.google.com ([2a00:1450:4864:20::42a]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sTy95-0000000CpxQ-2xDN for linux-riscv@lists.infradead.org; Wed, 17 Jul 2024 06:28:13 +0000 Received: by mail-wr1-x42a.google.com with SMTP id ffacd0b85a97d-367ab76d5e1so2119322f8f.3 for ; Tue, 16 Jul 2024 23:28:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1721197690; x=1721802490; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=JpMFH6VmSNUk91jIQ+0kjGx11FiPCz2ltKgHshZWUPg=; b=2fCa6ZYMB8Tl7dm8G+kbR7wyS7xr3zdDvnnnf6/c2wwxFI96JcgE9v3MFXc1Etb+uC wjNkE3B3CpMQp6Q9Ed91QUQ6lxKzwP3YrqUUVQ8ho7BDclGfIWEgxjvYjpFJkGw0Y9Wo fXipFCivni4C8WyITGOJ9llkCH1i29QC/gaqLiRX1aB0GXv8+ZI7QyugemIUw/mIGbSx tnJ7aXiAopk4tIJn1/bATXXLpAx3UU28tPPNaHLlHzQv/yR3pDE4hlr4avL4FimHSkwG yyQ+ySGrbmcrV2/ccw8jAiApEkf/EJy0e5Be/nURcJyFMnsiaAAzNeaaMoDyyBx5Fgo7 Nn8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721197690; x=1721802490; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JpMFH6VmSNUk91jIQ+0kjGx11FiPCz2ltKgHshZWUPg=; b=pjC6T6VHACFtlZnsyWGIEWdmxozmqGxpNMKtiHmD76H2COA1wkWaj3JF+9DyrsqOwc c7VUoNL00Kb+XQks6N285aLXitgfZDNIH1W7O4hTWbFrGBLe5+qrCZPlP+JlXshjETmB /9HieJTIjoTZUPHVHMzFL+Rw7Y9wXEqADNq/K2sxWu8988epLAiUbGruyOccxDFZ2GRS K1ivOVBYSrv23eucGivhl9AowoLGaaXdC3StFbQzNITFMi66X1c0utpPbX+fI5ji/rVt hTAU+t65vBWStZCJjGEGmfSCdGUHkVRgnA0HKGNtHu2Rjaoqxa7S6cxZ5bXDcNI8Rwtg +aqA== X-Forwarded-Encrypted: i=1; AJvYcCUfuBbGH2xvqBtWaJ6aeBUPjho9cBx9/o0CIjWy9BeNZVufFX2gKiMNu/lmFd5k9G+/zcPXQlxDQ4s7q4MjtYxBIdcgUYLwrBVBJp/DdZp9 X-Gm-Message-State: AOJu0YyjKhaCdZOYTVjCfcbSc9Xw7au8T3Yc4Yv3Y/gy730n+xlsAlij i4vpyHx9PK4VJjev7WeMW56Jojlkk/E7eVyBP+z0qBjH3WtlJ/L3PkwWhkpSm08= X-Google-Smtp-Source: AGHT+IFM/AGZ5F1dS74/5pYkBU8cAcbJDWrIIu9c3nYaG0zPFdfcFYCGZfuTjOfFxaw1xsX/SOzMGA== X-Received: by 2002:adf:e508:0:b0:366:f041:935d with SMTP id ffacd0b85a97d-3683179ebc7mr379203f8f.60.1721197690168; Tue, 16 Jul 2024 23:28:10 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3680daccacdsm10824862f8f.62.2024.07.16.23.28.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Jul 2024 23:28:09 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Guo Ren Subject: [PATCH v3 08/11] asm-generic: ticket-lock: Add separate ticket-lock.h Date: Wed, 17 Jul 2024 08:19:54 +0200 Message-Id: <20240717061957.140712-9-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240717061957.140712-1-alexghiti@rivosinc.com> References: <20240717061957.140712-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240716_232811_784574_1CCF9B33 X-CRM114-Status: GOOD ( 21.55 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Guo Ren Add a separate ticket-lock.h to include multiple spinlock versions and select one at compile time or runtime. Reviewed-by: Leonardo Bras Suggested-by: Arnd Bergmann Link: https://lore.kernel.org/linux-riscv/CAK8P3a2rnz9mQqhN6-e0CGUUv9rntRELFdxt_weiD7FxH7fkfQ@mail.gmail.com/ Signed-off-by: Guo Ren Signed-off-by: Guo Ren --- include/asm-generic/spinlock.h | 87 +--------------------- include/asm-generic/ticket_spinlock.h | 103 ++++++++++++++++++++++++++ 2 files changed, 104 insertions(+), 86 deletions(-) create mode 100644 include/asm-generic/ticket_spinlock.h diff --git a/include/asm-generic/spinlock.h b/include/asm-generic/spinlock.h index 4773334ee638..970590baf61b 100644 --- a/include/asm-generic/spinlock.h +++ b/include/asm-generic/spinlock.h @@ -1,94 +1,9 @@ /* SPDX-License-Identifier: GPL-2.0 */ -/* - * 'Generic' ticket-lock implementation. - * - * It relies on atomic_fetch_add() having well defined forward progress - * guarantees under contention. If your architecture cannot provide this, stick - * to a test-and-set lock. - * - * It also relies on atomic_fetch_add() being safe vs smp_store_release() on a - * sub-word of the value. This is generally true for anything LL/SC although - * you'd be hard pressed to find anything useful in architecture specifications - * about this. If your architecture cannot do this you might be better off with - * a test-and-set. - * - * It further assumes atomic_*_release() + atomic_*_acquire() is RCpc and hence - * uses atomic_fetch_add() which is RCsc to create an RCsc hot path, along with - * a full fence after the spin to upgrade the otherwise-RCpc - * atomic_cond_read_acquire(). - * - * The implementation uses smp_cond_load_acquire() to spin, so if the - * architecture has WFE like instructions to sleep instead of poll for word - * modifications be sure to implement that (see ARM64 for example). - * - */ - #ifndef __ASM_GENERIC_SPINLOCK_H #define __ASM_GENERIC_SPINLOCK_H -#include -#include - -static __always_inline void arch_spin_lock(arch_spinlock_t *lock) -{ - u32 val = atomic_fetch_add(1<<16, &lock->val); - u16 ticket = val >> 16; - - if (ticket == (u16)val) - return; - - /* - * atomic_cond_read_acquire() is RCpc, but rather than defining a - * custom cond_read_rcsc() here we just emit a full fence. We only - * need the prior reads before subsequent writes ordering from - * smb_mb(), but as atomic_cond_read_acquire() just emits reads and we - * have no outstanding writes due to the atomic_fetch_add() the extra - * orderings are free. - */ - atomic_cond_read_acquire(&lock->val, ticket == (u16)VAL); - smp_mb(); -} - -static __always_inline bool arch_spin_trylock(arch_spinlock_t *lock) -{ - u32 old = atomic_read(&lock->val); - - if ((old >> 16) != (old & 0xffff)) - return false; - - return atomic_try_cmpxchg(&lock->val, &old, old + (1<<16)); /* SC, for RCsc */ -} - -static __always_inline void arch_spin_unlock(arch_spinlock_t *lock) -{ - u16 *ptr = (u16 *)lock + IS_ENABLED(CONFIG_CPU_BIG_ENDIAN); - u32 val = atomic_read(&lock->val); - - smp_store_release(ptr, (u16)val + 1); -} - -static __always_inline int arch_spin_value_unlocked(arch_spinlock_t lock) -{ - u32 val = lock.val.counter; - - return ((val >> 16) == (val & 0xffff)); -} - -static __always_inline int arch_spin_is_locked(arch_spinlock_t *lock) -{ - arch_spinlock_t val = READ_ONCE(*lock); - - return !arch_spin_value_unlocked(val); -} - -static __always_inline int arch_spin_is_contended(arch_spinlock_t *lock) -{ - u32 val = atomic_read(&lock->val); - - return (s16)((val >> 16) - (val & 0xffff)) > 1; -} - +#include #include #endif /* __ASM_GENERIC_SPINLOCK_H */ diff --git a/include/asm-generic/ticket_spinlock.h b/include/asm-generic/ticket_spinlock.h new file mode 100644 index 000000000000..cfcff22b37b3 --- /dev/null +++ b/include/asm-generic/ticket_spinlock.h @@ -0,0 +1,103 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +/* + * 'Generic' ticket-lock implementation. + * + * It relies on atomic_fetch_add() having well defined forward progress + * guarantees under contention. If your architecture cannot provide this, stick + * to a test-and-set lock. + * + * It also relies on atomic_fetch_add() being safe vs smp_store_release() on a + * sub-word of the value. This is generally true for anything LL/SC although + * you'd be hard pressed to find anything useful in architecture specifications + * about this. If your architecture cannot do this you might be better off with + * a test-and-set. + * + * It further assumes atomic_*_release() + atomic_*_acquire() is RCpc and hence + * uses atomic_fetch_add() which is RCsc to create an RCsc hot path, along with + * a full fence after the spin to upgrade the otherwise-RCpc + * atomic_cond_read_acquire(). + * + * The implementation uses smp_cond_load_acquire() to spin, so if the + * architecture has WFE like instructions to sleep instead of poll for word + * modifications be sure to implement that (see ARM64 for example). + * + */ + +#ifndef __ASM_GENERIC_TICKET_SPINLOCK_H +#define __ASM_GENERIC_TICKET_SPINLOCK_H + +#include +#include + +static __always_inline void ticket_spin_lock(arch_spinlock_t *lock) +{ + u32 val = atomic_fetch_add(1<<16, &lock->val); + u16 ticket = val >> 16; + + if (ticket == (u16)val) + return; + + /* + * atomic_cond_read_acquire() is RCpc, but rather than defining a + * custom cond_read_rcsc() here we just emit a full fence. We only + * need the prior reads before subsequent writes ordering from + * smb_mb(), but as atomic_cond_read_acquire() just emits reads and we + * have no outstanding writes due to the atomic_fetch_add() the extra + * orderings are free. + */ + atomic_cond_read_acquire(&lock->val, ticket == (u16)VAL); + smp_mb(); +} + +static __always_inline bool ticket_spin_trylock(arch_spinlock_t *lock) +{ + u32 old = atomic_read(&lock->val); + + if ((old >> 16) != (old & 0xffff)) + return false; + + return atomic_try_cmpxchg(&lock->val, &old, old + (1<<16)); /* SC, for RCsc */ +} + +static __always_inline void ticket_spin_unlock(arch_spinlock_t *lock) +{ + u16 *ptr = (u16 *)lock + IS_ENABLED(CONFIG_CPU_BIG_ENDIAN); + u32 val = atomic_read(&lock->val); + + smp_store_release(ptr, (u16)val + 1); +} + +static __always_inline int ticket_spin_value_unlocked(arch_spinlock_t lock) +{ + u32 val = lock.val.counter; + + return ((val >> 16) == (val & 0xffff)); +} + +static __always_inline int ticket_spin_is_locked(arch_spinlock_t *lock) +{ + arch_spinlock_t val = READ_ONCE(*lock); + + return !ticket_spin_value_unlocked(val); +} + +static __always_inline int ticket_spin_is_contended(arch_spinlock_t *lock) +{ + u32 val = atomic_read(&lock->val); + + return (s16)((val >> 16) - (val & 0xffff)) > 1; +} + +/* + * Remapping spinlock architecture specific functions to the corresponding + * ticket spinlock functions. + */ +#define arch_spin_is_locked(l) ticket_spin_is_locked(l) +#define arch_spin_is_contended(l) ticket_spin_is_contended(l) +#define arch_spin_value_unlocked(l) ticket_spin_value_unlocked(l) +#define arch_spin_lock(l) ticket_spin_lock(l) +#define arch_spin_trylock(l) ticket_spin_trylock(l) +#define arch_spin_unlock(l) ticket_spin_unlock(l) + +#endif /* __ASM_GENERIC_TICKET_SPINLOCK_H */ From patchwork Wed Jul 17 06:19:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13735142 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 18AF3C3DA4B for ; Wed, 17 Jul 2024 06:29:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=ZRfBXR/uvQ9ppuE6GNmcPtoTTzop/NOFbxTgzcz8tM0=; b=x6M5Xeo2j5O5aO JPQb2x3QJfWNmRBGWSVQjFhaR3mI7K49msgs2MU0yu0d0BbzcVuFFRZltr+OyoDU7W5XFfqAUgGwy rlcYmeQmEslUgD1jEvEhtHYpb6cts8yCtZndjJLw23dkefPDvWPBBye9vMI+i5u7cMqEJtYlxSPyw ZFQ6DaL8xZUmILkfIsgRvC14DYAa/SeprMx/DdnWTTjxK9aiPUMH5QVDWWfIrWKIiB+ogwL6pf5Yu xGxMytmt545OKb2pU9GL9gNqvzZQaZ/2k6wjCbhJ/HQ3vhOxdIIDK0+V3uTeM8ZP3almMBh7V9Bc1 kAkAh4qMGIc+yozV9l9g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sTyA8-0000000CqK1-24FM; Wed, 17 Jul 2024 06:29:16 +0000 Received: from mail-wm1-x336.google.com ([2a00:1450:4864:20::336]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sTyA5-0000000CqJ4-0umX for linux-riscv@lists.infradead.org; Wed, 17 Jul 2024 06:29:14 +0000 Received: by mail-wm1-x336.google.com with SMTP id 5b1f17b1804b1-4279ca8af51so40471905e9.3 for ; Tue, 16 Jul 2024 23:29:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1721197751; x=1721802551; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qV10XIriQxrkSLT6hfJ9bzuJQGW5jGWUcHq8Bf3DvNw=; b=p/b34UEpTOWRz5pn4FRSOTyHWtU66U4EBdrM0VOYszGTrFgAVT2CeR0eK8NhG22A1u BFzfh6oEqrAC7VQY+zpX4sQFQJR2FQ9hM2WaYsKJJ9elxe7DtGvII6NdOqoY6CgjaJSH eSb7Kxje0qI9u1iNzwRbCveJxqpRq/vtLQx7k3V1SIxYKEaDzru1IAO7C+Wxe/W2taAZ TmhtykQ20D1na8+pAyNpaCaXUg6tPAf/hDMdGI16hthDktnzJDgt5mtJOQI0GYsEXUnf 21YwGGxNYU+Fg4cue4jC5bUIPgsLKx2zJsdzort6ThTADeUIIC2bev4wdW6J7l0A9tAt oVyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721197751; x=1721802551; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qV10XIriQxrkSLT6hfJ9bzuJQGW5jGWUcHq8Bf3DvNw=; b=S4RZf/hiHA24MjfYDyplX9BL38JOoKApxcBZBrp2Aq/VQrvfvakTwgD9r5X1grOmnm rQZvSJ4nobPnzb6/y3ZacLft41P+g6vTc9Tr+8m5iMU7eflOrB0/xLHRiWeKldQN19I4 dNWdIzJ7hgM/Eqni2VvLqemxKsz8owAFbjpA1dw0gdOYhL/9TmgCeU1MFGlpeZSHGLyp SkrK/pPJskRwa3yCvPFJlK1im4iU+mQzP/otZk7vvVonItIxk8lHV5aMmNyMDGOMT58R QiIaGWNXv7vcnOZE2qIO62C03t4jmLEEik6Shi0XC1zocWBN25ONqF/KeSzntP4CfKh2 4wPw== X-Forwarded-Encrypted: i=1; AJvYcCW+KruYWI5aFY8i2MogXMAIjxyB0ihIrD/wx73GkpT39BLB2PiPiLIM1Ii5RWr5mMDW7bgwsP6Z3Owsri9QoTYr+1Yp4uLOf/rhs/6RoB+R X-Gm-Message-State: AOJu0YxT/XKLdssEyFqJ1S8T5tNFPP2U7waXQ2lXgDpb+TjLVbEIARoX D5Py0UBftrYrgmOFy9lQ2PA2HQoPgCU9U5AWHrQsjCHRtg8ysjuAaJSn7ClHxL4= X-Google-Smtp-Source: AGHT+IGP/cuCt+F0ta7uxUlFKhwXsnKpljMYZclyaPYQLOpIG5TT5vjECTCQ1Rcr5BTRTziHhuytmQ== X-Received: by 2002:adf:fa03:0:b0:368:303b:8fe7 with SMTP id ffacd0b85a97d-36831600243mr520432f8f.7.1721197751437; Tue, 16 Jul 2024 23:29:11 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3680daccad8sm10840137f8f.60.2024.07.16.23.29.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Jul 2024 23:29:11 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH v3 09/11] riscv: Add ISA extension parsing for Ziccrse Date: Wed, 17 Jul 2024 08:19:55 +0200 Message-Id: <20240717061957.140712-10-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240717061957.140712-1-alexghiti@rivosinc.com> References: <20240717061957.140712-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240716_232913_266946_869FA688 X-CRM114-Status: GOOD ( 10.20 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Add support to parse the Ziccrse string in the riscv,isa string. Signed-off-by: Alexandre Ghiti --- arch/riscv/include/asm/hwcap.h | 1 + arch/riscv/kernel/cpufeature.c | 1 + 2 files changed, 2 insertions(+) diff --git a/arch/riscv/include/asm/hwcap.h b/arch/riscv/include/asm/hwcap.h index f71ddd2ca163..863b9b7d4a4f 100644 --- a/arch/riscv/include/asm/hwcap.h +++ b/arch/riscv/include/asm/hwcap.h @@ -82,6 +82,7 @@ #define RISCV_ISA_EXT_ZACAS 73 #define RISCV_ISA_EXT_XANDESPMU 74 #define RISCV_ISA_EXT_ZABHA 75 +#define RISCV_ISA_EXT_ZICCRSE 76 #define RISCV_ISA_EXT_XLINUXENVCFG 127 diff --git a/arch/riscv/kernel/cpufeature.c b/arch/riscv/kernel/cpufeature.c index c125d82c894b..93d8cc7e232c 100644 --- a/arch/riscv/kernel/cpufeature.c +++ b/arch/riscv/kernel/cpufeature.c @@ -306,6 +306,7 @@ const struct riscv_isa_ext_data riscv_isa_ext[] = { __RISCV_ISA_EXT_DATA(svnapot, RISCV_ISA_EXT_SVNAPOT), __RISCV_ISA_EXT_DATA(svpbmt, RISCV_ISA_EXT_SVPBMT), __RISCV_ISA_EXT_DATA(xandespmu, RISCV_ISA_EXT_XANDESPMU), + __RISCV_ISA_EXT_DATA(ziccrse, RISCV_ISA_EXT_ZICCRSE), }; const size_t riscv_isa_ext_count = ARRAY_SIZE(riscv_isa_ext); From patchwork Wed Jul 17 06:19:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13735156 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DD1E0C3DA42 for ; Wed, 17 Jul 2024 06:30:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=O0JB2SWSbLFKvpWRYBnQTgisIUXBj9XTP6KM9HI3u0k=; b=Azc+JpueI743ZT ebQz2RsdnNwrv9yfngKVb09pcGKfEbBDraGqRTHCQDua3nuLubeO/VnO/ZI0tJs6iUdCSLNLVRuoZ AUzlDA8ydrwveA5gHyosJGVYrAp16NF5jRYQJA5PJvahU3umr5hOcjG+P0VTHmkI0jBN1kKttCrG0 FReTrxDFaA+VT0/j8sFEAdGKBEEtfi4dKkfNebvGs2INTrwe1FL0LPBbmQ4tWYFWAy4mKCJyxLBpv sggSBBCr6TQiHKFxaCIUrDMKFxeswcWAFDWF1mcdRHtkTH+5kJhIxICHNkqgWQt7D1WPlV7T+BYIB 2mY3o86wdH6EEPQ+jIJg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sTyB6-0000000CqcF-3e61; Wed, 17 Jul 2024 06:30:16 +0000 Received: from mail-wm1-x32d.google.com ([2a00:1450:4864:20::32d]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sTyB4-0000000Cqan-15Fe for linux-riscv@lists.infradead.org; Wed, 17 Jul 2024 06:30:16 +0000 Received: by mail-wm1-x32d.google.com with SMTP id 5b1f17b1804b1-4267300145eso51278445e9.3 for ; Tue, 16 Jul 2024 23:30:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1721197813; x=1721802613; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=r/a3I9RG+3lbcPTW2WISOokc8gewDy3fkMsPDZhy/D4=; b=N42oS91AhHzkKUMo9Tid74LqHfSBv2OLbnn8rVeF1wmdOAsywtASCw4AEaglFPqoWM sjNsCco6vjnbB3mbG0kiUDGTjv2RV9QUOTyrIvW+BXa8qAtuVkeYV3/TQ15AD7AKlnjD MbhmEGFNkcSAvJzQhHoBmfa5xlambFBwmUCbnKBw7/pdyBvxzzSkiqD2mSupUhPWt3Qc e3Uw/xZsg5EcGg3cmgsofSXy4yfGW2FrwZv3Ebntd5NcZUmvZMI6XEKegrU41TPwvW82 fd7nb/v+B0BO9SD2yLGknXRxjYNu29lgWSdzfJIYDz24W3o0ZeyFGIvntNuTbzKtjyYS m/rA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721197813; x=1721802613; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=r/a3I9RG+3lbcPTW2WISOokc8gewDy3fkMsPDZhy/D4=; b=psDdkzDFMbarzogh6w0cP5Zwx10cyYeal/ISZlRze/YeZW1qK31f1kPawuZBz+DyFt jqUQ9zs8AIfjsUReFc5BkucUZJRvKz/SkVHtM9coYMHaffFeqOkV4LBEOea0kVPVghX7 XzTZZ6s2HJrGtWYkh8ER+QiWiV4RcsMupr6Sa3zADquzZlZtRh3FuTDLhAQm2eRVcImh m/jY6INzcD4Bdy92vVo8yemNEa2uAUzTtf6h0XDGSRtwO6bgg78mlcsKXXkkMnJVMTmI wJq9vR43cwsFXT3e0Z16PC6EBvsE3heC62oueYU/JMuBjVzxj2pWpDAD4oWamF1jsDfD r+Fg== X-Forwarded-Encrypted: i=1; AJvYcCXaEteMxuZ8voXUXm8HflFJWyr4k8lPHf8gRszZUTqxeAwl2PbQ2gSN1HTbz43dgbwJiDen5HPUkwvzKM+QYesKx91VqDScaXMVJ+eih1Tr X-Gm-Message-State: AOJu0YwWlBH6ubzgiE3a0y9iDeUK3/F40+R8l/0s+TKdLZykiDqPjAS/ ATCjhTng/xem36uUyx/eNMoePhUR4YsJUqjbxE+9eH9v62HkYWnjoCZRG4FOPcE= X-Google-Smtp-Source: AGHT+IHes7c6JkEf7yEOk72RnIcqQqTVYNuy3gJbWA9je9hHeJGPheqNOo3gwFeqz1xI+QDtIGxtPA== X-Received: by 2002:a05:600c:4ec9:b0:426:62c5:4731 with SMTP id 5b1f17b1804b1-427c2d01cafmr5720315e9.29.1721197812676; Tue, 16 Jul 2024 23:30:12 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-427a5ef460csm158127685e9.44.2024.07.16.23.30.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Jul 2024 23:30:12 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH v3 10/11] dt-bindings: riscv: Add Ziccrse ISA extension description Date: Wed, 17 Jul 2024 08:19:56 +0200 Message-Id: <20240717061957.140712-11-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240717061957.140712-1-alexghiti@rivosinc.com> References: <20240717061957.140712-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240716_233014_331824_EFB41FC3 X-CRM114-Status: UNSURE ( 8.77 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Add description for the Ziccrse ISA extension which was introduced in the riscv profiles specification v0.9.2. Signed-off-by: Alexandre Ghiti Reviewed-by: Guo Ren --- Documentation/devicetree/bindings/riscv/extensions.yaml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/Documentation/devicetree/bindings/riscv/extensions.yaml b/Documentation/devicetree/bindings/riscv/extensions.yaml index e6436260bdeb..b08bf1a8d8f8 100644 --- a/Documentation/devicetree/bindings/riscv/extensions.yaml +++ b/Documentation/devicetree/bindings/riscv/extensions.yaml @@ -245,6 +245,12 @@ properties: in commit 64074bc ("Update version numbers for Zfh/Zfinx") of riscv-isa-manual. + - const: ziccrse + description: + The standard Ziccrse extension which provides forward progress + guarantee on LR/SC sequences, as introduced in the riscv profiles + specification v0.9.2. + - const: zk description: The standard Zk Standard Scalar cryptography extension as ratified From patchwork Wed Jul 17 06:19:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13735157 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B0845C3DA42 for ; Wed, 17 Jul 2024 06:31:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=RSsKEkMD8ZzrOuBEjhs7PabDSZ53LyNKxJztvVa1PMQ=; b=wRP3z67IhCy7cc ooraB224Redt8qpyO6SJ81V2fXGKT83m0XSe108p1ryNaDyYEu+gpLSTVy/ZTBBYVmtnWhFjXBXPn /emO10b6awg0BY3irOFmPgoXLNvnUthPJOAcJlOciSdYjW1qdLdxBV3vZAAemoLR1QV+gWzpd3CXB zIEXQb8NPUfIyZ7le2ZBoVcYkNccRbDBH7ekqiHYmHQ5gLKT6YHVhM4fx6zxm2PWh1xq+ZcyfeOlg GXy1dxP7oJ5ewovYlMHVsNYCiC+79WgHEH4n6pZFpqN8877+0hZzSzQ/iTf99FHYIYcvn32fWo+kP IfbX50oazKIFHihLRrTw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sTyC5-0000000CqpN-42u8; Wed, 17 Jul 2024 06:31:17 +0000 Received: from mail-wm1-x32d.google.com ([2a00:1450:4864:20::32d]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sTyC3-0000000CqoS-2MMr for linux-riscv@lists.infradead.org; Wed, 17 Jul 2024 06:31:17 +0000 Received: by mail-wm1-x32d.google.com with SMTP id 5b1f17b1804b1-4266eda81c5so51445095e9.0 for ; Tue, 16 Jul 2024 23:31:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1721197874; x=1721802674; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kdclfDAD0BbbW9A8JO1TG59iSyD1aGCH+ewt/4gE4jc=; b=BpPhg29q85cSw7+KjfUoIiKraFv2aYLX9poUMxNiOS+RAtVYhKYrc2U4UCFijHvWN4 j1+8bC+Sb581vgDRsD72HwA5gBRzoTPVam0LSdNrt+aPxAr+40J8iA7KpDjCai0YJaPi XKKKKUvD9MF1Vb/oDFbVtHnyyiRbStXqPM3NMUkMx9hGesVmfecoU73hnLCg+ABD/fgB Q/dnIPKiqw5EDUURVvG5eRdVS72xzfEWlVGYFbgHcPoD6siv6hlttNswMZytEB2Axu4N h6ad4oWjXk7pH83WBxjaSjW1fRcoU5KW2+LYMhS03pixF+9iqBEc4YIQpzBHWqmOTRlB ONtQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721197874; x=1721802674; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kdclfDAD0BbbW9A8JO1TG59iSyD1aGCH+ewt/4gE4jc=; b=J7sdqq3XTn8M21PfYj3ou6ZRWkDURpO0+dG9Imuo6NUHgUTkzDv+r9CqguMJQGyt5o EZtFGGmLQzZV1HvPQQjEmmutHL8seabmDZNCpRuBMaV6FVhij2wsgnTIHSTS6RCV3vWv usX1e/i3bPDN0s4XjFBpU5pBWOk4+CHdwRS1lN3h9GfMh5FqMaP8fvUvkRn5+JaGALST KPMeRspruqEiBsD6vTqaibnP6Wej0cL/oPh2bPsa93DFRZZvzyHcu0IZX5ASzgHsRAJS dRpb/TBDGCqg3S3flvY5l1kCEM+kD2WYRY6u3x8zoWEN8OLBafXeSxJJd7rLXu1oEWCF DJ4g== X-Forwarded-Encrypted: i=1; AJvYcCWf92c2rx8oQtB1g5DRFxf3h/SibxZgcBa9Vagir4qzrjce54RRVMIhHf7GJdSPGEMvW5DmuH+wsebvRPnW4XxJ901hnhQ8yud8B/0Noy9t X-Gm-Message-State: AOJu0YyWI+YSXvxNtU8NPu1mBBpD8e1wdXfTXctnEWVdDXsV47JyhgVg /PKBRtatZ6QMcV/OI7YBsYSojBi1Jl9G27sn0ll4AKaTil/rXSRn8qOi1H1SrKc= X-Google-Smtp-Source: AGHT+IEb50phVixq0u+U2husY+tMg8VyZk4HDBQadM7JWcW6QpG95LwceBZFv+wDTFAxqwuJF0ajmQ== X-Received: by 2002:a05:600c:3544:b0:426:6822:5aa8 with SMTP id 5b1f17b1804b1-427c2cc8481mr4819935e9.18.1721197874118; Tue, 16 Jul 2024 23:31:14 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-427a5e996c1sm159710625e9.25.2024.07.16.23.31.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Jul 2024 23:31:13 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH v3 11/11] riscv: Add qspinlock support Date: Wed, 17 Jul 2024 08:19:57 +0200 Message-Id: <20240717061957.140712-12-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240717061957.140712-1-alexghiti@rivosinc.com> References: <20240717061957.140712-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240716_233115_636313_9527095B X-CRM114-Status: GOOD ( 23.24 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org In order to produce a generic kernel, a user can select CONFIG_COMBO_SPINLOCKS which will fallback at runtime to the ticket spinlock implementation if Zabha or Ziccrse are not present. Note that we can't use alternatives here because the discovery of extensions is done too late and we need to start with the qspinlock implementation because the ticket spinlock implementation would pollute the spinlock value, so let's use static keys. This is largely based on Guo's work and Leonardo reviews at [1]. Link: https://lore.kernel.org/linux-riscv/20231225125847.2778638-1-guoren@kernel.org/ [1] Signed-off-by: Guo Ren Signed-off-by: Alexandre Ghiti --- .../locking/queued-spinlocks/arch-support.txt | 2 +- arch/riscv/Kconfig | 29 ++++++++++++++ arch/riscv/include/asm/Kbuild | 4 +- arch/riscv/include/asm/spinlock.h | 39 +++++++++++++++++++ arch/riscv/kernel/setup.c | 33 ++++++++++++++++ include/asm-generic/qspinlock.h | 2 + include/asm-generic/ticket_spinlock.h | 2 + 7 files changed, 109 insertions(+), 2 deletions(-) create mode 100644 arch/riscv/include/asm/spinlock.h diff --git a/Documentation/features/locking/queued-spinlocks/arch-support.txt b/Documentation/features/locking/queued-spinlocks/arch-support.txt index 22f2990392ff..cf26042480e2 100644 --- a/Documentation/features/locking/queued-spinlocks/arch-support.txt +++ b/Documentation/features/locking/queued-spinlocks/arch-support.txt @@ -20,7 +20,7 @@ | openrisc: | ok | | parisc: | TODO | | powerpc: | ok | - | riscv: | TODO | + | riscv: | ok | | s390: | TODO | | sh: | TODO | | sparc: | ok | diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 0bbaec0444d0..5040c7eac70d 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -72,6 +72,7 @@ config RISCV select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP select ARCH_WANTS_NO_INSTR select ARCH_WANTS_THP_SWAP if HAVE_ARCH_TRANSPARENT_HUGEPAGE + select ARCH_WEAK_RELEASE_ACQUIRE if ARCH_USE_QUEUED_SPINLOCKS select BINFMT_FLAT_NO_DATA_START_OFFSET if !MMU select BUILDTIME_TABLE_SORT if MMU select CLINT_TIMER if RISCV_M_MODE @@ -482,6 +483,34 @@ config NODES_SHIFT Specify the maximum number of NUMA Nodes available on the target system. Increases memory reserved to accommodate various tables. +choice + prompt "RISC-V spinlock type" + default RISCV_COMBO_SPINLOCKS + +config RISCV_TICKET_SPINLOCKS + bool "Using ticket spinlock" + +config RISCV_QUEUED_SPINLOCKS + bool "Using queued spinlock" + depends on SMP && MMU + select ARCH_USE_QUEUED_SPINLOCKS + help + The queued spinlock implementation requires the forward progress + guarantee of cmpxchg()/xchg() atomic operations: CAS with Zabha or + LR/SC with Ziccrse provide such guarantee. + + Select this if and only if Zabha or Ziccrse is available on your + platform. + +config RISCV_COMBO_SPINLOCKS + bool "Using combo spinlock" + depends on SMP && MMU + select ARCH_USE_QUEUED_SPINLOCKS + help + Embed both queued spinlock and ticket lock so that the spinlock + implementation can be chosen at runtime. +endchoice + config RISCV_ALTERNATIVE bool depends on !XIP_KERNEL diff --git a/arch/riscv/include/asm/Kbuild b/arch/riscv/include/asm/Kbuild index 504f8b7e72d4..ad72f2bd4cc9 100644 --- a/arch/riscv/include/asm/Kbuild +++ b/arch/riscv/include/asm/Kbuild @@ -2,10 +2,12 @@ generic-y += early_ioremap.h generic-y += flat.h generic-y += kvm_para.h +generic-y += mcs_spinlock.h generic-y += parport.h -generic-y += spinlock.h generic-y += spinlock_types.h +generic-y += ticket_spinlock.h generic-y += qrwlock.h generic-y += qrwlock_types.h +generic-y += qspinlock.h generic-y += user.h generic-y += vmlinux.lds.h diff --git a/arch/riscv/include/asm/spinlock.h b/arch/riscv/include/asm/spinlock.h new file mode 100644 index 000000000000..4856d50006f2 --- /dev/null +++ b/arch/riscv/include/asm/spinlock.h @@ -0,0 +1,39 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef __ASM_RISCV_SPINLOCK_H +#define __ASM_RISCV_SPINLOCK_H + +#ifdef CONFIG_RISCV_COMBO_SPINLOCKS +#define _Q_PENDING_LOOPS (1 << 9) + +#define __no_arch_spinlock_redefine +#include +#include +#include + +DECLARE_STATIC_KEY_TRUE(qspinlock_key); + +#define SPINLOCK_BASE_DECLARE(op, type, type_lock) \ +static __always_inline type arch_spin_##op(type_lock lock) \ +{ \ + if (static_branch_unlikely(&qspinlock_key)) \ + return queued_spin_##op(lock); \ + return ticket_spin_##op(lock); \ +} + +SPINLOCK_BASE_DECLARE(lock, void, arch_spinlock_t *) +SPINLOCK_BASE_DECLARE(unlock, void, arch_spinlock_t *) +SPINLOCK_BASE_DECLARE(is_locked, int, arch_spinlock_t *) +SPINLOCK_BASE_DECLARE(is_contended, int, arch_spinlock_t *) +SPINLOCK_BASE_DECLARE(trylock, bool, arch_spinlock_t *) +SPINLOCK_BASE_DECLARE(value_unlocked, int, arch_spinlock_t) + +#else + +#include + +#endif + +#include + +#endif /* __ASM_RISCV_SPINLOCK_H */ diff --git a/arch/riscv/kernel/setup.c b/arch/riscv/kernel/setup.c index 4f73c0ae44b2..d7c31c9b8ead 100644 --- a/arch/riscv/kernel/setup.c +++ b/arch/riscv/kernel/setup.c @@ -244,6 +244,38 @@ static void __init parse_dtb(void) #endif } +DEFINE_STATIC_KEY_TRUE(qspinlock_key); +EXPORT_SYMBOL(qspinlock_key); + +static void __init riscv_spinlock_init(void) +{ + char *using_ext; + + if (IS_ENABLED(CONFIG_RISCV_ISA_ZACAS) && + IS_ENABLED(CONFIG_RISCV_ISA_ZABHA)) { + using_ext = "using Zabha"; + + asm goto(ALTERNATIVE("j %[no_zacas]", "nop", 0, RISCV_ISA_EXT_ZACAS, 1) + : : : : no_zacas); + asm goto(ALTERNATIVE("nop", "j %[qspinlock]", 0, RISCV_ISA_EXT_ZABHA, 1) + : : : : qspinlock); + } + +no_zacas: + using_ext = "using Ziccrse"; + asm goto(ALTERNATIVE("nop", "j %[qspinlock]", 0, + RISCV_ISA_EXT_ZICCRSE, 1) + : : : : qspinlock); + + static_branch_disable(&qspinlock_key); + pr_info("Ticket spinlock: enabled\n"); + + return; + +qspinlock: + pr_info("Queued spinlock %s: enabled\n", using_ext); +} + extern void __init init_rt_signal_env(void); void __init setup_arch(char **cmdline_p) @@ -295,6 +327,7 @@ void __init setup_arch(char **cmdline_p) riscv_set_dma_cache_alignment(); riscv_user_isa_enable(); + riscv_spinlock_init(); } bool arch_cpu_is_hotpluggable(int cpu) diff --git a/include/asm-generic/qspinlock.h b/include/asm-generic/qspinlock.h index 0655aa5b57b2..bf47cca2c375 100644 --- a/include/asm-generic/qspinlock.h +++ b/include/asm-generic/qspinlock.h @@ -136,6 +136,7 @@ static __always_inline bool virt_spin_lock(struct qspinlock *lock) } #endif +#ifndef __no_arch_spinlock_redefine /* * Remapping spinlock architecture specific functions to the corresponding * queued spinlock functions. @@ -146,5 +147,6 @@ static __always_inline bool virt_spin_lock(struct qspinlock *lock) #define arch_spin_lock(l) queued_spin_lock(l) #define arch_spin_trylock(l) queued_spin_trylock(l) #define arch_spin_unlock(l) queued_spin_unlock(l) +#endif #endif /* __ASM_GENERIC_QSPINLOCK_H */ diff --git a/include/asm-generic/ticket_spinlock.h b/include/asm-generic/ticket_spinlock.h index cfcff22b37b3..325779970d8a 100644 --- a/include/asm-generic/ticket_spinlock.h +++ b/include/asm-generic/ticket_spinlock.h @@ -89,6 +89,7 @@ static __always_inline int ticket_spin_is_contended(arch_spinlock_t *lock) return (s16)((val >> 16) - (val & 0xffff)) > 1; } +#ifndef __no_arch_spinlock_redefine /* * Remapping spinlock architecture specific functions to the corresponding * ticket spinlock functions. @@ -99,5 +100,6 @@ static __always_inline int ticket_spin_is_contended(arch_spinlock_t *lock) #define arch_spin_lock(l) ticket_spin_lock(l) #define arch_spin_trylock(l) ticket_spin_trylock(l) #define arch_spin_unlock(l) ticket_spin_unlock(l) +#endif #endif /* __ASM_GENERIC_TICKET_SPINLOCK_H */