From patchwork Wed Jul 31 07:23:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13748231 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3A06DC3DA64 for ; Wed, 31 Jul 2024 07:25:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=qFLgPDpqWGASX/ahEDMLTJe/LkCqIMLJY1B7JlmJ9ds=; b=nEO7HjagvmFtHW 6TCr757rmZlQ3zMzVQ1MBkgjJ5VdPnGkOcBMg5tS+VLOVvfwgW6opdOEcw1PbWQLSK7ttqJpbydcg tm2pLT+ykYolc9HwzPS/Q2tZ9wdWpHt/ZoxuKSuBGadOBKe1rHrQhc4bRYzZs5Ke9/Btofzs3JNNC ddGpArVkct4JmtGpGpLqoAOuJAvMtXUMk/TzD59na61yESnnVpwB5cnZXK+NRf5EDdjv7NNcboIjJ p7BtHaQUT+39g+hRpgEPZOeTIbw5XFsrdzVHZfYV90FocvVbRcdMDzCAtYwD8aT7ro0soHYTEzvlX XmRFng1hYp6nFW0cPYvQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sZ3hy-000000006vK-3KzA; Wed, 31 Jul 2024 07:25:14 +0000 Received: from mail-wm1-x333.google.com ([2a00:1450:4864:20::333]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sZ3hu-000000006uG-48nD for linux-riscv@lists.infradead.org; Wed, 31 Jul 2024 07:25:12 +0000 Received: by mail-wm1-x333.google.com with SMTP id 5b1f17b1804b1-42817bee9e8so31460115e9.3 for ; Wed, 31 Jul 2024 00:25:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1722410709; x=1723015509; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=vrgH74i2p3t2z7NFA5O4Z9Q/nrBRVoMRKy6jFwvD/Ug=; b=jmZ1X6zY4zZJzVO8+zw0QMgjLyiQjLaa/o+FJbnaX9BqaXGWXKRsu3qlxkmLsZfG5k 1rKVXf3lvpI6oux6gb8LeHwxrOfj0hXQnb2wkA80xgtFEglNRuB5KidzMWFlkTumDvR8 jbBEHE/KT7D7jxvIwS5sxjyuH6nXjJXTy7thKwZELE9k2OV2MjSbpvkaBz9q0ewQwNW2 2HoVrZyke56mB44hsLsqXpc4GuAiCZnY9G7JcooefyOj6EFTGyIbEuHvpowuKT57jU2w QmcHjOExZElaslMSubVkzwH8H+rMiV9pURgdbiBVLaH/Vw0MAtwiuGOqtdZkpdG3bzXQ mCSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722410709; x=1723015509; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vrgH74i2p3t2z7NFA5O4Z9Q/nrBRVoMRKy6jFwvD/Ug=; b=Oqutm8sKW6Hg6XUJdrKRqpdvYsicfUDAa5VWJecxBCv6DFcokGel+AyZ82qUOdnlaF Db07v8kI8Aw7O4pfpX6MQvabUtoKLWnLJAumPSMldp6zbw+uaiXM9QCKkR22UtVVT2Ez Ng4zBxkG0o9r0dNzR6vxD5CzmjKQ8FwqTMKEhxTcy3VVDnUJojXW5yVgPZf+6sEwtvnv NXU38Wg9dUWNl72ERdH3Q9dhxyPrbqiOyiT/OKLuUXYffRx/YdpI+yr6M0WvB+2cHrui +jIkDi2z1KVZVSc9QeCCqNV+EIigxORSWtnNOPfmjgF430z5syKGZvUfuQ1gDwO0Y9Md e6Pg== X-Forwarded-Encrypted: i=1; AJvYcCX32uqdgymmWcK3ybvUdtoH1F3MhkfJIqXg15QglYWfInM8ISo5zjy3vTVFfjJ3M8I8L5y8i+XTxlXsXWHjAt3QbEWJ2qv13nHTKr/at3eh X-Gm-Message-State: AOJu0YxZflAbBM1mXEA/n0DKhHnO1itR0ytcbPE36P2b18i1H5gfy9aU PQt0QUQpo0zEO6QWoESe2uDzLVAoSuOnDkd0nOsjThpb9zqZ0gh+MWkX1gNknm4= X-Google-Smtp-Source: AGHT+IH2fhRxtLzhZGksxU10vpYRAvkVREwWprJnjJIWdgKEaDVtJYoKkJRwph0ht118TITdUlVmFw== X-Received: by 2002:a05:600c:3511:b0:428:3b5:816b with SMTP id 5b1f17b1804b1-42811d6e2ddmr96720825e9.3.1722410708829; Wed, 31 Jul 2024 00:25:08 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4282bb226b0sm10529075e9.46.2024.07.31.00.25.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jul 2024 00:25:08 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH v4 01/13] riscv: Move cpufeature.h macros into their own header Date: Wed, 31 Jul 2024 09:23:53 +0200 Message-Id: <20240731072405.197046-2-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240731072405.197046-1-alexghiti@rivosinc.com> References: <20240731072405.197046-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240731_002511_043216_1A286D73 X-CRM114-Status: GOOD ( 15.60 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org asm/cmpxchg.h will soon need riscv_has_extension_unlikely() macros and then needs to include asm/cpufeature.h which introduces a lot of header circular dependencies. So move the riscv_has_extension_XXX() macros into their own header which prevents such circular dependencies by including a restricted number of headers. Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones --- arch/riscv/include/asm/cpufeature-macros.h | 66 ++++++++++++++++++++++ arch/riscv/include/asm/cpufeature.h | 56 +----------------- 2 files changed, 67 insertions(+), 55 deletions(-) create mode 100644 arch/riscv/include/asm/cpufeature-macros.h diff --git a/arch/riscv/include/asm/cpufeature-macros.h b/arch/riscv/include/asm/cpufeature-macros.h new file mode 100644 index 000000000000..c5f0bf75e026 --- /dev/null +++ b/arch/riscv/include/asm/cpufeature-macros.h @@ -0,0 +1,66 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright 2022-2024 Rivos, Inc + */ + +#ifndef _ASM_CPUFEATURE_MACROS_H +#define _ASM_CPUFEATURE_MACROS_H + +#include +#include + +#define STANDARD_EXT 0 + +bool __riscv_isa_extension_available(const unsigned long *isa_bitmap, unsigned int bit); +#define riscv_isa_extension_available(isa_bitmap, ext) \ + __riscv_isa_extension_available(isa_bitmap, RISCV_ISA_EXT_##ext) + +static __always_inline bool __riscv_has_extension_likely(const unsigned long vendor, + const unsigned long ext) +{ + asm goto(ALTERNATIVE("j %l[l_no]", "nop", %[vendor], %[ext], 1) + : + : [vendor] "i" (vendor), [ext] "i" (ext) + : + : l_no); + + return true; +l_no: + return false; +} + +static __always_inline bool __riscv_has_extension_unlikely(const unsigned long vendor, + const unsigned long ext) +{ + asm goto(ALTERNATIVE("nop", "j %l[l_yes]", %[vendor], %[ext], 1) + : + : [vendor] "i" (vendor), [ext] "i" (ext) + : + : l_yes); + + return false; +l_yes: + return true; +} + +static __always_inline bool riscv_has_extension_unlikely(const unsigned long ext) +{ + compiletime_assert(ext < RISCV_ISA_EXT_MAX, "ext must be < RISCV_ISA_EXT_MAX"); + + if (IS_ENABLED(CONFIG_RISCV_ALTERNATIVE)) + return __riscv_has_extension_unlikely(STANDARD_EXT, ext); + + return __riscv_isa_extension_available(NULL, ext); +} + +static __always_inline bool riscv_has_extension_likely(const unsigned long ext) +{ + compiletime_assert(ext < RISCV_ISA_EXT_MAX, "ext must be < RISCV_ISA_EXT_MAX"); + + if (IS_ENABLED(CONFIG_RISCV_ALTERNATIVE)) + return __riscv_has_extension_likely(STANDARD_EXT, ext); + + return __riscv_isa_extension_available(NULL, ext); +} + +#endif diff --git a/arch/riscv/include/asm/cpufeature.h b/arch/riscv/include/asm/cpufeature.h index 45f9c1171a48..c991672bb401 100644 --- a/arch/riscv/include/asm/cpufeature.h +++ b/arch/riscv/include/asm/cpufeature.h @@ -11,6 +11,7 @@ #include #include #include +#include /* * These are probed via a device_initcall(), via either the SBI or directly @@ -103,61 +104,6 @@ extern const size_t riscv_isa_ext_count; extern bool riscv_isa_fallback; unsigned long riscv_isa_extension_base(const unsigned long *isa_bitmap); - -#define STANDARD_EXT 0 - -bool __riscv_isa_extension_available(const unsigned long *isa_bitmap, unsigned int bit); -#define riscv_isa_extension_available(isa_bitmap, ext) \ - __riscv_isa_extension_available(isa_bitmap, RISCV_ISA_EXT_##ext) - -static __always_inline bool __riscv_has_extension_likely(const unsigned long vendor, - const unsigned long ext) -{ - asm goto(ALTERNATIVE("j %l[l_no]", "nop", %[vendor], %[ext], 1) - : - : [vendor] "i" (vendor), [ext] "i" (ext) - : - : l_no); - - return true; -l_no: - return false; -} - -static __always_inline bool __riscv_has_extension_unlikely(const unsigned long vendor, - const unsigned long ext) -{ - asm goto(ALTERNATIVE("nop", "j %l[l_yes]", %[vendor], %[ext], 1) - : - : [vendor] "i" (vendor), [ext] "i" (ext) - : - : l_yes); - - return false; -l_yes: - return true; -} - -static __always_inline bool riscv_has_extension_unlikely(const unsigned long ext) -{ - compiletime_assert(ext < RISCV_ISA_EXT_MAX, "ext must be < RISCV_ISA_EXT_MAX"); - - if (IS_ENABLED(CONFIG_RISCV_ALTERNATIVE)) - return __riscv_has_extension_unlikely(STANDARD_EXT, ext); - - return __riscv_isa_extension_available(NULL, ext); -} - -static __always_inline bool riscv_has_extension_likely(const unsigned long ext) -{ - compiletime_assert(ext < RISCV_ISA_EXT_MAX, "ext must be < RISCV_ISA_EXT_MAX"); - - if (IS_ENABLED(CONFIG_RISCV_ALTERNATIVE)) - return __riscv_has_extension_likely(STANDARD_EXT, ext); - - return __riscv_isa_extension_available(NULL, ext); -} - static __always_inline bool riscv_cpu_has_extension_likely(int cpu, const unsigned long ext) { compiletime_assert(ext < RISCV_ISA_EXT_MAX, "ext must be < RISCV_ISA_EXT_MAX"); From patchwork Wed Jul 31 07:23:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13748232 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A2077C3DA64 for ; Wed, 31 Jul 2024 07:26:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=dh2/ah0WhuFzFQtEkb6oymwZNKRdNjW//UfnhhxyAB8=; b=QI1uhIz7nYqPtX aGarSV1FQb64E+uSM+qyPVucl92BSvFMFRhsT1IcrEiGGIiuyG0RJrisjHFYAqslZngD2zbOYjPQf BVeI8fGUu0ScAXW/XolxkvCam25RxEdkx7Zwsw65mPZDWZgU4yVgP6Tf4dR7mmbYkjrru403xIKqV wzZ9etjypeSo4q+gJ4Z38lppZVfArUtezcNmNoooy72lDTKlEbr8t3kngy7aeEi97xs8N5oYqaLDY dVw7H45slRUKQUkFkOF9Ej1ZEyR8G3Ft+jYFXJFLUEuORf4Njr3ur6m3sdWv5hksKg7zwau4OSdGK ZeVfSQ+gE5PY7Ysbk/Ww==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sZ3iy-000000007A0-0FqN; Wed, 31 Jul 2024 07:26:16 +0000 Received: from mail-lf1-x12b.google.com ([2a00:1450:4864:20::12b]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sZ3iu-0000000078y-1VU2 for linux-riscv@lists.infradead.org; Wed, 31 Jul 2024 07:26:13 +0000 Received: by mail-lf1-x12b.google.com with SMTP id 2adb3069b0e04-52f025bc147so7670737e87.3 for ; Wed, 31 Jul 2024 00:26:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1722410770; x=1723015570; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=X5b/A6qU74AkfxdtYyxiBPFCfAn9Wby9ZSL7Rzcj1cY=; b=qjLy4lFXVxiuJmpXn0bEvI0cZANuH3O6CWfzGO34CDgrfebbYDGYcUadOHQeDVG234 STBKbyyiuOOQs9TgQ0d67oJk6oOM6EWt6Ty9a6XJO0+5I4jkeoIFlluZlKsbd9vCwjzK Wsmb0npKryWSSnsCkzjAd0x/vCGJ3PfGnwCdnG/JGNGlxq10KQ38cjJdykT+ZY2eDjM9 17NpNrJV0My4kqCP02Ur8EUSUxOisbFWnK79QY2KvGf8QuTQg7paGteGqeuEwyIrWmsX n2rOjC9yYMMHCKJWlh2qp2d2b4rWqM3RMyckJG1PwJNOnb6bb5huz1K3rFjYZbkGNQrZ dzLQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722410770; x=1723015570; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=X5b/A6qU74AkfxdtYyxiBPFCfAn9Wby9ZSL7Rzcj1cY=; b=Bld3sHrJJzvI7LgNMLFWvDS+r8UXbjnaiC8SvO5C7gILkn6xpjm9FKE6F8yB4WsPVZ crRd7o+N7VgAK2cHotHB2rXfKH9WEZqj1VAFdn/wMnyVCG3r7d+lCnRvGHm3OQz4loVV TOWUhPrphUEg+gkmSmd3sV6l/jZW1R6VcYwlT1zg4DYV4b18WFV078S0x1jPN6OGw7zv GFdFq1EM8b0QTbayxxSbnC5g9uTcGj97Z8zPP9yqIzjozAG+Wd1yn04ShOrQZhRWkBNc ynJ0t9bTAO+0K+Z7f5LqB+jQHH7PuGxOZQ+EAJZMt1k7UF7xr+cIa5NNqg6U0Q8wDCTw L5pA== X-Forwarded-Encrypted: i=1; AJvYcCV21bODghNz8Ry22/LsSsOXYVi2fgaCAUZqGSsgOaeoJdwHEtyaTXtTY3gqs9JI7x5zDWY1zq07TrAB+zbyxT49o9YRbPOEfiLQxLnCtv+z X-Gm-Message-State: AOJu0YyvpZveoUk4Ndcx4dpx9XPNwN4qscVuRM+vTgJPGDnNhnPdZ6f3 sPCZe2mMRTl07+kOWuq2C4+/UgqY5lGDouKr6OZiJlnWWr1EWCJkyrQfTo9LnZY= X-Google-Smtp-Source: AGHT+IElDfoIlakrFYsMBeeLlbntHhhP6Di7Erttb/4pd4dWg5QISN3W8tux4DC9SLwXgyAkaGYd7g== X-Received: by 2002:a19:8c58:0:b0:52c:f12a:d0e0 with SMTP id 2adb3069b0e04-5309b27a656mr7792329e87.28.1722410770171; Wed, 31 Jul 2024 00:26:10 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4282bac7349sm10525575e9.27.2024.07.31.00.26.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jul 2024 00:26:09 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH v4 02/13] riscv: Do not fail to build on byte/halfword operations with Zawrs Date: Wed, 31 Jul 2024 09:23:54 +0200 Message-Id: <20240731072405.197046-3-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240731072405.197046-1-alexghiti@rivosinc.com> References: <20240731072405.197046-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240731_002612_423454_AFF06470 X-CRM114-Status: GOOD ( 11.37 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org riscv does not have lr instructions on byte and halfword but the qspinlock implementation actually uses such atomics provided by the Zabha extension, so those sizes are legitimate. Then instead of failing to build, just fallback to the !Zawrs path. Signed-off-by: Alexandre Ghiti --- arch/riscv/include/asm/cmpxchg.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index ebbce134917c..9ba497ea18a5 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -268,7 +268,8 @@ static __always_inline void __cmpwait(volatile void *ptr, break; #endif default: - BUILD_BUG(); + /* RISC-V doesn't have lr instructions on byte and half-word. */ + goto no_zawrs; } return; From patchwork Wed Jul 31 07:23:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13748234 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 37443C3DA7F for ; Wed, 31 Jul 2024 07:27:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=B7K5O6x82GUucVVh75eKCf5oqyXDcYUQkHvQW0KJUjI=; b=e9um6/16dI11LS +wxeXLSsWQPrUhsqE9gbl3QEG0Wl2Y7dW9cULGlcQQWz3ZsB6bWN2URpYlZwZrchSqYk1cXnKGe24 DIJiqgt+hb0yI2K0KWcxW1HNIrNglxlBHO7DoSrq8QkvOgVtj/qcnW/cMea6brgjOzEOVMRFh9Kat EfsHLdYVoIvxD0JZmX9ST12tlYjEa6KcXFRMMSYlAgvYDEILREh7vbMTrlc/V6NuKa+6IrVzNm62p BkfhtcZ251kvNtnPV0JiYCeHX3KtTdfWpE/DNaub6rofNv8J3agkVPbnd55vcvsLMTFq3OJBoPVP8 4aVro/QBeNcKZruJTmdw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sZ3jv-000000007Sc-2Had; Wed, 31 Jul 2024 07:27:15 +0000 Received: from mail-wm1-x32b.google.com ([2a00:1450:4864:20::32b]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sZ3jt-000000007Ra-0V5m for linux-riscv@lists.infradead.org; Wed, 31 Jul 2024 07:27:14 +0000 Received: by mail-wm1-x32b.google.com with SMTP id 5b1f17b1804b1-42816ca797fso30104815e9.2 for ; Wed, 31 Jul 2024 00:27:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1722410832; x=1723015632; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=c+0f3E3q4VF2oUdETZhba5ndpciNr7WJjXKRA7n65fw=; b=RW6HaobHr5emls+XMurTEKtkdBd0WTNplSI9fLbsaKYiaoaKroAk+jCEoGEt++qKyl jwydcgK+UsHDtwMzhBduSRN/veNZSqZOWrnVXLTSpBY9A/Edj4r6xxPV2LW2BqPt465T hcECU6RmG/0mcfR7bB9AUP8z4jnaVYN6ISfcdmZ7BIlvRPl+uq6f0Gw2Zwh2zrA5PXgF ZL9oZ1TCeBubTCQc2JHZaXPqlFTHcUd46r5gvczKynCxnfzoXRTWvN80ak2+4a0KdtG2 WX2Zr1HC5y/thJ1OQk9IdxIwKviGNMj4q7gjlqjSdtEounhscxsUDGB+itAE6fUdk/1v Eq+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722410832; x=1723015632; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=c+0f3E3q4VF2oUdETZhba5ndpciNr7WJjXKRA7n65fw=; b=uM8Ci1clCv5GASv45EkjnWq1UpxYkZSNkidVk3ZDSbxpOWywHeMrqzUA7fpISWkpzd xcU5LD6+jU8aF8P82VsFfz3m+3qU7MrPA+4qKR2cq3lbLekpV9VgMVXFNh8Hkbt4fWL2 vKHYXd8l7LKz57pWRajFDYQgCok7xCjssRfHHSuYNGGutnUfL025ydNgaIfLtQH34vfI 3CirSjWzyLre8A4qGkC6YB5omp98xGQKSNPv4tHbOZjDeUT7fCfpAfVRrNvYUZPEeHWH swbPV88d3iIDjXmOxYbxuRO48abY/h4xmaNmOHNn9b3qhbqH2e7NAX7IaD2Zg2Rj2SsV h4vw== X-Forwarded-Encrypted: i=1; AJvYcCUmJIOf3w8UiPIGvEZUb//YVVzXJOt6Pn5IsmOFmnoAhnvJ0ZKuQISe0L+xFlhnjjyzDmvBXbK8Eil+6RAtasqa5nrzvXwKeguWeo1HQHhT X-Gm-Message-State: AOJu0Yz4L2ZjYaecYDzIgNzq4EGm3WIkteMmBUuH90ENhKqTpojXW3Ee THbEv7yMjN4zT2GUlKArBmMaylfTgE/EWkFO3CIQfsjYgXr+ihgBxuqxWoNxvMo= X-Google-Smtp-Source: AGHT+IF+zhrAE8xWu8482bc/xbqgEfe8Vh5QY9YSloK+qT0j5XnJu7tISiM0orNkpBJdFYpMo1Fyfg== X-Received: by 2002:a05:600c:4593:b0:426:6d1a:d497 with SMTP id 5b1f17b1804b1-42811d8c5b5mr82139885e9.12.1722410831634; Wed, 31 Jul 2024 00:27:11 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-36b367d95c7sm16218977f8f.37.2024.07.31.00.27.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jul 2024 00:27:11 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH v4 03/13] riscv: Implement cmpxchg32/64() using Zacas Date: Wed, 31 Jul 2024 09:23:55 +0200 Message-Id: <20240731072405.197046-4-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240731072405.197046-1-alexghiti@rivosinc.com> References: <20240731072405.197046-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240731_002713_181719_118E5C07 X-CRM114-Status: GOOD ( 14.86 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org This adds runtime support for Zacas in cmpxchg operations. Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones --- arch/riscv/Kconfig | 16 +++++++++++ arch/riscv/Makefile | 3 ++ arch/riscv/include/asm/cmpxchg.h | 48 +++++++++++++++++++++----------- 3 files changed, 50 insertions(+), 17 deletions(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 0f3cd7c3a436..d955c64d50c2 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -613,6 +613,22 @@ config RISCV_ISA_ZAWRS use of these instructions in the kernel when the Zawrs extension is detected at boot. +config TOOLCHAIN_HAS_ZACAS + bool + default y + depends on !64BIT || $(cc-option,-mabi=lp64 -march=rv64ima_zacas) + depends on !32BIT || $(cc-option,-mabi=ilp32 -march=rv32ima_zacas) + depends on AS_HAS_OPTION_ARCH + +config RISCV_ISA_ZACAS + bool "Zacas extension support for atomic CAS" + depends on TOOLCHAIN_HAS_ZACAS + depends on RISCV_ALTERNATIVE + default y + help + Enable the use of the Zacas ISA-extension to implement kernel atomic + cmpxchg operations when it is detected at boot. + If you don't know what to do here, say Y. config TOOLCHAIN_HAS_ZBB diff --git a/arch/riscv/Makefile b/arch/riscv/Makefile index 6fe682139d2e..f1788131d5fe 100644 --- a/arch/riscv/Makefile +++ b/arch/riscv/Makefile @@ -82,6 +82,9 @@ else riscv-march-$(CONFIG_TOOLCHAIN_NEEDS_EXPLICIT_ZICSR_ZIFENCEI) := $(riscv-march-y)_zicsr_zifencei endif +# Check if the toolchain supports Zacas +riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZACAS) := $(riscv-march-y)_zacas + # Remove F,D,V from isa string for all. Keep extensions between "fd" and "v" by # matching non-v and non-multi-letter extensions out with the filter ([^v_]*) KBUILD_CFLAGS += -march=$(shell echo $(riscv-march-y) | sed -E 's/(rv32ima|rv64ima)fd([^v_]*)v?/\1\2/') diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index 9ba497ea18a5..c626fe0d08ae 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -12,6 +12,7 @@ #include #include #include +#include #define __arch_xchg_masked(sc_sfx, prepend, append, r, p, n) \ ({ \ @@ -137,24 +138,37 @@ r = (__typeof__(*(p)))((__retx & __mask) >> __s); \ }) -#define __arch_cmpxchg(lr_sfx, sc_sfx, prepend, append, r, p, co, o, n) \ +#define __arch_cmpxchg(lr_sfx, sc_cas_sfx, prepend, append, r, p, co, o, n) \ ({ \ - register unsigned int __rc; \ + if (IS_ENABLED(CONFIG_RISCV_ISA_ZACAS) && \ + riscv_has_extension_unlikely(RISCV_ISA_EXT_ZACAS)) { \ + r = o; \ \ - __asm__ __volatile__ ( \ - prepend \ - "0: lr" lr_sfx " %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc" sc_sfx " %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - append \ - "1:\n" \ - : "=&r" (r), "=&r" (__rc), "+A" (*(p)) \ - : "rJ" (co o), "rJ" (n) \ - : "memory"); \ + __asm__ __volatile__ ( \ + prepend \ + " amocas" sc_cas_sfx " %0, %z2, %1\n" \ + append \ + : "+&r" (r), "+A" (*(p)) \ + : "rJ" (n) \ + : "memory"); \ + } else { \ + register unsigned int __rc; \ + \ + __asm__ __volatile__ ( \ + prepend \ + "0: lr" lr_sfx " %0, %2\n" \ + " bne %0, %z3, 1f\n" \ + " sc" sc_cas_sfx " %1, %z4, %2\n" \ + " bnez %1, 0b\n" \ + append \ + "1:\n" \ + : "=&r" (r), "=&r" (__rc), "+A" (*(p)) \ + : "rJ" (co o), "rJ" (n) \ + : "memory"); \ + } \ }) -#define _arch_cmpxchg(ptr, old, new, sc_sfx, prepend, append) \ +#define _arch_cmpxchg(ptr, old, new, sc_cas_sfx, prepend, append) \ ({ \ __typeof__(ptr) __ptr = (ptr); \ __typeof__(*(__ptr)) __old = (old); \ @@ -164,15 +178,15 @@ switch (sizeof(*__ptr)) { \ case 1: \ case 2: \ - __arch_cmpxchg_masked(sc_sfx, prepend, append, \ + __arch_cmpxchg_masked(sc_cas_sfx, prepend, append, \ __ret, __ptr, __old, __new); \ break; \ case 4: \ - __arch_cmpxchg(".w", ".w" sc_sfx, prepend, append, \ + __arch_cmpxchg(".w", ".w" sc_cas_sfx, prepend, append, \ __ret, __ptr, (long), __old, __new); \ break; \ case 8: \ - __arch_cmpxchg(".d", ".d" sc_sfx, prepend, append, \ + __arch_cmpxchg(".d", ".d" sc_cas_sfx, prepend, append, \ __ret, __ptr, /**/, __old, __new); \ break; \ default: \ From patchwork Wed Jul 31 07:23:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13748235 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E6463C3DA7F for ; Wed, 31 Jul 2024 07:28:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=YWgGWE/mdvxpgjPEx0VfvrMQc9QpT6mNU7WT9tXWuWM=; b=OiGAsKXW2cwEaU XsXU3vYOihVuQPaqWn+jum0LS+n3Lhi9CQG3cX8HTvraf0kJPn7BGYQ78sk4JW8mn4ST1fvd4Wzoy Fx/f6heEs68GK/akR6aW+1sRjhdOJsygjLU36Or5F2iM36JtiYUoT3VXdhv96p1/IXDel27/AKJi2 /QQ63q4foRuCEMxz2IlsDmc1zQ645kQ63jlshxDeCc6GpSeDPJ5g7WY7+vxDyYkYOB2fsyMtNQpvq a4WhSOrZmIaeCV4sJlu7nG2N7VT+KNq008fTUTe7zwpyvBjBAptj3UL1/YSvPbHo6D2nbF3HBEUf1 ODIy3X3bkFlfio5pZXeA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sZ3kx-000000007t6-0N6A; Wed, 31 Jul 2024 07:28:19 +0000 Received: from mail-lf1-x131.google.com ([2a00:1450:4864:20::131]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sZ3kt-000000007q2-3R0F for linux-riscv@lists.infradead.org; Wed, 31 Jul 2024 07:28:17 +0000 Received: by mail-lf1-x131.google.com with SMTP id 2adb3069b0e04-530ad969360so1330502e87.0 for ; Wed, 31 Jul 2024 00:28:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1722410893; x=1723015693; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ttjywx6Mo1/3X2D9jge6HvTlgUDsJ3yzsugn6JInGgA=; b=MI7+URP4Mm46KBbDdyazDorMMY3XLKQ3jeQHYorJApwDfcRUcygY5q6UiCevpPNtiS DQmFb0CcFMJl6ZcYbabbpcWI+462heWUeybxLMKeWrNft4aYnCUalNXCSZYKtXuxQqFq xt1Ep/Z0EwEg0qTHkzZxloDbEjy7gadAjB7Ty+KFxAj+Xyh90f2eHzQuCuqufN/eeuB+ qV0JvsQ35j/mZne23XDfoBi0s/n9auTwIk6ypThmMA4tcdahVOhXgq9InYCPXDfOlF67 OoFg+WGkEG2IBcBQQYABBtXh3GciQ/bQMO0M3PdM2rFcQNjO91h8js6yMdYziCH4AiUT iVXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722410893; x=1723015693; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ttjywx6Mo1/3X2D9jge6HvTlgUDsJ3yzsugn6JInGgA=; b=atDXQKQyNjvAkNFwR/tANqk3E8Lw5gW47S5fQfYjM/8J9qtRF6xnSFECa6m5ZtxLAO tnEAbME8PhzKuAGh3raNQwTvz5nltifDbxbq2ORKBthDLzwHSqM6Pm5Iml8uwI4PegYa OYlYayO3KfXolJXrPBb3jGqxkB9gwpjYIlpKMErhUUn+rOXD7fDzM5W6f73V/Q060Bu1 VoNzKVZ4Jza6Xpne+0hToVOckNY82tD7G8ntCLbTNFNLAWEKwt3aLbNSaKksoWX794vF RiVILvJmg4YR/J4RtmNklFKA30kjZgAXMf6SWGFILLXQW33OtUHYnLjIvAEe4M4LCASu ueiA== X-Forwarded-Encrypted: i=1; AJvYcCXTEt2kd1Bg+vvOUawQ7UViYiZi4BXh9RsgrnsxKL9qLUjb2Bp0G2S9+d9qNSUPWkWRKr6vgmRIbacNoP5VRYyYSMOyMpi1xaZSbiPxYSx3 X-Gm-Message-State: AOJu0YyFa1IQQ3OGZ5RTygyWb+w4rGQog0d1ZEVjkDjZ72zQZhU07IxV leRJOOrwLT7WNpCSt8Qv6t2we7ZJGS5yMRQyVyqlat/hwUMbIoJg750rsm2+DghYGGdQ/qTzGfc y X-Google-Smtp-Source: AGHT+IGwuD5WOsjtT2bfQ2zXne4CrgHWP6AcnVi1hE62VKVHJr7T6wk/7agmQ1tHhyxNcI20/8ivqg== X-Received: by 2002:ac2:5690:0:b0:52c:ebd0:609 with SMTP id 2adb3069b0e04-5309b259bb9mr12112105e87.7.1722410892990; Wed, 31 Jul 2024 00:28:12 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4282babaab3sm10562035e9.28.2024.07.31.00.28.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jul 2024 00:28:12 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH v4 04/13] dt-bindings: riscv: Add Zabha ISA extension description Date: Wed, 31 Jul 2024 09:23:56 +0200 Message-Id: <20240731072405.197046-5-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240731072405.197046-1-alexghiti@rivosinc.com> References: <20240731072405.197046-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240731_002815_869271_8AE27ADB X-CRM114-Status: UNSURE ( 9.56 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Add description for the Zabha ISA extension which was ratified in April 2024. Signed-off-by: Alexandre Ghiti Reviewed-by: Guo Ren Acked-by: Conor Dooley --- Documentation/devicetree/bindings/riscv/extensions.yaml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/Documentation/devicetree/bindings/riscv/extensions.yaml b/Documentation/devicetree/bindings/riscv/extensions.yaml index a06dbc6b4928..a63578b95c4a 100644 --- a/Documentation/devicetree/bindings/riscv/extensions.yaml +++ b/Documentation/devicetree/bindings/riscv/extensions.yaml @@ -171,6 +171,12 @@ properties: memory types as ratified in the 20191213 version of the privileged ISA specification. + - const: zabha + description: | + The Zabha extension for Byte and Halfword Atomic Memory Operations + as ratified at commit 49f49c842ff9 ("Update to Rafified state") of + riscv-zabha. + - const: zacas description: | The Zacas extension for Atomic Compare-and-Swap (CAS) instructions From patchwork Wed Jul 31 07:23:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13748236 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6D524C3DA64 for ; Wed, 31 Jul 2024 07:29:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=6WciRtr21cpSepoFkJ37sg0K5gj/vwXNNOiKDiQrL74=; b=Whi8IuXe7T6jsF Dn5AAgpfVOwtH+ly5yqqQr10Ij2DQRgOc2QtJB6NvBKQu/+3QRJGPPC+mT8b1Q5vKOijn48ZpJO9S +8HqcmHBpoJF694srSqEagw69MHXN49obmsQDFXcKjo+gx0oPuEKEYXjh15lIiF7QsBJ1Ku4b/bsQ KlkxBpTZ1CCHKj1FFn1C1P32sRKy/FOiAe6ZZMpkvmv0Bko+ZdfhCqRqrzcw/1bKL9Le9JAOwko/g jGNq6XqdNr1Low5qVjCliDqePkUHNEi2TCcDvDgncmjepI4V1y4aadYgjnfHYUenRjM67hdrtxoxf rl4+8u+ICNkZSjHsTk4A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sZ3lv-0000000088P-1cIZ; Wed, 31 Jul 2024 07:29:19 +0000 Received: from mail-wr1-x42e.google.com ([2a00:1450:4864:20::42e]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sZ3ls-0000000087Q-1Imk for linux-riscv@lists.infradead.org; Wed, 31 Jul 2024 07:29:17 +0000 Received: by mail-wr1-x42e.google.com with SMTP id ffacd0b85a97d-3685a564bafso2581426f8f.3 for ; Wed, 31 Jul 2024 00:29:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1722410954; x=1723015754; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=bTv3dCV8uOmdi0okEMo3U0BHPIZT88r4+7MFI/VgzDo=; b=YPtMYtagryO67AFh0QnHKffBKcp3UVrAfB1Errf3GN15VF92g5YXiNK95/qUP158s0 OENuMnw37HZ2N/2qFfzwh2wSnYYNb0EGXNTaV7yWnDQl448Xnx2VWZ+c3fOv9bLWisbG ++XGHMMDtq9+IJJDRKXSmlkdnE6w3QyMAIYOxeWs4MgyBlnOwiWd7crtRb2bHR61o1R8 eDmqzfs3Fy2nPMSA2lxSpAIr0zgNUHr+dRBz1uuv+aO8OLV19l7oUX2hHgVOrKkDQsIn SaMugw0Z9hpAGe2bmqsABgkMMc7n6apMkyQ6ORdCpT7js5Y5YHSPFhSxBSA1JsKWgTZq 7GMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722410954; x=1723015754; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=bTv3dCV8uOmdi0okEMo3U0BHPIZT88r4+7MFI/VgzDo=; b=wBv/+P8b2HM8rLi+b88uelY8nLkDXyBWIeeMDrUyMJBZA7Hl6tPn0+wU50XnuPaqRS EPHXog9EpImsxN/dGMVALgW8b152EyeVbg9qkFTunZRgL+it97Ru+2okeJs15Gl85yRV 7PQ5trRVAl06w/QEkvx9MTimVOwbbbLuyTqjWitcerSD/G01Hrupim21hZqEHt4hd53S CG6HDRDQ4Izgi3sJu8hOoQhpEjrFjWHywmr42XRpEfNFlIaKyGHHQyzz7Yz8NPxARNLW o4KLcdPVTlro9pNI2l2fSzhEE1GLXXnX4acEH7XyZd4GkZFQBORjJM6NvITVqBVvUG0s SkmA== X-Forwarded-Encrypted: i=1; AJvYcCV771PjYzZjgz8ws/xczlD3GDfAxYY8M+Zdm6qr+Bbu6dpFycjEdC5CywOxJ2jHE8tU4+jEHn5zPGKdcvCrWbPmoQsQtB7U63EvdB79bjG8 X-Gm-Message-State: AOJu0YwnKgHkQGe0A8Kd5jWxTWj8Oi8fur+06ghG1vGVbUbRv29ZhWhs NlhCFCrFlku0Yo4bByFlm/bc6y45+mJVUY7Xo8FMeCkI39REcTiQMz7/ZRMy8+o= X-Google-Smtp-Source: AGHT+IGdWI7pNX76CY5MDuyu7lWxBLphPJYSwh1U+oglkHzUOKa2xmfRAJ3FI2utc6jCiAfVd04UpA== X-Received: by 2002:a5d:440f:0:b0:367:f056:24ac with SMTP id ffacd0b85a97d-36b5ceff66emr7868257f8f.28.1722410954255; Wed, 31 Jul 2024 00:29:14 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-36b36862271sm16319776f8f.98.2024.07.31.00.29.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jul 2024 00:29:13 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH v4 05/13] riscv: Implement cmpxchg8/16() using Zabha Date: Wed, 31 Jul 2024 09:23:57 +0200 Message-Id: <20240731072405.197046-6-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240731072405.197046-1-alexghiti@rivosinc.com> References: <20240731072405.197046-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240731_002916_382289_5F1AC37B X-CRM114-Status: GOOD ( 16.89 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org This adds runtime support for Zabha in cmpxchg8/16() operations. Note that in the absence of Zacas support in the toolchain, CAS instructions from Zabha won't be used. Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones --- arch/riscv/Kconfig | 18 ++++++++ arch/riscv/Makefile | 3 ++ arch/riscv/include/asm/cmpxchg.h | 78 ++++++++++++++++++++------------ arch/riscv/include/asm/hwcap.h | 1 + arch/riscv/kernel/cpufeature.c | 1 + 5 files changed, 72 insertions(+), 29 deletions(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index d955c64d50c2..212ec2aab389 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -613,6 +613,24 @@ config RISCV_ISA_ZAWRS use of these instructions in the kernel when the Zawrs extension is detected at boot. +config TOOLCHAIN_HAS_ZABHA + bool + default y + depends on !64BIT || $(cc-option,-mabi=lp64 -march=rv64ima_zabha) + depends on !32BIT || $(cc-option,-mabi=ilp32 -march=rv32ima_zabha) + depends on AS_HAS_OPTION_ARCH + +config RISCV_ISA_ZABHA + bool "Zabha extension support for atomic byte/halfword operations" + depends on TOOLCHAIN_HAS_ZABHA + depends on RISCV_ALTERNATIVE + default y + help + Enable the use of the Zabha ISA-extension to implement kernel + byte/halfword atomic memory operations when it is detected at boot. + + If you don't know what to do here, say Y. + config TOOLCHAIN_HAS_ZACAS bool default y diff --git a/arch/riscv/Makefile b/arch/riscv/Makefile index f1788131d5fe..f6dc5ba7c526 100644 --- a/arch/riscv/Makefile +++ b/arch/riscv/Makefile @@ -85,6 +85,9 @@ endif # Check if the toolchain supports Zacas riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZACAS) := $(riscv-march-y)_zacas +# Check if the toolchain supports Zabha +riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZABHA) := $(riscv-march-y)_zabha + # Remove F,D,V from isa string for all. Keep extensions between "fd" and "v" by # matching non-v and non-multi-letter extensions out with the filter ([^v_]*) KBUILD_CFLAGS += -march=$(shell echo $(riscv-march-y) | sed -E 's/(rv32ima|rv64ima)fd([^v_]*)v?/\1\2/') diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index c626fe0d08ae..ebcd4a30ae60 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -108,34 +108,49 @@ * indicated by comparing RETURN with OLD. */ -#define __arch_cmpxchg_masked(sc_sfx, prepend, append, r, p, o, n) \ -({ \ - u32 *__ptr32b = (u32 *)((ulong)(p) & ~0x3); \ - ulong __s = ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \ - ulong __mask = GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \ - << __s; \ - ulong __newx = (ulong)(n) << __s; \ - ulong __oldx = (ulong)(o) << __s; \ - ulong __retx; \ - ulong __rc; \ - \ - __asm__ __volatile__ ( \ - prepend \ - "0: lr.w %0, %2\n" \ - " and %1, %0, %z5\n" \ - " bne %1, %z3, 1f\n" \ - " and %1, %0, %z6\n" \ - " or %1, %1, %z4\n" \ - " sc.w" sc_sfx " %1, %1, %2\n" \ - " bnez %1, 0b\n" \ - append \ - "1:\n" \ - : "=&r" (__retx), "=&r" (__rc), "+A" (*(__ptr32b)) \ - : "rJ" ((long)__oldx), "rJ" (__newx), \ - "rJ" (__mask), "rJ" (~__mask) \ - : "memory"); \ - \ - r = (__typeof__(*(p)))((__retx & __mask) >> __s); \ +#define __arch_cmpxchg_masked(sc_sfx, cas_sfx, prepend, append, r, p, o, n) \ +({ \ + if (IS_ENABLED(CONFIG_RISCV_ISA_ZABHA) && \ + IS_ENABLED(CONFIG_RISCV_ISA_ZACAS) && \ + riscv_has_extension_unlikely(RISCV_ISA_EXT_ZABHA) && \ + riscv_has_extension_unlikely(RISCV_ISA_EXT_ZACAS)) { \ + r = o; \ + \ + __asm__ __volatile__ ( \ + prepend \ + " amocas" cas_sfx " %0, %z2, %1\n" \ + append \ + : "+&r" (r), "+A" (*(p)) \ + : "rJ" (n) \ + : "memory"); \ + } else { \ + u32 *__ptr32b = (u32 *)((ulong)(p) & ~0x3); \ + ulong __s = ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \ + ulong __mask = GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \ + << __s; \ + ulong __newx = (ulong)(n) << __s; \ + ulong __oldx = (ulong)(o) << __s; \ + ulong __retx; \ + ulong __rc; \ + \ + __asm__ __volatile__ ( \ + prepend \ + "0: lr.w %0, %2\n" \ + " and %1, %0, %z5\n" \ + " bne %1, %z3, 1f\n" \ + " and %1, %0, %z6\n" \ + " or %1, %1, %z4\n" \ + " sc.w" sc_sfx " %1, %1, %2\n" \ + " bnez %1, 0b\n" \ + append \ + "1:\n" \ + : "=&r" (__retx), "=&r" (__rc), "+A" (*(__ptr32b)) \ + : "rJ" ((long)__oldx), "rJ" (__newx), \ + "rJ" (__mask), "rJ" (~__mask) \ + : "memory"); \ + \ + r = (__typeof__(*(p)))((__retx & __mask) >> __s); \ + } \ }) #define __arch_cmpxchg(lr_sfx, sc_cas_sfx, prepend, append, r, p, co, o, n) \ @@ -177,8 +192,13 @@ \ switch (sizeof(*__ptr)) { \ case 1: \ + __arch_cmpxchg_masked(sc_cas_sfx, ".b" sc_cas_sfx, \ + prepend, append, \ + __ret, __ptr, __old, __new); \ + break; \ case 2: \ - __arch_cmpxchg_masked(sc_cas_sfx, prepend, append, \ + __arch_cmpxchg_masked(sc_cas_sfx, ".h" sc_cas_sfx, \ + prepend, append, \ __ret, __ptr, __old, __new); \ break; \ case 4: \ diff --git a/arch/riscv/include/asm/hwcap.h b/arch/riscv/include/asm/hwcap.h index 5a0bd27fd11a..f5d53251c947 100644 --- a/arch/riscv/include/asm/hwcap.h +++ b/arch/riscv/include/asm/hwcap.h @@ -92,6 +92,7 @@ #define RISCV_ISA_EXT_ZCF 83 #define RISCV_ISA_EXT_ZCMOP 84 #define RISCV_ISA_EXT_ZAWRS 85 +#define RISCV_ISA_EXT_ZABHA 86 #define RISCV_ISA_EXT_XLINUXENVCFG 127 diff --git a/arch/riscv/kernel/cpufeature.c b/arch/riscv/kernel/cpufeature.c index 8f20607adb40..2a608056c89c 100644 --- a/arch/riscv/kernel/cpufeature.c +++ b/arch/riscv/kernel/cpufeature.c @@ -322,6 +322,7 @@ const struct riscv_isa_ext_data riscv_isa_ext[] = { __RISCV_ISA_EXT_DATA(zihintpause, RISCV_ISA_EXT_ZIHINTPAUSE), __RISCV_ISA_EXT_DATA(zihpm, RISCV_ISA_EXT_ZIHPM), __RISCV_ISA_EXT_DATA(zimop, RISCV_ISA_EXT_ZIMOP), + __RISCV_ISA_EXT_DATA(zabha, RISCV_ISA_EXT_ZABHA), __RISCV_ISA_EXT_DATA(zacas, RISCV_ISA_EXT_ZACAS), __RISCV_ISA_EXT_DATA(zawrs, RISCV_ISA_EXT_ZAWRS), __RISCV_ISA_EXT_DATA(zfa, RISCV_ISA_EXT_ZFA), From patchwork Wed Jul 31 07:23:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13748237 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D7631C3DA64 for ; Wed, 31 Jul 2024 07:30:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=2L6+l2hkEWfOT8Ygm9Pk/YyXjCL98bbYO4nxUyxWHzo=; b=Vcp6lShxtZEaEk 8HGOPMwlunoGE43IgGoFRAVkWhBq8J+QHgt89g4ZaGTjKglFGrOJ2gGmQJ6wYFoZH+Cynw/iKVKHJ b03YbMGaBai9fNG+ShGa9VhziN0/7aee8YIUBoMCLtvuAHokZxW3nKulFCCbLeHi2HZDh4CFLbYx2 pQQopkDaThDW2z4/osM4/rsOLQjtNQEXLxaPV4oh23HAVtYAU8jjA4zklk8DirWe/iIp0jEz/KSIV 7h9ctRSxfaeWia5EJY+nBkHHItxGIpIMUWEm55LDUNMJwS2xdNOJqIp0fF5sYKxBsm6Yoh582VSFu ZjU0a/R4X0sCVqefZYXg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sZ3mv-000000008Xr-1ki4; Wed, 31 Jul 2024 07:30:21 +0000 Received: from mail-wm1-x334.google.com ([2a00:1450:4864:20::334]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sZ3mr-000000008VW-1jol for linux-riscv@lists.infradead.org; Wed, 31 Jul 2024 07:30:19 +0000 Received: by mail-wm1-x334.google.com with SMTP id 5b1f17b1804b1-428178fc07eso31791775e9.3 for ; Wed, 31 Jul 2024 00:30:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1722411016; x=1723015816; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ZEhBxTOflsSVFsDNAT/T0q8dd0IMJwzl7NNb3foLEqQ=; b=uU90fA0KvcdmZo6fWkXctlQz3RAY2JYD9Q3UeidLfaMNeatzpG73oU4phRqc6NTVIa kMiWMOkQmbvG5QVbpZ5Y2zy4zAYk9fjWx20Cv5RlPwHUYKZi3KvX8eso19LAs9gb+5yt ifXa1EferWQlRXsax4Rtp3chwyDxfnoxW3wDGhVsNH9TYj2sAScgrec5sI0P2chx9ShZ aF1+KSIpbH8MLqiEQLh4l3iJ5les1ZwaCuwE/ZwvCZdT2VdO6a4+m1WnoQG2uakUIXDL fPq7WApKMsBRim+H5uCwzzagQEZmMf9zejXQr8A5GRhKR11H5JnvA6V0ezK1ZP6zI3dZ QElQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722411016; x=1723015816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ZEhBxTOflsSVFsDNAT/T0q8dd0IMJwzl7NNb3foLEqQ=; b=XunyZCC0FGTBZgbHpt7j3qCAsX/uDksUS+6g2j0EDeIZumXbfbEsUVmhmGPojk0fqz aewaixF8lCCA1i3FazEe3+2/Z+TA0Tp3VC94L6A45h2C+A1rW2an8jahUySa7u3ES25+ h9En7/8uiZ/1aWYCz9oVs8bbXdNU3bik778869FY/L2EcxGJAM7vP8RAsyh/UEkqw0WT +ALyrsFV7C1VMPR7lEXXg0+M2rQ6Zh2XjcOSl874jrQzjRzC7DRiqzwLErKXIpLHOH6n L3YoqOrnirZOueGGFoQSiF6w74WykT6VSLfHdBMOrpyt5Lvwb5BwaSTP1M0eSLTUcnJ2 hK5g== X-Forwarded-Encrypted: i=1; AJvYcCUYPL7H6kngCu0wiAIQSWl0EwumkQaEQcSlZKTebvz9FfSAP6EAPmQNGBVXEgIX2S6RPf48hdL606WhldQXrHCHeYqR7UYwjNoycDdKDgtK X-Gm-Message-State: AOJu0YwGGYJsRdja7qmrKa+EcIFi71J22P3JYjpmdmbCVtY1MWkY6yix xHd8zKuEb4TJxNSSRc+FDt/97+3Sd66sW0tyLpopaIq50iYAb7Ocyu3FrYvEFGE= X-Google-Smtp-Source: AGHT+IG/D6snF/aUaHw8KmNLUqH1g4Di4cdhfBmk7rNdHJS5BuCS2wlMZCx/S8dP1xXXPnipMharHQ== X-Received: by 2002:a05:600c:5492:b0:426:5b21:97fa with SMTP id 5b1f17b1804b1-42811dd19camr83992885e9.29.1722411015502; Wed, 31 Jul 2024 00:30:15 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4282bb9d54esm10549595e9.43.2024.07.31.00.30.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jul 2024 00:30:15 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti , Andrea Parri Subject: [PATCH v4 06/13] riscv: Improve zacas fully-ordered cmpxchg() Date: Wed, 31 Jul 2024 09:23:58 +0200 Message-Id: <20240731072405.197046-7-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240731072405.197046-1-alexghiti@rivosinc.com> References: <20240731072405.197046-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240731_003017_480693_961BDB1C X-CRM114-Status: GOOD ( 10.82 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org The current fully-ordered cmpxchgXX() implementation results in: amocas.X.rl a5,a4,(s1) fence rw,rw This provides enough sync but we can actually use the following better mapping instead: amocas.X.aqrl a5,a4,(s1) Suggested-by: Andrea Parri Signed-off-by: Alexandre Ghiti --- arch/riscv/include/asm/cmpxchg.h | 72 +++++++++++++++++++------------- 1 file changed, 44 insertions(+), 28 deletions(-) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index ebcd4a30ae60..391730367213 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -107,8 +107,10 @@ * store NEW in MEM. Return the initial value in MEM. Success is * indicated by comparing RETURN with OLD. */ - -#define __arch_cmpxchg_masked(sc_sfx, cas_sfx, prepend, append, r, p, o, n) \ +#define __arch_cmpxchg_masked(sc_sfx, cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append, \ + r, p, o, n) \ ({ \ if (IS_ENABLED(CONFIG_RISCV_ISA_ZABHA) && \ IS_ENABLED(CONFIG_RISCV_ISA_ZACAS) && \ @@ -117,9 +119,9 @@ r = o; \ \ __asm__ __volatile__ ( \ - prepend \ + cas_prepend \ " amocas" cas_sfx " %0, %z2, %1\n" \ - append \ + cas_append \ : "+&r" (r), "+A" (*(p)) \ : "rJ" (n) \ : "memory"); \ @@ -134,7 +136,7 @@ ulong __rc; \ \ __asm__ __volatile__ ( \ - prepend \ + sc_prepend \ "0: lr.w %0, %2\n" \ " and %1, %0, %z5\n" \ " bne %1, %z3, 1f\n" \ @@ -142,7 +144,7 @@ " or %1, %1, %z4\n" \ " sc.w" sc_sfx " %1, %1, %2\n" \ " bnez %1, 0b\n" \ - append \ + sc_append \ "1:\n" \ : "=&r" (__retx), "=&r" (__rc), "+A" (*(__ptr32b)) \ : "rJ" ((long)__oldx), "rJ" (__newx), \ @@ -153,16 +155,19 @@ } \ }) -#define __arch_cmpxchg(lr_sfx, sc_cas_sfx, prepend, append, r, p, co, o, n) \ +#define __arch_cmpxchg(lr_sfx, sc_sfx, cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append, \ + r, p, co, o, n) \ ({ \ if (IS_ENABLED(CONFIG_RISCV_ISA_ZACAS) && \ riscv_has_extension_unlikely(RISCV_ISA_EXT_ZACAS)) { \ r = o; \ \ __asm__ __volatile__ ( \ - prepend \ - " amocas" sc_cas_sfx " %0, %z2, %1\n" \ - append \ + cas_prepend \ + " amocas" cas_sfx " %0, %z2, %1\n" \ + cas_append \ : "+&r" (r), "+A" (*(p)) \ : "rJ" (n) \ : "memory"); \ @@ -170,12 +175,12 @@ register unsigned int __rc; \ \ __asm__ __volatile__ ( \ - prepend \ + sc_prepend \ "0: lr" lr_sfx " %0, %2\n" \ " bne %0, %z3, 1f\n" \ - " sc" sc_cas_sfx " %1, %z4, %2\n" \ + " sc" sc_sfx " %1, %z4, %2\n" \ " bnez %1, 0b\n" \ - append \ + sc_append \ "1:\n" \ : "=&r" (r), "=&r" (__rc), "+A" (*(p)) \ : "rJ" (co o), "rJ" (n) \ @@ -183,7 +188,9 @@ } \ }) -#define _arch_cmpxchg(ptr, old, new, sc_cas_sfx, prepend, append) \ +#define _arch_cmpxchg(ptr, old, new, sc_sfx, cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append) \ ({ \ __typeof__(ptr) __ptr = (ptr); \ __typeof__(*(__ptr)) __old = (old); \ @@ -192,22 +199,28 @@ \ switch (sizeof(*__ptr)) { \ case 1: \ - __arch_cmpxchg_masked(sc_cas_sfx, ".b" sc_cas_sfx, \ - prepend, append, \ - __ret, __ptr, __old, __new); \ + __arch_cmpxchg_masked(sc_sfx, ".b" cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append, \ + __ret, __ptr, __old, __new); \ break; \ case 2: \ - __arch_cmpxchg_masked(sc_cas_sfx, ".h" sc_cas_sfx, \ - prepend, append, \ - __ret, __ptr, __old, __new); \ + __arch_cmpxchg_masked(sc_sfx, ".h" cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append, \ + __ret, __ptr, __old, __new); \ break; \ case 4: \ - __arch_cmpxchg(".w", ".w" sc_cas_sfx, prepend, append, \ - __ret, __ptr, (long), __old, __new); \ + __arch_cmpxchg(".w", ".w" sc_sfx, ".w" cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append, \ + __ret, __ptr, (long), __old, __new); \ break; \ case 8: \ - __arch_cmpxchg(".d", ".d" sc_cas_sfx, prepend, append, \ - __ret, __ptr, /**/, __old, __new); \ + __arch_cmpxchg(".d", ".d" sc_sfx, ".d" cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append, \ + __ret, __ptr, /**/, __old, __new); \ break; \ default: \ BUILD_BUG(); \ @@ -216,16 +229,19 @@ }) #define arch_cmpxchg_relaxed(ptr, o, n) \ - _arch_cmpxchg((ptr), (o), (n), "", "", "") + _arch_cmpxchg((ptr), (o), (n), "", "", "", "", "", "") #define arch_cmpxchg_acquire(ptr, o, n) \ - _arch_cmpxchg((ptr), (o), (n), "", "", RISCV_ACQUIRE_BARRIER) + _arch_cmpxchg((ptr), (o), (n), "", "", \ + "", RISCV_ACQUIRE_BARRIER, "", RISCV_ACQUIRE_BARRIER) #define arch_cmpxchg_release(ptr, o, n) \ - _arch_cmpxchg((ptr), (o), (n), "", RISCV_RELEASE_BARRIER, "") + _arch_cmpxchg((ptr), (o), (n), "", "", \ + RISCV_RELEASE_BARRIER, "", RISCV_RELEASE_BARRIER, "") #define arch_cmpxchg(ptr, o, n) \ - _arch_cmpxchg((ptr), (o), (n), ".rl", "", " fence rw, rw\n") + _arch_cmpxchg((ptr), (o), (n), ".rl", ".aqrl", \ + "", RISCV_FULL_BARRIER, "", "") #define arch_cmpxchg_local(ptr, o, n) \ arch_cmpxchg_relaxed((ptr), (o), (n)) From patchwork Wed Jul 31 07:23:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13748238 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0F42FC3DA7F for ; Wed, 31 Jul 2024 07:31:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=JCeSUs4HXSUCwtUXm7G1vMoQX/wcz1BqM0xQP69qu9A=; b=Iu4Xtj/RvMiRVi tH4M5tpL2oNkRNCzZYEAlmagtkztS0Si0f0u0XyHZDJXRTSziGTf/5OAouFh3UOpv92ZylkphLLlT ICTMcjI5zkeBr2VYBzIRU1PJNom++uR+rBb8r/nCtxyX3nfhZMn+uQyvmU49Pk15tzY6h2V/l2XNP JVAYRvl9XV/BViRIYIFXE1QwQsoJsu9rOWDiEJrlGf7Ze/Ag3QiTHqsW9D9gTAZD2e5c68hRG6Wlu AKaiR0fgAw/9covrLMuB6ZXo5mbFsnKPRAHnvU/a1Vj7Ekd8Upz15oQtci1WnkDnOKIFb2WgksPV4 DAsQv16F8cR3NmMs7wAw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sZ3nt-000000008q2-1UgP; Wed, 31 Jul 2024 07:31:21 +0000 Received: from mail-wm1-x330.google.com ([2a00:1450:4864:20::330]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sZ3nq-000000008o5-2pvI for linux-riscv@lists.infradead.org; Wed, 31 Jul 2024 07:31:20 +0000 Received: by mail-wm1-x330.google.com with SMTP id 5b1f17b1804b1-428163f7635so33109255e9.2 for ; Wed, 31 Jul 2024 00:31:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1722411077; x=1723015877; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BHoHYllmH7EYuYJy+Jv8B241H4q5/CnBJ4HnToKFT0c=; b=R/QK68O61+nvVpCn3D/1OH+qS/KgSI3e2psV2xuvV31+m6GPbHFVroXay3lgHidjl8 Ne1NmlEU4f6/EEiEznG7V47baf4+1uS8gbIadURI5pdoiEzG8MjHjsm4UzunOs09oBCC OrvQjSH25eJT4Zqo1EJMNS2lmBO44RkVFUVwLWbZkWZzOF6UB74Kv+bDJu0mElKwV6yX Rbyd/U3Oozdr9+ccila9LywGSOGNOVOqqVpRYNYhbgzNZJpPixxc0ld7SoghbeTBTfG5 KMyt4/nmLuyqIAFsnheFBbiGcbhDgWYljCDMHTSprxK4bp8x6whe38kVCz+C52bXqYSb d9qA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722411077; x=1723015877; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BHoHYllmH7EYuYJy+Jv8B241H4q5/CnBJ4HnToKFT0c=; b=AHv7TKm7sTuKBH7nhWjOMDCcWiEMBb4yvVdOz8vZRkU4rtJbNTybjPJRKtbt15TmkP PYoQKqn3FBYDtoDhik+IOqCgGqkBzkoHu7kc4Z6ow09+SmTHJWIonMqeuTiDyfESUGDN c3T2JYrko3xH5pEjMJehAkc8QfnE6y7YevDp/QZYYmOfAwU5OFq0HxNqKinW5Ky/d6Br LKjAyKVQ9b0qi+KTaMlaDg+dVxlLESUwEr1ynf0AdV+nQRyFj+SkM5YUK/3NKs07scas hqz2rME8FYlAo9dYm2LuXgcGNgQ1uiFmVZS+2NWfuIdafPv9ejT1GrH70ku3UA5vIq94 7b1A== X-Forwarded-Encrypted: i=1; AJvYcCUDRuVPxKkdleNbbXzDVOklq7kKEf+dBgskFV61lp2/YudxIPH4kSMrUUhWd40Xmvu9vGcf7arJERg3y/HBVgdUsZnWMPfE4j3H7B4VchOm X-Gm-Message-State: AOJu0YxSY7h0hsO8/xuP07nj7aFjaAx9FkEBzqCqoRE2QNz46bHFLETn l4WPr2yqq/Sg0xU48mqvzoYh/em+SlK2YYyomOQ9rN3AcosVmaVc3O+TLzXLrzM= X-Google-Smtp-Source: AGHT+IFWKmJ8e4ndadH473CTdPU3AlRXobOboTCXTtcOkKGrZDLPp8Y0XiEkCzxGJJj4GbuZsEuDpQ== X-Received: by 2002:a05:600c:4447:b0:426:5c34:b19b with SMTP id 5b1f17b1804b1-42811d9af49mr92305945e9.20.1722411076723; Wed, 31 Jul 2024 00:31:16 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4282bac8069sm10520875e9.28.2024.07.31.00.31.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jul 2024 00:31:16 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH v4 07/13] riscv: Implement arch_cmpxchg128() using Zacas Date: Wed, 31 Jul 2024 09:23:59 +0200 Message-Id: <20240731072405.197046-8-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240731072405.197046-1-alexghiti@rivosinc.com> References: <20240731072405.197046-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240731_003118_760320_3E2E5EAF X-CRM114-Status: GOOD ( 12.93 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Now that Zacas is supported in the kernel, let's use the double word atomic version of amocas to improve the SLUB allocator. Note that we have to select fixed registers, otherwise gcc fails to pick even registers and then produces a reserved encoding which fails to assemble. Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones --- arch/riscv/Kconfig | 1 + arch/riscv/include/asm/cmpxchg.h | 38 ++++++++++++++++++++++++++++++++ 2 files changed, 39 insertions(+) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 212ec2aab389..ef55ab94027e 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -111,6 +111,7 @@ config RISCV select GENERIC_VDSO_TIME_NS if HAVE_GENERIC_VDSO select HARDIRQS_SW_RESEND select HAS_IOPORT if MMU + select HAVE_ALIGNED_STRUCT_PAGE select HAVE_ARCH_AUDITSYSCALL select HAVE_ARCH_HUGE_VMALLOC if HAVE_ARCH_HUGE_VMAP select HAVE_ARCH_HUGE_VMAP if MMU && 64BIT diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index 391730367213..ce9613516bbb 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -276,6 +276,44 @@ arch_cmpxchg_release((ptr), (o), (n)); \ }) +#if defined(CONFIG_64BIT) && defined(CONFIG_RISCV_ISA_ZACAS) + +#define system_has_cmpxchg128() riscv_has_extension_unlikely(RISCV_ISA_EXT_ZACAS) + +union __u128_halves { + u128 full; + struct { + u64 low, high; + }; +}; + +#define __arch_cmpxchg128(p, o, n, cas_sfx) \ +({ \ + __typeof__(*(p)) __o = (o); \ + union __u128_halves __hn = { .full = (n) }; \ + union __u128_halves __ho = { .full = (__o) }; \ + register unsigned long t1 asm ("t1") = __hn.low; \ + register unsigned long t2 asm ("t2") = __hn.high; \ + register unsigned long t3 asm ("t3") = __ho.low; \ + register unsigned long t4 asm ("t4") = __ho.high; \ + \ + __asm__ __volatile__ ( \ + " amocas.q" cas_sfx " %0, %z3, %2" \ + : "+&r" (t3), "+&r" (t4), "+A" (*(p)) \ + : "rJ" (t1), "rJ" (t2) \ + : "memory"); \ + \ + ((u128)t4 << 64) | t3; \ +}) + +#define arch_cmpxchg128(ptr, o, n) \ + __arch_cmpxchg128((ptr), (o), (n), ".aqrl") + +#define arch_cmpxchg128_local(ptr, o, n) \ + __arch_cmpxchg128((ptr), (o), (n), "") + +#endif /* CONFIG_64BIT && CONFIG_RISCV_ISA_ZACAS */ + #ifdef CONFIG_RISCV_ISA_ZAWRS /* * Despite wrs.nto being "WRS-with-no-timeout", in the absence of changes to From patchwork Wed Jul 31 07:24:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13748240 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 77424C3DA64 for ; Wed, 31 Jul 2024 07:32:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=+enbuZzYHKqIDiuxke7hDE7QvqaI/7rNH6LmiKTaprc=; b=fu6OjXJXLQ3SiI /YgUIjkFSuuyz5KnIFC2kT3i8I2cqRhLwGhUJLhH5Dj/AD2lLoHnkS8Ef9aprnGkeQb92yidhd/Kj /9DiFK5WT8lbBdmpJ9vnKHvm7/vL/e02y+TVIyeSViV9WYGCmfLS3qtH58ef5JF1t5jIpAyImVy7h thn/gosJ0VT2gTYxK7rvLOfWuU97wdHM960MAvfg6SWOOLn3oXrvmZYqc9jetVOTw9ZgR558w9leU UYbxIHqFGYH/IBB72vDtJTiBvg04M0MfKovfQFwyi4kITohZptCk4SQGm7v2BItrVAqa6soNEyTOs wmHU1pCMu4KMaFpcXTBw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sZ3ov-0000000093i-3JSF; Wed, 31 Jul 2024 07:32:25 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sZ3ou-00000000936-3007 for linux-riscv@bombadil.infradead.org; Wed, 31 Jul 2024 07:32:24 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Transfer-Encoding:MIME-Version :References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=xze2cs107FZ3zEBdldNBuzUZPwt1bYMtkFI7VhzQeZI=; b=WCgBz6MBacfKChetuQMPIoPSYD zTYJdQcdyS+DcAQNwj88Gv9xNUsDy1gDdMVfwIAcdH8NtxO0ee5VgYP3X1q28ndw701WKfP6CtT6Y B7DLAN925sPIRrxLQQ5R3Q/4vktIFmWDatZfbIDADyOzUfNeEWVX2eMroU2EQnzP8rdxyPy3FH4rI /oB3MS3098dWQfdJPAuW7ue2V32rB2meP01l1cZPiM1pIWJZSawUn6zk75sekwfzbNSQzGoeF6G9E yhKwIwZUVALQOErbL3gRu7nbLcb0g6ZMp//rGPyZN2H8Pc6y1vMSK2X1AVY0xSy1rgJ2V2WeSdBhH f4kBH0bg==; Received: from mail-wr1-x42c.google.com ([2a00:1450:4864:20::42c]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sZ3or-00000005A5p-0bL5 for linux-riscv@lists.infradead.org; Wed, 31 Jul 2024 07:32:23 +0000 Received: by mail-wr1-x42c.google.com with SMTP id ffacd0b85a97d-3685b3dbcdcso3021803f8f.3 for ; Wed, 31 Jul 2024 00:32:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1722411138; x=1723015938; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xze2cs107FZ3zEBdldNBuzUZPwt1bYMtkFI7VhzQeZI=; b=FFfGU9JETNx+z8nfY60wgnVLhJUo+casCeCIW10pdmacbb1m86p+TxYyoddLdbB4G5 tlxh3qa5JmHYqo0GMsrLb5XFYj9PwR0weYg//todxl8Nu996thJc4PwXSPM/ABS4e16Z CO4wpgF8eaVLBWYwaVEave0IlTBOsgrYav0X+CfSeS/eyZber2+XSIQnufPEaLJSBSwy 0BqoDwOXkwTKwF3T1kqnAenrvF47uYOHlSEoqhUkr6igQeqvKNmp/RfMyJVbf+d3gjbv 4g+U+cWbsunxiILnjaFQLIMTNB9Ng2wvyEx2KiaVdEE12LYxwD3ybbQu+mR5bWr1ZpR5 Zd6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722411138; x=1723015938; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xze2cs107FZ3zEBdldNBuzUZPwt1bYMtkFI7VhzQeZI=; b=ak41vziyjqxrxi919V5fqGXwohS9VLQXdlsdWAtCwePVaK9hDenm24FrBUvb8PbCIf FYDjI55BIpWIOkNUJ6TMFxkS+bVMvnrsKJvzn8H5g09NBk9RnDOLKa3QzuMcV5WBhXBD WeTw8tJ5Fezft+hV9kHYdBTMtiN7vtKra9T+66sgQ+5T3DPqTFqyPJXa6oRnGS0IYTwf D6zoKj9RuAwJ/SNy5z/Y8ujOwhJX4ZXrlODulPODA29ubsxv8cM61gd3wRir3bTuVRkF dVsyggxPHWOJ7ToXsf3o4f1S6Mqeav3Dth11jwNWq7eBC2dXnbbf57/dxvGgz642omyr L60A== X-Forwarded-Encrypted: i=1; AJvYcCXDWY/otvsTElIReqV+KZTxQM8IDSm/hUu1DZ+YDu7dK/dayppcjjGUudrjvltXlBb9S1SyMH6+MG6RNeYYkyn0TuoZxjLJrheaNjKjfRbb X-Gm-Message-State: AOJu0Yx/RzCfJKCCARGG8DnppeXdj8oFRicZjjji0fcTFBXmG2g6gOiq AqvQniG9IJgFLKIkr4ZI6qE4JrFM6oaBmNCS/u4ePnNOfmr5GCMkFetqAxUH3Qo= X-Google-Smtp-Source: AGHT+IHQ02v4jgJ0i8k8ZC9oEkha1vnkq1LaDLws9riLxBs06x93VaBTTty+hQP6mbP1ktT3LB3TnA== X-Received: by 2002:adf:e707:0:b0:367:9088:fecc with SMTP id ffacd0b85a97d-36b5d0b04a6mr9601634f8f.7.1722411137984; Wed, 31 Jul 2024 00:32:17 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-36b367c02casm16371948f8f.15.2024.07.31.00.32.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jul 2024 00:32:17 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH v4 08/13] riscv: Implement xchg8/16() using Zabha Date: Wed, 31 Jul 2024 09:24:00 +0200 Message-Id: <20240731072405.197046-9-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240731072405.197046-1-alexghiti@rivosinc.com> References: <20240731072405.197046-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240731_083221_448243_2D87CB98 X-CRM114-Status: GOOD ( 10.61 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org This adds runtime support for Zabha in xchg8/16() operations. Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones --- arch/riscv/include/asm/cmpxchg.h | 65 ++++++++++++++++++++------------ 1 file changed, 41 insertions(+), 24 deletions(-) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index ce9613516bbb..7484f063809e 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -14,29 +14,41 @@ #include #include -#define __arch_xchg_masked(sc_sfx, prepend, append, r, p, n) \ -({ \ - u32 *__ptr32b = (u32 *)((ulong)(p) & ~0x3); \ - ulong __s = ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \ - ulong __mask = GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \ - << __s; \ - ulong __newx = (ulong)(n) << __s; \ - ulong __retx; \ - ulong __rc; \ - \ - __asm__ __volatile__ ( \ - prepend \ - "0: lr.w %0, %2\n" \ - " and %1, %0, %z4\n" \ - " or %1, %1, %z3\n" \ - " sc.w" sc_sfx " %1, %1, %2\n" \ - " bnez %1, 0b\n" \ - append \ - : "=&r" (__retx), "=&r" (__rc), "+A" (*(__ptr32b)) \ - : "rJ" (__newx), "rJ" (~__mask) \ - : "memory"); \ - \ - r = (__typeof__(*(p)))((__retx & __mask) >> __s); \ +#define __arch_xchg_masked(sc_sfx, swap_sfx, prepend, sc_append, \ + swap_append, r, p, n) \ +({ \ + if (IS_ENABLED(CONFIG_RISCV_ISA_ZABHA) && \ + riscv_has_extension_unlikely(RISCV_ISA_EXT_ZABHA)) { \ + __asm__ __volatile__ ( \ + prepend \ + " amoswap" swap_sfx " %0, %z2, %1\n" \ + swap_append \ + : "=&r" (r), "+A" (*(p)) \ + : "rJ" (n) \ + : "memory"); \ + } else { \ + u32 *__ptr32b = (u32 *)((ulong)(p) & ~0x3); \ + ulong __s = ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \ + ulong __mask = GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \ + << __s; \ + ulong __newx = (ulong)(n) << __s; \ + ulong __retx; \ + ulong __rc; \ + \ + __asm__ __volatile__ ( \ + prepend \ + "0: lr.w %0, %2\n" \ + " and %1, %0, %z4\n" \ + " or %1, %1, %z3\n" \ + " sc.w" sc_sfx " %1, %1, %2\n" \ + " bnez %1, 0b\n" \ + sc_append \ + : "=&r" (__retx), "=&r" (__rc), "+A" (*(__ptr32b)) \ + : "rJ" (__newx), "rJ" (~__mask) \ + : "memory"); \ + \ + r = (__typeof__(*(p)))((__retx & __mask) >> __s); \ + } \ }) #define __arch_xchg(sfx, prepend, append, r, p, n) \ @@ -59,8 +71,13 @@ \ switch (sizeof(*__ptr)) { \ case 1: \ + __arch_xchg_masked(sc_sfx, ".b" swap_sfx, \ + prepend, sc_append, swap_append, \ + __ret, __ptr, __new); \ + break; \ case 2: \ - __arch_xchg_masked(sc_sfx, prepend, sc_append, \ + __arch_xchg_masked(sc_sfx, ".h" swap_sfx, \ + prepend, sc_append, swap_append, \ __ret, __ptr, __new); \ break; \ case 4: \ From patchwork Wed Jul 31 07:24:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13748241 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BBDABC3DA64 for ; Wed, 31 Jul 2024 07:33:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=81l1vnndeH6RGQ7RI3IttTQT/Ue2c84FixhX6Y1lqPk=; b=2cyQdW4fxxCG+W F+IUMW9xg8Rc97EtPYhkVJe7xf26dRGxc28cq5eSyU8LKRpvMKvEHA+Vjo/0Gieg+DDF4bsp+Bv4i /kM7Bw3TQTRm7wagsBqe4lZwKMIWyLgQ07yeNVDdt3CGFvKeH8o7ztFfCNs4DwTLLwqJcn1aMVmZD LSSLVuYXO5K7b2yrX9VBVd6JUOh8crtKxCwp0Ox+rG8XA5b+cWHLMjoDvDKpjw8P+XKxIb52sBuWQ SUucm8OuVh3+rTfqOUwjLQ3tqw6PNSb21kMbNyUH8c50UftEUaahZJPMENjGU0fc8VS/eojrP8tOj Fe98V+1PBsuhGEN9xlng==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sZ3pv-000000009G0-1pBi; Wed, 31 Jul 2024 07:33:27 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sZ3pt-000000009Fc-1wtI for linux-riscv@bombadil.infradead.org; Wed, 31 Jul 2024 07:33:25 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Transfer-Encoding:MIME-Version :References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=heAbIXv1oMx4SoDoWG1mJ8ALiEwmwn+mI3czPDKQjFY=; b=JT21GbOUqDK69Hd9t/zgge3TWd je2JkZweA2zs4jX+TYobPa9OGY5BzYR64wmE8/iHyf1qMvuuGg5x4uDVUxiw4vJeni/pb+eLteQox m8hI3ZvVhKxhhUTs4gwKIkxWFbxpnR5gm+JClXsAhvXSyfz1c48IFJ1aLrbeqw4rEPeNjF78RwjG9 2D68KctmFSx0YL7tZvjw1lY4pPkU+egDAscrtIbD+gpkV6LtBtnAk+onLaOXr0oLdr0xPQIOt+YmB FB1irdXNZNsZdrAKQkAySYSEvAznHJ9L9D3pV3yZpdcdnUqF/gkeWR0coPGFWcC46RQ5Mmmw/bas+ nJWoXZQg==; Received: from mail-wm1-x332.google.com ([2a00:1450:4864:20::332]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sZ3pp-00000005ADS-3eQV for linux-riscv@lists.infradead.org; Wed, 31 Jul 2024 07:33:24 +0000 Received: by mail-wm1-x332.google.com with SMTP id 5b1f17b1804b1-4280bbdad3dso34891675e9.0 for ; Wed, 31 Jul 2024 00:33:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1722411199; x=1723015999; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=heAbIXv1oMx4SoDoWG1mJ8ALiEwmwn+mI3czPDKQjFY=; b=a0xoB/rkQiV9AjSuoI6WDeph+g+0zwW71+GoSStWcV4nXTOpqKrga90SftP9yPcckN GWjTfzhri+VsBRlSVZ9z1XMLeNTDnpzhexy08PcvMz/X6ixfKlLTlzYsGxwqlq96JPER cLoQFKhssl9gSuDBAXyekoVntGaVkitHGSRz5udoGAQ4ro3iZjCBlNUFz/OrgMP/6cPx EDfKcFvSu+dl4zziEe+Ys7/b+fFjOd5FU5vZTL9xqtQ6F0jdxp+p1hZzvrfKficXpA4g 6ccEon2CGo2TyXlCrK60qABCmTtjVjdYt/sr/WVssyCkRA9dPKcpVwQMZqodO9nbPKyL x07g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722411199; x=1723015999; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=heAbIXv1oMx4SoDoWG1mJ8ALiEwmwn+mI3czPDKQjFY=; b=XkanqVbhLwIgrFGrViaBkllCC9duxzQ63nF7LxmKVkP56QZnooo4BoYh2kWHZMsTv3 pY5yA6CvIa1NOU6MWPN5lgU7Ej2ymSM6Kb5QzftlA9ixhaxp+AeQKFlYD+GwToCTdpy1 hQKA/vHTQQ64q60SiXEQdQnM42B7oDQq+vrQKdr9okgneHZoi/A1YRgOwqJnnw0swPfm rtIcMbGACCAcvGQLWk010RMagSbCnGpsZ4p8E5tpxpHFmStQ+9o3RODbadxLmfYma5+J pzHLed2B9spwaTAr8Dty7ib6ue46jUCgTEOmZHPG2yQY1Rbp3wfocksj/tVcRtjXoqvJ XxAw== X-Forwarded-Encrypted: i=1; AJvYcCUGgNZathgymfJrRwYcalmNEsbLJsUZt9Wr63/Mb/U2koGGG2ZlTFYpv26Kd/k1ZKDOMly7iTZ/kyRY7PGjosS0CpuWTy/ttUBNzW6PEgf7 X-Gm-Message-State: AOJu0YxpGD7ISCVQ/gxN9mj7pDM88WxAlmR0hzrY24vzbD2bnTfUqge+ 4llfOQqxvncHbwu/i5pRZshnc4BpzVHBgp61lDtVAIPgRSa7nOudkH/EZdPkMEo= X-Google-Smtp-Source: AGHT+IEcnY6AHoPV/M+h1TDSqxuXyVV5Hu7UR4S5fwQ9ffgAKn2jmSi+Y4GhJoM3ugRGgONDlr03/Q== X-Received: by 2002:a5d:6d81:0:b0:368:4c38:a669 with SMTP id ffacd0b85a97d-36b5cee2e5fmr11880218f8f.10.1722411199233; Wed, 31 Jul 2024 00:33:19 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-36b367e51f0sm16335609f8f.46.2024.07.31.00.33.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jul 2024 00:33:18 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Guo Ren Subject: [PATCH v4 09/13] asm-generic: ticket-lock: Reuse arch_spinlock_t of qspinlock Date: Wed, 31 Jul 2024 09:24:01 +0200 Message-Id: <20240731072405.197046-10-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240731072405.197046-1-alexghiti@rivosinc.com> References: <20240731072405.197046-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240731_083322_234812_DB781375 X-CRM114-Status: GOOD ( 13.42 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Guo Ren The arch_spinlock_t of qspinlock has contained the atomic_t val, which satisfies the ticket-lock requirement. Thus, unify the arch_spinlock_t into qspinlock_types.h. This is the preparation for the next combo spinlock. Reviewed-by: Leonardo Bras Suggested-by: Arnd Bergmann Link: https://lore.kernel.org/linux-riscv/CAK8P3a2rnz9mQqhN6-e0CGUUv9rntRELFdxt_weiD7FxH7fkfQ@mail.gmail.com/ Signed-off-by: Guo Ren Signed-off-by: Guo Ren --- include/asm-generic/spinlock.h | 14 +++++++------- include/asm-generic/spinlock_types.h | 12 ++---------- 2 files changed, 9 insertions(+), 17 deletions(-) diff --git a/include/asm-generic/spinlock.h b/include/asm-generic/spinlock.h index 90803a826ba0..4773334ee638 100644 --- a/include/asm-generic/spinlock.h +++ b/include/asm-generic/spinlock.h @@ -32,7 +32,7 @@ static __always_inline void arch_spin_lock(arch_spinlock_t *lock) { - u32 val = atomic_fetch_add(1<<16, lock); + u32 val = atomic_fetch_add(1<<16, &lock->val); u16 ticket = val >> 16; if (ticket == (u16)val) @@ -46,31 +46,31 @@ static __always_inline void arch_spin_lock(arch_spinlock_t *lock) * have no outstanding writes due to the atomic_fetch_add() the extra * orderings are free. */ - atomic_cond_read_acquire(lock, ticket == (u16)VAL); + atomic_cond_read_acquire(&lock->val, ticket == (u16)VAL); smp_mb(); } static __always_inline bool arch_spin_trylock(arch_spinlock_t *lock) { - u32 old = atomic_read(lock); + u32 old = atomic_read(&lock->val); if ((old >> 16) != (old & 0xffff)) return false; - return atomic_try_cmpxchg(lock, &old, old + (1<<16)); /* SC, for RCsc */ + return atomic_try_cmpxchg(&lock->val, &old, old + (1<<16)); /* SC, for RCsc */ } static __always_inline void arch_spin_unlock(arch_spinlock_t *lock) { u16 *ptr = (u16 *)lock + IS_ENABLED(CONFIG_CPU_BIG_ENDIAN); - u32 val = atomic_read(lock); + u32 val = atomic_read(&lock->val); smp_store_release(ptr, (u16)val + 1); } static __always_inline int arch_spin_value_unlocked(arch_spinlock_t lock) { - u32 val = lock.counter; + u32 val = lock.val.counter; return ((val >> 16) == (val & 0xffff)); } @@ -84,7 +84,7 @@ static __always_inline int arch_spin_is_locked(arch_spinlock_t *lock) static __always_inline int arch_spin_is_contended(arch_spinlock_t *lock) { - u32 val = atomic_read(lock); + u32 val = atomic_read(&lock->val); return (s16)((val >> 16) - (val & 0xffff)) > 1; } diff --git a/include/asm-generic/spinlock_types.h b/include/asm-generic/spinlock_types.h index 8962bb730945..f534aa5de394 100644 --- a/include/asm-generic/spinlock_types.h +++ b/include/asm-generic/spinlock_types.h @@ -3,15 +3,7 @@ #ifndef __ASM_GENERIC_SPINLOCK_TYPES_H #define __ASM_GENERIC_SPINLOCK_TYPES_H -#include -typedef atomic_t arch_spinlock_t; - -/* - * qrwlock_types depends on arch_spinlock_t, so we must typedef that before the - * include. - */ -#include - -#define __ARCH_SPIN_LOCK_UNLOCKED ATOMIC_INIT(0) +#include +#include #endif /* __ASM_GENERIC_SPINLOCK_TYPES_H */ From patchwork Wed Jul 31 07:24:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13748242 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3762BC3DA7F for ; Wed, 31 Jul 2024 07:34:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Ch1fU7HxtxblHEnw2eK8E63qnmnxpbMDLgaMVzTElew=; b=oCaC7ByLSBwfY3 /SMvW0whqqx9jfZ/n1Ls3lgIOY+rsUlHrNC3lYIQygdFYmatDYzjbbZEByCBkJDeWEFGjskhdltoZ tFS83+VNRx6z9ywFL99Cpq/DuBRR55pskvgO/f0SiPYn60paVE5Ciu0gHB8AD1dBANu5/NQw/t3qU 1Y+CeGw3euDLJLbbEojTHonx6EE+yA2Qy0joDRCMRku5aTNvR0uM6a9Yle/CGAOvGyS+CFFqg5vqT ZtxCd9RANCksHzq6YG80kPXddCzZXgCMr87n7A7uF3ZPDpLhQGVbyMn1oF6Dq/vAA4Blv2/KudqqO Vl1MEmJPZsyk7UA6LgFg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sZ3qr-000000009Tg-0xtE; Wed, 31 Jul 2024 07:34:25 +0000 Received: from mail-wm1-x32f.google.com ([2a00:1450:4864:20::32f]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sZ3qo-000000009Sk-0jVW for linux-riscv@lists.infradead.org; Wed, 31 Jul 2024 07:34:23 +0000 Received: by mail-wm1-x32f.google.com with SMTP id 5b1f17b1804b1-428178fc07eso31816855e9.3 for ; Wed, 31 Jul 2024 00:34:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1722411260; x=1723016060; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=JpMFH6VmSNUk91jIQ+0kjGx11FiPCz2ltKgHshZWUPg=; b=sSGTmgStsb67RsNyii0EHMohIPVI65C6DpzCUnF7OXNia5SWj5lju+6gGa/OY+bQgy As7zKy3bc6B+p5I0n5+ggKI/MkYw8qVDH3wx/WkvQRbRJbLYQZEvbD+3RLQJPfqDOCAh 38rmoVy+w/RcX0PG5dThQyuIffmEOA9V8UVlPioHOzi6hgVHjSw4Dmzzl8MpTk/QiPcb MfizLSVLUiF4mXGI3e0NIzYK0XK6WgkQaXih70CLzaPnuFaWgGmRBKfJNwE09SPrvpvE DdRDTv5GWcc0l0rgYQzkkQmej8vUIIwbhJKE6VFJ9rdVdO9Dlnk6nyifyZbfmb9Oca1I W4AA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722411260; x=1723016060; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JpMFH6VmSNUk91jIQ+0kjGx11FiPCz2ltKgHshZWUPg=; b=XnX3u5YXBcYOVvnpvWKsdKQwXXXCJgflNO0puKvKFYS04qzzx+bJ7IrCkkETJ0Iqzf D0k8YYmmYW6J3iEnVPOrGpd+kQjSguac8KdyskHe/f4Pb28B0jHfceT8RZQsPqO6dafD YuOXmCABIZSrvn/aIXhicsRkSqqGJDWpTZBEjTPhGI4YxYs7WNajxb4WzgSppcR9tYVD 06h/jFtpwMP56qOksqcGOuT2QCdN/rEzpqVxewm5fzJdLdD117a73WcmxERPZKOHWoQY pXib7FMnDwn/NqbnivHbNuzuwId9tIaOxz9EYAHzIODB2cKoRBAXT36vSdo3RLNbOC6b gCWA== X-Forwarded-Encrypted: i=1; AJvYcCV1WQHnErF28ANnWjdQZCbuQ9qedKcUIbnTKf4rVKo62YREadORYByXkaYGQD0xoaotCCACFnGOS6Kghs8O6TIZ70GnnZrSMFtwH2dr4Wkj X-Gm-Message-State: AOJu0YwlD+qBECvsmKiNxo17oIpgCo2pIEn+cRgTiFHhXjTNZZOn5QBk NiE5+bDHRz95mS22r1GSI7eofwHlts6UwlIws2gc6jC9g2fQNGgJoPfu5oBsIJEr29K5+pZXym4 u X-Google-Smtp-Source: AGHT+IGGoWgAithnQ+FsXC5/r7ATF1hMT98LcKsT+9Dz8AUlHrEPaEsw7gl/cSyZggIjyGsVTn08vg== X-Received: by 2002:a05:600c:4706:b0:427:d8f7:b718 with SMTP id 5b1f17b1804b1-42811dd19f0mr88083515e9.24.1722411260519; Wed, 31 Jul 2024 00:34:20 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4282b89a946sm10886525e9.6.2024.07.31.00.34.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jul 2024 00:34:20 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Guo Ren Subject: [PATCH v4 10/13] asm-generic: ticket-lock: Add separate ticket-lock.h Date: Wed, 31 Jul 2024 09:24:02 +0200 Message-Id: <20240731072405.197046-11-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240731072405.197046-1-alexghiti@rivosinc.com> References: <20240731072405.197046-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240731_003422_248548_A0B55EE4 X-CRM114-Status: GOOD ( 22.16 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Guo Ren Add a separate ticket-lock.h to include multiple spinlock versions and select one at compile time or runtime. Reviewed-by: Leonardo Bras Suggested-by: Arnd Bergmann Link: https://lore.kernel.org/linux-riscv/CAK8P3a2rnz9mQqhN6-e0CGUUv9rntRELFdxt_weiD7FxH7fkfQ@mail.gmail.com/ Signed-off-by: Guo Ren Signed-off-by: Guo Ren --- include/asm-generic/spinlock.h | 87 +--------------------- include/asm-generic/ticket_spinlock.h | 103 ++++++++++++++++++++++++++ 2 files changed, 104 insertions(+), 86 deletions(-) create mode 100644 include/asm-generic/ticket_spinlock.h diff --git a/include/asm-generic/spinlock.h b/include/asm-generic/spinlock.h index 4773334ee638..970590baf61b 100644 --- a/include/asm-generic/spinlock.h +++ b/include/asm-generic/spinlock.h @@ -1,94 +1,9 @@ /* SPDX-License-Identifier: GPL-2.0 */ -/* - * 'Generic' ticket-lock implementation. - * - * It relies on atomic_fetch_add() having well defined forward progress - * guarantees under contention. If your architecture cannot provide this, stick - * to a test-and-set lock. - * - * It also relies on atomic_fetch_add() being safe vs smp_store_release() on a - * sub-word of the value. This is generally true for anything LL/SC although - * you'd be hard pressed to find anything useful in architecture specifications - * about this. If your architecture cannot do this you might be better off with - * a test-and-set. - * - * It further assumes atomic_*_release() + atomic_*_acquire() is RCpc and hence - * uses atomic_fetch_add() which is RCsc to create an RCsc hot path, along with - * a full fence after the spin to upgrade the otherwise-RCpc - * atomic_cond_read_acquire(). - * - * The implementation uses smp_cond_load_acquire() to spin, so if the - * architecture has WFE like instructions to sleep instead of poll for word - * modifications be sure to implement that (see ARM64 for example). - * - */ - #ifndef __ASM_GENERIC_SPINLOCK_H #define __ASM_GENERIC_SPINLOCK_H -#include -#include - -static __always_inline void arch_spin_lock(arch_spinlock_t *lock) -{ - u32 val = atomic_fetch_add(1<<16, &lock->val); - u16 ticket = val >> 16; - - if (ticket == (u16)val) - return; - - /* - * atomic_cond_read_acquire() is RCpc, but rather than defining a - * custom cond_read_rcsc() here we just emit a full fence. We only - * need the prior reads before subsequent writes ordering from - * smb_mb(), but as atomic_cond_read_acquire() just emits reads and we - * have no outstanding writes due to the atomic_fetch_add() the extra - * orderings are free. - */ - atomic_cond_read_acquire(&lock->val, ticket == (u16)VAL); - smp_mb(); -} - -static __always_inline bool arch_spin_trylock(arch_spinlock_t *lock) -{ - u32 old = atomic_read(&lock->val); - - if ((old >> 16) != (old & 0xffff)) - return false; - - return atomic_try_cmpxchg(&lock->val, &old, old + (1<<16)); /* SC, for RCsc */ -} - -static __always_inline void arch_spin_unlock(arch_spinlock_t *lock) -{ - u16 *ptr = (u16 *)lock + IS_ENABLED(CONFIG_CPU_BIG_ENDIAN); - u32 val = atomic_read(&lock->val); - - smp_store_release(ptr, (u16)val + 1); -} - -static __always_inline int arch_spin_value_unlocked(arch_spinlock_t lock) -{ - u32 val = lock.val.counter; - - return ((val >> 16) == (val & 0xffff)); -} - -static __always_inline int arch_spin_is_locked(arch_spinlock_t *lock) -{ - arch_spinlock_t val = READ_ONCE(*lock); - - return !arch_spin_value_unlocked(val); -} - -static __always_inline int arch_spin_is_contended(arch_spinlock_t *lock) -{ - u32 val = atomic_read(&lock->val); - - return (s16)((val >> 16) - (val & 0xffff)) > 1; -} - +#include #include #endif /* __ASM_GENERIC_SPINLOCK_H */ diff --git a/include/asm-generic/ticket_spinlock.h b/include/asm-generic/ticket_spinlock.h new file mode 100644 index 000000000000..cfcff22b37b3 --- /dev/null +++ b/include/asm-generic/ticket_spinlock.h @@ -0,0 +1,103 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +/* + * 'Generic' ticket-lock implementation. + * + * It relies on atomic_fetch_add() having well defined forward progress + * guarantees under contention. If your architecture cannot provide this, stick + * to a test-and-set lock. + * + * It also relies on atomic_fetch_add() being safe vs smp_store_release() on a + * sub-word of the value. This is generally true for anything LL/SC although + * you'd be hard pressed to find anything useful in architecture specifications + * about this. If your architecture cannot do this you might be better off with + * a test-and-set. + * + * It further assumes atomic_*_release() + atomic_*_acquire() is RCpc and hence + * uses atomic_fetch_add() which is RCsc to create an RCsc hot path, along with + * a full fence after the spin to upgrade the otherwise-RCpc + * atomic_cond_read_acquire(). + * + * The implementation uses smp_cond_load_acquire() to spin, so if the + * architecture has WFE like instructions to sleep instead of poll for word + * modifications be sure to implement that (see ARM64 for example). + * + */ + +#ifndef __ASM_GENERIC_TICKET_SPINLOCK_H +#define __ASM_GENERIC_TICKET_SPINLOCK_H + +#include +#include + +static __always_inline void ticket_spin_lock(arch_spinlock_t *lock) +{ + u32 val = atomic_fetch_add(1<<16, &lock->val); + u16 ticket = val >> 16; + + if (ticket == (u16)val) + return; + + /* + * atomic_cond_read_acquire() is RCpc, but rather than defining a + * custom cond_read_rcsc() here we just emit a full fence. We only + * need the prior reads before subsequent writes ordering from + * smb_mb(), but as atomic_cond_read_acquire() just emits reads and we + * have no outstanding writes due to the atomic_fetch_add() the extra + * orderings are free. + */ + atomic_cond_read_acquire(&lock->val, ticket == (u16)VAL); + smp_mb(); +} + +static __always_inline bool ticket_spin_trylock(arch_spinlock_t *lock) +{ + u32 old = atomic_read(&lock->val); + + if ((old >> 16) != (old & 0xffff)) + return false; + + return atomic_try_cmpxchg(&lock->val, &old, old + (1<<16)); /* SC, for RCsc */ +} + +static __always_inline void ticket_spin_unlock(arch_spinlock_t *lock) +{ + u16 *ptr = (u16 *)lock + IS_ENABLED(CONFIG_CPU_BIG_ENDIAN); + u32 val = atomic_read(&lock->val); + + smp_store_release(ptr, (u16)val + 1); +} + +static __always_inline int ticket_spin_value_unlocked(arch_spinlock_t lock) +{ + u32 val = lock.val.counter; + + return ((val >> 16) == (val & 0xffff)); +} + +static __always_inline int ticket_spin_is_locked(arch_spinlock_t *lock) +{ + arch_spinlock_t val = READ_ONCE(*lock); + + return !ticket_spin_value_unlocked(val); +} + +static __always_inline int ticket_spin_is_contended(arch_spinlock_t *lock) +{ + u32 val = atomic_read(&lock->val); + + return (s16)((val >> 16) - (val & 0xffff)) > 1; +} + +/* + * Remapping spinlock architecture specific functions to the corresponding + * ticket spinlock functions. + */ +#define arch_spin_is_locked(l) ticket_spin_is_locked(l) +#define arch_spin_is_contended(l) ticket_spin_is_contended(l) +#define arch_spin_value_unlocked(l) ticket_spin_value_unlocked(l) +#define arch_spin_lock(l) ticket_spin_lock(l) +#define arch_spin_trylock(l) ticket_spin_trylock(l) +#define arch_spin_unlock(l) ticket_spin_unlock(l) + +#endif /* __ASM_GENERIC_TICKET_SPINLOCK_H */ From patchwork Wed Jul 31 07:24:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13748243 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 85525C3DA7F for ; Wed, 31 Jul 2024 07:35:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=DpsANbD8STIbda4XVlC/8pV9ZRHiWcC64qhfTiIBR84=; b=qu/hAINFRjOx4k nGOXvlo3zqyET6rmOh76wzUr55ipBd5kShLenEuVPgBN2UIHzW4jmD8gCdMEcdPFImAXr02dP13bI bL3eRgyWUByPfG/nuhNFKYTKO5AZicmmwnrSjbCkT3/kg+o6z0pMWepGjrpqD6PUnZfzM6Wi9zvtt iLQfaTsSl89kirxGaM+G41qv4PqZdwtH/S7/9gkbY2NkAoqbNzz+uD0gX5IbrdzqbEk6iSKSTh1aF 0ZlhHFM37N63aD5CK65mLYNWh5gwlCrfgufdJKydK1sEgUURtxxq6s2gfpE53iSJfe/mGzHebBO0u TK59nzeFPXebt6iXmlig==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sZ3rr-000000009mL-2wAC; Wed, 31 Jul 2024 07:35:27 +0000 Received: from mail-wr1-x435.google.com ([2a00:1450:4864:20::435]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sZ3rn-000000009kU-34mD for linux-riscv@lists.infradead.org; Wed, 31 Jul 2024 07:35:25 +0000 Received: by mail-wr1-x435.google.com with SMTP id ffacd0b85a97d-3687ea0521cso3389141f8f.1 for ; Wed, 31 Jul 2024 00:35:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1722411322; x=1723016122; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=z44ALrb7LNXYv1B0ZHkbMo5mYcvmK7MIKHKlm+pxL7E=; b=Lhyfw+evq8W93mFfw1SQQ/TfQB+zfRNJktpMtlWVzsPXYuPEidY2JGUF1FQ+XXjimr NJ8BMDtgit8W2OhVn9DZLvxk90Mqi5DksF88fu5LiLfXc1lRA57hmIzaJZCq4YNC/rFv xhyAViTI/priabSRW2E/an14xLC2TWBjCSRZ/QFu9fDDWGxXXqWQY7sulyclKageOp8s wGamx+fpp5NrqaVH3M7LH6CfuLHhDUh1nZgP2NPZX5QLCUq6n5LyKll3t0Zk6HwTz5s6 TNyEuaCoGlTi5xBRonK23ZN+eho+v9kr35jquHOIFmwt+ghRoZNViQETL9Wase0BOQpI OBlQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722411322; x=1723016122; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=z44ALrb7LNXYv1B0ZHkbMo5mYcvmK7MIKHKlm+pxL7E=; b=f6snapjP8d/+1JULlS2f0p/nESD3uWd5P885td/gNKuH6ZHx9zZ2HaZTZbJI+ewc/R Gs9bp/tJhFYwtb4V+gYY4+8jBjUzWESz+xIPC8QZMuaW/mCkgTq7NH7+5T+FaRx0Iyyv uE52xSmaWubngZgCa2DUXAgJdQZzJIJvWDApTV+M4KVf7/ja3tjppjWCmugdgeW1GTLZ oqiUI5JDk850u4y1GpmPkOKC5KIg2yXoYZfkF3JzkXXXkjOpmEjH5Zjpo4GK2ky8OJRK TGIOZ7mPE/AwaunpWkoeHrH2Ue+nPIJQ6n40GoViPs1dxI+C7e+5MjhdVWDaz40u+E30 F5Uw== X-Forwarded-Encrypted: i=1; AJvYcCX+QxfgewT/f7CSz1PrQCINhVnKmoQe8MUcY16r0ixp8+h13uSlsSSSKZIF3GGJYiiGtubh0/LYfonxdPhUo4Mg1YtTkRFCsK22YUVC/REp X-Gm-Message-State: AOJu0YyKmbyp3NtPVtl3ERsfGIIMpqTzEU2NRWgJl8UKfUZOIVffqag1 L4bhfZ+xRFDCj+8ZGPzlg8Dkyabm9FEj4iwrb6oNKHzj+VUlDZzJWREHy6+lBwQ= X-Google-Smtp-Source: AGHT+IG6U6uPzWRr6gyEB9gsCw8ZKy/LcXxiWXsEaCB8YLZ20e09CxhjZV72Tw4GvYFGKWtqsWRHog== X-Received: by 2002:a5d:6d87:0:b0:368:31c7:19da with SMTP id ffacd0b85a97d-36b5cf2520fmr13078467f8f.13.1722411321847; Wed, 31 Jul 2024 00:35:21 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-36b36862271sm16333146f8f.98.2024.07.31.00.35.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jul 2024 00:35:21 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH v4 11/13] riscv: Add ISA extension parsing for Ziccrse Date: Wed, 31 Jul 2024 09:24:03 +0200 Message-Id: <20240731072405.197046-12-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240731072405.197046-1-alexghiti@rivosinc.com> References: <20240731072405.197046-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240731_003523_777997_4653093B X-CRM114-Status: GOOD ( 10.70 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Add support to parse the Ziccrse string in the riscv,isa string. Signed-off-by: Alexandre Ghiti --- arch/riscv/include/asm/hwcap.h | 1 + arch/riscv/kernel/cpufeature.c | 1 + 2 files changed, 2 insertions(+) diff --git a/arch/riscv/include/asm/hwcap.h b/arch/riscv/include/asm/hwcap.h index f5d53251c947..9e228b079a6d 100644 --- a/arch/riscv/include/asm/hwcap.h +++ b/arch/riscv/include/asm/hwcap.h @@ -93,6 +93,7 @@ #define RISCV_ISA_EXT_ZCMOP 84 #define RISCV_ISA_EXT_ZAWRS 85 #define RISCV_ISA_EXT_ZABHA 86 +#define RISCV_ISA_EXT_ZICCRSE 87 #define RISCV_ISA_EXT_XLINUXENVCFG 127 diff --git a/arch/riscv/kernel/cpufeature.c b/arch/riscv/kernel/cpufeature.c index 2a608056c89c..097821df8642 100644 --- a/arch/riscv/kernel/cpufeature.c +++ b/arch/riscv/kernel/cpufeature.c @@ -314,6 +314,7 @@ const struct riscv_isa_ext_data riscv_isa_ext[] = { riscv_ext_zicbom_validate), __RISCV_ISA_EXT_SUPERSET_VALIDATE(zicboz, RISCV_ISA_EXT_ZICBOZ, riscv_xlinuxenvcfg_exts, riscv_ext_zicboz_validate), + __RISCV_ISA_EXT_DATA(ziccrse, RISCV_ISA_EXT_ZICCRSE), __RISCV_ISA_EXT_DATA(zicntr, RISCV_ISA_EXT_ZICNTR), __RISCV_ISA_EXT_DATA(zicond, RISCV_ISA_EXT_ZICOND), __RISCV_ISA_EXT_DATA(zicsr, RISCV_ISA_EXT_ZICSR), From patchwork Wed Jul 31 07:24:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13748248 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4EDF2C3DA7F for ; Wed, 31 Jul 2024 07:36:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=qmYDAIItpjkWNFi0mvFyvyKA/NF8b4Q8clRj9rK/Pzo=; b=10+UR0NoVu940k 43QwxqsAuAcJOzhTh8U5dfhl+FPzfn7em01kZuSvdwY0GSDr+31ubSu9QgsKVe0MgCgibc91vfN8x Pp4am21Dzw6KKDE2WJs2DXeYE5+Ve5H0TiLIjNuhOwNE34i2RUM8LUqu4wh5Y+xRIC/iFCLbI0tHS +DhePSLi2XQnx4yFbzTsaGIN/gWgIZx30cQwWk0+xCmCsFHRPIF9Q8Pi7WtFYm+eftJ37IDcgtXSJ yp1xrLbFzSRxmfwsQ2HZa7gAu7k9n9rbrSpJmG2zOx1Wu+HJCUQ5WRwv03I4IJzvshVTiIgPGapZ5 x8l1Zz7IU5m03VYX8ucw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sZ3sp-00000000A1C-3Xcf; Wed, 31 Jul 2024 07:36:27 +0000 Received: from mail-wm1-x329.google.com ([2a00:1450:4864:20::329]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sZ3sm-000000009zQ-2YpC for linux-riscv@lists.infradead.org; Wed, 31 Jul 2024 07:36:25 +0000 Received: by mail-wm1-x329.google.com with SMTP id 5b1f17b1804b1-4280c493719so32200775e9.3 for ; Wed, 31 Jul 2024 00:36:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1722411383; x=1723016183; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=StkHQugPFKmcMlCdkJBjlWwaxWZu+axwFXxjxfJ46xI=; b=3CJROQEqrtjBblg4WWUdMDE7HdaDESR1X1FO1xjA64AGcOWPBrm9Lx62jfYP7oe+vi BVIBWfTNTIYucZE4HrltSN7WPqxKz9gPcyycfYc74GpYX/5tfvyVh2+v3i03kWhbPHI5 kI6+Dx3A6DVW3cyT2EG+YavmKy2r3kyv+2rG4VUg1NiEHdUsjjdx41o+yVTX0B6vNfbs 9VHrhZe/nz5dJIbbu42U4pwG4vvjh4hQHNIsQQdzz+Xsk9gn1ze5ta6HVT6CkvtLKWL8 U0rradqOSKWlpnNSeFnV8N0J7O6HxKefxqx9BVSLBau+V8NFaasDd3Hwj5F5Qg7DJeDp mpqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722411383; x=1723016183; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=StkHQugPFKmcMlCdkJBjlWwaxWZu+axwFXxjxfJ46xI=; b=HnwP3oHgYVF1Mu4Zx0LSWFbT5SS1TluPqLWa//sBuCE6rR2hB1Imni6FIjrrwPS+cR PdIHGf/82PqPr3IHF2zXmFideP/r89YP4k03IdqCFMsW1WSB3vJdieqkZdWAmZHqVH07 obqZlCCXtdbaR8JI8Hh19bnvoggn4yj4On0Y/uJKDw+6YBw9ZQvU9YHvFqLANTFcv32H LCxTJjie5hADUvmcqge3B3ekU/+cm9GO4BZX7JMgD2UxjCgxadoVdfJTyUkK6GDv2yhx V1ki+oRbIzmTVRrBQI65nhXNCddJaIXT/YN3e60NM2Qyt21pQZ3SCl7S2/hA69ZfFWhC wEsA== X-Forwarded-Encrypted: i=1; AJvYcCUqeBacif1ZGl/TkQ+PdW0ZeBFtADq0Dq8a5qlMSqfsYRfezaqjWB4aLIRnzx6yM8f2dFEq5AWI3sfQF3rbX6RKpF3+/zIBv0l8vaZIcNWj X-Gm-Message-State: AOJu0YxySwAy7vTfCml0yAp3+LT4HHWiUu9NYWJCNviTyjCyenc74fYP 6FeCtLwO552zhYJS+nir2h0kUhID1MzV2D73UpX76y+T7N2VedZpW08Kc+/Eeec= X-Google-Smtp-Source: AGHT+IEiU0CaFWiD1hh70+ka7aQVmGlrhYH1/rnpyVPMCyVRk32YX0EEddiK4g7ZX2FGkW4AWaXtCw== X-Received: by 2002:a05:600c:45cb:b0:428:1007:6068 with SMTP id 5b1f17b1804b1-42811e0b70amr100189195e9.34.1722411383067; Wed, 31 Jul 2024 00:36:23 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-36b367fce64sm16325946f8f.61.2024.07.31.00.36.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jul 2024 00:36:22 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH v4 12/13] dt-bindings: riscv: Add Ziccrse ISA extension description Date: Wed, 31 Jul 2024 09:24:04 +0200 Message-Id: <20240731072405.197046-13-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240731072405.197046-1-alexghiti@rivosinc.com> References: <20240731072405.197046-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240731_003624_674495_653B63F0 X-CRM114-Status: UNSURE ( 9.48 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Add description for the Ziccrse ISA extension which was introduced in the riscv profiles specification v0.9.2. Signed-off-by: Alexandre Ghiti Reviewed-by: Guo Ren --- Documentation/devicetree/bindings/riscv/extensions.yaml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/Documentation/devicetree/bindings/riscv/extensions.yaml b/Documentation/devicetree/bindings/riscv/extensions.yaml index a63578b95c4a..22824dd30175 100644 --- a/Documentation/devicetree/bindings/riscv/extensions.yaml +++ b/Documentation/devicetree/bindings/riscv/extensions.yaml @@ -289,6 +289,12 @@ properties: in commit 64074bc ("Update version numbers for Zfh/Zfinx") of riscv-isa-manual. + - const: ziccrse + description: + The standard Ziccrse extension which provides forward progress + guarantee on LR/SC sequences, as introduced in the riscv profiles + specification v0.9.2. + - const: zk description: The standard Zk Standard Scalar cryptography extension as ratified From patchwork Wed Jul 31 07:24:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13748249 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2DDCFC3DA7F for ; Wed, 31 Jul 2024 07:37:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=kxezsi/N2Mu6EEYwXVnwJs5gyYsBFEZL5Xh0tNFEIkE=; b=U7HcHXvSKyXb3M VRicLjc8rQZwShzeW0LTs6diGyowROy/Z7Bm1aXWqbopeT4c/oBr5R2CeMn2rs88lvPrwCtNed4nC dRtuKgI8Wr9wPsPZGjOToE1nra8d8XfKkVyo3WEXYu+34M51BMAve28tcvBiKrWPU99ANSB9HcGxS fPwHwzl5FSPuTvqAAv6T1XRHOwICmizACdTh+TSwQizZDixzto1gCFyxB33I5x45s0cJIO2IIlBa0 FlGHWp6wr1N+M8EeBh1mrvq6lFkbs5fh6jRwV4s0jxOLL7z3Y/YXhBKOkZMWNl+k/OCesdFUIzgZS ofIxTa/yod3/gcL1eCqQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sZ3tp-00000000AKf-2cJ8; Wed, 31 Jul 2024 07:37:29 +0000 Received: from mail-wm1-x32f.google.com ([2a00:1450:4864:20::32f]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sZ3tl-00000000AJ0-3tZv for linux-riscv@lists.infradead.org; Wed, 31 Jul 2024 07:37:27 +0000 Received: by mail-wm1-x32f.google.com with SMTP id 5b1f17b1804b1-42816ca782dso33082985e9.2 for ; Wed, 31 Jul 2024 00:37:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1722411444; x=1723016244; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=IuzTs5i1Aydmrp9OuDhDQYyZMjhUNqn9zGoqbFH2HN0=; b=aOQxHBrKEH3ptEMrOcnq/q6RrMUMjSjcMwO1EnYimEB7VnJLDV9fWoxQUZ5ABX0P9b 0Gakpoiqzv4Kqu27Ts2gb26b2lZHj4KGusWr+lekZ0VxfC/Sp94Ogk3wBoxNwLQH/Ykn qiR0Nh6D+Jh/ZYtKbB+eRymIYFwu5DCEhw3p9vWtZqx0U8AM58L4esNlki6Mf32gHf02 MplH7bNOgpGpK3xSDt18pO+4XOIvp/DtRcHsxnmH/xzZYmxW4uRpzj+THrUmwoztmCTE CMIIjU6InN0PcFhGKvwfoEQsTDChb+pGbvRN472wSCaF9VNyU6BE8F2+ET1rAPqTd6I4 oidw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722411444; x=1723016244; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=IuzTs5i1Aydmrp9OuDhDQYyZMjhUNqn9zGoqbFH2HN0=; b=weG3RFZZMmA0MlHER3dt8vtWA/EpO2c1Bky5AIDws623KZ7Q/MjMhf8jdVrdj5niF9 pRDc6oAKX4nF0t7KaQivyMsoVXaocf5LyZV4p0yItvgZK1yYz3KawrHtNMwzlodjWIHn EkasAk/J3Zn9D1CAbRII2IAZK/W0++Hz7QjbRcw8X/Zm1QSyGpxlXzDjwopfQaH6QcRQ UU30wX0R8TuhYG4M2+TZDjJ/nN8HKHlmhvSU1oR5W5uHpBnEhYFoAOS++aPIK+rwHp3W ndkDJQ8yhgnQso9Qd5ry+dm/UC+Zbqz2fd+WVPCLuc17PnfZW/XqqV6YerNs9863Kq3k dDHg== X-Forwarded-Encrypted: i=1; AJvYcCUpWzr+cPJLKs18n5HbH5Ui2n+x1xsf2h6bh6qL1Jvuk68rAV2Cm5FNcxYl+AgPLYbtd3LbSnZ1QTaxts/Y0tTEKoG1BK3W+Ke1Uj+R8tJX X-Gm-Message-State: AOJu0YzDm+KIGuueCFzOMdD5Vzrry97QrrTnOrGB7tX577Ir9MOpq+9V wLNd6FSIlHhjvYC+uGK/T6bvxuYiDAiXIUCnnXFEjaGNMjsA1c020sFdruJeLw8= X-Google-Smtp-Source: AGHT+IFmxRjofwufvv+5ZesOhaxyTIL2IhSq0e317HP/ein1RfvIfnjlhNGswng/psJn1bIEYGSz1g== X-Received: by 2002:a05:600c:468d:b0:426:6153:5318 with SMTP id 5b1f17b1804b1-42811d9e0e7mr87431975e9.19.1722411444271; Wed, 31 Jul 2024 00:37:24 -0700 (PDT) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-36b367e51f0sm16343005f8f.46.2024.07.31.00.37.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jul 2024 00:37:24 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH v4 13/13] riscv: Add qspinlock support Date: Wed, 31 Jul 2024 09:24:05 +0200 Message-Id: <20240731072405.197046-14-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240731072405.197046-1-alexghiti@rivosinc.com> References: <20240731072405.197046-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240731_003725_997257_01C906D4 X-CRM114-Status: GOOD ( 25.21 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org In order to produce a generic kernel, a user can select CONFIG_COMBO_SPINLOCKS which will fallback at runtime to the ticket spinlock implementation if Zabha or Ziccrse are not present. Note that we can't use alternatives here because the discovery of extensions is done too late and we need to start with the qspinlock implementation because the ticket spinlock implementation would pollute the spinlock value, so let's use static keys. This is largely based on Guo's work and Leonardo reviews at [1]. Link: https://lore.kernel.org/linux-riscv/20231225125847.2778638-1-guoren@kernel.org/ [1] Signed-off-by: Guo Ren Signed-off-by: Alexandre Ghiti --- .../locking/queued-spinlocks/arch-support.txt | 2 +- arch/riscv/Kconfig | 29 +++++++++++++ arch/riscv/include/asm/Kbuild | 4 +- arch/riscv/include/asm/spinlock.h | 43 +++++++++++++++++++ arch/riscv/kernel/setup.c | 38 ++++++++++++++++ include/asm-generic/qspinlock.h | 2 + include/asm-generic/ticket_spinlock.h | 2 + 7 files changed, 118 insertions(+), 2 deletions(-) create mode 100644 arch/riscv/include/asm/spinlock.h diff --git a/Documentation/features/locking/queued-spinlocks/arch-support.txt b/Documentation/features/locking/queued-spinlocks/arch-support.txt index 22f2990392ff..cf26042480e2 100644 --- a/Documentation/features/locking/queued-spinlocks/arch-support.txt +++ b/Documentation/features/locking/queued-spinlocks/arch-support.txt @@ -20,7 +20,7 @@ | openrisc: | ok | | parisc: | TODO | | powerpc: | ok | - | riscv: | TODO | + | riscv: | ok | | s390: | TODO | | sh: | TODO | | sparc: | ok | diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index ef55ab94027e..c9ff8081efc1 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -79,6 +79,7 @@ config RISCV select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP select ARCH_WANTS_NO_INSTR select ARCH_WANTS_THP_SWAP if HAVE_ARCH_TRANSPARENT_HUGEPAGE + select ARCH_WEAK_RELEASE_ACQUIRE if ARCH_USE_QUEUED_SPINLOCKS select BINFMT_FLAT_NO_DATA_START_OFFSET if !MMU select BUILDTIME_TABLE_SORT if MMU select CLINT_TIMER if RISCV_M_MODE @@ -488,6 +489,34 @@ config NODES_SHIFT Specify the maximum number of NUMA Nodes available on the target system. Increases memory reserved to accommodate various tables. +choice + prompt "RISC-V spinlock type" + default RISCV_COMBO_SPINLOCKS + +config RISCV_TICKET_SPINLOCKS + bool "Using ticket spinlock" + +config RISCV_QUEUED_SPINLOCKS + bool "Using queued spinlock" + depends on SMP && MMU && NONPORTABLE + select ARCH_USE_QUEUED_SPINLOCKS + help + The queued spinlock implementation requires the forward progress + guarantee of cmpxchg()/xchg() atomic operations: CAS with Zabha or + LR/SC with Ziccrse provide such guarantee. + + Select this if and only if Zabha or Ziccrse is available on your + platform. + +config RISCV_COMBO_SPINLOCKS + bool "Using combo spinlock" + depends on SMP && MMU + select ARCH_USE_QUEUED_SPINLOCKS + help + Embed both queued spinlock and ticket lock so that the spinlock + implementation can be chosen at runtime. +endchoice + config RISCV_ALTERNATIVE bool depends on !XIP_KERNEL diff --git a/arch/riscv/include/asm/Kbuild b/arch/riscv/include/asm/Kbuild index 5c589770f2a8..1c2618c964f0 100644 --- a/arch/riscv/include/asm/Kbuild +++ b/arch/riscv/include/asm/Kbuild @@ -5,10 +5,12 @@ syscall-y += syscall_table_64.h generic-y += early_ioremap.h generic-y += flat.h generic-y += kvm_para.h +generic-y += mcs_spinlock.h generic-y += parport.h -generic-y += spinlock.h generic-y += spinlock_types.h +generic-y += ticket_spinlock.h generic-y += qrwlock.h generic-y += qrwlock_types.h +generic-y += qspinlock.h generic-y += user.h generic-y += vmlinux.lds.h diff --git a/arch/riscv/include/asm/spinlock.h b/arch/riscv/include/asm/spinlock.h new file mode 100644 index 000000000000..503aef31db83 --- /dev/null +++ b/arch/riscv/include/asm/spinlock.h @@ -0,0 +1,43 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef __ASM_RISCV_SPINLOCK_H +#define __ASM_RISCV_SPINLOCK_H + +#ifdef CONFIG_RISCV_COMBO_SPINLOCKS +#define _Q_PENDING_LOOPS (1 << 9) + +#define __no_arch_spinlock_redefine +#include +#include +#include + +DECLARE_STATIC_KEY_TRUE(qspinlock_key); + +#define SPINLOCK_BASE_DECLARE(op, type, type_lock) \ +static __always_inline type arch_spin_##op(type_lock lock) \ +{ \ + if (static_branch_unlikely(&qspinlock_key)) \ + return queued_spin_##op(lock); \ + return ticket_spin_##op(lock); \ +} + +SPINLOCK_BASE_DECLARE(lock, void, arch_spinlock_t *) +SPINLOCK_BASE_DECLARE(unlock, void, arch_spinlock_t *) +SPINLOCK_BASE_DECLARE(is_locked, int, arch_spinlock_t *) +SPINLOCK_BASE_DECLARE(is_contended, int, arch_spinlock_t *) +SPINLOCK_BASE_DECLARE(trylock, bool, arch_spinlock_t *) +SPINLOCK_BASE_DECLARE(value_unlocked, int, arch_spinlock_t) + +#elif defined(CONFIG_RISCV_QUEUED_SPINLOCKS) + +#include + +#else + +#include + +#endif + +#include + +#endif /* __ASM_RISCV_SPINLOCK_H */ diff --git a/arch/riscv/kernel/setup.c b/arch/riscv/kernel/setup.c index a2cde65b69e9..b811fa331982 100644 --- a/arch/riscv/kernel/setup.c +++ b/arch/riscv/kernel/setup.c @@ -244,6 +244,43 @@ static void __init parse_dtb(void) #endif } +#if defined(CONFIG_RISCV_COMBO_SPINLOCKS) +DEFINE_STATIC_KEY_TRUE(qspinlock_key); +EXPORT_SYMBOL(qspinlock_key); +#endif + +static void __init riscv_spinlock_init(void) +{ + char *using_ext = NULL; + + if (IS_ENABLED(CONFIG_RISCV_TICKET_SPINLOCKS)) { + pr_info("Ticket spinlock: enabled\n"); + return; + } + + if (IS_ENABLED(CONFIG_RISCV_ISA_ZABHA) && + IS_ENABLED(CONFIG_RISCV_ISA_ZACAS) && + riscv_isa_extension_available(NULL, ZABHA) && + riscv_isa_extension_available(NULL, ZACAS)) { + using_ext = "using Zabha"; + } else if (riscv_isa_extension_available(NULL, ZICCRSE)) { + using_ext = "using Ziccrse"; + } +#if defined(CONFIG_RISCV_COMBO_SPINLOCKS) + else { + static_branch_disable(&qspinlock_key); + pr_info("Ticket spinlock: enabled\n"); + + return; + } +#endif + + if (!using_ext) + pr_err("Queued spinlock without Zabha or Ziccrse"); + else + pr_info("Queued spinlock %s: enabled\n", using_ext); +} + extern void __init init_rt_signal_env(void); void __init setup_arch(char **cmdline_p) @@ -297,6 +334,7 @@ void __init setup_arch(char **cmdline_p) riscv_set_dma_cache_alignment(); riscv_user_isa_enable(); + riscv_spinlock_init(); } bool arch_cpu_is_hotpluggable(int cpu) diff --git a/include/asm-generic/qspinlock.h b/include/asm-generic/qspinlock.h index 0655aa5b57b2..bf47cca2c375 100644 --- a/include/asm-generic/qspinlock.h +++ b/include/asm-generic/qspinlock.h @@ -136,6 +136,7 @@ static __always_inline bool virt_spin_lock(struct qspinlock *lock) } #endif +#ifndef __no_arch_spinlock_redefine /* * Remapping spinlock architecture specific functions to the corresponding * queued spinlock functions. @@ -146,5 +147,6 @@ static __always_inline bool virt_spin_lock(struct qspinlock *lock) #define arch_spin_lock(l) queued_spin_lock(l) #define arch_spin_trylock(l) queued_spin_trylock(l) #define arch_spin_unlock(l) queued_spin_unlock(l) +#endif #endif /* __ASM_GENERIC_QSPINLOCK_H */ diff --git a/include/asm-generic/ticket_spinlock.h b/include/asm-generic/ticket_spinlock.h index cfcff22b37b3..325779970d8a 100644 --- a/include/asm-generic/ticket_spinlock.h +++ b/include/asm-generic/ticket_spinlock.h @@ -89,6 +89,7 @@ static __always_inline int ticket_spin_is_contended(arch_spinlock_t *lock) return (s16)((val >> 16) - (val & 0xffff)) > 1; } +#ifndef __no_arch_spinlock_redefine /* * Remapping spinlock architecture specific functions to the corresponding * ticket spinlock functions. @@ -99,5 +100,6 @@ static __always_inline int ticket_spin_is_contended(arch_spinlock_t *lock) #define arch_spin_lock(l) ticket_spin_lock(l) #define arch_spin_trylock(l) ticket_spin_trylock(l) #define arch_spin_unlock(l) ticket_spin_unlock(l) +#endif #endif /* __ASM_GENERIC_TICKET_SPINLOCK_H */