From patchwork Thu Mar 7 19:05:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Charlie Jenkins X-Patchwork-Id: 13586224 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A474DC54798 for ; Thu, 7 Mar 2024 20:06:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:In-Reply-To:References:Message-Id :MIME-Version:Subject:Date:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=w70xITaCma6OVdx6baqnQZBFD6T+MhCXDBEIGWSLu/c=; b=kg3DBblUpRE1Jr LpSRckXycw3Ctovz4JaJsRTPCEcY2SXLx2uku1rpKv4DXq6kovOIbyZ9Y9JLRl0HMDl/xd1vQmwg7 YRFiwpzfIoXqGAfIisTaATVkgeFVyqaoiom5enoCtdgv5+TX1/FL5pyPuLiXJ82R8lzHJSLJzAVoo aJiRXuqnuee2Pv8+zzQlfGvb2Cw7DQb4mI47YKUrj9YkHlV8XY5fCQ/imCxstOZ4dB0L5En/7SB7j KAbGfTWvk1afJCSWzqHt/KwOs4fzxAqdcizZfmmnrQvXPb+D53H3IX7aMl++DLbR0VsaEkPaGA+2X USoD6zOVcWO8pR/ljHlA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1riK0r-00000006Cxj-1ZmR; Thu, 07 Mar 2024 20:06:45 +0000 Received: from mail-pl1-x629.google.com ([2607:f8b0:4864:20::629]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1riJ3z-00000005yPy-3GFf for linux-riscv@lists.infradead.org; Thu, 07 Mar 2024 19:05:59 +0000 Received: by mail-pl1-x629.google.com with SMTP id d9443c01a7336-1dd611d5645so4486995ad.1 for ; Thu, 07 Mar 2024 11:05:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1709838354; x=1710443154; darn=lists.infradead.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=akMpdDedBfqF60/fNsk+O8npKk5JqF+ySZ0f9xW5/1M=; b=pFYq1MQ0RW+gjAE2WI59qoVnYMV9Y2mfb+3zZaY3I6sA1Hjyn0at273IDsgqhRyzAB qtmp3RZ2CBFCYGl2NopQnCC2QaHrz9+fu0VsgQJtKI0Gy5TXW3hKF8hGZqSgA23qK1Cb ZBzAi1KdPT4OSgl/mLypvfbBRBvj5rlx78m8JYgLaLSDAlrHp8jV00jf6dG2mzVRfdGs PvkDHt55kzwsTggtN41l17uvYcRHaNZGJyz/1kHG/SrbAnLuPm89HkNwpDShbarenVB/ LCOAwbaluIIcx38/ll5K16JCP4S7SRGOL/abf4P4zbhIDDiEppEzsYA+O8gBnj9BBh+y IWfA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709838354; x=1710443154; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=akMpdDedBfqF60/fNsk+O8npKk5JqF+ySZ0f9xW5/1M=; b=JaGQlAaevAVM/zzt2d7d9VKMe8QaGsMxjU0jdTZ2kMERUvTUKeu+NeSrG5d4wrfzTc fChi5Tj9B0O92kYlR3LbGAofZGxp+yxp0/7AQP9EaHQZ0RQzzzvocBauUGY1nAwQ6rZm aD7LnQs2NfHMd3qwjg3JO/83NzJ9d4D9Q3gzZ8JFmfyy5sNEz9KU1o9lp3pnefxgp0uB u6PRsbAh7vi8qPdtZU4Bj5/uPpl3qIj2P0dbZQAlNvGkhv+FrJKtGoruZB2bDwvE2SF4 Y1767pdQFJEev57kVMlvIxSNdTFcuDov7Q8eB7xyRthzRCCh2FO3Fkfppaw6fn+eR87U 0Vvw== X-Gm-Message-State: AOJu0YxlFoZ3WNcAgAahPaX90qrzDBnL8qFpn0+AS6uO4fhbx/6vW8hn S6vNNXMN48PRbq4HqMb4n2T6eO4gig5FEI9UPO3izSYuQYSSyZkpOQbCFPrXL2g= X-Google-Smtp-Source: AGHT+IGIkFO9XVaeP/1tKPmqCILvLsJZGaZz9P1TWEAdnDzXM5s2KMdvUPij6lw/K5WrTNkXwn1yVg== X-Received: by 2002:a17:902:fa90:b0:1dc:c161:bce6 with SMTP id lc16-20020a170902fa9000b001dcc161bce6mr2666593plb.15.1709838354594; Thu, 07 Mar 2024 11:05:54 -0800 (PST) Received: from charlie.ba.rivosinc.com ([64.71.180.162]) by smtp.gmail.com with ESMTPSA id h3-20020a170902680300b001dd526af36csm1747338plk.295.2024.03.07.11.05.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Mar 2024 11:05:53 -0800 (PST) From: Charlie Jenkins Date: Thu, 07 Mar 2024 11:05:45 -0800 Subject: [PATCH v8 1/4] riscv: lib: Introduce has_fast_unaligned_access() MIME-Version: 1.0 Message-Id: <20240307-disable_misaligned_probe_config-v8-1-55d696cb398b@rivosinc.com> References: <20240307-disable_misaligned_probe_config-v8-0-55d696cb398b@rivosinc.com> In-Reply-To: <20240307-disable_misaligned_probe_config-v8-0-55d696cb398b@rivosinc.com> To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Jisheng Zhang , Evan Green , =?utf-8?b?Q2zDqW1lbnQgTMOpZ2Vy?= , Eric Biggers , Elliot Berman , Charles Lohr , Conor Dooley Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, Charlie Jenkins X-Mailer: b4 0.12.3 X-Developer-Signature: v=1; a=ed25519-sha256; t=1709838351; l=3428; i=charlie@rivosinc.com; s=20231120; h=from:subject:message-id; bh=ejS3FVJ1L+bLL4KR96q6ZNDMwke6NlmAthP91enITHI=; b=QSgWXU48WzSbKsbVl21l4yUMXYbuR79AfVXchsjcwshH/vOtpegFqmabEpCqdScNEvHiGhn1R dUTNhTVGbssD6hYxGOAUgt/afIDbNbpK0lhR5laE9KWUauYvQHlnj1x X-Developer-Key: i=charlie@rivosinc.com; a=ed25519; pk=t4RSWpMV1q5lf/NWIeR9z58bcje60/dbtxxmoSfBEcs= X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240307_110555_910748_44C1AE04 X-CRM114-Status: GOOD ( 11.02 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Create has_fast_unaligned_access to avoid needing to explicitly check the fast_misaligned_access_speed_key static key. Signed-off-by: Charlie Jenkins Reviewed-by: Evan Green Reviewed-by: Conor Dooley --- arch/riscv/include/asm/cpufeature.h | 11 ++++++++--- arch/riscv/kernel/cpufeature.c | 6 +++--- arch/riscv/lib/csum.c | 7 ++----- 3 files changed, 13 insertions(+), 11 deletions(-) diff --git a/arch/riscv/include/asm/cpufeature.h b/arch/riscv/include/asm/cpufeature.h index 5a626ed2c47a..466e1f591919 100644 --- a/arch/riscv/include/asm/cpufeature.h +++ b/arch/riscv/include/asm/cpufeature.h @@ -1,6 +1,6 @@ /* SPDX-License-Identifier: GPL-2.0-only */ /* - * Copyright 2022-2023 Rivos, Inc + * Copyright 2022-2024 Rivos, Inc */ #ifndef _ASM_CPUFEATURE_H @@ -53,6 +53,13 @@ static inline bool check_unaligned_access_emulated(int cpu) static inline void unaligned_emulation_finish(void) {} #endif +DECLARE_STATIC_KEY_FALSE(fast_unaligned_access_speed_key); + +static __always_inline bool has_fast_unaligned_accesses(void) +{ + return static_branch_likely(&fast_unaligned_access_speed_key); +} + unsigned long riscv_get_elf_hwcap(void); struct riscv_isa_ext_data { @@ -135,6 +142,4 @@ static __always_inline bool riscv_cpu_has_extension_unlikely(int cpu, const unsi return __riscv_isa_extension_available(hart_isa[cpu].isa, ext); } -DECLARE_STATIC_KEY_FALSE(fast_misaligned_access_speed_key); - #endif diff --git a/arch/riscv/kernel/cpufeature.c b/arch/riscv/kernel/cpufeature.c index 89920f84d0a3..7878cddccc0d 100644 --- a/arch/riscv/kernel/cpufeature.c +++ b/arch/riscv/kernel/cpufeature.c @@ -810,14 +810,14 @@ static void check_unaligned_access_nonboot_cpu(void *param) check_unaligned_access(pages[cpu]); } -DEFINE_STATIC_KEY_FALSE(fast_misaligned_access_speed_key); +DEFINE_STATIC_KEY_FALSE(fast_unaligned_access_speed_key); static void modify_unaligned_access_branches(cpumask_t *mask, int weight) { if (cpumask_weight(mask) == weight) - static_branch_enable_cpuslocked(&fast_misaligned_access_speed_key); + static_branch_enable_cpuslocked(&fast_unaligned_access_speed_key); else - static_branch_disable_cpuslocked(&fast_misaligned_access_speed_key); + static_branch_disable_cpuslocked(&fast_unaligned_access_speed_key); } static void set_unaligned_access_static_branches_except_cpu(int cpu) diff --git a/arch/riscv/lib/csum.c b/arch/riscv/lib/csum.c index af3df5274ccb..7178e0acfa22 100644 --- a/arch/riscv/lib/csum.c +++ b/arch/riscv/lib/csum.c @@ -3,7 +3,7 @@ * Checksum library * * Influenced by arch/arm64/lib/csum.c - * Copyright (C) 2023 Rivos Inc. + * Copyright (C) 2023-2024 Rivos Inc. */ #include #include @@ -318,10 +318,7 @@ unsigned int do_csum(const unsigned char *buff, int len) * branches. The largest chunk of overlap was delegated into the * do_csum_common function. */ - if (static_branch_likely(&fast_misaligned_access_speed_key)) - return do_csum_no_alignment(buff, len); - - if (((unsigned long)buff & OFFSET_MASK) == 0) + if (has_fast_unaligned_accesses() || (((unsigned long)buff & OFFSET_MASK) == 0)) return do_csum_no_alignment(buff, len); return do_csum_with_alignment(buff, len); From patchwork Thu Mar 7 19:05:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Charlie Jenkins X-Patchwork-Id: 13586161 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E1F59C54E5E for ; Thu, 7 Mar 2024 19:06:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:In-Reply-To:References:Message-Id :MIME-Version:Subject:Date:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=JYRWCLe27HG9aNimRIlulGh19swqovIzRlEsSD8T7n4=; b=oA65Vvm0Zd1KbQ 1m8at0g/anv2FMGe63bzO2aeFzZ8qWv/iyf3cvhpKGkR95HOFSY36Dok1xdyAigrCsgfasmPe2umQ oWZnut82O1em0bDpwKCf8duDj/asgkT9EKsXlp5FOMjgOZJKtO16VUrHg77PqYvAWocoHktWel5zl j3baAEpdzR+S1TTjIqCu+pEWVFhL+774vSOvwN3NXXDNynAhxyf9k9HW5XV5PnmCa417M1U7EN4LE 4gtZanxL1u50stpCOwWT4ySFc9GbTSvaaYkK1nK/EVX3TcSYU+icy1p1fSAeZm9R6ten3TaEuXYh9 XvWXUse1Ec4074UdpMJw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1riJ4D-00000005ya2-4B4m; Thu, 07 Mar 2024 19:06:10 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1riJ45-00000005yUc-3sa7 for linux-riscv@bombadil.infradead.org; Thu, 07 Mar 2024 19:06:01 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Cc:To:In-Reply-To:References: Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Sender:Reply-To:Content-ID:Content-Description; bh=h8i7Hn0NjGdBjABWwTPM97LUVWqQ4V1pWIxyh4e32z8=; b=eBhH8eVt2QdSlCM4xTbcb7/okl vuVZj+RFu6peK13s0gK16SVUqbIvBKlnGPluxYKyjKhS4E2+Q56/hDLN5PrL+DdPbRx6vmQ8sKebx 3fkYZPMaMi3gWl0e/pDT3b9i5/PFptxgppk5aBgdhFv66dB+yvKL/4D5SUR+ESmscb34SDpMR25SG 2pmkKygrMyCgfCTvCe/slFJ03fddftix1eoRRBvfYjZO1C5IGlM1siGd1lwwj8cVmCCqHrXkoisXc NFiyICcLAAMmAItln/nqbB97Yp0QLlNaWMo258QTUjNLnZZ0ruan7EE4/pidzBJWDCeZw4n0BH/eP SconW3Wg==; Received: from mail-pl1-x62c.google.com ([2607:f8b0:4864:20::62c]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1riJ42-00000006rT2-2sUL for linux-riscv@lists.infradead.org; Thu, 07 Mar 2024 19:06:00 +0000 Received: by mail-pl1-x62c.google.com with SMTP id d9443c01a7336-1dd6412da28so2281885ad.3 for ; Thu, 07 Mar 2024 11:05:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1709838356; x=1710443156; darn=lists.infradead.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=h8i7Hn0NjGdBjABWwTPM97LUVWqQ4V1pWIxyh4e32z8=; b=CZ14umrb/744OqwmZaIGDNL7w0SkLMTpkl+H+xkGuS6xrtC2zMBEmOtqmqdhcmIrQM Ahd7hYgPD8eKwLCbkuePmDhZgKSP3QZMoM/fEF2inyrA4+HfJN8HIVPGWZMGhcC0nFz3 mNwi63QGPgsOTHseUnqXQ7wZJLevWs0KqOc2GHC1ULycymh0zvI5FRI3jewwQKUJR7yl 7dloKvmACL7Tk2EZTe3MuOS4UwtJKuGHX6hj3TP1D1XoZOK8Zg/G4XXV6/daVsiA95AV qbHCqk7LGfxvJHMUN4KKa3CdXPTDfxRnp16v63gFuV4NOaPhzkaY9bSVrnHyv2hOpbYz j3Kg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709838356; x=1710443156; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=h8i7Hn0NjGdBjABWwTPM97LUVWqQ4V1pWIxyh4e32z8=; b=VeL8SwJLBpT0EsoBNQlgPSU22/ZMKaX2SMpd9L/6NauO5NmW8E/ogxVZmsfHlLRPBm U4baGoffLLohhs4166Sf+1RVEbW4dZyJ7lS/EL+2vd9iU/hWQFazpsabIyL5ZTn+d3vK bdILXW/DXyXrLgIhlIwVGJIq/PhmwdF4MSgyqwXhmSsne2u/vC3eues07qCpeLitttEj hBxqh8Jo9tvpEmqyUaAC8z9PeGlP6tCsO/gOL9uQfs9eRuuFP6qUiAeUDBpuncLq2l4w GuUZfDVpJpwuwQzlCNpju1T5e2r0qylrmNuFullm390Z/9x60DZH17paAKrBzUDc3Tbz v0cg== X-Gm-Message-State: AOJu0YyLQHFlLuPwY35RM//D5ftu7mFCQreiorOeKsQ3aLftmWh5Q3WX bdmRCujDpWUuplQHrAjzsBtw8cLPO+0QUPqxZveKUPEw2XysdvkJspjxE4f4URM= X-Google-Smtp-Source: AGHT+IGWG50eIYNqJYtIEtMgtF56ZtFfU2/+liaC87qJQqHAa4lId+qqp/lStD4LT51B0M3EfKpmCg== X-Received: by 2002:a17:902:db0a:b0:1dd:a34:7321 with SMTP id m10-20020a170902db0a00b001dd0a347321mr10498088plx.25.1709838355931; Thu, 07 Mar 2024 11:05:55 -0800 (PST) Received: from charlie.ba.rivosinc.com ([64.71.180.162]) by smtp.gmail.com with ESMTPSA id h3-20020a170902680300b001dd526af36csm1747338plk.295.2024.03.07.11.05.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Mar 2024 11:05:55 -0800 (PST) From: Charlie Jenkins Date: Thu, 07 Mar 2024 11:05:46 -0800 Subject: [PATCH v8 2/4] riscv: Only check online cpus for emulated accesses MIME-Version: 1.0 Message-Id: <20240307-disable_misaligned_probe_config-v8-2-55d696cb398b@rivosinc.com> References: <20240307-disable_misaligned_probe_config-v8-0-55d696cb398b@rivosinc.com> In-Reply-To: <20240307-disable_misaligned_probe_config-v8-0-55d696cb398b@rivosinc.com> To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Jisheng Zhang , Evan Green , =?utf-8?b?Q2zDqW1lbnQgTMOpZ2Vy?= , Eric Biggers , Elliot Berman , Charles Lohr , Conor Dooley Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, Charlie Jenkins X-Mailer: b4 0.12.3 X-Developer-Signature: v=1; a=ed25519-sha256; t=1709838351; l=985; i=charlie@rivosinc.com; s=20231120; h=from:subject:message-id; bh=VSZiKx13zcdCVDYwqIRzCcJDp79icXmI6mc1wjrw+wY=; b=4lDes0vZrJms/4gMpoYeVGqXErBW/PLjmFFmXEprN4sEQrjl/O2TnHAvCflhkGywOBjrDhXZx brGOqXY3mv4DSqV1Bxg2PRf3Aizh4mrVFcQLGzt1urygWj2UkFOGR/j X-Developer-Key: i=charlie@rivosinc.com; a=ed25519; pk=t4RSWpMV1q5lf/NWIeR9z58bcje60/dbtxxmoSfBEcs= X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240307_190558_961853_922C4490 X-CRM114-Status: UNSURE ( 9.70 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org The unaligned access checker only sets valid values for online cpus. Check for these values on online cpus rather than on present cpus. Signed-off-by: Charlie Jenkins Reviewed-by: Conor Dooley Fixes: 71c54b3d169d ("riscv: report misaligned accesses emulation to hwprobe") --- arch/riscv/kernel/traps_misaligned.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/riscv/kernel/traps_misaligned.c b/arch/riscv/kernel/traps_misaligned.c index 8ded225e8c5b..c2ed4e689bf9 100644 --- a/arch/riscv/kernel/traps_misaligned.c +++ b/arch/riscv/kernel/traps_misaligned.c @@ -632,7 +632,7 @@ void unaligned_emulation_finish(void) * accesses emulated since tasks requesting such control can run on any * CPU. */ - for_each_present_cpu(cpu) { + for_each_online_cpu(cpu) { if (per_cpu(misaligned_access_speed, cpu) != RISCV_HWPROBE_MISALIGNED_EMULATED) { return; From patchwork Thu Mar 7 19:05:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Charlie Jenkins X-Patchwork-Id: 13586159 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 290A4C54798 for ; Thu, 7 Mar 2024 19:06:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:In-Reply-To:References:Message-Id :MIME-Version:Subject:Date:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=65B1dK/wLVNe3rc/un3fl+yafVflmbq4P3/ZxMZmIzU=; b=r9A+vflxSdyjMd BEVnpWNxZrV/sPghiLQvGDv6lchJ2ET0tpeC99gK6ciSgxEelOLii6Z3wGUqRLYlahRUo7wjnaoG4 OKsQvk3v8NKBYgcgvj6mXsOqREwoI3dKcoZEwdOEH4YtDrsKctCxgSTAGxDqFOHA3+8gQPJ/0YWjN TnLzkffl9zGzIfrurKyw3YhTeKxMaT4SbgFkCgXCLjVAULBwmJF1Nu5Cy1v9CcOwBRTFY46gfPsgh lv2W7zp2pc5ZBd9iUMNQDQLPblAH+/VmLT4DwjipDCi2doZ8Y3DvoNAzI7Bpwda/Qc+aLr0HDW3Vl ffLGljULf2kcB2ahUjqA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1riJ4E-00000005yaF-3SSy; Thu, 07 Mar 2024 19:06:10 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1riJ46-00000005yUv-19AB for linux-riscv@bombadil.infradead.org; Thu, 07 Mar 2024 19:06:02 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Cc:To:In-Reply-To:References: Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Sender:Reply-To:Content-ID:Content-Description; bh=rcFiymo+h5H2KYUaV+euAHpHhZgWsCrhqFiINsYj2NQ=; b=e6wJi5blvoIvSO+5r24tgS3TNk xZW1ZTKSSkW9Y0Dm6Br3V7dEThgota9EX76Akua6MAM9n9Ec8fZNi/N/Q1bHHSEmgdIgYcjTHa/pi XChE+yLJerim/Wiq6RRfsquF21tZRnXvB7eobvzGKXjsd0pCKiFPrD8WNtRfszGGoV7CE6fIzmjf0 OhEqLWfnNq6ubjT35AzZ8lGV+Vw5BcOyq9wXmQvHWa8LnewcXdCRfwtTFH9Dq2RURWlpLI9pVp+6U CGkHvuwpmjzIuu2ewDzgbliuVOegLKJVsQ/XvxbQAA7uq6V18DlSwdUuD1U7FIaKO+dn5PCwQ/+VV mGeoj73g==; Received: from mail-pl1-x636.google.com ([2607:f8b0:4864:20::636]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1riJ43-00000006rT4-0fM5 for linux-riscv@lists.infradead.org; Thu, 07 Mar 2024 19:06:01 +0000 Received: by mail-pl1-x636.google.com with SMTP id d9443c01a7336-1dc1ff3ba1aso10445685ad.3 for ; Thu, 07 Mar 2024 11:05:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1709838357; x=1710443157; darn=lists.infradead.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=rcFiymo+h5H2KYUaV+euAHpHhZgWsCrhqFiINsYj2NQ=; b=jTfSABJNSlHJc9AWxha3z/iweaZ7eDSPhm3Htm4beki9ybM3vonVqQ7ciytduD9Q+G KYlucK9MX2v/oo8BceF5yJdFQYW0pO6xnI/bdWwl52+BcqYbc9TlLk+9zW8ERBbixLCT Ias/YOyIuqAQ2Nw0rAQiMVXfFvHEKKYk1IIGUy3KeWdrIxIBxpLiFJQuQfWQ28Tl/Gi+ ye643f2ZF4uRg54GF5PKT4/1Fc5bK07aOkY+Lnz3aL4k3trwUUm2xXRMGWTq5svq1hIr jsAj0dL0NuTc3ScIJ8Y5+0+zW3JVgAtqfmXLD9ekizppouPw/1RvXZApW0mLyMtzm+Mu lRTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709838357; x=1710443157; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rcFiymo+h5H2KYUaV+euAHpHhZgWsCrhqFiINsYj2NQ=; b=TmyWwLi/wqamZkN/qC/WvUFerfe4TRSFRDIMk7i+z8/5q90OdaOvC/DSasO0faNEvk SzG19SMP5xM7kPidSg9XJfLPCX4W1yXzfFu5DqvIFrocujPH6/rGV0xhpEYzMx7OW9tN gsJmx86gPjI1x3EQAADcY5g+kcwRToawVKrIK1J28EVSXYEIomvSjR/lmALjMX7mqPu7 WChlWCP5ianff5v1dqBZ+ppmfKlG0mJXjqnGg1AxRJXLwBxYL49RPQ26E1jyegHU8rav yt9IlvK+yxPckrM10SnSpfkFRsfnTQMxhMt8wUMYMFs4niFkyDA4v9gXWSjwVWA6YfV1 Sclg== X-Gm-Message-State: AOJu0Yy4AQA9mt4KjLfdp0nKLA8gXTMTyqAOd0TEVCe+ojwdgcX9ARUD mhZAGU5cJ0MDWnwDepEu3fp2wU/zWG979P9ZKi59PG+20ctZbPlVn+fFuDfvG4o= X-Google-Smtp-Source: AGHT+IGWkAQ/GgJHU1goRENWlzlo1nOGmjqZXS1Ap4YlB/sVxOCqmhx+oHH8w+U3FcSQCMZmvqiyag== X-Received: by 2002:a17:902:b202:b0:1dc:fb12:abbc with SMTP id t2-20020a170902b20200b001dcfb12abbcmr7992411plr.61.1709838357144; Thu, 07 Mar 2024 11:05:57 -0800 (PST) Received: from charlie.ba.rivosinc.com ([64.71.180.162]) by smtp.gmail.com with ESMTPSA id h3-20020a170902680300b001dd526af36csm1747338plk.295.2024.03.07.11.05.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Mar 2024 11:05:56 -0800 (PST) From: Charlie Jenkins Date: Thu, 07 Mar 2024 11:05:47 -0800 Subject: [PATCH v8 3/4] riscv: Decouple emulated unaligned accesses from access speed MIME-Version: 1.0 Message-Id: <20240307-disable_misaligned_probe_config-v8-3-55d696cb398b@rivosinc.com> References: <20240307-disable_misaligned_probe_config-v8-0-55d696cb398b@rivosinc.com> In-Reply-To: <20240307-disable_misaligned_probe_config-v8-0-55d696cb398b@rivosinc.com> To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Jisheng Zhang , Evan Green , =?utf-8?b?Q2zDqW1lbnQgTMOpZ2Vy?= , Eric Biggers , Elliot Berman , Charles Lohr , Conor Dooley Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, Charlie Jenkins X-Mailer: b4 0.12.3 X-Developer-Signature: v=1; a=ed25519-sha256; t=1709838351; l=4202; i=charlie@rivosinc.com; s=20231120; h=from:subject:message-id; bh=IaoDcAIGwiU84s3iFQFa7IKKT21z3ZbSgM4n2pSHuEQ=; b=Gd/AFd9UPYuU9gsHq8mkyXcCsBgVRVzPdcuQYH+roI4KVNNrqQUITF4Emy6vygu23m3rPxwio U5tr57dBkG8BJE6C+mIDw5LFGt6Z3a+kGFeNFUsbIiIjDF7+dODt76C X-Developer-Key: i=charlie@rivosinc.com; a=ed25519; pk=t4RSWpMV1q5lf/NWIeR9z58bcje60/dbtxxmoSfBEcs= X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240307_190559_451716_440A489F X-CRM114-Status: GOOD ( 12.69 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Detecting if a system traps into the kernel on an unaligned access can be performed separately from checking the speed of unaligned accesses. This decoupling will make it possible to selectively enable or disable each of these checks. Signed-off-by: Charlie Jenkins Reviewed-by: Conor Dooley --- arch/riscv/include/asm/cpufeature.h | 2 +- arch/riscv/kernel/cpufeature.c | 25 +++++++++++++++++++++---- arch/riscv/kernel/traps_misaligned.c | 15 +++++++-------- 3 files changed, 29 insertions(+), 13 deletions(-) diff --git a/arch/riscv/include/asm/cpufeature.h b/arch/riscv/include/asm/cpufeature.h index 466e1f591919..6fec91845aa0 100644 --- a/arch/riscv/include/asm/cpufeature.h +++ b/arch/riscv/include/asm/cpufeature.h @@ -37,7 +37,7 @@ void riscv_user_isa_enable(void); #ifdef CONFIG_RISCV_MISALIGNED bool unaligned_ctl_available(void); -bool check_unaligned_access_emulated(int cpu); +bool check_unaligned_access_emulated_all_cpus(void); void unaligned_emulation_finish(void); #else static inline bool unaligned_ctl_available(void) diff --git a/arch/riscv/kernel/cpufeature.c b/arch/riscv/kernel/cpufeature.c index 7878cddccc0d..abb3a2f53106 100644 --- a/arch/riscv/kernel/cpufeature.c +++ b/arch/riscv/kernel/cpufeature.c @@ -719,7 +719,8 @@ static int check_unaligned_access(void *param) void *src; long speed = RISCV_HWPROBE_MISALIGNED_SLOW; - if (check_unaligned_access_emulated(cpu)) + if (IS_ENABLED(CONFIG_RISCV_MISALIGNED) && + per_cpu(misaligned_access_speed, cpu) != RISCV_HWPROBE_MISALIGNED_UNKNOWN) return 0; /* Make an unaligned destination buffer. */ @@ -896,8 +897,8 @@ static int riscv_offline_cpu(unsigned int cpu) return 0; } -/* Measure unaligned access on all CPUs present at boot in parallel. */ -static int check_unaligned_access_all_cpus(void) +/* Measure unaligned access speed on all CPUs present at boot in parallel. */ +static int check_unaligned_access_speed_all_cpus(void) { unsigned int cpu; unsigned int cpu_count = num_possible_cpus(); @@ -935,7 +936,6 @@ static int check_unaligned_access_all_cpus(void) riscv_online_cpu, riscv_offline_cpu); out: - unaligned_emulation_finish(); for_each_cpu(cpu, cpu_online_mask) { if (bufs[cpu]) __free_pages(bufs[cpu], MISALIGNED_BUFFER_ORDER); @@ -945,6 +945,23 @@ static int check_unaligned_access_all_cpus(void) return 0; } +#ifdef CONFIG_RISCV_MISALIGNED +static int check_unaligned_access_all_cpus(void) +{ + bool all_cpus_emulated = check_unaligned_access_emulated_all_cpus(); + + if (!all_cpus_emulated) + return check_unaligned_access_speed_all_cpus(); + + return 0; +} +#else +static int check_unaligned_access_all_cpus(void) +{ + return check_unaligned_access_speed_all_cpus(); +} +#endif + arch_initcall(check_unaligned_access_all_cpus); void riscv_user_isa_enable(void) diff --git a/arch/riscv/kernel/traps_misaligned.c b/arch/riscv/kernel/traps_misaligned.c index c2ed4e689bf9..e55718179f42 100644 --- a/arch/riscv/kernel/traps_misaligned.c +++ b/arch/riscv/kernel/traps_misaligned.c @@ -596,7 +596,7 @@ int handle_misaligned_store(struct pt_regs *regs) return 0; } -bool check_unaligned_access_emulated(int cpu) +static bool check_unaligned_access_emulated(int cpu) { long *mas_ptr = per_cpu_ptr(&misaligned_access_speed, cpu); unsigned long tmp_var, tmp_val; @@ -623,7 +623,7 @@ bool check_unaligned_access_emulated(int cpu) return misaligned_emu_detected; } -void unaligned_emulation_finish(void) +bool check_unaligned_access_emulated_all_cpus(void) { int cpu; @@ -632,13 +632,12 @@ void unaligned_emulation_finish(void) * accesses emulated since tasks requesting such control can run on any * CPU. */ - for_each_online_cpu(cpu) { - if (per_cpu(misaligned_access_speed, cpu) != - RISCV_HWPROBE_MISALIGNED_EMULATED) { - return; - } - } + for_each_online_cpu(cpu) + if (!check_unaligned_access_emulated(cpu)) + return false; + unaligned_ctl = true; + return true; } bool unaligned_ctl_available(void) From patchwork Thu Mar 7 19:05:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Charlie Jenkins X-Patchwork-Id: 13586160 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 04EBBC54E4A for ; Thu, 7 Mar 2024 19:06:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:In-Reply-To:References:Message-Id :MIME-Version:Subject:Date:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=N6KMV2y/UZNaN62biI+fvSxnQznlfdivjvZybucpvkA=; b=Ncy0npmgeWEJvz R0LpMEUo3a2kPiwLEdUWSSWxyoCeLWlJZCwolJV5M+eiFHk+3ye4fhKqPXGD0CNPxjOpLGDyPDtis Rh4VjmpgftxUp0aEMi7mdqwOvcyGbSQsA3zNJQBC3Yhua/xQhenEDjYTgTpJKrr5PufDw1gLJo1RY RzdHqfT8xdR23zMEDxYhTthjSBV2UJbGI3m16ewpM7Ach4hSV6lZP15p4uJAPgVijM1V5/wTa6oGq 7klVs/WxI9FgVNdgt0FJpUxwktIXzIEKfFIFx9F6U/hWvOK67d2d1w8y2JwlG9NSTPFadeihAbVdk dV0/LCMDE9Xd1hkA5f3Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1riJ4F-00000005yb9-2zsb; Thu, 07 Mar 2024 19:06:11 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1riJ47-00000005yVk-3H9Z for linux-riscv@bombadil.infradead.org; Thu, 07 Mar 2024 19:06:04 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Cc:To:In-Reply-To:References: Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Sender:Reply-To:Content-ID:Content-Description; bh=+kRgKvy2tDZROsLBE4TRNc3vK6NXitWNpMmv1T0d9X8=; b=lMMX5V/7dDuPCLrBK8Q8YKIaFo eAQqL2dQK79w7ygprTz4hkJX/auq6GkUWGR1AAlzuBzNQMlayM/rBWGOMeR+ubJ4djmauXHkKiZ9R d9ZmBBDFk4qwmrG91+ZjHwd4h9RPY4LmZY1ZV+y5MseXdnDuXgQNfZMUTaXi4NZ0HVuNS/G5RugPq yDGwwELuOhBJK1Y3fLi2aX3zOdNfsFFIlhfVoUyd1mChjrB+8ktjhluranJBpT8laC3rnw5wzIobK rofX6NJapzclFXSTyDWmJPAfX14c6DVJPWKZiYzTHQ9+AkLr0R5djMBN0yinJFZWaC0PWM1FsEegP xOKww2VQ==; Received: from mail-pl1-x62e.google.com ([2607:f8b0:4864:20::62e]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1riJ44-00000006rTW-2kHp for linux-riscv@lists.infradead.org; Thu, 07 Mar 2024 19:06:02 +0000 Received: by mail-pl1-x62e.google.com with SMTP id d9443c01a7336-1dcd6a3da83so9058175ad.3 for ; Thu, 07 Mar 2024 11:06:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1709838358; x=1710443158; darn=lists.infradead.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=+kRgKvy2tDZROsLBE4TRNc3vK6NXitWNpMmv1T0d9X8=; b=OQjr8r2lHubSnAtQwVRuWbo0wnLbIToqCDljEVrIvZlLvi8s+1FNKu4X/V8YhaSnIo aSSN6Q4QbDMCDMNCMLuxcINUuiV1qIV0gFR9Elzp7dKcoivnveue3uCoNLY3+eC8Aicv TTkcCE233dZC+bEvc0GRDX9nEQe2kiOZ+TrLdwL7KLo8j+SrAMNQH69nGWJtSpmu/M3o PRmrdZOBbD+thWuMtYmK4H4DdtUpNzXux1LA7gwKtq4fqPBkZzqf0h5Z0PsgDzp6DBDF h6gwlqvgB3BH+bYYO/o2EqYgwmijTOgJYnw+EOkj/K2hAMmdPHhiS5EwvHos7LgNgwyD Ticg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709838358; x=1710443158; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+kRgKvy2tDZROsLBE4TRNc3vK6NXitWNpMmv1T0d9X8=; b=PWvjHIAvPCCovy3mQeF7UUfbxGyuo9g4MqWLtRSVuHOCqZAyMTRHSWO17iIHbhRIvj JfAM05mRNb2tH/qDVySiXabmyOEq1mMOnd2UXGjQ+Wk80dZ2se4HauzvQjG3Dqd5Sw2Z S55mp6NQ3nP4UlVzjrmR+SGLaqUlIdCT4M14j1g7Lj24goiT9OE4sveud7MRadF9+vvj akhuFmr9OybQq38vOppHrbLVTTYbb6NJxbq39X823rzbFh0mNj4oK+1J7EvWdbaIQnFB j5HSWcpIF6NdsLhJ41K6LUh0S2UkkxMGuDfCnzxNDglHZ93RRARxd9S/wKQGHdKF6+6R 9vTw== X-Gm-Message-State: AOJu0YywzEFsTMGweuSH8GleKKPaBGSnJSpMw3KvpSaApzosFFNkPQu2 xnXCrx2WTUpncdfhi6CpuJELdjkAczTjZO6eqWoCEgSTP+57Nlk8KiwlyplJZvY= X-Google-Smtp-Source: AGHT+IH7dtXgBzXcjt1MX/4VQ6f1NSrjHQ7P0vdV3ZZCej7grdjYrZUrGgpFKLvzBknn631GDphA2w== X-Received: by 2002:a17:902:a986:b0:1dc:6e6f:5a33 with SMTP id bh6-20020a170902a98600b001dc6e6f5a33mr8114715plb.43.1709838358461; Thu, 07 Mar 2024 11:05:58 -0800 (PST) Received: from charlie.ba.rivosinc.com ([64.71.180.162]) by smtp.gmail.com with ESMTPSA id h3-20020a170902680300b001dd526af36csm1747338plk.295.2024.03.07.11.05.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Mar 2024 11:05:57 -0800 (PST) From: Charlie Jenkins Date: Thu, 07 Mar 2024 11:05:48 -0800 Subject: [PATCH v8 4/4] riscv: Set unaligned access speed at compile time MIME-Version: 1.0 Message-Id: <20240307-disable_misaligned_probe_config-v8-4-55d696cb398b@rivosinc.com> References: <20240307-disable_misaligned_probe_config-v8-0-55d696cb398b@rivosinc.com> In-Reply-To: <20240307-disable_misaligned_probe_config-v8-0-55d696cb398b@rivosinc.com> To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Jisheng Zhang , Evan Green , =?utf-8?b?Q2zDqW1lbnQgTMOpZ2Vy?= , Eric Biggers , Elliot Berman , Charles Lohr , Conor Dooley Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, Charlie Jenkins X-Mailer: b4 0.12.3 X-Developer-Signature: v=1; a=ed25519-sha256; t=1709838351; l=25831; i=charlie@rivosinc.com; s=20231120; h=from:subject:message-id; bh=lnMJWIpDWdTswrXX6v2faY2UZNpWSZLqFcxqdatd22s=; b=9Sds+u3zq+URUmp/F0mAoAW/zes1QItqJKmINvRKciP+mZ5VHikiXyLcdqNl7STKn83rdVwF9 oyRpZJ3yls7AkJryzU5/nPAHk54kq9NPjssBhPJeUm6L877aPB2iTwY X-Developer-Key: i=charlie@rivosinc.com; a=ed25519; pk=t4RSWpMV1q5lf/NWIeR9z58bcje60/dbtxxmoSfBEcs= X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240307_190600_974254_CEA17F2A X-CRM114-Status: GOOD ( 29.26 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Introduce Kconfig options to set the kernel unaligned access support. These options provide a non-portable alternative to the runtime unaligned access probe. To support this, the unaligned access probing code is moved into it's own file and gated behind a new RISCV_PROBE_UNALIGNED_ACCESS_SUPPORT option. Signed-off-by: Charlie Jenkins Reviewed-by: Conor Dooley --- arch/riscv/Kconfig | 60 ++++-- arch/riscv/include/asm/cpufeature.h | 24 +-- arch/riscv/kernel/Makefile | 4 +- arch/riscv/kernel/cpufeature.c | 272 ---------------------------- arch/riscv/kernel/sys_hwprobe.c | 13 ++ arch/riscv/kernel/traps_misaligned.c | 2 + arch/riscv/kernel/unaligned_access_speed.c | 282 +++++++++++++++++++++++++++++ 7 files changed, 361 insertions(+), 296 deletions(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index bffbd869a068..28c1e75ea88a 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -688,27 +688,63 @@ config THREAD_SIZE_ORDER affects irq stack size, which is equal to thread stack size. config RISCV_MISALIGNED - bool "Support misaligned load/store traps for kernel and userspace" + bool select SYSCTL_ARCH_UNALIGN_ALLOW - default y help - Say Y here if you want the kernel to embed support for misaligned - load/store for both kernel and userspace. When disable, misaligned - accesses will generate SIGBUS in userspace and panic in kernel. + Embed support for misaligned load/store for both kernel and userspace. + When disabled, misaligned accesses will generate SIGBUS in userspace + and panic in the kernel. + +choice + prompt "Unaligned Accesses Support" + default RISCV_PROBE_UNALIGNED_ACCESS + help + This determines the level of support for unaligned accesses. This + information is used by the kernel to perform optimizations. It is also + exposed to user space via the hwprobe syscall. The hardware will be + probed at boot by default. + +config RISCV_PROBE_UNALIGNED_ACCESS + bool "Probe for hardware unaligned access support" + select RISCV_MISALIGNED + help + During boot, the kernel will run a series of tests to determine the + speed of unaligned accesses. This probing will dynamically determine + the speed of unaligned accesses on the underlying system. If unaligned + memory accesses trap into the kernel as they are not supported by the + system, the kernel will emulate the unaligned accesses to preserve the + UABI. + +config RISCV_EMULATED_UNALIGNED_ACCESS + bool "Emulate unaligned access where system support is missing" + select RISCV_MISALIGNED + help + If unaligned memory accesses trap into the kernel as they are not + supported by the system, the kernel will emulate the unaligned + accesses to preserve the UABI. When the underlying system does support + unaligned accesses, the unaligned accesses are assumed to be slow. + +config RISCV_SLOW_UNALIGNED_ACCESS + bool "Assume the system supports slow unaligned memory accesses" + depends on NONPORTABLE + help + Assume that the system supports slow unaligned memory accesses. The + kernel and userspace programs may not be able to run at all on systems + that do not support unaligned memory accesses. config RISCV_EFFICIENT_UNALIGNED_ACCESS - bool "Assume the CPU supports fast unaligned memory accesses" + bool "Assume the system supports fast unaligned memory accesses" depends on NONPORTABLE select DCACHE_WORD_ACCESS if MMU select HAVE_EFFICIENT_UNALIGNED_ACCESS help - Say Y here if you want the kernel to assume that the CPU supports - efficient unaligned memory accesses. When enabled, this option - improves the performance of the kernel on such CPUs. However, the - kernel will run much more slowly, or will not be able to run at all, - on CPUs that do not support efficient unaligned memory accesses. + Assume that the system supports fast unaligned memory accesses. When + enabled, this option improves the performance of the kernel on such + systems. However, the kernel and userspace programs will run much more + slowly, or will not be able to run at all, on systems that do not + support efficient unaligned memory accesses. - If unsure what to do here, say N. +endchoice endmenu # "Platform type" diff --git a/arch/riscv/include/asm/cpufeature.h b/arch/riscv/include/asm/cpufeature.h index 6fec91845aa0..46061f5e9764 100644 --- a/arch/riscv/include/asm/cpufeature.h +++ b/arch/riscv/include/asm/cpufeature.h @@ -28,37 +28,39 @@ struct riscv_isainfo { DECLARE_PER_CPU(struct riscv_cpuinfo, riscv_cpuinfo); -DECLARE_PER_CPU(long, misaligned_access_speed); - /* Per-cpu ISA extensions. */ extern struct riscv_isainfo hart_isa[NR_CPUS]; void riscv_user_isa_enable(void); -#ifdef CONFIG_RISCV_MISALIGNED -bool unaligned_ctl_available(void); +#if defined(CONFIG_RISCV_MISALIGNED) bool check_unaligned_access_emulated_all_cpus(void); void unaligned_emulation_finish(void); +bool unaligned_ctl_available(void); +DECLARE_PER_CPU(long, misaligned_access_speed); #else static inline bool unaligned_ctl_available(void) { return false; } - -static inline bool check_unaligned_access_emulated(int cpu) -{ - return false; -} - -static inline void unaligned_emulation_finish(void) {} #endif +#if defined(CONFIG_RISCV_PROBE_UNALIGNED_ACCESS) DECLARE_STATIC_KEY_FALSE(fast_unaligned_access_speed_key); static __always_inline bool has_fast_unaligned_accesses(void) { return static_branch_likely(&fast_unaligned_access_speed_key); } +#else +static __always_inline bool has_fast_unaligned_accesses(void) +{ + if (IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS)) + return true; + else + return false; +} +#endif unsigned long riscv_get_elf_hwcap(void); diff --git a/arch/riscv/kernel/Makefile b/arch/riscv/kernel/Makefile index f71910718053..c8085126a6f9 100644 --- a/arch/riscv/kernel/Makefile +++ b/arch/riscv/kernel/Makefile @@ -38,7 +38,6 @@ extra-y += vmlinux.lds obj-y += head.o obj-y += soc.o obj-$(CONFIG_RISCV_ALTERNATIVE) += alternative.o -obj-y += copy-unaligned.o obj-y += cpu.o obj-y += cpufeature.o obj-y += entry.o @@ -62,6 +61,9 @@ obj-y += tests/ obj-$(CONFIG_MMU) += vdso.o vdso/ obj-$(CONFIG_RISCV_MISALIGNED) += traps_misaligned.o +obj-$(CONFIG_RISCV_MISALIGNED) += unaligned_access_speed.o +obj-$(CONFIG_RISCV_PROBE_UNALIGNED_ACCESS) += copy-unaligned.o + obj-$(CONFIG_FPU) += fpu.o obj-$(CONFIG_RISCV_ISA_V) += vector.o obj-$(CONFIG_RISCV_ISA_V) += kernel_mode_vector.o diff --git a/arch/riscv/kernel/cpufeature.c b/arch/riscv/kernel/cpufeature.c index abb3a2f53106..319670af5704 100644 --- a/arch/riscv/kernel/cpufeature.c +++ b/arch/riscv/kernel/cpufeature.c @@ -11,7 +11,6 @@ #include #include #include -#include #include #include #include @@ -21,20 +20,12 @@ #include #include #include -#include #include #include #include -#include "copy-unaligned.h" - #define NUM_ALPHA_EXTS ('z' - 'a' + 1) -#define MISALIGNED_ACCESS_JIFFIES_LG2 1 -#define MISALIGNED_BUFFER_SIZE 0x4000 -#define MISALIGNED_BUFFER_ORDER get_order(MISALIGNED_BUFFER_SIZE) -#define MISALIGNED_COPY_SIZE ((MISALIGNED_BUFFER_SIZE / 2) - 0x80) - unsigned long elf_hwcap __read_mostly; /* Host ISA bitmap */ @@ -43,11 +34,6 @@ static DECLARE_BITMAP(riscv_isa, RISCV_ISA_EXT_MAX) __read_mostly; /* Per-cpu ISA extensions. */ struct riscv_isainfo hart_isa[NR_CPUS]; -/* Performance information */ -DEFINE_PER_CPU(long, misaligned_access_speed); - -static cpumask_t fast_misaligned_access; - /** * riscv_isa_extension_base() - Get base extension word * @@ -706,264 +692,6 @@ unsigned long riscv_get_elf_hwcap(void) return hwcap; } -static int check_unaligned_access(void *param) -{ - int cpu = smp_processor_id(); - u64 start_cycles, end_cycles; - u64 word_cycles; - u64 byte_cycles; - int ratio; - unsigned long start_jiffies, now; - struct page *page = param; - void *dst; - void *src; - long speed = RISCV_HWPROBE_MISALIGNED_SLOW; - - if (IS_ENABLED(CONFIG_RISCV_MISALIGNED) && - per_cpu(misaligned_access_speed, cpu) != RISCV_HWPROBE_MISALIGNED_UNKNOWN) - return 0; - - /* Make an unaligned destination buffer. */ - dst = (void *)((unsigned long)page_address(page) | 0x1); - /* Unalign src as well, but differently (off by 1 + 2 = 3). */ - src = dst + (MISALIGNED_BUFFER_SIZE / 2); - src += 2; - word_cycles = -1ULL; - /* Do a warmup. */ - __riscv_copy_words_unaligned(dst, src, MISALIGNED_COPY_SIZE); - preempt_disable(); - start_jiffies = jiffies; - while ((now = jiffies) == start_jiffies) - cpu_relax(); - - /* - * For a fixed amount of time, repeatedly try the function, and take - * the best time in cycles as the measurement. - */ - while (time_before(jiffies, now + (1 << MISALIGNED_ACCESS_JIFFIES_LG2))) { - start_cycles = get_cycles64(); - /* Ensure the CSR read can't reorder WRT to the copy. */ - mb(); - __riscv_copy_words_unaligned(dst, src, MISALIGNED_COPY_SIZE); - /* Ensure the copy ends before the end time is snapped. */ - mb(); - end_cycles = get_cycles64(); - if ((end_cycles - start_cycles) < word_cycles) - word_cycles = end_cycles - start_cycles; - } - - byte_cycles = -1ULL; - __riscv_copy_bytes_unaligned(dst, src, MISALIGNED_COPY_SIZE); - start_jiffies = jiffies; - while ((now = jiffies) == start_jiffies) - cpu_relax(); - - while (time_before(jiffies, now + (1 << MISALIGNED_ACCESS_JIFFIES_LG2))) { - start_cycles = get_cycles64(); - mb(); - __riscv_copy_bytes_unaligned(dst, src, MISALIGNED_COPY_SIZE); - mb(); - end_cycles = get_cycles64(); - if ((end_cycles - start_cycles) < byte_cycles) - byte_cycles = end_cycles - start_cycles; - } - - preempt_enable(); - - /* Don't divide by zero. */ - if (!word_cycles || !byte_cycles) { - pr_warn("cpu%d: rdtime lacks granularity needed to measure unaligned access speed\n", - cpu); - - return 0; - } - - if (word_cycles < byte_cycles) - speed = RISCV_HWPROBE_MISALIGNED_FAST; - - ratio = div_u64((byte_cycles * 100), word_cycles); - pr_info("cpu%d: Ratio of byte access time to unaligned word access is %d.%02d, unaligned accesses are %s\n", - cpu, - ratio / 100, - ratio % 100, - (speed == RISCV_HWPROBE_MISALIGNED_FAST) ? "fast" : "slow"); - - per_cpu(misaligned_access_speed, cpu) = speed; - - /* - * Set the value of fast_misaligned_access of a CPU. These operations - * are atomic to avoid race conditions. - */ - if (speed == RISCV_HWPROBE_MISALIGNED_FAST) - cpumask_set_cpu(cpu, &fast_misaligned_access); - else - cpumask_clear_cpu(cpu, &fast_misaligned_access); - - return 0; -} - -static void check_unaligned_access_nonboot_cpu(void *param) -{ - unsigned int cpu = smp_processor_id(); - struct page **pages = param; - - if (smp_processor_id() != 0) - check_unaligned_access(pages[cpu]); -} - -DEFINE_STATIC_KEY_FALSE(fast_unaligned_access_speed_key); - -static void modify_unaligned_access_branches(cpumask_t *mask, int weight) -{ - if (cpumask_weight(mask) == weight) - static_branch_enable_cpuslocked(&fast_unaligned_access_speed_key); - else - static_branch_disable_cpuslocked(&fast_unaligned_access_speed_key); -} - -static void set_unaligned_access_static_branches_except_cpu(int cpu) -{ - /* - * Same as set_unaligned_access_static_branches, except excludes the - * given CPU from the result. When a CPU is hotplugged into an offline - * state, this function is called before the CPU is set to offline in - * the cpumask, and thus the CPU needs to be explicitly excluded. - */ - - cpumask_t fast_except_me; - - cpumask_and(&fast_except_me, &fast_misaligned_access, cpu_online_mask); - cpumask_clear_cpu(cpu, &fast_except_me); - - modify_unaligned_access_branches(&fast_except_me, num_online_cpus() - 1); -} - -static void set_unaligned_access_static_branches(void) -{ - /* - * This will be called after check_unaligned_access_all_cpus so the - * result of unaligned access speed for all CPUs will be available. - * - * To avoid the number of online cpus changing between reading - * cpu_online_mask and calling num_online_cpus, cpus_read_lock must be - * held before calling this function. - */ - - cpumask_t fast_and_online; - - cpumask_and(&fast_and_online, &fast_misaligned_access, cpu_online_mask); - - modify_unaligned_access_branches(&fast_and_online, num_online_cpus()); -} - -static int lock_and_set_unaligned_access_static_branch(void) -{ - cpus_read_lock(); - set_unaligned_access_static_branches(); - cpus_read_unlock(); - - return 0; -} - -arch_initcall_sync(lock_and_set_unaligned_access_static_branch); - -static int riscv_online_cpu(unsigned int cpu) -{ - static struct page *buf; - - /* We are already set since the last check */ - if (per_cpu(misaligned_access_speed, cpu) != RISCV_HWPROBE_MISALIGNED_UNKNOWN) - goto exit; - - buf = alloc_pages(GFP_KERNEL, MISALIGNED_BUFFER_ORDER); - if (!buf) { - pr_warn("Allocation failure, not measuring misaligned performance\n"); - return -ENOMEM; - } - - check_unaligned_access(buf); - __free_pages(buf, MISALIGNED_BUFFER_ORDER); - -exit: - set_unaligned_access_static_branches(); - - return 0; -} - -static int riscv_offline_cpu(unsigned int cpu) -{ - set_unaligned_access_static_branches_except_cpu(cpu); - - return 0; -} - -/* Measure unaligned access speed on all CPUs present at boot in parallel. */ -static int check_unaligned_access_speed_all_cpus(void) -{ - unsigned int cpu; - unsigned int cpu_count = num_possible_cpus(); - struct page **bufs = kzalloc(cpu_count * sizeof(struct page *), - GFP_KERNEL); - - if (!bufs) { - pr_warn("Allocation failure, not measuring misaligned performance\n"); - return 0; - } - - /* - * Allocate separate buffers for each CPU so there's no fighting over - * cache lines. - */ - for_each_cpu(cpu, cpu_online_mask) { - bufs[cpu] = alloc_pages(GFP_KERNEL, MISALIGNED_BUFFER_ORDER); - if (!bufs[cpu]) { - pr_warn("Allocation failure, not measuring misaligned performance\n"); - goto out; - } - } - - /* Check everybody except 0, who stays behind to tend jiffies. */ - on_each_cpu(check_unaligned_access_nonboot_cpu, bufs, 1); - - /* Check core 0. */ - smp_call_on_cpu(0, check_unaligned_access, bufs[0], true); - - /* - * Setup hotplug callbacks for any new CPUs that come online or go - * offline. - */ - cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN, "riscv:online", - riscv_online_cpu, riscv_offline_cpu); - -out: - for_each_cpu(cpu, cpu_online_mask) { - if (bufs[cpu]) - __free_pages(bufs[cpu], MISALIGNED_BUFFER_ORDER); - } - - kfree(bufs); - return 0; -} - -#ifdef CONFIG_RISCV_MISALIGNED -static int check_unaligned_access_all_cpus(void) -{ - bool all_cpus_emulated = check_unaligned_access_emulated_all_cpus(); - - if (!all_cpus_emulated) - return check_unaligned_access_speed_all_cpus(); - - return 0; -} -#else -static int check_unaligned_access_all_cpus(void) -{ - return check_unaligned_access_speed_all_cpus(); -} -#endif - -arch_initcall(check_unaligned_access_all_cpus); - void riscv_user_isa_enable(void) { if (riscv_cpu_has_extension_unlikely(smp_processor_id(), RISCV_ISA_EXT_ZICBOZ)) diff --git a/arch/riscv/kernel/sys_hwprobe.c b/arch/riscv/kernel/sys_hwprobe.c index a7c56b41efd2..8cae41a502dd 100644 --- a/arch/riscv/kernel/sys_hwprobe.c +++ b/arch/riscv/kernel/sys_hwprobe.c @@ -147,6 +147,7 @@ static bool hwprobe_ext0_has(const struct cpumask *cpus, unsigned long ext) return (pair.value & ext); } +#if defined(CONFIG_RISCV_PROBE_UNALIGNED_ACCESS) static u64 hwprobe_misaligned(const struct cpumask *cpus) { int cpu; @@ -169,6 +170,18 @@ static u64 hwprobe_misaligned(const struct cpumask *cpus) return perf; } +#else +static u64 hwprobe_misaligned(const struct cpumask *cpus) +{ + if (IS_ENABLED(CONFIG_RISCV_EFFICIENT_UNALIGNED_ACCESS)) + return RISCV_HWPROBE_MISALIGNED_FAST; + + if (IS_ENABLED(CONFIG_RISCV_EMULATED_UNALIGNED_ACCESS) && unaligned_ctl_available()) + return RISCV_HWPROBE_MISALIGNED_EMULATED; + + return RISCV_HWPROBE_MISALIGNED_SLOW; +} +#endif static void hwprobe_one_pair(struct riscv_hwprobe *pair, const struct cpumask *cpus) diff --git a/arch/riscv/kernel/traps_misaligned.c b/arch/riscv/kernel/traps_misaligned.c index e55718179f42..2adb7c3e4dd5 100644 --- a/arch/riscv/kernel/traps_misaligned.c +++ b/arch/riscv/kernel/traps_misaligned.c @@ -413,7 +413,9 @@ int handle_misaligned_load(struct pt_regs *regs) perf_sw_event(PERF_COUNT_SW_ALIGNMENT_FAULTS, 1, regs, addr); +#ifdef CONFIG_RISCV_PROBE_UNALIGNED_ACCESS *this_cpu_ptr(&misaligned_access_speed) = RISCV_HWPROBE_MISALIGNED_EMULATED; +#endif if (!unaligned_enabled) return -1; diff --git a/arch/riscv/kernel/unaligned_access_speed.c b/arch/riscv/kernel/unaligned_access_speed.c new file mode 100644 index 000000000000..52264ea4f0bd --- /dev/null +++ b/arch/riscv/kernel/unaligned_access_speed.c @@ -0,0 +1,282 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright 2024 Rivos Inc. + */ + +#include +#include +#include +#include +#include +#include +#include +#include + +#include "copy-unaligned.h" + +#define MISALIGNED_ACCESS_JIFFIES_LG2 1 +#define MISALIGNED_BUFFER_SIZE 0x4000 +#define MISALIGNED_BUFFER_ORDER get_order(MISALIGNED_BUFFER_SIZE) +#define MISALIGNED_COPY_SIZE ((MISALIGNED_BUFFER_SIZE / 2) - 0x80) + +DEFINE_PER_CPU(long, misaligned_access_speed); + +#ifdef CONFIG_RISCV_PROBE_UNALIGNED_ACCESS +static cpumask_t fast_misaligned_access; +static int check_unaligned_access(void *param) +{ + int cpu = smp_processor_id(); + u64 start_cycles, end_cycles; + u64 word_cycles; + u64 byte_cycles; + int ratio; + unsigned long start_jiffies, now; + struct page *page = param; + void *dst; + void *src; + long speed = RISCV_HWPROBE_MISALIGNED_SLOW; + + if (per_cpu(misaligned_access_speed, cpu) != RISCV_HWPROBE_MISALIGNED_UNKNOWN) + return 0; + + /* Make an unaligned destination buffer. */ + dst = (void *)((unsigned long)page_address(page) | 0x1); + /* Unalign src as well, but differently (off by 1 + 2 = 3). */ + src = dst + (MISALIGNED_BUFFER_SIZE / 2); + src += 2; + word_cycles = -1ULL; + /* Do a warmup. */ + __riscv_copy_words_unaligned(dst, src, MISALIGNED_COPY_SIZE); + preempt_disable(); + start_jiffies = jiffies; + while ((now = jiffies) == start_jiffies) + cpu_relax(); + + /* + * For a fixed amount of time, repeatedly try the function, and take + * the best time in cycles as the measurement. + */ + while (time_before(jiffies, now + (1 << MISALIGNED_ACCESS_JIFFIES_LG2))) { + start_cycles = get_cycles64(); + /* Ensure the CSR read can't reorder WRT to the copy. */ + mb(); + __riscv_copy_words_unaligned(dst, src, MISALIGNED_COPY_SIZE); + /* Ensure the copy ends before the end time is snapped. */ + mb(); + end_cycles = get_cycles64(); + if ((end_cycles - start_cycles) < word_cycles) + word_cycles = end_cycles - start_cycles; + } + + byte_cycles = -1ULL; + __riscv_copy_bytes_unaligned(dst, src, MISALIGNED_COPY_SIZE); + start_jiffies = jiffies; + while ((now = jiffies) == start_jiffies) + cpu_relax(); + + while (time_before(jiffies, now + (1 << MISALIGNED_ACCESS_JIFFIES_LG2))) { + start_cycles = get_cycles64(); + mb(); + __riscv_copy_bytes_unaligned(dst, src, MISALIGNED_COPY_SIZE); + mb(); + end_cycles = get_cycles64(); + if ((end_cycles - start_cycles) < byte_cycles) + byte_cycles = end_cycles - start_cycles; + } + + preempt_enable(); + + /* Don't divide by zero. */ + if (!word_cycles || !byte_cycles) { + pr_warn("cpu%d: rdtime lacks granularity needed to measure unaligned access speed\n", + cpu); + + return 0; + } + + if (word_cycles < byte_cycles) + speed = RISCV_HWPROBE_MISALIGNED_FAST; + + ratio = div_u64((byte_cycles * 100), word_cycles); + pr_info("cpu%d: Ratio of byte access time to unaligned word access is %d.%02d, unaligned accesses are %s\n", + cpu, + ratio / 100, + ratio % 100, + (speed == RISCV_HWPROBE_MISALIGNED_FAST) ? "fast" : "slow"); + + per_cpu(misaligned_access_speed, cpu) = speed; + + /* + * Set the value of fast_misaligned_access of a CPU. These operations + * are atomic to avoid race conditions. + */ + if (speed == RISCV_HWPROBE_MISALIGNED_FAST) + cpumask_set_cpu(cpu, &fast_misaligned_access); + else + cpumask_clear_cpu(cpu, &fast_misaligned_access); + + return 0; +} + +static void check_unaligned_access_nonboot_cpu(void *param) +{ + unsigned int cpu = smp_processor_id(); + struct page **pages = param; + + if (smp_processor_id() != 0) + check_unaligned_access(pages[cpu]); +} + +DEFINE_STATIC_KEY_FALSE(fast_unaligned_access_speed_key); + +static void modify_unaligned_access_branches(cpumask_t *mask, int weight) +{ + if (cpumask_weight(mask) == weight) + static_branch_enable_cpuslocked(&fast_unaligned_access_speed_key); + else + static_branch_disable_cpuslocked(&fast_unaligned_access_speed_key); +} + +static void set_unaligned_access_static_branches_except_cpu(int cpu) +{ + /* + * Same as set_unaligned_access_static_branches, except excludes the + * given CPU from the result. When a CPU is hotplugged into an offline + * state, this function is called before the CPU is set to offline in + * the cpumask, and thus the CPU needs to be explicitly excluded. + */ + + cpumask_t fast_except_me; + + cpumask_and(&fast_except_me, &fast_misaligned_access, cpu_online_mask); + cpumask_clear_cpu(cpu, &fast_except_me); + + modify_unaligned_access_branches(&fast_except_me, num_online_cpus() - 1); +} + +static void set_unaligned_access_static_branches(void) +{ + /* + * This will be called after check_unaligned_access_all_cpus so the + * result of unaligned access speed for all CPUs will be available. + * + * To avoid the number of online cpus changing between reading + * cpu_online_mask and calling num_online_cpus, cpus_read_lock must be + * held before calling this function. + */ + + cpumask_t fast_and_online; + + cpumask_and(&fast_and_online, &fast_misaligned_access, cpu_online_mask); + + modify_unaligned_access_branches(&fast_and_online, num_online_cpus()); +} + +static int lock_and_set_unaligned_access_static_branch(void) +{ + cpus_read_lock(); + set_unaligned_access_static_branches(); + cpus_read_unlock(); + + return 0; +} + +arch_initcall_sync(lock_and_set_unaligned_access_static_branch); + +static int riscv_online_cpu(unsigned int cpu) +{ + static struct page *buf; + + /* We are already set since the last check */ + if (per_cpu(misaligned_access_speed, cpu) != RISCV_HWPROBE_MISALIGNED_UNKNOWN) + goto exit; + + buf = alloc_pages(GFP_KERNEL, MISALIGNED_BUFFER_ORDER); + if (!buf) { + pr_warn("Allocation failure, not measuring misaligned performance\n"); + return -ENOMEM; + } + + check_unaligned_access(buf); + __free_pages(buf, MISALIGNED_BUFFER_ORDER); + +exit: + set_unaligned_access_static_branches(); + + return 0; +} + +static int riscv_offline_cpu(unsigned int cpu) +{ + set_unaligned_access_static_branches_except_cpu(cpu); + + return 0; +} + +/* Measure unaligned access speed on all CPUs present at boot in parallel. */ +static int check_unaligned_access_speed_all_cpus(void) +{ + unsigned int cpu; + unsigned int cpu_count = num_possible_cpus(); + struct page **bufs = kzalloc(cpu_count * sizeof(struct page *), + GFP_KERNEL); + + if (!bufs) { + pr_warn("Allocation failure, not measuring misaligned performance\n"); + return 0; + } + + /* + * Allocate separate buffers for each CPU so there's no fighting over + * cache lines. + */ + for_each_cpu(cpu, cpu_online_mask) { + bufs[cpu] = alloc_pages(GFP_KERNEL, MISALIGNED_BUFFER_ORDER); + if (!bufs[cpu]) { + pr_warn("Allocation failure, not measuring misaligned performance\n"); + goto out; + } + } + + /* Check everybody except 0, who stays behind to tend jiffies. */ + on_each_cpu(check_unaligned_access_nonboot_cpu, bufs, 1); + + /* Check core 0. */ + smp_call_on_cpu(0, check_unaligned_access, bufs[0], true); + + /* + * Setup hotplug callbacks for any new CPUs that come online or go + * offline. + */ + cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN, "riscv:online", + riscv_online_cpu, riscv_offline_cpu); + +out: + for_each_cpu(cpu, cpu_online_mask) { + if (bufs[cpu]) + __free_pages(bufs[cpu], MISALIGNED_BUFFER_ORDER); + } + + kfree(bufs); + return 0; +} + +static int check_unaligned_access_all_cpus(void) +{ + bool all_cpus_emulated = check_unaligned_access_emulated_all_cpus(); + + if (!all_cpus_emulated) + return check_unaligned_access_speed_all_cpus(); + + return 0; +} +#else /* CONFIG_RISCV_PROBE_UNALIGNED_ACCESS */ +static int check_unaligned_access_all_cpus(void) +{ + check_unaligned_access_emulated_all_cpus(); + + return 0; +} +#endif + +arch_initcall(check_unaligned_access_all_cpus);