From patchwork Thu Apr 11 00:51:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hector Martin X-Patchwork-Id: 13625206 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 59C41CD11C2 for ; Thu, 11 Apr 2024 00:51:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:In-Reply-To:References:Message-Id :MIME-Version:Subject:Date:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=b57Xc7JdrW4d24Ko6O81HkUBf80aFFXQjhFcv2yqtGY=; b=hQ8TADQkTHGkCc tMrexznX0PjdwjAZU43YwtyULhAXwtvCLL3EqkLaAr257fZqazEMaaLpho8WV0w6qk6E3Fnugkirc LZseGmh4ryIEKzFA4vDaamrsr0EKcIxpuq+in4bMSDyrYrSppjOKTZA1B/9RBgjI7ZRMoz96RSD51 mFgcmyBGbM5f7W1fCmrI2lirHuS47nOwzDVZpsTZA+CNobJ0pyddaoVIZLYy+5ELsPe6bFPUmlBt/ PYMrklQgxkQItsASNJed0GCmmo8OdDSexbQXPRCX+baJOI3h3QSL+vRDiceX1ZECoBcWxvqopOa0G npnXy8HlbU0Ucu7lan6w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1ruifH-00000009ie8-3YmE; Thu, 11 Apr 2024 00:51:43 +0000 Received: from marcansoft.com ([212.63.210.85] helo=mail.marcansoft.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1ruifA-00000009ibR-3m22 for linux-arm-kernel@lists.infradead.org; Thu, 11 Apr 2024 00:51:38 +0000 Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: sendonly@marcansoft.com) by mail.marcansoft.com (Postfix) with ESMTPSA id CAE1A425BC; Thu, 11 Apr 2024 00:51:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=marcan.st; s=default; t=1712796695; bh=m9CLW62XS8hN2CqVX6w4Wd1xurxuFzqdfTlLrD+lanE=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=NBMYzbZr6/k0E9id/N48LMb1DAGMmEgWK38+ilVhdJFsyslsW1Tjk9IfslGPpUn7q Lg5GAjyQ1tYLfPHKPVqfXMpLKHKjAo/DuUiJGBF23PdSc+8cQVo3FuXShADgAl9vkl 7xqq+QsdyROsylR3WQ1mM3ssN97T827ZPpX/6tzvWc1BiC1gvuCdoEEomncL2yNVS5 ZN6SMg4ZfdAAUlRtE7nKvRaElMu6aHxAp/Q7+1zuGbGbhxIieMmNZ2SMmFBnY8cJ/7 n97H3yNBzArs88UBgUzokbGAso06IslmxjNIthfQCu+YxV6WJxOvuk91d6V9PXsl3Y 6fpS3HLqyZKFQ== From: Hector Martin Date: Thu, 11 Apr 2024 09:51:20 +0900 Subject: [PATCH 1/4] prctl: Introduce PR_{SET,GET}_MEM_MODEL MIME-Version: 1.0 Message-Id: <20240411-tso-v1-1-754f11abfbff@marcan.st> References: <20240411-tso-v1-0-754f11abfbff@marcan.st> In-Reply-To: <20240411-tso-v1-0-754f11abfbff@marcan.st> To: Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland Cc: Zayd Qumsieh , Justin Lu , Ryan Houdek , Mark Brown , Ard Biesheuvel , Mateusz Guzik , Anshuman Khandual , Oliver Upton , Miguel Luis , Joey Gouly , Christoph Paasch , Kees Cook , Sami Tolvanen , Baoquan He , Joel Granados , Dawei Li , Andrew Morton , Florent Revest , David Hildenbrand , Stefan Roesch , Andy Chiu , Josh Triplett , Oleg Nesterov , Helge Deller , Zev Weiss , Ondrej Mosnacek , Miguel Ojeda , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Asahi Linux , Hector Martin X-Mailer: b4 0.13.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=4103; i=marcan@marcan.st; h=from:subject:message-id; bh=m9CLW62XS8hN2CqVX6w4Wd1xurxuFzqdfTlLrD+lanE=; b=owGbwMvMwCUm+yP4NEe/cRLjabUkhjRxE56vn3UL3add4zzFreKy5N7SuRskrXwvmNXGz/jVH /2GVbaso5SFQYyLQVZMkaXxRO+pbs/p59RVU6bDzGFlAhnCwMUpABOZK8TI0J7+dcND45wuhinf Zx1+a/UjQUFU0VD3o+HhBZcD/88+Y8jw34Xlnt6TrOcad6wXtJX9/L6x58CWeotVixma9sh2fYx w4wAA X-Developer-Key: i=marcan@marcan.st; a=openpgp; fpr=FC18F00317968B7BE86201CBE22A629A4C515DD5 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240410_175137_133672_CA2E7358 X-CRM114-Status: GOOD ( 20.18 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On some architectures, it is possible to query and/or change the CPU memory model. This allows userspace to switch to a stricter memory model for performance reasons, such as when emulating code for another architecture where that model is the default. Introduce two prctls to allow userspace to query and set the memory model for a thread. Two models are initially defined: - PR_SET_MEM_MODEL_DEFAULT requests the default memory model for the architecture. - PR_SET_MEM_MODEL_TSO requests the x86 TSO memory model. PR_SET_MEM_MODEL is allowed to set a stricter memory model than requested if available, in which case it will return successfully. If the requested memory model cannot be fulfilled, it will return an error. The memory model that was actually set can be queried by a subsequent call to PR_GET_MEM_MODEL. Examples: - On a CPU with not support for a memory model at least as strong as TSO, PR_SET_MEM_MODEL(PR_SET_MEM_MODEL_TSO) fails. - On a CPU with runtime-configurable TSO support, PR_SET_MEM_MODEL can toggle the memory model between DEFAULT and TSO at will. - On a CPU where the only memory model is at least as strict as TSO, PR_GET_MEM_MODEL will return PR_SET_MEM_MODEL_DEFAULT, and PR_SET_MEM_MODEL(PR_SET_MEM_MODEL_TSO) will return success but leave the memory model at PR_SET_MEM_MODEL_DEFAULT. This implies that the default is in fact at least as strict as TSO. Signed-off-by: Hector Martin --- include/linux/memory_ordering_model.h | 11 +++++++++++ include/uapi/linux/prctl.h | 5 +++++ kernel/sys.c | 21 +++++++++++++++++++++ 3 files changed, 37 insertions(+) diff --git a/include/linux/memory_ordering_model.h b/include/linux/memory_ordering_model.h new file mode 100644 index 000000000000..267a12ca6630 --- /dev/null +++ b/include/linux/memory_ordering_model.h @@ -0,0 +1,11 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __ASM_MEMORY_ORDERING_MODEL_H +#define __ASM_MEMORY_ORDERING_MODEL_H + +/* Arch hooks to implement the PR_{GET_SET}_MEM_MODEL prctls */ + +struct task_struct; +int arch_prctl_mem_model_get(struct task_struct *t); +int arch_prctl_mem_model_set(struct task_struct *t, unsigned long val); + +#endif diff --git a/include/uapi/linux/prctl.h b/include/uapi/linux/prctl.h index 370ed14b1ae0..961216093f11 100644 --- a/include/uapi/linux/prctl.h +++ b/include/uapi/linux/prctl.h @@ -306,4 +306,9 @@ struct prctl_mm_map { # define PR_RISCV_V_VSTATE_CTRL_NEXT_MASK 0xc # define PR_RISCV_V_VSTATE_CTRL_MASK 0x1f +#define PR_GET_MEM_MODEL 0x6d4d444c +#define PR_SET_MEM_MODEL 0x4d4d444c +# define PR_SET_MEM_MODEL_DEFAULT 0 +# define PR_SET_MEM_MODEL_TSO 1 + #endif /* _LINUX_PRCTL_H */ diff --git a/kernel/sys.c b/kernel/sys.c index f8e543f1e38a..6af659a9f826 100644 --- a/kernel/sys.c +++ b/kernel/sys.c @@ -45,6 +45,7 @@ #include #include #include +#include #include #include @@ -2442,6 +2443,16 @@ static int prctl_get_auxv(void __user *addr, unsigned long len) return sizeof(mm->saved_auxv); } +int __weak arch_prctl_mem_model_get(struct task_struct *t) +{ + return -EINVAL; +} + +int __weak arch_prctl_mem_model_set(struct task_struct *t, unsigned long val) +{ + return -EINVAL; +} + SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3, unsigned long, arg4, unsigned long, arg5) { @@ -2757,6 +2768,16 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3, case PR_RISCV_V_GET_CONTROL: error = RISCV_V_GET_CONTROL(); break; + case PR_GET_MEM_MODEL: + if (arg2 || arg3 || arg4 || arg5) + return -EINVAL; + error = arch_prctl_mem_model_get(me); + break; + case PR_SET_MEM_MODEL: + if (arg3 || arg4 || arg5) + return -EINVAL; + error = arch_prctl_mem_model_set(me, arg2); + break; default: error = -EINVAL; break; From patchwork Thu Apr 11 00:51:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hector Martin X-Patchwork-Id: 13625208 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3927FCD11C2 for ; Thu, 11 Apr 2024 00:51:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:In-Reply-To:References:Message-Id :MIME-Version:Subject:Date:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=yvzG0KBQSdcJDH/5WGwnAIurt0bXXkHeuM2PQCtlX54=; b=SOpBukn3MSChx9 gHVhhc+4OX6zAIUbOVL/XphlYpCQ1mtOpUbkRrQIoNFjiSeCnoxLn8HvAeRFYuiPZ5oSNSo+MFmF4 O/S2XC4rixbUOP593US33QHT0PSrMq+Kd4ByfR559r8Gk6sQERef5eXKBwZ4sfw/gVcbuMz2j4l+9 zkPhT9d3qusjIe5P1mvUF/JlxfkwbI8yCAwSw1K+h1B+rKaetmmPPIPGEUScG98+Y/ZOnhmjlWy+Y jDeOme08yxZc13uG09LsQ1Ij3WELMleXceeHEb7AA+ayCerZD1D/ELJbNhijad9wfURZ4ndeHQATn VZgFAwqnZk16unbJFuig==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1ruifM-00000009igs-3l51; Thu, 11 Apr 2024 00:51:49 +0000 Received: from marcansoft.com ([212.63.210.85] helo=mail.marcansoft.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1ruifF-00000009id7-3X5p for linux-arm-kernel@lists.infradead.org; Thu, 11 Apr 2024 00:51:43 +0000 Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: sendonly@marcansoft.com) by mail.marcansoft.com (Postfix) with ESMTPSA id E1A2A425BF; Thu, 11 Apr 2024 00:51:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=marcan.st; s=default; t=1712796700; bh=oISCn6HTlsZmKUOaxgKnHYg672B8CResDHmP3zCW1MM=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=nX/a7pNHfAXiCvRUmeEGsXfJVsBpAIZIlubeHMBosYwoGG9hmjlvdA6KaxDZqVlmF DQ8gk3hkbT1EJItGpRUeCt8GpUbQ6y/tVQFTq6tgETvz2+e9+DmSuF3ETKPyqrPKbX +Vcobr4fAoSgW+2VQQVIq6NB9i6nF2s6XQo0N7XfSFv6AgbTpUFXcnR9AnBN+/dAy4 UPWebzHr4TRcRy/TWjdq3mddtslfLBX3iPQmq354fAm4TS31TL3Rv0+XUVsVxQ6lKw nBWLYwF1QWsf4zFyV0WXi6BHwx+BDABMl+o4NvlHVRk5cK+i9lUBdZax/YpXNeWxlS Yz4v6kLwcQJLQ== From: Hector Martin Date: Thu, 11 Apr 2024 09:51:21 +0900 Subject: [PATCH 2/4] arm64: Implement PR_{GET,SET}_MEM_MODEL for always-TSO CPUs MIME-Version: 1.0 Message-Id: <20240411-tso-v1-2-754f11abfbff@marcan.st> References: <20240411-tso-v1-0-754f11abfbff@marcan.st> In-Reply-To: <20240411-tso-v1-0-754f11abfbff@marcan.st> To: Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland Cc: Zayd Qumsieh , Justin Lu , Ryan Houdek , Mark Brown , Ard Biesheuvel , Mateusz Guzik , Anshuman Khandual , Oliver Upton , Miguel Luis , Joey Gouly , Christoph Paasch , Kees Cook , Sami Tolvanen , Baoquan He , Joel Granados , Dawei Li , Andrew Morton , Florent Revest , David Hildenbrand , Stefan Roesch , Andy Chiu , Josh Triplett , Oleg Nesterov , Helge Deller , Zev Weiss , Ondrej Mosnacek , Miguel Ojeda , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Asahi Linux , Hector Martin X-Mailer: b4 0.13.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=7167; i=marcan@marcan.st; h=from:subject:message-id; bh=oISCn6HTlsZmKUOaxgKnHYg672B8CResDHmP3zCW1MM=; b=owGbwMvMwCUm+yP4NEe/cRLjabUkhjRxE57a5HP7e152SOzeJvh2p2AX49RolkDJPZGnTUpP5 n4Nf67cUcrCIMbFICumyNJ4ovdUt+f0c+qqKdNh5rAygQxh4OIUgIlINDEyvCuZ4TiNf9VBZyNn vnz36mhP85dRLcaxXYvPJi7UdnPRYfgr3/FliYvzRIPclbu/bHBMbTnw6Y+09xdhRZMVFxklbRJ 4AQ== X-Developer-Key: i=marcan@marcan.st; a=openpgp; fpr=FC18F00317968B7BE86201CBE22A629A4C515DD5 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240410_175142_074556_8D1C30F4 X-CRM114-Status: GOOD ( 21.73 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Some ARM64 implementations are known to always use the TSO memory model. Add trivial support for the PR_{GET,SET}_MEM_MODEL prctl, which allows userspace to learn this fact. Known TSO implementations: - Nvidia Denver - Nvidia Carmel - Fujitsu A64FX Signed-off-by: Hector Martin --- arch/arm64/Kconfig | 9 +++++++++ arch/arm64/include/asm/cpufeature.h | 4 ++++ arch/arm64/kernel/Makefile | 3 ++- arch/arm64/kernel/cpufeature.c | 11 +++++----- arch/arm64/kernel/cpufeature_impdef.c | 38 +++++++++++++++++++++++++++++++++++ arch/arm64/kernel/process.c | 24 ++++++++++++++++++++++ arch/arm64/tools/cpucaps | 1 + 7 files changed, 84 insertions(+), 6 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 7b11c98b3e84..f8e66fe44ff4 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -2162,6 +2162,15 @@ config ARM64_DEBUG_PRIORITY_MASKING If unsure, say N endif # ARM64_PSEUDO_NMI +config ARM64_MEMORY_MODEL_CONTROL + bool "Runtime memory model control" + help + Some ARM64 CPUs support runtime switching of the CPU memory + model, which can be useful to emulate other CPU architectures + which have different memory models. Say Y to enable support + for the PR_SET_MEM_MODEL/PR_GET_MEM_MODEL prctl() calls on + CPUs with this feature. + config RELOCATABLE bool "Build a relocatable kernel image" if EXPERT select ARCH_HAS_RELR diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 8b904a757bd3..fb215b0e7529 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -1032,6 +1032,10 @@ static inline bool cpu_has_lpa2(void) #endif } +void __init init_cpucap_indirect_list_impdef(void); +void __init init_cpucap_indirect_list_from_array(const struct arm64_cpu_capabilities *caps); +bool cpufeature_matches(u64 reg, const struct arm64_cpu_capabilities *entry); + #endif /* __ASSEMBLY__ */ #endif diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index 763824963ed1..5eaaee7b8358 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -33,7 +33,8 @@ obj-y := debug-monitors.o entry.o irq.o fpsimd.o \ return_address.o cpuinfo.o cpu_errata.o \ cpufeature.o alternative.o cacheinfo.o \ smp.o smp_spin_table.o topology.o smccc-call.o \ - syscall.o proton-pack.o idle.o patching.o pi/ + syscall.o proton-pack.o idle.o patching.o pi/ \ + cpufeature_impdef.o obj-$(CONFIG_COMPAT) += sys32.o signal32.o \ sys_compat.o diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 56583677c1f2..e39ab93ad683 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -1028,7 +1028,7 @@ static void init_cpu_ftr_reg(u32 sys_reg, u64 new) extern const struct arm64_cpu_capabilities arm64_errata[]; static const struct arm64_cpu_capabilities arm64_features[]; -static void __init +void __init init_cpucap_indirect_list_from_array(const struct arm64_cpu_capabilities *caps) { for (; caps->matches; caps++) { @@ -1540,8 +1540,8 @@ has_always(const struct arm64_cpu_capabilities *entry, int scope) return true; } -static bool -feature_matches(u64 reg, const struct arm64_cpu_capabilities *entry) +bool +cpufeature_matches(u64 reg, const struct arm64_cpu_capabilities *entry) { int val, min, max; u64 tmp; @@ -1594,14 +1594,14 @@ has_user_cpuid_feature(const struct arm64_cpu_capabilities *entry, int scope) if (!mask) return false; - return feature_matches(val, entry); + return cpufeature_matches(val, entry); } static bool has_cpuid_feature(const struct arm64_cpu_capabilities *entry, int scope) { u64 val = read_scoped_sysreg(entry, scope); - return feature_matches(val, entry); + return cpufeature_matches(val, entry); } const struct cpumask *system_32bit_el0_cpumask(void) @@ -3486,6 +3486,7 @@ void __init setup_boot_cpu_features(void) * handle the boot CPU. */ init_cpucap_indirect_list(); + init_cpucap_indirect_list_impdef(); /* * Detect broken pseudo-NMI. Must be called _before_ the call to diff --git a/arch/arm64/kernel/cpufeature_impdef.c b/arch/arm64/kernel/cpufeature_impdef.c new file mode 100644 index 000000000000..bb04a8e3d79d --- /dev/null +++ b/arch/arm64/kernel/cpufeature_impdef.c @@ -0,0 +1,38 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Contains implementation-defined CPU feature definitions. + */ + +#include + +#ifdef CONFIG_ARM64_MEMORY_MODEL_CONTROL +static bool has_tso_fixed(const struct arm64_cpu_capabilities *entry, int scope) +{ + /* List of CPUs that always use the TSO memory model */ + static const struct midr_range fixed_tso_list[] = { + MIDR_ALL_VERSIONS(MIDR_NVIDIA_DENVER), + MIDR_ALL_VERSIONS(MIDR_NVIDIA_CARMEL), + MIDR_ALL_VERSIONS(MIDR_FUJITSU_A64FX), + { /* sentinel */ } + }; + + return is_midr_in_range_list(read_cpuid_id(), fixed_tso_list); +} +#endif + +static const struct arm64_cpu_capabilities arm64_impdef_features[] = { +#ifdef CONFIG_ARM64_MEMORY_MODEL_CONTROL + { + .desc = "TSO memory model (Fixed)", + .capability = ARM64_HAS_TSO_FIXED, + .type = ARM64_CPUCAP_SYSTEM_FEATURE, + .matches = has_tso_fixed, + }, +#endif + {}, +}; + +void __init init_cpucap_indirect_list_impdef(void) +{ + init_cpucap_indirect_list_from_array(arm64_impdef_features); +} diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c index 4ae31b7af6c3..7920056bad3e 100644 --- a/arch/arm64/kernel/process.c +++ b/arch/arm64/kernel/process.c @@ -41,6 +41,7 @@ #include #include #include +#include #include #include @@ -513,6 +514,25 @@ void update_sctlr_el1(u64 sctlr) isb(); } +#ifdef CONFIG_ARM64_MEMORY_MODEL_CONTROL +int arch_prctl_mem_model_get(struct task_struct *t) +{ + return PR_SET_MEM_MODEL_DEFAULT; +} + +int arch_prctl_mem_model_set(struct task_struct *t, unsigned long val) +{ + if (alternative_has_cap_unlikely(ARM64_HAS_TSO_FIXED) && + val == PR_SET_MEM_MODEL_TSO) + return 0; + + if (val == PR_SET_MEM_MODEL_DEFAULT) + return 0; + + return -EINVAL; +} +#endif + /* * Thread switching. */ @@ -651,6 +671,10 @@ void arch_setup_new_exec(void) arch_prctl_spec_ctrl_set(current, PR_SPEC_STORE_BYPASS, PR_SPEC_ENABLE); } + +#ifdef CONFIG_ARM64_MEMORY_MODEL_CONTROL + arch_prctl_mem_model_set(current, PR_SET_MEM_MODEL_DEFAULT); +#endif } #ifdef CONFIG_ARM64_TAGGED_ADDR_ABI diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps index 62b2838a231a..daa6b9495402 100644 --- a/arch/arm64/tools/cpucaps +++ b/arch/arm64/tools/cpucaps @@ -52,6 +52,7 @@ HAS_STAGE2_FWB HAS_TCR2 HAS_TIDCP1 HAS_TLB_RANGE +HAS_TSO_FIXED HAS_VA52 HAS_VIRT_HOST_EXTN HAS_WFXT From patchwork Thu Apr 11 00:51:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hector Martin X-Patchwork-Id: 13625209 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AD766CD1297 for ; Thu, 11 Apr 2024 00:52:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:In-Reply-To:References:Message-Id :MIME-Version:Subject:Date:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=nLqn10itK6uzPFjxNTUJz4ib0VBNPEZobwcZzSUpHWg=; b=ACwwHgBvs71CuX zCbwX1yQDuhzbd6YFfQuvlOCkmhefoMSaVCsgtECxirNopEU1qoyqRx2cexRBTo9Yg1i3hZy0Am+C zLcUNdPpJ0IgxWmnQEAYO9fsnYDnFL+f4z5lUpNNiB8pyc0W6ThZIX/IOHp618fovxv/idr7b4BKF 06mmqmajwh8x0J7vuYDkiSTSamaIkDDF4HDCNmyvb5rT42ct0U5TyCqk9J7+3AKCNc1FWxcceNZ4M 2fxgILDOdPbLm6W4eZvSiBEwKLPUgXTeJTdELR0P/0P3lV5bbc3jFy5csHaLjfmZx7SkuK8tskgxv CcBa5a0i6zXiiZUxo9ZQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1ruifS-00000009ikY-2llQ; Thu, 11 Apr 2024 00:51:54 +0000 Received: from marcansoft.com ([212.63.210.85] helo=mail.marcansoft.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1ruifK-00000009if0-2rJ9 for linux-arm-kernel@lists.infradead.org; Thu, 11 Apr 2024 00:51:49 +0000 Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: sendonly@marcansoft.com) by mail.marcansoft.com (Postfix) with ESMTPSA id DBC20425C1; Thu, 11 Apr 2024 00:51:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=marcan.st; s=default; t=1712796705; bh=0K4TZBePvBxEpbZMaDmUuVsuBfSttKVJG4yDVJUb46w=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=hz1spajEMg+3ADUd7G6gmCypE69lWE+6evcxkkEXR5QPC9frxm8fR8hTbo0VSX2+W oRDocscZhWJPBLKxYn5lBn5ZB2YqAYG7vQRPpOWEr8MoRVLqhYDjEAJzqqBjlNiQkh gcoE2W++u8bMDTd3/QECYG3x4dPVjE2OlfDSgl782rl1c7bbV3kmx2mp1FVi5Rr/vK eOewYrvSRxYygENC6IbCw7dXNYtMv/oBLICvNsYDVqRONBPvBaJJWT8gV4ZYmA9Z4C Q4WDteGsrwQgxl6CL1GSLZisEZLmfgX4yuqt1ScTz7eOv1prVATQL/4Lcp4vNWN/I5 Aiu0gNFpiZzvw== From: Hector Martin Date: Thu, 11 Apr 2024 09:51:22 +0900 Subject: [PATCH 3/4] arm64: Introduce scaffolding to add ACTLR_EL1 to thread state MIME-Version: 1.0 Message-Id: <20240411-tso-v1-3-754f11abfbff@marcan.st> References: <20240411-tso-v1-0-754f11abfbff@marcan.st> In-Reply-To: <20240411-tso-v1-0-754f11abfbff@marcan.st> To: Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland Cc: Zayd Qumsieh , Justin Lu , Ryan Houdek , Mark Brown , Ard Biesheuvel , Mateusz Guzik , Anshuman Khandual , Oliver Upton , Miguel Luis , Joey Gouly , Christoph Paasch , Kees Cook , Sami Tolvanen , Baoquan He , Joel Granados , Dawei Li , Andrew Morton , Florent Revest , David Hildenbrand , Stefan Roesch , Andy Chiu , Josh Triplett , Oleg Nesterov , Helge Deller , Zev Weiss , Ondrej Mosnacek , Miguel Ojeda , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Asahi Linux , Hector Martin X-Mailer: b4 0.13.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=4561; i=marcan@marcan.st; h=from:subject:message-id; bh=0K4TZBePvBxEpbZMaDmUuVsuBfSttKVJG4yDVJUb46w=; b=owGbwMvMwCUm+yP4NEe/cRLjabUkhjRxEx6F/QufTKyTf5Q0scmR133RpmfxhySmHREz/Nxdn TJ72+2MjlIWBjEuBlkxRZbGE72nuj2nn1NXTZkOM4eVCWQIAxenAEykT5bhv1c/75ffmS942dRm ZCjZV8/tzJlxLCL/d9VPzpuLr/z7tYGRoUP3lmXFzym1hTP2lD8XrtzBImhu67X9df0GJg+Paqt uXgA= X-Developer-Key: i=marcan@marcan.st; a=openpgp; fpr=FC18F00317968B7BE86201CBE22A629A4C515DD5 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240410_175146_935555_47B5B273 X-CRM114-Status: GOOD ( 17.48 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Some CPUs expose IMPDEF features in ACTLR_EL1 that can be meaningfully controlled per-thread (like TSO control on Apple cores). Add the basic scaffolding to save/restore this register as part of context switching. This mechanism is disabled by default both by config symbol and via a runtime check, which ensures it is never triggered unless the system is known to need it for some feature (which also implies that the layout of ACTLR_EL1 is uniform between all CPU core types). Signed-off-by: Hector Martin --- arch/arm64/Kconfig | 3 +++ arch/arm64/include/asm/cpufeature.h | 5 +++++ arch/arm64/include/asm/processor.h | 3 +++ arch/arm64/kernel/process.c | 25 +++++++++++++++++++++++++ arch/arm64/kernel/setup.c | 8 ++++++++ 5 files changed, 44 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index f8e66fe44ff4..9b3593b34cce 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -408,6 +408,9 @@ config KASAN_SHADOW_OFFSET config UNWIND_TABLES bool +config ARM64_ACTLR_STATE + bool + source "arch/arm64/Kconfig.platforms" menu "Kernel Features" diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index fb215b0e7529..46ab37f8f4d8 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -909,6 +909,11 @@ static inline unsigned int get_vmid_bits(u64 mmfr1) return 8; } +static __always_inline bool system_has_actlr_state(void) +{ + return false; +} + s64 arm64_ftr_safe_value(const struct arm64_ftr_bits *ftrp, s64 new, s64 cur); struct arm64_ftr_reg *get_arm64_ftr_reg(u32 sys_id); diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h index f77371232d8c..d43c5791a35e 100644 --- a/arch/arm64/include/asm/processor.h +++ b/arch/arm64/include/asm/processor.h @@ -184,6 +184,9 @@ struct thread_struct { u64 sctlr_user; u64 svcr; u64 tpidr2_el0; +#ifdef CONFIG_ARM64_ACTLR_STATE + u64 actlr; +#endif }; static inline unsigned int thread_get_vl(struct thread_struct *thread, diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c index 7920056bad3e..117f80e16aac 100644 --- a/arch/arm64/kernel/process.c +++ b/arch/arm64/kernel/process.c @@ -372,6 +372,11 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args) if (system_supports_tpidr2()) p->thread.tpidr2_el0 = read_sysreg_s(SYS_TPIDR2_EL0); +#ifdef CONFIG_ARM64_ACTLR_STATE + if (system_has_actlr_state()) + p->thread.actlr = read_sysreg(actlr_el1); +#endif + if (stack_start) { if (is_compat_thread(task_thread_info(p))) childregs->compat_sp = stack_start; @@ -533,6 +538,25 @@ int arch_prctl_mem_model_set(struct task_struct *t, unsigned long val) } #endif +#ifdef CONFIG_ARM64_ACTLR_STATE +/* + * IMPDEF control register ACTLR_EL1 handling. Some CPUs use this to + * expose features that can be controlled by userspace. + */ +static void actlr_thread_switch(struct task_struct *next) +{ + if (!system_has_actlr_state()) + return; + + current->thread.actlr = read_sysreg(actlr_el1); + write_sysreg(next->thread.actlr, actlr_el1); +} +#else +static inline void actlr_thread_switch(struct task_struct *next) +{ +} +#endif + /* * Thread switching. */ @@ -550,6 +574,7 @@ struct task_struct *__switch_to(struct task_struct *prev, ssbs_thread_switch(next); erratum_1418040_thread_switch(next); ptrauth_thread_switch_user(next); + actlr_thread_switch(next); /* * Complete any pending TLB or cache maintenance on this CPU in case diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c index 65a052bf741f..35342f633a85 100644 --- a/arch/arm64/kernel/setup.c +++ b/arch/arm64/kernel/setup.c @@ -359,6 +359,14 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p) */ init_task.thread_info.ttbr0 = phys_to_ttbr(__pa_symbol(reserved_pg_dir)); #endif +#ifdef CONFIG_ARM64_ACTLR_STATE + /* Store the boot CPU ACTLR_EL1 value as the default. This will only + * be actually restored during context switching iff the platform is + * known to use ACTLR_EL1 for exposable features and its layout is + * known to be the same on all CPUs. + */ + init_task.thread.actlr = read_sysreg(actlr_el1); +#endif if (boot_args[1] || boot_args[2] || boot_args[3]) { pr_err("WARNING: x1-x3 nonzero in violation of boot protocol:\n" From patchwork Thu Apr 11 00:51:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hector Martin X-Patchwork-Id: 13625210 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 93C43CD1297 for ; Thu, 11 Apr 2024 00:52:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:In-Reply-To:References:Message-Id :MIME-Version:Subject:Date:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=OUxnTwh2zsU+/Svs6ZZjeldBb5WiD5zn/bnLGG+cTcQ=; b=Djx0RNoUqh8CRc UZMvH6MKgcZaUA1i3hD0FRBofrVGHuOYOigu5pekDzbL5rsuEiqeHlAiePG00IUV2eIzbeUc0xoG+ R6w7+I5l+60OoKJCy68GdyEmvKZgDbaLoaNBCTjcOAhNhsZg5AaYGTEpCkcSCJUHnYg/OVyz4pTsl QewIQni7BnyY/QdwUY0QqNO4ppw9Prwjm8Zf0efwTsLBimrI9GlhUPo1jTW9+CnvlNjuGllq+n8XB 75Ll2Bs7wKTJX+Pm8jjXFCarhdr8jFeVyv2vvKzNQGo61resE1ub5m+umdHoi4OJiTXad+4L7jMwY wZykBkvZ7iddSLYk6R4w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1ruifY-00000009io4-2CTD; Thu, 11 Apr 2024 00:52:00 +0000 Received: from marcansoft.com ([212.63.210.85] helo=mail.marcansoft.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1ruifP-00000009ii8-1vdg for linux-arm-kernel@lists.infradead.org; Thu, 11 Apr 2024 00:51:53 +0000 Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: sendonly@marcansoft.com) by mail.marcansoft.com (Postfix) with ESMTPSA id A9CB8425C7; Thu, 11 Apr 2024 00:51:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=marcan.st; s=default; t=1712796710; bh=A6cfBeGeoux68NpHgjY1yK+Y/P3gva5wYhlCSyteIQ8=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=hQsQ2KMFH+62Ms4wBxA9kOprBD45PuiudDTCI4Jc4FGX9LpD74wyBTzgEPPVjWUnB 953bY8Dqz9SO3OO4aOQL27CegKMagwfpLuV+N2j2IubpZ7mJZRXeV17q646tvLmMdj AQDO6gvir7cpikZZ9v9b5p9KjuO929lju6gRqXmEl6ayhx28tondrxz9ioiCDG4zG/ k8Tkna5i6wcEZy8+4LyPyH56PrRn/PB64V2/cZ1Ga6JK07yFX/7ipVjJ20wK9jbGED /rxmNI9VBoHwwwL0I2ru1B1ACbf8DbqhzbbOC/lshwf4EIjeUqmSxkMii2YRtjO8Uj BZfDAJl3lm9Rg== From: Hector Martin Date: Thu, 11 Apr 2024 09:51:23 +0900 Subject: [PATCH 4/4] arm64: Implement Apple IMPDEF TSO memory model control MIME-Version: 1.0 Message-Id: <20240411-tso-v1-4-754f11abfbff@marcan.st> References: <20240411-tso-v1-0-754f11abfbff@marcan.st> In-Reply-To: <20240411-tso-v1-0-754f11abfbff@marcan.st> To: Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland Cc: Zayd Qumsieh , Justin Lu , Ryan Houdek , Mark Brown , Ard Biesheuvel , Mateusz Guzik , Anshuman Khandual , Oliver Upton , Miguel Luis , Joey Gouly , Christoph Paasch , Kees Cook , Sami Tolvanen , Baoquan He , Joel Granados , Dawei Li , Andrew Morton , Florent Revest , David Hildenbrand , Stefan Roesch , Andy Chiu , Josh Triplett , Oleg Nesterov , Helge Deller , Zev Weiss , Ondrej Mosnacek , Miguel Ojeda , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Asahi Linux , Hector Martin X-Mailer: b4 0.13.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=5582; i=marcan@marcan.st; h=from:subject:message-id; bh=A6cfBeGeoux68NpHgjY1yK+Y/P3gva5wYhlCSyteIQ8=; b=owGbwMvMwCUm+yP4NEe/cRLjabUkhjRxE94WxUt9IucjBTdtv6v15/UKdQNda43uC39P229ef uim5u6wjlIWBjEuBlkxRZbGE72nuj2nn1NXTZkOM4eVCWQIAxenAEzkhCrDf8/nnCam8mvmbv3j rVJcmMGmb8BzQcSso17uv+gqxuKo9Qz/7O4K+/y0uul+TvFG7uNAH1NVHblvejunNfWVqW+SPqj PBAA= X-Developer-Key: i=marcan@marcan.st; a=openpgp; fpr=FC18F00317968B7BE86201CBE22A629A4C515DD5 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240410_175151_724455_25363E9C X-CRM114-Status: GOOD ( 19.44 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Apple CPUs may implement the TSO memory model as an optional configurable mode. This allows x86 emulators to simplify their load/store handling, greatly increasing performance. Expose this via the prctl PR_SET_MEM_MODEL_TSO mechanism. We use the Apple IMPDEF AIDR_EL1 register to check for the availability of TSO mode, and enable this codepath on all CPUs with an Apple implementer. This relies on the ACTLR_EL1 thread state scaffolding introduced earlier. Signed-off-by: Hector Martin --- arch/arm64/Kconfig | 2 ++ arch/arm64/include/asm/apple_cpufeature.h | 15 +++++++++++++++ arch/arm64/include/asm/cpufeature.h | 3 ++- arch/arm64/kernel/cpufeature_impdef.c | 23 +++++++++++++++++++++++ arch/arm64/kernel/process.c | 22 ++++++++++++++++++++++ arch/arm64/tools/cpucaps | 1 + 6 files changed, 65 insertions(+), 1 deletion(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 9b3593b34cce..2f3eedd955c9 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -2167,6 +2167,8 @@ endif # ARM64_PSEUDO_NMI config ARM64_MEMORY_MODEL_CONTROL bool "Runtime memory model control" + default ARCH_APPLE + select ARM64_ACTLR_STATE help Some ARM64 CPUs support runtime switching of the CPU memory model, which can be useful to emulate other CPU architectures diff --git a/arch/arm64/include/asm/apple_cpufeature.h b/arch/arm64/include/asm/apple_cpufeature.h new file mode 100644 index 000000000000..4370d91ffa3e --- /dev/null +++ b/arch/arm64/include/asm/apple_cpufeature.h @@ -0,0 +1,15 @@ +// SPDX-License-Identifier: GPL-2.0 + +#ifndef __ASM_APPLE_CPUFEATURES_H +#define __ASM_APPLE_CPUFEATURES_H + +#include +#include + +#define AIDR_APPLE_TSO_SHIFT 9 +#define AIDR_APPLE_TSO BIT(9) + +#define ACTLR_APPLE_TSO_SHIFT 1 +#define ACTLR_APPLE_TSO BIT(1) + +#endif diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 46ab37f8f4d8..a191000d88c2 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -911,7 +911,8 @@ static inline unsigned int get_vmid_bits(u64 mmfr1) static __always_inline bool system_has_actlr_state(void) { - return false; + return IS_ENABLED(CONFIG_ARM64_ACTLR_STATE) && + alternative_has_cap_unlikely(ARM64_HAS_TSO_APPLE); } s64 arm64_ftr_safe_value(const struct arm64_ftr_bits *ftrp, s64 new, s64 cur); diff --git a/arch/arm64/kernel/cpufeature_impdef.c b/arch/arm64/kernel/cpufeature_impdef.c index bb04a8e3d79d..9325d1eb12f4 100644 --- a/arch/arm64/kernel/cpufeature_impdef.c +++ b/arch/arm64/kernel/cpufeature_impdef.c @@ -4,8 +4,21 @@ */ #include +#include #ifdef CONFIG_ARM64_MEMORY_MODEL_CONTROL +static bool has_apple_feature(const struct arm64_cpu_capabilities *entry, int scope) +{ + u64 val; + WARN_ON(scope != SCOPE_SYSTEM); + + if (read_cpuid_implementor() != ARM_CPU_IMP_APPLE) + return false; + + val = read_sysreg(aidr_el1); + return cpufeature_matches(val, entry); +} + static bool has_tso_fixed(const struct arm64_cpu_capabilities *entry, int scope) { /* List of CPUs that always use the TSO memory model */ @@ -22,6 +35,16 @@ static bool has_tso_fixed(const struct arm64_cpu_capabilities *entry, int scope) static const struct arm64_cpu_capabilities arm64_impdef_features[] = { #ifdef CONFIG_ARM64_MEMORY_MODEL_CONTROL + { + .desc = "TSO memory model (Apple)", + .capability = ARM64_HAS_TSO_APPLE, + .type = ARM64_CPUCAP_SYSTEM_FEATURE, + .matches = has_apple_feature, + .field_pos = AIDR_APPLE_TSO_SHIFT, + .field_width = 1, + .sign = FTR_UNSIGNED, + .min_field_value = 1, + }, { .desc = "TSO memory model (Fixed)", .capability = ARM64_HAS_TSO_FIXED, diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c index 117f80e16aac..34a19ecfb630 100644 --- a/arch/arm64/kernel/process.c +++ b/arch/arm64/kernel/process.c @@ -44,6 +44,7 @@ #include #include +#include #include #include #include @@ -522,6 +523,10 @@ void update_sctlr_el1(u64 sctlr) #ifdef CONFIG_ARM64_MEMORY_MODEL_CONTROL int arch_prctl_mem_model_get(struct task_struct *t) { + if (alternative_has_cap_unlikely(ARM64_HAS_TSO_APPLE) && + t->thread.actlr & ACTLR_APPLE_TSO) + return PR_SET_MEM_MODEL_TSO; + return PR_SET_MEM_MODEL_DEFAULT; } @@ -531,6 +536,23 @@ int arch_prctl_mem_model_set(struct task_struct *t, unsigned long val) val == PR_SET_MEM_MODEL_TSO) return 0; + if (alternative_has_cap_unlikely(ARM64_HAS_TSO_APPLE)) { + WARN_ON(!system_has_actlr_state()); + + switch (val) { + case PR_SET_MEM_MODEL_TSO: + t->thread.actlr |= ACTLR_APPLE_TSO; + break; + case PR_SET_MEM_MODEL_DEFAULT: + t->thread.actlr &= ~ACTLR_APPLE_TSO; + break; + default: + return -EINVAL; + } + write_sysreg(t->thread.actlr, actlr_el1); + return 0; + } + if (val == PR_SET_MEM_MODEL_DEFAULT) return 0; diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps index daa6b9495402..62f9ca9ce44b 100644 --- a/arch/arm64/tools/cpucaps +++ b/arch/arm64/tools/cpucaps @@ -52,6 +52,7 @@ HAS_STAGE2_FWB HAS_TCR2 HAS_TIDCP1 HAS_TLB_RANGE +HAS_TSO_APPLE HAS_TSO_FIXED HAS_VA52 HAS_VIRT_HOST_EXTN