From patchwork Thu Aug 22 01:15:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 13776850 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2C6A6C5320E for ; Sun, 25 Aug 2024 18:06:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A75958D0019; Sun, 25 Aug 2024 14:06:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9F1A68D0018; Sun, 25 Aug 2024 14:06:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8279C8D001B; Sun, 25 Aug 2024 14:06:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 62F8E8D0016 for ; Sun, 25 Aug 2024 14:06:50 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 05CFBA07E5 for ; Sun, 25 Aug 2024 18:06:50 +0000 (UTC) X-FDA: 82491548580.02.5FA846F Received: from nyc.source.kernel.org (nyc.source.kernel.org [147.75.193.91]) by imf27.hostedemail.com (Postfix) with ESMTP id 4E04B40007 for ; Sun, 25 Aug 2024 18:06:48 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=spIIBu4G; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf27.hostedemail.com: domain of broonie@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=broonie@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724609113; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=aoTcPxl0VwQ++6GXGbVHutXxCUza0K7v10lQpUDcA4I=; b=tVMiZ8afWrxCbGsGbcn4qyZxTrbwueJVnwqctpWiQZk9iOsUue+aritWjZpaTfmkL/0Ajf W6NgX1jA4i2RNTFDThaMeBwtYGcHqmsUQS+WBMX43OQrwD/FQSdOqfjj3y8BZsD4N/UE9J ihE9352+6c1wuptXu79dLZlD3EuKM1k= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724609113; a=rsa-sha256; cv=none; b=K4uPhFk4QOTrQS0qJ2nQrTeSxmh7SvzRUQebta0s/tRk29SosiX3FLHZdf/n56R2ScAKDl MAd6U5TfRGnh5XjtJ+JX1E3mpUfujlSxKKUQSDVaIVOAvnxlVMeWBcFf7HziHmYrn83J8J PZ2vTDfuaoEixhM8uJm+yi6/uDM/fiI= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=spIIBu4G; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf27.hostedemail.com: domain of broonie@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=broonie@kernel.org Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id C83AEA42111; Thu, 22 Aug 2024 01:20:01 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 81CF3C4AF09; Thu, 22 Aug 2024 01:19:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1724289608; bh=ahP+LTv7b4XthpsSBycFF0WuUJionb06ERHs9uQMqB0=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=spIIBu4GeSPYVLSchl3Yy+CPSYnnh/AGFV9P30yga1/EwkCOvuGXE/lRU3Idarzsn ldpc9kbY/fYBlkZX7edjPkGY7Wut4tpr1xmRr2QHhF6vMuMsBx49HqGxCRcnyuFpJr xpKbjEu6xpDic42aTtPKuWCVXYuQ/imk5MYCoNpwGVTojGqYBblO25LMxJ6piXmY3i aRLW3AUUhzIk2g3XoXc9ZWzXwRdEhvEsj8G+43a5VVh8biGOzNzX8ts4jdM58xBKWL BPt9Q6X3ZPeiLSOjDYB34uin93NvXR43LI7WIjSwkWqIjCfX+ZtJiV5fQFmONqAUi3 i3rXPWmu/5lnA== From: Mark Brown Date: Thu, 22 Aug 2024 02:15:23 +0100 Subject: [PATCH v11 20/39] arm64/gcs: Context switch GCS state for EL0 MIME-Version: 1.0 Message-Id: <20240822-arm64-gcs-v11-20-41b81947ecb5@kernel.org> References: <20240822-arm64-gcs-v11-0-41b81947ecb5@kernel.org> In-Reply-To: <20240822-arm64-gcs-v11-0-41b81947ecb5@kernel.org> To: Catalin Marinas , Will Deacon , Jonathan Corbet , Andrew Morton , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Arnd Bergmann , Oleg Nesterov , Eric Biederman , Shuah Khan , "Rick P. Edgecombe" , Deepak Gupta , Ard Biesheuvel , Szabolcs Nagy , Kees Cook Cc: "H.J. Lu" , Paul Walmsley , Palmer Dabbelt , Albert Ou , Florian Weimer , Christian Brauner , Thiago Jung Bauermann , Ross Burton , Yury Khrustalev , Wilco Dijkstra , linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Mark Brown X-Mailer: b4 0.15-dev-37811 X-Developer-Signature: v=1; a=openpgp-sha256; l=6696; i=broonie@kernel.org; h=from:subject:message-id; bh=ahP+LTv7b4XthpsSBycFF0WuUJionb06ERHs9uQMqB0=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBmxpEzoEVzAmKLLFWRLpMs9clmoSlI6B8aerT4lEDr HEQHpGOJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCZsaRMwAKCRAk1otyXVSH0N89B/ wKEtFu6hbp0q1zALn5n+QKZjuVKSfBNrqE0RrWOFE8Z94hdCTI+y//U6UsFa4TiqxX4xr+l4twYbOM //DvZO4MQzswfdnT0o8G15wME25L/bOuhxVXEE7PjS3mTXqq74+2piWjX79jTaWHSTB0TbhjzVWkny S37hqjsYYpPRYxQCd+oE2zZRfto8VOhPQHhaoKo3TjL+H0tph2KAdYrFWXwV/dCVOeDaceRy8dgwxs zZoDabYZ6FEQqOWecUbQrAPkPXIpztAdOFFZW5wCQa4I7XMN54YRtPAiKyHAHCC/0ExuzI6eSO03/y waC60FONbYCCPGLAkhiTufm08+rOWq X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 4E04B40007 X-Stat-Signature: ex8ubqgiwanz83r8f8j9tzqqo7pn86dt X-Rspam-User: X-HE-Tag: 1724609208-215515 X-HE-Meta: U2FsdGVkX19pQ9OYz4UQXJBuLWkf18JF+RzlfmpW2ht7SpY088JNfIf39sarqIrmNrn/izGHwq1xYsID3ZCSuPprxxQNZ7/5Rs06KvBinWCLAWiMG7vQVuqE1E7t9O5rZmY8yldw1zO04ee+jreGrPAQ6QQ53O/nY1u0zvfQVrk2dBevyJWGEaw3hHStUyTAfJ/yMPnd1ay2tFR1P9fEH1i2uLfNISJgD3pkTfYNYzSo61u6Y/IUWuQf0pFZBcYJ+q2sXndlEBCwMSozj3nRa3cyPmvFuJW0h2WKYa1QkhIYP4AZ99V2XMyDTYzVP+qmkVNlT/U66l+ADlrnwixUjx7D0Up3Z6musjO4kHKoH7lQZexeIMp1prQojKKdlA1xG9NlhOFHrFs1iYiAkNZRJ0puzhDJTL7e8asWpr3UP0Jb6OZZRs2PV6JtksHPiLzzyVmnkbg+neA8K6zlaaCzRehKGi2lTkBTjbCZeXlIYgKgLGys9eNZWz41MSTPSoOGYCkQ9Pt5IRCR0bWTMcZdvVYfBCvCSuP/MDfOWJyvqabvVMgF4hkcOo3VBzG/1Dis/5O+xicIx25HP0L+GKkjJ7ih2Ze+lZIgBOo7iJo5a6IgR8upiEqVy+HGIm+HgnpZxz7MF3ODb6zZiJ9g9bHxDtMSy5lALDxWrEJPQZKcszpol+9nDB8o9FvRhjnQPeyxoB+gRYn0rOEmqE9M8gqSJrN5Xz6cJDGRDFL+3V/5scSgLMy5bAOJk/mCkRtgYyHlrdGFzNz4BPIUCva0kwU8XIRxU6h46Onjhl2tdrzhAGXyYdr8S1A9lEQS+1HYQVew7j1Od3lnwlbDFrlU72DaCc/nqnyBy1dY58QzpbCYNkMsg+VRpjyJ9YOQhjM2Kn/iOx5/1cGKwqe3/aw++idpOwOFrBoVauN+hfGGROYbUoJ5DmQwUTPfnYVJaMId0GY+QRzYQCA9ghWHUh9uoSi dFpu7Hjl Tz2FYU2VC9eytusrB7RH2JcKh6PdUv9o3reqz9Fj+f0164A3JLB21PNMXfnTFqjcKkxUxNI7pD5lFXKYCglZYFqRgvcwosXmZ6VMm/Qw1KagY6HC73Z847LM7nkmWt1T5mO59AWADpeU3n5wmQZK6flT7XwcsblmnN3B2opAHGlXhvWfqg6X4khcnTtbQlv+xce8CEbxPeYDUkJIn5ejDnwsf2c4AE74Go7Y64//gumWjxQP79GeCyJ14hjEWWQKNmc/mhWp2bI4z4C2tFUQmns4szmpK4YmY5H28p+tgjFntEuisjCE6GJhT0WNxWCI8c33P47lIwfMc1PThWIpBC4fhJr/LjBabxQz5LGkvZvFUdnQbSeubnN8crHLoSBAQaOevTH7I8pSu/p95FYaj0yH1pQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: There are two registers controlling the GCS state of EL0, GCSPR_EL0 which is the current GCS pointer and GCSCRE0_EL1 which has enable bits for the specific GCS functionality enabled for EL0. Manage these on context switch and process lifetime events, GCS is reset on exec(). Also ensure that any changes to the GCS memory are visible to other PEs and that changes from other PEs are visible on this one by issuing a GCSB DSYNC when moving to or from a thread with GCS. Since the current GCS configuration of a thread will be visible to userspace we store the configuration in the format used with userspace and provide a helper which configures the system register as needed. On systems that support GCS we always allow access to GCSPR_EL0, this facilitates reporting of GCS faults if userspace implements disabling of GCS on error - the GCS can still be discovered and examined even if GCS has been disabled. Signed-off-by: Mark Brown --- arch/arm64/include/asm/gcs.h | 24 ++++++++++++++++ arch/arm64/include/asm/processor.h | 6 ++++ arch/arm64/kernel/process.c | 56 ++++++++++++++++++++++++++++++++++++++ arch/arm64/mm/Makefile | 1 + arch/arm64/mm/gcs.c | 39 ++++++++++++++++++++++++++ 5 files changed, 126 insertions(+) diff --git a/arch/arm64/include/asm/gcs.h b/arch/arm64/include/asm/gcs.h index 7c5e95218db6..04594ef59dad 100644 --- a/arch/arm64/include/asm/gcs.h +++ b/arch/arm64/include/asm/gcs.h @@ -48,4 +48,28 @@ static inline u64 gcsss2(void) return Xt; } +#ifdef CONFIG_ARM64_GCS + +static inline bool task_gcs_el0_enabled(struct task_struct *task) +{ + return current->thread.gcs_el0_mode & PR_SHADOW_STACK_ENABLE; +} + +void gcs_set_el0_mode(struct task_struct *task); +void gcs_free(struct task_struct *task); +void gcs_preserve_current_state(void); + +#else + +static inline bool task_gcs_el0_enabled(struct task_struct *task) +{ + return false; +} + +static inline void gcs_set_el0_mode(struct task_struct *task) { } +static inline void gcs_free(struct task_struct *task) { } +static inline void gcs_preserve_current_state(void) { } + +#endif + #endif diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h index f77371232d8c..c55e3600604a 100644 --- a/arch/arm64/include/asm/processor.h +++ b/arch/arm64/include/asm/processor.h @@ -184,6 +184,12 @@ struct thread_struct { u64 sctlr_user; u64 svcr; u64 tpidr2_el0; +#ifdef CONFIG_ARM64_GCS + unsigned int gcs_el0_mode; + u64 gcspr_el0; + u64 gcs_base; + u64 gcs_size; +#endif }; static inline unsigned int thread_get_vl(struct thread_struct *thread, diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c index 4ae31b7af6c3..a4fd25585801 100644 --- a/arch/arm64/kernel/process.c +++ b/arch/arm64/kernel/process.c @@ -48,6 +48,7 @@ #include #include #include +#include #include #include #include @@ -271,12 +272,32 @@ static void flush_tagged_addr_state(void) clear_thread_flag(TIF_TAGGED_ADDR); } +#ifdef CONFIG_ARM64_GCS + +static void flush_gcs(void) +{ + if (!system_supports_gcs()) + return; + + gcs_free(current); + current->thread.gcs_el0_mode = 0; + write_sysreg_s(0, SYS_GCSCRE0_EL1); + write_sysreg_s(0, SYS_GCSPR_EL0); +} + +#else + +static void flush_gcs(void) { } + +#endif + void flush_thread(void) { fpsimd_flush_thread(); tls_thread_flush(); flush_ptrace_hw_breakpoint(current); flush_tagged_addr_state(); + flush_gcs(); } void arch_release_task_struct(struct task_struct *tsk) @@ -471,6 +492,40 @@ static void entry_task_switch(struct task_struct *next) __this_cpu_write(__entry_task, next); } +#ifdef CONFIG_ARM64_GCS + +void gcs_preserve_current_state(void) +{ + current->thread.gcspr_el0 = read_sysreg_s(SYS_GCSPR_EL0); +} + +static void gcs_thread_switch(struct task_struct *next) +{ + if (!system_supports_gcs()) + return; + + /* GCSPR_EL0 is always readable */ + gcs_preserve_current_state(); + write_sysreg_s(next->thread.gcspr_el0, SYS_GCSPR_EL0); + + if (current->thread.gcs_el0_mode != next->thread.gcs_el0_mode) + gcs_set_el0_mode(next); + + /* + * Ensure that GCS changes are observable by/from other PEs in + * case of migration. + */ + gcsb_dsync(); +} + +#else + +static void gcs_thread_switch(struct task_struct *next) +{ +} + +#endif + /* * ARM erratum 1418040 handling, affecting the 32bit view of CNTVCT. * Ensure access is disabled when switching to a 32bit task, ensure @@ -530,6 +585,7 @@ struct task_struct *__switch_to(struct task_struct *prev, ssbs_thread_switch(next); erratum_1418040_thread_switch(next); ptrauth_thread_switch_user(next); + gcs_thread_switch(next); /* * Complete any pending TLB or cache maintenance on this CPU in case diff --git a/arch/arm64/mm/Makefile b/arch/arm64/mm/Makefile index 60454256945b..1a7b3a2f21e6 100644 --- a/arch/arm64/mm/Makefile +++ b/arch/arm64/mm/Makefile @@ -11,6 +11,7 @@ obj-$(CONFIG_TRANS_TABLE) += trans_pgd.o obj-$(CONFIG_TRANS_TABLE) += trans_pgd-asm.o obj-$(CONFIG_DEBUG_VIRTUAL) += physaddr.o obj-$(CONFIG_ARM64_MTE) += mteswap.o +obj-$(CONFIG_ARM64_GCS) += gcs.o KASAN_SANITIZE_physaddr.o += n obj-$(CONFIG_KASAN) += kasan_init.o diff --git a/arch/arm64/mm/gcs.c b/arch/arm64/mm/gcs.c new file mode 100644 index 000000000000..b0a67efc522b --- /dev/null +++ b/arch/arm64/mm/gcs.c @@ -0,0 +1,39 @@ +// SPDX-License-Identifier: GPL-2.0-only + +#include +#include +#include +#include + +#include +#include + +/* + * Apply the GCS mode configured for the specified task to the + * hardware. + */ +void gcs_set_el0_mode(struct task_struct *task) +{ + u64 gcscre0_el1 = GCSCRE0_EL1_nTR; + + if (task->thread.gcs_el0_mode & PR_SHADOW_STACK_ENABLE) + gcscre0_el1 |= GCSCRE0_EL1_RVCHKEN | GCSCRE0_EL1_PCRSEL; + + if (task->thread.gcs_el0_mode & PR_SHADOW_STACK_WRITE) + gcscre0_el1 |= GCSCRE0_EL1_STREn; + + if (task->thread.gcs_el0_mode & PR_SHADOW_STACK_PUSH) + gcscre0_el1 |= GCSCRE0_EL1_PUSHMEn; + + write_sysreg_s(gcscre0_el1, SYS_GCSCRE0_EL1); +} + +void gcs_free(struct task_struct *task) +{ + if (task->thread.gcs_base) + vm_munmap(task->thread.gcs_base, task->thread.gcs_size); + + task->thread.gcspr_el0 = 0; + task->thread.gcs_base = 0; + task->thread.gcs_size = 0; +}