From patchwork Thu Aug 22 01:15:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 13772324 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 16A8AC52D6F for ; Thu, 22 Aug 2024 01:20:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9BA0A6B01F2; Wed, 21 Aug 2024 21:20:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9436D6B01F3; Wed, 21 Aug 2024 21:20:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7E4636B01F5; Wed, 21 Aug 2024 21:20:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 5D3526B01F2 for ; Wed, 21 Aug 2024 21:20:23 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 19AE7A7ED5 for ; Thu, 22 Aug 2024 01:20:23 +0000 (UTC) X-FDA: 82478125926.08.9768A34 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf12.hostedemail.com (Postfix) with ESMTP id 3B44F40003 for ; Thu, 22 Aug 2024 01:20:21 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=nwCz74RQ; spf=pass (imf12.hostedemail.com: domain of broonie@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=broonie@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724289540; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=iagVRYloCKIO0s2YwTVqShvpKyhoVqU0Akb7gWmT8wc=; b=uEefkXi3uO4MbsofkTgInoGgLZsG65MDj6iDrNvs0AwJjGo0m/RCyWMYb1QDyFL7/beuND X+Y1bwSzXmndASWRkWtEqwEkZy9I+SW/xLTp6gyaIo/dOSZLIYG4MEwUA3BnIle+j9+lif aVqeiSQ4usPdZe6PpqbnA3lm6K/wNcE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724289540; a=rsa-sha256; cv=none; b=Ml28nNfMTYsgCeJ4sJ6wKjxcnVnL78yu76j+atp+1VxEd+yjlETK4xqw3ZkTZuOQ9ed8r/ Gr+TZv9uNmJgUzRPbx6bZb1/cTlQSzgjBQoVei/p6bt1uUU1LNuRMlUvQXvKjrex2cKzRm Tpb1cw+wAOv7ZdgZ0fPW0wcmkxxLPMg= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=nwCz74RQ; spf=pass (imf12.hostedemail.com: domain of broonie@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=broonie@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 83FC160FE8; Thu, 22 Aug 2024 01:20:20 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id BD8C0C4AF0E; Thu, 22 Aug 2024 01:20:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1724289620; bh=RTGexDthYFr44xiqo+HLLY4BIMEvafRxD0gCV0rtMNw=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=nwCz74RQhv+6TSyNSnHLKjoCcoQq/n5I2qJ7ODUhcMUDknH3RoZx3+U53CZh4omCs pHlpjBu0mzP233pTtuA7VB8xr3tC1F27PU41ouRkj4j0YILmrAQYhvH+IQ/AIpiLWd yCrmS9Z173ji2C5qYBJAbIhFopDTAuft2OsfOTmMii4czZZwqSaGbyPc8M86YuV6GQ 8zg+wDQ3uvJZln9CEmmGwImupKGRxJGVCoDdmoejHCvGFxEaPzFSJN8e5UW4uQDQ6M Q/AtHRm8I3sty4ur4tY0HyZY3/wRv+xMvj5eiMth+Cly3U10CUC4hdrs5gCG4ApZzr IuTJr0ojQEVZQ== From: Mark Brown Date: Thu, 22 Aug 2024 02:15:24 +0100 Subject: [PATCH v11 21/39] arm64/gcs: Ensure that new threads have a GCS MIME-Version: 1.0 Message-Id: <20240822-arm64-gcs-v11-21-41b81947ecb5@kernel.org> References: <20240822-arm64-gcs-v11-0-41b81947ecb5@kernel.org> In-Reply-To: <20240822-arm64-gcs-v11-0-41b81947ecb5@kernel.org> To: Catalin Marinas , Will Deacon , Jonathan Corbet , Andrew Morton , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Arnd Bergmann , Oleg Nesterov , Eric Biederman , Shuah Khan , "Rick P. Edgecombe" , Deepak Gupta , Ard Biesheuvel , Szabolcs Nagy , Kees Cook Cc: "H.J. Lu" , Paul Walmsley , Palmer Dabbelt , Albert Ou , Florian Weimer , Christian Brauner , Thiago Jung Bauermann , Ross Burton , Yury Khrustalev , Wilco Dijkstra , linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Mark Brown X-Mailer: b4 0.15-dev-37811 X-Developer-Signature: v=1; a=openpgp-sha256; l=7114; i=broonie@kernel.org; h=from:subject:message-id; bh=RTGexDthYFr44xiqo+HLLY4BIMEvafRxD0gCV0rtMNw=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBmxpE0o1U0PhWyG7IJgmly5/ucjchbqyop+pwJgqc7 4cSP+qqJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCZsaRNAAKCRAk1otyXVSH0GMWB/ 9WICAq2SrwWqJTnYUuIjxl/qBe8At86oZHK+3XOdhOpLoKTmzyB8P3xYichKFWIDihs88bgvPdnzb/ FfVvXD4FUGaGPV2TvCx00aqavPExQC0TyVx2NX7eifWDcXNffy+PKHhZR7gC6bc8c9WqM0ugq9u9pO e0JljMz8HcjhiWnbNeiljOkYMmgOctlrBw3Jh0Shk2Jj4VvqD4oT3op9HIjbs5SVQkywjJN/Pw9YlJ HNCNVkWSzCREK3hYPv+QHMwbDWiN6dd2sfNJikCFMXzNeOB8VThNEaFYv8+d1jvmHBjn3muzY9RNzF MkKF+6jecDOjzR0lPpZOf1Su28jYgF X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB X-Stat-Signature: ruiqa97o9grc3echjtixr7qeggonr1m4 X-Rspamd-Queue-Id: 3B44F40003 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1724289621-247492 X-HE-Meta: U2FsdGVkX1/jtq/QFlTSI/CH0FlTHHCcx0/HngwFtkBvaiIkn5X1bCh+hVGRSpbItX/AeJFvEKUeQji3EsAgfXcqx5bEuhLj+PH5z0JOL0GeQk3n1/4KbT6m+M/FD/LTKUKYL6nT+I+OQuTm9Tif8aK7ctjyyO9rTqQpvbm4COBm39z/l1Rlp3SO1QRBUMKFPvXVMeFfGBuf3WXQib1dxxNmcu2pETblnF0Z8p77mfFEDQnI/JqBuK2vJvnVZ65z/YSYsHUjkwwOYgpxy/H3mvURzpPCinYGuC7h+i529RZZFokvZGN3D6gJDQ4vAGZUyr4jZ49xhzMYKTBf2JASd9TZ7rT8UUUDO0/t3pGqOnqXRC+4k+FnSSVLt9Gscr5J9WlZtOs5xweS+nKQypKCXo1h1MFUcQ0NeXCcJwhisdDnw2RN4kifjR/XCZASSWYP6CGxGrUtaO3VjSyeVF619rD7nc2XbvHNsyCIc7zwHWqmye3i/+PudTIPmo7462kBMZxfnYbjmh16eX65Hsu6O3teoERIAtvg9bD3SU68rspEe0aZi5h0gWT6pl5ev6OecZWAlxv9jvZRsXpvBxYuH7Bd4FiMls8iJOLDznou/Ql4yMmlbvusIXM7KZ0LE8CKf7YJ4y2bitJyTif2KVt3T9oXUuev76UesuBuTAez1AXBAV58Ch8v4E1NtJ5fxaR0zuOrH8RFsuhO0Q0IvCXRaLnAw+GNZ3d8Ik9LHsihKXyQh0C7Yutzu5jynEkXNoaZQSZuCUQAvCotXCzY4H+jyY2fZh27n1ICL4njjM/fVjNHzJDfgdhLtQSFw9kSpmXKRM1WwEYCIuQDP6vKUXsrRo0U6WqVC9Kwd0p6jEfhK2iDTDW/HZa3jYtR+Oe0gyRhzDQoLJ1M8fPlmli4l24ykFr/G6rRa8j2KfrwNXcxN4Ptn1RGeY3lOiRKlYemxys3mKOPBpvnNIVGABuiun1 z5w1lO5B kyHHkWXjpZRzolUrke3ohGO94F5UVvWJgIAEUT8qlmlLiQvN31Z/ml4RDaBJbjJ/M2m3BLbF3M7W7WhuSYnkXtBR3wfLnyEs/diU9MZMy6P+GPRsNfr1PA+kyOEi1sXwjXoSTNGx+X6awRctUH+oezxDXmHs3CTX7DccLbSnKhWnJ70WRhrPyBiOdmd4NSqWCpIJ7MMjZF8I/kygKC5bN2DVOZN4U6Oo1heFZHG39M+5HS6OPaCFNecqeIxT6nv9LjoNoZhyZwWBchycdiVtbfux1MXzIBg9EJvreon88kqYwZViDXK9SYWZ4cJ3hTAxFd8iXKfpdnfocJEQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When a new thread is created by a thread with GCS enabled the GCS needs to be specified along with the regular stack. Unfortunately plain clone() is not extensible and existing clone3() users will not specify a stack so all existing code would be broken if we mandated specifying the stack explicitly. For compatibility with these cases and also x86 (which did not initially implement clone3() support for shadow stacks) if no GCS is specified we will allocate one so when a thread is created which has GCS enabled allocate one for it. We follow the extensively discussed x86 implementation and allocate min(RLIMIT_STACK, 2G). Since the GCS only stores the call stack and not any variables this should be more than sufficient for most applications. GCSs allocated via this mechanism will be freed when the thread exits. Signed-off-by: Mark Brown Reviewed-by: Catalin Marinas --- arch/arm64/include/asm/gcs.h | 9 ++++++ arch/arm64/kernel/process.c | 38 ++++++++++++++++++++++-- arch/arm64/mm/gcs.c | 69 ++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 113 insertions(+), 3 deletions(-) diff --git a/arch/arm64/include/asm/gcs.h b/arch/arm64/include/asm/gcs.h index 04594ef59dad..c1f274fdb9c0 100644 --- a/arch/arm64/include/asm/gcs.h +++ b/arch/arm64/include/asm/gcs.h @@ -8,6 +8,8 @@ #include #include +struct kernel_clone_args; + static inline void gcsb_dsync(void) { asm volatile(".inst 0xd503227f" : : : "memory"); @@ -58,6 +60,8 @@ static inline bool task_gcs_el0_enabled(struct task_struct *task) void gcs_set_el0_mode(struct task_struct *task); void gcs_free(struct task_struct *task); void gcs_preserve_current_state(void); +unsigned long gcs_alloc_thread_stack(struct task_struct *tsk, + const struct kernel_clone_args *args); #else @@ -69,6 +73,11 @@ static inline bool task_gcs_el0_enabled(struct task_struct *task) static inline void gcs_set_el0_mode(struct task_struct *task) { } static inline void gcs_free(struct task_struct *task) { } static inline void gcs_preserve_current_state(void) { } +static inline unsigned long gcs_alloc_thread_stack(struct task_struct *tsk, + const struct kernel_clone_args *args) +{ + return -ENOTSUPP; +} #endif diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c index a4fd25585801..de59aa16919c 100644 --- a/arch/arm64/kernel/process.c +++ b/arch/arm64/kernel/process.c @@ -285,9 +285,29 @@ static void flush_gcs(void) write_sysreg_s(0, SYS_GCSPR_EL0); } +static int copy_thread_gcs(struct task_struct *p, + const struct kernel_clone_args *args) +{ + unsigned long gcs; + + gcs = gcs_alloc_thread_stack(p, args); + if (IS_ERR_VALUE(gcs)) + return PTR_ERR((void *)gcs); + + p->thread.gcs_el0_mode = current->thread.gcs_el0_mode; + p->thread.gcs_el0_locked = current->thread.gcs_el0_locked; + + return 0; +} + #else static void flush_gcs(void) { } +static int copy_thread_gcs(struct task_struct *p, + const struct kernel_clone_args *args) +{ + return 0; +} #endif @@ -303,6 +323,7 @@ void flush_thread(void) void arch_release_task_struct(struct task_struct *tsk) { fpsimd_release_task(tsk); + gcs_free(tsk); } int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) @@ -366,6 +387,7 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args) unsigned long stack_start = args->stack; unsigned long tls = args->tls; struct pt_regs *childregs = task_pt_regs(p); + int ret; memset(&p->thread.cpu_context, 0, sizeof(struct cpu_context)); @@ -407,6 +429,10 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args) p->thread.uw.tp_value = tls; p->thread.tpidr2_el0 = 0; } + + ret = copy_thread_gcs(p, args); + if (ret != 0) + return ret; } else { /* * A kthread has no context to ERET to, so ensure any buggy @@ -512,10 +538,16 @@ static void gcs_thread_switch(struct task_struct *next) gcs_set_el0_mode(next); /* - * Ensure that GCS changes are observable by/from other PEs in - * case of migration. + * Ensure that GCS memory effects of the 'prev' thread are + * ordered before other memory accesses with release semantics + * (or preceded by a DMB) on the current PE. In addition, any + * memory accesses with acquire semantics (or succeeded by a + * DMB) are ordered before GCS memory effects of the 'next' + * thread. This will ensure that the GCS memory effects are + * visible to other PEs in case of migration. */ - gcsb_dsync(); + if (task_gcs_el0_enabled(current) || task_gcs_el0_enabled(next)) + gcsb_dsync(); } #else diff --git a/arch/arm64/mm/gcs.c b/arch/arm64/mm/gcs.c index b0a67efc522b..adedbfc062f3 100644 --- a/arch/arm64/mm/gcs.c +++ b/arch/arm64/mm/gcs.c @@ -5,9 +5,68 @@ #include #include +#include #include #include +static unsigned long alloc_gcs(unsigned long addr, unsigned long size) +{ + int flags = MAP_ANONYMOUS | MAP_PRIVATE; + struct mm_struct *mm = current->mm; + unsigned long mapped_addr, unused; + + if (addr) + flags |= MAP_FIXED_NOREPLACE; + + mmap_write_lock(mm); + mapped_addr = do_mmap(NULL, addr, size, PROT_READ, flags, + VM_SHADOW_STACK | VM_WRITE, 0, &unused, NULL); + mmap_write_unlock(mm); + + return mapped_addr; +} + +static unsigned long gcs_size(unsigned long size) +{ + if (size) + return PAGE_ALIGN(size); + + /* Allocate RLIMIT_STACK/2 with limits of PAGE_SIZE..2G */ + size = PAGE_ALIGN(min_t(unsigned long long, + rlimit(RLIMIT_STACK) / 2, SZ_2G)); + return max(PAGE_SIZE, size); +} + +unsigned long gcs_alloc_thread_stack(struct task_struct *tsk, + const struct kernel_clone_args *args) +{ + unsigned long addr, size; + + if (!system_supports_gcs()) + return 0; + + if (!task_gcs_el0_enabled(tsk)) + return 0; + + if ((args->flags & (CLONE_VFORK | CLONE_VM)) != CLONE_VM) { + tsk->thread.gcspr_el0 = read_sysreg_s(SYS_GCSPR_EL0); + return 0; + } + + size = args->stack_size; + + size = gcs_size(size); + addr = alloc_gcs(0, size); + if (IS_ERR_VALUE(addr)) + return addr; + + tsk->thread.gcs_base = addr; + tsk->thread.gcs_size = size; + tsk->thread.gcspr_el0 = addr + size - sizeof(u64); + + return addr; +} + /* * Apply the GCS mode configured for the specified task to the * hardware. @@ -30,6 +89,16 @@ void gcs_set_el0_mode(struct task_struct *task) void gcs_free(struct task_struct *task) { + + /* + * When fork() with CLONE_VM fails, the child (tsk) already + * has a GCS allocated, and exit_thread() calls this function + * to free it. In this case the parent (current) and the + * child share the same mm struct. + */ + if (!task->mm || task->mm != current->mm) + return; + if (task->thread.gcs_base) vm_munmap(task->thread.gcs_base, task->thread.gcs_size);