From patchwork Thu Aug 22 01:15:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 13776852 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1BF9CC5321E for ; Sun, 25 Aug 2024 18:06:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 12C168D0018; Sun, 25 Aug 2024 14:06:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0A9D68D001B; Sun, 25 Aug 2024 14:06:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A665B8D001B; Sun, 25 Aug 2024 14:06:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 6FC998D0019 for ; Sun, 25 Aug 2024 14:06:50 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 1A56C1407FF for ; Sun, 25 Aug 2024 18:06:50 +0000 (UTC) X-FDA: 82491548580.24.3A1DAE5 Received: from nyc.source.kernel.org (nyc.source.kernel.org [147.75.193.91]) by imf29.hostedemail.com (Postfix) with ESMTP id 58D7A12000A for ; Sun, 25 Aug 2024 18:06:48 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=SKAPNy3r; spf=pass (imf29.hostedemail.com: domain of broonie@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=broonie@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724609095; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1zhSjceydU0lZvoQ55BhA1CSLcr+UutwwGTWuNFjHhY=; b=5V9zId9In1gSYXBhcc35gcadXKZziGjEM1CxG1mbdUGiyvH1PKETLRpNr9vO8DbjaVNW89 vtK/93D7iGLpGxL4X9vbIwIa2mxQaqPJ0WRhioj/uYMvfdrREqlfWTQvPqZdS05RFVHweo pmr7yBLrLRmCW8vurwd5bijf+aSWVIU= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=SKAPNy3r; spf=pass (imf29.hostedemail.com: domain of broonie@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=broonie@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724609095; a=rsa-sha256; cv=none; b=hwTrfxwU6v29O+rZms51v5fbp/UiAfURG68dJ0ogyJd9GepOI/CTWRIUm8NEkFkIOCJn3F GyAD9jpqQwjQkttFhoSzhsqQOrzoHJyzG5ghrQET2fjC4206T6eVxPYn+wK/3eSC3MsZGS lVSUATZHs8fPHtpprFiSfiyuEFmhL2Y= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id 8F6AAA4210F; Thu, 22 Aug 2024 01:19:50 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0594FC32781; Thu, 22 Aug 2024 01:19:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1724289596; bh=is2zgCRaR9rKZObHv5ljdEWRzYPoLx4wqWI+Auj5mNM=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=SKAPNy3rZqeWeOli+lFlVF+9Ltr6/Ptv46pvxt2SvY7N2+CL2Wsfw0kIYURI3MlWO h6ysmRaeEAz0QZaCLfVIEmy7IVq33c5/7zbnlkh3OHBsMhIBBXbGqIUDyWK61HNufB Pxe2zBJPGd6Jpr0TSEaC7Yy0QBLrATRNE22pDDxydTUY5wpRR4PDsFahkUr6euYtsY gy8Z0mv6ik1ATXuMU5+b0/vNqexzketHKOcKvai55sRSYu0PUBIoZf4P3xBnImyAT/ SS546dZuFYXDITKRB0eTCwf8NG6LwgZ8IYnTxsv6V4OIEZBRJA6wC8GzDcI1IVvc7J b7U4dzeu9d9TQ== From: Mark Brown Date: Thu, 22 Aug 2024 02:15:22 +0100 Subject: [PATCH v11 19/39] arm64/mm: Handle GCS data aborts MIME-Version: 1.0 Message-Id: <20240822-arm64-gcs-v11-19-41b81947ecb5@kernel.org> References: <20240822-arm64-gcs-v11-0-41b81947ecb5@kernel.org> In-Reply-To: <20240822-arm64-gcs-v11-0-41b81947ecb5@kernel.org> To: Catalin Marinas , Will Deacon , Jonathan Corbet , Andrew Morton , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Arnd Bergmann , Oleg Nesterov , Eric Biederman , Shuah Khan , "Rick P. Edgecombe" , Deepak Gupta , Ard Biesheuvel , Szabolcs Nagy , Kees Cook Cc: "H.J. Lu" , Paul Walmsley , Palmer Dabbelt , Albert Ou , Florian Weimer , Christian Brauner , Thiago Jung Bauermann , Ross Burton , Yury Khrustalev , Wilco Dijkstra , linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Mark Brown X-Mailer: b4 0.15-dev-37811 X-Developer-Signature: v=1; a=openpgp-sha256; l=3630; i=broonie@kernel.org; h=from:subject:message-id; bh=is2zgCRaR9rKZObHv5ljdEWRzYPoLx4wqWI+Auj5mNM=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBmxpEy7glYh5KZh88eOmHhzQChA3uW5DjxuojHjDOM f7/CBWOJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCZsaRMgAKCRAk1otyXVSH0BuQB/ 0VJlVzYf3tUrImZPBIFuEnpfaDF1wEv+1OaLwNM/WrMeioCAmQIx9ArEHaxWsU7S7F5s35RqrOaIue hnN1Me447fn9MfXIwJQOlrXluKT1yGT9eSMeQY4JFwMDDkHWeE6ZGfxho4NJUNUQY39AEyHRtx2PbY 8bLiwr9HkEhKIdeBH/E0jrg4mdutWTPYKWe3LdVOhqfzS5eCsZi0Budp7G4Jj9hCDmo4reIOYoXtLX hhRqX/hlSkbA6cNo4xUi9zwvEQ3kaDQL4aBBd8y9j59d7OQ41bZztuhKXt0FLMCfESuqXu+x2W8AP/ MPQfiDAvQjzPk9KyFgwnF29rmKqo++ X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 58D7A12000A X-Stat-Signature: 9kmurs4rfr4zen3gzkmeupzgsc9f3wta X-Rspam-User: X-HE-Tag: 1724609208-842973 X-HE-Meta: U2FsdGVkX18h0MCmZ4/FO2ClB90FDU4yG1TuY5Y6b9qHrvX2BIfKbsuPc7cmCVodiWc/5hZS36nxxwr4pboIWSGmcMKgGsmKFEat5r6RJdMn8rfROZfsd6xZ58p5M2JOowM3rGEzlClGgLsASqF7UWUwHCBjVRbSgEYp/G6+DeqFAjltHoEWGgkfb8f07w1JpWsX06G7Tfh6LLx4Sza2hlEwDD7T/jFk7dvRlMkUTJIQiIh9PevyJ6FAojAahuljpXuh0andU7FlymLrnqjfI4PfKFPaLA7K1+0rYUKSALp5zvRFFb5V6Cwx5KWaz3j4Tl1rbERpZtF6MIQi8sYK55/g+yeuuyb77npcVrtvlQYYimult6PMQ9c4rg1jW1ZnoKTuKbOSO33PDvh7s8sD84FNEWpnXiGcUm3lyMRemMJLSZgJHPcB5Ir5tdX7R9iO1iN5VeQOn2EgE7hjB69j+NaVd5IBbf2QwY4XXUFOrRCQTxPhG5hECYhKFgMVrK8X6NewpoJVgVzjO4sDZcQzmeFq7ntAoPtIFyqyDL6w2/QiKSqR0vUxRzEomcGNZAS21M1ValdnY/S6PqzIp9RzDsL+yVXaxRM7kAs4qL740q2F6Y109XRs5/xj0G9dVXW7vyiqhkzwaGJ+1ucdB6WhYOK4VCLimMTPh6lFpyLiM0mCcLbe3kBl287raoJnE/dMgcYuiiHq7GL/jQQ9ltaHgkRSV8aqCjhWi6kLBoxHG0KwXvGJnmLobWB6oIfJ37Pq9YvWqgit7OBMsgYU03+SZQGtWQXwgDxFU3zz3pjfPdWbJA35q6V4Hn/2Gq5u5PXxRArwQc366uIs6kTS6QdGTl4TXxPkLZmW8ecfIjKVkFXBPO8TPj3QI5CZKPBQXDOgoLI6pL9f5ROVuw5NhC/azJkHE8vN4GCS36/OPTL5b7W5TFssQFtUWy2uoxHbtr3jrd333wJCy+sszopW0BE uymVLtSE jbdcmZiz/hA385BoLYKKIL7PyAL9Wkdat81n7QX8bY1G4GC71BdIaQszswagc+sHGlq9mD8R60eeFzASZPItZDmTB9mClFc4dIydFKT/WXvF3JyVfm+2S6ad8KuuVr7xmzIwSxVtxSnaWVaidTPBYUFLSHTHeT7OengXoG3dx+YrIywSkV1fZcztL72CK0Wwa4WWLOxS1+htbeq33KDY8EIIb0NNSyZX+H162blPEyOHEbSdyplJH2snDaohZGOM4IbP3/cQaCH31NkHSCo6QEjJbN5dWNN25/a7bsZ/epuIMKNTmQ/cQrttUSlhfDtuxh07nvDfYAfuzNZz+aDUfnGlhZBBDqQRlmOweerIdYh3+9KbdJfl7wrBZfwfY7OU+NnUi4KWvN/dqpTBtduU6PZQEuCGcHWJ0N0JdERFd/o05p9I= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: All GCS operations at EL0 must happen on a page which is marked as having UnprivGCS access, including read operations. If a GCS operation attempts to access a page without this then it will generate a data abort with the GCS bit set in ESR_EL1.ISS2. EL0 may validly generate such faults, for example due to copy on write which will cause the GCS data to be stored in a read only page with no GCS permissions until the actual copy happens. Since UnprivGCS allows both reads and writes to the GCS (though only through GCS operations) we need to ensure that the memory management subsystem handles GCS accesses as writes at all times. Do this by adding FAULT_FLAG_WRITE to any GCS page faults, adding handling to ensure that invalid cases are identfied as such early so the memory management core does not think they will succeed. The core cannot distinguish between VMAs which are generally writeable and VMAs which are only writeable through GCS operations. EL1 may validly write to EL0 GCS for management purposes (eg, while initialising with cap tokens). We also report any GCS faults in VMAs not marked as part of a GCS as access violations, causing a fault to be delivered to userspace if it attempts to do GCS operations outside a GCS. Reviewed-by: Thiago Jung Bauermann Signed-off-by: Mark Brown --- arch/arm64/mm/fault.c | 40 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 451ba7cbd5ad..3ada31c2ac12 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -486,6 +486,14 @@ static void do_bad_area(unsigned long far, unsigned long esr, } } +static bool is_gcs_fault(unsigned long esr) +{ + if (!esr_is_data_abort(esr)) + return false; + + return ESR_ELx_ISS2(esr) & ESR_ELx_GCS; +} + static bool is_el0_instruction_abort(unsigned long esr) { return ESR_ELx_EC(esr) == ESR_ELx_EC_IABT_LOW; @@ -500,6 +508,23 @@ static bool is_write_abort(unsigned long esr) return (esr & ESR_ELx_WNR) && !(esr & ESR_ELx_CM); } +static bool is_invalid_gcs_access(struct vm_area_struct *vma, u64 esr) +{ + if (!system_supports_gcs()) + return false; + + if (unlikely(is_gcs_fault(esr))) { + /* GCS accesses must be performed on a GCS page */ + if (!(vma->vm_flags & VM_SHADOW_STACK)) + return true; + } else if (unlikely(vma->vm_flags & VM_SHADOW_STACK)) { + /* Only GCS operations can write to a GCS page */ + return is_write_abort(esr); + } + + return false; +} + static int __kprobes do_page_fault(unsigned long far, unsigned long esr, struct pt_regs *regs) { @@ -535,6 +560,14 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, /* It was exec fault */ vm_flags = VM_EXEC; mm_flags |= FAULT_FLAG_INSTRUCTION; + } else if (is_gcs_fault(esr)) { + /* + * The GCS permission on a page implies both read and + * write so always handle any GCS fault as a write fault, + * we need to trigger CoW even for GCS reads. + */ + vm_flags = VM_WRITE; + mm_flags |= FAULT_FLAG_WRITE; } else if (is_write_abort(esr)) { /* It was write fault */ vm_flags = VM_WRITE; @@ -568,6 +601,13 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, if (!vma) goto lock_mmap; + if (is_invalid_gcs_access(vma, esr)) { + vma_end_read(vma); + fault = 0; + si_code = SEGV_ACCERR; + goto bad_area; + } + if (!(vma->vm_flags & vm_flags)) { vma_end_read(vma); fault = 0;