From patchwork Tue Oct 1 22:58:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 13818920 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ABB35CF318A for ; Tue, 1 Oct 2024 23:03:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3A1B944015F; Tue, 1 Oct 2024 19:03:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 351C968002B; Tue, 1 Oct 2024 19:03:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1A57544015F; Tue, 1 Oct 2024 19:03:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id EC2B268002B for ; Tue, 1 Oct 2024 19:03:41 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id A940D1A1319 for ; Tue, 1 Oct 2024 23:03:41 +0000 (UTC) X-FDA: 82626562242.13.EEE5926 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf21.hostedemail.com (Postfix) with ESMTP id E31A81C0011 for ; Tue, 1 Oct 2024 23:03:39 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=h5QOvsyH; spf=pass (imf21.hostedemail.com: domain of broonie@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=broonie@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1727823692; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Or9D4svYoGrxBoA4BHrqTjTxUH2w3U+9VPV+mFsrKvw=; b=pfETi1YWIxdZGXPwbTg2VMFjUJPTMy/vRyaYR2amfQJJK+7LLV10jl/5IM8B1DXsxVsTum olC5KXu76Peeah1p7CVFcAm6AFBShijK8NZ/GH1VOv8QvZ0WhuO8hSK16GFVosWcI4yxXT x4ELZWFgWe9VNTkQemCi6Bo6cSHPVl4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1727823692; a=rsa-sha256; cv=none; b=V1dMAufXxpmPso1Z26C4SPq/CamN8vyzRxj68gh6LudI39eYlR5C9Pby1caTBWTNpCha8h IZUv0YR/Zak+to6JAmxTwQj3GaWJB+O/GaJYwTHkirCk4F0sVFeMkN8xYAkLoTVgxlBncH 7v7Jr6tMjdCnb133esAaz0vmWB5uXN4= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=h5QOvsyH; spf=pass (imf21.hostedemail.com: domain of broonie@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=broonie@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 220F55C1060; Tue, 1 Oct 2024 23:03:35 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B91A5C4CEC6; Tue, 1 Oct 2024 23:03:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1727823818; bh=JwTS5J5Kz+XCxYDRc3FG3N1klYZC99PAyz93oxxWHns=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=h5QOvsyHwmwYJXTikpeXsVEp2A3UPoVjGZRUucHDnIEP8s20RwqCtfvtaEgYASJ3A 4VvT7Z9FkWpjGLRRUyAM0E57+nFfV/1smmkjKGkRZKQARGtcXv4Kr/B1Fg5EqqXPTT w7iMpC+tcgCrK+qsiDR7XnELMA4PkJgDckI9vpw8SVgbifNvpYEuLSdt70HOnbcLrh bILOGqSFtZOKo1haHLpTrtPKpk0NC0HS0Pd2sgi3K8cPhb+7HZxIGLJOcB3QJHfl0A Gos56B/AegT6wsD2YwKq8Ie1aAdRX49w3baVX3709fSna41T6qIRdgrbFb2IW97c4V 0q6PzQ7x+JAPg== From: Mark Brown Date: Tue, 01 Oct 2024 23:58:59 +0100 Subject: [PATCH v13 20/40] arm64/mm: Handle GCS data aborts MIME-Version: 1.0 Message-Id: <20241001-arm64-gcs-v13-20-222b78d87eee@kernel.org> References: <20241001-arm64-gcs-v13-0-222b78d87eee@kernel.org> In-Reply-To: <20241001-arm64-gcs-v13-0-222b78d87eee@kernel.org> To: Catalin Marinas , Will Deacon , Jonathan Corbet , Andrew Morton , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Arnd Bergmann , Oleg Nesterov , Eric Biederman , Shuah Khan , "Rick P. Edgecombe" , Deepak Gupta , Ard Biesheuvel , Szabolcs Nagy , Kees Cook Cc: "H.J. Lu" , Paul Walmsley , Palmer Dabbelt , Albert Ou , Florian Weimer , Christian Brauner , Thiago Jung Bauermann , Ross Burton , David Spickett , Yury Khrustalev , Wilco Dijkstra , linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Mark Brown X-Mailer: b4 0.15-dev-99b12 X-Developer-Signature: v=1; a=openpgp-sha256; l=3733; i=broonie@kernel.org; h=from:subject:message-id; bh=JwTS5J5Kz+XCxYDRc3FG3N1klYZC99PAyz93oxxWHns=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBm/H7TVZ3jDKeIyJFS2xkZycVzFdIgkZ/2MVMSxegg PracWFmJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCZvx+0wAKCRAk1otyXVSH0PPFB/ 9jOVBZ0sZYLFYt8CYSgFwth5bQAMffCYR65UIbAKTDAZV8czeEm1xDTBVqLzh2y8cJ8L9rksWGRzsa KoNNdusUYbP0WwOzwiB8f26x4k5gKBP8dzKhvHobyjoVwM9dwHJ1P+31Rnk9S2VC0HlJ12ukGOll7A CZczYl0vb1teCaug4ldvi8DJRkAAMotUN8oeVWz1J0E9GwHb+eMDvTL9igdY9wrw4EyL8kkGp2K8aF B+pBMS0HXzUFSpXDbnSX5eVKfGA1myQSsEM0KQsGSA4aaHTZ8Guexvn78tQkU7ofyEMejYY/2DYTmb IZnC/hzjnKbZohafBaYTyRcPVaFiYq X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: E31A81C0011 X-Stat-Signature: u7jdaebg9po75fpxzrdngxembeyngfch X-HE-Tag: 1727823819-386737 X-HE-Meta: U2FsdGVkX1/HI/S5Lp6fF8oqmTQfRW+92MrMYTyJr6odYZt9FiG5sZWncCDSOgop45ARTYO5AJJS/VMxVMn8r6fmYxF8Ccye4Y14ElgtQgn49c4HDhJmvNmH3f5ILPxFj5c2jRM5yr7a48pITOpo+5I6hTDmXntmRHG/Zzs90mQ1AH2zaxadn5JZgkqjnyNX3Nzz9iuLHTTkYpDyytZUc9sYx/gvlO7PBHSaQdKC7EMJAaxgldEnXm3Hr2Ua/U1ylN/fd+kYtCKHYwspBTPd5qeJ3WvtYWC9Upwo7Vexzmo5Tl6agvrtqautSCkeGrItvuGBODfUw8FcHOuzyoSIAurNEmosSVZOcqVWjWtq2Wk665MMpqlTZvg77BD3MUBQcDpKdNbtau+eHOtpEexk1dj+KHvP12VD6ChhSAWsevjO0v9mSRz3nm4ZOzbUbcK38Jr+7mRGUk1/0XVF26azoJl8YOEiHYDx0bAVE9JREwV+4qmhU8DSelh2KIv8yE9T4uWUWQo0FpOPlH7/K2W7+e1dUBdRLd+yw6kxuc6bGGh2ce6Pi/m67tXJnt5CooW7QY04Q8j5lrD2ZZgksl3phF8IsJoR1G7UducjhuWdANUUxWn8BFHHFNgbNx8g3GSdu6+FaPhFi1ag2SYFkFrQzRpOgtTUTe9zOm1zsHcT0i05/LyOIGXLa5430ICmHCiJMBGp6YEe46u5wlXtg8W1bxcmmYg3SiZMjK21guun3O0SJmnnMs2TqR7wJFQV7Hnt4ygIhQZLZ5VJeGM8cxKVuG1yJIAf7kzV6PMfwTP/GDtd+OKLkjBcnwkZ0XWvv/5OtMzRSOGzPqBQ/xRejDF+bu/LiQy3B84LQRiMSHC5RKjOH8dk+9KB2vKPHjrwIGGt3ZwChd8l18dRLc8HEoa/MwQMV+dExXFNwZAaR+MrX7WSPKxQaK7xntqC9M+XnRZ5v4cGm93Xe/0AHX2uM05 MyR/YZ8b iS/MpaMi7Dl2SB8xClC+EqAtdWDnVuLBGTgUVlMr3yB2DlVASrjWLRILeRrXL0wXhFjc3MKBTdHq1K0YMioUp/cV06uIZ43FoDw28ePKwcIyGQWdlD5kuVrZFBQ/zAAsLkMJ2kM9brN7mBogUyiRUACoaq+//Sk64pWbr8nDTG59UwvRHsJv5e6KTaxR34Ny9nHUx/XkwpXmtOjOKWl48dM6x5qYm+LbYMHGwTVkJuXUnaF74bf1VkzKCdawRTVFtPTUu2jHn4RFCAZ1seWqg8glY8wtIdDSQz5Afub0pztnOalSZrxGuWmnRteesEFWyMLyk0klWe19WhjjKptJ5DpnkteSPGurLyBxwmNEs0Xs0uHuUF7gV3U2N23wxbvYvAr8D2i4yzzpfdE+XbTeYB4jcEgjZAaACSyW392qDsXf0Vy2n7Kzcr4v8rQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: All GCS operations at EL0 must happen on a page which is marked as having UnprivGCS access, including read operations. If a GCS operation attempts to access a page without this then it will generate a data abort with the GCS bit set in ESR_EL1.ISS2. EL0 may validly generate such faults, for example due to copy on write which will cause the GCS data to be stored in a read only page with no GCS permissions until the actual copy happens. Since UnprivGCS allows both reads and writes to the GCS (though only through GCS operations) we need to ensure that the memory management subsystem handles GCS accesses as writes at all times. Do this by adding FAULT_FLAG_WRITE to any GCS page faults, adding handling to ensure that invalid cases are identfied as such early so the memory management core does not think they will succeed. The core cannot distinguish between VMAs which are generally writeable and VMAs which are only writeable through GCS operations. EL1 may validly write to EL0 GCS for management purposes (eg, while initialising with cap tokens). We also report any GCS faults in VMAs not marked as part of a GCS as access violations, causing a fault to be delivered to userspace if it attempts to do GCS operations outside a GCS. Reviewed-by: Thiago Jung Bauermann Reviewed-by: Catalin Marinas Signed-off-by: Mark Brown --- arch/arm64/mm/fault.c | 40 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 8b281cf308b3..c2f89a678ac0 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -504,6 +504,14 @@ static bool fault_from_pkey(unsigned long esr, struct vm_area_struct *vma, false); } +static bool is_gcs_fault(unsigned long esr) +{ + if (!esr_is_data_abort(esr)) + return false; + + return ESR_ELx_ISS2(esr) & ESR_ELx_GCS; +} + static bool is_el0_instruction_abort(unsigned long esr) { return ESR_ELx_EC(esr) == ESR_ELx_EC_IABT_LOW; @@ -518,6 +526,23 @@ static bool is_write_abort(unsigned long esr) return (esr & ESR_ELx_WNR) && !(esr & ESR_ELx_CM); } +static bool is_invalid_gcs_access(struct vm_area_struct *vma, u64 esr) +{ + if (!system_supports_gcs()) + return false; + + if (unlikely(is_gcs_fault(esr))) { + /* GCS accesses must be performed on a GCS page */ + if (!(vma->vm_flags & VM_SHADOW_STACK)) + return true; + } else if (unlikely(vma->vm_flags & VM_SHADOW_STACK)) { + /* Only GCS operations can write to a GCS page */ + return esr_is_data_abort(esr) && is_write_abort(esr); + } + + return false; +} + static int __kprobes do_page_fault(unsigned long far, unsigned long esr, struct pt_regs *regs) { @@ -554,6 +579,14 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, /* It was exec fault */ vm_flags = VM_EXEC; mm_flags |= FAULT_FLAG_INSTRUCTION; + } else if (is_gcs_fault(esr)) { + /* + * The GCS permission on a page implies both read and + * write so always handle any GCS fault as a write fault, + * we need to trigger CoW even for GCS reads. + */ + vm_flags = VM_WRITE; + mm_flags |= FAULT_FLAG_WRITE; } else if (is_write_abort(esr)) { /* It was write fault */ vm_flags = VM_WRITE; @@ -587,6 +620,13 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, if (!vma) goto lock_mmap; + if (is_invalid_gcs_access(vma, esr)) { + vma_end_read(vma); + fault = 0; + si_code = SEGV_ACCERR; + goto bad_area; + } + if (!(vma->vm_flags & vm_flags)) { vma_end_read(vma); fault = 0;