From patchwork Tue Jun 25 14:57:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 13711442 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D61BCC2BBCA for ; Tue, 25 Jun 2024 15:02:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 66C246B00AB; Tue, 25 Jun 2024 11:02:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5F73A6B00AD; Tue, 25 Jun 2024 11:02:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 446F56B00AE; Tue, 25 Jun 2024 11:02:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 1F52D6B00AB for ; Tue, 25 Jun 2024 11:02:55 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id CE5F11A01EC for ; Tue, 25 Jun 2024 15:02:54 +0000 (UTC) X-FDA: 82269728268.24.CA25292 Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by imf16.hostedemail.com (Postfix) with ESMTP id 418AF18005A for ; Tue, 25 Jun 2024 15:02:50 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b="NqmuhU/F"; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf16.hostedemail.com: domain of broonie@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=broonie@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719327755; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=neNjxYSK5RinDqINCqqALmwc8YSiZ7aA4MfY4tBDfkM=; b=19/bHv1NIWl/2DHrgm/gvig+KRjkmcaJtq+cAnQzh6cWAihcRqaA/jxAcbYLSgWPnSsRbH MPzfiQATWbwXTBOP2sD1REf2eH6Y7/6T1W7Y1RKzPoeCYqSQxvdImpfGdZ7EZlqLCWZXj2 CSNwKfuB9ST1JCesgWwEWdP0C3IZPCU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1719327755; a=rsa-sha256; cv=none; b=QsNvgujGQV0BZx3TYqG0dhdi0L6eIXZq58NTytQTtnEv8dxQJH4KwqVFi8i77tkdRmVI32 ++kbR3q7a6XlTm0e2wCaGo+sa740lR0zVtK4qJqOxld1seEbIwUOlxP0lKcBpKmWbb4J9u CK49wHpQugOJR335Wz/XZV1GhFg+Yyg= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b="NqmuhU/F"; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf16.hostedemail.com: domain of broonie@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=broonie@kernel.org Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id 4DBDACE09F8; Tue, 25 Jun 2024 15:02:48 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0DDB4C32782; Tue, 25 Jun 2024 15:02:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1719327767; bh=Kh+IETSPL7qtolQq3lnlhkzFZ1gTXcIg84ZJjg4GwDo=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=NqmuhU/Ff0h3LtISDpIR4LELNEW47+ErWU3dX0Kgb2WwBpg/UNPMZk/3pQ0PYB1VI r/Sk2xkGmNKtOI/q8U2CgxZfhnnEM7SbHdW6fD9QDbpvh8GryRUlCOEwblVuQislXS UMhR4JP8WmDWmFbOAPfVkDy3agDNMQ1EjqGGBc6b767cwqVtcNWutiFFFJhGEz3fU0 CKb3bK8zw9PtvzVDEpX94l+RGWbaYZkGsx+H35ODgE8LNeBd7SZN9OxBfAMa72WOoc 14vmisyaBHOWWM1XshDHqpHBiAEi+387qIU1euy1VJrilb7MSQekyQzo0Ak+tu/xDi TWFJndJ8VyW4w== From: Mark Brown Date: Tue, 25 Jun 2024 15:57:46 +0100 Subject: [PATCH v9 18/39] arm64/mm: Handle GCS data aborts MIME-Version: 1.0 Message-Id: <20240625-arm64-gcs-v9-18-0f634469b8f0@kernel.org> References: <20240625-arm64-gcs-v9-0-0f634469b8f0@kernel.org> In-Reply-To: <20240625-arm64-gcs-v9-0-0f634469b8f0@kernel.org> To: Catalin Marinas , Will Deacon , Jonathan Corbet , Andrew Morton , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Arnd Bergmann , Oleg Nesterov , Eric Biederman , Shuah Khan , "Rick P. Edgecombe" , Deepak Gupta , Ard Biesheuvel , Szabolcs Nagy , Kees Cook Cc: "H.J. Lu" , Paul Walmsley , Palmer Dabbelt , Albert Ou , Florian Weimer , Christian Brauner , Thiago Jung Bauermann , Ross Burton , linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Mark Brown X-Mailer: b4 0.14-dev-d4707 X-Developer-Signature: v=1; a=openpgp-sha256; l=3651; i=broonie@kernel.org; h=from:subject:message-id; bh=Kh+IETSPL7qtolQq3lnlhkzFZ1gTXcIg84ZJjg4GwDo=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBmetuAlhPwwg74pRVVM89hL0m4VNM6ITOzanR6CdNZ sXVkSpCJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCZnrbgAAKCRAk1otyXVSH0EpYB/ 9DrxYdrSx1M6UsWZ8hhevoQ6LhgfMS1f0ctPjflnEYxAuXwqOT+eZL/xrSaLvCcaVqiQz7y+sS5b5K 3aQXMrDN2LkRkJbOxwsA89H6WTBhFvNPu0hfZiAAO2P6wEeEtdIs/efj9Jif5lxTx+4F75eu3qHJ7v 9I7XDppZFUhJS9qA/xcfwrgR9YydpToq5a/cS2H7r3aGSYe6+NkWAIsyObTDjgxANhadUN06ef2rjx A8f/hqcVj+Wlm3QLrLdhjfHiy3woa+yVkGfxH1peBijH7Nu1oW6NYBqXizTNPkLbv7F3077Y/SUzPJ yCsShRq/pkkfBsZeVJieKkZPSga30a X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 418AF18005A X-Stat-Signature: yawqs55txwkubdasckaut4jeghrkcs86 X-Rspam-User: X-HE-Tag: 1719327770-729258 X-HE-Meta: U2FsdGVkX1/Bq2f+tLtNBA7z6m8Du3eqBmKe8E57E9ye3/J90YkWc8Tn6XAujV7JxofVclwXxVyRuOL+7Zr75WIL9P9G6Y7we3vrjFENacD2KVmLeRUGpzqD3Ldf/sX6nL5SWlXc1BAH+iVniBz/cfWuWIqWiDtSHMRGdZteqGpv9EK4iLK6NRXc3Hutd6IJEZqbbluYL7yFZq2dESzEY3fzqOkgo6rLLEWaHEHs0zWjudU6m4VLgREpoTWTWwhSkT6oGF5HP+GPuOCSiqYE32mpIxfsE02IG3ioDP2QDw4W0aYXiSQ7Tcy40hPvGMUxJxkK7Ka/N+191Uo/oKvCnONTdTjNvCL1VzAlBsC1k3NUPuweqij7ovJFXZfzAZcg3PL/ktzsxdE0bmM+aqto8d/2BVQqXSi+AOGH2ultR5qETuZ2NLzKwWt0i96FqUHjPj0Ytc3IrkItr/ktHkKgqdpQqmYH2yH4/XjpOgeFI/h8qV/WMzBGw92LQp2li1WM+ZL72Y3PZHiqUiDDK3OYyRGbMbXXsPfYcoafk2oA3s07xn4EOS1twRM3HZErf5MB5xHaGwFvwjGKQKZmpq6YcD4dXjqxBVAmKjAJAtdAjxaaFsm24arcqZIHUsTgiHvn4liFEhI3yj8Wvw6ORzhEbSOmurFLZqso1Sly+pnturc4KcvUFbFrhtzD8wht6mp4FHDt60q8jIp3VFFHIgIbahA7qrCmm3zjA7h0zrE8RR95jC2z8RNaBcUiHRtJk3tZtFPU353+vhyTkIwNzEHb4i5ILJEoBgbx9cKaj93PD/9FzirhHkoQKwhK+HN9vvg+QMV6YHbqSAgy6aLAoigLxCNEzREY7vyY1pegl9winAevrv5CmYokIrX+SywiY3V3GE4o0q+t4CzQ1CFsU3vtYwSuopuSugYtS6quyqti6wzG5xKrLk5s6oDvwzjaF+vvZUnXMpxozEIdRzE/U1/ jGmWPUfN mO3a1Bh3xMbHSuYZouIlKNH4UngB93mXGvL9ABDUQ2ZOgME+69PX3bBAPCGzHkXeQAekALI6LlpypqFM26EJRQZ2QggzEUfsSwrDtHCghHVFNTn/XLSitEu32Ad8VziaU15SVwZAawml+x6XB6bgO8ZpnCO6Fe2u5ZhMLa1OGIq4mFgaQ6u1ln7BgGxVHtJs4cCtoxgC+IURdfNZngA2bb3okmnMY5oYj/entQc8cPZBIAZ5Zp2z8/Fvgl7F6RJa4m6tLVLXfvoDvsSNqO8M+jrbwtz9s4nFNLsd5DCGwhenJZ5DZIywIMrYnlhFumGspSxOB X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: All GCS operations at EL0 must happen on a page which is marked as having UnprivGCS access, including read operations. If a GCS operation attempts to access a page without this then it will generate a data abort with the GCS bit set in ESR_EL1.ISS2. EL0 may validly generate such faults, for example due to copy on write which will cause the GCS data to be stored in a read only page with no GCS permissions until the actual copy happens. Since UnprivGCS allows both reads and writes to the GCS (though only through GCS operations) we need to ensure that the memory management subsystem handles GCS accesses as writes at all times. Do this by adding FAULT_FLAG_WRITE to any GCS page faults, adding handling to ensure that invalid cases are identfied as such early so the memory management core does not think they will succeed. The core cannot distinguish between VMAs which are generally writeable and VMAs which are only writeable through GCS operations. EL1 may validly write to EL0 GCS for management purposes (eg, while initialising with cap tokens). We also report any GCS faults in VMAs not marked as part of a GCS as access violations, causing a fault to be delivered to userspace if it attempts to do GCS operations outside a GCS. Signed-off-by: Mark Brown --- arch/arm64/mm/fault.c | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 451ba7cbd5ad..bdc28588163d 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -486,6 +486,14 @@ static void do_bad_area(unsigned long far, unsigned long esr, } } +static bool is_gcs_fault(unsigned long esr) +{ + if (!esr_is_data_abort(esr)) + return false; + + return ESR_ELx_ISS2(esr) & ESR_ELx_GCS; +} + static bool is_el0_instruction_abort(unsigned long esr) { return ESR_ELx_EC(esr) == ESR_ELx_EC_IABT_LOW; @@ -500,6 +508,25 @@ static bool is_write_abort(unsigned long esr) return (esr & ESR_ELx_WNR) && !(esr & ESR_ELx_CM); } +static bool is_invalid_gcs_access(struct vm_area_struct *vma, u64 esr) +{ + if (!system_supports_gcs()) + return false; + + if (unlikely(is_gcs_fault(esr))) { + /* GCS accesses must be performed on a GCS page */ + if (!(vma->vm_flags & VM_SHADOW_STACK)) + return true; + if (!(vma->vm_flags & VM_WRITE)) + return true; + } else if (unlikely(vma->vm_flags & VM_SHADOW_STACK)) { + /* Only GCS operations can write to a GCS page */ + return is_write_abort(esr); + } + + return false; +} + static int __kprobes do_page_fault(unsigned long far, unsigned long esr, struct pt_regs *regs) { @@ -535,6 +562,14 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, /* It was exec fault */ vm_flags = VM_EXEC; mm_flags |= FAULT_FLAG_INSTRUCTION; + } else if (is_gcs_fault(esr)) { + /* + * The GCS permission on a page implies both read and + * write so always handle any GCS fault as a write fault, + * we need to trigger CoW even for GCS reads. + */ + vm_flags = VM_WRITE; + mm_flags |= FAULT_FLAG_WRITE; } else if (is_write_abort(esr)) { /* It was write fault */ vm_flags = VM_WRITE; @@ -568,6 +603,14 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, if (!vma) goto lock_mmap; + if (is_invalid_gcs_access(vma, esr)) { + pr_crit("INVALID GCS\n"); + vma_end_read(vma); + fault = 0; + si_code = SEGV_CPERR; + goto bad_area; + } + if (!(vma->vm_flags & vm_flags)) { vma_end_read(vma); fault = 0;